Event-Driven Service Design: Five Steps from Event Storming to OpenAPI and Camel Flow

Event storming workshops yield domain events and commands (among other artifacts). This post proposes a continuous refinement process to derive an API design and an executable integration flow from such event-command chains. Its five steps are tool-supported.

The post explains the essentials required to run the tool demo steps. It also motivates the steps to some extent, providing some process design rationale. Deeper background information is referenced.

Prerequisites: The instructions below assume that you have Context Mapper Version 6.6 and MDSL Tools Version 5.4 (just released!) installed, as well as access to a miro2cml deployment (as the online whiteboard platform miro serves as an example in Step 1). That said, intermediate results in each step are provided so that tool usage may be skipped.

Step 1: Sketch Commands and Events and Transfer them to Domain Model

Event storming results can be captured in drawing tools, for instance an online collaboration service such as miro. An exemplary event storming from the research domain is:

You can draw the above board yourself (or upload it from this backup). Next, please install miro2cml, a Spring Boot Web application, by cloning the repository https://github.com/socadk/miro2cml-generator and then following the build/run instructions in its ReadMe.md. Alternatively, a hosted version of the tool is available. In both cases you have to authorize with miro and specify a miro team for the miro2cml app.

Once miro2cml is up and running, try to find the board in its home screen (hint: use “ReferenceManagementDemo” in the board filter if you have uploaded the backup). Select “EventStorming” as “Miro Board Type” in the dropdown at the top of the page and then hit “Map Board”. Review the log messages and the mapping result, which looks as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
BoundedContext EventStormingBoardContext {
  Application EventStormingBoardContextApplication {
    CommandEvent Write_Paper
	CommandEvent Review_Paper
	CommandEvent Review_Feedback
	CommandEvent Accept_Paper
	CommandEvent Reject_Paper
	CommandEvent Publish_Authors_Copy
	Flow Write_Paper {
	  command Write_Paper [ initiated by "Researcher" ] delegates to Paper emits event Paper_Submitted_for_Review
	  event Paper_Submitted_for_Review triggers command Review_Paper
	  command Review_Feedback [ initiated by "Researcher" ] delegates to Paper emits event Paper_Will_Be_Revised
	  event Paper_Will_Be_Revised triggers command Write_Paper
	  command Review_Paper [ initiated by "Review_Committee" ] delegates to Paper emits event Acceptance_Decision_Made
	  event Acceptance_Decision_Made triggers command Accept_Paper X Reject_Paper
	  command Reject_Paper [ initiated by "Review_Committee" ] delegates to Review_Feedback emits event Feedback_Sent
	  event Feedback_Sent triggers command Review_Feedback
	  command Accept_Paper [ initiated by "Review_Committee" ] delegates to Conference_Program emits event Proceedings_Available
	  event Proceedings_Available triggers command Publish_Authors_Copy
	  command Publish_Authors_Copy [ initiated by "Researcher" ] delegates to Paper_Collection emits event Paper_Available_Online
	}
  }
  Aggregate Paper {
	Event Paper_Submitted_for_Review
	Event Paper_Will_Be_Revised
	Event Acceptance_Decision_Made
  }
  Aggregate Review_Feedback {
 	Event Feedback_Sent
  }
  Aggregate Conference_Program {
	Event Proceedings_Available
}
  Aggregate Paper_Collection {
	Event Paper_Available_Online
  }
}

The blue cards have been turned into Command Events (which is a term from Context Mapper and Sculptor, the Domain-Driven Design (DDD) tools we will use in the next step). The orange cards became Domain Events (a DDD pattern) wrapped in draft Aggregates (another DDD pattern), corresponding to the green cards in the source board. The control flow is preserved; in our example, it contains a choice and a loop (with still undefined selection and termination conditions). The user personas were transferred to (“initiated by ...”).

Note: miro2cml is the result of a student term project (see acknowledgements below). View it as an experimental technology preview that shows what is possible, but does not claim to be production ready. For instance, not each and every miro board layout and widget type is mapped at present; the online tutorial that comes with the tool documents expected input, performed mappings and known limitations.

For the next steps, copy-paste this mapping output or download it as a Context Mapper Language (CML) file as proposed by miro2cml; it is available here as well. The syntax and semantics of the application layer concepts in CML are explained on the Context Mapper website.

Step 2: Extract BPMN Process Flow from Domain Model

Context Mapper can turn the Step 1 model into story input for BPMN Sketch Miner, an in-browser tool that transforms textual process stories into visual sketches of Business Process Model and Notation (BPMN) specifications.

Open the downloaded .cml file in Context Mapper (for example, the Visual Studio Code extension) and run “Generate Sketch Miner Diagrams” to yield:

1
2
3
4
5
6
7
8
9
10
11
(InitialEvent)
Researcher: service WritePaper
(PaperSubmittedforReview)
Review_Committee: service ReviewPaper
(AcceptanceDecisionMade)
Review_Committee: service AcceptPaper
(ProceedingsAvailable)
Researcher: service PublishAuthorsCopy
(PaperAvailableOnline)

[...]

Process steps, events and their flow order are defined and can be traced back to CML (and the miro whiteboard) easily. When you copy-paste the complete input (available here), BPMN Sketch Miner analyzes its paths and creates a BPMN representation from them:

The BPMN process flow shown above contains command activities and events that mirror the input from previous steps; there is one lane per actor role (or persona) from the miro/CML input.

Not in the mood for copy-pasting? You can also go from Context Mapper to BPMN Sketch Miner directly via a quick fix, a feature explained on the Context Mapper website. Thanks to the link sharing taking place behind the scenes, you can even open the model in the browser by clicking on this link. This is possible because the entire input model is encoded in the URI and hyperlink in the previous sentence (try hovering over it without clicking); client-side JavaScript then takes care of the rendering.

The generated BPMN source is available here. It can be imported into BPMN tools such as Camunda Modeler (and used in other ways).

Step 3: Derive Service Layer with API Endpoints Defined in MDSL

We can do even more with our event storming output: Context Mapper also is able to create an initial (and incomplete) API/service design corresponding to the CML application flow.

In Visual Studio Code or Eclipse, select “Generate MDSL Service Contracts” in the “Context Mapper” menu to yield:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
data type Accept_PaperCommand P // the type Accept_PaperCommand has not been specified
data type Publish_Authors_CopyCommand P // the type Publish_Authors_CopyCommand has not been specified
data type Reject_PaperCommand P // the type Reject_PaperCommand has not been specified
data type Review_FeedbackCommand P // the type Review_FeedbackCommand has not been specified
data type Review_PaperCommand P // the type Review_PaperCommand has not been specified
data type Write_PaperCommand P // the type Write_PaperCommand has not been specified

endpoint type ReferenceManagementDemoContract
  exposes
    operation Write_Paper
      expecting payload Write_PaperCommand
    operation Review_Paper
      expecting payload Review_PaperCommand
    operation Review_Feedback
      expecting payload Review_FeedbackCommand
    operation Accept_Paper
      expecting payload Accept_PaperCommand
    operation Reject_Paper
     expecting payload Reject_PaperCommand
    operation Publish_Authors_Copy
      expecting payload Publish_Authors_CopyCommand

API provider ReferenceManagementDemoContextProvider
	offers ReferenceManagementDemoContextApplication
	at endpoint location "http://localhost:8000"
		via protocol "tbd"

What you see is an excerpt from an API description in the Microservice Domain Specific Language (MDSL); a more complete version of the generated MDSL, purged a bit, is available here. MDSL is a domain-specific language for platform-independent API and service design. MDSL is endpoint-centric and has mappings to the most popular API specification languages and protocols (such as OpenAPI for HTTP resource APIs). Our MDSL Tools support these mappings with generators (see below).

According to the above MDSL specification, each flow command (or process step) is now supported by an API operation (which still has to be implemented). Conceptually, this operation is an instance of the Business Activity Processor variant of the State Transition Operation API design pattern that foresees read and write access to server-side state.

One could flesh out the request and response data now (in CML or MDSL). But in this demo, we continue to work with the incomplete message specifications (expecting payload Write_PaperCommand etc.) and data placeholders (P) that were generated. Information about MDSL data modeling can be found here.

Users of the Eclipse version of the MDSL Tools have to create a simple project so that the generators are able to find or create output folder src-gen.

Step 4: Transform Candidate Service Design to OpenAPI, GraphQL, gRPC and Other Technology-/Platform-Specific API Contract Languages

We are getting closer to our target, but are not there yet. MDSL serves as an intermediate language here — what we are really looking for technology-specific API Descriptions, for instance for HTTP resource APIs.

Note that HTTP resource APIs have a different look and feel than those leveraging other remoting paradigms. For instance, an abstract endpoint with five operations that do not convey any information about their read-write behavior cannot be mapped to a single HTTP URI and resource easily — due to the REST constraint uniform interface. Hence, our MDSL specification from above has to be extended with an explicit HTTP binding that maps the single endpoint types in MDSL to four HTTP resources:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
API provider ReferenceManagementDemoProvider 
  offers ReferenceManagementDemoContract
  at endpoint location "http://localhost:8080"
  via protocol HTTP binding
  resource ResearchResource at "/authoring"
    operation Write_Paper to POST all elements realized as BODY parameters
    operation Review_Feedback to PATCH all elements realized as BODY parameters

  resource ReviewResource at "/reviewing"
    operation Review_Paper to POST all elements realized as BODY parameters

  resource ReviewDecisionCommunicationResource at "/notifying"
    operation Accept_Paper to POST all elements realized as BODY parameters
    operation Reject_Paper to PUT all elements realized as BODY parameters

  resource PublisherResource at "/publishing"
    operation Publish_Authors_Copy to POST all elements realized as BODY parameters

The available MDSL bindings and the HTTP mapping concepts are explained here and here.

The MDSL, available for download, is now ready to be converted to Open API and other formats. The required conversions are available in generators in the MDSL Tools; a separate OOAD/DDD tool demo features this step as Step 7. At present, five types of API specifications can be generated (select “MDSL” and then “Generate …”). The generator output for our example is available for download:

• OpenAPI Specification (OAS)
• gRPC Protocol Buffer
• GraphQL
• Jolie (which in turn can be converted to WSDL/XML Schema)
• ALPS

The MDSL Tools documentation pages explains the performed mapping/conversion in detail, and also demonstrates what can be done with the generated specifications.

Step 5: Turn API Orchestration Specification into Sample Integration Flow (Apache Camel Configuration)

You might have noticed that the MDSL created in Step 2 not only contains an endpoint type for the process (which we just worked with in Steps 3 and 4), but also an orchestration flow:¹

1
2
3
4
5
6
7
8
9
10
11
12
flow Write_Paper type APPLICATION_FLOW
  command Write_PaperCommand emits event Paper_Submitted_for_Review
  event Paper_Submitted_for_Review triggers command Review_PaperCommand
  command Review_FeedbackCommand emits event Paper_Will_Be_Revised
  event Paper_Will_Be_Revised triggers command Write_PaperCommand
  command Review_PaperCommand emits event Acceptance_Decision_Made
  event Acceptance_Decision_Made triggers command Accept_PaperCommand x Reject_PaperCommand
  command Reject_PaperCommand emits event Feedback_Sent
  event Feedback_Sent triggers command Review_FeedbackCommand
  command Accept_PaperCommand emits event Proceedings_Available
  event Proceedings_Available triggers command Publish_Authors_CopyCommand
  command Publish_Authors_CopyCommand emits event Paper_Available_Online

We recognize the events and commands from Step 1 and Step 2; note the exclusive, flow-splitting choice Accept_PaperCommand x Reject_PaperCommand.

Events and command definitions for the flow appear further up in the generated MDSL:

1
2
3
4
5
6
7
8
9
10
11
12
event type Paper_Submitted_for_Review
event type Paper_Will_Be_Revised
event type Acceptance_Decision_Made
event type Feedback_Sent
event type Proceedings_Available
event type Paper_Available_Online
command type Write_PaperCommand
command type Review_PaperCommand
command type Review_FeedbackCommand
command type Accept_PaperCommand
command type Reject_PaperCommand
command type Publish_Authors_CopyCommand

These flow concepts are explained in the MDSL language reference.

A Freemarker template ApacheCamelConfiguration.java.ftl, available for download, allows us to convert this flow into a basic configuration of Apache Camel. If the MDSL tool plugin is installed in Eclipse, the Freemarker template-based generator is available in the “MDSL” context menu: “Generate Text File with Freemarker Template”. Alternatively, the Command Line Interface (CLI) of the MDSL Tools can be used. The following excerpt from the generated, self-contained Java program shows the route configurations:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// routes for command invocation steps
from("direct:Paper_Submitted_for_Review").to("direct:Review_PaperCommand"); // event to command
from("direct:Paper_Will_Be_Revised").to("direct:Revise_PaperCommand"); // event to command
from("direct:Acceptance_Decision_Made").choice() // event to multiple commands 
    .when(simple("${header.Acceptance_Decision_MadeCommandInvocationCondition} == 'choiceOfAccept_PaperCommand'")).to("direct:Accept_PaperCommand") 
    .when(simple("${header.Acceptance_Decision_MadeCommandInvocationCondition} == 'choiceOfReject_PaperCommand'")).to("direct:Reject_PaperCommand")
    .otherwise().to("mock:bye").stop();
from("direct:Feedback_Sent").to("direct:Review_FeedbackCommand"); // event to command
from("direct:Proceedings_Available").to("direct:Publish_Authors_CopyCommand"); // event to command
// note: Paper_Available_Online does not invoke any commands 

// routes for domain event production steps
from("direct:Write_PaperCommand").process(new Write_PaperCommandProcessor("Write_PaperCommand")).to("direct:Paper_Submitted_for_Review"); // command to single event
from("direct:Review_PaperCommand").process(new Review_PaperCommandProcessor("Review_PaperCommand")).to("direct:Acceptance_Decision_Made"); // command to single event
from("direct:Review_FeedbackCommand").process(new Review_FeedbackCommandProcessor("Review_FeedbackCommand")).to("direct:Paper_Will_Be_Revised"); // command to single event
from("direct:Accept_PaperCommand").process(new Accept_PaperCommandProcessor("Accept_PaperCommand")).to("direct:Proceedings_Available"); // command to single event
from("direct:Reject_PaperCommand").process(new Reject_PaperCommandProcessor("Reject_PaperCommand")).to("direct:Feedback_Sent"); // command to single event
from("direct:Publish_Authors_CopyCommand").process(new Publish_Authors_CopyCommandProcessor("Publish_Authors_CopyCommand")).to("direct:Paper_Available_Online"); // command to single event
        
// routes to terminate flow
from("direct:Paper_Available_Online").to("mock:bye").stop();

These route definitions correspond to the flow steps (command, events) from previous steps. For instance, the choice is now called Accept_PaperCommand_Reject_PaperCommand_XORCommand.

A manual patch is required to actually send a message because of the paper revision loop because neither a dedicated init event nor any start command can be identified. Initiate the flow with:

1
template.sendBodyAndHeader("direct:Write_PaperCommand", testMessage, "Acceptance_Decision_MadeCommandInvocationCondition", "choiceOfAccept_PaperCommand");

Note: Do not try the second choice choiceOfReject_PaperCommand, unless you want to observe an endless loop in action.²

The complete generated Java program (which includes a main method) can be found here. Note the inlined helper classes for each command (for instance, Publish_Authors_CopyCommandProcessor) and the sample JoinAggregatorStrategy.

To build and run this program, the maven pom.xml requires this dependency (see this tutorial on Apache Camel for more setup information):

1
2
3
4
5
  <dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-core</artifactId>
    <version>3.12.0</version>
  </dependency>

When we run the program, we can see that some (but not all) of the steps of the original flow from the event storming output and CML application flow (see Steps 1 and 2 above) are executed. The chosen route is determined by the header value that we added to the message, and inlined utility classes such as Write_PaperCommandProcessor print and modify the test message for each executed command:

1
2
3
4
5
6
[...]
Command processor Write_PaperCommandProcessor activated, processing message: Test message for flow Write_Paper in EventStormingBoardContextAPI
Command processor Review_PaperCommandProcessor activated, processing message: Test message for flow Write_Paper in EventStormingBoardContextAPI, processed by Write_PaperCommandProcessor
Command processor Accept_PaperCommandProcessor activated, processing message: Test message for flow Write_Paper in EventStormingBoardContextAPI, processed by Write_PaperCommandProcessor, processed by Review_PaperCommandProcessor
Command processor Publish_Authors_CopyCommandProcessor activated, processing message: Test message for flow Write_Paper in EventStormingBoardContextAPI, processed by Write_PaperCommandProcessor, processed by Review_PaperCommandProcessor, processed by Accept_PaperCommandProcessor
[...]

Note: Please take the provided Freemarker template as a starting point, not as a final product. Consult the Apache Camel documentation to extend and adjust it to your needs and generate a configuration that suits your integration and process management requirements. For instance, you might want to replace the direct: routes with JMS queues or other connectors; you also might want to set some of the many Camel parameters and configuration options.

Wrap Up and Outlook

Summary. This post demonstrated how to progress from event storming output (as an example of a dynamic analysis artifact) to:

1. A BPMN representation of the process flow outlined by the events and commands.
2. A candidate service design in MDSL and its technology-specific API mappings (including OpenAPI).
3. An executable integration flow that comes as a basic Apache Camel configuration.

Five incremental steps were needed to get there, which are supported in research prototypes and open source tools: miro2cml, Context Mapper, BPMN Sketch Miner and MDSL Tools. Some of these tools are quite mature already, other have demonstrator or preview status.

Review and Reflection. Here are some questions you might want to (and be able to) answer once you have run though the above steps:

• How to model AND, OR, XOR conditions in an event storming? How are parallel branches, decision points and corresponding join conditions handled in the transformations?
• What happens to the data that the commands work with, for instance Aggregates, Entities and Value Objects? How are event conditions expressed?
• Can Subdomains and Bounded Contexts (from Domain-Driven Design) be mapped as well?

Such advanced features are not supported yet but opportunities for future work, on projects following the above (or similar) design steps or for continued tool development (contributions welcome!). Plain old xyz³ programming is better suited to address many of these design concerns; the demonstrated tool chain does not intend to generate complete applications — but starting points for rapid prototyping and agile development.

Next Steps. The API and orchestration design is only sketched and rapidly prototyped by now. Next up would be:

• Data modeling, for instance yielding tactic DDD in CML or data contracts in MDSL.
• System decomposition, either (micro-)service cutting or program-internal modularization.
• Process refinements, for instance normal case(s) and error situations (including compensation).
• API and flow implementation, including application state management and backend system integration.
• Automated testing, starting with defining suited tests and test data.

Not that we deliberately excluded explicit architectural decision making for the most part in this post, for instance about architectural style (event-driven architecture? CQRS?), composition paradigm (orchestration vs. choreography?) and integration technologies (HTTP resource API? queue-based messaging?). The Design Practice Repository (DPR) guides you through these (many) remaining human service design tasks. And the workshop paper “Architectural Decision Models as Micro-Methodology for Service-Oriented Analysis and Design” collects and organizes many of the decision points.⁴

Contact me if you like the proposed approach or have questions about it (see links below).

– Olaf (a.k.a. socadk)

Acknowledgements. miro2cml is the result of a term project by Saskia Stillhardt and Timothy Moos at the University of Applied Sciences of Eastern Switzerland (OST). The thesis report (in German) is available here.

The open source project Context Mapper has its roots in Stefan Kapferer’s term projects and master thesis (2018 to 2020) that I supervised at the Institute for Software at OST fka HSR.⁵

BPMN Sketch Miner is a research prototype originally from the Software Institute at USI Lugano led by Cesare Pautasso. It is also available as a Visual Studio Code extension.

MDSL is a language proposal whose tools that benefitted from continued research and development in two projects funded by the Hasler-Foundation (2020, 2021). Both Context Mapper and MDSL Tools could be validated and hardened in an innovation project funded by Innosuisse (2021).

Microservice API Patterns (MAP) is a community effort of Olaf Zimmermann, Mirko Stocker, Daniel Lübke, Uwe Zdun and Cesare Pautasso.

Known Limitations in the Tools. Some of the featured tools have limited functionality:

• Not all event storming elements are processed; as already indicated, automation is not yet implemented or does not make sense for all of them.
• Much richer CML could be generated from miro, for instance input for the system decomposition features of Context Mapper (including its Service Cutter integration).
• The transformations and code generators (for instance, CML to MDSL and MDSL to Apache Camel) do not support all source and target language features yet.

Notes.