Story-Driven Service Design: From Feature Request to Minimum Viable API Product

Goal-oriented API design (“contract-first”) is a recommended practice in the Web API community; data modeling is equally important to balance API granularity and client-provider coupling. Finally, refactoring to API patterns has been researched recently. These three complementary practices form a mini-method whose steps are partially supported by the Microservice Domain-Specific Language (MDSL) Tools.

Prerequisites: The instructions below assume that you have the MDSL Tools Version 5.4 (or higher) installed. Alternatively, experimental MDSL Web Tools can be found online.¹

Step 1: Write Integration Story

We start with a feature request, expressed as user story. The MDSL syntax for stories is similar to the agile format also supported in the Context Mapper Language (CML) for domain-driven design:

1
2
3
4
5
6
7
API description ReferenceManagementSystem

scenario PaperPublishing
  story PaperArchiving 
  a "Researcher" 
  wants to "create" a "PaperItem" with "title" with "authors" with "venue" in a "PaperCollection"
  so that "other researchers can find and cite the referenced paper easily, and the h-index goes up."

Rationale: “APIs should get to the POINT”, and starting with a story makes them purposeful. The linked blog post also lists some of the many books and blogs supporting this claim (under “What do other API designers recommend?”). Note that several variations of the story format exist, but a role-action-benefit triple is quite common.

Step 2: Transform Story to Endpoint Type Exposing Operation(s)

For now, let us assume that we can realize the story with a single API operation. Later on, we will decompose this operation and complement it with supporting ones.

The scenario-level transformation quick fix “Derive endpoint type supporting this scenario” in the MDSL plugin (or, in the MDSL Web Tools, the refactoring option “Add Endpoint with Operations for First Scenario and its First Story”, available on the second tab) analyzes the story and then adds the following candidate endpoint to the API description model:

1
2
3
4
5
6
7
endpoint type PaperPublishingRealizationEndpoint supports scenario PaperPublishing
exposes
operation create with responsibility STATE_CREATION_OPERATION 
  expecting payload {
    "paperItem":D, "title":D, "authors":D, "venue":D, "paperCollection":D
  } 
  delivering payload "data":D<string>

The story action was converted into an initial operation in the endpoint type; the objects from the story definition are available in the request message. The response message is just a stub that we will refine in the next step. Its data types are placeholders, for instance D (which is short for some untyped Data). In the MDSL documentation, the endpoint type syntax is explained in detail under “Service Endpoint Contracts in MDSL”.

Rationale: Adding this initial operation to the endpoint type (a.k.a. service contract) ensures that all that we know so far is transferred from analysis to design. The operation serves as a starting point for the further design and implementation work required to realize the story.

Next, we can add operations via Microservice API Patterns decorators. Two endpoint-level transformation quick fixes are in action here: “Decorate as Information Holder Resource” assigns an architectural role to the endpoint (under “serves as”) and “Add operations common/typical for role stereotypes” provides two retrieval operations. The result is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
data type PaperPublishingRealizationEndpointDTO "paperPublishingRealizationEndpoint":D

endpoint type PaperPublishingRealizationEndpoint supports scenario PaperPublishing
serves as INFORMATION_HOLDER_RESOURCE
exposes
 operation create with responsibility STATE_CREATION_OPERATION 
  expecting payload {
    "paperItem":D, "title":D, "authors":D, "venue":D, "paperCollection":D} 
  delivering payload {"data":D<string>}
  operation findAll with responsibility RETRIEVAL_OPERATION 
    expecting payload "query":{"queryFilter":MD<string>*} 
    delivering payload "result":{"responseDTO":PaperPublishingRealizationEndpointDTO}*
  operation findById with responsibility RETRIEVAL_OPERATION 
    expecting payload "resourceId":ID<int>
    delivering payload "responseDTO":PaperPublishingRealizationEndpointDTO

Note that STATE_CREATION_OPERATION and RETRIEVAL_OPERATION are two Microservice API Patterns (MAP) responsibility decorators, indicating the behavior of the operation (here: 1x write, 2x read access to API provider-side state). All available MAP decorators are listed in the “MDSL Grammar Quick Reference”.

Rationale: Making MDSL and the design transformations featured in this post aware of patterns speeds up the design work and removes technical risk (as a proven solution to a recurring problem is chosen and semi-automatically applied).

Step 3: Refine Data Types

The generated PaperPublishingRealizationEndpointDTO looks a bit odd and incomplete still. Let’s apply two transformation quick fixes to the data type definition to improve the data contract, “Add string as type” and “Include atomic parameter in key-value map”:²

1
2
3
4
data type PaperPublishingRealizationEndpointDTO 
  "mapOfPaperPublishingRealizationEndpointData":{
    "key":ID<string>, 
    "paperPublishingRealizationEndpointData":D<string>}

In the above snippet, the identifier of the generated DTO type structure has been changed to mapOfPaperPublishingRealizationEndpointData manually (this can be done via an “Rename Element” refactoring, provided by the underlying modeling frameworks EMF and Xtext).

Rationale: Key-value pairs are pretty common in programming languages and elsewhere (even in real life). Here, the "key" is specified to have an IDentifier role, and the value is a plain string. This value can be structured too (similar to JSON object nesting).

The MDSL data modeling concepts are quite close to those in JSON and Jolie (curly braces!). “MDSL Data Contracts” is the reference page in the MDSL documentation.

Step 4: Refactor Operations to Improve Quality Properties

When large results sets are returned, the Pagination pattern is often applied to balance comprehensive information needs with network- and processing-efficient message sizes. Pagination is not only described as a pattern in MAP but also as an API refactoring called Introduce Pagination. The refactoring steps are supported in three MDSL transformation quick fixes that correspond to pattern variants. Apply “Introduce Offset-Based Pagination” to findAll in the endpoint (that was created in the previous step) to change the operation interface to:

1
2
3
4
operation findAll with responsibility RETRIEVAL_OPERATION 
  expecting payload "query":{"queryFilter":MD<string>*, "limit":MD<int>, "offset":MD<int>} 
  delivering payload <<Pagination>> "result":{
    "responseDTO":PaperPublishingRealizationEndpointDTO, "offset-out":MD<int>, "limit-out":MD<int>, "size":MD<int>, "self":L<string>, "next":L<string>}*

Note the <<Pagination>> decorator and the additional parameters such as "limit".

When the amount of message exchanges should be reduced and larger messages are not an issue, you also may want to consider the Bundle Requests refactoring, typically applied to write operations. The MDSL Tools provide two separate transformations to refactor the request and response messages, “Bundle requests” and “Bundle responses”. The create operation originating from the story defined in Step 1 can serve as an example, realizing a bulk/batch upload here:

1
2
3
4
operation create with responsibility STATE_CREATION_OPERATION 
  expecting payload <<Request_Bundle>> {
    {"paperItem":D, "title":D, "authors":D, "venue":D, "paperCollection":D} }+ 
  delivering payload <<Response_Bundle>> { {"data":D<string>} }+

A third example of a quality refactoring is Add Wish List. Wish List is pattern to reduce the amount of response data returned (see here). We may want to apply it to the third operation in the endpoint, findById.

To do so, we first wrap the atomic request parameter and the type reference in the response message in a Parameter Tree (this can also be done automatically by the next transformation):

1
2
3
4
operation findById with responsibility RETRIEVAL_OPERATION 
  expecting payload "resourceIdWrapper":{"resourceId":ID<int>} 
  delivering payload "responseDTOWrapper":{"responseDTO":PaperPublishingRealizationEndpointDTO}

Now we can run the “Add Wish List” transformation on the operation to yield the following operation interface (note: the generated top-level message identifier was removed, it is not needed):

1
2
3
4
operation findById with responsibility RETRIEVAL_OPERATION 
  expecting payload {"resourceId":ID<int>, 
    <<Wish_List>> "desiredElements":MD<string>*} 
  delivering payload {"responseDTO":PaperPublishingRealizationEndpointDTO}

Outlook: Another twenty-something refactorings that match those in the refactoring catalog are available as well, and documented in the “MDSL Transformations” reference.

Step 5: Progress From Abstract Endpoint Types to HTTP Resources

MDSL is an intermediate, technology-neutral API contract language. But what we are really looking for is API Descriptions for popular integration technologies such as HTTP resource APIs.

HTTP resource APIs must respect the REST constraint uniform interface, which means that it is not possible to define the operations of a resource, identified by a URI, freely; only GET, POST, PUT, and in most environments, PATCH and DELETE are available. This constraint implies that an abstract endpoint that exposes more than five operations and/or multiple getters, cannot be translated to an HTTP resource API specification in a a straightforward way (this stands in contrast to APIs realized with gRPC Protocol Buffers, GraphQL, and so on). An explicit binding is required — sometimes more than one, as the relation from MDSL endpoint type to HTTP resource can be n:m.

In the MDSL Tools, multiple transformation quick fixes are available and in this transition; some manual work is required as well in this example. Let’s first create a binding (with the “Provide HTTP binding” quick fix on the endpoint) and then fix the reported mapping/binding problems , with “Move operation binding to new resource with URI template for PATH parameter” . We can also change the binding of the create operation from PUT to POST if we want to. The result of these three actions is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
API provider PaperPublishingRealizationEndpointProvider offers PaperPublishingRealizationEndpoint 
 at endpoint location "http://localhost:8080"
via protocol HTTP binding
resource PaperPublishingRealizationEndpointHome 
 at "/paperPublishingRealizationEndpointHome"
 operation create to POST 
  element "paperItem" realized as BODY parameter 
  element "title" realized as BODY parameter 
  element "authors" realized as BODY parameter 
  element "venue" realized as BODY parameter 
  element "paperCollection" realized as BODY parameter 
  element "requestedPage" realized as BODY parameter 
  element "requestedPageSize" realized as BODY parameter
  element "paperCollection" realized as BODY parameter
 operation findAll to GET element "queryFilter" realized as QUERY parameter
resource PaperPublishingRealizationEndpointHome_findById 
 at "/paperPublishingRealizationEndpointHome/{resourceId}"
 operation findById to GET element "resourceId" realized as PATH parameter

MDSL language reference: “Protocol Bindings” page and “Support for HTTP Resource APIs”.

Step 6: Turn MDSL into OpenAPI and other Platform-Specific Interfaces

The MDSL that we yield when running through Steps 1 to 5 is available for download here.

6a: OpenAPI for HTTP Resource APIs

In the MDSL Tools, a context menu option allows generating various target formats (select “MDSL” and then “Generate …”).³

In the Web-based Swagger editor, the OAS generated from the endpoint type and HTTP binding from the previous steps look like this (you may want to copy-paste it there and navigate around):

The paginated findAll operation appears as an HTTP GET method now. Metadata elements such as limit and offset control the pagination (as explained in the Pagination pattern):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
paths:
  /paperPublishingRealizationEndpointHome:
    summary: general data-oriented endpoint
    get:
      tags:
      - PaperPublishingRealizationEndpointProvider-PaperPublishingRealizationEndpointHome
      summary: findAll (read only method)
      description: '[Retrieval Operation](https://microservice-api-patterns.org/patterns/responsibility/operationResponsibilities/RetrievalOperation.html).'
      operationId: PaperPublishingRealizationEndpointHome-findAll
      parameters:
      - name: queryFilter
        in: query
        description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
          target="_blank">Metadata Element</a>
        required: false
        schema:
          type: array
          description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
            target="_blank">Metadata Element</a>
          items:
            type: string
      - name: limit
        in: query
        description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
          target="_blank">Metadata Element</a>
        required: true
        schema:
          type: integer
          description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
            target="_blank">Metadata Element</a>
          format: int32
      - name: offset
        in: query
        description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
          target="_blank">Metadata Element</a>
        required: true
        schema:
          type: integer
          description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
            target="_blank">Metadata Element</a>
          format: int32
      responses:
        "200":
          description: findAll successful execution
          content:
            application/json:
              schema:
                type: array
                items:
                  type: object
                  properties:
                    responseDTO:
                      $ref: '#/components/schemas/PaperPublishingRealizationEndpointDTO'
                    offsetout:
                      type: integer
                      description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
                        target="_blank">Metadata Element</a>
                      format: int32
                    limitout:
                      type: integer
                      description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
                        target="_blank">Metadata Element</a>
                      format: int32
                    size:
                      type: integer
                      description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/MetadataElement"
                        target="_blank">Metadata Element</a>
                      format: int32
                    self:
                      type: string
                      description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/LinkElement"
                        target="_blank">Link Element</a>
                    next:
                      type: string
                      description: <a href="https://microservice-api-patterns.org/patterns/structure/elementStereotypes/LinkElement"
                        target="_blank">Link Element</a>

Other target Interface Definition Languages (IDLs) such as gRPC Protocol Buffers, GraphQL schema and Jolie are supported as well; Step 4 of the sibling post “Event-Driven Service Design: Five Steps from Whiteboard to OpenAPI and Camel” features their generators.

Rationale: With respect to the POINT principles for API design, the MDSL contracts with their various bindings promote a technology-neutral but still style-oriented approach (REST is one such integration architectural style).

The “MDSL Tools: Users Guide” explains the performed mapping/conversion in detail, and also show what to do with the generated specifications. See for instance “OpenAPI Specification Generator”⁴.

6b (optional): AsyncAPI (via AsyncMDSL)

AsyncAPI is supposed to do what OpenAPI does, but for APIs provided by messaging systems and their applications. There is an MDSL extension supporting such channel/queue interface definition on the abstract level; similar to core MDSL, it provides (integration) patterns decorators. It is called AsyncMDSL.

AsyncMDSL can be created from scratch — or added to our interface with a transformation. “Create AsyncMDSL Specification” is available in the “MDSL” menu (if the MDSL Tools have been installed into Eclipse). The channel that corresponds to the findAll operation looks as follows:

1
2
3
4
5
channel PaperPublishingRealizationEndpoint_findAll
request message findAllRequestChannel on path "/findAllRequestChannelPath" 
expecting payload "query":{"queryFilter":MD<string>*}
reply message findAllReplyChannel on path "/findAllReplyChannelPath" 
delivering payload "result":{"responseDTO":PaperPublishingRealizationEndpointDTO}*

Rationale: Event-driven architectures and queue-based messaging are popular choices when integrating enterprise applications and service components. With AsyncMDSL, related modeling support is integrated into MDSL.

Another MDSL generator supports the transition from this abstract, pattern-oriented AsyncMDSL to AsyncAPI (“Generate AsyncAPI Specification”). The channel for findAll now looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
    findAllRequestChannel:
      name: findAllRequestChannel
      title: Find All Request Channel
      description: |
        No description specified
        Request message. Reply message is *findAllReplyChannel*. 
      payload:
        type: object
        properties:
          'queryFilter':
            type: array
            items: 
              type: string
    findAllReplyChannel:
      name: findAllReplyChannel
      title: Find All Reply Channel
      description: |
        No description specified       
        Reply message. Request message is *findAllRequestChannel*. 
      payload:
        type: array
        items:
          type: object
          required:
            -  'responseDTO'
          properties:
            'responseDTO':
              $ref: '#/components/schemas/PaperPublishingRealizationEndpointDTO'

The generated AsyncAPI is available for download here as well. You can test it in the AsyncAPI Playground.

6c (optional): Java Modulith

If you are not in the mood for remoting and have decided to build a modular monolith instead,⁵ you can also generate a set of Java classes and interfaces that are structured according to our intermediate API design.

Selecting “Generate Java Modulith” in the “MDSL” menu causes this code to be generated into the src-gen folder (multiple folders and files).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// ** service interface class:

CreateResponseDataTypeList create(CreateRequestDataTypeList anonymousInput);
FindAllResponseDataTypeList findAll(FindAllRequestDataType anonymousInput);
PaperPublishingRealizationEndpointDTO findById(FindByIdRequestDataType anonymousInput);

// ** DTOs used in interface

public class FindAllRequestDataType {
  private List<String> queryFilter; 
  private Integer limit; 
  private Integer offset; 
  [...]
}

public class FindAllResponseDataTypeList {
	private List<FindAllResponseDataType> entries; 
  [...]
}

public class FindAllResponseDataType {
	private PaperPublishingRealizationEndpointDTO responseDTO; 
	private Integer offsetout; 
	private Integer limitout; 
	private Integer size; 
	private String self; 
	private String next; 
  [...]
}

The in parameters are called anonymousInput because the generator was unable to get a name from the input MDSL (which would be possible). The generated code comes with a very basic server stub implementation and a JUnit test case. So you can run it straight away.

Rationale: Remoting introduces overhead, which sometimes is not justified or not tolerable. A standalone Java program with local interfaces also is very easy to run and test, which supported rapid prototyping and continuous design refinement.

Step 7: From API Description to API Implementation(s)

See Step 7c of the post “Domain-Driven Service Design with Context Mapper and MDSL” for instructions how to:

• Create REST controllers in a Spring Boot application from the Step 6a OpenAPI.
• Complete this application and test it.
• Deploy it to a public cloud.

More generators and text templates are available have been drafted and are available upon request:

• A simple REST controller for Spring can be generated and tested with curl.
• A template for a corresponding Apache Karate test suite (.feature file) exists as well.
• Basic sample data can be generated with templates too (as/if there is a provider specification in the MDSL).

Wrap Up and Next Steps

This post demonstrated how to progress from requirement elicitation/analysis to contract-first service design in a few incremental steps. These steps are supported by several MDSL Tools. We were able to progress from analysis to early design and experimentation quite rapidly:

1. Integration stories make sure that API operations are introduced for a reason, supporting the API first principle.
2. Interface data modeling was underrated so far, but is important to get request and response messages content right; it is supported by transformations in the MDSL tools.
3. An API design evolves continuously by refactoring them to patterns (which requires more than code refactoring).
4. OpenAPI and other contract formats are generated from intermediate and final MDSL specifications (that come out of Steps 1 to 3).
5. Mock implementations no longer have to be written manually but are generated too. Rapid API prototyping and testing is streamlined this way. Continuous API testing might then suggest further refactorings, leading back to earlier step in an iterative and incremental fashion.

The advantages of the resulting mini-method for service-oriented analysis and design include:

• An API feature requirement, expressed as story, is the starting point. Via a trace link, this goal is kept at hand while designing.
• We can experiment with different designs on a technology-independent, but still concrete level.
• Platform artifacts can be generated, which jump starts agile development.

Some of the consequences of leveraging such a tool-supported method are:

• A new language and supporting tools are involved, which causes some learning effort and context switches between tools while designing.
• Models (would) have to be reconciled if the tools were to support automated round-tripping.
• Modeling tools in general might be perceived as “anti agile” — if code is seen as the only abstraction that is required and suited (what is your opinion on that?).

Outlook: Having run through these seven steps, can we consider the API done? Well, not quite. The steps only yielded an early prototype and service stub. Next up would be providing backend connectivity, implementing API logic including transaction management, monitoring, backup and security design (to call out just a few particularly relevant design issues and stakeholder concerns).

The sibling post “Event-Driven Service Design: Five Steps from Event Storming to OpenAPI and Camel Flow” shares more of the decisions required (and options available). Our “Design Practice Repository (DPR)” can guide you through through some of the more advanced tasks. And the workshop paper “Architectural Decision Models as Micro-Methodology for Service-Oriented Analysis and Design” collects and organizes many of the decisions points (note: while the options might be dated, the issues still apply).

Latest reference information for the MDSL transformations featured in this post (and all additional ones) can be found here:

https://microservice-api-patterns.github.io/MDSL-Specification/soad.html

Final thought: Service and application design, implementation and integration might be business as usual for many of us now, but never gets boring!

Contact me if you have comments (see links below).

– Olaf (a.k.a. socadk)

This post is also available as a Medium story.

Acknowledgements and Notes

The refactorings featured in this post are joint work with my esteemed OST colleague Mirko Stocker. Mirko also reviewed an intermediate draft of this post.

AsyncMDSL and the AsyncAPI generator were developed by Giacomo Di Liberali in his master thesis at the University of Pisa.

While at OST, Stefan Kapferer originally developed several of the MDSL generators featured in this post.