MAP Retrospective and Outlook

Microservices are still trending, but not straightforward to design well. Patterns are a mature and elaborate form of knowledge sharing, so Microservice API Patterns (MAP) complement other pattern languages to fill the knowledge gap. Let’s recapitulate and reflect a bit where MAP currently stands — and see what’s coming.

Update (2/2023): Podcast about out book now available! Listen to Daniel Lübke and Vaughn Vernon discussing “Writing Patterns for API Design”.

Update (12/2022): The book is published! More information.

Update (08/2022): Our pattern language for API and service design forms the core of a forthcoming Signature Series book to be published by Pearson in December 2022! Read more.

Update 2/2022: Watch my fellow MAP writer Daniel Lübke explain the why and how of MAP in a 24-minute video on Erik Wilde’s YouTube channel “Getting APIs to Work”.

Why MAP?

Why yet another pattern language? Why one on service design? Why now?

I have long been a pattern enthusiast (I am Olaf a.k.a. ZIO, by the way). The inaugural Design Patterns book was among the first books I bought (and actually read) when developing distributed systems, middleware and tools had become my day job back in 1994. Having applied patterns from languages such as Patterns of Enterprise Application Architecture (PoEAA), Enterprise Integration Patterns (EIP), and Cloud Computing Patterns in research and development regularly since then, writing some myself had been on my mind for a long time. However, mining and documenting proven solutions to recurring problems and forces requires deep experience, both in a topic area and in technical writing — if you take the term pattern seriously.¹ Pattern writing calls for a substantial investment of time, commitment and perseverance. Hence, I was reluctant for many years.

I finally decided to embark on a pattern writing journey during a mini-sabbatical in 2016. At that time, microservices had become quite popular, so both project input and an audience were there. Many of the messages in the early articles on microservices reminded me of the positive experiences on SOA projects² I had gained more than a decade earlier.³ SOA and microservices share principles such as business alignment (via domain models), loose coupling (or service autonomy) and an emphasis on rightsizing services (not downsizing)⁴. Not convinced? This web page collects evidence and this journal paper analyzes in depth.⁵

Getting started

An early step when trying to contribute to the collective knowledge of a community (soon after ideation) should be to look for things that have been said and done already.⁶ The website microservices.io was there, the Sam Newman book was there, Web APIs and RESTful HTTP were pretty well covered in blogs and books too. And there even was an entire book on Service Design Patterns (a bit older).

All these sources are useful and good reads. But not enough focus seemed to have been put on the APIs, the contracts, the message structures. Service designers ought to know:

• How many services should be exposed? What is an adequate size for them?
• How to ensure that services are loosely coupled? How much data should they exchange, and how often does this happen?
• What are the most suitable message representations? How to agree on the meaning of each message?

I noticed another knowledge gap. Throwing Domain-Driven Design (DDD) terms such as Bounded Context into service design discussions is good, but only if you know a) how to find such contexts and b) how to translate them into API endpoints and operations. Advice on these two steps still is not easy to find online and in books (please correct me if I am wrong).

So we had our scope (I write “we” because in late 2016 I had recruited three co-authors: Daniel Lübke and Mirko Stocker and Uwe Zdun). Capturing design rationale is another favorite topic of mine. So let’s capture this scoping and notation decision properly:

In the context of microservice architectures and Web development, wanting to write about (and promote) API endpoint, operations, and message designs of quality and style,

we decided for the pattern format to mine and share community knowledge and neglected lighter forms such as if-then recipes or free-form “best practice” advice,

to achieve high content quality and good readability (for those familiar with the format)
accepting that the effort for the technical writing is much higher and learning is required.

This decision is structured as a Y-statement, explained in another post in this blog.

Content evolution

First, I studied and researched quite a bit (I was on mini-sabbatical after all). This yielded a list of 15+ candidate patterns. I discussed it with Daniel, which instantly lead to new candidates from his and my service projects — in pattern lingo: known uses.⁷

Next, I discussed the scope with pattern celebrity Gregor Hohpe during an IEEE Software board meeting. He also taught me the pattern template and author tools used for EIP. Thanks Gregor! Still looking forward to your Conversation Patterns.

Next up were “forces jams”: I presented the idea for the new pattern language to a number of architects and service designers, and asked them about the hard problems they had been facing (and they would like to read about). And we started writing pattern drafts.

A first round of comments in a EuroPLoP 2017 Writers Workshop got us very motivating feedback about four basic structural patterns as well the variants of Pagination we had found in various public Web APIs and company-internal integration architectures.

Moreover, we were lucky to receive a rich set of very nice pattern icons created by two of my students, Sebnem Kaslack and Nicolas Dipner, on two term thesis and bachelor thesis projects (many pattern names have changed since then, and IRP is an earlier name for our language):

In addition to the icons, the two projects got us early validation that APIs can be described with our patterns — and that our pattern names and intents do work (for the most part). Finally, Sebnem and Nicolas prototyped a tool that reads in Swagger contracts, looks for pattern instances and lets you zoom in and out of API endpoints and their operations:

In 2018 we had a very active EuroPLoP shepherd and were able to go through a writers’ workshop a second time, yielding five interface quality patterns including API Key and Wish List. My colleague Mirko led this effort and received coaching from Uwe and Cesare Pautasso (who had joined us at that time). Workshops with industry partners to validate our patterns followed. We also invested in creating our own single-source, multi-channel tooling.⁸

2019 was a busy and successful year. As pattern writers we focussed on evolution strategies and versioning, again going through the shepherding and peer-reviewing process at EuroPloP (Daniel had the lead). This is a rather controversial topic; decision making in enterprise application development deviates from some of the practices in the Web community. Backward compatibility and extensibility can be competing forces.[^88] Have a look at Two in Production and Version Identifier to whet your appetite. We also wrote a first tutorial and overview, and gave several presentations. To anybody who attended and commented: thanks again, your feedback served us well! Last but certainly not least, the MAP website went live, and we enhanced it several times since it launch. For instance, we recently added a quickstart page serving as a curated site map. More to come!

Our focus in 2020 was to apply and extend by Rebecca Wirfs-Brock’s Responsibility-Driven Design for remote, message-based APIs, forming the next MAP category. A tough nut to crack, actually: many different voices can be heard, from “REST equals CRUD on data resources” to “no entity services please” — and it is pretty hard to draw the line between API designs (external interfaces) and their implementations. Just think about transaction management and backup, availability, consistency tradeoffs (to learn more, have a look at the BAC theorem). Patterns excerpts are available online, and more detailed articles are available on this page. On the MAP website, a cheat sheet emerged.

In 2021, we added Context Representation, four element stereotypes, and six foundation patterns to the MAP website; these patterns did not go through a pattern conference (yet). We also enhanced the Pagination pattern from 2017.

For further updates, please check the news page.

Lessons learned

Would I do this again? Would I go down the same path? Well:

• Serious pattern writing is harder than it seems. For instance, it is not always clear where to put certain information (context? problem? solution?), and difficult not to unveil and give away the solution in the problem statement or the forces section.
• The advice from the elder might not be easy to follow instantly, but doing so pays off in the long run. Luckily we have two team-internal shepherds. If you are not as privileged, pattern writing advice can be found in “A Decade of Enterprise Integration Patterns” and in “20 Years of Pattern Impact”, two articles from IEEE Software. Try a submission to a patterns conference to receive early feedback! Some general advice on authoring is available on this blog.
• Going through the form-storm-norm-perform cycle on a pro-bono volunteer project can be even more painful and time consuming than on the day job and on commercial projects. Frustration tolerance and patience wanted!
• Markdown, pandoc, and Jekyll are great tools for authors (more on this in a later post).
• Readers appreciate deep advice, but also look for short and concise nuggets of advice and implementation examples. We still work on overcoming the “interesting but I will need to come back when I have more time” reaction on our website.⁹

Articulating these lessons learned seems to be easier than following them consistently. The last one in particular — this blog post got much longer than I thought 😉

What’s next?

We hope you find our patterns useful — feedback and input is always welcome and appreciated! Check our news page and authors page for updates and contact information.

– Olaf (in MAP Author role)