Common Process Challenges When Breaking Monoliths

3 difficult challenges that we often come across when our team works with clients as they try to break monoliths:

1. Silver Bullets - Enterprises have been burnt by vendor solutions that promise migration with a tool like BPM or some silver bullet methodology that promises seamless migration. The Truth is that disentangling your monolith's code and data is going to get messy. The entire business process will need to be disaggregated, visualized and seams will need to be identified to create a blueprint for a target architecture. Other than Pivotal/VMware there is no one else who has done this at enterprise scale. Our approach modernizes the monolith incrementally with demonstrated business value in weeks not years. 

2. Over Engineering - It is common to get distracted by technology choices and deployment options rather than focus on the difficult work of understanding the core domain. Identifying the business capabilities and assessing if they are core or supporting domains. Do what is sustainable and what your average (not rockstar) software engineers can support and focus on the outcomes.

3. Pressure Cooker:  When clients continuously keep changing their priorities or lose faith in the progress or micromanage the design then it subverts the process and the target architecture looks like the old system.  Breaking monoliths with Domain Driven Design is like landing the plane from 30,000 feet. You cannot skip phases and go straight to user story backlog generation with somebody else's domain model or DDD process. Don't short circuit steps in the process.  It is critical to follow the strategic and tactical patterns and land the plane with gradual descent to an organized user story backlog.  

also checkout Eight Pitfalls Of Application Modernization (m11n) when Practicing Domain Driven Design (DDD)  

Eight Pitfalls Of Application Modernization (m11n) when Practicing Domain Driven Design (DDD)

1. High Ceremony OKRs

An application modernization effort is necessary broad in scope. Overly constraining the goals and objectives often results in premature bias and could lead to solving the wrong problems. Driving out premature Key Results means that you are tracking and optimizing for the wrong outcomes. If you are uncertain of the goals of modernization or if this is your first time refactoring and rewriting a critical legacy system, it is best to skip this step and come back to it later. 

2. Mariana Trench Event Storming

There are multiple variants of Event Storming (ES). ES can be used for big picture, detailed system design, greenfield exploration, value stream mapping, journey mapping, event modeling etc., Beware of going too deep into existing system design with ES when modernizing a system. If the intent of ES is to uncover the underlying domains and the services it is counterproductive to look at ALL the existing constraints and hot spots in the system. Model enough to get started to understand and reveal the seams of the domain. If you go deep with Event Storming you will bias or over-correct the design of the new system with the baggage and pain of the old. 

They key to success with Event Storming is to elicit the business process i.e. how the system wants to behave and not the current hacked up process. When there is no representation from the business there is no point in doing event storming. 

3. Anemic Boris

When modeling relationships between bounded contexts it is critical to fully understand the persistence of data, flow of messages, visualization of UI and the definition of APIs. If these interactions are not fully flushed out the Boris diagram becomes anemic and this in turn reflects an anemic domain model. 

4. Picking Thin Slices

As you start modeling the new system design the end to end happy path of the user workflow first. Pick concrete scenarios and use cases that add value to the business.  Those that encounter the maximum pain. The idea here is to validate the new design cheaply with stickies instead of code and not get stuck in error or edge cases. If you don't maintain speed and cycle through multiple end to end steel threads you may may be prematurely restricting the solution space. 

5. Tactical Patterns - The Shit Sandwich

As you start implementing the new system and co-existing with the new there are a few smells to watch for the biggest one being the Shit Sandwich. “Shit Sandwich” - When you are stuck doing DDD on a domain that is sandwiched between upstream and downstream domains/services that cannot be touched thereby overly constraining your decomposition and leading to anemic modeling since you cannot truly decompose a thin slice. You spend all your time writing ACLs mapping data across domains. 

So the Top bun is the downstream service, the bottom bun is the upstream service, then there are two layers of cheese - which are the ACLs and then  your core domain is the burger patty in the middle - and now you have the :shit:  sandwich. Watch for this when you are called in to modernize ESBs and P2P integration layers. 

6. Over Engineering

Engineers are guilty of this all the time.

Ooooh ... we modeled the system with Event Storming as events ergo - we should implement the new system in Kotlin with Event Sourcing on Kafka and deploy to Kubernetes.

Yikes!!

Do what it is sustainable and what your average (not rockstar) software engineers can support. My colleague Shaun Anderson explains this best in The 3 Ks of the Apocalypse — Or how awesome technologies can get in the way of good solutions.

7. Pressure Cooker Stakeholder Management

When the stakeholders of the project continuously keep changing their priorities or they lose faith in the progress or if the domains are pre-decided then it is time to push back and reassert control. The top down process of Domain Driven Design is like landing the plane from 30,000 feet. You cannot skip phases and go straight to user story backlog generation with somebody else's domain model or DDD process. Don't short circuit steps in the process.  It is critical to follow the strategic and tactical patterns and land the plane with gradual descent to an organized user story backlog. 

8. Faith

The biggest way to fail with DDD when doing system design or modernization is when you lose faith  and start questioning the process. Another guaranteed failure mode is when you procrastinate and don't start the modeling activities i.e. you are stuck in analysis and paralysis phase. The only prerequisite to success is fearlessness and curiosity. You don't need to have a formal training to be a facilitator.  Start the journey, iterate and improve as you go.  

Mainframe Modernization Is Hard

Mainframe modernization is like a balloon payment on your 7 year ARM mortgage that has come true or an unhedged call option that has been called. [bad-code-isnt-technical-debt-its-an-unhedged-call-option](https://www.higherorderlogic.com/2010/07/23/bad-code-isnt-technical-debt-its-an-unhedged-call-option/). All code is technical debt and it is critical to understand the risk profile of your debt as you embark on a mainframe modernization project. [risk-profile-of-technical-debt](https://tanzu.vmware.com/content/intersect/risk-profile-of-technical-debt).  Also see derivatives of technical debt for a detailed treatment of this topic. 

Sticking with the financial analogy your payment is due, your option has expired and you are due a large amount. Our natural instinct is to to find an easy way out. Perhaps get a payday loan or swap out one kind of debt for another. Unfortunately none of these options work in the long term. To avoid bankruptcy we have to go through a debt restructuring or program where we retire and payout gradually the debt owed over time or in extreme cases declare bankruptcy. So what does all of this have to do with mainframe modernization ?

There are many enticing options when it comes to mainframe modernization like offloading work to cheaper processors on the mainframe, getting volume discounts from your mainframe provider, slapping REST APIs on top  of the mainframe systems, COBOL to Java Code code generators or outsourcing the refactoring and rewrite of code (debt) outside the company. These efforts generally are well intentioned and start well but quickly get stuck in the mud because they don't scale or the complexity and sustainability of the solution does not work.

At VMware Pivotal Labs we acknowledge that mainframe modernization is hard. The implementation of the program gets worse before it becomes better as concurrent development work streams have to be maintained both for the legacy and net new. Having helped multiple customers on this journey we have come up with a iterative phased approach to mainframe modernization that scales and yields ROI in days and weeks and not months and years.

1. Start with the end in mind. What is the critical business event or situation that has triggered the modernization. It is very important to understand why the modernization program is being funded so that we can create the right set of goals, objectives and key results. Are you doing because you cannot add features fast enough to the critical system  running on the mainframe. Are you doing this because you need a new digital 360 degree experience for your customers ? What are the key business drivers for the modernization. This alignment needs to be driven by both business and technology executives and refinforced by all the product owners, product managers and Tech. Leads. **The outcome of this phase is clearly articulated set of goals and objectives with quantified key results that provide the journey markers to understand if the program is on track.**

2. After goal alignment it is time to take an inventory of the business processes of the critical systems running on the mainframe. The business domain has to be analyzed and broken down into discrete independent business capabilities that can be modeled as microservices. This process of analyzing the core business domain and deriving modeling its constituent parts is called Event Storming. Event Storming enables decomposing massive monolith systems into smaller more granular independent software modules aka microservices. It allows for modeling new flows and ideas, synthesizing knowledge, and facilitating active group participation without conflict in order to ideate the next generation of a software system. Event Storming a group collaborative modeling exercise is used ot understand the top constraints, conflicts, inefficiencies of the system and reveal the underlying bounded contexts The seams of the current system will tell us about how the new distributed system should be designed. We also weaven in aspects of XP here like UCD, Design Thinking and interviews to ensure that our understanding of the system to keep it real. **The outcome of this phase is a set of service candidates also called as bounded contexts that represent the business capabilities of the core, supporting and generic business domain.** 

3. A critical system on the mainframe like commercial loan processing or Pharmacy Management has multiple end to end workflows. It is critical to understand all the key business flows across the event storm that provide a steel thread for modernization. We need to prioritize these key flows as they will drive out the system design. The first thin slice picked should be a happy path flow that provides end to end value and demonstrates incremental progress and redoubles the faith in the whole process. **The outcome of this phase is a set of prioritized thin slices that encompass the Event Storm.**

4. We have all the lego pieces, now its a matter of putting them together with flows of messaging, data, APIs and UIs so that we fulfill the needs fo the business. We now have to wire up all our domain services. We use a process called Boris invented at Pivotal for this phase [SWIFT](https://www.swiftbird.us/). Boris provides a structured way to design synchronous API driven and asynchronous event-driven service context interactions. We identify relationships between services to reveal the notional target system architecture and record them using SNAP. SNAP takes the understanding from the Boris diagram to understand the specific needs of every bounded context under the new proposed architecture. The SNAP exercise is done concurrently with the Boris exercise. We call out APIs, data needed, UIs, and risks that would apply to that bounded context as the thin slice is modeled across services.  **The outcome of this phase is a Notional Target Architecture of your new system with external interaction modes mapped out in terms of messaging, data and APIs.**

5. In some ways this process is like bringing a plane at 30000 feet to the ground. We are in full descent now and at the 10K feet level. At this point we have a target architecture. It is critical now to understand how we will develop these the old and new systems without disruption i.e. change the engine while the plane is descending to the ground.  We employ a set of key tactical patterns for modernization like Anti-Corruption Layer,  Facade, Data driven strangler to carve out a set of MVPs and user stories mapped to the MVP. These stories will realize the SNAP built earlier and implement the thin slices that were modeled. Creating a road map of all the quantum of work is critical as we start to make sure we are going in the right direction with speed. **The outcome of this phase is a set of user stories for modernization implementing the tactical pattern of co-existence with the monolith and a set of MVPs to track the key results.**

6. We no have a backlog of user stories and we are less than 1000 feet from the ground. At this point it is important to identify the biggest spikes and risks to the technical implementation like latency, performance, security, tenancy etc., and resolve them. We start building out contracts for our APIs so that other teams and dependencies may get unblocked. The stories are organized into Epics at Inception, product managers and engineers are allocated and the first iteration begins. The feedback loop from Product - Engineering - Business is set in motion. **This phase encompasses the first sprint or iteration of development. It is critical to establish demos, Iteration Planning Meeting, retrospectives and feedback loops in this phase as this will set a tone for the rest of the project. **

The six steps of mainframe modernization outlined here are not implemented like a waterfall. Six steps are sometimes run multiple times for different areas of a complex domain for a large domain and the results are stitched together. Steps or phases may be skipped altogether if we already know parts of the domain well. This six step process is what we call SWIFT. It is not dogmatic. Do what works at velocity to modernize the system in increments with a target architecture map in hand. Mainframe modernization is hard and there is no easy way out. Internalize this and start the journey of thousand miles swiftly with the first step. 

Top Ten Application Modernization Myths

Sometimes we tell little lies to ourselves. It is always good to take inventory of reality and introspect on what is true and what is not. Here are some of the little lies of application migration and modernization that I have observed over the last five years. 

1. Application has to be 12/15 factors compliant to land on PaaS. Apps can be modified on the cloud native spectrum. The more cloud native idiomatic changes to an app - the more ROI you get from the changes. See Myth #1 "Cloud Foundry can only run cloud-native, 12-factor apps." - FALSE https://tanzu.vmware.com/content/blog/debunking-cloud-foundry-myths
2. Applications need to be modified by developers before landing them on Kubernetes (TKG). In fact an enterprise can get significant cost savings by migrating one factor apps to Kubernetes. A one factor app simply has the capability to restart with no harmful side-effects See https://tanzu.vmware.com/content/intersect/vmware-tanzu-in-15-minutes Do you even have a 1-factor application?
3. Once technical debt on an application becomes unsurmountable the only recourse is to rewrite it. Surgical strikes with an emphasis on understanding the core domain can lead to incremental modernization of the most valuable parts of a big critical legacy system. A FULL rewrite is not the only option.  See technical debt like financial debt https://tanzu.vmware.com/content/intersect/risk-profile-of-technical-debt and https://tanzu.vmware.com/content/webinars/may-6-tech-debt-audit-how-to-prioritize-and-reduce-the-tech-debt-that-matters-most
4. There is a silver bullet for app migration. There is an increasing bevy of tools that have started promising a seamless migration of VMs to containers in the cloud. Remember in life nothing is free. You get what you put in. Migration is highly contextual and the OPEX and Developer efficiency returns are dependent on the workloads being ported. Migration of apps in VMs to Kubernetes stateful sets or automatic dockerization through buildpacks etc should be evaluated for the desired objectives of the Migration Project.
5. Microservices and event driven architecture is ALWAYS the right architecture choice for app modernization. Sometimes the answer is to step back simplify the domain and implement a modular monolithic system and sometimes the answer is to decompose the large system into a combination of microservices and functions. Understand the design and operational tradeoffs first before making the choice. Every tech choice like eventing, APIs, streaming etc has a spectrum. The fundamental job of an architect is to understand the sociotechnical factors and make the right choices from a process, people and implementation perspective. see https://tanzu.vmware.com/content/practitioners-blog/how-to-build-sustainable-modern-application-architectures
6. Decomposing and rearchitecture of an existing system can be done concurrently with forward development with little impact to exisrting release schedules. This is a dream. When working on two branches of an existing system a forward development branch and a rearchitecture branch > the total output often times gets worse before becoming better. WBB - This is because there is a period of time where dual maintenance and dual development and the coordination tax across two teams are levied without getting any of the benefits of modularization and refactoring. See The Capability Trap: Prevalence in Human Systems https://www.systemdynamics.org/assets/conferences/2017/proceed/papers/P1325.pdf https://rutraining.org/2016/05/02/dont-fall-into-the-capability-trap-does-your-organization-work-harder-or-smarter/
7. The fundamental problems of app modernization are technical. If developers  only had the rigor and discipline to write idiomatic code all problems would be fixed and we won't incur technical debt. Wrong- The fundamental problems of app modernization are team and people related. Incorrect team structure, wrong alignment of resources to core domains and messed up interaction patterns are far more responsible for the snail pace for feature addition rather than technical changes. The answer is team re-organization based on the reverse conway maneuver. See Team Topologies https://www.slideshare.net/matthewskelton/team-topologies-how-and-why-to-design-your-teams-alldaydevops-2017
8. Mainframe modernization can be accelerated by using lift-n-shift tools like emulators or code generation tools. In our experience a complex mainframe modernization almost always involves a fundamental rethink of the problem being solved and then rewriting a new system to address the core domain divorced from the bad parts of the existing intermingled complex system. Theory of constraints and a systems thinking help us reframe the system and implement a better simpler one.
9. Engineers, Developers and Technical Architects tend to think from a technical nuts and bolts perspective (the “how”) and, therefore, tend to look at modern technologies such as Cloud Foundry, Spring Boot, Steeltoe, Kafka and containerization as the definition of a modern application. This misses the mark. The Swift Method pioneered by Pivotal  helps bridge the gap in understanding between the non-technical, top down, way of thinking and the technical, bottom up thought process.  The end result is an architecture that maps to the way the system “wants to behave” rather than one that is dictated by the software frameworks of du jour. 
10. AWS or Azure or GKE/GCP etc provide an all encompassing suite of tools, services and platforms to enable and accelerate modernization and migration of workloads. While it is true that the major cloud providers have ALL the bells and whistles to migrate workloads, the economics of app modernization tend towards the app and not the platform. The more cloud native you make the app, the higher the optionality you get since it becomes cloud agnostic allowing enterprises to exact maximum leverage from all the providers. The focus needs to be on the app inside-out to get the best returns. In general the higher you are in the abstraction stack the more performance gains you will get so Architecture changes will yield a 10x more benefit than JVM or GC tuning which will yield a 10x more benefit than tuning assembly code and so on … If it is the database tier that you think is the problem - then you can put in multiple shock absorbers 1. caches 2. queues 3. partitioning first and focused on instead tuning the startup memory and app start times. Apps first, Platform second :-)  

Event Storming - A Pivotal Practice for decomposing applications

FIELDS + GUIDANCE	Guidance
Name of method	Event Storming https://speakerdeck.com/rkelapure/event-storming
What is this method ?	Event Storming is a cross functional facilitation technique for revealing the bounded contexts, microservices, vertical Slices, trouble spots and starting points for a system or business process.
Phases	Discovery, Kick Off
Suggested Time	1 - 2 hours.
Who participates?	SMEs, Core Team (see facilitator notes)
Why do it?	Event Storming enables decomposing monoliths into microservices. It allows for modeling new flows and ideas, synthesizing  knowledge and facilitating active group participation without conflict to time travel and ideate the next generation of a software system.
When to do it?	When you need to make sense of a Huge mess to enable Cross Perspective Communication as a force function for clarity.
What supplies are needed?	People, tools and supplies needed to conduct an ES session A large wall (for stickies) At least 4 different colored stickies Sharpies Blue painters tape Refreshments  like water and soda and juices for hydration Paper flip boards for readouts and breakouts
How to Use this Method	Event Storming is a group exercise to scientifically explore the domains and problem areas of a monolithic application. The most concise description of the process of event storming comes from Vaughn Vernon's DDD-Distilled book and the color around the process comes from Alberto Brandolini's book Event Storming. Storm the business process by creating a series of Domain events on sticky notes. The most popular color to use for domain events is orange. The DomainEvent is a verb stated in past tense and represent a state transition in the domain. Write the name of the DomainEvent on an orange sticky note. Place the sticky notes on your modeling surface in time order from left to right. As you go through the storming session, you will find trouble spots in your existing business process. Clearly mark these with a purple/red stick notes. Use vertical space to represent parallel processing. After all the events are posted experts will post locally ordered sequence of events and enforce a timeline. Enforcing a timeline triggers long awaited conversations and eventually STRUCTURE will emerge.  These event clumps or common groupings give us our notional service candidates (actors or aggregates depending on how rigid the team is with DDD definitions).  These will be used during the Boris Exercise.
Success/Expected Outcomes “You know you are done when…”	- Event Storming generates an immense backlog of user stories. - Perform  User Story Mapping to Map and organize stories into MVPs - Define scope of the problem - Confirm that you are solving the right problem ?
Facilitator Notes & Tips	Event Storming is a technique used to visualize complex systems and processes. This could range from monoliths to value streams. Event Storming is gamestorming technique for harnessing and capturing the information captured in a group’s minds. It surfaces conflicts and different perspectives of a complex system and bubbles up the top constraints and problem spots.   As an event storming facilitator you have one job - create a safe environment for the exchange and output of ideas and data. The job is 50% technical facilitation and 50% soft people facilitation where you are reading body language. A single facilitator can typically orchestrate groups of 15-20. For a group of 30 or more you need two facilitators.  ES is usually conducted in two phases. A high level event storm to identify the domains and then a subsequent ES into a top constraint - the core domain. The language of ES is stickies. In its simplest form ES is basically a facilitated group story telling. The stickies represent domain events -  or things that happened in the past. The trouble spots are identified by orange/red stickies. The color of the stickies does not matter. What does matter is that you start simple and then add the notation incrementally. Start simple and then add the information in layers. ES can serve many goals - break down a monolith into constituent bounded contexts, create a value stream - as a way to onboard employees, etc., There is no ONE correct style of ES. Every session is different from another based on the desired goals and outcomes. So don’t worry about getting it right- just do it and roll your own style. An ES is only successful if the right people are involved. This is a mix of business domain experts, customer executives, stakeholders, business analysts, software developers, architects, testers, and folks who support the product in production. Subject matter experts, product owners and developers that knows and understand the application domain. This process enables cross perspective conversation throughout the team as well as a standard definition of the terms used by both technical and non-technical team members.
Related practices	BORIS SNAP
Real world example	Deconstructing Monoliths With Domain Driven Design
Recommended reading	Motivation behind ES can be found here - Gamestorming: A Playbook for Innovators, Rulebreakers, and Changemakers. Note this book is available on safari books online: https://www.safaribooksonline.com/library/view/gamestorming/9781449391195/ This is the book we read on the flight to Boston Eventstorming https://leanpub.com/introducing_eventstorming Book is written by Alberto Brandolini the father and inventor of Eventstorming Domain Driven Design (DDD) - provides the theoretical underpinnings of decomposing  ‘/monoliths. DDD Distilled is the perfect book to understand the science of DDD and how ES fits into the grander scheme of things - how do the ES artifacts translate into  software design, architecture and an actual backlog.