Background
The rising popularity of microservices as an architecture has led to several questions around the role of the ESB. Traditionally the role of the ESB was to fulfill the promise of SOA by providing invocation, routing, mediation, messaging, process choreography, service orchestration, complex event processing, management, agnosticism, support for various message exchange patterns, adapters, transformation, validation, governance, enrichment, support for WS-* standards and abstraction. One can easily see how such a software can easily become the central bottleneck for all things enterprise aka the God object.
The term 'central' can also be used to reference the architectural style, namely everything had to be routed through the ESB. Not having to do that but still support message exchange patterns via Spring Integration's approach was a significant' distinction and is now even more important. The microservices approach favors dumb pipes and smart endpoints whereas an ESB works on an canonical data representation leading to dumb endpoints and smart pipes. There is an inherent conflict between microservices and ESBs. An ESB centric app architecture results in anemic services that are integrated via a smart centralized ESB.
The Microservice architecture pattern is SOA without the commercialization and perceived baggage of WS death star and an ESB. Furthermore creation of a SOA Center Of Excellence for Application Integration promoted a culture of centralized integration in the enterprise. Microservice-based applications favor simpler, lightweight protocols such as REST, rather than WS-*. They also very much avoid using ESBs and instead implement ESB-like functionality in the microservices themselves. The Microservice architecture pattern also rejects other parts of SOA, such as the concept of a canonical schema and an unified domain model across bounded contexts.. [
chris-richardson].
Why Microservices
The fundamental tenets of Microservices like modularization, cohesiveness, DRYness, loose coupling, replaceability etc., have been widely recognized in the software industry since the 70s. The recent popularity of microservices stems from lack of independent deployability, independent scalability and independent failure of systems. A new generation of distributed system platforms and programming models has emerged to address these issues i.e. Microservices [martin-fowler]. Most enterprise systems over time evolve to Big Balls of Mud [balls-of-mud]. Beyond a certain scale making modifications to this spaghetti code jungle becomes untenable. Microservices gives us a structure approach to shear these monoliths and develop and deploy them in an agile fashion. Decomposing a monolith requires patterns like the Strangler, Inverse-Conway, Facade, Adapter, smart routing, feature-flags, zero-downtime-deployment and Proxy. Cloud Foundry natively bakes these concerns into the platform.
Integration tools have historically been the domain of the specialists working in the Integration Competency Center. They have required distinct skill sets and domain knowledge – from enterprise application integration and enterprise service buses and services oriented architecture to extract, transformation and load and enterprise data warehousing and business intelligence. The citizen model of integration refers to the process of democratization of integration workflows upending traditional approaches to data and application integration put together by citizen developers with no specialized domain knowledge driven by the demand for self-service from the business. [rise-citizen-integrator]
The emergence of PaaS and iPaaS platforms is a consequence of the need for reduced business software time to value. Features need to be delivered in a time span of weeks instead of years. The Third Platform makes this possible by baking into its foundation microservice management capabilities like service discovery, routing, load balancing, application lifecycle management and operational capabilities like canary releases, zero downtime deployment,automation, monitoring and security. Many of the concerns taken care of by the ESB are now usurped by the PaaS and provided to all the applications transparently at web scale. Moreover these platforms are truly open to extension and can be deployed using next generation tooling with BOSH, Ansible etc.
People & Process
A successful deployment of microservices is contingent on people, process and technology. The development of microservices is done with the agile methodology. The roots of agile can be found in Mel Conway's seminal work "How Do Committees Invent" mel-conway in 1967.
Conway's First Law: A system’s design is a copy of the organization’s communication structure. Conway's first law tells us TEAM SIZE is important. Communication dictates design. Make the teams as small as necessary.
Conway's Second Law: There is never enough time to do something right, but there is always enough time to do it over. Conway's second law tells us PROBLEM SIZE is important. Make the solution as small as necessary.
Conway's Third Law: There is a homomorphism from the linear graph of a system to the linear graph of its design organization. Conway’s Third law tells us CROSS-TEAM INDEPENDENCE is important. A more modern variant of the same covenant is the ‘two pizza rule’ invented by Jeff Bezos i.e. teams shouldn’t be larger than what two pizzas can feed.
Conway's Fourth Law: The structures of large systems tend to disintegrate during development, qualitatively more so than with small systems. Conway’s fourth law tells us that TIME is against LARGE teams therefore it is critical to make release cycles short and small.
The ESB development model of design by committee is at odds with the fundamental tenets of agile and lean development i.e. smaller decentralized teams and faster release cycles. ESB development requires specialized knowledge and does not support rapid continuous integration and development.
The O-Ring Theory when applied to Devops states that when production depends on completing a series of tasks, failure or quality reduction of any task reduces the value of the entire product. [O-ring-theory]. Another interesting finding is that, the better you are the more value you get from improving your weakness and conversely if you are fairly poor across the board you won’t get as high a ROI on an improvement in one specific area. In order to boost productivity it is critical to pursue excellence across all IT processes.
Challenges With Microservices
When a business system is designed as a suite of microservices that prefer choreography over orchestration it behooves the advocates of the architecture to provide best practices and patterns to implement the abstraction. This is where microservices falls short. The programming frameworks for microservices choreography such as event sourcing and CQRS are woefully behind in terms of maturity and production capability with some notable exceptions like axon or akka-persistence.
An ESB provides a visual mode of development that allows domain modelers and business developers an opportunity to design and validate a business process without getting their hands dirty with code. A cross-bar architecture allows architects to enforce a central point of governance and control over design and implementation. DIY Integration frameworks lack the extensive user and process modeling tools that traditional ESB vendors provide. This is a shortcoming in the DIY Integration ecosystem with Spring Integration.
Features of the ESB like business activity monitoring and event correlation now have to be implemented in applications or supplied as cross-cutting libraries to applications. ESB has a whole suite of connector services and adaptors that will need to plugged in for an individual service instead of harvesting from a common core.
A mature enterprise has a fully developed and very mature suite of ESB services that makes development of features with ESBs easy and fast. The pain of development, training and deployment has been amortized over the years to the point where developing features with the ESB may be equivalent to writing a suite of microservices.
Migration from ESBs to Cloud Native Platforms
There are 5 phases to evolving an ESB infrastructure and accompanying business services to the cloud.
Phase 0 : Co-exist
Deploy the ESB alongside the PaaS. ESB services are exposed as user provided external services to the PaaS. This allows the enterprise to deploy apps to a platform like Cloud Foundry and keep its existing backend services in an ESB like TIBCO. The advantages here is the ability to move the web tier to the next generation platform without disrupting existing integration flows.
Phase 1 : Lift And Shift
Deploy the ESB in the PaaS. Increasingly ESB vendors have moved their offerings to the cloud. See TIBCO Business Works Cloud Foundry Edition or mulesoft iPaaS. In fact Gartner defines this as an entirely new category called iPaaS i.e. a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on premises and cloud-based processes, services, applications and data within individual or across multiple organizations.
The Cloud Foundry edition of TIBCO allows individual TIBCO business processes to be natively scaled and managed by Cloud Foundry. From a Cloud Foundry perspective the TIBCO business process is treated as a new language with its own buildpack. This is a reasonable middle ground between ESBs and Microservices. There are two issues with a cloud enabled ESB like TIBCO, the ESB needs to leverage native messaging capabilities (like FTL) that are not available natively in the PaaS. The ESB needs to leverage the managed data backing services natively available on the platform.
Phase 2 : Refactor
A refactoring of the ESB services offered by the ESB such that the ESB layer is kept as thin as possible. At this point there are many opportunities to migrate from the ESB to Spring Integration to achieve basic transformation, mediation, and routing. Consumer Services Expose APIs with formats/protocols desired by consumers. Consumer services transform formats provide sync/async, protocols and composite workflows. Provider Services integrate with legacy IT systems, implement required workflow and consume whatever protocol/format is exposed by legacy systems. Refactor services incrementally per bounded context, prioritizing the ones that are new or under active development.
Standardize consumer APIs over REST/JSON and reduce the cross-bar architecture complexity by not entertaining a variety of consumer preferences. Extend bounded contexts to include legacy app dependencies. For high performance data intensive workloads the cost of serialization/deserialization of messages from the enterprise bus may be too high. Consider a binary serialization mechanism like google protocol buffers or Kyro for internal messaging, APIs, caching, etc across services.
Phase 3 : Replace
Replacement of ESB services using integration pattern frameworks like Spring Cloud Stream. Spring Integration and its cloud successor Spring Cloud Stream provide successively higher layers of abstraction making it possible to write ESB flows and function using cloud native architectural constructs leveraging annotations and interfaces to specify business function. Spring Cloud Stream is a combination of Spring Cloud, Spring Boot and Spring Integration where integration modules are Spring Boot apps. The plumbing of putting together channel adapters and flows over message brokers like Kafka and RabbitMQ is done intelligently by the platform. Spring-cloud-dataflow can also orchestrate jobs so batch-processing and stream-processing are all manageable in a consistent way on a unified platform. Spring cloud "tasks" can handle spring-batch jobs as well as simple one-off runnable tasks. This approach does not rely on IDE tools to design the services and an additional step to deploy. Development of integration services is the same as development of any other business logic which eliminates the traditional impedance mismatch associated with ESBs and allows for a faster iteration of feature code. The citizen integration framework like Spring Integration provides developers a familiar Java developer's way to compose and operate on services.
One language (Java), one runtime (Tomcat) and one CI/CD process results in the holy grail of reduced time to value and increased efficiency. Both Apache Camel and Spring Integration have a vast number of components, modules and adapters to legacy systems that can aid in this transformation. REST APIs from zConnect should be leveraged natively on the mainframe where possible ditching the man-in-middle legacy adapter layer if possible. z/OS Connect enables z/OS systems such as CICS and IMS to provide RESTful APIs and accepts transactional JSON payloads via the WebSphere Liberty app server.
Apache Camel is in the same category as Spring Integration in terms of being a lightweight framework alternative to ESBs. Apache Camel like Spring Integration is a low level toolbox of components that can be used to implement EAI patterns. Unlike Spring Cloud Stream Apache Camel does not provide higher level microservice abstractions for designing flows other than a DSL.
Phase 4 : Transform
After phase 2 and 3 you are leveraging a lot of Integration patterns aka EAI constructs. Any ESB albeit even light weight cloud native ones inevitably result in context bleeding and a proliferation of interchange contexts resulting from sharing a canonical data model. Entities and Aggregates are outside their Bounded Context or even worse replicated outside, endorsed in a more generic representation, where the integration team has put an extra layer of complexity on top of it.enterprise-arch-in-practice.
Phase 4 is doing away with EAI patterns and leveraging a pure event driven messaging architecture. Decomposition of ESB integration services based on purpose alignment model to higher order vertically sliced domain based service. The intent is to make each service independent. The service provides a smart endpoint. The service fulfills a contract for a bounded context and is complete in all respects. Process orchestration is replaced by choreography. The Spring Cloud framework provides an umbrella of Netflix OSS tools that should be used when writing distributed systems. The service implemented with the help of Spring cloud and the platform are responsible for providing all the qualities of service provided earlier by the ESB. A fabric of independently operating coarsely grained microservices that are choreographed to achieve business function is the ultimate manifestation of the Domain Driven microservice philosophy.
In a choreography approach the central controller service is replaced by the asynchronous flow of events over a message bus. These events are picked up by interested services and appropriate actions are taken which may be message enrichment, transformation, etc,. Since business process flows are now transformed to event and activity flows, business activity monitoring becomes critical. Compensation of failed transactions and exception actions should be incorporated into business logic. A user journey now becomes a series of async interactions. Additional work is needed to ensure that you can monitor and track that the right things have happened.
The decomposition of a monolith to microservices is done by breaking the system into a shared kernel and identifying candidate bounded contexts using techniques like event storming and model-storming. Use an Anti-Corruption Layer to ring-fence clearly distinguishable bounded contexts from their neighbors to allow for extraction. Start with the core domain, move according to value. Transform the monolith using approaches such as Strangler, asset-capture, event-interception and reference-data. A transformation is not a refactoring it’s a debt restructuring program. Usual moves include Shared Kernel -> Customer-Supplier -> Open Host Services -> Published Language. References (ian-cooper, implementing-ddd)
