Moving Data Off An Mainframe

Recent developments have reinforced the need for change in our political system. In the same vein the blog post below provides a path to unlocking your data and compute from mainframe.  The content in this post is a straight ripoff from my buddy and colleague David Malone's postulation on this topic. The steps below from Dave and are the most cogent explanation of how to move data away from mainframe to distributed systems. It draw's from Dave's direct experience at a major retailer.  The same approach can be leveraged to starve and strangle data locked in any data store.

The Thin Line Between Application Replatforming and Refactoring

When migrating applications to the cloud there is a very thin line between application replatforming and refactoring. These terms are defined as follows: 

• Replatforming: Make the least amount of changes to move it to the cloud aka "lift-tinker-and-test-and-shift" 
• Refactoring: Modify the applications such that it becomes cloud native. Reimagine how the application is architected and developed, typically using cloud-native features.

What defines cloud native? Well we can use 12 factor/15 factor app as one set of heuristics to determine cloud nativeness. We will therefore explore the boundary between replatforming and refactoring along the 15 factors since they are as good as any, ubiquitous language to talk about the effort and the changes that differentiate lift-and-shift vs refactor. The examples below illustrate what it means to replatform vs refactor the application. The context of these words changes in some cases from applications to environment.

TL;DR: Always replatform first and then based on business and technical strategic objectives refactor. The line between replatform and refactor is fuzzy.

One Codebase, One Application:
Adding automated maven or ant based build such that each build runs consistently and generates a deployable artifact is replatforming. If the code is not in source control checking in the code and applying a bug tracking system counts as replatforming. Moving the code from svn to git or across SCMs is refactoring. Modifying the build such that the single codebase is  split and reorganized among teams dedicated to individual apps and microservices is refactoring. If you split your code base into multiple microservices or shared libraries or common API then it counts as refactoring. 

API First:
If you apply the extract API from class pattern and use it to strangulate the monolith then  that is replatforming. If you have an API designed by product designer based on UX or user story flow mapping and are retrofitting implementation to this new API then it counts as refactoring. Putting API management tools or ambitious API gateways counts as refactoring. If you are fixing the implementation of an existing API i.e. modernizing from Axis1/2 to Spring REST MVC or switching the payload from XML to JSON then it counts as replatforming. If you are making semantic changes or additions or deletions to an existing API that falls in the category of refactoring. Getting messaging working with WebSphere MQ counts as replatforming whereas replacing WebSphere MQ with JMS or AMQP  is refactoring.  

Dependency Management: 
Modifying the existing packaging of the application such that all of the dependencies including the application server are vendored into the app like the spring boot fat jar packaging qualifies as refactoring. Modifying the packaging of the application such that a single monolithic ear file is broken into multiple war files that are deployed separately falls under replatforming.  Using an application server buildpack to vendor in your app dependencies like the TomEE or JBOSS buildpacks counts as replatforming. Removing dependence on older frameworks and JavaEE and building the app with spring-boot-starters counts as refactoring. Spring bootification i.e. the process of converting an app to spring boot of the app, ultimately leads to mavenization conflicts where the newer dependencies dragged in by boot and its cousins fight with the older frameworks ossified in the app. Surprisingly a large part of refactoring is spent harmonizing these dueling sets of dependencies.  

Design, Build, Release, Run: 
Using a CI tool like Jenkins to manage the CI pipeline counts as replatforming; however if you define the deployment pipeline through code instead of configuring a running CI/CD tool then you are refactoring. If you leverage blue/green deployment then you are replatforming; however if you are using feature flags and dark launching i.e. adhering to NO breaking changes then you are refactoring. If you do no* upfront design and dive straight into making the code run on the platform then you are replatforming; however if you indulge in model design with bounded contexts and all the good stuff with DDD you are in refactoring territory. If you run the application with the actuators that come by default with Spring Boot you are replatforming; however if you find your self writing a ton of custom actuator metrics and health end points you may be refactoring.

Configuration, Credentials and Code: 

If you have brute forced your configuration by getting rid of all of your configuration files and then went back through your codebase and modified it to expect all of those values to be supplied  by environment variables then you are replatforming. If you have modified your code to expect configuration based on profiles from an external configuration server then you are refactoring. If your credentials are fetched from an external service then you are refactoring; however if your credential management requires encrypted keys as environment variables or hard coded in environment specific property files then you are replatforming.

Logs: 

If you modify your application send logs to stdout and stderr you have replatformed the app. If you have modified the traces across your apps to take advantage of distributed tracing frameworks like Zipkin and corresponding log aggregation, indexing and visualization tools like ELK or Grafana you have refactored the app.

Disposability: 

If your application starts slowly and you have tinkered with timeout settings of the platform to allow the health checks to pass then you have replatformed. If you have fixed the slow app startup and shutdown issues then you have refactored the application. If your application is a mix of web request processing and batch tasks then you have simply replatformed the app; however if you have separated batch processing from the request or message processing bits into separate apps then you have refactored.

Backing Services:

If your cache or sessions are within the JVM or offloaded them to an external DB then you have replatformed. If you have eliminated the need to keep persistent state across requests then you have indulged in refactoring. If the configuration of the backing service happens outside of the application  i.e. there is no coupling between the app and the specific backing service then you have refactored the application. If the configuration and binding of the backing service is done explicitly within the application then you have replatformed the app. If you have protected your backing service interaction with circuit breakers then you have refactored whereas if you have simply gotten the backend service configured correctly with the app with externalized configuration then you have replatformed the app.

Environment Parity: 
If you have 8 different environments across 3 different tiers, each with their own databases, firewall rules and app specific configurations in properties files then you have replatformed the environment; however if you have a one-touch CI/CD deployment to production with appropriate process gates then you have refactored your environment. Unless every commit is a candidate for deployment and the gap between production and test environments basically comes down to different prioritized infrastructure resource pools you are replatforming. 

Administrative Processes:

If you have separated your admin processes like crons, database migrations, singleton services from your parent app then you are a refactorer otherwise if your app is a jack of all trades implementing different concerns then you are a replatformer. If you have extracted your one-off processes and implemented them using batching frameworks like Spring Batch or Spring Cloud Streams or Spring Cloud Data Flow then you have certainly refactored the app.

Port Binding: 
Table stakes here is eliminating hard-coded dependencies on specific ports for startup. When you have eliminated all other network protocols and refactored the application to exclusively communicate exclusively over HTTP then you have refactored the app. If you have NOT micromanaged your port assignments and can run with the ports exposed by the container you have replatformed the app. If you rely on a dynamic service registry like Eureka or Consul to discover other micro services and their ports then you have refactored the app.

Stateless Processes:

If all the long running state is external to application persisted in backing services then you have replatformed the application. If you have rearchitected the application to get rid of the need to carry long running state then you have refactored the application. Remediating the app dependency on a persistent filesystem like NFS or local disk to instead leverage s3 or sftp  is replatforming; however relying on an external service that provides a file caching abstraction or a user level file system like HDFS is refactoring the application.

Concurrency:
If the most efficient way to scale the application, is to vertically scale resources like memory or CPU or I/O then all you have done is replatformed the app; however if the app can be horizontally scaled without bound to maintain the same SLAs with a linear increasing load then you have refactored the application.

Telemetry: 

If your approach to monitoring involves third party APM tools and other health checks baked into the platform with Spring boot actuator libraries then you have replatformed; however if you have baked in business domain specific monitoring and metrics that can be leveraged by the business for A/B testing and used by the team to discuss feature rollout or blue/green deployment then you have reached refactoring nirvana.

Authentication & Authorization:

If you rely on integration with SiteMinder or other agent based protocols like Kerberos and have them working in the cloud via  a route service or via an API gateway or via a route interceptor then you are replatforming. On the other hand if the application is refactored to use federated lighter-weight user and application space based identity and authorization protocols like OAuth2, OpenID Connect and JWT then you have a reformed app on your hands.