Distributed Cloud Native Transactions?

Does the Java world have a good cloud solution for distributed transactions?

@Wael from the Pivotal AppTx team asked this question which triggered the blog post ....

The traditional view of distributed transactions in the new cloud native microservices world is that they are a strict no-no. There are ways to workaround distributed transactions via compensation and eventual consistency and reverting to One-phase commit transactions. These alternatives although perfectly acceptable leave us wanting. Developers want everything and taking away distributed transactions is like pulling away a favorite teddy bear from a toddler.

But wait the tech industry evolves so fast there are other options available now ...  Let me explain ...

Java world NOW does have support for distributed transactions indirectly. For Heterogeneous distributed transactions > indirectly via Kafka and MongoDB and  for Homogeneous distributed transactions  via Hyper Scale Secret Sauce Cloud Provider Expensive databases like Spanner DB and CosmosDB and others ...

Unless you like to pay top dollar to Microsoft or Google I would stay away from proprietary hyper-scale cloud provider databases that provide auto-magic 2 phase commit ACID transactions via atomic clocks synced across datacenters. God forbid you need to migrate away from these databases.

The other options are more palatable :

1. MongoDB has recently added ACID Transaction support see https://www.mongodb.com/transactions and Spring-mongo-data supports that via https://spring.io/blog/2018/06/28/hands-on-mongodb-4-0-transactions-with-spring-data and https://www.baeldung.com/spring-data-mongodb-transactions

2. Kafka - Online Event Processing have enabled achieving consistency where distributed transactions have failed see https://queue.acm.org/detail.cfm?id=3321612 Support for distributed transactions across heterogeneous storage technologies is either nonexistent or suffers from poor operational and performance characteristics. By building on top of immutable persistent ordered event logs OLEP systems are used to provide good performance and strong consistency guarantees in such settings. Use Online Event Processing to implement distributed transactions. For more concrete guidance around distributed transactions with Kafka checkout https://kafka-summit.org/sessions/simplifying-distributed-transactions-sagas-kafka/ which introduces the new Simple Sagas library. Built using Kafka streams, it provides a scalable fault tolerance event-based transaction processing engine. We walk through a use case of coordinating a sequence of complex financial transactions.

Bottomline is that distributed transactions in the cloud are hard due to the heterogenity of storage technologies and the transaction managers for ACID transactions were all built for the pre-cloud era. You will need to approach transactions in the cloud native era with new patterns like sagas and compensation and eventual consistency and rely on the product characteristics like immutable persistent ordered logs or proprietary features to realize your dream of distributed transactions in the cloud.

The Top Ten App Transformations

Pivotal AppTx helps customers migrate applications to cloud native every day. Recently I wondered what are some of the most frequent app transformations and changes we need to make to get an app to cloud native. Is there a way to automate these app transformations of monolithic and legacy applications to Spring Boot ? Which manual transformation when automated will lead to the holy grail of automatic app migration to cloud ? If you are interested please read on ...

1. Spring Bootification : The process of injecting the Spring Boot framework and packaging the application as a fat jar. If you want to go through a comprehensive recipe for bootification look no further than https://pivotal.io/application-transformation-recipes/replatforming/spring-bootification-process . Don't forget the old faithful guide Building an Application with Spring Boot. 

2. Escaping the Dependency Hell : As part of the bootification you will drag in new versions of dependent java libraries. You will need to now go through an exercise of deduping and reconciling dependencies. Typically this well result in an upgrade all dependencies to latest possible versions as part of pom.xml or the build.gradle refactoring. Coexistence and upgrade  of older libraries and frameworks with Spring 2.x, 5.x starters. The maven dependency management plugin is a friend in these cases especially the dependency:tree  command that can be used to determine dependencies that are used and declared; used and undeclared; unused and declared. If you are still in classpath resolution hell and cannot figure out the origin of a class leverage the ClassGraph tool a super-fast and flexible classpath scanner.

3. Externalized Configuration : If you do manage to escape the dependency hell you now get the privilege of isolating and separating the configuration of your application and externalizing it. Typically configuration is strewn in java code, various property files, app server deployment descriptors and sometimes files on VM or bare-metal laid down by the Devops or shared services groups. When taking an app to the cloud the configuration needs to come from an external source so that it can be manipulated by the cloud platform in a consistent fashion. The best approach to take here is to fix the most egregious config options first like hardcoded ports and hardcoded file paths etc. After that take a look at the app-server deployment descriptors and port all the JNDI names and resource names to Spring Beans and configure these via Spring property sources. Eventually when most of the configuration has been externalized you can put it in a config server and organize it based on your deployment environment i.e config is segmented based on dev, test, perf, QA and prod spaces. There are some static code analysis tools that can help you understand the broad impact of these changes. Leverage app-server specific tools like windup and IBM migration-toolkit as well as classic static analysis tools like find-bugs that identify anti-cloud patterns. Pivotal has a tool in this space that identifies all the cloud remediations called Pivotal App Analyzer. Remove any app server specific deployment descriptors and clustering behavior since the clustering of app instances is now a feature provided by the cloud platform.

4. Connections to external data sources : Remove JNDI dependencies. Leverage user Provided data sources along with Spring Cloud Connectors to connect to backend and backing services in the clouds. Here is the recipe for replacing the JavaEE way of connecting to backend resources with the Spring way of datasource beans and wiring the datasources to endpoints in Pivotal Cloud Foundry. see. Spring, Java Buildpack and Pivotal Cloud Foundry does a lot of magic in this area. Remember this golden rule - NO application should *ever* use auto-reconfiguration in production. They should always use Spring Cloud Connectors directly. The right way is to leverage the java-cfenv which allows the developer direct control over parsing VCAP_SERVICES and configuring and connecting to downstream bound services.

5. Logging - Log to stdout and stderr. Before you get along too far you need some window into the state of the app in the cloud. For this purpose you should change the loggers of the app to log to stdout and stderr so these streams can be aggregated and captured in one tool provided by the platform or a trusted third-party like Splunk. Typically this involves 1. Remove logging related jars from the legacy pom.xml 2. Introduce the Cloud Native Logging Facade with a convenience library like lombok or Use SLF4J Logger Directly. Therafter configure Standard Logging by adding standard logging pattern to application.yml. This pattern will log all output to one line, and ensure that the log readers and can parse the log in a way consistent with other projects. Follow this comprehensive recipe for application logging in CloudFoundry.

6. Observability and Healthchecks : One of the cool features of Spring Boot is the ability to add production ready actuators and healthchecks to the project that introspect the datasource beans and configure the right healthchecks chained and invoked by the platform. This should be very easy to do following the step by step documentation defined in production-ready apps. It is also a good idea to define domain specific custom health indicators that give deep insight into the health of the app to the Cloud Foundry health monitor. Recipe for defining custom healthchecks can be found here.

7. Securing applications : Handle user credentials, backing service credentials with Vault or Credhub. Configure user authentication and role based access control authorization with the  Pivotal Tile SSO. Recipe for using Vault as a backing store Config Server. Recipe for Securing Applications with CredHub. Recipe for configuring the Pivotal SSO Tile for apps. Configure Pivotal SSO Automatic Resource Configuration.

8. Remove persistence of State: In-memory state offload to an external cache. Recipe for Offload HTTP Sessions with Spring Session and Redis. Conversion of Entity EJBs to Hibernate or OpenJPA or other ORM. Recipe for Converting an Existing EJB-Based Application to a Spring-Based Application. Conversion of 2pc transactions to 1pc transactions or eventual consistent state. Recipe for How To Deal With XA Global Transactions in the Cloud. Cloud Foundry doesn’t provide a (durable) file system. If your application requires a file system for durable persistence, consider using something like a MongoDB GridFS-based solution or an Amazon Web services-based S3 solution. If your application’s use of the file system is ephemeral—staging file uploads or something—then you can use the Cloud Foundry application’s temporary directory for the life of the request. forklifted-application Another alternative is to use volume service in PCF for persistent data. See recipe on external-file-system.

9. Messaging with distributed brokers. Leverage Spring Cloud Streams as an abstraction over Kafka or RabbitMQ. Recipe for running spring cloud streams with the right broker provider plugin on PCF.

10. CI/CD automation > Build and Deployment Pipelines. Auto generate CI/CD pipelines for your legacy app using Spring Pipelines. spring-cloud-pipelines provides scripts, configuration, and convention for automated deployment pipeline creation for Jenkins and Concourse with Cloud Foundry or Kubernetes. Project Crawler allows for auto generating cloud pipelines. recipe

Load Testing Tools

The mention of load testing tools evokes images of archaic heavy weight license bearing tools like LoadRunner. You are fighting to find time slots for the load clients to run and target your application!  yikes. 

There are lighter weight and convenient tools that every programmer should add to his tool belt for performance testing and load simulation. You will find a listing of such tools  below inspired by a question asked on slack by John Feminella.

A useful thing to note is whether your load generator is focused on generating requests or whether it tries to simulate users. The former is an “open queue” and the latter is a “closed queue”. For the “same” amount of traffic they will behave differently. see: http://www.cs.cmu.edu/~harchol/Papers/nsdi_camera.pdf  thanks Jacques Chester

Say you want to generate a lot of concurrent requests for demo purposes  and want something robust and scriptable then these are the tools you should consider :

1. Apache Bench (https://httpd.apache.org/docs/2.4/programs/ab.html) Prevalent in all major unix and linux based distros and easiest to use. ab -k -c 100 -n 20000 abc.com/hello.html for 100 simultaneous connections emitting 20,000 requests each as fast as they can.
2. Gatling (http://gatling.io/) Cloud based load testing. Load testing as code.
3. Tsung (http://tsung.erlang-projects.org/). Tsung  is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, AMQP, MQTT, LDAP and Jabber/XMPP servers. docs
4. Hey (https://github.com/rakyll/hey)  More heavy duty, Gatling and makes pretty ASCII graphs
5. wrk & wrk2 (https://github.com/wg/wrk) HTTP benchmarking tool that generates significant load when run on a single multi-core CPU. Also see wrk2
6. Siege (https://github.com/JoeDog/siege) Siege is an open source regression test and benchmark utility. It can stress test a single URL with a user defined number of simulated users, or it can read many URLs into memory and stress them simultaneously. siege -c255 -r1000 https://abc.com/greeter/hello
7. Httperf The httperf HTTP load generator. httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance.
8. Hey: https://github.com/rakyll/hey HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom. hey runs provide number of requests in the provided concurrency level and prints stats. It also supports HTTP2 endpoints.

Happy Load & Chaos Testing!

Power of PCF Metrics For Day 2 App Ops

PCF Metrics is a powerful free batteries included Application Monitoring and Management tool that is bundled with PCF. It showcases the power of PCF allowing an enterprise to look at one pane of glass for logging and metrics. PCF Metrics if used wisely reduces the cost of leveraging an expensive log aggregation tool like Splunk and eliminate the use of an APM like Dynatrace or AppDynamics or whatever new flavor shows up in the market claiming AIOps.

So let's talk about some concrete uses cases of app developers using PCF Metrics. First of all PCF Metric is a resource hog when it comes to the Tile install. So there will be some sticker shock.  You want to install the latest version of PCFMetrics i.e. PCF Metrics 1.6. You will also need to install the Metrics Forwarder tile . 

So why do I have to bother with the Metrics Forwarder ? - The metrics forwarder service enables the gathering of fine-grained metrics from the PCF Metrics deployment. The Metrics Forwarder for PCF is a service that allows apps to emit custom metrics to Loggregator and consume those metrics from the Loggregator Firehose. These metrics are then consumed by PCFMetrics via an internal nozzle.

PCF Metrics 1.6 gives you the capability of monitoring custom application metrics  and set monitors and alerts on them and graph them on the dashboard. Yes you can define any application metric using micrometer and that metric shows up in the PCF Metrics dashboard. You can set alerts on metrics and have PCF metrics page you on Slack. This applies not only to custom metrics but also to container metrics like HIGH CPU. What this means is that if you have a non cooperative Ops team - then you can control your destiny and be notified immediately via SLACK when your SLO's are violated and can undertake remediation or triage of a situation immediately by taking thread dumps or heap dumps.

How does all this magic work ?

First you need a certain version of the Java Buildpack v 4.2

You can use Spring Boot Actuators to emit metrics to the Metrics Forwarder API. To do this, perform the following steps:

1. Configure your app to use Spring Boot Actuators.
2. Create the Metrics Forwarder (Tile ver. 1.11.3 )  for PCF service.
3. Bind your app to the Metrics Forwarder for PCF service.
4. Push or restage your app using the Java buildpack v4.2 or later.

Behind the Scenes

https://content.pivotal.io/blog/how-pcf-metrics-helps-you-reduce-mttr-for-spring-boot-apps-and-save-money-too

When an app is bound to the Metrics Forwarder service, the app receives credentials and the URL of the Forwarder API. It uses this information to post Spring Boot Actuator metrics to the Metrics Forwarder tile. This configuration data is stored in VCAP_SERVICES environment variables.When you cf push or cf restage the app, the Java buildpack downloads an additional metrics exporter jar, and includes it within the application droplet. When the app is running, the metrics exporter jar now added to the application context reads the Actuator metrics from a metrics registry every minute. It then posts the data to the Metrics Forwarder URL. From there, the Metrics Forwarder service sends this data to Loggregator. PCF Metrics then reads from the Firehose to ingest metrics data for retention and visualization.

How do you add custom metrics to a Spring Boot App ?

Micrometer!

Micrometer is the metrics collection facility included in Spring Boot 2’s Actuator. It has also been backported to Spring Boot 1.5, 1.4, and 1.3 with the addition of the micrometer-spring-legacy dependency.

The PCF java buildpack includes a Cloud Foundry Spring Boot Metric Writer that  provides an extension to Spring Boot that writes Metrics to a Metric Forwarder service. Here are the gory details 

The CloudFoundryMetricWriterAutoConfiguration through spring boot autoconfig magic creates a RestOperationsMetricPublisher that publishes metrics to the Metrics Forwarder API. Therefore in order to publish metrics to  PCF Metrics you don't need a micrometer-registry. Core-micrometer is enough to get the  metrics published. You don't need to add any additional dependencies for Metric registry extensions. Spring Boot auto-configures a composite MeterRegistry and adds a registry to the composite for each of the supported implementations that it finds on the classpath. In PCF the Java Buildpack is configuring Spring Boot with the PCF Metrics Meter Registry.

You can take action on custom metrics by creating monitors that alert a slack endpoint. See https://docs.pivotal.io/pcf-metrics/1-6/using.html#monitors for configuring the right set of app specific alerts for your application.

For sample code for micrometer metrics in spring boot checkout https://github.com/micrometer-metrics/micrometer-samples-spring-boot and https://spring.io/blog/2018/05/02/spring-tips-metrics-collection-in-spring-boot-2-with-micrometer

Resources

• Using PCFMetrics with your apps 
• https://docs.pivotal.io/pcf-metrics/1-6/api/monitors.html#view-monitors
• https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics-getting-started
• https://micrometer.io/docs/ref/spring/1.5
• https://www.baeldung.com/micrometer
• https://www.youtube.com/watch?v=JAdxO1XboJY

How healthy is your Rabbit ?

We deal with RabbitMQ a lot in our AppTx engagements. If your RabbitMQ or microservices that deal with events and messaging are unhealthy this blog post has some hints towards fixing the issues.

Please remember these two tenets as you diagnose RabbitMQ performance issues with your application

• To achieve predictable RabbitMQ response times, you will want to dedicate an ODB service instance in PCF (if you can leverage isolated zones, the better). This is so that the results are not skewed due to noisy neighbors.
• 99% of the performance problems are generated by the applications and it is extremely rare to be related to misconfiguration of RabbitMQ. Therefore, we advise to simulate the customer's workload, taking into account peak and average load, and using the perf-test tool  to simulate it.

Pivotal Tile for RabbitMQ 3.7.11 has a number of enhancements for health checks of the deployment for an operator. If you are on a RabbitMQ deployment and are wondering about the deployment health point your operators to https://www.rabbitmq.com/blog/2019/02/07/this-month-in-rabbitmq-feb-7-2019/ .

Here are the links you need to benchmark the deployment against Pivotal Validated numbers > Pick a workload and compare.  https://github.com/rabbitmq/workloads

You can also run the performance test on PCF  https://github.com/rabbitmq/rabbitmq-perf-test-for-cf

To test the health of an individual deployment you can use  http://www.rabbitmq.com/monitoring.html#health-checks

If you want to run large workloads follow this link https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/#workloads-with-a-large-number-of-clients

In terms of application resiliency with regards RabbitMQ look at. https://github.com/rabbitmq/workloads/tree/master/resiliency for a number of recommendations.

Use RabbitMQ channel caching: Opening and closing channels frequently is a CPU bound process and does incur performance penalties. One of the best practices to implement is caching of connections and channels. Channels should NOT be shared across thread, but they sure can be shared in the same thread. Use Spring-AMQP abstraction to help with this. The auto configuration created by Spring AMQP creates a Caching ConnectionFactory, which allows connections and channels caching in a thread safe manner. You can read more about RabbitMQ best practices here : https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-common-errors.html

If you are using Spring Cloud Stream you are covered. By default, the RabbitMQ binder uses Spring Boot’s ConnectionFactory, and it therefore supports all Spring Boot configuration options for RabbitMQ and wires in the CachingConnectionFactory.

It is very difficult to determine if a shared PCF environment is in "good shape" - not because certainly the conditions (spare cpu, number of connections, spare memory, etc) of the benchmark are not going to be the same to the conditions at the time when we obtained the baseline. The best we can do is determine a number of metrics of what a healthy deployment looks like. And that will depend on the solution itself.

For instance, if it is massively critical that a hypothetical "incoming request" queue be always below a certain threshold then a metric would be the depth of the queue. Or if in normal circumstances we expect around 100+/10 connections, having more than 150 connections seems like we have a connection leak (something that usually occurs). The platform team should be monitoring RabbitMQ with a tool like prometheus connected with some alerting solution.

- The recommendation of scaling applications based on the depth of a RMQ queue sounds sensible but increasing the number of consumer connections/channels may also produce an excess of load in RabbitMQ defeating the purpose of scaling the number of consumers to help reduce the backlog of messages. So, we need to be cautious here.

- Analyzing "why there are so many messages in RMQ" puts us in the right direction. Monitoring the number of consumers on the queue and the consumer utilization is extremely valuable but also the monitoring the message ingress rate and the ack rate in the application itself.

Thanks to Marcial Rosales and Anwar Chirakkattil  for his guidance on Scaling and Health of RabbitMQ in PCF and Dan Frey for reviewing this article.  

Books I read in 2018 and Predictions

List of books I read in 2018 in prioritized order of importance and impact :

1. Algorithms To Live By

2. The Hard Thing About Hard Things

3. High Growth Handbook

4. High Output Management 

5. Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity 

6. Pre-suasion: A Revolutionary Way to Influence and Persuade

7. Kubernetes In Action 

8. Kubernetes Up & Running

9. Data Intensive Applications

10. The Managers Path

11. Reset
12. Rapid Modernization Of Java Apps

List Of Books In Progress

1. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures 

2. Sell with a Story

Here is a list of books I will read/ work through in 2019 :

1. A Programmers Introduction to Mathematics
2. Istio-In-Action

3. Philosophy Of Software Design
4. Chaos -Engineering eBook. infoQ- eMAG on chaos-engineering
5. Dare To Lead
6. Refactoring: Improving the Design of Existing Code (2nd Edition) 
7. Reactive Spring book by Josh Long
8. The McKinsey Edge: Success Principles from the World’s Most Powerful Consulting Firm 
9. Thinking Fast and Slow

Books I am looking forward to reading 

1. Super Thinking

Magazines That I read

1. ACM Queue
2. Prose Garden
3. OReilly News Letters
4. Thoughtworks Radar

Podcasts That I listen to .. 

Predictions of 2019

Please note that these are not my employers opinions and these predictions are just that => future indicators based on past experience.  

1. Middleware modernization to cloud native tech and services accelerates dramatically

2. Further consolidation in the PaaS space. Pivotal and Docker are all acquisition targets.

3. Kubernetes enters trough of disillusionment. Serverless becomes the new darling particularly from the cost savings perspective. 

4. Backlash against AWS monopoly with increased adoption of multi-cloud and poly-cloud. 

5. ML/AI is infused in every product/ software category. 

The Anti-Architect

A colleague of mine - fellow AppTx practice lead -  Shaun Anderson posed a very interesting question last night

"tell me about the behaviors of all the people you have not had success working with"

 which got me thinking. Our industry has some established norms on what a software architect should do; however we don't have prescriptive guidelines around what an architect should NOT do. What behaviors an architect should not display. To capture these ANTI-architect behaviors , I outline the the architect styles you should probably NOT copy.

1. *Mr. Boxes and Lines*, *Mr. Ivory Tower* : Knows all the latest frameworks and technology choices and answers questions the right way; however could not code up a simple program to save his life.

2. *Mr. Hoarder*: Architects the system in such a way that only he understands the intricacies and holds it close to his chest. Refuses to call his baby ugly.

3. *Mr. Framework*: Frameworkify everything even spring such that all the developers only need to extend the UBER framework at the edges. Assumes developers are dumb and cannot think for themselves and therefore need to be provided abstractions suitable for a seven year old. This   is an egregious form of a hoarder. Sometimes this is taken to the extreme where a meta framework will generate other child frameworks aka Mother Of All Frameworks.

4. *Mr. Kafka-lover*, *Mr. Serverless-lover*, *Mr. Reactive-lover* : Answer is Kafka OR Answer is Serverless... What is the question ? This is an architect who falls prey to Microservices-envy, Kafka-as-an-esb and layered-tech-driven architecture anti-patterns. Answer is r2dbc - Now what is the question ?

5. *Mr. Async* : Makes the genuine mistake of equating event driven architecture to event sourcing. Does not realize the four forms of event driven architecture. see  Many meanings of event driven architecture

6. *Mr. Pussyfoot* Hesitates to make any architectural decision without deferring to a central committee.

7. *Mr. Shit-it-all* : Periodically swoops down from an ivory tower to shit all over the current project architecture and implementation leaving behind a bunch of half-ass diktats without full context.

8. *Mr. Manager-upper*: Architect only shows up when his boss or adversary shows up and is largely absent without their presence.

9. *Mr. Idiomatic*: Architect does not engage in fruitful debate and unwilling to reassess their state. They have made up their mind and are not willing to experiment or concede that their world view may be wrong.

10. *Mr. Cowboy*: Solution first - ask questions later.  Indulges in premature optimization or premature solutioning without fully understanding the question, intent or context.

11. *Ms. Alice in Wonderland*: jumps deep into each and every rabbit hole one can find

12. *Ms. I Told You So* : As soon as something goes just a bit sideways drops the “I told you this was not going to work and here’s why” in front of your stakeholders.

13. *Mrs. Nit Picker*:  The architect who needs to understand every single detail prior to coding anything.

14. *Ms. But Why*: When you don’t understand something 100% and the only thing you can do to prove your presence is to ask “But Why” >  on repeat.

As an architect we all make these mistakes and there is no shame in admitting that; however recognizing some of these behaviors  in ourselves and others helps us improve and become better architects that build something real that delights our customers and stakeholders.

Performance Profiling a Ruby application on the PCF Pivotal Application Service

Here is how we would go about debugging a ruby app on
http://engineering.pivotal.io/post/debugging-ruby-memory-issues-cloud-foundry-cloud-controller/

You will also do well to read https://www.oreilly.com/library/view/ruby-performance-optimization/9781680501681/

This poor-man's stats gathering tool can be really helpful (credit Will Sulzer)
https://gist.github.com/kelapure/f24a0e27aa12f1d3364a93b5289e7ec2. However, you'd need to modify the code to run in a block passed to the measure method. This comes from the Ruby Performance Optimization Book on Safari.

Ruby-Prof is another nice tool (https://github.com/ruby-prof/ruby-prof).. and then the real heavy-weight tools trace back to the native calls using Valgrind etc.. Unfortunately, the linked script may tell you what the Ruby program running in the JVM is doing.. but maybe not why it's consuming JVM memory to load the interpreter, etc. By default JRuby disables ObjectSpace because it's the *cause* of memory problems (it causes the runtime to maintain an additional list of weak references), BTW.

Checkout https://github.com/jruby/jruby/wiki/PerformanceTuning for JRuby specifics. You can enable ObjectSpace if you have code that depends on it, but it's highly discouraged in JRuby, and would likely spoil the tests.

That begs the question why should you run Ruby apps on Pivotal Cloud Foundry ?  What advantages does the Ruby BP on PAS

• It allows for the code to run in isolation
• It allow for resiliency with healthchecks
• It allows for custom metrics and alerts via PCF Metrics
• Makes binding to external services easier
• Log aggregation
• Manage all the app instances in apps-manager
• Leverage Zipkin and Spring Cloud Services
• Leverage task support to run one-off tasks
• Ruby apps are one of the first ones to run in a PAS. Remember PCF was inspired from Heroku that only had support for Ruby initially.
• Support for ruby apps in Concourse (edited)
• Autoscale Ruby apps based on app metrics including latency, throughput, CPU, memory and custom ones.

What are the disadvantages

• Debugging is hard in production
• Memory limits on container are hard

via GIPHY

Many thanks to Chris Umbel and Will Sulzer who contributed the majority of this post. 

Pivotal AppTx

Pivotal's approach to Application Transformation with help from Rohit Kelapure + Yaroslav Novytskyy. Want more details? Check this whitepaper by Matt Russell: https://lnkd.in/gMrHmRJ

How to choose between PAS (Cloud Foundry - PaaS) and PKS(Kubernetes - CaaS)

This seems to be the question on the top of everybody's mind.  There are multiple ways of framing this decision. Several decision trees have been drawn up on this topic including these

credit @jxxf

App Transformation Decision Tree

These diagrams can be summarized as follows from a PAS and PKS perspective as follows : PKS is ideal for stateful and persistent pinned workloads, commercially packaged software, short-lived apps/workloads, software distributed via Helm chart, apps using non-standard port behavior and legacy, zero-factor, apps and complex apps already packaged as docker images and well along the containerization journey. PAS is ideal for custom-built software targeting Windows or Linux, software packaged as ear, jar and war files, web applications, APIs, batch jobs and streaming and reactive applications.

Looking at this decision tree someone who is further along on the dockerization journey may comment that the decision tree is biased towards PAS. You may take the view that if the workload(app) requires ANY or non-zero code changes to migrate to the PAS then the default destination should be PKS. How does one resolve this conflict?

The scientific method creates a hypothesis and then validate/refute the assumptions that led to the hypothesis. In this blog post, I will explain the science between choosing a destination (Cloud Foundry or Kubernetes) for your workload and establishing a migration factory that drives the transformation of all your apps to the right destination in the cloud.

Why should PAS (Cloud Foundry) be the default Choice?

First, let me explain why the default choice in the above picture is PAS aka Cloud foundry. Developers should operate at the highest level of abstraction. The easiest place to change and test your code is in in-place in the inner loop.  Cloud Foundry allows you to use the current currency of your developers i.e. war, jar, ear files.  Cloud Foundry provides a set of top-notch validated developer abstractions for running applications whereas Kubernetes provides a top-notch platform to build a platform. Kelsey Hightower has reinforced this

Kubernetes is an infrastructure framework. It's YAML based configuration files and the kubectl command line tool make it approachable to developers, but far from the developer productivity, you find in a PaaS or FaaS platform.

A common principle in manufacturing is that we should always detect and fix any problem in the production process at the lowest-value stage possible. When developing applications Cloud Foundry gives you a chance to develop faster and get productive by focusing on the inner loop of development and not worrying about non-functional concerns like service discovery, resiliency, stability, routing, security.  K8s is maturing very fast and these app concerns are being baked into the platform via various CNCF Projects and SIGs. There are some promising projects that are improving the K8s development experience including Istio, Knative, Buildpacks, and spring-cloud-kubernetes; however, the beauty of an opinionated platform like CF that these choices are already made and you don't have to roll out a custom stack every time you develop an app. See challenges of containerization here.  also don't realize that the constraints of first generation PaaS systems (Heroku, Google Cloud Engine, CF v1) are all gone.  Many developers, architects, and managers still think of those first-generation PaaS constraints when considering PCF, and specifically, the Pivotal Application Service (PAS). Richard Seroter has demolished this myth in his 5 part series.

What Outcomes are you driving for the Cloud Acceleration / Migration Factory?

Technical decisions taken in a vacuum without influence from business drivers are destined to fail. Therefore before picking a choice, it is critical to examine the business motivators and your specific constraints.  So what is the science between picking the right destination for your workload ?. Its a combination of three factors - 1. technical feasibility 2. business value and 3. human factors. Any factor can tilt the decision to go towards PAS or PKS. This decision is also on a per deployable basis. For a large logical app you may components and modules on both PAS or PKS. 

1. Technical Feasibility: Where does the app fall on the cloud-native spectrum of 15 factors? Cloud Nativity can be ascertained through a tool like SNAP or automation via the various scanners. This analysis is done on the current state and has NO cloud influence. A scoring system is established with 0 being completely cloud-non-native, persistent and stateful and with 15 being completely cloud-native. It is helpful to think of workloads along an axis like this one ...

2. Business Value: Next a determination needs to be made on the strategic business value of this application. Is this application under heavy development. Are the feature and functions of this application critical to the survival and growth of our organization. A scoring of the application under business factors needs to be made. So what are the business factors? They can look like these ...

• Ongoing Development Cost
• Infrastructure Cost 
• Software License Cost
• Operations Cost 
• Overall value to the Business  
• Lead time for Changes
• Business Priority
• Business Criticality
• User Satisfaction 

A score between 1-10 is determined based on a weighted average of all the contributed factors. 

3. Human Factors: Once the technical feasibility score and business value of an app is determined it is time to determine the wave of applications by arranging the apps in a matrix. Apps can be rearranged here arbitrarily based on knowns and unknowns that escaped the technical and business feasibility analysis. As the destination, PAS or PKS is chosen it is critical to understand the outcomes derived from each choice of platform and transformation activity such as rehost  (0 code changes also called lift' n ' shift, little or no configuration changes), replatform, refactor or rebuild.

• Rehost: Containerize to “lift and shift” into Pivotal Container Services (PKS)
• Replatform:  Upgrade an application from its existing platform adhering to the least possible 15 factors to get it to run on PAS, preserving existing functionality
• Refactor: Changes to apps with high business priority, transactional load  to get them to 15 factor Cloud Native using Cloud Native architectural patterns
• Rebuild: Leverage DDD techniques to deconstruct and migrate a complex and monolithic application to the cloud. 

The benefits of containerization to PKS include decreased infrastructure use, automated zero touch deployments with CI/CD,  reduce extensive & manual change management processes by using CI/CD and increased Multi-Cloud portability. If the underlying architecture and tech stack remain unchanged most of the gains are in OPEX related in operational efficiency.

However; rehosting to PKS will not eliminate the cost of proprietary stacks (RHEL, WebSphere, Weblogic, TIBCO) whereas replatforming will typically lead to the elimination of proprietary middleware licenses and decreased effort to patch & upgrade of software by the platform.

A more comprehensive refactoring or rebuild to a cloud-native application running on PAS yields CAPEX benefits like decreased time to scale, decreased MTTR for applications, proactive monitoring of KPI’s of applications,  increased deployment Frequency with CI/CD, reduced Lead Time, reduction of tight service coupling, increased automated testing & test data management as part of CI/CD pipeline and all this leads to increased developer productivity and satisfaction.

Containerization is the only fraction of the opportunity. Driving efficiency of the developer via XP practices and a product mindset on a PaaS is the whole opportunity. 80% of your development cost is labor and people, not infrastructure. Improve productivity by increasing leverage and producing better and faster with the same team.

What does the Cloud Migration Factory look like?

The process part of the migration factory where we take in a large number of raw materials and assemble meaningful parts was explained earlier. Once we have all the components in a factory it is time to assemble a coherent meaningful product. This is where a funnel and a codification of the process above helps. Once all the data is visualized you need to plan the waves of apps again based on the outcomes in the transformation program. It is critical that we measure key indicators and journey markers to ensure that we are realizing the outcomes planned before. These KPIs could be as varied as Percentage of portfolio running on PCF, Number of developers enabled on cloud-native, App Transformation decision framework in place, amount of time taken from idea to production and Developer Engagement with the platform and ROI from infrastructure and license consolidation.

In the end, none of the benefits of the cloud be it PAS or PKS or PFS can ONLY be realized with a change in the existing agilefall or waterfall-agile development process. It is critical to implement a value stream that emphasizes pace and progressive delivery. Without the confidence of automated tests and removal of headaches from continuous deployment, a cloud platform (Bare-metal, IaaS, CaaS, PaaS) from God won't help realize the desired outcomes.

Finally whatever your journey - please remember the law of the hammer - "If the only tool you have is a hammer, you treat everything as if it were a nail." Fight this cognitive bias and leverage the right choice PAS AND PKS for your apps driving the business outcomes that matter. 

Good Luck!

Credits:  

The blog post is a synthesis of the work of many colleagues in Pivotal and Pivotal AppTx including Richard Seroter, Joe Szodfridt, Shaun Anderson, Vinay Upadhya and others. You can checkout the Pivotal AppTx mission at https://pivotal.io/application-transformation and our whitepaper at https://content.pivotal.io/application-modernization/pivotal-practices-application-transformation