Spring Boot Microservices Observability > Pivotal PCF Metrics

We are often asked Why the PCF Platform team should install PCF Metrics when the customer already has NewRelic, AppDynamics, Dynatrace, <insert APM tool of choice> and a log aggregator like ELK, Splunk etc., After all PCF Metrics is a resource hog at least in the older versions. Note this has improved and is tunable in later versions.

Now we all know that developers like their shiny toys and given latitude will install the internet on the platform. However there are genuine reasons to install PCFMetrics for app developers.  First a picture is worth 1000 words. So please see graphic and check if the reasons make sense. I have tried to present a biased bulls and bears assessment of PCFMetrics for your developer enterprise needs below.

Wait but what is PCF Metrics ?

Pivotal Cloud Foundry Metrics stores logs, metrics data, and event data from apps running on PCF for the past 14 days. It graphically presents this data to help operators and developers better understand the health and performance of their apps. PCF Metrics will enable development teams to advance microservice resliency and scalability goals  by providing us a single pane of glass for logging, tracing and metrics giving insight into outages, interruptions and scaling events.

Wait but we already have all the data in Kibana and Splunk and NewRelic/AppDynamics ?

Here is what PCFMetrics provides that the current monitoring and logging solutions do not provide > 

1. The java buildpack has inbuilt integration with the metrics forwarder tile and gives us the ability to look at Spring Boot Actuator metrics and custom metrics in PCFMetrics without any additional work. We need the ability to see custom metrics and spring boot actuator metrics in a dashboard, see https://content.pivotal.io/blog/how-pcf-metrics-helps-you-reduce-mttr-for-spring-boot-apps-and-save-money-too for a behind the scenes look on how all this works.

Configuring the New Relic MeterRegistry or any of the other specific Micrometer registries has the benefit of skipping the potentially-lossy loggregator flow in favor of direct communication. Skipping the loggregator in favor of direct communication with a registry highly encouraged if your registry supports dimensional data even if you could sink loggregator metrics to that registry transparently. 

2. The metrics are correlated with logs and we can zoom in immediately into the section of logs that is pertinent to the metric spike. We can also visualize PCF autoscaling events in the dashboard giving us an accurate picture of when and why scaling occurred and the faults in the system. Both these capabilities are not provided by ELK/NewRelic without significant customization. see https://docs.pivotal.io/pcf-metrics/1-6/using.html  You can also configure alerts in PCF metrics via webhooks when values of metrics are crossed or when significant app events take place. This allows developers to build and own what they deliver to the platform and enables developers to engage in SRE practices. Now you can autoscale without PCF Metrics but you just won't have as much historical data on which to base these scaling decisions. One could start creating autoscaling rules without referencing that history. PCF Metrics gives you the tools to make better scaling decisions.

3. From a cost perspective this tile is free and you are entitled to long term support. Example resource configuration to store approximately 14 days of data for a small deployment, about 100 application instances are as follows. The tile be configured in S, M, L and XL resource configurations. From a cost perspective most Dev teams should be able to absorb this cost. We can start small and then resize the deployment when the development team validates the value. If this is a cost issue the sizing can be further reduced by storing only 7 days. 

Now "it's basically free" should never be used an argument for choosing the best tool for monitoring your applications in production in part because tools like ELK and Prometheus also come with no-price-but-the-infrastructure options. Windows like the default two weeks may work in development environments but may not be sufficient for real life app monitoring especially in domains like e-commerce that have distinct rhythm across the day/month/year. 

4. PCF metrics comes with an inbuilt distributed tracing dashboard for all the microservices providing an end to end view of microservice latencies in fanout configurations. This Zipkin  based dashboard that showcases spans and traces is baked in into PCFMetrics and is better than the corresponding e2e support in NewRelic. see https://content.pivotal.io/blog/distributed-tracing-in-pcf-metrics-breakthrough-insight-for-microservices The distributed tracing support in PCFmetrics provides a deep understanding in the causality of outages and other interruptions across microservice fanout and hierarchy calls. 

PCF Metrics can be installed in the non-production PCF environment to do a proper cost/benefit pro/cons analysis. Install the PCFMetrics and the Metrics Forwarder tile to achieve autoscaling to meet the observability, resiliency and meet the non-functional requirements of Microservices. PCF Metrics also supports the day-2 Ops goals of your organization http://cloud.rohitkelapure.com/2019/02/power-of-pcf-metrics.html

PCFMetrics does not displace your enterprise APM and logging tool. It does not provide support today for multi-dimensional metrics and only persists logs for 14 days by default. PCMetrics does not provide any profiling data. There needs to a proper telemetry and APM focused solution alongside PCF Metrics. And if you have something along with it ship metrics directly to that system using the right Micrometer registry. The fact that an application cannot publish metrics directly to PCF Metrics in any way and requires some additional platform support either metrics-forwarder or the new metrics-registrar also means product teams have extra coupling to the teams that maintain the platform or release pipelines.  In absence of PCFMetrics, Teams should have an external log aggregator and metrics platform in place for app teams to own their own destiny regarding how and what they monitor about their apps.

If the platform team offers Prometheus for metrics and visualization, developers should use it. Prometheus has fantastic support for polling of metrics. In most places developers need to open tickets to post custom metrics. For instance in NewRelic you need an Insights API key to post custom metrics and create dashboards. NR and ELK are better than PCFMetrics when it comes to specialized features;  however developers don’t visit these Portals. PCFMetrics also enables the configuration of autoscaling rules on custom metrics and alerts.  Developers are lazy and will do the most convenient thing i.e. pull up the portal from the console which leads me to the bottomline. 

PCFMetrics is the perfect on-boarding portal for developers in non-production environments that enables them to start the journey of observability and then get deeper with specialized tools.

Credit to my colleague Matt Campbell for contributing to this blog and keeping it real!

Tools To Create Chaos

These are interesting tools that I have come across in the last couple of days to create chaos one of the key SRE practices to determine if your production site can handle excess load ...

• ChaosBlade: An easy to use and powerful chaos engineering experiment toolkit from AliBaba. https://github.com/chaosblade-io/chaosblade

• Chaos Toolkit: Create Chaos in your spring apps . https://github.com/chaostoolkit-incubator/chaostoolkit-spring and the chaos monkey for Spring Boot apps https://codecentric.github.io/chaos-monkey-spring-boot/

• Gremlin: Chaos As a Service.  https://www.gremlin.com/docs/application-layer/attacks/ Resiliency through orchestrated chaos. Worth paying for this service if you have low confidence on the production readiness of your code or if you don't have SRE practices to shock the organization into operational readiness.

• General Load Testing tools: http://cloud.rohitkelapure.com/2019/05/load-testing-tools.html

• Istio Fault Injection: https://istio.io/docs/tasks/traffic-management/fault-injection/

Now that you have succeeded in creating chaos how should you instrument and fix the system to deal with the chaos. To understand how to deal with chaos start with Health Checks and Graceful Degradation in Distributed Systems and  Testing in Production- The Safe Way

Other Book Chapters to understand the theory and implementation of SRE practices when dealing with Chaos read the chapters on Handling Overload and Addressing Cascading Failures from the SRE Books. As a bonus read the chapter on Non Abstract Large System Design to understand the design process for designing large scale fault tolerant systems.

Lastly if you are in the Bay Area this looks like  an awesome conference https://chaosconf.io/

Happy SRE Practices! 

Distributed Cloud Native Transactions?

Does the Java world have a good cloud solution for distributed transactions?

@Wael from the Pivotal AppTx team asked this question which triggered the blog post ....

The traditional view of distributed transactions in the new cloud native microservices world is that they are a strict no-no. There are ways to workaround distributed transactions via compensation and eventual consistency and reverting to One-phase commit transactions. These alternatives although perfectly acceptable leave us wanting. Developers want everything and taking away distributed transactions is like pulling away a favorite teddy bear from a toddler.

But wait the tech industry evolves so fast there are other options available now ...  Let me explain ...

Java world NOW does have support for distributed transactions indirectly. For Heterogeneous distributed transactions > indirectly via Kafka and MongoDB and  for Homogeneous distributed transactions  via Hyper Scale Secret Sauce Cloud Provider Expensive databases like Spanner DB and CosmosDB and others ...

Unless you like to pay top dollar to Microsoft or Google I would stay away from proprietary hyper-scale cloud provider databases that provide auto-magic 2 phase commit ACID transactions via atomic clocks synced across datacenters. God forbid you need to migrate away from these databases.

The other options are more palatable :

1. MongoDB has recently added ACID Transaction support see https://www.mongodb.com/transactions and Spring-mongo-data supports that via https://spring.io/blog/2018/06/28/hands-on-mongodb-4-0-transactions-with-spring-data and https://www.baeldung.com/spring-data-mongodb-transactions

2. Kafka - Online Event Processing have enabled achieving consistency where distributed transactions have failed see https://queue.acm.org/detail.cfm?id=3321612 Support for distributed transactions across heterogeneous storage technologies is either nonexistent or suffers from poor operational and performance characteristics. By building on top of immutable persistent ordered event logs OLEP systems are used to provide good performance and strong consistency guarantees in such settings. Use Online Event Processing to implement distributed transactions. For more concrete guidance around distributed transactions with Kafka checkout https://kafka-summit.org/sessions/simplifying-distributed-transactions-sagas-kafka/ which introduces the new Simple Sagas library. Built using Kafka streams, it provides a scalable fault tolerance event-based transaction processing engine. We walk through a use case of coordinating a sequence of complex financial transactions.

Bottomline is that distributed transactions in the cloud are hard due to the heterogenity of storage technologies and the transaction managers for ACID transactions were all built for the pre-cloud era. You will need to approach transactions in the cloud native era with new patterns like sagas and compensation and eventual consistency and rely on the product characteristics like immutable persistent ordered logs or proprietary features to realize your dream of distributed transactions in the cloud.

The Top Ten App Transformations

Pivotal AppTx helps customers migrate applications to cloud native every day. Recently I wondered what are some of the most frequent app transformations and changes we need to make to get an app to cloud native. Is there a way to automate these app transformations of monolithic and legacy applications to Spring Boot ? Which manual transformation when automated will lead to the holy grail of automatic app migration to cloud ? If you are interested please read on ...

1. Spring Bootification : The process of injecting the Spring Boot framework and packaging the application as a fat jar. If you want to go through a comprehensive recipe for bootification look no further than https://pivotal.io/application-transformation-recipes/replatforming/spring-bootification-process . Don't forget the old faithful guide Building an Application with Spring Boot. 

2. Escaping the Dependency Hell : As part of the bootification you will drag in new versions of dependent java libraries. You will need to now go through an exercise of deduping and reconciling dependencies. Typically this well result in an upgrade all dependencies to latest possible versions as part of pom.xml or the build.gradle refactoring. Coexistence and upgrade  of older libraries and frameworks with Spring 2.x, 5.x starters. The maven dependency management plugin is a friend in these cases especially the dependency:tree  command that can be used to determine dependencies that are used and declared; used and undeclared; unused and declared. If you are still in classpath resolution hell and cannot figure out the origin of a class leverage the ClassGraph tool a super-fast and flexible classpath scanner.

3. Externalized Configuration : If you do manage to escape the dependency hell you now get the privilege of isolating and separating the configuration of your application and externalizing it. Typically configuration is strewn in java code, various property files, app server deployment descriptors and sometimes files on VM or bare-metal laid down by the Devops or shared services groups. When taking an app to the cloud the configuration needs to come from an external source so that it can be manipulated by the cloud platform in a consistent fashion. The best approach to take here is to fix the most egregious config options first like hardcoded ports and hardcoded file paths etc. After that take a look at the app-server deployment descriptors and port all the JNDI names and resource names to Spring Beans and configure these via Spring property sources. Eventually when most of the configuration has been externalized you can put it in a config server and organize it based on your deployment environment i.e config is segmented based on dev, test, perf, QA and prod spaces. There are some static code analysis tools that can help you understand the broad impact of these changes. Leverage app-server specific tools like windup and IBM migration-toolkit as well as classic static analysis tools like find-bugs that identify anti-cloud patterns. Pivotal has a tool in this space that identifies all the cloud remediations called Pivotal App Analyzer. Remove any app server specific deployment descriptors and clustering behavior since the clustering of app instances is now a feature provided by the cloud platform.

4. Connections to external data sources : Remove JNDI dependencies. Leverage user Provided data sources along with Spring Cloud Connectors to connect to backend and backing services in the clouds. Here is the recipe for replacing the JavaEE way of connecting to backend resources with the Spring way of datasource beans and wiring the datasources to endpoints in Pivotal Cloud Foundry. see. Spring, Java Buildpack and Pivotal Cloud Foundry does a lot of magic in this area. Remember this golden rule - NO application should *ever* use auto-reconfiguration in production. They should always use Spring Cloud Connectors directly. The right way is to leverage the java-cfenv which allows the developer direct control over parsing VCAP_SERVICES and configuring and connecting to downstream bound services.

5. Logging - Log to stdout and stderr. Before you get along too far you need some window into the state of the app in the cloud. For this purpose you should change the loggers of the app to log to stdout and stderr so these streams can be aggregated and captured in one tool provided by the platform or a trusted third-party like Splunk. Typically this involves 1. Remove logging related jars from the legacy pom.xml 2. Introduce the Cloud Native Logging Facade with a convenience library like lombok or Use SLF4J Logger Directly. Therafter configure Standard Logging by adding standard logging pattern to application.yml. This pattern will log all output to one line, and ensure that the log readers and can parse the log in a way consistent with other projects. Follow this comprehensive recipe for application logging in CloudFoundry.

6. Observability and Healthchecks : One of the cool features of Spring Boot is the ability to add production ready actuators and healthchecks to the project that introspect the datasource beans and configure the right healthchecks chained and invoked by the platform. This should be very easy to do following the step by step documentation defined in production-ready apps. It is also a good idea to define domain specific custom health indicators that give deep insight into the health of the app to the Cloud Foundry health monitor. Recipe for defining custom healthchecks can be found here.

7. Securing applications : Handle user credentials, backing service credentials with Vault or Credhub. Configure user authentication and role based access control authorization with the  Pivotal Tile SSO. Recipe for using Vault as a backing store Config Server. Recipe for Securing Applications with CredHub. Recipe for configuring the Pivotal SSO Tile for apps. Configure Pivotal SSO Automatic Resource Configuration.

8. Remove persistence of State: In-memory state offload to an external cache. Recipe for Offload HTTP Sessions with Spring Session and Redis. Conversion of Entity EJBs to Hibernate or OpenJPA or other ORM. Recipe for Converting an Existing EJB-Based Application to a Spring-Based Application. Conversion of 2pc transactions to 1pc transactions or eventual consistent state. Recipe for How To Deal With XA Global Transactions in the Cloud. Cloud Foundry doesn’t provide a (durable) file system. If your application requires a file system for durable persistence, consider using something like a MongoDB GridFS-based solution or an Amazon Web services-based S3 solution. If your application’s use of the file system is ephemeral—staging file uploads or something—then you can use the Cloud Foundry application’s temporary directory for the life of the request. forklifted-application Another alternative is to use volume service in PCF for persistent data. See recipe on external-file-system.

9. Messaging with distributed brokers. Leverage Spring Cloud Streams as an abstraction over Kafka or RabbitMQ. Recipe for running spring cloud streams with the right broker provider plugin on PCF.

10. CI/CD automation > Build and Deployment Pipelines. Auto generate CI/CD pipelines for your legacy app using Spring Pipelines. spring-cloud-pipelines provides scripts, configuration, and convention for automated deployment pipeline creation for Jenkins and Concourse with Cloud Foundry or Kubernetes. Project Crawler allows for auto generating cloud pipelines. recipe

Load Testing Tools

The mention of load testing tools evokes images of archaic heavy weight license bearing tools like LoadRunner. You are fighting to find time slots for the load clients to run and target your application!  yikes. 

There are lighter weight and convenient tools that every programmer should add to his tool belt for performance testing and load simulation. You will find a listing of such tools  below inspired by a question asked on slack by John Feminella.

A useful thing to note is whether your load generator is focused on generating requests or whether it tries to simulate users. The former is an “open queue” and the latter is a “closed queue”. For the “same” amount of traffic they will behave differently. see: http://www.cs.cmu.edu/~harchol/Papers/nsdi_camera.pdf  thanks Jacques Chester

Say you want to generate a lot of concurrent requests for demo purposes  and want something robust and scriptable then these are the tools you should consider :

1. Apache Bench (https://httpd.apache.org/docs/2.4/programs/ab.html) Prevalent in all major unix and linux based distros and easiest to use. ab -k -c 100 -n 20000 abc.com/hello.html for 100 simultaneous connections emitting 20,000 requests each as fast as they can.
2. Gatling (http://gatling.io/) Cloud based load testing. Load testing as code.
3. Tsung (http://tsung.erlang-projects.org/). Tsung  is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, AMQP, MQTT, LDAP and Jabber/XMPP servers. docs
4. Hey (https://github.com/rakyll/hey)  More heavy duty, Gatling and makes pretty ASCII graphs
5. wrk & wrk2 (https://github.com/wg/wrk) HTTP benchmarking tool that generates significant load when run on a single multi-core CPU. Also see wrk2
6. Siege (https://github.com/JoeDog/siege) Siege is an open source regression test and benchmark utility. It can stress test a single URL with a user defined number of simulated users, or it can read many URLs into memory and stress them simultaneously. siege -c255 -r1000 https://abc.com/greeter/hello
7. Httperf The httperf HTTP load generator. httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance.
8. Hey: https://github.com/rakyll/hey HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom. hey runs provide number of requests in the provided concurrency level and prints stats. It also supports HTTP2 endpoints.

Happy Load & Chaos Testing!

Power of PCF Metrics For Day 2 App Ops

PCF Metrics is a powerful free batteries included Application Monitoring and Management tool that is bundled with PCF. It showcases the power of PCF allowing an enterprise to look at one pane of glass for logging and metrics. PCF Metrics if used wisely reduces the cost of leveraging an expensive log aggregation tool like Splunk and eliminate the use of an APM like Dynatrace or AppDynamics or whatever new flavor shows up in the market claiming AIOps.

So let's talk about some concrete uses cases of app developers using PCF Metrics. First of all PCF Metric is a resource hog when it comes to the Tile install. So there will be some sticker shock.  You want to install the latest version of PCFMetrics i.e. PCF Metrics 1.6. You will also need to install the Metrics Forwarder tile . 

So why do I have to bother with the Metrics Forwarder ? - The metrics forwarder service enables the gathering of fine-grained metrics from the PCF Metrics deployment. The Metrics Forwarder for PCF is a service that allows apps to emit custom metrics to Loggregator and consume those metrics from the Loggregator Firehose. These metrics are then consumed by PCFMetrics via an internal nozzle.

PCF Metrics 1.6 gives you the capability of monitoring custom application metrics  and set monitors and alerts on them and graph them on the dashboard. Yes you can define any application metric using micrometer and that metric shows up in the PCF Metrics dashboard. You can set alerts on metrics and have PCF metrics page you on Slack. This applies not only to custom metrics but also to container metrics like HIGH CPU. What this means is that if you have a non cooperative Ops team - then you can control your destiny and be notified immediately via SLACK when your SLO's are violated and can undertake remediation or triage of a situation immediately by taking thread dumps or heap dumps.

How does all this magic work ?

First you need a certain version of the Java Buildpack v 4.2

You can use Spring Boot Actuators to emit metrics to the Metrics Forwarder API. To do this, perform the following steps:

1. Configure your app to use Spring Boot Actuators.
2. Create the Metrics Forwarder (Tile ver. 1.11.3 )  for PCF service.
3. Bind your app to the Metrics Forwarder for PCF service.
4. Push or restage your app using the Java buildpack v4.2 or later.

Behind the Scenes

https://content.pivotal.io/blog/how-pcf-metrics-helps-you-reduce-mttr-for-spring-boot-apps-and-save-money-too

When an app is bound to the Metrics Forwarder service, the app receives credentials and the URL of the Forwarder API. It uses this information to post Spring Boot Actuator metrics to the Metrics Forwarder tile. This configuration data is stored in VCAP_SERVICES environment variables.When you cf push or cf restage the app, the Java buildpack downloads an additional metrics exporter jar, and includes it within the application droplet. When the app is running, the metrics exporter jar now added to the application context reads the Actuator metrics from a metrics registry every minute. It then posts the data to the Metrics Forwarder URL. From there, the Metrics Forwarder service sends this data to Loggregator. PCF Metrics then reads from the Firehose to ingest metrics data for retention and visualization.

How do you add custom metrics to a Spring Boot App ?

Micrometer!

Micrometer is the metrics collection facility included in Spring Boot 2’s Actuator. It has also been backported to Spring Boot 1.5, 1.4, and 1.3 with the addition of the micrometer-spring-legacy dependency.

The PCF java buildpack includes a Cloud Foundry Spring Boot Metric Writer that  provides an extension to Spring Boot that writes Metrics to a Metric Forwarder service. Here are the gory details 

The CloudFoundryMetricWriterAutoConfiguration through spring boot autoconfig magic creates a RestOperationsMetricPublisher that publishes metrics to the Metrics Forwarder API. Therefore in order to publish metrics to  PCF Metrics you don't need a micrometer-registry. Core-micrometer is enough to get the  metrics published. You don't need to add any additional dependencies for Metric registry extensions. Spring Boot auto-configures a composite MeterRegistry and adds a registry to the composite for each of the supported implementations that it finds on the classpath. In PCF the Java Buildpack is configuring Spring Boot with the PCF Metrics Meter Registry.

You can take action on custom metrics by creating monitors that alert a slack endpoint. See https://docs.pivotal.io/pcf-metrics/1-6/using.html#monitors for configuring the right set of app specific alerts for your application.

For sample code for micrometer metrics in spring boot checkout https://github.com/micrometer-metrics/micrometer-samples-spring-boot and https://spring.io/blog/2018/05/02/spring-tips-metrics-collection-in-spring-boot-2-with-micrometer

Resources

• Using PCFMetrics with your apps 
• https://docs.pivotal.io/pcf-metrics/1-6/api/monitors.html#view-monitors
• https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics-getting-started
• https://micrometer.io/docs/ref/spring/1.5
• https://www.baeldung.com/micrometer
• https://www.youtube.com/watch?v=JAdxO1XboJY

How healthy is your Rabbit ?

We deal with RabbitMQ a lot in our AppTx engagements. If your RabbitMQ or microservices that deal with events and messaging are unhealthy this blog post has some hints towards fixing the issues.

Please remember these two tenets as you diagnose RabbitMQ performance issues with your application

• To achieve predictable RabbitMQ response times, you will want to dedicate an ODB service instance in PCF (if you can leverage isolated zones, the better). This is so that the results are not skewed due to noisy neighbors.
• 99% of the performance problems are generated by the applications and it is extremely rare to be related to misconfiguration of RabbitMQ. Therefore, we advise to simulate the customer's workload, taking into account peak and average load, and using the perf-test tool  to simulate it.

Pivotal Tile for RabbitMQ 3.7.11 has a number of enhancements for health checks of the deployment for an operator. If you are on a RabbitMQ deployment and are wondering about the deployment health point your operators to https://www.rabbitmq.com/blog/2019/02/07/this-month-in-rabbitmq-feb-7-2019/ .

Here are the links you need to benchmark the deployment against Pivotal Validated numbers > Pick a workload and compare.  https://github.com/rabbitmq/workloads

You can also run the performance test on PCF  https://github.com/rabbitmq/rabbitmq-perf-test-for-cf

To test the health of an individual deployment you can use  http://www.rabbitmq.com/monitoring.html#health-checks

If you want to run large workloads follow this link https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/#workloads-with-a-large-number-of-clients

In terms of application resiliency with regards RabbitMQ look at. https://github.com/rabbitmq/workloads/tree/master/resiliency for a number of recommendations.

Use RabbitMQ channel caching: Opening and closing channels frequently is a CPU bound process and does incur performance penalties. One of the best practices to implement is caching of connections and channels. Channels should NOT be shared across thread, but they sure can be shared in the same thread. Use Spring-AMQP abstraction to help with this. The auto configuration created by Spring AMQP creates a Caching ConnectionFactory, which allows connections and channels caching in a thread safe manner. You can read more about RabbitMQ best practices here : https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-common-errors.html

If you are using Spring Cloud Stream you are covered. By default, the RabbitMQ binder uses Spring Boot’s ConnectionFactory, and it therefore supports all Spring Boot configuration options for RabbitMQ and wires in the CachingConnectionFactory.

It is very difficult to determine if a shared PCF environment is in "good shape" - not because certainly the conditions (spare cpu, number of connections, spare memory, etc) of the benchmark are not going to be the same to the conditions at the time when we obtained the baseline. The best we can do is determine a number of metrics of what a healthy deployment looks like. And that will depend on the solution itself.

For instance, if it is massively critical that a hypothetical "incoming request" queue be always below a certain threshold then a metric would be the depth of the queue. Or if in normal circumstances we expect around 100+/10 connections, having more than 150 connections seems like we have a connection leak (something that usually occurs). The platform team should be monitoring RabbitMQ with a tool like prometheus connected with some alerting solution.

- The recommendation of scaling applications based on the depth of a RMQ queue sounds sensible but increasing the number of consumer connections/channels may also produce an excess of load in RabbitMQ defeating the purpose of scaling the number of consumers to help reduce the backlog of messages. So, we need to be cautious here.

- Analyzing "why there are so many messages in RMQ" puts us in the right direction. Monitoring the number of consumers on the queue and the consumer utilization is extremely valuable but also the monitoring the message ingress rate and the ack rate in the application itself.

Thanks to Marcial Rosales and Anwar Chirakkattil  for his guidance on Scaling and Health of RabbitMQ in PCF and Dan Frey for reviewing this article.  

Books I read in 2018 and Predictions

List of books I read in 2018 in prioritized order of importance and impact :

1. Algorithms To Live By

2. The Hard Thing About Hard Things

3. High Growth Handbook

4. High Output Management 

5. Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity 

6. Pre-suasion: A Revolutionary Way to Influence and Persuade

7. Kubernetes In Action 

8. Kubernetes Up & Running

9. Data Intensive Applications

10. The Managers Path

11. Reset
12. Rapid Modernization Of Java Apps

List Of Books In Progress

1. The Back of the Napkin: Solving Problems and Selling Ideas with Pictures 

2. Sell with a Story

Here is a list of books I will read/ work through in 2019 :

1. A Programmers Introduction to Mathematics
2. Istio-In-Action

3. Philosophy Of Software Design
4. Chaos -Engineering eBook. infoQ- eMAG on chaos-engineering
5. Dare To Lead
6. Refactoring: Improving the Design of Existing Code (2nd Edition) 
7. Reactive Spring book by Josh Long
8. The McKinsey Edge: Success Principles from the World’s Most Powerful Consulting Firm 
9. Thinking Fast and Slow

Books I am looking forward to reading 

1. Super Thinking

Magazines That I read

1. ACM Queue
2. Prose Garden
3. OReilly News Letters
4. Thoughtworks Radar

Podcasts That I listen to .. 

Predictions of 2019

Please note that these are not my employers opinions and these predictions are just that => future indicators based on past experience.  

1. Middleware modernization to cloud native tech and services accelerates dramatically

2. Further consolidation in the PaaS space. Pivotal and Docker are all acquisition targets.

3. Kubernetes enters trough of disillusionment. Serverless becomes the new darling particularly from the cost savings perspective. 

4. Backlash against AWS monopoly with increased adoption of multi-cloud and poly-cloud. 

5. ML/AI is infused in every product/ software category. 

The Anti-Architect

A colleague of mine - fellow AppTx practice lead -  Shaun Anderson posed a very interesting question last night

"tell me about the behaviors of all the people you have not had success working with"

 which got me thinking. Our industry has some established norms on what a software architect should do; however we don't have prescriptive guidelines around what an architect should NOT do. What behaviors an architect should not display. To capture these ANTI-architect behaviors , I outline the the architect styles you should probably NOT copy.

1. *Mr. Boxes and Lines*, *Mr. Ivory Tower* : Knows all the latest frameworks and technology choices and answers questions the right way; however could not code up a simple program to save his life.

2. *Mr. Hoarder*: Architects the system in such a way that only he understands the intricacies and holds it close to his chest. Refuses to call his baby ugly.

3. *Mr. Framework*: Frameworkify everything even spring such that all the developers only need to extend the UBER framework at the edges. Assumes developers are dumb and cannot think for themselves and therefore need to be provided abstractions suitable for a seven year old. This   is an egregious form of a hoarder. Sometimes this is taken to the extreme where a meta framework will generate other child frameworks aka Mother Of All Frameworks.

4. *Mr. Kafka-lover*, *Mr. Serverless-lover*, *Mr. Reactive-lover* : Answer is Kafka OR Answer is Serverless... What is the question ? This is an architect who falls prey to Microservices-envy, Kafka-as-an-esb and layered-tech-driven architecture anti-patterns. Answer is r2dbc - Now what is the question ?

5. *Mr. Async* : Makes the genuine mistake of equating event driven architecture to event sourcing. Does not realize the four forms of event driven architecture. see  Many meanings of event driven architecture

6. *Mr. Pussyfoot* Hesitates to make any architectural decision without deferring to a central committee.

7. *Mr. Shit-it-all* : Periodically swoops down from an ivory tower to shit all over the current project architecture and implementation leaving behind a bunch of half-ass diktats without full context.

8. *Mr. Manager-upper*: Architect only shows up when his boss or adversary shows up and is largely absent without their presence.

9. *Mr. Idiomatic*: Architect does not engage in fruitful debate and unwilling to reassess their state. They have made up their mind and are not willing to experiment or concede that their world view may be wrong.

10. *Mr. Cowboy*: Solution first - ask questions later.  Indulges in premature optimization or premature solutioning without fully understanding the question, intent or context.

11. *Ms. Alice in Wonderland*: jumps deep into each and every rabbit hole one can find

12. *Ms. I Told You So* : As soon as something goes just a bit sideways drops the “I told you this was not going to work and here’s why” in front of your stakeholders.

13. *Mrs. Nit Picker*:  The architect who needs to understand every single detail prior to coding anything.

14. *Ms. But Why*: When you don’t understand something 100% and the only thing you can do to prove your presence is to ask “But Why” >  on repeat.

As an architect we all make these mistakes and there is no shame in admitting that; however recognizing some of these behaviors  in ourselves and others helps us improve and become better architects that build something real that delights our customers and stakeholders.

Performance Profiling a Ruby application on the PCF Pivotal Application Service

Here is how we would go about debugging a ruby app on
http://engineering.pivotal.io/post/debugging-ruby-memory-issues-cloud-foundry-cloud-controller/

You will also do well to read https://www.oreilly.com/library/view/ruby-performance-optimization/9781680501681/

This poor-man's stats gathering tool can be really helpful (credit Will Sulzer)
https://gist.github.com/kelapure/f24a0e27aa12f1d3364a93b5289e7ec2. However, you'd need to modify the code to run in a block passed to the measure method. This comes from the Ruby Performance Optimization Book on Safari.

Ruby-Prof is another nice tool (https://github.com/ruby-prof/ruby-prof).. and then the real heavy-weight tools trace back to the native calls using Valgrind etc.. Unfortunately, the linked script may tell you what the Ruby program running in the JVM is doing.. but maybe not why it's consuming JVM memory to load the interpreter, etc. By default JRuby disables ObjectSpace because it's the *cause* of memory problems (it causes the runtime to maintain an additional list of weak references), BTW.

Checkout https://github.com/jruby/jruby/wiki/PerformanceTuning for JRuby specifics. You can enable ObjectSpace if you have code that depends on it, but it's highly discouraged in JRuby, and would likely spoil the tests.

That begs the question why should you run Ruby apps on Pivotal Cloud Foundry ?  What advantages does the Ruby BP on PAS

• It allows for the code to run in isolation
• It allow for resiliency with healthchecks
• It allows for custom metrics and alerts via PCF Metrics
• Makes binding to external services easier
• Log aggregation
• Manage all the app instances in apps-manager
• Leverage Zipkin and Spring Cloud Services
• Leverage task support to run one-off tasks
• Ruby apps are one of the first ones to run in a PAS. Remember PCF was inspired from Heroku that only had support for Ruby initially.
• Support for ruby apps in Concourse (edited)
• Autoscale Ruby apps based on app metrics including latency, throughput, CPU, memory and custom ones.

What are the disadvantages

• Debugging is hard in production
• Memory limits on container are hard

via GIPHY

Many thanks to Chris Umbel and Will Sulzer who contributed the majority of this post.