Data The Forgotten Middle Child of the Cloud Native Family

Most of the discussion around cloud native revolves around writing cloud native or cloud friendly 15 factor apps, decomposing monolithic applications with DDD or containerizing legacy apps with dockerization to Kubernetes, Cloud Foundry or other container orchestration platforms.

Unfortunately everyone forgets about Data!! 

The best resource I have found for data migration is https://www.martinfowler.com/articles/evodb.html and https://www.slideshare.net/ThoughtWorks/data-patterns-andrew-jones-by-thoughtworks. The last chapter of the Cloud Native Patterns book by Cornelia Davis has a chapter dedicated to this topic. 

Q. How do we manage the migration of data from the legacy services to the modernized services?  Some of our tables have millions of records and hundreds of columns.  

[Branch-by-abstraction] enables rapid deployment with feature development that requires large changes to the codebase. For example, consider the delicate issue of migrating data from an existing store to a new one. This can be broken down as follows:

1. Require a transition period during which both the original and new schemas exist in production
2. Encapsulate access to the data in an appropriate data type. Expose a Facade service to  encapsulate DB changes.
3. Modify the implementation to store data in both the old and the new stores. Move logic and constraints to the edge aka services
4. Bulk migrate existing data from the old store to the new store.
5. This is done in the background in parallel to writing new data to both stores.
6. Modify the implementation to read from both stores and compare the obtained data. Implement retry and compensations. Database Transformation Patterns cataloged like Data sync, data replication and migrating data.
7. Leverage techniques like TCP Proxy for JDBC to understand the flow of data and transparently intercept traffic. Use Change Data Capture tooling to populate alternate datastores.
8. When convinced that the new store is operating as intended, switch to using the new store exclusively (the old store may be maintained for some time to safeguard against unforeseen problems).

Managing Persistence: You will need to choose between creating a new DB or letting the old and new implementations share the same datastore. Separating the DBs is more complex if you need to keep them in sync, but it gives you a lot more freedom. If your old and new applications share a datastore, you’ll need to build a translation layer to translate between the old and new models. If you give your old and new applications separate datastores, be prepared to invest a lot of effort in tooling to synchronize the two DBs. If your DB synchronization mechanism writes directly to the DB, be careful you don’t violate any assumptions the application makes about being the sole writer.[Re-engineering Legacy Software]. Splitting data for microservices involves breaking foreign key relationships and managing constraints in the resulting services rather than at the database level. For shared mutable data you may need to split the schemas , keep the service together before splitting the application out into separate microservices. By splitting the schemas  but keeping the application code together, we can revert our changes or continue to tweak things without impacting any consumers of our service. Once we are satisfied that the DB separation makes sense, we can then think about splitting out the application code into two services.[Refactoring Databases]

Role of Streaming in Data Decomposition

The road to data flows through Kafka - We count on Kafka for consistency, strict ordering, replay, durability and auditability. Using microservices gives the ability to plug in new modules for encoding and enrichment in real time.CDC, data pumps, facades, caching and Event Shunting are emerging patterns  that allow streaming data platform teams to offer mainframe and legacy RDBMS events to microservices teams. Each team can build appropriate persistence and achieve multi-DC replication with streaming platform.

https://www.confluent.io/blog/spring-for-apache-kafka-deep-dive-part-2-apache-kafka-spring-cloud-stream

Realtime Inventory with Spring, Kafka and Cassandra - a three way punch

Spring , Kafka and Cassandra combination are quickly becoming the market standard reference architecture for realtime inventory and related systems.

Here a quick list of compiled reference links provided by Patrick Druley from Confluent 

Nordstrom's

https://www.youtube.com/watch?v=wLQbArGw29o&feature=youtu.be

Target

https://www.youtube.com/watch?v=JLO83v6JdbM

Walmart (via Walmart Labs)

https://kafka-summit.org/sessions/kafka-streams-scale-2/

Macy's 

https://www.youtube.com/watch?v=bzH86d_jOg0

If you are wondering how to engineer  a realtime view of the inventory and want to modernize from a batch based systems you should start with these videos. More on this theme in later blog posts. Stay tuned. 

Power of Six Week Projects at 3.25 Engineers

shhh! I have a secret to tell.  As a Principal Solutions Architect in the VMware Pivotal Labs App Modernization practice I have to often scope app modernization projects where we decompose or rewrite massive monoliths, Batch processes, ETL or port legacy apps at scale to Cloud Foundry or Kubernetes platforms.  After a scoping session sometimes you struggle determining the exact duration of the project. At Pivotal we always bias shipping to production and working in incremental chunks i.e. fixed time variable scope. In situations like these where I am not sure about the scope my go to default  - from the gut after working in this space for 5 years - is a team of 3.25 @ 6 weeks.

As humans we always seek approval and are especially prey to confirmatory Bias. The recent ShapeUp Book from Jason Fried and DHH validated my natural affinity for 6 week projects. They have articulated brilliantly what I have been struggling to say ...

" After years of experimentation we arrived at six weeks. Six weeks is long enough to finish something meaningful and still short enough to see the end from the beginning. "

In fact they have a full chapter dedicated to Six Week Cycles in their Book ShapeUp. If you would like to go on a confirmatory bias victory lap on a weekend, you would do no wrong by binge reading this chapter and the full book.

Now the next question to ask how about the 3.25 size projects ? First of all the .25 is a product manager who helps in Inception and is a client liaison  at critical times. Sometimes for a challenging project we have a full time Product Manager. The rest of the team are 3 engineers or 2 engineers and 1 designer depending on a mix of frontend and backend work.  But WHY a team of 3.25 and not 2.25 or 4.25 or 5.25. Here I put forth a picture to explain the communication overhead as

Brook's Law

A team size of one is almost never recommended unless the project duration is less than a week. This leads to loneliness and burnout over the long term. There is a reason most startup founders are asked to look for a co-founder. Our projects are mini-startup projects that sometimes involve the same level of stress and entrepreneurship. With a team of two there is no fallback i.e. if one engineer is out of commission then the entire load falls on the other engineer and we are back to burnout and loneliness.

The ideal starter team is 3 people with high fidelity communication and load balancing across team members. When the team size bumps to 4, you now need 6 lines of communication adding unnecessary overhead.  At a size of 4, if each developer eats 3 slices of pizza, then you are at 12 slices and two pizza team. In a product development R&D setup that is fine; however when the team is on the road on an entirely unfamiliar territory you need to extremely nimble and tight. Therefore from a people happiness, productivity and mental health perspective the ideal team size is 3.25. In case of big projects it is OK to ramp this up to 4.25; however when team size increases to more than 4 and you should split the team into two teams of 3 engineers.

TL;DR When in doubt go with 3.25 @ 6 weeks.

Happy Hunting!

Event Storming - A Pivotal Practice for decomposing applications

FIELDS + GUIDANCE	Guidance
Name of method	Event Storming https://speakerdeck.com/rkelapure/event-storming
What is this method ?	Event Storming is a cross functional facilitation technique for revealing the bounded contexts, microservices, vertical Slices, trouble spots and starting points for a system or business process.
Phases	Discovery, Kick Off
Suggested Time	1 - 2 hours.
Who participates?	SMEs, Core Team (see facilitator notes)
Why do it?	Event Storming enables decomposing monoliths into microservices. It allows for modeling new flows and ideas, synthesizing  knowledge and facilitating active group participation without conflict to time travel and ideate the next generation of a software system.
When to do it?	When you need to make sense of a Huge mess to enable Cross Perspective Communication as a force function for clarity.
What supplies are needed?	People, tools and supplies needed to conduct an ES session A large wall (for stickies) At least 4 different colored stickies Sharpies Blue painters tape Refreshments  like water and soda and juices for hydration Paper flip boards for readouts and breakouts
How to Use this Method	Event Storming is a group exercise to scientifically explore the domains and problem areas of a monolithic application. The most concise description of the process of event storming comes from Vaughn Vernon's DDD-Distilled book and the color around the process comes from Alberto Brandolini's book Event Storming. Storm the business process by creating a series of Domain events on sticky notes. The most popular color to use for domain events is orange. The DomainEvent is a verb stated in past tense and represent a state transition in the domain. Write the name of the DomainEvent on an orange sticky note. Place the sticky notes on your modeling surface in time order from left to right. As you go through the storming session, you will find trouble spots in your existing business process. Clearly mark these with a purple/red stick notes. Use vertical space to represent parallel processing. After all the events are posted experts will post locally ordered sequence of events and enforce a timeline. Enforcing a timeline triggers long awaited conversations and eventually STRUCTURE will emerge.  These event clumps or common groupings give us our notional service candidates (actors or aggregates depending on how rigid the team is with DDD definitions).  These will be used during the Boris Exercise.
Success/Expected Outcomes “You know you are done when…”	- Event Storming generates an immense backlog of user stories. - Perform  User Story Mapping to Map and organize stories into MVPs - Define scope of the problem - Confirm that you are solving the right problem ?
Facilitator Notes & Tips	Event Storming is a technique used to visualize complex systems and processes. This could range from monoliths to value streams. Event Storming is gamestorming technique for harnessing and capturing the information captured in a group’s minds. It surfaces conflicts and different perspectives of a complex system and bubbles up the top constraints and problem spots.   As an event storming facilitator you have one job - create a safe environment for the exchange and output of ideas and data. The job is 50% technical facilitation and 50% soft people facilitation where you are reading body language. A single facilitator can typically orchestrate groups of 15-20. For a group of 30 or more you need two facilitators.  ES is usually conducted in two phases. A high level event storm to identify the domains and then a subsequent ES into a top constraint - the core domain. The language of ES is stickies. In its simplest form ES is basically a facilitated group story telling. The stickies represent domain events -  or things that happened in the past. The trouble spots are identified by orange/red stickies. The color of the stickies does not matter. What does matter is that you start simple and then add the notation incrementally. Start simple and then add the information in layers. ES can serve many goals - break down a monolith into constituent bounded contexts, create a value stream - as a way to onboard employees, etc., There is no ONE correct style of ES. Every session is different from another based on the desired goals and outcomes. So don’t worry about getting it right- just do it and roll your own style. An ES is only successful if the right people are involved. This is a mix of business domain experts, customer executives, stakeholders, business analysts, software developers, architects, testers, and folks who support the product in production. Subject matter experts, product owners and developers that knows and understand the application domain. This process enables cross perspective conversation throughout the team as well as a standard definition of the terms used by both technical and non-technical team members.
Related practices	BORIS SNAP
Real world example	Deconstructing Monoliths With Domain Driven Design
Recommended reading	Motivation behind ES can be found here - Gamestorming: A Playbook for Innovators, Rulebreakers, and Changemakers. Note this book is available on safari books online: https://www.safaribooksonline.com/library/view/gamestorming/9781449391195/ This is the book we read on the flight to Boston Eventstorming https://leanpub.com/introducing_eventstorming Book is written by Alberto Brandolini the father and inventor of Eventstorming Domain Driven Design (DDD) - provides the theoretical underpinnings of decomposing  ‘/monoliths. DDD Distilled is the perfect book to understand the science of DDD and how ES fits into the grander scheme of things - how do the ES artifacts translate into  software design, architecture and an actual backlog.

Learnings from Implementing enterprise event driven architecture

There are four different types of event driven architecture[3]. Docket Based Choreography pattern is one of our inventions that allows us to design and operationalize an event driven architecture for a legacy system. It involves event notification but is a specific implementation. Typical event driven reengineering of a monolith involves both sync and async flows. CQRS and Event Sourcing - one of the en vogue forms of event driven architecture is hard resulting in to 50% failure rate in projects. Practicing domain driven design and carving out bounded contexts and vertical slices is hard. You have to stick to first principles after the system is decomposed to stay true to the event driven architecture. We leverage Kafka a lot primarily as a messaging broker. Developers struggle with ACID guarantees in Event driven systems. Online event processing provides a way to make a system eventually consistent [6].

Typical pitfalls encountered in engineering to microservices from monoliths are Incidental Coupling of microservices and shared data model across microservices. in some cases the microservices reverted to shared canonical domain models. CDC driven decomposition of monoliths is rare due to data silos. Successful architecture and app transformation requires change in culture. Event Shunting Pattern [4] allows for gradual transformation for legacy to a modern stream event driven architecture. Among many things, the pace of change dictates the boundary of the microservice. Leverage past experience to see which features/components change over time. Avoid glorious central model. Central authority make changes to the canonical model. Avoid the trap of making a giant canonical model by staying true to guiding principles. Large messages as events vs small messages build on this. Large canonical messages can force many changes to the model happen. Managing regulatory, security, PII, encrypted data at rest are painful. Small messages that trigger data fetches through an API have been much more successful towards a sustainable architecture.

References

1. Evolutionary_design_still_requires_up_front_thinking.html
2. http://nealford.com/memeagora/2015/03/30/architecture_is_abstract_until_operationalized.html
3. https://martinfowler.com/articles/201701-event-driven.html
4. https://medium.com/@KevinHoffman/migrating-apps-to-the-cloud-shunting-the-event-stream-8c2f6f309242
5. https://fs.blog/2018/04/reversible-irreversible-decisions/
6. https://queue.acm.org/detail.cfm?id=3321612
7. https://content.pivotal.io/blog/agile-architecture

Death To the Kubernetes YAML! Long Live the Manifest Generators

You are wallowing in a wall of YAML, wondering where your inner loop developer productivity disappeared. You are wandering the annals of the internet to find the kubectl equivalent of cf push. You are a refugee from the land of the Platform As A Service wandering aimlessly in the Container As a Service world. The land of Kubernetes is intimidating for those of us who are used to the higher order abstractions afforded by platforms like Cloud Foundry or Heroku

In this series of blog posts a refugee from PaaS who has crossed  the chasm, will be your coach as you navigate this perilous journey. We will cover equivalence of concepts from Cloud Foundry to Kubernetes. We will draw out the distinctions of architecture & developer workflows across K8s and Cloud Foundry. You will get a deep understanding of the tools & the confidence necessary to make YAML your new best friend and conquer K8s to 10x your productivity developing in the Kubernetes native way.

So here it goes - episode 1 - Death to YAML long live the manifest generators!

What are the options when it comes to a cf push like experience on kubernetes. I am going to keep a running list of K8s YAML template generators here that generate K8s manifests for app deployments. Developers start with these tools below to escape death by YAML on first contact with kubernetes.

1. Fabric8.io has a java DSL for kubernetes. The fabric8.io kubernetes java client. Istio and other projects use it.  fabric8 library is a single jar - It works in airgapped environments. Provides a Typesafe Kubernetes-manifest DSL for JVM-based apps
2. The primary interface to kubectl is YAML. Pulumi exposes a rich, multi-language SDK to create API resources, and additionally supports execution of Kubernetes YAML manifests and Helm charts. 
3. kf provides cloud foundry users a familiar workflow experience on top of Knative.
4. Google Cloud Code - Google Cloud Code provides IDE support for the full development cycle of Kubernetes applications, from creating a cluster to deploying your finished application. https://cloud.google.com/code/docs/intellij/
5. Lift - Spring 2 Cloud Internal hygen inspired YAML generator. Templating engine is pluggable - and based on mustache/handlebars ATM. Pivotal internal only at the moment. 
6. Docker Enterprise 3.0 Simplifies Kubernetes Management  - It can identify and build, from Docker Hub, the containers needed and creates the Docker Compose and Kubernetes YAML files, Helm charts, and other required configuration settings.
7. Develop with Java on Kubernetes using Azure Dev Spaces - Generate the Docker and Helm chart assets for running the application in Kubernetes using the azds prep command.

In the next blogpost I will cover the automation toolchain for building docker images from source. Tools like jib, pivotal build service, cloud-native-buildpacks, s2i etc fall in that category. 

Honorable Mentions

• Helm 3.0
• ytt - k14s tools
• Zeit 
• Kabanero & Appsody 

The Future Of Observability and Developer Business Intersect Dashboards

Not sure if y'all came across https://thenewstack.io/observability-a-3-year-retrospective/

I really liked this part and for me resonated with the value of PCF Metrics. We have a single unified firehose of information available to us which helps us achieve the things Susan mentions i.e. figure out the unknown unknowns ... 

The Future of Observability

Three short years into this ride, I ponder the question; What’s next and where will this movement take us? I believe that in the next ~3 years, all three of those categories — APM, monitoring/metrics, logs, and possibly others — are likely to cease to exist. There will only be one category: observability. And it will contain all the insights you need to understand any state your system can get itself into.
After all, metrics, logs, and traces can trivially be derived from arbitrarily wide structured events; the reverse is not true.
Users are going to start to figure out that they are paying multiple times to store single data sets they should only have to store once. There is no reason to invest budget with separate monitoring vendors, logs vendors, tracing vendors, or APM vendors. If you collect data in arbitrarily wide structured events, you can infer metrics from those, and if you automatically append some simple span identifiers, you can use those same events for tracing views. Not only can you cut spending by 3-4X, but it’s phenomenally more powerful if you can use a single tool and fluidly flip back and forth between the big picture (“there’s a spike”) and drilling down to the exact raw events with the errors. Next, compute what outlier values they have in common, trace one of them, locate wherein the trace a problem lives, and figure out who else is impacted by that specific outlier behavior. All conducted in one single solution with all teams getting the same level of visibility.
Right now this is either a) impossible, or b) a human being has to copy-paste an ID from one system to another to the next. This is wasteful, slow, and cumbersome, and extremely frustrating for the teams that have to do this when trying to solve a problem. Tools create silos and siloed teams spend too much time arguing about the nature of reality instead of the problem at hand.

In the same vein - wonder what a perfect App Metrics dashboard looks like for the organization ?
Here is a sample soup 2 nuts source to business OKRs dashboard that you should emulate

Failures in Microservices

As microservices evolve into a tangled mess of synchronous and asynchronous flows with multi-level fanouts it becomes important to think about failure and resiliency since that is pretty much a guaranteed outcome when the availability of the whole system is a multiplicative of all its downstream microservices and dependencies.

How does one systematically think about handling load, graceful degradation and load shedding in the face of impaired operation and sustained high load ?  Google's SRE books contain excellent high level advice as it pertains to handling load and addressing cascading failures. I have prepared a actionable summary of a couple of chapters dealing with resiliency to win in the face of failure. Follow the notes here to create

Rigor and governance around Microservices frameworks and templates to enable systematic resiliency through circuit breakers and autoscaling for sustainable scale out of your System of Systems.

https://landing.google.com/sre/sre-book/chapters/addressing-cascading-failures/

https://landing.google.com/sre/workbook/chapters/managing-load/

Different types of resources can be exhausted

Insufficient CPU > all requests become slower > various secondary effects

  1. Increased number of inflight requests

  2. Excessively long queue lengths

    - steady state rate of incoming requests > rate at which the server can process requests

  3. Thread starvation

  4. CPU or request starvation

  5. Missed RPC deadlines

  6. Reduced CPU caching benefits

Memory Exhaustion - as more in-flight requests consume more RAM, response, and RPC objects

  1. Dying containers due to OOM Killers

  2. A vicious cycle - (Increased rate of GC in Java, resulting in increased CPU usage)

  3. Reduction in app level cache hit rates

Threads (Tomcat HTTP )

  1. Thread starvation can directly cause errors or lead to health check failures.

  2. If the server adds threads as needed, thread overhead can use too much RAM.

  3. In extreme cases, thread starvation can also cause you to run out of process IDs.

File descriptors

• Running out of file descriptors can lead to the inability to initialize network connections, which in turn can cause health checks to fail.

Dependencies among resources  

•   Resource exhaustion scenarios feed from one another
•   DB Connections (Negative Indicator)

All this can ultimately lead to Service Unavailability > Resource exhaustion can lead to servers crashing leading to snowball effect.

How To Prevent Server Overload

1.  Load test the server’s capacity limits,
2. Serve degraded results
3.  Instrument servers to reject requests when overloaded - fail early and cheaply
4. Instrument higher-level systems to reject requests at reverse proxies, by limiting the volume of requests by criteria such as IP address, At the load balancers, by dropping requests when the service enters global overload and at At individual tasks
5. Perform capacity planning

Making Eureka Service Discovery Responsive on PCF

Making Service Discovery More Responsive on PCF

Eureka service registries and Eureka clients are tuned for cloud scale deployment of applications. This results in tuning their respective server and client side caches to account for network brownouts and self-preservation in case of network partitions, failed compute or storage.

All this has the effect of the service registry getting stale sometimes especially in autoscaling or auto descaling scenarios. If autoscaling and scaling down is happening very fast then the REST or HTTP clients sometimes experience timeouts since the service IPs are outdated and the service registry has not been updated to the latest set of microservices app instances.

We ran into one such issue on PCF and configured Service Discovery in three ways to eliminate timeouts. Much of this detailed experimentation was done by my colleague Rohit Bajaj. 

1. Ribbon Ping Configuration
2. BOSH DNS Polyglot discovery 
3. Eureka server configuration to eliminate server timeouts 

Summary

We Determined that Native Polyglot service discovery provided by Cloud Foundry as the optimal Configuration for Service Discovery. Error rate drops to 0-1% in the auto descaling scenarios  as opposed to > 2% with other settings.

For your average  spring, spring-boot,  Java app that requires service discovery Eureka Service Registry fits the bill nicely; however if your workload is highly dynamic and you need close to 0 error rate when load balancing across transient service instances then BOSH DNS is better. When using Bosh DNS you don’t use Ribbon. So Custom Circuit Breaker + BOSH DNS replaces Ribbon + Eureka. 

1. Ribbon Ping Configuration

See https://thepracticaldeveloper.com/2017/06/28/how-to-fix-eureka-taking-too-long-to-deregister-instances/

2. BOSH DNS Discovery

Polyglot service discovery introduces new capabilities with a familiar workflow. An app developer can configure an internal route to create a DNS entry for their app. This makes the app discoverable on the container network. A DNS lookup on an internal route returns a list of container IPs for applications corresponding to that particular internal route. docs

We recommend pairing BOSH service discovery with a robust circuit breaker like Resilience4J Sample code on Reslience4J + BOSH Polyglot discovery

3 Eureka Settings

Ideal setting for tuning Eureka clients and servers to be super responsive 

# Ribbon Settings 

ribbon.ServerListRefreshInterval = 50

# Eureka Client

eureka.instance.lease-renewal-interval-in-seconds = 10
eureka.client.initialInstanceInfoReplicationIntervalSeconds = 10
eureka.client.instanceInfoReplicationIntervalSeconds = 10
eureka.client.registryFetchIntervalSeconds = 10

# Eureka Server

eureka.instance.lease-expiration-duration-in-seconds = 30
eureka.server.eviction-interval-timer-in-ms = 20 * 1000
eureka.server.responseCacheUpdateIntervalMs = 10 * 1000
eureka.server.getWaitTimeInMsWhenSyncEmpty = 10 * 1000

Typically changing Eureka server side settings is not possible when the Eureka server is provisioned by the Spring Cloud Services tile.

# Eureka Server Self-Preservation

eureka.server.enableSelfPreservation = false

Self Preservation window is every 15 minutes
  DEFAULT = 30s interval for `lease-renewal-interval`
  N = 24
  Number of heartbeats per minute to trigger self-preservation < 2 * 24 * 0.85 = 41 heartbeats

  NEW = 10s interval , N = 24
  Number of heartbeats per minute to trigger self-preservation < 6 * 24 * 0.85 = 123 heartbeats

# References

- Spring Cloud Netflix Eureka - The Hidden Manual https://blog.asarkar.org/technical/netflix-eureka/
- The Mystery of Eureka self-preservation https://medium.com/@fahimfarookme/the-mystery-of-eureka-self-preservation-c7aa0ed1b799
- Eureka http://netflix.github.io/ribbon/ribbon-eureka-javadoc/index.html
- Spring Cloud Ribbon https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-ribbon.html
- B/g for Eureka https://app-transformation-cookbook-internal.cfapps.io/duplicates/replatforming/blue-green-with-eureka/3e7f94f6c49795ef347b70141a36c134/

# CODE

https://github.com/Netflix/eureka/blob/master/eureka-client/src/main/java/com/netflix/discovery/EurekaClientConfig.java
https://github.com/Netflix/eureka/wiki/Eureka-REST-operations
https://github.com/Netflix/eureka/tree/master/eureka-core/src/main/java/com/netflix/eureka/registry

# Monitoring

https://docs.newrelic.com/docs/apis/get-started/intro-apis/understand-new-relic-api-keys
https://github.com/micrometer-metrics/micrometer/blob/master/implementations/micrometer-registry-new-relic/src/main/java/io/micrometer/newrelic/NewRelicMeterRegistry.java
https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/send-custom-events-event-api
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html#production-ready-metrics-export-newrelic
https://micrometer.io/docs/registry/new-relic
https://github.com/TechPrimers/spring-boot-1.5-micrometer-prometheus-example
https://github.com/cloudfoundry/java-buildpack-metric-writer/tree/master/java-buildpack-metric-writer-common/src/main/java/org/cloudfoundry/metrics
https://docs.pivotal.io/pivotalcf/2-6/metric-registrar/using.html

# Autoscaling

https://docs.pivotal.io/pivotalcf/2-6/appsman-services/autoscaler/using-autoscaler.html
https://www.toptal.com/devops/scaling-microservices-applications
https://speakerdeck.com/adriancole/observability-3-ways-logging-metrics-and-tracing?slide=3
https://github.com/TechPrimers/spring-boot-1.5-micrometer-prometheus-example/blob/master/pom.xml
https://medium.com/finc-engineering/autoscaling-microservices-on-aws-part-1-c8488c64f6d1
https://docs.pivotal.io/pivotalcf/2-4/appsman-services/autoscaler/using-autoscaler-cli.html
https://nephely-io.github.io/app-autoscaling-calculator/?source=post_page
https://docs.pivotal.io/pivotalcf/2-6/appsman-services/autoscaler/using-autoscaler.html

# References 

How to improve the eviction policy in the Eureka Service Registry
https://thepracticaldeveloper.com/2017/06/28/how-to-fix-eureka-taking-too-long-to-deregister-instances/

Working with load balancers
https://github.com/Netflix/ribbon/wiki/Working-with-load-balancers

Autoscaling using HTTP Throughput & Latency metrics
https://community.pivotal.io/s/article/autoscaling-using-http-throughput-latency-metrics

Client Side Load Balancer: Ribbon
https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-ribbon.

Spring Boot Actuator: Production-ready features
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html

Observability
https://speakerdeck.com/adriancole/observability-3-ways-logging-metrics-and-tracing?slide=21

On Scaling Microservices

THOUGHTS ON SCALING MICROSERVICES

Much of this is a rehash of Susan Fowler's excellent book
 Production-Ready Microservices Published by O'Reilly Media, Inc., 2016 http://shop.oreilly.com/product/0636920053675.do

# Qualitative and Quantitative growth scales of a Microservice

## Quantitative

Qualitative growth scales allow the scalability of a service to tie in with higher-level business metrics: a microservice may, for example, scale with the number of users, with the number of people who open a phone application (“eyeballs”), or with the number of orders (for a food delivery service). These metrics, these qualitative growth scales, aren’t tied to an individual microservice but to the overall system or product(s).
Quantitative

•     Business Metrics 
•     Number of Health Care Claims Adjudicated
•     Number of Insurance claims processed

## Qualitative

If the qualitative growth scale of our microservice is measured in “eyeballs”, and each “eyeball” results in two requests to our microservice and one database transaction, then our quantitative growth scale is measured in terms of requests and transactions, resulting in requests per second and transactions per second as the two key quantities determining our scalability. 

•     RequestsPerSecond/QueriesPerSecond/TransactionsPerSecond
•     HTTP Throughput
•     CPU Utilization
•     Memory
•     Latency
•     (negative - scaling) Threadpool saturation
•     (negative - scaling) Number of open database connections _is it near the conn limit_

When dealing with complex dependency chains, making sure that all microservice teams tie the scalability of their services to high-level business metrics can make sure that all services are properly prepared for expected growth, even when cross-team communication becomes difficult. 

## What To Monitor For Each Microservice Availability

### Infrastructure Metrics

• CPU utilized by the microservice across all containers
• RAM utilized by the microservice across all containers,
• The available threads,
• The microservice’s open file descriptors (FD)
• The number of database connections that the microservice has to any databases it uses.

### Monitor the availability of the service

•     Service-level agreement (SLA) of the service,
•     Latency (of both the service as a whole and its API endpoints), -
•     Success of API endpoints, Responses
•    Average response times of API endpoints, the services (clients) from which API requests originate (along with which endpoints they send requests to),
•    Errors and exceptions (both handled and unhandled), and the health and status of dependencies.

# Monitoring ADVICE

1. CUSTOM DASHBOARD FOR EACH MICROSERVICE that ALERT FOR EACH MICROSERVICE FOR KEY METRICS   
2. Normal, Warning and Critical Alerts
3. On call Runbook procedure for remediating all alerts
4. Low level Remediations should be automated

A microservice should never experience the same exact problem twice.