Making Eureka Service Discovery Responsive on PCF

Making Service Discovery More Responsive on PCF

Eureka service registries and Eureka clients are tuned for cloud scale deployment of applications. This results in tuning their respective server and client side caches to account for network brownouts and self-preservation in case of network partitions, failed compute or storage.

All this has the effect of the service registry getting stale sometimes especially in autoscaling or auto descaling scenarios. If autoscaling and scaling down is happening very fast then the REST or HTTP clients sometimes experience timeouts since the service IPs are outdated and the service registry has not been updated to the latest set of microservices app instances.

We ran into one such issue on PCF and configured Service Discovery in three ways to eliminate timeouts. Much of this detailed experimentation was done by my colleague Rohit Bajaj. 

1. Ribbon Ping Configuration
2. BOSH DNS Polyglot discovery 
3. Eureka server configuration to eliminate server timeouts 

Summary

We Determined that Native Polyglot service discovery provided by Cloud Foundry as the optimal Configuration for Service Discovery. Error rate drops to 0-1% in the auto descaling scenarios  as opposed to > 2% with other settings.

For your average  spring, spring-boot,  Java app that requires service discovery Eureka Service Registry fits the bill nicely; however if your workload is highly dynamic and you need close to 0 error rate when load balancing across transient service instances then BOSH DNS is better. When using Bosh DNS you don’t use Ribbon. So Custom Circuit Breaker + BOSH DNS replaces Ribbon + Eureka. 

1. Ribbon Ping Configuration

See https://thepracticaldeveloper.com/2017/06/28/how-to-fix-eureka-taking-too-long-to-deregister-instances/

2. BOSH DNS Discovery

Polyglot service discovery introduces new capabilities with a familiar workflow. An app developer can configure an internal route to create a DNS entry for their app. This makes the app discoverable on the container network. A DNS lookup on an internal route returns a list of container IPs for applications corresponding to that particular internal route. docs

We recommend pairing BOSH service discovery with a robust circuit breaker like Resilience4J Sample code on Reslience4J + BOSH Polyglot discovery

3 Eureka Settings

Ideal setting for tuning Eureka clients and servers to be super responsive 

# Ribbon Settings 

ribbon.ServerListRefreshInterval = 50

# Eureka Client

eureka.instance.lease-renewal-interval-in-seconds = 10
eureka.client.initialInstanceInfoReplicationIntervalSeconds = 10
eureka.client.instanceInfoReplicationIntervalSeconds = 10
eureka.client.registryFetchIntervalSeconds = 10

# Eureka Server

eureka.instance.lease-expiration-duration-in-seconds = 30
eureka.server.eviction-interval-timer-in-ms = 20 * 1000
eureka.server.responseCacheUpdateIntervalMs = 10 * 1000
eureka.server.getWaitTimeInMsWhenSyncEmpty = 10 * 1000

Typically changing Eureka server side settings is not possible when the Eureka server is provisioned by the Spring Cloud Services tile.

# Eureka Server Self-Preservation

eureka.server.enableSelfPreservation = false

Self Preservation window is every 15 minutes
  DEFAULT = 30s interval for `lease-renewal-interval`
  N = 24
  Number of heartbeats per minute to trigger self-preservation < 2 * 24 * 0.85 = 41 heartbeats

  NEW = 10s interval , N = 24
  Number of heartbeats per minute to trigger self-preservation < 6 * 24 * 0.85 = 123 heartbeats

# References

- Spring Cloud Netflix Eureka - The Hidden Manual https://blog.asarkar.org/technical/netflix-eureka/
- The Mystery of Eureka self-preservation https://medium.com/@fahimfarookme/the-mystery-of-eureka-self-preservation-c7aa0ed1b799
- Eureka http://netflix.github.io/ribbon/ribbon-eureka-javadoc/index.html
- Spring Cloud Ribbon https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-ribbon.html
- B/g for Eureka https://app-transformation-cookbook-internal.cfapps.io/duplicates/replatforming/blue-green-with-eureka/3e7f94f6c49795ef347b70141a36c134/

# CODE

https://github.com/Netflix/eureka/blob/master/eureka-client/src/main/java/com/netflix/discovery/EurekaClientConfig.java
https://github.com/Netflix/eureka/wiki/Eureka-REST-operations
https://github.com/Netflix/eureka/tree/master/eureka-core/src/main/java/com/netflix/eureka/registry

# Monitoring

https://docs.newrelic.com/docs/apis/get-started/intro-apis/understand-new-relic-api-keys
https://github.com/micrometer-metrics/micrometer/blob/master/implementations/micrometer-registry-new-relic/src/main/java/io/micrometer/newrelic/NewRelicMeterRegistry.java
https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/send-custom-events-event-api
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html#production-ready-metrics-export-newrelic
https://micrometer.io/docs/registry/new-relic
https://github.com/TechPrimers/spring-boot-1.5-micrometer-prometheus-example
https://github.com/cloudfoundry/java-buildpack-metric-writer/tree/master/java-buildpack-metric-writer-common/src/main/java/org/cloudfoundry/metrics
https://docs.pivotal.io/pivotalcf/2-6/metric-registrar/using.html

# Autoscaling

https://docs.pivotal.io/pivotalcf/2-6/appsman-services/autoscaler/using-autoscaler.html
https://www.toptal.com/devops/scaling-microservices-applications
https://speakerdeck.com/adriancole/observability-3-ways-logging-metrics-and-tracing?slide=3
https://github.com/TechPrimers/spring-boot-1.5-micrometer-prometheus-example/blob/master/pom.xml
https://medium.com/finc-engineering/autoscaling-microservices-on-aws-part-1-c8488c64f6d1
https://docs.pivotal.io/pivotalcf/2-4/appsman-services/autoscaler/using-autoscaler-cli.html
https://nephely-io.github.io/app-autoscaling-calculator/?source=post_page
https://docs.pivotal.io/pivotalcf/2-6/appsman-services/autoscaler/using-autoscaler.html

# References 

How to improve the eviction policy in the Eureka Service Registry
https://thepracticaldeveloper.com/2017/06/28/how-to-fix-eureka-taking-too-long-to-deregister-instances/

Working with load balancers
https://github.com/Netflix/ribbon/wiki/Working-with-load-balancers

Autoscaling using HTTP Throughput & Latency metrics
https://community.pivotal.io/s/article/autoscaling-using-http-throughput-latency-metrics

Client Side Load Balancer: Ribbon
https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-ribbon.

Spring Boot Actuator: Production-ready features
https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-metrics.html

Observability
https://speakerdeck.com/adriancole/observability-3-ways-logging-metrics-and-tracing?slide=21

On Scaling Microservices

THOUGHTS ON SCALING MICROSERVICES

Much of this is a rehash of Susan Fowler's excellent book
 Production-Ready Microservices Published by O'Reilly Media, Inc., 2016 http://shop.oreilly.com/product/0636920053675.do

# Qualitative and Quantitative growth scales of a Microservice

## Quantitative

Qualitative growth scales allow the scalability of a service to tie in with higher-level business metrics: a microservice may, for example, scale with the number of users, with the number of people who open a phone application (“eyeballs”), or with the number of orders (for a food delivery service). These metrics, these qualitative growth scales, aren’t tied to an individual microservice but to the overall system or product(s).
Quantitative

•     Business Metrics 
•     Number of Health Care Claims Adjudicated
•     Number of Insurance claims processed

## Qualitative

If the qualitative growth scale of our microservice is measured in “eyeballs”, and each “eyeball” results in two requests to our microservice and one database transaction, then our quantitative growth scale is measured in terms of requests and transactions, resulting in requests per second and transactions per second as the two key quantities determining our scalability. 

•     RequestsPerSecond/QueriesPerSecond/TransactionsPerSecond
•     HTTP Throughput
•     CPU Utilization
•     Memory
•     Latency
•     (negative - scaling) Threadpool saturation
•     (negative - scaling) Number of open database connections _is it near the conn limit_

When dealing with complex dependency chains, making sure that all microservice teams tie the scalability of their services to high-level business metrics can make sure that all services are properly prepared for expected growth, even when cross-team communication becomes difficult. 

## What To Monitor For Each Microservice Availability

### Infrastructure Metrics

• CPU utilized by the microservice across all containers
• RAM utilized by the microservice across all containers,
• The available threads,
• The microservice’s open file descriptors (FD)
• The number of database connections that the microservice has to any databases it uses.

### Monitor the availability of the service

•     Service-level agreement (SLA) of the service,
•     Latency (of both the service as a whole and its API endpoints), -
•     Success of API endpoints, Responses
•    Average response times of API endpoints, the services (clients) from which API requests originate (along with which endpoints they send requests to),
•    Errors and exceptions (both handled and unhandled), and the health and status of dependencies.

# Monitoring ADVICE

1. CUSTOM DASHBOARD FOR EACH MICROSERVICE that ALERT FOR EACH MICROSERVICE FOR KEY METRICS   
2. Normal, Warning and Critical Alerts
3. On call Runbook procedure for remediating all alerts
4. Low level Remediations should be automated

A microservice should never experience the same exact problem twice.