Cloud Foundry High Availability

This blog post will explore all the intricacies of  High Availability with apps and services deployed to Cloud Foundry Components within one Availability Zone. The modes of recovery for CF components and services within one availability zone are as follows:

Types of HA - What is Running	Owner - Who Recovers It
CF Process running within a VM or a Container	Monit Supervisor Process running within the VM/Container
VM or Container	BOSH Agent + Resurrector plugin
Application Instance (JVM / Node.js / Ruby/ Python)	Health Monitor CF Job
Availability Zone (AZ)	IaaS specific HA (VMWare HA, AWS CloudWatch + EC2 AutoRecovery)

 Jobs in a typical CF 1 AZ  Deployment (PCF 1.5 AWS Deployment using CloudFormation)
+----------------------------------------------------------------+
| Job/index                                                      |
+----------------------------------------------------------------+
| clock_global-partition-489dca900676578cb063/0                  |
| cloud_controller-partition-489dca900676578cb063/0              |
| cloud_controller_worker-partition-489dca900676578cb063/0       |
| consul_server-partition-489dca900676578cb063/0                 |
| dea-partition-489dca900676578cb063/0                           |
| doppler-partition-489dca900676578cb063/0                       |
| etcd_server-partition-489dca900676578cb063/0                   |
| health_manager-partition-489dca900676578cb063/0                |
| loggregator_trafficcontroller-partition-489dca900676578cb063/0 |
| nats-partition-489dca900676578cb063/0                          |
| router-partition-489dca900676578cb063/0                        |
| uaa-partition-489dca900676578cb063/0                           |
+----------------------------------------------------------------+

VM Instances used to Manage and Operate Cloud Foundry
i-4c40e0b2 (Ops Manager optional - only in PCF) 
i-af65c551 (NAT Instance)
i-df53f321 (microbosh-bb550035e754b2468ad0) 

Databases used in Cloud Foundry  (Critical ones highlighted)
+--------------------+
| Database           |
+--------------------+
| app_usage_service  |
| autoscale          |
| bosh               |
| ccdb               |
| console            |
| notifications      |
| uaa                |
+--------------------+

BlobStore to store droplets, buildpacks and compiled packages

pcf-rohitk2-pcfopsmanagers3bucket-1fj4lxkpzysvv

Now lets look at the potential impact of Failure for these components

[CF STATELESS]  
System can be recovered quickly by spawning another instance of the same process type. Failure of these processes is not catastrophic. The four levels of HA provide resiliency and recovery of the CF core jobs.

• clock_global-partition - This component is stateless.  The instance does not need to persist state, but can be restored using the BOSH ressurector.
• cloud_controller-partition & cloud_controller_worker-partition - CC relies on on access to the CCDB and the blobstore. No API requests are available. Current apps and services continue to run. Service instances cannot be bound/unbound created/deleted.Crashed apps are not restarted.
• consul_server-partition - This consensus based KV store is not currently deployed by default as part of cf-release. Required by Diego and routing-api features. Therefore this component can be considered optional. 
• dea-partition - DEA need to be sized to handle failure as applications will be migrated dynamically. With two DEA VMs, 50% over-provisioning will guarantee all app instances restart in the remaining DEA VM. Three DEA's require 33%, etc. App instances will fail to start if there is no capacity remaining. This may affect performance. 
• doppler-partition & loggregator_trafficcontroller - These are stateless components. Loss of these components results in App and system logs not drained correctly or streamed back to the clients
• nats-partition - Cloud Foundry continues to run any apps that are already running even when NATS is unavailable for short periods of time. The components publishing messages to and consuming messages from NATS are resilient to NATS failures. As soon as NATS recovers, operations such as health management and router updates resume.
• router-partition - Dependent on NATs to actually receive route updates for most clients. Will definitely impact client traffic since it reverse proxies all requests/responses from the DEA. 
• uaa-partition - Depends on the uaadb. Loss of logged in state. New OAuth access tokens cannot be obtained from the UAA.

[CF-STATEFUL]

These components together represent the entire runtime and deployment state of CF.  Failure of any one of these DBs represents a significant loss of end-user experience. 

• uaa - Authentication and authorization
• ccdb - API requests and Managed Services
• console - Apps Manager/ Developer Console
• bosh - Current state of a CF Deployment in terms of job and VMs
• blobstore - Currently running apps continue to run. Apps can not be successfully pushed. Already pushed apps cannot be staged. Health management will fail to start apps because droplets cannot be downloaded

Within a single AZ, periodic DB snapshots and and IaaS capability to backup and restore DBs is critical when DBs fail. The only solution to the true- HA of these databases is to host them in an external customer-provided DB deployment can provide a multi-AZ capability that a CF deployment can leverage. Using something like AWS RDS or external blob storage such as S3 provides multi-AZ capabilities as a feature from the provider. Using an external blob storage such as S3-compatible blob provider inherits the HA capabilities of the external provider. For example, AWS S3 is multi-AZ and HA.

Its time to consider even dire scenarios with our CF deployment:

For a treatment of CF across Availability Zones in a multi-AZ deployment please read https://docs.google.com/document/d/1PhtKMboRNeiihoIBHw7kqnimVm9kyDU4ddV7YWCYEzQ/edit?usp=sharing

It is critical to establish processes and procedures for Disaster Recovery in the event of a complete loss of multiple AZs. Duncan Winn covers this extensively in http://blog.pivotal.io/pivotal-cloud-foundry/features/restoring-pivotal-cloud-foundry-after-disaster

Note this blog does not cover the process for recovering your own persistence tier say Oracle, Tibco and other middleware which often provides state to apps deployed on PCF. You will need to establish a coherent HA/DR strategy for data services in line with what has been presented for the elastic runtime above.