Cloud Foundry High Availability

This blog post will explore all the intricacies of  High Availability with apps and services deployed to Cloud Foundry Components within one Availability Zone. The modes of recovery for CF components and services within one availability zone are as follows:

Types of HA - What is Running	Owner - Who Recovers It
CF Process running within a VM or a Container	Monit Supervisor Process running within the VM/Container
VM or Container	BOSH Agent + Resurrector plugin
Application Instance (JVM / Node.js / Ruby/ Python)	Health Monitor CF Job
Availability Zone (AZ)	IaaS specific HA (VMWare HA, AWS CloudWatch + EC2 AutoRecovery)

 Jobs in a typical CF 1 AZ  Deployment (PCF 1.5 AWS Deployment using CloudFormation)
+----------------------------------------------------------------+
| Job/index                                                      |
+----------------------------------------------------------------+
| clock_global-partition-489dca900676578cb063/0                  |
| cloud_controller-partition-489dca900676578cb063/0              |
| cloud_controller_worker-partition-489dca900676578cb063/0       |
| consul_server-partition-489dca900676578cb063/0                 |
| dea-partition-489dca900676578cb063/0                           |
| doppler-partition-489dca900676578cb063/0                       |
| etcd_server-partition-489dca900676578cb063/0                   |
| health_manager-partition-489dca900676578cb063/0                |
| loggregator_trafficcontroller-partition-489dca900676578cb063/0 |
| nats-partition-489dca900676578cb063/0                          |
| router-partition-489dca900676578cb063/0                        |
| uaa-partition-489dca900676578cb063/0                           |
+----------------------------------------------------------------+

VM Instances used to Manage and Operate Cloud Foundry
i-4c40e0b2 (Ops Manager optional - only in PCF) 
i-af65c551 (NAT Instance)
i-df53f321 (microbosh-bb550035e754b2468ad0) 

Databases used in Cloud Foundry  (Critical ones highlighted)
+--------------------+
| Database           |
+--------------------+
| app_usage_service  |
| autoscale          |
| bosh               |
| ccdb               |
| console            |
| notifications      |
| uaa                |
+--------------------+

BlobStore to store droplets, buildpacks and compiled packages

pcf-rohitk2-pcfopsmanagers3bucket-1fj4lxkpzysvv

Now lets look at the potential impact of Failure for these components

[CF STATELESS]  
System can be recovered quickly by spawning another instance of the same process type. Failure of these processes is not catastrophic. The four levels of HA provide resiliency and recovery of the CF core jobs.

• clock_global-partition - This component is stateless.  The instance does not need to persist state, but can be restored using the BOSH ressurector.
• cloud_controller-partition & cloud_controller_worker-partition - CC relies on on access to the CCDB and the blobstore. No API requests are available. Current apps and services continue to run. Service instances cannot be bound/unbound created/deleted.Crashed apps are not restarted.
• consul_server-partition - This consensus based KV store is not currently deployed by default as part of cf-release. Required by Diego and routing-api features. Therefore this component can be considered optional. 
• dea-partition - DEA need to be sized to handle failure as applications will be migrated dynamically. With two DEA VMs, 50% over-provisioning will guarantee all app instances restart in the remaining DEA VM. Three DEA's require 33%, etc. App instances will fail to start if there is no capacity remaining. This may affect performance. 
• doppler-partition & loggregator_trafficcontroller - These are stateless components. Loss of these components results in App and system logs not drained correctly or streamed back to the clients
• nats-partition - Cloud Foundry continues to run any apps that are already running even when NATS is unavailable for short periods of time. The components publishing messages to and consuming messages from NATS are resilient to NATS failures. As soon as NATS recovers, operations such as health management and router updates resume.
• router-partition - Dependent on NATs to actually receive route updates for most clients. Will definitely impact client traffic since it reverse proxies all requests/responses from the DEA. 
• uaa-partition - Depends on the uaadb. Loss of logged in state. New OAuth access tokens cannot be obtained from the UAA.

[CF-STATEFUL]

These components together represent the entire runtime and deployment state of CF.  Failure of any one of these DBs represents a significant loss of end-user experience. 

• uaa - Authentication and authorization
• ccdb - API requests and Managed Services
• console - Apps Manager/ Developer Console
• bosh - Current state of a CF Deployment in terms of job and VMs
• blobstore - Currently running apps continue to run. Apps can not be successfully pushed. Already pushed apps cannot be staged. Health management will fail to start apps because droplets cannot be downloaded

Within a single AZ, periodic DB snapshots and and IaaS capability to backup and restore DBs is critical when DBs fail. The only solution to the true- HA of these databases is to host them in an external customer-provided DB deployment can provide a multi-AZ capability that a CF deployment can leverage. Using something like AWS RDS or external blob storage such as S3 provides multi-AZ capabilities as a feature from the provider. Using an external blob storage such as S3-compatible blob provider inherits the HA capabilities of the external provider. For example, AWS S3 is multi-AZ and HA.

Failure Testing of Components in Cloud Foundry

Update:
- Checkout cf-attack A cf cli plugin to perf/load test an app
- Chaos as a Service: https://github.com/skibum55/chaos-as-a-service
- MONKEYS & LEMURS ANDLOCUSTS … OH MY! ANTI-FRAGILE PLATFORMS

Failure testing hardens your systems against failure. Failure Testing prevents larger outages by enabling the practice of immunization i.e. imbibe poison/virus in small quantities to become immune. It is critical that you know how the system responds to failure. When failure does occur it is prudent to plan for it and establish an Incident Control System that first mitigates, isolates and resolves the issue. The entire episode ends with a blameless post-mortem that enables the product and failure response process to address gaps.

This blog post explains the various scenarios you need to cover in your end to end failure testing plan for apps deployed to Cloud Foundry.  We only get into the details of which scenarios to consider and how to affect them. Response to failure aka High Availability of CF and Apps will be covered in the next blog post.

1. Application Failure Scenarios

1. [HEAP-OOM]  App instance allocates excess Java Heap Memory. App repetitively suffers java heap OOM.
2. [NATIVE-OOM] App instance allocates excess native memory leading to crashes of the warden container.
3. [EXCESS-CPU] App instance is pegged at > 90% CPU for a sustained period of time.
4. [DEADLOCK]  App instance is pegged < 50% CPU inspite of sustained increasing request traffic.
5. [GC-PAUSES]  App unresponsive for random periods of time.
6. [RANDOM-DEATH] App instance randomly exits.

Apps for Affecting App Failure Scenarios:
- https://t.co/50H0HA10ym - Problem Diagnostics Toolkit that provides an app for creating thread concurrency issues, memory leaks, high CPU usage and JVM crashes. Unzip the ear, ignore all the other files and just push the war file. The browser load of the first page will take a  long time since it loads the DOJO toolkit on the first web request.  Credit for this app goes to IBM. Thanks @jpapejr For a running example see http://pdt.cfapps.pez.pivotal.io/
- https://github.com/cf-platform-eng/cf-scale-boot - Endpoint for randomly killing app instances
- https://github.com/pivotalservices/autoscaler-test-app - Hogs Memory to create OOMs
- https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army#grow-your-own-chaos-monkey - Customize the Netflix chaos monkey for your own purposes.

2. Service Failure Scenarios

1. Logging Infrastructure service (say Splunk or LogStash) is unresponsive / not accessible
2. Monitoring Service (say AppDynamics) is unresponsive / not accessible
3. Data/Persistence services (say Postgresql, MySQL) are not available/ accessible

Tools to simulate troublesome service interactions
- Failure Injection Testing
- Spring Integration Testing to introduce faulty service dependencies.

3. Cloud Foundry Stateful and Stateless Job (Component) Failure Scenarios

1. [CF-STATELESS] Random failure of critical CF stateless components. DEA VMs, Collector, Doppler & Loggregator Traffic Controller VMs, Health Manager VM, ETCD nodes,  Cleanup  jobs and Errands fall in this category.
2. [CF-STATEFUL] Loss of storage volumes and persistence stores in a CF deployment. This includes the loss of critical databases like the BOSHDB, CCDB, UAADB, AppsManager-DB, NFS-Server/Blobstore,etc.,
3. [CF-CRITICAL] Loss of critical CF components like the front end loadbalancer, UAA, gorouter, NATS, & CloudController.
4. [CF-CONFIGURATION] Loss of BOSH Director, OpsManager VM, MicroBosh, Jumpbox, DNS Server
5. [CF-AZ-LOSS] Loss of an entire AZ.

Tools for killing CF components 
- https://github.com/pivotal-cf/chaos-lemur - Adaptation of the Chaos Monkey concept to the SpringXD BOSH release
- https://github.com/BrianMMcClain/cloudfoundry-chaos-monkey - Chaos Monkey style service for CF

4. Hardware Failure Scenarios

1. [NETWORK-PARTITION] Entire ESXi Hypervisor hosting DEAs and other CF components goes offline.
2. [HYPERVISOR-DEATH]  Entire ESXi Hypervisor hosting DEAs and other CF components dies.

Tools To Blow Up Your Favorite Hypervisor
- Simulating VMware High Availability failover (2056634)
- Testing a VMware Fault Tolerance configuration (1020058)
- Simian Army for Creating Havoc on AWS

The best way to test resiliency of Cloud Foundry is to do the following:
1. Kill VMs from vSphere vCenter or AWS IaaS console.
2. Run bosh tasks —no-filter in a loop to watch the resurrector bring them up. i.e. watch -n 2 "bosh tasks —no-filter"
3. Randomly ssh into a container bosh ssh and sudo kill -9 -1
4. bosh ssh into a DEA and kill a container

Footnote* 

Failure is NOT a once/twice a year event. If your site gets a billion queries/requests/hits then once in a billion events is happening 20 times a day. Plan for it and embrace it. Last but not the least always adhere to the rules of  designing and developing operations-friendly services.

Top 50 Cloud Foundry Github Repositories

Tools, Hidden gems and Utilities for developing on and deploying Cloud Foundry

1. https://github.com/pivotalservices/cfops -This is simply an automation that is based on the supported way to back up Pivotal Cloud Foundry.
2. https://github.com/pivotalservices/shekel - Tool for capacity planning, sizing and costing a production deployment of Cloud Foundry.
3. https://start.spring.io/ - A web-based quickstart generator for Spring projects  https://github.com/spring-io/initializr
4. https://github.com/cloudfoundry-community/cf-boshworkspace -  Easily deploy CloudFoundry in various configurations, small to large, in AWS.
5. https://github.com/logsearch/logsearch-for-cloudfoundry - A Logsearch addon that customizes Logsearch to work with Cloud Foundry data.
6. https://github.com/logsearch/logsearch-boshrelease - A BOSH-scalable Elasticsearch + Logstash + Kibana release http://www.logsearch.io
7. https://github.com/dpinto-pivotal/cf-SpringBootTrader - Microservices  showcase for Spring Trader using spring boot + Spring Cloud also https://github.com/spring-cloud-samples
8. https://github.com/pivotalservices/spring-music - sample app for using database services on Cloud Foundry with the Spring Framework. Great for testing spring cloud connectors. See https://github.com/pivotal-cf-experimental/spring-music-on-cf-benchmarking for Apache JMeter stress tests.
9. https://github.com/cloudfoundry/cli - A CLI for Cloud Foundry written in Go. Look in https://github.com/cloudfoundry/cli/releases for downloads.
10.  http://plugins.cfapps.io/ui/ - Repository of cf-cli plugins https://github.com/cloudfoundry-incubator/cli-plugin-repo - public repository for community created CF CLI plugins
11. https://github.com/pivotalservices/pse-training-cf-hw-service-broker - Lab showcases Cloud foundry service broker concepts. Writing, adding and using Brokered Services
12. https://github.com/cloudfoundry/cli/tree/master/plugin_examples -  Instructions for writing your own Cloud Foundry CLI plugin 
13. https://github.com/pivotalservices/bosh-lite-install - Scripts to install bosh-lite on your local machine
14. https://github.com/pivotalservices/pse-training-cf-bp-module - Obtain the buildpack source code, extend it, package it, deploy it, and test it.
15. https://github.com/pivotalservices/pivotal-cf-irules - Library of F5 iRules which may be useful when working with Pivotal CF.
16. https://github.com/pivotalservices/java-r-buildpack - Heroku forked 'R' buildpack for CF that includes Rserve for supporting remote callers using an Rserve client library.
17. https://github.com/cloudfoundry-community/spring-boot-cf-service-broker - Spring Boot project for creating Cloud Foundry service brokers.
18. https://github.com/cloudfoundry-community/terraform-aws-cf-install - One click deploy of Cloud Foundry into an AWS VPC
19. https://github.com/cloudfoundry-community/cf-boshworkspace - easily deploy CloudFoundry in various configurations, small to large, in AWS
20. https://github.com/cloudfoundry-community/bosh-bootstrap - Simplest way to get a micro bosh running and upgrade/destroy it over time
21. https://github.com/cloudfoundry-community/firehose-to-syslog - Send firehose events from Cloud Foundry to syslog.
22. https://github.com/cloudfoundry-community/delete-all-bindings-and-services/blob/master/bin/delete.sh - cf-cli curl script to delete all bindings and services
23. https://github.com/cloudfoundry-community/kibana-me-logs/blob/master/bin/upgrade-all.sh - cf-cli curl script to upgrade all apps
24. https://github.com/krujos/cf-org-usage-report - Script to report on the amount of memory used against the currently applied quota. CLI Plugin; https://github.com/krujos/usagereport-plugin/  https://github.com/malston/notification-quota & https://github.com/malston/cloudfoundry-orphan-space-removal - Quota & Space Mgmt.
25. https://github.com/Pivotal-Field-Engineering/PCF-demo/tree/micro-services - Demonstration of CF as an application platform
26. https://github.com/Pivotal-Field-Engineering/postgres-bosh-release -   Simple bosh release for PostgreSQL. 
27. https://github.com/Pivotal-Field-Engineering/pcf-workshop-devops - Workshop for CF Devops
28. https://github.com/pivotalservices/cf-egress-tester - A quick app to test network egress from a CF space. Used for security group testing.  Helps test firewall access.
29. https://github.com/pivotalservices/uaaldapimport - Pre-populate the users with UAA api and CloudController api. So they can have all the roles before user logs in.
30. https://github.com/pivotalservices/cloudfoundry-java-logging - Injects CF VCAP_APPLICATION environment variable into a variety of Java logging frameworks. 
31. https://github.com/pivotalservices/cf-riak-cs-release - A BOSH release for riak and riak-cs
32. https://github.com/cloudfoundry-community/cf-docs-contrib/wiki - scrapbook of  design docs, Pivotal Trackers and info and links about CF. 
33. https://blog.starkandwayne.com/2014/07/10/resurrecting-bosh-with-binary-boshes - Highly available BOSH.
34. https://github.com/cloudfoundry-community/docker-services-boshworkspace - Fastest way to deploy Docker Services in combination with Cloud Foundry onto bosh-lite. 
35. https://github.com/cloudfoundry-incubator/route-registrar - A standalone executable written in golang that continuously broadcasts a route using NATS to the CF router.
36. https://github.com/cloudfoundry-community/cf-java-component - A set of libraries for building Cloud Foundry components on the Java platform
37. https://github.com/cloudfoundry-community - BOSH releases for haproxy, docker-services, route-registrar, memcache, hazelcast, ...
38. https://github.com/cloudfoundry-community/.net-buildpack - Cloud Foundry buildpack for running .NET applications
39. https://github.com/Pivotal-Field-Engineering/Drupal - Deploy Drupal CMS to CF
40. https://github.com/cloudfoundry-community/traveling-cf-admin - packages CLIs used by CF admins - the cf CLI and useful admin plugins, and the uaac CLI without requiring ruby.
41. https://bosh.io/docs/create-release.html - Simplest way to create a CF BOSH release. also see https://github.com/cloudfoundry/bosh-notes & https://bosh.io/docs Honorable mention - Generators for creating and sharing BOSH releases https://github.com/cloudfoundry-community/bosh-gen
42. https://github.com/cloudfoundry-incubator/admin-ui - An application for viewing Cloud Foundry metrics and operations data
43. https://github.com/cloudfoundry-incubator/lattice - Playground for next-gen CF based on Deigo
44. https://github.com/cloudfoundry-incubator/diego-release - BOSH release for deploying Diego and associated tasks. see https://github.com/cloudfoundry-incubator/diego-design-notes
45. https://github.com/cloudfoundry-incubator/pat - Super-simple load generation/performance testing framework for quickly and easily running load against CF
46. https://github.com/cloudfoundry/cf-acceptance-tests - test suite exercises a full Cloud Foundry deployment using the golang cf CLI and curl. Includes the DORA test app. Also see https://github.com/cloudfoundry/cf-smoke-tests - Smoke tests for CloudFoundry that are safe to run in a production environment
47. https://github.com/cloudfoundry-incubator/notifications - A notifications component that parts of CF can use to send email to end users.
48. https://github.com/concourse/concourse - BOSH release for the Concourse CI. Tutorial - https://github.com/starkandwayne/concourse-tutorial
49. https://github.com/cloudfoundry/java-buildpack-support - Access logging, lifecycle support and tomcat logging support for the Java Buildpack.
50. https://github.com/spring-io/sagan - The spring.io site and reference app. https://github.com/bclozel/spring-resource-handling - Spring 4.1 Resource handling
51. https://github.com/xchapter7x/autopilot - cf plugin for hands-off, Blue/Green zero downtime application deploys & https://github.com/krujos/scaleover-plugin - Uses cf scale and a time interval to move from B-> G. Look for these two getting merged in the future.
52. Report that shows which apps are using which buildpacks - https://github.com/krujos/oohimtelling

Cloud Foundry Java Buildpack Debugging ProTips and Common extension scenarios

Java Buildpack Debugging - During Staging of an Application

The best technique to debug an application that fails during staging is to set the JBP_LOG_LEVEL env. variable to DEBUG and restage the app.

cf set-env <APP> JBP_LOG_LEVEL DEBUG

Tomcat Container Debugging in Cloud Foundry

To enable system component logging of Tomcat  from within the app. Place the following logging.properties in the WEB-INF/classes dir. of your app. 

handlers = com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler

.handlers = com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler

com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler.level = FINE

org.apache.catalina.core.level = ALL 

org.apache.catalina.realm.level = ALL

org.apache.catalina.authenticator.level = ALL

org.apache.catalina.session.level= ALL

This properties file relies on Tomcat to load the logging.properties from both the conf directory and the application itself.  Tomcat's implementation of the java.util.logging API is enabled by default, and supports per classloader configuration, in addition to the regular global java.util.logging configuration. This means that logging can be configured at the following layers:

• Globally. That is usually done in the ${catalina.base}/conf/logging.properties file. The file is specified by the java.util.logging.config.file System property which is set by the startup scripts. If it is not readable or is not configured, the default is to use the ${java.home}/lib/logging.properties file in the JRE.
• In the web application. The file will be WEB-INF/classes/logging.properties

This technique allows you to get DEBUG level Tomcat container logs without forking (modifying java-buildpack/blob/master/resources/tomcat/conf/logging.properties) the Java buildpack 

Extending the Java Buildpack

Traditionally extending the Java Buildpack meant forking the buildpack. The recommendation is that fork changes should be kept at a minimum. JBP 3.0 introduced support for user configuration via environment variables. Where possible the modification to the java buildpack function should be done via environment variables. Fork changes should be restricted to features that absolutely cannot be implemented in an app like adding strong cryptography or adding certificates to the JVM keystore.

Buildpack configuration can be overridden with an environment variable matching the configuration file you wish to override minus the .yml extension and with a prefix of JBP_CONFIG. The value of the variable should be valid inline yaml. For example, to change the default version of Java to 7 and adjust the memory heuristics apply this environment variable to the application. doc

cf set-env my-application JBP_CONFIG_OPEN_JDK_JRE '[jre: {version: 1.7.0_+}, memory_calculator: {memory_heuristics: {heap: 85, stack: 10}}]'

Common scenarios for Extending the Java Buildpack

1. Configuring LDAP Security to protect service endpoints with LDAP in Cloud Foundry

The LDAP connection and other configuration in an app can be done via a META-INF/context.xml.This technique allows context files to be packaged within WARs. In Tomcat, a Context represents a single web application. Tomcat uses the Context configuration element to contain information about components required by a given application, such as databases, realms, or custom loaders. Additionally, the Context element can be configured with a wide variety of attributes that control things such as logging, reload permissions, caching, and more.

Typically add the following in the web.xml

<security-constraint>

    <web-resource-collection>

      <web-resource-name>AppConfigResource</web-resource-name>

           <url-pattern>/s-music/p/__admin/console</url-pattern>

      </web-resource-collection>

    <auth-constraint>

      <role-name>*</role-name>

    </auth-constraint>

</security-constraint>

There needs to be a corresponding JNDIRealm definition in the application context. i.e. the META-INF dir. of the app should contain a context.xml

<?xml version="1.0" encoding="UTF-8"?>
<Context>          
       <Realm className="org.apache.catalina.realm.JNDIRealm"
       debug="99"
       resourceName="AuthIn"
       connectionURL="ldap://${LDAP-url}"
       authentication="simple"
       referrals="follow"
       connectionName="CN=Service Account\, ztcserver,OU=Service,OU=Accounts,OU=Administration,DC=rohitkelapure,DC=local"
       connectionPassword="${LDAPconnectPassword}"
       userSubtree="true"
       userBase="OU=Accounts,OU=Administration,DC=rohitkelapure,DC=local"
       userSearch="(sAMAccountName={0})"
       roleBase="OU=Permissions,OU=Groups,OU=Administration,DC=rohitkelapure,DC=local"
       roleName="cn"
       roleSubtree="true"
       roleSearch="(member={0})"
        />
</Context>

When the JNDIRealm is not authenticating correctly it is very helpful to enable tomcat container logging for the org.apache.catalina.realm and org.apache.catalina.authenticator loggers.

Note the use of system properties in the context.xml. Tomcat supports parameter substitution in its configuration files using the ANT style ${var}. The variables available are pulled in via System.getProperties() automatically. Further you can push variables into this area from the command line using the -Dname=value option for the Tomcat launch. These system properties can be set when pushing the app using the JAVA_OPTS env. variable

cf se spring-music JAVA_OPTS "-DLDAP-url=ldap://ldap.example.com/dc=example,dc=com -DLDAPconnectPassword=passw0rd"

2. Adding SSLCerts to the JVM

Leverage the Java Buildpack resources support to override and add files into the buildpack. The JBP copy_resources support overlays the contents of the resources directory onto a component's sandbox.

To Add your own SSL certs to the Oracle JRE:

mkdir -p resources/oracle_jdk_jre/lib/security/cacerts
Copy your certs into the newly created cacerts directory.

3. Add strong JCE support to the JRE

mkdir -p resources/open_jdk_jre/lib/security/
Copy the local_policy.jar into the newly created security directory.

4. Enable GC logging

1. Uncomment and modify the value of the java_opts setting of the config/java_opts.yml file.
2. As an example see https://github.com/pivotalservices/oracle-jre-java-buildpack/blob/master/config/java_opts.yml#L19

Upstream Maintenance of a forked Buildpack

Credit for this section goes to my colleague David Malone (@malonedave). 

Pulling Upstream Changes  into your fork of the JBP 

1. Configure Ruby to use your local Gem repo
2. Install all of the necessary tools; Ruby, RVM, Bundler 
3. git clone http://git.mycompany.com/java-buildpack.git
4. cd java-buildpack
5. git remote add upstream https://github.com/cloudfoundry/java-buildpack.git
6. git pull upstream master
7. git push origin master

Packaging the Java Buildpack

The offline package is a version of the buildpack designed to run without access to a network. It packages the latest version of each dependency (as configured in the config/ directory) and disables remote_downloads. This package is about 180M in size. To create the offline package, use the OFFLINE=true argument:

1. cd java-buildpack
2. export BUNDLE_GEMFILE=$PWD/Gemfile
3. bundle install
4. bundle exec rake (runs all unit & integration tests)- requires CentOS
5. bundle exec rake package VERSION=3.1 OFFLINE=true VERSION=MY_JBP_OFFLINE_3.1

Hey Coud Foundry Where my MBeans @ ?

If you have pushed a java application to Cloud Foundry you have experienced the pain of NOT having an accessible Mbean server.  Cloud Foundry by default opens only ONE ingress port over the HTTP protocol.

The Java Management Extensions Instrument and Agent Specification defines the concept of connectors. A connector makes a Java Management Extensions (JMX) technology MBean server accessible to remote Java technology-based clients. The client end of a connector exports essentially the same interface as the MBean server.

Typical connectors connect to the MBean server over RMI. Herein lies the problem because the RMI-IIOP port is NOT open on the container running the CF app. Therefore an alternative mechanism is needed that allows a client to talk to the Mbean server over HTTP.

This is where Jolokia comes to the rescue. Jolokia is an agent based approach for remote JMX access. It is an alternative to standard JSR 160 connectors. The communication between client and agent goes over HTTP (either GET or POST), where the request and response payload is represented in JSON.

The best way to incorporate Jolokia in your app is by incorporating the Jolokia agent into the application by adding the following to the pom.xml or build.gradle

//Jolokia

runtime 'org.jolokia:jolokia-core:1.3.1'

runtime 'org.jolokia:jolokia-jsr160:1.3.1'

The Agent Servlet is also listed in the web.xml

<web-app xmlns="http://java.sun.com/xml/ns/javaee"

     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

     xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 

     http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"

     version="3.0">

  <display-name>Spring-Music</display-name>

  

  <servlet>

    <servlet-name>jolokia-agent</servlet-name>

    <servlet-class>org.jolokia.http.AgentServlet</servlet-class>

    <load-on-startup>1</load-on-startup>

  </servlet>

  <servlet-mapping>

    <servlet-name>jolokia-agent</servlet-name>

    <url-pattern>/jolokia/*</url-pattern>

  </servlet-mapping>

</web-app>

Thereafter rebuild the app and then the metrics can be accessed over the browser using requests like these: 

Request:

http://spring-music-nonhesitant-cornetist.cfapps.pez.pivotal.io/jolokia/read/java.lang:type=Memory/HeapMemoryUsage

Response

{

• request: 
  

  {
  

  • mbean: "java.lang:type=Memory",
  • attribute: "HeapMemoryUsage",
  • type: "read"

  },
• value: 
  

  {
  

  • init: 805306368,
  • committed: 771751936,
  • max: 771751936,
  • used: 203904200

  },
• timestamp: 1435594883,
• status: 200

}

The JMX metrics can then be easily incorporated into a dashboard. In addition to Java there are other Jolokia clients available as well. The Javascript Jolokia client is full-featured and supports charting with Cubism, a D3.js plugin for plotting time series data.

Please note that some of the commercial app servers like Glassfish, WebSphere Liberty Profile, WebLogic provide JMX HTTP connectors.  The advantage of using these connectors over Jolokia is that , existing JMXClients like Jconsole and VisualVM plugins work as-is out of the box with JMX over HTTP. There is no need to do the querying for individual request/responses or write custom code. You can leverage the existing ecosystem of JMX client tooling.

Warden and Deigo container preservation in Cloud Foundry

There are a number of use cases for Warden container preservation in Cloud Foundry -

1. Post-mortem analysis of a compromised or hacked container
2. Problem Determination and Troubleshooting
3. Audit and compliance

There are two ways to achieve this before and after Diego i.e BD and AD

Before Deigo

This is a world where apps run in warden containers. Warden containers are garbage collected after a predetermined time period dea_next.crash_lifetime_secs Container container stays around on the DEA for about an hour or so by default before being removed. In addition to setting the dea_next.crash_lifetime_secs, also set dd the container_grace_time parameter to 3600 in the Warden configuration file stored in the warden.yml. container_grace parameter controls defines time before DEA deletes -containers.

After Deigo

Diego does not keep around containers after crashes. Diego kills containers when they "crash" this can be because:

•  the application exited
• the health check failed (today this is a port check however it will be possible to have a custom health check in the future)
• the application exceeds its memory use

Till CF explicitly introduces support for keeping crashed containers around in Deigo  - there are some options to implement post-mortem container forensics :

1. Modify the java buildpack release ruby script to upload the bits under /home/vcap rootfs to a s3 compatible blob store on JVM process exit. add OOM heapdump script that uploads heapdump to S3 storage
2. Create a JVM shutdown hook that introspects and copies the bits of the app and the file-system to a Riak-S2 blob store. Runtime.getRuntime().addShutdownHook(shutdownHook); see
3. Leverage a Tomcat Lifecycle Listener org.apache.catalina.LifecycleListener to do post-mortem activities. For a prototype look at the current Java buildpack’s ApplicationStartupFailureDetectingLifecycleListener
4. As a part of the release phase, a postStop.sh script is executed after the actual server has stopped or crashed. This script will report the death of the instance along with instructions on downloading any relevant files from the specific instance and also provide a default grace period of 30 seconds before the warden container gets estroyed. This approach is implemented by the cloud foundry weblogic buildpack
5. For the app in question explicitly specify a start command by adding a  ";sleep 1d" The push command would like this - cf push <app_name> -c "<original_command> ;sleep 1d". This will keep the container around for a day after process within the container has exited.  For a complete guide on troubleshooting  CF app issues take a look at 10-common-errors-when-pushing-apps-to-cloud-foundry.
6. The easiest way to achieve container preservation without any extra work is to simply snapshot the DEA VM that contains the warden container.  The DEA VM mapping to the container in question will need to be ascertained by looking in the DEA logs for the app GUID. The mapping function can be scripted in a log analytics engine like Splunk or  ELK.
7. For apps that are NOT running on the JVM , you can generate a  raw binary dump of the process memory by issuing a kill -6 ${PID} or kill -11 ${PID}. These dumps will need to either pulled manually or pushed from the warden container by an async task scheduled by the buildpack of the app runtime.  For a detailed discussion of various troubleshooting options with JVMs and Operating Systems refer to this cookbook from Kevin Grigorenko.
8. If you want to download contents of a running app's file directory from the warden container  use the cf-download plugin. cf download spring-music Usage: cf download APP_NAME [PATH] [--overwrite] [--verbose] [--omit omitted_path] [-i instance]

Session Persistence in Cloud Foundry

Lets start from the basics:

1. By default, the Tomcat instance is configured to store all Sessions and their data in memory. Under certain circumstances it my be appropriate to persist the Sessions and their data to a repository.

2. The Cloud Foundry Go Router maintains session affinity. So users will keep hitting their session in the same container. Furthermore when the user logs out or his session expires he will be placed on a different container and a new session will start. If for some reason the container that contains the session dies then the user gets kicked out a new session begins on a different container if session persistence is not configured. The following blog post from James Bayer is a good intro to CF session persistence. 

There are a couple of issues relating to the update of the _vcap_id cookie when a  new session is created. Please ensure that your CF release has the following fixes ... 

1. cloudfoundry/gorouter #76: Sticky Sessions lost on App Failure
2. After app instances are recreated every request round robins to a different app instance 

3. If the session contains state of the user that cannot be rebuilt in a stateless fashion then I recommend session persistence using some kind of database like (Postgres) or a NoSQL store (like Redis).

4. The java-buildpack already has in-built support for session persistence with Redis. To enable Redis-based session replication, simply bind a Redis service containing a name, label, or tag that has session-replication as a substring. see

5. Another option you have is to use Spring Session project that implements session facade across app servers and protocol tiers like REST and WebSocket.

Sticky sessions are an optimization and not a guarantee; however you should be able to rely on CF pulling out the right session content if all the app instances are bound to a session persistence service. 

Microservices Security at the Edge

There are two approaches to Microservices security 

1. Network-centric-approach

Keep the private microservices on an externally unroutable private shared sub-domain. Private services can only be accessed among themselves and from public microservices. Inbound security is implemented by adding a HAProxy acting as a layer 7 HTTP filter  behind the public ELB. Egress security is configured with PCF Application security groups. 

1. https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry/
2. PCF CookBook - AWS Private Sub-Domains 

2. Application-centric-approach

Leverage Spring-cloud-Zuul + spring-security-oauth2 to secure microservices reverse proxied by Zuul.  The API calls proxied by Zuul are protected using the OAuth2 protocol. Zuul proxied APIs can be protected using any security mechanism - not just Spring-Security. Spring security makes it easier to protect resources with less boilerplate.

The software based approach is explained in 

1. https://spring.io/blog/2015/02/03/sso-with-oauth2-angular-js-and-spring-security-part-v
2. https://spring.io/blog/2015/01/28/the-api-gateway-pattern-angular-js-and-spring-security-part-iv

These articles explain how to how to build an API Gateway to control the authentication and access to the backend resources using Spring Cloud.  Please note that when using the application-centric approach, the service endpoints are not blocked, they are protected with the security scheme put in place with Spring. Some clever hacker could still figure out the endpoint of the internal service bypassing the API Gateway tier; however since the resources are protected they will not be able to access anything. 

Debugging DEA Issues in Cloud Foundry

Scenario 0: Failure of the app at staging or runtime and the container still exists
cloudfoundry-community/cf-ssh

Scenario 1: Tracking down app by CPU usage, Accessing and Preserving a warden container, Monitoring the NATs message bus

Troubleshooting Applications | Cloud Foundry Docs

Scenario 2: Failures in Staging with no access to BOSH

1. Do a push with 'CF_TRACE=true' and look for a HTTP header called 'X-App-Staging-Log'. This will contain a route to the staging DEA's directory server. 

2. Then ssh onto a router VM and curl http://XXXX:XXXX@ROUTER_IP:PORT/routes. The username, password, and port can be found in /var/vcap/jobs/gorouter/config/gorouter.yml.

3. Look for the route to the directory server in the router's routes, which will tell you the IP of the staging DEA.

4. When you get on the DEA, you can look for your app's name in the dea_next/dea_next.log file.

5. Look for a warden_handle_id in the same log line as your app's name within the dea_next.log.

6. Once you get the warden_handle cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

Scenario 3: How can i determine where an app deployed is actually running ?  I need to debug something between an app and an external service

1. Find the GUID of the app using the following command CF_TRACE=true cf app APP_NAME

2. Use cf curl /v2/apps/GUID/stats to find the DEA VM host/port

3. If its a connection issue from the dea to the service .. get into the container and try to connect ... also look at the security groups

4. look for the app GUID in a  file called /var/vcap/data/dea_next/db/instances.json on the DEA VM. instances.json lists the state of the container, the app guid, and the warden_container_path.

5. Once you get the warden_handle then cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

An Architecture for Microservices using Spring on Cloud Foundry

Microservices Architecture with Spring on Cloud Foundry