Warden and Deigo container preservation in Cloud Foundry

There are a number of use cases for Warden container preservation in Cloud Foundry -

1. Post-mortem analysis of a compromised or hacked container
2. Problem Determination and Troubleshooting
3. Audit and compliance

There are two ways to achieve this before and after Diego i.e BD and AD

Before Deigo

This is a world where apps run in warden containers. Warden containers are garbage collected after a predetermined time period dea_next.crash_lifetime_secs Container container stays around on the DEA for about an hour or so by default before being removed. In addition to setting the dea_next.crash_lifetime_secs, also set dd the container_grace_time parameter to 3600 in the Warden configuration file stored in the warden.yml. container_grace parameter controls defines time before DEA deletes -containers.

After Deigo

Diego does not keep around containers after crashes. Diego kills containers when they "crash" this can be because:

•  the application exited
• the health check failed (today this is a port check however it will be possible to have a custom health check in the future)
• the application exceeds its memory use

Till CF explicitly introduces support for keeping crashed containers around in Deigo  - there are some options to implement post-mortem container forensics :

1. Modify the java buildpack release ruby script to upload the bits under /home/vcap rootfs to a s3 compatible blob store on JVM process exit. add OOM heapdump script that uploads heapdump to S3 storage
2. Create a JVM shutdown hook that introspects and copies the bits of the app and the file-system to a Riak-S2 blob store. Runtime.getRuntime().addShutdownHook(shutdownHook); see
3. Leverage a Tomcat Lifecycle Listener org.apache.catalina.LifecycleListener to do post-mortem activities. For a prototype look at the current Java buildpack’s ApplicationStartupFailureDetectingLifecycleListener
4. As a part of the release phase, a postStop.sh script is executed after the actual server has stopped or crashed. This script will report the death of the instance along with instructions on downloading any relevant files from the specific instance and also provide a default grace period of 30 seconds before the warden container gets estroyed. This approach is implemented by the cloud foundry weblogic buildpack
5. For the app in question explicitly specify a start command by adding a  ";sleep 1d" The push command would like this - cf push <app_name> -c "<original_command> ;sleep 1d". This will keep the container around for a day after process within the container has exited.  For a complete guide on troubleshooting  CF app issues take a look at 10-common-errors-when-pushing-apps-to-cloud-foundry.
6. The easiest way to achieve container preservation without any extra work is to simply snapshot the DEA VM that contains the warden container.  The DEA VM mapping to the container in question will need to be ascertained by looking in the DEA logs for the app GUID. The mapping function can be scripted in a log analytics engine like Splunk or  ELK.
7. For apps that are NOT running on the JVM , you can generate a  raw binary dump of the process memory by issuing a kill -6 ${PID} or kill -11 ${PID}. These dumps will need to either pulled manually or pushed from the warden container by an async task scheduled by the buildpack of the app runtime.  For a detailed discussion of various troubleshooting options with JVMs and Operating Systems refer to this cookbook from Kevin Grigorenko.
8. If you want to download contents of a running app's file directory from the warden container  use the cf-download plugin. cf download spring-music Usage: cf download APP_NAME [PATH] [--overwrite] [--verbose] [--omit omitted_path] [-i instance]

Session Persistence in Cloud Foundry

Lets start from the basics:

1. By default, the Tomcat instance is configured to store all Sessions and their data in memory. Under certain circumstances it my be appropriate to persist the Sessions and their data to a repository.

2. The Cloud Foundry Go Router maintains session affinity. So users will keep hitting their session in the same container. Furthermore when the user logs out or his session expires he will be placed on a different container and a new session will start. If for some reason the container that contains the session dies then the user gets kicked out a new session begins on a different container if session persistence is not configured. The following blog post from James Bayer is a good intro to CF session persistence. 

There are a couple of issues relating to the update of the _vcap_id cookie when a  new session is created. Please ensure that your CF release has the following fixes ... 

1. cloudfoundry/gorouter #76: Sticky Sessions lost on App Failure
2. After app instances are recreated every request round robins to a different app instance 

3. If the session contains state of the user that cannot be rebuilt in a stateless fashion then I recommend session persistence using some kind of database like (Postgres) or a NoSQL store (like Redis).

4. The java-buildpack already has in-built support for session persistence with Redis. To enable Redis-based session replication, simply bind a Redis service containing a name, label, or tag that has session-replication as a substring. see

5. Another option you have is to use Spring Session project that implements session facade across app servers and protocol tiers like REST and WebSocket.

Sticky sessions are an optimization and not a guarantee; however you should be able to rely on CF pulling out the right session content if all the app instances are bound to a session persistence service. 

Microservices Security at the Edge

There are two approaches to Microservices security 

1. Network-centric-approach

Keep the private microservices on an externally unroutable private shared sub-domain. Private services can only be accessed among themselves and from public microservices. Inbound security is implemented by adding a HAProxy acting as a layer 7 HTTP filter  behind the public ELB. Egress security is configured with PCF Application security groups. 

1. https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry/
2. PCF CookBook - AWS Private Sub-Domains 

2. Application-centric-approach

Leverage Spring-cloud-Zuul + spring-security-oauth2 to secure microservices reverse proxied by Zuul.  The API calls proxied by Zuul are protected using the OAuth2 protocol. Zuul proxied APIs can be protected using any security mechanism - not just Spring-Security. Spring security makes it easier to protect resources with less boilerplate.

The software based approach is explained in 

1. https://spring.io/blog/2015/02/03/sso-with-oauth2-angular-js-and-spring-security-part-v
2. https://spring.io/blog/2015/01/28/the-api-gateway-pattern-angular-js-and-spring-security-part-iv

These articles explain how to how to build an API Gateway to control the authentication and access to the backend resources using Spring Cloud.  Please note that when using the application-centric approach, the service endpoints are not blocked, they are protected with the security scheme put in place with Spring. Some clever hacker could still figure out the endpoint of the internal service bypassing the API Gateway tier; however since the resources are protected they will not be able to access anything. 

Debugging DEA Issues in Cloud Foundry

Scenario 0: Failure of the app at staging or runtime and the container still exists
cloudfoundry-community/cf-ssh

Scenario 1: Tracking down app by CPU usage, Accessing and Preserving a warden container, Monitoring the NATs message bus

Troubleshooting Applications | Cloud Foundry Docs

Scenario 2: Failures in Staging with no access to BOSH

1. Do a push with 'CF_TRACE=true' and look for a HTTP header called 'X-App-Staging-Log'. This will contain a route to the staging DEA's directory server. 

2. Then ssh onto a router VM and curl http://XXXX:XXXX@ROUTER_IP:PORT/routes. The username, password, and port can be found in /var/vcap/jobs/gorouter/config/gorouter.yml.

3. Look for the route to the directory server in the router's routes, which will tell you the IP of the staging DEA.

4. When you get on the DEA, you can look for your app's name in the dea_next/dea_next.log file.

5. Look for a warden_handle_id in the same log line as your app's name within the dea_next.log.

6. Once you get the warden_handle cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

Scenario 3: How can i determine where an app deployed is actually running ?  I need to debug something between an app and an external service

1. Find the GUID of the app using the following command CF_TRACE=true cf app APP_NAME

2. Use cf curl /v2/apps/GUID/stats to find the DEA VM host/port

3. If its a connection issue from the dea to the service .. get into the container and try to connect ... also look at the security groups

4. look for the app GUID in a  file called /var/vcap/data/dea_next/db/instances.json on the DEA VM. instances.json lists the state of the container, the app guid, and the warden_container_path.

5. Once you get the warden_handle then cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

An Architecture for Microservices using Spring on Cloud Foundry

Microservices Architecture with Spring on Cloud Foundry

How to Prevent a SONY style hack in your enterprise

The sony hack of critical application systems of both personal and company data has sent shivers down the spine of most corporate organizations.  How do you combat such an egregious targeted hacking of your infrastructure.  I wonder if a radical rethink of security systems is required. Security software needs to  evolve from a preventive to a reactive mindset. i.e. Just like distributed systems are designed around the fallacies of network computing, security software ought to be designed assuming that one or more of the attack vectors listed in the OWASP top 10 will succeed.  Securing data in transit and rest is critical for prevention; however an approach where even if the critical data is exposed no harm results will be key for future security.

From a microservices perspective follow these are the current best practices for application security -

1. Multi-factor authentication utilizing a Time-based One-time Password Algorithm http://www.insaneprogramming.be/blog/2014/05/14/two-factor-otp-security.  Universal 2nd Factor (U2F), a solution managed by the FIDO Alliance based on public key cryptography and inexpensive USB hardware tokens.

2. Perimeter security - Applying different security policies to public and private microservices via network segmentation https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry/

3. Authenticate - HTTP Endpoints using OAuth2, OpenID,  SAML, etc.,

4  Authorize - resources  in multiple layers, inspect HTTP request and do authorization logic based on URL patterns followed by method based security.

5. Out Of The Box Protection: Built-in framework protection against common attacks - like session fixation, XSS, SQL Injection, etc., Examples of frameworks include spring-security, Java EE and Java SE security models, etc.,

6. Data Parsimony : Datensparsamkeit taken from German privacy legislation describes the idea to only store as much personal information as is absolutely required for the business or applicable laws. Reduce data breach and data loss risk by anonymizing all information and capturing as little as possible and not reselling the data to third parties.

7. Data Loss Prevention Strategy - Prevent data loss and theft by putting in a comprehensive data loss and data theft policy in place.

8. Key Management: The only way to effectively secure data is strong encryption combined with rigorous hygiene around key management and access controls.

9. Web application Security Best Practices : Isolate critical information in the DB, Protect from Remote Code Execution, Flood and spam protection, Hash passwords with unique salts, Global XSS protection, Protect from SQL injection vulnerability, Protect from CSRF, Ask for password before modifying critical information, HTTP Strict Transport Security, Use HTTPS throughout the application and Validate session against browser and location

Microservices Reading List

As the year draws to the close I have decided to resume an activity that I really enjoyed i.e. Blogging. To this end this marks my first blog spot in six months.

As the microservices (MS) buzzword catches fire in the industry I have tried to dig underneath the surface and understand they why, how and when of microservices. When is it a suitable approach and when it is not. Why should you adopt a MS architecture. Is it simply a SOA implementation concern of a radical rethink of software architecture.  As 2015 progresses I intend to blog more about this topic and get into the nitty-gritty that will help regular software developers sort out the hype from the reality and understand what it takes to implement a MS based system.

For now I will paste here words of caution and optimism from the Thoughtworks Technology Radar Dec 2014

We remain convinced that microservices can offer significant advantages to organizations, in terms of improving team autonomy and faster frequency of change. The additional complexity that comes from distributed systems requires an additional level of maturity and investment. We are concerned that some teams are rushing in to adopting microservices without understanding the changes to development, test, and operations that are required to do them well. Our general advice remains simple. Avoid microservice envy and start with one or two services before rushing headlong into developing more, to allow your teams time to adjust and understand the right level of granularity

I encourage all software developers to form their own opinion on this subject. I have compiled a core list of books that are prerequisite reading to understand MS deeply. This list is to be delved into once you have done an initial reading of the microservices article and related videos. 

1. Implementing DDD 
2. Refactoring Databases
3. Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications 
4. Lean Enterprise: How High Performance Organizations Innovate at Scale
5. Reactive Enterprise with Actor Model: Application and Integration Patterns for Scala and Akka
6. Big Data Principles and best practices of scalable realtime data systems

Go Read!!