Debugging DEA Issues in Cloud Foundry

Scenario 0: Failure of the app at staging or runtime and the container still exists
cloudfoundry-community/cf-ssh

Scenario 1: Tracking down app by CPU usage, Accessing and Preserving a warden container, Monitoring the NATs message bus

Troubleshooting Applications | Cloud Foundry Docs

Scenario 2: Failures in Staging with no access to BOSH

1. Do a push with 'CF_TRACE=true' and look for a HTTP header called 'X-App-Staging-Log'. This will contain a route to the staging DEA's directory server. 

2. Then ssh onto a router VM and curl http://XXXX:XXXX@ROUTER_IP:PORT/routes. The username, password, and port can be found in /var/vcap/jobs/gorouter/config/gorouter.yml.

3. Look for the route to the directory server in the router's routes, which will tell you the IP of the staging DEA.

4. When you get on the DEA, you can look for your app's name in the dea_next/dea_next.log file.

5. Look for a warden_handle_id in the same log line as your app's name within the dea_next.log.

6. Once you get the warden_handle cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

Scenario 3: How can i determine where an app deployed is actually running ?  I need to debug something between an app and an external service

1. Find the GUID of the app using the following command CF_TRACE=true cf app APP_NAME

2. Use cf curl /v2/apps/GUID/stats to find the DEA VM host/port

3. If its a connection issue from the dea to the service .. get into the container and try to connect ... also look at the security groups

4. look for the app GUID in a  file called /var/vcap/data/dea_next/db/instances.json on the DEA VM. instances.json lists the state of the container, the app guid, and the warden_container_path.

5. Once you get the warden_handle then cd /var/vcap/data/warden/depot/<warden_handle>; ./bin/wsh

An Architecture for Microservices using Spring on Cloud Foundry

Microservices Architecture with Spring on Cloud Foundry

How to Prevent a SONY style hack in your enterprise

The sony hack of critical application systems of both personal and company data has sent shivers down the spine of most corporate organizations.  How do you combat such an egregious targeted hacking of your infrastructure.  I wonder if a radical rethink of security systems is required. Security software needs to  evolve from a preventive to a reactive mindset. i.e. Just like distributed systems are designed around the fallacies of network computing, security software ought to be designed assuming that one or more of the attack vectors listed in the OWASP top 10 will succeed.  Securing data in transit and rest is critical for prevention; however an approach where even if the critical data is exposed no harm results will be key for future security.

From a microservices perspective follow these are the current best practices for application security -

1. Multi-factor authentication utilizing a Time-based One-time Password Algorithm http://www.insaneprogramming.be/blog/2014/05/14/two-factor-otp-security.  Universal 2nd Factor (U2F), a solution managed by the FIDO Alliance based on public key cryptography and inexpensive USB hardware tokens.

2. Perimeter security - Applying different security policies to public and private microservices via network segmentation https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry/

3. Authenticate - HTTP Endpoints using OAuth2, OpenID,  SAML, etc.,

4  Authorize - resources  in multiple layers, inspect HTTP request and do authorization logic based on URL patterns followed by method based security.

5. Out Of The Box Protection: Built-in framework protection against common attacks - like session fixation, XSS, SQL Injection, etc., Examples of frameworks include spring-security, Java EE and Java SE security models, etc.,

6. Data Parsimony : Datensparsamkeit taken from German privacy legislation describes the idea to only store as much personal information as is absolutely required for the business or applicable laws. Reduce data breach and data loss risk by anonymizing all information and capturing as little as possible and not reselling the data to third parties.

7. Data Loss Prevention Strategy - Prevent data loss and theft by putting in a comprehensive data loss and data theft policy in place.

8. Key Management: The only way to effectively secure data is strong encryption combined with rigorous hygiene around key management and access controls.

9. Web application Security Best Practices : Isolate critical information in the DB, Protect from Remote Code Execution, Flood and spam protection, Hash passwords with unique salts, Global XSS protection, Protect from SQL injection vulnerability, Protect from CSRF, Ask for password before modifying critical information, HTTP Strict Transport Security, Use HTTPS throughout the application and Validate session against browser and location

Microservices Reading List

As the year draws to the close I have decided to resume an activity that I really enjoyed i.e. Blogging. To this end this marks my first blog spot in six months.

As the microservices (MS) buzzword catches fire in the industry I have tried to dig underneath the surface and understand they why, how and when of microservices. When is it a suitable approach and when it is not. Why should you adopt a MS architecture. Is it simply a SOA implementation concern of a radical rethink of software architecture.  As 2015 progresses I intend to blog more about this topic and get into the nitty-gritty that will help regular software developers sort out the hype from the reality and understand what it takes to implement a MS based system.

For now I will paste here words of caution and optimism from the Thoughtworks Technology Radar Dec 2014

We remain convinced that microservices can offer significant advantages to organizations, in terms of improving team autonomy and faster frequency of change. The additional complexity that comes from distributed systems requires an additional level of maturity and investment. We are concerned that some teams are rushing in to adopting microservices without understanding the changes to development, test, and operations that are required to do them well. Our general advice remains simple. Avoid microservice envy and start with one or two services before rushing headlong into developing more, to allow your teams time to adjust and understand the right level of granularity

I encourage all software developers to form their own opinion on this subject. I have compiled a core list of books that are prerequisite reading to understand MS deeply. This list is to be delved into once you have done an initial reading of the microservices article and related videos. 

1. Implementing DDD 
2. Refactoring Databases
3. Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications 
4. Lean Enterprise: How High Performance Organizations Innovate at Scale
5. Reactive Enterprise with Actor Model: Application and Integration Patterns for Scala and Akka
6. Big Data Principles and best practices of scalable realtime data systems

Go Read!!