About Me

My photo
Rohit is an investor, startup advisor and an Application Modernization Scale Specialist working at Google.

Monday, October 31, 2016

Why Bother ? The Need for Rabbit and Redis through Pivotal Cloud Foundry

1. Need for self-provisonable reliable and production capable messaging and non-relational backing services in a time-bound fashion (typically less than 5 mins).

2. Reduce time to value for the application features and increase developer flow by giving the ability to create and bind services during application development avoiding ticket based speed bumps.

3. Existing IBM based messaging and relational stores are too heavyweight for cloud based applications and don't have quick turnaround for new instances.

4.  Pre-canned  recipes and collateral to leverage Redis and Rabbit to accelerate the migration to the Cloud. see blog posts https://medium.com/@KevinHoffman/migrating-apps-to-the-cloud-shunting-the-event-stream-8c2f6f309242#.5h3dpckl4 and https://medium.com/@KevinHoffman/orchestration-to-choreography-19686684fd44#.nxp73bxxk

5. Redis and RabbitMQ are the best-in-breed when it comes to session caching, event sourcing and messaging. Deep Integration and Ease of Usage with the Spring and the Spring Boot frameworks that automatically configure connections and credentials to Redis and Rabbit when an application is deployed to Pivotal Cloud Foundry.

6. Developer ramp-up time to Redis and RabbitMQ is much lower than other technologies in the same space.

7. These Pivotal Tech will help reduce IBM spend on DB2 and provide a pathway for migration away from MQ and DB2.

Wednesday, October 26, 2016

Snap Analysis of applications

We are often called upon to analyze applications for suitability of porting to Cloud Foundry. We quickly draw up the following chart and prepare a forrester style 4 quadrant graph of Technical suitability vs Business Criticality. I am embedding the technical criteria for evaluating an app below :









The quadrants on the SWOT analysis graph below are the axis of whatever is important to you to prioritize the apps - typically the axes are - Technical complexity and effort vs Strategic business importance. It is a way of categorizing your apps based on your goals and interests that come out during the inception.

Typically the X axis is (technical complexity - how much do you know about the app) - Y axis is (Business criticality - How important is this to the business) - other factors come into play as well


image credit goes to my buddy Joe Szodfridt

CQRS and Event Sourcing in Java with Spring Framework

Recently I have had the opportunity to delve deep in to the world of Event Sourcing and CQRS for a modernization project where we are refactoring a 1M LOC a decade old classic java application that does more than a billion dollars of revenue across multiple lines of business. If you have to embark on a similar journey these are the resources I found useful:

1. DDD Distilled : This is the best summary of the classic Eric Evan's DDD book and Vernon's iDDD book. There is also a more practical Tools and Techniques section which summarizes Event Storming technique very well. We used a 3 day event storming inception to discover the complex business and technical domain and figure out the bounded contexts and problem areas.

2. Event Storming from Alberto Brandolini: A very easy read on the why and how of event storming. Alberto's writing style is very entertaining. This is probably the most interesting book written in this space. Alberto and Paul Raynor's videos(1, 2) on event storming provide a good getting-started guide to the world of event storming.

3. This Gist from Szymon provides an excellent collection of videos to understand the basics of modeling, Domain-Driven Design, CQRS and Event Sourcing.

4. Online Class: Greg Young's entire Domain Driven Design, CQRS, and Event Sourcing class all online.

5. Read Event Sourcing and CQRS chapter from Cloud Native Go: Building Web Applications and Microservices for the Cloud with Go and React on Safari Online.

6. Can't say enough good things about Michael Ploed's talk on building microservices with event sourcing and CQRS at SpringOne 2016. His practical advice is sorely needed in understanding the production concerns when refactoring a monolith to event-sourcing.

7. Peruse significant discussions on the DDD CQRS Group google mailing list.

8. Blogs & Articles

9. Code Repositories:

10. Videos & Webinars:

Tuesday, August 2, 2016

Running WebSphere Liberty Profile or Tomcat in Cloud Foundry

Why should you chose to run your apps on a plain vanilla Tomcat/Jetty/Undertow servers with the Spring Framework instead of running on WebSphere Liberty Profile or any proprietary app server.
  • Oracle has stopped investing in JavaEE. Java-Gaurdians. Java EE innovation driven through the JCP process is driven to a complete halt thanks to lack of investment by its major supporters and Oracle hosting the Java EE standard  hostage with the proprietary TCK accessible only to Oracle and corporate licenses. [reboot]. Meanwhile innovation in the Spring community has taken off. 
  • Don't take my word on it. Read the analyst Stephen O' Grady's opinion on Spring Boot ... Better than ten years removed from the initial release of Rails, it seems strange to be writing about the “new” emphasis on projects intended to simplify the boostrapping process. But in spite of the more recent successes of projects like Bootstrap and Spring Boot, such projects are not the first priority for most technical communities. Perhaps because of the tendency to admire elegant solutions to problems of great difficulty, frameworks and on ramps to new community entrants tend to be an afterthought. In spite of this, they frequently prove to be immensely popular, because in any given community the people who know little about it far outnumber the experts. Even in situations, then, when the boot-oriented project and its opinions are outgrown, boot-style projects can have immense value simply because they widen the funnel of incoming developers. 
  • Thoughtworks technology radar now recommends Spring Boot as an ADOPT technology.  A lot of work has gone into Spring Boot to reduce complexity and dependencies, which largely alleviates our previous reservations. If you live in a Spring ecosystem and are moving to microservices, Spring Boot is now the obvious choice. For those not in Springland, Dropwizard is also worthy of serious consideration.
  • Developing applications using the full feature set of the WebSphere Liberty Profile will result in your application being non-portable to other app servers or migration from WebSphere Classic versions to the Liberty Profile. Furthermore you are offloading the control of your applications features to the app-server. Using Spring and vendored frameworks instead of provided dependencies at runtime yields control back to the application leading to a deterministic set of compile and runtime dependencies. Packaging apps with Spring lead to immutable artifacts. 
  • Idiomatic Applications written with the Spring Framework and Spring Boot on Pivotal Cloud Foundry running on a plain vanilla app server written in an idiomatic fashion are inherently 15 factor cloud native app. The applications are production-ready from the get-go. 
  • Organizations are now comfortable running business critical applications in production. Take a look at a recent Java Application Server survey report from Plumbr where the Tomcat installation base exceeded the 50% threshold for the second year in a row. The 58.22% share of the pie left no question about the winner. 


Tuesday, June 14, 2016

Multi-line Java stack traces out of order in Logstash and Splunk

Do you have customers frustrated by getting multi-line Java stack traces out of order? We're working on a clean solution in our enhanced metrics work, but here is a workaround courtesy @DavidLaing .
With the Java Logback library you can do this by adding 

"%replace(%xException){'\n','\u2028'}%nopex" 

to your logging config [1] , and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will display as a new line.


mutate {
  gsub => [ "[@message]", '\u2028', "
"]
^^^ Seems that passing a string with an actual newline in it is the only way to make gsub work
}

Thursday, June 9, 2016

Top 10 KPIs Cloud Foundry

The cloud foundry firehose and syslog streams generate tons of metrics and logs. What should a busy devops engineer look at ? My colleague Ford Donald put this awesome list of the top 10 KPIs of Cloud Foundry.

+----------------------------------+---------+--------------------+
| KPI                              | Description
+----------------------------------+---------+--------------------+
| rep.capacityRemainingMemory      | Available cell memory (sum for all cells)
| rep.capacityRemainingDisk        | Available cell disk (sum for all cells)
| bbs.LRPsRunning                  | Equivalent to # apps running
| bbs.RequestLatency               | CF of API is slow if this rises
| bbs.ConvergingLRPduration        | Apps or staging crashing if this rises
| stager.StagingRequestFailed      | Indicates bad app or buildpack pushes
| auctioneer.LRPauctionsFailed     | CF can’t place an app (out of container space)
| router.totalRoutes               | Size in routes of a PCF install (indicates uptake)
| router.requests                  | Tracks traffic flow thru a PCF
| bosh.healthmonitor.system.healthy| Indicates BOSH/VM health
+----------------------------------+---------+--------------------+

New PCF v1.10 "Monitoring PCF" docs section went live!  - https://docs.pivotal.io/pivotalcf/1-10/monitoring/index.html




Replatforming .NET applications to Cloud Foundry - Migrating .NET apps to Cloud Foundry


A lot of developers do not realize that Cloud Foundry can run a process from any runtime including .NET. Windows applications are managed side-by-side with Linux applications. .NET apps get support for key PCF features like scaling including autoscaling, monitoring, and high availability, logs and metrics, notifications, services including user-provided services and PCF set environment variables. BOSH and OpsManager now supports deployments on Microsoft Azure via pre-built ARM templates.This blog post provides details of the Windows support and provides guidance on migrating .NET apps to Cloud Foundry . This post build extensively on the work done by the Pivotal product engineering and the solutions team particularly Kevin Hoffman, Chris Umbel Zach Brown, Sam Cooper and Aaron Fortener.

There are two ways to run ASP.NET  apps on Cloud Foundry

1. Windows 2012 R2 Stack
 • Support Introduced in Nov. 2015
 • Supports.NET 3.5-4.5+
 • Requires Diego-Windows, Garden-Windows
 • BOSH-Windows support currently underway
 • App is pre-compiled, no detect. Leverage binary_buildpack  to push app
 • cf push mydotNetApp -s windows2012R2 -b binary_buildpack
 • Runs on Garden-Windows container
 • Win 2k12 provides process isolation primitives similar to those found in the Linux kernel
 • IronFrame abstracts these primitives into an API utilized byGarden-Windows
 • Win 2k12’s low-level API Linux kernel’s API
 • Therefore Garden-Windows process isolation functionality ≈ Garden-Linux
 • Developer OS is Windows

2. Linux cflinuxfs2 Stack
 • ASP.NET Core CLR only
 • Runs on Diego or DEA
 • Standard CF Ubuntu Stack
 • Community ASP.NET 5 Buildpack
 • cf push mydotNetApp -s cflinuxfs2 -b <asp.net5 bp>
 • Garden-Linux is the backing container
 • Garden-Linux containers are based on LXC (Linux cgroups)
 • Developer OS is Win, Mac OSX and Linux

ASP.NET Core vs ASP.NET 4.6


 ASP.NET Core
  • Web API
  • ASP.NET MVC
  • No Web Forms (yet)
  • No SignalR
ASP.NET 4.6
  • Battle-tested, hardened
  • Many years of testing and improvement
  • MVC, Web Forms, SignalR, etc.

Migrating Existing Applications

• For Greenfield Applications Try “Core First” approach
• For Replatforming and migrating Existing Apps Probably ASP.NET 4.x
Dependencies and framework requirements will likely dictate choice
• All versions of the .NET SDK supported by Windows Server 2012 R2 are supported on PCF including 3.5, 4.5, 4.5.1 and 4.6

Here are some of the details on how Garden-Windows secures applications


Non Cloud-Native stuff you will want to change

 • Reading/Writing to the Registry.
•  Super old versions of ASP.NET. Upgrade to .NET core or ASP.NET 4.x
 • Reading/Writing to the local disk. File systems are ephemeral. Replace usage of the file system e.g. disk logging even temp files with a S3 compatible blob store or external file store.
 • Does the app use integrated Windows Authentication? Replace  Integrated Windows Auth. with ADFS/OAuth2 to leverage UAA and Pivotal SSO Tiles
 • In-Process Session State / Sticky Sessions. Replace InProc, StateServer with out-of-process data store e.g. Redis, SQL Server
 • Externalize  environment-specific config in web.config into VCAP environment variables. If value changes between environments, don’t get it from web.config. Don’t use externalization methods that only work in Azure. If app relies on machine.config or applicationHost.config, could have issues in PCF.
 • For MSI-Installed services or drivers, Bin deploy dependencies with app.
 • Nearly every app will need refactoring to use Console logging and tracing. Customers who mess with applicationHost.config can interfere with tracing, logging, etc.
 • 32 bit ASP-Net builds/References/Builds/apps depending on 32-bit libraries will not work. Use of 32-bit dependencies or when the app is itself a 32-bit app this is a red flag.
 • Does the app run as a service or a WCF Self-hosted services as opposed to IIS-hosted ? These services will need to be self-hosted in IIS as part of the replatforming to PCF.
 • Apps requiring assemblies in Global Assembly Cache won't work in CF.
 • Backing Service Drivers requiring native installers / DLLs may not work in CF.
 • On Demand Apps and ones triggered by job controller will need to be refactored to run in CF to use CF one-off task support coming in the CC v3 API.
• MSMQ and tight integration with other MSFT server products via on-server dependencies will need to be refactored to use external backing services. .NET Core is now supported in the RabbitMQ .NET client.
• Routing to non-HTTP WCF endpoints and listeners for third party middleware will not work since only HTTP based inbound traffic is routed to the windows containers.
• Reliance on Microsoft-specific infrastructure? (e.g. MSMQ, BizTalk, SharePoint, etc). These apps do NOT make good candidates to port to PCF.

.NET Application Selection Criteria


References

Wednesday, June 8, 2016

Creating Chaos in Cloud Foundry

One of the key tenets of operational readiness is to be prepared for every emergency. The best way to  institutionalize this discipline is by repeatedly creating chaos in your own production deployment and monitor the system recovery. The list below is a listing a tools from the PCF Solutions team @ pivotal and others to cause chaos at all levels in the stack in Cloud Foundry.

Tools, Presentations & Repos:
https://github.com/xchapter7x/chaospeddler
https://github.com/xchapter7x/cf-app-attack
https://github.com/strepsirrhini-army/chaos-lemur
https://github.com/FidelityInternational/chaos-galago
https://github.com/skibum55/chaos-as-a-service
Monkeys & Lemurs and Locusts Oh My - Anti-Fragile Platforms

Type of test/event/task



1. BOSH
* bosh target (director ip)
* bosh login (director username/password obtained from Ops Man)
* bosh download manifest cf-(hash) ~/cf.yml
* bosh deployment ~/cf.yml
* bosh vms/cck
* bosh ssh
* bosh logs
* bosh debug (gives you the job/task logs)
2. VM Recovery
* Terminate a VM by deleting it in vSphere, watch it come back up
3. App Recovery
* Terminate an app by using cf plugin, watch it come back up.
4. Correlate logs?
* Watch logs for steps above
5. Chaos Monkeys
* Execute Chaos Lemur and watch bosh/cf respond
6. Director
* Shut VM down/delete in vCenter
* When its down, what app still runs?
* Once VM is gone, how do you get it back/rebuild?
7. Network switch
8. Hypervisor
9. Credentials that expire:
* Certs that have expiration date
* System Accounts (internal CF system accounts)
* vCenter API Account that CF uses
10. Log Insight goes down
11. Kill container
12. Kill VM
13. Kill DEA
14. Kill Router
15. Kill Health Manager
16. Kill Binary Repository
* Then scale
17. Over-allocate Hardware (how do we do it?)
18. Execute and backout a change to CF
19. Bulid Pack Upgrade and Roll Back
20. Right Apps have right build pack
21. Licensing server scenario (for example, can't connect)
22. Double single components (for example, 2 BOSH's)
23. Kill internal message bus
24. DNS
25. Clock drift



Chaos Testing Procedure: 
Kil vms from vsphere; used bosh tasks —no-filter in a loop to watch resurrector bring them up
bosh ssh and sudo kill -9 -1 are also fun
bosh ssh’d into a dea and killed a container

Monday, May 16, 2016

Application Transformation - The Pivotal Way

Chapter One: Perspective

Monolithic apps aka legacy applications evolve into a mess of of spaghetti code and features that support production workloads that run from airline systems to nuclear reactors to angrybird servers.

With the evolution of the cloud and the death of traditional enterprise vendors there is a huge wave of transformation sweeping across the entire industry neutering all the traditional heavyweight enterprise vendors unhinging apps from their app servers and running them on lighter weight runtimes or even serverless platforms.   

This wave has unleashed a rethinking of application bundling, packaging and deployment constructs at the logical and physical level in terms of domain modeling and immutable deployments, all leading to microservices based architectures that provide the ability to rapidly innovate with people and code.

From a software perspective a monolith is a combination of business capabilities or bounded contexts all enmeshed with a core context. The process of transforming  an app to a microservices based architecture entails separation of each of these bounded contexts into their own contexts.


Chapter Two: Process of breaking apart a monolith

1. Identify the seams of the application
2. Get started along a seam i.e. with a particular business capability. Ideally this should be something that must be isolated and adds business value.
3. Put fences around the existing feature function by writing tests and  exposing an API

API Fencing and Contracts
4. The API and the tests serve as the moat giving you the confidence to rewrite the existing business capability as a microservice.
5. Ensure that both old and new services are talking to the mothership. Validate that the microservice satisfies all the parent dependencies via canary testing. Optionally the Microservice communicates with the parent through an anti-corruption layer. An anti-corruption layer allows for the refactored microservice domain to remain pure. Think of it as a translator that speaks the languages of both the parent and the child.
6. Route traffic to both the monolith and the microservice with Blue/Green deployment.
7. Wrap the older feature code within the monolith with a feature flag disabled by default.
8. Once the microservice is functioning as expected then disable the same feature in the monolith.
9. Delete the Feature Flag  and the code associated with the feature in the monolith.
10. You have now successfully decomposed the business capability from the parent.
11. Scrub and repeat for all bounded contexts.
https://leanpub.com/Practicing-DDD
 Refactoring Legacy code with an Anti-Corruption Layer https://leanpub.com/Practicing-DDD 

Chapter Three: Identify the seams

One of the most difficult aspect of transformation of an app is figuring out where the seams lie.

The best way to do this is to talk to maintainers and users of the application. Users will describe difficulties in interacting with the application giving hints into which portions of the app need to be rewritten. The app maintainer has intimate details about the sections of code that are stable and those that change very frequently.

Static code analysis with code coverage tools will provide insight into code complexity, external dependencies and coupling. Runtime code analysis with logs and sequence diagrams will provide a map of data of event flows through the application.

Exercises like event storming will map out all the epics and user flows through the monolith. Event Storming is a useful way to do rapid "outside-in" domain modeling: starting with the events that occur in the domain rather than a static data model. Run as a facilitated workshop, it focuses on discovering key domain events, placing them along a timeline, identifying their triggers and then exploring their relationships. Event Storming provides a way to discover the big picture, with the goal of collectively understanding the domain in all of its complexity, before diving into solutions.

The onus is not to do a lot of upfront analysis; rather do just enough to get started. It is important to pick one and get started. Let the process of doing fuel the discovery. In many ways this is similar to hill climbing algorithms.

Chapter Four: People and Process Aspects of Transformation

It is critical NOT to ignore the softer side of app transformation.  The 12 tenets of lean app transformation as laid out in An Agile Approach To A Legacy System are
1. Don't reproduce legacy
2. Always ask the user what the problem is
3. Refactor a legacy application by delivering business value
4. Incrementally build trust - prove that you can do the hardest part of the problem
5. Build a small, self-selected team
6. Don't get hungup on process
7. Involve the whole team with larger refactorings so the team can move on as quickly as possible
8. Effective Teams need breakpoints
9. Don't sat no, say later. Treat politics as a user requirement
10. A System that connects to a legacy system must be tested using live feeds
11. Engage users and they not only won't they turn it off, they will fight some of your battles for you
12. Keep giving a good team motivated by giving them new hard problems - don't waste a good team

Risk Profile with the Pivotal Way
To be successful at application transformation, application decomposition, application replatforming and application refactoring it is critical that you get both the process and technology pieces right. One cannot succeed without the other. That is where a company like Pivotal that builds the PCF platform and runs Pivotal Labs an in house eXtreme Programming consultancy for customer success can help. We believe that the feedback cycle should be as short as possible. Automate everything you can with CI/CD. Build new skills through pairing and by doing. Failure is Productive. Experimentation Informs Strategy and Reduces Risk. Watch One > Do One > Teach One.

12 Factor App Transformation

Taking inspiration from the morning paper, today's blog post is a synopsis of the excellent work published In An Agile Approach to a Legacy System where Chris Stevenson and Andy Pols lay down the 12 the core principles to transform a monolith:

 1. Don't reproduce legacy

 2. Always ask the user what the problem is

 3. Refactor a legacy application by delivering business value

 4. Incrementally build trust - prove that you can do the hardest part of the problem

 5. Build a small, self-selected team

 6. Don't get hungup on process

 7. Involve the whole team with larger refactorings so the team can move on as quickly as possible

 8. Effective Teams need breakpoints

 9. Don't sat no, say later. Treat politics as a user requirement

 10. A System that connects to a legacy system must be tested using live feeds

 11. Engage users and they not only won't they turn it off, they will fight some of your battles for you

 12. Keep giving a good team motivated by giving them new hard problems - don't waste a good team