Supplying an external configuration to the Java Buildpack

I wanted to highlight a very important commit recently made to the Java Buildpack i.e.You can now configure the Java Buildpack with a custom server.xml via a repository.

Extension point for supplying an external Tomcat configuration

This allows a CF Platform Operator to provide a vetted server.xml that contains custom Tomcat configuration to the Java Buildpack without forking the buildpack. This feature is OFF by default. In order to enable it set the following env variable in the manifest:

Example in a manifest.yml
env: JBP_CONFIG_TOMCAT: "{ tomcat: { external_configuration_enabled: true }, external_configuration: { repository_root: \"http://repository...\" } }"

The artifacts that the repository provides must be in TAR format and must follow the Tomcat archive structure:

tomcat
|__conf
   |__context.xml
   |__server.xml
   |__web.xml
   |...

You can expect this feature to show up in the next release of the JavaBuildpack and its downstream clones like TomEE.

Deigo BBS, ETCD and Consul - 3 is a crowd

PCF 1.6 has now been completely redesigned on the Diego runtime. In case you are wondering all this all works now checkout the picture below -

Diego internally uses a couple of registries to store internal state primarily Consul for NAT-less service registry and ETCD as an ephemeral data store for the Diego bulletin board service. Once there are corruption issues due to connectivity or disks, these distributed registries  start arguing. They cannot reconcile the state of the world with the CloudController due to quorum or issues with leader election or distributed key lockout. Basically the instances need to get the same key.  You can specify the key in the newer versions, but there is no enforcement of state in the case of corruption.

I am sure there is a proper technical reason why these distributed K,V stores behave they way they do in these situations; however my intent here is to teach you how to get out of this situation if somehow you end up here.

So in situations where Consul and Etcd and Diego Brain BBS jobs act like errant brothers and fight with another to reconstitute the state of the world from the Cloud Controller, leverage the following recipe to keep sanity in the house.

1. Restart all the VMs of the Consul, ETCD and Diego BBS nodes in that order.

2. Wipe out member DB files i.e. nuke the data dir. on the ETCD VMs. monit restart all services on the ETCD nodes.To delete the contents of the etcd data store see:

1. Run bosh vms to identify the etcd nodes in your deployment.
2. Run bosh ssh to open a secure shell into each etcd node.
3. Run monit stop etcd on each etcd node.
4. Delete or move the etcd storage directory, /var/vcap/store, on each etcd node.
5. Run monit start etcd on each etcd node.

3. Reduce the number of Consul instances to 1 and  deploy the BOSH release again. If this fails, follow the same process for the other errant jobs. In the end you may end up with just one instance of these 3 jobs. Once you have a healthy deployment bump up the # of Job instances in the Resources page of the OpsManager.

4. Etcd may sometimes experience orphaned processes that leads to the Diego BBS job failing. This can be remedied by killall etcd on the Diego BBS VMs and subsequent redeployment.

5.  Nuke and Pave ... Take backup of your PCF deployment with cfops and do a brand new deployment followed by a restore.

State of JavaEE

There is a lot of chatter on the internet about Java EE being dead. As someone who spent a lot of time implementing some of the core specs like CDI and also helping customers adopt JavaEE I was interested in determining if this is true.

Unfortunately the anecdotal evidence points to JavaEE dying slowly. This is the evidence now points towards JavaEE being dead:

1. Many prominent Java EE evangelists - Folks who in some cases are friends and I admire deeply have left their respective Java EE shepherd companies. Reza Rahman from Oracle, Kenny Bastani from Oracle and Arun Gupta from Redhat to name just a few ...

2. Oracle, Redhat and IBM - the three most prominent supporters of JavaEE are now spending their money and developers innovating elsewhere. Oracle cloud,OpenShift and Bluemix respectively.

3. The advent of microservices has rendered the monolithic app-server dead. Gone are the days that you have the pack the kitchen sink in your app-server runtime. Nowadays developers ONLY want to use the runtime they want - case in point the rise of the micro-servers - Spring Boot, Dropwizard, Vert.x, Ratpack etc. The next evolution is the disappearance of these micro-servers and the move to serverless computing i.e. the developers are coding chained reactive lambda functions in the cloud. Not sure how I feel about that....

4. The pace of JavaEE specification implementation by vendors and the evolution of the specification itself has slowed to a glacial pace. Where is Java EE8 or Java EE9 ? Where is the Java EE equivalent of all the resiliency patterns like Circuit Breaker, Bulkhead and microservice patterns like service discovery, logging everything to stdout etc. CDI is the only major hope left for JavaEE. They need to rebase the entire platform on CDI and start innovating like crazy. Unfortunately this cannot be done by committee.

There is still a LOT of code running on application servers. WebSphere, WebLogic and JBOSS were the goto runtime and middleware OS in the 90s and early 2Ks.  I care about these apps and if you do too start decomposing them and running them on lightweight runtimes like Spring Boot, Jetty whatever blows your horn. Then move these services to the cloud using Docker or Buildpacks.  If the apps don't add any business value then move them to lighter runtimes within the JavaEE stack i.e. move to WebSphere Liberty or JBOSS Wildfly.

This should serve as a cautionary tale. I am sure a lot of people with disagree with what I have written. But hey this my opinion you should form one too ....

Au Revoir

Update: There is a new organization called Java EE Gaurdians https://dzone.com/articles/who-are-the-java-ee-guardians-and-why-you-should-c that seems to be JavaEEs best shot at survival.