Cloud Foundry Java Buildpack Debugging ProTips and Common extension scenarios

Java Buildpack Debugging - During Staging of an Application

The best technique to debug an application that fails during staging is to set the JBP_LOG_LEVEL env. variable to DEBUG and restage the app.

cf set-env <APP> JBP_LOG_LEVEL DEBUG

Tomcat Container Debugging in Cloud Foundry

To enable system component logging of Tomcat  from within the app. Place the following logging.properties in the WEB-INF/classes dir. of your app. 

handlers = com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler

.handlers = com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler

com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler.level = FINE

org.apache.catalina.core.level = ALL 

org.apache.catalina.realm.level = ALL

org.apache.catalina.authenticator.level = ALL

org.apache.catalina.session.level= ALL

This properties file relies on Tomcat to load the logging.properties from both the conf directory and the application itself.  Tomcat's implementation of the java.util.logging API is enabled by default, and supports per classloader configuration, in addition to the regular global java.util.logging configuration. This means that logging can be configured at the following layers:

• Globally. That is usually done in the ${catalina.base}/conf/logging.properties file. The file is specified by the java.util.logging.config.file System property which is set by the startup scripts. If it is not readable or is not configured, the default is to use the ${java.home}/lib/logging.properties file in the JRE.
• In the web application. The file will be WEB-INF/classes/logging.properties

This technique allows you to get DEBUG level Tomcat container logs without forking (modifying java-buildpack/blob/master/resources/tomcat/conf/logging.properties) the Java buildpack 

Extending the Java Buildpack

Traditionally extending the Java Buildpack meant forking the buildpack. The recommendation is that fork changes should be kept at a minimum. JBP 3.0 introduced support for user configuration via environment variables. Where possible the modification to the java buildpack function should be done via environment variables. Fork changes should be restricted to features that absolutely cannot be implemented in an app like adding strong cryptography or adding certificates to the JVM keystore.

Buildpack configuration can be overridden with an environment variable matching the configuration file you wish to override minus the .yml extension and with a prefix of JBP_CONFIG. The value of the variable should be valid inline yaml. For example, to change the default version of Java to 7 and adjust the memory heuristics apply this environment variable to the application. doc

cf set-env my-application JBP_CONFIG_OPEN_JDK_JRE '[jre: {version: 1.7.0_+}, memory_calculator: {memory_heuristics: {heap: 85, stack: 10}}]'

Common scenarios for Extending the Java Buildpack

1. Configuring LDAP Security to protect service endpoints with LDAP in Cloud Foundry

The LDAP connection and other configuration in an app can be done via a META-INF/context.xml.This technique allows context files to be packaged within WARs. In Tomcat, a Context represents a single web application. Tomcat uses the Context configuration element to contain information about components required by a given application, such as databases, realms, or custom loaders. Additionally, the Context element can be configured with a wide variety of attributes that control things such as logging, reload permissions, caching, and more.

Typically add the following in the web.xml

<security-constraint>

    <web-resource-collection>

      <web-resource-name>AppConfigResource</web-resource-name>

           <url-pattern>/s-music/p/__admin/console</url-pattern>

      </web-resource-collection>

    <auth-constraint>

      <role-name>*</role-name>

    </auth-constraint>

</security-constraint>

There needs to be a corresponding JNDIRealm definition in the application context. i.e. the META-INF dir. of the app should contain a context.xml

<?xml version="1.0" encoding="UTF-8"?>
<Context>          
       <Realm className="org.apache.catalina.realm.JNDIRealm"
       debug="99"
       resourceName="AuthIn"
       connectionURL="ldap://${LDAP-url}"
       authentication="simple"
       referrals="follow"
       connectionName="CN=Service Account\, ztcserver,OU=Service,OU=Accounts,OU=Administration,DC=rohitkelapure,DC=local"
       connectionPassword="${LDAPconnectPassword}"
       userSubtree="true"
       userBase="OU=Accounts,OU=Administration,DC=rohitkelapure,DC=local"
       userSearch="(sAMAccountName={0})"
       roleBase="OU=Permissions,OU=Groups,OU=Administration,DC=rohitkelapure,DC=local"
       roleName="cn"
       roleSubtree="true"
       roleSearch="(member={0})"
        />
</Context>

When the JNDIRealm is not authenticating correctly it is very helpful to enable tomcat container logging for the org.apache.catalina.realm and org.apache.catalina.authenticator loggers.

Note the use of system properties in the context.xml. Tomcat supports parameter substitution in its configuration files using the ANT style ${var}. The variables available are pulled in via System.getProperties() automatically. Further you can push variables into this area from the command line using the -Dname=value option for the Tomcat launch. These system properties can be set when pushing the app using the JAVA_OPTS env. variable

cf se spring-music JAVA_OPTS "-DLDAP-url=ldap://ldap.example.com/dc=example,dc=com -DLDAPconnectPassword=passw0rd"

2. Adding SSLCerts to the JVM

Leverage the Java Buildpack resources support to override and add files into the buildpack. The JBP copy_resources support overlays the contents of the resources directory onto a component's sandbox.

To Add your own SSL certs to the Oracle JRE:

mkdir -p resources/oracle_jdk_jre/lib/security/cacerts
Copy your certs into the newly created cacerts directory.

3. Add strong JCE support to the JRE

mkdir -p resources/open_jdk_jre/lib/security/
Copy the local_policy.jar into the newly created security directory.

4. Enable GC logging

1. Uncomment and modify the value of the java_opts setting of the config/java_opts.yml file.
2. As an example see https://github.com/pivotalservices/oracle-jre-java-buildpack/blob/master/config/java_opts.yml#L19

Upstream Maintenance of a forked Buildpack

Credit for this section goes to my colleague David Malone (@malonedave). 

Pulling Upstream Changes  into your fork of the JBP 

1. Configure Ruby to use your local Gem repo
2. Install all of the necessary tools; Ruby, RVM, Bundler 
3. git clone http://git.mycompany.com/java-buildpack.git
4. cd java-buildpack
5. git remote add upstream https://github.com/cloudfoundry/java-buildpack.git
6. git pull upstream master
7. git push origin master

Packaging the Java Buildpack

The offline package is a version of the buildpack designed to run without access to a network. It packages the latest version of each dependency (as configured in the config/ directory) and disables remote_downloads. This package is about 180M in size. To create the offline package, use the OFFLINE=true argument:

1. cd java-buildpack
2. export BUNDLE_GEMFILE=$PWD/Gemfile
3. bundle install
4. bundle exec rake (runs all unit & integration tests)- requires CentOS
5. bundle exec rake package VERSION=3.1 OFFLINE=true VERSION=MY_JBP_OFFLINE_3.1