Top 50 Cloud Foundry Github Repositories

Tools, Hidden gems and Utilities for developing on and deploying Cloud Foundry

1. https://github.com/pivotalservices/cfops -This is simply an automation that is based on the supported way to back up Pivotal Cloud Foundry.
2. https://github.com/pivotalservices/shekel - Tool for capacity planning, sizing and costing a production deployment of Cloud Foundry.
3. https://start.spring.io/ - A web-based quickstart generator for Spring projects  https://github.com/spring-io/initializr
4. https://github.com/cloudfoundry-community/cf-boshworkspace -  Easily deploy CloudFoundry in various configurations, small to large, in AWS.
5. https://github.com/logsearch/logsearch-for-cloudfoundry - A Logsearch addon that customizes Logsearch to work with Cloud Foundry data.
6. https://github.com/logsearch/logsearch-boshrelease - A BOSH-scalable Elasticsearch + Logstash + Kibana release http://www.logsearch.io
7. https://github.com/dpinto-pivotal/cf-SpringBootTrader - Microservices  showcase for Spring Trader using spring boot + Spring Cloud also https://github.com/spring-cloud-samples
8. https://github.com/pivotalservices/spring-music - sample app for using database services on Cloud Foundry with the Spring Framework. Great for testing spring cloud connectors. See https://github.com/pivotal-cf-experimental/spring-music-on-cf-benchmarking for Apache JMeter stress tests.
9. https://github.com/cloudfoundry/cli - A CLI for Cloud Foundry written in Go. Look in https://github.com/cloudfoundry/cli/releases for downloads.
10.  http://plugins.cfapps.io/ui/ - Repository of cf-cli plugins https://github.com/cloudfoundry-incubator/cli-plugin-repo - public repository for community created CF CLI plugins
11. https://github.com/pivotalservices/pse-training-cf-hw-service-broker - Lab showcases Cloud foundry service broker concepts. Writing, adding and using Brokered Services
12. https://github.com/cloudfoundry/cli/tree/master/plugin_examples -  Instructions for writing your own Cloud Foundry CLI plugin 
13. https://github.com/pivotalservices/bosh-lite-install - Scripts to install bosh-lite on your local machine
14. https://github.com/pivotalservices/pse-training-cf-bp-module - Obtain the buildpack source code, extend it, package it, deploy it, and test it.
15. https://github.com/pivotalservices/pivotal-cf-irules - Library of F5 iRules which may be useful when working with Pivotal CF.
16. https://github.com/pivotalservices/java-r-buildpack - Heroku forked 'R' buildpack for CF that includes Rserve for supporting remote callers using an Rserve client library.
17. https://github.com/cloudfoundry-community/spring-boot-cf-service-broker - Spring Boot project for creating Cloud Foundry service brokers.
18. https://github.com/cloudfoundry-community/terraform-aws-cf-install - One click deploy of Cloud Foundry into an AWS VPC
19. https://github.com/cloudfoundry-community/cf-boshworkspace - easily deploy CloudFoundry in various configurations, small to large, in AWS
20. https://github.com/cloudfoundry-community/bosh-bootstrap - Simplest way to get a micro bosh running and upgrade/destroy it over time
21. https://github.com/cloudfoundry-community/firehose-to-syslog - Send firehose events from Cloud Foundry to syslog.
22. https://github.com/cloudfoundry-community/delete-all-bindings-and-services/blob/master/bin/delete.sh - cf-cli curl script to delete all bindings and services
23. https://github.com/cloudfoundry-community/kibana-me-logs/blob/master/bin/upgrade-all.sh - cf-cli curl script to upgrade all apps
24. https://github.com/krujos/cf-org-usage-report - Script to report on the amount of memory used against the currently applied quota. CLI Plugin; https://github.com/krujos/usagereport-plugin/  https://github.com/malston/notification-quota & https://github.com/malston/cloudfoundry-orphan-space-removal - Quota & Space Mgmt.
25. https://github.com/Pivotal-Field-Engineering/PCF-demo/tree/micro-services - Demonstration of CF as an application platform
26. https://github.com/Pivotal-Field-Engineering/postgres-bosh-release -   Simple bosh release for PostgreSQL. 
27. https://github.com/Pivotal-Field-Engineering/pcf-workshop-devops - Workshop for CF Devops
28. https://github.com/pivotalservices/cf-egress-tester - A quick app to test network egress from a CF space. Used for security group testing.  Helps test firewall access.
29. https://github.com/pivotalservices/uaaldapimport - Pre-populate the users with UAA api and CloudController api. So they can have all the roles before user logs in.
30. https://github.com/pivotalservices/cloudfoundry-java-logging - Injects CF VCAP_APPLICATION environment variable into a variety of Java logging frameworks. 
31. https://github.com/pivotalservices/cf-riak-cs-release - A BOSH release for riak and riak-cs
32. https://github.com/cloudfoundry-community/cf-docs-contrib/wiki - scrapbook of  design docs, Pivotal Trackers and info and links about CF. 
33. https://blog.starkandwayne.com/2014/07/10/resurrecting-bosh-with-binary-boshes - Highly available BOSH.
34. https://github.com/cloudfoundry-community/docker-services-boshworkspace - Fastest way to deploy Docker Services in combination with Cloud Foundry onto bosh-lite. 
35. https://github.com/cloudfoundry-incubator/route-registrar - A standalone executable written in golang that continuously broadcasts a route using NATS to the CF router.
36. https://github.com/cloudfoundry-community/cf-java-component - A set of libraries for building Cloud Foundry components on the Java platform
37. https://github.com/cloudfoundry-community - BOSH releases for haproxy, docker-services, route-registrar, memcache, hazelcast, ...
38. https://github.com/cloudfoundry-community/.net-buildpack - Cloud Foundry buildpack for running .NET applications
39. https://github.com/Pivotal-Field-Engineering/Drupal - Deploy Drupal CMS to CF
40. https://github.com/cloudfoundry-community/traveling-cf-admin - packages CLIs used by CF admins - the cf CLI and useful admin plugins, and the uaac CLI without requiring ruby.
41. https://bosh.io/docs/create-release.html - Simplest way to create a CF BOSH release. also see https://github.com/cloudfoundry/bosh-notes & https://bosh.io/docs Honorable mention - Generators for creating and sharing BOSH releases https://github.com/cloudfoundry-community/bosh-gen
42. https://github.com/cloudfoundry-incubator/admin-ui - An application for viewing Cloud Foundry metrics and operations data
43. https://github.com/cloudfoundry-incubator/lattice - Playground for next-gen CF based on Deigo
44. https://github.com/cloudfoundry-incubator/diego-release - BOSH release for deploying Diego and associated tasks. see https://github.com/cloudfoundry-incubator/diego-design-notes
45. https://github.com/cloudfoundry-incubator/pat - Super-simple load generation/performance testing framework for quickly and easily running load against CF
46. https://github.com/cloudfoundry/cf-acceptance-tests - test suite exercises a full Cloud Foundry deployment using the golang cf CLI and curl. Includes the DORA test app. Also see https://github.com/cloudfoundry/cf-smoke-tests - Smoke tests for CloudFoundry that are safe to run in a production environment
47. https://github.com/cloudfoundry-incubator/notifications - A notifications component that parts of CF can use to send email to end users.
48. https://github.com/concourse/concourse - BOSH release for the Concourse CI. Tutorial - https://github.com/starkandwayne/concourse-tutorial
49. https://github.com/cloudfoundry/java-buildpack-support - Access logging, lifecycle support and tomcat logging support for the Java Buildpack.
50. https://github.com/spring-io/sagan - The spring.io site and reference app. https://github.com/bclozel/spring-resource-handling - Spring 4.1 Resource handling
51. https://github.com/xchapter7x/autopilot - cf plugin for hands-off, Blue/Green zero downtime application deploys & https://github.com/krujos/scaleover-plugin - Uses cf scale and a time interval to move from B-> G. Look for these two getting merged in the future.
52. Report that shows which apps are using which buildpacks - https://github.com/krujos/oohimtelling

Cloud Foundry Java Buildpack Debugging ProTips and Common extension scenarios

Java Buildpack Debugging - During Staging of an Application

The best technique to debug an application that fails during staging is to set the JBP_LOG_LEVEL env. variable to DEBUG and restage the app.

cf set-env <APP> JBP_LOG_LEVEL DEBUG

Tomcat Container Debugging in Cloud Foundry

To enable system component logging of Tomcat  from within the app. Place the following logging.properties in the WEB-INF/classes dir. of your app. 

handlers = com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler

.handlers = com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler

com.gopivotal.cloudfoundry.tomcat.logging.CloudFoundryConsoleHandler.level = FINE

org.apache.catalina.core.level = ALL 

org.apache.catalina.realm.level = ALL

org.apache.catalina.authenticator.level = ALL

org.apache.catalina.session.level= ALL

This properties file relies on Tomcat to load the logging.properties from both the conf directory and the application itself.  Tomcat's implementation of the java.util.logging API is enabled by default, and supports per classloader configuration, in addition to the regular global java.util.logging configuration. This means that logging can be configured at the following layers:

• Globally. That is usually done in the ${catalina.base}/conf/logging.properties file. The file is specified by the java.util.logging.config.file System property which is set by the startup scripts. If it is not readable or is not configured, the default is to use the ${java.home}/lib/logging.properties file in the JRE.
• In the web application. The file will be WEB-INF/classes/logging.properties

This technique allows you to get DEBUG level Tomcat container logs without forking (modifying java-buildpack/blob/master/resources/tomcat/conf/logging.properties) the Java buildpack 

Extending the Java Buildpack

Traditionally extending the Java Buildpack meant forking the buildpack. The recommendation is that fork changes should be kept at a minimum. JBP 3.0 introduced support for user configuration via environment variables. Where possible the modification to the java buildpack function should be done via environment variables. Fork changes should be restricted to features that absolutely cannot be implemented in an app like adding strong cryptography or adding certificates to the JVM keystore.

Buildpack configuration can be overridden with an environment variable matching the configuration file you wish to override minus the .yml extension and with a prefix of JBP_CONFIG. The value of the variable should be valid inline yaml. For example, to change the default version of Java to 7 and adjust the memory heuristics apply this environment variable to the application. doc

cf set-env my-application JBP_CONFIG_OPEN_JDK_JRE '[jre: {version: 1.7.0_+}, memory_calculator: {memory_heuristics: {heap: 85, stack: 10}}]'

Common scenarios for Extending the Java Buildpack

1. Configuring LDAP Security to protect service endpoints with LDAP in Cloud Foundry

The LDAP connection and other configuration in an app can be done via a META-INF/context.xml.This technique allows context files to be packaged within WARs. In Tomcat, a Context represents a single web application. Tomcat uses the Context configuration element to contain information about components required by a given application, such as databases, realms, or custom loaders. Additionally, the Context element can be configured with a wide variety of attributes that control things such as logging, reload permissions, caching, and more.

Typically add the following in the web.xml

<security-constraint>

    <web-resource-collection>

      <web-resource-name>AppConfigResource</web-resource-name>

           <url-pattern>/s-music/p/__admin/console</url-pattern>

      </web-resource-collection>

    <auth-constraint>

      <role-name>*</role-name>

    </auth-constraint>

</security-constraint>

There needs to be a corresponding JNDIRealm definition in the application context. i.e. the META-INF dir. of the app should contain a context.xml

<?xml version="1.0" encoding="UTF-8"?>
<Context>          
       <Realm className="org.apache.catalina.realm.JNDIRealm"
       debug="99"
       resourceName="AuthIn"
       connectionURL="ldap://${LDAP-url}"
       authentication="simple"
       referrals="follow"
       connectionName="CN=Service Account\, ztcserver,OU=Service,OU=Accounts,OU=Administration,DC=rohitkelapure,DC=local"
       connectionPassword="${LDAPconnectPassword}"
       userSubtree="true"
       userBase="OU=Accounts,OU=Administration,DC=rohitkelapure,DC=local"
       userSearch="(sAMAccountName={0})"
       roleBase="OU=Permissions,OU=Groups,OU=Administration,DC=rohitkelapure,DC=local"
       roleName="cn"
       roleSubtree="true"
       roleSearch="(member={0})"
        />
</Context>

When the JNDIRealm is not authenticating correctly it is very helpful to enable tomcat container logging for the org.apache.catalina.realm and org.apache.catalina.authenticator loggers.

Note the use of system properties in the context.xml. Tomcat supports parameter substitution in its configuration files using the ANT style ${var}. The variables available are pulled in via System.getProperties() automatically. Further you can push variables into this area from the command line using the -Dname=value option for the Tomcat launch. These system properties can be set when pushing the app using the JAVA_OPTS env. variable

cf se spring-music JAVA_OPTS "-DLDAP-url=ldap://ldap.example.com/dc=example,dc=com -DLDAPconnectPassword=passw0rd"

2. Adding SSLCerts to the JVM

Leverage the Java Buildpack resources support to override and add files into the buildpack. The JBP copy_resources support overlays the contents of the resources directory onto a component's sandbox.

To Add your own SSL certs to the Oracle JRE:

mkdir -p resources/oracle_jdk_jre/lib/security/cacerts
Copy your certs into the newly created cacerts directory.

3. Add strong JCE support to the JRE

mkdir -p resources/open_jdk_jre/lib/security/
Copy the local_policy.jar into the newly created security directory.

4. Enable GC logging

1. Uncomment and modify the value of the java_opts setting of the config/java_opts.yml file.
2. As an example see https://github.com/pivotalservices/oracle-jre-java-buildpack/blob/master/config/java_opts.yml#L19

Upstream Maintenance of a forked Buildpack

Credit for this section goes to my colleague David Malone (@malonedave). 

Pulling Upstream Changes  into your fork of the JBP 

1. Configure Ruby to use your local Gem repo
2. Install all of the necessary tools; Ruby, RVM, Bundler 
3. git clone http://git.mycompany.com/java-buildpack.git
4. cd java-buildpack
5. git remote add upstream https://github.com/cloudfoundry/java-buildpack.git
6. git pull upstream master
7. git push origin master

Packaging the Java Buildpack

The offline package is a version of the buildpack designed to run without access to a network. It packages the latest version of each dependency (as configured in the config/ directory) and disables remote_downloads. This package is about 180M in size. To create the offline package, use the OFFLINE=true argument:

1. cd java-buildpack
2. export BUNDLE_GEMFILE=$PWD/Gemfile
3. bundle install
4. bundle exec rake (runs all unit & integration tests)- requires CentOS
5. bundle exec rake package VERSION=3.1 OFFLINE=true VERSION=MY_JBP_OFFLINE_3.1

Hey Coud Foundry Where my MBeans @ ?

If you have pushed a java application to Cloud Foundry you have experienced the pain of NOT having an accessible Mbean server.  Cloud Foundry by default opens only ONE ingress port over the HTTP protocol.

The Java Management Extensions Instrument and Agent Specification defines the concept of connectors. A connector makes a Java Management Extensions (JMX) technology MBean server accessible to remote Java technology-based clients. The client end of a connector exports essentially the same interface as the MBean server.

Typical connectors connect to the MBean server over RMI. Herein lies the problem because the RMI-IIOP port is NOT open on the container running the CF app. Therefore an alternative mechanism is needed that allows a client to talk to the Mbean server over HTTP.

This is where Jolokia comes to the rescue. Jolokia is an agent based approach for remote JMX access. It is an alternative to standard JSR 160 connectors. The communication between client and agent goes over HTTP (either GET or POST), where the request and response payload is represented in JSON.

The best way to incorporate Jolokia in your app is by incorporating the Jolokia agent into the application by adding the following to the pom.xml or build.gradle

//Jolokia

runtime 'org.jolokia:jolokia-core:1.3.1'

runtime 'org.jolokia:jolokia-jsr160:1.3.1'

The Agent Servlet is also listed in the web.xml

<web-app xmlns="http://java.sun.com/xml/ns/javaee"

     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

     xsi:schemaLocation="http://java.sun.com/xml/ns/javaee 

     http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"

     version="3.0">

  <display-name>Spring-Music</display-name>

  

  <servlet>

    <servlet-name>jolokia-agent</servlet-name>

    <servlet-class>org.jolokia.http.AgentServlet</servlet-class>

    <load-on-startup>1</load-on-startup>

  </servlet>

  <servlet-mapping>

    <servlet-name>jolokia-agent</servlet-name>

    <url-pattern>/jolokia/*</url-pattern>

  </servlet-mapping>

</web-app>

Thereafter rebuild the app and then the metrics can be accessed over the browser using requests like these: 

Request:

http://spring-music-nonhesitant-cornetist.cfapps.pez.pivotal.io/jolokia/read/java.lang:type=Memory/HeapMemoryUsage

Response

{

• request: 
  

  {
  

  • mbean: "java.lang:type=Memory",
  • attribute: "HeapMemoryUsage",
  • type: "read"

  },
• value: 
  

  {
  

  • init: 805306368,
  • committed: 771751936,
  • max: 771751936,
  • used: 203904200

  },
• timestamp: 1435594883,
• status: 200

}

The JMX metrics can then be easily incorporated into a dashboard. In addition to Java there are other Jolokia clients available as well. The Javascript Jolokia client is full-featured and supports charting with Cubism, a D3.js plugin for plotting time series data.

Please note that some of the commercial app servers like Glassfish, WebSphere Liberty Profile, WebLogic provide JMX HTTP connectors.  The advantage of using these connectors over Jolokia is that , existing JMXClients like Jconsole and VisualVM plugins work as-is out of the box with JMX over HTTP. There is no need to do the querying for individual request/responses or write custom code. You can leverage the existing ecosystem of JMX client tooling.