About Me

My photo
Rohit is an investor, startup advisor and an Application Modernization Scale Specialist working at Google.

Thursday, May 5, 2016

Structuring orgs spaces in Cloud Foundry


with inputs from various Pivotal Experts (Joe Fitzgerald, Stuart Radnidge)... 


Let’s firstly look at orgs and spaces. An organization, or an org, is a set of users grouped together for management purposes, and all members of an org share the same resource quota plan, services availability, and their custom domain. In an nutshell, organizations allow you to create a multi-tenant environment within your pivotal cloud foundry deployment.
Organizations can be defined any way that you like. Typically, they’re defined around things like line of businesses, or particular projects, or a specific division, or an initiative. There’s no sort of right way to do it. There’s the way that suits your organization, is really the way to do it. You don’t want to overdo the orgs, but orgs give you a level of abstraction that enables you to define who can do what in a particular environment.
Associated with these orgs are quota plans, and a quota will apply to all the activities within their particular organization, and in particular they provide limits and tracking around memory and disk use. Memory that is used by application instances; the services the consume, so how many services you can allocate to applications within that environment; and the number of routes, or ‘roots’, depending on where you come from, to applications that are allowed to be configured. These are custom or multiple domains that can be configured so you can control limits on all of those.
Within each organization, you can have multiple spaces. Each org will have at least one space, you have to have one, and every application and service is scoped to a space. Spaces are a very important additional level of abstraction for organizing applications and services. A space gives you access to a shared location for application development, deployment, and maintenance, and users will have specific space-related roles.
What do we normally organize our spaces into? The kind of canonical version is really having a development space, a test space, a staging space, and a production space. That’s one example. It’s not the only example, but it shows you how you can move an application through different spaces based upon your development pipeline, and have different roles and responsibilities allocated to that space, and different things happening in those particular spaces.

In non-prod deployments, it makes sense to allow developers to experiment with Cloud Foundry in their own org. Heritage Pivotal Elastic Zones (PEZ), Pivotal’s Cloud Foundry offering for all Pivots, implements this model. In this scenario, a developer is given their own organization, and the developer is made the OrgManager of that org. A limited quota is assigned to this org, which that developer can view but can’t change. As a result, the developer can create as many spaces as they want to within this org, can bind service instances, etc, but can only consume a limited amount of resources with the application instances that they push to Cloud Foundry. This enables them to explore the platform, test various scenarios, etc without cluttering up other orgs and spaces.

So our recommendation in  a nutshell is – Create an org for a business unit or line of business and then one space per logical environment like dev, test and performance. Start simple and then customize as dictated by requirements.  Org= sensible business unit – a space per environment.

Alternate Approach: All the people with access to an org get read access to everything in that org, which means service credentials for every app in the org,  if you map an org to a business unit, and have multiple apps. To prevent this kind of credential and access leakage across apps in an org, another way to structure apps is one org per app and a space per environment. If the inventory and chargeback model is around apps then it is easier to map orgs to apps for that too.

References

Sunday, May 1, 2016

Create WebSphere Liberty Server Offline Buildpack

A number times I get queries on how to create an offline version of the WebSphere Liberty Profile Buildpack. Here are the high level steps with some color added, where the instructions are not clear for creating an Offline version of the liberty buildpack. 

The instructions below assume access to the internet. Once the buildpack is created then you can push applications to it. The compile process of the buildpack will find all the dependencies within the buildpack.

The steps below should be automated in a CI pipeline. For an example of a CI pipeline that generates the liberty offline buildpack head over to travis.

## Instructions Loosely Followed:
If you have customization to WebSphere  and JRE binaries.
- [Forking The Buildpack](https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack/blob/master/docs/forking.md)

If you have NO customization to WebSphere and JRE binaries.
- [How to install the Buildpack into your Cloud Foundry release](https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack/blob/master/docs/installation.md)

## Download WAS and JRE Binaries
https://developer.ibm.com/assets/wasdev/#asset/runtimes-wlp-runtime wlp runtime jar and others needed needed to setup repository see below ...

https://www-01.ibm.com/marketing/iwm/iwm/web/preLogin.do?source=swg-sdk8&S_PKG=amd64_8.0.3.0&S_TACT=105AGX05&S_CMP=JDK
- IBM 64-bit SDK for Linux on AMD64/EM64T architecture, Java Technology Edition
  Version  8.0.3.0
- Simple unzip with license (InstallAnywhere root not required)
  ibm-java-jre-8.0-3.0-x86_64-archive.bin  (111 MB)


~/Downloads/liberty8559❯
-rw-r--r--@   1 kelapr  720748206  147038277 Apr 30 00:06 ibm-java-x86_64-sdk-8.0-3.0.bin
-rw-r--r--@   1 kelapr  720748206   62753431 Apr 30 00:00 wlp-runtime-8.5.5.9.jar

## Structure the binaries into a repository and create the appropriate index.yml files 

https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack/blob/master/docs/util-repositories.md#setting-up-your-web-server

Copy license files from
https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/wlp/8.5.5.9/lafiles/runtime/en.html
https://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/wasdev/downloads/jre/index.yml

Push these binaries to the cloud using the static file buildpack
Dir. Structure before push...

~/Downloads/liberty8559❯ tree -f .
.
├── ./Staticfile
└── ./buildpack
    ├── ./buildpack/jre
    │   ├── ./buildpack/jre/ibm-java-jre-8.0-3.0-x86_64-archive-License.html
    │   ├── ./buildpack/jre/ibm-java-jre-8.0-3.0-x86_64-archive.bin
    │   └── ./buildpack/jre/index.yml
    └── ./buildpack/wlp
        ├── ./buildpack/wlp/index.yml
        ├── ./buildpack/wlp/wlp-developers-runtime-8.5.5.9-License.html
        ├── ./buildpack/wlp/wlp-javaee7-8.5.5.9.zip
        ├── ./buildpack/wlp/wlp-kernel-8.5.5.9.zip
        ├── ./buildpack/wlp/wlp-runtime-8.5.5.9.jar
        └── ./buildpack/wlp/wlp-webProfile7-8.5.5.9.zip



Modify the index.yml files after pushing for the first time and then repush

~/Downloads/liberty8559❯ cf push liberty-repo
Updating app liberty-repo in org pivot-rkelapure / space development as rkelapure@pivotal.io...
OK

See http://liberty-repo.cfapps.pez.pivotal.io/ for a functioning repository and index.yml files for a minimal repo of customized liberty and jre.

Another way of creating a repository is to run the dependency-builder; however configuring and running the dependency builder takes time and is error prone. Your best chance of success is to handcraft the repository based on your custom websphere binaries. If you have NO access to the internet when running the package rake task THEN you will need to setup a repository with not only WAS and JRE binaries but also all the third party binaries that are used like dynatrace, appdynamics etc.,

Modify the code in ibm-websphere-liberty-buildpack/config/ibmjdk.yml to point to the pushed JRE.
Modify the code in ibm-websphere-liberty-buildpack/config/liberty.yml to point to pushed Liberty.

Create the offline buildpack
https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack/blob/master/docs/installation.md
Create a licenses.yml based on the license files uploaded above

It is critical that the repository index.yml (specifically the wlp/index.yml)  are setup right otherwise you will see errors like these...

rake aborted!
undefined method `gsub' for nil:NilClass
/Users/kelapr/git/ibm-websphere-liberty-buildpack/resources/download_buildpack_cache.rb:155:in `filename'
/Users/kelapr/git/ibm-websphere-liberty-buildpack/resources/download_buildpack_cache.rb:64:in `block in download_cache'
/Users/kelapr/git/ibm-websphere-liberty-buildpack/resources/download_buildpack_cache.rb:47:in `each'
/Users/kelapr/git/ibm-websphere-liberty-buildpack/resources/download_buildpack_cache.rb:47:in `download_cache'
/Users/kelapr/git/ibm-websphere-liberty-buildpack/Rakefile:95:in `block (2 levels) in <top (required)>'
/Users/kelapr/git/ibm-websphere-liberty-buildpack/Rakefile:78:in `block in <top (required)>'
/Users/kelapr/.rvm/gems/ruby-1.9.3-p545/bin/ruby_executable_hooks:15:in `eval'
/Users/kelapr/.rvm/gems/ruby-1.9.3-p545/bin/ruby_executable_hooks:15:in `<main>'


## Generate the Buildpack

kelapr @ usxxkelaprm2 in ~/git/ibm-websphere-liberty-buildpack on git:master x [18:31:11]
$ bundle exec rake 'package[,*,]'

This will generate the offline IBM WebSphere Buildpack one dir. above
ibm-websphere-liberty-buildpack.zip

You can confirm the output location from the following line
/Users/kelapr/git/ibm-websphere-liberty-buildpack.zip as a buildpack zip output file

Once the buildpack is created. you will find all the cached dependencies inside

$ unzip -vl ibm-websphere-liberty-buildpack.zip

Archive:  ibm-websphere-liberty-buildpack.zip
 Length   Method    Size  Ratio   Date   Time   CRC-32    Name
--------  ------  ------- -----   ----   ----   ------    ----
       0  Stored        0   0%  05-01-16 18:26  00000000  admin_cache/
31318727  Defl:N 30838619   2%  05-01-16 18:24  7d5deee0  admin_cache/http%3A%2F%2Fdownloads.dynatracesaas.com%2F6.3%2Fdynatrace-agent-unix.jar.cached
      82  Defl:N       76   7%  05-01-16 18:24  edd887d4  admin_cache/http%3A%2F%2Fdownloads.dynatracesaas.com%2Fcloudfoundry%2Fbuildpack%2Fjava%2Findex.yml.cached
   71744  Defl:N    20247  72%  05-01-16 18:24  e514cbb9  admin_cache/http%3A%2F%2Fliberty-repo.cfapps.pez.pivotal.io%2Fbuildpack%2Fjre%2Fibm-java-jre-8.0-3.0-x86_64-archive-License.html.cached
115512196  Defl:N 114652626   1%  05-01-16 18:25  7b5cfed8  admin_cache/http%3A%2F%2Fliberty-repo.cfapps.pez.pivotal.io%2Fbuildpack%2Fjre%2Fibm-java-jre-8.0-3.0-x86_64-archive.bin.cached
     234  Defl:N      124  47%  05-01-16 18:24  f5c67578  admin_cache/http%3A%2F%2Fliberty-repo.cfapps.pez.pivotal.io%2Fbuildpack%2Fjre%2Findex.yml.cached
     596  Defl:N      171  71%  05-01-16 18:25  f94f0939  admin_cache/http%3A%2F%2Fliberty-repo.cfapps.pez.pivotal.io%2Fbuildpack%2Fwlp%2Findex.yml.cached
   60989  Defl:N    18625  70%  05-01-16 18:25  44573b38  admin_cache/http%3A%2F%2Fliberty-repo.cfapps.pez.pivotal.io%2Fbuildpack%2Fwlp%2Fwlp-developers-runtime-8.5.5.9-License.html.cached
67959662  Defl:N 67748071   0%  05-01-16 18:26  588faaec  admin_cache/http%3A%2F%2Fliberty-repo.cfapps.pez.pivotal.io%2Fbuildpack%2Fwlp%2Fwlp-webProfile7-8.5.5.9.zip.cached
    1487  Defl:N      183  88%  05-01-16 18:25  f74bed4e  admin_cache/https%3A%2F%2Fdl.zeroturnaround.com%2Fjrebel%2Findex.yml.cached
27633836  Defl:N 25152183   9%  05-01-16 18:25  04291789  admin_cache/https%3A%2F%2Fdl.zeroturnaround.com%2Fjrebel%2Freleases%2Fjrebel-6.4.3-nosetup.zip.cached
11656197  Defl:N 11640688   0%  05-01-16 18:24  a9417282  admin_cache/https%3A%2F%2Fdownload.run.pivotal.io%2Fapp-dynamics%2Fapp-dynamics-4.1.8_5.zip.cached

....

## Update PCF with Buildpack and push application with custom buildpack

# kelapr @ usxxkelaprm2 in ~/git [19:11:47]
$ cf create-buildpack  rliberty ./ibm-websphere-liberty-buildpack.zip 10 --enable

cf push spring-music -p build/libs/spring-music.war -b rliberty

Saturday, April 9, 2016

Debugging Node.js high CPU, crash and memory issues in Cloud Foundry - part 2

This blog post goes into step-by step run through of how to take a core dump of a specific application instance in cloud foundry. Please note you WILL need access to the PCF OpsManager. Typically the ability to target the BOSH director to your CF deployment is only possessed by sysadmins. So if you are reading this as a developer of Node.js apps in PCF, your best shot is to build the diagnostics into the app like so

  1. Use the restify observable REST framework
  2. Modify the node start command for your app like so node --abort_on_uncaught_exception --expose-gc
  3. Install heapdump with npm install heapdump and require it in server.js. Use chrome dev tools to analyze the heapdumps.
  4. Develop Debugging Techniques as explained here. 
Step 0: App Listing
cf app word-finder
     state     since                    cpu    memory         disk           details
#0   running   2016-04-09 02:13:26 AM   0.0%   107.4M of 1G   164.8M of 1G
#1   running   2016-04-09 10:31:47 PM   0.0%   96.2M of 1G    44.6M of 1G
#2   running   2016-04-09 10:31:47 PM   0.0%   95.4M of 1G    44.6M of 1G

Step 1: Determine app GUID
cf app word-finder --guid
5e46745d-9849-42c3-9549-be9ba5fe1e57

Step 2: Determine Host and Port of the app instance in question
Lets assume that it is instance #2 that is suffering the HIGH CPU.

export CF_TRACE=true
cf curl /v2/apps/5e46745d-9849-42c3-9549-be9ba5fe1e57/stats

  "2": {
      "state": "RUNNING",
      "stats": {
         "name": "word-finder",
         "uris": [
            "word-finder.cfapps.paas.pivotal.io"
         ],
         "host": "10.65.187.46",
         "port": 60540,
         "uptime": 560,
         "mem_quota": 1073741824,
         "disk_quota": 1073741824,
         "fds_quota": 16384,
         "usage": {
            "time": "2016-04-10T02:41:08.873469312Z",
            "cpu": 0.00019609362320671737,
            "mem": 103034880,
            "disk": 46796800
         }
      }
Step 3: bosh ssh into the Diego Cell VM 

ssh into the OpsManager VM and then target the BOSH director to the correct deployment

Target the BOSH director after looking up the Director IP address from the OpsManager console


bosh deployment /var/tempest/workspaces/default/deployments/cf-cdcc8064df9477b983f3.yml

Issue a bosh vms command to determine the name of the Diego cell that houses the app instance container. Note the host-name should match from step-1.

ubuntu@pivotal-ops-manager:~$ bosh vms cf-cdcc8064df9477b983f3 --details
Acting as user 'director' on deployment 'cf-cdcc8064df9477b983f3' on 'p-bosh-1667ec8b8a5e1edbe71b'


| Job/index| State   | Resource Pool | IPs | CID | Agent ID | Resurrection |
|diego_cell-partition-e8ca738002f1f150c602/0 | running | diego_cell-partition-e8ca738002f1f150c602 | 10.65.187.46 | vm-b2f2b447-043b-41ec-bad8-9fd950e9f9d7 | f1ec0fc6-9942-4c18-a55e-b507014684e3 | active       |

ubuntu@pivotal-ops-manager:~$ bosh ssh diego_cell-partition-e8ca738002f1f150c602/0

Step 4: Locate the Garden container that contains the app instance # 2.

First start in the right directory
cd var/vcap/data

Install Veritas
wget https://github.com/pivotal-cf-experimental/veritas/releases/download/latest/veritas
chmod a+x veritas

Configure Veritas
chmod a+x veritas

bosh_4auu9hoye@f1ec0fc6-9942-4c18-a55e-b507014684e3:/var/vcap/data$ ./veritas autodetect
export VITALS_ADDRS=garden:0.0.0.0:17013,rep:0.0.0.0:17008

root@9148d07d-9e64-4308-88ec-6ec90a04be37:/var/vcap/data# export GARDEN_ADDR=/var/vcap/data/garden/garden.sock

root@9148d07d-9e64-4308-88ec-6ec90a04be37:/var/vcap/data# export GARDEN_NETWORK=unix

root@9148d07d-9e64-4308-88ec-6ec90a04be37:/var/vcap/data# export VITALS_ADDRS=garden:0.0.0.0:17013,rep:0.0.0.0:17008

root@9148d07d-9e64-4308-88ec-6ec90a04be37:/var/vcap/data# ./veritas garden-containers

Derive a garden container mapping of the app instances. Now find the garden container based on the port exposed for that instance from step #2.

b60ba43b-d927-4d92-b3775e46745d-9849-42c3-9549-be9ba5fe1e57-773eaa66-667c-40b2-863e-51c3cd9570f8-bf91aa00-fabb-45a9-4311-1e326ad5998e - active @ /var/vcap/data/garden/depot/f8gqc8vnqfl
  Memory: 98.797 MB
  Disk: Total:802.640 MB 0 Inodes, Exclusive:44.629 MB 0 Inodes
  10.254.0.25=>5e46745d-9849-42c3-9549-be9ba5fe1e57-773eaa66-667c-40b2-863e-51c3cd9570f8-bf91aa00-fabb-45a9-4311-1e326ad5998e: 60542:8080,60543:2222
  Running: 2 processes
  Properties:
    executor:owner: executor

5e46745d-9849-42c3-9549-be9ba5fe1e57-773eaa66-667c-40b2-863e-51c3cd9570f8-fc296ea9-1e02-4f97-6c95-553388a3370c - active @ /var/vcap/data/garden/depot/f8gqc8vnqfm
  Memory: 98.297 MB
  Disk: Total:802.640 MB 0 Inodes, Exclusive:44.629 MB 0 Inodes
  10.254.0.29=>5e46745d-9849-42c3-9549-be9ba5fe1e57-773eaa66-667c-40b2-863e-51c3cd9570f8-fc296ea9-1e02-4f97-6c95-553388a3370c: 60540:8080,60541:2222
  Running: 2 processes
  Properties:
    executor:owner: executor

5e46745d-9849-42c3-9549-be9ba5fe1e57-773eaa66-667c-40b2-863e-51c3cd9570f8-8ce28a9f-a9ef-4aff-73a6-4e095822e0db - active @ /var/vcap/data/garden/depot/f8gqc8vnqbr
  Memory: 107.430 MB
  Disk: Total:922.765 MB 0 Inodes, Exclusive:164.754 MB 0 Inodes
  10.254.0.21=>5e46745d-9849-42c3-9549-be9ba5fe1e57-773eaa66-667c-40b2-863e-51c3cd9570f8-8ce28a9f-a9ef-4aff-73a6-4e095822e0db: 60538:8080,60539:2222
  Running: 2 processes
  Properties:
    executor:owner: executor

At this point you have nailed down the path to your specific garden container i.e. /var/vcap/data/garden/depot/f8gqc8vnqfm

Step 5: wsh into the container and issue the gcore command by attaching to the node.js process

Determine the PID of the node.js app process
root@f1ec0fc6-9942-4c18-a55e-b507014684e3:~# cd /var/vcap/data/garden/depot/f8gqc8vnqfm

garden/depot/f8gqc8vnqfm$ sudo su -

root@f1ec0fc6-9942-4c18-a55e-b507014684e3:/var/vcap/data/garden/depot/f8gqc8vnqfm# ./bin/wsh

# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 02:31 ?        00:00:00 initd -dropCapabilities=false -title="wshd: f8gqc8vnqfm"
vcap        14     1  0 02:31 ?        00:00:00 node server.js
vcap        17     1  0 02:31 ?        00:00:00 /tmp/lifecycle/diego-sshd -address=0.0.0.0:2222 -hostKey=-----BEGIN RSA PRIVATE KEY----- MIICXAIBAAKBgQCxtuDXcInp3mD2x9s8oI6c6Ca8wgEYvPoFwichBYOklyxJHMSy
root       330     1  0 03:07 pts/0    00:00:00 /bin/sh
root       332   330  0 03:08 pts/0    00:00:00 ps -ef

Attach a debugger to the process
# gdb --pid=14
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Attaching to process 14
Reading symbols from /home/vcap/app/.heroku/node/bin/node...done.
Reading symbols from /lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libssl.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 25]
[New LWP 24]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f8baa3c4859 in syscall () from /lib/x86_64-linux-gnu/libc.so.6


Generate a core dump
(gdb) gcore
warning: target file /proc/14/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 8192 bytes at 0x7fff6e7d9000.
Saved corefile core.14
(gdb) exit
Undefined command: "exit".  Try "help".
(gdb) quit
A debugging session is active.

Inferior 1 [process 14] will be detached.

Quit anyway? (y or n) y
Detaching from program: /home/vcap/app/.heroku/node/bin/node, process 14


# ls
core.14  firstboot.sh
# ls -al
total 114044
drwx------  2 root root      4096 Apr 10 03:08 .
drwxr-xr-x 31 root root      4096 Apr 10 03:08 ..
-rw-r--r--  1 root root      3106 Feb 20  2014 .bashrc
-rw-r--r--  1 root root       140 Feb 20  2014 .profile
-rw-r--r--  1 root root 116764456 Apr 10 03:08 core.14
-rwxr-xr-x  1 root root       213 Feb 22 19:32 firstboot.sh
#

Step 6 - Transfer core-dump to a VM that has the mdb debugger installed. 
After generating the coredump change ownership of the core to vcap:vcap  and move it to the /tmp dir. so that the core can scp'ed from the container to your Joyent or SunOS box

cf ssh-code | pbcopy scp -P 2222 -o User=cf:0b9216bc-5606-4c40-80cc-ae177a072435/0 ssh.run.haas-35.pez.pivotal.io:/tmp/core.16 . 

Step 7 Load the Core dump on the Solaris VM
Fetch a Solaris VM from the Joyent cloud. My specific machine is a SunOS 6de24ae5-f178-4d11-aa0f-c027c81e9f84 5.11 joyent_20151029T052920Z i86pc i386 i86pc Solaris

# mdb ./node-v4.2.2-linux/node-v4.2.2-linux-x64/bin/node ./core.14

> ::load ./mdb_v8_amd64.so
mdb_v8 version: 1.1.1 (release, from 28cedf2)
V8 version: 143.156.132.195
Autoconfigured V8 support from target
C++ symbol demangling enabled

Step 8 Issue mdb commands for introspecting the dump for stacktraces and data
::help <cmd>
::jsstack
::jsprint
::jssource
::jsconstructor
::findjsobjects
::jsfunctions

Step 9: Figure out the exact stack frame executing on the heap when the crash occurred or High CPU situation occurred.

Happy hunting ... 

Debugging Node.js high CPU, crash and memory issues in Cloud Foundry - part 1

These are the scenarios that keep a Devops engineer awake in the night - an instance of the node.js app is running at 100% CPU in Cloud Foundry with NO logging or resiliency protection like circuit breakers. This is the third occurrence of the problem in the last month. Management is dragging you into these long drawn day long meetings where each side blames each other - the app is restarted - the problem goes away and the cycle repeats. 

With apps increasingly being wrapped in russian doll layers of abstraction  [baremetal => VM => Container => Node process] getting to the root cause has become increasingly difficult.

So if you are a Java guy that has no experience in debugging a production node.js app. After drinking an appropriate amount of coffee and fruity alcoholic drinks you have landed on http://techblog.netflix.com/2015/12/debugging-nodejs-in-production.html

The recommendations from Yunong Xiao in the presentation essentially boils down to:
1. Using node-restify as a REST framework in the node app to log and observe app performance
2. Use Linux Perf Events to statistically sample stack traces  to construct CPU flame-graphs. This technique can reliably find the proverbial needle in the haystack.
3. Take Coredumps to dump the app state for post-mortem debugging with the mdb debugger. Unfortunately the mdb debugger can only run on Solaris- therefore you have to transfer the coredump to a debug solaris VM instance [1] [2]. 

Unfortunately both options 1 and 2 are not possible. 1 because the app-dev team will need to get involved and they don't look upon kindly on a devops guy advising a REST framework. 2 because CF does not bake perf events into the OS stemcell. Yikes!!! Looks like good old gcore is the only way to go  ... 

ok so with the introduction of Diego, finding your way into the container should be as easy as doing a cf-ssh and then issuing the gcore command rite ? - NOPE Wrong - Once ssh'ed into the container there is no way to elevate the privilege and attach gdb to your node.js process. You will need to find another way to elevate privilege and take a core dump.

This other way is using BOSH and Veritas. The gist of the method is that you bosh ssh into the correct Diego cell by first examining the app metadata. Thereafter on the Diego cell install veritas the Garden explorer tool. Use Veritas to map from the app instance to the container instance. Thereafter ./wsh into the garden container. You can now elevate your privilege with a sudo su - and then issue a  gdb, gcore of the node.js server process.

Once the dump has been generated sftp it to a debug solaris instance and run diagnostics with mdb commands like jsstack etc ... see this video from Joyent that explains how to walk the dump.

Part 2 of the blog goes into the gory detail of discovery and  generation of the core dump.

Friday, March 4, 2016

Supplying an external configuration to the Java Buildpack

I wanted to highlight a very important commit recently made to the Java Buildpack i.e.You can now configure the Java Buildpack with a custom server.xml via a repository.
Extension point for supplying an external Tomcat configuration
This allows a CF Platform Operator to provide a vetted server.xml that contains custom Tomcat configuration to the Java Buildpack without forking the buildpack. This feature is OFF by default. In order to enable it set the following env variable in the manifest:

Example in a manifest.yml
env: JBP_CONFIG_TOMCAT: "{ tomcat: { external_configuration_enabled: true }, external_configuration: { repository_root: \"http://repository...\" } }"

The artifacts that the repository provides must be in TAR format and must follow the Tomcat archive structure:
tomcat
|__conf
   |__context.xml
   |__server.xml
   |__web.xml
   |...
You can expect this feature to show up in the next release of the JavaBuildpack and its downstream clones like TomEE.

Thursday, March 3, 2016

Deigo BBS, ETCD and Consul - 3 is a crowd

PCF 1.6 has now been completely redesigned on the Diego runtime. In case you are wondering all this all works now checkout the picture below -



Diego internally uses a couple of registries to store internal state primarily Consul for NAT-less service registry and ETCD as an ephemeral data store for the Diego bulletin board service. Once there are corruption issues due to connectivity or disks, these distributed registries  start arguing. They cannot reconcile the state of the world with the CloudController due to quorum or issues with leader election or distributed key lockout. Basically the instances need to get the same key.  You can specify the key in the newer versions, but there is no enforcement of state in the case of corruption.

I am sure there is a proper technical reason why these distributed K,V stores behave they way they do in these situations; however my intent here is to teach you how to get out of this situation if somehow you end up here.

So in situations where Consul and Etcd and Diego Brain BBS jobs act like errant brothers and fight with another to reconstitute the state of the world from the Cloud Controller, leverage the following recipe to keep sanity in the house.

1. Restart all the VMs of the Consul, ETCD and Diego BBS nodes in that order.

2. Wipe out member DB files i.e. nuke the data dir. on the ETCD VMs. monit restart all services on the ETCD nodes.To delete the contents of the etcd data store see:
  1. Run bosh vms to identify the etcd nodes in your deployment.
  2. Run bosh ssh to open a secure shell into each etcd node.
  3. Run monit stop etcd on each etcd node.
  4. Delete or move the etcd storage directory, /var/vcap/store, on each etcd node.
  5. Run monit start etcd on each etcd node.
3. Reduce the number of Consul instances to 1 and  deploy the BOSH release again. If this fails, follow the same process for the other errant jobs. In the end you may end up with just one instance of these 3 jobs. Once you have a healthy deployment bump up the # of Job instances in the Resources page of the OpsManager.

4. Etcd may sometimes experience orphaned processes that leads to the Diego BBS job failing. This can be remedied by killall etcd on the Diego BBS VMs and subsequent redeployment.

5.  Nuke and Pave ... Take backup of your PCF deployment with cfops and do a brand new deployment followed by a restore.

State of JavaEE

There is a lot of chatter on the internet about Java EE being dead. As someone who spent a lot of time implementing some of the core specs like CDI and also helping customers adopt JavaEE I was interested in determining if this is true.

Unfortunately the anecdotal evidence points to JavaEE dying slowly. This is the evidence now points towards JavaEE being dead:

1. Many prominent Java EE evangelists - Folks who in some cases are friends and I admire deeply have left their respective Java EE shepherd companies. Reza Rahman from Oracle, Kenny Bastani from Oracle and Arun Gupta from Redhat to name just a few ...

2. Oracle, Redhat and IBM - the three most prominent supporters of JavaEE are now spending their money and developers innovating elsewhere. Oracle cloud,OpenShift and Bluemix respectively.

3. The advent of microservices has rendered the monolithic app-server dead. Gone are the days that you have the pack the kitchen sink in your app-server runtime. Nowadays developers ONLY want to use the runtime they want - case in point the rise of the micro-servers - Spring Boot, Dropwizard, Vert.x, Ratpack etc. The next evolution is the disappearance of these micro-servers and the move to serverless computing i.e. the developers are coding chained reactive lambda functions in the cloud. Not sure how I feel about that....

4. The pace of JavaEE specification implementation by vendors and the evolution of the specification itself has slowed to a glacial pace. Where is Java EE8 or Java EE9 ? Where is the Java EE equivalent of all the resiliency patterns like Circuit Breaker, Bulkhead and microservice patterns like service discovery, logging everything to stdout etc. CDI is the only major hope left for JavaEE. They need to rebase the entire platform on CDI and start innovating like crazy. Unfortunately this cannot be done by committee.

There is still a LOT of code running on application servers. WebSphere, WebLogic and JBOSS were the goto runtime and middleware OS in the 90s and early 2Ks.  I care about these apps and if you do too start decomposing them and running them on lightweight runtimes like Spring Boot, Jetty whatever blows your horn. Then move these services to the cloud using Docker or Buildpacks.  If the apps don't add any business value then move them to lighter runtimes within the JavaEE stack i.e. move to WebSphere Liberty or JBOSS Wildfly.

This should serve as a cautionary tale. I am sure a lot of people with disagree with what I have written. But hey this my opinion you should form one too ....

Au Revoir

Update: There is a new organization called Java EE Gaurdians https://dzone.com/articles/who-are-the-java-ee-guardians-and-why-you-should-c that seems to be JavaEEs best shot at survival.

Saturday, February 27, 2016

Backup and Restore Pivotal Cloud Foundry

Customers often ask us "What is the proper way to backup and restore Pivotal Cloud Foundry ?"

I want to go over all the options you have to successfully and expediently backup and restore Pivotal Cloud Foundry. The advice here is generally applicable to open source cloud foundry as well to a certain extent.

First a bit of a background. Pivotal Cloud Foundry is comprised of a number of tiles which roughly map to a BOSH release in open source. Each of these tiles need to backed up and restored individually. The OpsManager Tile is special in that it provides the inception VM and is responsible for deploying the BOSH Director. 

Manual Backup and Restore:
This is the recommendation from the product team itself. See 
Backup => https://docs.pivotal.io/pivotalcf/customizing/backup-restore/backup-pcf.html and Restore => https://docs.pivotal.io/pivotalcf/customizing/backup-restore/restore-pcf.html.  So what exactly happens during a backup and restore is explained by the pictures below.

Backup
Restore
Automated Backup and Restore
Manually backing up a foundation every night is not practical. Therefore the Pivotal Cloud Foundry Solutions team has an open source tool that automates the manual process for taking a backup and restoring it. This tool is called cfops => https://github.com/pivotalservices/cfops
Documentation on how to run backup/restore with cfops* can be found Here http://www.cfops.io/docs/  The basic process here is  Backup all Tiles and place the backup on a reliable file system. During a restore, restore each tile, apply changes and rinse and repeat for all tiles. 

cfops as of today does not support backing up or restoring data service tiles like MySQL, RabbitMQ etc., The tile settings are restored; however the data is not imported. So you will need an independent backup/restore strategy for data stores. This is where a data backup a tool like Shield => https://github.com/starkandwayne/shield from Stark and Wayne looks attractive. The project is in its infancy. Another option for backing up  the data services is leveraging Concourse.

Concourse can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off tasks, either locally or in another CI system. Concourse encourages practices that keep your project loosely coupled to the details of your continuous integration infrastructure.

If for some reason your BOSH Director disappears or the Director VM gets nuked you can bring it back using the procedure described here : https://blog.pivotal.io/pivotal-cloud-foundry/features/restoring-pivotal-cloud-foundry-after-disaster

Please note that the Backup and Restore process can be done with zero downtime if done right particularly because of the canary support provided by BOSH.

Canary instances are first instances updated within an instance group. Any update error in a canary instance causes the deployment to stop. Since only canaries are affected before an update stops, problem jobs and packages are prevented from taking over all instances.



At some point Pivotal Cloud Foundry will provide an official tool and automation strategy for backup & restore. Till then we can leverage cfops backup and restore function- automated with cron or in a CI pipeline with (Wercker/Concourse/Jenkins) to ensure that our sysadmins can sleep safe and sound.






* Please note cfops is not officially supported by the Pivotal Product Team; however we as Pivotal make our best effort to support this tool and keep it functioning across product upgrades.

Wednesday, February 3, 2016

13 Factor apps for .NET - Cloud Foundry on Azure


Why bother with Cloud Foundry on Azure
Quick Answer: To optimize for the cloud programming model and build apps that can scale agnostically across cloud platforms. Microsoft haas published 13 recommended patterns from MS for cloud development in BuildingCloudAppsMsAzure. Cloud Foundry through it platform contracts, partner tiles and programming model constraints enables the best way to develop, design and code cloud apps on Azure.

• Automate everything
   • Use scripts to maximize efficiency and minimize errors in repetitive processes. aka Use Cloud Foundry

• Source control
   • Set up branching structures in source control to facilitate a DevOps workflow.
   • Example: add scripts to source control.
   • Example: keep sensitive data out of source control.
   • Example: use Git in Visual Studio.

• Continuous integration and delivery
   • Automate build and deployment with each source control check-in.

• Web development best practices
   • Keep web tier stateless
   • Example: scaling and autoscaling in Cloud Foundry.
   • Avoid session state.
   • Use a Content Delivery Network (CDN).
   • Use an asynchronous programming model.
   • Example: async in ASP.NET MVC and Entity Framework.

• Single sign-on
   • Introduction to Azure Active Directory.
   • Example: create an ASP.NET app that uses Azure Active Directory.

• Data storage options
   • Types of data stores.
   • How to choose the right data store.
   • Example: Azure SQL Database.

• Data partitioning strategies
   • Partition data vertically, horizontally, or both to facilitate scaling a relational database.

• Unstructured blob storage
   • Store files in the cloud by using the Blob service.
   • Example: using blob storage

• Design to survive failures
   • Types of failures.
   • Failure scope.
   • Understanding SLAs.

• Monitoring and telemetry
   • Why you should both buy a telemetry app and write your own code to instrument your app.
   • Example: New Relic for Azure
   • Example: logging code in the Fix It app.
   • Example: built-in logging support in Cloud Foundry.
   • Example: built-in Telemetry for websites and systems in Cloud Foundry

• Transient fault handling
   • Use smart retry/back-off logic to mitigate the effect of transient failures.
   • Example retry/back-off in Entity Framework 6.

• Distributed caching
   • Improve scalability and reduce database transaction costs by using distributed caching.
   • Example: Using Gemfire in ASP.NET app

• Queue-centric work pattern
   • Enable high availability and improve scalability by loosely coupling web and worker tiers.
   • Example: Using RabbitMQ 

Write sample Java apps using the User Authentication Authorization (UAA) API on Cloud Foundry


Do you want to use UAA as an OAuth2 authorization server (eg SSO, token creation / verification)? 

Take a look at github.com/pivo...ity-sample-apps for basic examples of integrating with Pivotal SSO which is UAA with subdomain multi-tenancy, and comes free with PCF 1.5, or look at github.com/will...ervice-security and spring.io/blog/...-cloud-security for a more involved example of using straight UAA to secure microservices.

For PCF customers and interested in using Pivotal SSO and Spring Cloud Services to manage a secure microservice architecture, github.com/will...ran/freddys-bbq is the the microservice-security example and goes the extra mile and integrates with all those products.

Do you want use UAA APIs to dynamically create OAuth2 clients and manage user roles?

There aren't many examples in the OSS world for that, but we do some of that in both Pivotal SSO and Spring Cloud Services. There are API docs however: github.com/clou...cs/UAA-APIs.rst

THANK YOU Will Tran