Saturday, October 10, 2015

Boiler plate apps for all buildpacks in Cloud Foundry

If you are tired of searching for sample apps of different types that exercise all the buildpacks in Cloud Foundry, the list of boilerplate apps below comes to your rescue. Most of these apps need a backing service like MySQL which exercises the service binding code in CF and your app.

List of all the Buildpacks supported by CF: http://docs.cloudfoundry.org/buildpacks/

java_buildpack: Spring-Music : Allows binding based on profiles to mysql, postgres, in-memory, etc., https://github.com/pivotalservices/spring-music

go_buildpack: pong_matcher_go: This is an app to match ping-pong players with each other. It's currently an API only, so you have to use curl to interact with it. Requires mysql
https://github.com/cloudfoundry-samples/pong_matcher_go

nodejs_buildpack: node-tutorial-for-frontend-devs: Node.js sample app with mongodb backend:
https://github.com/cwbuecheler/node-tutorial-for-frontend-devs

php_buildpack: PHPMyAdmin : out-of-the-box implementation of PHPMyAdmin 4.2.2. Requires mysql
https://github.com/dmikusa-pivotal/cf-ex-phpmyadmin

binary_buildpack: pezdispenser: Admin portal for Cloud Foundry
https://github.com/pivotal-pez/pezdispenser

ruby_buildpack: Rails app to match ping-pong players with each other. Requires mysql.

https://github.com/cloudfoundry-samples/pong_matcher_rails

python_buildpack: Buildpack uses pip to install dependencies. Needs a requirement.txt

Flask app: https://github.com/michaljemala/hello-python

PyData app: https://gist.github.com/ihuston/d6aab5e4a811fe582fa7 Does not use pip. Uses conda.

staticfile_buildpack: Put a Staticfile in any directory and do a cf push. If directory browsing is needed Add a line to your Staticfile that begins with directory: visible

Spring Cloud services Tile:
- https://github.com/spring-cloud-samples/fortune-teller
- https://github.com/dpinto-pivotal/cf-SpringBootTrader

.NET sample app: Contoso University

- https://github.com/pivotalservices/contoso-university

Method to get to the root cause of a memory leak in the Cloud

From an application developer perspective our apps are increasingly run in russian doll deployments i.e. app code wrapped in a app server container which is wrapped by a LXC container running within a VM , running on one or multiple hypervisors running on bare metal. In such a scenario determining the root cause of memory leaks becomes difficult. Please find below a process that could be used to get to the eureka moment.

The basic principle to get to root cause is that of eliminating all the variables one by one. We start at the top of the stack and work our way down. Remember the JVM is like an iceberg. There is java heap that is above the water and an unbound native memory portion underneath the surface. Java heap OutOfMemory errors are easier to fix than native memory leaks. Native leaks are generally caused by errant libraries/frameworks, JDKs, app-servers or some unexplained OS-container-JDK interaction.

ok lets get to it ...

First establish a process to measure the JVM heap and native process size perhaps using a dump script like https://github.com/dmikusa-pivotal/cf-debug-tools#use-profiled-to-dump-the-jvm-native-memory. Remember to take heapdumps before during and after the load run. Once the test is close to completion take native process core dumps using kill -6 or kill -11. This procedure of running the test is then repeated as you eliminate each variable below.

1. [app] First look at the application source for the usual memory leak anti-patterns like DirectByteBuffers, threadlocal, Statics, Classloader retention, resource cleanup etc. This is where you will get the maximum bang for the buck. Take a heapdump of the JVM process and analyze in EclipseMemoryAnalyzer of HeapAnalyzer.

2. [jdk] Eliminate the JDK as a factor of the leak by switching JVM implementations i.e. moving from OpenJDK to Hotspot or from OpenJDK to IBM JDK etc .. see the entire list of JVM impls https://en.wikipedia.org/wiki/List_of_Java_virtual_machines.

3. [app-server] If simple eyeballing does not help then switch the app-server i.e. move from tomcat to jetty, undertow to tomcat. If your app runs on WebSphere or WebLogic and cannot be ported then my apologies. Call 1-800-IBM-Support.

4. [container] If your droplet (app + libraries/frameworks + jvm) is running within a container in Cloud Foundry or Docker then try switching out the containers. i.e. If running within the warden container then run the same app within Docker container. Try changing Docker base images and see if the leak goes away.

5. [hypervisor] If running on AWS switch to OpenStack or vSphere and vice versa. You get the idea. Cloud Foundry makes this easy since you can standup the same CF deployment on all three providers.

6. [bare-metal] Run the app on the bare metal server to check if the leak persists.

7. [sweep-under-the-rug] Once you are ready to pull your hair out, resort to tuning the JDK. Start playing with JVM options like -Xms234M -Xmx234M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=128M -Xss228K. In cloud foundry this is set by the memory_calculator that is influenced by setting the memory_heuristics and memory_sizes env vars.

JBP_CONFIG_OPEN_JDK_JRE: '[memory_heuristics: {heap: 55, metaspace: 30}, memory_sizes: {metaspace: 4096m..}]'
JBP_CONFIG_OPEN_JDK_JRE: '[memory_calculator: {memory_sizes: {stack: 228k, heap: 225M}}]
JBP_CONFIG_OPEN_JDK_JRE: '[memory_calculator: {memory_heuristics: {stack: .01, heap: 10, metaspace: 2}}]'

As you can see, the options become increasingly cumbersome as you keep doing down this list. Your best hope is to catch it at 1, 2 or 3. Good Luck Hunting!

Thursday, October 1, 2015

Chasing Cloud Foundry OutOfMemory Errors - OOM

If you are ever unfortunate enough to troubleshoot application, java heap or native process OOM issues in Cloud Foundry follow the playbook below to get to the root cause:

1. Include the attached script dump.sh at the root of your JAR/WAR file. You can edit the LOOP_WAIT variable in the script to configure how often it will dump the Java NMT info. I'd suggest somewhere between 5 and 30 seconds, depending on how long it takes for the problem to occur. If the problem happens pretty quick, go with a lower number. If it takes hours, then go with something higher.

2. Make a .profile.d directory, also in the root of the JAR / WAR file. For a detailed explanation on using .profile.d to profile native memory checkout this note from CF support engineer Daniel Mikusa.

3. In that directory, add this script.
#!/bin/bash
$HOME/dump.sh &

This script will be run before the application starts. It starts the dump.sh script and backgrounds it. The dump.sh script will loop and poll the Java NMT stats, dumping them to STDOUT. As an example see the simple-java-web-for-test application. There is also an accompanying load plan here.

4. Add the following parameters to JAVA_OPTS:
JAVA_OPTS: "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:./jvm-gc.log -XX:NativeMemoryTracking=detail"
The -XX:NativeMemoryTracking will enable native OOM analysis [1]. More on this later. The -Xloggc will allow you to pipe all GC output to a log that you can later analyze with tool like PMAT or GCMV

5. Push or restage the application.

6. In a terminal open cf logs app-name > app-name.log. That will dump the app logs & the Java NMT info to the file. Please try to turn off as much application logging as possible as this will make it easier to pick out the Java NMT dumps.

7. Kick off the load tests.

The nice thing about Java NMT is that it will take a snapshot of the memory usage when it first runs and every other time we poll the stats we'll see a diff of the initial memory usage. This is helpful as we really only need the last Java NMT dump prior to the crash to know what parts of the memory have increased and by how much. It's also nice because it gives us insight into non-heap usage. Given the Java NMT info, it should be easier to make some suggestions for tuning the JVM so that it doesn't exceed the memory limit of the application and cause a crash.

8. If you have the ability to ssh into the container use the following commands to trigger heapdumps and coredumps
JVM Process Id:
PID=` ps -ef | grep java | grep -v "bash\|grep" | awk '{print $2}'`
Heapdump:
./jmap -dump:format=b,file=/home/vcap/app/test.hprof $PID

Coredump:

kill -6 $PID should produce a core file and leave the server running

Analyze these dumps using tools like Eclipse Memory Analyzer and IBM HeapDump Analyzer

9. If you have the ability to modify the application then enable the Spring Boot Actuator feature for your app if it is Boot app else integrate a servlet or script like DumpServlet and HeapDumpServlet into the app

Salient Notes:

- The memory statistics reported by CF in cf app is in fact used_memory_in_bytes which is a summation of rss and the active and inactive caches. [2] and [3]. This is the number watched by the cgroup linux OOM killer.
- The Cloud Foundry Java Buildpack by default enables the following parameter -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
Please do NOT be lulled into a false sense of complacency by this parameter. I have never seen this work in production. Your best bet is to be proactive when triggering and pulling dumps using servlets, JMX, kill commands, whatever ...

References:

[1] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

[2] https://github.com/cloudfoundry/dea_ng/commit/e5fd523aaf2cddeda01d4134ecd2a929829c4147

[3] https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/6M8BDV_tq7w/VCoJmWtJJncJ
[4] https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
[5] http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
[6] Native Memory Best Practices
[7] http://www-01.ibm.com/support/docview.wss?uid=swg21255223
[8] https://publib.boulder.ibm.com/httpserv/cookbook/cookbook.html

Tuesday, September 29, 2015

Spring Boot Activator metrics collection in a spreadsheet

If your app is a spring boot app that has the actuator enabled then use this nifty script from Greg Turnquist's Learning Spring Boot book with some changes from me to collect all the metrics in a csv

package learningspringboot

@Grab("groovy-all")
import groovy.json.*

package learningspringboot

@Grab("groovy-all")
import groovy.json.*

@EnableScheduling
class MetricsCollector {

def url = "http://fizzbuzz.cfapps.io/metrics"
def slurper = new JsonSlurper()
def keys = slurper.parse(new URL(url)).keySet()
def header = false;
@Scheduled(fixedRate = 1000L)
void run() {
if (!header) {
println(keys.join(','))
header = true
}

def metrics = slurper.parse(new URL(url))

println(keys.collect{metrics[it]}.join(','))
}
}

Debugging DEA Issues on Cloud Foundry - Halloween Edition

Ever wondered why Cloud Foundry does not support ssh'ing into the warden container. Such a feature could be useful in so many situations - TechOps, Troubleshooting, Debugging etc. cf-ssh is coming to Cloud Foundry via Diego; however till you deploy Diego in production you will need to live with the process below to ssh into a particular application instance warden container.

# Step 1 - Find app guid

C:\Users\Administrator\workspace\FizzBuzz>cf app --guid fizzbuzz
90291bd7-ce52-43ee-aaa1-ed0405863c4a

# Step 2 - Find DEA IP address and port

C:\Users\Administrator\workspace\FizzBuzz>cf curl /v2/apps/90291bd7-ce52-43ee-aaa1-ed0405863c4a/stats
{
"0": {
"state": "RUNNING",
"stats": {
"name": "fizzbuzz",
"uris": [
"fizzbuzz.kelapure.cloud.pivotal.io"
],
"host": "192.168.200.27",
"port": 61015,
"uptime": 1489673,
"mem_quota": 1073741824,
"disk_quota": 1073741824,
"fds_quota": 16384,
"usage": {
"time": "2015-09-29 02:36:12 +0000",
"cpu": 0.003104336638753874,
"mem": 546877440,
"disk": 187445248
}
}
}
}

# Step 3 - Locate DEA Job that maps to the DEA IP from previous step

# Step 4 Login into the DEA VM

ubuntu@pivotal-ops-manager:~$ bosh ssh dea-partition-0d7a243620d08147fd3a/0 --public_key y.pub
Acting as user 'director' on deployment 'cf-938e3d9bec67dbffeacc' on 'microbosh-38a3a7433db69fa7d159'
Enter password (use it to sudo on remote host): ********
Target deployment is `cf-938e3d9bec67dbffeacc'

see https://github.com/cloudfoundry/bosh-lite/issues/134 if stuck on bosh ssh

# Step 5 Locate the warden container path

If you grep for `fizzbuzz`and the instance port gleaned from step-2 in the list below you will find the following

"warden_container_path": "/var/vcap/data/warden/depot/18tlhc59f3v",

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot$ sudo cat /var/vcap/data/dea_next/db/instances.json

[sudo] password for bosh_yms06qjkj:
{
"time": 1443494275.6732097,
"instances": [

{
"cc_partition": "default",
"instance_id": "07a4a39d7ec14249863303246c73dfa2",
"instance_index": 0,
"private_instance_id": "5e15175c18644561828e52534b7c7a71fad5773f13b045058eaed76793618b1f",
"warden_handle": "18tlhc59erj",
"limits": {
"mem": 512,
"disk": 1024,
"fds": 16384
},
"health_check_timeout": null,
"environment": {
"CF_PROCESS_TYPE": "web"
},
"services": [

],
"application_id": "093a9e11-0b06-45f1-b3c4-e801ad0aca81",
"application_version": "2febead7-c0fe-43a5-b4a4-724c5581c037",
"application_name": "tmfnodetest",
"application_uris": [
"tmfnodetest.kelapure.cloud.pivotal.io"
],
"droplet_sha1": "fcba91c3099ba1218febfcb043d3792ccc9bba97",
"droplet_uri": null,
"start_command": null,
"state": "RUNNING",
"warden_job_id": 461,
"warden_container_path": "/var/vcap/data/warden/depot/18tlhc59erj",
"warden_host_ip": "10.254.2.41",
"warden_container_ip": "10.254.2.42",
"instance_host_port": 61127,
"instance_container_port": 61127,
"syslog_drain_urls": [

],
"state_starting_timestamp": 1442435859.1693246
},

{
"cc_partition": "default",
"instance_id": "9383f43f0e3549ad9c29c511c5f4211e",
"instance_index": 0,
"private_instance_id": "23065b9faddb43d2ae296802c1db8cbdbf2a2d00dd2343849c8825bcfc7eb044",
"warden_handle": "18tlhc59enu",
"limits": {
"mem": 1024,
"disk": 1024,
"fds": 16384
},
"health_check_timeout": null,
"environment": {
"CF_PROCESS_TYPE": "web"
},
"services": [
{
"credentials": {
"agent-name": "nginx-hello",
"host-name": "ca-apm.springapps.io",
"port": "5001"
},
"options": {

},
"syslog_drain_url": "",
"label": "user-provided",
"name": "ca_apm_10",
"tags": [

]
}
],
"application_id": "90291bd7-ce52-43ee-aaa1-ed0405863c4a",
"application_version": "09f093ae-c9e8-46ba-89c8-e56d7b84b671",
"application_name": "fizzbuzz",
"application_uris": [
"fizzbuzz.kelapure.cloud.pivotal.io"
],
"droplet_sha1": "e5fc94cc79eb72489e94c6b620887c5b72244b89",
"droplet_uri": "http://staging_upload_user:c601ae7f5ae3745d40ee@192.168.200.18:9022/staging/droplets/90291bd7-ce52-43ee-aaa1-ed0405863c4a/download",
"start_command": null,
"state": "RUNNING",
"warden_job_id": 92,
"warden_container_path": "/var/vcap/data/warden/depot/18tlhc59enu",
"warden_host_ip": "10.254.0.85",
"warden_container_ip": "10.254.0.86",
"instance_host_port": 61015,
"instance_container_port": 61015,
"syslog_drain_urls": [
""
],
"state_starting_timestamp": 1442004499.0728624

},

{
"cc_partition": "default",
"instance_id": "fa057152a7094ee6bfd0cd28c8cb76dc",
"instance_index": 0,
"private_instance_id": "18b2515c046742ae979fbfa58428120608960bc04f7444759983d1c91621e7da",
"warden_handle": "18tlhc59f3v",
"limits": {
"mem": 1024,
"disk": 1024,
"fds": 16384
},
"health_check_timeout": null,
"environment": {
"CF_PROCESS_TYPE": "web"
},
"services": [

],
"application_id": "d9969088-1f7b-40b3-a048-c71814d172c4",
"application_version": "431e7f3d-774b-4c24-ad4e-c6fda884dab1",
"application_name": "spring-music",
"application_uris": [
"spring-music.kelapure.cloud.pivotal.io"
],
"droplet_sha1": "9e35ddfd97a7c1a32e2a69d5e1c5f90c6d4b7e06",
"droplet_uri": "http://staging_upload_user:c601ae7f5ae3745d40ee@192.168.200.18:9022/staging/droplets/d9969088-1f7b-40b3-a048-c71814d172c4/download",
"start_command": null,
"state": "CRASHED",
"warden_job_id": 1265,
"warden_container_path": "/var/vcap/data/warden/depot/18tlhc59f3v",
"warden_host_ip": "10.254.2.113",
"warden_container_ip": "10.254.2.114",
"instance_host_port": 61395,
"instance_container_port": 61395,
"syslog_drain_urls": [

],
"state_starting_timestamp": 1443494262.4416354
}
],
"staging_tasks": [

]
}

# Step 6 wsh into the warden container

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot$ cd 18tlhc59enu

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot/18tlhc59enu$ ls
bin destroy.sh etc jobs lib mnt net_rate.sh net.sh run setup.sh snapshot.json start.sh stop.sh tmp

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot/18tlhc59enu$ sudo ./bin/wsh
wsh wshd

bosh_yms06qjkj@437bbc76-7d1c-4ce9-916d-4e6a6355537f:/var/vcap/data/warden/depot/18tlhc59enu$ sudo ./bin/wsh
root@18tlhc59enu:~# ls
firstboot.sh
root@18tlhc59enu:~# cd /home
root@18tlhc59enu:/home# cd vcap/
root@18tlhc59enu:/home/vcap# ls
app logs run.pid staging_info.yml tmp

root@18tlhc59enu:/home/vcap# ll
total 40
drwxr-xr-x 5 vcap vcap 4096 Sep 11 20:48 ./
drwxr-xr-x 3 root root 4096 Sep 11 20:48 ../
drwxr--r-- 7 vcap vcap 4096 Sep 11 20:47 app/
-rw-r--r-- 1 vcap vcap 220 Apr 9 2014 .bash_logout
-rw-r--r-- 1 vcap vcap 3637 Apr 9 2014 .bashrc
drwxr-xr-x 2 vcap vcap 4096 Sep 11 20:47 logs/
-rw-r--r-- 1 vcap vcap 675 Apr 9 2014 .profile
-rw------- 1 vcap vcap 3 Sep 11 20:48 run.pid
-rw-r--r-- 1 vcap vcap 2000 Sep 11 20:47 staging_info.yml
drwxr-xr-x 3 vcap vcap 4096 Sep 11 20:48 tmp/
root@18tlhc59enu:/home/vcap#

# Step 7 - Now you are free to ftp files out or take heap dumps or thread-dumps

For instance to take a heapdump
root@18tlhc59enu:/home/vcap/app/.java-buildpack/open_jdk_jre/bin# ll
total 384
drwxr-xr-x 2 vcap vcap 4096 Sep 11 20:47 ./
drwxr-xr-x 5 vcap vcap 4096 Sep 11 20:47 ../
-rwxr-xr-x 1 vcap vcap 8798 Jul 16 09:29 java*
-rwxr-xr-x 1 vcap vcap 8909 Jul 16 09:29 jcmd*
-rwxr-xr-x 1 vcap vcap 8909 Jul 16 09:29 jjs*
-rwxr-xr-x 1 vcap vcap 8973 Jul 16 09:29 jmap*
-rwxr-xr-x 1 vcap vcap 8981 Jul 16 09:29 jstack*
-rwxr-xr-x 1 vcap vcap 8917 Jul 16 09:29 keytool*
-rwxr-xr-x 1 vcap vcap 1146 Sep 11 20:47 killjava.sh*
-rwxr-xr-x 1 vcap vcap 8981 Jul 16 09:29 orbd*
-rwxr-xr-x 1 vcap vcap 8917 Jul 16 09:29 pack200*
-rwxr-xr-x 1 vcap vcap 8917 Jul 16 09:29 policytool*
-rwxr-xr-x 1 vcap vcap 8909 Jul 16 09:29 rmid*
-rwxr-xr-x 1 vcap vcap 8917 Jul 16 09:29 rmiregistry*
-rwxr-xr-x 1 vcap vcap 8917 Jul 16 09:29 servertool*
-rwxr-xr-x 1 vcap vcap 8989 Jul 16 09:29 tnameserv*
-rwxr-xr-x 1 vcap vcap 217462 Jul 16 09:29 unpack200*

root@18tlhc59enu:/home/vcap/app/.java-buildpack/open_jdk_jre/bin# su vcap

vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ PID=` ps -ef | grep java | grep -v "bash\|grep" | awk '{print $2}'`
vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ echo $PID
29

vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ ./jmap -dump:format=b,file=/home/vcap/app/test.hprof $PID
Dumping heap to /home/vcap/app/test.hprof ...
Heap dump file created

vcap@18tlhc59enu:~/app/.java-buildpack/open_jdk_jre/bin$ ls -al /home/vcap/app
total 230616
drwxr--r-- 7 vcap vcap 4096 Sep 29 03:06 .
drwxr-xr-x 5 vcap vcap 4096 Sep 11 20:48 ..
drwxr-xr-x 5 vcap vcap 4096 Sep 11 20:47 .java-buildpack
-rw-r--r-- 1 vcap vcap 82155 Sep 11 20:47 .java-buildpack.log
drwxr--r-- 3 vcap vcap 4096 Sep 11 20:46 META-INF
drwxr--r-- 3 vcap vcap 4096 Sep 11 20:46 my-resources
drwxr--r-- 3 vcap vcap 4096 Sep 11 20:46 org
-rw------- 1 vcap vcap 236032777 Sep 29 03:06 test.hprof
drwxr--r-- 5 vcap vcap 4096 Sep 11 20:46 WEB-INF

Sunday, September 20, 2015

Migrating The Monolith

Wednesday, August 19, 2015

Porting strategies for migrating apps to Cloud Foundry

Migration Recipes

You should consider the following options when lifting and shifting apps to Cloud Foundry:

1. Deploy app as a VM/Job with BOSH + persistent disk
2. Dockerize with patterns and Recipes
3. Lift and Shift JavaEE Full profile apps to TomEE BP, JBOSS BP, Liberty Profile BP, WebLogic BP
4. Lift and Shift JavaEE Web profile apps to Tomcat Java BP.
5. Refactor heavily used portions of app with decomposition patterns.
6. Bootify: Create a self-contained fat jar, including the app server runtime into the app.
7. Covert app to a managed service by creating a app-service broker that provisions and manages the app. Expose the app as a service bindable to other apps.
8. Employ VM Templating technologies like TOSCA, vRealize or vCaC brokers to deploy app VMs.
9. Rewrite app following 12 factor, modern cloud native app architectures

Apps cannot live without data. In addition to the app-code the persistence tier may also need to be refactored to accommodate the migrated app.

Key Features in Cloud Foundry that Enable App Migration

1. Deigo
2. CF v3 app-service
3. TCP Routing
4. Persistence Support for Deigo
5. Context Path Routing
6. Routing service

Remediations when moving apps to a PaaS

Anti-Cloud Pattern	Remedy
Multiple Inbound protocols like RMI, JMX, Custom-TCP	TCP Routing, Tunneling
Persistence in local VMs like Caches, Transaction logs, CMS	External managed data services, Persistent volume support, sshfs service, EBS & S3 blob store, mount external NFSTransations
JTA & 2-pc commit	Use standalone transaction managers like Atomikos and Bitronix. Introduce eventual consistency patterns
Spaghetti Configuration	Externalize, plugin config via Config Server , init script in .profile.d directory that injects the config. as environment variables
Logging and Monitoring	Punch holes in outbound security groups, CF v3 API, Different process types

Wednesday, July 15, 2015

Migrating WebSphere Application Server Applications to Cloud Foundry

Migration of WebSphere Application Server apps to Cloud Foundry entails dissecting the app for API usage and then configuring the implementation of the same APIs on a servlet container or an alternate open source EE server. My app server of choice for Java EE APIs is Tomcat + Spring Framework or Apache TomEE.

Migrations of Java EE applications vary in complexity, depending on a series of factors that are identified during the migration:

Source and target Java SE specification (JRE 1.5 to JRE 1.8)
Source and target Java EE specifications (J2EE 1.4 to Java EE 7)
Usage of specific Java components (EJB, JSPs, JSF, etc.,)
Usage of proprietary code that is specific for an application server
Vendor-specific deployment descriptors
Extended capabilities
Vendor-specific server configurations (JMS providers and JDBC connection pooling)
Class loading issues .
Integration with complex external systems such as a CRM or SAP

From a code perspective an application owner has to ask questions related to Java, EJB usage, Servlets & JSPs, Web Services, database access, JMS, JNDI naming, Application trace & logging, UI frameworks, Transactions, Threads, sockets and XML/REST/JSON. The configuration and usage of these technologies will affect the migration of the app to Cloud Foundry. From a deployment perspective the following areas need to be probed: Current hardware, current network edge load balancing solution and expectation around session affinity, expected downtime during upgrade, current administration constructs, Details of the HTTP Server/Proxy Tier. hardware details & security (How exactly is the security of the app and app-server managed). From an architectural perspective to understand the technical barriers and issues in porting apps to cloud native CF platform please read Roadblocks to PaaS.

Process

Get a lay of the land. Identify ALL the open source and proprietary APIs used by the app. Determine what modifications you will need to make to the app if the target runtime packaged by the buildpack does not include the said libraries.

APIs
- Make a list of all WebSphere Proprietary APIs used by your app,
- Make a list of all the Java EE APIs used by your app,
- Make a list of all Spring Framework APIs used the app,
- Make a list of all 3rd party open source frameworks used by the app,
Analysis Tools
- Leverage static analysis tools like FindBugs, Checkstyle, PMD, Coverity and SonarQube. Write custom rules that will provide a breakdown of the external api usage. From the The Wise Developers' Guide to Static Code Analysis - PMD is a very customizable tool, so the performance and installation outcome really depends on how one has instructed the tool to find problems. Coverity is a commercial tool that it does a much deeper analysis than other tools and it also keeps track of the issues that you’ve found and fixed. SonarQube gives the best combination of usefulness and transparency against the relative complexity of setting it up and solving weird configuration issues.
- Run Google code search on your app source repository. Code Search is a tool for indexing and then performing regular expression searches over large bodies of source code. It is a set of command-line programs written in Go.
- Use a modern IDE like Netbeans, Eclipse and IntelliJ to find usage and references of classes and packages in your app code.
Look at all the XML files bundled within the app. These XML Deployment descriptors can be categorized into two categories. Standard discriptors and app-server specific ones. You will need to replace the vendor specific depkoyment descriptors with TomEE descriptors or Spring Beans in Tomcat.
If possible disaggregate the EAR file into multiple war files and tie them together as one logical app. This will increase loose coupling and force you to disentangle a web of dependencies that will make the transition to a microservices architecture much easier. You can chose to lift and shift your ear file as-is too since some of the buildpacks below will allow the staging of ear files. Typically it is OK to duplicate code like utility jars across multiple war files it leads to loose coupling and better isolation of function. Note as of today only the Liberty buildpack supports the cf push of ear files. The Tomcat based Java buildpack does not support ear files.
Look at shared libraries configured for the app server across apps. These server scoped shared libraries will need to be bundled in the app when it is pushed to CF.
Take advantage of Java EE6 and collapse EJBs and Servlets in a single war file. No classloader boundaries between Servlets and EJBs. EJBs and Servlets can share all third-party libraries (like Spring!), Use annotations and free your code from deployment descriptor hell.
Do not concern yourself with anything that deals with clustering, workload management, high availability, smart routing or proxying. These concerns are now taken care of by the underlying platform i.e. CF.
Check if there is any native code and ensure that the native library is included with the app and built on the linuxfs32 file system. When porting across Java EE versions please consider the breaking changes across EE specifications detailed in 3.3 J2EE to Java EE migration considerations
Address runtime Migration Concerns:
- A common issue in migrating JMS resources is that the way each application server configures JMS resources varies from vendor to vendor. Before migrating JMS definition to WebSphere, you should decide which messaging engine to use to exchange messages asynchronously. CF provides multiple JMS & AMQP messaging providers.
- Reconfiguring the JDBC resources is a common migration step. Use the appropriate JDBC driver to avoid problems with new implementations for new databases. Java Buildpack makes it easy to consume SQL and NoSQL data stores in the app running on the platform. You can auto-configure connections to back-end services or use spring-cloud-connectors to explictly control the JDBC connection to the persistence tier in the cloud.
- There are several areas where you configure security resources for your migrated applications and the administrative functions and roles, such as authentication mechanisms, authorization mechanisms, and security for web services. Configure the target app server in Cloud Foundry with the correct AAA mechanisms to port all your security apparatus. Apache Tomcat 8 Security Considerations
- Port JVM arguments, class path, and system properties to the buildpack and cf push manifests in Cloud Foundry.
- Understand the external factors that affect migration like interaction with external CRM/ERP systems via XML or legacy web-service stacks.
By default, the Tomcat instance provided by the Java Buildpack is configured to store all sessions and their data in memory. If your app relies on session persistence then configure session replication in CF by binding the app to a DB like redis session replication on Cloud Foundry. In some cases the configuration of the app to use the session database may need to be done explicitly based on the service variables injected via the VCAP_SERVICES environment variable. CF by default supports sticky session i.e. honors the JSESSIONID cookie.
If the app does not run on WebSphere Liberty Profile then use the WebSphere Migration Toolkit to get the app running on the WebSphere Liberty Profile. The Liberty Tech Preview tool helps you move WebSphere Application Server full profile applications to Liberty profile, which can be running inside or outside of the cloud. Follow the steps outlined by Cindy High on moving applications to Liberty using the migration toolkit.
You have now come to a fork in the road. You have multiple choices [AppServer(WebSphere | TomEE | JBOSS) * Cloud packager(Buildpack | Docker)] in deploying your app to CF.
1. Run the app in Cloud Foundry using the WebSphere Liberty Profile Buildpack. WebSphere Liberty Profile provides production support for Java EE7 and requires zero-migration across versions of the app server. The licensing model of running Liberty in production are NOT clear. The way I read the license, is that WebSphere Liberty profile can be used in production on CF for JVM heaps <= 2GB. So this gives you the license to install Liberty and fixes thereafter; not entitlement to support. No serious enterprise can run all their business on 2GB JVM heap. The catch here is that all your apps in total across the org cannot exceed 2 GB in heap. You can share 2GB across as many instances of Liberty as you like as long as the total JVM heap space across the instances does not exceed 2GB. ;) . Basically the first instance is free and thereafter you pay. Of course this licensing does not apply to Bluemix where you can run as many instances as you like as long as you pay Bluemix GB/hr cost.
2. Run the app in Cloud Foundry using TomEE. See this blog post from Romain Manni-Bucau that explains how to get the app running on Bluemix which is the IBM derivative of CF. The same technique will work on open source and other distributions of CF. This technique stages and runs the app as an embedded TomEE server within the app. 50 Shades of TomEE explains the many ways an executable fat jar can be cooked with your app and TomEE binaries. Another popular packaging option is to follow recipes and prepare a docker container of the app using the official TomEEDocker image from TomEETribe. Deployment of Docker image in Cloud Foundry is possible with the Diego BOSH release. Diego supports running of Docker images natively within CF. The recommendation here is to ONLY package the app in Docker and rely on the platform to inject service metadata. A great guide on packaging apps with Docker and keeping your docker images can be found in Adam Bien's blog JAVA EE 7 + THIN WARS + DOCKER = GREAT PRODUCTIVITY.
3. Stage and run the app in JBoss now renamed as WildFly, JavaEE certified application server using the jboss-buildpack. Redhat has provided a detailed guide on How-to migrate application running on WebSphere to WildFly was2jboss. If your application is not cloud-native then package and deploy your app using the official WildFly Docker image.
4. Refactor the app based on decomposition recipes outlined by Matt Stine in Migrating to Cloud-Native Application Architectures such as the anti-corruption layer and strangling the monolith. As you refactor the app to a microservices based architecture use SpringCloud OSS to implement the distributed services patterns necessary to overcome the fallacies of distributed computing. A playbook on app refactoring is provided by Josh Kruck here. Run the refactored slmmed down app on a specialized micro-server like Spring-Boot or DropWizard, What we want is a simple, lightweight, fire-and-forget packaging tool for server-side code taking advantage of executable JARs’ simplicity and platform independence. Merging all dependencies into a single JAR can be done with Tools like Maven’s Shade plugin or Gradle’s Shadow plugin. Before deciding on Spring Boot or DropWizard understand the tradeoffs as explained by the good folk at Takipi.
5. If you do not want to put any effort in refactoring the app since it is of low strategic value and business impact then simply packing the app in a Docker container and pushing it to PCF will allow you to deploy the app to the platform and leverage the operational & cost benefits with very little development cost. Run the IBM WebSphere Application Server Full and Liberty profiles in a Docker Container following documentation in FullWasInDocker and LibertyInDocker. Again Licensing headaches apply :-(. To get started with running in WebSphere go through this IBM InterConnect 2015 session on WebSphere Application Server Liberty Profile and Docker.

Tuesday, July 14, 2015

IBM Bluemix & Pivotal Cloud Foundry - A Tale of Two PaaS

Parameters for comparison
- Ramp for Developers
- Devops Experience
- Pushing Docker Images
- Services (Beta and Production)
- Cost for a stress test
-

PaaS and the Disruption of the Financial Tech Industry

A number of companies from the financial services industry are looking at Cloud Foundry or already have production deployment of Cloud Foundry. Why is the deployment and uptake of a platform, particularly strong in Financial Tech ? The reason is that this is an industry that is in a fervor of disruption and disaggregation. The incumbents have taken notice and are coming to the battle with guns drawn. Take a look at this picture.

http://fermi.vc/post/72559525330/disaggregation-of-a-bank

As we look at the future of enterprises with CF, as senior consultants and technical leaders it behooves us to paint a picture of the future. For a look at the ideal digital future of this kind of platform driven transformation you need to look no further than the European bank ING who is undergoing/underwent a complete digital transformation. ING CEO Koos Timmermans spoke at the Goldman Sachs European Financials Conference in Rome on June 16/2015. Take a look at this slide deckhttp://www.slideshare.net/ING/goldman-sachs-16-june-2015 in particular slides 6 and 9. Note ING used a home-grown PaaS and not CF. To quote Koos from the slide deck -

Investment of EUR 200 mln to further simplify, standardize and automate IT: • Decommissioning 40% of application landscape • Moving 80% of applications to zero-touch private cloud.

I also really enjoyed the following four part series from ThoughtWorks where the authors share their experiences and insights on ushering tech. fueled innovation in incumbent financial services organizations.

Much of what these articles state we already know; however the information is provided in the context of the financial tech industry and peppered with customer names/ case studies. These articles propose looking at all IT through a bi-modal view - Strategic and Utility IT. Thereafter they propose running strategic IT via a LEAN methodology in an innovation lab

Shift from a project mentality to a product paradigm
Break down large-scale, high-risk product development efforts into smaller, low-risk experiments.
Provide these teams with the budget discretion to quickly test multiple ideas in parallel.Teams have toc learly define the problem and identify appropriate metrics to measure success.
Collaborate early and often with stakeholders from technology, business, marketing, advertising, product, etc.

Subscribe to: Posts (Atom)

Saturday, October 10, 2015

Boiler plate apps for all buildpacks in Cloud Foundry

ruby_buildpack: Rails app to match ping-pong players with each other. Requires mysql.

https://github.com/cloudfoundry-samples/pong_matcher_rails

python_buildpack: Buildpack uses pip to install dependencies. Needs a requirement.txt

Flask app: https://github.com/michaljemala/hello-python

PyData app: https://gist.github.com/ihuston/d6aab5e4a811fe582fa7 Does not use pip. Uses conda.

.NET sample app: Contoso University

- https://github.com/pivotalservices/contoso-university

Method to get to the root cause of a memory leak in the Cloud

JBP_CONFIG_OPEN_JDK_JRE: '[memory_heuristics: {heap: 55, metaspace: 30}, memory_sizes: {metaspace: 4096m..}]'
JBP_CONFIG_OPEN_JDK_JRE: '[memory_calculator: {memory_sizes: {stack: 228k, heap: 225M}}]
JBP_CONFIG_OPEN_JDK_JRE: '[memory_calculator: {memory_heuristics: {stack: .01, heap: 10, metaspace: 2}}]'

As you can see, the options become increasingly cumbersome as you keep doing down this list. Your best hope is to catch it at 1, 2 or 3. Good Luck Hunting!

Thursday, October 1, 2015

Chasing Cloud Foundry OutOfMemory Errors - OOM

Coredump:

kill -6 $PID should produce a core file and leave the server running

Analyze these dumps using tools like Eclipse Memory Analyzer and IBM HeapDump Analyzer

Salient Notes:

References:

[1] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

[2] https://github.com/cloudfoundry/dea_ng/commit/e5fd523aaf2cddeda01d4134ecd2a929829c4147

Tuesday, September 29, 2015

Spring Boot Activator metrics collection in a spreadsheet

Debugging DEA Issues on Cloud Foundry - Halloween Edition

# Step 1 - Find app guid

# Step 3 - Locate DEA Job that maps to the DEA IP from previous step

# Step 4 Login into the DEA VM

# Step 5 Locate the warden container path

# Step 6 wsh into the warden container

# Step 7 - Now you are free to ftp files out or take heap dumps or thread-dumps

Sunday, September 20, 2015

Migrating The Monolith

Wednesday, August 19, 2015

Porting strategies for migrating apps to Cloud Foundry

Migration Recipes

Key Features in Cloud Foundry that Enable App Migration

1. Deigo
2. CF v3 app-service
3. TCP Routing
4. Persistence Support for Deigo
5. Context Path Routing
6. Routing service

Remediations when moving apps to a PaaS

Anti-Cloud Pattern	Remedy
Multiple Inbound protocols like RMI, JMX, Custom-TCP	TCP Routing, Tunneling
Persistence in local VMs like Caches, Transaction logs, CMS	External managed data services, Persistent volume support, sshfs service, EBS & S3 blob store, mount external NFSTransations
JTA & 2-pc commit	Use standalone transaction managers like Atomikos and Bitronix. Introduce eventual consistency patterns
Spaghetti Configuration	Externalize, plugin config via Config Server , init script in .profile.d directory that injects the config. as environment variables
Logging and Monitoring	Punch holes in outbound security groups, CF v3 API, Different process types

Wednesday, July 15, 2015

Migrating WebSphere Application Server Applications to Cloud Foundry

Migrations of Java EE applications vary in complexity, depending on a series of factors that are identified during the migration:

Source and target Java SE specification (JRE 1.5 to JRE 1.8)
Source and target Java EE specifications (J2EE 1.4 to Java EE 7)
Usage of specific Java components (EJB, JSPs, JSF, etc.,)
Usage of proprietary code that is specific for an application server
Vendor-specific deployment descriptors
Extended capabilities
Vendor-specific server configurations (JMS providers and JDBC connection pooling)
Class loading issues .
Integration with complex external systems such as a CRM or SAP

Process

Get a lay of the land. Identify ALL the open source and proprietary APIs used by the app. Determine what modifications you will need to make to the app if the target runtime packaged by the buildpack does not include the said libraries.

APIs
- Make a list of all WebSphere Proprietary APIs used by your app,
- Make a list of all the Java EE APIs used by your app,
- Make a list of all Spring Framework APIs used the app,
- Make a list of all 3rd party open source frameworks used by the app,
Analysis Tools
- Leverage static analysis tools like FindBugs, Checkstyle, PMD, Coverity and SonarQube. Write custom rules that will provide a breakdown of the external api usage. From the The Wise Developers' Guide to Static Code Analysis - PMD is a very customizable tool, so the performance and installation outcome really depends on how one has instructed the tool to find problems. Coverity is a commercial tool that it does a much deeper analysis than other tools and it also keeps track of the issues that you’ve found and fixed. SonarQube gives the best combination of usefulness and transparency against the relative complexity of setting it up and solving weird configuration issues.
- Run Google code search on your app source repository. Code Search is a tool for indexing and then performing regular expression searches over large bodies of source code. It is a set of command-line programs written in Go.
- Use a modern IDE like Netbeans, Eclipse and IntelliJ to find usage and references of classes and packages in your app code.
Look at all the XML files bundled within the app. These XML Deployment descriptors can be categorized into two categories. Standard discriptors and app-server specific ones. You will need to replace the vendor specific depkoyment descriptors with TomEE descriptors or Spring Beans in Tomcat.
If possible disaggregate the EAR file into multiple war files and tie them together as one logical app. This will increase loose coupling and force you to disentangle a web of dependencies that will make the transition to a microservices architecture much easier. You can chose to lift and shift your ear file as-is too since some of the buildpacks below will allow the staging of ear files. Typically it is OK to duplicate code like utility jars across multiple war files it leads to loose coupling and better isolation of function. Note as of today only the Liberty buildpack supports the cf push of ear files. The Tomcat based Java buildpack does not support ear files.
Look at shared libraries configured for the app server across apps. These server scoped shared libraries will need to be bundled in the app when it is pushed to CF.
Take advantage of Java EE6 and collapse EJBs and Servlets in a single war file. No classloader boundaries between Servlets and EJBs. EJBs and Servlets can share all third-party libraries (like Spring!), Use annotations and free your code from deployment descriptor hell.
Do not concern yourself with anything that deals with clustering, workload management, high availability, smart routing or proxying. These concerns are now taken care of by the underlying platform i.e. CF.
Check if there is any native code and ensure that the native library is included with the app and built on the linuxfs32 file system. When porting across Java EE versions please consider the breaking changes across EE specifications detailed in 3.3 J2EE to Java EE migration considerations
Address runtime Migration Concerns:
- A common issue in migrating JMS resources is that the way each application server configures JMS resources varies from vendor to vendor. Before migrating JMS definition to WebSphere, you should decide which messaging engine to use to exchange messages asynchronously. CF provides multiple JMS & AMQP messaging providers.
- Reconfiguring the JDBC resources is a common migration step. Use the appropriate JDBC driver to avoid problems with new implementations for new databases. Java Buildpack makes it easy to consume SQL and NoSQL data stores in the app running on the platform. You can auto-configure connections to back-end services or use spring-cloud-connectors to explictly control the JDBC connection to the persistence tier in the cloud.
- There are several areas where you configure security resources for your migrated applications and the administrative functions and roles, such as authentication mechanisms, authorization mechanisms, and security for web services. Configure the target app server in Cloud Foundry with the correct AAA mechanisms to port all your security apparatus. Apache Tomcat 8 Security Considerations
- Port JVM arguments, class path, and system properties to the buildpack and cf push manifests in Cloud Foundry.
- Understand the external factors that affect migration like interaction with external CRM/ERP systems via XML or legacy web-service stacks.
By default, the Tomcat instance provided by the Java Buildpack is configured to store all sessions and their data in memory. If your app relies on session persistence then configure session replication in CF by binding the app to a DB like redis session replication on Cloud Foundry. In some cases the configuration of the app to use the session database may need to be done explicitly based on the service variables injected via the VCAP_SERVICES environment variable. CF by default supports sticky session i.e. honors the JSESSIONID cookie.
If the app does not run on WebSphere Liberty Profile then use the WebSphere Migration Toolkit to get the app running on the WebSphere Liberty Profile. The Liberty Tech Preview tool helps you move WebSphere Application Server full profile applications to Liberty profile, which can be running inside or outside of the cloud. Follow the steps outlined by Cindy High on moving applications to Liberty using the migration toolkit.
You have now come to a fork in the road. You have multiple choices [AppServer(WebSphere | TomEE | JBOSS) * Cloud packager(Buildpack | Docker)] in deploying your app to CF.
1. Run the app in Cloud Foundry using the WebSphere Liberty Profile Buildpack. WebSphere Liberty Profile provides production support for Java EE7 and requires zero-migration across versions of the app server. The licensing model of running Liberty in production are NOT clear. The way I read the license, is that WebSphere Liberty profile can be used in production on CF for JVM heaps <= 2GB. So this gives you the license to install Liberty and fixes thereafter; not entitlement to support. No serious enterprise can run all their business on 2GB JVM heap. The catch here is that all your apps in total across the org cannot exceed 2 GB in heap. You can share 2GB across as many instances of Liberty as you like as long as the total JVM heap space across the instances does not exceed 2GB. ;) . Basically the first instance is free and thereafter you pay. Of course this licensing does not apply to Bluemix where you can run as many instances as you like as long as you pay Bluemix GB/hr cost.
2. Run the app in Cloud Foundry using TomEE. See this blog post from Romain Manni-Bucau that explains how to get the app running on Bluemix which is the IBM derivative of CF. The same technique will work on open source and other distributions of CF. This technique stages and runs the app as an embedded TomEE server within the app. 50 Shades of TomEE explains the many ways an executable fat jar can be cooked with your app and TomEE binaries. Another popular packaging option is to follow recipes and prepare a docker container of the app using the official TomEEDocker image from TomEETribe. Deployment of Docker image in Cloud Foundry is possible with the Diego BOSH release. Diego supports running of Docker images natively within CF. The recommendation here is to ONLY package the app in Docker and rely on the platform to inject service metadata. A great guide on packaging apps with Docker and keeping your docker images can be found in Adam Bien's blog JAVA EE 7 + THIN WARS + DOCKER = GREAT PRODUCTIVITY.
3. Stage and run the app in JBoss now renamed as WildFly, JavaEE certified application server using the jboss-buildpack. Redhat has provided a detailed guide on How-to migrate application running on WebSphere to WildFly was2jboss. If your application is not cloud-native then package and deploy your app using the official WildFly Docker image.
4. Refactor the app based on decomposition recipes outlined by Matt Stine in Migrating to Cloud-Native Application Architectures such as the anti-corruption layer and strangling the monolith. As you refactor the app to a microservices based architecture use SpringCloud OSS to implement the distributed services patterns necessary to overcome the fallacies of distributed computing. A playbook on app refactoring is provided by Josh Kruck here. Run the refactored slmmed down app on a specialized micro-server like Spring-Boot or DropWizard, What we want is a simple, lightweight, fire-and-forget packaging tool for server-side code taking advantage of executable JARs’ simplicity and platform independence. Merging all dependencies into a single JAR can be done with Tools like Maven’s Shade plugin or Gradle’s Shadow plugin. Before deciding on Spring Boot or DropWizard understand the tradeoffs as explained by the good folk at Takipi.
5. If you do not want to put any effort in refactoring the app since it is of low strategic value and business impact then simply packing the app in a Docker container and pushing it to PCF will allow you to deploy the app to the platform and leverage the operational & cost benefits with very little development cost. Run the IBM WebSphere Application Server Full and Liberty profiles in a Docker Container following documentation in FullWasInDocker and LibertyInDocker. Again Licensing headaches apply :-(. To get started with running in WebSphere go through this IBM InterConnect 2015 session on WebSphere Application Server Liberty Profile and Docker.

Tuesday, July 14, 2015

IBM Bluemix & Pivotal Cloud Foundry - A Tale of Two PaaS

Parameters for comparison
- Ramp for Developers
- Devops Experience
- Pushing Docker Images
- Services (Beta and Production)
- Cost for a stress test
-

PaaS and the Disruption of the Financial Tech Industry

http://fermi.vc/post/72559525330/disaggregation-of-a-bank

Investment of EUR 200 mln to further simplify, standardize and automate IT: • Decommissioning 40% of application landscape • Moving 80% of applications to zero-touch private cloud.

Shift from a project mentality to a product paradigm
Break down large-scale, high-risk product development efforts into smaller, low-risk experiments.
Provide these teams with the budget discretion to quickly test multiple ideas in parallel.Teams have toc learly define the problem and identify appropriate metrics to measure success.
Collaborate early and often with stakeholders from technology, business, marketing, advertising, product, etc.

About Me

Saturday, October 10, 2015

Thursday, October 1, 2015

Tuesday, September 29, 2015

Monday, September 28, 2015

# Step 3 - Locate DEA Job that maps to the DEA IP from previous step

# Step 5 Locate the warden container path

# Step 6 wsh into the warden container

Sunday, September 20, 2015

Wednesday, August 19, 2015

Migration Recipes

Key Features in Cloud Foundry that Enable App Migration

Remediations when moving apps to a PaaS

Wednesday, July 15, 2015

Migrating WebSphere Application Server Applications to Cloud Foundry

Process

APIs

Analysis Tools

Tuesday, July 14, 2015

# Step 3 - Locate DEA Job that maps to the DEA IP from previous step

# Step 5 Locate the warden container path

# Step 6 wsh into the warden container

Migration Recipes

Key Features in Cloud Foundry that Enable App Migration

Remediations when moving apps to a PaaS

Migrating WebSphere Application Server Applications to Cloud Foundry

Process

APIs

Analysis Tools