Backup and Restore Pivotal Cloud Foundry

Customers often ask us "What is the proper way to backup and restore Pivotal Cloud Foundry ?"

I want to go over all the options you have to successfully and expediently backup and restore Pivotal Cloud Foundry. The advice here is generally applicable to open source cloud foundry as well to a certain extent.

First a bit of a background. Pivotal Cloud Foundry is comprised of a number of tiles which roughly map to a BOSH release in open source. Each of these tiles need to backed up and restored individually. The OpsManager Tile is special in that it provides the inception VM and is responsible for deploying the BOSH Director. 

Manual Backup and Restore:

This is the recommendation from the product team itself. See 

Backup => https://docs.pivotal.io/pivotalcf/customizing/backup-restore/backup-pcf.html and Restore => https://docs.pivotal.io/pivotalcf/customizing/backup-restore/restore-pcf.html.  So what exactly happens during a backup and restore is explained by the pictures below.

Backup

Restore

Automated Backup and Restore: 

Manually backing up a foundation every night is not practical. Therefore the Pivotal Cloud Foundry Solutions team has an open source tool that automates the manual process for taking a backup and restoring it. This tool is called cfops => https://github.com/pivotalservices/cfops

Documentation on how to run backup/restore with cfops* can be found Here http://www.cfops.io/docs/  The basic process here is  Backup all Tiles and place the backup on a reliable file system. During a restore, restore each tile, apply changes and rinse and repeat for all tiles. 

cfops as of today does not support backing up or restoring data service tiles like MySQL, RabbitMQ etc., The tile settings are restored; however the data is not imported. So you will need an independent backup/restore strategy for data stores. This is where a data backup a tool like Shield => https://github.com/starkandwayne/shield from Stark and Wayne looks attractive. The project is in its infancy. Another option for backing up  the data services is leveraging Concourse.

Concourse can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off tasks, either locally or in another CI system. Concourse encourages practices that keep your project loosely coupled to the details of your continuous integration infrastructure.

If for some reason your BOSH Director disappears or the Director VM gets nuked you can bring it back using the procedure described here : https://blog.pivotal.io/pivotal-cloud-foundry/features/restoring-pivotal-cloud-foundry-after-disaster

Please note that the Backup and Restore process can be done with zero downtime if done right particularly because of the canary support provided by BOSH.

Canary instances are first instances updated within an instance group. Any update error in a canary instance causes the deployment to stop. Since only canaries are affected before an update stops, problem jobs and packages are prevented from taking over all instances.

At some point Pivotal Cloud Foundry will provide an official tool and automation strategy for backup & restore. Till then we can leverage cfops backup and restore function- automated with cron or in a CI pipeline with (Wercker/Concourse/Jenkins) to ensure that our sysadmins can sleep safe and sound.

* Please note cfops is not officially supported by the Pivotal Product Team; however we as Pivotal make our best effort to support this tool and keep it functioning across product upgrades.