Auto Generating cloud pipelines for existing applications - THANKS TO Marcin Grzejszczak

This is a guest post where I copy - paste complete nuggets of wisdom from Spring and Pipelines guru Marcin Grzejszczak

The question I posed Marcin was how do we mass generate CI pipelines for existing enterprise applications ?

Marcin indicated that this functionality already exists in spring-cloud-pipelines via project crawler. However this functionality is currently only restricted to Jenkins because there’s no Java API to work with Concourse YAML or Concourse as such.

Project Crawler -  In Jenkins, you can generate the deployment pipelines by passing an environment variable with a comma separated list of repositories. This however doesn’t scale. We would like to automatically fetch a list of all repositories from a given organization / team.

Documentation for Crawler support 

Cloud Pipelines Concourse comes with a crawler.groovy file that allows to go through the repositories in an organization in an SCM server (e.g. Github, Gitlab, Bitbucket) and for each repo it will create a pipeline in Concourse.

How does this work with Jenkins ?

This is the logic for Jenkins that does the creation of pipelines -

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/jobs/jenkins_pipeline_crawler_sample.groovy

This is the crawler logic that creates the crawler -

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/jobs/jenkins_pipeline_crawler_sample.groovy#L34-L41

This is the logic that fetches the repos -

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/jobs/jenkins_pipeline_crawler_sample.groovy#L43-L45


Here you have the logic that iterates over repos and does some stuff (generates pipelines)

https://github.com/spring-cloud/spring-cloud-pipelines/blob/master/jenkins/src/main/groovy/org/springframework/cloud/pipelines/common/PipelineFactory.groovy#L40-L73


What can we do for Concourse ?

The following Groovy script to mass - generate pipelines:
// crawl the org
ProjectCrawler crawler = new ProjectCrawler(OptionsBuilder
    .builder().rootUrl(urlRoot)
    .username(gitUsername)
    .password(gitPassword)
    .token(gitToken)
    .exclude(repoProjectsExcludePattern)
    .repository(repoType).build())

// get the repos from the org
ListRepository repositories = crawler.repositories(org)
MapString, Exception errors = [:]

repositories.each { Repository repo ->
            try {
                // fetch the descriptor
                String descriptor = projectCrawler.fileContent(org, repo.name, repo.requestedBranch, "sc-pipelines.yml")
                // parse it
                PipelineDescriptor pipeline = PipelineDescriptor.from(descriptor)
                if (pipeline.hasMonoRepoProjects()) {
                    // for monorepos treat the single repo as multiple ones
                    pipeline.pipeline.project_names.each { String monoRepo ->
                        Repository monoRepository = new Repository(monoRepo, repo.ssh_url, repo.clone_url, repo.requestedBranch)
                        // generate a concourse pipeline for monoRepository
                    }
                } else {
                        // generate a concourse pipeline for pipeline
                }
            } catch (Exception e) {
                errors.put(repo.name, e)
            }
        }

if (generatedJobs.hasErrors()) {
    println "\n\n\nWARNING, THERE WERE ERRORS WHILE TRYING TO BUILD PROJECTS\n\n\n"
    generatedJobs.errors.each { String key, Exception error ->
        println "Exception for project [${key}], [${error}]"
        println "Stacktrace:"
        error.printStackTrace()
    }
}


In the future the spring cloud pipelines developer workflows shall be simplified such that

As a developer, you will clone one of our sc-pipelines prepared sample apps, alter it to suit your needs (and maintain the conventions that are already there). You’re done.

As an ops, ensure that the Jenkins seed job is running every now or then (or is triggered when a new repo is added to your Github org). That way a pipeline is created out of the box.

Thank you Marcin!!!