X
    Categories Industry Insights

Jenkins: Multiple Branches, Multiple Job Configs

AppNeta no longer blogs on DevOps topics like this one.

Feel free to enjoy it, and check out what we can do for monitoring end user experience of the apps you use to drive your business at www.appneta.com.

In the name of continuous delivery, we recently changed our Software Development Cycle (SDLC). Instead of working on just two major branches using SVN (current and future sprint), we create a branch for every new feature and issue using git. Once verified and tested, these feature branches get merged into the master branch. With Jenkins being our choice for build automation we faced some new challenges moving to the new SDLC. In the old SDLC we only had three Jenkins workflows: current sprint, future sprint and previous sprint. When a job of the current sprint workflow needed to be reconfigured then the corresponding job of the future sprint workflow had to be reconfigured as well. Now this wasn’t ideal either but still manageable for three workflows.

But doing this for 6-7 active feature branches, as we have now with the new SDLC, would have been a frustrating and error-prone venture. Just consider this use case: We have branches “master”, “A” and “B” and each branch has its own workflow in Jenkins. Workflow A introduces a new job that builds a new artifact used by some other jobs. We need to

  1. create a new job in workflow A
  2. register this new job in the parent job to build when workflow A builds
  3. add a new build step to an existing job to copy the new artifact over

So we touch at least three job configs in workflow A.

Workflow A builds fine – so far so good. The problems arise when it is time to merge A back into master. It is painful and annoying to go manually through each corresponding job in master and to apply the same changes again. By that time we might have already forgotten the config changes we did in workflow A back then.

To make it even more complicated let’s say workflow B needs the changes from workflow A before branch A got merged to master. We would have to manually redo the config changes to B and apply them to master as soon as branch B is merged back to master along with all other config changes that happened in B only.

Is there a better way to manage Jenkins job configs?

Of course: put it in version control! We figured we could add our Jenkins job configs to git to keep them in sync with our feature branches. This way, each branch knows exactly how to test itself, and if there are merge conflicts, they get handled as soon as the branches come together. But that alone was not enough. We also needed a mechanism to automatically update the configs if something has changed (e.g. merge from a different branch) and a simple way to create a new workflow using existing configs. There are some plugins for Jenkins out there that let you sync your job configs to a SCM repository (e.g. SCM Sync configuration plugin) or help you generate jobs for new branches using templates (e.g. Build Per Branch) but none of them really seemed to fit our needs. So we came up with our own solution using the Jenkins Rest-API.

Basic ideas:

  • Commit a template version of each job config to the master branch
  • Have a job “Branch-Create-Workflow” that creates a new workflow for a given branch
  • Automatically update configs before running a build

Commit a template version of each job config to the master branch

This is a one-time step to get the initial job configs into SCM. We can’t just commit the unmodified configs of the master workflow because the job names will be slightly different in each workflow (e.g. Master_Systest vs. BranchA_Systest). Converting to template means that we have to replace all occurrences of “Master” in a config xml with a placeholder (e.g. Master_Systest becomes @@WORKFLOW_NAME@@_Systest). Another value that will be different in each workflow is the branch to check out from so we create a placeholder for that as well (repo1/master becomes repo1/@@REPO1_BRANCH@@ and repo2/master becomes repo2/@@REPO2_BRANCH@@). After we’re done creating the template configs we commit each template to /jenkins/configs/<Jobname>_config.xml (e.g. Systest_config.xml) on the master branch of the corresponding repository.

Example config.xml:


<scm class="hudson.plugins.git.GitSCM" plugin="git@1.5.0">
<userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig>
...
<url>git@github.com:myOrg/repo1.git</url>
</hudson.plugins.git.UserRemoteConfig>
</userRemoteConfigs>
<branches>
<hudson.plugins.git.BranchSpec>
<name>@@REPO1_BRANCH@@</name>
</hudson.plugins.git.BranchSpec>
</branches>
...
</scm>
...
<hudson.plugins.copyartifact.CopyArtifact plugin="copyartifact@1.25">
<projectName>@@WORKFLOW_NAME@@_Systest</projectName>
<filter>systest*.deb</filter>
<target>./systest/</target>
<selector class="hudson.plugins.copyartifact.StatusBuildSelector"/>
</hudson.plugins.copyartifact.CopyArtifact>
...

Now when we create a new feature branch from master we can work on that branch and also make changes to job configs. We just need to make sure that every change is made to the corresponding template config in the feature branch as well.

Have a job “Branch-Create-Workflow” that creates a new workflow for a given branch

This is a parameterized job which takes the workflow name and branch names from user input to replace the placeholders we created in the first step:

The job needs to get the templates by checking out the corresponding branches from each repository. Then it will do the following using an ‘Execute shell’ build step:

  1. Create a new view (Jenkins API)
  2. Replace placeholders (@@WORKFLOW_NAME@@, …) with user input values for each config file
  3. Create a new job for each config file (Jenkins API)
  4. Add each new job to the just created view (Jenkins API)

Shell script (simplified):


# Create View
wget -qO- --post-data='name='$WORKFLOW_NAME'&amp;mode=hudson.model.ListView&amp;json={"name": "'$WORKFLOW_NAME'", "mode": "hudson.model.ListView"}' http://jenkins-master:8080/createView >/dev/null
# Create Jobs
for CFG in $(ls /tmp/jenkins/$WORKFLOW_NAME/*_config.xml); do
JOB=$(echo $(basename $CFG) | sed 's/_config.xml$//g')
JOB=${WORKFLOW_NAME}_$JOB
sed -i 's/@@REPO1_BRANCH@@/'$REPO1_BRANCH'/g' $CFG
sed -i 's/@@REPO2_BRANCH@@/'$REPO2_BRANCH'/g' $CFG
sed -i 's/@@WORKFLOW_NAME@@/'$WORKFLOW_NAME'/g' $CFG
wget -qO- --header="Content-Type: text/xml" --post-file=$CFG http://jenkins-master:8080/createItem?name=$JOB >/dev/null
# Add Job to View
wget -qO- --post-data= http://jenkins-master:8080/view/$WORKFLOW_NAME/addJobToView?name=$JOB >/dev/null
done

Automatically update configs before running a build

When a new build runs there could have been a merge from a different branch in the meantime. So we need to make sure that all jobs of the workflow are up-to-date. To accomplish this we need to create a “pre-build” job in that workflow which fetches all templates (same way as Branch-Create-Workflow), replaces the placeholders and then updates the job configs (see shell script). Once that’s done it will trigger a build on the parent job.

Shell script (simplified):


# Iterate over each Config File
for CFG in $(ls /tmp/jenkins/Master/*_config.xml); do
JOB=$(echo $(basename $CFG) | sed 's/_config.xml$//g')
JOB=Master_$JOB
if [ "$JOB" != "Master_Nightly-Build" ]; then
sed -i 's/@@REPO''1_BRANCH@@/'$REPO1_BRANCH'/g' $CFG  # added '' to prevent string replace
sed -i 's/@@REPO''2_BRANCH@@/'$REPO2_BRANCH'/g' $CFG
sed -i 's/@@WORK''FLOW_NAME@@/Master/g' $CFG
# Check if Job exists
set +e
wget -qO- http://jenkins-master:8080/job/$JOB >/dev/null
if [ $? = 0 ]; then
set -e
# Update
wget -qO- --post-file=$CFG http://jenkins-master:8080/job/$JOB/config.xml >/dev/null
else
set -e
# Create
wget -qO- --header="Content-Type: text/xml" --post-file=$CFG http://jenkins-master:8080/createItem?name=$JOB >/dev/null
# Add to View
wget -qO- --post-data= http://jenkins-master:8080/view/Master/addJobToView?name=$JOB >/dev/null
fi
fi
done

Not mentioned here, but we also have a job called Branch-Remove-Workflow that gets rid of a workflow using the Jenkins REST-API call to “removeJobFromView” and “doDelete”.

Conclusion

You can do quite a lot with the Jenkins REST-API without actually writing a Jenkins plugin. All in all with these tricks we found a convenient and automated way to manage our job configs throughout all feature branches. Merging a branch back to master doesn’t need any manual edit of the master jobs anymore because all jobs are updated automatically. A nice side effect is that we have a backup copy of all job configs in our SCM.

Daniel Schnabel: