This page will describe myGrid's best practice on how to set up a Jenkins job for:
- Building a Maven project from a corresponding Github repository
- Automatically builds and unit tests
- Triggered by git commits
- Triggered by upstream snapshot changes
- Nightly (cron-like)
- Automatically deploy snapshots to a Maven repository
- One-click release (versioning/tagging/deploy)
This relies on:
- GitHub source code repositories - http://github.com/taverna/
- Artifactory open source Maven repository - http://repository.mygrid.org.uk/
- Jenkins build server - http://build.mygrid.org.uk/ci/ - configured with:
jenkinsfor Artifactory server, using the Jenkins Artifactory Plugin
jenkins-mygridon Github. SSH keys for both master and slave, github user added as contributor to repositories (in order to tag).
- 1 Creating a Jenkins job
- 2 Building a snapshot
- 3 Building a release
Creating a Jenkins job
In order to create a job at myGrid's Jenkins server, you need to be a member of the Crowd group
jenkins-users, using the same credentials as for the myGrid JIRA or Confluence. Contact firstname.lastname@example.org if you are unable to log in to Jenkins or would like help with setting up the Jenkins job.
For the purpose of this documentation, we are going to create a Jenkins job for building the taverna-wsdl-activity repository.
Start by creating a New Item:
The Item name should reflect the Github repository name
If you want to create releases on several branches, e.g. both for a Taverna 2.x build (
For the purpose of completeness we will check :
|Copy existing item|
Most projects should be able to use from a pre-configured project like
Discard old builds
The Discard Old Builds setting should be set to:
- Max # of builds to keep: 30
The Jenkins configuration screen can be overwhelming as a multitude of options are presented. Most of these options can luckily be left at their (typically empty) defaults.
The JDK should normally be set to JDK7:
Source code management
Select Git for the source code repository.
For the Repository URL paste in the SSH repository from the Github repository page, after logging in to Github:
You should not need to fill in Credentials as the SSH keys on the Jenkins master/slave will be picked up automatically if you filled in the SSH repository path above.
Modify Branches if this build job is for a different branch, e.g. for Taverna 2.x use
maintenance. It is not recommended to use the blank for any default as this means releases and snapshots in the Maven repository could come from an unpredictable branch.
You can set up the Repository Browser location to the githubweb URL for your repository (Ensure this URL does not end with
The Jenkins build triggers allow different kind of events to trigger a new build. The recommended options are:
- Build whenever a SNAPSHOT dependency is built (one Jenkins job triggering another)
- Poll SCM (Build after detecting Git commits)
H/7 * * * *(every 7th minute on average)
For integrating projects (e.g. the workbench) you might want to also enable Build after other projects are built and select the virtual project
t3-nightly in order to have a nightly build. For most projects this is not necessary and therefore not recommended.
Enable Artifactory release management should be ticked, as it allows one-click releases initiated from Jenkins.
Let the Git release branch name prefix remain as
REL-BRANCH- (however this feature should not generally be used when making releases as it unnecessarily clutters the Github repository)
For the product builds like taverna-workbench-product, the Alternate Maven goals and options should be filled in to enable the installers, e.g.
clean install -Pdist.
Tick the Resolve artifacts for Artifactory, as it speeds up the builds by retrieving dependencies through the Artifactory proxy repositories.The myGrid artifactory server
should be pre-selected.
As of 2014-05-08, there is a bug in that the Resolution repository drop-down list can't be selected unless you select the Artifactory Server, click Save and then click Configure again for the Jenkins job. This also applies to the Post-build Action Deploy artitfacts to Artifactory (see further down).
After reopening the configuration, select the Resolution repository
repo (which cache all relevant internal and external Maven repositories), unless this is a standalone-type library that don't have Taverna dependencies, in which case
remote-repos is a better choice.
The default Maven version 3.2.1 is generally OK, but for the Goals and options, specify
Click Add post-build action (not Add post-build step!) and select Deploy artifacts to Artifactory.
The myGrid artifactory server
http://repository.mygrid.org.uk/artifactory should be preselected. The bug mentioned earlier also apply here, so after selecting the server, Save and then Configure again.
Now make sure you change the target repositores:
When should the other repositories be used?
The remaining defaults (e.g. Deploy Maven artifacts and Capture and publish build info are fine as they are.
Now click to Save the configuration.
Building a snapshot
Click Build Now to manually trigger a build.
Under Build History a progress bar will show how the build is progressing.
If the build was successful, the end of the build's Console Log should show that the artifacts were deployed to the Artifactory server:
The build snapshots are now deployed to the myGrid snapshot repository. – If not, ensure that the correct sub-repository (e.g.
libs-snapshot-local) was deployed to above.
Additionally the build page will have a link to the Artifactory build info for the deployed snapshots:
Similarly, the Artifactory server relates the artifacts back to the Jenkins job:
Building a release
Creating a release is done entirely within Jenkins using the Artifactory Release Staging functionality.
This functionality includes:
- Assigning stable version numbers, e.g.
1.5.0as opposed to
- Tagging the version in git, e.g.
- Optionally: Create a build branch, e.g.
- Optionally: Create a build branch, e.g.
- Bumping the SNAPSHOT version on the master branch, e.g.
- Deploying to Artifactory release repository, e.g.
The Artifactory staging does not have a way to lock down SNAPSHOT dependencies. Attempting to releasing with those would fail early with:
To prevent such an error, the project
pom.xml should therefore first be manually stabilized with release versions of all dependencies, plugins and parents – except for the version of the module(s) that are part of the build which should remain as SNAPSHOT. For instance, the pom.xml containing:
Must be updated to: (Notice how wsdl-activity remains at SNAPSHOT)
<parent> should be updated manually. For any remaining -SNAPSHOT dependencies,
mvn versions:use-releases will be able to lock down versions. To verify resolved dependencies, use
Commit the changed
pom.xml back to the
|Should I update all dependencies?|
A question might arise - should you update all dependencies, even if they are not SNAPSHOT?
Checking the build
Before starting the Artifactory Release Staging process, run a new build of the job in Jenkins to ensure that it is stable. This step is also recommended to ensure that the next step picks up the correct 'next version'.
Ensure that the build is running of the desired branch, e.g.
master. This can be checked in the Configuration under Git, but also in the beginning of the Console Output:
Ensure that the build did not accidentally release a stable (but untagged) version to Artifactory.
Artifactory Release Staging
To start the release process, click the Artifactory Release Staging button:
Module Version Configuration
The suggested version numbers is normally fine, using One version for all modules:
The Version per module can be used where modules advance independently, e.g.
workflowmodel-api 1.5.3 and
workflowmodel-impl 1.6.2. Do not attempt to re-release an existing submodule as the same version (it will fail the deployment to Artifactory). If it a submodule has not changed at all, then you can instead add a fourth qualifier element, e.g.
workflowmodel-api 18.104.22.168 - nothing outside would ever need to depend on this version.
Under VCS Configuration, please untick "Use release branch" as it would litter the Github repositories with branches like
- This option is NOT for building from a pre-existing release branch. The
The "Create VCS tag" should match the release version above, e.g. the suggested
wsdl-activity-1.6. Please check the naming convention for tags in the Github repository - some standalone repositories (e.g.
scufl2) use tags like
The defaults for Tag comments and Next development version comment are normally OK.
Click Build and Release to Artifactory to continue.
Inspecting the Console Log for the created build will show its progress. Things to check:
- Correct branch / commit
- Correct version numbers
- Successful deployment to correct repository
- Correct tag and successful github push
At the end of the build, the log should say something like:
If successful, the artifact has been tagged and released.
|No commit access?|
In some cases the build may fail to push the tag or branch to Github:
[RELEASE] Pushing branch 'REL-BRANCH-wsdl-activity-1.6.test' to 'email@example.com:taverna/taverna-wsdl-activity.git' > /usr/bin/git push firstname.lastname@example.org:taverna/taverna-wsdl-activity.git refs/heads/REL-BRANCH-wsdl-activity-1.6.test
Failure in post build SCM action: remote file operation failed: /var/lib/jenkins/workspace/taverna-wsdl-activity-maintenance at hudson.remoting.Channel@2a230d0f:jenkins-slave > /usr/bin/git checkout maintenance ..
This error will happen if the
Also ensure that the Jenkins job Git section is configured with the SSH repository, e.g.
If a release-version artifact already exists in a Artifactory repository (for instance because the job failed while tagging), deployment will fail:
[INFO] Deploying artifact: http://repository.mygrid.org.uk/artifactory/libs-release-local/net/sf/taverna/t2/activities/wsdl-activity/1.6.test/wsdl-activity-1.6.test-sources.jar [INFO] I/O exception (java.net.SocketException) caught when processing request: Broken pipe [INFO] Retrying request [INFO] I/O exception (java.net.SocketException) caught when processing request: Broken pipe [INFO] Retrying request [INFO] I/O exception (java.net.SocketException) caught when processing request: Broken pipe [INFO] Retrying request [ERROR] org.jfrog.build.extractor.maven.BuildInfoRecorder.sessionEnded() listener has failed: java.lang.RuntimeException: Error occurred while publishing artifact to Artifactory: /var/lib/jenkins/workspace/taverna-wsdl-activity-maintenance/target/wsdl-activity-1.6.test-sources.jar. Skipping deployment of remaining artifacts (if any) and build info.
If you really intend to overwrite the old version (because you deployed it just before), log in as an admin user on http://repository.mygrid.org.uk/artifactory to delete the offending version from the Tree Browser to navigate to the correct repository and sub-tree, and then use Delete versions.
Keeping the build
On the page for the successful build, click Keep this build forever. This build will then not be deleted by Discard Old Build. (Note: this button won't show if you forgot to turn on automatic discard)
Now, just give a better name to the build. Click Edit Build Information and change DisplayName to the version number, e.g.
The build is automatically linked to from the corresponding Artifactory build entry.
New snapshot build
After a release, do another Build Now to ensure a regular snapshot build still works as it should with its new SNAPSHOT version number. This will also deploy the new SNAPSHOT to the repository.