Skip to end of metadata
Go to start of metadata

This page will describe myGrid's best practice on how to set up a Jenkins job for:

  • Building a Maven project from a corresponding Github repository
  • Automatically builds and unit tests
    • Triggered by git commits
    • Triggered by upstream snapshot changes
    • Nightly (cron-like)
  • Automatically deploy snapshots to a Maven repository
  • One-click release (versioning/tagging/deploy)

This relies on:

Content

Creating a Jenkins job

In order to create a job at myGrid's Jenkins server, you need to be a member of the Crowd group jenkins-users, using the same credentials as for the myGrid JIRA or Confluence. Contact support@mygrid.org.uk if you are unable to log in to Jenkins or would like help with setting up the Jenkins job.

For the purpose of this documentation, we are going to create a Jenkins job for building the taverna-wsdl-activity repository.

New item

Start by creating a New Item:

The Item name should reflect the Github repository name

  • e.g. taverna-wsdl-activity

 

Several branches

If you want to create releases on several branches, e.g. both for a Taverna 2.x build (maintenance branch), and for Taverna 3.x (master) then create one build job for each (using Copy existing), and append the branch name to the project name for the non-masters:

    • taverna-wsdl-activity (master)
    • taverna-wsdl-activity-maintenance (maintenance branch)

For the purpose of completeness we will check :

Copy existing item

Most projects should be able to use from a pre-configured project like databundle and only customize project specific items like the Git repository.

Discard old builds

The Discard Old Builds setting should be set to:

  • Max # of builds to keep: 30

Job configuration

The Jenkins configuration screen can be overwhelming as a multitude of options are presented. Most of these options can luckily be left at their (typically empty) defaults.

The JDK should normally be set to JDK7:

Source code management

Select Git for the source code repository.

For the Repository URL paste in the SSH repository from the Github repository page, after logging in to Github:

e.g.  git@github.com:taverna/taverna-wsdl-activity.git

 

You should not need to fill in Credentials as the SSH keys on the Jenkins master/slave will be picked up automatically if you filled in the SSH repository path above.

Modify Branches if this build job is for a different branch, e.g. for Taverna 2.x use maintenance. It is not recommended to use the blank for any default as this means releases and snapshots in the Maven repository could come from an unpredictable branch.

You can set up the Repository Browser location to the githubweb URL for your repository (Ensure this URL does not end with .git)

Build triggers

The Jenkins build triggers allow different kind of events to trigger a new build. The recommended options are:

  • Build whenever a SNAPSHOT dependency is built (one Jenkins job triggering another)
  • Poll SCM  (Build after detecting Git commits)
    • Schedule:   H/7 * * * *   (every 7th minute on average)

For integrating projects (e.g. the workbench) you might want to also enable Build after other projects are built and select the virtual project t3-nightly in order to have a nightly build. For most projects this is not necessary and therefore not recommended.

Build environment

Enable Artifactory release management should be ticked, as it allows one-click releases initiated from Jenkins.

Let the Git release branch name prefix remain as REL-BRANCH- (however this feature should not generally be used when making releases as it unnecessarily clutters the Github repository)

For the product builds like taverna-workbench-product, the Alternate Maven goals and options should be filled in to enable the installers, e.g. clean install -Pdist.

Tick the Resolve artifacts for Artifactory, as it speeds up the builds by retrieving dependencies through the Artifactory proxy repositories.The myGrid artifactory server http://repository.mygrid.org.uk/artifactory should be pre-selected.

As of 2014-05-08, there is a bug in that the Resolution repository drop-down list can't be selected unless you select the Artifactory Server, click Save and then click Configure again for the Jenkins job. This also applies to the Post-build Action Deploy artitfacts to Artifactory (see further down).

After reopening the configuration, select the Resolution repository repo (which cache all relevant internal and external Maven repositories), unless this is a standalone-type library that don't have Taverna dependencies, in which case remote-repos is a better choice.

Build

The default Maven version 3.2.1 is generally OK, but for the Goals and options, specify clean install

Post-build actions

Click Add post-build action (not Add post-build step!) and select Deploy artifacts to Artifactory.

The myGrid artifactory server http://repository.mygrid.org.uk/artifactory should be preselected. The bug mentioned earlier also apply here, so after selecting the server, Save and then Configure again.

Now make sure you change the target repositores:

When should the other repositories be used?

  • If the Jenkins job is building a third-party library, e.g. jsonld-java, use ext-*-local
  • If the Jenkins job is building a Maven plugin or Maven archetypes, e.g. taverna-maven-archetypes, use plugins-*-local (Not for Taverna plugins!)

The remaining defaults (e.g. Deploy Maven artifacts and Capture and publish build info are fine as they are.

Now click to Save  the configuration.

Building a snapshot

Click Build Now to manually trigger a build.

Under Build History a progress bar will show how the build is progressing.

If the build was successful, the end of the build's Console Log should show that the artifacts were deployed to the Artifactory server:

The build snapshots are now deployed to the myGrid snapshot repository. – If not, ensure that the correct sub-repository (e.g. libs-snapshot-local) was deployed to above.

Additionally the build page will have a link to the Artifactory build info for the deployed snapshots:

Similarly, the Artifactory server relates the artifacts back to the Jenkins job:

Building a release

Creating a release is done entirely within Jenkins using the Artifactory Release Staging functionality.

This functionality includes:

  • Assigning stable version numbers, e.g. 1.5.0 as opposed to 1.5.0-SNAPSHOT
  • Tagging the version in git, e.g. wsdl-activity-1.5.0
    • Optionally: Create a build branch, e.g. REL-BRANCH-1.5.0
  • Bumping the SNAPSHOT version on the master branch, e.g. 1.5.1-SNAPSHOT
  • Deploying to Artifactory release repository, e.g. libs-release-local

Preparations

The Artifactory staging does not have a way to lock down SNAPSHOT dependencies. Attempting to releasing with those would fail early with:

To prevent such an error, the project pom.xml should therefore first be manually stabilized with release versions of all dependencies, plugins and parents – except for the version of the module(s) that are part of the build which should remain as SNAPSHOT. For instance, the pom.xml containing:

Must be updated to:  (Notice how wsdl-activity remains at SNAPSHOT)

The top <parent> should be updated manually. For any remaining -SNAPSHOT dependencies, mvn versions:use-releases will be able to lock down versions. To verify resolved dependencies, use mvn dependency:list.

Commit the changed pom.xml back to the master or maintenance branch.

Should I update all dependencies?

A question might arise - should you update all dependencies, even if they are not SNAPSHOT?

  • For Taverna 2.x, the answer is probably yes, as you want to ensure your artifact is using dependencies for the latest Taverna release
  • For Taverna 3.x, the answer is probably no, as you want your artifact to have as low OSGi dependencies as possible, and thus be compatible with existing installations.

Checking the build

Before starting the Artifactory Release Staging process, run a new build of the job in Jenkins to ensure that it is stable. This step is also recommended to ensure that the next step picks up the correct 'next version'.

Ensure that the build is running of the desired branch, e.g. maintenance or master. This can be checked in the Configuration under Git, but also in the beginning of the Console Output:

Ensure that the build did not accidentally release a stable (but untagged) version to Artifactory.

Artifactory Release Staging

To start the release process, click the Artifactory Release Staging button:

Module Version Configuration

The suggested version numbers is normally fine, using One version for all modules:

Which version?
  • For Taverna 3 and Taverna 2.x Products (Workbench, Command line, Server), use Semantic Versioning, e.g. 1.6.0 and 1.6.1-SNAPSHOT.
    • Backwards compatible changes that do not modify behaviour or API (typically bug fixes) can be reflected with increase in patch version, e.g. 1.6.1.
      • Includes change of Maven dependencies, unless those dependencies' interface has minor/major changes.
    • Any modifications to API (new method, new class or changed behaviour) MUST be reflected with a new minor version number, e.g. 1.7.0. 
    • Breaking changes (refactoring and deleting methods/classes) SHOULD be reflected with a new major version, e.g. 2.0.0.
  • Taverna 2.x modules have traditionally followed a looser Maven-plugin-like versions with optional patch levels, e.g. 1.5, followed by 1.6, followed by 1.6.1, followed by 1.7

The Version per module can be used where modules advance independently, e.g. workflowmodel-api 1.5.3 and workflowmodel-impl 1.6.2. Do not attempt to re-release an existing submodule as the same version (it will fail the deployment to Artifactory). If it a submodule has not changed at all, then you can instead add a fourth qualifier element, e.g. workflowmodel-api 1.5.3.1 - nothing outside would ever need to depend on this version.

VCS configuration

Under VCS Configuration, please untick "Use release branch" as it would litter the Github repositories with branches like REL-BRANCH-wsdl-activity-1.6

  • This option is NOT for building from a pre-existing release branch. The

The "Create VCS tag" should match the release version above, e.g. the suggested wsdl-activity-1.6. Please check the naming convention for tags in the Github repository - some standalone repositories (e.g. scufl2) use tags like 1.6.

The defaults for Tag comments and Next development version comment are normally OK.

Click Build and Release to Artifactory to continue.

Console Log

Inspecting the Console Log for the created build will show its progress. Things to check:

  • Correct branch / commit
  • Correct version numbers
  • Successful deployment to correct repository
  • Correct tag and successful github push

At the end of the build, the log should say something like:

If successful, the artifact has been tagged and released.

No commit access?

In some cases the build may fail to push the tag or branch to Github:

[RELEASE] Pushing branch 'REL-BRANCH-wsdl-activity-1.6.test' to 'git@github.com:taverna/taverna-wsdl-activity.git'
 > /usr/bin/git push git@github.com:taverna/taverna-wsdl-activity.git refs/heads/REL-BRANCH-wsdl-activity-1.6.test

Failure in post build SCM action: remote file operation failed: /var/lib/jenkins/workspace/taverna-wsdl-activity-maintenance at hudson.remoting.Channel@2a230d0f:jenkins-slave
 > /usr/bin/git checkout maintenance
..
Finished: FAILURE

This error will happen if the jenkins-mygrid user on Github does not have write access to the repository.

  • For the taverna repositories, ensure the repository is listed on the developers team.
  • For the myGrid repositories, ensure the repository is listed on the automatic tools team.
  • For other repositories, add jenkins-mygrid directly as a contributor.

Also ensure that the Jenkins job Git section is configured with the SSH repository, e.g. git@github.com:taverna/taverna-wsdl-activity.git and not https://github.com/taverna/taverna-wsdl-activity.git

Already deployed?

If a release-version artifact already exists in a Artifactory repository (for instance because the job failed while tagging), deployment will fail:

[INFO] Deploying artifact: http://repository.mygrid.org.uk/artifactory/libs-release-local/net/sf/taverna/t2/activities/wsdl-activity/1.6.test/wsdl-activity-1.6.test-sources.jar
[INFO] I/O exception (java.net.SocketException) caught when processing request: Broken pipe
[INFO] Retrying request
[INFO] I/O exception (java.net.SocketException) caught when processing request: Broken pipe
[INFO] Retrying request
[INFO] I/O exception (java.net.SocketException) caught when processing request: Broken pipe
[INFO] Retrying request
[ERROR] org.jfrog.build.extractor.maven.BuildInfoRecorder.sessionEnded() listener has failed: java.lang.RuntimeException: Error occurred while publishing artifact to Artifactory: /var/lib/jenkins/workspace/taverna-wsdl-activity-maintenance/target/wsdl-activity-1.6.test-sources.jar.
 Skipping deployment of remaining artifacts (if any) and build info.

If you really intend to overwrite the old version (because you deployed it just before), log in as an admin user on http://repository.mygrid.org.uk/artifactory to delete the offending version from the Tree Browser to navigate to the correct repository and sub-tree, and then use Delete versions.

Keeping the build

On the page for the successful build, click Keep this build forever. This build will then not be deleted by Discard Old Build. (Note: this button won't show if you forgot to turn on automatic discard)

Now, just give a better name to the build. Click Edit Build Information and change DisplayName to the version number, e.g. 1.6.0.

The build is automatically linked to from the corresponding Artifactory build entry.

New snapshot build

After a release, do another Build Now to ensure a regular snapshot build still works as it should with its new SNAPSHOT version number. This will also deploy the new SNAPSHOT to the repository.

Labels
  • None