Upgrade a Jenkins farm

| Robert Magnus

We have been using successfully Jenkins for a long time, but our Jenkins environment was very outdated. The master and its slaves were still running on JDK1.6 with Jenkins version 1.456. So it was “very” old. Even the installation of new plugins was impossible  because these were usually based on Java7. Overall we have 20 Jenkins slaves for 27 projects and 200 jobs. The projects are in several states: just started, ongoing, release or maintanence. The normal project work should not be disturbed by this upgrade.

Therefore we decided not to update the existing installation but to use a completely new Jenkins and update the JDK too.

There are two options: First is to update the server and pray. The second is to set up  a new master and then  move the jobs successively from the old to the new master. We opted for the second method. Initially the old master was duplicated. The copy got a new Jenkins and a new JDK and became the new master. All slaves in the new master were deactivated. Then one slave after the other was activated on the new master and deactivated on the old. In addition, prior to the switching of the slaves to the new master  the current build history of all jobs for this slave was updated (via rsync) on the new master. Doing this we ran into these problems:

  1. The parallel operation of two masters (with different versions) on the same slave doesn’t work. The new master updates the jar files on the slave and breaks the old master. Therefore, it is important to disable slaves on the old master after the switch
  2. Installed users get new API tokens (see https://wiki.jenkins-ci.org/display/JENKINS/Authenticating+scripted+clients). So if you have external calls to trigger Jenkins builds that use tokens (z. B. SCM Manager), these calls must be updated..

Steps

Our enviroment consists of one master and a dozen slaves (RedHat, Windows, Ubuntu) and crowd as authentication system. The master controls the slaves via ssh .

Install master

  1. Install Java8 (e.g. Ubuntu):
    apt-get install software-properties-common
    add-apt-repository     ppa:webupd8team/java
    aptitude update
    aptitude install oracle-java8-installer
  2. Install Jenkins. We recommend the  LTS-version (http://jenkins-ci.org/#stable). There are different installation  ways. Important: If you have projects still using JDK1.6 you can’t install a version higher than 1.612 (see https://issues.jenkins-ci.org/browse/JENKINS-29004).
    1. via package management (e.g. Ubuntu):
      1. add the jenkins repository to the apt sources:
        wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
        sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
        aptitude update
      2. install the Jenkins via aptitude safe-upgrade. It makes sense to mark the Jenkins status as hold.
    2. Download the archive directly from  https://jenkins-ci.org/ and install it.
  3. Start Jenkins

Update the Authentication

  1. Download the Crowd Plugin (not Crowd2!)
    cd /opt/jenkins/plugins
    rm -rf crowd*
    curl -OL http://updates.jenkins-ci.org/latest/crowd.hpi
  2. Fix the bug https://issues.jenkins-ci.org/browse/JENKINS-17288
    mv /var/run/jenkins/war/WEB-INF/lib/spring-context-support-2.5.6.SEC03.jar /var/run/jenkins/war/WEB-INF/lib/spring-context-support-2.5.6.SEC03.jar.orig
  3. Update the configuration in config.xml
    <securityRealm class="com.ds.tools.hudson.crowd.CrowdSecurityRealm" plugin="crowd@1.2">
      <url>http://XXXX/crowd/services</url>
      <applicationName>XXXXX</applicationName>
      <applicationPassword>XXXXXX</applicationPassword>
    </securityRealm>

Additional Clean Up

  1. Update all plugins
  2. Switch the “Tabbar for Views” to “Default Views Tabbar”
  3. In Manage Jenkins, then Configure Global Security enable “Enable Slave → Master Access Control” and set “allow all /.*” as File Access Rules.
  4. In Credentials, then Global credentials set up the internal user with password.

Conclusion   
 

The update involved 27 projects with a total of 200 jobs. The chosen method allows each project itself to decide when it is moved to the new Jenkins. When problems ocurred, there was a simple fallback: clean up the slave (remove the new jenkins jar files) and just re-enabled the slave on the old master.