Provisioning Maven artifacts with Puppet

Our last blog post introduced how Puppet can be used to achieve Infrastructure-As-Code, and how to deploy Play applications following this practice. However, we didn’t address how the applications are actually copied to the host.

Apache Maven is a widely used build tool adopted by more and more companies to support their build process from compilation to deployment. Deploying, in the  Maven world,  means uploading the artifact to a Maven repository. Such Maven repositories are managed using Sonatype Nexus or JFrog Artifactory. However, this sort of deployment does not address the real provisioning of the application, i.e the deployment on the production servers.

This blog post presents a Puppet module to download Maven artifacts from a Nexus repository. This module closes the gap between the development team deploying their artifacts to a Maven repository, and the administration team responsible for installing and configuring the application.

Why use Puppet to download Maven artifacts ?

The first question you’re probably wondering is “why?”. At akquinet, Apache Maven and Nexus are cornerstones of our build processes. However, dealing with the live deployment still requires manual configuration, copies and so on. This process is error-prone and puts a lot of pressure on the development team. So, we started adopting Continuous Delivery and Infrastructure as Code to improve this last deployment step. But, all our applications and artifacts were stored in Nexus repositories, and the deployment to the production servers requires copying them onto a different type of server (generally via SSH/SCP) to be then deployed to the production servers. Even if these copies were automated using a continuous integration server (such as Jenkins), we’re duplicating the artifacts, making harder to trace the origin of each file. So, it makes a lot of sense to avoid this second server and to retrieve the artifacts directly from Nexus.

The puppet-nexus module is a Puppet extension downloading Maven artifacts from Nexus. Why Nexus? Because it relies on the REST API of Nexus. This Puppet extension supports authentication and repository selection. Actually, we have developed a Bash script to do the job. This script can be used outside of the Puppet module and is just wrapped inside Puppet. But we needed a solution integrated with Puppet and our deployment tool, and which simplified usage of the script.

The Puppet Nexus Module

The Puppet Nexus module defines two entities: The Nexus class and the artifact resource.

The Nexus class is configured with the base url of your Nexus installation and the credentials of the user:

class {'nexus':
    url      => "http://edge.spree.de/nexus",
    username => "nexus",
    password => "********"
}

Once done, you can declare nexus::artifacts as follows:

nexus::artifact {'commons-io':
   gav        => "commons-io:commons-io:2.1",
   repository => "public",
   output     => "/tmp/commons-io-2.1.jar"
}

The artifact is identified using the GAV coordinates (groupId:artifactId:version). The module supports SNAPSHOT versions (downloading the latest snapshot). The classifier and packaging attributes allow you to select the right file:

nexus::artifact {'chameleon web distribution':
   gav        => "org.ow2.chameleon:distribution-web:0.3.0-SNAPSHOT",
   classifier => 'distribution',
   packaging  => 'zip',
   repository => "public-snapshots",
   output     => "/tmp/distribution-web-0.3.0-SNAPSHOT.zip"

The repository attribute indicates the repository to use, generally a group (i.e. a group of proxies and hosted repositories) such as the public, public-snapshots, central. Check your nexus installation to know which repository you should use.

The output attribute indicates the location of the downloaded artifact. It must be an absolute path (following the Puppet convention).

The ensure parameter can be set to:

  • present : does not update the file if already there
  • absent : deletes the file if there
  • update : updates the file (default)
# Checks that the file is present, downloads it if needed
nexus::artifact {'ipojo-2':
    gav        => "org.apache.felix:org.apache.felix.ipojo:1.8.0",
    repository => "public",
    output     => "/tmp/ipojo-1.8.jar",
    ensure     => present
}
# Delete the file
nexus::artifact {'ipojo-1':
    gav        => "org.apache.felix:org.apache.felix.ipojo:1.8.0",
    repository => "public",
    output     => "/tmp/ipojo-1.8.jar",
    ensure     => absent
}
# Update the file
nexus::artifact {'ipojo-3':
    gav        => "org.apache.felix:org.apache.felix.ipojo:1.8.0",
    repository => "public",
    output     => "/tmp/ipojo-1.8.jar",
    ensure     => update
}

Getting and using the module

To use the presented module, clone the git repository. Once done, configure the modulepath of Puppet in the Puppet configuration file (/etc/puppet/puppet.conf). Once the module is installed correctly, you can download any Maven artifacts from the repositories hosted on your Nexus instance as presented above.

The common pattern is to download an archive containing your application (EAR, ZIP, WAR) and to unzip it (if needed) to the right location such as in the deploy folder of your application server.

Conclusion

This blog post has presented a Puppet Module closing the gap between Maven deployment and the production installation. The Nexus Puppet Module allows you to download Maven artifacts on your production servers. Artifacts are still organized and managed by Nexus. They are downloaded when the application is installed using Puppet.

12 Responses to Provisioning Maven artifacts with Puppet

  1. ming says:

    The Nexus plugin is exactly what I was looking for. Thank you.
    Our Maven build assembles tar.gz files and store them in Nexus.
    Can you recommend a way for Puppet to deploy those?
    Puppetlab recommends packaging RPM instead, but we want to stick with tarballs.

  2. Clement Escoffier says:

    Hi,

    In our case we’re using ZIP files. We just unzip them to a specific location using an ‘exec’ executing ‘unzip’. You can simply do the same for tar.gz files.

  3. Pranav Kumar Varshney says:

    Is there a Puppet Artifactory module?

  4. Clement Escoffier says:

    Not as far as I know. This module is specific to Nexus because it relies on the REST API provided by Nexus.

  5. David Gordon says:

    Just what I need – thanks. I had issues with the script when used with our Nexus installation, though: having both -L and –location-trusted on the curl line resulted in me jsut getting some text about the redirect URL instead of the artifact. Removing –location-trusted got it working, however..

  6. Andi says:

    Great idea.
    I’m not sure why, but puppet always tells me that the class nexus couldn’t be found.

    In which file i need to create the instance of nexus like

    class {‘nexus':
    url => “http://edge.spree.de/nexus”,
    username => “nexus”,
    password => “********”
    }

    in init.pp (/etc/puppet/modules/puppet-nexus/manifests/)
    or in e.g. site.pp (/etc/puppet/manifests/)?

    My modulepath is set to /etc/puppet/modules/
    I’m working with SLES 11.2 and puppet 2.6.12

    Thanks from Munich

  7. Clement Escoffier says:

    The nexus class must be configured within your site.pp file. But the nexus module must be in /etc/puppet/modules (or whatever is your modules directory).

  8. I love your blog.. very nice colors & theme. Did you make this
    website yourself or did you hire someone to do it for
    you? Plz answer back as I’m looking to create my own blog and would like to find out where u got this from. appreciate it

  9. Matt Fleming says:

    In 2.7 the way I got the module to work was by:
    sudo git clone https://github.com/cescoffier/puppet-nexus.git /usr/share/puppet/modules/nexus

    The key here was to name the module ‘nexus’ not ‘puppet-nexus’.

  10. Mani says:

    There is also a puppet module for Artifactory check this site – https://forge.puppetlabs.com/jcraigbrown/artifactory

  11. Great article. We would like to feature it on Sonatype’s new community (launching soon). Who should we contact about that?

  12. jon shanks says:

    I am not a huge fan of dropping files randomly on boxes, i think it is a better approach to facilitate using native package managers and using the maven plugins that represent these to build either rpms or deb’s and push them into a repository (pulp, aptly, createrepo) or whatever seems to fit the bill. If you are trying to have CI / CD, then you want your dev environments to always be fetching the latest version, if you are using snapshots rather than versions, it’s hard to decipher specifically what was put on the box. Does app1-snapshot.jar really contain the code changes i think it does? If you use versions (app1-1.2.3.jar), managing versions which end up being curl or wget requests to a jar string, means you will have to constantly be modifying the version data based on the build that has occurred (pushing the version number to another job after the unit tests pass to modify hiera data on the fly for example). These end up being quite fudge like solutions and hard to maintain, when you can simply always get the latest utilising the native package manager under the hood by utilising package provider in puppet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 202 other followers

%d bloggers like this: