07.12. One-click Deployment

Today’s topic is deployment. It’s called one-click deployment for a reason: Developers are lazy.
It’s hard to do less than clicking on one button, so that’s our goal.

With the growing need for lower time-to-market and faster response to user feedback it is inevitable to not be limited by technical factors (there are enough other obstacles already). The focus lies on reproducible results.

So, what do we need? Actually, not much. Disregarding the tools and practices that build the foundation of agile software development, you only need a central build server. But you’ve already got that one covered, right?

If you don’t, you should get one. It’s a huge help to discover errors quickly and be alerted instantly. This usually leads to a shorter time frame until a fix is done. Tests are run continuously and new parts are integrated into the whole code base.

Everything written here is based on Apache ant, so the easy choices here are Hudson and Jenkins. We’re focusing on Jenkins here, but we’ve successfully used some variation of this continuous integration/deployment script on both.

We’ve also used CruiseControl/phpUnderControl and Atlassian Bamboo in the past for continuous integration, so the deployment parts should work as well.

Back to the hard facts. We’re using the Template for Jenkins Jobs for PHP Projects here for our basic checks, so let’s assume you got it installed and your build server is working.

The basics

The typical workflow for a developer now consists of these steps:

• write some unit tests
• write some lines of code to get the tests to green
• commit to git/svn, let’s call it „release 1.32c“
• build server integrates the commit automatically and runs all configured checks
• build status is green (hopefully)

There’s not much deployment in this, so a few more steps:

• after the build is done, a new ant task is started, called „create-package“
• developer clicks on „deploy-to-staging“ (yes, another ant task)
• some 5 minutes later the staging environment has „release 1.32c“ installed

How does this work?

There are a few preconditions in our environment that haven’t been named yet. Not all of them are hard requirements, but they make some things a lot easier:

• all developers have a VM image for this project, based on Ubuntu 11.04
• this VM image is personalized to a certain degree (user/hostname)
• it is managed by puppet, all developers can edit puppet manifests
• the staging environment also consists of 4 VMs
  • Ubuntu 11.04 for the 2 frontend nodes
  • Ubuntu 10.04 for the 2 backend nodes
  • (hindsight is 20/20, we should’ve used the same version on all 4, but it’s not that bad)
• the machine that Jenkins is running on is also based on that same development image. This helps fight issues with incompatible versions and build tools.

The frontend nodes use a setup including varnish, nginx, PHP 5.3+APC, ZF 1.11, Dojo 1.6 and Doctrine 2.1.
The backend nodes use MySQL(Master+Slave), Apache Solr, RabbitMQ, memcached and ejabberd.

Back to ant.

The task create-package is building a .deb package that is to be deployed to all the staging hosts.

It basically executes these steps:

• git pull
• ant phpunit (unit tests, integration tests)
• ant clean (remove build artifacts)
• ant compilejs (using Google Closure Compiler)
• ant compilecss (compiling SASS to CSS)
• dh_make (read the Debian New Maintainers‘ Guide for details)

And that’s it.

The manually executed task is called deploy-to-staging and consists of these steps:

• ant deb-sign (sign the package with the development team’s key, optional)
• ant db-staging-up (we’re using Liquibase, so roll out the database changes)
• copy the .deb to all staging nodes via scp

Here’s one caveat: We’ve set up our staging (and production) nodes to include a directory (like /opt/repository) in /etc/apt/sources.list and puppet periodically checks if there are new packages in this directory, and installs it. Of course you could just call dpkg -i through ssh as well.

So far, so good. After waiting 5mins we’ve got our „release 1.32c“ deployed to all 4 staging nodes and the testers can do their work.

Wasn’t that easy?

We’re not quite done yet though. The testers really liked release 1.32c and we want to deploy to live now!

So this means:

• developer clicks on deploy-to-live (yes, you’ve guessed it, an ant task)
• some 5mins later „release 1.32c“ is live

In greater detail:

• fetch the latest „known good“ debian package from a staging host
• ant db-production-up (Liquibase doing it’s magic again)
• copy the .deb to all production nodes via scp
• there’s no step 4

If everything went well, you just deployed a new release!

Of course it’s not all perfect, there are some disadvantages and shortcomings.

The first thing we noticed is the build time. The build used to take 15 minutes because of massive disk I/O. Mostly all the analysis/metrics tools are to blame, if you have a few of those, every single one has to read and check all of your hundreds of files even if you exclude the external dependencies and libraries (which is a good idea, anyway). We put the files on a ram disk and brought the build from 15 minutes down to 4. Good enough.

The setup is quite complex (as you’ve seen if you’re still reading). It takes a bit to get used to and there are some moving parts. But so far it’s working nicely.

Jenkins is a single point of failure. If your build server goes the way of the dodo, it’s very hard to get a release going. There’s an easy fix, however: Have a second machine ready that’s running an identical copy of your Jenkins setup. If your main build server is down you can still use that one then. As your developer VMs are using the same ant build.xml for daily work, even using that as a last resort is possible.

There are also some goodies that aren’t crucial, but nice:

• everyone can deploy (even the product owner)
• 30 minutes from bug report to live deploy is not uncommon with this setup and it’s still following a fixed procedure
• this setup still allows us to do hotfixes (check out the tag deployed to production, do a fix, commit to a branch and HEAD, create .deb, deploy)
• Jenkins has some graphing functions, you can do commit stats or graph your Facebook fans

Having talked with developers from other companies there seem to be quite a few similar setups including chef, rake, phing or even make – so you’re surely not forced to use ant or a debian-based distribution.

Good luck simplifying your own deployment!