How do you deploy applications?

Monday, 30 May 2011

I've got a few projects which are hosted in mercurial repositories. To deploy them I manually checkout the repository, create symlinks by hand, then update apache thttpd to make them work.

When I want to update my applications I manually become the correct user, find the repository and run "hg pull --update".

I think it is about time that I sat down and started doing things neatly. I made a start at this by writing a shell script for each site called .deploy then I drive it like so:

#!/bin/sh
#
# ~/bin/deploy  execute the .deploy file associated with this project.
#
while true; do

    #
    #  If we're at the root directory we're done.
    #
    if [ $PWD = "/" ]; then
        echo "Reached /"
        exit
    fi

    # found our file?
    #
    if [ -x ".deploy" ]; then

       ./.deploy
       exit
    fi

    cd ..
done

It seems the main candidate is capistrano, which was previously very Ruby on Rails centric, but these days seems to be divorced from it.

Alternatively there is the python-based fabric project which has been stalled for two years, vlad the deployer (great name!) which is another Rake-based and thus Ruby-loving system, and finally whiskey disk which is limited to Git-based projects as far as I can tell.

In short each of these projects is very similar, and each relies upon being able to do two things:

  • SSH to remote machine(s) and run a command.
  • Copy files to the remote command / pull a repository from a known location.

I've automated SSH before, and I've automed SCP/rsync. The hard part is doing both "copy" and "command" over one SSH channel - such that you don't get prompted for passwords multiple times - and handling the case of runnign sudo where appropriate. Still most of the initial stages are trivial.

I wonder what project I should be using:

  • I like perl. Perl is good.
  • I use mercurial. Mercurial is good.
  • Rake is perhaps permissable, but too ruby-centric == not for me.

Anything I've missed? Or pointers to good documentation?

ObQuote: "We need to be a little more constructive here, okay? " - Terminator 2

| 11 comments.

 

Comments On This Entry

[gravitar] John Eikenberry

Submitted at 15:05:46 on 30 may 2011

Fabric is quite actively developed. They just release 1.01 not to long ago and have an active community. Their primary web presense and documentation are at .

You missed the more 'pull' variety of deployment tools like Puppet and Chief . For a nice overview of 'push' vs 'pull' this is a nice post .

[gravitar] TomDV

Submitted at 15:09:16 on 30 may 2011

"I've automated SSH before, and I've automed SCP/rsync. The hard part is doing both "copy" and "command" over one SSH channel - such that you don't get prompted for passwords multiple times"

Use SSH keys to eliminate the need for a password to log in.
When configured - properly - you could also eliminate the need to use passwords for - certain - commands too. E.g. 'service apache2 reload'.

[author] Steve Kemp

Submitted at 15:13:17 on 30 may 2011

Thanks for the comment regarding Fabric, now I've used the correct URL I see it is current. Also packaged for Debian, which is a nice bonus.

Regarding push v.s. pull I'm very familiar with CFEngine and I've toyed with Chef & Puppet. But I don't really wish to mix the administration of my systems with the deployment of my applications - because they are both separately maintained and deployed.

Having a recipe to deploy project X is useful to me, and might be useful for other people who wish to deploy it - but if it gets hardwired into the deployment and management of a complete system then things get harder to take elsewhere.

[gravitar] TomDV

Submitted at 16:08:54 on 30 may 2011

s/you could/sudo could/

[author] Steve Kemp

Submitted at 19:01:44 on 30 may 2011

OK I've now written Fabric configuration files to deploy two web applications:

  • SSH into the host.
  • Update the repository.
  • Restart the thttpd server.
  • Run the test suite remotely.

Life is good, even if I still dislike python ;)

[gravitar] sime

Submitted at 23:46:02 on 30 may 2011

I use the railless-deploy gem with capistrano.

Easy enough to setup rules like the following:

$ cap staging deploy

etc.

[gravitar] Mike Greb

Submitted at 00:44:20 on 31 may 2011

I quite like your .deploy and ~/bin/deploy solution, I have a few mini project-ish things that this would be perfect for.

I'll have to take a look at Fabric for the slightly more complex things, I wasn't aware of it.

[gravitar] natxo

Submitted at 19:33:31 on 1 june 2011

Just curious: if you already use cfengine, why not use it to deploy your web applications? If you did not use it, I can see the point, but why learn just another middle-ware and add just another layer of complexity? Specially if you need root access to deploy the applications (restart daemons and such).

We distribute apps/files with rsync from cfengine just the same way we install packages to the servers from it, there really is no difference. You could even package the applications.

In the policy master we keep everything in version control (production, staging, testing - mercurial is good ;-) ) but I see no added value in keeping distributed copies of the repositories in the servers where the apps will run. No one is going to use those repositories anyway. So we just rsync a folder hierarchy from cfengine.

[author] Steve Kemp

Submitted at 20:55:45 on 1 june 2011

Natxo - There are two main reasons, one of which I previously alluded to that I regard chef/puppet/cfengine as taking care of systems and not applications.

(Obviously there is some overlap in what is a system software, and what is an application utility, library or dependency.)

But the bigger reason is that my CFEngine rules/recipes are not public whereas my applications have public repositories.

In the real world chances are nobody will deploy one of my web applications - such as my todo list - but it is possible, as that repository is both public & self-contained.

If I move my application code inside my cfengine repository it will no longer be public, and would be significantly harder for an average person to install.

Obviously there is some overlap. I could have a public application repository which would be deployed via the cfengine rules, but then you're mixing two unrelated things in a way that doesn't feel clever.

To my mind the deployment of a project or application is something that belongs, self-containedly, inside that applications repository. Whether that is "make install", or some remote tool, or a list of instructions in a README.

[gravitar] Yaarg

Submitted at 19:58:34 on 2 june 2011

Debian packages and Mcollective.

  • make debian packages of your applications so that the code extracts to the right place.
  • make a post-install script that creates a symlink if required.
  • use mc-package to update the package.
  • use mc-service to restart the web server.

alternatively you could write an mcollective agent that does some variation of the above.

mcollective makes fabric and capistrano look like cheap toys..

[gravitar] Phil Hollenback

Submitted at 14:10:56 on 10 june 2011

At Yahoo we use a really fantastic internally developed tool called Pogo. Pogo excels at massively parallel ssh execution and includes a constraint mechanism so you can for example limit the number of hosts on a server cluster that are deployed to simultaneously. There's also a pre/post hook mechanism to do things like take hosts out of rotation and perform healthchecks. If a host fails the healthcheck after a push, it's failure is included in the constraint consideration so you can avoid taking whole clusters of servers down accidentally.

My group (Yahoo Mail) uses Pogo to do software pushes to up to several thousand machines at a time.

Pogo is open source and can be found at https://github.com/ytoolshed/pogo/wiki. Unfortunately the open source version is not really ready for general use, but the developers assure me it will be Real Soon Now.

 

Comments are closed on posts which are more than ten days old.

Recent Posts

Recent Tags