Yesterday I wrote briefly about my development to deployment pipeline, and I very briefly mentioned Fabric, which is a Capistrano-like tool written in Python that helps remove mindless repetition from deployment.

For a long time I was content with the SSH into the server and do stuff approach to deployment (I'm not managing any large deployments, and am usually dealing with small projects without the resources for staging servers and such), but I've been intoxicated by Fabric's sweet poison, and I think it's a noticable improvement--even for the smallest of deployments.

Briefly, I think using Fabric (or something similar) is a real win because:

1. You never have to leave your system. You get to use your tools, your files, your setup.

2. You only have to type one command to perform the deployment. Even for simple deployments you're still typing one or two dozen shell commands (changing directories, pulling from version control, rebooting the server), and you just don't have to type those commands anymore.

   Even if that seems like a small gain, I think it's more than that, because you can actually deploy without breaking your programming flow.

3. With a good deployment solution, you type once and then run it many times, so you're more likely to do things the right way instead of the quick way. (Repetition leads to boredom, boredom to horrifying mistakes, horrifying mistakes to God-I-wish-I-was-still-bored, and it goes downhill from there.)

4. Other developers, even those who wake up with a cold sweat when dreaming about linux administration, can deploy and rollback as well. If you throw a five minute GUI on top, then even your grandmother can be out there deploying and rolling back your production servers.

   By abstracting the deployment process into a simple deployment script you're making it easiser for other people to do your job for you. Which is a good thing. Unless you're worried that you seem unproductive at work. Then you might not like Fabric.

5. It scales well to large environments.

Now let's take a brief stroll through my Fabric setup.

Using Fabric

You can install Fabric using easy_install, but I'm using the current Git checkout, and the project is still young enough that code written for the latest packaged release will not work for the current development head¹. So I recommend you install it from its Git repository.

git clone git://git.savannah.nongnu.org/fab.git
cd fab
sudo python setup.py build install

Now go into your Django project directory and create a file named fabfile.py. Let's start out by adding a few utility functions I use for managing Git repositories.

def git_pull():
    "Updates the repository."
    run("cd ~/git/$(repo)/; git pull $(parent) $(branch)")

def git_reset():
    "Resets the repository to specified version."
    run("cd ~/git/$(repo)/; git reset --hard $(hash)")

Note a few important things:

1. The connection between Fabric and your server is not a persistent connection. That means you cannot use commands like cd in multiple commands and expect them to work.

   # this works as expected
   run("cd ~/git/$(repo)/; git pull $(parent) $(branch)")
   
   # this will run the command in the wrong directory
   run("cd ~/git/$(repo)/")
   run("git pull $(parent) $(branch)")
   

2. Fabric uses its own system for string interpolation. Usage is like this:

   # standard python string interpolation
   run("git pull %s %s" % ("origin", "master"))
   
   # Fabric string interpolation
   config.parent = "origin"
   config.branch = "master"
   run("git pull $(parent) $(branch)")
   

   Fabric's approach makes it easier to pass state around between various commands. Well, that's their story anyway. I'm not going to defend the approach beyond saying that I suspect they're working around some pain point in their architecture.

Now that we have those Git utilities, lets fill in the rest of the fabfile.py so that it can take advantage of the utilities.

The first thing you should do is create one or more functions, one for each set of servers you want to handle in a different way. For example, if you have a cluster of production servers and a cluster of staging servers, then you'd write something like this:

def production():
    config.fab_hosts = ['a.example.com','b.example.com']
    config.repos = (('project','origin','master'),
                    ('app','origin','master'))

def staging():
    config.fab_hosts = ['a.staging_example.com','b.staging_example.com']
    config.repos = (('project','origin','master'),
                    ('app','origin','master'))

The fab_hosts value is a special value that Fabric uses to determine the servers to run commands on, but the repos value is something that I added in myself for my scripts. You can add as many arbitrary values to config as your heart desires.

Now to put together the three functions I use for managing deployment.

def reboot():
    "Reboot Apache2 server."
    sudo("apache2ctl graceful")

def pull():
    require('fab_hosts', provided_by=[production])
    for repo, parent, branch in config.repos:
        config.repo = repo
        config.parent = parent
        config.branch = branch
        invoke(git_pull)

def reset(repo, hash):
    """
    Reset all git repositories to specified hash.
    Usage:
        fab reset:repo=my_repo,hash=etcetc123
    """
    require("fab_hosts", provided_by=[staging, production])
    config.hash = hash
    config.repo = repo
    invoke(git_reset)

To understand this code, it helps to know that there are a few magic functions that Fabric uses for deployment.

• run runs a command on another machine.
• local runs a command on your local machine.
• sudo runs a command on another machine as root.
• put sends a file to another machine from your local machine.
• invoke runs another function with the current function's configuration context.
• require creates dependencies among various functions, to prevent functions from being run without their prerequisites.

Now deployment is as easy as:

cd directory_with_fabfile/
# reboot the server
fab production reboot

# you can do all this on staging too
fab staging reboot

# update the git repositories
fab production pull

# update repositories then reboot
fab production pull reboot

# reset repositories then reboot
fab production reset:repo=my_app_repo,hash=13klafeomaeio23 reboot

This makes administration painless and quick as hell. But, hey, lets add just a little more spice. Let's add a deployment option that will only deploy your project if all its tests pass:

def test():
    local("python manage.py test", fail="abort")

And now you could deploy if and only if all tests pass by using this command:

fab test pull reboot

Now, let's say that you have some bad apples on your development team and even with the test option they still keep pushing broken code to your server. You could update pull to look like this:

def pull():
    require('fab_hosts', provided_by=[production])
    invoke(test)
    for repo, parent, branch in config.repos:
        config.repo = repo
        config.parent = parent
        config.branch = branch
        invoke(git_pull)

Now they won't be able to deploy their changes unless all the tests pass successfully. (Or they edit fabfile.py, but that isn't the point. ;)

I hope you're starting to get some ideas about how integrating Fabric into your deployment workflow can make your life a bit happier. It's useful, and it's fun. What more can you ask for?

Full Source

Here is the complete code we put together in this article.

REPOS = (("my_project", "origin", "master"),
        ("my_app", "origin", "master"))

def production():
    config.fab_hosts = ['a.example.com']
    config.repos = REPOS

def staging():
    config.fab_hosts = ['a.staging_example.com']
    config.repos = REPOS

def git_pull():
    "Updates the repository."
    run("cd ~/git/$(repo)/; git pull $(parent) $(branch)")

def git_reset():
    "Resets the repository to specified version."
    run("cd ~/git/$(repo)/; git reset --hard $(hash)")

def reboot():
    "Reboot Apache2 server."
    sudo("apache2ctl graceful")

def pull():
    require('fab_hosts', provided_by=[production])
    for repo, parent, branch in config.repos:
        config.repo = repo
        config.parent = parent
        config.branch = branch
        invoke(git_pull)

def test():
    local("python manage.py test", fail='abort')

def reset(repo, hash):
    """
    Reset all git repositories to specified hash.
    Usage:
        fab reset:repo=my_repo,hash=etcetc123
    """
    require("fab_hosts", provided_by=[production])
    config.hash = hash
    config.repo = repo
    invoke(git_reset)

As always, comments, suggestions and complaints are welcome. If you're looking to do more complex things with Fabric, then I highly recommend browsing he mailing lists, which managed to answer all my questions without me actually asking any.