One of the most important things I've done to improve my experience working with Django is to develop a consistent pipeline between development and deployment. Here I'll quickly outline some of the things I've done to make the experience easier, but I'm curious to hear about the techniques that you use as well.
local_settings.py++
One of the best tricks for working with Django deployment is to end your
settings.py file with these four lines:
try:
from local_settings import *
except ImportError:
pass
Doing that lets you override settings.py with the contents
of local_settings.py. For a public shareable project, you
can't take this much further than providing a local_settings.py
template which users can customize, but for a private project where
you control the production environment and the development environment,
then you can take this approach a bit further.
I create a production_settings.py file with settings for production servers, and
a devel_settings.py for local development. I add both of them to
version control, and then creating a local_settings.py for my local
checkout that looks like this:
from devel_settings import *
and for my production checkout local_settings.py looks like:
from production_settings import *
Finally, I make sure that local_settings.py is ignored by
version control.
I like this setup because it (like the standard local_settings.py
trick) makes it easy to modify shared settings, but in addition this trick also makes it
easy to modify the production settings withing SSHing into the
production server, and is simpler than maintaining a production and
development branch in the repository (which requires a lot of pointless
merging).
devel_settings.py and production_settings.py
My devel_settings.py file almost always looks exactly
the same regardless of the project it is part of. Yours may
look a bit different, but I find that it saves a few precious
moments of thinking to just keep a standardized project template
somewhere (instead of using djangoadmin startproject to create new projects).
Mine looks like this:
# Settings for devel server
import os
ROOT_PATH = os.path.dirname(__file__)
DEBUG = True
TEMPLATE_DEBUG = DEBUG
COMPRESS = False # django-compress setting
CACHE_BACKEND="locmem:///"
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = os.path.join(ROOT_PATH, 'devel.sqlite')
DATABASE_USER = ''
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = ''
MEDIA_ROOT = os.path.join(ROOT_PATH, 'media')
MEDIA_URL = 'http://127.0.0.1:8000/media/'
ADMIN_MEDIA_PREFIX = '/media/admin/'
The production_settings.py file varies a bit more, since
it depends on the Postgres and Memcached setup, but
very roughly it looks something like:
# Django settings for codernode.com project
COMPRESS = True # django-compress setting
COMPRESS_VERSION = True # django-compress setting
DEBUG = True
TEMPLATE_DEBUG = DEBUG
ADMINS = (('Mr Admin','admin@example.com'),)
MANAGERS = ADMINS
CACHE_BACKEND="memcached://127.0.0.1:11211"
EMAIL_SUBJECT_PREFIX = "[My Project]"
SERVER_EMAIL = "django@example.com"
DEFAULT_FROM_EMAIL = 'info@example.com'
DATABASE_ENGINE = 'postgresql_psycopg2'
DATABASE_NAME = 'some_database'
DATABASE_USER = 'some_user'
DATABASE_PASSWORD = 'some_password'
DATABASE_HOST = ''
DATABASE_PORT = ''
TIME_ZONE = 'America/Chicago'
LANGUAGE_CODE = 'en-us'
SITE_ID = 1
USE_I18N = True
MEDIA_ROOT = '/path/to/media/root/'
MEDIA_URL = 'http://example.com/media/'
ADMIN_MEDIA_PREFIX = '/media/admin/'
TEMPLATE_DIRS = ()
Yours will likely look quite different.
A Tale of Three Repositories
The foundation of my development-deployment pipeline is three repositories (it's a bit simpler using distributed version control, but you could accomplish more or less the same with Subversion or CVS).
A master repository hosted in a non-local and non-production location. I use my Slicehost slice that hosts my blog for all my private git repositories (it has daily and weekly backups, and if it does lag or crash, I can fix it myself), but you could use a paid GitHub account or one of the other similar services.
A local repository for each developer, which they push out to the master server at regular intervals.
A production repository on each production machine serving the project.
Developers do most of their work on their local machines, and push it to an appropriate branch on the master repository. They also pull from the master server occasionally to keep the repositories synchronized.
The production repositories are kept up to date by either:
Using Fabric to command them to pull the newest changes. This is the best option for large, medium and small deployments. That is, this is always the best option. Don't bother reading the others.
SSHing into the production servers and manually pulling the change set. Alternatively you could write a batch script that does this. This approach is more than sufficient for one machine deployment, but for anything more complex than that you'll want to use a deployment solution.
Setup a cronjob that pulls the changes periodically. This is the simplest and laziest of the approaches, but is a bad idea in most circumstances.
Using this setup, pushing a change on the development server to the production servers is lovable two steps:
git push
fab deploy
I highly recommend adding a fab revert hash-to-revert-to command
to the mix as well, for those awkward moments when your deployment
doesn't go quite as well as you hoped.
Production Server Setup
I keep my directory layouts very simple and uniform. All production servers
have a django user whose password has been disabled and doesn't have
access to SSH. The developers have accounts on the production machines, and they
are members of the django group, and thus can modify all of the django
user's files (having SSH setup to work without passwords will save days of your life).
I organize libraries into folders based on the version control system they use,
so I'll typically have a git folder and a svn folder, but I might have an
hg or bzr folder as well depending on the libraries that are being used.
(Okay, this is a lie, I can't remember ever having a bzr folder, but I wanted to
be inclusive.) Then I symlink them all into the /usr/python2.?/site-packages/
folder (I've been using Python 2.5 pretty exclusively of late, but I imagine in the next
year or two I may move up to 2.6 if there are any compelling performance gains).
I prefer linking directly from the checkouts (as opposed to using setup.py to
install the libraries), since it makes it easier to update the libraries across all
production servers at a later point if necessary.
For serving media I symlink the necessary folders from the project/application
repositories into /var/www/example.com/media/, and let Nginx handle
serving it. (I'm still quite happy with my Nginx/Apache2/mod_python stack,
even if it has fallen out of favor with the coolest kids for WSGI.)
Testing...
There are a lot of creative things you can do with integrating testing into the mix (deploy script only pushes changes to production if all the tests pass, or using a post-commit hook for the master repository to run the test battery and email developers if any tests fail, etc), but I'm still at a stage where I run tests manually. This will probably be the next area that I start improving upon my setup.
Do you use other tricks that I haven't mentioned here? Or perhaps have a better overall design? I'm curious to see how others approach this universal problem.
Your localsettings.py trick is really nice and I think I'll try to use the same approach for a PHP Projet I have to maintain and also for my own django projetcs. Thanks a lot for this idea !
Small remark on your production settings file you mentioned, Debug should of course be set to off ;-)
I look forward on reading the next episodes...
local_settings.py is a neat trick. Even neater is to sniff the hostname and import specific settings files from to that.
http://www.djangosnippets.org/snippets/600/
I love the article ... and like reading your blog in general. Darned good topic, thanks for sharing your details.
I do the hostname matching with a large if-else in the middle of the settings.py
I am thinking that I really need to move those pieces out into separate files (dev_settings.py, prod_settings.py).
I don't understand the win of using Fab. I do what you do -- commit changes to repository, ssh to production, svn update and restart/validate.
Some multiple/parallel SSH-enabled pusher or FUSE filesystem would be what I'd look for, once I have more than a single production target.
-- joe
We do more or less the same. Instead of importing a Python module we just read an ini file (just a matter of preference compared to a python module). We have a module to locate this ini file, so that it either append the hostname and look for it in a standard location, or it takes the value of DJANGO_SETTINGS_INI env variable and append. This means that we on development machines just place e.g a settings-<hostname>.ini in conf/ dir, while on the production machines (we have 4 atm) we set the env variable in the apache config, so that the load balanced machines used the same settings.ini file, instead of one pr. machine.
We also use Fabric for deployment, and are really happy about it. Only running one command is so much easier and error-free. I also find my self deploying more often since it's painless and leaves less room for mistakes (with deadlines all the time, this really makes the difference between keep up with the work instead of lacking way behind).
I have two question about you deployment process.
First, how do you deal with iterlibrary dependencies? For example, if appA does not works with appB < 0.1.2, or appA broke when appB > 1.0.0 is installed?
Second, how do you upgrade complex libraries or frameworks like django? If you install all applications and libraries into the common site-packages, than it would be difficult to upgrade all projects to the latest django, for example, if there are any backward incompartibilities. Don't you think, that it would be better to create separate python environment for each project?
I have two situations I deal with:
For both those scenarios I am comfortable with either a) not upgrading anything or b) upgrading everything. If I wasn't, then I do agree that using separate python environments for each project would be a good idea.
As for interlibrary dependencies, that just isn't something that crops up very often for me, and when it does I handle it manually. If it was a consistent problem (like if I was running a webhost), then I'd probably resort to separate Python environments.
This is a perfect use case for virtualenv - creating a simple python installation so you can be sure of having the right versions. Have a look at the django recipe for buildout - a tool that's come out of the Zope community.
Cheers, Dan
Nice read. That solves quite a few problems that I had before.
I've noticed that some variables set in local settings don't stick -- notably CACHE_BACKEND.
In settings.py, I have
Django uses the one in settings.py
Have you seen this sort of thing?
In one of my project, we have some pain when we try to convert the database from sqlite(used in development environment) to postgres(used in production environment). That's because sqlite doesn't have enough type checking and data constraint. So when you try to create the initial data based on sqlite database and import the initial data to the postgresql, you will found a lot of errors. How do you deal with this kind of transition?
You don't, on the sqlite db you keep testing data, the real data stay on the db in production. At least that how I do it and I thought about it from the beginning when I read this.
If you have been so ingenuous to fill an sqlite development db with production data then you need a set of tools to convert and validate the data out of the sqlite into whatever, painfull... or you could take a look at sqlalchemy...
We did that same trick for a while. But we found that it began to be a pain, esp. when setting up new projects. You have to add that little bit to every settings file in every new project you create. And, if it gets any more complicated, you have to be very careful to get it right.
I was documenting our complex customization to settings.py, when I realized that Django already solves the problem for me. You can specify "--settings" when you run "manage.py". Just create your other "devel_settings.py" and "import * from settings" before you override stuff.
We also use WSGI, so all we have to do is specify the 'production_settings' there.
I've learned that being able to run "./manage.py" consistently from box to box is less important than being able to consistently use Django out-of-the-box. In other words: Why mess with a perfectly good framework? The Django guys are a lot smarter than I am, and have already solved more problems than I've hit yet. Way to go, Django framework team! :)