Creating my Dream Server for Django

July 13, 2007. Filed under djangolighttpdmemcachepostgresqlslicehostubuntu

2009/2/13 An updated version of this article using Ubuntu Intrepid can be viewed here.

Note: This article has been reworked to use Nginx as frontend to Apache2, instead of the fairly unusual configuration shown in this article. You probably want to view the reworked version instead.

When you are building something important to you, you want to build it on a solid foundation. There is nothing more frustrating than trying to build a great piece of software and then running it on a poorly configured server. Recently I decided to stop having that feeling and I rebuilt my server, here are my instructions.

The end product is an Ubuntu Feisty server using Apache2 and mod_python to serve Django, and lighttpd to serve static media. It uses memcached as its caching backend, and uses Postgres8.2 as its database. The machine built using these instructions is in fact running this blog, which is a Django application. I performed this installation on a SliceHost 256 meg slice, but they would apply equally well to any Ubuntu server (not so well to shared hosting).

On to the fun stuff.

Credit

This setup guide is a consolidation of a huge number of other resources written by a wide number of individuals. Here are the resources I used during this setup process, I will try to also link to them in the context that I used them.

  • [Guide to doing anything server related with feisty][server]
  • [Starter guide to setting up feisty][start]
  • [Upgrading your SliceHost slice to Ubuntu Feisty][feisty]
  • [Setting up a mySQL Django server on SliceHost][start]
  • [Installing Postgres on Ubuntu][postgres]
  • [Setting up CentOS on SliceHost for Django with Postgres by DjangoJoe][centos]
  • [Configuring Lighttpd as a file server for Apache][lighttpd]

Upgrading Ubuntu Dapper to Feisty

These instructions were taken from [here][feisty].

Upgrading from Ubuntu Dapper (SliceHost's current Ubuntu OS) to Feisty is quick and painless. You do need to do the upgrades in order though, skipping straight to feisty is likely to make your installation unstable. The final 'lsb_release -a' is simply to confirm that the upgrade has occured. It should show that you now have Ubuntu Feisty installed.

### My article

1. My list entry one
2. My list entry two

<code class="python">def x (a, b):
    return a * b</code>

Adding Apache & Other Libraries

Mostly taken from [here][apache].

We install a number of libraries here. Some of them will have fun little installation screens to walk through. None of them were particularly difficult though. You may notice that psyco_pg2 is not being installed via apt-get, thats because the psyco_pg2 in the apt-get repository didn't work for me (kept causing segment faults).

 ### My article

 1. My list entry one
 2. My list entry two

 @@ python
 def x (a, b):
     return a * b

This will be the easiest section of the installation. Still nice while it lasted.

PostgreSQL

Instructions taken from [here][postgres] and [here][centos].

Make sure to use your own password instead of just 'password' in the code below.

import myproject.myapp.markdownpp as markdownpp

And then we edit the pg_hba.conf file.

class Entry(models.Model):
     title = models.CharField(maxlength=200)
     body = models.TextField()
     body_html = models.TextField(blank=True, null=True)
     use_markdown = models.BooleanField(default=True)

Go to the end of the file and comment out all lines that start with host (unless you will be accessing your database remotely). Finally the local line should look like

    def save(self):
        if self.use_markdown:
            self.body_html = markdownpp.markdown(self.html)
        else:
            self.body_html = self.body
        super(Entry,self).save()

Finally we'll want to restart Postgres:

class Resource(models.Model):
    title = models.CharField(maxlength=50)
    markdown_id = models.CharField(maxlength=50)
    content = models.FileField(upload_to="myapp/resource")

Configuring memcached

Now we are going to setup memcached. This is pretty easy to do, and it is going to provide a great caching system for your Django system. The first step is:

class Entry(models.Model):
     title = models.CharField(maxlength=200)
     body = models.TextField()
     body_html = models.TextField(blank=True, null=True)
     use_markdown = models.BooleanField(default=True)

On Ubuntu the user www-data is the user that runs the webserver, making it very similar to the apache user on some other distributions. Running memcached with the -u www-data option means that we'll be running memcached with the same user as the webserver. Port 11211 is the default for memcached, and probably should not be changed unless you are running multiple memcached instances. I chose to start my memcached instance with 32 megs of memory because my slice only has 256 meg total, and my Django app simply doesn't have very much information to cache. You may want to devote more of your memory to memcached, especially if you have a larger slice.

Next we need to get python-memcached, which is a python memcached client. There is an alternate cmemcached library for Python that is twice as fast as python-memcached, but I had trouble getting it to compile (I believe because I installed memcached from a repository instead of from source). Python-memcached is easy to get:

    def save(self):
        if self.use_markdown:
            pieces = [self.html,]
            for res in Resource.objects.all():
                 ref = u'\n\n[%s]: %s "%s"\n\n' % (
                    res.markdown_id,
                    res.get_content_url(),
                    res.title,
                    )
                pieces.append(res)
            content = u"\n".join(pieces)
            self.body_html = markdownpp.markdown(content)
        else:
            self.body_html = self.body
        super(Entry,self).save()

And that should be that.

At this exact moment (July 12, 2007) the current svn version of Django is broken with python-memcached. A patch has been submitted and is working its way through the submission system, so hopefully this issue will resolve itself soon. If anyone is affected by this situation, send me an email or leave a comment and I'll update this guide to include the required modifications.

Setting Up a Non-Root Account

Now for some security minded things. Not that I know very much about security, but every bit helps.

refs = entry.resources.all()

Copy the line for root and replicate it for your new account.

Now you'll want to close ssh and ssh back in as the new user your just made.

Then you'll want to disable the root account.

def save:
    super(Entry,self).save()
    res = self.resources.all()
    # etc etc
    super(Entry,self).save()

The line of thought behind disabling the root account is that it is bad to have a commonly known name that is available to SSH. Its also bad to have it accessible to a quick su. Admittedly your new account will still be accessible, but at least it will be your account and not an universally known one.

Setting up automatic SSH login with keys

This is more of a convience solution, but it is also a boost to your security once we disable password based access (although you might opt not to). I am assuming you have a set of rsa keys already on your local machine at ~/.ssh . If you don't have them (or you are using Windows, etc), you'd be better off looking for a more detailed guide.

import time, thread
from django.db import models
from django.dispatch import dispatcher
from django.db.models import signals

#######################
### your models go here ###
#######################

Remote login should now be automatic. Now we will restrict SSH a bit more

You will want to make these changes

Now log out, ssh in. Everything worked, right? If so we are now going to disable passwords for ssh (have to have an enable key). Once again

def resave_object(sender, instance, signal, *args, **kwargs):
    def do_save():
        time.sleep(3)
        try:
            instance.save()
        except:
            pass
    id = unicode(instance) + unicode(instance.id)
    try:
        should_resave = resave_hist[id]
    except KeyError:
        resave_hist[id] = True
        should_resave = True
    if should_resave is True:
        resave_hist[id] = False
        thread.start_new_thread(do_save, ())
    else:
        resave_hist[id] = True

and append this line

resave_hist = {}
dispatcher.connect(resave_object, signal=signals.post_save, sender=Entry)

and make this change

(PAM helps prevent brute force SSH logins, which no longer applies since password based login is disabled.)

Now only users with valid keys can ssh in... which means only your one account can ssh in. If you want to allow others to log in you can have them send you their public rsa keys, or you could temporarily enable password based login while they set up ssh.

Configuring iptables

Yep... I don't know how to do this yet on Ubuntu. Which is to say I don't know where the config file is. The command line tool is killing me. I refuse to learn another mini-language! Do we really need mini-languages for every mundane tasks?

That said, you should do this. Hopefully I'll crack open the helpfiles soon and update this afterwards.

Setting up Django, etc

Now we are going to setup Django. We will first create a postgres user for our Django app:

Hint: Write down the database table, user and password we'll be using them again in a couple of minutes, just far enough in the future to completely forget them all.

Now we need to give the www-data user access to our files (www-data is the user that runs the web server).

Then we'll want to create some folders for Django. You can feel free to play with the folder layout, its mostly a personal thing, but you'll have to keep any changes you make in mind when you follow the remaining instructions.

qaodmasdkwaspemas18ajkqlsmdqpakldnzsdfls

Now we check out the Django source and link the checked out source into the site-packages so that the python interpreter can find it.

qaodmasdkwaspemas19ajkqlsmdqpakldnzsdfls

Now we get to create our first Django project.

qaodmasdkwaspemas20ajkqlsmdqpakldnzsdfls

Now edit your newly minted settings.py file.

qaodmasdkwaspemas21ajkqlsmdqpakldnzsdfls

You'll need to make a number of changes, these are the lines you will need to add (not alter) to your settings.py:

qaodmasdkwaspemas22ajkqlsmdqpakldnzsdfls

I personally use a very long middleware cache because my pages don't change much (I am primarily serving a blog, and thus the caching entire pages is okay for my situation), the standard value is much lower, around 300 or so. The key prefix is used to distinguish caches between multiple Django projects using the same caching backend. If you are only planning to have one project using your caching backend then it is fine to leave the key as an empty string, if you do plan on ahving multiple projects each should have a unique key prefix. Finally, the anonymous only option means that logged in users will not recieve cached pages. For my application, where only the admin will ever be logged in, this is an appropriate setting, your milleage may vary.

And here are the already existing lines you will need to modify in settings.py:

qaodmasdkwaspemas23ajkqlsmdqpakldnzsdfls

Extending this Django setup later: you will eventually want to add your own templates and your own media along with your own appl ication. To provide access to your media files you will want to create a symlink from wherever they are into your /var/www/yourdomain.com/media folder like this:

qaodmasdkwaspemas24ajkqlsmdqpakldnzsdfls

For templates you'll just need to append a path to your template directory to the TEMPLATE_DIRS variable in the setting s.py file. Configuring Django applications and projects is sometimes more of an art than a science. Go try painting for a while, but feel free to ask for help if you need it.

After creating our Django project we need to finish setting up Apache. First we need to make a directory for our error logs:

qaodmasdkwaspemas25ajkqlsmdqpakldnzsdfls

Then we'll need to edit our apache config file:

qaodmasdkwaspemas26ajkqlsmdqpakldnzsdfls

That will initially be an empty file, and you'll be adding this to it:

qaodmasdkwaspemas27ajkqlsmdqpakldnzsdfls

Finally a few more symlinks to seal the deal:

qaodmasdkwaspemas28ajkqlsmdqpakldnzsdfls

Go ahead and restart apache and a stock Django page should show up:

qaodmasdkwaspemas29ajkqlsmdqpakldnzsdfls

Install Lighttpd

Basically I followed [the instructions here][lighttpd]. They are fantastic instructions, although I did end up playing with them a bit to get things working with my specific installation details.

First we open up the config file:

qaodmasdkwaspemas30ajkqlsmdqpakldnzsdfls

Next you need to uncomment line 60

qaodmasdkwaspemas31ajkqlsmdqpakldnzsdfls

Add the following (perhaps at line 118 like he does [here][lighttpd]):

qaodmasdkwaspemas32ajkqlsmdqpakldnzsdfls

Then we start the lighttpd server:

qaodmasdkwaspemas33ajkqlsmdqpakldnzsdfls

And enable some apache2 mods...

qaodmasdkwaspemas34ajkqlsmdqpakldnzsdfls

And finally edit the proxy.conf file

qaodmasdkwaspemas35ajkqlsmdqpakldnzsdfls

You will need to modify the file to look like this (you will be changing the existing config file to look like what is below, not just appending the text below to the config file):

qaodmasdkwaspemas36ajkqlsmdqpakldnzsdfls

You don't need to enable proxy routing/etc or modify any parts of the file outside of the tags

Next you need to modify the VirtualHost file:

qaodmasdkwaspemas37ajkqlsmdqpakldnzsdfls

Add these lines inside your VirtualHost (anywhere, except in the media location, is fine. I put them near the top):

qaodmasdkwaspemas38ajkqlsmdqpakldnzsdfls

Load changes into Apache and restart it.

qaodmasdkwaspemas39ajkqlsmdqpakldnzsdfls

The [instructions I borrowed from][lighttpd] recommended using curl to verify you are serving pages from lighttpd, but its easier (for me) to just use Firefox/Firebug and look at the response headers for the static media files you think lighttpd should be serving. Either way you should now be serving all files out of the /media folder using lighttpd.

Finished

At this point you should have a pretty kickin' Django server. You may want to spruce up the security a bit (I'll be looking into this and adding some more details), but I think this, security aside, is about as close to a production server as I can come. Mod_python for Django, lighttpd for static files, memcached for caching, Python 2.5 (2.4 would be a bit more stable, but at this point I think 2.5 is more than sufficiently mature), it works pretty well for me.

Let me know if you have any questions about any of the setup and I'd be glad to help. Also, if anyone has any ideas about improving this setup I'd really like to hear them.