Building a Django App Server with Chef: Part 2

Alternate title: Actually doing something useful.

Yesterday we covered the basics to getting started with Chef. You should have a remote server configured with chef, and have curl installed! Now lets go ahead and get some useful bits for your Django application.

What we’ll need

So this is going to be based around the way that I set up my servers, so if this is different than you, I’m sorry. However, I think it is a pretty solid way of managing them. A lot of the ideas here are stolen from Travis when he set up the server for Pypants.

So lets assemble a list of things we’re going to want in order to get a super basic Django configuration running:

  • A user to run our code as and who’s home directory we’ll store the data.
  • A basic global python ecosystem, including setuptools and pip
  • A virtualenv to store all the project-specific packages and code in
  • A copy of the project that we’ll be running

Let’s get started.

The finished code for today is located on github, with the tag blog-post-2. It is a copy of the completed steps, so feel free to peek through that and come back here for clarification (or to ask questions).

Setting up our user

For RTD, I run everything under the user docs. So we’ll go ahead and set up that user so that we can get our site set up. We’re going to go ahead and replace our “default” recipe, because right now it isn’t doing anything much useful. The relevant part is below:

cookbooks/main/recipes/default.rb

node[:base_packages].each do |pkg|
    package pkg do
        :upgrade
    end
end

node[:users].each_pair do |username, info|
    group username do
       gid info[:id]
    end

    user username do
        comment info[:full_name]
        uid info[:id]
        gid info[:id]
        shell info[:disabled] ? "/sbin/nologin" : "/bin/bash"
        supports :manage_home => true
        home "/home/#{username}"
    end

    directory "/home/#{username}/.ssh" do
        owner username
        group username
        mode 0700
    end

    file "/home/#{username}/.ssh/authorized_keys" do
        owner username
        group username
        mode 0600
        content info[:key]
    end
end

node[:groups].each_pair do |name, info|
    group name do
        gid info[:gid]
        members info[:members]
    end
end

There’s a lot of stuff going on here, so lets go over it. First you might notice that there’s this node variable, the node data structure is the JSON that you have in your node.json file. It is looping over the keys and values with ruby’s each_pair and pair functions.

The base_packages bit is a cool example of the power of the chef configuration. We have a list of packages that we want to install in our Attributes, and we’re looping over them and setting using the package Resource.

I realize I skipped over the run_list part yesterday, but it basically is just a list of recipes to run. Each of the resources in the default.rb file should be pretty self explanatory. The Chef Resource Documentation is really comprehensive, and will probably be the most referenced document that you use. The main resource’s that we used were group, user, file, directory, let’s take a look at the User declaration in particular.

Everything there should be pretty obvious, as it’s the information that goes into /etc/passwd for the user. However, the supports keyword isn’t obvious at first. This is part of the Common Attributes that can be set on all Resources. It’s a way of passing along configuration options to the Resource. manage_home actually just makes it so that the users home directory is created when the user is created.

So we’re going to have to go ahead and put some data in there for it to work with. Our node.json will now look like this:

node.json

{
  "run_list": [ "main::default", "main::python", "main::readthedocs" ],
  "base_packages": ["git-core", "bash-completion"],

  "users": {
    "docs": {
      "id": 1001,
      "full_name": "Docs User",
      "key": "ssh-rsa key-goes-here eric@Bahamut"
    }
  },

  "groups": {
    "docs": {
      "gid": 201,
      "members": ["docs"]
    }
  }
}

Adding a Basic Python Environment

Now lets go ahead and add a python recipe to build out some basic python stuff that we’ll be needing.

cookbooks/main/recipes/python.rb

node[:ubuntu_python_packages].each do |pkg|
    package pkg do
        :upgrade
    end
end

# System-wide packages installed by pip.
# Careful here: most Python stuff should be in a virtualenv.

node[:pip_python_packages].each_pair do |pkg, version|
    execute "install-#{pkg}" do
        command "pip install #{pkg}==#{version}"
        not_if "[ `pip freeze | grep #{pkg} | cut -d'=' -f3` = '#{version}' ]"
    end
end

Additions to node.json

"ubuntu_python_packages": ["python-setuptools", "python-pip", "python-dev", "libpq-dev"],
"pip_python_packages": {"virtualenv": "1.5.1", "mercurial": "1.7"},

Here we’re adding some global packages that we need. We’re going to install setuptools and pip so that we can install further python packages. python-dev and libpq-dev are so that we have the headers for libraries that need to compile against postgres and python. We’ll also be installing virtualenv and mercurial globally so that we can create our virtualenv and install packages from mercurial.

Creating a virtualenv

We’re going to introduce the first new Chef concept here, which is called a Definition.

  • Definition (cookbooks/*/definitions/*.rb)

    A definition is a custom Resource that you build to abstract a set of operations. Pretty simple

This is a definition that Jacob published and then I updated to make the permissions correct. It allows you to set up a virtualenv:

cookbooks/main/definitions/virtualenv.rb

define :virtualenv, :action => :create, :owner => "root", :group => "root", :mode => 0755, :packages => {} do
    path = params[:path] ? params[:path] : params[:name]
    if params[:action] == :create
        # Manage the directory.
        directory path do
            owner params[:owner]
            group params[:group]
            mode params[:mode]
        end
        execute "create-virtualenv-#{path}" do
            user params[:owner]
            group params[:group]
            command "virtualenv #{path}"
            not_if "test -f #{path}/bin/python"
        end
        params[:packages].each_pair do |package, version|
            pip = "#{path}/bin/pip"
            execute "install-#{package}-#{path}" do
                user params[:owner]
                group params[:group]
                command "#{pip} install #{package}==#{version}"
                not_if "[ `#{pip} freeze | grep #{package} | cut -d'=' -f3` = '#{version}' ]"
            end
        end
    elsif params[:action] == :delete
        directory path do
            action :delete
            recursive true
        end
    end
end

As you can see, it takes a bunch of arguments, then just wraps up a bunch of Resource definitions in a nice little package. There is a little bit of magic with the pip freezing things, but it’s basically just how we’re checking to make sure that a package isn’t instead before we install it. We are using only using the directory and execute Resources here.

Now we’re going to use this virtualenv Definition, and create the home virtualenv for our site. I like to keep my virtualenv’s in ~/sites/<domain>, so this will go into /home/docs/sites/readthedocs.org/. Since this is becoming specific to the site we’re building, it’s going to go into a readthedocs recipe:

cookbooks/main/recipes/readthedocs.rb

directory "/home/docs/sites/" do
    owner "docs"
    group "docs"
    mode 0775
end

virtualenv "/home/docs/sites/readthedocs.org" do
    owner "docs"
    group "docs"
    mode 0775
end

This will set up a basic virtualenv in our directory.

Getting our site set up

To get our site set up, we need to pull in the source code, and make sure our virtualenv has all the requirements. This code is a little bit hacky, and could probably be abstracted out a bit, but it will work for now. We’re going to go ahead and add some things to our readthedocs Recipe.

Additions to cookbooks/main/recipes/readthedocs.rb

directory "/home/docs/sites/readthedocs.org/run" do
    owner "docs"
    group "docs"
    mode 0775
end

git "/home/docs/sites/readthedocs.org/checkouts/readthedocs.org" do
  repository "git://github.com/rtfd/readthedocs.org.git"
  reference "HEAD"
  user "docs"
  group "docs"
  action :sync
end

script "Install Requirements" do
  interpreter "bash"
  user "docs"
  group "docs"
  code <<-EOH
  /home/docs/sites/readthedocs.org/bin/pip install -r /home/docs/sites/readthedocs.org/checkouts/readthedocs.o
rg/deploy_requirements.txt
  EOH
end

I like to have my runtime files in the venv/run directory, so we’ll go ahead and create that directory. Then comes the fun part.

We are checking the Readthedocs source out of github with the git Resource. Chef only supports git and svn as far as I can tell, so luckily I’m using git.

Then we’re going to install from the pip requirements file. This is using the script Resource, which allows you to inline a bash, ruby, python, or more script inside your Recipe. This is using a hard coded bash script to install the requirements, which sucks, but will work for now.

Note: Chef appears to buffer output and not show itself as doing anything when running the script Resource here, so it will look like your build will hang while it installs your pip requirements file for the first time.

Done for now

Alright, this post has gotten long enough, so we’re done for today. But we’re in a pretty awesome spot, I think. We now have our app server set up with a runnable version of our code. You can go ssh in and play around, you should be able to run simple manage.py commands inside the virtualenv and whatnot (after a syncdb).

Tomorrow we’ll talk about deploying our code with Nginx and Gunicorn. I’ve been having trouble with Upstart, so we might switch our deployment to Supervisord, but we’ll see how it goes.

Don’t forget to check out the finished code on Github to see the actual running examples.

This Post

Published 2010-11-09

Find me online

Email me

eric@ericholscher.com

Subscribe

RSS

Conferences

Write the Docs is happening in Europe and North America this year:

Fund my work

I am currently working to improve open source documentation. If you think this work is important, please support me on Gittip.

Table Of Contents