.. post:: 2010-11-09 17:00:00 Building a Django App Server with Chef: Part 2 ============================================== Alternate title: Actually doing something useful. `Yesterday `_ we covered the basics to getting started with Chef. You should have a remote server configured with chef, and have curl installed! Now lets go ahead and get some useful bits for your Django application. What we'll need --------------- So this is going to be based around the way that I set up my servers, so if this is different than you, I'm sorry. However, I think it is a pretty solid way of managing them. A lot of the ideas here are stolen from `Travis `_ when he set up the server for Pypants. So lets assemble a list of things we're going to want in order to get a super basic Django configuration running: - A user to run our code as and who's home directory we'll store the data. - A basic global python ecosystem, including setuptools and pip - A virtualenv to store all the project-specific packages and code in - A copy of the project that we'll be running Let's get started. The finished code for today is located on github, with the `tag blog-post-2 `_. It is a copy of the completed steps, so feel free to peek through that and come back here for clarification (or to ask questions). Setting up our user ------------------- For `RTD `_, I run everything under the user docs. So we'll go ahead and set up that user so that we can get our site set up. We're going to go ahead and replace our "default" recipe, because right now it isn't doing anything much useful. The relevant part is below: ``cookbooks/main/recipes/default.rb`` :: node[:base_packages].each do |pkg| package pkg do :upgrade end end node[:users].each_pair do |username, info| group username do gid info[:id] end user username do comment info[:full_name] uid info[:id] gid info[:id] shell info[:disabled] ? "/sbin/nologin" : "/bin/bash" supports :manage_home => true home "/home/#{username}" end directory "/home/#{username}/.ssh" do owner username group username mode 0700 end file "/home/#{username}/.ssh/authorized_keys" do owner username group username mode 0600 content info[:key] end end node[:groups].each_pair do |name, info| group name do gid info[:gid] members info[:members] end end There's a lot of stuff going on here, so lets go over it. First you might notice that there's this node variable, the node data structure is the JSON that you have in your node.json file. It is looping over the keys and values with ruby's each\_pair and pair functions. The base\_packages bit is a cool example of the power of the chef configuration. We have a list of packages that we want to install in our Attributes, and we're looping over them and setting using the package Resource. I realize I skipped over the run\_list part yesterday, but it basically is just a list of recipes to run. Each of the resources in the default.rb file should be pretty self explanatory. The `Chef Resource Documentation `_ is really comprehensive, and will probably be the most referenced document that you use. The main resource's that we used were **group, user, file, directory**, let's take a look at the `User `_ declaration in particular. Everything there should be pretty obvious, as it's the information that goes into /etc/passwd for the user. However, the ``supports`` keyword isn't obvious at first. This is part of the `Common Attributes `_ that can be set on all Resources. It's a way of passing along configuration options to the Resource. manage\_home actually just makes it so that the users home directory is created when the user is created. So we're going to have to go ahead and put some data in there for it to work with. Our node.json will now look like this: ``node.json`` :: { "run_list": [ "main::default", "main::python", "main::readthedocs" ], "base_packages": ["git-core", "bash-completion"], "users": { "docs": { "id": 1001, "full_name": "Docs User", "key": "ssh-rsa key-goes-here eric@Bahamut" } }, "groups": { "docs": { "gid": 201, "members": ["docs"] } } } Adding a Basic Python Environment --------------------------------- Now lets go ahead and add a python recipe to build out some basic python stuff that we'll be needing. ``cookbooks/main/recipes/python.rb`` :: node[:ubuntu_python_packages].each do |pkg| package pkg do :upgrade end end # System-wide packages installed by pip. # Careful here: most Python stuff should be in a virtualenv. node[:pip_python_packages].each_pair do |pkg, version| execute "install-#{pkg}" do command "pip install #{pkg}==#{version}" not_if "[ `pip freeze | grep #{pkg} | cut -d'=' -f3` = '#{version}' ]" end end Additions to ``node.json`` :: "ubuntu_python_packages": ["python-setuptools", "python-pip", "python-dev", "libpq-dev"], "pip_python_packages": {"virtualenv": "1.5.1", "mercurial": "1.7"}, Here we're adding some global packages that we need. We're going to install setuptools and pip so that we can install further python packages. python-dev and libpq-dev are so that we have the headers for libraries that need to compile against postgres and python. We'll also be installing virtualenv and mercurial globally so that we can create our virtualenv and install packages from mercurial. Creating a virtualenv --------------------- We're going to introduce the first new Chef concept here, which is called a `Definition `_. - Definition (cookbooks/\*/definitions/\*.rb) A definition is a custom Resource that you build to abstract a set of operations. Pretty simple This is a definition that `Jacob published `_ and then I updated to make the permissions correct. It allows you to set up a virtualenv: ``cookbooks/main/definitions/virtualenv.rb`` :: define :virtualenv, :action => :create, :owner => "root", :group => "root", :mode => 0755, :packages => {} do path = params[:path] ? params[:path] : params[:name] if params[:action] == :create # Manage the directory. directory path do owner params[:owner] group params[:group] mode params[:mode] end execute "create-virtualenv-#{path}" do user params[:owner] group params[:group] command "virtualenv #{path}" not_if "test -f #{path}/bin/python" end params[:packages].each_pair do |package, version| pip = "#{path}/bin/pip" execute "install-#{package}-#{path}" do user params[:owner] group params[:group] command "#{pip} install #{package}==#{version}" not_if "[ `#{pip} freeze | grep #{package} | cut -d'=' -f3` = '#{version}' ]" end end elsif params[:action] == :delete directory path do action :delete recursive true end end end As you can see, it takes a bunch of arguments, then just wraps up a bunch of Resource definitions in a nice little package. There is a little bit of magic with the pip freezing things, but it's basically just how we're checking to make sure that a package isn't instead before we install it. We are using only using the **directory and execute** Resources here. Now we're going to use this virtualenv Definition, and create the home virtualenv for our site. I like to keep my virtualenv's in ``~/sites/``, so this will go into ``/home/docs/sites/readthedocs.org/``. Since this is becoming specific to the site we're building, it's going to go into a readthedocs recipe: ``cookbooks/main/recipes/readthedocs.rb`` :: directory "/home/docs/sites/" do owner "docs" group "docs" mode 0775 end virtualenv "/home/docs/sites/readthedocs.org" do owner "docs" group "docs" mode 0775 end This will set up a basic virtualenv in our directory. Getting our site set up ----------------------- To get our site set up, we need to pull in the source code, and make sure our virtualenv has all the requirements. This code is a little bit hacky, and could probably be abstracted out a bit, but it will work for now. We're going to go ahead and add some things to our readthedocs Recipe. Additions to ``cookbooks/main/recipes/readthedocs.rb`` :: directory "/home/docs/sites/readthedocs.org/run" do owner "docs" group "docs" mode 0775 end git "/home/docs/sites/readthedocs.org/checkouts/readthedocs.org" do repository "git://github.com/rtfd/readthedocs.org.git" reference "HEAD" user "docs" group "docs" action :sync end script "Install Requirements" do interpreter "bash" user "docs" group "docs" code <<-EOH /home/docs/sites/readthedocs.org/bin/pip install -r /home/docs/sites/readthedocs.org/checkouts/readthedocs.o rg/deploy_requirements.txt EOH end I like to have my runtime files in the ``venv/run`` directory, so we'll go ahead and create that directory. Then comes the fun part. We are checking the Readthedocs source out of github with the `git `_ Resource. Chef only supports git and svn as far as I can tell, so luckily I'm using git. Then we're going to install from the pip requirements file. This is using the `script Resource `_, which allows you to inline a bash, ruby, python, or more script inside your Recipe. This is using a hard coded bash script to install the requirements, which sucks, but will work for now. **Note**: Chef appears to buffer output and not show itself as doing anything when running the script Resource here, so it will look like your build will hang while it installs your pip requirements file for the first time. Done for now ------------ Alright, this post has gotten long enough, so we're done for today. But we're in a pretty awesome spot, I think. We now have our app server set up with a runnable version of our code. You can go ssh in and play around, you should be able to run simple manage.py commands inside the virtualenv and whatnot (after a syncdb). Tomorrow we'll talk about deploying our code with Nginx and Gunicorn. I've been having trouble with Upstart, so we might switch our deployment to Supervisord, but we'll see how it goes. Don't forget to check out the finished code `on Github `_ to see the actual running examples.