Eric Holscher - Posted in 2011

Read the Docs Update

2011-04-11T19:22:37+00:00

It’s been a while since I last talked about Read the Docs, and there has been a lot of activity. This is an update on the latest and greatest new features.

PSF Funding

The biggest news that has happened is that we have been given a grant from the Python Software Foundation to help host the site. Thanks PSF! They have blogged about it, and I am grateful that they have given us support. With the funds they have offered, we have been able to make Read the Docs a lot faster, and more robust. I will outline some of the changes below.

This also means we won’t be going away any time soon!

New Theme

We have a fancy new theme for documentation on Read the Docs! If you have the ‘default’ theme for your project, it will show up on the build of your docs on the site. I think it is pretty great, thanks to the designers who spent their time making it awesome. A really great feature is that the new theme is mobile ready. Go ahead and view a project using it on your phone, or make your browser smaller and you will see the fanciness. Having a custom theme will give us a base to build lots of other neat features on top of.

Better architecture

We had some connectivity trouble in between our servers a while back, and this prompted me to make the site respond better to these conditions. Every time you view documentation on a subdomain of readthedocs.org, your request will never hit Django. So all of these requests will work without a database. We have also added a second application server with a load balance in front, which means that one of the app servers could go away and your documentation would still get served.

That leaves our load balancer as the main single point of failure at the moment. We’re using Varnish for the load balancer, and we’ve implemented strong caching of data. Varnish will cache your docs for up to a week, and it will be actively purged when you rebuild your docs. This means that your docs will usually be served out of memory, and without dependence on any other server but that one. We have plans to elininate Varnish as a single POF, and then it would only be our hosting provider that would be a single point of failure (famous last words).

Intersphinx support

Intersphinx is an awesome feature of Sphinx that allows you to reference remote sphinx documentation easily. RTD now supports it for every project that we host.

Improved rtfd.org

I’ve always had big ideas for rtfd.org, since it can act as a short-url for things. projectname.rtfd.org has always redirects to the projects docs, but now we have something a lot better. Inspired by Jacob Kaplan-Moss and his work on django.me, we now support human-edited deep-linking within documentation hosted on RTD.

Taking another page from Jacob’s book, we seeded the index of our projects with their Intersphinx data, so a lot of references will automatically work. This works best with API reference docs, but anything people have put links to in their documentation should have been picked up. A couple of examples:

If you go to a non-existent link on rtfd.org, you will be prompted to enter a suggested URL. This will help build the data, and make it more useful for everyone.

rtd command line utility

RTD has had an API for a while now, and with the addition of the support for rtfd.org, I thought it would be neat to make it easier to access docs from the command line. With a simple pip install rtd, you will get an rtd utility that will open docs on RTD. It supports 2 arguments, the first being a project name, and the second being a slug to append for the rtfd.org functionality. So like the example above:

-> rtd pip
Pip Installs Packages.
Opening browser to http://pip.rtfd.org/
-> rtd celery Task
Distributed task queue
Opening browser to http://celery.rtfd.org/Task

It hits the RTD API to see if the project is on the site, and only opens your browser if it doesn’t exist. I hope that in the future we’ll make it easy to upload a project from the shell, and more.

More docs

Since we are a documentation site, we’ve always had documentation. I’ve been adding more as time has gone on, and most of the features I’ll be talking about today are already documented. I also broke the documentation up into sections for users of the site, and developers on the codebase, so it should be easier to find for everyone to find what they are looking for.

Conclusion

I think that RTD can be doing a lot more to help out the community with regards to documentation. I’ll write another post about that soon. But if you are interested in helping out with the effort, all of the code is open source and we love people to contribute. Feel free to jump in #readthedocs on Freenode as well, if you have any questions or thoughts.

Using Reviewboard with Git

2011-01-23T21:25:40+00:00

Reviewboard is a great tool for managing the process of Code Reviews. It has pretty good git support, but it might not be obvious what the best way is to use it. At work, I have a couple of different ways of pushing up code for reviews, which I’ll talk about.

This guide is assuming you are using your own bare repositories, on the server hosting the Reviewboard instance. It’s mainly here so that I can remember how to do this next time I need to. Also, thanks to Travis Cline for the initial pointers for this post.

Setting up Reviewboard

Once you have Reviewboard installed, you need to add a Repository in the admin, which is located at /admin/db/scmtools/repository/. The required fields have the following values:

Name: The name of the project
Hosting service: Custom
Repository type: Git
Path: The path to the local checkout of the git repository (ex. /var/lib/git/name)
Mirror Path: The Url to the repository (ex. ssh://git@your.server.com/var/lib/git/name)

The Repository Documentation has more about why you need this screwy configuration.

post Review

Before we get started, you’re going to want to get the post-review tool that works along with Reviewboard. The easiest way to get it is to pip install RBTools.

Pointing to the right Reviewboard Instance

The easiest way to make sure your pointing at the right Reviewboard instance is the .reviewboardrc file in your home directory. You only need to add one line to that file, which is:

REVIEWBOARD_URL = "https://path.to.your.instance"

If you are working with multiple instances that map to different repos, you can set the Reviewboard instance for the specific repo as well:

git config reviewboard.url https://path.to.your.instance

Reviewing a Branch

Alright, now you are all setup to start posting reviews. The easiest way to do that is to branch off of master, and start committing. If you are following something similar to this awesome branching model, this should be your normal workflow. Once your branch is ready to be merged back into your project, you want to get it reviewed. You can send a review equivalent to “this branch diffed against master” like so:

post-review --guess-summary --guess-description

Reviewing one commit

Another thing I find myself doing a lot is working on my master, and only having one commit to review. In theory this should probably be done on a bugfix branch, but such is life. There are other good use cases for only reviewing the latest commit as well. It’s done like so:

post-review --guess-summary --guess-description --parent=HEAD^

Reviewing arbitrary number of commits

It’s also possible to review any number of previous commits. It looks a lot like the previous command:

post-review -o --guess-summary --guess-description --parent=HEAD~4 #To review last 4 commits.

If you are familiar with git, you will understand that there is a lot more that you can do with the –parent argument. I’ll leave the possibilities up to your imagination.

Read the Docs Updates

2011-01-11T02:00:00+00:00

Documentation writing will always be hard work. It’s a much different mind-set than programming, and people that write good code might not necessarily write good docs. However, this is a known issue, and something that can’t really be solved.

What you can do is make it easier to write documentation. Every step along the way that you can give yourself an excuse to not write documentation is another undocumented open source project.

Luke Plant has a great post up about how important documentation is, and I completely agree. I imagine a lot of the people using Django are using it because of the documentation. I think as members of the Django community, we need to build a culture of documentation within the greater Python world. Python does tend to have better documentation than a lot of languages, but it’s still not nearly what it could be.

Read the Docs exists to make it easier to host your Sphinx documentation. Over the weekend, Bobby, Jonas, and I added a bunch of new features to the site. I think it’s getting to the point where there isn’t an easier or better way to host the documentation for your Django project, and we’re only going to keep improving it!

A different Eric added a really nice Getting Started guide for RTD, that shows how easy it is to get your projects hosted with us.

Anyway, on to the new features that we added.

New Features

Versions

Versions of projects are easily one of the biggest requested features on the site. For a long time we just supported building the latest versions of your documentation. Now we support versions of your documentation that are tagged in your VCS (hg/git only).

A lot of larger projects need versioning because they support one or two versions, as well as developing in the trunk. Django was the main project we were thinking of, but some other projects have put this to good use. A couple of examples are:

Django’s latest stable build
Fabric 0.9.3,
django-admin-tools’s awesome integration that has all it’s versions hosted with us.

PDF Support

Sphinx has interesting support for PDF generation through Latex. In my testing it was pretty unreliable, but I was able to rangle it into working well enough to expose in the UI. So now almost every project will have a “Download PDF” button. This code has version support as well, so we can offer PDFs of certain versions.

Another interesting part of this feature is that this building code has been abstracted out, so we can support epub, plain text, and all the other Sphinx output formats that people want.

Badges on the project pages

We killed the RTD header on hosted documentation pages in favor of a Badge in the lower right hand corner. The header clashed with a lot of the themes, and the badge is nice because it gives us a place to put functionality that is always visible, but is obviously not part of the hosted documentation. We want to build some more functionality into the badge, like switching between versions and linking back to the project’s RTD page, once we build a good UI for it.

Sponsorship

Revsys has agreed to sponsor the hosting costs for RTD. Jacob Kaplan-Moss has always been a big proponent of documentation, and I’m glad that he and Frank Wiles are helping us keep Read the Docs around and get better. We tried to make the sponsorship subtle and not intrusive, so please let me know if it bothers you and we can try and figure something out.

Conclusion

I think that these features are really starting to make RTD a compelling platform for hosting your documentation. We are planning more awesome features that will make RTD even better. I’m really excited about the project and I hope that you either host your docs with us, or find docs that we host useful.

Handling Django Settings Files

2011-01-10T19:09:27+00:00

I have seen a lot of talk over the past couple years about how to handle different settings files and databases, synced between production and development. I have happened onto a way of doing it that makes me happy, and figured I would share it with the world.

File structure

I use a file structure that looks like this:

project/
    settings/
        __init__.py (empty)
        base.py
        sqlite.py
        postgres.py

The base.py contains all of the configuration options that are shared among the databases. INSTALLED_APPS, etc. All of the DATABASE settings should be specified in the more-specific files. As well as things that differ by environment, like remote servers, cache settings, cookie domains, and other things.

This allows you to run the sqlite settings file, and have it be set to localhost, or whatever your development settings are. Then in production you just run against the postgres settings.

A good example of this being used in practice is on Read the Docs.

But wait, there’s more!

manage.py for dev

./manage.py is great for development. It is the easiest way to get started, and it automatically sets up your paths and stuff. With my setup, I actually explicitly set manage.py’s settings file to the sqlite file.

This means that whenever you are using manage.py, you are in a development context. So, what do you do about production?

django-admin.py is for production

In production, you set your DJANGO_SETTINGS_MODULE to the postgres settings file. So whenever you use django-admin.py, you will be running against the production database.

I really like this scheme, because it gives you a logical distinction between production and development in your code, and in your interface on the CLI. When you are developing, you are using manage.py and editing the sqlite settings file. The reverse for production.

Eric Holscher - Posted in 2011

Read the Docs Update

PSF Funding

New Theme

Better architecture

Intersphinx support

Improved rtfd.org

rtd command line utility

More docs

Conclusion

Using Reviewboard with Git

Setting up Reviewboard

post Review

Pointing to the right Reviewboard Instance

Reviewing a Branch

Reviewing one commit

Reviewing arbitrary number of commits

Other useful post-review flags

Read the Docs Updates

New Features

Versions

PDF Support

Badges on the project pages

Sponsorship

Conclusion

Handling Django Settings Files

File structure

manage.py for dev

django-admin.py is for production