Adding testing to pip

« Making Template Tag Parsing Easier | Large Problems in Django, Mostly Solved: Database Migrations »

Python packaging has been in a bit of a state of disarray for as long as I've been using it. Pip has come along to make installing python packages easier. It has a lot of features that are useful, but they have been talked about in many other blog posts.

Today I want to talk about adding testing to pip. If you are familiar with the Perl community, then you probably know about CPAN. It is basically Pypi for Perl. They have a command, called cpan, which allows you to install packages in a similar way to pip.

One of the steps that a package goes through before being installed on your system is that the tests are run. This allows you to know if the package that you have installed is actually going to work on your system. It may be broken on your platform, or you may be missing a library that it thought you had. Currently, pip has no way to test packages when they are being installed. I went looking for a way to make that happen.

It should be noted that pip is based on setuptools. Setuptools is what parses and understands most of the logic inside of your setup() function in the setup.py for your project (which you have, right?). Setuptools has an option called test_suite, which allows you to run setup.py test on your package, and have it run the unit tests. This is done by calling whatever python function is defined in test_suite.

I added the ability for pip to run setup.py test on a package that it is installing. It is executed by running pip install --test <package>. The implementation is on a ticket on bitbucket, and in a repository on github.

If you want to check it out, go ahead and clone my repo and check out the test_command branch. Then you can simply run

python pip.py install --test wsgiref

for an example package. If a package doesn't have a test_suite, then it simply doesn't run anything.

Note that if the tests fail, it doesn't impact the installation of the package. The python community's tests aren't quite good enough,and almost any Django package you try this on will not have any tests. I wrote about how to add testing to your django package, but the process is long and involved. I'm working to improve the situation for Django and hopefully having the ability to run tests in the package management tool will spur people to add testing ability to their setup scripts!

Nose makes this really easy, by simply adding test_suite = 'nose.collector' to your setup.py, nose will run your tests correctly. This is the level of support that I am hoping to implement for Django.

On a side node, I talked to Ian Bicking about this, and he suggested writing the test command as a separate command, so you would be able to do pip test wsgiref, if it was installed. This has some other problems, which I will talk about after I have implemented this functionality.

I would love to hear feedback, or if anyone has ideas for improving testing in the python and django communities. I have lots of ideas, and I will be writing more of them up over the following weeks.

Comments

1 schmichael says...

I would love for Python packaging tools (whether pip, easy_install, setuptools/distribute, or distutils) to make running tests on installation easier.

In fact it'd be nice if there were a way to make it run tests on installation by default and refuse to install packages with failing tests (without --force) like CPAN.

That being said, I don't think Python packages should ever default to that setting out of the box like CPAN. I'm not a Perl developer, but I use Perl tools from time to time which means trips to CPAN. Inevitably a test fails, and I end up scoring the Internet for help before giving up and just adding --force to the install. Training users to use --force by default is just sad, so I hope Python never finds itself in a similar position.

Posted at 3:40 a.m. on November 6, 2009

2 Kevin Teague says...

I'm not to sure about the value of such a feature. There are few reasons why I don't think this feature is too useful:

1) Install time and run time are not the same thing. It's possible for the environment or dependencies not to be ready or available at install time. If a dependant library can't be imported at install time doesn't mean it can't be imported at run time. For example, you might be adding in a particular dependency by modifying PYTHONPATH, or by source'ing ./bin/activate in a virtualenv. You could satisfy this environmental condition at install time, but when you go to run the software you might be in a different login shell with a different environment.

2) Testing is for developers. Automated testing is great, but it's value is in being able to maintain or refactor code with out breaking anything. You change some code, run the tests, then fix the code until the tests pass. Hopefully you run and pass all your tests before you release. If you pick up failing tests at install time, you don't have the source code handy, so how often are you going to from tests failing at install time all the way to providing a fix back or bug report back upstream? Also, in the spirit of "release early, release often", early in the lifecycle of a project it's valid to release with failing tests or poorly written tests as you may not yet fully defined the software. So if you have failing tests at install time, is it because: things aren't installed correctly? the test suite for that software is poorly written? the software is still early in it's lifecycle? or the developer's got sloppy and made a brown bag release?

(continued in next post due to 3000 character limit of commenting system ...)

Posted at 7:41 a.m. on November 6, 2009

3 Kevin Teague says...

3) The 'test_suite' metadata is just a single field, but often you'll divide suites into more than one. Primarily you might have 'fast tests' (aka "real" unit tests) and integration/functional tests. If you do have both types of tests, then I suppose the 'test_suite' field would only be applicable to the fast test suite. But many projects only have functional and integration tests, so it's not clear if there is any value in using the test_suite field for that, since something external outside of the installation of the library is necessary to setup the rest of what's required to run functional tests. e.g. Run 'python setup.py test' for fast tests, and run './bin/test' for functional tests. For "pluggable apps" in a web application, there is value in being able to run functional tests to ensure that it's been properly integrated into whatever it's being plugged into, but I think those types of tests need to be written to target a test discovery system that sits at a higher level than 'python setup.py test'.

So "yay!" for testing, but I think that testing at install time is mostly a feature looking for a reason to exist. Well, I've only given scenarios where having failing tests at install time is merely a false positive. I know there are plenty of scenarios where these failing tests could actually provide useful information.

However my hunch is that it's too easy for tests to fail at install time, and too hard to deduce productive steps to take as a result of those tests failing. But maybe I'm prejudiced by my experience with the CPAN client and it's obnoxious insistence on refusing to install if any tests fail. I've worked in organizations where it's system administrator's who do the installation, under a root account with a root environment, and developer's who run the software under an different account with a different environment. Here install time and run time are very different. A lot of time is wasted while developer's wait for software with "installation bugs" or people get the idea that the software has "quality problems" when in reality it's just one or two naive tests in otherwise fine suite of hundreds of tests that are making silly checks about whether or not the library is installed into the same type of environment that the developer's were testing the software in, and a lot of time is really wasted over nothing.

Surfing in Kansas