Encouraging Testing in Django

« The importance of not deleting blog posts (read: ideas) | Should reusable apps have templates? »

I was having a conversation with Jacob tonight about testing in Django. He has shot down testmaker for being too specific for Django core, which I almost agree with, given my grandiose plans for it before the month is out. I'm quite okay with it staying a third party app for a little while longer.

However, that got us on the topic of testing, and I think it's interesting enough to post here to get some feedback and to tell people what's up. First we talked about trying to stub out some tests for people in the startapp command in Django. Like Rails does, except there is a really hard question about what to provide in that file. Should we provide a simple test that passes and makes people feel good about testing? Do we be evil and provide a test that fails, with assert False, 'Write some tests yo!'?

So the idea then progressed into perhaps having a command that can be called later in the process to stub out your tests files with real data. Perhaps stubbing a test for each view in your URLconf and each method in your Models or something like that.

We also talked about the possibility of adding a fifth part to the Django Intro Tutorial about testing. I think that this is a really great idea, and would help further the testing culture inside of the Django community. I volunteered to write the first draft of that document, so expect that to be posted to this blog sometime next week.

So I'm just kind of curious what people think is a good way to get testing integrated into the Django community better. I am trying to write some tools that will help people write tests, which would help them have tests :). But I think that there is a lot more that can be done to get people thinking about testing their applications.

Should we be encouraging people to be testing from when they start a new application in Django? If so, what should we put in the tests.py file when they create an application? Should we just stub out an empty tests.py file to remind them that they should be writing tests? Should we be pushing best practices from the beginning in that form, or giving people a builtin option to perhaps then stub it out later?

I think that the Django community is lacking in the testing realm these days, and I'm curious what we can be doing to get more people excited about testing. It's a great tool, and something that everyone should be doing. So I'd love to hear feedback or ideas about what people think can and should be done with regards to testing.

Comments

1 Paul Hummer says...

Have you seen Pythoscope (http://pythoscope.org/)? I've considered on more than one occasion adding support for walking through a Django project and making its own tests.

I really think pythoscope and testmaker could be a good solution for stubbing out and writing tests.

Posted at 3:10 a.m. on November 14, 2008

2 Julien Phalip says...

Hello again Eric ;) I totally agree that testing needs (a hell of a lot) more attention in the Django community. Recently I started compiling an argument about that. I'll try to wrap it up and write a blog post about it. Will let you know!

Posted at 3:29 a.m. on November 14, 2008

3 James Tauber says...

Well, you can probably guess my thoughts on this matter, but I want your stuff in Pinax and I really want Pinax to lead the way in how to test, particularly at the project level.

Posted at 7 a.m. on November 14, 2008

4 Ben says...

I think that in a production environment it's sometimes useful to be able to test against a (anonomized) copy of the live data. The two main "barriers to entry" I've notices testing our project were: - The setup for each test takes a long time (the flush/load) cycle. There are some tests that we'd like to run without doing a test setup. - Fixtures incur a lot of overhead when the database schema is changing rapidly.

Our solution to that was to take the django nose plugin and add some options to it (around whether to setup a test db or not), enable doctesting (with ELLIPSIS and WHITESPACE options set) and run all tests in a transaction.

So what we now have is each project with a 'doctests' directory into which we put all of our sphinx documentation and all of our doctests (which are also linked into the docs).

Posted at 9:38 a.m. on November 14, 2008

5 yml says...

I like the idea of the django command extension that write stub tests for all the views defined in urls.py. Ideally these test would use reverse(..., kwargs=...) and be a unittest rather than doctest. Unittest are then much easier to introspect using pdb of ipdb. Thank you for all the great working you are doing around this important topic. --yml

Posted at 12:21 p.m. on November 14, 2008

6 Justin Lilly says...

I'd personally like to see testing stubbed out as such:

## VIEWS

## MODELS

## TEMPLATE TAGS

or some such so you remember what all you can test in django.

Posted at 1:12 p.m. on November 14, 2008

7 Chris Pratt says...

I don't think Rails necessarily leads the way in this area. On principle, setting up all the files so they're ready to go makes sense (and should probably be mimicked in Django), but it's also really easy to just ignore it and not test.

Putting in failing tests seems short-sighted as well. I see it being a huge road block to beginners, who will be forced to learn to run before they know how to walk. Though some could argue that learning how to test first might be a good thing.

I think the key is to beef up the documentation. I think a large part of the reason testing is as prevalent as it is in the Rails community is that there's a TON of documentation on testing. Even the pre-eminent beginner's Rails book, Agile Web Development with Rails, has a section on testing in it (sorely missing from the Django Book).

You can't force good behavior, but I think a wealth of documentation on testing would go a long way in encouraging better behavior.

Posted at 3:55 p.m. on November 14, 2008

8 V says...

I think the first question is whom are you writing? From the last part I had the feeling that you would like to convince people who don't do any (or not much) testing today (by extending the docs and raising attention at startapp), and you would like to make it easier as well (by developing tools).

As I am hacking only in my spare time, I will focus exclusively on the "extending the docs" part, and will present it through my own experience to testing. First I didn't knew about it at all (the "you don't know what you don't know phase"), but as I like to read on programming I've read about it as well ("I knew what I don't know"), and after a while I decided to give it a try. It was a failure.

It was a failure, because I was used to design my applications "on paper", and then just code it straight, and simply my thinking missed the idea that I should start with (or rather write simultaneously) the tests as well.

Last summer, I was working with a "real" programmer (real=he has a degree in computer science), and he was amazingly against it, as his thinking followed the same logics then mine. Even though he wrote nice code, that was all. He couldn't find any reason why to get used to testing. So, we went along with our project, without many tests, but thinking about them all the time. Finally, I found two reasons that made me force myself to test, and later came a third one as well.

The first, well known reason in the "literature" is that tests keeps you safe from screwing up your whole application with a small change. Even if this is a well known reason to write tests, I think it can not be overemphasised.

The second, that is actually just a corollay of the first, but I had to figure it out myself, was that testing helps connecting separate modules as well. Especially, when we were working together on distinct parts of the code, we could (have) ensured "good communication" just by using the tests. I think that for open source projects this reason is just as important then the first.

Finally, once I got used to testing came the best reason. I simply feel much more confident about my code, and it seems much quicker to develop a whole application, because it really requires me to think about the exact working of every method, and not just the "design on paper".

Posted at 4:44 p.m. on November 14, 2008

9 Eric Holscher says...

Lots of great replies!

Paul: I did not know about that, is looks really useful!

Julian: Awesome, let me know and I'll link to it or something.

James: I would love to see Pinax lead the way in Testing best practices, as it seems to be leading the way in lots of other best practices as well.

Ben: Yea, Unit tests and flushing are a huge pain. I know there is some talk (and a 1.1 feature proposal) about running them in a transaction, so it would hopefully alleviate those problems. I also agree that fixture data tends to be pretty fragile (even without an often changing schema) due to content types and such. I think a lot of these issues are being addressed with 1.1

yml: Yea, that's along the lines of what I was thinking. I think giving people something in between the output of testmaker, and just a blank file would be useful.

Justin: Yea, I think that this should be done at the very minimum. Perhaps if Django stubbed this out, a third party app (testmaker-lite or a modified pythoscope) could have an option to go in and just stub

Chris: Yea, Rails doesn't necessarily lead the way, it just does something along the lines of what I was proposing. I agree with adding more to the docs, making the testing doc pointed to in more places, and adding testing to the tutorial would be awesome.

V: I agree! Testing is one of those things that seems annoying until you do it, and then you don't know how you lived without it. I think that we should certainly be promoting it to Django people a lot louder than it currently it. As you said, if people don't know about it, how can they do it??

Posted at 4:57 p.m. on November 14, 2008

10 Varikin says...

Writing some good docs on testing would be great. Especially walking through testing the app made in the tutorial. If it is in the tutorial, what about having a bug in the tutorial that is then caught later with testing it? Finding bugs through testing seems to be what convinces testing is worthwhile.

Of course, purposefully putting a bug in the tutorial might not be acceptable, but maybe just a small tiny insignificant bug.

What ever the outcome, I would love to see Django committed to improving the testing framework.

Posted at 5:20 p.m. on November 14, 2008

11 Gregor Müllegger says...

Code coverage would also be a nice feature in the future of django's testsuite. It's always good to now how much code is actually tested (ofcourse code coverage is not a really measure for good tests, but a first hint and gives you bad feelings if there pops up that you only execute 50% of your code in the tests).

Posted at 4:03 p.m. on November 15, 2008

12 pulleytut says...

I am here at a forum newcomer. Until I read and deal with the forum. Let's learn!

Posted at 5:27 p.m. on December 4, 2008

13 ignireeastess says...

TST

Posted at 7:47 a.m. on December 12, 2008

Comments support markdown

Comments are closed.

Comments have been close for this post.

About this post

Posted at 7:43 p.m. on November 13, 2008

Comments: 13

Tags: django , post-a-day , testing

Surfing in Kansas