garethr / django-test-extensions

A set of custom assertions and examples for use testing django applications

PyUnit provides a basic set of assertions which can get you started with unit testing python, but it’s always useful to have more. Django also has a few specific requirements and common patterns when it comes to testing. This set of classes aims to provide a useful starting point for both these situations.

The application also overrides the default Django test runner, adding a few useful features:

Installation

Just add the project to your INSTALLED_APPS.

INSTALLED_APPS = (
	'test_extensions',
)

Note that this application steals the test command from django, overriding it with extra toys. If another application in your INSTALLED_APPS does this too then the last one in the list will win. South migrations does this in order to use the django syncdb command for testing, which test_extensions does too. As of 0.4 test_extensions works with South, just as long as you include it after south in the list of installed apps.

Assertions

See the examples directory in the src/test_extensions directory for details of a large number of useful assertions for testing django apps:

• assert_response_contains
• assert_response_doesnt_contain
• assert_regex_contains
• assert_render_matches
• assert_code
• assert_render
• assert_render_matches
• assert_doesnt_render
• assert_render_contains
• assert_render_doesnt_contain

Test Runners

XMLUnit

Sometimes it’s nice to have a file reporting the results of a test run. Some applications such as CruiseControl can use this to display the results in a user interface.

python manage.py test --xml

Code Coverage

If you want to know what code is being run when you run your test suite then codecoverage is for you. These two flags use two different third party libraries to calculate coverage statistics. The first dumps the results to stdout, —xmlcoverage creates a cobertura-compatible xml output, and the last one creates a series of files displaying the results.

python manage.py test --coverage

python manage.py test --xmlcoverage

python manage.py test --figleaf

No Database

Sometimes your don’t want the overhead of setting up a database during testing, probably because your application just doesn’t use it.

python manage.py test --nodb

python manage.py test --nodb --coverage

python manage.py test --nodb --xmlcoverage

WARNING Don’t use this if you use the ORM in your app. An outstanding issue means that you can get into trouble. Your tests will still hit the database, but it will be your non test data.

Local Continuous Integration Command

Thanks to Roberto Aguilar (http://github.com/rca) for providing a auto-reloading version of the test runner. Run the runtester command and it should run your test suite whenever you change a file (similar to how runserver reloads the server each time you change something.)

See this thread
from the Django Developer list of more information and discussion.

python manage.py runtester

Licence

XMLUnit is included out of convenience. It was written by Marc-Elian Begin <[email protected]> and is Copyright © Members of the EGEE Collaboration. 2004. http://www.eu-egee.org

The rest of the code is licensed under an MIT license:

Copyright © 2008 Gareth Rushgrove <[email protected]>

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the “Software”), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

django-test-extensions's Issues

In xmloutput runner, stacktraces from exceptions do not include line endings

Stacktraces generated during tests when using the xmloutput runner do not appear to include line endings, which makes them difficult to read.

Figleaf Coverage directory creation

If you don't have a directory temp/figleaf it will throw an error:

OSError: [Errno 2] No such file or directory: 'temp/figleaf'

Fix is very simple :)

diff --git a/src/test_extensions/testrunners/figleafcoverage.py b/src/test_extension /testrunners/figleafcoverage.py
index 7a20e25..565aa05 100755
--- a/src/test_extensions/testrunners/figleafcoverage.py
+++ b/src/test_extensions/testrunners/figleafcoverage.py
@@ -11,7 +11,7 @@ def run_tests(test_labels, verbosity=1, interactive=True, extra_tests=[]):
     figleaf.start()
     test_results = django_test_runner(test_labels, verbosity, interactive, extra_tests)
     figleaf.stop()
-    if not os.path.isdir(os.path.join("temp", "figleaf")): os.mkdir(os.path.join("temp", "figleaf"))
+    if not os.path.isdir(os.path.join("temp", "figleaf")): os.makedirs(os.path.join("temp", "figleaf"))
     file_name = "temp/figleaf/test_output.figleaf"
     figleaf.write_coverage(file_name)
     output = commands.getoutput("figleaf2html " + file_name + " --output directory=temp/figleaf")

'test --coverage' mode causes external modules to fail

possibly because it tries to interpret every source file (django-haystack fails with a missing dependency Exception for a backend i haven't configured). Is there a way to fix that ?

modules causing problems:

• django.contrib.gis (not finding GEOSPrepare in the c library)
• django-haystack (trying to find the woosh backend even when solr is configured)
• django-paypal

Runing tests with --xml fails with UnicodeEncodeError

This runs correctly without --xml

 Traceback (most recent call last):
   File "./manage.py", line 11, in <module>
     execute_manager(settings)
   File "/usr/lib/pymodules/python2.6/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/lib/pymodules/python2.6/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/pymodules/python2.6/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/pymodules/python2.6/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.6/dist-packages/django_test_extensions-0.9-py2.6.egg/test_extensions/management/commands/test.py", line 125, in handle
    failures = test_runner(**test_options).run_tests(test_labels)
  File "/usr/lib/pymodules/python2.6/django/test/simple.py", line 314, in run_tests
    result = self.run_suite(suite)
  File "/usr/local/lib/python2.6/dist-packages/django_test_extensions-0.9-py2.6.egg/test_extensions/testrunners/xmloutput.py", line 12, in run_suite
    return XMLTestRunner(verbosity=self.verbosity).run(suite)
  File "/usr/local/lib/python2.6/dist-packages/django_test_extensions-0.9-py2.6.egg/test_extensions/testrunners/xmlunit/unittest.py", line 815, in run
    self._writeReport(result,self.totalTime)
  File "/usr/local/lib/python2.6/dist-packages/django_test_extensions-0.9-py2.6.egg/test_extensions/testrunners/xmlunit/unittest.py", line 833, in _writeReport
    self.output.write(self.stdout.read())
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 569: ordinal not in range(128)

--nodb not safe for database-backed apps!

If you happen to run tests with the --nodb option on an app that does in fact use the ORM to do CRUD operations on your database, it WILL corrupt whatever your default (not test) database is. You can test this by making a simple app, let's call it "myapp", with a single model, set up to use a SQLite3 database:

from django.db import models

class MyTable(models.Model):
    a = models.IntegerField(primary_key=True)

And a single test:

from django.test import TestCase
from myapp.models import MyTable

class SimpleTest(TestCase):
    def test_blow_away_everything(self):
        MyTable.objects.all().delete()

Now run a syncdb and then manually insert a few rows into "myapp_mytable", and pretend this is your production DB. Run ./manage.py test normally and it is fine, of course. But run ./manage.py test --nodb; all your rows will have been deleted.

I presume this happens because --nodb doesn't do anything to prevent database access, nor does it set up a test database to replace/act as a proxy for your dev/production/whatever database.

Oddly enough, the problem does not seem to exist if using django.db.connection directly, instead of the ORM. I haven't looked into why that is.

I'd suggest 1) a strongly worded warning in the README and anywhere else appropriate, not to use this option, ever, on a database-backed app, 2) if possible, a modification to disable the ORM or access to the default database with this option turned on. Perhaps the runner could just quit if it encounters a test trying to use the ORM/database. --nodb is very useful for certain sets of tests that don't hit the database, but the user would need to make sure they specify the particular tests to run correctly every time, or else suffer potentially dire consequences. (As I unfortunately did.)

Does it not run with Django 1.4.10?

Does it not run with Django 1.4.10 and Python 2.6?

/env/lib/python2.7/site-packages/test_extensions/testrunners/codecoverage.py", line 6, in
from django.test.simple import run_tests as django_test_runner
ImportError: cannot import name run_tests

--coverage displays coverage report twice

When using --coverage, cov.report() is called twice.

xml test runner fails unless old-style DATABASE_* settings are used

The xml test runner depends on the DATABASE_NAME setting. In 1.2 this is pending deprecation in favor of DATABASES.

SKIP_TESTS only examines last part of app name

When you have 'django.contrib.admin' in the SKIP_TESTS setting, it still runs tests on this app. You need to just have 'admin'.

The method of identifying which apps to skip should mirror the way apps are identified in INSTALLED_APPS.

EXCLUDE_FROM_COVERAGE only looks at first part of module name

I like to exclude the django.contrib stuff from my coverage testing. If you put the 'django.contrib.*' stuff into this settings variable, they still get coverage tested. If you just use 'django', then they are not tested:

codecoverage.py line 37 only looks at the first part of the path.

coverage.CoverageException: Only one coverage object allowed.

After fixing issue 9 (see 568d192), I still get this traceback when using --coverage:

$ ./manage.py test --coverage
Traceback (most recent call last):
  File "testproject/manage.py", line 11, in <module>
    execute_manager(settings)
  File "django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 218, in execute
    output = self.handle(*args, **options)
  File "test_extensions/management/commands/test.py", line 123, in handle
    failures = test_runner(test_labels, **test_options)
  File "test_extensions/testrunners/codecoverage.py", line 96, in run_tests
    cov = coverage.coverage()
  File "lib/python2.6/site-packages/coverage-2.85-py2.6.egg/coverage.py", line 305, in __init__
    raise CoverageException("Only one coverage object allowed.")
coverage.CoverageException: Only one coverage object allowed.

This is Django 1.2.1 with django-test-extensions a5ef307 patched with 568d192.

I am using django-test-extensions for coverage, but the coverage score is incorrect

Hi,

Here is my issue:

http://bitbucket.org/ned/coveragepy/issue/45/when-i-run-my-tests-imports-statement-and-function-declaration-are-not-counted-as

No such file or directory: 'temp/xml' when running tests with --xml option

When I run ./manage test --xml APP, without a temp directory, I get the following error:

OSError                                   Traceback (most recent call last)
OSError: [Errno 2] No such file or directory: 'temp/xml'

The problem is in src/test_extensions/testrunners/xmlunit/unittest.py . I tries to create multiple dirs, but is using the wrong function. Line 787 looks like:

if not os.path.isdir(os.path.join("temp", "xml")): os.mkdir(os.path.join("temp", "xml"))

It should be using makedirs instead:

if not os.path.isdir(os.path.join("temp", "xml")): os.makedirs(os.path.join("temp", "xml"))

Unit test times wrong?

Hudson log reports Ran 182 tests in 39.448s
Hudson test result page shows (root) 2.3 sec 1 0 182

xml file shows
testsuite errors="0" failures="1" name="" tests="182" time="39.448">
but individual test times are way too short

I suppose the unit test xml file somehow contains invalid times for individual tests.

runtester fails with Django 1.2

This is due to the support for quitting tests using a signal handler: signal handlers cannot be installed into threads other than the main thread.

codecoverage causes re-registration of admin models

runtester fails with Django 1.3

The command fail with message: ValueError: signal only works in main thread.

Failfast always enabled

With the django 1.2 (trunk) test runner, and test_extensions, there is an option called failfast: with test_extensions, this appears to always be enabled.

IndexError when running test runner with option --xml

Hi,

I have IndexError when i try to run testrunner with option --xml
I have tested it on django 1.1 and 1.2.

File "bin/django-1.2", line 24, in
djangorecipe.manage.main('example_project.development')
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/eggs/djangorecipe-0.20-py2.6.egg/djangorecipe/manage.py", line 16, in main
management.execute_manager(mod)
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/parts/django-1.2/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/parts/django-1.2/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/parts/django-1.2/django/core/management/base.py", line 191, in run_from_argv
self.execute(_args, *_options.dict)
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/parts/django-1.2/django/core/management/base.py", line 218, in execute
output = self.handle(_args, *_options)
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/eggs/django_test_extensions-0.7-py2.6.egg/test_extensions/management/commands/test.py", line 66, in handle
interactive=interactive)
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/eggs/django_test_extensions-0.7-py2.6.egg/test_extensions/testrunners/xmloutput.py", line 32, in run_tests
result = XMLTestRunner(verbosity=verbosity).run(suite)
File "/Users/MacBookPro/Documents/Aptana Studio Workspace/django-moderation/eggs/django_test_extensions-0.7-py2.6.egg/test_extensions/testrunners/xmlunit/unittest.py", line 808, in run
fileName = "%s" % string.split("%s" % t._tests[0]._tests[0].class,'.')[0]
IndexError: list index out of range

ImportError: cannot import name run_tests

I've installed the app and when I tried python manage.py test --coverage I got the following error:

ImportError: cannot import name run_tests

It seems the cause of the problem is at this file: https://github.com/garethr/django-test-extensions/blob/master/src/test_extensions/testrunners/codecoverage.py

from django.test.simple import run_tests as django_test_runner

Apparently django doesn't have run_tests function anymore. Instead they encapsulated it with DjangoTestSuiteRunner class.

It'd be great if anyone fix this. Btw, I use Django 1.5