kmmbvnr / django-jenkins

Plug and play continuous integration with django and jenkins

django-jenkins

Plug and play continuous integration with Django and Jenkins

Installation

From PyPI:

$ pip install django-jenkins

Or by downloading the source and running:

$ python setup.py install

Latest git version:

$ pip install -e git+git://github.com/kmmbvnr/django-jenkins.git#egg=django-jenkins
$ pip install coverage

Installation for Python 3:

Works out of the box

Usage

Add 'django_jenkins' to your INSTALLED_APPS list. Configure Jenkins to run the following command:

$ ./manage.py jenkins --enable-coverage

This will create reports/ directory with junit xml, Coverage and Pylint reports.

For more details see the generic tutorial: https://sites.google.com/site/kmmbvnr/home/django-jenkins-tutorial

Settings

• PROJECT_APPS

  If present, it is supposed to be a list/tuple of django apps for Jenkins to run. Tests, reports, and coverage are generated only for the apps from this list.

• JENKINS_TASKS

  List of Jenkins reporters executed by ./manage.py jenkins command.

  Default value:

  JENKINS_TASKS = ()

• JENKINS_TEST_RUNNER

  The name of the class to use for starting the test suite for jenkins command. Class should be inherited from django_jenkins.runner.CITestSuiteRunner

Reporters

Here is the reporters prebuild with django-jenkins

• django_jenkins.tasks.run_pylint

  Runs Pylint over selected Django apps.

  Task-specific settings: PYLINT_RCFILE

• django_jenkins.tasks.run_pep8

  Runs pep8 tool over selected Django apps. Creates Pylint compatible report for Jenkins

  You should have pep8 python package (>=1.3) installed to run this task.

  Task-specific settings: PEP8_RCFILE

• django_jenkins.tasks.run_pyflakes

  Runs Pyflakes tool over selected Django apps. Creates Pylint compatible report for Jenkins.

  You should have Pyflakes python package installed to run this task.

• django_jenkins.tasks.run_flake8

  Runs flake8 tool over selected Django apps. Creates pep8 compatible report for Jenkins.

  You should have flake8 python package installed to run this task.

Changelog

GIT Version

• csslint/jshint/sccsLint/sloccount task removed (could be used as standalone tools)

Contribution guide

• Set up local jenkins

• Set up django-jenkins:

  npm install jshint
  npm install csslint
  PATH=$PATH:$WORKSPACE/node_modules/.bin
  tox

• Ensure that everything works
• Modify the the only one thing
• Ensure that everything works again
• Fix pep8/pyflakes errors and minimize pylint's warnings
• Pull request!

Authors

Created and maintained by Mikhail Podgurskiy <[email protected]>

Contributors: https://github.com/kmmbvnr/django-jenkins/graphs/contributors

Special thanks, for all github forks authors for project extensions ideas and problem identifications.

django-jenkins's Issues

jenkins can't parse jslint output xml

Hi,

I configured django-jenkins and Jenkins to use jslint support but it always give me the following error:

ERROR: Publisher hudson.plugins.violations.ViolationsPublisher aborted due to exception
hudson.util.IOException2: Cannot parse s3p/reports/jslint.xml
at hudson.plugins.violations.parse.AbstractTypeParser.parse(AbstractTypeParser.java:62)
at hudson.plugins.violations.ViolationsCollector.doType(ViolationsCollector.java:187)
at hudson.plugins.violations.ViolationsCollector.invoke(ViolationsCollector.java:114)
at hudson.plugins.violations.ViolationsCollector.invoke(ViolationsCollector.java:25)
at hudson.FilePath.act(FilePath.java:784)
at hudson.FilePath.act(FilePath.java:766)
at hudson.plugins.violations.ViolationsPublisher.perform(ViolationsPublisher.java:74)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:697)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:672)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:650)
at hudson.model.Build$RunnerImpl.post2(Build.java:162)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:619)
at hudson.model.Run.run(Run.java:1429)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
Caused by: org.xmlpull.v1.XmlPullParserException: entity reference names can not start with character '&' (position: TEXT seen ...=' and instead saw '!='." evidence="\t\t\tif(p.recreateForm===true &&... @38:114)
at org.xmlpull.mxp1.MXParser.parseEntityRef(MXParser.java:2212)
at org.xmlpull.mxp1.MXParser.parseAttribute(MXParser.java:2047)
at org.xmlpull.mxp1.MXParser.parseStartTag(MXParser.java:1799)
at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1127)
at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
at hudson.plugins.violations.parse.AbstractParser.skipToTag(AbstractParser.java:189)
at hudson.plugins.violations.types.jslint.JsLintParser.parseFileElement(JsLintParser.java:42)
at hudson.plugins.violations.types.jslint.JsLintParser.execute(JsLintParser.java:30)
at hudson.plugins.violations.parse.AbstractTypeParser.parse(AbstractTypeParser.java:57)
... 16 more
Finished: FAILURE

use pep8 native jenkins support

Rearranging output to be like pylint not required anymore

Remove stale README.md

Looks like a stale README.md file was introduced in a merge (?).

django-jenkins 0.10 backwards incompatibility with test args --noinput and -v2

Hello,

In django-hudson 0.9 we where able to do the following.

python2.6 manage.py hudson contact signup -v2 --noinput

in django-hudson 0.10 it fails hard with --noinput

Usage: manage.py jenkins [options] [appname ...]
Run CI process
manage.py: error: no such option: --noinput

and -v2 is totally ignored

----------------------------------------------------------------------
Ran 21 tests in 3.725s

OK

csslint error trying to run task

When i configure to run csslint task, I get the following error:

teardown_test_environment
signals.teardown_test_environment.send(sender=self)
File "/home/jenkins/.virtualenvs/django-ca4/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, **named)
File "/home/jenkins/.virtualenvs/django-ca4/local/lib/python2.7/site-packages/django_jenkins/tasks/run_csslint.py", line 51, in teardown_test_environment
csslint_output = check_output([self.intepreter, self.implementation, '--format=%s' % fmt] + files)
File "/home/jenkins/.virtualenvs/django-ca4/local/lib/python2.7/site-packages/django_jenkins/functions.py", line 42, in check_output
raise exception
subprocess.CalledProcessError: Command '['rhino', '/home/jenkins/.virtualenvs/django-ca4/local/lib/python2.7/site-packages/django_jenkins/tasks/csslint/release/csslint-rhino.js', '--format=lint-xml']' returned non-zero exit status 1

Support for coverage for parallel tasks

Allow to configure in hudson downstream builds, and covers it results

run lettuces in jtest command

Need to checked is lettuce enabled and run it in jtest command

Not all applications could be found by jenkins.

I've got the following exception when run manage.py jenkins:
django.core.exceptions.ImproperlyConfigured: App with label utils could not be found

The reason is that 'utils' app does not have 'models.py'. I would not like to create 'models.py' for each of my project application (listed in PROJECT_APPS), e. g. for such apps as 'utils'.

coverage broken if project is an installed package

a)
coverage determines tested apps from given test_labels which is empty by default.

b)
coverage is initialized under the assumption that PROJECTPATH is on PYTHONPATH.

Initializing coverage with test_apps instead of test_labels solves both problems.

diff --git a/django_jenkins/tasks/with_coverage.py b/django_jenkins/tasks/with_coverage.py
index 4f82528..40f9a30 100644
--- a/django_jenkins/tasks/with_coverage.py
+++ b/django_jenkins/tasks/with_coverage.py
@@ -33,7 +33,7 @@ class Task(BaseTask):
         self.html_dir = options['coverage_html_report_dir']

         self.coverage = coverage(branch=options['coverage_measure_branch'],
-                                 source=test_labels or None,
+                                 source=self.test_apps,
                                  config_file=options['coverage_rcfile'] or Task.default_config_path())

     def setup_test_environment(self, **kwargs):

No python24/python25 support

Dies in misterious ways

Your plugin is great, im using it for my projects, it works good in one of them, but i have another similar project and its not working, all i can see is the exit status at 1 but it not shows the reason.

I also try the debug, verbosity and other options but any of them give me hints about whats wrong.

Pylint launched by Django Jenkins ignores files in project

I have registered my app within PROJECT_APPS in settings.py, and it is tested successfully by Jenkins, however pylint doesn't test the other files in the app - forms.py, admin.py etc. it only tests urls.py, views.py and tests.py. Do I have to explicitly make it test the forms, admin etc. files?

$ ls appdir
    admin.py
    forms.py
    models.py
    tests.py
    urls.py
    views.py

Yet only urls, views and tests are pylinted. I'd have expected the default behaviour to test the lot of them. Is there a setting somewhere I'm missing or is this a deficiency?

Thanks,

J

Add django_lint support

jslint reporting ignore .min.js files

Is there a way to make jslint reporting ignore javascript files terminated with .min.js ?

Unable to use custom settings in jenkins task

I'mv trying to launch the manage.py jenkins task using a custom settings module:

(env)niko@NikoBook:~/Sites/taxeo/taxeo$ ./manage.py jenkins --settings=settings_jenkins
Unknown command: 'jenkins'
Type 'manage.py help' for usage.
(env)niko@NikoBook:~/Sites/taxeo/taxeo$ ./manage.py help jenkins
Usage: manage.py jenkins [options] [appname ...]

Run CI process

Options:
  -v VERBOSITY, --verbosity=VERBOSITY
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=all output
  --settings=SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
(<snip/>)
(env)niko@NikoBook:~/Sites/taxeo/taxeo$ ./manage.py jenkins
Creating test database 'default'...

Please note:

• that the settings_jenkins.py file actually exists and contains no python error (works with other django tasks)
• running manage.py jenkins without any option works flawlessly (but uses the settings.py standard module)

Doesn't support django-nose's test generators

In django-nose, a test generator class can't be implemented as a unittest.TestCase-derived class and thus django.test.TestCase. However, django-jenkins re-uses Django's own suite builder which expects test class to be a TestCase so that it's not possible to imlpement test generators in a django-jenkins-based setup.

pylint not recognizing admin 'qs' name as a good-name

Shouldn't "qs" admin site variable be recognized as a good-name like other django related variables (pk, setUp, etc...) in pylint.rc ?
https://docs.djangoproject.com/en/1.3/ref/contrib/admin/#django.contrib.admin.ModelAdmin.queryset
For now, it is marked as an invalid name by pylint

Start url http://tutorial.getwindmill.com

Please remove http://tutorial.getwindmill.com or create a setting for start url.

jenkins windmill not works with FF4

Exception: No default or local profile has been set.

Write some docs :)

It will be great if

a) they are some docs to document all the settings.py flags
b) a sample config on the hudson side.

pep8 support

python-cdb required

Need add python-cdb into install_requires

csslint support

Hi,

It could be nice adding support to csslint.

existing csslib http://cthedot.de/cssutils/

Thanks

testing mail.outbox length fails in django_jenkins but not in direct test

running manage.py test against my app and the test passes, however running manage.py jenkins causes the test to fail. upping the verbosity doesn't yield any feedback.

any thoughts on this?

Add celery support task

https://github.com/ask/django-celery/blob/master/djcelery/contrib/test_runner.py

Or, maybe support test runners as subtask?

No browser information in failed windmill tests

If test fails, there is no info under what browser it runs

No remote windmill service support in windmill task

Need to specify remote windmill server parameters

--with-browser=firefox,192.169.0.2:4444

Ignoring django core in integration

When I get jenkins to run pylint on my project, it looks for violations in all of the imports - including django's own files, which I don't want to check. I have been unable to find any way to get the pylint command to ignore django all together (if calling pylint independently, --ignore=django could be used) however, I can't find a way to get jenkins to use this option when it does its django_jenkins tasks... Is this possible?

Thanks,

J

Extend docs, Fix hudson->jenkins renaming

Extend documentation

• cmd parameters
• settings.py
• windmill support

Lots of junk in coverage report

Getting a lot of junk modules in the coverage report:

.var.lib.jenkins..virtualenvs.coursebook.lib.python2.6.site-packages.south.migration
.var.lib.jenkins..virtualenvs.coursebook.lib.python2.6.site-packages.django.core.mail

And a lot of other assorted Django project modules. Any way to get these excluded like with the tests?

Optionally Disable reports/*.xml generation

This is largely a matter of opinion, but I would like a way to enable / disable reports/*.xml generation, and just run the tests. This runner would otherwise be the same as manage.py jenkins.

Jenkins / CI servers are great, but I'd really like people to test their code, then have Jenkins as a fallback to make sure everything is running smoothly.

Let me know what you think. I'd be happy to set something like this up.

What about a command "alltests" or "jalltests"?

Move windmill task to new project `django-jenkins-js`

pylint UnicodeEncodeError

I was following the tutorial over at:

http://sites.google.com/site/kmmbvnr/home/django-hudson-tutorial/

when I got to the part:

"Now running python manage.py hudson will do the same job as python manage.py test"

my traceback looked like:

pylint/reporters/__init__.py", line 55, in writeln
print >> self.out, string
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbb' 
in position 117: ordinal not in range(128)

A couple of things to note:

• I'm using Mac OS X 10.6.4 Snow Leopard 64-bit built using UCS-4
• The project we're working on uses unittest2

No pure js tests support in windmill task

use jslint native output

It seems jslint.js have no xml output, but it could be implemented

Jenkins Violation plugin have support for jslint4java xml format.

fix want_module in coverage

When app name is the substring of project name, the want_module function allows top project modules, such as settings.py

Ignore migrations by default

Hi, when a django project uses South it creates a lot of files in migrations folder, they should be ignored by default. What do you think?

Invalid Syntax Error

Exception:

Traceback (most recent call last):
  File "./bin/django", line 27, in <module>
    djangorecipe.manage.main('testapp.settings')
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/eggs/djangorecipe-0.20-py2.5.egg/djangorecipe/manage.py", line 16, in main
    management.execute_manager(mod)
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/parts/django/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/parts/django/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/parts/django/django/core/management/__init__.py", line 257, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/parts/django/django/core/management/__init__.py", line 67, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/parts/django/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/var/lib/hudson/jobs/django-cms-new/workspace/tests/eggs/django_hudson-0.9.0-py2.5.egg/django_hudson/management/commands/hudson.py", line 52
    output_file=path.join(output_dir,'pylint.report'))
              ^
SyntaxError: invalid syntax

Document PROJECT_APPS setting.

It would be nice to be able to ignore certain patterns of directory and filenames, based on either glob or regex patterns. Ideally, this would be a setting.

For instance, it's not very helpful to see how well the Django project and all my third-party packages adhere to pylint's standards. It just adds noise to the report.

document how to pass options to tasks

e.g. How do I pass a --pylint-rcfile option to the run_pylint task?

Sorry to file this as an issue when it's really a support request. Where should I be posting this to?

run_jslint task doesn't find any files unless ...

diff --git a/django_jenkins/tasks/run_jslint.py b/django_jenkins/tasks/run_jslint.py
index 634f0ae..1cfd7bb 100644
--- a/django_jenkins/tasks/run_jslint.py
+++ b/django_jenkins/tasks/run_jslint.py
@@ -65,7 +65,8 @@ class Task(BaseTask):
             from django.contrib.staticfiles import finders

             for finder in finders.get_finders():
-                for path, _ in finder.list(self.exclude):
+                for path, storage in finder.list(self.exclude):
+                    path = os.path.join(storage.location, path)
                     if path.endswith('.js') and in_tested_locations(path):
                         yield path
         else:

path returned by finder.list() contains only the relative path to the static dir (eg.: js/foo.js) so you have to join it with the storage's location (eg.: app_dir/static). if you don't do that the in_tested_locations() check will always returns False.

Command style attribute error

If you override the __init__ method of BaseCommand, you must still call the BaseCommand constructor. When Django attempts to output an error (raised by CommandError) to the screen, it attempts to style it. Errors will be printed in bold red text or something similar based on your terminal settings.

In some cases:
AttributeError: ‘Command’ object has no attribute ‘style’

You need the following in your TaskListCommand constructor:

class TaskListCommand(BaseCommand):
def init(self):
# Call the BaseCommand constructor
super(TaskListCommand, self).init()

    # ...

Jenkins tasks problem

Hi, when i try to set this setting:

JENKINS_TASKS = (
'django_jenkins.tasks.run_pylint',
'django_jenkins.tasks.with_coverage',
)

and run ./manage.py jenkins it exit with code '1' and no output to console.

I just dont need to run tests.

DjangoTestRunner is deprecated post-1.3

Post-1.3, DjangoTestRunner is deprecated.

The deprecation warning states:

"DjangoTestRunner is deprecated; it's functionality is indistinguishable from TextTestRunner"

Great project, by the way. Make's Jenkin's setup a breeze.

cannot import name _strclass

Hi kmmbvnr,
I get the traceback below when I use django-jenkins with python 2.7.
Thanks
Michael

Removing the import gets things working again.

Traceback (most recent call last):
File "bin/django", line 50, in
djangorecipe.manage.main('lfs_project.settings')
File "/home/mick/src/lfs-buildout-development/eggs/djangorecipe-0.99-py2.7.egg/djangorecipe/manage.py", line 16, in main
management.execute_manager(mod)
File "/home/mick/src/lfs-buildout-development/eggs/Django-1.3-py2.7.egg/django/core/management/init.py", line 438, in execute_manager
utility.execute()
File "/home/mick/src/lfs-buildout-development/eggs/Django-1.3-py2.7.egg/django/core/management/init.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/mick/src/lfs-buildout-development/eggs/Django-1.3-py2.7.egg/django/core/management/init.py", line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/mick/src/lfs-buildout-development/eggs/Django-1.3-py2.7.egg/django/core/management/init.py", line 68, in load_command_class
return module.Command()
File "/home/mick/src/lfs-buildout-development/eggs/django_jenkins-0.11.0-py2.7.egg/django_jenkins/management/commands/init.py", line 48, in init
self.tasks_cls = [import_module(module_name).Task for module_name in self.get_task_list()]
File "/home/mick/src/lfs-buildout-development/eggs/Django-1.3-py2.7.egg/django/utils/importlib.py", line 35, in import_module
import(name)
File "/home/mick/src/lfs-buildout-development/eggs/django_jenkins-0.11.0-py2.7.egg/django_jenkins/tasks/windmill_tests.py", line 10, in
from unittest import _strclass
ImportError: cannot import name _strclass

tasks.django_tests not honors PROJECT_APPS settings

jslint_runner.js missing from setup.py

... and thus is not installed and cannot be found by rhino....

without tests in project jenkins exit with code 1

project with no tests fail with exit code 1 and no output to console.

jslint - *.js files in one location

Thanks for great job with django-jenkins.

I would love to see in django-jenkins an option, to tell run_jslint to check all *.js files in the media directory.

Start coverage earlier in the process

coverage reports currently have a lot of noise for files like models.py which are loaded before django-jenkins can start the coverage report. The best solution for this would probably be something like a separate "django-jenkins.py" script which could start coverage before doing anything which will cause Django to load modules.