spotify / dh-virtualenv

Python virtualenvs in Debian packages

dh-virtualenv

Contents

• Overview
• Presentations, Blogs & Other Resources
• Using dh-virtualenv
• How does it work?
• Running tests
• Building the package in a Docker container
• Building the documentation locally
• Releasing a new version
• Code of conduct
• License

Overview

dh-virtualenv is a tool that aims to combine Debian packaging with self-contained virtualenv based Python deployments.

The idea behind dh-virtualenv is to be able to combine the power of Debian packaging with the sandboxed nature of virtualenvs. In addition to this, using virtualenv enables installing requirements via Python Package Index instead of relying on the operating system provided Python packages. The only limiting factor is that you have to run the same Python interpreter as the operating system.

For complete online documentation including installation instructions, see the online documentation.

Presentations, Blogs & Other Resources

Here are a few external resources that can help you to get a more detailed first impression of dh-virtualenv, or advocate its use in your company or project team.

• How We Deploy Python Code – Building, packaging & deploying Python using versioned artifacts in Debian packages at Nylas.
• DevOps Tool Bazaar – Slide deck presented at the DevOps Karlsruhe Meetup in February 2018, regarding Python software deployment for Debian with a practical example.
• The Architecture of Open Source Applications: Python Packaging – This provides a lot of background (and possibly things you didn't know) about the Python side of packaging.

Using dh-virtualenv

Using dh-virtualenv is fairly straightforward. First, you need to define the requirements of your package in requirements.txt file, in the format defined by pip.

To build a package using dh-virtualenv, you need to add dh-virtualenv in to your build dependencies and write following debian/rules file:

  %:
          dh $@ --with python-virtualenv

Note that you might need to provide additional build dependencies too, if your requirements require them.

Also, you are able to define the root path for your source directory using --sourcedirectory or -D argument:

  %:
          dh $@ --with python-virtualenv --sourcedirectory=root/srv/application

NOTE: Be aware that the configuration in debian/rules expects tabs instead of spaces!

Once the package is built, you have a virtualenv contained in a Debian package and upon installation it gets placed, by default, under /opt/venvs/<packagename>.

For more information and usage documentation, check the accompanying documentation in the doc folder, also available at Read the Docs.

How does it work?

To do the packaging, dh-virtualenv extends debhelper's sequence by inserting a new dh_virtualenv command, which effectively replaces the following commands in the original sequence:

• dh_auto_clean
• dh_auto_build
• dh_auto_test
• dh_auto_install
• dh_python2
• dh_pycentral
• dh_pysupport

In the new sequence, dh_virtualenv is inserted right before dh_installinit.

Running tests

$ nosetests ./test/test_deployment.py

Building the package in a Docker container

To build dh-virtualenv itself in a Docker container, call docker build --tag dh-venv-builder .. This will build the DEB package for Debian stable by default. Add e.g. --build-arg distro=ubuntu:bionic to build for Ubuntu LTS instead.

The resulting files must be copied out of the build container, using these commands:

mkdir -p dist && docker run --rm dh-venv-builder tar -C /dpkg -c . | tar -C dist -xv

There is also a short-cut for all this, in the form of invoke bdist_deb [--distro=‹id›:‹codename›].

Building the documentation locally

If you execute the following commands in your clone of the repository, a virtualenv with all necessary tools is created. invoke docs then builds the documentation into doc/_build/.

command . .env --yes --develop
invoke docs

To start a watchdog that auto-rebuilds documentation and reloads the opened browser tab on any change, call invoke docs -w -b (stop the watchdog using the -k option).

Releasing a new version

Follow these steps when creating a new release:

1. Check version in dh_virtualenv/_version.py and debian/changelog.
2. Make sure doc/changes.rst is complete.
3. Bump release date in debian/changelog (dch -r).
4. Tag the release and git push --tags.
5. Edit release entry on GitHub (add changes).
6. Update the Ubuntu PPA.
7. Bump to next release version in dh_virtualenv/_version.py.
8. Open new section in debian/changelog (with …-0.1+dev added).
9. Open a new section in doc/changes.rst, so it can be maintained as features are added!

Code of conduct

This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

License

Copyright (c) 2013-2017 Spotify AB

dh-virtualenv is licensed under GPL v2 or later. Full license is available in the LICENSE file.

dh-virtualenv's Issues

Empty packages

Hello,

I'm trying to make à package with dh-virtualenv but at the end the package are always empty :

(venv)xmppserialcom|master⚡ ⇒ dpkg -c ../s2x_0.1-1_all.deb
drwxr-xr-x root/root         0 2014-02-12 17:09 ./
drwxr-xr-x root/root         0 2014-02-12 17:09 ./usr/
drwxr-xr-x root/root         0 2014-02-12 17:09 ./usr/share/
drwxr-xr-x root/root         0 2014-02-12 17:09 ./usr/share/doc/
drwxr-xr-x root/root         0 2014-02-12 17:09 ./usr/share/doc/s2x/
-rw-r--r-- root/root       137 2014-02-06 17:39 ./usr/share/doc/s2x/changelog.Debian.gz

(venv)xmppserialcom|master⚡ ⇒ ls
bin    config.yaml  requirements.txt  s2x_0.1-1.dsc     setup.py
build  debian       s2x               s2x_0.1-1.tar.gz  venv

I have the issue on raspian (debian 7 on raspberry pi) and on ubuntu 13.04.

cannot write to /root/.ssh

When running dpkg-buildpackage (as an unprivileged user) with git+ssh URLs in requirements.txt, the following error is generated:

Could not create directory '/root/.ssh'.

pip should be using the user's .ssh directory, not root's.

version info:
dh-virtualenv: 0.6
python: 2.7
pip: 1.0
setuptools: 2.0.1

Missing package

i creates a debian package using your tool.
The content of this debian package is available here [1]. My entry point is well generated:

$ cat /usr/share/python/nimbus-stats/bin/nimbus-stats-monthly 
#!/usr/share/python/nimbus-stats/bin/python
# PBR Generated from u'console_scripts'

import sys

from nimbusstats.monthly import main


if __name__ == "__main__":
    sys.exit(main())

If i try the script :

$ /usr/share/python/nimbus-stats/bin/nimbus-stats-monthly 
Traceback (most recent call last):
  File "/usr/share/python/nimbus-stats/bin/nimbus-stats-monthly", line 6, in <module>
    from nimbusstats.monthly import main
  File "/usr/share/python/nimbus-stats/local/lib/python2.7/site-packages/nimbusstats/monthly.py", line 12, in <module>
    import datetime
ImportError: No module named datetime

datetime is not found in python interpreter :

$ /usr/share/python/nimbus-stats/bin/python
Python 2.7.3 (default, Mar 13 2014, 11:03:55) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named datetime
>>> 
$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> 

Any idea ?

[1] : https://gist.github.com/nlamirault/32e1679c9616a30a33d4

Trouble building Numpy with dh-virtualenv

Disclaimer: My Debian Package knowledge is super weak.

Context: I can do the following:
$ sudo pip install numpy (on the system python)
$ pip install numpy (on a new blank virtualenv)

But I cannot get numpy to build within dh-virualenv

Version: git clone (3-31-15)
Machine: 14.04 Ubuntu (trusty)

Replication:

1. Follow the instructions here http://dh-virtualenv.readthedocs.org/en/latest/tutorial.html
2. put 'numpy' in the requirements.txt file
3. dpkg-buildpackage -us -uc

Error:

/usr/bin/gfortran -Wall -g -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o -L/usr/lib/atlas-base/atlas -L/usr/lib/atlas-base -Lbuild/temp.linux-x86_64-2.7 -llapack -lf77blas -lcblas -latlas -lgfortran -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 0 has invalid symbol index 11
/usr/bin/ld: /usr/lib/debug/usr/lib/x86_64-linux-gnu/crt1.o(.debug_info): relocation 1 has invalid symbol index 12
...
/usr/lib/gcc/x86_64-linux-gnu/4.9/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'

So I noticed that the FLAGS to gfortran were different, the first two that worked had these flags

/usr/bin/gfortran -Wall -g -Wall -g -shared 
 build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o 
 build/temp.linux-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o 
 -L/usr/lib/atlas-base/atlas 
 -L/usr/lib/atlas-base -Lbuild/temp.linux-x86_64-2.7 
 -llapack -lf77blas -lcblas -latlas -lgfortran 
 -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so

The dh-virtualenv version had different FLAGS

 /usr/bin/gfortran -Wall -g -Wl,-Bsymbolic-functions -Wl,-z,relro 
 build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o
 build/temp.linux-x86_64-2.7/numpy/linalg/lapack_lite/python_xerbla.o 
 -L/usr/lib/atlas-base/atlas 
 -L/usr/lib/atlas-base 
 -Lbuild/temp.linux-x86_64-2.7 
 -llapack -lf77blas -lcblas -latlas -lgfortran 
 -o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so

Any suggestions are greatly appreciated :)

Upload to Debian

Hi!

Would you be interested to upload dh-virtualenv to Debian? I could sponsor an upload if you are interested.

Create PPA for Ubuntu Trusty

In order for it to be easier to install 0.7 (and newer) to Trusty, create a PPA for that. Maybe even support Precise.

Add 0.7 to backports for Debian Wheezy

To make all fixes in 0.7 and future versions available, add the 0.7 to backports for wheezy.

Investigate use of pyenv to embed what ever Python version you want

In order not to be confined with system Python interpreters, investigate the possibility of using a tool like pyenv to integrate full python interpreter within dh-virtualenv packages

Consider reusing libs on current virtualenv

For projects with many libs, the time for installing libs how: numpy and scipy is long time!
Consider use libs on current virtualenv and no compiling all requirements on project again.

Tag 0.8.1 release

Please and thank you! :)

Consider changing install location

Currently dh-virtualenv installs packages into /usr/share/python/$packagename. As far as I understand /usr/share/python is for system python packages only and other packages should be installed somewhere else (/usr/share/$packagename maybe?).

Installing into /usr/share/python may break system python packages.

For example, I have run into this issue:

I have a dh-virtualenv based package that includes gunicorn. When it is installed, python-minimal package cannot be upgraded or reinstalled:

Setting up python-minimal (2.7.3-0ubuntu2.2) ...
dpkg: error processing python-minimal (--configure):
 subprocess installed post-installation script returned error exit status 1

The reason is this line in python-minimal.postinst script:

python2.7 -m compileall /usr/share/python/ >/dev/null

Which fails with this:

[...]
Compiling /usr/share/python/<my-package-name>/lib/python2.7/site-packages/gunicorn/workers/_gaiohttp.py ...
SyntaxError: ('invalid syntax', ('/usr/share/python/<my-package-name>/lib/python2.7/site-packages/gunicorn/workers/_gaiohttp.py', 68, 26, '                yield from self.wsgi.close()\n'))
[...]

To upgrade python-minimal package I need to remove my package first and then reinstall it after. Which is neither obvious nor convenient.

How to include config files?

I want to build a .deb file for a Django project that includes uWSGI and its config files. I'd like to include the uWSGI ini file in /etc/uwsgi/package.ini but can't get this working with dh-virtualenv. Please can you explain how to do this?

I've tried including it in debian/etc/uwsgi/ but it doesn't install - I've also tried adding it to debian/conffiles and debian/install but without success.

Apologies for the dumb question.

Install requirements only from local links

In our company we are installing dependencies from an internal HTTP directory, in order to avoid failures because of network issues or packages no more available.

To rely only on the internal repository and never download anything from the external network, we are executing virtualenv and pip with the following options:

virtualenv --never-download 
pip install --no-index --no-dependencies --find-links=$REPO

Could it be helpful to add an option (such as --local-links or whatever) which will activate all of the above options?

If helpful, I may write a patch.

Bad VIRTUAL_ENV path in installed activate scripts

Resulting activate script (and activate.csh, activate.fish) contains VIRTUAL_ENV variable prefixed with the path where the script was built. Instead that variable should be set to the installation root, as set with $DH_VIRTUALENV_INSTALL_ROOT/my-package (or /usr/share/python/my-package by default)

For example if setup.py for my-package is in /home/agilevic/hgdev/my-package, running dpkg-buildpackage -us -uc results in activate script's VIRTUAL_ENV set to /home/agilevic/hgdev/my-package/debian/my-package/usr/share/python/my-package; should be VIRTUAL_ENV=/usr/share/python/my-package.

Resulting virtualenvs are unusable.

Creating a .desktop app

dh-virtualenv is working very well for packaging python projects so thanks very much.

I would now like to include a desktop shortcut to my app using a .deskop file. What is the best way to include this with dh-virtualenv?

My knowledge of debian packaging is very limited.

help supervising a modification to stdeb to directly integrate dh-virtualenv

astraw/stdeb#94

I am begining to play with stdeb and dh_virtualenv I have the feeling both can integrate fairly well.

I am not sure of my code though.

Tutorial

First off, this is awesome!

I was curious if you could provide a basic tutorial on how to package something using dh_virtualenv. I know for a lot of people Debian Python packaging is already quite confusing and this adds something to the mix. Perhaps use a simple application like the Django tutorial?

If you happen to have some up to date documentation (or know of) on Debian Python packaging I think that would be very helpful too to a lot of people.

Trigger virtualenv recreation on Python updates

virtualenvs tend to break horribly after certain host Python updates – those that change internal, statically linked modules, or make symlinked .so modules incompatible to the old "python" binary contained in the virtualenv.

Just putting the idea out there, I might do it if and when I find the time.

Package build fail in pbuilder

Hi,

I'm currently working on a package. The build is fine using dpkg-buildpackage (on debian sid) but it fails when I'm building the package using gbp and pbuilder (target is Debian Wheezy).

It seems pip is failling during logging:

  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/backwardcompat.py", line 98, in fwrite
    f.write(s)
ValueError: I/O operation on closed file

but the error seems to be when fetching pypi page:

Downloading/unpacking Cheetah==2.4.4 (from -r ./requirements.txt (line 1))
  Cannot fetch index base URL http://pypi.python.org/simple/

When manually running the pip install command in the pbuilder, I manage to install the pip package.

I've got the following error:

New python executable in debian/scirius-pro/usr/share/python/scirius-pro/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
Downloading/unpacking Cheetah==2.4.4 (from -r ./requirements.txt (line 1))
  Cannot fetch index base URL http://pypi.python.org/simple/
  Could not find any downloads that satisfy the requirement Cheetah==2.4.4 (from -r ./requirements.txt (line 1))
No distributions at all found for Cheetah==2.4.4 (from -r ./requirements.txt (line 1))
Storing complete log in /tmp/buildd/.pip/pip.log
Traceback (most recent call last):
  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/bin/pip", line 9, in <module>
    load_entry_point('pip==1.1', 'console_scripts', 'pip')()
  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/__init__.py", line 116, in main
    return command.main(args[1:], options)
  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/basecommand.py", line 140, in main
    logger.fatal('Storing complete log in %s' % log_fn)
  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/log.py", line 50, in fatal
    self.log(self.FATAL, msg, *args, **kw)
  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/log.py", line 77, in log
    pip.backwardcompat.fwrite(consumer, rendered)
  File "/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg/pip/backwardcompat.py", line 98, in fwrite
    f.write(s)
ValueError: I/O operation on closed file
Traceback (most recent call last):
  File "/usr/bin/dh_virtualenv", line 101, in <module>
    sys.exit(main() or 0)
  File "/usr/bin/dh_virtualenv", line 81, in main
    deploy.install_dependencies()
  File "/usr/lib/python2.7/dist-packages/dh_virtualenv/deployment.py", line 131, in install_dependencies
    subprocess.check_call(self.pip('-r', requirements_path))
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/bin/python', '/tmp/buildd/scirius-pro-0.9.8/debian/scirius-pro/usr/share/python/scirius-pro/bin/pip', 'install', '--log=/tmp/tmplhhr7e', '-r', './requirements.txt']' returned non-zero exit status 1

Thanks for any fix, tips or idea.

Increase a build speed by caching virtualenv dependencies

It takes long when requirements for the project include several binary packages, like uwsgi, crypto and postgresql. Build time significantly increase.

What if there would be some way to eliminate re-building from scratch the all those things each time by using for instance https://github.com/paylogic/pip-accel?

Add the ability to use dh-virtualenv with Pythons that were built using pyenv/python-build

This will allow deploying with upgraded Python versions, different configurationsm patches etc.

dh-virtualenv should cache package downloads

My package have about 10 pypi dependencies. Each time I build same files are downloaded from pypi over and over again. This increase deb building some somewhat.

It would be nice if dh-virtualenv could cache downloaded packages and reuse them in subsequent builds.

dh_virtualenv doesn't honor dh's -a option

Related Debian bug and description of the bug here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=738964

In a nutshell, dh_virtualenv should expect to be passed -a and -i options. It should probably just ignore them.

Release new version to publish -S/--use-system-packages

Would you mind tagging a new version that would contains the recent "-S/--use-system-packages" feature? Once tagged, the Debian package can be easily refreshed and thus the change be made available easily to the whole community.

Python3 compat

Any words on Python3 projects? Any known issues or recommendations?

Best

dh: unable to load addon python-virtualenv

I'm attempting to follow your tutorial with little success. So I wasn't entirely sure what's the best way to install dh-virtualenv first, so I've just installed it into my home directory:

┌─[~]
└─▪ pip install --user git+https://github.com/spotify/dh-virtualenv
Downloading/unpacking git+https://github.com/spotify/dh-virtualenv
  Cloning https://github.com/spotify/dh-virtualenv to /tmp/pip-i1waBN-build
  Running setup.py egg_info for package from git+https://github.com/spotify/dh-virtualenv

Installing collected packages: dh-virtualenv
  Running setup.py install for dh-virtualenv
    changing mode of build/scripts-2.7/dh_virtualenv from 664 to 775

    changing mode of /home/dgillies/.local/bin/dh_virtualenv to 775
Successfully installed dh-virtualenv
Cleaning up...

So then I create a directory and add my requirements.txt the contents of:

sentry

A debian directory in it with some basics:

─[python-virtualenv-sentry]
└─▪ ls debian/
changelog  compat  control  rules  source

debian/rules looks like as per the docs:

#!/usr/bin/make -f
%:
        dh $@ --with python-virtualenv

Attempt to build, but then get:

┌─[python-virtualenv-sentry]
└─▪ dpkg-buildpackage -uc -us
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor): -D_FORTIFY_SOURCE=2
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor): -Wl,-Bsymbolic-functions -Wl,-z,relro
dpkg-buildpackage: source package python-virtualenv-sentry
dpkg-buildpackage: source version 6.3.0-1
dpkg-buildpackage: source changed by Dave <dg@foobarbash>
dpkg-buildpackage: host architecture amd64
 dpkg-source --before-build python-virtualenv-sentry
 fakeroot debian/rules clean
dh clean --with python-virtualenv
dh: unable to load addon python-virtualenv: Can't locate Debian/Debhelper/Sequence/python_virtualenv.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .) at (eval 3) line 2.
BEGIN failed--compilation aborted at (eval 3) line 2.

make: *** [clean] Error 2
dpkg-buildpackage: error: fakeroot debian/rules clean gave error exit status 2

Add support for console script entry points in /usr/bin/

If a package installs a console script using a setuptools entrypoint, there will be a generated wrapper script in $VIRTUAL_ENV/bin/. See https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation

With dh-virtualenv this works as well, but the generated scripts are somewhere in /usr/share/.../$venv/bin/, and not in /usr/bin/. This means any command line tools are not available by default.

It would be nice if dh-virtualenv had an option to make any console scripts (preferably configurable since some libraries also provide console scripts) available in /usr/bin/.

Work around too long shebang lines when installing

This issue is related to #10, but since this is an issue in a different place of the pipeline (in build rather than after installation), it's filed separately.

Some installation scripts might want to run other scripts when running setup.py install. This can be problematic if they want to run a script that got installed by a dependency, since the shebang of those scripts is ${BUILDDIR}/usr/share/python/<packagename>/bin/python. This easily grows over 127 characters, effectively halting the build.

See if dh_auto_build could be removed

Currently debhelepr sequencer runs dh_auto_build, which tries to deduct which kind of package is in question. As setup.py exists, it thinks the packages made for dh_virtualenv as python packages and thus fires up a build.

As we anyway completely build the package later on, we should probably remove dh_auto_build from sequence when using dh-virtualenv.

pip acts up when asked to install '.' into the virtualenv

On newer Debian systems, you always get that pesky "local" directory or symlink (which causes all sorts of problems). When I packaged "nodeenv", the pip process spawned by dh_virtualenv basically went into a recursive copy (until path lengths grew too large), due to this:

$ ls -l debian/nodeenv/usr/share/python/nodeenv/local
lrwxrwxrwx 1 jhe jhe 1 Jun 17 17:22 debian/nodeenv/usr/share/python/nodeenv/local -> .

The fix is having a more complicated "rules" file (see here), I'd suggest to embed the additional "setup.py sdist" step directly into dh-virtualenv, since it avoids this and other foreseeable problems with temp dirs in your work dir.

Also, only files explicitly mentioned in the MANIFEST get added to the DEB package, which IMHO is what's expected.

All options passed to dh_virtualenv are removed.

The issue is this line:

sys_args_o = [o[2:] for o in sys.argv[1:] if o.startswith('-O-')]
options, args = parser.parse_args(sys_args_o + os.environ.get('DH_OPTIONS', '').split())

Any options that don't start with '-O-' is dropped.

A fix is:

sys_args_o = [(o[2:] if o.startswith('-O-') else o) for o in sys.argv[1:]]

I can't really figure out what the point of this line was in the first place. A comment would be nice.

Move dh-virtualenv to be ran after dh_prep instead of dh_perl

Current model where we run dh_virtualenv after dh_perl (blatantly copied from dh_python2) does not work if the user wants to use <packagename>.install files because dh_install* commands have been ran before building the virtualenv. So now the only option is the <packagename>.links.

Moving dh_virtualenv after dh_prep should solve this and probably #59.

Wrong path of dh_virtualenv.pm?

When building with the new experimental build mode, the Perl script can't be found on my box:

dh clean --buildsystem=dh_virtualenv
   dh_testdir -O--buildsystem=dh_virtualenv
   dh_auto_clean -O--buildsystem=dh_virtualenv
dh_auto_clean: unable to load build system class 'dh_virtualenv': Can't locate Debian/Debhelper/Buildsystem/dh_virtualenv.pm in @INC (you may need to install the Debian::Debhelper::Buildsystem::dh_virtualenv module) (@INC contains: /etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18 /usr/share/perl/5.18 /usr/local/lib/site_perl .) at (eval 4) line 2.

If I move dh_virtualenv.pm from Debian/Debhelper/BuildSystem to Debian/Debhelper/Buildsystem the script can be found on the build works fine.

Latest master (7952f34) on Ubuntu 14.04.

Build directories for documentation not being created

Building dh-virtualenv with dpkg-buildpackage -us -uc on HEAD gives:

...
make[1]: Entering directory `/home/vagrant/dh-virtualenv'
python setup.py build_sphinx
running build_sphinx
error: error in 'build_dir' option: 'doc/_build' does not exist or is not a directory
make[1]: *** [override_dh_installdocs] Error 1
make[1]: Leaving directory `/home/vagrant/dh-virtualenv'
make: *** [binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2

It looks like the directories used to build the sphinx documentation in setup.cfg are not created if they do not already exist. I also noted that the directory passed to dh_installdocs in debian/rules is hard-coded.

I am currently working around this by creating the directories in the debian/rules file, which is not the general solution.

Figure out workaround for too long shebang lines

Due to the shebang manipulation the shebang lines can grow a lot in size, depending on the project name and installation location.

Unfortunately shebang line usually limits to 127 characters and the only way to tweak it is to recompile the kernel. Proper workaround might be to ship a custom shebang runner with this

Package's activate script has PATH set for the build environment

Snippet from my package's bin/activate script, built on a vagrant VM:

VIRTUAL_ENV="/home/vagrant/my_component/debian/my_package/usr/share/python/my_package"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

The value for VIRTUAL_ENV is not valid outside of the build VM. I believe it should be /usr/share/python/my_package.

I think for the moment I can fix it up in my postinst script, but it looks like another step is required to fix up the virtualenv's activate script. If you agree that is the best way forward (ie I haven't missed something here), I'll see if I can find some time to patch this.

Create a cookiecutter for dh-virtualenv

It would help both new adapters plus other people converting existing repos to use dh-virtualenv if we would provide a cookiecutter template for it.

Please document USENETWORK=yes

Whenever using pbuilder (such as via git-buildpackage) we definitely need a network access to fetch dependencies from pypi. I eventually found the solution on #70 which is to pass USENETWORK=yes.

The variable is explained in man 5 pbuilderrc:

USENETWORK=no

Specify yes when you do not want to disable network access during build. Network is not available on a Debian buildd, so you might want to keep the default. Disabling network access currently only works on Linux.

I have set it globally in my ~/.pbuiderrc , since apparently passing it to git-buildpackage / cowbuilder does not work :(

Allow passing `--system-site-packages` to virtualenv

In some cases it might be useful to have --system-site-package flag passed to virtualenv to overcome some issues in legacy systems.

Support for the flag could be added with a bold disclaimer of what this flag might do.

Reuse created virtualenv on multiple binary packages

When more than one binary package is defined on debian/control, dh_virtualenv does not reuse the frozen virtualenv it creates.

In my usage scenario, the difference between "binaries" are solely on debian/xxx.install and debian/xxx.postinst files, but since the virtualenv is recreated for each define binary, the build takes much longer.

Add to it the fact that I am cross compiling using a slow ARM machine (QEMU / raspberry), and it gets real bad.

Can't build on trusty

To be able to use dh-virtualenv every day and easily, I try (and will ;) ) to create a ppa for dh-virtual env.

Here is the beginning of my work:
https://launchpad.net/dh-virtualenv
https://launchpad.net/~dh-virtualenv
https://code.launchpad.net/~dh-virtualenv/dh-virtualenv/trunk
https://code.launchpad.net/~dh-virtualenv/+recipe/dh-virtualenv-daily
https://code.launchpad.net/~dh-virtualenv/+archive/daily
https://launchpad.net/~dh-virtualenv/+archive/stable (not provisionned for now)

so this repo can be used to install dh-virtualenv

 deb http://ppa.launchpad.net/dh-virtualenv/daily/ubuntu precise main

My problem come from trusty (14.04);
https://launchpadlibrarian.net/162948633/buildlog.txt.gz

It now fail to build with format '3.0 (native)' and prefer format '3.0 (quilt)'.

Is it possible to change it?

Python updates can break virtualenvs

Sometimes python updates break existing virtualenvs, and you have to copy a new python in or rebuild them to keep things working. I think this really breaks dh-virtualenv as a way of distributing things that don't have a regular release pipeline.

The most recent example of this is 2.6.8 and 2.7.3 which cause import random to fail with "cannot import name urandom" when the package was built with and earlier python. Before that I saw it happen with the MAXREPEAT constant.

I can't think of a good solution for this.

We can't symlink the python binary itself without breaking the virtualenv machinery.

We could possibly use dpkg triggers to automatically update the virtualenv when the system python changes. A bit risky.

Perhaps the cleanest option is to break any symlinks in the virtualenv's stdlib. Replace them with a full copy. This is what virtualenv does on platforms that don't support symlink (unfortunately it's not exposed as a command line option). However that's only a subset of the stdlib, so not enough. We'd have to copy the entire stdlib. Which is lame.

At the very least we could just note in the docs that this can happen.

The --no-test option has no default value preventing tests from running

Hi,

The --no-test option is currently not set up with a default value of true. This means that options.test is either None or False in dh_virtualenv.

https://github.com/spotify/dh-virtualenv/blob/master/dh_virtualenv/cmdline.py#L64
https://github.com/spotify/dh-virtualenv/blob/master/bin/dh_virtualenv#L71

We can put together a pull request if you'd like?

Tom

Fix dependencies for systems that don't sport `virtualenv` package

As mentioned in fdc57c3#commitcomment-6632167 the building and using the package on a system that does not have package virtualenv is broken after the commit fdc57c3.

Correct way most likely would be to have the dependency to use virtualenv or python-virtualenv, but we need to wait for the Debian to get the virtualenv in unstable to be able to test.

dh_auto_tests can break installation

dh_auto_tests runs python setup.py test with the system's interpreter. This can unfortunately break the installation.

One solution is to create dh_virtualenv_tests or similar and make it run on the ready-made virtualenv.

This relates to #11 and probably fixing this fixes it too

Should automatically honor DEB_BUILD_OPTIONS=nocheck

dh-virtualenv has a --no-test parameter to skip runing tests, it should also skip test whenever DEB_BUILD_OPTIONS contains the tag nocheck. The tag is standardized in Debian Policy 4.9.1 debian/rules and DEB_BUILD_OPTIONS.

A workaround is:

override_dh_virtualenv:
ifneq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
    $(eval DH_VIRTUALENV_NO_TEST := "--no-test")
endif
    dh_virtualenv $(DH_VIRTUALENV_NO_TEST)

Don't include .pyc files in the .deb

What is the reason for shipping *.pyc files inside the virtualenv? Is this just because everything is copied in, or does it serve a specific purpose?

split build and install part

For the moment, dh-virtualenv is running at the end of "install" part of debhelper.

It would be great to be able to have a "build" part. I explain:

• In build part, we are creating a virtualenv in a temporary dir (with packages etc).
• In install part, we are coping this venv to the correct place in package.

We are needing this, because we have to do some special command on venv to be able to build our software.
For example, to do a collectstatic on a django app we needing a complete venv. This static will be put in the right place in package with mypackage-static.install file.

I begin to work on it, here is for example command I used in python_virtualenv.pm:

insert_before("dh_auto_build", "dh_virtualenv_build");
insert_before("dh_auto_install", "dh_virtualenv_install");
insert_before("dh_clean", "dh_virtualenv_clean");

Depend on virtualenv instead of python-virtualenv

Due to the change in Debian packaging, we need to depend on virtualenv instead of python-virtualenv:

http://www.mail-archive.com/[email protected]/msg1227948.html

Relevant Debian bug #751192

Add hook for running tests inside the virtual environment

People using continuous integration and such (like us) would like to run tests inside the virtualenv.

This should probably happen prior to the shebang-line rewrite, as the rewrite will make calling scripts cumbersome or then a helper needs to be provided.

Now you can get around this with override_dh_virtualenv, but it is far from optimal.