sjkingo / virtualenv-api

An API for virtualenv/pip

virtualenv-api - an API for virtualenv

virtualenv is a tool to create isolated Python environments. Unfortunately, it does not expose a native Python API. This package aims to provide an API in the form of a wrapper around virtualenv.

It can be used to create and delete environments and perform package management inside the environment.

Full support is provided for all supported versions of Python.

Installation

The latest stable release is available on PyPi:

$ pip install virtualenv-api

Please note that the distribution is named virtualenv-api, yet the Python package is named virtualenvapi.

Alternatively, you may fetch the latest version from git:

$ pip install git+https://github.com/sjkingo/virtualenv-api.git

Usage

To begin managing an environment (it will be created if it does not exist):

from virtualenvapi.manage import VirtualEnvironment
env = VirtualEnvironment('/path/to/environment/name')

If you have already activated a virtualenv and wish to operate on it, simply call VirtualEnvironment without the path argument:

env = VirtualEnvironment()

The VirtualEnvironment constructor takes some optional arguments (their defaults are shown below):

• python=None - specify the Python interpreter to use. Defaults to the default system interpreter (new in 2.1.3)
• cache=None - existing directory to override the default pip download cache
• readonly=False - prevent all operations that could potentially modify the environment (new in 2.1.7)
• system_site_packages=False - include system site packages in operations on the environment (new in 2.1.14)

Operations

Once you have a VirtualEnvironment object, you can perform operations on it.

• Check if the mezzanine package is installed:

>>> env.is_installed('mezzanine')
False

• Install the latest version of the mezzanine package:

>>> env.install('mezzanine')

• A wheel of the latest version of the mezzanine package (new in 2.1.4):

>>> env.wheel('mezzanine')

• Install version 1.4 of the django package (this is pip’s syntax):

>>> env.install('django==1.4')

• Upgrade the django package to the latest version:

>>> env.upgrade('django')

• Upgrade all packages to their latest versions (new in 2.1.7):

>>> env.upgrade_all()

• Uninstall the mezzanine package:

>>> env.uninstall('mezzanine')

Packages may be specified as name only (to work on the latest version), using pip’s package syntax (e.g. django==1.4) or as a tuple of ('name', 'ver') (e.g. ('django', '1.4')).

• A package may be installed directly from a git repository (must end with .git):

>>> env.install('git+git://github.com/sjkingo/cartridge-payments.git')

New in 2.1.10:

• A package can be installed in pip's editable mode by prefixing the package name with -e (this is pip's syntax):

>>> env.install('-e git+https://github.com/stephenmcd/cartridge.git')

New in 2.1.15:

• Packages in a pip requirements file can be installed by prefixing the requirements file path with -r:

>>> env.install('-r requirements.txt')

• Instances of the environment provide an installed_packages property:

>>> env.installed_packages
[('django', '1.5'), ('wsgiref', '0.1.2')]

• A list of package names is also available in the same manner:

>>> env.installed_package_names
['django', 'wsgiref']

• Search for a package on PyPI (changed in 2.1.5: this now returns a dictionary instead of list):

>>> env.search('virtualenv-api')
{'virtualenv-api': 'An API for virtualenv/pip'}
>>> len(env.search('requests'))
231

• The old functionality (pre 2.1.5) of env.search may be used:

>>> list(env.search('requests').items())
[('virtualenv-api', 'An API for virtualenv/pip')]

Verbose output from each command is available in the environment's build.log file, which is appended to with each operation. Any errors are logged to build.err.

virtualenv-api's Issues

Shebang length exceeded in pip executable

See the following issue on pip: pypa/pip#1773

The accepted solution seems to be running the following:

/some/path/to/python -m pip

I would make a PR but I am not sure on how you want to go about supporting that.

README.md is not in the tarball

Steps to reproduce:

1. Go to https://pypi.python.org/pypi/virtualenv-api
2. Download https://pypi.python.org/packages/source/v/virtualenv-api/virtualenv-api-2.1.7.tar.gz
3. Unpack it and cd in
4. virtualenv-3.4 .env
5. . .env/bin/activate
6. python setup.py install

Actual result:

Installed /home/churchyard/tmp/virtualenv-api-2.1.7/.eggs/pypandoc-1.1.3-py3.4.egg
Traceback (most recent call last):
  File "setup.py", line 15, in <module>
    packages=find_packages(),
  File "/usr/lib64/python3.4/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/home/churchyard/tmp/virtualenv-api-2.1.7/.env/lib/python3.4/site-packages/setuptools/dist.py", line 269, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib64/python3.4/distutils/dist.py", line 280, in __init__
    self.finalize_options()
  File "/home/churchyard/tmp/virtualenv-api-2.1.7/.env/lib/python3.4/site-packages/setuptools/dist.py", line 325, in finalize_options
    ep.load()(self, ep.name, value)
  File "/home/churchyard/tmp/virtualenv-api-2.1.7/.eggs/setuptools_markdown-0.2-py3.4.egg/setuptools_markdown.py", line 22, in long_description_markdown_filename
    output = pypandoc.convert(markdown_filename, 'rst')
  File "/home/churchyard/tmp/virtualenv-api-2.1.7/.eggs/pypandoc-1.1.3-py3.4.egg/pypandoc/__init__.py", line 50, in convert
    outputfile=outputfile, filters=filters)
  File "/home/churchyard/tmp/virtualenv-api-2.1.7/.eggs/pypandoc-1.1.3-py3.4.egg/pypandoc/__init__.py", line 68, in _convert
    raise RuntimeError('Missing format!')
RuntimeError: Missing format!

The error above says Missing format, but is actually about a missing file. See JessicaTegner/pypandoc#86

Excepted result: Installs.

See also fedora-python/pyp2rpm#27 and msabramo/setuptools-markdown#7 (comment)

Support editable installation

pip help:

Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.

As I see, this is not possible now with the virtualenv-api.
I tried env.install("-e /path/to/module"), but it raised an exception.

(virtualenv-api==2.1.9)

BTW this package is pretty cool :)

Package install hangs

I am trying to create a new virtual env and install a specific package (paramiko==1.7.7.1), but it seems that every time I do that, the installation hangs in a specific point.

My setup:

import shutil

from virtualenvapi.manage import VirtualEnvironment

if __name__ == '__main__':
    venv = VirtualEnvironment('./bla-env')
    venv.install(('paramiko', '1.7.7.1'), options=['--no-cache'])
    print venv.installed_packages
    shutil.rmtree('./bla-env')

Using PyCharm, I have been able to track down where it hangs:

Traceback (most recent call last):
  File "/home/alacea/pycharm-2018.3.5/helpers/pydev/pydevd.py", line 1741, in <module>
    main()
  File "/home/alacea/pycharm-2018.3.5/helpers/pydev/pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/alacea/pycharm-2018.3.5/helpers/pydev/pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/alacea/test-venv/src/vapi-test.py", line 7, in <module>
    venv.install(('paramiko', '1.7.7.1'), options=['--no-cache'])
  File "/home/alacea/test-venv/env/lib/python2.7/site-packages/virtualenvapi/manage.py", line 223, in install
    self._execute_pip(['install'] + package_args + options)
  File "/home/alacea/test-venv/env/lib/python2.7/site-packages/virtualenvapi/manage.py", line 127, in _execute_pip
    return self._execute(exec_args, log=log)
  File "/home/alacea/test-venv/env/lib/python2.7/site-packages/virtualenvapi/manage.py", line 137, in _execute
    output, error = proc.communicate()
  File "/opt/test/base/lib/python2.7/subprocess.py", line 483, in communicate
    return self._communicate(input)
  File "/opt/test/base/lib/python2.7/subprocess.py", line 1124, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/opt/test/base/lib/python2.7/subprocess.py", line 1178, in _communicate_with_poll
    ready = poller.poll()
KeyboardInterrupt

Do you have any idea what might be causing this?

Installing the same package from the CLI (--no-cache as well) works just fine. I didn't come across other packages causing this so far.

No such option: --disable-pip-version-check

When I run:

>>> env = VirtualEnvironment('/tmp/venv')
>>> env.is_installed('jinja2')

The last command fails with an error:
subprocess.CalledProcessError: Command '<subprocess.Popen object at 0x7fb69ffbacd0>' returned non-zero exit status 2

error output: ('', '\nUsage: \n /tmp/venv/bin/python -m pip [options]\n\nno such option: --disable-pip-version-check\n') (it seems raw_pip argument of _execute_pip method is False)

Arguments of subprocess are: ARGS:['bin/python', '-m', 'pip', '--disable-pip-version-check', '-V']

pip version is 8.1.2

Build time blowout

Tests are now taking over 10 minutes to run. This increased slightly when Python 3.5 support was added in 72bf154 but over time I can see they've increased gradually.

Need to investigate what is slowing them down and seeing if we can do anything about speeding it up.

https://travis-ci.org/sjkingo/virtualenv-api/builds

Packages with capitalized names give the wrong result for is_installed

Try this:

$ pip install Pillow
$ ipython
Python 3.6.1 (v3.6.1:69c0db5050, Mar 21 2017, 01:21:04)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from virtualenvapi.manage import VirtualEnvironment

In [2]: env = VirtualEnvironment('.venv')

In [3]: env.installed_package_names
Out[3]:
[
 ...
 'pillow',
 'pip',
 ...
]

In [4]: env.is_installed('Pillow')
Out[4]: False

In [5]: env.installed_packages
Out[5]:
[
 ('Pillow', '5.3.0'),
 ('pip', '18.1'),
 ...
]

It looks like installed_package_names would lowercase all the package names, so the check for whether the package is installed would wrongly return False.

Additional functionality

virtualenv-api is quite useful. Thanks.

I presume it's possible to use a VirtualEnvironment() from the Python that created the VirtualEnvironment(). The code would look like this, for example:

env = VirtualEnvironment('/path/to/environment/name')
env.install('package_foo')
# modify sys.path to use env
# now import in env's context
import bar
# perhaps modify sys.path to stop using env

If so, these two methods would be helpful:

env.activate()      # modify sys.path so that imports occur in env's context
env.deactivate()    # undo env.activate()

Thanks, Arthur

Run scripts from created venv

The VirtualEnvironment class provide methods to install/uninstall packages and list them. But there is no way to activate created venv and/or run scripts using this brand new environment.

Do you plan to add something like these methods or it is outside of project's scope and doesn't worth to be implemented?

Add test case for `pip_version` and `pip_upgrade` functions

Package installed in wrong virtual env

In example.py the line
env.install(payments_repo)
installs the package cartridge-payments in my currently active 'real' environment, not the one set up in the example.

Also, shouldn't the checks for the package being installed check for 'cartridge-payments' rather than payments_repo?

Relativ paths does not work for Windows

We use this package for a project and we discovered a bug for Windows users. If you want to install dependencies on windows you will receive an OSError - File not found.

We looked a bit deeper into this bug and found out that the path to the python binary (on windows \.venv\Scripts\python.exe) is passed as a relative path. The problem is that Popen on windows does not execute this path from the current working directory (see: https://stackoverflow.com/a/21412727).

Therefore the argument for the Popen call needs to have an absolute path. This could be fixed with (if path contains the absolute path)

@property
def _python_rpath(self):
    """The relative path (from environment root) to python."""
    # Windows virtualenv installation installs pip to the [Ss]cripts
    # folder. Here's a simple check to support:
    if sys.platform == 'win32':
        return os.path.join(self.path, 'Scripts', 'python.exe')
    return os.path.join('bin', 'python')

Alternatively, it could be solved with a os.path.abspath for Windows.

package name on PyPI is not importable

If I 'import virtualenvapi' can the name of the package on PyPI be 'virtualenvapi' to match that?

Add tests for specifying path=None to VirtualEnvironment

This looks up the virtualenv path in the env variable VIRTUAL_ENV.

2.1.18: pytest is failing in four units

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-virtualenv-api-2.1.18-6.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-virtualenv-api-2.1.18-6.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra tests.py
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.13, pytest-7.1.1, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/virtualenv-api-2.1.18
collected 10 items

tests.py .....FFFF.                                                                                                                                                  [100%]

================================================================================= FAILURES =================================================================================
________________________________________________________________________ SearchTestCase.test_search ________________________________________________________________________

self = <tests.SearchTestCase testMethod=test_search>

    def test_search(self):
        for pack in all_packages_for_tests:
>           result = self.virtual_env_obj.search(pack)

tests.py:106:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
virtualenvapi/manage.py:307: in search
    results = self._execute_pip(['search', term], log=False)  # Don't want to log searches
virtualenvapi/manage.py:132: in _execute_pip
    return self._execute(exec_args, log=log)
virtualenvapi/manage.py:156: in _execute
    raise e
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <virtualenvapi.manage.VirtualEnvironment object at 0x7f54fe45cf10>, args = ['bin/python', '-m', 'pip', '--disable-pip-version-check', 'search', 'pep8'], log = False

    def _execute(self, args, log=True):
        """Executes the given command inside the environment and returns the output."""
        if not self._ready:
            self.open_or_create()
        output = ''
        error = ''
        try:
            proc = subprocess.Popen(args, cwd=self.path, env=self.env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            output, error = proc.communicate()
            returncode = proc.returncode
            if returncode:
>               raise subprocess.CalledProcessError(returncode, proc, (output, error))
E               subprocess.CalledProcessError: Command '<subprocess.Popen object at 0x7f54fe45cb50>' returned non-zero exit status 1.

virtualenvapi/manage.py:145: CalledProcessError
_____________________________________________________________________ SearchTestCase.test_search_names _____________________________________________________________________

self = <tests.SearchTestCase testMethod=test_search_names>

    def test_search_names(self):
        for pack in all_packages_for_tests:
>           result = self.virtual_env_obj.search_names(pack)

tests.py:114:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
virtualenvapi/manage.py:323: in search_names
    return list(self.search(term).keys())
virtualenvapi/manage.py:307: in search
    results = self._execute_pip(['search', term], log=False)  # Don't want to log searches
virtualenvapi/manage.py:132: in _execute_pip
    return self._execute(exec_args, log=log)
virtualenvapi/manage.py:156: in _execute
    raise e
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <virtualenvapi.manage.VirtualEnvironment object at 0x7f54fe477790>, args = ['bin/python', '-m', 'pip', '--disable-pip-version-check', 'search', 'pep8'], log = False

    def _execute(self, args, log=True):
        """Executes the given command inside the environment and returns the output."""
        if not self._ready:
            self.open_or_create()
        output = ''
        error = ''
        try:
            proc = subprocess.Popen(args, cwd=self.path, env=self.env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            output, error = proc.communicate()
            returncode = proc.returncode
            if returncode:
>               raise subprocess.CalledProcessError(returncode, proc, (output, error))
E               subprocess.CalledProcessError: Command '<subprocess.Popen object at 0x7f54fe477820>' returned non-zero exit status 1.

virtualenvapi/manage.py:145: CalledProcessError
________________________________________________________________ PythonArgumentTestCase.test_python_version ________________________________________________________________

self = <tests.PythonArgumentTestCase testMethod=test_python_version>

    def setUp(self):
        self.env_path = tempfile.mkdtemp()
        self.python = which('python')
>       self.assertIsNotNone(self.python)
E       AssertionError: unexpectedly None

tests.py:127: AssertionError
___________________________________________________________ SystemSitePackagesTest.test_no_system_site_packages ____________________________________________________________

self = <tests.SystemSitePackagesTest testMethod=test_no_system_site_packages>

    def test_no_system_site_packages(self):
        """
        test that creating a venv with system_site_packages=False
        results in a venv that contains the no-global-site-packages
        file

        """
        venv = VirtualEnvironment(self.dir)
        venv._create()
        expected = os.path.join(venv.path, self.no_global)
>       self.assertTrue(
            os.path.exists(expected)
        )
E       AssertionError: False is not true

tests.py:187: AssertionError
========================================================================= short test summary info ==========================================================================
FAILED tests.py::SearchTestCase::test_search - subprocess.CalledProcessError: Command '<subprocess.Popen object at 0x7f54fe45cb50>' returned non-zero exit status 1.
FAILED tests.py::SearchTestCase::test_search_names - subprocess.CalledProcessError: Command '<subprocess.Popen object at 0x7f54fe477820>' returned non-zero exit status 1.
FAILED tests.py::PythonArgumentTestCase::test_python_version - AssertionError: unexpectedly None
FAILED tests.py::SystemSitePackagesTest::test_no_system_site_packages - AssertionError: False is not true
======================================================================= 4 failed, 6 passed in 57.93s =======================================================================

Install in editable mode does not use name in setup.py

Using virtualenv-api version 2.1.17,

I can install my_package and then print the package name:

from virtualenvapi.manage import VirtualEnvironment
env = VirtualEnvironment('.')
env.install('.')
print(env.installed_package_names)

Result:

['pip', 'setuptools', 'my_package', 'wheel']

Nice!

But, doing the same in editable mode does not give me the package name in setup.py.

from virtualenvapi.manage import VirtualEnvironment
env = VirtualEnvironment('.')
env.install('-e .')
print(env.installed_package_names)

Result:

['pip', 'setuptools', '-e git+https://github.com/marskar/package_name.git@6e7d262c1d9ad5047ada8b8ad471f3f1852dad87#egg=try', 'wheel']

According to the method docstring, -e . is supported.
https://github.com/sjkingo/virtualenv-api/blob/master/virtualenvapi/manage.py#L193
Is this the expected behavior?

install() method has 'sticky options'

If install() is called with force=True and/or upgrade=True but options= not specified then the pip options created by install() are used for subsequent calls of install() regardless of force= and upgrade= settings if options= is again not specified.

To show this problem I added the the following as the first executable line in _execute():

        print('_execute: ' + ' '.join(args))

Then, running the following script:

#!/usr/bin/env python
import tempfile

from virtualenvapi.manage import VirtualEnvironment

def example(path=tempfile.mkdtemp('virtualenv.test')):
    pkg1 = 'tzlocal'
    pkg2 = 'pymongo'
    print('Env path', path)
    env = VirtualEnvironment(path)

    print('1. install package 1 with force and upgrade both True')
    env.install(pkg1, force=True, upgrade=True)
    print()
    print('2. install package 2 with default force and upgrade')
    env.install(pkg2)

if __name__ == '__main__':
    example()

produces this output:

(new_env)$ ./opts.py 
Env path /var/folders/rc/y53n1qss1pd3d2d593vpf2wm0000gn/T/tmpwk_i515uvirtualenv.test
1. install package 1 with force and upgrade both True
_execute: bin/pip install tzlocal --upgrade --force-reinstall

2. install package 2 with default force and upgrade
_execute: bin/pip freeze -l
_execute: bin/pip install pymongo --upgrade --force-reinstall
(new_env)$ 

The --upgrade and --force-reinstall options in step 2 have been carried over from step 1.

I believe this problem happens because of a well-known Python 'gotcha': use of =[] to specify the default value of the options keyword parameter of install(). Default values are evaluated only once, when the definition of install() is processed, and so the options argument is always bound to the same default object unless the caller supplies a value. Since [] creates a mutable object, any strings appended to it by code in install() will still be there the next time the default value for options= is needed.

Tests failing because of wheel installation error

@rmb938: there is a failure in the tests because it can't find MarkupSafe to install as a wheel (a dependency of flask). It's definitely on pypi and I can install it successfully.

Build log is here: https://travis-ci.org/sjkingo/virtualenv-api/jobs/119961786

The pip output says it's ignoring indexes and using cache only? What's the --no-index flag to pip for in this instance?

Would you be able to take a look and see if you can fix?

pip options

Hi,
In the context of some of my projects, I need to use local pip repository, defined with the --extra-index-url option, as well as the --allow-all-external. It would be nice to have a mechanism to set those (or any) options via the API...

Thanks.

Small error in docs

In the last example

>>> list(env.search('requests').items())
[('virtualenv-api', 'An API for virtualenv/pip')]

the search looks for 'requests' but the results are for 'virtualenv-api'.