paylogic / pip-accel Goto Github PK

View Code? Open in Web Editor NEW

307.0 307.0 35.0 644 KB

pip-accel: Accelerator for pip, the Python package manager

Home Page: https://pypi.python.org/pypi/pip-accel

License: MIT License

Python 95.89% Makefile 0.98% Shell 2.35% Batchfile 0.79%

pip-accel's People

Contributors

Stargazers

Watchers

pip-accel's Issues

The file (path-to-package-name).zip exists. (i)gnore, (w)ipe, (b)ackup

I use pip-accel with automated testing, however when I updated the Python package today, my automated tests cannot use it. I get the following prompt when trying to install packages:

The file /var/cache/pip-accel/sources/(package-name).zip exists. (i)gnore, (w)ipe, (b)ackup

I also get this prompt when running in 'silent mode' specified by the '-q' option.
Is there a way to automatically ignore this message?

Not able to use pip-accel both globally and inside a virtual environment

I've installed pip-accel globally, but when I try to use it inside a virtual environment I get this error:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip_accel/cli.py", line 56, in main
    accelerator = PipAccelerator(config)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip_accel/__init__.py", line 109, in __init__
    self.validate_environment()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip_accel/__init__.py", line 141, in validate_environment
    """, environment=environment, prefix=sys.prefix)
EnvironmentMismatchError: You are trying to install packages in environment #1 which is different from environment #2 where pip-accel is installed! Please install pip-accel under environment #1 to install packages there.

This is caused by validate_environment, which doesn't make much sense. Since pip-accel uses pip under the hood, why should I have to install another version of pip-accel inside every virtual environment? The kicker is that if pip-accel is already installed globally, pip won't even let you install it inside a virtual environment. It's either one or the other.

install_binary_dist doesn't respect directory permissions

It appears that the install_binary_dist function tries to create a missing subdirectory even if it doesn't have the permissions to do so.
More specifically, the issue appears while trying to install the greenlet package, which is supposed to install a header file in the include directory of the virtualenv. However pip-accel tries to create the directory $VIRTUAL_ENV/include/python2.7/greenlet which is problematic since $VIRTUAL_ENV/include/python2.7 is a symbolic link to /usr/local/include/python2.7 which is owned by root.
Pip works around this issue by installing the header file in $VIRTUAL_ENV/include/site/python2.7/.

Currently, since the installation procedure is interrupted, the virtual environment is left in a "semi-broken" state because greenlet is not really installed, but the egg-info is in place making pip think that greenlet is installed. Of course, running pip uninstall greenlet && pip install greenlet fixes the environment.

Check out terrarium

Hi,

We just came across your project and are excited at the prospect of integrating it into our development work flow. For requirements management, we currently use a tool that we developed internally and released called terrarium (https://github.com/PolicyStat/terrarium).

My initial feeling is that pip-accel and terrarium could work well when used together, so I thought I would share our project. Let us know what you think!

multiple parallels pip-accel can fail

even with a virtualenv.

We got parallel branches being build/tested with jenkins and it uses pip-accel to install packages into a virtual env. But it seems sometimes we get an error about missing files. My guess is that the other pip-accel deleted it.

015-04-14 11:31:43 stagingpitchup.dh.bytemark.co.uk pip_accel.bdist[4504] INFO Deleting old functools32 (3.2.3.post1) binary (source is newer) ..
2015-04-14 11:31:44 stagingpitchup.dh.bytemark.co.uk pip_accel.cli[4504] ERROR Caught unhandled exception!
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/cli.py", line 55, in main
    accelerator.install_from_arguments(arguments)
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 234, in install_from_arguments
    return self.install_requirements(requirements, **kw)
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 520, in install_requirements
    self.bdists.install_binary_dist(binary_distribution, **kw)
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/bdist.py", line 329, in install_binary_dist
    for member, from_handle in members:
  File "/var/lib/jenkins/jobs/pitchup.com-mj/workspace/venv/local/lib/python2.7/site-packages/pip_accel/bdist.py", line 76, in get_binary_dist
    os.unlink(cache_file)
OSError: [Errno 2] No such file or directory: '/var/lib/jenkins/.pip-accel/binaries/v7/functools32:3.2.3.post1:CPython-2.7.tar.gz'

Allow anonymous connections to the S3 cache backend

S3 has the ability to set ACLs for anonymous access to buckets. This is wired into boto's s3.connection.S3Connection method via the anon parameter.

Modify the S3 cache backend code so that if the user does not have credentials, it will connect as an anonymous user rather than erroring with "No handler was ready to authenticate... Check your credentials" and disabling the S3CacheBackend.

'NoneType' object has no attribute 'endswith'

I recently started getting the following error trying to do installs with pip-accel. I don't hardcode the version of pip-accel I install, so it might be something introduced in a recent release, but I haven't narrowed it down yet:

2014-01-09 19:30:46 build01 pip-accel[19661] INFO Unpacked local source distributions in 3.81 seconds.
Traceback (most recent call last):
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/bin/pip-accel", line 9, in <module>
    load_entry_point('pip-accel==0.11.2', 'console_scripts', 'pip-accel')()
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip_accel/__init__.py", line 99, in main
    requirements = unpack_source_dists(arguments, build_directory)
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip_accel/__init__.py", line 177, in unpack_source_dists
    return sorted([Requirement(r) for r in requirement_set.requirements.values()],
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip_accel/req.py", line 46, in __init__
    self.old_interface = (self.name, self.version, self.source_directory)
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip_accel/req.py", line 80, in version
    return self.pip_requirement.installed_version
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip/req.py", line 390, in installed_version
    return self.pkg_info()['version']
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip/req.py", line 359, in pkg_info
    logger.warn('No PKG-INFO file found in %s' % display_path(self.egg_info_path('PKG-INFO')))
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/site-packages/pip/req.py", line 306, in egg_info_path
    base = os.path.join(self.source_dir, 'pip-egg-info')
  File "/home/buildbot/jenkins/workspace/gemstone-server-integration-prod/gemstone-server/virtualenv/lib/python2.6/posixpath.py", line 67, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

pip is version 1.4.1, which I've been using for a while. And although the exception is down in pip itself, if I replace pip-accel in my script with pip, it works fine.

I've tried nuking my ~/.pip-accel directory but that didn't work. Any other ideas?

Repeating output from pip after retries

When pip-accel runs pip more than once (as it does whenever you're installing a new package) the output of pip starts to repeat line by line, once for each time that pip runs (it gets really awkward after a bunch of retries).

I introduced this bug while converting pip-accel to run pip using the pip "API" instead of running it as a subprocess. However that change had other important aspects going for it so I went ahead anyway.

I'd still like to fix this though. It may involve beating pip into submission, but that might just be worth it if the change isn't too ugly and/or involved :-)

Exception during installation from an empty file

Pip just return a simple message. But when i try install requirements from empty file, pip-accel generate exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/pip_accel/cli.py", line 53, in main
    accelerator.install_from_arguments(arguments)
  File "/usr/local/lib/python3.4/site-packages/pip_accel/__init__.py", line 179, in install_from_arguments
    requirements = self.get_requirements(arguments)
  File "/usr/local/lib/python3.4/site-packages/pip_accel/__init__.py", line 197, in get_requirements
    return self.unpack_source_dists(arguments)
  File "/usr/local/lib/python3.4/site-packages/pip_accel/__init__.py", line 244, in unpack_source_dists
    requirements = self.get_pip_requirement_set(arguments, use_remote_index=False)
  File "/usr/local/lib/python3.4/site-packages/pip_accel/__init__.py", line 316, in get_pip_requirement_set
    return self.transform_pip_requirement_set(pip.requirement_set)
  File "/usr/local/lib/python3.4/site-packages/pip_accel/__init__.py", line 335, in transform_pip_requirement_set
    for requirement in requirement_set.requirements.values():
AttributeError: 'NoneType' object has no attribute 'requirements'

is_wheel fails when source_directory is None

I am trying to accelerate devstack execution using pip_accell. After forcing devstack to use pip_accell I had below error as soon as devstack tries to install/update setuptools package.

The failing command line is this:

sudo -H pip-accel install -v -c upper-constraints.txt -U setuptools

I have attached the whole error log and upper-constraints.txt file.

Produced error is this:

2015-12-28 09:52:23 control pip_accel.cli[19019] ERROR Caught unhandled exception!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/cli.py", line 57, in main
    accelerator.install_from_arguments(arguments)
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/__init__.py", line 234, in install_from_arguments
    have_wheels = any(req.is_wheel for req in requirements)
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/__init__.py", line 234, in <genexpr>
    have_wheels = any(req.is_wheel for req in requirements)
  File "/usr/local/lib/python2.7/dist-packages/cached_property.py", line 26, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python2.7/dist-packages/pip_accel/req.py", line 167, in is_wheel
    probably_sdist = os.path.isfile(os.path.join(self.source_directory, 'setup.py'))
  File "/usr/lib/python2.7/posixpath.py", line 77, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

--exists-action option does not work

While running command:
pip-accel install --exists-action=s -r requirements.txt

I get:
Executing command: pip install --download-cache=/home/.../.pip_download_cache --find-links=file:///... --build-directory=/tmp/... --exists-action=s -r requirements.txt --no-install An error occurred during configuration: option --exists-action: invalid choice: ['s'] (choose from 's', 'i', 'w', 'b')

Using pip 1.4.1, pip-accel 0.10.2, Python 2.7.3

pip-accel does not support running as a module

I've tried to run pip-accel as a
python -m pip_accel install -r requirements.txt

but it does not support running this way:
No module named pip_accel.__main__; 'pip_accel' is a package and cannot be directly executed

python==2.7, pip==7.1.2, pip-accel==0.38

0.28.2 raises ValueError on some package install

During requirements installation at some point I get:

Traceback (most recent call last):
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/pip_accel/cli.py", line 55, in main
accelerator.install_from_arguments(arguments)
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/pip_accel/__init__.py", line 234, in install_from_arguments
return self.install_requirements(requirements, **kw)
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/pip_accel/__init__.py", line 520, in install_requirements
self.bdists.install_binary_dist(binary_distribution, **kw)
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/pip_accel/bdist.py", line 328, in install_binary_dist
for member, from_handle in members:
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/pip_accel/bdist.py", line 74, in get_binary_dist
if cache_file and requirement.last_modified > os.path.getmtime(cache_file):
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/cached_property.py", line 26, in __get__
value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/tmp/test-virtualenv3.3/lib/python3.3/site-packages/pip_accel/req.py", line 127, in last_modified
return max(mtimes) if mtimes else time.time()
ValueError: max() arg is an empty sequence

It works on 0.27 and other previous releases. Locally on Py 3.4 I couldn't reproduce it, so it also may be specific to Py 3.3.

ipython fails to install with pip-accel: pkg_resources.DistributionNotFound: gnureadline

preparing environment for pip-accel

$ export PS1='$ '
$ mkvirtualenv xxx
(xxx)$ python -V
Python 2.7.8
(xxx)$ pip install pip-accel
Downloading/unpacking pip-accel
  Downloading pip-accel-0.13.3.tar.gz
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/pip-accel/setup.py) egg_info for package pip-accel

Downloading/unpacking coloredlogs>=0.4.6 (from pip-accel)
  Downloading coloredlogs-0.8.tar.gz
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/coloredlogs/setup.py) egg_info for package coloredlogs

Downloading/unpacking humanfriendly>=1.6 (from pip-accel)
  Downloading humanfriendly-1.9.6.tar.gz
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/humanfriendly/setup.py) egg_info for package humanfriendly

Downloading/unpacking pip>=1.4,<1.5 (from pip-accel)
  Downloading pip-1.4.1.tar.gz (445kB): 445kB downloaded
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/pip/setup.py) egg_info for package pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip-accel, coloredlogs, humanfriendly, pip
  Running setup.py install for pip-accel

    Installing pip-accel script to /Users/jquast/.virtualenvs/xxx/bin
  Running setup.py install for coloredlogs

    Installing ansi2html script to /Users/jquast/.virtualenvs/xxx/bin
  Running setup.py install for humanfriendly

  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /Users/jquast/.virtualenvs/xxx/bin
    Installing pip-2.7 script to /Users/jquast/.virtualenvs/xxx/bin
Successfully installed pip-accel coloredlogs humanfriendly pip
Cleaning up...

installing ipython==2.3.0 with pip-accel (fails)

(xxx)$ pip-accel install ipython==2.3.0
2014-10-24 16:27:22 IO-PHX-L-468 pip_accel[40389] INFO Unpacking local source distributions ..
2014-10-24 16:27:22 IO-PHX-L-468 pip_accel[40389] INFO Executing command: pip install --download-cache=/Users/jquast/.pip/download-cache --find-links=file:///Users/jquast/.pip-accel/sources --build-directory=/var/folders/k3/ssq0mds55pn0lg4v0lr9cmp48g76w2/T/tmpStcttu --no-index ipython==2.3.0 --no-install
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking ipython==2.3.0
  Running setup.py egg_info for package ipython
                  readline: yes

Successfully downloaded ipython
2014-10-24 16:27:24 IO-PHX-L-468 pip_accel[40389] INFO Unpacked local source distributions in 1.08 second.
2014-10-24 16:27:24 IO-PHX-L-468 pip_accel[40389] INFO Installing from binary distributions ..
2014-10-24 16:27:25 IO-PHX-L-468 pip_accel[40389] INFO Finished installing all requirements in 1.04 second.
2014-10-24 16:27:25 IO-PHX-L-468 pip_accel[40389] INFO Done! Took 2.12 seconds to install 1 package.
(xxx)$ ipython -V
Traceback (most recent call last):
  File "/Users/jquast/.virtualenvs/xxx/bin/ipython", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 2829, in <module>
    working_set = WorkingSet._build_master()
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 449, in _build_master
    ws.require(__requires__)
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 742, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gnureadline

installing ipython==2.3.0 with pip (succeeds)

(xxx)$ pip-accel uninstall ipython
Uninstalling ipython:
  /Users/jquast/.virtualenvs/xxx/bin/ipcluster
  /Users/jquast/.virtualenvs/xxx/bin/ipcluster2
  /Users/jquast/.virtualenvs/xxx/bin/ipcontroller
  /Users/jquast/.virtualenvs/xxx/bin/ipcontroller2
  /Users/jquast/.virtualenvs/xxx/bin/ipengine
  /Users/jquast/.virtualenvs/xxx/bin/ipengine2
  /Users/jquast/.virtualenvs/xxx/bin/iptest
  /Users/jquast/.virtualenvs/xxx/bin/iptest2
  /Users/jquast/.virtualenvs/xxx/bin/ipython
  /Users/jquast/.virtualenvs/xxx/bin/ipython2
  /Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/IPython
  /Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/ipython-2.3.0-py2.7.egg-info
Proceed (y/n)? y
  Successfully uninstalled ipython
(xxx)$ pip install ipython==2.3.0
Downloading/unpacking ipython==2.3.0
  Downloading ipython-2.3.0.tar.gz (11.9MB): 11.9MB downloaded
  Running setup.py egg_info for package ipython
                  readline: yes

Installing collected packages: ipython
  Running setup.py install for ipython
                  readline: yes
    checking package data

    Installing ipengine2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing iptest script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipython2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcluster2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcluster script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipython script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcontroller2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcontroller script to /Users/jquast/.virtualenvs/xxx/bin
    Installing iptest2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipengine script to /Users/jquast/.virtualenvs/xxx/bin
Successfully installed ipython
Cleaning up...
(xxx)$ ipython -V
2.3.0

uses old pip

in the requirements.txt it pins pip to an old version:

pip >= 1.4, < 1.5

But pip is up to version 6.0.8 https://pypi.python.org/pypi/pip

Any reason it uses an old version of pip?

Doesn't work with pip 7+

pip is pinned to between versions 6.0.8 and 6.2. pip is on version 7.1 now. Can we get pip-accel updated to work with newer pip? We need some of the newer features of pip and would like to use pip-accel.

Implement the ability to read configuration from a file

There are a few OS environment variables that control the behavior of pip-accel: 'PIP_ACCEL_CACHE', 'PIP_DOWNLOAD_CACHE', 'PIP_ACCEL_S3_BUCKET', 'PIP_ACCEL_S3_PREFIX', 'PIP_ACCEL_AUTO_INSTALL'.

It would be great if there was the option of reading these values out of configuration file as well.

My use case: I am creating jenkins worker AMIs via a jenkins job that uses Packer to trigger an ansible script.

Just to get this working, I'm using not setting the CACHE vars and I've hard coded the values for the PIP_ACCEL_S3_ env vars. I know, it's awful!

I'm doing this because there are too many levels of indirection for this to work easily otherwise. I can pass OS environment variables to the shell command that runs the play, but that doesn't flow through to the command that is run on the remote AMI as the worker is being built. More detailed info here.

This could probably be sorted out, but it would be much more straightforward if I could instead just write out a config file via a template, like I am for boto.

Perhaps instead of current implementation, config could be refactored to create a configuration object. Then it would be easier to set the values either via a config file or as extra vars on the command line.

UPDATE:
Note that the links above are now obsolete, as they point to a branch which now has subsequent commits and will probably be deleted in the future after merging into master.

Cache contention?

We tried using pip-accel on our CI environment. It failed with strange errors. Notice the error near the bottom of the trace where it says CRC check failed. Multiple jobs run on our jenkins slave simultaneously so am not sure if there is a cache contention since those jobs are sharing the same cache location ~/.pip-accel and ~/.pip

Traceback (most recent call last):
11:02:42   File "/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/bin/pip-accel", line 9, in <module>
11:02:42     load_entry_point('pip-accel==0.12', 'console_scripts', 'pip-accel')()
11:02:42   File "/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 106, in main
11:02:42     install_requirements(requirements)
11:02:42   File "/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 233, in install_requirements
11:02:42     install_binary_dist(members, prefix=install_prefix, python=python)
11:02:42   File "/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/local/lib/python2.7/site-packages/pip_accel/bdist.py", line 206, in install_binary_dist
11:02:42     for member, from_handle in members:
11:02:42   File "/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/local/lib/python2.7/site-packages/pip_accel/bdist.py", line 76, in get_binary_dist
11:02:42     for member in archive.getmembers():
11:02:42   File "/usr/lib/python2.7/tarfile.py", line 1805, in getmembers
11:02:42     self._load()        # all members, we first have to
11:02:42   File "/usr/lib/python2.7/tarfile.py", line 2380, in _load
11:02:42     tarinfo = self.next()
11:02:42   File "/usr/lib/python2.7/tarfile.py", line 2315, in next
11:02:42     self.fileobj.seek(self.offset)
11:02:42   File "/usr/lib/python2.7/gzip.py", line 429, in seek
11:02:42     self.read(1024)
11:02:42   File "/usr/lib/python2.7/gzip.py", line 256, in read
11:02:42     self._read(readsize)
11:02:42   File "/usr/lib/python2.7/gzip.py", line 303, in _read
11:02:42     self._read_eof()
11:02:42   File "/usr/lib/python2.7/gzip.py", line 342, in _read_eof
11:02:42     hex(self.crc)))
11:02:42 IOError: CRC check failed 0x6eda328f != 0x74cdee77L
11:02:42 [ERROR] '/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/bin/python /var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/bin/pip-accel install -U -r /var/lib/jenkins/workspace/build-component-virtualenv-debian7/etc/venv-requirements.txt' failed with return code 1
11:02:42 
11:02:42 '/var/lib/jenkins/workspace/build-component-virtualenv-debian7/virtualenv/bin/python /var/lib/jenkins/workspace/build-component-virtual

A build directory in pip cache causes failure

pip can create a temporary build directory as a subdirectory of the download cache. If this hasn't been cleaned up successfully, then pip-accel fails:

$ ~/.virtualenvs/jenkins-job-builder/bin/pip-accel install jenkins-job-builder
Traceback (most recent call last):
  File "/home/davidf/.virtualenvs/jenkins-job-builder/bin/pip-accel", line 9, in <module>
    load_entry_point('pip-accel==0.13.2', 'console_scripts', 'pip-accel')()
  File "/home/davidf/.virtualenvs/jenkins-job-builder/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 104, in main
    initialize_directories()
  File "/home/davidf/.virtualenvs/jenkins-job-builder/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 426, in initialize_directories
    update_source_dists_index()
  File "/home/davidf/.virtualenvs/jenkins-job-builder/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 358, in update_source_dists_index
    archive_path = os.path.join(source_index, add_extension(download_path, archive_name))
  File "/home/davidf/.virtualenvs/jenkins-job-builder/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 385, in add_extension
    handle = open(download_path, 'rb')
IOError: [Errno 21] Is a directory: '/home/davidf/Download/pip-cache/build'

Support S3 compatible as cache backend

Riak CS and Swift are widely used, it will be great to be able to store binary cache in Riak CS or Swift.

Fallback to 'python setup.py bdist' when 'python setup.py bdist_dumb' fails?

One of the entries in my requirements file is paver, but pip-accel is choking on installing it.
You can reproduce with:

pip-accel install paver -vv --no-clean --build /tmp

This will give you something like:

pip_accel.bdist[79437] DEBUG Executing external command: /Users/foo/.virtualenvs/test/bin/python setup.py bdist_dumb --format=tar
Traceback (most recent call last): paver (1.2.3) .. 
  File "/Users/foo/.virtualenvs/test/bin/pip-accel", line 9, in <module>
    load_entry_point('pip-accel==0.13.5', 'console_scripts', 'pip-accel')()
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/__init__.py", line 117, in main
    install_requirements(requirements)
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/__init__.py", line 280, in install_requirements
    install_binary_dist(members, prefix=install_prefix, python=python)
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/bdist.py", line 248, in install_binary_dist
    for member, from_handle in members:
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/bdist.py", line 84, in get_binary_dist
    raw_file = build_binary_dist(package, version, directory, python=python)
  File "/Users/foo/.virtualenvs/test/lib/python2.7/site-packages/pip_accel/bdist.py", line 156, in build_binary_dist
    filenames = os.listdir(dist_directory)
OSError: [Errno 2] No such file or directory: '/tmp/paver/dist'

And now check out paver's build directory with:

ls -l /tmp/paver

Windows / Appveyor - same version of cached file - invalidated and built again

I am using pip-accel with appveyor in my project and pip-accel invalidates even the same version of a package that is already cached:

pip_accel.bdist    INFO Invalidating old numpy (1.10.1) binary (source is newer) ..
pip_accel.bdist    INFO Building numpy (1.10.1) binary distribution ..

Url for appveyor build: https://ci.appveyor.com/project/matysek/pyinstaller-ewwmo/build/181/job/6aj9u6f8lsfbpej9

In the build appveyor is instructed to cache these paths:

C:\Users\appveyor\AppData\Local\pip
C:\Users\appveyor\AppData\Roaming\.pip-accel

I've seen already similar issue when porting pip-accel to windows.
My workaround was to ignore the mtime.
It looks like mtime is not reliable when appveyor caches some files to any external storage between builds.

Doesn't install packages from source repository

I'm having issues with pip-accel when installing from a repository. This is the command I am using:

source virtenv/bin/activate
pip-accel install -e git+git://github.com/asavoy/django-mobile.git@b6088e1c34#egg=django-mobile

It appears to install OK. But when I enter a python console:

>>> import django_mobile
ImportError: No module named django_mobile

The same install command works when using pip.

It seems to fail with other packages too. I'm actually installing from a requirements.txt file, and other packages in there that are to install from a source repository, also don't seem to be importable.

pip 1.5 error - need more than 2 values to unpack error

pip-accel seems incompatible with pip v1.5

$ pip-accel install -r requirements.txt
INFO Unpacking local source distributions ..
INFO Executing command: pip install --download-cache=/root/.pip/download-cache --find-links=file:///var/cache/pip-accel/sources --build-directory=/tmp/tmpLyTF4T --no-index -r requirements.txt --no-install
Traceback (most recent call last):
File "/srv/virtualenv/1388883995/bin/pip-accel", line 9, in <module>
  load_entry_point('pip-accel==0.11', 'console_scripts', 'pip-accel')()
File "/srv/virtualenv/1388883995/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 99, in main
  requirements = unpack_source_dists(arguments, build_directory)
File "/srv/virtualenv/1388883995/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 175, in unpack_source_dists
  build_directory=build_directory)
File "/srv/virtualenv/1388883995/local/lib/python2.7/site-packages/pip_accel/__init__.py", line 256, in run_pip
    cmd_name, options, args, parser = parseopts(command_line[1:])
ValueError: need more than 2 values to unpack

pip-accel does not honor -q (--quiet)

pip-accel -q has output

(xxx)$ pip-accel -q install blessed
2014-10-24 16:34:05 IO-PHX-L-468 pip_accel[40554] INFO Unpacking local source distributions ..
2014-10-24 16:34:05 IO-PHX-L-468 pip_accel[40554] INFO Executing command: pip -q install --download-cache=/Users/jquast/.pip/download-cache --find-links=file:///Users/jquast/.pip-accel/sources --build-directory=/var/folders/k3/ssq0mds55pn0lg4v0lr9cmp48g76w2/T/tmpaen61Z --no-index blessed --no-install
  Could not find any downloads that satisfy the requirement blessed
No distributions at all found for blessed
Storing complete log in /Users/jquast/.pip/pip.log
2014-10-24 16:34:05 IO-PHX-L-468 pip_accel[40554] WARNING We don't have all source distributions yet!
2014-10-24 16:34:05 IO-PHX-L-468 pip_accel[40554] INFO Downloading source distributions ..
2014-10-24 16:34:05 IO-PHX-L-468 pip_accel[40554] INFO Executing command: pip -q install --download-cache=/Users/jquast/.pip/download-cache --find-links=file:///Users/jquast/.pip-accel/sources --build-directory=/var/folders/k3/ssq0mds55pn0lg4v0lr9cmp48g76w2/T/tmpaen61Z blessed --no-install
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel[40554] INFO Finished downloading source distributions in 1.39 second.
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel[40554] WARNING pip failed, retrying (2/10) ..
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel[40554] INFO Unpacking local source distributions ..
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel[40554] INFO Executing command: pip -q install --download-cache=/Users/jquast/.pip/download-cache --find-links=file:///Users/jquast/.pip-accel/sources --build-directory=/var/folders/k3/ssq0mds55pn0lg4v0lr9cmp48g76w2/T/tmpaen61Z --no-index blessed --no-install
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel[40554] INFO Unpacked local source distributions in 0.33 seconds.
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel[40554] INFO Installing from binary distributions ..
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel.bdist[40554] INFO Building binary distribution of blessed (1.9.4) ..
2014-10-24 16:34:07 IO-PHX-L-468 pip_accel.bdist[40554] INFO Finished building blessed (1.9.4) in 0.32 seconds.
2014-10-24 16:34:08 IO-PHX-L-468 pip_accel.bdist[40554] INFO Building binary distribution of wcwidth (0.1.1) ..
2014-10-24 16:34:08 IO-PHX-L-468 pip_accel.bdist[40554] INFO Finished building wcwidth (0.1.1) in 0.21 seconds.
2014-10-24 16:34:08 IO-PHX-L-468 pip_accel[40554] INFO Finished installing all requirements in 0.91 seconds.
2014-10-24 16:34:08 IO-PHX-L-468 pip_accel[40554] INFO Done! Took 2.65 seconds to install 2 packages.
(xxx)$ pip-accel uninstall blessed
Proceed (y/n)? y

whereas standard pip does not

(xxx)$ pip -q install blessed
(xxx)$

pip-accel not updating package

Maybe I'm making an obvious mistake, or my use case might not be suitable for pip-accel. Sorry in advance :)

I'm using pip to install/update packages in a ve. When using pip-accel my package is not updated to the latest version present on my pypi server, probably because of the --no-index option used for pip? Is there any way I can configure pip-accel to scan my pypi server to check for an updated package?

I was expecting this behaviour only when using pinned versions.

> (ve)[idgen@test3-ipw idgen]$ pip-accel -v install -U --extra-index-url https://user:[email protected]/simple/ datamodel
>2015-07-01 11:10:55 test3-ipw pip_accel[21722] INFO Unpacking distribution(s) ..
>2015-07-01 11:10:55 test3-ipw pip_accel[21722] INFO Executing command: pip install --download=/home/idgen/.pip-accel/sources --find-links=file:///home/idgen/.pip-accel/sources --no-index --no-clean --build-directory=/tmp/pip-accel-build-dir-F5f9LE -v -U --extra-index-url https://user:[email protected]/simple/ datamodel
>2015-07-01 11:10:55 test3-ipw pip.commands.install[21722] INFO Ignoring indexes: https://pypi.python.org/simple,https://user:[email protected]/simple/
>2015-07-01 11:10:55 test3-ipw pip.index[21722] DEBUG 0 location(s) to search for versions of datamodel:

Implement source distribution archive caching for Amazon S3 backend

Pull request #33 introduced a second level cache in Amazon S3 which is currently only used to cache binary distribution archives.

Due to pip-accel's current architecture it can't do anything useful without unpacked source distributions. The result is that, even though your complete requirement set has been cached as binary distributions, you still need to fetch all of the source distributions from PyPI (or the local download cache) before you can install the cached binary distributions.

In the case of ephemeral local storage, the download cache may frequently be empty so pip-accel has to use pip to search around PyPI which is slow. By storing the download cache in Amazon S3 as well this process could be further optimized.

I consider this a "nice to have" feature. It's also a bit non trivial to implement as explained by me in a comment on pull request #33:

Right now only the binary cache can be stored in Amazon S3. It is possible but non trivial to add support for caching source distribution archives in Amazon S3. If @jzoldak really sees value in this I may try to implement it soon. The main difficulty is that for the binary cache a simple get() / put() interface suffices, but the source index requires an index.html that can be scanned by pip install --find-links=.... Because Amazon S3 does not support server side directory listings this will have to be implemented in pip-accel (one way or another).

--no-install option is deprecated for pip 1.5

--no-install option is deprecated for pip 1.5

http://www.pip-installer.org/en/latest/news.html#id1

pypa/pip#906

Upgrade to pip 8

I'm creating this issue just to let pip-accel users know that I'm aware of pip 8 being released and will be upgrading pip-accel to use pip 8 soon. For now everything is fine due to the way pip-accel pins the pip dependency (pip will just spam messages to the terminal about not being up to date). Judging by pip's release notes most of the changes shouldn't have a big impact on pip-accel. Only the following items might prove to be relevant and/or tricky from a first cursory reading:

Implement a top-level pip download command and deprecate pip install --download.
Include the functionality of peep into pip, allowing hashes to be baked into a requirements file and ensuring that the packages being downloaded match one of those hashes. This is an additional, opt-in security measure that, when used, removes the need to trust the repository.

bug in setup.py

The same modules are defined with packages and py_modules, causing warnings at install time. You should keep packages and remove py_modules.

Binary cache doesn't work with scipy

Steps to reproduce:

virtualenv ve
source ve/bin/active
pip install pip-accel
pip-accel install scipy

Then destroy virtualenv and repeat, scipy still compiles.

Reduce printed output by default?

pip-accel prints out thousands and thousands of lines of stuff which is typically uninteresting. Things like:

    Skipping link file:///home/buildbot/.pip-accel/sources/fde86a64744f4e1e22df3327596e7f680fac340e.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/coverage-3.4.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/iso8601-0.1.4.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/ssh-1.8.0.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/mock-0.7.2.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/python-openid-2.2.5.zip; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/envoy-0.0.2.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/python-novaclient-2.9.0.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/httplib2-0.7.6.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/prettytable-0.5.zip; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/setproctitle-1.1.7.zip; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/wsgi_intercept-0.5.0.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/prettytable-0.6.tar.bz2; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/Werkzeug-0.8.3.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/six-1.2.0.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/simplejson-3.1.3.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/beanstalkc-0.3.0.tar.gz; wrong project name (not keyring)
    Skipping link file:///home/buildbot/.pip-accel/sources/chardet-1.0.1.tar.gz; wrong project name (not keyring)

Would it be possible to suppress all that output unless a --verbose flag is passed in?

How to use with tox

At the moment we use tox to build our virtualenvs before tests.

It doesn't seem possible to call an externally installed pip-accel from within another virtualenv (it installs it to the host virtualenv), nor to install pip-accel into the virtualenv before the install process of tox starts

Doesn't run on windows

Fresh install of Python 3.4.3, ran python -m pip install pip-accel, success.

pip-accel -h, then gives ImportError: No module named 'pwd'

Support custom pip command

I have my python3 versions of pip and pip-accel installed as pip3 and pip-accel3 respectively. However, when I run pip-accel3, it installs packages with pip (the python2 version). Is there a way to specify which pip version to use?

[BUG] pip-accel 0.12.8 breaks Ubuntu compatibility

Version 0.12.8 moves installed packages from dist-packages into site-packages, however, Ubuntu 12.04 sys.path does not contain site-packages. A temporary workaround is to either revert to 0.12.7 or use PYTHONPATH environment variable.

Wheel binary format

Great tool!
Have you considered using Wheels as binary format? They're pretty advanced and got accepted through some PEPs. Here it is a couple of interesing links:

https://wheel.readthedocs.org
https://bitbucket.org/dholth/wheel

Enable binary cache format revisions to coexist

Problem:

In pull request #33 a new cache backend was introduced that stores binary distribution archives on Amazon S3. However the CACHE_FORMAT_REVISION that is used to clear the binary cache when a change to the format is made is only stored in the ~/.pip-accel directory on the local file system.

Solution:

Similar to ~/.pip-accel/version.txt the cache format revision should be stored inside the Amazon S3 bucket so that the bucket contents can be cleared when the cache format revision changes.

Errors when building/installing ssh==1.7.14

Installing ssh==1.7.14 leads to this error, when installed with pip-accel:

(pip-accel) Building binary distribution of ssh (1.7.14) ..
running bdist_dumb
running build
running build_py
installing to build/bdist.macosx-10.8-intel/dumb
running install
running install_lib
running install_egg_info
running egg_info
writing requirements to ssh.egg-info/requires.txt
writing ssh.egg-info/PKG-INFO
writing top-level names to ssh.egg-info/top_level.txt
writing dependency_links to ssh.egg-info/dependency_links.txt
reading manifest file 'ssh.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ssh.egg-info/SOURCES.txt'
removing 'build/bdist.macosx-10.8-intel/dumb/Users/nvie/.virtualenvs/foo/lib/python2.7/site-packages/ssh-1.7.14-py2.7.egg-info' (and everything under it)
Copying ssh.egg-info to build/bdist.macosx-10.8-intel/dumb/Users/nvie/.virtualenvs/foo/lib/python2.7/site-packages/ssh-1.7.14-py2.7.egg-info
running install_scripts
Traceback (most recent call last):
  File "setup.py", line 67, in <module>
    **kw
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/bdist_dumb.py", line 123, in run
    owner=self.owner, group=self.group)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 392, in make_archive
    owner=owner, group=group)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/archive_util.py", line 237, in make_archive
    filename = func(base_name, base_dir, **kwargs)
TypeError: make_tarball() got an unexpected keyword argument 'owner'
(pip-accel) Failed to build binary distribution!

Installing with pip works just fine. Any clues?

Python 3.4 issue

After running:

$ pip-accel install -r requirements/devel.txt

I get the following error:

2014-05-23 15:26:25 benjamin pip-accel[24787] INFO Requirement already satisfied: cov-core>=1.6 in /home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages (from pytest-cov==1.6->-r requirements/testing.txt (line 6)).
2014-05-23 15:26:25 benjamin pip-accel[24787] INFO Requirement already satisfied: argh>=0.8.1 in /home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages (from watchdog==0.7.1->-r requirements/testing.txt (line 10)).
2014-05-23 15:26:25 benjamin pip-accel[24787] INFO Requirement already satisfied: coverage>=3.4 in /home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages (from cov-core>=1.6->pytest-cov==1.6->-r requirements/testing.txt (line 6)).
2014-05-23 15:26:25 benjamin pip-accel[24787] INFO Installing from binary distributions ..
Traceback (most recent call last):
  File "/home/bhe/.virtualenvs/mx-py34/bin/pip-accel", line 9, in <module>
    load_entry_point('pip-accel==0.12.2', 'console_scripts', 'pip-accel')()
  File "/home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages/pip_accel/__init__.py", line 113, in main
    install_requirements(requirements)
  File "/home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages/pip_accel/__init__.py", line 240, in install_requirements
    install_binary_dist(members, prefix=install_prefix, python=python)
  File "/home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages/pip_accel/bdist.py", line 210, in install_binary_dist
    for member, from_handle in members:
  File "/home/bhe/.virtualenvs/mx-py34/lib/python3.4/site-packages/pip_accel/bdist.py", line 59, in get_binary_dist
    tag = hashlib.sha1(version + url).hexdigest() if url else version
TypeError: Unicode-objects must be encoded before hashing

I'm on Python 3.4.0. OS: Archlinux x86_64

Disabling S3CacheBackend because it failed

Testing pip-accel and running into this error. There is a config file for both boto and pip-accel.

2016-02-24 17:22:04 ad60d292765e pip_accel.caches[8287] ERROR Disabling S3CacheBackend because it failed: 'module' object has no attribute 'get'
Traceback (most recent call last):
  File "/opt/wp/app/helios/virtualenv/local/lib/python2.7/site-packages/pip_accel/caches/__init__.py", line 192, in put
    backend.put(filename, handle)
  File "/opt/wp/app/helios/virtualenv/local/lib/python2.7/site-packages/pip_accel/caches/s3.py", line 190, in put
    with PatchedBotoConfig():
  File "/opt/wp/app/helios/virtualenv/local/lib/python2.7/site-packages/pip_accel/caches/s3.py", line 359, in __init__
    self.unbound_method = ConfigParser.get
AttributeError: 'module' object has no attribute 'get'

Usage of corrupted package build

During building celery, occurred error:

2014-11-13 11:23:46 ns3314108.ovh.net pip_accel.bdist[12645] INFO Building binary distribution of celery (3.1.16) ..
2014-11-13 11:23:47 ns3314108.ovh.net pip_accel.bdist[12645] INFO Finished building celery (3.1.16) in 1.31 second.
Traceback (most recent call last):
  File "/tmp/test-virtualenv/bin/pip-accel", line 9, in <module>
    load_entry_point('pip-accel==0.14.1', 'console_scripts', 'pip-accel')()
  File "/tmp/test-virtualenv/lib/python3.3/site-packages/pip_accel/__init__.py", line 121, in main
    install_requirements(requirements, cache)
  File "/tmp/test-virtualenv/lib/python3.3/site-packages/pip_accel/__init__.py", line 285, in install_requirements
    install_binary_dist(members, prefix=install_prefix, python=python)
  File "/tmp/test-virtualenv/lib/python3.3/site-packages/pip_accel/bdist.py", line 247, in install_binary_dist
    for member, from_handle in members:
  File "/tmp/test-virtualenv/lib/python3.3/site-packages/pip_accel/bdist.py", line 91, in get_binary_dist
    with open(transformed_file, 'rb') as handle:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery-3.1.16.linux-x86_64.tar'

This error didn't stop pip-accel from using not built version of celery, and caused celery package to missfunction. Only way to fix this for that version of celery is manually deletion of that celery build.

In my opinion manual fixing shouldn't be required for every error that occurs during building packages.
Additionally tmp folder doesn't guarantee file persistence event between two shell commands, so opening of temporary files should be better handled.

Downgrading a package doesn't work

$ pip-accel install xlrd==0.9.2
...
$ pip-accel install xlrd==0.7.1 --upgrade

(pip-accel) Unpacking local source distributions ..
(pip-accel) Executing command: pip install --download-cache=/home/harrovanderklauw/.pip/cache --find-links=file:///home/harrovanderklauw/.pip-accel/sources --no-index xlrd==0.7.1 --upgrade -v -v --no-install
  Ignoring indexes: http://pypi.python.org/simple/
  Downloading/unpacking xlrd==0.7.1
    Running setup.py egg_info for package xlrd
      running egg_info
      writing pip-egg-info/xlrd.egg-info/PKG-INFO
      writing top-level names to pip-egg-info/xlrd.egg-info/top_level.txt
      writing dependency_links to pip-egg-info/xlrd.egg-info/dependency_links.txt
      warning: manifest_maker: standard file '-c' not found
      reading manifest file 'pip-egg-info/xlrd.egg-info/SOURCES.txt'
      writing manifest file 'pip-egg-info/xlrd.egg-info/SOURCES.txt'
    Requested xlrd==0.7.1, but installing version 0.9.2
  Successfully downloaded xlrd
(pip-accel) Unpacked local source distributions in 0.57 seconds.
(pip-accel) Found 0 requirements in pip's output.
(pip-accel) No requirements found in pip's output, probably there's nothing to do.

As you can see it installs 0.9.2 instead of downgrading to 0.7.1!

Using --download-cache can cause builds to fail

If you add the --download-cache option to pip-accel install, it can result in a build directory being present which stops pip continuing the build on the next run (because pip complains that the source build is already present).

Without using the --download-cache option, it works fine.

I guess this is a change in pip since the latest release, as it used to (prior to about a week ago) work fine.

Running pip-accel remote sudoed as a user without home folder breaks on line 493

File "/srv/sites/trello_burndown/releases/8acf54903e6b-1304290842/.env/bin/pip-accel", line 8, in
load_entry_point('pip-accel==0.7.2', 'console_scripts', 'pip-accel')()
File "/srv/sites/trello_burndown/releases/8acf54903e6b-1304290842/.env/local/lib/python2.7/site-packages/pip_accel.py", line 89, in main
initialize_directories()
File "/srv/sites/trello_burndown/releases/8acf54903e6b-1304290842/.env/local/lib/python2.7/site-packages/pip_accel.py", line 493, in initialize_directories
os.makedirs(directory)
File "/srv/sites/trello_burndown/releases/8acf54903e6b-1304290842/.env/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/srv/sites/trello_burndown/releases/8acf54903e6b-1304290842/.env/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/harro/.pip'

Add option for terse logging.

When using pip-accel for e.g. Travis-CI, the log output is much to long:

2015-10-19 23:50:55 testing-worker-linux-docker-357adccc-3199-linux-7 pip.req.req_set[3303] INFO Requirement already satisfied (use --upgrade to upgrade): wheel in /home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages (from -r tests/requirements.txt (line 9))
2015-10-19 23:50:55 testing-worker-linux-docker-357adccc-3199-linux-7 pip.download[3303] INFO File was already downloaded /home/travis/.cache/pip-accel/sources/check_manifest-0.25-py2.py3-none-any.whl

Neither timestamp nor hostname, not even the process-id is meaning full here. Instead aligning the length of the logger names would ease reading the logs.

Please add an option for setting the log format, or at least for disabling timestamp and hostname. Another option would be to add an option to use the standard logging module instead of colorlogs.

(When thinking about this, the later option would a) quickly be implemented, b) allow configuring the logformat later c) reuse the pip-accel config-file for defining the loggers (see https://docs.python.org/2/howto/logging.html#configuring-logging)

py.test in python 2.6 environment doesn't install correctly

Steps:

Make sure pip-accel cache contains no pytest
mkvirtualenv test -ppython2.6
workon test
pip-accel install pytest==2.3.5
py.test

Traceback (most recent call last):
  File "/home/harrovanderklauw/virtualenvs/test/bin/py.test", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/home/harrovanderklauw/virtualenvs/test/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg/pkg_resources.py", line 2815, in <module>
    working_set.require(__requires__)
  File "/home/harrovanderklauw/virtualenvs/test/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg/pkg_resources.py", line 690, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/harrovanderklauw/virtualenvs/test/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg/pkg_resources.py", line 588, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pytest==2.3.5
zsh: exit 1     py.test

Traceback (most recent call last):
File "/home/harrovanderklauw/virtualenvs/test/bin/py.test", line 5, in
from pkg_resources import load_entry_point
File "/home/harrovanderklauw/virtualenvs/test/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg/pkg_resources.py", line 2815, in
working_set.require(requires)
File "/home/harrovanderklauw/virtualenvs/test/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg/pkg_resources.py", line 690, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/harrovanderklauw/virtualenvs/test/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg/pkg_resources.py", line 588, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pytest==2.3.5
zsh: exit 1 py.test

Binary cache doesn't take Python version into account

A colleague of mine pointed out that when switching Python versions, pip-accel blindly installs cached binary distributions. Instead, the Python version should be included in the filenames used inside the binary cache. Should be an easy fix, but there are several other variables to consider...

Ignore already installed requirements

I have several custom packages that I install before running pip-accel to install the rest of my requirements, however I like to keep those custom ones in the requirements file, as pip just skips them over since they're already present (and it's a problem if they're not).

I just upgraded to the latest pip-accel 0.26.2 and pip 6.0.8 and pip-accel is now failing when installing my requirements.txt because it can't find these already installed packaged. Is there an option to make this work?

This works using pip 1.4.1 and pip-accel 0.22.2.

Thanks!

Store cache on AWS S3?

Hi,

Would you be interested in a pull request for some code that lets you put the caches (PIP_DOWNLOAD_CACHE, PIP_ACCEL_CACHE) in an AWS S3 bucket?

We're using the Atlassian Elastic Bamboo Continuous Integration system, and want to store our binary distributions made by pip-accel somewhere all our Elastic Bamboo build workers can get them. This would make our builds a lot faster.

cheers
adam

paylogic / pip-accel Goto Github PK

pip-accel's People

Contributors

Stargazers

Watchers

Forkers

pip-accel's Issues

preparing environment for pip-accel

installing ipython==2.3.0 with pip-accel (fails)

installing ipython==2.3.0 with pip (succeeds)

pip-accel -q has output

whereas standard pip does not

Recommend Projects

Recommend Topics

Recommend Org

Jobs