mardiros / pyshop

A Private PyPI server written in pyramid

pyshop

Important

This project is unmaintained

Getting Started

Pyshop is a private repository for python packages.

The aim is to split private projects in distinct private packages and keep a setup.py clean and working, by declaring all dependencies, exactly as public packages on PyPI.

Pyshop also proxies and caches packages from PyPI safely using SSL and checking server certificate.

Pyshop uses clear and simple ACLs to manage privileges:

• an installer group that can only download release files,
• a developer group that can download and upload release files and browse the website,
• an admin group that has developer privileges and accounts management.

Since pyshop is intended to host private packages, every user, including pip, must be authenticated by login and password.

Installation

Using A virtualenv with Python 3

$ cd /srv
$ sudo mkdir pyshop
$ sudo chown $(whoami) pyshop
$ cd pyshop
$ pyvenv .
(pyshop)$ source bin/activate
(pyshop)$ pip install "pyshop[waitress]"
(pyshop)$ cp pyshop.sample.ini pyshop.ini
(pyshop)$ vim pyshop.ini  # change at least the pyshop.cookie_key setting
(pyshop)$ pyshop_setup pyshop.ini  # Create the database
(pyshop)$ pserve pyshop.ini        # start pyshop CTRL+C to stop

Note

If you are using python2 and the pyshop you may not have the pyshop.sample.ini file, you can download it with:

(pyshop)$ curl -o pyshop.ini https://raw.githubusercontent.com/mardiros/pyshop/master/pyshop.sample.ini

You should edit the pyshop.ini file in order to configure the pyshop.cookie_key and the host:port that hosts the service. When the server is running visit the website, http://localhost:8000/ by default, to check everything is fine.

For production usage, you should create accounts with the developer group. Visit http://localhost:8000/pyshop/user with the admin account to create accounts.

You also should also use an https reverse proxy. Python packaging core uses HTTP basic authentication: it sends user/password in clear.

The pythop.sample.ini file use waitress as the default WSGI server, but, if you are familiar with another WSGI server that support paste format, you could use it.

Daemonize with systemd on linux

Pyramid 1.8 has removed deamonized options you have to use a process manager.

Here is a simple way to daemonise it undex linux that use systemd

(pyshop)$ curl -o pyshop.service https://raw.githubusercontent.com/mardiros/pyshop/master/pyshop.sample.service
(pyshop)$ sudo mv pyshop.service /etc/systemd/system/pyshop.service

Note

you may edit the pyshop.service file to adapt path in case you install it

_`Pyramid 1.8 has removed deamonized options`: http://docs.pylonsproject.org/projects/pyramid/en/latest/whatsnew-1.8.html#backwards-incompatibilities

Using Docker

Currently, there is an image of pyshop used for development purpose, it support both MySQL and PostgreSQL. The PostgreSQL integration is fully operation, you can run a new Pyshop install using docker-compose, with the command:

docker-compose up pgpyshop

It will create the database with the default pyshop users:

• privileged user: login admin, password: changeme
• unprivileged user: login pip, password changeme

If you want to use a different orchestrator, you have to link the postgresql container to Pyshop container with the name postgresql.localdomain

The MySQL support does not automate the database setup right now.

The official Docker image of Pyshop is available here on the Docker Hub:

https://registry.hub.docker.com/u/mardiros/pyshop/

Configuring your environment

Here are all configuration files you will need to modify for usual python tools to use your newly deployed private repository.

~/.pip/pip.conf

Configuration used by pip. This is a user file, you can set a developer or the generic pip account.

[global]
# when mirroring a package, pyshop retrieves information from PyPI and
# stores it in its database. Be patient, it is not so long.
default-timeout = 120
timeout = 120

[install]
index-url = http://pip:changeme@localhost:8000/simple/

[search]
index = http://pip:changeme@localhost:8000/pypi

Note

If you are using a WSGI server that kills requests if it is too long, like uWSGI or gunicorn, set an appropriate timeout for this service too.

Note

The search funciton is not working with Python 3

setup.cfg and pydistutils.cfg

setup.cfg and pydistutils.cfg are used when running python setup.py develop to install your package or when using easy_install. You should use a generic account with installer privileges only, shared by all developers.

This setting can be set per project or in user $HOME (see setuptools documentation for details)

[easy_install]
index-url = http://pip:changeme@localhost:8000/simple/

This should work now:

python setup.py develop

~/.pypirc

Configuration used by setuptools to upload files. All developers should have this configuration in their $HOME to upload packages.

[distutils]
index-servers =
    pyshop

[pyshop]
username: admin  # or create an account in pyshop admin interface
password: changeme
repository: http://localhost:8000/simple/

This should work now:

python setup.py sdist upload -v -r pyshop

Alternatives

• pypiserver: https://pypi.python.org/pypi/pypiserver
• localshop: http://pypi.python.org/pypi/localshop
• djangopypi: http://pypi.python.org/pypi/djangopypi
• chishop: http://pypi.python.org/pypi/chishop

pyshop's Issues

LDAP Documentation/Implementation Suggestions

Hey there, so a couple suggestions for fixing your LDAP documentation and development of Pyshop based on our companies woes with the documentation/implementation:

# name of the atrribute where login is
# ex : uid
pyshop.ldap.login_attr = uid ##This didn't work
pyshop.ldap.login_attr = uid={0} ##This worked

• Should mention that uid might need to be set equal to {0} (not positive if that's an issue specific to us)

pyshop.ldap.login_attr =
pyshop.ldap.email_attr =

• These should be mandatory, I agree

pyshop.ldap.first_name_attr =
pyshop.ldap.last_name_attr =

• These should be either optional, or allow you to set a full_name_attr if the LDAP server's implementation does not store first and last names apart.
  • You can't assume that you can get a fullname and there won't be any spaces in the first or last name, so we are going to have to edit our local source so we're able to use it for now (probably just have the fullname value in the first name and a space in the last name)

Will update this more as we progress through getting the rest of the LDAP plugin to work!.

Packages not updated, sometimes...

I have repeated experienced an issue where PyPi will release a new version of a package and pyshop is not updating properly.

I submitted a patch #46 that fixes part of this, but I am currently unable to located the source of the issue.

Current problem with Pecan:

They released a new version 0.8.3 on 1/12/2015. You can see from the logs below that it recognized refresh interval (24 hours) and issued a refresh on the package. But for some reason, it did not find the new version and cache it.

2015-01-12 19:52:47,675 INFO  [pyshop.security][Dummy-4] [104.130.1.123] GET http://wheel.cldbckp.com/test/simple/pecan/
28303 2015-01-12 19:52:47,854 INFO  [pyshop.views.base][Dummy-4] dispatch view Show
28304 2015-01-12 19:52:47,855 DEBUG [pyshop.views.simple][Dummy-4] "2 days, 21:12:23.652369" > "1 day, 0:00:00": True
28305 2015-01-12 19:52:47,855 INFO  [pyshop.views.simple][Dummy-4] refresh package pecan
28306 2015-01-12 19:52:47,954 INFO  [pyshop.views.simple][Dummy-4] package pecan mirrored
28307 2015-01-12 19:52:48,281 INFO  [pyshop.security][Dummy-3] [104.130.1.123] GET http://wheel.cldbckp.com/test/simple/pecan/
28308 2015-01-12 19:52:48,458 INFO  [pyshop.views.base][Dummy-3] dispatch view Show
28309 2015-01-12 19:52:48,460 DEBUG [pyshop.views.simple][Dummy-3] "0:00:00.779265" > "1 day, 0:00:00": False
28310 2015-01-12 19:52:48,464 INFO  [pyshop.views.simple][Dummy-3] package pecan mirrored

If I check manually with pyshop_shell, I can see that the pypi.proxy finds the package and lists it.

(test)root@bus1a-cbo-iad:/opt/pyshop/test# pyshop_shell test.ini
Welcome to PyShop shell.
In [1]: pkg = Package.by_name(session, 'pecan')

In [2]: pkg.versions
Out[2]:
[u'0.8.2',
 u'0.8.1',
 u'0.8.0',
 u'0.7.0',
 u'0.6.1',
 u'0.6.0',
 u'0.5.0',
 u'0.4.5',
 u'0.4.4',
 u'0.4.3',
 u'0.4.2',
 u'0.4.1',
 u'0.3.2',
 u'0.3.1',
 u'0.3.0',
 u'0.2.4',
 u'0.2.3',
 u'0.2.2',
 u'0.2.1',
 u'0.2.0',
 u'0.2.0b2',
 u'0.2.0a']

In [3]: from pyshop.helpers import pypi

In [4]: pypi.proxy.package_releases('pecan', True)
Out[4]:
['0.8.3',
 '0.8.2',
 '0.8.1',
 '0.8.0',
 '0.7.0',
 '0.6.1',
 '0.6.0',
 '0.5.0',
 '0.4.5',
 '0.4.4',
 '0.4.3',
 '0.4.2',
 '0.4.1',
 '0.3.2',
 '0.3.1',
 '0.3.0',
 '0.2.4',
 '0.2.3',
 '0.2.2',
 '0.2.1',
 '0.2.0',
 '0.2.0b2',
 '0.2.0a']

When I force a refresh by using the refresh button, it pulls it in perfectly.

2015-01-14 18:33:00,290 INFO  [pyshop.views.base][Dummy-4] dispatch view Show
3307995 2015-01-14 18:33:00,291 INFO  [pyshop.views.simple][Dummy-4] refresh package pecan
3307996 2015-01-14 18:33:00,338 INFO  [pyshop.views.simple][Dummy-4] Create release 0.8.3 for package pecan
3307997 2015-01-14 18:33:00,339 INFO  [pyshop.views.simple][Dummy-4] Looking for author Jonathan LaCour
3307998 2015-01-14 18:33:01,486 INFO  [pyshop.views.simple][Dummy-4] package pecan mirrored

PEP 503 package name canonicalization

PyShop doesn't seems to canonicalize package names according the PEP 503. See: pypa/pip@8e236dd

As a result a package mirrored by PyShop can't be install if containing a dot with pip >= 8.1.2 because of the non-canonicalized name.

Old version in PyPi

Hey!

A heads up that the upstream version in pypi doesn't seem to integrate the non-interactive changes to the install script from 6e5f842.

I pip installed pyshop, and my install bin __main__ methods looks like:

def main(argv=sys.argv):
    if len(argv) != 2:
        usage(argv)
    config_uri = argv[1]
    setup_logging(config_uri)
    settings = get_appsettings(config_uri)
    engine = create_engine('pyshop', settings, scoped=False)
    populate(engine)
    dispose_engine('pyshop')

Using python2.7, ubuntu trusty.

Great job, loving this package 😄

Cannot resolve some packages like django

pyshop does not seem to be resolving certain packages but others are just file, e.g. I do not seem to be able to do pip install django but pip install flask is fine.

Collecting django-roles
  Downloading http://pyshop......com/repository/12425/django_roles-0.0.4.tar.gz
Collecting django>=1.7.3 (from django-roles)
  Could not find a version that satisfies the requirement django>=1.7.3 (from django-roles) (from versions: )
No matching distribution found for django>=1.7.3 (from django-roles)

pip install django:

Collecting django
  Could not find a version that satisfies the requirement django (from versions: )
No matching distribution found for django

Can't mirror new package version

Hi mardiros, it's me again ;)

I have no idea what has happened, but after upgrading to 0.8 I mirrored a package I recently created pyramid_fullauth, with version 0.0.3 at the time.
However after uploading 0.0.4 to pypi, pyshop can't seem to get a handle of the new version... even if I try ti install it with a specific version number required... I can't seem to force the refresh on the package here.

Just a quick thought, could it be related to recent cheeseshop change of serving download links?

Wrong sorting of package versions

If you add a package version 0.0.9 and 0.0.10, the 0.0.9 will be considered as the latest package, when it should be the 0.0.10.

It's at least for the web display, will have to check how it looks from the pip end.

Notes:

http://www.python.org/dev/peps/pep-0386/#the-new-versioning-algorithm

PyPI search broken

Last night, it appears that PyPi might have made a change to their search functionality. Several packages that have been installed 8,700 times or more are suddenly breaking and it is because they are no longer being returned in the _search_package function.

It appears to me that packages that include uppercase characters are being downcased by pip. Pyshop cannot make a match because the package names are stored as camel-case and so pyshop searches PyPi for a match and then uses that to find the package. But since PyPi's search is broken, its not returning any results which mean pyshop returns package not found.

Case in point, Babel and MarkupSafe.

If you search for babel on PyPi, it returns no results on their webpage or in pyshop's search_results. But the package is listed in /simple and you can navigate to them.

I added some debug lines.

015-05-19 20:17:30,964 INFO  [pyshop.security][Dummy-2] [162.209.100.94] GET http://wheel.cldbckp.com/prod/simple/babel/
2015-05-19 20:17:31,140 INFO  [pyshop.views.base][Dummy-2] dispatch view Show
2015-05-19 20:17:31,143 INFO  [pyshop.views.simple][Dummy-2] refresh package babel
2015-05-19 20:17:31,262 DEBUG [pyshop.views.simple][Dummy-2] Found 17 packages
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Type: <type 'list'>
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.3.5', 'name': 'babel-cli', 'summary': 'A command line interface for sending anything anywhere.'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.7.1', 'name': 'babel-obviel', 'summary': 'Obviel Template message extractor for Babel'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.1', 'name': 'babelchart', 'summary': 'Pluggable timeseries data converter'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.5.4', 'name': 'babelfish', 'summary': 'A module to work with countries and languages'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.4.0', 'name': 'django-babel', 'summary': 'Utilities for using Babel in Django'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.3.0', 'name': 'django-babel-underscore', 'summary': 'Implements a underscore extractor for django-babel.'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.0', 'name': 'flask-babel-utclocal-utils', 'summary': 'UTC to local (and vice versa) datetime conversion utilities for use with Flask-Babel.'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.2.2', 'name': 'frasco-babel', 'summary': 'I18n and L10n support for Frasco'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1', 'name': 'gears-babel', 'summary': 'Gears compiler for Babel (previously 6to5) http://babeljs.io/'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.0', 'name': 'genibabel', 'summary': 'Genetic Imaging Babel (GenIBabel)'}
2015-05-19 20:17:31,263 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '1.1.1', 'name': 'gpsbabel', 'summary': 'Python wrapper for GPSBabel project'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '2.0.0', 'name': 'nibabel', 'summary': 'Access a multitude of neuroimaging data formats'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '1.8.2', 'name': 'openbabel', 'summary': 'Python interface to the Open Babel chemistry library'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.3', 'name': 'pyifbabel', 'summary': 'A pure-Python implementation of the Treaty of Babel'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.0', 'name': 'radiobabel', 'summary': 'Interact with a number of online music services using a single unified API.'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.43', 'name': 'unbabel-py', 'summary': 'Python Wrapper around Unbabel HTTP API'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found: {'_pypi_ordering': False, 'version': '0.1.3', 'name': 'webassets-babel', 'summary': 'Babel filter for Webassets'}
2015-05-19 20:17:31,264 DEBUG [pyshop.views.simple][Dummy-2] Found 17, matched 0
2015-05-19 20:17:31,264 INFO  [pyshop.views.simple][Dummy-2] package babel has no versions

Search for markupsafe, PyPi shows a match but Pyshop doesnt list any.

2015-05-19 20:17:51,616 INFO  [pyshop.security][Dummy-1] [162.209.100.94] GET http://wheel.cldbckp.com/prod/simple/markupsafe/
2015-05-19 20:17:51,794 INFO  [pyshop.views.base][Dummy-1] dispatch view Show
2015-05-19 20:17:51,798 INFO  [pyshop.views.simple][Dummy-1] refresh package markupsafe
2015-05-19 20:17:52,006 DEBUG [pyshop.views.simple][Dummy-1] Found 0 packages
2015-05-19 20:17:52,006 DEBUG [pyshop.views.simple][Dummy-1] Type: <type 'list'>
2015-05-19 20:17:52,006 INFO  [pyshop.views.simple][Dummy-1] package markupsafe has no versions

Here is the output for Babel yesterday. It finds the package and lists the versions.

934918 2015-05-18 22:47:07,730 INFO  [pyshop.security][Dummy-4] [162.209.108.184] GET http://wheel.cldbckp.com/prod/simple/babel/
934919 2015-05-18 22:47:07,907 INFO  [pyshop.views.base][Dummy-4] dispatch view Show
934920 2015-05-18 22:47:07,910 INFO  [pyshop.views.simple][Dummy-4] refresh package babel
934921 2015-05-18 22:47:08,336 DEBUG [pyshop.views.simple][Dummy-4] Found 29, matched 1
934922 2015-05-18 22:47:08,350 INFO  [pyshop.views.simple][Dummy-4] mirror package Babel now
934923 2015-05-18 22:47:08,368 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user cmlenz
934924 2015-05-18 22:47:08,381 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user asmodai
934925 2015-05-18 22:47:08,398 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user asmodai
934926 2015-05-18 22:47:08,400 INFO  [pyshop.views.simple][Dummy-4] Looking for non local user mitsuhiko
934927 2015-05-18 22:47:08,408 DEBUG [pyshop.views.simple][Dummy-4] refreshing Babel package
934928 2015-05-18 22:47:08,411 INFO  [pyshop.views.simple][Dummy-4] No new version to mirror
934929 2015-05-18 22:47:08,411 DEBUG [pyshop.views.simple][Dummy-4] pypi versions: [u'1.3', u'1.2', u'1.1', u'1.0', u'0.9.6', u'0.9.5', u'0.9.4', u'0.9.3', u'0.9.2', u'0.9.1', u'0.9', u'0.8.1', u'0.8']
934930 2015-05-18 22:47:08,412 DEBUG [pyshop.views.simple][Dummy-4] mirrored versions: [u'1.3', u'1.2', u'1.1', u'1.0', u'0.9.6', u'0.9.5', u'0.9.4', u'0.9.3', u'0.9.2', u'0.9.1', u'0.9', u'0.8.1', u'0.       8']
934931 2015-05-18 22:47:08,412 INFO  [pyshop.views.simple][Dummy-4] package Babel mirrored
934932 2015-05-18 22:47:08,609 INFO  [pyshop.security][Dummy-3] [162.209.108.184] GET http://wheel.cldbckp.com/prod/repository/261/Babel-1.3.tar.gz

I would be happy to work on this with you as I need a fix asap. My CICD pipeline is down and therefore my shop until this is resolved.

visit /pypi error

when I visit host:8000/pypi,error occurs

find error in log

This error results in that I can't use "pip search"

Error pkg_resources.DistributionNotFound: waitress while starting

I followed the installation procedure described on page https://pypi.python.org/pypi/pyshop with release 1.2.2. The server fails to start with following error in log file :

Traceback (most recent call last):
  File "/home/pyshop/pyshop/bin/pserve", line 9, in <module>
    load_entry_point('pyramid==1.5.3', 'console_scripts', 'pserve')()
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/pyramid/scripts/pserve.py", line 58, in main
    return command.run()
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/pyramid/scripts/pserve.py", line 325, in run
    relative_to=base, global_conf=vars)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/pyramid/scripts/pserve.py", line 367, in loadserver
    server_spec, name=name, relative_to=relative_to, **kw)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 255, in loadserver
    return loadobj(SERVER, uri, name=name, **kw)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
    global_conf=global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
    section)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
    global_conf=global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
    object_type, name=name)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/paste/deploy/loadwsgi.py", line 640, in find_egg_entry_point
    pkg_resources.require(self.spec)
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py", line 648, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/pyshop/pyshop/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: waitress
Removing PID file pyramid.pid

missing information about installation

Helo,

Following the documentation, i had to do these 2 extra steps to get it working:
1/ create a 'repository' folder, otherwise the uploads failed.
2/ pip install waitress otherwise pserve did not work.

Ability to disable file renaming

Hi there,

file renaming, although a good idea on a open mirror does not actually cover enough cases to be left active for me.

I wish to submit eggs of packages with compiled libraries and renaming the file would break easy_install ability to select the appropriate egg for os/architecture.

A quick look at the code suggests no easy fix exists to support this so I'd like to see an option to actually disable renaming since it my pyshop instance will not be public.

Error When Attempting to Download Requests package

Hello,
I have a fresh install of pyshop and up until today everything seemed to be in order. However, I am getting an error when attempting to pip install requests. It seems that is attempts to catch every version of the the requests package, and then fails when it tries to serve it to the client. I didn't include the entire log entry because its very long as it lists every version of request that it attempt to mirror. I only included the last version it attempt to mirror before serving the request.

The package is also never written to disk. I don't see a r/requests directory/package on disk after the failure.

Any help would be greatly appreciated!

.....
Aug 21 23:01:32 sd-repo-001 python[15578]: 2017-08-21 23:01:32,973 INFO  [pyshop.views.simple][waitress] Mirroring version 2.17.3
Aug 21 23:01:33 sd-repo-001 python[15578]: 2017-08-21 23:01:33,519 INFO  [pyshop.views.simple][waitress] Create release 2.17.3 for package requests
Aug 21 23:01:33 sd-repo-001 python[15578]: 2017-08-21 23:01:33,520 INFO  [pyshop.views.simple][waitress] Looking for author Kenneth Reitz
Aug 21 23:01:34 sd-repo-001 python[15578]: 2017-08-21 23:01:34,174 INFO  [pyshop.views.simple][waitress] package requests mirrored
Aug 21 23:01:34 sd-repo-001 python[15578]: 2017-08-21 23:01:34,789 INFO  [pyshop.security][waitress] [127.0.0.1] GET http://localhost:8000/pyshop/package
Aug 21 23:01:34 sd-repo-001 python[15578]: 2017-08-21 23:01:34,790 INFO  [pyshop.views.base][waitress] dispatch view Login
Aug 21 23:01:35 sd-repo-001 python[15578]: 2017-08-21 23:01:35,200 ERROR [waitress][waitress] Exception when serving /simple/requests/
Aug 21 23:01:35 sd-repo-001 python[15578]: Traceback (most recent call last):
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/tweens.py", line 12, in _error_handler
Aug 21 23:01:35 sd-repo-001 python[15578]: response = request.invoke_exception_view(exc_info)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/view.py", line 756, in invoke_exception_view
Aug 21 23:01:35 sd-repo-001 python[15578]: raise HTTPNotFound
Aug 21 23:01:35 sd-repo-001 python[15578]: pyramid.httpexceptions.HTTPNotFound: The resource could not be found.
Aug 21 23:01:35 sd-repo-001 python[15578]: During handling of the above exception, another exception occurred:
Aug 21 23:01:35 sd-repo-001 python[15578]: Traceback (most recent call last):
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/waitress/channel.py", line 338, in service
Aug 21 23:01:35 sd-repo-001 python[15578]: task.service()
Aug 21 23:01:35 sd-repo-001.saildrone.com python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/waitress/task.py", line 169, in service
Aug 21 23:01:35 sd-repo-001 python[15578]: self.execute()
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/waitress/task.py", line 399, in execute
Aug 21 23:01:35 sd-repo-001 python[15578]: app_iter = self.channel.server.application(env, start_response)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/router.py", line 270, in __call__
Aug 21 23:01:35 sd-repo-001 python[15578]: response = self.execution_policy(environ, self)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/router.py", line 278, in default_execution_policy
Aug 21 23:01:35 sd-repo-001 python[15578]: return request.invoke_exception_view(reraise=True)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/view.py", line 755, in invoke_exception_view
Aug 21 23:01:35 sd-repo-001 python[15578]: reraise_(*exc_info)
Aug 21 23:01:35 sd-repo-001.saildrone.com python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/compat.py", line 148, in reraise
Aug 21 23:01:35 sd-repo-001 python[15578]: raise value
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/router.py", line 276, in default_execution_policy
Aug 21 23:01:35 sd-repo-001 python[15578]: return router.invoke_request(request)
Aug 21 23:01:35 sd-repo-001.saildrone.com python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/router.py", line 249, in invoke_request
Aug 21 23:01:35 sd-repo-001. python[15578]: response = handle_request(request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid_tm/__init__.py", line 171, in tm_tween
Aug 21 23:01:35 sd-repo-001. python[15578]: reraise(*exc_info)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid_tm/compat.py", line 36, in reraise
Aug 21 23:01:35 sd-repo-001 python[15578]: raise value
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid_tm/__init__.py", line 136, in tm_tween
Aug 21 23:01:35 sd-repo-001 python[15578]: response = handler(request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/tweens.py", line 41, in excview_tween
Aug 21 23:01:35 sd-repo-001 python[15578]: response = _error_handler(request, exc)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/tweens.py", line 16, in _error_handler
Aug 21 23:01:35 sd-repo-001 python[15578]: reraise(*exc_info)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/compat.py", line 148, in reraise
Aug 21 23:01:35 sd-repo-001 python[15578]: raise value
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/tweens.py", line 39, in excview_tween
Aug 21 23:01:35 sd-repo-001 python[15578]: response = handler(request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/router.py", line 156, in handle_request
Aug 21 23:01:35 sd-repo-001 python[15578]: view_name
Aug 21 23:01:35 sd-repo-001. python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/view.py", line 642, in _call_view
Aug 21 23:01:35 sd-repo-001 python[15578]: response = view_callable(context, request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/viewderivers.py", line 301, in secured_view
Aug 21 23:01:35 sd-repo-001 python[15578]: return view(context, request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/viewderivers.py", line 462, in rendered_view
Aug 21 23:01:35 sd-repo-001 python[15578]: request, result, view_inst, context)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/renderers.py", line 435, in render_view
Aug 21 23:01:35 sd-repo-001 python[15578]: return self.render_to_response(response, system, request=request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/renderers.py", line 458, in render_to_response
Aug 21 23:01:35 sd-repo-001 python[15578]: result = self.render(value, system_values, request=request)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid/renderers.py", line 454, in render
Aug 21 23:01:35 sd-repo-001 python[15578]: result = renderer(value, system_values)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyramid_jinja2/__init__.py", line 265, in __call__
Aug 21 23:01:35 sd-repo-001 python[15578]: return template.render(system)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
Aug 21 23:01:35 sd-repo-001 python[15578]: return original_render(self, *args, **kwargs)
Aug 21 23:01:35 sd-repo-001. python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/jinja2/environment.py", line 1008, in render
Aug 21 23:01:35 sd-repo-001 python[15578]: return self.environment.handle_exception(exc_info, True)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
Aug 21 23:01:35 sd-repo-001 python[15578]: reraise(exc_type, exc_value, tb)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
Aug 21 23:01:35 sd-repo-001 python[15578]: raise value.with_traceback(tb)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyshop/templates/pyshop/simple/show.html", line 2, in top-level template code
Aug 21 23:01:35 sd-repo-001 python[15578]: {% for r in package.sorted_releases %}
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/jinja2/environment.py", line 430, in getattr
Aug 21 23:01:35 sd-repo-001 python[15578]: return getattr(obj, attribute)
Aug 21 23:01:35 sd-repo-001 python[15578]: File "/opt/pyshop/lib64/python3.6/site-packages/pyshop/models.py", line 518, in sorted_releases
Aug 21 23:01:35 sd-repo-001 python[15578]: releases.sort(reverse=True)
Aug 21 23:01:35 sd-repo-001 python[15578]: TypeError: '<' not supported between instances of 'Release' and 'Release'

NameError: name 'raw_input' is not defined...

...when running the command "pyshop_setup pyshop.ini" using Python 3.4. This is after installing Version 1.0 of PyShop from python.pypi.org. I verified that the issue clears and the command completes without errors if I change all "raw_input" to "input" in the bin/install.py file.

I found one of the better Python 2.x/3.x compatibility fixes for this here: http://stackoverflow.com/a/7321970/1916161

try: input = raw_input
except NameError: pass
print("Hi " + input("Say something: "))

User creation for ldap user unnecessarily copies password

When using ldap authentication, the by_ldap_credentials methods checks if a corresponding local User object exists in the database, and creates one if it does not:

if user_ldap is None:
log.debug('create user %s'%login)
user_ldap = User()
user_ldap.login = login
user_ldap.password = password
..snip..

Currently, the ldap password is getting copied into the model instance, even though the model's password will never be used, since the ldap server will be queried every time an ldap user attempts to log in. Copying the ldap password into the User instance seems to be an unnecessary exposure of an ldap user's password - bcrypt potentially reduces the risk somewhat, but it is still possible someone could grab the pyshop db and use it as a vector for getting ldap passwords.

I propose that in the case of an ldap login, any User instance creation should refrain from copying the ldap password, rather some "dummy" password could be used - an empty string, a random digit, anything at all really, given that the dummy password will never be used.

Thoughts?

Can't run pyshop tests

When I try to run pyshop tests, I get 43 errors

All basically complaining about the same thing:

Traceback (most recent call last):
  File "/home/fizyk/workspace/pyshop/pyshop/tests/case.py", line 16, in setUp
    self.session = DBSession()
  File "/home/fizyk/workspace/pyshop/pyshop/models.py", line 23, in <lambda>
    DBSession = lambda: SessionFactory.get('pyshop')()
  File "/home/fizyk/workspace/pyshop/pyshop/helpers/sqla.py", line 133, in get
    return cls.sessions[name]
KeyError: 'pyshop'

Off-thread mirroring

Really enjoying pyshop for my personal projects 😄

I was wondering if doing the upstream package mirroring off-thread would be a valuable contribution? I know this increases complexity greatly, but it would be cool to have the mirroring not block requests to the web service, since it sometimes can take a long time.

Implementation ideas:

• Have a celery worker queue that handles requests to upstream pypi rather than firing them on the request handler
• Have UI in the web interface where a requirements file or package list could be specified, which would trigger mirroring immediately on submission.

The major risk of this would be increased complexity and added dependencies.

Python3 Support

Hey there,
With the 1.2.4 update, it seems that it's not quite supporting Python3 yet, at least not the pyramid package xmlrpc!

user@pyshop:~/pyshop-1.2.4$ pserve pyshop.ini
Traceback (most recent call last):
File "/home/user/pyshop-1.2.4/lib/python3.5/site-packages/pyramid/path.py", line 380, in >_.zope_dottedname_style
found = getattr(found, n)
AttributeError: module 'pyshop.views' has no attribute 'xmlrpc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/pyshop-1.2.4/lib/python3.5/site-packages/pyshop/views/xmlrpc.py", line 11, in >
from pyramid_xmlrpc import XMLRPCView
File "/home/user/pyshop-1.2.4/lib/python3.5/site-packages/pyramid_xmlrpc/init.py", line 1, in >
import xmlrpclib
ImportError: No module named 'xmlrpclib'

If I missed something in your Python3 setup guide, just let me know, but that was following it as I saw it!

Working under Python2.7 of course though

Support for X-accel headers

Pyshop could be improved by implementation of X-accel headers.
Static content of packages could be served by respective webserver.
It could improve performance of serving packages.

Invalid dependency against PyPI

I'm seeing an uninstallable version of requests specified in the requirements file?

Could not find a version that satisfies the requirement requests<0.1 (from versions: 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.8.7, 0.8.8, 0.8.9, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.10.8, 0.11.1, 0.11.2, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.13.4, 0.13.5, 0.13.6, 0.13.7, 0.13.8, 0.13.9, 0.14.0, 0.14.1, 0.14.2, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.2.1, 2.3.0, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.5.0, 2.5.1)
  No distributions matching the version for requests<0.1

I'm curious about the older spec for requests (0.1)?

Upgrade dependencies

You're using features of pyramid 1.5 but depend on pyramid >=1.2 which is working fine for new installations, since pip downloads the last version released, but not for existing installations since the dependency is satisfied when pyramid <1.5 but >=1.2 is already installed.

Can you consider update the setup.py and depend on pyramid >=1.5?

How to force mirrored package updates

I've been using pyshop for a while now and its worked well. I am confused about how I can force a mirrored package to grab the latest version in PyPI.

https://pypi.python.org/pypi?name=planetary_test_data&version=0.3.2&:action=display

It appears if I wait a day then the package will update. Is there a way to force a package to update? I've tried purging a package and refreshing a package without the desired effect.

package name is case sensitive

package name in setup.py is case sensitive

Errors during installation

I hit a snag pretty early during the installation instructions - after doing pip install "pyshop[waitress]" (which looked successful), the next step should be cp pyshop.sample.ini pyshop.ini, but no such file exists:

(pyshop) [KenMacBook-2:/tmp/pyshop] % ls -l
total 8
drwxr-xr-x  51 kwilliams  wheel  1734 Mar 28 17:01 bin/
drwxr-xr-x   3 kwilliams  wheel   102 Mar 28 17:00 include/
drwxr-xr-x   3 kwilliams  wheel   102 Mar 28 17:00 lib/
-rw-r--r--   1 kwilliams  wheel    60 Mar 28 17:00 pip-selfcheck.json

Am I supposed to find this file somewhere else?

Anonymous pip install

Is there a way to configure pyshop such that pip install does not require authentication?
It is fairly common to want a pypi instance which requires authentication to upload packages but allows anonymous installs.

Currently, pip does not provide a mechanism to specify username and password in pip.conf, so this in combination with no anonymous read access in pyshop makes it very difficult to automate the usage of pip+pyshop.

Upload failed (500): Internal Server Error

Hi,

I tried to use pyshop here, when uploading my first package I get Upload failed (500): Internal Server Error

pyshop (1.2.2)

What's wrong? :)

2015-06-18 16:59:28,570 ERROR [waitress][Dummy-2] Exception when serving /simple/ 
Traceback (most recent call last): 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/waitress/channel.py", line 337, in service 
    task.service() 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/waitress/task.py", line 173, in service 
    self.execute() 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/waitress/task.py", line 392, in execute 
    app_iter = self.channel.server.application(env, start_response) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/router.py", line 223, in __call__ 
    response = self.invoke_subrequest(request, use_tweens=True) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/router.py", line 198, in invoke_subrequest 
    response = handle_request(request) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/tweens.py", line 20, in excview_tween 
    response = handler(request) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 99, in tm_tween 
    reraise(*exc_info) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 80, in tm_tween 
    response = handler(request) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/router.py", line 145, in handle_request 
    view_name 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/view.py", line 527, in _call_view 
    response = view_callable(context, request) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/config/views.py", line 243, in _secured_view 
    return view(context, request) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/config/views.py", line 352, in rendered_view 
    result = view(context, request) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyramid/config/views.py", line 480, in _class_requestonly_view 
    response = inst() 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyshop/views/base.py", line 44, in __call__ 
    response = self.render() 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyshop/views/simple.py", line 144, in render 
    create_if_not_exists=True) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyshop/models.py", line 427, in by_name 
    classifier = cls.first(session, where=(cls.name == name,)) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/pyshop/helpers/sqla.py", line 52, in first 
    return cls.build_query(session, join, where, order_by).first() 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2444, in first 
    ret = list(self[0:1]) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2280, in __getitem__ 
    return list(res) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2515, in __iter__ 
    return self._execute_and_instances(context) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2530, in _execute_and_instances 
    result = conn.execute(querycontext.statement, self._params) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 
    return meth(self, multiparams, params) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 
    return connection._execute_clauseelement(self, multiparams, params) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 
    compiled_sql, distilled_params 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 
    context) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception 
    exc_info 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 
    reraise(type(exception), exception, tb=exc_tb) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 
    context) 
  File "/opt/hosting/run/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 
    cursor.execute(statement, parameters) 
InterfaceError: (sqlite3.InterfaceError) Error binding parameter 0 - probably unsupported type. [SQL: u'SELECT classifier.id AS classifier_id, classifier.created_at AS classifier_created_at, classifier.name AS classifier_name, classifier.parent_id AS classifier_parent_id, classifier.category AS classifier_category \nFROM classifier \nWHERE classifier.name = ?\n LIMIT ? OFFSET ?'] [parameters: (FieldStorage('classifiers', u'Development Status :: 5 - Production/Stable'), 1, 0)] 

Installation fails on Ubuntu 10 with missing SQLAlchemy relationship

(pyshop)test@repo:~/pyshop$ pyshop_setup pyshop.ini
Traceback (most recent call last):
File "/home/test/pyshop/bin/pyshop_setup", line 9, in
load_entry_point('pyshop==0.9.10', 'console_scripts', 'pyshop_setup')()
File "/home/test/pyshop/local/lib/python2.7/site-packages/pkg_resources.py", line 352, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/test/pyshop/local/lib/python2.7/site-packages/pkg_resources.py", line 2307, in load_entry_point
return ep.load()
File "/home/test/pyshop/local/lib/python2.7/site-packages/pkg_resources.py", line 2021, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/home/test/pyshop/local/lib/python2.7/site-packages/pyshop/init.py", line 9, in
from .security import groupfinder, RootFactory
File "/home/test/pyshop/local/lib/python2.7/site-packages/pyshop/security.py", line 10, in
from .models import DBSession, User, Group
File "/home/test/pyshop/local/lib/python2.7/site-packages/pyshop/models.py", line 500, in
class Release(Base):
File "/home/test/pyshop/local/lib/python2.7/site-packages/pyshop/models.py", line 532, in Release
cascade='all, delete-orphan'))
File "", line 2, in relationship
File "/home/test/pyshop/local/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py", line 666, in init
self.strategy_class = self._strategy_lookup(("lazy", self.lazy))
File "/home/test/pyshop/local/lib/python2.7/site-packages/sqlalchemy/orm/interfaces.py", line 505, in _strategy_lookup
raise Exception("can't locate strategy for %s %s" % (cls, key))
Exception: can't locate strategy for <class 'sqlalchemy.orm.relationships.RelationshipProperty'> (('lazy', 'join'),)

Error during mirrored package refresh.

I am going to look into it sometime today, hopefully. This is having quite an impact on my environment.

2014-12-08 15:56:14,727 INFO  [pyshop.security][Dummy-4] [72.3.234.177] GET http://wheel.cldbckp.com/test/simple/oslosphinx/
2014-12-08 15:56:14,903 INFO  [pyshop.views.base][Dummy-4] dispatch view Show
2014-12-08 15:56:14,905 DEBUG [pyshop.views.simple][Dummy-4] "3 days, 0:44:43.355074" > "1 day, 0:00:00": True
2014-12-08 15:56:14,905 INFO  [pyshop.views.simple][Dummy-4] refresh package oslosphinx
2014-12-08 15:56:15,070 ERROR [pyshop.views.base][Dummy-4] Error on view Show
Traceback (most recent call last):
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyshop/views/simple.py", line 372, in render
    session_users)
UnboundLocalError: local variable 'session_users' referenced before assignment
2014-12-08 15:56:15,071 ERROR [waitress][Dummy-4] Exception when serving /test/simple/oslosphinx/
Traceback (most recent call last):
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/waitress/channel.py", line 337, in service
    task.service()
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 82, in tm_tween
    reraise(*exc_info)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 63, in tm_tween
    response = handler(request)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/config/views.py", line 245, in _secured_view
    return view(context, request)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyramid/config/views.py", line 475, in _class_requestonly_view
    response = inst()
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyshop/views/simple.py", line 372, in render
    session_users)
UnboundLocalError: local variable 'session_users' referenced before assignment

Must use HTTPS with Pypi

For two days now my Pyshop setup has failed to contact Pypi.

This is the error every time:

  File "/home/pyshop/prod/local/lib/python2.7/site-packages/pyshop/helpers/pypi.py", line 66, in request
    str(e), resp.headers)
ProtocolError: <ProtocolError for http://pypi.python.org/pypi: 403 403 Client Error: Must access using HTTPS instead of HTTP>

Suggested fix: use_https = True in RequestsTransport. It works and it would fix the problem for everyone who mirrors Pypi directly (it could be an issue for users who mirror a non-SSL aware server).

Is there a way to upload a bunch of packages quickly into PyShop?

I work in an environment where PyShop is unable to access PyPi, it's in an entirely isolated subnetwork.

Is there a way to bulk upload a bunch of eggs/wheels/sdists into PyShop?

LDAP users cannot change password

Support for this is not implemented and I doubt it would be any useful anyway so it would be best to just hide this menu entry from LDAP users.

This is made a bit difficult though because there is currently no bit of information in the model that allows us to known whether the model maps an external authentication backend or refers to a purely database bound user.

Ability to edit package description

Just like the usual pypi server has, ability to edit package description.

Can't mirror sphinx_bootstrap_theme

Just tried to install sphinx_bootstrap_theme, and for reasons unknown to me at the moment. pyshop doesn't want to even mirror this package.

2013-04-12 13:27:30,469 INFO  [pyshop.security][Dummy-1] [127.0.0.1] GET http://localhost:8000/simple/sphinx_bootstrap_theme/
2013-04-12 13:27:30,520 INFO  [pyshop.security][Dummy-2] [127.0.0.1] GET http://localhost:8000/simple/sphinx_bootstrap_theme/
/home/fizyk/.virtualenvs/pyshop/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py:455: SAWarning: Unicode type received non-unicode bind param value.
  param.append(processors[key](compiled_params[key]))
2013-04-12 13:27:30,815 INFO  [pyshop.views.base][Dummy-2] dispatch view Show
2013-04-12 13:27:31,589 INFO  [pyshop.views.simple][Dummy-2] package sphinx_bootstrap_theme has no versions

~.pip/pip.log

/home/fizyk/.virtualenvs/test/bin/pip run on Fri Apr 12 13:27:31 2013
Downloading/unpacking sphinx-bootstrap-theme

  Getting page http://pip:changeme@localhost:8000/simple/sphinx_bootstrap_theme/
  URLs to search for versions for sphinx-bootstrap-theme:
  * http://pip:changeme@localhost:8000/simple/sphinx_bootstrap_theme/
  Analyzing links from page http://localhost:8000/simple/sphinx_bootstrap_theme/
  Could not find any downloads that satisfy the requirement sphinx-bootstrap-theme

No distributions at all found for sphinx-bootstrap-theme

Exception information:
Traceback (most recent call last):
  File "/home/fizyk/.virtualenvs/test/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/home/fizyk/.virtualenvs/test/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py", line 266, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/fizyk/.virtualenvs/test/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 1026, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/home/fizyk/.virtualenvs/test/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/index.py", line 171, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for sphinx-bootstrap-theme

mirroring couses problems

When trying to install a package through pyshop, first I get is an error about not beeing able to find package on pyshop:

Getting page http://pip:changeme@localhost:8000/simple/virtualenv
Could not fetch URL http://pip:changeme@localhost:8000/simple/virtualenv: HTTP Error 404: Not Found
Will skip URL http://pip:changeme@localhost:8000/simple/virtualenv when looking for download links for virtualenv in /usr/local/lib/python2.7/dist-packages (from virtualenvwrapper)
URLs to search for versions for virtualenv in /usr/local/lib/python2.7/dist-packages (from virtualenvwrapper):
* http://pip:changeme@localhost:8000/simple/virtualenv/
Getting page http://pip:changeme@localhost:8000/simple/virtualenv/
Could not fetch URL http://pip:changeme@localhost:8000/simple/virtualenv/: timed out
Will skip URL http://pip:changeme@localhost:8000/simple/virtualenv/ when looking for download links for virtualenv in /usr/local/lib/python2.7/dist-packages (from virtualenvwrapper)
Could not find any downloads that satisfy the requirement virtualenv in /usr/local/lib/python2.7/dist-packages (from virtualenvwrapper)

Downloading/unpacking virtualenv (from virtualenvwrapper)

No distributions at all found for virtualenv in /usr/local/lib/python2.7/dist-packages (from virtualenvwrapper)

Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip-1.2.1-py2.7.egg/pip/basecommand.py", line 107, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip-1.2.1-py2.7.egg/pip/commands/install.py", line 256, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/local/lib/python2.7/dist-packages/pip-1.2.1-py2.7.egg/pip/req.py", line 1010, in prepare_files
    raise not_found
DistributionNotFound: No distributions at all found for virtualenv in /usr/local/lib/python2.7/dist-packages (from virtualenvwrapper)

I got this first for virtualenwrapper, but second run installed the package. I'd expect rather being forwarded immediately to pypi.python.org, (or am I able to configure that in pip.conf/pydisutils.conf to look in two sources)?

Full gist:
https://gist.github.com/fizyk/5335485

Package installation fails until cached.

Clean install following documented instructions.

Two requests for pymongo and the associated logs.

$ pip install pymongo
Downloading/unpacking pymongo
User for localhost:8000: pip
Password:
  http://localhost:8000/simple/pymongo/ uses an insecure transport scheme (http). Consider using https if localhost:8000 has it available
  Could not find any downloads that satisfy the requirement pymongo
Cleaning up...
No distributions at all found for pymongo
Storing debug log for failure in /Users/chri7485/.pip/pip.log

$ pip install pymongo
Downloading/unpacking pymongo
User for localhost:8000: pip
Password:
  http://localhost:8000/simple/pymongo/ uses an insecure transport scheme (http). Consider using https if localhost:8000 has it available
  Downloading pymongo-2.7.2-cp27-none-macosx_10_9_intel.whl (231kB): 231kB downloaded
Installing collected packages: pymongo
Successfully installed pymongo
Cleaning up...

$ cat pyshop.log
Starting server in PID 24226.
serving on http://0.0.0.0:8000
2014-09-26 11:34:20,114 INFO  [pyshop.security][Dummy-1] [127.0.0.1] GET http://localhost:8000/simple/pymongo/
2014-09-26 11:34:23,515 INFO  [pyshop.security][Dummy-2] [127.0.0.1] GET http://localhost:8000/simple/pymongo/
2014-09-26 11:34:23,742 INFO  [pyshop.views.base][Dummy-2] dispatch view Show
2014-09-26 11:34:23,745 INFO  [pyshop.views.simple][Dummy-2] refresh package pymongo
2014-09-26 11:34:24,524 INFO  [pyshop.views.simple][Dummy-2] mirror package pymongo now
2014-09-26 11:34:38,518 INFO  [pyshop.security][Dummy-3] [127.0.0.1] GET http://localhost:8000/simple/
2014-09-26 11:34:38,745 INFO  [pyshop.views.base][Dummy-3] dispatch view List
2014-09-26 11:34:38,780 INFO  [pyshop.security][Dummy-4] [127.0.0.1] GET http://localhost:8000/simple/pymongo/
2014-09-26 11:34:39,003 INFO  [pyshop.views.base][Dummy-4] dispatch view Show
2014-09-26 11:34:39,005 INFO  [pyshop.views.simple][Dummy-4] refresh package pymongo
2014-09-26 11:34:39,339 INFO  [pyshop.views.simple][Dummy-4] mirror package pymongo now
2014-09-26 11:34:44,284 ERROR [pyshop.views.base][Dummy-4] Error on view Show
Traceback (most recent call last):
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyshop/views/simple.py", line 312, in render
    self.session.flush()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
    rec.execute(self)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
    uow
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj
    mapper, table, insert)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 602, in _emit_insert_statements
    execute(statement, params)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) database is locked u'INSERT INTO package (created_at, update_at, name, local, downloads) VALUES (CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?)' (u'pymongo', 0, 0)
2014-09-26 11:34:44,288 ERROR [waitress][Dummy-4] Exception when serving /simple/pymongo/
Traceback (most recent call last):
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/waitress/channel.py", line 337, in service
    task.service()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 82, in tm_tween
    reraise(*exc_info)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 63, in tm_tween
    response = handler(request)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/config/views.py", line 245, in _secured_view
    return view(context, request)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyramid/config/views.py", line 475, in _class_requestonly_view
    response = inst()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/pyshop/views/simple.py", line 312, in render
    self.session.flush()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
    rec.execute(self)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
    uow
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj
    mapper, table, insert)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 602, in _emit_insert_statements
    execute(statement, params)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/Users/chri7485/Repos/pyshop/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) database is locked u'INSERT INTO package (created_at, update_at, name, local, downloads) VALUES (CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?)' (u'pymongo', 0, 0)
2014-09-26 11:35:31,179 INFO  [pyshop.views.simple][Dummy-2] package pymongo mirrored
2014-09-26 11:36:01,023 INFO  [pyshop.security][Dummy-1] [127.0.0.1] GET http://localhost:8000/simple/pymongo/
2014-09-26 11:36:05,376 INFO  [pyshop.security][Dummy-3] [127.0.0.1] GET http://localhost:8000/simple/pymongo/
2014-09-26 11:36:05,599 INFO  [pyshop.views.base][Dummy-3] dispatch view Show
2014-09-26 11:36:05,600 DEBUG [pyshop.views.simple][Dummy-3] "-1 day, 20:00:34.600507" > "1 day, 0:00:00": False
2014-09-26 11:36:05,605 INFO  [pyshop.views.simple][Dummy-3] package pymongo mirrored
2014-09-26 11:36:06,371 INFO  [pyshop.security][Dummy-4] [127.0.0.1] GET http://localhost:8000/repository/8/pymongo-2.7.2-cp27-none-macosx_10_9_intel.whl
2014-09-26 11:36:06,494 INFO  [pyshop.helpers.download][Dummy-4] Downloading https://pypi.python.org/packages/2.7/p/pymongo/pymongo-2.7.2-cp27-none-macosx_10_9_intel.whl

Failed upload results in 200 response.

Building a package and uploading it will result in a 200 being returned even if the package fails to upload. Also, the returned error HTTPForbidden() makes it appear to be related to a permissions issue, so I was double checking my accounts when in fact it was the upload sanitation on the version that was failing.

$ python setup.py bdist_wheel upload -v -r test
<output skipped>
running install_egg_info
Copying app_api.egg-info to build/bdist.linux-x86_64/wheel/app_api-2.0.0.dev309.g3121c39-py2.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/app_api-2.0.0.dev309.g3121c39.dist-info/WHEEL
running upload
Submitting /home/jenkins/workspace/python-build/dist/app_api-2.0.0.dev309.g3121c39-py2-none-any.whl to https://server/test/simple/
Server response (200): OK

And the pyshop resulting logs...

2014-10-01 13:39:32,059 INFO  [pyshop.security][Dummy-1] [162.209.108.184] POST http://server/test/simple/
2014-10-01 13:39:32,243 INFO  [pyshop.views.base][Dummy-1] dispatch view
2014-10-01 13:39:32,247 ERROR [pyshop.views.base][Dummy-1] Error on view UploadReleaseFile
Traceback (most recent call last):
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/opt/pyshop/test/local/lib/python2.7/site-packages/pyshop/views/simple.py", line 77, in render
    raise exc.HTTPForbidden()
HTTPForbidden: Access was denied to this resource.
2014-10-01 13:39:32,351 INFO  [pyshop.security][Dummy-2] [162.209.108.184] GET http://server/test/simple/
2014-10-01 13:39:32,537 INFO  [pyshop.views.base][Dummy-2] dispatch view List

AttributeError: 'NoneType' object has no attribute 'startswith'

I had not had time to work on this myself so far, just checking, but, when trying to install package from pyshop's pypi, I get:

AttributeError: 'NoneType' object has no attribute 'startswith'

On line:

File "...python2.7/site-packages/pyshop/views/repository.py", line 10, in get_release_file

Which is if url.startswith('http://pypi.python.org'):

I have just one file in my repository, and trying to install these files.

Everything is installed locally.

Add ldap backend for authentification

We use LDAP for auth in my company, it would be nice to be able to use it to login to pyshop.

Release pyshop

Hey,

This fix isn't in a release, so the last version pyshop is broken.

Can you release the v1.1.2 including this commit?

pyshop ldap Not available？

Does PYSHOP now support LDAP?
My configuration is correct.
but,unable login to LDAP USER.
Do not know where the problem lies.

Error during install pymlconf

When I try:

python setup.py develop

Searching for pymlconf>=0.2.10a
Reading http://***:***@pyshop.i.***.pl/simple/pymlconf/
Reading http://github.com/pylover/pymlconf
Reading http://packages.python.org/pymlconf
Best match: pymlconf 0.2.10a
Downloading http://***:***@pyshop.i.***.pl/repository/47/pymlconf-0.2.10a-py2.7.egg#592f8c812ce91c7e72b8228670fc8713
error: Unexpected HTML page found at http://***:***@pyshop.i.***.pl/repository/47/pymlconf-0.2.10a-py2.7.egg#592f8c812ce91c7e72b8228670fc8713

I don't know, what it meaning...

Unexpected HTML page found

It's my conf files (on dev and on test, these files are the same):

~/.pip/pip.conf

[global]
timeout = 60
[install]
index-url = http://***:***@pyshop.i.***.pl/simple/

~/.pydistutils.cfg

[easy_install]
index-url: http://***:***@pyshop.i.***.pl/simple/

If I type:

python setup.py develop

On test server all works fine (downloaded tar.gz packages), but on dev I received this (egg packages). This tweepy packages is only for instance:

Searching for tweepy
Reading http://pip:[email protected]/simple/tweepy/
Reading http://github.com/tweepy/tweepy
Reading http://github.com/joshthecoder/tweepy
Best match: tweepy 2.0
Downloading http://***:***@pyshop.i.***.pl/repository/1698/tweepy-2.0-py2.7.egg#193993b0df7c4e28fba5bb457e401d91
error: Unexpected HTML page found at http://***:***@pyshop.i.***.pl/repository/1698/tweepy-2.0-py2.7.egg#193993b0df7c4e28fba5bb457e401d91

But, e.g for zc.lockfile package works (downloading egg, but form pypi.python.org:

Searching for zc.lockfile
Reading http://***:***@pyshop.i.***.pl/simple/zc.lockfile/
Reading http://www.python.org/pypi/zc.lockfile
Best match: zc.lockfile 1.1.0
Downloading https://pypi.python.org/packages/source/z/zc.lockfile/zc.lockfile-1.1.0.zip
Processing zc.lockfile-1.1.0.zip
Running zc.lockfile-1.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-62_8Ve/zc.lockfile-1.1.0/egg-dist-tmp-UGHU9P
warning: no files found matching '*.rst'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
Adding zc.lockfile 1.1.0 to easy-install.pth file
Installed /home/***/.virtualenv/virtual_name/lib/python2.7/site-packages/zc.lockfile-1.1.0-py2.7.egg

Fails to distribute already mirrored packages with _ or -

Some packages like riak_pb or factory_boy use an _ in their name but their name is converted to using - using bdist_wheel. Not sure if this is a bug with wheel or not but this brought my attention to a "fix" to handle this case that has been added to pypiserver.

When requesting a package name like factory-boy, pypiserver will answer a redirect to the correct package name which is factory_boy.

This issue is a bit different than the one reported in #10 because that issue was about mirroring the package and this one is about serving the package to the client.

This seems to be no issue for regular pip or easy_install, but it fails for pip --use-wheel.

pyshop didn't find local packages by using pip install

Problem

I want to use pyhsop for internal deployment of our python packages. I created a instance of pyshop and deployed some of our packages to the pyshop server. Afterwards i tired to use pip to install these packages on my local system. The result is that the package is not available

(test)❯ pip install nexiles.gateway.fileservice
    Downloading/unpacking nexiles.gateway.fileservice
        Could not find any downloads that satisfy the requirement       
    nexiles.gateway.fileservice
    No distributions at all found for nexiles.gateway.fileservice
    Storing complete log in /Users/jboerner/.pip/pip.log

But as you can see in the UI of pyshop the packages are available on the server

I also use the right pip configuration

[global]
# when mirroring a package, pyshop retrieves information from PyPI and
# stores it in its database. Be patient, it is not so long.
default-timeout = 120
timeout = 120

[install]
index-url = http://admin:admin@localhost:9000/simple/

And if we have a look on the server log, we can see that response is actually positiv

2015-07-15 10:36:34,690 INFO  [pyshop.security][Dummy-1] [127.0.0.1] GET http://localhost:9000/simple/nexiles.gateway.fileservice/
2015-07-15 10:36:34,708 INFO  [pyshop.security][Dummy-2] [127.0.0.1] GET http://localhost:9000/simple/nexiles.gateway.fileservice/
2015-07-15 10:36:34,900 INFO  [pyshop.views.base][Dummy-2] dispatch view Show

Any idea why this is not working?

"Database is locked" errors?

when doing an initial (no caching yet) install of a requirements.txt file, I get errors like below. If I re-run, it'll do it on a different package. eventually (once everything is cached) it seems to work. Sometimes it'll do 2-3 new packages before failing again.

Python 2.6.5 on Ubuntu Lucid. Running via supervisord (only one process, I checked), and no other shells/etc. Apache timeout/proxytimeout is set to 300. pip 1.5.2. I've dumped/reloaded the database to check it wasn't a corrupted sqlite file.

Any ideas?

2014-02-04 00:54:36,479 INFO  [pyshop.views.base][Dummy-4] dispatch view List
2014-02-04 00:54:36,494 INFO  [pyshop.security][Dummy-3] [10.0.0.1] GET Current request matches no route
2014-02-04 00:54:36,500 INFO  [pyshop.security][Dummy-1] [10.0.01] GET http://example.com/simple/ipython/
2014-02-04 00:54:36,708 INFO  [pyshop.views.base][Dummy-1] dispatch view Show
2014-02-04 00:54:36,712 INFO  [pyshop.views.simple][Dummy-1] refresh package ipython
2014-02-04 00:54:36,989 INFO  [pyshop.views.simple][Dummy-1] mirror package ipython now
2014-02-04 00:54:42,000 ERROR [pyshop.views.base][Dummy-1] Error on view Show
Traceback (most recent call last):
  File "/home/pyshop/lib/python2.6/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/home/pyshop/lib/python2.6/site-packages/pyshop/views/simple.py", line 311, in render
    self.session.flush()
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1879, in flush
    self._flush(objects)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1997, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 57, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1961, in _flush
    flush_context.execute()
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 370, in execute
    rec.execute(self)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 523, in execute
    uow
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
    mapper, table, insert)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 594, in _emit_insert_statements
    execute(statement, params)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 717, in execute
    return meth(self, multiparams, params)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 814, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 927, in _execute_context
    context)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1076, in _handle_dbapi_exception
    exc_info
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 920, in _execute_context
    context)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 425, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) database is locked u'INSERT INTO package (created_at, update_at, name, local, downloads) VALUES (CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?)' (u'ipython', 0, 0)
2014-02-04 00:54:42,008 ERROR [waitress][Dummy-1] Exception when serving /simple/ipython/
Traceback (most recent call last):
  File "/home/pyshop/lib/python2.6/site-packages/waitress/channel.py", line 337, in service
    task.service()
  File "/home/pyshop/lib/python2.6/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/home/pyshop/lib/python2.6/site-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/router.py", line 251, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/router.py", line 227, in invoke_subrequest
    response = handle_request(request)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid_tm/__init__.py", line 82, in tm_tween
    reraise(*exc_info)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid_tm/__init__.py", line 63, in tm_tween
    response = handler(request)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/router.py", line 161, in handle_request
    response = view_callable(context, request)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/config/views.py", line 237, in _secured_view
    return view(context, request)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/config/views.py", line 347, in rendered_view
    result = view(context, request)
  File "/home/pyshop/lib/python2.6/site-packages/pyramid/config/views.py", line 467, in _class_requestonly_view
    response = inst()
  File "/home/pyshop/lib/python2.6/site-packages/pyshop/views/base.py", line 44, in __call__
    response = self.render()
  File "/home/pyshop/lib/python2.6/site-packages/pyshop/views/simple.py", line 311, in render
    self.session.flush()
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1879, in flush
    self._flush(objects)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1997, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 57, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1961, in _flush
    flush_context.execute()
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 370, in execute
    rec.execute(self)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 523, in execute
    uow
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
    mapper, table, insert)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 594, in _emit_insert_statements
    execute(statement, params)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 717, in execute
    return meth(self, multiparams, params)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 814, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 927, in _execute_context
    context)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1076, in _handle_dbapi_exception
    exc_info
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 920, in _execute_context
    context)
  File "/home/pyshop/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 425, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (OperationalError) database is locked u'INSERT INTO package (created_at, update_at, name, local, downloads) VALUES (CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, ?, ?, ?)' (u'ipython', 0, 0)
2014-02-04 00:54:46,593 INFO  [pyshop.views.simple][Dummy-2] package ipython mirrored

Administration tab should be visible to admin only

Since this view is simply denied to non-admin roles, it would be best to just not show it to users not concerned with it.

Supporting Search Results With Hyphens (Replace with Underscores)

After some weird testing, it seems I've come across an edge case that pip accounts for using the normal pypi.python.org index, but Pyshop does not (yet).

When using the index for Pyshop, I was trying to install pylint, but it kept failing about being unable to find any matches for 'backports-functools-lru-cache'. After commenting out my indexer and going directly to pypi.python.org, it found it immediately, leading me to believe it was a fault with Pyshop. After investigating the logged outputs, I came across this:

INFO [pyshop.security][waitress] [10.1.22.95] GET http://pyshop.<company>.com/simple/backports-functools-lru-cache/ INFO [pyshop.security][waitress] [10.1.22.95] GET http://pyshop.<company>.com/simple/backports-functools-lru-cache/ INFO [pyshop.views.base][waitress] dispatch view Show INFO [pyshop.views.base][waitress] dispatch view Show INFO [pyshop.views.simple][waitress] refresh package backports-functools-lru-cache INFO [pyshop.views.simple][waitress] refresh package backports-functools-lru-cache DEBUG [urllib3.connectionpool][waitress] Starting new HTTPS connection (1): pypi.python.org DEBUG [urllib3.connectionpool][waitress] https://pypi.python.org:443 "POST /pypi HTTP/1.1" 200 None DEBUG [urllib3.connectionpool][waitress] Starting new HTTPS connection (1): pypi.python.org DEBUG [urllib3.connectionpool][waitress] https://pypi.python.org:443 "POST /pypi HTTP/1.1" 200 None DEBUG [pyshop.views.simple][waitress] Found 151, matched 0 DEBUG [pyshop.views.simple][waitress] Found 151, matched 0 DEBUG [urllib3.connectionpool][waitress] Starting new HTTPS connection (1): pypi.python.org DEBUG [urllib3.connectionpool][waitress] https://pypi.python.org:443 "POST /pypi HTTP/1.1" 200 None DEBUG [pyshop.views.simple][waitress] Found 151, matched 0 DEBUG [pyshop.views.simple][waitress] Found 151, matched 0 INFO [pyshop.views.simple][waitress] package backports-functools-lru-cache has no versions INFO [pyshop.views.simple][waitress] package backports-functools-lru-cache has no versions

The offending hyphen's ('-') in the name of the search seem to cause Pyshop not to register the package (It finds 151 results that potentially match the package name, but none are exact matches due to the hyphens).

From what I can tell from searching using pypi.python.org though, hyphens and underscores are interchangeable (well, hyphens get replaced with underscores)

Looking forward to this fix!

Instructions on docker setup

could you please provide some detailed instructions on how to use the docker image?
and on a slightly different note, what makes it not production ready?

xmlrpc.py package_releases() method results in 500 Internal Server Error

Running this:

import xmlrpclib
import pprint

client = xmlrpclib.ServerProxy('http://192.168.197.42:8000/pypi/')
# this works fine:
# pprint.pprint(client.list_packages())
# this doesn't:
pprint.pprint(client.package_releases('somepackage'))

...results in the following stack trace:

Traceback (most recent call last):
  File "/Users/some.user/PycharmProjects/testproject/xmlrpctest.py", line 8, in <module>
    pprint.pprint(client.package_releases('somepackage'))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xmlrpclib.py", line 1312, in single_request
    response.msg,
xmlrpclib.ProtocolError: <ProtocolError for 192.168.197.42:8000/pypi/: 500 Internal Server Error>

Meanwhile, over on the server, pyshop.log provides more information:

2014-12-11 23:05:22,980 ERROR [waitress][Dummy-2] Exception when serving /pypi/
Traceback (most recent call last):
  File "/home/some.user/pyshop/lib/python2.7/site-packages/waitress/channel.py", line 337, in service
    task.service()
  File "/home/some.user/pyshop/lib/python2.7/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/home/some.user/pyshop/lib/python2.7/site-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 92, in tm_tween
    reraise(*exc_info)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 73, in tm_tween
    response = handler(request)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid/config/views.py", line 385, in viewresult_to_response
    result = view(context, request)
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid/config/views.py", line 489, in _class_view
    response = inst()
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyramid_xmlrpc/__init__.py", line 119, in __call__
    return xmlrpc_response(getattr(self,method)(*params))
  File "/home/some.user/pyshop/lib/python2.7/site-packages/pyshop/views/xmlrpc.py", line 36, in package_releases
    releases = Release.by_package_name(session, package_name, show_hidden)
AttributeError: type object 'Release' has no attribute 'by_package_name'

Indeed, xmlrpc.py does call Release.by_package_name() at line 36:

30    def package_releases(self, package_name, show_hidden=False):
31        """
32        Retrieve a list of the releases registered for the given package_name.
33        Returns a list with all version strings if show_hidden is True or
34        only the non-hidden ones otherwise."""
35        session = DBSession()
36        releases = Release.by_package_name(session, package_name, show_hidden)
37        session.rollback()
38        return releases

...but the Release class in models.py does not appear to define by_package_name() at all. Just saying, y'all might want to take a look. ;-)

Thanks!

Package purge fails with postgresql backend

I implemented the purge functionality before and it works well with SQLite, but blows up with Postgres and I am assuming MySQL. I know what the problem is and I would love to fix it myself, but my database knowledge isn't strong enough.

Any help would be greatly appreciated!

2015-03-26 16:08:18,491 INFO  [pyshop.security][Dummy-4] [72.3.234.177] GET http://wheel.cldbckp.com/prev/
2015-03-26 16:08:18,498 INFO  [pyshop.views.base][Dummy-4] dispatch view Index
2015-03-26 16:08:18,568 INFO  [pyshop.security][Dummy-1] [72.3.234.177] GET http://wheel.cldbckp.com/prev/pyshop/package
2015-03-26 16:08:18,575 INFO  [pyshop.views.base][Dummy-1] dispatch view List
2015-03-26 16:08:20,507 INFO  [pyshop.security][Dummy-2] [72.3.234.177] GET http://wheel.cldbckp.com/prev/pyshop/purge/package/156
2015-03-26 16:08:20,515 INFO  [pyshop.views.base][Dummy-2] dispatch view Purge
2015-03-26 16:08:23,782 INFO  [pyshop.security][Dummy-3] [72.3.234.177] GET http://wheel.cldbckp.com/prev/pyshop/purge/package/156?form.submitted=
2015-03-26 16:08:23,790 INFO  [pyshop.views.base][Dummy-3] dispatch view Purge
2015-03-26 16:08:33,698 ERROR [pyshop.views.base][Dummy-3] Error on view Purge
Traceback (most recent call last):
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyshop/views/base.py", line 48, in __call__
    self.session.flush()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 367, in execute
    n.execute_aggregate(self, set_)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute_aggregate
    uow)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 117, in delete_obj
    cached_connections, mapper, table, delete)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 694, in _emit_delete_statements
    c = connection.execute(statement, del_objects)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 928, in _execute_context
    context)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 433, in do_executemany
    cursor.executemany(statement, parameters)
IntegrityError: (IntegrityError) update or delete on table "classifier" violates foreign key constraint "classifier__release_classifier_id_fkey" on table "classifier__release"
DETAIL:  Key (id)=(4) is still referenced from table "classifier__release".
 'DELETE FROM classifier WHERE classifier.id = %(id)s' ({'id': 4}, {'id': 5}, {'id': 11}, {'id': 14}, {'id': 15}, {'id': 48}, {'id': 58}, {'id': 178})
2015-03-26 16:08:33,699 ERROR [waitress][Dummy-3] Exception when serving /prev/pyshop/purge/package/156
Traceback (most recent call last):
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/waitress/channel.py", line 337, in service
    task.service()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/waitress/task.py", line 173, in service
    self.execute()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/router.py", line 242, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/router.py", line 217, in invoke_subrequest
    response = handle_request(request)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 82, in tm_tween
    reraise(*exc_info)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid_tm/__init__.py", line 63, in tm_tween
    response = handler(request)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/config/views.py", line 245, in _secured_view
    return view(context, request)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyramid/config/views.py", line 475, in _class_requestonly_view
    response = inst()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/pyshop/views/base.py", line 48, in __call__
    self.session.flush()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 367, in execute
    n.execute_aggregate(self, set_)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute_aggregate
    uow)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 117, in delete_obj
    cached_connections, mapper, table, delete)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 694, in _emit_delete_statements
    c = connection.execute(statement, del_objects)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 928, in _execute_context
    context)
  File "/opt/pyshop/preview/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 433, in do_executemany
    cursor.executemany(statement, parameters)
IntegrityError: (IntegrityError) update or delete on table "classifier" violates foreign key constraint "classifier__release_classifier_id_fkey" on table "classifier__release"
DETAIL:  Key (id)=(4) is still referenced from table "classifier__release".
 'DELETE FROM classifier WHERE classifier.id = %(id)s' ({'id': 4}, {'id': 5}, {'id': 11}, {'id': 14}, {'id': 15}, {'id': 48}, {'id': 58}, {'id': 178})