pypi / warehouse

The Python Package Index

Warehouse

Warehouse is the software that powers PyPI. See our development roadmap, documentation, and architectural overview.

Getting Started

You can run Warehouse locally in a development environment using docker. See Getting started documentation for instructions on how to set it up.

The canonical deployment of Warehouse is in production at pypi.org.

Discussion

You can find help or get involved on:

• Github issue tracker for reporting issues
• IRC: on Libera, channel #pypa for general packaging discussion and user support, and #pypa-dev for discussions about development of packaging tools
• The PyPA Discord for live discussions
• The Packaging category on Discourse for discussing new ideas and community initiatives

Testing

Read the running tests and linters section of our documentation to learn how to test your code. For cross-browser testing, we use an open source account from BrowserStack. If your pull request makes any change to the user interface, it will need to be tested to confirm it works in our supported browsers.

_

Code of Conduct

Everyone interacting in the Warehouse project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PSF Code of Conduct.

warehouse's Issues

Port Existing XMLRPC API

• list_packages
• package_releases
• release_urls
• release_data
• search
• browse
• updated_releases
• changelog
• changelog_last_serial
• changelog_since_serial
• changed_packages
• release_downloads
• package_roles
• user_packages
• package_hosting_mode
• top_packages
• list_packages_with_serial

Replace hard coded path assumptions with something like Stockpile

Deduplicate Mainainers

If a user has both the maintainer and the owner role for a package they will show up twice in the Maintainers list. This should be set to be a distinct query so that they only show up once.

Make Keywords editable in the Django Admin

Keywords are an ArrayField and currently is not editable in the Django Admin due to issues with djorm-ext-pgarray.

Replace hardcoded Varnish/Fastly assumptions with something pluggable

Currently there are some headers and processes that expect Varnish, or more explicitly Fastly. As reusable software Warehouse really shouldn't depend on Fastly for proper operation. This should be something pluggable.

Implement Persona Login

It would be nice to support logging into Warehouse via Persona.

Use web test to implement a functional test suite

We need some functional tests to ensure that Warehouse functions at a high level. Ideally this would use webtest to avoid needing to parse http and simply use the wsgi application.

Switch back to using an upstream webassets

Currently we are using a forked copy of webassets (webassets-py3k). Once upstream has released a Python 3 compatible release we should switch back to upstream.

Switch Recliner back to using upstream bleach

Currently recliner is using a forked copy of bleach (bleach-py3k) in order to support Python 3. Once upstream has released an updated version that should be switched back.

Figure out how to make HSTORE keys case insenstive

It would be nice for them to at least be case insensitive unique, but great if they were just in general case insensitive like the citext module.

Ensure usernames are case insensitive

This is a current issue on PyPI. Usernames are case sensitive so I can have dstufft, Dstufft, etc.

We need to both make sure this cannot happen in Warehouse, and figure out what sort of transition plan we will use.

Implement OpenID Logins

PyPI allows logging in via OpenID, we need to either implement this or deprecate it and provide a migration path.

Implement a Fastly Utility Decorator

Copy and Pasting the Surrogate-Key handling code all over the place is far from optimal, we should figure out how to turn it into a decorator (ideally) or at the very least make it a utility function that can be called.

Implement a UI

The current PyPI UI is pretty horrible. Crate tried to improve on this but while it's an improvement it's not all that great itself. Perhaps we can find a really great designer/ux person to contribute?

Here's a list of UI centric routes from the current PyPI to give an idea of what sort of views we need.

• home
• browse
• index
• search
• display_pkginfo
• remove_pkg
• pkg_edit
• submit_form
• display
• register_form
• user
• user_form
• forgotten_password_form, forgotten_password
• password_reset, pw_reset, pw_reset_change
• role, role_form
• list_classifiers
• login, logout
• files
• urls
• show_md5
• addkey, delkey
• about
• /mirrors
• /security

GPG Short IDs are vulnerable to collision attacks

As @alex mentioned in #23 the GPG Short IDs used by PyPI are vulnerable to collision attacks. Once we strangle out PyPI's legacy code we should update it to use long ids or full fingerprints.

Display the Owners and Maintainers of a Project

Need to be able to see what user accounts own/maintain a particular project. It would be cool if this included some sort of avatar support instead of just raw usernames.

Existing Package Upload APIs

• submit - Submit metadata
• submit_pkg_info - Submit metadata as a PKG-INFO file
• file_upload - Upload a File
• doc_upload - Upload documentation

Remove the precomputed description_html from the DB

We don't really need to do this. It makes it harder to ever adjust how we render the pages and overall caching is a better method of dealing with any sort of expensiveness in rendering.

Switch to South

Once models settle down a bit and it makes sense to, we need to switch from syncdb to using South for migrations.

Switch from south to Django migrations

Link dependency names to the actual dependency

When viewing dependencies on the project page, the names should be a link that take you to that dependency's project page.

Prevent cacading deletes in the packaging app

Cascading deletes make it easier to programatically delete whole packages (or releases). While this is convenient it does mean that that we must protect against the case where we don't want to allow cascaded deletes.

On the other hand if we prevent cascading deletes that forces people to be explicit by default if they want to delete whole chunks of data and that seems like a better option to me.

Clean the date_joined field in the database

-infinity is actually a terrible "we don't have a value" value, so make date_joined nullable and set anything with -infinty to NULL.

Ensure version is sorted in a sane direction.

Implement CSP

Ideally the entire site will be protected by a CSP policy. However because of the admin we might need to exclude /admin from that, at least until the Django admin no longer uses inline javascript or CSS.

Link to release notifications help on project detail page

This would include the ability to upload a requirements.txt adn get notified for everything in it.

---

note by @brainwane 2018-03-22: see update

Implement a report URI for the Content Security Policy

In #93 a Content Security Policy was added, however reports are not being sent anywhere. Ideally they would go to something like Sentry (perhaps even sentry itself?).

X-Powered-By Headers

It'd be useful to add a X-Powered-By header to enable easily checking to see what version of Warehouse is running.

Figure out an assest pipeline

For development ease css & js will be in multiple files and unoptimized. Figured out a decent asset pipeline to use to handle both development and production.

Refactor the Link header to have a helper to generate them

We're copy/pasting some hardcoded method for generating a Link url, ideally we should be able to simply modify an attribute on a Response to set this.

Clean database of UNKNOWN and validates against it

Currently the database is littered with "UNKNOWN". This "helpfully" comes from distutils who will fill it in for a missing required value. We should strip these from the database and strip it from new incoming data.

Standardize Database Handling of Timezone

The fields that came from Django have timezone=True, the old fields have timezone=False, we should pick one and use it.

Support more than X-Accel-Redirect for Package hosting

Implement OpenID Provider

PyPI offers the ability to be an OpenID provider, we need to add this feature to Warehouse or deprecate it from PyPI itself.

Logging configuration

There's already some logging being done by the app - the web requests. There will be more logging done explicitly by code as users do things (dumb things or good things).

I'm going to add a new section to the configuration file called logging which will have the following structure (closely mirroring the standard logging configuration file structure, but not requiring us to actually have a separate INI syntax file).

logging:
    formatters:
        simpleFormater:
            format: '%(asctime)s - %(levelname)s: %(message)s'
            datefmt: '%Y/%m/%d %H:%M:%S'

    handlers:
        console:
            class: logging.StreamHandler
            formatter: simpleFormater
            level: DEBUG
            stream: ext://sys.stdout
        file:
            class : logging.FileHandler
            formatter: simpleFormater
            level: WARNING
            filename: output.log

    loggers:
        clogger:
            level: DEBUG
            handlers: [console]
        flogger:
            level: WARNING
            handlers: [file]

    root:
        level: DEBUG
        handlers: [console, file]

This example is way more complex than I imagine any given configuration would actually be.

This configuration is then loaded with some code like:

# config = loaded yaml config
logging_conf = config['logging']
logging_conf.setdefault('version', 1)
logging.config.dictConfig(logging_conf)

For a more concrete dev configuration I anticipate something like:

logging:
    handlers:
        console:
            class: logging.StreamHandler
            level: DEBUG
            stream: ext://sys.stdout

    root:
        level: DEBUG
        handlers: [console]

(which will hopefully work; actual results may vary of course given it's the logging module we're talking about)

Inform Users of Latest Version

When the users are manually on a page other than the latest version, they should get a link/message telling them that and pointing them to the latest version.

Ensure that the database handles the ON DELETE for Emails

Implement ON DELETE at the database level for the ForeignKey from warehouse.accounts.models.Email to warehouse.accounts.models.User.

The ability to relocate virtual environments

The ability to relocate virtual environments was called out specifically.

Restrict the img-src CSP directive and fix HTTPS images

Currently PyPI allows using the tag to host images anywhere, this has a few problems. Namely that those images are often not available via HTTPS, making those pages have mixed content, and it forces us to allow anything in the img-src directive in the CSP policy.

Possibly this can be solved similarly to how github solved this and use https://github.com/atmos/camo.

/cc @coderanger

Protect from users having multiple primary addresses

The database should (if supported) create a constraint that prevents any particular using from having more than one primary email address at any one time.

Port Existing APIs to Warehouse

• The "Simple" API (/simple/)
• The Package URLs (/packages/)
• The daytime URL (/daytime) (Is this something we need?)
• XMLRPC #59
• JSON (/pypi/<packagename>/json)
• RSS (Includes "Last Hour", and "Packages")
• DOAP (Is this something we need?)
• Explicit URLs (Do we need a legacy copy of this?)
• Package Upload APIs #57

Validate Complexity of Passwords

Ensure that user passwords always match some level of password complexity/strength.

Link classifiers in the project description

The classifiers need to be linked to some sort of "browse" page.

Verify ownership of GPG key

Currently there's no verification that a person owns the GPG key they claim they do. We should verify this before allowing this key.

Refactor out project name normalization

It would be a good idea to refactor out the project name normalization.

Show reverse dependencies on the package page

When using wheel and twine to upload a package, PyPI and Warehouse will have information about the dependencies of said package. It would be very nice to be able to look up the reverse relationship, the dependents, on the package's listing in Warehouse.

Mock the database in the unit test suite

switch from /~username/ to /user/username/

Handle the case where there is no long_description

Long description can be None instead of "". Right now this cases an exception.

Stacktrace (most recent call last):

  File "raven/middleware.py", line 31, in __call__
    iterable = self.application(environ, start_response)
  File "site-packages/guard.py", line 62, in __call__
    return self.application(environ, _start_response)
  File "warehouse/middleware.py", line 28, in __call__
    return self.app(environ, _start_response)
  File "werkzeug/wsgi.py", line 40, in <lambda>
    return update_wrapper(lambda *a: f(*a)(*a[-2:]), f)
  File "warehouse/application.py", line 260, in wsgi_app
    return view(self, request, **kwargs)
  File "warehouse/utils.py", line 90, in wrapper
    resp = fn(app, request, *args, **kwargs)
  File "warehouse/utils.py", line 173, in wrapper
    resp = fn(app, request, *args, **kwargs)
  File "warehouse/packaging/views.py", line 78, in project_detail
    description_html = htmlize(release["description"])
  File "recliner/renderer.py", line 152, in htmlize
    html = render(text)
  File "recliner/renderer.py", line 108, in render
    settings_overrides=settings,
  File "docutils/core.py", line 448, in publish_parts
    enable_exit_status=enable_exit_status)
  File "docutils/core.py", line 662, in publish_programmatically
    output = pub.publish(enable_exit_status=enable_exit_status)
  File "docutils/core.py", line 217, in publish
    self.settings)
  File "docutils/readers/__init__.py", line 71, in read
    self.input = self.source.read()
  File "docutils/io.py", line 426, in read
    return self.decode(self.source)
  File "docutils/io.py", line 99, in decode
    data_encoding = self.determine_encoding_from_data(data)
  File "docutils/io.py", line 142, in determine_encoding_from_data
    if data.startswith(start_bytes):

Resolving this issue should resolve https://app.getsentry.com/pypi/warehouse/group/9475461/ as well.

Support Package Specific Mimetypes

It would be nice if we had real mimetypes for packages and not just application/x-tar or application/octect-stream.