heroku / heroku-buildpack-python

Heroku's buildpack for Python applications.

Heroku Buildpack: Python

This is the official Heroku buildpack for Python apps.

Recommended web frameworks include Django and Flask, among others. The recommended webserver is Gunicorn. There are no restrictions around what software can be used (as long as it's pip-installable). Web processes must bind to $PORT, and only the HTTP protocol is permitted for incoming connections.

See it in Action

$ ls
my-application		requirements.txt	runtime.txt

$ git push heroku main
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 276 bytes | 276.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> Installing python
remote: -----> Installing pip
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Collecting flask (from -r /tmp/build_c2c067ef79ff14c9bf1aed6796f9ed1f/requirements.txt (line 1))
remote:          Downloading ...
remote:        Installing collected packages: Werkzeug, click, MarkupSafe, Jinja2, itsdangerous, flask
remote:        Successfully installed Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 click-7.0 flask-1.0.2 itsdangerous-1.1.0
remote:
remote: -----> Discovering process types
remote:        Procfile declares types -> (none)
remote:

A requirements.txt must be present at the root of your application's repository to deploy.

To specify your python version, you also need a runtime.txt file - unless you are using the default Python runtime version.

Current default Python Runtime: Python 3.12.3

Alternatively, you can provide a setup.py file, or a Pipfile. Using pipenv will generate runtime.txt at build time if one of the field python_version or python_full_version is specified in the requires section of your Pipfile.

Specify a Buildpack Version

You can specify the latest production release of this buildpack for upcoming builds of an existing application:

$ heroku buildpacks:set heroku/python

Specify a Python Runtime

Supported runtime options include:

• python-3.12.3 on all supported stacks
• python-3.11.9 on all supported stacks
• python-3.10.14 on all supported stacks
• python-3.9.19 on Heroku-20 and Heroku-22 only
• python-3.8.19 on Heroku-20 only

heroku-buildpack-python's Issues

Django createsuper user fails on heroku

createsuperuser management command fails with:

return getpass.getuser().decode(locale.getdefaultlocale()[1]) TypeError: decode() argument 1 must be string, not None
related to:

https://code.djangoproject.com/ticket/16017

fixed by:

http://stackoverflow.com/questions/8981065/on-heroku-locale-getdefaultlocale-is-returning-none-none-breaking-django

heroku config:add LANG=en_US.UTF-8

Perhaps the fix is just to add to the compile script - but the link in the read me to build pack docs is broken - so I'm not sure if this is the best place for the fix?

Resource not found

When I try this command:

heroku create --stack cedar --buildpack [email protected]:heroku/heroku-buildpack-python.git

it says:

Creating morning-winter-2803... done, stack is cedar
 !    Resource not found

So I can't deploy my django 1.4 application, why I use --buildpack option?

cause it only detects my django application if django version equals to 1.3 and I need support for django 1.4.

Am I wrong in something??

best regards

thanks

Unable to push to Heroku, Django(1.4.a1) installed from github (in requirements.txt)

Installation of Django 1.4-a1 (to install the Django from the github HEAD)
pip install -e git+https:////github.com/django/django.git#egg=Django-dev

after freezing using pip freeze -l > requirements.txt I get
-e git+https://github.com/django/django.git@4bdd63bfd7a2318f0e3108ee714abf2defe25593#egg=Django-dev

I can use the above generated requirements.txt to reinstall the dependencies(to replicate the project), but it fails when it push it to github with the following error. I can figure out why this is failing.

 Command /usr/bin/git tag -l had error code 128
 Complete output from command /usr/bin/git rev-parse "fatal: Not a git repository: '.'":
 fatal: Not a git repository: '.'`

Just to provide a better idea, I will put the whole heroku push dump here

 Counting objects: 119, done.
 Delta compression using up to 2 threads.
 Compressing objects: 100% (67/67), done.
 Writing objects: 100% (119/119), 26.75 KiB, done.
 Total 119 (delta 35), reused 105 (delta 29)

 -----> Heroku receiving push
 -----> Python/Django app detected
  !     You have a virtualenv checked in. You should ignore the appropriate paths in your repo. See http://devcenter.heroku.com/articles/gitignore for more info.
 -----> Preparing virtualenv version 1.6.4
        New python executable in ./bin/python
        Installing setuptools............done.
        Installing pip...............done.
 -----> Django settings injection
        Injecting code into conf/settings.py to read from DATABASE_URL
 -----> Installing dependencies using pip version 1.0.2
        Obtaining Django from git+https://github.com/django/django.git@4bdd63bfd7a2318f0e3108ee714abf2defe25593#egg=Django-dev (from -r requirements.txt (line 1))
          Cloning https://github.com/django/django.git (to 4bdd63bfd7a2318f0e3108ee714abf2defe25593) to ./src/django
          Command /usr/bin/git tag -l had error code 128
          Complete output from command /usr/bin/git rev-parse "fatal: Not a git repository: '.'":
          fatal: Not a git repository: '.'

        ----------------------------------------
        Command /usr/bin/git rev-parse "fatal: Not a git repository: '.'" failed with error code 128
        Storing complete log in /app/.pip/pip.log
  !     Heroku push rejected, failed to compile Python/django app

 To [email protected]:ekoll.git
  ! [remote rejected] master -> master (pre-receive hook declined)

Can somebody explain what's going on here ?

Do not overwrite already specified paths with those set in config_vars

Currently, only the PATH environment variable will preserve it's original value, and append any additional paths to itself in case they are defined in the bin/release script under config_vars. However, others, like LD_LIBRARY_PATH will simply be overwritten by the values specified in config_vars, so one has to copy the original values there as well (at the moment only '/app/.heroku/vendor/lib'). Anyway, it would be convenient if LD_LIBRARY_PATH(and perhaps others) could preserve it's original value, and simply append the additional ones from config_vars. Custom buildpacks could benefit greatly from this, for instance when merging the latest updates from the original heroku buildpack, as there will be no need to track down what are the eventual path changes, and copy them to config_vars.

Detecting Django Applications with Settings Packages

Per discussion with @kennethreitz in #38:

Now that settings injection is disabled (in favor of using dj-database-url) it is possible to support recognition of Django projects (documentation) that use either a settings.py or a settings package.

Pip version is insecure.

Pip prior to version 1.3 used http to download from pypi http://www.pip-installer.org/en/latest/installing.html

As far as I can see from the commit log, the pip vendored into this buildpack is based on v1.2 and hasn't patched it to use ssl. Is there any reason not to update the vendored pip to use >= 1.3?

pylibmc not available during compile

Desipte the pylibmc requirement being detected correctly and libmemcached being made available at runtime, it is not available at compile time, in particular during the django/collectstatic run.

If you're using the CachedStaticFilesMixin this causes collectstatic to fail.

Not precisely sure what the issue is here (pylibmc detection and bootstrapping happens before any of the Django stuff) but if I track it down I'll update this ticket.

Don't inject code into my code; provide me with a helpful function.

The current buildpack injects code into my settings.py in order to set DB settings. This is bad and has lots of breakage.

I would be happy if you injected a function at the top of my settings.py so that I could call the function later. e.g.

### HEROKU INJECTED CODE BEGIN
def heroku_setup_database():
    # do magic
### HEROKU INJECTED CODE END

my_settings = 'fascinating'
my_other_settings = 'boring'

# Use Heroku helper to get DB settings from ENV
heroku_setup_database()

# or, rather than munging my locals, send me the dict :
DATABASES = heroku_parse_database()

Cannot Choose Requirements File

Many projects have multiple individual deployments, such as dev, staging and production. Often these environments have different requirements.

Breaking up these requirements is easy to do locally but since the Heroku buildpack always uses the root requirements.txt it's hard to do on Heroku, even with cascading.

Is there a way to do this? If not, is this something we could add?

Django 1.4 layout

If you follow heroku guide on django, but use 1.4, it won't be started on release. In logs:

Error H14 (No web processes running)

I assume this is due to new layout, specifically settings.py and manage.py being on different levels and default web process will not be able to find manage.py

Provide a way to disable Django settings injection.

Output from collectstatic

When using S3 (or similar backends) for collectstatic, various issues can arise during deployment that are silenced, hence the user is unaware of issues.

I am proposing some form of environment variable (or flag) to output debug information from collectstatic. Can I get some discussion going around this feature implementation?

Strip -e's from requirements.txt at build time.

This depends on an upcoming upstream fix to pip's url parsing mechanism.

Build explodes if user checks in virtualenv lib/ dir

Confirm in output that injection is disabled.

If the user has injection disabled via .heroku/injection_disabled , it would be nice to output this when the script runs so they know it's working.

Build error: pip 1.3 -> No such file or directory

I am using master head of the build pack and am getting the following error:

Counting objects: 113, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (94/94), done.
Writing objects: 100% (100/100), 17.84 KiB, done.
Total 100 (delta 63), reused 6 (delta 3)

-----> Deleting 0 files matching .slugignore patterns.
-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python.git
=====> Detected Framework: Python
-----> Found python-2.7.3, removing.
-----> Preparing Python runtime (python-3.3.0)
-----> Installing Distribute (0.6.35)
-----> Installing Pip (1.3.1)
-----> Installing dependencies using Pip (1.3.1)
/tmp/buildpackA8wRQ/bin/compile: line 164: /app/.heroku/python/bin/pip: No such file or directory
 !     Heroku push rejected, failed to compile Multipack app

To [email protected]:proximate-staging.git
 ! [remote rejected] dev -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:proximate-staging.git'

Better Django Detection

Be more liberal with the django checks, but check for Django in requirements.txt.

This will solve cases like #15

Django 1.4 project fails to initialize

Heroku does not seem to recognize my project. I'm not sure if this is at all related to #30.

See gist: https://gist.github.com/2ee3ddc51470ffeb3c76

No CLEAN_VIRTUALENV?

The handler for rebuilding virtualenv is gone from the compile script. What is the current method of purging virtualenv?

serving static files via gunicorn

although serving files works just fine when using aws s3 or rackspace cloudfiles it doesn't work for stuff where "same origin policy" blocks using files from CDN.
there are several ways to serve static files through python app (wsgi, django, ...) so it would be great if an official recommendation what to use when deploying on heroku was part of buildpack given that performance will be affected.

git pip requirements are not updated when the reference changes

If I specify a git+... requirement, it never updates after the first install, even if I change the ref to a new commit hash. It even uses the full git url + new hash in the line where it claims that the requirement is satisfied.

This does not occur if I run the same requirements file through pip locally (v 1.0.2, 1.1 or 1.2.1).

The only way to get it to update is to nuke everything via the CLEAN_VIRTUALENV workaround here: http://stackoverflow.com/questions/8937905/how-to-pip-uninstall-with-virtualenv-on-heroku-cedar-stack

Django: Check for manage.py too

There's a few people out there that exclude it for some reason ;)

GeoDjango

Consider supporting GeoDjango required libraries?

reference:
https://github.com/cirlabs/heroku-buildpack-geodjango/

thx

Document new assumptions from versions branch

I am working on some buildpack-compatible tools that are broken after the recent changes. The breakages center around the new requirement that Python be inside /app/.heroku/python. When trying to create stubs in /app so the buildpack will at least run, I get "cp: cannot stat `/app/!(tmp)': No such file or directory".

Ideally the buildpack would make no assumptions about /app existing (the pre-Dec 19 behavior). Short of that, it would be great to know what's now required in the host environment in order for the buildpack to work.

Looks like a typo in bin/compile

I think

fp#!/usr/bin/env bash

should be

!/usr/bin/env bash

deleted files remain in cache

If I add a file to bin/ in my repo, push to heroku, then remove that file from my repo, the file persists in the bin dir cache.

This is leading me to think that . isn't the best place for the virtualenv, perhaps use ENV/ as the docs suggest?

pre/post-compile hooks w/o using custom buildpacks

would you be open to add support for something like:
$ heroku config:set BUILDPACK_PRE_COMPILE=custom_script.sh

which would enable custom actions on build time (like using "django-admin.py collectstatic" w/ use-env-compile labs feature) where $BUILDPACK_PRE_COMPILE script must exists in git repo.

this would remove requirement to create custom buildpack for simple stuff like mentioned collectstatic problem.

Reclaim ./bin/ by moving virtualenv to ./venv/

Not 100% sure of the recommended directory by the virtualenv maintainers, but there is mounting evidence that . is not the best place for the virtualenv, as it can conflict with a users repo bin/ or lib/ dir, etc.

The virtualenv should be moved to a different sub directory and set PATH appropriately.

Explain required project structure better

The heroku docs and this project's readme should explain the format better. I had to read the code to adjust my existing application.

Most of the 3rd party docuemtnation I've read captured the fact that you need requirements.txt in the root and everything else in an appname/ subdirectory.

None of them captured that settings.py needed to be at appname/settings.py. Mine was in another subdirectory and was particularly hard to debug because Heroku was recognizing my app as a Python app. Took me a long time to figure out that's what was wrong.

I considered proposed another issue suggesting that the project structure be made more flexible (where your folders are, where settings.py is, where requirements.txt is), but I'll pass on that for now as long as there is some better documentation.

push fails for python buildpack with pip 1.3.1

-----> Fetching custom git buildpack... done
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.3.
-----> Preparing Python runtime (python-2.7.3)
-----> Installing Distribute (0.6.35)
-----> Installing Pip (1.3.1)
-----> Installing dependencies using Pip (1.3.1)
/tmp/buildpack_1ehndh7bfq63/bin/compile: line 164: /app/.heroku/python/bin/pip: No such file or directory
 !     Heroku push rejected, failed to compile Python app

Works with pip 1.2.1 (v16 of buildpack)

Allow for Mercurial pip Installs

Pip supports installing from an mercurial repository, but obviously requires hg to be installed first. Is it possible to install hg in the pre-compile step?

Vendored venv re-uses existing python binary over system binary

A stack image with Python 2.7.2 is rolling out. Pushing an existing app against this, however, still keeps the Python 2.7.1 binary from the existing venv.

The only explanation is that virtualenv is re-using the existing/old bin/python when it does the "New python executable in ./bin/python" step.

Need to dig into virtualenv to see how its finds a python binary, and get it to always use the system one.

Run `manage.py collectstatic` in buildpack

This would make scenarios like this unnecessary.

pip command not found on recent commit

I'm using the multi buildpack with node and python:
BUILDPACK_URL: https://github.com/ddollar/heroku-buildpack-multi.git

My .buildpacks contains:

https://github.com/heroku/heroku-buildpack-python.git
https://github.com/heroku/heroku-buildpack-nodejs.git

This is the output from my git push heroku master:

[gamen] 16:01:27 [email protected] gamen master ? git push heroku master
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 843 bytes, done.
Total 9 (delta 7), reused 0 (delta 0)
-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python.git
=====> Detected Framework: Python
-----> No runtime.txt provided; assuming python-2.7.3.
-----> Preparing Python runtime (python-2.7.3)
-----> Installing Distribute (0.6.32)
-----> Installing pip (1.2.1)
/tmp/buildpackGKM0H/bin/compile: line 177: pip: command not found
 !     Heroku push rejected, failed to compile Multipack app

To [email protected]:gamen.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:gamen.git'

I've been using this setup for about 2 weeks. I've noticed that there was a commit yesterday and it seems to have broken my app. Any light on this?

Thanks!

Give nicer feedback when PyPi appears to be down.

Problem with 2.7 libraries on cedar?

Reported by a Herkou user, not yet verified by myself.

Would be nice to find a way to reproduce without RubyPython...

heroku run console --app myapp-cedar 
Running console attached to terminal... up, run.7 
irb(main):001:0> require 'rubypython' 
=> true 
irb(main):002:0> RubyPython.start 
LoadError: Could not open library '/usr/local/lib/libpython2.7.a': /usr/local/lib/libpython2.7.a: invalid ELF header 

pip uninstall/removing packages does not work

I know this issue has been raised multiple times, but it remains to exist.

Right now, the only way to force a rebuild of your virtualenv seems to be to set a different runtime: https://devcenter.heroku.com/articles/python-runtimes#changing-runtimes

But I don't want to change my python runtime just so that I can remove a tiny package that I have modified, and instead want to use from my source tree.

I think being able to uninstall pagckages, even if you have set some flag to enable it, is a very important functionality. Thank you.

Change default python without changing existing python

Django app not detected

Heroku detects a Django app as a Python app when requirements.txt points to django default requirements files

--requirement=<myapp>/requirements/project.txt

I initially did it that way because pip freeze doesn't preserve --extra-index-url links.

Please use explicit commit messages

I'm going through the buildpack history trying to figure out at what point projects stopped to compile and if there were any changes related to pylibmc/libmemcached.

Having a clean commit log with messages explaining what actually has been changed and why would be great (and much more helpful than a stream of random kill the source, better, remove bunk, much better).

See http://ablogaboutcode.com/2011/03/23/proper-git-commit-messages-and-an-elegant-git-history/ for a suggestion of how proper git messages can be structured. Thanks.

package installed but not in requirements.txt

package that was installed in previous push wasn't removed after it (django-piston) was removed from requirements.txt.
when i try to run "pip install -r requirements.txt" in a new "virtualenv --no-site-packages env" it doesn't install piston.

when connected w/ "heroku run bash":

~ $ pwd
/app

~ $ ls lib/python2.7/site-packages/
piston
django_piston-0.2.3-py2.7.egg-info
...

~ $ cat requirements.txt |grep piston

django-piston

user-env-compile for custom S3 bucket

for staging

No changes on my end, scratching my head at "error fetching custom build-pack"

Counting objects: 16, done.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 13.34 KiB, done.
Total 9 (delta 6), reused 1 (delta 0)

-----> Heroku receiving push
-----> Fetching custom buildpack... failed
 !     Heroku push rejected, error fetching custom buildpack

To [email protected]:pbmobile.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]'

Really don't know where to start debugging. If you recall, I'm the guy with the weird settings setup whereby I have to run
bin/python manage.py {development/production} syncdb

See #46 for setup.

bin/steps/pylibmc should use list_requirements function

The detect script uses a list_requirements function which supports -r inclusion of other files. It would be nice if the pylibmc script made use of this as well.

Looks like a few people have been caught out by this: rdegges/django-heroku-memcacheify#4

support sqlite3 module

Extension Mechanism

The third-party buildpacks are in general really really good but sometimes it would be useful to simply extend the master one.

I was thinking of a way to add extra build steps so an app could still use this buildpack but set paths to extra plugins (maybe a url to a github repo?). In this manner I could vendor simple extras as I've done here with wkhtmltopdf: https://github.com/ghickman/heroku-buildpack-python/blob/master/bin/steps/wkhtmltopdf

This way you could stay up to date with the changes made here while using the extensions you need and avoiding having to pull in upstream changes all the time.

corrupt virtualenv breaks app deployment

While trying to deploy a database-less django app, I did the following:

1. push to heroku
2. realize I don't want a database (and don't have psycopg2 in requirements.txt)
3. Remove the database addon
4. Set DISABLE_INJECTION=1
5. push to heroku, see injection was still attempted
6. run heroku labs:enable user_env_compile
7. push to heroku

The last push resulted in:

Counting objects: 14, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (14/14), 3.75 KiB, done.
Total 14 (delta 0), reused 0 (delta 0)

-----> Heroku receiving push
-----> Python/Django app detected
-----> Preparing Python interpreter (2.7.2)
ImportError: No module named site
-----> Creating Virtualenv version 
 !     Virtualenv corrupt, rebuilding.
ImportError: No module named site

-----> Activating virtualenv
/app/slug-compiler/lib/../buildpacks/python/bin/compile: line 172: pip: command not found
-----> Installing dependencies using Pip version 
/app/slug-compiler/lib/../buildpacks/python/bin/compile: line 173: pip: command not found
 !     Heroku push rejected, failed to compile Python/django app

To [email protected]:restpaste.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:restpaste.git'

Trying to re-push doesn't seem to help.

Since this is a toy app I will destroy and recreate it, but I thought you might be interested.

Issue With Bootstrapping Detection

Heya,

I've hit an issue with this buildpack's bootstrapping detection. Below is my attempt to explain this in the best way possible, if this isn't clear, please let me know ^^

I've got a Django project, whose top-level directory looks like so:

requirements.txt
requirements/common.txt
requirements/dev.txt
requirements/prod.txt

My requirements.txt (which Heroku detects) looks like this:

-r requirements/prod.txt

Essentially--I'm using my top-level requirements.txt file as a symlink, so that I tell Heroku to only install my production requirements (which are specified in requirements/prod.txt).

The issue is this:

My production requirements (requirements/prod.txt) include pylibmc==1.2.3. If I specify pylibmc==1.2.3 in my top-level requirements.txt file, all is good because when I push to Heroku, it detects pylibmc and 'bootstraps' its c dependencies.

If I simply leave pylibmc in my requirements/prod.txt file, it gets installed--but Heroku doesn't detect it and 'bootstrap' it, so my application crashes.

As a workaround I've modified my top-level requirements.txt file to include:

-r requirements/prod.txt
pylibmc==1.2.3

But this feels like a hack.

My thought (and suggestion), is that you scan all included requirement files and detect any bootstrappable dependencies, not just the top-level requirements.txt file.

Detect a settings module as Django project type

I have a project with the following structure (following the recent advice of BDFL @jacobian, http://www.slideshare.net/jacobian/the-best-and-worst-of-django slide 51)

root/
    Procfile
    requirements.txt
    project/
        settings/
            __init__.py (empty)
            base.py
            sqlite.py
            postgres.py

Unfortunately, Heroku doesn't detect this as a Django project due to the fact that a settings.py file does not exist.

Error collecting static data Django 1.4

My app, which was previously working fine, now errors on push:

-----> Collecting static files
Unknown command: '--noinput

The one unorthodox part of my application is that I store separate settings for my production settings in a subfolder

.
├── application
│   ├── settings
│   └── web
├── requirements

in settings sub folder

├── __init__.py
├── base.py
├── development.py
└── production.py

and my manage.py

#!/usr/bin/env python

import os
import sys


if __name__ == "__main__":
    from django.core.management import execute_from_command_line

    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings.{}".format(sys.argv[1]))
    execute_from_command_line([sys.argv[0]] + sys.argv[2:])