storborg / python-packaging

Tutorial on how to structure Python packages

This tutorial is hosted at http://www.scotttorborg.com/python-packaging/

Feedback is welcome, please submit a pull request or email [email protected].

python-packaging's Issues

minimal.rst doesn't work

As written the tutorial is incorrect.

Delete project because out of date

The last commit is 6 years old and so is the information's.

Your nice looking readthedocs version of that repo expose a lot of informations about python and packaging that are not only not up to date but very problematic.

Today python itself has a lot of very good resources where you can learn how to package something. There is no need for such a 3rd party documentation anymore. That is why I would say don't upgrade your repo because there is no need for this anymore.

It will save a lot of time if this outdated information's wouldn't exist anymore. I often have people pointing me to your readthedocs and I have to explain them why the advices from there are not up to date anymore.

Please drop the reference to 'dependency_links' - it is no longer supported

The section Packages Not On PyPI recommends using the 'dependency_links' argument to setup().

According to the User Guide

Warning: Dependency links support has been dropped by pip starting with version 19.0 (released 2019-01-22).

Please update the Guide.

`test_require` is deprecated.

https://github.com/storborg/python-packaging/blob/35daf9938a63f7deb4b25cb1c5c65578c73f28e4/testing.rst recommends

setup(
    ...
    test_suite='nose.collector',
    tests_require=['nose'],
)
...
$ python setup.py test

If you actually try this, you'll get

WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.

(from pypa/setuptools#1878). There's a long thread (pypa/setuptools#931) where the PyPA devs talk about their plan to drop setup.py test entirely, apparently in favour of expecting you to tox instead. Except the tox maintainer told me that tox isn't really meant to be used that way.

Anyway, in the middle of all this confusion pypa/setuptools#1684 (comment) claims

I don't think we've ever tried to disguise this. For several years now we have been actively warning people not to invoke setup.py directly at every opportunity, and we've been quite explicit that the end goal is for setuptools to lose as many of its extraneous features as possible and become a standard build tool.

but I'd never seen any of those warnings. You, as one of the top hits for "python packaging" was also unaware. So I don't really know whose advice to trust. What do you think should be recommended?

Note any significant distinctions between distutils and setuptools

Aloha,

"you’ll need to supply include_package_data=True to the setup() function."

/usr/local/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'include_package_data'
warnings.warn(msg)

It seems that (at least with python3) this option is no longer available.

Is this guide outdated?

Is this guide outdated?

Could you please say something about wheel? Even one sentence can be helpful.

Installing nose fails in `python setup.py test`, works second time through

I'm working on a homework assignment for our budding Python packagers, based on this repo.

We're using Python 2.7.13 built via spack.

The script sets up working directory and a "dev virtual environment" (perhaps not strictly necessary) the clones this repo and tries to run python setup.py test.

The first time it runs it tries to install nose-cover3 twice and then complains about not being able to install nose and exits. If I run python setup.py test again it installs nose and then things work as they should.

I discovered that if I reverse the order of the test dependencies in setup.py then it succeeds, e.g.:

--- a/funniest/setup.py
+++ b/funniest/setup.py
@@ -26,7 +26,7 @@ setup(name='funniest',
           'markdown',
       ],
       test_suite='nose.collector',
-      tests_require=['nose', 'nose-cover3'],
+      tests_require=['nose-cover3', 'nose'],
       entry_points={
           'console_scripts': ['funniest-joke=funniest.command_line:main'],
       },

I'm not enough of a Python guru to understand what's going on. Perhaps nose-cover3 has messed up metadata or .... In any case it doesn't seem to be a bug here but I'm hoping you might be able to shed some light.

Below is a cut-and-paste of a sample session.

Thanks!

g.

[me@blah ~]$ # make a directory for the demo
[me@blah ~]$ DEMO_DIR=$(mktemp -t -d ${USER}-PYTHON_DEMO.XXXXX)
[me@blah ~]$
[me@blah ~]$ # set up a virtualenv for our dev work
[me@blah ~]$ # this might not be *required*, but it's safer when you're me...
[me@blah ~]$ V_DEV=${DEMO_DIR}/dev_virtenv
[me@blah ~]$ virtualenv ${V_DEV}
 test deps....  SIGH...
python setup.py test
python setup.py test
New python executable in /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin/python2.7
Also creating executable in /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin/python
Installing setuptools, pip, wheel...done.
[me@blah ~]$ source ${V_DEV}/bin/activate
(dev_virtenv) [me@blah ~]$
(dev_virtenv) [me@blah ~]$ # clone the python packaging repo (w/ funniest project)
(dev_virtenv) [me@blah ~]$ PP=${DEMO_DIR}/python-packaging
(dev_virtenv) [me@blah ~]$ git clone https://github.com/storborg/python-packaging ${PP}
Cloning into '/tmp/me-PYTHON_DEMO.tNpOB/python-packaging'...
remote: Counting objects: 167, done.
remote: Total 167 (delta 0), reused 0 (delta 0), pack-reused 167
Receiving objects: 100% (167/167), 34.13 KiB | 0 bytes/s, done.
Resolving deltas: 100% (79/79), done.
(dev_virtenv) [me@blah ~]$ cd ${PP}/funniest
(dev_virtenv) [me@blah funniest]$
(dev_virtenv) [me@blah funniest]$ # install prerequisites
(dev_virtenv) [me@blah funniest]$ python setup.py develop
running develop
running egg_info
creating funniest.egg-info
writing requirements to funniest.egg-info/requires.txt
writing funniest.egg-info/PKG-INFO
writing top-level names to funniest.egg-info/top_level.txt
writing dependency_links to funniest.egg-info/dependency_links.txt
writing entry points to funniest.egg-info/entry_points.txt
writing manifest file 'funniest.egg-info/SOURCES.txt'
reading manifest file 'funniest.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'funniest.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/lib/python2.7/site-packages/funniest.egg-link (link to .)
Adding funniest 0.1 to easy-install.pth file
Installing funniest-joke script to /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin

Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest
Processing dependencies for funniest==0.1
Searching for markdown
Reading https://pypi.python.org/simple/markdown/
Downloading https://pypi.python.org/packages/29/82/dfe242bcfd9eec0e7bf93a80a8f8d8515a95b980c44f5c0b45606397a423/Markdown-2.6.9.tar.gz#md5=56547d362a9abcf30955b8950b08b5e3
Best match: Markdown 2.6.9
Processing Markdown-2.6.9.tar.gz
Writing /tmp/easy_install-f7q0Sx/Markdown-2.6.9/setup.cfg
Running Markdown-2.6.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-f7q0Sx/Markdown-2.6.9/egg-dist-tmp-80aaNi
zip_safe flag not set; analyzing archive contents...
Moving Markdown-2.6.9-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/lib/python2.7/site-packages
Adding Markdown 2.6.9 to easy-install.pth file
Installing markdown_py script to /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin

Installed /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/lib/python2.7/site-packages/Markdown-2.6.9-py2.7.egg
Finished processing dependencies for funniest==0.1
(dev_virtenv) [me@blah funniest]$
(dev_virtenv) [me@blah funniest]$ # edit, test, edit, test, yay
(dev_virtenv) [me@blah funniest]$ # edit...
(dev_virtenv) [me@blah funniest]$ # install test deps fails to install nose the first time around...
(dev_virtenv) [me@blah funniest]$ # do it again and it works.  Sigh...
(dev_virtenv) [me@blah funniest]$ # or swap order of 'nose' and 'nose-cover3' in test deps....  SIGH...
(dev_virtenv) [me@blah funniest]$ python setup.py test
running test
Searching for nose-cover3
Reading https://pypi.python.org/simple/nose-cover3/
Downloading https://pypi.python.org/packages/f0/17/8c55242e86830a006bbaa0463f4a1da44f332ef7cd5a402f459c8dbaaf84/nose-cover3-0.1.0.tar.gz#md5=82f981eaa007b430679899256050fa0c
Best match: nose-cover3 0.1.0
Processing nose-cover3-0.1.0.tar.gz
Writing /tmp/easy_install-81NDHC/nose-cover3-0.1.0/setup.cfg
Running nose-cover3-0.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-81NDHC/nose-cover3-0.1.0/egg-dist-tmp-V4JbFB
warning: no previously-included files matching '*.pyc' found anywhere in distribution
creating /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Extracting nose_cover3-0.1.0-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs

Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Searching for nose
Downloading https://pypi.python.org/packages/f0/17/8c55242e86830a006bbaa0463f4a1da44f332ef7cd5a402f459c8dbaaf84/nose-cover3-0.1.0.tar.gz#md5=82f981eaa007b430679899256050fa0c
Best match: nose cover3-0.1.0
Processing nose-cover3-0.1.0.tar.gz
Writing /tmp/easy_install-OZM0ym/nose-cover3-0.1.0/setup.cfg
Running nose-cover3-0.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-OZM0ym/nose-cover3-0.1.0/egg-dist-tmp-OmgVZ0
warning: no previously-included files matching '*.pyc' found anywhere in distribution
removing '/tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg' (and everything under it)
creating /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Extracting nose_cover3-0.1.0-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs

Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 34, in <module>
    zip_safe=False)
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-fiaifkmalam23l42s4lyqg2kithvppfi/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-fiaifkmalam23l42s4lyqg2kithvppfi/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-fiaifkmalam23l42s4lyqg2kithvppfi/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/setuptools/command/test.py", line 199, in run
    installed_dists = self.install_dists(self.distribution)
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/setuptools/command/test.py", line 195, in install_dists
    tr_d = dist.fetch_build_eggs(dist.tests_require or [])
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/setuptools/dist.py", line 375, in fetch_build_eggs
    replace_conflicting=True,
  File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/pkg_resources/__init__.py", line 854, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'nose' distribution was not found and is required by the application
(dev_virtenv) [me@blah funniest]$ python setup.py test
running test
Searching for nose-cover3
Best match: nose-cover3 0.1.0
Processing nose_cover3-0.1.0-py2.7.egg

Using /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Searching for nose
Reading https://pypi.python.org/simple/nose/
Downloading https://pypi.python.org/packages/58/a5/0dc93c3ec33f4e281849523a5a913fa1eea9a3068acfa754d44d88107a44/nose-1.3.7.tar.gz#md5=4d3ad0ff07b61373d2cefc89c5d0b20b
Best match: nose 1.3.7
Processing nose-1.3.7.tar.gz
Writing /tmp/easy_install-LokDOS/nose-1.3.7/setup.cfg
Running nose-1.3.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-LokDOS/nose-1.3.7/egg-dist-tmp-YF_UWz
no previously-included directories found matching 'doc/.build'
creating /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose-1.3.7-py2.7.egg
Extracting nose-1.3.7-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs

Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose-1.3.7-py2.7.egg
running egg_info
writing requirements to funniest.egg-info/requires.txt
writing funniest.egg-info/PKG-INFO
writing top-level names to funniest.egg-info/top_level.txt
writing dependency_links to funniest.egg-info/dependency_links.txt
writing entry points to funniest.egg-info/entry_points.txt
reading manifest file 'funniest.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'funniest.egg-info/SOURCES.txt'
running build_ext
test_basic (funniest.tests.test_command_line.TestCmd) ... ok
test_is_string (funniest.tests.test_joke.TestJoke) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.030s

OK

Test not detected

Hi!
Im following the tutorial and copied the code, but the
python setup.py test
seems like not detecting the test.

I found the solution here:
https://stackoverflow.com/questions/17001010/how-to-run-unittest-discover-from-python-setup-py-test
In which the test_suite in the setup.py file is changed to

test_suite="tests",

readme function fails on install

How can I solve this issue ? Thanks!

Collecting test
Downloading http://pypi.example.com/repository/2/test-0.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 20, in
File "/tmp/pip-build-EAllOQ/test/setup.py", line 11, in
long_description=readme(),
File "/tmp/pip-build-EAllOQ/test/setup.py", line 4, in readme
with open('README.md') as f:
IOError: [Errno 2] No such file or directory: 'README.md'

Non-code-files guidance requires unnecessary repetition

Thanks for these docs. I was searching for how to handle package data and it took me to http://python-packaging.readthedocs.io/en/latest/non-code-files.html. I followed that technique in one project before I realized there's a more automated way to include such files.

Including setuptools_scm as a setup_requires declaration will invoke the setuptools file_finder to automatically include any files found in the SCM (e.g. Git) repository. Thus, the developer doesn't have to maintain a separate manifest (one in a MANIFEST.in and another in the SCM repo) and run the risk of the two being inconsistent.

I believe non-code-files deserves at least a nod to file finders as a more automated solution to the problem of non-code-files.

`.pypirc` file not mentioned

Hi,

This issue (quite extensive) revolves around the lack of a .pypirc file and lack of clarity around the fact that it is needed.

I found that the python-packaging docs don't mention this file (specifically, this page), so I got stuck at the upload step.

I used this python.org page, which does mention the file, to get un-stuck.

Place a link to the tutorial

Under the project description, there is a dedicated field for a URL. I suggest placing http://www.scotttorborg.com/python-packaging there so the tutorial is easy to find (and visit) from this repo.

gh_pages

It would be really cool to have a little section about hosting your sphinx generated site hosted through gh_pages!

Top level tests directory

https://github.com/storborg/python-packaging/blob/35daf9938a63f7deb4b25cb1c5c65578c73f28e4/testing.rst recommends

tests [..] should be placed in a submodule of funniest. so that they can be imported, but won't pollute the global namespace.

There's a good case to be made that tests should not live inside your package, but instead inside a separate top-level tests/. This is how one of the PyPA core devs does it. This is also what https://packaging.python.org/tutorials/packaging-projects/ (which makes sense, it's written by the same group). The reason for this is it lets you cull tests from your binary distribution, since people installing from binary almost never care to run your tests or have them take up space -- though you need to go out of your way to make sure to re-include them in your sdist, but that's a one-line addition.

https://stackoverflow.com/questions/61151/where-do-the-python-unit-tests-go covers the three main options:

1. Side-by-side your code (and says they prefer it; to be honest, so do I, it feels more right)
2. In a top level tests/
3. In a tests/ subpackage

The current recommendation in this doc is for 3. PyPA thinks you should use 2. What are your thoughts?

Publishing on PyPi register and upload methods deprecated/not-recommended

python3 setup.py register
is deprecated and as per the official guide is not recommended due to security issues. Similar issues with
python2 setup.py upload

The recommended way is to use twine to register and upload the package.

Git repo name incorrect in setup.py

In setup.py:
https://github.com/storborg/python-packaging/blob/master/funniest/setup.py

the url (line 20) is listed as follows:

  url='http://github.com/storborg/funniest',

This repo does not exist, I believe it ought to be:

  url='http://github.com/storborg/python-packaging',

ensuring submodules work

hi storborg,

perhaps you should include a section on how to include submodules and ensure that they work

ie either specify the submodules specifically by name on the packages section or include
packages=setuptools.find_packages(),
in your setup.py

thanks

Install the package from github

You might mention this option:

pip install git+https://github.com/myaccount/mypackage

or even

pip install git+https://github.com/myaccount/mypackage@version

where version can be either a branch, a tag or a commit

Handle files created on the fly.

Hi, thanks for this great tutorial, I wanted to ask a question -

My python script which I have packed writes to a file, this file is not shipped with the package so in essence it is created on the fly for the user (that is when he runs the package.).

Now I want to know, how do I handle such a file?

Currently I have not specified it in my setup.py because it has not been shipped with the package, do I need to define it in setup.py and maybe ship it is an empty file?

Setup.py inconsistency

Setup.py refers to "README.rst.example" but everything.rst refers simply to README.rst - if you go strictly by http://python-packaging.readthedocs.org/en/latest/everything.html it won't work because it expects the README.rst.example, which isn't called out as a needed file on the page.

I'd suggest just renaming README.rst.example to README.rst and updating the reference in setup.py accordingly.

(I assume it was renamed to keep it from rendering the readme on the github page for https://github.com/storborg/python-packaging/tree/master/funniest - an empty README.md if present should take priority in rendering for the purpose of how github's UI displays things - see github/markup#360 )

Please change basestring to str

Would you please change basestring to str. The basestring is no longer available in Python 3, and it was removed. Use str instead.

Shadowing of stdlib "cmd.py" module can cause confusion

This was by far the best tutorial I've read on creating pip-compatible setups. However, the use of "cmd.py" in the examples could possibly shadow the standard library's cmd.py so I'd recommend changing it to something else in the locations it exists.

I think this project brings more confusion than benefit.

I think this project brings more confusion than benefit.

If the upstream docs are missing something, then please talk to the upstream.

According to this page, the guide is eight years old: https://python-packaging.readthedocs.io/en/latest/about.html

Unfortunately python-packaging is the first match if you google for "setup.py dependencies".

Instead of more documentation, we need a clear central place for the python packaging docs.

requirements.txt

Could you add an explicit statement about a requirements.txt file in the repo if your package relies on something besides the python core library? I can make the actual documentation update if I knew what the answer was.

Needs updating for Python 3.x

At the time of writing, Python 2.7 is EOL in under 10 months, but https://python-packaging.readthedocs.io/en/latest/ says:

At this time, this documentation focuses on Python 2.x only, and may not be as applicable to packages targeted to Python 3.x.

So as increasing numbers move to Python 3.x, it needs a sanity check to make sure it's not accidentally misleading people who miss that caveat.

Really "opening" this project means to have a license

Problem

The source code and tutorial are both available online, but the default copyright in most countries forbid people from using it legally

Solution

Add a license to the repository and all it's files and other people can use it legally.

Because it is not really source code it shouldn't be a program license like GPL.

My advice:

• CC-BY-SA - this is used by many many human language projects like Wikipedia (!)
• GFDL - the internet says it should also be good, but I've read that the CC-BY-SA is used much more often

If you want help in implementing one or another, tell me. I'm happy to help!

Split out the "funniest" module into its own repo?

Hi,

Thanks for your great tutorial!

At the end, though, I was thinking: is there somewhere I can go and download the source to funniest so that I can use it as a template for my own package? Turns out, yes, there's this repository - but its not quite set up to do that. Would you mind splitting this into two repositories, so that funniest could be forked, downloaded, and tested independently of the tutorial?

Please, do not recommend nose

I am a maintainer of Python packages in openSUSE, and I am on my crusade of eliminating nose1 from our distribution. When I look at its repository on https://github.com/nose-devs/nose, the last release 1.3.7 was on 2 Jun 2015, and even the last commit on the master branch was on 4 Mar 2016. Moreover, nose is not compatible with Python >= 3.9.

So, it was with horror I was looking at testing.rst and see this:

The best way to get these tests going (particularly if you're not sure what to use) is Nose.

Please, do not recommend this wreck. python3 -munittest discover -v works usually just fine, and if you have to use alternative test runner, then please recommend pytest.