ucl-arc / python-tooling

Python package template for new research software projects

UCL ARC Python Recommendations

This repository collects the UCL ARC recommendations for a research software project in Python. It contains a template for new Python packages and a website documenting our recommendations. We've turned on discussions for this repo, and we welcome questions there or in the #helpme channel on the UCL research programming hub Slack.

🍪 Our template is a cookiecutter template which automatically creates new Python packages with our recommended tooling set up and ready to go.

Note If you're making a package within a community that has an existing package template (e.g., scikit-hep), we recommend using their template instead of this one.

Using this template

1. Install cookiecutter in a Conda or venv environment (commented lines for Conda example).

   # conda create --channel conda-forge --name new-env-name
   # conda activate new-env-name
   # conda install pip
   pip install cookiecutter
   

2. Run cookiecutter in the desired directory

   cookiecutter gh:ucl-arc/python-tooling
   

   If you have this repo locally (this may be the case if you are developing), you can run the following:

   cookiecutter /path/to/your/checkout/of/python-tooling
   

3. A series of questions will pop up to configure the project. Type the answer or hit return to use the default option (shown in square brackets).

   Note that these project variables are defined in the cookiecutter.json file.

4. This will create a specific directory structure.

   For example, for a project with the following variables:

   project_name [Python Template]: PROJECT_NAME
   project_slug [python-template]: PROJECT_SLUG
   

   we will get a project folder named PROJECT_SLUG, structured like this:

   PROJECT_SLUG
   ├── ...
   ├── README.md
   ├── pyproject.toml
   ├── src
   │   └── PROJECT_SLUG
   │       └── PROJECT_SLUG.py
   └── tests
       └── test_dummy.py
   

   And the PROJECT_NAME will appear in the README.md as the human-readable name of the project.

   cat PROJECT_SLUG/README.md
   # PROJECT_NAME
   ...
   

5. To work on your project, initialise a git repository and install it in editable mode.

   cd PROJECT_SLUG
   git init
   python -m pip install -e ".[dev]"
   

6. Build your package

   python -m build
   

Notes for developers

Click to expand...

First, clone repository

• (Optional) Generate your SSH keys as suggested here
• (Optional) GitHub CLI as suggested here
• Clone the repository by typing (or copying) the following line in a terminal at your selected path in your machine:

git clone [email protected]:UCL-ARC/python-tooling.git
cd python-tooling

Developing the cookiecutter template

• To create and remove your virtual environment

  conda create -n ptoolingVE pip -c conda-forge
  conda activate ptoolingVE
  conda deactivate ptoolingVE
  conda remove -n ptoolingVE --all
  

• To run template in the same path of this repo. We do a test cookiecut of a dummy package and install it to ensure the template setup works.

  cookiecutter .
  cd python-template
  git init
  python -m pip install -e ".[dev]"
  

• To run cookiecutter using a specific branch of the template:

  cookiecutter https://github.com/UCL-ARC/python-tooling --checkout <branch-name>
  

• To run the tests locally:

  pytest -s
  

Developing the recommended tooling pages

Pages all live in the docs/pages sub-directory, and are written in markdown.

To build the webpage locally (for testing)

1. Install jekyll
2. Run bundle install from the docs/ directory of this repository to install dependencies.
3. Run bundle exec jekyll serve from the root directory of this repository. This should fire up a local web server and tell you its address. By default the server will automatically refresh the HTML pages if any changes are made to the markdown sources.

See the jekyll docs for more info.

python-tooling's Issues

Initialise a `git` repository if one doesn't already exist

See @p-j-smith's suggestion here: #13 (comment)

Start packaging page

Drop `3.8` support

https://numpy.org/neps/nep-0029-deprecation_policy.html

Start development tools page

Start CI page

Ruff should be green

Start documentation page

Review existing content

Would be good for someone who isn't me to proof read and suggest improvements to the initial content on the site.

Merge queues

Feature request

To the project admin to set up merge queue rule.

Better/easier ways to contribute to the repo

Contributing to this repo via fork is a bit annoying than doing to what it is already suggested https://github.com/UCL-ARC/python-tooling/blob/main/README.md. Perhaps current contributors need to be added as writers or admins in this repo to help with PR reviews or add other potential contributors.

Adding scikit-surgery python template based on Cookiecutter to index.md

to add https://github.com/SciKit-Surgery/PythonTemplate

The CI should build an example cookiecutter project and check that it worked.

Current CI just runs tox. Be nice to make an elaborate CI.yml that creates a cookie-cutter project and checks things are sensible.

Could maybe do the whole thing from bash commands on the CI runner.

A switch between manual and automatic versioning

@harryjmoss has a use case that requires static versioning and so maybe a flag to select whether to use setuptools-scm or not

Add codespell to pre-commit config

A suggestion to add https://github.com/codespell-project/codespell. I've found it useful in the past.

Nice way to pull the corresponding license.

The cookiecutter examples seem to recommend a giant if clause in LICENSE.

Not sure this is the smartest way, and it feels like there's probably a better solution.

Other tools to manage environments

Perhaps add some other tools manage environments (e.g, Virtualenv, venv or pyenv)

Investigate a package template to recommend

Start linting page

Discorse and Gitter should be added to the community page.

• Discourse
• Gitter

Explain why items are green

Should we add werkzeug to the benchmarking page?

• https://pypi.org/project/Werkzeug/

@sfmig you've been using it... which makes you expert. So here's a ticket 😁.

If the answer is "yes": please add it. If "no": please close with wontfix.

Start testing page

Automatically bump pre-commit hook versions

Some of the pre-commit hooks (e.g. ruff) have newer versions

We could use something like pre-commit.ci or renovate to automatically keep the versions up to date?

Should we separate benchmarking and profiling tools?

The benchmarking section right now includes profiling and benchmarking tools.... should we split them for clarity?

Reformat linting.md

Split into subcategories of linting, put 🟢 first.

Add memory_profiler to benchmarking page

https://github.com/pythonprofilers/memory_profiler

Can we specify a list of python versions and loop through them?

The combination of https://github.com/UCL-ARC/python-template/blob/49898d88e6f46e0e214b0d4eecee2d07c19cad82/cookiecutter.json#L9 and https://github.com/UCL-ARC/python-template/blob/49898d88e6f46e0e214b0d4eecee2d07c19cad82/%7B%7Bcookiecutter.project_slug%7D%7D/pyproject.toml#L109-L129 doesn't work at the moment. What I think would be good would be specifying either a list of python versions to support or a min and/or max version and do something clever.

README adding git remote

Should we maybe add a quick note about creating an empty remote repo and then following the instructions on github.com to link the local repo with the remote one?

Make setuptools green

Add CITATION.cff

Should be able to split cookiecutter.full_name to grab the first name and surname for CITATION.cff.
Introduced in the history of #3 but that was broken.

• Can we have multiple authors in cookie-cutter templates?
• Remove citation section from README?

Add our 🟢 tools.

Here's a list of our 🟢 🚦 from the webpage.

• pyinstrument

• snakeviz

• asv

• github actions

• codecov

• sphinx

• sphinx-gallery

• (binder)

• pre-commit

• black

• toml

• isort

• (ruff? autoflake)

• mypy

• build

• setuptools

• cibuildwheel

• pytest

• tox

• pytest-cov

• pytest-mock

• mamba

Adding further details to community section

Table can be improved adding used by and #87

Clarify some (obvious to us) best practices about repo organisation etc.

We probably want somewhere to write down, i.e. one package per repo. And link to package vs module distinction (probably somewhere in the official python docs).

These kinds of things are obvious to us, but likely not to new python developers.

Not really sure where this should go. But probably should be somewhere here.

Relates to

• #53

Add conda to list of virtual envs

• maybe also mamba?
• conda should be 🟢 item of virtual envs.

tox configuration requires tox version 4

While using this template for https://github.com/UCL/dxh (thanks all for the work in getting this set up 😁) I found that the default legacy_tox_ini in the [tool.tox] section of the generated pyproject.toml

https://github.com/UCL-ARC/python-template/blob/ddf6cc2f8388efa9b9b2c8873f2ea5e837247b01/%7B%7Bcookiecutter.project_slug%7D%7D/pyproject.toml#L144-L169

needed a few tweaks before I could get tox running locally with the version I had installed (v3.24.5)

• Removing the whitespace indentation within the """ .... """ multiline string
• Adding isolated_build = true to the [tox] section

Might be worth documenting this and/or pinning the tox version in the dev list to tox>=4 in optional-dependencies . Potentially may also be worth adding version pin when tox is installed in Actions workflows though not sure if it is likely that a previous version will get installed on Actions runners in practice.

Add setuptools-scm to packaging page

Start community page

Start Licensing page

Which licence, how to avoid unnecessary bloat and best practices on maintaining and licensing a library might be part of this page.

Some pointers:

• https://github.com/belle2/basf2
• https://www.gnu.org/licenses/gpl-faq.html#NoticeInSourceFile
• https://spdx.dev/ids/
• https://manifest.fm/10
• History log of licences as pytorch does: https://github.com/pytorch/pytorch/blob/master/LICENSE
• Have LICENSES_bundled.txt as https://github.com/numpy/numpy/blob/main/LICENSES_bundled.txt
• Have General licence and licence each file adding " Copyright (c) MONAI Consortium" as https://github.com/Project-MONAI/MONAI
• Have one general license as Scikit-Surgery libraries which are under BSD

Link each tool to the ARC projects that use it (or vice versa?)

Mention RPHub slack workspace in community.md

Members of UCL could be encouraged to create a channel on RPHub for their specific community

Start Planning page

The Python Template might come with a doc/requirements.rst file which is a good place to write up initial thoughts on what the library will or won't do. There are various planning elements and good practices for users to consider. Here some pointed that can be added in the planning page:

• https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002802
• https://github.com/lorensen/VTKExamples/blob/master/src/VTKBook/02Chapter2.md#22-goals-of-good-software-design
• https://en.wikipedia.org/wiki/Test-driven_development
• http://andtr.com/the-importance-of-software-requirements
• https://github.com/SciKit-Surgery/scikit-surgerytutorial02/blob/master/doc/04_Planning.rst

Add CI to test package template

Start benchmarking page

Jupyter notebook formatting

Wonder whether we could make them better in git using this https://github.com/nbQA-dev/nbQA in pre-commit

Add bindings section

Add documentation

A really lazy way: paddyroddy/python-template#9 (+ paddyroddy/python-template#10 + paddyroddy/python-template#11) as I'm an idiot) which allows you to just use github-pages deploying from a gh-pages branch with these settings

A better example that I just did looks like: https://astro-informatics.github.io/sleplet/

Add build, and GraySkull to packaging.md

• build is @dstansby 's recommended so 🟢 .
• GraySkull.

How to integrate `setuptools-scm`?

Ideally, we'd have something like the following included in the pyproject.toml.

[build-system]
requires = [
    "setuptools-scm",
]

[project]
dynamic = [
    "version",
]

[tool.setuptools_scm]

This has the advantage that all files tracked by git are included in the build - avoiding the need for MANIFEST.in etc. Also, we can do dynamic versioning.

If we have this, I run into issues as the templated generated code isn't a git repo. How can we get this to work nicely?

Migrate python-tooling to here in this repo's `docs/`

https://github.com/UCL-ARC/python-tooling

Should be migrated here.

• Code
• GH pages deployment
• Migrate issues

Add line_profiler to benchmarking page

https://github.com/pyutils/line_profiler