GithubHelp home page GithubHelp logo

duckinator / bork Goto Github PK

View Code? Open in Web Editor NEW
12.0 4.0 2.0 629 KB

A Python build and release management tool.

Home Page: https://bork.readthedocs.io

License: MIT License

Python 99.45% Dockerfile 0.55%
python pep517 pypi github-releases

bork's Introduction

bork Build Status PyPI Documentation Status

A frontend for building and releasing PEP 517 compliant projects, including support for generating a ZipApp.

Includes a basic task runner, in the form of bork run <task name>. Tasks are defined in your pyproject.toml file.

Bork requires Python 3.10 or newer.

Installation

$ pip3 install bork

Or download the latest zipapp releases

Usage

Example usage information is provided below. Additional documentation can be found at bork.readthedocs.io.

Downloading Existing Builds

To download a release from GitHub:

$ bork download gh:duckinator/emanate # download latest .pyz for Emanate
$ bork download gh:duckinator/emanate --directory bin/ # put files in ./bin
$ bork download gh:ppb/pursuedpybear --files '*.tar.gz' # download latest .tar.gz file

To download a wheel from a PyPi release:

$ bork download pypi:emanate 6.0.0 --files '*.whl'

To download a wheel from a release on PyPi's test instance:

$ bork download testpypi:whaledo 1.0.1 --files '*.whl'

Building and Releasing

Assuming a project is PEP 517 compliant, you can just do:

$ bork clean # Remove anything in build/, dist/, *.egg-info/
$ bork build # Build the project
$ bork release # Release to PyPI

ZipApp Support

If you want to build a ZipApp, add this to your pyproject.toml:

[tool.bork.zipapp]
enabled = true
main = "<entrypoint>"

Where <entrypoint> is of the form "module.submodule:function", and may be equivalent to a console_script entrypoint in setup.cfg.

Uploading To GitHub Releases

If you want to upload assets to GitHub Releases, you can add the following configuration to your pyproject.toml:

[project]
name = "<project name>"

[tool.bork.release]
# If true, release to PyPi; otherwise, don't.
pypi = true
# If true, release to GitHub; otherwise, don't.
github = true # release to GitHub
# GitHub repository, e.g. "duckinator/bork".
github_repository = "<owner>/<repo>"
# List of file globs to include in GitHub Releases.
github_release_globs = ["dist/*.pyz", "dist/*.whl"]
# If true, zipapps are named "<name>.pyz", otherwise "<name>-<version>.pyz".
strip_zipapp_version = true

Aliases (Basic task runner)

Bork includes a very basic task runner, for single-line commands.

As an example, here is what Bork uses:

[tool.bork.aliases]
lint = [
    "pylint bork tests",
    "mypy bork",
]
# Runs all tests.
test = "pytest --verbose"
# Runs fast tests.
test-fast = "pytest --verbose -m 'not slow'"
# Runs slow tests.
test-slow = "pytest --verbose -m slow"
# Build docs
docs = "mkdocs build"

Then you can run bork aliases to get the list of aliases:

~/bork$ bork aliases
lint
test
test-fast
test-slow
~/bork$

And run bork run <alias> to run that alias:

~/bork$ bork run docs
mkdocs build
INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /usr/home/puppy/bork/site
INFO     -  Documentation built in 0.25 seconds
~/bork$

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/duckinator/bork. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

The code for Bork is available under the MIT License.

bork's People

Contributors

astraluma avatar bors[bot] avatar duckinator avatar nbraud avatar pyup-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

astraluma

bork's Issues

Warnings when running `bork run lint`/`bork run test`

Getting some annoying/weird warnings that I've been unable to track down the cause of so far:

/usr/local/lib/python3.8/site-packages/setuptools/distutils_patch.py:25
  /usr/local/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
    warnings.warn(
Full logs
============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /tmp/cirrus-ci-build, configfile: pytest.ini
plugins: mypy-0.6.2, flake8-1.0.6, pylint-0.17.0
collecting ... collected 50 items
--------------------------------------------------------------------------------
Linting files
................
--------------------------------------------------------------------------------
setup.py::/tmp/cirrus-ci-build/setup.py::PYLINT PASSED                   [  1%]
setup.py::FLAKE8 PASSED                                                  [  3%]
setup.py::mypy PASSED                                                    [  5%]
bork/__init__.py::/tmp/cirrus-ci-build/bork/__init__.py::PYLINT PASSED   [  7%]
bork/__init__.py::FLAKE8 PASSED                                          [  9%]
bork/__init__.py::mypy PASSED                                            [ 11%]
bork/api.py::/tmp/cirrus-ci-build/bork/api.py::PYLINT PASSED             [ 13%]
bork/api.py::FLAKE8 PASSED                                               [ 15%]
bork/api.py::mypy PASSED                                                 [ 17%]
bork/asset_manager.py::/tmp/cirrus-ci-build/bork/asset_manager.py::PYLINT PASSED [ 19%]
bork/asset_manager.py::FLAKE8 PASSED                                     [ 21%]
bork/asset_manager.py::mypy PASSED                                       [ 23%]
bork/builder.py::/tmp/cirrus-ci-build/bork/builder.py::PYLINT PASSED     [ 25%]
bork/builder.py::FLAKE8 PASSED                                           [ 27%]
bork/builder.py::mypy PASSED                                             [ 29%]
bork/cli.py::/tmp/cirrus-ci-build/bork/cli.py::PYLINT PASSED             [ 31%]
bork/cli.py::FLAKE8 PASSED                                               [ 33%]
bork/cli.py::mypy PASSED                                                 [ 35%]
bork/filesystem.py::/tmp/cirrus-ci-build/bork/filesystem.py::PYLINT PASSED [ 37%]
bork/filesystem.py::FLAKE8 PASSED                                        [ 39%]
bork/filesystem.py::mypy PASSED                                          [ 41%]
bork/github.py::/tmp/cirrus-ci-build/bork/github.py::PYLINT PASSED       [ 43%]
bork/github.py::FLAKE8 PASSED                                            [ 45%]
bork/github.py::mypy PASSED                                              [ 47%]
bork/github_api.py::/tmp/cirrus-ci-build/bork/github_api.py::PYLINT PASSED [ 49%]
bork/github_api.py::FLAKE8 PASSED                                        [ 50%]
bork/github_api.py::mypy PASSED                                          [ 52%]
bork/log.py::/tmp/cirrus-ci-build/bork/log.py::PYLINT PASSED             [ 54%]
bork/log.py::FLAKE8 PASSED                                               [ 56%]
bork/log.py::mypy PASSED                                                 [ 58%]
bork/pypi.py::/tmp/cirrus-ci-build/bork/pypi.py::PYLINT PASSED           [ 60%]
bork/pypi.py::FLAKE8 PASSED                                              [ 62%]
bork/pypi.py::mypy PASSED                                                [ 64%]
bork/pypi_api.py::/tmp/cirrus-ci-build/bork/pypi_api.py::PYLINT PASSED   [ 66%]
bork/pypi_api.py::FLAKE8 PASSED                                          [ 68%]
bork/pypi_api.py::mypy PASSED                                            [ 70%]
bork/version.py::/tmp/cirrus-ci-build/bork/version.py::PYLINT PASSED     [ 72%]
bork/version.py::FLAKE8 PASSED                                           [ 74%]
bork/version.py::mypy PASSED                                             [ 76%]
tests/helpers.py::/tmp/cirrus-ci-build/tests/helpers.py::PYLINT PASSED   [ 78%]
tests/helpers.py::FLAKE8 PASSED                                          [ 80%]
tests/helpers.py::mypy PASSED                                            [ 82%]
tests/test_cmd_aliases.py::/tmp/cirrus-ci-build/tests/test_cmd_aliases.py::PYLINT PASSED [ 84%]
tests/test_cmd_aliases.py::FLAKE8 PASSED                                 [ 86%]
tests/test_cmd_aliases.py::mypy PASSED                                   [ 88%]
tests/test_cmd_aliases.py::test_cmd_aliases PASSED                       [ 90%]
tests/test_cmd_download.py::/tmp/cirrus-ci-build/tests/test_cmd_download.py::PYLINT PASSED [ 92%]
tests/test_cmd_download.py::FLAKE8 PASSED                                [ 94%]
tests/test_cmd_download.py::mypy PASSED                                  [ 96%]
tests/test_cmd_download.py::test_download PASSED                         [ 98%]
::mypy PASSED                                                            [100%]
=============================== warnings summary ===============================
/usr/local/lib/python3.8/site-packages/setuptools/distutils_patch.py:25
  /usr/local/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
    warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
===================================== mypy =====================================
Success: no issues found in 16 source files
======================== 51 passed, 1 warning in 11.95s ========================
/usr/local/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
  warnings.warn(

Support `python -m bork`

In theory, I think putting this in bork/__main__.py may work:

import . import cli

cli.main()

This is useful for situations where Python is in your PATH, but things installed via pip are not.
I encounter this sometimes in CI — it feels silly to have to adjust the PATH to run a single command, and it's very fragile ime to try to guess the correct path to the executable. Easier to let Python figure it out.

Release automation

I'd like to automate bork's release process.

What this specifically means in this case is: I want a commit to master which increments the value of bork.__version__ to initiate a new release.

Add ability to run test suite

To my knowledge, there's no PEP for testing that's equivalent to PEP 517.

Given this, the simplest approach is probably just to add aliases that just run a shell command.

Below I'll document some approaches, with some pros and cons of each.

All of these examples assume a setup.cfg with a section alog the lines of this:

[options.extras_require]
linting =
    flake8
    pylint

testing =
    bork[linting]
    pytest
    pytest-flake8
    pytest-pylint

[bork.aliases]
test = pytest --pylint --flake8 --verbose

Generic aliases, no special syntax

Run with: bork test

Pros:

  • only one program to work with!

Cons:

  • confusing as to whether a command is built-in or not
  • potential for conflicting with future bork sub-commands

Generic aliases, with a prefix

Run with: bork +test (Possibly different single-character prefix.)

Pros:

  • only one program to work with!
  • disambiguates sub-commands vs aliases

Cons:

  • arguably a bit of a kludge

Generic aliases, subcommand

Run with: bork run test

Pros:

  • only one program to work with!
  • disambiguates sub-commands vs aliases

Cons:

  • longer command to type

Generic aliases, separate program

Run with: bork-run test

Pros:

  • disambiguates sub-commands vs aliases

Cons:

  • longer command to type
  • two programs to work with. :(

Feature Request: Flags related to logging

I'm thinking --log <level> and --logfile <filename>.

WRT --log <level>, you can just pass the value as a string along to logging.getLogger().setLevel('<level>') to set it on the root logger, and it'll propagate to everything else.

WRT --logfile, maybe something like this?: logging.addHandler(logging.FileHandler(filename, mode='w'))

logging.basicConfig(handlers=(...)) may be useful, idk.

Use type annotations

We should probably be using type annotations for things at this point.

Possibly also mypy or similar. (I'm not sure what the state of that kind of stuff is.)

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

container: pip warnings:

https://cirrus-ci.com/task/5187350944808960

bork build
pep517.envbuild ERROR WARNING: You are using pip version 20.2; however, version 20.2.2 is available.
pep517.envbuild ERROR You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
pep517.envbuild ERROR WARNING: You are using pip version 20.2; however, version 20.2.2 is available.
pep517.envbuild ERROR You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
running sdist
running egg_info

Re-build Docker Hub image on Python dependency releases?

It may make sense to re-build the Docker Hub image when Pip is released.

There's a RSS feed for Pip releases: https://pypi.org/rss/project/pip/releases.xml
It also looks like the Python Docker image is re-released whenever Pip is: https://hub.docker.com/layers/python/library/python/3.8-slim/images/sha256-282fc7428f74cf3317d78224349b9215f266fc9cb4197cce58dad775cb565ed3

I'm not sure if I want to automatically rebuild+push the Docker Hub image, but it'd be nice to at least get an alert somehow that would let me manually initiate the process.

Improve how zipapps are built by using bdists

What prompted this: Files with a .py~ suffix are backups (e.g. from vim) and __pycache__ is cached files. Bork probably shouldn't include these in distributions.

The current approach to installing:

  1. make an empty directory
  2. copy the source (./emanate, ./bork, etc)
  3. grab the list of install_requires dependencies from setup.cfg, then run (basically) <python executable> -m pip install --target <directory from step 1> dep1 dep2 ... depN

However, step 2 of the current approach already requires a bdist be built, so why not this?:

  1. ensure bdist is built
  2. grab the list of install_requires dependencies from setup.cfg, and run (basically) <python executable> -m pip install --target <directory from step 1> dep1 dep2 ... depN <bdist path>

Fix `bork download`

Yes, I apparently already broke it.

It seems gh: URLs save a JSON blob instead of the actual file, for some reason.

Investigate better bors/GitHub integration?

Follow-up to #234.

My idea is, basically:

  1. Have a Cirrus CI task named something like "CI_Success_task", which does nothing but relies on all other CI tasks.
  2. Have bors rely on just that task.

This lets us have all the CI-related configuration that's prone to changing over time in one place.

@nbraud: what are your thoughts on this approach?

Ability to list all aliases

The command would be something like bork alias or bork aliases, I guess.

I'm thinking something along the lines of

$ cat pyproject.toml
[build-system]
# Specify the required build system.
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[tool.bork.zipapp]
# bork can not build a functional zipapp of Emanate yet.
enabled = false
main = "emanate.cli:main"

[tool.bork.aliases]
# Runs *only* pylint and flake8. (Not the actual tests.)
lint = "pytest -k 'flake8 or pylint' --pylint --flake8 --verbose"
# Runs tests, pylint, and flake8.
test = "pytest --pylint --flake8 --verbose"
docs = "env PYTHONPATH=./ pdoc3 --html --output-dir ./html --force emanate"
$ bork aliases
lint
test
docs

Not sure if a comma-separated list, space-separated list, or a newline-separated list would be better.

cc @nbraud @astronouth7303

Feature Request: New Project generation

I've lost track myself. Can you add a command to generate template pyproject.toml/setup.{cfg,py} files?

(I include setup.py because pips that don't support setup.py-less packages are still super common)

Make `bork release` as close to atomic as possible

The main guarantee an atomic release provides is that if releasing to any service fails, releases to all services are cancelled.

A release to multiple services can never, as far as I know, be truly atomic. However, you can get far closer than bork currently does.

The basic way that I plan to accomplish this is:

  1. make a draft release on every service.
  2. if any draft-release failed, cancel the release and clean up.
  3. if everything worked, publish all the releases.

This can not be done on PyPi until pypi/warehouse#726 is resolved, so I have an intermediate "improved implementation" before we reach the ideal one.

Current Implementation

At the moment, bork release will just:

  1. Release to PyPi.
  2. Release to GitHub Releases.

Any failures will probably keep it from continuing, but that's not guaranteed and the cleanup is completely manual.

Notably, a fail to GitHub Releases will result in it being released to PyPi.

Improved Implementation

A better workflow, which only works for the combination of PyPi and GitHub:

  1. Make a draft GitHub Release.
  2. Release to PyPi.
  3. If step 2 fails, delete the draft GitHub Release as well.

(Long Term) The Ideal Workflow

Here is the ideal release workflow I would love to have long-term, because it can be scaled to support more services without breaking assumptions:

  1. Upload a draft release to every service being used (PyPi, GitHub Releases).
  2. If any fail, abort the release immediately.
  3. If everything succeeded, mark them all as published.

bork run could color stderr

When running interactively, it's hard to distinguish the output on bork run on stdout and stderr.
Do you think it would be a good idea to automatically colorise stderr (say, in red?) while still letting commands override it (i.e. if a command sets a color, it should be honored).

Looks like there are libraries that produce cross-platform terminal color (like colorama), but I can't find anything that provides a colored, stderr-like object. Though perhaps what I'm looking for is actually a logging lib...

Support projects without setup.cfg

It seems like setup.cfg is used exclusively for the project name. Providing a way to specify it in pyproject.toml might be useful, for folks not using setuptools (or who are doing a quick migration and still use setup.py).

I explicitly do not want to try to process setup.py. It can't be done without running it.

Feature Request: Container

I just did the thing where I have a multi-stage Dockerfile, and one of the stages is "run bork build on my project.

Any chance of a bork container?

Add download command

For projects that have CIs doing their builds, being able to download the files is pretty helpful.

  • From github releases
  • From test.pypi.org
  • From pypi.org (less useful, but for completeness?)

Check pep517 API + expose anything useful

The pep517 library's pep517.build.parser.parse_args(...) accepts more arguments than what we use.

It's probably worth checking which ones can be used, and exposing them via an API when appropriate.

`bork run lint` is broken

$ bork run lint
====================================== test session starts ======================================
platform linux -- Python 3.7.4+, pytest-5.1.3, py-1.8.0, pluggy-0.13.0 -- /home/nicoo/devel/py/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/nicoo
plugins: flake8-1.0.4, pylint-0.14.1
collected 16 items                                                                              
-----------------------------------------------------------------
Linting files
........
-----------------------------------------------------------------

bork/__init__.py FAILED                                                                   [  6%]
bork/__init__.py::FLAKE8 PASSED                                                           [ 12%]
bork/asset_manager.py FAILED                                                              [ 18%]
bork/asset_manager.py::FLAKE8 SKIPPED                                                     [ 25%]
bork/builder.py FAILED                                                                    [ 31%]
bork/builder.py::FLAKE8 SKIPPED                                                           [ 37%]
bork/cli.py FAILED                                                                        [ 43%]
bork/cli.py::FLAKE8 SKIPPED                                                               [ 50%]
bork/filesystem.py FAILED                                                                 [ 56%]
bork/filesystem.py::FLAKE8 SKIPPED                                                        [ 62%]
bork/github.py FAILED                                                                     [ 68%]
bork/github.py::FLAKE8 SKIPPED                                                            [ 75%]
bork/pypi.py FAILED                                                                       [ 81%]
bork/pypi.py::FLAKE8 SKIPPED                                                              [ 87%]
bork/pypi_api.py FAILED                                                                   [ 93%]
bork/pypi_api.py::FLAKE8 SKIPPED                                                          [100%]

=========================================== FAILURES ============================================
____________________________ [pylint] devel/py/bork/bork/__init__.py ____________________________
F:  1, 0: No module named devel/py/bork/bork/__init__.py (fatal)
_________________________ [pylint] devel/py/bork/bork/asset_manager.py __________________________
F:  1, 0: No module named devel/py/bork/bork/asset_manager.py (fatal)
____________________________ [pylint] devel/py/bork/bork/builder.py _____________________________
F:  1, 0: No module named devel/py/bork/bork/builder.py (fatal)
______________________________ [pylint] devel/py/bork/bork/cli.py _______________________________
F:  1, 0: No module named devel/py/bork/bork/cli.py (fatal)
___________________________ [pylint] devel/py/bork/bork/filesystem.py ___________________________
F:  1, 0: No module named devel/py/bork/bork/filesystem.py (fatal)
_____________________________ [pylint] devel/py/bork/bork/github.py _____________________________
F:  1, 0: No module named devel/py/bork/bork/github.py (fatal)
______________________________ [pylint] devel/py/bork/bork/pypi.py ______________________________
F:  1, 0: No module named devel/py/bork/bork/pypi.py (fatal)
____________________________ [pylint] devel/py/bork/bork/pypi_api.py ____________________________
F:  1, 0: No module named devel/py/bork/bork/pypi_api.py (fatal)
============================ 8 failed, 1 passed, 7 skipped in 0.70s =============================

My clone of the bork repo lives at ~/devel/py/bork; not sure why the path ends up there...
This was with the current state of master (i.e. 7428c3c)

Improve how bork.builder.zipapp() gets version information

At the moment, bork.builder.zipapp() assumes bork.builder.dist() is called first exclusively because it uses the generated sdist file to determine the version number.

We should, uh, probably not do that, because it's incredibly silly.

Related: #59

twine is asking for my username even though i configured my pypi credentials

I have my ~/.pypirc set up like this

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
repository=https://upload.pypi.org/legacy/
username=...
password=...

[testpypi]
repository=https://test.pypi.org/legacy/
username=...
password=...

I can run twine upload dist/<file>.tar.gz just fine, but when I try to run twine upload --repository-url https://upload.pypi.org/legacy/ dist/<file>.tar.gz, as bork does, twine asks me for my username.

Make Bork work more consistently across differently environments

I am incredibly tired of having to throw this in .cirrus.yml for projects using Bork:

# This makes Click, a Bork dependency, behave itself.
env:
  LC_ALL: C.UTF-8
  LANG: C.UTF-8

I'm unsure whether I want to switch away from Click, or if it's better to just add a kludge of some sort.

Document bork aliases

I just realized everything related to bork aliases/bork run is undocumented in the README.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.