GithubHelp home page GithubHelp logo

fedora-python-tox's Introduction

Fedora + Pythons + Tox in docker image

CI/CD status

This repository contains source code for Docker image designed to help you run your Tox tests with all available Python versions in Docker container.

Docker image is hosted on Docker Hub and build automatically after every push to master branch.

Fedora base image version

We usually add the newest Pythons to all supported Fedora releases. We also have to remove the oldest ones from time to time. These are the reasons why we move this container image from Fedora N to N+1 when N goes EOL.

Installed packages

On top of stable Fedora, this Docker image contains:

  • Python 2.7, 3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12, 3.13
  • Pypy2, Pypy3.9 and Pypy3.10
  • python3-pip
  • gcc, gcc-c++, git-core and make
  • tox
  • virtualenv

Docker and Podman

The image works very well with both Docker and Podman so you can use containers management tool you prefer.

Example usage with local source code

If you want to use this Docker image as environmet for your test, use this command:

cd example_project
docker run --rm -it -v $PWD:/src -w /src fedorapython/fedora-python-tox

py27 create: /src/.tox/py27
py27 installdeps: pytest < 3.3
py27 installed: py==1.5.3,pytest==3.2.5
py27 runtests: PYTHONHASHSEED='85819776'
py27 runtests: commands[0] | pytest
==== test session starts ====
platform linux2 -- Python 2.7.18, pytest-3.2.5, py-1.5.3, pluggy-0.4.0
rootdir: /src, inifile:
collected 10 items

test_fac.py ..........

==== 10 passed in 0.02 seconds ====

py35 create: /src/.tox/py35
py35 installdeps: pytest < 3.3
py35 installed: py==1.5.3,pytest==3.2.5
py35 runtests: PYTHONHASHSEED='1223497559'
py35 runtests: commands[0] | pytest
==== test session starts ====
platform linux -- Python 3.5.10, pytest-3.2.5, py-1.5.3, pluggy-0.4.0
rootdir: /src, inifile:
collected 10 items

test_fac.py ..........

==== 10 passed in 0.04 seconds ====
... etc ...

This command will download an image from Docker hub for you

Unable to find image 'fedorapython/fedora-python-tox:latest' locally
Trying to pull repository docker.io/fedorapython/fedora-python-tox ...
sha256:d4b51931dd86a33552b6a08f39575db336b606f4ccebbc48f390d7f728367a6f: Pulling from docker.io/fedorapython/fedora-python-tox

Command and argument description:

  • docker run - run docker image
  • --rm - automatically remove the container when it exits
  • -i - keep STDIN open even if not attached
  • -t - allocate a pseudo-TTY
  • -v $PWD:/src - mount your current working directory to /src inside container
  • -w - use /src (mountpoint) as working directory inside container
  • fedorapython/fedora-python-tox - name of Docker image

Tox is run automatically with this container. How cool!

You also can limit Tox tests by passing environment variable to Docker. For example:

cd example_project
docker run --rm -it -v $PWD:/src -w /src -e TOXENV=pypy3 fedorapython/fedora-python-tox
pypy3 create: /src/.tox/pypy3
pypy3 installdeps: pytest
pypy3 installed: cffi==1.10.1,greenlet==0.4.12,py==1.4.34,pytest==3.2.2,readline==6.2.4.1
pypy3 runtests: PYTHONHASHSEED='68938415'
pypy3 runtests: commands[0] | pytest
==== test session starts ====
platform linux -- Python 3.5.3[pypy-5.8.0-beta], pytest-3.2.2, py-1.4.34, pluggy-0.4.0
rootdir: /src, inifile:
collected 10 items

test_fac.py ..........

==== passed in 0.24 seconds ====

  pypy3: commands succeeded
  congratulations :)

Parallel run and other tox features

You can adjust the behavior of tox by setting TOX_PARAMS variable with any combination of CLI parameters. It's useful for example for parallel run of your tests.

docker run --rm -it -v $PWD:/src -w /src -e TOX_PARAMS="-p auto" fedorapython/fedora-python-tox
✔ OK py37 in 1.664 seconds
✔ OK py35 in 1.893 seconds
✔ OK py27 in 1.902 seconds
✔ OK py38 in 1.913 seconds
✔ OK py36 in 1.987 seconds
✔ OK pypy in 2.904 seconds
✔ OK pypy3 in 3.688 seconds
__________________________________________ summary ___________________________________________
  py27: commands succeeded
  py35: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
  py38: commands succeeded
  pypy: commands succeeded
  pypy3: commands succeeded
  congratulations :)

Example usage with GIT URL

Instead of mounting local folder with source code into container, you can provide HTTPS URL of your GIT repository and it will be cloned automatically.

For example:

docker run --rm -it -e GIT_URL=https://github.com/fedora-python/pyp2rpm.git fedorapython/fedora-python-tox
Cloning into 'temp_project_dir'...
remote: Counting objects: 3075, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 3075 (delta 1), reused 3 (delta 1), pack-reused 3069
Receiving objects: 100% (3075/3075), 2.97 MiB | 2.57 MiB/s, done.
Resolving deltas: 100% (2188/2188), done.
GLOB sdist-make: /temp_project_dir/setup.py
py27 create: /temp_project_dir/.tox/py27

... etc ...

Installing RPM packages from Fedora

Sometimes, your tests need some system packages (such as C libraries). You can install them via dnf by setting the $DNF_INSTALL environment variable:

docker run --rm -it -v $PWD:/src -w /src -e DNF_INSTALL="libyaml-devel libgit2-devel" fedorapython/fedora-python-tox
...
Installed:
  http-parser-2.9.3-2.fc32.x86_64    libgit2-1.0.1-2.fc32.x86_64
  libgit2-devel-1.0.1-2.fc32.x86_64  libssh2-1.9.0-5.fc32.x86_64
  libyaml-devel-0.2.2-3.fc32.x86_64  openssl-devel-1:1.1.1g-1.fc32.x86_64
  pcre2-devel-10.35-4.fc32.x86_64    pcre2-utf16-10.35-4.fc32.x86_64
  pcre2-utf32-10.35-4.fc32.x86_64    zlib-devel-1.2.11-21.fc32.x86_64

Complete!
py27 create: /src/.tox/py27
...

You can install packages by any RPM Provides, for example:

  • Fedora package names, e.g. libgit2-devel,
  • pkgconfig names, e.g. pkgconfig(libgit2), or
  • commands, e.g. /usr/bin/cowsay.

WORKDIR

If your tox.ini file is not in the root directory of your project, set WORKDIR to the path you want to cd to before the tests are executed.

docker run --rm -it -e TOXENV=py310-minimal -e GIT_URL=https://github.com/trezor/trezor-firmware.git -e WORKDIR=python fedora-python-tox

License

MIT

fedora-python-tox's People

Contributors

abompard avatar encukou avatar frenzymadness avatar hroncok avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

fedora-python-tox's Issues

RFE: Add and support detox

Detox (fedora package python-detox, or python3-detox when built) allow to run tox environments in parallel. Would you accept a PR that adds it?

Jython & pip issue

Tests with Jython fail because Jython doesn't work with the latest pip and therefore cannot install pytest dependency.

Upstream bug: pypa/pip#5121

Python 3.7

Add python37. (I don't have time to do it right now, so opening an issue, but I'll gladly do it next week.)

Jython

Should we add Jython?

Remove Jython from the image?

Jython is basically not supported by tox/pip/pytest etc. and with every new release of those components, a new way it fails appears.

Do we want to keep it in the image and document that it has a limited support and tox (which is the main tool we use here) does not support it, or just remove it?

Add PyPy 3.10

This is a placeholder issue to add PyPy 3.10 to the container once available in Fedora 37.

Progress:

Pypy3 from COPR doesn't work

During build with Python 3.7 I found out that there is something wrong with the newest Pypy3 from COPR.

$ docker run --rm -it -v $PWD:/src -w /src frenzymadness/fedora-python-tox
# Pypy is ok
pypy installed: cffi==1.10.1,greenlet==0.4.12,py==1.5.2,pytest==3.2.5,readline==6.2.4.1
pypy runtests: PYTHONHASHSEED='618530766'
pypy runtests: commands[0] | pytest
========================================== test session starts ==========================================
platform linux2 -- Python 2.7.13[pypy-5.8.0-final], pytest-3.2.5, py-1.5.2, pluggy-0.4.0
rootdir: /src, inifile:
collected 10 items                                                                                       

test_fac.py ..........

======================================= 10 passed in 0.05 seconds =======================================
# But Pypy3 not
Traceback (most recent call last):
  File "/usr/bin/tox", line 11, in <module>
    load_entry_point('tox==2.7.0', 'console_scripts', 'tox')()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 39, in main
    retcode = Session(config).runcommand()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 392, in runcommand
    return self.subcommand_test()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 543, in subcommand_test
    if self.setupenv(venv):
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 461, in setupenv
    envlog.set_python_info(commandpath)
  File "/usr/lib/python3.6/site-packages/tox/result.py", line 49, in set_python_info
    "import sys; "
  File "/usr/lib/python3.6/site-packages/py/_path/local.py", line 714, in sysexec
    stdout, stderr,)
py.process.cmdexec.Error: ExecutionFailed: 1  /src/.tox/pypy3/bin/python
debug: WARNING: Library path not found, using compiled-in sys.path, with
debug: WARNING: sys.prefix = '/builddir/build/BUILD/pypy2-v5.8.0-src'
debug: WARNING: Make sure the pypy binary is kept inside its tree of files.
debug: WARNING: It is ok to create a symlink to it from somewhere else.
debug: OperationError:
debug:  operror-type: ImportError
debug:  operror-value: No module named os

Because I don't have time to inspect it now, I decided to remove COPR repo from Dockerfile and use standard Pypy3 from Fedora repositories.. frenzymadness@ff004e7

@hroncok IMHO you know Pypy better than me.

Build the image cronly

When there are no changes in here, the Docker image can get a bit outdated. Would you consider building it weekly by cron?

Problem on ppc64le

The latest build by Travis cron failed on ppc64le:

$ docker run --rm -e TOXENV -e GIT_URL=https://github.com/[secure]/python-tox-example.git fedorapython/fedora-python-tox:$TRAVIS_CPU_ARCH
Cloning into 'temp_project_dir'...
fatal: unable to access 'https://github.com/[secure]/python-tox-example.git/': OpenSSL SSL_connect: Connection reset by peer in connection to github.com:443 
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
The command "docker run --rm -e TOXENV -e GIT_URL=https://github.com/[secure]/python-tox-example.git fedorapython/fedora-python-tox:$TRAVIS_CPU_ARCH" exited with 1.

I thought it might be some problem in OpenSSL on ppc64le but I'm not able to reproduce the issue. I've opened an issue on Travis CI support so we'll see what they'll come with.

[RFE] Auto-clone project

It would be nice to pass somehow git URL into the docker so that it would automatically clone the requested repository and run tox command on top of it using the supplied tox.ini from fedora-python-tox.

It could be done using env. variables, extending docker entry point with additional/custom arguments taken from docker run, some special config file (with multiple git URLs) etc.

Backward compatibility for Github Action

We have discussed this briefly with @encukou today. When we switch to F35, we might break someone's CI because F35 does not contain Python 3.5.

My idea is to build container images and create manifests with the fedora version in their name and release a new version of Github action every time we switch to a newer Fedora release.

We can build all the container images with the original tags like amd64 and amd64_fedora34. The same applies to the latest manifest where we can use latest and latest_fedora34. We can then change the github action to use latest_fedora34 explicitly. Then, when we switch to Fedora 35, the tags and manifests using _fedora34 suffix will stay the same, and therefore the github action will stay stable and we can release a new version of the github action using _fedora35. Also, those who use the images directly can choose between a stable and possibly obsolete version of the images or rolling tags.

Does this make sense?

tox 3 vs tox 4

Tox 4.0.0 was released. We need to decide what to do about that. Since this docker image uses stable Fedora, it will contain tox3 for a while. Should we document that and let it be? Should we package a conflicting version of tox4 into Fedora? etc.

Travis build is broken, image is outdated

The latest successful Travis build happened 3 months ago (with Python 3.10.0b1) and since June 15th, 2021, the building on travis-ci.org is ceased.

As a result, the Pythons are outdated:

My machine:

$ rpm -qa | grep -F python3. | sort
python3.10-3.10.0~rc1-1.fc33.x86_64
python3.5-3.5.10-5.fc33.x86_64
python3.6-3.6.14-1.fc33.x86_64
python3.7-3.7.11-1.fc33.x86_64
python3.8-3.8.11-1.fc33.x86_64
$ rpm -q python3
python3-3.9.6-2.fc33.x86_64

This image:

[root@9808fb7fd92c /]# rpm -qa | grep -F python3. | sort
python3.10-3.10.0~b1-1.fc33.x86_64
python3.5-3.5.10-5.fc33.x86_64
python3.6-3.6.13-1.fc33.x86_64
python3.7-3.7.10-1.fc33.x86_64
python3.8-3.8.10-1.fc33.x86_64
[root@9808fb7fd92c /]# rpm -q python3
python3-3.9.5-1.fc33.x86_64

The only up to date Python is Python 3.5, because it is EOL

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.