amrox / asdf-pyapp

A generic Python Application plugin the asdf version manager.

asdf-pyapp

A generic Python Application plugin the asdf version manager.

What is a "Python Application"?

For purposes of this plugin, a Python Application is program that happens to be written in Python, but otherwise behaves like a regular command-line tool. The term "Python Application" comes from pipx.

Examples of Python Applications are awscli and conan. See below for more compatible applications.

Dependencies

• python/python3 >= 3.6 with pip and venv
• OR asdf-python installed

Install

Plugin:

asdf plugin add <python app> https://github.com/amrox/asdf-pyapp.git
# for example
asdf plugin add cowsay https://github.com/amrox/asdf-pyapp.git

Example using cowsay:

# Show all installable versions
asdf list-all cowsay

# Install specific version
asdf install cowsay latest

# Set a version globally (on your ~/.tool-versions file)
asdf global cowsay latest

# Now cowsay commands are available
cowsay "Hi!"

Compatible Python Applications

This is a non-exhaustive list of Python Applications that work with this plugin.

App	Command to add Plugin	Notes
ansible	ASDF_PYAPP_INCLUDE_DEPS=1 asdf plugin add ansible https://github.com/amrox/asdf-pyapp.git	(info)
awscli	asdf plugin add awscli https://github.com/amrox/asdf-pyapp.git
awsebcli	asdf plugin add awsebcli https://github.com/amrox/asdf-pyapp.git
aws-sam-cli	asdf plugin add aws-sam-cli https://github.com/amrox/asdf-pyapp.git
aws-ssm-tools	asdf plugin add aws-ssm-tools https://github.com/amrox/asdf-pyapp.git
black	asdf plugin add black https://github.com/amrox/asdf-pyapp.git
bpython	asdf plugin add bpython https://github.com/amrox/asdf-pyapp.git
conan	asdf plugin add conan https://github.com/amrox/asdf-pyapp.git
cowsay	asdf plugin add cowsay https://github.com/amrox/asdf-pyapp.git
dbt	ASDF_PYAPP_INCLUDE_DEPS=1 asdf plugin add dbt https://github.com/amrox/asdf-pyapp.git	(info)
doit	asdf plugin add doit https://github.com/amrox/asdf-pyapp.git
flake8	asdf plugin add flake8 https://github.com/amrox/asdf-pyapp.git
hy	asdf plugin add hy https://github.com/amrox/asdf-pyapp.git	The latest stable version of hy (0.20.0 atm) requires python > 3.8
meson	asdf plugin add meson https://github.com/amrox/asdf-pyapp.git
mypy	asdf plugin add mypy https://github.com/amrox/asdf-pyapp.git
pipenv	asdf plugin add pipenv https://github.com/amrox/asdf-pyapp.git
pre-commit	asdf plugin add pre-commit https://github.com/amrox/asdf-pyapp.git
salt	asdf plugin add salt https://github.com/amrox/asdf-pyapp.git
sphinx	asdf plugin add sphinx https://github.com/amrox/asdf-pyapp.git
yawsso	asdf plugin add sphinx https://github.com/amrox/asdf-pyapp.git
youtube-dl	asdf plugin add sphinx https://github.com/amrox/asdf-pyapp.git

Check asdf readme for more instructions on how to install & manage versions.

How it Works

asdf-pyapp is a lot more complex than most asdf plugins since it's designed to work with generic Python Applications, and challenges that come with Python itself.

asdf-pyapp uses the same technique as asdf-hashicorp to use a single plugin for multiple tools.

When installing a tool, asdf-pyapp creates a fresh virtual environment and pip-installs the package matching the plugin name. Then it uses pipx under the hood to extract the entrypoints for the package exposes them to asdf.

Python Resolution

To run Python Applications, you need Python:

1. If ASDF_PYAPP_DEFAULT_PYTHON_PATH is set - use it
2. Else if the asdf-python plugin is installed - use the global python3**.
3. Finally, just use python3 in our path.

** We use the global python3 to avoid picking up local python versions inside projects, which would result in inconsistent tool installations. If you want to install a tool with a specific version of Python see the following section on asdf-python Integration.

asdf-python Integration (Experimental)

Here we color outside the lines a bit :)

asdf-python supports installing a Python App with a specific Python version using a special syntax. This feature requires the asdf-python plugin to be installed.

The general form is:

asdf <app> install <app-version>@<python-version>

For example, to install cowsay 3.0 with Python 3.9.1:

asdf cowsay install [email protected]

Python Apps with different python versions and python itself can all happily co-exist in the same project. For example, take this .tool-versions:

python 3.8.5
awscli 1.19.93
cowsay [email protected]
conan [email protected]

• awscli will be installed with the global Python (see Python Resolution), in an isolated virtual environment
• Python 3.9.1 will be installed, and then cowsay will be installed using that Python (in a venv)
• conan will be installed with Python 3.8.5, but isolated from the project's Python, which is also 3.8.5.

Configuration

Environment Variables

• ASDF_PYAPP_INCLUDE_DEPS - when set to 1, this plugin will consider the executables of the dependencies of the installed package as well. For example, when installing ansible, the ansible command actually comes from its depency, ansible-core. This is the same as Pipx's --include-deps flag.
• ASDF_PYAPP_DEFAULT_PYTHON_PATH - Path to a python/python3 binary this plugin should use. Default is unset. See Python Resolution section for more details.
• ASDF_PYAPP_VENV_COPY_MODE:
  • 0: (default) Add --copies flag to venvs created with a specific Python version. Symlinks otherwise.
  • 1: Prefer --copies whenever possible (--copies does not work with /usr/bin/python3 on macOS).
• ASDF_PYAPP_DEBUG - Set to 1 for additional logging

Background and Inspiration

asdf-pyapp was inspired by asdf-hashicorp and pipx - which is also used under the hood. Big thanks to the creators, contributors, and maintainers of both these projects.

Contributing

Contributions of any kind welcome! See the contributing guide.

Thanks goes to these contributors!

License

See LICENSE © Andy Mroczkowski

asdf-pyapp's Issues

Can I switch between ansible-base and ansible-core?

Is your feature request related to a problem? Please describe.
Can I switch between ansible-base and ansible-core?
I use both ansible 2.10.x and 2.11.x or higher.
I installed ansible-core and ansible-base, asdf uses the first installed.
When switching between ansible-base and ansible-core, one of them had to be removed

Describe the solution you'd like
Switch between ansible-base and ansible-core without remove the ansible-core or ansible-base.

Describe alternatives you've considered
Nothing.

Additional context
The script messages when I installed both.

okamoto_shinichi[~] asdf list 2>&1 | grep -A 1 ansible
ansible-core
  2.12.4
okamoto_shinichi[~] ansible --version
ansible [core 2.12.4]
  config file = None
  configured module search path = ['/Users/okamoto_shinichi/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/okamoto_shinichi/.asdf/installs/ansible-core/2.12.4/venv/lib/python3.10/site-package
  ansible collection location = /Users/okamoto_shinichi/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/okamoto_shinichi/.asdf/installs/ansible-core/2.12.4/bin/ansible
  python version = 3.10.1 (main, Dec 23 2021, 17:57:48) [Clang 13.0.0 (clang-1300.0.27.3)]
  jinja version = 3.1.1
  libyaml = True
okamoto_shinichi[~] asdf plugin add ansible-base https://github.com/amrox/asdf-pyapp.git
okamoto_shinichi[~] asdf install ansible-base 2.10.17

okamoto_shinichi[~] asdf global ansible-base 2.10.17
okamoto_shinichi[~] asdf list 2>&1 | grep -A 1 ansible
ansible-base
  2.10.17
ansible-core
  2.12.4
okamoto_shinichi[~] asdf reshim
okamoto_shinichi[~] ansible --version
ansible [core 2.12.4]
  config file = None
  configured module search path = ['/Users/okamoto_shinichi/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/okamoto_shinichi/.asdf/installs/ansible-core/2.12.4/venv/lib/python3.10/site-package
  ansible collection location = /Users/okamoto_shinichi/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/okamoto_shinichi/.asdf/installs/ansible-core/2.12.4/bin/ansible
  python version = 3.10.1 (main, Dec 23 2021, 17:57:48) [Clang 13.0.0 (clang-1300.0.27.3)]
  jinja version = 3.1.1
  libyaml = True

Installation of semgrep - error

Is your feature request related to a problem? Please describe.

I stumbled onto this because I was trying to build an asdf-semgrep but couldn't figure out why it was crashing. It seems this repository does a lot of heavy lifting for python applications and it's very much appreciated.

I ran into an error when running asdf plugin add semgrep https://github.com/amrox/asdf-pyapp.git and then asdf latest semgrep. Semgrep can be found in Pypi.

When running asdf latest semgrep or asdf list-all semgrep. This is the error:

Plugin semgrep's list-all callback script failed with output:
asdf-pyapp: [ERROR] Unable to parse versions for 'semgrep'

I'm running Python 3.11.8 and pip 24.0.

Describe the solution you'd like

semgrep to be installed and running

Describe alternatives you've considered

I'll probably look into the parser to see how versions are being parsed.

Figure out how to get lint action working

Describe the bug

Lint jobs fail with the error:

/home/runner/.asdf/plugins/nodejs/bin/import-release-team-keyring: No such file or directory

I don't understand this, it came from the asdf-plugin-template

Steps to reproduce

Run CI Lint

dbt support

Is your feature request related to a problem? Please describe.

The linking doesn't seem to work with DBT, and as dbt isn't in the list of Compatible Python Applications I'm assuming it's not "officially" supported, and thus a feature and not a bug.

Describe the solution you'd like

the dbt command available as a shim

asdf plugin add dbt-snowflake https://github.com/amrox/asdf-pyapp.git
ls ~/.asdf/installs/dbt-snowflake/1.0.0rc1/venv/bin lists dbt as a binary there
ls ~/.asdf/installs/dbt-snowflake/1.0.0rc1/bin but is not linked here, thus no shim

perhaps because the egg/wheel is named dbt-snowflake and not just dbt ? (not sure haven't verified in the code as of yet, and not sure if this is something you would want to officially support)

Describe alternatives you've considered

I could set up another asdf plugin exclusively for dbt.

Additional context

list-all ansible-base contains invalid output

Describe the bug

Running asdf list-all ansible-base returns version numbers surrounded by invalid output.

The output I got was

➜ asdf list-all ansible-base
-atplotlib
-atplotlib
(/home/me/.local/lib/python3.9/site-packages)
(/home/me/.local/lib/python3.9/site-packages
0.0.1a1
0.0.1a2
2.10.0
2.10.0b1
2.10.0rc1
2.10.0rc2
2.10.0rc3
2.10.0rc4
2.10.1
2.10.1rc1
2.10.1rc2
2.10.1rc3
2.10.2
2.10.2rc1
2.10.3
2.10.3rc1
2.10.4
2.10.4rc1
2.10.5
2.10.5rc1
2.10.6
2.10.6rc1
2.10.7
2.10.7rc1
2.10.8
2.10.8rc1
2.10.9
2.10.9rc1
2.10.10
2.10.10rc1
2.10.11
2.10.11rc1
2.10.12
2.10.12rc1
2.10.13
2.10.13rc1
2.10.14
2.10.14rc1
2.10.15
2.10.15rc1
2.10.16)
2.10.16rc1
ERROR:
Ignoring
Ignoring
No
WARNING:
WARNING:
ansible-base==
distribution
distribution
distribution
for
found
invalid
invalid
matching

Steps to reproduce

asdf plugin add ansible-base https://github.com/amrox/asdf-pyapp.git
asdf list-all ansible-base

Expected behavior

I expected to get a valid list of installable ansible versions.

Edit
asdf install ansible-base latest is also broken as it tries to install ansible-base==2.10.16)

Expected executables are not available when installing Ansible

Description

When installing ansible through this plugin, only the executable ansible-community is available, all other ansible executables are missing.

Sorry if I am misunderstanding or doing something wrong, but the result I am getting is neither expected nor useful.

Steps to reproduce

ASDF_PYAPP_INCLUDE_DEPS=1 asdf plugin add ansible https://github.com/amrox/asdf-pyapp.git
ASDF_PYAPP_INCLUDE_DEPS=1 asdf install ansible 8.4.0
asdf local ansible 8.4.0
ansible-playbook --version

zsh: command not found: ansible-playbook

I have tried with and without ASDF_PYAPP_INCLUDE_DEPS=1, and it's the same result in both cases.

I have tried using both asdf and rtx, and it's the same result in both cases.

I have tried both version 8.3.0 and 8.4.0, and it's the same result in both cases.

Expected behavior

All regular ansible executables should be available, for example: ansible, ansible-playbook, ansible-galaxy, ansible-vault.
That is what happens when installing ansible through pip.

python -m venv venv
. ./venv/bin/activate
pip install ansible==8.4.0
ansible-playbook --version

ansible-playbook [core 2.15.4]

asdf-pyapp fails when pip version string not in major.minor.patch format

Describe the bug

asdf-pyapp fails to install an app with pip version 21.0

Steps to reproduce

$ python3 -m pip --version
pip 21.0 from /usr/lib/python3.10/site-packages/pip (python 3.10)

$ asdf install yamllint
asdf-pyapp: [ERROR] Unable to parse pip version

With shell trace:

++ local 'regex=pip (.+) from.*\(python (.+)\)'
++ [[ pip 21.0 from /usr/lib/python3.10/site-packages/pip (python 3.10) =~ pip (.+) from.*\(python (.+)\) ]]
++ echo -n 21.0
+ pip_version=21.0
+ [[ 21.0 =~ ^([0-9]+)\.([0-9]+)\. ]]
+ fail 'Unable to parse pip version'
+ echo -e 'asdf-pyapp: [ERROR] Unable to parse pip version'
asdf-pyapp: [ERROR] Unable to parse pip version

Expected behavior

asdf-pyapp working with PIP version specified in major.minor format.

Equivalent of `pipx inject` for asdf-pyapp?

Pipx supports the very useful "inject" feature, which allows you to easily pip install additional packages into the environment of a particular Python application, e.g.:

pipx install jupyterlab
pipx inject jupyterlab ipywidgets ipympl

Is this supported at all in the ASDF-PyApp plugin? If not, can it be? Is it even possible to support this, within the ASDF framework?

`pipx` installed via `pyapp` complains `pip` isn't found

Describe the bug

pipx complains no pip is available when installing poetry, or any other apps.

Steps to reproduce

$ asdf install pipx latest                                    
Requirement already satisfied: setuptools in ./.asdf/installs/pipx/1.2.0/venv/lib/python3.11/site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-67.7.2-py3-none-any.whl (1.1 MB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
Successfully installed setuptools-67.7.2
Requirement already satisfied: pip in ./.asdf/installs/pipx/1.2.0/venv/lib/python3.11/site-packages (22.3.1)
Collecting pip
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Collecting wheel
  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, pip
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-23.1.2 wheel-0.40.0
Collecting pipx==1.2.0
  Using cached pipx-1.2.0-py3-none-any.whl (57 kB)
Collecting argcomplete>=1.9.4 (from pipx==1.2.0)
  Downloading argcomplete-3.0.8-py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.0/40.0 kB 1.7 MB/s eta 0:00:00
Collecting packaging>=20.0 (from pipx==1.2.0)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting userpath>=1.6.0 (from pipx==1.2.0)
  Using cached userpath-1.8.0-py3-none-any.whl (9.0 kB)
Collecting click (from userpath>=1.6.0->pipx==1.2.0)
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Installing collected packages: packaging, click, argcomplete, userpath, pipx
Successfully installed argcomplete-3.0.8 click-8.1.3 packaging-23.1 pipx-1.2.0 userpath-1.8.0
Collecting pipx==0.16.5
  Downloading pipx-0.16.5-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.2/54.2 kB 2.8 MB/s eta 0:00:00
Collecting userpath>=1.6.0
  Using cached userpath-1.8.0-py3-none-any.whl (9.0 kB)
Collecting argcomplete<2.0,>=1.9.4
  Downloading argcomplete-1.12.3-py2.py3-none-any.whl (38 kB)
Collecting packaging>=20.0
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting click
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Installing collected packages: argcomplete, packaging, click, userpath, pipx
Successfully installed argcomplete-1.12.3 click-8.1.3 packaging-23.1 pipx-0.16.5 userpath-1.8.0

$ pipx --version
1.2.0

$ pip --version      
pip 23.1.2 from /home/us-er/.asdf/installs/python/3.11.3/lib/python3.11/site-packages/pip (python 3.11)

$ pipx install poetry
Fatal error from pip prevented installation. Full pip output in file:
    /home/us-er/.local/pipx/logs/cmd_2023-05-04_15.15.45_pip_errors.log

Error installing poetry

$ cat /home/us-er/.local/pipx/logs/cmd_2023-05-04_15.15.45_pip_errors.log
PIP STDOUT
----------

PIP STDERR
----------
/home/us-er/.local/pipx/venvs/poetry/bin/python: No module named pip

Expected behavior

pipx works properly.

---

I'm aware that I can use pyapp directly, but I've reasons not to.

[Information] Working with aws-ssm-tools and yawsso

Is your feature request related to a problem? Please describe.
Not really a problem.
Your plugin works like charm with two other packages:

• aws-ssm-tools
• yawsso

Describe the solution you'd like
Probably nothing ;-)
Except maybe these tools are listed in README.md (but I know this list is non-exhaustive).
Or add them in asdf-plugins (I did not look exactly how it happens).

Probably I can make a PR for this two suggestions, let me know!

edit: typo in link

No versions listed after `2.10` But upstream is at `2.14`

Describe the bug

Listing latest releases with the plugin ends at 2.10.17, but the latest release (found in pip) is 2.14.4 (as of this writing)

Steps to reproduce

asdf add plugin ansible-base
asdf list all ansible-base

Expected behavior

Latest should keep track with pip releases (to my understanding)

Screenshots

asdf list all ansible-base
0.0.1a1
0.0.1a2
2.10.0
2.10.0b1
2.10.0rc1
2.10.0rc2
2.10.0rc3
2.10.0rc4
2.10.1
2.10.1rc1
2.10.1rc2
2.10.1rc3
2.10.2
2.10.2rc1
2.10.3
2.10.3rc1
2.10.4
2.10.4rc1
2.10.5
2.10.5rc1
2.10.6
2.10.6rc1
2.10.7
2.10.7rc1
2.10.8
2.10.8rc1
2.10.9
2.10.9rc1
2.10.10
2.10.10rc1
2.10.11
2.10.11rc1
2.10.12
2.10.12rc1
2.10.13
2.10.13rc1
2.10.14
2.10.14rc1
2.10.15
2.10.15rc1
2.10.16
2.10.16rc1
2.10.17
2.10.17rc1

Additional context

Just a thanks 🧇 for making this!

migrate/integrate integration test suite into Github actions

Ansible not compatible with its ecosystem

Describe the bug

When using ansible from asdf, ansible-lint won't work due to the following error:

❯ ansible-lint playbook.yml
Ansible CLI (2.14.6) and python module (2.15.2) versions do not match. This indicates a broken execution environment.

Steps to reproduce

1. Install ansible using asdf as reported in the README
2. configure asdf and direnv in a project for using older versions of ansible (e.g. 7.6.0)
3. In that project, try to run ansible-lint

Expected behavior
I expect ansible-lint to run without reporting the error of a broken execution environment.

ansible-base seems not working

Describe the bug

ansible-base seems not working

Steps to reproduce

❯ asdf plugin add ansible-base https://github.com/amrox/asdf-pyapp.git

❯ asdf current
ansible-base ______ No version is set. Run "asdf <global|shell|local> ansible-base "
awscli 2.4.0 /root/.tool-versions
packer 1.7.8 /root/.tool-versions
terraform 1.0.11 /root/.tool-versions

❯ asdf list all ansible-base
Plugin ansible-base's list-all callback script failed with output:

❯ asdf install ansible-base latest
Plugin ansible-base's list-all callback script failed with output:

ansible-base is already installed

`black latest` is not working

Describe the bug

Steps to reproduce

• Install this plugin for black
• note that latest version of black appears to be whitespace

❯ asdf install black latest
black  is already installed