flyingcircusio / appenv Goto Github PK
View Code? Open in Web Editor NEWSelf-contained bootstrapping/updating of Python applications deployed through shared repositories
License: BSD 3-Clause "New" or "Revised" License
Self-contained bootstrapping/updating of Python applications deployed through shared repositories
License: BSD 3-Clause "New" or "Revised" License
@wosc if you could dig up the issues you had w/ Debian then I'd be happy to investigate a potential generic way of getting environment-specific workarounds available.
Hi,
with the newest version of AppEnv, I get a pkg_resources.RequirementParseError
if the requirement.txt contains a custom index server:
# appenv-python-preference: 3.8,3.7
--extra-index-url https://my.custompypi.de/simple
-e .
When running ./appenv update-lockfile
I get:
Traceback (most recent call last):
File "./appenv", line 515, in <module>
main()
File "./appenv", line 507, in main
appenv.meta()
File "./appenv", line 280, in meta
args.func(args, remaining)
File "./appenv", line 466, in update_lockfile
spec = list(pkg_resources.parse_requirements(line))[0]
File "/Users/sweh/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3094, in parse_requirements
yield Requirement(line)
File "/Users/sweh/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3103, in __init__
raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'--extra-'"
Is there another way to specify a custom index server?
This depends how we perform releases. We could leverage our venv management and create a temporary install that extracts the appenv script from a distribution.
frlan@trabant:~/quellen/myprj$ ./appenv update-lockfile
Update lockfile with with /usr/bin/python3.12.
If you want to use a different version, set it via
`# appenv-python-preference:` in requirements.txt.
Updating lockfile
Creating venv ...
Activating broken distutils/ensurepip stdlib workaround ...
ensurepip
distutils
Traceback (most recent call last):
File "/home/frlan/quellen/myprj/./appenv", line 77, in ensure_venv
import ensurepip # noqa: F401 imported but unused
^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'ensurepip'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/frlan/quellen/myprj/./appenv", line 533, in <module>
main()
File "/home/frlan/quellen/myprj/./appenv", line 525, in main
appenv.meta()
File "/home/frlan/quellen/myprj/./appenv", line 283, in meta
args.func(args, remaining)
File "/home/frlan/quellen/myprj/./appenv", line 442, in update_lockfile
ensure_venv(tmpdir)
File "/home/frlan/quellen/myprj/./appenv", line 102, in ensure_venv
shutil.copytree(
File "/usr/lib/python3.12/shutil.py", line 598, in copytree
with os.scandir(src) as itr:
^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpk63bq9cr/Python-3.12.2/Lib/distutils'
My current working dir looks like
frlan@trabant:~/quellen/myprj$ ls -la
insgesamt 32
drwxr-xr-x 4 frlan frlan 84 23. Mär 20:23 .
drwxr-xr-x 86 frlan frlan 4096 23. Mär 20:10 ..
drwxr-xr-x 3 frlan frlan 24 23. Mär 20:23 .appenv
-rwxr-xr-x 1 frlan frlan 20553 23. Mär 20:23 appenv
lrwxrwxrwx 1 frlan frlan 6 23. Mär 20:23 batou -> appenv
drwxr-xr-x 7 frlan frlan 119 23. Mär 20:11 .git
-rw-r--r-- 1 frlan frlan 6 23. Mär 20:23 requirements.txt
Currently, run()
directly exec
's the setuptools-generated entrypoint script, which in turn finds the correct python
by including the full path to the {env_dir}/bin/python
executable as a shebang line.
Shebang lines are limited to 127 characters, which is very easy to exceed with appenv, since the venv hash already takes up 64.
I propose to specify the python executable in appenv itself, instead of relying on the shebang line, like this:
diff --git a/batou b/batou
index 8f03df7..455def7 100755
--- a/batou
+++ b/batou
@@ -200,7 +200,10 @@ def run(argv, meta_args):
env_dir = _prepare(meta_args)
# Allow called programs to find out where the wrapper lives
os.environ['APPENV_BASEDIR'] = meta_args.base
- os.execv(os.path.join(env_dir, 'bin', meta_args.appname), argv)
+ interpreter = os.path.join(env_dir, 'bin', 'python')
+ entrypoint = os.path.join(env_dir, 'bin', meta_args.appname)
+ argv[0] = interpreter
+ argv.insert(1, entrypoint)
+ os.execv(interpreter, argv)
This works fine with batou, but argv[0]
is not preserved (since at least under Debian we must set argv[0]
to the venv/bin/python, otherwise the venv is not recognized and the spawned python runs with the system site-packages), so programs that are interested in that will have trouble. Not sure if that's something we have to worry about, though.
Pip already automatically checks for hashes, but we don't generate them into the lockfile. Need to check whether we can do that
I needed to put in full path instead of e.g. ./
frlan@trabant:~/quellen/myprj$ curl -sL https://github.com/flyingcircusio/appenv/raw/master/bootstrap | sh
Let's create a new appenv project.
What should the command be named? batou
What is the main dependency as found on PyPI? [batou]
Where should we create this? [/tmp/tmp.yaqHoNo9fu/batou] /home/frlan/quellen/myprj/
Creating appenv setup in /home/frlan/quellen/myprj ...
Done. You can now `cd ../../home/frlan/quellen/myprj` and call `./batou` to bootstrap and run it.
Maybe defaulting to ./ anyway?
I accidentally rsynced an .appenv directory from Mac to Linux and appenv didn't automatically recover from that:
subprocess.CalledProcessError: Command '['./.appenv/unclean/bin/python -m pip install -r requirements.txt --upgrade']' returned non-zero exit status 126.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./batou", line 444, in <module>
main()
File "./batou", line 438, in main
appenv.run(application_name, sys.argv[1:])
File "./batou", line 234, in run
self._prepare()
File "./batou", line 253, in _prepare
cmd('{env_dir}/bin/python -m pip install -r requirements.txt --upgrade'
File "./batou", line 43, in cmd
raise ValueError(e.output.decode("ascii"))
ValueError: /nix/store/9ywr69qi622lrmx5nn88gk8jpmihy0dz-bash-4.4-p23/bin/sh: ./.appenv/unclean/bin/python: cannot execute binary file: Exec format error
The tests of the package should be run on GHA and a separate check running the linter would be nice as the appenv.py
is checked in into customer repositories it should not violate flake8 rules.
Ok. This is going to be painful (for others to watch), but:
If we can't fix the system Python - and I don't want to compile my own - then here's what I'm going to try:
Hopefully any venvs we create will properly inherit this ... I'll try this out.
Currently updates are done by hand, i.e. using curl to get a new version.
Further it would be nice to have at least a notification about a new version when running update-lockfile
.
(Maybe update appenv here as well?)
Looking at the confusion between batou and appenv I suggest we implement a new scheme where appenv becomes its own script and the actual (managed) utilities are symlinks (so that appenv determines which script to run by looking at the argv[0]) by which it was called. This removes the need for a "meta" argument parser:
ctheune@fourteen-3 /tmp/asdf $ ls -lah
total 0
drwxr-xr-x 4 ctheune wheel 128B Feb 23 06:54 .
drwxrwxrwt 494 root wheel 15K Feb 23 06:54 ..
-rw-r--r-- 1 ctheune wheel 0B Feb 23 06:54 appenv
lrwxr-xr-x 1 ctheune wheel 6B Feb 23 06:54 batou -> appenv
This also means we can easily add more symlinks, think harder about updating appenv itself and maybe providing it as a real, versioned package instead of just a curl'ed file.
Line 223 in 5ff385f
I get this error but also have python, als well as python3 installed. With which python
I get the path /usr/local/bin/python
in the appenv script current_python
variable has the value /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9
which is also a working python.
Even though I cannot run the script and get the error Could not find the minimal preferred Python version.
F-strings are faster and more readable. They got introduced in Python 3.6.
By default we choose /usr/bin/env python3 and suggest users specify a custom specific version. If you'd like to use "whatever is better than X" you could say "3.5 is fine, but use 3.6 if available" so it works even if 3.5 is not available...
In downstream packages that use appenv (like... batou :) it would be helpful to know which appenv version they bundled. For example, in the version bundled with batou-2.0b9, the appenv-update-lockfile
command does not work (it exits with "unrecognized arguments: -u").
I've solved my immediate issue by downloading https://github.com/flyingcircusio/appenv/blob/dc208ac9df334962b013001c60cd93fb88949c98/src/appenv.py and naming it batou
, but I think this points to an issue how to manage bundling and versions and somesuch.
So, here's a weird one: a) The default values for python -m venv
and import venv; venv.create()
are not the same. The former uses symlinks=True, but the latter, which is used by batou/appenv, has symlinks=False). b) At least with my brew install python
, the variant with symlinks=False
has recently(?) stopped working:
$ python3 -m venv venv; venv/bin/python --version; rm -rf venv
Python 3.11.5
$ python3 -c 'import venv; venv.create("venv")'; venv/bin/python --version; rm -rf venv
dyld: Library not loaded: @loader_path/../../../../Python.framework/Versions/3.11/Python
Referenced from: /Users/schnerring/Downloads/venv/bin/python
Reason: image not found
fish: Job 1, 'venv/bin/python --version' terminated by signal SIGABRT (Abort)
$ python3 -c 'import venv; venv.create("venv", symlinks=True)'; venv/bin/python --version; rm -rf venv
Python 3.11.5
I don't understand neither (a) nor (b), and the closest related bug I could find was python/cpython#82886 (but that talks about Apple's Python, not Homebrew), so my proposal would be this:
diff --git a/batou b/batou
index e509e153..691d9cca 100755
--- a/batou
+++ b/batou
@@ -69,7 +69,7 @@ def ensure_venv(target):
version = sys.version.split()[0]
python_maj_min = ".".join(str(x) for x in sys.version_info[:2])
print("Creating venv ...")
- venv.create(target, with_pip=False)
+ venv.create(target, with_pip=False, symlinks=True)
So,
theoretically we could speed things up even further if we include more data in the hash and then could just copy over build results from other locations/machines similar to nix. Obviously this isn't going to be perfect, but maybe good enough to be helpful.
We'd need at least:
I was wondering if one could replace zc.buildout
environments with appenv
, particularly if one's using zc.buildout
only for package installation and version pinning.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.