facundobatista / pyempaq

A simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere.

PyEmpaq

A simple but powerful Python packer to run any project with any virtualenv dependencies anywhwere.

With PyEmpaq you can convert any Python project (see limitations below) into a single .pyz file with all the project's content packed inside.

Packing is super simple, see this demo:

That single file is everything that needs to be distributed. When the final user executes it, the original project will be expanded, its dependencies installed in a virtualenv, and then executed. Note that no special permissions or privileges are required, as everything happens in the user environment.

See that in action:

Both the packaging and the execution are fully multiplatorm. This means that you can pack a project in Linux, Windows, Mac or whatever, and it will run ok in Linux, Windows, Mac or whatever. The only requirement is Python to be already installed.

You can try yourself some packed with PyEmpaq examples, very easy, just download any of these files and run it with Python:

• in a terminal: a very small pure terminal example (this, of course, needs to be run in a terminal)
• a game: a simple game using the Python Arcade library (actually, it's the example #6 from their tutorial)
• desktop app: a full-fledged desktop application using PyQt5 (this Encuentro app)

You can install pyempaq directly from PyPI; see instructions below.

PyEmpaq is security friendly, there is nothing obscure or secretly shipped when you distribute your project with it: anybody can just open the pyz file (it's just a ZIP) and inspect it.

It's also safe regarding licenses when distributing your packed software. Unlike packing mechanisms that rely on putting inside a big blob, with PyEmpaq you don't have to worry about the licenses of your software dependencies (and the dependencies' dependencies) in regard to distributing them.

Limitations:

There are some limitations:

• Only Python >= 3.7 is supported

• Only Linux, Windows and Mac is supported

• Only pip-installable dependencies are supported (from PyPI or whatever).

• Only dependencies that are pure Python or provide wheels are supported.

All this means that the most majority of the projects could be packed and run by PyEmpaq. If you have any ideas on how to overcome any of these limitations, let's talk!

How Does this Work?

There are two phases: packing and execution.

The packing phase is executed by the project developer, only once, before distribution. It's a simple step where the developer runs PyEmpaq indicating all needed info, and PyEmpaq will produce a single <projectname>.pyz file. That's all, and that only file is what is needed for distribution.

In this packing phase, PyEmpaq builds the indicated packed file, putting inside:

• the payload project, with all the indicated modules and binary files

• an unpacker script from PyEmpaq, which will be run during the execution phase

• few more stuff: some needed infrastructure details for the .pyz to run correctly

After packing, the developer will distribute the packed file, final users will download/receive/get it, and execute it.

In the execution phase all that needs to be done by the final user is to run it using Python, which can be done from the command line (e.g. python3 supergame.pyz) or by doing double click from the file explorer in those systems that relate the .pyz extension to Python (e.g. Windows).

In this execution phase, the unpacker script put by PyEmpaq inside the packed file will be run, running the following steps:

• check if the needed setup exists from a previous run; if yes, it will just run the payload project with almost no extra work; otherwise...

• create a directory in the user data dir (or the indicated one, see below), and expand the .pyz file there

• create a virtualenv in that new directory, and install all the payload's project dependencies

• run the payload's project inside that virtualenv

The verification that the unpacker does to see if has a reusable setup from the past is based on the .pyz timestamp; if it changed (a new file was distributed), a new setup will be created and used.

The environment variable PYEMPAQ_UNPACK_BASE_PATH can be used to specify in which base directory PyEmpaq will unpack the different projects. If indicated, the path must exist and be a directory.

PyEmpaq transparently returns the payload's exit code except when it must exit before the execution of the payload. In that case, it uses special codes equal or above 64, meaning:

• 64: unpack-restrictions are not met on the final user's system.
• 65: the indicated PYEMPAQ_UNPACK_BASE_PATH does not exist
• 66: the indicated PYEMPAQ_UNPACK_BASE_PATH is not a directory
• 67: the indicated PYEMPAQ_ACTION is not valid

Special Actions

When running the packed project, some special actions can be indicated through the environment variable PYEMPAQ_ACTION. Note that if it is specified, the special action will take place and the packed program itself will not be run.

Current special actions:

• info: Show information about installations of the given project

• uninstall: Remove all installations of the given project

Command Line Options

There is one mandatory parameter:

• source: it needs to point the configuration file or to the directory where the configuration will be located (it will be searched by its default name pyempaq.yaml).

Examples:

• pyempaq .

• pyempaq /data/project/

• pyempaq ~/repo/proj/config.yaml

You can control verbosity by adding these command line parameters to control logging levels:

• -v, --verbose - Show detailed information, typically of interest only when diagnosing problems.
• -q, --quiet - Only events of WARNING level and above will be tracked.

All that said, there is an special option -V, --version (new in v0.3) that if used will just print the version and exit.

Note In the execution phase, if you have an environment variable PYEMPAQ_DEBUG=1 it will show the Pyempaq log lines during the execution.

The Configuration File

All the information that PyEmpaq needs to pack a project comes from a configuration file, which the developer would normally have in the project itself.

The following is the structure of the pyempaq.yaml configuration file:

• name [mandatory]: the name of the project.

• basedir [optional]: the project's base directory; if not included it defaults to where the configuration file is located.

• exec [mandatory]: the section where is defined the information to execute the project once unpacked; it holds different subkeys (some subkeys are marked with †: ONE of those keys must be present, but only ONE):

  • script [†]: the filepath of the python script to run; when unpacking PyEmpaq will do python3 SCRIPT.

  • module [†]: the name of the module to invoke; when unpacking PyEmpaq will do python3 -m MODULE.

  • entrypoint [†]: freeform, as a list of strings; when unpacking PyEmpaq will only insert the proper python3 at the beginning: python3 STR1 STR2 ....

  • default-args [optional]: the default arguments to be passed to the script/module/entrypoint (if not overridden when the distributed .pyz is executed).

• requirements [optional]: a list of filepaths pointing to the requirement files with pip-installable dependencies.

• dependencies [optional]: a list of strings to directly specify packages to be installed by pip without needing to have a requirement file.

• include [optional, new in v0.3]: a list of strings, each one specifying a pattern to decide what files and directories to include in the pack (see below); if not included it defaults to ["./**"] which means all the files and directories in the project.

• exclude [optional, new in v0.3]: a list of strings, each one specifying a pattern to decide what files and directories to exclude from the pack (see below).

• unpack-restrictions [optional, new in v0.3]: a section to specify different restrictions to be verified/enforced during unpack.

  • minimum-python-version [optional, new in v0.3]: a string specifying the minimum version possible to run correctly.

All specified filepaths must exist inside the project and must be relative (to the project's base directory), with the exception of basedir itself which can be absolute or relative (to the configuration file location).

Both include and exclude options use pattern matching according to the rules used by the Unix shell, using *, ?, and character ranges expressed with []. Also ** will match any files and zero or more directories. For more information or subtleties check the glob.glob documentation.

Note that the final user may ignore/bypass any unpack restrictions using the PYEMPAQ_IGNORE_RESTRICTIONS environment variable with a value of comma-separated names of which restrictions to ignore.

The following are examples of different configuration files (which were the ones used to build the packed examples mentioned before):

• the terminal script

• the simple game

• the full desktop app

Context of the Project Being Run

When the packed script/module/whatever is run in the unpacking phase, it will be executed inside the virtualenv with the needed requirements/dependencies (freshly created and installed or reused from a previous run).

Also, the following environment variable will be set:

• PYEMPAQ_PYZ_PATH: the absolute path to the .pyz run by the user

How to Install PyEmpaq

Directly from PyPI:

pip install --user --upgrade --ignore-installed pyempaq

It's handy to install it using pipx, if you have it:

pipx install pyempaq

If you have fades you don't even need to install pyempaq, just run it:

fades -d pyempaq -x pyempaq

In the future there will be more ways to install it. Please open an issue if you desire an installation method (extra points if you specify how), thanks!

Try Packing an Example Project

PyEmpaq sources come with a small example project if you want to play a little packing it. Just a couple of dir/files under examples/srcproject:

• a src and media, with stuff to be imported and accessed.

• a pyempaq.yaml with the configuration for PyEmpaq.

• a ep.py file which is the project's entrypoint; all it does is to inform it started, import the internal module, access the media files, and use the declared dependency, reporting every step.

• a secrets.txt file that must not be included in the packed file.

This explores most of the needs of any project. You can try this example, and surely after you will be ready to actually pack any other project you want.

So, let's pack the example source project. As you're working with the PyEmpaq project itself (as you're packing its example), you don't really need to have it installed yet. In that case, if you have fades installed is super easy:

fades -r requirements.txt -m pyempaq examples/srcproject/

Otherwise, you would need to create and use a virtual environment:

python3 -m venv env
source env/bin/activate
pip install -r requirements
python3 -m pyempaq examples/srcproject/

After running that command, you will see a pyempaq-example.pyz file. That is the whole project encoded in a single file.

At this point you may move that pyempaq-example.pyz to another directory, or to another machine, even that other machine having another operating system.

Then, try it:

python3 pyempaq-example.pyz

You should see the project's reportings that we mentioned above (note: these lines will be surrounded by debug ones that will be hidden by default in the future):

Hello world
Code access ok .../pyempaq/projectname-20210722013526/orig/src/foo.py
Media access ok
Module requests imported .../pyempaq/projectname-20210722013526/venv/lib/python3.8/site-packages/requests/__init__.py

This shows that what you've run actually started, accessed the internal modules and other files, and imported correctly a custom-installed dependency.

How PyEmpaq Relates to Other Similar Tools?

There are other tools that are similar in their work. The more known are Shiv and Pex.

Shiv would create a fully self-contained Python zip app with all their dependencies included, which means that it ships not only your project but also the dependencies! this makes the resulting .pyz NOT multiplatform (unless your project and ALL your dependencies are pure Python...).

On the other hand, PyEmpaq will create the virtualenv and install the needed dependencies the first time the .pyz is executed in the destination machine. This allows for example to pack a fully graphical desktop program that works with Qt, like the example shown above).

A side detail of this is that we normally include dependencies in our programs, and those dependencies have sub-dependencies, etc. How sure are you that the whole set of dependencies and sub dependencies have a license that allows you to distribute them? Shiv will distribute them, PyEmpaq will not!

Pex looks like it also includes the dependencies in the resulting file, but goes an extra mile pointing to the Python that was used to build the thing. So for example I built a .pex with Py3.10 and moved the file to a machine that has only 3.9, and it didn’t work, and both were Ubuntus! No multiplatform at all, no even mention to pack in Linux (once!) and run in Windows (or everywhere), as PyEmpaq allows to do.

These both tools are "ahead of time" dense (larger size) archives that include application code and dependencies for a single target platform. Hermetic and deterministic installation. Something that you would also get if you snap your project.

On the other hand PyEmpaq is a sparse (smaller size) archive. It only includes the application code and at first-run will create and install the rest into a virtualenv. A bit like using pipx to install an application, except without needing to teach the end-user about pipx. It’s self installing. Non-hermetic and non-deterministic installation, but may not matter in practice.

How to Contribute to the Project?

Please check the how to contribute instructions.

pyempaq's Issues

Update test matrix for modern platforms/pythons

Currently we have:

        os: [ubuntu-20.04, ubuntu-18.04, windows-2019, windows-2022, macos-11, macos-12]
        python-version: [3.7, 3.8, 3.9, "3.10"]

• Ubuntu 18.04 is for sure deprecated, but we need to add Ubuntu 22.04. What about Windows and Mac?
• See what Python versions are still supported by python.org

Use `logged_exec` in unpacker, instead of `subprocess.run`

Add a shebang to the `.pyz` to make it "more executable"

This way, people would only need to do chmod +x to it and will be ready to go.

It looks Windows may also support it, according to this conversation

What would happen if we double-click on it in Linux?

Dump a file with installation metadata in the project directory

When creating (first time) installation a file with metadata about that installation should be dumped in the root of the project's directory, called installation.json, with the following info:

• timestamp: current timestamp
• pyz_path: the full path of the originating PYZ
• pyz_hash: full sha256 hash of the originating PYZ

Show the unpacking process in a GUI window

The reason for this is that if the environment creation takes long (downloading and installing packages!) the user will see that "something is going on".

This should be done if a terminal is not available (so we don't open GUI windows if the user is using the terminal) and if tkinter is available (use that GUI framework so we don't need extra dependencies).

The opened window title should be something like "Unpacking process (powered by PyEmpaq)" and it would be just a big text entry where the log lines will be added. After finishing, before the final process is executed, the window should be closed.

The `packaging` library is missing in the produced `.pyz` file

After packaging a simple project without dependencies I tried to run it and received this error message:

$ python3.11 Something.pyz 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/tmp.EQhCLfHNpy/Something.pyz/__main__.py", line 20, in <module>
ModuleNotFoundError: No module named 'packaging'

It seems to me like the same way appdirs is installed in the venv directory inside the .pyz output file, packaging should be installed as well.

Also I think you should consider migrating from appdirs to platformdirs, which is more up-to-date (as far as I know, it is vendored-in by pip itself and thus will keep receiving a lot of attention in terms of continuous maintenance).

Add a -V/--version option

It should just output the version and quit.

Make `logged_exec` a common function and test it

Note that as will be used from the unpacker, the new common module needs to be imported also from there, so needs to be "packed" in the final project.

Support `setup.py` as source of the information to pack

IOW, if the user does pyempaq /path/to/project/setup.py all the information to pack the project (basedir, files to include, script to execute, etc.) should be obtained from that file.

Also, setup.py needs to be searched for if the user specifies a directory as "source".

Support `setup.cfg` as source of the information to pack

IOW, if the user does pyempaq /path/to/project/setup.cfg all the information to pack the project (basedir, files to include, script to execute, etc.) should be obtained from that file.

Also, setup.cfg needs to be searched for if the user specifies a directory as "source".

Pack PyEmpaq itself as a snap

Name pyempaq already registered.

At unpacking, flag the successful installation of dependencies and use that to check the reusing

Currently, when the unpacker starts, it checks that the project dir exists to reuse it:

    if project_dir.exists():

If not, it's created, and dependencies are installed. The problem is that because of underlying virtualenv magic (which varies from system to system, and we don't want to get into that, just be robust) the venv.create() call may end up in the process being exited, without properly finishing the whole directory setup.

So, we need to:

• create a flag file inside project_dir, something like project-setup-done.flag (empty is enough)
• when verifying that "the previous process did all fine", use this new file, like:

    if project_dir.exists() and (project_dir / project-setup-done.flag).exists() :

Publish documentation (README) on ReadTheDocs

Support different forms of installation, if source project is prepared

If the source project is prepared and opts-in, the first run of a project and everytime the packed project changes, the unpacker should do stuff for the unpacked project so it remains "installed" in the final system.

• Linux: the project could have a .desktop file, and the unpacker could put it in the proper place (if exists)

• Windows: maybe add a "symlink" ("direct access", are they called?) in the user's desktop, pointing to the "expanded" location

  • as this dirties the user environment, maybe the project should "opt in"
  • how can the project be registered so it appears "in the menu"?

• Mac OS: no clue on what is needed here

Validate that the payload is not corrupted

The original project should not be included as "directories and files", but it should be zipped into orig.zip (of course, will be unzipped by unpacker, the net effect is not changed by this issue).

Then, a hash will be created using the bytes from:

• that orig.zip
• the created metadata json
• unpacker.py
• the common.py library

That hash will be added to metadata.json before saving.

At unpacking time, the unpacker will get the metadata, and as a first step validate the integrity: remove the "integrity hash" (as it was added after hashing when packing), re hash everything (taking in consideration that unpacker.py is now __main__.py), and verify values.

Support for handling the "install caches" client side

When the final user executes the .pyz a new install (virtualenv + dependencies) will be created. Running exactly the same .pyz will reuse that installation, but newer versions of that program, other programs, etc, all will create new installs.

So far there is nothing in the distributed .pyz to handle those installations (beyond running it with PYEMPAQ_DEBUG so the installation path is shown among other messages). We would like to have a way to handle those "install caches", for example to:

• uninstall a .pyz: this would remove all installs for the given program (all its versions, and all created by different Python versions after #58 is implemented)
• show info about the install cache: path location, size, maybe even a pip freeze of its virtualenv
• something else?

There is to be decided how these behaviours need to be triggered. @sinoroc correctly suggests that it should be part of executing the program's .pyz (I said that maybe we could have an extra tool, but he's right, using the same .pyz is more natural).

I can think of two ways to trigger these behaviours, then. Through a command line option or an environment variable (the later also suggested by @sinoroc).

• command line option: this would be through special options that have low probability on clashing with the options from the end program itself, like python3 myprogram.pyz --pyempaq-action=uninstall; note this is not 100% new, other systems/frameworks do the same (e.g. when running GTK-based programs you can pass --gtk-module=module)

• environment variable: something like PYEMPAQ_ACTION=uninstall python3 myprogram.pyz

In any case the end program will NOT be run, only the specific PyEmpaq action.

Other detail to consider here is that all these options are "terminal based", the user will never be able to run these functionalities by "double clicking the pyz". This is somewhat ok, for the 99% of basic cases, you'll never need to handle install caches at all. But if you want, there is a way.

Remove the installation if indicated

We shall support the PYEMPAQ_EPHEMERAL environment variable; if true, the installation directory shall be removed after running the unpacked project.

It's fine if the project's directory already exists when running PyEmpaq in ephemeral mode . In that case a WARNING message should alert that the directory previously existed, then run the unpacked project, and finally remove that directory.

Make `find_venv_bin` a common function and test it

Note that as will be used from the unpacker, the new common module needs to be imported also from there, so needs to be "packed" in the final project.

Allow to specify the minimum required Python version to run the packed content

The configuration should have a key called unpack_restrictions. It's a "dict", currently will have only one key: `minimum_python_version". Example:

unpack_restrictions:
    minimum_python_version: "3.8"

When packing, the code should:

• validate that the value is a string, not a float (a common mistake is writing the YAML like minimum_python_version: 3.8, which ends up being a float).
• annotate this inside the metadata, for the unpacker

When unpacking, it should:

• very early in the process verify if that metadata is present, and if yes, it should validate that the Python used to run the .pyz is fine.
• the comparison should be done using parse_version (from pkg_resources)
• if fails, it should error out saying "Unsupported Python version, the minimum required is XXX" (where XXX is the repr() of the string that the developer configured)
• however, if the PYEMPAQ_IGNORE_RESTRICTIONS environment variable is set, it should NOT fail (but still show a warning with the same text)

The project dir name should include the Python version

Currently, if you run the .pyz with Python X, it will create all directories and install everything. If you run the same .pyz with Python Y, it will reuse that installation, but is wrong, as the virtualenv and dependencies installation may depend on the Python version to work properly).

So, the project dir name should be composed by the project name and .pyz timestamp (as today) plus the Python version.

This way, in the situation aforementioned, the unpacker would just create a new directory with a clean install and everything would just work.

Specify exactly what to include in the project's config

Add a key include to pyempaq.yaml to specify what will be included in the project. It will be a list of strings, supporting wildcards, in special the double star one.

Also, add a key ignore to specify what to be ignored (no matter if it was included before). This is to be able to specify a whole directory or tree, but at the same time ignore temp files, *.pyc, etc.

Each indication line in both lists will be managed using glob.glob(indication, recursive=True), doc is here.

Particularly, one include entry being ./** means "everything in the project directory", but use with caution, you may have secrets or other files that you don't want to distribute.

Properly document this.

Use Python's logging in `main`

Use a simple format:

• timestamps (with microseconds), but no date
• timestamp, the level, and the message

Stop using print:

• all DEBUG prints should be logger.debug
• stuff inside logged_exec also .debug
• the rest should be .info

Support `pyproject.toml` as source of the information to pack

IOW, if the user does pyempaq /path/to/project/pyproject.toml all the information to pack the project (basedir, files to include, script to execute, etc.) should be obtained from that file.

Also, pyproject.toml needs to be searched for if the user specifies a directory as "source".

Expose the .pyz path to the packaged program

I have a program that I want to distribute as a pyempaq .pyz, but when running I want to create and use a data directory in the same location the user placed the .pyz file. So in my ,py I would need a mechanism to get the path of the .pyz from which it that was executed.

Refactor source of information

We need to stop using command line arguments to gather the needed information to pack a project, it does not scale.

We will use a YAML file, pyempaq.yaml, that will hold all the required information:

• basedir: the project's base dir (absolute or relative to the YAML; the rest of the paths can be absolute or relative to this basedir)
• exec, which hold different subkeys
  • an XOR of the following ones:
    • script: the filepath of the python script to run; when unpacking PyEmpaq will do python3 SCRIPT
    • module: the name of the module to invoke; when unpacking PyEmpaq will do python3 -m MODULE
    • entrypoint: freeform, as a list of strings, when unpacking PyEmpaq will only insert the proper python3 at the beginning
  • also, default-args (not now, see this issue)
• requirements: a list of filepaths pointing to the requirement files with pip-installable dependencies

Note that we would need to heavily adapt/fix the README after this change. In particular, we need a section explaining all the pyempaq.yaml keys.

The command line will end up being VERY simple:

• the --verbose and/or --quiet options to control verbosity (see this issue)
• a source mandatory parameter: the path to a file or a directory; if it's a file, it will be source of information, if it's a directory PyEmpaq will search for the source there (sequentially pyempaq.yaml, setup.py, etc.)

Properly explain i the README how the command line works.

Python’s `venv.create` does not work in GA context for Windows

That's why in tests/test_integration.py there is some commented out code (search for XXX).

I opened this post in the Github Community forum for this, and there is a PR with a special test case that only calls venv.create, to easily see the problem.

Support PyBI

Add support for packing a Python interpreter in the PyBI format, and using that interpreter at runtime.

Obviously right now that format is still in draft, but perhaps it's not much work to implement and could make PyEmpaq able to run without Python installed (or at least with an arbitrary version).

The unpacker should execute the entrypoint with parameters, if available

The parameters may come from two different sources:

• a default-args key in the metadata (under exec)
• the .pyz execution itself (which overrides default-args, if any)

Include and exclude settings seem to be broken

(Using pyempaq v0.2.2, python 3.8.10)

If I specify the include: and/or exclude: fields in the pyempaq.yaml file, when trying to build the package I get this error:

simpyt ❯ pyempaq .
Parsing configuration in '.'
Problem(s) found in configuration file:
- 'exclude': extra fields not permitted
- 'include': extra fields not permitted

Example failing config:

name: simpyt
exec:
  script: simpyt.py
requirements:
  - requirements.txt
include: 
  - simpyt.py
  - assets
  - example_user_dir
  - templates
  - README.md
exclude:
  - tests

I tried fixing it, found out that the Config class (from config_manager.py) doesn't have the required fields:

class Config(ModelConfigDefaults, validate_all=False):
    """Definition of PyEmpaq's configuration."""

    name: str
    basedir: pathlib.Path  # the default for this is managed on YAML load time
    exec: Executor
    requirements: List[RelativeFile] = []
    dependencies: List[str] = []

Tried adding them:

(...)
    include: List[RelativeFile] = []
    exclude: List[RelativeFile] = []

And the error went away, but then the settings seem to have no effect whatsoever, as the resulting package includes all the files from the source directory, not just the ones specified in include:, and even including the tests folder specified in exclude:.

Add `codespell` to the automatic verifications

We should run codespell before merging anything to main...

Allow to change the location where the files get unpacked

This shall be done through the PYEMPAQ_UNPACK_PATH environment variable. The indicated path must exist and be a directory, otherwise it's an error.

It will override the usage of appdirs.user_data_dir() / 'pyempaq'; in other words, the project directory will be created directly in the indicated path.

For example: PYEMPAQ_UNPACK_PATH=/bigdisk/installs python3 superapp.pyz.

Control verbosity in `main`

Add --verbose and --quiet command line to control logging levels.

Migrate from 'appdirs' to 'platformdirs'.

The former is stalled, the later is the lib to use.

Test the `pack` function

Big and complicated, but should be tested to ensure functionality

Log the return code of the executed project

After running the unpacked project, it should log its return code.

Consider stop creating a virtualenv for unpacker

Currently when packing a virtualenv is created for unpacker to run, providing the following two modules: packaging and appdirs.

@sinoroc in #55 indicates to consider"migrating from appdirs to platformdirs, which is more up-to-date (as far as I know, it is vendored-in by pip itself and thus will keep receiving a lot of attention)". The problem here is that pip may not be installed in the destination system.

Regarding packaging, we'd need a way to compare versions integrated in the stdlib or similar...

Control default verbosity in `unpacker`

Only show the lines in the log function when PYEMPAQ_DEBUG=1. Annotate this "trick" in the README.