sio / makefile.venv Goto Github PK

I'm using Big Sur 11.2.

Somehow, sha256sum is not pre-installed on my machine. So I have to change from:

echo "5afbcf51a82f629cd65ff23185acde90ebe4dec889ef80bbdc12562fbd0b2611 *Makefile.fetched" \
		| sha256sum --check - \
		&& mv Makefile.fetched Makefile.venv

to:

echo "5afbcf51a82f629cd65ff23185acde90ebe4dec889ef80bbdc12562fbd0b2611 *Makefile.fetched" \
		| shasum -a 256 --check - \
		&& mv Makefile.fetched Makefile.venv

Wonder if it should be updated...

Scheduled test runs have started failing today on windows-latest:

First failure: https://github.com/sio/Makefile.venv/runs/5257374669?check_suite_focus=true

This run was using:

Python 3.9.10 (tags/v3.9.10:f2f3f53, Jan 17 2022, 15:14:21) [MSC v.1929 64 bit (AMD64)]
Windows-10-10.0.20348-SP0
GNU Make 4.2.1

Previous successful run on the same codebase: https://github.com/sio/Makefile.venv/runs/5121494065?check_suite_focus=true

This run was using:

Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:58:18) [MSC v.1900 64 bit (AMD64)]
Windows-10-10.0.17763-SP0
GNU Make 4.2.1

Probable reasons for failure

The obvious change in CI runner is Python upgrade from 3.7.9 to 3.9.10. The failure is not inherent to that Python version: macos and cygwin were already using it successfully in the same run.

Rerun eliminated test failures related to command timeouts, but this one has remained:

FAIL: test_setup_cfg (tests.test_dependencies.TestDependencies)
Check that setup.cfg is being processed correctly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\Makefile.venv\Makefile.venv\tests\test_dependencies.py", line 33, in test_setup_cfg
    make = self.make('hello', makefile=makefile)
  File "D:\a\Makefile.venv\Makefile.venv\tests\common.py", line 62, in make
    self.check_returncode(process, returncode)
  File "D:\a\Makefile.venv\Makefile.venv\tests\common.py", line 67, in check_returncode
    self.assertEqual(
AssertionError: 2 != 0 : make exited with code 2 (expected 0)

stdout:
mingw32-make[1]: Entering directory 'C:/Users/RUNNER~1/AppData/Local/Temp/Makefile.venv_test_7o0hpfn3'
./.venv/Scripts/pip install -e ./
Obtaining file:///C:/Users/RUNNER~1/AppData/Local/Temp/Makefile.venv_test_7o0hpfn3
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: pyfiglet in c:\users\runneradmin\appdata\local\temp\makefile.venv_test_7o0hpfn3\.venv\lib\site-packages (from hello==0.0.0) (0.8.post1)
Installing collected packages: hello
  Attempting uninstall: hello
    Found existing installation: hello 0.0.0
mingw32-make[1]: Leaving directory 'C:/Users/RUNNER~1/AppData/Local/Temp/Makefile.venv_test_7o0hpfn3'

stderr:
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\cli\req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\commands\install.py", line 405, in run
    installed = install_given_reqs(
  File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\req\__init__.py", line 68, in install_given_reqs
    uninstalled_pathset = requirement.uninstall(auto_confirm=True)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\req\req_install.py", line 637, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\Makefile.venv_test_7o0hpfn3\.venv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 530, in from_dist
    assert link_pointer == dist_location, (
AssertionError: Egg-link c:\users\runner~1\appdata\local\temp\makefile.venv_test_7o0hpfn3 does not match installed location of hello (at c:\users\runneradmin\appdata\local\temp\makefile.venv_test_7o0hpfn3)
mingw32-make[1]: *** [D:\a\Makefile.venv\Makefile.venv\Makefile.venv:223: .venv/Scripts/.initialized-with-Makefile.venv] Error 2

The last part indicates that the issue is caused by pip failing to correctly compare shortened Windo~1 path to a full version:

AssertionError: 
Egg-link c:\users\runner~1\appdata\local\temp\makefile.venv_test_7o0hpfn3 
does not match installed location of hello 
(at c:\users\runneradmin\appdata\local\temp\makefile.venv_test_7o0hpfn3)

I'll wait this out a little in hopes that upstream pip will fix the issue in the next update. Will check pip bug reports now.

See docs/howto-poetry.md, docs/howto-pipenv.md

GitHub Actions provide Windows environment with half of Unix installed on top. This is not exactly representative of an average Windows PC.

We should run CI tests in a more bare environment: just Python from python.org, some build of GNU Make for orchestration (choco?) and cmd.exe for shell. There should be no touch command, no /dev/null to write to, no sh.exe to fall back to... Code path for such environment will be significantly different to what we test now, and some new issues may be discovered.

One way to achieve this would be to use Docker Windows containers.

As with many things Windows, running Docker is annoying:

• Be prepared to waste 2GB of RAM on a Linux VM even if you intend to run zero Linux containers
• Windows container OS version must match host OS, say goodbye to reproducibility unless you're ready to spin up the same version of Windows Server GHA is currently using.
• Typical container size for Windows 10 is about 8GB. Nanoserver images exist, but they are not compatible with Windows 10.

Currently I have no patience for this obnoxious stuff. Leaving it here in case someone else is interested.

Hi, when Makefile.venv creates a venv, it does this:

I have a Makefile to automate the build process. It needs to do some steps before the build, otherwise the build will fail. I don't really understand why Makefile.venv tries to install the local project. In my eyes, Makefile.venv manages the venv using requirements.txt and the other Makefile manages the build of the project. Am I misinterpreting something?

It would be wonderful if there were a way to specify that optional dependencies are to be installed; e.g., in my pyproject.toml, I have:

[project.optional-dependencies]
dev = ["mypy", "black", "pylint", "pytest", "pytest-md", "validate-pyproject"]

This would allow projects to avoid having a separate requirements.txt file if they choose not to.

venv is a .PHONY target and rules that depend on it will be executed every time make is run. This behavior is sensible as default because most Python projects use Makefiles for running development chores, not for artifact building. In cases where that is not desirable use order-only prerequisite syntax:

For the majority of my use, venv being phony is fine, as I can set it as an order-only dependency.

But I have a make rule that invokes pyinstaller, and so I want my output artefacts to be rebuilt if venv changes. What's the best way to do this? For now I've just gone for $(VENV)/$(MARKER), but is that the best solution? (Especially as it means I have to include Makefile.venv near the top)

I want to override the virtual environment dir to include the Python version on the path.

Here's the relevant part of my Makefile:

PYTHON_VERSION := 3.7
PY := python$(PYTHON_VERSION)
VENVDIR := .venv/$(PYTHON_VERSION)
include release/Makefile.venv

And here is the output for a simple command

$ make show-venv
release/Makefile.venv:134: warning: overriding recipe for target '.venv/3.7'
release/Makefile.venv:129: warning: ignoring old recipe for target '.venv/3.7'
release/Makefile.venv:174: warning: overriding recipe for target '.venv/3.7'
release/Makefile.venv:134: warning: ignoring old recipe for target '.venv/3.7'
release/Makefile.venv:173: *** mixed implicit and normal rules: deprecated syntax
make: Circular .venv/3.7 <- .venv/3.7 dependency dropped.
make: Circular /bin/.initialized-with-Makefile.venv <- .venv/3.7 dependency dropped.
.venv/3.7  /bin/pip install -r requirements.txt
make: .venv/3.7: No such file or directory
make: *** [release/Makefile.venv:134: /bin/.initialized-with-Makefile.venv] Error 127

Hardcoding the VENVDIR without variables doesn't help, and neither does turning it into a regular variable (VENVDIR = .venv/3.7).
Maybe I'm doing something wrong?

I'm using version "v2020.05.05"

Hi there, thanks for this cool tool :)

Is there any way to pass additional arguments to the pip install that installs the requirements files? Specifically, my workflow involves dependencies that are in a private repository so I need to pass --extra-index-url.

Looking at the source it seems it is not possible at moment. Perhaps something like this could do the job?

REQUIREMENTS_TXT?=requirements.txt  # Multiple paths are supported (space separated)
PIP_INSTALL_ARGS?=
...
ifneq ($(_REQUIREMENTS),)
	$(VENV)/pip install $(foreach path,$(_REQUIREMENTS),-r $(path)) $(PIP_INSTALL_ARGS)
endif
...

If the above is sensible I can throw a PR :)

It seems normal for projects to make custom rules for generating *.txt from *.in, and such a rule will need to use venv for pip-compile.

However when we tried it, it creates a circular dependency since Makefile.venv has a venv dependency on the REQUIREMENTS_TXT files.

Can Makefile.venv support this use case?

Perhaps Makefile.venv is doing too much implicitly-- could it do less and do it explicitly?

cc: @alegonz

If you run make with -R, there is no value for $(RM) set, which causes clean-venv to fail. This is probably worth fixing with a simple ?= rule so that this works without relying on default variable values.

Hi,
Thanks for the code. I had to replace spaces by underscore in my project directory name to make it work properly.
Minh

First, let me say I love this project. I started down this road myself a few hours before finding it. Super helpful, saved me a lot of time, and showed me things I hadn't even thought of yet.

I'm using GitHub actions which are dependent on environment variables that should not be stored in git. A local .env file which ii sourced by targets while testing supports this workflow.

Enabling the shell targets to source a .env file would remove a step if doing more interactive manual testing.

After using this in several projects (thanks for sharing this!), I found myself needing project-specific shell scripts for some tasks.

I wanted to re-use the venv created by this tool, since it has all the necessary tools already installed, but this is not easy to do, since I'm writing a shell script, not Python code. After some trial and error, I found this to work very well:

Expose the VENV location to the child processes

On the Makefile, use:

export VENV

Scripts

On the scripts themselves, I use this snippet on the top:

#!/bin/bash
if [ -n "$VENV" ]; then
	echo "# Augment PATH with '$VENV'" >&2
	PATH="$(realpath "$VENV"):$PATH"
	export PATH
fi

That last export re-exports the PATH to child processes, so that they can also use the venv tools. These scripts also work inside the shell from make bash, with duplicated PATH.

Another (worse) alternative is to adjust all calls, using something like this:

"${VENV+$VENV/}python" --version

Makefile Targets

That way you can define target in the "usual" way in the Makefile, and all the venv tools are available on the PATH:

output.txt: input.txt | $(VENV)
    location/script <$< >$@

This requires changes on the scripts themselves, but I think this is the cleanest way.

Just putting this out here in case anyone runs into the same problem.

currently Makefile.venv tries to install the latest pip version at venv creation time

But we found that having an arbitrary version can cause problems. The use case is when packaging wheel files so that we can have a local, "wheel-only" install (i.e. no indexes or network connection at install time).

1. pip wheel is used, and the selected whl files are determined by the pip version
2. at install time, a different pip version is used which doesn't support the packages wheel files

(for example, see the table at https://github.com/pypa/manylinux)

So it may be useful to have an option to pin the pip version.

Hi, I stumbled upon a problem: Recently Windows user tend to use the Python Launcher for Windows to execute python programs. So they use, e.g., py -3 script.py instead of python3 script.py to launch a script with Python 3.

Unfortunately that also means that the python executable is not be on the path with its *nix/Linux name.

This leads to a problem due to PY using the *nix/Linux name

It is very annoying that Windows continuously creates problems for cross-platform tools (like Makefile.venv). I don't have a good idea how to solve that. Maybe detect if py.exe can be found on PATH before the OS detection and if so replace 'python' with 'py -' in PY? What do you think?

I built this snippet that seems to work:

# Detect Windows py launcher and use it to find the given python version
ifeq ($(OS),Windows_NT)
ifneq (, $(shell where py 2>NUL))
PY := $(subst python,py -,$(PY))
endif
endif

Some more infos:

• running py -3.7 -m venv .venv results in a proper .venv/ with python.exe and pip.exe in .venv/Scripts/
• running py -3.7 -c "import __future__, sys; print(sys.platform)" prints win32
• related to #2

I had to replace the block where OS is detected in order to support cygwin.
I believed that the following doesn't break anything, but you may want to double check before doing the change:

VENV=$(VENVDIR)/bin
EXE=
ifdef OS  # Windows support
ifneq ($(shell echo $$OSTYPE),cygwin) #Cygwin support
VENV=$(VENVDIR)/Scripts
EXE=.exe
endif #OSTYPE
endif #OS

hi, thanks for the nice timesaving package - would you happen to know how I can integrate usage of pyproject.toml over requirements.txt?

Cygwin behaves quite differently from both Windows and Linux. Running CI tests in Cygwin seems reasonable.

Related: #2