eight04 / vpip

A CLI which aims to provide an npm-like experience when installing Python packages.

vpip

A CLI which aims to provide an npm-like experience when installing Python packages.

Features

• Install packages to isolated global virtual environments.
  • Executables are linked to the Python Scripts folder so you can still use the CLI without activating the venv.
  • This allows you to install different CLI tools without worrying about dependency conflicts.
• Install packages to a local virtual environment.
  • Dependencies are stored in requirements.txt (development) and setup.cfg/pyproject.toml (production).
• When removing a package, also remove its sub-dependencies.
• Easily run commands in the local virtual environment.
• Generate a lock file (requirements-lock.txt).

Installation

vpip is hosted on pypi:

pip install vpip

After installing vpip as a CLI, you can use it to install other packages (globally or locally).

Usage example

Install:

• vpip install - Create/activate a local venv and install all dependencies.
• vpip install configupdater - Create/activate a local venv, install configupdater, and add to production dependency.
• vpip install -g youtube-dl - Create a venv under ~/.vpip, install youtube-dl, and link the executable (youtube-dl.exe) to the script folder.
• vpip install -g https://github.com/eight04/ComicCrawler/archive/refs/heads/master.zip - You can also install global CLI from a URL.

Uninstall:

• vpip uninstall pylint - Activate the local venv, uninstall pylint, and remove pylint from both dev/prod dependency.

Update:

• vpip update pylint - Upgrade pylint to the compatible version.
• vpip update pylint --latest - Upgrade pylint to the latest release.

Execute command:

• vpip run python - Launch python REPL in the local venv.
• vpip run pylint my_proj - Run pylint installed in the local venv.

List dependencies:

• vpip list - List development/production dependencies.
• vpip list --outdated - List development/production dependencies that are outdated.
• vpip list -g - List globally installed packages.

Compatibility

vpip is a cross-platform CLI. Please report any compatibility issues.

Documentation

https://vpip.readthedocs.io/en/latest/index.html

Similar projects

• pipm - which doesn't use virtualenv.
• pipx - like vpip install -g.
• pdm - a more feature-rich depdenency manager.

Changelog

• 0.9.2 (Feb 4, 2024)
  • Fix: ignore unsupported version number in list --outdated.
• 0.9.1 (Oct 22, 2023)
  • Fix: type error on Python 3.9
• 0.9.0 (Oct 22, 2023)
  • Change: bump to python>=3.9, update dependencies.
  • Add: support pyproject.toml.
  • Add: link command now accepts an optional package name.
  • Fix: install -g error when using a specifier.
  • Fix: JSON error in pip_api.list_.
• 0.8.0 (Apr 23, 2022)
  • Change: now vpip would try to avoid sub-dependencies conflicts by passing all dependencies to pip install when installing/updating packages.
  • Fix: now vpip update won't install packages whose env marker evaluates to false.
• 0.7.0 (Feb 9, 2022)
  • Change: now wheel is also pre-installed in venv like pip.
• 0.6.0 (Jan 25, 2022)
  • Fix: make sure the script folder is in env variable path when vpip link.
  • Add: support installing global CLI from a URL.
  • Change: pip_api.install now accepts multiple packages.
  • Change: vpip update now updates sub-dependencies.
• 0.5.0 (Jan 5, 2022)
  • Change: bump to [email protected]+
  • Change: drop pkg_resources, improve performance.
  • Change: don't use pip internal when finding global scripts folder.
  • Fix: use utf8 encoding when reading/writing setup.cfg or requirements.txt.
• 0.4.3 (Jan 31, 2020)
  • Fix: don't break sub-dependencies when uninstall.
• 0.4.2 (Nov 9, 2019)
  • Fix: generate setup.py automatically if needed.
  • Fix: use utf8 encoding when parsing pip output.
• 0.4.1 (Nov 2, 2019)
  • Nothing is changed. Updated README and corrected some errors.
• 0.4.0 (Nov 1, 2019)
  • Fix: rebuild egg files after doing an incompatible update.
  • Fix: clean unused packages after uninstall.
  • Add: update_venv command.
  • Breaking: pip_api.show and pip_api.uninstall now accept multiple packages.
• 0.3.0 (Oct 31, 2019)
  • Support Unix system.
  • Add: generate a lock file.
• 0.2.3 (Feb 10, 2019)
  • Fix: pypi.is_compatible treat 0.1.0 and 0.2.0 as compatible.
  • Fix: don't include pre-release when checking updates.
  • Update dependencies.
• 0.2.2 (Feb 2, 2019)
  • Add: link command.
• 0.2.1 (Nov 16, 2018)
  • Add: user defined commands.
• 0.2.0 (Nov 16, 2018)
  • Add documentation.
  • Change: console scripts will be overwritten by default, matching pip's default behavior.
• 0.1.0 (Nov 13, 2018)
  • First release

vpip's Issues

Does pinning dependencies really make sense in python project?

In node, package-lock contains the information of the entire environment.

However, packages in python are so platform dependent that I started feeling generating the lock file makes no sense.

You get different packages for different os, different python version.
Some packages can be built on windows, some others require unix.
Some packages provides wheel, some others don't.
Some of those issues raise installation errors, some others raise runtime errors.

In fact, there is 80% of the time that I have to delete the lock file and re-install all packages after upgrading python.

`vpip update` should ignore packages if env marker evaluated to false

https://stackoverflow.com/questions/53417969/how-to-evaluate-pep-508-style-platform-markers

pip backtrack

I was trying to update prod dependencies of ptt-mail-backup

vpip update uao wcwidth --latest

It is fast while installing packages:

Requirement already satisfied: uao in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (0.1.1)
Collecting uao
  Downloading uao-0.2.0-py3-none-any.whl (229 kB)
     |████████████████████████████████| 229 kB 1.1 MB/s
Installing collected packages: uao
  Attempting uninstall: uao
    Found existing installation: uao 0.1.1
    Uninstalling uao-0.1.1:
      Successfully uninstalled uao-0.1.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ptt-article-parser 0.5.0 requires uao<0.2,>=0.1, but you have uao 0.2.0 which is incompatible.
ptt-mail-backup 0.3.0 requires uao~=0.1.1, but you have uao 0.2.0 which is incompatible.
Successfully installed uao-0.2.0
Requirement already satisfied: wcwidth in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (0.1.9)
Collecting wcwidth
  Downloading wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Installing collected packages: wcwidth
  Attempting uninstall: wcwidth
    Found existing installation: wcwidth 0.1.9
    Uninstalling wcwidth-0.1.9:
      Successfully uninstalled wcwidth-0.1.9
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ptt-mail-backup 0.3.0 requires uao~=0.1.1, but you have uao 0.2.0 which is incompatible.
ptt-mail-backup 0.3.0 requires wcwidth~=0.1.7, but you have wcwidth 0.2.5 which is incompatible.
Successfully installed wcwidth-0.2.5

However, it did a 40 mins backtrack while running pip install -e .

Obtaining file:///C:/Users/eight04/dev/ptt-mail-backup
Requirement already satisfied: paramiko~=2.6 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from ptt-mail-backup==0.3.0) (2.7.2)
Requirement already satisfied: ptt-article-parser~=0.5.0 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from ptt-mail-backup==0.3.0) (0.5.0)
Requirement already satisfied: uao~=0.2.0 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from ptt-mail-backup==0.3.0) (0.2.0)
Requirement already satisfied: wcwidth~=0.2.5 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from ptt-mail-backup==0.3.0) (0.2.5)
Requirement already satisfied: cryptography>=2.5 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from paramiko~=2.6->ptt-mail-backup==0.3.0) (3.4.7)
Requirement already satisfied: pynacl>=1.0.1 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from paramiko~=2.6->ptt-mail-backup==0.3.0) (1.4.0)
Requirement already satisfied: bcrypt>=3.1.3 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from paramiko~=2.6->ptt-mail-backup==0.3.0) (3.2.0)
Requirement already satisfied: cffi>=1.1 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from bcrypt>=3.1.3->paramiko~=2.6->ptt-mail-backup==0.3.0) (1.14.5)
Requirement already satisfied: six>=1.4.1 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from bcrypt>=3.1.3->paramiko~=2.6->ptt-mail-backup==0.3.0) (1.16.0)
Requirement already satisfied: pycparser in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko~=2.6->ptt-mail-backup==0.3.0) (2.20)
Requirement already satisfied: safeprint~=0.2.0 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from ptt-article-parser~=0.5.0->ptt-mail-backup==0.3.0) (0.2.0)
Requirement already satisfied: docopt~=0.6.2 in c:\users\eight04\dev\ptt-mail-backup\.venv\lib\site-packages (from ptt-article-parser~=0.5.0->ptt-mail-backup==0.3.0) (0.6.2)
INFO: pip is looking at multiple versions of cryptography to determine which version is compatible with other requirements. This could take a while.
Collecting cryptography>=2.5
  Using cached cryptography-3.4.7-cp36-abi3-win32.whl (1.4 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Downloading cryptography-3.4.6-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 1.1 MB/s
  Downloading cryptography-3.4.5-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 3.3 MB/s
  Downloading cryptography-3.4.4-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 6.4 MB/s
  Downloading cryptography-3.4.3-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 3.3 MB/s
  Downloading cryptography-3.4.2-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 6.4 MB/s
  Downloading cryptography-3.4.1-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 6.4 MB/s
Collecting setuptools-rust>=0.11.4
  Downloading setuptools_rust-0.12.1-py3-none-any.whl (22 kB)
INFO: pip is looking at multiple versions of cryptography to determine which version is compatible with other requirements. This could take a while.
Collecting cryptography>=2.5
  Downloading cryptography-3.4-cp36-abi3-win32.whl (1.4 MB)
     |████████████████████████████████| 1.4 MB 6.4 MB/s
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Downloading cryptography-3.3.2-cp36-abi3-win32.whl (1.3 MB)
...

Ended with the following error:

ERROR: Cannot install cryptography==3.4.7 and paramiko because these package versions have conflicting dependencies.

The conflict is caused by:
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.7 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.6 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.5 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.4 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.3 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.2 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4.1 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.4 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.3.2 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.3.1 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.3 depends on cffi>=1.12
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.2.1 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.2 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.1.1 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.1 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 3.0 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.9.2 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.9.1 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.9 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.8 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.7 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.6.1 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.6 depends on cffi!=1.11.3 and >=1.8
    bcrypt 3.1.3 depends on cffi>=1.1
    cryptography 2.5 depends on cffi!=1.11.3 and >=1.8

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Upgraded pip not used

See You are using pip version 9.0.1, however version 19.3.1 is available. in

> vpip install -g story
building venv at /home/jayvdb/.vpip/pkg_venvs/story
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.3.1
Collecting story
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/ab/0d/35476f94059d480573ddd46b84f318721d9de8bc120d763cf11fd831d1ab/story-0.21.2.tar.gz (76kB)
    100% |████████████████████████████████| 81kB 499kB/s 
Collecting click-completion==0.5.1 (from story)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/77/fd/2d7ec2b86cd4d487abf0b13dce58e98413096c45b9645470be0cb8de6ff2/click-completion-0.5.1.tar.gz
Collecting click-help-colors==0.5 (from story)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/4e/88/10c3472e038bad3a504496df1cb657a8caf505b5cf4c6b95f3cae4cd4b10/click-help-colors-0.5.tar.gz
Collecting click-spinner==0.1.8 (from story)
  Cache entry deserialization failed, entry ignored
Requirement already satisfied: click==7.0 in /usr/lib/python2.7/site-packages (from story)
Collecting emoji==0.5.0 (from story)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/47/e7/cf76e0774b13459e61ffe16a042a35f42e292aeebba060b919c05b0854ab/emoji-0.5.0.tar.gz
Collecting raven==6.9.0 (from story)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/11/3a/b3e46b279b8bdd9eb55857d0e95044cad31732c80f628bb75e1e9e881a32/raven-6.9.0-py2.py3-none-any.whl
Requirement already satisfied: requests~=2.20 in /usr/lib/python2.7/site-packages (from story)
Collecting storyscript==0.25.2 (from story)
  Cache entry deserialization failed, entry ignored
  Could not find a version that satisfies the requirement storyscript==0.25.2 (from story) (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.0.14, 0.0.15, 0.0.16, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 0.9.9, 0.9.10, 0.9.11, 0.9.12, 0.9.13, 0.9.14, 0.9.15, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.15.0, 0.15.1, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.17.2, 0.17.3, 0.18.0)
No matching distribution found for storyscript==0.25.2 (from story)
You are using pip version 9.0.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/usr/bin/vpip", line 11, in <module>
    load_entry_point('vpip==0.2.3', 'console_scripts', 'vpip')()
  File "/usr/lib/python3.7/site-packages/vpip/cli.py", line 57, in cli
    module.run(ns)
  File "/usr/lib/python3.7/site-packages/vpip/commands/install.py", line 33, in run
    install_global(ns.PACKAGE)
  File "/usr/lib/python3.7/site-packages/vpip/commands/install.py", line 59, in install_global
    pip_api.install(pkg, upgrade=upgrade, latest=latest)
  File "/usr/lib/python3.7/site-packages/vpip/pip_api.py", line 39, in install
    execute_pip("{} {}".format(cmd, package))
  File "/usr/lib/python3.7/site-packages/vpip/pip_api.py", line 119, in execute_pip
    return execute(prefix + cmd, capture)
  File "/usr/lib/python3.7/site-packages/vpip/execute.py", line 30, in execute
    list(do_execute())
  File "/usr/lib/python3.7/site-packages/vpip/execute.py", line 27, in do_execute
    raise subprocess.CalledProcessError(process.returncode, cmd)
subprocess.CalledProcessError: Command 'python -m pip install story' returned non-zero exit status 1.

ContextualVersionConflict error when building rpm

vpip fails due to needing an old setuptools

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (setuptools 41.0.1 (/home/python3.7/lib/python3.7/site-packages), Requirement.parse('setuptools~=40.7'), {'vpip'})

I guess setuptools>=40.7 would solve that, but the other pins also need to be re-checked.

Pre-install wheel and setuptools?

If setuptools/wheel is pinned in the lock file, we may get some errors: eight04/ComicCrawler#320 (comment)

Maybe they should stick with the platform just like pip.

pylint vs six breaking CI

build of current master https://travis-ci.org/jayvdb/vpip/builds/601646066

pip install emits and ignores error

astroid 2.3.2 has requirement six==1.12, but you'll have six 1.11.0 which is incompatible.

and then

$ python cute.py test
> Task: test
> Cmd: pylint vpip cute.py test.py
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/bin/pylint", line 10, in <module>
    sys.exit(run_pylint())
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/__init__.py", line 20, in run_pylint
    Run(sys.argv[1:])
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/lint.py", line 1608, in __init__
    linter.check(args)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/lint.py", line 938, in check
    self._do_check(files_or_modules)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/lint.py", line 1071, in _do_check
    self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/lint.py", line 1154, in check_astroid_module
    walker.walk(ast_node)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/utils.py", line 1269, in walk
    self.walk(child)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/utils.py", line 1266, in walk
    cb(astroid)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pylint/checkers/variables.py", line 1582, in visit_import
    module = next(node.infer_name_module(parts[0]))
AttributeError: 'Import' object has no attribute 'infer_name_module'
Command 'pylint vpip cute.py test.py' returned non-zero exit status 1.

Don't run "pip install -e ." when there is no setup.py?

I think there is a workflow without setup.py in recent versions of setuptools. Let's check the document.

pip will install packages to user site by default

pypa/pip#1668
pypa/pip#7002
Investigate how does it work with venv.

Run install_editable after updating a prod dependency

Otherwise, something like this will happen:

> Task: readme_build
> Cmd: python setup.py --long-description | x-pipe build/readme/index.rst
Traceback (most recent call last):
  File "D:\Dev\pyXcute\.venv\lib\site-packages\pkg_resources\__init__.py", line
583, in _build_master
    ws.require(__requires__)
  File "D:\Dev\pyXcute\.venv\lib\site-packages\pkg_resources\__init__.py", line
900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "D:\Dev\pyXcute\.venv\lib\site-packages\pkg_resources\__init__.py", line
791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (natsort 6.0.0 (d:\dev\pyxcute\.venv\li
b\site-packages), Requirement.parse('natsort<6,>=5.0.2'), {'pyxcute'})

Add lock file?

Every time pylint updates astroid, the CI build failed. It is pretty annoying.

If we use requirements.txt to pin all packages, where should we store dev dependencies?

The folder getter didn't check if the path is in the path/exists

vpip/vpip/venv.py

Line 37 in 02e2de9

return self.folders

We should score the folder and sort them:

1. If the folder exists, +1
2. If the folder is included in path, +1

Then remove folders with zero score.

Lowercase envvar path

The code uses lowercase envvar name path instead of PATH.

This results in

Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/site-packages/vpip/__main__.py", line 2, in <module>
    cli()
  File "/usr/lib/python3.7/site-packages/vpip/cli.py", line 57, in cli
    module.run(ns)
  File "/usr/lib/python3.7/site-packages/vpip/commands/list.py", line 20, in run
    print_local_packages(check_outdated=ns.outdated)
  File "/usr/lib/python3.7/site-packages/vpip/commands/list.py", line 42, in print_local_packages
    vv = venv.get_current_venv()
  File "/usr/lib/python3.7/site-packages/vpip/venv.py", line 50, in get_current_venv
    return Venv(".venv")
  File "/usr/lib/python3.7/site-packages/vpip/venv.py", line 98, in __init__
    get_path_without_venv(os.environ["path"], self.old_env_dir)
  File "/usr/lib64/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'path'

patch coming

Drop pkg_resources

pypa/setuptools#510 (comment)

Don't uninstall a package if it is a subdependency

Instead, just remove the dependency from requirements.txt or setup.cfg.

Uninstall sub-dependencies

https://github.com/cls1991/pef
https://github.com/naiquevin/pipdeptree
https://github.com/wimglenn/johnnydep

Some options:

1. Inject a script into venv to get all installed packages.
2. Read the site-packages folders from venv folder and check their deps.
3. Use the information from pip list --not-required.

Invalid version on paramiko

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\__main__.py", line 2, in <module>
    cli()
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\cli.py", line 54, in cli
    module.run(ns)
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\commands\outdated.py", line 19, in run
    list_.print_local_packages(check_outdated=True)
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\commands\list.py", line 69, in print_local_packages
    for info in get_infos(prod_requires):
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\commands\list.py", line 57, in get_infos
    if check_outdated and not info.check_update():
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\commands\list.py", line 103, in check_update
    self.update_result = pypi.check_update(self.name, self.version)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\vpip\pypi.py", line 39, in check_update
    all_versions = [parse_version(v) for v in r.json()["releases"].keys()]
                    ^^^^^^^^^^^^^^^^
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\packaging\version.py", line 54, in parse
    return Version(version)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\eight04\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\packaging\version.py", line 200, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '0.1-bulbasaur'

https://pypi.org/project/paramiko/0.1-bulbasaur/

How should we handle legacy version? Just ignore them?

Upgrading subdependencies may cause dependency conflicts?

Sphinx 4.5.0 requires docutils < 0.18. After running vpip update coverage twine, docutils also upgrades to 0.18.1.

Add "update-venv" command

1. Update home and version in pyvenv.cfg. (Does it work? Should we rebuild the entire venv?)
2. Update pip version in the venv.

UnicodeDecodeError while running vpip install -g poetry

Got the Following error while attempting an install of poetry on python 3.7.5 using vpip install -g poetry. Same install doesnt fail using pip. Thanks for the amazing library, I can give more details if necessary. I'm quite rusty in python and have to spend a little longer to find the guilty code.

Please see the full output bellow:

C:\Users\aperez>vpip install -g poetry
building venv at C:\Users\aperez\.vpip\pkg_venvs\poetry
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1
Collecting poetry
  Using cached https://files.pythonhosted.org/packages/c9/76/214acc400055aa0045ed21ce25378225b1887c5c091f99476955bf9f6984/poetry-0.12.17-py2.py3-none-any.whl
Collecting tomlkit<0.6.0,>=0.5.1
  Using cached https://files.pythonhosted.org/packages/3e/30/7c2693fc50bd466285ec22bf02ee344be1bde3e2e8267e302fdc82d11f2d/tomlkit-0.5.8-py2.py3-none-any.whl
Collecting jsonschema<4.0,>=3.0a3
  Using cached https://files.pythonhosted.org/packages/ce/6c/888d7c3c1fce3974c88a01a6bc553528c99d3586e098eee23e8383dd11c3/jsonschema-3.1.1-py2.py3-none-any.whl
Collecting pyparsing<3.0,>=2.2
  Using cached https://files.pythonhosted.org/packages/a3/c4/7828cf9e71ce8fbd43c1e502f3fdd0498f069fcf9d1c268205ce278ae201/pyparsing-2.4.4-py2.py3-none-any.whl
Collecting cachecontrol[filecache]<0.13.0,>=0.12.4
  Using cached https://files.pythonhosted.org/packages/5e/f0/2c193ed1f17c97ae539da7e1c2d48b80d8cccb1917163b26a91ca4355aa6/CacheControl-0.12.5.tar.gz
Collecting shellingham<2.0,>=1.1
  Using cached https://files.pythonhosted.org/packages/2d/ac/e8a34d4b3d24bf554f40651b2aac549a3fc7223725bf10fbdfe2083b6372/shellingham-1.3.1-py2.py3-none-any.whl
Collecting requests-toolbelt<0.9.0,>=0.8.0
  Using cached https://files.pythonhosted.org/packages/97/8a/d710f792d6f6ecc089c5e55b66e66c3f2f35516a1ede5a8f54c13350ffb0/requests_toolbelt-0.8.0-py2.py3-none-any.whl
Collecting html5lib<2.0,>=1.0
  Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
Collecting cachy<0.3,>=0.2
  Using cached https://files.pythonhosted.org/packages/26/37/8ce3e7b330078b6797a34e79a80a8ad6935e404a3b903765417182c9ce19/cachy-0.2.0-py2.py3-none-any.whl
Collecting cleo<0.7.0,>=0.6.7
  Using cached https://files.pythonhosted.org/packages/a7/b9/270301a3a87587f09bc3985973f2e362ffa45fa5fcd5128501516b2f5e31/cleo-0.6.8-py2.py3-none-any.whl
Collecting pyrsistent<0.15.0,>=0.14.2
  Using cached https://files.pythonhosted.org/packages/8c/46/4e93ab8a379d7efe93f20a0fb8a27bdfe88942cc954ab0210c3164e783e0/pyrsistent-0.14.11.tar.gz
Collecting requests<3.0,>=2.18
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting pkginfo<2.0,>=1.4
  Using cached https://files.pythonhosted.org/packages/e6/d5/451b913307b478c49eb29084916639dc53a88489b993530fed0a66bab8b9/pkginfo-1.5.0.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in c:\users\aperez\.vpip\pkg_venvs\poetry\lib\site-packages (from jsonschema<4.0,>=3.0a3->poetry) (41.2.0)
Collecting attrs>=17.4.0
  Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting importlib-metadata
  Using cached https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl
Collecting six>=1.11.0
  Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl
Collecting msgpack
  Downloading https://files.pythonhosted.org/packages/41/0a/49b522c5b23006d60669ec8dc97558e91880673e170108e0e9e21fc452e3/msgpack-0.6.2-cp37-cp37m-win_amd64.whl (68kB)
     |████████████████████████████████| 71kB 353kB/s
Collecting lockfile>=0.9
  Using cached https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl
Collecting webencodings
  Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
Collecting pastel<0.2.0,>=0.1.0
  Using cached https://files.pythonhosted.org/packages/09/b9/ca6434777928cc4e25cd17a8d2a913f3c2877ce54c5a7b7c18bf646ff257/pastel-0.1.1-py2.py3-none-any.whl
Collecting pylev<2.0,>=1.3
  Using cached https://files.pythonhosted.org/packages/40/1c/7dff1d242bf1e19f9c6202f0ba4e6fd18cc7ecb8bc85b17b2d16c806e228/pylev-1.3.0-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17
  Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Collecting zipp>=0.5
  Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
Collecting more-itertools
  Using cached https://files.pythonhosted.org/packages/45/dc/3241eef99eb45f1def35cf93af35d1cf9ef4c0991792583b8f33ea41b092/more_itertools-7.2.0-py3-none-any.whl
Installing collected packages: tomlkit, six, pyrsistent, attrs, more-itertools, zipp, importlib-metadata, jsonschema, pyparsing, urllib3, idna, chardet, certifi, requests, msgpack, lockfile, cachecontrol, shellingham, requests-toolbelt, webencodings, html5lib, cachy, pastel, pylev, cleo, pkginfo, poetry
    Running setup.py install for pyrsistent ... done
    Running setup.py install for cachecontrol ... done
Successfully installed attrs-19.3.0 cachecontrol-0.12.5 cachy-0.2.0 certifi-2019.9.11 chardet-3.0.4 cleo-0.6.8 html5lib-1.0.1 idna-2.8 importlib-metadata-0.23 jsonschema-3.1.1 lockfile-0.12.2 more-itertools-7.2.0 msgpack-0.6.2 pastel-0.1.1 pkginfo-1.5.0.1 poetry-0.12.17 pylev-1.3.0 pyparsing-2.4.4 pyrsistent-0.14.11 requests-2.22.0 requests-toolbelt-0.8.0 shellingham-1.3.1 six-1.13.0 tomlkit-0.5.8 urllib3-1.25.6 webencodings-0.5.1 zipp-0.6.0
Traceback (most recent call last):
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\aperez\AppData\Local\Programs\Python\Python37\Scripts\vpip.exe\__main__.py", line 9, in <module>
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\site-packages\vpip\cli.py", line 57, in cli
    module.run(ns)
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\site-packages\vpip\commands\install.py", line 33, in run
    install_global(ns.PACKAGE)
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\site-packages\vpip\commands\install.py", line 58, in install_global
    pip_api.install(pkg, upgrade=upgrade, latest=latest)
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\site-packages\vpip\pip_api.py", line 40, in install
    return show([require.name])[0]
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\site-packages\vpip\pip_api.py", line 76, in show
    for line in execute_pip("{} {}".format(cmd, " ".join(packages)), True):
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\site-packages\vpip\execute.py", line 24, in do_execute
    for line in process.stdout:
  File "c:\users\aperez\appdata\local\programs\python\python37\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 130: invalid continuation byte

Support github repo (and tag or branch?)

This will only work with -g flag. It will make it easier for users who want to test the latest code e.g. eight04/ComicCrawler#320.

Version is not always numbers

vpip/vpip/pypi.py

Line 77 in ad6bc73

return tuple(int(n) for n in str(version).split("."))

https://pypi.org/pypi/setuptools/json

link_console_script global install tries to write to /usr/bin

On Unix, global installs of console scripts attempt to write to /usr/bin.

> ~/.local/bin/vpip install -g story
building venv at /home/jayvdb/.vpip/pkg_venvs/story
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.3.1
...
link console script 'story'
Traceback (most recent call last):
  File "/home/jayvdb/.local/bin/vpip", line 11, in <module>
    load_entry_point('vpip', 'console_scripts', 'vpip')()
  File "/home/jayvdb/projects/python/vpip/vpip/cli.py", line 57, in cli
    module.run(ns)
  File "/home/jayvdb/projects/python/vpip/vpip/commands/install.py", line 33, in run
    install_global(ns.PACKAGE)
  File "/home/jayvdb/projects/python/vpip/vpip/commands/install.py", line 60, in install_global
    link_console_script(pkg, overwrite=True)
  File "/home/jayvdb/projects/python/vpip/vpip/commands/link.py", line 43, in link_console_script
    linker.dest.unlink()
  File "/usr/lib64/python3.7/pathlib.py", line 1287, in unlink
    self._accessor.unlink(self)
PermissionError: [Errno 13] Permission denied: '/usr/bin/story'

The script story is already installed in /home/jayvdb/.vpip/pkg_venvs/story/bin, and the next step would be to be create a symlink in /home/jayvdb/.local/bin/ , but instead it is writing to /usr/bin/ which isnt allowed.

install.py includes # TODO: make pip support install_scripts which may be referring to some part of this problem.

However the comment also mentions pypa/pip#3934 , which has a recent comment indicating it has been fixed indirectly, but not in a way that helps with a target-scripts-dir different from the target-dir.

I think it is a good idea to let the scripts be installed into the venv as normal, and then copy them to the GLOBAL_SCRIPT_FOLDER. And the global script directory for unix should be ~/.local/bin rather than /usr/bin

update_venv may break the environment if python version bumped?

Dependency-less vpip

pipx is running into a few issues wrt having dependencies, esp wrt Click. see pypa/pipx#233

It would be nice for vpip to aim to be an alternative that is dependency free, avoiding those problems, even if it comes at the cost of being less pretty/featured/etc.

Also it might want to do vendoring of any libraries it needs, to avoid those problems. Also aiming to be frozen e.g. pypa/pipx#244

Unpinned pylint errors

Unpinned pylint errors

> Cmd: pylint vpip cute.py test.py

************* Module vpip.commands.link

vpip/commands/link.py:5:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/link.py:11:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/link.py:21:4: C0415: Import outside toplevel (shutil) (import-outside-toplevel)

vpip/commands/link.py:22:4: C0415: Import outside toplevel (os) (import-outside-toplevel)

vpip/commands/link.py:23:4: C0415: Import outside toplevel (configparser) (import-outside-toplevel)

vpip/commands/link.py:24:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/link.py:51:8: C0415: Import outside toplevel (pathlib) (import-outside-toplevel)

vpip/commands/link.py:67:8: C0415: Import outside toplevel (pathlib) (import-outside-toplevel)

vpip/commands/link.py:72:8: C0415: Import outside toplevel (os) (import-outside-toplevel)

************* Module vpip.commands.uninstall

vpip/commands/uninstall.py:27:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/uninstall.py:38:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

************* Module vpip.commands.list

vpip/commands/list.py:27:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/list.py:40:4: C0415: Import outside toplevel (packaging.utils) (import-outside-toplevel)

vpip/commands/list.py:41:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/list.py:102:8: C0415: Import outside toplevel (%s) (import-outside-toplevel)

************* Module vpip.commands.update

vpip/commands/update.py:24:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/update.py:47:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

************* Module vpip.commands.install

vpip/commands/install.py:48:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/install.py:73:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/install.py:89:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

************* Module vpip.commands.run

vpip/commands/run.py:6:4: C0415: Import outside toplevel (subprocess) (import-outside-toplevel)

vpip/commands/run.py:7:4: C0415: Import outside toplevel (%s) (import-outside-toplevel)

vpip/commands/run.py:8:4: C0415: Import outside toplevel (execute) (import-outside-toplevel)

vpip/commands/run.py:25:12: R1722: Consider using sys.exit() (consider-using-sys-exit)

************* Module cute

cute.py:5:4: C0415: Import outside toplevel (livereload) (import-outside-toplevel)

************* Module test

test.py:2:4: C0415: Import outside toplevel (vpip.pypi) (import-outside-toplevel)

Command to update sub dependencies?

For example: certifi. Currently, to update sub-dependencies, you have to

vpip install -D certifi  # pin it to dev-deps
vpip update certifi --latest  # update

For global packages, you have to reinstall.

Try to use the executable directly when running scripts on Windows?

https://bugs.python.org/issue35217

Also, document the solution in the readme.