ido50 / morgan Goto Github PK

PyPI Mirror for Restricted/Offline Environments

License: Apache License 2.0

Python 100.00%

morgan's Introduction

Ido Perlmuter

Hi there. I am Ido Perlmuter, also known by the ridiculous net handle "ido50". I am a Software Engineer (and former Mechanical Engineer) with over two decades of experience.

I run my own business called Shift-Delete, which provides software development, training and consultancy services to both early-stage startups and established corporations.

My personal blog is at ido50.net.

Maintained Free Software Projects

Name	Written In	Type	Description
morgan	Python 3	exe/lib	PyPI mirror for restricted networks
sheepwool	C, Lua	exe	Semi-static site generator/server
sqlz	Go	lib	SQL query builder
requests	Go	lib	API-centric HTTP client
aiac	Go	exe	AI-generated IaC files
lmdrouter	Go	lib	HTTP router for AWS Lambda functions
mccs	Perl	exe/lib	Fully-featured static file server
esquery	Go	lib	Query builder for ElasticSearch
WWW::Postmark	Perl	lib	Client library for postmarkapp.com email service
Svsh	Perl	exe	Process supervision shell
Brannigan	Perl	lib	Input validation library
Locale::Wolowitz	Perl	lib	Easy, JSON-based l10n/i18n for Perl apps
Tenjin	Perl	lib	Template engine with support for embedded Perl
MQUL	Perl	lib	General purpose impl. of MongoDB's query lang.
Abilities	Perl	lib	Extensible, flexible authorization library

morgan's People

Contributors

Stargazers

Watchers

Forkers

yozh2 i077 shamilbi

morgan's Issues

Multiple Versions of a Single Package

Unless I'm missing something I don't see an obvious way of pulling down multiple versions of a single package. For example, opencv-python requires a certain version of numpy for python 3.8 and another for 3.9 and since morgan pulls the latest of all dependencies the install breaks.

Using:
numpy = ==1.17.3;python_version < '3.9'

works, but I can't say x for python 39 and y for python > 310.

What would be great is to allow either

(a) another line that says (requires at the least a change to the configparser/duplicateoptionerror):
numpy = ==1.19.3;python_version < '3.10'

or
(b) (arguably better/simpler) be able to say
numpy = ==1.17.3 && ==1.19.3 && >=1.20.0
which will pull down all three versions

Have you considered allowing this? or is it already possible? I'd be happy to try to work on it, if you have an idea of the best way to approach allowing multiple versions.

selenium: fetches wrong last version

morgan fetches selenium-2.0-dev-9429.tar.gz instead of selenium-4.18.1.tar.gz

morgan.ini:

[env.None]
os_name = posix
sys_platform = linux
platform_machine = x86_64
platform_python_implementation = CPython
platform_system = Linux
python_version = 3.11
python_full_version = 3.11.5
implementation_name = cpython

[requirements]
selenium =

rm -rf selenium
morgan
ls selenium
selenium-2.0-dev-9429.tar.gz
selenium-2.0-dev-9429.tar.gz.hash
selenium-2.0-dev-9429.tar.gz.metadata

but the last version of selenium is 4.18.1
https://pypi.org/project/selenium/

json changes? size limits? pypi package json incomplete

Our installation was working well until a couple months ago? when we stopped being able to download past a certain version of x package - mostly from more popular packages. I think I've traced the issue down to incomplete json being downloaded, possibly related to this ( https://discuss.python.org/t/backwards-incompatible-change-to-pypi-json-api/17154 )

Right now if I have in [requirements]
numpy = ==1.17.3
This works fine

But as of a few weeks/months ago?
numpy = ==1.26.0, for example, or anything above 1.19.5 errors out with "No files match requirement".

(this also happens with transformers and a handful of other packages that were working fine up until a certain point)

In examining the json from https://pypi.org/pypi/numpy/json for example, it seems to have 1.26.1. It's not invalid, so I don't think it's a string/request size limit.

I'm still trying to debug this, but I'm wondering if you have any input on this issue?

v0.13.2 error: packaging.specifiers.InvalidSpecifier

morgan-0.13.2, python-3.11, linux-x86_64

morgan.ini:
[requirements]
flit =

morgan mirror

output (only in v0.13.2):

flit
Traceback (most recent call last):
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/packaging/specifiers.py", line 634, in __init__
    parsed.add(Specifier(specifier))
               ^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/apps/morgan/.venv/bin/morgan", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 472, in main
    mirror(args.index_path)
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 435, in mirror
    m.mirror("{}{}".format(package, m.config["requirements"][package]))
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 74, in mirror
    deps = self._mirror(requirement)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 150, in _mirror
    files = self._filter_files(requirement, files)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 232, in _filter_files
    files = list(filter(
            ^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 233, in <lambda>
    lambda file: self._matches_environments(file), files))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 250, in _matches_environments
    spec_set = packaging.specifiers.SpecifierSet(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/packaging/specifiers.py", line 636, in __init__
    parsed.add(LegacySpecifier(specifier))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/packaging/specifiers.py", line 253, in __init__
    super().__init__(spec, prereleases)
  File "~/apps/morgan-0.13.2/.venv/lib64/python3.11/site-packages/packaging/specifiers.py", line 98, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
packaging.specifiers.InvalidSpecifier: Invalid specifier: '3'

feature request: morgan.ini multiline requirement values

To be able to download several versions of one library, allow multiline requirement values:

urllib3 =
        <1.27
        >=2
        [brotli,socks,zstd]

v0.13.3 error: IndexError: list index out of range

morgan-0.13.3, python-3.11, linux-x86_64

morgan.ini:

[requirements]
pywinauto = <1

[env.None]
os_name = posix
sys_platform = linux
platform_machine = x86_64
platform_python_implementation = CPython
platform_system = Linux
python_version = 3.11
python_full_version = 3.11.5
implementation_name = cpython

morgan mirror

output:

pywinauto<1
[pywinauto<1]: six
[pywinauto<1]: pywin32
Skipping pywin32, no file matches environments
Traceback (most recent call last):
  File "~/apps/morgan/.venv/bin/morgan", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "~/apps/morgan-0.13.3/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 483, in main
    mirror(args.index_path)
  File "~/apps/morgan-0.13.3/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 446, in mirror
    m.mirror("{}{}".format(package, m.config["requirements"][package]))
  File "~/apps/morgan-0.13.3/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 81, in mirror
    more_deps = self._mirror(
                ^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.3/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 150, in _mirror
    files = self._filter_files(requirement, files)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-0.13.3/.venv/lib64/python3.11/site-packages/morgan/__init__.py", line 240, in _filter_files
    latest_version = files[0]["version"]
                     ~~~~~^^^
IndexError: list index out of range

Can't mirror tensorflow properly for 3.11 when 3.12 is also configured

If I try to mirror tensorflow with python environments for both 3.11 and 3.12 then it won't grab tensorflow-io-gcs-filesystem which is required for python <3.12. From what I can gather the trouble is that there's no version of tensorflow-io-gcs-filesystem for 3.12 and so morgan gives up and decides not to get it. Not exactly sure why that's not fatal, but more to the point morgan should be handling the different environments somewhat more independently. Environment a needs package x, then get a version of package x that fulfills the requirement for environment a. Don't care if it doesn't work for environment b, you still need it for environment a. If you want to get fancy it ends up being a packing problem. I would suggest just doing the trivial "find a version that works for the environment I need it in" and don't worry about trying to reduce duplication.

windows files?

Hello,

First, thanks for the work on this - we are trying to get it going into a closed off PHI environment. I have it working from the linux side of things, but I was trying to both install and test the ability of windows machines to use the mirror as well (not sure if this is possible, but it seemed so).

I tried the following ini:
[env.Windows]
os_name = nt
sys_platform = windows (also tried "win32")
platform_machine = AMD64 (also tried "win_amd64")
platform_system = Windows
... then all the same as linux python_version = 3.10, etc.

I then tried to pull down a package on linux (since I don't have a good way to test this from windows locally) using:

pip3 install pandas --platform win32 --only-binary=:all: --trusted-host 0.0.0.0 --target=./windows

I always get a no matching distribution could be found. Whereas pointing this to pypi:
pip3 install pandas --platform win32 --only-binary=:all: --target=./windows
works fine.

Any obvious things I am missing to get the windows side of this working?

Inconsistent name when installing from local PyPi mirror

Steps to reproduce:

Use the following morgan.ini:

[env.None]
os_name = posix
sys_platform = darwin
platform_machine = arm64
platform_python_implementation = CPython
platform_system = Darwin
python_version = 3.10
python_full_version = 3.10.9
implementation_name = cpython
[requirements]
fastapi= ==0.88.0

Run morgan mirror
Run morgan serve
Try to install the package pip install FastAPI --index-url http://localhost:8080

Expected: the package installed successfully.
What happens instead: failure

Looking in indexes: http://localhost:8080
Collecting FastAPI
  Obtaining dependency information for FastAPI from http://localhost:8080/fastapi/fastapi-0.88.0-py3-none-any.whl.metadata
  Downloading http://localhost:8080/fastapi/fastapi-0.88.0-py3-none-any.whl.metadata
     - 24.7 kB 241.5 MB/s 0:00:00
Discarding http://localhost:8080/fastapi/fastapi-0.88.0-py3-none-any.whl#sha256=263b718bb384422fe3d042ffc9a0c8dece5e034ab6586ff034f6b4b1667c3eee (from http://localhost:8080/fastapi/): Requested FastAPI from http://localhost:8080/fastapi/fastapi-0.88.0-py3-none-any.whl#sha256=263b718bb384422fe3d042ffc9a0c8dece5e034ab6586ff034f6b4b1667c3eee has inconsistent Name: expected 'FastAPI', but metadata has 'fastapi'
  Obtaining dependency information for FastAPI from http://localhost:8080/fastapi/fastapi-0.88.0.tar.gz.metadata
  Downloading http://localhost:8080/fastapi/fastapi-0.88.0.tar.gz.metadata
     - 24.7 kB 249.5 MB/s 0:00:00
Discarding http://localhost:8080/fastapi/fastapi-0.88.0.tar.gz#sha256=915bf304180a0e7c5605ec81097b7d4cd8826ff87a02bb198e336fb9f3b5ff02 (from http://localhost:8080/fastapi/): Requested FastAPI from http://localhost:8080/fastapi/fastapi-0.88.0.tar.gz#sha256=915bf304180a0e7c5605ec81097b7d4cd8826ff87a02bb198e336fb9f3b5ff02 has inconsistent Name: expected 'FastAPI', but metadata has 'fastapi'
ERROR: Could not find a version that satisfies the requirement FastAPI (from versions: 0.88.0)
ERROR: No matching distribution found for FastAPI

http error 404

morgan.ini

[env.None]
os_name = posix
sys_platform = linux
platform_machine = x86_64
platform_python_implementation = CPython
platform_system = Linux
python_version = 3.11
python_full_version = 3.11.9
implementation_name = cpython

[requirements]
fail2ban =

morgan mirror
fail2ban
Traceback (most recent call last):
  File "~/apps/morgan/.venv/bin/morgan", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "~/apps/morgan-http404/.venv/lib64/python3.12/site-packages/morgan/__init__.py", line 484, in main
    mirror(args.index_path)
  File "~/apps/morgan-http404/.venv/lib64/python3.12/site-packages/morgan/__init__.py", line 447, in mirror
    m.mirror("{}{}".format(package, m.config["requirements"][package]))
  File "~/apps/morgan-http404/.venv/lib64/python3.12/site-packages/morgan/__init__.py", line 74, in mirror
    deps = self._mirror(requirement)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/apps/morgan-http404/.venv/lib64/python3.12/site-packages/morgan/__init__.py", line 132, in _mirror
    with urllib.request.urlopen(request) as response:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/urllib/request.py", line 215, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/urllib/request.py", line 521, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/urllib/request.py", line 630, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/urllib/request.py", line 559, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib64/python3.12/urllib/request.py", line 639, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

multiple morgan.ini files?

I am utilizing this tool in my company for my team to centrally pull many packages into our locked down environment, Since every project has their own set of python version, environment, and packages, a single morgan.ini isn't enough. My current workaround is to overwrite morgan.ini content with morgan-[project].ini before running morgan mirror and loop for every .ini files.

Do you have any plan to support --config or --outputDir argument so it can build a library of packages for many projects instead of just one, or am I missing the docs?

Unsupported metadata version 1.1, only support 1.0

Slackware Linux x86_64, python 3.10.10, pip-23.0.1, setuptools-67.4.0

morgan.ini:
wheel =

output:
wheel
Traceback (most recent call last):
File "apps/morgan/.venv/bin/morgan", line 8, in
sys.exit(main())
File "apps/morgan_/.venv/lib64/python3.10/site-packages/morgan/init.py", line 458, in main
mirror(args.index_path)
File "apps/morgan_/.venv/lib64/python3.10/site-packages/morgan/init.py", line 421, in mirror
m.mirror("{}{}".format(package, m.config["requirements"][package]))
File "apps/morgan_/.venv/lib64/python3.10/site-packages/morgan/init.py", line 74, in mirror
deps = self.mirror(requirement)
File "apps/morgan/.venv/lib64/python3.10/site-packages/morgan/init.py", line 136, in _mirror
raise Exception(
Exception: Unsupported metadata version 1.1, only support 1.0

morgan missing optional requirements when mirroring jupyterlab

Morgan version 0.13.4

Mirroring jupyterlab ==4.0.9 misses the fqdn requirement of the format-nongpl optional-dependency of jsonschema.
This dependency is pulled in by jupyter_events which chains up by jupyter_server > jupyterlab_server > jupyterlab.

From pyproject.toml of jupyter-events (which is included in the mirror index):

...
[project]
...
dependencies = [
    "referencing",
    "jsonschema[format-nongpl]>=4.18.0",
    "python-json-logger>=2.0.4",
    "pyyaml>=5.3",
    "traitlets>=5.3",
    # The following are necessary to address an issue where pyproject.toml normalizes extra dependencies
    # such that 'format_nongpl' is normalized to 'format-nongpl' which prevents these two validators from
    # from being installed when jsonschema is <= 4.9 because jsonschema uses 'format_nongpl' in those releases.
    "rfc3339-validator",
    "rfc3986-validator>=0.1.1",
]
...

From pyproject.toml of jsonschema (which is also included in the mirror index):

...
[project.optional-dependencies]
...
format-nongpl = [
  "fqdn",
  "idna",
  "isoduration",
  "jsonpointer>1.13",
  "rfc3339-validator",
  "rfc3986-validator>0.1.0",
  "uri_template",
  "webcolors>=1.11",
]
...

fqdn is NOT included in the mirror index (hence the problem) and causes a pip install against the mirror to fail for jupyterlab (or simpler and smaller for jupyter-event).

Beyond what I've described I don't understand what's going on (how morgan calculates dependencies), but I'm happy to help debug.

morgan.ini: "cffi = >=1.15.1" doesn't work, but "cffi = ==1.15.1" does

morgan-0.12.3, python-3.10, linux-x86_64
morgan mirror