Comments (5)
What is happening? I'm guessing it's downloading lots of python3 only packages you don't want? Describing that is always an advantage when you open an issue to help people find the bug / config error. Attaching run logs is always super helpful too.
I always recommend using as little plugins as possible, as there is very little testing of applying them all together. Interoperability of all the filtering plugins has never been super tested and the apply order isn't documented / tested.
I feel here cause you have on setuptools in allowlist
is the only package you pull?
from bandersnatch.
so I removed all plugins and kept the following as you suggested
[mirror]
directory = /data/MINI_PYPI/bandersnatch/packages/
master = https://pypi.org
timeout = 20
workers = 3
hash-index = false
stop-on-error = false
json = true
[plugins]
enabled =
allowlist_project
blocklist_project
exclude_platform
[allowlist]
packages =
setuptools
[blocklist]
platforms =
windows
macos
freebsd
py3
py3.0
py3.1
py3.2
py3.3
py3.4
py3.5
py3.6
py3.7
py3.8
py3.9
py3.10
py3.11
I thought this would block the sync of all python3-compatible versions of setuptools. But it pulled all versions.
This is the log:
2023-07-12 09:22:55,420 INFO: No status file to move (/data/MINI_PYPI/bandersnatch/packages/status) - Full sync will occur (main.py:187)
2023-07-12 09:22:55,420 INFO: Selected storage backend: filesystem (configuration.py:131)
2023-07-12 09:22:55,420 INFO: Selected compare method: hash (configuration.py:179)
2023-07-12 09:22:55,431 INFO: Initialized project plugin allowlist_project, filtering ['setuptools'] (allowlist_name.py:33)
2023-07-12 09:22:55,432 INFO: Initialized project plugin blocklist_project, filtering [] (blocklist_name.py:27)
2023-07-12 09:22:55,446 INFO: Initialized exclude_platform plugin with ['.win32', '-win32', 'win_amd64', 'win-amd64', 'macosx_', 'macosx-', '.freebsd', '-freebsd', '-cp30-', '-pp30-', '-ip30-', '-jy30-', '-py3.0-', '-py3.0.', '-cp31-', '-pp31-', '-ip31-', '-jy31-', '-py3.1-', '-py3.1.', '-cp32-', '-pp32-', '-ip32-', '-jy32-', '-py3.2-', '-py3.2.', '-cp33-', '-pp33-', '-ip33-', '-jy33-', '-py3.3-', '-py3.3.', '-cp34-', '-pp34-', '-ip34-', '-jy34-', '-py3.4-', '-py3.4.', '-cp35-', '-pp35-', '-ip35-', '-jy35-', '-py3.5-', '-py3.5.', '-cp36-', '-pp36-', '-ip36-', '-jy36-', '-py3.6-', '-py3.6.', '-cp37-', '-pp37-', '-ip37-', '-jy37-', '-py3.7-', '-py3.7.', '-cp38-', '-pp38-', '-ip38-', '-jy38-', '-py3.8-', '-py3.8.', '-cp39-', '-pp39-', '-ip39-', '-jy39-', '-py3.9-', '-py3.9.', '-cp310-', '-pp310-', '-ip310-', '-jy310-', '-py3.10-', '-py3.10.'] (filename_name.py:108)
2023-07-12 09:22:55,449 INFO: Status file /data/MINI_PYPI/bandersnatch/packages/status missing. Starting over. (mirror.py:566)
2023-07-12 09:22:55,449 INFO: Syncing with https://pypi.org. (mirror.py:57)
2023-07-12 09:22:55,449 INFO: Current mirror serial: 0 (mirror.py:278)
2023-07-12 09:22:55,449 INFO: Syncing all packages. (mirror.py:293)
2023-07-12 09:23:10,527 INFO: Package 'setuptools' is allowlisted (allowlist_name.py:90)
2023-07-12 09:23:10,735 INFO: Trying to reach serial: 18856834 (mirror.py:310)
2023-07-12 09:23:10,735 INFO: 1 packages to sync. (mirror.py:312)
2023-07-12 09:23:10,745 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:76)
2023-07-12 09:23:10,746 INFO: No release filters are enabled. Skipping release filtering (mirror.py:78)
2023-07-12 09:23:10,746 INFO: Fetching metadata for package: setuptools (serial 18561969) (package.py:58)
To answer your question, for now, I only need setup-tools. My aim is to sync only needed packages. I don't want all of the packages present in my repo
from bandersnatch.
Thanks for the logs etc. - But in future please add --debug
to get more logging. So I was able to repro locally with your condif:
python3 -m venv --upgrade-deps /tmp/tb
/tmp/tb/bin/pip install -e .
vim /tmp/bandersnatch.conf
mkdir /tmp/pypi
/tmp/tb/bin/bandersnatch -c /tmp/bandersnatch.conf --debug mirror
Can confirm reading the code that all sdists are expected. But the -py3- wheels are not. So need to workout if the bug is:
- Caused by the
allowlist
being used withdenylist
causing the latter to be ignored - Is there a bug with the release filename plugin itself
- It does have tests tho so we could maybe use them, to test this theory
All help here is welcome. PRs welcome. I am down on time ...
from bandersnatch.
I believe I have fixed the bug. That said, you might want to just use version filtering for package in an allow list to just pull version that support python 2 as this method is still going to pull down all sdists (source distributions) of all new versions for the allow listed package.
For example:
[plugins]
enabled =
allowlist_release
[allowlist]
packages =
setuptools<=45.0.0
- I believe 45.0.0 was the last python 2 supported version of setuptools
https://bandersnatch.readthedocs.io/en/latest/filtering_configuration.html#packages
(I haven't tested this tho)
from bandersnatch.
I lie - CI does use this feature: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch/tests/ci.conf#L26
from bandersnatch.
Related Issues (20)
- Since the Bump s3path from 0.4.2 to 0.5.0 HOT 2
- package exist on pypi but not sync HOT 8
- Clean/verify packages in another way HOT 5
- Test bandersnatch in 3.12 + cut over docker HOT 1
- Move CI back to latest python once aiohttp supports 3.12 HOT 1
- Latest x releases: version sorting broken HOT 3
- Latest x packages breaks pinned version functionality HOT 1
- Issues serving via S3 static website HOT 5
- HTTP(S) Proxy Support HOT 10
- bandersnatch mirror completeness HOT 12
- bandersnatch repeatedly executes synchronously but keeps getting stuck on the same package(this time is mpf) HOT 6
- Incremental Synchronization Issue with Bandersnatch HOT 2
- Can I edit the file "todo"? I encountered a lot of "no longer exists on PyPI" HOT 2
- Add SOCKS support to proxy configuration parameter
- ImportError on S3DirEntry from 'mirror' operation HOT 2
- Enhance `bandersnatch mirror` to optionally delete packages detected as no longer found HOT 2
- Problems creating symlinks on Windows 10 HOT 4
- why are we replace the "." into "-",and how can i change it ? HOT 2
- configparser.NoOptionError: No option 'storage-backend' in section: 'mirror' HOT 2
- Help port bandersnatch to latest S3Path APIs + python 3.12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bandersnatch.