package exist on pypi but not sync about bandersnatch

[cooperlees]

as log below:

[root@softsource-agent-pypi-0 /data]# bandersnatch sync simple
2023-08-28 14:24:55,082 bandersnatch: INFO Selected storage backend: filesystem
2023-08-28 14:24:55,082 bandersnatch: INFO Selected compare method: stat
2023-08-28 14:24:55,082 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-08-28 14:24:55,236 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['pyagrum-nightly', 'tf-nightly', 'tf-nightly-cpu', 'tf-nightly-gpu', 'uselesscapitalquiz', 'appium', 'tensorflow-io-nightly']
2023-08-28 14:24:55,241 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-08-28 14:24:55,258 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('pyagrum-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('appium')>, <Requirement('uselesscapitalquiz')>, <Requirement('tf-nightly-gpu')>, <Requirement('tf-nightly')>, <Requirement('tensorflow-io-nightly')>]
2023-08-28 14:24:55,308 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-08-28 14:24:55,309 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-08-28 14:24:55,309 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-08-28 14:24:55,309 bandersnatch.package: INFO Fetching metadata for package: simple (serial 0)
2023-08-28 14:24:57,399 bandersnatch.package: INFO simple no longer exists on PyPI
2023-08-28 14:24:57,399 bandersnatch.simple: INFO Generating global index page.

however，"simple" package exist on pypi :https://pypi.org/project/simple/

Simple's Simple API page has no packages, so we can not mirror non existent packages: https://pypi.org/simple/simple/

It could be a bug with PyPI maybe?

[89ao]

same issue happens on flask-ipfilter,Prix-Carburant-FR-Client packages

[89ao]

I agree， seems that the "simple" package has a version 0.1.1 but doesn't have a package,interesting.

[89ao]

how about this one:adafruit-python-dht
https://pypi.org/project/Adafruit_Python_DHT/ http returns 200
yet "bandersnatch sync adafruit_python_dht" log below :

[root@softsource-agent-pypi-0 /data]# bandersnatch sync adafruit_python_dht
2023-08-31 10:27:19,106 bandersnatch: INFO Selected storage backend: filesystem
2023-08-31 10:27:19,107 bandersnatch: INFO Selected compare method: stat
2023-08-31 10:27:19,107 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-08-31 10:27:19,242 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['tf-nightly-cpu', 'appium', 'tf-nightly-gpu', 'tensorflow-io-nightly', 'uselesscapitalquiz', 'pyagrum-nightly', 'tf-nightly']
2023-08-31 10:27:19,246 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-08-31 10:27:19,260 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('appium')>, <Requirement('uselesscapitalquiz')>, <Requirement('tf-nightly-gpu')>, <Requirement('pyagrum-nightly')>, <Requirement('tf-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('tensorflow-io-nightly')>]
2023-08-31 10:27:19,307 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-08-31 10:27:19,307 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-08-31 10:27:19,307 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-08-31 10:27:19,307 bandersnatch.package: INFO Fetching metadata for package: adafruit-python-dht (serial 0)
**2023-08-31 10:27:20,931 bandersnatch.package: INFO adafruit-python-dht no longer exists on PyPI**
2023-08-31 10:27:20,931 bandersnatch.simple: INFO Generating global index page.

Seems like there's a bit of an issue with the conversion of the characters '-' and '_' ？

[89ao]

one more case :
https://pypi.org/project/thrift-sasl/
Pypi.org returns 200 yet bandersnatch cannot sync it locally.

[root@softsource-agent-pypi-0 /data]# bandersnatch sync thrift-sasl 
2023-09-05 20:06:52,749 bandersnatch: INFO Selected storage backend: filesystem
2023-09-05 20:06:52,750 bandersnatch: INFO Selected compare method: stat
2023-09-05 20:06:52,750 bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn
2023-09-05 20:06:52,929 bandersnatch: INFO Initialized project plugin blocklist_project, filtering ['tf-nightly', 'tensorflow-io-nightly', 'tf-nightly-cpu', 'appium', 'tf-nightly-gpu', 'uselesscapitalquiz', 'pyagrum-nightly']
2023-09-05 20:06:52,935 bandersnatch: INFO Initialized regex release plugin with [re.compile('.+-nightly.*')]
2023-09-05 20:06:52,955 bandersnatch: INFO Initialized release plugin blocklist_release, filtering [<Requirement('uselesscapitalquiz')>, <Requirement('appium')>, <Requirement('tf-nightly')>, <Requirement('tf-nightly-cpu')>, <Requirement('tensorflow-io-nightly')>, <Requirement('tf-nightly-gpu')>, <Requirement('pyagrum-nightly')>]
2023-09-05 20:06:53,011 bandersnatch.mirror: INFO Syncing with https://pypi.org/.
2023-09-05 20:06:53,012 bandersnatch.mirror: INFO No metadata filters are enabled. Skipping metadata filtering
2023-09-05 20:06:53,012 bandersnatch.mirror: INFO No release file filters are enabled. Skipping release file filtering
2023-09-05 20:06:53,012 bandersnatch.package: INFO Fetching metadata for package: thrift-sasl (serial 0)
===2023-09-05 20:06:54,631 bandersnatch.package: INFO thrift-sasl no longer exists on PyPI =====
...

[cooperlees]

how about this one:adafruit-python-dht https://pypi.org/project/Adafruit_Python_DHT/ http returns 200 yet "bandersnatch sync adafruit_python_dht" log below :

Seems like there's a bit of an issue with the conversion of the characters '-' and '_' ？

Worked for me installing bandersnatch from master and running on my laptop:

crl-m1:bandersnatch cooper$ /tmp/tb/bin/bandersnatch sync adafruit_python_dht
2023-09-05 21:21:11,292 INFO: Selected storage backend: filesystem (configuration.py:132)
2023-09-05 21:21:11,292 INFO: Selected compare method: hash (configuration.py:180)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/simple (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/packages (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/local-stats/days (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/json (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Setting up mirror directory: /tmp/pypi/web/pypi (mirror.py:505)
2023-09-05 21:21:11,323 INFO: Generation file missing. Reinitialising status files. (mirror.py:545)
2023-09-05 21:21:11,325 INFO: Status file /tmp/pypi/status missing. Starting over. (mirror.py:567)
2023-09-05 21:21:11,325 INFO: Syncing with https://pypi.org. (mirror.py:58)
2023-09-05 21:21:11,325 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:77)
2023-09-05 21:21:11,325 INFO: No release filters are enabled. Skipping release filtering (mirror.py:79)
2023-09-05 21:21:11,325 INFO: No release file filters are enabled. Skipping release file filtering (mirror.py:81)
2023-09-05 21:21:11,325 INFO: Fetching metadata for package: adafruit-python-dht (serial 0) (package.py:58)
2023-09-05 21:21:11,511 INFO: Downloading: https://files.pythonhosted.org/packages/2c/ed/482d4e0aaf7221996db3173044de4d09fc2c2a7e3589050ad09b6d24758c/Adafruit_Python_DHT-1.1.0.tar.gz (mirror.py:876)
2023-09-05 21:21:11,736 INFO: Downloading: https://files.pythonhosted.org/packages/09/99/fd16d79b5ae210aa10c96fa60aeb85a13b87bd8aafb2d7b4b7e0b365ad32/Adafruit_Python_DHT-1.1.1.tar.gz (mirror.py:876)
2023-09-05 21:21:11,767 INFO: Downloading: https://files.pythonhosted.org/packages/73/f3/b5e473c3d058870b04b401210139f9bdba85cc2dbfbfb2b57a6b6ffb2318/Adafruit_Python_DHT-1.1.2.tar.gz (mirror.py:876)
2023-09-05 21:21:11,859 INFO: Downloading: https://files.pythonhosted.org/packages/52/aa/418b82e36fc57316137679c2a12b8bae89ab73f211cbca91a3275e0b5fc2/Adafruit_Python_DHT-1.3.2.tar.gz (mirror.py:876)
2023-09-05 21:21:11,948 INFO: Downloading: https://files.pythonhosted.org/packages/b0/76/bdce34fd6e58cd91a6e4fc39a06a6fd362fd0cdec71fb77ad1c6a9f0c545/Adafruit_Python_DHT-1.3.3-py3.5-linux-x86_64.egg (mirror.py:876)
2023-09-05 21:21:12,006 INFO: Downloading: https://files.pythonhosted.org/packages/aa/9d/21e7f902e693c65a2a6c46ca327b8a15751fc04c3cf0ecb5e692540f6ab8/Adafruit_Python_DHT-1.3.3.tar.gz (mirror.py:876)
2023-09-05 21:21:12,096 INFO: Downloading: https://files.pythonhosted.org/packages/d6/85/af539216f80d10e917eca17c52085963765646a3500a9dd11ed8e9d9b7df/Adafruit_Python_DHT-1.3.4.tar.gz (mirror.py:876)
2023-09-05 21:21:12,185 INFO: Downloading: https://files.pythonhosted.org/packages/e9/e5/8261ce3480ad08567123286501628bfe1452b13bc4f20e98b46e456b060b/Adafruit_Python_DHT-1.4.0.tar.gz (mirror.py:876)
2023-09-05 21:21:12,279 INFO: Storing index page(s): adafruit-python-dht - in /tmp/pypi/web/simple/a/adafruit-python-dht (mirror.py:699)
2023-09-05 21:21:12,282 INFO: Generating global index page. (simple.py:259)
2023-09-05 21:21:12,385 INFO: 1 packages had changes (mirror.py:991)
2023-09-05 21:21:12,386 INFO: Writing diff file to mirrored-files (mirror.py:1001)

So did thrift-sasl:

crl-m1:bandersnatch cooper$ /tmp/tb/bin/bandersnatch sync thrift-sasl
2023-09-05 21:24:23,109 INFO: Selected storage backend: filesystem (configuration.py:132)
2023-09-05 21:24:23,110 INFO: Selected compare method: hash (configuration.py:180)
2023-09-05 21:24:23,144 ERROR: Removing inconsistent todo list. (mirror.py:401)
2023-09-05 21:24:23,144 INFO: Removing file: /tmp/pypi/todo (filesystem.py:188)
2023-09-05 21:24:23,145 INFO: Status file /tmp/pypi/status missing. Starting over. (mirror.py:567)
2023-09-05 21:24:23,145 INFO: Syncing with https://pypi.org. (mirror.py:58)
2023-09-05 21:24:23,145 INFO: No metadata filters are enabled. Skipping metadata filtering (mirror.py:77)
2023-09-05 21:24:23,145 INFO: No release filters are enabled. Skipping release filtering (mirror.py:79)
2023-09-05 21:24:23,145 INFO: No release file filters are enabled. Skipping release file filtering (mirror.py:81)
2023-09-05 21:24:23,145 INFO: Fetching metadata for package: thrift-sasl (serial 0) (package.py:58)
2023-09-05 21:24:23,319 INFO: Downloading: https://files.pythonhosted.org/packages/ca/24/120d2f48efc65204d859081e4eb0a18695e7531c4575bb136a25746a1381/thrift_sasl-0.1.0-py2.7.egg (mirror.py:876)
2023-09-05 21:24:23,453 INFO: Downloading: https://files.pythonhosted.org/packages/5b/e4/371dbc680035ec788d40216b9daf0526a79b329070ba7272d4e3fc4d2c4e/thrift_sasl-0.1.0.tar.gz (mirror.py:876)
2023-09-05 21:24:23,550 INFO: Downloading: https://files.pythonhosted.org/packages/0f/d6/ac7fcc6dbc1475aa3b23dbee126788d59b487054e3f458c446f3ce15d1d7/thrift_sasl-0.2.0.tar.gz (mirror.py:876)
2023-09-05 21:24:23,641 INFO: Downloading: https://files.pythonhosted.org/packages/80/36/16dfe92d32df63cc2b7b7be8d0e4a736617b7e52daaa7f83ae386a89d179/thrift_sasl-0.2.1.tar.gz (mirror.py:876)
2023-09-05 21:24:23,676 INFO: Downloading: https://files.pythonhosted.org/packages/50/fe/89cbc910809e3757c762f56ee190ca39e0f28b7ea451835232c0c988d706/thrift_sasl-0.3.0.tar.gz (mirror.py:876)
2023-09-05 21:24:23,702 INFO: Downloading: https://files.pythonhosted.org/packages/1b/df/a744ed6929808c52d1ee0648557f1a1430400d364f0ffdd9ea28d7127579/thrift_sasl-0.4.1.tar.gz (mirror.py:876)
2023-09-05 21:24:23,729 INFO: Downloading: https://files.pythonhosted.org/packages/73/d3/588654faef5511afadc1a091d32fcdbb24ae5f2d90b380874aee68a717f9/thrift_sasl-0.4.2.tar.gz (mirror.py:876)
2023-09-05 21:24:23,756 INFO: Downloading: https://files.pythonhosted.org/packages/c3/9e/636c24ce1c0d46ce3020c5836c5a375d8e862fa81a240e0e352cc991dcf8/thrift_sasl-0.4.3-py2.py3-none-any.whl (mirror.py:876)
2023-09-05 21:24:23,785 INFO: Downloading: https://files.pythonhosted.org/packages/fa/fb/5d9ba73be60b32f6fda080a5556cff25e15daf33bafe043c226404b39b25/thrift_sasl-0.4.3.tar.gz (mirror.py:876)
2023-09-05 21:24:23,812 INFO: Downloading: https://files.pythonhosted.org/packages/f2/08/912c32ecbe6e5b8ed4eec93bee3e0b9cce8ca335576194bbd07c1931be46/thrift_sasl-0.4.3a1-py2.py3-none-any.whl (mirror.py:876)
2023-09-05 21:24:23,908 INFO: Downloading: https://files.pythonhosted.org/packages/42/cb/ee76ab613e3b4e01dfb6044558b8e9838f9f97e0a37cc2e0ecffc6a7c606/thrift_sasl-0.4.3a1.tar.gz (mirror.py:876)
2023-09-05 21:24:23,996 INFO: Downloading: https://files.pythonhosted.org/packages/d0/0a/92e961e6a30222ef1775d15a0c11f89309101f6e98e4ba709462760f8a40/thrift_sasl-0.4.3a2-py2.py3-none-any.whl (mirror.py:876)
2023-09-05 21:24:24,033 INFO: Downloading: https://files.pythonhosted.org/packages/39/ac/713d03a8d6d73e5d35e7ae06ad8e10543c7adb64329eee8b1570af38f583/thrift_sasl-0.4.3a2.tar.gz (mirror.py:876)
2023-09-05 21:24:24,126 INFO: Downloading: https://files.pythonhosted.org/packages/61/fd/a24996119bd346ca06a182135c1894609ccb5d35f5e0e81be83877fc4005/thrift_sasl-0.4a1.tar.gz (mirror.py:876)
2023-09-05 21:24:24,226 INFO: Storing index page(s): thrift-sasl - in /tmp/pypi/web/simple/t/thrift-sasl (mirror.py:699)
2023-09-05 21:24:24,230 INFO: Generating global index page. (simple.py:259)
2023-09-05 21:24:24,333 INFO: 1 packages had changes (mirror.py:991)
2023-09-05 21:24:24,333 INFO: Writing diff file to mirrored-files (mirror.py:1001)

The config was:

crl-m1:bandersnatch cooper$ cat /etc/bandersnatch.conf
[mirror]
directory = /tmp/pypi
json = true
cleanup = true
master = https://pypi.org
timeout = 60
global-timeout = 18000
workers = 3
hash-index = true
stop-on-error = true
storage-backend = filesystem
verifiers = 3
keep_index_versions = 2
compare-method = hash

So don't know what to fix here sorry ...

[89ao]

tks @cooperlees ，maybe I need to update to a new version and have a try!

[cooperlees]

Looking again at your logs, either the alternative mirror you're using is broken or we have a bug when using an alternative mirror:

bandersnatch: INFO Selected alternative download mirror https://mirrors.tuna.tsinghua.edu.cn