Comments (3)
Howdy,
This isn't really a bandersnatch question. This is all a limitation of lots of small files on your storage backend.
The only ideas we could possibly try:
- Use the JSON metadata in parallel and check if a simple dir exists and if so just sum up all the packages
- Many bugs, but if you use filtering, that won't be applied
- Use the JSON metadata in parallel and check if the files exist, but I think this will be just as expensive as
du
(but not sure all the operations du does under the covers)
Another hack I've generally recommended is making a dedicated partition or volume for each part of bandersnatch's storage - e.g. simple and packages directories to be in their own filesystems and then df -h
can give quicker insight too.
- If you use
hash-index = true
you could also create a volume/file system per shard to get further insight
I don't have the cycles to look into these ideas, but would take a PR add docs or a bandersnatch du
like command that works out the sizes quicker if possible. But I feel we'd need to use a lower level language than python to get true speed here. Will leave open incase someone smarter comes along with better ideas.
from bandersnatch.
Thank you for your reply, and look forward to adding the du parameter to bandersmatch.
from bandersnatch.
Awesome. Yeah I’ll be surprised if it’s much faster and will be hard to get accurate without checking if the files exist, which is the expensive part. It might surprise us and be much quicker than du …
from bandersnatch.
Related Issues (20)
- High memory usage for `verify --delete` due to deletion occurring at end HOT 3
- No sync of new packages when the package list changes after the first sync HOT 3
- bandersnatch mirror cannot get update data HOT 4
- Update bandersnatch to latest packaging (22.0) to unblock tox 4.0
- Make CI pass in 3.11 + Docker build in 3.11 HOT 2
- Stop issuing PURGE requests on stale serial as PyPI requires authentication now
- heavy io doing glob('**') in /web/simple for global index HOT 4
- bandersnatch.master.XmlRpcError: Unable to get full list of packages HOT 1
- Will bandersnatch delete packages that are not on pypi.org but exist locally when synchronizing? HOT 1
- Exception with UTF-16 encoded requirements.txt file HOT 3
- stale serial errors blocking synchronization HOT 1
- Improve mirror filtering for latest releases HOT 2
- Add TODO file cleanup to avoid a single package blocking the entire sync process HOT 5
- package json digest dict mapped to simple json hashes dict causes pip >23 to fail HOT 3
- Generate SimpleDigests object from what metadata offers
- Bandersnatch mirror do not mirror the dependencies for the packages mentioned on allow list ? HOT 2
- Ensure bandersnatch implements pep700 fields
- bandersnatch verify --delete occurs error HOT 2
- Sync only python 2 packages HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bandersnatch.