Make Whitelist Optionally Resolve Dependencies about bandersnatch

[GreatBahram]

Things that I think maybe useful to achieve this feature. At the /srv/pypi/web/json file you can find the list of package dependencies, for example bandersnatch['info']['requires_dist'] it'll show these packages:

['aiohttp', 'filelock', 'packaging', 'requests', 'setuptools', 'xmlrpc2']

These data can be retrieve online throughf'https://pypi.org/pypi/{package_name}/json' address. whitelist plugin must do more things than blacklist, I mean after getting package names from config file it should calculates the package dependencies then starts to download them.

I couldn't find out how you really used whitelist I've just seen they're live in completely separated directory and it seems at the begging when bandersnatch runs you just get a list of filtered packages. for whitelist I think that should be different allow the program to get the pacakge's data (f'https://pypi.org/pypi/{package_name}/json') and after calculation of dependencies stuffs download all of them(?).

[cooperlees]

I’ve tried to keep bandersnatch as simple as possible, this is the main reason what you’re proposing is not in bandersnatch. Adding dependencies is not going to be as easy as you think, thus why it was left out.

Each version of a package can have different dependencies (so you’d have to lookup every version on pypi to “be sure”), and some people also miss deps in their install_requires in their setup.py, only using requirements.txt which means no deps are in the JSON.

I feel this will be a lot of work but if you want to give it a go I won’t stop you. I’d like the following:

1. option for white-list-dep-finding
2. a dep finder module (file) that uses an already existing dependency resolver:

• I’d suggest looking at ‘poetry’ or ‘piplock’ to do the dependency resolution.

[GreatBahram]

I know dependencies are a mess in python and checking each version make it more harder, but currently I need that option, So I'm working on it. Thanks for suggestions

[GreatBahram]

Currently, there are two projects that we can use their code for this purpose. First is pip-tools and second one is JohnnyDep!. Among them I found pip-tools is much more faster and reliable and it's just need some time to change their code in order to use it as a library.
There are some caveats about how to use this library with bandersnatch. Consider requests package for example, when we're resolving this package's dependencies at the end we have a list of packages with their specific version and bandersnatch is going to get all version of those packages and I think that's going to be useless. because requests only needs urllib==2.0.0 and getting all version of urllib does make sense, right?
I'm not sure about how we can merge this new library with bandersnatch, I mean at the end we should change the bandersnatch in whitelist option to only gets those specific packages. To me, it's OK we're mirroring like always but this time we're mirroring a limited packages.

[cooperlees]

Can we keep the logic simple and if a package is a dependency we just mirror ALL versions please.

I don’t think the complication of the code and chance for bugs worth it. We are a tool to mirror PyPI, not be a perfect dep resolver.

[GreatBahram]

OK, so in this case there is no need to change bandersnatch.

Thanks

[cooperlees]

Closing as I think due to the complexity @GreatBahram has decided to not go forward with this and I don't think this is of huge value worth the effort it would take. Please reopen if this is not the case.

[GreatBahram]

Hi,
Sorry for the delay, I've implemented it but it wasn't satisfying. Firstly, as I talked to you about it, each version of python packages may use different dependencies so I think it would be completely nonsense to clone all version of those dependencies. Besides, that's not the goal of bandersnatch project. Overall, you're right.

Then, I've tried to solve this problem with devpi project, it was much more efficient in terms of disk space and network usage. The only thing you need to do in devpi is to ask devpi-client to install all versions of that specific package you want.

Thanks @cooperlees