johannestaas / evilpackages Goto Github PK

Hi,

As I mentioned on Reddit, I was going to take a look at this. It took longer than I thought. :) While the diff works well when I tested it using the included stats.json file, I ran into a pretty big issue with the stats subcommand.

As far as I can tell, the current implementation of evilpackages stats does not respect the posted etiquette for pypistats.org:

If you plan on using the API to download historical data for every python package in the database (e.g. for some personal data exploration), DON'T. This website runs on limited resources and you will degrade the site performance by doing this. It will also take a very long time.

Because of this, fetching the stats does not work on my machine. It gives me a JSON error, which I traced back to a 429 Too Many Requests error and found the etiquette section on the website. The 429 response apparently doesn't have a body GRequests can convert into JSON. I tried this a few times and apparently more than a few dozen requests are throttled. (Arguably I shouldn't have run the script once I knew it wasn't respecting the etiquette, but the rate throttling seemed aggressive enough that it wouldn't cause any problems, and I would have shut it off had it started to download more than a few hundred packages.)

The alternative suggested by both PyPi Stats and PyPi itself is to instead use BigQuery's free tier to access the full dataset. This may actually be simpler than the current code, but would require the user to provide Google credentials and a new import to interface with BigQuery. I'd be willing to make this change.