PeeringDB anonymous API throttling about arouteserver

[ccaputo]

• implement a retry strategy with an incremental back-off time in response to 426;

429

[grizz]

• add support for peeringdb-py

I'd be happy to help with that one! It would also be a good excuse to make the client use a local cache.

[pierky]

v1.15.1 is out, with a way to set the API key. Using authenticated API queries should reduce the impact of rate limiting.

This issue will remain opened until I’ll check how to implement bulk queries, which should lower the number of interactions between ARouteServer and the PeeringDB API.

[pierky]

I pushed some commits here to implement the bulk queries (150 ASNs being queried at once to get IRR info and max-prefix limits) and also to add the user-agent header.

The CI/CD pipeline is running for 1.16.0-alpha1, which includes these changes: once it will be completed, it will be possible to install and test out the pre-release following these instructions.

[pierky]

The (optional) environment variable SECRET_PEERINGDB_API_KEY can now be used to set the API key to be used for PeeringDB API queries. This should mitigate the effects of the anonymous API throttling mechanism.

The CI/CD pipeline is running for 1.15.1-alpha1: once it will be completed, it will be possible to install and test out the pre-release following these instructions.

The new env var will be shipped as part of 1.15.1.

[pierky]

Other options:

• implement a retry strategy with an incremental back-off time in response to 429; (done in 5461cdb)
• add support for peeringdb-py
• use bulk queries (asn__in=$list_of_ASN_separated_by_comma) (done in 3a2f827)

[pierky]

A retry-on-failure with back-off has been added, and also a way to pass the API key using a local file at a well-known path.

The CI/CD pipeline is running for 1.15.1-alpha2, which includes these changes: once it will be completed, it will be possible to install and test out the pre-release following these instructions.

[ccaputo]

If not using peeringdb-py, also needed:

• Add a delay in between queries that is randomly between 2 and 2.5 seconds, to reduce thundering herd. This delay will mean arouteserver queries PeeringDB at most 30 hits per minute.

[ccaputo]

Another thing that would be great:

• Set User-Agent to include "arouteserver" and the current version number. Right now it just shows up as, for example, python-requests/2.27.1. Would be better if it was arouteserver/x.y.z python-requests/a.b.c.

[pierky]

v1.16.0 is out with the improvements mentioned above.

I’ll file another ticket to analyze a potential integration with peeringdb-py, or to check whether it’s possible to add support for it.