Interaction with tenacity retry about aiometer

[florimondmanca]

Hello :)

Don't really know, never tried using them in combination.

Do you have a working snippet of something you tried to do? Are you encountering blockers?

[Midnighter]

This doesn't work for me. Maybe you can see why. And by doesn't work I mean that it exits with an uncaught RetryError.

import asyncio
from io import StringIO
from typing import NamedTuple

import aiometer
import httpx
import pandas as pd
import tenacity
# import trio
from tenacity import stop_after_attempt, wait_random_exponential


class KEGGArguments(NamedTuple):

    client: httpx.AsyncClient
    identifier: str


async def fetch_kegg_list(
    client: httpx.AsyncClient,
    database: str,
) -> StringIO:
    """Fetch the tabular overview of a KEGG database."""
    text = StringIO()
    async with client.stream("GET", f"list/{database}") as response:
        async for chunk in response.aiter_text():
            text.write(chunk)
    # We set cursor to beginning such that the buffer can be read like a file.
    text.seek(0)
    return text


@tenacity.retry(stop=stop_after_attempt(5), wait=wait_random_exponential(max=30))
async def fetcher(args: KEGGArguments):
    response = await args.client.get(f"get/{args.identifier}")
    if response.status_code == 403:
        # Raise an exception here such that we get to retry.
        response.raise_for_status()
    return args.identifier, response.status_code


async def main(
    url: str = "http://rest.kegg.jp/",
    requests_per_second: int = 50,
):
    async with httpx.AsyncClient(
        base_url=url,
        pool_limits=httpx.Limits(
            max_keepalive_connections=requests_per_second,
            max_connections=requests_per_second * 2,
        ),
        timeout=None,
    ) as client:
        # async with trio.open_nursery():
        print("Fetch list")
        reactions = await fetch_kegg_list(client, "reaction")
        df = pd.read_csv(
            reactions,
            sep="\t",
            header=None,
            index_col=False,
            names=["id", "description"],
        )
        identifiers = [
            KEGGArguments(client=client, identifier=i)
            for i in df["id"].str[len("rn:") :].unique()
        ]
        print("Fetch reactions")
        async with aiometer.amap(
            fetcher, identifiers, max_per_second=requests_per_second
        ) as results_iter:
            async for rxn in results_iter:
                # async for rxn in tqdm(
                #     results_iter,
                #     total=len(identifiers),
                #     desc="Reaction",
                #     unit_scale=True,
                # ):
                print(rxn)


if __name__ == "__main__":
    # trio.run(main)
    asyncio.run(main())

[Midnighter]

Sorry, it may actually be working correctly... unfortunately, doesn't seem to recover once a retry was hit but that may be due to my application code.

[Midnighter]

Okay, it's a bit tricky. I have added logging to the retry:

@tenacity.retry(
    stop=stop_after_attempt(8),
    wait=wait_random_exponential(max=30),
    before_sleep=before_sleep_log(logger, logging.INFO),
)

which shows that as soon as one retry is attempted, another request is made by aiometer.amap. In my case, this is undesirable. As soon as I get a 403 response (this is the status code when I hit the rate limit), I would like to pause all requests and exponentially wait until I can make requests again. Then I would like to continue making requests with the given limit.

[Midnighter]

Closing this as I should find a solution on my end.