processing urls in batches about wg-async

[nikomatsakis]

The basic format is that the host asks you (and others) a bunch of questions about your experiences and then we try to collectively write a story about one of the characters. It's fun. :)

[nikomatsakis]

Can you say a bit more, @rgreinho? What caused you not to succeed, for example?

[rgreinho]

We're still investigating, so hopefully we will find solutions soon :)

But here are our two big blockers so far:

1. Not sure how to limit the number of requests sent at the same time asynchronously. In Python the
   aiohttp web client can be configured with a smaller connection pool size and handles it for us. We hoped that reqwest would do the same. Note that we're not specifically attached to reqwest and could use surf or hreq, but unfortunately we found out that they behave the same way.

   Our second hope was that the FuturesUnordered container would allow us to manage this.

   A comment in the SO question pointed us to this question, where they create a Stream and apply the buffer_unordered() method on it. That will be our next attempt.

   Coming from a Python background, we hoped we could simply use the same pattern, where the asyncio.gather() function from the stdlib executes the coroutines and collects the results, that's why we went with the FuturesUnordered container first.

2. We do not know how to retry failed requests. We did find the backoff crate and the tokio-retry one, but they don't seem to work well with FuturesUnordered. Or at least we did not succeed to get them to work together.

   In Python we use tenacity to decorate our functions, and if an exception is caught, it tries to re run them for us.

We are also having problems with the error handling. We could not get the map_err() and or_else() method to work as expected. But this is probably simply do to the fact that we're new to this and did not use them properly. I'm sure we will figure it out soon. Same thing with the logging, the tracing-rs library looks fantastic.

[nikomatsakis]

@rgreinho any chance you want to join a "vision doc writing session" and talk about this? What time zone are you in? :)

[nikomatsakis]

This week's writing sessions -- I expect we'll schedule more for next week.

[rgreinho]

Sure thing! Not sure how that works, or what exactly is expected from participants, but I'll be glad to help.

I am in the CDT time zone.