JoinHandle not completing when too many tasks or opened files present about tokio-uring

[Noah-Kennedy]

I'll get a fix up in a bit.

[Noah-Kennedy]

@FrankReh could you review the RT changes?

[Noah-Kennedy]

You've hit an issue I found while fixing something else months ago, decided I would fix the next day, and then promptly forgot about until now lmao.

io_uring has an unfortunate footgun where you can put more events in-flight than the completion queue can hold, thus allowing you to overflow the queue. If the queue overflows, the extra completions are yeeted into the void, never to be received.

The solution here is to add a semaphore to make sure we can't put more events in-flight than the cqueue can hold.

[Darksonn]

Huh? My understanding is that this was true in the first versions of the driver, but hasn't this been fixed? As far as I know, the driver should hold on to completions that don't fit until you empty the completion queue.

[Noah-Kennedy]

@Darksonn no, still an issue, its a weird edge in the kernel APIs. The kernel drops completions in the event of an overflow. A semaphore is currently the workaround.

[FrankReh]

@Noah-Kennedy There is a CQ_OVERFLOW bit in the submission queue now and the rust io-uring bindings exposes it with the method cq_overflow on the ring's squeue. I believe the intent for this bit is to let the user space know when the kernel couldn't add to the cq so after the user space has drained some or all of the cq ring, it is up to the user space to kick the kernel again so the kernel will again fill the cq. I don't think the kernel is actually dropping information, it is probably just being queued up by an internal mechanism.

Getting tokio-uring to use this mechanism seems safer than trying to add a semaphore because there will be other reasons for the cq filling up too, the multi-shot operations come to mind.

[Noah-Kennedy]

@FrankReh I will need to look at this, but I agree that this sounds like a much better option!

[ollie-etl]

@FrankReh I think i've hit this too - I will also take a look

[FrankReh]

I think I saw the problem fixed locally by using the CQ_OVERFLOW bit to say when tick should be run again. But I was waiting for the Rc/RefCell driver rework. My solution had tick to using a loop and checking the squeue bit itself for deciding when to run again. I wasn't thrilled with the memory sync options that were available and saw a recent PR for io-uring that would have separated read and write fences but it wasn't accepted yet.

The Rc removal hasn't been committed yet. And I wanted to see if the RefCell couldn't be pushed further down so tick didn't need to be given a mutable driver either.

Not to say you shouldn't look at the problem. Maybe you'll find a better way.

[Noah-Kennedy]

Let's go with your approach once the rt changes get committed.

[FrankReh]

If the rt changes don't go today or tomorrow, I may send a PR that shows what I had done so far anyway. Could help since a few people have run into cases where the CQ_OVERFLOW mechanism would have helped.

[FrankReh]

This has been solved. Waiting for tokio-rs/io-uring#152 or its equivalent to be merged and then this crate's version for io-uring to be bumped appropriately.