Unexplained race condition in v0.16 causing "runtime dropped the dispatch task" about bollard

[fussybeaver]

I did find a fix, if you have the time, I'd appreciate if you can check if it works for you.. #390

Related to this hyperium/hyper#2312

[fussybeaver]

Thanks for the report... I see there's a v1.2.0 version of Hyper out and version v0.1.3 of hyper-util, let's see if we can reproduce this on those versions.

[fussybeaver]

I actually can't reproduce this problem. Can you give more detail on your system, and maybe any dockerd logs you find ? you can turn on debug logging in the daemon using the following configuration in /etc/docker/daemon.json :

{
	"debug": true,
	"raw-logs": true
}

[Nikita240]

That's very strange. I'm able to replicate this on two different machines running different docker versions.

[Nikita240]

I think the issue here is caused by the statically stored Docker instance static DOCKER: OnceCell<Docker>.

When running tokio tests with multi_thread, tokio will actually run the tests concurrently, but spawn a unique runtime for each one of them.

As of [email protected], somehow, the Docker instance "absorbs" the first tokio runtime it sees, and if that runtime is dropped while someone else is making a request, you get the error "runtime dropped the dispatch task".

[fussybeaver]

Ah yes, I see it now if you run them all together..

[fussybeaver]

I put this test scenario into bollard's CI system, and it seems to fail on all connectors (http / ssl / named pipe / unix socket), so that excludes any issue with any individual connector. I also checked locally running against the latest master branch of hyper and it still fails (albeit less often).

[Nikita240]

I just got around to test your fix. I can confirm it works!

Thank you so much for your support on this ❤️