Blinking test_full about aw-server-rust

[johan-bjareholt]

I've ran this for an hour on my local machine and it has failed once, not sure what the issue can be really.
But at least it is possible to reproduce.

[xylix]

Could it be related to the hard coded test port being in use on travis? Or related to ram limits. I've no other idea what could it be.

…

On Tue, 3 Mar 2020, 23.27 Johan Bjäreholt, ***@***.***> wrote: I've ran this for an hour on my local machine and it has failed once, not sure what the issue can be really. But at least it is possible to reproduce. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#92?email_source=notifications&email_token=ADGEMKGUFO7JVH7TP2IK6YLRFVY2XA5CNFSM4K7UAGN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENVG5TY#issuecomment-594177743>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGEMKCS67FEX2G7JQBZW3LRFVY2XANCNFSM4K7UAGNQ> .

[johan-bjareholt]

@xylix I've also considered if it's relate dto the hard-coded port, but then realized that the test is only run once and inside a container so it really shouldn't interfere with anything else.

I've also considered if it's because we don't cleanly shut down, but it's the same thing there that the test runs once and inside a container so it shouldn't interfere. I'd like to fix this anyway though but it will be almost impossible before rocket releases version 0.5 with its async support and updated hyper where it should be possible to gracefully shut down.

[xylix]

Looking some more at aw-server-rust's aw-client, running the tests in my ubuntu virtual machine until failure,
// Start testserver and wait 10s for it to start up // TODO: Properly shutdown

Also just as a tip for reproducing: in my ubuntu VM it fails every 1/10 - 1/20 runs.

I don't think the test_full is actually waiting 10 seconds. Could the problem just be it trying to get stuff from the setup server before the server is ready?
Looking into it now I'm starting to think most likely the server is closing the connection between the get_bucket and get_buckets call in the test_full. If this is the case it's not really a problem with the test, it's problem with the client / reqwest timing out. It's not even a long session so this shouldn't happen.

Could try adding some sort of keep_alive / ensure_is_alive thing to all the client methods or could try changing from using a pooling client to just making singular requests with aw-client-rust.

@johan-bjareholt What are the planned use cases for aw-client-rust anyway? If it's stuff like the watchers that might update every heartbeat or much slower does it even make sense to keep a connection alive or should we just send single requests with the client?

I did the changes to change it to make new clients for every request, and it fixes the test. Changes are here #94 . We probably do not want to implement them this way though, since it'll add an amount to overhead to aw-client-rust. I coded it this way to find out if this is the actual issue.

Adding some kind of check "if old connection is alive, use it, otherwise create a new connection" is probably the more correct way, but it'll also take more programming. Might even be necessary to use some other library than reqwest.

[johan-bjareholt]

Well done!

Looking into it now I'm starting to think most likely the server is closing the connection between the get_bucket and get_buckets call in the test_full. If this is the case it's not really a problem with the test, it's problem with the client / reqwest timing out. It's not even a long session so this shouldn't happen.

Well, the proper thing to do would be to figure out why the server is closing the connection.
Can't really see a reason why this would occur.

What are the planned use cases for aw-client-rust anyway? If it's stuff like the watchers that might update every heartbeat or much slower does it even make sense to keep a connection alive or should we just send single requests with the client?

For now it's just integration tests and my prototype aw-watcher-wayland. In the future it would be nice if it could be a proper client library but I don't think we should prioritize doing that now.

[xylix]

Yeah. Rocket supports an option for keep-alive, will probably need to try a longer option for that at least. Altough the default probably isn't 0 anyway.

But looking at the code it doesn't seem like aw-server-rust handles a cut connection in any way since it only initializes the reqwest client once. Or does reqwest automatically re-connect on demand?

[johan-bjareholt]

@xylix The default keep_alive seems to be 5 seconds, seeing what happens when you increase it would be interesting.

https://api.rocket.rs/v0.4/src/rocket/config/config.rs.html#231

[xylix]

Explicitly setting it to 10 or 20 seconds did nothing :/. Weirdly setting it to 0 (which disables keep-alive connections) actually seems to make the test fail less (0 failure in quite a while), so maybe reqwest can actually figure out to reconnect if it knows connections have been closed.

After testing with keep_alive at 0, rocket accidentally dropping connections on high cpu usage / low available cpu resources could explain the failures with a non 0 amount of keep_alive, since in my VM and travis it happens quite often but earlier you said it took almost an hour of running tests on your machine to get one failure.

https://github.com/SergioBenitez/Rocket/issues?q=is%3Aissue+drop+connection I think we'll need to file a new upstream issue. (I filed one, rwf2/Rocket#1254 , waiting for questions about details now)

One explanation could be that aw-client-rust isn't actually consuming the previous request fully, which would lead to the server not allowing another request from the same connection (see this rwf2/Rocket#580 (comment) )
I'll try to see if running the tests with rocket in production config changes anything, but looking here (it didn't change anything)

[xylix]

Upstream suggested we wait for Rocket 0.5 and see if it's fixed in that.

They mentioned possible solutions including setting keep_alive to 0 and running rocket behind a reverse proxy.

@johan-bjareholt So we should probably just set the keep_alive to 0 for now as a workaround, and then get back to this when rocket 0.5 releases.

[xylix]

I created #97 , but we should still leave this issue open until the upstream fix possible in 0.5 is confirmed / denied.

[johan-bjareholt]

Has not occurred for a long time, not sure if it was because of one of our workarounds or the switch to GitHub actions though.

Closing