[Feature Request] HTTP connection reuse about rmega

[topac]

I've just took a look at this old code and it seems so.
As you suggest, the correct solution should be 1 connection kept-alive and per host per thread.

I'll ping you for a review of the new code soon :)

[baconwaifu]

Some testing says that there's nothing that breaks if using a per-thread connection pool (technically per-fiber, using Thread.current) but there is another issue stopping that from doing anything: Pool creates a new thread for every single chunk, rather than keeping threads alive for as long as possible.

Modifying Pool so that threads live until they run out of work, only creating new threads up to the limit seems to work fine. but that reveals another issue: there's no coherent place to call 'finish' for HTTP requests. It would require quite a bit more "upwards visibility" in the network stack, or an open/close interface in Net (what I'm about to try)

[baconwaifu]

tested adding an http_wrap method in Net, and wrapping the each_chunk in it. Opens a connection to the storage server even if one may not be needed, but it seems to work? it's at least re-using the HTTP objects. Another possible (albeit hacky) solution is to simply keep all userstorage.mega.co.nz connections open for the lifetime of the thread, and let the GC close them.

[baconwaifu]

Just tested a dirty workaround (let the GC close the userstorage connections for us), can confirm persistent connections make a huge difference in speed. it's now capable of hitting 100Mbps on 4 threads.

[topac]

checkout this new branch https://github.com/topac/rmega/tree/conn_pool

[topac]

@baconwaifu have you tried using the changes from the conn_pool branch?

[baconwaifu]

Was busy with schoolwork and car stuff. Will take a look later today.

[vijayanandnandam]

using Typhoeus instead of plain net/http should work. Typhoeus already handles http connection pooling

[topac]

@vijayanandnandam sure but I would like to not have dependencies on other gems

[topac]

this was merged