http2 performance issue (2x slower than http1.1 and vs h2load) about oha

[hatoo]

#540 solves on my side.

Could you check on your side?

[hatoo]

Ok
Binary here

cargo build --release --target x86_64-unknown-linux-musl

oha.zip

[hatoo]

In my observation, #540 constantly did 100 downloads by around 15.5s which is about the same as h2load
But adaptive_window may work vary on runtime (although I don't know what adaptive_window means).

I've found that h2loads's default window size setting is larger than hyper's default
I've added on #541. It takes time the same as h2load in my tests.

Could you check it? Binary is attached in the PR.

@hatoo On main builds (and even on branches) - What if we saved the built binaries as artifacts that we could download them when we need to test a specific version or pre-release. Thoughts?

It looks good but I don't know how to do it without spamming https://github.com/hatoo/oha/releases page

[reneleonhardt]

Wow, awesome analysis, thank you! 🚀

[hatoo]

Thank you for reporting it.

I've been able to reproduce it.
tested at AWS CloudShell in ap-northeast-1

$ time h2load -c 1 -n 100 -m 1 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
starting benchmark...
spawning thread #0: 1 total client(s). 100 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 15.74s, 6.35 req/s, 15.59MB/s
requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 245.37MB (257291904) total, 7.97KB (8159) headers (space savings 82.86%), 245.23MB (257140500) data
                     min         max         mean         sd        +/- sd
time for request:   138.92ms    900.98ms    153.99ms     76.99ms    98.00%
time for connect:   337.22ms    337.22ms    337.22ms         0us   100.00%
time to 1st byte:   451.57ms    451.57ms    451.57ms         0us   100.00%
req/s           :       6.35        6.35        6.35        0.00   100.00%

real    0m15.755s
user    0m0.318s
sys     0m0.198s

$ time ./oha-linux-amd64 -c 1 -p 1 -n 100 --http2 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif                                                                                                                                                                                                                                               
Summary:
  Success rate: 100.00%
  Total:        25.3859 secs
  Slowest:      0.8901 secs
  Fastest:      0.2342 secs
  Average:      0.2505 secs
  Requests/sec: 3.9392

  Total data:   245.23 MiB
  Size/request: 2.45 MiB
  Size/sec:     9.66 MiB

Response time histogram:
  0.234 [1]  |
  0.300 [98] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.365 [0]  |
  0.431 [0]  |
  0.497 [0]  |
  0.562 [0]  |
  0.628 [0]  |
  0.693 [0]  |
  0.759 [0]  |
  0.825 [0]  |
  0.890 [1]  |

Response time distribution:
  10.00% in 0.2351 secs
  25.00% in 0.2396 secs
  50.00% in 0.2456 secs
  75.00% in 0.2461 secs
  90.00% in 0.2485 secs
  95.00% in 0.2543 secs
  99.00% in 0.8901 secs
  99.90% in 0.8901 secs
  99.99% in 0.8901 secs


Details (average, fastest, slowest):
  DNS+dialup:   0.0000 secs, 0.0000 secs, 0.0000 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs

Status code distribution:
  [200] 100 responses

real    0m25.459s
user    0m1.256s
sys     0m0.425s

And I've created a simple load generator to find whether the difference is came from oha or hyper which is backend of oha.
https://github.com/hatoo/oha-538

$ time ./oha-538 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
....................................................................................................
real    0m24.733s
user    0m0.664s
sys     0m0.642s

I guess the difference came from hyper.
I'll investigate further.

[huntharo]

Oh cool will try

[huntharo]

Hmm... it's kind of tricky to do given that I'm on a Mac and CloudShell doesn't seem to have enough space to setup the Rust toolchain. I could do it a thousand other way but it would be so nice if the linux binary with that fix was available?

[huntharo]

@hatoo On main builds (and even on branches) - What if we saved the built binaries as artifacts that we could download them when we need to test a specific version or pre-release. Thoughts?

[huntharo]

#540 solves on my side.

Could you check on your side?

@hatoo did you check the performance vs h2load with the fix? What I see is:

• oha before fix: 25 seconds for 100 copies of file
• oha with fix: 18 seconds
• h2load: 14.5 seconds

The fix has closed 64% of the gap with h2load, which is GREAT!

But there is something else it seems? I wonder if Rustls is the remaining slowness or if we have to change any other parameters on hyper?