New compression algorithms brotli about haproxy

[git001]

@TimWolla have send a patchset to the mailing list which needs a review.
https://www.mail-archive.com/[email protected]/msg32807.html

[wtarreau]

So you already have two other solutions available. Just use what suits you best for a given use case. If you're interested, there's also an implementation in rust running as an external module called from Lua, that was presented at HaproxyConf: https://www.youtube.com/watch?v=VNidpAXOrpk

But at this stage, I'm sorry to say we don't have the bandwidth to develop and even less to maintain and handle problem reports associated with such possibly heavy compressors.

[wtarreau]

Brotli is expected to compress slightly better than gzip at the same speed and much better at a much lower speed. I'm a bit worried about the CPU time and RAM usage needed to achieve this though, and when dealing with large traffic I wouldn't be surprized if we have to throttle to preserve memory, resulting in a lower overall compression ratio. This is already how SLZ achieves better average compression ratios than zlib, since the former uses 3 times less CPU and doesn't require to keep a heavy memory context.

But possibly that it's not that difficult to link with libbrotli and run some tests.

[lukastribus]

Also see the discourse thread Brotli compression support, which also contains comments from Google's Jyrki Alakuijala, which afaik was involved in the development of brotli.

[RedXanadu]

@jriall It looks like this idea was shelved over concerns about resource usage. See:
https://www.mail-archive.com/[email protected]/msg32964.html
https://discourse.haproxy.org/t/brotli-compression-support/1931/9

[lukastribus]

live the choice on the user of haproxy to add more cpu/memory to achieve less bandwith and less latency....

s/live/leave/

The maintenance as well as the support burden is something that a OSS project needs to consider when accepting new features.

This is even more true for cpu and memory intensive features in a load balancer.

So no, we definitely can't just "leave this up to the user".

[wtarreau]

No it does not depend on a point of view, and it's not a matter of new modules, it's a matter of regular breakage caused by changing APIs for code that is maintained out of tree. And I think I do qualify particularly well to know how painful that can be because the company I work for does have some modules developed and maintained by very few people who carefully follow the core's evolution, and despite this, about half of the issues that are reported are definitely caused by subtle incompatibility issues that happen each time a new major version is emitted. There's a reason the kernel developers make huge efforts to integrate out-of-tree code instead of keeping it outside and out-of-date: it's basically impossible to maintain out-of-tree code working fine in a same memory space. We even had the issue with device detection add-ons because they were not regularly built due to their dependency on an external library. The issue mostly stopped when they provided dummy libs that permitted to integrate them into every single build developers do.

The points you mention regarding what to prepare for when using opensource are sane. I wish more than 5% of opensource users would think like this and not just see it as software someone else developed for free. We do have awesome users who contribute a lot by running development versions, reporting core dumps, reading code, suggesting fixes and improvements. But these users are rare, and besides a few even rarer exceptions, the larger the company the rarer it is to even know they're using the software. Or you learn about them when they start to get in touch with support teams, complaining about instabilities that cause them some grief but still without exchanging any useful info!

[breisig]

This would be a great feature to add.

[jriall]

Is this still under consideration?

[git001]

Let's close this issue as it looks like that the brotli algorithm uses more resources then the other algorithms

[mcarbonneaux]

brotli consume more ressource but permit to have better latency if you add more power to compress.
leave the choice on the user of haproxy to add more cpu/memory to achieve less bandwith and less latency....

[mcarbonneaux]

i'm sad about you way of thinking... and how you integrated need of haproxy user...

about brotli, majority of browser support it, nginx and envoy support it...

• browser support : https://caniuse.com/brotli
• nginx : https://github.com/google/ngx_brotli (made by google, because nginx as documented module inteface)
• envoy : https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/compression/brotli/compressor/v3/brotli.proto#extension-envoy-compression-brotli-compressor (integrated in envoy).
• apache : https://httpd.apache.org/docs/2.4/fr/mod/mod_brotli.html

[mcarbonneaux]

push documentation for developping module, and the community while develop for you (like apache/nginx/envoy)...

i'm sade because haproxy is french, and i prefere using it than other solution... and it's really performant... but each time i try to replicate what can do with other opensource (or non opensource...) solution the response is the same...

[wtarreau]

A module would only make things worse, it was what made Apache 1.3 so unreliable by then. Having out-of-tree code that doesn't follow internal API changes as fast, and that is only mastered by people who are not those dealing with bug reports here is a total horror, because in the end the crashes you get for memory corruption, loops etc are caused by any code that lies in the same memory space. That's exactly one of the reasons why we designed SPOE, to be able to load external modules running in a separate memory space, and possibly even developed in other languages to suit developers' preference. But it never took off, nobody's interested in developing external code for it nor participating to improve it, so it's going to be dropped (probably after 3.0).

Just to give you an idea, chasing nasty bugs like I mention above take more than 50% of the developers' time. Some features are postponed from version to version because too few people are available/able to chase such bugs. It's not a pleasure to say "no to this or that", it's really a matter of survival of the core.

Now if someone is able to contribute a brotli compressor to merge in mainline, that can:

• bound its memory usage per request to something reasonable (a few tens of kB, not 256kB like zlib)
• bound its latency per call to something reasonable in microseconds not milliseconds
• works by chunks, not a full object at once
• doesn't rely on a fast-moving random external lib that doesn't care about long-term maintenance

Then it would certainly be welcome. At the moment I'm not aware of anything like this.

[chipitsine]

чт, 9 нояб. 2023 г. в 04:41, Willy Tarreau ***@***.***>:

A module would only make things worse, it was what made Apache 1.3 so unreliable by then. Having out-of-tree code that doesn't follow internal API changes as fast, and that is only mastered by people who are not those dealing with bug reports here is a total horror, because in the end the crashes you get for memory corruption, loops etc are caused by *any* code that lies in the same memory space. That's exactly one of the reasons why we designed SPOE, to be able to load external modules running in a separate memory space, and possibly even developed in other languages to suit developers' preference. But it never took off, nobody's interested in developing external code for it nor participating to improve it, so it's going to be dropped (probably after 3.0). Just to give you an idea, chasing nasty bugs like I mention above take *more* than 50% of the developers' time. Some features are postponed from version to version because too few people are available/able to chase such bugs. It's not a pleasure to say "no to this or that", it's really a matter of survival of the core. Now if someone is able to contribute a brotli compressor to merge in mainline, that can: - bound its memory usage per request to something reasonable (a few tens of kB, not 256kB like zlib) - bound its latency per call to something reasonable in microseconds not milliseconds - works by chunks, not a full object at once

brotli works on full objects. In theory you can compress static with brotli in advance, but that applies to web servers, not reverse proxies.

…

- doesn't rely on a fast-moving random external lib that doesn't care about long-term maintenance Then it would certainly be welcome. At the moment I'm not aware of anything like this. — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ5KUC2LGBNC3WQQEM4U7TYDRGEXAVCNFSM4GRVEYYKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBQGMYTCNBTHA3Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mcarbonneaux]

That's exactly one of the reasons why we designed SPOE

spoe idea is nice, but cannot manipulate body as filter... sade...

apache..

ok but my example is not about perf but about extensibility... and now is about event like reverse proxy like nginx or envoy that are extensible...
ok module can be pain if bad coded.... but in my experience majority of the time is not a problem...
i've used apache for long time without problem... with many module....

[wtarreau]

spoe idea is nice, but cannot manipulate body as filter... sade...

That was the initial point, however it takes a lot of time to rework on this and sadly nobody's interested in working 6 months on something that nobody uses :-(

ok module can be pain if bad coded.... but in my experience majority of the time is not a problem...

That's always the #1 problem. It's not a matter of "poorly coded", it's a matter of complexity of internal APIs and interaction between elements in fast moving APIs. What I'm describing is not specific to haproxy and is the same for plenty of other mechanisms as well. Look at the kernel for example. Also Apache and nginx can tolerate modules well when they work in multi-process mode where a blocking module doesn't put all the traffic to a halt, and where a crashing process will automatically reload. I'm not kidding, I remember seeing "tail -f" on Apache logs scrolling so fast we had to ctrl-C to discover that processes were just crashing and restarting in turn because one of the modules was under attack...

[chipitsine]

On Thu, Nov 9, 2023, 22:00 Willy Tarreau ***@***.***> wrote: spoe idea is nice, but cannot manipulate body as filter... sade... That was the initial point, however it takes a lot of time to rework on this and sadly nobody's interested in working 6 months on something that nobody uses :-( ok module can be pain if bad coded.... but in my experience majority of the time is not a problem... That's always the #1 <#1> problem. It's not a matter of "poorly coded", it's a matter of complexity of internal APIs and interaction between elements in fast moving APIs. What I'm describing is not specific to haproxy and is the same for plenty of other mechanisms as well. Look at the kernel for example. Also Apache and nginx can tolerate modules well when they work in multi-process mode where a blocking module doesn't put all the traffic to a halt, and where a crashing process will automatically reload. I'm not kidding, I remember seeing "tail -f" on Apache logs scrolling so fast we had to ctrl-C to discover that processes were just crashing and restarting in turn because one of the modules was under attack...

That also happened to nginx brotli, it failed due to "fast changes in internal API", and master process just kept restarting workers. I had that experience)) —

…

Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ5KUHVNXWE4HTHDRFZYA3YDT42HAVCNFSM4GRVEYYKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBQGQYTANRVGY2Q> . You are receiving this because you commented.Message ID: ***@***.***>

[mcarbonneaux]

depend on point of view....
new module are instable, don't use at start... like new major version of open source software... haproxy included...
qualify use of module before go in production...
don't use every module you found on internet... avaluate the maturity of the module before use... wait the maturity before use...
be prepared to analyse core dump... or don't use opensource software...
be prepared to read source code to find the root cause... or don't use opensource software...
be prepared to contribute... or don't use opensource software...

or use proprietary software...

[mcarbonneaux]

it's a matter of regular breakage caused by changing APIs for code that is maintained out of tree.

for apache the api has been stable for long time... but passing from 1.3 to 2.0 it as brocken many thing, but the communauty as followed... ok is challanging to conceive an stable api... but when is done is abel to add many value...

the api must be suffisantly abstracted to be stable ...

quasi all of opensource project are plugable... it's not impossible to do...must be just correctly documented...

for example keycloak while be not interesting without plugability... ok he had changed multiple time the api... but it's work...
i can found many exemple like that...

spoa is an api... but you miss the possibility to add body filter... but i think is not to complicate to add it because you conceived the protocol to be upgradable...

And I think I do qualify particularly well to know how painful that can be because the company I work for does have some modules developed and maintained by very few people who carefully follow the core's evolution, and despite this, about half of the issues that are reported are definitely caused by subtle incompatibility issues that happen each time a new major version is emitted. There's a reason the kernel developers make huge efforts to integrate out-of-tree code instead of keeping it outside and out-of-date: it's basically impossible to maintain out-of-tree code working fine in a same memory space. We even had the issue with device detection add-ons because they were not regularly built due to their dependency on an external library. The issue mostly stopped when they provided dummy libs that permitted to integrate them into every single build developers do.

it's normal life of opensource project ! and software ! :)

The points you mention regarding what to prepare for when using opensource are sane.

you must diferenciate who are only consumer, who are potentiel contributer...

[wtarreau]

You really seem to underestimate what a multi-protocol internal proxy API looks like. By the time you manage to document 0.1% of it, it is already obsolete. We even thought about trying to provide an abstraction layer to an external type of modules, but these would still need to access all the rest (tasks, etc).

spoa is an api... but you miss the possibility to add body filter...

It's not that we missed it, it was one of the very first goals. The problem is that it was provided as an early PoC for a first version and that it takes an amazing amount of work to rework on it, that nobody is interested to engage into, hence my intent to just get rid of it :-( It's just too annoying to regularly have to deal with issues there that result from an early attempt, knowing that we're just trying to dirtily work around basic issues while what is needed is to break the engine and rewrite it differently after figuring what we want to support and what not (e.g. streaming a body may or may not be able to affect headers and that's not compatible with the current sequence of internal processing).

[mcarbonneaux]

You really seem to underestimate what a multi-protocol internal proxy API looks like. By the time you manage to document 0.1% of it, it is already obsolete. We even thought about trying to provide an abstraction layer to an external type of modules, but these would still need to access all the rest (tasks, etc).

i know perfectly... i've developed many plugin for apache... and many plugin for opensource project in different language....
you need to autodocument your code to generate your doc (like apache and many other project)... documentation is more important that the code itself in fact... if you want people start to work on it...

It's not that we missed it, it was one of the very first goals. The problem is that it was provided as an early PoC for a first version and that it takes an amazing amount of work to rework on it, that nobody is interested to engage into

i use it on my internal project and it's work fine... we've devloped spoa service in go for my internal need... we just need body to completely do what i need !

envoy is developping the same fonctionality than spoe, that are named external processing...
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter

and use grpc to pass the processing... and work on body also... and is multiplexed (http/2 and grpc)... the interface are declared in normalized way with protobuf... can be done async (all body) or sync (chunk by chunk)...