jmx_exporter should collect metrics asynchronously about jmx_exporter

[brian-brazil]

The Prometheus architecture is that metrics are collected upon request, as any form of caching is going to cause aliasing and related weirdness.

[iksaif]

And this is probably valid when metric collection is cheap, which clearly isn't the case with the jmx exporter. Having multiple clients fetching data from /metrics makes all that worse.

I totally agree that in this case the software should get fixed as fetching metrics should not really do actual work in general. But since changing all that in something like Cassandra isn't trivial it would be good to have a (discouraged) workaround.

I'll try to profile what is taking so much time exactly tomorrow, and post results here, maybe there is a quick win that can make that less necessary.

[brian-brazil]

The problem with C* was latency when I last investigated it, there's lots of waiting in the various JMX calls. Thus multiple clients doesn't make a difference.

[iksaif]

The thing is that some of these jmx calls create more load on the machine (the SnapshotSize for example does actually I/O).

This surely should be fixed, but my point is that since not all software will be fixable it might be a good idea to provide a way to mitigate issues when they happen. One solution is to collect metrics in the background.

If you feel that this is not something you want in the official client, that's fine too, we will just build it on top of JMX Exporter :)

[brian-brazil]

Caching is usually considered bad practice, so it's not something I'm inclined to add.