Consider automatically computing statistics about gen_stage

[onetapbeyond]

Just a quick heads-up for those interested: an updated version of the GenMetrics library is coming soon and will add support for pushing metrics out to any statsd agent which makes it possible to analyze, visualize etc. the metrics in existing tools and services like Graphana and Datadog.

You simply activate the service using the existing statistics option, for example:

alias GenMetrics.GenStage.Pipeline
pipeline = %Pipeline{name: "demo",
                     producer: [Data.Producer],
                     producer_consumer: [Data.Scrubber, Data.Analyzer],
                     consumer: [Data.Consumer],
                     opts: [statistics: :statsd]}
GenMetrics.monitor_pipeline(pipeline)

The original in-memory statistics option, statistics: true, is still available as before. The upcoming changes simply introduce support for :statsd and :datadog on the option. Here is a screenshot of GenMetrics in action within a Datadog dashboard for the classic A -> B -> C GenStage pipeline:

This new feature continues to support both GenServer and GenStage applications. I'll update this thread once I've pushed the new release in a couple of days.

[onetapbeyond]

A quick update for everybody.

A new v0.2.0 release of the GenMetrics library has just gone live. You can now collect and report metrics on any GenServer or GenStage application and in addition to existing in-memory metrics for real-time analysis you can optionally redirect metrics to a statsd agent making it possible to analyze, and visualize these metrics within existing tools and services like Graphana and Datadog.

The screenshot above in my previous post gives you a very small sample of the metrics data captured by the library being rendered in a Datadog dashboard.

The github repo and the hexdocs for the project have been updated to reflect these changes. As ever, I remain very interested in any feedback from the Elixir core team or the wider community. Enjoy.

ps. The GitPitch presentation for the GenMetrics library has now also been updated and it's a great way to get a quick overview of the metrics gathering and reporting capabilities of GenMetrics.

[josevalim]

@fishcakez mentioned that we could measure the "relative time" which is the time between sending a demand and finally getting the event for it.

[onetapbeyond]

Hi @josevalim, @fishcakez et al.

Being new to the Elixir community I wanted to bootstrap my learning by sinking my teeth into an interesting problem. So I decided to see what I could do to help with delivering automatically computed statistics for GenStage applications.

The result is GenMetrics. It is a newly released Hex package that supports the collection and publication of runtime metrics for GenServer and GenStage applications.

You can find detailed documentation for the GenMetrics library here on HexDocs. There is also example code demonstrating the use of GenMetrics alongside GenServer and GenStage applications in the GitHub repo.

As you will see, the approach I have taken is based on introspection. As a consequence, no instrumentation changes are needed within the GenServer or GenStage libraries themselves. Indeed no changes are needed within existing GenServer or GenStage application source code for these applications to take full advantage of GenMetrics right away.

I hope this is something that may be of interest to the Elixir community. I've also just started initial work on a GenMetrics Console, essentially a dashboard powered by GenMetrics that will allow users to visualize in realtime what is going on inside their GenServer and GenStage applications.

I greatly appreciate any feedback, criticisms or advice you might have to help improve and promote this library. Thanks.

[josevalim]

Thank you for the updates and good work @onetapbeyond! ❤️

[omnibs]

I ended up here investigating whether I can write something to poke inside GenStage to pick out statistics for reporting.

Specifically, we have a GenStage running reports and we'd live to observe the stages in our dashboard. I was looking for an alternative to sprinkling instrumentation calls inside our GenStages.

If there was a set of instrumentation callbacks that just report statistics on GenStage I could write something pluggable like:

defmodule MySQLIngestor do
  use GenStage
  use GenStageCloudwatchMonitor, logging: [:demand, :counter]
...

Which would take care of sending data about that GenStage to AWS CloudWatch, keeping charts like this for GenServers in production:

(taken from Measuring and visualizing GenStage/Flow with Gnuplot)

Not sure though if having instrumentation callbacks are the way, but enabling that kind of simple drop-in instrumentation above would be super nice.

[fishcakez]

Hi @onetapbeyond! Thanks for looking into this. There's quite a lot there, so I will have to look next week! Sorry for the delay.

[isaacsanders]

If there were a "GenMetric" type behaviour (there isn't AFAIK) that had an interface for this sort of thing, I think this would be useful for the initial release. As it stands, there isn't a unified interface for storing metrics, so I am inclined to think that this is not very important immediately.

Then again, I am just one. Others may differ.

---

On the other hand, there could be a meta-producer (or a series of them) inherent in each entity that would vend these statistics to anything that subscribes.

[onetapbeyond]

Just fyi, GenMetrics v0.3.0 has just been released with support for metrics sampling. This feature can be used to greatly reduce the runtime overhead of the GenMetrics monitoring agent when working with high volume/throughput/load GenServer and GenStage applications.

You can activate sampling using the new sample_rate option, for example:

alias GenMetrics.GenStage.Pipeline
pipeline = %Pipeline{name: "demo",
                     producer: [Data.Producer],
                     producer_consumer: [Data.Scrubber, Data.Analyzer],
                     consumer: [Data.Consumer],
                     opts: [sample_rate: 0.1]}
GenMetrics.monitor_pipeline(pipeline)

A detailed GenMetrics benchmarks performance guide is also available.

[seivan]

Does this mean the metrics will be left for third party solutions like GenMetrics (which btw looks amazing) or will GenStage incorporate its own solution albeit more spartan?

[josevalim]

It is still to be answered. None of the other behaviors have metrics out of the box, so we need to find out the best way to act consistently. -- *José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D*

[josevalim]

Closing this. I think today the best option to move this forward is to add telemetry to your stage implementations and then build the metrics from that.