stackexchange / stackexchange.metrics Goto Github PK

Need to use suffixes for each aggregator in aggregator gauges. Right now I'm using tags, but that needs to change. See: bosun-monitor/bosun#419

An application may want to use a metric name prefix, but still have the ability to send some metrics without that prefix (global metrics). A method called GetMetricWithoutPrefix should be added which is identical to GetMetric except that it does not prepend the prefix.

I've been taking advantage of the flexibility of passing around IDoubleGauges to keep the specific choice of gauge as configuration. Now, I'd like the ability to do the same with MetricGroups of them. This will require creating an interface IMetricGroup, since variant type annotations can only be applied to interfaces, not classes.

Here are the expected variances of the type parameters:

• TMetric is clearly covariant, since you are using it for output. Namely, you want to be able to perform this assignment legally:

  IMetricGroup<bool, IDoubleGauge> group = GetMetricGroup<bool, EventGauge>(...);
  

• The other type parameters are probably contravariant, since they are used as inputs. I believe you would like the following assignment to be legal:

  IMetricGroup<short, EventGauge> group = GetMetricGroup<int, EventGauge>(...);
  

The reasoning in the latter case can be considered like this: later calls to the group variable will only ever try to produce a new metric with a tag value of the derived type, but the MetricGroup can handle anything that's of the base type. I could also see the case made that it should just be invariant.

There's a big caveat to all of this: MetricGroup has mutable state. That means, if you cast to a group type with a more generic TMetric and then try to add an incompatible metric you'll get a runtime type error. Fortunately, the actual creation of the metric added is handled by the factory, which is type-specific, so I believe we're golden.

Adding this interface would lock in two specific aspects of the MetricGroup API:

• we couldn't add a TMetric as an input parameter to a method on MetricGroup (e.g. void AddCustom(T1 key, TMetric metric))
• we couldn't add a method that returns values of the other parameter types (e.g. IEnumerable<T1> GetKeys())

The first point seems like a reasonable limitation, but I could see how the second point is too limiting. That's why it might be useful to leave the other parameters as invariant. Also, I can't think of a great use case for really needing contravariance on those parameters, whereas for the covariance of TMetric the value is obvious.

This may be a little bit of effort, but I think it's worth it to promote MetricGroup to the same level of flexibility that the first-level gauges and counters have. Also I'm totally willing to hack at it, as long as the idea seems reasonable.

some docs

• Defining Variant Generic Interfaces and Delegates
• Variance in Generic Interfaces

Going to be annoying to consume if it's not in nuget.

I have a small console app that gets triggered by windows after certain events. It's supposed to fire up, read a few files and record metrics, then close. It does minimal work and is active for a very short period of time.

What I've found is this means it doesn't reliably send meta data. I'm calling Shutdown() to ensure metrics are sent, but I couldn't find an equivalent for sending the meta data.

Is there something I'm missing?
Should the meta data be sent from within the Shutdown method (if it wasn't previously sent)?
Should a public method be made available to force sending meta data?

Some wonkiness around Git versioning, may be fixed with a package upgrade

For aggregate gauges, there is a lock around the Record code, and the snapshot-swap. This could get contentious and expensive, and it's likely a good candidate for using SpinWait instead of a lock.

It is very hard to know whether it is working, as there are no API hooks that allow you to see metadata such as "last upload", "number of uploads", "number of metrics", or whatever - I don't know quite what should be exposed, but it would be good to have some mechanism to provide high level info; heck, just a ToString() status might be enough

Right now there are counters and aggregate gauges, but we also need snapshot gauges which are much simpler than aggregate gauges (they just report the last value).

They should also have the ability to hook into an event or callback which fires right before metrics are going to be snapshotted.

Right now, the only way to bind a type to a metric name is implicitly the first time you create a metric with that name. There should be a way to declare this beforehand.

For example, this fails:

            [BosunTag]
            public readonly int Response;

but this works fine:

            [BosunTag]
            public readonly string Response;

It seems like the library could perform the string conversion itself?

I'm using String.Join to put together the metric list. It may or may not perform better if I just wrote the individual strings out to the response stream. Need to benchmark this before making any changes to the library.

Right now, flush is called when metrics are snapshotted, but it should really be running in an infinite loop which pauses for a second when the queue is empty, and five seconds when an API POST fails. This is the way SCollector is implemented.

Bosun is going to start using an access token for sending metrics to it. This will take the form of an HTTP X-Access-Token header. BosunReporter should add this to the Options class as string AccessToken.

I'm not sure yet what a good way to expose the metadata API via this library is going to be. In the meantime, I can at least start sending whether each metric is a gauge or a counter since that's already available.

Right now, there is no way to remove a metric from the reporter after it has been added. There should be a way to do this.

Hi!

Are there people who use bosun and BosunReporter.NET in production to measure app metrics like service or page calls.

I am very interested in the kind of throughput that people are able to achieve.

So far, I only got the information that StackExchange's setup can cope with 15k entries per seconds which amount to about 3GB of data a day.

I'm looking for a metrics library to embed in my software. This one looks very promising, I like the way you can define custom metrics.

Our company uses Zabbix for it's monitoring purposes. Is there a way a ZabbixMetricHandler can be added to this library? I can put together a PR if you guys would accept that.

One last question, will this library also be available as a Nuget package on nuget.org?

Right now, ThrowOnPostFail only applies to >= 500 status codes, but it should also apply if the connection is unavailable.

I always thought [BosunTag] public readonly string TagName; was awkward. You have to remember the attribute, remember to use a field, rather than a property, and remember the readonly keyword.

If I remember right, the choice of readonly fields pre-dated C# 6 and its read-only auto-properties. Furthermore, since most fields/properties on a metric are intended to be tags, I think an opt-out system would be cleaner than opt-in.

If I were designing the Metric class today, I would probably support public string TagName { get; } with no attribute. Attributes could be used to opt-out a field/property or mark it as an optional tag.

Immutability is still important for preventing accidental duplicate metrics. So, at the very least, I would want to make sure the property has no setter, and perhaps see if it has a matching auto-generated field. You can always actively choose to shoot yourself in the foot with reflection, but you don't want people to accidentally do it.

Fyi, I noticed BosunTagAttribute was renamed to MetricTagAttribute, but there are still documentation references to [BosunTag] which should be cleaned up.

It's a breaking change and might not be worth the hassle (part of why I never did it), but since you're transforming BosunReporter into StackExchange.Metrics and there are other breaking changes, I thought now might be a good time to suggest it.

I was originally thinking that default tags could be handled as part of inheritance, but since there are three different metric classes (four if you include the base BosunMetric class) you can inherit from, it's going to get tedious to make users implement it that way. Instead, I'm going to provide a Dictionary<string, string> for default tags in BosunReporterOptions.

I want to provide the ability to specify a Funct<string, string> which property names are run through to generate tag names.

The general convention for public member names in C# is PublicMemberName, but you want that to translate to public_member_name. This option would let you do that.

I should also provide a ToSnakeCase which can be used, since that seems like the most common use case.

Need to implement a way to set a standard interval for snapshotting the metrics (in seconds). It would be neat to integrate schyntax into this when there is a C# port of that.

We want a _count suffix and aggregator for AggregateGauge which reports how many records were reported during the previous reporting interval.

I think I'm 90% done, but Bosun is complaining about something, so it's going to take a bit of debugging work. Should be done soon.

I couldn't see an API to describe metadata for tags; does this exist? For example:

The fact that BosunReporter has a background exception handler isn't even documented at the moment. Yet, it can crash your entire process if it's not set.

The next major version of BosunReporter should require it, rather than making it optional.

Right now I'm just using a list, with some optimization hacks for when it's empty or you're dequeuing everything. A standard Queue or ConcurrentQueue didn't seem like good choices, but I should revisit this in the near future.

Not all tag values are valid; the library includes features to check for invalid tags, but it would be nice if it exposed something on the public API to sanitize tag values (perhaps simply removing all illegal characters)

Just do it.

Hi,

I am currently investigating if I could add a custom handler to write metrics to influxdb using the line protocol which consists of sending HTTP text data with 1 line per metric.

I was reading the code, and found HttpJsonMetricHandler which already handle sending data using HttpClient but specific to json. But the only code I found specific to json seems to be the content-type header, which is hardcoded.

Perhaps this class could be renamed to handle more generic http messages ?

I am certainly short-sighted and the ReadOnlySequenceContent type will certainly need more customization endpoint, for example to add custom auth headers... So in the end, I will still need to do a copy of the class.