signalfx / golib Goto Github PK

Hi,

While the dependency for github.com/signalfx/gohistogram can still be collected, the repo itself is now gone it appears. Was this done on purpose? Is this dependency going to be changed? dropped?

I ask since as a part of the telegraf project we ensure we can get to all our dependencies and validate their licences.

We're switching from the https://github.com/zvelo/go-signalfx package to the official signalfx one, but noticed that there is no support for counters, but you do support cumulative counters?

Is there any reason why there is no support for regular counters?

Thanks,
Owen

Hi There!

We've recently added functionality to our metrics pipeline that allows us to send datapoints using an ingestion token based off a certain dimension value on our metrics. Previously we sent all datapoints using a single ingestion token using the HTTPSink.AddDatapoints method. With this change, we have migrated to using the AsyncMultiTokenSink.AddDatapointsWithToken method. With this method, when a batch of metrics comes in, we split the datapoints by their intended token and for any metrics that don't have their own token, we fall back to the same default token as in our old approach. For a given 'flush', we see about 60,000 datapoints per second hitting the default token and about 3000 datapoints per second for individual service tokens (these numbers are for a single instance across ~30 c5.xlarge EC2 instances).

In switching to the new approach, we have noticed a fairly significant change in terms of the memory pressure / heap allocations. I suspect this is in part due to how the AsyncMultiTokenSink buffers metrics before sending onto the SignalFx ingestion endpoint (the HTTPSink.AddDatapoints method has no buffering).

Given the number of tuning options the AsyncMultiTokenSink provides, i'm afraid we haven't optimised these parameters to suit our workload / shape of data and was looking for some guidance from those that have experience running this sink type in production.

In particular, given the numbers above, we are handling about 90M DPM across currently running with the following settings in production:

NumChannels=1 
// we only choose a single channel, because selecting multiple channels when 90%~ of our metrics go to the default token would mean we'd have a 'hot channel' and the other channels would be heavily under-utilised. We instead scale by adding more workers

NumWorkers=1000
// my understanding of this parameter is the larger this number is, the more simultaneous requests to the SignalFx ingestion API we'll have

BufferSize=1000
// we only buffer if our workers aren't able to keep up with the workload. We haven't really seen this in production

BatchSize=10000
// the max number of datapoints we'll send in a single request to SignalFx

I'm wondering if anyone has recommendations / a formula for these parameters given a theoretic workload. For instance:

• What is the optimal BatchSize we should use / can SignalFx handle?
• Given an incoming number of datapoints per second / request and an instance size (e.g 4 CPUs / 8GB RAM) what should uor target number of channels / workers be?

Running a heap pprof shows that the bulk of our objects in use is the translation step we run to convert our statsd metrics to the SignalFx Datapoint format, which to me indicates something about the new Sink holding onto that memory and not freeing it as quickly as we used to, though the pprof shows no indication that it's a leak within the AsyncMultiTokenSink code. What's interesting is that the memory utilisation does not drop when we stop sending requests to the service and our CPU profile is approximately the same between the old and new approach.

Given we also see a drop from around 10~ GC operations per second to <1 per second, my gut feel here is that we're not seeing the GC being triggered enough (because our allocation hits a high enough peak early on to never see the heap double in size, which is the default behaviour of GOGC). I will be investigating whether making the GOGC value more aggressive helps alleviate the memory pressure.

I've attached the change in memory profile:

Any guidance would be greatly appreciated!

Hi,

We use git tags to version dependencies. There was a breaking change when you renamed context. Can you push a new git tag?

Thanks!

Problem
Go mod tidy has stopped working for go.mod that imports signalfx due to issues with the smartystreets imports. This seems to happen specifically when the dependency is 2 or more levels deep (this does not happen to the go.mod directly importing signalfx). This is likely due to the recent rebranding that cause smarty/smartystreets libs to rename their imports/modules.

To replicate:

1. import github.com/signalfx/golib/v3 in a go.mod and push to a remote repository (eg git push example-branch in a repo github.com/example-project).
2. Import the branch in git repo via a separate go.mod (eg require github.com/example-project <example-branch> or require github.com/example-project + replace github.com/example-project => github.com/example-project <exampe-branch>)
3. Run go mod tidy on new project

As stated above, running go mod tidy on the module directly seems to work. This seems to only happen if the dependency is 2 layers down or more. I suspect it might have something to with the way golang attempts to download dependencies and find tags for branches (this also happens for master if master is requiring signalfx/golib/v3 and you require as github.com/example-project master instead of the explicit tag). Also open to workarounds if anyone has any, but I propose a potential solution below:

Proposed Solution
In go.mod, update goconvey to a later version to
Also update smartystreets/assertions OR add replace directives to point this to smarty/assertions instead.

Please change the default branch name for this repo to main.

At a glance, it doesn't look like any code changes will be required. The linter already looks at the main branch too. The other references to the antiquated name are broken links anyway.

Thanks!

Hello!
I can't use dep tool anymore with my project (here: https://github.com/olx-global/signalfx2terraform) , because I am receiving a lot of errors about signalfx/golib:

        v3.1.0: Could not introduce github.com/signalfx/[email protected], as its subpackage github.com/signalfx/golib/v3/pointer is missing. (Package is required by github.com/s
ignalfx/[email protected].)
        v3.0.0: Could not introduce github.com/signalfx/[email protected], as its subpackage github.com/signalfx/golib/v3/pointer is missing. (Package is required by github.com/s
ignalfx/[email protected].)

and so one.
I am using

dep version
dep:
 version     : v0.5.4
 build date  : 
 git hash    : 
 go version  : go1.12.6
 go compiler : gc
 platform    : linux/amd64
 features    : ImportDuringSolve=false

What did happen with golib?

jaeger before version 1.18.1 has known vulnerability: https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2020-10750
It was fixed in 1.18.1: https://github.com/jaegertracing/jaeger/releases/tag/v1.18.1

So github.com/jaegertracing/jaeger in version v1.15.1 is detected as a vulnerability during static analysis:

Upgrading github.com/jaegertracing/jaeger to 1.18.1 (or newer) should fix this problem.

Shouldn't sfxclient be deprecated in favor of https://github.com/signalfx/splunk-otel-go and https://pkg.go.dev/go.opentelemetry.io/otel/metric?

Or maybe we should simply refer to https://github.com/signalfx/splunk-otel-go instead of sfxclient in the main README.md:

CC @mdubbyap @jgheewala