collectd / go-collectd

Utilities for using collectd together with Golang.

go-collectd

Utilities for using collectd together with Go.

Synopsis

package main

import (
    "context"
    "time"
    
    "collectd.org/api"
    "collectd.org/exec"
)

func main() {
    vl := api.ValueList{
        Identifier: api.Identifier{
            Host:   exec.Hostname(),
            Plugin: "golang",
            Type:   "gauge",
        },
        Time:     time.Now(),
        Interval: exec.Interval(),
        Values:   []api.Value{api.Gauge(42)},
    }
    exec.Putval.Write(context.Background(), &vl)
}

Description

This is a very simple package and very much a Work in Progress, so expect things to move around and be renamed a lot.

The repository is organized as follows:

• Package collectd.org/api declares data structures you may already know from the collectd source code itself, such as ValueList.
• Package collectd.org/exec declares some utilities for writing binaries to be executed with the exec plugin. It provides some utilities (getting the hostname, e.g.) and an executor which you may use to easily schedule function calls.
• Package collectd.org/format declares functions for formatting ValueLists in other format. Right now, only PUTVAL is implemented. Eventually I plan to add parsers for some formats, such as the JSON export.
• Package collectd.org/network implements collectd's binary network protocol. It offers client and server implementations, see network.Client and network.ListenAndWrite() for more details.

Install

To use this package in your own programs, simply use go get to fetch the packages you need, for example:

go get collectd.org/api

Author

Florian "octo" Forster <ff at octo.it>

go-collectd's Issues

conversion error message is too opaque to action

When go-collectd is used inside of influxdb, there can be 10s of servers sending 100s of metrics in and only one reports an error. The error message as it stands is like this:

2017/09/07 16:55:07 unable to convert values according to TypesDB: len(args) = 3, want 4

There are not enough details present to properly troubleshoot the situation.

This patch:

diff -u ./go-collectd/network/parse.go ./go/src/collectd.org/network/parse.go
--- ./go-collectd/network/parse.go	2017-09-08 19:32:59.949965463 +1000
+++ ./go/src/collectd.org/network/parse.go	2017-09-08 23:18:41.574964616 +1000
@@ -110,7 +110,7 @@
 				// Returns an error if the number of values is incorrect.
 				v, err := ds.Values(ifValues...)
 				if err != nil {
-					log.Printf("unable to convert values according to TypesDB: %v", err)
+					log.Printf("unable to convert metric %v values %v according to %v in TypesDB: %v", state, ifValues, ds, err)
 					continue
 				}
 				vl.Values = v

Produces a message like this:

2017/09/08 23:15:11 unable to convert metric hostname/plugin-instance/type-type_instance values [0 20690 0 48630396 0] according to &{type [{field1 api.Counter 0 NaN} {field2 api.Counter 0 NaN} {field3 api.Counter 0 NaN} {field4 api.Counter 0 NaN} {bytesdrop api.Counter 0 NaN} {field5 api.Counter 0 NaN}]} in TypesDB: len(args) = 5, want 6

... which I can then track down to a source very easily. Possibly the text presentation of "ds" is more detailed than it needs to be, but this doesn't get in the way of resolving the problem.

Guidance on reading collectd xml

As I understand it, most collectd monitors allow them to be configurable by defining a XML definition.

I was wonder if that assumption is correct and how you would write a plugin to understand those values.

Any advice would be wonderful

"unknown type" in Buffer.writeValues()

Turns out that Parse() can handle counter metrics, but Buffer.writeValues() can't write them.

panic: unknown type

goroutine 1 [running]:
collectd.org/network.Fuzz(0x7fe8fa833000, 0xf, 0x200000, 0x10)
        /tmp/go-fuzz-build999482993/src/collectd.org/network/fuzz.go:47 +0x339
github.com/dvyukov/go-fuzz/go-fuzz-dep.Main(0x652568)
        /usr/local/google/home/octo/go/src/github.com/dvyukov/go-fuzz/go-fuzz-dep/main.go:47 +0x13d
main.main()
        /tmp/go-fuzz-build999482993/src/go-fuzz-main/main.go:10 +0x2a

Release 0.4.0

It looks like it's been a long time since a release was tagged. With Go modules, git tags are used to lock vendoring by default. There's a lot of changes. Maybe time to cut a new version.

Package config: Unmarshal: Add support for field tags.

Similar to encoding/json and other packages, it would be nice to use struct field tags to control the unmarshalling behavior. For example:

• Write block values (arguments of config blocks) to an arbitrary field (currently hard-coded to Args).
• Control if and where to store Block.Key (used to look up fields in a struct but not stored in the target).

Package plugin: populate ValueList.DSNames when calling write callbacks.

Plugin writes do not include new dsnames but instead default to value. To confirm this issue, I used the following code from the docs but changed the dsname. I listened for the messages using the amqp1.

package main

import (
        "context"
        "fmt"
        "time"

        "collectd.org/api"
        "collectd.org/plugin"
)

type examplePlugin struct{}

func (examplePlugin) Read(ctx context.Context) error {
        vl := &api.ValueList{
                Identifier: api.Identifier{
                        Host:   "example.com",
                        Plugin: "goplug",
                        Type:   "gauge",
                },
                Time:     time.Now(),
                Interval: 10 * time.Second,
                Values:   []api.Value{api.Gauge(42)},
                DSNames:  []string{"read"},
        }
        if err := plugin.Write(ctx, vl); err != nil {
                return fmt.Errorf("plugin.Write: %w", err)
        }

        return nil
}

func init() {
	plugin.RegisterRead("example", examplePlugin{})
}

func main() {}

Received Message

[
  {
    "values": [
      42
    ],
    "dstypes": [
      "gauge"
    ],
    "dsnames": [
      "value"
    ],
    "time": 1591894480.918,
    "interval": 10,
    "host": "example.com",
    "plugin": "goplug",
    "plugin_instance": "",
    "type": "gauge",
    "type_instance": ""
  }
]

Expected Message

[
  {
    "values": [
      42
    ],
    "dstypes": [
      "gauge"
    ],
    "dsnames": [
      "read"
    ],
    "time": 1591894480.918,
    "interval": 10,
    "host": "example.com",
    "plugin": "goplug",
    "plugin_instance": "",
    "type": "gauge",
    "type_instance": ""
  }
]

The dsnames field of the message does not container read, but rather the default string value

checksum mismatch with v0.5.0

The sum.golang.org reports hashes of:

collectd.org v0.5.0 h1:y4uFSAuOmeVhG3GCRa3/oH+ysePfO/+eGJNfd0Qa3d8=
collectd.org v0.5.0/go.mod h1:A/8DzQBkF6abtvrT2j/AU/4tiBgJWYyh0y/oB/4MlWE=

However, if a user attempts to use collectd.org and run GOPROXY=direct go mod tidy, meaning directly download the package from the version control system (i.e. github) as opposed to the module proxy, the result is a checksum mismatch:

go: downloading collectd.org v0.5.0
verifying [email protected]: checksum mismatch
	downloaded: h1:mRTLdljvxJNXPMMO9RSxf0PANDAqu/Tz+I6Dt6OjB28=
	go.sum:     h1:y4uFSAuOmeVhG3GCRa3/oH+ysePfO/+eGJNfd0Qa3d8=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

This can be reproduced with:

package main

import _ "collectd.org/api"

func main() {}

module test

go 1.17

require collectd.org v0.5.0

This was reported to me via influxdata/telegraf#10408

Licensing terms

Hi @octo

Could you please elaborate on the License for the project? Specifically interested in https://github.com/collectd/go-collectd/tree/master/plugin

Thank you!

"index out of range" in decryptAES256()

panic: runtime error: index out of range

goroutine 1 [running]:
collectd.org/network.decryptAES256(0x7f2c77702004, 0x1, 0x1ffffc, 0x7f2c788a7a80, 0x4c20801e120, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/go-fuzz-build146054357/src/collectd.org/network/crypto.go:216 +0x1122
collectd.org/network.parseEncryptAES256(0x7f2c77702004, 0x1, 0x1ffffc, 0x7f2c788a7a80, 0x4c20801e120, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /tmp/go-fuzz-build146054357/src/collectd.org/network/parse.go:221 +0x9d
collectd.org/network.parse(0x7f2c77702000, 0x5, 0x200000, 0x0, 0x7f2c788a7a80, 0x4c20801e120, 0x0, 0x0, 0x0, 0x0, ...)
        /tmp/go-fuzz-build146054357/src/collectd.org/network/parse.go:103 +0xcc1
collectd.org/network.Parse(0x7f2c77702000, 0x5, 0x200000, 0x7f2c788a7a80, 0x4c20801e120, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /tmp/go-fuzz-build146054357/src/collectd.org/network/parse.go:34 +0x97
collectd.org/network.Fuzz(0x7f2c77702000, 0x5, 0x200000, 0x3)
        /tmp/go-fuzz-build146054357/src/collectd.org/network/fuzz.go:36 +0x112
github.com/dvyukov/go-fuzz/go-fuzz-dep.Main(0x652508)
        /usr/local/google/home/octo/go/src/github.com/dvyukov/go-fuzz/go-fuzz-dep/main.go:47 +0x13d
main.main()
        /tmp/go-fuzz-build146054357/src/go-fuzz-main/main.go:10 +0x2a
exit status 2

Unable to send metrics using network.Client

I'm trying to send some metrics to a server from my application but I've been unsuccessful. I've been able to send it using CollectD just fine from the same location so it's likely not a networking issue. Is there something wrong with my code? I used your test_client.go to help me. Or is there an issue with the library?

package main

import (
    "collectd.org/api"
    "collectd.org/network"
    "collectd.org/exec"
    "net"
    "time"
    "log"
)

func main() {
    options := network.ClientOptions{
        SecurityLevel: network.Encrypt,
        Username: "my_username",
        Password: "my_password",
    }

    conn, err := network.Dial(net.JoinHostPort("23.x.x.x", "25825"), options)
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    vl := api.ValueList{
        Identifier: api.Identifier{
            Host:   "testing.bryan",
            Plugin: "cpu",
            Type:   "cpu",
        },
        Time:     time.Now(),
        Interval: time.Second * 1,
        Values:   []api.Value{api.Gauge(42)},
    }    

    exec.Putval.Write(vl)

    for {
        if err := conn.Write(vl); err != nil {
            log.Fatal(err)
        }
        conn.Write(vl)
        log.Println("Metrics Sent")
        time.Sleep(time.Second * 1)
    }
}

This is the config for my network plugin for collectd

LoadPlugin network

<Plugin network>
        <Server "23.x.x.x" "25825">
                SecurityLevel "Encrypt"
                Username "my_username"
                Password "my_password"
        </Server>
</Plugin>

Package config: Add types for common configuration needs.

The "config".Unmarshaler interface can be used to implement unmarshaling behavior that differs from the default. That can be used to support some recurring needs:

• A type Port uint16 that accepts numeric and string values. If a string, "net".LookupPort is used to convert it to an integer. If the numeric value is outside of the 1–65535 range, an error is returned.
• A type ServiceName string that is the inverse: a numeric value (in the 1–65535 range) is converted to a string.
• Control mapping of multiple values onto a scalar type. For example, when mapping ["foo", "bar", "qux"] to a single string, should this result in an error (current behavior), the first value ("foo"), the last value ("qux"), or a random value? Special types could handle that. The "last value wins" / value is overwritten behavior is not uncommon in C plugins.
• Provide minimum / maximum values for numeric values.
• Check string values with a regular expression.

type *tar.Header has no field or method PAXRecords

Hi,

When I try to test archive, diff/walking and services/tasks, they faileds with

+ go test -compiler gc -ldflags ' -extldflags '\''-Wl,-z,relro  '\'''
Testing archive…
~/build/BUILD/containerd-1.0.1/_build/src/github.com/containerd/containerd/archive ~/build/BUILD/containerd-1.0.1
# github.com/containerd/containerd/archive
./tar.go:406:10: hdr.PAXRecords undefined (type *"github.com/dmcgowan/go-tar".Header has no field or method PAXRecords)
./tar.go:407:8: hdr.PAXRecords undefined (type *"github.com/dmcgowan/go-tar".Header has no field or method PAXRecords)
./tar.go:409:7: hdr.PAXRecords undefined (type *"github.com/dmcgowan/go-tar".Header has no field or method PAXRecords)
./tar.go:535:29: hdr.PAXRecords undefined (type *"github.com/dmcgowan/go-tar".Header has no field or method PAXRecords)
FAIL    github.com/containerd/containerd/archive [build failed]

Thank you for your help

Obtaining an info from types.db

When obtaining metrics from collectd with Prometheus time series database system, using collectd binary protocol misses some important information - the names of data sources if dataset contains metrics from multiple sources. This is the info you'll find from types.db(5) file.

Parsing types.db should probably belong to go-collectd.

collectd.org is down

says "refuse to connect".

go get collectd.org hence does not work.

RFC: Repo location for 'Go' based plugins

There was a discussion on mailing list based on @octo 's RFC now that collectd.org/plugin is available. Moving this to github to enable better conversation and possibly make a decision soon.

Two aspects that were raised in the mailing list:

• For Go-based plugins, i.e. plugins based on "collectd.org/plugin", create a separate repository, e.g.
  github.com/collectd/go-plugins, and maintain those there
• Instead unify the golang plugins and go-collectd packages under same directory structure. Meaning provide a home for golang plugins under "github.com/collectd/go-collectd/plugins". This would mean either moving "collectd.org/plugin" to another location or renaming it.

Thoughts?

Suggestion: Continuous Fuzzing

Hi, I'm Yevgeny Pats Founder of Fuzzit - Continuous fuzzing as a service platform.

I saw that you implemented Fuzz targets but they are currently not running as part of the CI.

We have a free plan for OSS and I would be happy to contribute a PR if that's interesting.
The PR will include the following

• Continuous Fuzzing of master branch which will generate new corpus and look for new crashes
• Regression on every PR that will run the fuzzers through all the generated corpus and fixed crashes from previous step. This will prevent new or old bugs from crippling into master.

You can see our basic example here and you can see an example of "in the wild" integration here.

Let me know if this is something worth working on.

Cheers,
Yevgeny

Package config: Implement the "encoding".TextMarshaler interface.

The result should be a format that collectd could then parse. Supporting TextUnmarshaler would also be great. Extra bonus points for writing a fuzz target for the config parsing.

malloc() issue with more than 3 values

Hi @octo,

I ran into a strange error when trying to implement a plugin using collect.org/api/plugin.

When I send more than 3 values in []api.Value{}, I get malloc() errors in Collectd.

For example:

package main

import (
    "time"

    "collectd.org/api"
    "collectd.org/plugin"
)

type Realloc3 struct{}

func (Realloc3) Read() error {
    if err := plugin.Write(&api.ValueList{
        Identifier: api.Identifier{
            Host:         "example",
            Plugin:       "realloc3",
            Type:         "realloc3",
            TypeInstance: "demo",
        },
        Time:     time.Now(),
        Interval: 10 * time.Second,
        DSNames:  []string{"a", "b", "c"},
        Values:   []api.Value{api.Derive(1), api.Derive(2), api.Derive(3)},
    }); err != nil {
        plugin.Error(err)
    }

    return nil
}

func init() {
    plugin.RegisterRead("realloc3", &Realloc3{})
}

func main() {}

gives me:

# CGO_CPPFLAGS='-I/usr/include/collectd/core/daemon -I/usr/include/collectd/core' go build -buildmode=c-shared -o realloc3.so realloc3.go

# collectd -f
Initialization complete, entering read-loop.
tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
^CExiting normally.
collectd: Stopping 5 read threads.
collectd: Stopping 5 write threads.

but with (which basically adds a fourth value):

package main

import (
    "time"

    "collectd.org/api"
    "collectd.org/plugin"
)

type Realloc4 struct{}

func (Realloc4) Read() error {
    if err := plugin.Write(&api.ValueList{
        Identifier: api.Identifier{
            Host:         "example",
            Plugin:       "realloc4",
            Type:         "realloc4",
            TypeInstance: "demo",
        },
        Time:     time.Now(),
        Interval: 10 * time.Second,
        DSNames:  []string{"a", "b", "c", "d"},
        Values:   []api.Value{api.Derive(1), api.Derive(2), api.Derive(3), api.Derive(4)},
    }); err != nil {
        plugin.Error(err)
    }

    return nil
}

func init() {
    plugin.RegisterRead("realloc4", &Realloc4{})
}

func main() {}

I get:

# CGO_CPPFLAGS='-I/usr/include/collectd/core/daemon -I/usr/include/collectd/core' go build -buildmode=c-shared -o realloc4.so realloc4.go

# collectd -f
Initialization complete, entering read-loop.
collectd: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.
Aborted

# collectd -f
Initialization complete, entering read-loop.
*** Error in `collectd': malloc(): memory corruption: 0x00007fca08000900 ***
tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
Aborted

Did I miss something important or did I do something wrong?

Kind regards,
Vincent

Writing bad stream should return an error

I use go 1.5.2 and see the following:

[   60s] /usr/lib64/go/pkg/tool/linux_amd64/link -o $WORK/collectd.org/network/_test/network.test -L $WORK/collectd.org/network/_test -L $WORK -L /home/abuild/rpmbuild/BUILD/go/pkg/linux_amd64 -w -extld=gcc -buildmode=exe $WORK/collectd.org/network/_test/main.a
[   60s] $WORK/collectd.org/export/_test/export.test
[   61s] $WORK/collectd.org/api/_test/api.test
[   61s] ok     collectd.org/api    0.012s
[   61s] ok     collectd.org/cdtime 0.009s
[   61s] ok     collectd.org/exec   0.009s
[   61s] ok     collectd.org/export 0.023s
[   61s] $WORK/collectd.org/format/_test/format.test
[   61s] ok     collectd.org/format 0.009s
[   62s] $WORK/collectd.org/network/_test/network.test
[   62s] --- FAIL: TestUnknownType (0.00s)
[   62s]    buffer_test.go:159: Writing bad stream should return an error
[   62s] FAIL
[   62s] FAIL   collectd.org/network    0.013s

please version and tag this project

Hello,

Can you please tag and version this project?

I am the Debian Maintainer for go-collectd and versioning would help Debian keep up with development.

commit 54def43854675971ffc402ef91f24280f6bf4879 breaks with golang 1.6

../collectd.org/api/main.go:5:2: cannot find package "context" in any of:
/root/go/src/vendor/context (vendor tree)
/usr/lib/go/src/context (from $GOROOT)
/root/go/src/context (from $GOPATH)

Maybe problem similar to this one
https://groups.google.com/forum/#!topic/golang-nuts/kpmMRtqvCiQ

collectd server crashes with empty listening address or address without host part

See prometheus/collectd_exporter#18.

When calling ListenAndWrite() on a collectd server that has either an empty Addr or one that only contains the port (:12345), there's an index-out-of-range panic:

panic: runtime error: index out of range

goroutine 10 [running]:
net.IP.IsMulticast(0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/ip.go:132 +0x9b
collectd.org/network.(*Server).ListenAndWrite(0xc2080505a0, 0x0, 0x0)
    /home/julius/gosrc/src/github.com/prometheus/collectd_exporter/.build/gopath/src/collectd.org/network/server.go:49 +0x125
main.func·001()
    /home/julius/gosrc/src/github.com/prometheus/collectd_exporter/main.go:248 +0x27
created by main.startCollectdServer
    /home/julius/gosrc/src/github.com/prometheus/collectd_exporter/main.go:249 +0x2a9

This is because net.ResolveUDPAddr() returns a nil IP when the host part is missing here:

https://github.com/collectd/go-collectd/blob/master/network/server.go#L43

Given this code:

https://github.com/collectd/go-collectd/blob/master/network/server.go#L38-L41

...it looks like the code is intended to work with empty / port-only addresses, so this looks like something that should be fixed in go-collectd.

Setting the the size of the operating system's receive buffer

Default size of the receive buffer OSes are assigning to connections isn't often big enough for receiving a lot of traffic and it needs to be adjusted. There isn't a way to do it at the moment from application using go-collectd and I have to patch go-collectd itself - it shouldn't be necessary.

Cannot perform fuzzing in external applications which include `collectd.org/network`

Cannot perform fuzzing in external applications (using gofuzz build tag) which include collectd.org/network.

It is caused by network/fuzz.go failing:

> go test -v -tags gofuzz ./network/... 
network/fuzz.go:43:20: not enough arguments in call to s1.Write
        have (*api.ValueList)
        want (context.Context, *api.ValueList)
network/fuzz.go:58:20: not enough arguments in call to s2.Write
        have (*api.ValueList)
        want (context.Context, *api.ValueList)

It has been failing since this change: https://github.com/collectd/go-collectd/pull/21/files:
Two calls in network/fuzz.go to func (b *Buffer) Write(_ context.Context, vl *api.ValueList) error don't contain context.Context argument.

Package network: Buffer.Write() does not support api.Counter metrics.

The appropriate case is missing in the writeValues() method.

Cannot install `collectd.org/api`

Reported to collectd/collectd.github.io#3 since I believe the issue was caused by the most recent deployment, but linking it here for awareness.

Installation of the package appears broken.

> go get collectd.org
go: unrecognized import path "collectd.org": parse https://collectd.org/?go-get=1: no go-import meta tags ()

> go get collectd.org/api
go: unrecognized import path "collectd.org/api": reading https://collectd.org/api?go-get=1: 404 Not Found

FR: Add option to specify interval to plugin.RegisterRead.

I'm writing a simple plugin that needs to be run less often than every 10s, in order to achieve this I have the following values:

	vl := api.ValueList{
		Identifier: api.Identifier{
			Host:         exec.Hostname(),
			Plugin:       "foo",
			Type:         "gauge",
			TypeInstance: "ticker",
		},
		Time: time.Now(),
		Interval: 60 * time.Second,
		DSNames:  pairs,
		Values:   vals,
	}

Nevertheless, the plugin runs every 10 seconds.

I also tried overriding using the <LoadPlugin> method in collectd.conf without success.

Am I hitting a bug or did I miss something?

Thanks