florianl / go-conntrack

c-binding free API for golang to communicate with the conntrack subsystem

go-conntrack

This is go-conntrack and it is written in golang. It provides a C-binding free API to the conntrack subsystem of the Linux kernel.

Example

package main

import (
	"fmt"

	"github.com/florianl/go-conntrack"
)

func main() {
	nfct, err := conntrack.Open(&conntrack.Config{})
	if err != nil {
		fmt.Println("could not create nfct:", err)
		return
	}
	defer nfct.Close()

	// Get all IPv4 entries of the expected table.
	sessions, err := nfct.Dump(conntrack.Expected, conntrack.IPv4)
	if err != nil {
		fmt.Println("could not dump sessions:", err)
		return
	}

	// Print out all expected sessions.
	for _, session := range sessions {
		fmt.Printf("%#v\n", session)
	}
}

Requirements

• A version of Go that is supported by upstream

go-conntrack's Issues

filter on IPv6 (or IPv4) hides events from the other IP family

I am playing with the library, using the example code provided, but using RegisterFiltered

var filterNoIPv6Loopback = []ct.ConnAttr{
	{ct.AttrOrigIPv6Src,
		net.IPv6loopback
		net.CIDRMask(128, net.IPv6len*8),
		true,
	},
}

func ExampleNfctRegister(ctx context.Context) {
	nfct, err := ct.Open(&ct.Config{AddConntrackInformation: true})
	if err != nil {
		log.Fatal(err)
		return
	}
	go func() {
		<-ctx.Done()
		nfct.Close()
	}()

	if err := nfct.RegisterFiltered(ctx, ct.Conntrack, ct.NetlinkCtNew, filterNoIPv6Loopback, printer); err != nil {
		fmt.Println("could not register callback:", err)
		return
	}
	fmt.Println("registered")
}

func printer(c ct.Con) int {
	jsonOut, _ := json.MarshalIndent(c, "", "  ")
	fmt.Println(string(jsonOut))
	fmt.Println("-------------------------")
	return 0
}

func main() {
	ctx, ctxCancel := context.WithCancel(context.Background())
	ExampleNfctRegister(ctx)

	c := make(chan os.Signal, 1)
	signal.Notify(c, os.Interrupt)
	go func() {
		for sig := range c {
			_ = sig
			ctxCancel()
			return
		}
	}()

	fmt.Println("waiting for SIGINT signal")
	<-ctx.Done()
}

Seems that by registering a filter for an IPv6 IP, only IPv6 events are received, but no IPv4 event.
The opposite is also true: registering a IPv4 filter will exclude all IPv6 events.

By registering no filter, both IPv4 and IPv6 events are received.

Is this expected?

Unable to 'go get' versions after v0.1.0

In might be my go mod skills are deficient, but
$ go list -m -versions github.com/florianl/go-conntrack
github.com/florianl/go-conntrack v0.1.0
$ git tag
v0.1.0
v2.0.0
v3.0.0

but I can't build with v3.0.0.
$ mkdir new
$ cd new
$ cp ~/git/go-conntrack/example_DumpCPUStats_test.go d.go
$ go mod init
$ go build d.go
go: finding module for package github.com/florianl/go-conntrack
go: found github.com/florianl/go-conntrack in github.com/florianl/go-conntrack v0.1.0

github.com/florianl/go-conntrack

../../pkg/mod/github.com/florianl/[email protected]/conntrack_nonlinux.go:55:9: undefined: extractAttributes

Edit d.go to ct "github.com/florianl/go-conntrack/v3"
$ go build d.go
go: finding module for package github.com/florianl/go-conntrack/v3
d.go:6:2: module github.com/florianl/go-conntrack@latest found (v0.1.0), but does not contain package github.com/florianl/go-conntrack/v3

Any ideas?

Netlink socket read timeouts

Hi 👋
I'm not sure if this behavior is intentional or not so apologize in advance for the confusion. 😅
I noticed users of this library using Go >1.12 can set a read timeout for the underlying Netlink socket via the ReadTimeout configuration option.
However if a socket read operation times out, the polling Go routine will simply terminate and the registered callback will no longer receive data.
I believe these errors should be handled somehow as in

if opError, ok := err.(*netlink.OpError); ok {
	if opError.Timeout() || opError.Temporary() {
		// some backoff logic
		continue
	}
}

around

go-conntrack/conntrack.go

Lines 310 to 313 in f722615

	if err != nil {
	nfct.logger.Printf("receiving error: %v", err)
	return
	}

or perhaps there should be a way to know that the event stream is no longer being consumed.
Thanks!

ExampleNfct_RegisterFiltered Unknown message type

when I test func ExampleNfct_RegisterFiltered, the program print
Unknown message type: 0x00
map[]
is there something wrong with Func RegisterFiltered?

Please expose the netlink.Config.DisableNSLockThread option for configuration

I continue to develop my monitoring program using your library, and my rather pathological test case, which is the current release of the
edgex package, fires up about 18 docker containers that then proceed to furiously talk to each other (50,000 active
network connections).
I hit an interesting situation where my program would sit on about 40% CPU while monitoring; a large amount of the
processes CPU time being in runtime.futex (35%) and runtime.findrunnable (%18), which, is to my understanding are locking/scheduling related. I have 7-8 quite active go-routines running, on a quad core box.
I shutdown the docker containers, which normally triggered a failure on go-conntrack, which I then restarted.
I then restart the containers. My program now sat on about %20 CPU and those functions would drop to nearly nothing.
If I started my program before starting edgex, it would also be well behaved.
Looking at the different run profiles, I found that after restarting go-conntrack, I was not shutting down the worker go-routine so I had two sitting around.
To me I suspect this is related to the underlying netlink library thread locking its go-routine to clear up issues around
the setNS system call operating on a thread basis. I'm using setNS (and sock_diag) elsewhere, and I can appreciate
why you would want to lock your go-routine to a real thread. In my case I setNS, open a socket, then setNS back.
For me this seems to work (the socket is in the alternative namespace and I can keep on talking to it).

When I modified the arguments to netlink to set DisableNSLockThread to true, my problems seemed to go away.
I know what I'm doing elsewhere with setNS and am will to take the chance with turning this option on.
My problem is the go-conntrack library does not expose this option. Could I request you add it to you
conntrack.Config structure and pass it through.
I believe I'm seeing a 'semi-random' go-routine scheduling issue because a very active go-routine is being thread locked.

thanks

Question: Delete both IPv4 and IPv6 entry

Please does anyone knows how I can use this library to delete IPv4 and IPv6 conntrack entry?

Please help.

Thanks

API-design

The current API design makes the use of go-conntrack fairly hard to use, without deeper knowledge of conntrack. E.g. the returned ConnAttrs are from type []byte with variable size instead of proper Go types. So the returned values for AttrOrigIPv4Src could be an net.IP instead of []byte.

So in the future it could look something like:

type IPTuple struct {
       Src   net.IP
       Dst   net.IP
      ...
}

type Con struct {
    Origin      *IPTuple
    Reply      *IPTuple
    ID            *uint32
   ...
}

As not all attributes, which are possible, are provided every time, the returned Con struct contains pointers.These pointers can be nil, if the given attribute is not provided.

Using NetlinkCtDestroy do not work

use RegisterFiltered with NetlinkCtDestroy, no message will receive.

ICMP type getting overwritten.

I'm continuing to replicate conntrack(1) with your library and have hit another issue regarding ICMP type/code values. What seems to be happening is that inside extractTuple(), while it gets called twice for the two IP directions, in both cases it contains an ICMP protocol Tuple which gets decoded. The first (for a ping) has a type of 8, then the second has a type of 0, which overwrites the 8. My understanding of ICMP and how conntrack handles it is a little light, but the linux header files indicates that it makes no sense to have both a Orig/Repl for these fields. So I suspect the correct solution is to only update them when 'dir' is 0.
It seems to me that there should not even be a second embedded protocol tuple for ICMP.

This patch replicates the output from conntrack(1).

diff --git a/attribute.go b/attribute.go
index 15d4841..15effef 100644
--- a/attribute.go
+++ b/attribute.go
@@ -224,17 +224,29 @@ func extractProtocolTuple(conn Conn, dir int, data []byte) error {
                        eleType := map[int]ConnAttrType{dirOrig: AttrOrigPortDst, dirReply: AttrReplPortDst, dirMaster: AttrMasterPortDst}[dir]
                        conn[eleType] = attr.Data
                case ctaProtoIcmpID:
-                       conn[AttrIcmpID] = attr.Data
+                       if dir == 0 {
+                               conn[AttrIcmpID] = attr.Data
+                       }
                case ctaProtoIcmpType:
-                       conn[AttrIcmpType] = attr.Data
+                       if dir == 0 {
+                               conn[AttrIcmpType] = attr.Data
+                       }
                case ctaProtoIcmpCode:
-                       conn[AttrIcmpCode] = attr.Data
+                       if dir == 0 {
+                               conn[AttrIcmpCode] = attr.Data
+                       }
                case ctaProtoIcmpv6ID:
-                       conn[AttrIcmpID] = attr.Data
+                       if dir == 0 {
+                               conn[AttrIcmpID] = attr.Data
+                       }
                case ctaProtoIcmpv6Type:
-                       conn[AttrIcmpType] = attr.Data
+                       if dir == 0 {
+                               conn[AttrIcmpType] = attr.Data
+                       }
                case ctaProtoIcmpv6Code:
-                       conn[AttrIcmpCode] = attr.Data
+                       if dir == 0 {
+                               conn[AttrIcmpCode] = attr.Data
+                       }
                default:
                        fmt.Printf("Tuple %d is not yet implemented: %v\n", attr.Type, attr.Data)
                }

Problem with negative filters

Hi,
I really like this library and I would like to use it in my project, but I found a problem which is kind of show stopper for me.
I have a server with a lot of traffic, but I need to analyse small portion of it.

So I wanted to use RegisterFiltered method for filtration of traffic which is irrelevant for me. But I can't make it work.

I have created filters to remove traffic going from/to localhost in this example:

package main

import (
	"context"
	"github.com/florianl/go-conntrack"
	"log"
	"os"
	"time"
)

func main() {
	var err error

	logger := log.Logger{}
	logger.SetOutput(os.Stdout)

	nfct, err := conntrack.Open(&conntrack.Config{
		Logger:                  &logger,
		AddConntrackInformation: true,
	})
	if err != nil {
		logger.Fatalln("could not create nfct: %v", err)
	}

	var processor = func(c conntrack.Con) int {
		if c.Origin.Proto.DstPort == nil {
			logger.Printf("Port for %s is nil\n", c.Origin.Dst.String())
			return 0
		}

		logger.Printf("Conntrack connection: src: %s - dest: %s:%d\n", c.Origin.Src.String(), c.Origin.Dst.String(), *c.Origin.Proto.DstPort)
		return 0
	}

	filters := make([]conntrack.ConnAttr, 0)

	filterLocalhostSrc := conntrack.ConnAttr{Type: conntrack.AttrOrigIPv4Src, Data: []byte{0x7f, 0x0, 0x0, 0x1}, Mask: []byte{0xff, 0xff, 0xff, 0xff}, Negate: true}
	filterLocalhostDst := conntrack.ConnAttr{Type: conntrack.AttrOrigIPv4Dst, Data: []byte{0x7f, 0x0, 0x0, 0x1}, Mask: []byte{0xff, 0xff, 0xff, 0xff}, Negate: true}

	filters = append(filters, filterLocalhostSrc, filterLocalhostDst)

	err = nfct.RegisterFiltered(context.Background(), conntrack.Conntrack, conntrack.NetlinkCtNew|conntrack.NetlinkCtUpdate|conntrack.NetlinkCtDestroy, filters, processor)

	if err != nil {
		logger.Printf("could not register callback: %v\n", err)
	}

	time.Sleep(2 * time.Second)

	err = nfct.Close()
	if err != nil {
		logger.Printf("nfct close error: %v\n", err)
	}

	time.Sleep(1 * time.Second)

	logger.Println("closing")
}

But it doesn't work for me. This traffic is still forwarder to the userspace. Example from console:

Conntrack connection: src: 127.0.0.1 - dest: 127.0.0.1:8125
Conntrack connection: src: 127.0.0.1 - dest: 127.0.0.1:8125
Conntrack connection: src: 127.0.0.1 - dest: 127.0.0.1:55586

I noticed that following error is produced when I call the Close method:

receiving error: netlink receive: use of closed file
could not remove filter: netlink remove-bpf: setsockopt: bad file descriptor
could not unsubscribe from group: netlink leave-group: setsockopt: bad file descriptor

But I'm not sure if it is relevant.

I have also noticed that if I use positive filter (for example to get just traffic from localhost}, it seems that it works:

filterLocalhostSrc := conntrack.ConnAttr{Type: conntrack.AttrOrigIPv4Src, Data: []byte{0x7f, 0x0, 0x0, 0x1}, Mask: []byte{0xff, 0xff, 0xff, 0xff}, Negate: false}

So probably just negative filters don't work.

Any suggestion would be appreciated.

Thank you,
Martin

Building a prometheus exporter using this lib

Hi!

Would it me possible to build a tool like this: https://github.com/hiveco/conntrack_exporter using this library?

I haven't fully checked this project but the hiveco one does everything in this file: https://github.com/hiveco/conntrack_exporter/blob/master/src/connection_table.cc

Notification of Fatal Errors

I have an application using this library, and when on a heavily stressed system, I get a
'error: netlink receive: recvmsg: no buffer space available' error.
The library can be configured to log it, but I want to receive notification so I can act upon this
information.
I've appended the patch I'm using to give me what I'm after. I allow an error channel to be passed in with the Config structure.

diff --git a/conntrack.go b/conntrack.go
index febf035..3cb0d13 100644
--- a/conntrack.go
+++ b/conntrack.go
@@ -341,6 +341,9 @@ func (nfct *Nfct) register(ctx context.Context, t Table, groups NetlinkGroup, fi
}
}
nfct.logger.Printf("receiving error: %v", err)

•   		if nfct.errorChan != nil {
  

•   			nfct.errorChan <- err
  

•   		}
    		return
    	}
  

diff --git a/conntrack_gteq_1.12.go b/conntrack_gteq_1.12.go
index f5bdba4..6d098a3 100644
--- a/conntrack_gteq_1.12.go
+++ b/conntrack_gteq_1.12.go
@@ -44,5 +44,9 @@ func Open(config *Config) (*Nfct, error) {
nfct.setWriteTimeout = func() error { return nil }
}

• if config.Error != nil {

•   nfct.errorChan = config.Error
  

• }

• return &nfct, nil
  }
  diff --git a/conntrack_lt_1.12.go b/conntrack_lt_1.12.go
  index 37f3b3a..5e30728 100644
  --- a/conntrack_lt_1.12.go
  +++ b/conntrack_lt_1.12.go
  @@ -28,5 +28,9 @@ func Open(config *Config) (*Nfct, error) {
  nfct.setReadTimeout = func() error { return nil }
  nfct.setWriteTimeout = func() error { return nil }

• if config.Error != nil {

•   nfct.errorChan = config.Error
  

• }

• return &nfct, nil
  }
  diff --git a/types.go b/types.go
  index de46107..69b51ab 100644
  --- a/types.go
  +++ b/types.go
  @@ -24,6 +24,9 @@ type Config struct {
  // Time till a write action times out - only available for Go >= 1.12
  WriteTimeout time.Duration

• // If there is an error while using a handler (Register), the error

• // that closed the reader routine will be sent on this channel

• Error chan error
  // Interface to log internals.
  Logger *log.Logger
  }
  @@ -35,6 +38,10 @@ type Nfct struct {

  logger *log.Logger

• // If a fatal error occurs, the error will be written to this channel

• // before the reader go-routine exits

• errorChan chan error

• setReadTimeout func() error
  setWriteTimeout func() error
  }

Can not update ICMPv6 sessions

An issue has been reported, that ICMPv6 sessions can not be altered.

What is the minimum support Linux kernel version？

Hi, have a problem：
What is the minimum support Linux kernel version?

[Question] Is this project suitable for fast-changing rules?

Hi!

First, thanks for your work on this tool @florianl!

I have an use case where I'll be deleting like 50~100 rules per second, do you think this lib can do the trick? (if not, any other way of doing that?)

I've tried it and sometimes it's giving me a lot of delay to delete some rules, maybe I'm using it the wrong way... I'm trying it on an ubuntu 18 VM.

Thanks!

Issue around Delete events having zero content.

I've already responded on the other issue as to why Delete events were not registering, and I believe I have to solution as to why there is zero data.

In conntrack.go,

      func parseConnectionMsg(msg netlink.Message, reqType int) (Conn, error) {
           if msg.Header.Type&netlink.Error == netlink.Error {

should be

if msg.Header.Type == netlink.Error {

from the relevant linux header

     #define NLMSG_NOOP              0x1     /* Nothing.             */
     #define NLMSG_ERROR             0x2     /* Error                */
     #define NLMSG_DONE              0x3     /* End of a dump        */
     #define NLMSG_OVERRUN           0x4     /* Data lost            */

The current code does a bitmask on the Header.Type (uint16) with netlink.Error.
When a normal event comes in, it is 0x0100, so all is good. For delete, it is 0x0102, which is causing the code to bail and not decode. My reading of the headers is that it is a full 16bits, of numbers, not bitmap. Note that LMSG_DONE would also trigger your error condition.

#define NLMSG_MIN_TYPE 0x10 /* < 0x10: reserved control messages */

So anything above that should be ok. In a few places, you mask the reqType before making the call to parseConnectionMsg().
From netfilter/nfnetlink.h

/* netfilter netlink message types are split in two pieces:
 * 8 bit subsystem, 8bit operation.
 */
#define NFNL_SUBSYS_ID(x)       ((x & 0xff00) >> 8)
#define NFNL_MSG_TYPE(x)        (x & 0x00ff)
#define NFNL_SUBSYS_CTNETLINK 1

The SUBSYS is NFNL_SUBSYS_CTNETLINK, but I'm not sure what the MSG_TYPES are otherwise. I'm rather new to netlink/conntrack, but I'm reasonably sure this is why the events are not being decoded when they should be. I'm also not sure what would get broken with these changes

Deletion of git tags

Currently, this project has three different git tags

• v3.0.0
• v2.0.0
• v0.1.0

All three of them are not reflected in https://github.com/florianl/go-conntrack/blob/master/go.mod and vice versa. In addtion, this leads to confusion for some users - see #25.

As most users of this pkg are using go modules, I will delete these git tags to reduce confusion in the future.

Therefore, on September 1st 2020 I will delete the git tags.

If there are concerns with the removal of these git tags, please speak up.