synfinatic / netflow2ng

NetFlow v9 collector for ntopng

netflow2ng

NetFlow v9 collector for ntopng

TL;DR

ntopng is a free/commercial NetFlow/sFlow analysis console suitible for a variety of use cases. However, if you want to collect NetFlow or sFlow data and load that into ntopng you currently have no choice but to spend 199Euro on nProbe which in my case is more expensive than the Ubiquiti USG that I wanted to collect NetFlow stats from.

Hence, I created netflow2ng.

Installing

Build From Source

1. Make sure you have a recent version of go. I used 1.14.2. Older versions may have problems.
2. git clone https://github.com/synfinatic/netflow2ng.git
3. cd netflow2ng
4. make
5. The binary should now be in the dist directory. Copy it somewhere appropriate and create the necessary startup script(s).

Install via Docker

1. Pull the latest docker image
2. Use the optional docker-compose.yaml file

Configuration

1. For a list of configuration arguments, run netflow2ng -h
2. Configure your network device(s) to send NetFlow stats to netflow2ng
3. Configure your ntopng service to read from netflow2ng: ntopng -i tcp://192.168.1.1:5556 where "192.168.1.1" is the IP address of your netflow2ng server.

Features

• Collect NetFlow v9 stats from one or more probes
• Run a ZMQ Publisher for ntopng to collect metrics from
• Prometheus metrics
• NetFlow Templates

NetFlow v9 Support

netflow2ng utilizes goflow for NetFlow decoding. For more information on what NetFlow fields are supported in netflow2ng, please read the goflow docs.

sFlow/IPFIX/etc support?

In theory, adding sFlow/IPFIX/NetFlow v5 support should be pretty trivial, but isn't something I plan on doing due to lack of hardware for testing/need.

How is netflow2ng different from nProbe?

• Not 199Euro
• Doesn't support any probe features (sniffing traffic directly)
• Can't write stats to MySQL/disk or act as a NetFlow proxy
• Not tested with lots of probes or on 10Gbit networks
• Targeted for Home/SOHO use.
• No commercial support, etc.
• May not support the latest versions/features of ntopng
• Written in GoLang instead of C/C++

netflow2ng's Issues

Huawei IP netstream

when i use a huawei ne40 to export netflow v9 data to netflow2ng i don't have the correct information.
on an interface with approximately 5gb of traffic it is displayed as if it had only 5mbps of traffic.

thanks

In Mikrotik router this information are ok

IPFIX support (seems to work fine)

I successfully tested netflow collection from OpenBSD's pflow(4) interface using IPFIX.
Initially I expected some minor tweaks in the source code, however it seems to work fine out of the box.
Just FYI, the README can be updated in case. 😄

sFlow Support

How difficult would it be to add sFlow support? I was looking to implement this as a drop in replacement for nProbe and its ridiculous fee, but our Brocade uses sFlow instead of netflow.

runtime error: invalid memory address or nil pointer dereference

Hello Sir, I can run the program successfully on my compiling machine, but I got the below error message when I move the executable file to another machine. Any idea on this?

---

root@UBUNTU_18:/mnt/ubuntu/netflow2ng-master/dist# ./netflow2ng--linux-x86_64
INFO[0000] Starting netflow2ng
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7cd9eb]

goroutine 1 [running]:
github.com/pebbe/zmq4.(*Context).NewSocket(0x0, 0x1, 0xc000099130, 0xc0000d0c00, 0x8a87e0)
/root/go/pkg/mod/github.com/pebbe/[email protected]/zmq4.go:834 +0x3b
github.com/synfinatic/netflow2ng/transport.StartZmqProducer(0x8cbabc, 0xc, 0x8c9086, 0x4, 0x9742e0, 0xc00010a460, 0xc0000d7df8, 0x2d4b439a, 0xf9a6cd421a4e21ad)
/mnt/ubuntu/netflow2ng-master/transport/zmq.go:105 +0xc7
github.com/synfinatic/netflow2ng/transport.StartZmqProducerFromArgs(...)
/mnt/ubuntu/netflow2ng-master/transport/zmq.go:133
main.main()
/mnt/ubuntu/netflow2ng-master/cmd/netflow2ng/netflow2ng.go:78 +0x243

memory leak?

showing 2.5GB resident memory used after 24 days?

arm docker support

Can we get arm64 docker support for rasp 4 and 5s?

Issue building from source

I tried to build from source and keep getting the following errors on my Ubuntu server

mkdir -p dist/
go build -ldflags='-X "main.Version=0.0.5" -X "main.Delta=''" -X "main.Buildinfos=2023-12-08T21:58:11+0000" -X "main.Tag=v0.0.5-3-ga5c82b5" -X "main.CommitID=a5c82b51ab1be0a18dd6363e679869d05afd463e"' -o dist/netflow2ng-0.0.5 ./cmd/...
# github.com/pebbe/zmq4
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:10:4: undefined: State
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:11:9: undefined: State
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:21:16: undefined: Socket
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:23:12: undefined: Poller
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:61:34: undefined: Socket
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:61:49: undefined: State
../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:68:37: undefined: Socket
../go/pkg/mod/github.com/pebbe/[email protected]/utils.go:17:12: undefined: Socket
../go/pkg/mod/github.com/pebbe/[email protected]/utils.go:24:12: undefined: Socket
../go/pkg/mod/github.com/pebbe/[email protected]/utils.go:28:12: undefined: Socket
../go/pkg/mod/github.com/pebbe/[email protected]/utils.go:28:12: too many errors
make: *** [Makefile:60: dist/netflow2ng-0.0.5] Error 1

It would be better if a deb or binary package is made available in addition to the Docker one.

Thanks
ZeeKay

praise: thank you very much!!

saved me 199 bucks!!

Question: console logging

Hi,

Firstly, thank you for the timer you invested in creating netflow2ng and supporting it.

I've been struggling with getting it to work with ntopng. I have ntopng, redis, and netflow2ng, each in their own container with a common network. Each container can see the others and I have softflowd on pfsense sending packets to the netflow2ng (exposed on 0.0.0.0:2055).

My docker compose file is here:

https://raw.githubusercontent.com/robgarrett/Docker/master/ntopng/docker-compose.yml

If I run docker compose up flowd, should I be able to see traffic on the console. I was able to see data in the metrics HTTP page, but nothing that shows active flows.

ntopng shows no flows.

I'd appreciate any help you can offer.

Binary build on github CI.

Would it be possible to add a binary release build by the means of github actions?

Since Dockerfile for building already exists, it should be fairly simple to add support for automatically making "releases" with binary files when a tag is pushed?

I am not very familiar with the flow, but it should be something like:

1. Use Github Actions to run a script on new tags. Like this guy does: https://github.com/codebam/resume/blob/master/.github/workflows/build.yml
2. In that script use ghr to create a github release. https://pkg.go.dev/github.com/lowply/ghr
3. The pattern for running on new tags is the following: https://stackoverflow.com/questions/61891328/trigger-github-action-only-on-new-tags

The command for building is already in the Makefile, so this should not be hard?

ntopng local host names

Hello.
First of all, thank you for creating netflow2ng.
Now i started receive flows to ntopng.
But unfortioantly a can't receive name of my local hosts.
There are ip adress instead of name of local host.
I configured dns-mode in ntopng.conf file (i used 1, 2 and 3) then restarted ntopng but it didn't help.

I get flows from mikrotik (trafic flow v9 and port 2055)

Please could you help and tell how can i get host names instead of ip addresses?
Thank's in advance.

Compile fails on USG & linux cross-compile?

I really love this project, but I am failing to get the code compiled.

I assume that you are compiling your code directly on the USG as you did not mention any cross-compiling?

If so:
Mine does not have "go" installed. I tried it to install and it was not available. I've added these sources to sources.list (had to use the archive as wheezy is no longer maintained and I believe USG is based on wheezy?):

• deb http://archive.debian.org/debian wheezy main contrib non-free # wheezy #
• deb http://archive.debian.org wheezy main # wheezy-security #

Trying to install golang-go fails:

• sudo apt-get install golang-go
• Package golang-go is not available, but is referred to by another package.
• This may mean that the package is missing, has been obsoleted, or
• is only available from another source

So I thought I'd try to cross-compile netflow2ng for "linux mips64" on my Linux system, but that doesn't work either:

• GOOS=linux GOARCH=mips make
• ...

• github.com/pebbe/zmq4

• ../../go/pkg/mod/github.com/pebbe/[email protected]/reactor.go:10:4: undefined: State

Could you shed some light on how you got yours to compile? Thx!

runtime error: invalid memory address or nil pointer dereference

# ./netflow2ng-v0.0.2-8-g887b99b-linux-x86_64 -addr 192.168.1.2 -port 2055
INFO[0000] Starting netflow2ng
INFO[0001] Started ZMQ listener on: tcp://*:5556
INFO[0001] Listening on UDP 192.168.1.2:2055             Type=NetFlow
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8619cc]

goroutine 1 [running]:
github.com/cloudflare/goflow/v3/utils.UDPRoutine(0x9b4cb2, 0x7, 0xc000152090, 0x1, 0x7ffcc729f60d, 0xb, 0x807, 0x0, 0xa74ac0, 0xc000132620, ...)
        /root/go/pkg/mod/github.com/cloudflare/goflow/[email protected]/utils/utils.go:184 +0x100c
github.com/cloudflare/goflow/v3/utils.(*StateNetFlow).FlowRoutine(0xc000067280, 0x1, 0x7ffcc729f60d, 0xb, 0x807, 0x0, 0x2, 0x0)
        /root/go/pkg/mod/github.com/cloudflare/goflow/[email protected]/utils/netflow.go:354 +0x15a
main.main()
        /root/netflow2ng/cmd/netflow2ng/netflow2ng.go:88 +0x4ca

Running on CentOS 8 4.18.0-147.el8.x86_64 #1 SMP Wed Dec 4 21:51:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux inside HyperV.
Thought it might be related to #8 but this was compiled and running on same machine.

Can't get flows from mikrotik

Hello! Getting some issue here.
Compiled as described in readme.
Launching:

root@adguard:~/netflow2ng/dist# ./netflow2ng-v0.0.2-8-g887b99b-linux-x86_64 -loglevel=debug -addr 192.168.69.22 -port 2056
INFO[0000] Starting netflow2ng
INFO[0001] Started ZMQ listener on: tcp://*:5556
INFO[0001] Listening on UDP 192.168.69.22:2056           Type=NetFlow

Verifying ports:

tcp        0      0 0.0.0.0:5556            0.0.0.0:*               LISTEN      18320/./netflow2ng-
tcp6       0      0 :::8080                 :::*                    LISTEN      18320/./netflow2ng-
udp        0      0 192.168.69.22:2056      0.0.0.0:*                           18320/./netflow2ng-

ntopng 5, added in config:

-i=tcp://127.0.0.1:5556

Can see interface in webui now but everything is empty and ntopng reports "no data on interface" alert.

ZMQ is reachable:

lex@adguard:~$ telnet 127.0.0.1 5556
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

Can also see netflow packets with tcpdump.
Router is configured to send netflow v9 to 192.168.69.22:2056

Migrate to goflow2?

Probably better supported long term: https://github.com/netsampler/goflow2

Not working anymore with Netflow v9?

I pulled the docker image and run it with docker run <image id> but no flow is collected nor exported

not sending zmq message?

using openvswitch with IPFIX pointed to netflow2ng, and started ntopng with zmq pointed to netflow2ng,
but no data shown on ntopng,
command I use: ./dist/netflow2ng-0.0.5 -m 0.0.0.0:8009 -z tcp://127.0.0.1:5556 --log-level debug --reuse

zmq format for ntopng incorrect? (openflow support)

I am trying to push some data into ntopng 6.0, but it doesn't receive any flows, despite receiving zmq messages:

So I went ahead and compared the output of netflow2ng (0.0.5) with the output of nprobe (10.5) and they seem to be very different:
netflow2ng output

{
    "1": 1421,
    "10": 0,
    "11": 33923,
    "12": "61.250.232.215",
    "130": "192.168.0.1",
    "14": 0,
    "15": "61.250.232.215",
    "16": 0,
    "17": 0,
    "2": 6,
    "21": 1706022328,
    "22": 1706022318,
    "29": 0,
    "30": 0,
    "32": 0,
    "4": 6,
    "44": 0,
    "45": 0,
    "5": 0,
    "52": 0,
    "54": 0,
    "58": 0,
    "59": 0,
    "6": 27,
    "60": 4,
    "61": 0,
    "7": 22,
    "8": "89.0.69.55",
    "80": "00:00:00:00:00:00",
    "81": "00:00:00:00:00:00",
    "88": 0,
    "89": 0
}

nprobe output

{
    "56": "00:00:00:00:00:00",
    "57": "00:00:00:00:00:00",
    "10": 0,
    "14": 0,
    "58": 0,
    "8": "192.168.0.1",
    "12": "192.168.0.4",
    "7": 53,
    "11": 51996,
    "27": "::",
    "28": "::",
    "60": 4,
    "4": 17,
    "35632.118": "5",
    "35632.560": 0,
    "1": 64,
    "2": 1,
    "23": 0,
    "24": 0,
    "22": 1706022259,
    "21": 1706022259,
    "35632.78": 0,
    "35632.79": 0,
    "35632.509": 0,
    "35632.527": 0,
    "130": "192.168.0.1",
    "61": 2,
    "34": 1,
    "42": 18,
    "35632.471": "192.168.3.28",
    "35632.122": "",
    "58051": 1,
    "225": "0.0.0.0",
    "226": "0.0.0.0",
    "227": 0,
    "228": 0
}

I am using the @NTOPNG@ template on nprobe, which expands to
%IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORT

Looking at https://github.com/cloudflare/goflow/tree/master/decoders/netflow, I can see how I could manipulate the JSON output by incorporating those additional fields into https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go. However, I am not sure how those "combined" fields (i.e. "35632.527") are created / what they mean. Also, I am not sure why for example the MAC addresses use different identifiers on the nprobe string (for example 56 - IPFIX_FIELD_sourceMacAddress, 57 - IPFIX_FIELD_postDestinationMacAddress) compared to the identifiers netflow2ng uses (for example 80 - IPFIX_FIELD_destinationMacAddress, 81 - IPFIX_FIELD_postSourceMacAddress). Same goes for a couple of the other fields - I guess the template used by netflow2ng at the moment is an old version, compatible with previous versions of ntopng!?

So, for now at least, ntopng doesn't receive any flows from netflow2ng. Could you give any pointers on how to incorporate the changes neccessary to support a more current ntopng version?

Which version of Ntopng work well with current Netflow2ng

Hello Sir,

I want to try this project , and I want to know which version of Ntopng I should use to avoid the unexpected error on latest Ntopng. I also see some compile error report in closed Issue. Could you please suggest which OS (Fedora , Centos , Debian ?) and which version I should use to avoid unexpected compile error.

Thank you.

More explanation on how to use

Very glad to see you created this project, I have same situation and commercial nProbe has 25000 flows limitation, I can not use this for my home router since it will reach out it limitation in 30min.

Would you give more information how to utilize these codes?

Ports must be open?

Hello
docker run synfinatic/netflow2ng
Run the command and it automatically showed me the following output:
level=info msg="Starting netflow2ng"
level=info msg="Started ZMQ listener on: tcp://*:5556"
level=info msg="Listening on UDP 0.0.0.0:2055" Type=NetFlow

Then run the ntop command, to start ntop on port 5556
Collecting flows on tcp://172.16.88.7:5556

It showed me the following line, but in the web browser, it doesn't show any result.. my sopgos xg firewall directs netflow traffic to port 2055 of my server.. did i miss something? I opened the udp ports and I can't get it to work.

Some help?

Thank you