ntop / ntopng Goto Github PK

Web-based Traffic and Security Network Traffic Monitoring

License: GNU General Public License v3.0

C++ 30.09% C 1.16% Shell 0.86% Ruby 0.01% Makefile 0.17% Lua 55.78% HTML 0.36% Roff 0.12% JavaScript 3.27% Python 0.85% NASL 0.19% SCSS 0.30% Vue 6.75% M4 0.10%

ntopng realtime network sflow ipfix traffic-monitoring packet-analyser packet-processing netflow snmp

ntopng's Introduction

ntopng

Introduction

ntopng® is a web-based network traffic monitoring application released under GPLv3. It is the new incarnation of the original ntop written in 1998, and now revamped in terms of performance, usability, and features.

While you can read more about ntopng on the ntop web site (http://www.ntop.org), we suggest you to start reading the doc/README.md file for learning how to compile and use ntopng.

If instead of source code you prefer to use a pre-built package, please go to http://packages.ntop.org

We build binary packages for the following platforms:

Debian/Ubuntu LTS x64
CentOS/RedHat/RockyLinux/AlmaLinux Linux x64
Windows x64
RaspberryPI/Debian ARM
FreeBSD/OPNsense/pfSense

Enjoy.

Documentation

If you want to learn more about ntopng please visit the User's Guide and the API Documentation.

Details

For more information about ntopng, please visit http://ntop.org

ntopng is a registered trademark in the US and EU.

ntopng's People

Contributors

Stargazers

Watchers

Forkers

thaolt initdotlove tomarch chris4269 lorenzobrunetti yagami0939 alenauta etrotta-grupposintesi philippeszanto pr0gg3d mloporchio zhouska ipernet geezyx vittgam jerric12 holmser manickamsp sjas beratx infranetntop kunliu1210 kargh lucgagnon paulosv dunn alagoutte slachiewicz simonemainardi bigjazzer lmangani girishprb martinreidy m-ust nazhu benintech p4tch3r msbrogli haochuanxjtu zhhwin embededc x19e pombredanne seismiccollision stevewsneaker kuffz emanuelemazza sgryphon roccotocco fengpeiyuan ggbg ericlin315 cyndz136 changqiaowang ghiltanas dlsu-sysads shinsec xgerligand bobyou xtiankisutsa markfoodyburton shihliu pquast nicopayet tianzhishuizhu rtdpng linkp2p rdel-hgj nhurman madpilot78 daurenamanbayev gengzhi1024 niko78 cbrdiv masaeedu dobriboyadzhiev routenull0 emanuele-f malesa-mx deeper-think mathieufrh chinasun021 blakerouse fortitudepub luhengsw xephon-contrib misteryoung i7446047 qxip digideskio wyxy2005 mattparr prasincs aakapoor fmeerkoetter kldeng sauropollastrini matiasfigue jeffreymingyue cequencer

ntopng's Issues

Add ability to start ntopng with promiscuous mode disabled

In essence it must be implemented a new command line option to disable promiscuous mode similar to what happens for instance with "tcpdump -p"

Add netfilter support

Add ability for ntopng to sniff from a netfilter interface (Linux only).

bandwidth graph at bottom reports local-local traffic also

in the past, the graph only reported local-external and external-local traffic. now it seems to report all traffic, which is of no use to me.

Error compile dev branch

Hi,

I have clone the default (dev) branch of ntopng and the compiling process quit with an error. On the same machine I have no problem compiling the master branch. My OS is

Linux SMX8SIL 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Is there a bug in the Makefile?. Thanks.

Make message with error below

checking GeoIP.h usability... yes
checking GeoIP.h presence... yes
checking for GeoIP.h... yes
checking for GeoIP_lib_version in -lGeoIP... yes
checking for sqlite3_open in -lsqlite3... yes
configure: creating ./config.status
config.status: creating packages/ntopng.spec
config.status: creating packages/ntopng-data.spec
config.status: creating Makefile
config.status: WARNING: 'Makefile.in' seems to ignore the --datarootdir setting
config.status: creating doc/doxygen.conf
config.status: creating include/config.h
config.status: include/config.h is unchanged
configure: WARNING: unrecognized options: --disable-libdbi, --disable-libwrap, --disable-rrdcgi, --disable-libtool-lock, --disable-nls, --disable-rpath, --disable-perl, --disable-ruby, --disable-lua, --disable-tcl, --disable-python, --disable-dependency-tracking, --disable-rrd_graph

You are now ready to compile typing make
Please do not forget to download GeoIP databases doing: make geoip
make[1]: Entering directory /home/smart/ntopng/src' make[1]: *** No rule to make targetlibrrd_th.la'. Stop.
make[1]: Leaving directory `/home/smart/ntopng/src'
make: *** [/third-party/rrdtool-1.4.8/src/.libs/librrd_th.a] Error 2

Unresponsive script

We use our NTOP installation as part of our NOC. While running the main page (Top Talkers with actual traffic and the graphs corresponding those) we get this message after a few hours:

We use Firefox version 37.0.2. Also, if we run this page in IE, after a while the screen freezes without any error.

Anyone else that had this issue?

Realtime traffic calculation when using nProbe

When ntopng is used as flow collector, the traffic graphs are not accurate as ntopng computes the bytes/packets from flows. This is not correct, as flows contain average value and are sent to ntop periodically thus invalidating the realtime view of traffic.

Upgraded to ntopng 2.0 and get the error HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 3000

We were running stable on version 1.99. Everything was working fine until the update. I performed the update to hopefully get historical data to work when coming from nprobe. The server is running CentOS 6.6 with the ntop.repo enabled. The ntopng service did not start, so I ran ntopng at the command line. Whether I use my ntopng.conf file or not, I get the same error.

16/Jun/2015 10:25:54 [src/HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 3000,3001s Success

port 3000 is not in use

I do see 'User changed to nobody' before the error occurs.

ntopng version is

ntopng-data-2.0.150616-203.noarch
ntopng-2.0.150616-203.x86_64

Max Size of Dump Files Unit

In "Max Size of Dump Files" (e.g. http://localhost:3000/lua/if_stats.lua?if_name=XXX&page=packetdump) the unit (bytes) is too small. Change it to KB or better MB to make it easier for the user to set it.

cannot install on debian

seems ntopng package isn't in the ubutu repos.
exist for x64 but not for "all" folder

graph legend

on report page, clicking on the print icon, a report i produced but graphs have no legend or a "cut" on

RRD configuration preferences

ntopng saves raw metrics in RRDs with 1 sec granularity for 1 day, 1 min resolution for 1 month...

We need to create a preference panel where such values can be set by the user. The preferences will be written in redis, and the graph_utils.lua script (whenever a new rrd has to be created) will fetch the values from rrd. Existing rrds will not be affected by changes to preferences, but just those created after that the preference has been set.

Loading of historical data broken when using external flow-collector

i have the same problem like described in this mailing-list:
http://www.gossamer-threads.com/lists/ntop/misc/37261

Is there any timerange in which this refactoring will be done?

Ntop Linux Debian

'errno 10 while sending page to client'
Please, someone can help me ?
Thanks 👍

Aggregated view graph

In ntopng when two interfaces are aggregated through a view the traffic is merged. It would be nice on
http://localhost:3000/lua/if_stats.lua?if_name=en0,lo0&page=historical not to see the sum of traffic directions but rather a stacked chart (of different colours) for each interface

Project about Graphite

In our project we decide to use StatsD to take network data in SNMP protocol and put them in the Graphite's DB Carbon, but StatsD can't do this, so we need help to do this changes.
Thank you

#include error when building from source

Hi,

I'm running an Arch Linux machine, so there is no official package. I'm trying to get to know ntopng and ran into problems building it. I followed http://www.ntop.org/get-started/download/. Below is what I did:

git clone https://github.com/ntop/nDPI.git
cd nDPI
git checkout -b 1.6-stable
./autogen.sh
make
sudo make install
cd ..
git clone https://github.com/ntop/ntopng.git
cd ntopng
git checkout -b 2.0-stable
./autogen.sh
./configure
make geoip
make

Unfortunately, that yields the following output:

moschroe@X /media/VOLATILE/ntopng (git)-[2.0-stable] % LANG=C make
/usr/bin/g++ -g -Wall -I/media/VOLATILE/ntopng -I/media/VOLATILE/ntopng/include -I/usr/local/include -I /media/VOLATILE/ntopng/third-party/hiredis -I/media/VOLATILE/ntopng/third-party/mongoose -I/usr/include/json-c  -I/usr/include//libndpi  -I/media/VOLATILE/ntopng/third-party/LuaJIT-2.0.3/src    -I/media/VOLATILE/ntopng -I/media/VOLATILE/ntopng/include -I/usr/local/include -I/media/VOLATILE/ntopng/third-party/http-client-c/src/ -I/media/VOLATILE/ntopng/third-party/EWAHBoolArray/headers  -DDATA_DIR='"/usr/local/share"' -I/media/VOLATILE/ntopng/third-party/libgeohash -I/media/VOLATILE/ntopng/third-party/patricia   -c src/HTTPBL.cpp -o src/HTTPBL.o
In file included from /media/VOLATILE/ntopng/include/ntop_includes.h:153:0,
                 from src/HTTPBL.cpp:22:
/media/VOLATILE/ntopng/include/NetworkInterface.h:90:18: error: 'NUM_BREEDS' was not declared in this scope
     breeds_bytes[NUM_BREEDS];
                  ^
/media/VOLATILE/ntopng/include/NetworkInterface.h: In member function 'char* NetworkInterface::get_ndpi_proto_breed_name(u_int)':
/media/VOLATILE/ntopng/include/NetworkInterface.h:190:19: error: 'ndpi_get_proto_breed' was not declared in this scope
                 id))); };
                   ^
/media/VOLATILE/ntopng/include/NetworkInterface.h:190:20: error: 'ndpi_get_proto_breed_name' was not declared in this scope
                 id))); };
                    ^
In file included from /media/VOLATILE/ntopng/include/ntop_includes.h:181:0,
                 from src/HTTPBL.cpp:22:
/media/VOLATILE/ntopng/include/Flow.h: At global scope:
/media/VOLATILE/ntopng/include/Flow.h:183:10: error: 'ndpi_protocol_breed_t' does not name a type
   inline ndpi_protocol_breed_t get_protocol_breed() { return(ndpi_get_proto_breed(iface->get_ndpi_struct(), ndpi_detected_protocol)); };
          ^
/media/VOLATILE/ntopng/include/Flow.h: In member function 'char* Flow::get_protocol_breed_name()':
/media/VOLATILE/ntopng/include/Flow.h:186:40: error: 'ndpi_get_proto_breed' was not declared in this scope
                  ndpi_detected_protocol))); };
                                        ^
/media/VOLATILE/ntopng/include/Flow.h:186:41: error: 'ndpi_get_proto_breed_name' was not declared in this scope
                  ndpi_detected_protocol))); };
                                         ^
Makefile:131: recipe for target 'src/HTTPBL.o' failed
make: *** [src/HTTPBL.o] Error 1

Switching to the dev branch yields the same result.
The directory /usr/include/libndpi/does contain all the header files. -I/usr/include//libndpi should set up g++ to find them. So I concluded that at least one #include is missing or in the wrong place.
Obviously, you are able to build packages, so I am confused 😕.

Am I missing something? Are you missing some information I could provide?

Thank you very much
moschroe

Missed Top Talker

Just install 2.0.150601 on CentOS 6.6

RPM version do not start, so compiled and installed from source

Still Top Talkers are not showed (like was on previous version) on Historical NIC Activity.
This is a very useful feature!
See also bug 538 on https://kpn.ntop.org/bugzilla/show_bug.cgi?id=538

It's also be mentioned that this windows in noticeably slower than previous version, when moving cursor on Traffic part.

Regards,
Paolo.

host_details.lua historical

historical tab in host_details.lua does not show the graph in with any time series selected.
This could be cause by a "<script>" string inside a javascript section (probably just after the drawRRD call?). Chrome inspection tool reports as follow:
....
$.ajax ({
type: 'GET',
url: '/lua/modules/get_real_epochs.lua?epoch='+point.value.x, data: { epoch: point.value.x },
async: false,
success: function(content) {
var res = content.split(" ");
seconds = parseInt(res[0]) - parseInt(res[1]);
}
});

infoHTML += "

99% dropped packets after update to ntopng Community v.2.0.150619

Hey guys, I’ve got 99% dropped packets after update to NtopNG Community v.2.0.150619.
Only in the first 2 minutes the flow information are shown after reset the NtopNG settings via Nbox .
Could you please help? BR Alex

footer line is not updated when looking a collector interface

ntopng Community v.2.0.150619
I have two ntopng instances on separate systems - with one acting as collector. The collector does not
update the bps in the footer.
0 bps [0 pps]
Uptime: 10 min, 52 sec
800 Hosts 4,206 Flows
also I don't see top talker graph.
If I do a tcpdump on port 3456 - the port I am using send the data on - I see a lot of traffic bewteen the
two ntopng systems.

on the sender I see
18.67 Mbps [6,521 pps]
Uptime: 29 min, 46 sec
1,501 Hosts 1,368 Aggregations 6,748 Flows
So there is plenty of data -

but the collector show very little
Name tcp://10.0.133.87:3456
Save Name
Family zmq
Bytes
6.00 KB

-e switch causes flow export to ElasticSearch to not work correctly

But here I started ntopng without the -e flag and I got 3562 packets on port 9200 in a minute and a half.
sudo /usr/local/bin/ntopng -U ntop -ip4p1 -D all -E all -A 2 -H -n 1 -F"es;flows;ntopng2-%Y.%m.%d;http://localhost:9200/_bulk" -d/var/lib/ntop -w 3000 -n 1 -m 192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
09/Jun/2015 10:34:11 [src/Prefs.cpp:685] Using ElasticSearch for data dump [flows][ntopng2-%Y.%m.%d][http://localhost:9200/_bulk]

Tue Jun 9 10:33:08 EDT 2015
P301002:~
$ sudo tcpdump -w test1.pcap -X -s1500 -nnli lo port 9200
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 1500 bytes
^C3562 packets captured
7126 packets received by filter
0 packets dropped by kernel
Tue Jun 9 10:34:29 EDT 2015
The pcap file is attached as test1.pcap.gz

Here I started ntopng same command but with the -e flag.

$ sudo /usr/local/bin/ntopng -e -U ntop -ip4p1 -D all -E all -A 2 -H -n 1 -F"es;flows;ntopng2-%Y.%m.%d;http://localhost:9200/_bulk" -d/var/lib/ntop -w 3000 -n 1 -m 192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
09/Jun/2015 10:50:40 [src/Prefs.cpp:685] Using ElasticSearch for data dump [flows][ntopng2-%Y.%m.%d][http://localhost:9200/_bulk]
09/Jun/2015 10:50:40 [src/Prefs.cpp:793] Logging into /var/lib/ntop/ntopng.log
09/Jun/2015 10:50:40 [src/Ntop.cpp:781] Setting local networks to 192.168.0.0/16,10.0.0.0/8,172.16.0.0/12
09/Jun/2015 10:50:40 [src/Redis.cpp:93] Successfully connected to redis 127.0.0.1:6379@0
09/Jun/2015 10:50:40 [pro/NtopPro.cpp:100] ERROR: [LICENSE] Invalid or missing ntopng License [Empty license file]
09/Jun/2015 10:50:40 [pro/NtopPro.cpp:111] WARNING: [LICENSE] ntopng will now run in pro mode for 10 minutes
09/Jun/2015 10:50:40 [pro/NtopPro.cpp:113] WARNING: [LICENSE] before returning to community mode
09/Jun/2015 10:50:40 [pro/NtopPro.cpp:114] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org
09/Jun/2015 10:50:40 [pro/NtopPro.cpp:115] WARNING: [LICENSE] or run ntopng in community mode starting
09/Jun/2015 10:50:40 [pro/NtopPro.cpp:116] WARNING: [LICENSE] ntopng --community
09/Jun/2015 10:50:40 [src/Ntop.cpp:755] Parent process is exiting (this is normal)
Tue Jun 9 10:50:40 EDT 2015

After about 5 minutes I stopped ntopng and the packet capture - 8 packets captured.
Tue Jun 9 10:49:45 EDT 2015
P301002:~
$ sudo tcpdump -w test2.pcap -X -s1500 -nnli lo port 9200
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 1500 bytes
^C8 packets captured
16 packets received by filter
0 packets dropped by kernel
Tue Jun 9 10:56:09 EDT 2015

here is the data:
$ tcpdump -r test2.pcap -nn
reading from file test2.pcap, link-type EN10MB (Ethernet)
10:51:27.908125 IP 127.0.0.1.60295 > 127.0.0.1.9200: Flags [.], ack 116553322, win 537, options [nop,nop,TS val 104525777 ecr 104450777], length 0
10:51:27.908141 IP 127.0.0.1.9200 > 127.0.0.1.60295: Flags [.], ack 1, win 529, options [nop,nop,TS val 104525777 ecr 100040817], length 0
10:52:42.908074 IP 127.0.0.1.60295 > 127.0.0.1.9200: Flags [.], ack 1, win 537, options [nop,nop,TS val 104600777 ecr 104525777], length 0
10:52:42.908084 IP 127.0.0.1.9200 > 127.0.0.1.60295: Flags [.], ack 1, win 529, options [nop,nop,TS val 104600777 ecr 100040817], length 0
10:53:57.908048 IP 127.0.0.1.60295 > 127.0.0.1.9200: Flags [.], ack 1, win 537, options [nop,nop,TS val 104675777 ecr 104600777], length 0
10:53:57.908066 IP 127.0.0.1.9200 > 127.0.0.1.60295: Flags [.], ack 1, win 529, options [nop,nop,TS val 104675777 ecr 100040817], length 0
10:55:12.908331 IP 127.0.0.1.60295 > 127.0.0.1.9200: Flags [.], ack 1, win 537, options [nop,nop,TS val 104750777 ecr 104675777], length 0
10:55:12.908347 IP 127.0.0.1.9200 > 127.0.0.1.60295: Flags [.], ack 1, win 529, options [nop,nop,TS val 104750777 ecr 100040817], length 0
Tue Jun 9 10:57:33 EDT 2015

Missing dependency on OS/X Yosemite

I haven't built ntopng in a while. When trying to build it on Yosemite I get the following linker error:

/usr/bin/clang++ src/Ntop.o src/DnsStats.o src/Prefs.o src/GenericHash.o src/Categorization.o src/EthStats.o src/HTTPserver.o src/StringHost.o src/VirtualHost.o src/StringHash.o src/PacketDumperTuntap.o src/HistoricalInterface.o src/AddressResolution.o src/HostContacts.o src/main.o src/NetworkInterfaceView.o src/Mutex.o src/NetworkInterface.o src/TrafficStats.o src/ExportInterface.o src/EppStats.o src/Redis.o src/Flow.o src/Host.o src/IpAddress.o src/DB.o src/RuntimePrefs.o src/ActivityStats.o src/PacketDumper.o src/LocalTrafficStats.o src/PeriodicActivities.o src/PacketStats.o src/HTTPBL.o src/GenericHost.o src/VirtualHostHash.o src/AlertCounter.o src/HTTPStats.o src/FlowHash.o src/Geolocation.o src/NtopGlobals.o src/PcapInterface.o src/PF_RINGInterface.o src/ParserInterface.o src/Trace.o src/CollectorInterface.o src/Utils.o src/HostHash.o src/SimpleStringHost.o src/ProtoStats.o src/GenericHashEntry.o src/StatsManager.o src/Lua.o src/NdpiStats.o -Wall ./nDPI/src/lib/.libs/libndpi.a -lpcap ./third-party/LuaJIT-2.0.3/src/libluajit.a third-party/rrdtool-1.4.8/src/.libs/librrd_th.a -lm -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0 ./third-party/zeromq-4.0.5/src/.libs/libzmq.a third-party/json-c/.libs/libjson-c.a -lsqlite3 -pagezero_size 10000 -image_base 100000000 -L/usr/local/lib -ldl -lcurl -lm -lpthread -lstdc++.6 -o ntopng

ld: library not found for -lgobject-2.0

I have glib2 installed via macports but it apparently is missing lgobject-2.0. Is there another port that contains lgobject-2.0?

ntopng license inside file

Adding ntopng license in /etc/ntopng.license

ERROR: Unable to start HTTP server

I have installed ntopng and redis server successfully on one of my VMs

OS Version: 2.6.39-400.109.6.el6uek.x86_64
Installed by compiling from source checked out from svn

Not able to start ntopng. The error that I am facing is as follows

[root@ucf2b-docs-psr0427-ms1 bin]# ./ntopng
30/Apr/2015 10:48:53 [Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
30/Apr/2015 10:48:53 [Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
30/Apr/2015 10:48:54 [main.cpp:201] ERROR: Startup error: missing super-user privileges ? (I am already running as su!)

[root@ucf2b-docs-psr0427-ms1 bin]# ./ntopng -i eth0
30/Apr/2015 10:48:57 [Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
30/Apr/2015 10:48:57 [Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
30/Apr/2015 10:48:57 [PcapInterface.cpp:81] Reading packets from interface eth0...
30/Apr/2015 10:48:57 [Ntop.cpp:1027] Registered interface view eth0 [id: 0]
30/Apr/2015 10:48:57 [Ntop.cpp:997] Registered interface eth0 [id: 0]
30/Apr/2015 10:48:57 [Utils.cpp:277] User changed to nobody
30/Apr/2015 10:48:57 [main.cpp:227] PID stored in file /var/tmp/ntopng.pid
30/Apr/2015 10:48:57 [HTTPserver.cpp:449] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
30/Apr/2015 10:48:57 [HTTPserver.cpp:451] Please read https://svn.ntop.org/svn/ntop/trunk/ntopng/README.SSL if you want to enable SSL.
30/Apr/2015 10:48:57 [HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 3000 (no su error now!)

Confirmed that 3000 port is not in use
[root@ucf2b-docs-psr0427-ms1 bin]# netstat -nap|grep 3000
[root@ucf2b-docs-psr0427-ms1 bin]#

Tried to start it on a different port just in case
[root@ucf2b-docs-psr0427-ms1 bin]# ./ntopng -i eth0 -w 25436
30/Apr/2015 10:58:15 [Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
30/Apr/2015 10:58:15 [Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
30/Apr/2015 10:58:15 [PcapInterface.cpp:81] Reading packets from interface eth0...
30/Apr/2015 10:58:15 [Ntop.cpp:1027] Registered interface view eth0 [id: 0]
30/Apr/2015 10:58:15 [Ntop.cpp:997] Registered interface eth0 [id: 0]
30/Apr/2015 10:58:15 [Utils.cpp:277] User changed to nobody
30/Apr/2015 10:58:15 [main.cpp:227] PID stored in file /var/tmp/ntopng.pid
30/Apr/2015 10:58:15 [HTTPserver.cpp:449] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
30/Apr/2015 10:58:15 [HTTPserver.cpp:451] Please read https://svn.ntop.org/svn/ntop/trunk/ntopng/README.SSL if you want to enable SSL.
30/Apr/2015 10:58:15 [HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 25436

Enabled SSL and tried to start it
[root@ucf2b-docs-psr0427-ms1 bin]# ./ntopng -i eth0 -w 25436 -s
30/Apr/2015 10:58:19 [Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
30/Apr/2015 10:58:19 [Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
30/Apr/2015 10:58:19 [PcapInterface.cpp:81] Reading packets from interface eth0...
30/Apr/2015 10:58:19 [Ntop.cpp:1027] Registered interface view eth0 [id: 0]
30/Apr/2015 10:58:19 [Ntop.cpp:997] Registered interface eth0 [id: 0]
30/Apr/2015 10:58:19 [main.cpp:227] PID stored in file /var/tmp/ntopng.pid
30/Apr/2015 10:58:19 [HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 25436,3001s

Checked /var/log/messages and /var/tmp/ntopng to see if there are any logs but couldn’t find anything relevant.

Continuously duplicated data when outputting to ElasticSearch

When running the following command:
ntopng -n 3 -i [path to pcap] -F "es;flows;ntopng-%Y.%m.%d;http:/;locahost:9200/_bulk;" -v
I notice that the program sends some data to ES, then loops for 5 seconds or longer (once each second, 5+ times) and outputs to the screen:

29/Jun/2015 12:31:17 [src/PeriodicActivities.cpp:91] Starting script /usr/share/ntopng/scripts/callbacks/second.lua
29/Jun/2015 12:31:17 [src/Lua.cpp:2555] ntop_get_dirs() called
29/Jun/2015 12:31:17 [src/Lua.cpp:2515] ntop_is_pro() called
29/Jun/2015 12:31:17 [src/Lua.cpp:2515] ntop_is_pro() called
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:60] About to decompress 3434 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:69] Decompressed 5540 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:60] About to decompress 3656 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:69] Decompressed 5540 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:60] About to decompress 3662 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:69] Decompressed 5540 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:60] About to decompress 3105 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:69] Decompressed 5540 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:60] About to decompress 1671 bytes
29/Jun/2015 12:31:17 [pro/LuaHandler.cpp:69] Decompressed 2292 bytes
29/Jun/2015 12:31:17 [src/Lua.cpp:216] ntop_get_interface_names() called

The command will run continuously until it is manually terminated (^C) even on a single packet pcap file. After running the command on the single packet PCAP file for a minute, minute it generated 24 entries in ES before I ended the process.

Environment specs:

Ubuntu 14.04 x64 desktop VM
ntopng -V: 2.0.150531
ES version: 1.4.5

svn files in rpm package

ntopng-2.0.150601-165.x86_64.rpm contains .svn directories.
rpm -qvlp ntopng-2.0.150601-165.x86_64.rpm | grep svn

Compiling from source Debian 7

echo "deb http://ftp.de.debian.org/debian wheezy-backports main" > /etc/apt/sources.list.d/backports.list
apt-get update && apt-get dist-upgrade
apt-get install git screen vim build-essential devscripts libnetfilter-queue-dev autoconf libtool libpcap-dev libjson0-dev autogen automake autoconf libtool libjsoncpp-dev
libjson-glib-dev libjson-c-dev libcurl4-openssl-dev libsqlite3-dev libgeoip-dev libxml2-dev

mkdir /usr/xxx
cd /usr/xxx
git clone https://github.com/ntop/ntopng.git
git clone https://github.com/ntop/nDPI.git

cd nDPI
./autogen.sh
./configure
make
make install

cd ../ntopng
./autogen.sh
./configure
make

Same exact error on a fresh build vm instance of Debian 7.8 w/latest updates applied

src/NetfilterInterface.cpp: In function ‘int netfilter_callback(nfq_q_handle_, nfgenmsg_, nfq_data_, void_)’:
src/NetfilterInterface.cpp:86:64: error: invalid conversion from ‘unsigned char**’ to ‘char**’ [-fpermissive]
In file included from /usr/xxx/ntopng/include/ntop_includes.h:107:0,
from src/NetfilterInterface.cpp:22:
/usr/include/libnetfilter_queue/libnetfilter_queue.h:99:12: error: initializing argument 2 of ‘int nfq_get_payload(nfq_data_, char__)’ [-fpermissive]
make: *_* [src/NetfilterInterface.o] Error 1

Please explain your build environment

Per-host configurable alerts

Add per-host thresholds so that alerts are not generated with a global threshold that might be too high for some hosts and too little for others.
Alerts need to be generated for local hosts only and ignored for others. Example if local host X is under flood attack by remote host Y, the alert should report this (i.e. both X and Y must be named) but no state for host Y must be kept.

bootstrap.css.map

With the authentication active (and without an active session), the login page includes a reference to bootstrap.css.map that could not be retrieved (due to the login) and several requests to that file are done (each request is redirected to the login page). This issue do not have any side effect on the usability, but it is still and non working condition. When the user is logged this file is however requested several times causing useless network traffic to moving back and forth from the ntopng server

Rework host interactions

It is necessary to rework /lua/hosts_interaction.lua as it is barely usable on large networks

ERROR: Unable to start HTTP server (ntopng version v.1.99.150505)

Reopening issue #2

Pulled latest code from git and tried to start ntopng. However it still fails to start

[root@ucf2b-docs-psr0427-ms1 paasusr]# /usr/local/bin/ntopng --version
v.1.99.150505

[root@ucf2b-docs-psr0427-ms1 paasusr]# /usr/local/bin/ntopng
05/May/2015 08:44:23 [src/Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
05/May/2015 08:44:23 [src/Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
05/May/2015 08:44:23 [src/main.cpp:201] ERROR: Startup error: missing super-user privileges ?

[root@ucf2b-docs-psr0427-ms1 paasusr]# /usr/local/bin/ntopng -i eth0
05/May/2015 08:44:43 [src/Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
05/May/2015 08:44:43 [src/Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
05/May/2015 08:44:43 [src/PcapInterface.cpp:81] Reading packets from interface eth0...
05/May/2015 08:44:43 [src/Ntop.cpp:1027] Registered interface view eth0 [id: 0]
05/May/2015 08:44:43 [src/Ntop.cpp:997] Registered interface eth0 [id: 0]
05/May/2015 08:44:43 [src/Utils.cpp:277] User changed to nobody
05/May/2015 08:44:43 [src/main.cpp:227] PID stored in file /var/tmp/ntopng.pid
05/May/2015 08:44:43 [src/HTTPserver.cpp:449] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
05/May/2015 08:44:43 [src/HTTPserver.cpp:451] Please read https://svn.ntop.org/svn/ntop/trunk/ntopng/README.SSL if you want to enable SSL.
05/May/2015 08:44:43 [src/HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 3000

[root@ucf2b-docs-psr0427-ms1 paasusr]# /usr/local/bin/ntopng -i eth0 -w 3060
05/May/2015 08:44:49 [src/Ntop.cpp:779] Setting local networks to 192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,127.0.0.0/8,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
05/May/2015 08:44:49 [src/Redis.cpp:93] Successfully connected to Redis 127.0.0.1:6379@0
05/May/2015 08:44:49 [src/PcapInterface.cpp:81] Reading packets from interface eth0...
05/May/2015 08:44:49 [src/Ntop.cpp:1027] Registered interface view eth0 [id: 0]
05/May/2015 08:44:49 [src/Ntop.cpp:997] Registered interface eth0 [id: 0]
05/May/2015 08:44:49 [src/Utils.cpp:277] User changed to nobody
05/May/2015 08:44:49 [src/main.cpp:227] PID stored in file /var/tmp/ntopng.pid
05/May/2015 08:44:49 [src/HTTPserver.cpp:449] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
05/May/2015 08:44:49 [src/HTTPserver.cpp:451] Please read https://svn.ntop.org/svn/ntop/trunk/ntopng/README.SSL if you want to enable SSL.
05/May/2015 08:44:49 [src/HTTPserver.cpp:486] ERROR: Unable to start HTTP server (IPv4) on ports 3060

OS version is 2.6.39-400.109.6.el6uek.x86_64

[root@ucf2b-docs-psr0427-ms1 paasusr]# ifconfig
eth0 Link encap:Ethernet HWaddr C6:B0:3D:21:E4:CA
inet addr:10.241.239.42 Bcast:10.241.239.43 Mask:255.255.255.252
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:47709210 errors:0 dropped:4 overruns:0 frame:0
TX packets:45548955 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:42809387833 (39.8 GiB) TX bytes:42008874445 (39.1 GiB)
Interrupt:80

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2132 errors:0 dropped:0 overruns:0 frame:0
TX packets:2132 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:178372 (174.1 KiB) TX bytes:178372 (174.1 KiB)

Add Interface Alerts

Currently alerts are defined only for hosts. It is requested to implements alerts per interface

Nprobe usage

I have installed ntopng on Ubuntu machine to work as a collector for IPFIX traffic. I would like to know if nprobe should be installed to monitor the IPFIX traffic or it's possible to do that only using ntopng.
Please advise.

Lua out of memory

Running on centos6 against a very busy Cisco ASA ntopng is able to run briefly but once we click on a tab to get any data out of the interface we get messages like the following:

13/May/2015 15:05:19 [src/Lua.cpp:4667] WARNING: Script failure [/tmp/ntopng/scripts/lua/iface_ports_list.lua][not enough memory]
13/May/2015 15:05:19 [src/Lua.cpp:4667] WARNING: Script failure [/tmp/ntopng/scripts/lua/iface_ports_list.lua][not enough memory]
13/May/2015 15:05:19 [src/Lua.cpp:4667] WARNING: Script failure [/tmp/ntopng/scripts/lua/iface_ports_list.lua][not enough memory]
13/May/2015 15:05:19 [src/Lua.cpp:4398] WARNING: Script failure [/tmp/ntopng/scripts/callbacks/minute.lua][not enough memory]

Filtering report

After saving the report filter with no protocols, re-clicking the report filter the http protocol results selected.

Add layer-2 view

In the old ntop (non -ng) it was possible to show layer 2 statistics and to display mac address symbolic name (e.g. see https://code.wireshark.org/review/gitweb?p=wireshark.git;a=blob_plain;f=manuf).

Ntopng Service Status - Debian 8.0 Jessie package

In a virtual machine I have Debian 8.0 Jessie for testing. With this Debian version I can install ntopng from repository instead of sources. This is a great because for Wheezy it was not available.

Ntopng installs fine as a service in the system and it works as expected (except Similarity bug I reported earlier). When I run:

root@server:~# service ntopng status

I get two warnings:

May 09 02:01:02 server1 ntopng[656]: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/share/ntopng/httpdocs/geoip/GeoLiteCity.dat
May 09 02:01:02 server1 ntopng[656]: [Geolocation.cpp:59] WARNING: Unable to read GeoIP database /usr/share/ntopng/httpdocs/geoip/GeoLiteCityv6.dat
May 09 02:01:02 server1 sh[509]: Error Opening file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
May 09 02:01:02 server1 sh[509]: Error Opening file /usr/share/ntopng/httpdocs/geoip/GeoLiteCity.dat
May 09 02:01:02 server1 sh[509]: Error Opening file /usr/share/ntopng/httpdocs/geoip/GeoLiteCityv6.dat
May 09 03:33:28 server1 ntopng[656]: [Lua.cpp:2775] WARNING: Script failure [/usr/share/ntopng/scripts/lua/host_details.lua][/usr/share/ntopng/scripts/lua/host_details.lua:1253: bad argument #1...cted, got nil)]
May 09 03:35:50 server1 ntopng[656]: [Lua.cpp:2775] WARNING: Script failure [/usr/share/ntopng/scripts/lua/host_details.lua][/usr/share/ntopng/scripts/lua/host_details.lua:1253: bad argument #1...cted, got nil)]
May 09 03:36:17 server1 ntopng[656]: [Lua.cpp:2775] WARNING: Script failure [/usr/share/ntopng/scripts/lua/host_details.lua][/usr/share/ntopng/scripts/lua/host_details.lua:1253: bad argument #1...cted, got nil)]
May 09 03:40:40 server1 ntopng[656]: [Lua.cpp:2775] WARNING: Script failure [/usr/share/ntopng/scripts/lua/host_details.lua][/usr/share/ntopng/scripts/lua/host_details.lua:1253: bad argument #1...cted, got nil)]
May 09 03:41:18 server1 ntopng[656]: [Lua.cpp:2775] WARNING: Script failure [/usr/share/ntopng/scripts/lua/host_details.lua][/usr/share/ntopng/scripts/lua/host_details.lua:1253: bad argument #1...cted, got nil)]

I attach a screenshot from my Putty session to see them too. These warnings are related to "GeoIP missing files" and "Script failure".

For the first warning I guess those files are missing. But for the second warning I don't have any idea what this could be. This warning is related to Similarity bug I posted earlier.

Thank you.

Compiling from source Debian 7

Similarity information is not working

First of all thank you for this wonderful monitoring software.

I installed ntopng on Debian 7.8 Wheezy from source with geoip (online server) and Debian 8.0 Jessie from Debian's repository (vm server). In both I get an error as follows:

Go to [Flows] section. Click in column "Server" then on your server IP address.
You will see a lot of tabs from [Traffic] to [Historical]. Please click on [Similarity] tab. Page loads and you will see an error:

HTTP/1.1 500 Internal server error Content-Type: text/html Connection: close Script "/usr/share/ntopng/scripts/lua/host_details.lua" returned an error:

/usr/share/ntopng/scripts/lua/host_details.lua:1253: bad argument #1 to 'pairs' (table expected, got nil)

I attach a screenshot to see this error in my browser.

Thank you.

segmentation fault when reading pcap

I just update my ntopng (bade on git dev repo) and now it segfault (it was working 2 week ago):

my command:
ntopng --pid /var/tmp/ntopng2.pid -i /tmp/dump_firewall.txt -w 3000 -n 2 -A 2 -F "db" -m "192.168.201.0/24,192.168.202.0/24,192.168.203.0/24,192.168.204.0/24,192.168.205.0/24,192.168.206.0/24"

the log:
18/Jun/2015 11:37:58 [src/Ntop.cpp:785] Setting local networks to 192.168.201.0/24,192.168.202.0/24,192.168.203.0/24,192.168.204.0/24,192.168.205.0/24,192.168.206.0/24
18/Jun/2015 11:37:58 [src/Redis.cpp:93] Successfully connected to redis 127.0.0.1:6379@0
18/Jun/2015 11:37:58 [src/Ntop.cpp:1033] Registered interface view /tmp/dump_firewall.txt [id: 7]
18/Jun/2015 11:37:58 [src/Ntop.cpp:1003] Registered interface /tmp/dump_firewall.txt [id: 0]
18/Jun/2015 11:37:58 [src/Ntop.cpp:1033] Registered interface view Historical [id: 4]
18/Jun/2015 11:37:58 [src/Ntop.cpp:1003] Registered interface Historical [id: 1]
18/Jun/2015 11:37:58 [src/Utils.cpp:292] User changed to nobody
18/Jun/2015 11:37:58 [src/main.cpp:236] PID stored in file /var/tmp/ntopng2.pid
18/Jun/2015 11:37:58 [src/HTTPserver.cpp:449] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
18/Jun/2015 11:37:58 [src/HTTPserver.cpp:451] Please read https://svn.ntop.org/svn/ntop/trunk/ntopng/README.SSL if you want to enable SSL.
18/Jun/2015 11:37:58 [src/HTTPserver.cpp:492] Web server dirs [/usr/local/share/ntopng/httpdocs][/usr/local/share/ntopng/scripts]
18/Jun/2015 11:37:58 [src/HTTPserver.cpp:495] HTTP server listening on port 3000
18/Jun/2015 11:37:58 [src/main.cpp:287] Working directory: /var/tmp/ntopng
18/Jun/2015 11:37:58 [src/main.cpp:289] Scripts/HTML pages directory: /usr/local/share/ntopng
18/Jun/2015 11:37:58 [src/Ntop.cpp:271] Welcome to ntopng x86_64 v.2.0.150618 - (C) 1998-15 ntop.org
18/Jun/2015 11:37:58 [src/PeriodicActivities.cpp:53] Started periodic activities loop...
18/Jun/2015 11:37:58 [src/RuntimePrefs.cpp:32] Dumping alerts into syslog
18/Jun/2015 11:37:58 [src/NetworkInterface.cpp:1204] Setting affinity of interface /tmp/dump_firewall.txt to core 7
18/Jun/2015 11:37:58 [src/NetworkInterface.cpp:1208] Started packet polling on interface /tmp/dump_firewall.txt [id: 7]...
18/Jun/2015 11:37:58 [src/NetworkInterface.cpp:1208] Started packet polling on interface Historical [id: 4]...
18/Jun/2015 11:37:59 [src/PcapInterface.cpp:131] Reading packes from pcap file /tcpdump/firewall/dump_2015-06-01_00:17:34.pcap
Segmentation fault

Historical interface

I use last SVN version of ntopng.
When I try to load files via historical interface, my ntopng crash...

Known issue ?

aggregated_hosts_stats.lua

with a reverse proxy configuration environment no data are displayed in the aggregated_hosts_stats.lua page. Javascript console reports an error as follows:
aggregated_hosts_stats.lua:164 Uncaught SyntaxError: Unexpected token ILLEGAL
issue is probably caused by a misplaced ' " '
var url_update ="/lua/get_hosts_data.lua?aggregated=1";

Revised Historical Interface

The current historical interface is not too useful as

When there are too many flows it is slow
With the pro version, you can have reports and top X without too much hassle.

Recode the historical interface so that users can drill down to flows starting from charts Example if you look at HTTP traffic chart on eth1 of host 1.2.3.4, then a new drill down element should appear for selecting/visualising all flows of such host. Flows can be fetched from ES but it would be nice to add support for additional DBs such as MySQL.

Add per-network traffic chart

Show a chart for traffic volume for a given network in the web GUI.

add DSCP/ToS flag in bridge mode

I (and many others) would like to put ntopng in front of our networks to identify and mark traffic types so that those marks can be used in our firewall and traffic shapers

For example:
ntopng as 'bump in wire' on the wan side of a traffic shaper.
ntopng marks bittorrent with a DSCP/ToS tag.
In the traffic shaper, mark connections with the bittorrent DSCP tag so that those connections can be shaped.

another example:
same physical layout, but this time identify windows update and mark with a different DSCP tag.
Router can now route those packets (via identifying the DSCP tag) through a windows update proxy.

You can detect flows that are apparently "bad". For bad flows, ntopng (when used in inline mode), should refrain from forwarding traffic of these flows.

Per-Interface Alarms

Implement per-interface alarms similar to what happens with host alarms