GithubHelp home page GithubHelp logo

dns-oarc / dsc Goto Github PK

View Code? Open in Web Editor NEW
105.0 22.0 25.0 3.04 MB

DNS Statistics Collector

Home Page: https://www.dns-oarc.net/oarc/data/dsc

License: Other

Makefile 1.51% Shell 2.00% C 86.66% M4 2.29% Roff 6.97% Python 0.57%
dsc dns-statistics-collector collector dns statistics pcap monitoring c packet-capture dnstap

dsc's Introduction

DNS Statistics Collector

Total alerts Bugs Security Rating

DNS Statistics Collector (DSC) is a tool used for collecting and exploring statistics from busy DNS servers. It can be set up to run on or near nameservers to generate aggregated data that can then be transported to central systems for processing, displaying and archiving.

Together with dsc-datatool the aggregated data can be furthur enriched and converted for import into for example InfluxDB which can then be accessed by Grafana for visualzation, see this wiki on how to set up that:

DSC data transforming and enriching tool can be found here:

More information about DSC may be found here:

Issues should be reported here:

General support and discussion:

Dependencies

dsc requires a couple of libraries beside a normal C compiling environment with autoconf, automake, libtool and pkgconfig.

dsc has a non-optional dependency on the PCAP library and optional dependency on the MaxMindDB library (for the asn and country indexer).

To install the dependencies under Debian/Ubuntu:

apt-get install -y libpcap-dev

To install the dependencies under CentOS (with EPEL/PowerTools enabled):

yum install -y libpcap-devel

To install the dependencies under FreeBSD 10+ using pkg:

pkg install -y libpcap

NOTE: It is recommended to install the PCAP library from source/ports on OpenBSD since the bundled version is an older and modified version.

DNSTAP support

To enable DNSTAP support, first install the necessary dependencies and then run configure with --enable-dnstap.

  • Debian/Ubuntu: apt-get install -y libdnswire-dev libuv1-dev
  • CentOS: yum install -y dnswire-devel libuv-devel
  • FreeBSD: pkg install -y libuv
  • OpenBSD: pkg_add libuv

dnswire packages for Debian, Ubuntu and CentOS exists at https://dev.dns-oarc.net/packages/, for other distributions please see https://github.com/DNS-OARC/dnswire.

Building from source tarball

The source tarball from DNS-OARC comes prepared with configure:

tar zxvf dsc-version.tar.gz
cd dsc-version
./configure [options]
make
make install

NOTE: If building fails on FreeBSD/OpenBSD, try adding these configure options: --with-extra-cflags="-I /usr/local/include" --with-extra-ldflags="-L/usr/local/lib".

Building from Git repository

If you are building dsc from it's Git repository you will first need to initiate the Git submodules that exists and later create autoconf/automake files, this will require a build environment with autoconf, automake, libtool and pkgconfig to be installed.

git clone https://github.com/DNS-OARC/dsc.git
cd dsc
git submodule update --init
./autogen.sh
./configure [options]
make
make install

NOTE: If building fails on FreeBSD/OpenBSD, try adding these configure options: --with-extra-cflags="-I /usr/local/include" --with-extra-ldflags="-L/usr/local/lib".

Puppet

John Bond at ICANN DNS Engineering team has developed a puppet module for DSC, the module and code can be found here:

dsc's People

Contributors

aabdnn avatar jelu avatar jtkristoff avatar kdrenard avatar klaus-nicat avatar mayasd avatar mcstork avatar mfr00t avatar mimuret avatar wessels avatar wfms avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dsc's Issues

Datasets are non-unique

When configuring datasets the name is not checked for duplicates so you can specify the same named dataset multiple times with the same or different metrics.

@wessels is this a design choice or a bug?

New config parser *buff[4096] is too small

The new version can not load our bfp filter which is around 5500 characters:

in src/parse_conf.c:
int parse_conf(const char* file) {
FILE* fp;
char buffer[4096];

As a workaround we increased buf to [40960]. Maybe you can make it dynamically or just increase the default value.

response time statistics

Is it possible in DSC to correspond responses with requests? Then we could write a response-time analyzer

PID file empty

dsc logs "writing PID to ...." but the PID file is empty (0 bytes).

But once I kill dsc, the dsc file suddenly contains the PID, although dsc is not running anymore.

indexer support for A RDATA in DSC collector

Need to capture RDATA for TYPE A from the DNS response message. Currently DSC collector does not have support for this.

For A RDATA format I am referring to RFC at - https://www.ietf.org/rfc/rfc1035.txt

3.4.1. A RDATA format

+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    ADDRESS                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

where:

ADDRESS A 32 bit Internet address.

Wrong statistics with version 2.3 and threads

Hi!
We tested todays version from master (commit c02a170).

The problem is, that it does not report all requests, and even less responses. We started with 100q/s, increased to 200q/s, and then switched back to our old DSC version (reported as 2.0.0 rc1).

Requests

dsc-grapher pl

Responses

dsc-grapher-responses pl

New DSC uses the same config as the old DSC.

When using newest DSC with option -T the count is correct (200q/s)

Pcap input support

Add support for reading pcap files rather then capturing on interfaces.

This is needed in order to be able to write consistently automated tests for DSC.

Use Geo::IP as fallback

Based on the Debian package patches Geo::IP could be used as a fallback if IP::Country does not exist.

--- a/presenter/perllib/DSC/grapher/plot.pm
+++ b/presenter/perllib/DSC/grapher/plot.pm
@@ -249,14 +249,12 @@
     names  => [ qw(Unknown IANA LACNIC APNIC RIPE ARIN) ],
     colors => [ qw(black red purple yellow blue brightgreen) ],
     label_func => sub {
-   use IP::Country;
-   my $l = new IP::Country::Fast;
-   $l->inet_atocc(shift) || '??';
+   use Geo::IP;
+   my $l = Geo::IP->new(GEOIP_MEMORY_CACHE|GEOIP_CHECK_CACHE);
+   $l->country_code_by_addr(shift) || '??';
     },
     color_func => sub {
-   use IP::Authority;
-   my $l = new IP::Authority;
-   $l->inet_atoauth(shift) || '??';
+   "??";
     },
     data_reader => \&DSC::extractor::read_data2,
     data_summer => \&DSC::grapher::data_summer_0d,

building dsc on solaris 11

Has anyone succefully built dsc on solaris11

make > make.out.1

pcap_layers/pcap_layers.c: In function ‘handle_ipv4’:
pcap_layers/pcap_layers.c:327:16: error: ‘IPPROTO_GRE’ undeclared (first use in this function)
pcap_layers/pcap_layers.c:327:16: note: each undeclared identifier is reported only once for each function it appears in
pcap_layers/pcap_layers.c: In function ‘handle_ipv6’:
pcap_layers/pcap_layers.c:396:16: error: ‘IPPROTO_GRE’ undeclared (first use in this function)
make: Fatal error: Command failed for target pcap_layers/pcap_layers.o' Current working directory /export/home/dsc/dsc-2.1.1/src make: Fatal error: Command failed for targetall-recursive'
Current working directory /export/home/dsc/dsc-2.1.1/src
make: Fatal error: Command failed for target all' Current working directory /export/home/dsc/dsc-2.1.1/src make: Fatal error: Command failed for targetall-recursive'

dsc.zip

reports_on_exit have wrong stop time

dump_reports_on_exit does work now, but the timestamps do not show the real end-date, but the planned end-date, e.g, - dsc was killed after 8 seconds, but the end time is aligned to the next minute:

-rw-rw-r-- 1 root root 26600 2017-01-25 09:00:00.015471993 +0000 1485334800.dscdata.xml
-rw-rw-r-- 1 root root 23000 2017-01-25 09:00:08.415754582 +0000 1485334860.dscdata.xml

#head 1485334860.dscdata.xml
<dscdata>
<array name="pcap_stats" dimensions="2" start_time="1485334800" stop_time="1485334860">
  <dimension number="1" type="ifname"/>

Segfault in 2.2.0

Hello,

We encountered an issue with the latest version (2.2.0) or DSC. Hard to tell what exactly went wrong; all we know is that it crashed right after generating the first file.

The point is that we wanted to rollback to the previous version of DSC but we couldn't, since the package repository (http://pkg.dns-oarc.net/debian/pool/main/d/dsc/) only contains the latest version (2.2.0) of DSC.

May I suggest to keep all the past versions of DSC to ease a rollback after a failed upgrade.

Memory leaking

Hi!

dsc seems to leak memory:
image

We are seeing similiar behavior on all our nodes. The more traffic a node has, the faster the memory usage grows. We are using version 2.4.0, but we saw this behavior also with old versions (e.g pre2.0). I can't remember if we had this problem also with very old Debian version (201203250530).

I once had a problem with the old 201203250530 where I found out this special memory leak (not slowly over days but consuming all memory within a few minutes) was caused by some crappy TCP traffic which caused dsc to loop inside the TCP reassembly code.

So, now I do not have an idea where this is coming from. Maybe you add some memory debugging to DSC (eg reported also in the XML files) to find out where this leak is coming from.

allow network for local_address statement

--- Feature Request ---
Hi!
We use plenty of IP addresses on our name servers, Thus we have around 1000 lines of:
local_address 1.1.1.1
local_address 1.1.1.2
local_address 1.1.1....
local_address 1.1.1.255
in our config.
The capture filter is smart by using "and net 1.1.1.0/24" but adding all this local_addresses bloats the config. It would be great if we could configure a network/mask too.

Thanks
Klaus

Warning about userland filtering always appears

I receive the detected non-fatal error during pcap activation, filters may run in userland [11]: Resource temporarily unavailable warning also when using short filters which do not cause any libcap warnings.

Setting statistics interval to: 60
BPF program is: (net 11.22.32.98/32 or net 2a02:fff:4:4::98/128 or net 11.22.32.103/32 or net 2a02:fff:4:4::103/128) and (udp port 53 or tcp port 53)
Opening interface eth1
Opening interface eth2
creating dataset qtype
...
writing PID to /var/run/regdns.dsc.pid
detected non-fatal error during pcap activation, filters may run in userland [11]: Resource temporarily unavailable
Running

dsc stopped writing reports with 2.3.0

Hello and happy new year !

We installed dsc 2.3.0 on two servers shortly after the version release. Three days later, the daemon of one of the servers stopped writing the reports (but dsc kept running). The same situation occurred on the second server, 10 days after the installation of dsc 2.3.0.
It seems that no logs were written when the situations occurred.

We performed today a rollback to 2.1.1 which was the previously installed version on these servers.

Linking to unused libraries

dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/dsc/usr/bin/dsc was not linked against libresolv.so.2 (it uses none of the library's symbols)
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/dsc/usr/bin/dsc was not linked against libnsl.so.1 (it uses none of the library's symbols)

Allow multiple bpf_program and append content

Related to #108, suggestion is to allow for multiple bpf_program and append content to previous including one space as delimiter.

bpf_program "udp port 53";
bpf_program "and (";
bpf_program "host 192.168.0.1";
bpf_program "or host 192.168.0.2";
bpf_program ")";

Presenter graphic doesn't show CC for IPv6 clients

DSC's presenter portion doesn't display ISO Country Codes for IPv6 subnets that are clients of the DNS server being monitored. Instead, there's a double ?? displayed. This seems to be related to the use of the IP::Country module which is an old PERL module that only supports IPv4 whois lookups.

There's another implementation in CPAN similar to IP::Country, but requires upkeep of a database. Alternatively, the portion doing the CC code lookup code be replaced by something else entirely to get the appropriate info.

Conform to FHS

Change all paths into .in and let make generate them with defaults that conform to FHS and as closely as possible matches the old paths if installed into /usr/local.

Patches are from the Debian source package dsc-statistics for an old version, may not cover all places:

--- a/presenter/cron/refile-and-grok.sh
+++ b/presenter/cron/refile-and-grok.sh
@@ -1,7 +1,7 @@
 #!/bin/sh
 set -e

-cd /usr/local/dsc/data
+cd /var/lib/dsc-statistics/data

 PROG=`basename $0`
 exec >$PROG.stdout
@@ -9,7 +9,7 @@
 #set -x
 date

-EXECDIR=/usr/local/dsc/libexec
+EXECDIR=/usr/share/dsc-statistics-presenter
 export EXECDIR SERVER NODE

 for SERVER in * ; do
--- a/presenter/perllib/DSC/grapher.pm
+++ b/presenter/perllib/DSC/grapher.pm
@@ -27,8 +27,8 @@

 # CONSTANTS
 my $dbg_lvl = 0;   # also set debug_file in dsc-grapher.cfg
-my $DATAROOT = '/usr/local/dsc/data';
-my $DEFAULTCONFIG = '/usr/local/dsc/etc/dsc-grapher.cfg';
+my $DATAROOT = '/var/lib/dsc-statistics/data';
+my $DEFAULTCONFIG = '/etc/dsc-statistics/dsc-grapher.cfg';
 my $CacheImageTTL = 60;        # 1 min
 my $expires_time = '+1m';
 my $sublist_item ='&rsaquo;&nbsp;';
@@ -180,7 +180,7 @@
                $self->make_image($cache_name);
            }
        }
-       my $source = "/usr/local/dsc/share/html/plot.page";
+       my $source = "/usr/share/dsc-statistics-presenter/html/plot.page";
        my $t = Text::Template->new(
            TYPE => 'FILE',
            SOURCE => $source,
@@ -1005,13 +1005,13 @@
 sub cache_image_path {
    my $self = shift;
    my $prefix = shift || die;
-   "/usr/local/dsc/cache/$prefix.png";
+   "/var/cache/dsc-statistics-presenter/$prefix.png";
 }

 sub cache_mapfile_path {
    my $self = shift;
    my $prefix = shift || confess "cache_mapfile_path: no prefix given";
-   "/usr/local/dsc/cache/$prefix.map";
+   "/var/cache/dsc-statistics-presenter/$prefix.map";
 }

 # return 0 if we should generate a cached image
@@ -1587,11 +1587,11 @@
    my $self = shift;
    my $icon = shift;   # should be like 'foo.png"
    my $buf;
-   if (open(F, "/usr/local/dsc/share/html/$icon")) {
+   if (open(F, "/usr/share/dsc-statistics-presenter/html/$icon")) {
        $buf .= $_ while (<F>);
        close(F);
    } else {
-       warn "/usr/local/dsc/htdocs/$icon: $!\n";
+       warn "/usr/share/dsc-statistics-presenter/html/$icon: $!\n";
    }
    $buf;
 }
--- a/collector/cron/upload-prep.pl
+++ b/collector/cron/upload-prep.pl
@@ -20,8 +20,8 @@
 sleep 3;


-foreach my $conf (</usr/local/dsc/etc/*.conf>) {
+foreach my $conf (</var/run/dsc-statistics-collector/*.cfg>) {
    next unless open (CONF, $conf);
    my $rundir = undef;
    while (<CONF>) {

libpcap double reads pcap data from file under Ubuntu 16.04

Weird problem, libpcap seems to reset the file handle after clone() and then rereads all data. Need to report upstreams.

Strace diff output:

 open("./1458044657.pcap.dist", O_RDONLY) = 3
 fstat(3, {st_mode=S_IFREG|0664, st_size=1020, ...}) = 0
 read(3, "\324\303\262\241\2\0\4\0\0\0\0\0\0\0\0\0\377\377\0\0\1\0\0\0\357\376\347V \354\3\0"..., 4096) = 1020
-clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f76ad71aa10) = 7448
...
-lseek(3, -996, SEEK_CUR <unfinished ...>
...
-read(3, "\357\376\347V \354\3\0I\0\0\0I\0\0\0>1\220\201\223\263\2B\254\21\0\20\10\0E\0"..., 4096) = 996
...
 read(3, "", 4096)                 = 0

Additional output JSON

Hello,
Here is a proposal for adding an additional DSC output to the already existing XML one.

I already discussed a little of this matter on the mailing list but did not get quite a finale answer, except that some subscribers of the mailing list seem to be interested.

Basically I suggest to add an option 'add_output' that allows to generate JSON or Extended JSON (MongoDB) along with XML files.
JSON is a popular format and can be processed by many databases: MongoDB, ElasticSearch, PostgreSQL, SQL Server 2016, ...

This would allow users to have easy processing of DSC outputs for their businesses.

Code changes can be seen in PR #2. Any feedback is welcome.

aggregation of data / depth of tree

What would be the best strategy for dsc presenter to show/aggregate dsc data if the node structure is:

some.tld - master aggregation of all nodes
some.tld.anycast1 - first anycast node
some.tld.anycast1-node1
some.tld.anycast1-node2
some.tld.anycast2 - second anycast node
some.tld.anycast2-node1
some.tld.anycast2-node2

Currently looking at configuration i can only aggregate one level not two levels like would be required in this scenario, how do people resolve issue like that today with dsc ?

Compile failure on centos

I've just tried to compile 2.0.0-rc.1 on centos 6 and centos 7. Both builds fail. I've attached the build logs of both to this report.

Mainly, it seems like the compile is failing because it's expecting some libpcap feature that appears not to be present in the libpcap packages of centos.

centos6-build.txt

centos7-build.txt

dump_reports_on_exit does not work

Using dsc from today 94f680a
After kill <pid>:
dsc[25206]: Received signal 15 while dumping reports, exiting later
dsc[25206]: unable to pcap thread run: system error, check errno
dsc[25206]: system error [4]: Interrupted system call (select())

IMO the first log message is confusing, it should be something like: "Received signal 15 while capturing, dumping last report, exiting later"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.