bittorrent maching is not working properly about ndpi-netfilter

[rsweerarathna]

I installed the latest ndpi version from "https://github.com/ntop/nDPI" and run the ndpiReader example. it seems that torrent maching is works fine. Can i compile latest ndpi version with ndpi-netfilter?

[kong156]

This matter was already discussed. Please check the older topics.

[kong156]

Please see "kernel panic and build issue w/ current" #8

[rsweerarathna]

yes i read them,
for some torrents, it's working fine, but some are goes as unknown traffic.
is there anyway to block unknown traffic?

[kong156]

Please test this particular rule iptables -A FORWARD -m ndpi --bittorrent -j ACCEPT and check if the counters are matching most of the traffic and with what ratio.

[rsweerarathna]

I applied this rule to a particular ip and started downloading a torrent size of 9.82MB. But rule matched very less amount of packets as follows.

Chain INPUT (policy ACCEPT 3294 packets, 215K bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 205K packets, 135M bytes)
pkts bytes target prot opt in out source destination
4216 5955K ACCEPT all -- * * 0.0.0.0/0 192.168.200.15 protocol BITTORRENT
3451 214K ACCEPT all -- * * 192.168.200.15 0.0.0.0/0 protocol BITTORRENT

Chain OUTPUT (policy ACCEPT 3281 packets, 543K bytes)
pkts bytes target prot opt in out source destination

[rsweerarathna]

It's nearly half of the full size of torrent.

[dwipujono]

Same issue here, only matched 78MB of 744MB, downloading torrent using aria2. But match all with rtorrent.

Using ubuntu 14.04 server, kernel 3.19.0-51.

[kong156]

Same results with ntopng?

[kong156]

Please try to connmark the bittorent connections on POSTROUTING, restore on PREROUTING and match your marks on FORWARDING. You could do the same thing with a local bt client and match on INPUT and please compare the results. thanks...

[dwipujono]

Hi Kong,

ntopng shows most of the protocol is Unknown (TCP) when using aria2 (BT 13.3%, Unknown 86.6%).
Using rtorrent: BT 97,3%, Others: 2.7%.

I will try connmark later.

[dwipujono]

I test with transmission-cli, ntopng result: BT 54,8%, Unknown: 45,2%

[dwipujono]

Below are the result of connmark bittorrent connections

************* Download torrent using aria2 *********************

Chain PREROUTING (policy ACCEPT 571K packets, 842M bytes)
pkts bytes target prot opt in out source destination
571K 842M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore

Chain POSTROUTING (policy ACCEPT 411K packets, 25M bytes)
pkts bytes target prot opt in out source destination
425K 26M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
14162 994K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x0
754 95787 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0protocol BITTORRENT MARK set 0x1
411K 25M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

Chain INPUT (policy ACCEPT 555K packets, 819M bytes)
pkts bytes target prot opt in out source destination
16021 23M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1

---

************* Download torrent using rtorrent *********************

Chain PREROUTING (policy ACCEPT 539K packets, 818M bytes)
pkts bytes target prot opt in out source destination
539K 818M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore

Chain POSTROUTING (policy ACCEPT 17748 packets, 1164K bytes)
pkts bytes target prot opt in out source destination
489K 30M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
472K 28M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x0
103 12841 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0protocol BITTORRENT MARK set 0x1
17748 1164K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

Chain INPUT (policy ACCEPT 19196 packets, 27M bytes)
pkts bytes target prot opt in out source destination
520K 792M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1

---

[kong156]

Same torrent? If so, It looks like ndpi cannot deal with the way aria2 downloads the torrent. Could you set a default DROP rule on INPUT chain and accept only the marked ones?

[rsweerarathna]

I tried with default DROP and allow ndpi marked torrent traffic. But it drops all the packets, because DROP rule is in layer 4 and ndpi marking is at layer7. As Drop rule cannot identify layer 7 marks it DRO everything, So it's not possible to mark torrent traffic at layer4.

[kong156]

Could you dump the input chain? also make sure that you have a related, established rule set on input chain.

[kong156]

I've just tried aria2c and yes, this client passes ndpi inspection successfully. I think this is a matter of ndpi library protocol dissection. You could solve this by sending a pcap file to ndpi developers in order to fix this issue, if possible...

[freeyoung]

Same question as @rsweerarathna. I used -j ACCEPT just to make sure if ndpi-netfilter could recognise the BT traffic but no luck. The counters were always 0.

With ndpiReader, torrent matching works perfectly fine.

[rsweerarathna]

Hi bitolj and kong156,
nDPI version 1.8 has released. But ndpi-netfilter is still using old version. Are you not going to update the code? I like to contribute to update it with latest nDPI library. I tried to compile with new library but couldn't. It needs some code change, right? If you are not going to update the code, Please give some necessary instructions me to update it.

[betolj]

Hi,

I'm working on it in this week. I can build the binaries now, but I'm reviewing
the spinlocks calls.

2016-06-16 3:05 GMT-04:00 rsweerarathna [email protected]:

Hi bitolj and kong156,
nDPI version 1.8 has released. But ndpi-netfilter is still using old
version. Are you not going to update the code? I like to contribute to
update it with latest nDPI library. I tried to compile with new library but
couldn't. It needs some code change, right? If you are not going to update
the code, Please give some necessary instructions me to update it.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#25 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AEBma2oSVcuKF4b1oqeW7cXTQrGSTrNaks5qMPW_gaJpZM4HtcWS
.

[rsweerarathna]

Good to hear that, im eagerly waiting for it. What are the improvments made in ndpi-netfilter? I mean max no of rules, performance,etc. New nDPI has some new protocols and improved decectors. But what about the performance and scalability?

[elico]

@betolj I was wondering if there is a version of OS and kernel that is not affected by:
#11 (comment) ?

[betolj]

Try the lastest ndpi-netfilter update (v3.0).

• You can load with debug messages (but avoid it in a production environment)
  modprobe xt_ndpi debug_dpi=1

[betolj]

The encrypted bittorrent traffic detection does not work well.

[betolj]

Good results with ktorrent without encrypt option.
No kernel panics (spin_lock fix).