accepting only one service is not working about ndpi-netfilter

[aewne]

Netfilter processes rules sequentially, meaning you need to add your drop rule to the end of your chain or table. Otherwise, a packet hitting the FORWARD table will be immediately dropped since the drop rule is the first one to match.

[rsweerarathna]

I tested both ways, but no luck. it drop all the traffic.
I saw,some has said that ndpi engine identified the protocols using first several packets going through the device up and down. As my drop rule blocking those initial packets as well, ndpi may not identify the protocols properly. Is it may be the problem?

[kong156]

First of all, default drop is always the last rule applied. default drop takes action at layer 4 which means that layer 7 inspection cannot be completed as long as no tcp connection is established.
Best practice is to accept as default rule and block at layer 7.
Did you try to negate a ndpi protocol in case of a drop rule?

[rsweerarathna]

as i know, iptable match option doesn't have a negation.
could you please give an example how to negate ndpi protocol.?

[kong156]

You are right... bad luck :)

[kong156]

Whitelisting some of protocols have major implications such as if you permit dns traffic at level 4 and block it at layer 7 doesn't make sense :)

[betolj]

You can try something like this, but its'nt so fair with other exist "http proto" like amazon and twitter.

iptables -N drop_http
iptables -A drop_http -m ndpi --facebook -j ACCEPT
iptables -A drop_http -m ndpi --youtube -j ACCEPT
iptables -A drop_http -m ndpi --http -j DROP
iptables -A drop_http -m ndpi --ssl -j DROP

iptables -I FORWARD -j drop_http

[kong156]

Best fit for ndpi-netfilter is to classify and prioritize l7 traffic. There is no need to block the undesirable traffic. Just apply QoS rules and I think it's enough.

[kong156]

By the way, betolj. Is there any way to mark as "unknown" the unlisted/unidentified ndpi protocols?

[betolj]

Hi, i also agree that the ideal is only make classifier rules for QoS strategy.

The "unknown" protocol id was suppressed from ndpi-netfilter checks becouse ndpi need some packet samples to detect the application type. So, i dont know how coding the proper match yet.

You can use MARK rules to do this...
For example (0x100 for unknown traffic):

iptables -t mangle -N mark_http
iptables -t mangle -A mark_http -j CONNMARK --restore-mark
iptables -t mangle -A mark_http -m mark --mark 0x0 -j MARK --set-mark 0x100
iptables -t mangle -A mark_http -m mark ! --mark 0x100 -j RETURN
iptables -t mangle -A mark_http -m ndpi --facebook -j MARK --set-mark 0x101
iptables -t mangle -A mark_http -m ndpi --youtube -j MARK --set-mark 0x102

iptables -t mangle -I PREROUTING -j mark_http
iptables -t mangle -A POSTROUTING -m mark ! --mark 0x100 -j CONNMARK --save-mark

iptables -t mangle -I OUTPUT -j mark_http

[betolj]

And, you can add a limit detection for 10 packets too:
iptables -t mangle -I mark_http 4 -m connbytes --connbytes-dir both --connbytes-mode packets --connbytes 10 -j RETURN

[kong156]

Is there any info about how many packets have to be exchanged to detect every listed ndpi protocols?

[betolj]

Not exactly, but the http (or based) need little packets exchange to detect the protocol. We know that only with '3 way-handshake' it's need at least three packets, if there is no retransmission. And, with packet limit, we could avoid the analysis of large downloads.

But... Anyway, i coded in ndpi-netfilter a 30 sec interval as detection limit.

And once the connection is detected, there is no DPI inspections any more - the result is cached.

[kong156]

In this situation, I ask you if it is suitable to present the connections as "unknown" after that 30s limit and the traffic being identified as "ongoing" or something like that. It would simplify iptables configuration.

[kong156]

the Mikrotik way :) ...

"L7 matcher collects the first 10 packets of a connection or the first 2KB of a connection and searches for the pattern in the collected data. If the pattern is not found in the collected data, the matcher stops inspecting further. Allocated memory is freed and the protocol is considered as unknown."

[rsweerarathna]

Hi, Is it possible to mark youtube packets and apply a traffic shaper for only that protocol using tc command? I tried several times, but no luck. Please give some working example if possible.

[betolj]

Take a look in this example:
https://github.com/betolj/ndpi-netfilter/blob/master/ingress-qos.sh

It's a little complex... need kernel 4.2 and iproute2 4.1.1 to shape ingress flow.

[rsweerarathna]

I successfully implement the traffic shaper for youtube using tc and CLASSIFY target. It works fine. Here is my rule set.

sudo tc qdisc add dev eth1 root handle 1: htb default 10
sudo tc class add dev eth1 parent 1: classid 1:1 htb rate 1Mbps
sudo tc class add dev eth1 parent 1:1 classid 1:10 htb rate 100kbps ceil 100kbps prio 10

sudo iptables -t mangle -A POSTROUTING -s 192.168.200.27 -m ndpi --youtube -j CLASSIFY --set-class 1:10
sudo iptables -t mangle -A POSTROUTING -d 192.168.200.27 -m ndpi --youtube -j CLASSIFY --set-class 1:10

This thing works very well when i use firefox for browsing. But the problem is when i use google chrome it doesn't work.
Is google chrome altering the packet header or something? Anyone else have figure this out before?

[kong156]

when using ndpi library youtube is seen frequently as quic protocol.

[rsweerarathna]

Yes, i drop QUIC first and then apply the shaping. then it works fine. thanks kong156.
Have another problem now. I use bittorent and transmission torrent clients. i need to drop or shape torrent traffic. Although i drop bittorrent for my whole network by a single rule, it downloads fine. any other supporting protocols need to be blocked to implement this?
Below is my rules. why this is not working?

sudo iptables -A FORWARD -d 192.168.200.0/24 -m ndpi --bittorrent -j DROP
sudo iptables -A FORWARD -s 192.168.200.0/24 -m ndpi --bittorrent -j DROP

[rsweerarathna]

And bellow error is continuously logging in /var/log/kern.log

Feb 26 12:41:01 machine kernel: [15216.622249] [NDPI] Internal error: protocol BitTorrent/37 has been already registered
Feb 26 12:41:52 machine kernel: [15267.730375] [NDPI] Internal error: protocol SSL/91 has been already registered
Feb 26 12:41:52 machine kernel: [15267.730375] [NDPI] Internal error: protocol DNS/5 has been already registered

what is the reason for this?

[kong156]

Would you dump your iptables config?

[rsweerarathna]

No, This error appears when i match same protocol twice. when the first rule added no problem. and when adding the second rule matching the same ndpi protocol, this error appear in kernel log.

[kong156]

Is this true for every proto you match twice? ex SSL, DNS?

[rsweerarathna]

yes. now i'm going to install new machine and test again from the beginning.
Because i update the kernel and it's giving some error when loading the xt_ndpi kernel module.
I'l try from the beginning and update you is it solve.

[mmanoj]

Hi betolj & Kong,
If we have same scenarios with more than 100,000 IP addresses, what is the best practice to handle QoS with IP table+ndpi and classes ? As per our test ipset not work for QoS bandwidth control.

We mainly focus on manageability and IP table performance for large individual rule set for QoS.

P.S we check http://www.hipac.org/ project but not having full features of IP-Table yet.

[kong156]

What is your classification criteria?

[mmanoj]

Hi Kong,
for example simile to above discuss scenarios youtube / bittorrent etc. need to control via traffic classes either give more bandwidth or downgrade based on the logic per each user. Because each user belongs to different kind of profile (home user, student etc.)

[kong156]

Did you try to chain your ipset match with ndpi rules which can be used for traffic marking and then sending to the right htb pipe?

[mmanoj]

Thank you for the quick reply, can you provide me some example how to do it. Will try and update you.

[kong156]

Try to group your ipset rules into a ipset specific chain and for every match jump to a ndpi chain where you can classify based on ndpi protocols. You already have some examples across a few topics around here :)

[mmanoj]

1. create student set --> add IP belongs to student's profiles users to the set
2. create IP table rule with above set for ndpi check with classify

we done simile rules, the issue we found out bandwidth allocated to youtube group(1Mbps) will divide among the users in the IPset group. I cannot guaranteed every user in the youtube group will get 1Mbps. Assuming I have enough bandwidth for main class. Any idea on this ? how to resolve bandwidth sharing problem. The bottom line is we need to give every user in the student set 1Mbps download bandwidth for youtube etc.

[betolj]

1. To shape a fixed rate per client, you can configure delay_pool (class 2) in squid proxy server.
2. But, with tc qdisc you can make packet priority based in ndpi classification and share the bandwith with a fair queue algorithm like sfq or fq_codel.
   https://wiki.gentoo.org/wiki/Traffic_shaping

[mmanoj]

Hi Betolj,

Thank you for the feedback, it's mean we have to forward traffic to transparent proxy for further down to control the bandwidth after identify from ndpi ? give more details on this scenario.

Will check the other suggested solution also and update you based on test.

[betolj]

With transparent proxy you can shape any http traffic using delay_pools and domain acl criterea, without ndpi classification. But, i prefer shape by subnet and split bandwith using a fair queue algo.

[rsweerarathna]

Hi Betolj,
It means that we can not use transparent proxy for shaping youtube or bittorrent traffic? only whole traffic to one ip or a subnet?

[betolj]

You can. But, only for web traffic. The bittorrent, for example, tends to open connections without going through the proxy (it's a direct routing). With Squid proxy you can configure different scenaries - a fixed rates with delay_pools (class 2) or re-mark packets using "tcp_outgoing_mark". There are many options.

To share the bandwith equally, you will need a fair queue discipline (like sfq). In the most cases (large networks) itsn't practicable to create a qos class per ip. So, you can create a subnet hierarchy subdivided by service type (like interactive, normal_traffic and p2p, for example). To split p2p traffic you can include a SFQ ou FQ_CODEL leaf.

[mmanoj]

Hi,
One clarification as per our scenario , we cannot create predefined subnet, because client IP is dynamic and same IP can get various profiles each time one it assign to different user (based on user profile), so one time IP can get more services and bandwidth, next time with different user it goes low profile bandwidth/shaped etc.

Please advice.

[kong156]

You may stick with ipset solution but the scalability and performance will suffer. Better is to integrate with IP address leasing infrastructure if possible.

[mmanoj]

Yes I agreed with you. But IP address leasing will lead to errors from business logic and tracking. Is their any way to do via customised module or kind of code level change.

[kong156]

I see no reason for errors as long as there is a minimal logic to map a client id to a user profile such as dhcp user class option or pppoe user auth.

[rsweerarathna]

I tried with sfq, and it divides the whole bandwidth equally to all users in the ipset. It's somewhat fair, but will not solve our problem.

[mmanoj]

Hi betolj & Kong

Any plan to extend this ndpi module to NFTable ? we mainly looking to improve the TP table lookup performance as per our unique scenario, individual rules giving better results and we have granular level control of each user. Please advice how to achieve this/ to get start to working on it.
Thanks in advance.

[faizatahir94]

Hi @betolj , @kong156

As commented by rsweerarathna for the chrome issue , I am not able to control bandwidth of YouTube on Microsoft Edge.
After adding drop rules for QUIC protocol, I was able to achieve specified YouTube's bandwidth on chrome successfully. But this does not work for Microsoft Edge browser. Can you please suggest me any possible solution?