max197616 / nfqfilter Goto Github PK

View Code? Open in Web Editor NEW

34.0 34.0 22.0 474 KB

License: GNU General Public License v3.0

Shell 0.03% C++ 71.34% C 24.96% Makefile 0.41% M4 3.25%

nfqfilter's People

Contributors

Stargazers

Watchers

Forkers

lumenods egrekov surgeonalex zar1n ne-vlezay80 nightweb hotid ellund foxdavion vans1 alex-eri i39 5un5h1n3 mzhumakhanov enterstudio kiwinet stroboscope kaowul 2561024 a-pog int2ecall

nfqfilter's Issues

Блокировка домена по маске.

https://eais.rkn.gov.ru/docs/Recomendation.pdf
п. 12
Блокировка домена по маске *.domain.com

Блокировка всех страниц если в urllist присутствует корневая страница сайта.

Небольшой баг
При наличии в urllist корневой страницы сайта domain/ блокируются все страницы.
При блокировке url вида domain/123 блокируюся все url вида domain/123*

Точка в домене (URL лист)

Если в списке url есть что типа osago-blank.co./index.htm то nfqfilter пропускает его.
Это проблему лучше решать на этапе формирования списков ?

Блокировка url с учетом фрагминтации

Имееv url:
http://spys.ru/
Если у клиента маленькое мту, то url будет таким:
http://spys.r
u/
Короче, как идея реализовать добавление url в фильтр с учетом фрагментации?

Не собралось на ubuntu

Есть комп:

uname -a
Linux nas 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

при попытки собрать, получил:

# make
Making all in src
make[1]: Entering directory '/home/nfqfilter/src'
g++ -DHAVE_CONFIG_H -I. -I../include  -I../nDPI/src/include   -std=c++0x -O2 -pthread -MT main.o -MD -MP -MF .deps/main.Tp                      o -c -o main.o main.cpp
In file included from /usr/include/Poco/Util/ServerApplication.h:44:0,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Util/Application.h:358:35: error: ‘typedef class std::vector<std::__cxx11::basic_string<char> > Poco::Ut                      il::Application::ArgVec’ is private
  typedef std::vector<std::string> ArgVec;
                                   ^
In file included from main.cpp:23:0:
../include/main.h:150:17: error: within this context
  int main(const ArgVec& args);
                 ^
main.cpp: In member function ‘virtual void nfqFilter::initialize(Poco::Util::Application&)’:
main.cpp:125:60: error: no matching function for call to ‘Poco::Logger::debug(const char [30], std::__cxx11::string&)’
  logger().debug("URL additional info set to %s", add_p_type);
                                                            ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 2 provided
main.cpp:132:50: error: no matching function for call to ‘Poco::Logger::debug(const char [20], std::__cxx11::string&)’
   logger().debug("HTTP code set to %s", http_code);
                                                  ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 2 provided
main.cpp:147:69: error: no matching function for call to ‘Poco::Logger::information(const char [25], int&)’
  logger().information("Starting up on queue: %d",_config.queueNumber);
                                                                     ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:409:13: note: candidate: void Poco::Logger::information(const string&)
 inline void Logger::information(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:409:13: note:   candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:463:13: note: candidate: bool Poco::Logger::information() const
 inline bool Logger::information() const
             ^
/usr/include/Poco/Logger.h:463:13: note:   candidate expects 0 arguments, 2 provided
In file included from /usr/include/Poco/Util/ServerApplication.h:44:0,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Util/Application.h: In member function ‘virtual int nfqFilter::main(const ArgVec&)’:
/usr/include/Poco/Util/Application.h:358:35: error: ‘typedef class std::vector<std::__cxx11::basic_string<char> > Poco::Ut                      il::Application::ArgVec’ is private
  typedef std::vector<std::string> ArgVec;
                                   ^
main.cpp:261:27: error: within this context
 int nfqFilter::main(const ArgVec& args)
                           ^
main.cpp: In member function ‘void nfqFilter::loadDomains(std::__cxx11::string&, AhoCorasickPlus*, DomainsMatchType*)’:
main.cpp:312:86: error: no matching function for call to ‘Poco::Logger::warning(const char [51], std::__cxx11::string&, st                      d::__cxx11::string&)’
       logger().warning("Pattern '%s' already present database from file %s",insert,fn);
                                                                                      ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:397:13: note: candidate: void Poco::Logger::warning(const string&)
 inline void Logger::warning(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:397:13: note:   candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:451:13: note: candidate: bool Poco::Logger::warning() const
 inline bool Logger::warning() const
             ^
/usr/include/Poco/Logger.h:451:13: note:   candidate expects 0 arguments, 3 provided
main.cpp:314:85: error: no matching function for call to ‘Poco::Logger::error(const char [45], std::__cxx11::string&, int&                      , std::__cxx11::string&)’
       logger().error("Failed to add '%s' from line %d from file %s",insert,lineno,fn);
                                                                                     ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:391:13: note: candidate: void Poco::Logger::error(const string&)
 inline void Logger::error(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:391:13: note:   candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:445:13: note: candidate: bool Poco::Logger::error() const
 inline bool Logger::error() const
             ^
/usr/include/Poco/Logger.h:445:13: note:   candidate expects 0 arguments, 4 provided
main.cpp:320:85: error: no matching function for call to ‘Poco::Logger::debug(const char [48], std::__cxx11::string&, int&                      , std::__cxx11::string&)’
       logger().debug("Inserted domain: '%s' from line %d from file %s",str,lineno,fn);
                                                                                     ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 4 provided
main.cpp:322:84: error: no matching function for call to ‘Poco::Logger::debug(const char [47], std::__cxx11::string&, int&                      , std::__cxx11::string&)’
       logger().debug("Updated domain: '%s' from line %d from file %s",str,lineno,fn);
                                                                                    ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 4 provided
main.cpp: In member function ‘void nfqFilter::loadURLs(std::__cxx11::string&, AhoCorasickPlus*)’:
main.cpp:360:94: error: no matching function for call to ‘Poco::Logger::warning(const char [62], std::__cxx11::string&, st                      d::__cxx11::string&)’
       logger().warning("Pattern '%s' already present in the URL database from file %s",str,fn);
                                                                                              ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:397:13: note: candidate: void Poco::Logger::warning(const string&)
 inline void Logger::warning(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:397:13: note:   candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:451:13: note: candidate: bool Poco::Logger::warning() const
 inline bool Logger::warning() const
             ^
/usr/include/Poco/Logger.h:451:13: note:   candidate expects 0 arguments, 3 provided
main.cpp:362:82: error: no matching function for call to ‘Poco::Logger::error(const char [45], std::__cxx11::string&, int&                      , std::__cxx11::string&)’
       logger().error("Failed to add '%s' from line %d from file %s",str,lineno,fn);
                                                                                  ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:391:13: note: candidate: void Poco::Logger::error(const string&)
 inline void Logger::error(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:391:13: note:   candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:445:13: note: candidate: bool Poco::Logger::error() const
 inline bool Logger::error() const
             ^
/usr/include/Poco/Logger.h:445:13: note:   candidate expects 0 arguments, 4 provided
main.cpp: In member function ‘void nfqFilter::loadHosts(std::__cxx11::string&, IPPortMap*)’:
main.cpp:394:47: error: no matching function for call to ‘Poco::Logger::debug(const char [17], std::__cxx11::string&, std:                      :__cxx11::string&)’
      logger().debug("IP is %s port %s",ip,port);
                                               ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 3 provided
main.cpp:397:45: error: no matching function for call to ‘Poco::Logger::debug(const char [19], std::__cxx11::string&)’
      logger().debug("IP %s without port", ip);
                                             ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 2 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 2 provided
main.cpp:406:57: error: no matching function for call to ‘Poco::Logger::debug(const char [24], std::__cxx11::string&, std:                      :__cxx11::string&)’
       logger().debug("Adding port %s to ip %s", port, ip);
                                                         ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 3 provided
main.cpp:410:63: error: no matching function for call to ‘Poco::Logger::debug(const char [29], std::__cxx11::string&, int&                      )’
      logger().debug("Inserted ip: %s from line %d", ip, lineno);
                                                               ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 3 provided
main.cpp:412:75: error: no matching function for call to ‘Poco::Logger::debug(const char [37], std::__cxx11::string&, int&                      , std::__cxx11::string&)’
      logger().debug("Adding port %s from line %d to ip %s", port,lineno,ip);
                                                                           ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:415:13: note: candidate: void Poco::Logger::debug(const string&)
 inline void Logger::debug(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:415:13: note:   candidate expects 1 argument, 4 provided
/usr/include/Poco/Logger.h:469:13: note: candidate: bool Poco::Logger::debug() const
 inline bool Logger::debug() const
             ^
/usr/include/Poco/Logger.h:469:13: note:   candidate expects 0 arguments, 4 provided
main.cpp: In member function ‘void nfqFilter::loadSSLIP(const string&, Patricia*)’:
main.cpp:440:102: error: no matching function for call to ‘Poco::Logger::information(const char [61], std::__cxx11::string                      &, int&)’
      logger().information("Unable to add IP address %s from line %d to the SSL IPs list", str, lineno);
                                                                                                      ^
In file included from /usr/include/Poco/Util/Application.h:48:0,
                 from /usr/include/Poco/Util/ServerApplication.h:44,
                 from ../include/main.h:23,
                 from main.cpp:23:
/usr/include/Poco/Logger.h:409:13: note: candidate: void Poco::Logger::information(const string&)
 inline void Logger::information(const std::string& msg)
             ^
/usr/include/Poco/Logger.h:409:13: note:   candidate expects 1 argument, 3 provided
/usr/include/Poco/Logger.h:463:13: note: candidate: bool Poco::Logger::information() const
 inline bool Logger::information() const
             ^
/usr/include/Poco/Logger.h:463:13: note:   candidate expects 0 arguments, 3 provided
Makefile:390: recipe for target 'main.o' failed
make[1]: *** [main.o] Error 1
make[1]: Leaving directory '/home/nfqfilter/src'
Makefile:340: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1

Время вывода статистики в логи.

В конфиге nfqfilter.ini:

; время вывода статистики по использованию памяти, минут
statistic_interval = 10

В логах:

2016-08-15 04:20:16.363 [30753] Information Application - nDPI memory (once): *** KB
2016-08-15 04:20:16.363 [30753] Information Application - nDPI memory per flow: *** KB
2016-08-15 04:20:16.363 [30753] Information Application - nDPI current memory usage: * MB
2016-08-15 04:20:16.363 [30753] Information Application - nDPI maximum memory usage: * GB
2016-08-15 04:20:16.363 [30753] Information Application - Total seen packets: _, Total seen bytes: *, Average packet size: *_* bytes, Traffic throughput: * pps
2016-08-15 04:20:16.363 [30753] Information Application - Total matched by ip/port: _, Total matched by ssl: *, Total matched by ssl/ip: _
2016-08-15 04:20:16.364 [30753] Information Application - Total redirected domains _, Total redirected urls: *, Total marked ssl: *, Total marked hosts: *, Total rst sended: _
2016-08-15 04:20:26.364 [30753] Information Application - nDPI memory (once): *** KB
2016-08-15 04:20:26.364 [30753] Information Application - nDPI memory per flow: *** KB
2016-08-15 04:20:26.364 [30753] Information Application - nDPI current memory usage: *** MB
2016-08-15 04:20:26.364 [30753] Information Application - nDPI maximum memory usage: * GB
2016-08-15 04:20:26.364 [30753] Information Application - Total seen packets: _, Total seen bytes: *, Average packet size: * bytes, Traffic throughput: * pps
2016-08-15 04:20:26.364 [30753] Information Application - Total matched by ip/port: _, Total matched by ssl: *__, Total matched by ssl/ip: *`

То есть на самом деле это не 10 минут а 10 секунд.
Наверно стоит поправить описание в nfqfilter.ini, или в коде. Не знаю где правильнее.

Не фильтруются "длинные" домены

Если в файл domains добавить доменное имя длиннее чем 191 символ (определили опытным путем), то он не фильтруется.
В тоже время, если этот же домен добавить в ulrs, то он удачно блокируется.
В данный момент времени в реестре есть одна запись длинной 206 символов:
www.legal-pills.net.kupit-tabletki-kadillak-adam-boby-yasnost-eyv-lyubov-ulybka.snezhok-vitaminki-diski-tably-kolosa-bubliki-kalachi-muzyka.ekstazi-kruglye-yeshki-eksta-eks-ti-si-tapki-bubly-pilsy-eyfor.net

Отладочное сообщение о нахождении домена в списке

Имеем сообщения (все) такого вида):
Host poker.123365-sb.com present in domain (file line [ERRFMT]) list from ip ХХ.ХХ.ХХ.ХХ
Вместо [ERRFMT] явно должен быть номер строки...

Матчинг маркированных пакетов в NAT

Добрый день. Маркированные пакеты не попадают в NAT таблицу.

Счетчики на этом правиле растут
iptables -A FORWARD -m mark --mark 17 -p tcp -j ACCEPT

А на этом нет
iptables -t nat -A PREROUTING -m mark --mark 17 -p tcp -j ACCEPT

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble