zr-hebo / sniffer-agent Goto Github PK
View Code? Open in Web Editor NEWSniffer MySQL package: capture TCP package, parsed with mysql protocol, optional you can send query info to Kafka. 抓取tcp包解析出mysql语句
Sniffer MySQL package: capture TCP package, parsed with mysql protocol, optional you can send query info to Kafka. 抓取tcp包解析出mysql语句
长期抓包会不会oom , 如果某些会话 一直不断开跑批量任务,或者pcap漏掉了fin包,那sessionPool 就会很大吧?
capture_packet_rate设置为1
测试mysql抓包的时候,发现并发不高的情况下,也有很多语句抓不到,这块能否优化下呢?
./sniffer-agent --interface=lo --port=3306 本机root是localhost用户无法抓取审计信息。
执行go buid:
xxxx
Username for 'https://github.com':
Password for 'https://github.com':
xxxx
mysql client
select * from t_user limit 1;update t_user set name='ADFAFD' limit 1;
sniffer-agent的输出
{"sip":"10.10.20.176","sport":3306,"cpr":1,"bt":1681099755827,"cip":"10.10.20.177","cport":55697,"user":"root","db":"test_fly","sql":"update t_user set name='ADFA","cms":4}
{"sip":"10.10.20.176","sport":3306,"cpr":1,"bt":1681099755838,"cip":"10.10.20.177","cport":55697,"user":"root","db":"test_fly","sql":"update t_user set name='ADFAFD' limit 1","cms":11}
不能准确的抓取第一个SQL语句的内容
[root@host-172-16-10-12 sniffer-agent-1.2.0]# ./sniffer-agent --interface=eth0 --port=3307 --export_type=cli
set config capture_packet_rate: 1
INFO[0000] parsed local ip address:172.16.10.12
set config capture_packet_rate: 1
panic: package size bigger than max buffer size need deal:0
goroutine 24 [running]:
github.com/zr-hebo/sniffer-agent/util.(*SliceBufferPool).DequeueWithInit(0xc0000a8fe0, 0x54, 0x0, 0x0, 0x0)
/usr/local/sniffer-agent-1.2.0/util/cache_pool.go:43 +0x165
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).readFromClient(0xc000174000, 0x64ab3a25, 0xc000258042, 0x58, 0xffbe)
/usr/local/sniffer-agent-1.2.0/session-dealer/mysql/session.go:182 +0xe6
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).ReceiveTCPPacket(0xc000174000, 0xc00016c060)
/usr/local/sniffer-agent-1.2.0/session-dealer/mysql/session.go:91 +0x90
github.com/zr-hebo/sniffer-agent/capture.readToServerPackage(0x0, 0x0, 0xc000172020, 0xd2e8, 0xc0001648c0, 0xc0000872c0, 0x0, 0x0)
/usr/local/sniffer-agent-1.2.0/capture/network.go:295 +0x19b
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).parseTCPPackage(0xc0000a9480, 0xb3ac00, 0xc0000ecc60, 0x0)
/usr/local/sniffer-agent-1.2.0/capture/network.go:212 +0x22c
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal.func1(0xc0000a9480)
/usr/local/sniffer-agent-1.2.0/capture/network.go:164 +0x41a
created by github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal
/usr/local/sniffer-agent-1.2.0/capture/network.go:107 +0x3f
启动2个docker容器,一个容器跑的centos并安装了mysql8.0,此外安装并执行sniffer agent以及tcpdump。
另外一个docker容器,允许mysql连接到第一个容器中去执行sql。
tcpdump可以抓到包,但是sniffer agent抓不到数据,什么也不显示。
您好,请问一下有计划在目前 MySQL 协议基础上,添加“返回行数”和“执行状态(成功/失败)”这两个功能
请问下编译报这个错误如何处理:
[root@vm10-2-103-156 sniffer-agent]# go build -mod=readonly
/root/go/pkg/mod/github.com/google/[email protected]/pcap/pcap_unix.go:34:18: fatal error: pcap.h: No such file or directory
#include <pcap.h>
^
compilation terminated.
[root@vm10-2-103-156 sniffer-agent]#
DEBU[0009] receive an unexpect packet
INFO[0009] expect receive size is bigger than max deal size: 131072
INFO[0009] expect receive size is bigger than max deal size: 131072
INFO[0009] expect receive size is bigger than max deal size: 131072
[root@test_5_186 sniffer-agent]# ./sniffer-agent --interface=ens160 --port=3306 --log_level=debug --capture_packet_rate=1000 --export_type=cli --max_packet_length=13107200
set config capture_packet_rate: 1
INFO[0000] parsed local ip address:10.6.5.186
INFO[0000] begin listen
set config capture_packet_rate: 1000
另一边模拟数据库登录,执行sql,但sniffer-agent后面无任何输出,
作者您好,我关注这个项目已经很久了,感谢您的开源,看到您最近更新了 macOs版本的agent,想请问一下 什么时候有windows mysql版本呢,或者有这个计划在内吗。
set config capture_packet_rate: 1
INFO[0000] parsed local ip address:10.128.0.105
set config capture_packet_rate: 1
panic: package size bigger than max buffer size need deal:0
goroutine 23 [running]:
github.com/zr-hebo/sniffer-agent/util.(*SliceBufferPool).DequeueWithInit(0xc0000a4cc0, 0x8e, 0x0, 0x0, 0x0)
/root/sniffer-agent/util/cache_pool.go:43 +0x190
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).readFromClient(0xc0001fc000, 0x5f2c57e1, 0xc0001e6042, 0x92, 0xffbe)
/root/sniffer-agent/session-dealer/mysql/session.go:182 +0xdc
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).ReceiveTCPPacket(0xc0001fc000, 0xc00010bad0)
/root/sniffer-agent/session-dealer/mysql/session.go:91 +0x8b
github.com/zr-hebo/sniffer-agent/capture.readToServerPackage(0x0, 0x0, 0xc000097e30, 0xabd4, 0xc0001f8000, 0xc0000b0d20, 0x0, 0x0)
/root/sniffer-agent/capture/network.go:295 +0x19f
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).parseTCPPackage(0xc0000a5180, 0xba39a0, 0xc0000ef760, 0x0)
/root/sniffer-agent/capture/network.go:212 +0x245
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal.func1(0xc0000a5180)
/root/sniffer-agent/capture/network.go:151 +0x601
created by github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal
/root/sniffer-agent/capture/network.go:107 +0x3f
麻烦帮忙看下什么原因
我通过命令:
./sniffer-agent --interface=eth0 --port=6033 --log_level=info --strict_mode=true --admin_user=user_microloan_proxyA --admin_passwd=xxx> res
抓出到的SQL日志为:
{"sip":"10.106.3.180","sport":6033,"cpr":1,"bt":1581173313742,"cip":"10.106.3.31","cport":6033,"user":null,"db":null,"sql":"INSERT INTO thirdparty_statistic_customer
(id
, user_account_id
, order_id
, mobile
, api
, api_md5
, api_fee
, call_count
, success_call_count
, hit_call_count
, cutomer_total_cost
, record_type
, ctime
, utime
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)","cms":2}
发现两个问题
1、开启了--strict_mode=true 但是抓取的db,user信息仍是NULL
2、程序是先prepare语句, 然后执行的语句的时候赋值,但是 sniffer-agent 生成的SQL 仍无法抓取到赋值。
go version go1.12.17 linux/amd64:
cannot find module providing package github.com/zr-hebo/validator
panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
goroutine 1 [running]:
github.com/zr-hebo/sniffer-agent/exporter.NewKafkaExporter(0xa9d750)
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/exporter/kafka.go:83 +0x329
github.com/zr-hebo/sniffer-agent/exporter.NewExporter(0xc00018feb8, 0x512d9b)
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/exporter/model.go:26 +0x8c
main.mainServer()
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/main.go:50 +0x34
main.main()
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/main.go:46 +0x9a
能否提供一个编译后可执行的包?或者提供一下go env的配置参数?
谢啦
DEBU[8871] receive an unexpect packet
DEBU[8871] receive an unexpect packet
DEBU[8871] receive an unexpect packet
DEBU[8871] receive an unexpect packet
请问这个报错是什么原因?
zr-hebo,你好,我找了很多的抓包工具,目前最好的就是你这个抓包工具了,感谢作者的开源
我的golang水平不太高。关于源码里有一些疑惑
就是BaseQueryPiece这个结构体的Recovery()明明没有做任何事情 是怎么产生出抓包功能的呢?
PooledMysqlQueryPiece的结构体带的Recovery()函数是有具体作用
但是我从main函数入口去看 ,只有一个 nc.receiver <- model.NewBaseQueryPiece(localIPAddr, nc.listenPort, capturePacketRate) 这个明明是返回的BaseQueryPiece 所以很疑惑
如题,sniffer可以动态的调整抓包率,但是调整的依据尚不明确,sniffer是否可以提供一个实时qps查询的功能,之后用户根据实时的qps值调整抓包率
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.