GithubHelp home page GithubHelp logo

zr-hebo / sniffer-agent Goto Github PK

View Code? Open in Web Editor NEW
93.0 93.0 42.0 2.23 MB

Sniffer MySQL package: capture TCP package, parsed with mysql protocol, optional you can send query info to Kafka. 抓取tcp包解析出mysql语句

Makefile 0.02% Go 91.39% Python 5.71% Shell 2.89%

sniffer-agent's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sniffer-agent's Issues

长期抓包 会oom么?

长期抓包会不会oom , 如果某些会话 一直不断开跑批量任务,或者pcap漏掉了fin包,那sessionPool 就会很大吧?

mysql8.0抓包很多语句抓不到

capture_packet_rate设置为1
测试mysql抓包的时候,发现并发不高的情况下,也有很多语句抓不到,这块能否优化下呢?

[Bug]MySQL5.7 客户端一次发送给MySQL多条SQL,截取的sql文本不正确

mysql client

 select * from t_user limit 1;update t_user set name='ADFAFD' limit 1;

sniffer-agent的输出

{"sip":"10.10.20.176","sport":3306,"cpr":1,"bt":1681099755827,"cip":"10.10.20.177","cport":55697,"user":"root","db":"test_fly","sql":"update t_user set name='ADFA","cms":4}
{"sip":"10.10.20.176","sport":3306,"cpr":1,"bt":1681099755838,"cip":"10.10.20.177","cport":55697,"user":"root","db":"test_fly","sql":"update t_user set name='ADFAFD' limit 1","cms":11}

不能准确的抓取第一个SQL语句的内容

执行运行失败

[root@host-172-16-10-12 sniffer-agent-1.2.0]# ./sniffer-agent --interface=eth0 --port=3307 --export_type=cli
set config capture_packet_rate: 1
INFO[0000] parsed local ip address:172.16.10.12
set config capture_packet_rate: 1
panic: package size bigger than max buffer size need deal:0

goroutine 24 [running]:
github.com/zr-hebo/sniffer-agent/util.(*SliceBufferPool).DequeueWithInit(0xc0000a8fe0, 0x54, 0x0, 0x0, 0x0)
/usr/local/sniffer-agent-1.2.0/util/cache_pool.go:43 +0x165
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).readFromClient(0xc000174000, 0x64ab3a25, 0xc000258042, 0x58, 0xffbe)
/usr/local/sniffer-agent-1.2.0/session-dealer/mysql/session.go:182 +0xe6
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).ReceiveTCPPacket(0xc000174000, 0xc00016c060)
/usr/local/sniffer-agent-1.2.0/session-dealer/mysql/session.go:91 +0x90
github.com/zr-hebo/sniffer-agent/capture.readToServerPackage(0x0, 0x0, 0xc000172020, 0xd2e8, 0xc0001648c0, 0xc0000872c0, 0x0, 0x0)
/usr/local/sniffer-agent-1.2.0/capture/network.go:295 +0x19b
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).parseTCPPackage(0xc0000a9480, 0xb3ac00, 0xc0000ecc60, 0x0)
/usr/local/sniffer-agent-1.2.0/capture/network.go:212 +0x22c
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal.func1(0xc0000a9480)
/usr/local/sniffer-agent-1.2.0/capture/network.go:164 +0x41a
created by github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal
/usr/local/sniffer-agent-1.2.0/capture/network.go:107 +0x3f

docker容器运行mysql,无法抓包

启动2个docker容器,一个容器跑的centos并安装了mysql8.0,此外安装并执行sniffer agent以及tcpdump。
另外一个docker容器,允许mysql连接到第一个容器中去执行sql。
tcpdump可以抓到包,但是sniffer agent抓不到数据,什么也不显示。

功能咨询

您好,请问一下有计划在目前 MySQL 协议基础上,添加“返回行数”和“执行状态(成功/失败)”这两个功能

编译问题请教

请问下编译报这个错误如何处理:
[root@vm10-2-103-156 sniffer-agent]# go build -mod=readonly

github.com/google/gopacket/pcap

/root/go/pkg/mod/github.com/google/[email protected]/pcap/pcap_unix.go:34:18: fatal error: pcap.h: No such file or directory
#include <pcap.h>
^
compilation terminated.
[root@vm10-2-103-156 sniffer-agent]#

无法抓取mysql client 登录后执行的SQL信息

DEBU[0009] receive an unexpect packet
INFO[0009] expect receive size is bigger than max deal size: 131072
INFO[0009] expect receive size is bigger than max deal size: 131072
INFO[0009] expect receive size is bigger than max deal size: 131072

编译报错

centos 7.4
go 1.13.15
go build main.go 输出如上图 是因为什么
image

无任何输出,感觉没抓到

[root@test_5_186 sniffer-agent]# ./sniffer-agent --interface=ens160 --port=3306 --log_level=debug --capture_packet_rate=1000 --export_type=cli --max_packet_length=13107200
set config capture_packet_rate: 1
INFO[0000] parsed local ip address:10.6.5.186
INFO[0000] begin listen
set config capture_packet_rate: 1000
另一边模拟数据库登录,执行sql,但sniffer-agent后面无任何输出,

关于 windows版 mysql agent

作者您好,我关注这个项目已经很久了,感谢您的开源,看到您最近更新了 macOs版本的agent,想请问一下 什么时候有windows mysql版本呢,或者有这个计划在内吗。

运行报panic: package size bigger than max buffer size need deal:0错误

set config capture_packet_rate: 1
INFO[0000] parsed local ip address:10.128.0.105         
set config capture_packet_rate: 1

panic: package size bigger than max buffer size need deal:0

goroutine 23 [running]:
github.com/zr-hebo/sniffer-agent/util.(*SliceBufferPool).DequeueWithInit(0xc0000a4cc0, 0x8e, 0x0, 0x0, 0x0)
        /root/sniffer-agent/util/cache_pool.go:43 +0x190
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).readFromClient(0xc0001fc000, 0x5f2c57e1, 0xc0001e6042, 0x92, 0xffbe)
        /root/sniffer-agent/session-dealer/mysql/session.go:182 +0xdc
github.com/zr-hebo/sniffer-agent/session-dealer/mysql.(*MysqlSession).ReceiveTCPPacket(0xc0001fc000, 0xc00010bad0)
        /root/sniffer-agent/session-dealer/mysql/session.go:91 +0x8b
github.com/zr-hebo/sniffer-agent/capture.readToServerPackage(0x0, 0x0, 0xc000097e30, 0xabd4, 0xc0001f8000, 0xc0000b0d20, 0x0, 0x0)
        /root/sniffer-agent/capture/network.go:295 +0x19f
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).parseTCPPackage(0xc0000a5180, 0xba39a0, 0xc0000ef760, 0x0)
        /root/sniffer-agent/capture/network.go:212 +0x245
github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal.func1(0xc0000a5180)
        /root/sniffer-agent/capture/network.go:151 +0x601
created by github.com/zr-hebo/sniffer-agent/capture.(*networkCard).listenNormal
        /root/sniffer-agent/capture/network.go:107 +0x3f

麻烦帮忙看下什么原因

prepare语句中 抓绑定变量的值的问题

我通过命令:
./sniffer-agent --interface=eth0 --port=6033 --log_level=info --strict_mode=true --admin_user=user_microloan_proxyA --admin_passwd=xxx> res

抓出到的SQL日志为:

{"sip":"10.106.3.180","sport":6033,"cpr":1,"bt":1581173313742,"cip":"10.106.3.31","cport":6033,"user":null,"db":null,"sql":"INSERT INTO thirdparty_statistic_customer (id, user_account_id, order_id, mobile, api, api_md5, api_fee, call_count, success_call_count, hit_call_count, cutomer_total_cost, record_type, ctime, utime) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)","cms":2}

发现两个问题
1、开启了--strict_mode=true 但是抓取的db,user信息仍是NULL
2、程序是先prepare语句, 然后执行的语句的时候赋值,但是 sniffer-agent 生成的SQL 仍无法抓取到赋值。

go build失败

go version go1.12.17 linux/amd64:
cannot find module providing package github.com/zr-hebo/validator

进程退出,panic err

panic: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

goroutine 1 [running]:
github.com/zr-hebo/sniffer-agent/exporter.NewKafkaExporter(0xa9d750)
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/exporter/kafka.go:83 +0x329
github.com/zr-hebo/sniffer-agent/exporter.NewExporter(0xc00018feb8, 0x512d9b)
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/exporter/model.go:26 +0x8c
main.mainServer()
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/main.go:50 +0x34
main.main()
/export/project/golang/src/github.com/zr-hebo/sniffer-agent/main.go:46 +0x9a

MySQL8.0报错 receive an unexpect packet

DEBU[8871] receive an unexpect packet
DEBU[8871] receive an unexpect packet
DEBU[8871] receive an unexpect packet
DEBU[8871] receive an unexpect packet

请问这个报错是什么原因?

关于源码的一些疑惑

zr-hebo,你好,我找了很多的抓包工具,目前最好的就是你这个抓包工具了,感谢作者的开源

我的golang水平不太高。关于源码里有一些疑惑

就是BaseQueryPiece这个结构体的Recovery()明明没有做任何事情 是怎么产生出抓包功能的呢?

PooledMysqlQueryPiece的结构体带的Recovery()函数是有具体作用

但是我从main函数入口去看 ,只有一个 nc.receiver <- model.NewBaseQueryPiece(localIPAddr, nc.listenPort, capturePacketRate) 这个明明是返回的BaseQueryPiece 所以很疑惑

根据qps自动调节抓包率

如题,sniffer可以动态的调整抓包率,但是调整的依据尚不明确,sniffer是否可以提供一个实时qps查询的功能,之后用户根据实时的qps值调整抓包率

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.