Comments (5)
Unfortunately the stack trace is not very useful, because all goroutines seem to be waiting for IO or in a system call. Based on the logs it looks like there might be an issue parsing the MySQL messages. Which version of MySQL do you have?
Also, if this happens quickly enough, can you please run packetbeat like this? It will add more debugging info:
packetbeat -e -c /etc/packetbeat/packetbeat.conf -d mysql,mysqldetailed
Thanks.
from beats.
It happens quickly enough, hope this helps gaining insights:
....
mysql.go:118: DBG extract mysql message
mysql.go:133: DBG Seq=1, type=0, start=0
mysql.go:154: DBG OK response
mysql.go:440: DBG fields: 0
mysql.go:448: DBG response raw: {}
mysql.go:455: DBG Mysql transaction completed: map[query:UPDATE agents SET heartbeat_timestamp='2014-05-09 08:11:44.224058' WHERE agents.id = '5621ac36-b411-4cae-bfeb-4ce8f8e4b27b
' tables: num_fields:%!s(int=0) error_code:%!s(int=0) isok:%!s(bool=true) iserror:%!s(bool=false) method:UPDATE affected_rows:%!s(int=1) num_rows:%!s(int=0) insert_id:%!s(int=0) si
ze:%!s(uint64=0) error_message:]
mysql.go:118: DBG extract mysql message
mysql.go:133: DBG Seq=1, type=0, start=0
mysql.go:154: DBG OK response
mysql.go:440: DBG fields: 0
mysql.go:448: DBG response raw: {}
mysql.go:455: DBG Mysql transaction completed: map[isok:%!s(bool=true) affected_rows:%!s(int=1) insert_id:%!s(int=0) num_rows:%!s(int=0) num_fields:%!s(int=0) error_message: table
s: query:UPDATE agents SET heartbeat_timestamp='2014-05-09 08:11:44.224058' WHERE agents.id = '5621ac36-b411-4cae-bfeb-4ce8f8e4b27b' method:UPDATE size:%!s(uint64=0) iserror:%!s(bo
ol=false) error_code:%!s(int=0)]
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
main.decodePktEth(0x71, 0xc2003ee460)
/home/vagrant/src/packetbeat/main.go:147 +0x711
main.main()
/home/vagrant/src/packetbeat/main.go:336 +0x1467
...
MySQL in this case means Percona 5.5.
from beats.
Hello,
I'm seeing this error as well, it randomly crashes, with mysql 5.0 on Centos 5.9 using the binary package.
Also I'm trying it in FreeBSD 9 and one time it crashed with the same error but parsing http (nginx proxy), although only happened once for now (in a server with 3000 reqs/minute):
http.go:495: WARN Response from unknown transaction. Ignoring.
http.go:434: WARN Two requests without a response. Dropping old request
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
runtime.panic(0x6774e0, 0xb8450a)
/usr/local/go/src/pkg/runtime/panic.c:266 +0xb6
main.httpMessageParser(0xc2102a4cc0, 0xc213886036)
/root/software/packetbeat-0.1.0/http.go:316 +0x1b3d
main.ParseHttp(0xc213952b80, 0xc2100fd700, 0xc213952b00)
/root/software/packetbeat-0.1.0/http.go:362 +0x2d1
main.(*TcpStream).AddPacket(0xc2100fd700, 0xc213952b80, 0xc21e00c296f10010)
/root/software/packetbeat-0.1.0/tcp.go:89 +0x10d
main.FollowTcp(0xc213886022, 0x14, 0x145c, 0xc213952b80)
/root/software/packetbeat-0.1.0/tcp.go:135 +0x22b
main.decodePktEth(0x1, 0xc210cc6f00)
/root/software/packetbeat-0.1.0/main.go:149 +0x6d8
main.main()
/root/software/packetbeat-0.1.0/main.go:336 +0x13ae
goroutine 3 [syscall]:
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1394
goroutine 8 [chan receive]:
main.(*PublisherType).UpdateTopologyPeriodically(0xb9c760)
/root/software/packetbeat-0.1.0/publish.go:178 +0x3e
created by main.(*PublisherType).Init
/root/software/packetbeat-0.1.0/publish.go:319 +0x58c
goroutine 6 [runnable]:
net/http.(*persistConn).readLoop(0xc210058a00)
/usr/local/go/src/pkg/net/http/transport.go:778 +0x68f
created by net/http.(*Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:528 +0x607
goroutine 7 [select]:
net/http.(*persistConn).writeLoop(0xc210058a00)
/usr/local/go/src/pkg/net/http/transport.go:791 +0x271
created by net/http.(*Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:529 +0x61e
goroutine 3928 [select]:
net/http.(*persistConn).writeLoop(0xc21294f480)
/usr/local/go/src/pkg/net/http/transport.go:791 +0x271
created by net/http.(*Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:529 +0x61e
goroutine 3927 [IO wait]:
net.runtime_pollWait(0x800bc99d8, 0x72, 0x0)
/usr/local/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc210944d10, 0x72, 0x800bc8190, 0x23)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc210944d10, 0x23, 0x800bc8190)
/usr/local/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).Read(0xc210944cb0, 0xc21304f000, 0x1000, 0x1000, 0x0, ...)
/usr/local/go/src/pkg/net/fd_unix.go:204 +0x2a0
net.(*conn).Read(0xc210942120, 0xc21304f000, 0x1000, 0x1000, 0x755dc0, ...)
/usr/local/go/src/pkg/net/net.go:122 +0xc5
bufio.(*Reader).fill(0xc211a9f840)
/usr/local/go/src/pkg/bufio/bufio.go:91 +0x110
bufio.(*Reader).Peek(0xc211a9f840, 0x1, 0x423f98, 0xc211a9f3b0, 0x42e689, ...)
/usr/local/go/src/pkg/bufio/bufio.go:119 +0xcb
net/http.(*persistConn).readLoop(0xc21294f480)
/usr/local/go/src/pkg/net/http/transport.go:687 +0xb7
created by net/http.(*Transport).dialConn
/usr/local/go/src/pkg/net/http/transport.go:528 +0x607
from beats.
We have just released v0.1.1 which should not crash anymore in these cases. Instead it will print a stacktrace to the logs and continue. Please continue to post us the stacktraces if you see them in the logs.
Btw, I'm glad to hear that it works in FreeBSD, apart of the above issues :).
from beats.
This fixes the stack trace from @badcrc. The one from @srenatus might still be there, but hopefully you'll get a better trace when running v0.1.1.
from beats.
Related Issues (20)
- [metricbeat/system][windows] - Metricbeat reports DEGRADED while running in privileged mode HOT 3
- [filebeat/input/redis/slowlog] Add field for the Redis replication role HOT 1
- [add_Kubernetes_metadata processor] Implement a new matcher for pattern matching
- x-pack/metricbeat: multiple unit tests failures on macOS arm64 HOT 1
- x-pack/heartbeat: "build constraints exclude all Go files" for Win tests HOT 2
- x-pack/filebeat: AWS test failure HOT 3
- Early event encoding obscures raw event content in the event log HOT 1
- Windows service for Beat does not stop when output is unreachable HOT 1
- Investigate the best way to decide when to read system logs from files or journald HOT 1
- [metricbeat/system module] should differentiate kernel and user-space threads HOT 6
- [metricbeat:elasticsearch module] Missing jvm.mem.pools metrics from node.stats HOT 1
- Filebeat should fail to start when multiple filestream inputs have the same input ID HOT 1
- [Elastic Agent] The system process metricsets should not report as degraded when metrics are partially collected HOT 7
- [Elastic Agent] Add feature flag to the system metricsets to control whether metrics failures mark agent as degraded HOT 1
- Filebeat gets stuck on one unhealthy Logstash
- Correction to Documentation: prospector.scanner.check_interval HOT 1
- [RFC] Standardization of azure-eventhub Input Metadata Field HOT 1
- Add synthetics to the new Wolfi image of Heartbeat HOT 4
- [Packetbeat] Add support for ignoring interfaces HOT 1
- [AWS] Enable OwningAccount for cross account monitoring HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from beats.