victorialogs panic about victoriametrics

[kk-kwok]

Also, try using VictoriaLogs v0.18.0. It is built with Go1.22.4, which may contain the fix for this issue. See the list of fixes in Go1.22.4

Yesterday, it was updated to v0.18.0, and panic often occurred.

[valyala]

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at golang/go#66383 , so they could fix it quickly.

victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

[valyala]

@kk-kwok , thanks for filing the bug report! It looks similar to golang/go#66383 and golang/go#67255 .

@kk-kwok , if you run VictoriaLogs on arm64, then it would be great if you could help Go core team narrowing down the issue at golang/go#66383

[valyala]

Also, try using VictoriaLogs v0.18.0. It is built with Go1.22.4, which may contain the fix for this issue. See the list of fixes in Go1.22.4

[kk-kwok]

@kk-kwok , thanks for filing the bug report! It looks similar to golang/go#66383 and golang/go#67255 .

@kk-kwok , if you run VictoriaLogs on arm64, then it would be great if you could help Go core team narrowing down the issue at golang/go#66383

Sorry, I don't have an arm64 environment

[valyala]

Yesterday, it was updated to v0.18.0, and panic often occurred.

Does the stack trace for the panic look the same every time? Does it contain SIGSEGV: segmentation violation and runtime.(*mspan).typePointersOfUnchecked every time?

If not, then it would be great to see other examples of the panic.

Also, could you build VictoriaLogs from the v0.18.0-victorialogs release tag with the make victoria-logs-race command? This command builds VictoriaLogs executable with the built-in data race detector and put the built binary in bin directory. The binary name should be bin/victoria-logs-race. Then run this binary and see whether it logs additional errors (data race errors) before and during the panic.

[kk-kwok]

Yesterday, it was updated to v0.18.0, and panic often occurred.

Does the stack trace for the panic look the same every time? Does it contain SIGSEGV: segmentation violation and runtime.(*mspan).typePointersOfUnchecked every time?

If not, then it would be great to see other examples of the panic.

Also, could you build VictoriaLogs from the v0.18.0-victorialogs release tag with the make victoria-logs-race command? This command builds VictoriaLogs executable with the built-in data race detector and put the built binary in bin directory. The binary name should be bin/victoria-logs-race. Then run this binary and see whether it logs additional errors (data race errors) before and during the panic.

Yes, the stack trace of Panic is the same every time.

OK, I'll try.

[valyala]

@kk-kwok , are there any updates regarding the build with Go1.21? Feel free asking additional assistance if it is needed from our side.

[kk-kwok]

@kk-kwok , are there any updates regarding the build with Go1.21? Feel free asking additional assistance if it is needed from our side.

I'm sorry that I only saw this reply now. I'm now trying to use the image produced by go1.21.11.
Thank you

[kk-kwok]

@valyala Another question, a colleague reported that the json size of several log records exceeds 25KB, but it is not found in victorylogs. What is the limit on the size of a single log?

[kk-kwok]

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at golang/go#66383 , so they could fix it quickly.

victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

Now that the new image compiled with go1.21.11 is used to run, I will observe whether panic occurs in the next day. Previous versions including v0.19.0 will panic after running for a few hours.

[kk-kwok]

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at golang/go#66383 , so they could fix it quickly.
victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

Now that the new image compiled with go1.21.11 is used to run, I will observe whether panic occurs in the next day. Previous versions including v0.19.0 will panic after running for a few hours.

@valyala LGTM!
I have been running the image compiled by go1.21.11 for 25 hours now, and it has been running normally without panic.