break hangs program; dlv unresponsive about delve

[derekparker]

If you're building from source off of master, please build with Go 1.4. There was a commit I added and several other commits that went into Go 1.4 stdlib that fixed some bugs. Please try rebuilding with Go 1.4 and see if the issue still persists.

[derekparker]

Any update on this? Have you tried after compiling Delve with Go 1.4?

[ddietterich]

I just spun up a system, removed 1.3.3 and installed 1.4.
Compiled my code - that worked. But failed trying to go get dlv:

go get github.com/derekparker/delve/cmd/dlv
# github.com/derekparker/delve/proctl
In file included from ../src/github.com/derekparker/delve/proctl/breakpoints_linux_amd64.go:5:
/usr/include/sys/user.h:32: error: expected specifier-qualifier-list before '__uint16_t'

Any clues for me?

[derekparker]

Hmm.. what kernel version / distro are you using?

[ddietterich]

I am using CentOS. uname -a says:

Linux ip-10-100-22-74 2.6.32-504.1.3.el6.x86_64 #1 SMP Tue Nov 11 17:57:25 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[derekparker]

Alright, it seems there are issues with <sys/user.h> on certain versions of glibc or something. I think I have a fix, I'll push it up soon.

[derekparker]

Would you mind pulling from the fix-sys-user-issue branch and trying to build and see if it fixes your problem? I'm relying on some C libs for setting hardware breakpoints, and I'd rather not, I may be able to remove those dependencies.

[derekparker]

@ddietterich any update on either the original issue or the hardware breakpoint issue?

[ddietterich]

Apologies for not getting to this sooner. I pulled and checked out fix-sys-user-issue.
I cd to cmd/dlv and did go install .
That seemed to produce a new dlv in $GOPATH/bin successfully

I ran dlv on my program. Set a breakpoint. Hit the rest API and dlv stopped at the breakpoint. Yay!
Then I typed "n", and the program hung. I had to kill -9.

So it seems like a good fix for the build problem, but something is still amiss.

[pnasrat]

I think I can reproduce this I'll try to get some more data.

[pnasrat]

So I'm getting different behavior depending on if I use delve to attach to a running pid versus using delve to lauch the program.

Base environment (I used GCE to set up a centos-6 instance)

$ uname -a 
Linux delve02 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ go version
go version go1.4.1 linux/amd64
$ go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/pnasrat/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Delve at 19c62d0 manually added

#include <bits/types.h>

Setup:

go get github.com/golang/example/outyet

Case 1 Delve launching process

dlv $GOPATH/bin/outyet
(dlv) break main.(*Server).ServeHTTP
Breakpoint 1 set at 0x401300 for main.(*Server).ServeHTTP /home/pnasrat/go/src/github.com/golang/example/outyet/main.go:109
c
new thread spawned 20542

In second terminal

curl http://localhost:8080/
curl: (7) couldn't connect to host

 lsof -p 20532
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
outyet  20532 pnasrat  cwd    DIR    8,1     4096 276213 /home/pnasrat/go/src/github.com/derekparker/delve
outyet  20532 pnasrat  rtd    DIR    8,1     4096      2 /
outyet  20532 pnasrat  txt    REG    8,1  7333600 154623 /home/pnasrat/go/bin/outyet
outyet  20532 pnasrat  mem    REG    8,1  1921176 529409 /lib64/libc-2.12.so
outyet  20532 pnasrat  mem    REG    8,1   142640 529482 /lib64/libpthread-2.12.so
outyet  20532 pnasrat  mem    REG    8,1   154624 529397 /lib64/ld-2.12.so
outyet  20532 pnasrat    0r   CHR    1,3      0t0   3829 /dev/null
outyet  20532 pnasrat    1u   CHR  136,0      0t0      3 /dev/pts/0
outyet  20532 pnasrat    2u   CHR  136,0      0t0      3 /dev/pts/0

cat /proc/20532/status 
Name:   outyet
State:  T (tracing stop)
Tgid:   20532
Pid:    20532
PPid:   20530
TracerPid:  20530

[pnasrat]

Case 2 - pid debugging

outyet &
 dlv --pid 22081
Type 'help' for list of commands.
(dlv) break main.(*Server).ServeHTTP
Breakpoint 1 set at 0x401300 for main.(*Server).ServeHTTP /home/pnasrat/go/src/github.com/golang/example/outyet/main.go:109
c

Here curl works - returns and doesn't hit the breakpoint at all.

 cat /proc/22081/status
Name:   outyet
State:  S (sleeping)
Tgid:   22081
Pid:    22081
PPid:   26676

[pnasrat]

Adding some printf debugging I can see us go into the loop below (but not exit it)

https://github.com/derekparker/delve/blob/master/proctl/proctl.go#L537

$ cat /proc/23656/status
Name:   outyet
State:  S (sleeping)
Tgid:   23653
Pid:    23656
PPid:   23649
TracerPid:  23649
Uid:    500 500 500 500
Gid:    500 500 500 500
Utrace: 1f0f
FDSize: 64

[pnasrat]

Further fmt.Printf("xx %v %v\n", m.procid, msg) in the loop shows we're getting

xx 0 24701
xx 0 24701
xx 0 24701
xx 0 24701
...

[derekparker]

Hmm.. Yeah that loop is a suboptimal solution I think - but it solved the problem of odd behavior on VMs. I think the ideal solution would be the hardware assisted memory write trap that I detailed in the comments. Otherwise it's hard to have a guarantee that procid ever gets set correctly. But if you're stuck in that loop it must mean that procid is never getting set, which is very strange.

Can you reproduce this on another OS, or is this just consistently happening on CentOS for some reason?

[pnasrat]

I'll try some other OS and cloud/vm provider combos, but that may not be for a few days. I'll file another issue about the pid connection path not seeming to actually set the breakpoint and try fix that separately.

[pnasrat]

Tried kill -6 the program under test to get thread state and then killing off delve

SIGABRT: abort
PC=0x43e96d

goroutine 0 [idle]:
runtime.usleep(0x2710, 0x0, 0xaf2f41fe01a, 0x2, 0x0, 0xdf8475800, 0xaf03b4bee13, 0x45d964b800, 0x1bf08eb000, 0x0, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:82 +0x2d
sysmon()
    /usr/local/go/src/runtime/proc.c:2892 +0xfb
mstart()
    /usr/local/go/src/runtime/proc.c:859 +0xe1
runtime.mstart()
    /usr/local/go/src/runtime/proc.c:836 +0xa4

goroutine 1 [stack growth, locked to thread]:
runtime.init·2()
    /usr/local/go/src/runtime/mem.go:64
runtime.init()
    /usr/local/go/src/runtime/zversion.go:7 +0x657
runtime.main()
    /usr/local/go/src/runtime/proc.go:46 +0x94
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

rax     0xfffffffffffffffc
rbx     0xaf2f41fe01a
rcx     0xffffffffffffffff
rdx     0x0
rdi     0x0
rsi     0x0
rbp     0x1
rsp     0x7ffff7e40da0
r8      0x7ffff7e40da0
r9      0x7ffff7e41700
r10     0x0
r11     0x246
r12     0x7fffffffe440
r13     0x7ffff7e419c0
r14     0x0
r15     0x3
rip     0x43e96d
rflags  0x246
cs      0x33
fs      0x0
gs      0x0

[derekparker]

@pnasrat @ddietterich any updates on this? Have you been able to track down a minimal reproducible case?

[ddietterich]

Hmm... @pnasrat seems to have reproduced the problem. I haven't paid it attention since.
Have you been able to reproduce it with that (perhaps not minimal) technique?

[derekparker]

I'll try it out on an Ubuntu VM see if I can reproduce this, if not try and spin up a CentOS VM and see what happens.

[derekparker]

So I was able to reproduce at least the 1st case on an Ubuntu machine - I'll start looking into a fix for this now.

[derekparker]

I've tracked down what is likely causing this and I believe it to be triggered by programs using cgo. It seems that if a go program uses glibc clone instead of runtime.clone, m->procid is never set, causing us to hang forever in the loop waiting for that to be set.

This affects #67 as well.

[ddietterich]

I did not do anything to use cgo. I simply go build, go test, go install.

[derekparker]

The net/http package, as far as I know, will trigger a cgo build due to linking against C networking libraries.

[derekparker]

Closing in favor of #70.