kinvolk / traceloop

Now moved into `github.com/inspektor-gadget/inspektor-gadget/pkg/gadget-collection/gadgets/traceloop`. Tracing system calls in cgroups using BPF and overwritable ring buffers

traceloop

traceloop is a command line tool to trace system calls in a similar way to strace but with some differences:

• traceloop uses BPF instead of ptrace
• traceloop's tracing granularity is the control group (cgroup) instead of a process
• traceloop's traces are recorded in a fast, in-memory, overwritable ring buffer like a flight recorder. The tracing could be permanently enabled and inspected in case of crash.

traceloop can be used directly on the command line or via an HTTP interface.

traceloop has been written to trace Kubernetes Pods with Inspektor Gadget, but it can as easily be used with systemd services that are in their own control groups (look for .service and .scope directories inside /sys/fs/cgroup/unified/).

On the command line

Example with an existing systemd service:

sudo -E ./traceloop cgroups /sys/fs/cgroup/unified/system.slice/sshd.service

Example with a custom command:

sudo systemd-run -t  --unit=test42.service  /bin/sh -c 'for i in $(seq 1 1000) ; do sleep 4 ; echo 2*3*7 | bc > /dev/null ; echo Multiplication $i done. ; done'
...
sudo -E ./traceloop cgroups /sys/fs/cgroup/unified/system.slice/test42.service
...
00:04.022260640 cpu#0 pid 23981 [bc] brk(brk=0) = 94045092683776
00:04.022346588 cpu#0 pid 23981 [bc] ioctl(fd=0, cmd=21505, arg=140721805741680) = 18446744073709551591
00:04.022361201 cpu#0 pid 23981 [bc] read(fd=0, buf=94045092586128 "2*3*7\n", count=8192) = 6
00:04.022401517 cpu#0 pid 23981 [bc] fstat() = 0
00:04.022414650 cpu#0 pid 23981 [bc] ioctl(fd=1, cmd=21505, arg=140721805741312) = 18446744073709551591
00:04.022440173 cpu#0 pid 23981 [bc] write(fd=1, buf=94045092602832 "42\n", count=3) = 3
00:04.022460351 cpu#0 pid 23981 [bc] read(fd=0, buf=94045092586128 "", count=8192) = 0
00:04.022475888 cpu#0 pid 23981 [bc] ioctl(fd=0, cmd=21505, arg=140721805741616) = 18446744073709551591
00:04.022525326 cpu#0 pid 23981 [bc] exit_group(error_code=0)...
00:04.022833827 cpu#2 pid 23961 [sh] ...wait4() = 23981

With Docker

docker run --rm -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/fs/bpf:/sys/fs/bpf -v /run:/run --privileged kinvolk/traceloop

With HTTP interface

sudo -E ./traceloop serve
...

$ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/add?name=sshd&cgrouppath=/sys/fs/cgroup/unified/system.slice/sshd.service'
added as id 0
$ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/list'
0: [sshd] /sys/fs/cgroup/unified/system.slice/sshd.service
$ sudo curl --unix-socket /run/traceloop.socket 'http://localhost/dump-by-cgroup?cgroup=/sys/fs/cgroup/unified/system.slice/sshd.service'
...

Talk at Linux Plumbers Conference 2020

A comprehensive presentation was held at LPC 2020 in the Networking and BPF Summit. See the slides here.

After feedback to include a comparison to perf trace we reran the benchmark but omitted the synchronous write syscall case that logs the buffer contents because dumping the buffers is not implemented in perf trace. Here the results:

traceloop's Issues

HTTP API does not return HTTP error codes

If registering fails or if dumping fails because no trace can be found, traceloop should return HTTP error codes so that clients can react on errors (e.g., curl -f).

Currently it's always a 200 OK response even if dumping fails because there is no trace (prog with name "MYNAME" not found).

Display errors not as unsigned big numbers but -1 and lookup error codes

Seen with running an unpriv. date -s '2000-01-01 00:00'

open(filename=5275524 "/etc/TZ", flags=0, mode=0) = 18446744073709551614
…
settimeofday(tv=140723222619760, tz=0) = 18446744073709551615

Keep compressed logs of retired tracers in a separate data structure

When the k8s subcommand notices that a container is deleted, it keeps the tracelet for one minute. Directly retire the tracelet without any timeouts after dumping it a final time and store the rendered string compressed in a second list (which has a maximal length, appending if full will push the oldest element out).
If the API functions for dump, close etc can't find an active tracelet, they fall back to the list of compressed dumps. The API function for list should include the list of compressed dumps.

Trace identifier can clash if same pod is run twice

Maybe use a timestamp or incremented suffix if a trace with the same name exists.

Registering a cgroup sometimes fails

I did not save the error message but it seems it's related to loading the tailcall program.

Don't print buffer when read() returns 0 on EOF

When the read system call returns zero because it reached the end of the file, the content of the buffer is not meaningful.

[cat] open("/proc/version", 0, 18446744073709424640) = 3
...
[cat] read(3, "Linux version 5.2.13-flatcar (jenkins@flatcar-jenkins-worker) (gcc version 8.3.0 (Gentoo Hardened 8.3.0-r1 p1.1)) #1 SMP Thu Sep", 131072) = 150
[cat] write(1, "Linux version 5.2.13-flatcar (jenkins@flatcar-jenkins-worker) (gcc version 8.3.0 (Gentoo Hardened 8.3.0-r1 p1.1)) #1 SMP Thu Sep", 150) = 150
[cat] read(3, "(Pointer deref failed!)", 131072) = 0

It should not matter that the buffer couldn't be read from the BPF program.

Instead, it should print an empty string like strace does:

$ strace cat /proc/version 2>&1 | grep 'read.*= 0$'
read(3, "", 131072)                     = 0

Add support for ARM

traceloop currently only works on x86_64.

Non-exhaustive list of tasks:

• The syscall table (syscall_table.go) needs to be per-architecture
• straceback-guess-bpf.c hard code some syscall numbers

if (exit != 0 && id != 165) { // mount

if (id != 269) { // faccessat()

• errno computation is x86_64 specific (if errNo >= -4095 && errNo <= -1 in event.go)

Upstream all changes for gobpf

There needs to be a PR to https://github.com/iovisor/gobpf/ for
kinvolk/gobpf@a01317a (can be found in https://github.com/kinvolk/gobpf/tree/alban/straceback)

Add mntns filter support

Current situation
[ Please describe the current situation you would like to have improved ]
Currently bcc-tools supported filter by mntns and cgroup. so could we add mntns filter here to support the old kernel that
haven't cgroup-v2.

Impact
[ Please describe the impact the lack of the feature requested is creating ]

Ideal future situation
[ Please describe the future situation after the improvement was implemented ]

Implementation options
[ Optional: please provide one or more options for implementing the feature requested ]

Additional information
[ Please Add any information that does not fit into any of the above sections here ]

fix Go 11 modules

The project should build when cloned outside of the GOPATH. (This worked before the rename.)

read()/write() buffers sometimes not printed correctly

Following the instructions from demo-traceloop.md, I sometimes have this problem:

• The buffer of the read() syscall ("3*7*2\n") is printed on a separate line.
• The buffer of the write() syscall is not printed at all, even though it returns 3.

00:00.481572346 cpu#3 pid 388925 [bc] read("", 5330944, 4096)...
00:00.481963718 "3*7*2\n"
00:00.482122689 cpu#5 pid 388924 [sh] ...write() = 6
00:00.482136409 cpu#5 pid 388924 [sh] exit_group(0)...
00:00.482504750 cpu#3 pid 388925 [bc] ...read() = 6
00:00.482628831 cpu#3 pid 388925 [bc] write(1, "", 3) = 3
00:00.482914162 cpu#3 pid 388925 [bc] read("", 5330944, 4096)...
00:00.482996992 cpu#3 pid 388925 [bc] exit_group(0)...

Option to enable/disable log to stdout

Currently traceloop logs all traces to stdout every second. Having a general log collection system means a lot of unnecessary traffic. Would it be possible to add an option to either disable or enable logs via a flag?

Populate the buffer of the read syscall at __sys_exit

Use bpf_probe_read and bpf_perf_event_output in __sys_exit which needs a map to from thread ID to the (last) arg struct. See https://github.com/weaveworks/tcptracer-bpf/blob/afee59868bf5a60c40489a7cb41da40374f29aea/tcptracer-bpf.c#L48-L58

trace creation date sometimes missing

Symptoms:

$ kubectl gadget traceloop list
PODNAME           PODUID      INDEX    TRACEID             CONTAINERID    STATUS
multiplication    41f6677e    0        0000055fa8cf3342    85983624       deleted 9 minutes ago
multiplication    679b1343    0        000005d0d786e53e    ba580597       deleted About a minute ago
nginx             bd7983a4    0        0000052a3d3493ec    27a353fc       created a while ago (parsing time "" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "2006")

The json for this nginx trace without creation date is:

   {
      "capabilities" : 136316359,
      "containerid" : "cri-o://27a353fc1074066ba64c8e7cdbf948b362b3aabfd45625a80158bced241b3087",
      "namespace" : "default",
      "node" : "crc-rk2fc-master-0",
      "podname" : "nginx",
      "status" : "ready",
      "traceid" : "0000052a3d3493ec",
      "uid" : "bd7983a4-95b5-11ea-8585-0242ac11003a"
   },

Some other traces have the timecreation field.

Live tracing

Implement a snooping mode which attaches its own ring buffer (as done in bcc tools). Need to take care of the parameter reassembly which might have to be redone when new events come in.

• in live mode: update window contents by rendering last N bytes (configurable by parameter)
• allow to dump events in binary format to capture everything and have a Inspektor Gadget subcommand to convert them to text.

Wrong syscalls reported for i386 binaries

When a 32 bit binary is run via

podman run --rm -ti i386/alpine

and, for example, just the Enter key is hit a few times, I get wrong syscalls reported (expected is poll and read(0):

sudo -E ./traceloop cgroups /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/user.slice/libpod-66540563be22be03c226f6f0c5d0f1d4f92eda982412c61c4da1218397f9ded7.scope
00:00.519821885 cpu#5 pid 776610 [sh] swapoff(4287264624) = 1
00:00.531230324 cpu#5 pid 776610 [sh] close(0) = 1
00:00.531236722 cpu#5 pid 776610 [sh] swapoff(4287264624) = 1
00:00.531240220 cpu#5 pid 776610 [sh] close(0) = 1
00:00.531243272 cpu#5 pid 776610 [sh] swapoff(4287264624) = 1
00:00.531246027 cpu#5 pid 776610 [sh] close(0) = 1
00:00.531249071 cpu#5 pid 776610 [sh] swapoff(4287264624) = 1
00:00.531251834 cpu#5 pid 776610 [sh] close(0) = 1
00:00.531254858 cpu#5 pid 776610 [sh] swapoff(4287264624) = 1
00:00.531257615 cpu#5 pid 776610 [sh] close(0) = 1

because i386's read is interpreted as amd64's close and poll is interpreted as swapoff since they have the same syscall number:

grep close /usr/include/asm/unistd_64.h 
#define __NR_close 3
grep ' 3$' /usr/include/asm/unistd_32.h
#define __NR_read 3
grep swapoff /usr/include/asm/unistd_64.h 
#define __NR_swapoff 168
grep 168 /usr/include/asm/unistd_32.h
#define __NR_poll 168

traceloop fails to publish traces on cri-o

traceloop expects cgroup paths of Kubernetes pods to follow this format:

traceloop/pkg/straceback/straceback.go

Lines 244 to 248 in 9bc843e

	// Two examples of params. Both are supported:
	// /sys/fs/cgroup/systemd/kubepods/besteffort/pod91a8fc3a-0ecf-48b4-81bf-78a7275d348c/f75aff467357c5d0ddd47cb7ad87ed38746e018992586ff66198a5c11218f634
	// /sys/fs/cgroup/systemd/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5759a1ae_36ca_48a8_b20a_b0b5c8a90fb8.slice/docker-c8b38413c88eefe063b8cd3f01c16be5e3bda9693a19a68a88807baca9feb937.scope
	paramRegexp1, _ := regexp.Compile("^/sys/fs/cgroup/systemd.*/kubepods.*[/-]pod([a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}).*/([0-9a-f]{64})")
	paramRegexp2, _ := regexp.Compile("^/sys/fs/cgroup/systemd.*/kubepods.*[/-]pod([a-f0-9]{8}_[a-f0-9]{4}_[a-f0-9]{4}_[a-f0-9]{4}_[a-f0-9]{12}).*/docker-([0-9a-f]{64}).scope")

This does not work with cri-o, where the following format is used:

1:name=systemd:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12345678_1234_1234_1234_123456789012.slice/crio-f75aff467357c5d0ddd47cb7ad87ed38746e018992586ff66198a5c11218f634.scope

Also, traceloop has the assumption that podStatus has container ids with the format docker://. However, with cri-o, it could be in this format:

cri-o://f75aff467357c5d0ddd47cb7ad87ed38746e018992586ff66198a5c11218f634

syscall fstat() parameters are not displayed

fstat has the following prototype:

       int fstat(int fd, struct stat *statbuf);

However, straceback currently does not show the parameters:

fstat() = 0

This is because we don't have a entry for /sys/kernel/debug/tracing/events/syscalls/sys_enter_fstat/, unlike other syscalls. We should investigate why.

Print all syscalls with the same details as strace

Currently we have some syscalls annotated to dereference buffer pointers. This could be automated with a script having a heuristic for the data in /sys/kernel/debug/tracing/events/syscalls/sys_enter_NAME/format where const and char * are useful hints. We will still need to fix some corner cases like dereferencing a read buffer at exit.
Besides dereferencing buffer pointers we have to display constants and structs in a nice way.
It would be good to see if we can use BTF and also if we can share code with strace.

fatal error: concurrent map iteration and map write

On image: docker.io/kinvolk/gadget:v0.3.1

{"level":"info","msg":"BPF Tracer ready","time":"2021-12-10T14:04:22Z"}
{"level":"info","msg":"Starting proc informer","time":"2021-12-10T14:04:22Z"}
{"level":"info","msg":"Starting annotation publisher","time":"2021-12-10T14:04:22Z"}
{"level":"info","msg":"Starting updater loop","time":"2021-12-10T14:04:22Z"}
fatal error: concurrent map iteration and map write

goroutine 280 [running]:
runtime.throw(0x1608852, 0x26)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/panic.go:774 +0x72 fp=0xc012a6cf60 sp=0xc012a6cf30 pc=0x431282
runtime.mapiternext(0xc012a6d078)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/map.go:858 +0x579 fp=0xc012a6cfe8 sp=0xc012a6cf60 pc=0x4126f9
github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).GetPodFromContainerID(0xc0000cf540, 0xc0129f8d80, 0x40, 0x1, 0x1, 0x0)
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:148 +0xae fp=0xc012a6d0e8 sp=0xc012a6cfe8 pc=0x12e15ee
github.com/kinvolk/traceloop/pkg/straceback.(*StraceBack).updater(0xc0001165a0, 0x0, 0x0)
        /home/runner/work/traceloop/traceloop/pkg/straceback/straceback.go:402 +0x158d fp=0xc012a6dfc8 sp=0xc012a6d0e8 pc=0x130ee8d
runtime.goexit()
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc012a6dfd0 sp=0xc012a6dfc8 pc=0x45e631
created by github.com/kinvolk/traceloop/pkg/straceback.NewTracer
        /home/runner/work/traceloop/traceloop/pkg/straceback/straceback.go:246 +0x114b

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f184c498e38, 0x72, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc01298da98, 0x72, 0x0, 0x0, 0x15e6a28)
        /opt/hostedtoolcache/go/1.13.15/x64/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /opt/hostedtoolcache/go/1.13.15/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc01298da80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc01298da80, 0xc00064e380, 0x7f1850709d98, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/fd_unix.go:238 +0x42
net.(*UnixListener).accept(0xc01297d890, 0xc000029b18, 0x40fc68, 0x30)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/unixsock_posix.go:162 +0x32
net.(*UnixListener).Accept(0xc01297d890, 0x1505c40, 0xc01297d980, 0x142d9c0, 0x2212180)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/unixsock.go:260 +0x47
net/http.(*Server).Serve(0xc00036c0e0, 0x1821f60, 0xc01297d890, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/http/server.go:2925 +0x280
main.main()
        /home/runner/work/traceloop/traceloop/traceloop.go:273 +0x1389

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        /opt/hostedtoolcache/go/1.13.15/x64/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /opt/hostedtoolcache/go/1.13.15/x64/src/os/signal/signal_unix.go:29 +0x41

goroutine 7 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x22926e0)
        /home/runner/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1169 +0x8b
created by k8s.io/klog/v2.init.0
        /home/runner/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:417 +0xdd

goroutine 8 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x2292600)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
        /home/runner/go/pkg/mod/k8s.io/[email protected]/klog.go:411 +0xd6

goroutine 11 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000248060)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:198 +0xe0
created by k8s.io/client-go/util/workqueue.newQueue
        /home/runner/go/pkg/mod/k8s.io/[email protected]/util/workqueue/queue.go:58 +0x132

goroutine 13 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000248180)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x405
created by k8s.io/client-go/util/workqueue.newDelayingQueue
        /home/runner/go/pkg/mod/k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x184

goroutine 14 [chan receive]:
github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).Run(0xc0000cf540, 0x1, 0xc00004a480)
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:296 +0x2b6
created by github.com/kinvolk/traceloop/pkg/podinformer.NewPodInformer
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:128 +0x6a7

goroutine 23 [sync.Cond.Wait]:
runtime.goparkunlock(...)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0002900c8, 0x3)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0002900b8)
        /opt/hostedtoolcache/go/1.13.15/x64/src/sync/cond.go:56 +0x9d
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0002900a0, 0xc000120f30, 0x0, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/delta_fifo.go:493 +0xaa
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000116120)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:183 +0x40
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00025bfb0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00025bfb0, 0x17ef3c0, 0xc00024c000, 0xc0001ac001, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00025bfb0, 0x3b9aca00, 0x0, 0xc000392c01, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0xaa
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc000116120, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:154 +0x2f6
created by github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).Run
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:284 +0x157

goroutine 323 [syscall]:
github.com/iovisor/gobpf/elf._C2func_poll(0xc00c1cfe00, 0x8, 0x1f4, 0x0, 0x0, 0x0)
        _cgo_gotypes.go:352 +0x55
github.com/iovisor/gobpf/elf.perfEventPoll(0xc000044f60, 0x8, 0x8, 0xc0003380f0, 0xc00007fe40)
        /home/runner/go/pkg/mod/github.com/kinvolk/[email protected]/elf/perf.go:425 +0xe6
github.com/iovisor/gobpf/elf.(*PerfMap).PollStart.func1(0xc0003380f0, 0xc00003da80, 0xc0003948c0)
        /home/runner/go/pkg/mod/github.com/kinvolk/[email protected]/elf/perf.go:329 +0x792
created by github.com/iovisor/gobpf/elf.(*PerfMap).PollStart
        /home/runner/go/pkg/mod/github.com/kinvolk/[email protected]/elf/perf.go:312 +0xbf

goroutine 50 [chan receive]:
k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc00004a480, 0xc000116120)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:130 +0x34
created by k8s.io/client-go/tools/cache.(*controller).Run
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:129 +0x9f

goroutine 51 [select]:
k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc0001d0000, 0xc064f577dead9950, 0x32468e0, 0x22921a0, 0x17fab20, 0xc00095fec0, 0xc000c3bb98, 0xc00026fe00, 0xc00004a480, 0x0, ...)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:463 +0x1ab
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0001d0000, 0xc00004a480, 0x0, 0x0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:427 +0x669
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:221 +0x38
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00033f6d0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000c3bed0, 0x17ef3a0, 0xc0006a00a0, 0x1, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xa3
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0001d0000, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:220 +0x1b5
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001ac010, 0xc0002ae020)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x59
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x62

goroutine 16 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func2(0xc0001d0000, 0xc00004a480, 0xc000976300, 0xc00026fe00)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:373 +0x17e
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
        /home/runner/go/pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:367 +0x2bf

goroutine 279 [chan send]:
github.com/kinvolk/traceloop/pkg/procinformer.(*ProcInformer).update(0xc0002af3c0, 0xc000343dd0, 0x2)
        /home/runner/work/traceloop/traceloop/pkg/procinformer/procinformer.go:131 +0x6dc
github.com/kinvolk/traceloop/pkg/procinformer.NewProcInformer.func1(0xc0002af3c0)
        /home/runner/work/traceloop/traceloop/pkg/procinformer/procinformer.go:50 +0x128
created by github.com/kinvolk/traceloop/pkg/procinformer.NewProcInformer
        /home/runner/work/traceloop/traceloop/pkg/procinformer/procinformer.go:41 +0xd7

goroutine 38 [IO wait]:
internal/poll.runtime_pollWait(0x7f184c498f08, 0x72, 0xffffffffffffffff)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000254398, 0x72, 0xb600, 0xb63a, 0xffffffffffffffff)
        /opt/hostedtoolcache/go/1.13.15/x64/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /opt/hostedtoolcache/go/1.13.15/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000254380, 0xc00017e000, 0xb63a, 0xb63a, 0x0, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000254380, 0xc00017e000, 0xb63a, 0xb63a, 0x203000, 0x7f184c441ba8, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000010a90, 0xc00017e000, 0xb63a, 0xb63a, 0x0, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc00091c000, 0xc00017e000, 0xb63a, 0xb63a, 0x1f, 0x17ee400, 0xc00010f920)
        /opt/hostedtoolcache/go/1.13.15/x64/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc000071758, 0x17ee260, 0xc00091c000, 0x40d3d5, 0x1459560, 0x1587420)
        /opt/hostedtoolcache/go/1.13.15/x64/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc000071500, 0x17ef840, 0xc000010a90, 0x5, 0xc000010a90, 0x9)
        /opt/hostedtoolcache/go/1.13.15/x64/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000071500, 0x0, 0x0, 0x442a9f)
        /opt/hostedtoolcache/go/1.13.15/x64/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
        /opt/hostedtoolcache/go/1.13.15/x64/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc000071500, 0xc000374000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc00027c720, 0xc0001d0118, 0x9, 0x9, 0xc00010fcc0, 0x0, 0x90d7b5)
        /opt/hostedtoolcache/go/1.13.15/x64/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x17ee0e0, 0xc00027c720, 0xc0001d0118, 0x9, 0x9, 0x9, 0xc000082060, 0x0, 0x17ee400)
        /opt/hostedtoolcache/go/1.13.15/x64/src/io/io.go:310 +0x87
io.ReadFull(...)
        /opt/hostedtoolcache/go/1.13.15/x64/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001d0118, 0x9, 0x9, 0x17ee0e0, 0xc00027c720, 0x0, 0x0, 0xc0002c4918, 0x101)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001d00e0, 0xc00c8e8030, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00010ffb8, 0x0, 0x0)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1819 +0xbe
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00066af00)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:1741 +0xa3
created by golang.org/x/net/http2.(*Transport).newClientConn
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:705 +0x68b

goroutine 67 [sync.Cond.Wait]:
runtime.goparkunlock(...)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0002510c0, 0x2)
        /opt/hostedtoolcache/go/1.13.15/x64/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0002510b0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/sync/cond.go:56 +0x9d
golang.org/x/net/http2.(*pipe).Read(0xc0002510a8, 0xc003d84001, 0x1dff, 0x1dff, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/pipe.go:65 +0xa6
golang.org/x/net/http2.transportResponseBody.Read(0xc000251080, 0xc003d84001, 0x1dff, 0x1dff, 0x0, 0x0, 0x0)
        /home/runner/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2108 +0xac
encoding/json.(*Decoder).refill(0xc000251340, 0xc00027000a, 0x9)
        /opt/hostedtoolcache/go/1.13.15/x64/src/encoding/json/stream.go:161 +0xeb
encoding/json.(*Decoder).readValue(0xc000251340, 0x0, 0x0, 0x1430c40)
        /opt/hostedtoolcache/go/1.13.15/x64/src/encoding/json/stream.go:136 +0x1dc
encoding/json.(*Decoder).Decode(0xc000251340, 0x144d480, 0xc0002700e0, 0x0, 0x0)
        /opt/hostedtoolcache/go/1.13.15/x64/src/encoding/json/stream.go:63 +0x79
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00098ac90, 0xc003d88000, 0x2000, 0x2600, 0x17f8320, 0x0, 0x38)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/framer/framer.go:152 +0x1a1
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0000cfe50, 0x0, 0x17fa760, 0xc0009bf600, 0x0, 0x0, 0x0, 0xc000976478, 0x4584d0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/streaming/streaming.go:77 +0x89
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc000990ce0, 0xc00010df58, 0x8, 0x17f8320, 0xc012af8000, 0x0, 0x0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:49 +0x7c
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00095fec0)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:104 +0x17e
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:71 +0xbe

goroutine 68 [chan send]:
github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).syncToStdout(0xc0000cf540, 0xc012a88b20, 0x18, 0x13e2200, 0xc0001503d0)
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:238 +0x633
github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).processNextItem(0xc0000cf540, 0x0)
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:185 +0xf4
github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).runWorker(0xc0000cf540)
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:301 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000688130)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000688130, 0x17ef3c0, 0xc0003ac000, 0x1, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000688130, 0x3b9aca00, 0x0, 0x1, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0xaa
k8s.io/apimachinery/pkg/util/wait.Until(0xc000688130, 0x3b9aca00, 0xc00004a480)
        /home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/kinvolk/traceloop/pkg/podinformer.(*PodInformer).Run
        /home/runner/work/traceloop/traceloop/pkg/podinformer/podinformer.go:293 +0x22d

Restart traces

When a trace was dumped, it got stopped. Allow to restart tracing (has to be done before closing).

Build failure on getting github.com/fatih/hclfmt

Description

With the current master branch, a simple build by running make fails like that:

Step 5/6 : RUN go get -u github.com/fatih/hclfmt
 ---> Running in faddb8dc120a
package github.com/hashicorp/hcl/hcl/printer: cannot find package "github.com/hashicorp/hcl/hcl/printer" in any of:
        /usr/lib/golang/src/github.com/hashicorp/hcl/hcl/printer (from $GOROOT)
        /go/src/github.com/hashicorp/hcl/hcl/printer (from $GOPATH)
The command '/bin/sh -c go get -u github.com/fatih/hclfmt' returned a non-zero code: 1
make: *** [Makefile:34: build-docker-image] Error 1

That's why the CI for #58 failed.

error reporting with no such file or directory when running traceloop with docker

Just encountered an error when running traceloop with docker:

[root@k8s-master ~]# docker run --rm -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/fs/bpf:/sys/fs/bpf -v /run:/run --privileged kinvolk/traceloop
/usr/bin/docker-current: Error response from daemon: mkdir /sys/fs/bpf: no such file or directory.

Below is my environment:

[root@k8s-master ~]# uname
Linux
[root@k8s-master ~]# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
[root@k8s-master ~]# 
[root@k8s-master ~]# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.centos.1.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Tue Jan 30 09:17:00 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.centos.1.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Tue Jan 30 09:17:00 2018
 OS/Arch:         linux/amd64
[root@k8s-master ~]# 

Use Go 11 Modules and document setup

kernel trace with "General protection fault in user access. Non-canonical address?"

On newer kernels, traceloop sometimes triggers a kernel stack to be printed in dmesg:

kernel: General protection fault in user access. Non-canonical address?

kernel: ------------[ cut here ]------------
kernel: General protection fault in user access. Non-canonical address?
kernel: WARNING: CPU: 1 PID: 4564 at ../../../../../../../../usr/src/linux-5.4.16-coreos/arch/x86/mm/extable.c:126 ex_handler_uaccess+0x4d/0x60
kernel: Modules linked in: veth xt_statistic xt_nat ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_nat ip6_tables iptable_mangle xt_comment xt_mark xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_u>
kernel: CPU: 1 PID: 4564 Comm: traceloop Not tainted 5.4.16-flatcar #1
kernel: Hardware name: Amazon EC2 t3.small/, BIOS 1.0 10/16/2017
kernel: RIP: 0010:ex_handler_uaccess+0x4d/0x60
kernel: Code: 83 c4 08 b8 01 00 00 00 5b c3 80 3d f4 d1 48 01 00 75 dc 48 c7 c7 a0 72 15 90 48 89 34 24 c6 05 e0 d1 48 01 01 e8 6c 11 01 00 <0f> 0b 48 8b 34 24 eb bd 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44
kernel: RSP: 0018:ffffa13f80bfb9e8 EFLAGS: 00010282
kernel: RAX: 0000000000000000 RBX: ffffffff8fa0272c RCX: 0000000000000000
kernel: RDX: 000000000000003f RSI: ffffffff9344475f RDI: 0000000000000246
kernel: RBP: 000000000000000d R08: ffffffff93444720 R09: 000000000000003f
kernel: R10: 0000000000000000 R11: 00000000000011d4 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
kernel: FS:  00007f394bb2c700(0000) GS:ffff93867b500000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 00000000731f6003 CR4: 00000000007606e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: PKRU: 55555554
kernel: Call Trace:
kernel:  fixup_exception+0x43/0x56
kernel:  do_general_protection+0x49/0x150
kernel:  general_protection+0x28/0x30
kernel: RIP: 0010:copy_user_generic_unrolled+0x86/0xc0
kernel: Code: 4c 8b 5e 38 4c 89 47 20 4c 89 4f 28 4c 89 57 30 4c 89 5f 38 48 8d 76 40 48 8d 7f 40 ff c9 75 b6 89 d1 83 e2 07 c1 e9 03 74 12 <4c> 8b 06 4c 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10
kernel: RSP: 0018:ffffa13f80bfbaf8 EFLAGS: 00050202
kernel: RAX: 0000000000000002 RBX: ffff9386169c0000 RCX: 0000000000000001
kernel: RDX: 0000000000000000 RSI: 0100000000000000 RDI: ffffa13f80bfbc90
kernel: RBP: 0000000000000008 R08: ffffffff8f18840a R09: ffff93867572b000
kernel: R10: ffff93867572b630 R11: ffff93867adad000 R12: 00007ffffffff000
kernel: R13: 0100000000000000 R14: ffffa13f80bfbc90 R15: 0000000000000001
kernel:  ? ___bpf_prog_run+0x94a/0x13b0
kernel:  __probe_kernel_read+0x54/0x80
kernel:  bpf_probe_read+0x2d/0x60
kernel:  ___bpf_prog_run+0xa01/0x13b0
kernel:  __bpf_prog_run512+0x3e/0x60
kernel:  ? ___bpf_prog_run+0x94a/0x13b0
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? legitimize_path.isra.44+0x2d/0x60
kernel:  ? unlazy_walk+0x42/0x70
kernel:  ? terminate_walk+0x7a/0xe0
kernel:  ? path_lookupat.isra.50+0xa3/0x220
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? alloc_htab_elem+0x180/0x280
kernel:  trace_call_bpf+0x82/0x100
kernel:  ? htab_map_update_elem+0x25b/0x450
kernel:  perf_trace_run_bpf_submit+0x4e/0xc0
kernel:  perf_trace_sys_enter+0xf7/0x160
kernel:  syscall_trace_enter+0x2a9/0x2c0
kernel:  do_syscall_64+0xdb/0x120
kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: RIP: 0033:0x4b07f0
kernel: Code: 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 49 c7 c2 00 00 00 00 49 c7 c0 00 00 00 00 49 c7 c1 00 00 00 00 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
kernel: RSP: 002b:000000c001445690 EFLAGS: 00000206 ORIG_RAX: 000000000000010d
kernel: RAX: ffffffffffffffda RBX: 000000c00004a000 RCX: 00000000004b07f0
kernel: RDX: 0000000000000000 RSI: 000000c001a9647a RDI: ffffffffffffff9c
kernel: RBP: 000000c0014456e8 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
kernel: R13: 0000000000000088 R14: 0000000000000087 R15: 0000000000000200
kernel: ---[ end trace a69fe79250905b65 ]---

For more information:

• torvalds/linux@00c4237 (warning added in Linux 5.1)
• torvalds/linux@6ae08ae (BPF helpers bpf_probe_read_{user, kernel}() added in Linux 5.5)

Traceloop should not use bpf_probe_read() but should use the new bpf_probe_read_{user, kernel}() when possible, depending on the Linux version (see kernel-versions.md). I'm hoping it will help, but I have not tried it.

Gobpf currently does not allow to load different implementation of a BPF program depending on the kernel version. However, @shang-wang wrote DataDog/gobpf#1 which could help with the function SetKprobeForSection(). It could be upstreamed ;) There are some examples how to use that function here.