failed to load unwind_stop about otel-profiling-agent

[inge4pres]

hey @guolifu 😄
This line

load program: no space left on device: 174:

seems to indicate that the BPF programs cannot be loaded due to limited resources.
Not sure exactly where the shortage is, pretty sure it's not on disk and the message is misleading.

I tested on three different kernels

Are you using VMs? If yes, how are they configured?

[athre0z]

That's weird. The bpf syscall is documented to return that status code under the following circumstances:

The eBPF program is too large or a map reached the
max_entries limit (maximum number of elements).

All kernel versions that you listed should have a limit of 1M instructions whereas we are trying to stay below 4K instructions (for compat with old kernels), so this is probably not it. At the same time, the map sizes that are being printed in your verbose output also don't look outrageously large.

What architecture is this? How many cores does the machine have? How did you build, make agent (uses Docker) or just plain make (uses host LLVM toolchain)?

[guolifu]

That's weird. The bpf syscall is documented to return that status code under the following circumstances:

The eBPF program is too large or a map reached the
max_entries limit (maximum number of elements).

All kernel versions that you listed should have a limit of 1M instructions whereas we are trying to stay below 4K instructions (for compat with old kernels), so this is probably not it. At the same time, the map sizes that are being printed in your verbose output also don't look outrageously large.

What architecture is this? How many cores does the machine have? How did you build, make agent (uses Docker) or just plain make (uses host LLVM toolchain)?

One of them is this

root@ubuntu2204:/opt/github/euspace# uname -a
Linux ubuntu2204 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

cpu cores : 2

root@ubuntu2204:/opt/github/euspace# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 7373
max locked memory           (kbytes, -l) 250420
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 7373
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

make agent (uses Docker)

1. make docker-image
2. make agent

[rockdaboot]

@guolifu I could reproduce the issue in a fresh Ubuntu22.04 VM.
The work-around is to drop the -bpf-log-level=2 option.
We have to investigate what exactly happens when the log level is set.

[guolifu]

@guolifu我可以在新的 Ubuntu22.04 VM 中重现该问题。 解决方法是删除该-bpf-log-level=2选项。 我们必须调查设置日志级别时到底会发生什么。

You are right. Setting -bpf-log-level to either 1 or 2 will cause problems, while 0 will not cause any problems.
It's strange.😂

[rockdaboot]

You have to set the -bpf-log-size values high enough (possibly depends on kernel version).

Works here on 6.7.9-amd64:
-bpf-log-level=1 -bpf-log-size=524288
and
-bpf-log-level=2 -bpf-log-size=8388608

[guolifu]

You have to set the -bpf-log-size values high enough (possibly depends on kernel version).

Works here on 6.7.9-amd64: -bpf-log-level=1 -bpf-log-size=524288 and -bpf-log-level=2 -bpf-log-size=8388608

Your help was very much appreciated,I'll give it a try👍