brendangregg / perf-tools Goto Github PK

I want to use the perf_event_open() to monitor the events (such as mem_load_l3_miss_retired.remote_pmm、mem_load_retired.local_pmm). The doucument of linux\perf\design.txt says as follow. But how to get the event_id of a specific event (such as mem_load_l3_miss_retired.remote_pmm、mem_load_retired.local_pmm)?

The 'config' field specifies what the counter should count. It is divided into 3 bit-fields:
raw_type: 1 bit (most significant bit) 0x8000_0000_0000_0000
type: 7 bits (next most significant) 0x7f00_0000_0000_0000
event_id: 56 bits (least significant) 0x00ff_ffff_ffff_ffff
If 'raw_type' is 1, then the counter will count a hardware event specified by the remaining 63 bits of event_config. The encoding is machine-specific.

Thanks, Looking forward to your reply.

I'm using kali 1.0.9a and trying to use opensnoop. I get an error statring "events/syscalls/sys_exit_open/enable". Wondering which of the prerequisites I'm missing.

chandranshu@chandranshu-laptop:~$ ./iosnoop 
Tracing block I/O. Ctrl-C to end.
./iosnoop: line 122: cd: /sys/kernel/debug/tracing: Permission denied
ERROR: accessing tracing. Root user? Kernel has FTRACE?
chandranshu@chandranshu-laptop:~$ sudo ./iosnoop 
Tracing block I/O. Ctrl-C to end.
ERROR: ftrace may be in use by PID 7690 /var/tmp/.ftrace-lock

Looking at the code, this problem is present at all the places where die is being used. We can improve the die function to clean up the lock and any other resources before dying.

A reminder that for 4.x kernels, syscount should switch to using either eBPF (Alexei Starovoitov) or hist triggers (Tom Zanussi), either of which can do these aggregations in-kernel.

After collecting perf trace using perf record -e 'raw_syscalls:sys_enter' --call-graph dwarf,4096, I use perf script to look at the collect data. In most part, it is as expected. However, one issue is that my application binary has symbols stripped. I noticed that whenever perf script cannot resolve the symbol of an item on the stack, it stops printing the stack. I don't see an option that says keep printing stack trace even if you cannot resolve a symbol. Did I miss something or is this by design?

Hi, all. I am having problem with memory on one server, and I am don't understanding how is fixed ?
It's file /proc/meminfo
MemTotal: 24592560 kB
MemFree: 532324 kB
Buffers: 236832 kB
Cached: 459252 kB
SwapCached: 60448 kB
Active: 7312736 kB
Inactive: 1607264 kB
Active(anon): 6851356 kB
Inactive(anon): 1372752 kB
Active(file): 461380 kB
Inactive(file): 234512 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2097148 kB
SwapFree: 1424980 kB
Dirty: 136 kB
Writeback: 24 kB
AnonPages: 8199460 kB
Mapped: 69492 kB
Shmem: 192 kB
Slab: 14876100 kB
SReclaimable: 14810428 kB
SUnreclaim: 65672 kB
KernelStack: 23496 kB
PageTables: 38716 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14393428 kB
Committed_AS: 8999040 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 213232 kB
VmallocChunk: 34359392960 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2250752 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10240 kB
DirectMap2M: 25155584 kB

I seeing what slab used 14 GB memory, it's not normal.
I have second node, but them slab's having size 1.6Gb.
Maybe something now, why can slab have big size ?

Why does opensnoop traces the open syscall and not fs/do_sys_open?
That covers both open and openat for you.

Hi, I'm confused what's the differnece between sched_process_free and sched_process_exit. And when I want to listen to a process to ensure that it is fully "terminated" and removed from memory, which is the best choice?

When blocks are merged it is possible that the resulting latency will be very high simply because the reply that will be taken will be from much later from a later IO.

See the following snippet:

  nb_truck_1-5402  [002]  8018.233160: block_rq_insert: 65,192 WS 0 () 184067232 + 16 [nb_truck_1]
  nb_truck_1-5402  [002]  8018.233161: block_rq_issue: 65,192 WS 0 () 184067232 + 16 [nb_truck_1]
  nb_truck_3-5404  [004]  8108.333833: block_rq_complete: 65,192 R () 184067232 + 16 [0]

The IO took 90 seconds to return which is completely unlikely, the IO that returned is a read whereas the original IO was a write.

Hi Brendan,

We encountered the below errors while trying to generate the flame graphs from the host.
After successful perf record we invoked perf script command on perf.data
perf script | ./stackcollapse-perf.pl > out.perf-folded got the below errors. How could we overcome these errors. I believe few are related perfdata file not available in host since it resides in container. This is a multi container env.

Failed to open /tmp/perf-12964.map, continuing without symbols
Failed to open /tmp/perf-20444.map, continuing without symbols
Failed to open /lib/x86_64-linux-gnu/libpthread-2.23.so, continuing without symbols
Failed to open /x/web/LIVE/keymakeragent/keymakeragent/infra/lib/linux_x86_py27/_faststat.so, continuing without symbols
Failed to open /x/opt/pp/bin/python2.7, continuing without symbols
Failed to open /x/web/LIVE/keymakeragent/keymakeragent/infra/lib/linux_x86_py27/greenlet.so, continuing without symbols
Failed to open /applicationpackages/manifests/active/JDK/cronus/scripts/jdk1.8.0_60/jre/lib/amd64/server/libjvm.so, continuing without symbols
Failed to open /tmp/perf-29726.map, continuing without symbols
Failed to open /applicationpackages/manifests/active/JDK/cronus/scripts/jdk1.8.0_60/jre/lib/amd64/libnet.so, continuing without symbols
Failed to open /lib/x86_64-linux-gnu/libc-2.23.so, continuing without symbols
Failed to open /tmp/perf-5808.map, continuing without symbols
Failed to open /tmp/perf-28806.map, continuing without symbols
Failed to open /lib/ld-musl-x86_64.so.1, continuing without symbols
Failed to open /tmp/perf-25990.map, continuing without symbols
Failed to open /lib/libpthread-2.5.so, continuing without symbols
Failed to open /lib/libc-2.5.so, continuing without symbols
Failed to open /tmp/perf-14795.map, continuing without symbols
Failed to open /applicationpackages/manifests/active/JSW/cronus/scripts/JSW/bin/wrapper, continuing without symbols
Failed to open /tmp/perf-22375.map, continuing without symbols
Failed to open /tmp/perf-14695.map, continuing without symbols
Failed to open /usr/bin/ppregistrator, continuing without symbols
Failed to open /x/web/LIVE/keymakeragent/keymakeragent/infra/lib/linux_x86_py27/gevent/core.so, continuing without symbols
Failed to open /tmp/perf-14639.map, continuing without symbols
Failed to open /tmp/perf-23526.map, continuing without symbols
Failed to open /tmp/perf-8628.map, continuing without symbols
Failed to open /applicationpackages/manifests/active/JDK/cronus/scripts/jdk1.8.0_60/jre/lib/amd64/libjava.so, continuing without symbols
Failed to open /usr/lib/libstdc++.so.6.0.8, continuing without symbols
Failed to open /x/web/LIVE/caldaemon/caldaemon, continuing without symbols
Failed to open /tmp/perf-10390.map, continuing without symbols
Failed to open /lib/librt-2.5.so, continuing without symbols
Failed to open /tmp/perf-3844.map, continuing without symbols
no symbols found in /bin/dash, maybe install a debug package?
Failed to open /applicationpackages/manifests/active/JDK/cronus/scripts/jdk1.8.0_60/jre/lib/amd64/libnio.so, continuing without symbols
Failed to open /x/opt/pp/lib/python2.7/lib-dynload/select.so, continuing without symbols
Failed to open /x/opt/pp/lib/python2.7/lib-dynload/time.so, continuing without symbols
Failed to open /applicationpackages/manifests/active/JDK/cronus/scripts/jdk1.8.0_60/jre/lib/amd64/libmanagement.so, continuing without symbols
Failed to open /usr/bin/socat, continuing without symbols
Failed to open /x/opt/pp/lib/python2.7/lib-dynload/_socket.so, continuing without symbols

touch loop succeeds until syscount is run.

$ echo $$
32922
$ while  touch /home/hydra/minion; do sleep .1; done
-bash: fork: Operation not permitted

./syscount -c  -p 32922

2.6.32-504.el6.x86_64

I grabbed the isnoop script but unfortunately when I run it I end up with:

/usr/local/bin/iosnoop: line 231: warning: here-document at line 62 delimited by end-of-file (wanted `END')
/usr/local/bin/iosnoop: line 232: syntax error: unexpected end of file

I'm slightly confused by why this is popping up, as the HEREDOC opened on line 62 is closed on line 74 with the END, or at least it should be.

Hi team,

When we try to collect statistics pertaining to caches using perf we got the below message stating not supported. I could do a perf top successfully but not a statistics with event. Could you please help us.
We use ubuntu distro 4.4.0-97-generic #120~14.04.1-Ubuntu SMP

perf stat -e cycles,instructions,cache-references,cache-misses,bus-cycles -a sleep 10

Performance counter stats for 'system wide':

cycles
instructions
cache-references
cache-misses
bus-cycles

  10.000910410 seconds time elapsed

Thanks

When not using a probe name, and naming args, kprobe parses the probe incorrectly. Eg:

# ./kprobe 'p:udp_send_skb si=%si'
Tracing kprobe udp_send_skb. Ctrl-C to end.
./kprobe: line 179: echo: write error: No such file or directory
ERROR: adding kprobe "p:udp_send_skb si=%si".
Last 2 dmesg entries (might contain reason):
    [1680629.084807] Event kprobes/udp_send_skb doesn't exist.
    [1685530.939679] Could not insert probe at si=%si+0: -2
Exiting.

Workaround:

# ./kprobe 'p:myprobe udp_send_skb si=%si'
Tracing kprobe myprobe. Ctrl-C to end.
        postgres-16392 [000] 2844852.131332: myprobe: (udp_send_skb+0x0/0x2a0) si=ffff8800e6f349e0
        postgres-16392 [000] 2844852.131349: myprobe: (udp_send_skb+0x0/0x2a0) si=ffff8800e6f349e0
        postgres-16392 [000] 2844852.131355: myprobe: (udp_send_skb+0x0/0x2a0) si=ffff8800e6f349e0
        postgres-16392 [000] 2844852.131362: myprobe: (udp_send_skb+0x0/0x2a0) si=ffff8800e6f349e0
[...]

But this should be fixed.

@brendangregg when i perf record or perf top -p pid
Error:
The sys_perf_event_open() syscall returned with 3 (No such process) for event (cycles:ppp).
/bin/dmesg may provide additional information.
No CONFIG_PERF_EVENTS=y kernel support configured?

but Not for all of the process

jibanes@wopr:~$ sudo ./opensnoop -d 1 omfg
Tracing open()s for filenames containing "omfg" for 1 seconds (buffered)...
COMM PID FD FILE
mawk: line 19: syntax error at or near ,

Ending tracing...

Using opensnoop from Ubuntu 16.04 on a WINE installation.

sudo opensnoop 2>&1 | grep "^wine"
(...)
wineserver       6666   0x72 /home/me/wineprefix/dosdevices/c:/
wineserver       6666   0x72 Files/
wineserver       6666   0x72 Files/Notepad++/
(...)

The last two lines are wrong. The files are in

/home/me/wineprefix/dosdevices/c\:/Program\ Files/Notepad++/
# a.k.a.
"/home/me/wineprefix/dosdevices/c:/Program Files/Notepad++/"

Seemingly opensnoop does not handle paths with blanks correctly?

To reproduce:

sudo dpkg --add-architecture i386 
wget https://dl.winehq.org/wine-builds/Release.key
sudo apt-key add Release.key
sudo apt-add-repository 'https://dl.winehq.org/wine-builds/ubuntu/'
sudo apt-get update
sudo apt-get -y install winehq-stable 

mkdir -p wineprefix
export WINEPREFIX=$(readlink -f wineprefix)
export WINEARCH=win32

wget  https://raw.githubusercontent.com/Winetricks/winetricks/master/src/winetricks
chmod +x winetricks 

bash winetricks npp

sudo apt -y install perf-tools-unstable
sudo opensnoop

Not able to get execsnoop to work on OS X even after disabling SIP.

$ sw_vers
ProductName:	Mac OS X
ProductVersion: 10.15.7

$ csrutil status
System Integrity Protection status: disabled.

bash-3.2# ./execsnoop
Tracing exec()s. Ctrl-C to end.
./execsnoop: line 160: cd: /sys/kernel/debug/tracing: No such file or directory
ERROR: accessing tracing. Root user? Kernel has FTRACE?
    debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)

Is there a workaround or config that enables tracing on Mac?

Thanks!

I found running execsnoop has quite different results with do_execve() and stub_execve().

There is an example in Fedora with Kernel 3.11 and stub_execve(),:
'bash -x' results: http://paste.ubuntu.com/8431228/
'cat /sys/kernel/debug/tracing/trace_pipe' http://paste.ubuntu.com/8431209/.

And Fedora with 3.16 and do_execve():
'bash -x' http://paste.ubuntu.com/8431251/
'cat /sys/kernel/debug/tracing/trace_pipe' http://paste.ubuntu.com/8431253/

We could find a process with name starts with 'neutron-openvsw...' has execsnoop_stub_execve output from 'trace_pipe' but another not. In both of tests, that process has 'sched_process_fork' output. I don't know the develop history of stub_execve and do_execve and have to guess that are there any function call format changes between those, different limitations, or something else...

Hi Brendan Gregg:

Currently, functrace and funcgraph only supports single function.
There might be cases when we need to monitor multiple function in single tracing.

I had to manually edit and multiple function using.

function1 >current_tracer
function2>> current_tracer

Hi,
I'd like to invoke killsnoop from inside a docker container. I've tried adding:

  --privileged \
  --cap-add=ALL \

to its launch commands, but I still see:

usr/sbin/killsnoop.bt -p 3519776 
ERROR: tracepoint not found: syscalls:sys_enter_kill

I can run it from outside the container, if it launched with --pid=host that can make sense, however, thats just one of my 3 usecases where I need to trap things.

This command works when I run with sudo, but If I don't have permissions I get

posix4e@posix4e-P27GV2:~$ perf stat -e 'ext4:' -a
invalid or unsupported event: 'ext4:'
Run 'perf list' for a list of valid events

usage: perf stat [] []

-e, --event <event>   event selector. use 'perf list' to list available events

Anyway to make this give a sign to a user that they should enable permissions

busybox and other cat tools do not have -v, added a check if the option is present

--- bin/execsnoop
+++ bin/execsnoop
@@ -58,6 +58,7 @@ tracing=/sys/kernel/debug/tracing
 flock=/var/tmp/.ftrace-lock; wroteflock=0
 opt_duration=0; duration=; opt_name=0; name=; opt_time=0; opt_reexec=0
 opt_argc=0; argc=8; max_argc=16; ftext=
+VOPT=-v
 trap ':' INT QUIT TERM PIPE HUP        # sends execution to end tracing section

 function usage {
@@ -156,6 +157,9 @@ else
        fi
 fi

+# not all cat have -v, e.g busybox
+echo test | cat -v 2> /dev/null | grep -q test || VOPT=
+
 ### check permissions
 cd $tracing || die "ERROR: accessing tracing. Root user? Kernel has FTRACE?
     debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)"
@@ -224,10 +228,10 @@ warn "echo > trace"
 ( if (( opt_duration )); then
        # wait then dump buffer
        sleep $duration
-       cat -v trace
+       cat $VOPT trace
 else
        # print buffer live
-       cat -v trace_pipe
+       cat $VOPT trace_pipe
 fi ) | $awk -v o=$offset -v opt_name=$opt_name -v name=$name \
     -v opt_duration=$opt_duration -v opt_time=$opt_time -v kname=$kname \
     -v opt_reexec=$opt_reexec '

Hi,
when i am using "perf record -F 99 -g -p -- sleep 300" during heavy load, it is crashing. Only when i reduce the frequency to around 30, i am getting results.Ideally, the frequency should be high to get better results(around 997 samples/sec). Is there any solution for this.

It's not compatible with ash because of using some bash-specific syntax.

How do we get the device 202,1 ? I cant figure it out . can you share few examples.

By modifying the cachestat, i was able to get the caches miss rate of L1. I would like to use that result in my scheduler program where the tasks will be allocated based on L1 cache miss rate.How can that be done?How can i use the results from cachestat on my scheduler on LITMUS-RT testbed

Hi!

I understand that these tools are not really meant for amateur use, but I figured I'd mention this if only because you may run into it yourself at some point.

I'm trying to figure out how/why/where/etc a process is resolving a hostname and this is what I tried:

uprobe -d 1 p:libc:gethostbyname2

That works if I hack uprobe's set_path to take the first file; the little awk script returns 3 lines which of course doesn't work. To be clear:

ERROR: resolved "libc" to "/lib/x86_64-linux-gnu/libc-2.21.so
/lib32/libc-2.21.so
/libx32/libc-2.21.so", but file missing

Additionally, and this may be me not understanding something, the check for a library's existence is with -x, and on my system (ubuntu 15.10) almost none of the .so's in /lib/x86_64-linux-gnu are actually executable.

Hi,
i dont think this is normal:

# sh cachestat -D 5
Counting cache functions... Output every 5 seconds.
    HITS   MISSES  DIRTIES    RATIO   BUFFERS_MB   CACHE_MB  DEBUG
   -3075        0     5748   100.0%          299     119830  (2673 5748 2023 5741)
   -3798        0     7365   100.0%          299     119841  (3567 7365 3069 7383)
   -2333        0     5245   100.0%          299     119850  (2912 5245 1929 5240)
   -2469        0     4877   100.0%          299     119858  (2408 4877 2068 4832)
   -1782        0     4761   100.0%          299     119867  (2979 4761 2506 4742)
   -1981        0     4393   100.0%          299     119876  (2412 4393 2194 4379)
    2811        0     3719   100.0%          299     119883  (6530 3719 1832 3709)
       7        0     2145   100.0%          299     119886  (2152 2145 873 2140)
   53008        0    10504   100.0%          299     119902  (63512 10504 4412 10420)
   -5710        0    15480   100.0%          299     119902  (9770 15480 7456 15471)

This is on RHEL.

does it wrap around or something ?

cachestat: line 7: syntax error near unexpected token `newline'

I was lately trying to run execsnoop on my workstation and seeing nothing being printed out. Confused, I read the script and tried catting all sorts of values in /sys/kernel/debug/tracing. To my surprise, tracing_on contained 0. Doing echo 1 > tracing_on thus fixed the problem.

I see that execsnoop doesn't deal with tracing_on, so maybe it would be good to just put a echo 1 > tracing_on in there?

I find it strange that I only encounter this now, as a cursory Google search tells me that tracing_on has been present in Linux kernels since forever.

First, thank you for his tool.
I'm profiling an application that spends 20% of its time in system calls. With the option -c I have the list with the topmost frequent syscalls. It would be also helpful to have them sorted by time spent within them.

As always with the kernel-colored glasses, the -p switch in some tools uses set_ftrace_pid, which is the kernel pid notion -- user thread id. So tracing a process with multiple threads using the -p switch only traces the first thread. I've run into this with funcslower but I assume it affects additional tools as well.

I can submit a PR that fixes this by parsing the thread ids from /proc/PID/task, or by parsing ps output. Thoughts?

I would like to package tagged, numbered releases of perf-tools for Arch; we have a -git package, but that requires continual manual updating by the end user. Would you be willing to git tag releases so that we can grab github.com generated tarballs from the releases/ page? I'm sure this would assist other distributions in packaging "stable" as well...

Using iosnoop output BYTES！Where does the bytes size of the output come from? Can this be adjusted?

@brendangregg hi，I admire you for a long time,but now i have a question that When I am with 'perf record' ,it is over before the sleep time was reached,and did not show any information. the flame graph shows: Unrecognized line: Was the 'perf record' command properly terminated?The at FlameGraph/stackcollapse - perf. Pl line 339, < > the line 2.
Can you give me some advice?

On a 64-bit OS, with 32-bit compatibility, 32-bit processes are not traced. Only sys_execve systemcalls are traced, not the compat_sys_execve systemcalls.

After collecting perf data using perf record, I use perf script -i perf.data -F +insn to get the collected data. But there is nothing in the insn field.

        perf 32224 15889777.194652:          1 cycles:ppp:  ffffffff8826e448 native_write_msr ([kernel.kallsyms]) insn:
        perf 32224 15889777.194654:          1 cycles:ppp:  ffffffff8826e448 native_write_msr ([kernel.kallsyms]) insn:
        perf 32224 15889777.194655:         11 cycles:ppp:  ffffffff8826e448 native_write_msr ([kernel.kallsyms]) insn:
        perf 32224 15889777.194657:        285 cycles:ppp:  ffffffff8826e44a native_write_msr ([kernel.kallsyms]) insn:
        perf 32224 15889777.194658:       7550 cycles:ppp:  ffffffff88237a58 native_sched_clock ([kernel.kallsyms]) insn:
          sh 32224 15889777.194660:     191221 cycles:ppp:  ffffffff886219bd apparmor_bprm_committing_creds ([kernel.kallsyms]) insn:
          sh 32224 15889777.194707:    2700450 cycles:ppp:      7fc66bd036d7 [unknown] (/lib/x86_64-linux-gnu/ld-2.27.so) insn:
          sh 32225 15889777.195051:          1 cycles:ppp:  ffffffff8826e448 native_write_msr ([kernel.kallsyms]) insn:
          sh 32225 15889777.195054:          1 cycles:ppp:  ffffffff8826e448 native_write_msr ([kernel.kallsyms]) insn:
          sh 32225 15889777.195055:         13 cycles:ppp:  ffffffff8826e448 native_write_msr ([kernel.kallsyms]) insn:
          sh 32225 15889777.195056:        302 cycles:ppp:  ffffffff8826e44a native_write_msr ([kernel.kallsyms]) insn:
          sh 32225 15889777.195057:       7713 cycles:ppp:  ffffffff88237a58 native_sched_clock ([kernel.kallsyms]) insn:
          sh 32225 15889777.195060:     192158 cycles:ppp:  ffffffff88418cdb __handle_mm_fault ([kernel.kallsyms]) insn:
          sh 32225 15889777.195119:    2580729 cycles:ppp:  ffffffff88ba9d97 clear_page_erms ([kernel.kallsyms]) insn:

Is it a bug or just because I use it in wrong way?

# ./opensnoop -d 2
Tracing open()s for 2 seconds (buffered)...
COMM             PID      FD FILE
<...>            22794   0x3 
<...>            22799   0x3 P[���������7
<...>            22799   0x3 P[�����t�kڠ�
<...>            22798   0x3 ��%��������7
<...>            22798   0x3 ��%����ƜO;��
<...>            22798   0x3 ��%����ӅO;��
<...>            22798   0x3 ��%����t�O;��
<...>            22799   0x3 P[�����p���7
<...>            22798   0x3 ��%����p���7
<...>            22798   0x3 ��%��������7
���.>            22798   0x3 ��%����@y
<...>            22798    -1 ��%����
<...>            22798    -1 ��%�����@��
<...>            22798    -1 ��%�����*��
<...>            22798    -1 ��%�����+��
<...>            22798    -1 ��%����0A��
<...>            22798    -1 ��%�����+��
cat              22800   0x3 ���������7
cat              22800   0x3 �����tdT\h�
cat              22800   0x3 �����p���7

Ending tracing...

Tried with centos6 kernels 2.6.32-504.el6.x86_64 and 2.6.32-431.17.1.el6.x86_64 but of course I can't really comment on what's in them.

hi

This implementation is designed to work on older kernel versions, and without kernel debuginfo.

would be nice if it would also work on more recent kernel versions. I have tried changing the makeprobe call use __x64_sys_execve, but then half the output is gibberish. Unsure on how I have to adjust the output format of the probe.

uprobe fails to work when a library used by a binary is present in different locations. In my case, it is libc which was present at 3 different locations. The fix is to choose one of them and proceed, I am pasting the line here.

function set_path {
145 name=$1
146
147 path=$(which $name)
148 if [[ "$path" == "" ]]; then
149 path=$(ldconfig -v 2>/dev/null | awk -v lib=$name '
150 $1 ~ /:/ { sub(/:/, "", $1); path = $1 }
151 { sub(/..*/, "", $1); }
152 $1 == lib { print path "/" $3 }')
153 if [[ "$path" == "" ]]; then
154 die "ERROR: segment "$name" ambiguous."
155 "Program or library? Try a full path."
156 fi
157 fi
158 path=$(echo $path | cut -f2 -d" ")
159 if [[ ! -x $path ]]; then
160 die "ERROR: resolved "$name" to "$path", but file missing"
161 fi
162 }

When the kernel informs us that we lost events "CPU:%d [LOST %lu EVENTS]\n", we cannot assume that a filename we saved from a do_sys_open line corresponds to the rval we get from the following sys_open line. With the existing logic, we do, and under certain conditions it's possible to get a spurious "valid" open of a nonexistent file in the opensnoop output.

(I'll have a PR for this shortly.)

when I run:

perf script | ./stackcollapse-perf.pl > out.perf-folded`

Failed to open /tmp/perf-21607.map, continuing without symbols
Failed to open /tmp/perf-12967.map, continuing without symbols
Failed to open /tmp/perf-28472.map, continuing without symbols
Failed to open /tmp/perf-24981.map, continuing without symbols
Failed to open /tmp/perf-28048.map, continuing without symbols
Failed to open /tmp/perf-7905.map, continuing without symbols
Failed to open /tmp/perf-28432.map, continuing without symbols
Failed to open /tmp/perf-28669.map, continuing without symbols
Failed to open /tmp/perf-8011.map, continuing without symbols

Setup

- Debian 10 
- Perf version 4.19.152

Thanks

I have a question about the perf. When I use the perf to record my code.I find three chose lbr dwarf and pf,what is different for this.

I am trying to run cachestat with ./cachestat 5 but it fails with following output

Counting cache functions... Output every 5 seconds.
./cachestat: line 115: function_profile_enabled: Permission denied
ERROR: enabling function profiling. Have CONFIG_FUNCTION_PROFILER? Exiting.

The kernel version is

Linux version 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03)

And there isn't any file named function_profile_enabled in /sys/kernel/debug/tracing directory.

Is there any solutions for this problem?
Thanks a lot!