iovisor / ply

Dynamic Tracing in Linux

PLUMgrid iovisor source code

• net/ - kernel module

• compat/ - compatiblity code for older kernels

• bld/ - .ko build

Distributed bridge demo using BPF programs

• test_l2ls.sh - demonstrates L2 learning switch functionality defining a Topology with two Linux namespaces connected by a switch.

• test_l2ls_tun.sh - demonstrates distributed L2 learning switch functionality defining a Topology with two switches connected through Linux bridge and kernel vxlan/gre drivers. note: requires kernel 3.13+

• l2ls.c - C code for BPF bridge

• l2ls_bpf.h - compiled BPF code

• tunnel_port.c - C code for BPF tunnel

• tunnel_port_bpf.h - compiled BPF code

ply's Issues

Something's wrong even using --with-kerneldir

Hi,

First of all, thank you for your work.
I have a problem, maybe it's something stupid, but the first example is still failing for me:

$ ./configure --with-kerneldir=/usr/src/linux-4.10_p3-pf/
$ make

$ ./src/ply -v
ply-v1-beta1(9e810b1) (linux-version:264704~4.10.0)

$ uname -a
Linux erthalion 4.10.0-pf3 #2 SMP Sat Jul 15 20:07:25 CEST 2017 x86_64 Intel(R) Core(TM) i5-4210U CPU @ 1.70GHz GenuineIntel GNU/Linux

$ ./src/ply -d -c 'kprobe:SyS_*{ @[func()].count(); }'
dbg memlock_uncap       : unlimited memlock
dbg annotate_script     : ok
dbg map_setup           : @: type:1 ksize:0x8 vsize:0x8 nelem:0x400
dbg compile_probe       : kprobe:SyS_*
ERR probe_load          : kprobe:SyS_* : Invalid argument
ERR probe_load          : no output from kernel verifier
ERR probe_load          : was ply built against the running kernel?

$ ./src/ply -D -d -c 'kprobe:SyS_*{ @[func()].count(); }'
dbg memlock_uncap       : unlimited memlock
symtable:
@(kprobe:SyS_*)                          (type:map/int size:0x8 loc:stack/-0x8)
dbg annotate_script     : ok
ast:
`-> <script> (type:script/none size:0x0 loc:nowhere)
    `-> kprobe:SyS_* (type:probe/none size:0x0 loc:nowhere)
        `-> <method> (type:method/none size:0x0 loc:nowhere)
            `-> @ (type:map/int size:0x8 loc:stack/-0x8)
                `-> <rec> (type:rec/rec size:0x8 loc:stack/-0x10)
                    `-> <auto>.func (type:call/int size:0x8 loc:stack/-0x10)
                        `-> 0x10 (type:int/none size:0x0 loc:virtual)
            `-> method.count (type:call/int size:0x8 loc:nowhere)
dbg compile_probe       : kprobe:SyS_*
  0:    mov     ctx, r1
dmp compile_post        : > func (call/int/0x8)
  1:    mov     r0, #0x0
  2:    stdw    [sp - 0x10], r0
  3:    mov     r1, sp
  4:    add     r1, #-0x10
  5:    mov     r2, #0x8
  6:    mov     r3, ctx
  7:    add     r3, #0x80
  8:    call    probe_read
dmp compile_post        : < func (call/int/0x8)
dmp compile_post        : > <rec> (rec/rec/0x8)
dmp compile_post        : < <rec> (rec/rec/0x8)
dmp compile_post        : > @ (map/int/0x8)
  9:    mov     r0, #0x0
 10:    stdw    [sp - 0x8], r0
 11:    lddw    r1, r1
 12:    ldw     r0, #0x0
 13:    mov     r2, sp
 14:    add     r2, #-0x10
 15:    call    map_lookup_elem
 16:    jeq     r0, #0x0, +5
 17:    mov     r1, sp
 18:    add     r1, #-0x8
 19:    mov     r2, #0x8
 20:    mov     r3, r0
 21:    call    probe_read
dmp compile_post        : < @ (map/int/0x8)
dmp compile_post        : > count (call/int/0x8)
 22:    lddw    r0, [sp - 0x8]
 23:    add     r0, #0x1
 24:    stdw    [sp - 0x8], r0
dmp compile_post        : < count (call/int/0x8)
dmp compile_post        : > <method> (method/none/0)
 25:    lddw    r1, r1
 26:    ldw     r0, #0x0
 27:    mov     r2, sp
 28:    add     r2, #-0x10
 29:    mov     r3, sp
 30:    add     r3, #-0x8
 31:    mov     r4, #0x0
 32:    call    map_update_elem
dmp compile_post        : < <method> (method/none/0)
 33:    mov     r0, #0x0
 34:    exit

Any idea what am I doing wrong?

problem on linux 4.19.91

root@vm_compiler-s8:~/ply# ply -dS 'k:schedule { exit(0); }'

-- globals
int stdbuf{u32}
k:schedule
{}void
bwritevoid
ctxvoid __bpf *
stdbufint{u32}
:structstruct :anon_0x24058e0
0u64
:structstruct :anon_0x2405880
0int

-- locals
void __bpf *ctx
-- ir
0 movq r2, r1
;; >pre {}()
;; >pre bwrite()
;; >pre ctx()
;; >post ctx()
;; >pre stdbuf()
;; >post stdbuf()
;; >pre :struct()
1 stw [bp - 0x4], #0x0
;; >pre <0>
;; >post <0>
;; >pre :struct()
;; >pre <0>
;; >post <0>
;; >post :struct()
2 stw [bp - 0x8], #0x0
;; >post :struct()
3 stq [bp - 0x10], #0x0
;; >post bwrite()
4 movq r1, r2
5 ldmap r2, stdbuf
6 ldw r0, #0x0
7 movw r3, #-0x1
8 movq r4, bp
9 addq r4, #-0x10
10 movq r5, #0x10
11 call perf_event_output
;; >post {}()
12 exit
error: could not link map to queue
error: unable to create buffer 'stdbuf'
ERR:-22

unknown word needs better error message

Mistyping arg(0) as arg0:

# ply -c 'kprobe:SyS_open { printf("open %s\n", arg0); }'
ERR probe_load          : kprobe:SyS_open : Permission denied
ERR probe_load          : output from kernel bpf verifier:
0: (bf) r9 = r1
1: (62) *(u32 *)(r10 +0) = 0
invalid stack off=0 size=4

Same as:

# ply -c 'kprobe:SyS_open { printf("open %s\n", foo); }'
ERR probe_load          : kprobe:SyS_open : Permission denied
ERR probe_load          : output from kernel bpf verifier:
0: (bf) r9 = r1
1: (62) *(u32 *)(r10 +0) = 0
invalid stack off=0 size=4

Should say something like: "foo" unknown.

Some ply functions can't work?

hi, I enabled ply in our embedded target board, and can successfully run commands like:

 ply 'tracepoint:irq/irq_handler_entry { @[data->irq] = count(); }'
 ply 'kretprobe:vfs_read { @[pid] = quantize(retval); }'
 ply 'tracepoint:sched/sched_switch { @[kstack] = count();}'

But when I want to print syscall's argument, it can't be printed out. For example:
ply 'kprobe:do_sys_open { printf("opened:%s\n", str(arg1));}'_
then I opened a file. But only prints out:
opened
The str(arg1) seems empty...

Also met below errors when I tried to run commands in https://www.slideshare.net/brendangregg/linux-4x-tracing-performance-analysis-with-bccbpf

_sh-4.4# ply 'kprobe:do_sys_open { printf("%s %ld\n", comm, nsecs()); }'
<input>:1:46-51: error: type of symbol 'nsecs' is unknown
<input>:1:21-54: error: type of symbol ':struct' is unknown
<input>:1:21-54: error: type of symbol ':struct' is unknown

sh-4.4# ply 'kretprobe:vfs_read { @[func()] = quantize(retval); }'                                                                                                                                          
<input>:1:21-22: error: type of symbol '@' is unknown
<input>:1:23-27: error: type of symbol 'func' is unknown
<input>:1:22-30: error: type of symbol ':struct' is unknown

Is it because func() and nsecs() are not available in current ply? Are there any alternative functions?Thanks.

How to concatenate strings?

Is it possible to do this? It'd be nice to have this, so that I can implement execsnoop with return code check. Currently, I can only do stuff like this:

kprobe:sys_execve {
  x = 0;

  printf("%s: ", mem(arg(0), "128s"));

  unroll (10) {
    if (mem(arg(1) + x*sizeof("p"), "p")) {
      printf("%s ", mem(mem(arg(1) + x*sizeof("p"), "p"), "128s"));
      x=x+1;
    }
  }

  printf("\n");
}

Segmentation fault due to null pointer dereference in ply_compile in type_base

Hello,

the following file, found by fuzzing ply, attached below, causes ply to segfault when it is ran with the file.
nullptr.zip

The issue seems to be caused by a null dereference in type.h in type_base. As struct type *t is NULL, accessing t->ttype accesses invalid memory.

The following patch seems to fix the issue:

diff --git a/src/libply/built-in/memory.c b/src/libply/built-in/memory.c
index b3780e6..2dbe056 100644
--- a/src/libply/built-in/memory.c
+++ b/src/libply/built-in/memory.c
@@ -261,6 +261,7 @@ static int struct_deref_type_infer(const struct func *func, struct node *n)
        if (t->ttype != T_POINTER) {
                _ne(n, "%N is not a pointer (type '%T').\n",
                    sou, sou->sym->type);
+               return -EINVAL;
        }

        t = type_base(t->ptr.type);

The unpatched version produces the following output, so I guess that exiting after the issue has been detected should work as a fix:

[root@fuzz ply]# ./src/ply/.libs/ply ./nullptr
1: error: unknown token
<input>:1:3-9: error: 2 is not a pointer (type 'int').
Segmentation fault (core dumped)

Regards,
Juraj

error when trying to use "strcmp" and "execname" on ubuntu 17.04

This is my first time trying to use ply, so I apologize if this is a self induced issue, but I'm trying to execute the following command (example taken from the man page):

$ sudo ply -c 'kprobe:SyS_* / !strcmp(execname, "dd") / { @[func()].count() }'
ERR static_post         : node:<auto>.strcmp (type:call/none size:0x0 loc:nowhere) : Invalid argument
ERR annotate_script     : static type inference failed

Any clues as to what might be causing the error? and/or maybe point me to some docs/source code so I can try to read up on how this is supposed to work?

I built ply from source, and am using an Ubuntu 17.04 system:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.04
Release:        17.04
Codename:       zesty

$ uname -a
Linux ubuntu 4.10.0-19-generic #21-Ubuntu SMP Thu Apr 6 17:04:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ ply -v
ply-v1-beta1(9e810b1) (linux-version:264712~4.10.8)

I had to download Ubuntu's kernel sources, and use the --with-kerneldir option to workaround #7

stack() results are unexpectedly short (on kernel 4.9.0)

Use-case: I'm trying to trace long page faults.

#!/usr/sbin/ply

kprobe:handle_mm_fault
{
    @lat[pid()] = nsecs();
}

kretprobe:handle_mm_fault
{
    start = @lat[pid()]; # Fetch the time at handle_mm_fault() entry.
    if (start == 0) {
        return; # Don't proveed if we don't have a start time.
    }
    lat = nsecs() - start;
    limit = 1000000;
    if (lat > limit) {
        printf("%16s(%5d): handle_mm_fault took more than %dus (%dus)", comm(), pid(), limit / 1000, lat / 1000);
        printf("%v\n\n", stack());
    }
    @lat[pid()] = nil; # Clear map.
}

This produces output like the following:

             git(191300): handle_mm_fault took more than 1000us (9121us)
	kretprobe_trampoline
	trace_page_fault+0x28

             git(191300): handle_mm_fault took more than 1000us (1524us)
	kretprobe_trampoline
	trace_page_fault+0x28

            cron(191662): handle_mm_fault took more than 1000us (1028us)
	kretprobe_trampoline
	trace_page_fault+0x28

             zsh(190053): handle_mm_fault took more than 1000us (2037us)
	kretprobe_trampoline
	__wake_up_parent
	trace_page_fault+0x28
	__wake_up_parent
	__put_user_4+0x1c
	schedule_tail+0x40
	ret_from_fork+0x8

          chrome(72654): handle_mm_fault took more than 1000us (3971us)
	kretprobe_trampoline
	trace_page_fault+0x28

            nvim(190453): handle_mm_fault took more than 1000us (2483us)
	kretprobe_trampoline
	trace_page_fault+0x28
	__clear_user+0x21
	load_elf_binary+0x12e6
	search_binary_handler+0xb8
	search_binary_handler+0xa0
	do_execveat_common.isra.37+0x5aa
	sys_execve+0x35
	kretprobe_trampoline
	return_from_SYSCALL_64

Sometimes the stacktrace looks like its rooted at kernel entry (return_from_SYCALL_64), but most of the time I only see the kretprobe_trampoline and trace_page_fault.

I'm running a Linux 4.9.0 kernel. Looking for trace_page_fault, I only find assembly:

#ifdef CONFIG_TRACING
ENTRY(trace_page_fault)
	ASM_CLAC
	pushl	$trace_do_page_fault
	jmp	error_code
END(trace_page_fault)
#endif

Could that be what's messing with the stack creation?

I tried running the same with trace(8) from bcc-tools and it appears that this is not ply's fault:

$ sudo trace -K -I linux/mm.h 'r::handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) "%d", retval'
46540   46540   chrome          handle_mm_fault  0
        kretprobe_trampoline+0x0 [kernel]
        page_fault+0x28 [kernel]

46540   46540   chrome          handle_mm_fault  0
        kretprobe_trampoline+0x0 [kernel]
        page_fault+0x28 [kernel]

Strangely enough, the referenced function is called trace_fault here, instead of trace_page_fault. Does ply have custom code to avoid printing any trace_ prefix?

Another question/suggestions: perhaps ply should filter out the kretprobe_trampoline stuff (unless in debugging mode)?

So I guess this is just a heads up. I notice the `page_fault

Final note: non-kretprobe tracing has saner stacks:

$ sudo trace -K -I linux/mm.h 'p::handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) "%d", address
45283   45283   chrome          handle_mm_fault  72798280
        handle_mm_fault+0x1 [kernel]
        __do_page_fault+0x25c [kernel]
        page_fault+0x28 [kernel]

45283   45283   chrome          handle_mm_fault  72798280
        handle_mm_fault+0x1 [kernel]
        __do_page_fault+0x25c [kernel]
        page_fault+0x28 [kernel]

Although I didn't expect it to be this short, it makes ~sense.

Thanks for building a great low-dependency tool, its code is a joy to read (the parts I've seen, at least).

wild cards not working, quantize() not working

Hi,

This should maybe be two issues but I'm posting them as one for now because maybe the root cause is the same i.e. user error on my part.

I've added ply to buildroot and have basic scripts like the snippet below working on x86_64. I'm using kernel 5.4. buildroot is building that kernel alongside the toolchain so I don't think this is a problem with the kernel headers and toolchain being out of sync.

kprobe:vfs_read
{
        @[caller] = count();
}

I'm having problems getting wildcards and quantize() working.

if I change vfs_read to vfs_* in the above snippet it looks like attaching the probes fails:

openat(AT_FDCWD, "/sys/kernel/debug/tracing/kprobe_events", O_RDWR|O_CREAT|O_APPEND, 0666) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
write(7, "p:ply195/p610b50_vfs_fadvise vfs"..., 3119) = -1 EINVAL (Invalid argument)
close(7)                                = 0
close(6)                                = 0
close(5)                                = 0
munmap(0x7fef69ff4000, 3425280)         = 0
close(4)                                = 0
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x5, 0x1), ...}) = 0
ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0
write(1, "ERR:-22\n", 8ERR:-22
)                = 8
exit_group(1)                           = ?
+++ exited with 1 +++

If I change the above snippet to this:

#!/usr/bin/env ply

kretprobe:vfs_read
{
        @["rdsz"] = quantize(retval);
}

I get a huge wall of output like this:

from 72 to 75: R0=inv28 R1=inv1 R2=inv0 R6=ctx(id=0,off=0,imm=0) R10=fp0 fp-8=mmmmmmmm fp-16=mmmmmmmm fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm fp-80=mmmmmmmm fp-88=mmmmmmmm fp-96=mmmmmmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=mmmmmmmm fp-128=mmmmmmmm fp-136=mmmmmmmm fp-144=mmmmmmmm fp-152=mmmmmmmm fp-160=mmmmmmmm fp-168=mmmmmmmm fp-176=mmmmmmmm fp-184=mmmmmmmm fp-192=mmmmmmmm fp-200=mmmmmmmm fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=mmmmmmmm fp-232=mmmmmmmm fp-240=mmmmmmmm fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=mmmmmmmm fp-272=mmmmmmmm
75: (67) r0 <<= 2
76: (bf) r1 = r10
77: (07) r1 += -264
78: (0f) r1 += r0
last_idx 78 first_idx 69
regs=1 stack=0 before 77: (07) r1 += -264
regs=1 stack=0 before 76: (bf) r1 = r10
regs=1 stack=0 before 75: (67) r0 <<= 2
regs=1 stack=0 before 72: (b5) if r1 <= 0x1 goto pc+2
regs=1 stack=0 before 69: (b5) if r1 <= 0x3 goto pc+2
 R0_rw=invP28 R1_rw=inv(id=0,umin_value=1,umax_value=68719476735,var_off=(0x0; 0xfffffffff)) R2=inv0 R6=ctx(id=0,off=0,imm=0) R10=fp0 fp-8_r=mmmmmmmm fp-16_r=mmmmmmmm fp-24_r=mmmmmmmm fp-32_r=mmmmmmmm fp-40_r=mmmmmmmm fp-48_r=mmmmmmmm fp-56_r=mmmmmmmm fp-64_r=mmmmmmmm fp-72_r=mmmmmmmm fp-80_r=mmmmmmmm fp-88_r=mmmmmmmm fp-96_r=mmmmmmmm fp-104_r=mmmmmmmm fp-112_r=mmmmmmmm fp-120_r=mmmmmmmm fp-128_r=mmmmmmmm fp-136_r=mmmmmmmm fp-144_r=mmmmmmmm fp-152_r=mmmmmmmm fp-160_r=mmmmmmmm fp-168_r=mmmmmmmm fp-176_r=mmmmmmmm fp-184_r=mmmmmmmm fp-192_r=mmmmmmmm fp-200_r=mmmmmmmm fp-208_r=mmmmmmmm fp-216_r=mmmmmmmm fp-224_r=mmmmmmmm fp-232_r=mmmmmmmm fp-240_r=mmmmmmmm fp
ERR:-28

This seems to be output from the bpf verification in the kernel? I'm pretty new to bpf so I don't think I'll have much hope of debugging that. If posting the complete output/the code ply generates would be helpful I can do that.

Thanks,

Daniel

Is it supposed to work on armv7 and 5.10.x kernel?

---

# cat /proc/version 
Linux version 5.10.2-debug (oe-user@oe-host) (arm-resy-linux-gnueabi-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35.0.20200730) #1 SMP Mon Dec 21 12:30:08 UTC 2020

---

# cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 6.00
Features        : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

processor       : 1
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 6.00
Features        : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

processor       : 2
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 6.00
Features        : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

processor       : 3
model name      : ARMv7 Processor rev 10 (v7l)
BogoMIPS        : 6.00
Features        : half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x2
CPU part        : 0xc09
CPU revision    : 10

Hardware        : Freescale i.MX6 Quad/DualLite (Device Tree)
Revision        : 0000
Serial          : 0000000000000000

---

# ply  -T
Verifying kernel config (/proc/config.gz)... OK
Ensuring that debugfs is mounted... OK
Verifying kprobe... OK
Verifying tracepoint... OK

---

# ply -v
ply 2.1.1-14-ge25c913 (linux-version:330240~5.10.0)

This is what's currently on master.

# ply 'kretprobe:vfs_read { @["size"] = quantize(retval); }'
ply: active
^Cply: deactivating

@:
{ size    }: 
                 < 0           1 ┤▏                               │
        [   0,    1]         111 ┤████████████████▎               │
        [   2,    3]          19 ┤██▊                             │
        ...
        [  16,   31]           2 ┤▎                               │
        [  32,   63]          66 ┤█████████▊                      │
        ...
        [ 512,   1k)           5 ┤▊                               │
        ...
        [  4k,   8k)          14 ┤██                              │

root@multi-v7-ml:~/projects/ply/scripts# 

---

# ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'
ply: active

executed find and cat

^Cply: deactivating

---

# ./opensnoop.ply
ply: active

cd /
find | grep linux

^Cply: deactivating

path:

---

# ./execsnoop.ply
ERR:-2

---

# ./net-rx.ply 
ERR:-2

---

Can you please advise on the test cases?
In case you want me to run more tests, or different things please let me know.

printing strings

# ply -c 'kprobe:SyS_open { printf("open %s\n", arg(0)); }'
1 probe active
open pr??
open ?q??
open ?!A
open pr??
open ?q??

Ok, I need to use mem(). This ticket is to explore either:

A) not needing mem(). Previous command would just work.

B) a shortcut, like:

# ply -c 'kprobe:SyS_open { printf("open %s\n", str(arg(0))); }'

where str() could be:

#define str(p)    (mem(p, "128s"))

Could also make the 128 a global tunable.

kretprobe support

arg(0) could be the return value

Out of bounds write due to size_t underflow in ply_fparse

Hello,

fuzzing ply_fparse has revealed an issue that causes a segmentation fault. It can also be reproduced by running ply with the generated file directly. The file is attached below, as an attachment to this issue, as it contains unprintable characters.
crash.zip

I've reproduced the crash by running src/ply/.libs/ply with the file. Valgrind's output is shown below:

==1716== Invalid write of size 1
==1716==    at 0x483CFF4: strncpy (vg_replace_strmem.c:550)
==1716==    by 0x4873FA1: node_string (node.c:201)
==1716==    by 0x4870641: yylex (lexer.l:65)
==1716==    by 0x4867C0D: yyparse (grammar.c:1504)
==1716==    by 0x487AE22: ply_fparse (libply.c:148)
==1716==    by 0x10A809: main (ply.c:194)
==1716==  Address 0x4a82d80 is 0 bytes after a block of size 0 alloc'd
==1716==    at 0x483BB65: calloc (vg_replace_malloc.c:760)
==1716==    by 0x4873F5D: xcalloc (utils.h:65)
==1716==    by 0x4873F5D: node_string (node.c:200)
==1716==    by 0x4870641: yylex (lexer.l:65)
==1716==    by 0x4867C0D: yyparse (grammar.c:1504)
==1716==    by 0x487AE22: ply_fparse (libply.c:148)
==1716==    by 0x10A809: main (ply.c:194)

After analyzing the issue, the cause seems to be the statement len = strlen(data) - 2;. In this case strlen(data) is 1, so subtracting 2 causes an underflow, which then causes the strncpy in node.c:201 to write outside of the bounds of the unquoted buffer.

This patch seems to fix the crash and if it is appropriate I can open a pull request with it.

diff --git a/src/libply/node.c b/src/libply/node.c
index 5a14a07..8885639 100644
--- a/src/libply/node.c
+++ b/src/libply/node.c
@@ -195,7 +195,10 @@ struct node *node_string(const struct nloc *loc, char *data)
        if (data[0] == '"') {
                char *unquoted;

-               len = strlen(data) - 2;
+               len = strlen(data);
+               if (len > 1 && data[len - 1] == '"') {
+                       len -= 2;
+               }

                unquoted = xcalloc(1, len + 1);
                strncpy(unquoted, data + 1, len);

However, I'm not sure if that is appropriate and whether a fix in lexer.l would work better, as I'm not sure why the regex in lexer.l:65 matched the single quote as a string.

If additional information about the crash is required, or if I should change something in the patch, please let me know.

Regards,
Juraj

Test failure

Hi, I've tried to run the test but it failed like below

ply$ cd  test/
test$ ls
Makefile  README.md  test.conf  test.sh  test-wrapper.sh
test$ make
mkdir -p work/basis-ply
mkdir -p cache
wget -O cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2 https://toolchains.bootlin.com/downloads/releases/toolchains/armv5-eabi/tarballs/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2
--2021-09-22 12:41:46--  https://toolchains.bootlin.com/downloads/releases/toolchains/armv5-eabi/tarballs/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2
Resolving toolchains.bootlin.com (toolchains.bootlin.com)... 51.254.149.199
Connecting to toolchains.bootlin.com (toolchains.bootlin.com)|51.254.149.199|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 121154542 (116M) [application/x-bzip2]
Saving to: ‘cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2’

cache/armv5-eabi--glibc--bleedi 100%[=======================================================>] 115.54M   168KB/s    in 11m 0s  

2021-09-22 12:52:47 (179 KB/s) - ‘cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2’ saved [121154542/121154542]

tar -C cache -maxf cache/armv5-eabi--glibc--bleeding-edge-2020.08-1.tar.bz2
cd work/basis-ply cache/armv5-eabi--glibc--bleeding-edge-2020.08-1 && PATH=/home/namhyung/.local/bin:/home/namhyung/.cargo/bin:/home/namhyung/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/namhyung/go/bin:/home/namhyung/project/ply/test/cache/armv5-eabi--glibc--bleeding-edge-2020.08-1/bin ../../../configure CFLAGS="-Wall -Wextra -Werror" --host=arm-linux --prefix=
/bin/sh: line 1: cd: too many arguments
make: *** [Makefile:151: work/basis-ply/Makefile] Error 1

how can i get a variable value in function with kprobe.

kernel version :5.7
I want to get variable ’duration‘ value in watchdog_timer_fn, how i should do ?

such as:
ply 'kprobe:watchdog_timer_fn+ offset {xxxxx}'

Can you provide a full command for me ?

Thanks.

Example script has a syntax error

I've been trying to get the trace provider working with a predicate, but running the example script gives me this error:

error(4): syntax error, unexpected '{'

Here's the script:

#!/usr/bin/env ply

trace:raw_syscalls/sys_exit / ret() < 0 /
{
	@[comm()].count()
}

syscall-fail.ply failed with syntax error

Hi there,

When I test syscall-fail.ply script, it reports error:

root@linaro-developer:~/ply/scripts# ply syscall-fail.ply
error(6): syntax error, unexpected '{'

I tried to modify for ply script, but have no luck to fix it. Could you check for this?

what's the difference with ktap

seems the same. like both lightweight and the samilar syntax

ply keeps failing with probe_load error

I compiled and installed ply, but it keeps throwing probe_load errors when I try to run.

$ sudo ply -d -c 'kretprobe:SyS_read{ @.quantize(retval()); }'
dbg memlock_uncap       : unlimited memlock
dbg annotate_script     : ok
dbg map_setup           : @: type:1 ksize:0x10 vsize:0x8 nelem:0x400
dbg compile_probe       : kretprobe:SyS_read
ERR probe_load          : kretprobe:SyS_read : Invalid argument
ERR probe_load          : no output from kernel verifier
ERR probe_load          : was ply built against the running kernel?

Is there some obvious step I am missing?

I keep hitting the same error for some of the other examples in the README as well.

quantize histogram output as an ASCII diagram

# ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2)); }'
1 probe active
^Cde-activating probes

$bytes:
[   0,    1]         698
[  64,  128)          31
[ 128,  256)           2
[ 512,   1k)         109
[  1k,   2k)          18
[  4k,   8k)          28
[ 16k,  32k)           3

This is a good terse output for power-of-2 histograms, however, I think it should be optional, and the default should be something more easily readable, like:

      value          : count     distribution
       0 -> 1        : 3        |                                      |
       2 -> 3        : 0        |                                      |
       4 -> 7        : 211      |**********                            |
       8 -> 15       : 0        |                                      |
      16 -> 31       : 0        |                                      |
      32 -> 63       : 0        |                                      |
      64 -> 127      : 1        |                                      |
     128 -> 255      : 800      |**************************************|

Note that zero rows are included, as well as an ASCII distribution. (The ASCII distribution would make less sense if the zero rows were elided). Optionally, the heading "value" could be changed. Eg, in this case, to "bytes".

This could be accomplished a number of ways. Here's one suggestion:

# emits the raw output (as it does today):
ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2), raw=true); }' 

# emits the sample output above:
ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2)); }'

# emits the sample output, but with the value column renamed:
ply -c 'kprobe:SyS_read { $bytes.quantize(arg(2), value="bytes"); }'

I've opted against just adding optional arguments to quantize (eg, "quantize(arg(2), true, "bytes"). I think positional arguments like those should be for the common use cases. In cases where the argument would be uncommonly used, a key=value approach may be easier for users to learn and remember, since it self documents.

Type error with stored char * variable:

# ./opensnoop.ply 
ERR static_post         : node:<auto>.mem (type:call/none size:0x0 loc:nowhere) : Invalid argument
ERR annotate_script     : static type inference failed

code:

#!/usr/bin/env ply
/*
 * opensnoop    trace file opens.
 */

kprobe:do_sys_open
{
    @path[pid()] = arg(1);
}

kretprobe:do_sys_open / @path[pid()] /
{
    if (retval() > 0) {
        fd = retval();
        errno = 0;
    } else {
        fd = /* minus */ 1;
        errno = /* minus */ retval();
    }

    printf("%-6d %-16s %4d %3d %s\n", pid(), comm(), fd, errno,
        mem(@path[pid()], "128s"));

    @path[pid()] = nil;
}

(Ignore the "/* minus */", which are pending #11 .)

This code is pretty nice BTW.

build ply for arm64 target

Hi,
I'm trying to build ply for arm64 target but somehow I'm unable to do it:
I used configure like this:
./configure --target=arm64 --with-kerneldir=/home/kedaru/proj/base/linux/

But I still get x86_64 ply file after a make. How do I compile ply executable for arm64 target ?

Thanks.

arm linux 4.19 'could not link map to queue'

I am having some trouble getting ply going on an arm chromebook.

Linux localhost 4.19.184 #29 SMP PREEMPT Mon Apr 12 17:19:19 AEST 2021 aarch64 GNU/Linux

I compiled using the gentoo ebuild: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-util/ply

Which uses ply-2.1.1 + an arm patch: https://github.com/iovisor/ply/commit/1bc183af8703023e377f92716ecd3f339ffffd11.patch

# ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }'
error: could not link map to queue
error: unable to create buffer 'stdbuf'
ERR:-22

Which looks like this issue: #48
I am not sure the best testcase to run, nothing works that I can see.

Here is the strace snippet:

openat(AT_FDCWD, "/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
read(3, "0-3\n", 8192)                  = 4
close(3)                                = 0
access("k:schedule { exit(0); }", R_OK) = -1 ENOENT (No such file or directory)
getpid()                                = 19458
stat64("/var/tmp/ply-ksyms", {st_mode=S_IFREG|0644, st_size=192, ...}) = 0
openat(AT_FDCWD, "/var/tmp/ply-ksyms", O_RDWR) = 3
mmap2(NULL, 192, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0xead4c000
stat64("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
stat64("/var/tmp/ply-ksyms", {st_mode=S_IFREG|0644, st_size=192, ...}) = 0
ugetrlimit(RLIMIT_MEMLOCK, {rlim_cur=64*1024, rlim_max=64*1024}) = 0
prlimit64(0, RLIMIT_MEMLOCK, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}, NULL) = 0
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=4, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0}, 48) = 4
perf_event_open({type=PERF_TYPE_SOFTWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=PERF_COUNT_SW_BPF_OUTPUT, ...}, -1, 0, -1, 0) = 5
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=4, key=0xffe63d34, value=0x106f57e4, flags=BPF_ANY, ...}, 48) = -1 EINVAL (Invalid argument)
write(2, "error: could not link map to que"..., 35error: could not link map to queue

Let me know what else I can try

config.txt

assertion with syscall tracepoint

Hi,

I got an assertion with sys_enter_openat tracepoint but not with sys_enter_open as below.

root@osboxes:/home/sgkim/work/tmp/ply# ply -d 'tracepoint:syscalls/sys_enter_openat { printf("%v", comm); }'
ply: provider/tracepoint.c:197: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.
Aborted
root@osboxes:/home/sgkim/work/tmp/ply# ply -d 'tracepoint:syscalls/sys_enter_open { printf("%v", comm); }'
ply: active
^Cply: deactivating

I think it is because the offset of dfd is not matching 12 = previous offset(8) + size of __syscall_nr(4) but the offset of "filename" is padded in sys_enter_open case.

root@osboxes:/sys/kernel/debug/tracing# cat events/syscalls/sys_enter_openat/format
name: sys_enter_openat
ID: 607
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:int __syscall_nr; offset:8;       size:4; signed:1;
        field:int dfd;  offset:16;      size:8; signed:0;
        field:const char * filename;    offset:24;      size:8; signed:0;
        field:int flags;        offset:32;      size:8; signed:0;
        field:umode_t mode;     offset:40;      size:8; signed:0;

print fmt: "dfd: 0x%08lx, filename: 0x%08lx, flags: 0x%08lx, mode: 0x%08lx", ((unsigned long)(REC->dfd)), ((unsigned long)(REC->filename)), ((unsigned long)(REC->flags)), ((unsigned long)(REC->mode))
root@osboxes:/sys/kernel/debug/tracing# cat events/syscalls/sys_enter_open/format
name: sys_enter_open
ID: 609
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;

        field:int __syscall_nr; offset:8;       size:4; signed:1;
        field:const char * filename;    offset:16;      size:8; signed:0;
        field:int flags;        offset:24;      size:8; signed:0;
        field:umode_t mode;     offset:32;      size:8; signed:0;

print fmt: "filename: 0x%08lx, flags: 0x%08lx, mode: 0x%08lx", ((unsigned long)(REC->filename)), ((unsigned long)(REC->flags)), ((unsigned long)(REC->mode))

This is tested with the commit aa5b9ac and kernel version 5.0.0-23, so could you have a look on this?
I tried to check it by myself, but have no idea how to fix it without breaking other cases.

newer kernel support

Maybe I'm doing something wrong:

(root) /mnt/src/ply # ply -c 'kprobe:SyS_* { @syscalls[func()].count() }'
ERR probe_load          : kprobe:SyS_* : Invalid argument
ERR probe_load          : no output from kernel verifier
ERR probe_load          : was ply built against the running kernel?

(root) /mnt/src/ply # ply -c 'kprobe:tcp_sendmsg { @names[func()].count() }'
ERR probe_load          : kprobe:tcp_sendmsg : Invalid argument
ERR probe_load          : no output from kernel verifier
ERR probe_load          : was ply built against the running kernel?

(root) /mnt/src/ply # uname -a
Linux bgregg-xenia 4.9.0-rc5-virtual #1 SMP Fri Nov 18 22:01:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

(root) /mnt/src/ply # ply -v
ply-v1-beta1(e2cc48e) (linux-version:263203~4.4.35)

(root) /mnt/src/ply # ply -D -c 'kprobe:tcp_sendmsg { @names[func()].count() }'
symtable:
@names(kprobe:tcp_sendmsg)               (type:map/int size:0x8 loc:stack/-0x8)
ast:
`-> <script> (type:script/none size:0x0 loc:nowhere)
    `-> kprobe:tcp_sendmsg (type:probe/none size:0x0 loc:nowhere)
        `-> <method> (type:method/none size:0x0 loc:nowhere)
            `-> @names (type:map/int size:0x8 loc:stack/-0x8)
                `-> <rec> (type:rec/rec size:0x8 loc:stack/-0x10)
                    `-> <auto>.func (type:call/int size:0x8 loc:stack/-0x10)
                        `-> 0x10 (type:int/none size:0x0 loc:virtual)
            `-> method.count (type:call/int size:0x8 loc:nowhere)
  0:	mov	ctx, r1
dmp compile_post        : > func (call/int/0x8)
  1:	mov	r0, #0x0
  2:	stdw	[sp - 0x10], r0
  3:	mov	r1, sp
  4:	add	r1, #-0x10
  5:	mov	r2, #0x8
  6:	mov	r3, ctx
  7:	add	r3, #0x80
  8:	call	probe_read
dmp compile_post        : < func (call/int/0x8)
dmp compile_post        : > <rec> (rec/rec/0x8)
dmp compile_post        : < <rec> (rec/rec/0x8)
dmp compile_post        : > @names (map/int/0x8)
  9:	mov	r0, #0x0
 10:	stdw	[sp - 0x8], r0
 11:	lddw	r1, r1
 12:	ldw	r0, #0x0
 13:	mov	r2, sp
 14:	add	r2, #-0x10
 15:	call	map_lookup_elem
 16:	jeq	r0, #0x0, +5
 17:	mov	r1, sp
 18:	add	r1, #-0x8
 19:	mov	r2, #0x8
 20:	mov	r3, r0
 21:	call	probe_read
dmp compile_post        : < @names (map/int/0x8)
dmp compile_post        : > count (call/int/0x8)
 22:	lddw	r0, [sp - 0x8]
 23:	add	r0, #0x1
 24:	stdw	[sp - 0x8], r0
dmp compile_post        : < count (call/int/0x8)
dmp compile_post        : > <method> (method/none/0)
 25:	lddw	r1, r1
 26:	ldw	r0, #0x0
 27:	mov	r2, sp
 28:	add	r2, #-0x10
 29:	mov	r3, sp
 30:	add	r3, #-0x8
 31:	mov	r4, #0x0
 32:	call	map_update_elem
dmp compile_post        : < <method> (method/none/0)
 33:	mov	r0, #0x0
 34:	exit

Various basic example failures

I want to package ply for ALT Linux. But, most basic eamples I try failing with different errors. What this could be?

5.4.51-std-def-alt1:~# cat > count-syscallt.ply
#!/usr/bin/env ply

kprobe:SyS_*
{
    @syscalls[caller] = count();
}
5.4.51-std-def-alt1:~# ply count-syscallt.ply
  0     movq    v0, r1
;; >pre  {}()
;; >pre  =()
;; >pre  []()
;; >pre  @syscalls()
;; >post @syscalls()
;; >pre  caller()
;; >pre  .()
;; >pre  u*()
;; >pre  regs()
;; >pre  ctx()
;; >post ctx()
;; >post regs()
;; >post u*()
;; >pre  "rip"
;; >post "rip"
;; >post .()
  1     movq    r3, v0
  2     addq    r3, #0x80
  3     movq    r2, #0x8
  4     movq    r1, bp
  5     addq    r1, #-0x10
  6     call    probe_read
;; >post caller()
  7     ldq     r0, [bp - 0x10]
  8     stq     [bp - 0x18], r0
;; >post []()
;; >pre  count()
;; >post count()
  9     ldq     r0, [bp - 0x8]
 10     addq    r0, #0x1
 11     stq     [bp - 0x8], r0
;; >post =()
ply: ir.c:389: ir_emit_sym_to_stack: Assertion `0' failed.
Aborted
5.4.51-std-def-alt1:~#

5.4.51-std-def-alt1:~# ply 'kretprobe:vfs_read { @["size"] = quantize(retval); }'
  0     movq    v0, r1
;; >pre  {}()
;; >pre  =()
;; >pre  []()
;; >pre  @()
;; >post @()
;; >pre  "size"
;; >post "size"
  1     stw     [bp - 0x108], #0x657a6973
  2     stw     [bp - 0x104], #0x0
;; >post []()
;; >pre  quantize()
;; >pre  retval()
;; >pre  .()
;; >pre  u*()
;; >pre  regs()
;; >pre  ctx()
;; >post ctx()
;; >post regs()
;; >post u*()
;; >pre  "rax"
;; >post "rax"
;; >post .()
  3     movq    r3, v0
  4     addq    r3, #0x50
  5     movq    r2, #0x8
  6     movq    r1, bp
  7     addq    r1, #-0x110
  8     call    probe_read
;; >post retval()
  9     ldq     v1, [bp - 0x110]
;; >post quantize()
 10     movq    r0, #0x0
 11     movq    r1, v1
 12     movq    r2, r1
 13     rshq    r2, #0x20
 14     jeq     r2, #0x0, +2
 15     addq    r0, #0x20
 16     movq    r1, r2
 17     jle     r1, #0xffff, +2
 18     addq    r0, #0x10
 19     rshq    r1, #0x10
 20     jle     r1, #0xff, +2
 21     addq    r0, #0x8
 22     rshq    r1, #0x8
 23     jle     r1, #0xf, +2
 24     addq    r0, #0x4
 25     rshq    r1, #0x4
 26     jle     r1, #0x3, +2
 27     addq    r0, #0x2
 28     rshq    r1, #0x2
 29     jle     r1, #0x1, +2
 30     addq    r0, #0x1
 31     rshq    r1, #0x1
 32     lshq    r0, #0x2
 33     movq    r1, bp
 34     addq    r1, #-0x100
 35     addq    r1, r0
 36     movq    r0, #0x1
 37     stw     r1, r0
;; >post =()
ply: ir.c:389: ir_emit_sym_to_stack: Assertion `0' failed.
Aborted
5.4.51-std-def-alt1:~#

5.4.51-std-def-alt1:~# ply 'kprobe:dev_queue_xmit { @[stack] = count(); }'
  0     movq    v0, r1
;; >pre  {}()
;; >pre  =()
;; >pre  []()
;; >pre  @()
;; >post @()
;; >pre  stack()
;; >pre  ctx()
;; >post ctx()
;; >pre  :stackmap()
;; >post :stackmap()
;; >post stack()
  1     movq    r1, v0
  2     ldmap   r2, :stackmap
  3     ldw     r0, #0x0
  4     movq    r3, #0x0
  5     call    get_stackid
  6     stw     [bp - 0xc], r0
;; >post []()
;; >pre  count()
;; >post count()
  7     ldq     r0, [bp - 0x8]
  8     addq    r0, #0x1
  9     stq     [bp - 0x8], r0
;; >post =()
ply: ir.c:389: ir_emit_sym_to_stack: Assertion `0' failed.
Aborted
5.4.51-std-def-alt1:~#

5.4.51-std-def-alt1:~# cat execsnoop.ply
#!/sbin/ply

kprobe:SyS_execve {
        execs[kpid] = str(arg0, 48);
}

kretprobe:SyS_execve {
        printf("(%4u) %v %3ld\n", uid, execs[kpid], retval);
}
5.4.51-std-def-alt1:~# ply execsnoop.ply
error: unable to load kretprobe:SyS_execve, errno:13
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-13
5.4.51-std-def-alt1:~#

• ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }' this example works!

failing on untracable functions

vfs_caches_init() can't be traced on my machine. It's in /proc/kallsyms, but it is not in /sys/kernel/debug/tracing/available_filter_functions. This is what happens with bcc:

# ./bcc/tools/trace.py vfs_caches_init
check dmesg output for possible cause
Failed to attach BPF to kprobe
dmesg | tail -1
[ 6679.602959] Could not insert probe at vfs_caches_init+0: -22
# ./bcc/tools/funccount.py 'vfs_*'
Tracing 49 functions for "vfs_*"... Hit Ctrl-C to end.
^C
FUNC                                    COUNT
vfs_fstat                                   1
vfs_getattr_nosec                           1
vfs_getattr                                 1
vfs_read                                   32
vfs_open                                   40
vfs_write                                  48

Which is ok. funccount is skipping it entirely, and trace is trying (I'm not sure it should) but failing.

With ply:

# ply -c 'kprobe:vfs_caches_init { printf("hi\n"); }'
ERR probe_event_id      : "kprobes/p_vfs_caches_init_0" : No such file or directory
# ply -c 'kprobe:vfs* { @[func()].count(); }'
ERR probe_event_id      : "kprobes/p_vfs_caches_init_early_0" : No such file or directory
ERR probe_event_id      : "kprobes/p_vfs_caches_init_0" : No such file or directory
#

The problem is the second invocation. Maybe ply should just skip probes not in available_filter_functions?

Segmentation fault due to invalid pointer dereferences when calling ply_fparse

Hello,

the files attached below cause a segmentation fault in ply. The crashes were found by fuzzing ply_fparse, like #55 and the crashes can be reproduced the same way, by feeding the files into ply.
crashes.zip

The first input, in the invalidptr1 file causes a null pointer dereference in node_nloc_valid. However, checking whether n is NULL in node_vfprintxf doesn't prevent all the crashes.

The second file invalidptr2 still causes the same crash, and bypasses the NULL pointer check, as n in that case takes a value of 1.

If any additional information is needed regarding the crash, please let me know.

Regards,
Juraj

Assertion fails in tracepoint_parse

ply: provider/tracepoint.c:198: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.

I've tried this workaround, but it didn't work:

	/* Find all basic scalars. */
	/* if (!strcmp(str, "char")) */
	/* 	return explicit_sign ? (sign ? &t_schar : &t_uchar) : &t_char; */
	/* else if (!strcmp(str, "short")) */
	/* 	return explicit_sign ? (sign ? &t_sshort : &t_ushort) : &t_short; */
	/* else if (!strcmp(str, "int")) */
	/* 	return explicit_sign ? (sign ? &t_sint : &t_uint) : &t_int; */
	/* else if (!strcmp(str, "long")) */
	/* 	return explicit_sign ? (sign ? &t_slong : &t_ulong) : &t_long; */
	/* else if (!strcmp(str, "long long")) */
	/* 	return explicit_sign ? (sign ? &t_sllong : &t_ullong) : &t_llong; */

Format:

~# cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/format
name: sys_enter
ID: 19
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;
        field:unsigned short common_migrate_disable;    offset:8;       size:2; signed:0;
        field:unsigned short common_padding;    offset:10;      size:2; signed:0;

        field:long id;  offset:16;      size:4; signed:1;
        field:unsigned long args[6];    offset:20;      size:24;        signed:0;

print fmt: "NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]

I've added some logs:

~# ply 'tracepoint:raw_syscalls/sys_enter { @[pid, comm] = count(); }'
name: common_type, offs: 0, type_offsetof: 0
name: common_flags, offs: 2, type_offsetof: 2
name: common_preempt_count, offs: 3, type_offsetof: 3
name: common_pid, offs: 4, type_offsetof: 4
name: common_migrate_disable, offs: 8, type_offsetof: 8
name: common_padding, offs: 10, type_offsetof: 10
name: id, offs: 16, type_offsetof: 12
ply: provider/tracepoint.c:198: tracepoint_parse: Assertion `offs == type_offsetof(t, t->sou.fields[n - 1].name)' failed.
Aborted (core dumped)

ASCII histogram width

# ply -c 'kprobe:vfs_read { @start[pid()] = nsecs(); } kretprobe:vfs_read / @start[pid()] / { @us.quantize((nsecs() - @start[pid()]) / 1000); @start[pid()] = 0; }'
2 probes active
^Cde-activating probes
[...]
@us:
        
	           0	     103 ┤█▏                              │
	           1	     748 ┤████████                        │
	[   2,    3]	     811 ┤████████▋                       │
	[   4,    7]	     727 ┤███████▊                        │
	[   8,   15]	     540 ┤█████▊                          │
	[  16,   31]	      30 ┤▍                               │
	[  32,   63]	       6 ┤▏                               │
	[  64,  127]	       5 ┤                                │
	[ 128,  255]	       2 ┤                                │
	[ 256,  511]	       0 ┤                                │
	[ 512,   1k)	       6 ┤▏                               │
	[  1k,   2k)	       2 ┤                                │
	[  2k,   4k)	       2 ┤                                │
	[  4k,   8k)	       6 ┤▏                               │
	[  8k,  16k)	       2 ┤                                │
	[ 16k,  32k)	       8 ┤▏                               │
	[ 32k,  64k)	       2 ┤                                │
	[ 64k, 128k)	       0 ┤                                │
	[128k, 256k)	       2 ┤                                │
	[256k, 512k)	       0 ┤                                │
	[512k,   1M)	       1 ┤                                │

ASCII graph is nice (and I like how -A is available), but could it use the full width?

type tests on printf

typo: %s instead of %d:

# ply -c 'kprobe:do_sys_open { printf("pid:%s\n", pid()); }'
1 probe active
pid:s
pid:?
pid:?_
pid:s
pid:?
pid:s
pid:?
pid:s
[...]

I suppose it is doing what I told it to do, but it would be nice if it threw an error or warning.

static linking fails

Hello,

First, thanks for the great piece of software. I can confirm it works on (armv7l) raspberry pi zero, with kernel 5.10.
I am trying to do static linking to try it on embedded system with read-only filesystem.
I am building with:

• ./autogen.sh
• ./configure CFLAGS="-static"
• make LDFLAGS="-all-static"

and getting following linking error. Any idea how to build ply as a static library?

...
Making all in ply
make[3]: Entering directory '/home/pi/ply/ply/src/ply'
  CC       ply-ply.o
cd /home/pi/ply/ply/src/ply && \
/usr/bin/ld -r -b binary -o /home/pi/ply/ply/src/ply/self-test.o self-test.sh
  CCLD     ply
/usr/bin/ld: ../libply/.libs/libply.a(libply_la-provider.o): in function `provider_get':
provider.c:(.text+0xb8): undefined reference to `__start_providers'
/usr/bin/ld: provider.c:(.text+0xbc): undefined reference to `__stop_providers'
collect2: error: ld returned 1 exit status
make[3]: *** [Makefile:388: ply] Error 1
make[3]: Leaving directory '/home/pi/ply/ply/src/ply'
make[2]: *** [Makefile:356: all-recursive] Error 1
make[2]: Leaving directory '/home/pi/ply/ply/src'
make[1]: *** [Makefile:457: all-recursive] Error 1
make[1]: Leaving directory '/home/pi/ply/ply'
make: *** [Makefile:368: all] Error 2

ply-complete-build-output.txt

Can I use it on the board through cross compilation?

I use the following instructions to compile the installation and then copy the files in the build directory to the board.

./autogen.sh
./configure --host=aarch64-linux-gnu --prefix=~/build
make
make install

After the files in the lib directory are copied to the /lib64 directory of the development board, I try to run ply in sbin directory. However, the following error message is displayed. Is the cross compilation chain incorrect?

ply: line 3: syntax error: unexpected "("

Problems loading with 4.9.16 kernel on powerpc

I am working on a 32 bit powerpc (ppc440) and all the examples I run give EINVAL (-22).

As far as I can tell the kernel header version that I am using is what is running. The --with-kerneldir does not seem to have any effect.

I am using the most recent code.
Thanks,
Neil

arm64 kernel4.19 kprobe & kretprobe all failed. why ?

ply 'kprobe:do_sys_open { printf("opened:%s\n", str(arg1));}'
[1] 41739
error: unable to load kprobe:do_sys_open, errno:22
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-22

ply 'kprobe:watchdog_timer_fn { @[stack] = count(); }'
error: unable to load kprobe:watchdog_timer_fn, errno:22
error: no output from kernel bpf verifier, retry with debugging enabled
ERR:-22

Build dependency on 'ronn'

When making the man pages, 'ronn' is required, which I didn't have:

Making all in man
make[2]: Entering directory 'HOME/ply/man'
ronn <ply.1.ronn >ply.1
/bin/sh: ronn: command not found
Makefile:520: recipe for target 'ply.1' failed

Easiest thing for me was to install ruby-ronn, at least on debian.

Error returned when some examples are tried

Hi there !

I have a problem with ply. When I try to execute some of your README examples, I have an error code returned :

apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kretprobe:SyS_read { @["size"] = quantize(retval); }'
ERR:-22
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kprobe:SyS_read / arg2 > 1024 / { @[pid] = quantize(arg2); }'
ERR:-2
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kprobe:SyS_* { @[caller] = count(); }'
ERR:-22
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo ply 'kprobe:SyS_* { @[comm, pid] = count(); }'
ERR:-22

I think this problem is similar to the issue #7 , but I'm not sure. So I tried the Brendan's solution:

apitronix@cuillere: ~/Documents/Sources/PLY $ ./autogen.sh        # This is fine
apitronix@cuillere: ~/Documents/Sources/PLY $ ./configure --with-kerneldir=/usr/linux
# This option don't exist anymore, then I'm not sure to use it right...
apitronix@cuillere: ~/Documents/Sources/PLY $ make        # This is fine
apitronix@cuillere: ~/Documents/Sources/PLY $ sudo make install        # This is fine too

Some additional information about the system:

OS version:
Linux cuillere 5.3.11-100.fc29.x86_64 #1 SMP Tue Nov 12 20:41:25 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Ply version:
ply 7978c1a (linux-version:328459~5.3.11)

accept negative values

# ply -c 'kprobe:do_sys_open { i = 42; printf("%d\n", i); }'
1 probe active
42
42
42
42
42
42
42
42
42
42
42
42
42
42
42
^Cde-activating probes

# ply -c 'kprobe:do_sys_open { i = -42; printf("%d\n", i); }'
error(1): syntax error, unexpected '-'

How to specify correct kernel version?

Hi, I failed to run kprobe command in ply due to the KERNEL_VERSION_CODE checking failed. The kernel image version code actually is 265829, but the ply version printed out is 200960. How can I include the correct version.h file in ply configure command? Currently the configuration command I use is "./configure --host=aarch64-linux-gnu --prefix=/home/data/bpf/ply/install CPPFLAGS="-I /home/data/stpu/kernel/linux/include/uapi -I /home/data/stpu/kernel/linux/include/". Thanks.

easier install without kernel source

I found this works, with the kernel headers package only (Ubuntu Yakkety):

$ git clone https://github.com/iovisor/ply
$ cd ply
$ ./autogen.sh
$ ./configure --with-kerneldir=/usr/src/linux-headers-4.8.0-30-generic
$ make
Making all in src
make[1]: Entering directory '/root/ply/src'
  LEX      lang/lex.c
  YACC     lang/parse.c
updating lang/parse.h
  HEADERS  /usr/src/linux-headers-4.8.0-30-generic(x86_64)
make -C /usr/src/linux-headers-4.8.0-30-generic ARCH=x86_64 INSTALL_HDR_PATH=/root/ply/src/.kernel headers_install
make[2]: Entering directory '/usr/src/linux-headers-4.8.0-30-generic'
  CHK     include/generated/uapi/linux/version.h
make[3]: *** No rule to make target 'arch/x86/entry/syscalls/syscall_32.tbl', needed by 'arch/x86/entry/syscalls/../../include/generated/asm/syscalls_32.h'.  Stop.
arch/x86/Makefile:191: recipe for target 'archheaders' failed
make[2]: *** [archheaders] Error 2
make[2]: Leaving directory '/usr/src/linux-headers-4.8.0-30-generic'
Makefile:1089: recipe for target '.kernel/include/linux/version.h' failed
make[1]: *** [.kernel/include/linux/version.h] Error 2
make[1]: Leaving directory '/root/ply/src'
Makefile:402: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
$ vi src/Makefile
[...]
.kernel/include/linux/version.h:
    @echo "  HEADERS  /usr/src/linux-headers-4.8.0-30-generic(x86_64)"
#   make -C /usr/src/linux-headers-4.8.0-30-generic $(kernelenv) headers_install
[...]
$ make
$ sudo make install
$ ply -c 'kprobe:schedule { @cs[comm()].count(); }'
1 probe active
^Cde-activating probes

@cs:
gmain           	       1
watchdog/0      	       1
ply             	       3
kworker/0:1     	       4
kworker/u2:2    	       4
rcu_sched       	       5
iscsid          	       7
kworker/u2:0    	      10
sshd            	      10
swapper/0       	      18

So... Do we need the headers_install?

BEGIN

Some way to run code (eg, printf()s) at the start of the program. Useful for printing headings. awk has BEGIN, which is even used in the ply source already:

[...]
autom4te.cache/output.1:BEGIN {
config.status:ac_cs_awk_cr=`$AWK 'BEGIN { print "a\rb" }' </dev/null 2>/dev/null`
config.status:echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
config.status:BEGIN {
configure:ac_cs_awk_cr=`$AWK 'BEGIN { print "a\rb" }' </dev/null 2>/dev/null`
configure:echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
configure:BEGIN {
Makefile:  $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \
Makefile:  BEGIN { nonempty = 0; } \
Makefile.in:  $(AWK) 'BEGIN { files["."] = "" } { files[$$2] = files[$$2] " " $$1; \
Makefile.in:  BEGIN { nonempty = 0; } \
[...]

How to compile latest ply version 2.1.1 with custom kernel-dir?

Hi,

Ply looks like an awesome tool, and my embedded software team is very excited to try/use it.

We successfully cross-compiled the tool for arm-linux using arm-linux-androideabi-clang, and we pushed the tool to our embedded device.

The tool compiled correctly. i.e. the following two commands work on our arm embedded device:

./ply
error: no input
ply - Dynamic tracing utility

Usage:
  ply [options] <ply-text>
  ply [options] <ply-file>

Options:
  -c COMMAND     Run COMMAND in a shell, exit upon completion.
  -d             Enable debug output.
  -e             Exit after compiling.
  -h             Print usage message and exit.
  -k             Keep going in face of trace buffer overruns.
  -S             Show generated BPF.
  -v             Print version information.

./ply -v                                                   
ply 15c164b (linux-version:328450~5.3.2)

However, we are unable to run a basic ply test:

ply 'kprobe:do_sys_open { printf("%v(%v): %s\n", comm, uid, str(arg1)); }' -d

When running the above command, we get the following error:

info: creating kallsyms cache
warning: unable to create kallsyms cache: No such file or directory
error: unable to load kprobe:do_sys_open, errno:22
warning: was ply built against the running kernel?
ERR:-22

In ply 1.x, there was a configure param --with-kerneldir that devs could use to set a custom kernel distro (other than kernel version on host build machine).

In 2.1.1 it seems this option is unrecognized as part of the configure process.

configure: WARNING: unrecognized options: --with-kerneldir

How can we compile ply 2.1.1 for the kernel version used on our embedded device, rather than the kernel version of our local development linux machines?

We are using kernel v4.14 on embedded device, but version 4.15 on our local dev ubuntu machines.

Thanks,

Eric

Segmentation fault in ply_fparse due to null pointer dereference in node_append

Hello,

the file attached below causes a segmentation fault. It was found by fuzzing ply_fparse and can be reproduced by running ply with the file as an argument.
nullptr.zip

The issue seems to be in node_append in node.c:263. When the file is parsed, tail is NULL, so tail->prev is a NULL pointer dereference. Returning NULL if tail is NULL seems to fix the issue and ply then successfully reports an error with the following output:

[root@fuzz ply]# ./src/ply/.libs/ply ./nullptr
1: error: syntax error, unexpected IDENT, expecting ';'
ERR:-22

Regards,
Juraj

Does ply work on kernel 4.8?

I built ply cfc56c7 against Linux 4.8.4_1, but it doesn't seem to like the bpf code:

% sudo strace ./src/ply -c 'kprobe:SyS_open { $opens.count() }'
...
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=8, value_size=8, max_entries=512}, 48) = 3
open("/sys/kernel/debug/tracing/kprobe_events", O_RDWR|O_CREAT|O_APPEND, 0666) = 4
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=29, insns=0x1315558, license="GPL", log_level=1, log_size=131072, log_buf=0x611780, kern_version=262415}, 48) = -1 EINVAL (Invalid argument)

testing strings in predicates

# ./opensnoop.ply
ERR compile_pred        : predicate @path was not in a register as expected

source:

#!/usr/bin/env ply
/*
 * opensnoop    trace file opens.
 */

kprobe:do_sys_open
{
	# should just save ptr, pending issue #13
	@path[pid()] = mem(arg(1), "128s");
}

kretprobe:do_sys_open / @path[pid()] /
{
	if (retval() > 0) {
		fd = retval();
		errno = 0;
	} else {
		fd = -1;
		errno = -1 * retval();
	}

	printf("%-6d %-16s %4d %3d %s\n", pid(), comm(), fd, errno, @path[pid()]);
	@path[pid()] = nil;
}

don't enable all tracepoints

I noticed trace_pipe was full of tracepoint events after using ply; I think the problem is that it's enabling all tracepoints:

/sys/kernel/debug/tracing # for f in `find . -name enable`; do read i < $f; (( i )) && echo $f $i; done
[...]
./events/xen/xen_mmu_set_domain_pte/enable 1
./events/xen/xen_mmu_set_pte_at/enable 1
./events/xen/xen_mmu_pte_clear/enable 1
./events/xen/xen_mmu_set_pmd/enable 1
./events/xen/xen_mmu_pmd_clear/enable 1
./events/xen/xen_mmu_set_pud/enable 1
./events/xen/xen_mmu_set_pgd/enable 1
./events/xen/xen_mmu_pud_clear/enable 1
./events/xen/xen_mmu_pgd_clear/enable 1
./events/xen/xen_mmu_ptep_modify_prot_start/enable 1
./events/xen/xen_mmu_ptep_modify_prot_commit/enable 1
./events/xen/xen_mmu_alloc_ptpage/enable 1
./events/xen/xen_mmu_release_ptpage/enable 1
./events/xen/xen_mmu_pgd_pin/enable 1
./events/xen/xen_mmu_pgd_unpin/enable 1
./events/xen/xen_mmu_flush_tlb_all/enable 1
./events/xen/xen_mmu_flush_tlb/enable 1
./events/xen/xen_mmu_flush_tlb_single/enable 1
./events/xen/xen_mmu_flush_tlb_others/enable 1
./events/xen/xen_mmu_write_cr3/enable 1
./events/xen/xen_cpu_write_ldt_entry/enable 1
./events/xen/xen_cpu_write_idt_entry/enable 1
./events/xen/xen_cpu_load_idt/enable 1
./events/xen/xen_cpu_write_gdt_entry/enable 1
./events/xen/xen_cpu_set_ldt/enable 1
./events/xen/enable 1
./events/vsyscall/emulate_vsyscall/enable 1
./events/vsyscall/enable 1
./events/raw_syscalls/sys_enter/enable 1
./events/raw_syscalls/sys_exit/enable 1
./events/raw_syscalls/enable 1
/sys/kernel/debug/tracing # for f in `find . -name enable`; do read i < $f; (( i )) && echo $f $i; done|wc
   1804    3608   79468

Question: clear up map?

Hi there,

Before we can clear map with assignment 'nil', seems now the latest code base doesn't support 'nil' anymore. So e.g. below program cannot work.

kprobe:kmem_cache_alloc_node {                
        s[cpu,kpid] = stack();                
}                                             
                                              
kretprobe:kmem_cache_alloc_node {             
        @[retval] = s[cpu,kpid];              
        s[cpu,kpid] = nil;                    
}                                             
                                              
kprobe:kmem_cache_free {                      
        @[arg1] = nil;                        
}                                             

If I change 'nil' to 0, the program can run successfully. But the assignment for the sentence "@[retval] = s[cpu,kpid]" also cannot work as expected, I observe the result as below:

{ -281474254472384 }:
{ -281474254472000 }:
{ -281474231713792 }:
{ -281474177160960 }:
{ -281474148187360 }:
{ -281474147794048 }:
{ -281474147622912 }:
{ -281474132892352 }:
{ -281474132891456 }:
{ -281474132887872 }:
{ -281474132887424 }:
{ -281474132886528 }:

Thanks for the suggestion!

Plans for uprobe support?

My current understanding is that ply doesn't support uprobes yet. Is that planned, or is anyone working on it?

Looking at the providers, the support to add say profiling provide in kprobes was wonderfully concise. Might uprobes be similar, or is it a quite different undertaking? Apologies in advance for my lack of understanding there.

yacc requirements?

I'm trying to build ply on an in-house enterprise distribution which has a backported GCC but is still using a very old YACC (version 1.9 20070509); your parser doesn't compile with that version of YACC:

% make
Making all in src
make[1]: Entering directory `/home/build/pg/ply/src'
  YACC   lang/parse.c
/home/build/pg/ply/src/lang/parse.y:20.21-27: syntax error, unexpected identifier
make[1]: *** [lang/parse.c] Error 1
make[1]: Leaving directory `/home/build/pg/ply/src'
make: *** [all-recursive] Error 1

Do you know what version of YACC I would need to backport in order to get this to build?

Plans for x86 arch support?

Hi,

Can I use ply for i586 architecture? When I try to compile ply I see the follow error:

| make  all-recursive
| make[1]: Entering directory `/local/workspace/ply/2.1.0-r2'
| Making all in include
| make[2]: Entering directory `/local/workspace/ply/2.1.0-r2/include'
| make[2]: Nothing to be done for `all'.
| make[2]: Leaving directory `/local/workspace/ply/2.1.0-r2/include'
| Making all in src
| make[2]: Entering directory `/local/workspace/ply/2.1.0-r2/src'
| Making all in libply
| make[3]: Entering directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make  all-am
| make[4]: Entering directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make[4]: *** No rule to make target `arch/i586.c', needed by `arch/libply_la-i586.lo'.  Stop.
| make[4]: Leaving directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make[3]: *** [all] Error 2
| make[3]: Leaving directory `/local/workspace/ply/2.1.0-r2/src/libply'
| make[2]: *** [all-recursive] Error 1
| make[2]: Leaving directory `/local/workspace/ply/2.1.0-r2/src'
| make[1]: *** [all-recursive] Error 1
| make[1]: Leaving directory `/local/workspace/ply/2.1.0-r2'
| make: *** [all] Error 2
| autoreconf: make failed with exit status: 2