bpftrace not able to attack kprobe with nf_nat with kernel 6.1.66-1 about bpftrace

[pkopensrc]

As attached It was working fine till 6.1.55 and broken recently.
It it was inline it shouldn't show in bpftrace -l.

Here is output of llvm-dwarfdump:
root@pk-ntest-12new:/boot# llvm-dwarfdump-14 vmlinuz-6.1.0-15-cloud-amd64 | grep -A 5 DW_TAG_inlined_subroutine | grep -C 1 nf_nat_ipv4_manip_pkt
root@pk-ntest-12new:/boot#

No changes in function.
static bool nf_nat_ipv4_manip_pkt(struct sk_buff *skb,
unsigned int iphdroff,
const struct nf_conntrack_tuple *target,
enum nf_nat_manip_type maniptype)

[pkopensrc]

@jordalgo here is output which is same in both.
Debian 6.1.55-1

debian:~$ sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep "nf_nat_ipv4_manip_pkt"
nf_nat_ipv4_manip_pkt [nf_nat]
debian:~$ uname -a
Linux connector1 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
debian~$

Debian - 6.1.66-1

~$ sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep "nf_nat_ipv4_manip_pkt"
nf_nat_ipv4_manip_pkt [nf_nat]
~$ uname -a
Linux  6.1.0-15-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.66-1 (2023-12-09) x86_64 GNU/Linux
~$

[jordalgo]

I use modprobe to load kernel modules: https://opensource.com/article/18/5/how-load-or-unload-linux-kernel-module
I believe it has to do with kconfig settings that determines which ones get loaded by default.

Can i use tracepoints with bpf and use bpf maps?

Sure, but there might not be 1 to 1 matching of kprobe to tracepoint. You'll probably have to dig around in the kernel code to see if there is something you can use.

[jordalgo]

My first guess is that function was inlined. Can you try running this against both vmlinux binaries?
llvm-dwarfdump /path/to/vmlinux | grep -A 5 DW_TAG_inlined_subroutine | grep -C 1 nf_nat_ipv4_manip_pkt

[danobi]

Hmm, weird. You've already established that it's the change in kernel - not in bpftrace. That is good information to know. But nothing obvious stands out to me when looking at nf_nat_ipv4_manip_pkt in master, 6.1, or 6.0. I'm afraid the only way I know to debug this is basically to printk debug the kernel or start bisecting kernels. Neither of which we can really help you with from here.

Bisecting might be easier if you're not familiar with the kernel codepaths in question.

[jordalgo]

@pkopensrc Are you able to duplicate this behavior with a raw bcc or libbpf program? Then at least we'd know if it is isolated to bpftrace.

[pkopensrc]

@jordalgo Libbpf i tried and it failed. I don't know how to use bcc for same.

@danobi
I am not a kernel expert and don't know how to compile kernel and trying my hands in bpf.

As given in bug output in kernel 6.1.55 it works fine using bpftrace.
I have 2 gcloud VM with different debian and saw issue.

I am not sure its issue with bpftrace or kernel itself.
Basic libbpf program works fine.

If its not bpftrace issue How to raise this issue with kernel team?
I tried to post query to netfilter-dev team but didn't got any respond.

[jordalgo]

Libbpf i tried and it failed

Ah interesting. I wonder if the bug is in bpftrace's listing of kprobes that are not actually available to attach to. I'll see if I can dig in a bit more this weekend.

[pkopensrc]

@jordalgo
Yes that can be also true. If its bug in listing of kprobes. Can you please check that.

But i am getting confused on how it was working in 1.55 and not in 1.66 means something changed recently.
A Kernel developer with an expertise in bpftrace can tell us why its not working.

[jordalgo]

@pkopensrc Mind trying one more thing, can you run this on both kernel versions?
sudo cat /sys/kernel/debug/tracing/available_filter_functions | grep "nf_nat_ipv4_manip_pkt"

[pkopensrc]

Just to update.
I was trying to check same in latest ubuntu kernel and seen same issue there with kernel 6.2.0.
even bpftrace -l gives empty output.

Here is output:

# uname -a
Linux 6.2.0-1019-gcp #21~22.04.1-Ubuntu SMP Thu Nov 16 18:18:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# bpftrace -e 'kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
stdin:1:1-29: WARNING: nf_nat_ipv4_manip_pkt is not traceable (either non-existing, inlined, or marked as "notrace"); attaching to it will likely fail
kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Attaching 1 probe...
cannot attach kprobe, probe entry may not exist
ERROR: Error attaching probe: 'kprobe:nf_nat_ipv4_manip_pkt'
#

[jordalgo]

Well this is definitely a strange one. I'm not able to repro this behavior on ubuntu, centos, fedora, or debian (6.1.0-13-arm64, which is after loading the nf_nat kernel module).

[pkopensrc]

Maybe you can try latest kernel 6.1.66 after Dec 12th release

[jordalgo]

Ok, I was finally able to repro. Also asked around and it might be that this patch needs to get backported to fix this issue:
https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/

[pkopensrc]

Thank you for finding out.
but it has to go to main stream kernel. Otherwise cloud providers will not work.

One more strange thing was if you see latest kernel Linux 6.2.0-1019-gcp even nat symbols are not present in bpftrace.

# bpftrace -e 'kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }'
stdin:1:1-29: WARNING: nf_nat_ipv4_manip_pkt is not traceable (either non-existing, inlined, or marked as "notrace"); attaching to it will likely fail
kprobe:nf_nat_ipv4_manip_pkt { printf("func called\n"); }
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Attaching 1 probe...
cannot attach kprobe, probe entry may not exist
ERROR: Error attaching probe: 'kprobe:nf_nat_ipv4_manip_pkt'
#

When i asked in dev channel they said kprobe/kfuncs are unreliable and unstable and can change anytime.

[jordalgo]

It was just merged into 6.1 stable:
https://lore.kernel.org/stable/[email protected]/T/#u

One more strange thing was if you see latest kernel Linux 6.2.0-1019-gcp even nat symbols are not present in bpftrace.

Did you load the nt_nat module?

kprobe/kfuncs are unreliable and unstable and can change anytime

This is true and oftentimes why people choose to use tracepoints instead.

[pkopensrc]

No how to load nf_nat module? I was thinking it will always load by default.
Generally in 6.1.55 i don't have to load anything it used to come by default.

Can i use tracepoints with bpf and use bpf maps?
Basically i want to tap to get source ip and natted ip from function after iptables nat the connection while routing the packet.