Support capturing via fentry / fexit trace points about xdpcap

[wenlxie]

I did a try to do implementation for this.
Here are the steps:

• Get the prog info from id by NewProgramFromID()
  The id is the xdp program's prog id
  p, err := NewProgramFromID(id)
• Then I set the program spec (which used to load the program)to be:

 	progSpec.Name = "fentry"
	progSpec.AttachTarget =  xdp program get by NewProgramFromID(id)
	progSpec.AttachTo = xdp program's name
	progSpec.AttachType = ebpf.AttachTraceFEntry

• The key part is for how to generate the instructions for the progSpec.Instructions
  The previous asm code are generated for xdp programs with xdp_md as args.
  But for fentry to hook xdp programs, the input arg had been changed to xdp_buff, which means the asm code need to be changed.
  At least that this need to be changed:

  xdpcap/cmd/xdpcap/program.go

  Lines 96 to 99 in 417804f

  	asm.LoadMem(asm.R0, asm.R6, 0, asm.Word),
  	// Packet end
  	asm.LoadMem(asm.R1, asm.R6, 4, asm.Word),

• I tried to change the asm code to make it works for xdp_buff, but got stucked.

I changed to asm code for the data and data_end to be :

		// Packet start
		asm.LoadMem(asm.R9, asm.R6, 0, asm.DWord),
		asm.LoadMem(asm.R0, asm.R9, 0, asm.DWord),

		// Packet end
		asm.LoadMem(asm.R1, asm.R9, 8, asm.DWord),

But I got error

permission denied: 21: (69) r2 = *(u16 *)(r0 +12): R0 invalid mem access 'inv' (26 line(s) omitted)

@arthurfabre Any thoughts for this issue? Could you help for how to make these asm codes works for xdp_buff? Thanks
Looks like fentry has more strict verify than xdp

Here are the asm instructions:

 0: MovReg dst: r6 src: r1
	  1: LoadMapPtr dst: r1 fd: 23
	  3: MovReg dst: r2 src: rfp
	  4: AddImm dst: r2 imm: -4
	  5: StMemW dst: r2 src: r0 off: 0 imm: 0
	  6: Call FnMapLookupElem
	  7: JEqImm dst: r0 off: -1 imm: 0 <exit>
	  8: MovReg dst: r7 src: r0
	  9: LdXMemDW dst: r9 src: r6 off: 0 imm: 0
	 10: LdXMemDW dst: r0 src: r9 off: 0 imm: 0
	 11: LdXMemDW dst: r1 src: r9 off: 8 imm: 0
	 12: MovReg dst: r8 src: r1
	 13: SubReg dst: r8 src: r0
	 14: JLEImm dst: r8 off: -1 imm: 12 <exit>
	 15: LdXMemDW dst: r2 src: r7 off: 0 imm: 0
	 16: AddImm dst: r2 imm: 1
	 17: StXMemDW dst: r7 src: r2 off: 0 imm: 0
filter_block_0:
	 18: MovReg dst: r4 src: r0
	 19: AddImm dst: r4 imm: 21
	 20: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 21: LdXMemH dst: r2 src: r0 off: 12 imm: 0
	 22: SwapBE dst: r2 imm: 16
	 23: JNEImm dst: r2 off: -1 imm: 34525 <filter_block_7>
filter_block_2:
	 24: LdXMemB dst: r2 src: r0 off: 20 imm: 0
	 25: JEqImm dst: r2 off: -1 imm: 6 <filter_block_10>
filter_block_4:
	 26: JNEImm dst: r2 off: -1 imm: 44 <filter_block_11>
filter_block_5:
	 27: MovReg dst: r4 src: r0
	 28: AddImm dst: r4 imm: 55
	 29: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 30: LdXMemB dst: r2 src: r0 off: 54 imm: 0
	 31: JEqImm dst: r2 off: -1 imm: 6 <filter_block_10>
	 32: JaImm dst: r0 off: -1 imm: 0 <filter_block_11>
filter_block_7:
	 33: JNEImm dst: r2 off: -1 imm: 2048 <filter_block_11>
filter_block_8:
	 34: MovReg dst: r4 src: r0
	 35: AddImm dst: r4 imm: 24
	 36: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 37: LdXMemB dst: r2 src: r0 off: 23 imm: 0
	 38: JNEImm dst: r2 off: -1 imm: 6 <filter_block_11>
filter_block_10:
	 39: Mov32Imm dst: r2 imm: 1
	 40: JaImm dst: r0 off: -1 imm: 0 <result>
filter_block_11:
	 41: Mov32Imm dst: r2 imm: 0
	 42: JaImm dst: r0 off: -1 imm: 0 <result>
filter_nomatch:
	 43: MovImm dst: r2 imm: 0
	 44: JaImm dst: r0 off: -1 imm: 0 <result>
result:
	 45: JEqImm dst: r2 off: -1 imm: 0 <exit>
	 46: LdXMemDW dst: r0 src: r7 off: 8 imm: 0
	 47: AddImm dst: r0 imm: 1
	 48: StXMemDW dst: r7 src: r0 off: 8 imm: 0
	 49: MovReg dst: r1 src: r6
	 50: LoadMapPtr dst: r2 fd: 3
	 52: MovReg dst: r3 src: r8
	 53: LShImm dst: r3 imm: 32
	 54: LdImmDW dst: r0 imm: 4294967295
	 56: OrReg dst: r3 src: r0
	 57: MovReg dst: r4 src: rfp
	 58: AddImm dst: r4 imm: -8
	 59: StXMemDW dst: r4 src: r8 off: 0 imm: 0
	 60: AddImm dst: r4 imm: -8
	 61: StMemDW dst: r4 src: r0 off: 0 imm: 1
	 62: MovImm dst: r5 imm: 16
	 63: Call FnPerfEventOutput
	 64: JEqImm dst: r0 off: -1 imm: 0 <exit>
	 65: LdXMemDW dst: r0 src: r7 off: 16 imm: 0
	 66: AddImm dst: r0 imm: 1
	 67: StXMemDW dst: r7 src: r0 off: 16 imm: 0
exit:
	 68: MovImm dst: r0 imm: 1
	 69: Exit

[wenlxie]

@ptzianos Could you also help to take a look. Thanks

[arthurfabre]

Thanks for giving it a go! Looking at the definition of xdp_buff (https://elixir.bootlin.com/linux/latest/source/include/net/xdp.h#L81), data is a normal pointer to the packet, you don't need to immediately dereference it (it happens later on every time we access the packet).

I think the only difference between xdp_buff and xdp_md is that xdp_buff uses 64 bit pointers on a 64 bit system but xdp_md' always uses 32 bits. Maybe something like this works on a 64 bit system:

 // Packet start
 asm.LoadMem(asm.R0, asm.R6, 0, asm.DWord), 
  
 // Packet end 
 asm.LoadMem(asm.R1, asm.R6, 8, asm.DWord), 

[wenlxie]

Thanks for giving it a go! Looking at the definition of xdp_buff (https://elixir.bootlin.com/linux/latest/source/include/net/xdp.h#L81), data is a normal pointer to the packet, you don't need to immediately dereference it (it happens later on every time we access the packet).

I think the only difference between xdp_buff and xdp_md is that xdp_buff uses 64 bit pointers on a 64 bit system but xdp_md' always uses 32 bits. Maybe something like this works on a 64 bit system:

 // Packet start
 asm.LoadMem(asm.R0, asm.R6, 0, asm.DWord), 
  
 // Packet end 
 asm.LoadMem(asm.R1, asm.R6, 8, asm.DWord), 

@arthurfabre
Actually I tried this, then I got the error: func 'xxxxxxx' doesn't have 2-th argument

Trace type prog should have the r1 for the address to arg arrays, which is different with xdp prog.

[wenlxie]

@arthurfabre

Add more details for the issue I met now:

1. My tcp filter expr is just "tcp"

2. In the genenrated cbpf to ebpf codes, it failed to pass the verifier in kernel, the error is: permission denied: 21: (69) r2 = *(u16 *)(r0 +12): R0 invalid mem access 'inv' (26 line(s) omitted)

3. What I changed is to change the asm code to read the xdp_buff->data and xdp_buff->data_end

		// Packet start
		asm.LoadMem(asm.R9, asm.R6, 0, asm.DWord),
		asm.LoadMem(asm.R0, asm.R9, 0, asm.DWord),

		// Packet end
		asm.LoadMem(asm.R1, asm.R9, 8, asm.DWord),

5. The error code part is it is try to check for the type field in the mac header.

	 19: AddImm dst: r4 imm: 21
	 20: JGTReg dst: r4 off: -1 src: r1 <filter_nomatch>
	 21: LdXMemH dst: r2 src: r0 off: 12 imm: 0

Since there are already check code ( data + 21 > data_end) before load the data in data+ 12 , so I'd think the generated asm code is fine

6. So I still think this may related with the code that I changed to get the xdp_buff->data and xdp_buff->data_end
   But not sure for how to change for this part.

7. I checked the xdp-dump tools,
   I tried xdp-dump and xlated the trace program that used to hook for xdp prog

When fetch for data and data_end, there is always a convert to long operation. Then I guess maybe this is the issue why I got the error.

I tried to check for the xlated code the trace prograom, I got following codes:

nt trace_on_entry(unsigned long long * ctx):
; int BPF_PROG(trace_on_entry, struct xdp_buff *xdp)
   0: (79) r1 = *(u64 *)(r1 +0)
; void *data = (void *)(long)xdp->data;
   1: (79) r3 = *(u64 *)(r1 +0)
; void *data_end = (void *)(long)xdp->data_end;
   2: (79) r2 = *(u64 *)(r1 +8)
; if (data >= data_end ||
   3: (3d) if r3 >= r2 goto pc+38
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   4: (79) r4 = *(u64 *)(r1 +32)
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   5: (79) r4 = *(u64 *)(r4 +0)
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   6: (61) r5 = *(u32 *)(r4 +208)
; trace_cfg.capture_if_ifindex != xdp->rxq->dev->ifindex)
   7: (18) r4 = map[id:2635][0]+0
   9: (61) r0 = *(u32 *)(r4 +0)
; if (data >= data_end ||
  10: (5d) if r0 != r5 goto pc+31
; metadata.prog_index = trace_cfg.capture_prog_index;
  11: (61) r5 = *(u32 *)(r4 +8)
; metadata.prog_index = trace_cfg.capture_prog_index;
  12: (6b) *(u16 *)(r10 -10) = r5
; metadata.ifindex = xdp->rxq->dev->ifindex;
  13: (79) r5 = *(u64 *)(r1 +32)
; metadata.pkt_len = (__u16)(data_end - data);
  14: (1f) r2 -= r3
; metadata.ifindex = xdp->rxq->dev->ifindex;
  15: (79) r3 = *(u64 *)(r5 +0)
; metadata.ifindex = xdp->rxq->dev->ifindex;
  16: (61) r3 = *(u32 *)(r3 +208)
; metadata.ifindex = xdp->rxq->dev->ifindex;
  17: (63) *(u32 *)(r10 -24) = r3
; metadata.cap_len = min(metadata.pkt_len, trace_cfg.capture_snaplen);
  18: (61) r3 = *(u32 *)(r4 +4)
  19: (bf) r4 = r2
  20: (57) r4 &= 65535
  21: (2d) if r3 > r4 goto pc+1
  22: (bf) r4 = r3
; metadata.rx_queue = xdp->rxq->queue_index;
  23: (79) r5 = *(u64 *)(r1 +32)
; ((__u64) metadata.cap_len << 32) |
  24: (bf) r3 = r4
  25: (67) r3 <<= 32
  26: (18) r0 = 0xffffffff
; ((__u64) metadata.cap_len << 32) |
  28: (4f) r3 |= r0
; metadata.rx_queue = xdp->rxq->queue_index;
  29: (61) r5 = *(u32 *)(r5 +8)
; metadata.pkt_len = (__u16)(data_end - data);
  30: (6b) *(u16 *)(r10 -16) = r2
  31: (b7) r2 = 0
;
  32: (6b) *(u16 *)(r10 -12) = r2
; metadata.action = action;
  33: (63) *(u32 *)(r10 -8) = r2
; metadata.cap_len = min(metadata.pkt_len, trace_cfg.capture_snaplen);
  34: (6b) *(u16 *)(r10 -14) = r4
; metadata.rx_queue = xdp->rxq->queue_index;
  35: (63) *(u32 *)(r10 -20) = r5
  36: (bf) r4 = r10
; metadata.rx_queue = xdp->rxq->queue_index;
  37: (07) r4 += -24
; bpf_xdp_output(xdp, &xdpdump_perf_map,
  38: (18) r2 = map[id:2634]
  40: (b7) r5 = 20
  41: (85) call bpf_xdp_event_output#8687200
; int BPF_PROG(trace_on_entry, struct xdp_buff *xdp)
  42: (b7) r0 = 0
  43: (95) exit

Look like the long convert is actually no needed.

So got a bit confused here.

@arthurfabre Any insights for this? Appreciate that if you can some time and have a check.

[wenlxie]

I also genearted the cbpf to c code

// True if packet matches, false otherwise
__attribute__((__always_inline__)) static inline
uint32_t test(const uint8_t *const data, const uint8_t *const data_end) {
	__attribute__((unused))
	uint32_t a, x, m[16];
	__attribute__((unused))
	const uint8_t *indirect;


block_0:
__attribute__((unused));
	if (data + 21 > data_end) return 0;
	a = ntohs(*((uint16_t *) (data + 12)));
	if (a != 34525) goto block_7;

block_2:
__attribute__((unused));
	if (data + 21 > data_end) return 0;
	a = *(data + 20);
	if (a == 6) goto block_10;

block_4:
__attribute__((unused));
	if (a != 44) goto block_11;

block_5:
__attribute__((unused));
	if (data + 55 > data_end) return 0;
	a = *(data + 54);
	if (a == 6) goto block_10; else goto block_11;

block_7:
__attribute__((unused));
	if (a != 2048) goto block_11;

block_8:
__attribute__((unused));
	if (data + 24 > data_end) return 0;
	a = *(data + 23);
	if (a != 6) goto block_11;

block_10:
__attribute__((unused));
	return 1;

block_11:
__attribute__((unused));
	return 0;

}

The block0 can't pass the ebpf verify

block_0:
__attribute__((unused));
	if (data + 21 > data_end) return 0;
	a = ntohs(*((uint16_t *) (data + 12)));
	if (a != 34525) goto block_7;