sysprogs / kgdboe

A network interface for GDB for Linux Kernel

KGDBoE - Kernel Debug over Ethernet

KGDBoE is a kernel module to debug Linux kernel on a bare metal the network connection. It is useful on modern PCs that don't have a serial port or a JTAG connector and it's much faster than using a COM port.

KGDBoE is inspired by the original kgdboe patches by Jason Wessel, but goes well beyond the capabilities of the original tool. The key features are:

• Works with stock kernels without rebuilding them. Tested on Linux Kernels: 3.8.0-5.15.0, 5.19.0, 6.3.x
• Easy configuration. No need to specify the IP or MAC addresses
• Supports modern multi-core systems
• It was tested on pcnet32, r8169 and e1000 network drivers

Build

git clone https://github.com/sysprogs/kgdboe.git
cd kgdboe
make -C /lib/modules/$(uname -r)/build M=$(pwd)

Load

sudo insmod kgdboe.ko kallsyms_lookup_name_address=0x$(sudo cat /proc/kallsyms | grep 'T kallsyms_lookup_name' | awk '{print $1}') device_name=eth0

Then, you should see log information whether kgdboe succeeded or failed to load in the output of sudo dmesg --follow.

Use (remote)

gdb /boot/vmlinuz
target remote udp:<IP>:31337

Note: it is best that the GDB client is running on Linux (an attempt to use MacOS did not work). Also, prefer to use latest GDB. GDB 13.1 should work.

Debug kernel module

Find out module address

sudo grep modulename /proc/modules
modulename 1073152 5 - Live 0xffffffffa0120000

In the gdb (remote)

add-symbol-file drivers/char/modulename.ko 0xffffffffa0120000

Integration with Visual Studio

For the most intuitive debugging experience, kernel source code navigation and tracing, check out VisualKernel.

Configuration / kernel module parameters

• kallsyms_lookup_name_address=0x... -- address of the kallsyms_lookup_name symbol
• device_name=eth0 -- Ethernet device to use for debugging.
• local_ip -- Local IP address to bind to. Auto-detected if not specified.
• udp_port=31337 -- UDP port to use for debugging.
• force_single_core=1 -- Disable all cores except #0 when the module is loaded. This setting is recommended unless you are debugging SMP-specific issues, as it avoids many synchronization problems. KGDBoE can reliably work in the SMP mode, but it has not been tested on all network drivers, so use caution if you decide to disable this.

Limitations

KGDBoE uses some of the network stack code to communicate with GDB. Setting breakpoints in the code that is used by it would deadlock your debugging session. Follow the tips below to avoid it:

• Don't set breakpoints in the network code
• Don't set breakpoints in mod_timer() unless you're using the single-CPU mode
• Use the single-CPU mode unless you absolutely need SMP. Although KGDBoE includes workarounds for multi-CPU mode, they are based on making assumptions about the network driver internals and can be safely avoided by disabling all CPUs except #0 during debugging.

License

The source code is available under the GPL license.

Source: http://sysprogs.com/VisualKernel/kgdboe/

kgdboe's Issues

__module_address undefined

Attempting to build kgdboe against 5.10.0-rc7 and I'm getting an undefined symbol error during modpost for "__module_address" Looking through git history it appears this patch in upstream Linux broke kgdboe:

commit 34e64705ad415ed7a816e60ef62b42fe6d1729d9
Author: Christoph Hellwig <[email protected]>
Date:   Thu Jul 30 08:10:24 2020 +0200

    modules: unexport __module_address

    __module_address is only used by built-in code.

    Signed-off-by: Christoph Hellwig <[email protected]>
    Signed-off-by: Jessica Yu <[email protected]>

diff --git a/kernel/module.c b/kernel/module.c
index 6ee1739..e85d061 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -4464,7 +4464,6 @@ struct module *__module_address(unsigned long addr)
        }
        return mod;
 }
-EXPORT_SYMBOL_GPL(__module_address);

 /*
  * is_module_text_address - is this address inside module code?

Kgdboe does not support CONFIG_KDB_KGDB

Hi,

TL;DR: Kgdboe does not support CONFIG_KGDB_KGDB. If this kernel config is enabled, a GDB connection to kgdboe will halt the target system. I haven't exhaustively analyzed this issue, but kdb makes kgdboe gdbstub to return junk kdb shell messages instead of speaking the expected gdbserver protocol.

Full story below.

I have been trying to run kgdboe against Ubuntu 22.04 (5.19.0-38-generic) running on ryzen 5950x cpu and on three different network cards and drivers:

• I211 Gigabit Network Connection with igb driver (firmware 0. 6-1)
• RTL8125 2.5GbE Controller with r8169 driver (firmware 72.daa05125.0)
• A "Satechi" ethernet/usb adapter with r8152 driver (driverversion=v1.12.13, firmeware=rtl8153a-4 v2 02/07/20)

For this, I build the latest kgdboe from this repo with a -D__NO_FORTIFY definition to compile it (to prevent kernel's fortify source mitigation errors, e.g., on memcpy calls on whole structs when the struct has padding etc.) as well as I signed the module with my own and enrolled MOK key (I have used this Ubuntu tutorial for that).

Then, all my attempts to run kgdboe ended up with my target system halting even though I used a tested network driver (r8169) and trying out different GDB versions on the host (12.1 on MacOS, 12.0.90 on Ubuntu an self compiled 13.1 on Ubuntu). For completeness, here is how I inserted kgdboe to the kernel:

sudo insmod kgdboe.ko kallsyms_lookup_name_address=0x$(sudo cat /proc/kallsyms | grep 'T kallsyms_lookup_name' | awk '{print $1}') device_name=enp5s0

For what is worth, I also ran the GDB with set debug remote 1 to investigate what is going on. Here is the full log from such run: https://gist.github.com/disconnect3d/0c4c68ea6ff72830aa581e0a1c9185b1#file-kgdboe-attempt-log

A friend of mine (Thx noodly from irc.gynvael.live) also tested kgdboe on his own Gentoo machine with 6.3.x kernel and it worked for him with a tested (e1000) network driver.

We then compared our kernel configs regarding KGDB. Here is the config from Ubuntu with kernel 5.19.0-38:

dc@dc:~$ cat /boot/config-5.19.0-38-generic | grep -i kgdb
CONFIG_SERIAL_KGDB_NMI=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_HONOUR_BLOCKLIST=y
CONFIG_KGDB_SERIAL_CONSOLE=y
# CONFIG_KGDB_TESTS is not set
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y

And here is the config from Gentoo with 6.3.x kernel - on which kgdboe worked fine:

# CONFIG_SERIAL_KGDB_NMI is not set
--
  | CONFIG_HAVE_ARCH_KGDB=y
  | CONFIG_KGDB=y
  | CONFIG_KGDB_HONOUR_BLOCKLIST=y
  | CONFIG_KGDB_SERIAL_CONSOLE=y
  | CONFIG_KGDB_TESTS=y
  | # CONFIG_KGDB_TESTS_ON_BOOT is not set
  | # CONFIG_KGDB_LOW_LEVEL_TRAP is not set
  | # CONFIG_KGDB_KDB is not set

Clearly, the configs differ. I went ahead and rebuild an unsigned Ubuntu kernel and changed my config to make it similar to my friend's. Here it is now:

dc@dc:~$ cat /boot/config-5.19.0-38-generic-unsigned | grep -i kgdb
CONFIG_SERIAL_KGDB_NMI=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_HONOUR_BLOCKLIST=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_TESTS=y
# CONFIG_KGDB_TESTS_ON_BOOT is not set
# CONFIG_KGDB_LOW_LEVEL_TRAP is not set
# CONFIG_KGDB_KDB is not set

On this unsigned kernel, then kgdboe works fine with the r8169 network driver (I haven't really tested others).

However, given all this testing as well as the GDB remote debug output, which said something like:

Entering kdb (current=0xfff..., pid 14) on processor 0 due to NonMaskable Interrupt @ 0xfff...
[0] kdb>
diag: 22: Permission Denied
[0] kdb> :multiprocess

Along with [remote] Junk: gqSupporteddiag: -22: Permission denied and timeouts at the end, I believe the problem results from kdb being enabled for kgdb. At least... disabling it as well as CONFIG_KGDB_LOW_LEVEL_TRAP worked for me, though, I am not sure if CONFIG_KGDB_LOW_LEVEL_TRAP changes anything.

---

With all this, I suggest that kgdboe should error out to compile if KGDB_KDB is enabled. I will send a PR with such a change.

Module fails to load with 'Missing tracepoints' error

I built a custom kernel with KGDB and CONFIG_TRACEPOINTS enabled, and also built KGDBoE from that tree. When loading the module, i get an error like
kgdboe: Missing tracepoints for netif_receive_skb. Aborting

It is unclear to me why this is happening, the __tracepoint_netif_receive_skb symbol seems to exist in the kernel image, but cannot be found by kallsyms_lookup_name

I tried both a recent master-branch (4.20-rc2), and the 4.15 release tag (because there's a PR that mentions it as working).

Error during insmod

On dmesg i see this error:
kgdboe: this kernel version requires specifying the address of kallsyms_lookup_name explicitly. Please append 'kallsyms_lookup_name_address=0x...' to insmod command line.
I'm not sure of what I'm supposed to do... How can I locate that address?

My linux version:
Linux ubuntu 5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

GDB connection fails with "Remote replied unexpectedly to 'vMustReplyEmpty': timeout"

(note actual IP address replaced with X.X.X.X)

insmod succeeds and dmesg log looks good:

[ 1275.281811] kgdboe: loading out-of-tree module taints kernel.
[ 1275.281811] kgdboe: module verification failed: signature and/or required key missing - tainting kernel
[ 1275.298549] netpoll: kgdboe: local IP X.X.X.X
[ 1275.298549] kgdboe: single-core mode enabled. Shutting down all cores except #0. This is slower, but safer.
[ 1275.298549] kgdboe: you can try using multi-core mode by specifying the following argument:
[ 1275.298549]  insmod kgdboe.ko force_single_core = 0
[ 1275.339718] KGDB: Registered I/O driver kgdboe
[ 1275.358895] kgdboe: Successfully initialized. Use the following gdb command to attach:
[ 1275.358895]  target remote udp:X.X.X.X:31337

But when I run target remote udp:X.X.X.X:31337 it fails with the following:

Remote debugging using udp:X.X.X.X:31337
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout

I don't have a great idea of how I can fix this.

GDB version is 8.1
Kernel version is 5.6.0

Compile Error With 4.14 preempt-rt Kernel with

getting build error with 4.14 preempt-rt Kernel

make: Entering directory '/opt/PAOS/paos/build/tmp/work/n03-poky-linux/linux-yocto-rt/4.14.67+gitAUTOINC+c43c9e19a2_af1b926c91-r0/linux-n03-preempt-rt-build'
CC [M] /home/zaki/Workspace/Playground/kgdboe/nethook.o
/home/zaki/Workspace/Playground/kgdboe/nethook.c: In function ‘nethook_initialize’:
/home/zaki/Workspace/Playground/kgdboe/nethook.c:224:83: error: ‘spinlock_t {aka struct spinlock}’ has no member named ‘rlock’; did you mean ‘lock’?
if (!hook_spinlock(nethook.spinhook, &timerhook_get_spinlock(nethook.timerhook)->rlock))
^~~~~
lock
/home/zaki/Workspace/Playground/kgdboe/nethook.c:231:64: error: ‘spinlock_t {aka struct spinlock}’ has no member named ‘rlock’; did you mean ‘lock’?
if (!hook_spinlock(nethook.spinhook, &nethook.netdev_api_lock.rlock))
^~~~~
lock
/home/zaki/Workspace/Playground/kgdboe/nethook.c:241:81: error: ‘spinlock_t {aka struct spinlock}’ has no member named ‘rlock’; did you mean ‘lock’?
if (!hook_spinlock(nethook.spinhook, &netdev_get_tx_queue(dev, i)->_xmit_lock.rlock))
^~~~~
lock
/home/zaki/Workspace/Playground/kgdboe/nethook.c:251:60: error: ‘spinlock_t {aka struct spinlock}’ has no member named ‘rlock’; did you mean ‘lock’?
if (!hook_spinlock(nethook.spinhook, &jiffies_lock->lock.rlock))
^~~~~
lock
/opt/PAOS/paos/build/tmp/work-shared/n03/kernel-source/scripts/Makefile.build:328: recipe for target '/home/zaki/Workspace/Playground/kgdboe/nethook.o' failed
make[3]: *** [/home/zaki/Workspace/Playground/kgdboe/nethook.o] Error 1
/opt/PAOS/paos/build/tmp/work-shared/n03/kernel-source/Makefile:1523: recipe for target 'module/home/zaki/Workspace/Playground/kgdboe' failed
make[2]: *** [module/home/zaki/Workspace/Playground/kgdboe] Error 2
Makefile:146: recipe for target 'sub-make' failed
make[1]: *** [sub-make] Error 2
Makefile:24: recipe for target '__sub-make' failed
make: *** [__sub-make] Error 2
make: Leaving directory '/opt/PAOS/paos/build/tmp/work/n03-poky-linux/linux-yocto-rt/4.14.67+gitAUTOINC+c43c9e19a2_af1b926c91-r0/linux-n03-preempt-rt-build'

Potential to upstream

I want to understand what is keeping this work from being upstreamed into the mainline kernel.

I did some digging and found it was proposed in patches many years ago but I couldn't find why it was never merged. Are the problems technical? Philosophical?

I think this module is great and it would be awesome if it could eventually be officially supported upstream. I would be interested in contributing to this effort if to limitation is just technical.

Fails with "Packet error"

I built the module and loaded it successfully.

# insmod kgdboe.ko device_name=enp2s0f0
# dmesg | tail
...
[  127.055363] kgdb: Registered I/O driver kgdboe.
[  127.055450] kgdboe: Successfully initialized. Use the following gdb command to attach:
[  127.055454]  target remote udp:10.5.221.4:31337

However, when trying to connect with gdb I'm getting packet errors and debug session does not start:

(gdb) target remote udp:10.5.221.4:31337
warning: The remote protocol may be unreliable over UDP.
Some events may be lost, rendering further debugging impossible.
Remote debugging using udp:10.5.221.4:31337
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
warning: Invalid remote reply: timeout

---

OS and NIC details:

Linux lb-tlvb-emu6 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# lshw -class network
  *-network:0
       description: Ethernet interface
       product: I350 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: enp2s0f0
       version: 01
       serial: 00:1e:67:9b:10:c2
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.4.0-k duplex=full firmware=1.48, 0x80000707 ip=10.5.221.4 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:33 memory:d1160000-d117ffff ioport:2060(size=32) memory:d11b0000-d11b3fff memory:d12a0000-d12bffff memory:d1280000-d129ffff

Build fails for kernel 4.16

uname -r:
4.16.0-2-amd64
make log:
makelog.txt

EDIT:
nevermind, My kernel didn't have kgdb compiled in.

insmod gets invalid parameters error

I was creating a "Hello World" kernel module project using a customized kernel built using VisualKernel. The kernel version is 4.15.18. Then I was trying to debug the kernel module using kgdboe. At the beginning the kgdboe has build error but after using the most updated source files of kgdboe from here the build passed. But then when the kgdboe.ko was being loaded an error reported saying

"insmod: ERROR: could not insert module kgdboe.ko: Invalid parameters".

The following log message is obtained from dmesg command after loading kgdboe.ko

[ 6641.067703] netpoll: kgdboe: local IP 10.0.0.20
[ 6641.067714] netpoll: kgdboe: wlp3s0 doesn't support polling, aborting
[ 6641.067717] kgdboe: Failed to setup netpoll for wlp3s0, code -524
[ 6966.810172] netpoll: kgdboe: local IP 10.0.0.20
[ 6966.810174] netpoll: kgdboe: wlp3s0 doesn't support polling, aborting
[ 6966.810177] kgdboe: Failed to setup netpoll for wlp3s0, code -524
[ 7043.346409] kgdboe: Cannot find network device by name: eth0
[ 9780.671684] kgdboe: Cannot find network device by name: eth0

Ignoring packet error, continuing...

(gdb) target remote udp:192.162.8.116:31337
warning: The remote protocol may be unreliable over UDP.
Some events may be lost, rendering further debugging impossible.
Remote debugging using udp:192.162.8.116:31337
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
Ignoring packet error, continuing...
warning: Invalid remote reply: timeout

target:
insmod kgdboe.ko device_name=eth0 kallsyms_lookup_name_address=

Supported on 5.x kernels?

We're able to successfully attach using a 4.14 kernel but 5.4 and 5.6 kernels fail the handshake during attach. The kgdboe module seems to load fine on the 5.x kernels and gdb triggers the breakpoint on the target node but the following messages appears in gdb:

Remote replied unexpectedly to 'vMustReplyEmpty': timeout

This leaves the target node hung because it's at the kgdb breakpoint and we can't issue the command to resume.