Fix for 5.7+ Linux kernel and example code for ch13 percpu_var.c about linux-kernel-programming

[kaiwan]

Hey @brian-wood !
Apologies for the delay... just saw this.
Excellent work-around (for a book at least right), thank you.

Y'know, am using this very same approach for the 2nd Ed...
Thanks,
Kaiwan.

[kaiwan]

This is the commit:
7f04301

[kaiwan]

Hi Kaiwan,

I was working through the example for ch13 "percpu_var" module and noticed that since 5.7+ kernel's the kallsyms_lookup_name() is no longer exported (I'm using latest stable of 6.4.9 for all the examples in the book). Anyway, I came up with a workaround that solves the issue (it's kind of hacky), but thought you might get a kick out of it. :-)

/* HACK: To we can't easily use the kallsyms_lookup_name() on 5.7+ Linux kernels

• since this function no longer has EXPORT_SYMBOL() support, so needed an
• alternate workaround (using a module parameter seemed to be the quickest solution).
  */
  #if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 7, 0)
  static unsigned long kallsyms_sched_setaffinity;
  module_param(kallsyms_sched_setaffinity, ulong, 0660);
  MODULE_PARM_DESC(kallsyms_sched_setaffinity, "With 5.7+ Linux kernel's we need to use alternate method for"
  " instead of using kallsyms_lookup_name() for a pointer to sched_setaffinity(). To use kernel module"
  " load passing in the kallsyms_sched_setaffinity module parameter:\n"
  " $ insmod percpu_var.ko kallsyms_sched_setaffinity=0x$(sudo grep -w 'sched_setaffinity$' /proc/kallsyms | cut -d' ' -f1)\n");
  #endif

Then down in the module init function 'init_percpu_var' I added the Linux version conditional check:

    /* Following line won't work for 5.7+ kernels since this function no longer has
     * EXPORT_SYMBOL support:
     *      ptr_sched_setaffinity = (void *)kallsyms_lookup_name("sched_setaffinity");
     * Possible workaround in the code below, see the module parameter at the
     * beginning of this file (or use 'modinfo percpu_var.ko' from cmdline) to see it's use.
     */
    ret = -ENOSYS;

#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 7, 0) ptr_sched_setaffinity = (void *)kallsyms_sched_setaffinity; #else ptr_sched_setaffinity = (void *)kallsyms_lookup_name("sched_setaffinity"); #endif

With these changes I'm able to run the driver as expected and get the dmesg results (I did make a few alterations to the values displayed for tx/rx increments just so I could see the changes a little easier):

dmesg of insmod (looks like my 3 CPU virtual machine was interleaving output I guess compared to screenshot from book :-) ): [ 390.116111] percpu_var:init_percpu_var(): inserted [ 390.116584] percpu_var:thrd_work(): *** kthread PID 11503 on cpu 0 now *** [ 390.116868] percpu_var:thrd_work(): thrd_0/cpu0: pcpa = +1 [ 390.117101] percpu_var:thrd_work(): thrd_0/cpu0: pcp ctx: tx = 100, rx = 10 [ 390.117408] percpu_var:thrd_work(): thrd_0/cpu0: pcpa = +2 [ 390.117556] percpu_var:thrd_work(): *** kthread PID 11504 on cpu 1 now *** [ 390.117653] percpu_var:thrd_work(): thrd_0/cpu0: pcp ctx: tx = 200, rx = 20 [ 390.117660] percpu_var:thrd_work(): thrd_0/cpu0: pcpa = +3 [ 390.117664] percpu_var:thrd_work(): thrd_0/cpu0: pcp ctx: tx = 300, rx = 30 [ 390.117824] percpu_var:thrd_work(): thrd_1/cpu1: pcpa = -1 [ 390.117828] percpu_var:disp_vars(): 000) [thrd_0/0]:11503 | .N.0 /* disp_vars() / [ 390.118002] percpu_var:thrd_work(): thrd_1/cpu1: pcp ctx: tx = 20, rx = 200 [ 390.118007] percpu_var:disp_vars(): cpu 0: pcpa = +3, rx = 30, tx = 300 [ 390.118138] percpu_var:thrd_work(): thrd_1/cpu1: pcpa = -2 [ 390.118140] percpu_var:thrd_work(): thrd_1/cpu1: pcp ctx: tx = 40, rx = 400 [ 390.118143] percpu_var:thrd_work(): thrd_1/cpu1: pcpa = -3 [ 390.118432] percpu_var:disp_vars(): cpu 1: pcpa = -3, rx = 400, tx = 40 [ 390.118440] percpu_var:disp_vars(): cpu 2: pcpa = +0, rx = 0, tx = 0 [ 390.118446] percpu_var:thrd_work(): Our kernel thread #0 exiting now... [ 390.120060] percpu_var:thrd_work(): thrd_1/cpu1: pcp ctx: tx = 60, rx = 600 [ 390.120238] percpu_var:disp_vars(): 001) [thrd_1/1]:11504 | .N.0 / disp_vars() */ [ 390.120427] percpu_var:disp_vars(): cpu 0: pcpa = +3, rx = 30, tx = 300 [ 390.120597] percpu_var:disp_vars(): cpu 1: pcpa = -3, rx = 600, tx = 60 [ 390.120767] percpu_var:disp_vars(): cpu 2: pcpa = +0, rx = 0, tx = 0 [ 390.120937] percpu_var:thrd_work(): Our kernel thread #1 exiting now...

dmesg of rmmod: [ 501.720553] percpu_var:exit_percpu_var(): kthread #0 stopped [ 501.720703] percpu_var:exit_percpu_var(): kthread #1 stopped [ 501.720839] percpu_var:disp_vars(): 001) rmmod :14779 | ...0 /* disp_vars() */ [ 501.721023] percpu_var:disp_vars(): cpu 0: pcpa = +3, rx = 30, tx = 300 [ 501.721194] percpu_var:disp_vars(): cpu 1: pcpa = -3, rx = 600, tx = 60 [ 501.721661] percpu_var:disp_vars(): cpu 2: pcpa = +0, rx = 0, tx = 0 [ 501.721862] percpu_var:exit_percpu_var(): removed, bye

One other thing I noticed when compiling was a stack size warning from the compiler (prior to the changes above and after): warning: the frame size of 1232 bytes is larger than 1024 bytes [-Wframe-larger-than=]
Hey Brian... ok...

I noticed that if I commented out the following lines in function set_cpuaffinity() the warning went away:

// cpumask_clear(&mask); // cpumask_set_cpu(cpu, &mask); // 1st param is the CPU number, not bitmask /* !HACK! sched_setaffinity() is NOT exported, we can't call it * sched_setaffinity(0, &mask); // 0 => on self * so we invoke it via it's function pointer */ // ret = (*ptr_sched_setaffinity)(0, &mask); // 0 => on self

hmmm, ok but we need the sched_setaffinity() in order to guarantee the threads run in parallel on different cpu cores, updating and using the per-cpu variables independently, a key point of the demo.

Is the quick explanation that this struct is so large that when putting it on the local stack frame it's bigger than the default recommended stack frame size by GCC?
Yes. It's tunable via make menuconfig...

Thanks,

Brian W.

P.S. I'm definitely going to give this book an excellent write-up on Amazon. :-)