openucx / xpmem Goto Github PK
View Code? Open in Web Editor NEWThis project forked from hpc/xpmem
Linux Cross-Memory Attach
License: GNU Lesser General Public License v2.1
This project forked from hpc/xpmem
Linux Cross-Memory Attach
License: GNU Lesser General Public License v2.1
Possible patch to handle API change with read-only public field:
Compile error:
CC [M] /usr/src/xpmem-git/kernel/xpmem_attach.o
/usr/src/xpmem-git/kernel/xpmem_attach.c: In function ‘xpmem_attach’:
/usr/src/xpmem-git/kernel/xpmem_attach.c:588:23: error: assignment of read-only member ‘vm_flags’
588 | vma->vm_flags |=
| ^~
make[4]: *** [scripts/Makefile.build:252: /usr/src/xpmem-git/kernel/xpmem_attach.o] Error 1
*** /usr/src/xpmem-git/kernel/xpmem_attach.c 2023-06-21 09:17:37.614617257 -0400
--- xpmem_attach.c 2023-06-06 09:38:40.664864853 -0400
***************
*** 585,592 ****
--- 585,598 ----
xpmem_mmap_write_unlock(current->mm);
vma->vm_private_data = att;
+
+ #if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 3, 0)
+ vm_flags_set(vma, vma->vm_flags |
+ VM_DONTCOPY | VM_DONTDUMP | VM_IO | VM_DONTEXPAND | VM_PFNMAP);
+ #else
vma->vm_flags |=
VM_DONTCOPY | VM_DONTDUMP | VM_IO | VM_DONTEXPAND | VM_PFNMAP;
+ #endif
vma->vm_ops = &xpmem_vm_ops;
att->at_vma = vma;
Could be all 6.x.y versions, not sure.
I have previously filed issue hpc#45, but got no answer there. Maybe you have an idea what's going on?
We are running on RH 7.7, kernel 3.10.0-1062.9.1.el7.x86_64, on AMD EPYC 7702 64-Core Processor. Quite often after an OpenMPI job, which uses UCX for all communication, is killed before it finalizes cleanly the processes get stuck in D-state and cannot be killed, the node has to be rebooted. Looking at /proc/<PID>/stack
shows this type of stacks:
b1118-mn: [<ffffffffb9192d48>] call_rwsem_down_read_failed+0x18/0x30
b1118-mn: [<ffffffffc088f73d>] xpmem_clear_PTEs_range+0x24d/0x300 [xpmem]
b1118-mn: [<ffffffffc088f80b>] xpmem_clear_PTEs+0x1b/0x20 [xpmem]
b1118-mn: [<ffffffffc088d8d0>] xpmem_remove_seg+0x50/0xf0 [xpmem]
b1118-mn: [<ffffffffc088dbfa>] xpmem_remove_segs_of_tg+0x4a/0x80 [xpmem]
b1118-mn: [<ffffffffc088d82f>] xpmem_teardown+0x3f/0x90 [xpmem]
b1118-mn: [<ffffffffc0891011>] xpmem_mmu_release+0x111/0x180 [xpmem]
b1118-mn: [<ffffffffb901c2f7>] __mmu_notifier_release+0x57/0x140
b1118-mn: [<ffffffffb8ffb065>] exit_mmap+0x175/0x1a0
b1118-mn: [<ffffffffb8e982d7>] mmput+0x67/0xf0
b1118-mn: [<ffffffffb8ea1ff8>] do_exit+0x288/0xa50
b1118-mn: [<ffffffffb8ea283f>] do_group_exit+0x3f/0xa0
b1118-mn: [<ffffffffb8eb364e>] get_signal_to_deliver+0x1ce/0x5e0
b1118-mn: [<ffffffffb8e2c527>] do_signal+0x57/0x6f0
b1118-mn: [<ffffffffb8e2cc32>] do_notify_resume+0x72/0xc0
b1118-mn: [<ffffffffb958457c>] retint_signal+0x48/0x8c
b1118-mn: [<ffffffffffffffff>] 0xffffffffffffffff
I haven't seen this happening after a job exits cleanly, and UCX is finalized. So I get an impression that when I kill a job and UCX cannot release the xpmem pointers, something goes wrong with the process cleanup in the kernel.
Do you have any ideas as to what might be the reason?
Recent master, ./configure && (cd kernel && make)
/labhome/tzafrirc/sysgwork/ofed/xpmem/xpmem/kernel/xpmem_pfn.c: In function 'xpmem_pin_page':
/labhome/tzafrirc/sysgwork/ofed/xpmem/xpmem/kernel/xpmem_pfn.c:265:25: error: 'struct task_struct' has no member named 'cpus_allowed'; did you mean 'nr_cpus_allowed'?
265 | saved_mask = current->cpus_allowed;
| ^~~~~~~~~~~~
| nr_cpus_allowed
If I manually edit config.h to define HAVE_STRUCT_TASK_STRUCT_CPUS_MASK , this works OK, but continues to fail on an error with PDE.
So why does it fail? On config.log I see:
configure:12967: gcc -c -g -O2 -include /lib/modules/5.10.0-13-amd64/build//include/linux/kconfig.h -include /lib/modules/5.10.0-13-amd64/build//include/linux/compiler.h -D__KERNEL__ -I/lib/modules/5.10.0-13-amd64/build//include -I/lib/modules/5.10.0-13-amd64/build//include/uapi -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include/uapi -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include/generated -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include/generated/uapi conftest.c >&5
cc1: fatal error: /lib/modules/5.10.0-13-amd64/build//include/linux/kconfig.h: No such file or directory
compilation terminated.
And indeed /lib/modules/5.10.0-13-amd64/build//include/linux/kconfig.h is not there. So how does kernel build work?
$ cat /lib/modules/5.10.0-13-amd64/build/Makefile
include /usr/src/linux-headers-5.10.0-13-common/Makefile
I also see:
$ ls -l /lib/modules/5.10.0-13-amd64/ | grep -- '->'
lrwxrwxrwx 1 root root 38 Mar 17 2022 build -> /usr/src/linux-headers-5.10.0-13-amd64
lrwxrwxrwx 1 root root 39 Mar 17 2022 source -> /usr/src/linux-headers-5.10.0-13-common
and /lib/modules/5.10.0-13-amd64/source/include/linux/kconfig.h is the right file.
I am unable to use xpmem on my local laptop (Linux Mint 20.2 Uma). I built xpmem, signed the kernel module, loaded it and when I run xpmem_make
it returns -1. The last two messages below seem to be appended to dmesg every time I call xpmem_make
:
[ 2450.718135] XPMEM kernel module v2.6.5 loaded
[ 3165.159169] nouveau 0000:02:00.0: Enabling HDA controller
[ 3165.262314] nouveau 0000:02:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ PRIVRING ]
Linux kernel version:
$ uname -r
5.13.0-27-generic
Has anyone seen this? Any idea what is going wrong here?
$ make -C test/gtest test GTEST_FILTER=shm_ib/test_ucp_tag_nbx.fallback/0
<hang>
...
$ cat /proc/32638/stack
[<ffffffff9c5e15b2>] __get_user_pages+0x1d2/0x7a0
[<ffffffff9c5e2032>] get_user_pages+0x52/0x60
[<ffffffffc02b6124>] xpmem_pin_page+0xc4/0x200 [xpmem]
[<ffffffffc02b6660>] xpmem_ensure_valid_PFN+0x30/0x40 [xpmem]
[<ffffffffc02b4595>] xpmem_fault_handler+0x1e5/0x560 [xpmem]
[<ffffffff9c5e41da>] __do_fault.isra.59+0x8a/0x100
[<ffffffff9c5e478c>] do_read_fault.isra.61+0x4c/0x1b0
[<ffffffff9c5e9134>] handle_pte_fault+0x2f4/0xd10
[<ffffffff9c5ebc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffff9cb6f5e3>] __do_page_fault+0x203/0x500
[<ffffffff9cb6f915>] do_page_fault+0x35/0x90
[<ffffffff9cb6b758>] page_fault+0x28/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
make[2]: Entering directory '/dev/shm/ape/1716881881/xpmem-2.7.3/kernel'
make -C /lib/modules/6.9.1-2.el9.elrepo.x86_64/build/ M=/dev/shm/ape/1716881881/xpmem-2.7.3/kernel
make[3]: Entering directory '/usr/src/kernels/6.9.1-2.el9.elrepo.x86_64'
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_main.o
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_main.c:404:1: warning: no previous prototype for ‘xpmem_init’ [-Wmissing-prototypes]
404 | xpmem_init(void)
| ^~~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_main.c:492:1: warning: no previous prototype for ‘xpmem_exit’ [-Wmissing-prototypes]
492 | xpmem_exit(void)
| ^~~~~~~~~~
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_make.o
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_get.o
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_attach.o
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.o
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c: In function ‘xpmem_vaddr_to_pte_offset’:
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:51:24: error: implicit declaration of function ‘pud_large’; did you mean ‘pud_page’? [-Werror=implicit-function-declaration]
51 | #define pud_is_huge(p) pud_large(p)
| ^~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:151:18: note: in expansion of macro ‘pud_is_huge’
151 | else if (pud_is_huge(*pud)) {
| ^~~~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:50:24: error: implicit declaration of function ‘pmd_large’; did you mean ‘pmd_page’? [-Werror=implicit-function-declaration]
50 | #define pmd_is_huge(p) pmd_large(p)
| ^~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:161:18: note: in expansion of macro ‘pmd_is_huge’
161 | else if (pmd_is_huge(*pmd)) {
| ^~~~~~~~~~~
cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:244: /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.o] Error 1
Please advise
Building recent master (93ae5ca) on Ubuntu 18.04 fails:
CC [M] /var/lib/dkms/xpmem/2.6.5/build/kernel/xpmem_mmu_notifier.o
In file included from ./include/linux/kernel.h:10:0,
from ./include/linux/list.h:9,
from ./include/linux/module.h:9,
from /var/lib/dkms/xpmem/2.6.5/build/kernel/xpmem_mmu_notifier.c:13:
./arch/powerpc/include/asm/uaccess.h: In function ‘clear_user’:
./arch/powerpc/include/asm/uaccess.h:437:23: error: ‘VERIFY_WRITE’ undeclared (first use in this function); did you mean ‘MAY_WRITE’?
if (likely(access_ok(VERIFY_WRITE, addr, size))) {
^
(Log attached)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.