openucx / xpmem Goto Github PK
View Code? Open in Web Editor NEWThis project forked from hpc/xpmem
Linux Cross-Memory Attach
License: GNU Lesser General Public License v2.1
xpmem's People
Contributors
Stargazers
Watchers
xpmem's Issues
Linux Kernel 6.3 issue: read-only vma flags
Possible patch to handle API change with read-only public field:
Compile error:
CC [M] /usr/src/xpmem-git/kernel/xpmem_attach.o
/usr/src/xpmem-git/kernel/xpmem_attach.c: In function ‘xpmem_attach’:
/usr/src/xpmem-git/kernel/xpmem_attach.c:588:23: error: assignment of read-only member ‘vm_flags’
588 | vma->vm_flags |=
| ^~
make[4]: *** [scripts/Makefile.build:252: /usr/src/xpmem-git/kernel/xpmem_attach.o] Error 1*** /usr/src/xpmem-git/kernel/xpmem_attach.c 2023-06-21 09:17:37.614617257 -0400
--- xpmem_attach.c 2023-06-06 09:38:40.664864853 -0400
***************
*** 585,592 ****
--- 585,598 ----
xpmem_mmap_write_unlock(current->mm);
vma->vm_private_data = att;
+
+ #if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 3, 0)
+ vm_flags_set(vma, vma->vm_flags |
+ VM_DONTCOPY | VM_DONTDUMP | VM_IO | VM_DONTEXPAND | VM_PFNMAP);
+ #else
vma->vm_flags |=
VM_DONTCOPY | VM_DONTDUMP | VM_IO | VM_DONTEXPAND | VM_PFNMAP;
+ #endif
vma->vm_ops = &xpmem_vm_ops;
att->at_vma = vma;
Could be all 6.x.y versions, not sure.
Unkillable processes caused by xpmem
I have previously filed issue hpc#45, but got no answer there. Maybe you have an idea what's going on?
We are running on RH 7.7, kernel 3.10.0-1062.9.1.el7.x86_64, on AMD EPYC 7702 64-Core Processor. Quite often after an OpenMPI job, which uses UCX for all communication, is killed before it finalizes cleanly the processes get stuck in D-state and cannot be killed, the node has to be rebooted. Looking at /proc/<PID>/stack shows this type of stacks:
b1118-mn: [<ffffffffb9192d48>] call_rwsem_down_read_failed+0x18/0x30
b1118-mn: [<ffffffffc088f73d>] xpmem_clear_PTEs_range+0x24d/0x300 [xpmem]
b1118-mn: [<ffffffffc088f80b>] xpmem_clear_PTEs+0x1b/0x20 [xpmem]
b1118-mn: [<ffffffffc088d8d0>] xpmem_remove_seg+0x50/0xf0 [xpmem]
b1118-mn: [<ffffffffc088dbfa>] xpmem_remove_segs_of_tg+0x4a/0x80 [xpmem]
b1118-mn: [<ffffffffc088d82f>] xpmem_teardown+0x3f/0x90 [xpmem]
b1118-mn: [<ffffffffc0891011>] xpmem_mmu_release+0x111/0x180 [xpmem]
b1118-mn: [<ffffffffb901c2f7>] __mmu_notifier_release+0x57/0x140
b1118-mn: [<ffffffffb8ffb065>] exit_mmap+0x175/0x1a0
b1118-mn: [<ffffffffb8e982d7>] mmput+0x67/0xf0
b1118-mn: [<ffffffffb8ea1ff8>] do_exit+0x288/0xa50
b1118-mn: [<ffffffffb8ea283f>] do_group_exit+0x3f/0xa0
b1118-mn: [<ffffffffb8eb364e>] get_signal_to_deliver+0x1ce/0x5e0
b1118-mn: [<ffffffffb8e2c527>] do_signal+0x57/0x6f0
b1118-mn: [<ffffffffb8e2cc32>] do_notify_resume+0x72/0xc0
b1118-mn: [<ffffffffb958457c>] retint_signal+0x48/0x8c
b1118-mn: [<ffffffffffffffff>] 0xffffffffffffffff
I haven't seen this happening after a job exits cleanly, and UCX is finalized. So I get an impression that when I kill a job and UCX cannot release the xpmem pointers, something goes wrong with the process cleanup in the kernel.
Do you have any ideas as to what might be the reason?
xpmem kernel fails to build on debian and SLES due to kernel headers location
Recent master, ./configure && (cd kernel && make)
/labhome/tzafrirc/sysgwork/ofed/xpmem/xpmem/kernel/xpmem_pfn.c: In function 'xpmem_pin_page':
/labhome/tzafrirc/sysgwork/ofed/xpmem/xpmem/kernel/xpmem_pfn.c:265:25: error: 'struct task_struct' has no member named 'cpus_allowed'; did you mean 'nr_cpus_allowed'?
265 | saved_mask = current->cpus_allowed;
| ^~~~~~~~~~~~
| nr_cpus_allowed
If I manually edit config.h to define HAVE_STRUCT_TASK_STRUCT_CPUS_MASK , this works OK, but continues to fail on an error with PDE.
So why does it fail? On config.log I see:
configure:12967: gcc -c -g -O2 -include /lib/modules/5.10.0-13-amd64/build//include/linux/kconfig.h -include /lib/modules/5.10.0-13-amd64/build//include/linux/compiler.h -D__KERNEL__ -I/lib/modules/5.10.0-13-amd64/build//include -I/lib/modules/5.10.0-13-amd64/build//include/uapi -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include/uapi -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include/generated -I/lib/modules/5.10.0-13-amd64/build//arch/x86/include/generated/uapi conftest.c >&5
cc1: fatal error: /lib/modules/5.10.0-13-amd64/build//include/linux/kconfig.h: No such file or directory
compilation terminated.
And indeed /lib/modules/5.10.0-13-amd64/build//include/linux/kconfig.h is not there. So how does kernel build work?
$ cat /lib/modules/5.10.0-13-amd64/build/Makefile
include /usr/src/linux-headers-5.10.0-13-common/Makefile
I also see:
$ ls -l /lib/modules/5.10.0-13-amd64/ | grep -- '->'
lrwxrwxrwx 1 root root 38 Mar 17 2022 build -> /usr/src/linux-headers-5.10.0-13-amd64
lrwxrwxrwx 1 root root 39 Mar 17 2022 source -> /usr/src/linux-headers-5.10.0-13-common
and /lib/modules/5.10.0-13-amd64/source/include/linux/kconfig.h is the right file.
xpmem_make returns -1, MMIO read fault?
I am unable to use xpmem on my local laptop (Linux Mint 20.2 Uma). I built xpmem, signed the kernel module, loaded it and when I run xpmem_make it returns -1. The last two messages below seem to be appended to dmesg every time I call xpmem_make:
[ 2450.718135] XPMEM kernel module v2.6.5 loaded
[ 3165.159169] nouveau 0000:02:00.0: Enabling HDA controller
[ 3165.262314] nouveau 0000:02:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ PRIVRING ]
Linux kernel version:
$ uname -r
5.13.0-27-generic
Has anyone seen this? Any idea what is going wrong here?
hang with UCX test
$ make -C test/gtest test GTEST_FILTER=shm_ib/test_ucp_tag_nbx.fallback/0
<hang>
...
$ cat /proc/32638/stack
[<ffffffff9c5e15b2>] __get_user_pages+0x1d2/0x7a0
[<ffffffff9c5e2032>] get_user_pages+0x52/0x60
[<ffffffffc02b6124>] xpmem_pin_page+0xc4/0x200 [xpmem]
[<ffffffffc02b6660>] xpmem_ensure_valid_PFN+0x30/0x40 [xpmem]
[<ffffffffc02b4595>] xpmem_fault_handler+0x1e5/0x560 [xpmem]
[<ffffffff9c5e41da>] __do_fault.isra.59+0x8a/0x100
[<ffffffff9c5e478c>] do_read_fault.isra.61+0x4c/0x1b0
[<ffffffff9c5e9134>] handle_pte_fault+0x2f4/0xd10
[<ffffffff9c5ebc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffff9cb6f5e3>] __do_page_fault+0x203/0x500
[<ffffffff9cb6f915>] do_page_fault+0x35/0x90
[<ffffffff9cb6b758>] page_fault+0x28/0x30
[<ffffffffffffffff>] 0xffffffffffffffff
error: implicit declaration of function ‘pud_large’
make[2]: Entering directory '/dev/shm/ape/1716881881/xpmem-2.7.3/kernel'
make -C /lib/modules/6.9.1-2.el9.elrepo.x86_64/build/ M=/dev/shm/ape/1716881881/xpmem-2.7.3/kernel
make[3]: Entering directory '/usr/src/kernels/6.9.1-2.el9.elrepo.x86_64'
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_main.o
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_main.c:404:1: warning: no previous prototype for ‘xpmem_init’ [-Wmissing-prototypes]
404 | xpmem_init(void)
| ^~~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_main.c:492:1: warning: no previous prototype for ‘xpmem_exit’ [-Wmissing-prototypes]
492 | xpmem_exit(void)
| ^~~~~~~~~~
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_make.o
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_get.o
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_attach.o
CC [M] /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.o
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c: In function ‘xpmem_vaddr_to_pte_offset’:
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:51:24: error: implicit declaration of function ‘pud_large’; did you mean ‘pud_page’? [-Werror=implicit-function-declaration]
51 | #define pud_is_huge(p) pud_large(p)
| ^~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:151:18: note: in expansion of macro ‘pud_is_huge’
151 | else if (pud_is_huge(*pud)) {
| ^~~~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:50:24: error: implicit declaration of function ‘pmd_large’; did you mean ‘pmd_page’? [-Werror=implicit-function-declaration]
50 | #define pmd_is_huge(p) pmd_large(p)
| ^~~~~~~~~
/dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.c:161:18: note: in expansion of macro ‘pmd_is_huge’
161 | else if (pmd_is_huge(*pmd)) {
| ^~~~~~~~~~~
cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:244: /dev/shm/ape/1716881881/xpmem-2.7.3/kernel/xpmem_pfn.o] Error 1
Please advise
Fail to build on Ubuntu 18.04 ppc64le
Building recent master (93ae5ca) on Ubuntu 18.04 fails:
CC [M] /var/lib/dkms/xpmem/2.6.5/build/kernel/xpmem_mmu_notifier.o
In file included from ./include/linux/kernel.h:10:0,
from ./include/linux/list.h:9,
from ./include/linux/module.h:9,
from /var/lib/dkms/xpmem/2.6.5/build/kernel/xpmem_mmu_notifier.c:13:
./arch/powerpc/include/asm/uaccess.h: In function ‘clear_user’:
./arch/powerpc/include/asm/uaccess.h:437:23: error: ‘VERIFY_WRITE’ undeclared (first use in this function); did you mean ‘MAY_WRITE’?
if (likely(access_ok(VERIFY_WRITE, addr, size))) {
^
(Log attached)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.