Zombie/defunct processes caused by xpmem? about xpmem

[angainor]

@hjelmn not sure if this is important, but I've noticed that I get the deadlock less often when I don't call xpmem_remove explicitly in my code. This makes me wonder: is there a possible cleanup problem when the publisher calls xpmem_remove on a region, which is still attached to by the peers? In other words, publisher calls xpmem_remove and only then the peer calls xpmem_detach and xpmem_release.

[cvmeq]

@angainor have you found a solution or root cause for this? We are experiencing very similar crashes resulting in zombie/defunct processes in our AMD cluster running RHEL 8.6 and MLNX_OFED_LINUX-5.8-1.0.1.1:

[Wed Jun 14 17:28:43 2023] Call Trace:
[Wed Jun 14 17:28:43 2023]  __schedule+0x2d1/0x840
[Wed Jun 14 17:28:43 2023]  schedule+0x35/0xa0
[Wed Jun 14 17:28:43 2023]  schedule_timeout+0x278/0x300
[Wed Jun 14 17:28:43 2023]  ? number+0x324/0x360
[Wed Jun 14 17:28:43 2023]  ? get_futex_key+0x98/0x3e0
[Wed Jun 14 17:28:43 2023]  wait_for_completion+0x96/0x100
[Wed Jun 14 17:28:43 2023]  __synchronize_srcu.part.17+0x83/0xb0
[Wed Jun 14 17:28:43 2023]  ? __bpf_trace_rcu_utilization+0x10/0x10
[Wed Jun 14 17:28:43 2023]  ? synchronize_srcu+0xad/0xf0
[Wed Jun 14 17:28:43 2023]  mmu_notifier_unregister+0xa6/0xe0
[Wed Jun 14 17:28:43 2023]  xpmem_flush+0x14a/0x170 [xpmem]
[Wed Jun 14 17:28:43 2023]  filp_close+0x31/0x70
[Wed Jun 14 17:28:43 2023]  put_files_struct+0x70/0xc0
[Wed Jun 14 17:28:43 2023]  do_exit+0x32f/0xb10
[Wed Jun 14 17:28:43 2023]  do_group_exit+0x3a/0xa0
[Wed Jun 14 17:28:43 2023]  get_signal+0x158/0x870
[Wed Jun 14 17:28:43 2023]  do_signal+0x36/0x690
[Wed Jun 14 17:28:43 2023]  ? do_send_sig_info+0x63/0x90
[Wed Jun 14 17:28:43 2023]  ? recalc_sigpending+0x17/0x60
[Wed Jun 14 17:28:43 2023]  exit_to_usermode_loop+0x89/0x100
[Wed Jun 14 17:28:43 2023]  do_syscall_64+0x19c/0x1b0
[Wed Jun 14 17:28:43 2023]  entry_SYSCALL_64_after_hwframe+0x61/0xc6

[angainor]

@cvmeq Unfortunately no, I still see those issues sometimes, mostly when you kill / interrupt a large job, or at job cleanup. Then only solution for me was not to use xpmem transport in OpenMPI / UCX