ompi-x / libomp Goto Github PK

Adding this as issue to track status while testing. Using gv_pmix branch at da5ea2c (after PR #4 was merged).

I enabled debug prints/instrumentation to __kmp_affinity_set_init_mask() to show the binding of OMP threads to places. I see that some threads show all places instead of a specific place X.

Here's the command I used on machine with 12 cores (libomp/moc test show NCPUs=12).

NOTE: I lowered the KA_TRACE() lines to debug level=5 in code, otherwise use KMP_A_DEBUG=101.

  beaker:$ env KMP_A_DEBUG=6 mpirun \
                   -x OMP_PLACES=threads \
                  -x OMP_CPU_AFFINITY=TRUE \
                  -x OMP_PROC_BIND=spread \ 
                  --map-by core -np 2 \
                 ./mpi_omp_hello.dbg

Here's the relevant output sorted for easier reading and with the PID info:

  __kmp_affinity_set_init_mask: binding T#0 (PID:13615) to place 0
  __kmp_affinity_set_init_mask: binding T#1 (PID:13615) to all places
  __kmp_affinity_set_init_mask: binding T#2 (PID:13615) to place 1
  __kmp_affinity_set_init_mask: binding T#3 (PID:13615) to place 1
  __kmp_affinity_set_init_mask: binding T#4 (PID:13615) to place 1

  __kmp_affinity_set_init_mask: binding T#0 (PID:13614) to place 0
  __kmp_affinity_set_init_mask: binding T#1 (PID:13614) to all places
  __kmp_affinity_set_init_mask: binding T#2 (PID:13614) to all places
  __kmp_affinity_set_init_mask: binding T#3 (PID:13614) to place 0
  __kmp_affinity_set_init_mask: binding T#4 (PID:13614) to place 0

Implement the hooks for the creation of places so we can do more advanced partitioning of available resources when more than 1 MPI rank is running on the node.
We already identified the __kmp_partition_places() function (runtime/src/kmp_partition.cpp) that seems to initialize places based on the binding strategy; it might be a good place to start.

Also, the goal would ultimately be to control where and how Open MP places are created while sharing resources between MPI ranks.

I just ran a test against the latest gv_pmix branch and noticed that the omp_get_thread_num() returns one less than expected. For example, if NumCpus=12 and NumRanks=2, would expect to have 6 threads but instead we get 5 threads.

  [PID:1037] DBG: LINE:1780 nthreads=5, _n_local_cpus=12, _n_local_ranks=2
  [PID:1036] DBG: LINE:1780 nthreads=5, _n_local_cpus=12, _n_local_ranks=2
 ...<snip>...
  [mpi_omp_hello.c:main:48] In parallel loop: calling omp_get_thread_num()...
  [mpi_omp_hello.c:main:50] omp_get_thread_num() succeeded
  Number of threads = 5

Looks like this is related to kmp_runtime.cpp#L1779.