5enxia / parallel-krylov

Krylov Subspace Method Modules made by Ikuno labolatory in Tokyo University of Technology

krylov

Krylov subspace methods module.

Methods

• CG
• MrR
• k-skip CG
• k-skip MrR
• Adaptive k-skip MrR

directories

• v3
  • cpu
    • mpi
      • cg
      • mrr
      • kskipcg
      • kskipmrr
      • adaptivekskipmrr
    • cg
    • mrr
    • kskipcg
    • kskipmrr
    • adaptivekskipmrr
  • gpu
    • mpi
      • cg
      • mrr
      • kskipcg
      • kskipmrr
      • adaptivekskipmrr
    • cg
    • mrr
    • kskipcg
    • kskipmrr
    • adaptivekskipmrr

requirements

C libs

only CPU

• C Compiler
  • GCC
  • Intel C Compiler
  • etc...
• BLAS library
  • OpenBLAS
  • AppleBLAS
  • etc..

with GPU

• CUDA(10.1)

with MPI

• MPI library
  • OpenMPI(https://www.open-mpi.org/)
  • MVAPICH2(http://mvapich.cse.ohio-state.edu/)
  • Intel MPI
  • etc...

Pyhton3 modules

only CPU

• numpy
• scipy

with GPU

• cupy9.6.0

with MPI

• mpi4py

only exec with cuda and mpiexec.hydra(Intel MPI)

• fastrlock

settings

on macOS(10.14.6)

export below param

export PMIX_MCA_gds=hash

with cupy

expand mermory allocator limit

pool = cp.cuda.MemoryPool(cp.cuda.malloc_managed)
cp.cuda.set_allocator(pool.malloc)

parallel-krylov's Issues

IndexError: Index 204 is out of bounds for axis 0 with size 136

Doesn't converged, k-skip MrR with GPU + MPI

Fri Dec 24 17:12:03 JST 2021
ESC[32m# ================ INFO ================ #ESC[0m
Method: k-skip MrR + gpu + mpi
Initial_k: 0
Time: 0.5320529937744141 s
Status: diverged
Iteration: 112 times
Final_Residual: 0.2209790717824514
ESC[32m# ====================================== #ESC[0m

InfiniBand cache error

[scb0115:mpi_rank_0][MPIDI_CH3_Init]

Please set LD_PRELOAD to the full path of libmpi.so of your MVAPICH2-GDR installation to avoid unexpected errors for GPU-based transfers

E.g. LD_PRELOAD=<PATH_TO_MVAPICH2_GDR_INSTALL>/lib64/libmpi.so

WARNING: Error in initializing MVAPICH2 ptmalloc library.Continuing without InfiniBand registration cache support.

The output of the MPI adaptive k skip MrR method does not match a single version.

single

============== INFO =================

Method: adaptive k-skip MrR
k: 5
time: 0.013791053999739233 s
status: converged
iteration: 6 times
residual: 3.4114063610239287e-09
final k: 5

=====================================

MPI

============== INFO =================

Method: adaptive k-skip MrR
k: 5
time: 0.130841 s
status: converged
iteration: 7 times
residual: 2.9166983973371867e-10
final k: 5

=====================================

CUDA_VISIBLE_DEVICE Operation

Make interface scipy-like

Reuse calculated vector x

Implement with NCCL

Calculation result becomes strange at the time of asynchronous execution with memcpyPeerAsync()

def dot(cls, local_A, x, out):
        # Copy vector data to All devices
        for i in range(cls.begin, cls.end+1):
            index = i-cls.begin
            cp.cuda.runtime.memcpyPeerAsync(cls.x[index].data.ptr, i, x.data.ptr, cls.end, cls.nbytes, cls.streams[index].ptr)
        # dot
        for i in range(cls.begin, cls.end+1):
            index = i-cls.begin
            Device(i).use()
            cls.streams[index].synchronize()
            cls.y[index] = cls.A[index].dot(cls.x[index])
        # Gather caculated element from All devices
        for i in range(cls.begin, cls.end+1):
            Device(i).synchronize()
            index = i-cls.begin
            cp.cuda.runtime.memcpyPeerAsync(cls.out[index*cls.local_local_N].data.ptr, cls.end, cls.y[index].data.ptr, i, cls.local_local_nbytes, cls.streams[index].ptr)