mpi4py / mpi4py-fft

mpi4py-fft is a Python package for computing Fast Fourier Transforms (FFTs).

mpi4py-fft

mpi4py-fft is a Python package for computing Fast Fourier Transforms (FFTs). Large arrays are distributed and communications are handled under the hood by MPI for Python (mpi4py). To distribute large arrays we are using a new and completely generic algorithm that allows for any index set of a multidimensional array to be distributed. We can distribute just one index (a slab decomposition), two index sets (pencil decomposition) or even more for higher-dimensional arrays.

mpi4py-fft comes with its own Python interface to the serial FFTW library. This interface can be used much like pyfftw, and even for real-to-real transforms, like discrete cosine or sine transforms.

Further documentation is found at readthedocs.

Installation

The mpi4py-fft package can be installed using:

pip install mpi4py-fft

or, to get the latest version from GitHub:

pip install git+https://github.com/mpi4py/mpi4py-fft@master

Install with conda from the coda-forge channel:

conda install -c conda-forge mpi4py-fft

or build it with conda build from the main source directory:

conda build -c conda-forge conf/
conda create --name mpi4py_fft mpi4py_fft --use-local

which will pull in the required dependencies from the conda-forge channel.

Note that mpi4py-fft depends on Python packages

• mpi4py
• numpy
• cython

and the serial C-library

• FFTW

Note in particular that mpi4py requires that you have a working MPI installation, with the compiler wrapper mpicc, on your search path. The FFTW header and libraries must also be available on the search path, and we will build wrappers for any precision found of the FFTW libraries.

All of the above dependencies are available and will be downloaded through the conda-forge channel if conda is used for installation. However, pip will not help you with MPI or FFTW.

For IO you need to install either h5py or netCDF4 with support for MPI. Both are available from the coda-forge channel through:

conda install -c conda-forge h5py=*=mpi* netcdf4=*=mpi*

mpi4py-fft's Issues

Problem in slab decomposition docs

From following the slab decomposition docs I have the file pfft_example.py, which is copied below. When I run it with two cores I get a slightly different output from what is suggested.

1 (64, 128, 128) 0 (64, 128, 128) 1 (128, 32, 128) 0 (128, 33, 128)

It does run but I imagine the output should be modified slightly to reflect this difference.

`import numpy as np
from mpi4py import MPI
from mpi4py_fft import PFFT, newDistArray
N = np.array([128, 128, 128], dtype=int)
#fft = PFFT(MPI.COMM_WORLD, N, axes=(0, 1, 2), dtype=np.float, grid=(-1,))
fft = PFFT(MPI.COMM_WORLD, N, axes=(2, 0, 1), dtype=np.float)

u = newDistArray(fft, False)
u[:] = np.random.random(u.shape).astype(u.dtype)
u_hat = fft.forward(u, normalize=True) # Note that normalize=True is default and ca
n be omitted
uj = np.zeros_like(u)
uj = fft.backward(u_hat, uj)
assert np.allclose(uj, u)
print(MPI.COMM_WORLD.Get_rank(), u.shape)

u_hat = newDistArray(fft, True)
print(MPI.COMM_WORLD.Get_rank(), u_hat.shape)
`

errors in test

Hello,

After having installed mpi4py-fft I am trying some of the tests and they are failing. I have installed netcdf4 adn h5py, as requested but when I try trying test_io and test_darray I get that they both fail because of no attribute 'set_fapl_mpio'. See below. Can you tell me what it is that I might be missing?

Francis

fpoulin@amlt21:~/software/mpi4py-fft/tests$ python3 -m test_io Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/fpoulin/software/mpi4py-fft/tests/test_io.py", line 180, in <module> test_3D(bnd, kind) File "/home/fpoulin/software/mpi4py-fft/tests/test_io.py", line 82, in test_3D h0file = writer[backend]('uv'+filename, domain=domain) File "/home/fpoulin/.local/lib/python3.6/site-packages/mpi4py_fft/io/h5py_file.py", line 33, in __init__ self.f = h5py.File(h5name, mode, driver="mpio", comm=comm, **kw) File "/home/fpoulin/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 405, in __init__ fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, **kwds) File "/home/fpoulin/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 135, in make_fapl set_fapl(plist, **kwds) File "/home/fpoulin/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 48, in _set_fapl_mpio plist.set_fapl_mpio(**kwargs) AttributeError: 'h5py.h5p.PropFAID' object has no attribute 'set_fapl_mpio' fpoulin@amlt21:~/software/mpi4py-fft/tests$ python3 -m test_darray Traceback (most recent call last): File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/fpoulin/software/mpi4py-fft/tests/test_darray.py", line 129, in <module> test_2Darray() File "/home/fpoulin/software/mpi4py-fft/tests/test_darray.py", line 36, in test_2Darray k = a.get((0,)*rank+(0, slice(None))) File "/home/fpoulin/.local/lib/python3.6/site-packages/mpi4py_fft/distarray.py", line 218, in get f = h5py.File('tmp.h5', 'w', driver="mpio", comm=comm) File "/home/fpoulin/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 405, in __init__ fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, **kwds) File "/home/fpoulin/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 135, in make_fapl set_fapl(plist, **kwds) File "/home/fpoulin/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 48, in _set_fapl_mpio plist.set_fapl_mpio(**kwargs) AttributeError: 'h5py.h5p.PropFAID' object has no attribute 'set_fapl_mpio'

How to fill a distributed array with function values?

Hello!

I'm just starting to study your library, so my question may seem naive.
But how to fill a distributed array with function values correctly? For example, there is a coordinate grid (NxN) on which the values of the function are set. I want to do a Fourier transform for this function. With newDistArray, I can create a distributed array of full dimension (NxN). And how to fill it with function values after that?

I could create a distributed array for the coordinate grid, then pass it as an argument to the function and as a result get a distributed array with the values of the function.
But then the question arises, how to fill in a distributed array for a coordinate grid?

How to approach this problem correctly?
(Something similar is being done in your example spectral_dns_solver.py, but I haven't fully figured out this code)

I will be glad to your comments!

switching to real to real transforms

Hello Mikael,

I have figured out how to go between fftn/ifftn and rfftn/irfftn and I think this is working nicely.

I am now trying to figure out how to use cosine and sine transforms. My first attempt seemed to crash and burn. I defined the following but it seems to keep things periodic at the boundaries.

from mpi4py_fft.fftw import dstn, idstn, dctn, idctn

import functools
dst = functools.partial(dstn, type=3)
idst = functools.partial(idstn, type=3)

r2r = PFFT(MPI.COMM_WORLD, N, axes=(0, 1), dtype=np.float,
           grid=(-1,), transforms={(0,1): (dstn, idstn)} )

I also used the same wavenumbers that I used for the rfftn. Is there anything else that I should be doing?

reading in h5 file

I am following section 1.5 of the docs to read in an h5 file and everything does seem to work well. But I do have questions about some of the details.

• It seems that we need the transform type, what is called T in the example. Is this really necessary?
• In the example, you need to define the grid, then then the transform, and then you read in the array. Can you read in the array and then figure that out?
• Also, in addition to figuring out the grid, can you figure out the number of time steps that are stored?

gathering arrays to plot

I am using spectralDNS as a guide to learn how to solve PDEs using mpi4py-fft and having a problem with the plotting. In particular, I want to gather the distributed arrays, namely coordinates and a field in order to be able to plot it with matplotlib.

I have included below my current attempt it doesn't work. What is the correct way of doing this given how newDistArrays are defined?

`# Initialize arrays
xx = np.zeros((N[0],N[1]))
yy = np.zeros((N[0],N[1]))
uu = np.zeros((N[0],N[1]))

Gather fields

xx = MPI.COMM_WORLD.gather(X[0], root=0)
yy = MPI.COMM_WORLD.gather(X[1], root=0)
uu = MPI.COMM_WORLD.gather(ur, root=0)

Plot

plt.figure(figsize=(16,10))
plt.pcolormesh(uu)
plt.colorbar()
plt.show()
`

Example reproduction error

Hello everyone,

I tried out your great package and tested this example (https://github.com/mpi4py/mpi4py-fft/blob/master/examples/darray.py) with minor attachment to print out s0,s1,s2:

darray.txt

I've tried to run the example code on a total of 1-8 nodes with python 3.11.6, but the code runs successfully only in the case of 1, 2, 3 and 5 nodes, otherwise error assertion arises:
assertion_error.txt

Attached are output of different processor numbers:
nproc1.txt
nproc2.txt
nproc3.txt
nproc4.txt
nproc5.txt
nproc6.txt
nproc7.txt
nproc8.txt

As you can see, the output changes for certain processor numbers, but I am unsure why. Help would be appreciated.
Regards
Hanru

mpi4py-fft with Apple Silicon M1/M2?

Looks like mpi4py-fft cannot easily be installed on an M1 or M2 arm-based architecture. Using conda does not work, because only recipes for osx-64 and linux-64 are given. Using pip fails (for me) with

      [...]
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      cimport fftwf_xfftn
              ^
      ------------------------------------------------------------
      
      mpi4py_fft/fftw/fftwf_xfftn.pyx:1:8: 'fftwf_xfftn.pxd' not found
      [...]

Any ideas?

install FFTW with conda

Hi
Did you try to use the FFTW library that is installed automatically from conda-forge when you install mpi4py-fft? That version is configured appropriately and should work. You do not need FFTW built with MPI.

Originally posted by @mikaem in #15 (comment)
Thanks, actually, I installed the FFTW library manually and copied it to the conda library, it may incompatible with the mpi4py-fft, I used your method, and it works well

AssertionError on dtype mismatch in pencil.py for small arrays

3D transforms with a small index-2 lengths encounter AssertionError in pencil.py (link to line).

import numpy as np
from mpi4py import MPI
from mpi4py_fft import PFFT, newDistArray

# No assertion error for (5, 5, 3).
shape = np.array((5, 5, 2), dtype=int)

fft = PFFT(MPI.COMM_WORLD, shape=shape, dtype=float)

u = newDistArray(fft, forward_output=False)
u[...] = np.random.random(u.shape).astype(u.dtype)

u_hat = newDistArray(fft, forward_output=True)

# AssertionError
fft.forward(u, u_hat)

An array with dimensions (5, 5, 3) is fine, but (5, 5, 2) doesn't work.

I am using version 2.0.4.

How to apply to already distributed array?

In the Ramba distributed array package (https://github.com/Python-for-HPC/ramba), the arrays are already distributed and Ramba has a mode where inter-node communication is handled by MPI. Ramba tries to provide the NumPy API so would it be possible for Ramba's version of ramba.fft.fft to use mpi4py-fft internally to implement that function?

more cores takes longer

I have put together a simple code from the codes that will time how long it takes the compute the forward and inverse transforms of a matrix 100 times.

When I do tests on my laptop (4 cores) I get that with more cores the code actually slows down.

This is run using: mpiexec -np 4 python pfft_2d.py

np = 1 37 sec
np = 2 48 sec
np = 4 49 sec

I am surely doing something wrong and would really appreciate how I can fix my code to get some speed up instead slow down.

`import numpy as np
import timeit
import sys

SETUP_CODE = '''
import numpy as np
from mpi4py import MPI
from mpi4py_fft import PFFT, newDistArray

nx = 4096
N = np.array([nx, nx], dtype=int)
fft = PFFT(MPI.COMM_WORLD, N, axes=(0, 1), dtype=np.float, grid=(-1,))

u = newDistArray(fft, False)
u[:] = np.random.random(u.shape).astype(u.dtype)
'''

TEST_CODE = '''
u_hat = fft.forward(u)
uj = np.zeros_like(u)
uj = fft.backward(u_hat, uj)
'''

times = timeit.timeit(setup = SETUP_CODE,
stmt = TEST_CODE,
number = 100)

print('Time = %6.4f seconds' % times)
`

setup.py and docs still reference bitbucket repo

I guess the source url should be updated from https://bitbucket.org/mpi4py/mpi4py-fft to https://github.com/mpi4py/mpi4py-fft in setup.py and in docs.

How to redistribute an already declared array?

Hello, Mikaem!

In the Parallel FFT examples in the documentation, a distributed array is always created first, and then filled with numbers (in this case, random numbers). For instance:

N = np.array([128, 128, 128], dtype=int)
fft = PFFT(MPI.COMM_WORLD, N, axes=(0, 1, 2), dtype=np.float, grid=(-1,))
u = newDistArray(fft, False)
u[:] = np.random.random(u.shape).astype(u.dtype)

Can you please tell me how to create a numpy array first and then redistribute it for parallel FFT?

mpi4py-fft and numpy differ for a noisy field

I have been converting some numpy based fft code to mpi4py-fft and finding some odd results. When I choose the initial field to be random, as one does, I get that the derivatives in the x-directions match very well, but the derivatives in the y-directions are rather different.

Question: should they yield the same results and if yes why are these results different?

`import numpy as np
from numpy.fft import fftn, ifftn
import sys
from mpi4py import MPI
from mpi4py_fft import PFFT, newDistArray
from mpi4py_fft.fftw import rfftn, irfftn

import matplotlib.pyplot as plt

def get_local_mesh(FFT, L):
"""Returns local mesh."""
X = np.ogrid[FFT.local_slice(False)]
N = FFT.global_shape()
for i in range(len(N)):
X[i] = (X[i]*L[i]/N[i])
X = [np.broadcast_to(x, FFT.shape(False)) for x in X]
return X

def get_local_wavenumbermesh(FFT, L):
"""Returns local wavenumber mesh."""
s = FFT.local_slice()
N = FFT.global_shape()
# Set wavenumbers in grid
k = [np.fft.fftfreq(n, 1./n).astype(int) for n in N[:-1]]
k.append(np.fft.rfftfreq(N[-1], 1./N[-1]).astype(int))
K = [ki[si] for ki, si in zip(k, s)]
Ks = np.meshgrid(K, indexing='ij', sparse=True)
Lp = 2np.pi/L
for i in range(2):
Ks[i] = (Ks[i]*Lp[i]).astype(float)
return [np.broadcast_to(k, FFT.shape(True)) for k in Ks]

def main():

# Grid Parameters                                                
M = 3
N = np.array([2**M, 2**M],       dtype=int)
L = np.array([np.pi, np.pi], dtype=float)
dx = L/N

# Create parallel FFT
#r2c = PFFT(MPI.COMM_WORLD, N, axes=(0, 1), dtype=np.float,
#           grid=(-1,), transforms={(0,1): (rfftn, irfftn)} )
r2c = PFFT(MPI.COMM_WORLD, N)

# Declare variables
pr   = newDistArray(r2c, False)      
qr   = newDistArray(r2c, False)      
er   = newDistArray(r2c, False)      
rr   = newDistArray(r2c, False)      
qk   = newDistArray(r2c)      
fk   = newDistArray(r2c)      

# Grid and wavenumber
X    = get_local_mesh(r2c, L)
K    = get_local_wavenumbermesh(r2c, L)
K    = np.array(K).astype(float)

# Initial Conditions
np.random.seed(1)
qr[:] = 2.0*(np.random.random(qr.shape).astype(qr.dtype)-0.5)

### Method 1: compute gradient using mpi4py-fft

qk = r2c.forward(qr)
fk = 1j*K[1]*qk
pr = r2c.backward(fk, pr)

qk = r2c.forward(qr)
fk = 1j*K[0]*qk
rr = r2c.backward(fk, rr)

print('Method 1: mpi4py-fft (extreme values)')
print('q_x:', np.amin(pr), np.amax(pr))
print('q_y:', np.amin(rr), np.amax(rr))
print(' ')

### Method 2: compute gradient using numpy fft

# Grid
x = np.linspace(0,L[0]-dx[0],N[0])
y = np.linspace(0,L[1]-dx[1],N[1])
xx,yy = np.meshgrid(x,y)

# Wavenumber
kx = 2*np.pi/L[0]*N[0]*np.fft.fftfreq(N[0])
ky = 2*np.pi/L[1]*N[1]*np.fft.fftfreq(N[1])
kxx, kyy = np.meshgrid(kx,ky)

# Initial Conditions
np.random.seed(1)
q  = 2.0*(np.random.random((N[1],N[0])) - 0.5)

q_hat = fftn(q)
q_x = (ifftn( 1j*kxx*q_hat)).real
q_y = (ifftn( 1j*kyy*q_hat)).real

print('Method 2: numpy (extreme values)')
print('q_x:', np.amin(q_x), np.amax(q_x))
print('q_y:', np.amin(q_y), np.amax(q_y))
print(' ')

print('Good news: q_x are almost identical')
print('Bad news:  q_y are very different')

main()
`

How to load array from the disk, such as in the format of '.npy' ?

Hi,
I find that mpi4py-fft is an excellent tool for performing the FFT in parallel, which greatly improves computational efficiency. However, I don't see anything in the documentation about how to load data from the disk. So, Can you specify the methods of loading data from the disk?

AssertionError for number of processors equals array dimension

This probably doesn't happen often in practice, but using 3 processors for the below code triggers an assertion error in pencil.py (link to line). 2 processors causes no problems.

import numpy as np
from mpi4py import MPI
from mpi4py_fft import PFFT, newDistArray

# This works on 2 processors, but not 3.
shape = np.array((2, 2, 3), dtype=int)

fft = PFFT(MPI.COMM_WORLD, shape=shape, dtype=float)

u = newDistArray(fft, forward_output=False)
u[...] = np.random.random(u.shape).astype(u.dtype)

u_hat = newDistArray(fft, forward_output=True)

# AssertionError
fft.forward(u, u_hat)

If this case is not supported, I would like to request that this raise an exception with descriptive text rather than just hit an assert.

incorrect link on pypi

https://pypi.org/project/mpi4py-fft/ still links to the bitbucket page.

doing multiple transforms together

I am trying to improve the efficiency of my code that uses mpi4py-fft but having problems with accuracy. Basically, I want to compute the gradient of a vorticity vector and then invert to compute the velocity. I hoped that I could do one 2D forward transform and then many derivatives without having to store too much. Below tries to compute psi_y*q_x where q is the vorticity.

What is weird is that if I uncomment the 4th line I get a different answer, which suggests to me that qk changes somewhere in either line 2 or 3. Can you help me to understand what's happening?

qk = fft.forward(qr)
fk = -ikyoK2*qk
fr = fft.backward(fk, fr)

#qk = fft.forward(qr)
fk =  ikx*qk
gr = fft.backward(fk, gr)

pip installation error

Hi,

I'm aiming to install mpi4py-fft using pip.
However, I ended up with the following error

[...]
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    cimport fftwf_xfftn
            ^
    ------------------------------------------------------------
    
    mpi4py_fft/fftw/fftwf_xfftn.pyx:1:8: 'fftwf_xfftn.pxd' not found

[...]
 Cython.Compiler.Errors.CompileError: /scratch/pip-build-bwiiwole/mpi4py-fft/mpi4py_fft/fftw/fftwf_xfftn.pyx

The error is reproducible under various conditions

• mpi4py_fft versions (master, 2.0.5 - 2.0.0)
• gnu compiler version (11, 9)
• python version (3.6,3.7,3.9,3.10)
• operating systems (Ubuntu 22.04.3 LTS, Red Hat Enterprise Linux release 8.6 (Ootpa))

I would be grateful for any hint on how to resolve this issue.

Best,
Sebastian

docs lead to ImportError

Hello

I am brand new to mpi4py-fft and have read through the documents and now wanting to try out the different examples to learn from experience. Unfortunately, one of the first importing steps seems to fail. My library claims to be successfully installed. Does this mean I have a faulty installation?

Francis

`>>> from mpi4py_fft import Pencil, Subcomm

Traceback (most recent call last):

File "", line 1, in

ImportError: cannot import name 'Pencil'
`

fftw_threads symbols

I get an import error when executing: from mpi4py_fft.fftw import fftw_xfftn

In [1]: from mpi4py_fft.fftw import fftw_xfftn

ImportError Traceback (most recent call last)
in
----> 1 from mpi4py_fft.fftw import fftw_xfftn

ImportError: dlopen(/Users/colincotter/firedrake-fftw2/lib/python3.8/site-packages/mpi4py_fft/fftw/fftw_xfftn.cpython-38-darwin.so, 2): Symbol not found: _fftw_cleanup_threads
Referenced from: /Users/colincotter/firedrake-fftw2/lib/python3.8/site-packages/mpi4py_fft/fftw/fftw_xfftn.cpython-38-darwin.so
Expected in: flat namespace
in /Users/colincotter/firedrake-fftw2/lib/python3.8/site-packages/mpi4py_fft/fftw/fftw_xfftn.cpython-38-darwin.so

I have built fftw with --enable-threads --enable-shared (I'm asking PETSc to build it because I need to ensure that I'm using the right mpi library). libfftw3_threads can be found in the PETSc lib and I am adding the PETSc lib and include directories to FFTW_INCLUDE_DIR and FFTW_LIBRARY_DIR. How can I get mpi4py_fft to pick up these symbols?

using PFFT is causing error, not sure why!

Hi,

I am following the example as in this thread just to make sure that everything works fine,
from mpi4py import MPI
from mpi4py_fft import PFFT, newDistArray
FFT = PFFT(MPI.COMM_WORLD, [64, 64, 64])
u = newDistArray(FFT, False, rank=1)
u_hat = newDistArray(FFT, True, rank=1)

but when I call PFFT(), the following error occurs,

PFFT(MPI.COMM_WORLD, [64, 64, 64])
Traceback (most recent call last):
File "", line 1, in
File "/home/ahmad68/.conda/envs/kmc/lib/python3.9/site-packages/mpi4py_fft/mpifft.py", line 315, in init
xfftn = FFT(pencil.subshape, axes, dtype, padding, backend=backend,
File "/home/ahmad68/.conda/envs/kmc/lib/python3.9/site-packages/mpi4py_fft/libfft.py", line 387, in init
self.fwd, self.bck = plan(self.shape, self.axes, self.dtype, transforms, kw)
File "/home/ahmad68/.conda/envs/kmc/lib/python3.9/site-packages/mpi4py_fft/libfft.py", line 73, in _Xfftn_plan_fftw
xfftn_fwd = plan_fwd(U, s=s, axes=axes, threads=threads, flags=flags)
File "/home/ahmad68/.conda/envs/kmc/lib/python3.9/site-packages/mpi4py_fft/fftw/xfftn.py", line 239, in rfftn
return get_planned_FFT(input_array, output_array, axes, kind, threads,
File "/home/ahmad68/.conda/envs/kmc/lib/python3.9/site-packages/mpi4py_fft/fftw/factory.py", line 106, in get_planned_FFT
return _fft.FFT(input_array, output_array, axes, kind, threads, flags,
File "mpi4py_fft/fftw/fftw_xfftn.pyx", line 153, in mpi4py_fft.fftw.fftw_xfftn.FFT.cinit
RuntimeError: Failure creating FFTW plan

thanks a lot,

ttest2D_cr0.h5 fails trying to append to file which does not yet exist

Issue ported from https://bitbucket.org/mpi4py/mpi4py-fft/issues/21

I'm trying to run the tests for mpi4py-fft 2.0.4 with h5py 3.3.0

Tests initially run successfully, but then fail:

$ H5PY_ALWAYS_USE_MPI=1 ./runtests.sh 
/usr/lib/python3/dist-packages/mpi4py_fft/libfft.py:182: ComplexWarning: Casting complex values to real discards the imaginary part
  self.output_array[...] = self.xfftn(self.input_array, **self.opt)
backend:  scipy 0.07071471214294434
backend:  numpy 0.06495881080627441
backend:  fftw 0.014360666275024414
Traceback (most recent call last):
  File "/home/drew/projects/python/build/mpi4py-fft/tests/test_io.py", line 181, in <module>
    test_2D(bnd, kind)
  File "/home/drew/projects/python/build/mpi4py-fft/tests/test_io.py", line 46, in test_2D
    u.write('t'+filename, 'u', 0)
  File "/usr/lib/python3/dist-packages/mpi4py_fft/distarray.py", line 400, in write
    f = writer(filename, domain=domain, mode='a')
  File "/usr/lib/python3/dist-packages/mpi4py_fft/io/h5py_file.py", line 33, in __init__
    self.f = h5py.File(h5name, mode, driver="mpio", comm=comm, **kw)
  File "/usr/lib/python3/dist-packages/h5py/_debian_h5py_mpi/_hl/files.py", line 444, in __init__
    fid = make_fid(name, mode, userblock_size,
  File "/usr/lib/python3/dist-packages/h5py/_debian_h5py_mpi/_hl/files.py", line 211, in make_fid
    fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
  File "h5py/_debian_h5py_mpi/_objects.pyx", line 54, in h5py._debian_h5py_mpi._objects.with_phil.wrapper
  File "h5py/_debian_h5py_mpi/_objects.pyx", line 55, in h5py._debian_h5py_mpi._objects.with_phil.wrapper
  File "h5py/_debian_h5py_mpi/h5f.pyx", line 100, in h5py._debian_h5py_mpi.h5f.open
OSError: Unable to open file (MPI_ERR_NO_SUCH_FILE: no such file or directory)

‌

Tracing the value of filename shows that this error occurs with ttest2D_cr0.h5, accessing with mode 'a'.

Preceding tests completed succcessfully (e.g.uvtest_cr1.h5, uvtest_rr1.h5, uvtest_rr2.h5). Tracing their file access conditions, I see that they were first accessed with mode 'w', and later with mode 'a'.

The error seems to be happening since ttest2D_cr0.h5 is immediately being accessed with mode 'a' without first creating the file with mode 'w'. Evidentally mode 'a' is failing since the file does not exist yet.

RuntimeError: Failure creating FFTW plan

Hi,
I have install mpi4py-fft from command conda install -c conda-forge mpi4py-fft, And it output no error when I import it.
However , when I run the test file, it output the errors like that:

Traceback (most recent call last):
  File "/home/tristan/learn_fft/mpi4py-fft/tests/test_fftw.py", line 162, in <module>
    test_fftw()
  File "/home/tristan/learn_fft/mpi4py-fft/tests/test_fftw.py", line 101, in test_fftw
    dct = fftw.dctn(input_array, None, axes, type, threads, fflags, output_array=oa)
  File "/home/tristan/.local/software/anaconda3/lib/python3.8/site-packages/mpi4py_fft/fftw/xfftn.py", line 397, in dctn
    return get_planned_FFT(input_array, output_array, axes, kind, threads,
  File "/home/tristan/.local/software/anaconda3/lib/python3.8/site-packages/mpi4py_fft/fftw/factory.py", line 106, in get_planned_FFT
    return _fft.FFT(input_array, output_array, axes, kind, threads, flags,
  File "mpi4py_fft/fftw/fftwf_xfftn.pyx", line 153, in mpi4py_fft.fftw.fftwf_xfftn.FFT.__cinit__
RuntimeError: Failure creating FFTW plan

I can't run the pfft_example.py with the same error, I think it is from the FFTW library.
I built my fftw library like this :

./configure --prefix=$HOME/.local/software/fftw-3.3.10 --enable-shared --enable-mpi --enable-float

and I have the bin file in my $PATH, also the lib file in my $LD_LIBRARY_PATH, is there something wrong I did?
Could you give me some advice, please?
Thanks

pip installation error

Hi,

I'm aiming to install mpi4py-fft using pip.
However, I ended up with the following error

error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for mpi4py-fft (setup.py) ... error
  ERROR: Failed building wheel for mpi4py-fft
  Running setup.py clean for mpi4py-fft
Failed to build mpi4py-fft
ERROR: Could not build wheels for mpi4py-fft, which is required to install pyproject.toml-based projects

Use mpi4py-fft with CuPy arrays?

@dalcinl I had a tiny bit of time over the holidays to pick up this old task that has been on my mind for a couple of years. I noticed DistArray is the workhorse behind mpi4py-fft, however it requires subclassing numpy.ndarray

mpi4py-fft/mpi4py_fft/distarray.py

Line 10 in ac510f8

class DistArray(np.ndarray):

which makes it not possible to use cupy.ndarray as a drop-in replacement, as CuPy currently does not support subclassing (cupy/cupy#3972).

Is it absolutely necessary to make DistArray a subclass? Can it be tweaked (hopefully mildly) to become an array container to support both NumPy and CuPy arrays (and potentially other array/tensor libraries)? Could this change break backward compatibility? From my first glimpse it seems possible but I'd like to discuss further and see if I miss anything.

mpi4py-fft, array storage

Would there be any sense for mpi4py-fft to provide methods for storing arrays in physical or spectral spaces and under various formats (hdf, netcdf or other) ?

I was playing with mpi4py-fft and wished there was such feature.

cheers