errors in test about mpi4py-fft

[mikaem]

Hi Francis
Do you have h5py installed with support for mpi?

[francispoulin]

Thank you for the reply.

I should have mentioned that I am running ubuntu 18.

Using apt I have python3-h5py installed and I also pip3 installed h5py. I have removed them both.

As for the parallel part, I now see I should be installing h5pyp, the parallel version. Sorry for not seeing this before. I am trying to do it now using pip3 and it's progressing but I get a "Failed building wheel for h5pyp", which is not a good sign.

But I suppose I should ask the h5pyp group about that?

[francispoulin]

I've been trying to run the simple scripts that show to store data.

https://mpi4py-fft.readthedocs.io/en/latest/io.html

Unfortunately, I can't write in either format. I have tried on a couple of ubuntu machines, and both give the same error. Copied below.

I have netCDF4 and h5py installed but i'm guessing not the parallel versions.

Are the parallel versions things that we can install with pip or do we need to build them?

Thanks in advance.
`>>> f0 = HDF5File('h5test.h5', mode='w')
Traceback (most recent call last):
File "", line 1, in
File "/home/fpoulin/software/firedrake/lib/python3.6/site-packages/mpi4py_fft/io/h5py_file.py", line 33, in init
self.f = h5py.File(h5name, mode, driver="mpio", comm=comm, **kw)
File "/home/fpoulin/software/firedrake/lib/python3.6/site-packages/h5py/_hl/files.py", line 405, in init
fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, **kwds)
File "/home/fpoulin/software/firedrake/lib/python3.6/site-packages/h5py/_hl/files.py", line 135, in make_fapl
set_fapl(plist, **kwds)
File "/home/fpoulin/software/firedrake/lib/python3.6/site-packages/h5py/_hl/files.py", line 48, in _set_fapl_mpio
plist.set_fapl_mpio(**kwargs)
AttributeError: 'h5py.h5p.PropFAID' object has no attribute 'set_fapl_mpio'

f1 = NCFile('nctest.nc', mode='w')
Traceback (most recent call last):
File "", line 1, in
File "/home/fpoulin/software/firedrake/lib/python3.6/site-packages/mpi4py_fft/io/nc_file.py", line 54, in init
comm=comm, **kw)
File "netCDF4/_netCDF4.pyx", line 2157, in netCDF4._netCDF4.Dataset.init
ValueError: parallel mode requires MPI enabled netcdf-c
`

[mikaem]

Hi Francis
I believe the error is that you are using the serial hdf5 and netcdf libraries and not parallel?

[francispoulin]

Hello,

I think you are at least half right. I have not been able to install h5pyp and have received no support from the issue I posted on their site. However, I do have python3-netcdf installed which I think is parallel.

Could maybe help me figure out exactly which libraries I should install?

[mikaem]

python3-netcdf is serial. I don't even know if there's a parallel version available on ubuntu. I have built a parallel version myself, but that is only available with conda (https://anaconda.org/spectraldns/netcdf4-parallel).

[francispoulin]

Thanks. I will work on that and may go the way of conda since that would be a lot easier.

[francispoulin]

I have installed conda and tried to install spectraldns, mpifftpy nad netcdf4-parallel and each of them fails because of conflicts.

Any advice on how I might fix this?

`(conda_python) fpoulin@amlt21:~/software$ conda install -c spectraldns spectraldns
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions
`

[mikaem]

If you just want to install mpi4py-fft with parallel hdf5 do

conda install -c conda-forge mpi4py-fft h5py=*=mpi_*

You are trying to install the spectraldns solvers. (The project here: https://github.com/spectralDNS/spectralDNS) The h5py stuff is not really needed, but it does not hurt either to specify it. You can choose which mpi provider, e.g., h5py=*=mpi_mpich* to get mpich and h5py=*=mpi_openmpi* for openmpi.

[francispoulin]

Thanks again. I didn't know I should get these from conda-forge.

Also, I will go to git directly to access spectralDNS

From conda-forge I installed: mpi4py-fft, netcdf and h5py. Note that the latter don't necessarily involve MPI. I need to check on those.

When I tried the command you suggested, it took a long time, and seemed to get further, but it did fail and I believe that I sent a report, in case that would be helpful.

Unfortunately, when I use the basic operations to write, it still fails in about the same way.

I am sorry that I'm not able to figure this out. I suppose if I could install your netcdf4-parallel, that would allow me to write but as it stands I can't even do it in serial.

[mikaem]

h5py should pick a parallel build if you do as I suggested above. However, you need to get netCDF4 from the spectralDNS channel, because conda-forge does not (at least they did not the last time I checked) have parallel NetCDF4

conda install -c spectraldns netcdf4-parallel

[francispoulin]

Unfortunately, the build that you suggested failed to install it.

When I try the above, it seems to complain about what version of python I'm using. Essentially, I'm using 3.7 but it wants something different.

`(base) fpoulin@amlt21:~/software/mpi4py-fft/my_tests$ conda install -c spectraldns netcdf4-parallel
...
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

• netcdf4-parallel -> python[version='>=3.5,<3.6.0a0|>=3.8,<3.9.0a0']

Your python: defaults/linux-64::python==3.7.6=h0371630_2

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
`

I installed python 3.5 and then it still seemed to complain about my version of python. I suppose I can try 3.6 and see if that works better.

`Specifications:

• netcdf4-parallel -> python[version='2.7.|>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|3.6.']

Your python: python=3.5.6
`

[mikaem]

Just checked and apparently you can get netCDF4 in parallel from conda-forge now, just like h5py. Just do

conda install -c conda-forge netcdf4=*=mpi*

and you should get a parallel build.

[francispoulin]

I have tried both the commands you suggested

conda install -c conda-forge netcdf4=*=mpi* conda install -c conda-forge mpi4py-fft h5py=*=mpi_*

They each take a really long time and they each fail, because an error has occurred. A conda report is prepared for each an below I've copied one of the two. I am a bit at a loss as to what to try. Any suggestions are welcome.

' environment variables:
CIO_TEST=
CONDA_DEFAULT_ENV=py37
CONDA_EXE=/home/fpoulin/software/anaconda3/bin/conda
CONDA_PREFIX=/home/fpoulin/software/anaconda3/envs/py37
CONDA_PREFIX_1=/home/fpoulin/software/anaconda3
CONDA_PROMPT_MODIFIER=(py37)
CONDA_PYTHON_EXE=/home/fpoulin/software/anaconda3/bin/python
CONDA_ROOT=/home/fpoulin/software/anaconda3
CONDA_SHLVL=2
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
LMOD_DEFAULT_MODULEPATH=/opt/modulefiles/Linux:/etc/lmod/modules:/usr/share/lmod/lmod/modulefi
les
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
MANPATH=/usr/share/lmod/lmod/share/man::
MODULEPATH=/opt/modulefiles/Linux:/etc/lmod/modules:/usr/share/lmod/lmod/modulefi
les
PATH=/home/fpoulin/software/anaconda3/bin:/home/fpoulin/software/anaconda3/
envs/py37/bin:/home/fpoulin/software/anaconda3/condabin:/home/fpoulin/
software/VisIt/bin:/home/fpoulin/software/julia-1.0.3/bin:/home/fpouli
n/.local/bin:/home/fpoulin/bin:/usr/local/sbin:/usr/local/bin:/usr/sbi
n:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/maple
2017/bin
PYTHONPATH=/home/fpoulin/software/pyro2:/home/fpoulin/software/SPINSpy:
REQUESTS_CA_BUNDLE=
SSL_CERT_FILE=
WINDOWPATH=2

 active environment : py37
active env location : /home/fpoulin/software/anaconda3/envs/py37
        shell level : 2
   user config file : /home/fpoulin/.condarc

populated config files : /home/fpoulin/.condarc
conda version : 4.8.2
conda-build version : 3.18.9
python version : 3.7.4.final.0
virtual packages : __glibc=2.27
base environment : /home/fpoulin/software/anaconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/fpoulin/software/anaconda3/pkgs
/home/fpoulin/.conda/pkgs
envs directories : /home/fpoulin/software/anaconda3/envs
/home/fpoulin/.conda/envs
platform : linux-64
user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.4 Linux/4.15.0-88-generic ubuntu/18.04.4 glibc/2.27
UID:GID : 16273791:100
netrc file : None
offline mode : False '

[mikaem]

I'm sorry, conda is not something mastered in a day. But when you get the hang of it it's great:-) On my mac everything works right out of the box using just conda-forge. I start in the base conda environment. I make sure there is nothing there from the conda-forge or other channels (see output of conda list). I then create and activate the mpi4py-fft environment like this

>>> conda create --name mpi4py-fft mpi4py-fft mpich h5py=*=mpi* netcdf4=*=mpi* 
>>> conda activate mpi4py-fft

I usually also install pip and matplotlib in the new environment. And all this takes less than 2 minutes I think. But I have good internet:-)

[francispoulin]

Thanks for the continued help. It is greatly appreciated and learning a lot, which I appreciate it.
I was running things with an ethernet connection at my university so the internet should have been excellent but will try it at home.

I tried what you suggested but sadly it complained. Do I need to create first than install?

`(py37) fpoulin@amlt21:~/software/netcdf4-python$ conda create --name mpi4py-fft mpi4py-fft mpich h5py==mpi netcdf4==mpi
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

• h5py[build=mpi*]
• netcdf4[build=mpi*]
• mpi4py-fft

Current channels:

• https://repo.anaconda.com/pkgs/main/linux-64
• https://repo.anaconda.com/pkgs/main/noarch
• https://repo.anaconda.com/pkgs/r/linux-64
• https://repo.anaconda.com/pkgs/r/noarch
  `

[mikaem]

You need the coda-forge channel. Add it to your .condarc file or use '-c conda-forge' in the command.

[francispoulin]

Thanks MIkael that makes a lot of sense and should have guessed that.

I'm happy to say success! This worked beautifully in a few minutes, all while my laptop was tethered to my phone. I'm not sure exactly what it was I did differently yesterday but very happy that it's finally working.

Now to get back to developing my code.

In case anyone else is interested, this is the command that worked.

conda create --name mpi4py-fft -c conda-forge mpi4py-fft mpich h5py=*=mpi* netcdf4=*=mpi*