pnorbert / adiosvm Goto Github PK
View Code? Open in Web Editor NEWPackages and howtos for creating a linux system for ADIOS tutorials
Packages and howtos for creating a linux system for ADIOS tutorials
adiosvm ======= Packages and howtos for creating a linux system for ADIOS tutorials Required steps to get plain ADIOS working: I.1-6,8 II.1,4 III.1 Steps: I. Set up a Linux VM ==================== 1. Install VirtualBox 2. Get a linux ISO image We currently use Lubuntu 16.04 64bit and the descriptions below all refer to that system. Debian based systems use 'apt-get' to install packages. 3. Create a new VM Type=Linux, System=Ubuntu (32bit) Memory: at least 2048MB, we use 3072MB just for the sake of it Virtual hard drive: VMDK type, dynamically allocated, about 16GB - the more the better, but an exported images should fit on an USB stick... Video memory: 64MB - we don't really know how much is needed - do not enable 2D video or 3D acceleration Processor: 2 CPUs - 1 is enough, 2 builds codes faster Storage: load the linux iso image into the DVD drive. - Start the VM and install linux, preferably with updates. Your name: ADIOS Your computer's nane: adiosVM Username: adios password: adios Log in automatically: yes, because this is a tutorial vm Note: if you use a computer name other than adiosVM, make sure you substitute for the name everywhere in this document (e.g. Flexpath install) - Instead of restart, shut down and remove the linux DVD, start again Whenever Lubuntu offers updates, do it... 4. LXTerminal: Bottom left corner of screen has the start menu, open System Tools/LXTerminal - LXTerminal setup (if you don't like the default one) - Start LXTerminal - Edit/Profile preferences - Display tab: increase the scrollback to a few thousand lines (e.g. 5120) - Style tab: set font and background/foreground to your style (Monospace 12 and black font on white background for the tutorial VM) - Add terminal icon to task bar: right-click on task bar, select Add/Remove Panel Items Select Application Launch Bar, click Preferences Add System Tools/LXTerminal to the Launchers list 5. Virtualbox Guest Additions $ sudo apt-get update $ sudo apt-get install dkms - Dynamic Kernel Modules, to ensure rebuilding Guest Additions at future kernel updates - or just reinstall Guest Additions each time if you don't want dkms installed - VirtualBox VM menu: Devices/Insert Guest Additions CD Image - under Lubuntu it will not autorun, so in a terminal $ cd /media/adios/VBOXADDITIONS*/ $ sudo ./VBoxLinuxAdditions.run - this allows for resizing the window and for copy/paste between the VM and your host machine (and sharing folders between your host machine and this VM if you want) - set in VirtualBox main menu: Devices/Shared Clipboard/Bidirectional) Note: This has to be repeated when updating or recompiling the kernel unless the dkms package is installed - reboot the machine 6. Install some linux packages (sudo apt-get install or can use sudo synaptic) $ sudo apt-get install apt-file $ sudo apt-file update $ sudo apt-get install build-essential git-core libtool libtool-bin autoconf subversion $ sudo apt-get install gfortran $ sudo apt-get install pkg-config $ sudo apt-get autoremove -- this one is to remove unused packages, probably 0 at this point - Set HISTORY to longer: $ vi ~/.bashrc increase HISTSIZE (to 5000) and HISTFILESIZE (to 10000) - Turn off screen saver and lock Start menu / Preferences / Power Manager Display tab: Turn off flag to Handle display power management Set Blank after to the max (60 minutes on Lubuntu) Security tab: Automatically lock the session: Never Turn off flag to lock screen when system is going to sleep 7. Shared folder between your host machine and the VM (optional) This is not needed for tutorial, just if you want to share files between host and vm. - In VirtualBox, while the VM is shut down, set up a shared folder, with auto-mount. - Start VM. You can see a folder /media/sf_<your folder name> - Need to set group rights for adios user to use the folder $ sudo vi /etc/group add "adios" to the vboxfs entry so that it looks like this vboxsf:x:999:adios - log out and back, run 'groups' to check if you got the group rights. 8. Download this repository You can postpone step 6 and 7 after 8 if you have a github account and want to edit this repository content. $ cd $ git clone https://github.com/pnorbert/adiosvm.git 9. VIM setup $ sudo apt-get install vim copy from this repo: vimrc to ~/.vimrc $ cp ~/adiosvm/vimrc .vimrc On Lubuntu for some reason, the root creates ~/.viminfo which we need to remove This will allow vi/vim to create it again under adios user and use it to remember positions in files opened before $ sudo rm ~/.viminfo 10. Github access setup This step is only needed for ADIOS developers to get write access to the ADIOS repository from github. We need an account to github and a config for ssh. A minimum .ssh/config is found in this repository: $ cd $ mkdir .ssh $ cp ~/adiosvm/ssh_config ~/.ssh/config If the shared folder has access to you .ssh: $ cp /media/sf_<yourfolder>/.ssh/config .ssh $ cp /media/sf_<yourfolder>/.ssh/id_dsa_github* .ssh If you need a proxy to get to GitHub, use corkscrew and edit ~/.ssh/config to add the proxy command to each entry $ sudo apt-get install corkscrew If you have a github account and the config already, and postponed step 6, get the adiosvm repo now: $ git clone github:pnorbert/adiosvm.git Git settings: $ git config --global user.name "<your name>" $ git config --global user.email "<your email>" $ git config --global core.editor vim Of course, set an editor what you like. II. Preparations to install ADIOS ================================= 1. Linux Packages $ sudo apt-get install openmpi-common openmpi-bin libopenmpi-dev $ sudo apt-get install gfortran $ sudo apt-get install python-cheetah python-yaml 2. CMake We need a newer CMake version than what's available in linux distros. Download from: https://cmake.org/download/ Choose the self-extracting archive package for the Linux x86_64 platform and install it. 3. Staging support a. libfabric is required by the SST staging engine ------------ Download latest release from Github https://github.com/ofiwg/libfabric/releases and extract in ~/Software $ cd libfabric-1.6.0 $ ./configure --disable-verbs --prefix=/opt/libfabric $ make $ sudo make install In ~/.bashrc, add to LD_LIBRARY_PATH "/opt/libfabric/lib" b. ZeroMQ is required by the DataMan staging engine --------- $ sudo apt-get install libzmq5 libzmq3-dev 4. Compression libraries Only if you want to demo the transform library. zlib and bzip2 are installed as linux packages: -------------- $ sudo apt-get install bzip2 libbz2-dev zlib1g zlib1g-dev SZ is provided in adiospackages/ -- $ cd ~/Software $ tar zxf ~/adiosvm/adiospackages/sz-1.4.13.0.tar.gz $ cd sz-1.4.13.0 $ ./configure --prefix=/opt/SZ --with-pic --disable-shared --disable-fortran --disable-maintainer-mode $ make $ sudo make install BLOSC is available on GitHub: ----- $ cd ~/Software $ git clone https://github.com/Blosc/c-blosc.git $ cd c-blosc $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX=/opt/blosc .. $ make $ sudo make install In ~/.bashrc, add to LD_LIBRARY_PATH "/opt/blosc/lib" 5. Parallel HDF5 support Only if you want HDF5 read/write in ADIOS. $ cd ~/Software $ tar jxf ~/adiosvm/adiospackages/hdf5-1.8.17.tar.bz2 $ mv hdf5-1.8.17 hdf5-1.8.17-parallel $ cd hdf5-1.8.17-parallel $ ./configure --with-zlib=/usr --without-szlib --prefix=/opt/hdf5-1.8.17-parallel --enable-parallel --enable-fortran --with-pic CC=mpicc FC=mpif90 Verify that in the Features list: Parallel HDF5: yes Note: the -fPIC option is required for building parallel NetCDF4 later $ make -j 4 $ sudo make install 6. Python/Numpy support To build Adios python wrapper, install following packages by: $ sudo apt-get install python3 python3-dev Note: To use a parallel version, we need mpi4py. $ sudo apt-get install python3-pip python3-tk $ sudo -H pip3 install numpy mpi4py matplotlib Alternatively, we can install from a source code too: $ wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-2.0.0.tar.gz $ tar xvf mpi4py-2.0.0.tar.gz $ cd mpi4py-2.0.0 $ python3 setup.py build $ sudo python3 setup.py install 9. Fastbit indexing support (needed for queries) for ADIOS 1.x $ cd ~/Software $ svn co https://code.lbl.gov/svn/fastbit/trunk fastbit username and password: anonsvn $ cd fastbit/ $ ./configure --with-pic -prefix=/opt/fastbit $ make -- this will be slooooow $ sudo make install $ make clean -- src/ is about 1.4GB after build In ~/.bashrc, add to LD_LIBRARY_PATH "/opt/fastbit/lib" 10. Alacrity indexing and query support for ADIOS 1.x $ cd ~/Software either $ tar zxf ~/adiosvm/adiospackages/alacrity-1.0.0.tar.gz $ cd alacrity-1.0.0 or $ git clone https://github.com/ornladios/ALACRITY-ADIOS.git $ cd ALACRITY-ADIOS $ . ./runconf or $ ./configure CFLAGS="-g -fPIC -fno-common -Wall" CXXFLAGS="-g -fPIC -fno-exceptions -fno-rtti" --prefix=/opt/alacrity $ make $ sudo make install III. ADIOS Installation ======================= 1. Download ADIOS 2. Download ADIOS master from repository $ cd ~/Software $ git clone github:ornladios/ADIOS2.git OR $ git clone https://github.com/ornladios/ADIOS2.git $ cd ADIOS2 2. Build ADIOS Then: $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX=/opt/adios2 \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_PREFIX_PATH="/opt/blosc;/opt/zfp/0.5.5;/opt/SZ/2.0.2.1;/opt/MGARD;/opt/hdf5-parallel" \ -DADIOS2_USE_MPI=ON \ -DADIOS2_USE_Python=ON \ -DADIOS2_USE_Profiling=ON \ -DADIOS2_BUILD_TESTING=OFF \ .. $ make -j 4 3. Test ADIOS a bit $ ctest 4. Install $ sudo make install In ~/.bashrc, add to LD_LIBRARY_PATH "/opt/adios2/lib" and add to PATH "/opt/adios2/bin" 5. Python wrapper IV. ADIOS 1.x for compression and queries ========================================= 1. Download ADIOS 1. ADIOS 1.13.1 is in this repo: $ cd ~/Software $ tar zxf ~/adiosvm/adiospackages/adios-1.13.1.tar.gz $ cd adios-1.13.1 2. Download ADIOS master from repository $ cd ~/Software $ git clone github:ornladios/ADIOS.git OR $ git clone https://github.com/ornladios/ADIOS.git $ cd ADIOS $ ./autogen.sh 2. Build ADIOS Then: Edit runconf and change install path for the adiosVM target to /opt/adios1 $ mkdir build $ cd build $ ~/adiosvm/adiospackages/runconf.adios1 Configure ADIOS 1.x on VirtualBox. ... $ make -j 4 3. Test ADIOS a bit $ make check $ cd tests/suite $ ./test.sh 01 and so on up to 16 tests Some test may fail because multiple processes write a text file which is compared to a reference file, but some lines may get mixed up. Try running the same test a couple of times. One 'OK' means the particular test is okay. 4. Install $ sudo make install In ~/.bashrc, add to LD_LIBRARY_PATH "/opt/adios1/lib" and add to PATH "/opt/adios1/bin" 5. Build and install python wrapper To build Adios python wrapper, install following packages by: $ sudo apt-get install python3 python3-dev python3-tk Note: To use a parallel version, we need mpi4py. $ sudo apt-get install python3-pip $ sudo -H pip3 install numpy mpi4py matplotlib Alternatively, we can install from a source code too: $ wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-2.0.0.tar.gz $ tar xvf mpi4py-2.0.0.tar.gz $ cd mpi4py-2.0.0 $ python3 setup.py build $ sudo python3 setup.py install Then, we are ready to install adios and adios_mpi python module. An easy way is to use "pip3". $ sudo -H "PATH=$PATH" pip3 install --upgrade \ --global-option build_ext --global-option -lrt adios adios_mpi If there is any error, we can build from source. Go to the wrapper/numpy directory under the adios source directory: $ cd wrapper/numpy Type the following to build Adios python wrapper: $ python3 setup.py build_ext If not working, add " -lrt" The following command is to install: $ sudo "PATH=$PATH" python3 setup.py install Same for adios_mpi module: $ python3 setup_mpi.py build_ext -lrt $ sudo "PATH=$PATH" python3 setup_mpi.py install Test: A quick test can be done: $ cd wrapper/numpy/tests $ python3 test_adios.py $ mpirun -n 4 python3 test_adios_mpi.py V. ADIOS Tutorial code ======================= The tutorial is included in this repository ~/adiosvm/Tutorial 1. Linux Packages KSTAR demo requires gnuplot $ sudo apt-get install gnuplot Brusselator demo requires FFTW3 $ sudo apt-get install libfftw3-dev VI. Build VTK-M from source =========================== $ cd ~/Software $ git clone https://gitlab.kitware.com/vtk/vtk-m.git $ cd vtk-m $ mkdir build $ cd build $ cmake -DCMAKE_INSTALL_PREFIX=/opt/vtk-m .. $ make -j 4 $ sudo make install In ~/.bashrc, add to LD_LIBRARY_PATH "/opt/vtk-m/lib" VI. Build Visit from release =========================== Simple way: install binary release - download a release from https://wci.llnl.gov/simulation/computer-codes/visit/executables E.g.: $ cd /opt $ sudo tar zxf visit2_10_2.linux-x86_64-ubuntu14.tar.gz $ sudo mv visit2_10_2.linux-x86_64 visit Complicated way: - Need to have many linux packages installed. - Build a Visit release using it's build script that downloads/builds a lot of dependencies. 1. Linux packages $ sudo apt-get install dialog gcc-multilib subversion libx11-dev tcl tk $ sudo apt-get install libglu1-mesa-dev $ sudo libxt-dev $ sudo apt-get install xutils-dev For Silo reader to believe in good Qt installation $ sudo apt-get install libxmu-dev libxi-dev Qt development packages (required or visit will build its own) $ sudo apt-get libqt4-dev This will install all dependent packages 2. Build latest Visit release (with many dependencies) https://wci.llnl.gov/codes/visit/source.html Visit 2.7.2 release uses adios 1.6.0 but it downloads and builds its own version without compression or staging. $ cd ~/Software $ mkdir -p visit $ cd visit $ wget http://portal.nersc.gov/svn/visit/trunk/releases/2.7.2/build_visit2_7_2 Or download the latest build script from the website https://wci.llnl.gov/codes/visit/source.html $ chmod +x build_visit2_7_2 $ ./build_visit2_7_2 --parallel --mesa --mxml --adios --hdf5 --xdmf --zlib --silo $ ./build_visit --system-qt --parallel --mesa --mxml --adios --hdf5 --xdmf --zlib --silo --console This script should be started again and again after fixing build problems. All log is founf in build_visit2_7_2_log, appended at each try. In the dialogs, just accept everything | You many now try to run VisIt by cd'ing into the │ │ visit2.7.2/src/bin directory and invoking "visit". │ │ │ │ To create a binary distribution tarball from this │ │ build, cd to │ │ /home/adios/Software/visit/visit2.7.2/src │ │ then enter: "make package" │ │ │ │ This will produce a tarball called │ │ visitVERSION.ARCH.tar.gz, where VERSION is the │ │ version number, and ARCH is the OS architecure. │ │ │ │ To install the above tarball in a directory called │ │ "INSTALL_DIR_PATH", enter: svn_bin/visit-install │ │ VERSION ARCH INSTALL_DIR_PATH | 3. Build Visit from svn trunk - This has to be built on a second scratch disk. It requires ~10GB space. The installation is just 0.5GB. $ cd /work/adios $ mkdir visit.build $ cd visit.build/ $ svn co http://visit.ilight.com/svn/visit/trunk/src/svn_bin svn_bin $ ln -s svn_bin/build_visit . $ ln -s svn_bin/bv_support . $ git clone github:ornladios/ADIOS2.git $ ./build_visit --system-cmake --system-python --adios2 --silo --szip --zlib --hdf5 --makeflags -j4 --prefix /opt/visit when it is failed or succeeded with building the Visit source (not just the third-parties), i.e. when trunk/ already exists $ cp adiosVM.cmake trunk/src/config-site -- edit trunk/src/config-site/adiosVM.cmake and point to a SERIAL ADIOS2 installation VISIT_OPTION_DEFAULT(VISIT_ADIOS2_DIR /opt/adios2-serial) VISIT_OPTION_DEFAULT(VISIT_ADIOS2_PAR_DIR /opt/adios2) and build manually after this: $ cd trunk $ mkdir -p build $ cd build $ cmake ../src $ make -j 4 $ sudo make install This should put the installation files into /opt/visit/<version> and the executable script into /opt/visit/bin VII. Build Plotter ================= If you still want to use the our own plotter instead of / besides visit. 1. Install grace or build it ---------------------------- $ sudo apt-get install grace -- use XMGRACE=/usr in Makefile.adiosVM of the plotter package below or $ sudo apt-get install libmotif-common libmotif-dev $ tar zxf ~/adiosvm/plotterpackages/grace-5.1.25.tar.gz $ cd grace-5.1.25 $ ./configure --prefix=/opt/plotter $ make -j 4 $ sudo make install -- use XMGRACE=/opt/plotter/grace in Makefile.adiosVM of the plotter package below If not installed Visit yet: --------------------------- $ sudo apt-get install libglu1-mesa-dev libxt-dev $ mkdir ~/Software/plotter $ cd ~/Software/plotter 1. Build Mesa library (libosmesa6-dev package does not seem to let vtk build) $ cd ~/Software/plotter $ tar zxf ~/adiosvm/plotterpackages/Mesa-7.8.2.tar.gz $ cd Mesa-7.8.2 $ ./configure CFLAGS="-I/usr/include/i386-linux-gnu -O2 -DUSE_MGL_NAMESPACE -fPIC -DGLX_USE_TLS" CXXFLAGS="-O2 -DUSE_MGL_NAMESPACE -fPIC -DGLX_USE_TLS" --prefix=/opt/plotter --with-driver=osmesa --disable-driglx-direct $ make -j 4 $ sudo make install 2. Build an old VTK library $ tar zxf ~/adiosvm/plotterpackages/visit-vtk-5.8.tar.gz $ mkdir vtk-5.8-build $ cd vtk-5.8-build $ export LD_LIBRARY_PATH=/opt/plotter/lib:$LD_LIBRARY_PATH $ cmake -DCMAKE_INSTALL_PREFIX:PATH=/opt/plotter -DCMAKE_C_FLAGS=-DGLX_GLXEXT_LEGACY -DCMAKE_CXX_FLAGS=-DGLX_GLXEXT_LEGACY -Wno-dev ../visit-vtk-5.8 Check if CMakeCache.txt has VTK_OPENGL_HAS_OSMESA:BOOL=ON, if not, turn on and rerun cmake. It has to find the Mesa options and have this in CMakeCache.txt //Path to a file. OSMESA_INCLUDE_DIR:PATH=/opt/plotter/include //Path to a library. OSMESA_LIBRARY:FILEPATH=/opt/plotter/lib/libOSMesa.so ... //Use mangled Mesa with OpenGL. VTK_USE_MANGLED_MESA:BOOL=OFF <-- This is OFF!! $ make -j 4 $ sudo make install 3. Build plotter $ cd ~/Software $ svn co https://svn.ccs.ornl.gov/svn-ewok/wf.src/trunk/plotter ...OR... $ tar zxf ~/adiosvm/plotterpackages/plotter.tar.gz $ cd plotter Edit Makefile.adiosVM to point to the correct ADIOS, HDF5 (sequential), NetCDF (sequential), Grace and VTK libraries. $ make $ sudo INSTALL_DIR=/opt/plotter INSTALL_CMD=install make install In ~/.bashrc, add to PATH "/opt/plotter/bin" VIII. Clean-up a bit ================== Not much space left after building visit and plotter. You can remove this big offenders 1.9GB ~/Software/plotter/vtk-5.8-build 1.3GB ~/Software/visit/qt-everywhere-opensource-src-4.8.3 ~/Software/visit/qt-everywhere-opensource-src-4.8.3.tar.gz ~/Software/visit/VTK*-build IX. Installing R and pbdR =========================== This is for the pbdR tutorial. Not required for an ADIOS-only tutorial. Packages that will be needed: $ sudo apt-get install libcurl4-gnutls-dev Install R --------- See instructions on http://cran.r-project.org/bin/linux/ubuntu/ $ sudo vi /etc/apt/sources.list add a line to the end: deb http://mirrors.nics.utk.edu/cran/bin/linux/ubuntu xenial/ Add the key for this mirror $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 $ sudo apt-get update $ sudo apt-get install r-base r-base-dev $ sudo apt-get install openssl libssl-dev libssh2-1-dev libcurl4-openssl-dev If these above don't work, you need to resort building from source. from http://cran.r-project.org/sources.html e.g. http://cran.r-project.org/src/base/R-3/R-3.1.2.tar.gz --- non-SUDO version below, SUDO below --- $ sudo mkdir -p /opt/R/library $ sudo chgrp -R adios /opt/R $ sudo chmod -R g+w /opt/R $ export R_LIBS_USER=/opt/R/library $ R install.packages("ggplot2", repos="https://cran.cnr.berkeley.edu/", lib="/opt/R/library") install.packages("rlecuyer", repos="http://mirrors.nics.utk.edu/cran/", lib="/opt/R/library") install.packages("raster", repos="http://mirrors.nics.utk.edu/cran/", lib="/opt/R/library") install.packages("devtools", repos="http://mirrors.nics.utk.edu/cran/") library(devtools) install_github("RBigData/pbdMPI") install_github("RBigData/pbdSLAP") install_github("wrathematics/RNACI") install_github("RBigData/pbdBASE") install_github("RBigData/pbdDMAT") install_github("RBigData/pbdDEMO") install_github("RBigData/pbdPAPI") install.packages("pbdPROF", repos="http://mirrors.nics.utk.edu/cran/") quit() $ cd ~/Software $ git clone https://github.com/sgn11/pbdADIOS.git $ R CMD INSTALL pbdADIOS --configure-args="--with-adios-home=/opt/adios1/" --no-test-load Add to ~/.bashrc export R_LIBS_USER=/opt/R/library --- SUDO version below, non-SUDO above --- Install 'rlecuyer' ------------------ from http://cran.r-project.org/web/packages/rlecuyer/index.html $ sudo R > install.packages("rlecuyer", repos="http://mirrors.nics.utk.edu/cran/") should get messages ending with: * DONE (rlecuyer) > q() quit R (or ctrl+D) Test if it is installed correctly: $ R > library(help=rlecuyer) > q() > install.packages("raster", repos="http://mirrors.nics.utk.edu/cran/") Install pbdR ------------- $ sudo R install.packages("devtools", repos="http://mirrors.nics.utk.edu/cran/") library(devtools) install_github("RBigData/pbdMPI") install_github("RBigData/pbdSLAP") install_github("wrathematics/RNACI") install_github("RBigData/pbdBASE") install_github("RBigData/pbdDMAT") install_github("RBigData/pbdDEMO") install_github("RBigData/pbdPAPI") install.packages("pbdPROF", repos="http://mirrors.nics.utk.edu/cran/") > q() Install pbdADIOS ---------------- $ cd ~/Software $ git clone https://github.com/sgn11/pbdADIOS.git $ sudo R CMD INSTALL pbdADIOS --configure-args="--with-adios-home=/opt/adios1/" --no-test-load -- quick test $ R > library(pbdADIOS) Loading required package: pbdMPI Loading required package: rlecuyer > quit() --- end of SUDO version above --- Quick test of pbdR ------------------ $ cd ~/adiosvm $ mpirun -np 2 Rscript test_pbdR.r COMM.RANK = 0 [1] 0 COMM.RANK = 1 [1] 1 Download pbdR tutorial examples ------------------------------- $ cd $ wget https://github.com/RBigData/RBigData.github.io/blob/master/tutorial/scripts.zip?raw=true $ unzip scripts.zip?raw=true or get it using a browser from http://r-pbd.org/tutorial Quick test of scripts: ---------------------- $ scripts/pbdMPI/quick_examples $ mpirun -np 4 Rscript 1_rank.r COMM.RANK = 0 [1] 0 COMM.RANK = 1 [1] 1 COMM.RANK = 2 [1] 2 COMM.RANK = 3 [1] 3 Install R Studio ---------------- $ sudo apt-get install libjpeg62 $ cd adiosvm/Rpackages if not there, get it from the web $ wget http://download1.rstudio.org/rstudio-0.98.1091-i386.deb $ sudo dpkg -i rstudio-0.98.1091-i386.deb VII. Others =========== Enable GDB debugging -------------------- See http://askubuntu.com/questions/146160/what-is-the-ptrace-scope-workaround-for-wine-programs-and-are-there-any-risks Ubuntu prohibits ptrace to see other processes so gdb will fail with permissions. Enable GDB binary to see your processes: $ sudo apt-get install libcap2-bin $ sudo setcap cap_sys_ptrace=eip /usr/bin/gdb GLOBAL/GTAGS ------------- GTAGS is useful for quickly finding definitions of functions in source codes in VIM It needs to be installed from source to make it work in VIM $ cd ~/Software $ tar zxf ~/adiosvm/linuxpackages/global-6.3.4.tar.gz $ cd global-6.3.4 $ ./configure $ make $ sudo make install $ mkdir -p ~/.vim/plugin $ cp ~/adiosvm/linuxpackages/gtags.vim ~/.vim/plugin/ NOTE: .vimrc already contains the flag to turn it on Generate the tags for a source $ cd ~/Software/ADIOS/src $ gtags $ ls G* GPATH GRTAGS GSYMS GTAGS $ vi write/adios_posix.c /adios_posix_read_version Hit Ctrl-\ Ctrl-\ Vi should open new file core/adios_bp_v1.c and jump to the definition of adios_posix_read_version :b# -- to go back to the previous file If there are multiple finds, you can move among them with Ctrl-\ Ctrl-[ and Ctrl-\ Ctrl-] The commands can be modified at the bottom of ~/.vim/plugin/gtags.vim Add a path of any software so that gtags will search the GTAGS file there to find external references. In .bashrc, add export GTAGSLIBPATH=/home/adios/Software/ADIOS/src You also need to run gtags in your application (top) source directory too, to get global/gtag working.
Building on top of the latest adios2
, the python package installed via conda
, I successfully ran:
$ ./build/gray-scott simulation/settings-files.json
But when following the instructions to create a plot, I obtain the error:
$ python3 plot/gsplot.py -i gs.bp
Traceback (most recent call last):
File "plot/gsplot.py", line 105, in <module>
fr = adios2.open(args.instream, "r", mpi.comm_app,"adios2.xml", "SimulationOutput")
TypeError: open(): incompatible function arguments. The following argument types are supported:
1. (name: str, mode: str, engine_type: str='BPFile') -> adios2::py11::File
2. (name: str, mode: str, config_file: str, io_in_config_file: str) -> adios2::py11::File
Invoked with: 'gs.bp', 'r', <mpi4py.MPI.Intracomm object at 0x7f1e9c6a5d30>, 'adios2.xml', 'SimulationOutput'
Other relevant info:
$ python3 --version
Python 3.7.5
$ conda --version
conda 4.7.12
I am building toward the Gray-Scott in situ pipeline. I cannot import adios2. I can import mpi4py , but import adios2 results in:
Traceback (most recent call last):
File "import_adios.py", line 5, in
import adios2
ImportError: ERROR: mpi4py not loaded correctly
I am running python3, openmpi, py-mpi4py, py-numpy from Spack.
My ADIOS2 cmake reported:
-- Found MPI_C: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/openmpi-3.1.4-6icbx4wzmo4tuzfidmwtqu2takxv265z/lib/libm
pi.so (found version "3.1")
-- Found MPI_CXX: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/openmpi-3.1.4-6icbx4wzmo4tuzfidmwtqu2takxv265z/lib/li
bmpi_cxx.so (found version "3.1")
-- Found MPI_Fortran: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/openmpi-3.1.4-6icbx4wzmo4tuzfidmwtqu2takxv265z/li
b/libmpi_usempif08.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: C Fortran CXX
-- Found PythonInterp: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/python-3.7.3-r6r7ezh2rfmjlsxrivsa2lx3g5mozxzy/bin/python3 (found version "3.7.3")
-- Found PythonLibs: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/python-3.7.3-r6r7ezh2rfmjlsxrivsa2lx3g5mozxzy/lib/libpython3.7m.so (found version "3.7.3")
-- Found PythonModule_numpy: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/py-numpy-1.16.2-nc6pfi2ejbmmytfaewoytodexakcvpdm/lib/python3.7/site-packages/numpy
-- Found PythonModule_mpi4py: /usb1/wozniak/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/py-mpi4py-3.0.1-frisvuxcphcxkwrybf4ptv7k7bfcatbv/lib/python3.7/site-packages/mpi4py
At runtime, I do:
spack load openmpi
spack load [email protected]
spack load py-numpy
spack load py-mpi4py
then append the ADIOS2 PYTHONPATH and LD_LIBRARY_PATH entries. What else can I check?
load modules
module unload intel-mkl/2017.3.196-v7uuj6z
module load intel/17.0.4-74uvhji
module load intel-mpi/2017.3-dfphq6k
module load libpsm2/10.3-17
module load cmake/3.12.2-4zllpyo
module load jdk/8u141-b15-mopj6qr
module load tcl/8.6.6-x4wnbsg
module load bzip2/1.0.6-hcvyuh5
module load zlib/1.2.11-6632jqd
install ADIOS2
export ROOT=$PWD/install
git clone https://github.com/ornladios/ADIOS2.git
cd ADIOS2
mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=$ROOT/adios2 ..
make -j8
make install
cd ../..
export PATH=$ROOT/adios2/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOT/adios2/lib64
Build the application
git clone https://github.com/pnorbert/adiosvm.git
cd adiosvm/Tutorial/gray-scott
mkdir build && cd build
cmake -DADIOS2_DIR=$ROOT/adios2 ..
make -j8
cd ..
Run the application
Set the engine type of simulation as SST
$ mpirun -n 4 build/gray-scott simulation/settings.json
Simulation writes data using engine type: SST
#========================================
grid: 48x48x48
steps: 600
plotgap: 25
F: 0.02
k: 0.048
dt: 1
Du: 0.2
Dv: 0.1
noise: 0.01
output: gs.bp
adios_config: adios2.xml
decomposition: 2x2x1
grid per process: 24x24x48
#========================================
Simulation at step 0 writing output step 0
terminate called after throwing an instance of 'std::invalid_argument'
terminate called after throwing an instance of 'std::invalid_argument'
terminate called after throwing an instance of 'std::invalid_argument'
terminate called after throwing an instance of 'std::invalid_argument'
what(): ERROR: Engine derived class SstWriter doesn't implement function DoPut
what(): ERROR: Engine derived class SstWriter doesn't implement function DoPut
what(): ERROR: Engine derived class SstWriter doesn't implement function DoPut
gray-scott:120537 terminated with signal 6 at PC=2b2622ee1207 SP=7ffc5b80e1d8. Backtrace:
what(): ERROR: Engine derived class SstWriter doesn't implement function DoPut
gray-scott:120538 terminated with signal 6 at PC=2abb680cb207 SP=7ffcd35bce68. Backtrace:
gray-scott:120539 terminated with signal 6 at PC=2ac17f721207 SP=7ffc878d2bd8. Backtrace:
gray-scott:120536 terminated with signal 6 at PC=2b027ce3a207 SP=7ffc5c54fcf8. Backtrace:
/lib64/libc.so.6(gsignal+0x37)[0x2b027ce3a207]
/lib64/libc.so.6(abort+0x148)[0x2b027ce3b8f8]
/lib64/libc.so.6(gsignal+0x37)[0x2b2622ee1207]
/lib64/libc.so.6(abort+0x148)[0x2b2622ee28f8]
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x2b26226ec7d5]
/lib64/libc.so.6(gsignal+0x37)[0x2abb680cb207]
/lib64/libc.so.6(abort+0x148)[0x2abb680cc8f8]
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x2abb678d67d5]
/lib64/libc.so.6(gsignal+0x37)[0x2ac17f721207]
/lib64/libc.so.6(abort+0x148)[0x2ac17f7228f8]
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x2ac17ef2c7d5]
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x2b027c6457d5]
/lib64/libstdc++.so.6(+0x5e746)[0x2b027c643746]
/lib64/libstdc++.so.6(+0x5e746)[0x2b26226ea746]
/lib64/libstdc++.so.6(+0x5e746)[0x2abb678d4746]
/lib64/libstdc++.so.6(+0x5e746)[0x2ac17ef2a746]
/lib64/libstdc++.so.6(+0x5e773)[0x2b26226ea773]
/lib64/libstdc++.so.6(+0x5e773)[0x2abb678d4773]
/lib64/libstdc++.so.6(+0x5e773)[0x2ac17ef2a773]
/lib64/libstdc++.so.6(+0x5e773)[0x2b027c643773]
/lib64/libstdc++.so.6(+0x5e993)[0x2b027c643993]
/lib64/libstdc++.so.6(+0x5e993)[0x2b26226ea993]
/lib64/libstdc++.so.6(+0x5e993)[0x2abb678d4993]
/lib64/libstdc++.so.6(+0x5e993)[0x2ac17ef2a993]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZNK6adios24core6Engine7ThrowUpESs+0x102)[0x2b027a8bada2]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZNK6adios24core6Engine7ThrowUpESs+0x102)[0x2b2620961da2]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZNK6adios24core6Engine7ThrowUpESs+0x102)[0x2abb65b4bda2]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZNK6adios24core6Engine7ThrowUpESs+0x102)[0x2ac17d1a1da2]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZN6adios24core6Engine5DoPutERNS0_8VariableIdEERNS0_4SpanIdEEmRKd+0x2d)[0x2b027a8bb89d]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZN6adios24core6Engine5DoPutERNS0_8VariableIdEERNS0_4SpanIdEEmRKd+0x2d)[0x2b262096289d]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZN6adios24core6Engine5DoPutERNS0_8VariableIdEERNS0_4SpanIdEEmRKd+0x2d)[0x2abb65b4c89d]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(_ZN6adios24core6Engine5DoPutERNS0_8VariableIdEERNS0_4SpanIdEEmRKd+0x2d)[0x2ac17d1a289d]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios24core6Engine3PutIdEERNS0_8VariableIT_E4SpanERS5_mRKS4+0x135)[0x2b027a8d2d05]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios24core6Engine3PutIdEERNS0_8VariableIT_E4SpanERS5_mRKS4+0x135)[0x2b2620979d05]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios24core6Engine3PutIdEERNS0_8VariableIT_E4SpanERS5_mRKS4+0x135)[0x2abb65b63d05]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios24core6Engine3PutIdEERNS0_8VariableIT_E4SpanERS5_mRKS4+0x135)[0x2ac17d1b9d05]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4_mRKS3+0xb9)[0x2abb65e22929]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4_mRKS3+0xb9)[0x2ac17d478929]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4_mRKS3+0xb9)[0x2b027ab91929]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4_mRKS3+0xb9)[0x2b2620c38929]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4+0x1a)[0x2abb65e229ba]
build/gray-scott[0x405d45]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4+0x1a)[0x2ac17d4789ba]
build/gray-scott[0x405d45]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4+0x1a)[0x2b027ab919ba]
build/gray-scott[0x405d45]
/home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2(ZN6adios26Engine3PutIdEENS_8VariableIT_E4SpanES4+0x1a)[0x2b2620c389ba]
build/gray-scott[0x405d45]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2abb680b73d5]
build/gray-scott[0x4041a9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac17f70d3d5]
build/gray-scott[0x4041a9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b027ce263d5]
build/gray-scott[0x4041a9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b2622ecd3d5]
build/gray-scott[0x4041a9]
gray-scott:120539 terminated with signal 11 at PC=2ac1864e75bf SP=2ac18a319e20. Backtrace:
SST stream open at exit, unlinking contact file /blues/gpfs/home/tshu/project/bebop/HPC_Autotuner/adiosvm/Tutorial/gray-scott/gs.bp.sst
$ ldd gray-scott
linux-vdso.so.1 => (0x00007ffde95aa000)
libadios2.so.2 => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2.so.2 (0x00002af278e06000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002af279584000)
libmpifort.so.12 => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/intel-mpi-2017.3-dfphq6kavje2olnichisvjjndtridrok/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib/libmpifort.so.12 (0x00002af2797a0000)
libmpi.so.12 => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/intel-mpi-2017.3-dfphq6kavje2olnichisvjjndtridrok/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib/libmpi.so.12 (0x00002af279b49000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002af27a871000)
librt.so.1 => /lib64/librt.so.1 (0x00002af27aa75000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00002af27ac7d000)
libm.so.6 => /lib64/libm.so.6 (0x00002af27af84000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002af27b286000)
libc.so.6 => /lib64/libc.so.6 (0x00002af27b49c000)
libadios2_sst.so.2 => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2_sst.so.2 (0x00002af27b869000)
libbz2.so.1.0 => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/intel-17.0.4/bzip2-1.0.6-hcvyuh5a7fzugpdvauilafgcvhqc2ajb/lib/libbz2.so.1.0 (0x00002af27ba88000)
libtaustubs.so => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libtaustubs.so (0x00002af27bcd8000)
/lib64/ld-linux-x86-64.so.2 (0x00002af278be2000)
libfabric.so.1 => /lib64/libfabric.so.1 (0x00002af27bedc000)
libadios2_evpath.so => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2_evpath.so (0x00002af27c233000)
libadios2_ffs.so.1 => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2_ffs.so.1 (0x00002af27c4a7000)
libadios2_atl.so.2 => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/libadios2_atl.so.2 (0x00002af27c70d000)
libimf.so => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-17.0.4-74uvhjiulyqgvsmywifbbuo46v5n42xc/compilers_and_libraries_2017.4.193/linux/compiler/lib/intel64_lin/libimf.so (0x00002af27c91b000)
libsvml.so => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-17.0.4-74uvhjiulyqgvsmywifbbuo46v5n42xc/compilers_and_libraries_2017.4.193/linux/compiler/lib/intel64_lin/libsvml.so (0x00002af27ce08000)
libirng.so => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-17.0.4-74uvhjiulyqgvsmywifbbuo46v5n42xc/compilers_and_libraries_2017.4.193/linux/compiler/lib/intel64_lin/libirng.so (0x00002af27dd21000)
libintlc.so.5 => /blues/gpfs/home/software/spack-0.10.1/opt/spack/linux-centos7-x86_64/gcc-4.8.5/intel-17.0.4-74uvhjiulyqgvsmywifbbuo46v5n42xc/compilers_and_libraries_2017.4.193/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00002af27e096000)
librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00002af27e301000)
libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00002af27e518000)
libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00002af27e72f000)
libpsm_infinipath.so.1 => /lib64/libpsm_infinipath.so.1 (0x00002af27e99c000)
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00002af27ebf2000)
libpsm2.so.2 => /soft/bebop/libpsm2/10.3-17/usr/lib64/libpsm2.so.2 (0x00002af27ee13000)
libadios2_dill.so.2 => /home/tshu/project/bebop/HPC_Autotuner/install/adios2/lib64/../lib64/libadios2_dill.so.2 (0x00002af27f072000)
libinfinipath.so.4 => /lib64/libinfinipath.so.4 (0x00002af27f2a7000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00002af27f4b6000)
libnuma.so.1 => /lib64/libnuma.so.1 (0x00002af27f6bb000)
I am trying to build the Gray-Scott example using a Spack-based MPI and mpi4py. I was able to build ADIOS2 with these. It does not find the desired MPI, or even the mpicc/mpicxx in PATH. Do you know how to do this? Thanks
$ MPI=$HOME/proj/spack/clean/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/openmpi-3.1.4-uwu4odepmdk5p7ajh26eti5q6y3iry56
$ MPI_C_LIBRARIES=$MPI/lib/libmpi.so
$ MPI_CXX_INCLUDE=$MPI/include
$ cmake -DADIOS2_DIR=$ADIOS -DMPI_C_LIBRARIES=$MPI_C_LIBRARIES -DMPI_C_INCLUDE=$MPI_C_INCLUDE ..
-- The C compiler identification is GNU 7.4.0
...
-- Found MPI_C: /usb1/wozniak/sfw/mpich-3.2.1/lib/libmpi.so (found version "3.1")
-- Could NOT find MPI_CXX (missing: MPI_CXX_WORKS)
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find MPI (missing: MPI_CXX_FOUND) (found version "3.1")
Call Stack (most recent call first):
/usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.10/Modules/FindMPI.cmake:1444 (find_package_handle_standard_args)
CMakeLists.txt:7 (find_package)
Compiling the gray-scott...
$ make
Scanning dependencies of target gray-scott
Scanning dependencies of target pdf_calc
[ 16%] Building CXX object CMakeFiles/pdf_calc.dir/analysis/pdf_calc.cpp.o
[ 33%] Building CXX object CMakeFiles/gray-scott.dir/simulation/gray-scott.cpp.o
[ 50%] Building CXX object CMakeFiles/gray-scott.dir/simulation/main.cpp.o
[ 66%] Building CXX object CMakeFiles/gray-scott.dir/simulation/settings.cpp.o
/home/tshu/project/MPI_in_MPI/t7810/adiosvm/Tutorial/gray-scott/analysis/pdf_calc.cpp: In function ‘int main(int, char**)’:
/home/tshu/project/MPI_in_MPI/t7810/adiosvm/Tutorial/gray-scott/analysis/pdf_calc.cpp:212:59: error: ‘NextAvailable’ is not a member of ‘adios2::StepMode’
adios2::StepStatus read_status = reader.BeginStep(adios2::StepMode::NextAvailable, 10.0f);
^
CMakeFiles/pdf_calc.dir/build.make:62: recipe for target 'CMakeFiles/pdf_calc.dir/analysis/pdf_calc.cpp.o' failed
make[2]: *** [CMakeFiles/pdf_calc.dir/analysis/pdf_calc.cpp.o] Error 1
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/pdf_calc.dir/all' failed
make[1]: *** [CMakeFiles/pdf_calc.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 83%] Linking CXX executable gray-scott
[ 83%] Built target gray-scott
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
We get segfault running the brusselator on Summit early in the run process with gcc/6.4.0. This appears to be happening in the heat2d_fft code, in the call to MPI_CART_SUB. I can reproduce this with a vanilla installation of 2dcomp_fft with the gnu compiler (but not with xl.) It looks like the MPI call is being redirected to a PMPI call, and I suspect something is broken on the installation there. I have a minimal example that reproduces the issue, and I am submitting an OLCF ticket.
I am getting the following error using git master in ADIOS2 and adiosvm - do I need to use some other versions?
$ adiosvm/Tutorial/gray-scott/build> make
[ 16%] Building CXX object CMakeFiles/gray-scott.dir/simulation/main.cpp.o
/home/wozniak/proj/adiosvm/Tutorial/gray-scott/simulation/main.cpp: In function ‘int main
(int, char**)’:
/home/wozniak/proj/adiosvm/Tutorial/gray-scott/simulation/main.cpp:74:69: error: no match
ing function for call to ‘adios2::ADIOS::ADIOS(std::__cxx11::string&, ompi_communicator_t
*&, const bool&)’
adios2::ADIOS adios(settings.adios_config, comm, adios2::DebugON);
^
In file included from /home/wozniak/Public/sfw/ADIOS2/include/adios2.h:18:0,
from /home/wozniak/proj/adiosvm/Tutorial/gray-scott/simulation/main.cpp:5:
/home/wozniak/Public/sfw/ADIOS2/include/cxx11/ADIOS.h:104:5: note: candidate: adios2::ADIOS::ADIOS(adios2::ADIOS&&)
ADIOS(ADIOS &&) = default;
^~~~~
/home/wozniak/Public/sfw/ADIOS2/include/cxx11/ADIOS.h:104:5: note: candidate expects 1 argument, 3 provided
I fail to compile heat2d/cpp . Could you please help me fix the compiling issue? Thank you very much!
in make.settings:
ADIOS2_DIR=.../install/adios2
CXX=mpicxx
in Makefile
override ADIOS2_INC=-I${ADIOS2_DIR}/include
override ADIOS2_LIB=-L${ADIOS2_DIR}/lib64
$ make all
mpicxx -g -std=gnu++11 -O2 -c -I/lcrc/project/PACC/tong/HPC_Autotuner/install/adios2/include -o simulation/HeatTransfer.o simulation/HeatTransfer.cpp
mpicxx -g -std=gnu++11 -O2 -c -I/lcrc/project/PACC/tong/HPC_Autotuner/install/adios2/include -o simulation/IO_adios2.o simulation/IO_adios2.cpp
mpicxx -g -std=gnu++11 -O2 -c -I/lcrc/project/PACC/tong/HPC_Autotuner/install/adios2/include -o simulation/Settings.o simulation/Settings.cpp
mpicxx -g -std=gnu++11 -O2 -c -I/lcrc/project/PACC/tong/HPC_Autotuner/install/adios2/include -o simulation/heatSimulation.o simulation/heatSimulation.cpp
mpicxx -g -std=gnu++11 -O2 -o heatSimulation simulation/HeatTransfer.o simulation/IO_adios2.o simulation/Settings.o simulation/heatSimulation.o -L/lcrc/project/PACC/tong/HPC_Autotuner/install/adios2/lib64
simulation/IO_adios2.o: In function IO::~IO()': /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:89: undefined reference to
adios2::Engine::Close(int)'
simulation/IO_adios2.o: In function IO::write(int, HeatTransfer const&, Settings const&, int)': /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:109: undefined reference to
adios2::Engine::BeginStep()'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:115: undefined reference to void adios2::Engine::Put<double>(adios2::Variable<double>, double const*, adios2::Mode)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:116: undefined reference to
adios2::Engine::EndStep()'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:99: undefined reference to adios2::Engine::BeginStep()' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:101: undefined reference to
adios2::Variable::Span adios2::Engine::Put(adios2::Variable)'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:103: undefined reference to adios2::detail::Span<double>::data() const' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:105: undefined reference to
adios2::Engine::EndStep()'
simulation/IO_adios2.o: In function IO::IO(Settings const&, int)': /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:25: undefined reference to
adios2::ADIOS::ADIOS(std::string const&, int, bool)'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:27: undefined reference to adios2::ADIOS::DeclareIO(std::string)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:28: undefined reference to
adios2::IO::InConfigFile() const'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:32: undefined reference to adios2::IO::SetEngine(std::string)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:33: undefined reference to
adios2::IO::SetParameters(std::map<std::string, std::string, std::lessstd::string, std::allocator<std::pair<std::string const, std::string> > > const&)'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:37: undefined reference to adios2::IO::AddTransport(std::string, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:53: undefined reference to
adios2::Variable adios2::IO::DefineVariable(std::string const&, std::vector<unsigned long, std::allocator > const&, std::vector<unsigned long, std::allocator > const&, std::vector<unsigned long, std::allocator > const&, bool)'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:56: undefined reference to adios2::Attribute<std::string> adios2::IO::DefineAttribute<std::string>(std::string const&, std::string const&, std::string const&, std::string)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:58: undefined reference to
adios2::Variable::Name() const'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:58: undefined reference to adios2::Attribute<std::string> adios2::IO::DefineAttribute<std::string>(std::string const&, std::string const&, std::string const&, std::string)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:77: undefined reference to
adios2::Attributestd::string adios2::IO::DefineAttributestd::string(std::string const&, std::string const&, std::string const&, std::string)'
/lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:79: undefined reference to adios2::IO::Open(std::string const&, adios2::Mode, int)' /lcrc/project/PACC/tong/HPC_Autotuner/adiosvm/Tutorial/heat2d/cpp/simulation/IO_adios2.cpp:84: undefined reference to
adios2::Engine::LockWriterDefinitions()'
collect2: error: ld returned 1 exit status
make: *** [heatSimulation] Error 1
Could you please add
import matplotlib
matplotlib.use('agg')
before "import matplotlib.pyplot as plt" in gsplot.py and pdfplot.py ?
Otherwise, the following issue appear.
$ mpiexec -n 1 python3 plot/gsplot.py -i gs.bp -o BPgsplot/img
GS Plot step 0 processing simulation output step 0 or computation step 10
/soft/anaconda3/5.2.0/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
Traceback (most recent call last):
File "plot/gsplot.py", line 139, in
Plot2D ('yz', data, args, fullshape, sim_step[0], fontsize)
File "plot/gsplot.py", line 42, in Plot2D
fig = plt.figure(1, figsize=(8,8))
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/pyplot.py", line 548, in figure
**kwargs)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
return cls.new_figure_manager_given_figure(num, fig)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 167, in new_figure_manager_given_figure
canvas = cls.FigureCanvas(figure)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 24, in init
super(FigureCanvasQTAgg, self).init(figure=figure)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backends/backend_qt5.py", line 234, in init
_create_qApp()
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backends/backend_qt5.py", line 125, in _create_qApp
raise RuntimeError('Invalid DISPLAY variable')
RuntimeError: Invalid DISPLAY variable
$ mpiexec -n 2 python3 plot/pdfplot.py -i pdf.bp -o BPpdfplot/fig
PDF Plot step 0 processing analysis step 0 simulation step 10
/soft/anaconda3/5.2.0/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
/soft/anaconda3/5.2.0/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216, got 192
return f(*args, **kwds)
Traceback (most recent call last):
File "plot/pdfplot.py", line 113, in
PlotPDF (pdf, bins, args, start, count, sim_step[0], fontsize)
File "plot/pdfplot.py", line 35, in PlotPDF
fig = plt.figure(1, figsize=(8,8))
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/pyplot.py", line 548, in figure
**kwargs)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
return cls.new_figure_manager_given_figure(num, fig)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 167, in new_figure_manager_given_figure
canvas = cls.FigureCanvas(figure)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backends/backend_qt5agg.py", line 24, in init
super(FigureCanvasQTAgg, self).init(figure=figure)
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backends/backend_qt5.py", line 234, in init
_create_qApp()
File "/soft/anaconda3/5.2.0/lib/python3.6/site-packages/matplotlib/backends/backend_qt5.py", line 125, in _create_qApp
raise RuntimeError('Invalid DISPLAY variable')
RuntimeError: Invalid DISPLAY variable
I run the simulation gray-scott in Bebop. It can work over the following parameter values for the number of processes and the number of processes per node.
$ mpiexec -n 162 -ppn 31
$ mpiexec -n 68 -ppn 13
$ mpiexec -n 64 -ppn 32
simulation/settings-files.json:
{
"L": 512,
"Du": 0.2,
"Dv": 0.1,
"F": 0.01,
"k": 0.05,
"dt": 2.0,
"plotgap": 10,
"steps": 250,
"noise": 0.0000001,
"output": "gs.bp",
"checkpoint": false,
"checkpoint_freq": 10,
"checkpoint_output": "gs_ckpt.bp",
"adios_config": "adios2.xml",
"adios_span": false,
"adios_memory_selection": false,
"mesh_type": "image"
}
However, gray-scott cannot work for some reasonable parameter values in terms of the number of processes and the number of processes per node. For example,
$ mpiexec -n 73 -ppn 26 -hosts bdw-0034,bdw-0478,bdw-0514 ./build/gray-scott simulation/settings-files.json > output_gray-scott.txt 2>&1
Fatal error in PMPI_Type_vector: Invalid argument, error stack:
PMPI_Type_vector(163): MPI_Type_vector(count=514, blocklength=-64, stride=-62, MPI_DOUBLE, new_type_p=0x7ffeb16750a8) failed
PMPI_Type_vector(128): Invalid value for blocklen, must be non-negative but is -64
gray-scott cannot work for the following parameter values either.
$ mpiexec -n 304 -ppn 34
$ mpiexec -n 536 -ppn 35
Could you please help fix this issue and let me know the reason? Thank you very much for your help!
Split the MPI communicator in the Gray-Scott code to support InSituMPI.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.