ncar / xtar Goto Github PK

When using HPC cluster, compute node hosting jupyternotebook mounts the tar file through ratarmount, but the dask worker nodes on other compute node does not.

Hi @andersy005 , @kmpaul
https://github.com/tinaok/xtar/blob/master/notebooks/tar-netcdf-HPCrealdata.ipynb
I was trying with NFS disk space on our HPC system.
ratarmount installed in my conda environment but can not access; may be I missunderstanding something?

As you can see, I installed libfuse in the same conda environment.

hello @kmpaul @andersy005

I was wondering if it would be possible to have dummy netcdf test data, like the one used on Pangeo-data/benchmarking;

1. With flex input size for total lattitude, longitude and time, and time interval (for example if we say lat=360, lon=180, time=120 hourly, it will create lat: -180, 180 divided in 360 data points, lon: -90 , 90 is divided in 90 points, time: makes hourly data, starts year 2000 Jan, 1st 0:00 and ends on 5th
2. Then separated the file in flex chunk sized netcdf file; in lat_chunk, lon_chunk and time_chunk. for example if we say 10,10,10, which make 1000 netcdf file. 0.0.0.nc: corresponds to lat: -180 to -145 degree, lon: -90, -73 degree time: 2000 Jan 1st 0:00 to 9:00

This way we can bench what is the best tar, and netcdf chunked format to be stored at the mpi computation on oceanophysics or atmospheric computation; to perform Pangeo post-analysis computation later on.

@kmpaul & @tinaok

I just put together a simple notebook demonstrating how to mount the tar archive via ratarmount. I then ran some smoke tests to see what the performance looks like:

• Native/Original netCDF files

• netCDF files in the tar archive

Here's a rendered version of the notebook: https://nbviewer.jupyter.org/github/NCAR/xtar/blob/master/notebooks/tar-netcdf.ipynb. I included dask's performance report in the notebook itself.