simone-silvestri / oceanscalingtests.jl

Scaling experiments for the multi-GPU GCM mode of Oceananigans.jl

1. Add Julia to path export PATH=/TO/JULIA:${PATH}
2. EXPERIMENT:
   • "Quiescent" no flow, no bathymetry and zero flux boundary conditions
   • "DoubleDrake" setup similar to https://doi.org/10.1175/2009JCLI3197.1, default
   • "RealisticOcean" a realistic ocean setup with initial conditions and boundary forcing from ECCO2 Version 4 climatological data. It assumes data is available to load in the data folder
3. PROFILE: 0 (run simulation), 1 (profile 100 time steps marked with NVTX, default)
4. NZ: vertical levels (default 120)
5. WITHFLUXES: 0, 1 (only relevant for RealisticOcean, default 1)
6. RESTART: restart file iterations (only relevant for RealisticOcean, default "")

Satori

EXPERIMENT="DoubleDrake" RESOLUTION=6 sbatch -N1 satori_job.sh

Packages:

• Oceananigans.jl.git#ss-vc/overlay

Setup

• Julia 1.9.0-rc2
• Instantiate Julia environment JULIA_LOAD_PATH=:$(pwd)/perlmutter julia --project=. -e "import Pkg; Pkg.instantiate()"
• Warmup GPUCompiler cache JULIA_LOAD_PATH=:$(pwd)/perlmutter julia --project=. experiments/run.jl

Example

vorticity (left) and free surface (right) for the "DoubleDrake" experiment at resolution = 3 ran on 2 mpi processors (9 years integration)

double_drake_LQ.mp4