uturuncoglu / regesm

Regional Earth System Model

RegESM: Regional Earth System Model

still under testing

Supported Components

• Atmosphere (ATM):
  • RegCM version >4.5 (it needs extra patch for co-processing)
  • WRF version 3.8.1
• Ocean (OCN):
  • ROMS revision 809
  • ROMS-Ice: The driver is not tested with new CICE implementation !!!
  • MITgcm version c63s
• River Routing (RTM):
  • HD version 1.0.2
• Wave (WAV):
  • WAM version Cycle_4.5.3_MPI
• Co-processing (COP):
  • ParaView version 5.4.1: It requires Catalyst module and also GPU with graphics mode enabled.

Prerequisites

• RegESM itself (as a driver)
• Earth System Modeling Framework (ESMF) Library: version 7.1.0
• Model Components (patched and installed with coupling support - see documentation for more information)

Documentation

Table of Contents:

1. Model Design
2. Benchmark
3. Installation
4. Usage
5. Co-processing
6. Limitations
7. Known Issues

Tutorial

A tutorial for two component coupling (RegCM+ROMS) over Black Sea can be found in here. This repository is the same one that is used in First Regional Earth Sistem Modeling Summer School 2016 held in ITU, Turkey.

Talks

• Turuncoglu, U.U. and Mahendra Roppa, 2018. A New Modeling Framework for In-Situ Visualization Integrated in Earth System Models, GTC2018 GPU Technology Conference, March 26-29, 2018, San Jose, CA, USA. pdf

• Turuncoglu, U.U., 2017. Towards in situ visualization integrated model coupling framework for earth system science, Fourth Workshop on Coupling Technologies for Earth System Models, Princeton, NJ, USA. pdf

Co-processing Component Generated Movies

• Vimeo Site

References

Please cite following publications in case of using RegESM coupled modeling system or any variants of it:

• Turuncoglu, U.U., Onol, B., and Ilicak, M., 2019. A new approach for in situ analysis in fully coupled earth system models. In Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV '19). ACM, New York, NY, USA, 6-11, https://doi.org/10.1145/3364228.3364231. link

• Turuncoglu, U.U., 2019. Toward modular in situ visualization in Earth system models: the regional modeling system RegESM 1.1, Geosci. Model Dev., 12, 233-259, https://doi.org/10.5194/gmd-12-233-2019. link

• Turuncoglu, U.U., 2017. Integration of ParaView Catalyst with Regional Earth System Model, Kitware Source (Software Developer's Quarterly), 42, 7-10. link

• Turuncoglu, U.U., Sannino, G., 2016. Validation of newly designed regional earth system model (RegESM) for Mediterranean Basin, Climate Dynamics, 48(9), 2919–2947, doi:10.1007/s00382-016-3241-1 link

Previous publications with early version of the two-component (atmosphere-ocean) coupled model:

• Turuncoglu, U. U., Giuliani, G., Elguindi, N., and Giorgi, F., 2013. Modelling the Caspian Sea and its catchment area using a coupled regional atmosphere-ocean model (RegCM4-ROMS): model design and preliminary results, Geosci. Model Dev., 6, 283-299, doi:10.5194/gmd-6-283-2013 link

Release Notes

Updates

New in Version 1.2

• The COP component is optimized for very-high resolution applications

New in Version 1.1

• New co-processing component is introduced to allow in situ visualization applications for multi-component earth system model configurations
• Updated and revisited model documentation (under doc/ directory)

regesm's Issues

array index problem in mod_utils.F90

Reported by: Vittorio Ruggiero (cineca.it)

The code gives following error and exits for coupling with MITgcm

forrtl: severe (408): fort: (3): Subscript #2 of the array PTRMASK has value 0 which is less than the lower bound of 133

Image PC Routine Line Source
regesm.x 00000000011711E0 Unknown Unknown Unknown
regesm.x 00000000004B716E mod_utils_mp_util 635 mod_utils.F90
regesm.x 000000000042CDC8 mod_esmf_cpl_mp_c 829 mod_esmf_cpl.F90

How Can I Define "Enable PerfCheck" in Namelist.rc

Here comes out another question. After successfully compiled all things regesm is needed and a "regesm.x" excutable file is generated, I run regesm with a command like this "mpirun -n 20 ./regesm.x regcm.in_10km bsea.in". However, it comes out an error like this:
**20180424 100531.164 INFO PET00 Running with ESMF Version 7.1.0r
20180424 100531.167 ERROR PET00 util/mod_config.F90:181 Not found - Passing error in return code
20180424 100531.167 ERROR PET00 regesm.F90:58 Not found - Passing error in return code
20180424 100531.167 INFO PET00 Finalizing ESMF
20180424 100628.222 INFO PET00 Running with ESMF Version 7.1.0r
20180424 100701.114 INFO PET00 Running with ESMF Version 7.1.0r
20180424 100701.116 ERROR PET00 util/mod_config.F90:181 Not found - Passing error in return code
20180424 100701.116 ERROR PET00 regesm.F90:58 Not found - Passing error in return code
20180424 100701.116 INFO PET00 Finalizing ESMF**

I checked the ${regesm_dir}/util/mod_config.F90 and find that this error is probably resulted from the missing definition of "EnablePerfCheck".
call ESMF_ConfigGetAttribute(cf, enablePerfCheck, & label='EnablePerfCheck:', rc=rc) if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, & line=__LINE__, file=FILENAME)) return

ALL SEPARATE TESTS FOR REGCM AND ROMS ARE PASSED (regcm-4.6.1,roms is the summer school version).

The link to your regesm user guide seems to be a dead one.

Problem in model grids under ESMF 7.0.0

The model gives following error message after upgrading the code from ESMF 7.0.0_beta_38 to ESMF 7.0.0.

20160130 211836.877 ERROR PET95 ~~~~~~~~~~~~~~~~~~~~ Degenerate Element Detected ~~~~~~~~~~~~~~~~~~~~
20160130 211836.877 ERROR PET95 degenerate elem. id=149839
20160130 211836.877 ERROR PET95
20160130 211836.877 ERROR PET95 degenerate elem. coords (lon [-180 to 180], lat [-90 to 90]) (x,y,z)
20160130 211836.877 ERROR PET95 -----------------------------------------------------------------
20160130 211836.878 ERROR PET95 0 (31.097986, 44.492306) (0.610826, 0.368445, 0.700813)
20160130 211836.878 ERROR PET95 1 (31.180412, 44.485279) (0.610369, 0.369368, 0.700726)
20160130 211836.878 ERROR PET95 2 (31.180412, 44.485279) (0.610369, 0.369368, 0.700726)
20160130 211836.878 ERROR PET95 3 (31.097986, 44.492306) (0.610826, 0.368445, 0.700813)
20160130 211836.878 ERROR PET95 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
20160130 211836.878 ERROR PET95 ESMCI_Mesh_Regrid_Glue.C:155 c_esmc_regrid_create() Invalid argument - - Src contains a cell that has corners close enough that the cell collapses to a line or point
20160130 211836.896 ERROR PET95 ESMCI_Mesh_Regrid_Glue.C:460 c_esmc_regrid_create() Invalid argument - Internal subroutine call returned Error
20160130 211836.896 ERROR PET95 /home/netapp/clima-users/users/uturunco/progs/esmf-7.0.0/src/Infrastructure/Mesh/src/ESMCI_MeshCap.C:514 MeshCap::regrid_create() Invalid argument - Internal subroutine call returned Error
20160130 211836.896 ERROR PET95 ESMF_Regrid.F90:342 ESMF_RegridStore Invalid argument - Internal subroutine call returned Error
20160130 211836.896 ERROR PET95 ESMF_FieldRegrid.F90:1163 ESMF_FieldRegridStoreNX Invalid argument - Internal subroutine call returned Error
20160130 211836.896 ERROR PET95 mod_esmf_cpl.F90:344 Invalid argument - Passing error in return code
20160130 211836.896 ERROR PET95 OCN-TO-ATM:src/addon/NUOPC/src/NUOPC_Connector.F90:1882 Invalid argument - Passing error in return code
20160130 211836.896 ERROR PET95 regesm:src/addon/NUOPC/src/NUOPC_Driver.F90:1301 Invalid argument - Phase 7 Initialize for connectorComp 2 -> 1: OCN-TO-ATM did not return ESMF_SUCCESS
20160130 211836.896 ERROR PET95 regesm:src/addon/NUOPC/src/NUOPC_Driver.F90:879 Invalid argument - Passing error in return code
20160130 211836.896 ERROR PET95 regesm.F90:107 Invalid argument - Passing error in return code

wind rotation for the projections except LC

I have just realize that the wind rotation part of the driver only valid for Lambert Conformal projection but as you already know that we have more domains now and they are using different map projections. So, i got following list from Laura, ICTP

Africa, Central America and South-East Asia is NORMER
India and South America is ROTMER

In this case, we need to update wind rotation code to support NORMER and ROTMER also. Otherwise, these domains might have some problems in the wind directions (also affects the currents). I did not consider about it before because we were using LC projection only.

The wind rotation code can be found at the end of the mod_esmf_atm.F90 in the driver.

https://github.com/uturuncoglu/RegESM/blob/master/mod_esmf_atm.F90

Anyway, i checked the RegCM code and i found PreProc/ICBC/mod_uvrot.f90 module. I think that it is possible to use uvrot4nx subroutine to implement ROTMER but i could not find any code for NORMER. I also sent a mail to Graziano, ICTP about this issue but he could not replay yet. Somehow, it is necessary to clarify it. I think that this is a major problem for the domains that use NORMER and ROTMER (i have no idea about POLSTR).

Lost configure file

configure file is not in the master branch. Are there any easier way to compile this thing without configure file ？

Matching between interactive and prescribed SST

When the dynamic ocean domain is smaller than the atmospheric domain, there could be regions of the coupled grid which are ocean/sea but are not part of the oceanic grid. In this case the atmosphere reads the SST from an input file. If the prescribed SST is in contact with grid points from the oceanic grid, sharp gradients could be generated as the atmospheric domain merges both SST fields which are not consistent.
A smoothing algorithm along the boundary of the ocean model that reduces SST differences gradually to prevent sharp gradients should be implemented. This should be done at the level of the driver/coupler in order to be generic and support both MITgcm and ROMS.

create doc for RegESM

The initial version of the user guide must be created.

• wiki + pdf file
• simple use case ?

Strange error in MPI_Bcast

The coupled model gives following error

[node80:71487] *** An error occurred in MPI_Bcast
[node80:71487] *** on communicator MPI COMMUNICATOR 26 CREATE FROM 15
[node80:71487] *** MPI_ERR_TRUNCATE: message truncated
[node80:71487] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort

in the ocean component (ROMS) when it is trying to broadcast time data of the forcing fields. The error is same for different MPI implementations (Intel MPI and openMPI).

Observations:

• model does not give any error in two component coupling case (ocn+atm)
• model does not give any error in three component coupling case (ocn+atm+rtm) when MITgcm is selected as an ocean component.
• model does not give any error in three component coupling case when there is no any active river (disabled in namelist.rc)

problem for compile atm-cop (GTC2018_demo)

Dear Sir:
I use my Ubuntu 18.04 installed with gcc, g++, gfortran (v7.5) and mpich2-3.3 to compile the GTC2018_demo tutorial, and some compilation problems happened. Before this, I have installed all the libraries and ATM, OCN, Paraview_5.4 successfully. But When I compile the driver (RegESM_1.1), the errors are:
/usr/bin/ld: cop/libcop.a(adaptor.o): undefined reference to symbol '_ZNSt8ios_base4InitD1Ev@@GLIBCXX_3.4'
/usr/lib/gcc/x86_64-linux-gnu/7//libstdc++.so: error adding symbols: DSO missing from command line

In RegESM, the command I used is:
./bootstrap.sh
./configure --prefix=pwd --with-atm=../../atm/r6146 --with-paraview=/home/lijian/paraview-5.4.1/build CC=gcc FC=gfortran MPIFC=mpif90 CXX=g++

The solution on the Internet tells me to use g++ instead of gcc, which can solve the problem. But I find mpicxx was used to compile the cop including adaptor.cpp and grid.cpp.
Could you please help me to analyze the configuration?
Thanks
Li Jian

implement two step interpolation for unmapped grid points

1. Use bilinear regridding with the atm and ocn masks on to generate a routeHandle. Call this routehandle: RH_B
2. From the pictures it looks like you know which points in the ocean are unmapped that you want mapped. (if not let me know and I think that I have a procedure for finding them). In the mask array for the ocean grid mark all the unmapped points with a new mask number say 3. In that mask array also mark all points that are NOT already masked as land and that are NOT the problem unmapped ocean points with another new mask number say 4. (i.e. mark all the ocean points that are being interpolated correctly as 4).
3. Use NEAREST_STOD interpolation generate a routehandle called RH_N from a field on the ocn grid to the same field on the ocn grid. For the source turn on the land mask and the unmapped point mask (3). For the destination turn on the land mask and the already mapped points mask (4). It should be ok to interpolate from a grid to itself, it might not be ok to interpolate from the same Field to itself. If it complains about having the same Array or Field then we can just create a temporary Field in the middle.
4. To do the interpolation use RH_B to interpolate from the atm to ocn. Then use RH_N to interpolate from ocn to itself. When doing the RH_N, set zeroregion=ESMF_REGION_SELECT to not erase the previos interpolation results.

Another technique to find unmapped points (aside from the unmappedDstList) is to interpolate a constant field (e.g. 1.0) any unmapped points should remain 0.0. I mention this if you don't want to convert the sequence indices in the unmappedDstList to locations in the Grid.

Reproducibility problem ...

TICKET: #3613762: ESMF and reproducibility ...

I am trying to test the reproducibility of the results in coupled model.,For this reason, i am using "-fp-model precise" compiler flag in all the model components and i tested them in standalone mode. I confirmed that the results of the individual components (i.e. atmosphere and wave) are reproducible. Then, i tested,coupled model but i could not reproduce the results. When i compare the,two different run with exactly same configuration, the results are,different. So, i just want to find the source of this issue and i would,like to narrow down the possible sources. My question is that do i need to,compile ESMF also with "-fp-model precise" compiler flag? Is there any,ESMF test to check the reproducibility in ESMF side? What do you suggest,about it? You could find the detail of my work environment as follows

ifort (IFORT) 14.0.1 20131008
openmpi-1.6.5 (compiled with same compiler)
esmf-7.0.0 (beta 38)
CentOS release 6.5 (Final) 2.6.32-431.29.2.el6.x86_64
Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

Failed to compile mod_esmf_esm.F90

When building on Darwin.gfortranclang.64.openmpi.ESMF_7_1_0r with

./bootstrap.sh && CC=clang CXX=clang ./configure  && make

I get

gfortran -Iutil -I/opt/esmf/mod/modg/Darwin.gfortranclang.64.openmpi.ESMF_7_1_0r -I/opt/esmf/include -I/opt/local/include -I/opt/local/include       -O3 -mtune=native -g -O2 -c  mod_esmf_esm.F90
mod_esmf_esm.F90:113:55:

                                  comp=child, rc=rc)
                                                   1
Error: There is no specific subroutine for the generic 'nuopc_driveraddcomp' at (1)

add support for representation of sea-ice in ATM component

The current version of the ATM component (RegCM 4.3.5.6) does not allow to change the land-use type. The model crashes when the land-use type and masking is changed.

problem in field time stamp information in three component case (atm+ocn+rtm)

Ticket: #3613711 (ESMF Support)

ExecuteRouteHandle method of Connector does not working as it is expected. The OCN component is interacting with both ATM (fast time step - every 1 hour) and RTM (slow time step, every day) and OCN
component import state includes the all the fields come from both ATM (i.e. net heat flux) and RTM (i.e. river discharge).

When the code passes from ExecuteRouteHandle method for the case of ATM-OCN direction, it updates the time steps for all the fields (both net heat flux and river discharge) in the field bundle using
"NUOPC_FieldBundleUpdateTime" call (in NUOPC_Connector.F90, around line 1762). The problem is that if the field (i.e. river discharge) does not belong to the components that are interacting (i.e. ATM-OCN), then the code updates the time stamp of the field as full of zeros and only "net heat flux" will have the correct time stamp information.

Then the second ExecuteRouteHandle method will perform for RTM-OCN direction and it follows the same logic. It updates the time stamp of the fields (stored in the OCN import state) and put zeros to the "net heat flux" (it was correct in the previous step) and correct time stamp information to the "river discharge" (it was wrong in the previous step). So, at the end of this process, only "river discharge" will have the correct time stamp information because if the component interacts with more than one component that last interaction (in this case from RTM to OCN) will overwrites the time stamps. The OCN component CheckImport will fail due to the wrong time stamp information assigned to the "net heat flux" field.

I know that it is hard to explain but this situation only appears in the concurrent type execution and not in sequential one. It could be related with NUOPC_FieldBundleUpdateTime call

   call NUOPC_FieldBundleUpdateTime(genIS%wrap%srcFields, &
                                    genIS%wrap%dstFields, rc=rc)
   if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU,    &
       line=__LINE__, file=FILENAME)) return

because it updates the destination field's time stamp information only for the PETs that exports the data and rest of the PETs will be zeros.

It seems that there is a "race condition" for the "time stamp" attribute of the exchange field. The possible solution is to add an extra code to fix the overwritten time stamp information after " ExecuteRouteHandle"
execution but i am not sure. I might have a design problem and maybe there is no need to store all fields in the import state but i am not sure. Anyway, if you have any suggestion just let me know.

PS: ESMF version is esmf-6.3.0rp1.

There is no any control for the allocation of importField and exportField

In util/mod_config.F90, there is no any control for the allocation of importField and exportField. This might cause error when the code is compiled with debug options like following

-O0 -g -check bounds -traceback -mt_mpi

add support to ATM component for wetting and drying scheme

Options:
1 - interpolate time dependent masking data like other exchange fields and handle the land points in ATM component
2 - put large value to the dried cells before interpolation and use these large values as a masking to handle dried/wet cells

Some other solution can be implemented in ESMF level (sparse matrix multiplication routine) ???

error when one step interpolation is selected

The model gives following error when one step interpolation (no extrapolation support) is selected in exfield.tbl.

forrtl: severe (193): Run-Time Check Failure. The variable 'mod_esmf_cpl_mp_cpl_computerh_$RH2EXIST' is being used without being defined
Image PC Routine Line Source
libirc.so 00002B0C7A175A1E Unknown Unknown Unknown
libirc.so 00002B0C7A1744B6 Unknown Unknown Unknown
regesm.x 000000000138B9F2 Unknown Unknown Unknown
regesm.x 0000000001325BCB Unknown Unknown Unknown
regesm.x 000000000049F817 mod_esmf_cpl_mp_c 549 mod_esmf_cpl.F90

error in exchange field definition

Observations: If two different component will exchange data with same name (mask variable is defined in both ocean and atmosphere model), then the model gives error related with internal field dictionary.
Error:
20170301 231238.889 ERROR PET00 OCN-TO-COP:src/addon/NUOPC/src/NUOPC_Connector.F90:745 Invalid argument - Ambiguous connection status, multiple connections with identical bondLevel found for: land_sea_mask
20170301 231238.889 ERROR PET00 regesm:src/addon/NUOPC/src/NUOPC_Driver.F90:1601 Invalid argument - Phase 'IPDv05p2b' Initialize for connectorComp 2 -> 3: OCN-TO-COP did not return ESMF_SUCCESS
20170301 231238.889 ERROR PET00 regesm:src/addon/NUOPC/src/NUOPC_Driver.F90:837 Invalid argument - Passing error in return code
20170301 231238.889 ERROR PET00 regesm:src/addon/NUOPC/src/NUOPC_Driver.F90:309 Invalid argument - Passing error in return code

I want to get the wave model in RegESM

Dear Sir:
Sorry for send this mail in github because the email I sent was drawed back.
Nowadays I study the RegESM which is a great tool in regional simulation. Some questions I should consult with you:
(1) Coupld you please send me the WAM model with coupling support?
(2) Has the non-hydrostatic stable problem solved in the recent release of RegCM model? Have you integrated the WRF model into RegESM?
Thanks in advance! These helps will help me to develop RegESM in the future, now I try to integrate unstructured-mesh model like SCHISM into RegESM.
Best Regards,
Li Jian
From China University of Geoscience (Wuhan)

implement the vector field (wind or wind stress) rotation.

Two step procedure:
1) Rotate the ATM component (RegCM) wind on a rectangular north-south east-west oriented grid.
2) After this first rotation we need to rotate again the wind on the OCN component (ROMS or MITgcm) grid.

ESMF-MITGCM

Hi, I downloaded RegESM-1.0.0-beta.2 but it seems I cannot find any mitgcm interface. Where is it supposed to be in the main folder? Is it already stable? I know it's a beta code but I don't know if there are some parts which are rather stable actually.

Thanks in advance.