umr-cnrm / phyex

PHYsique EXternalisée

PHYEX

PHYsique EXternalisée

Documentation can be found in the docs directory.

Several presentations were done, the materials can be found on the wiki.

Prerequisites:

• an internet connexion (with access to the github servers) is needed only for the installation and, for offline tests (testprogs), when the fiat version to use change
• python > 3.8 (but only tested with version 3.10)
• some python packages (available on PyPI):
  • numpy and pandas for the testprogs
  • epygram and matplotlib for AROME
  • xarray for Meso-NH
  • ctypesForFortran to use the python binding
• a C compiler (tested with cc 11.4.0)
• a FORTRAN compiler (tested with ifort and gfortran, but automatic installation only works with gfortran >= 10)
• some classical unix tools: wget, tar, make and git

Quick Start Guide:

• open a terminal on a system satisfying the prerequisites and enter the following commands
• if you don't have a github ssh key or don't know what it is:

  git clone https://github.com/UMR-CNRM/PHYEX.git
  ./PHYEX/tools/INSTALL.sh --ALL --test

• if you have a github ssh key:

  git clone [email protected]:UMR-CNRM/PHYEX.git
  ./PHYEX/tools/INSTALL.sh --ALL --test --ssh

• If all goes well, the last line should be "SUCCESS, files are identical"

phyex's Issues

3D arrays suppression from rain_ice

The rain fraction diagnostic must be computed before calling ice4_tendencies in order to suppress all the 3D arrays in ice4_tendencies.
This modification will induce an approximation that should be acceptable.

do concurrent suppression

Replace the do concurrent in rain_ice by a do with the "$DIR$ IVDEP" directive

Signed zero

The code should give the same results whether the zeros are signed or not. Is it the case?

Results different for rain_ice_old

On a small 3D case with AROME (small_2D_alt9), the results given by rain_ice_old are different between commits 37885e5 and 6e28495.
It could be due to the merging of branch 'rain_ice_old' of https://github.com/rolfhm/PHYEX or due to the branch itself.
Investigation and/or bug correction needed.

Seg fault when adding budget of rs, rh or rg in turb

At the integration of changes from mesonh 5.6->5.7 a new process changes the some mixing ratio in turb.F90, so budgets are added around turb_ver such as :
IF( BUCONF%LBUDGET_RR ) CALL BUDGET_STORE_INIT_PHY(D, TBUDGETS(NBUDGET_RR), 'VTURB', PRRS (:,:, 3) )
IF( BUCONF%LBUDGET_RS ) CALL BUDGET_STORE_INIT_PHY(D, TBUDGETS(NBUDGET_RS), 'VTURB', PRRS (:,:, 5) )
IF( BUCONF%LBUDGET_RG ) CALL BUDGET_STORE_INIT_PHY(D, TBUDGETS(NBUDGET_RG), 'VTURB', PRRS (:,:, 6) )
IF( BUCONF%LBUDGET_RH .AND. KRR==7 ) CALL BUDGET_STORE_INIT_PHY(D, TBUDGETS(NBUDGET_RH), 'VTURB', PRRS (:,:, 7) )
and call budget_store_end_phy.

(Budgets on ri, PRRS(:,:,4) was already present).

At the call of BUDGET_STORE_INIT_PHY, the default test case of arome raise at first timestep the error on call of budget of RS
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
[EC_DRHOOK:sxphynh:4:1:26619:26619] [20231222:115855:2.594] [LinuxTraceBack] [03]: MASTER() [0x136de72] : __mode_budget_phy_MOD_budget_ddh() at mode_budget_phy.F90:173
[EC_DRHOOK:sxphynh:4:1:26619:26619] [20231222:115855:2.594] [LinuxTraceBack] [04]: MASTER() [0x136ee10] : __mode_budget_phy_MOD_budget_store_end() at mode_budget_phy.F90:30 (discriminator 4)
[EC_DRHOOK:sxphynh:4:1:26619:26619] [20231222:115855:2.594] [LinuxTraceBack] [05]: MASTER() [0x136eef6] : mode_budget_phy_MOD_budget_store_end_phy() at mode_budget_phy.F90:39
[EC_DRHOOK:sxphynh:4:1:26619:26619] [20231222:115855:2.594] [LinuxTraceBack] [06]: MASTER() [0x150cac7] : turb() at turb.F90:1131 (discriminator 1)
[EC_DRHOOK:sxphynh:4:1:26619:26619] [20231222:115855:2.594] [LinuxTraceBack] [07]: MASTER() [0x121f49e] : aro_turb_mnh() at aro_turb_mnh.F90:389

In mesonh and testprogs, there is no problem.
If the new budgets of RS is commented, the error triggs on the call of budget or RG.
The call budget on RR is fine.

The problem does not come from the changes in mesonh 5.6 : I tried in the master branch of PHYEX at commit 7c3b2cb (20/12, before merging changes from mesonh) to add the call of BUDGET_STORE_INIT_PHY in turb : the error is the same.
On 7c3b2cb, I tried :

• to add a call of init/end budget on r_s at the end of rain_ice.F90 : it works
• to add the call budget_init on r_s at the first line of turb.F90 : same error

It looks like a problem of the shape of PRRS at entering of turb. I checked apl_arome and aro_turb_mnh without sucess.

Workaround : in commit 32e554f , I controled the call BUDGET_STORE_INIT/END_PHY on r_s and the others impacted new mxing ratios by the namelist key LTURB_PRECIP (False by default in AROME) that` is responsible of the changes (and the budget term).
But using LTURB_PRECIP in AROME would then not work for now

LICENCE with minpack.f90 and LIMA

With the inclusion of LIMA in PHYEX, minpack.f90 is introduced (726e04c). This routine is attached with a LICENCE in MesoNH pack
LICENCE.txt
. Is this LICENCE compatible with PHYEX's and how to distribute minpack within PHYEX folder ? (in micro ?)

modd_turb_cloud suppression

Would it be possible to move the modd_turb_cloud keys into modd_turbn?
This would allow a better flexibility (activation of cloud mixing length modification in several models) and the suppression of the KMI argument of turb.

vertical limitation of updraft in EDKF

Fleur C. :
Durant son stage, Aude avait soulevé un bug dans la limitation de l'updraft dans EDKF qui dès que l'updraft dépasse une certaine hauteur tout l'updraft est coupé alors que l'idée initiale (cf les commentaires) était plutot de faire progressivement diminuer le flux de masse.
Actuellement si le nuage dépasse la hauteur max des nuages bas, alors le flux de masse est nul pour toute la colonne.

Work as done in MesoNH 5.5.1. Must :

• merge to current version of mode_compute_updraft.F90
• add a namelist key to activate the correction or not
• test it on MesoNH
• test it on AROME on production cycle at integration

Array syntax in call statements

Issue found by Ph. Marguinaud in mode_mf_turb (line 198):
CALL TRIDIAG_MASSFLUX(D,PTHLM,PFLXZTHMF,-PEMF,PTSTEP,PIMPL,...
The "-PEMF" part should be replaced by a temporary variable.

The same kind of array syntax in call statements exist in other places. They must be suppressed.

Install simplification

At the beginning, it wasn't possible to use the official versions of AROME and Meso-NH to plug PHYEX because of the too many differences in the source code and in particular the changes in the source directory organisation.

Starting from cy49t1 for arome and v5.7 for Meso-NH, we could:

• use "official" pre-compiled pack for AROME (maybe through the same tools as those used by davaï)
• use official tar.gz for Meso-NH
• put Meso-NH test cases in a separate directory (like the conf_tests one used by AROME), except if all the tests are available in the official tar.gz file
• modify the testing.sh script to allow for automatic running of the reference versions when missing

This evolution needs some modifications in the check_commit_ial.sh and check_commit_mesonh.sh scripts and allows the suppression of the 'pack' subdirectory in the PHYEX repository. Documentation must also be updated.

Moreover, a limited set of testprogs data could be made available on a web page to ease the use of these test programs.

A small install script could be written to install filepp, mnh_expand and the testprogs data.

rain_ice_old fusion

Is it possible to merge the arome and mesonh versions of rain_ice_old?

Implementation of the NPROMICRO mechanism in the LPACK_MICRO=F case

In mode_ice4_pack, we can compute the microphysics on a selection of points by packing them in new arrays before calling ice4_stepping (LPACK_MICRO=T) or call directly ice4_stepping on all the points (LPACK_MICRO=F).

With the packing, we can activate the NPROMICRO mechanism; but this mechanism is not available without the packing.

This issue describes how to implement NPROMICRO without packing:

• add variables for lower and upper bounds of the DO loops used in the different processes
• these variables will evaluate to 1 and KSIZE in the packing case (to reproduce the present computation)
• without the packing, these variables will target a moving slice, with a DO loop around the call to ice4_stepping

Code cleaning

Several cleanings are needed:

• suppression of unused local variables
• addition of missing "IMPLICIT NONE"
• addition of missing "INTENT" attributes

Tkemin increment

XTKEMIN seems to be incremented at each time step (tested in LMDZ)

ini_snow suppression

It seems that ini_snow code is very similar to in_rain_ice and could be replaced by a control switch in in_rain_ice to limit code duplication.

Use a mean to close the PMMC09 scheme

Merge the modifications done by B. Viala

Testprogs results different with --noexpand

Running the testprogs, the results for turb are different depending on the --noexpand option

HPROGRAM

This key must be replaced by some namelist individual keys

Add more possibilities to call ice4_stepping from ice4_pack

In mode_ice4_pack, we can compute the microphysics on a selection of points by packing them in new arrays before calling ice4_stepping (LPACK_MICRO=T) or call directly ice4_stepping on all the points (LPACK_MICRO=F).

On NEC, it could be more efficient to use indirection.

This issue describes how to implement an indirection mechanism in the LPACK_MICRO=.F. case. But this way of implementing it was not fully validated by Ryad.

• compute an indirection array (INDIRARRAY), which will be given to ice4_stepping
• in ice4_stepping and lower in the call tree, all "DO JL=1, KSIZE" loops will be transformed into
  "DO JLL=1,KSIZEINDIR; JL=INDIR(JLL, INDIRARRAY)"
• the following piece of code will be included in the different files:
  #ifdef INDIR_MICRO
  #define INDIR(JLL, INDIRARRAY) INDIRARRAY(JLL)
  #else
  #define INDIR(JLL, INDIRARRAY) JLL
  #endif

Use statuses instead of comments for continous integration

Presently the testing.sh script uses commit comments to store and display the result of the integration tests.
Statuses are a a better feature to display the result of the integration tests but cannot be used to display result details. Details must be stored elsewhere and a web link can be added to the statuses.

We must found a public place to store these details and modify the testing.sh script to use statuses.

LES_MEAN_SUBGRID cleaning

1. use LES_MEAN_SUBGRID_PHY instead of LES_MEAN_SUBGRID in the horizontal turbulence
2. rename the arome/turb and mesonh/turb routines into les_mean_subgrid.F90
3. replace call (suppress _PHY)
4. remove old routines

small_3D_lima runs x10 longer since MesoNH 5.6->5.7 contribution

I forgot to send a PR for the contrib...
Does it come from the ELEC contrib or already before ?
Check the branch check_mesonh_dev_0.6.3_to_master

branch master : commit 7f5af68 : runtime OK
branch check_mesonh_dev_0.6.3_to_master, commit 5b8e8ce (before ELEC) : runtime OK

PMMC09 and vertical resolution

When vertical resolution increases near the surface, the scheme closure happens too near the surface and can lead to an overestimated transport.
The idea of Valery is to modify the closure mechanism to take into account the environment variables values higher and start the updraft at a given height above ground (maybe around 20m). The difficulty is to impose updraft characteristics under this point to allow the mixing on the whole column and not only starting from the 20m level.

KFB scheme

Is it possible to merge the mesonh and arome version of the shallow part of KFB

Remove CTOM='TOM06' option

The third moments in turb is not used in Méso-NH, AROME-FR, HARMONIE-AROME, AROME and ALARO from Austria, Hungary and Poland (mail from [email protected]).
ALARO-CMC use their own TOM options called under the key LCOEFK_TOMS=.T. (NAMPHY) and fully coded in acdifv3.F90

Cleaning of this option will help a lot for GPU porting

Tools reorganisation

Rewrite/reorganise the present tools to have:

1. a transformation tool (replacing filepp, MNH_Expand_Array, correct_indent.py, verify_mnh_expand.py and the file renaming part of prep_code.sh)
2. the prep_code.sh script (without the renaming part) to deal with commits and call the transformation tool
3. the check_commit_* applicative tools (which call the prep_code script)

The small scripts needed to control the results (comp_DDH.py, compare.py, diffNODE.001_01) may be, or not, in-lined in the different check_commit_* scripts

LPACK_MICRO=FALSE bug in MesoNH

The option LPACK_MICRO=FALSE doesn't work in Meso-NH

ANY() and COUNT() on scalar or array

On branch demo-dwarf, commits :
6f801ec
ef6b29a
c6e9033

transforms by hand COUNT() and ANY() in the case of the argument is a scalar.
(on master branch, CPU, arguments are arrays)

For a better way of handling both cases (array or scalar), one must code a version of ANY() and COUNT() that

• returns the scalar if the argument is a scalar
• apply the operator if the argument is an array

By extending the intrinsic functions (possible ? which version of FORTRAN ?) or re-write new home-made function

Add the missing dynamical production term of TKE from the shallow convection scheme

The EDKF shallow convection scheme already computes a thermal production term of TKE but no dynamical one. This term must be added.

LIMA testprogs

A tesprog (and code to generate data) must be written for the LIMA scheme.

GPU porting of ELEC scheme

ELEP and ELECD structures from modd_elec_* have been introduced into PHYEX (in rain_ice and lima) to apply sedimentation of electric charges with respect to the microphysics scheme.
In mode_elec_tendencies, there is still some USE MODD of variables from LIMA and ELEC routines to be included in a structure.
For ELECP and ELECD, there are a lot of missing variables included in the structure already created...to be continued.

The full electricity scheme could be included further in PHYEX.

For now, it must be excluded from the GPU transformations applied for IAL.

modd_conf cleaning

MODD_CONF might be suppressed from PHYEX:

• rewrite horizontal turbulence to make use of structures instead of module variables
• suppression of modd_conf from the PHYEX repository
• move aro_ini_conf from mpa to mse (AROME)

move deposition in ice4_tendencies

The deposition process must be moved in ice4_tendencies

rain_ice_old testprog crash with profile mesonh-LXgfortran

At line 559 of file /home/riette/TESTPROGS/COMMITe070d16/build/with_fcm/arch_mesonh-LXgfortran/src/rain_ice_old/getdata_rain_ice_old_mod.F90
Fortran runtime error: Index '16' of dimension 2 of array 'p_in' above upper bound of 15

Error termination. Backtrace:
#0 0x7fca6753bad0 in ???
#1 0x7fca6753c649 in ???
#2 0x7fca6753cc46 in ???
#3 0x55af1011a112 in __getdata_rain_ice_old_mod_MOD_npromize3
at /home/riette/TESTPROGS/COMMITe070d16/build/with_fcm/arch_mesonh-LXgfortran/src/rain_ice_old/getdata_rain_ice_old_mod.F90:559
#4 0x55af1012e656 in __getdata_rain_ice_old_mod_MOD_getdata_rain_ice_old
at /home/riette/TESTPROGS/COMMITe070d16/build/with_fcm/arch_mesonh-LXgfortran/src/rain_ice_old/getdata_rain_ice_old_mod.F90:415
#5 0x55af0fe4cef2 in main_rain_ice_old
at /home/riette/TESTPROGS/COMMITe070d16/build/with_fcm/arch_mesonh-LXgfortran/src/rain_ice_old/main_rain_ice_old.F90:357
#6 0x55af0fe5e515 in main
at /home/riette/TESTPROGS/COMMITe070d16/build/with_fcm/arch_mesonh-LXgfortran/src/rain_ice_old/main_rain_ice_old.F90:3

Include and check performance metrics

Include performance metrics (DrHook, total execution time, analyse of the compilation logs for checking vectorisation).
Test on other architectures (NEC?)

Heat conservation

Energy is not conserved in:

• ice_adjust
• rain_ice (microphysical processes and sedimentation)
• th_r_from_thl_rt and thl_rt_from_th_r
• transfer between soil and atmosphere (precipitation, evaporation)

Vertical gradients and shuman

In Meso-NH, arrays are overdimensioned along the vertical axis.
In AROME, we can use "normal" arrays for all parametrisation except the turbulence one.

This issue list the steps to realise in order to suppress the unphysical points for AROME:

1. Update vertical halo before calling the parametrizations in Meso-NH: At least when calling the PMMC09 scheme, the values in the unphysical level under ground is not equal to the values on the first physical level. Could it be possible to update the vertical halo before calling each parametrisation?
2. Try to suppress computation on unphysical points in shuman_mf, keeping the results unchanged
3. If the preceding rewritting is successful, EDKF can be rewritten to suppress computation on unphysical points
4. Again, if the shuman_mf is successful, the turb scheme could use these routines (all the physics could use the same set of routines to compute gradients and shuman)
5. The last step would be suppress the computations on unphysical points in the turbulence scheme, giving the possibility to suppress the overdimensioning in apl_arome

KSPLIT argument of turb

Is there a reason to not include the KSPLIT argument of the turbulence scheme inside the modd_turbn module? If not we must do it.

Add the call to sources_neg_correct in AROME

In AROME and co, in turb.f90 the correction of negativities values are not called (an empty routine is present in src/common).
The true routine is in src/mesonh/aux.

Adding this will change the results, it must be tested correctly.

Meanwhile, for GPU porting with OpenACC within routines, as sources_neg_corecct is directly moved out of PHYEX in the integration of MesoNH (because it is called in other parts of MesoNH), the routine is ported with hard-coded (not with pyft usage) MNH_MEM_GET, MPPDB_CHECKED and so on to ease the porting of sources_neg_correct in the GPU branch of MesoNH (arrays are declared in array-syntax style for ex).

Activation of the snow related modifications in AROME (work of Jean Wurtz)

These modifications have not been introduced in the AROME version as they break the bit-reproducibility and, thus, need a more complex validation.
There are two solutions:

• rewrite the code to enable the bit-reproducibility
• validate the new results

Control to implement: when lred=F, lsnow must be F

Be careful, in arome, to keep consistency with ini_snow (easier if code is rewritten to enable bit-reproducibility)

Towards version 1.0

This issue lists the modifications to introduce before issuing the version 1.0 of PHYEX:

• check_commit_* cleaning to remove parts useful only for the earliest versions of PHYEX source code (code move, rename, missing json). This modification isn't backward compatible.
• use of pyft instead of filepp
• offline driver with python binding
• enable compilation/execution on GPU with transformations
• full automatic installation with INSTALL.sh
• versioning of all the test cases for Méso-NH (and automatic installation of theses cases)
• #35

update init of rain_ice_old testprog

Since the introduction of the rain_ice_old testprog, the initialisation of the PHYEX parametrisations has evolved.
The rain_ice_old testprog should be modified to use the new setup mechanism

Wrong results for LCRIAUTI=F

Results are different for test case small_3D_alt10