CDEPS does not support data files on unstructured grid about cdeps

[jedwards4b]

@uturuncoglu Were you ever able to get this working? I've had another request for this feature and thought I would ask before starting over.

[jedwards4b]

I don't think that it would be too hard, I made a start in this branch.
remote: Create a pull request for 'unstructured_data_support' on GitHub by visiting:
remote: https://github.com/jedwards4b/CDEPS/pull/new/unstructured_data_support
remote:
To github.com:jedwards4b/CDEPS.git

• [new branch] unstructured_data_support -> unstructured_data_support

Can you give it a try with your data?

[klindsay28]

CASEROOT=/glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD has datm reading fields with dimensions (time, atmImp_ny, atmImp_nx) atmImp_nx = 48600 atmImp_ny = 1 Perhaps it will work for you if you add a singleton dimension to your data.

…

On Thu, Oct 19, 2023 at 1:06 PM Jim Edwards ***@***.***> wrote: I don't think that it would be too hard, I made a start in this branch. remote: Create a pull request for 'unstructured_data_support' on GitHub by visiting: remote: https://github.com/jedwards4b/CDEPS/pull/new/unstructured_data_support remote: To github.com:jedwards4b/CDEPS.git - [new branch] unstructured_data_support -> unstructured_data_support Can you give it a try with your data? — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADWZPO3Z3GY5GBXJLNVK4PLYAF24HAVCNFSM6AAAAAA6HUSG3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRGU2TMOBTGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[uturuncoglu]

@jedwards4b Okay. Thanks. Let me try.
@klindsay28 Adding singleton dimension also a good idea. Do you need to do anything special in the ESMF mesh file?

[klindsay28]

I'm not very knowledgeable about ESMF mesh files, so I don't know what counts as special. That said, the mesh file that datm is being pointed to in the case above is at
/glade/p/cesmdata/cseg/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
and its netCDF header is

netcdf ne30pg3_ESMFmesh_cdf5_c20211018 {
dimensions:
	nodeCount = 48602 ;
	elementCount = 48600 ;
	maxNodePElement = 4 ;
	coordDim = 2 ;
variables:
	double nodeCoords(nodeCount, coordDim) ;
		nodeCoords:units = "degrees" ;
	int elementConn(elementCount, maxNodePElement) ;
		elementConn:long_name = "Node indices that define the element connectivity" ;
		elementConn:_FillValue = -1 ;
	int numElementConn(elementCount) ;
		numElementConn:long_name = "Number of nodes per element" ;
	double centerCoords(elementCount, coordDim) ;
		centerCoords:units = "degrees" ;
	double elementArea(elementCount) ;
		elementArea:units = "radians^2" ;
		elementArea:long_name = "area weights" ;
	int elementMask(elementCount) ;

// global attributes:
		:gridType = "unstructured mesh" ;
		:version = "0.9" ;
		:inputFile = "/glade/work/aherring/grids/SCRIP_files/ne30pg3_scrip_170611.nc" ;
		:timeGenerated = "Mon Jan 18 17:52:10 2021" ;
}

[uturuncoglu]

I am getting following error after pull your changes,

/work/noaa/nems/tufuk/COASTAL/ufs-coastal/CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90(1985): error #6311: A block was not terminated before an ELSEIF statement was reached.
    else if (ndims == 3) then
----^
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90(2005): error #6321: An unterminated block exists.
    else if (ndims == 4) then

I think there is a missing end if in the previous if statement. The if (sdat%mainproc) then needs to be closed before pio_initdecomp. It is also same for ndims == 4 case but I am not sure about the logic in there. When I run the case, I got following error,

112: forrtl: error (76): Abort trap signal
112: Image              PC                Routine            Line        Source             
112: fv3_coastal.exe    0000000001EEAF5B  Unknown               Unknown  Unknown
112: libpthread-2.17.s  00002B1D309D95D0  Unknown               Unknown  Unknown
112: libc-2.17.so       00002B1D30C1C2C7  gsignal               Unknown  Unknown
112: libc-2.17.so       00002B1D30C1D9B8  abort                 Unknown  Unknown
112: libc-2.17.so       00002B1D30C5EE17  Unknown               Unknown  Unknown
112: libc-2.17.so       00002B1D30C68E16  Unknown               Unknown  Unknown
112: libc-2.17.so       00002B1D30C6B32C  __libc_malloc         Unknown  Unknown
112: libc-2.17.so       00002B1D30C6D89C  posix_memalign        Unknown  Unknown
112: libucs.so.0.0.0    00002B1E728E7098  ucs_posix_memalig     Unknown  Unknown
112: libucs.so.0.0.0    00002B1E728E93AD  ucs_rcache_get        Unknown  Unknown
112: libuct_ib.so.0.0.  00002B1E72E655C4  Unknown               Unknown  Unknown
112: libucp.so.0.0.0    00002B1E723DA490  ucp_mem_rereg_mds     Unknown  Unknown
112: libucp.so.0.0.0    00002B1E723DDC84  ucp_request_memor     Unknown  Unknown
112: libucp.so.0.0.0    00002B1E72404677  Unknown               Unknown  Unknown
112: libucp.so.0.0.0    00002B1E7240490B  ucp_rndv_progress     Unknown  Unknown
112: libucp.so.0.0.0    00002B1E72409029  ucp_rndv_receive      Unknown  Unknown
112: libucp.so.0.0.0    00002B1E7241B518  ucp_tag_recv_nbx      Unknown  Unknown
112: libucp.so.0.0.0    00002B1E7241B7A3  ucp_tag_recv_nb       Unknown  Unknown
112: libmlx-fi.so       00002B1E7216ADBB  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F599320  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F2D830D  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F2D91D6  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F3272C5  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F3067EC  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F4495B0  Unknown               Unknown  Unknown
112: libmpi.so.12.0.0   00002B1D2F2D9A05  PMPI_Alltoallw        Unknown  Unknown
112: libpioc.so         00002B1D2D19AD5B  pio_swapm             Unknown  Unknown
112: libpioc.so         00002B1D2D1A0E04  compute_counts        Unknown  Unknown
112: libpioc.so         00002B1D2D19EAE4  box_rearrange_cre     Unknown  Unknown
112: libpioc.so         00002B1D2D198C69  PIOc_InitDecomp       Unknown  Unknown
112: libpiof.so         00002B1D2CF28016  piolib_mod_mp_pio     Unknown  Unknown
112: libpiof.so         00002B1D2CF25F3E  piolib_mod_mp_pio     Unknown  Unknown
112: libpiof.so         00002B1D2CF282E7  piolib_mod_mp_pio     Unknown  Unknown
112: fv3_coastal.exe    0000000001A1EDDC  dshr_strdata_mod_        1975  dshr_strdata_mod.F90
112: fv3_coastal.exe    0000000001A291B6  dshr_strdata_mod_        1509  dshr_strdata_mod.F90
112: fv3_coastal.exe    0000000001A25823  dshr_strdata_mod_        1368  dshr_strdata_mod.F90
112: fv3_coastal.exe    0000000001A202B6  dshr_strdata_mod_         934  dshr_strdata_mod.F90
112: fv3_coastal.exe    00000000019CEC2A  cdeps_datm_comp_m         720  atm_comp_nuopc.F90
112: fv3_coastal.exe    00000000019D1D59  cdeps_datm_comp_m         506  atm_comp_nuopc.F90

Let me also try to add extra dimension to the file and see what happens.

[jedwards4b]

@uturuncoglu I fixed that branch for you to try again.

[uturuncoglu]

@jedwards4b still getting same error. Let me try with fresh copy since the last one had conflicts and I fixed them but maybe after getting last updates, it did not work as expected.

[uturuncoglu]

@jedwards4b Okay. Tried again with fresh copy and getting same error.

[uturuncoglu]

@klindsay28 Okay. I restructure the netcdf file like following. In this case, I also added node_y a dimension with size of 1.

netcdf Florence_HWRF_HRRR_HSOFS_shifted_v3 {
dimensions:
	time = UNLIMITED ; // (522 currently)
	node_x = 1813443 ;
	node_y = 1 ;
	element = 3564104 ;
	noel = 3 ;
variables:
	float P(time, node_x, node_y) ;
		P:field = "V, scalar, series" ;
		P:long_name = "Pressure Reduced to MSL" ;
		P:missing_value = 9.96921e+36f ;
		P:units = "Pa" ;
	double latitude(node_x, node_y) ;
		latitude:field = "lat, scalar, series" ;
		latitude:long_name = "latitude" ;
		latitude:units = "degree_north" ;
	double longitude(node_x, node_y) ;
		longitude:field = "lon, scalar, series" ;
		longitude:long_name = "longitude" ;
		longitude:units = "degree_east" ;
	double time(time) ;
		time:standard_name = "time" ;
		time:long_name = "julian day (UT)" ;
		time:units = "days since 1990-01-01 00:00:00" ;
		time:calendar = "gregorian" ;
		time:axis = "T" ;
	int tri(element, noel) ;
	float uwnd(time, node_x, node_y) ;
		uwnd:field = "U, scalar, series" ;
		uwnd:long_name = "eastward_wind" ;
		uwnd:missing_value = 9.96921e+36f ;
		uwnd:units = "m s-1" ;
	float vwnd(time, node_x, node_y) ;
		vwnd:field = "V, scalar, series" ;
		vwnd:long_name = "northward_wind" ;
		vwnd:missing_value = 9.96921e+36f ;
		vwnd:units = "m s-1" ;

but I am getting following error trace (this is the original CDEPS without @jedwards4b mods),

 40: fv3_coastal.exe    0000000001A2C9C2  dshr_strdata_mod_        1993  dshr_strdata_mod.F90
 40: fv3_coastal.exe    0000000001A26F96  dshr_strdata_mod_        1509  dshr_strdata_mod.F90
 40: fv3_coastal.exe    0000000001A23603  dshr_strdata_mod_        1368  dshr_strdata_mod.F90
 40: fv3_coastal.exe    0000000001A1E096  dshr_strdata_mod_         934  dshr_strdata_mod.F90
 40: fv3_coastal.exe    00000000019CEBD5  cdeps_datm_comp_m         709  atm_comp_nuopc.F90
 40: fv3_coastal.exe    00000000019D1C79  cdeps_datm_comp_m         497  atm_comp_nuopc.F90

and still getting log like,

(shr_strdata_set_stream_iodesc) setting iodesc for : uwnd with dimlens(1), dimlens(2) =        1   1813443   variable as time dimension time

So, maybe this modification is not enough. @klindsay28 I could not find the run directory in /glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD. If you could, can you point me the data file. Then, I could try to mimic the same in my file and test again.

[klindsay28]

The run directory is on cheyenne/casper at
/glade/scratch/oleson/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD/run.
There are multiple data files in the directory
/glade/scratch/hannay/archive/f.cam6_3_112.FLT1850.ne30.landspinup.001/cpl/hist

[uturuncoglu]

@klindsay28 I could not access those directories (probably permission issue). Since, I know the case, maybe I could try to find the original data under CESMDATA.

[uturuncoglu]

@klindsay28 Okay. I could aces to the data: /glade/scratch/hannay/archive/f.cam6_3_112.FLT1850.ne30.landspinup.001/cpl/hist/f.cam6_3_112.FLT1850.ne30.landspinup.001.cpl.hx.atm.1h.inst.0027-05-28-03600.nc In this case, lat, lon variables have also time dimension. Other than that it seems very similar to my file. Maybe that is the trick.

[klindsay28]

@uturuncoglu , I made a world-readable copy of CASEROOT at
/glade/scratch/klindsay/for_turuncu/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD. As is usual, the CaseDocs subdirectory has component namelists and CDEPS stream xml files. There is a single source file in SourceMods/src.datm, but the diff (see below) seems to be related to the shortwave computation time variable, not how data is processed.

diff /glade/work/oleson/cesm2_3_alpha14a/components/cdeps/datm/atm_comp_nuopc.F90 SourceMods/src.datm/
875,876c875,880
<        if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
<           nextsw_cday = julday + 2*dtime/shr_const_cday
---
> !KO       if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
> !KO          nextsw_cday = julday + 2*dtime/shr_const_cday
> !KO
>        if (mod(tod,delta_radsw) == 0 .and. stepno > 0) then
>           nextsw_cday = julday + 1*dtime/shr_const_cday
> !KO

[uturuncoglu]

@klindsay28 Thanks for all your help. I am trying to restructure my file at this point and then I'll try again. Keep you updated.

[klindsay28]

@uturuncoglu, FYI, the experiment that I'm pointing you to is using cdeps1.0.14. I don't follow CDEPS development closely, so I don't know if updates to CDEPS between that tag and the most recent would interact with CDEPS' ability to process 1D-like data.

[jedwards4b]

@uturuncoglu I haven't been able to reproduce your error in my branch - can you supply a reproducer please?

[uturuncoglu]

@jedwards4b I am using CDEPS under UFS with additional data mode and Orion. Let me try to create reproducer for you, it could take little bit time.

[jedwards4b]

@uturuncoglu has this issue been resolved?

[uturuncoglu]

@jedwards4b I could not continue to that work but I'll do. Let's keep this open if you don't mind. I have very similar mesh file to the case that work on CESM side but I my case it is not working. I have some doubt that our data file has some issue since it has more elements than the nodes which seems weird to me. I am discussing with UFS coastal developer to fix the data file then I'll create mesh again and test it. It will probably take time.

[uturuncoglu]

@jedwards4b first off all sorry for late response. last couple of weeks were crazy but I hope I'll be fine after AMS. Anyway, I could cot continue to testing this. It was failing in my case. If you have time and want to implement something that is fine. I could test it with one of the case from UFS Coastal to see what happens.