Comments (21)
@uturuncoglu Were you ever able to get this working? I've had another request for this feature and thought I would ask before starting over.
from cdeps.
I don't think that it would be too hard, I made a start in this branch.
remote: Create a pull request for 'unstructured_data_support' on GitHub by visiting:
remote: https://github.com/jedwards4b/CDEPS/pull/new/unstructured_data_support
remote:
To github.com:jedwards4b/CDEPS.git
- [new branch] unstructured_data_support -> unstructured_data_support
Can you give it a try with your data?
from cdeps.
from cdeps.
@jedwards4b Okay. Thanks. Let me try.
@klindsay28 Adding singleton dimension also a good idea. Do you need to do anything special in the ESMF mesh file?
from cdeps.
I'm not very knowledgeable about ESMF mesh files, so I don't know what counts as special. That said, the mesh file that datm is being pointed to in the case above is at
/glade/p/cesmdata/cseg/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
and its netCDF header is
netcdf ne30pg3_ESMFmesh_cdf5_c20211018 {
dimensions:
nodeCount = 48602 ;
elementCount = 48600 ;
maxNodePElement = 4 ;
coordDim = 2 ;
variables:
double nodeCoords(nodeCount, coordDim) ;
nodeCoords:units = "degrees" ;
int elementConn(elementCount, maxNodePElement) ;
elementConn:long_name = "Node indices that define the element connectivity" ;
elementConn:_FillValue = -1 ;
int numElementConn(elementCount) ;
numElementConn:long_name = "Number of nodes per element" ;
double centerCoords(elementCount, coordDim) ;
centerCoords:units = "degrees" ;
double elementArea(elementCount) ;
elementArea:units = "radians^2" ;
elementArea:long_name = "area weights" ;
int elementMask(elementCount) ;
// global attributes:
:gridType = "unstructured mesh" ;
:version = "0.9" ;
:inputFile = "/glade/work/aherring/grids/SCRIP_files/ne30pg3_scrip_170611.nc" ;
:timeGenerated = "Mon Jan 18 17:52:10 2021" ;
}
from cdeps.
I am getting following error after pull your changes,
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90(1985): error #6311: A block was not terminated before an ELSEIF statement was reached.
else if (ndims == 3) then
----^
/work/noaa/nems/tufuk/COASTAL/ufs-coastal/CDEPS-interface/CDEPS/streams/dshr_strdata_mod.F90(2005): error #6321: An unterminated block exists.
else if (ndims == 4) then
I think there is a missing end if in the previous if statement. The if (sdat%mainproc) then
needs to be closed before pio_initdecomp
. It is also same for ndims == 4
case but I am not sure about the logic in there. When I run the case, I got following error,
112: forrtl: error (76): Abort trap signal
112: Image PC Routine Line Source
112: fv3_coastal.exe 0000000001EEAF5B Unknown Unknown Unknown
112: libpthread-2.17.s 00002B1D309D95D0 Unknown Unknown Unknown
112: libc-2.17.so 00002B1D30C1C2C7 gsignal Unknown Unknown
112: libc-2.17.so 00002B1D30C1D9B8 abort Unknown Unknown
112: libc-2.17.so 00002B1D30C5EE17 Unknown Unknown Unknown
112: libc-2.17.so 00002B1D30C68E16 Unknown Unknown Unknown
112: libc-2.17.so 00002B1D30C6B32C __libc_malloc Unknown Unknown
112: libc-2.17.so 00002B1D30C6D89C posix_memalign Unknown Unknown
112: libucs.so.0.0.0 00002B1E728E7098 ucs_posix_memalig Unknown Unknown
112: libucs.so.0.0.0 00002B1E728E93AD ucs_rcache_get Unknown Unknown
112: libuct_ib.so.0.0. 00002B1E72E655C4 Unknown Unknown Unknown
112: libucp.so.0.0.0 00002B1E723DA490 ucp_mem_rereg_mds Unknown Unknown
112: libucp.so.0.0.0 00002B1E723DDC84 ucp_request_memor Unknown Unknown
112: libucp.so.0.0.0 00002B1E72404677 Unknown Unknown Unknown
112: libucp.so.0.0.0 00002B1E7240490B ucp_rndv_progress Unknown Unknown
112: libucp.so.0.0.0 00002B1E72409029 ucp_rndv_receive Unknown Unknown
112: libucp.so.0.0.0 00002B1E7241B518 ucp_tag_recv_nbx Unknown Unknown
112: libucp.so.0.0.0 00002B1E7241B7A3 ucp_tag_recv_nb Unknown Unknown
112: libmlx-fi.so 00002B1E7216ADBB Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F599320 Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F2D830D Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F2D91D6 Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F3272C5 Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F3067EC Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F4495B0 Unknown Unknown Unknown
112: libmpi.so.12.0.0 00002B1D2F2D9A05 PMPI_Alltoallw Unknown Unknown
112: libpioc.so 00002B1D2D19AD5B pio_swapm Unknown Unknown
112: libpioc.so 00002B1D2D1A0E04 compute_counts Unknown Unknown
112: libpioc.so 00002B1D2D19EAE4 box_rearrange_cre Unknown Unknown
112: libpioc.so 00002B1D2D198C69 PIOc_InitDecomp Unknown Unknown
112: libpiof.so 00002B1D2CF28016 piolib_mod_mp_pio Unknown Unknown
112: libpiof.so 00002B1D2CF25F3E piolib_mod_mp_pio Unknown Unknown
112: libpiof.so 00002B1D2CF282E7 piolib_mod_mp_pio Unknown Unknown
112: fv3_coastal.exe 0000000001A1EDDC dshr_strdata_mod_ 1975 dshr_strdata_mod.F90
112: fv3_coastal.exe 0000000001A291B6 dshr_strdata_mod_ 1509 dshr_strdata_mod.F90
112: fv3_coastal.exe 0000000001A25823 dshr_strdata_mod_ 1368 dshr_strdata_mod.F90
112: fv3_coastal.exe 0000000001A202B6 dshr_strdata_mod_ 934 dshr_strdata_mod.F90
112: fv3_coastal.exe 00000000019CEC2A cdeps_datm_comp_m 720 atm_comp_nuopc.F90
112: fv3_coastal.exe 00000000019D1D59 cdeps_datm_comp_m 506 atm_comp_nuopc.F90
Let me also try to add extra dimension to the file and see what happens.
from cdeps.
@uturuncoglu I fixed that branch for you to try again.
from cdeps.
@jedwards4b still getting same error. Let me try with fresh copy since the last one had conflicts and I fixed them but maybe after getting last updates, it did not work as expected.
from cdeps.
@jedwards4b Okay. Tried again with fresh copy and getting same error.
from cdeps.
@klindsay28 Okay. I restructure the netcdf file like following. In this case, I also added node_y
a dimension with size of 1.
netcdf Florence_HWRF_HRRR_HSOFS_shifted_v3 {
dimensions:
time = UNLIMITED ; // (522 currently)
node_x = 1813443 ;
node_y = 1 ;
element = 3564104 ;
noel = 3 ;
variables:
float P(time, node_x, node_y) ;
P:field = "V, scalar, series" ;
P:long_name = "Pressure Reduced to MSL" ;
P:missing_value = 9.96921e+36f ;
P:units = "Pa" ;
double latitude(node_x, node_y) ;
latitude:field = "lat, scalar, series" ;
latitude:long_name = "latitude" ;
latitude:units = "degree_north" ;
double longitude(node_x, node_y) ;
longitude:field = "lon, scalar, series" ;
longitude:long_name = "longitude" ;
longitude:units = "degree_east" ;
double time(time) ;
time:standard_name = "time" ;
time:long_name = "julian day (UT)" ;
time:units = "days since 1990-01-01 00:00:00" ;
time:calendar = "gregorian" ;
time:axis = "T" ;
int tri(element, noel) ;
float uwnd(time, node_x, node_y) ;
uwnd:field = "U, scalar, series" ;
uwnd:long_name = "eastward_wind" ;
uwnd:missing_value = 9.96921e+36f ;
uwnd:units = "m s-1" ;
float vwnd(time, node_x, node_y) ;
vwnd:field = "V, scalar, series" ;
vwnd:long_name = "northward_wind" ;
vwnd:missing_value = 9.96921e+36f ;
vwnd:units = "m s-1" ;
but I am getting following error trace (this is the original CDEPS without @jedwards4b mods),
40: fv3_coastal.exe 0000000001A2C9C2 dshr_strdata_mod_ 1993 dshr_strdata_mod.F90
40: fv3_coastal.exe 0000000001A26F96 dshr_strdata_mod_ 1509 dshr_strdata_mod.F90
40: fv3_coastal.exe 0000000001A23603 dshr_strdata_mod_ 1368 dshr_strdata_mod.F90
40: fv3_coastal.exe 0000000001A1E096 dshr_strdata_mod_ 934 dshr_strdata_mod.F90
40: fv3_coastal.exe 00000000019CEBD5 cdeps_datm_comp_m 709 atm_comp_nuopc.F90
40: fv3_coastal.exe 00000000019D1C79 cdeps_datm_comp_m 497 atm_comp_nuopc.F90
and still getting log like,
(shr_strdata_set_stream_iodesc) setting iodesc for : uwnd with dimlens(1), dimlens(2) = 1 1813443 variable as time dimension time
So, maybe this modification is not enough. @klindsay28 I could not find the run directory in /glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD
. If you could, can you point me the data file. Then, I could try to mimic the same in my file and test again.
from cdeps.
The run directory is on cheyenne/casper at
/glade/scratch/oleson/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD/run
.
There are multiple data files in the directory
/glade/scratch/hannay/archive/f.cam6_3_112.FLT1850.ne30.landspinup.001/cpl/hist
from cdeps.
@klindsay28 I could not access those directories (probably permission issue). Since, I know the case, maybe I could try to find the original data under CESMDATA.
from cdeps.
@klindsay28 Okay. I could aces to the data: /glade/scratch/hannay/archive/f.cam6_3_112.FLT1850.ne30.landspinup.001/cpl/hist/f.cam6_3_112.FLT1850.ne30.landspinup.001.cpl.hx.atm.1h.inst.0027-05-28-03600.nc In this case, lat, lon variables have also time dimension. Other than that it seems very similar to my file. Maybe that is the trick.
from cdeps.
@uturuncoglu , I made a world-readable copy of CASEROOT
at
/glade/scratch/klindsay/for_turuncu/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_fullout_1850AD
. As is usual, the CaseDocs
subdirectory has component namelists and CDEPS stream xml files. There is a single source file in SourceMods/src.datm
, but the diff (see below) seems to be related to the shortwave computation time variable, not how data is processed.
diff /glade/work/oleson/cesm2_3_alpha14a/components/cdeps/datm/atm_comp_nuopc.F90 SourceMods/src.datm/
875,876c875,880
< if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
< nextsw_cday = julday + 2*dtime/shr_const_cday
---
> !KO if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
> !KO nextsw_cday = julday + 2*dtime/shr_const_cday
> !KO
> if (mod(tod,delta_radsw) == 0 .and. stepno > 0) then
> nextsw_cday = julday + 1*dtime/shr_const_cday
> !KO
from cdeps.
@klindsay28 Thanks for all your help. I am trying to restructure my file at this point and then I'll try again. Keep you updated.
from cdeps.
@uturuncoglu, FYI, the experiment that I'm pointing you to is using cdeps1.0.14
. I don't follow CDEPS development closely, so I don't know if updates to CDEPS between that tag and the most recent would interact with CDEPS' ability to process 1D-like data.
from cdeps.
@uturuncoglu I haven't been able to reproduce your error in my branch - can you supply a reproducer please?
from cdeps.
@jedwards4b I am using CDEPS under UFS with additional data mode and Orion. Let me try to create reproducer for you, it could take little bit time.
from cdeps.
@uturuncoglu has this issue been resolved?
from cdeps.
@jedwards4b I could not continue to that work but I'll do. Let's keep this open if you don't mind. I have very similar mesh file to the case that work on CESM side but I my case it is not working. I have some doubt that our data file has some issue since it has more elements than the nodes which seems weird to me. I am discussing with UFS coastal developer to fix the data file then I'll create mesh again and test it. It will probably take time.
from cdeps.
@jedwards4b first off all sorry for late response. last couple of weeks were crazy but I hope I'll be fine after AMS. Anyway, I could cot continue to testing this. It was failing in my case. If you have time and want to implement something that is fine. I could test it with one of the case from UFS Coastal to see what happens.
from cdeps.
Related Issues (20)
- Add better error check for cases when data files don't contain the data fields asked for
- user_nl_datm comment gives incorrect advice
- ssp370 default aerosol file starts in 2014 HOT 2
- ERA5 DATM forcing option is missing from config_component.xml, so hence does NOT work HOT 1
- Add PLUMBER2 option for CLM_USRDAT_NAME for datm similar to handling of NEON
- overzealous error checking
- Running a case with files with zero size doesn't give a graceful error HOT 5
- SSTICE_DATA_FILENAME for "2000" cases changed after switching to CDEPS HOT 5
- SSP cases get rcp45 anomalies HOT 1
- CTSM ISSP460 test gives ERROR: No stream_entry presaero.SSP4-6.0 found
- Add _clean methods to shr_strdata_mod to cleanup memory usage
- Increase stream filename lengths from shr_kind_cl to shr_kind_cx
- Use relative paths for NEON datasets to make them more readable HOT 1
- Update aux_cdeps testlist from Cheyenne to Derecho HOT 8
- testcases should not pull in 50 years of data
- Bad format in error reporting for dtlimit in dshr_strdata_mod HOT 2
- New DOCN mode to ingest multi level ocean fields to send to CISM HOT 11
- cdeps1.0.32 causes all of the cdeps aux tests to fail HOT 2
- Allow streams files without a time axis HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cdeps.