Add the MERRA2 model support about raider HOT 17 CLOSED

jlmaurer commented on July 23, 2024

Add the MERRA2 model support

from raider.

Comments (17)

leiyangleon commented on July 23, 2024 1

@dbekaert @jlmaurer problem solved after adding the application NASA GESDISC DATA ARCHIVE to my earth data profile as @dbekaert suggested. I am going to write a file linked from the main page to account for this.

@dbekaert thanks for the notes. Yes, we will slice the array based on the bounding box and utc time, etc.

from raider.

leiyangleon commented on July 23, 2024 1

@leiyangleon can you replace the model to point to inst3_3d_asm_Nv instead of inst6_3d_ana_Nv.
That will make it the same as for GMAO. Note that this is a 3Hourly model output.

I will verify with the atmospheric scientist to see if this is the best one to use.

@dbekaert Yes, I can use that instead. That one looks almost identical to the GMAO model.

from raider.

piyushrpt commented on July 23, 2024

Here is some code from pyaps - might help get started. nc4 might be most relevant.
http://earthdef.caltech.edu/projects/pyaps/repository/entry/merra.py

from raider.

dbekaert commented on July 23, 2024

An example of an actual download is here, although looking to leverage OpenDAP for retrieval of the specific layers and some slice&dice operations, not a complete file download.

MERRA-2 data

there are pressure and model outputs. We should aim to use the model level files.

Preferred inst6_3d_ana_Nv: 3d,6-Hourly,Instantaneous,Model-Level,Analysis,Analyzed Meteorological Fields V5.12.4 (M2I6NVANA 5.12.4)
inst6_3d_ana_Np: 3d,6-Hourly,Instantaneous,Pressure-Level,Analysis,Analyzed Meteorological Fields V5.12.4 (M2I6NPANA 5.12.4)

OpenDAP for slice and dice of data

We made an initial test with OpenDAP but were in contact with GSFC to resolve some issue. The idea is to contruct a URL which has all the subsetting included. Here is relavant documentation:

http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.html
Earthdata and OpenDAP: https://earthdata.nasa.gov/collaborate/open-data-services-and-software/api/opendap
OpenDAP python background: https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+PyDAP

Below i capture the example we worked on for MERRA-2 with Ray:

import pydap.client
import pydap.cas.urs
url = 'https://goldsmr5.gesdisc.eosdis.nasa.gov:443/opendap/MERRA2/M2I6NPANA.5.12.4/1980/01/MERRA2_100.inst6_3d_ana_Np.19800101.nc4'
session = pydap.cas.urs.setup_session('username', 'password', check_url=url)
ds = pydap.client.open_url(url, session=session)
ds['T'][:]

The problem experienced when getting the temperature:

        <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
        <html><head>
        <title>302 Found</title>
        </head><body>
        <h1>Found</h1>
        <p>The document has moved <a href="https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fgoldsmr5.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cHM6Ly9nb2xkc21yNS5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL01FUlJBMi9NMkk2TlBBTkEuNS4xMi40LzE5ODAvMDEvTUVSUkEyXzEwMC5pbnN0Nl8zZF9hbmFfTnAuMTk4MDAxMDEubmM0LmRvZHM%2FVFswOjE6M11bMDoxOjQxXVswOjE6MzYwXVswOjE6NTc1XQ">here</a>.</p>
        </body></html>

Feedback from the GSFC colleagues was the following link:
https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+PyDAP

from raider.

dbekaert commented on July 23, 2024

Also there is a lot of similarity between MERRA-2 and the GMAO models, both leverage OpenDAP and also have same filename, format and layers. So we should be able to re-use it.

That say we are working closely with GMAO team, so we could try to get that to work with openDAP first, as now we have their support in case of questions.

from raider.

leiyangleon commented on July 23, 2024

@dbekaert could you ask the MERRA-2 team for the updated url? Jeremy and I both encountered the 302 found error, saying the "the document has moved", when using the OpenDAP to retrieve the data.

from raider.

dbekaert commented on July 23, 2024

@leiyangleon might just be a typo in the documentation. See the parent folder of the URL: https://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2I6NPANA.5.12.4/1980/01/

Also, the URL changes depending on the year its being accessed; so one would need to construct it based on the user request. See the above link that @piyushrpt provided or look within the code of TRAIN to get an idea on URLS.

from raider.

leiyangleon commented on July 23, 2024

@dbekaert the above URL link is the same as the one @jlmaurer and I tried. It won't work. I also looked at the URLs that you and Piyush mentioned above (in Pyaps) and it is indeed as you mentioned year-month-date-hr dependent. So I need to get the right format of these. Well I am guessing for example, year=1980, month=01, date=19800101 and hr=00. However, it would be really helpful to have a documentation or updated URL database. Could you direct me to either of these resources? Otherwise I have no idea/experience what I am manipulating and expecting.

from raider.

jlmaurer commented on July 23, 2024

Hey @leiyangleon I just went to the URL and copied this link an example file:
https://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2I6NPANA.5.12.4/1980/01/MERRA2_100.inst6_3d_ana_Np.19800101.nc4.html. Perhaps let's try to experiment with pydap to get this to work. maybe a permission issue with opendap or something.

from raider.

dbekaert commented on July 23, 2024

We reversed engineered the link to know what it its.
See here for TRAIN and merra-2: https://github.com/dbekaert/TRAIN/blob/6c93feb95ae95eaf4c8468e89ec0b8325eac946f/matlab/aps_merra_files.m#L149

The baseurl can change depending on the year. See the Pyaps or TRAIN code for it.
Then YYYY/MM/MERRA2_100.inst6_3d_ana_Np.YYYYMMDD.nc4

from raider.

dbekaert commented on July 23, 2024

@leiyangleon looked a bit in more detail on the internet.
The MERRA-2 data is also on GES DISC so its behind Earthdata log-in.

This means that there needs to be a .netrc file with Earthdata credentials.
In addition the users needs to add the following application to their earthdata profile: GES DISC

I tried it now:

In [7]: import pydap.client 
   ...: import pydap.cas.urs 
   ...: url = 'https://goldsmr5.gesdisc.eosdis.nasa.gov:443/opendap/MERRA2/M2I6NPANA.5.12.4/1980/01/MERRA2_100.inst6_3d_ana_Np.19800101.nc4' 
   ...: session = pydap.cas.urs.setup_session('username', 'password', check_url=url) 
   ...: ds = pydap.client.open_url(url, session=session) 

In [6]: ds['T'][:]                                                                                                                                                            

Out[6]: 
<BaseType with data array([[[[9.9999999e+14, 9.9999999e+14, 9.9999999e+14, ...,
          9.9999999e+14, 9.9999999e+14, 9.9999999e+14],
         [9.9999999e+14, 9.9999999e+14, 9.9999999e+14, ...,
          9.9999999e+14, 9.9999999e+14, 9.9999999e+14],
         [9.9999999e+14, 9.9999999e+14, 9.9999999e+14, ...,
          9.9999999e+14, 9.9999999e+14, 9.9999999e+14],
         ...,

Can you add documentation to the RAIDER README.md as well?
Perhaps have a new file linked from the main page with in difffernt sections the details for each model what users would need to do in addition to just running the code.
e.g. ECMWF, how do users get an account and how to they set up there .cdsapi.
e.g. MERRA2, how to get Earthdata account, the need for making a .netrc file, the need for adding in the application to their profile.
i.e.

from raider.

dbekaert commented on July 23, 2024

@jlmaurer: When a user submits a call and asks for a region of interest, do you automatically expand the model bbox with extra nodes prior to calling the download code or does @leiyangleon need to account for that himself?

@leiyangleon the example provided above does not slice and dice yet. Next step would be to slice and dice based on the bbox, the utc time etc using opendap; and then just load the complete array.

from raider.

leiyangleon commented on July 23, 2024

@dbekaert Thanks for spotting the problem. That is probably the reason why I cannot get access to the specific data values. Sounds good. Will update the docs then.

from raider.

leiyangleon commented on July 23, 2024

@dbekaert @jlmaurer I checked the above-mentioned pressure-level (inst6_3d_ana_Np, M2I6NPANA 5.12.4) and model-level (inst6_3d_ana_Nv, M2I6NVANA 5.12.4) data in the MERRA2 docs. Please see attached the two figures extracted from the docs. For the pressure level, surface pressure and Geopotential height are provided, which can be used to calculate the pressure and height; for model level, only pressure thickness is provided for each model layer without height information.

I further checked the docs, which says all of the 72 model layers come with fixed 1-D vector of top pressure of each layer, and the pressure can be calculated with this top pressure plus the layer pressure thickness (DELP, provided in the model level data) all the way from the top layer (index=0) down to the bottom one (index=71). So it seems we can compute the pressure information using this way; however, in the docs, it also says this is good for the current version of data, but may be changed later. So it is recommended to use the 3D pressure data as distributed, which is only available for pressure-level.

Also, even though we can find the pressure for model-level, we still need the 1-D or 3-D height, which is not available for model-level data but does exist in pressure-level. By referring to the docs again, it says all of these model-level data are defined on a terrain-following hybrid sigma-p coordinate or hybrid-sigma, see below
For the MERRA-2 products documented here, all model-level fields are on a hybrid-sigma coordinate, and their vertical location could be obtained from the “ak-bk” relationship as well. But this may change in future GMAO systems. We thus recommend that users rely on the reported 3D pressure distribution, and not use those computed from the “ak” and “bk”.

So I need your input about how we are gonna proceed, e.g.

pressure-level or model-level?
if model-level, how to compute height using ak and bk for the hybrid-sigma coordinate? There is no information in the docs about this calculation. BTW, I can find ak and bk from the metadata.

from raider.

dbekaert commented on July 23, 2024

@leiyangleon can you replace the model to point to inst3_3d_asm_Nv instead of inst6_3d_ana_Nv.
That will make it the same as for GMAO. Note that this is a 3Hourly model output.

I will verify with the atmospheric scientist to see if this is the best one to use.

from raider.

leiyangleon commented on July 23, 2024

I am closing this issue because MERRA-2 mode support has be completed as here.

from raider.

jlmaurer commented on July 23, 2024

Closing as addressed by #112

from raider.

Add the MERRA2 model support about raider HOT 17 CLOSED

Comments (17)

MERRA-2 data

OpenDAP for slice and dice of data

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs