cgentemann / cloud_science

NASA cloud science tutorials and science use cases

Tutorial materials

This repository tutorial materials for tutorials on cloud data. The links below will launch an interactive environment on binder.pangeo.io Note that binder environments are ephemeral. Any changes you make will be lost once your session ends, and you shouldn't store passwords.

To explore Pangeo data on GCP (eg. CCMP), select the button 'Pangeo Binder GCP US-central1', and then, once the binder initializes, select 'tutorials'.

| |

To explore 3 difference cloud-optimized datasets (MUR SST, GOES, ERA5), select button below 'Pangeo Binder AWS US-west1', and then, once the binder initializes, select 'tutorials'.

Tutorial Highlights

---

• About Pangeo: Pangeo is a community effort for big data in the geosciences using Python. A key component of the Pangeo effort is the improved integration of Xarray and Dask to enable analysis of very large datasets.
• About Jupyter: Project Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages. The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.
• About Xarray: Xarray is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures.
• About Dask: Dask is a flexible parallel computing library for analytic computing.
• About Geopandas: Geopandas is a library to facilitate analysis of geospatial vector data
• About Intake: Intake is a cataloging system designed to "Take the pain out of data access and distribution"

Workshops

• 2020 OceanHackWeek: OceanHackWeek2020 tutorials.
• 2018 AGU workshop: Scalable Geoscience Tools in Python — Xarray, Dask, and Jupyter.
• 2019 AGU workshop: Pangeo: Hands on with JupyterHub and Open-source Python Tools for Scalable Analysis of Big Data in the Geosciences

Acknowledgements

At its core, Pangeo is a community effort built around open-source software. As such, the credit for the developments of the software described here belongs with the community that created it.

Elements of this tutorial were taken from the xarray, Dask, Cartopy, Holoviews, and Geoviews documentation. Some pieces of text in the xarray portion of the tutorial were adapted from Hoyer and Hamman (2016).

Pangeo is supported by the National Science Foundation (NSF) via the EarthCube Program and the National Aeronautics and Space Administration via the ACCESS Program. NCAR is separately supported by the National Science Foundation (NSF).

Google provided compute credits on Google Compute Engine. Amazon provided compute credits on AWS

This work is licensed under a Creative Commons Attribution 4.0 International License.

cloud_science's Issues

fix time loop in Collocate_cloud_data_cvs

@marisolgr possible to remove time loop?
Can we update cvs read to use parse_time? https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

add mur

In this repository there is a Access_MUR_SST_example.ipynb that shows how to read AWS MUR global SST.
update tutorial to use MUR SST from AWS.
to do this, you may need to open a dask cluster.
@caitlinkroeger

podaacpy

@lewismc
This notebook has the example code. You can launch binder here: https://github.com/cgentemann/cloud_science/tree/master/GHRSST_tutorial
Notebook: https://github.com/cgentemann/cloud_science/blob/master/GHRSST_tutorial/Intro_09_Xarray_advanced_hurricane_case_study.ipynb

podaacpy is doing a couple of funny things.

1. In cell [4] I use podaacpy to read in a list of files. The time array is for a month but it only returns the first 7 daily files instead of all 30.

2. in cell [6] I use podaacpy to search for ccmp files 8/15/2011 - 9/15/2011. It returns 4 files from 1987 (the beginning of the dataset)

Thank you!

MUR SST Zarr `fill_value` should be `nan`?

Chelle mentioned that

[In] printing out points over land, they should be nan. All files have nan except the rechunked zarr file created after our version 1 zarr. I tried to read zarr two different ways, write it out to see if that was the issue but wasn't able to recreate. somehow in the 2nd zarr file all the land values got set to a flag value.

The flag value is -32768 - listed in the zarr metadata

The data is probably coming down as the official cloud version goes up, but we are just trying to figure out how this happened, if we any problem with xarray/zarr or something easy to do that we need to be careful about when converting files.

Chelle also mentioned that this issue is the same as posted by @rsignell-usgs: pangeo-data/rechunker#59
Chelle tried both cloud s3 & local: It would create the final but not intermediate file. This is code to try on a subset of mur data to check if it is the rechunker step that is filling the fill_value: https://github.com/cgentemann/cloud_science/blob/master/make_zarr/test_mur_rechunker.ipynb

I will try replicating the error and post any updates here.

cc @cgentemann

add more markdown comments to collocate_cloud_cvs

add comments to tutorial to explain what we are trying to do.
update to new .cvs file
@jeffdorman