ds604 / w10n

NASA's Webification

NASA/JPL's Webification

All of NASA/JPL's science data is available to the public. Webification (w10n) is an open source NASA/JPL data access framework that enables developers to easily access data in web-friendly formats for visualizations like this:

Webification simplifies use of the large and often complex data files published by NASA's Distributed Active Archive Centers, like PO.DAAC. The PO.DAAC w10n Service is open source software, that bridges the gap between Earth science data formats like NetCDF, HDF4 & HDF5 to JSON and XML.

Re:Invent Challenge

From here:

Getting Started

1, Explore the interactive tutorial here

2, Test our our live API here:

# Open to explore in your web browser and note the available attributes:
http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/?output=html

# bash
# - Get Metadata with '/'
curl "http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/lat/?output=json"
# - Get 100 datapoints with '[0:100]'
curl "http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/lat\[0:100\]?output=json"

# python
import requests
# - Get Metadata with '/'
requests.get('http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/lat/?output=json')
# - Get 100 datapoints with '[0:100]'
requests.get('http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/lat[0:100]?output=json')

// Javascript
// - Get Metadata with '/'
http.get("http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/lat/?output=json", function(res) {
  res.on("data", function(chunk) {
    console.log("data: " + chunk);
  });
});
// - Get 100 datapoints with '[0:100]'
http.get("http://54.212.253.164/data/GRACE.CSR.LAND.RL05.DS.G200KM.nc/lat[0:100]?output=json", function(res) {
  res.on("data", function(chunk) {
    console.log("data: " + chunk);
  });
});

3, Spin up a webification machine of your own (AMI ID: ami-3eef740e, named: 'w10n-reinvent' in US-West-2 ) with w10n pre-configured. Notes:

• Suggested instance type: m1.medium
• Security Group should allow inboud traffic from only ports 22 & 80
• Setup your own server simply by downloading & unzipping this API:
  • https://s3-us-west-2.amazonaws.com/w10n/taiga-1.6.6-linux-x86_64-b.tar.gz

4, SSH into your instance

#log into the machine you just spun up
> ssh -i <your ssh key.pem> ubuntu@<address of the machine you just created>

# Configure this new w10n server to use port 80
ubuntu$ sudo ./taiga/taiga-1.6.6-linux-x86_64-b/bin/taiga-service config -p 80 -d /vol/data

# Start your server
ubuntu$ sudo ./taiga/taiga-1.6.6-linux-x86_64-b/bin/taiga-service start

# Verify that your server is running & logout
ubuntu$ curl localhost
ubuntu$ exit

## Now point your browser to the <address of the machine you just created> and interact with your favorite API!

5, Hack something masterful! And/or get in touch with us (Alex Smith or Rob Witoff):

• Realtime Collaborative Q&A with the developers here
• Realtime video chat with the developers here

AWS Resources

n.b. All resources are located in the Oregon us-west-2 AWS Region

• AMI: Pre-built machine images
  • ami-3eef740e: Pre-packaged volume snapshots and configured machine.
• EBS: Ready-to-go filesystems
  • snap-42c7797d: All the below earth science data
• S3: Data Bucket
• Grace Land data [29.9 MB]
• Grace Ocean data [30.4 MB]
• Modis data [568.5 MB]
• Oscar data [55.5 MB]
• Terabytes of additional oceanography data is below

Background Mission Data:

• Grace: NASA's Gravity Recovery & Climate Satellite here. Grace launched in March of 2002 to investigate water and land data through detailedgravitational field measurements.
• Information on Grace's Webified datasets are here: water and land
• Modis: NASA's Moderate-Resolution Imaging Spectroradiometer here was launched in '99 with instruments to measure the Earh's sea-surface tempuratue.
• Information on Modis's Webified datasets is here
• Oscar: Wind current speed & velocity
• Information on Oscar's Webified datasets is here

Resources:

• w10n Homepage
• w10n Tutorial
• w10n Visualization Toolkit
• w10n Interactive Examples
• re:Invent Hacathon Homepage

More Data:

• Expert Data Bucket here [~100 GB]
• NASA NEX Data Bucket
• AMSRE Data
• Tellus Ocean Data
• Tellus Land Data
• Odyssea Data

Developers

• Zhangfan Xing
• Alex Smith
• Jeremy Arca
• And many more...