gridsst-hackathon / data-pre-processing

Data Pre Processing

This repository is home to the access/importing of L2 Data Resources:

From this a consensus emerged on high level objective and secondary needs (L1/L2 system engineering requirements)

Level 1 requirements (the foundational requirements)

Study Area & Temporal Coverage:

• Time period 2011-present (forward stream)
• Start with VIIRS era (2012?)
• North and South Atlantic Ocean Basins (GOES-16)

5 sensors to blend from polar orbiting IR, MW and Geostationary IR using night+day, AM and PM orbits

• L2P S-NPP VIIRS (PM) [2012-Feb-01 to Present]
• L2P MetopB AVHRR (AM) [2012-Oct-19 to Present]
• L2P GCOM AMSR2 and realtime dataset (PM) [2012-Jul-2 to Present]
• L2P GOES-16 [2017-Dec-15 to Present]
• L2P SEVIRI (Metosat)

Reference SST is CMC L4

Level 2 requirements (probably not attainable this week)

Reference is the ASTR/SLSTR series IR radiometers or in situ data SST adjusted using a diurnal warming model Motion compensation A global output vs. regional proof of concept

L2 & L3 Aggregate Product Deliverables

• Native geo-location retained for each pixel (no super obs)
• Re-Packing includes SST, Quality Level (3-5), SSES, Wind, and others
• Re-Packing includes flagged daytime screened SST using a wind speed (which one?)

NOTES: Daily 24 hour product as diurnal free as possible Reference is the ASTR/SLSTR series IR radiometers or in situ data

data-pre-processing's Issues

Bias Correction Clarity

Starting this issue as thread on the topic for bias correction code. Quick status update of where things are, and where some clarity on next steps could be.

Currently it seems like the following pre-processing steps have been implemented & turned into functions:

1. Lookup and access s3 granules for L2P and L3 (CMC) resources
2. Pre-process L2P Sensor Data in preparation for bias-correction
3. Load CMC data to use for bias-correction

What is somewhat unclear, or in a pseudo-code form is what reshaping needs to happen, and what processes need to be performed to atually do the bias-correction. @andy123harris has sent me pseudo code for the process, and I have included it at the end of this notebook https://github.com/gridSST-hackathon/data-pre-processing/blob/main/bias_correct_pseudo.ipynb

Please respond here with any comments/feedback on that assesment or any details I've missed. HUGE thanks to all of y'all for letting me be a part of this group, cheers!