Comments (6)
Pathway discussed Sept 20: calculate partial LUT for min/max parameters (e.g., wind, AOT, etc.) impact on difference between M99 and Z17 rather than calculated Z17 for each spectrum or LUTs for all possible parameterizations.
from hypercp.
pySAS, M99, turning off satellite convolution to Aqua reduces timing from to 2 min.
The snag is in instrument.rrsHyperUNCFACTORY is quite slow, ~160s per ensemble/xSlice:
Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
Process Single Level
ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
Convolving MODIS Aqua (ir)radiances in the slice
67 spectra in slice (ensemble).
7 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.3 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 160.2 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717
… [Repeat for each ensemble]
Filtering reflectance spectra for negative values.
0.0% of Rrs_HYPER spectra flagged
0.0% of nLw_HYPER spectra flagged
Plotting Rrs
Plotting Rrs
Plotting Es
Plotting Li
Plotting Lt
Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL
L2 file produced:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Output SeaBASS for HDF:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Writing PDF Report...
Level 1A
Level 1AQC
Adding deglitching plots...
Level 1B
Adding interpolation plots...
Level 1BQC
Adding spectral filter plots
Level 2
Adding radiometry plots
Adding ocean color product plots
processFilesSingleLevel, single file - DONE
Time elapsed: 14 minutes
from hypercp.
Without Aqua convolution takes < 1 minute:
Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
Process Single Level
ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
67 spectra in slice (ensemble).
7 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.3 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717
80 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.1 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.679655711474021e-06; offset(nLw) = -0.033131963419616335
78 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.1 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.3 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -2.845188536517354e-05; offset(nLw) = -0.034560316042568845
76 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = 1.2613785225814207e-06; offset(nLw) = -0.030815234070160086
49 spectra in slice (ensemble).
5 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = 8.215846949638485e-05; offset(nLw) = -0.020145435286546286
Filtering reflectance spectra for negative values.
0.0% of Rrs_HYPER spectra flagged
0.0% of nLw_HYPER spectra flagged
Plotting Rrs
Plotting Rrs
Plotting Es
Plotting Li
Plotting Lt
Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL
L2 file produced:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Output SeaBASS for HDF:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Writing PDF Report...
Level 1A
Level 1AQC
Adding deglitching plots...
Level 1B
Adding interpolation plots...
Level 1BQC
Adding spectral filter plots
Level 2
Adding radiometry plots
Adding ocean color product plots
processFilesSingleLevel, single file - DONE
Time elapsed: 1 minutes
from hypercp.
Also, while Z17 is apparently not the logjam here, I have so far found that updating Z17 has only increased its size, considering the newly added AOD to 0.5. From 2.47 GB, it goes to 3.28 GB. I can bring it back down to 2.26 GB by truncating AOD of 0 and SZA of 0 (i.e., starting at 0.05 and 10, respectively). Does not seem worth it yet, but will continue to experiment.
from hypercp.
Two issues to address: Z17 uncertainties at L2 are too slow (i.e., processing time is nearly. as long as data collection time). For this we need a LUT for M99 - Z17.
The second issue is that the convolutions to satellite bands are excessively slow, making even M99 processing interminable. Setting created to forgo convolution uncertainties until a resolution can be identified.
from hypercp.
One confusing element about the slowness of running Z17 uncertainties is that when zhangWrapper runs Z17 for uncertainty estimates, each run takes much longer than the run of Z17 to do the glint correction:
...
6 spectra in slice (ensemble).
Wavelengths extend beyond model limits. Truncating to 350 - 1000 nm.
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 9.1 s
Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.5 SST: 13.1 SSS: 35.6
Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 135.0
zhangWrapper Z17 Elapsed Time: 35.8 s
Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.7 SST: 15.7 SSS: 34.8
Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 133.5
zhangWrapper Z17 Elapsed Time: 35.5 s
...
from hypercp.
Related Issues (20)
- Main.py: paths defined in script are defined wrt current working directory and not code's directory HOT 1
- Optimise L2 uncertainties by saving Py6s output from L1B HOT 1
- ES, LI and LT should be used to calculate band convolved Rrs and LW HOT 1
- Polarization correction for FRM branch (longer term issue) HOT 1
- New RADCAL with old FidRadDB characterizations HOT 2
- py6S Cloud flag automation HOT 3
- L2 Broadcast input array error; NewRhoHyper HOT 1
- Add bincount to SeaBASS files HOT 1
- M02 BRDF implementation error HOT 2
- BRDF: Unify M02 and L11 workflows HOT 1
- Negative value intolerance in comet-maths HOT 1
- Issue with SeaBASS output HOT 1
- Potential Reports issue with station extraction HOT 1
- Single ancillary timestamp kept in `ProcessL1aqc.py` HOT 4
- New TriOS Config instruments (frames) HOT 1
- TriOS; Factory files (Cal, Back) have NANs for many pixels HOT 3
- Error with comet _maths and negative values HOT 3
- Indexerror at L1B->L1BQC - possibly due to long station time HOT 5
- SEABASS output file BRDF correction HOT 1
- Tilt-Heading sensor HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hypercp.