Slow processing of L2 uncertainties about hypercp

[oceancolorcoder]

Pathway discussed Sept 20: calculate partial LUT for min/max parameters (e.g., wind, AOT, etc.) impact on difference between M99 and Z17 rather than calculated Z17 for each spectrum or LUTs for all possible parameterizations.

@ARamsay17

[oceancolorcoder]

pySAS, M99, turning off satellite convolution to Aqua reduces timing from to 2 min.

The snag is in instrument.rrsHyperUNCFACTORY is quite slow, ~160s per ensemble/xSlice:

Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
Process Single Level
ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
Convolving MODIS Aqua (ir)radiances in the slice
67 spectra in slice (ensemble).
7 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.3 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 160.2 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717

… [Repeat for each ensemble]

Filtering reflectance spectra for negative values.

0.0% of Rrs_HYPER spectra flagged
0.0% of nLw_HYPER spectra flagged
Plotting Rrs
Plotting Rrs
Plotting Es
Plotting Li
Plotting Lt
Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL
L2 file produced:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Output SeaBASS for HDF:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Writing PDF Report...
Level 1A
Level 1AQC
Adding deglitching plots...
Level 1B
Adding interpolation plots...
Level 1BQC
Adding spectral filter plots
Level 2
Adding radiometry plots
Adding ocean color product plots
processFilesSingleLevel, single file - DONE
Time elapsed: 14 minutes

[oceancolorcoder]

Without Aqua convolution takes < 1 minute:

Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
Process Single Level
ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
67 spectra in slice (ensemble).
7 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.3 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717
80 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.1 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.679655711474021e-06; offset(nLw) = -0.033131963419616335
78 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.1 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.3 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -2.845188536517354e-05; offset(nLw) = -0.034560316042568845
76 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = 1.2613785225814207e-06; offset(nLw) = -0.030815234070160086
49 spectra in slice (ensemble).
5 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = 8.215846949638485e-05; offset(nLw) = -0.020145435286546286
Filtering reflectance spectra for negative values.
0.0% of Rrs_HYPER spectra flagged
0.0% of nLw_HYPER spectra flagged
Plotting Rrs
Plotting Rrs
Plotting Es
Plotting Li
Plotting Lt
Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL
L2 file produced:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Output SeaBASS for HDF:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Writing PDF Report...
Level 1A
Level 1AQC
Adding deglitching plots...
Level 1B
Adding interpolation plots...
Level 1BQC
Adding spectral filter plots
Level 2
Adding radiometry plots
Adding ocean color product plots
processFilesSingleLevel, single file - DONE
Time elapsed: 1 minutes

[oceancolorcoder]

Also, while Z17 is apparently not the logjam here, I have so far found that updating Z17 has only increased its size, considering the newly added AOD to 0.5. From 2.47 GB, it goes to 3.28 GB. I can bring it back down to 2.26 GB by truncating AOD of 0 and SZA of 0 (i.e., starting at 0.05 and 10, respectively). Does not seem worth it yet, but will continue to experiment.

[oceancolorcoder]

Two issues to address: Z17 uncertainties at L2 are too slow (i.e., processing time is nearly. as long as data collection time). For this we need a LUT for M99 - Z17.

The second issue is that the convolutions to satellite bands are excessively slow, making even M99 processing interminable. Setting created to forgo convolution uncertainties until a resolution can be identified.

[oceancolorcoder]

One confusing element about the slowness of running Z17 uncertainties is that when zhangWrapper runs Z17 for uncertainty estimates, each run takes much longer than the run of Z17 to do the glint correction:

...
6 spectra in slice (ensemble).
Wavelengths extend beyond model limits. Truncating to 350 - 1000 nm.
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 9.1 s
Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.5 SST: 13.1 SSS: 35.6
Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 135.0
zhangWrapper Z17 Elapsed Time: 35.8 s
Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.7 SST: 15.7 SSS: 34.8
Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 133.5
zhangWrapper Z17 Elapsed Time: 35.5 s
...