nasa / hypercp

Hi 👋, welcome to the NASA org on github.com!

Github.com/nasa has one of the largest collections of NASA open-source code repositories. Members of the NASA org can find instructions for github.com/nasa in http://nasa.github.io/.

🔭 Additional open-source code repositories resides in a variety of locations other than github.com/nasa. To discover code across all of these locations, we suggest you use code.nasa.gov & software.nasa.gov. These are two different sites holding metadata that describe code projects. Any code released through the NASA Software Release Authority process should be cataloged on those sites.

Code.nasa.gov —

Is a page with short descriptions of all of NASA's open-source code. Code.nasa.gov feeds into code.gov, which covers open-source and government-source code from many different U.S. governmental agencies. To assist in discovery, code projects described on code.nasa.gov have both human and A.I.-generated tags. These can be useful for finding related code projects.

software.nasa.gov —

Contains metadata descriptions for all code projects in code.nasa.gov as well as government-source code projects only sharable with other government agencies. It is part of the large https://technology.nasa.gov/ that also includes patents and spinoffs. To help discoverability, software.nasa.gov puts each code project into one fo the following categories: Business Systems and Project Management, System Testing, Operations, Design and Integration Tools, Vehicle Management (Space/Air/Ground), Data Servers Processing and Handling, Propulsion, Structures and Mechanisms, Crew and Life Support, Data and Image Processing, Materials and Processes, Electronics and Electrical Power, Environmental Science (Earth, Air, Space, Exoplanet), Autonomous Systems, and Aeronautics.

NOTE - PROFILE READMES CURRENTLY DON'T WORK FOR ORG PROFILES ONLY USER PROFILES :(

https://github.community/t/readme-for-organization-front-page/2920

hypercp's Issues

Class-based cals L1B

L1B pathway for class-based characterization and calibration is incomplete for SeaBird. For TriOS, this path is coded to do the same thing as factory calibration (is this intentional?)

Zhang method fail in L2 processing

Error when running Zhang instead of Mobley.
Happens at L2 processing:

File "/mount/internal/work-st/projects/tartu-1281/FRM4SOCv2/workspace/aderu/nasa/HyperInSPACE/Source/ProcessL2.py", line 1616, in ensemblesReflectance
rhoVector, rhoUNC = RhoCorrections.ZhangCorr(WINDSPEEDXSlice,AODXSlice, CloudXSlice, SZAXSlice, SSTXSlice,

File "/mount/internal/work-st/projects/tartu-1281/FRM4SOCv2/workspace/aderu/nasa/HyperInSPACE/Source/RhoCorrections.py", line 170, in ZhangCorr
rhoDelta = Propagate.zhangWrapper(mean_vals=varlist, uncertainties=ulist,

TypeError: Propagate.zhangWrapper() got an unexpected keyword argument 'mean_vals'

Minor Improvements

Minor improvements
+ Update run as python package:
+ Added example on how to run multiple instances from command line (run.py)
+ Added environment variable HYPERINSPACE_CMD
+ Prevent showing error window when running in command line (error still show in standard output and logs)
+ Answered question in Main.py regarding error with pylint and dependencies: major refactoring is required which would improve the behaviour of the code when used as a python package instead of with the GUI
+ Removed unnecessary dependencies in OCproductsWindow, ProcessL2, Utilities
+ Improve row deletion of hdf files in Process L1aqc, ProcessL2, Utilities

Incorporate TriOS instrumentation

Interpreter modules required for raw TriOS sensor data for ingestion at RAW and propagation through Level-2.

SeaBird Non-FRM Class-based

ProcessL2.ensembleReflectances:

SeaBird in "Factory" mode should calculate class-based uncertainties using class-based RADCAL uncertainties for SeaBird.

@ARamsay17

Z17 pathway needs to truncate to Zhang database

350 - 1000 nm. Should eliminated pesky NaNs. Databases should be truncated, not filled with zeros.

Automatically download FidRadDB characterization files

Instrument characterization files stored in EUMETSAT OCDB can be automatically downloaded and stored locally as needed

ECMWF error

dev branch

GetAnc_ecmwf.EAC4_download_ensemble returns error "EAC4 atmospheric data could not be retrieved. Check inputs."

Currently called in L1BQC, but soon to be moved to L1B.

Zhang rho efficiency

Running the Zhang et al. 2017 glint correction is inefficient and slow.

Windows h5py/hdf5 conflict

Windows OS only. h5py fails to import

See description here: https://stackoverflow.com/questions/68296054/importerror-dll-load-failed-while-importing-defs-the-specified-procedure-could

I tried the solution to conda install h5py=3.6.0 -c pkgs/main within the hypercp environment. I can at least now activate hypercp, launch python, and import h5py, but when HyperCP HDFRoot.py tries (also within hypercp environment), I still get the error.

L1B CosCorr, AOT reorganization

MERRA2 AOT to be imported at beginning of L1B or earlier.

Cosine correction (CosCorr) Py6S computation times need to be reduced. Potentially use time-averaged direct/diffuse ratios. (E.g., use L2 ensemble times of ~300s to reduce Lsky and Es from O(2000) spectra to O(20) after dark correction.) Problem still remains that calibrations are applied prior to timestamp/waveband matching. Possible to move CosCorr to after matching?

Likely two PRs to resolve:
DAA: Move MERRA2 acquisition to begin L1B
AD: Move Py6S CosCorr to end L1B and use ensemble-length averages for direct/diffuse

Pre-process L1A-L1AQC when launching Anomaly Analysis

Currently, Anomaly analysis has to open a L1AQC file to evaluate deglitching paramterizations prior to processing to L1AQC. This requires running L1AQC with no deglitching first, then going back and opening it in the AnomAnal tool.

This should open L1A file and process with no deglitching prior to running the tool. Slower, but less confusing and error prone.

TriOS Factory L2 fail

TriOS in Factory pathway fails when trying to plot nLw, because nLw_HYPER_unc has 139 np.arrays (one for each waveband), but each array has 194 values instead of 1 value for the uncertainty in nLw in that band.

Ir/radiances and Rrs all have _unc datasets set to all zeroes. nLw does not output nLw_HYPER_unc.

negative degrees not correctly converted

https://github.com/nasa/HyperInSPACE/blob/951206f742dd280db7f8d4ad2722bdce6f55a2c5/Source/Utilities.py#L187-L193

-0.05 degrees is 3 minutes (South or West).
ddToDm(-0.05) returns 3.0, meaning 3 minutes (North or East).

-1.05 degrees is 1 degree 3 minutes (South or West).
ddToDm(-1.05) returns -97.0, meaning 97 minutes (South or West), equivalent to 1 degree 37 minutes.

Conda virtual environment takes too long to build and may have errors

Solution: simplify the conda environment.yml to include less and specify build versions where possible.

AVW/QWIP don't trigger Aqua by default

Full-characterization file names have changed

Rewrite file ingestion codes to accommodate new file/folder/format from Tartu

Py6S.sixs_exceptions.ExecutionError: 6S executable not found

Occasionally, when running HyperCP in VSCode, this error is triggered in L1AQC->L1B processing. The problem appears to be that -- for whatever reason -- VSCode did not activate the hypercp environment properly (even though it appeared to and every other dependency is found). Shutting down HyperCP and restarting it appears to resolve the issue.

Similar issues were reported with Py6S here:
robintw/Py6S#52

nLw uncertainties

Need to confirm that DA's RSS implementation of F0 uncertainties from Coddingtion TSIS-1 F0 into nLw uncertainties (ProcessL2 L456) conforms.

@ARamsay17

Add Morel f/Q BRDF uncertainties

Based on today's discussion, these uncertainties are known, but have not been rolled in. Hopefully we can incorporate these also by next week for the merge to master.

@ARamsay17

Test issue

Test issue for pull request (PR) testing.

Revisit f/Q BRDF correction

The Morel & Gentili 1996/2002 f/Q BRDF correction implementation needs attention. Determination of gothic R is ambiguous, and no iteration currently implemented. Implications of scipy spline to hyperspectral f/Q are unclear.

New environment.yml still causing issue on some Nix systems

On Ubuntu 20.04.6 LTS (but not all of them) and conda 23.3.1, using environment.yml fe65804

(test) daurin@gs616-analysis702:~/GitRepos/HyperInSPACE$ python Main.py
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, vnc, webgl, xcb.

Slow processing of L2 uncertainties

Ideas to pursue:

Reduce the size of Z17 by confining it to protocol geometries (DA).

Build LUTs in lieu of full calculations for given conditions (AR).

Fix L1B_FRM for SolarTracker, no Anc

Conflict. Resolution pending. Also involves L1B Interp

pySAS FRM-Full L2 fail

Error in L2 process:
File "/mount/internal/work-st/projects/tartu-1281/FRM4SOCv2/workspace/aderu/miniconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/random/generate_sample.py", line 262, in generate_sample_random
sample = sample_pdf * u_param[sli_par] + param[sli_par]
ValueError: operands could not be broadcast together with shapes (100,255) (1,165)

Something to do with a missing transpose...
@aramsay, @AlexisDeru

TriOS Factory (7a97d7c)

File "/Users/daurin/GitRepos/HyperCP/Source/ProcessL2.py", line 1718, in ensemblesReflectance
xUNC = instrument.factory(node, uncGroup,
^^^^^^^^^^^^^^^^^^
AttributeError: 'Trios' object has no attribute 'factory'. Did you mean: 'Factory'?

Full uncertainty budget

Modules are required to incorporate FRM-style full sensor characterization files (individual and class-based) and to propagate associated uncertainties from RAW through Level-2 including improved uncertainty estimates for glint and other corrections/adjustments to the radiometry.

ProcessInstrumentUncertainties.py "mean_values" and "uncertainty"

Near line 116 and 126, it tries to broadcast into lists (mean_values and uncertainty) data from stats with 255 elements with data from Coeff and Cal with 10 elements. resulting in, e.g.:

File "/Users/daurin/GitRepos/HyperCP/Source/ProcessL2.py", line 1690, in ensemblesReflectance
xSlice.update(instrument.Default(uncGroup, stats)) # update the xSlice dict with uncertianties and samples
File "/Users/daurin/GitRepos/HyperCP/Source/ProcessInstrumentUncertainties.py", line 116, in Default
mean_values = [stats['ES']['ave_Light'], ones*stats['ES']['ave_Dark'],
ValueError: operands could not be broadcast together with shapes (10,) (255,)

ProcessInstrumentUncertainties.py raw_data math

L1383 is trying to do math with raw_data, but raw_data is an OrderedDict with a numpy array for each waveband, keyed by waveband string name.

Z17 pySAS Factory

File "/Users/daurin/GitRepos/HyperCP/Source/ProcessInstrumentUncertainties.py", line 209, in Factory
radcal_cal = pd.DataFrame(uncGrp.getDataset(sensor + "_RADCAL_CAL").data)['2']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'data'

Datetime x-axis broken in Anomaly Analysis tool

Utilities.plotRadiometry is called by ProcessL1aqc_deglitch.py and AnomalyDetection.realTimePlot used to be able to project the x-axes in Light and Dark plots onto a datetime rather than serial number using matplotlib. Updates to the environment have broken this feature, and it was reverted to serial, which makes cross-referencing Light/Dark difficult.

AnomalyDetection.plotButtonPressed > Utilities.saveDeglitchPlots still exports to PNG with datetime x-axes.

Consider docker distribution

It may be more user friendly to use a Docker to contain the code for distribution to avoid conda environment issues.

L2 uncertainties missing

Running SEABIRD_SOLARTRACKER in the Factory Only pathway calculates uncertainties, as evidenced by the very slow use of Z17 despite running for M99 (i.e., to find the difference and add as an uncertainty contribution), but the L2 files have all zeros for ir/radiance and reflectance _unc datasets, and no nLw_unc dataset.

HyperCP updates to README, Guidelines, Changelog, License, etc.

To bring HyperCP into compliance with the agreed-upon guidelines, the language of the following files needs to be updated or added to the frm4soc2 branch:

README.md
HyperCP-Collaboration_Guidelines.md
Changelog.md
LICENSE.txt

Additional language for the README may be needed from NPL and ACRI for updates to instructions regarding end-to-end uncertainties and TriOS implementation.

TriOS calMap broken

I clearly broke the GUI for TriOS cal files when I automated the SeaBird cal file recognition logic.

ProcessInstrumentUncertainties.py TriOS FRM-Class

Fails on similar broadcast error to SeaBird, but at a different location:

File "/Users/daurin/GitRepos/HyperCP/Source/ProcessL2.py", line 1690, in ensemblesReflectance
xSlice.update(instrument.Default(uncGroup, stats)) # update the xSlice dict with uncertianties and samples
File "/Users/daurin/GitRepos/HyperCP/Source/ProcessInstrumentUncertainties.py", line 137, in Default
ES_unc, LI_unc, LT_unc, ES_rel, LI_rel, LT_rel = PropagateL1B.propagate_Instrument_Uncertainty(mean_values,
File "/Users/daurin/GitRepos/HyperCP/Source/Uncertainty_Analysis.py", line 63, in propagate_Instrument_Uncertainty
sensor = self.instruments(mean_vals)
File "/Users/daurin/GitRepos/HyperCP/Source/Uncertainty_Analysis.py", line 185, in instruments
return np.array((ESLIGHT - ESDARK)ESCalESStabESLinESStrayEST*ESCos),
ValueError: operands could not be broadcast together with shapes (208,) (10,)

Speed up computation of rho with Zhang et al. 2017

Speedup computation of rho with Zhang et al. 2017
+ Improved performance of get_prob x1.5
+ Added cache:
+ Keep LUT from Zhang et al. 2017 loaded in memory
+ Round environmental parameters and added lru_cache on get_prob
+ Replaced interpn from xarray by scipy function for increased speed
+ Added option to run interpn by chunks for datasets exceeding CPU cache (still room for improvements)
+ Limited use of xarray to only reading matlab file, all arrays are with numpy
+ Replaced OrderedDict by normal Dictionary as since python 3.6 Dictionary are ordered and code didn't require data to be ordered.
+ Updated vector direction to avoid unnecessary 2D arrays and have matrix shapes closer to the original matlab code
=> Committed code is up to 9x faster than past code on macBook Pro 2019 with Intel i9 CPU

IMKClient Stall detected

Pesky new error message pops up frequently after upgrade to macOS Ventura 13.4

2023-06-09 10:11:30.284 python[10409:3541769] IMKClient Stall detected, please Report your user scenario attaching a spindump (or sysdiagnose) that captures the problem - (imkxpc_bundleIdentifierWithReply:) block performed very slowly (2.74 secs).

Easy to reproduce, but difficult to track to a cause or troubleshoot. Appears harmless, but annoying.

specify precision for degree/decimal conversion

https://github.com/nasa/HyperInSPACE/blob/951206f742dd280db7f8d4ad2722bdce6f55a2c5/Source/Utilities.py#L177-L193

It is reasonable to expect that ddToDm(dmToDd(n)) == n for all n. However, for n = 31 (that is, 0 degrees 31 minutes), this equality does not hold because of the inexact nature of floating point numbers - the value differs from expected by 3.5e-15 (it may depend on hardware and implementation). Curiously, this behavior does not occur for <anything other than 0> degrees 31 minutes.

I recommend that dmToDd accept an optional precision parameter (defaulting to, say, 3 or 6 decimal places) and round the results accordingly.

Add 3C glint correction

Philipp M. M. Groetsch, Peter Gege, Stefan G. H. Simis, Marieke A. Eleveld, and Steef W. M. Peters, "Validation of a spectral correction procedure for sun and sky reflections in above-water reflectance measurements," Opt. Express 25, A742-A761 (2017).

PG to adapt in python to L2 processing in module called by ProcessL2.py>RhoCorrections.py. Uncertainty analysis to follow separately.

MainConfig.settings load in ProcessL1a

MainConfig.settings called in ProcessL1a.py before running loadConfig.

Automate SeaBird Cal/TDF file interpretations in GUI

Need to simplify interface by interpreting SeaBird cal files on import in ConfigWindow to recognize

HED/HLD -> Dark Frame
HSE/HSL - > Light Frame

And enable each by default.

Add IOP-based BRDF correction option

Non-positive definite covariance matrix

When running for SeaBird_noTracker (factory mode), I get several warnings:

Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/tcenas/proj/ocean/color/conda/miniconda3/envs/hypercp/lib/python3.10/site-packages/comet_maths/linear_algebra/matrix_calculation.py:284: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.

FRM Branch Uncertainty Budget

Updates are required to apply proper uncertainty analysis and propagation for FRM branch processing, so that the uncertainties correspond to corrected outputs.

Propagated L1AQC data filtering and units

L1AQC data propagated to L2 for uncertainties have calibrations applied in error.

Need to confirm propagated L1AQC are filtered the same as other groups in L1BQC and L2 (they should be).

Demo the Issue-Fork-Branch-PR

This issue is to demonstrate the process of submitting an Issue, Forking, creating a development Branch, and submitting a Pull Request (PR). The "fix" will involve demonstrating each of these steps and it will be testing by completing them according to the Guidelines.

L1B py6S error

ProcessL1B_FRMCal.py throws an error on call (near line 69):

wavelengths, res = Py6S.SixSHelpers.Wavelengths.run_wavelengths(s, 1e-3*wvl)

TypeError: only size-1 arrays can be converted to Python scalars

Tidy up command line functionality

Need to update how version is passed along to MainConfig
Model Main.Command MainConfig settings after GitLab version that includes a new .config file for command line runs
Clean up run.py to be able to use the sample data provided with the master repo as and example. Rename to run_sample.py and provide guidance in the README on how it can be adapted (and renamed) by the user.