opera-adt / rtc Goto Github PK

OPERA Radiometric Terrain-Corrected (RTC) Product

License: Apache License 2.0

Dockerfile 0.42% Python 99.40% Shell 0.18%

rtc's Introduction

RTC

NASA's Observational Products for End-Users from Remote Sensing Analysis (OPERA) Radiometric Terrain-Corrected (RTC) SAR backscatter from Sentinel-1 (RTC-S1) Science Application Software developed by the OPERA Algoritm Development Team at NASA's Jet Propulsion Laboratory (JPL).

Install

Instructions to install RTC under a conda environment.

Download the source code:

git clone https://github.com/opera-adt/RTC.git RTC

Install isce3:

conda install -c conda-forge isce3

Install s1-reader via pip:

git clone https://github.com/opera-adt/s1-reader.git s1-reader
conda install -c conda-forge --file s1-reader/requirements.txt
python -m pip install ./s1-reader

Install RTC via pip:

git clone https://github.com/opera-adt/s1-reader.git s1-reader
python -m pip install ./RTC

Usage

The command below generates the RTC product:

rtc_s1.py <path to rtc yaml file>

To compare the RTC-S1 products, use rtc_compare.py.

python rtc_s1.py <1st product HDF5> <2nd product HDF5>

License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of Caltech nor its operating division, the Jet Propulsion Laboratory, nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

rtc's People

Contributors

Stargazers

Watchers

Forkers

gshiroma seongsujeong tyler-g-hudson scottstanie kimbaldry forrestfwilliams

rtc's Issues

[New Feature]: Validation script

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Describe the feature request

We need a script to validate the output product with the golden dataset.

CircieCI and tester module

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Describe the feature request

I need or want [...]

GDAL `No such file or directory` when using unzipped SAFE file

Currently, rtc_s1.py fails with a GDAL No such file or directory error if the input Sentinel-1 data is unzipped.

This can be recreated by processing the Los Angeles sample data hosted on zenodo, unzipping the provide SAFE file, then running rtc_s1.py with this configuration file.

This error occurs because calling s1reader's Sentinel1BurstSlc.slc_to_vrt_file() method writes a VRT representing the burst SLC data with a relative path to the parent geotiff that assumes the VRT is in the same directory as the SAFE file. However, in RTC this VRT is written to a temporary subdirectory with this structure:

input_dir
└── S1_SLC.SAFE
scratch_dir
└── temp_*
   └── temp_*
      └── BURST_ID
         └── slc_VV.vrt

Thus, GDAL is not able to locate the parent geotiff using the information in the VRT.

I can think of two ways to remedy this:

Modify Sentinel1BurstSlc.slc_to_vrt_file() so that it uses absolute file paths
Force RTC to write burst VRTs into the same directory as the input SAFE file

Personally, I prefer option 1 because it fixes the problem for other projects as well, and requires fewer code changes.
error.txt

Docker image: ImportError: cannot import name 'get_orbit_file_from_list' from 's1reader.s1_orbit'

Facing the following issue while trying to run a container from a successfully built docker image. I tried to install it manually but import RTC is working instead import rtc again throwing the same error as below (after fixing for case sensitive import ). Could you please help.

On DOCKER

ERROR conda.cli.main_run:execute(41): `conda run rtc_s1.py` failed. (See above for error)
Traceback (most recent call last):
  File "/home/rtc_user/miniconda3/envs/isce3_rtc/bin/rtc_s1.py", line 20, in <module>
    from rtc.runconfig import RunConfig
  File "/home/rtc_user/miniconda3/envs/isce3_rtc/lib/python3.9/site-packages/rtc/runconfig.py", line 19, in <module>
    from s1reader.s1_orbit import get_orbit_file_from_list
ImportError: cannot import name 'get_orbit_file_from_list' from 's1reader.s1_orbit' (/home/rtc_user/miniconda3/envs/isce3_rtc/lib/python3.9/site-packages/s1reader/s1_orbit.py)

Manual Installation

Traceback (most recent call last):
  File "//home/RTC/app/rtc_s1.py", line 20, in <module>
    from rtc.runconfig import RunConfig
  File "/home/RTC/src/rtc/runconfig.py", line 19, in <module>
    from s1reader.s1_orbit import get_orbit_file_from_list
ImportError: cannot import name 'get_orbit_file_from_list' from 's1reader.s1_orbit' (/usr/local/lib/python3.10/dist-packages/s1reader/s1_orbit.py)

Update required for `environment.yml`

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Describe the feature request

I need or want [...]

[Bug]: Process hanging indefinitely with apply_shadow_masking : true

Checked for duplicates

Yes - I've already checked

Describe the bug

I am creating some backscatter products over the Antarctic region (coastal and inland). The workflow works as expected for a wide range of products I have tested. However, when I configure the workflow to apply and save the layover shadow mask, the process seems to hang indefinitely on certain bursts. This will only happen on some products.

E.g. Here are the logs for a test scene where I encountered the issue below. I let the process sit for approx. 10 hours. There was still high CPU usage so the process seemed to be ongoing.

Identification:
    product type: RTC_S1
    product version: 1.0.1
Ancillary input(s):
    DEM file: /data/dem/glo_30/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE_dem.tif
Processing parameters:
    apply RTC: True
    apply thermal noise correction: True
    apply absolute radiometric correction: True
    apply valid samples sub-swath masking: True
    apply shadow masking: True
    apply bistatic delay correction: False
    apply static tropospheric delay correction: False
    skip if already processed: False
    scratch dir: /data/OPERA/scratch/temp_1707445567.2457469/temp_1707452886.070618
    output dir: /data/OPERA/outdir/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE
    save bursts: True
    save mosaics: False
    save browse: True
    output imagery format: COG
    output imagery compression: DEFLATE
    output imagery nbits: 32
    save secondary layers as HDF5 files: True
    check ancillary coverage: False
Save layers:
    Mask Layer: True
    RTC area normalization factor: True
    RTC area normalization factor Gamma0 to Beta0: True
    Number of Looks: True
    Incidence Angle: False
    Local Incidence Angle: True
    Projection Angle: False
    RTC Area Normalization Factor Gamma0 to Beta0 (Projection Angle - ProjectionAngle): True
    Range Slope: False
    Digital Elevation Model (DEM): True
Browse images:
    burst height: 2048
    burst width: None
Check ancillary inputs' coverage:
    DEM file coverage: (not tested)
Number of bursts to process: 1
Processing burst: t164_351771_iw2 (1/1)
    product ID: OPERA_L2_RTC-S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE
    burst geogrid:
        start X: -732300.000000
        end X: -700140.000000
        start Y: 1519860.000000
        end Y: 1417680.000000
        spacing X: 20.000000
        spacing Y: -20.000000
        width: 1608
        length: 5109
        epsg: 3031
    reading burst SLCs
    applying thermal noise correction to burst SLC
    applying absolute radiometric correction to burst SLC
    computing layover shadow mask for t164_351771_iw2
journal:
 -- DEM EPSG: 4326
 -- Output EPSG: 4326
journal:
 -- Processing block: 1 
 --   - line start: 0
 --   - line end  : 1000
 --   - dopplers near mid far: 0 0 0 
journal:
 -- Actual DEM bounds used:
 -- Top Left: -27.2192 -74.4911
 -- Bottom Right: -24.6192 -75.6261 
 -- Spacing: 0.000833333 -0.000277778
 -- Dimensions: 3121 4087

Topo progress (block 1/2): 100%

it hung here.

What did you expect?

I ran the same workflow again with both apply_shadow_masking : false and save_mask : false set and it worked as expected:

apply_shadow_masking : false and save_mask : false -> SUCCESS (approx 50 mins)
apply_shadow_masking : true and save_mask: true -> HANGING (process still ongoing after 10 hours)

If I had to guess I thought it could potentially be a DEM coverage issue? In some areas there can be a large amount of no-data in the DEM.

Here is the .yaml file I am using. Details on the scene can be found here.

runconfig:
  name: rtc_s1_workflow_default

  groups:
      primary_executable:
          product_type: RTC_S1

      pge_name_group:
          pge_name: RTC_S1_PGE

      input_file_group:
          # Required. List of SAFE files (min=1)
          safe_file_path: [/data/scenes/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE.SAFE]

          orbit_file_path: [/data/osv/POEORB/S1B_OPER_AUX_POEORB_OPOD_20210315T112322_V20210222T225942_20210224T005942.EOF]

          # Optional. Burst ID to process (empty for all bursts)
          # burst_id: [t069_147173_iw1, t069_147174_iw1]
          # burst_id: [t071_151225_iw1, t071_151226_iw1]

      dynamic_ancillary_file_group:
          # Digital elevation model
          # dem_file: /data/dem/glo_30/S1B_IW_SLC__1SSH_20190223T222639_20190223T222706_015079_01C2E9_1D63_dem.tif
          dem_file: /data/dem/glo_30/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE_dem.tif

          # Digital elevation model description
          dem_file_description:

      static_ancillary_file_group:

          # burst database sqlite file
          burst_database_file:

      product_group:
          processing_type: 'CUSTOM'

          # Directory where PGE will place results
          product_path: .

          # Directory where SAS writes temporary data
          scratch_path: /data/OPERA/scratch

          # If option `save_bursts` is set, output bursts are saved to:
          #     {output_dir}/{burst_id}/{product_id}{suffix}.{ext}
          # If option `save_mosaics` is set, output mosaics are saved to:
          #     {output_dir}/{product_id}{suffix}.{ext}
          # If the field `product_id`` is left empty, the prefix "rtc_product"
          # will be used instead.
          # `suffix` is only used when there are multiple output files.
          # `ext` is determined by geocoding_options.output_imagery_format.
          output_dir: /data/OPERA/outdir/S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE
          product_id: OPERA_L2_RTC-S1B_IW_SLC__1SSH_20210223T233056_20210223T233124_025740_031194_E7BE

          save_bursts: True
          save_mosaics: True
          output_imagery_format: COG
          output_imagery_compression: DEFLATE
          output_imagery_nbits: 32

          # Optional. Save secondary layers (e.g., inc. angle) within 
          # the HDF5 file
          save_secondary_layers_as_hdf5: True

          # Save RTC-S1 metadata in the HDF5 format
          # Optional for `output_imagery_format` equal to 'ENVI', 'GTiff', or
          # 'COG', and enabled by default for `output_imagery_format` equal
          # to 'HDF5' or 'NETCDF' or `save_secondary_layers_as_hdf5` is True
          save_metadata: True

      processing:

          # Check if ancillary input cover entirely output products
          check_ancillary_inputs_coverage: False

          polarization: co-pol

          # Options to run geo2rdr
          geo2rdr:
              threshold: 1.0e-8
              numiter: 25

          # Options to run rdr2geo
          rdr2geo:
              threshold: 1.0e-7
              numiter: 25

          # Apply absolute radiometric correction
          apply_absolute_radiometric_correction: True
 
          # Apply thermal noise correction
          apply_thermal_noise_correction: True

          # OPTIONAL - Apply RTC
          apply_rtc: True

          # Apply bistatic delay correction
          apply_bistatic_delay_correction: False

          # Apply static tropospheric delay correction
          apply_static_tropospheric_delay_correction: False

          # DEM interpolation method
          # Choices - 'sinc', 'bilinear', 'bicubic', 'nearest', 'biquintic'
          dem_interpolation_method: bilinear

          # OPTIONAL - to control behavior of RTC module
          # (only applicable if geocode.apply_rtc is True)
          rtc:
              # OPTIONAL - Choices:
              # "gamma0" (default)
              # "sigma0"
              output_type: gamma0

              # OPTIONAL - Choices:
              # "bilinear_distribution" (default)
              # "area_projection"
              algorithm_type: area_projection

              # OPTIONAL - Choices:
              # "beta0" (default)
              # "sigma0"
              input_terrain_radiometry: beta0

              # OPTIONAL - Minimum RTC area factor in dB
              rtc_min_value_db: -30

              # RTC DEM upsampling
              dem_upsampling: 2

          # OPTIONAL - Mechanism to specify output posting and DEM
          geocoding:

              # OPTIONAL - Apply valid-samples sub-swath masking
              apply_valid_samples_sub_swath_masking: True

              # OPTIONAL - Apply shadow masking
              apply_shadow_masking: True

              # OPTIONAL -
              algorithm_type: area_projection

              # OPTIONAL - Choices: "single_block", "geogrid", "geogrid_radargrid", and "auto" (default)
              memory_mode: auto

              # OPTIONAL - Processing upsampling factor applied to input geogrid
              geogrid_upsampling: 2
               
              # Save the incidence angle
              save_incidence_angle: False
               
              # Save the local-incidence angle
              save_local_inc_angle: True

              # Save the projection angle
              save_projection_angle: False

              # Save the RTC ANF compuated with the projection angle method
              save_rtc_anf_projection_angle: True

              # Save the range slope angle
              save_range_slope: False

              # Save the number of looks used to compute RTC-S1
              save_nlooks: True
               
              # Save the RTC area normalization factor (ANF) used to generate
              # the RTC product
              save_rtc_anf: True

              # Save the RTC area normalization factor (ANF) gamma0 to sigma0
              save_rtc_anf_gamma0_to_sigma0: True

              # Save interpolated DEM used to compute RTC-S1
              save_dem: True

              # Save layover shadow mask
              save_mask: True

              # OPTIONAL - Absolute radiometric correction
              abs_rad_cal: 1

              # OPTIONAL - Clip values above threshold
              clip_max:

              # OPTIONAL - Clip values below threshold
              clip_min:

              # OPTIONAL - Double sampling of the radar-grid
              # input sampling in the range direction
              upsample_radargrid: False

              # Fields to populate the products' metadata required by
              # CEOS Analysis Ready Data specifications
              estimated_geometric_accuracy_bias_x:
              estimated_geometric_accuracy_bias_y:
              estimated_geometric_accuracy_stddev_x:
              estimated_geometric_accuracy_stddev_y:

              bursts_geogrid:
                  output_epsg: 3031
                  x_posting: 20
                  y_posting: 20
                  x_snap: 20
                  y_snap: 20
                  top_left:
                      x:
                      y:
                  bottom_right:
                      x:
                      y:


          mosaicking:
              # OPTIONAL - Choices: "average", "first", "bursts_center" (default)
              mosaic_mode: first
              mosaic_geogrid: 
                  output_epsg: 3031
                  x_posting: 20
                  y_posting: 20
                  x_snap: 20
                  y_snap: 20
                  top_left:
                      x:
                      y:
                  bottom_right:
                      x:
                      y:

Reproducible steps

- Download SLC and associated metadata to necessary folders 
- Cop 30m DEM downloaded with - https://pypi.org/project/dem-stitcher/ and padded with no data if the scene extent wasn't covered. The DEM can be found here - https://drive.google.com/file/d/19Z-vbNhoGelBQ_uDwmer0ADYKXG4B50F/view
- docker run rtc_s1.py {path_to_above_config}.yaml

Environment

- docker image - opera/rtc:final_1.0.1
- fedora linux
- aws t2.xlarge instance (4xCPU, 8GB Ram)

[StdErr] Traceback (most recent call last):
[StdErr]   File "/home/rtc_user/miniconda3/envs/RTC/bin/rtc_s1.py", line 41, in <module>
[StdErr]     main()
[StdErr]   File "/home/rtc_user/miniconda3/envs/RTC/bin/rtc_s1.py", line 36, in main
[StdErr]     run_parallel(cfg, path_logfile_parent, flag_full_log_formatting)
[StdErr]   File "/home/rtc_user/miniconda3/envs/RTC/lib/python3.9/site-packages/rtc/rtc_s1.py", line 627, in run_parallel
[StdErr]     os.rmdir(output_dir)
[StdErr] PermissionError: [Errno 13] Permission denied: '/home/rtc_user/output_dir'

When running the SAS in the SDS, the output directory is always mounted from the file system outside of the container, so the SAS should never try to delete this directory (deleting the contents within the output directory is OK though)

What did you expect?

Please confirm if the "DSI or ACC record missing" error is due to a defect in the input SLC granule, or if this can be fixed within the RTC code. The RTC code should also be updated to remove this block of code, but this is less critical.

Reproducible steps

Inputs and a sample RunConfig can be found in the following S3 location: s3://opera-dev-triage-fwd-collinss/triaged_job-SCIFLO_L2_RTC_S1__issue_775_disp_s1_v3.0.0-rc.2.0_integration-S1A_IW_SLC__1SDV_20230114T161231_20230114T161258_046782_059BD6_D15F-r2-20240403T165711.673086Z_task-dbe0456c-a6ec-432e-9651-d94a076386e0/

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...

What did you expect?

I expect overview resampling to take into account NaN values such that they are averaged out at lower resolution levels. Also factor-of-2 zoom levels could be more useful given the starting resolution as the smaller zoom pyramids are extremely tiny.

Seems this could be the place for such changes?

RTC/src/rtc/core.py

Line 59 in 70d4216

def save_as_cog(filename, scratch_dir='.', logger=None,

Reproducible steps

import xarray as xr #2023.2.0
import matplotlib.pyplot as plt
import os 

# Uses ~/.netrc with nasa credentials
os.environ['GDAL_DISABLE_READDIR_ON_OPEN']='EMPTY_DIR'
os.environ['GDAL_HTTP_COOKIEJAR']='/tmp/cookies.txt'
os.environ['GDAL_HTTP_COOKIEFILE']='/tmp/cookies.txt'

rtcFile = 'OPERA_L2_RTC-S1_T013-026571-IW1_20231112T142222Z_20231113T010035Z_S1A_30_v1.0_VH.tif'
url = f'https://datapool.asf.alaska.edu/RTC/OPERA-S1/{rtcFile}'


fig,axes = plt.subplots(1,5,
                        figsize=(16,6), 
                        sharex=True, sharey=True, 
                       )

for i in range(-1,4):
    ax=axes[i+1]
    da = xr.open_dataarray(url, engine='rasterio',
                           open_kwargs={"overview_level": i}
                          ).squeeze()
    
    da.isnull().plot.imshow(ax=ax, add_colorbar=False)
    ax.set_title(f'overview_level={i}\n{da.shape}')
    ax.set_aspect(1)

Environment

https://opensciencelab.asf.alaska.edu

[New Feature]: Suggestions for RTC-S1 metadata

Checked for duplicates

Yes - I've already checked

Alternatives considered

Yes - and alternatives don't suffice

Describe the feature request

Based on the main branch RTC/h5_prep.py

platform_id and sensor in the identification group contain the same metadata value i.e., platform_id
acquisitionMode seems to be hard-coded to IW. Sentinel supports also EW which can be used to generate RTC products.
listOfFrequencies seems to be a NISAR heritage. Sentinel will not have multiple frequency bands. Consider removing it.
isUrgentObservation no urgent observation foreseen for Sentinel or by the project
diagnosticModeFlag not pertinent to Sentinel
softwareVersion should not be contained in identification but in processing_information. Unclear which software correspond to (RTC? ISCE3?)
ISCEVersion should be ISCE3Version
Include RTC software version in processingInformation/algorithms
configFiles will not be populated with anything meaningful. Most likely PCM will archive the runconfigs and this metadata will not contain the location of the archived runconfig. Consider replacing it with the content of the runconfiguration file
beamID May be subswath?

For CSLC-S1 we decided to use

snake case (RTC is using camel case)
Remove /science/SENTINEL1
Rename the grids group to data
Remove frequencyA nomenclature

opera-adt / rtc Goto Github PK

rtc's Introduction

RTC

Install

Usage

License

rtc's People

Contributors

Stargazers

Watchers

Forkers

rtc's Issues

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

On DOCKER

Manual Installation

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Describe the bug

What did you expect?

Reproducible steps

Environment

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Describe the bug

What did you expect?

Reproducible steps

Environment

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Checked for duplicates

Describe the bug

What did you expect?

Reproducible steps

Environment

Checked for duplicates

Alternatives considered

Related problems

Describe the feature request

Recommend Projects

Recommend Topics

Recommend Org

Jobs