dshean / demcoreg

Utilities for DEM and point cloud co-registration

demcoreg

Python and shell scripts for co-registration of rasters, specifically horizontal and vertical alignment of digital elevation models (DEMs).

Overview

All DEMs have some horizontal and vertical geolocation error. It is important to remove relative offsets when differencing DEMs for elevation change analyses. These tools offer several options to solve this problem. Most solve for the sub-pixel horizontal shift and vertical offset required to minimize errors over "static" control surfaces. The ASP pc_align tool can also solve for more complex transformations with rotations and scaling.

Features

• Multiple co-registration algorithms:
  • Automated, iterative implementation of the algorithm outlined by Nuth and Kaab (2011)
  • Wrappers around the Ames Stereo Pipeline Iterative Closest Point (ICP) pc_align utility (https://stereopipeline.readthedocs.io/en/latest/tools/pc_align.html)
  • Normalized cross-correlation (NCC) with sub-pixel refinment
  • Sum of absolute differences (SAD)
• Command-line utilities for raster differencing with necessary resampling (compute_diff.py), raster stats (robust_stats.py), and raster sampling at point locations (sample_raster_at_pts.py)
• Mask preparation and automatic determination of static control surfaces (i.e., exposed bedrock) for a user-specified combination of:
  • RGI glacier polygons
  • Land-use/Land-cover classification:
    • National Land Cover Database (NLCD) 30-m products for CONUS with pre-configured filter combinations (e.g., 'not_forest+not_water')
    • Global Bare Ground 30-m products
  • Snow mask:
    • Thresholded MODSCAG fSCA from ~2-week period around DEM timestamp
    • Thresholded SNODAS model
    • Thresholded Top-of-atmosphere reflectance values from corresponding orthoimage (requires pregeneration)

Some useful command-line utilities (run with -h option for complete usage)

• dem_align.py - robust raster DEM co-registration (e.g., Nuth and Kaab [2011]) for surfaces with variable slope and aspect (e.g., mountains)
• dem_mask.py - pre-generate mask to identify "stable" surfaces to use during co-registration
• pc_align_wrapper.sh - wrapper around NASA Ames Stereo Pipeline pc_align utility for iterative closest point co-registration
• apply_dem_translation.py - update raster geotransform and remove vertical offset
• compute_diff.py - simple DEM difference calculation with intuitive resampling options
• robust_stats.py - print out robust raster statistics (e.g,. for DEM difference map before/after co-registration)

Sample output

dem_align.py

Sample command: dem_align.py ref_dem.tif src_dem.tif

dem_mask.py

Sample command: dem_mask.py --toa --bareground --glaciers src_dem.tif

filter_glas.py output

Example applications

High-mountain Asia

• Co-registration of ~35000 high-resolution DEMs from multiple sensors
• https://github.com/dshean/hma_mb_paper

Pine Island Glacier, Antarctica

• Least-squares optimization to correct for offset and "tilt" of ~800 high-resolution DEMs with limited ground control
• https://github.com/dshean/pig_dem_meltrate

Installation

We are hoping to clean up the code, remove unnecessary dependencies, and streamline installation using conda. For now, we recommend following the "Building from Latest Source" instructions below, to obtain latest features/bugfixes.

If unfamiliar with this process, or if you are new to Python, bash, and/or git/github, start with these more detailed instructions and notes: Beginner's guide for installation and basic usage

Building from Latest Source (recommended)

1. Assuming you have working Python3 install with GDAL and NumPy, install pygeotools
2. Clone the demcoreg repository: git clone https://github.com/dshean/demcoreg.git
3. Perform developer install with pip: pip install -e demcoreg
   • The -e flag ("editable mode", setuptools "develop mode") will allow you to modify source code and immediately see changes. Useful if you need to make minor tweaks or bugfixes (please submit a Pull Request!)
4. Optionally, append the demcoreg subdirectory containing scripts to your PATH: export PATH=${PATH}:$PWD/demcoreg/demcoreg (replacing $PWD with the absolute path to the cloned demcoreg repository)
   • To make this permanent, add that line to your shell config file (e.g., ~/.bashrc).

Simple install with PyPI

pip install demcoreg

Documentation

http://demcoreg.readthedocs.io (autogenerated from source code, may be out of date)

License

This project is licensed under the terms of the MIT License.

Citation

If you use any of this software for research applications that result in publications, please cite:
Shean, D. E., O. Alexandrov, Z. Moratto, B. E. Smith, I. R. Joughin, C. C. Porter, Morin, P. J., An automated, open-source pipeline for mass production of digital elevation models (DEMs) from very high-resolution commercial stereo satellite imagery, ISPRS J. Photogramm. Remote Sens, 116, 101-117, doi: 10.1016/j.isprsjprs.2016.03.012, 2016.

demcoreg's Issues

dem_align tiltcorr error

Hi, I am running into an issue when I attempt to use the tiltcorr function when aligning dems. Everything works great running the basic command without the addition of tiltcorr but when I add it I get an error message (copied below). Not sure what this is trying to tell me and was hoping someone here could point me in the right direction!

Thanks, Jackson

---

Calculating 'tiltcorr' 2D polynomial fit to residuals with order 1

---

Excluding values outside of percentile range: 0.00 to 100.00
Excluding values outside of range: -19.771729 to 19.140015
[-4.48856504e-11 4.17859625e-06 -3.66286135e-05 2.25317262e-11]
Z shift offset array mean: 0.34901226247618106
Creating plot of polynomial fit to residuals
Traceback (most recent call last):
File "/home/jacksonbodtker/src/demcoreg/demcoreg/dem_align.py", line 603, in
main()
File "/home/jacksonbodtker/src/demcoreg/demcoreg/dem_align.py", line 400, in main
ax = pltlib.iv(valgrid, ax=axa[1], cmap='RdBu', clim=dz_clim,
File "/home/jacksonbodtker/src/imview/imview/lib/pltlib.py", line 100, in iv
res = geolib.get_res(ds, c_srs)[0]
File "/home/jacksonbodtker/src/pygeotools/pygeotools/lib/geolib.py", line 1256, in get_res
extent = ds_extent(ds, t_srs)
File "/home/jacksonbodtker/src/pygeotools/pygeotools/lib/geolib.py", line 1384, in ds_extent
ul = ct.TransformPoint(*ul)
File "/home/jacksonbodtker/miniconda3/envs/demcoreg_env1/lib/python3.10/site-packages/osgeo/osr.py", line 1150, in TransformPoint
return _osr.CoordinateTransformation_TransformPoint(self, *args)
NotImplementedError: Wrong number or type of arguments for overloaded function 'CoordinateTransformation_TransformPoint'.
Possible C/C++ prototypes are:
OSRCoordinateTransformationShadow::TransformPoint(double [3])
OSRCoordinateTransformationShadow::TransformPoint(double [4])
OSRCoordinateTransformationShadow::TransformPoint(double [3],double,double,double)
OSRCoordinateTransformationShadow::TransformPoint(double [4],double,double,double,double)

Skysat_stereo Problem

Hello,

I'm trying to run the skysat_stereo scripts.
The installation seems fine, but when I run ./skysat_triplet_pipeline.py I have this error:

Traceback (most recent call last):
File "./skysat_triplet_pipeline.py", line 5, in
from rpcm import geo
File "/home/ganci/.local/lib/python3.8/site-packages/rpcm/init.py", line 8, in
import srtm4
File "/home/ganci/.local/lib/python3.8/site-packages/srtm4/init.py", line 7, in
from srtm4.raster import crop
File "/home/ganci/.local/lib/python3.8/site-packages/srtm4/raster.py", line 15, in
pyproj.network.set_network_enabled(active=True)

I don't know how to solve this problem.
Can you please help me with this issue?
Many thanks

Clean up mask functionality

Overhaul needed after many incremental modifications. Update dem_mask.py and dem_align.py, perhaps move some shared functionality to new library.

The dem_align static_mask code is a mess, flags are confusing. There were several attempts for shortcuts to use existing mask rather than recomputing. This makes the code confusing and is leading to bugs somewhere (the "Before" histogram still has a spike for the 0 bin).

Should also set default to us no masking, so the user doesn't have to deal with downloading all of the glacier, LULC datasets, etc.

Missing requirement: demtools/stats.py

Hello, I'm trying to use the dem_coreg.sh script, and the execution fails on line 94 with the error

/home/USER/src/demtools/stats.py: No such file or directory

I could not find in the README or documentation which demtools should be used to complete the functionality.

Thanks for any information.

Error during gdal reproject: "Object given is not a Python function"

Running into a problem when GDAL is reprojecting my DEMs. Here is the code up to the error

`Reference: 2021_10_03_Discovery_Mine_dsm2.tif
Source: Discovery_Mine_dsm2.tif
Mode: nuth
Output: Discovery_Mine_dsm2_dem_align\Discovery_Mine_dsm2_2021_10_03_Discovery_Mine_dsm2

Warping all inputs to the following:
Resolution: 0.031370000000001015
Extent: [353337.1446387405, 7008245.025648932, 354702.3585187405, 7010228.931808715]
Projection: '+proj=utm +zone=12 +datum=WGS84 +units=m +no_defs'
Resampling alg: cubic

1 of 2: 2021_10_03_Discovery_Mine_dsm2.tif
nl: 63242 ns: 43520 res: 0.031

Traceback (most recent call last):
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\dem_align.py", line 605, in
main()
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\dem_align.py", line 273, in main
ref_dem_ds, src_dem_ds_align = warplib.memwarp_multi([ref_dem_ds, src_dem_ds_align],
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\venv\lib\site-packages\pygeotools\lib\warplib.py", line 520, in memwarp_multi
return warp_multi(src_ds_list, res, extent, t_srs, r, warptype=memwarp, verbose=verbose, dst_ndv=dst_ndv)
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\venv\lib\site-packages\pygeotools\lib\warplib.py", line 512, in warp_multi
dst_ds = warptype(ds, res, extent, t_srs, r, outdir, dst_ndv=dst_ndv, verbose=verbose)
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\venv\lib\site-packages\pygeotools\lib\warplib.py", line 216, in memwarp
return warp(src_ds, res, extent, t_srs, r, driver=driver, dst_ndv=dst_ndv, verbose=verbose)
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\venv\lib\site-packages\pygeotools\lib\warplib.py", line 192, in warp
gdal.ReprojectImage(src_ds, dst_ds, src_srs.ExportToWkt(), t_srs.ExportToWkt(), gra, 0.0, 0.0, prog_func)
File "E:\01_Projects\2022\22-17-NN Discovery Mine\07_GIS\dist\demcoreg-0.5.0\demcoreg\venv\lib\site-packages\osgeo\gdal.py", line 3914, in ReprojectImage
return _gdal.ReprojectImage(*args, **kwargs)
RuntimeError: Object given is not a Python function
`

I tried reprojecting into NAD83(CSRS) UTM zone 12 but ran into the same problem. Both rasters have the same resolution and mostly overlap, one is a few m wider than the other.

Any help is greatly appreciated

Two different reference surfaces

Hi

I want to co-register two different digital elevation models

1): Lidar dem (vertical datum: geoid EGM96)
2): stereoscopic Pléiades dem (vertical datum: ellipsoid WGS84)

Do i have to apply a vertical datum transformation first before using dem_align.py?

issue installing on MacOS M1

hi, anyone else had an issue installing on MacOS M1? I had no issues installing on the Intel OS, now on M1 I get and error when following the installation (with pip),

pip install demcoreg

which says "Failed building wheels for gdal"

then
× Encountered error while trying to install package.
╰─> gdal

I checked, wheel package is installed and I am using this in a separate environment (not the base).
thanks-

Memory stacking up when looping the dem_align main function

Hi,

I noticed when making a loop to run the dem_align.main() on many files the memory usage keeps increasing.

It seems to be due to the plots that are never closed.
Adding a matplotlib.pyplot.close('all') at the end of the main function does the trick.

Might be worth looking into to close each plot as soon as they are not needed to improve the memory usage even when running it only once as the plots seems to be quite memory intensive.

RuntimeError: Object given is not a Python function

When I run dem_align.py, the following error occurs. Is there anyone who knows how to solve it

Reference: ref_dem.tif
Source: src_dem.tif
Mode: nuth
Output: src_dem_dem_align\src_dem_ref_dem

Warping all inputs to the following:
Resolution: 0.0002777777777777829
Extent: [75.89041678772851, 35.88458367232465, 76.36625000000001, 36.28958333333334]
Projection: '+proj=longlat +datum=WGS84 +no_defs'
Resampling alg: cubic

1 of 2: ref_dem.tif
nl: 1458 ns: 1713 res: 0.000
Traceback (most recent call last):
File "D:\iceprogram\demcoreg-master\demcoreg-master\d\dem_align.py", line 608, in
main()
File "D:\iceprogram\demcoreg-master\demcoreg-master\d\dem_align.py", line 271, in main
ref_dem_ds, src_dem_ds_align = warplib.memwarp_multi([ref_dem_ds, src_dem_ds_align],
File "C:\Anaconda3\lib\site-packages\pygeotools\lib\warplib.py", line 520, in memwarp_multi
return warp_multi(src_ds_list, res, extent, t_srs, r, warptype=memwarp, verbose=verbose, dst_ndv=dst_ndv)
File "C:\Anaconda3\lib\site-packages\pygeotools\lib\warplib.py", line 512, in warp_multi
dst_ds = warptype(ds, res, extent, t_srs, r, outdir, dst_ndv=dst_ndv, verbose=verbose)
File "C:\Anaconda3\lib\site-packages\pygeotools\lib\warplib.py", line 216, in memwarp
return warp(src_ds, res, extent, t_srs, r, driver=driver, dst_ndv=dst_ndv, verbose=verbose)
File "C:\Anaconda3\lib\site-packages\pygeotools\lib\warplib.py", line 192, in warp
gdal.ReprojectImage(src_ds, dst_ds, src_srs.ExportToWkt(), t_srs.ExportToWkt(), gra, 0.0, 0.0, prog_func)
File "C:\Anaconda3\lib\site-packages\osgeo\gdal.py", line 3915, in ReprojectImage
return _gdal.ReprojectImage(*args, **kwargs)
RuntimeError: Object given is not a Python function

Process finished with exit code 1

Input images do not intersect

Hello!
I try to use dem_align.py co-register a SRTM DEM and a COPDEM (more details about COPDEM) by typing python dem_align.py dem1_path dem2_path. I've conformed that the two DEM is intersected (the SRTM DEM is larger than COPDEM, so COPDEM is 100% intersected with SRTM DEM).

At the beginning, there are six consecutive errors:

proj_create: Error -38: failed to load datum shift file                                                  
proj_create: Error -38: failed to load datum shift file                                                  
proj_create: Error -38: failed to load datum shift file                                                  
proj_create: Error -38: failed to load datum shift file                                                  
proj_create: Error -38: failed to load datum shift file                                                  
proj_create: Error -38: failed to load datum shift file

But the script finished the first iteration and saved a "Nuth and Kaab plot" in the disk.

While at the beginning of the second iteration, a Input images do not intersect appeared and the script is over.

Did I miss any thing?

---

I insalled imview, pygeotools, and demcoreg by using git and pip

My demcoreg lib version is 0.5.0, and pygeotools lib version is 0.6.0, and imview lib version is 0.3.0

My python version is 3.7.6

My operating system is Ubuntu 20.04

Programme being killed after 1 interaction

Hi, I keep getting this issue after 1 Interaction:

~/src/demcoreg/demcoreg/dem_align.py -max_offset 250 28-06-2012_E_BI.tif 22-04-2016_E_BI.tif

Reference: 28-06-2012_E_BI.tif
Source: 22-04-2016_E_BI.tif
Mode: nuth
Output: 22-04-2016_E_BI_dem_align/22-04-2016_E_BI_28-06-2012_E_BI

Warping all inputs to the following:
Resolution: 2.0
Extent: [-1970264.0, -517090.0, -1899022.0, -482764.0]
Projection: '+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs'
Resampling alg: cubic

1 of 2: 28-06-2012_E_BI.tif
nl: 17163 ns: 35621 res: 2.000
0...10...20...30...40...50...60...70...80...90...100 - done.
2 of 2: [memory]

Reference DEM res: 2.00
Source DEM res: 2.00
Resolution for coreg: mean (2.00 m)

*** Iteration 1 ***

Warping all inputs to the following:
Resolution: 2.0
Extent: [-1970264.0, -517090.0, -1899022.0, -482764.0]
Projection: '+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs'
Resampling alg: cubic

1 of 2: 28-06-2012_E_BI.tif
2 of 2: [memory]
Killed

I'm using a Ubuntu Virtual Machine with minicoda 3

beginners_doc - get_rgi.sh fails if libraries are not added to PATH

I followed the beginners_doc, but did not add paths to the three libraries (pygeotools, demcoreg, imview) to my PATH env var.

Upon trying to run dem_align.py, I was advised to run the get_rgi.sh script.

Missing rgi glacier data source. If already downloaded, specify correct datadir. If not, run `get_rgi.sh` to download

I ran the script and, after the large download completed, a "command not found" error then an OGR error was thrown.

Merging region shp for global shp
.../demcoreg/demcoreg/get_rgi.sh: line 80: ogr_merge.sh: command not found

Clipping to site: CONUS
POLYGON ((-125 49, -104 49, -104 32, -125 32, -125 49))
+proj=aea +lat_1=36 +lat_2=49 +lat_0=43 +lon_0=-115 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs
FAILURE:
Unable to open datasource `rgi60_merge.shp' with the following drivers.
  -> `PCIDSK'
  -> `netCDF'
  -> `JP2OpenJPEG'
  ......<long list of drivers>

The script is failing because it calls pygeotools' ogr_merge.sh script on line 80. If you have not added pygeotools to PATH, those files are not created, and we also get the red herring OGR error

Unable to open datasource `rgi60_merge.shp' with the following drivers.

because that file does not exist.

I resolved the issue by adding the paths to the three libraries to my PATH env var. Maybe the beginner docs could be updated to say that adding the libraries to your path is a requirement or perhaps get_rgi.sh can be modified as to not require that. No real bug here but I think walking through the beginner docs would be smoother if this was mentioned.

Update water/shoreline masking and integrate in dem_mask

I started this process with the HYDROBasins products for lakes/rivers (https://github.com/dshean/demcoreg/blob/master/demcoreg/get_hydro.sh) but never actually added support in the dem_mask functionality. Should be pretty straightforward - can use the RGI polygon handling as a template, or better yet, update with latest GDAL/rasterio rasterization approaches.

Should also add global shoreline vector data handling: https://shoreline.noaa.gov/

Upon quick inspection, the global NGA product might be good (https://dnc.nga.mil/PrototypeGSD.php). Though if we can find a shoreline dataset that can be pulled in on the fly, that would be better than another set of shp.zip that we download and combine...

Ultimately, we would like to have better support for classification of corresponding orthoimages (if available for the DEM, i.e. from stereo satellite imagery), mapping actual distribution of water, veg, snow/ice, but hard to have a general solution for this when supporting DEMs from many sources.

Add checks for ref DEM, require projected coordinate system

Continuing to receive emails from users who are using input DEMs in EPSG:4326.

Simple solution is to exit with message about reprojecting with gdalwarp.

Maybe add a -t_srs option to allow the user to specify desired projection, or select local orthographic based on input DEM intersection, and convert the translation vector appropriately back to EPSG:4326.

Support for user-provided vector mask

At least add some notes on how to do this to FAQ/doc.

Can rasterize polygon shp (see gdal_rasterize or other tutorials), and then create a masked version of reference dem using the pygeotools apply_mask.py function, preserving the "static/stable" pixels (see invert option if necessary). Then provide the masked reference DEM to dem_align.py

Review code that does the masking using the RGI polygons. Could accept vector fiilename there, but likely issues with BYO files, missing projection info, etc.

Probably best to review latest material on rasterio/geopandas for these operations, and to start updating this functionality in demcoreg tools.

Dem_mask and dem_align glaciers mask not working.

Hi there,

I have been trying to use the glacier mask to run the DEM co-registration. When I run it with mask_list in dem_align it doesn't register that I put "glaciers". When I run it in dem_mask I can see that the rgi data set is not being access, I fixed the datadir issue but the MD5 check sum is failing. So the merging and the rest of the get_rgi.sh functionality is lost.

The DM5 checksum that can be downloaded from the GLIMS site is 9ddfdfe9e65c9550bdb8c5c98c4e1aff 00_rgi60.zip

The one being used in the get_rgi.sh code is
Check MD5 sum
#95bd0486301073bbbe26988303fdaa1d 00_rgi60.zip
orig_zip_md5=95bd0486301073bbbe26988303fdaa1d

I get this error when I run get_rgi.sh:
MD5 checksum failed for rgi60.zip:
Expected: 95bd0486301073bbbe26988303fdaa1d
Downloaded: 9ddfdfe9e65c9550bdb8c5c98c4e1aff

Is this what is causing the issues with running the glaciers mask? How do I fix this?

Thanks in advance.

Improved installation: create conda environment.yml

Can add pip install steps, which should simplify setup.

Should also review and remove unnecessary dependencies. Likely requires a full review of pygeotools as well. Not sure we need imview dependency for demcoreg.

Not creating an out - FIXED

Hi

When I'm using dem_align.py in a Unbuntu virtual machine it not creating an out, but creates the folder where it would be stored.

Mode: nuth
Output: Dem_1_dem_align/Dem_1_Dem_2

Warping all inputs to the following:
Resolution: 2.0
Extent: [-1959344.0, -505326.0, -1902156.0, -486468.0]
Projection: '+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs'
Resampling alg: cubic

1 of 2: Dem_2.tif
nl: 9429 ns: 28594 res: 2.000
0...10...20...30...40...50...60...70...80...90...100 - done.
2 of 2: [memory]
nl: 9429 ns: 28594 res: 2.000
0...10...20...30...40...50...60...70...80...90...100 - done.

Reference DEM res: 2.00
Source DEM res: 2.00
Resolution for coreg: mean (2.00 m)

*** Iteration 1 ***

Warping all inputs to the following:
Resolution: 2.0
Extent: [-1959344.0, -505326.0, -1902156.0, -486468.0]
Projection: '+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs'
Resampling alg: cubic

1 of 2: Dem_2.tif
2 of 2: [memory]
Elevation difference stats for uncorrected input DEMs (src - ref)
Removing outliers
Initial pixel count:
85109616
Absolute dz filter: 100.00
Killed

which version of python?

I would like to ask which version of python you are using? Does it apply to Python3？
Besides,the 198 line of the dem_align.py Program（dem1_ds = warplib.memwarp_multi_fn([dem1_fn,], res=dem2_ds, extent=dem2_ds, t_srs=dem2_ds)[0]）
Should it be written like this:
dem1_ds = warplib.memwarp_multi_fn([dem1_fn,dem2_fn], res=dem2_ds, extent=dem2_ds, t_srs=dem2_ds)[0]

ValueError: zero-size array to reduction operation maximum which has no identity

Hi
I receive following error. The DEM is without any doubt of very bad quality but I'm interest what causes the error.

*** Iteration 1 ***

`Warping` all inputs to the following:
Resolution: 1.0
Extent: [780221.0, 2052737.0, 780622.0, 2053138.0]
Projection: '+proj=utm +zone=18 +datum=WGS84 +units=m +no_defs '
Resampling alg: cubic

1 of 2: /media/ref.tif
2 of 2: [memory]
Elevation difference stats for uncorrected input DEMs (src - ref)
Removing outliers
Initial pixel count:
107630
Absolute dz filter: 100.00
107630
Excluding values outside of range: nan +/- 3*nan
Excluding values outside of range: nan to nan
107630
Computing slope
Slope filter: 0.10 - 40.00
Initial count: 65729
Excluding values outside of range: 0.100000 to 40.000000
37142
Computing aspect
Filtered difference map
/home/pygeotools/pygeotools/lib/malib.py:1665: UserWarning: Warning: converting a masked element to nan.
  print("count: %i min: %0.2f max: %0.2f mean: %0.2f std: %0.2f med: %0.2f mad: %0.2f q1: %0.2f q2: %0.2f iqr: %0.2f mode: %0.2f p16: %0.2f p84: %0.2f spread: %0.2f" % stats)
count: 37142 min: nan max: nan mean: nan std: nan med: nan mad: nan q1: nan q2: nan iqr: nan mode: -20.60 p16: nan p84: nan spread: nan
Computing sub-pixel offset between DEMs using mode: nuth
Computing common mask
Initial sample count:
37142
Removing outliers
3-sigma filter: nan - nan
0
Computing 1-degree bin statistics: count
Computing 1-degree bin statistics: median
Traceback (most recent call last):
  File "/home/demcoreg/demcoreg/dem_align.py", line 602, in <module>
    main()
  File "/home/demcoreg/demcoreg/dem_align.py", line 289, in main
    mask_list=mask_list, max_dz=max_dz, slope_lim=slope_lim, plot=True)
  File "/home/demcoreg/demcoreg/dem_align.py", line 147, in compute_offset
    fit_param, fig = coreglib.compute_offset_nuth(diff, slope, aspect, plot=plot)
  File "/home/demcoreg/demcoreg/coreglib.py", line 297, in compute_offset_nuth
    bin_ptp = np.cos(np.radians(bin_centers)).ptp()
  File "/home/.local/lib/python3.7/site-packages/numpy/core/_methods.py", line 247, in _ptp
    umr_maximum(a, axis, None, out, keepdims),
ValueError: zero-size array to reduction operation maximum which has no identity

`

Error: Not enough dh samples - What does it mean?

Hello, i don't have experience in using Linux or Coding, but want to Match to Point Clouds. So I followed the Instructions to install demcoreg and used the following Code: (demcoreg_env) bene@Benes-PC:~/src$ ~/src/demcoreg/demcoreg/dem_align.py -slope_lim 20 80 -mode nuth 1cm_02082022_SfM.tif 1cm_02082022_TLS.tif. The following Text appears

Reference: 1cm_02082022_SfM.tif
Source: 1cm_02082022_TLS.tif
Mode: nuth
Output: 1cm_02082022_TLS_dem_align/1cm_02082022_TLS_1cm_02082022_SfM


Warping all inputs to the following:
Resolution: 0.01
Extent: [-41.639, -6.809000000000001, -11.599000000000004, 16.221]
Projection: ''
Resampling alg: cubic

1 of 2: 1cm_02082022_SfM.tif
2 of 2: [memory]

Reference DEM res: 0.01
Source DEM res: 0.01
Resolution for coreg: mean (0.01 m)

*** Iteration 1 ***

Warping all inputs to the following:
Resolution: 0.01
Extent: [-41.639, -6.809000000000001, -11.599000000000004, 16.221]
Projection: ''
Resampling alg: cubic

1 of 2: 1cm_02082022_SfM.tif
2 of 2: [memory]
Elevation difference stats for uncorrected input DEMs (src - ref)
Removing outliers
Initial pixel count:
607632
Absolute dz filter: 100.00
607632
Excluding values outside of range: 0.010 +/- 3*0.066
Excluding values outside of range: -0.188392 to 0.208914
541814
Computing slope
Slope filter: 20.00 - 80.00
Initial count: 554157
Excluding values outside of range: 20.000000 to 80.000000
0
Computing aspect
Filtered difference map
/home/bene/src/pygeotools/pygeotools/lib/malib.py:1663: UserWarning: Warning: converting a masked element to nan.
  print("count: %i min: %0.2f max: %0.2f mean: %0.2f std: %0.2f med: %0.2f mad: %0.2f q1: %0.2f q2: %0.2f iqr: %0.2f mode: %0.2f p16: %0.2f p84: %0.2f spread: %0.2f" % stats)
count: 0 min: nan max: nan mean: nan std: nan med: nan mad: nan q1: 0.00 q2: 0.00 iqr: 0.00 mode: 0.00 p16: 0.00 p84: 0.00 spread: 0.00
Computing sub-pixel offset between DEMs using mode: nuth
Not enough dh samples`

Sadly I don't know what the Error Not enough dh samples, can anybody help me what this means?
Is my Area to small or is it another problem?

Thanks

beginners-doc

Hi I followed the beginners-doc.

I keep having this error whenever I ran dem_align.py -h

problem occurring in coregistering dems

dear sir I am trying to co register SRTM dem and TanDEM-X dem for all indus basin of high mountain asia. I am using WSL Ubuntu terminal on windows 10. when I process the code it ends in crashing my system. the attached image is taken just before it crashes the system and restarts it. kindly help me how can I deal with this problem.

int64 src and float64 offset results in type error

A user reported this error when dz_offset is added to the src_dem_array: TypeError: Cannot cast ufunc add output from dtype('float64') to dtype('int64') with casting rule 'same_kind' .

• This happens when src DEM has int values and ref DEM has float values.
• I think converting src DEM with int dtype (if present) to float should be harmless here ? Let me know what you think @dshean and I can submit PR.

Unable to create glacier mask (TypeError: data type 'Bool' not understood)

Hi,
I have been trying to create a mask based on the RGI outlines. I ran "get_rgi.sh" which seems to work even if I got the message

ERROR`` 1: TopologyException: Input geom 0 is invalid: Ring Self-intersection at or near point 75.186070378000068 36.165444900000068 at 75.186070378000068 36.165444900000068`

When I run dem_mask.py --glaciers --outdir "mask" 2012_geoeye.tif I get the error message

Traceback (most recent call last): File "/mnt/c/Ben/Ben/Other/demcoreg/demcoreg/demcoreg/dem_mask.py", line 647, in <module> main() File "/mnt/c/Ben/Ben/Other/demcoreg/demcoreg/demcoreg/dem_mask.py", line 625, in main newmask = get_mask(dem_ds, mask_list, dem_fn=dem_fn, writeout=args.writeout, outdir=args.outdir, args=args) File "/mnt/c/Ben/Ben/Other/demcoreg/demcoreg/demcoreg/dem_mask.py", line 439, in get_mask icemask = get_icemask(dem_ds) File "/mnt/c/Ben/Ben/Other/demcoreg/demcoreg/demcoreg/dem_mask.py", line 107, in get_icemask icemask = geolib.shp2array(glac_shp_fn, ds) File "/mnt/c/Ben/Ben/Other/demcoreg/pygeotools/pygeotools/lib/geolib.py", line 799, in shp2array a = ~(a.astype('Bool')) TypeError: data type 'Bool' not understood

Thanks,

Ben

How can I use ICESat-2 points as GCPs to co-register a DEM?

I want to use ICESat-2 points as GCPs to coregister a DEM. Is it possible that I can do it using demcoreg tool?. if not is there any other tool that I can use for this purpose?

swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.

Help please, i have this mensaje, I don´t know how to fix.

(demcoreg) enver@lynx-pc:~$ dem_align.py -h
usage: dem_align.py [-h] [-mode {ncc,sad,nuth,none}]
[-mask_list {toa,snodas,modscag,bareground,glaciers,nlcd,none} [{toa,snodas,modscag,bareground,glaciers,nlcd,none} ...]]
[-tiltcorr] [-polyorder POLYORDER] [-tol TOL] [-max_offset MAX_OFFSET] [-max_dz MAX_DZ]
[-res {min,max,mean,common_scale_factor}] [-slope_lim SLOPE_LIM SLOPE_LIM]
[-max_iter MAX_ITER] [-outdir OUTDIR]
ref_fn src_fn

Perform DEM co-registration using multiple algorithms

positional arguments:
ref_fn Reference DEM filename
src_fn Source DEM filename to be shifted

options:
-h, --help show this help message and exit
-mode {ncc,sad,nuth,none}
Type of co-registration to use (default: nuth)
-mask_list {toa,snodas,modscag,bareground,glaciers,nlcd,none} [{toa,snodas,modscag,bareground,glaciers,nlcd,none} ...]
Define masks to use to limit reference surfaces for co-registration (default: [])
-tiltcorr After preliminary translation, fit polynomial to residual elevation offsets and remove
(default: False)
-polyorder POLYORDER Specify order of polynomial fit (default: 1)
-tol TOL When iterative translation magnitude is below this tolerance (meters), break and write
out corrected DEM (default: 0.02)
-max_offset MAX_OFFSET
Maximum expected horizontal offset in meters, used to set search range for ncc and sad
modes (default: 100)
-max_dz MAX_DZ Maximum expected vertical offset in meters, used to filter outliers (default: 100)
-res {min,max,mean,common_scale_factor}
Warp intputs to this resolution (default: mean)
-slope_lim SLOPE_LIM SLOPE_LIM
Minimum and maximum surface slope limits to consider (default: (0.1, 40))
-max_iter MAX_ITER Maximum number of iterations, if tol is not reached (default: 30)
-outdir OUTDIR Output directory (default: None)
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.
swig/python detected a memory leak of type 'OSRSpatialReferenceShadow *', no destructor found.

Create a `demcoreg_init.sh` wrapper for all of the `get_*.sh` scripts

Ideally, we would fetch all of these layers (e.g., NLCD, bareground) on the fly through services like "Earth on AWS" registry: https://aws.amazon.com/earth/

At present, they still require local download, extraction and processing.

We should give the user the option to get all demcoreg layers in one shot, or instructions on how to run the necessary get_*.sh script. Right now, when a user runs dem_align.py it starts with a bunch of downloads - no good.

Alternatively, we could include all auxiliary data in a docker image, or store ourselves in the cloud. Should discuss with @scottyhq

AttributeError: module 'numpy' has no attribute 'float'.

I am following the beginners guide to run the codes but getting the following error, even after updating numpy version.

Traceback (most recent call last):
File "/home/lpatel/src/demcoreg/demcoreg/dem_align.py", line 21, in
from imview.lib import pltlib
File "/home/lpatel/src/imview/imview/init.py", line 3, in
from .lib import *
File "/home/lpatel/src/imview/imview/lib/pltlib.py", line 19, in
cpt_rainbow = gmtColormap.get_rainbow()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lpatel/src/imview/imview/lib/gmtColormap.py", line 19, in get_rainbow
d = gmtColormap(fn, reverse=rev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/lpatel/src/imview/imview/lib/gmtColormap.py", line 74, in gmtColormap
x = np.array( x , np.float)
^^^^^^^^
File "/home/lpatel/anaconda3/envs/ev1/lib/python3.11/site-packages/numpy/init.py", line 319, in getattr
raise AttributeError(former_attrs[attr])
AttributeError: module 'numpy' has no attribute 'float'.
np.float was a deprecated alias for the builtin float. To avoid this error in existing code, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'cfloat'?

Please see

Regards
Lavkush

I can't find a program that generates "dem_align_stats.pkl".

Hello!
For the "dem_align_post.ipynb".I wonder how should I get "dem_align_stats.pkl". I could only find the program to generate "dem_align_stats_201909_rerun.pkl".

question about dem_mask.py

hello! When I run the program, an error shows that"http://edcintl.cr.usgs.gov/downloads/sciweb1/shared/gtc/downloads/bare2010.zip" can't access

error:No mould named 'Resource'

Hello, I would like to know if this program can run on Windows?
I found that some people say that this mould is on a Linux system.

Program stopped after one iteration using dem_align.py command

Hi, Thank you for sharing. I am trying to register SRTM and Tandem x, but there is an error that I cannot search for the reason.
Typed this command in the terminal: python3 dem_align.py -mode nuth ../data/SRTM.tif ../data/20131017.dem.tif
Stopped after only one iteration.

Below is the running log:

Reference: ../data/SRTM.tif
Source: ../data/20131017.dem.tif
Mode: nuth
Output: ../data/20131017.dem_dem_align/20131017.dem_SRTM

wandi@mefe54 172 <.../demcoreg/demcoreg>:python3 dem_align.py -mode nuth ../data/SRTM.tif ../data/20131017.dem.tif

Reference: ../data/SRTM.tif
Source: ../data/20131017.dem.tif
Mode: nuth
Output: ../data/20131017.dem_dem_align/20131017.dem_SRTM

Warping all inputs to the following:
Resolution: 0.0001701388891388889
Extent: [109.15996525581751, 30.199972163502004, 109.2100347006625, 30.240027719378002]
Projection: '+proj=longlat +datum=WGS84 +no_defs'
Resampling alg: cubic

1 of 2: ../data/SRTM.tif
nl: 235 ns: 294 res: 0.000
0...10...20...30...40...50...60...70...80...90...100 - done.
2 of 2: [memory]
nl: 235 ns: 294 res: 0.000
0...10...20...30...40...50...60...70...80...90...100 - done.

Reference DEM res: 0.00
Source DEM res: 0.00
Resolution for coreg: mean (0.00 m)

*** Iteration 1 ***

Warping all inputs to the following:
Resolution: 0.0001701388891388889
Extent: [109.15996525581751, 30.200045080430364, 109.20998608922434, 30.240027719378002]
Projection: '+proj=longlat +datum=WGS84 +no_defs'
Resampling alg: cubic

1 of 2: [memory]
2 of 2: [memory]
Elevation difference stats for uncorrected input DEMs (src - ref)
Removing outliers
Initial pixel count:
69090
Absolute dz filter: 100.00
69090
Excluding values outside of range: -26.283 +/- 3*4.283
Excluding values outside of range: -39.131017 to -13.435633
63174
Computing slope
Slope filter: 0.10 - 40.00
Initial count: 68036
Excluding values outside of range: 0.100000 to 40.000000
65121
Computing aspect
Filtered difference map
count: 60601 min: -39.12 max: -13.44 mean: -25.90 std: 4.35 med: -26.30 mad: 3.69 q1: -28.43 q2: -23.39 iqr: 5.04 mode: -26.51 p16: -29.81 p84: -21.66 spread: 4.07
Computing sub-pixel offset between DEMs using mode: nuth
Computing common mask
Initial sample count:
60601
Removing outliers
3-sigma filter: -780.30 - 543.09
60074
Computing 1-degree bin statistics: count
Computing 1-degree bin statistics: median
Computing fit
[-11.40113551 -53.66257759 -79.27599968]
Generating Nuth and Kaab plot
/home/wandi/lz/demcoreg/demcoreg/demcoreg/demcoreg/coreglib.py:337: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
bp = ax.boxplot(np.array(output)[idx][::s], positions=bin_centers[::s], widths=widths[::s], showfliers=False,
Median dz: -26.30
Nuth dz: -27.61
Incremental offset: dx=-9.18m, dy=+6.76m, dz=+26.30m
Cumulative offset: dx=-9.18m, dy=+6.76m, dz=+26.30m
Writing offset plot: ../data/20131017.dem_dem_align/20131017.dem_SRTM_nuth_iter01_plot.png
Segmentation fault (core dumped)

No overlapping, unmasked pixels shared between input DEMs

Hello,
I hope you can help me, I keep getting this error even tho when I plot my data on qgis is overlapping. What can I be doing wrong?
I used:
gdalwarp -r near -tr 0.00005 0.00005 -te
to clip all the files to the same extent. Is there any other file I need beside the .tif?

Error -38

I run the dem_align.py command successfully and get an output dem after four iterations. However, the following error message pops immediately after I run the dem_align.py command. It appears three times in succession. Wondering what is causing this error message, and if it will cause any problems? Both of my input dems are in a projected coordinate system. Thanks!

proj_create: Error -38: failed to load datum shift file

Fix polynomial fits to residuals

The initial dem_align.py planar fit to remove post-co-registration residuals worked well. The current n-order polynomial fit functionality is largely untested. User reports in Dec 2019 indicated this functionality is not working correctly, something that I was able to reproduce with quick tests, but didn't have the time to document and fix.

Change from geolib.ma_fitplane to geolib.ma_fitpoly:

demcoreg/demcoreg/dem_align.py

Line 374 in abd6be7

vals, resid, coeff = geolib.ma_fitpoly(diff_align_filt, order=polyorder, gt=gt, perc=(0,100), origmask=False)

This pygeotools/geolib commit updated underlying fitting routines to support arbitrary higher-order polynomials:
dshean/pygeotools@fbd7b97#diff-0c9ae3a15902d51d1f8374a5976f4d79

Notes suggest we should consider numpy.polynomial.polynomial here. I need to revisit local branches - I thought I started this, but did not merge in master.

May be an issue of when/how the resulting "tiltcorr" correction is applied. I wanted another iteration of the main Nuth and Kaab optimization after the residual correction was removed, as it's possible that the final translation would be different after removing systematic residuals.

Should also rename command line option from -tiltcorr to -remove_residuals or something along those lines.

Update beginners doc to install directly from github

4. Install pygeotools and demcoreg from latest github source:

• python -m pip install git+https://github.com/dshean/pygeotools.git
• python -m pip install git+https://github.com/dshean/demcoreg.git
• python -m pip install git+https://github.com/dshean/imview.git

Requires some reworking of PATH and ability to call scripts.