mapbox / rio-cloudmask Goto Github PK
View Code? Open in Web Editor NEWRasterio plugin for identifying clouds in multi-spectral satellite imagery
License: MIT License
Rasterio plugin for identifying clouds in multi-spectral satellite imagery
License: MIT License
We should test each function in equations.py
using hypothesis to suss out any edge cases.
The output of rio-cloudmask is somewhat more opinionated than the original algorithm in that we mush boil the entire analysis down to a binary mask. Currently we just ignore the potential snow layer. Should we do something with it?
From the cloudmask
function:
# remove cloud shadow outliers
pcsl = minimum_filter(pcsl, size=(5, 5))
# grow around the edges
pcl = maximum_filter(pcl, size=(21, 21))
pcsl = maximum_filter(pcsl, size=(13, 13))
These should be configurable from the CLI and the cloudmask function. Naming TBD
Hi @perrygeo,
I love all the work you've done and put on github, it has taught me a lot and made my life of working with landsat data easier!
I'm putting your implementation of fmask equations in a class and noticed a possible misuse of numpy.fmax:
def variability_prob(ndvi, ndsi, whiteness):
return 1.0 - np.fmax(np.absolute(ndvi), np.absolute(ndsi), whiteness)
where fmax should only evaluate two arrays at a time, e.g. in the following code indices in which c is greatest are ignored:
>>> a
array([[177, 44, 99],
[198, 108, 206],
[180, 10, 35]])
>>> b
array([[101, 64, 53],
[ 5, 169, 82],
[206, 111, 252]])
>>> c
array([[149, 87, 36],
[210, 111, 83],
[189, 208, 106]])
>>> np.fmax(a, b, c)
array([[177, 64, 99],
[198, 169, 206],
[206, 111, 252]])
This is fixed with
ndi_max = np.fmax(np.absolute(self.ndvi), np.absolute(self.ndsi))
f_max = 1.0 - np.fmax(ndi_max, whiteness)
return f_max
I'm planning on continued work with fmask. If you are open to PRs, I might work on equations.py (and tests) and seperate the functions from my fmask class.
Cheers
Great initiative! If you were not already aware of this, you may want to checkout pythonfmask: http://pythonfmask.org and sourcecode @ https://bitbucket.org/chchrsc/python-fmask
I have been using it for Sentinel-2 imagery since quite recently. The code is running well, but I have not spent alot of time checking the results yet...
Hello,
I found a small typo in the algorithm.
According to the paper, it is band 5 that is used for brightness test, i.e. swir1 and not nir as coded in rio-cloudmask. This messes a lot of things over water.
Many thanks for the clean code style!
Thomas
There are a few global operations (scene-wide quantiles, others?) that preclude a simplistic block iteration approach. But we should be able to trade a bit of speed, maybe doing two passes, in order to reduce the memory footprint which is upwards of 5+ GB. We should profile memory usage and think of some ways to reduce it.
When the TIRS band 10 is all-zero, as is the case with e.g. LC80460282016097LGN00, the equations 7&8 are filled with nans
/Users/mperry/env/mapbox35/lib/python3.5/site-packages/numpy/lib/nanfunctions.py:1001: RuntimeWarning: All-NaN slice encountered
warnings.warn("All-NaN slice encountered", RuntimeWarning)
Which leads to the masking of the entire scene, as if there were 100% clouds.
The fmask algorithm is capable of dealing with the absence of a thermal band. We just need to
Proximate failure: the temp_land
function returns a single nan
which prevents the tlow, thigh = temp_land(pcps, water, tirs1)
from unpacking (expects two variables)
But the bigger question is, what should the algorithm do in this case? It's not well defined in the publication so I'll have to dig into the implemetation.
Allowing us to compare results to other vetted approaches and assess the impact of changes.
@perrygeo I prefer your clean framework here a million times over the prohibiting interfaces of the other implementations of FMask. ๐
But are you planning on adding some of the missing features / want contributions in that direction? You mentioned some under https://github.com/mapbox/rio-cloudmask#status
In particular, I need cloud shadow detection and full Sentinel 2 support.
I can offer some, but maybe not all of the work. So if this is high enough on your agenda, we could maybe do this together rather soon and perhaps also get some more current users of python-fmask
interested...
What do you say?
The naive cloudshadow approach in #3 is better than nothing but barely acceptable. Let's study the full-blown cloud/shadow matching and implement it here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.