earthlab / earthpy Goto Github PK
View Code? Open in Web Editor NEWA package built to support working with spatial data using open source python
Home Page: https://earthpy.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
A package built to support working with spatial data using open source python
Home Page: https://earthpy.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
Let's follow the rasterio model and add a contributing file to earthpy. I think we can start with their exact file and can adapt/ adjust it as we go.
hey @choldgraf i moved the earthlab module over here! i had attempted to preserve the entire history but that didn't work. I may try again but let me know if you think that history is imperative!
not sure if this is a thing yet or not but...
Running setup.py bdist_wheel for earthpy ... done
Stored in directory: /private/var/folders/43/4q82487d5xsfpxdx6nl_c1wmhckx08/T/pip-ephem-wheel-cache-9jj3bwkx/wheels/3f/df/1c/6aad6ef509d5c33973e83bdcc68b02460d390b7661e2a6bb8f
Successfully built earthpy
geocoder 1.33.0 requires future, which is not installed.
contextily 0.99.0 requires pytest, which is not installed.
boto3 1.5.14 has requirement botocore<1.9.0,>=1.8.28, but you'll have botocore 1.5.92 which is incompatible.
Installing collected packages: earthpy
Found existing installation: earthpy 0.0.1-git
Uninstalling earthpy-0.0.1-git:
Successfully uninstalled earthpy-0.0.1-git
Successfully installed earthpy-0.0.1-git
@mbjoseph can you tell me what code cov is testing for? pretty please. i haz needz clarificationzzzz.
Updating the docstring for get_data() in the EarthlabData class in io.py to match the updated parameters that the function takes. Additionally, updating the utils fix_paths() function docstring in order to note the inputs and returns.
Currently we have a normalized_diff()
function which calculated norm diff of 2 bands.
It's nice and generic in that it can be used on ANY raster data BUT question
norm_diff()
?hey @mbjoseph @joemcglinchy @jlpalomino have a look at the function and see what you think. I won't change it during the semester but am open to changing it in December if you guys think it should be the opposite? Or is it ok?
Add vignette to normalized_diff function in spatial.py
Add vignette to plot_bands function in spatial.py
import numpy as np
import earthpy.spatial as es
red_band = np.array([[1, 2, 3, 4, 5],[11,12,13,14,15]]) nir_band = np.array([[6, 7, 8, 9, 10],[16,17,18,19,20]]) # Calculate normalized difference ndiff = es.normalized_diff(b2=nir_band, b1=red_band)
import earthpy.spatial as es
titles = ["Red Band", "Green Band", "Blue Band", "Near Infrared (NIR) Band"]
# Plot all bands of a raster tif es.plot_bands(naip_image, title=titles, figsize=(12,5), cols=2)
We have none. and we desperately need to begin documenting this package. This is the start of this effort...
example change in plot_rgb:
rgb: list, indices of the three bands to be plotted (default = 0,1,2)
would be modified to:
rgb: list (default = 0,1,2)
Indices of the three bands to be plotted
Here is a start at some functions to handle cloud masks. Currently they are very modular and super focused as written. Let's start a discussion around what types of functions would be most useful for typical modis and landsat workflows where you are removing cloudy pixels.
def make_cloud_mask(mask_arr, vals):
"""Take an input single band mask layer such as a pixel_qa
layer for MODIS or Landsat and apply a mask given a range of vals to mask
Parameters
-----------
mask_arr : numpy array
A array... to open the pixel_qa or mask raster of interest
vals : list of numbers (int or float)
A list of values that represent no data in the provided raster layer (arr)
Returns
-----------
arr : numpy array
A numpy array with values that should be masked set to 1 for True (Boolean)
"""
# this might require a lambda
for cval in vals:
mask_arr[mask_arr == cval] = 1
return(mask_arr)
def apply_cloud_mask(arr, the_mask):
# Create a mask for all bands in the landsat scene
cl_mask = np.broadcast_to(the_mask == 1,
arr.shape)
# If the user provides a masked array, combine masks
if isinstance(arr, np.ma.MaskedArray):
cl_mask = np.logical_or(arr.mask, cl_mask)
# Return combined mask
return (ma.masked_array(arr, mask=cl_mask))
def make_apply_mask(arr, mask_arr, vals):
cl_mask = make_cloud_mask(mask_arr, vals, arr)
return (apply_cloud_mask(arr, cl_mask))
Hey @mbjoseph just curious at this point with our testing setup if we want a yaml file
This one: is not representative of our dependencies. there are many more things there that we don't need. it's also an old version of the ea-environment. i'd like to understand what this file should contain IF we need it!
https://github.com/earthlab/earthpy/blob/master/environment.yml
Correct spelling in Line 16. Misspelled word in BOLD text:
All files in the list must be in the same Coordinate **Refenence** System (CRS) and
Reword content to better flow below:
The stack_raster_tifs function takes a list of raster paths and turns that list into an
1. a stacked geotiff on your hard drive and
2. (optionally) an output raster stack in numpy format with associated metadata.
Correction made in BOLD text for spatial-raster.rst:
All files in the list must be in the same Coordinate Reference System (CRS) and
The stack_raster_tifs function takes a list of raster paths and turns the list into the following:
Use the Sphinx documentation to describe the following element and options in index.rst:
Element:
.. toctree::
Options:
:maxdepth: 2
:caption: Contents:
Include a comment that states the elements and options are applied to the following documents"
Documents:
get-started
spatial-raster
spatial-vector
Line 32 in earthpy/clip.py defines shp_sub
, but is commented out with an alternative line of code below that defines object. This commented out line will not be used and should be removed (penultimate line below)
# Get a list of id's for each road line that overlaps the bounding box and subset the data to just those lines`
sidx = list(spatial_index.intersection(bbox))
#shp_sub = shp[shp.index.isin(sidx)]
shp_sub = shp.iloc[sidx]
Remove line 32
# Get a list of id's for each road line that overlaps the bounding box and subset the data to just those lines`
sidx = list(spatial_index.intersection(bbox))
shp_sub = shp.iloc[sidx]
Earth Analytics class repository
Add .DS_Store to .gitignore file on Earth Analytics class repository
the spatial-raster.rst file is missing a parameter.
Spatial.py has a lot of code that needs to be put in PEP8 format. Comments don't start with capitalized first letter, some comments are notes to other contributors and should be rephrased to be a "to do" instead of a comment to oneself.
The arr_out function is not in the .rst file but it is in the spatial.py file.
Starting letter in comments are not capitalized. Comments can also be more clear.
This comment should be rephrased.
https://github.com/earthlab/earthpy/blob/master/earthpy/spatial.py#L66
This comment referencing an issue with the "current" matplotlib but does not specify what the current matplotlib is.
https://github.com/earthlab/earthpy/blob/master/earthpy/spatial.py#L252
This comment should be changed
https://github.com/earthlab/earthpy/blob/master/earthpy/spatial.py#L345
clip_shp function in clip.py needs documentation, and clip_point function needs RST documentation.
Add docstring documentation to clip_shp and remove commented out code from crop_image
I propose adding this .rst documentation for both functions identified above.
Currently a bit of looping is required to plot a bunch of histograms in a raster stack
colors = ['r', 'g', 'b', 'k']
titles = ['red band','green band','blue band','nir band']
fig, axs = plt.subplots(2, 2, figsize=(10, 10), sharex=True, sharey=True)
for band, color, the_title, ax in zip(naip_csf, colors, titles, axs.ravel()):
ax.hist(band.ravel(), bins=20, color=color, alpha=.8)
ax.set_title(the_title)
Create a small function to handle hist plotting without all of the loops. it's just better that way.
potential arguments:
** i wonder if i could send a **kwargs dict object to be unpacked with a set of all the hist options. that would be much more flexible than the way i'm writing things now...
Currently there is a "Feature" in numpy where if you mask an array with no actual maskable values you get a pseudo mask array. Now... this breaks plot_Rsb...more later
Make this a function to plot each band in a stack. it's way too much for to plot it as it is now.
inputs
# calculate the total rows that will be required to plot each band
# this could be a function too
plot_rows = int(np.ceil(landsat_pre_fire.shape[0] / 3))
total_layers = landsat_pre_fire.shape[0]
# plot all bands
fig, axs = plt.subplots(plot_rows, 3, figsize=(15, 15))
axs_ravel = axs.ravel()
for ax, i in zip(axs_ravel, range(total_layers)):
band = i+1
ax.imshow(et.spatial.bytescale(landsat_pre_fire[i]), cmap='Greys')
ax.set(title='Band %i' %band)
ax.set(xticks=[], yticks=[])
# this loop clears out the plots for bands 8-9 which are empty
# but you have to populate them in matplotlib when you specify plot rows and cols
for ax in axs_ravel[total_layers:]:
ax.set_axis_off()
ax.set(xticks=[], yticks=[])
plt.tight_layout()
We don't need this any more! Yay.
The utils.fix_paths() has the following comment:
"This should be moved to the new package that handles data download"
Perhaps it should live in io.py then?
I would like to add documentation to two functions: clip_points and clip_line_poly. Currently there is no documentation describing the functions or the arguments. Both files found in clip.py.
I would like to add a description of the function and a description of each argument in each function. Ensuring that PEP8 is used throughout function. Adding a statement on where and how each argument is used.
crop_image has commented out code
Delete commented out code from crop_image
remove it. replaced with cold-springs-fire
Currently this repo contains an environment.yml file that is very similar (but not identical) to the environment.yml
file for the earth analytics class conda environment.
But, this package has dependencies that aren't needed for the class, and the class has dependencies that aren't needed for this package. One solution to this is to deal with the environment for this package separately from the environment for the class (but still make sure that students using the class environment can use this package).
From what I can gather from https://python-packaging.readthedocs.io/en/latest/dependencies.html, the convention is to use install_requires
in setup.py
(https://github.com/earthlab/earthpy/blob/master/setup.py#L33-L34) to specify the minimal set of requirements for users to install and use the package. But, it also seems like we could use a requirements.txt
file or an environment.yml
file.
So my question is: Which of the three options (install_requires
, requirements.txt
, and environment.yml
) is better in this case for ensuring that this package complies with typical practice, and that the package works with the earth analytics class conda environment?
Currently there are a suite of helper functions that are specific to earth analytics courses and website build. Create a separate module that handles these tasks so earthpy scope remains focused on spatial and remote sensing data only.
Edit the documentation to match pep8 guidelines and rearrange the import libraries to follow pep8 guidelines wherein more general libraries are listed first and more specific libraries after.
Also delete comments that seem unnecessary, and add comments to suggest a change to normalized_diff function to make it less confusing
Documentation should begin with a space and follow with a capital letter (i.e ' # Comment here')
Importing should begin with the most general libraries first
Delete comments such as
1.) #ndvi[np.isnan(ndvi)] = 0 : commented out in normalized_diff function
2.) # Should this wrap around show instead of plotting as it does? : commented in plot_rgb
3.) #ax.set(xticks=[], yticks=[]) :commented out in the hist function
Add comments
1.) For normalized_diff function to suggest a change ( # Could be confusing if one does not know that b2 is subtracted from b1.
#An individual might assume the first input is subtracted from the second.
#For example, it could be more commonly assumed : (b1-b2)/(b1+b2) )
2.) Add comment for stack_raster_tifs function to include .sort()
I am going to delete commented code in the io.py section of earthpy and update the fix_paths function section in the earthpy docs
I am going to delete lines 17-19, 25, and 29-30, which are commented-out code
before:
# 'week_02': [('https://ndownloader.figshare.com/files/7010681', 'boulder-precip.csv', 'file'),
# ('https://ndownloader.figshare.com/files/7010681', 'temperature_example.csv', 'file'),
# ('https://ndownloader.figshare.com/files/7426738', '.', 'zip')],
'co-flood-extras': [('https://ndownloader.figshare.com/files/7010681', 'boulder-precip.csv', 'file'),
('https://ndownloader.figshare.com/files/7010681', 'temperature_example.csv', 'file')],
'colorado-flood': ('https://ndownloader.figshare.com/files/12395030', '.', 'zip'),
'spatial-vector-lidar': ('https://ndownloader.figshare.com/files/12459464', '.', 'zip'),
'cold-springs-modis-h5': ('https://ndownloader.figshare.com/files/10960112', '.', 'zip'),
#'week_05': ('https://ndownloader.figshare.com/files/7525363', '.', 'zip'),
'cold-springs-fire': ('https://ndownloader.figshare.com/files/10960109', '.', 'zip'),
'cs-test-naip': ('https://ndownloader.figshare.com/files/10960211?private_link=18f892d9f3645344b2fe', '.', 'zip'),
'cs-test-landsat': ('https://ndownloader.figshare.com/files/10960214?private_link=fbba903d00e1848b423e', '.', 'zip'),
#'week_08': [('https://ndownloader.figshare.com/files/9666637?private_link=480fba92b3e882c4d35d', 'week_08', 'zip'),
# ('https://ndownloader.figshare.com/files/9666637?private_link=480fba92b3e882c4d35d', 'week_08-hw', 'tar')],
}```
after:
```# deprecating week 2 & 8 data given i've reorganized these data into single repos
'co-flood-extras': [('https://ndownloader.figshare.com/files/7010681', 'boulder-precip.csv', 'file'),
('https://ndownloader.figshare.com/files/7010681', 'temperature_example.csv', 'file')],
'colorado-flood': ('https://ndownloader.figshare.com/files/12395030', '.', 'zip'),
'spatial-vector-lidar': ('https://ndownloader.figshare.com/files/12459464', '.', 'zip'),
'cold-springs-modis-h5': ('https://ndownloader.figshare.com/files/10960112', '.', 'zip'),
'cold-springs-fire': ('https://ndownloader.figshare.com/files/10960109', '.', 'zip'),
'cs-test-naip': ('https://ndownloader.figshare.com/files/10960211?private_link=18f892d9f3645344b2fe', '.', 'zip'),
'cs-test-landsat': ('https://ndownloader.figshare.com/files/10960214?private_link=fbba903d00e1848b423e', '.', 'zip'),
}```
I am going to add docstring documentation for the fix_paths function in earthpy
Currently the colorbar function is supposed to match the height of a plot. This works but it also now covers up the entire plot horizontally...
SO Question: https://stackoverflow.com/questions/53121012/matplotlib-color-bar-height-equals-image-in-matplotlib-v-3-0-python-3-x
Currently when you run norm_diff if there is a divide by 0 issue you get the following warning:
/Users/leah-su/Documents/github/1-python/earthpy/earthpy/spatial.py:76: RuntimeWarning: divide by zero encountered in true_divide
n_diff = (b2 - b1) / (b2 + b1)
This warning comes from numpy and just informs the user that they were a divide by 0 issue! But it actually simply provides that cell with a "na" value so it works as you might think it would. Look into whether it's possible to throw a slightly more friendly error that says - warning: divide by 0. Cells that have this issue calculated with be assigned "NA". I'm not sure if there is a clever way to actually do this or not given the warning is thrown via numpy but it's worth looking into .
utils.py is not following pep8 conventions in regards to the order of imported libraries.
There is some code that has been commented out in _init_.py.
I will reorder the order of libraries being imported in utils.py following pep8 conventions.
I will delete the commented out code from _init_.py
In the readme.md file, earth py is spelled with two words rather than one. I'd like to propose this fix.
a. Remove spaces around keyword assignments and adding spaces after commas on the following lines: 138, 170, 174, 175, 250, 282, 332, 338, 339, 340, 341, 342, 343, 344, 393, 394, 401, 402, 403, 404.
b. Fix indentation on the following lines: 131, 132, 133, 134, 135, 301, 321, 322, 393, 425, 444.
c. Break up lines that are greater than 120 characters (in a couple of locations).
d. Remove an unused import from rasterio.mask import mask
a. Function docstrings that need a description of the return added: normalized_diff()
, stack_raster_tifs()
.
b. Function docstrings that need their return description updated: plot_bands()
(Function doesn't actually return anything. Should this return the plt and/or ax object?), plot_rgb()
(Function doesn't actually return anything. Should this return the ax object as described?), hist()
(Function doesn't actually return anything. Should this return the plt and/or ax object?)
Currently there is a draw_legend function that is supposed to make creating a nice legend for a plot easier. However as it is built, it's not general enough to use consistently.
I think i should remove it for the time being until it's fully tested but just noting here that right now it's in the package and really does not work.
with
statement.Plot RGB
The ``plot_rgb`` function plots each layer in a raster stack converted into a numpy array for quick visualization.
``plot_rgb`` takes 8 input parameters:
``arr``: numpy array
An n-dimension numpy array in rasterio band order (bands, x, y)
``rgb``: list
Indices of the three bands to be plotted (default = 0,1,2)
``extent``: tuple - optional
The extent object that matplotlib expects (left, right, bottom, top)
``title``: string- optional
String representing the title of the plot
``ax``: matplotlib AxesSubplot
The ax object where the ax element should be plotted. Default = none
``figsize``: tuple
The x and y integer dimensions of the output plot if preferred to set.
``stretch``: boolean
If True, a linear stretch will be applied
``str_clip``: int
The % of clip to apply to the stretch. Default = 2 (2 and 98)
The ``plot_rgb`` function returns the following:
``fig, ax``: figure object, axes object
The figure and axes object associated with the 3 band image. If the ax keyword is specified, the figure return will be None.
Example:
.. code-block:: python
import matplotlib as plt
import earthpy.spatial as es
fig, ax1 = plt.subplots(figsize=(12, 6))
es.plot_rgb(naip_image,
rgb=[0, 1, 2],
extent=naip_extent,
title="NAIP 2017 Post Fire RGB Image",
ax=ax1)
2.
Histogram
The hist()
function plots a histogram of each layer in a raster stack converted into a numpy array for quick visualization.
hist()
takes 6 input parameters:
arr
: numpy array
An dimension numpy array
title
: list
A list of title values that should either equal the number of bands or be empty, default = none
colors
: list
A list of color values that should either equal the number of bands or be a single color, (purple = default)
cols
: int
The number of columns you want to plot in
bins
: int
The number of bins to calculate for the histogram
figsize
: tuple
The figsize if you'd like to define it. default: (12, 12)
The hist()
function returns the following:
fig, ax or axs
: figure object, axes object
The figure and axes object(s) associated with the histogram.
Example:
.. code-block:: python
import earthpy.spatial as es
colors = ['r', 'k', 'b', 'g', 'k', 'y', 'y']
titles = ["Red Band", "Near Infrared (NIR) Band", "Blue/Green Band",
"Green Band", "Near Infrared (NIR) Band",
"Mid-infrared Band", "Mid-infrared Band"]
# Plot histogram
es.hist(modis_bands_pre_data,
colors=colors,
title=titles,
cols=2)
Issue 1: Fix capitalization issues throughout spatial.py so that first word is capitalized and there is a space after the pound sign.
Issue 2: Remove extraneous lines that are commented out from io.py
Line 16 in d2c1e31
First test for an example.
Currently we do'nt have any data to implement unit testing on. However given the nature of the packages that we are using, we should have some data to test against. Let's follow the rasterio model and add a small data subset to be used to testing purposes in earthpy.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.