cosmiq / cw-geodata

CosmiQ Works Geospatial Processing Toolkit for Machine Learning

This repository is no longer being updated. Future development of code tools for geospatial machine learning analysis will be done at https://github.com/cosmiq/solaris.

CosmiQ Works Geospatial Data Processing Tools for ML

This package is currently under active development. Check back soon for a mature version.

• Installation Instructions
• API Documentation
• Dependencies
• License

---

This package is built to:

• Enable management and interconversion of geospatial data files without requiring understanding of coordinate reference systems, geospatial transforms, etc.
• Enable creation of training targets for segmentation and object detection from geospatial vector data (i.e. geojsons of labels) without requiring understanding of ML training target formats.

Installation Instructions

Several packages require binaries to be installed before pip installing the other packages. We recommend creating a conda environment and installing dependencies there from environment.yml, then using pip to install this package.

First, clone this repo to your computer and navigate into the folder:

git clone https://github.com/cosmiq/cw-geodata.git
cd cw-geodata

Next, create a conda environment with dependencies installed as defined in the environment.yml file.

conda env create -f environment.yml
source activate cw-geodata

Finally, use pip to install this package.

pip install .

For bleeding-edge versions (use at your own risk), pip install from the dev branch of this repository:

pip install --upgrade git+https://github.com/CosmiQ/cw-geodata.git@dev

API Documentation

API documentation can be found here

Dependencies

All dependencies can be found in environment.yml

License

See LICENSE.

cw-geodata's Issues

Enable conversion of data to TFRecords

It would be great to be able to create TFRecords from data (and potentially vice versa) for improved compatibility with the TF Object Detection API. @avanetten, do you have code to do this already for SIMRDWN?

fix import tests

Tests currently don't import the right things - from the old structure.

Re-structure submodules for vector, raster

The submodule names are inconsistent with the common geospatial scheme of "raster" and "vector" (which are unfortunately inconsistent with the common ML/CV nomenclature of "image" and "label"). Going to re-name them to rasterimage and vectorlabel, and create sub-modules beneath them for a lot of the functionality. The current "label" module is getting unwieldy and I still haven't added in road network/graph support.

Write raster mask to vector functions

I added functionality to convert raster images to vector masks, but we need something for the reverse process still. This will also be good to do for road networks (@avanetten).

Write tests for CLI functions

Implement unit tests for vector.graph

@avanetten, do you think you could do this? If not, can you drop me a quick example file to build a graph from alongside a pickled graph object to compare it to? Ideally from a low-density tile or something like that so that I can include the files in the testing submodule.

Thanks!

Create a documentation front page

We need a docs front page to describe what the package does and where its different functionalities are located, as well as links to the various different usage resources (cookbook recipes, API docs, etc.)

cw_geodata.vector_label.polygon.geojson_to_px_gdf im_path

cw_geodata.vector_label.polygon.geojson_to_px_gdf

im_path : str
Path to a georeferenced image (ie a GeoTIFF) that geolocates to the
same geography as the geojson(s). If a directory, the bounds of each
GeoTIFF will be loaded in and all overlapping geometries will be
transformed.

Based on my reading of this I assumed I could feed a directory containing multiple tif images, which fails.

Implement unit tests for vector_label.polygon

Add functionality to make roads masks

@avanetten, I don't know if you already have this somewhere you want to leave it (e.g. APLS). If so, we can just import it from there; alternatively, if you'd rather port the code into cw-geodata we can probably put it in cw_geodata.vector_label.mask.

Remove osmnx dependency

osmnx is a fantastic package, but it's a bit hard to work with due to limited documentation of what resides in its node and path dictionaries, and it's tricky to get installed (particularly if you're using conda). It's also making it impossible to get the readthedocs built. We only use it for working with graphs in cw_geodata.vector_label.graph, so I'm going to remove the dependency and re-write vector_label.graph to remove the need for it.

Parallelize graphtools operations

Implement unit tests for utils.geo

Implement unit tests for `raster_image.image`

Set up readthedocs

Depends on #3. Set up a readthedocs.yml and readthedocs_environment.yml to build docs for cw-geodata.readthedocs.io (once that's created).

Put cw-geodata on PyPI

The version is probably now stable enough for a public beta accessible from PyPI.

Create cookbook and write usage recipes

Usages that need documenting:

• Creating masks from vector labels and csvs
• Creating and visualizing graph objects
• Convert a pixel-coordinate csv to a georegistered object
• Convert a georegistered geojson to a pixel-coordinate csv

May make sense to wait to do these until the entire 0.1.0 codebase is complete so I can add more to the list.

Fix intersphinx refs for various packages

The intersphinx references for a few packages are missing or incorrect (don't point to the directory or URL containing the objects.inv file). We need to go through the documentation, find the broken links, and fix the intersphinx call in docs/conf.py.

Functionality to interconvert roads labels between competition and geojson formats

@avanetten , If you get a chance to drop me a sample submission from SN3 I'd appreciate it - I just want to make sure I know what the exact submission format is so I can get the interconversion right. Thanks!

Implement unit tests for utils.core

Convert inputs to DarkNet format

@avanetten, do you think you could implement this functionality? I think we essentially need something that will take in geojson or competition-formatted footprints, or bounding boxes in other formats, and convert them to DarkNet format for YOLT/SIMRDWN. If you'd rather not do it, just point me to where it's implemented already and I'll add it in.

Create CLI

Create scripts and add setup.py entry points for CLI functions to complete various tasks through this repo.

This will likely be a meta-issue to organize various tasks for this repo.

Add graph functionality

Starting from @avanetten's https://github.com/cosmiq/apls/src/graphTools.py, re-write code to load a graph from a geojson/write a graph to a geojson.

Implement unit tests for vector.mask

set up Sphinx docs

Enable COG parsing

We currently have some tools implemented in https://github.com/spacenetchallenge/ml-export-tool and https://github.com/spacenetchallenge/ml-lambda-api, but it would be good to port those over to CosmiQ and clean up the functionality.

Enable creation of bounding boxes from polygons

@avanetten, I think you/we might have code to do this already for a variety of formats (TFRecord, COCO, Darknet) - can we try to port this in?

Make graph creation more efficient

The current implementation of cw_geodata.vector_label.graph is rather slow. It needs to be sped up at some point.

Write tests for graph export