nscc-cogs / aestheta

An Industrial Think Tank Focused on Developing and Promoting AI Technology for Geospatial Applications [Please note this group was formed as an academic exercise for educational purposes and does not represent a real world organization]

Aestheta

An Industrial Think Tank Focused on Developing and Promoting AI Technology for Geospatial Applications

Let's get this thing rolling!

Starting out: lets have a look at the previous tutorials from GDA2030 to get an idea where we can begin. There we can learn some basics of using google colab, we can see some basic image loading examples in python, see an example of accessing landsat data, see some more advanced image maniuplation techniques, and finally get an example of building a model in scikitlearn.

let's do this!

Check out our latest 'Hello Earth' satellite classification and other weird tests in our experiments section...

We're Live!

kind of... You can see a website of sorts at https://nscc-cogs.github.io/Aestheta/

Getting Started

To get going fast, try the following in a new Google Colab notebook

!git clone https://github.com/NSCC-COGS/Aestheta.git

import Aestheta.Library.core as core

core.getTile(source = 'google_sat', show=True)

You should see earth appear - represented as a small numpy array! Stay tuned for more simple examples of what were doing with this data.

Requirements

We reccomend 64-bit python version 3.7.10 and higher. We use scikit-learn which includes numpy, imageIO.

Note for windows users!

To get access the magical GIS tools in GDAL, rasterio, and fiona on windows - we recommend you donwload the appropriate wheel files for your vesrion on python from the glorious website of Christoph Gohlke.

Once you have these wheel files, you can run the following commands in command line ...

cd c:\downloads

py -3.9 -m pip install GDAL-3.2.2-cp39-cp39-win_amd64.whl

py -3.9 -m pip install rasterio-1.2.1-cp39-cp39-win_amd64.whl

py -3.9 -m pip install Fiona-1.8.18-cp39-cp39-win_amd64.whl

these are for example having downloaded the wheel files to c:\downloads and for 64bit python version 3.9

Earth is Special

We're teaching an AI to understand what it means to be looking at our lovely planet.

There is lots of great data available and tools we can use! Early results are very promising.

aestheta's Issues

A link to our core example should be placed in our ReadMe

Place a link to our core_example_2.ipynb file so the user can jump right into using our code in google colab.
The following link should suffice:

https://colab.research.google.com/github/NSCC-COGS/Aestheta/blob/main/Experiments/core_example_2.ipynb

The readme can be edited directly at the following link:

https://github.com/NSCC-COGS/Aestheta/edit/main/README.md

(OR, the readme can be modified on your local machine by 'pulling' the Aestheta git repo, modifying the file, commiting the changes, and 'pushing' it back. )

You should add some accompanying text explaining the link to the readme.
You may also consider adding some commends or functionality to the core_example_2.ipynb such as demonstrating different geographic locations.

swap lat and lon in getTile()

and test the tutorials etc

Add Landsat scene loading functions to core

Some functions exist in the previous tutorial 2 from this course wherein we grab a large list of landsat scene metadata and links where they can be downloaded from Amazon Web Services. We parse this file and perform simple look ups to find landsat scenes for a given location. These functions should be integrated intro our core library and tested. Examples of the code being used (or tested) can be seen here. There is also an example python notebook which illustrates the results.

More information on the original source of landsat data used in these tools:
https://registry.opendata.aws/landsat-8/
https://docs.opendata.aws/landsat-pds/readme.html

**note this source of landast data is being depreciated! We may need to investigate new means of obtaining the data
the direct replacement for the above service seems to operate quite differently
https://www.usgs.gov/core-science-systems/nli/landsat/landsat-commercial-cloud-data-access

Write a mission Statement

Create a new python Notebook illustrating all data sources for getTile()

Building on the basic functionality of our core example python notebook:
https://colab.research.google.com/github/NSCC-COGS/Aestheta/blob/main/Experiments/core_example_2.ipynb

Create a new python notebook which illustrates each of the available options for data sources found in the aesthera core.py
A list of possible sources (such as 'google_sat') can be found in the getTile() function located below:

Aestheta/Library/core.py

Line 32 in 229dbbf

def getTile(xyz=[0,0,0], source='google_map', show=False):

The python notebook can be created in google colab, and uploaded to the /Experiments section in Aestheta:
https://github.com/NSCC-COGS/Aestheta/tree/main/Experiments

Build the new ipython notebook which illustrates all avilable data sources of getTile() and upload it to the experiments section
optional: you may want to add a link to the new ipython notebook in the getting started section of the readme on the git.

loadModel() function currently wont accept a name argument

Aestheta/Library/core.py

Line 108 in 229dbbf

def loadModel(name = None, model_dir = 'Models/'):

The function works when no name is set by finding the most recent available model in the /Models directory in the git repo:
https://github.com/NSCC-COGS/Aestheta/tree/main/Models

However, if an actual model name is specified - the loadModel() function fails.

Add the basic functionality to specify a model by name in the loadModel() function.
Optionally: test to see if the specified named model exists - or add other features like finding the SECOND most recent model etc...

Model Loading is now broken.

@donuty-party
It seems that you're addition to to the model loading aspect of the script no longer functions. I've manually rolled back and removed the import of the new model.py code so that other features can be used and people can call things in colab etc. We may need to reassess how this model system works. We can either roll back to the original functionality or setup a hierarchy of packages where core.py and model.py are both sub-packages.

I'd like to get back to our original state for now. We may need to discuss this before going forward. Currently we cannot load models.

Traceback (most recent call last):
  File "c:\_dev\Aestheta\Library\core.py", line 22, in <module>
    from .model import Model # currently not working
ImportError: attempted relative import with no known parent package

What are the limits of the usage on google free map api

build a lat/long GPS sample function

we need a tool that grabs the classification value from a lat/long

This is quite similar in principal to issue #14

Investigate Microsoft based alternative to Google Colab

We should explore the Microsoft's Offerings for Cloud based Python Development Environments

This is not directly related to our code base necessarily. To date we have been discussing and using google colab as a baseline web based python develop environment where we can easily cue up a python environment in the browser where libraries like google tensforflow are readily available.

as we look to processing larger datasets stored on the NSCC-azure/onedrive server (a Microsoft product) It maybe wise to investigate options for Microsoft based alternatives to google colab. Perhaps these tools would integrate better with our data servers, as well as gitHub (which is own by Microsoft), and Micosoft's VS code (the IDE we've been mostly using it seems)

So! It seems that Microsoft has retired its direct alternative to google colab, called Azure Notebooks .

A specific good option may be GitHub Codespaces which is currently under a beta and you must apply for early access.

Move GeoAI_ESRI to discussion

Aestheta/Experiments/GeoAI_ESRI.txt

Line 7 in 84fe8bb

After eelgrass project is complete, should we consider mimicking one of these projects to learn the workflow?

Hey this seems out of place in /Experiments, perhaps we should move it to the discussion section

Models don't load without setting python working environment

Aestheta/Library/core.py

Line 108 in 229dbbf

def loadModel(name = None, model_dir = 'Models/'):

Test and speed up core.classifyImage()

Aestheta/Library/core.py

Line 65 in 0fe6e4f

def classifyImage(img_RGB,classModel,classes):

Merge shpreader to main branch

Aestheta/Library/core.py

Line 9 in 75f33c0

def shpreader(fname, show = False):

shpreader can be used to load shapefiles into numpy arrays and only exists in the coreJG branch. We will need this functionality for the zostera project. We should move this over to the main branch.

Move, copy, or merge core.shpreader() into the main branch from the coreJG branch

Migrate test script from Library.core.__main__ to new Tests folder

Develop function to load available files directly from OneDrive

Our Eelgrass data is stored on NSCC's OneDrive (a microsoft Azure server). We should investigate if we can load files directly from OneDrive easily using python. This would remove the requirement to manage the data locally or specifically download the files as we need them. This basic functionality could be useful when accessing large eelgrass mapping support files - for example the large aerial photography geotiffs required for project zostera.

It seems this can be accomplished through the OneDrive SDK for Python which may be rather straightforward but in our case I suspect accessing our data will require authentication.

Implement a simple function to load shapefiles

We will require shapefiles periodically as training data and to check our classification results. Lets investigate some methods for achieving this. A simple library that may facilitate this is pyshape: https://github.com/GeospatialPython/pyshp

An initial iteration of this function could simply load a given shapefile and display it using matplotlib. The function could be done as an experiment or included into the core.py functions.

Explore how we can use leaflet

Our landing page could possibly be spruced up with an interactive leaflet webmap.
https://leafletjs.com/examples.html

Our current landing page can be seen here:
https://nscc-cogs.github.io/Aestheta/

Examples of other doing similar things on github can be seen here:
https://github.com/StephSaephan/leaflet-map-example

With an example of the leaflet webmap here:
https://stephsaephan.github.io/leaflet-map-example/
(note that the link is github.io!)

A video which may overview the process may be here; (I have not reviewed the entire video)
https://www.youtube.com/watch?v=liMd1L8KDpc&ab_channel=JerryShannon-UGA

Investigate these resources and others and determine if and perhaps how we can use leaflet in this way. Prepare a brief report indicating the idea and how we would go about achieving it. Brainstorm what sort of information or functionality we could demonstrate on our leaflet map.

Classifier hard coded for 256x256 tiles

Aestheta/Library/core.py

Line 72 in 0fe6e4f

img_class = np.reshape(arr_label_model,(256,256,4)) #hard coded for 256x256 images!

core.py crashes on currently enabled test

Hey Kevin, do you know offhand what's going on here?

/content/Aestheta/Library/core.py in loadModel(name, model_dir)
    125     filename = model_dir+newest_model
    126     print(filename)
--> 127     classModel, classes = pickle.load(open(filename, 'rb'))
    128 
    129     return classModel, classes

sklearn/tree/_tree.pyx in sklearn.tree._tree.Tree.__cinit__()

ValueError: Buffer dtype mismatch, expected 'SIZE_t' but got 'int'

Lets put some thought into the "labels" we use for Issues

These help organize our issues and make them easier to filter down. We should also consider the colors so that they can be differentiated easily. Perhaps we should include:

• Dev
• Ops
• Research
• Testing
• Critical
• Non-Critical
• Complex
• Basic
• Bug

Any more good ones?

Experiment with Keras tools using Overhead-MNIST dataset

Keras is a very approachable and 'high-level' python library for designing and running pretty wild machine learning tools such as convolutional neural nets and the like. The library can be accessed quickly and easily inside google colab. Using freely available machine learning datasets we can pretty quickly establish and train models to identify features, segment objects, and even generate new images.

Using the Overhead-MNIST dataset, we should be able to cook up some pretty neat machine learning based analysis techniques such as image segmentation using u-nets. This would probably be easiest to quickly put together using google colab. If you succeed building an .ipynb demonstrating something like the u-net segmentation - upload it to the /Experiments section on the git.

** Hey! And don't forget how great kaggle is! You can explore some example python notebooks exploring the Overhead-MNIST dataset directly on the kaggle site!

** Some additional information get downloading kaggle data directly to your google colab environment can be found here: https://www.kaggle.com/general/74235

Refactor and comment “Hello Earth.ipynb”

calculate lat/lon extents of getTile()

We're experimenting with displaying our tiles interactively using the python library folium. This library can display numpy arrays directly to the map. For our tiles to show up correctly on the map, we will need to record the latitude and longitude extents of the tile.

Create a function that calculates the lat/lon extents of a given tile

Investigate the application of this GAN technique

https://github.com/CompVis/taming-transformers

Test modelLoad() function

this should automatically load the most recent file in Models path based on timestamp in the file name.
Classifying images should now be possible without specifying a model or class list. Ie,
img_class = classifyImage(img_RGB)

Otile WMS source not working

I don't believe it is working. #

Determine a list of ‘classes’ as per google maps various colors

may be quite out of date.
https://blog.google/products/maps/discover-action-around-you-with-updated/

Google maybe a dead end for us as I understand their class colors can be quite vague and overlap (they are essentially for visual purposes only I suppose!!

Implement Normalizing Difference image function in core.py

create core.norm_diff() as outlined below:

the function should accept an RGB numpy array such as the google_sat images we have been displaying so far (we have been using the variable name IMG_RGB). The function will a allow the user to specify two image bands (B1 and B2) such that a standard normalized difference (ND) can be calculated, wherein:

ND = (B1-B2) / (B1 + B2)

This function will return numerical values ranging from -1.0 to 1.0 and will require a 32 bit floating point array,

The function should return a single band image array array of the same x and y dimensions of the input image. The result should optionally be able to be plotted using the matplotlib function imshow() and the plot should display details of the image (meaning that the ND values may require being stretched or manipulated so that the image displays well given the new numerical range)

An input image may appear as such:

IMG_RGB = core.getTile([-63.5752,44.6488,2],source='google_sat')

and an output from the new function may appear as such:

ND = core.norm_diff(IMG_RGB, B1=1, B2=2)

Review Andrew Glassner's Deep Learning Crash Course Video

Andrew Glassner's video Deep Learning Crash Course is posted in the discussion section at the following link.

#33

Review the video in its entirety and post a brief summary outlining the basic concepts covered by the video in the Aestheta discussion section.

The classifier we train should be saved to the git

Aestheta/Library/core.py

Line 63 in 0fe6e4f

return classModel, classes

getTiles_3x3() function does not work with lat/lon as input

Aestheta/Library/core.py

Line 180 in 229dbbf

def getTiles_3x3(xyz=[0,0,0], source = 'google_map', show=False):

Running the following test:

import Aestheta.Library.core as core
core.getTiles_3x3([-63.5752,44.6488,2],source='google_sat',show=True)

Gives the following result:

We can determine the tile for these coordinates using our tile_from_cords() function:

print(core.tile_from_coords(63.5752,44.6488,2))

which returns:

[2, 1, 2]

If we plug these into the getTiles_3x3() we see the correct results:

core.getTiles_3x3([2,1,2],source='google_sat',show=True)

Implement a fix in the code base such that this error is resolved

Create and Begin to populate new "Tutorials" Section

A new 'Tutorials' folder in needed at the root of the git which includes a series of straightforward example python notebooks that demonstrate functionality and concepts of our code.

Proof read wikipages

Implement "Points of Interest" table

We could benefit from a clean and clear way to tracking certain 'points of interest' - specific locations where we may like to test our models repeatedly. Currently, out getTile() can accept either tile coordinates integers, or lat/lon float values. We could design this function to accept strings as well whereby we provide a list of specific strings which relate to a list of coordinates.

for example: the Eiffel tower is at 48.8584° N, 2.2945° E

we can create a lookup table dictionary such that:

{'eiffel' : [2.2945,48.8584]}

which should give get tile the following functionality:

core.getTile('eiffel', show=true)

Implement a system which can acceive the above funcatinaity in the core.getTile() function.
Consider: how could we incorporate both a default zoom level for a given POI and also the ability to use various zoom levels for a given AOI when calling getTile()

Once built, feel free to add some points of interest of your own!

Setup repo as a proper Python package

I was slightly trying to avoid this - which was silly of me. But there is a proper way to set up a python package! If we follow these relatively straightforward steps - we should be able to build our library and install using pip.

This would look something like this, if we were to call the command in goggle colab for example:
!pip install git+https://github.com/NSCC-COGS/Aestheta.git

What we are currently doing is the following in many of our tutorial and experiments:

!git clone https://github.com/NSCC-COGS/Aestheta.git

What the above command does is simply clones the git repo to our instance of google colab. This is rather nice and straightforward as it gives us quick access to all the files available in our git. This includes all out python code, our model files, and any other data we place on the git repo.

What this method does not do - is install our code as a package. This has various implications such as any specific external packages we require to run our code (such as shapefile) will not be automatically included. There are a few other differences but lets not get concerned with those right now.

So, lets get our code setup as a proper package!

Otile WMS source not working

This needs to be tested

Aestheta/Library/core.py

Line 30 in 2302c48

elif source == 'otile':

Design specific Project Steps for Project Zostera

Setup requirements.txt

A requirements.txt file is a simple python housekeeping file that lets us (and others) keep easy track of that external python libraries our code base requires in order to run. It really is a simple text file basically listing the required libraries in a way in which python's package manger (pip) can scan and ensure they are installed.

An overview explaining how the file is created and used can be found here. https://note.nkmk.me/en/python-pip-install-requirements/

getTiles_3x3 does not work if source = "google_map"

It works if source = "google_sat", but return error if source = "google_map"

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-65-c7e137954c9f> in <module>()
----> 1 img_features = core.getTiles_3x3(xyz,source="google_map", show=True)

/content/Aestheta/Library/core.py in getTiles_3x3(xyz, source, show)
    153             x0 = (j+1)*256
    154             y0 = (k+1)*256
--> 155             img[y0:y0+256,x0:x0+256] = tile_img
    156 
    157     if show:

ValueError: could not broadcast input array from shape (256,256,4) into shape (256,256,3)

Establish the point of Contact for Dr. Webster representing the group.

We need a member of the team to take ownership of the direct communications with Dr Tim Webster. This person will provide Dr Webster with updates on the status of the project, clarify project goals, and communicate information back to the group.

"Hello Mind" Experiment

I believe we would benefit from building a simple neural network example using keras and demonstrated in an ipython notebook (.ipynb). Such examples are readily available pre-built in google colab - but perhaps we can make our own basic example.

Create new "/Projects/Zostera" Section

To get started and organize the code for project Zostera we should create a new folder location /Projects/Zostera were we could include scripts, meta data, and some information relating to the zostera project. Python scripts in this directory could perform tasks like grabbing the project data from oneDrive, performing the georeferencing, computing the image classification model, and applying the results.

Make a new folder in the git /Projects/Zostera and inset a new python file process.py where we can begin scripting the Zostera processing steps.

determine a source for Landsat data

What classifier (from scikit-learn) can add to simpleClassifier()

perhaps xgboost would be a good option
https://www.kaggle.com/stuarthallows/using-xgboost-with-scikit-learn

Test the classifier across scenes and scales

Implement confusion matrix validation

We should implement a simple cross check of our classification results wherein we our compare the classification results against for example a google map tile.

Some more information on implementing confusion matrix using scikit learn can be found here:
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html

pytest is commented out in action workflow

Aestheta/.github/workflows/python-package.yml

Line 40 in aa8f00e

# pytest

Convert GPS coordinate to WMS tile

Aestheta/Library/core.py

Line 7 in 1db5026

def getTile(xyz=[0,0,0], source = 'google_map', show=False):

the getTile function should have the option to get tiles based on a GPS coordinate instead of the tile indexes. This would be more intuitive. The simple math to facilitate can be seen here: https://wiki.openstreetmap.org/wiki/Slippy_map_tilenames

It would be wise to write this conversion as a separate function and then perhaps call that conversion inside getTile if a certain argument is set (such as latlon = True, or if floating point values are passed to x,y)

Test pip install on package

Once the package is setup properly and built
we should be able to properly install our library.

Test this by attempting the following command in a new google colab notebook
!pip install git+https://github.com/NSCC-COGS/Aestheta.git

note: there may be some small adjustments to this line of code to get it to work
optionally: upload the .ipynb to the tutorials section