wrfitch / fyp Goto Github PK

It might be worth balancing the dataset closer towards the more random cases that include water, concrete, greenery etc in different capacities.

one day they were fine, the next they weren't. Find out what changed before commit e79f585

• Combine datasets to be one easy import
• Is my GBR object ok?
• define upscaled and downscaled resolutions

Cut a disso release branch with clean code and minimal notebook output, depending on where the notebook is needed. This may also include a model, though that should be kept out of the git tree.

In the image import pipeline, the data is being continually scaled and rescaled - optimise this pipeline by reducing the use of scaling until absolutely necessary.

Fix each notebook and script to be of a consistent standard of quality that can be submitted.

Write it out into log files or something if necessary - it's unbearably messy in situ.

Validate model against all types of terrain in the dataset, including aquatic, urban, manufacturing, forest, farms, suburban etc. Split this by land use classification.

There's a lot of code that doesn't need to be sat in these notebooks, cluttering the notebooks so they aren't so readable. Extract shared code and unnecessary verbosity.

It's too cluttered. It's a handy place to store research links, but the PDFs are getting out of hand.

Current accuracy given 224px images =~ 11% +- 2
Target <= 5%

Options:

• Experiment with different network architecture - see fastai model zoo.
  • compare networks, see what works in background literature.
• tune other hyperparameters
  • optimisers
  • loss function
  • batch size
  • epochs
• Create an ensemble of networks and combine predictive power.
  • Existing kernel measurement idea re: take a load of nearby measurements and incorporate them into a tab recommender.
• Data
  • Give augmentation another go
    • only use rotations OR flips?
  • Import far more data to get a good prediction?

Look into implementing a cnn_learner that can handle multidimensional y_ranges

Currently the PNG exports of the GeoTIFF files are distorted. Examine whether the GeoTIFF files are distorted in GIMP and resolve however possible. This may be due to a scaling issue somewhere in the export pipeline.

Now that we've definitively proven the old 160px image pipeline is inferior to the new 224px, replace immediately

Incidentally, testing it with even larger images may be useful.

I mean, dear god. fix file structure and remove unnecessary rubbish. Write import script into files if streaming architecture ends up being necessary.

Could also cross my fingers and hope google doesn't notice, but I need their materials. Incidentally, hinging my entire FYP on what one company thinks of me was probably not a smart move.

Implement a local script that allows imports to be run locally.

This thread is to keep note of general optimisations as they are found. AI optimisations should go in #21

Currently it's organised in a very awkward way, based around these notebooks. Look for options to make more elegant.

All the images are weirdly dark. This could be resolved in fastai, but I'd prefer to limit that kind of operation where possible.

add in latlong data while you're at it.

For some reason the data isn't aligned properly.

If the filepath doesn't exist, getFilepath(coords) takes a very long time to notice.