Download Tier 2 images for remote locations about coastsat

[bruna-ambrosio]

Ok, Kilian. No problem. I can do it. :)
Could you send me a warning when you have finished updating the codes, please?

[kvos]

hi Bruna, here is a cool video on how the Tiers are organised for Landsat: https://www.usgs.gov/media/videos/landsat-collections-what-are-tiers

CoastSat only looks for Tier1 images but you could modify the SDS_download.py function to also look for Tier2 images (you just need to change the name of the Google Earth Engine collection to LANDSAT/LC08/C01/T2_TOA for L8 for example).

The issue with Tier2 images is that they need to be georeferenced (RMSE error against GCPs is more than 12m) before using the for time-series analysis. Probably the problem on your archipelago is that there are not enough ground control points (GCPs) to georeference the images (the same happens on small Pacific islands).

[bruna-ambrosio]

Hi Kilian,

Excellent explanation. Thank you so much for sharing this video. Satellite imagery processing is still a new world for me.

In my archpelago there are many constructions, airport and port, because the main island is inhabited. However, I do not know if these are enough to ensure the quality of georeferencing, as they are concentrated in a specific portion of the island.

I found and replaced T1 with T2 in the SDS_download file (3 occurrences), and ran the CoastSat again to test the change. However, the code is continuously processing in the download cell, without return any error or 0 images message. I requested images from all satellites for the year 2010, and I verified that there are two Landsat 7 (T2) images for this period (from the USGS). I have already waited for hours and the CoastSat is not able to respond from Landsat 7. I thought it could be my internet connection, but the same problem persisted when I tested it using a better connection.

Should I have to modify anything else in the SDS_download file or in another file? I also noticed that in Narra's example the polygon is formed by 5 points and I used only 4. Could this be a problem?

Thanks!

[kvos]

Yes better to use 5 points, where the last point is the same as the first point.
So what is happening in SDS_download is that I have put try/except structures around the download bits of code so that the script doesn't crash if the internet connection is temporarily interrupted. Therefore it will try downloading until successful in an infinite loop.

What is probably happening in your case is that there is an error because it was changed from T1 to T2 and it is try again in an infinite loop. I recommend you do some debugging as some things may be changing when using T2 instead of T1 in terms of metadata, bands etc...

[bruna-ambrosio]

Perfect. I will use 5 points in the next times.

That bad luck. I don't know if I have enough knowledge in Python to debug the script and find the error, but I will try.

Do you think I could download and clip the T2 satellite images manually from the USGS page and then move on to the next CoastSat steps? If I properly name and organize the images and metadatas in the correct folders, can it work?

Thank you for your help, Kilian.

[kvos]

I think it's easier to include T2 images in CoastSat's download functions, I will have a go at it so you can test it. The only problem is that usually T2 images are not recommended for time-series analysis.

[kvos]

ok @bruna-ambrosio , could you download the development branch and try the following:

• in your jupyter notebook, just after you have defined the inputs (second cell) with you polygon etc.., add this line:
  SDS_download.check_images_available(inputs)

then run the the cell and it should show how many images are available in each Tier. If you can send me a screenshot of what shows up, then we can go from there to include those Tier 2 images

[bruna-ambrosio]

I think that I will have to do a more rigorous quality control of the CoastSat results, due to the use of T2 images. But as my aim is a qualitative analysis of the coastline evolution, I hope this will not be a big problem.

Sure. Here is my polygon (with 5 points):
[-32.445446, -3.857915]
[-32.448738, -3.850517]
[-32.399821, -3.828781]
[-32.398889, -3.836099]
[-32.445446, -3.857915]

Thank you for your help.

[bruna-ambrosio]

Ok. I will do it.

[bruna-ambrosio]

Here's the screenshot now:

I have been waiting for several minutes, but didn’t finish to run. I think it found some problem.

[bruna-ambrosio]

I was analyzing the images available on the USGS page and noticed that there are only T2 images with L1GT processing correction level. And analyzing the SDS_download file, I don't know if the CoastSat is looking for images that belong to this level of processing. Could this be the problem?

I understand that the CoastSat is looking for only TOA images, but I'm not sure.

[kvos]

so when I run it with your polygon (all dates, all satellites) it shows this:

have you been able to download images at other sites (like Narrabeen)? it could be an issue with your internet connection or link to earth engine server...

[kvos]

if you get it working I will set up an option to download T2 images (those 152 images of Noronha could be helpful)

[bruna-ambrosio]

What a good new! Certainly those 152 images of Noronha will be very helpful for me. That's excellent. Thank you so much, Kilian.

I've been trying to discover why it doesn't work here...

I tried to run it using the Narrabeen polygon and the same problem occurs...the second cell doesn't finish to run and doesn't show the number of images available. So, I remotely accessed my university desktop on which I successfully ran all the code in the past (for Narrabeen polygon), but the same problem occurred. I also tried to use another browser (Google Chrome) to open the jupyter notebook, but was unsuccessful. I didn't notice any issue with my internet connection and the google engine authentication was successful. So I decided to reinstall the Anaconda3 and restart from step zero. At the anaconda prompt, I accessed the CoastSat-development folder, created the coastsat environment, activated it, authenticated the google engine and opened the jupyter notebook (with google chrome). I opened the example_jupyter file and added the line SDS_download.check_images_available(inputs) in the second cell, but the same problem persists.

Now I finally discovered that when I remove the additional line (SDS_download.check_images_available(inputs)) from the second cell, it works normally. Then the third cell works, too. For the Narrabeen polygon the output shows the number of images and these images are download. For the Noronha polygon the output only shows the number of images and this error. I think it was expected, right? You know how to set up an option to download these 152 images, right? :D

[bruna-ambrosio]

Sorry for my english mistakes and for spending yor time, Kilian. I'm very grateful for your help. Have a nice weekend.

[kvos]

ook @bruna-ambrosio , it was my fault, I forgot to add ee.Initialize() inside the check_image_available function and that is why it didn't work (sorry you had to reinstall everything). I'll fix that and create an option to download those T2 images. Everything will be on the development branch so it can be tested.

[kvos]

hi @bruna-ambrosio , I have updated the development: to download the Tier 2 images available, you have to add this line inputs['include_T2'] = True before calling SDS_download.retrieve_images . It will then download the Tier 2 images as well. Let me know how you go

[bruna-ambrosio]

Hi Kilian. Perfect. I'll test it and so I'll tell you how it worked. Thanks.

[bruna-ambrosio]

Hi Kilian. I tested it for the year 2010 and it worked perfectly :D The CoastSat have been able to find and download the two Tier 2 Landsat 7 images available for this period. Thank you very much for providing this new setup for downloading T2 images.

[bruna-ambrosio]

I have two more questions...now, about the steps after downloading the images.

1. Is it possible to set up the CoastSat to correct / fill these gaps in Landsat 7 (MS) images? I don't know if I need to call any functions or change any input parameters. Are there already any functions that do this?

2. I noticed that the extracted coastline plot is inverted because it uses a north hemisphere reference system. It is possible to change the projection system for the south hemisphere, right? Do I just have to change the 'output_epsg' to resolve it?

Should I post it in a new issue? And sorry for so many questions, Kilian.

[kvos]

1. The gaps are present in all L7 images from 2002 due to a hardware malfunction onboard the satellite, there is nothing that can be done about it. Fortunately, the bands move constantly from frame to frame.

2. Yes you have to change 'output_epsg' to a projected coordinate system that is valid for your area of interest (be careful, if you input a spherical coordinate system it won't work). If you're not sure you can always use 3857, which is Google Web mercator (basically google maps)

[bruna-ambrosio]

OK, Kilian. I think this will not be a problem. Thank you so much for all your help :)