Keep getting a "No such file:" type error when trying to show input target pair? about pytorch-2d-3d-unet-tutorial

[johschmidt42]

The dataset class 'Segmentation Dataset' expects a list of existing URLs (strings). It seems that at least one of your URLs does not exist. You can iterate over your list of URLs and check if each file can be found for example using the os module or the pathlib library. You should also test if you can read every file with the skimage.io.imread() function. I will provide an example once I get home.

[johschmidt42]

Try something like this:

# file 1 and 2 exist while file 3 does not exist on my machine

list_of_urls = [
    r'C:\Users\johan\Desktop\UNET\PyTorch-2D-3D-UNet-Tutorial\Carvana\Input\0cdf5b5d0ce1_01.png',
    r'C:\Users\johan\Desktop\UNET\PyTorch-2D-3D-UNet-Tutorial\Carvana\Input\0cdf5b5d0ce1_02.png',
    r'C:\Users\johan\Desktop\UNET\PyTorch-2D-3D-UNet-Tutorial\Carvana\Input\0cdf5b5d0ce1_42.png'
]

# iterate over urls and make a new list with pathlib.Path objects (makes it easier to handle paths/directories)

import pathlib
from skimage.io import imread

list_of_paths = [pathlib.Path(url) for url in list_of_urls]

# iterate over the paths and check if they actually exist
for path in list_of_paths:
    print(f'Path: {path} exists: {path.exists()}')

# iterate over the paths and check if you can read them with skimage.io.imread().
# if the file exists but you can't read it, check the error message that is thrown to you.
for path in list_of_paths:
    if path.exists():
        img = imread(path)

This will output something like this:

Path: C:\Users\johan\Desktop\UNET\PyTorch-2D-3D-UNet-Tutorial\Carvana\Input\0cdf5b5d0ce1_01.png exists: True
Path: C:\Users\johan\Desktop\UNET\PyTorch-2D-3D-UNet-Tutorial\Carvana\Input\0cdf5b5d0ce1_02.png exists: True
Path: C:\Users\johan\Desktop\UNET\PyTorch-2D-3D-UNet-Tutorial\Carvana\Input\0cdf5b5d0ce1_42.png exists: False

Thanks for getting back to me so quickly mate. I have taken a different approach and I am attempting to load in my own dataset from Google Drive. The same format as Cardana though i.e. PNG files split into images and masks. The only things I changed from your code was the following:

# Your code
root = pathlib.Path('/Carvana')
def get_filenames_of_path(path: pathlib.Path, ext: str = '*'):
    """Returns a list of files in a directory/path. Uses pathlib."""
    filenames = [file for file in path.glob(ext) if file.is_file()]
    return filenames

to

# My Code
from google.colab import drive

drive.mount('/content/drive', force_remount=True)

FOLDERNAME = '/content/drive/MyDrive/marvel/Image Dataset'

assert FOLDERNAME is not None, "[!] Enter the foldername."

and then proceeded as follows to get the respective lists of PNG files from each mask and image folder:

inputs = os.listdir(FOLDERNAME + '/images')
targets   = os.listdir(FOLDERNAME + '/masks')

inputs.sort()
targets.sort()

I just checked out the problem using skimage.io.imread() as you suggested, and it seems that when trying to read the file it is missing the '/images' part of the path that I had specified earlier. Which is weird, because when generating the inputs
and targets lists, it uses the '/images' and '/masks' part of the filepath to build a list of the files from each folder.

Seems to work fine if I change the current directory to that of each folder i.e. /masks and /images, as below:

Do you know if there is any way to do this without having to change the directory each time?

[johschmidt42]

os.listdir returns a list containing the names of the entries in the directory given by path. But you want the full path of your files, not the relative. It seems that when you try to read an image with imread and your input parameter is a relative path, it will take your cwd (current working directory) and combine it with the image name. Check your cwd with os.getcwd(). This should be /content/drive/MyDrive/marvel/Image Dataset. Now check your list of image names in inputs.

My suggestion when dealing with paths and directories: pathlib
You can basically use the function get_filenames_of_path() like this:

root = pathlib.Path(FOLDERNAME) # this will transform FOLDERNAME into a pathlib object
image_path = root / 'images' # note that you can connect a pathlib.Path object with a '/' operator 
target_path = root / 'masks'

list_of_full_paths_images = get_filenames_of_path(image_path)
list_of_full_paths_masks = get_filenames_of_path(target_path)

With pathlib you get a lot of very useful features. Print these for an example:

list_of_full_paths_images[0].stem
list_of_full_paths_images[0].name
list_of_full_paths_images[0].with_suffix('.tif')
list_of_full_paths_images[0].parent

Hi mate, I was just trying out your advice and now for some reason all of the ipynbs in your repository won't load?
Have you taken this repository down?

Cheers

[johschmidt42]

I haven't changed anything, the last commit was 2 months ago. What do you exactly mean you can't load the jupyter notebooks?
Can you not download (e.g. git clone) the repo or can you not open them? Or do you mean that you cannot execute the code within a notebook?
Please make sure that you start the jupyter notebook from your environment and select the correct interpreter (python). If you just open the notebooks in GitHub, just reload them. Github kind of never loads them the first time.

Sorry, I'm brand new to git so not too sure what is going on. Just for the .ipynb notebooks, your parts 1-4, none of them seem to be loading and I'm not too sure why. Keeps giving this error:

I've tried reloading, different browsers and restarting my internet. Also forked them to my own repository and can't get them open there either. I'll try downloading them instead and see if I can get around it that way.

Also mad impressed that you're not from a computer science background. Your Unet tutorial is by far the cleanest and most concise I have come across to date. Really solid work. Hopefully, I can get my dataset up and running this week following the remaining tutorials. Thanks for your help.

[johschmidt42]

I don't know what causes the notebooks not to open on your site, but downloading should be possible.
Here are the steps I performed on my Windows10 machine (Anaconda/Miniconda Powershell Prompt):

• open Anaconda Powershell Prompt
• conda activate <my_conda_env>
• mkdir C:\Users\<username>\Desktop\UNET
• cd C:\Users\<username>\Desktop\UNET
• git clone https://github.com/johschmidt42/PyTorch-2D-3D-UNet-Tutorial # command to download this project
• jupyter-notebook.exe # .exe is only for windows of course. You could also use jupyter lab
• execute code in jupyter notebook

Try these steps and tell me if you encounter any issues.

Thanks for the feedback, I appreciate it! I know the struggle with computer science and deep learning, programming etc. I just started doing this all at the end of my bachelors and also faced similar issues. Consider using a higher level API for training once you get it all up and running. My code probably works for prototyping but that's it. The transformations I wrote are also very boilerplate! I tried to reduce that in my other tutorial. I'm going to update the repo accordingly some time in the future and might also include a docker container.

That worked perfectly , cheers.

So should I be able to use your tutorials / code to train my dataset on Unet in its entirety? Or will I most likely have to use some other external resources to be able to do so?

I don't suppose you have a discord / signal / linkedin that I could shoot you some questions on do you? Saves me messing up your GitHub, plus I'd really value the input of someone from a non-comp sci background who is a little further ahead in their deep learning journey than I am.

[johschmidt42]

Sure, the code works without trouble. It's good for prototyping, but you'll need to think about logging, plotting, early stopping and many more things if you continue like this. So once you manage to start training and get some decent results, you should switch to something more robust.

If you want, I could give you my semantic segmentation pytorch lightning model (if I can still find it on my computer) in combintation with neptune.ai for experiment tracking.

Send me a friend request on linkedin, you'll find the link on medium.com - I'm glad if I can help you on your journey.
I don't think you're messing up the GitHub repo, because your questions could be beneficial for other people that are in the same situation.