Mismatched images and labels for semantic segmentation about bdd100k

[griffintin]

@rod409 I found similar mismatches and my instance training results was very bad.
In addition to the above problem you pointed out, I found some labels are wrong for instances.

images:
val/7d128593-0ccfea4c.jpg : background was labeled as truck

val/a640b39e-f556d329.jpg background grass/curb was labeled as train

val/a98e2bc9-9e2a0b07.jpg poles was labeled as car, some cars labeled as bus....

(please ignore the color, I forgot to change RGB order)
@fyu
Hope you also noticed these wrong labels for instances.

[XiaLiPKU]

I found a few images and labels that do not match in the images/10k images for semantic segmentation and the corresponding bitmasks downloaded in labels/sem_seg/masks. These are the following files

In images path but not in labels:

• train/3d581db5-2564fb7e.jpg
• train/52e3fd10-c205dec2.jpg
• train/781756b0-61e0a182.jpg
• train/78ac84ba-07bd30c2.jpg
• val/80a9e37d-e4548ac1.jpg
• val/9342e334-33d167eb.jpg

In labels path but not in images:

• train/fee92217-63b3f87f.png
• train/ff1e4d6d-f4d85cfd.png
• train/ff3d3536-04986e25.png
• train/ff3da814-c3463a43.png
• val/ff55861e-a06b953c.png
• val/ff7b98c7-3cb964ac.png

I am able to train a model successfully after removing these images but wanted to bring this to your attention.

Thanks.

Thanks for informing!
We also notice this problem and would like to publish a new version of the image zip these days

[griffintin]

@XiaLiPKU

could you please have a look at the above validation images of wrong labels for instances segmentation?
I would like to mention that, in training labels, I found similar wrong labels.
But sorry that I am not able to check all 7000 images to give you details.

[XiaLiPKU]

@XiaLiPKU

could you please have a look at the above validation images of wrong labels for instances segmentation?
I would like to mention that, in training labels, I found similar wrong labels.
But sorry that I am not able to check all 7000 images to give you details.

All thanks for your mentioning.
Currently, we have no method to deal with these errors, except to find them and let the labelers re-label them.
This issue is significant to us, so we will try to solve it as quickly as we can.

[XiaLiPKU]

I found a few images and labels that do not match in the images/10k images for semantic segmentation and the corresponding bitmasks downloaded in labels/sem_seg/masks. These are the following files
In images path but not in labels:

• train/3d581db5-2564fb7e.jpg
• train/52e3fd10-c205dec2.jpg
• train/781756b0-61e0a182.jpg
• train/78ac84ba-07bd30c2.jpg
• val/80a9e37d-e4548ac1.jpg
• val/9342e334-33d167eb.jpg

In labels path but not in images:

• train/fee92217-63b3f87f.png
• train/ff1e4d6d-f4d85cfd.png
• train/ff3d3536-04986e25.png
• train/ff3da814-c3463a43.png
• val/ff55861e-a06b953c.png
• val/ff7b98c7-3cb964ac.png

I am able to train a model successfully after removing these images but wanted to bring this to your attention.
Thanks.

Thanks for informing!
We also notice this problem and would like to publish a new version of the image zip these days

To quickly solve this current problem, you can use the patch here: patch.tar.gz. Hope this can help you.

Moreover, we are updating downloading files, you shall see them today or tomorrow.

[pcicales]

Hello, were the download files updated? I downloaded last week.

[XiaLiPKU]

Hello, were the download files updated? I downloaded last week.

Sorry to let you wait.
We published a new version of image downloading files.
You can now download them on the website: https://bdd-data.berkeley.edu/
Now, 100k and 10k images are separated into two files.
For more information, you may refer to: https://doc.bdd100k.com/download.html#id1

[pcicales]

@XiaLiPKU are the labels updated as well? I am getting poor accuracy with a cascade RCNN on the sem_seg_20 task for example, perhaps the same issue mentioned by @griffintin? Perhaps we could have a list of annotations that are defective in the meantime, so that we may exclude them? This seems like a serious issue @fyu ... can you please confirm if these errors exist?

I am also running the ins_seg task, and my code is currently showing that only 1696 images have associated annotations. Performance so far (using a 3 stage cascade RCNN) is also low. Im still looking into this and will post back here over the weekend.

[guifentian]

@XiaLiPKU @fyu

I am trying to use the latest 10k images/labels , and the newest code for instance segmentation.
The labels are still wrong as reported here #103 (comment)

what should we users do with current partial wrong labels, any advices?
How can we clarify the cause for low AP , the network, the labels?
Or can you provide your results using current labels on typical models like Panoptic-FPN, DeepLab, etc..?

[thomasehuang]

These issues should be already addressed, let us know if it's still an issue.