SOTA about ultralytics

[bakeryproducts]

No way professionals like Ultralytics claiming 'new standards in SOTA real-time detection and segmentation' without proper comparison on different resolutions, datasets, batch-sizes, GPU-archs.

Surely Ultralytics got better SOTA comparisons then two .png files of accuracy plots vs other YOLO models.
Here some reference that you brought up by yourself in plots from #230 : https://arxiv.org/pdf/2207.02696.pdf

YOLOv7 <...> has the highest accuracy 56.8% AP among all known real-time object detectors with 30 FPS or higher on GPU V100

Whats YOLOv8 highest AP on COCO val with 30 FPS or higher (on A100 GPU as you did in #230)?

[HarishGuragol]

@bakeryproducts you can go through this medium blog for understanding of the SOTA comparison and results
[medium blog] (https://kandiraju31.medium.com/sota-metrics-comparison-c1c65755f346)

[AyushExel]

@bakeryproducts @HarishGuragol we'll release comparison plots soon with exact commands to reproduce. There are multiple tasks so it takes comparatively more time. Please be patient.

[glenn-jocher]

@bakeryproducts SOTA comparisons added in #260

[AyushExel]

@bakeryproducts yes, that's the standard to compare mAP vs speed on coco test set. Every repository does that and thats the main plot. We've provided size comparison too. And more plots are coming in the future. And all these plots will be available in the paper too. FPS too can be calculated using the latency. We need to prioritize bug reports too.

Regarding 56.8 map, those are p6 models. That's a different family of models. Base models (trainer at 640 pixel) and larger p6 models( trained at 1280 pixels). 53.9 is the highest-ever map achieved by the base model still being faster and smaller than the next model in the line.
We cannot compare the different classes of models with very different sizes. v6 had 5 base models and v7 has 2. Both are included in our charts. Our p6 models will be available soon if you're interested in that. We will keep following the release on a rolling basis. hope I answered your questions. If not, please check back in a few days.

[bonlime]

@glenn-jocher totally agree with @bakeryproducts on that. According to the numbers you're still beyond the Yolo v7 from the paper above. Comparing only to your own results and calling it a "SOTA" is a strong exaggeration

[bakeryproducts]

So right now highest AP is 53.9 (2.9p behind real-time YOLOv7 56.8) and you are already willing to call that 'setting new standards in SOTA'? Are there any results at higher resolution for v8?

[Laughing-q]

@bakeryproducts hi actually the highest model we got is 57.0 with resolution 1280. But we noticed that the settings we used for the model is suboptimal after the training finished, so the 57.0 is suboptimal. We think it's not ready to release if it's suboptimal. And we don't have enough time to train another before release the code. Thanks for the attention. :) We'll release the highest model soon.

[wesboyt]

has there been a followup here @bakeryproducts ?

[bakeryproducts]

Hey @wesboyt, no, but when the Ultralytics team says "soon," it could take years. No need to worry though! It's been only a year since SOTA claims, and we've been waiting for YOLO papers from them for about three years i think. fingers crossed!