Do datasets have a size limit? about hub

[kalenmike]

@Latzi To improve the training speed you will want to train on a system that has as much memory as you can afford. We are currently working on a cloud training solution but it is not ready for release. Your best option for the moment is to select bring your own agent from the Ultralytics HUB and connect to a machine with more powerful hardware.

[Latzi]

I understand the need for more powerful hardware and I was hoping that you could point me towards some options ? Where I can find such services apart from Google AUTO ML or AWS ? The AWS looks like it is able to handle the training and it will take around 3.5 days to finish the model. Also the price is only $6.18 / h so not too bad. Do you know of any other options ? I have a few more models to train, happy to pay and I am trying to avoid waiting for 4 days :-) . Thanks again and I can hardly wait to test your cloud training solution when it becomes available.

[glenn-jocher]

Lambda has some good prices also. You'll get the best deals with reserved instances, but they've got competitive spot prices as well.
https://lambdalabs.com/service/gpu-cloud/pricing

[Latzi]

Thanks Glenn. For now I was able to train a model with ~100k images (640 px) using AWS . $6.18 /h took around 4 days just over $1k . I got what I wanted so hopefully next models won't require these kind of numbers :-) . I basically stripped 280k images from the COCO 2017 dataset by eliminating images with classes I didn't need and adding my own images to the effect of 20% of total. Trained and the training metrics are very similar to the original model but the model performs better in the real life having the images specific to the application embedded in the model .

[kalenmike]

@Latzi We have no limits on the size for an individual dataset, currently the Free plan has a storage limit of 100GB. The processing can take some time for larger datasets. I will take a look into the logs and see if there are any issues recently.

[kalenmike]

@Latzi I found an issue in generating the previews that is causing a slow down on the dataset processing. We will look into getting this resolved. Your dataset should now be processed. Please let me know if there is anything else we can help with.

[Latzi]

Hi Mike. Yes I am training the model using the dataset. It works okay but looking at the training speed it will take some 18 days ! There is no way Colab will keep my instance alive for that long. Is there an other way that you know of, where I could train a model with 100k images ? Of course I'd be happy to pay for that and for the use of resources. I am doing a training session with AWS right now but that appears painfully slow as well. Please let me know, if you know, what other resources would eb available ? Thanks again :-)

[Denizzje]

I understand the need for more powerful hardware and I was hoping that you could point me towards some options ? Where I can find such services apart from Google AUTO ML or AWS ? The AWS looks like it is able to handle the training and it will take around 3.5 days to finish the model. Also the price is only $6.18 / h so not too bad. Do you know of any other options ? I have a few more models to train, happy to pay and I am trying to avoid waiting for 4 days :-) . Thanks again and I can hardly wait to test your cloud training solution when it becomes available.

Check out datacrunch.io . 1x V100 for $1, 1x A100 for $2.25. Also available in 2x, 4x and 8x for V100 and A100. The A100 is the full sized 80GB SXM beast GPU and blows everything out of the water from other cloud providers I have tried/seen.

[glenn-jocher]

@Latzi awesome, nice work! Hopefully we can get cloud training deployed around October at prices significantly below the large cloud providers.