Error Code 2: Internal Error (Assertion cublasStatus == CUBLAS_STATUS_SUCCESS failed. ) about ultralytics

[github-actions]

👋 Hello @wanghuajia, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

Hello @wanghuajia,

Thank you for bringing this issue to our attention. The error you're encountering seems to be related to TensorRT and cuBLAS during the export process.

To assist you better, could you please provide a minimal reproducible code example? This will help us replicate the issue on our end and investigate a solution. You can find guidelines on creating a minimal reproducible example here: Minimum Reproducible Example.

Additionally, please ensure that you are using the latest versions of torch and ultralytics. You can update your packages using the following commands:

pip install --upgrade torch ultralytics

Your suggested workaround involving the set_tactic_sources method is interesting. If you have already tested this and it resolves the issue, feel free to share the modified code snippet. However, we still need to reproduce the issue to ensure a comprehensive fix.

Looking forward to your response!

[wanghuajia]

Hello @wanghuajia,

Thank you for bringing this issue to our attention. The error you're encountering seems to be related to TensorRT and cuBLAS during the export process.

To assist you better, could you please provide a minimal reproducible code example? This will help us replicate the issue on our end and investigate a solution. You can find guidelines on creating a minimal reproducible example here: Minimum Reproducible Example.

Additionally, please ensure that you are using the latest versions of torch and ultralytics. You can update your packages using the following commands:

pip install --upgrade torch ultralytics

Your suggested workaround involving the set_tactic_sources method is interesting. If you have already tested this and it resolves the issue, feel free to share the modified code snippet. However, we still need to reproduce the issue to ensure a comprehensive fix.

Looking forward to your response!

I have solved the problem. The problem is mainly caused by cuda10.2.
The solution is as follows
A workaround is adding some code after
/home/huajia/code/ultralytics/ultralytics/engine/exporter.py config = builder.create_builder_config() config.set_tactic_sources(1<<int(trt.TacticSource.CUBLAS)) # add the code
reference the linkhttps://github.com/open-mmlab/mmdeploy/issues/261

[glenn-jocher]

Hello @wanghuajia,

Thank you for the update and for sharing your solution! It's great to hear that you were able to resolve the issue by adjusting the TensorRT configuration.

For others who might encounter a similar problem, your workaround of adding config.set_tactic_sources(1<<int(trt.TacticSource.CUBLAS)) after config = builder.create_builder_config() in the exporter.py file is valuable. This adjustment helps ensure that the cuBLAS tactic source is correctly set, which can address compatibility issues with CUDA 10.2.

If you have any further questions or run into other issues, feel free to reach out. We're here to help! 😊