How to run yolov8 on OpenVino? about ultralytics

[glenn-jocher]

@akashAD98 you don't need any external notebook or custom script to run OpenVINO models, you run them with the ultralytics package just like any other export format, i.e.:

CLI

yolo predict model=yolov8n_openvino_model/
yolo predict model=yolov8n.onnx
yolo predict model=yolov8n.engine
# ... etc.

Python

from ultraltyics import YOLO

model = YOLO('yolov8n_openvino_model/')
results = model(img)

See https://docs.ultralytics.com/modes/export for details

[glenn-jocher]

@achuntelolan hi there! 😊 To create an inference pipeline using your YOLOv8 model trained and exported for OpenVINO, you'll first need to ensure you have OpenVINO installed and set up. Then, you can use the OpenVINO Core API to load your model and the InferRequest API for inference. Here's a quick example to get you started:

from openvino.runtime import Core

# Initialize OpenVINO Core
core = Core()

# Read the network and weights from the model file
model_path = 'path/to/your/yolov8_openvino_model.xml'  # Path to .xml file
model = core.read_model(model_path)

# Compile the model for a specific device
compiled_model = core.compile_model(model=model, device_name="CPU")

# Create an infer request
infer_request = compiled_model.create_infer_request()

# Prepare your input data (adjust the shape according to your model's needs)
input_data = ...  # Your input data here

# Perform inference
results = infer_request.infer(inputs={0: input_data})

# Process results
...

Regarding loading the YAML file, it's typically used for configuration during training with YOLOv8. For inference with OpenVINO, you usually don't need it since the model architecture and weights are already encapsulated in the exported .xml and .bin files.

For more detailed examples and guides, the OpenVINO documentation is a fantastic resource. Also, keep an eye on the Ultralytics GitHub for any updates or example projects. Happy coding! 🚀

[dhaval-zala]

Thanks

I run yolov8n int8 IR converted model using this notebook, it consume 40% of my CPU on 10 FPS live stream. But if I restrict thread using taskset command it consume only 8-10% CPU on the same stream. So, why it is required to restrict CPU and how I can fix this?

[adrianboguszewski]

I see. We don't have a simple script for that, but you can look here for the simpler solution.

[glenn-jocher]

@akashAD98 Yes, you can customize the YOLO code to add features such as setting the number of threads or tuning other parameters. One way to do this is to modify the YOLO code directly and add functions that allow you to do this customization, or you could create a new class that inherits from the YOLO class and add the new features there. Just be careful when you modify the code, because it might affect other parts of the YOLO codebase. The safer way to add these features is by creating a wrapper function that loads the OpenVINO model and then sets the desired parameters before calling the YOLO function with the loaded model.

[glenn-jocher]

@achuntelolan you're welcome! If you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[adrianboguszewski]

@achuntelolan OV 2021 is a very old version. Any chance to update it to something newer? I think it may resolve your issue.

[adrianboguszewski]

Hey @dhaval-zala-aivid, please try this notebook: https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/230-yolov8-optimization/230-yolov8-optimization.ipynb

[adrianboguszewski]

@yury-gorbachev, any idea who can help with that?

[yury-gorbachev]

@dmitry-gorokhov or @wangleis can you guys look at this question?

[wangleis]

@dhaval-zala-aivid @dhaval-zala Could you please share the full command which restrict thread using taskset command?

[wangleis]

@dhaval-zala-aivid @dhaval-zala Could you please share log of lscpu and lscpu -e as well?

[dhaval-zala]

Im using taskset -c 0 python scripy.py to restrict CPU

lscpu and lscpu -e

[wangleis]

@dhaval-zala-aivid @dhaval-zala OpenVINO uses all CPU resources provided for application for inference. But when there is not enough input, the CPU will idle.

yolo_openvino_demo.py you used loaded network to OpenVINO with 2 infer request. The following benchmark app command also loads network to OpenVINO with 2 infer request. Run this command with yolov8n int8 IR converted by 230-yolov8-optimization.ipynb on 4 cores 8 CPUs platform, the throughput performance is 20.31 FPS.

• benchmark_app -d CPU -m openvino_notebooks/notebooks/230-yolov8-optimization/yolov8n_openvino_model/yolov8n.xml -shape [1,3,640,640] -nireq 2

If run benchmark app in throughput mode as below comments, the throughput performance can achieve 30.04 FPS.

• benchmark_app -d CPU -m openvino_notebooks/notebooks/230-yolov8-optimization/yolov8n_openvino_model/yolov8n.xml -shape [1,3,640,640] -hint throughput

So 10 FPS live stream cannot provide enough inputs in you case. Please try the two benchmark app commands, all CPU resource on you platform will be used.

[HENNESSYxie]

Hey @dhaval-zala-aivid, please try this notebook: https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/230-yolov8-optimization/230-yolov8-optimization.ipynb

I have tried this notebook for inference my model, but I'm recieving error:
`RuntimeError Traceback (most recent call last)
in
32
33 input_image = np.array(Image.open(IMAGE_PATH))
---> 34 detections = detect(input_image, det_compiled_model)[0]
35 image_with_boxes = draw_results(detections, input_image, label_map)
36

2 frames
/usr/local/lib/python3.8/dist-packages/ultralytics/yolo/utils/ops.py in non_max_suppression(prediction, conf_thres, iou_thres, classes, agnostic, multi_label, labels, max_det, nc, max_time_img, max_nms, max_wh)
198
199 t = time.time()
--> 200 output = [torch.zeros((0, 6 + nm), device=prediction.device)] * bs
201 for xi, x in enumerate(prediction): # image index, image inference
202 # Apply constraints

RuntimeError: Trying to create tensor with negative dimension -73: [0, -73]`

[eaidova]

@HENNESSYxie if I correctly understand, does your model trained on a different dataset? there is a parameter inside detect function, when calling non_maximum_suppression, nc - it is responsible for a number of classes known by the model, please try to modify it according to your model supported a number of classes

[HENNESSYxie]

@HENNESSYxie if I correctly understand, does your model trained on a different dataset? there is a parameter inside detect function, when calling non_maximum_suppression, nc - it is responsible for a number of classes known by the model, please try to modify it according to your model supported a number of classes

Its worked for me. Thanks!

[akashAD98]

@wangleis @adrianboguszewski is there any simple script to run yolov8 openvino model?
the notebook which you mentioned is very big & in order to run file need to run all notebook.

model = YOLO("yolov8n.pt")
start_time = time.time()
src='/content/inputvideo.mp4'
results = model.track(source=src, tracker="botsort.yaml", verbose=False, save=True)

i just want to use openvino weights insted of yolov8.pt , its possible to get that script ?

something like

from openvino.runtime import Core
import collections

core = Core()
# read converted model
#pat of object detector
model = core.read_model("object_detector/best_weapons22_int8_IRnew.xml")
# load model on CPU device
model = core.compile_model(model, 'CPU')

[akashAD98]

i tested with tracker & this is the speed im getting, ill try yolov8s_int format for getting more fps.
@glenn-jocher thanks a lot , you're always supporting & helping.

yolov8n.xml --> Speed: 0.8ms preprocess, 130.6ms inference, 1.2ms postprocess per image at shape (1, 3, 640, 640)
yolov8.pt --> Speed: 0.6ms preprocess, 130.7ms inference, 1.0ms postprocess per image at shape (1, 3, 640, 640)
yolov8s.xml. --> Speed: 1.0ms preprocess, 396.0ms inference, 1.3ms postprocess per image at shape (1, 3, 640, 640
yolov8s.pt -->Speed: 0.5ms preprocess, 416.1ms inference, 1.1ms postprocess per image at shape (1, 3, 640, 640)

[akashAD98]

#1735

[glenn-jocher]

@akashAD98 OpenVINO should show speedups on Intel CPUs, i.e. if you look at the CI benchmarks here. On other CPUs ONNX may be faster:

[akashAD98]

@glenn-jocher its fp32 openvino model or int8.xml model?
or if its fp32 model, is there any way to convert it into int8 format?

[akashAD98]

@glenn-jocher when i didi the fps check this is how im getting, i tried it on 16 core macine & yolov8s.pt has higher fps than openvino yolov8s.xml file, may i know why its like this?
also yolov8.pt models using 384*640 size during predictions -may i know why its like this?

[glenn-jocher]

@akashAD98 It's curious that you are getting better results with yolov8s.pt than with openvino yolov8s.xml on an Intel CPU, because the OpenVINO tools are optimized specifically for Intel hardware. However, there may be other factors that affect the results, such as the size of the model, the preprocessing, or the batch size.

Regarding your second question, most YOLO models do not require input images that are multiples of 32, but instead accept any image size that is divisible by two. 384x640 is probably used here as an arbitrary image size that gives good results.

[akashAD98]

@adrianboguszewski is there any way to improve fps using yolo code & openvino weights? like passing the number of threads , asynchronous queue or any other methods which we can integrate with current Yolo code?

[adrianboguszewski]

Yes. We created a new notebook showing how to improve the performance in OpenVINO. It hasn't been merged yet, but you can see some tricks here

[glenn-jocher]

@akashAD98 Yes, there are several ways to improve the performance of OpenVINO models. One of them is to use number of threads, i.e. you can try to set the num_threads parameter to optimize your hardware. Another way is to use asynchronous inference, which allows GPU to work overlap with CPU. Yet another method is to use OpenVINO's own performance testing tools, such as the benchmark_app tool, in order to optimize your models.

[akashAD98]

@glenn-jocher i want to add this code , is there nay way to customise yolo code?
im directlyy loading openvino modelmbut i want to add few parameters to it,

from ultraltyics import YOLO
model = YOLO('yolov8n_openvino_model/')
results = model(img)

below are the some improvment things i want to try

num_cores = os.cpu_count()
ov_cpu_config_model = core.compile_model(ov_model, device_name="CPU", config={"INFERENCE_NUM_THREADS": num_cores})

core = Core()
model = core.read_model("object_detector/yolv8.xml)
setting = {"INFERENCE_NUM_THREADS":"16", "AFFINITY":"NUMA"}
model = core.compile_model(model=model, device_name="CPU", config=setting)

[akashAD98]

@glenn-jocher i was using int8_format of openvino weight . i compaired fps it with yolov8_openvino weight & im exactly getting same fps, isexport='openvino' bydefault converting .pt weights into int8 format?

also in the export of openvino model there is metadata.yaml file is generating
even if i pass batch=10,its not reflect any changes ,getting exactly same fps

[glenn-jocher]

@akashAD98 The export parameter in the YOLO class specifies how the model should be exported. If you set it to openvino, the YOLO class will convert the model to OpenVINO format, and by default, it will use the int8 data format. However, you can change this behavior by specifying the precision parameter when exporting the model. The metadata.yaml file is used to store information about the exported model, such as its input and output shapes, batch size, and precision. If you're not seeing any changes in FPS when changing the batch size, it could be because the model is being executed with the same set of inputs. You may need to modify the input data to reflect the new batch size.

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[achuntelolan]

i need to create a separate inference pipeline for object prediction using the model i trained using yolov8 in open vino so how i import the architecture and how i load the yaml file extra can anyone share me some example blogs or tell me how

[achuntelolan]

Ok thank you @glenn-jocher

[achuntelolan]

Hi is Yolo v8 openvino model will work with openvio version 2022 and below and Python 3.6? because my current project is running on this configuration that's why, can anyone please help me?

[glenn-jocher]

@achuntelolan hello! 👋 YOLOv8 models exported for OpenVINO should be compatible with OpenVINO 2022 versions and work fine with Python 3.6. However, it's always good to test with your specific setup. If you encounter any issues, make sure to have the latest version of the YOLOv8 repository and consider updating OpenVINO if possible. Here's a quick snippet on how to load and use the model:

from openvino.runtime import Core

core = Core()
model = core.read_model(model="path/to/your/model.xml")
compiled_model = core.compile_model(model=model, device_name="CPU")
# Now you're ready to make predictions with `compiled_model`!

If you run into any specific errors, feel free to share them here! 😊

[achuntelolan]

hii @glenn-jocher while i trying to load the openvino model of yolov8 I am getting error like this I am using open vino 2021 version:
Traceback (most recent call last):
File "yolo-old-v8.py", line 199, in
start_function()
File "/home/anoop/AnoopAJ/ML/new_code_workshop/YOLOV8/yolov4/lib/python3.8/site-packages/memory_profiler.py", line 1188, in wrapper
val = prof(func)(*args, **kwargs)
File "/home/anoop/AnoopAJ/ML/new_code_workshop/YOLOV8/yolov4/lib/python3.8/site-packages/memory_profiler.py", line 761, in f
return func(*args, **kwds)
File "yolo-old-v8.py", line 132, in start_function
yolo = predict(input_image_size,batch_size,thresh, common_path)
File "yolo-old-v8.py", line 16, in init
self.yolo = YOLO(inp_size=inp_size,
File "/home/anoop/AnoopAJ/ML/new_code_workshop/YOLOV8/yolov8.py", line 71, in init
net = ie.read_network(model=model_xml,weights=model_bin)
File "ie_api.pyx", line 368, in openvino.inference_engine.ie_api.IECore.read_network
File "ie_api.pyx", line 411, in openvino.inference_engine.ie_api.IECore.read_network
RuntimeError: Check 'false' failed at src/frontends/common/src/frontend.cpp:53:
Converting input model
Cannot create Interpolate layer /model.10/Resize id:164 from unsupported opset: opset11

[achuntelolan]

Hii @adrianboguszewski updating to a newer version is a little difficult because I am running this project individually in 88 systems that are in different locations in India so updating the version is the last way before we try to resolve using the older version

[adrianboguszewski]

Could you report your bug here: https://github.com/openvinotoolkit/openvino/issues? I think our developers will be able to help :)

[achuntelolan]

Hii when i trying to run yolo v8 on openvino version 2022.2.0 with python 3.6 getting this error:
Check 'false' failed at frontends/common/src/frontend.cpp:54:
Converting input model

[adrianboguszewski]

Did you convert the model to OV with 2022.2 as well? The best way is to convert the model with the version you want to use for inference

[achuntelolan]

how i can ?

[adrianboguszewski]

Have you tried this?

from ultralytics import YOLO

# Load a YOLOv8n PyTorch model
model = YOLO('yolov8n.pt')

# Export the model
model.export(format='openvino')  # creates 'yolov8n_openvino_model/'

# Load the exported OpenVINO model
ov_model = YOLO('yolov8n_openvino_model/')

# Run inference
results = ov_model('https://ultralytics.com/images/bus.jpg')

[achuntelolan]

This is just exporting the pytorch model to the openvino model right, this I have done I need how to convert to the openvino 2022 version supported model

[adrianboguszewski]

You just need to run the code above, when you have OpenVINO 2022 installed.