ultralytics_stream_example do not work about supervision

[SkalskiP]

Hi @tgbaoo 👋🏻 I just run ultralytics_stream_example.py and it runs without any issue. This is how I run it:

python ultralytics_stream_example.py \
--zone_configuration_path "data/checkout/config.json" \
--rtsp_url "rtsp://localhost:8554/live0.stream" \
--weights "yolov8x.pt" \
--device "mps" \
--classes 0 \
--confidence_threshold 0.5 \
--iou_threshold 0.5

It's true that deprecation warnings appear for every frame, which can be quite annoying, but the code still functions correctly.

SupervisionWarnings: __call__ is deprecated: `FPSMonitor.__call__` is deprecated and will be removed in `supervision-0.22.0`. Use `FPSMonitor.fps` instead.

[SkalskiP]

Hi @tgbaoo 👋🏻 you discovered exactly what I did over the past 30 minutes. Looks like this issue is related to inference 0.9.22 release that introduced breaking changes in the API. I just merged the PR that pinned the version of inference to to maximum 0.9.21 so if you delete your Python environment and and install once again from current version of develop branch you should be fine.

You can probably also:

pip uninstall inference
pip install `inference==0.9.21`

I'm closing the issue, but if you will have more questions let us know.

[LinasKo]

Hi @tgbaoo 👋

Thank you for reporting the issue!

I see you're willing to contribute a PR to help us out. I'll assign the issue to you - let me know if you need any help 😉

[tgbaoo]

@LinasKo Please check my bug and help me with the second bug that need to be fixed!

[LinasKo]

There's a little bit in your explanation that I don't understand. Could you please share a Colab showing where the error appears?

[tgbaoo]

Sure @LinasKo, here is more information about my problem:
First bug that I mention above should be fixed like this:

    def inference_callback(frame: VideoFrame) -> sv.Detections:
        results = model(frame[0].image, verbose=True, conf=confidence, device=device)
        results_0 = results[0]
        print("result: ", results_0)
        detections = sv.Detections.from_ultralytics(results_0).with_nms(threshold=iou)
        print("[ultralytics] detections: ", detections)
        print("[ultralytics] detections class_id attr: ", detections.class_id)

        return detections

• The second bug I think come from this part of code:

    def on_prediction(self, detections: sv.Detections, frame: VideoFrame) -> None:
        self.fps_monitor.tick()
        fps = self.fps_monitor.fps
        detections = detections[find_in_list(detections.class_id, self.classes)]
        detections = self.tracker.update_with_detections(detections)

full code from ultralytics_stream_example.py for more context:

class CustomSink:
    def __init__(self, zone_configuration_path: str, classes: List[int]):
        self.classes = [classes] if isinstance(classes, int) else classes
        self.tracker = sv.ByteTrack(minimum_matching_threshold=0.8)
        self.fps_monitor = sv.FPSMonitor()
        self.polygons = load_zones_config(file_path=zone_configuration_path)
        self.timers = [ClockBasedTimer() for _ in self.polygons]
        self.zones = [
            sv.PolygonZone(
                polygon=polygon,
                triggering_anchors=(sv.Position.CENTER,),
            )
            for polygon in self.polygons
        ]

    def on_prediction(self, detections: sv.Detections, frame: VideoFrame) -> None:
        self.fps_monitor.tick()
        fps = self.fps_monitor.fps

        detections = detections[find_in_list(detections.class_id, self.classes)]
        detections = self.tracker.update_with_detections(detections)
        annotated_frame = frame.image.copy()
        annotated_frame = sv.draw_text(
            scene=annotated_frame,
            text=f"{fps:.1f}",
            text_anchor=sv.Point(40, 30),
            background_color=sv.Color.from_hex("#A351FB"),
            text_color=sv.Color.from_hex("#000000"),
        )
        for idx, zone in enumerate(self.zones):
            annotated_frame = sv.draw_polygon(
                scene=annotated_frame, polygon=zone.polygon, color=COLORS.by_idx(idx)
            )
            detections_in_zone = detections[zone.trigger(detections)]
            time_in_zone = self.timers[idx].tick(detections_in_zone)
            custom_color_lookup = np.full(detections_in_zone.class_id.shape, idx)
            annotated_frame = COLOR_ANNOTATOR.annotate(
                scene=annotated_frame,
                detections=detections_in_zone,
                custom_color_lookup=custom_color_lookup,
            )
            labels = [
                f"#{tracker_id} {int(time // 60):02d}:{int(time % 60):02d}"
                for tracker_id, time in zip(detections_in_zone.tracker_id, time_in_zone)
            ]
            annotated_frame = LABEL_ANNOTATOR.annotate(
                scene=annotated_frame,
                detections=detections_in_zone,
                labels=labels,
                custom_color_lookup=custom_color_lookup,
            )
        cv2.imshow("Processed Video", annotated_frame)
        cv2.waitKey(1)
def main(
    rtsp_url: str,
    zone_configuration_path: str,
    weights: str,
    device: str,
    confidence: float,
    iou: float,
    classes: List[int],
) -> None:
    model = YOLO(weights)
    def inference_callback(frame: VideoFrame) -> sv.Detections:
        results = model(frame[0].image, verbose=True, conf=confidence, device=device)
        results_0 = results[0]
        print("result: ", results_0)
        detections = sv.Detections.from_ultralytics(results_0).with_nms(threshold=iou)
        print("[ultralytics] detections: ", detections)
        print("[ultralytics] detections class_id attr: ", detections.class_id)
        return detections
    sink = CustomSink(zone_configuration_path=zone_configuration_path, classes=classes)
    pipeline = InferencePipeline.init_with_custom_logic(
        video_reference=rtsp_url,
        on_video_frame=inference_callback,
        on_prediction=sink.on_prediction,
    )

    pipeline.start()
    try:
        pipeline.join()
    except KeyboardInterrupt:
        pipeline.terminate()

the log I get when run the code:

SupervisionWarnings: __call__ is deprecated: FPSMonitor.__call__ is deprecated and will be removed in supervision-0.22.0. Use FPSMonitor.fps instead.
0: 384x640 4 persons, 1 truck, 1 handbag, 1 tv, 942.1ms
Speed: 8.0ms preprocess, 942.1ms inference, 8.5ms postprocess per image at shape (1, 3, 384, 640)
result:  ultralytics.engine.results.Results object with attributes:
boxes: ultralytics.engine.results.Boxes object
keypoints: None
masks: None
names: {0: 'person', 1: 'bicycle', 2: 'car', 3: 'motorcycle', 4: 'airplane', 5: 'bus', 6: 'train', 7: 'truck', 8: 'boat', 9: 'traffic light', 10: 'fire hydrant', 11: 'stop sign', 12: 'parking meter', 13: 'bench', 14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe', 24: 'backpack', 25: 'umbrella', 26: 'handbag', 27: 'tie', 28: 'suitcase', 29: 'frisbee', 30: 'skis', 31: 'snowboard', 32: 'sports ball', 33: 'kite', 34: 'baseball bat', 35: 'baseball glove', 36: 'skateboard', 37: 'surfboard', 38: 'tennis racket', 39: 'bottle', 40: 'wine glass', 41: 'cup', 42: 'fork', 43: 'knife', 44: 'spoon', 45: 'bowl', 46: 'banana', 47: 'apple', 48: 'sandwich', 49: 'orange', 50: 'broccoli', 51: 'carrot', 52: 'hot dog', 53: 'pizza', 54: 'donut', 55: 'cake', 56: 'chair', 57: 'couch', 58: 'potted plant', 59: 'bed', 60: 'dining table', 61: 'toilet', 62: 'tv', 63: 'laptop', 64: 'mouse', 65: 'remote', 66: 'keyboard', 67: 'cell phone', 68: 'microwave', 69: 'oven', 70: 'toaster', 71: 'sink', 72: 'refrigerator', 73: 'book', 74: 'clock', 75: 'vase', 76: 'scissors', 77: 'teddy bear', 78: 'hair drier', 79: 'toothbrush'}
obb: None
orig_img: array([[[ 91,  91,  91],
        [111, 111, 111],
        [115, 115, 115],
        ...,
        [135, 135, 135],
        [125, 125, 125],
        [ 53,  53,  53]],
       [[ 85,  85,  85],
        [103, 103, 103],
        [110, 110, 110],
        ...,
        [135, 135, 135],
        [125, 125, 125],
        [ 53,  53,  53]],
       [[ 76,  76,  76],
        [ 91,  91,  91],
        [101, 101, 101],
        ...,
        [135, 135, 135],
        [125, 125, 125],
        [ 53,  53,  53]],
       ...,
       [[ 89, 102,  96],
        [ 89, 102,  96],
        [ 89, 102,  96],
        ...,
        [233, 194, 180],
        [233, 194, 180],
        [109,  70,  56]],
       [[ 89, 102,  96],
        [ 89, 102,  96],
        [ 89, 102,  96],
        ...,
        [233, 194, 180],
        [233, 194, 180],
        [109,  70,  56]],
       [[ 89, 102,  96],
        [ 89, 102,  96],
        [ 89, 102,  96],
        ...,
        [233, 194, 180],
        [233, 194, 180],
        [109,  70,  56]]], dtype=uint8)
orig_shape: (720, 1280)
path: 'image0.jpg'
probs: None
save_dir: 'runs\\detect\\predict'
speed: {'preprocess': 8.001565933227539, 'inference': 942.1312808990479, 'postprocess': 8.53729248046875}
[ultralytics] detections:  Detections(xyxy=array([[     695.06,      61.616,      795.59,      310.53],
       [     878.28,      132.54,      959.91,      369.58],
       [     1035.3,      27.613,      1105.2,      191.51],
       [     59.808,      385.06,      335.94,         643],
       [     519.25,      150.87,      593.72,      385.25],
       [     1099.2,      348.75,      1154.4,      431.54],
       [     612.75,      110.03,      695.27,       217.1]], dtype=float32), mask=None, confidence=array([    0.89195,     0.82298,     0.69308,     0.60449,      0.5467,     0.38043,     0.33792], dtype=float32), class_id=array([ 0,  0,  0,  7,  0, 26, 62]), tracker_id=None, data={'class_name': array(['person', 'person', 'person', 'truck', 'person', 'handbag', 'tv'], dtype='<U7')})
[ultralytics] detections class_id attr:  [ 0  0  0  7  0 26 62]

[04/29/24 18:45:46] WARNING  Error in results dispatching - 'tuple' object has no attribute 'class_id'                                            inference_pipeline.py:892
SupervisionWarnings: __call__ is deprecated: FPSMonitor.__call__ is deprecated and will be removed in supervision-0.22.0. Use FPSMonitor.fps instead.

• One more thing is this warning below just keep looping logging to my terminal, have a llitle bit annoying, if I comment that decorated on my virtual env, will it disappear?
  SupervisionWarnings: __call__ is deprecated: FPSMonitor.__call__ is deprecated and will be removed in supervision-0.22.0. Use FPSMonitor.fps instead. (DONE)

[tgbaoo]

Dear @LinasKo and @SkalskiP Maybe the problem come from the new version of roboflow or I clone the new PR?

I run on window with CPU, the CLI same as you setup.

I have check the roboflow inference documentation and see different implementation of custom sink with before and after 0.9.18 version.

Sorry because I am newbie on computer vision, from your expertise and experience, which direction should I investigate for debugging this code?

Thanks for you passionate and for your patient to help me out.

[tgbaoo]

I must give a big thanks to @LinasKo and @SkalskiP a lot, you guys are so 🔥🔥🔥 You guys just saved My day 🚀🚀🚀