zfturbo / weighted-boxes-fusion

Set of methods to ensemble boxes from different object detection models, including implementation of "Weighted boxes fusion (WBF)" method.

Weighted boxes fusion

Repository contains Python implementation of several methods for ensembling boxes from object detection models:

• Non-maximum Suppression (NMS)
• Soft-NMS [1]
• Non-maximum weighted (NMW) [2]
• Weighted boxes fusion (WBF) [3] - new method which gives better results comparing to others

Requirements

Python 3.*, Numpy, Numba

Installation

pip install ensemble-boxes

Usage examples

Coordinates for boxes expected to be normalized e.g in range [0; 1]. Order: x1, y1, x2, y2.

Example of boxes ensembling for 2 models below.

• First model predicts 5 boxes, second model predicts 4 boxes.
• Confidence scores for each box model 1: [0.9, 0.8, 0.2, 0.4, 0.7]
• Confidence scores for each box model 2: [0.5, 0.8, 0.7, 0.3]
• Labels (classes) for each box model 1: [0, 1, 0, 1, 1]
• Labels (classes) for each box model 2: [1, 1, 1, 0]
• We set weight for 1st model to be 2, and weight for second model to be 1.
• We set intersection over union for boxes to be match: iou_thr = 0.5
• We skip boxes with confidence lower than skip_box_thr = 0.0001

from ensemble_boxes import *

boxes_list = [[
    [0.00, 0.51, 0.81, 0.91],
    [0.10, 0.31, 0.71, 0.61],
    [0.01, 0.32, 0.83, 0.93],
    [0.02, 0.53, 0.11, 0.94],
    [0.03, 0.24, 0.12, 0.35],
],[
    [0.04, 0.56, 0.84, 0.92],
    [0.12, 0.33, 0.72, 0.64],
    [0.38, 0.66, 0.79, 0.95],
    [0.08, 0.49, 0.21, 0.89],
]]
scores_list = [[0.9, 0.8, 0.2, 0.4, 0.7], [0.5, 0.8, 0.7, 0.3]]
labels_list = [[0, 1, 0, 1, 1], [1, 1, 1, 0]]
weights = [2, 1]

iou_thr = 0.5
skip_box_thr = 0.0001
sigma = 0.1

boxes, scores, labels = nms(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr)
boxes, scores, labels = soft_nms(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr, sigma=sigma, thresh=skip_box_thr)
boxes, scores, labels = non_maximum_weighted(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr, skip_box_thr=skip_box_thr)
boxes, scores, labels = weighted_boxes_fusion(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr, skip_box_thr=skip_box_thr)

Single model

If you need to apply NMS or any other method to single model predictions you can call function like that:

from ensemble_boxes import *
# Merge boxes for single model predictions
boxes, scores, labels = weighted_boxes_fusion([boxes_list], [scores_list], [labels_list], weights=None, method=method, iou_thr=iou_thr, thresh=thresh)

More examples can be found in example.py

3D version

There is support for 3D boxes in WBF method with weighted_boxes_fusion_3d function. Check example of usage in example_3d.py

1D version

There is support for 1D line segments in WBF method with weighted_boxes_fusion_1d function. Check example of usage in example_1d.py. It was reported that 1D variant can be useful in Named-entity recognition (NER) type of tasks for Natural Language Processing (NLP) problems. Check discussion here.

Benchmarks

• Benchmark for Open Images Dataset (5 models)
• Benchmark for COCO Dataset (10 models)
• Benchmark for NLP Dataset (10 models) - example for one-dimensional WBF variant

Description of WBF method and citation

• https://arxiv.org/abs/1910.13302 (updated: 2020.08)
• https://authors.elsevier.com/c/1ca0dxnVK3cWY

If you find this code useful please cite:

@article{solovyev2021weighted,
  title={Weighted boxes fusion: Ensembling boxes from different object detection models},
  author={Solovyev, Roman and Wang, Weimin and Gabruseva, Tatiana},
  journal={Image and Vision Computing},
  pages={1-6},
  year={2021},
  publisher={Elsevier}
}

weighted-boxes-fusion's Issues

weighted boxes fusion on single model

hello, today i found a repo: https://github.com/kentaroy47/ODA-Object-Detection-ttA
this repo also use the wbf, you can find the code in adach/oda.py/line 192 class TTAWrapper. you can see the example.ipynb in that repo, i found oda use the wbf just only one model. I remember that you say that wbf on single model, the performance will drop. how do i choose the nms option in class TTAWrapper in oda? just nms='nms' not nms='wbf', thanks?

How to use WBF in mmdetection?

How to use WBF in mmdetection? Has anyone done it?

label list on SSD

Hello, I'm building an ensemble object detection model with two SSD head detectors.
In this implementation the labels of the predicted model are stored with the corresponding conf scores and the bbox coordinates.

In SSD, though, the model doesn't produce the corresponding label during the detection.

Is there any way to make sense this implementation when using SSD as the detector or does anyone have any input on this matter?
Looking forward for sharing!

Would you mind if you clarify the license?

Would you mind if you clarify the license?

Bbox is 'Nan' when weights have 0

Hello, thanks for your nice work!

Recently, I used your method in my own project, and the mAP is improved indeed. However, When I check the JSON format result, most of the bounding boxes are Nan.

The below are the parameters:
iou_thr=0.5, skip_box_thr=0.01, weights=[1, 0]

when weights have 0, there will be warnings:
RuntimeWarning: invalid value encountered in true_divide box[4:] /= conf
But there are also 0 in the benchmark you have given.

Could you give me some suggestions?
Looking forward to your reply, Thanks!

how to extract yolov5 features to add another deeplearining model

I want to hybrid another deeplearning model like ssd
after yolov5

how to extract yolov5 features

How to find the optimal hyper-parameters in the ensembling process

Hello, I would like to ask how to find the best parameters combination, such as the model importances, IOU threshold when performing the ensemble process? Now I got prediction results from different models and try to ensemble these predictions, but I cannot know how to find the optimal parameters. Any suggestions will be helpful!
Thank you in advance.

will WBF be better than nms in a single model?

Sometimes output boxes have zero area

run NMS on empty bounding box from one of ensembled models

Hi, I tried to ensemble 5 models. if there is no bounding box detected from one of 5 models, the nms gives error as follow :

ValueError                                Traceback (most recent call last)
Input In [39], in <cell line: 14>()
     12 lebar = dict_pred[img.split('\\')[-1].split('.')[0]]['image_width']
     13 tinggi = dict_pred[img.split('\\')[-1].split('.')[0]]['image_height']
---> 14 boxes, scores, labels = nms_method(bbox_list, conf_list, cat_list, method=2, weights=weights, iou_thr=0.3, sigma=0.05, thresh=0.001)

File ~\Anaconda3\envs\SiT\lib\site-packages\ensemble_boxes\ensemble_boxes_nms.py:187, in nms_method(boxes, scores, labels, method, iou_thr, sigma, thresh, weights)
    184             scores[i] = (np.array(scores[i]) * weights[i]) / weights.sum()
    186 # We concatenate everything
--> 187 boxes = np.concatenate(boxes)
    188 scores = np.concatenate(scores)
    189 labels = np.concatenate(labels)

File <__array_function__ internals>:5, in concatenate(*args, **kwargs)

ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 3 has 1 dimension(s)

If I used wbf method, it runs succesfully. is there any solution to run nms in this situation like wbf does. please advise

thank you
regards

the results come from two different models

I want to ask, if the results come from two different models, such as Yolo and frcnn, can I use this method?

wbf weights

Hello. Can you tell me how to assign different wbf weights to models?

What is the standard of normalization

Suppose I have the results obtained by the two models now, what standard should I adopt to normalize, so that the results after the fusion can be restored back.For example, I normalize the maximum and minimum values of all results of the two models. Is this correct?

I've found typo in your title!

Set of methods to ensemble boxes from object detection models, inlcuding implementation of "Weighted boxes fusion (WBF)" method.

inlcuding -> including

Thank you!

how to extract yolov5 features to add another deeplearining model

I want to hybrid another deeplearning model like ssd
after yolov5

how to extract yolov5 features

unable to send absolute coordinates

I wanted to send absolute coordinates as I don't have the width and height of the image.

Question about different scores of two different model

Two models with different structures get different confidencefor the same dataset. For example, most of the confidence obtained by the first model are more than 0.9, while most of the confidence of the second model are above 0.8. However, their final map is almost the same. How should I set the weight of WBF? Or how can I change the confidence of the two models?

Add ensemble boxes to conda-forge

We should add this package to conda-forge, it should be very simple since it is already in pypi so the grayskull recipe generator can be used (more here)

single model

the wbf can be used for only single model? If I only use the yolov5 to get the result, can i apply the wbf to refine the detection?

Compatibility for Instance Segmentation

I'm not actually suggesting to do wbf or nms on masks but I think those methods should be compatible with masks for instance segmentation. I haven't tried any methods in this repository for instance segmentation yet but I think it makes sense to get more refined masks with wbf and nms.

I'm currently doing this by adding masks argument to prepare_boxes and nms_method functions.

def prepare_boxes(boxes, scores, labels, masks=None):
    result_boxes = boxes.copy()

    cond = (result_boxes < 0)
    cond_sum = cond.astype(np.int32).sum()
    if cond_sum > 0:
        print('Warning. Fixed {} boxes coordinates < 0'.format(cond_sum))
        result_boxes[cond] = 0

    cond = (result_boxes > 1)
    cond_sum = cond.astype(np.int32).sum()
    if cond_sum > 0:
        print('Warning. Fixed {} boxes coordinates > 1. Check that your boxes was normalized at [0, 1]'.format(cond_sum))
        result_boxes[cond] = 1

    boxes1 = result_boxes.copy()
    result_boxes[:, 0] = np.min(boxes1[:, [0, 2]], axis=1)
    result_boxes[:, 2] = np.max(boxes1[:, [0, 2]], axis=1)
    result_boxes[:, 1] = np.min(boxes1[:, [1, 3]], axis=1)
    result_boxes[:, 3] = np.max(boxes1[:, [1, 3]], axis=1)

    area = (result_boxes[:, 2] - result_boxes[:, 0]) * (result_boxes[:, 3] - result_boxes[:, 1])
    cond = (area == 0)
    cond_sum = cond.astype(np.int32).sum()
    if cond_sum > 0:
        print('Warning. Removed {} boxes with zero area!'.format(cond_sum))
        result_boxes = result_boxes[area > 0]
        scores = scores[area > 0]
        labels = labels[area > 0]
        if masks is not None:
            masks = masks[area > 0]

    return result_boxes, scores, labels, masks

def nms_method(boxes, scores, labels, masks=None, method=3, iou_thr=0.5, sigma=0.5, thresh=0.001, weights=None):
    """
    :param boxes: list of boxes predictions from each model, each box is 4 numbers. 
    It has 3 dimensions (models_number, model_preds, 4)
    Order of boxes: x1, y1, x2, y2. We expect float normalized coordinates [0; 1] 
    :param scores: list of scores for each model 
    :param labels: list of labels for each model
    :param method: 1 - linear soft-NMS, 2 - gaussian soft-NMS, 3 - standard NMS
    :param iou_thr: IoU value for boxes to be a match 
    :param sigma: Sigma value for SoftNMS
    :param thresh: threshold for boxes to keep (important for SoftNMS)
    :param weights: list of weights for each model. Default: None, which means weight == 1 for each model

    :return: boxes: boxes coordinates (Order of boxes: x1, y1, x2, y2). 
    :return: scores: confidence scores
    :return: labels: boxes labels
    """

    # If weights are specified
    if weights is not None:
        if len(boxes) != len(weights):
            print('Incorrect number of weights: {}. Must be: {}. Skip it'.format(len(weights), len(boxes)))
        else:
            weights = np.array(weights)
            for i in range(len(weights)):
                scores[i] = (np.array(scores[i]) * weights[i]) / weights.sum()

    # We concatenate everything
    boxes = np.concatenate(boxes)
    scores = np.concatenate(scores)
    labels = np.concatenate(labels)
    if masks is not None:
        masks = np.concatenate(masks)

    # Fix coordinates and removed zero area boxes
    boxes, scores, labels, masks = prepare_boxes(boxes, scores, labels, masks)

    # Run NMS independently for each label
    unique_labels = np.unique(labels)
    final_boxes = []
    final_scores = []
    final_labels = []
    if masks is not None:
        final_masks = []

    for l in unique_labels:
        condition = (labels == l)
        boxes_by_label = boxes[condition]
        scores_by_label = scores[condition]
        labels_by_label = np.array([l] * len(boxes_by_label))
        if masks is not None:
            masks_by_label = masks[condition]

        if method != 3:
            keep = cpu_soft_nms_float(boxes_by_label.copy(), scores_by_label.copy(), Nt=iou_thr, sigma=sigma, thresh=thresh, method=method)
        else:
            # Use faster function
            keep = nms_float_fast(boxes_by_label, scores_by_label, thresh=iou_thr)

        final_boxes.append(boxes_by_label[keep])
        final_scores.append(scores_by_label[keep])
        final_labels.append(labels_by_label[keep])
        if masks is not None:
            final_masks.append(masks_by_label[keep])
    final_boxes = np.concatenate(final_boxes)
    final_scores = np.concatenate(final_scores)
    final_labels = np.concatenate(final_labels)
    final_masks = np.concatenate(final_masks)

    return final_boxes, final_scores, final_labels, final_masks

def nms(boxes, scores, labels, masks=None, iou_thr=0.5, weights=None):
    """
    Short call for standard NMS 
    
    :param boxes: 
    :param scores: 
    :param labels: 
    :param iou_thr: 
    :param weights: 
    :return: 
    """
    return nms_method(boxes, scores, labels, masks, method=3, iou_thr=iou_thr, weights=weights)


def soft_nms(boxes, scores, labels, masks=None, method=2, iou_thr=0.5, sigma=0.5, thresh=0.001, weights=None):
    """
    Short call for Soft-NMS
     
    :param boxes: 
    :param scores: 
    :param labels: 
    :param method: 
    :param iou_thr: 
    :param sigma: 
    :param thresh: 
    :param weights: 
    :return: 
    """
    return nms_method(boxes, scores, labels, masks, method=method, iou_thr=iou_thr, sigma=sigma, thresh=thresh, weights=weights)

I know it looks like error prone with this fast work around but do you think this feature deserves a pull request? @ZFTurbo

All Performance Metrics are 0

I wanna use this repo with mmdetection result.I edited Json and converted to CSV format.However I get 0 for mAP and Recall.
I can share my csv and ground_truth.json files via drive link.In addition you can see my detection output for model_1 below;

You can see json2csv script on below;

with open('fsaf_r50.csv', 'w', encoding='UTF8',newline='') as ff:
    writer = csv.writer(ff)
    writer.writerow(header)
    for i in range(0,len(data)):

            real_data=data[i]['bbox']

            
            real_data[2]=real_data[0]+real_data[2]
            real_data[3]=real_data[1]+real_data[3]

            normalized_arr = preprocessing.normalize([real_data])
            
            image_id=data[i]['image_id'][:]
            csv_data = [int(image_id), normalized_arr[0][0],normalized_arr[0][1],normalized_arr[0][2],normalized_arr[0][3], data[i]['score'],data[i]['category_id'],]
            writer.writerow(csv_data)

I think normalization process is wrong my code or ground_truth.json configuration is false.Could you help me ?
You can access all code with this URL:

https://github.com/ozanpkr/Weighted-Boxes-Fusion/tree/master/ozan

Could you normalize this bbox?
format=[x1,y1,width,height] !!!(x1,y1) refer to upper left
[148,186,72,96] =??

WBF on Polygon instead of Bounding Box

I've read this algorithm utilizes confidence scores of all proposed bounding boxes to constructs the averaged boxes. However, I'm interested to know, is it possible to utilize the algorithm with an arbitrarily number of polygons, such as in the scene text spotting problem? I hope you get my point. Any pointer? Thanks.

Implemetation on mobile phone

I am using this repo to ensemble multiple models but I want to convert to mobile application. Is there a way to convert to this method?

Why is coordinates for boxes expected to be normalized e.g in range [0; 1]?

Can it exceed the limit?Whether the limit has mathematical meaning?
I use WBF,but got worse results.So I wonder if that's the reason

when i run example_oid.py, the error occured "ModuleNotFoundError: No module named 'map_boxes'",how can I solve the error ? thanks

Incorrect allows_overflow=False mode

Example:

from ensemble_boxes import *

weigths = [0.2, 0.2, 0.2, 0.2, 0.2]
pred_boxes = []
pred_scores = []
pred_labels = []
for _ in range(5):
    pred_boxes.append([[0.    , 0.    , 0.0001, 0.0001]])
    pred_scores.append([1.])
    pred_labels.append([0])

pred_boxes, pred_scores, pred_labels = weighted_boxes_fusion(
    pred_boxes,
    pred_scores,
    pred_labels,
    weights=weigths,
    iou_thr=0.4,
    skip_box_thr=0.,
    allows_overflow=False
)
print(pred_scores)

Actual result: score [0.2]
Expected result: score [1]

Probably we need to change the line
weighted_boxes[i][1] = weighted_boxes[i][1] * min(weights.sum(), len(clustered_boxes)) / weights.sum()
->
weighted_boxes[i][1] = weighted_boxes[i][1] * min(len(weights), len(clustered_boxes)) / weights.sum()

does the 3d wbf support rotation?

most of the 3d object-detector need to predict a heading angle(yaw angle),but is seems that the 3d wbf don't use it
and another problem is the normalization,.by 2D case it is quite easy because we just need to divided by width and height,
but 3D detection are the coordinate in real world with respect to camera coordinate(not just image plane)

how to normalize bbox in WBF?

should I divided bbox by image width and height? thanks for your reply!

no module map_boxes

请问为什么老是显示没有这个库，安装了也不行

about yolov5

do you embeded the WBF into yolov5? I have the question about using WBF in Yolov5

How to normalize confidence?

How to normalize confidence? ?thanks for your reply!

Strange behavior of weighted_boxes_fusion

@ZFTurbo

boxes = [
    [[410, 464, 354, 410],
    [511, 89, 470, 32],
    [503, 213, 447, 166],
    [300, 444, 252, 391],
    [290, 184, 234, 133]],
    
    [[354, 412, 409, 463],
    [447, 165, 504, 212],
    [251, 392, 299, 444],
    [470, 35, 511, 90],
    [187, 316, 240, 378]],

    [[447, 166, 503, 213],
    [355, 412, 411, 464],
    [470, 33, 511, 88],
    [251, 391, 300, 444],
    [234, 132, 289, 184]],

    [[251, 391, 300, 445],
    [354, 412, 410, 463],
    [447, 166, 503, 212],
    [235, 134, 289, 184],
    [191, 316, 239, 380]],

    [[410, 465, 355, 412],
    [511, 88, 470, 33],
    [504, 213, 448, 167],
    [299, 444, 251, 392],
    [289, 185, 236, 133]]
]

scores = [
    [0.893,0.886,0.865,0.864,0.801],
    [0.915,0.881,0.873,0.852,0.844],
    [0.896, 0.895,0.860,0.855,0.801],
    [0.900,0.889,0.861,0.838,0.831],
    [0.897,0.877, 0.877,0.861,0.820],
]

labels = [
    [1, 1, 1, 1, 1],
    [1, 1, 1, 1, 1],
    [1, 1, 1, 1, 1],
    [1, 1, 1, 1, 1],
    [1, 1, 1, 1, 1]
]

wbf_boxes, wbf_scores, wbf_labels = ensemble_boxes.ensemble_boxes_wbf.weighted_boxes_fusion(boxes, scores, labels, weights=None, iou_thr=0.43, skip_box_thr=0.44)

wbf_boxes.shape
>>> (16, 4)

I think we see excess wbf_boxes, it is very strange. Picture with demonstration:

Python 3.7.6
numpy==1.18.1
ensemble_boxes==1.0.1

Building wheel for llvmlite ... error

hi there, maybe this is not a issue but an error that I faced when I install ensemble_boxes with the following command:

pip install ensemble_boxes

My environment is Jetson Xavier:

• Ubuntu:18.04 LTS
• ARMv8 Processor rev 0 (v8l) × 6
• NVIDIA Tegra Xavier (nvgpu)/integrated
• Conda 4.10.3

To reproduce:
1 create a conda environment named pytorch1
2 install pytorch via the following link https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-10-now-available/72048
In my case, I used the python 3.6 and pytorch v1.10.0
3 After successful installed use
'pip install ensemble_boxes'

4 No error if I use python 3.8 and with cpu version pytorch.

Error occurs:

(pytorch1) user@GTW-XNX-C1P0IUL1:~/Github/$ pip install ensemble_boxes
Collecting ensemble_boxes
Using cached ensemble_boxes-1.0.8-py3-none-any.whl (21 kB)
Collecting numba
Using cached numba-0.53.1-cp36-cp36m-linux_aarch64.whl
Requirement already satisfied: numpy in /home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages (from ensemble_boxes) (1.19.5)
Requirement already satisfied: pandas in /home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages (from ensemble_boxes) (1.1.5)
Collecting llvmlite<0.37,>=0.36.0rc1
Using cached llvmlite-0.36.0.tar.gz (126 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: setuptools in /home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages (from numba->ensemble_boxes) (59.5.0)
Requirement already satisfied: python-dateutil>=2.7.3 in /home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages (from pandas->ensemble_boxes) (2.8.2)
Requirement already satisfied: pytz>=2017.2 in /home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages (from pandas->ensemble_boxes) (2021.3)
Requirement already satisfied: six>=1.5 in /home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages (from python-dateutil>=2.7.3->pandas->ensemble_boxes) (1.16.0)
Building wheels for collected packages: llvmlite
Building wheel for llvmlite (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/user/anaconda3/envs/pytorch1/bin/python3.6 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/setup.py'"'"'; file='"'"'/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-lykc90s6
cwd: /tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/
Complete output (11 lines):
running bdist_wheel
/home/user/anaconda3/envs/pytorch1/bin/python3.6 /tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py
LLVM version... Traceback (most recent call last):
File "/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py", line 220, in
main()
File "/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py", line 210, in main
main_posix('linux', '.so')
File "/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py", line 134, in main_posix
raise RuntimeError(msg) from None
RuntimeError: Could not find a llvm-config binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help.
error: command '/home/user/anaconda3/envs/pytorch1/bin/python3.6' failed with exit status 1

ERROR: Failed building wheel for llvmlite
Running setup.py clean for llvmlite
Failed to build llvmlite
Installing collected packages: llvmlite, numba, ensemble-boxes
Running setup.py install for llvmlite ... error
ERROR: Command errored out with exit status 1:
command: /home/user/anaconda3/envs/pytorch1/bin/python3.6 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/setup.py'"'"'; file='"'"'/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-hho726u9/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/anaconda3/envs/pytorch1/include/python3.6m/llvmlite
cwd: /tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/
Complete output (16 lines):
running install
/home/user/anaconda3/envs/pytorch1/lib/python3.6/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
setuptools.SetuptoolsDeprecationWarning,
running build
got version from file /tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/llvmlite/_version.py {'version': '0.36.0', 'full': 'e6bb8d137d922bec8beeb01a237254778759becd'}
running build_ext
/home/user/anaconda3/envs/pytorch1/bin/python3.6 /tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py
LLVM version... Traceback (most recent call last):
File "/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py", line 220, in
main()
File "/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py", line 210, in main
main_posix('linux', '.so')
File "/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/ffi/build.py", line 134, in main_posix
raise RuntimeError(msg) from None
RuntimeError: Could not find a llvm-config binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help.
error: command '/home/user/anaconda3/envs/pytorch1/bin/python3.6' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/user/anaconda3/envs/pytorch1/bin/python3.6 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/setup.py'"'"'; file='"'"'/tmp/pip-install-v9ddsi1d/llvmlite_c662f38a09a547d3aac9a4c30baebe97/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-hho726u9/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/anaconda3/envs/pytorch1/include/python3.6m/llvmlite Check the logs for full command output.

How to use in Yolov5

我注意到你们给出了一个转换yolov5数据的代码，转换成csv文件。
进一步处理这个csv后使用wbf的demo有吗

Incorrect conf_type='max' mode

Example:

from ensemble_boxes import *

boxes_list = [[
    [0.1, 0.1, 0.2, 0.2],
], [
    [0.1, 0.1, 0.2, 0.2],
]]
scores_list = [[1], [1]]
labels_list = [[0], [0]]
weights = [1, 3]

iou_thr = 0.5
skip_box_thr = 0.0001

boxes, scores, labels = weighted_boxes_fusion(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr,
                                              skip_box_thr=skip_box_thr, allows_overflow=True, conf_type='max'
                                              )
print(scores)

Actual result: [1.5]
Expected result: [1]

completed wbf for rotate

Sould the "break" change to "continue"

Weighted-Boxes-Fusion/ensemble_boxes/ensemble_boxes_wbf.py

Line 35 in d48402e

if score < thr:

Hi I do not see why this should break,if one prediction is bad,we should just ignore instead of jumping out of loop unless this list is sorted?

Thanks!

Rotate Box example

I tried to solve the problem of Rotate Boxes. There is no problem in debugging, but I am not sure whether the result is correct. So I hope the author can provide an e-mail address. I will provide the code.
it is the result of Traditional WBF:
3
[[0.10666665 0.31666666 0.71333331 0.62 ]
[0. 0.51 0.81 0.91 ]
[0.04 0.56 0.84 0.92 ]]
it is the result of Rotate Box WBF:
3
[[0.10666665 0.62 0.10666665 0.31666666 0.71333331 0.31666666
0.71333331 0.62 ]
[0. 0.91 0. 0.51 0.81 0.51
0.81 0.91 ]
[0.04 0.92 0.04 0.56 0.84 0.56
0.84 0.92 ]]

Why does my ensemble still have so many repetitive (IoU > thresh I set) bboxes?

Thank you for your code.

However, when I use this method, I have so many repetitive bboxes. Do you have some advices?

Code is following:
boxes, score12, label = example_nms_2_models(boxes_list, labels_list, scores_list, draw_image=False, method=3, iou_thr=0.5, thresh=0.1, sigma=0.5)

I just use nms method because the limitation of running time. An example of results is following:
(Pdb) boxes
array([[0.73385417, 0.34259259, 0.81041667, 0.45648148],
[0.74930206, 0.36653722, 0.79911662, 0.45000624]])
The labels are same. Obviously, the IoU of these two boxes should be over than 0.5. Looks like NMS doen not work in ensemble. Right?

Single instance NMS

I'm after NMS over classes, but by default most implementations just do within class. Can't work out if over classes is implemented by this tool / repo?

See yolo discussion ultralytics/yolov5#2162

(struggling to work with that implementation, not obvious what the format of the input predictions is)

Can work with polygon object ?

Thank you for your work . I has a question how to use with output is polygon not rectangle ?

configure WBF to mmdetection

How do I configure WBF to mmdetection

Rotate Box

Hello，How to use it in rotate boxes?

how to use it in yolov5?

Is the result after NMS in YoloV5 passed to WBF for secondary screening?@ZFTurbo

Errors?

According to the function, the results of NMW score shouldn't be larger than 1, however, I can get many scores that are larger than 1. I can test this error in your example if we set all elements as 1.
#for i in range(len(weights)):

scores_list[i] = (np.array(scores_list[i]) * weights[i]) / weights.sum()

Strange result if boxes are intersected inside one model

Example:

from ensemble_boxes import *

boxes_list = [[
    [0.1, 0.1, 0.2, 0.2],
    [0.1, 0.1, 0.2, 0.2],
], [
    [0.3, 0.3, 0.4, 0.4],
]]
scores_list = [[1.0, 1.0], [0.5]]
labels_list = [[0, 0], [0]]
weights = [2, 1]

iou_thr = 0.5
skip_box_thr = 0.0001

boxes, scores, labels = weighted_boxes_fusion(boxes_list, scores_list, labels_list, weights=weights, iou_thr=iou_thr,
                                              skip_box_thr=skip_box_thr, allows_overflow=False
                                              )
print(scores)

Actual result: [1.33333337 0.16666667]
Expected result: scores are <= 1.

How to set weights in wbf correctly?

Hi, @ZFTurbo ! Thank you for your amazing approach and code!

I have a question: what is a right way of choosing weights parameter? I have several models and in case if I choose weights greater than 1, for example [2,1], I get confidence values greater than 1 in result. But if I choose normalized weights (the sum of weights is equal to 1), I get values less than 0.5 in result.

confidence score > 1 issue

elif not allows_overflow:
                weighted_boxes[i][1] = weighted_boxes[i][1] * min(len(weights), len(clustered_boxes)) / weights.sum()

Hi there, I noticed this line of code indicating whether the confidence score is allowed to overflow 1 or not. However, I didn't get it. Suppose I have four models with weights: 0.5, 0.5, 0.5, 1. we thus have weights.sum=2.5

Let's say the current cluster have 5 boxes, then len(clustered_boxes)=5. Now we have min(len(weights), len(clustered_boxes)) = min(4,5)=4 > weights.sum()

Do weighted_boxes[i][1] will overflow... Did I misunderstand something here? Thanks!

Is WBF consider to be part of Ensemble Learning

hi
i need to apply Ensemble Learning on My object detection project and
I saw your github, can you please confirm that WBF considers to be part of Ensemble Learning?
and is it like ADA boost? Does it work with voting Aggregation

2d bbox wbf compute result error

from ensemble_boxes import *

box_list = [[[0.3186783194541931, 0.7567034912109375, 0.3458101451396942, 0.8225753173828125],
             [0.2893345057964325, 0.7414910888671875, 0.30395767092704773, 0.82741845703125],
             [0.08234874159097672, 0.7192938232421875, 0.10110706090927124, 0.7978226318359375],
             [0.03901633247733116, 0.6389908447265625, 0.08065500855445862, 0.794277587890625],
             [0.10205534100532532, 0.7005220947265625, 0.3009164333343506, 0.8869298095703125],
             [0.013241510838270187, 0.7410292358398437, 0.4873046875, 0.9892860717773437],
             [0.20103125274181366, 0.6678619384765625, 0.2735580503940582, 0.8197706298828125],
             [0.2807173728942871, 0.748069091796875, 0.34884029626846313, 0.8265133056640624]],
            [[499.8333263993263, 0.7574194946289062, 499.8582033813, 0.8213037719726562],
             [499.80139984190464, 0.744067626953125, 499.8154309540987, 0.83922900390625],
             [499.79503624141216, 0.7509530639648437, 499.86309084296227, 0.8289489135742187],
             [499.52070143818855, 0.7752119750976563, 500.0, 0.9840308227539063],
             [499.5126953125, 0.5068761901855469, 499.98771003447473, 0.74507958984375]],
            [[0.31953710317611694, 0.758130859375, 0.34634509682655334, 0.821462158203125],
             [0.2890157401561737, 0.7444622192382813, 0.3044307231903076, 0.8266027221679687],
             [0.008551719598472118, 0.4673032836914062, 0.36901113390922546, 0.7606300659179688],
             [0.0, 0.4736612548828125, 0.4354751706123352, 0.738594970703125],
             [0.004452536813914776, 0.7197838134765625, 0.4873046875, 1.0818572998046876],
             [0.0, 0.7836503295898437, 0.4873046875, 0.9610354614257812],
             [0.0, 0.4444543762207031, 0.4644381105899811, 0.7153197021484375],
             [0.0, 0.4595667419433594, 0.45853325724601746, 0.745251220703125]],
            [[0.31933701038360596, 0.7575816040039063, 0.3459422290325165, 0.8217849731445312],
             [0.2892712652683258, 0.742080322265625, 0.3039073348045349, 0.82669189453125],
             [0.08082691580057144, 0.7194573364257812, 0.10114574432373047, 0.7929706420898438],
             [0.014002567157149315, 0.73406103515625, 0.4873046875, 1.049856201171875],
             [0.0058621931821107864, 0.485712646484375, 0.39766108989715576, 0.765383544921875],
             [0.0058621931821107864, 0.485712646484375, 0.39766108989715576, 0.765383544921875],
             [0.0010756775736808777, 0.7905551147460937, 0.4873046875, 0.9734806518554687]],
            [[0.3187095522880554, 0.7571224365234375, 0.34541282057762146, 0.82177685546875],
             [0.2895776629447937, 0.7409693603515625, 0.3036201298236847, 0.8277974853515625],
             [0.09533613175153732, 0.6792424926757813, 0.2974565327167511, 0.8962586059570312],
             [0.08239483088254929, 0.720238525390625, 0.10080346465110779, 0.7970889892578125],
             [0.03715477138757706, 0.6399656372070313, 0.08126747608184814, 0.7963726196289063],
             [0.11268552392721176, 0.6827256469726563, 0.12404713779687881, 0.7254093627929687],
             [0.13349150121212006, 0.6368695678710937, 0.14964361488819122, 0.6750675659179688],
             [0.15557613968849182, 0.6638201904296875, 0.28306907415390015, 0.858079833984375],
             [0.01832456886768341, 0.7675958251953126, 0.47474271059036255, 0.944915283203125],
             [0.4468492865562439, 0.6279349975585937, 0.4620358347892761, 0.6685007934570313]],
            [[0.3194415271282196, 0.7588555908203125, 0.3447743058204651, 0.8150806884765625],
             [0.0741792619228363, 0.6543595581054688, 0.3116285800933838, 0.9023289184570312],
             [0.08234858512878418, 0.7222454223632813, 0.10103598982095718, 0.7950789184570313],
             [0.13459675014019012, 0.6382381591796875, 0.1480436772108078, 0.6662626953125],
             [0.28921717405319214, 0.738470703125, 0.3036784827709198, 0.8197808837890626],
             [0.007326365448534489, 0.7826550903320313, 0.4597592055797577, 0.9532874145507813],
             [0.0, 0.6689840087890625, 0.0, 0.70613916015625]]]

score_list = [[0.83423, 0.79483235, 0.6388512, 0.18261029, 0.14935824, 0.088351235, 0.08091657, 0.07652432],
              [0.879186, 0.60111153, 0.14815152, 0.14187415, 0.063758574],
              [0.5681963, 0.4344975, 0.31340924, 0.21414337, 0.14648718, 0.07502777, 0.06623978, 0.06619557],
              [0.7660579, 0.6813378, 0.24239238, 0.11751874, 0.11032326, 0.10366057, 0.074862376],
              [0.8430377, 0.82931787, 0.7683554, 0.6823021, 0.28933194, 0.18235974, 0.17097037, 0.09869627, 0.08994506, 0.079594925],
              [0.78075427, 0.7309819, 0.6295794, 0.48739368, 0.39831838, 0.073874675, 0.059466254]]

label_list = [[6, 6, 6, 7, 7, 9, 7, 6],
              [6, 6, 6, 9, 11],
              [6, 6, 13, 11, 13, 9, 8, 0],
              [6, 6, 6, 13, 11, 13, 9],
              [6, 6, 7, 6, 7, 6, 6, 7, 9, 6],
              [6, 7, 6, 6, 6, 9, 6]]

weights = [1, 1, 1, 1, 1, 1]
box_list, score_list, label_list = weighted_boxes_fusion(box_list, score_list, label_list,
                                                         weights=weights, iou_thr=0.5,
                                                         skip_box_thr=0.1)
print(box_list)
print(score_list)
print(label_list)

and the result is:

[[0.31910411 0.75763094 0.34561539 0.82052845]
[0.28932601 0.7415092  0.30388758 0.82627851]
[0.08219483 0.72045314 0.10099649 0.79627055]
[0.08656451 0.67013788 0.3040534  0.89810479]
[0.13430972 0.63788271 0.14845915 0.66854924]
[0.03787507 0.63958848 0.0810305  0.79556197]
[0.00788325 0.47187883 0.37613192 0.76181149]
[0.00199323 0.47775888 0.42261785 0.74770349]
[0.00870361 0.72613913 0.48730466 1.        ]
[0.11268552 0.68272565 0.12404714 0.72540936]]
[0.63204601 0.52305063 0.36552083 0.2747826  0.10972734 0.07865704
0.06951164 0.05407777 0.04400099 0.03039329]
[ 6.  6.  6.  7.  6.  7. 13. 11. 13.  6.]

The low confidence box that lower than 0.1 (skip_box_thr) not be filtrated.