anuragxel / salt Goto Github PK

When you add it, you get this extra piece. How to solve

Error Message:

...//salt/salt/display_utils.py", line 33, in __convert_ann_to_mask
    rles = coco_mask.frPyObjects(poly, height, width)
  File "pycocotools/_mask.pyx", line 308, in pycocotools._mask.frPyObjects
Exception: input type is not supported.

 .../salt/salt/display_utils.py", line 33, in __convert_ann_to_mask
    rles = coco_mask.frPyObjects(poly, height, width)
  File "pycocotools/_mask.pyx", line 294, in pycocotools._mask.frPyObjects
TypeError: Argument 'bb' has incorrect type (expected numpy.ndarray, got list)

My image is pretty obscure so sometimes SAM segmentation include very small mask which has poly of 4 elements.

This seems to coincide with this issue.
cocodataset/cocoapi#139

Maybe there need to be some changes that make len(poly[i]) > 4

Package libffi conflicts for:
certifi=2022.12.7 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
libffi=3.4.2
wheel=0.38.4 -> python[version='>=3.10,<3.11.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
setuptools=65.6.3 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
pip=23.0.1 -> python[version='>=3.9,<3.10.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']

Models folder would have all onnx models in a specified format e.g. ".<res>.onnx" and the ONNX caller picks up the "right" model for the given image.

Hi
Thanks for your quick and great job!

I want to test on 3 images but get the following error when calling segment_anything_annotator.py
any insights?

Traceback (most recent call last): File "segment_anything_annotator.py", line 26, in <module> editor = Editor( File "/mnt/data/polymore/salt/salt/editor.py", line 57, in __init__ ) = self.dataset_explorer.get_image_data(self.image_id) File "/mnt/data/polymore/salt/salt/dataset_explorer.py", line 146, in get_image_data image_name = self.coco_json["images"][image_id]["file_name"] IndexError: list index out of range

Thanks in advance

while annotating, the areas which are excluded as background is still getting enclosed within the annotation image after clicking add. If we annotate this image, https://www.electriduct.com/assets/images/water-filled-barricade-female-angle.jpg, the hollow parts in the barricade gets annotated even if its not annotated or manually exclude using right click.

When I use the generate_onnx.py to generate the onnx file of SAM, it would return the exception:
Traceback (most recent call last):
File "generate_onnx.py", line 177, in
run_export(
File "generate_onnx.py", line 149, in run_export
torch.onnx.export(
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/init.py", line 305, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 118, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 719, in _export
_model_to_graph(model, args, verbose, input_names,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 503, in _model_to_graph
graph = _optimize_graph(graph, operator_export_type,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 232, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/init.py", line 354, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 1061, in _run_symbolic_function
return symbolic_fn(g, *inputs, **attrs)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/symbolic_opset13.py", line 267, in repeat_interleave
return torch.onnx.symbolic_opset9.repeat_interleave(g, self, repeats, final_dim)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py", line 2152, in repeat_interleave
for idx, r_split in enumerate(r_splits):
TypeError: 'torch._C.Value' object is not iterable
(Occurred when translating repeat_interleave).
I don‘t konw how to solve the exception. May I beg a favour？Thanks for your help.

@anuragxel @Daa98
I faced this issue when running the app in windows system. The function was not updated with the click event.

   def get_side_panel(self):
        panel = QWidget()
        panel_layout = QVBoxLayout(panel)
        categories = self.editor.get_categories()
        for category in categories:
            label = QPushButton(category)
            label.clicked.connect(lambda: self.editor.select_category(category))
            panel_layout.addWidget(label)
        return panel

the lambda function inside the loop is capturing the reference to the same category variable every time, and when the lambda function is executed later, it uses the last value that category had in the loop.
To fix this, create a separate function that takes the category as an argument and returns a lambda function that calls self.editor.select_category(category). Now lambda function will capture the value of category at the time it was created.

    def get_side_panel(self):
        panel = QWidget()
        panel_layout = QVBoxLayout(panel)
        categories = self.editor.get_categories()
        for category in categories:
            label = QPushButton(category)
            label.clicked.connect(self.create_category_handler(category))
            panel_layout.addWidget(label)
        return panel

    def create_category_handler(self, category):
        return lambda: self.editor.select_category(category)

How to do box labeling mode? I left clicked just marked the mask？

Thanks for your nice work!
I don't have any image that it's shape is [1080,1920]，therefor i get this erro when i call segment_anything_annotator.py :

Traceback (most recent call last):
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\segment_anything_annotator.py", line 26, in <module>
    editor = Editor(
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\editor.py", line 58, in __init__
    self.onnx_helper = OnnxModels(onnx_models_path)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 16, in __init__
    self.set_image_resolution(image_width, image_height)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 29, in set_image_resolution
    self.__init_model(onnx_model_path)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 19, in __init_model
    self.ort_session = onnxruntime.InferenceSession(
  File "C:\Users\26655\.conda\envs\pt2\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 360, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\Users\26655\.conda\envs\pt2\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 397, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE : Load model from model data\sam_onnx.1080_1920.onnx failed:Load model model data\sam_onnx.1080_1920.onnx failed. File doesn't exist

It's clear that it happens when class Editor initializes

When I clicked Split, some graphics inexplicably connected to the top left corner, which felt like a bug.

Thank you for sharing the awesome work!
I think it is necessary to adjust the transparency of the current predicted mask during the annotation process, so that it is easier to check the accuracy of the mask boundaries.
Not much code to adjust，

    def step_up_transparency(self):
        self.display = self.image_bgr.copy()
        self.du.increase_transparency()

        if self.curr_inputs.curr_mask is not None:
            self.display = self.du.draw_points(
            self.display, self.curr_inputs.input_point, self.curr_inputs.input_label)
            self.display = self.du.overlay_mask_on_image(self.display, self.curr_inputs.curr_mask)
        
        if self.show_other_anns:
            self.draw_known_annotations()

    def step_down_transparency(self):
        self.display = self.image_bgr.copy()
        self.du.decrease_transparency()

        if self.curr_inputs.curr_mask is not None:
            self.display = self.du.draw_points(
            self.display, self.curr_inputs.input_point, self.curr_inputs.input_label)
            self.display = self.du.overlay_mask_on_image(self.display, self.curr_inputs.curr_mask)
        
        if self.show_other_anns:
            self.draw_known_annotations()

Error: Request failed with status code 500. "datumaro.components.errors.AnnotationImportError: Failed to import item (0, 'default') annotation: Polygon has invalid value count 4. Expected at least 3 (x, y) pairs.\n"

Hi!
when I press the middle mouse button (usually by accident) I get this error:

QLayout::addChildLayout: layout "" already has a parent
Traceback (most recent call last):
  File "/home/maciej/PycharmProjects/salt/salt/interface.py", line 97, in mousePressEvent
    self.editor.add_click([int(x), int(y)], label, selected_annotations)
UnboundLocalError: local variable 'label' referenced before assignment
Aborted (core dumped)

I will try to add fix of these bug later

Thanks for your tools. I am trying to use this tool in my equipment. And i meet some problems during run the segment_anything_annotator.py.

I have a server with GPU, which is a Linux system without GUI. It's totally terminal system. I use my Macbook to link the server with SSH tool. And I have created embedding and onnx model in server.

When I want to annotate my images, the PyQt doesn't work well and export error:Could not connect to any X display and QLibraryPrivate::unload succeeded on "/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so". I have tried to fix it by reinstall pyqt and install PYQT Integration in my VSCode. But it is not useful for me.

I would like to ask two questions:

1. How can i solve my PyQt problems? Can you give me some advices?
2. How to use your tool in Macbook and GPU server more more convenient?

I am looking forward to your answer

Add ability to provide box input along with point input (with nice looking mousedown animation). Though I'm not sure if it's needed to be honest, point input works really well in general.

Let's say you annotated an object and forgot to change the category, go into this UI mode, change the radio selection and fix the category of that object with a click.

Issue1: TypeError: Argument 'bb' has incorrect type (expected numpy.ndarray, got list).

Issue2: ValueError: list.remove(x): x not in list

This is actually crashing the py page everytime

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_149' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,7,256}, requested shape:{1,10,8,32}

Reference: https://github.com/ChaoningZhang/MobileSAM

Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.

MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

Best Wishes,

Qiao

When I view the image, there is a pink line generated from the top left corner of the image. I tried to investigate the issue by printing variable "sc" from the file dataset_explorer.py, and I found a random value "6.91669498947497e-310, 6.91669498947497e-310" generated at the beginning. To overcome this issue, I added this line "sc = [x for x in sc if x > 0.1]" in the file dataset_explorer.py.

Screenshot:

Please refer to the attached screenshots for better understanding.

Expected Outcome:

The pink line should not be generated from the top left corner of the image.

Steps to Reproduce:

1. Run segment_anything_annotator.py.
2. Label any object.
3. Select the label name and click on the save button, then click on the add button.
4. The error will be found on the image.

i took 5 images in folder,
i clicked on object its showing mask on it, then i clicked on save button, reted same process for all images & genrated annotations.json is not correct

{"info": {"description": "SAM Dataset", "url": "", "version": "1.0", "year": 2023, "contributor": "Sam", "date_created": "2021/07/01"}, "images": [{"id": 0, "file_name": "images\\Cigaretee_i_Weapon_Pragati137.jpg", "width": 1000, "height": 1500}, {"id": 1, "file_name": "images\\Cigaretee_i_Weapon_Pragati138.jpg", "width": 1600, "height": 2400}, {"id": 2, "file_name": "images\\Cigaretee_i_Weapon_Pragati139.jpg", "width": 1600, "height": 1067}, {"id": 3, "file_name": "images\\Cigaretee_i_Weapon_Pragati140.jpg", "width": 1600, "height": 2400}, {"id": 4, "file_name": "images\\Cigaretee_i_Weapon_Pragati141.jpg", "width": 1600, "height": 2400}], "annotations": [{"id": 0, "image_id": 0, "category_id": 0, "bbox": [468.0, 675.0, 53.0, 197.0], "area": 10441.0, "iscrowd": 0, "segmentation": [[505.0, 754.5, 500.0, 754.5, 498.0, 753.5, 493.5, 748.0, 489.5, 737.0, 488.5, 736.0, 484.5, 726.0, 475.5, 707.0, 473.5, 699.0, 471.5, 696.0, 467.5, 684.0, 467.5, 678.0, 471.0, 674.5, 477.0, 674.5, 480.5, 681.0, 485.5, 686.0, 489.5, 692.0, 492.5, 700.0, 495.5, 703.0, 495.5, 705.0, 498.5, 711.0, 500.5, 713.0, 504.5, 721.0, 506.5, 727.0, 510.5, 734.0, 511.5, 738.0, 513.5, 740.0, 513.5, 747.0, 507.0, 753.5, 505.0, 754.5], [518.0, 871.5, 514.5, 870.0, 516.0, 867.5, 519.0, 867.5, 520.5, 869.0, 518.0, 871.5]]}], "categories": [{"id": 0, "name": "'cigaretee'", "supercategory": "'cigaretee'"}]}

Sorry for the ignorance, can I run this remotely? This is, using a sever - client paradigm or similarly?

Thank for your work! It's very useful for me!
Would you add fuction to delete existing labels, thanks!

If possible then please integrate this repo with labelstudio because there are lots of other things that they have got it right.

i set 5 classes in it ,but i dont know how to choose these class. it seems no thing happend whether click the class button on the right of the windows

Need to think of a UI (how to select the annotation) and then implement corresponding deletion functionality.

thanks for the amazing work, may i know where is the code for converting the mask into segmented support format coco or yolo format data? i m trying to load my custom detector & want to detect objects & get its coordinates. thanks

as the blow image show

Traceback (most recent call last):
File "……/segment_anything_annotator.py", line 27, in
editor = Editor(
File "……\salt\editor.py", line 57, in init
) = self.dataset_explorer.get_image_data(self.image_id)
File "……\salt\dataset_explorer.py", line 144, in get_image_data
image_name = self.coco_json["images"][image_id]["file_name"]
IndexError: list index out of range

Segment Anything model doesn't always work. Let users add a custom polygon if they realize that's faster to do.