anuragxel / salt

Segment Anything Labelling Tool

Segment Anything Labelling Tool (SALT)

Uses the Segment-Anything Model By Meta AI and adds a barebones interface to label images and saves the masks in the COCO format.

Under active development, apologies for rough edges and bugs. Use at your own risk.

Installation

1. Install Segment Anything on any machine with a GPU. (Need not be the labelling machine.)
2. Create a conda environment using conda env create -f environment.yaml on the labelling machine (Need not have GPU).
3. (Optional) Install coco-viewer to scroll through your annotations quickly.

Usage

1. Setup your dataset in the following format <dataset_name>/images/* and create empty folder <dataset_name>/embeddings.
   • Annotations will be saved in <dataset_name>/annotations.json by default.
2. Copy the helpers scripts to the base folder of your segment-anything folder.
   • Call extract_embeddings.py to extract embeddings for your images.
   • Call generate_onnx.py generate *.onnx files in models.
3. Copy the models in models folder.
4. Symlink your dataset in the SALT's root folder as <dataset_name>.
5. Call segment_anything_annotator.py with argument <dataset_name> and categories cat1,cat2,cat3...
   • There are a few keybindings that make the annotation process fast.
   • Click on the object using left clicks and right click (to indicate outside object boundary).
   • n adds predicted mask into your annotations. (Add button)
   • r rejects the predicted mask. (Reject button)
   • a and d to cycle through images in your your set. (Next and Prev)
   • l and k to increase and decrease the transparency of the other annotations.
   • Ctrl + S to save progress to the COCO-style annotations file.
6. coco-viewer to view your annotations.
   • python cocoviewer.py -i <dataset> -a <dataset>/annotations.json

Demo

Contributing

Follow these guidelines to ensure that your contributions can be reviewed and merged. Need a lot of help in making the UI better.

If you have found a bug or have an idea for an improvement or new feature, please create an issue on GitHub. Before creating a new issue, please search existing issues to see if your issue has already been reported.

When creating an issue, please include as much detail as possible, including steps to reproduce the issue if applicable.

Create a pull request (PR) to the original repository. Please use black formatter when making code changes.

License

The code is licensed under the MIT License. By contributing to SALT, you agree to license your contributions under the same license as the project. See LICENSE for more information.

salt's Issues

IndexError: list index out of range

Traceback (most recent call last):
File "……/segment_anything_annotator.py", line 27, in
editor = Editor(
File "……\salt\editor.py", line 57, in init
) = self.dataset_explorer.get_image_data(self.image_id)
File "……\salt\dataset_explorer.py", line 144, in get_image_data
image_name = self.coco_json["images"][image_id]["file_name"]
IndexError: list index out of range

genrated annotion are not correct

i took 5 images in folder,
i clicked on object its showing mask on it, then i clicked on save button, reted same process for all images & genrated annotations.json is not correct

{"info": {"description": "SAM Dataset", "url": "", "version": "1.0", "year": 2023, "contributor": "Sam", "date_created": "2021/07/01"}, "images": [{"id": 0, "file_name": "images\\Cigaretee_i_Weapon_Pragati137.jpg", "width": 1000, "height": 1500}, {"id": 1, "file_name": "images\\Cigaretee_i_Weapon_Pragati138.jpg", "width": 1600, "height": 2400}, {"id": 2, "file_name": "images\\Cigaretee_i_Weapon_Pragati139.jpg", "width": 1600, "height": 1067}, {"id": 3, "file_name": "images\\Cigaretee_i_Weapon_Pragati140.jpg", "width": 1600, "height": 2400}, {"id": 4, "file_name": "images\\Cigaretee_i_Weapon_Pragati141.jpg", "width": 1600, "height": 2400}], "annotations": [{"id": 0, "image_id": 0, "category_id": 0, "bbox": [468.0, 675.0, 53.0, 197.0], "area": 10441.0, "iscrowd": 0, "segmentation": [[505.0, 754.5, 500.0, 754.5, 498.0, 753.5, 493.5, 748.0, 489.5, 737.0, 488.5, 736.0, 484.5, 726.0, 475.5, 707.0, 473.5, 699.0, 471.5, 696.0, 467.5, 684.0, 467.5, 678.0, 471.0, 674.5, 477.0, 674.5, 480.5, 681.0, 485.5, 686.0, 489.5, 692.0, 492.5, 700.0, 495.5, 703.0, 495.5, 705.0, 498.5, 711.0, 500.5, 713.0, 504.5, 721.0, 506.5, 727.0, 510.5, 734.0, 511.5, 738.0, 513.5, 740.0, 513.5, 747.0, 507.0, 753.5, 505.0, 754.5], [518.0, 871.5, 514.5, 870.0, 516.0, 867.5, 519.0, 867.5, 520.5, 869.0, 518.0, 871.5]]}], "categories": [{"id": 0, "name": "'cigaretee'", "supercategory": "'cigaretee'"}]}

integrate this with labelstudio

If possible then please integrate this repo with labelstudio because there are lots of other things that they have got it right.

Add UI button/mode to change category of an annotation.

Let's say you annotated an object and forgot to change the category, go into this UI mode, change the radio selection and fix the category of that object with a click.

Excluded regions are still getting annotated within the annotated area of the image.

while annotating, the areas which are excluded as background is still getting enclosed within the annotation image after clicking add. If we annotate this image, https://www.electriduct.com/assets/images/water-filled-barricade-female-angle.jpg, the hollow parts in the barricade gets annotated even if its not annotated or manually exclude using right click.

IndexError: list index out of range

Hi
Thanks for your quick and great job!

I want to test on 3 images but get the following error when calling segment_anything_annotator.py
any insights?

Traceback (most recent call last): File "segment_anything_annotator.py", line 26, in <module> editor = Editor( File "/mnt/data/polymore/salt/salt/editor.py", line 57, in __init__ ) = self.dataset_explorer.get_image_data(self.image_id) File "/mnt/data/polymore/salt/salt/dataset_explorer.py", line 146, in get_image_data image_name = self.coco_json["images"][image_id]["file_name"] IndexError: list index out of range

Thanks in advance

Classes Not updated while clicking on the GUI option

@anuragxel @Daa98
I faced this issue when running the app in windows system. The function was not updated with the click event.

   def get_side_panel(self):
        panel = QWidget()
        panel_layout = QVBoxLayout(panel)
        categories = self.editor.get_categories()
        for category in categories:
            label = QPushButton(category)
            label.clicked.connect(lambda: self.editor.select_category(category))
            panel_layout.addWidget(label)
        return panel

the lambda function inside the loop is capturing the reference to the same category variable every time, and when the lambda function is executed later, it uses the last value that category had in the loop.
To fix this, create a separate function that takes the category as an argument and returns a lambda function that calls self.editor.select_category(category). Now lambda function will capture the value of category at the time it was created.

    def get_side_panel(self):
        panel = QWidget()
        panel_layout = QVBoxLayout(panel)
        categories = self.editor.get_categories()
        for category in categories:
            label = QPushButton(category)
            label.clicked.connect(self.create_category_handler(category))
            panel_layout.addWidget(label)
        return panel

    def create_category_handler(self, category):
        return lambda: self.editor.select_category(category)

how can i draw different class?

i set 5 classes in it ,but i dont know how to choose these class. it seems no thing happend whether click the class button on the right of the windows

some bug :Load model from model sam_onnx.1080_1920.onnx failed:

Thanks for your nice work!
I don't have any image that it's shape is [1080,1920]，therefor i get this erro when i call segment_anything_annotator.py :

Traceback (most recent call last):
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\segment_anything_annotator.py", line 26, in <module>
    editor = Editor(
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\editor.py", line 58, in __init__
    self.onnx_helper = OnnxModels(onnx_models_path)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 16, in __init__
    self.set_image_resolution(image_width, image_height)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 29, in set_image_resolution
    self.__init_model(onnx_model_path)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 19, in __init_model
    self.ort_session = onnxruntime.InferenceSession(
  File "C:\Users\26655\.conda\envs\pt2\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 360, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\Users\26655\.conda\envs\pt2\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 397, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE : Load model from model data\sam_onnx.1080_1920.onnx failed:Load model model data\sam_onnx.1080_1920.onnx failed. File doesn't exist

It's clear that it happens when class Editor initializes

Add Box Input

Add ability to provide box input along with point input (with nice looking mousedown animation). Though I'm not sure if it's needed to be honest, point input works really well in general.

When you add it, you get this extra piece. How to solve

When you add it, you get this extra piece. How to solve

When zooming in to more than full screen, holding down the middle mouse button to drag the image may cause the result to get stuck and flicker back.

Traceback (most recent call last):
  File "D:\pyworkspaces\SAM\SAM-Tool\salt\interface.py", line 67, in mousePressEvent
    self.editor.add_click([int(x), int(y)], label)
UnboundLocalError: local variable 'label' referenced before assignment

There is a problem with the reshape node exported by ONNX. I checked the network structure and found that it is only a simple reshape node. Have you encountered this problem

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_149' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,7,256}, requested shape:{1,10,8,32}

Unreasonable segmentation results

When I clicked Split, some graphics inexplicably connected to the top left corner, which felt like a bug.

Package libffi conflict

Package libffi conflicts for:
certifi=2022.12.7 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
libffi=3.4.2
wheel=0.38.4 -> python[version='>=3.10,<3.11.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
setuptools=65.6.3 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
pip=23.0.1 -> python[version='>=3.9,<3.10.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']

check this problem, has been quite persistent, when creating a label which is partially covered under a label. Crashing Frequntly

Issue1: TypeError: Argument 'bb' has incorrect type (expected numpy.ndarray, got list).

Issue2: ValueError: list.remove(x): x not in list

This is actually crashing the py page everytime

Unable to adjust the transparency of the predicted mask

Thank you for sharing the awesome work!
I think it is necessary to adjust the transparency of the current predicted mask during the annotation process, so that it is easier to check the accuracy of the mask boundaries.
Not much code to adjust，

salt/salt/editor.py

Line 96 in 5ed8652

def step_up_transparency(self):

Amend the following after the line of code:

    def step_up_transparency(self):
        self.display = self.image_bgr.copy()
        self.du.increase_transparency()

        if self.curr_inputs.curr_mask is not None:
            self.display = self.du.draw_points(
            self.display, self.curr_inputs.input_point, self.curr_inputs.input_label)
            self.display = self.du.overlay_mask_on_image(self.display, self.curr_inputs.curr_mask)
        
        if self.show_other_anns:
            self.draw_known_annotations()

    def step_down_transparency(self):
        self.display = self.image_bgr.copy()
        self.du.decrease_transparency()

        if self.curr_inputs.curr_mask is not None:
            self.display = self.du.draw_points(
            self.display, self.curr_inputs.input_point, self.curr_inputs.input_label)
            self.display = self.du.overlay_mask_on_image(self.display, self.curr_inputs.curr_mask)
        
        if self.show_other_anns:
            self.draw_known_annotations()

why can't i correctlyopen coco jason file with cocoviewer correctly?

Pink line generated from the top left corner of the image

When I view the image, there is a pink line generated from the top left corner of the image. I tried to investigate the issue by printing variable "sc" from the file dataset_explorer.py, and I found a random value "6.91669498947497e-310, 6.91669498947497e-310" generated at the beginning. To overcome this issue, I added this line "sc = [x for x in sc if x > 0.1]" in the file dataset_explorer.py.

Screenshot:

Please refer to the attached screenshots for better understanding.

1. 
2. 

Expected Outcome:

The pink line should not be generated from the top left corner of the image.

Steps to Reproduce:

1. Run segment_anything_annotator.py.
2. Label any object.
3. Select the label name and click on the save button, then click on the add button.
4. The error will be found on the image.

A bug occur when I want to change the categories

I want to change the categories from 'cat' and 'dog' to other labels, but as I want to change the categories with the command:
python segment_anything_annotator.py --onnx-model-path sam_onnx.onnx --dataset-path ../shot --categories i,j,k,l,o

It turns out that the categories in the interface are still cat and dog, and I cannot change the categories whatever I type in.
So I wonder if someone who once faced the same problem can give me an explanation and help me slove this problem, thank you so much!!! ^-^

how to get annotations from mask?

thanks for the amazing work, may i know where is the code for converting the mask into segmented support format coco or yolo format data? i m trying to load my custom detector & want to detect objects & get its coordinates. thanks

Add fuction of deleting existing labels

Thank for your work! It's very useful for me!
Would you add fuction to delete existing labels, thanks!

Add Delete Functionality.

Need to think of a UI (how to select the annotation) and then implement corresponding deletion functionality.

Error when clicking middle mouse button

Hi!
when I press the middle mouse button (usually by accident) I get this error:

QLayout::addChildLayout: layout "" already has a parent
Traceback (most recent call last):
  File "/home/maciej/PycharmProjects/salt/salt/interface.py", line 97, in mousePressEvent
    self.editor.add_click([int(x), int(y)], label, selected_annotations)
UnboundLocalError: local variable 'label' referenced before assignment
Aborted (core dumped)

I will try to add fix of these bug later

An error occurred when importing the labeling results into cvat, has anyone encountered it?

Error: Request failed with status code 500. "datumaro.components.errors.AnnotationImportError: Failed to import item (0, 'default') annotation: Polygon has invalid value count 4. Expected at least 3 (x, y) pairs.\n"

Add manual polygon mode to let a user manually add a polygon.

Segment Anything model doesn't always work. Let users add a custom polygon if they realize that's faster to do.

Error for low quality image

Error Message:

...//salt/salt/display_utils.py", line 33, in __convert_ann_to_mask
    rles = coco_mask.frPyObjects(poly, height, width)
  File "pycocotools/_mask.pyx", line 308, in pycocotools._mask.frPyObjects
Exception: input type is not supported.

 .../salt/salt/display_utils.py", line 33, in __convert_ann_to_mask
    rles = coco_mask.frPyObjects(poly, height, width)
  File "pycocotools/_mask.pyx", line 294, in pycocotools._mask.frPyObjects
TypeError: Argument 'bb' has incorrect type (expected numpy.ndarray, got list)

My image is pretty obscure so sometimes SAM segmentation include very small mask which has poly of 4 elements.

This seems to coincide with this issue.
cocodataset/cocoapi#139

Maybe there need to be some changes that make len(poly[i]) > 4

Select ONNX model depending on the input resolution

Models folder would have all onnx models in a specified format e.g. ".<res>.onnx" and the ONNX caller picks up the "right" model for the given image.

TypeError: 'torch._C.Value' object is not iterable

When I use the generate_onnx.py to generate the onnx file of SAM, it would return the exception:
Traceback (most recent call last):
File "generate_onnx.py", line 177, in
run_export(
File "generate_onnx.py", line 149, in run_export
torch.onnx.export(
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/init.py", line 305, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 118, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 719, in _export
_model_to_graph(model, args, verbose, input_names,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 503, in _model_to_graph
graph = _optimize_graph(graph, operator_export_type,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 232, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/init.py", line 354, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 1061, in _run_symbolic_function
return symbolic_fn(g, *inputs, **attrs)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/symbolic_opset13.py", line 267, in repeat_interleave
return torch.onnx.symbolic_opset9.repeat_interleave(g, self, repeats, final_dim)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py", line 2152, in repeat_interleave
for idx, r_split in enumerate(r_splits):
TypeError: 'torch._C.Value' object is not iterable
(Occurred when translating repeat_interleave).
I don‘t konw how to solve the exception. May I beg a favour？Thanks for your help.

Suggestion - Integrate MobileSAM into the pipeline for lightweight and faster inference

Reference: https://github.com/ChaoningZhang/MobileSAM

Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.

MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

Best Wishes,

Qiao

How to do box labeling mode? I left clicked just marked the mask

How to do box labeling mode? I left clicked just marked the mask？

The mask does not exactly coincide with the original image

as the blow image show

Some GUI problems

Thanks for your tools. I am trying to use this tool in my equipment. And i meet some problems during run the segment_anything_annotator.py.

I have a server with GPU, which is a Linux system without GUI. It's totally terminal system. I use my Macbook to link the server with SSH tool. And I have created embedding and onnx model in server.

When I want to annotate my images, the PyQt doesn't work well and export error:Could not connect to any X display and QLibraryPrivate::unload succeeded on "/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so". I have tried to fix it by reinstall pyqt and install PYQT Integration in my VSCode. But it is not useful for me.

I would like to ask two questions:

1. How can i solve my PyQt problems? Can you give me some advices?
2. How to use your tool in Macbook and GPU server more more convenient?

I am looking forward to your answer

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (8,) + inhomogeneous part.

remote labeling

Sorry for the ignorance, can I run this remotely? This is, using a sever - client paradigm or similarly?

Failed to run, maybe caused by parsing annotations.json

python segment_anything_annotator.py --onnx-models-path models --dataset-path seg_imgs --categories dog
Traceback (most recent call last):
  File "segment_anything_annotator.py", line 26, in <module>
    editor = Editor(
  File "C:\pyprojects\salt\salt\editor.py", line 45, in __init__
    self.dataset_explorer = DatasetExplorer(
  File "C:\pyprojects\salt\salt\dataset_explorer.py", line 126, in __init__
    max(self.coco_json["annotations"], key=lambda x: x["id"])["id"] + 1
ValueError: max() arg is an empty sequence