GithubHelp home page GithubHelp logo

salt's Introduction

Segment Anything Labelling Tool (SALT)

Uses the Segment-Anything Model By Meta AI and adds a barebones interface to label images and saves the masks in the COCO format.

Under active development, apologies for rough edges and bugs. Use at your own risk.

Installation

  1. Install Segment Anything on any machine with a GPU. (Need not be the labelling machine.)
  2. Create a conda environment using conda env create -f environment.yaml on the labelling machine (Need not have GPU).
  3. (Optional) Install coco-viewer to scroll through your annotations quickly.

Usage

  1. Setup your dataset in the following format <dataset_name>/images/* and create empty folder <dataset_name>/embeddings.
    • Annotations will be saved in <dataset_name>/annotations.json by default.
  2. Copy the helpers scripts to the base folder of your segment-anything folder.
    • Call extract_embeddings.py to extract embeddings for your images.
    • Call generate_onnx.py generate *.onnx files in models.
  3. Copy the models in models folder.
  4. Symlink your dataset in the SALT's root folder as <dataset_name>.
  5. Call segment_anything_annotator.py with argument <dataset_name> and categories cat1,cat2,cat3...
    • There are a few keybindings that make the annotation process fast.
    • Click on the object using left clicks and right click (to indicate outside object boundary).
    • n adds predicted mask into your annotations. (Add button)
    • r rejects the predicted mask. (Reject button)
    • a and d to cycle through images in your your set. (Next and Prev)
    • l and k to increase and decrease the transparency of the other annotations.
    • Ctrl + S to save progress to the COCO-style annotations file.
  6. coco-viewer to view your annotations.
    • python cocoviewer.py -i <dataset> -a <dataset>/annotations.json

Demo

How it Works Gif!

Contributing

Follow these guidelines to ensure that your contributions can be reviewed and merged. Need a lot of help in making the UI better.

If you have found a bug or have an idea for an improvement or new feature, please create an issue on GitHub. Before creating a new issue, please search existing issues to see if your issue has already been reported.

When creating an issue, please include as much detail as possible, including steps to reproduce the issue if applicable.

Create a pull request (PR) to the original repository. Please use black formatter when making code changes.

License

The code is licensed under the MIT License. By contributing to SALT, you agree to license your contributions under the same license as the project. See LICENSE for more information.

salt's People

Contributors

anuragxel avatar brollb avatar daa98 avatar dtock89 avatar tdnb avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

salt's Issues

IndexError: list index out of range

Traceback (most recent call last):
File "……/segment_anything_annotator.py", line 27, in
editor = Editor(
File "……\salt\editor.py", line 57, in init
) = self.dataset_explorer.get_image_data(self.image_id)
File "……\salt\dataset_explorer.py", line 144, in get_image_data
image_name = self.coco_json["images"][image_id]["file_name"]
IndexError: list index out of range

genrated annotion are not correct

image

i took 5 images in folder,
i clicked on object its showing mask on it, then i clicked on save button, reted same process for all images & genrated annotations.json is not correct


{"info": {"description": "SAM Dataset", "url": "", "version": "1.0", "year": 2023, "contributor": "Sam", "date_created": "2021/07/01"}, "images": [{"id": 0, "file_name": "images\\Cigaretee_i_Weapon_Pragati137.jpg", "width": 1000, "height": 1500}, {"id": 1, "file_name": "images\\Cigaretee_i_Weapon_Pragati138.jpg", "width": 1600, "height": 2400}, {"id": 2, "file_name": "images\\Cigaretee_i_Weapon_Pragati139.jpg", "width": 1600, "height": 1067}, {"id": 3, "file_name": "images\\Cigaretee_i_Weapon_Pragati140.jpg", "width": 1600, "height": 2400}, {"id": 4, "file_name": "images\\Cigaretee_i_Weapon_Pragati141.jpg", "width": 1600, "height": 2400}], "annotations": [{"id": 0, "image_id": 0, "category_id": 0, "bbox": [468.0, 675.0, 53.0, 197.0], "area": 10441.0, "iscrowd": 0, "segmentation": [[505.0, 754.5, 500.0, 754.5, 498.0, 753.5, 493.5, 748.0, 489.5, 737.0, 488.5, 736.0, 484.5, 726.0, 475.5, 707.0, 473.5, 699.0, 471.5, 696.0, 467.5, 684.0, 467.5, 678.0, 471.0, 674.5, 477.0, 674.5, 480.5, 681.0, 485.5, 686.0, 489.5, 692.0, 492.5, 700.0, 495.5, 703.0, 495.5, 705.0, 498.5, 711.0, 500.5, 713.0, 504.5, 721.0, 506.5, 727.0, 510.5, 734.0, 511.5, 738.0, 513.5, 740.0, 513.5, 747.0, 507.0, 753.5, 505.0, 754.5], [518.0, 871.5, 514.5, 870.0, 516.0, 867.5, 519.0, 867.5, 520.5, 869.0, 518.0, 871.5]]}], "categories": [{"id": 0, "name": "'cigaretee'", "supercategory": "'cigaretee'"}]}

integrate this with labelstudio

If possible then please integrate this repo with labelstudio because there are lots of other things that they have got it right.

IndexError: list index out of range

Hi
Thanks for your quick and great job!

I want to test on 3 images but get the following error when calling segment_anything_annotator.py
any insights?

Traceback (most recent call last): File "segment_anything_annotator.py", line 26, in <module> editor = Editor( File "/mnt/data/polymore/salt/salt/editor.py", line 57, in __init__ ) = self.dataset_explorer.get_image_data(self.image_id) File "/mnt/data/polymore/salt/salt/dataset_explorer.py", line 146, in get_image_data image_name = self.coco_json["images"][image_id]["file_name"] IndexError: list index out of range

Thanks in advance

Classes Not updated while clicking on the GUI option

@anuragxel @Daa98
I faced this issue when running the app in windows system. The function was not updated with the click event.

   def get_side_panel(self):
        panel = QWidget()
        panel_layout = QVBoxLayout(panel)
        categories = self.editor.get_categories()
        for category in categories:
            label = QPushButton(category)
            label.clicked.connect(lambda: self.editor.select_category(category))
            panel_layout.addWidget(label)
        return panel

the lambda function inside the loop is capturing the reference to the same category variable every time, and when the lambda function is executed later, it uses the last value that category had in the loop.
To fix this, create a separate function that takes the category as an argument and returns a lambda function that calls self.editor.select_category(category). Now lambda function will capture the value of category at the time it was created.

    def get_side_panel(self):
        panel = QWidget()
        panel_layout = QVBoxLayout(panel)
        categories = self.editor.get_categories()
        for category in categories:
            label = QPushButton(category)
            label.clicked.connect(self.create_category_handler(category))
            panel_layout.addWidget(label)
        return panel

    def create_category_handler(self, category):
        return lambda: self.editor.select_category(category)

how can i draw different class?

i set 5 classes in it ,but i dont know how to choose these class. it seems no thing happend whether click the class button on the right of the windows

some bug :Load model from model sam_onnx.1080_1920.onnx failed:

Thanks for your nice work!
I don't have any image that it's shape is [1080,1920],therefor i get this erro when i call segment_anything_annotator.py :

Traceback (most recent call last):
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\segment_anything_annotator.py", line 26, in <module>
    editor = Editor(
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\editor.py", line 58, in __init__
    self.onnx_helper = OnnxModels(onnx_models_path)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 16, in __init__
    self.set_image_resolution(image_width, image_height)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 29, in set_image_resolution
    self.__init_model(onnx_model_path)
  File "D:\Users\26655\Desktop\CAT\library\others\SAM\salt\onnx_model.py", line 19, in __init_model
    self.ort_session = onnxruntime.InferenceSession(
  File "C:\Users\26655\.conda\envs\pt2\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 360, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "C:\Users\26655\.conda\envs\pt2\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 397, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.NoSuchFile: [ONNXRuntimeError] : 3 : NO_SUCHFILE : Load model from model data\sam_onnx.1080_1920.onnx failed:Load model model data\sam_onnx.1080_1920.onnx failed. File doesn't exist

It's clear that it happens when class Editor initializes

Add Box Input

Add ability to provide box input along with point input (with nice looking mousedown animation). Though I'm not sure if it's needed to be honest, point input works really well in general.

There is a problem with the reshape node exported by ONNX. I checked the network structure and found that it is only a simple reshape node. Have you encountered this problem

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_149' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,7,256}, requested shape:{1,10,8,32}

Package libffi conflict

Package libffi conflicts for:
certifi=2022.12.7 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
libffi=3.4.2
wheel=0.38.4 -> python[version='>=3.10,<3.11.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
setuptools=65.6.3 -> python[version='>=3.7,<3.8.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']
pip=23.0.1 -> python[version='>=3.9,<3.10.0a0'] -> libffi[version='>=3.2.1,<3.3a0|>=3.3,<3.4.0a0']

Unable to adjust the transparency of the predicted mask

Thank you for sharing the awesome work!
I think it is necessary to adjust the transparency of the current predicted mask during the annotation process, so that it is easier to check the accuracy of the mask boundaries.
Not much code to adjust,

def step_up_transparency(self):

Amend the following after the line of code:

    def step_up_transparency(self):
        self.display = self.image_bgr.copy()
        self.du.increase_transparency()

        if self.curr_inputs.curr_mask is not None:
            self.display = self.du.draw_points(
            self.display, self.curr_inputs.input_point, self.curr_inputs.input_label)
            self.display = self.du.overlay_mask_on_image(self.display, self.curr_inputs.curr_mask)
        
        if self.show_other_anns:
            self.draw_known_annotations()

    def step_down_transparency(self):
        self.display = self.image_bgr.copy()
        self.du.decrease_transparency()

        if self.curr_inputs.curr_mask is not None:
            self.display = self.du.draw_points(
            self.display, self.curr_inputs.input_point, self.curr_inputs.input_label)
            self.display = self.du.overlay_mask_on_image(self.display, self.curr_inputs.curr_mask)
        
        if self.show_other_anns:
            self.draw_known_annotations()

Pink line generated from the top left corner of the image

When I view the image, there is a pink line generated from the top left corner of the image. I tried to investigate the issue by printing variable "sc" from the file dataset_explorer.py, and I found a random value "6.91669498947497e-310, 6.91669498947497e-310" generated at the beginning. To overcome this issue, I added this line "sc = [x for x in sc if x > 0.1]" in the file dataset_explorer.py.

Screenshot:

Please refer to the attached screenshots for better understanding.

  1. Screenshot from 2023-05-02 15-36-22
  2. Screenshot from 2023-05-02 15-36-29

Expected Outcome:

The pink line should not be generated from the top left corner of the image.

Steps to Reproduce:

  1. Run segment_anything_annotator.py.
  2. Label any object.
  3. Select the label name and click on the save button, then click on the add button.
  4. The error will be found on the image.

A bug occur when I want to change the categories

I want to change the categories from 'cat' and 'dog' to other labels, but as I want to change the categories with the command:
python segment_anything_annotator.py --onnx-model-path sam_onnx.onnx --dataset-path ../shot --categories i,j,k,l,o

It turns out that the categories in the interface are still cat and dog, and I cannot change the categories whatever I type in.
So I wonder if someone who once faced the same problem can give me an explanation and help me slove this problem, thank you so much!!! ^-^

how to get annotations from mask?

thanks for the amazing work, may i know where is the code for converting the mask into segmented support format coco or yolo format data? i m trying to load my custom detector & want to detect objects & get its coordinates. thanks

Add Delete Functionality.

Need to think of a UI (how to select the annotation) and then implement corresponding deletion functionality.

Error when clicking middle mouse button

Hi!
when I press the middle mouse button (usually by accident) I get this error:

QLayout::addChildLayout: layout "" already has a parent
Traceback (most recent call last):
  File "/home/maciej/PycharmProjects/salt/salt/interface.py", line 97, in mousePressEvent
    self.editor.add_click([int(x), int(y)], label, selected_annotations)
UnboundLocalError: local variable 'label' referenced before assignment
Aborted (core dumped)

I will try to add fix of these bug later

Error for low quality image

Error Message:

...//salt/salt/display_utils.py", line 33, in __convert_ann_to_mask
    rles = coco_mask.frPyObjects(poly, height, width)
  File "pycocotools/_mask.pyx", line 308, in pycocotools._mask.frPyObjects
Exception: input type is not supported.
 .../salt/salt/display_utils.py", line 33, in __convert_ann_to_mask
    rles = coco_mask.frPyObjects(poly, height, width)
  File "pycocotools/_mask.pyx", line 294, in pycocotools._mask.frPyObjects
TypeError: Argument 'bb' has incorrect type (expected numpy.ndarray, got list)

My image is pretty obscure so sometimes SAM segmentation include very small mask which has poly of 4 elements.

This seems to coincide with this issue.
cocodataset/cocoapi#139

Maybe there need to be some changes that make len(poly[i]) > 4

TypeError: 'torch._C.Value' object is not iterable

When I use the generate_onnx.py to generate the onnx file of SAM, it would return the exception:
Traceback (most recent call last):
File "generate_onnx.py", line 177, in
run_export(
File "generate_onnx.py", line 149, in run_export
torch.onnx.export(
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/init.py", line 305, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 118, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 719, in _export
_model_to_graph(model, args, verbose, input_names,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 503, in _model_to_graph
graph = _optimize_graph(graph, operator_export_type,
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 232, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/init.py", line 354, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/utils.py", line 1061, in _run_symbolic_function
return symbolic_fn(g, *inputs, **attrs)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/symbolic_opset13.py", line 267, in repeat_interleave
return torch.onnx.symbolic_opset9.repeat_interleave(g, self, repeats, final_dim)
File "/HOME/scz0bck/.conda/envs/seg-tool/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py", line 2152, in repeat_interleave
for idx, r_split in enumerate(r_splits):
TypeError: 'torch._C.Value' object is not iterable
(Occurred when translating repeat_interleave).
I don‘t konw how to solve the exception. May I beg a favour?Thanks for your help.

Suggestion - Integrate MobileSAM into the pipeline for lightweight and faster inference

Reference: https://github.com/ChaoningZhang/MobileSAM

Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.

MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

image

image

Best Wishes,

Qiao

Some GUI problems

Thanks for your tools. I am trying to use this tool in my equipment. And i meet some problems during run the segment_anything_annotator.py.

I have a server with GPU, which is a Linux system without GUI. It's totally terminal system. I use my Macbook to link the server with SSH tool. And I have created embedding and onnx model in server.

When I want to annotate my images, the PyQt doesn't work well and export error:Could not connect to any X display and QLibraryPrivate::unload succeeded on "/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/libqxcb.so". I have tried to fix it by reinstall pyqt and install PYQT Integration in my VSCode. But it is not useful for me.

I would like to ask two questions:

  1. How can i solve my PyQt problems? Can you give me some advices?
  2. How to use your tool in Macbook and GPU server more more convenient?

I am looking forward to your answer

remote labeling

Sorry for the ignorance, can I run this remotely? This is, using a sever - client paradigm or similarly?

Failed to run, maybe caused by parsing annotations.json

python segment_anything_annotator.py --onnx-models-path models --dataset-path seg_imgs --categories dog
Traceback (most recent call last):
  File "segment_anything_annotator.py", line 26, in <module>
    editor = Editor(
  File "C:\pyprojects\salt\salt\editor.py", line 45, in __init__
    self.dataset_explorer = DatasetExplorer(
  File "C:\pyprojects\salt\salt\dataset_explorer.py", line 126, in __init__
    max(self.coco_json["annotations"], key=lambda x: x["id"])["id"] + 1
ValueError: max() arg is an empty sequence

image
image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.