anminhhung / segment-anything-clip

Connecting segment-anything's output masks with the CLIP model; Awesome-Segment-Anything-Works

Conect Segment-Anything with CLIP

We aim to classify the output masks of segment-anything with the off-the-shelf CLIP models. The cropped image corresponding to each mask is sent to the CLIP model.

Other Nice Works

Editing-Related Works

1. sail-sg/EditAnything
2. IDEA-Research/Grounded-Segment-Anything
3. geekyutao/Inpaint-Anything

Segmentation-Related Works

3. maxi-w/CLIP-SAM
4. Curt-Park/segment-anything-with-clip
5. kadirnar/segment-anything-video
6. fudan-zvg/Semantic-Segment-Anything
7. continue-revolution/sd-webui-segment-anything
8. RockeyCoss/Prompt-Segment-Anything
9. ttengwang/Caption-Anything
10. ngthanhtin/owlvit_segment_anything
11. lang-segment-anything
12. helblazer811/RefSAM
13. Hedlen/awesome-segment-anything
14. ziqi-jin/finetune-anythin
15. ylqi/Count-Anything

Todo

1. We plan connect segment-anything with MaskCLIP.
2. We plan to finetune on the COCO and LVIS datasets.

Run Demo

Download the sam_vit_h_4b8939.pth model from the SAM repository and put it at ./SAM-CLIP/. Follow the instructions to install segment-anything and clip packages using the following command.

cd SAM-CLIP; pip install -e .
pip install git+https://github.com/openai/CLIP.git

Then run the following script:

sh run.sh

Example

Input an example image and a point (250, 250) to the SAM model. The input image and output three masks as follows:

The three masks and corresponding predicted category are as follows:

You can change the point location at L273-274 of scripts/amp_points.py.

## input points 
input_points_list = [[250, 250]]
label_list = [1]