yformer / efficientsam Goto Github PK
View Code? Open in Web Editor NEWEfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything
License: Apache License 2.0
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything
License: Apache License 2.0
In this code:
def run_ours_box(img_path, pts_sampled, model):
bbox = torch.reshape(torch.tensor(pts_sampled), [1, 1, 2, 2])
bbox_labels = torch.reshape(torch.tensor([2, 3]), [1, 1, 2])
image = cv2.imread(image_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predicted_logits, predicted_iou = model(
img_tensor[None, ...].cuda(),
bbox.cuda(),
bbox_labels.cuda(),
)
... what are the bbox_labels doing? Why are they [2,3], and why does the model need them? I couldn't find that described in the paper either. Maybe I missed it...
thx!
Thanks for your great work.
Here is the Segment Anything CPP Wrapper for macOS and it corresponds to MobileSAM, EdgeSAM, EfficientSAM, HQ-SAM, and Segment Anything models.
Please let us know your opinion.
Hello ! I found a problem when I test a image from coco datasets. For the same point prompt, the output mask has different result with tiny and small model, as the following screenshots. The coordinate of the point prompt is (380,250).
I would like to be able to get a full person segmentation with EfficientSAM-S model, how can I achieve this?
Thank you for your suggestions
Dear EfficientSAM Contributors,
I hope this message finds you well. I have been closely following the development of EfficientSAM and am thoroughly impressed with the strides made in efficient image segmentation. The recent release of the torchscript version and the accompanying Colab notebook have been particularly helpful.
However, I am curious about the model's performance and adaptability in low-resource environments, which is often a challenge in the field of computer vision. Specifically, I am interested in understanding how EfficientSAM fares on devices with limited computational power and memory constraints.
Could you provide insights or benchmarks on the following aspects?
I believe addressing these queries would greatly benefit researchers and practitioners working in regions with limited access to high-end hardware, thereby broadening the scope of EfficientSAM's applicability.
Thank you for your time and the remarkable work on this project. I look forward to your response and any guidance you can provide.
Best regards,
yihong1120
Thank you for you have released the project code , it's a pretty good work ! Could you upload the notebook code of the saliency segment code
? Looking forward to you reply!
Great work! I was wondering what's happening with SqueezeSAM as it's been added to this repo and then subsequently removed. I'd like to try it for its salient segmentation features. Thanks!
Hi there - I noticed the new notebooks added for segment anything example. I am wondering if you can give any advice for how to finetune it on a custom dataset for semantic segmentation? Will there be another notebook released which will cover this?
Thanks
Why do I find that the JIT model provided initially performs significantly better than the current PT model or the converted ONNX model during testing? Additionally, I noticed three nodes in the JIT model that are missing in the PT models.
Hi, firstly very appreciate your work. However, I noticed that the jit encapsulation made it impossible for me to inspect the intermediate output (e.g. encoder feature, decoder feature). Is there any possibility for you to release the model code?
Hello author. I just tried your latest uploaded code(sample_inference.py) and it runs perfectly. However, when I set the device to cuda, it will prompt that there is not enough graphics memory. May I ask why this is?
Hello, thank you for your nice job and I want to use effieicnet Sam to Identify the salientest object in the picture and mask it.
But I didn't find this keyword when I searched in the github repository
Best~!
Hello author,
when running the sample_inference.py you provided the day before yesterday, it can run on the CPU but shows insufficient graphics memory when placed on the GPU. May I ask why this is happening? Additionally, I would like to ask if it is possible to directly call the
build_efficient_sam if I want to make fine-tuning on my own dataset?
`input_points = torch.tensor([[[[580, 350], [650, 350]]]])
input_labels = torch.tensor([[[1, 1]]])
redicted_logits, predicted_iou = model(
sample_image_tensor[None, ...],
input_points,
input_labels,
)
`
Hello, I'm grateful for your research.
I have a question about checkpoints. Is the change in weights of efficentsam compared to sam only the image encoder part.
Or can I get the correct result by processing the weights of the image encoder of efficentsam with the weights of the prompt encoder and mask decoder of the original sam?
Would it be possible to export this model to CoreML? In my attempt I got:
ValueError: Torch var fusion_type.1 not found in context
Thanks for your great work! I want to know how the Cross-Attention Decoder is implemented. Do we need to determine the positional relationship between mask tokens and unmask tokens?
Thank you for your amazing research. I would like to use a text prompt. However, when I examined the code, there was no code related to the text prompt. I'm curious if you have any plans to add it in the future or if there's a specific reason why text prompts are not available. Thank you.
the efficientSAM model do not support input multi-bounding box?
I have tried to export the encoder of the model to ONNX, but it informs me that the export has failed. Can anyone who has done relevant work give some advice?
Hello Yunyang @yformer ! Thanks for your nice work! We've alrealy support grounded-efficient-sam demo under Grounded-Segment-Anything! Hope for the training code release!
Congrats on the release of the EfficientSAM models! I noticed the files were hosted in Dropbox and created a quick fork to host in GitHub itself (using the XetData extension):
https://github.com/xetdata/EfficientSAM
This lets you work with the models locally, keep them in the Git repo, and just push up to GitHub without having to upload in separate place. I'd love to work with y'all to do the same with this repo too if you're interested!
'keys and values derive from both unmasked features from encoder and masked features.' But I have no idea where do the masked features come from? what do the 'masked features' refer to? And how did you merge these with the unmasked features from the encoder? I can't have a clear idea from the paper.
Thank you a lot!
Dear author, thank for releasing your model and Colab notebook on example usage. But in your Colab notebook, you demonstrated how to perform inference with box prompt and point prompt inputs. What about "segment everything" and "salient instance segmentation"? How do we do that? Can you please provide some examples.
Thank you for sharing the great work ! I want to export the onnx model and when I run export_to_onnx.py
, the following error encounters:
torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::tile' to ONNX opset version 18 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues.
Based on the documents of ONNX supported TorchScript operators, the aten:tile is supported since opset version 13. Could you give me some suggestions ?
When I use the nvidia 3090 GPU (Memory 24576M) to run the “EfficientSAM_segment_everything_example” code, the model used is vitt, GRID_SIZE = 32, which indicates that the graphics card memory is not enough and cannot run。
when I adjust parameter 32 to 16, it can be run but the segment effect is not good
I tested the ‘S’ model on my own dataset.
I intended to use negative points to eliminate some confusing regions.
However, it appears that the model shows almost no response to the negative points.
I quickly glanced through the paper but found no relevant information.
Is this behavior of the model by design, or have I overlooked any important detail?
There are no training codes with SAMI Module since it's necessary for training custom datasets. Finetuning config settings and codes are also necessary for reproduce the results. Thanks.
Hello, I'm grateful for your research. I tried segment-anything with the code you shared. I though it would be fast, but it shows a very slow speed with an average of 13000ms. Can you tell me the reason why?
from efficient_sam.build_efficient_sam import build_efficient_sam_vits
import zipfile
with zipfile.ZipFile("weights/efficient_sam_vits.pt.zip", 'r') as zip_ref:
zip_ref.extractall("weights")
efficient_sam_vits_model = build_efficient_sam_vits()
efficient_sam_vits_model.eval()
import os
image_path = "dataset/data1"
image_list = os.listdir(image_path)
import time
for image_name in image_list:
image_path1 = os.path.join(image_path, image_name)
st = time.time()
mask_efficient_sam_vits = run_everything_ours(image_path1, efficient_sam_vits_model)
et = time.time()
print('inference time:', (et - st)*1000)
Output:
inference time: 17762.072563171387
inference time: 13303.216695785522
inference time: 13251.076221466064
inference time: 12843.31202507019
image_size : 640x640
GPU : RTX A6000
Hey,
Thanks for your work and model!
Is there a plan to integrate this model to work with the segment_anything package, specifically with SamAutomaticMaskGenerator?
For example, this is how we use it today:
self.sam = sam_model_registry["vit_h"](checkpoint=model_checkpoint_path).to(self.device)
self.mask_generator = SamAutomaticMaskGenerator(self.sam, pred_iou_thresh=0.88, stability_score_thresh=0.8, min_mask_region_area=200)
will we be able to use the same with the EfficiantSAM model?
Thanks!
Hi author,
When you do pretraining, how do you generate masked tokens? Initiated by random?
Thanks for the author's effort, this is a very meaningful work, and it has been a great help to me.
I found that there is a parameter decoder_max_num_input_points available in the EfficientSAM, which specifies the maximum number of points accepted by the model. I would like to know why 6 is set here, and if we want to change this parameter, is there a maximum value?
Moreover, the current model does not seem to support negative sample points and mask input. Can you give more details about EfficientSAM's training?
I've tested your torchscript models on linux with A6000 gpu.
Regression 1:
Running the original sam huge model, I get 3 output masks from a point prompt, but efficient-sam only outputs 1 mask.
Regression 2:
The original sam offers 2 stage inference using SamPredictor.setimage and SamPredictor.predict but efficient-sam has only 1 stage of inference.
Speed:
Original sam (huge) inference stage 1 - about 450ms, stage 2 - 25ms. This means interactive segmentation adjusting points/boxes can be done at 40fps.
Efficient sam (small) inference one stage - about 105ms. Overall 4-5 times faster but now rather blocky for interactive picking.
Efficient sam tiny - about 75ms. Mask quality not so great.
I haven't looked into the mask quality much and whether I should be comparing one of the smaller sam models. I'd be interested to know your thoughts/plans. Thanks for the work.
Hi,
I found that in the paper, there is evaluation results on COCO dataset,
would you mind providing the corresponding code?
Thanks
When I run EfficientSAM_example.py file, a RuntimeError appears: expected scale type float but found double
This would involve adding the optional parameter to EfficientSam.forward
, and then passing it to EfficientSam.predict_masks
.
def forward(
self,
batched_images: torch.Tensor,
batched_points: torch.Tensor,
batched_point_labels: torch.Tensor,
scale_to_original_image_size: bool = True,
multimask_output: bool = True
) -> Tuple[torch.Tensor, torch.Tensor]:
"""
Predicts masks end-to-end from provided images and prompts.
If prompts are not known in advance, using SamPredictor is
recommended over calling the model directly.
Arguments:
batched_images: A tensor of shape [B, 3, H, W]
batched_points: A tensor of shape [B, num_queries, max_num_pts, 2]
batched_point_labels: A tensor of shape [B, num_queries, max_num_pts]
multimask_output: If True, generate multiple masks for each query. Otherwise, generate one mask per query.
Returns:
A list tuples of two tensors where the ith element is by considering the first i+1 points.
low_res_mask: A tensor of shape [B, 256, 256] of predicted masks
iou_predictions: A tensor of shape [B, max_num_queries] of estimated IOU scores
"""
batch_size, _, input_h, input_w = batched_images.shape
image_embeddings = self.get_image_embeddings(batched_images)
return self.predict_masks(
image_embeddings,
batched_points,
batched_point_labels,
multimask_output=multimask_output,
input_h=input_h,
input_w=input_w,
output_h=input_h if scale_to_original_image_size else -1,
output_w=input_w if scale_to_original_image_size else -1,
)
I'm happy to make a PR for this, but I figure it may be easier to just throw this in as part of the ongoing updates you all are making.
Thanks for releasing this code and updating it so frequently!
Edit: I tried the code above, and using multimask_output=False
seems to be giving me broken masks, so I'm probably missing something and this may be more involved than I'd thought. The bug could also be in my postprocessing code.
For the dog example image and points, this is what I get with, and without multimask:
Hi, it's a nice work and we're interested in it!
Could you release the pre-trained models (ViT-S or ViT-Ti)? We believe the pre-trained models also matter for the community! 😊
Hello I was wondering what is the expected value range for the input tensors. It seems that it is float32, [0,1], but asking just in case.
Thanks a lot for your help.
Regards,
CArlos
HEllo, I would like to use masks as input prompts to EfficientSAm, but with what I have seen, it seems that it is not possible with the current implementation.
Could I use the outputs of the original SAM prompt encoder and combine them with the result of the VIT small to get the same results as with all the "official" EfficientSAM stuff?
Regards,
Carlos
Thanks for sharing this repo.
On the demo collab file, how we can pass multi bounding boxes to the model as prompt?
I have a widget which gets the bboxs by the users, and I want to pass it to the model like FastSAM.
File "Grounded-Segment-Anything/EfficientSAM/app.py", line 60, in efficient_sam_box_prompt_segment
predicted_logits, predicted_iou = model(
^^^^^^
File "/root/anaconda3/envs/ml_zb/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: shape '[0, 3, 1, 2]' is invalid for input of size 1572864
after add env like this:
export PYTORCH_NVFUSER_DISABLE=fallback && python EfficientSAM/app.py
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/anaconda3/envs/ml_zb/lib/python3.11/site-packages/gradio/queueing.py", line 501, in process_events
response = await self.call_prediction(awake_events, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/ml_zb/lib/python3.11/site-packages/gradio/queueing.py", line 465, in call_prediction
raise Exception(str(error) if show_error else None) from error
Exception: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: thread_predicates_.find(tv_inp) != thread_predicates_.end() INTERNAL ASSERT FAILED at "../third_party/nvfuser/csrc/lower_thread_predicate.cpp":221, please report a bug to PyTorch. Thread predicate map was not initialized, couldn't find T6_l[ 0 ]
can anyone help me ?
The "Box segmentatoin" section doesn't work at all. There are multiple run-time errors.
Hello! When I use multiple prompt inputs, the inference speed becomes slow. When the number of prompt inputs is 1, the inference time is approximately 0.11 seconds. However, when the number of prompt inputs is 30, the inference time exceeds 3 seconds. What could be the reason for this?
In Section 4.1, it seems that only IN1K is used for pretraining. But in Table 1, both SA-1b and IN1K are used in pretraining. Which is the correct one?
Dear author, I see you have released a new model SqueezeSAM. However, I couldn't find much information regarding its performance relative to the other EfficientSAM (Tiny, Small) in your paper or this repo's README. Can you shed some information on this?
Hi Amazing work,
can you upload jit models for Mac ("mps") and cpu?
It will be awesome thank you (:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.