junyi42 / sd-dino Goto Github PK

Thank you for the amazing work! I am trying to visualize the feature maps for dino and SD. Do you have a collab notebook, that I can use to run it?

Hello,I'm sorry to bother you again. I've encountered a version issue. My TensorFlow and Keras versions are 2.13.1, and I'm getting this error. Could you please let me know the Keras version requirements for this code? I couldn't find any helpful answers online, and despite using a global search, I haven't found any occurrences of the "is_tensor" function in the code.
Thanks!

Hi Junyi,

I am confused about how to extract sd features. Actually the file extractor_sd.py seems to output a feature in shape of [1, 1280, 16, 16] without obvious semantic information. And it seems to use the model weights from project ODISE. Could you please provide a script to easily extract and visualize the sd features using publicly available stable diffusion model weights? Thanks a lot!

Hi,

Thanks for this awesome work! 🤩

The image resolution is 518 in the Dinov2-pretrained-model, why you can use the 840.

looking forward your reply.

Hello Junyi, GREAT JOB! It seems that everything works well when calling get_features in extractor_sd.py using cuda:3
but the inference process failed even I change
def inference(model, aug, image, vocab, label_list):
from
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)

pred = demo.predict(np.array(image))
to
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)

demo.model = demo.model.to(torch.device("cuda:3"))

pred = demo.predict(np.array(image))

I guess the main problem lies in wrongly loading the decoder part of the model, but I'm not sure how to fix it.

Hello @Junyi42 ,
Thanks for your contribution. I am facing the an installation issue when running the "pip install -e ." command. This is giving the error as follows:

Emitting ninja build file /BS/keytr_neus/work/supplementary/sd-dino/third_party/Mask2Former/build/temp.linux-x86_64-cpython-39/build.ninja...

error: [Errno 2] No such file or directory: '/BS/keytr_neus/work/supplementary/sd-dino/third_party/Mask2Former/build/temp.linux-x86_64-cpython-39/build.ninja'

ERROR: Failed building wheel for mask2former

ERROR: Could not build wheels for mask2former, which is required to install pyproject.toml-based projects

Please help me in this

Hi,

Thanks for this awesome work! 🤩

DINO and StableDiffusion works have MIT licenses. Is your work also MIT?

Best,
Iago.

Hi, I met the problem when I run pck_spair_pascal.py
Would you mind to telling me how to fix the issue?
Thanks!

Hello, I am very interested in your work, but I encountered some difficulties when setting up the environment. I followed the steps in the README, but there seems to be some problem somewhere, and I don't know how to fix it.

RuntimeError: Panoptic/odise_label_coco_50e.py not available in Model Zoo!
would you mind to telling me how to fix the issue.

Hi!
when running the demo,

src_img_path = "data/images/dog_00.jpg"
trg_img_path = "data/images/dog_59.jpg"
result = process_images(src_img_path, trg_img_path)

I found that the get_mask function cannot return a valid mask but an all-1 matrix. Is this a bug?

if DRAW_DENSE:
                if not Anno:
                    mask1 = get_mask(model, aug, img1, category[0])
                    mask2 = get_mask(model, aug, img2, category[-1])

Hello @Junyi42 , Thanks for your contribution. I ran the "demo_vis_features.ipynb on the dog that was given in the default image folder. My results are coming different than yours. Yours masked pca result was

while I am getting

Also, my clustering is

I didn't change anything in the code only dumped everything from the ipynb to .py file and I am getting these outputs in the results_vis folder in the form of png files.

Hello, I would like to know whether the 2, 5, 8-layer features mentioned in the paper refer to the actual 2, 5, 8 layers or the layers after processing with the UpSample block. Does it mean the results obtained after the UpSample block processing? I find it a bit challenging to understand the feature extraction in the code. I hope to receive your reply. Thank you!

https://github.com/Junyi42/sd-dino/blob/48278d9c3c1cc2386ca08438d527a35dff902c9d/extractor_sd.py#L260C1-L269C100
I guess the dimensions of tensor are [N, HW, C], I don’t know why transformed_tensor here only transforms tensor[0] and discards other samples, so that new features will only have one sample features, i.e., [1, HW, C].

Hi, thanks for sharing the codes.

I found a problem when running the demo codes. I followed all the setup in readme without changing anything, but it seems the download pre-trained weights mismatch the model:

so I got the results which are very different from yours:

This problem also occurs when I run Geoaware-SC. Could you give me some advice on how to solve this?

Hi Junyi,
Could you please provide the code for Object Swapping (with refinement process), current object swapping result seems blur