How can I load an adaptor, IP-Adapter

yaseryacoob

I am trying to load https://github.com/tencent-ailab/IP-Adapter to add conditioning in the generation to ScaleCrafter. Is it possible to do that?

Comments

YingqingHe

Yes. It is possible. Simply switch the Stable Diffusion model used in the IP adapter to one with ScaleCrafter. It may require some coding to deal with it. Thanks.

Can you please be a little more clear. If I understand you correctly here is what you suggest 1. Build on the IP-Adaptor code, which currently has something like ``` pipe = StableDiffusionXLCustomPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, add_watermarker=False) ip_model = IPAdapterPlusXL(pipe, image_encoder_path, ip_ckpt, device, num_tokens=16) ``` 2. Replace the stabilityai/stable-diffusion-xl-base-1.0 with ScaleCrafter Model. Except I don't see a ScaleCrafterModel anywhere, or if it can be saved given your code. Also not sure how the IPAdapter would deal with higher resolution you generate? If you meant the opposite, which is loading the IPAdaptor into the ScaleCrafter code, then it appears more tricky since there is overlapping code to be figured out between your code and the IPAdaptor Please provide me with more details as I don't have experience with integrating multiple models.