RuntimeError: Input type (c10::Half) and bias type (float) should be the same about diffusers

[tolgacangoz]

Hi Kadir, thanks for bringing up this issue!
I noticed that a similar issue was also mentioned in a comment on the implementation PR after it had been merged: #6074 (comment). I'd like to tag @manuelbrack here as well for his information.
As I see, to ensure dtype alignment between image
and vae in such situations, StableDiffusionXLReferencePipeline preferred this before vae's encoding:

diffusers/examples/community/stable_diffusion_xl_reference.py

Lines 156 to 160 in 5823736

	if self.vae.dtype == torch.float16 and self.vae.config.force_upcast:
	self.upcast_vae()
	refimage = refimage.to(next(iter(self.vae.post_quant_conv.parameters())).dtype)
	if refimage.dtype != self.vae.dtype:
	refimage = refimage.to(dtype=self.vae.dtype)

[yiyixuxu]

@standardAI would you like to open a PR to fix this?