I2V architecture about vgen

[Steven-SWZhang]

Thank you for your interest in our work.

1. Yes, we are using the LDM method for videos.
2. In the base stage, we input the image (extracting CLIP features and latent represents separately) and combine it with noise to input into the 3D U-Net to get the output video.
3. In practice, we treat the video as a whole during input and use a denoising and diffusion process. For the temporal encoding process, you can refer to the design of our 3D U-Net, Fig.3.
   Thank you.

[Sanath91009]

Thanks for the reply,
2. Input to 3D Unet is 2D noise or 3D noise ?
3. So input to refinement stage pre-trained LDM is resized output of base stage LDM ? Because any LDM takes noise as input and denoises it to generate video if it takes resized video (non-noise) as input, how it is denoises when there is no noise ?