bahjat-kawar / ddrm Goto Github PK

Dear Authors,

Many thanks for your excellent work. And I am very interested in your work.

May I ask the approximate time of code release?

Thanks

Hi, we are having a problem with the code. When we copy python main.py --ni --config imagenet_256.yml --doc imagenet --timesteps 20 --eta 0.85 --etaB 1 --deg sr4 --sigma_0 0.05 in the prompt this is the error that occurs:

C:\Users\aless>python main.py --ni --config imagenet_256.yml --doc imagenet --timesteps 20 --eta 0.85 --etaB 1 --deg sr4 --sigma_0 0.05
INFO - main.py - 2023-04-21 16:50:48,504 - Using device: cpu
INFO - main.py - 2023-04-21 16:50:48,504 - Writing log file to exp\logs\imagenet
INFO - main.py - 2023-04-21 16:50:48,504 - Exp instance id = 976
INFO - main.py - 2023-04-21 16:50:48,504 - Exp comment =
ERROR - main.py - 2023-04-21 16:51:11,266 - Traceback (most recent call last):
File "C:\Users\aless\main.py", line 164, in main
runner.sample()
File "C:\Users\aless\runners\diffusion.py", line 135, in sample
model.load_state_dict(torch.load(ckpt, map_location=self.device))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\aless\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\serialization.py", line 797, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\aless\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\serialization.py", line 283, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

Can somebody help us? We are new to coding, so we don't know where to start. Thank you very much

Hello, thank you Bahjat very much for uploading this code and making it so easy to reproduce your group's results. It is truly much appreciated. I had a question regarding the possibility for defining a new H function class (in the svd_replacement.py code), and was hoping you might be able to offer some insight. What is the general thought process you followed in order to define the functions for Vt, V, U, Ut, and the singulars in this code? If I wanted to do it for another linear function (i.e. only super-resolve right half of image or specific pixel region within image), how could I do this? I have defined the H-matrix for the operation I want to do, but it is not clear to me how to define the class so that it works properly with the rest of the code. Would you be able to offer any useful resources or guidelines to follow for help with this? Thank you!

Hi, I tried the CelebA example from the README, using your environment and the provided images, but I get the error

ERROR - main.py - 2022-06-17 10:28:39,626 - Traceback (most recent call last):
  File "/home/wizard/dev/ddrm/main.py", line 164, in main
    runner.sample()
  File "/home/wizard/dev/ddrm/runners/diffusion.py", line 163, in sample
    self.sample_sequence(model, cls_fn)
  File "/home/wizard/dev/ddrm/runners/diffusion.py", line 237, in sample_sequence
    H_funcs = SuperResolution(config.data.channels, config.data.image_size, blur_by, self.device)
  File "/home/wizard/dev/ddrm/functions/svd_replacement.py", line 179, in __init__
    self.U_small, self.singulars_small, self.V_small = torch.svd(H, some=False)
RuntimeError: cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSgesvdj( handle, jobz, econ, m, n, A, lda, S, U, ldu, V, ldv, static_cast<float*>(dataPtr.get()), lwork, info, params)`

Probably there is a mismatch between the CUDA versions. Which CUDA did you use for your experiments?

Hi, thank you for this excellent work. I am interested in real-world image restoration, and I want to get some advice on how to apply the codes to real-world cases. I tried to use Imagenet ood dataset for this purpose, but it seems that a class lable is needed for each input image. (Please correct me if my understanding is wrong.)

Thanks in advance for your reply!

hello
thanks for you great works
You have release code but i do not find some demos about Colorization, so can you give me a examle for how to run it for Colorization?
thanks!
@jiamings @bahjat-kawar

Hi. Thank you for sharing!
I am trying to run with :
--ni --config celeba_hq.yml --doc celeba --timesteps 20 --eta 0.85 --etaB 1 --deg sr4 --sigma_0 0.05 -i celeba_hq_sr4_sigma_0.05
but I am getting:
Attribute Error: Can't pickle local object 'Diffusion.sample_sequence..seed_worker'
in the runners/diffusion.py?
is there any way around this?
thank you

I'm trying to test the 256 ImageNet model on the deblurring task on the OOD data you provide in your adiacent repository. I'm getting this error:

ERROR - main.py - 2022-07-25 10:25:13,026 - Traceback (most recent call last):
  File "/Users/mbejan/Documents/diffusion/ddrm/main.py", line 164, in main
    runner.sample()
  File "/Users/mbejan/Documents/diffusion/ddrm/runners/diffusion.py", line 161, in sample
    self.sample_sequence(model, cls_fn)
  File "/Users/mbejan/Documents/diffusion/ddrm/runners/diffusion.py", line 249, in sample_sequence
    for x_orig, classes in pbar:
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 438, in __iter__
    return self._get_iterator()
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 384, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1048, in __init__
    w.start()
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/mbejan/opt/anaconda3/envs/ddrm/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Diffusion.sample_sequence.<locals>.seed_worker'

This is the script that creates the behaviour from above:

python main.py --ni \
  --config imagenet_256_cc.yml \
  --doc ood \
  --timesteps 20 \
  --eta 0.85 \
  --etaB 1 \
  --deg deblur_uni \
  --sigma_0 0.05 \

My imagenet_256_cc.yml is the same as the one your provide apart from the out_of _distribution argument, which is set to true.

Hi, thank you for awesome work.
I'm interested in the theoretical contents in the original paper, and taking hard time to understand
and derivate the equation 17.
Can you explain about the derivation of equation 17?

Hi
@jiamings @bahjat-kawar
I have retrain the diffusion model from guide-diffusion, and my setting as follow:
MODEL_FLAGS="--image_size 256 --num_channels 256 --num_res_blocks 2 --learn_sigma True --use_scale_shift_norm true --attention_resolutions 32,16,8 --num_head_channels 64"
DIFFUSION_FLAGS="--resblock_updown True --diffusion_steps 1000 --noise_schedule linear --rescale_learned_sigmas False --rescale_timesteps False "
TRAIN_FLAGS="--lr 1e-4 --microbatch 4 --dropout 0.1"

but it was wrong with this ckpt:
Missing key(s) in state_dict: "temb.dense.0.weight", "temb.dense.0.bias", "temb.dense.1.weight", "temb.dense.1.bias", "conv_in.weight", "conv_in.bias", "down.0.block.0.norm1.weight", "down.0.block.0.norm1.bias", "down.0.block.0.conv1.weight", "down.0.block.0.conv1.bias", "down.0.block.0.temb_proj.weight", "down.0.block.0.temb_proj.bias", "down.0.block.0.norm2.weight", "down.0.block.0.norm2.bias", "down.0.block.0.conv2.weight", "down.0.block.0.conv2.bias", "down.0.block.1.norm1.weight", "down.0.block.1.norm1.bias", "down.0.block.1.conv1.weight", "down.0.block.1.conv1.bias", "down.0.block.1.temb_proj.weight", "down.0.block.1.temb_proj.bias", "down.0.block.1.norm2.weight", "down.0.block.1.norm2.bias", "down.0.block.1.conv2.weight", "down.0.block.1.conv2.bias", "down.0.downsample.conv.weight", "down.0.downsample.conv.bias", "down.1.block.0.norm1.weight", "down.1.block.0.norm1.bias", "down.1.block.0.conv1.weight", "down.1.block.0.conv1.bias", "down.1.block.0.temb_proj.weight", "down.1.block.0.temb_proj.bias", "down.1.block.0.norm2.weight", "down.1.block.0.norm2.bias", "down.1.block.0.conv2.weight", "down.1.block.0.conv2.bias", "down.1.block.1.norm1.weight", "down.1.block.1.norm1.bias", "down.1.block.1.conv1.weight", "down.1.block.1.conv1.bias", "down.1.block.1.temb_proj.weight", "down.1.block.1.temb_proj.bias", "down.1.block.1.norm2.weight", "down.1.block.1.norm2.bias", "down.1.block.1.conv2.weight", "down.1.block.1.conv2.bias", "down.1.downsample.conv.weight", "down.1.downsample.conv.bias", "down.2.block.0.norm1.weight", "down.2.block.0.norm1.bias", "down.2.block.0.conv1.weight", "down.2.block.0.conv1.bias", "down.2.block.0.temb_proj.weight", "down.2.block.0.temb_proj.bias", "down.2.block.0.norm2.weight", "down.2.block.0.norm2.bias", "down.2.block.0.conv2.weight", "down.2.block.0.conv2.bias", "down.2.block.0.nin_shortcut.weight", "down.2.block.0.nin_shortcut.bias", "down.2.block.1.norm1.weight", "down.2.block.1.norm1.bias", "down.2.block.1.conv1.weight", "down.2.block.1.conv1.bias", "down.2.block.1.temb_proj.weight", "down.2.block.1.temb_proj.bias", "down.2.block.1.norm2.weight", "down.2.block.1.norm2.bias", "down.2.block.1.conv2.weight", "down.2.block.1.conv2.bias", "down.2.downsample.conv.weight", "down.2.downsample.conv.bias", "down.3.block.0.norm1.weight", "down.3.block.0.norm1.bias", "down.3.block.0.conv1.weight", "down.3.block.0.conv1.bias", "down.3.block.0.temb_proj.weight", "down.3.block.0.temb_proj.bias", "down.3.block.0.norm2.weight", "down.3.block.0.norm2.bias", "down.3.block.0.conv2.weight", "down.3.block.0.conv2.bias", "down.3.block.1.norm1.weight", "down.3.block.1.norm1.bias", "down.3.block.1.conv1.weight", "down.3.block.1.conv1.bias", "down.3.block.1.temb_proj.weight", "down.3.block.1.temb_proj.bias", "down.3.block.1.norm2.weight", "down.3.block.1.norm2.bias", "down.3.block.1.conv2.weight", "down.3.block.1.conv2.bias", "down.3.downsample.conv.weight", "down.3.downsample.conv.bias", "down.4.block.0.norm1.weight", "down.4.block.0.norm1.bias", "down.4.block.0.conv1.weight", "down.4.block.0.conv1.bias", "down.4.block.0.temb_proj.weight", "down.4.block.0.temb_proj.bias", "down.4.block.0.norm2.weight", "down.4.block.0.norm2.bias", "down.4.block.0.conv2.weight", "down.4.block.0.conv2.bias", "down.4.block.0.nin_shortcut.weight", "down.4.block.0.nin_shortcut.bias", "down.4.block.1.norm1.weight", "down.4.block.1.norm1.bias", "down.4.block.1.conv1.weight", "down.4.block.1.conv1.bias", "down.4.block.1.temb_proj.weight", "down.4.block.1.temb_proj.bias", "down.4.block.1.norm2.weight", "down.4.block.1.norm2.bias", "down.4.block.1.conv2.weight", "down.4.block.1.conv2.bias", "down.4.attn.0.norm.weight", "down.4.attn.0.norm.bias", "down.4.attn.0.q.weight", "down.4.attn.0.q.bias", "down.4.attn.0.k.weight", "down.4.attn.0.k.bias", "down.4.attn.0.v.weight", "down.4.attn.0.v.bias", "down.4.attn.0.proj_out.weight", "down.4.attn.0.proj_out.bias", "down.4.attn.1.norm.weight", "down.4.attn.1.norm.bias", "down.4.attn.1.q.weight", "down.4.attn.1.q.bias", "down.4.attn.1.k.weight", "down.4.attn.1.k.bias", "down.4.attn.1.v.weight", "down.4.attn.1.v.bias", "down.4.attn.1.proj_out.weight", "down.4.attn.1.proj_out.bias", "down.4.downsample.conv.weight", "down.4.downsample.conv.bias", "down.5.block.0.norm1.weight", "down.5.block.0.norm1.bias", "down.5.block.0.conv1.weight", "down.5.block.0.conv1.bias", "down.5.block.0.temb_proj.weight", "down.5.block.0.temb_proj.bias", "down.5.block.0.norm2.weight", "down.5.block.0.norm2.bias", "down.5.block.0.conv2.weight", "down.5.block.0.conv2.bias", "down.5.block.1.norm1.weight", "down.5.block.1.norm1.bias", "down.5.block.1.conv1.weight", "down.5.block.1.conv1.bias", "down.5.block.1.temb_proj.weight", "down.5.block.1.temb_proj.bias", "down.5.block.1.norm2.weight", "down.5.block.1.norm2.bias", "down.5.block.1.conv2.weight", "down.5.block.1.conv2.bias", "mid.block_1.norm1.weight", "mid.block_1.norm1.bias", "mid.block_1.conv1.weight", "mid.block_1.conv1.bias", "mid.block_1.temb_proj.weight", "mid.block_1.temb_proj.bias", "mid.block_1.norm2.weight", "mid.block_1.norm2.bias", "mid.block_1.conv2.weight", "mid.block_1.conv2.bias", "mid.attn_1.norm.weight", "mid.attn_1.norm.bias", "mid.attn_1.q.weight", "mid.attn_1.q.bias", "mid.attn_1.k.weight", "mid.attn_1.k.bias", "mid.attn_1.v.weight", "mid.attn_1.v.bias", "mid.attn_1.proj_out.weight", "mid.attn_1.proj_out.bias", "mid.block_2.norm1.weight", "mid.block_2.norm1.bias", "mid.block_2.conv1.weight", "mid.block_2.conv1.bias", "mid.block_2.temb_proj.weight", "mid.block_2.temb_proj.bias", "mid.block_2.norm2.weight", "mid.block_2.norm2.bias", "mid.block_2.conv2.weight", "mid.block_2.conv2.bias", "up.0.block.0.norm1.weight", "up.0.block.0.norm1.bias", "up.0.block.0.conv1.weight", "up.0.block.0.conv1.bias", "up.0.block.0.temb_proj.weight", "up.0.block.0.temb_proj.bias", "up.0.block.0.norm2.weight", "up.0.block.0.norm2.bias", "up.0.block.0.conv2.weight", "up.0.block.0.conv2.bias", "up.0.block.0.nin_shortcut.weight", "up.0.block.0.nin_shortcut.bias", "up.0.block.1.norm1.weight", "up.0.block.1.norm1.bias", "up.0.block.1.conv1.weight", "up.0.block.1.conv1.bias", "up.0.block.1.temb_proj.weight", "up.0.block.1.temb_proj.bias", "up.0.block.1.norm2.weight", "up.0.block.1.norm2.bias", "up.0.block.1.conv2.weight", "up.0.block.1.conv2.bias", "up.0.block.1.nin_shortcut.weight", "up.0.block.1.nin_shortcut.bias", "up.0.block.2.norm1.weight", "up.0.block.2.norm1.bias", "up.0.block.2.conv1.weight", "up.0.block.2.conv1.bias", "up.0.block.2.temb_proj.weight", "up.0.block.2.temb_proj.bias", "up.0.block.2.norm2.weight", "up.0.block.2.norm2.bias", "up.0.block.2.conv2.weight", "up.0.block.2.conv2.bias", "up.0.block.2.nin_shortcut.weight", "up.0.block.2.nin_shortcut.bias", "up.1.block.0.norm1.weight", "up.1.block.0.norm1.bias", "up.1.block.0.conv1.weight", "up.1.block.0.conv1.bias", "up.1.block.0.temb_proj.weight", "up.1.block.0.temb_proj.bias", "up.1.block.0.norm2.weight", "up.1.block.0.norm2.bias", "up.1.block.0.conv2.weight", "up.1.block.0.conv2.bias", "up.1.block.0.nin_shortcut.weight", "up.1.block.0.nin_shortcut.bias", "up.1.block.1.norm1.weight", "up.1.block.1.norm1.bias", "up.1.block.1.conv1.weight", "up.1.block.1.conv1.bias", "up.1.block.1.temb_proj.weight", "up.1.block.1.temb_proj.bias", "up.1.block.1.norm2.weight", "up.1.block.1.norm2.bias", "up.1.block.1.conv2.weight", "up.1.block.1.conv2.bias", "up.1.block.1.nin_shortcut.weight", "up.1.block.1.nin_shortcut.bias", "up.1.block.2.norm1.weight", "up.1.block.2.norm1.bias", "up.1.block.2.conv1.weight", "up.1.block.2.conv1.bias", "up.1.block.2.temb_proj.weight", "up.1.block.2.temb_proj.bias", "up.1.block.2.norm2.weight", "up.1.block.2.norm2.bias", "up.1.block.2.conv2.weight", "up.1.block.2.conv2.bias", "up.1.block.2.nin_shortcut.weight", "up.1.block.2.nin_shortcut.bias", "up.1.upsample.conv.weight", "up.1.upsample.conv.bias", "up.2.block.0.norm1.weight", "up.2.block.0.norm1.bias", "up.2.block.0.conv1.weight", "up.2.block.0.conv1.bias", "up.2.block.0.temb_proj.weight", "up.2.block.0.temb_proj.bias", "up.2.block.0.norm2.weight", "up.2.block.0.norm2.bias", "up.2.block.0.conv2.weight", "up.2.block.0.conv2.bias", "up.2.block.0.nin_shortcut.weight", "up.2.block.0.nin_shortcut.bias", "up.2.block.1.norm1.weight", "up.2.block.1.norm1.bias", "up.2.block.1.conv1.weight", "up.2.block.1.conv1.bias", "up.2.block.1.temb_proj.weight", "up.2.block.1.temb_proj.bias", "up.2.block.1.norm2.weight", "up.2.block.1.norm2.bias", "up.2.block.1.conv2.weight", "up.2.block.1.conv2.bias", "up.2.block.1.nin_shortcut.weight", "up.2.block.1.nin_shortcut.bias", "up.2.block.2.norm1.weight", "up.2.block.2.norm1.bias", "up.2.block.2.conv1.weight", "up.2.block.2.conv1.bias", "up.2.block.2.temb_proj.weight", "up.2.block.2.temb_proj.bias", "up.2.block.2.norm2.weight", "up.2.block.2.norm2.bias", "up.2.block.2.conv2.weight", "up.2.block.2.conv2.bias", "up.2.block.2.nin_shortcut.weight", "up.2.block.2.nin_shortcut.bias", "up.2.upsample.conv.weight", "up.2.upsample.conv.bias", "up.3.block.0.norm1.weight", "up.3.block.0.norm1.bias", "up.3.block.0.conv1.weight", "up.3.block.0.conv1.bias", "up.3.block.0.temb_proj.weight", "up.3.block.0.temb_proj.bias", "up.3.block.0.norm2.weight", "up.3.block.0.norm2.bias", "up.3.block.0.conv2.weight", "up.3.block.0.conv2.bias", "up.3.block.0.nin_shortcut.weight", "up.3.block.0.nin_shortcut.bias", "up.3.block.1.norm1.weight", "up.3.block.1.norm1.bias", "up.3.block.1.conv1.weight", "up.3.block.1.conv1.bias", "up.3.block.1.temb_proj.weight", "up.3.block.1.temb_proj.bias", "up.3.block.1.norm2.weight", "up.3.block.1.norm2.bias", "up.3.block.1.conv2.weight", "up.3.block.1.conv2.bias", "up.3.block.1.nin_shortcut.weight", "up.3.block.1.nin_shortcut.bias", "up.3.block.2.norm1.weight", "up.3.block.2.norm1.bias", "up.3.block.2.conv1.weight", "up.3.block.2.conv1.bias", "up.3.block.2.temb_proj.weight", "up.3.block.2.temb_proj.bias", "up.3.block.2.norm2.weight", "up.3.block.2.norm2.bias", "up.3.block.2.conv2.weight", "up.3.block.2.conv2.bias", "up.3.block.2.nin_shortcut.weight", "up.3.block.2.nin_shortcut.bias", "up.3.upsample.conv.weight", "up.3.upsample.conv.bias", "up.4.block.0.norm1.weight", "up.4.block.0.norm1.bias", "up.4.block.0.conv1.weight", "up.4.block.0.conv1.bias", "up.4.block.0.temb_proj.weight", "up.4.block.0.temb_proj.bias", "up.4.block.0.norm2.weight", "up.4.block.0.norm2.bias", "up.4.block.0.conv2.weight", "up.4.block.0.conv2.bias", "up.4.block.0.nin_shortcut.weight", "up.4.block.0.nin_shortcut.bias", "up.4.block.1.norm1.weight", "up.4.block.1.norm1.bias", "up.4.block.1.conv1.weight", "up.4.block.1.conv1.bias", "up.4.block.1.temb_proj.weight", "up.4.block.1.temb_proj.bias", "up.4.block.1.norm2.weight", "up.4.block.1.norm2.bias", "up.4.block.1.conv2.weight", "up.4.block.1.conv2.bias", "up.4.block.1.nin_shortcut.weight", "up.4.block.1.nin_shortcut.bias", "up.4.block.2.norm1.weight", "up.4.block.2.norm1.bias", "up.4.block.2.conv1.weight", "up.4.block.2.conv1.bias", "up.4.block.2.temb_proj.weight", "up.4.block.2.temb_proj.bias", "up.4.block.2.norm2.weight", "up.4.block.2.norm2.bias", "up.4.block.2.conv2.weight", "up.4.block.2.conv2.bias", "up.4.block.2.nin_shortcut.weight", "up.4.block.2.nin_shortcut.bias", "up.4.attn.0.norm.weight", "up.4.attn.0.norm.bias", "up.4.attn.0.q.weight", "up.4.attn.0.q.bias", "up.4.attn.0.k.weight", "up.4.attn.0.k.bias", "up.4.attn.0.v.weight", "up.4.attn.0.v.bias", "up.4.attn.0.proj_out.weight", "up.4.attn.0.proj_out.bias", "up.4.attn.1.norm.weight", "up.4.attn.1.norm.bias", "up.4.attn.1.q.weight", "up.4.attn.1.q.bias", "up.4.attn.1.k.weight", "up.4.attn.1.k.bias", "up.4.attn.1.v.weight", "up.4.attn.1.v.bias", "up.4.attn.1.proj_out.weight", "up.4.attn.1.proj_out.bias", "up.4.attn.2.norm.weight", "up.4.attn.2.norm.bias", "up.4.attn.2.q.weight", "up.4.attn.2.q.bias", "up.4.attn.2.k.weight", "up.4.attn.2.k.bias", "up.4.attn.2.v.weight", "up.4.attn.2.v.bias", "up.4.attn.2.proj_out.weight", "up.4.attn.2.proj_out.bias", "up.4.upsample.conv.weight", "up.4.upsample.conv.bias", "up.5.block.0.norm1.weight", "up.5.block.0.norm1.bias", "up.5.block.0.conv1.weight", "up.5.block.0.conv1.bias", "up.5.block.0.temb_proj.weight", "up.5.block.0.temb_proj.bias", "up.5.block.0.norm2.weight", "up.5.block.0.norm2.bias", "up.5.block.0.conv2.weight", "up.5.block.0.conv2.bias", "up.5.block.0.nin_shortcut.weight", "up.5.block.0.nin_shortcut.bias", "up.5.block.1.norm1.weight", "up.5.block.1.norm1.bias", "up.5.block.1.conv1.weight", "up.5.block.1.conv1.bias", "up.5.block.1.temb_proj.weight", "up.5.block.1.temb_proj.bias", "up.5.block.1.norm2.weight", "up.5.block.1.norm2.bias", "up.5.block.1.conv2.weight", "up.5.block.1.conv2.bias", "up.5.block.1.nin_shortcut.weight", "up.5.block.1.nin_shortcut.bias", "up.5.block.2.norm1.weight", "up.5.block.2.norm1.bias", "up.5.block.2.conv1.weight", "up.5.block.2.conv1.bias", "up.5.block.2.temb_proj.weight", "up.5.block.2.temb_proj.bias", "up.5.block.2.norm2.weight", "up.5.block.2.norm2.bias", "up.5.block.2.conv2.weight", "up.5.block.2.conv2.bias", "up.5.block.2.nin_shortcut.weight", "up.5.block.2.nin_shortcut.bias", "up.5.upsample.conv.weight", "up.5.upsample.conv.bias", "norm_out.weight", "norm_out.bias", "conv_out.weight", "conv_out.bias".
Unexpected key(s) in state_dict: "time_embed.0.weight", "time_embed.0.bias", "time_embed.2.weight", "time_embed.2.bias", "input_blocks.0.0.weight", "input_blocks.0.0.bias", "input_blocks.1.0.in_layers.0.weight", "input_blocks.1.0.in_layers.0.bias", "input_blocks.1.0.in_layers.2.weight", "input_blocks.1.0.in_layers.2.bias", "input_blocks.1.0.emb_layers.1.weight", "input_blocks.1.0.emb_layers.1.bias", "input_blocks.1.0.out_layers.0.weight", "input_blocks.1.0.out_layers.0.bias", "input_blocks.1.0.out_layers.3.weight", "input_blocks.1.0.out_layers.3.bias", "input_blocks.2.0.in_layers.0.weight", "input_blocks.2.0.in_layers.0.bias", "input_blocks.2.0.in_layers.2.weight", "input_blocks.2.0.in_layers.2.bias", "input_blocks.2.0.emb_layers.1.weight", "input_blocks.2.0.emb_layers.1.bias", "input_blocks.2.0.out_layers.0.weight", "input_blocks.2.0.out_layers.0.bias", "input_blocks.2.0.out_layers.3.weight", "input_blocks.2.0.out_layers.3.bias", "input_blocks.3.0.in_layers.0.weight", "input_blocks.3.0.in_layers.0.bias", "input_blocks.3.0.in_layers.2.weight", "input_blocks.3.0.in_layers.2.bias", "input_blocks.3.0.emb_layers.1.weight", "input_blocks.3.0.emb_layers.1.bias", "input_blocks.3.0.out_layers.0.weight", "input_blocks.3.0.out_layers.0.bias", "input_blocks.3.0.out_layers.3.weight", "input_blocks.3.0.out_layers.3.bias", "input_blocks.4.0.in_layers.0.weight", "input_blocks.4.0.in_layers.0.bias", "input_blocks.4.0.in_layers.2.weight", "input_blocks.4.0.in_layers.2.bias", "input_blocks.4.0.emb_layers.1.weight", "input_blocks.4.0.emb_layers.1.bias", "input_blocks.4.0.out_layers.0.weight", "input_blocks.4.0.out_layers.0.bias", "input_blocks.4.0.out_layers.3.weight", "input_blocks.4.0.out_layers.3.bias", "input_blocks.5.0.in_layers.0.weight", "input_blocks.5.0.in_layers.0.bias", "input_blocks.5.0.in_layers.2.weight", "input_blocks.5.0.in_layers.2.bias", "input_blocks.5.0.emb_layers.1.weight", "input_blocks.5.0.emb_layers.1.bias", "input_blocks.5.0.out_layers.0.weight", "input_blocks.5.0.out_layers.0.bias", "input_blocks.5.0.out_layers.3.weight", "input_blocks.5.0.out_layers.3.bias", "input_blocks.6.0.in_layers.0.weight", "input_blocks.6.0.in_layers.0.bias", "input_blocks.6.0.in_layers.2.weight", "input_blocks.6.0.in_layers.2.bias", "input_blocks.6.0.emb_layers.1.weight", "input_blocks.6.0.emb_layers.1.bias", "input_blocks.6.0.out_layers.0.weight", "input_blocks.6.0.out_layers.0.bias", "input_blocks.6.0.out_layers.3.weight", "input_blocks.6.0.out_layers.3.bias", "input_blocks.7.0.in_layers.0.weight", "input_blocks.7.0.in_layers.0.bias", "input_blocks.7.0.in_layers.2.weight", "input_blocks.7.0.in_layers.2.bias", "input_blocks.7.0.emb_layers.1.weight", "input_blocks.7.0.emb_layers.1.bias", "input_blocks.7.0.out_layers.0.weight", "input_blocks.7.0.out_layers.0.bias", "input_blocks.7.0.out_layers.3.weight", "input_blocks.7.0.out_layers.3.bias", "input_blocks.7.0.skip_connection.weight", "input_blocks.7.0.skip_connection.bias", "input_blocks.8.0.in_layers.0.weight", "input_blocks.8.0.in_layers.0.bias", "input_blocks.8.0.in_layers.2.weight", "input_blocks.8.0.in_layers.2.bias", "input_blocks.8.0.emb_layers.1.weight", "input_blocks.8.0.emb_layers.1.bias", "input_blocks.8.0.out_layers.0.weight", "input_blocks.8.0.out_layers.0.bias", "input_blocks.8.0.out_layers.3.weight", "input_blocks.8.0.out_layers.3.bias", "input_blocks.9.0.in_layers.0.weight", "input_blocks.9.0.in_layers.0.bias", "input_blocks.9.0.in_layers.2.weight", "input_blocks.9.0.in_layers.2.bias", "input_blocks.9.0.emb_layers.1.weight", "input_blocks.9.0.emb_layers.1.bias", "input_blocks.9.0.out_layers.0.weight", "input_blocks.9.0.out_layers.0.bias", "input_blocks.9.0.out_layers.3.weight", "input_blocks.9.0.out_layers.3.bias", "input_blocks.10.0.in_layers.0.weight", "input_blocks.10.0.in_layers.0.bias", "input_blocks.10.0.in_layers.2.weight", "input_blocks.10.0.in_layers.2.bias", "input_blocks.10.0.emb_layers.1.weight", "input_blocks.10.0.emb_layers.1.bias", "input_blocks.10.0.out_layers.0.weight", "input_blocks.10.0.out_layers.0.bias", "input_blocks.10.0.out_layers.3.weight", "input_blocks.10.0.out_layers.3.bias", "input_blocks.10.1.norm.weight", "input_blocks.10.1.norm.bias", "input_blocks.10.1.qkv.weight", "input_blocks.10.1.qkv.bias", "input_blocks.10.1.proj_out.weight", "input_blocks.10.1.proj_out.bias", "input_blocks.11.0.in_layers.0.weight", "input_blocks.11.0.in_layers.0.bias", "input_blocks.11.0.in_layers.2.weight", "input_blocks.11.0.in_layers.2.bias", "input_blocks.11.0.emb_layers.1.weight", "input_blocks.11.0.emb_layers.1.bias", "input_blocks.11.0.out_layers.0.weight", "input_blocks.11.0.out_layers.0.bias", "input_blocks.11.0.out_layers.3.weight", "input_blocks.11.0.out_layers.3.bias", "input_blocks.11.1.norm.weight", "input_blocks.11.1.norm.bias", "input_blocks.11.1.qkv.weight", "input_blocks.11.1.qkv.bias", "input_blocks.11.1.proj_out.weight", "input_blocks.11.1.proj_out.bias", "input_blocks.12.0.in_layers.0.weight", "input_blocks.12.0.in_layers.0.bias", "input_blocks.12.0.in_layers.2.weight", "input_blocks.12.0.in_layers.2.bias", "input_blocks.12.0.emb_layers.1.weight", "input_blocks.12.0.emb_layers.1.bias", "input_blocks.12.0.out_layers.0.weight", "input_blocks.12.0.out_layers.0.bias", "input_blocks.12.0.out_layers.3.weight", "input_blocks.12.0.out_layers.3.bias", "input_blocks.13.0.in_layers.0.weight", "input_blocks.13.0.in_layers.0.bias", "input_blocks.13.0.in_layers.2.weight", "input_blocks.13.0.in_layers.2.bias", "input_blocks.13.0.emb_layers.1.weight", "input_blocks.13.0.emb_layers.1.bias", "input_blocks.13.0.out_layers.0.weight", "input_blocks.13.0.out_layers.0.bias", "input_blocks.13.0.out_layers.3.weight", "input_blocks.13.0.out_layers.3.bias", "input_blocks.13.0.skip_connection.weight", "input_blocks.13.0.skip_connection.bias", "input_blocks.13.1.norm.weight", "input_blocks.13.1.norm.bias", "input_blocks.13.1.qkv.weight", "input_blocks.13.1.qkv.bias", "input_blocks.13.1.proj_out.weight", "input_blocks.13.1.proj_out.bias", "input_blocks.14.0.in_layers.0.weight", "input_blocks.14.0.in_layers.0.bias", "input_blocks.14.0.in_layers.2.weight", "input_blocks.14.0.in_layers.2.bias", "input_blocks.14.0.emb_layers.1.weight", "input_blocks.14.0.emb_layers.1.bias", "input_blocks.14.0.out_layers.0.weight", "input_blocks.14.0.out_layers.0.bias", "input_blocks.14.0.out_layers.3.weight", "input_blocks.14.0.out_layers.3.bias", "input_blocks.14.1.norm.weight", "input_blocks.14.1.norm.bias", "input_blocks.14.1.qkv.weight", "input_blocks.14.1.qkv.bias", "input_blocks.14.1.proj_out.weight", "input_blocks.14.1.proj_out.bias", "input_blocks.15.0.in_layers.0.weight", "input_blocks.15.0.in_layers.0.bias", "input_blocks.15.0.in_layers.2.weight", "input_blocks.15.0.in_layers.2.bias", "input_blocks.15.0.emb_layers.1.weight", "input_blocks.15.0.emb_layers.1.bias", "input_blocks.15.0.out_layers.0.weight", "input_blocks.15.0.out_layers.0.bias", "input_blocks.15.0.out_layers.3.weight", "input_blocks.15.0.out_layers.3.bias", "input_blocks.16.0.in_layers.0.weight", "input_blocks.16.0.in_layers.0.bias", "input_blocks.16.0.in_layers.2.weight", "input_blocks.16.0.in_layers.2.bias", "input_blocks.16.0.emb_layers.1.weight", "input_blocks.16.0.emb_layers.1.bias", "input_blocks.16.0.out_layers.0.weight", "input_blocks.16.0.out_layers.0.bias", "input_blocks.16.0.out_layers.3.weight", "input_blocks.16.0.out_layers.3.bias", "input_blocks.16.1.norm.weight", "input_blocks.16.1.norm.bias", "input_blocks.16.1.qkv.weight", "input_blocks.16.1.qkv.bias", "input_blocks.16.1.proj_out.weight", "input_blocks.16.1.proj_out.bias", "input_blocks.17.0.in_layers.0.weight", "input_blocks.17.0.in_layers.0.bias", "input_blocks.17.0.in_layers.2.weight", "input_blocks.17.0.in_layers.2.bias", "input_blocks.17.0.emb_layers.1.weight", "input_blocks.17.0.emb_layers.1.bias", "input_blocks.17.0.out_layers.0.weight", "input_blocks.17.0.out_layers.0.bias", "input_blocks.17.0.out_layers.3.weight", "input_blocks.17.0.out_layers.3.bias", "input_blocks.17.1.norm.weight", "input_blocks.17.1.norm.bias", "input_blocks.17.1.qkv.weight", "input_blocks.17.1.qkv.bias", "input_blocks.17.1.proj_out.weight", "input_blocks.17.1.proj_out.bias", "middle_block.0.in_layers.0.weight", "middle_block.0.in_layers.0.bias", "middle_block.0.in_layers.2.weight", "middle_block.0.in_layers.2.bias", "middle_block.0.emb_layers.1.weight", "middle_block.0.emb_layers.1.bias", "middle_block.0.out_layers.0.weight", "middle_block.0.out_layers.0.bias", "middle_block.0.out_layers.3.weight", "middle_block.0.out_layers.3.bias", "middle_block.1.norm.weight", "middle_block.1.norm.bias", "middle_block.1.qkv.weight", "middle_block.1.qkv.bias", "middle_block.1.proj_out.weight", "middle_block.1.proj_out.bias", "middle_block.2.in_layers.0.weight", "middle_block.2.in_layers.0.bias", "middle_block.2.in_layers.2.weight", "middle_block.2.in_layers.2.bias", "middle_block.2.emb_layers.1.weight", "middle_block.2.emb_layers.1.bias", "middle_block.2.out_layers.0.weight", "middle_block.2.out_layers.0.bias", "middle_block.2.out_layers.3.weight", "middle_block.2.out_layers.3.bias", "output_blocks.0.0.in_layers.0.weight", "output_blocks.0.0.in_layers.0.bias", "output_blocks.0.0.in_layers.2.weight", "output_blocks.0.0.in_layers.2.bias", "output_blocks.0.0.emb_layers.1.weight", "output_blocks.0.0.emb_layers.1.bias", "output_blocks.0.0.out_layers.0.weight", "output_blocks.0.0.out_layers.0.bias", "output_blocks.0.0.out_layers.3.weight", "output_blocks.0.0.out_layers.3.bias", "output_blocks.0.0.skip_connection.weight", "output_blocks.0.0.skip_connection.bias", "output_blocks.0.1.norm.weight", "output_blocks.0.1.norm.bias", "output_blocks.0.1.qkv.weight", "output_blocks.0.1.qkv.bias", "output_blocks.0.1.proj_out.weight", "output_blocks.0.1.proj_out.bias", "output_blocks.1.0.in_layers.0.weight", "output_blocks.1.0.in_layers.0.bias", "output_blocks.1.0.in_layers.2.weight", "output_blocks.1.0.in_layers.2.bias", "output_blocks.1.0.emb_layers.1.weight", "output_blocks.1.0.emb_layers.1.bias", "output_blocks.1.0.out_layers.0.weight", "output_blocks.1.0.out_layers.0.bias", "output_blocks.1.0.out_layers.3.weight", "output_blocks.1.0.out_layers.3.bias", "output_blocks.1.0.skip_connection.weight", "output_blocks.1.0.skip_connection.bias", "output_blocks.1.1.norm.weight", "output_blocks.1.1.norm.bias", "output_blocks.1.1.qkv.weight", "output_blocks.1.1.qkv.bias", "output_blocks.1.1.proj_out.weight", "output_blocks.1.1.proj_out.bias", "output_blocks.2.0.in_layers.0.weight", "output_blocks.2.0.in_layers.0.bias", "output_blocks.2.0.in_layers.2.weight", "output_blocks.2.0.in_layers.2.bias", "output_blocks.2.0.emb_layers.1.weight", "output_blocks.2.0.emb_layers.1.bias", "output_blocks.2.0.out_layers.0.weight", "output_blocks.2.0.out_layers.0.bias", "output_blocks.2.0.out_layers.3.weight", "output_blocks.2.0.out_layers.3.bias", "output_blocks.2.0.skip_connection.weight", "output_blocks.2.0.skip_connection.bias", "output_blocks.2.1.norm.weight", "output_blocks.2.1.norm.bias", "output_blocks.2.1.qkv.weight", "output_blocks.2.1.qkv.bias", "output_blocks.2.1.proj_out.weight", "output_blocks.2.1.proj_out.bias", "output_blocks.2.2.in_layers.0.weight", "output_blocks.2.2.in_layers.0.bias", "output_blocks.2.2.in_layers.2.weight", "output_blocks.2.2.in_layers.2.bias", "output_blocks.2.2.emb_layers.1.weight", "output_blocks.2.2.emb_layers.1.bias", "output_blocks.2.2.out_layers.0.weight", "output_blocks.2.2.out_layers.0.bias", "output_blocks.2.2.out_layers.3.weight", "output_blocks.2.2.out_layers.3.bias", "output_blocks.3.0.in_layers.0.weight", "output_blocks.3.0.in_layers.0.bias", "output_blocks.3.0.in_layers.2.weight", "output_blocks.3.0.in_layers.2.bias", "output_blocks.3.0.emb_layers.1.weight", "output_blocks.3.0.emb_layers.1.bias", "output_blocks.3.0.out_layers.0.weight", "output_blocks.3.0.out_layers.0.bias", "output_blocks.3.0.out_layers.3.weight", "output_blocks.3.0.out_layers.3.bias", "output_blocks.3.0.skip_connection.weight", "output_blocks.3.0.skip_connection.bias", "output_blocks.3.1.norm.weight", "output_blocks.3.1.norm.bias", "output_blocks.3.1.qkv.weight", "output_blocks.3.1.qkv.bias", "output_blocks.3.1.proj_out.weight", "output_blocks.3.1.proj_out.bias", "output_blocks.4.0.in_layers.0.weight", "output_blocks.4.0.in_layers.0.bias", "output_blocks.4.0.in_layers.2.weight", "output_blocks.4.0.in_layers.2.bias", "output_blocks.4.0.emb_layers.1.weight", "output_blocks.4.0.emb_layers.1.bias", "output_blocks.4.0.out_layers.0.weight", "output_blocks.4.0.out_layers.0.bias", "output_blocks.4.0.out_layers.3.weight", "output_blocks.4.0.out_layers.3.bias", "output_blocks.4.0.skip_connection.weight", "output_blocks.4.0.skip_connection.bias", "output_blocks.4.1.norm.weight", "output_blocks.4.1.norm.bias", "output_blocks.4.1.qkv.weight", "output_blocks.4.1.qkv.bias", "output_blocks.4.1.proj_out.weight", "output_blocks.4.1.proj_out.bias", "output_blocks.5.0.in_layers.0.weight", "output_blocks.5.0.in_layers.0.bias", "output_blocks.5.0.in_layers.2.weight", "output_blocks.5.0.in_layers.2.bias", "output_blocks.5.0.emb_layers.1.weight", "output_blocks.5.0.emb_layers.1.bias", "output_blocks.5.0.out_layers.0.weight", "output_blocks.5.0.out_layers.0.bias", "output_blocks.5.0.out_layers.3.weight", "output_blocks.5.0.out_layers.3.bias", "output_blocks.5.0.skip_connection.weight", "output_blocks.5.0.skip_connection.bias", "output_blocks.5.1.norm.weight", "output_blocks.5.1.norm.bias", "output_blocks.5.1.qkv.weight", "output_blocks.5.1.qkv.bias", "output_blocks.5.1.proj_out.weight", "output_blocks.5.1.proj_out.bias", "output_blocks.5.2.in_layers.0.weight", "output_blocks.5.2.in_layers.0.bias", "output_blocks.5.2.in_layers.2.weight", "output_blocks.5.2.in_layers.2.bias", "output_blocks.5.2.emb_layers.1.weight", "output_blocks.5.2.emb_layers.1.bias", "output_blocks.5.2.out_layers.0.weight", "output_blocks.5.2.out_layers.0.bias", "output_blocks.5.2.out_layers.3.weight", "output_blocks.5.2.out_layers.3.bias", "output_blocks.6.0.in_layers.0.weight", "output_blocks.6.0.in_layers.0.bias", "output_blocks.6.0.in_layers.2.weight", "output_blocks.6.0.in_layers.2.bias", "output_blocks.6.0.emb_layers.1.weight", "output_blocks.6.0.emb_layers.1.bias", "output_blocks.6.0.out_layers.0.weight", "output_blocks.6.0.out_layers.0.bias", "output_blocks.6.0.out_layers.3.weight", "output_blocks.6.0.out_layers.3.bias", "output_blocks.6.0.skip_connection.weight", "output_blocks.6.0.skip_connection.bias", "output_blocks.6.1.norm.weight", "output_blocks.6.1.norm.bias", "output_blocks.6.1.qkv.weight", "output_blocks.6.1.qkv.bias", "output_blocks.6.1.proj_out.weight", "output_blocks.6.1.proj_out.bias", "output_blocks.7.0.in_layers.0.weight", "output_blocks.7.0.in_layers.0.bias", "output_blocks.7.0.in_layers.2.weight", "output_blocks.7.0.in_layers.2.bias", "output_blocks.7.0.emb_layers.1.weight", "output_blocks.7.0.emb_layers.1.bias", "output_blocks.7.0.out_layers.0.weight", "output_blocks.7.0.out_layers.0.bias", "output_blocks.7.0.out_layers.3.weight", "output_blocks.7.0.out_layers.3.bias", "output_blocks.7.0.skip_connection.weight", "output_blocks.7.0.skip_connection.bias", "output_blocks.7.1.norm.weight", "output_blocks.7.1.norm.bias", "output_blocks.7.1.qkv.weight", "output_blocks.7.1.qkv.bias", "output_blocks.7.1.proj_out.weight", "output_blocks.7.1.proj_out.bias", "output_blocks.8.0.in_layers.0.weight", "output_blocks.8.0.in_layers.0.bias", "output_blocks.8.0.in_layers.2.weight", "output_blocks.8.0.in_layers.2.bias", "output_blocks.8.0.emb_layers.1.weight", "output_blocks.8.0.emb_layers.1.bias", "output_blocks.8.0.out_layers.0.weight", "output_blocks.8.0.out_layers.0.bias", "output_blocks.8.0.out_layers.3.weight", "output_blocks.8.0.out_layers.3.bias", "output_blocks.8.0.skip_connection.weight", "output_blocks.8.0.skip_connection.bias", "output_blocks.8.1.norm.weight", "output_blocks.8.1.norm.bias", "output_blocks.8.1.qkv.weight", "output_blocks.8.1.qkv.bias", "output_blocks.8.1.proj_out.weight", "output_blocks.8.1.proj_out.bias", "output_blocks.8.2.in_layers.0.weight", "output_blocks.8.2.in_layers.0.bias", "output_blocks.8.2.in_layers.2.weight", "output_blocks.8.2.in_layers.2.bias", "output_blocks.8.2.emb_layers.1.weight", "output_blocks.8.2.emb_layers.1.bias", "output_blocks.8.2.out_layers.0.weight", "output_blocks.8.2.out_layers.0.bias", "output_blocks.8.2.out_layers.3.weight", "output_blocks.8.2.out_layers.3.bias", "output_blocks.9.0.in_layers.0.weight", "output_blocks.9.0.in_layers.0.bias", "output_blocks.9.0.in_layers.2.weight", "output_blocks.9.0.in_layers.2.bias", "output_blocks.9.0.emb_layers.1.weight", "output_blocks.9.0.emb_layers.1.bias", "output_blocks.9.0.out_layers.0.weight", "output_blocks.9.0.out_layers.0.bias", "output_blocks.9.0.out_layers.3.weight", "output_blocks.9.0.out_layers.3.bias", "output_blocks.9.0.skip_connection.weight", "output_blocks.9.0.skip_connection.bias", "output_blocks.10.0.in_layers.0.weight", "output_blocks.10.0.in_layers.0.bias", "output_blocks.10.0.in_layers.2.weight", "output_blocks.10.0.in_layers.2.bias", "output_blocks.10.0.emb_layers.1.weight", "output_blocks.10.0.emb_layers.1.bias", "output_blocks.10.0.out_layers.0.weight", "output_blocks.10.0.out_layers.0.bias", "output_blocks.10.0.out_layers.3.weight", "output_blocks.10.0.out_layers.3.bias", "output_blocks.10.0.skip_connection.weight", "output_blocks.10.0.skip_connection.bias", "output_blocks.11.0.in_layers.0.weight", "output_blocks.11.0.in_layers.0.bias", "output_blocks.11.0.in_layers.2.weight", "output_blocks.11.0.in_layers.2.bias", "output_blocks.11.0.emb_layers.1.weight", "output_blocks.11.0.emb_layers.1.bias", "output_blocks.11.0.out_layers.0.weight", "output_blocks.11.0.out_layers.0.bias", "output_blocks.11.0.out_layers.3.weight", "output_blocks.11.0.out_layers.3.bias", "output_blocks.11.0.skip_connection.weight", "output_blocks.11.0.skip_connection.bias", "output_blocks.11.1.in_layers.0.weight", "output_blocks.11.1.in_layers.0.bias", "output_blocks.11.1.in_layers.2.weight", "output_blocks.11.1.in_layers.2.bias", "output_blocks.11.1.emb_layers.1.weight", "output_blocks.11.1.emb_layers.1.bias", "output_blocks.11.1.out_layers.0.weight", "output_blocks.11.1.out_layers.0.bias", "output_blocks.11.1.out_layers.3.weight", "output_blocks.11.1.out_layers.3.bias", "output_blocks.12.0.in_layers.0.weight", "output_blocks.12.0.in_layers.0.bias", "output_blocks.12.0.in_layers.2.weight", "output_blocks.12.0.in_layers.2.bias", "output_blocks.12.0.emb_layers.1.weight", "output_blocks.12.0.emb_layers.1.bias", "output_blocks.12.0.out_layers.0.weight", "output_blocks.12.0.out_layers.0.bias", "output_blocks.12.0.out_layers.3.weight", "output_blocks.12.0.out_layers.3.bias", "output_blocks.12.0.skip_connection.weight", "output_blocks.12.0.skip_connection.bias", "output_blocks.13.0.in_layers.0.weight", "output_blocks.13.0.in_layers.0.bias", "output_blocks.13.0.in_layers.2.weight", "output_blocks.13.0.in_layers.2.bias", "output_blocks.13.0.emb_layers.1.weight", "output_blocks.13.0.emb_layers.1.bias", "output_blocks.13.0.out_layers.0.weight", "output_blocks.13.0.out_layers.0.bias", "output_blocks.13.0.out_layers.3.weight", "output_blocks.13.0.out_layers.3.bias", "output_blocks.13.0.skip_connection.weight", "output_blocks.13.0.skip_connection.bias", "output_blocks.14.0.in_layers.0.weight", "output_blocks.14.0.in_layers.0.bias", "output_blocks.14.0.in_layers.2.weight", "output_blocks.14.0.in_layers.2.bias", "output_blocks.14.0.emb_layers.1.weight", "output_blocks.14.0.emb_layers.1.bias", "output_blocks.14.0.out_layers.0.weight", "output_blocks.14.0.out_layers.0.bias", "output_blocks.14.0.out_layers.3.weight", "output_blocks.14.0.out_layers.3.bias", "output_blocks.14.0.skip_connection.weight", "output_blocks.14.0.skip_connection.bias", "output_blocks.14.1.in_layers.0.weight", "output_blocks.14.1.in_layers.0.bias", "output_blocks.14.1.in_layers.2.weight", "output_blocks.14.1.in_layers.2.bias", "output_blocks.14.1.emb_layers.1.weight", "output_blocks.14.1.emb_layers.1.bias", "output_blocks.14.1.out_layers.0.weight", "output_blocks.14.1.out_layers.0.bias", "output_blocks.14.1.out_layers.3.weight", "output_blocks.14.1.out_layers.3.bias", "output_blocks.15.0.in_layers.0.weight", "output_blocks.15.0.in_layers.0.bias", "output_blocks.15.0.in_layers.2.weight", "output_blocks.15.0.in_layers.2.bias", "output_blocks.15.0.emb_layers.1.weight", "output_blocks.15.0.emb_layers.1.bias", "output_blocks.15.0.out_layers.0.weight", "output_blocks.15.0.out_layers.0.bias", "output_blocks.15.0.out_layers.3.weight", "output_blocks.15.0.out_layers.3.bias", "output_blocks.15.0.skip_connection.weight", "output_blocks.15.0.skip_connection.bias", "output_blocks.16.0.in_layers.0.weight", "output_blocks.16.0.in_layers.0.bias", "output_blocks.16.0.in_layers.2.weight", "output_blocks.16.0.in_layers.2.bias", "output_blocks.16.0.emb_layers.1.weight", "output_blocks.16.0.emb_layers.1.bias", "output_blocks.16.0.out_layers.0.weight", "output_blocks.16.0.out_layers.0.bias", "output_blocks.16.0.out_layers.3.weight", "output_blocks.16.0.out_layers.3.bias", "output_blocks.16.0.skip_connection.weight", "output_blocks.16.0.skip_connection.bias", "output_blocks.17.0.in_layers.0.weight", "output_blocks.17.0.in_layers.0.bias", "output_blocks.17.0.in_layers.2.weight", "output_blocks.17.0.in_layers.2.bias", "output_blocks.17.0.emb_layers.1.weight", "output_blocks.17.0.emb_layers.1.bias", "output_blocks.17.0.out_layers.0.weight", "output_blocks.17.0.out_layers.0.bias", "output_blocks.17.0.out_layers.3.weight", "output_blocks.17.0.out_layers.3.bias", "output_blocks.17.0.skip_connection.weight", "output_blocks.17.0.skip_connection.bias", "out.0.weight", "out.0.bias", "out.2.weight", "out.2.bias".

can you help me?
Thanks!

Thank you for your awesome work!

I would like to test the diffusion from an input image.
Example. I create a blurry image then use a diffusion model to clean it.
By looking at your code for de-blurring, you take clear images from a folder and auto blur them before you do diffusion.
Is there a way to just use my already blurred input image? Or do I have to change the code to do that?

Thank you.

Can everyone understand the theoretical derivation in the original text? Is there any video lecture material related to the revision of the thesis, you can share it, I hope to use it to further understand

It seems that access to the link provided for the .ckpt file of the celeba_hq dataset (https://image-editing-test-12345.s3-us-west-2.amazonaws.com/checkpoints/celeba_hq.ckpt), also used in this repo https://github.com/ermongroup/SDEdit/tree/main, is denied.

Hi, @bahjat-kawar

Thank you for sharing your nice implementation!
I guess that your code includes only VP SDEs.
Would you provide a code for VE SDEs?

I've tested you model on all available configurations in Google Colab and got this when running image_256_cc.yml.

ERROR - main.py - 2022-07-21 11:00:58,211 - Traceback (most recent call last):
  File "main.py", line 164, in main
    runner.sample()
  File "/content/ddrm/runners/diffusion.py", line 135, in sample
    model.load_state_dict(torch.load(ckpt, map_location=self.device))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1605, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for UNetModel:
	Missing key(s) in state_dict: "label_emb.weight". 

Could you recommend a possible fix?

Hi, I just run the following code from the README and get "RuntimeError: CUDA error: out of memory". The GPU is NVIDIA Quadro RTX 8000 with 48 GB.
python main.py --ni --config imagenet_256.yml --doc imagenet --timesteps 20 --eta 0.85 --etaB 1 --deg sr4 --sigma_0 0.05

ERROR - main.py - 2023-01-04 15:05:00,845 - Traceback (most recent call last):
  File "E:/leisen-workspace/codelife/super-resolution/ddrm-master/main.py", line 164, in main
    runner.sample()
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\runners\diffusion.py", line 163, in sample
    self.sample_sequence(model, cls_fn)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\runners\diffusion.py", line 310, in sample_sequence
    x, _ = self.sample_image(x, model, H_funcs, y_0, sigma_0, last=False, cls_fn=cls_fn, classes=classes)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\runners\diffusion.py", line 338, in sample_image
    etaB=self.args.etaB, etaA=self.args.eta, etaC=self.args.eta, cls_fn=cls_fn, classes=classes)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\functions\denoising.py", line 53, in efficient_generalized_steps
    et = model(xt, t)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\parallel\data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\guided_diffusion\unet.py", line 657, in forward
    h = module(h, emb)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\guided_diffusion\unet.py", line 75, in forward
    x = layer(x, emb)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\guided_diffusion\unet.py", line 233, in forward
    self._forward, (x, emb), self.parameters(), self.use_checkpoint
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\guided_diffusion\nn.py", line 139, in checkpoint
    return func(*inputs)
  File "E:\leisen-workspace\codelife\super-resolution\ddrm-master\guided_diffusion\unet.py", line 242, in _forward
    h = in_conv(h)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\conv.py", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\envs\transenet\lib\site-packages\torch\nn\modules\conv.py", line 443, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Do you know why this error happens?

I tried to test the performance of DDRM by adding a photo of myself to the ood_celeba directory and running

python main.py --ni --config celeba_hq.yml --doc celeba --timesteps 20 --eta 0.85 --etaB 1 --deg sr4 --sigma_0 0.05 -i celeba_hq_sr4_sigma_0.05

While the examples from the original CelebA set are reconstructed very accurately, results on this additional photo are less than optimal, let's say :-) Now I'm not a celebrity, but I was wondering whether you have an idea of what is happening here


.

Is it possible to use this DDRM code to sample from p_theta(x)? For example, if I start with unit variance zero-mean gaussian noise, could this code be used similar to DDIM to unconditionally generate a sample? By the way, thank you for the excellent work on this repository. The code has been very useful so far in my group's research!

I've tested you model on all available configurations in Google Colab and got this when running image_512_cc.yml.

Error when using image_256_cc.yml #7
ERROR - main.py - 2022-07-21 11:16:31,872 - Traceback (most recent call last):
  File "main.py", line 164, in main
    runner.sample()
  File "/content/ddrm/runners/diffusion.py", line 135, in sample
    model.load_state_dict(torch.load(ckpt, map_location=self.device))
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 713, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 920, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xef'.

Could you recommend a possible fix?

Thank you for your good work?where can I find the train process code?

Hi,

Thank you for your inspiring paper and excellent code. I am really interested in this work and trying to reproduce the experimental results for the inpainting task. However, I didn't find how to use the code for inpainting. Could you please add the inpainting function to this repo? Your help would be greatly appreciated!

Regards,

Thanks for the beautiful work, can you give a short instruction on how to use model to train our own data?