New modalities about diffusers

[richardrl]

@anton-l @patrickvonplaten Thanks for your input thus far.

I took the latest commit (as of this moment) and made a minimum reproduction of a 1D MLP model and training.

I had to make an additional modification to pipeline_ddpm.py to support noise samples of the right shape.

bimodal_testt.zip

python3 train_unconditional_gaussian_test.py This runs a test on a bimodal gaussian distribution centered at +33, -33 with low variance

It seems to not capture the -33 mode after an epoch or two. Am running the training overnight to see what happens.

Welcome you guys to try running this to see if there's anything I did wrong

[patrickvonplaten]

Taking a look at the function as is:

    def training_step(self, original_samples: torch.Tensor, noise: torch.Tensor, timesteps: torch.Tensor):
        if timesteps.dim() != 1:
            raise ValueError("`timesteps` must be a 1D tensor")

        device = original_samples.device
        batch_size = original_samples.shape[0]
        timesteps = timesteps.reshape(batch_size, 1, 1, 1)

        sqrt_alpha_prod = self.alphas_cumprod[timesteps] ** 0.5
        sqrt_one_minus_alpha_prod = (1 - self.alphas_cumprod[timesteps]) ** 0.5
        noisy_samples = sqrt_alpha_prod.to(device) * original_samples + sqrt_one_minus_alpha_prod.to(device) * noise
        return noisy_samples

Note that the input can be both torch and numpy tensors -> this should be changed.

Also there shouldn't be any .to(device) statements, nor framework and modality spefific .reshape(...) operation.

I'd be in favor of implementing framework specific (one for PT one for TF) functions called

def extract(....) in SchedulerMixin that have if framework == "pt" statements in them. Also note that we shouldn't assume to know the dimension of the input original_samples

[patrickvonplaten]

@anton-l - we need to make sure that training_step is both framework agnostic and shape agnostic

[patrickvonplaten]

BTW, it's super nice to get all your feedback here @richardrl - thanks a lot!

[patil-suraj]

Thanks for reporting @richardrl !
Indeed the plan is to support multiple modalities, but we haven't yet tested the schedulers with 1D data.

cc @patrickvonplaten @anton-l

[anton-l]

Now we use match_shape(timestaps, original_samples) for everything, which is framework- and shape-agnostic: https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_ddpm.py#L146

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

@anton-l as you've re-opened the issue -> are you planning on doing something with it?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[anton-l]

I think we have support for all shapes now, agreed with the stalebot :)