Nan appears during sd3 training about diffusers

[sayakpaul]

We don't support full text to image fine-tuning. However, you can refer to which provides this support: https://github.com/bghira/SimpleTuner.

[Luciennnnnnn]

I also encounter NAN problem when adapting SD3 to another task

[bghira]

please try --max_grad_norm=1

[bghira]

also #8592 might help

[Luciennnnnnn]

@bghira I do use these strategies, however, nan still exists.

[bghira]

can you try adding QK Norm blocks back?

bghira/SimpleTuner#469

[bghira]

this will extend the model to 3B parameters for higher performance (the 2B seems limited) and reintroduces the missing qk norm blocks

prepare for a longer training session

[Luciennnnnnn]

Hi @bghira, thank you for your suggestion! It seems be hard to train for many users by introducing extra 1B parameters, i.e. extensive computing power and longer training iterations.

[Luciennnnnnn]

I mean in order to fine-tuning a model, it is not wise to introduce at least 1B parameters. That should not be the correct direction for SD3.

[bghira]

someone has to!

[AmericanPresidentJimmyCarter]

I mean in order to fine-tuning a model, it is not wise to introduce at least 1B parameters. That should not be the correct direction for SD3.

You can just use the model there with the original number of blocks and copy in the parameters except QK norms, then init them as I had.

[sayakpaul]

My question is what happens when you try a lower learning rate or try to overfit a single batch of training data? In any case, this seems more like a discussion to me and not an "issue".

[HaozheZhao]

I also encounter NAN problem when tuning the SD3 for image editing. Is there any new updates?

[sayakpaul]

Did you try the "logit_normal" weighting scheme?

[rardz]

try --mixed_precision="bf16" may help

[BJQ123456]

Where is this file located? train_text_to_imagesd3.py, I couldn't find it.Thanks

[sayakpaul]

Closing this since switching to "logit_normal" weighting scheme for loss resolves this issue in most cases. If not, please re-open and I will turn it into a discussion.