Corrupted Model after training about everydream2trainer

[victorchall]

P100 is pretty old tech (1070/1080 era hardware), its possible its going to need a special compiled xformers if it will work at all with xformers, I'm not sure I can help much with this.

[victorchall]

Discussed on the discord, but I ran a test and no problems. The error is in inference after something already went wrong saving the files and they are corrupt, maybe process was killed? I would look at your output on the training cell to spot the actual problem, its too late at this point to fix the corrupt files.

[CypherpunkSamurai]

I checked the logs and the training had completed successfully, and the files also got copied to two remotes. after training (checkpoint and diffuser)

The training was running on a P100. I tried training on a T4 and a P100 and turns out in T4 trained model are of right sizes, where's on a P100 the trained models were half the original sizes. The training notebook and configs were copied in both cases.

I also reverted back to commit 27b66ab54f9d2515211a373c7f3127d5b66b4921 with git reset --hard and the trained model worked fine on the P100.

Not sure what's happening here you might wanna tag this "additional information required" for now. I'll try running tests again / providing you the notebooks.