Title: CUDA RuntimeError: Unspecified Launch Failure during Training about transformers

[Sharan1712]

Update: I downgraded my PEFT to 10.0 and Transformers to 4.39.0 and it is working fine now

[Hongjie1Chu]

and when i set :
device_map["model.embed_tokens"] = 0
device_map["model.norm.weight"] = 0

it will not error at start ,but it will error after:

[younesbelkada]

Hi @Hongjie1Chu !
In principle the device order shouldn't affect the training behaviour - can you let us know what happens when you run the training script with CUDA_LAUNCH_BLOCKING=1 ? Also do you run your training script with accelerate launch xxx or python xxx.py?

[Sharan1712]

I too am facing a similar issue.
I haven't made any changes to my code but all of a sudden, my code gives this error after training for like 30 steps.

[Hongjie1Chu]

thanks for your answer!

[Sharan1712]

Has there been a solution for this yet? I tried using the latest version of transformers and it still gave this issue. I want to use some of the new quantization methods.

[Sharan1712]

@ArthurZucker @younesbelkada @muellerzr

[younesbelkada]

Hi !
It is hard for us to debug without a proper error trace, can you re-run the training script with CUDA_LAUNCH_BLOCKING=1 and paste the error trace here?