Strange interaction for DeepZero via text-generation-webui? about autoawq

[casper-hansen]

I’m not quite sure. I have seen this error before, but I am not sure what is causing it to happen.

Did you compile from main or did you use the pypi package?

[cal066]

This is from main, git revision 8788fe1. I pointed pip to github.

[casper-hansen]

I think I understand better now. DeepSpeed is trying to copy data that does not exist. This happens because we start by initializing the model with empty weights:

AutoAWQ/awq/models/base.py

Lines 337 to 338 in 8788fe1

	with init_empty_weights():
	model = AutoModelForCausalLM.from_config(config=config, torch_dtype=torch_dtype, trust_remote_code=trust_remote_code)

In general, I am not sure AutoAWQ is even compatible with DeepSpeed since we rely on accelerate.
huggingface/accelerate#1530 (comment)

[cal066]

That was what I suspected when I saw some comments on DeepSpeed about copying empty weights. Hopefully there is a way to stop DeepSpeed from kicking in for model types that are incompatible.

[casper-hansen]

Closing this since we use accelerate.