[LOADING TIME ISSUE] 2hrs of loading time Mistral 8x7b in AWS sagemaker about autotrain-advanced

[abhishekkrthakur]

mixtral on h100s: https://huggingface.co/blog/abhishek/autotrain-mixtral-dgx-cloud-local

[abhishekkrthakur]

you seem to have quantization params but peft is not enabled which effectively disables quantization. try adding --peft param.

also, the time to load the model depends on download time and hardware used.

[RishubAmpliforce]

I have tried with Peft as well

On hardware regards I used 8xA10G GPUs and the model is already downloaded.

The delay I am facing is because it loads model in the RAM first and then later the GPU.

I tried finetuning mistral 8x7b across both the cases.

Any advice to reduce the model load time. Preferrably load to the GPU directly.

If this is expected behaviour curious why we load the model to the RAM and then to the GPU later.

[abhishekkrthakur]

it takes time to load but it takes like a few minutes not hours. are you using ssd?

[RishubAmpliforce]

I am using SSD.