freezing of LLM during pretrain stage about onellm

[csuhan]

@eugenelet Thanks for pointing out that! The current code is modified from the inference code, so we MISSED some functions such as parameter freezing. We will fix it soon. For now, you can temporally add a few lines to freeze the parameters in

OneLLM/model/LLM/onellm.py

Line 332 in db02333

# TODO: Freeze some parameters at here. Freeze LLM for pretraining and Projection for finetuining.

For example, freezing LLM:

for param in list(self.layers.parameters()) + list(self.tok_embeddings.parameters()) + list(self.norm.parameters()) + list(self.output.parameters()):
            param.requires_grad = False

[eugenelet]

Thanks for point this out @csuhan ! I'll use this workaround for now. Keep on the great work!