[BUG] Config.json file not generated after finetuning model using Autotrain with Nvidia DGX about autotrain-advanced

[abhishekkrthakur]

you must use latest version of transformers and peft if you want to use colab or any other machine. for inferencce endpoints im checking internally

[yitzshapiro]

I also have this issue

[abhishekkrthakur]

you have trained an adapter model (peft=true) which doesnt generate config.json. it doesnt matter though, the model can still be loaded with AutoModelForCausalLM or with Inference endpoints. are you getting an error?

[RishubAmpliforce]

Error Trace:
Server message:Endpoint failed to start. 203Z","level":"ERROR","fields":{"message":"Download encountered an error: \nTraceback (most recent call last):\n\n File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py", line 15, in download_and_unload_peft\n model = AutoPeftModelForCausalLM.from_pretrained(\n\n File "/opt/conda/lib/python3.10/site-packages/peft/auto.py", line 72, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n File "/opt/conda/lib/python3.10/site-packages/peft/config.py", line 151, in from_pretrained\n return cls.from_peft_type(**kwargs)\n\n File "/opt/conda/lib/python3.10/site-packages/peft/config.py", line 118, in from_peft_type\n return config_cls(**kwargs)\n\nTypeError: LoraConfig.init() got an unexpected keyword argument 'layer_replication'\n\n\nDuring handling of the above exception, another exception occurred:\n\n\nTraceback (most recent call last):\n\n File "/opt/conda/bin/text-generation-server", line 8, in \n sys.exit(app())\n\n File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 225, in download_weights\n utils.download_and_unload_peft(\n\n File "/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/peft.py", line 23, in download_and_unload_peft\n model = AutoPeftModelForSeq2SeqLM.from_pretrained(\n\n File "/opt/conda/lib/python3.10/site-packages/peft/auto.py", line 72, in from_pretrained\n peft_config = PeftConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n File "/opt/conda/lib/python3.10/site-packages/peft/config.py", line 151, in from_pretrained\n return cls.from_peft_type(**kwargs)\n\n File "/opt/conda/lib/python3.10/site-packages/peft/config.py", line 118, in from_peft_type\n return config_cls(**kwargs)\n\nTypeError: LoraConfig.init() got an unexpected keyword argument 'layer_replication'\n"},"target":"text_generation_launcher","span":{"name":"download"},"spans":[{"name":"download"}]} Error: DownloadError

I tried using inference endpoints

[abhishekkrthakur]

does it work if you dont choose tgi container?

[RishubAmpliforce]

I have'nt tried that

[RishubAmpliforce]

I tried downloading the model in colab