lyccyl1
### 提交前必须检查以下项目 - [X] 请确保使用的是仓库最新代码（git pull），一些问题已被解决和修复。 - [X] 我已阅读[项目文档](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki)和[FAQ章节](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki/常见问题)并且已在Issue中对问题进行了搜索，没有找到相似问题和解决方案。 - [X] 第三方插件问题：例如[llama.cpp](https://github.com/ggerganov/llama.cpp)、[LangChain](https://github.com/hwchase17/langchain)、[text-generation-webui](https://github.com/oobabooga/text-generation-webui)等，同时建议到对应的项目中查找解决方案。 ### 问题类型 模型训练与精调 ### 基础模型 Chinese-LLaMA-2 (7B/13B) ### 操作系统 Linux ### 详细描述问题 ``` # 请在此处粘贴运行代码（请粘贴在本代码块里） ``` 尝试运行ZeRO3 实现模型并行报错 ``` # 运行脚本前请仔细阅读wiki(https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki/sft_scripts_zh) # Read the wiki(https://github.com/ymcui/Chinese-LLaMA-Alpaca-2/wiki/sft_scripts_zh) carefully before running the script lr=1e-4 lora_rank=64 lora_alpha=128 # llama1 lora lora_trainable="q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj" # falcon lora # lora_trainable="query_key_value,dense,dense_h_to_4h,dense_4h_to_h" modules_to_save="embed_tokens,lm_head" lora_dropout=0.05 pretrained_model=/data0/user/ycliu/work1/models/chinese-llama2-13b chinese_tokenizer_path=/data0/user/ycliu/work1/models/chinese-llama2-13b dataset_dir=/data0/user/ycliu/work1/data/sft_dataset/train_dataset_900k per_device_train_batch_size=1 per_device_eval_batch_size=1 gradient_accumulation_steps=2 max_seq_length=512 output_dir=/data0/user/ycliu/work1/data/Chinese-LLaMA-Alpaca-2/scripts/training/output_weight_chinese-llama2-13b_float32_zero3/output_weight_900k validation_file=/data0/user/ycliu/work1/data/sft_dataset/valid_dataset/translation2019zh_valid_clean.json deepspeed_config_file=/data0/user/ycliu/work1/llama/alpaca-lora/ds_config_zero3.json # device num_nodes=1 num_gpu_per_node=4 master_addr=gpu02 master_port=6108 node_rank=0 export CUDA_VISIBLE_DEVICES=1,2,3,6 torchrun --nnodes ${num_nodes} \ --nproc_per_node ${num_gpu_per_node}\ --node_rank ${node_rank} \ --master_addr ${master_addr} \ --master_port ${master_port} \ run_clm_sft_with_peft.py \ --deepspeed ${deepspeed_config_file} \ --model_name_or_path ${pretrained_model} \ --tokenizer_name_or_path ${chinese_tokenizer_path} \ --dataset_dir ${dataset_dir} \ --per_device_train_batch_size ${per_device_train_batch_size} \ --per_device_eval_batch_size ${per_device_eval_batch_size} \ --do_train \ --do_eval \ --seed $RANDOM \ --fp16 \ --num_train_epochs 1 \ --lr_scheduler_type cosine \ --learning_rate ${lr} \ --warmup_ratio 0.03 \ --weight_decay 0 \ --logging_strategy steps \ --logging_steps 10 \ --save_strategy steps \ --save_total_limit 3 \ --evaluation_strategy steps \ --eval_steps 10000 \ --save_steps 1000 \ --gradient_accumulation_steps ${gradient_accumulation_steps} \ --preprocessing_num_workers 8 \ --max_seq_length ${max_seq_length} \ --output_dir ${output_dir} \ --overwrite_output_dir \ --ddp_timeout 30000 \ --logging_first_step True \ --lora_rank ${lora_rank} \ --lora_alpha ${lora_alpha} \ --trainable ${lora_trainable} \ --lora_dropout ${lora_dropout} \ --modules_to_save ${modules_to_save} \ --torch_dtype float32 \ --validation_file ${validation_file} \ --load_in_kbits 16 \ --save_safetensors False \ --gradient_checkpointing \ --ddp_find_unused_parameters False ``` ### 依赖情况（代码类问题务必提供） ``` # 请在此处粘贴依赖情况（请粘贴在本代码块里） ``` ### 运行日志或截图 ``` 0%| | 0/112822 [00:00<?, ?it/s] RuntimeError: shape '[-1, 0]' is invalid for input of size 4478976 ret_val = func(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1822, in forward shift_logits = shift_logits.view(-1, self.config.vocab_size) RuntimeError: shape '[-1, 0]' is invalid for input of size 3760128 loss = self.module(*inputs, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl Traceback (most recent call last): File "/data0/user/ycliu/work1/data/Chinese-LLaMA-Alpaca-2/scripts/training/run_clm_sft_with_peft.py", line 524, in <module> result = forward_call(*args, **kwargs) File "/data0/user/ycliu/work1/data/Chinese-LLaMA-Alpaca-2/scripts/training/peft/peft_model.py", line 529, in forward main() File "/data0/user/ycliu/work1/data/Chinese-LLaMA-Alpaca-2/scripts/training/run_clm_sft_with_peft.py", line 497, in main return self.base_model( File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl train_result = trainer.train(resume_from_checkpoint=checkpoint) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/transformers/trainer.py", line 1553, in train return self._call_impl(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs)return inner_training_loop( File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/transformers/trainer.py", line 1835, in _inner_training_loop File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 848, in forward shift_logits = shift_logits.view(-1, self.config.vocab_size) RuntimeError: shape '[-1, 0]' is invalid for input of size 4644864tr_loss_step = self.training_step(model, inputs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/transformers/trainer.py", line 2679, in training_step loss = self.compute_loss(model, inputs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/transformers/trainer.py", line 2704, in compute_loss outputs = model(**inputs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn ret_val = func(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1822, in forward loss = self.module(*inputs, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) File "/data0/user/ycliu/work1/data/Chinese-LLaMA-Alpaca-2/scripts/training/peft/peft_model.py", line 529, in forward return self.base_model( File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1568, in _call_impl result = forward_call(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 848, in forward shift_logits = shift_logits.view(-1, self.config.vocab_size) RuntimeError: shape '[-1, 0]' is invalid for input of size 4202496 0%| | 0/112822 [00:15<?, ?it/s] [2024-07-01 23:53:13,049] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 92660) of binary: /data0/user/ycliu/.conda/envs/newnlp/bin/python Traceback (most recent call last): File "/data0/user/ycliu/.conda/envs/newnlp/bin/torchrun", line 8, in <module> sys.exit(main()) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper return f(*args, **kwargs) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/distributed/run.py", line 806, in main run(args) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/distributed/run.py", line 797, in run elastic_launch( File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/data0/user/ycliu/.conda/envs/newnlp/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ run_clm_sft_with_peft.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-07-01_23:53:13 host : gpu02 rank : 1 (local_rank: 1) exitcode : 1 (pid: 92661) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-07-01_23:53:13 host : gpu02 rank : 2 (local_rank: 2) exitcode : 1 (pid: 92662) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2024-07-01_23:53:13 host : gpu02 rank : 3 (local_rank: 3) exitcode : 1 (pid: 92663) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-07-01_23:53:13 host : gpu02 rank : 0 (local_rank: 0) exitcode : 1 (pid: 92660) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ ```