I can successfully fine-tune the model with 1 Epoch by using the latest commits in the repo. (all arguments are default)
However, when I set Epoch to 2, it goes all the way to the end but it fails with many No such file or directory:
. I was wondering if something is hard-coded for 1 Epoch.
Loading best model from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400 (score: 1.3070000410079956).
[2023-03-28 12:06:44,437] [INFO] [logging.py:93:log_dist] [Rank 0] DeepSpeed info: version=0.8.3, git-hash=unknown, git-branch=unknown
[2023-03-28 12:06:44,450] [INFO] [logging.py:93:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module fused_adam, skipping build step...
Loading extension module fused_adam...
Time to load fused_adam op: 0.0007417201995849609 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0001647472381591797 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module fused_adam, skipping build step...
Loading extension module fused_adam...
Time to load fused_adam op: 0.0008955001831054688 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module fused_adam, skipping build step...
Loading extension module fused_adam...
Time to load fused_adam op: 0.0008783340454101562 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module fused_adam, skipping build step...
Loading extension module fused_adam...
Time to load fused_adam op: 0.0008599758148193359 seconds
[2023-03-28 12:06:44,561] [INFO] [logging.py:93:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00018143653869628906 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00016498565673828125 seconds
[2023-03-28 12:06:44,572] [INFO] [logging.py:93:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2023-03-28 12:06:44,572] [INFO] [utils.py:55:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'>
[2023-03-28 12:06:44,573] [INFO] [logging.py:93:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2023-03-28 12:06:44,756] [INFO] [utils.py:829:see_memory_usage] Stage 3 initialize beginning
[2023-03-28 12:06:44,756] [INFO] [utils.py:830:see_memory_usage] MA 22.7 GB Max_MA 31.48 GB CA 46.49 GB Max_CA 46 GB
[2023-03-28 12:06:44,757] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 23.99 GB, percent = 3.2%
[2023-03-28 12:06:44,758] [INFO] [stage3.py:113:__init__] Reduce bucket size 16777216
[2023-03-28 12:06:44,758] [INFO] [stage3.py:114:__init__] Prefetch bucket size 15099494
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0002541542053222656 seconds
[2023-03-28 12:06:44,893] [INFO] [utils.py:829:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2023-03-28 12:06:44,893] [INFO] [utils.py:830:see_memory_usage] MA 22.7 GB Max_MA 22.7 GB CA 46.49 GB Max_CA 46 GB
[2023-03-28 12:06:44,894] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 23.99 GB, percent = 3.2%
Parameter Offload: Total persistent parameters: 811008 in 114 params
[2023-03-28 12:06:45,040] [INFO] [utils.py:829:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2023-03-28 12:06:45,040] [INFO] [utils.py:830:see_memory_usage] MA 22.7 GB Max_MA 22.7 GB CA 46.49 GB Max_CA 46 GB
[2023-03-28 12:06:45,040] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 23.99 GB, percent = 3.2%
[2023-03-28 12:06:45,177] [INFO] [utils.py:829:see_memory_usage] Before creating fp16 partitions
[2023-03-28 12:06:45,177] [INFO] [utils.py:830:see_memory_usage] MA 22.7 GB Max_MA 22.7 GB CA 46.49 GB Max_CA 46 GB
[2023-03-28 12:06:45,177] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 24.0 GB, percent = 3.2%
[2023-03-28 12:06:47,552] [INFO] [utils.py:829:see_memory_usage] After creating fp16 partitions: 2
[2023-03-28 12:06:47,553] [INFO] [utils.py:830:see_memory_usage] MA 25.52 GB Max_MA 25.52 GB CA 28.33 GB Max_CA 46 GB
[2023-03-28 12:06:47,553] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 23.99 GB, percent = 3.2%
[2023-03-28 12:06:47,689] [INFO] [utils.py:829:see_memory_usage] Before creating fp32 partitions
[2023-03-28 12:06:47,689] [INFO] [utils.py:830:see_memory_usage] MA 25.52 GB Max_MA 25.52 GB CA 28.33 GB Max_CA 28 GB
[2023-03-28 12:06:47,690] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 23.99 GB, percent = 3.2%
[2023-03-28 12:06:47,842] [INFO] [utils.py:829:see_memory_usage] After creating fp32 partitions
[2023-03-28 12:06:47,843] [INFO] [utils.py:830:see_memory_usage] MA 31.16 GB Max_MA 32.1 GB CA 35.85 GB Max_CA 36 GB
[2023-03-28 12:06:47,843] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 24.07 GB, percent = 3.2%
[2023-03-28 12:06:47,979] [INFO] [utils.py:829:see_memory_usage] Before initializing optimizer states
[2023-03-28 12:06:47,980] [INFO] [utils.py:830:see_memory_usage] MA 31.16 GB Max_MA 31.16 GB CA 35.85 GB Max_CA 36 GB
[2023-03-28 12:06:47,980] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 24.07 GB, percent = 3.2%
[2023-03-28 12:06:48,644] [INFO] [utils.py:829:see_memory_usage] After initializing optimizer states
[2023-03-28 12:06:48,644] [INFO] [utils.py:830:see_memory_usage] MA 42.43 GB Max_MA 46.19 GB CA 50.88 GB Max_CA 51 GB
[2023-03-28 12:06:48,644] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 24.0 GB, percent = 3.2%
[2023-03-28 12:06:48,645] [INFO] [stage3.py:376:_setup_for_real_optimizer] optimizer state initialized
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0003037452697753906 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00035309791564941406 seconds
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00029277801513671875 seconds
[2023-03-28 12:06:48,919] [INFO] [utils.py:829:see_memory_usage] After initializing ZeRO optimizer
[2023-03-28 12:06:48,919] [INFO] [utils.py:830:see_memory_usage] MA 45.28 GB Max_MA 46.05 GB CA 65.12 GB Max_CA 65 GB
[2023-03-28 12:06:48,919] [INFO] [utils.py:838:see_memory_usage] CPU Virtual Memory: used = 24.31 GB, percent = 3.2%
[2023-03-28 12:06:48,920] [INFO] [logging.py:93:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2023-03-28 12:06:48,920] [INFO] [logging.py:93:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR
[2023-03-28 12:06:48,920] [INFO] [logging.py:93:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7fbdb36c2fa0>
[2023-03-28 12:06:48,920] [INFO] [logging.py:93:log_dist] [Rank 0] step=0, skipped=0, lr=[1e-05], mom=[[0.9, 0.999]]
[2023-03-28 12:06:48,920] [INFO] [config.py:1018:print] DeepSpeedEngine configuration:
[2023-03-28 12:06:48,920] [INFO] [config.py:1022:print] activation_checkpointing_config {
"partition_activations": false,
"contiguous_memory_optimization": false,
"cpu_checkpointing": false,
"number_checkpoints": null,
"synchronize_checkpoint_boundary": false,
"profile": false
}
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] amp_enabled .................. False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] amp_params ................... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] autotuning_config ............ {
"enabled": false,
"start_step": null,
"end_step": null,
"metric_path": null,
"arg_mappings": null,
"metric": "throughput",
"model_info": null,
"results_dir": "autotuning_results",
"exps_dir": "autotuning_exps",
"overwrite": true,
"fast": true,
"start_profile_step": 3,
"end_profile_step": 5,
"tuner_type": "gridsearch",
"tuner_early_stopping": 5,
"tuner_num_trials": 50,
"model_info_path": null,
"mp_size": 1,
"max_train_batch_size": null,
"min_train_batch_size": 1,
"max_train_micro_batch_size_per_gpu": 1.024000e+03,
"min_train_micro_batch_size_per_gpu": 1,
"num_tuning_micro_batch_sizes": 3
}
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] bfloat16_enabled ............. True
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] checkpoint_parallel_write_pipeline False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] checkpoint_tag_validation_enabled True
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] checkpoint_tag_validation_fail False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fbdb82352e0>
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] communication_data_type ...... None
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] curriculum_enabled_legacy .... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] curriculum_params_legacy ..... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] data_efficiency_enabled ...... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] dataloader_drop_last ......... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] disable_allgather ............ False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] dump_state ................... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] dynamic_loss_scale_args ...... None
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_enabled ........... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_gas_boundary_resolution 1
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_layer_name ........ bert.encoder.layer
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_layer_num ......... 0
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_max_iter .......... 100
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_stability ......... 1e-06
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_tol ............... 0.01
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] eigenvalue_verbose ........... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] elasticity_enabled ........... False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] flops_profiler_config ........ {
"enabled": false,
"profile_step": 1,
"module_depth": -1,
"top_modules": 1,
"detailed": true,
"output_file": null
}
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] fp16_auto_cast ............... None
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] fp16_enabled ................. False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] fp16_master_weights_and_gradients False
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] global_rank .................. 0
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] grad_accum_dtype ............. None
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] gradient_accumulation_steps .. 1
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] gradient_clipping ............ 1.0
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] gradient_predivide_factor .... 1.0
[2023-03-28 12:06:48,921] [INFO] [config.py:1022:print] initial_dynamic_scale ........ 1
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] load_universal_checkpoint .... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] loss_scale ................... 1.0
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] memory_breakdown ............. False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] nebula_config ................ {
"enabled": false,
"persistent_storage_path": null,
"persistent_time_interval": 100,
"num_of_version_in_retention": 2,
"enable_nebula_load": true,
"load_path": null
}
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] optimizer_legacy_fusion ...... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] optimizer_name ............... adamw
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] optimizer_params ............. {'lr': 1e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] pld_enabled .................. False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] pld_params ................... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] prescale_gradients ........... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] scheduler_name ............... WarmupLR
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 1e-05, 'warmup_num_steps': 0}
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] sparse_attention ............. None
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] sparse_gradients_enabled ..... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] steps_per_print .............. 2000
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] train_batch_size ............. 32
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] train_micro_batch_size_per_gpu 8
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] use_node_local_storage ....... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] wall_clock_breakdown ......... False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] world_size ................... 4
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] zero_allow_untested_optimizer False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=16777216 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=15099494 param_persistence_threshold=40960 model_persistence_threshold=sys.maxsize max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] zero_enabled ................. True
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] zero_force_ds_cpu_optimizer .. True
[2023-03-28 12:06:48,922] [INFO] [config.py:1022:print] zero_optimization_stage ...... 3
[2023-03-28 12:06:48,922] [INFO] [config.py:1007:print_user_config] json = {
"bf16": {
"enabled": true
},
"optimizer": {
"type": "AdamW",
"params": {
"lr": 1e-05,
"betas": [0.9, 0.999],
"eps": 1e-08,
"weight_decay": 0.0
}
},
"scheduler": {
"type": "WarmupLR",
"params": {
"warmup_min_lr": 0,
"warmup_max_lr": 1e-05,
"warmup_num_steps": 0
}
},
"zero_optimization": {
"stage": 3,
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1.000000e+09,
"reduce_bucket_size": 1.677722e+07,
"stage3_prefetch_bucket_size": 1.509949e+07,
"stage3_param_persistence_threshold": 4.096000e+04,
"stage3_max_live_parameters": 1.000000e+09,
"stage3_max_reuse_distance": 1.000000e+09,
"stage3_gather_16bit_weights_on_model_save": true
},
"gradient_accumulation_steps": 1,
"gradient_clipping": 1.0,
"steps_per_print": 2.000000e+03,
"train_batch_size": 32,
"train_micro_batch_size_per_gpu": 8,
"wall_clock_breakdown": false
}
Using /home/maziyar/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.00025963783264160156 seconds
Attempting to resume from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400
[2023-03-28 12:06:48,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loading checkpoint from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400/global_step1400/zero_pp_rank_0_mp_rank_00_model_states.pt...
[2023-03-28 12:06:48,976] [INFO] [torch_checkpoint_engine.py:25:load] [Torch] Loaded checkpoint from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400/global_step1400/zero_pp_rank_0_mp_rank_00_model_states.pt.
[2023-03-28 12:06:48,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loading checkpoint from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400/global_step1400/zero_pp_rank_0_mp_rank_00_model_states.pt...
[2023-03-28 12:06:49,028] [INFO] [torch_checkpoint_engine.py:25:load] [Torch] Loaded checkpoint from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400/global_step1400/zero_pp_rank_0_mp_rank_00_model_states.pt.
[2023-03-28 12:06:49,469] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loading checkpoint from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400/global_step1400/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
[2023-03-28 12:06:56,282] [INFO] [torch_checkpoint_engine.py:25:load] [Torch] Loaded checkpoint from local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-1400/global_step1400/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
[2023-03-28 12:06:56,283] [INFO] [engine.py:3043:_get_all_zero_checkpoint_state_dicts] successfully read 4 ZeRO state_dicts for rank 0
[2023-03-28 12:07:00,465] [INFO] [engine.py:2983:_load_zero_checkpoint] loading 4 zero partition checkpoints for rank 0
{'train_runtime': 15522.5667, 'train_samples_per_second': 6.568, 'train_steps_per_second': 0.205, 'train_loss': 1.0332834323995606, 'epoch': 2.0}
Deleting older checkpoint [local_dolly_training/dolly__2023-03-28T07:46:32/checkpoint-3000] due to args.save_total_limit
2023-03-28 12:07:01 ERROR [__main__] main failed
Traceback (most recent call last):
File "/home/maziyar/apps/dolly/training/trainer.py", line 260, in <module>
main()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 252, in main
train(**kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 215, in train
trainer.train()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train
return inner_training_loop(
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
shutil.rmtree(checkpoint)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'rng_state_3.pth'
Traceback (most recent call last):
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/runpy.py", line 197, in _run_module_as_main
2023-03-28 12:07:01 ERROR [__main__] main failed
Traceback (most recent call last):
File "/home/maziyar/apps/dolly/training/trainer.py", line 260, in <module>
main()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 252, in main
train(**kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 215, in train
trainer.train()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train
return inner_training_loop(
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
shutil.rmtree(checkpoint)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'rng_state_3.pth'
Traceback (most recent call last):
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/maziyar/apps/dolly/training/trainer.py", line 260, in <module>
return _run_code(code, main_globals, None,
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/runpy.py", line 87, in _run_code
main()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
exec(code, run_globals)
File "/home/maziyar/apps/dolly/training/trainer.py", line 260, in <module>
main()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1053, in main
return self.main(*args, **kwargs)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
rv = self.invoke(ctx)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 252, in main
train(**kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 215, in train
return __callback(*args, **kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 252, in main
trainer.train()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train
train(**kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 215, in train
trainer.train()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train
return inner_training_loop(
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
return inner_training_loop(
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
shutil.rmtree(checkpoint)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 734, in rmtree
shutil.rmtree(checkpoint)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())_rmtree_safe_fd(fd, path, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
onerror(os.unlink, fullname, sys.exc_info())
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd
FileNotFoundError: [Errno 2] No such file or directory: 'rng_state_3.pth'
os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'rng_state_3.pth'
2023-03-28 12:07:01 ERROR [__main__] main failed
Traceback (most recent call last):
File "/home/maziyar/apps/dolly/training/trainer.py", line 260, in <module>
main()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 252, in main
train(**kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 215, in train
trainer.train()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train
return inner_training_loop(
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
shutil.rmtree(checkpoint)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt'
Traceback (most recent call last):
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/maziyar/apps/dolly/training/trainer.py", line 260, in <module>
main()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 252, in main
train(**kwargs)
File "/home/maziyar/apps/dolly/training/trainer.py", line 215, in train
trainer.train()
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1527, in train
return inner_training_loop(
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/site-packages/transformers/trainer.py", line 1920, in _inner_training_loop
shutil.rmtree(checkpoint)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 734, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 690, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/maziyar/anaconda3/envs/dolly/lib/python3.9/shutil.py", line 688, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt'
[2023-03-28 12:07:02,780] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 235676
[2023-03-28 12:07:04,022] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 235677
[2023-03-28 12:07:04,023] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 235678
[2023-03-28 12:07:04,037] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 235679
[2023-03-28 12:07:04,051] [ERROR] [launch.py:324:sigkill_handler] ['/home/maziyar/anaconda3/envs/dolly/bin/python', '-u', '-m', 'training.trainer', '--local_rank=3', '--deepspeed', '/home/maziyar/apps/dolly/config/ds_z3_bf16_config.json', '--epochs', '2', '--local-output-dir', 'local_dolly_training/dolly__2023-03-28T07:46:32', '--dbfs-output-dir', 'dolly_training/dolly__2023-03-28T07:46:32', '--per-device-train-batch-size', '8', '--per-device-eval-batch-size', '8', '--lr', '1e-5'] exits with return code = 1