Comments (7)
abhishekkrthakur
commented on July 4, 2024
it seems like your dataset has very short sentences? if thats the case, reduce blocksize to quite a small value like 32 or even 16
from autotrain-advanced.
treeoflovely
commented on July 4, 2024
I bumped it down to 32 (anything less would truncate the vast majority) and it still throws the same error.
🚀 INFO | 2024-04-02 16:28:52 | main🚋332 - model dtype: torch.float16
🚀 INFO | 2024-04-02 16:28:52 | main🚋340 - preparing peft model...
🚀 INFO | 2024-04-02 16:29:04 | main🚋391 - Using block size 32
🚀 INFO | 2024-04-02 16:29:04 | main🚋453 - creating trainer
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 0 examples [00:00, ? examples/s]
❌ ERROR | 2024-04-02 16:29:05 | autotrain.trainers.common:wrapper:115 - train has failed due to an exception: Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1676, in _prepare_split_single
for key, record in generator:
File "/app/env/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 30, in _generate_examples
for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 517, in data_generator
yield from constant_length_iterator
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 457, in iter
buffer.append(self.formatting_func(next(iterator)))
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 434, in
self.formatting_func = lambda x: x[dataset_text_field]
KeyError: 'text'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 520, in _prepare_packed_dataloader
packed_dataset = Dataset.from_generator(
File "/app/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1072, in from_generator
).read()
File "/app/env/lib/python3.10/site-packages/datasets/io/generator.py", line 47, in read
self.builder.download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
super()._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1555, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1712, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/src/autotrain/trainers/common.py", line 112, in wrapper
return func(*args, **kwargs)
File "/app/src/autotrain/trainers/clm/main.py", line 480, in train
trainer = SFTTrainer(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 282, in init
train_dataset = self._prepare_dataset(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 419, in _prepare_dataset
return self._prepare_packed_dataloader(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 524, in _prepare_packed_dataloader
raise ValueError(
ValueError: Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
❌ ERROR | 2024-04-02 16:29:05 | autotrain.trainers.common:wrapper:116 - Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
from autotrain-advanced.
treeoflovely
commented on July 4, 2024
Same for block size 16: ===== Application Startup at 2024-04-02 15:59:30 =====
==========
== CUDA ==
CUDA Version 12.1.1
Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.
Use the NVIDIA Container Toolkit to start this container with GPU support; see
https://docs.nvidia.com/datacenter/cloud-native/ .
Found existing installation: autotrain-advanced 0.7.43.dev0
Uninstalling autotrain-advanced-0.7.43.dev0:
Successfully uninstalled autotrain-advanced-0.7.43.dev0
Collecting autotrain-advanced
Downloading autotrain_advanced-0.7.42-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: albumentations==1.3.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.3.1)
Requirement already satisfied: codecarbon==2.2.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.2.3)
Requirement already satisfied: datasets~=2.14.0 in ./env/lib/python3.10/site-packages (from datasets[vision]=2.14.0->autotrain-advanced) (2.14.7)=2.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (2.1.3)
Requirement already satisfied: evaluate==0.3.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.3.0)
Requirement already satisfied: ipadic==1.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.0.0)
Requirement already satisfied: jiwer==3.0.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.0.2)
Requirement already satisfied: joblib==1.3.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.3.1)
Requirement already satisfied: loguru==0.7.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.7.0)
Requirement already satisfied: pandas>=1.4 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.2.1)
Requirement already satisfied: nltk==3.8.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.8.1)
Requirement already satisfied: optuna==3.3.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.3.0)
Requirement already satisfied: Pillow==10.0.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (10.0.0)
Requirement already satisfied: protobuf==4.23.4 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.23.4)
Requirement already satisfied: sacremoses==0.0.53 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.0.53)
Requirement already satisfied: scikit-learn==1.3.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.3.0)
Requirement already satisfied: sentencepiece==0.1.99 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.99)
Requirement already satisfied: tqdm==4.65.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.65.0)
Requirement already satisfied: werkzeug==2.3.6 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.3.6)
Requirement already satisfied: xgboost==1.7.6 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.7.6)
Requirement already satisfied: huggingface-hub>=0.16.4 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.22.2)
Requirement already satisfied: requests==2.31.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.31.0)
Requirement already satisfied: einops==0.6.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.6.1)
Requirement already satisfied: invisible-watermark==0.2.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.2.0)
Requirement already satisfied: packaging==23.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (23.1)
Requirement already satisfied: tensorboard in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.16.2)
Requirement already satisfied: peft==0.10.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.10.0)
Requirement already satisfied: trl==0.8.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.8.1)
Requirement already satisfied: tiktoken==0.6.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.6.0)
Requirement already satisfied: transformers==4.39.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (4.39.0)
Requirement already satisfied: accelerate==0.28.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.28.0)
Requirement already satisfied: diffusers==0.27.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.27.2)
Requirement already satisfied: bitsandbytes==0.43.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.43.0)
Requirement already satisfied: rouge-score==0.1.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.2)
Requirement already satisfied: py7zr==0.20.6 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.20.6)
Requirement already satisfied: fastapi==0.104.1 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.104.1)
Requirement already satisfied: uvicorn==0.22.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.22.0)
Requirement already satisfied: python-multipart==0.0.6 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.0.6)
Requirement already satisfied: gradio==3.41.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (3.41.0)
Requirement already satisfied: pydantic==2.4.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.4.2)
Requirement already satisfied: hf-transfer in ./env/lib/python3.10/site-packages (from autotrain-advanced) (0.1.6)
Requirement already satisfied: pyngrok==7.0.3 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (7.0.3)
Requirement already satisfied: authlib==1.3.0 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.3.0)
Requirement already satisfied: itsdangerous==2.1.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (2.1.2)
Requirement already satisfied: seqeval==1.2.2 in ./env/lib/python3.10/site-packages (from autotrain-advanced) (1.2.2)
Requirement already satisfied: numpy>=1.17 in ./env/lib/python3.10/site-packages (from accelerate==0.28.0->autotrain-advanced) (1.26.4)
Requirement already satisfied: psutil in ./env/lib/python3.10/site-packages (from accelerate==0.28.0->autotrain-advanced) (5.9.8)
Requirement already satisfied: pyyaml in ./env/lib/python3.10/site-packages (from accelerate==0.28.0->autotrain-advanced) (6.0.1)
Requirement already satisfied: torch>=1.10.0 in ./env/lib/python3.10/site-packages (from accelerate==0.28.0->autotrain-advanced) (2.2.2)
Requirement already satisfied: safetensors>=0.3.1 in ./env/lib/python3.10/site-packages (from accelerate==0.28.0->autotrain-advanced) (0.4.2)
Requirement already satisfied: scipy>=1.1.0 in ./env/lib/python3.10/site-packages (from albumentations==1.3.1->autotrain-advanced) (1.12.0)
Requirement already satisfied: scikit-image>=0.16.1 in ./env/lib/python3.10/site-packages (from albumentations==1.3.1->autotrain-advanced) (0.22.0)
Requirement already satisfied: qudida>=0.0.4 in ./env/lib/python3.10/site-packages (from albumentations==1.3.1->autotrain-advanced) (0.0.4)
Requirement already satisfied: opencv-python-headless>=4.1.1 in ./env/lib/python3.10/site-packages (from albumentations==1.3.1->autotrain-advanced) (4.9.0.80)
Requirement already satisfied: cryptography in ./env/lib/python3.10/site-packages (from authlib==1.3.0->autotrain-advanced) (42.0.5)
Requirement already satisfied: arrow in ./env/lib/python3.10/site-packages (from codecarbon==2.2.3->autotrain-advanced) (1.3.0)
Requirement already satisfied: pynvml in ./env/lib/python3.10/site-packages (from codecarbon==2.2.3->autotrain-advanced) (11.5.0)
Requirement already satisfied: py-cpuinfo in ./env/lib/python3.10/site-packages (from codecarbon==2.2.3->autotrain-advanced) (9.0.0)
Requirement already satisfied: fuzzywuzzy in ./env/lib/python3.10/site-packages (from codecarbon==2.2.3->autotrain-advanced) (0.18.0)
Requirement already satisfied: click in ./env/lib/python3.10/site-packages (from codecarbon==2.2.3->autotrain-advanced) (8.1.7)
Requirement already satisfied: importlib-metadata in ./env/lib/python3.10/site-packages (from diffusers==0.27.2->autotrain-advanced) (7.1.0)
Requirement already satisfied: filelock in ./env/lib/python3.10/site-packages (from diffusers==0.27.2->autotrain-advanced) (3.13.1)
Requirement already satisfied: regex!=2019.12.17 in ./env/lib/python3.10/site-packages (from diffusers==0.27.2->autotrain-advanced) (2023.12.25)
Requirement already satisfied: dill in ./env/lib/python3.10/site-packages (from evaluate==0.3.0->autotrain-advanced) (0.3.7)
Requirement already satisfied: xxhash in ./env/lib/python3.10/site-packages (from evaluate==0.3.0->autotrain-advanced) (3.4.1)
Requirement already satisfied: multiprocess in ./env/lib/python3.10/site-packages (from evaluate==0.3.0->autotrain-advanced) (0.70.15)
Requirement already satisfied: fsspec>=2021.05.0 in ./env/lib/python3.10/site-packages (from fsspec[http]>=2021.05.0->evaluate==0.3.0->autotrain-advanced) (2023.10.0)
Requirement already satisfied: responses<0.19 in ./env/lib/python3.10/site-packages (from evaluate==0.3.0->autotrain-advanced) (0.18.0)
Requirement already satisfied: anyio<4.0.0,>=3.7.1 in ./env/lib/python3.10/site-packages (from fastapi==0.104.1->autotrain-advanced) (3.7.1)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in ./env/lib/python3.10/site-packages (from fastapi==0.104.1->autotrain-advanced) (0.27.0)
Requirement already satisfied: typing-extensions>=4.8.0 in ./env/lib/python3.10/site-packages (from fastapi==0.104.1->autotrain-advanced) (4.9.0)
Requirement already satisfied: aiofiles<24.0,>=22.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (23.2.1)
Requirement already satisfied: altair<6.0,>=4.2.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (5.3.0)
Requirement already satisfied: ffmpy in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (0.3.2)
Requirement already satisfied: gradio-client==0.5.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (0.5.0)
Requirement already satisfied: httpx in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (0.27.0)
Requirement already satisfied: importlib-resources<7.0,>=1.3 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (6.4.0)
Requirement already satisfied: jinja2<4.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (3.1.3)
Requirement already satisfied: markupsafe
Requirement already satisfied: matplotlib~=3.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (3.8.3)
Requirement already satisfied: orjson~=3.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (3.10.0)
Requirement already satisfied: pydub in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (0.25.1)
Requirement already satisfied: semantic-version~=2.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (2.10.0)
Requirement already satisfied: websockets<12.0,>=10.0 in ./env/lib/python3.10/site-packages (from gradio==3.41.0->autotrain-advanced) (11.0.3)
Requirement already satisfied: PyWavelets>=1.1.1 in ./env/lib/python3.10/site-packages (from invisible-watermark==0.2.0->autotrain-advanced) (1.6.0)
Requirement already satisfied: opencv-python>=4.1.0.25 in ./env/lib/python3.10/site-packages (from invisible-watermark==0.2.0->autotrain-advanced) (4.9.0.80)
Requirement already satisfied: rapidfuzz==2.13.7 in ./env/lib/python3.10/site-packages (from jiwer==3.0.2->autotrain-advanced) (2.13.7)
Requirement already satisfied: alembic>=1.5.0 in ./env/lib/python3.10/site-packages (from optuna==3.3.0->autotrain-advanced) (1.13.1)
Requirement already satisfied: cmaes>=0.10.0 in ./env/lib/python3.10/site-packages (from optuna==3.3.0->autotrain-advanced) (0.10.0)
Requirement already satisfied: colorlog in ./env/lib/python3.10/site-packages (from optuna==3.3.0->autotrain-advanced) (6.8.2)
Requirement already satisfied: sqlalchemy>=1.3.0 in ./env/lib/python3.10/site-packages (from optuna==3.3.0->autotrain-advanced) (2.0.29)
Requirement already satisfied: texttable in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (1.7.0)
Requirement already satisfied: pycryptodomex>=3.6.6 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (3.20.0)
Requirement already satisfied: pyzstd>=0.14.4 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (0.15.10)
Requirement already satisfied: pyppmd<1.1.0,>=0.18.1 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (1.0.0)
Requirement already satisfied: pybcj>=0.6.0 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (1.0.2)
Requirement already satisfied: multivolumefile>=0.2.3 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (0.2.3)
Requirement already satisfied: brotli>=1.0.9 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (1.0.9)
Requirement already satisfied: inflate64>=0.3.1 in ./env/lib/python3.10/site-packages (from py7zr==0.20.6->autotrain-advanced) (1.0.0)
Requirement already satisfied: annotated-types>=0.4.0 in ./env/lib/python3.10/site-packages (from pydantic==2.4.2->autotrain-advanced) (0.6.0)
Requirement already satisfied: pydantic-core==2.10.1 in ./env/lib/python3.10/site-packages (from pydantic==2.4.2->autotrain-advanced) (2.10.1)
Requirement already satisfied: charset-normalizer<4,>=2 in ./env/lib/python3.10/site-packages (from requests==2.31.0->autotrain-advanced) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in ./env/lib/python3.10/site-packages (from requests==2.31.0->autotrain-advanced) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./env/lib/python3.10/site-packages (from requests==2.31.0->autotrain-advanced) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.10/site-packages (from requests==2.31.0->autotrain-advanced) (2024.2.2)
Requirement already satisfied: absl-py in ./env/lib/python3.10/site-packages (from rouge-score==0.1.2->autotrain-advanced) (2.1.0)
Requirement already satisfied: six>=1.14.0 in ./env/lib/python3.10/site-packages (from rouge-score==0.1.2->autotrain-advanced) (1.16.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in ./env/lib/python3.10/site-packages (from scikit-learn==1.3.0->autotrain-advanced) (3.4.0)
Requirement already satisfied: tokenizers<0.19,>=0.14 in ./env/lib/python3.10/site-packages (from transformers==4.39.0->autotrain-advanced) (0.15.2)
Requirement already satisfied: tyro>=0.5.11 in ./env/lib/python3.10/site-packages (from trl==0.8.1->autotrain-advanced) (0.7.3)
Requirement already satisfied: h11>=0.8 in ./env/lib/python3.10/site-packages (from uvicorn==0.22.0->autotrain-advanced) (0.14.0)
Requirement already satisfied: pyarrow>=8.0.0 in ./env/lib/python3.10/site-packages (from datasets~=2.14.0->datasets[vision]=2.14.0->autotrain-advanced) (15.0.2)=2.14.0->datasets[vision]
Requirement already satisfied: pyarrow-hotfix in ./env/lib/python3.10/site-packages (from datasets=2.14.0->autotrain-advanced) (0.6)=2.14.0->datasets[vision]
Requirement already satisfied: aiohttp in ./env/lib/python3.10/site-packages (from datasets=2.14.0->autotrain-advanced) (3.9.3)=2.14.0->datasets[vision]
Requirement already satisfied: python-dateutil>=2.8.2 in ./env/lib/python3.10/site-packages (from pandas>=1.4->autotrain-advanced) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in ./env/lib/python3.10/site-packages (from pandas>=1.4->autotrain-advanced) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in ./env/lib/python3.10/site-packages (from pandas>=1.4->autotrain-advanced) (2024.1)
Requirement already satisfied: grpcio>=1.48.2 in ./env/lib/python3.10/site-packages (from tensorboard->autotrain-advanced) (1.62.1)
Requirement already satisfied: markdown>=2.6.8 in ./env/lib/python3.10/site-packages (from tensorboard->autotrain-advanced) (3.6)
Requirement already satisfied: setuptools>=41.0.0 in ./env/lib/python3.10/site-packages (from tensorboard->autotrain-advanced) (68.2.2)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in ./env/lib/python3.10/site-packages (from tensorboard->autotrain-advanced) (0.7.2)
Requirement already satisfied: Mako in ./env/lib/python3.10/site-packages (from alembic>=1.5.0->optuna==3.3.0->autotrain-advanced) (1.3.2)
Requirement already satisfied: jsonschema>=3.0 in ./env/lib/python3.10/site-packages (from altair<6.0,>=4.2.0->gradio==3.41.0->autotrain-advanced) (4.21.1)
Requirement already satisfied: toolz in ./env/lib/python3.10/site-packages (from altair<6.0,>=4.2.0->gradio==3.41.0->autotrain-advanced) (0.12.1)
Requirement already satisfied: sniffio>=1.1 in ./env/lib/python3.10/site-packages (from anyio<4.0.0,>=3.7.1->fastapi==0.104.1->autotrain-advanced) (1.3.1)
Requirement already satisfied: exceptiongroup in ./env/lib/python3.10/site-packages (from anyio<4.0.0,>=3.7.1->fastapi==0.104.1->autotrain-advanced) (1.2.0)
Requirement already satisfied: aiosignal>=1.1.2 in ./env/lib/python3.10/site-packages (from aiohttp->datasets=2.14.0->autotrain-advanced) (1.3.1)=2.14.0->datasets[vision]
Requirement already satisfied: attrs>=17.3.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets=2.14.0->autotrain-advanced) (23.2.0)=2.14.0->datasets[vision]
Requirement already satisfied: frozenlist>=1.1.1 in ./env/lib/python3.10/site-packages (from aiohttp->datasets=2.14.0->autotrain-advanced) (1.4.1)=2.14.0->datasets[vision]
Requirement already satisfied: multidict<7.0,>=4.5 in ./env/lib/python3.10/site-packages (from aiohttp->datasets=2.14.0->autotrain-advanced) (6.0.5)=2.14.0->datasets[vision]
Requirement already satisfied: yarl<2.0,>=1.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets=2.14.0->autotrain-advanced) (1.9.4)=2.14.0->datasets[vision]
Requirement already satisfied: async-timeout<5.0,>=4.0 in ./env/lib/python3.10/site-packages (from aiohttp->datasets=2.14.0->autotrain-advanced) (4.0.3)=3.0->gradio==3.41.0->autotrain-advanced) (1.2.0)
Requirement already satisfied: contourpy>=1.0.1 in ./env/lib/python3.10/site-packages (from matplotlib
Requirement already satisfied: cycler>=0.10 in ./env/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.41.0->autotrain-advanced) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in ./env/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.41.0->autotrain-advanced) (4.50.0)
Requirement already satisfied: kiwisolver>=1.3.1 in ./env/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.41.0->autotrain-advanced) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in ./env/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.41.0->autotrain-advanced) (3.1.2)
Requirement already satisfied: networkx>=2.8 in ./env/lib/python3.10/site-packages (from scikit-image>=0.16.1->albumentations==1.3.1->autotrain-advanced) (3.1)
Requirement already satisfied: imageio>=2.27 in ./env/lib/python3.10/site-packages (from scikit-image>=0.16.1->albumentations==1.3.1->autotrain-advanced) (2.34.0)
Requirement already satisfied: tifffile>=2022.8.12 in ./env/lib/python3.10/site-packages (from scikit-image>=0.16.1->albumentations==1.3.1->autotrain-advanced) (2024.2.12)
Requirement already satisfied: lazy_loader>=0.3 in ./env/lib/python3.10/site-packages (from scikit-image>=0.16.1->albumentations==1.3.1->autotrain-advanced) (0.3)
Requirement already satisfied: greenlet!=0.4.17 in ./env/lib/python3.10/site-packages (from sqlalchemy>=1.3.0->optuna==3.3.0->autotrain-advanced) (3.0.3)
Requirement already satisfied: sympy in ./env/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.28.0->autotrain-advanced) (1.12)
Requirement already satisfied: docstring-parser>=0.14.1 in ./env/lib/python3.10/site-packages (from tyro>=0.5.11->trl==0.8.1->autotrain-advanced) (0.16)
Requirement already satisfied: rich>=11.1.0 in ./env/lib/python3.10/site-packages (from tyro>=0.5.11->trl==0.8.1->autotrain-advanced) (13.7.1)
Requirement already satisfied: shtab>=1.5.6 in ./env/lib/python3.10/site-packages (from tyro>=0.5.11->trl==0.8.1->autotrain-advanced) (1.7.1)
Requirement already satisfied: types-python-dateutil>=2.8.10 in ./env/lib/python3.10/site-packages (from arrow->codecarbon==2.2.3->autotrain-advanced) (2.9.0.20240316)
Requirement already satisfied: cffi>=1.12 in ./env/lib/python3.10/site-packages (from cryptography->authlib==1.3.0->autotrain-advanced) (1.16.0)
Requirement already satisfied: httpcore==1.* in ./env/lib/python3.10/site-packages (from httpx->gradio==3.41.0->autotrain-advanced) (1.0.5)
Requirement already satisfied: zipp>=0.5 in ./env/lib/python3.10/site-packages (from importlib-metadata->diffusers==0.27.2->autotrain-advanced) (3.18.1)
Requirement already satisfied: pycparser in ./env/lib/python3.10/site-packages (from cffi>=1.12->cryptography->authlib==1.3.0->autotrain-advanced) (2.22)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in ./env/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.41.0->autotrain-advanced) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in ./env/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.41.0->autotrain-advanced) (0.34.0)
Requirement already satisfied: rpds-py>=0.7.1 in ./env/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.41.0->autotrain-advanced) (0.18.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in ./env/lib/python3.10/site-packages (from rich>=11.1.0->tyro>=0.5.11->trl==0.8.1->autotrain-advanced) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in ./env/lib/python3.10/site-packages (from rich>=11.1.0->tyro>=0.5.11->trl==0.8.1->autotrain-advanced) (2.17.2)
Requirement already satisfied: mpmath>=0.19 in ./env/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate==0.28.0->autotrain-advanced) (1.3.0)
Requirement already satisfied: mdurl~=0.1 in ./env/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=11.1.0->tyro>=0.5.11->trl==0.8.1->autotrain-advanced) (0.1.2)
Downloading autotrain_advanced-0.7.42-py3-none-any.whl (250 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 250.2/250.2 kB 12.6 MB/s eta 0:00:00
Installing collected packages: autotrain-advanced
Successfully installed autotrain-advanced-0.7.42
INFO: Will watch for changes in these directories: ['/app']
WARNING: "workers" flag is ignored when reloading is enabled.
INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
INFO: Started reloader process [35] using StatReload
WARNING Parameters not supplied by user and set to default: valid_split, prompt_text_column, optimizer, use_flash_attention_2, lora_dropout, rejected_text_column, evaluation_strategy, lora_r, repo_id, username, logging_steps, auto_find_batch_size, model, weight_decay, scheduler, lr, text_column, model_max_length, merge_adapter, max_grad_norm, dpo_beta, train_split, batch_size, save_total_limit, data_path, disable_gradient_checkpointing, seed, add_eos_token, token, push_to_hub, trainer, lora_alpha, model_ref, project_name, warmup_ratio, gradient_accumulation
WARNING Parameters not supplied by user and set to default: batch_size, save_total_limit, valid_split, optimizer, data_path, save_strategy, seed, evaluation_strategy, repo_id, username, logging_steps, auto_find_batch_size, model, token, push_to_hub, weight_decay, max_seq_length, scheduler, epochs, lr, text_column, log, project_name, gradient_accumulation, max_grad_norm, warmup_ratio, target_column, train_split
WARNING Parameters not supplied by user and set to default: batch_size, save_total_limit, valid_split, optimizer, data_path, save_strategy, seed, image_column, evaluation_strategy, repo_id, username, logging_steps, auto_find_batch_size, model, token, push_to_hub, weight_decay, scheduler, epochs, lr, log, project_name, gradient_accumulation, max_grad_norm, warmup_ratio, target_column, train_split
WARNING Parameters not supplied by user and set to default: valid_split, optimizer, max_target_length, lora_dropout, evaluation_strategy, lora_r, repo_id, username, logging_steps, auto_find_batch_size, model, weight_decay, max_seq_length, scheduler, lr, epochs, text_column, quantization, max_grad_norm, peft, train_split, batch_size, save_total_limit, data_path, seed, token, push_to_hub, lora_alpha, project_name, warmup_ratio, gradient_accumulation, target_column
WARNING Parameters not supplied by user and set to default: valid_split, data_path, target_columns, seed, repo_id, task, username, numerical_columns, categorical_columns, model, token, push_to_hub, num_trials, project_name, time_limit, id_column, train_split
WARNING Parameters not supplied by user and set to default: center_crop, validation_epochs, resume_from_checkpoint, prior_generation_precision, class_image_path, validation_images, scale_lr, repo_id, username, allow_tf32, model, dataloader_num_workers, scheduler, epochs, validation_prompt, adam_beta2, local_rank, max_grad_norm, tokenizer, class_labels_conditioning, xl, pre_compute_text_embeddings, image_path, seed, num_cycles, class_prompt, adam_weight_decay, tokenizer_max_length, adam_epsilon, token, prior_loss_weight, push_to_hub, adam_beta1, warmup_steps, prior_preservation, checkpoints_total_limit, num_class_images, revision, rank, sample_batch_size, num_validation_images, project_name, lr_power, text_encoder_use_attention_mask, logging, checkpointing_steps
WARNING Parameters not supplied by user and set to default: batch_size, save_total_limit, valid_split, optimizer, data_path, save_strategy, seed, evaluation_strategy, repo_id, username, logging_steps, auto_find_batch_size, model, tags_column, token, push_to_hub, weight_decay, max_seq_length, scheduler, epochs, lr, tokens_column, log, project_name, gradient_accumulation, max_grad_norm, warmup_ratio, train_split
/app/env/lib/python3.10/site-packages/huggingface_hub/utils/endpoint_helpers.py:247: FutureWarning: 'ModelFilter' is deprecated and will be removed in huggingface_hub>=0.24. Please pass the filter parameters as keyword arguments directly to thelist_modelsmethod.
warnings.warn(
INFO: Started server process [44]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 10.16.7.189:63274 - "GET /?custom_models=mistralai%2FMixtral-8x7B-v0.1&logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2M2QyZmQ0ZmI3MzRlYWE0ZDRmODM5MjgiLCJ1c2VyIjoiamVmZmVyeWxvdmVseSJ9LCJpYXQiOjE3MTIwNzM4NTYsInN1YiI6Ii9zcGFjZXMvQWlNYXZlbkFpL252aWRpYS1kZ3gtdHJhaW4iLCJleHAiOjE3MTIxNjAyNTYsImlzcyI6Imh0dHBzOi8vaHVnZ2luZ2ZhY2UuY28ifQ.KDM5fBC1boOpfLfQ2RzSJyncooQFUsPs1YduTNCKUpM1lQaakpv-GczqV8sT0JpRWNbFchKFglpTw52xAwOhBw HTTP/1.1" 200 OK
INFO: 10.16.8.237:2318 - "GET /logo.png HTTP/1.1" 200 OK
INFO: 10.16.7.189:7015 - "GET /login/huggingface HTTP/1.1" 302 Found
INFO: 10.16.41.118:51618 - "GET /auth?code=gmmkxCUYxOibDbew&state=4aCkwMa9YI14sTLPVFNEmohTttsWEu HTTP/1.1" 307 Temporary Redirect
INFO: 10.16.41.118:35077 - "GET /?__sign=eyJhbGciOiJFZERTQSJ9.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sIm9uQmVoYWxmT2YiOnsia2luZCI6InVzZXIiLCJfaWQiOiI2M2QyZmQ0ZmI3MzRlYWE0ZDRmODM5MjgiLCJ1c2VyIjoiamVmZmVyeWxvdmVseSJ9LCJpYXQiOjE3MTIwNzM4NjgsInN1YiI6Ii9zcGFjZXMvQWlNYXZlbkFpL252aWRpYS1kZ3gtdHJhaW4iLCJleHAiOjE3MTIxNjAyNjgsImlzcyI6Imh0dHBzOi8vaHVnZ2luZ2ZhY2UuY28ifQ.2tN4iQU3DvmG4S0Scz_KOzRy4MyF5znMHL3puv4HzvGD0cf4QJxE8SQa19Fv5B7DR2lOw6f8ngeLIy6LC4O1CQ HTTP/1.1" 200 OK
INFO Task: llm:sft
INFO: 10.16.7.189:2099 - "GET /params/llm%3Asft/basic HTTP/1.1" 200 OK
INFO: 10.16.25.191:32337 - "GET /model_choices/llm%3Asft HTTP/1.1" 200 OK
INFO Task: llm:sft
INFO: 10.16.8.237:38096 - "GET /params/llm%3Asft/full HTTP/1.1" 200 OK
INFO hardware: NVCF 8xH100
WARNING Parameters not supplied by user and set to default: add_eos_token, dpo_beta, valid_split, model_ref
INFO Starting NVCF training
INFO job_name: AiMavenAi-autotrain-16b1t-sxo3s
INFO backend: nvcf-h100x8
INFO AiMavenAi-autotrain-16b1t-sxo3s: POST - Configuring NVCF job_submit.
INFO AiMavenAi-autotrain-16b1t-sxo3s: POST - Successfully submitted NVCF job. Polling reqId for completion
INFO AiMavenAi-autotrain-16b1t-sxo3s: nvcfRequestId: 6e9b0606-93fe-4947-acf2-5bf62324b65a
INFO AiMavenAi-autotrain-16b1t-sxo3s: Polling : https://huggingface.co/api/dgx/v1/status/6e9b0606-93fe-4947-acf2-5bf62324b65a
INFO: 10.16.8.237:12638 - "POST /create_project HTTP/1.1" 200 OK
INFO AUTOTRAIN_USERNAME: AiMavenAi
INFO PROJECT_NAME: autotrain-16b1t-sxo3s
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO TASK_ID: 9
INFO DATA_PATH: jefferylovely/autotrain-data-MavenMixtralbase
INFO MODEL: mistralai/Mixtral-8x7B-v0.1
INFO OUTPUT_MODEL_REPO: AiMavenAi/autotrain-16b1t-sxo3s
INFO: Started server process [23476]
INFO: Waiting for application startup.
INFO {"model":"mistralai/Mixtral-8x7B-v0.1","project_name":"autotrain-16b1t-sxo3s","data_path":"jefferylovely/autotrain-data-MavenMixtralbase","train_split":"train","valid_split":null,"add_eos_token":true,"block_size":512,"model_max_length":2048,"padding":"right","trainer":"sft","use_flash_attention_2":false,"log":"tensorboard","disable_gradient_checkpointing":false,"logging_steps":-1,"evaluation_strategy":"epoch","save_total_limit":1,"save_strategy":"no","auto_find_batch_size":false,"mixed_precision":"fp16","lr":0.00003,"epochs":3,"batch_size":2,"warmup_ratio":0.1,"gradient_accumulation":4,"optimizer":"adamw_torch","scheduler":"linear","weight_decay":0.0,"max_grad_norm":1.0,"seed":42,"chat_template":"none","quantization":"int4","target_modules":"all-linear","merge_adapter":true,"peft":true,"lora_r":128,"lora_alpha":256,"lora_dropout":0.05,"model_ref":null,"dpo_beta":0.1,"prompt_text_column":"prompt","text_column":"text","rejected_text_column":"rejected_text","push_to_hub":true,"repo_id":"AiMavenAi/autotrain-16b1t-sxo3s","username":"AiMavenAi","token":"hf_oauth_eyJhbGciOiJFZERTQSJ9.eyJzY29wZSI6WyJvcGVuaWQiLCJwcm9maWxlIiwicmVhZC1yZXBvcyIsIndyaXRlLXJlcG9zIiwibWFuYWdlLXJlcG9zIiwiaW5mZXJlbmNlLWFwaSIsInJlYWQtYmlsbGluZyJdLCJhdWQiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIiwib2F1dGhBcHAiOiJhY2IwZmI3NC01NGFmLTRjZmQtYmQ4NC1mNDdkNWZmMzdkYTAiLCJzZXNzaW9uSWQiOiI2NTk0YWI3MTA3NTI0NWVhZGQ2MWIwNjIiLCJpYXQiOjE3MTIwNzM4NjcsInN1YiI6IjYzZDJmZDRmYjczNGVhYTRkNGY4MzkyOCIsImV4cCI6MTcxMjEwMjY2NywiaXNzIjoiaHR0cHM6Ly9odWdnaW5nZmFjZS5jbyJ9.GH4WO05V5ri1rnM5VrAYjKrgUtCoQcstLwimZn80eXXJ6GSxuSSOm3_1Xe456LajrHsFSvk5hiBWyD8nUm0HAg"}
INFO ['accelerate', 'launch', '--multi_gpu', '--num_machines', '1', '--num_processes', '8', '--mixed_precision', 'fp16', '-m', 'autotrain.trainers.clm', '--training_config', '/tmp/model/training_params.json']
INFO Started training with PID 23478
INFO Running jobs: [23478]
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
The following values were not passed toaccelerate launchand had defaults used instead:
--dynamo_backendwas set to a value of'no'
To avoid this warning pass in values for each of the problematic parameters or runaccelerate config.
[2024-04-02 16:06:14,465] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:14,729] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:15,148] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:15,377] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:15,423] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:15,466] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:15,496] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:06:15,574] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Downloading readme: 0%| | 0.00/581 [00:00<?, ?B/s]
Downloading readme: 100%|██████████| 581/581 [00:00<00:00, 4.85MB/s]
Downloading data files: 0%| | 0/2 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/277k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 393kB/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 393kB/s]
Downloading data files: 50%|█████ | 1/2 [00:00<00:00, 1.39it/s]
Downloading data: 0%| | 0.00/277k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 339kB/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 339kB/s]
Downloading data files: 100%|██████████| 2/2 [00:01<00:00, 1.29it/s]
Downloading data files: 100%|██████████| 2/2 [00:01<00:00, 1.30it/s]
Extracting data files: 0%| | 0/2 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 2/2 [00:00<00:00, 3383.87it/s]
Generating train split: 0%| | 0/924 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 924/924 [00:00<00:00, 250632.92 examples/s]
Generating validation split: 0%| | 0/924 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 924/924 [00:00<00:00, 465362.26 examples/s]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
🚀 INFO | 2024-04-02 16:06:31 | main:process_input_data:108 - Train data: Dataset({
features: ['autotrain_text'],
num_rows: 924
})
🚀 INFO | 2024-04-02 16:06:32 | main:process_input_data:109 - Valid data: None
🚀 INFO | 2024-04-02 16:06:35 | main🚋197 - creating training arguments...
🚀 INFO | 2024-04-02 16:06:35 | main🚋211 - Logging steps: 25
🚀 INFO | 2024-04-02 16:06:36 | main🚋252 - loading model config...
🚀 INFO | 2024-04-02 16:06:36 | main🚋264 - loading model...
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 5%|▌ | 1/19 [00:05<01:39, 5.50s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:38, 5.49s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:38, 5.50s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:38, 5.50s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:39, 5.50s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:38, 5.50s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:39, 5.51s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:39, 5.51s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:42, 6.01s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:42, 6.00s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:41, 6.00s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:41, 6.00s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:42, 6.00s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:42, 6.03s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:42, 6.04s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:42, 6.05s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.16s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.17s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.16s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.16s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.17s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.17s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.18s/it]
Downloading shards: 16%|█▌ | 3/19 [00:18<01:38, 6.18s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.70s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.70s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.70s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.70s/it]
Downloading shards: 21%|██ | 4/19 [00:23<01:25, 5.71s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.53s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.54s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.54s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.53s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.54s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.54s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.55s/it]
Downloading shards: 26%|██▋ | 5/19 [00:28<01:17, 5.55s/it]> INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.80s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.80s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.80s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.81s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.81s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.81s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.81s/it]
Downloading shards: 32%|███▏ | 6/19 [00:34<01:15, 5.82s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.38s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.38s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.39s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.39s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.39s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.39s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.39s/it]
Downloading shards: 37%|███▋ | 7/19 [00:42<01:16, 6.40s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.04s/it]
Downloading shards: 42%|████▏ | 8/19 [00:47<01:06, 6.05s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.75s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.76s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.75s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.76s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.76s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.76s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.76s/it]
Downloading shards: 47%|████▋ | 9/19 [00:52<00:57, 5.76s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.54s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.54s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.54s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.55s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.55s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.55s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.56s/it]
Downloading shards: 53%|█████▎ | 10/19 [00:57<00:49, 5.55s/it]> INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.54s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.54s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.54s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.54s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.55s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.55s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.55s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:03<00:44, 5.55s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.74s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.74s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:09<00:40, 5.74s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.90s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.90s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.90s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.90s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.90s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.90s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.91s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:15<00:35, 5.91s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.38s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.38s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.39s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.39s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.39s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.39s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.39s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:23<00:31, 6.39s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.02s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.01s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.01s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.02s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.02s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.02s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.03s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:28<00:24, 6.03s/it]> INFO Running jobs: [23478]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.33s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.33s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.34s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.34s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.34s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.34s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.35s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:35<00:19, 6.34s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.96s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:40<00:11, 5.95s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:45<00:05, 5.75s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:45<00:05, 5.75s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:45<00:05, 5.75s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:45<00:05, 5.75s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:46<00:05, 5.76s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:45<00:05, 5.75s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:45<00:05, 5.75s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:46<00:05, 5.76s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.39s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.38s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.39s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.39s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.38s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.40s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.40s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.39s/it]
Downloading shards: 100%|██████████| 19/19 [01:50<00:00, 5.82s/it]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:15, 4.22s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:21, 4.54s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:21, 4.51s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:22, 4.57s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:22, 4.57s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:22, 4.56s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:21, 4.50s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:23, 4.62s/it]> INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:09, 4.09s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:13, 4.30s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:14, 4.38s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:14, 4.36s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:14, 4.40s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:15, 4.41s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:15, 4.43s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:14, 4.40s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:12<01:04, 4.05s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:12<01:07, 4.19s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:08, 4.30s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:12<01:08, 4.30s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:09, 4.33s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:09, 4.33s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:09, 4.35s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:09, 4.37s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:16<00:59, 3.98s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:16<01:01, 4.10s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:03, 4.20s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:02, 4.19s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:03, 4.21s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:03, 4.21s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:03, 4.22s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:03, 4.26s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:20<00:55, 3.97s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:20<00:57, 4.11s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.24s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.24s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.25s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.26s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.26s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<01:00, 4.31s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:24<00:51, 4.00s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:24<00:53, 4.09s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:55, 4.24s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:55, 4.26s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:55, 4.27s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:55, 4.27s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:55, 4.28s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:26<00:56, 4.32s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:27<00:47, 3.96s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:28<00:48, 4.05s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:50, 4.17s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:50, 4.19s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:50, 4.20s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:50, 4.22s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:50, 4.25s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:30<00:50, 4.23s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:31<00:43, 3.95s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:32<00:44, 4.02s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:45, 4.15s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:45, 4.16s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:45, 4.17s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:45, 4.16s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:46, 4.20s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:46, 4.19s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:35<00:39, 3.95s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:36<00:40, 4.02s/it]> INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 47%|████▋ | 9/19 [00:37<00:41, 4.12s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:37<00:41, 4.13s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:41, 4.15s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:41, 4.15s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:41, 4.17s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:41, 4.16s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:39<00:35, 3.92s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:40<00:36, 4.01s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:41<00:36, 4.08s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:41<00:36, 4.08s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:36, 4.11s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.12s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.13s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.12s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:43<00:31, 3.93s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:44<00:32, 4.04s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:45<00:32, 4.07s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:45<00:32, 4.07s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:32, 4.09s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:32, 4.11s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:32, 4.11s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:32, 4.11s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:47<00:27, 3.93s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:48<00:28, 4.03s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:49<00:28, 4.06s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:49<00:28, 4.06s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:28, 4.08s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:28, 4.10s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:28, 4.10s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:28, 4.10s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:51<00:23, 3.94s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:52<00:24, 4.04s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.08s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.06s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.09s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.10s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.10s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.12s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:55<00:19, 3.91s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:56<00:20, 4.01s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.10s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.14s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.06s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.07s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.07s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.08s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [00:59<00:15, 3.91s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:00<00:16, 4.01s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.14s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.09s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.13s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.07s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.10s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.10s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:03<00:11, 3.90s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:04<00:12, 4.02s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.17s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.13s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.16s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.10s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.12s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.12s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:07<00:07, 3.89s/it]> INFO Running jobs: [23478]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:08<00:07, 3.99s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:10<00:08, 4.18s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:10<00:08, 4.17s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:10<00:08, 4.12s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:10<00:08, 4.15s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:10<00:08, 4.14s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:10<00:08, 4.09s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:10<00:03, 3.90s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:12<00:03, 3.99s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:14<00:00, 3.72s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:14<00:00, 3.91s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.17s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.16s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:14<00:04, 4.13s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:14<00:04, 4.15s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.09s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:16<00:00, 3.81s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:16<00:00, 4.01s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.99s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.98s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.99s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.98s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.95s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.99s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
🚀 INFO | 2024-04-02 16:09:49 | main🚋332 - model dtype: torch.float16
🚀 INFO | 2024-04-02 16:09:49 | main🚋340 - preparing peft model...
🚀 INFO | 2024-04-02 16:10:01 | main🚋391 - Using block size 512
🚀 INFO | 2024-04-02 16:10:01 | main🚋453 - creating trainer
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 0 examples [00:00, ? examples/s]
❌ ERROR | 2024-04-02 16:10:06 | autotrain.trainers.common:wrapper:115 - train has failed due to an exception: Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1676, in _prepare_split_single
for key, record in generator:
File "/app/env/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 30, in _generate_examples
for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 517, in data_generator
yield from constant_length_iterator
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 457, in iter
buffer.append(self.formatting_func(next(iterator)))
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 434, in
self.formatting_func = lambda x: x[dataset_text_field]
KeyError: 'text'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 520, in _prepare_packed_dataloader
packed_dataset = Dataset.from_generator(
File "/app/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1072, in from_generator
).read()
File "/app/env/lib/python3.10/site-packages/datasets/io/generator.py", line 47, in read
self.builder.download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
super()._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1555, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1712, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/src/autotrain/trainers/common.py", line 112, in wrapper
return func(*args, **kwargs)
File "/app/src/autotrain/trainers/clm/main.py", line 480, in train
trainer = SFTTrainer(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 282, in init
train_dataset = self._prepare_dataset(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 419, in _prepare_dataset
return self._prepare_packed_dataloader(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 524, in _prepare_packed_dataloader
raise ValueError(
ValueError: Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
❌ ERROR | 2024-04-02 16:10:06 | autotrain.trainers.common:wrapper:116 - Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 202 - Polling reqId for completion
INFO Running jobs: [23478]
[rank7]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank6]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank5]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank1]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank2]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank3]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank4]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
INFO AiMavenAi-autotrain-16b1t-sxo3s: GET - 200 - Polling completed
INFO Running jobs: [23478]
INFO Killing PID: 23478
INFO No running jobs found. Shutting down the server.
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [23476]
INFO hardware: NVCF 8xH100
INFO: 10.16.25.191:23209 - "POST /create_project HTTP/1.1" 409 Conflict
INFO hardware: NVCF 8xH100
WARNING Parameters not supplied by user and set to default: add_eos_token, dpo_beta, valid_split, model_ref
INFO Starting NVCF training
INFO job_name: AiMavenAi-Mixtral111
INFO backend: nvcf-h100x8
INFO AiMavenAi-Mixtral111: POST - Configuring NVCF job_submit.
INFO AiMavenAi-Mixtral111: POST - Successfully submitted NVCF job. Polling reqId for completion
INFO AiMavenAi-Mixtral111: nvcfRequestId: 9887e07b-6212-4491-bb75-66fb14088566
INFO AiMavenAi-Mixtral111: Polling : https://huggingface.co/api/dgx/v1/status/9887e07b-6212-4491-bb75-66fb14088566
INFO: 10.16.25.191:5909 - "POST /create_project HTTP/1.1" 200 OK
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO AUTOTRAIN_USERNAME: AiMavenAi
INFO PROJECT_NAME: Mixtral111
INFO TASK_ID: 9
INFO DATA_PATH: jefferylovely/autotrain-data-MavenMixtralbase
INFO MODEL: mistralai/Mixtral-8x7B-v0.1
INFO OUTPUT_MODEL_REPO: AiMavenAi/Mixtral111
INFO: Started server process [30136]
INFO: Waiting for application startup.
INFO {"model":"mistralai/Mixtral-8x7B-v0.1","project_name":"Mixtral111","data_path":"jefferylovely/autotrain-data-MavenMixtralbase","train_split":"train","valid_split":null,"add_eos_token":true,"block_size":32,"model_max_length":2048,"padding":"right","trainer":"sft","use_flash_attention_2":false,"log":"tensorboard","disable_gradient_checkpointing":false,"logging_steps":-1,"evaluation_strategy":"epoch","save_total_limit":1,"save_strategy":"no","auto_find_batch_size":false,"mixed_precision":"fp16","lr":0.00003,"epochs":3,"batch_size":2,"warmup_ratio":0.1,"gradient_accumulation":4,"optimizer":"adamw_torch","scheduler":"linear","weight_decay":0.0,"max_grad_norm":1.0,"seed":42,"chat_template":"none","quantization":"int4","target_modules":"all-linear","merge_adapter":true,"peft":true,"lora_r":128,"lora_alpha":256,"lora_dropout":0.05,"model_ref":null,"dpo_beta":0.1,"prompt_text_column":"prompt","text_column":"text","rejected_text_column":"rejected_text","push_to_hub":true,"repo_id":"AiMavenAi/Mixtral111","username":"AiMavenAi","token":"hf_oauth_eyJhbGciOiJFZERTQSJ9.eyJzY29wZSI6WyJvcGVuaWQiLCJwcm9maWxlIiwicmVhZC1yZXBvcyIsIndyaXRlLXJlcG9zIiwibWFuYWdlLXJlcG9zIiwiaW5mZXJlbmNlLWFwaSIsInJlYWQtYmlsbGluZyJdLCJhdWQiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIiwib2F1dGhBcHAiOiJhY2IwZmI3NC01NGFmLTRjZmQtYmQ4NC1mNDdkNWZmMzdkYTAiLCJzZXNzaW9uSWQiOiI2NTk0YWI3MTA3NTI0NWVhZGQ2MWIwNjIiLCJpYXQiOjE3MTIwNzM4NjcsInN1YiI6IjYzZDJmZDRmYjczNGVhYTRkNGY4MzkyOCIsImV4cCI6MTcxMjEwMjY2NywiaXNzIjoiaHR0cHM6Ly9odWdnaW5nZmFjZS5jbyJ9.GH4WO05V5ri1rnM5VrAYjKrgUtCoQcstLwimZn80eXXJ6GSxuSSOm3_1Xe456LajrHsFSvk5hiBWyD8nUm0HAg"}
INFO ['accelerate', 'launch', '--multi_gpu', '--num_machines', '1', '--num_processes', '8', '--mixed_precision', 'fp16', '-m', 'autotrain.trainers.clm', '--training_config', '/tmp/model/training_params.json']
INFO Started training with PID 30138
INFO Running jobs: [30138]
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
The following values were not passed toaccelerate launchand had defaults used instead:
--dynamo_backendwas set to a value of'no'
To avoid this warning pass in values for each of the problematic parameters or runaccelerate config.
[2024-04-02 16:24:52,740] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,141] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,255] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,288] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,294] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,344] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,377] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:24:53,412] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Downloading readme: 0%| | 0.00/581 [00:00<?, ?B/s]
Downloading readme: 100%|██████████| 581/581 [00:00<00:00, 6.43MB/s]
Downloading data files: 0%| | 0/2 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/277k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 713kB/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 712kB/s]
Downloading data files: 50%|█████ | 1/2 [00:00<00:00, 2.55it/s]
Downloading data: 0%| | 0.00/277k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 940kB/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 939kB/s]
Downloading data files: 100%|██████████| 2/2 [00:00<00:00, 2.98it/s]
Downloading data files: 100%|██████████| 2/2 [00:00<00:00, 2.91it/s]
Extracting data files: 0%| | 0/2 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 2/2 [00:00<00:00, 2750.36it/s]
Generating train split: 0%| | 0/924 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 924/924 [00:00<00:00, 256539.15 examples/s]
Generating validation split: 0%| | 0/924 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 924/924 [00:00<00:00, 480716.56 examples/s]
🚀 INFO | 2024-04-02 16:25:03 | main:process_input_data:108 - Train data: Dataset({
features: ['autotrain_text'],
num_rows: 924
})
🚀 INFO | 2024-04-02 16:25:04 | main:process_input_data:109 - Valid data: None
🚀 INFO | 2024-04-02 16:25:07 | main🚋197 - creating training arguments...
🚀 INFO | 2024-04-02 16:25:07 | main🚋211 - Logging steps: 25
🚀 INFO | 2024-04-02 16:25:07 | main🚋252 - loading model config...
🚀 INFO | 2024-04-02 16:25:08 | main🚋264 - loading model...
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]> INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Downloading shards: 5%|▌ | 1/19 [00:18<05:24, 18.02s/it]
Downloading shards: 5%|▌ | 1/19 [00:17<05:23, 17.98s/it]
Downloading shards: 5%|▌ | 1/19 [00:17<05:23, 17.99s/it]
Downloading shards: 5%|▌ | 1/19 [00:18<05:24, 18.02s/it]
Downloading shards: 5%|▌ | 1/19 [00:17<05:23, 17.99s/it]
Downloading shards: 5%|▌ | 1/19 [00:18<05:24, 18.02s/it]
Downloading shards: 5%|▌ | 1/19 [00:18<05:24, 18.02s/it]
Downloading shards: 5%|▌ | 1/19 [00:18<05:24, 18.04s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:52, 10.17s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:52, 10.15s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:52, 10.16s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:52, 10.17s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:52, 10.17s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:53, 10.18s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:53, 10.18s/it]
Downloading shards: 11%|█ | 2/19 [00:22<02:53, 10.18s/it]> INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.33s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.32s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.32s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.32s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.33s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.33s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.33s/it]
Downloading shards: 16%|█▌ | 3/19 [00:37<03:17, 12.34s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.64s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.64s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.64s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.64s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.64s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.64s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.66s/it]
Downloading shards: 21%|██ | 4/19 [00:45<02:39, 10.66s/it]
Downloading shards: 26%|██▋ | 5/19 [00:52<02:12, 9.45s/it]
Downloading shards: 26%|██▋ | 5/19 [00:53<02:12, 9.45s/it]
Downloading shards: 26%|██▋ | 5/19 [00:52<02:12, 9.45s/it]
Downloading shards: 26%|██▋ | 5/19 [00:53<02:12, 9.46s/it]
Downloading shards: 26%|██▋ | 5/19 [00:52<02:12, 9.45s/it]
Downloading shards: 26%|██▋ | 5/19 [00:53<02:12, 9.46s/it]
Downloading shards: 26%|██▋ | 5/19 [00:52<02:12, 9.46s/it]
Downloading shards: 26%|██▋ | 5/19 [00:53<02:12, 9.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:49, 8.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:49, 8.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:49, 8.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:49, 8.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:49, 8.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:50, 8.47s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:50, 8.46s/it]
Downloading shards: 32%|███▏ | 6/19 [00:59<01:50, 8.47s/it]> INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.64s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 37%|███▋ | 7/19 [01:08<01:43, 8.65s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.53s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.54s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.53s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.53s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.53s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.53s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.54s/it]
Downloading shards: 42%|████▏ | 8/19 [01:13<01:22, 7.54s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.74s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.74s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.74s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.75s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.74s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.75s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.75s/it]
Downloading shards: 47%|████▋ | 9/19 [01:18<01:07, 6.75s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.14s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.14s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.16s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.16s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:23<00:55, 6.16s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.07s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.06s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.07s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.07s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.07s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.08s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.07s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:29<00:48, 6.08s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.73s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.74s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:34<00:40, 5.74s/it]> INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.57s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.57s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.57s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.57s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.57s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.58s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.58s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:39<00:33, 5.58s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.45s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.45s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.45s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.45s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.45s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.46s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.46s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:44<00:27, 5.46s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.04s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.04s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.04s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.04s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.04s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.05s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.05s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:52<00:24, 6.06s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.80s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.80s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.80s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.80s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.81s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.81s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.81s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:57<00:17, 5.82s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.61s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.61s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.61s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.61s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.62s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.62s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.62s/it]
Downloading shards: 89%|████████▉ | 17/19 [02:02<00:11, 5.62s/it]> INFO Running jobs: [30138]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.59s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.58s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.59s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.59s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.59s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.60s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.60s/it]
Downloading shards: 95%|█████████▍| 18/19 [02:08<00:05, 5.60s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.25s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.25s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.25s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.25s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.26s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.26s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.26s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 5.26s/it]
Downloading shards: 100%|██████████| 19/19 [02:12<00:00, 6.98s/it]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:14, 4.12s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:28, 4.92s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:29, 4.97s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:29, 4.95s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:28, 4.92s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:28, 4.90s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:05<01:32, 5.15s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:05<01:32, 5.13s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:11, 4.21s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:26, 5.08s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:26, 5.08s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:26, 5.07s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:26, 5.08s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:26, 5.09s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:26, 5.06s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:10<01:25, 5.06s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:12<01:07, 4.24s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.72s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.70s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.72s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.72s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.72s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.73s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:14<01:15, 4.71s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:16<01:02, 4.19s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:08, 4.54s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:08, 4.54s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:08, 4.54s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:07, 4.53s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:08, 4.54s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:08, 4.56s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:18<01:08, 4.55s/it]> INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 26%|██▋ | 5/19 [00:20<00:58, 4.19s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.51s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.52s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.52s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.52s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.53s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.53s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:23<01:03, 4.52s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:55, 4.26s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:59, 4.54s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:58, 4.53s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:59, 4.55s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:59, 4.55s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:59, 4.55s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:59, 4.55s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:27<00:59, 4.55s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:51, 4.31s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.56s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.56s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.56s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.57s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.57s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.57s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:32<00:54, 4.57s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:46, 4.26s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:48, 4.43s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:48, 4.45s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:49, 4.45s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:49, 4.46s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:48, 4.45s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:49, 4.46s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:36<00:49, 4.47s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.24s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.41s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.41s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.42s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.42s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.42s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.42s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:40<00:44, 4.42s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.19s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.51s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.52s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.51s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.51s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.51s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.53s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:45<00:40, 4.52s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.18s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.57s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.57s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.57s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.57s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.58s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.57s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:50<00:36, 4.58s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.16s/it]> INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:24, 4.14s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.69s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.69s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.69s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.69s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.69s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.69s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:55<00:32, 4.70s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.11s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.72s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.72s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.73s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.73s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.73s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.74s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [01:00<00:28, 4.74s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.13s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.55s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.55s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.56s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.56s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.56s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.56s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [01:04<00:22, 4.57s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.14s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.71s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.71s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.71s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.71s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.71s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.72s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:09<00:18, 4.72s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.13s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.64s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.64s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.64s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.64s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.64s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.65s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:13<00:13, 4.65s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.14s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.68s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.69s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.69s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.69s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.69s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.69s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:18<00:09, 4.69s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.95s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
INFO Running jobs: [30138]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.72s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.72s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.72s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.73s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.73s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.74s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:23<00:04, 4.74s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.59s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.60s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.59s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.61s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.60s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.59s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.59s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.60s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:27<00:00, 4.62s/it]
🚀 INFO | 2024-04-02 16:28:52 | main🚋332 - model dtype: torch.float16
🚀 INFO | 2024-04-02 16:28:52 | main🚋340 - preparing peft model...
🚀 INFO | 2024-04-02 16:29:04 | main🚋391 - Using block size 32
🚀 INFO | 2024-04-02 16:29:04 | main🚋453 - creating trainer
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 0 examples [00:00, ? examples/s]
❌ ERROR | 2024-04-02 16:29:05 | autotrain.trainers.common:wrapper:115 - train has failed due to an exception: Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1676, in _prepare_split_single
for key, record in generator:
File "/app/env/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 30, in _generate_examples
for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 517, in data_generator
yield from constant_length_iterator
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 457, in iter
buffer.append(self.formatting_func(next(iterator)))
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 434, in
self.formatting_func = lambda x: x[dataset_text_field]
KeyError: 'text'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 520, in _prepare_packed_dataloader
packed_dataset = Dataset.from_generator(
File "/app/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1072, in from_generator
).read()
File "/app/env/lib/python3.10/site-packages/datasets/io/generator.py", line 47, in read
self.builder.download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
super()._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1555, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1712, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/src/autotrain/trainers/common.py", line 112, in wrapper
return func(*args, **kwargs)
File "/app/src/autotrain/trainers/clm/main.py", line 480, in train
trainer = SFTTrainer(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 282, in init
train_dataset = self._prepare_dataset(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 419, in _prepare_dataset
return self._prepare_packed_dataloader(
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 524, in _prepare_packed_dataloader
raise ValueError(
ValueError: Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
❌ ERROR | 2024-04-02 16:29:05 | autotrain.trainers.common:wrapper:116 - Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
INFO AiMavenAi-Mixtral111: GET - 202 - Polling reqId for completion
INFO Running jobs: [30138]
[rank3]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank7]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank5]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank4]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank2]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank6]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank1]:[W socket.cpp:432] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
INFO Running jobs: [30138]
INFO Killing PID: 30138
INFO No running jobs found. Shutting down the server.
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [30136]
INFO AiMavenAi-Mixtral111: GET - 200 - Polling completed
INFO hardware: NVCF 8xH100
INFO: 10.16.8.237:65120 - "POST /create_project HTTP/1.1" 409 Conflict
INFO hardware: NVCF 8xH100
WARNING Parameters not supplied by user and set to default: add_eos_token, dpo_beta, valid_split, model_ref
INFO Starting NVCF training
INFO job_name: AiMavenAi-Mixtral112
INFO backend: nvcf-h100x8
INFO AiMavenAi-Mixtral112: POST - Configuring NVCF job_submit.
INFO AiMavenAi-Mixtral112: POST - Successfully submitted NVCF job. Polling reqId for completion
INFO AiMavenAi-Mixtral112: nvcfRequestId: 18f278ca-ed14-4c41-b023-66b3a0704b7b
INFO AiMavenAi-Mixtral112: Polling : https://huggingface.co/api/dgx/v1/status/18f278ca-ed14-4c41-b023-66b3a0704b7b
INFO: 10.16.8.237:46152 - "POST /create_project HTTP/1.1" 200 OK
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
INFO AUTOTRAIN_USERNAME: AiMavenAi
INFO PROJECT_NAME: Mixtral112
INFO TASK_ID: 9
INFO DATA_PATH: jefferylovely/autotrain-data-MavenMixtralbase
INFO MODEL: mistralai/Mixtral-8x7B-v0.1
INFO OUTPUT_MODEL_REPO: AiMavenAi/Mixtral112
INFO: Started server process [60]
INFO: Waiting for application startup.
INFO {"model":"mistralai/Mixtral-8x7B-v0.1","project_name":"Mixtral112","data_path":"jefferylovely/autotrain-data-MavenMixtralbase","train_split":"train","valid_split":null,"add_eos_token":true,"block_size":16,"model_max_length":2048,"padding":"right","trainer":"sft","use_flash_attention_2":false,"log":"tensorboard","disable_gradient_checkpointing":false,"logging_steps":-1,"evaluation_strategy":"epoch","save_total_limit":1,"save_strategy":"no","auto_find_batch_size":false,"mixed_precision":"fp16","lr":0.00003,"epochs":3,"batch_size":2,"warmup_ratio":0.1,"gradient_accumulation":4,"optimizer":"adamw_torch","scheduler":"linear","weight_decay":0.0,"max_grad_norm":1.0,"seed":42,"chat_template":"none","quantization":"int4","target_modules":"all-linear","merge_adapter":true,"peft":true,"lora_r":128,"lora_alpha":256,"lora_dropout":0.05,"model_ref":null,"dpo_beta":0.1,"prompt_text_column":"prompt","text_column":"text","rejected_text_column":"rejected_text","push_to_hub":true,"repo_id":"AiMavenAi/Mixtral112","username":"AiMavenAi","token":"hf_oauth_eyJhbGciOiJFZERTQSJ9.eyJzY29wZSI6WyJvcGVuaWQiLCJwcm9maWxlIiwicmVhZC1yZXBvcyIsIndyaXRlLXJlcG9zIiwibWFuYWdlLXJlcG9zIiwiaW5mZXJlbmNlLWFwaSIsInJlYWQtYmlsbGluZyJdLCJhdWQiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIiwib2F1dGhBcHAiOiJhY2IwZmI3NC01NGFmLTRjZmQtYmQ4NC1mNDdkNWZmMzdkYTAiLCJzZXNzaW9uSWQiOiI2NTk0YWI3MTA3NTI0NWVhZGQ2MWIwNjIiLCJpYXQiOjE3MTIwNzM4NjcsInN1YiI6IjYzZDJmZDRmYjczNGVhYTRkNGY4MzkyOCIsImV4cCI6MTcxMjEwMjY2NywiaXNzIjoiaHR0cHM6Ly9odWdnaW5nZmFjZS5jbyJ9.GH4WO05V5ri1rnM5VrAYjKrgUtCoQcstLwimZn80eXXJ6GSxuSSOm3_1Xe456LajrHsFSvk5hiBWyD8nUm0HAg"}
INFO ['accelerate', 'launch', '--multi_gpu', '--num_machines', '1', '--num_processes', '8', '--mixed_precision', 'fp16', '-m', 'autotrain.trainers.clm', '--training_config', '/tmp/model/training_params.json']
INFO Started training with PID 62
INFO Running jobs: [62]
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
The following values were not passed toaccelerate launchand had defaults used instead:
--dynamo_backendwas set to a value of'no'
To avoid this warning pass in values for each of the problematic parameters or runaccelerate config.
[2024-04-02 16:51:14,345] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,208] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,208] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,241] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,300] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,329] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,331] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-04-02 16:51:15,331] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Downloading readme: 0%| | 0.00/581 [00:00<?, ?B/s]
Downloading readme: 100%|██████████| 581/581 [00:00<00:00, 7.41MB/s]
Downloading data files: 0%| | 0/2 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/277k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 436kB/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 435kB/s]
Downloading data files: 50%|█████ | 1/2 [00:00<00:00, 1.49it/s]
Downloading data: 0%| | 0.00/277k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 483kB/s]
Downloading data: 100%|██████████| 277k/277k [00:00<00:00, 483kB/s]
Downloading data files: 100%|██████████| 2/2 [00:01<00:00, 1.63it/s]
Downloading data files: 100%|██████████| 2/2 [00:01<00:00, 1.60it/s]
Extracting data files: 0%| | 0/2 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 2/2 [00:00<00:00, 3194.44it/s]
Generating train split: 0%| | 0/924 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 924/924 [00:00<00:00, 243254.89 examples/s]
Generating validation split: 0%| | 0/924 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 924/924 [00:00<00:00, 447418.25 examples/s]
🚀 INFO | 2024-04-02 16:51:25 | main:process_input_data:108 - Train data: Dataset({
features: ['autotrain_text'],
num_rows: 924
})
🚀 INFO | 2024-04-02 16:51:26 | main:process_input_data:109 - Valid data: None
🚀 INFO | 2024-04-02 16:51:30 | main🚋197 - creating training arguments...
🚀 INFO | 2024-04-02 16:51:30 | main🚋211 - Logging steps: 25
🚀 INFO | 2024-04-02 16:51:30 | main🚋252 - loading model config...
🚀 INFO | 2024-04-02 16:51:31 | main🚋264 - loading model...
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
low_cpu_mem_usagewas None, now set to True since model is quantized.
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]
Downloading shards: 0%| | 0/19 [00:00<?, ?it/s]> INFO Running jobs: [62]
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Downloading shards: 5%|▌ | 1/19 [00:05<01:35, 5.31s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:35, 5.29s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:35, 5.29s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:35, 5.31s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:36, 5.33s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:36, 5.34s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:35, 5.32s/it]
Downloading shards: 5%|▌ | 1/19 [00:05<01:36, 5.35s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.56s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.56s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.56s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.57s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.58s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.59s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.58s/it]
Downloading shards: 11%|█ | 2/19 [00:11<01:34, 5.58s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.42s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.41s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.42s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.43s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.42s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.43s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.42s/it]
Downloading shards: 16%|█▌ | 3/19 [00:16<01:26, 5.43s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.68s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.69s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.70s/it]
Downloading shards: 21%|██ | 4/19 [00:22<01:25, 5.70s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.60s/it]
Downloading shards: 26%|██▋ | 5/19 [00:27<01:18, 5.61s/it]> INFO Running jobs: [62]
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.33s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.33s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.33s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.33s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.33s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.33s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.34s/it]
Downloading shards: 32%|███▏ | 6/19 [00:35<01:22, 6.34s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.78s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.78s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.78s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.78s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.79s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.80s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.80s/it]
Downloading shards: 37%|███▋ | 7/19 [00:43<01:21, 6.80s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.39s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.38s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.39s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.39s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.39s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.39s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.40s/it]
Downloading shards: 42%|████▏ | 8/19 [00:48<01:10, 6.40s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.15s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.15s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.15s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.15s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.15s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.16s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.16s/it]
Downloading shards: 47%|████▋ | 9/19 [00:54<01:01, 6.16s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.14s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]
Downloading shards: 53%|█████▎ | 10/19 [01:00<00:55, 6.15s/it]> INFO Running jobs: [62]
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Downloading shards: 58%|█████▊ | 11/19 [01:06<00:49, 6.20s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:06<00:49, 6.21s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:06<00:49, 6.21s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:06<00:49, 6.22s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:07<00:49, 6.21s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:07<00:49, 6.21s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:06<00:49, 6.21s/it]
Downloading shards: 58%|█████▊ | 11/19 [01:07<00:49, 6.21s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.66s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.66s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.66s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.66s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.67s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.67s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.67s/it]
Downloading shards: 63%|██████▎ | 12/19 [01:14<00:46, 6.66s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.39s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.39s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.39s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.39s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.39s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.39s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.38s/it]
Downloading shards: 68%|██████▊ | 13/19 [01:20<00:38, 6.40s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.08s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.07s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.07s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.07s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.07s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.09s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.08s/it]
Downloading shards: 74%|███████▎ | 14/19 [01:25<00:30, 6.09s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.07s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.07s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.07s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.08s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.07s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.08s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.08s/it]
Downloading shards: 79%|███████▉ | 15/19 [01:31<00:24, 6.08s/it]> INFO Running jobs: [62]
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.75s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.75s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.75s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.75s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.75s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.75s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.76s/it]
Downloading shards: 84%|████████▍ | 16/19 [01:36<00:17, 5.76s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.51s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.51s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.52s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.52s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.52s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.52s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.52s/it]
Downloading shards: 89%|████████▉ | 17/19 [01:41<00:11, 5.53s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.11s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.12s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:55<00:08, 8.11s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.12s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.12s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.11s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.12s/it]
Downloading shards: 95%|█████████▍| 18/19 [01:56<00:08, 8.12s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.01s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 7.02s/it]
Downloading shards: 100%|██████████| 19/19 [02:00<00:00, 6.34s/it]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/19 [00:00<?, ?it/s]> INFO Running jobs: [62]
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:16, 4.23s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:27, 4.85s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:25, 4.75s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:27, 4.85s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:27, 4.88s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:28, 4.90s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:04<01:23, 4.63s/it]
Loading checkpoint shards: 5%|▌ | 1/19 [00:05<01:30, 5.00s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:13, 4.32s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:09<01:18, 4.59s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:08<01:15, 4.44s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:09<01:18, 4.64s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:09<01:18, 4.64s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:09<01:18, 4.61s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:09<01:19, 4.67s/it]
Loading checkpoint shards: 11%|█ | 2/19 [00:09<01:19, 4.65s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:12<01:08, 4.26s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:10, 4.39s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:08, 4.31s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:10, 4.41s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:10, 4.41s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:10, 4.42s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:11, 4.44s/it]
Loading checkpoint shards: 16%|█▌ | 3/19 [00:13<01:11, 4.45s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:16<01:03, 4.22s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:04, 4.28s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:03, 4.24s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:04, 4.31s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:04, 4.32s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:04, 4.33s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:04, 4.33s/it]
Loading checkpoint shards: 21%|██ | 4/19 [00:17<01:05, 4.35s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:58, 4.19s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.23s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:58, 4.20s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.23s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.24s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.24s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<00:59, 4.26s/it]
Loading checkpoint shards: 26%|██▋ | 5/19 [00:21<01:00, 4.29s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:54, 4.19s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:54, 4.18s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:54, 4.17s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:54, 4.20s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:54, 4.22s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:26<00:54, 4.23s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:25<00:54, 4.23s/it]
Loading checkpoint shards: 32%|███▏ | 6/19 [00:26<00:55, 4.26s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:49, 4.16s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:49, 4.14s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:49, 4.12s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:49, 4.16s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:49, 4.16s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:29<00:49, 4.16s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:30<00:50, 4.17s/it]
Loading checkpoint shards: 37%|███▋ | 7/19 [00:30<00:50, 4.21s/it]> INFO Running jobs: [62]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:45, 4.14s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:33<00:45, 4.14s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:45, 4.15s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:45, 4.13s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:45, 4.13s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:45, 4.14s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:45, 4.15s/it]
Loading checkpoint shards: 42%|████▏ | 8/19 [00:34<00:45, 4.18s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:37<00:42, 4.21s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.22s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.21s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.20s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.21s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:41, 4.20s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.21s/it]
Loading checkpoint shards: 47%|████▋ | 9/19 [00:38<00:42, 4.24s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:41<00:37, 4.18s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.18s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.17s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.17s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.16s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.15s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.14s/it]
Loading checkpoint shards: 53%|█████▎ | 10/19 [00:42<00:37, 4.21s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.17s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.20s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.20s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.16s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.19s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.16s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:46<00:33, 4.18s/it]
Loading checkpoint shards: 58%|█████▊ | 11/19 [00:47<00:33, 4.19s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.17s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.18s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.20s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.18s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.20s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.18s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:50<00:29, 4.20s/it]
Loading checkpoint shards: 63%|██████▎ | 12/19 [00:51<00:29, 4.19s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:25, 4.18s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:55<00:25, 4.18s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:54<00:25, 4.20s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:55<00:25, 4.19s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:55<00:25, 4.19s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:55<00:25, 4.18s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:55<00:25, 4.19s/it]
Loading checkpoint shards: 68%|██████▊ | 13/19 [00:55<00:25, 4.21s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.15s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:59<00:20, 4.13s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:58<00:20, 4.12s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:59<00:20, 4.16s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:59<00:20, 4.15s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:59<00:20, 4.16s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:59<00:20, 4.18s/it]
Loading checkpoint shards: 74%|███████▎ | 14/19 [00:59<00:20, 4.19s/it]> INFO Running jobs: [62]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.16s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:02<00:16, 4.12s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:03<00:16, 4.13s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:03<00:16, 4.14s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:03<00:16, 4.13s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:03<00:16, 4.16s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:03<00:16, 4.15s/it]
Loading checkpoint shards: 79%|███████▉ | 15/19 [01:03<00:16, 4.17s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:06<00:12, 4.16s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:07<00:12, 4.15s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:07<00:12, 4.16s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:07<00:12, 4.16s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:07<00:12, 4.16s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:07<00:12, 4.16s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:07<00:12, 4.17s/it]
Loading checkpoint shards: 84%|████████▍ | 16/19 [01:08<00:12, 4.20s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.18s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.12s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.11s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.15s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.14s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.15s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:11<00:08, 4.16s/it]
Loading checkpoint shards: 89%|████████▉ | 17/19 [01:12<00:08, 4.16s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.17s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.12s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.12s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.15s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.16s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.18s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:15<00:04, 4.18s/it]
Loading checkpoint shards: 95%|█████████▍| 18/19 [01:16<00:04, 4.16s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.97s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.15s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 3.93s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:18<00:00, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 3.94s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 4.16s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 3.97s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 4.17s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 3.97s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 4.18s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 3.97s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 4.18s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 3.97s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 4.18s/it]
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 3.98s/it]
Loading checkpoint shards: 100%|██████████| 19/19 [01:19<00:00, 4.20s/it]
🚀 INFO | 2024-04-02 16:54:55 | main🚋332 - model dtype: torch.float16
🚀 INFO | 2024-04-02 16:54:55 | main🚋340 - preparing peft model...
INFO AiMavenAi-Mixtral112: GET - 202 - Polling reqId for completion
🚀 INFO | 2024-04-02 16:55:07 | main🚋391 - Using block size 16
🚀 INFO | 2024-04-02 16:55:07 | main🚋453 - creating trainer
INFO Running jobs: [62]
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 0 examples [00:00, ? examples/s]
❌ ERROR | 2024-04-02 16:55:07 | autotrain.trainers.common:wrapper:115 - train has failed due to an exception: Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1676, in _prepare_split_single
for key, record in generator:
File "/app/env/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 30, in _generate_examples
for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 517, in data_generator
yield from constant_length_iterator
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 457, in iter
buffer.append(self.formatting_func(next(iterator)))
File "/app/env/lib/python3.10/site-packages/trl/trainer/utils.py", line 434, in
self.formatting_func = lambda x: x[dataset_text_field]
KeyError: 'text'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py", line 520, in _prepare_packed_dataloader
packed_dataset = Dataset.from_generator(
File "/app/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1072, in from_generator
).read()
File "/app/env/lib/python3.10/site-packages/datasets/io/generator.py", line 47, in read
self.builder.download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 954, in download_and_prepare
self._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
super()._download_and_prepare(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
self._prepare_split(split_generator, **prepare_split_kwargs)
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1555, in _prepare_split
for job_id, done, content in self._prepare_split_single(
File "/app/env/lib/python3.10/site-packages/datasets/builder.py", line 1712, in _prepare_split_single
raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
from autotrain-advanced.
abhishekkrthakur
commented on July 4, 2024
is it possible for you to share the dataset?
from autotrain-advanced.
treeoflovely
commented on July 4, 2024
Not possible. I can try to recreate the error with one I can.
from autotrain-advanced.
github-actions
commented on July 4, 2024
This issue is stale because it has been open for 15 days with no activity.
from autotrain-advanced.
github-actions
commented on July 4, 2024
This issue was closed because it has been inactive for 2 days since being marked as stale.
from autotrain-advanced.
Related Issues (20)
- [FEATURE REQUEST] Extend AutoTrain API with Start Training Request HOT 1
- [FEATURE REQUEST] Add W&B Logging to AutoTrain UI and API
- Training process crashes suddenly HOT 2
- Autotrain CLI suddenly crashes HOT 4
- [BUG] Config name is missing for Datasets with no default config HOT 1
- [BUG] AutoTrain_LLM.ipynb quantization = none HOT 7
- [BUG] Error when running autotrain CLI from huggingface - unrecognized arguments HOT 1
- [Push to Hub fails for local data paths] HOT 9
- [BUG] Distant computer (4 GPUs 10GiB of VRAM each) crashes the second i launch the finetuning using AutoTrain localy of mistral 7B HOT 9
- [BUG] Invalid YAML in README.md error after fine tuning is completed HOT 1
- [BUG] Autotrain fails due to missing metadata.jsonl, after removing my metadata.jsonl file. HOT 1
- [BUG] ObjectDetectionParams for uploaded Archive.zip has null data_path HOT 4
- [BUG] Autotrain Object Detection Error: KeyError: 'autotrain_label' HOT 13
- [BUG] Object Detection AutoTrain Error: iteration over a 0-d tensor HOT 12
- Text Classification (Multi-Class) not supported anymore? HOT 2
- Image Classification Training Issue with Hugging Face's AutoTrain HOT 1
- [BUG]Tabular Classification Error HOT 2
- Sort models alphabetically in the UI HOT 1
- [FEATURE REQUEST]SimPO support
- [BUG] BuilderConfig 'qa' not found, when finetunnig custom embedding models HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autotrain-advanced.