Validation sanity check: 0it [00:00, ?it/s]2020-12-14 19:32:07,863|transformers.modeling_longformer|INFO> Input ids are automatically padded from 1315 to 1536 to be a multiple of config.attention_window
: 512
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [162,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1607370172916/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [174,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize
failed.
Traceback (most recent call last):
File "main.py", line 408, in
main(main_args)
File "main.py", line 95, in main
trainer.fit(model)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 470, in fit
results = self.accelerator_backend.train()
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 66, in train
results = self.train_or_test()
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 65, in train_or_test
results = self.trainer.train()
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 492, in train
self.run_sanity_check(self.get_model())
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 690, in run_sanity_check
_, eval_results = self.run_evaluation(test_mode=False, max_batches=self.num_sanity_val_batches)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 606, in run_evaluation
output = self.evaluation_loop.evaluation_step(test_mode, batch, batch_idx, dataloader_idx)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 178, in evaluation_step
output = self.trainer.accelerator_backend.validation_step(args)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 84, in validation_step
return self._step(self.trainer.model.validation_step, args)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 76, in _step
output = model_step(*args)
File "/home/admin02/code/sss/longformer/transformersum/src/extractive.py", line 762, in validation_step
outputs, mask = self.forward(**batch)
File "/home/admin02/code/sss/longformer/transformersum/src/extractive.py", line 284, in forward
outputs = self.word_embedding_model(**inputs, **kwargs)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/transformers/modeling_longformer.py", line 1000, in forward
encoder_outputs = self.encoder(
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/transformers/modeling_longformer.py", line 695, in forward
layer_outputs = layer_module(hidden_states, attention_mask, output_attentions,)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/transformers/modeling_longformer.py", line 658, in forward
self_attn_outputs = self.attention(hidden_states, attention_mask, output_attentions=output_attentions,)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/transformers/modeling_longformer.py", line 642, in forward
self_outputs = self.self(hidden_states, attention_mask, output_attentions,)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/admin02/.conda/envs/transformersum/lib/python3.8/site-packages/transformers/modeling_longformer.py", line 299, in forward
is_global_attn = any(is_index_global_attn.flatten())
RuntimeError: CUDA error: device-side assert triggered
Did I get the running environment wrong? Or something else is wrong? I'm not sure.
Thank you in advance.