935963004 / labram Goto Github PK
View Code? Open in Web Editor NEWICLR 2024 spotlight
ICLR 2024 spotlight
祝贺您!很高兴看到您的团队发表的这篇充满见解的文章,关于其中的实验细节,我想向您询问一下。我看到您在文章中说,是否使用CodeBook对实验的结果不会有太大的提升。关于这个实验我很好奇,我想向您询问,有无codebook的实验是怎么做的呢?
Hi @935963004 , thanks for opening the excellent work!
As title illustrated, in finetuning phase line 117, the training accuracy is calculated as the class_acc = (output.max(-1)[-1] == targets).float().mean() , since the output has the tensor with dimension (64, 3) and target (64, 1), after max(-1)[-1], the pred dimension is (64), directly calling == will use the pytorch broadcasting mechanism,
Here is the minimal reproducible code:
import torch
pred = torch.tensor([0, 0, 2, 1, 0, 2, 2, 2, 1, 1, 0, 2, 0, 2, 0, 0, 1, 1, 2, 0, 2, 0, 0, 2,
0, 0, 0, 0, 0, 2, 2, 0, 1, 2, 0, 2, 0, 2, 1, 2, 0, 0, 1, 0, 2, 1, 2, 0,
2, 1, 1, 2, 1, 1, 2, 2, 0, 1, 0, 0, 1, 1, 0, 2])
target = torch.tensor([[0],[0],[2],[1],[0],[2],[2], [2],[1],[1],[0],[2],[0],[2],[0],[1],[1], [1], [2], [0], [2], [1], [0], [2],[0],[0], [0],[0], [0], [2],[2],[0],[1],[2],
[0],[2],[0],[2],[1],[2],[0],[0],[1],[0],[2],[1], [2],[0],[2],[1],[1],[2], [1], [1],[2],[2],[0],[1],[0],[0],[1],[1],[0],[2]])
comparison = (pred == target).float()
comparison_squeeze = (pred == target.squeeze()).float()
accuracy = comparison.mean()
accuracy_squeeze = comparison_squeeze.mean()
accuracy.item(), accuracy_squeeze.item()
(0.3408203125, 0.96875)
If the dataset is a cnt file, what code needs to be used or written to use the model?I seem to be seeing a couple of preprocessed python scripts:make_h5dataset_for_pretrain.py、dataset.py. Do I just modify the folder path and run it directly, or do I need to write my own code? If I need to write my own code, what do I need to pay attention to? I've seen a lot of people have questions about how to run the model on their own dataset, can you provide some help? Thank you
作者您好,根据您提供的代码,我制作了相应的多个数据集文件用于训练vqnsp,每个数据集单独保存为一个.hdf5文件。并且数据集借助仓库中提供的ShockDataset()类进行加载。但是在加载时发现,使用shuffle=True创建的Sampler会大大降低数据集加载的速度,导致GPU一直处于闲置的状态(数据集加载过慢)。当shuffle=False时,训练速度才恢复正常。我想这和dataloader对样本进行采样的逻辑有关,当随机采样时,需要频繁访问多个磁盘位置的数据;当按顺序采样时,只需要采样物理存储位置较近的相邻的数据即可。所以顺序采样加载数据集的速度更快。但是这其实不符合小批量随机梯度下降的训练逻辑。请问您在训练过程中有遇到这样的问题吗,不知您是如何解决的?
Thank you very much for the code. I would like to ask you, you mentioned in the code that you need to provide different time windows to ensure that the sequence length of each data set is equal to 256, but for data sets with the number of channels less than 64/32, do we need to complete it to 64/32? 32?
I put the modified cnt dataset on the model and ran it sending some errors.
D:\ProgramData\anaconda3\envs\labram\python.exe E:\lab\DL\LaBraM-main\run_class_finetuning.py
Not using distributed mode
Namespace(batch_size=64, epochs=30, update_freq=1, save_ckpt_freq=5, robust_test=None, model='labram_base_patch200_200', qkv_bias=True, rel_pos_bias=True, abs_pos_emb=True, layer_scale_init_value=0.1, input_size=200, drop=0.0, attn_drop_rate=0.0, drop_path=0.1, disable_eval_during_finetuning=False, model_ema=False, model_ema_decay=0.9999, model_ema_force_cpu=False, opt='adamw', opt_eps=1e-08, opt_betas=None, clip_grad=None, momentum=0.9, weight_decay=0.05, weight_decay_end=None, lr=0.0005, layer_decay=0.9, warmup_lr=1e-06, min_lr=1e-06, warmup_epochs=5, warmup_steps=-1, smoothing=0.1, reprob=0.25, remode='pixel', recount=1, resplit=False, finetune='', model_key='model|module', model_prefix='', model_filter_name='gzp', init_scale=0.001, use_mean_pooling=True, disable_weight_decay_on_rel_pos_bias=False, nb_classes=4, output_dir='E:/lab/DL/LaBraM-main/checkpoints/finetune_MI_base', log_dir='E:/lab/DL/LaBraM-main/log/finetune_MI_base', device='cuda', seed=0, resume='', auto_resume=True, save_ckpt=True, start_epoch=0, eval=False, dist_eval=False, num_workers=10, pin_mem=True, world_size=1, local_rank=-1, dist_on_itp=False, dist_url='env://', enable_deepspeed=False, dataset='MI', distributed=False)
2199 399 680
Sampler_train = <torch.utils.data.distributed.DistributedSampler object at 0x000001F93E230D50>
Patch size = 200
Model = NeuralTransformer(
(patch_embed): TemporalConv(
(conv1): Conv2d(1, 8, kernel_size=(1, 15), stride=(1, 8), padding=(0, 7))
(gelu1): GELU(approximate='none')
(norm1): GroupNorm(4, 8, eps=1e-05, affine=True)
(conv2): Conv2d(8, 8, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1))
(gelu2): GELU(approximate='none')
(norm2): GroupNorm(4, 8, eps=1e-05, affine=True)
(conv3): Conv2d(8, 8, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1))
(norm3): GroupNorm(4, 8, eps=1e-05, affine=True)
(gelu3): GELU(approximate='none')
)
(pos_drop): Dropout(p=0.0, inplace=False)
(blocks): ModuleList(
(0): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): Identity()
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(1): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.00909090880304575)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(2): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.0181818176060915)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(3): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.027272727340459824)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(4): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.036363635212183)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(5): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.045454543083906174)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(6): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.054545458406209946)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(7): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.06363636255264282)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(8): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.0727272778749466)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(9): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.08181818574666977)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(10): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.09090909361839294)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
(11): Block(
(norm1): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(attn): Attention(
(qkv): Linear(in_features=200, out_features=600, bias=False)
(q_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(k_norm): LayerNorm((20,), eps=1e-06, elementwise_affine=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=200, out_features=200, bias=True)
(proj_drop): Dropout(p=0.0, inplace=False)
)
(drop_path): DropPath(p=0.10000000149011612)
(norm2): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(mlp): Mlp(
(fc1): Linear(in_features=200, out_features=800, bias=True)
(act): GELU(approximate='none')
(fc2): Linear(in_features=800, out_features=200, bias=True)
(drop): Dropout(p=0.0, inplace=False)
)
)
)
(norm): Identity()
(fc_norm): LayerNorm((200,), eps=1e-06, elementwise_affine=True)
(head): Linear(in_features=200, out_features=4, bias=True)
)
number of params: 5825540
LR = 0.00050000
Batch size = 64
Update frequent = 1
Number of training examples = 2199
Number of training training per epoch = 34
Assigned values = [0.2541865828329001, 0.2824295364810001, 0.31381059609000006, 0.3486784401000001, 0.3874204890000001, 0.4304672100000001, 0.4782969000000001, 0.531441, 0.5904900000000001, 0.6561, 0.7290000000000001, 0.81, 0.9, 1.0]
Skip weight decay name marked in model: {'time_embed', 'cls_token', 'pos_embed'}
Param groups = {
"layer_0_no_decay": {
"weight_decay": 0.0,
"params": [
"cls_token",
"pos_embed",
"patch_embed.conv1.bias",
"patch_embed.norm1.weight",
"patch_embed.norm1.bias",
"patch_embed.conv2.bias",
"patch_embed.norm2.weight",
"patch_embed.norm2.bias",
"patch_embed.conv3.bias",
"patch_embed.norm3.weight",
"patch_embed.norm3.bias"
],
"lr_scale": 0.2541865828329001
},
"layer_13_no_decay": {
"weight_decay": 0.0,
"params": [
"time_embed",
"fc_norm.weight",
"fc_norm.bias",
"head.bias"
],
"lr_scale": 1.0
},
"layer_0_decay": {
"weight_decay": 0.05,
"params": [
"patch_embed.conv1.weight",
"patch_embed.conv2.weight",
"patch_embed.conv3.weight"
],
"lr_scale": 0.2541865828329001
},
"layer_1_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.0.gamma_1",
"blocks.0.gamma_2",
"blocks.0.norm1.weight",
"blocks.0.norm1.bias",
"blocks.0.attn.q_bias",
"blocks.0.attn.v_bias",
"blocks.0.attn.q_norm.weight",
"blocks.0.attn.q_norm.bias",
"blocks.0.attn.k_norm.weight",
"blocks.0.attn.k_norm.bias",
"blocks.0.attn.proj.bias",
"blocks.0.norm2.weight",
"blocks.0.norm2.bias",
"blocks.0.mlp.fc1.bias",
"blocks.0.mlp.fc2.bias"
],
"lr_scale": 0.2824295364810001
},
"layer_1_decay": {
"weight_decay": 0.05,
"params": [
"blocks.0.attn.qkv.weight",
"blocks.0.attn.proj.weight",
"blocks.0.mlp.fc1.weight",
"blocks.0.mlp.fc2.weight"
],
"lr_scale": 0.2824295364810001
},
"layer_2_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.1.gamma_1",
"blocks.1.gamma_2",
"blocks.1.norm1.weight",
"blocks.1.norm1.bias",
"blocks.1.attn.q_bias",
"blocks.1.attn.v_bias",
"blocks.1.attn.q_norm.weight",
"blocks.1.attn.q_norm.bias",
"blocks.1.attn.k_norm.weight",
"blocks.1.attn.k_norm.bias",
"blocks.1.attn.proj.bias",
"blocks.1.norm2.weight",
"blocks.1.norm2.bias",
"blocks.1.mlp.fc1.bias",
"blocks.1.mlp.fc2.bias"
],
"lr_scale": 0.31381059609000006
},
"layer_2_decay": {
"weight_decay": 0.05,
"params": [
"blocks.1.attn.qkv.weight",
"blocks.1.attn.proj.weight",
"blocks.1.mlp.fc1.weight",
"blocks.1.mlp.fc2.weight"
],
"lr_scale": 0.31381059609000006
},
"layer_3_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.2.gamma_1",
"blocks.2.gamma_2",
"blocks.2.norm1.weight",
"blocks.2.norm1.bias",
"blocks.2.attn.q_bias",
"blocks.2.attn.v_bias",
"blocks.2.attn.q_norm.weight",
"blocks.2.attn.q_norm.bias",
"blocks.2.attn.k_norm.weight",
"blocks.2.attn.k_norm.bias",
"blocks.2.attn.proj.bias",
"blocks.2.norm2.weight",
"blocks.2.norm2.bias",
"blocks.2.mlp.fc1.bias",
"blocks.2.mlp.fc2.bias"
],
"lr_scale": 0.3486784401000001
},
"layer_3_decay": {
"weight_decay": 0.05,
"params": [
"blocks.2.attn.qkv.weight",
"blocks.2.attn.proj.weight",
"blocks.2.mlp.fc1.weight",
"blocks.2.mlp.fc2.weight"
],
"lr_scale": 0.3486784401000001
},
"layer_4_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.3.gamma_1",
"blocks.3.gamma_2",
"blocks.3.norm1.weight",
"blocks.3.norm1.bias",
"blocks.3.attn.q_bias",
"blocks.3.attn.v_bias",
"blocks.3.attn.q_norm.weight",
"blocks.3.attn.q_norm.bias",
"blocks.3.attn.k_norm.weight",
"blocks.3.attn.k_norm.bias",
"blocks.3.attn.proj.bias",
"blocks.3.norm2.weight",
"blocks.3.norm2.bias",
"blocks.3.mlp.fc1.bias",
"blocks.3.mlp.fc2.bias"
],
"lr_scale": 0.3874204890000001
},
"layer_4_decay": {
"weight_decay": 0.05,
"params": [
"blocks.3.attn.qkv.weight",
"blocks.3.attn.proj.weight",
"blocks.3.mlp.fc1.weight",
"blocks.3.mlp.fc2.weight"
],
"lr_scale": 0.3874204890000001
},
"layer_5_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.4.gamma_1",
"blocks.4.gamma_2",
"blocks.4.norm1.weight",
"blocks.4.norm1.bias",
"blocks.4.attn.q_bias",
"blocks.4.attn.v_bias",
"blocks.4.attn.q_norm.weight",
"blocks.4.attn.q_norm.bias",
"blocks.4.attn.k_norm.weight",
"blocks.4.attn.k_norm.bias",
"blocks.4.attn.proj.bias",
"blocks.4.norm2.weight",
"blocks.4.norm2.bias",
"blocks.4.mlp.fc1.bias",
"blocks.4.mlp.fc2.bias"
],
"lr_scale": 0.4304672100000001
},
"layer_5_decay": {
"weight_decay": 0.05,
"params": [
"blocks.4.attn.qkv.weight",
"blocks.4.attn.proj.weight",
"blocks.4.mlp.fc1.weight",
"blocks.4.mlp.fc2.weight"
],
"lr_scale": 0.4304672100000001
},
"layer_6_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.5.gamma_1",
"blocks.5.gamma_2",
"blocks.5.norm1.weight",
"blocks.5.norm1.bias",
"blocks.5.attn.q_bias",
"blocks.5.attn.v_bias",
"blocks.5.attn.q_norm.weight",
"blocks.5.attn.q_norm.bias",
"blocks.5.attn.k_norm.weight",
"blocks.5.attn.k_norm.bias",
"blocks.5.attn.proj.bias",
"blocks.5.norm2.weight",
"blocks.5.norm2.bias",
"blocks.5.mlp.fc1.bias",
"blocks.5.mlp.fc2.bias"
],
"lr_scale": 0.4782969000000001
},
"layer_6_decay": {
"weight_decay": 0.05,
"params": [
"blocks.5.attn.qkv.weight",
"blocks.5.attn.proj.weight",
"blocks.5.mlp.fc1.weight",
"blocks.5.mlp.fc2.weight"
],
"lr_scale": 0.4782969000000001
},
"layer_7_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.6.gamma_1",
"blocks.6.gamma_2",
"blocks.6.norm1.weight",
"blocks.6.norm1.bias",
"blocks.6.attn.q_bias",
"blocks.6.attn.v_bias",
"blocks.6.attn.q_norm.weight",
"blocks.6.attn.q_norm.bias",
"blocks.6.attn.k_norm.weight",
"blocks.6.attn.k_norm.bias",
"blocks.6.attn.proj.bias",
"blocks.6.norm2.weight",
"blocks.6.norm2.bias",
"blocks.6.mlp.fc1.bias",
"blocks.6.mlp.fc2.bias"
],
"lr_scale": 0.531441
},
"layer_7_decay": {
"weight_decay": 0.05,
"params": [
"blocks.6.attn.qkv.weight",
"blocks.6.attn.proj.weight",
"blocks.6.mlp.fc1.weight",
"blocks.6.mlp.fc2.weight"
],
"lr_scale": 0.531441
},
"layer_8_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.7.gamma_1",
"blocks.7.gamma_2",
"blocks.7.norm1.weight",
"blocks.7.norm1.bias",
"blocks.7.attn.q_bias",
"blocks.7.attn.v_bias",
"blocks.7.attn.q_norm.weight",
"blocks.7.attn.q_norm.bias",
"blocks.7.attn.k_norm.weight",
"blocks.7.attn.k_norm.bias",
"blocks.7.attn.proj.bias",
"blocks.7.norm2.weight",
"blocks.7.norm2.bias",
"blocks.7.mlp.fc1.bias",
"blocks.7.mlp.fc2.bias"
],
"lr_scale": 0.5904900000000001
},
"layer_8_decay": {
"weight_decay": 0.05,
"params": [
"blocks.7.attn.qkv.weight",
"blocks.7.attn.proj.weight",
"blocks.7.mlp.fc1.weight",
"blocks.7.mlp.fc2.weight"
],
"lr_scale": 0.5904900000000001
},
"layer_9_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.8.gamma_1",
"blocks.8.gamma_2",
"blocks.8.norm1.weight",
"blocks.8.norm1.bias",
"blocks.8.attn.q_bias",
"blocks.8.attn.v_bias",
"blocks.8.attn.q_norm.weight",
"blocks.8.attn.q_norm.bias",
"blocks.8.attn.k_norm.weight",
"blocks.8.attn.k_norm.bias",
"blocks.8.attn.proj.bias",
"blocks.8.norm2.weight",
"blocks.8.norm2.bias",
"blocks.8.mlp.fc1.bias",
"blocks.8.mlp.fc2.bias"
],
"lr_scale": 0.6561
},
"layer_9_decay": {
"weight_decay": 0.05,
"params": [
"blocks.8.attn.qkv.weight",
"blocks.8.attn.proj.weight",
"blocks.8.mlp.fc1.weight",
"blocks.8.mlp.fc2.weight"
],
"lr_scale": 0.6561
},
"layer_10_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.9.gamma_1",
"blocks.9.gamma_2",
"blocks.9.norm1.weight",
"blocks.9.norm1.bias",
"blocks.9.attn.q_bias",
"blocks.9.attn.v_bias",
"blocks.9.attn.q_norm.weight",
"blocks.9.attn.q_norm.bias",
"blocks.9.attn.k_norm.weight",
"blocks.9.attn.k_norm.bias",
"blocks.9.attn.proj.bias",
"blocks.9.norm2.weight",
"blocks.9.norm2.bias",
"blocks.9.mlp.fc1.bias",
"blocks.9.mlp.fc2.bias"
],
"lr_scale": 0.7290000000000001
},
"layer_10_decay": {
"weight_decay": 0.05,
"params": [
"blocks.9.attn.qkv.weight",
"blocks.9.attn.proj.weight",
"blocks.9.mlp.fc1.weight",
"blocks.9.mlp.fc2.weight"
],
"lr_scale": 0.7290000000000001
},
"layer_11_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.10.gamma_1",
"blocks.10.gamma_2",
"blocks.10.norm1.weight",
"blocks.10.norm1.bias",
"blocks.10.attn.q_bias",
"blocks.10.attn.v_bias",
"blocks.10.attn.q_norm.weight",
"blocks.10.attn.q_norm.bias",
"blocks.10.attn.k_norm.weight",
"blocks.10.attn.k_norm.bias",
"blocks.10.attn.proj.bias",
"blocks.10.norm2.weight",
"blocks.10.norm2.bias",
"blocks.10.mlp.fc1.bias",
"blocks.10.mlp.fc2.bias"
],
"lr_scale": 0.81
},
"layer_11_decay": {
"weight_decay": 0.05,
"params": [
"blocks.10.attn.qkv.weight",
"blocks.10.attn.proj.weight",
"blocks.10.mlp.fc1.weight",
"blocks.10.mlp.fc2.weight"
],
"lr_scale": 0.81
},
"layer_12_no_decay": {
"weight_decay": 0.0,
"params": [
"blocks.11.gamma_1",
"blocks.11.gamma_2",
"blocks.11.norm1.weight",
"blocks.11.norm1.bias",
"blocks.11.attn.q_bias",
"blocks.11.attn.v_bias",
"blocks.11.attn.q_norm.weight",
"blocks.11.attn.q_norm.bias",
"blocks.11.attn.k_norm.weight",
"blocks.11.attn.k_norm.bias",
"blocks.11.attn.proj.bias",
"blocks.11.norm2.weight",
"blocks.11.norm2.bias",
"blocks.11.mlp.fc1.bias",
"blocks.11.mlp.fc2.bias"
],
"lr_scale": 0.9
},
"layer_12_decay": {
"weight_decay": 0.05,
"params": [
"blocks.11.attn.qkv.weight",
"blocks.11.attn.proj.weight",
"blocks.11.mlp.fc1.weight",
"blocks.11.mlp.fc2.weight"
],
"lr_scale": 0.9
},
"layer_13_decay": {
"weight_decay": 0.05,
"params": [
"head.weight"
],
"lr_scale": 1.0
}
}
Optimizer config: {'lr': 0.0005, 'weight_decay': 0.0, 'eps': 1e-08}
Use step level LR scheduler!
Set warmup steps = 170
Set warmup steps = 0
Max WD = 0.0500000, Min WD = 0.0500000
criterion = LabelSmoothingCrossEntropy()
Auto resume checkpoint:
Start training for 30 epochs
Traceback (most recent call last):
File "E:\lab\DL\LaBraM-main\run_class_finetuning.py", line 582, in
main(opts, ds_init)
File "E:\lab\DL\LaBraM-main\run_class_finetuning.py", line 496, in main
train_stats = train_one_epoch(
^^^^^^^^^^^^^^^^
File "E:\lab\DL\LaBraM-main\engine_for_finetuning.py", line 77, in train_one_epoch
loss, output = train_class_batch(
^^^^^^^^^^^^^^^^^^
File "E:\lab\DL\LaBraM-main\engine_for_finetuning.py", line 19, in train_class_batch
outputs = model(samples, ch_names)
^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ProgramData\anaconda3\envs\labram\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ProgramData\anaconda3\envs\labram\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\lab\DL\LaBraM-main\modeling_finetune.py", line 395, in forward
x = self.forward_features(x, input_chans=input_chans, return_patch_tokens=return_patch_tokens, return_all_tokens=return_all_tokens, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\lab\DL\LaBraM-main\modeling_finetune.py", line 362, in forward_features
x = x + pos_embed
~~^~~~~~~~~~~
RuntimeError: The size of tensor a (341) must match the size of tensor b (286) at non-singleton dimension 1
进程已结束,退出代码为 1
Hello @935963004,
I would like to starting say thank you for your work, I think it is a fundamental and necessary work in EEG decoding. Thank you for that!
So, I am trying to understand and run your code, but some things are not working, and I would like to request your assistance. From the beginning, with a toy example.
import torch
from torch import nn
from modeling_finetune import NeuralTransformer
# As commment in the meet, the expect input is:
# Batch size, channels, time//patch_size, patch_size
in_chans = 1 # **Not working if in_chans is different of 1. Issue with temporal_embedding.**
batch_size = 1
patch_size = 200
n_time_points_patched = 16 # Max number for patch, the value is hardcode in
# the model
EEG_size = 1600
# Generating an empty vector just to get the output.
X = torch.zeros(batch_size, in_chans, n_time_points_patched, patch_size)
# Everything is default
model = NeuralTransformer(
EEG_size=EEG_size,
patch_size=patch_size,
in_chans=in_chans,
out_chans=8,
num_classes=1000,
embed_dim=200,
depth=12,
num_heads=10,
mlp_ratio=4.,
qkv_bias=False,
qk_norm=None,
qk_scale=None,
drop_rate=0.,
attn_drop_rate=0.,
drop_path_rate=0.,
norm_layer=nn.LayerNorm,
init_values=0, # default value is not working, changed from None to zero.
use_abs_pos_emb=False, # Not working
use_rel_pos_bias=False,
use_shared_rel_pos_bias=False,
use_mean_pooling=True,
init_scale=0.001,
)
with torch.no_grad():
y_pred = model(X)
My questions are:
In my naive intuition if I change the in_chans everything should working because of the TemporalConv module, but it's not.
FYI @LemonFace0309, @jonxuxu and @shahbuland, @RashikShahjahan
All the best!
作者,您好!
首先非常感谢您分享如此精彩的工作。我有一个小小的困惑,恳请您能给予解答。
就是在预训练阶段是否划分了验证集进行模型预训练效果评估,如果没有验证集,单纯基于所有预训练数据,如何判断预训练达到多少epoch后停止可以得到最优的预训练模型。
祝好!
Hi @935963004,
Thanks so much for your work with this project - really exciting stuff. I was trying out fine-tuning on the TUAB dataset and came across some unexpected numbers. It could be something on my end and I'll provide update(s) as I dig into it further but wanted to get a thread started. Thanks for any help!
Created the TUAB dataset with make_TUAB.py and then running fine tuning with the recommended settings in the readme (except for setting to 1 gpu, increasing batch size, and num workers, and removing --dist_eval tag):
!OMP_NUM_THREADS=1 torchrun --nnodes=1 --nproc_per_node=1 run_class_finetuning.py \
--output_dir ./checkpoints/finetune_tuab_base/ \
--log_dir ./log/finetune_tuab_base \
--model labram_base_patch200_200 \
--finetune ./checkpoints/labram-base.pth \
--weight_decay 0.05 \
--batch_size 512 \
--lr 5e-4 \
--update_freq 1 \
--warmup_epochs 5 \
--epochs 50 \
--layer_decay 0.65 \
--drop_path 0.1 \
--save_ckpt_freq 5 \
--disable_rel_pos_bias \
--abs_pos_emb \
--dataset TUAB \
--disable_qkv_bias \
--seed 0 \
--num_workers 12 \
Below is some sample output I'm seeing:
...
Use step level LR scheduler!
Set warmup steps = 605
Set warmup steps = 0
Max WD = 0.0500000, Min WD = 0.0500000
criterion = BCEWithLogitsLoss()
Auto resume checkpoint:
Start training for 50 epochs
[W reducer.cpp:1346] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
Warning: NaN or Inf found in input tensor.
Epoch: [0] [ 0/121] eta: 3 days, 16:01:13 lr: 0.000000 min_lr: 0.000000 loss: 0.6931 (0.6931) class_acc: 1.0000 (1.0000) loss_scale: 32768.0000 (32768.0000) weight_decay: 0.0500 (0.0500) grad_norm: inf (inf) time: 2618.7900 data: 2615.6816 max mem: 34391
Warning: NaN or Inf found in input tensor.
Warning: NaN or Inf found in input tensor.
Epoch: [0] [ 10/121] eta: 7:21:42 lr: 0.000008 min_lr: 0.000000 loss: 0.6929 (0.6927) class_acc: 1.0000 (0.9998) loss_scale: 8192.0000 (11170.9091) weight_decay: 0.0500 (0.0500) grad_norm: 6.2091 (inf) time: 238.7637 data: 237.7895 max mem: 34433
Epoch: [0] [ 20/121] eta: 3:31:06 lr: 0.000017 min_lr: 0.000000 loss: 0.6915 (0.6911) class_acc: 1.0000 (0.9999) loss_scale: 8192.0000 (9752.3810) weight_decay: 0.0500 (0.0500) grad_norm: 6.1851 (inf) time: 0.7382 data: 0.0030 max mem: 34433
Epoch: [0] [ 30/121] eta: 2:09:22 lr: 0.000025 min_lr: 0.000000 loss: 0.6854 (0.6872) class_acc: 1.0000 (0.9999) loss_scale: 8192.0000 (9249.0323) weight_decay: 0.0500 (0.0500) grad_norm: 6.0339 (inf) time: 0.9033 data: 0.1924 max mem: 34433
Epoch: [0] [ 40/121] eta: 1:27:24 lr: 0.000033 min_lr: 0.000000 loss: 0.6681 (0.6793) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8991.2195) weight_decay: 0.0500 (0.0500) grad_norm: 5.7410 (inf) time: 1.0505 data: 0.3342 max mem: 34433
Epoch: [0] [ 50/121] eta: 1:01:48 lr: 0.000041 min_lr: 0.000000 loss: 0.6368 (0.6673) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8834.5098) weight_decay: 0.0500 (0.0500) grad_norm: 5.5797 (inf) time: 0.9733 data: 0.2568 max mem: 34433
Epoch: [0] [ 60/121] eta: 0:44:31 lr: 0.000050 min_lr: 0.000000 loss: 0.5923 (0.6513) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8729.1803) weight_decay: 0.0500 (0.0500) grad_norm: 5.6199 (inf) time: 0.8588 data: 0.1478 max mem: 34433
Epoch: [0] [ 70/121] eta: 0:32:04 lr: 0.000058 min_lr: 0.000000 loss: 0.5391 (0.6321) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8653.5211) weight_decay: 0.0500 (0.0500) grad_norm: 5.5495 (inf) time: 0.7452 data: 0.0358 max mem: 34433
Epoch: [0] [ 80/121] eta: 0:22:40 lr: 0.000066 min_lr: 0.000000 loss: 0.4821 (0.6105) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8596.5432) weight_decay: 0.0500 (0.0500) grad_norm: 5.2271 (inf) time: 0.7910 data: 0.0819 max mem: 34433
Epoch: [0] [ 90/121] eta: 0:15:19 lr: 0.000075 min_lr: 0.000000 loss: 0.4252 (0.5875) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8552.0879) weight_decay: 0.0500 (0.0500) grad_norm: 4.8080 (inf) time: 1.0040 data: 0.2929 max mem: 34433
Epoch: [0] [100/121] eta: 0:09:22 lr: 0.000083 min_lr: 0.000000 loss: 0.3711 (0.5638) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8516.4356) weight_decay: 0.0500 (0.0500) grad_norm: 4.3451 (inf) time: 0.9471 data: 0.2359 max mem: 34433
Epoch: [0] [110/121] eta: 0:04:28 lr: 0.000091 min_lr: 0.000000 loss: 0.3215 (0.5400) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8487.2072) weight_decay: 0.0500 (0.0500) grad_norm: 3.8768 (inf) time: 0.7331 data: 0.0248 max mem: 34433
Epoch: [0] [120/121] eta: 0:00:22 lr: 0.000099 min_lr: 0.000000 loss: 0.2762 (0.5166) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8462.8099) weight_decay: 0.0500 (0.0500) grad_norm: 3.4341 (inf) time: 0.7080 data: 0.0001 max mem: 34433
Epoch: [0] Total time: 0:45:20 (22.4855 s / it)
Averaged stats: lr: 0.000099 min_lr: 0.000000 loss: 0.2762 (0.5166) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8462.8099) weight_decay: 0.0500 (0.0500) grad_norm: 3.4341 (inf)
Val: [ 0/20] eta: 2:50:41 loss: 0.2346 (0.2346) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 512.0632 data: 511.3996 max mem: 34433
Val: [10/20] eta: 0:09:48 loss: 0.2346 (0.2346) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 58.8147 data: 58.4929 max mem: 34433
Val: [19/20] eta: 0:01:01 loss: 0.2346 (0.2346) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 61.3067 data: 61.0130 max mem: 34433
Val: Total time: 0:20:26 (61.3127 s / it)
* loss 0.235
Accuracy of the network on the 14718 val EEG: 0.00%
Test: [ 0/11] eta: 1:34:08 loss: 0.2346 (0.2346) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 513.4699 data: 513.1569 max mem: 34433
Test: [10/11] eta: 0:00:48 loss: 0.2346 (0.2346) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 48.5245 data: 48.2462 max mem: 34433
Test: Total time: 0:08:53 (48.5333 s / it)
* loss 0.235
Accuracy of the network on the 14718 test EEG: 0.00%
Max accuracy val: 0.00%, max accuracy test: 0.00%
Epoch: [1] [ 0/121] eta: 0:25:29 lr: 0.000100 min_lr: 0.000000 loss: 0.2362 (0.2362) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 3.0189 (3.0189) time: 12.6372 data: 11.8822 max mem: 34435
Epoch: [1] [ 10/121] eta: 0:03:19 lr: 0.000108 min_lr: 0.000000 loss: 0.2185 (0.2189) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 2.8253 (2.8273) time: 1.7946 data: 1.0805 max mem: 34435
Epoch: [1] [ 20/121] eta: 0:02:23 lr: 0.000117 min_lr: 0.000000 loss: 0.1991 (0.2033) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 2.6067 (2.6502) time: 0.8571 data: 0.1461 max mem: 34435
Epoch: [1] [ 30/121] eta: 0:01:58 lr: 0.000125 min_lr: 0.000000 loss: 0.1711 (0.1894) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 2.2782 (2.4876) time: 1.0270 data: 0.3148 max mem: 34435
Epoch: [1] [ 40/121] eta: 0:01:40 lr: 0.000133 min_lr: 0.000000 loss: 0.1473 (0.1768) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 1.9917 (2.3386) time: 1.0426 data: 0.3300 max mem: 34435
Epoch: [1] [ 50/121] eta: 0:01:25 lr: 0.000142 min_lr: 0.000001 loss: 0.1273 (0.1656) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 1.7436 (2.2026) time: 1.0637 data: 0.3517 max mem: 34435
Epoch: [1] [ 60/121] eta: 0:01:12 lr: 0.000150 min_lr: 0.000001 loss: 0.1103 (0.1555) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 1.5298 (2.0785) time: 1.1007 data: 0.3882 max mem: 34435
Epoch: [1] [ 70/121] eta: 0:00:57 lr: 0.000158 min_lr: 0.000001 loss: 0.0962 (0.1463) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 1.3468 (1.9653) time: 0.9096 data: 0.1978 max mem: 34435
Epoch: [1] [ 80/121] eta: 0:00:46 lr: 0.000166 min_lr: 0.000001 loss: 0.0842 (0.1381) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 1.1898 (1.8619) time: 0.9218 data: 0.2102 max mem: 34435
Epoch: [1] [ 90/121] eta: 0:00:34 lr: 0.000175 min_lr: 0.000001 loss: 0.0741 (0.1306) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 1.0556 (1.7673) time: 1.1428 data: 0.4293 max mem: 34435
Epoch: [1] [100/121] eta: 0:00:23 lr: 0.000183 min_lr: 0.000001 loss: 0.0654 (0.1238) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.9381 (1.6807) time: 1.1746 data: 0.4622 max mem: 34435
Epoch: [1] [110/121] eta: 0:00:12 lr: 0.000191 min_lr: 0.000001 loss: 0.0580 (0.1176) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.8384 (1.6012) time: 1.2109 data: 0.4988 max mem: 34435
Epoch: [1] [120/121] eta: 0:00:01 lr: 0.000200 min_lr: 0.000001 loss: 0.0518 (0.1120) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.7521 (1.5281) time: 0.9662 data: 0.2559 max mem: 34435
Epoch: [1] Total time: 0:02:13 (1.1073 s / it)
Averaged stats: lr: 0.000200 min_lr: 0.000001 loss: 0.0518 (0.1120) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.7521 (1.5281)
Val: [ 0/20] eta: 0:05:09 loss: 0.0461 (0.0461) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 15.4873 data: 15.0481 max mem: 34435
Val: [10/20] eta: 0:00:16 loss: 0.0461 (0.0461) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.6699 data: 1.3682 max mem: 34435
Val: [19/20] eta: 0:00:01 loss: 0.0461 (0.0461) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.3343 data: 1.0552 max mem: 34435
Val: Total time: 0:00:26 (1.3377 s / it)
* loss 0.046
Accuracy of the network on the 14718 val EEG: 0.00%
Test: [ 0/11] eta: 0:02:29 loss: 0.0461 (0.0461) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 13.6329 data: 13.3226 max mem: 34435
Test: [10/11] eta: 0:00:01 loss: 0.0461 (0.0461) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.4797 data: 1.2113 max mem: 34435
Test: Total time: 0:00:16 (1.4852 s / it)
* loss 0.046
Accuracy of the network on the 14718 test EEG: 0.00%
Max accuracy val: 0.00%, max accuracy test: 0.00%
Epoch: [2] [ 0/121] eta: 0:21:41 lr: 0.000200 min_lr: 0.000001 loss: 0.0463 (0.0463) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.6762 (0.6762) time: 10.7563 data: 10.0010 max mem: 34435
Epoch: [2] [ 10/121] eta: 0:03:00 lr: 0.000209 min_lr: 0.000001 loss: 0.0437 (0.0438) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.6413 (0.6420) time: 1.6236 data: 0.9095 max mem: 34435
Epoch: [2] [ 20/121] eta: 0:02:06 lr: 0.000217 min_lr: 0.000001 loss: 0.0410 (0.0416) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.6035 (0.6112) time: 0.7737 data: 0.0637 max mem: 34435
Epoch: [2] [ 30/121] eta: 0:01:43 lr: 0.000225 min_lr: 0.000001 loss: 0.0370 (0.0396) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.5469 (0.5830) time: 0.8707 data: 0.1589 max mem: 34435
Epoch: [2] [ 40/121] eta: 0:01:27 lr: 0.000233 min_lr: 0.000001 loss: 0.0335 (0.0378) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.4963 (0.5570) time: 0.9061 data: 0.1927 max mem: 34435
Epoch: [2] [ 50/121] eta: 0:01:14 lr: 0.000242 min_lr: 0.000001 loss: 0.0304 (0.0361) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.4528 (0.5330) time: 0.9294 data: 0.2166 max mem: 34435
Epoch: [2] [ 60/121] eta: 0:01:02 lr: 0.000250 min_lr: 0.000001 loss: 0.0277 (0.0345) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.4138 (0.5108) time: 0.9026 data: 0.1909 max mem: 34435
Epoch: [2] [ 70/121] eta: 0:00:49 lr: 0.000258 min_lr: 0.000001 loss: 0.0252 (0.0331) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.3788 (0.4902) time: 0.7824 data: 0.0718 max mem: 34435
Epoch: [2] [ 80/121] eta: 0:00:39 lr: 0.000267 min_lr: 0.000001 loss: 0.0231 (0.0317) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.3477 (0.4711) time: 0.7819 data: 0.0711 max mem: 34435
Epoch: [2] [ 90/121] eta: 0:00:29 lr: 0.000275 min_lr: 0.000001 loss: 0.0212 (0.0305) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.3201 (0.4533) time: 0.8553 data: 0.1451 max mem: 34435
Epoch: [2] [100/121] eta: 0:00:19 lr: 0.000283 min_lr: 0.000001 loss: 0.0195 (0.0293) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2956 (0.4367) time: 0.8650 data: 0.1550 max mem: 34435
Epoch: [2] [110/121] eta: 0:00:10 lr: 0.000291 min_lr: 0.000001 loss: 0.0180 (0.0283) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2735 (0.4211) time: 0.8804 data: 0.1692 max mem: 34435
Epoch: [2] [120/121] eta: 0:00:00 lr: 0.000300 min_lr: 0.000001 loss: 0.0166 (0.0272) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2533 (0.4066) time: 0.7990 data: 0.0883 max mem: 34435
Epoch: [2] Total time: 0:01:51 (0.9207 s / it)
Averaged stats: lr: 0.000300 min_lr: 0.000001 loss: 0.0166 (0.0272) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2533 (0.4066)
Val: [ 0/20] eta: 0:04:56 loss: 0.0154 (0.0154) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 14.8285 data: 14.5373 max mem: 34435
Val: [10/20] eta: 0:00:15 loss: 0.0154 (0.0154) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.5975 data: 1.3217 max mem: 34435
Val: [19/20] eta: 0:00:01 loss: 0.0154 (0.0154) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.2781 data: 1.0134 max mem: 34435
Val: Total time: 0:00:25 (1.2811 s / it)
* loss 0.015
Accuracy of the network on the 14718 val EEG: 0.00%
Test: [ 0/11] eta: 0:02:27 loss: 0.0154 (0.0154) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 13.4538 data: 13.1486 max mem: 34435
Test: [10/11] eta: 0:00:01 loss: 0.0154 (0.0154) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.4634 data: 1.1955 max mem: 34435
Test: Total time: 0:00:16 (1.4693 s / it)
* loss 0.015
Accuracy of the network on the 14718 test EEG: 0.00%
Max accuracy val: 0.00%, max accuracy test: 0.00%
Epoch: [3] [ 0/121] eta: 0:22:54 lr: 0.000300 min_lr: 0.000001 loss: 0.0154 (0.0154) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2353 (0.2353) time: 11.3583 data: 10.6085 max mem: 34435
Epoch: [3] [ 10/121] eta: 0:03:06 lr: 0.000309 min_lr: 0.000001 loss: 0.0148 (0.0148) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2266 (0.2267) time: 1.6779 data: 0.9647 max mem: 34435
Epoch: [3] [ 20/121] eta: 0:02:11 lr: 0.000317 min_lr: 0.000001 loss: 0.0142 (0.0143) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2171 (0.2189) time: 0.7983 data: 0.0875 max mem: 34435
Epoch: [3] [ 30/121] eta: 0:01:45 lr: 0.000325 min_lr: 0.000001 loss: 0.0132 (0.0138) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.2025 (0.2116) time: 0.8703 data: 0.1595 max mem: 34435
Epoch: [3] [ 40/121] eta: 0:01:27 lr: 0.000334 min_lr: 0.000001 loss: 0.0122 (0.0133) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1886 (0.2047) time: 0.8464 data: 0.1368 max mem: 34435
Epoch: [3] [ 50/121] eta: 0:01:13 lr: 0.000342 min_lr: 0.000001 loss: 0.0114 (0.0129) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1765 (0.1982) time: 0.8564 data: 0.1469 max mem: 34435
Epoch: [3] [ 60/121] eta: 0:01:01 lr: 0.000350 min_lr: 0.000001 loss: 0.0107 (0.0125) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1654 (0.1920) time: 0.8801 data: 0.1703 max mem: 34435
Epoch: [3] [ 70/121] eta: 0:00:49 lr: 0.000358 min_lr: 0.000001 loss: 0.0100 (0.0121) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1552 (0.1862) time: 0.7979 data: 0.0883 max mem: 34435
Epoch: [3] [ 80/121] eta: 0:00:39 lr: 0.000367 min_lr: 0.000001 loss: 0.0094 (0.0117) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1456 (0.1807) time: 0.7810 data: 0.0717 max mem: 34435
Epoch: [3] [ 90/121] eta: 0:00:29 lr: 0.000375 min_lr: 0.000001 loss: 0.0088 (0.0114) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1370 (0.1755) time: 0.8510 data: 0.1414 max mem: 34435
Epoch: [3] [100/121] eta: 0:00:19 lr: 0.000383 min_lr: 0.000001 loss: 0.0083 (0.0110) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1290 (0.1706) time: 0.8608 data: 0.1515 max mem: 34435
Epoch: [3] [110/121] eta: 0:00:10 lr: 0.000392 min_lr: 0.000001 loss: 0.0078 (0.0107) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1217 (0.1659) time: 0.8725 data: 0.1636 max mem: 34435
Epoch: [3] [120/121] eta: 0:00:00 lr: 0.000400 min_lr: 0.000001 loss: 0.0073 (0.0104) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1148 (0.1614) time: 0.7904 data: 0.0820 max mem: 34435
Epoch: [3] Total time: 0:01:50 (0.9140 s / it)
Averaged stats: lr: 0.000400 min_lr: 0.000001 loss: 0.0073 (0.0104) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1148 (0.1614)
Val: [ 0/20] eta: 0:04:54 loss: 0.0069 (0.0069) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 14.7361 data: 14.4306 max mem: 34435
Val: [10/20] eta: 0:00:15 loss: 0.0069 (0.0069) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.5994 data: 1.3224 max mem: 34435
Val: [19/20] eta: 0:00:01 loss: 0.0069 (0.0069) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.2757 data: 1.0101 max mem: 34435
Val: Total time: 0:00:25 (1.2790 s / it)
* loss 0.007
Accuracy of the network on the 14718 val EEG: 0.00%
Test: [ 0/11] eta: 0:02:28 loss: 0.0069 (0.0069) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 13.4998 data: 13.2017 max mem: 34435
Test: [10/11] eta: 0:00:01 loss: 0.0069 (0.0069) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.4676 data: 1.2003 max mem: 34435
Test: Total time: 0:00:16 (1.4737 s / it)
* loss 0.007
Accuracy of the network on the 14718 test EEG: 0.00%
Max accuracy val: 0.00%, max accuracy test: 0.00%
Epoch: [4] [ 0/121] eta: 0:22:45 lr: 0.000401 min_lr: 0.000001 loss: 0.0069 (0.0069) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1085 (0.1085) time: 11.2845 data: 10.5294 max mem: 34435
Epoch: [4] [ 10/121] eta: 0:03:05 lr: 0.000409 min_lr: 0.000002 loss: 0.0067 (0.0067) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1053 (0.1055) time: 1.6719 data: 0.9574 max mem: 34435
Epoch: [4] [ 20/121] eta: 0:02:10 lr: 0.000417 min_lr: 0.000002 loss: 0.0065 (0.0065) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.1020 (0.1027) time: 0.7890 data: 0.0790 max mem: 34435
Epoch: [4] [ 30/121] eta: 0:01:45 lr: 0.000425 min_lr: 0.000002 loss: 0.0061 (0.0063) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0966 (0.1000) time: 0.8712 data: 0.1616 max mem: 34435
Epoch: [4] [ 40/121] eta: 0:01:27 lr: 0.000434 min_lr: 0.000002 loss: 0.0058 (0.0062) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0916 (0.0974) time: 0.8668 data: 0.1574 max mem: 34435
Epoch: [4] [ 50/121] eta: 0:01:14 lr: 0.000442 min_lr: 0.000002 loss: 0.0055 (0.0060) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0869 (0.0950) time: 0.8685 data: 0.1592 max mem: 34435
Epoch: [4] [ 60/121] eta: 0:01:01 lr: 0.000450 min_lr: 0.000002 loss: 0.0052 (0.0059) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0826 (0.0927) time: 0.8772 data: 0.1675 max mem: 34435
Epoch: [4] [ 70/121] eta: 0:00:49 lr: 0.000459 min_lr: 0.000002 loss: 0.0049 (0.0057) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0785 (0.0904) time: 0.7934 data: 0.0831 max mem: 34435
Epoch: [4] [ 80/121] eta: 0:00:39 lr: 0.000467 min_lr: 0.000002 loss: 0.0047 (0.0056) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0747 (0.0883) time: 0.7916 data: 0.0816 max mem: 34435
Epoch: [4] [ 90/121] eta: 0:00:29 lr: 0.000475 min_lr: 0.000002 loss: 0.0045 (0.0054) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0712 (0.0863) time: 0.8624 data: 0.1533 max mem: 34435
Epoch: [4] [100/121] eta: 0:00:19 lr: 0.000483 min_lr: 0.000002 loss: 0.0042 (0.0053) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0678 (0.0843) time: 0.8471 data: 0.1383 max mem: 34435
Epoch: [4] [110/121] eta: 0:00:10 lr: 0.000492 min_lr: 0.000002 loss: 0.0040 (0.0052) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0647 (0.0824) time: 0.8458 data: 0.1372 max mem: 34435
Epoch: [4] [120/121] eta: 0:00:00 lr: 0.000500 min_lr: 0.000002 loss: 0.0039 (0.0051) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0618 (0.0806) time: 0.7797 data: 0.0708 max mem: 34435
Epoch: [4] Total time: 0:01:50 (0.9124 s / it)
Averaged stats: lr: 0.000500 min_lr: 0.000002 loss: 0.0039 (0.0051) class_acc: 1.0000 (1.0000) loss_scale: 8192.0000 (8192.0000) weight_decay: 0.0500 (0.0500) grad_norm: 0.0618 (0.0806)
Val: [ 0/20] eta: 0:04:55 loss: 0.0037 (0.0037) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 14.7555 data: 14.4493 max mem: 34435
Val: [10/20] eta: 0:00:15 loss: 0.0037 (0.0037) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.5907 data: 1.3137 max mem: 34435
Val: [19/20] eta: 0:00:01 loss: 0.0037 (0.0037) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.2756 data: 1.0098 max mem: 34435
Val: Total time: 0:00:25 (1.2788 s / it)
* loss 0.004
Accuracy of the network on the 14718 val EEG: 0.00%
Test: [ 0/11] eta: 0:02:27 loss: 0.0037 (0.0037) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 13.4220 data: 13.1036 max mem: 34435
Test: [10/11] eta: 0:00:01 loss: 0.0037 (0.0037) accuracy: 0.0000 (0.0000) balanced_accuracy: 0.0000 (0.0000) pr_auc: 0.0000 (0.0000) roc_auc: 0.0000 (0.0000) time: 1.4606 data: 1.1914 max mem: 34435
Test: Total time: 0:00:16 (1.4663 s / it)
* loss 0.004
Accuracy of the network on the 14718 test EEG: 0.00%
Max accuracy val: 0.00%, max accuracy test: 0.00%
...
Traceback (most recent call last):
File "/root/autodl-tmp/LaBraM/run_labram_pretraining.py", line 28, in
import modeling_pretrain
File "/root/autodl-tmp/LaBraM/modeling_pretrain.py", line 16, in
from modeling_finetune import Block, _cfg, PatchEmbed, RelativePositionBias
ImportError: cannot import name 'RelativePositionBias' from 'modeling_finetune' (/root/autodl-tmp/LaBraM/modeling_finetune.py)
There is not 'RelativePositionBias' in modeling_finetune.py. How to solve?
Hi!
Thanks for the exciting work. I have a question regarding the preprocessing of the pertaining datasets.
In the readme, you say, "Notably, you can also write your own codes for preprocessing EEG data. Make sure that the preprocessing is consistent with that of our paper, that is, removing useless channels, filtering between 0.1 Hz and 75 Hz, notch filtering of 50 Hz, resampling to 200 Hz, and setting the unit to uV".
I'm wondering if you have some example code of precisely this type of preprocessing and if not, what do you mean by "removing useless channels," i.e., what is a useless channel?
我想对 LaBraM 在EEG数据上的编码向量进行分析(模型权重从labram-base.pth导入,之后就冻结,不需要作训练和微调)。
假设信号已完成0.1Hz-75Hz的带通滤波,50Hz的陷波滤波, 重采样至200Hz
输入:
sig.shape :(64, 2006060) # 64通道,200Hz采样率,一小时时长的EEG信号
输出:
LaBraM模型对该eeg的表征向量:V
V.shape :(64, n, V_dim)
64:通道数
n:EEG信号根据窗口宽度(1秒?),被分割为n=3600段
模型对信号的编码向量维度(好像是32维)
V = LaBraM( sig )
einops.EinopsError: Error while processing rearrange-reduction pattern "B N (A T) -> B N A T".
Input tensor shape: torch.Size([23, 2000]). Additional info: {'T': 200}.
Wrong shape: expected 3 dims. Received 2-dim tensor.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 790759) of binary: /d/miniconda3/envs/lb_labram/bin/python
The TUAB data set and labram-base.pth model used,
But the input and output do not match
How to solve this error? Thank you so much
class TemporalConv(nn.Module):
""" Image to Patch Embedding
"""
def __init__(self, in_chans=1, out_chans=8):
super().__init__()
self.conv1 = nn.Conv2d(in_chans, out_chans, kernel_size=(1, 15), stride=(1, 8), padding=(0, 7))
self.gelu1 = nn.GELU()
self.norm1 = nn.GroupNorm(4, out_chans)
self.conv2 = nn.Conv2d(out_chans, out_chans, kernel_size=(1, 3), padding=(0, 1))
self.gelu2 = nn.GELU()
self.norm2 = nn.GroupNorm(4, out_chans)
self.conv3 = nn.Conv2d(out_chans, out_chans, kernel_size=(1, 3), padding=(0, 1))
self.norm3 = nn.GroupNorm(4, out_chans)
self.gelu3 = nn.GELU()
def forward(self, x, **kwargs):
x = rearrange(x, 'B N A T -> B (N A) T')
B, NA, T = x.shape
x = x.unsqueeze(1)
x = self.gelu1(self.norm1(self.conv1(x)))
x = self.gelu2(self.norm2(self.conv2(x)))
x = self.gelu3(self.norm3(self.conv3(x)))
x = rearrange(x, 'B C NA T -> B NA (T C)')
return x
Hello. I am analyzing your code to utilize it, and in the forward method of the TemporalConv class in modeling_pretrain.py above, in the part where einops is rearranged, the input dimension is listed as 4-dimensional. I thought B is the batch size, N is the number of electrodes, and T is the sample length, but I couldn't figure out what A means. Also, I have a question about whether N is the number of electrodes because of A.
Hello, thanks so much for making your code available!
I'd like to embed EEG data using the model and wanted to check whether the following makes sense to you. Given a pretrained model, e.g. your provided base model, I'd avoid adding a classification head to NeuralTransformer and catch the output of self.forward_features(...). Given input signals of [batch_size, channels, samples] and default repo parameters, the model would then provide embeddings [batch_size, embed_dim].
Does that seem OK or would you recommend another approach?
Thanks for your help!
你好,谢谢你们分享如此精彩的项目,我仔细看了一下labram的代码,发现总览图中下半部分的模型就是neural tokenizer的模型,只是head不一样。也就是说在第一步训练tokenizer的时候就把位置编码和时间编码弄进去了。然后建模的时候是把单个patch变成feature,对所有patch一起建模的。我没想通的是为什么要在第二步训练下面这个LaBram模型的时候完全重新训练一个,把LaBram作为student模型。直接用第一步训好的neural tokenizer,加载这个权重,然后只换一个head不行吗?就像你们在第三步微调的时候那样,加载原有的模型权重,只重新训练head。真诚发问,希望可以得到回复。
Hello! I would like to keep the entire network frozen but only update the classification head. Could you please provide an idea about how I can freeze the model?
Thank you so much.
Hi! Thank you for your great work. I am trying to load the pre-trained base model to extract feature embeddings (no fine-tuning). When I load the model, I get the following error:
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for NeuralTransformerForMEM: Unexpected key(s) in state_dict: "logit_scale".
Any solution? Also, do you have any pointers like which python file should I run to extract the feature embeddings from the pre-trained model?
Hi Weibang,
Thanks for sharing your excellent work! There's one thing that I'm not super clear about. When preprocessing a dataset like TUEP, in which the number of channels ranges from 19 to 23, how did you deal with this variation so that different samples could be batched and dumped into hdf5 files for later use?
Thanks in advance.
Test: [7/8] eta: 0:00:06 loss: 1.3861 (1.3866) accuracy: 0.2396 (0.2500) balanced_accuracy: 0.2500 (0.2471) cohen_kappa: 0.0000 (0.0000) f1_weighted: 0.0926 (0.1013) time: 6.2649 data: 4.9884 max mem: 6735
Test: Total time: 0:00:51 (6.4069 s / it)
Hi again @935963004,
I want to check if you use these two functions in any part of your code, and I would like to ask if not it would be okay to erase both functions.
Line 399 in 522da1d
Line 439 in 522da1d
您好,感谢您的优秀工作!我最近在复现您的实验,在处理原始TUAB和TUEV数据到h5数据集时遇到一些问题,您提供的代码是读取.cnt文件而,make_TUAB.py 和 make_TUEV.py 处理后的文件是存到.pkl格式的,请问这个如何处理成h5 dataset呢?我理解pretrain 模型的输入都是h5 dataset格式的,提前感谢您的耐心回复!
Hello, our EEG data format is SEEG and we would like to ask if we can directly use the data preprocessing method of changing the paper. Thank you very much for your reply!
Hello! I get the following error while training the vqnsp.
raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'VQNSP' object has no attribute 'module'. Did you mean: 'modules'?
If I use "modules" instead of "module", then the code works. Is it required to update your code or am I missing something?
I'm trying to fine tune the model and it's resulting in the following error
Start training for 50 epochs
/opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [52,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed. /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [13,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed. /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [25,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed. /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [18,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed. /opt/conda/conda-bld/pytorch_1682343995622/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [0,0,0], thread: [60,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"
failed.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 3392161 closing signal SIGTERM ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 1 (pid: 3392162) of binary: /home/anaconda3/envs/labram/bin/python
Traceback (most recent call last):
File "/home/anaconda3/envs/labram/bin/torchrun", line 33, in
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
NVCC
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Nov_22_10:17:15_PST_2023
Cuda compilation tools, release 12.3, V12.3.107
Build cuda_12.3.r12.3/compiler.33567101_0
Trying to run it on a node with 4 RTX A5000
Hello, and thank you for the awesome work!
I got the following error which is (I think) related to distributed training while try to run your example on colab pro plus
Traceback (most recent call last):
File "/content/LaBraM/run_class_finetuning.py", line 565, in
main(opts, ds_init)
File "/content/LaBraM/run_class_finetuning.py", line 232, in main
utils.init_distributed_mode(args)
File "/content/LaBraM/utils.py", line 428, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 399, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
Traceback (most recent call last):
File "/content/LaBraM/run_class_finetuning.py", line 565, in
main(opts, ds_init)
File "/content/LaBraM/run_class_finetuning.py", line 232, in main
utils.init_distributed_mode(args)
File "/content/LaBraM/utils.py", line 428, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 399, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
| distributed init (rank 0): env://, gpu 0
Traceback (most recent call last):
File "/content/LaBraM/run_class_finetuning.py", line 565, in
main(opts, ds_init)
File "/content/LaBraM/run_class_finetuning.py", line 232, in main
utils.init_distributed_mode(args)
File "/content/LaBraM/utils.py", line 428, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 399, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
Traceback (most recent call last):
File "/content/LaBraM/run_class_finetuning.py", line 565, in
main(opts, ds_init)
File "/content/LaBraM/run_class_finetuning.py", line 232, in main
utils.init_distributed_mode(args)
File "/content/LaBraM/utils.py", line 428, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 399, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
Traceback (most recent call last):
File "/content/LaBraM/run_class_finetuning.py", line 565, in
main(opts, ds_init)
File "/content/LaBraM/run_class_finetuning.py", line 232, in main
utils.init_distributed_mode(args)
File "/content/LaBraM/utils.py", line 428, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 399, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
Traceback (most recent call last):
File "/content/LaBraM/run_class_finetuning.py", line 565, in
main(opts, ds_init)
File "/content/LaBraM/run_class_finetuning.py", line 232, in main
utils.init_distributed_mode(args)
File "/content/LaBraM/utils.py", line 428, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py", line 399, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
Root Cause (first observed failure):
[0]:
time : 2024-06-30_22:56:38
host : 7283cc30eeb0
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 20760)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Hi, @935963004, thanks for your work. I'm wondering if you are planning to release the checkpoints of LaBraM-Large and LaBraM-Huge?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.