vita-group / bert-tickets Goto Github PK

May I know the transformer library version that was used for this work? I couldn't find it in the project readme file and the latest version raises many errors. Thanks.

Hi, I couldn't find information about the dataset for running (IMP)MLM. Is it WikiText-2 or WikiText-103?

Hi,
Thanks for sharing your codebase.
I wonder what exactly is doing the following part of your code as "pruning_model_custom".

Hello,

Thanks for this repo. I am trying to extract the GraSP code from:
https://github.com/VITA-Group/BERT-Tickets/blob/master/pretrain_grasp.py

But in https://github.com/VITA-Group/BERT-Tickets/blob/master/pretrain_grasp.py#L431 there is no import or definition of this function inside the .py. I mean, I have not tried to run the file but if it is not declared or imported from anywhere I imagine that it is not going to work.

Here you have commented the imports. Should I uncomment this line to import the pruning_model_custom, see_weight_rate, and pruning_model?

Is for this reason that I don't like to do absolute imports with * because you use to loose the trace. :(

Big anticipated thanks.
Best regards

I'm afraid that the rewinding doesn't work, because the code uses an assignment other than torch.clone function. It means the original weights is lost and would be optimised when training.

Hello,

I have extracted the GraSP algorithm from here:
https://github.com/VITA-Group/BERT-Tickets/blob/master/transformers-master/examples/GraSP.py

I am getting CUDA out of memory on whatever size my architecture have. I have migrated your version of transformers to the new one.

Here is my trainer.py script:
https://gist.github.com/gaceladri/f301f33779785c401c8c1e549bcc1144

I have just adapted your pretrain_grasp.py to the trainer.py script on Transformers 3.3.1

The error comes at the backward pass in the (2) iteration:

Could it be that we are doing the backward pass with the main model loaded and we are duplicating the weights?

Thanks

Hello :-)

I found your paper very interesting and would like to play with the sparse models that it produced. Are you going to open-source them soon?

The implementation doesn't seem to follow the description in the paper:

Learning rate decays linearly from initial value to zero

When running the LT_glue.py command shown in README, the script sets --num_train_epochs as 30 and it means training each pruned subnetwork takes 3 epochs (and the [0%, 10%, ...90%]-pruned subnets take 30 epochs in total).
However, the learning rate scheduler depends on the --num_train_epochs (30) rather than 3:

t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs
...
scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
)

So, the last learning rate after 3 epochs is 1.8e-5 (= 2e-5 * (1 - 3/30)) (rather than zero) if we start from 2e-5.
Is my understanding correct?

Sure this minor difference doesn't hurt the great contributions of the work. I just wanna confirm it and know the detail for reproduction and my own further research. Thanks!