How to reproduce results on Marco about tevatron

[MXueguang]

Hi @WenzhengZhang,
In the paper, we use

a learning rate of 5e-6 with batch size 64 for 3 epochs

The batch size 64 here is 8 queries each batch x 8 documents each query.
i.e. the 8x8 setting in the github repo.

The following command should be able to reproduce the results (on 4x GPU):

If you want to train on single GPU, remove -m torch.distributed.launch --nproc_per_node=4 and negatives_x_device, and change per_device_train_batch_size to 8

python -m torch.distributed.launch --nproc_per_node=4 -m tevatron.driver.train \
  --output_dir ./retriever_model \
  --do_train \
  --model_name_or_path bert-base-uncased \
  --model_name_or_path bert-base-uncased \
  --dataset_name Tevatron/msmarco-passage \
  --save_steps 20000 \
  --q_max_len 16 \
  --p_max_len 128 \
  --fp16 \
  --per_device_train_batch_size 2 \
  --train_n_passages 8 \
  --learning_rate 5e-6 \
  --num_train_epochs 3 \
  --dataloader_num_workers 4 \
  --negatives_x_device

[WenzhengZhang]

Hi @WenzhengZhang, In the paper, we use

a learning rate of 5e-6 with batch size 64 for 3 epochs

The batch size 64 here is 8 queries each batch x 8 documents each query. i.e. the 8x8 setting in the github repo.

The following command should be able to reproduce the results (on 4x GPU):

If you want to train on single GPU, remove -m torch.distributed.launch --nproc_per_node=4 and negatives_x_device, and change per_device_train_batch_size to 8

python -m torch.distributed.launch --nproc_per_node=4 -m tevatron.driver.train \
  --output_dir ./retriever_model \
  --do_train \
  --model_name_or_path bert-base-uncased \
  --model_name_or_path bert-base-uncased \
  --dataset_name Tevatron/msmarco-passage \
  --save_steps 20000 \
  --q_max_len 16 \
  --p_max_len 128 \
  --fp16 \
  --per_device_train_batch_size 2 \
  --train_n_passages 8 \
  --learning_rate 5e-6 \
  --num_train_epochs 3 \
  --dataloader_num_workers 4 \
  --negatives_x_device

Hi @MXueguang ,
Thanks for your reply! Have you ever tried to set train_n_passages to be 2 for marco? Will only using one BM25 hard negative perform much worse than 7 BM25 hard negatives?

[MXueguang]

Hi @WenzhengZhang,
If using train_n_passages=2, it may require a larger batch size to get the level of MRR. (e,g, batch size=128)

[WenzhengZhang]

Thanks a lot!