yuanbit / finbert-qa-notebooks

Notebooks for fine-tuning a BERT model and training a LSTM model for financial QA

FinBERT-QA-notebooks

This repo includes notebooks for training a QA-LSTM model and fine-tuning a pre-trained BERT model for the Opionated Financial Question and Answering FiQA dataset.

Colab

The notebooks can be opened in Colab which provides a low-effort way (no installation and access to free GPU) to train and evaluate the models.

• FinBERT-QA: [Colab] [Github]
• Further pre-training of BERT on FiQA: [Colab] [Github]
• QA-LSTM: [Colab] [Github]

Click "Open in Playground" to replicate the results!

finbert-qa-notebooks's Issues

Cannot reproduce the numbers on the paper with this Notebook

Hey! Thanks for open-sourcing this amazing project!
Just a quick question, I folllowed strictly with this notebook FinBERT_QA.ipynb and cannot reproduce the numbers reported on the paper.
More specificly, with using bert-qa as the starting point, after

config = {'bert_model_name': bert-qa,
      'max_seq_len': 512,
      'batch_size': 16,
      'learning_rate': 3e-6,
      'weight_decay': 0.01,
      'n_epochs': 3,
      'num_warmup_steps': 10000}

The results I had was
Epoch 2:

Train Loss: 0.069 | Train Accuracy: 98.39%
Validation Loss: 0.089 | Validation Accuracy: 98.09%
Average nDCG@10 for 333 queries: 0.476
MRR@10 for 333 queries: 0.442
Average Precision@1 for 333 queries: 0.381

Epoch 3:

Train Loss: 0.055 | Train Accuracy: 98.75%
Validation Loss: 0.097 | Validation Accuracy: 98.1%

Average nDCG@10 for 333 queries: 0.471
MRR@10 for 333 queries: 0.427
Average Precision@1 for 333 queries: 0.357

but the reported nDCG@10 should be 0.481.

Any ideas/suggestions? Thank you!