t5_finetune's Introduction

A simple example for finetuning T5.

To set up, you'll need to pip install transformers and all that normal stuff.

The important details:

Dataset

You'll have a dataset of some sort mapping source => target. To use this you'll want to set up a data_dir (which you will specify in train.py) to have

train.source
train.target
val.source
val.target
test.source
test.target

Where each source/train pair has one example per line. So, e.g., for QA you would have

questions in .source
answers in .target
with one line for each of them

You can generate a split using sklearn

Config

You'll want to tweak the k_* parameters at the top of train.py

Tensorboard

To run tensorboard, just pip install tensorboard and then tensorboard --logdir=

Notes

Test is not implemented, so if you want to test on a holdout dataset, you'll want to tweak the code to generate a dataset on test.source and test.target and evaluate the metrics you want.
If your task is similar to one of the originally trained tasks like summarization, you might benefit from prepending a task label to your inputs, like "summarize: " or "translate English to Russian: "

t5_finetune's People

Contributors

Stargazers

Watchers

t5_finetune's Issues

Partially generating output text

Hi ,
I'm use this T5 model finetune for text generation task.

I have a issue of partially generating the text.

for example this is the generated text:
"< pad > Kasun has 7 books and gave Nimal 2 of the books. How many book did Ka"

This is the full output.

This is not the full sentence and not end with < /s > token.
Output are maximum 17 words. Giving model.generate( . . . , min_length=30 ) is also not working.

(only small amount of word contains sentence are giving full output sentence)
How can I fix this issue?
Thank you.