Finetuning Document classification about donut

[sandorkonya]

@logan-markewich thank you for your comment!

My concerns were rather technical, due to the output of the decoder (i had experience with CNNs until now, there a finetuning was somewhat simpler) but in the meanwhile i learned much about transformers, NLP and pytorch-lightning, and managed to fine-tune on donut-proto on colab with custom small dataset!

[logan-markewich]

In my opinion, that will depend on your training data.

Do you have any overlap with your 8 classes vs. the existing 16? Is your dataset small? If so, I would try using the fine-tuned rvlcdip model. But you will run the risk of still predicting one of the existing 16 classes, but that can be handled with post-processing.

If you have a lot of training data, or your classes don't overlap, you are probably better off using donut-base

[sandorkonya]

@qustions i had success on very few examples! I went down to 50 / class and it showed already great accuracy.
Hint: If there is a specific word in the title of the page, make that word the class name (instead using arbitrary class names).
Consider using this repo to generate endless training samples: https://github.com/sparkfish/augraphy

[qustions]

Hello @sandorkonya I am also trying to do the same I have small dataset
If possible could you please share the colab of donut fine-tune

[sandorkonya]

@qustions hi there,
i do the same as here: https://github.com/clovaai/donut#training and use the naver-clova-ix/donut-base model.
I advice putting the data on gDrive (you will have to restart and reinstall the runtime many times to get things working =) ) , i changed
--dataset_name_or_paths '["naver-clova-ix/cord-v2"]'
to
--dataset_name_or_paths ["../gdrive/MyDrive/folderyouhaveyourstuffin"]

note, the "gdrive/MyDrive/folderyouhaveyourstuffin" points to where your train/ and validation/ folders are.

i think the most difficult is however to generate the grounf truth data correctly,, one line from my groung truth file:
{"file_name": "1234.jpg", "ground_truth": "{"gt_parse": {"class": "myimageclass"}}"}

hope this helps

[qustions]

@sandorkonya Thanks for sharing any idea on how many minimum images require for training