Comments (6)
@logan-markewich thank you for your comment!
My concerns were rather technical, due to the output of the decoder (i had experience with CNNs until now, there a finetuning was somewhat simpler) but in the meanwhile i learned much about transformers, NLP and pytorch-lightning, and managed to fine-tune on donut-proto on colab with custom small dataset!
from donut.
In my opinion, that will depend on your training data.
Do you have any overlap with your 8 classes vs. the existing 16? Is your dataset small? If so, I would try using the fine-tuned rvlcdip model. But you will run the risk of still predicting one of the existing 16 classes, but that can be handled with post-processing.
If you have a lot of training data, or your classes don't overlap, you are probably better off using donut-base
from donut.
@qustions i had success on very few examples! I went down to 50 / class and it showed already great accuracy.
Hint: If there is a specific word in the title of the page, make that word the class name (instead using arbitrary class names).
Consider using this repo to generate endless training samples: https://github.com/sparkfish/augraphy
from donut.
Hello @sandorkonya I am also trying to do the same I have small dataset
If possible could you please share the colab of donut fine-tune
from donut.
@qustions hi there,
i do the same as here: https://github.com/clovaai/donut#training and use the naver-clova-ix/donut-base model.
I advice putting the data on gDrive (you will have to restart and reinstall the runtime many times to get things working =) ) , i changed
--dataset_name_or_paths '["naver-clova-ix/cord-v2"]'
to
--dataset_name_or_paths ["../gdrive/MyDrive/folderyouhaveyourstuffin"]
note, the "gdrive/MyDrive/folderyouhaveyourstuffin" points to where your train/ and validation/ folders are.
i think the most difficult is however to generate the grounf truth data correctly,, one line from my groung truth file:
{"file_name": "1234.jpg", "ground_truth": "{"gt_parse": {"class": "myimageclass"}}"}
hope this helps
from donut.
@sandorkonya Thanks for sharing any idea on how many minimum images require for training
from donut.
Related Issues (20)
- Request: Dataset and pretrained model for language detection
- Problem with getting predictions HOT 7
- Does synthdog data has MiT or afl-3.0 license? HOT 1
- Error "A configuraton of type donut cannot be instantiated because not both `encoder` and `decoder` sub-configurations are passed" when run inference after finetuned docvqa without pushing to hugging face? HOT 1
- custom json schema - ASAP HOT 2
- Multi GPU support for fine tuning
- How to extract complete text from the document? HOT 3
- confidence 값의 공식적인 지원
- Classification inference
- Update donut-python Python Package to be compatible with latest versions of transformers
- donut inference시 sub task가 변경?
- Not getting prediction correctly using the model trained on the custom dataset (similar format as CORD-V2 dataset) HOT 6
- not work this app.py
- Can synthdog insert text for a specified bbox? HOT 1
- Where is the fine-tuned model?
- Why is the output of the intermediate verification empty after training?
- Donut generate ONLY <s><s>...<s></s> HOT 7
- Performance of the model HOT 1
- How to improve OCR accuracy for Japanese characters? HOT 2
- Early Stopping
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from donut.