Comments (4)
Just an idea, but you could probably model document hierarchy by adding special tokens to the tokenizer.
For example, DocVQA is trained using an input like '<s_docvqa><s_question>my question?</s_question><s_answer>', and the output then completes the input -> '<s_docvqa><s_question>my question?</s_question><s_answer>my answer</s_answer></s_docvqa>'
So following that logic, you could construct an input prompt as simply '<s_hierarchy>' and then the output could be '<s_hierarchy><s_title>My title<s_paragraph>My paragraph text</s_paragraph></s_title></s_hierarchy>'
This would work for training, as long as you add the proper special tokens.
In terms of the token limit though, you would definitely run into some problems. If you cared more about the structure, maybe you could just predict the first line of each hierarchical element to limit the prediction size. If you are working with synthetically generated data, this would be pretty easy to do!
In terms of doing DLA, by outputting the hierarchy you are already performing a type of layout analysis. Adapting existing datasets might be tough though, but maybe check out DocBank. They have token-level layout annotations that would work well for a text based model like this one
from donut.
@logan-markewich I like your approach.
What's your thoughts on replacing the BART encoder with one of the long-rage Transformer architecture (e.g. LongFormer or Performer)?
I don't have much experience with them but I understand that have O(n) complexity instead of O(n2) with the sequence length.
from donut.
@jordanparker6 I don't have any experience using long-range transformers, but it's probably worth trying! Assuming you have the resources needed to load and train the model ๐๐ป
Thankfully looking at the code, swapping out BART for anything else should be pretty straightforward, and you won't have to repeat the OCR pre-training either.
from donut.
Any updates on this? Why is Donut difficult for Document Layout Analysis task?
from donut.
Related Issues (20)
- Error "A configuraton of type donut cannot be instantiated because not both `encoder` and `decoder` sub-configurations are passed" when run inference after finetuned docvqa without pushing to hugging face? HOT 1
- custom json schema - ASAP HOT 2
- Multi GPU support for fine tuning
- How to extract complete text from the document? HOT 3
- confidence ๊ฐ์ ๊ณต์์ ์ธ ์ง์
- Classification inference
- Update donut-python Python Package to be compatible with latest versions of transformers
- donut inference์ sub task๊ฐ ๋ณ๊ฒฝ?
- Not getting prediction correctly using the model trained on the custom dataset (similar format as CORD-V2 dataset) HOT 6
- not work this app.py
- Can synthdog insert text for a specified bbox? HOT 1
- Where is the fine-tuned model?
- Why is the output of the intermediate verification empty after training?
- Donut generate ONLY <s><s>...<s></s> HOT 7
- Performance of the model HOT 1
- How to improve OCR accuracy for Japanese characters? HOT 2
- Early Stopping
- How many documents(invoices) are required for training model for document information extraction?
- What should be the configuration of the machine to train the model?
- Hindi Synthdog
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from donut.