Task | Understanding Paragraphs & Document Layout Analysis about donut

[logan-markewich]

Just an idea, but you could probably model document hierarchy by adding special tokens to the tokenizer.

For example, DocVQA is trained using an input like '<s_docvqa><s_question>my question?</s_question><s_answer>', and the output then completes the input -> '<s_docvqa><s_question>my question?</s_question><s_answer>my answer</s_answer></s_docvqa>'

So following that logic, you could construct an input prompt as simply '<s_hierarchy>' and then the output could be '<s_hierarchy><s_title>My title<s_paragraph>My paragraph text</s_paragraph></s_title></s_hierarchy>'

This would work for training, as long as you add the proper special tokens.

In terms of the token limit though, you would definitely run into some problems. If you cared more about the structure, maybe you could just predict the first line of each hierarchical element to limit the prediction size. If you are working with synthetically generated data, this would be pretty easy to do!

In terms of doing DLA, by outputting the hierarchy you are already performing a type of layout analysis. Adapting existing datasets might be tough though, but maybe check out DocBank. They have token-level layout annotations that would work well for a text based model like this one

[jordanparker6]

@logan-markewich I like your approach.

What's your thoughts on replacing the BART encoder with one of the long-rage Transformer architecture (e.g. LongFormer or Performer)?

I don't have much experience with them but I understand that have O(n) complexity instead of O(n2) with the sequence length.

[logan-markewich]

@jordanparker6 I don't have any experience using long-range transformers, but it's probably worth trying! Assuming you have the resources needed to load and train the model 👍🏻

Thankfully looking at the code, swapping out BART for anything else should be pretty straightforward, and you won't have to repeat the OCR pre-training either.

[jlia0]

Any updates on this? Why is Donut difficult for Document Layout Analysis task?