Comments (6)
We initialize the encoder and decoder weights with pretrained model weights. Then we train on a data mix including PMC with simple non-markup targets for layout diversity. So yes, it is a simple OCR pretraining objective. In the following step the IDL data is removed from the training.
from nougat.
yes, sounds right. With most of the weight on arxiv since it is the cleanest source. PMC's math is not always in a parsable format (eg images) or inline math is just italic
from nougat.
Thanks a ton, this is helpful!
from nougat.
@lukas-blecher one more thing, this data mix that you mention from what I understood it has PMC and IDL data and this simple non-markup data and when you want to train the model completely you use PMC + Arxiv with full markup abilities that are generated by the dataset generation code?
from nougat.
@lukas-blecher Thank you for your response but I am still unclear about the whole procedure for pretraining the model
- If the loss function is the same, why two stages? If the loss function is not the same, then what is the objectives for 1st stage and what is for the 2nd stage?
- If two stages, what is the training schedule? In the paper, you only mentioned training the model for 3 epochs. Is that for pretraining (stage 1) or training (stage 2)
- For pretraining, what sources do your data contain, is it Arxiv + PMC + IDL? and do the papers used in pretraining for example from Arxiv used again in training?
- For pretraining, you mentioned only use non-markup data. Does that mean you use masking to mask out the markup data to compute the losses, and if not for no-markup do you have a simple script which only keeps pages that have no-markup for training, and is this script run on the .mmd files generated by the dataset generation script?
- Finally for training (stage 2) are some PMC files removed based on some criteria which is applied to the .mmd file, for cases where PMC's math is not always parsable?
from nougat.
Also one more question. I looked into the IDL data the JSON object returned for each file, do you convert it into a text document by parsing the JSON response?
from nougat.
Related Issues (20)
- How to set the train_nougat.yaml to finetune the nougat base model ?
- Please add Rocm support HOT 1
- Cannot close object, library is destroyed. HOT 1
- Having issue on -> Cannot close object, library is destroyed. This may cause a memory leak! HOT 9
- Recompute option in nougat_api HOT 1
- Have anybody trained a Chinese version?
- Why citations come out like this?
- Why can't I run nougat-ocr on pdfs? HOT 1
- What does this issue mean?
- TypeError: BARTDecoder.prepare_inputs_for_inference() got an unexpected keyword argument 'cache_position' HOT 11
- Big Bug: Can not detect some kinds of whitespace in arxiv computer science paper.
- UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3587.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] and TypeError: BARTDecoder.prepare_inputs_for_inference() got an unexpected keyword argument 'cache_position' -> Cannot close object, library is destroyed. This may cause a memory leak! HOT 4
- Questions Regarding the Nougat Model's Pre-training Process
- pydantic error HOT 8
- The command to generate mmd from pdf, the output result is empty
- TypeError: BARTDecoder HOT 2
- Low amount of recognised pages
- pydantic_core._pydantic_core.ValidationError: 1 validation error for InitSchema | Windows, Python 3.11.5 HOT 2
- Training set format problem
- nought项目可以下载直接部署到服务器对外暴露接口提供使用吗
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nougat.