Comments (4)
-
Since GPT2 only supports sequences shorter 1024 tokens, this line skips dialogs longer than that. Though depending on your dataset, you can use a fix-sized window to extract dialogs.
-
Yes. It is set by the maximum length. If your dialog is longer than that, you can use a window to sample from the dialogs.
from ardm.
Thanks for answering.
-
Does it check if there’s a dialogue longer than 1024 in the batch, and if so, it skips the entire batch? I thought it’s this way since I saw my GPU not being utilized fully even on 128 batch size. Or is it just skipping per dialogue? Or is checking the sum of all the length of dialogues in a batch and then skipping them all if that’s higher than 1024?
-
What’s the 100 and the 1500 constant numbers represent then?
Please, excuse my lack of understanding. Thanks again.
from ardm.
The batch size is forced to be 1 now, which probably consumes all the memory of 11GB GPU. However, I have included batch support in the actual implementation. You can check out camrest and multiwoz. If there is a dialogue longer than 1024, it will skip the dialogue (batch size=1).
100 means dataset split. The first 100 dialogues are used for validation. 1500 is maximum length of your dialogue. It is only used for the tokenizer to tokenize your dialogue.
from ardm.
Okay, thanks a lot!
from ardm.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ardm.