Comments (6)
You can set fixed window size and shift it during the generation. Thus, you can manage the length to be under 1024.
from ardm.
Thus, you can manage the length to be under 1024.
Longer than 1024, you mean?
Also, would you be able to support this in your code, if it's not too much of an effort? I'm lacking the abilities to implement it myself.
from ardm.
You have a sampler to sample dialogs sequentially. I have attached a version that implements random sampling. You can just modify it.
class DialogFragmentSampler:
def __init__(self, max_len=1024):
"""Sample dialog fragments from a dialog
"""
self.max_tokens_len = max_len
def __call__(self, dialog):
"""dialog is a dict which has key "token_ids" and "text" with list of turns
"""
dialog_fragment = {}
lengths = np.array([len(item) for item in dialog['token_ids']])
# if the entire dialog is smaller than the max len
if lengths.sum() < self.max_tokens_len:
return dialog
cumsum_len = lengths.cumsum()
reverse_cumsum_len = cumsum_len[-1] - cumsum_len
# based on the reverse cumsum, we can have a range to select from
start_turns = np.arange(
len(reverse_cumsum_len))[reverse_cumsum_len > self.max_tokens_len]
# remove odd numbers
start_turns = [idx for idx in start_turns if idx % 2 == 0]
# randomly choose one
random_start_turn = random.choice(start_turns)
new_cumsum_len = cumsum_len - cumsum_len[random_start_turn]
# find the maximum end turn (only odd turn)
for i in reversed(range(len(new_cumsum_len))):
if i % 2 == 1 and new_cumsum_len[i] < self.max_tokens_len:
random_end_turn = i
break
dialog_fragment["text"] = dialog['text'][random_start_turn:
random_end_turn + 1]
dialog_fragment["token_ids"] = dialog['token_ids'][random_start_turn:
random_end_turn + 1]
return dialog_fragment
from ardm.
Isn't sampling for testing? I want the model to be able to learn to represent dialogs longer than 1024, not just generating. In any case, it seems like I'm inadequate to comprehend your code and what you're saying.
from ardm.
The current model can't encode more than 1024. You can try to apply the idea to models like XL-Net, which might solve your problem.
from ardm.
Gotcha, thanks a lot.
from ardm.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ardm.