How can I download the dataset? about metaseq

[stephenroller]

I'll post my post processing changes soon.

[DrMatters]

I'm particularly interested in how you concatenated messages of the Pushshift dataset (what delimiter did you use e.g.), as I'm trying to get a hand on fine-tuning OPT as a dialogue model.
It's not about how you flattened it, but more about the final format of data.

[corlder]

Hello, thank you so much for your excellent work. I want to reproduce the training results. Would you mind providing the complete data preprocessing scripts?

[suchenzang]

Since the datasets are already publicly available, we aren't hosting and offering the dataset for download.

[rtaori]

I see, thanks for the quick reply.

Section 2.3 of the papaer mentions some filtering & preprocessing done for each of the corpora. Are these scripts released? Where can I find them?

[rtaori]

Great, thanks so much! Looking forward!

[stephenroller]

I'm particularly interested in how you concatenated messages of the Pushshift dataset (what delimiter did you use e.g.), as I'm trying to get a hand on fine-tuning OPT as a dialogue model.

It's not about how you flattened it, but more about the final format of data.

One message per line. I wish I had done better.

[stephenroller]

https://gist.github.com/stephenroller/8738a3e4fcbeae23dc5dbb87c8745d87 regex processing here