yale-lily / dart Goto Github PK
View Code? Open in Web Editor NEWDataset for NAACL 2021 paper: "DART: Open-Domain Structured Data Record to Text Generation"
License: MIT License
Dataset for NAACL 2021 paper: "DART: Open-Domain Structured Data Record to Text Generation"
License: MIT License
Hi ! ๐
Great data augmentation you've done here!
I have noticed missing lexicalizations/annotations for some entries in your test set . (For example, eid=Id1392 in your dart-v1.1.1-full-test.xml file).
Do you plan to add those ?
Thanks!
Recently, I used GPT to do generation with DART dataset. However, I found that the test set may be different from other works. In fact, I can only get 5,097 samples for testing, while GEM website says their test set is 12,552. And the data provied in (Li, et al 2021) (https://github.com/XiangLi1999/PrefixTuning) also has 12,552 samples but they do not provide gold references.
Through the official evaluation scripts and test set, I obtain about 37-38 BLEU, which is much lower than the results (46-47 BLEU) reported by (Li, et al 2021) and other works (like the leaderboard in github: https://github.com/Yale-LILY/dart). So, I am confused that which one is right.
Could you please answer these questions if possible? I will be appreciate.
Reference
I noticed that the generate_input_dart.py only used 3 references for evaluation. However, some examples have many more references. I was wondering if you could provide more details about the results in the paper. Can't seem to replicate your results. Also if you can share the fine-tuned T5 and Bart model on dart this would be very helpful.
The references in /evaluation/dart_reference are not for the current version. Can you replace with the new references and share the tokenization script that is done to the predictions.
I am getting very different BLEU scores depending on tokenization, and how many references I use.
As there are up to ~30 for a few examples.
I would like to directly compare against the README leaderboard.
The example number of Dart declaimed in Readme file and paper is 82,191, while the actual number of training set in file dart-v1.1.1-full-train.json is 30,526 which is much smaller than 82,191.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.