isabella232 / begin-dataset

A benchmark dataset for evaluating dialog system and natural language generation metrics.

BEGIN

https://github.com/google/BEGIN-dataset

A benchmark dataset for evaluating dialog system and natural language generation metrics. See our paper "Evaluating Groundedness in Dialogue Systems: The BEGIN Benchmark" at https://arxiv.org/abs/2105.00071 .

This is not an officially supported Google product. This means that Google may not regularly release updates or provide support in conjunction with your use of this dataset.

Authors

• Nouha Dziri
• Hannah Rashkin [email protected]
• Tal Linzen
• David Reitter [email protected]

Versions

• Example release 6/1/2021: Only examples included
• First full release 6/1/2021

Documentation

Note that the dataset has been revised since we wrote the preprint below. Reproducing the work, you will find small differences in your metrics compared to the preprint. We will update the paper at a later point in time. That said, please refer to the following preprint for documentation:

Nouha Dziri, Hannah Rashkin, Tal Linzen, David Reitter. Evaluating Groundedness in Dialogue Systems: The BEGIN Benchmark. https://arxiv.org/abs/2105.00071 [Submitted on 30 Apr 2021]

Format

The format of the columns is:

• evidence: The selected document span (coming from Wizard of Wikipedia (Dinan et. al 2019))
• previous turn: The previous utterance in the conversation (coming from Wizard of Wikipedia (Dinan et. al 2019))
• response: A response generated by a large LM (either GPT2 or T5)
• gold label: An aggregated label from human annotators which is either: entailment, contradiction, hallucination, generic, or off-topic (for five-way classification)
• coarse label: An aggregated label from human annotators which is either: entailment, contradiction, or neutral (for three-way classification)
• full label set: The set of annotations from humans as a ";" delimited segment with annotations for whether the response is:
  • "generic"/"not-generic" - whether it was labelled as generic or not,
  • "off-topic"/"on-topic" - whether it was labelled as off-topic or not,
  • "cooperative"/"uncooperative" - whether it was labelled as cooperative or not,
  • "not-info/faithful/contradiction/hallucination-personal/hallucination-info" - whether it was intended to be informative and how faithful and supported the information is with respect to the evidence. We distinguish between "not-info" (examples that are not marked specifically as intending to be informative), "hallucination-personal" (examples that are marked as containing personal/subjective experiences or opinions) and "hallucination-info" (unsupported information)