youjiangxu / movieqa_benchmark

Benchmark data and code for Question-Answering on Movie stories

MovieQA

MovieQA: Understanding Stories in Movies through Question-Answering
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, June 2016.
Project page | arXiv preprint | Read the paper | Explore the data

---

Benchmark Data

The data is made available in simple JSON / text files for easy access in any environment. We provide Python scripts to help you get started by providing simple data loaders.

To obtain access to the stories, and evaluate new approaches on the test data, please register at our benchmark website.

Python data loader

import MovieQA
mqa = MovieQA.DataLoader()

Explore

Movies are indexed using their corresponding IMDb keys. For example
mqa.pprint_movie(mqa.movies_map['tt0133093'])

QAs are stored as a standard Python list
mqa.pprint_qa(mqa.qa_list[0])

Use

Get the list of movies in a particular split, use
movie_list = mqa.get_split_movies(split='train')

To get train or test splits of the QA along with a particular story, use
story, qa = mqa.get_story_qa_data('train', 'plot')

Supported splits are: train, val, test, full and story forms are: plot, subtitle, dvs, script

Video lists can be obtained per QA, or per movie using
vl_qa, _ = get_video_list('train', 'qa_clips') % per QA
vl_movie, _ = get_video_list('train', 'all_clips') % per movie

Build your own data/story loaders

We provide a simple interface to load all the data (QAs, movies) and stories through the code above. If you wish to modify something, you are welcome to use your own data loaders and access the raw data directly. The evaluation server submissions are simple text files (explained after login) and are independent of any data loaders.

---

qa.json

• qid: A unique id for every question. Also indicates, train|val|test sets
• imdb_key: The movie this question belongs to
• question: The question string
• answers: The five answer options
• correct_index: Correct answer option (indexed by 0)
• plot_alignment: split_plot file line numbers, to which this question corresponds
• video_clips: Clips that are aligned with the question, to be used for answering

movies.json

• imdb_key: A unique id for every movie. Corresponds to that used by IMDb
• name: Movie title
• year: Movie release year
• genre: Movie genre classification
• text: Text sources that are available for that movie

---

Data Releaselog

• 2017.01.14: Alignments between question and plot sentence, plot sentence and video clips
• 2016.11.08: Patch for 65 missing video clips
• 2016.09.10: Video meta-data released: Shot boundaries, frame-timestamp correspondence
• 2016.04.06: Removed missing video clips from qa.json
• 2016.03.30: v1.0 data release

---

Requirements

• numpy
• pysrt