Are You Smarter Than A Sixth Grader?
Textbook Question Answering for Multimodal Machine Comprehension
Abstract
We introduce the task of Multi-Modal Machine Comprehension (M3C), which aims at answering multimodal
questions given a context of text, diagrams and images.
We present the Textbook Question Answering (TQA) dataset
that includes 1,076 lessons and 26,260 multi-modal questions, taken from middle school science curricula. Our
analysis shows that a significant portion of questions require complex parsing of the text and the diagrams and reasoning, indicating that our dataset is more complex compared to previous machine comprehension and visual question answering datasets. We extend state-of-the-art methods for textual machine comprehension and visual question
answering to the TQA dataset. Our experiments show that
?The majority of the work was done while the author was interning at
the Allen Institute for Artificial Intelligence