Psycholinguistics meets Continual Learning:
Measuring Catastrophic Forgetting in Visual Question Answering
Abstract
We study the issue of catastrophic forgetting in
the context of neural multimodal approaches
to Visual Question Answering (VQA). Motivated by evidence from psycholinguistics, we
devise a set of linguistically-informed VQA
tasks, which differ by the types of questions
involved (Wh-questions and polar questions).
We test what impact task difficulty has on continual learning, and whether the order in which
a child acquires question types facilitates computational models. Our results show that dramatic forgetting is at play and that task diffi-
culty and order matter. Two well-known current continual learning methods mitigate the
problem only to a limiting degree