Abstract
In this paper, we exploit memory-augmented neural networks to predict accurate answers to visual questions, even
when those answers rarely occur in the training set. The
memory network incorporates both internal and external
memory blocks and selectively pays attention to each training exemplar. We show that memory-augmented neural networks are able to maintain a relatively long-term memory
of scarce training exemplars, which is important for visual
question answering due to the heavy-tailed distribution of
answers in a general VQA setting. Experimental results in
two large-scale benchmark datasets show the favorable performance of the proposed algorithm with the comparison to
state of the art.