Abstract

We propose a model to learn visually grounded word embeddings (vis-w2v) to capture visual notions of semantic relatedness. While word embeddings trained using text have been extremely successful, they cannot uncover notions of semantic relatedness implicit in our visual world. For in-stance, although “eats” and “stares at” seem unrelated in text, they share semantics visually. When people are eating something, they also tend to stare at the food. Groundingdiverse relations like “eats” and “stares at” into vision re-mains challenging, despite recent progress in vision. We note that the visual grounding of words depends on semantics, and not the literal pixels. We thus use abstract scenes created from clipart to provide the visual grounding. Wefind that the embeddings we learn capture fine-grained, vi-sually grounded notions of semantic relatedness. We show improvements over text-only word embeddings (word2vec) on three tasks: common-sense assertion classification, visual paraphrasing and text-based image retrieval. Our code and datasets are available online.