Abstract
As the intermediate level task connecting image captioning and object detection, visual relationship detection
started to catch researchers’ attention because of its descriptive power and clear structure. It detects the objects
and captures their pair-wise interactions with a subjectpredicate-object triplet, e.g. hperson-ride-horsei. In this
paper, each visual relationship is considered as a phrase
with three components. We formulate the visual relationship
detection as three inter-connected recognition problems and
propose a Visual Phrase guided Convolutional Neural Network (ViP-CNN) to address them simultaneously. In ViPCNN, we present a Phrase-guided Message Passing Structure (PMPS) to establish the connection among relationship
components and help the model consider the three problems
jointly. Corresponding non-maximum suppression method
and model training strategy are also proposed. Experimental results show that our ViP-CNN outperforms the stateof-art method both in speed and accuracy. We further pretrain ViP-CNN on our cleansed Visual Genome Relationship dataset, which is found to perform better than the pretraining on the ImageNet for this task.