Abstract

towards Artificial General Intelligence. When presented in complex scenes, humans can quickly identifyelements across different scenes and infer relations between them. For example, when you are using a pile ofdifferent types of LEGO bricks to assemble a spaceship, you are actively inferring relations between eachLEGO brick, such as in what ways they can fit together. This type of abstract reasoning, particularly in thevisual domain, is a crucial key to human ability to build complex things.Many tests have been proposed to measure human ability for abstract reasoning. The most popular test in thevisual domain is the Raven Progressive Matrices (RPM) test (Raven (2000)). In the RPM test, the participantsare asked to view a sequence of contextual diagrams, usually given as a 3 × 3 matrices of diagrams with thebottom-right diagram left blank. Participants should infer abstract relationships in rows or columns of thediagram, and pick from a set of candidate answers the correct one to fill in the blank. Figures 1 (a) shows anexample of RPM tasks containing XOR relations across diagrams in rows. More examples can be found inAppendix C. Another widely used test for measuring reasoning in psychology is Diagram Syllogism task(Sato et al. (2015)), where participants need to infer conclusions based on 2 given premises. Figure 1c showsan example of Euler Diagram Syllogism task.Barrett et al. (2018) recently published a large and comprehensive RPM-style dataset named ProcedurallyGenerated Matrices ‘PGM’, and proposed Wild Relation Network (WReN), a state-of-the-art neural net forRPM-style tasks. While WReN outperforms other state-of-the-art vision models such as Residual Network Heet al. (2016), the performance is still far from deep neural nets’ performance on other vision or naturallanguage processing tasks. Recently, there has been a focus on object-level representations (Yi et al. (2018);Hu et al. (2017); Hudson & Manning (2018); Mao et al. (2019); Teney et al. (2017); Zellers et al. (2018)) for 1