Abstract
This paper establishes the existence of observable footprints that reveal the “causal dispositions” of the object
categories appearing in collections of images. We achieve
this goal in two steps. First, we take a learning approach to
observational causal discovery, and build a classifier that
achieves state-of-the-art performance on finding the causal
direction between pairs of random variables, given samples
from their joint distribution. Second, we use our causal direction classifier to effectively distinguish between features of
objects and features of their contexts in collections of static
images. Our experiments demonstrate the existence of a
relation between the direction of causality and the difference
between objects and their contexts, and by the same token,
the existence of observable signals that reveal the causal
dispositions of objects.