Abstract

In this paper, we study the challenging problem of pre-dicting the dynamics of objects in static images. Given a query object in an image, our goal is to provide a physi-cal understanding of the object in terms of the forces actingupon it and its long term motion as response to those forces.Direct and explicit estimation of the forces and the motionof objects from a single image is extremely challenging. Wedefine intermediate physical abstractions called Newtonianscenarios and introduce Newtonian Neural Network (N 3 )that learns to map a single image to a state in a Newto-nian scenario. Our evaluations show that our method can reliably predict dynamics of a query object from a single image. In addition, our approach can provide physical reasoning that supports the predicted dynamics in terms of velocity and force vectors. To spur research in this direction we compiled Visual Newtonian Dynamics (VIND) dataset that includes more than 6000 videos aligned with Newtonian scenarios represented using game engines, and morethan 4500 still images with their ground truth dynamics.