Abstract

This paper presents a novel generalization of the optical flow equation to the case of refraction, and it describes a method for recover- ing the refractive structure of an ob ject from a video sequence acquired as the background behind the refracting ob ject moves. By structure here we mean a representation of how the ob ject warps and attenuates (or amplifies) the light passing through it. We distinguish between the cases when the background motion is known and unknown. We show that when the motion is unknown, the refractive structure can only be estimated up to a six-parameter family of solutions without additional sources of information. Methods for solving for the refractive structure are descri- bed in both cases. The performance of the algorithm is demonstrated on real data, and results of applying the estimated refractive structure to the task of environment matting and compositing are presented.