Abstract
We present the first end-to-end approach for real-time material estimation for general object shapes with uniform material that only requires a single color image as input. In
addition to Lambertian surface properties, our approach
fully automatically computes the specular albedo, material
shininess, and a foreground segmentation. We tackle this challenging and ill-posed inverse rendering problem using recent
advances in image-to-image translation techniques based
on deep convolutional encoder–decoder architectures. The
underlying core representations of our approach are specular
shading, diffuse shading and mirror images, which allow to
learn the effective and accurate separation of diffuse and specular albedo. In addition, we propose a novel highly efficient
perceptual rendering loss that mimics real-world image formation and obtains intermediate results even during run time.
The estimation of material parameters at real-time frame rates
enables exciting mixed-reality applications, such as seamless
illumination-consistent integration of virtual objects into realworld scenes, and virtual material cloning. We demonstrate
our approach in a live setup, compare it to the state of the art,
and demonstrate its effectiveness through quantitative and
qualitative evaluation