Abstract

We propose an approach for dense semantic 3D recon-struction which uses a data term that is defined as po-tentials over viewing rays, combined with continuous sur-face area penalization. Our formulation is a convex re-laxation which we augment with a crucial non-convex con-straint that ensures exact handling of visibility. To tackle thnon-convex minimization problem, we propose a majorizeminimize type strategy which converges to a critical point. We demonstrate the benefits of using the non-convex con-straint experimentally. For the geometry-only case, we seta new state of the art on two datasets of the commonly used Middlebury multi-view stereo benchmark. Moreover, our general-purpose formulation directly reconstructs thin objects, which are usually treated with specialized algorithms. A qualitative evaluation on the dense semantic 3D reconstruction task shows that we improve significantly over previous methods.