Abstract

We propose an adaptive multi-resolution formulation of se-mantic 3D reconstruction. Given a set of images of a scene,semantic 3D reconstruction aims to densely reconstructboth the 3D shape of the scene and a segmentation intosemantic object classes. Jointly reasoning about shape andclass allows one to take into account class-specific shapepriors (e.g., building walls should be smooth and vertical,and vice versa smooth, vertical surfaces are likely to bebuilding walls), leading to improved reconstruction results.So far, semantic 3D reconstruction methods have beenlimited to small scenes and low resolution, because of theirlarge memory footprint and computational cost. To scalethem up to large scenes, we propose a hierarchical scheme which refines the reconstruction only in regions that are likely to contain a surface, exploiting the fact that both highspatial resolution and high numerical precision are only required in those regions. Our scheme amounts to solving a sequence of convex optimizations while progressively removing constraints, in such a way that the energy, in each iteration, is the tightest possible approximation of the underlying energy at full resolution. In our experiments the method saves up to 98% memory and 95% computation time, without any loss of accuracy.