Abstract

We propose a novel three-layered approach for semantic seg- mentation of building facades. In the first layer, starting from an over- segmentation of a facade, we employ the recently introduced machine learning technique Recursive Neural Networks (RNN) to obtain a prob- abilistic interpretation of each segment. In the second layer, initial label- ing is augmented with the information coming from specialized facade component detectors. The information is merged using a Markov Ran- dom Field. In the third layer, we introduce weak architectural know ledge, which enforces the final reconstruction to be architecturally plausible and consistent. Rigorous tests performed on two existing datasets of building facades demonstrate that we significantly outperform the current-state of the art, even when using outputs from earlier layers of the pipeline. Also, we show how the final output of the third layer can be used to create a procedural reconstruction.