Abstract

Our goal is to recognize material categories using im-ages and geometry information. In many applications, suchas construction management, coarse geometry informationis available. We investigate how 3D geometry (surface nor-mals, camera intrinsic and extrinsic parameters) can beused with 2D features (texture and color) to improve ma-terial classification. We introduce a new dataset, GeoMat,which is the first to provide both image and geometry datain the form of: (i) training and testing patches that were ex-tracted at different scales and perspectives from real worldexamples of each material category, and (ii) a large scaleconstruction site scene that includes 160 images and over800,000 hand labeled 3D points. Our results show thatusing 2D and 3D features both jointly and independentlyto model materials improves classification accuracy acrossmultiple scales and viewing directions for both materialpatches and images of a large scale construction site scene.