Abstract. We present a deep model that can accurately produce dense
depth maps given an RGB image with known depth at a very sparse
set of pixels. The model works simultaneously for both indoor/outdoor
scenes and produces state-of-the-art dense depth maps at nearly realtime speeds on both the NYUv2 and KITTI datasets. We surpass the
state-of-the-art for monocular depth estimation even with depth values
for only 1 out of every ? 10000 image pixels, and we outperform other
sparse-to-dense depth methods at all sparsity levels. With depth values for 1/256 of the image pixels, we achieve a mean error of less than
1% of actual depth on indoor scenes, comparable to the performance of
consumer-grade depth sensor hardware. Our experiments demonstrate
that it would indeed be possible to efficiently transform sparse depth
measurements obtained using e.g. lower-power depth sensors or SLAM
systems into high-quality dense depth maps