Abstract. Robust data association is a core problem of visual odometry,
where image-to-image correspondences provide constraints for camera
pose and map estimation. Current state-of-the-art direct and indirect
methods use short-term tracking to obtain continuous frame-to-frame
constraints, while long-term constraints are established using loop closures. In this paper, we propose a novel visual semantic odometry (VSO)
framework to enable medium-term continuous tracking of points using
semantics. Our proposed framework can be easily integrated into existing direct and indirect visual odometry pipelines. Experiments on challenging real-world datasets demonstrate a significant improvement over
state-of-the-art baselines in the context of autonomous driving simply
by integrating our semantic constraints