Abstract Video-based person re-identifification (re-id) is an important application in practice. However, only a few methods have been presented for this problem. Since large variations exist between different pedestrian videos, as well as within each video, it’s challenging to conduct re-identifification between pedestrian videos. In this paper, we propose a simultaneous intra-video and inter-video distance learning (SI2DL) approach for video-based person re-id. Specififically, SI2DL simultaneously learns an intra-video distance metric and an inter-video distance metric from the training videos. The intravideo distance metric is to make each video more compact, and the inter-video one is to make that the distance between two truly matching videos is smaller than that between two wrong matching videos. To enhance the discriminability of learned metrics, we design a video relationship model, i.e., video triplet, for SI2DL. Experiments on the public iLIDS-VID and PRID 2011 image sequence datasets show that our approach achieves the stateof-the-art performance.