Abstract An anaglyph is a single image created by selecting complementary colors from a stereo color pair; the user can perceive depth by viewing it through color-fifiltered glasses. We propose a technique to reconstruct the original color stereo pair given such an anaglyph. We modifified SIFT-Flow and use it to initially match the different color channels across the two views. Our technique then iteratively refifines the matches, selects the good matches (which defifines the “anchor” colors), and propagates the anchor colors. We use a diffusion-based technique for the color propagation, and added a step to suppress unwanted colors. Results on a variety of inputs demonstrate the robustness of our technique. We also extended our method to anaglyph videos by using optic flflow between time frames