Abstract. Images captured by fifisheye lenses violate the pinhole camera assumption and suffffer from distortions. Rectifification of fifisheye images is therefore a crucial preprocessing step for many computer vision applications. In this paper, we propose an end-to-end multi-context collaborative deep network for removing distortions from single fifisheye images. In contrast to conventional approaches, which focus on extracting hand-crafted features from input images, our method learns high-level semantics and low-level appearance features simultaneously to estimate the distortion parameters. To facilitate training, we construct a synthesized dataset that covers various scenes and distortion parameter settings. Experiments on both synthesized and real-world datasets show that the proposed model signifificantly outperforms current state of the art methods. Our code and synthesized dataset will be made publicly available