Abstract For object detection, the two-stage approach (e.g., Faster R-CNN) has been achieving the highest accuracy, whereas the one-stage approach (e.g., SSD) has the advantage of high effificiency. To inherit the merits of both while overcoming their disadvantages, in this paper, we propose a novel single-shot based detector, called RefifineDet, that achieves better accuracy than two-stage methods and maintains comparable effificiency of one-stage methods. Re- fifineDet consists of two inter-connected modules, namely, the anchor refifinement module and the object detection module. Specififically, the former aims to (1) fifilter out negative anchors to reduce search space for the classififier, and (2) coarsely adjust the locations and sizes of anchors to provide better initialization for the subsequent regressor. The latter module takes the refifined anchors as the input from the former to further improve the regression accuracy and predict multi-class label. Meanwhile, we design a transfer connection block to transfer the features in the anchor refifinement module to predict locations, sizes and class labels of objects in the object detection module. The multitask loss function enables us to train the whole network in an end-to-end way. Extensive experiments on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO demonstrate that RefifineDet achieves state-of-the-art detection accuracy with high effificiency. Code is available at https: //github.com/sfzhang15/RefineDet