Abstract Binarized convolutional neural networks (BCNNs) are widely used to improve memory and computation effificiency of deep convolutional neural networks (DCNNs) for mobile and AI chips based applications. However, current BCNNs are not able to fully explore their corresponding full-precision models, causing a signifificant performance gap between them. In this paper, we propose rectifified binary convolutional networks (RBCNs), towards optimized BCNNs, by combining full-precision kernels and feature maps to rectify the binarization process in a unifified framework. In particular, we use a GAN to train the 1-bit binary network with the guidance of its corresponding full-precision model, which signifificantly improves the performance of BCNNs. The rectifified convolutional layers are generic and flflexible, and can be easily incorporated into existing DCNNs such as WideResNets and ResNets. Extensive experiments demonstrate the superior performance of the proposed RBCNs over state-of-the-art BCNNs. In particular, our method shows strong generalization on the object tracking task