Abstract Cross-modal transfer is helpful to enhance modalityspecifific discriminative power for scene recognition. To this end, this paper presents a unifified framework to integrate the tasks of cross-modal translation and modalityspecifific recognition, termed as Translate-to-Recognize Network (TRecgNet). Specififically, both translation and recognition tasks share the same encoder network, which allows to explicitly regularize the training of recognition task with the help of translation, and thus improve its fifinal generalization ability. For translation task, we place a decoder module on top of the encoder network and it is optimized with a new layer-wise semantic loss, while for recognition task, we use a linear classififier based on the feature embedding from encoder and its training is guided by the standard cross-entropy loss. In addition, our TRecgNet allows to exploit large numbers of unlabeled RGB-D data to train the translation task and thus improve the representation power of encoder network. Empirically, we verify that this new semi-supervised setting is able to further enhance the performance of recognition network. We perform experiments on two RGB-D scene recognition benchmarks: NYU Depth v2 and SUN RGB-D, demonstrating that TRecgNet achieves superior performance to the existing state-of-theart methods, especially for recognition solely based on a single modality.