Abstract In this paper, we propose a novel method to improve object recognition accuracies of convolutional neural networks (CNNs) by embedding the proposed Min-Max objective into a high layer of the models during the training process. The MinMax objective explicitly enforces the learned object feature maps to have the minimum compactness for each object manifold and the maximum margin between different object manifolds. The Min-Max objective can be universally applied to different CNN models with negligible additional computation cost. Experiments with shallow and deep models on four benchmark datasets including CIFAR10, CIFAR-100, SVHN and MNIST demonstrate that CNN models trained with the Min-Max objective achieve remarkable performance improvements compared to the corresponding baseline models.