Abstract This paper presents a novel adaptively connected neural network (ACNet) to improve the traditional convolutional neural networks (CNNs) in two aspects. First, ACNet employs a flflexible way to switch global and local inference in processing the internal feature representations by adaptively determining the connection status among the feature nodes (e.g., pixels of the feature maps) 1 . We can show that existing CNNs, the classical multilayer perceptron (MLP), and the recently proposed non-local network (NLN) [48] are all special cases of ACNet. Second, ACNet is also capable of handling non-Euclidean data. Extensive experimental analyses on a variety of benchmarks (i.e., ImageNet-1k classififi- cation, COCO 2017 detection and segmentation, CUHK03 person re-identifification, CIFAR analysis, and Cora document categorization) demonstrate that ACNet cannot only achieve state-of-the-art performance but also overcome the limitation of the conventional MLP and CNN 2 . The code is available at https://github.com/wanggrun/ Adaptively-Connected-Neural-Networks.