Abstract Because of playing one of the most important roles in the artificial intelligent systems like robots, visual understanding has gained vast interests in the past few decades. Most of the existing approaches need human labelled training data to train the learning models for visual understanding and in the most recent years, significant performance gain was obtained relying on unparalleled tremendous amount of human labelled training data. Under this circumstance, people are endowed with great burden to cost energy and time on the tedious data annotation for the traditional visual understanding approaches. To alleviate this problem, we propose to develop novel visual understanding algorithms which can learn informative visual patterns under minimal (none or very weak) supervision and thus facilitate higher-level intelligence of the visual understanding systems. Specifically, we focus on three subtopics, i.e., saliency detection, co-saliency detection, and weakly supervised learning based object detection, which can be used in both the image and video understanding systems. The experimental results have demonstrated the effectiveness of the proposed algorithms.