Abstract Learning visual features from unlabeled image data is an important yet challenging task, which is often achieved by training a model on some annotation-free information. We consider spatial contexts, for which we solve so-called jigsaw puzzles, i.e., each image is cut into grids and then disordered, and the goal is to recover the correct confifiguration. Existing approaches formulated it as a classifification task by defifining a fifixed mapping from a small subset of confifigurations to a class set, but these approaches ignore the underlying relationship between different confifigurations and also limit their applications to more complex scenarios. This paper presents a novel approach which applies to jigsaw puzzles with an arbitrary grid size and dimensionality. We provide a fundamental and generalized principle, that weaker cues are easier to be learned in an unsupervised manner and also transfer better. In the context of puzzle recognition, we use an iterative manner which, instead of solving the puzzle all at once, adjusts the order of the patches in each step until convergence. In each step, we combine both unary and binary features of each patch into a cost function judging the correctness of the current confifiguration. Our approach, by taking similarity between puzzles into consideration, enjoys a more effificient way of learning visual knowledge. We verify the effectiveness of our approach from two aspects. First, it solves arbitrarily complex puzzles, including high-dimensional puzzles, that prior methods are diffificult to handle. Second, it serves as a reliable way of network initialization, which leads to better transfer performance in visual recognition tasks including classifification, detection and segmentation