Abstract This paper presents a novel semantic regularized matrix factorization method for learning descriptive visual bag-of-words (BOW) representation. Although very influential in image classification, the traditional visual BOW representation has one distinct drawback. That is, for efficiency purposes, this visual representation is often generated by directly clustering the low-level visual feature vectors extracted from local keypoints or regions, without considering the high-level semantics of images. In other words, this visual representation still suffers from the semantic gap and may lead to significant performance degradation in more challenging tasks (e.g., classification of community-contributed images with large intra-class variations). To overcome this drawback, we develop a semantic regularized matrix factorization method for learning descriptive visual BOW representation by adding Laplacian regularization defined with the tags (easy to access although noisy) of community-contributed images into matrix factorization. Experimental results on two benchmark datasets show the promising performance of the proposed method.