Abstract Beauty is always an attractive topic in the human society, not only artists and psychologists, but also scientists have been searching for an answer – what is beautiful. This paper presents an approach to learning the human sense toward facial beauty. Different from previous study, the human sense is represented by a label distribution, which covers the full range of beauty ratings and indicates the degree to which each beauty rating describes the face. The motivation is that the human sense of beauty is generally quite subjective, thus it might be inappropriate to represent it with a single scalar, as most previous work does. Therefore, we propose a method called Beauty Distribution Transformation (BDT) to covert the k-wise ratings to label distributions and propose a learning method called Structural Label Distribution Learning (SLDL) based on structural Support Vector Machine to learn the human sense of facial beauty.