Abstract Medical image analysis has two important research areas: disease grading and fifine-grained lesion segmentation. Although the former problem often relies on the latter, the two are usually studied separately. Disease severity grading can be treated as a classifification problem, which only requires image-level annotations, while the lesion segmentation requires stronger pixel-level annotations. However, pixel-wise data annotation for medical images is highly time-consuming and requires domain experts. In this paper, we propose a collaborative learning method to jointly improve the performance of disease grading and lesion segmentation by semi-supervised learning with an attention mechanism. Given a small set of pixel-level annotated data, a multi-lesion mask generation model fifirst performs the traditional semantic segmentation task. Then, based on initially predicted lesion maps for large quantities of imagelevel annotated data, a lesion attentive disease grading model is designed to improve the severity classifification accuracy. Meanwhile, the lesion attention model can refifine the lesion maps using class-specifific information to fifine-tune the segmentation model in a semi-supervised manner. An adversarial architecture is also integrated for training. With extensive experiments on a representative medical problem called diabetic retinopathy (DR), we validate the effectiveness of our method and achieve consistent improvements over state-of-the-art methods on three public datasets