Abstract

Content based image retrieval is highly relevant in med-ical imaging, since it makes vast amounts of imaging dataaccessible for comparison during diagnosis. Finding im-age similarity measures that reflect diagnostically relevantrelationships is challenging, since the overall appearancevariability is high compared to often subtle signatures ofdiseases. To learn models that capture the relationship be-tween semantic clinical information and image elements at scale, we have to rely on data generated during clinical rou-tine (images and radiology reports), since expert annotationis prohibitively costly. Here we show that re-mapping vi-sual features extracted from medical imaging data based onweak labels that can be found in corresponding radiologyreports creates descriptions of local image content captur-ing clinically relevant information. We show that these se-mantic profiles enable higher recall and precision duringretrieval compared to visual features, and that we can evenmap semantic terms describing clinical findings from radi-ology reports to localized image volume areas.