TieNet: Text-Image Embedding Network for Common Thorax Disease
Classification and Reporting in Chest X-rays
Abstract
Chest X-rays are one of the most common radiological examinations in daily clinical routines. Reporting thorax diseases using chest X-rays is often an entry-level task
for radiologist trainees. Yet, reading a chest X-ray image
remains a challenging job for learning-oriented machine
intelligence, due to (1) shortage of large-scale machinelearnable medical image datasets, and (2) lack of techniques that can mimic the high-level reasoning of human
radiologists that requires years of knowledge accumulation
and professional training. In this paper, we show the clinical free-text radiological reportscan be utilized as a priori
knowledge for tackling these two key problems. We propose
a novel Text-Image Embedding network (TieNet) for extracting the distinctive image and text representations. Multilevel attention models are integrated into an end-to-end
trainable CNN-RNN architecture for highlighting the meaningful text words and image regions. We first apply TieNet
to classify the chest X-rays by using both image features
and text embeddings extracted from associated reports. The
proposed auto-annotation framework achieves high accuracy (over 0.9 on average in AUCs) in assigning disease
labels for our hand-label evaluation dataset. Furthermore,
we transform the TieNet into a chest X-ray reporting system.
It simulates the reporting process and can output disease
classification and a preliminary report together. The classi-
fication results are significantly improved (6% increase on
average in AUCs) compared to the state-of-the-art baseline
on an unseen and hand-labeled dataset (OpenI).