Translating and Segmenting Multimodal Medical Volumes with Cycle- and
Shape-Consistency Generative Adversarial Network
Abstract
Synthesized medical images have several important applications, e.g., as an intermedium in cross-modality image
registration and as supplementary training samples to boost
the generalization capability of a classifier. Especially, synthesized computed tomography (CT) data can provide Xray attenuation map for radiation therapy planning. In
this work, we propose a generic cross-modality synthesis
approach with the following targets: 1) synthesizing realistic looking 3D images using unpaired training data, 2)
ensuring consistent anatomical structures, which could be
changed by geometric distortion in cross-modality synthesis
and 3) improving volume segmentation by using synthetic
data for modalities with limited training samples. We show
that these goals can be achieved with an end-to-end 3D convolutional neural network (CNN) composed of mutuallybeneficial generators and segmentors for image synthesis
and segmentation tasks. The generators are trained with an
adversarial loss, a cycle-consistency loss, and also a shapeconsistency loss, which is supervised by segmentors, to reduce the geometric distortion. From the segmentation view,
the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors
prompt each other alternatively in an end-to-end training
fashion. With extensive experiments on a dataset including
a total of 4,496 CT and magnetic resonance imaging (MRI)
cardiovascular volumes, we show both tasks are beneficial
to each other and coupling these two tasks results in better
performance than solving them exclusively