PM-GANs: Discriminative Representation
Learning for Action Recognition Using
Partial-modalities
Abstract. Data of different modalities generally convey complimentary
but heterogeneous information, and a more discriminative representation
is often preferred by combining multiple data modalities like the RGB
and infrared features. However in reality, obtaining both data channels is
challenging due to many limitations. For example, the RGB surveillance
cameras are often restricted from private spaces, which is in conflict
with the need of abnormal activity detection for personal security. As
a result, using partial data channels to build a full representation of
multi-modalities is clearly desired. In this paper, we propose a novel
Partial-modal Generative Adversarial Networks (PM-GANs) that learns
a full-modal representation using data from only partial modalities. The
full representation is achieved by a generated representation in place
of the missing data channel. Extensive experiments are conducted to
verify the performance of our proposed method on action recognition,
compared with four state-of-the-art methods. Meanwhile, a new InfraredVisible Dataset for action recognition is introduced, and will be the first
publicly available action dataset that contains paired infrared and visible
spectrum