Abstract
Learning representations such that the source
and target distributions appear as similar as
possible has benefited transfer learning tasks
across several applications. Generally it requires labeled data from the source and only
unlabeled data from the target to learn such
representations. While these representations
act like a bridge to transfer knowledge learned
in the source to the target; they may lead
to negative transfer when the source specific
characteristics detract their ability to represent
the target data. We present a novel neural
network architecture to simultaneously learn
a two-part representation which is based on
the principle of segregating source specific
representation from the common representation. The first part captures the source specific
characteristics while the second part captures
the truly common representation. Our architecture optimizes an objective function which
acts adversarial for the source specific part if
it contributes towards the cross-domain learning. We empirically show that two parts of the
representation, in different arrangements, outperforms existing learning algorithms on the
source learning as well as cross-domain tasks
on multiple datasets.