Abstract
We address the problem of image feature learning for the
applications where multiple factors exist in the image generation process and only some factors are of our interest.
We present a novel multi-task adversarial network based on
an encoder-discriminator-generator architecture. The encoder extracts a disentangled feature representation for the
factors of interest. The discriminators classify each of the
factors as individual tasks. The encoder and the discriminators are trained cooperatively on factors of interest, but in
an adversarial way on factors of distraction. The generator
provides further regularization on the learned feature by reconstructing images with shared factors as the input image.
We design a new optimization scheme to stabilize the adversarial optimization process when multiple distributions
need to be aligned. The experiments on face recognition
and font recognition tasks show that our method outperforms the state-of-the-art methods in terms of both recognizing the factors of interest and generalization to images
with unseen variations