X2Face: A network for controlling face
generation using images, audio, and pose codes
Abstract. The objective of this paper is a neural network model that
controls the pose and expression of a given face, using another face or
modality (e.g. audio). This model can then be used for lightweight, sophisticated video and image editing.
We make the following three contributions. First, we introduce a network,
X2Face, that can control a source face (specified by one or more frames)
using another face in a driving frame to produce a generated frame with
the identity of the source frame but the pose and expression of the face in
the driving frame. Second, we propose a method for training the network
fully self-supervised using a large collection of video data. Third, we show
that the generation process can be driven by other modalities, such as
audio or pose codes, without any further training of the network.
The generation results for driving a face with another face are compared to state-of-the-art self-supervised/supervised methods. We show
that our approach is more robust than other methods, as it makes fewer
assumptions about the input data. We also show examples of using our
framework for video face editing