资源论文IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation

IRC-GAN: Introspective Recurrent Convolutional GAN for Text-to-video Generation

2019-10-08 | |  71 |   74 |   0

Abstract Automatically generating videos according to the given text is a highly challenging task, where visual quality and semantic consistency with text are two critical issues. In existing methods, when generating a specifific frame, the information in those frames generated before is not fully exploited. And an effective way to measure the semantic consistency between videos and given text remains to be established. To address these issues, we present a novel Introspective Recurrent Convolutional GAN (IRC-GAN) approach. First, we propose a recurrent transconvolutional generator, where LSTM cells are integrated with 2D transconvolutional layers. As 2D transconvolutional layers put more emphasis on the details of each frame than 3D ones, our generator takes both the defifinition of each video frame and temporal coherence across the whole video into consideration, and thus can generate videos with better visual quality. Second, we propose mutual-information introspection to semantically align the generated video to text. Unlike other methods simply judging whether the video and the text match or not, we further take mutual information to concretely measure the semantic consistency. In this way, our model is able to introspect the semantic distance between the generated video and the corresponding text, and try to minimize it to boost the semantic consistency. We conduct experiments on 3 datasets and compare with state-ofthe-art methods. Experimental results demonstrate the effectiveness of our IRC-GAN to generate plausible videos from given text

上一篇:iDev: Enhancing Social Coding Security by Cross-platform User Identification Between GitHub and Stack Overflow

下一篇:MAT-Net: Medial Axis Transform Network for 3D Object Recognition

用户评价
全部评价

热门资源

  • A Mathematical Mo...

    Direct democracy, where each voter casts one vo...

  • Learning to Predi...

    Much of model-based reinforcement learning invo...

  • The Variational S...

    Unlike traditional images which do not offer in...

  • Hierarchical Task...

    We extend hierarchical task network planning wi...

  • Shape-based Autom...

    We present an algorithm for automatic detection...