Towards Generating Long and Coherent Text
with Multi-Level Latent Variable Models
Abstract
Variational autoencoders (VAEs) have received much attention recently as an end-toend architecture for text generation with latent variables. However, previous works typically focus on synthesizing relatively short
sentences (up to 20 words), and the posterior collapse issue has been widely identified
in text-VAEs. In this paper, we propose to
leverage several multi-level structures to learn
a VAE model for generating long, and coherent text. In particular, a hierarchy of stochastic layers between the encoder and decoder
networks is employed to abstract more informative and semantic-rich latent codes. Besides, we utilize a multi-level decoder structure to capture the coherent long-term structure inherent in long-form texts, by generating
intermediate sentence representations as highlevel plan vectors. Extensive experimental results demonstrate that the proposed multi-level
VAE model produces more coherent and less
repetitive long text compared to baselines as
well as can mitigate the posterior-collapse issue