Abstract
We propose a segmental neural language
model that combines the generalization power
of neural networks with the ability to discover
word-like units that are latent in unsegmented
character sequences. In contrast to previous
segmentation models that treat word segmentation as an isolated task, our model unifies
word discovery, learning how words fit together to form sentences, and, by conditioning the model on visual context, how words’
meanings ground in representations of nonlinguistic modalities. Experiments show that
the unconditional model learns predictive distributions better than character LSTM models,
discovers words competitively with nonparametric Bayesian word segmentation models,
and that modeling language conditional on visual context improves performance on both