Abstract
In this paper, we study the problem of hybrid language modeling, that is using models
which can predict both characters and larger
units such as character ngrams or words. Using such models, multiple potential segmentations usually exist for a given string, for example one using words and one using characters only. Thus, the probability of a string is
the sum of the probabilities of all the possible segmentations. Here, we show how it is
possible to marginalize over the segmentations
efficiently, in order to compute the true probability of a sequence. We apply our technique
on three datasets, comprising seven languages,
showing improvements over a strong character
level language model.