Abstract
Text classification approaches have usually required task-specific model architectures and
huge labeled datasets. Recently, thanks to the
rise of text-based transfer learning techniques,
it is possible to pre-train a language model in
an unsupervised manner and leverage them to
perform effectively on downstream tasks. In
this work we focus on Japanese and show the
potential use of transfer learning techniques in
text classification. Specifically, we perform binary and multi-class sentiment classification
on the Rakuten product review and Yahoo
movie review datasets. We show that transfer learning-based approaches perform better
than task-specific models trained on 3 times
as much data. Furthermore, these approaches
perform just as well for language modeling
pre-trained on 1
30 of Wikipedia. We release our
pre-trained models and code as open source.