Are we there yet? Encoder-decoder neural networks
as cognitive models of English past tense inflection
Abstract
The cognitive mechanisms needed to account
for the English past tense have long been a
subject of debate in linguistics and cognitive
science. Neural network models were proposed early on, but were shown to have clear
flaws. Recently, however, Kirov and Cotterell
(2018) showed that modern encoder-decoder
(ED) models overcome many of these flaws.
They also presented evidence that ED models demonstrate humanlike performance in a
nonce-word task. Here, we look more closely
at the behaviour of their model in this task.
We find that (1) the model exhibits instability across multiple simulations in terms of
its correlation with human data, and (2) even
when results are aggregated across simulations
(treating each simulation as an individual human participant), the fit to the human data is
not strong—worse than an older rule-based
model. These findings hold up through several alternative training regimes and evaluation
measures. Although other neural architectures
might do better, we conclude that there is still
insufficient evidence to claim that neural nets
are a good cognitive model for this task