Abstract
Sequential learning of tasks using gradient descent
leads to an unremitting decline in the accuracy of
tasks for which training data is no longer available,
termed catastrophic forgetting. Generative models
have been explored as a means to approximate the
distribution of old tasks and bypass storage of real
data. Here we propose a cumulative closed-loop
memory replay GAN (CloGAN) provided with external regularization by a small memory unit selected for maximum sample diversity. We evaluate incremental class learning using a notoriously
hard paradigm, “single-headed learning,” in which
each task is a disjoint subset of classes in the overall
dataset, and performance is evaluated on all previous classes. First, we show that when constructing a dynamic memory unit to preserve sample heterogeneity, model performance asymptotically approaches training on the full dataset. We then show
that using a stochastic generator to continuously
output fresh new images during training increases
performance significantly further meanwhile generating quality images. We compare our approach
to several baselines including fine-tuning by gradient descent (FGD), Elastic Weight Consolidation
(EWC), Deep Generative Replay (DGR) and Memory Replay GAN(MeRGAN). Our method has very
low long-term memory cost, the memory unit, as
well as negligible intermediate memory storage