Riemannian Walk for Incremental Learning:
Understanding Forgetting and Intransigence
Abstract. Incremental learning (IL) has received a lot of attention recently, however, the literature lacks a precise problem definition, proper evaluation settings,
and metrics tailored specifically for the IL problem. One of the main objectives of
this work is to fill these gaps so as to provide a common ground for better understanding of IL. The main challenge for an IL algorithm is to update the classifier
whilst preserving existing knowledge. We observe that, in addition to forgetting,
a known issue while preserving knowledge, IL also suffers from a problem we
call intransigence, its inability to update knowledge. We introduce two metrics
to quantify forgetting and intransigence that allow us to understand, analyse, and
gain better insights into the behaviour of IL algorithms. Furthermore, we present
RWalk, a generalization of EWC++ (our efficient version of EWC [6]) and Path
Integral [25] with a theoretically grounded KL-divergence based perspective. We
provide a thorough analysis of various IL algorithms on MNIST and CIFAR-100
datasets. In these experiments, RWalk obtains superior results in terms of accuracy, and also provides a better trade-off for forgetting and intransigence