Amalgamating Filtered Knowledge:
Learning Task-customized Student from Multi-task Teachers
Abstract
Many well-trained Convolutional Neural Network (CNN) models have now been released online by developers for the sake of effortless reproducing. In this paper, we treat such pre-trained networks as teachers, and explore how to learn a target
student network for customized tasks, using multiple teachers that handle different tasks. We assume
no human-labelled annotations are available, and
each teacher model can be either single- or multitask network, where the former is a degenerated
case of the latter. The student model, depending on
the customized tasks, learns the related knowledge
filtered from the multiple teachers, and eventually
masters the complete or a subset of expertise from
all teachers. To this end, we adopt a layer-wise
training strategy, which entangles the student’s network block to be learned with the corresponding
teachers. As demonstrated on several benchmarks,
the learned student network achieves very promising results, even outperforming the teachers on the
customized tasks.