Abstract
Active learning was proposed to improve learning
performance and reduce labeling cost. However,
traditional relabeling-based schemes seriously limit the ability of active learning because human may
repeatedly make similar mistakes, without improving their expertise. In this paper, we propose a
Bidirectional Active Learning with human Training (BALT) model that can enhance human related expertise during labeling and improve relabeling quality accordingly. We quantitatively capture
how gold instances can be used to both estimate labelers’ previous performance and improve their future correctness ratio. Then, we propose the backward relabeling scheme that actively selects the
most likely incorrectly labeled instances for relabeling. Experimental results on three real datasets
demonstrate that our BALT algorithm significantly
outperforms representative related proposals