Abstract We present an approach and a system that explores the application of interactive machine learning to a branching program-based boosting algorithm—Martingale Boosting. Typically, its performance is based on the ability of a learner to meet a fixed objective and does not account for preferences (e.g. low false positives) arising from an underlying classification problem. We use user preferences gathered on holdout data to guide the two-sided advantages of individual weak learners and tune them to meet these preferences. Extensive experiments show that while arbitrary preferences might be difficult to meet for a single classifier, a non-linear ensemble of classifiers as the one constructed by martingale boosting, performs better.