FIESTA: Fast IdEntification of State-of-The-Art models using adaptive
bandit algorithms
Abstract
We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify
state-of-the-art performance from large collections of candidate models. Despite being
known to produce unreliable comparisons, it
is still common practice to compare model
evaluations based on single choices of random
seeds. We show that reliable model selection also requires evaluations based on multiple train-test splits (contrary to common practice in many shared tasks). Using bandit theory from the statistics literature, we are able to
adaptively determine appropriate numbers of
data splits and random seeds used to evaluate
each model, focusing computational resources
on the evaluation of promising models whilst
avoiding wasting evaluations on models with
lower performance. Furthermore, our userfriendly Python implementation produces con-
fidence guarantees of correctly selecting the
optimal model. We evaluate our algorithms
by selecting between 8 target-dependent sentiment analysis methods using dramatically
fewer model evaluations than current model
selection approaches.