Abstract The automation of hiring decisions is a well-studied topic in crowdsourcing. Existing hiring algorithms make a common assumption—that each worker has a level of task competence that is static and does not vary over time. In this work, we explore the question of how to hire workers who can learn over time. Using a medical time series classification task as a case study, we conducted experiments to show that workers’ performance does improve with experience and that it is possible to model and predict their learning rate. Furthermore, we propose a dynamic hiring mechanism that accounts for workers’ learning potential. Through both simulation and real-world crowdsourcing data, we show that our hiring procedure can lead to high-accuracy outcomes at lower cost compared to other mechanisms.