Abstract Recognition of defects in concrete infrastructure, especially in bridges, is a costly and time consuming crucial fifirst step in the assessment of the structural integrity. Large variation in appearance of the concrete material, changing illumination and weather conditions, a variety of possible surface markings as well as the possibility for different types of defects to overlap, make it a challenging real-world task. In this work we introduce the novel COncrete DEfect BRidge IMage dataset (CODEBRIM) for multi-target classifification of fifive commonly appearing concrete defects. We investigate and compare two reinforcement learning based metalearning approaches, MetaQNN and effificient neural architecture search, to fifind suitable convolutional neural network architectures for this challenging multi-class multi-target task. We show that learned architectures have fewer overall parameters in addition to yielding better multi-target accuracy in comparison to popular neural architectures from the literature evaluated in the context of our application.