Abstract. We propose dynamic task prioritization for multitask learning. This allows a model to dynamically prioritize difficult tasks during
training, where difficulty is inversely proportional to performance, and
where difficulty changes over time. In contrast to curriculum learning,
where easy tasks are prioritized above difficult tasks, we present several
studies showing the importance of prioritizing difficult tasks first. We observe that imbalances in task difficulty can lead to unnecessary emphasis
on easier tasks, thus neglecting and slowing progress on difficult tasks.
Motivated by this finding, we introduce a notion of dynamic task prioritization to automatically prioritize more difficult tasks by adaptively
adjusting the mixing weight of each task’s loss objective. Additional ablation studies show the impact of the task hierarchy, or the task ordering,
when explicitly encoded in the network architecture. Our method outperforms existing multitask methods and demonstrates competitive results
with modern single-task models on the COCO and MPII datasets