Abstract Image repurposing is a commonly used method for spreading misinformation on social media and online forums, which involves publishing untampered images with modifified metadata to create rumors and further propaganda. While manual verifification is possible, given vast amounts of verifified knowledge available on the internet, the increasing prevalence and ease of this form of semantic manipulation call for the development of robust automatic ways of assessing the semantic integrity of multimedia data. In this paper, we present a novel method for image repurposing detection that is based on the real-world adversarial interplay between a bad actor who repurposes images with counterfeit metadata and a watchdog who verififies the semantic consistency between images and their accompanying metadata, where both players have access to a reference dataset of verifified content, which they can use to achieve their goals. The proposed method exhibits state-of-the-art performance on location-identity, subject-identity and painting-artist verififi- cation, showing its effificacy across a diverse set of scenarios.