Abstract. Modelling human free-hand sketches has become topical recently, driven by practical applications such as fine-grained sketch based
image retrieval (FG-SBIR). Sketches are clearly related to photo edgemaps, but a human free-hand sketch of a photo is not simply a clean
rendering of that photo’s edge map. Instead there is a fundamental
process of abstraction and iconic rendering, where overall geometry is
warped and salient details are selectively included. In this paper we study
this sketching process and attempt to invert it. We model this inversion
by translating iconic free-hand sketches to contours that resemble more
geometrically realistic projections of object boundaries, and separately
factorise out the salient added details. This factorised re-representation
makes it easier to match a free-hand sketch to a photo instance of an
object. Specifically, we propose a novel unsupervised image style transfer
model based on enforcing a cyclic embedding consistency constraint. A
deep FG-SBIR model is then formulated to accommodate complementary discriminative detail from each factorised sketch for better matching
with the corresponding photo. Our method is evaluated both qualitatively and quantitatively to demonstrate its superiority over a number
of state-of-the-art alternatives for style transfer and FG-SBIR