Abstract. We present a novel method that is able to track a complex
deformable object in interaction with a hand. This is achieved by formulating and solving an optimization problem that jointly considers the
hand, the deformable object and the hand/object contact points. The optimization evaluates several hand/object contact configuration hypotheses and adopts the one that results in the best fit of the object’s model
to the available RGBD observations in the vicinity of the hand. Thus,
the hand is not treated as a distractor that occludes parts of the deformable object, but as a source of valuable information. Experimental
results on a dataset that has been developed specifically for this new
problem illustrate the superior performance of the proposed approach
against relevant, state of the art solutions