Abstract

We present a novel, non-intrusive approach for estimat-ing contact forces during hand-object interactions relyingsolely on visual input provided by a single RGB-D camera.We consider a manipulated object with known geometricaland physical properties. First, we rely on model-based vi-sual tracking to estimate the object’s pose together with thatof the hand manipulating it throughout the motion. Fol-lowing this, we compute the object’s first and second orderkinematics using a new class of numerical differentiation operators. The estimated kinematics is then instantly fed into a second-order cone program that returns a minimal force distribution explaining the observed motion. However, humans typically apply more forces than mechanically re-quired when manipulating objects. Thus, we complete ourestimation method by learning these excessive forces andtheir distribution among the fingers in contact. We provide a full validity analysis of the proposed method by evaluating it based on ground truth data from additional sensors such as accelerometers, gyroscopes and pressure sensors. Experimental results show that force sensing from vision (FSV) is indeed feasible.