DUST: Dual Union of Spatio-Temporal Subspaces for Monocular
Multiple Object 3D Reconstruction
Abstract
We present an approach to reconstruct the 3D shape of
multiple deforming objects from incomplete 2D trajectories
acquired by a single camera. Additionally, we simultaneously provide spatial segmentation (i.e., we identify each of
the objects in every frame) and temporal clustering (i.e., we
split the sequence into primitive actions). This advances existing work, which only tackled the problem for one single
object and non-occluded tracks. In order to handle several
objects at a time from partial observations, we model point
trajectories as a union of spatial and temporal subspaces,
and optimize the parameters of both modalities, the nonobserved point tracks and the 3D shape via augmented Lagrange multipliers. The algorithm is fully unsupervised and
results in a formulation which does not need initialization.
We thoroughly validate the method on challenging scenarios with several human subjects performing different activities which involve complex motions and close interaction.
We show our approach achieves state-of-the-art 3D reconstruction results, while it also provides spatial and temporal
segmentation