Abstract
Humans effortlessly “program” one another by communicating goals and desires in natural language. In contrast,
humans program robotic behaviours by indicating desired
object locations and poses to be achieved [5], by providing RGB images of goal configurations [19], or supplying
a demonstration to be imitated [17]. None of these methods generalize across environment variations, and they convey the goal in awkward technical terms. This work proposes joint learning of natural language grounding and instructable behavioural policies reinforced by perceptual detectors of natural language expressions, grounded to the
sensory inputs of the robotic agent.
Our supervision is narrated visual demonstrations
(NVD), which are visual demonstrations paired with verbal narration (as opposed to being silent). We introduce a dataset of NVD where teachers perform activities
while describing them in detail. We map the teachers’ descriptions to perceptual reward detectors, and use them
to train corresponding behavioural policies in simulation.
We empirically show that our instructable agents (i) learn
visual reward detectors using a small number of examples by exploiting hard negative mined configurations from
demonstration dynamics, (ii) develop pick-and-place policies using learned visual reward detectors, (iii) benefit from
object-factorized state representations that mimic the syntactic structure of natural language goal expressions, and
(iv) can execute behaviours that involve novel objects in
novel locations at test time, instructed by natural language