Abstract We consider the problem of robots following natural language commands through previously unknown outdoor environments. A robot receives commands in natural language, such as “Navigate around the building to the car left of the fifire hydrant and near the tree.” The robot needs fifirst to classify its surrounding objects into categories, using images obtained from its sensors. The result of this classifification is a map of the environment, where each object is given a list of semantic labels, such as “tree” or “car”, with varying degrees of confifidence. Then, the robot needs to ground the nouns in the command, i.e., mapping each noun in the command into a physical object in the environment. The robot needs also to ground a specifified navigation mode, such as “navigate quickly” or “navigate covertly”, as a cost map. In this work, we show how to ground nouns and navigation modes by learning from examples demonstrated by humans