Abstract We explore the possibility of automatically constructing hierarchical schemas from low-level sensory data. Here we suggest a hierarchical event network to build the hierarchical schemas and describe a novel machine learning method to learn the network from the data. The traditional methods for describing schemas defifine the primitives and the relationships between them in advance. Therefore it is diffificult to adapt the constructed schemas in new situations. However, the proposed method constructs the schemas automatically from the data. Therefore, it has a novelty that the constructed schemas can be applied to new and unexpected situations flflexibly. The key idea of constructing the hierarchical schema is selecting informative sensory data,integrating them sequentially and extracting high-level information. For the experiments, we collected sensory data using multiple wearable devices in restaurant situations. The experimental results demonstrate the real hierarchical schemas, which are probabilistic scripts and action primitives, constructed from the methods. Also, we show the constructed schemas can be used to predict the corresponding event to the low-level sensor data. Moreover, we show the prediction accuracy outperforms the conventional method signifificantly