Abstract Hierarchical Reinforcement Learning (HRL) outperforms many ‘flflat’ Reinforcement Learning (RL) algorithms in some application domains. However, HRL may need longer time to obtain the optimal policy because of its large action space. Potential Based Reward Shaping (PBRS) has been widely used to incorporate heuristics into flflat RL algorithms so as to reduce their exploration. In this paper, we investigate the integration of PBRS and HRL, and propose a new algorithm: PBRS-MAXQ- 0. We prove that under certain conditions, PBRSMAXQ-0 is guaranteed to converge. Empirical results show that PBRS-MAXQ-0 signifificantly outperforms MAXQ-0 given good heuristics, and can converge even when given misleading heuristics