Abstract This paper studies a decision-making problem for heterogeneous multi-agent systems with safety density constraints. An individual agent’s decisionmaking problem is modeled by the standard Markov Decision Process (MDP) formulation. However, an important special case occurs when the MDP states may have limited capacities, hence upper bounds on the expected number of agents in each state are imposed. We refer to these upper bound constraints as “safety” constraints. If agents follow unconstrained policies (policies that do not impose the safety constraints), the safety constraints might be violated. In this paper, we devise algorithms that provide safe decision-making policies. The set of safe decision policies can be shown to be convex, and hence the policy synthesis is tractable via reliable and fast Interior Point Method (IPM) algorithms. We evaluate the effectiveness of proposed algorithms first using a simple MDP, and then using a dynamic traffic assignment problem. The numerical results demonstrate that safe decision-making algorithms in this paper significantly outperform other baselines.