Hierarchical method for cooperative multi-agent reinforcement learning in Markov decision processes

In the rapidly evolving field of reinforcement learning, combination of hierarchical and multi-agent learning methods presents unique challenges and opens up new opportunities. This paper discusses a combination of multi-level hierarchical learning with subgoal discovery and multi-agent reinforcement learning with hindsight experience replay. Combining these approaches leads to the creation of Multi-Agent Subgoal Hierarchy Algorithm (MASHA) that allows multiple agents to learn efficiently in complex environments, including environments with sparse rewards. We demonstrate the results of the proposed approach in one of these environments inside the StarCraft II strategy game, in addition to making comparisons with other existing approaches. The proposed algorithm is developed in the paradigm of centralized learning with decentralized execution, which makes it possible to achieve a balance between coordination and autonomy of agents.