marl

Multi-Agent Reinforcement Learning (MARL) is the study of how multiple agents learn to act within a shared environment, where each agent seeks to maximize its own cumulative reward. MARL covers a range of settings from fully cooperative to competitive or mixed interactions. Unlike single-agent reinforcement learning, the environment in MARL is non-stationary from the perspective of any individual learner because other agents are continually adapting their policies.

Formalization usually uses the framework of Markov games (also called stochastic games). A Markov game generalizes a Markov decision process to multiple agents. The environment has states S, each agent i has an action space A_i, and the joint action a = (a_1, ..., a_N) leads to transitions P(s' | s, a) and rewards r_i(s, a, s') for each agent. Each agent i follows a policy π_i that maps observations (and possibly history) to actions. The objective is to maximize the expected discounted sum of rewards for each agent, often under considerations of cooperation, competition, or a combination.

Approaches in MARL frequently involve centralized training with decentralized execution (CTDE), especially in deep learning contexts. Popular methods include value-based approaches like QMIX and VDN, policy-gradient or actor-critic methods such as MADDPG and COMA, and various scalable architectures to address credit assignment and coordination. Independent Q-learning is simple but can suffer from non-stationarity.

Challenges in MARL include non-stationarity due to simultaneous learning, credit assignment across agents, partial observability, heterogeneity among agents, and scaling to many agents or actions. Applications span robotics coordination, traffic signal control, autonomous driving, drone swarms, and large-scale simulations. Benchmark environments and libraries, such as StarCraft II workspaces and multi-agent benchmarks in PettingZoo, are used to evaluate performance and scalability.