Section 10.7: PettingZoo for multi-agent

A Careful Control Loop

What This Section Builds

PettingZoo becomes operational when a multi-agent task names its agents, timing, and per-agent returns. Gymnasium assumes one learning agent acts at each step. PettingZoo adds agent identities, per-agent observations, per-agent rewards, and per-agent episode endings.

The goal is to choose the right multi-agent API before writing the task. Use AEC when turn order matters. Use the Parallel API when agents act simultaneously and the environment resolves their joint action at the end of a cycle.

Theory

PettingZoo's AEC API models an Agent Environment Cycle: the environment selects one active agent, the agent observes, acts, and the environment advances to the next agent. That model fits turn-based or order-sensitive settings, such as a robot handing off an object after another robot clears space.

The Parallel API steps all live agents together through dictionaries keyed by agent id. It fits simultaneous control, such as two mobile robots moving in the same timestep. Both APIs keep terminations, truncations, rewards, observations, and infos per agent, because different agents may leave the task for different reasons.

Worked Example

Code Fragment 10.7.1 builds a tiny Parallel API environment with two agents on a one-dimensional line. It is intentionally small so the return dictionaries are easy to inspect.

# Implement the minimal shape of a PettingZoo ParallelEnv.
# Observations, rewards, terminations, truncations, and infos are keyed by agent.
from gymnasium import spaces
from pettingzoo.utils.env import ParallelEnv

class TwoRobotLine(ParallelEnv):
    metadata = {"name": "two_robot_line_v0"}

    def __init__(self):
        self.possible_agents = ["picker", "carrier"]
        self.observation_spaces = {agent: spaces.Box(0, 4, shape=(1,), dtype=int) for agent in self.possible_agents}
        self.action_spaces = {agent: spaces.Discrete(3) for agent in self.possible_agents}

    def reset(self, seed=None, options=None):
        self.agents = self.possible_agents[:]
        self.positions = {"picker": 0, "carrier": 4}
        observations = {agent: [pos] for agent, pos in self.positions.items()}
        infos = {agent: {} for agent in self.agents}
        return observations, infos

    def step(self, actions):
        moves = {0: -1, 1: 0, 2: 1}
        for agent, action in actions.items():
            self.positions[agent] = min(4, max(0, self.positions[agent] + moves[int(action)]))
        observations = {agent: [pos] for agent, pos in self.positions.items()}
        rewards = {agent: float(self.positions["picker"] == self.positions["carrier"]) for agent in self.agents}
        terminations = {agent: rewards[agent] == 1.0 for agent in self.agents}
        truncations = {agent: False for agent in self.agents}
        infos = {agent: {"position": self.positions[agent]} for agent in self.agents}
        if any(terminations.values()):
            self.agents = []
        return observations, rewards, terminations, truncations, infos

env = TwoRobotLine()
observations, infos = env.reset(seed=42)
print(observations)
print(env.step({"picker": 2, "carrier": 0})[1:4])

The expected output exposes two independent agent views at reset, then three dictionaries keyed by agent id for rewards, terminations, and truncations after one joint step. Readers should interpret the all-false ending dictionaries as evidence that neither robot has yet reached the meeting condition.

Practical Recipe

1. Choose AEC when turn order affects state, legality, or reward assignment.
2. Choose Parallel API when all live agents submit actions for the same environment tick.
3. Define possible_agents, per-agent spaces, and live agents explicitly.
4. Return observations, rewards, terminations, truncations, and infos keyed by agent id.
5. Log agent-specific failure labels, because one agent can truncate or terminate before another.

PettingZoo matters because multi-agent environments are not only bigger Gymnasium environments. The unit of action can be one active agent or a dictionary of simultaneous agent actions. The unit of reward can be individual, shared, or both. The unit of termination can differ by agent.

The graduate-level habit is to state the game form before coding: agents, observation timing, action timing, reward ownership, termination ownership, and whether roles are symmetric. Once those are explicit, the PettingZoo API choice becomes a modeling decision rather than a software afterthought.

A robust implementation starts with the interaction schedule. After that, write the per-agent spaces and the step return structure. Only then should you connect policies or trainers.

1. List agents and roles before writing reward code.
2. Choose AEC or Parallel API from the action timing, not from trainer convenience.
3. Declare per-agent observation and action spaces.
4. Return per-agent dictionaries for observations, rewards, terminations, truncations, and infos.
5. Run PettingZoo API tests before trusting a custom environment.

# Sketch the official AEC loop shape for turn-based environments.
# The active agent receives last(), then submits one action with step().
def run_aec_policy(env, policy):
    env.reset(seed=42)
    for agent in env.agent_iter():
        observation, reward, termination, truncation, info = env.last()
        if termination or truncation:
            action = None
        else:
            action = policy(agent, observation, info)
        env.step(action)
    env.close()

print("AEC loop handles one active agent at a time.")

The expected output is intentionally verbal rather than numeric. It marks the core AEC interpretation: each iteration belongs to exactly one currently active agent, so per-agent observation, reward, and ending logic must be read turn by turn rather than as one simultaneous batch.

When an experiment about pettingzoo for multi-agent fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.