Section 2.1: Agents and environments formally

"The environment is the part of the experiment that gets a vote after every action."

A Boundary-Conscious Embodied AI Agent

This section turns the agent-environment loop into a precise object you can specify, implement, test, and log. The point is not to admire a loop diagram. The point is to know exactly what happens when a robot receives a sensor packet, chooses an action, waits for the world to respond, and records evidence about the result.

The distinction matters because many embodied failures are interface failures. A policy may be competent, but the environment wrapper may hide time-limit truncation. A simulator may expose privileged state that the real robot never observes. A logger may record reward but omit the action clipping that changed the actual command.

Theory

At time $t$, an agent receives an observation $o_t$, chooses an action $a_t$, and receives a consequence that usually includes a new observation $o_{t+1}$, a reward or score $r_{t+1}$, and episode status. The environment owns the transition dynamics. The agent owns the decision rule. The evaluator owns the claim about whether behavior was good.

For a fully specified single-agent environment, the minimal contract is close to the Gymnasium pattern: reset starts an episode and returns an initial observation plus metadata; step(action) advances the world and returns observation, reward, terminated, truncated, and info. Terminated means the task reached a natural end. Truncated means an external limit, such as a time limit, stopped the episode.

For multi-agent settings, the contract also needs turn order or simultaneous actions. PettingZoo makes that distinction explicit through sequential and parallel APIs. This matters for embodied systems with people, other robots, traffic participants, or adversarial agents.

Worked Example

Code Fragment 2.1.1 builds a tiny environment contract without any reinforcement learning library. The example is deliberately small so that every returned field is visible.

# Section 2.1: runnable checkpoint for Environment dynamics and transition functions.
# Keep the output small so the evidence record can be inspected directly.
from dataclasses import dataclass

@dataclass
class Transition:
    observation: dict
    action: str
    reward: float
    terminated: bool
    truncated: bool
    info: dict

def step(position, action, time_step):
    delta = 1 if action == "right" else -1
    next_position = position + delta
    reached_goal = next_position >= 3
    timed_out = time_step >= 4
    reward = 1.0 if reached_goal else -0.01
    return Transition(
        observation={"position": next_position},
        action=action,
        reward=reward,
        terminated=reached_goal,
        truncated=timed_out and not reached_goal,
        info={"latency_ms": 12, "action_clipped": False},
    )

print(step(position=2, action="right", time_step=3))

Practical Recipe

1. Name the agent process and the environment process separately.
2. Specify reset, observation, action, reward, termination, truncation, and info fields before training.
3. Track latency for observation capture, policy inference, transport, and action execution.
4. Log the full transition tuple before aggregating success rate or return.
5. Run a no-op or safe-action baseline to verify that the environment boundary behaves as expected.

The formal interface is useful only if it prevents silent changes in the experiment. A policy result should say which environment version produced it, how reset sampled initial state, which action space was accepted, whether the command was clipped, why the episode ended, and whether the ending was a task termination or an external truncation.

The most common mistake is to treat the environment as background code. In embodied AI the environment is part of the scientific claim. If a wrapper changes observations, action limits, time limits, rewards, or diagnostic info, it changes the meaning of the result.

Before training, run an interface audit on one transition. The audit should fail if a required field is missing, if termination and truncation are confused, or if action clipping is hidden.

1. Declare the reset output and step output as named fields.
2. Check that observation and action types match the declared spaces.
3. Log termination and truncation as different fields.
4. Put latency, clipping, safety gate status, and wrapper version in info.
5. Save the first transition from every experiment run as a smoke-test artifact.

# Audit one transition record before trusting an environment result.
transition = {
    "observation": {"position": 3},
    "action": "right",
    "reward": 1.0,
    "terminated": True,
    "truncated": False,
    "info": {"latency_ms": 12, "action_clipped": False, "wrapper_version": "v2"},
}

def audit_transition(row: dict[str, object]) -> list[str]:
    required = {"observation", "action", "reward", "terminated", "truncated", "info"}
    problems = [f"missing {key}" for key in sorted(required - row.keys())]
    if row.get("terminated") and row.get("truncated"):
        problems.append("terminated and truncated cannot both explain the same ending")
    info = row.get("info", {})
    for key in ["latency_ms", "action_clipped", "wrapper_version"]:
        if key not in info:
            problems.append(f"info missing {key}")
    return problems

print(audit_transition(transition))

When an agent-environment experiment fails, first inspect the transition boundary. Check reset distribution, action clipping, wrapper order, time-limit handling, reward emission, and diagnostic info before changing the policy.