Section 10.2: Observation and action spaces

A Careful Control Loop

What This Section Builds

Observation and action spaces are the typed boundary of an embodied task. The space is where the simulator tells the learning code what can be sensed, what can be commanded, and which samples are invalid before the policy ever trains.

The goal is to make every observation and action inspectable. A camera tensor, proprioceptive vector, gripper command, and discrete mode switch should each have shape, dtype, bounds, and units that a teammate can verify.

Theory

A Gymnasium space is a mathematical set with enough metadata for code to sample, validate, batch, flatten, or reject values. Box handles continuous tensors, Discrete handles one categorical choice, MultiDiscrete handles several categorical choices, and Dict keeps structured observations readable instead of forcing every signal into one anonymous vector.

The design question is not "what can my simulator emit?" The better question is "what should the policy be allowed to observe and command?" That distinction prevents hidden state leaks, mismatched units, and action dimensions that a robot controller cannot execute.

Worked Example

Code Fragment 10.2.1 builds a structured observation space for a tabletop robot. The camera, arm joints, and gripper state stay separate, so debugging can identify which signal violated the contract.

# Define a structured observation space for a tabletop robot.
# Dict spaces keep images, joints, and gripper state inspectable.
from gymnasium import spaces

observation_space = spaces.Dict({
    "image": spaces.Box(0, 255, shape=(64, 64, 3), dtype="uint8"),
    "joint": spaces.Box(-1.0, 1.0, shape=(7,), dtype="float32"),
    "gripper": spaces.Discrete(2),
})

sample = observation_space.sample()
print(observation_space.contains(sample))
print(sample["image"].shape, sample["joint"].shape, sample["gripper"])

The expected output starts with True, which confirms that the sampled observation satisfies the declared Dict space exactly. The listed shapes then make the contract visible: one RGB image tensor, one seven-joint vector, and one binary gripper state.

Practical Recipe

1. Choose Dict for heterogeneous sensor packets instead of flattening too early.
2. Choose Box for bounded continuous vectors such as joint positions, velocities, images, or normalized actions.
3. Choose Discrete or MultiDiscrete for mode choices, buttons, symbolic skills, or gripper states.
4. Check observation_space.contains(obs) after reset and step while developing a custom environment.
5. Document units and scaling near the space declaration, not in a separate notebook.

Observation and action spaces are where simulation shortcuts often leak into results. If the observation includes exact object pose from the simulator while the real robot would only have pixels, the task has changed. If the action space accepts arbitrary forces while the real controller accepts joint targets, the policy may solve a fantasy control problem.

The graduate-level habit is to audit spaces as claims about sensing and actuation. A space declaration says what information is available, what commands are legal, and what transformations happen before the learning algorithm sees the data.

A robust implementation writes the spaces before writing the dynamics. That order forces the author to decide what the agent may know and do, then makes the simulator produce values that match the declared contract.

1. Declare the observation and action spaces in the environment constructor.
2. Run contains checks after every custom reset and step during development.
3. Keep simulator-only diagnostic variables in info, not in the policy observation.
4. Record space definitions in the experiment artifact, including units and scaling.
5. Change the policy architecture only after the environment contract is stable.

# Define action choices separately from continuous observations.
# This makes controller legality visible before training starts.
from gymnasium import spaces

action_space = spaces.MultiDiscrete([3, 3, 2])
action_space.seed(4)

action = action_space.sample()
meaning = {
    "x_motion": ["left", "hold", "right"][int(action[0])],
    "y_motion": ["back", "hold", "forward"][int(action[1])],
    "gripper": ["open", "close"][int(action[2])],
}

print(action.tolist())
print(meaning)
print(action_space.contains(action))

The expected output shows the integer action tuple and its physical decoding side by side, then confirms with True that the action is legal under the declared space. This is the interpretation readers should want in a robot log: not only what integers were sampled, but what motion command they meant.

When an experiment about observation and action spaces fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.