Section 10.6: Evaluation protocol and seeding

A Careful Control Loop

What This Section Builds

Evaluation protocol and seeding are the guardrails for a valid comparison. A seed policy controls the starting randomness; an evaluation protocol controls which tasks, wrappers, metrics, and failure labels are compared.

The goal is one auditable comparison artifact. If two methods are compared, they should be evaluated in one pass on the same environment panel, same wrapper stack, same seed list, and same metric definitions.

Theory

Seeding is not a guarantee that every library, process, and physics engine will behave identically across machines. It is a contract for controlled comparison: the experiment declares how initial conditions, action sampling, environment randomness, and evaluation panels are generated.

Gymnasium supports this discipline by passing a seed into reset and allowing the action space to be seeded for reproducible sampling. A strong protocol records both, then evaluates methods under the same seed list instead of reporting numbers from separate runs.

Worked Example

Code Fragment 10.6.1 demonstrates a minimal seed smoke test. The same seed reproduces the first sampled action and transition; a different seed changes the trace.

# Check whether the environment and action sampling are seeded together.
# The same seed should reproduce the first action and first transition.
import gymnasium as gym

def first_step(seed):
    env = gym.make("CartPole-v1")
    observation, info = env.reset(seed=seed)
    env.action_space.seed(seed)
    action = env.action_space.sample()
    next_observation, reward, terminated, truncated, info = env.step(action)
    env.close()
    return round(float(next_observation[0]), 5), int(action), terminated, truncated

print(first_step(21))
print(first_step(21))
print(first_step(22))

The expected output repeats the first trace exactly under the repeated seed, then changes when the seed changes. That is the correct interpretation for a seed smoke test: determinism within a seed, variation across seeds, and no hidden episode ending in any of the one-step traces.

Practical Recipe

1. Declare the seed list before running methods.
2. Evaluate every compared method on the same environment ids, wrappers, and seeds.
3. Save per-seed results before aggregating means or confidence intervals.
4. Report termination, truncation, and failure labels with the primary success metric.
5. Save one machine-readable artifact that contains all compared numbers.

Evaluation protocols prevent accidental storytelling. Without a shared protocol, one method can benefit from easier starts, different time limits, missing wrappers, or a different failure classifier. The table still has numbers, but the comparison does not answer the claimed question.

The graduate-level habit is to make the comparison artifact the source of truth. Every plotted mean, table entry, and claim should be derivable from one saved panel where method, seed, environment, and metric are all columns.

A robust implementation produces a method-by-seed table first, then derives summaries from it. This keeps each claim traceable to a concrete run under the same protocol.

1. Write a protocol config with environment id, wrappers, seed list, max steps, and metrics.
2. Loop over methods inside the same evaluation script.
3. Loop over seeds inside each method and write one row per episode.
4. Aggregate only after all raw rows are saved.
5. Audit every table number by recomputing it from the saved artifact.

# Build a small co-computed evaluation panel.
# Each method is evaluated on the same seed list in one artifact.
import gymnasium as gym

def rollout(method_name, seed):
    env = gym.make("CartPole-v1", max_episode_steps=5)
    observation, info = env.reset(seed=seed)
    env.action_space.seed(seed)
    total_reward = 0.0
    terminated = truncated = False

    while not (terminated or truncated):
        action = 0 if method_name == "always_left" else env.action_space.sample()
        observation, reward, terminated, truncated, info = env.step(action)
        total_reward += float(reward)

    env.close()
    return {"method": method_name, "seed": seed, "reward": total_reward, "truncated": truncated}

panel = [rollout(method, seed) for method in ["always_left", "random"] for seed in [1, 2]]
print(panel)

The expected output is one co-computed row per method and seed, not one summary per method. Even though the toy panel gives the same reward in all four rows, the important interpretation is that the evidence format makes a valid comparison possible because every row was produced under the same protocol.

When an experiment about evaluation protocol and seeding fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.