Section 14.2: Policies and value functions

A Careful Control Loop

This section links back to Chapter 7: Control for AI Practitioners and Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The central distinction is simple: a policy chooses actions, while a value function estimates the future those actions create.

This section develops the technical contract for policies and value functions. First we define stochastic and deterministic policies, then we connect state values and action values through Bellman equations, then we run a small policy-evaluation example.

The key question is practical: when an embodied agent stands in a state with several possible actions, how does it separate "what I will do" from "how good this situation is if I keep doing it"?

Theory

A policy maps information to actions. In a fully observed MDP, a stochastic policy is written $\pi(a\mid s)$, the probability of choosing action $a$ in state $s$. In a partially observed robot task, the implemented policy often uses $\pi(a\mid o)$ or $\pi(a\mid \hat s)$, where $\hat s$ is the state estimate produced by perception and filtering.

The state-value function for a fixed policy is the expected return from state $s$:

$$V^\pi(s)=\mathbb E_\pi[G_t\mid S_t=s].$$

The action-value function asks the same question after forcing the first action:

$$Q^\pi(s,a)=\mathbb E_\pi[G_t\mid S_t=s,A_t=a].$$

These functions obey Bellman expectation equations because the return can be split into immediate reward plus discounted future value:

$$V^\pi(s)=\sum_a \pi(a\mid s)\sum_{s'}P(s'\mid s,a)\left[R(s,a,s')+\gamma V^\pi(s')\right].$$

The equation is recursive, not circular. It says the value of a state under policy $\pi$ equals the policy-weighted average of one-step outcomes plus the discounted value of the next state. For embodied systems, the hidden assumption is strong: the state or belief must contain enough physical information for the transition model to be meaningful.

Worked Example

Code Fragment 1 evaluates a fixed two-state policy for a charging robot. The state `low` means the battery is low, and the state `ready` means the robot can inspect objects; the policy recharges aggressively when low and mostly inspects when ready.

# Evaluate a fixed policy with Bellman expectation backups.
# The values forecast long-run return without changing the policy.
states = ["low", "ready"]
gamma = 0.8
policy = {
    "low": {"recharge": 0.9, "inspect": 0.1},
    "ready": {"recharge": 0.2, "inspect": 0.8},
}
transitions = {
    ("low", "recharge"): ("ready", 1.0),
    ("low", "inspect"): ("low", -2.0),
    ("ready", "recharge"): ("ready", 0.2),
    ("ready", "inspect"): ("low", 3.0),
}

values = {state: 0.0 for state in states}
for _ in range(8):
    new_values = {}
    for state in states:
        total = 0.0
        for action, prob in policy[state].items():
            next_state, reward = transitions[(state, action)]
            total += prob * (reward + gamma * values[next_state])
        new_values[state] = total
    values = new_values

for state in states:
    print(f"V({state}) = {values[state]:.3f}")

The numbers are not labels supplied by a dataset. They are self-consistent forecasts under the policy. If the policy changed, the Bellman equations would describe a different behavior and the values would need to be recomputed.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Bellman equations are useful because they turn a long-horizon prediction into repeated one-step backups. The backup is only as good as the state representation it conditions on. If a value function sees a camera frame but not the gripper force that predicts slip, it may assign the same value to physically different situations.

For policy-gradient methods in the next chapter, the value function often becomes a baseline or critic. That critic is not an optional decoration: it controls variance, estimates advantage, and can quietly mislead the update when trained on the wrong rollout distribution.

A robust implementation keeps policy outputs, value targets, and evaluation returns separate in logs. This prevents a common confusion where the action selected by the policy is mistaken for evidence that the value estimate was correct.

1. Define whether the policy receives $s$, $o$, or $\hat s$.
2. Log action probabilities or deterministic actions for each decision.
3. Compute Monte Carlo returns from traces before fitting a value network.
4. Compare Bellman targets and observed returns on a small batch.
5. Audit high-value states for physical plausibility with video or state logs.

When a value function fails, compare three quantities for the same episodes: predicted value at the start, realized discounted return, and the first few Bellman targets. Large disagreement can indicate reward bugs, truncation bugs, distribution shift, or missing state information.