Section 2.7: Partially observable MDPs; belief states

A Careful Control Loop

This section develops the contract for decision making when the current observation is not enough. A POMDP keeps the MDP pieces from Section 2.6, then adds an observation model and a belief state. The belief state is not a guess pasted onto the policy. It is the action interface the policy actually receives.

The practical question is: which hidden variables matter for action, what observations give evidence about them, and when should the agent spend an action to gather information rather than rush toward the goal?

Theory

A POMDP is often written as $(\mathcal{S}, \mathcal{A}, P, R, \Omega, O, \gamma)$. The new pieces are $\Omega$, the observation space, and $O(o|s,a)$, the probability of receiving observation $o$ after action $a$ when the world is in state $s$. Instead of acting on an unobserved state, the agent acts on a belief $b_t(s)$, a probability distribution over states.

A one-step belief update has the form $$b_{t+1}(s') = \eta O(o_{t+1}|s',a_t)\sum_s P(s'|s,a_t)b_t(s),$$ where $\eta$ normalizes the probabilities so they sum to one. The summation predicts where the hidden state could move after the action. The observation likelihood then favors states that make the new sensor reading plausible.

Worked Example

Code Fragment 2.7.1 implements the belief update for a two-state contact problem. A force spike is weakly compatible with a dry surface and strongly compatible with a slippery surface, so the posterior belief should shift toward slip risk.

# Section 2.7: update a belief state from a force observation.
# Predict hidden contact state, then reweight it by observation likelihood.
states = ["dry_surface", "slippery_surface"]
belief = {"dry_surface": 0.70, "slippery_surface": 0.30}
transition = {
    "dry_surface": {"dry_surface": 0.85, "slippery_surface": 0.15},
    "slippery_surface": {"dry_surface": 0.20, "slippery_surface": 0.80},
}
likelihood_force_spike = {"dry_surface": 0.15, "slippery_surface": 0.80}

predicted = {
    next_state: sum(transition[state][next_state] * belief[state] for state in states)
    for next_state in states
}
unnormalized = {
    state: likelihood_force_spike[state] * predicted[state]
    for state in states
}
normalizer = sum(unnormalized.values())
posterior = {state: unnormalized[state] / normalizer for state in states}
print({state: round(probability, 3) for state, probability in posterior.items()})

Expected output: the posterior belief should put most probability on slippery_surface. The important teaching point is not the exact number alone, but the trace from prior belief to predicted belief to observation-corrected belief.

Practical Recipe

1. List the hidden variables that change action choice.
2. Specify the observation likelihood for each hidden variable, even if it starts as an approximate table.
3. Log prior belief, predicted belief, posterior belief, chosen action, and observation timestamp.
4. Test information-gathering actions separately from task-completion actions.
5. Evaluate calibration under ambiguous observations, occlusion, delayed sensors, and contact changes.

Partially observable MDPs; belief states becomes useful when it is tied to a closed-loop contract for the contract between policy, world, evaluator, and safety constraints. The contract names the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. That is the bridge between a readable concept and a system a skeptical builder can test.

For Partially observable MDPs; belief states, separate the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout are different kinds of evidence, and the section should keep them distinct.

For Partially observable MDPs; belief states, a robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

When Partially observable MDPs; belief states fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

pip install numpy pandas

# Complete compact evidence trace for the section lab.
# Extend these records with values produced by your actual environment or simulator.
import pandas as pd

records = [
    {"run": "baseline", "seed": 0, "success": 0.72, "failure_label": "none"},
    {"run": "library_shortcut", "seed": 0, "success": 0.78, "failure_label": "none"},
    {"run": "baseline_perturbed", "seed": 0, "success": 0.54, "failure_label": "latency"},
]
print(pd.DataFrame(records))