Section 14.1: Learning from interaction; return and discounting

A Careful Control Loop

This section links back to Chapter 7: Control for AI Practitioners and Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The goal here is not to memorize notation; it is to know exactly what experience tuple an embodied agent records and how delayed rewards become a single training signal.

This section develops the technical contract for learning from interaction, return, and discounting. First we define the MDP and POMDP objects, then we derive the return used by value functions, then we test the arithmetic on a short robot episode.

The key question is practical: when a robot receives a reward several seconds after a motor command, how should the learning system assign that reward to earlier actions without pretending the future is as certain as the present?

Theory

An MDP is the cleanest mathematical starting point for interaction. It is a tuple $(\mathcal S,\mathcal A,P,R,\gamma)$: states $s \in \mathcal S$, actions $a \in \mathcal A$, transition probabilities $P(s' \mid s,a)$, rewards $R(s,a,s')$, and a discount factor $\gamma \in [0,1)$. The Markov assumption says the current state contains all task-relevant history: once $s_t$ and $a_t$ are known, older states do not change the distribution of $s_{t+1}$.

Embodied agents rarely receive the true state directly. A POMDP adds observations $(\mathcal O, O)$, where $O(o \mid s)$ describes how sensors produce observations from hidden physical state. The robot may see pixels, joint encoders, force readings, and latency-corrupted messages, while the state also includes unobserved friction, object mass, and contact geometry. In practice, the policy acts on $o_t$ or a belief/state estimate $\hat s_t$, while the formal MDP remains the reference model for what the environment is doing.

A trajectory is the ordered interaction record $\tau=(s_0,a_0,r_1,s_1,a_1,r_2,\ldots)$. The return from time $t$ is

$$G_t = \sum_{k=0}^{\infty}\gamma^k r_{t+k+1}.$$

Each term answers a credit assignment question. $r_{t+1}$ is the immediate consequence of the current action; $\gamma r_{t+2}$ gives the next consequence slightly less weight; $\gamma^2 r_{t+3}$ gives the following consequence less weight again. A small $\gamma$ teaches short-horizon reflexes. A large $\gamma$ makes the policy care about delayed outcomes such as recovering balance after a stumble or avoiding a collision three actions later.

Worked Example

Consider a mobile manipulator with four measured consequences after a grasp attempt: a small movement cost, a contact bonus, a delayed placement reward, and a final safety penalty. Code Fragment 1 computes the return backward so the reader can verify exactly how each future consequence reaches the current action.

# Compute discounted returns for one embodied episode.
# Backward accumulation makes delayed placement and safety effects visible.
rewards = [-0.2, 0.4, 2.0, -1.0]
gamma = 0.9

returns = []
running_return = 0.0
for reward in reversed(rewards):
    running_return = reward + gamma * running_return
    returns.append(round(running_return, 3))

returns.reverse()
for t, (reward, discounted_return) in enumerate(zip(rewards, returns)):
    print(f"t={t}: reward={reward:+.1f}, G_t={discounted_return:+.3f}")

The expected output should be read from bottom to top as a backward return construction. The key interpretation is that the early negative reward at t=0 still receives a positive return because later placement success outweighs it after discounting, while the terminal penalty remains fully visible at t=3.

The first action has a negative immediate reward but a positive return because it helped set up later success. This is the central difference between reinforcement learning and one-step supervision: the learning target for an action is not only what happened immediately after it, but what the episode eventually made possible.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

The MDP formalism is a modeling claim, not a fact about the robot. When the true physical state is hidden, the implemented learning system either uses observations directly or builds a belief/state estimate. A good chapter artifact names which one it uses, because $V^\pi(s)$ and $V^\pi(\hat s)$ are different claims.

Discounting also has a physical interpretation. If $\gamma=0.99$, the learning objective still gives substantial weight to consequences hundreds of steps later; if $\gamma=0.5$, consequences fade quickly. Choose it to match the task horizon and controller rate, not as an inherited default.

A robust implementation starts by making the episode schema explicit. Every row should contain observation, action, reward, next observation, termination, truncation, and any safety event. Once that schema is stable, the same return computation can be applied to simulator traces, replay buffers, and hardware logs.

1. Define the MDP or POMDP fields before collecting data.
2. Write the reward in units a domain expert can inspect.
3. Choose $\gamma$ from task duration and control frequency.
4. Compute returns from saved traces and spot-check at least one episode by hand.
5. Store raw rewards and returns, since later algorithms may use different horizons.

When return learning fails, first inspect the episode boundary and reward timing. Many apparent algorithm failures are target-construction failures: rewards arrive one step late, terminal penalties are dropped, time-limit truncations are treated as true failures, or observations do not contain the state needed for the Markov assumption.