Section 18.3: Goal-conditioned policies; hindsight experience replay

A Careful Control Loop

For Goal-conditioned policies; hindsight experience replay, reward design must expose objective term, safety interaction, exploration effect, and deployment risk instead of hiding them inside one scalar return.

This section develops the contract for policies of the form $\pi(a \mid s,g)$, where $g$ is a desired goal such as a target object pose, waypoint, door angle, drawer position, or language-grounded task state. The reward is also goal-indexed: $r_g(s,a,s')$ asks whether the transition moved the world toward that particular goal.

The key question is practical: how can a robot learn from an attempt that missed the requested goal but still reached a different, useful state?

Theory

A goal-conditioned replay buffer stores transitions as $(s_t, a_t, s_{t+1}, g, r_g)$. In sparse-goal tasks, the original reward may be zero for most failed trials. HER adds extra replay entries by replacing the desired goal $g$ with a goal $\tilde g$ that was achieved later in the same episode, then recomputing the reward $r_{\tilde g}$.

The mechanism works because the physical transition did happen. The same push, step, or grasp attempt can teach the value of moving from $s_t$ to $s_{t+1}$ under a different goal label. The evaluation, however, must remain on the original goal distribution, otherwise relabeling becomes a way to lower the task rather than learn it.

Worked Example

Suppose a block was supposed to end at position 10, but the robot pushed it to position 7. The original transition failed. HER can relabel part of the replay entry with goal 7, because that is what the trajectory actually achieved. Code Fragment 1 shows the relabeling step.

# Relabel failed transitions with goals the episode actually achieved.
# The original task remains unchanged for final evaluation.
episode = [
    {"state": 0, "action": "push", "achieved_goal": 3},
    {"state": 3, "action": "push", "achieved_goal": 7},
]
requested_goal = 10

for transition in episode:
    original_reward = int(transition["achieved_goal"] == requested_goal)
    relabeled_goal = transition["achieved_goal"]
    hindsight_reward = int(transition["achieved_goal"] == relabeled_goal)
    print(transition["action"], "g=", requested_goal, "r=", original_reward, "her_g=", relabeled_goal, "her_r=", hindsight_reward)

Expected output: the same physical transitions are visible under two labels: failed for goal 10, successful for goals 3 and 7. The relabeled reward is a replay trick, not a deployment metric.

Practical Recipe

1. Define the goal representation in task coordinates, not only in pixels.
2. Log both desired and achieved goals at every step.
3. Write a reward function that can be recomputed for any candidate goal.
4. Relabel replay entries with achieved goals from the same episode.
5. Evaluate only on held-out requested goals, with the original success threshold.

The main design choice is the goal space. A pose goal, image goal, language goal, and contact-state goal create different generalization problems. Pose goals are easy to score but may miss semantic intent. Image goals can capture rich state but may reward visual coincidence. Language goals are flexible but need grounding into measurable state.

The graduate-level habit is to separate relabeling from evaluation. Relabeling changes the training distribution in the replay buffer. Evaluation samples desired goals from the task distribution and computes success without hindsight. Mixing these two distributions invalidates the comparison because the training trick becomes part of the reported task.

A robust implementation starts by testing the reward recomputation function independently. If the function cannot score arbitrary achieved-goal and desired-goal pairs, the replay buffer cannot relabel transitions safely.

1. Choose a goal representation and success tolerance.
2. Verify reward recomputation on hand-written state-goal pairs.
3. Store desired goal, achieved goal, action, next achieved goal, and done flag.
4. Relabel a controlled fraction of replay samples with future achieved goals.
5. Report original-goal success, hindsight sample ratio, and goal-distribution coverage.

Code Fragment 2 captures the audit fields that make a HER run interpretable.

# Build one HER audit record for a goal-conditioned policy.
# The record separates replay relabeling from deployment evaluation.
from dataclasses import dataclass, asdict

@dataclass
class HERAudit:
    section: str
    goal_space: str
    relabel_strategy: str
    reward_recompute_test: str
    report_metrics: list[str]

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = HERAudit(
    section="18.3",
    goal_space="object xy pose in meters",
    relabel_strategy="future achieved goals from the same episode",
    reward_recompute_test="score three hand-written achieved/desired goal pairs",
    report_metrics=["requested_goal_success", "hindsight_ratio", "goal_coverage"],
)
print(record.as_row())

When HER fails, check whether the achieved goals are too narrow, too noisy, or not physically meaningful. Then inspect whether the relabeled goals match states the robot can intentionally reproduce. The failure label should distinguish exploration failure, goal-representation failure, reward-recompute bug, and evaluation-distribution mismatch.