Section 18.5: Human preferences and learned reward models (RLHF for control)

A Careful Control Loop

For Human preferences and learned reward models (RLHF for control), reward design must expose objective term, safety interaction, exploration effect, and deployment risk instead of hiding them inside one scalar return.

This section develops the contract for reward models in embodied control. A learned reward model maps a trajectory segment $\tau$ to a scalar score $\hat R(\tau)$. The training data is not a numeric reward label for every state. It is usually a set of comparisons such as "clip A is safer, smoother, or more successful than clip B."

The key question is practical: when should the builder trust learned preferences more than a hand-written reward, and how can the builder detect when the policy starts exploiting the reward model?

Theory

A common preference model uses a Bradley-Terry form. If a rater prefers trajectory $\tau_A$ over $\tau_B$, the model predicts

$$P(\tau_A \succ \tau_B)=\frac{\exp(\hat R(\tau_A))}{\exp(\hat R(\tau_A))+\exp(\hat R(\tau_B))}.$$

Training minimizes the negative log probability of the observed preference. In embodied control, the rater prompt must be precise: should the rater prioritize task completion, smoothness, speed, clearance from humans, object damage, or recovery behavior? A vague prompt trains a vague reward model.

Worked Example

Suppose raters prefer a slower grasp that avoids scraping the table over a faster grasp that succeeds but collides. Code Fragment 1 computes the probability assigned to the preferred clip and the corresponding loss.

# Compute one preference-model loss for two trajectory clips.
# The preferred clip should receive the higher learned reward score.
from math import exp, log

reward_safe = 1.4
reward_fast_collision = 0.2
prob_safe_preferred = exp(reward_safe) / (exp(reward_safe) + exp(reward_fast_collision))
loss = -log(prob_safe_preferred)

print("P(safe preferred)=", round(prob_safe_preferred, 3))
print("preference_loss=", round(loss, 3))

Expected output: the probability should rise when the preferred clip receives a higher score. If the score gap grows on training data but fails on held-out clips, the reward model has memorized the comparison set rather than learned the rater's criterion.

Practical Recipe

1. Define the rater rubric before collecting comparisons.
2. Sample diverse trajectory pairs, including failures, recoveries, near misses, and easy successes.
3. Track rater disagreement and remove or investigate ambiguous pairs.
4. Validate the reward model on held-out clips and adversarial high-score rollouts.
5. Report policy performance using task success and safety costs, not reward-model score alone.

Learned rewards are most useful when the desired behavior is hard to write as a formula but easy for trained raters to compare. Smooth recovery, respectful distance, gentle contact, and task style often fit this pattern. They are least useful when the rater cannot observe the relevant evidence, such as hidden force, internal wear, or a delayed safety consequence.

The graduate-level habit is to treat reward-model optimization as a distribution shift. The model is trained on comparison clips, then the policy searches for actions that maximize it. High reward-model scores should therefore trigger adversarial review, uncertainty checks, and independent embodied metrics.

A robust implementation records the human-labeling process as carefully as the RL run. Without the rater protocol and comparison distribution, a learned reward score is not interpretable.

1. Write a rater rubric with ranked criteria and tie rules.
2. Collect comparison pairs across easy, hard, unsafe, and ambiguous cases.
3. Train the reward model and report held-out pair accuracy plus calibration.
4. Optimize the policy under the reward model while saving high-score rollouts.
5. Evaluate with independent success, safety, and intervention metrics.

Code Fragment 2 records the minimum audit fields for a learned reward used in control.

# Build one preference-reward audit record for control.
# The fields connect human labels, model validation, and policy evaluation.
from dataclasses import dataclass, asdict

@dataclass
class PreferenceRewardAudit:
    section: str
    rater_rubric: str
    validation_check: str
    exploitation_probe: str
    deployment_metrics: list[str]

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = PreferenceRewardAudit(
    section="18.5",
    rater_rubric="prefer task success, then gentle contact, then smooth recovery",
    validation_check="held-out pair accuracy plus disagreement review",
    exploitation_probe="top reward-model rollouts inspected for hidden contact",
    deployment_metrics=["task_success", "contact_cost", "human_intervention_rate"],
)
print(record.as_row())

When a learned reward policy fails, assign the failure to rater ambiguity, dataset coverage, reward-model overfitting, optimization exploit, or missing physical evidence. Then add comparison pairs that target that failure and rerun the policy evaluation on the same seed panel.