Section 17.6: Throughput, wall-clock, and cost engineering

A Careful Control Loop

For Throughput, wall-clock, and cost engineering, GPU RL depends on simulator fidelity, PPO rollout semantics, reward terms, and reset distribution being versioned in the same training artifact.

This section develops the cost contract for massively parallel RL. We define the denominator for every claim: environment steps per second, policy updates per minute, wall-clock to target success, GPU memory at peak, evaluation time, and dollars per successful checkpoint.

The key question is practical: did the optimization make learning cheaper, or did it only move time from rollout collection into compilation, synchronization, logging, or evaluation?

Theory

Let a run collect $S$ environment steps in $W$ wall-clock seconds. The raw throughput is $S/W$, but the cost metric should include training, evaluation, checkpointing, and failed runs. If the target is a 90 percent held-out success rate, the relevant question is how many dollars and minutes were spent before the first checkpoint reached that target.

There is a common tension between throughput and sample efficiency. Larger batches can improve device utilization, but they may reduce update frequency or increase policy lag. Smaller batches may learn with fewer samples but waste the accelerator.

Worked Example

Code Fragment 17.6.1 computes the metrics that should appear together in a cost report. The same artifact contains throughput, wall-clock to target, evaluation overhead, and compute spend.

# Compute throughput and cost from one RL training run.
# The target-success checkpoint, not peak steps per second, drives the decision.
env_steps = 1_200_000_000
train_minutes = 42.0
eval_minutes = 6.0
gpu_dollars_per_hour = 2.20
heldout_success = 0.92
target_success = 0.90

total_minutes = train_minutes + eval_minutes
steps_per_second = env_steps / (train_minutes * 60)
total_cost = (total_minutes / 60) * gpu_dollars_per_hour
cost_per_billion_steps = total_cost / (env_steps / 1_000_000_000)

print(f"train throughput: {steps_per_second:,.0f} env steps/s")
print(f"total wall-clock: {total_minutes:.1f} min")
print(f"held-out success: {heldout_success:.2f}")
print(f"target reached: {heldout_success >= target_success}")
print(f"total compute cost: ${total_cost:.2f}")
print(f"cost per billion steps: ${cost_per_billion_steps:.2f}")

Expected output: the trace should include both systems and learning metrics. A run that reports 476,190 steps per second but omits held-out success has not shown that those steps bought a deployable policy.

Practical Recipe

1. Define the success target before training, such as 90 percent held-out success across 256 evaluation seeds.
2. Measure rollout, learning, evaluation, logging, and checkpoint time separately.
3. Track peak GPU memory, utilization, host-device transfer time, and compilation time where relevant.
4. Report dollars per target-reaching checkpoint, not only dollars per hour.
5. Repeat the run across seed panels when the result will support a paper table or hardware decision.

The idea in this section becomes useful when every number has a denominator. Steps per second uses training seconds. Wall-clock to target uses training plus evaluation seconds. Cost uses all billable time. Paper tables and engineering decisions should not mix these denominators.

The graduate-level habit is to report the bottleneck breakdown. A low throughput run may be simulation-bound, learner-bound, memory-bound, evaluation-bound, logging-bound, or compilation-bound. Each bottleneck implies a different fix.

A robust implementation starts with a cost ledger. The ledger makes it impossible to claim a throughput win from one run and a success win from another run.

1. Start the ledger before launch with instance type, price, batch size, target metric, and evaluation cadence.
2. Update the ledger at each checkpoint with train seconds, eval seconds, peak memory, success, and reset reasons.
3. Mark failed runs explicitly so cost estimates include search and debugging, not only the winning run.
4. Save profiler summaries with the same run ID as the policy checkpoint.
5. Compare cost only when success is computed on the same held-out panel.

# Record one cost ledger row for a GPU RL checkpoint.
# Keep systems metrics and held-out behavior in the same artifact.
from dataclasses import dataclass, asdict

@dataclass
class CostLedgerRow:
    checkpoint: int
    train_minutes: float
    eval_minutes: float
    peak_gpu_gb: float
    heldout_success: float
    billable_cost_usd: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

row = CostLedgerRow(
    checkpoint=320,
    train_minutes=42.0,
    eval_minutes=6.0,
    peak_gpu_gb=18.4,
    heldout_success=0.92,
    billable_cost_usd=1.76,
)
print(row.as_row())

When a cost result disappoints, avoid changing the algorithm first. Identify whether the budget was lost to simulator stepping, learner updates, GPU memory pressure, host-device transfers, compilation, evaluation, logging, or failed hyperparameter searches. Each cause has a different repair.