"A simulator is not a smaller world. It is a contract about which parts of the world you promise to measure."
A Calibration-Minded AI Agent
Physics simulators sit between the mathematical dynamics of Chapter 6 and the benchmark and transfer workflows of Chapter 12 and Chapter 13. This chapter teaches the mechanism first: bodies, joints, contacts, friction, time stepping, state, action, and observation. Then it turns that mechanism into a tool choice: MuJoCo, MJX, MuJoCo Warp, Isaac Sim, Isaac Lab, Newton, Genesis, Drake, SAPIEN, ROS 2, and Gazebo.
This section applies the dynamics background from Chapter 6: Dynamics and Simulation Math to the simulator stack introduced in Chapter 9: Why Simulation Is Central. It also prepares the GPU training workflows in Chapter 17: Massively Parallel and GPU RL and the randomization workflow in Chapter 13: Domain Randomization and Synthetic Data by tying tool choice to measurable task risk.
A simulator is not neutral infrastructure. It decides which contacts exist, how friction is approximated, which sensors are modeled, how many worlds can run in parallel, and which failure modes are visible. Good embodied AI work records those choices as carefully as model architecture and random seeds.
Chapter Overview
By the end of this chapter, you should be able to read a simulator recommendation and ask the right engineering questions. Does the task need contact-rich manipulation, high-throughput reinforcement learning, differentiable rollouts, photorealistic rendering, ROS 2 integration, model-based control, or a benchmark suite? Those are different needs, and they point to different tools.
The chapter uses a paired teaching rhythm. Section 11.1 builds the physics contract from scratch with a tiny deterministic stepper. Sections 11.2 through 11.7 map that contract onto the major simulator families. Section 11.8 closes with a decision table and a hands-on lab that turns simulator selection into a repeatable audit.
Learning Objectives
- Explain what rigid-body simulators model: bodies, joints, constraints, contacts, friction, actuators, sensors, and time integration.
- Choose between MJCF, URDF, USD, and ROS 2 descriptions based on the task and integration path.
- Distinguish CPU MuJoCo, MJX, MuJoCo Warp, Isaac Lab, Newton, Genesis, Drake, SAPIEN, and Gazebo by their real strengths.
- Design a simulator benchmark that measures throughput, stability, fidelity, and task transfer risk in one artifact.
- State when a fast simulator is the wrong choice because the missing physics or missing systems integration matters more than frames per second.
Prerequisites
This chapter assumes the perception-action loop from Chapter 2, coordinate frames from Chapter 4, basic dynamics from Chapter 6, and environment interfaces from Chapter 10. You do not need prior MuJoCo, Isaac, ROS, or Drake experience.
Chapter Roadmap
- 11.1 What physics simulators modelBodies, joints, contacts, friction, actuators, sensors, and the error budget behind every rollout.
- 11.2 MuJoCo and MJCF or URDF model formatsWhy MuJoCo remains the small, sharp tool for dynamics, control, and contact-rich robot learning.
- 11.3 MJX and MuJoCo WarpHow JAX and Warp move MuJoCo-style simulation toward parallel, differentiable, accelerator-native learning.
- 11.4 Isaac Sim and Isaac LabThe NVIDIA stack for USD scenes, synthetic sensors, GPU RL, and the migration away from Isaac Gym.
- 11.5 Newton and OpenUSDA frontier engine built on Warp and OpenUSD, with open governance and robot-learning ambitions.
- 11.6 Genesis and generative multi-physicsWhere Pythonic multi-physics, rendering, and generated assets may change simulation workflows.
- 11.7 Drake, SAPIEN, ROS 2, and GazeboThe systems and research tools that complete the simulator landscape.
- 11.8 Choosing a simulatorA practical decision guide, recency table, and hands-on audit lab.
| Need | Strong default | Why | Watch out for |
|---|---|---|---|
| Dynamics, control, small robot tasks | MuJoCo | Fast, readable MJCF, mature contact modeling, strong Python API | Renderer and full systems integration are not its main purpose |
| JAX parallel RL or differentiable rollout experiments | MJX | JAX transformations make batched simulation and gradients natural | Not every MuJoCo feature maps identically to accelerator execution |
| NVIDIA GPU parallel simulation with MuJoCo lineage | MuJoCo Warp | Warp targets NVIDIA hardware and high-throughput parallel simulation | Use current docs because the Newton ecosystem is moving quickly |
| Photoreal sensors, USD scenes, GPU robot learning | Isaac Sim plus Isaac Lab | Combines rendering, sensors, assets, and RL or IL workflows | Heavier stack and NVIDIA-centered hardware assumptions |
| Optimization-based control and verification | Drake | Strong systems framework, solvers, planning, control, and analysis | Not primarily a massive RL environment factory |
| ROS 2 integration and robot-system testing | Modern Gazebo | Works with ROS 2 nodes, controllers, sensors, and middleware | Gazebo Classic is end of life, use current Gazebo releases |
If you are building manipulation benchmarks, read Sections 11.1, 11.2, 11.6, 11.7, and 11.8. If you are training locomotion policies at scale, focus on Sections 11.3, 11.4, 11.5, and 11.8. If you are integrating with a real robot stack, do not skip Section 11.7.
Choose a simulator by task contract, not reputation. Record the dominant physical risk, required sensor model, throughput target, asset format, and integration boundary, then run the same task panel before comparing tools.
Hands-On Lab: Build A Simulator Decision Audit
Objective
Build a small simulator-selection report for one embodied task, using the same criteria the chapter teaches.
What You'll Practice
- Defining task physics and sensor requirements
- Separating training throughput from transfer fidelity
- Checking tool currency and deprecation risk
- Writing an evidence-backed simulator recommendation
Setup
No simulator install is required for the chapter-level lab. Use the decision table in Section 11.8, then optionally run one of the section code fragments after installing the relevant tool.
Steps
- Choose one task: block pushing, pick and place, legged locomotion, drone navigation, or ROS 2 system testing.
- List required physics: rigid contact, friction, deformables, fluids, sensors, terrain, or differentiability.
- List engineering constraints: hardware, operating system, rendering, ROS 2, parallelism, and team familiarity.
- Score at least three simulators on fidelity, throughput, integration, maturity, and debugging ergonomics.
- Write a one-page recommendation with one fallback choice and one thing to verify before committing.
Expected Output
Your output is a simulator audit that names a primary tool, a fallback tool, and the exact experiment that would falsify your choice. A good answer might choose MuJoCo for a small contact-control study, Isaac Lab for GPU locomotion training, or modern Gazebo for ROS 2 integration testing.
Stretch Goals
- Run the tiny scoring script in Section 11.8 with your own weights.
- Add a transfer-risk column that links to Chapter 13.
- Repeat the audit after reading Chapter 17 and revise your throughput assumptions.
Continue with Section 11.1, where we make the hidden physics contract explicit before selecting any simulator.
This is the foundational MuJoCo reference behind many robot-learning benchmarks. Read it to understand why contact dynamics, constraints, and control-friendly modeling shaped the simulator's design.
Drake Development Team. "Drake: Model-Based Design and Verification for Robotics."
Drake is the reference point for optimization-based robotics workflows. It is most relevant when this chapter discusses control, verification, and model-based design rather than high-throughput RL alone.
Google DeepMind. "MuJoCo Documentation."
The official documentation covers MuJoCo, MJCF, URDF loading, MJX, and MuJoCo Warp documentation paths. Practitioners should use it for current install commands and API details before running chapter code.
Isaac Lab Project. "Isaac Lab Documentation."
Isaac Lab is the maintained robot-learning framework built on Isaac Sim. It is the right starting point for readers replacing old Isaac Gym workflows with the current NVIDIA stack.
NVIDIA, Google DeepMind, and Disney Research. "Newton Physics Engine."
Newton is the newest simulator family in the chapter and should be treated as a frontier tool. The official page explains its Warp and OpenUSD foundations and its Linux Foundation governance path.
Genesis Team. "Genesis World Documentation."
Genesis is relevant for multi-physics, rendering, and Python-first simulator workflows. Researchers should read it as an emerging option, then validate performance and fidelity on their own task.