"Dexterity is contact planning with consequences."
A Dexterous Systems Team
This chapter studies grasping and dexterity as the design of useful contact sets over time. The object is not just acquired, it is stabilized, reoriented, and transferred under disturbance.
The main lesson is that dexterity increases task breadth by increasing contact options, but that same richness expands sensing, calibration, and control burden. Strong systems make that tradeoff explicit.
Chapter Overview
Chapter 43 starts with analytic and learned grasp synthesis, then compares hand morphologies, moves into in-hand manipulation, adds demonstration-bootstrapped RL, and closes with sim-to-real transfer for dexterous skills.
The practical stack emphasizes Dex-Net and GQ-CNN lineage, MoveIt feasibility checks, MuJoCo dexterous hands, tactile libraries, robomimic, ManiSkill, and transfer ledgers that compare simulation and hardware traces directly.
Prerequisites
Readers should know the basics of manipulation, control, and policy learning. This chapter asks them to reason more explicitly about contact families, disturbance resistance, and the cost of richer hands.
Chapter Roadmap
- 43.1 Grasp synthesis: analytic and learned (Dex-Net lineage)Grasp synthesis asks which contacts should be made, not only where the gripper should move. Analytic and learned methods differ in how they estimate grasp robustness, but both must answer the same contact question.
- 43.2 Parallel-jaw vs. multi-finger handsThe choice between parallel-jaw grippers and multi-finger hands is not philosophical. It is an engineering tradeoff across contact geometry, control complexity, sensing, speed, reliability, and recovery.
- 43.3 In-hand manipulation and reorientationIn-hand manipulation changes object pose while preserving useful control over contact. The task is not just gripping harder, but moving through a sequence of contact modes without dropping the object or losing task intent.
- 43.4 Dexterous RL with demonstrationsDexterous manipulation is one of the clearest cases where demonstrations and reinforcement learning complement each other. Demonstrations bootstrap the policy into plausible contact regimes, while RL refines robustness and recovery.
- 43.5 Sim-to-real for dexterityDexterous sim-to-real transfer is hard because every hidden modeling error in friction, compliance, sensing, delay, and calibration becomes a new contact failure mode.
Use analytic contact tools and learned scorers together. Dexterous systems become legible when contact mechanics, embodiment feasibility, and policy learning all remain visible in the same experiment ledger.
Hands-On Lab: Build the Chapter System
Objective
Build a grasp-and-reorientation benchmark that compares a simple gripper workflow to a dexterous workflow, then adds one demonstration-bootstrapped learning experiment and one transfer-gap audit.
Steps
- Define the object panel, disturbance model, and success metrics for grasp and reorientation.
- Implement or document one grasp-synthesis path and one dexterous reorientation path.
- Compare hand or policy choices on the same objects and failure taxonomy.
- Add one demonstration-initialized policy and record when RL fine-tuning begins to help.
- Finish with a transfer ledger that names the first real-world mismatch encountered.
What's Next?
Continue with Section 43.1: Grasp synthesis: analytic and learned (Dex-Net lineage), where the chapter moves from framing to the first concrete system contract.
Each section should leave behind a contact-centered artifact: a grasp score table, a hand-selection decision matrix, a contact-transition graph, a demo-to-RL curriculum note, or a sim-to-real mismatch ledger.
| Tool or Library | Where It Pays Off |
|---|---|
| Dex-Net and GQ-CNN | Synthetic supervision and learned grasp scoring |
| MoveIt 2 | Reachability and collision filtering for grasp candidates |
| MuJoCo | Dexterous-hand simulation and contact-rich rollouts |
| robomimic and LeRobot | Demonstration-based policy learning pipelines |
| Tactile libraries | Slip, contact, and reorientation state feedback |
Extend the lab by adding one perturbation, one recovery behavior, and one failure taxonomy. Save configuration, logs, metrics, and two representative traces in the same folder.
Teach this chapter through tradeoffs rather than hero demos. Ask what contact family the task demands, what extra burden richer hands introduce, and what evidence would convince a skeptical engineer that the dexterous path is worth its cost.
It helps to make embodiment variation explicit at the index level too. Parallel-jaw grippers, suction tools, tendon hands, and anthropomorphic hands all reshape the same grasping problem differently, so readers should be primed to compare representation, contact, sensing, and controller choices instead of looking for one universal dexterity recipe.
For builder audiences, the chapter index should already surface the minimum experiment contract. A grasping result needs an object panel, a grasp robustness metric, a disturbance protocol, and a recovery definition. A dexterous result adds contact-mode sequencing, in-hand pose estimation quality, and a statement of how slip or loss of contact is detected. Putting those requirements at the chapter level keeps the later sections grounded in auditable system work instead of drifting into vague capability claims.
| If your task looks like... | Start with... | Upgrade only when... |
|---|---|---|
| Pick-and-place with rigid objects and wide tolerances | Parallel-jaw grasping plus reachability filtering | Failures come from orientation, not access or clutter alone |
| Reorientation, tool use, or contact-rich insertion | Structured grasp planner plus tactile or proprioceptive feedback | Recovery and contact-mode reasoning become the bottleneck |
| Continuous in-hand pose change under disturbance | Dexterous hand, contact-state estimator, and demonstration bootstrapping | The added hand complexity still improves task-level evidence after transfer |
Before leaving the chapter, the reader should be able to justify a hand choice, explain a grasp robustness metric, sketch a reorientation path, and describe the first mismatch they would expect during dexterous transfer.
Dexterity is not a monolith. It is a stack of contact choices, sensing choices, and learning choices whose value should be argued in task-specific terms.
Agent Checklist Integration
This chapter has been reviewed as a teaching and builder unit with attention to depth, code pedagogy, diagrams, exercises, scientific framing, and practical stacks.
Use the chapter to teach students how dexterity arguments fail when they are underspecified. A convincing section does not just say that a hand is more capable. It shows what contact family became available, what sensing or calibration burden increased, how success was measured under disturbance, and whether the transfer story remained intact once the skill left simulation.
That same logic makes the chapter usable as a project spine. A semester project can move from robust parallel-jaw grasping to a single in-hand reorientation task and then to a demo-bootstrapped dexterous policy, while preserving one evidence ledger across all three stages. The educational payoff is that students see dexterity as a sequence of controlled commitments, not as a sudden jump from easy robotics to impossible robotics.
A dexterity claim is ready only when it states the contact family, disturbance model, embodiment constraints, success and recovery metrics, and any transfer gap observed between simulation and hardware.
Bibliography & Further Reading
Primary Sources, Tools, and References
Synthetic grasping dataset and robust-grasp project page.
Official learned-grasp-scoring package documentation.
Manipulation imitation-learning benchmark library.
Simulation reference for dexterous hands and contact-rich control.