"The hard part of dexterity is not holding on, it is changing contact without losing meaning."
A Dexterous Manipulation Bench
In-hand manipulation changes object pose while preserving useful control over contact. The task is moving through a sequence of contact modes without dropping the object or losing task intent.
This section treats in-hand manipulation as a sequence of contact transitions such as rolling, pivoting, finger gaiting, and regrasping. The robot must reason over object pose and hand configuration together.
It extends grasping from one-shot acquisition to object reorientation, tool positioning, and dexterous correction, which is where simple grasp scores stop being enough.
A secure initial grasp is only the opening move. In-hand manipulation succeeds when the robot can change contact intentionally while keeping the object inside a controllable region of the hand.
Theory
The state includes object orientation, fingertip contacts, joint configuration, and latent slip state. The action is often a combination of finger motion and controlled object motion induced by rolling or pivoting contacts.
This is why reorientation cannot be reduced to a single grasp quality number. The planner must care about whether the current contact set admits a path to the target pose through reachable intermediate contacts.
$$ R_{o,T} = R(\Delta \theta_K)\cdots R(\Delta \theta_2)R(\Delta \theta_1)R_{o,0},\qquad c_{k+1} \in \mathcal{R}(c_k, q_k)\qquad \text{while}\ \Pr(\text{drop}) < \tau $$
The system estimates the object pose in the hand, chooses a contact-mode transition such as rolling or finger gaiting, executes the transition under tactile and proprioceptive feedback, and verifies progress toward the target orientation after each step.
- Estimate object pose and current contacts in the hand frame.
- Search for a contact sequence that reaches the target orientation without violating joint or force limits.
- Execute one local contact transition and measure slip or pose residual immediately.
- Regrasp or backtrack if the next transition becomes unreachable under the current contact state.
Worked Example
# Choose the next in-hand move from orientation progress and slip risk.
moves = [
{"name": "roll", "progress_deg": 12, "slip_risk": 0.22},
{"name": "pivot", "progress_deg": 8, "slip_risk": 0.10},
{"name": "finger_gait", "progress_deg": 15, "slip_risk": 0.45},
]
ranked = []
for m in moves:
score = round(m["progress_deg"] - 20 * m["slip_risk"], 2)
ranked.append((m["name"], score))
ranked.sort(key=lambda row: row[1], reverse=True)
print(ranked)Expected output: The expected ranking prefers rolling because it makes strong orientation progress with moderate slip risk. In a real controller, ties would be broken using reachability of the next contact set.
MuJoCo dexterous hand models, tactile processing libraries, and learned contact policies can help, but the missing abstraction in many systems is still the contact-sequence ledger that explains why a reorientation path exists or fails.
Practical Recipe
- Represent object pose in the hand frame and update it after every contact transition.
- Log contact set, orientation residual, and slip estimate together.
- Use short local transitions with frequent verification instead of long open-loop finger motions.
- Reserve explicit regrasp states when the current contact family cannot reach the target orientation.
- Evaluate on held-out object shapes, not only on one friendly benchmark object.
Secure grasping can hide a dead-end contact topology. The hand may hold the object stably while making the target orientation unreachable without a deliberate regrasp.
Screwdriver pickup, package label presentation, and connector alignment all benefit from in-hand reorientation because moving the whole arm for every orientation change is slow and often unstable.
A hand that can hold a lemon very confidently is still not dexterous if every attempt to rotate it turns into an unplanned citrus launch.
Current systems combine tactile sensing, vision, and policy learning to infer latent object pose inside the hand. The strongest results tend to come from better state estimation and contact-transition supervision rather than from larger networks alone.
Could your system explain why the current contact set can or cannot reach the target orientation without a regrasp?
This section highlights the conceptual jump from grasping to dexterity. The object is no longer only constrained. It becomes a controlled body moving through a sequence of intermediate contact states.
A powerful classroom exercise here is to draw a contact graph over reorientation states. Students quickly see that dexterity is partly about planning over graph connectivity, not only about local control precision.
| Tool or Library | Role in the Topic | Builder Advice |
|---|---|---|
| MuJoCo dexterous hand models | Simulation of contact transitions | Use them to prototype rolling, pivoting, and regrasp routines with rich contact signals. |
| Tactile sensors | Slip and contact-state feedback | Critical for deciding whether a transition is proceeding or failing. |
| Replay traces | Contact-sequence debugging | Save pose, contact, and slip together so failed transitions can be explained. |
Implement a three-step reorientation planner for a simple object and show where a regrasp becomes necessary when one finger joint limit is tightened.
If orientation progress stalls, distinguish between bad state estimation, risky local transition choice, and unreachable next contact set. Each cause implies a different repair.
Section References
Widely used simulator for dexterous hand control and contact-rich rollouts.
Visuo-tactile in-hand perception project connecting touch, vision, and reorientation.
Open tactile-learning library relevant for contact and slip processing.
In-hand manipulation is successful contact-sequence planning under slip, reachability, and orientation constraints.
Sketch a contact-transition graph for reorienting a rectangular object by 90 degrees inside a multi-finger hand. Mark where a regrasp might be required.