"Vision sees where contact might happen, touch says what contact actually became."
A Tactile Systems Group
Touch matters because many decisive task variables, slip, local compliance, micro-geometry, and incipient failure, become observable only after contact begins.
This section explains why tactile sensing is not an exotic add-on. For contact-rich tasks, it is often the only channel that exposes the local state the robot must react to within tens of milliseconds.
It joins perception, manipulation, and control by showing how touch converts hidden contact state into measurable evidence that can change the next action.
Touch is valuable not because it duplicates vision, but because it reveals the variables vision cannot reliably infer once the scene is occluded by the hand or the object itself.
Theory
Once contact begins, the robot's latent state includes normal force, tangential shear, contact patch shape, and local compliance. These quantities drive success or failure, yet they are often weakly observed or fully hidden from vision.
Tactile sensing matters most when the correct action depends on these hidden states, such as increasing grip force before slip, searching for insertion alignment, or distinguishing a rigid stop from a soft obstacle.
$$ \text{slip risk} \propto \|\mathbf{f}_t\| - \mu f_n,\qquad o_t = [I_t, q_t, z_t^{\text{tactile}}],\qquad a_t = \pi(o_t) $$
The robot observes tactile signals at contact, infers a contact state such as stable hold, incipient slip, or misalignment, adjusts force or motion accordingly, and then verifies whether the contact stabilized or deteriorated.
- Instrument one contact-rich task with tactile and non-tactile observations collected in sync.
- Label which action decisions require contact information rather than only vision or proprioception.
- Train or script a policy that can react to those tactile cues before visible failure occurs.
- Evaluate on cases where vision becomes ambiguous or fully occluded during contact.
Worked Example
# Compute a simple tactile slip margin from tangential and normal force.
mu = 0.55
normal_force = 8.0
tangential_force = 3.8
margin = round(mu * normal_force - tangential_force, 2)
status = "stable" if margin > 0.0 else "slip_risk"
print({"slip_margin_N": margin, "status": status})Expected output: The expected result reports a positive slip margin, meaning the current grip should hold under the simple friction model. As that margin shrinks toward zero, the controller should react before visible object motion appears.
DIGIT, GelSight, ReSkin or AnySkin style hardware, and tactile-processing libraries can expose the raw tactile stream quickly. The real engineering work is connecting that stream to the right control decision and verifier.
Practical Recipe
- Synchronize tactile, vision, and robot-state logs before modeling anything.
- Identify one task decision that genuinely depends on contact evidence.
- Derive simple tactile baselines such as slip margins or contact-onset detectors first.
- Compare tactile and vision-only policies on occluded or slippery cases.
- Store tactile traces beside controller actions and success labels.
Teams often add touch and then evaluate on tasks where vision already solves everything. That guarantees disappointment because the extra modality is never given a chance to matter.
Grasping a smooth bottle, inserting a plug, or opening a child-safe container all benefit from touch because the crucial success cues appear only after the hand blocks the camera or local contact starts to deform.
If the robot learns that the object is slipping only after the object is halfway to the floor, the tactile sensor has become a historian instead of a teammate.
The frontier includes richer tactile skins, visuo-tactile pretraining, and whole-hand contact representations. The stable benchmark remains simple: does the extra modality change the next action in a way that reduces failure?
Can you name one failure in your manipulation loop that becomes detectable earlier with touch than with vision?
Touch is especially important pedagogically because it shifts students away from camera-centric thinking. Many contact-rich problems are not missing intelligence so much as missing observability of the right state variables.
This section also frames tactile sensing as an information-value problem. The modality is worth its hardware and software cost only when it changes action quality on the hard cases, not when it confirms what vision already knew.
| Tool or Library | Role in the Topic | Builder Advice |
|---|---|---|
| DIGIT or GelSight | High-resolution local contact sensing | Use them when surface geometry and slip cues matter at the fingertip. |
| Force-torque sensors | Global contact loads | Helpful for complementing localized tactile images with overall wrench information. |
| PyTouch | Tactile data processing | Useful for prototyping tactile-learning pipelines and feature extraction. |
Build a slip detector using tactile and force data, then show on a held-out object why the detector fires before a vision-only baseline notices failure.
If tactile signals do not help, ask whether the task truly requires contact information, whether the sensor was synchronized correctly, or whether the policy ignores the tactile channel entirely.
Section References
Compact high-resolution tactile sensor widely used for manipulation research.
Replaceable magnetic tactile sensing platform for robust robot touch.
Machine-learning library for tactile-signal processing and modeling.
Touch matters when hidden contact variables decide the next action faster than vision can observe the failure.
Describe one contact-rich task where touch should change the action earlier than vision. Name the exact tactile feature you would monitor.