Section 8.2: Cameras, depth (stereo/structured light/ToF), LiDAR

A Careful Control Loop

This section develops the technical contract for Cameras, depth (stereo/structured light/ToF), LiDAR into a usable mental model. First we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation.

The key question in Cameras, depth (stereo/structured light/ToF), LiDAR is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

For Cameras, depth (stereo/structured light/ToF), LiDAR, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example: Stereo Depth and the Point Cloud

A stereo pair recovers depth from disparity. The same world point projects to slightly different horizontal pixels in the left and right images, and that shift in pixels is the disparity $d$. With baseline $b$ (the distance between the two camera centers) and focal length $f$ in pixels, the depth is

This relation carries two practical consequences. First, depth resolution degrades with the square of distance: since $z \propto 1/d$, a one-pixel disparity error causes a depth error that grows like $z^2/(bf)$, which is why stereo rigs are accurate up close and vague far away, and why a wider baseline buys range. Second, zero disparity means infinite depth, so distant or textureless regions where disparity cannot be measured produce holes. LiDAR sidesteps disparity entirely by timing a laser pulse, giving direct metric range with near-uniform accuracy across distance, at the cost of sparser angular sampling and moving parts. Once depth per pixel is known, back-projection through the intrinsics turns the depth image into a 3D point cloud.

# Stereo depth z = b*f/d, then back-project a depth image to a 3D point cloud.
import numpy as np

b = 0.12      # baseline between the two cameras, meters
f = 700.0     # focal length, pixels
cx, cy = 320.0, 240.0   # principal point, pixels

# A tiny 2x3 disparity map (pixels). Larger disparity => nearer surface.
disparity = np.array([[40.0, 35.0, 30.0],
                      [50.0, 45.0, 20.0]])
H, W = disparity.shape
us, vs = np.meshgrid(np.arange(W), np.arange(H))

z = b * f / disparity            # metric depth per pixel
x = (us - cx) * z / f            # back-project to the camera frame
y = (vs - cy) * z / f
cloud = np.stack([x, y, z], axis=-1).reshape(-1, 3)

print("depth range (m):", round(z.min(), 3), "to", round(z.max(), 3))
print("first 3 points (m):")
print(np.round(cloud[:3], 3))

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Production Pattern

Cameras, depth (stereo/structured light/ToF), LiDAR sits inside the Part II robotics contract: geometry defines where things are, kinematics defines what motion is possible, dynamics defines what motion costs, control defines how errors are corrected, and sensing defines what the agent can know on time.

Camera, depth, and LiDAR pipelines should carry intrinsics, extrinsics, range limits, and missing-data behavior. This makes the section useful to students, builders, and researchers at the same time: the idea has an intuitive role, a formal interface, a runnable check, and a failure mode that can be reproduced.

Use this recipe when turning Cameras, depth (stereo/structured light/ToF), LiDAR into code, a simulator experiment, or a robot diagnostic. The point is not to use every library. The point is to keep the hand-built baseline and the maintained-tool path comparable.

1. Define each sensor message with units, frame, timestamp source, calibration file, and covariance meaning.
2. Run a static test, a slow-motion test, and a dropout test before fusing streams.
3. Compare the hand filter with FilterPy or ROS 2 robot_localization using identical measurements and noise settings.
4. Log innovation, covariance, delayed messages, rejected measurements, and downstream control effect.
5. Treat perception output as a belief with uncertainty, not as ground truth handed to the controller.

Vision and depth failures often come from exposure, rolling shutter, stereo texture, ToF multipath, lidar sparsity, reflective surfaces, or occlusion. Reproduce one calibration or range case by hand before retraining a perception model.