Module 03: The Bedrock (Calibration & Transforms)

By Gopi Krishna Tummala

The Story: The Most Under-Appreciated Part

The Analogy: If you don’t know where your eyes are relative to your feet, you trip. If you don’t know where your cameras are relative to your LiDAR, your perception fails.

Calibration is the most under-appreciated part of the autonomous stack. It’s invisible when it works, catastrophic when it fails. A 1-degree error in camera-LiDAR calibration can cause a 10cm error at 10m distance. At 30 mph, that’s the difference between hitting a pedestrian and missing them.

The “Oh S**t” Scenario: The Misaligned Sensor

The Failure Mode: Your vehicle has been driving for 6 months. A camera-LiDAR calibration drifts by 0.5 degrees (thermal expansion, vibration, or a minor impact). You don’t notice it — the error is small.

Then you encounter a scenario: a pedestrian is 20m ahead. Your camera sees them. Your LiDAR sees them. But because of the calibration error, when you project the LiDAR point onto the camera image, it’s offset by 10cm.

Your fusion algorithm thinks: “The camera sees a person here, but the LiDAR sees something 10cm away. These don’t match. Must be a false positive.”

Result: The pedestrian is ignored. Near-miss collision.

Why This Happens:

1. Calibration drift: Sensors move relative to each other over time
2. No detection: Small errors are hard to detect without explicit monitoring
3. Cascading failure: Small calibration errors cause large perception errors

The Solution: Online calibration — continuously monitor and correct calibration errors in real-time.

Intrinsics: Lens Distortion

Intrinsics describe the internal properties of a camera — how it maps 3D rays to 2D pixels.

The Pinhole Model (Ideal)

The Math:

Where $K$ is the intrinsic matrix:

Parameters:

• $f_x, f_y$ = focal lengths (in pixels)
• $c_x, c_y$ = principal point (image center, in pixels)

Lens Distortion (Real-World)

The Problem: Real lenses have distortion — straight lines in the world appear curved in the image.

Types of Distortion:

1. Radial Distortion: Caused by lens shape (barrel or pincushion)
2. Tangential Distortion: Caused by lens misalignment

The Math:

Radial Distortion:

Where:

• $r^2 = x^2 + y^2$ (distance from image center)
• $k_1, k_2, k_3$ = radial distortion coefficients

Tangential Distortion:

Where $p_1, p_2$ = tangential distortion coefficients

The Calibration Problem: Estimate $K$ and distortion coefficients $(k_1, k_2, k_3, p_1, p_2)$ from images of a known pattern (e.g., checkerboard).

Extrinsics: Rigid Body Transforms

Extrinsics describe the position and orientation of one sensor relative to another (or relative to the vehicle frame).

The Transform

The Math:

Where:

• $X_{\text{source}}$ = point in source frame
• $X_{\text{target}}$ = point in target frame
• $R$ = rotation matrix (3×3)
• $t$ = translation vector (3×1)

Example: Transform a LiDAR point to the camera frame:

Why This Matters

The Fusion Problem: To fuse camera and LiDAR data, you need to know:

• Where is the LiDAR relative to the camera? (extrinsics)
• How does the camera project 3D to 2D? (intrinsics)

The Error Propagation:

If calibration is off by angle $\theta$ and distance $d$:

At range $r$:

• Angular error: $\Delta x = r \cdot \sin(\theta) \approx r \cdot \theta$ (for small $\theta$)
• Distance error: $\Delta x = d$

Example: At $r = 20$m, if $\theta = 0.5°$:

That’s the width of a person. A calibration error can cause you to miss a pedestrian.

Homogeneous Coordinates

The Problem: Rotation and translation are separate operations. This makes composition of transforms awkward.

The Solution: Homogeneous coordinates — represent rotation and translation as a single matrix operation.

The Math

Homogeneous Representation:

Where the 4×4 transformation matrix is:

Composition of Transforms:

If you have two transforms $T_1$ and $T_2$:

The Intuition: Transform from frame A → B → C is the same as transforming A → C directly.

Example: Transform from LiDAR → Vehicle → Camera:

SE(3): Lie Groups and Lie Algebras

SE(3) is the Special Euclidean Group — the set of all rigid body transforms (rotations + translations).

Why SE(3) Matters

The Problem: How do you optimize over rotations? Rotation matrices have constraints:

• $R^T R = I$ (orthonormal)
• $\det(R) = 1$ (no reflection)

These constraints make optimization difficult.

The Solution: Lie Groups and Lie Algebras

Lie Algebra: $\mathfrak{se}(3)$

The Math:

A transform $T \in SE(3)$ can be represented by its Lie algebra $\xi \in \mathfrak{se}(3)$:

Where:

• $\rho \in \mathbb{R}^3$ = translation component
• $\phi \in \mathbb{R}^3$ = rotation component (axis-angle representation)

The Exponential Map:

Where $\xi^\wedge$ is the “hat” operator that converts $\xi$ to a 4×4 matrix.

The Logarithm Map:

Why This Helps

Optimization: Instead of optimizing over constrained rotation matrices, you optimize over unconstrained Lie algebra elements $\xi \in \mathbb{R}^6$.

The Calibration Problem:

Minimize reprojection error:

Where:

• $X_i$ = 3D point (from LiDAR)
• $x_i$ = 2D observation (from camera)
• $\pi$ = projection function
• $T(\xi)$ = transform parameterized by Lie algebra

Gradient-based optimization (e.g., Levenberg-Marquardt) can now optimize over $\xi$ directly.

Calibration Methods: Online vs. Offline

Offline Calibration

The Setup: Use a known calibration target (checkerboard, AprilTag, etc.) in a controlled environment.

The Process:

1. Collect data: Capture images/point clouds of the calibration target from multiple viewpoints
2. Extract features: Detect corners/markers in images, extract points in LiDAR
3. Optimize: Minimize reprojection error to estimate calibration parameters

The Math:

Where:

• $X_j$ = 3D point on calibration target
• $x_{ij}$ = observed 2D point in image $i$
• $\pi$ = projection function (with distortion)

Pros:

• Accurate (millimeter-level precision)
• Can calibrate all parameters at once
• Well-understood, standard approach

Cons:

• Requires controlled environment
• Time-consuming (30-60 minutes)
• Doesn’t handle calibration drift

Online Calibration

The Setup: Continuously estimate calibration from natural scene observations (no calibration target needed).

The Process:

1. Detect correspondences: Find matching features between camera and LiDAR
2. Estimate transform: Use RANSAC or optimization to find best transform
3. Monitor drift: Track calibration over time, detect when it drifts

The Math:

Correspondence-based:

Given correspondences $(X_i^{\text{LiDAR}}, x_i^{\text{camera}})$:

RANSAC: Robustly handle outliers (false correspondences).

Pros:

• No calibration target needed
• Handles calibration drift
• Can run continuously

Cons:

• Less accurate than offline (centimeter-level)
• Requires good correspondences
• Computationally expensive

Production Practice: A hybrid approach is commonly used:

• Offline calibration at factory (high accuracy baseline)
• Online calibration in the field (monitor and correct drift)

Time Synchronization: PTP and Timestamps

The Problem: Different sensors capture data at different times. If you fuse camera and LiDAR data, but the camera image is from time $t$ and the LiDAR scan is from time $t + 50$ms, you’re fusing data from different moments.

The Real-World Twist: At 30 mph, in 50ms you travel 2.2 feet. If you fuse a camera image with a LiDAR scan that’s 50ms later, objects will be misaligned by 2.2 feet.

PTP (Precision Time Protocol)

The Setup: All sensors synchronize to a master clock using PTP (IEEE 1588).

The Math:

Clock Synchronization:

Where:

• $\text{offset}$ = clock offset (measured via PTP)
• $\text{delay}$ = network delay (measured via PTP)

PTP achieves microsecond-level synchronization — good enough for sensor fusion.

Timestamping

The Process:

1. Hardware timestamping: Each sensor timestamps data at capture time (not processing time)
2. PTP synchronization: All timestamps are in the same time reference
3. Temporal alignment: When fusing, align data by timestamp (not by arrival time)

The Challenge: Different sensors have different latencies:

• Camera: 10-20ms (readout + processing)
• LiDAR: 50-100ms (scan time)
• Radar: 5-10ms (processing)

The Solution: Predict forward or interpolate backward to align timestamps.

Example: If camera image is at $t$ and LiDAR scan is at $t + 50$ms:

• Option 1: Predict LiDAR points forward to time $t$ (using motion model)
• Option 2: Interpolate camera image backward to time $t + 50$ms (not possible, so use closest frame)

The Intuition: Laser Pointer on a Unicycle

The Analogy: You’re holding a laser pointer while riding a unicycle. The laser pointer jitters. Is the jitter because:

1. Your hand is shaking? (sensor noise)
2. The unicycle is wobbling? (vehicle motion)
3. Both? (combined effect)

The Calibration Problem: Similarly, if a LiDAR point appears to move, is it because:

1. The LiDAR measurement is noisy? (sensor noise)
2. The vehicle is moving? (ego motion)
3. The calibration is wrong? (extrinsic error)
4. All of the above? (combined effect)

The Solution: Motion compensation — account for vehicle motion, then analyze residual error to detect calibration drift.

The Math:

Motion Compensation:

Where $T_{\text{vehicle}}(t)$ accounts for vehicle motion between timesteps.

Calibration Monitoring:

If calibration is correct, after motion compensation, corresponding points from camera and LiDAR should align. If they don’t, calibration has drifted.

Summary: The Bedrock of Perception

Calibration is the foundation of sensor fusion:

1. Intrinsics: Know how each sensor maps the world to measurements
2. Extrinsics: Know where sensors are relative to each other
3. Time sync: Know when each sensor captured its data
4. Monitoring: Continuously verify calibration hasn’t drifted

The Path Forward:

With calibrated sensors, we can now:

• Fuse sensor data (Module 6)
• Localize the vehicle (Module 4)
• Detect objects (Module 5)
• Track them over time (Module 6)

Graduate Assignment: Camera-LiDAR Calibration

Task: Implement offline calibration between a camera and LiDAR.

Setup:

• Calibration target: Checkerboard (known 3D positions)
• Data: Camera images + LiDAR point clouds of the checkerboard

Deliverables:

1. Extract checkerboard corners from camera images
2. Extract checkerboard points from LiDAR scans
3. Implement optimization to estimate $R, t$ (extrinsics) and $K$ (intrinsics)
4. Visualize: Project LiDAR points onto camera image — do they align?

Extension: Implement online calibration using natural scene correspondences.

Further Reading

• Module 2: Eyes and Ears (Sensors)
• Module 6: Merging Senses (Sensor Fusion & Tracking)
• AutoCalib Research: Automatic Camera Calibration

This is Module 3 of “The Ghost in the Machine” series. Module 4 will explore localization — knowing where you are in the world with centimeter-level accuracy.