What is the difference between image plane coordinates and pixel coordinates?

Image plane coordinates are continuous and centered at the image origin. Pixel coordinates are discrete, typically starting from a corner of the image, and require scaling and rounding from the continuous image plane coordinates.

What are homogeneous coordinates and why are they used in computer vision?

Homogeneous coordinates add an extra dimension to standard Cartesian coordinates, allowing non-linear transformations like projection to be represented as linear transformations (matrix multiplications). This simplifies calculations, especially for camera calibration.

What is the camera matrix (K) and what parameters does it contain?

The camera matrix, often denoted as K, is a 3x3 matrix containing the intrinsic parameters of the camera. These include the focal length (related to alpha and beta) and the coordinates of the principal point (U0, V0).

How does the transformation from world coordinates to camera coordinates work?

The transformation involves rotation (R) and translation (T). If the origins coincide, it's pure rotation. If they differ, it's rotation followed by translation, representing the camera's pose relative to the world.

How is camera calibration performed in practice?

Camera calibration involves using a known pattern (like a chessboard) with a defined world coordinate system. By observing the projected pattern in images, corresponding 3D-to-2D point pairs are used to estimate the unknown intrinsic (K) and extrinsic (R, T) parameters.

What is the minimum number of correspondences needed for camera calibration?

To accurately estimate the 11 degrees of freedom in the camera projection model (excluding the scale factor), at least six correspondences are theoretically needed. In practice, more correspondences are used to improve accuracy and account for measurement errors.

How are the rotation (R) and translation (T) components extracted from the M matrix?

The M matrix is decomposed using techniques like RQ decomposition. The R component is found to be the rotation matrix, and the translation vector T is derived by inverting the relationship between the scaled M matrix and K.

What is lens distortion and why is it important?

Lens distortion, such as barrel or pincushion distortion, occurs when magnification varies with distance from the image center. While often small, it can affect calibration accuracy and requires an additional parameter to be estimated.

What is stereo vision and how does it help reconstruct 3D scenes?

Stereo vision uses two cameras with a known relative pose to capture slightly different views of a scene. By finding corresponding points in both images and using triangulation, the 3D position of points can be accurately determined.

Key Moments

Stanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Camera Models & Calibration

Q: Why can't we directly recover a 3D point from a single 2D image?

A single 2D image suffers from inherent depth ambiguity. Any point along the ray from the camera's optical center through the projected 2D point could be the true 3D location, making precise depth recovery impossible without additional information.

Stanford Online

Education6 min read79 min video

Oct 16, 2026|4,083 views|23

Stanford Stanford Online Robotics

Save to Pod

Key Moments

TL;DR

Camera calibration is a tedious but essential process for robots to understand their environment, requiring precise internal and external parameters to accurately reconstruct 3D scenes from 2D images.

Key Insights

The process of reconstructing a 3D world from 2D images involves three steps: projecting a real-world point to the image plane, converting to pixel coordinates, and relating world-frame to camera-frame.

Camera calibration is crucial because manufacturing specifications for internal parameters like focal length and pixel size are often insufficiently accurate for robotic applications.

Homogeneous coordinates are a mathematical trick used to convert non-linear projection transformations into linear transformations, simplifying calculations in computer vision.

The camera matrix (K) encapsulates intrinsic camera parameters such as focal length (alpha, beta) and image plane origin (U0, V0), which must generally be estimated through calibration.

Camera calibration involves estimating the projection matrix (M), which is then decomposed into intrinsic (K), rotation (R), and translation (T) parameters, requiring at least six point correspondences.

Reconstructing 3D structure from 2D images is inherently ambiguous due to depth ambiguity; methods like stereo vision or structure from motion are needed to overcome this.

The fundamental problem of mapping 3D to 2D and back

The core challenge in computer vision for robotics is establishing a relationship between points in the real world and their projection onto a 2D image plane. This relationship is critical for enabling robots to reconstruct and understand their 3D environment from the 2D images they capture. This process is typically broken down into three sequential steps: first, understanding how a point P, expressed in the camera's reference frame, projects onto the image plane; second, converting this projected point into pixel coordinates as used by the camera; and third, relating the point P, expressed in the camera frame, to its representation in a global world frame. The ultimate goal is to work backward from images to reconstruct the 3D world.

Why camera calibration is a necessary, albeit tedious, step

While modern cameras provide pixel coordinates, this raw data is insufficient for tasks like 3D reconstruction. The process requires accurate internal camera parameters such as focal length and pixel resolution. However, manufacturer specifications for these parameters are often imprecise due to manufacturing variations. Therefore, any robotic system using a camera for environmental reconstruction must perform a calibration process. This calibration is described as one of the most bothersome and time-consuming operations, with personal anecdotes highlighting its challenging nature, even in professional settings like NASA.

From image plane to pixel coordinates: accounting for offsets and scaling

After projecting a 3D point onto the image plane, the next step is to map these continuous image plane coordinates to discrete pixel coordinates. This involves two primary considerations: firstly, the camera's coordinate system is typically not centered on the image plane but rather at one of its corners (e.g., bottom-right or top-left). This requires applying an offset based on the coordinates of the image plane's origin relative to the camera's coordinate system. Secondly, pixel coordinates are read as rows and columns. To convert image plane coordinates to pixel coordinates, one must account for the number of pixels per unit distance (e.g., pixels per centimeter). Multiplying the image plane coordinate by this density (e.g., Kx for the x-axis and Ky for the y-axis) and then rounding converts the measurement into pixel units. This process inherently involves some loss of information due to quantization, where a range of continuous values maps to a single pixel.

Leveraging homogeneous coordinates to linearize projections

The projection transformation from 3D to 2D is inherently non-linear because it involves division by the Z-coordinate (depth). This non-linearity poses a mathematical challenge, as it prevents the transformation from being represented as a simple matrix-vector multiplication. A common technique in computer vision to overcome this is the use of homogeneous coordinates. By adding an extra dimension, 3D points (X, Y, Z) are represented as (X, Y, Z, 1) or more generally as (λX, λY, λZ, λ). This mathematical construct allows non-linear projective transformations to be expressed as linear transformations (matrix multiplication) in a higher-dimensional space. The non-linearity is reintroduced only when converting back from homogeneous to non-homogeneous coordinates by dividing by the last component. This approach simplifies calculations and is particularly useful for camera calibration.

The camera matrix: intrinsic parameters and their estimation

The 3x3 camera matrix (K) encapsulates the intrinsic parameters of a camera, which are properties related to its manufacturing. These include parameters like alpha (related to focal length and pixel scale in the x-direction) and beta (similarly for the y-direction), and U0, V0, which represent the principal point or the origin of the pixel coordinate system within the image plane. Often, a skew coefficient (gamma) is also considered, accounting for non-orthogonality of the camera's reference frame axes, though it's frequently assumed to be zero for simplicity in many applications. These parameters are not precisely known and must generally be estimated from data through a calibration procedure.

Transforming between world and camera coordinate frames

To fully define the projection, we need to relate the point's coordinates in the global world frame to its coordinates in the camera's frame. This transformation involves two components: rotation (R) and translation (T). The rotation matrix R describes the orientation of the camera's coordinate system relative to the world coordinate system, effectively capturing how the camera is rotated. The translation vector T describes the position of the world origin relative to the camera origin. If the origins coincide, the transformation is purely rotational. If they differ, it involves both rotation and translation. The combined transformation allows coordinates of a point in the world frame (Pw) to be converted to coordinates in the camera frame (Pc) using the formula Pc = R * Pw + T. This relative pose information (R and T) is crucial for tasks like visual odometry or SLAM, where the robot estimates its movement based on camera views.

Decomposing the projection matrix for calibration

The overall transformation from a 3D point in the world frame to its pixel coordinates can be expressed as a single matrix multiplication in homogeneous coordinates: P_pixel_homogeneous = K * [R | T] * P_world_homogeneous. Here, [R | T] is the extrinsic camera matrix, combining rotation and translation, and K is the intrinsic camera matrix. The calibration process aims to estimate this complete projection matrix, often denoted as M (a 3x4 matrix). This M matrix is then decomposed to recover the individual K, R, and T parameters. This decomposition typically involves estimating M from multiple known 3D-to-2D point correspondences and then using techniques like QR or RQ decomposition to separate the intrinsic parameters (K) from the extrinsic parameters (R and T).

Challenges in 3D reconstruction and alternative methods

Even with a calibrated camera and its projection matrix, recovering the 3D structure of a scene from a single 2D image is fundamentally ambiguous. Any point along the ray extending from the camera's optical center through the projected 2D point could be the actual 3D point. This 'depth ambiguity' means a single image alone cannot determine the size or distance of objects. To overcome this, several methods are employed: semantic understanding (using prior knowledge about object sizes), depth from focus (analyzing image sharpness at different focal planes), stereo vision (using two cameras to triangulate points), and structure from motion (recovering 3D structure from a sequence of images taken from different viewpoints). Stereo vision is a common approach, utilizing the parallax between two known camera positions to triangulate 3D points. The accuracy relies heavily on precise camera calibration and reliable matching of corresponding points between the stereo images.

Mentioned in This Episode

●Software & Apps

●Organizations

Common Questions

Modern cameras provide pixel data, but the specifications for internal parameters like focal length are often inaccurate due to manufacturing tolerances. Calibration is crucial for robots to accurately reconstruct environments using camera data.

Topics

AI & Machine Learning Technology & Innovation Science & Mathematics 3D Reconstruction Computer Vision Camera Models Camera Calibration Homogeneous Coordinates Projection Matrix Stereo Vision

Mentioned in this video

Organizations

NASA

The speaker shares personal anecdotes about calibrating cameras while working at NASA, highlighting the challenging nature of the process.

Software & Apps

NumPy

A Python library mentioned for performing matrix decompositions like RQ analysis.

Locations

Pisa

Used as an example to illustrate the ambiguity of depth from a single 2D image.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free