Every deep-learning-heavy vision project we are called into eventually hits the same wall: the model works, and the numbers still lie. The bounding box is correct but the part measures 41.2 mm instead of 40.0 mm. The stereo disparity map is beautiful and the depth is off by 8% at the far end of the conveyor. Almost always the cause is not the network — it is calibration.
This tutorial covers the geometry side of OpenCV, which changed shape in OpenCV 5, and walks a full monocular and stereo calibration through to a depth number you can defend in a tolerance report.
1. Where calibration lives in OpenCV 5
In OpenCV 4.x everything geometric lived in calib3d. OpenCV 5 splits that surface: the general 3D vision primitives moved into the new 3d module, while calibration-specific functionality sits in calib, and the ArUco/ChArUco detectors are in objdetect (they left contrib back in 4.7). If you are porting 4.x code, that means new headers and new CMake component names:
// OpenCV 4.x
#include <opencv2/calib3d.hpp>
// OpenCV 5.x
#include <opencv2/3d.hpp> // undistort, rectify, PnP-adjacent 3D utilities
#include <opencv2/calib.hpp> // calibrateCamera, stereoCalibrate
#include <opencv2/objdetect/charuco_detector.hpp>
In Python the flat cv2. namespace hides most of this — cv2.calibrateCamera still resolves — but the ArUco API is the modern object-oriented one (cv2.aruco.CharucoDetector), not the deprecated free functions. If your calibration script still calls cv2.aruco.interpolateCornersCharuco, it is written against an API that no longer exists.
See our OpenCV 4.x to 5.0 migration checklist for the wider module map.
2. Use a ChArUco board, not a chessboard
A plain chessboard requires the entire board to be visible and unoccluded in every frame. A ChArUco board — a chessboard with ArUco markers in the white squares — gives each interior corner a unique identity, so partial views still contribute. That single property is why ChArUco wins in practice: you can push the board into the frame corners, where lens distortion actually lives, without losing the detection.
import cv2
import numpy as np
SQUARES_X, SQUARES_Y = 9, 6
SQUARE_LEN = 0.030 # metres, measured on the printed board
MARKER_LEN = 0.022 # metres
dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_5X5_100)
board = cv2.aruco.CharucoBoard((SQUARES_X, SQUARES_Y), SQUARE_LEN, MARKER_LEN, dictionary)
# Generate a print-ready image at 300 DPI
img = board.generateImage((int(SQUARES_X * SQUARE_LEN * 300 / 0.0254),
int(SQUARES_Y * SQUARE_LEN * 300 / 0.0254)), marginSize=40)
cv2.imwrite("charuco_9x6.png", img)
Three rules about the physical board that cost more projects than any code bug:
- Print flat and mount rigid. Tape on cardboard warps. Use foam board, aluminium composite, or an ordered calibration target. A 1 mm bow across an A3 board is a systematic error you cannot fit away.
- Measure the printed square, do not trust the printer. Printers scale. Put callipers on ten squares and average.
SQUARE_LENis the only thing that ties your calibration to real-world units; if it is 2% wrong, every distance you ever compute is 2% wrong. - Match the board scale to the working distance. The board should fill roughly a third to a half of the frame at your real operating distance.
3. Capture protocol
A good capture set is not "lots of images", it is diverse images. Aim for 20–40 accepted frames covering:
- board centred, then in all four corners of the frame,
- tilted roughly ±30–45° about both axes (this is what separates focal length from distance — without tilt the calibration is degenerate),
- near and far within your depth of field,
- rotated in-plane.
Lock the lens. Autofocus, image stabilisation and any auto-zoom must be off, and the aperture and focus ring taped, because focus changes the effective focal length. Recalibrate any time the lens is touched.
all_corners, all_ids, image_size = [], [], None
detector = cv2.aruco.CharucoDetector(board)
for path in sorted(glob.glob("captures/*.png")):
frame = cv2.imread(path)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
image_size = gray.shape[::-1]
ch_corners, ch_ids, _, _ = detector.detectBoard(gray)
if ch_corners is not None and len(ch_corners) >= 12:
all_corners.append(ch_corners)
all_ids.append(ch_ids)
else:
print(f"rejected {path}: {0 if ch_ids is None else len(ch_ids)} corners")
print(f"{len(all_corners)} usable views")
Reject frames with fewer than ~12 identified corners and any frame with visible motion blur. Blurred corners do not fail loudly; they just bias the result. Capture on a tripod with the board moving, or with a global-shutter camera — rolling shutter plus a moving board is its own error source.
4. Run the calibration and read the numbers honestly
flags = cv2.CALIB_RATIONAL_MODEL # k4..k6, useful for wide lenses
rms, K, dist, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco(
all_corners, all_ids, board, image_size, None, None, flags=flags)
print("RMS reprojection error (px):", rms)
print("fx, fy:", K[0, 0], K[1, 1])
print("cx, cy:", K[0, 2], K[1, 2])
How to interpret it:
- RMS below ~0.5 px is a healthy result for a decent industrial camera and a rigid board. Between 0.5 and 1.0 px, look for board flex or blur. Above 1.0 px, something is structurally wrong.
- A suspiciously low RMS (say 0.05 px) with 8 images is not a good calibration. It is an overfit. More free distortion parameters always lower RMS; they do not always improve accuracy.
- Sanity-check the principal point.
cx, cyshould land near the image centre, within a few percent of width. A principal point 200 px off centre on a 1280-wide sensor means your capture set lacked tilt diversity and the optimiser traded focal length against position. - Check fx ≈ fy for a square-pixel sensor. A ratio more than ~1% from 1.0 is a red flag unless you genuinely have non-square pixels.
Then look at per-view error, not just the global RMS, and drop the worst offenders before refitting:
errors = []
for i, (c, ids) in enumerate(zip(all_corners, all_ids)):
obj_pts = board.getChessboardCorners()[ids.flatten()]
proj, _ = cv2.projectPoints(obj_pts, rvecs[i], tvecs[i], K, dist)
e = cv2.norm(c.reshape(-1, 2), proj.reshape(-1, 2), cv2.NORM_L2) / len(proj)
errors.append(e)
worst = np.argsort(errors)[-3:]
print("worst views:", [(int(i), round(errors[i], 3)) for i in worst])
A final, non-negotiable step: validate on a held-out target of known size. Photograph a gauge block or a printed ruler at the working distance, undistort, measure in pixels, convert, and compare against the true dimension. Reprojection error measures how well the model fits the data you gave it. Only an independent measurement tells you whether it is right.
5. Fisheye and wide-angle lenses
Beyond roughly 120° field of view, the standard Brown–Conrady polynomial model stops behaving — you will see RMS creep up and undistorted straight lines bend near the edges. Switch to the fisheye (equidistant) model:
rms, K, D, _, _ = cv2.fisheye.calibrate(
obj_points, img_points, image_size, None, None,
flags=cv2.fisheye.CALIB_RECOMPUTE_EXTRINSIC | cv2.fisheye.CALIB_FIX_SKEW,
criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-6))
The fisheye functions want object points shaped (N, 1, 3) and image points (N, 1, 2) in float64; shape mismatches here produce cryptic assertion failures rather than useful messages. When undistorting, use cv2.fisheye.estimateNewCameraMatrixForUndistortRectify with a balance parameter — balance=0 crops to valid pixels only, balance=1 keeps the full field with black wedges.
6. Stereo: rectify, then measure
For stereo depth, calibrate each camera individually first, then fix those intrinsics while solving for the relative pose:
flags = cv2.CALIB_FIX_INTRINSIC
rms, K1, d1, K2, d2, R, T, E, F = cv2.stereoCalibrate(
obj_points, img_points_l, img_points_r, K1, d1, K2, d2, image_size,
flags=flags,
criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-5))
print("stereo RMS:", rms, "baseline (mm):", np.linalg.norm(T) * 1000)
R1, R2, P1, P2, Q, roi1, roi2 = cv2.stereoRectify(
K1, d1, K2, d2, image_size, R, T, alpha=0)
map1x, map1y = cv2.initUndistortRectifyMap(K1, d1, R1, P1, image_size, cv2.CV_16SC2)
map2x, map2y = cv2.initUndistortRectifyMap(K2, d2, R2, P2, image_size, cv2.CV_16SC2)
The printed baseline is your first sanity check: it must match the physical distance between the lens centres to within a millimetre or two. If it does not, the calibration is wrong regardless of what the RMS says.
Verify rectification visually before touching a matcher — remap both images, stack them side by side, and draw horizontal lines. Corresponding features must sit on the same row. If they do not, disparity search is looking in the wrong place and no amount of matcher tuning will save it.
Then match and reproject:
matcher = cv2.StereoSGBM_create(
minDisparity=0, numDisparities=128, blockSize=5,
P1=8 * 3 * 5 ** 2, P2=32 * 3 * 5 ** 2,
uniquenessRatio=10, speckleWindowSize=100, speckleRange=2,
mode=cv2.STEREO_SGBM_MODE_SGBM_3WAY)
left_r = cv2.remap(left, map1x, map1y, cv2.INTER_LINEAR)
right_r = cv2.remap(right, map2x, map2y, cv2.INTER_LINEAR)
disp = matcher.compute(left_r, right_r).astype(np.float32) / 16.0
points_3d = cv2.reprojectImageTo3D(disp, Q) # metres, if SQUARE_LEN was in metres
7. The depth error budget
Before promising an accuracy figure to a client, do the arithmetic. Stereo depth error grows with the square of range:
Z = f * B / d
dZ = Z^2 / (f * B) * dd
where f is focal length in pixels, B the baseline in metres, d disparity in pixels and dd your disparity uncertainty (realistically 0.2–0.5 px for SGBM on textured surfaces, worse on bland ones).
Worked example: f = 1400 px, B = 0.12 m, dd = 0.25 px. At 2 m the depth uncertainty is 2² × 0.25 / (1400 × 0.12) ≈ 6 mm. At 5 m it is ≈ 37 mm. If the requirement is ±10 mm at 5 m, no amount of software tuning gets you there — you need a longer baseline, a longer lens, or a different sensing modality. That conversation is far cheaper in week one than in month four.
Two practical follow-ups: on low-texture surfaces add a pattern projector, and stabilise thermally — a stereo rig's baseline drifts as the housing warms, so let cameras reach thermal equilibrium before calibrating, and re-verify extrinsics on a schedule for anything vibrating or vehicle-mounted.
8. Treat calibration as a versioned artefact
Calibration is not a one-off ritual performed by whoever set up the rig. Store the result like code:
fs = cv2.FileStorage("calib_cam0_v3.yml", cv2.FILE_STORAGE_WRITE)
fs.write("camera_matrix", K); fs.write("dist_coeffs", dist)
fs.write("image_size", np.array(image_size))
fs.write("rms", rms); fs.write("square_len_m", SQUARE_LEN)
fs.write("captured_utc", datetime.utcnow().isoformat())
fs.write("lens_serial", "LN-88213")
fs.release()
Bind each file to a camera and lens serial number, keep the raw capture images so it can be re-fit later, and add a startup check that refuses to run if the loaded image size does not match the live stream. Swapping a camera without swapping the calibration file is one of the most common silent failures in deployed vision systems — and because the pipeline keeps producing plausible-looking output, it can run wrong for months.
Where this fits
Detection tells you what and where in the image. Calibration is what turns that into where in the world, and it is the part that decides whether a measurement passes a tolerance audit. If your project involves dimensional measurement, robot guidance, multi-camera fusion or stereo depth, get the geometry right before optimising a single millisecond of inference.
SentientSight's OpenCV consultants build and audit calibration pipelines — target design, capture protocols, accuracy budgets and drift monitoring — for measurement and robotics systems in production. If you need a depth or measurement accuracy claim you can stand behind, get in touch.