Nobody asks for a panorama. What clients actually ask for is: "we scan a 4-metre composite panel with a camera on a gantry and we need one image an inspector can annotate", or "the drone flies the roof in 180 shots and we need a single orthophoto with defect locations in metres", or "the web moves at 90 m/min and we want a continuous roll image per batch". All three are the same problem: mosaicking — registering many overlapping frames into one geometrically consistent image, with a known pixel-to-millimetre scale.
OpenCV has shipped a stitching pipeline for over a decade, and most teams meet it through cv2.Stitcher, get a beautiful holiday panorama, then discover it fails silently on their low-texture panels or produces a mosaic that is pretty but not measurable. This tutorial unpacks the pipeline stage by stage in OpenCV 5, swaps the classical matcher for a learned one via ONNX where it earns its keep, and ends with the accuracy budget you need before you quote a defect-location tolerance.
1. Decide first: is this a panorama or a map?
The single biggest source of wasted weeks. Two very different jobs wear the same clothes.
| Panorama (rotational) | Mosaic / map (planar) | |
|---|---|---|
| Camera motion | Rotates about the optical centre | Translates across a flat-ish surface |
| Warp | Spherical / cylindrical | Homography or affine onto a plane |
cv2.Stitcher mode | Stitcher_PANORAMA | Stitcher_SCANS |
| Scale | Meaningless (angular) | Millimetres per pixel — the whole point |
| Typical use | Site context imagery | Inspection, web scanning, drone mapping |
If a client wants to measure anything on the output, you are in the right-hand column and you must use Stitcher_SCANS (affine, no exposure-driven resizing surprises) or hand-build the pipeline. Using PANORAMA mode on a translating gantry camera is the classic failure: it estimates a rotation-only model for translational motion, and the seams drift.
And a third case that is neither: if the surface is genuinely 3D (a turbine blade, a walkaround of a vehicle), stitching is the wrong tool entirely — you want structure-from-motion. See walkaround video to a measurable 3D model.
2. The five-minute baseline you should always run first
Before building anything, run the stock pipeline. If it works, you have just saved the client a fortnight.
import cv2, glob
imgs = [cv2.imread(p) for p in sorted(glob.glob("captures/*.png"))]
stitcher = cv2.Stitcher.create(cv2.Stitcher_SCANS)
status, pano = stitcher.stitch(imgs)
if status == cv2.STITCHER_OK:
cv2.imwrite("mosaic.png", pano)
else:
print({
cv2.STITCHER_ERR_NEED_MORE_IMGS: "not enough overlap / features",
cv2.STITCHER_ERR_HOMOGRAPHY_EST_FAIL: "match found but model estimation failed",
cv2.STITCHER_ERR_CAMERA_PARAMS_ADJUST_FAIL: "bundle adjustment diverged",
}.get(status, f"unknown status {status}"))
Two notes for OpenCV 5. The stitching module still exists and the high-level Stitcher API is stable, but the geometry underneath moved: calib3d was split into the new 3d and calib modules, so C++ code that included opencv2/calib3d.hpp for findHomography now wants opencv2/3d.hpp. And Stitcher::stitch no longer accepts the old two-argument masks overload in some builds — check your header, do not trust a 4.x snippet. The broader map is in our 4.x to 5.0 migration checklist.
STITCHER_ERR_NEED_MORE_IMGS is the status you will see most, and it almost never means "add more images". It means the feature matcher could not link the set. That is a capture problem or a matcher problem, and the rest of this tutorial is about both.
3. Capture protocol: where mosaicking projects are won
Software cannot recover information the capture never had. Fix these before writing code.
- Overlap: 30–40% between adjacent frames, minimum 25%. Below ~20% you are relying on a thin strip of features and a single bad frame breaks the chain. Above 60% you are paying compute for nothing. For drone grids, 70–80% forward / 60–70% side overlap is standard because the surface is not perfectly planar.
- Lock exposure, white balance, focus and gain. Auto-exposure across a mosaic gives you banding that the exposure compensator then has to guess its way out of. Manual, fixed, verified. Our input-quality gate tutorial has the settings checklist.
- Global shutter for anything moving. Rolling shutter turns a translating frame into a sheared frame, and a sheared frame does not fit a homography. This is non-negotiable for web scanning.
- Diffuse, constant illumination. Specular highlights move with the camera, so the matcher sees them as features that translate independently of the surface — a reliable way to poison a model estimate. Cross-polarise if the surface is glossy.
- Add texture to blank surfaces. A white composite panel or a plain steel sheet has no features. Options: a projected speckle pattern (cheap, effective), fiducial markers around the region of interest, or — best of all — encoder/robot pose as a prior so you do not need features to find the rough alignment at all.
- Fly or scan a serpentine grid with cross-strips. A single long strip accumulates drift with nothing to close it against. A boustrophedon pattern plus one or two perpendicular passes gives bundle adjustment loop constraints, and drift collapses.
Write this down as a capture spec and hand it to whoever operates the rig. Half of "the stitching is broken" tickets are capture-spec violations.
4. Building the pipeline by hand
When the stock stitcher fails, do not tweak it blindly — unpack it. The detail namespace exposes every stage.
4.1 Features and matching, classical
import numpy as np
detector = cv2.SIFT_create(nfeatures=4000)
def features(img):
g = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
g = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(g) # helps low contrast
return detector.detectAndCompute(g, None)
kp_a, des_a = features(imgs[0])
kp_b, des_b = features(imgs[1])
matcher = cv2.FlannBasedMatcher(dict(algorithm=1, trees=5), dict(checks=64))
raw = matcher.knnMatch(des_a, des_b, k=2)
good = [m for m, n in raw if m.distance < 0.75 * n.distance] # Lowe ratio
print(f"{len(good)} good matches of {len(raw)}")
SIFT is patent-free and in the main modules since 4.4 — use it as the default for mosaicking. ORB is faster and fine for high-texture surfaces at high frame rates, but its descriptor is noticeably weaker across scale and illumination change, which is exactly what you get on a curved or unevenly lit panel.
Rule of thumb: fewer than ~30 inlier matches between a pair and you should not trust the homography, however good the reprojection error looks.
4.2 Estimate the right model, not the most general one
src = np.float32([kp_a[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
dst = np.float32([kp_b[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
H, mask = cv2.findHomography(src, dst, cv2.USAC_MAGSAC, ransacReprojThreshold=3.0,
maxIters=10000, confidence=0.9999)
inliers = int(mask.sum())
print("inliers:", inliers, "ratio:", round(inliers / len(good), 2))
Two things matter here.
Use USAC_MAGSAC, not plain RANSAC. The USAC family has been in OpenCV since 4.5 and is strictly better behaved — it is less threshold-sensitive and degrades more gracefully on contaminated match sets. On marginal inspection data the difference between RANSAC and USAC_MAGSAC is often the difference between a mosaic and an error status.
Constrain the model to the physics. A full homography has 8 degrees of freedom. A camera translating on a rigid gantry over a flat panel has 2 (or 4 if you allow small rotation and scale). Fitting 8 DoF to 2 DoF of real motion lets noise express itself as spurious perspective, and the mosaic visibly keystones. Prefer:
M, mask = cv2.estimateAffinePartial2D(src, dst, method=cv2.RANSAC,
ransacReprojThreshold=3.0) # 4 DoF: t, R, s
for gantry and web scanning. Reserve the homography for handheld and drone capture. Model choice is the highest-leverage decision in the whole pipeline and it is one line of code.
4.3 Learned matchers: when SIFT is not enough
This is where mosaicking has actually moved in the last two years. SuperPoint + LightGlue (and the SuperPoint/DISK/ALIKED family generally) substantially outperform SIFT + ratio test on low-texture, repetitive-texture and large-viewpoint-change pairs. Both export cleanly to ONNX, which means OpenCV's DNN module — rebuilt in OpenCV 5 with a new engine and better ONNX coverage — can run them inside your existing pipeline with no extra runtime dependency.
# SuperPoint: 1x1xHxW grayscale float32 in [0,1]; outputs keypoints, scores, descriptors
sp = cv2.dnn.readNetFromONNX("superpoint.onnx")
def superpoint(img, size=(1024, 1024)):
g = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
h0, w0 = g.shape
g = cv2.resize(g, size)
blob = (g.astype(np.float32) / 255.0)[None, None]
sp.setInput(blob)
kpts, scores, desc = sp.forward(sp.getUnconnectedOutLayersNames())
kpts = kpts.reshape(-1, 2) * [w0 / size[0], h0 / size[1]] # back to original pixels
return kpts.astype(np.float32), desc.reshape(kpts.shape[0], -1)
# LightGlue takes both keypoint sets + descriptors and returns matches directly
lg = cv2.dnn.readNetFromONNX("lightglue.onnx")
Honest caveats, because this is where people lose a week:
- Rescale keypoints back to original image coordinates. Both networks want a fixed input size; your homography must be estimated in the original pixel frame or every downstream millimetre number is wrong by the resize ratio.
- Dynamic shapes are the failure mode. LightGlue ONNX graphs with dynamic keypoint counts can hit unsupported-op errors. Export with a fixed maximum keypoint count and pad, or run the matcher in ONNX Runtime alongside OpenCV for the rest of the pipeline. Validate the exact export against your OpenCV build before you design around it.
- Budget the cost. SuperPoint + LightGlue on a 1024×1024 pair is tens of milliseconds on a decent GPU and several hundred on CPU. For a 200-frame drone set run offline that is irrelevant. For a real-time web scanner it is not — use classical features for the live path and the learned matcher only for the offline refinement or for pairs that failed.
- Try SIFT first. If SIFT gives 400 inliers on your data, a learned matcher buys you nothing but dependencies. Reach for it when inlier counts are marginal, not as a default.
4.4 Global alignment: bundle adjustment and drift
Chaining pairwise transforms is the second classic failure. Frame 50's position is the product of 49 estimates, each with a fraction of a pixel of error, and a 200-frame strip ends up metres out of place. You must estimate all camera poses jointly:
from cv2 import detail
matcher = detail.BestOf2NearestMatcher_create(try_use_gpu=False, match_conf=0.3)
matches = matcher.apply2(feats)
matcher.collectGarbage()
indices = detail.leaveBiggestComponent(feats, matches, 0.3)
print("frames retained:", len(indices), "of", len(imgs))
estimator = detail.AffineBasedEstimator()
ok, cameras = estimator.apply(feats, matches, None)
adjuster = detail.BundleAdjusterAffinePartial()
adjuster.setConfThresh(1.0)
ok = adjuster.apply(feats, matches, cameras)
leaveBiggestComponent deserves attention: it silently drops frames that could not be linked. If it retains 140 of 200 frames, your mosaic has a hole and the pipeline will not tell you — it will just produce a smaller mosaic. Always log retained-versus-submitted counts and fail the job above a threshold. Silent partial output is worse than an error, because an inspector will happily sign off a panel whose defective third was never in the image.
For drone and vehicle capture, feed GNSS/RTK or robot-encoder pose in as a prior and as a constraint. Features get you relative alignment; only an external measurement stops global drift and gives you real-world coordinates.
4.5 Seams, exposure and blending
compensator = detail.ExposureCompensator_createDefault(detail.ExposureCompensator_GAIN_BLOCKS)
compensator.feed(corners, warped_imgs, warped_masks)
seam_finder = detail.SeamFinder_createDefault(detail.SeamFinder_DP_SEAM)
seam_finder.find([w.astype(np.float32) for w in warped_imgs], corners, warped_masks)
blender = detail.Blender_createDefault(detail.Blender_MULTI_BAND)
blender.prepare(dst_roi)
for img, mask, corner in zip(warped_imgs, warped_masks, corners):
blender.feed(img.astype(np.int16), mask, corner)
result, result_mask = blender.blend(None, None)
Critical judgement call for inspection work: multi-band blending makes mosaics look good and can make defects disappear. A hairline crack or a 3-pixel scratch that falls on a seam gets feathered into the background. If the mosaic is the input to a human inspector or a defect detector, prefer Blender_NO or a hard DP_SEAM cut with no feathering, accept the visible seam lines, and keep per-frame provenance so any finding can be traced back to the original frame. Pretty and diagnostic are different products; ask which one the client is buying.
Run defect detection on the original frames wherever you can, and use the mosaic only to place findings in panel coordinates. Warping resamples, resampling blurs, and blurring costs you small defects. That also keeps your detector operating on the image statistics it was trained on — see unsupervised defect detection.
5. From pixels to millimetres, and the accuracy you can promise
A mosaic without a scale is a picture. With one, it is a measurement instrument.
Ground sample distance is where every quote starts:
GSD (mm/px) = sensor_pixel_pitch_mm * working_distance_mm / focal_length_mm
Worked example, gantry inspection: 3.45 µm pitch, 16 mm lens, 600 mm standoff → GSD = 0.00345 × 600 / 16 ≈ 0.13 mm/px. A 5-pixel defect is therefore about 0.65 mm, and to resolve a 0.3 mm crack reliably (3+ pixels across) you need roughly 0.1 mm/px — so this configuration is borderline and the honest answer is "move to 400 mm standoff or a 25 mm lens", not "we will try".
Then the error budget for a defect position in panel coordinates:
- Residual mosaic alignment error: the RMS reprojection error from bundle adjustment, typically 0.3–1.5 px for a well-captured planar set. Read it, do not assume it.
- Scale error: comes from your calibration and standoff measurement. A 1% standoff error is a 1% error on every distance across a 4 m panel — that is 40 mm. Measure the standoff, or better, put two known-separation fiducials in the scene and solve the scale from the mosaic itself.
- Lens distortion residual: undistort every frame before stitching. Stitching distorted frames forces the homography to absorb radial distortion, and it cannot — you get systematic bowing near frame edges. Get the intrinsics from a proper ChArUco calibration.
- Non-planarity: a homography assumes a plane. A panel bowed by 5 mm at a 600 mm standoff introduces roughly a 0.8% local parallax error. For anything meaningfully 3D, switch to SfM.
Combine in quadrature, then validate against ground truth: put two fiducials a calliper-measured distance apart at opposite corners of the scan area, measure that distance in the mosaic, and report the delta. That single number — "distance error 1.8 mm over a 3.2 m span, n = 20 scans" — is what a client can put in an acceptance test. An RMS from bundle adjustment is not, because it only says the model fits the data it was given.
6. Production concerns
- Memory. A 200-frame 20 MP mosaic does not fit in RAM as float32. Work at reduced resolution for registration (
work_megapix ≈ 0.6), estimate the seams even smaller (seam_megapix ≈ 0.1), then compose at full resolution — that is exactly what the stock stitcher does, and it is the right architecture. For very large outputs, compose tiles and write a pyramidal tiled TIFF or COG rather than one flat PNG. - Runtime. Feature detection and matching dominate, and matching is O(n²) in frames if you are naive. Use capture order or pose priors to only match plausible neighbours; on a serpentine grid that turns hundreds of candidate pairs into a handful per frame. Profile before optimising — see where your frame time actually goes.
- Determinism and regression tests. RANSAC is stochastic. Seed it, pin your OpenCV version, and keep a golden capture set with an expected frame-retention count and fiducial distance so a library upgrade or a lens swap trips an alarm rather than shipping silently degraded output. Our pipeline regression testing tutorial covers the harness.
- Provenance. Store, per mosaic: the frame list, per-frame transform, bundle-adjustment RMS, retained-frame count, GSD, calibration file version and fiducial validation result. When an inspector disputes a finding six months later, that record is the difference between a five-minute answer and a re-scan.
7. Decision summary
| Situation | Approach |
|---|---|
| Handheld panorama, context imagery | Stitcher_PANORAMA, ship it |
| Gantry / web scan, flat surface, measurable | Stitcher_SCANS or hand-built affine + BundleAdjusterAffinePartial |
| Drone grid, near-planar terrain | Homography + bundle adjustment + RTK priors, or dedicated photogrammetry software |
| Low-texture or repetitive surface | Projected speckle or fiducials; SuperPoint + LightGlue via ONNX if capture cannot change |
| Genuinely 3D object | Not stitching — SfM / multi-view stereo |
| Defect detection on the output | Detect on original frames, use mosaic for coordinates only, no feather blending |
Where this fits
Mosaicking looks like a solved problem because the demo is a one-liner. The engineering is everywhere the demo is silent: choosing a 4-DoF model instead of 8, noticing that 60 frames were dropped, refusing to feather a seam over a crack, and converting a bundle-adjustment residual into a millimetre figure someone will sign.
SentientSight's OpenCV consultants design and audit large-area inspection and mapping pipelines — capture specs, registration architecture, accuracy budgets and the validation harness that keeps them honest in production. If you need a mosaic you can measure from, get in touch.