From walkaround video to a measurable 3D model: OpenCV 5 frame curation, SfM poses, and Gaussian splatting

Clients rarely ask us for "structure from motion". They ask for something like: someone walks around the asset with a phone, and we want a 3D model we can measure, plus a viewer the site team can spin. Over the last two years the second half of that request has changed completely — 3D Gaussian splatting and its descendants made photoreal scene capture cheap — but the first half has not. Whether the output is a mesh, a point cloud or a splat, everything downstream depends on camera poses and image quality, and that is where OpenCV 5 earns its keep.

This tutorial is the pipeline we actually deploy: OpenCV 5 for capture curation and pose sanity-checking, a structure-from-motion (SfM) stage for poses and sparse geometry, then either a mesh or a splat for the deliverable — with a metric scale step so the numbers on the model mean millimetres, not "units".

1. The pipeline, and where each piece can fail

video / image set
   │  ① OpenCV 5: frame extraction, blur & exposure rejection, de-duplication
   ▼
curated frames + intrinsics prior
   │  ② SfM (COLMAP / GLOMAP): feature matching → poses + sparse cloud
   ▼
poses (world→cam) + sparse points
   │  ③ dense: MVS mesh   OR   radiance: 3D Gaussian splatting
   ▼
   │  ④ OpenCV 5: metric scale, reprojection QA, measurement overlays
   ▼
deliverable + an error statement

Failures cluster in stage ①. Motion-blurred frames, rolling-shutter wobble, and 400 near-identical frames from a stationary pause all degrade matching. Fix the input and stage ② mostly takes care of itself.

2. Frame curation in OpenCV 5

Three cheap filters remove most bad capture: sharpness, exposure, and redundancy.

import cv2, numpy as np

cap = cv2.VideoCapture("walkaround.mp4")
kept, prev_gray, idx = [], None, -1

def sharpness(gray):
    # variance of Laplacian: higher = sharper. Scale-dependent, so
    # always compare on frames resized to the same working size.
    return cv2.Laplacian(gray, cv2.CV_64F).var()

def exposure_ok(gray, clip=0.02):
    hist = cv2.calcHist([gray], [0], None, [256], [0, 256]).ravel()
    hist /= hist.sum()
    return hist[:4].sum() < clip and hist[-4:].sum() < clip

while True:
    ok, frame = cap.read()
    if not ok:
        break
    idx += 1
    small = cv2.resize(frame, (960, 540), interpolation=cv2.INTER_AREA)
    gray = cv2.cvtColor(small, cv2.COLOR_BGR2GRAY)

    if sharpness(gray) < 60.0:          # calibrate this per camera, see below
        continue
    if not exposure_ok(gray):
        continue

    # redundancy: require real parallax since the last kept frame
    if prev_gray is not None:
        p0 = cv2.goodFeaturesToTrack(prev_gray, 400, 0.01, 8)
        p1, st, _ = cv2.calcOpticalFlowPyrLK(prev_gray, gray, p0, None)
        if p0 is not None and st is not None:
            d = np.linalg.norm((p1 - p0)[st.ravel() == 1], axis=1)
            if len(d) and np.median(d) < 6.0:   # px of motion at 960x540
                continue

    kept.append(idx)
    prev_gray = gray
    cv2.imwrite(f"frames/{idx:06d}.png", frame)   # full-res for SfM

print(len(kept), "frames kept")

Notes that matter in production:

• Never tune the blur threshold by feel. Record 30 seconds of deliberately good capture and 30 of deliberately bad, plot the two Laplacian-variance histograms, and put the threshold at the crossing point. It is camera- and texture-specific; a threshold copied from a blog post will silently throw away half a good dataset.
• Write PNG, not JPEG, for the SfM input if disk allows. Recompression artefacts cost you matches on low-texture surfaces.
• Median flow, not mean flow, for the parallax test — a few moving objects should not qualify a stationary frame.
• Rolling shutter: if the capture device is a phone in a moving vehicle or on a drone, drop the frame rate at the source rather than relying on curation. OpenCV cannot undo per-row exposure skew.

3. Give SfM an intrinsics prior

SfM will happily self-calibrate, but a good prior converges faster and avoids the classic focal-length/depth ambiguity that makes a whole scene come out uniformly 12% too small. If the capture camera is available, calibrate it once with a ChArUco board — see our camera calibration tutorial — and feed fx, fy, cx, cy, k1, k2, p1, p2 in as fixed or lightly-refined parameters.

In OpenCV 5 the geometry surface moved: undistort, rectification and the 3D utilities live in the new 3d module, calibration itself in calib, and ChArUco detection in objdetect. In Python the flat cv2. namespace still resolves; in C++ your includes change.

Two habits from the field:

1. Lock focus and exposure during capture. A phone that refocuses mid-orbit gives you a different focal length per frame, and no single prior is correct.
2. Do not pre-undistort the frames if the SfM tool models distortion itself. Doing both once is correct; doing it twice bakes in an error you cannot see until you measure the model.

4. Reading the SfM result honestly

The stage-② report is the first place a project quietly goes wrong. Three numbers to check before anyone renders anything:

Metric	Healthy range (phone walkaround, ~200 frames)	What a bad value means
Registered images	> 95% of input	Weak overlap, blur, or texture-poor surfaces
Mean reprojection error	0.4 – 1.0 px	> 1.5 px: bad intrinsics prior, rolling shutter, or mismatches
Mean track length	> 4 images per point	Sparse matching; capture with more overlap

If 30% of frames failed to register, the answer is a re-capture, not more compute. Say that early: it is much cheaper than delivering a model with a hole in it.

5. Verify poses with OpenCV before you trust them

A useful, unglamorous check: take the SfM poses and sparse points, reproject the points into a handful of frames with OpenCV, and look at them.

import cv2, numpy as np

# R, t: world -> camera for one registered frame; K: 3x3 intrinsics
pts3d = sparse_points          # (N,3) float64, world coords
rvec, _ = cv2.Rodrigues(R)
proj, _ = cv2.projectPoints(pts3d, rvec, t, K, dist)
proj = proj.reshape(-1, 2)

img = cv2.imread(frame_path)
for (u, v) in proj:
    if 0 <= u < img.shape[1] and 0 <= v < img.shape[0]:
        cv2.circle(img, (int(u), int(v)), 2, (0, 255, 0), -1)
cv2.imwrite("qa_reproj.png", img)

If the green points sit on the edges and corners you expect, the pose set is sane. If they drift systematically towards one side of the frame, suspect the principal point or an unmodelled distortion term. This ten-line check has caught more bad reconstructions for us than any dashboard.

6. Mesh or splat? Pick by deliverable, not by novelty

Multi-view stereo → mesh when the deliverable is geometry: volumes, clearances, clash detection, CAD comparison, anything that goes into an engineering decision. A mesh has explicit surfaces you can measure, decimate and export as OBJ/PLY/glTF for existing tooling.

3D Gaussian splatting when the deliverable is appearance: a photoreal, spinnable capture for review, marketing, remote inspection, or as-found documentation. Splats handle thin structures, foliage and specular surfaces that wreck MVS meshing, and render fast in a browser. What they are not is a clean measurable surface — extracting reliable geometry from splats is an active research area, not a delivery guarantee.

Both stages consume the same stage-② poses. That is the practical argument for treating pose estimation as the product of the pipeline: you can change the renderer later without re-capturing.

Plenty of projects want both, and that is fine — one capture, one SfM run, two outputs.

7. Metric scale, or the model measures nothing

SfM output is scale-free. Three ways to fix scale, in increasing order of trustworthiness:

1. Known baseline in scene. Put a scale bar, a printed ArUco board of known edge length, or two survey targets in the capture. Detect the markers with OpenCV 5 objdetect, solve pose with solvePnP, and compare the reconstructed distance to the true one.
2. Measured tape distance between two identifiable features, applied as a single scale factor.
3. Sensor/GNSS metadata (drone RTK, ARKit/ARCore odometry). Convenient, and the least controlled — verify it against option 1 before quoting tolerances.

# scale from an ArUco marker of known physical edge length
det = cv2.aruco.ArucoDetector(
    cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50),
    cv2.aruco.DetectorParameters())
corners, ids, _ = det.detectMarkers(gray)

# reconstructed_edge_m comes from the model; marker_edge_m is ground truth
scale = marker_edge_m / reconstructed_edge_m
print(f"apply scale {scale:.4f} to the reconstruction")

Then hold back a check measurement — a distance you never used to fit the scale — and report the residual. "Scaled from a 300 mm target; an independent 1.240 m span reconstructs at 1.247 m, +0.6%" is a defensible statement. "It's to scale" is not.

8. An error budget you can put in a report

For a phone walkaround of a machine-sized object, curated as above, with marker-based scale and clean SfM statistics, we typically quote something in this shape:

• Pose consistency: mean reprojection error 0.5–0.9 px
• Scale accuracy: ±0.5–1.5% of the measured span
• Local surface noise (MVS mesh): 1–3 mm at 1 m stand-off
• Absolute point-to-point measurement: ±1% of span, with a stated check measurement

Those figures move with camera, lighting, texture and stand-off — the point is that every deliverable ships with numbers and the method that produced them. If a client needs sub-millimetre absolute accuracy on a 5 m asset, photogrammetry from a phone is the wrong tool and structured light or laser scanning is the right one. Saying so in week one is worth more than a heroic pipeline in week nine.

9. Common failure modes, ranked by how often we see them

1. Stationary pauses — hundreds of zero-parallax frames. The redundancy filter in §2 fixes it.
2. Autofocus/auto-exposure drift — inconsistent intrinsics. Lock them at capture.
3. Texture-poor surfaces (white walls, machined metal, glass). Add temporary texture: projected pattern, removable stickers, chalk. Or accept holes and say where they are.
4. Reflective and transparent parts — MVS produces phantom geometry; splats look fine but measure badly. Mask these regions and exclude them from measurement claims.
5. Loop not closed — the operator walked 300° instead of 360°, so accumulated drift never gets corrected. A 30-second overlap back to the start costs nothing.
6. Scale fitted and verified on the same measurement — reports zero error, and is meaningless.

10. A capture protocol worth handing to the operator

• Orbit the subject at roughly constant radius, ~70–80% overlap between consecutive kept frames.
• Three passes at different heights; add a high-angle pass for anything with a top surface.
• Keep moving; do not stop and hold.
• Lock focus and exposure before starting; avoid direct sun/deep shadow boundaries where possible.
• Place at least two scale references, visible in multiple frames, ideally in different parts of the scene.
• Close the loop: end where you started, with overlap.
• Record camera, lens, resolution and frame rate in the job notes. Future-you will need them.

Reconstruction quality is decided during those five minutes of capture. Everything afterwards is bookkeeping — good bookkeeping, with OpenCV 5 doing the curation, the QA and the metric anchoring, but bookkeeping.

---

Need a 3D capture pipeline that produces numbers you can sign off? SentientSight places senior OpenCV and photogrammetry engineers on project and subcontract engagements — from capture protocol design and SfM tuning to metric QA and viewer delivery. Get in touch with your asset type, accuracy requirement and deliverable format, and we will tell you honestly whether photogrammetry is the right tool.