Two projects out of three that arrive at our door with "the detector is unreliable" turn out to have a camera-motion problem rather than a model problem. Drone inspection footage of a bridge soffit. A handheld endoscope. A camera bolted to a machine that vibrates at 34 Hz. The frames are individually sharp enough, but between frames the world moves, so trackers break, frame differencing lights up everywhere, and every downstream count or measurement inherits jitter that nobody budgeted for.
This tutorial is about the motion layer of an OpenCV 5 pipeline: how to estimate inter-frame motion, how to stabilise video without inventing motion that was never there, and how to report the result as a number rather than "it looks smoother".
1. Decide what you are actually removing
Before any code, separate three things:
- Intended motion — the operator panning along the weld seam, the drone flying the facade. You want to keep it.
- Unintended motion — hand tremor, rotor vibration, road ripple. You want to remove it.
- Scene motion — the person walking, the part moving on the conveyor. You must never remove it, and you must not let it contaminate your camera-motion estimate.
Every stabilisation failure we are asked to debug comes from confusing the third with the second. A truck fills 60% of the frame, the global motion estimate locks onto the truck, and the stabiliser swings the entire background to keep the truck still. Robust estimation is not a detail here; it is the whole job.
2. Where the motion APIs live in OpenCV 5
OpenCV 5 keeps the classic estimators in video, and the deep-flow models arrive through the DNN module as ONNX. The relevant surface:
import cv2
cv2.calcOpticalFlowPyrLK # sparse Lucas-Kanade, video module
cv2.goodFeaturesToTrack # Shi-Tomasi corners, imgproc
cv2.DISOpticalFlow_create # dense DIS flow, video module, CPU-real-time
cv2.calcOpticalFlowFarneback # older dense flow, still present, slower/blurrier
cv2.estimateAffinePartial2D # RANSAC similarity fit from point correspondences
Two migration notes if you are coming from 4.x: the old videostab module and the optflow extras from contrib are not where you should be building new work — DIS is in the main video module and is both faster and better than Farneback. And anything that relied on cv2.motempl or the legacy cv2.tracking free functions needs revisiting; see our OpenCV 4.x to 5.0 migration checklist.
3. The workhorse: sparse flow plus a robust similarity fit
For 90% of stabilisation jobs you do not need dense flow at all. Track a few hundred corners, fit a constrained transform with RANSAC, accumulate, smooth, and re-render.
import cv2, numpy as np
def frame_motions(path, max_corners=600, mask=None):
cap = cv2.VideoCapture(path)
ok, prev = cap.read()
prev_g = cv2.cvtColor(prev, cv2.COLOR_BGR2GRAY)
motions = []
while True:
ok, cur = cap.read()
if not ok:
break
cur_g = cv2.cvtColor(cur, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(prev_g, max_corners, 0.01, 12, mask=mask,
blockSize=7)
if p0 is None:
motions.append(np.eye(2, 3, dtype=np.float32)); prev_g = cur_g; continue
p1, st, _ = cv2.calcOpticalFlowPyrLK(prev_g, cur_g, p0, None,
winSize=(21, 21), maxLevel=3)
p0b, stb, _ = cv2.calcOpticalFlowPyrLK(cur_g, prev_g, p1, None,
winSize=(21, 21), maxLevel=3)
# forward-backward consistency: the only cheap lie-detector you get
fb = np.linalg.norm(p0.reshape(-1, 2) - p0b.reshape(-1, 2), axis=1)
good = (st.ravel() == 1) & (stb.ravel() == 1) & (fb < 1.0)
a, inliers = cv2.estimateAffinePartial2D(
p0[good], p1[good], method=cv2.RANSAC,
ransacReprojThreshold=2.0, maxIters=3000, confidence=0.995)
if a is None:
a = np.eye(2, 3, dtype=np.float32)
motions.append(a)
prev_g = cur_g
cap.release()
return motions
Three choices in there are worth defending in a design review:
estimateAffinePartial2D, not findHomography. A partial affine gives you translation, rotation and uniform scale — four degrees of freedom. A homography gives you eight, and the extra four will happily model a non-existent perspective warp from noisy correspondences on a near-planar scene. Use a homography only when the scene really is a plane you are viewing obliquely (document capture, road surface from a fixed mast).
Forward-backward checking. Lucas-Kanade always returns a point. The status flag tells you the tracker converged, not that it converged on the right thing. Tracking forwards then backwards and rejecting points that fail to return home removes most occlusion and repeated-texture failures for one extra tracker call.
A mask. If you know where the scene motion is — a conveyor band, the lower third where vehicles pass — exclude it from goodFeaturesToTrack. Feed the mask from your detector if you already run one: dilate the boxes by 10% and forbid corners inside them. This is the single cheapest fix for the truck-fills-the-frame failure.
4. Smoothing the trajectory, not the frames
Accumulate the per-frame motions into an absolute trajectory, smooth it, then warp each frame by the difference between the smoothed and raw trajectory. Smoothing the trajectory is what preserves the intended pan while removing the tremor.
def stabilise(motions, radius=30):
# accumulate dx, dy, dtheta
traj, x, y, th = [], 0.0, 0.0, 0.0
for a in motions:
x += a[0, 2]; y += a[1, 2]; th += np.arctan2(a[1, 0], a[0, 0])
traj.append((x, y, th))
traj = np.array(traj)
k = np.ones(2 * radius + 1) / (2 * radius + 1)
smooth = np.stack([np.convolve(np.pad(traj[:, i], radius, 'edge'), k, 'valid')
for i in range(3)], axis=1)
corr = smooth - traj
out = []
for (a, (dx, dy, dth)) in zip(motions, corr):
c, s = np.cos(dth), np.sin(dth)
out.append(np.array([[c, -s, dx], [s, c, dy]], dtype=np.float32))
return out
The radius parameter is the whole product decision. A small radius (5–10 frames) removes high-frequency vibration and follows the operator instantly; a large radius (60+) produces cinematic glide but lags a deliberate pan and swings the frame afterwards. For inspection work we almost always start at 20–30 frames at 30 fps and tune against real operator footage, not a test clip.
Two practical guards:
- Crop, do not fill. Warping leaves empty borders. Zooming in by a fixed 4–8% and cropping is honest. Border reflection or inpainting invents pixels that a downstream defect detector will happily classify.
- Clamp the correction. If
|dx|exceeds your crop margin, the stabiliser is about to expose a border. Clamp, and log it — a burst of clamps means the motion estimate has come off the rails, and that is a quality signal worth surfacing.
5. When you need dense flow: DIS first, RAFT if you must
Sparse flow gives you global camera motion. Dense flow gives you a vector per pixel, which you need for motion segmentation, frame interpolation, flow-guided denoising, or measuring how fast individual things move.
DIS (Dense Inverse Search) is the default. It runs real-time on CPU and needs no model file:
dis = cv2.DISOpticalFlow_create(cv2.DISOPTICAL_FLOW_PRESET_MEDIUM)
dis.setUseSpatialPropagation(True)
flow = dis.calc(prev_g, cur_g, None) # HxWx2 float32, pixels
mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])
RAFT-style learned flow is markedly more accurate on large displacements, thin structures and low texture — the cases where DIS smears. Export a RAFT (or a small RAFT variant) to ONNX and run it through the DNN module:
net = cv2.dnn.readNetFromONNX("raft_small_fp32.onnx")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA) # or DEFAULT / OPENVINO
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
b1 = cv2.dnn.blobFromImage(prev, 1/255.0, (imgW, imgH), swapRB=True)
b2 = cv2.dnn.blobFromImage(cur, 1/255.0, (imgW, imgH), swapRB=True)
net.setInputsNames(["img1", "img2"])
net.setInput(b1, "img1"); net.setInput(b2, "img2")
flow = net.forward() # 1x2xHxW, in the *resized* pixel scale
flow = flow[0].transpose(1, 2, 0)
flow[..., 0] *= origW / imgW # rescale or your magnitudes are wrong
flow[..., 1] *= origH / imgH
Three traps we hit repeatedly on client RAFT exports:
- Iteration count is baked in at export. RAFT refines iteratively; the ONNX graph fixes that count. Twelve iterations and four iterations are different models with different latency. Decide before you export.
- Input dimensions must be multiples of 8 (the feature pyramid stride). Pad, do not squash the aspect ratio — a squashed input produces a directionally biased flow field.
- Flow is in resized pixels. Forget the rescale above and every velocity you report is wrong by a constant factor that reviewers will not spot.
Rough budget from our own benchmarks on 1080p, per frame pair: DIS medium ~8–15 ms on a modern desktop CPU core-set; RAFT-small at 12 iterations ~25–40 ms on a mid-range discrete GPU and multiple hundreds of ms on CPU. On Jetson-class hardware, plan on TensorRT rather than the DNN CUDA backend — see real-time detection on Jetson Orin Nano for the export path.
6. Separating camera motion from scene motion
Once you have a global transform and a dense field, the residual is the interesting part:
h, w = prev_g.shape
gx, gy = np.meshgrid(np.arange(w), np.arange(h))
ones = np.ones_like(gx)
pts = np.stack([gx, gy, ones], axis=-1).astype(np.float32)
global_flow = pts @ A.T - np.stack([gx, gy], axis=-1) # A = 2x3 camera motion
residual = flow - global_flow
moving = cv2.magnitude(residual[..., 0], residual[..., 1]) > 1.5
That moving mask is a motion-segmentation primitive that costs nothing extra and works on a moving camera, which frame differencing does not. It feeds three useful things: a mask for the next round of goodFeaturesToTrack (closing the loop on the truck problem), a region proposal for a detector, and an activity trigger so you only run the expensive model on frames where something actually moved.
7. Validating a stabiliser without hand-waving
"It looks smoother" is not acceptance criteria. Three measurements we put in every stabilisation deliverable:
Residual inter-frame motion. Re-run the motion estimator on the stabilised output. Report the RMS of per-frame translation and rotation, before and after. A typical drone-inspection result: 3.4 px RMS translation before, 0.5 px after, with the intended pan preserved in the low-frequency trajectory.
Interest-point stability. Pick a static feature, track it through 300 stabilised frames, report its standard deviation in pixels. This is the number a customer intuitively understands.
A synthetic ground-truth clip. Take a static high-resolution capture, apply a known jitter trajectory plus a known pan, run the pipeline, and compare the recovered trajectory to the one you injected. This is the only way to prove the stabiliser is not eating real motion. Keep the clip in your regression suite alongside the golden frames from testing OpenCV 5 pipelines like software.
Also track crop loss (what percentage of the field of view you gave up) and clamp rate (what fraction of frames hit the correction limit). Both are honest costs, and quoting them up front prevents the awkward conversation where the customer discovers the edges of their inspection frame have vanished.
8. Where it slots into the pipeline
Order matters. Stabilise after undistortion and before detection, tracking or measurement. Undistorting afterwards applies a lens model to pixels that have already been warped, and the two corrections fight. If you are also doing metric work, remember that stabilisation changes the effective camera pose per frame — if you are back-projecting to world coordinates, either carry the correction transform through or do the metric work on the unstabilised stream and stabilise only the display copy. That second option is usually the right one: stabilisation is for humans and trackers; measurement should happen on the raw geometry you calibrated. See camera calibration in OpenCV 5 for the calibration half of that argument.
Getting help with a motion problem
If your detector is flickering, your tracker keeps swapping IDs, or your drone footage is unusable to the inspectors who have to review it, the fix is often a day of motion-layer work rather than a new model. SentientSight places senior OpenCV engineers on exactly these problems — as consultants, as contractors, or as a subcontracted team behind an agency. Get in touch with a clip and what you need it to measure, and we will tell you what is fixable and what it will cost.