Testing OpenCV 5 pipelines like software: golden frames, model-swap gates, and drift alarms

Every OpenCV project we are asked to rescue has the same symptom and the same cause. The symptom: "it worked last month." The cause: nothing in the pipeline was pinned, measured or re-measured. A wheel upgraded a minor version, someone swapped an ONNX file, a camera firmware update changed the white balance, and the accuracy number nobody was watching quietly moved.

The algorithm posts on this blog cover how to build detectors, trackers, depth and OCR pipelines. This one covers the part that decides whether they survive contact with production: reproducible builds, golden-frame regression tests in CI, model-swap gates, and drift alarms that fire before your client notices.

1. Pin the whole stack, not just OpenCV

A vision pipeline's output is a function of far more than your source code. Pin all of it:

# requirements.lock (generated, committed)
opencv-python-headless==5.0.0.86
numpy==2.2.6
onnxruntime==1.23.0

Then record the pieces pip cannot pin. Every run of the pipeline should emit a provenance block:

import cv2, numpy as np, hashlib, json, platform, subprocess

def provenance(model_path: str) -> dict:
    with open(model_path, "rb") as f:
        model_sha = hashlib.sha256(f.read()).hexdigest()[:16]
    return {
        "opencv": cv2.__version__,
        "opencv_build_hash": cv2.getBuildInformation().split("\n")[1].strip(),
        "numpy": np.__version__,
        "model_sha256": model_sha,
        "git_sha": subprocess.run(["git", "rev-parse", "--short", "HEAD"],
                                  capture_output=True, text=True).stdout.strip(),
        "cpu_baseline": cv2.getBuildInformation().split("CPU/HW features")[-1][:200],
        "threads": cv2.getNumThreads(),
        "platform": platform.platform(),
    }

print(json.dumps(provenance("models/detector.onnx"), indent=2))

Three details that matter more than they look:

• cv2.getNumThreads() is part of your result. Several OpenCV operations are not bit-exact across thread counts, and CI runners have different core counts from your workstation. Set cv2.setNumThreads(N) explicitly in tests.
• The CPU baseline changes the arithmetic. A build dispatching AVX-512 will not always produce byte-identical output to an AVX2 path. This is why golden tests need tolerances, not equality (§3).
• Hash the model file, do not trust the filename. detector_v3.onnx has been silently overwritten in every organisation that has ever shipped a model.

Do the same for the pipeline's own parameters: keep thresholds, ROIs, homographies and calibration files in a versioned config, not in code, and log the config hash next to the provenance block. See our camera calibration guide for why calibration in particular must be a versioned artefact.

2. Build a golden-frame corpus that is actually representative

The corpus is the test suite. Twenty pictures of the happy path prove nothing. Aim for 100–300 frames, sampled deliberately:

• Nominal frames — the boring, well-lit, in-spec case. Roughly half.
• Known-hard frames — every image that has ever caused a production incident. These are the highest-value assets your project owns. Name them after the ticket.
• Environmental spread — shift start and shift end, day and night, different cameras of the same model, one wet lens, one dirty lens, one partly occluded view.
• Degenerate inputs — a black frame, a fully saturated frame, a 1-pixel-wide crop, a corrupt JPEG, a frame with zero detections, a frame with 300 detections.

That last group catches an outsized share of real crashes. Empty-input handling is where OpenCV pipelines throw, because cv2.dnn NMS calls and np.argmax on empty arrays behave differently from the non-empty case.

Store frames as lossless PNG, never re-encoded JPEG — a re-save changes pixels and your golden values drift for no reason. Keep them out of the git object store with Git LFS or DVC, and pin the corpus by version:

git lfs track "tests/corpus/**/*.png"
echo "corpus_version: 2026.09.1" >> tests/corpus/MANIFEST.yml

Each frame gets a small sidecar with its expectation and provenance:

{
  "frame": "corpus/hard/INC-4412_wet_lens_dusk.png",
  "camera": "cam-07",
  "captured_utc": "2026-06-11T18:42:03Z",
  "expect": {"objects": 3, "min_confidence": 0.45},
  "notes": "Regression for INC-4412: rain on dome caused duplicate boxes."
}

3. Golden tests with tolerances, not equality

Pixel-exact assertions on a vision pipeline are a trap: they fail on a harmless SIMD or driver difference and get muted within a fortnight, taking your real coverage with them. Assert on decisions and metrics with explicit tolerances instead.

Three layers, cheapest first:

Layer 1 — geometry and decisions. For a detector, compare against stored boxes by IoU matching:

def match_boxes(expected, actual, iou_thresh=0.5):
    """Greedy IoU match; returns (matched, missed, spurious)."""
    unmatched = list(range(len(actual)))
    matched = 0
    for e in expected:
        best, best_iou = None, 0.0
        for j in unmatched:
            i = iou(e, actual[j])
            if i > best_iou:
                best, best_iou = j, i
        if best is not None and best_iou >= iou_thresh:
            unmatched.remove(best)
            matched += 1
    return matched, len(expected) - matched, len(unmatched)


def test_golden_detections(pipeline, corpus):
    misses = spurious = total = 0
    for frame, expected in corpus:
        _, missed, extra = match_boxes(expected, pipeline(frame))
        misses += missed; spurious += extra; total += len(expected)
    recall = 1.0 - misses / total
    assert recall >= 0.97, f"recall regressed to {recall:.3f}"
    assert spurious <= 4, f"{spurious} spurious detections (budget 4)"

Note the shape of the assertion: a corpus-level budget, not a per-frame absolute. Per-frame equality is brittle; a corpus-wide recall floor is meaningful and stable.

Layer 2 — image outputs. When a stage genuinely produces an image (undistortion, registration, a mask), compare numerically with a tolerance:

def assert_image_close(actual, golden, max_mean_abs=0.6, max_bad_pixel_frac=0.001):
    assert actual.shape == golden.shape, f"shape {actual.shape} != {golden.shape}"
    diff = cv2.absdiff(actual, golden).astype(np.float32)
    mean_abs = float(diff.mean())
    bad_frac = float((diff > 12).mean())
    assert mean_abs <= max_mean_abs, f"mean |diff| {mean_abs:.3f}"
    assert bad_frac <= max_bad_pixel_frac, f"{bad_frac:.4%} pixels differ badly"

For masks, use IoU or Dice rather than pixel diff — a one-pixel boundary shift is not a regression.

Layer 3 — invariants that need no golden at all. These are cheap and catch a surprising amount:

def test_invariants(pipeline):
    black = np.zeros((720, 1280, 3), np.uint8)
    assert pipeline(black) == []                 # no hallucinations, no crash
    assert pipeline(np.full_like(black, 255)) is not None
    frame = cv2.imread("corpus/nominal/0001.png")
    a, b = pipeline(frame), pipeline(frame.copy())
    assert a == b                                # determinism
    assert pipeline(frame[:1, :1]) is not None   # degenerate size

Determinism deserves special attention. Run the same frame twice in the same process; if the results differ, you have non-determinism from threading, an uninitialised buffer, or a stateful tracker leaking between calls — and no test suite above it can be trusted.

4. Gate model swaps like code changes

A new ONNX file is a production change with a bigger blast radius than most commits. Require the same evidence:

def test_model_contract():
    net = cv2.dnn.readNet("models/detector.onnx")
    # Input/output shapes are a contract; changes break postprocessing silently.
    out = net.forward(np.zeros((1, 3, 640, 640), np.float32))
    assert out.shape == (1, 300, 6), f"unexpected output shape {out.shape}"

Then a champion/challenger comparison on the corpus, reported per slice:

slice	frames	champion recall	challenger recall	Δ
nominal	148	0.991	0.993	+0.002
dusk / wet lens	41	0.902	0.874	−0.028
cam-07 only	33	0.964	0.967	+0.003

Aggregate accuracy went up; the hard slice went down. That is the single most common way a model upgrade causes an incident, and only per-slice reporting exposes it. Rule of thumb we apply on client projects: no slice may regress by more than 1 point of recall, regardless of the headline number — and any exception is a written decision, not a merge.

Also gate latency and size, not just accuracy. Record p50/p95 per-stage frame time from your profiling harness and fail the build if p95 exceeds the device's budget. A challenger that is 0.4 points better and 2.3× slower is a rejection on an embedded target.

5. Make CI reproducible with a container

Vision CI on a bare runner drifts as the runner image drifts. Build the test environment once:

FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
      libgl1 libglib2.0-0 git git-lfs && rm -rf /var/lib/apt/lists/*
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock
ENV OPENCV_LOG_LEVEL=WARNING OMP_NUM_THREADS=2

libgl1 and libglib2.0-0 are the two packages whose absence produces the classic ImportError: libGL.so.1 on slim images — or avoid them entirely by using opencv-python-headless, which is the right choice for any CI or server deployment.

A workable GitHub Actions shape:

jobs:
  vision-tests:
    runs-on: ubuntu-latest
    container: ghcr.io/acme/vision-ci:2026.09.1
    steps:
      - uses: actions/checkout@v4
        with: { lfs: true }
      - run: pytest tests/ -m "not gpu" --junitxml=report.xml
      - run: python tools/regression_report.py --out report.md
      - uses: actions/upload-artifact@v4
        with: { name: regression-report, path: |
                  report.md
                  artifacts/diffs/** }

Two practices that make failures actionable:

• Upload the visual diff. On any failing frame, write a side-by-side of golden, actual and amplified difference into artifacts/diffs/. A reviewer resolves in thirty seconds what a numeric assertion message takes an hour to explain.
• Keep GPU tests separate. Mark CUDA/TensorRT tests @pytest.mark.gpu and run them on a self-hosted runner (or nightly on the actual target device). CPU tests must stay fast — under five minutes — or people stop running them.

6. Golden-value updates need a ritual

Goldens must be updatable, or the suite becomes a wall people climb over. Give it a deliberate path:

pytest tests/ --update-goldens   # rewrites goldens, prints a summary table
git add tests/goldens && git commit -m "chore: update goldens (detector v4, +2 frames)"

And the rules around it:

1. --update-goldens never runs in CI. Ever.
2. The commit updating goldens contains nothing else, so the diff is reviewable.
3. The message states why — new model, new camera, fixed bug — and links the ticket.
4. Regenerating more than ~10% of goldens triggers a human review of whether the change was intended.

7. Drift monitoring: the same idea, pointed at production

CI protects you from your own changes. It cannot protect you from the world changing. That needs the same measurements running on live traffic.

You will usually not have live labels, so monitor input distributions and output statistics and alarm on movement:

import numpy as np, collections

class DriftMonitor:
    """Rolling input/output statistics against a frozen reference window."""

    def __init__(self, ref_stats, window=2000):
        self.ref = ref_stats
        self.buf = collections.deque(maxlen=window)

    @staticmethod
    def frame_features(frame, dets):
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        return {
            "mean_luma": float(gray.mean()),
            "rms_contrast": float(gray.std()),
            "sharpness": float(cv2.Laplacian(gray, cv2.CV_64F).var()),
            "n_det": len(dets),
            "mean_conf": float(np.mean([d.conf for d in dets])) if dets else 0.0,
        }

    def push(self, feats):
        self.buf.append(feats)
        if len(self.buf) < self.buf.maxlen:
            return {}
        alerts = {}
        for k, (mu, sigma) in self.ref.items():
            cur = np.mean([f[k] for f in self.buf])
            z = (cur - mu) / max(sigma, 1e-6)
            if abs(z) > 3.0:
                alerts[k] = {"z": round(float(z), 2), "current": round(float(cur), 3)}
        return alerts

What each signal actually tells you:

• sharpness falling — lens dirt, condensation, or a focus knock. The most common real-world cause of quiet accuracy loss, and it is fixable with a cloth rather than a retrain.
• mean_luma shifting — seasonal daylight, a failed IR illuminator, a changed exposure profile after a firmware update.
• n_det per hour changing shape — either the scene changed or the model did. Compare against a business counter (units produced, vehicles through the gate) to tell those apart.
• mean_conf sagging while n_det holds — classic covariate shift. The model is still finding things, less certain about each. This one usually precedes a visible failure by weeks, which is exactly what you want.

Sample and store the frames that trigger alarms. They become next quarter's corpus additions — the loop from production incident back into the test suite is the whole point, and it is the step most teams never close.

Two more things worth wiring on day one: a shadow deployment path so a challenger model can run on live frames without acting on them, and a canary on one camera or one line before a fleet rollout. Both are cheap once the metrics pipeline above exists.

8. A minimal maturity ladder

You do not need all of this in week one. In the order we usually recommend:

1. Pinned dependencies plus a provenance block in every log line. (One afternoon.)
2. Twenty golden frames, corpus-level assertions, running locally. (Two days.)
3. The same suite in a container in CI, with visual diffs uploaded. (Two days.)
4. Model-swap gate with per-slice reporting and a latency budget. (Three days.)
5. Input/output drift monitoring on production, alarm frames retained. (One week.)
6. Shadow and canary deployment for model changes. (Ongoing.)

Steps 1–3 cost roughly a week and eliminate the majority of "it worked last month" incidents. Step 4 is what turns a model upgrade from a leap of faith into a decision. Step 5 is what lets you answer, credibly, the question every client eventually asks: how do you know it is still working?

Where this fits

Accuracy is a claim. Repeatability is what makes the claim worth anything, and it is almost always the difference between a vision prototype and a system somebody will sign off on.

SentientSight's OpenCV consultants build regression harnesses, model-promotion gates and drift monitoring into client vision pipelines — including retrofitting them onto systems already in production. If your pipeline's accuracy number is a memory of a spreadsheet from six months ago, get in touch.