Every visual inspection project starts with the same conversation. The client has 40,000 images of good parts and eleven images of defects — three scratches, five contamination spots, and three of something nobody can name. Then somebody proposes training a defect detector, and the project spends six months collecting labels for defects that occur once per 20,000 units.
There is a better default for surface inspection: don't model the defects, model normal. Unsupervised (one-class) anomaly detection trains only on good parts, scores every pixel by how far it deviates from the learned normal manifold, and flags anything unusual — including defect modes that have never been seen. On MVTec AD-style benchmarks these methods reach image-level AUROC in the high 90s from a few hundred good images and zero defect labels.
This tutorial builds that pipeline end to end: OpenCV 5 for capture, registration and post-processing; a PatchCore-style embedding model exported to ONNX and run inside OpenCV's DNN engine; and — the part that decides whether the system ships — an honest threshold and an error budget.
1. When one-class beats supervised detection
Use unsupervised anomaly detection when:
- defects are rare, diverse, or not enumerable in advance ("anything that shouldn't be there"),
- you have plentiful good samples and few bad ones,
- the part presentation is repeatable — same lighting, same pose, same scale.
Use a supervised detector or segmentation model when:
- you have a fixed, known defect taxonomy with hundreds of examples each,
- the customer needs defect classification ("scratch vs. burr") not just reject/accept,
- normal appearance varies enormously (natural products, printed packaging with many SKUs).
Plenty of production lines end up with both: a one-class model as a catch-all gate, plus a small supervised classifier on the crops it flags, so the report says what kind of defect and not merely that one exists.
The honest limitation: one-class models detect deviation, not defect. A dust speck, a legitimate laser mark, and a hairline crack all deviate. Deciding which deviations matter is a specification problem, and it belongs in the requirements document before any code is written.
2. Fix the imaging before you touch the model
More inspection projects are rescued by lighting than by architecture. Before any training run:
- Lock exposure, gain and white balance. Auto-anything makes today's normal different from tomorrow's normal, and the model will score the drift as a defect.
- Choose the illumination for the defect physics. Scratches and dents need low-angle/darkfield light; contamination and print errors want diffuse dome light; transparent or specular parts often need coaxial or polarised light with a crossed polariser on the lens. A defect you cannot see in the raw frame will not be recovered by a network.
- Use a global shutter if the part moves, and trigger the camera off an encoder or photo-eye so every image is taken at the same station position.
- Keep it repeatable. One-class methods assume the normal distribution is narrow. Every millimetre of pose jitter widens it and eats your detection margin.
3. Registration and ROI in OpenCV 5
Even with a good fixture, parts shift. Register every frame to a canonical template so a patch at pixel (x, y) always means the same physical location.
import cv2
import numpy as np
template = cv2.imread("golden_part.png", cv2.IMREAD_GRAYSCALE)
# ORB is a solid default; for rigid parts with strong edges it is fast and licence-clean.
orb = cv2.ORB_create(nfeatures=2000)
kp_t, des_t = orb.detectAndCompute(template, None)
matcher = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
def register(frame_gray):
kp_f, des_f = orb.detectAndCompute(frame_gray, None)
if des_f is None or len(kp_f) < 20:
return None
matches = sorted(matcher.match(des_t, des_f), key=lambda m: m.distance)[:200]
if len(matches) < 12:
return None
src = np.float32([kp_t[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst = np.float32([kp_f[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)
H, inliers = cv2.findHomography(dst, src, cv2.RANSAC, 3.0)
if H is None or inliers.sum() < 10:
return None
return cv2.warpPerspective(frame_gray, H, (template.shape[1], template.shape[0]),
flags=cv2.INTER_LINEAR)
Two notes for OpenCV 5. First, findHomography, warpPerspective and friends now live in the new 3d/imgproc split rather than the old monolithic calib3d; in Python the flat cv2. namespace hides this, but C++ builds need the new headers — see our OpenCV 4.x to 5.0 migration checklist. Second, if the part is truly planar and fixtured, prefer estimateAffinePartial2D (4 DoF: rotation, uniform scale, translation) over a full homography. Fewer degrees of freedom means less opportunity for the registration to "absorb" a real defect into a warp.
Return failed registrations as a machine fault, not a good part. A pipeline that silently passes unregistered frames is a pipeline that passes defects.
Then crop to a fixed ROI and mask out regions that legitimately vary — date codes, serial numbers, sprue marks:
roi = registered[Y0:Y1, X0:X1]
roi = cv2.bitwise_and(roi, roi, mask=static_mask) # static_mask: 255 = inspect, 0 = ignore
Masking the variable regions is the cheapest false-positive reduction available. Do it before you tune anything else.
4. The model: PatchCore-style memory bank
PatchCore is the workhorse of this family and is easy to reason about:
- Push each good image through a frozen ImageNet-pretrained backbone (WideResNet-50 or ResNet-18 for edge).
- Take mid-level feature maps (layer2 + layer3) — deep enough to be semantic, shallow enough to be local — and locally average-pool them so each spatial position describes a small neighbourhood.
- Store all those patch vectors in a memory bank, then subsample it with greedy coreset selection down to ~1–10% of the vectors.
- At inference, score each patch by its distance to the nearest neighbour in the bank. The pixel-level anomaly map is those distances upsampled; the image-level score is the max (or a top-k mean, which is far less jittery).
Nothing is trained. That is the point: a few hundred good images and a few minutes on a GPU, no labels, no annotation contract.
The practical route is anomalib, which packages PatchCore, PaDiM, FastFlow, EfficientAD, Reverse Distillation and a common export path. EfficientAD is worth benchmarking whenever you need millisecond-scale latency; PatchCore is worth it when accuracy matters more than throughput.
pip install anomalib[full]
anomalib train --model Patchcore \
--data.root ./dataset --data.category widget \
--data.train_batch_size 8 --data.image_size 256
anomalib export --model Patchcore --export_type onnx \
--ckpt_path results/.../model.ckpt
Fold the coreset ratio and the top-k score into your validation sweep. A 1% coreset is much faster and usually within a point of AUROC of 10%; verify on your own data, not on the paper's.
5. Running it from OpenCV 5's DNN engine
OpenCV 5 shipped a rewritten DNN engine with much better ONNX graph coverage, so a mid-level feature extractor plus a distance head usually imports cleanly. Keep the preprocessing identical to training — this is the single most common source of "it scored differently in production".
net = cv2.dnn.readNetFromONNX("patchcore.onnx")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU)
MEAN = (0.485, 0.456, 0.406)
STD = (0.229, 0.224, 0.225)
def score(roi_bgr):
blob = cv2.dnn.blobFromImage(roi_bgr, scalefactor=1/255.0,
size=(256, 256), swapRB=True, crop=False)
blob = (blob - np.array(MEAN).reshape(1, 3, 1, 1)) / np.array(STD).reshape(1, 3, 1, 1)
net.setInput(blob.astype(np.float32))
anomaly_map, image_score = net.forward(net.getUnconnectedOutLayersNames())
return anomaly_map[0, 0], float(image_score)
Assert the preprocessing rather than trusting it. Run one fixed image through both the training framework and the OpenCV path and compare scores; anything beyond ~1e-3 relative difference means a mismatch in resize interpolation, channel order or normalisation, and you should fix it before tuning thresholds. If the graph refuses to import, export with a fixed batch size and opset_version=17, and run onnxsim over it first. For NVIDIA edge targets, TensorRT will normally beat the DNN engine — the same pattern as in our Jetson Orin Nano + TensorRT walkthrough.
6. From heat map to reject decision
The raw anomaly map is a float image. Turning it into an accept/reject needs OpenCV, not the network:
amap, img_score = score(roi)
amap = cv2.resize(amap, (roi.shape[1], roi.shape[0]), interpolation=cv2.INTER_CUBIC)
amap = cv2.GaussianBlur(amap, (0, 0), sigmaX=4) # suppress single-pixel noise
amap = cv2.bitwise_and(amap, amap, mask=static_mask.astype(bool))
binary = (amap > PIXEL_THRESHOLD).astype(np.uint8) * 255
binary = cv2.morphologyEx(binary, cv2.MORPH_OPEN,
cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3, 3)))
n, labels, stats, centroids = cv2.connectedComponentsWithStats(binary, connectivity=8)
defects = [i for i in range(1, n)
if stats[i, cv2.CC_STAT_AREA] >= MIN_BLOB_PX]
reject = len(defects) > 0
Two thresholds, two jobs. PIXEL_THRESHOLD decides what counts as anomalous; MIN_BLOB_PX encodes the customer's actual acceptance criterion ("scratches under 0.5 mm pass"). Convert MIN_BLOB_PX from millimetres using your measured pixel pitch so the rule stays meaningful when the optics change — and calibrate that pitch properly rather than eyeballing it (see our ChArUco calibration guide).
7. Choosing the threshold without lying to yourself
This is where inspection projects are won or lost, and it is not a modelling problem — it is a cost problem.
Build three sets: train (good only), validation (good + every defect you have), holdout (never touched until acceptance). Then sweep the threshold and read the confusion matrix at each point:
from sklearn.metrics import roc_curve, auc
fpr, tpr, thr = roc_curve(y_true, scores) # y_true: 1 = defect
print("AUROC", auc(fpr, tpr))
# Pick by cost, not by "best F1".
TARGET_RECALL = 0.99 # escape rate the customer will accept
i = np.argmax(tpr >= TARGET_RECALL)
print(f"threshold={thr[i]:.4f} recall={tpr[i]:.3f} false-reject={fpr[i]:.3f}")
Ask the customer two numbers before you pick: the cost of an escape (a defect shipped) and the cost of a false reject (a good part scrapped or re-inspected). In most industrial settings escapes cost 10–1000× more, so you run at high recall and pay in false rejects — but at a 2% false-reject rate on a 100,000-unit day, someone is manually re-inspecting 2,000 parts, and if you did not tell them that in advance, the system gets switched off in week three.
Also report AUPRC, not just AUROC. With one defect in 5,000 parts, AUROC flatters everything; precision-recall tells the truth about the alarm load.
8. Drift is the real failure mode
The model was fit to one week's normal. Then a new resin lot changes the surface gloss, a lamp ages 10%, or the day shift wipes the lens. Every score shifts and the false-reject rate quietly triples.
Ship a monitoring layer from day one:
- log every image score, plus a mean grey level and a Laplacian-variance focus metric per frame;
- chart the p50/p95 of good-part scores on a control chart; alarm on the distribution, not on individual parts;
- keep a small golden-sample set and re-run it on every shift start — a fixed part that suddenly scores higher means the imaging drifted, not the part;
- store rejects with their heat maps so operator dispositions become the retraining set.
Refitting a PatchCore memory bank takes minutes, so plan for scheduled re-fits on recent good production and treat the bank as a versioned, signed artefact tied to a lighting configuration.
9. A realistic build order
- Week 1 — imaging: fixture, lighting, trigger, lens. Collect 500+ good images and every defect the plant can find.
- Week 2 — registration, ROI, masking, and a baseline classical check (thresholding, template difference). Sometimes classical wins outright, and it is far cheaper to maintain.
- Week 3 — train PatchCore and one fast alternative (EfficientAD), export to ONNX, verify the OpenCV path scores identically.
- Week 4 — threshold selection against costs, holdout evaluation, PLC/MES integration, operator UI showing the heat map.
- Ongoing — drift monitoring, periodic re-fit, and a documented procedure for what happens when registration fails.
Note what is missing: no annotation project, no defect taxonomy negotiation, no six-month label-collection phase. That is the whole argument for one-class inspection.
Where this fits
Unsupervised anomaly detection is the highest-leverage pattern in industrial vision right now, and also the easiest to deploy badly — because the hard parts are optics, registration, thresholds and drift, not the network. The model is a week of the project; the other three weeks are what makes it survive a year on the line.
SentientSight's OpenCV consultants design and audit inspection systems end to end: lighting and optics selection, OpenCV 5 registration and post-processing, anomaly-model selection and export, threshold and cost analysis, and the drift monitoring that keeps false-reject rates stable after handover. If you have thousands of good parts, a handful of defects, and a decision to make, get in touch.