SentientSight | Training a detector when the customer has no images: synthetic data, domain randomization, and an OpenCV 5 sim-to-real loop

The most common blocker on a new computer vision engagement is not the model, the camera or the compute. It is that the customer has no labelled images — because the product is not in production yet, because the defect they care about happens twice a month, or because their images live behind a privacy review that will take a quarter to clear.

Waiting is usually the wrong answer. You can build, train and validate a detector on synthetic imagery, then close the gap with a small real-image set collected once hardware exists. This tutorial is the workflow we actually use: what to render, what to randomize, which OpenCV 5 calls do the compositing and augmentation, and — most importantly — how to prove the model transferred rather than hoping it did.

1. Decide whether synthetic data is the right tool

Synthetic data transfers well when the object's geometry carries the signal and badly when its statistics do.

Good candidates:

Rigid, CAD-defined parts: fasteners, connectors, moulded housings, PCBs, pallets, cartons.
Pose and keypoint estimation, where ground truth is essentially free from the renderer.
Counting and presence/absence of known SKUs.
Barcode, label and fiducial reading, where you can generate the payload and the print defects.

Bad candidates:

Fine-grained surface texture defects — corrosion, subtle scratches, organic contamination. Here an unsupervised approach on real good-only images is usually stronger; see our write-up on unsupervised defect detection with PatchCore.
Anything where "normal" is defined by the customer's process noise rather than by shape.
Open-ended classes you cannot enumerate. For those, start zero-shot: open-vocabulary detection with YOLO-World and SAM 2 will get you a baseline without training at all.

A useful rule from engagements: if a human can recognise the object from a grey untextured CAD render, synthetic data will probably work. If they cannot, render less and label more.

2. The pipeline in one picture

CAD / mesh ──▶ renderer (Blender, Isaac Sim, Unity)
                  │  RGB + instance mask + 6-DoF pose
                  ▼
        OpenCV 5 compositing & degradation
                  │  backgrounds, lens blur, noise, ISP artefacts
                  ▼
        COCO-format dataset  ──▶ train (Ultralytics / MMDet)
                  │
                  ▼
        ONNX export ──▶ OpenCV 5 DNN inference
                  │
                  ▼
        REAL validation set (few hundred images, hand-labelled)

Two things are non-negotiable in that diagram. First, the renderer emits instance masks, not just boxes — masks let you re-composite, re-crop and derive boxes later without re-rendering. Second, the validation set is real. A synthetic validation set will tell you your model is excellent and teach you nothing.

3. Render for variation, not for beauty

Photorealism is expensive and, past a point, not what drives transfer. Domain randomization works because a network trained across an absurdly wide distribution treats real images as just another sample from it. Concretely, per frame randomize:

Pose: full rotation range the fixture permits, plus a few degrees beyond it.
Camera: focal length ±15% around the real lens, principal point jitter of a few pixels, working distance across the depth of field.
Lighting: 1–4 sources, colour temperature 3000–7000 K, intensity across two stops, plus one frame in ten with harsh specular grazing light.
Materials: base colour jitter, roughness 0.1–0.9, metallic on/off for the same part.
Distractors: other SKUs, hands, gloves, tooling, cable, swarf — occluding the target 0–40%.

If you know the real optics, encode them rather than guessing. Calibrate the real camera once and feed its intrinsics and distortion into the render config; our ChArUco calibration walkthrough produces exactly the numbers you need. Renderers use ideal pinhole projection, so apply the measured distortion afterwards in OpenCV — that step alone closes a surprising amount of the gap for wide-angle installs.

4. Compositing and degradation in OpenCV 5

This is where OpenCV earns its place in the loop: cheap, deterministic, scriptable image degradation that makes clean renders look like sensor output.

import cv2, numpy as np, random

rng = np.random.default_rng(0)

def composite(fg_bgr, mask_u8, bg_bgr):
    """Alpha-composite a render over a real background, with a soft edge."""
    h, w = fg_bgr.shape[:2]
    bg = cv2.resize(bg_bgr, (w, h), interpolation=cv2.INTER_AREA)
    # feather the mask so the object does not look cut out with scissors
    a = cv2.GaussianBlur(mask_u8, (0, 0), sigmaX=1.2).astype(np.float32) / 255.0
    a = a[..., None]
    return (fg_bgr * a + bg * (1.0 - a)).astype(np.uint8)

def apply_optics(img, K, dist):
    """Push ideal-pinhole renders through the measured lens distortion."""
    h, w = img.shape[:2]
    mapx, mapy = cv2.initInverseRectificationMap(
        K, dist, np.eye(3), K, (w, h), cv2.CV_32FC1)
    return cv2.remap(img, mapx, mapy, cv2.INTER_LINEAR,
                     borderMode=cv2.BORDER_REPLICATE)

def degrade(img):
    out = img
    # 1. motion blur along a random direction (line speed)
    if rng.random() < 0.4:
        k = int(rng.integers(3, 13)) | 1
        kern = np.zeros((k, k), np.float32); kern[k // 2, :] = 1.0 / k
        ang = rng.uniform(0, 180)
        M = cv2.getRotationMatrix2D((k / 2 - 0.5, k / 2 - 0.5), ang, 1.0)
        kern = cv2.warpAffine(kern, M, (k, k))
        out = cv2.filter2D(out, -1, kern)
    # 2. defocus
    if rng.random() < 0.3:
        out = cv2.GaussianBlur(out, (0, 0), sigmaX=rng.uniform(0.4, 2.0))
    # 3. exposure / white balance drift in LAB
    lab = cv2.cvtColor(out, cv2.COLOR_BGR2LAB).astype(np.float32)
    lab[..., 0] *= rng.uniform(0.75, 1.25)
    lab[..., 1] += rng.uniform(-6, 6)
    lab[..., 2] += rng.uniform(-6, 6)
    out = cv2.cvtColor(np.clip(lab, 0, 255).astype(np.uint8), cv2.COLOR_LAB2BGR)
    # 4. sensor noise: Poisson-ish shot noise then read noise
    f = out.astype(np.float32)
    f = rng.poisson(np.clip(f, 0, None) * 0.8) / 0.8
    f += rng.normal(0, rng.uniform(1.0, 5.0), f.shape)
    out = np.clip(f, 0, 255).astype(np.uint8)
    # 5. the codec the customer actually streams through
    q = int(rng.integers(45, 92))
    ok, buf = cv2.imencode('.jpg', out, [cv2.IMWRITE_JPEG_QUALITY, q])
    return cv2.imdecode(buf, cv2.IMREAD_COLOR) if ok else out

Four notes from production use:

Match the real acquisition chain, artefact for artefact. If frames arrive as H.264 over RTSP at 4 Mbps, your synthetic set must contain compression blocking. Models are extremely good at noticing that your training images were pristine PNGs.
Feather masks. Hard composite edges are a shortcut feature; the network will learn the edge rather than the object, then collapse on real data.
Backgrounds should be real. A few thousand photos from the actual plant — even unlabelled, even from a phone — are worth more than any procedural background. This is usually the one data ask a customer can satisfy on day one.
Keep degradation deterministic per seed. You will want to re-generate an identical dataset when a training run misbehaves.

Derive boxes from the transformed masks rather than transforming boxes, and drop instances whose visible area falls below roughly 1% of the frame or whose occlusion exceeds your inspection spec.

5. Balance the mix: 5% real goes a long way

Pure synthetic training is a starting point, not the destination. The cheapest large win in sim-to-real is a small amount of real data mixed in — a few hundred labelled real frames alongside tens of thousands of synthetic ones, oversampled so real images make up 10–20% of each epoch. In engagement after engagement, that mix beats both pure-synthetic and pure-real-with-too-few-images.

A sensible staging plan:

Stage	Data	What you can honestly claim
0	20k synthetic only	Feasibility demo; ranking of camera/lens options
1	+ 200 real, labelled	Pilot-grade model; first real recall number
2	+ 1–2k real from the line, mined by uncertainty	Production candidate with a defensible spec

Stage 2 is an active-learning loop, not a bulk-collection exercise: run the stage-1 model over the line, keep the frames where confidence sits near the threshold or where tracking disagrees with detection, and label only those. It also shows the customer exactly where the model is weak, which is much easier to discuss than an aggregate mAP.

6. Validate on real images, and on the right metric

Hold out real images from the start and never train on them. Then look at:

Per-class recall at the operating threshold, not mAP. Nobody's tolerance spec is written in mAP.
Sim-to-real gap: the same model's score on a synthetic holdout minus its score on the real holdout. Shrinking that number is the whole objective; if synthetic score rises while the gap widens, you are overfitting the renderer.
Failure clustering: group false negatives by pose, illumination and occlusion bucket. Synthetic data's superpower is that you can fix a gap by rendering 3,000 more frames in exactly that bucket overnight.
Calibration of confidence: a model trained on synthetic data is often overconfident on real frames. Re-fit the threshold on the real holdout, not on the synthetic one.

7. Deploy through OpenCV 5's DNN engine

Export to ONNX with a fixed input size and static batch, then run it where the pipeline already lives:

net = cv2.dnn.readNetFromONNX("detector.onnx")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)   # or DEFAULT / OPENVINO
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)

blob = cv2.dnn.blobFromImage(frame, 1/255.0, (640, 640), swapRB=True, crop=False)
net.setInput(blob)
dets = net.forward()

OpenCV 5's rewritten DNN engine has much better ONNX operator coverage than 4.x, but the preprocessing contract still has to match training exactly — same resize interpolation, same letterbox padding colour, same channel order. A silent mismatch there looks exactly like a failed sim-to-real transfer and wastes days. Our notes on running YOLO26 in OpenCV 5's DNN engine cover the export flags, and golden-frame regression testing is how you keep preprocessing honest once more than one person touches the repo.

One more habit worth building: version the generator, not just the dataset. Store the render config, the randomization ranges and the degradation seed next to the model weights. Six months later, when the customer changes lighting, regenerating a matched dataset is a config edit rather than an archaeology project.

8. What to promise a client

Be precise about what synthetic data buys. It buys a working model before hardware is final, a way to start when privacy review is blocking real images, and free ground truth for pose and keypoints. It does not remove the need for real validation data, and any accuracy number quoted from a synthetic holdout is marketing, not engineering.

A typical honest scope: two weeks to stand up the render and degradation pipeline and produce a stage-0 feasibility model, then a pilot number once a few hundred real frames exist. That framing keeps expectations where they belong and turns "we have no data" from a project blocker into a scheduling detail.

If you are staring at a vision problem with no images yet, that is a normal place to start a project — not a reason to delay it. Tell us about the part and the line and we will tell you whether synthetic data is the right lever or whether you should be collecting real frames first.