+1 (415) 360-7596
SentientSight

Open-vocabulary detection with zero training data: YOLO-World and SAM 2 in an OpenCV 5 pipeline

The single most common reason a computer vision project stalls before it starts is not the model. It is the dataset. A client asks whether OpenCV can find "empty pallet positions" or "workers without gloves" or "cracked insulators", and the honest 2024-era answer was: yes, once you have collected and labelled a few thousand images. That is six weeks and a five-figure annotation bill before anyone sees a demo.

Open-vocabulary ("zero-shot") detectors changed the shape of that first six weeks. You describe the class in text, get boxes back, and find out on day one whether the idea is viable at all. This tutorial shows how to run that pipeline inside OpenCV 5 — including the parts that go wrong — and where the boundary sits between a good prototype and something you can put on a production line.

1. What open-vocabulary detection actually is

A classic detector (YOLO, SSD, Faster R-CNN) has a fixed output layer: 80 COCO classes, or the 3 classes you fine-tuned. An open-vocabulary detector instead embeds text and image regions into a shared space and scores them against each other. Change the prompt, change the classes — no retraining, no new weights.

The families you will meet in practice:

  • YOLO-World — CNN/YOLOv8-derived, region-text contrastive head, real-time on a GPU and the easiest of the group to export to ONNX. Best fit for OpenCV DNN.
  • Grounding DINO / GroundingDINO 1.5 — transformer-based, noticeably stronger on unusual phrasing, considerably heavier and harder to export cleanly.
  • OWL-ViT / OWLv2 — good at image-conditioned queries ("find more things that look like this crop"), useful when your class is hard to name in words.
  • SAM 2 (Segment Anything 2) — not a detector. It is a promptable segmenter with video memory: give it a box or a point, get a mask, and have that mask tracked across frames.

The pattern that works well in production prototyping is a two-stage one: an open-vocabulary detector proposes boxes from a text prompt, and SAM 2 promotes the good boxes to pixel-accurate masks (and, on video, carries them forward).

2. Prompt engineering is now part of the vision pipeline

Before any code: your class names are hyperparameters. A prompt list of ["crack"] and one of ["hairline crack in white ceramic", "chipped edge"] produce dramatically different recall on the same images.

Rules of thumb we apply on every engagement:

  • Prefer concrete noun phrases over jargon. "safety glove on hand" beats "PPE compliance". The text encoder was trained on web captions, not your quality manual.
  • Add negative-ish distractor classes. If you only prompt for "box", everything box-shaped is a box. Prompt ["cardboard box", "plastic crate", "pallet", "floor"] and the softmax has somewhere else to go.
  • Keep the vocabulary short at inference time. YOLO-World's speed depends on the number of prompt embeddings baked in. 4-10 classes is the sweet spot; 200 is a research demo.
  • Version your prompt list like code. It is the single most impactful configuration in the whole pipeline and it belongs in git, not in a notebook cell.

3. Exporting YOLO-World to ONNX for OpenCV

OpenCV's DNN module does not implement a text encoder, so you cannot change prompts at runtime inside OpenCV. The workaround is the one YOLO-World was designed for: re-parameterisation. You bake a fixed vocabulary into the model, and the exported graph becomes an ordinary N-class detector with a normal box-and-score output — exactly the kind of graph DNN is good at.

from ultralytics import YOLOWorld

CLASSES = ["cardboard box", "plastic crate", "wooden pallet", "forklift"]

model = YOLOWorld("yolov8s-worldv2.pt")
model.set_classes(CLASSES)          # re-parameterise into a fixed-vocab detector
model.save("warehouse_world.pt")    # persist the baked vocabulary

model.export(format="onnx", opset=17, simplify=True, dynamic=False, imgsz=640)

Two export details that cause most of the "it works in Python but not in C++" tickets:

  1. dynamic=False and a fixed imgsz. Dynamic shapes are supported unevenly across engines; a static 1x3x640x640 graph is boring and portable. Ship boring.
  2. Write the class list next to the weights. Once the vocabulary is baked, the ONNX file has no idea what index 2 means. A warehouse_world.json beside it with {"classes": [...]} prevents the classic off-by-one label bug six months later.

Verify the graph before you go near OpenCV:

python -m onnxruntime.tools.check_onnx_model_mobile_usability warehouse_world.onnx  # optional
python - <<'PY'
import onnx
m = onnx.load("warehouse_world.onnx")
onnx.checker.check_model(m)
print([(i.name, [d.dim_value for d in i.type.tensor_type.shape.dim]) for i in m.graph.input])
print([(o.name, [d.dim_value for d in o.type.tensor_type.shape.dim]) for o in m.graph.output])
PY

If the output is (1, 4 + N, 8400), you have a YOLOv8-style head with N = len(CLASSES). That is what the decode below assumes.

4. Running it in OpenCV 5's DNN engine

OpenCV 5 ships two DNN engines: the new graph engine (default) and the classic 4.x one. For an exported YOLO-World graph the new engine is usually fine and faster to load, but pin it explicitly so a library upgrade cannot silently change your numerics.

import cv2
import numpy as np
import json

CLASSES = json.load(open("warehouse_world.json"))["classes"]
CONF_T, NMS_T = 0.25, 0.50

net = cv2.dnn.readNetFromONNX("warehouse_world.onnx")
# OpenCV 5: choose the engine deliberately.
try:
    net.setPreferableBackend(cv2.dnn.DNN_BACKEND_OPENCV)
    net.setPreferableTarget(cv2.dnn.DNN_TARGET_CPU)
except AttributeError:
    pass

def detect(bgr):
    h0, w0 = bgr.shape[:2]
    # letterbox to 640x640 so aspect ratio survives
    s = min(640 / w0, 640 / h0)
    nw, nh = int(round(w0 * s)), int(round(h0 * s))
    canvas = np.full((640, 640, 3), 114, np.uint8)
    canvas[:nh, :nw] = cv2.resize(bgr, (nw, nh), interpolation=cv2.INTER_LINEAR)

    blob = cv2.dnn.blobFromImage(canvas, 1 / 255.0, (640, 640), swapRB=True, crop=False)
    net.setInput(blob)
    out = net.forward()                      # (1, 4+N, 8400)
    pred = out[0].T                          # (8400, 4+N)

    boxes, scores, ids = [], [], []
    cls_scores = pred[:, 4:]
    best = cls_scores.argmax(1)
    best_score = cls_scores[np.arange(len(pred)), best]
    keep = best_score > CONF_T
    for (cx, cy, w, h), sc, ci in zip(pred[keep, :4], best_score[keep], best[keep]):
        boxes.append([int((cx - w / 2) / s), int((cy - h / 2) / s), int(w / s), int(h / s)])
        scores.append(float(sc))
        ids.append(int(ci))

    idx = cv2.dnn.NMSBoxes(boxes, scores, CONF_T, NMS_T)
    return [(boxes[i], scores[i], CLASSES[ids[i]]) for i in np.array(idx).flatten()] if len(idx) else []

Things worth knowing here:

  • Letterbox, do not stretch. blobFromImage with a raw resize will squash tall objects and cost you several points of recall on anything non-square. The manual canvas above keeps the mapping invertible with a single scale factor.
  • Class-agnostic vs per-class NMS. NMSBoxes as written is class-agnostic. With overlapping prompts ("crate" and "cardboard box" on the same object) that is usually what you want; with genuinely distinct classes, use cv2.dnn.NMSBoxesBatched or offset boxes per class.
  • Confidence is not calibrated. Zero-shot scores are not comparable across prompt lists. Re-tune CONF_T every time you change the vocabulary, and do it on a held-out set of real frames, not on the three images in your README.

For the wider DNN engine picture — quantisation, backend selection, and where the classic engine is still required — see running YOLO26 in OpenCV 5's new DNN engine.

5. Promoting boxes to masks with SAM 2

Boxes are enough for counting and presence checks. They are not enough for area, shape, or "how much of the surface is corroded". SAM 2 turns a box prompt into a mask, and on video it keeps that mask attached to the object across frames using its memory bank.

SAM 2 does not export to a single clean ONNX graph as easily as YOLO-World; the usual production split is encoder-once, decoder-per-prompt, and many teams simply run SAM 2 in PyTorch or ONNX Runtime alongside OpenCV rather than inside DNN. That is a legitimate architecture — OpenCV remains the I/O, geometry, and post-processing layer, which is where it earns its keep.

import numpy as np, cv2
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2.1-hiera-small")

def masks_from_boxes(bgr, dets):
    rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
    predictor.set_image(rgb)                      # encoder runs once per frame
    xyxy = np.array([[x, y, x + w, y + h] for (x, y, w, h), _, _ in dets], dtype=np.float32)
    masks, iou_pred, _ = predictor.predict(box=xyxy, multimask_output=False)
    out = []
    for m, q, (box, score, label) in zip(masks, iou_pred, dets):
        mask = (m.squeeze() > 0).astype(np.uint8)
        # clean up: SAM edges are good but not morphologically tidy
        mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, np.ones((3, 3), np.uint8))
        cnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
        area_px = float(sum(cv2.contourArea(c) for c in cnts))
        out.append({"label": label, "det_score": score, "mask_quality": float(q),
                    "area_px": area_px, "mask": mask})
    return out

The useful trick: mask_quality is a free rejection signal. In our experience a low predicted-IoU mask on a high-confidence box almost always means the detector boxed a group of objects, a reflection, or a shadow. Thresholding on it removes a class of failure that no amount of detector tuning fixes.

If you convert area_px into square millimetres, everything depends on calibration, not on the model. Our OpenCV 5 camera calibration guide covers the error budget behind that conversion; a mask is only as trustworthy as the scale factor you multiply it by.

6. The highest-value use: auto-labelling, not deployment

Here is the recommendation we give clients most often, and it is deliberately unglamorous: use zero-shot models to build your dataset, then ship a small supervised model.

The workflow:

  1. Collect a few thousand unlabelled frames from the real camera, real lighting, real dirt.
  2. Run the open-vocabulary detector plus SAM 2 over all of them offline. Speed does not matter here; a slow, heavy Grounding DINO configuration is fine.
  3. Auto-reject with the signals above (low mask_quality, implausible area, boxes touching the frame edge, detections outside the ROI).
  4. Have a human review only the survivors — correcting boxes is roughly 5-10x faster than drawing them.
  5. Fine-tune a small YOLO or segmentation model on the result and export it to ONNX for OpenCV DNN.

What you end up with is a 6-15 MB model that runs at frame rate on a Jetson or an x86 industrial PC, has predictable latency, and can be validated to a fixed accuracy number — while the two-week annotation phase became two days. That is the real value of the technology today, and it is why we treat zero-shot models as tooling on most engagements rather than as the deliverable.

7. Where zero-shot is not enough

Be blunt with stakeholders about the limits. Open-vocabulary detectors are weak exactly where industrial vision is demanding:

  • Fine-grained and sub-pixel defects. "Scratch", "burr", "hairline crack" are under-represented in web-scale caption data. Recall collapses on low-contrast defects.
  • Domain-specific parts. Your part number is not a word. Image-conditioned prompting (OWLv2) helps; a fine-tuned model helps more.
  • Latency and determinism. A transformer detector plus a segmentation encoder is hundreds of milliseconds per frame on edge hardware. If you have a 30 fps line-rate budget, this is not the architecture. See choosing an embedded vision platform for the hardware side of that arithmetic.
  • Validation and audit. A regulated customer will ask for accuracy on a fixed test set. A prompt-driven model whose behaviour changes with a text string is hard to freeze. A supervised model with a versioned checkpoint is not.
  • Licensing. Check the weights, not just the code. Several strong open-vocabulary checkpoints are research-licensed, and this discovery belongs in week one, not in your customer's legal review.

A pragmatic default architecture

For most projects that walk in the door asking about "AI that just knows what to look for", the architecture we end up recommending is:

  • OpenCV 5 for capture, calibration, ROI geometry, morphology, tracking, and overlay — the deterministic scaffolding.
  • A zero-shot detector, offline, as a labelling engine and feasibility probe.
  • A small supervised ONNX model, online, in OpenCV DNN or TensorRT, for the actual production decision.
  • SAM 2 selectively, where pixel-accurate area or shape genuinely drives the business rule.

That combination gets a demo in front of a stakeholder in days and a defensible production system in weeks, without pretending the prototype was the product.

If you have a use case you are trying to size — including "is this even possible without a dataset?" — our team does short feasibility studies exactly like the one above. Get in touch with a description of the scene, the class you need found, and the frame rate you have to hit, and we will tell you which half of this pipeline you actually need.