Every second RFP we read this year has a line in it that says something like "and the system should be able to describe what it sees" or "use AI to explain the reject". What the customer has seen is a vision-language model (VLM) demo: upload a photo, ask a question in English, get a paragraph back. What they want is that flexibility inside a machine that has to run 24/7 on a factory line, in a vehicle, or on a camera pointed at a loading dock.
Those two things are not the same system, but they can live in the same pipeline. This tutorial is about how we actually deploy VLMs for clients: not as the detector, but as a second stage behind a conventional OpenCV 5 pipeline, with constrained output, hard guardrails and a test harness.
1. The one architectural rule: a VLM is a triage stage, not a measurement stage
A VLM will happily tell you a gap is "about 2 millimetres". It has no calibration, no scale, no lens model. It is guessing from pixels and from language priors. Never let a VLM produce a number that goes into a tolerance report, a count that goes into a production total, or a safety-critical stop signal. Those belong to the deterministic parts of your pipeline: calibrated geometry (see our camera calibration guide), a trained detector, a tracker with an association rule.
What a VLM is genuinely good at:
- Triage of the uncertain tail. Your classifier is confident on 96% of frames. The remaining 4% go to a human today. A VLM can pre-sort that tail into likely categories and attach a short rationale, cutting review time.
- Open-ended attributes on a cropped region. "Is there text on this label? Is the seal intact? Is the operator wearing gloves?" — questions nobody defined at training time and that change monthly.
- Describing the novel. Anomaly detection (see our PatchCore piece) tells you that a frame is weird. A VLM can produce a first-pass description of how it is weird, which is worth a lot in a defect log.
- Document- and sign-like understanding where layout matters and a plain OCR string is not enough — although the reading itself should still come from a real OCR stage.
So the shape is: OpenCV 5 does capture, geometry, detection and gating. The VLM is called on a small fraction of frames, on a cropped region, with a constrained answer format.
2. Model choices in 2026, honestly
The practical options split three ways.
Small local models (2B–8B). Qwen2.5-VL 3B/7B, Florence-2 base/large, Moondream, SmolVLM, InternVL2 2B. These run on a single mid-range GPU, or quantised on a Jetson-class board, and can be exported to ONNX / run under llama.cpp or vLLM. Florence-2 is the odd one out and often the most useful for engineers: it is small, fast and task-tokened — you prompt it with <OD>, <CAPTION>, <OCR_WITH_REGION> rather than free text, which makes its output far easier to parse and far harder to derail.
Hosted frontier models. Best quality on genuinely hard reasoning, useless for anything with a hard per-frame latency budget or a no-egress data policy. Budget 1–4 s per call and treat the network as unreliable.
Task-specific alternatives you may not need a VLM for at all. If the question is "which of these 12 categories", a CLIP/SigLIP embedding plus a nearest-centroid classifier is 50× cheaper, deterministic, and trivially retrainable from 20 examples per class. We kill roughly a third of proposed VLM features at this step, and clients are happier for it.
3. The pipeline: gate first, crop second, ask third
The gating logic is the whole design. A VLM call is 100–3000 ms; your pipeline has maybe 30 ms per frame. So the VLM must be asynchronous and rare.
import cv2, queue, threading
cap = cv2.VideoCapture(SOURCE, cv2.CAP_GSTREAMER)
vlm_q = queue.Queue(maxsize=8) # bounded: drop, never block the pipeline
def gate(det_score, anomaly_score):
"""Only escalate the uncertain tail."""
return 0.35 < det_score < 0.75 or anomaly_score > ANOMALY_T
while True:
ok, frame = cap.read()
if not ok:
break
dets = detector(frame) # ONNX/TensorRT, deterministic
for d in dets:
if gate(d.score, d.anomaly):
x, y, w, h = expand_box(d.box, frame.shape, pad=0.25)
crop = frame[y:y+h, x:x+w]
crop = cv2.resize(crop, (448, 448), interpolation=cv2.INTER_AREA)
try:
vlm_q.put_nowait((d.id, crop))
except queue.Full:
metrics.inc("vlm_dropped") # visible in the dashboard, not silent
Three details that matter more than the model choice:
- Crop and pad. A VLM given a 4K frame and asked about a 60-pixel object will hallucinate. Give it the region plus ~25% context, resized to the model's native resolution with
INTER_AREA. Accuracy improvements from cropping alone are usually larger than from swapping models. - Colour order. OpenCV is BGR; every VLM preprocessor expects RGB.
cv2.cvtColor(crop, cv2.COLOR_BGR2RGB)before handing off. Skipping this degrades results subtly rather than obviously, which is worse. - Bounded queue, explicit drops. Never let the escalation path apply backpressure to capture. Count drops and alarm on the rate.
4. Constrain the output or you will be parsing prose forever
Free-text answers are not an interface. Force a schema and validate it.
SCHEMA = {
"type": "object",
"properties": {
"category": {"enum": ["scratch", "contamination", "print_defect", "none", "unclear"]},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"evidence": {"type": "string", "maxLength": 160}
},
"required": ["category", "confidence", "evidence"],
"additionalProperties": False
}
PROMPT = (
"You are inspecting a cropped image of a moulded plastic housing.\n"
"Classify the visible surface anomaly. If you cannot see the region clearly, "
"answer 'unclear'. Do not estimate sizes. Reply as JSON matching the schema."
)
Use the serving stack's structured-output support (grammar-constrained decoding in llama.cpp, guided JSON in vLLM, Florence-2's task tokens) rather than asking politely in the prompt. Then validate against the schema anyway and treat a validation failure as unclear, not as an exception that kills the worker.
Note the explicit unclear class and the instruction not to estimate sizes. A VLM without a permitted escape hatch will invent an answer 100% of the time; with one, refusal rate becomes a useful health signal.
5. Guardrails
These are the ones we put in every deployment:
- The VLM can never stop the line. Its output is advisory: it routes an item to review, tags a defect log, or annotates a clip. Deterministic stages own the actuation.
- Confidence floor plus human review. Below the floor, the item goes to a person exactly as it does today. The VLM is measured on how much review work it removes, not on whether it is right.
- Prompt-injection surface. If the camera can see text — packaging, a screen, a hand-written note — that text is untrusted input to a language model. There are documented cases of models following instructions found in an image. Keep the system prompt out of reach, never let the output drive a shell command, database write or API call directly, and prefer enum outputs over free text.
- PII and egress. Faces and plates in escalated crops go through redaction before they leave the box (see our anonymisation pipeline). If the model is hosted, the data-protection question is now a contract question — get it answered before the pilot, not after.
- Cost and rate caps. A hard ceiling on calls per hour, enforced in code. A gate that drifts from 4% to 40% of frames is a pipeline bug, and with a hosted model it is also an invoice.
6. Testing something non-deterministic
A VLM stage still has to pass CI, and the same discipline applies as in our regression testing guide, with three adjustments:
- Fix the seed and temperature to 0, pin the model file by hash, and pin the serving-stack version. Quantisation format changes outputs; treat a re-quantised model as a new model.
- Score a labelled escalation set, not individual strings. Keep 200–500 real escalated crops with human labels. The gate for a model swap is a confusion matrix and a refusal rate, not exact-match text.
- Track the metric the client actually bought: review minutes saved per shift, and the number of true defects the VLM labelled
none(the only genuinely expensive error). Report both weekly.
Also keep a shadow mode deployment period. Run the VLM stage live, log every answer, surface nothing to operators for two weeks, then compare against what the humans decided. That comparison is the business case, and it is far more persuasive than a benchmark score.
7. A realistic budget
For a single-line inspection cell, 30 fps, ~2% escalation rate, Qwen2.5-VL 3B quantised to 4-bit on an RTX A2000-class GPU sharing the box with the detector: expect 250–600 ms per call, comfortable headroom at ~0.6 escalations/s, and about 2 GB of VRAM for the VLM. On a Jetson Orin-class module, expect 1–3 s per call and plan the escalation rate down accordingly — or run the detector at the edge and the VLM on one shared on-prem server for a fleet of cells, which is usually the better economics.
Where this goes wrong
The failure mode we get called in to fix is always the same: somebody put the VLM in the primary path. Every frame goes to the model, the frame rate collapses, the numbers are not reproducible, and nobody can explain why yesterday's good part is today's reject. Pull it back to a triage stage behind a gate and most of those problems disappear at once.
SentientSight's senior OpenCV consultants build hybrid pipelines like this one — deterministic OpenCV 5 capture, calibration and detection, with a constrained VLM stage where it genuinely pays for itself — and we are equally happy to tell you when a SigLIP classifier does the job for a fraction of the cost.
If you have a review queue you want to shrink, or an RFP with "AI that explains what it sees" in it, get in touch with your frame rate, escalation volume and data-egress constraints, and we will size the honest version of it.