INT8 quantisation and NPU inference for OpenCV 5 pipelines: OpenVINO, calibration sets, and an accuracy loss you can sign off

Most of the OpenCV 5 deployments we are asked to cost out in 2026 no longer start with "which GPU". They start with "the customer bought forty industrial PCs with an NPU in them, can we use it?" Intel Core Ultra machines, Qualcomm and Rockchip boxes, and the Jetson replacements people are choosing after the 2026 EOL wave all ship a neural accelerator that delivers excellent frames-per-watt — on one condition. The NPU will not run your FP32 model. It wants INT8 (sometimes INT4 or FP8), and getting there without quietly wrecking accuracy is the actual engineering work.

This tutorial is the workflow we use: export cleanly, quantise with a calibration set that resembles production, measure the accuracy delta on a held-out set, fall back per-layer where it hurts, benchmark the devices honestly, and hand the client a report they can sign.

1. Where the NPU sits relative to OpenCV

Be clear about the split of responsibilities, because this is where most first attempts go wrong:

• OpenCV 5 owns the pipeline: capture, decode, colour conversion, resize, ROI, letterboxing, post-processing, NMS, drawing, tracking, and the frame budget.
• The NPU owns one thing: the forward pass of a quantised network.

OpenCV 5's DNN engine has been rebuilt and is a fine choice on CPU and for GPU backends, but NPUs are reached through the vendor runtime — OpenVINO on Intel, QNN/LiteRT on Qualcomm, RKNN on Rockchip, and so on. Do not fight this. Keep cv::Mat as your frame currency, hand a tensor to the vendor runtime, get a tensor back, and post-process in OpenCV. The clean seam is a small inference class with submit(Mat) -> vector<float>; swap the implementation per platform and the rest of the pipeline never learns which device it ran on.

A warning that costs people a week: NPU inference is often asynchronous with a deep queue. Its throughput number looks fantastic and its single-frame latency does not. If your product has a control deadline — a reject gate, a robot pick — measure latency, not throughput. Our profiling tutorial covers how to instrument the whole frame path so you know which of the two you are actually bound by.

2. ONNX export hygiene (this is 80% of quantisation pain)

Quantisation tools are unforgiving about sloppy graphs. Before you attempt anything, fix the export:

• Static shapes. Dynamic height and width defeat most NPU compilers. Export at the exact inference resolution: 1x3x640x640, not 1x3xHxW. If you need two resolutions, export two models.
• Opset and simplification. Export at a recent opset, then run a simplifier (onnxsim) to fold constants and remove training-time detritus.
• Move post-processing out of the graph. Exported detectors frequently bake NMS, TopK, NonZero or scatter ops into the tail. These rarely map to an NPU and force an expensive fallback right at the end of the graph. Cut the model at the raw head outputs and do decode plus NMS in OpenCV (cv::dnn::NMSBoxesBatched) — it is a fraction of a millisecond on CPU.
• Normalisation: pick one home and write it down. Either the graph does mean/scale, or your OpenCV pre-processing does. Doing it in both places halves your input range and produces a model that detects nothing — and because it fails silently rather than crashing, teams lose days to it.

import onnx, onnxsim

model = onnx.load("detector_fp32.onnx")
model, ok = onnxsim.simplify(model)
assert ok
onnx.save(model, "detector_fp32_simplified.onnx")

for i in model.graph.input:
    print(i.name, [d.dim_value or d.dim_param for d in i.type.tensor_type.shape.dim])
for o in model.graph.output:
    print(o.name, [d.dim_value or d.dim_param for d in o.type.tensor_type.shape.dim])

If any dimension prints as a string rather than a number, go back and re-export.

3. The calibration set is the whole ballgame

Post-training quantisation (PTQ) estimates an activation range per tensor by running a few hundred representative images through the network. "Representative" is doing enormous work in that sentence.

Rules we hold to:

• 200–500 images, drawn from the deployed camera, lens, lighting and product mix. Not the public dataset the model was trained on.
• Include the hard tails: the dark shift after the factory lights change, the glare from the morning sun on the east window, the dirty lens, the rare product variant. Activation ranges are set by extremes; if your calibration set never sees glare, the first glare frame in production saturates and the output goes to noise.
• No duplicates and no augmentation. Synthetic brightness jitter invents ranges the camera cannot produce and widens your scales for nothing.
• Pre-process exactly as production does. Same resize interpolation, same letterbox padding colour, same channel order. A calibration set built with PIL RGB for a pipeline that feeds OpenCV BGR is a classic and infuriating bug.

Version the calibration set like code, next to the model. When accuracy drifts in six months you will want to know which images set the scales.

import cv2, glob, numpy as np, nncf, openvino as ov

SIZE = 640

def preprocess(path):
    img = cv2.imread(path)                      # BGR, exactly as the pipeline
    img = cv2.resize(img, (SIZE, SIZE), interpolation=cv2.INTER_LINEAR)
    blob = img.astype(np.float32) / 255.0
    return np.expand_dims(blob.transpose(2, 0, 1), 0)

files = sorted(glob.glob("calib_set_v2/*.png"))
assert 200 <= len(files) <= 600, len(files)
calib = nncf.Dataset(files, preprocess)

model = ov.Core().read_model("detector_fp32_simplified.onnx")
int8 = nncf.quantize(
    model, calib,
    preset=nncf.QuantizationPreset.MIXED,       # per-channel weights, safer for detectors
    subset_size=len(files),
)
ov.save_model(int8, "detector_int8.xml")

MIXED (symmetric weights, asymmetric activations) is the preset we reach for with detection heads; plain PERFORMANCE is faster but more likely to clip the wide activation ranges that regression outputs produce.

4. Measure the accuracy delta on a held-out set — then decide

Never accept a quantised model on eyeball evidence. You need the same task metric, FP32 versus INT8, on images the calibration set never saw.

Model	mAP@0.5	Recall @ 0.5 conf	False positives / 1k frames
FP32 (CPU)	0.914	0.961	3
INT8 (NPU)	0.902	0.948	5

What matters is not mAP. It is the number the client is actually paying for. On an inspection line that is usually escape rate at a fixed false-reject rate — so evaluate at the operating point, not across the whole curve. A 1.2-point mAP drop that all lands on the smallest defect class can be a project failure even though the headline looks fine. Break the delta out per class and per size bucket.

Our rule of thumb, which we state in proposals: under 1% relative drop on the operating-point metric, ship it. 1–3%, investigate and usually fix. Over 3%, something specific is broken — do not accept it as the cost of quantisation.

Diagnose by comparing intermediate tensors rather than guessing. Run FP32 and INT8 on the same frame, dump per-layer outputs, and compute cosine similarity. The layer where similarity falls off a cliff is your culprit — typically a detection head's regression branch, a sigmoid/exp tail, a depthwise layer with one enormous outlier channel, or a concat that mixes tensors with wildly different ranges.

5. Fixes, in the order we try them

1. Exclude the bad layers from quantisation. Keep them in FP16 and let everything else be INT8. Most tools accept an ignore list (nncf.IgnoredScope(names=[...])). Leaving the final head in FP16 typically costs a few percent of speed and recovers nearly all the accuracy — it is by far the best return on effort.
2. Widen or rebalance the calibration set. If the failure only appears on dark or glary frames, the set was not representative.
3. Per-channel weight quantisation where it is not already on, especially for depthwise-separable backbones.
4. Smooth the outliers — SmoothQuant-style range rebalancing between weights and activations moves difficulty from activations to weights, which tolerate it better.
5. Quantisation-aware training (QAT) only if you own the training pipeline and PTQ genuinely fails. It works, and it costs a retraining cycle plus a lot of care; treat it as a last resort, not the default plan.
6. Change the model. Sometimes the honest answer is that this architecture quantises badly. A slightly smaller model that survives INT8 cleanly beats a bigger one propped up by FP16 fallbacks.

6. Benchmark like an engineer, not a vendor

Run the whole pipeline, warm, for minutes — not a single forward() in a loop:

import time, numpy as np, openvino as ov

core = ov.Core()
print(core.available_devices)      # e.g. ['CPU', 'GPU', 'NPU']

for device in ["CPU", "GPU", "NPU"]:
    compiled = core.compile_model("detector_int8.xml", device)
    req = compiled.create_infer_request()
    x = preprocess(files[0])
    for _ in range(20):            # warm-up: compile, clocks, memory
        req.infer({0: x})
    lat = []
    for _ in range(300):
        t0 = time.perf_counter(); req.infer({0: x}); lat.append((time.perf_counter() - t0) * 1e3)
    lat = np.array(lat)
    print(f"{device}: mean {lat.mean():.1f} ms  p50 {np.percentile(lat,50):.1f}  "
          f"p95 {np.percentile(lat,95):.1f}  p99 {np.percentile(lat,99):.1f}")

Report p95 and p99, not the mean. A deadline-driven system is defined by its worst frames. Things to hold yourself to:

• Include pre- and post-processing. We have seen an NPU cut inference from 28 ms to 6 ms on a pipeline whose resize, colour conversion and NMS cost 19 ms. The end-to-end win was 45%, not 4x, and the proposal had promised the latter.
• Measure thermally soaked. Fanless industrial boxes throttle. A 60-second benchmark on a cold unit is marketing. Run 30 minutes with the lid on.
• Measure with the real number of streams. One camera on an NPU is not four cameras on an NPU; see the multi-camera ingest budget for how quickly decode competes with inference.
• Watch power and the other consumers. The NPU's whole appeal is watts. Log package power and note that on Intel Core Ultra the iGPU is often faster than the NPU while the NPU is far more efficient and leaves the GPU free for decode. Which one wins depends on your product, and you should be able to say why in one sentence.
• First-inference compile cost is real. NPU model compilation can take seconds. Cache the compiled blob (ov.Core().set_property({"CACHE_DIR": ...})) or your service looks broken at startup.

7. Make it a reproducible, auditable artefact

An INT8 model is a build output, not a file someone produced on a laptop once. Alongside the containerised build, store and version:

• the FP32 ONNX and its exact export script,
• the calibration set (or an immutable manifest of hashes),
• the quantisation config, including any ignore list and why each entry is on it,
• the accuracy report, FP32 vs INT8, on a named held-out set,
• the per-device latency table with the hardware, driver and runtime versions,
• a golden-frame test that fails CI if the INT8 output drifts beyond tolerance — the same gate pattern as our regression testing tutorial.

That last one matters more than it sounds. NPU drivers and runtimes update, and a driver bump can change numerics. If your CI does not notice, your customer will.

8. The one-page sign-off

What we put in front of a client before anything goes to the line:

On held-out set val-2026-09 (4,180 frames, 6 defect classes): INT8 recall at the fixed 2% false-reject operating point is 94.8% versus 96.1% FP32, a 1.3-point absolute drop, concentrated in the hairline-crack class. End-to-end p99 latency on the target NPU box, thermally soaked with 4 streams, is 41 ms against a 50 ms budget, at 11 W package power versus 34 W for the GPU path. The detection head is retained in FP16. Recommendation: accept for classes A–E; route hairline-crack through the FP16 GPU path until the calibration set is extended with the 300 crack images being collected.

That is the deliverable. Not "we got it running on the NPU" — a measured trade, with the residual risk named and an owner for it.

Where this fits

Quantisation is where computer vision projects meet procurement. The hardware is already bought, the power budget is fixed, and someone has to determine whether the accuracy that survives INT8 is good enough for the thing being inspected. That is an engineering judgement backed by measurement, not a checkbox in a conversion tool.

SentientSight's OpenCV consultants do this work end to end — export hygiene, calibration set design, PTQ and QAT, per-device benchmarking, and the accuracy report your quality team can accept. If you have NPU hardware in hand and need to know honestly what it will run, get in touch.