Most "OpenCV is too slow" engagements we get called into are not model problems. The customer has a detector that runs in 9 ms on the GPU and a pipeline that delivers 6 frames per second, and the assumption is that they need a bigger card. Nine times out of ten the frame time is being spent somewhere nobody has measured: an H.264 stream decoded on the CPU, a cvtColor on a 4K frame, a resize with the wrong interpolation, a cudaMemcpy per stage, or eight threads fighting over four physical cores.
This tutorial is about the boring part of computer vision performance work: measuring where the milliseconds go, then choosing the cheapest fix. It is deliberately not a list of GPU tricks. Every optimisation below is only worth doing if your own numbers say so, and we show how to produce those numbers.
1. Build a frame-time budget before you optimise anything
A production vision pipeline has five or six stages, and each one needs a line in a budget:
| Stage | Typical cost (1080p, modern x86 core) |
|---|---|
| Capture / decode | 2–15 ms CPU, ~0.5 ms with hardware decode |
| Colour conversion + resize | 1–6 ms |
| Normalise / blob construction | 1–4 ms |
| Inference | 5–40 ms |
| Post-processing (NMS, tracking) | 0.5–5 ms |
| Encode / draw / publish | 1–10 ms |
The rule we apply on every engagement: if inference is less than half of your frame time, do not touch the model. Buying a faster GPU to fix a pipeline where 70% of the time is CPU pre-processing is the most expensive way to make no difference.
2. Measure with TickMeter, not with wall-clock guesses
OpenCV ships a perfectly good stopwatch. Use one per stage, report the mean and the 95th percentile, and warm up before you record.
import time
import cv2
import numpy as np
from collections import defaultdict
class StageTimer:
"""Per-stage timings in milliseconds, with percentiles."""
def __init__(self):
self.samples = defaultdict(list)
class _Scope:
def __init__(self, parent, name):
self.parent, self.name = parent, name
def __enter__(self):
self.t0 = time.perf_counter()
def __exit__(self, *exc):
dt = (time.perf_counter() - self.t0) * 1000.0
self.parent.samples[self.name].append(dt)
def stage(self, name):
return StageTimer._Scope(self, name)
def report(self, warmup=20):
total = 0.0
for name, xs in self.samples.items():
a = np.asarray(xs[warmup:])
if a.size == 0:
continue
total += a.mean()
print(f"{name:<22} mean {a.mean():7.2f} ms p95 {np.percentile(a, 95):7.2f} ms n={a.size}")
print(f"{'TOTAL (mean)':<22} {total:7.2f} ms -> {1000.0 / max(total, 1e-6):5.1f} fps")
timer = StageTimer()
cap = cv2.VideoCapture("input.mp4")
net = cv2.dnn.readNet("detector.onnx")
while True:
with timer.stage("1_decode"):
ok, frame = cap.read()
if not ok:
break
with timer.stage("2_preprocess"):
blob = cv2.dnn.blobFromImage(frame, 1 / 255.0, (640, 640), swapRB=True, crop=False)
with timer.stage("3_infer"):
net.setInput(blob)
out = net.forward()
with timer.stage("4_postprocess"):
# NMS, tracking, business logic
pass
timer.report()
In C++ the equivalent is cv::TickMeter with start()/stop(), getAvgTimeMilli() and getFPS(). Two measurement rules that matter more than the tooling:
- Discard the first 10–20 iterations. Lazy backend initialisation, kernel compilation (OpenCL) and page faults all land in the first frames. Reporting them as steady-state cost is how people conclude that OpenCL is slow.
- Report p95, not just the mean. A pipeline whose mean is 25 ms and p95 is 90 ms will drop frames on an RTSP stream. Averages hide the failure mode you were hired to fix.
For Python pipelines where you do not yet know which call is hot, sample the whole process instead of instrumenting blindly:
pip install py-spy
py-spy record -o profile.svg -- python pipeline.py # flame graph
py-spy top -- python pipeline.py # live view
On the C++ side, perf record -g ./pipeline && perf report will tell you within a minute whether you are inside cv::resize, a memcpy, or the DNN backend.
3. Fix decode before you fix maths
Decode is the most commonly mis-attributed cost. cv2.VideoCapture on a 1080p25 H.264 RTSP feed will happily burn a core and a half in software decode, and that cost scales linearly with the number of cameras — which is why a "working" 4-camera demo collapses at 16 cameras on the same box.
The fix is a capture pipeline that decodes on fixed-function hardware and hands you frames in the layout you need:
# NVIDIA (nvv4l2decoder on Jetson, nvh264dec on desktop) via the GStreamer backend
pipeline = (
"rtspsrc location=rtsp://cam/stream latency=200 protocols=tcp ! "
"rtph264depay ! h264parse ! nvv4l2decoder ! "
"nvvidconv ! video/x-raw,format=BGRx ! "
"videoconvert ! video/x-raw,format=BGR ! "
"appsink drop=true max-buffers=2 sync=false"
)
cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)
Three details that decide whether this helps:
drop=true max-buffers=2on theappsink. Without it, a slow consumer builds an unbounded queue and your latency grows until the pipeline stalls. Live vision systems should drop frames, not buffer them.- Check the backend you actually got:
cap.getBackendName(). A malformed GStreamer string silently falls back to FFmpeg software decode and you optimise nothing. - The
videoconvertto BGR is a full-frame CPU pass. If your inference backend can take NV12 or RGBA, delete that step; it is often 3–4 ms per 1080p frame on its own.
On Intel hardware the equivalent is VA-API (vaapih264dec), and on constrained SBCs v4l2h264dec. Any of them beats software decode by an order of magnitude in CPU time.
4. Do fewer full-frame passes
Before reaching for a GPU, delete work. The cheapest wins we see repeatedly:
- Fuse resize and colour conversion into the blob step.
cv2.dnn.blobFromImagedoes scale, resize, swapRB and mean subtraction in one pass. A hand-rolledcvtColor→resize→astype(np.float32)→transposechain allocates four intermediate 1080p buffers and is typically 2–3× slower. - Downscale first, then filter. Blurring at 4K and then resizing to 640 is pure waste.
cv2.resizewithINTER_AREAfor downscale is both faster and better-looking thanINTER_CUBIChere. - Reuse buffers. Every OpenCV function with a
dstargument can write into a preallocatedMat/ndarray. In a 30 fps pipeline, allocator churn on 1080p three-channel frames is measurable and, worse, causes jitter. - Only process the ROI. If the conveyor occupies 40% of the frame, crop first. This is the single largest speedup available in most inspection systems and it costs nothing.
roi = frame[y0:y1, x0:x1] # a view, no copy
small = cv2.resize(roi, (640, 384), interpolation=cv2.INTER_AREA)
blob = cv2.dnn.blobFromImage(small, 1 / 255.0, (640, 384), swapRB=True)
5. UMat / OpenCL: the cheap GPU path (when it is cheap)
OpenCV's Transparent API routes operations to OpenCL when the data is in a UMat. It is a two-line change and works on integrated Intel and AMD GPUs, which is why it is worth trying before a CUDA rebuild.
print(cv2.ocl.haveOpenCL()) # is a device available?
cv2.ocl.setUseOpenCL(True)
umat = cv2.UMat(frame) # upload
gray = cv2.cvtColor(umat, cv2.COLOR_BGR2GRAY) # runs on device
blur = cv2.GaussianBlur(gray, (7, 7), 1.5)
edges = cv2.Canny(blur, 60, 160)
result = edges.get() # download
The rule that decides whether this wins: keep the whole chain in UMat and call .get() once. Each UMat(...)/.get() pair is a host-device transfer of several megabytes. A pipeline that uploads, does one cvtColor, downloads, does NMS on the CPU, uploads again, will be slower than pure CPU code — and this is the most common reason teams write off OpenCL after an afternoon.
Also beware: not every function has an OpenCL kernel. Unsupported calls fall back to the CPU path and force a download. Profile with cv2.ocl.setUseOpenCL(False) as a control run; if the delta is under ~15%, keep the CPU version and save yourself the driver support matrix.
6. CUDA: real speedups, on the condition that data stays resident
If you have an NVIDIA GPU and a build with opencv_cudaarithm, cudaimgproc and cudawarping, the cv2.cuda namespace gives you explicit control — which is exactly what you want, because the enemy is transfers, not maths.
stream = cv2.cuda.Stream()
gpu_frame = cv2.cuda.GpuMat()
gpu_small = cv2.cuda.GpuMat()
clahe = cv2.cuda.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
while True:
ok, frame = cap.read()
if not ok:
break
gpu_frame.upload(frame, stream) # one upload
gpu_gray = cv2.cuda.cvtColor(gpu_frame, cv2.COLOR_BGR2GRAY, stream=stream)
gpu_eq = clahe.apply(gpu_gray, stream=stream)
cv2.cuda.resize(gpu_eq, (640, 384), dst=gpu_small, stream=stream)
stream.waitForCompletion()
small = gpu_small.download() # one download
Practical notes from real deployments:
- Use pinned (page-locked) host memory (
cv2.cuda.HostMem) for the frames you upload every iteration. Pageable-memory transfers can be 2× slower and cannot overlap with compute. - Use streams and double-buffer. Upload frame n+1 while frame n computes. Without overlap you pay transfer and compute serially and lose most of the theoretical win.
- Time with events, not with the CPU clock. CUDA calls are asynchronous; a
perf_counteraround an async launch measures the launch, not the work. Synchronise explicitly before you stop the timer. - Keep DNN inference on the same device. Mixing
cv2.cudapre-processing with a CPU DNN backend means you download the blob just to upload it again inside the backend. Either setnet.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)/DNN_TARGET_CUDA_FP16, or run pre-processing on the CPU and let a TensorRT/ONNX Runtime engine own the device. Our Jetson Orin Nano + TensorRT tutorial walks the zero-copy version of that hand-off on embedded hardware.
7. G-API when the graph, not the operation, is the bottleneck
cv::gapi lets you declare the pipeline as a graph and compile it, so OpenCV can fuse operations, avoid intermediate buffers and pick a backend per node. It is most valuable when you have many small operations per frame — exactly the case where per-call overhead and memory traffic dominate.
#include <opencv2/gapi.hpp>
#include <opencv2/gapi/core.hpp>
#include <opencv2/gapi/imgproc.hpp>
cv::GMat in;
cv::GMat gray = cv::gapi::BGR2Gray(in);
cv::GMat blur = cv::gapi::gaussianBlur(gray, cv::Size(5, 5), 1.2);
cv::GMat edges = cv::gapi::Canny(blur, 60, 160);
cv::GComputation pipeline(cv::GIn(in), cv::GOut(edges));
// Compile once, reuse for every frame; swap in the OpenCL/Fluid backend here.
auto compiled = pipeline.compile(cv::descr_of(firstFrame),
cv::compile_args(cv::gapi::core::fluid::kernels()));
compiled(cv::gin(frame), cv::gout(out));
Be honest about the trade-off: G-API is a C++-first API with a real learning curve and a smaller kernel library than the classic modules. We reach for it on high-throughput streaming systems and leave straightforward per-frame pipelines alone.
8. Threading: the accidental 2× loss
OpenCV parallelises many primitives internally. If you also run four Python worker processes, each with OpenCV spawning threads across all cores, every worker oversubscribes and total throughput drops while every core reads 100% busy.
The rule for multi-camera and multi-process deployments:
import cv2
cv2.setNumThreads(1) # one OpenCV thread per worker process
Then scale by processes (one per camera or per shard), pinned if necessary, and re-measure. Related traps worth checking in the same pass:
OMP_NUM_THREADS/OPENBLAS_NUM_THREADSfor whatever NumPy and the inference runtime pull in — set them explicitly, before importing.- The Python GIL: threads help for I/O (RTSP reads, disk writes) and for time spent inside OpenCV C++ calls that release the GIL; they do not help for Python-level maths. Move per-detection loops into vectorised NumPy.
- One reader thread per camera with a depth-1 queue that overwrites, so a slow consumer drops old frames instead of accumulating latency.
9. A benchmark harness you can hand to a customer
Optimisation claims should be reproducible. The harness we ship on performance engagements records, for every configuration:
- Build and environment:
cv2.getBuildInformation()(backend, OpenCL, CUDA arch), driver version, CPU model, governor, GPU clocks. - Fixed input: the same 500-frame clip, decoded identically, no live camera.
- Per-stage mean and p95, plus end-to-end fps and peak RSS/GPU memory.
- A correctness check: mean absolute difference of outputs against the CPU baseline, so a "speedup" that changed the answer is caught immediately. FP16 and INT8 paths make this non-negotiable.
- A one-line verdict per config: kept or rejected, with the number that decided it.
That last point is what turns performance work from folklore into engineering. Every accepted change should have a before/after row someone can re-run six months later.
10. Where teams usually end up
Across the pipelines we have profiled, the ranked list of wins is remarkably stable:
- Hardware decode instead of software decode (biggest win on multi-camera systems).
- Deleting redundant colour conversions and full-frame passes; cropping to the ROI.
- Correct thread configuration for the process topology.
- A single resident GPU pipeline (CUDA or a TensorRT/OpenVINO engine) with pinned memory and streams.
- Model-level work — smaller input, pruning, INT8 — last, because it is the only one that can change your accuracy.
If your OpenCV pipeline is missing its frame budget and you would rather have the measurements than the guesswork, get in touch. We do short, fixed-scope profiling engagements: a per-stage budget for your real pipeline on your real hardware, a ranked list of fixes with expected gains, and a benchmark harness your team keeps.