Detection and tracking get the attention, but a large share of the OpenCV work that actually pays is reading characters: serial numbers stamped into metal, expiry codes ink-jetted onto a bottle, container IDs on a moving chassis, meter dials, batch labels, shipping documents scanned on a phone. Clients call it "just OCR" and expect a two-week job. It is rarely two weeks, because the hard part is not the recogniser — it is the pixels you feed it and the confidence rules you wrap around it.
This tutorial builds a scene-text pipeline in OpenCV 5: detect text regions, straighten them, recognise the characters, and then decide — with an explicit, testable rule — whether the read is trustworthy enough to act on.
1. Know which OCR problem you have
Three problems get lumped together and they need different pipelines:
- Document OCR — flat page, dense text, high resolution, controlled lighting. A document engine (Tesseract, PaddleOCR, a cloud API) is usually the right answer, and OpenCV's job is dewarping and binarisation.
- Scene text — signs, labels, packaging in an uncontrolled field of view. Arbitrary rotation, curvature, glare, motion. This is what OpenCV's
dnntext pipeline is built for. - Constrained industrial code reading — a fixed camera, a known font, a known number of characters, a known location within a jig. This is the easiest to make accurate and the one people over-engineer. If you can fix the geometry, half the pipeline below disappears.
Ask which one you have before writing any code. A client asking for "99.9% OCR accuracy" on scene text at 1080p with no lighting control is asking for something that no engine delivers; the same client with a fixed camera, a bar light and a 12 mm lens can often get there.
2. The models: DB for detection, CRNN for recognition
OpenCV ships high-level wrappers so you do not have to hand-roll the pre- and post-processing:
cv2.dnn_TextDetectionModel_DB— differentiable-binarisation detector, outputs quadrilaterals. Robust, ONNX-exportable, works on curved and rotated text.cv2.dnn_TextDetectionModel_EAST— older, faster, weaker on small and curved text. Still fine for large fixed-position labels.cv2.dnn_TextRecognitionModel— CRNN-style recogniser with CTC decoding and a supplied vocabulary file.
In OpenCV 5 these live behind the rewritten DNN engine, which prefers ONNX and drops much of the legacy Caffe/Darknet importer surface. If you are porting a 4.x script that loaded .caffemodel weights, convert them first — see our Caffe/Darknet to ONNX modernisation guide.
import cv2
import numpy as np
# --- detector ---
detector = cv2.dnn_TextDetectionModel_DB("DB_TD500_resnet50.onnx")
detector.setBinaryThreshold(0.3) # lower = more candidate regions
detector.setPolygonThreshold(0.5) # confidence to keep a box
detector.setUnclipRatio(2.0) # box padding; raise if characters get clipped
detector.setMaxCandidates(200)
detector.setInputParams(
scale=1.0 / 255.0,
size=(736, 736), # must be a multiple of 32
mean=(122.67891434, 116.66876762, 104.00698793),
swapRB=True)
# --- recogniser ---
recogniser = cv2.dnn_TextRecognitionModel("crnn_cs.onnx")
recogniser.setDecodeType("CTC-greedy")
with open("alphabet_94.txt") as f:
recogniser.setVocabulary([line.strip() for line in f])
recogniser.setInputParams(
scale=1.0 / 127.5, size=(100, 32), mean=(127.5, 127.5, 127.5), swapRB=True)
Two details that cause most first-run failures. The DB input size must be a multiple of 32 or you get a shape assertion deep in the graph. And the recogniser's vocabulary file must match the exported model exactly — a 36-character alphanumeric model fed a 94-character vocabulary produces confident, fluent nonsense, which is far worse than an error.
3. Detect, then crop with a real perspective warp
The detector returns quadrilaterals, not axis-aligned rectangles. Cropping the bounding rectangle of a rotated quad drags in neighbouring text and background; the recogniser then reads the mess. Warp instead:
def four_point_warp(image, quad, out_h=32):
quad = order_corners(np.array(quad, dtype=np.float32))
(tl, tr, br, bl) = quad
width = int(max(np.linalg.norm(tr - tl), np.linalg.norm(br - bl)))
height = int(max(np.linalg.norm(bl - tl), np.linalg.norm(br - tr)))
if width < 8 or height < 4:
return None
out_w = max(8, int(round(width * out_h / height)))
dst = np.array([[0, 0], [out_w - 1, 0], [out_w - 1, out_h - 1], [0, out_h - 1]],
dtype=np.float32)
M = cv2.getPerspectiveTransform(quad, dst)
return cv2.warpPerspective(image, M, (out_w, out_h), flags=cv2.INTER_CUBIC)
def order_corners(pts):
s, d = pts.sum(axis=1), np.diff(pts, axis=1).ravel()
return np.array([pts[np.argmin(s)], pts[np.argmin(d)],
pts[np.argmax(s)], pts[np.argmax(d)]], dtype=np.float32)
order_corners matters more than it looks. DB does not guarantee corner ordering, and an unordered quad produces a warp that is mirrored or rotated 180°. The classic symptom is a pipeline that reads perfectly on half your test images and returns reversed strings on the other half.
Now run the pipeline:
def read_text(image):
quads, confs = detector.detect(image)
results = []
for quad, det_conf in zip(quads, confs):
crop = four_point_warp(image, quad)
if crop is None:
continue
text = recogniser.recognize(crop)
if text:
results.append({"text": text, "quad": quad, "det_conf": float(det_conf)})
return results
If text appears vertically or upside down in your scenes, try each crop at 0° and 180° and keep whichever produces the higher-scoring read — cheap, and it removes a whole class of failures.
4. Get the pixels right before blaming the model
Recognition accuracy is dominated by input quality, and the fixes are old-fashioned image processing.
Resolution. CRNN recognisers want roughly 20–32 pixels of character height. Below about 12 px, accuracy collapses regardless of model. Measure your smallest character in pixels at the real working distance; if it is 8 px, the answer is a longer lens or a closer camera, not a better network.
Glare and uneven lighting on curved or metallic surfaces. CLAHE on the luminance channel, not on BGR:
lab = cv2.cvtColor(crop, cv2.COLOR_BGR2LAB)
l, a, b = cv2.split(lab)
l = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(l)
crop = cv2.cvtColor(cv2.merge([l, a, b]), cv2.COLOR_LAB2BGR)
Low-contrast engraved or dot-peen marking. A morphological black-hat pulls dark characters off a bright, uneven background far better than global thresholding:
gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (13, 13))
gray = cv2.morphologyEx(gray, cv2.MORPH_BLACKHAT, kernel)
gray = cv2.normalize(gray, None, 0, 255, cv2.NORM_MINMAX)
Document capture. For a phone-photographed page, find the page contour, warp it flat, then use adaptive thresholding — cv2.adaptiveThreshold with a block size around 1/30th of the image width beats Otsu whenever illumination varies across the page.
Motion blur. Do not deblur; fix the exposure. A 5 ms exposure with a brighter light and a global-shutter sensor solves what no algorithm will. On a conveyor, compute the maximum exposure from line speed and required blur: t_max = blur_px * pixel_size / line_speed.
Lighting is the highest-leverage variable in industrial OCR and it is not a software decision. Coaxial or dome lighting for shiny surfaces, low-angle grazing light for embossed and engraved characters, backlight for silhouettes. Budget for it in week one.
5. Constrain the output — this is where accuracy comes from
A raw recogniser produces the most likely character string. Your application usually knows far more than that, and every constraint you can apply is free accuracy.
import re
PATTERN = re.compile(r"^[A-Z]{3}\d{7}$") # e.g. container-style code
CONFUSIONS = str.maketrans({"O": "0", "I": "1", "l": "1", "S": "5", "B": "8", "Z": "2"})
def normalise(raw):
s = raw.upper().replace(" ", "").replace("-", "")
if PATTERN.match(s):
return s, 1.0
candidate = s.translate(CONFUSIONS)
if PATTERN.match(candidate):
return candidate, 0.8
return None, 0.0
Better still, use a real check digit if the code format has one (ISO 6346 container codes, GS1 identifiers, IMEI, VIN all do). A check digit converts "probably right" into "verified", and it is the single cheapest accuracy improvement available in code reading.
Temporal voting is the other big win. If you get 15 frames of the same object, do not act on one read:
from collections import Counter
def vote(reads, min_frames=3, min_share=0.6):
counts = Counter(r for r in reads if r)
if not counts:
return None
text, n = counts.most_common(1)[0]
if n >= min_frames and n / max(1, len(reads)) >= min_share:
return text
return None
Requiring the same string from at least three frames and a clear majority typically cuts the error rate by an order of magnitude versus a single-frame read, at the cost of a few hundred milliseconds.
6. Define "no read" and mean it
The most damaging OCR failure is not a missing read — it is a confident wrong one. A missed read routes to a human; a wrong read silently ships the wrong pallet.
Build an explicit three-way outcome and expose the thresholds as configuration:
- Accepted — passes the format pattern (and check digit), voted across frames, detector and recogniser confidences above threshold.
- Review — plausible but unconfirmed. Route to a human queue with the crop attached.
- No read — nothing usable. Trigger a retry, a re-present, or an operator prompt.
Report accuracy in those terms, not as a single percentage. The number a client actually needs is a pair: read rate (share of items automatically accepted) and error rate within accepted reads (share of accepted reads that were wrong). A 92% read rate with a 0.05% error rate is a good industrial system. A 98% read rate with a 2% error rate is usually worse than useless, because downstream nobody can trust any read.
Set the thresholds by sweeping them against a labelled validation set and plotting error rate versus read rate. That curve, not a demo video, is what should go into the acceptance criteria.
7. Make the test set real
Collect ground truth from the actual installation, not from clean samples the client emailed you. A usable validation set has at least a few hundred images and deliberately includes the ugly cases: worn and partially rubbed stamps, condensation, dirt, low sun and glare, night-time and IR illumination, damaged or overprinted labels, and the near-miss characters your confusion map touches.
# regression harness: fail the build if accuracy on the golden set drops
import json
golden = json.load(open("golden.json")) # [{"path":..., "expected":...}, ...]
accepted = correct = 0
for item in golden:
img = cv2.imread(item["path"])
text, _ = normalise(best_read(read_text(img)))
if text:
accepted += 1
correct += (text == item["expected"])
read_rate = accepted / len(golden)
error_rate = 1 - (correct / max(1, accepted))
print(f"read_rate={read_rate:.3f} error_rate={error_rate:.4f}")
assert read_rate > 0.90 and error_rate < 0.005, "OCR regression"
Run it in CI on every model or preprocessing change. Vision pipelines drift quietly; a threshold someone nudged during a site visit will otherwise show up as a support ticket six weeks later.
8. Performance notes
On a desktop CPU the DB detector at 736×736 costs roughly 60–120 ms per frame, and each recogniser crop a few milliseconds. Practical levers:
- Do not detect every frame. Detect at 3–5 Hz, track the boxes in between, and only re-recognise when a region changes materially.
- Restrict the ROI. If the label is always in the same third of the frame, crop before detection — the cheapest speedup available, and it also reduces false positives.
- Batch the crops through the recogniser rather than one call per box.
- Pick the backend deliberately.
DNN_BACKEND_CUDAon a discrete GPU, OpenVINO on Intel, TensorRT on Jetson. See our Jetson Orin Nano guide for the embedded path. - Quantise the recogniser to INT8 if you are CPU-bound; character recognition tolerates quantisation well, but re-run the golden-set harness afterwards rather than assuming it.
Where this fits
OCR projects fail on the boring parts: character height in pixels, lighting geometry, corner ordering, a vocabulary file that does not match the model, and the absence of an honest accept/reject rule. Get those right and a fairly ordinary DB + CRNN pipeline will outperform a much fancier model dropped into a bad capture setup.
SentientSight's OpenCV consultants design and audit text-reading systems end to end — optics and lighting specification, detection and recognition pipelines, check-digit and voting logic, and the read-rate/error-rate acceptance testing that makes the result contractual rather than anecdotal. If you have codes, labels or documents that need reading reliably, get in touch.