Thermal cameras have quietly become cheap enough to put on everything. A 256x192 or 640x512 LWIR core now costs less than the industrial machine-vision camera sitting next to it, and clients arrive with the same request in three different industries: electrical substations, rooftop solar strings, refrigerated logistics, livestock barns, building envelopes. "We want to see hot spots automatically." Then a second sentence follows, and that is where the project actually lives: "...and we need to know which component is hot."
A thermal frame on its own rarely answers the second question. Radiometric imagery has no text, weak texture and, at 256x192, very little geometry. The RGB frame beside it knows exactly what it is looking at — asset labels, panel boundaries, cable IDs, object classes from a detector you already trust. The value is in fusing the two: detect and identify in visible light, measure temperature in thermal, and report a number tied to a named component.
This tutorial builds that pipeline in OpenCV 5. Nothing here is exotic: it is calibration, registration, sampling discipline and a small amount of honesty about what a temperature reading is worth. We have not covered multi-sensor registration anywhere else in these tutorials, and it is the part clients most consistently underestimate.
1. What a thermal camera actually gives you
Before any code, get the data path right. Most fusion projects fail at this step and never recover.
A thermal core exposes two very different streams:
- A visualisation stream — 8-bit, already auto-gained, often already colour-mapped (Ironbow, White Hot). Pretty. Worthless for measurement.
- A radiometric stream — 16-bit per pixel, where the integer encodes sensor response you can convert to temperature. On many cores the convention is centikelvin:
T_celsius = raw / 100.0 - 273.15.
If your capture code hands you a three-channel colour image, you have already thrown the measurement away. Auto-gain rescales frame to frame, so the same pixel value means a different temperature in consecutive frames.
import cv2
import numpy as np
cap = cv2.VideoCapture(thermal_index)
# Critical: stop OpenCV converting to 8-bit BGR
cap.set(cv2.CAP_PROP_CONVERT_RGB, 0)
cap.set(cv2.CAP_PROP_FORMAT, -1) # deliver the raw frame as-is
ok, raw = cap.read()
print(raw.dtype, raw.shape) # expect uint16 (or uint8 with 2x width)
if raw.dtype == np.uint8:
# Some UVC cores present 16-bit data as double-width 8-bit.
raw = raw.view(np.uint16).reshape(height, width)
temp_c = raw.astype(np.float32) / 100.0 - 273.15
Two habits worth building in from day one:
- Store the raw 16-bit array, not the picture. Save frames as 16-bit PNG or in a container that preserves them. A colour-mapped JPEG is a screenshot, not a measurement.
- Log the emissivity and reflected-temperature assumptions. Vendor SDKs bake an emissivity value into their conversion. Bare metal is around 0.1 and painted steel is around 0.95; getting that wrong can move an apparent temperature by tens of degrees. If you cannot control it, record it, so the number in your report is reproducible.
2. Calibrate both cameras — with a target both can see
Fusion needs the geometric relationship between the two sensors. That means stereo calibration across modalities, and the standard printed chessboard is invisible to a thermal camera because paper ink and paper have nearly identical emissivity.
Three targets that work in practice, roughly in order of how much we like them:
- Cut-out board. Laser-cut or drill a grid of holes in thin aluminium or foam board, and backlight it with something warm and diffuse (a heated plate, or simply a sunlit wall). The pattern appears as a real thermal contrast and as a real visible contrast. Detect circle-grid centres rather than chessboard corners:
cv2.findCirclesGrid. - Emissivity-contrast board. A chessboard printed onto aluminium with matte black squares. Uniform temperature, but the emissivity difference makes the pattern visible in LWIR. Works best against a cool background.
- Heated resistive board. A PCB with copper squares that self-heat. Best contrast, most work to build.
The board must be planar and the geometry known to sub-millimetre accuracy, because every later number depends on it.
Calibrate each camera separately first, then solve for the transform between them. Note the OpenCV 5 module layout: calibration entry points sit in calib, the 3D utilities in 3d; in Python the flat cv2. namespace still resolves them. Our OpenCV 5 camera calibration tutorial covers reading reprojection error honestly, and everything there applies to each camera in isolation.
flags = cv2.CALIB_FIX_INTRINSIC # trust the per-camera solves
ret, K_rgb, d_rgb, K_th, d_th, R, T, E, F = cv2.stereoCalibrate(
object_points, img_points_rgb, img_points_th,
K_rgb, d_rgb, K_th, d_th, size_th,
flags=flags,
criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 1e-6),
)
print("stereo RMS:", ret)
Expect a worse RMS than an RGB-only stereo rig. Thermal corner localisation is fundamentally blurrier: an LWIR lens is germanium, the optics are slower and the effective pixel is larger. An RMS of 0.3-0.6 px on the thermal side is a good result. If you see 2 px, your target contrast is too low — fix the target, not the optimiser.
3. Registration: pick the model your geometry justifies
There are three honest choices here, and consultants get into trouble by reaching for the most sophisticated one by default.
Homography (planar scene). If the subject is effectively a plane at a roughly constant working distance — a solar panel photographed from a drone at 20 m, a printed circuit board on a fixed inspection stand, a building facade — a single 3x3 homography maps thermal pixels into the RGB frame and is accurate to a pixel or two. It is also cheap and easy to explain.
H, mask = cv2.findHomography(pts_th, pts_rgb, cv2.USAC_MAGSAC, 3.0)
warped = cv2.warpPerspective(temp_c, H, (w_rgb, h_rgb),
flags=cv2.INTER_NEAREST, # do NOT interpolate temperatures
borderValue=np.nan)
Two details matter more than the choice of solver. Use USAC_MAGSAC rather than plain RANSAC — it is the modern robust estimator in OpenCV and it is markedly better behaved with the modest point counts a cross-modal calibration gives you. And warp with INTER_NEAREST: bilinear interpolation between a 60 °C pixel and a 25 °C pixel invents a 42 °C reading that no physical object produced. If you must resample, resample the mask and the temperatures separately and be explicit about it.
Depth-aware reprojection (non-planar scene, known depth). If the scene has real relief and you have depth — from a stereo pair, a ToF sensor, or a monocular metric depth model as in our monocular metric depth tutorial — project each RGB pixel into 3D using its depth, transform by R, T, and reproject into the thermal camera. This is the only approach that is correct at multiple distances, and the extra term you now carry is depth error.
Feature-based per-frame alignment. Tempting and usually wrong. SIFT and ORB descriptors are built on gradient statistics that simply do not correspond across LWIR and visible light; a warm wall can be a dark wall. If you have no calibration option at all, mutual-information alignment or a cross-modal learned matcher is the research-grade route — but budget for it as research, not integration. Fixed-rig calibration is cheaper and repeatable.
Whichever model you use, validate parallax explicitly. Put the calibration target at the near, mid and far ends of the real working range and measure the residual misalignment in RGB pixels at each. A homography fitted at 5 m will be tens of pixels wrong at 1 m. That table is what tells the client the minimum object size the system can measure — and it belongs in the deliverable.
4. Detect in RGB, measure in thermal
Now the pipeline pays off. Run whichever detector or segmentation model already works on your visible imagery through OpenCV 5's DNN engine or ONNX Runtime, get a mask per component, warp the mask into thermal space, and reduce the temperatures under it.
def component_stats(temp_th, mask_rgb, H_inv, erode_px=3):
"""Temperature statistics for one RGB-space mask, sampled in thermal space."""
h, w = temp_th.shape
m = cv2.warpPerspective(mask_rgb, H_inv, (w, h), flags=cv2.INTER_NEAREST)
# Shrink the mask: edge pixels straddle the boundary and mix backgrounds.
if erode_px:
k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (erode_px, erode_px))
m = cv2.erode(m, k)
vals = temp_th[m > 0]
if vals.size < 20: # too few pixels to trust
return None
return {
"n_px": int(vals.size),
"p50": float(np.percentile(vals, 50)),
"p99": float(np.percentile(vals, 99)),
"max": float(vals.max()),
}
Three sampling rules we apply on every engagement:
- Erode the mask. A boundary pixel in a 256x192 thermal frame covers a big patch of the world and averages the component with whatever is behind it. Eroding costs you a little area and removes most of the false extremes.
- Report a high percentile, not the max. A single hot pixel is as likely to be sensor noise or a dead element as a fault.
p99over a component with 300 valid pixels is stable;maxis not. State which you used. - Enforce a minimum pixel count. Below roughly 3x3 valid pixels a reading is dominated by the point spread function of the lens, and the true peak is smeared low. Anything smaller than that gets flagged "too small to measure", not given a number.
5. Decide on deltas, not absolutes
The single most useful modelling decision in thermal inspection: absolute temperature thresholds are fragile, differential ones are robust.
Ambient temperature drifts. Sun load varies. Emissivity is an estimate. But identical components under identical load should be at identical temperatures, and that comparison cancels most of the error.
def flag_outliers(components, k=3.0, min_delta=5.0):
"""Flag components hot relative to their own peer group."""
temps = np.array([c["p99"] for c in components])
med = np.median(temps)
mad = np.median(np.abs(temps - med)) + 1e-6
flags = []
for c, t in zip(components, temps):
robust_z = 0.6745 * (t - med) / mad
delta = t - med
if robust_z > k and delta > min_delta:
flags.append({**c, "delta_k": round(float(delta), 1),
"peer_median": round(float(med), 1)})
return flags
Median and MAD rather than mean and standard deviation, because a genuine fault would otherwise inflate the very statistic used to detect it. The min_delta floor stops the system reporting a 0.4 K spread as an anomaly on a perfectly healthy panel — statistically significant, physically meaningless.
Standards practice already works this way. Solar thermography guidance is written in terms of temperature differences between comparable cells and modules, and electrical inspection uses delta-T against similar components under similar load. Align your thresholds with the client's standard and the report writes itself.
6. What to hand over
A fusion deliverable that survives review contains more than code:
- A calibration record: intrinsics, distortion,
R/T, stereo RMS, target type, date, and the operator. Rigs get bumped; you need to know when the numbers were last true. - A parallax table: residual misalignment in pixels at the near, mid and far ends of the working range, with the resulting minimum measurable component size.
- A measurement-uncertainty statement: sensor accuracy (typically ±2 K or ±2% for an uncooled core), emissivity assumption, reflected-temperature assumption, registration error, and how they combine. Clients who are going to act on your output need to know that "68.3 °C" means roughly 68 ± 3 K, and that the delta is far tighter than the absolute.
- A drift check: a shutter/NUC event log and a periodic re-check against a blackbody or a known reference surface. Uncooled microbolometers drift; scheduled non-uniformity correction is not optional in a long-running deployment.
- Golden frames in CI, exactly as in our pipeline regression testing tutorial — store the 16-bit thermal frames and the RGB frames together and assert that component temperatures and flags stay stable when a model or a library version changes.
7. Common failure modes
- Colour-mapped input. Someone swaps in the visualisation stream during an integration sprint and every temperature silently becomes a colour-lookup artefact. Assert
dtype == uint16at the boundary of your pipeline and fail loudly. - Shutter frames. Most uncooled cores periodically close an internal shutter for non-uniformity correction, producing a flat frame. Detect and drop those, or your maxima will be nonsense.
- Reflections. Polished metal reflects the sky, the operator, a nearby hot motor. A shiny surface reading 90 °C may be a mirror. Low-emissivity surfaces need either an applied high-emissivity patch or an explicit "not measurable" flag.
- Sun load. Outdoor surveys are frequently repeated at dawn, and for good reason: differential solar heating creates hot spots with no electrical fault behind them.
- Focus. LWIR lenses are often manually focused and easy to leave wrong. A defocused thermal frame reads peak temperatures low, which is the dangerous direction for an inspection system.
Where this typically goes next
Once temperature is a per-component attribute rather than a picture, the rest is ordinary engineering: time-series per asset, trend alarms rather than single-frame alarms, and a review UI where an inspector sees the RGB crop and the thermal crop side by side with the delta. That combination — identify in visible light, measure in thermal, decide on deltas — is what turns a thermal camera from a demo into an inspection system.
If you are scoping an RGB-thermal fusion build, the parts worth getting expert help on early are the calibration target and the registration model, because every number downstream inherits their error. We do this work as a fixed-scope engagement: rig and target design, cross-modal calibration, registration validated across the real working range, and the uncertainty statement to go with it.
Talk to us about your thermal fusion project and we will tell you honestly whether a homography will do or whether you need depth.