Single-camera tracking is a solved-enough problem: detect, associate, count. The question that follows it in almost every real deployment is harder. Is the person who just left camera 3 the same person who appeared on camera 7 eleven seconds later? Retail wants dwell time across a store. Logistics wants a pallet's route across a yard. Safety wants to know whether the operator who entered the cell is the one who signed in at the door.
That is cross-camera re-identification, and it is where a lot of otherwise competent vision projects quietly fall apart — because teams reach for an appearance embedding first, when the cheap win is geometry. This tutorial builds the handover layer properly in OpenCV 5: ground-plane homography so every camera reports metres instead of pixels, appearance embeddings via ONNX as a tie-breaker, a gated association rule, and the identity-accuracy numbers you need before you promise anything.
It assumes you already have per-camera detection and tracking working. If you do not, start with multi-object tracking and counting in OpenCV 5, and with scaling to 32 RTSP cameras for the ingest side.
1. Decide what "same" has to mean before you build anything
Re-ID projects fail at the requirements stage far more often than at the model stage. Three very different systems get requested with the same sentence:
- Soft analytics. "Roughly how long do people spend in the store?" Aggregate statistics; individual mistakes wash out. Achievable, cheap, and usually what the business actually needs.
- Hard identity linkage. "Show me this specific person's full path and let an operator act on it." Every error is visible and attributable. Expensive, and legally loaded.
- Biometric identification. "Who is this?" Different system, different law, different conversation. Appearance re-ID is not this, and you should say so explicitly in writing.
Also pin the tolerated error direction. A false link (two people merged into one track) corrupts the analytics silently. A missed link (one person split into two tracks) inflates your counts. You cannot minimise both; ask which one the customer would rather have, and tune the gate accordingly.
One more requirement to nail early: retention. Appearance embeddings computed from people are personal data under GDPR and, in the EU, deployments like this fall under the AI Act's transparency obligations. Decide the retention window (hours, not months, for embeddings), write it into the design, and see the redaction patterns in video anonymisation in OpenCV 5 for the storage side.
2. Ground-plane homography: the step people skip
If cameras overlook a flat floor or yard, a homography maps each camera's image plane to a shared world plane. Once you have it, two detections in different cameras can be compared in metres — and most association ambiguity disappears before an embedding is ever computed.
Pick four or more well-separated points visible in the camera and measurable on the floor: column bases, floor-marking corners, painted line intersections, or surveyed markers. Measure them on site with a tape or laser distance meter in a single site coordinate frame.
import cv2
import numpy as np
# Image points (px) clicked in the camera view
img_pts = np.array([
[ 412, 688], [1180, 651], [1502, 902], [ 233, 961], [ 860, 742],
], dtype=np.float32)
# Corresponding world points (metres) on the shop-floor plane
world_pts = np.array([
[ 0.00, 0.00], [ 6.40, 0.00], [ 6.40, 3.20], [ 0.00, 3.20], [ 3.20, 1.10],
], dtype=np.float32)
H, inliers = cv2.findHomography(img_pts, world_pts, cv2.RANSAC, 3.0)
print("inliers:", inliers.ravel(), "\nH:\n", H)
Then validate it, because an unvalidated homography is a confident source of wrong numbers:
def to_world(H, pts_px):
pts = np.asarray(pts_px, np.float32).reshape(-1, 1, 2)
return cv2.perspectiveTransform(pts, H).reshape(-1, 2)
# Hold out two measured points that were NOT used in the fit
check_px = np.array([[1044, 833], [ 640, 700]], np.float32)
check_world = np.array([[ 4.80, 2.10], [ 2.00, 0.60]], np.float32)
err = np.linalg.norm(to_world(H, check_px) - check_world, axis=1)
print("held-out error (m):", err) # want < ~0.25 m across the working area
Four practical notes that decide whether this works:
- Use the foot point, not the box centre. A person's world position is where they touch the floor: the bottom-centre of the bounding box. Box centres float around chest height and project metres away from the true ground position. If the feet are occluded, estimate them from the visible box and flag the track as low-confidence.
- Undistort first. Feed the homography rectified coordinates from your camera calibration, or lens distortion leaks straight into your world error. See camera calibration in OpenCV 5.
- Accuracy degrades with distance. Error grows fast toward the horizon, where a pixel spans a large ground distance. Compute the held-out error per zone and define an explicit "trusted region" polygon per camera; treat detections outside it as detections only, never as positions.
- Homography assumes one flat plane. Ramps, stairs and mezzanines each need their own plane, or a different approach entirely.
Store each camera's calibration as a versioned artefact — H, the measured control points, held-out error, the trusted-region polygon, and the date. Cameras get bumped, and a homography that silently went stale produces plausible, wrong analytics for months.
3. Time synchronisation, which is not optional
Association across cameras compares timestamps. If camera clocks drift by two seconds, a person walking at 1.4 m/s appears to teleport 2.8 m, and your gate either rejects true matches or admits false ones.
- Run NTP (or PTP for tighter needs) on every camera and on the inference hosts, and monitor offset as a metric you alarm on.
- Timestamp frames at the capture end, not when your Python loop happens to get them. Pull the RTP/RTCP timestamp from the stream where available.
- Measure and record per-camera pipeline latency (decode + inference); it differs per stream and biases association if ignored.
- Log the offset with every track record so a reviewer can tell whether a suspicious link was a clock problem.
4. Appearance embeddings as the tie-breaker
Now, and only now, add appearance. A person re-ID network maps a cropped detection to a unit-length vector where the same identity lands close together. OSNet and similar architectures export cleanly to ONNX and run through OpenCV 5's DNN engine.
net = cv2.dnn.readNetFromONNX("osnet_x0_25_market.onnx")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)
MEAN = (0.485, 0.456, 0.406)
STD = (0.229, 0.224, 0.225)
def embed(crops):
"""crops: list of BGR person crops -> (N, D) L2-normalised embeddings."""
blob = cv2.dnn.blobFromImages(
crops, scalefactor=1 / 255.0, size=(128, 256), # (W, H) for person re-ID
mean=(0, 0, 0), swapRB=True, crop=False)
blob = (blob - np.array(MEAN).reshape(1, 3, 1, 1)) / np.array(STD).reshape(1, 3, 1, 1)
net.setInput(blob.astype(np.float32))
feats = net.forward()
return feats / (np.linalg.norm(feats, axis=1, keepdims=True) + 1e-9)
Two details cost most of the accuracy people lose here. First, aspect ratio: person re-ID models expect a 1:2 crop, and blobFromImages with crop=False will squash a wide box. Pad the box to 1:2 around the detection instead of stretching it. Second, crop hygiene: embed only clean observations. Reject crops that are heavily occluded, smaller than roughly 64 px wide, badly blurred, or clipped at the frame edge. A tracklet described by three good crops beats one described by thirty bad ones.
And do not embed every frame. Keep a small gallery per tracklet — say the five highest-quality crops, spread over time — and compare set to set:
from collections import deque
class Tracklet:
def __init__(self, cam_id, track_id, max_gallery=5):
self.cam_id, self.track_id = cam_id, track_id
self.gallery = deque(maxlen=max_gallery) # list of (quality, embedding)
self.last_world = None
self.last_t = None
def offer(self, quality, emb):
self.gallery.append((quality, emb))
def feats(self):
return np.stack([e for _, e in self.gallery]) if self.gallery else None
def appearance_distance(a: Tracklet, b: Tracklet):
fa, fb = a.feats(), b.feats()
if fa is None or fb is None:
return None
sim = fa @ fb.T # cosine, embeddings are L2-normalised
return float(1.0 - np.max(sim)) # best-of-set distance
Be honest about the domain gap. Public re-ID benchmarks report high accuracy on datasets that look nothing like a warehouse at 3 a.m. under sodium lighting, seen from a 4 m ceiling mount. Expect a large drop. Two people in the same hi-vis uniform are close to indistinguishable by appearance — which is exactly why geometry carries the decision and appearance only breaks ties.
5. The gated association rule
Combine the two signals with geometry as a hard gate, then appearance as the cost. This is the whole handover logic:
MAX_GAP_S = 25.0 # plausible transit time between the two views
MAX_SPEED_MPS = 2.0 # walking; raise for forklifts, lower for queues
APP_THRESHOLD = 0.35 # tuned on YOUR site, see section 6
def can_link(a: Tracklet, b: Tracklet, transit_prior_s=0.0):
"""Geometric feasibility gate. a exits, b appears later."""
if a.cam_id == b.cam_id:
return False
dt = b.last_t - a.last_t
if dt <= 0 or dt > MAX_GAP_S:
return False
dist = np.linalg.norm(b.last_world - a.last_world)
# allow the known minimum transit time for this camera pair
usable = max(dt - transit_prior_s, 0.1)
return dist / usable <= MAX_SPEED_MPS
def link_cost(a, b, transit_prior_s=0.0):
if not can_link(a, b, transit_prior_s):
return None
d_app = appearance_distance(a, b)
if d_app is None or d_app > APP_THRESHOLD:
return None
return d_app
Then solve the assignment globally rather than greedily — greedy matching locks in an early cheap pair and strands the correct one:
from scipy.optimize import linear_sum_assignment
def match(exits, entries, priors):
BIG = 1e6
C = np.full((len(exits), len(entries)), BIG)
for i, a in enumerate(exits):
for j, b in enumerate(entries):
c = link_cost(a, b, priors.get((a.cam_id, b.cam_id), 0.0))
if c is not None:
C[i, j] = c
rows, cols = linear_sum_assignment(C)
return [(exits[i], entries[j], C[i, j])
for i, j in zip(rows, cols) if C[i, j] < BIG]
Three things make this work in the field:
- A camera topology graph. Not every pair of cameras is reachable, and the ones that are have a minimum walking time. Encode adjacency and a per-pair transit prior; it removes a large share of candidate links for free and costs an afternoon with a site map.
- Entry and exit zones. Score handovers on the zone the track left through and the zone it appeared in. A track that vanished mid-frame is an occlusion or a tracker failure, not a handover, and should not be offered for linkage.
- Deferred decisions. Do not commit a link at the instant a track appears. Buffer a few seconds, let the tracklet accumulate quality crops, then decide. Latency is almost always cheaper than a wrong merge.
For overlapping fields of view, skip the appearance model for the overlap: two detections at the same world point at the same timestamp are the same object, and a nearest-neighbour match in metres with a sub-metre threshold is more reliable than any embedding.
6. Measuring it, and the numbers to quote
"It looks right on the demo clip" is not a result. Build an evaluation set and report defensible metrics.
Label a held-out window — 30 to 60 minutes covering a shift change, a busy period and a quiet one — with global identities across cameras. This is tedious; it is also the only thing that turns your system from a hope into a specification. Then report:
- IDF1 and ID switches for the multi-camera tracks (the standard MOT identity metrics).
- Handover precision and recall for the thing you actually built: of the links you made, how many were correct; of the true handovers, how many did you find.
- Rank-1 / mAP on your own cropped gallery, not on Market-1501, so you know the real domain gap.
- Held-out homography error in metres, per zone, from section 2.
Sweep APP_THRESHOLD on labelled data and plot precision against recall to choose the operating point deliberately. A tight threshold buys precision and loses handovers; a loose one merges strangers. Which way you lean is the requirements answer from section 1, not a default.
Then put it in CI. A fixed multi-camera clip plus expected identity metrics catches the day a model swap or a camera firmware update quietly halves your handover recall — see testing OpenCV 5 pipelines like software.
7. Running it in production
Some hard-won operational advice:
- Keep per-camera tracking at the edge and do association centrally. Send compact tracklet records — world coordinates, timestamps, a handful of embeddings — not video. It is a fraction of the bandwidth and keeps raw imagery local, which the privacy review will like.
- Cap global identity lifetime. An identity that has not been seen for longer than a plausible dwell time should be retired, not kept in the gallery forever. Unbounded galleries slow matching down and steadily raise the false-link rate.
- Monitor drift signals, not just uptime. Track handover rate per camera pair, mean appearance distance of accepted links, and detection counts per camera per hour. A camera that was nudged, refocused or partially occluded shows up in these long before anyone notices bad analytics.
- Provide a merge/split audit trail. Every global identity should record which tracklets were joined, when, on what cost, and under which calibration version. When someone questions a number — and they will — you need to be able to reconstruct the decision.
- Budget the compute. Embedding is cheap per crop but you do it many times per second across many cameras. Measure it with the same discipline as the rest of the pipeline; profiling OpenCV 5 pipelines applies unchanged.
The short version
Do geometry first. A validated ground-plane homography, synchronised clocks and a camera topology graph resolve most cross-camera ambiguity and cost you a site visit and an afternoon of arithmetic. Add appearance embeddings after that, as a gated tie-breaker with strict crop hygiene and a threshold tuned on your own labelled footage — and quote handover precision and recall from that footage, never from a public benchmark.
SentientSight's OpenCV consultants design and audit multi-camera tracking systems — site calibration, handover logic, identity metrics, privacy posture and drift monitoring — for analytics and safety deployments in production. If you need cross-camera numbers you can defend, get in touch.