Detection is the part clients ask for. Tracking is the part that decides whether the system works.
Almost every "count the people/vehicles/parcels" project we get called into arrives with a working detector and a broken number. The detector is 94% mAP and the daily count is 30% high. The cause is never the network — it is the association step between frames, the geometry of the counting line, and a video ingest layer that quietly drops frames when the network hiccups.
This tutorial builds a production-shaped tracking-and-counting pipeline with OpenCV 5: robust RTSP ingest, a ByteTrack-style association loop written in plain NumPy, line and zone counting that survives loitering, and the evaluation metrics that let you defend the number.
1. Detection is not counting
The naive pipeline — detect per frame, count boxes that cross a line — fails for three separate reasons:
- A person is not an event. One person crossing a line produces detections on 15 consecutive frames. Without identity you count 15 crossings, or you deduplicate with a timeout and lose two people walking together.
- Detectors flicker. Even a good model drops a box for one or two frames under occlusion or motion blur. Every drop becomes a new "object" if you have no memory.
- The camera sees pixels, not the floor. A line drawn across the image is a line in the image plane. A tall person's box crosses it several frames before their feet do.
Tracking fixes the first two. Geometry — see section 5 — fixes the third.
2. Ingest first: the frames you never see
Before any tracking code, fix the source. cv2.VideoCapture("rtsp://...") with default settings buffers frames internally; when your loop is slower than the stream, you process increasingly stale frames until the buffer blows and OpenCV silently drops a burst. Tracking on a discontinuous stream produces ID switches that look like a tracker bug and are not.
Use a GStreamer pipeline with hardware decode and a leaky, depth-1 queue, so the decoder drops old frames deterministically and you always get the latest one:
import cv2
PIPELINE = (
"rtspsrc location={url} latency=100 protocols=tcp ! "
"rtph264depay ! h264parse ! avdec_h264 ! " # or nvv4l2decoder on Jetson
"videoconvert ! video/x-raw,format=BGR ! "
"appsink max-buffers=1 drop=true sync=false"
)
cap = cv2.VideoCapture(PIPELINE.format(url=RTSP_URL), cv2.CAP_GSTREAMER)
if not cap.isOpened():
raise RuntimeError("GStreamer pipeline failed to open — check plugins with gst-inspect-1.0")
Two rules worth writing into your runbook:
protocols=tcpfor anything over WiFi or a shared LAN. UDP RTSP loses packets, and lost packets become smeared macroblocks that wreck detection quality in ways no metric on your validation set will predict.- Reconnect, don't crash. IP cameras reboot, PoE switches renegotiate, DHCP leases expire. Wrap capture in a supervisor that rebuilds the pipeline on N consecutive failed reads, and reset the tracker state when it does — carrying stale tracks across a 40-second gap is worse than starting clean.
consecutive_failures = 0
while True:
ok, frame = cap.read()
if not ok:
consecutive_failures += 1
if consecutive_failures > 30:
cap.release(); cap = open_capture(RTSP_URL); tracker.reset()
consecutive_failures = 0
continue
consecutive_failures = 0
process(frame)
Also log a real FPS counter and the wall-clock timestamp of each processed frame. If you are counting events per hour, you need to know that you processed 12 FPS of a 25 FPS stream — because your effective sampling rate is exactly what determines how fast an object can move before it is missed entirely.
3. What the OpenCV trackers are, and are not
OpenCV ships single-object trackers in the tracking module — KCF, CSRT, MIL, and the DNN-based Nano/ViT trackers. In OpenCV 5 these live in opencv_contrib's video/tracking surface (cv2.TrackerCSRT_create, cv2.TrackerNano_create, cv2.TrackerVit_create).
They solve a different problem than multi-object tracking:
| Need | Use |
|---|---|
| User draws one box, follow it | TrackerCSRT (accurate, ~25 FPS/object) or TrackerNano/TrackerVit |
| Follow N detected objects, assign stable IDs | Detector + association tracker (SORT/ByteTrack family) |
| Bridge 1–2 frame detector dropouts on one object | TrackerKCF as a short-lived filler |
Running one CSRT instance per object in a 40-person scene does not scale, and it has no mechanism for births, deaths or ID management. For multi-object work, use detection-based tracking. The good news is that the strongest current approach — ByteTrack — is about 120 lines of NumPy and needs no extra dependency.
4. A ByteTrack-style tracker in plain NumPy
ByteTrack's insight is simple and worth internalising: do not throw away low-confidence detections. Occluded objects produce low-confidence boxes. Standard trackers threshold at 0.5, discard those boxes, and lose the track. ByteTrack matches high-confidence detections first, then does a second association pass using the leftovers against still-unmatched tracks.
Start with IoU and a constant-velocity motion model:
import numpy as np
from scipy.optimize import linear_sum_assignment
def iou_matrix(a, b):
"""a: (N,4), b: (M,4) in xyxy. Returns (N,M) IoU."""
if len(a) == 0 or len(b) == 0:
return np.zeros((len(a), len(b)), dtype=np.float32)
x1 = np.maximum(a[:, None, 0], b[None, :, 0])
y1 = np.maximum(a[:, None, 1], b[None, :, 1])
x2 = np.minimum(a[:, None, 2], b[None, :, 2])
y2 = np.minimum(a[:, None, 3], b[None, :, 3])
inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
area_a = (a[:, 2] - a[:, 0]) * (a[:, 3] - a[:, 1])
area_b = (b[:, 2] - b[:, 0]) * (b[:, 3] - b[:, 1])
return (inter / (area_a[:, None] + area_b[None, :] - inter + 1e-9)).astype(np.float32)
class Track:
_next_id = 1
def __init__(self, box, score):
self.id = Track._next_id; Track._next_id += 1
self.box = np.asarray(box, dtype=np.float32)
self.velocity = np.zeros(2, dtype=np.float32)
self.score = score
self.age = 0 # frames since birth
self.hits = 1 # successful associations
self.time_since_update = 0
self.history = [] # centroid trail, for line crossing
@property
def centroid(self):
x1, y1, x2, y2 = self.box
return np.array([(x1 + x2) / 2, (y1 + y2) / 2], dtype=np.float32)
def predict(self):
self.box[[0, 2]] += self.velocity[0]
self.box[[1, 3]] += self.velocity[1]
self.age += 1
self.time_since_update += 1
return self.box
def update(self, box, score, alpha=0.6):
new = np.asarray(box, dtype=np.float32)
old_c = self.centroid
self.box = alpha * new + (1 - alpha) * self.box
self.velocity = 0.7 * self.velocity + 0.3 * (self.centroid - old_c)
self.score = score
self.hits += 1
self.time_since_update = 0
self.history.append(self.centroid.copy())
if len(self.history) > 60:
self.history.pop(0)
The association loop:
class ByteTracker:
def __init__(self, high_thresh=0.5, low_thresh=0.1,
match_thresh=0.2, max_age=30, min_hits=3):
self.high_thresh, self.low_thresh = high_thresh, low_thresh
self.match_thresh, self.max_age, self.min_hits = match_thresh, max_age, min_hits
self.tracks = []
def reset(self):
self.tracks = []
def _associate(self, tracks, dets, thresh):
if not tracks or len(dets) == 0:
return [], list(range(len(tracks))), list(range(len(dets)))
ious = iou_matrix(np.stack([t.box for t in tracks]), dets)
rows, cols = linear_sum_assignment(-ious)
matches = [(r, c) for r, c in zip(rows, cols) if ious[r, c] >= thresh]
m_r = {r for r, _ in matches}; m_c = {c for _, c in matches}
return (matches,
[i for i in range(len(tracks)) if i not in m_r],
[j for j in range(len(dets)) if j not in m_c])
def update(self, boxes, scores):
boxes = np.asarray(boxes, dtype=np.float32).reshape(-1, 4)
scores = np.asarray(scores, dtype=np.float32).reshape(-1)
for t in self.tracks:
t.predict()
high = scores >= self.high_thresh
low = (scores >= self.low_thresh) & ~high
# Pass 1: confident detections against all tracks
matches, un_tracks, un_dets = self._associate(
self.tracks, boxes[high], self.match_thresh)
hi_boxes, hi_scores = boxes[high], scores[high]
for r, c in matches:
self.tracks[r].update(hi_boxes[c], hi_scores[c])
# Pass 2: low-confidence detections against what's left (occlusion recovery)
leftover = [self.tracks[i] for i in un_tracks]
lo_boxes, lo_scores = boxes[low], scores[low]
matches2, un_tracks2, _ = self._associate(leftover, lo_boxes, 0.5)
for r, c in matches2:
leftover[r].update(lo_boxes[c], lo_scores[c])
# Births from unmatched confident detections only
for j in un_dets:
self.tracks.append(Track(hi_boxes[j], hi_scores[j]))
# Deaths
self.tracks = [t for t in self.tracks if t.time_since_update <= self.max_age]
return [t for t in self.tracks
if t.hits >= self.min_hits and t.time_since_update == 0]
Note the second pass uses a stricter IoU threshold (0.5). Low-confidence boxes are noisy; only accept them when the spatial evidence is strong.
Tuning notes that matter more than the algorithm:
max_ageis a time, not a frame count. 30 frames is 1 second at 30 FPS and 2.5 seconds at 12 FPS. Set it from seconds and your measured FPS, or your tracker behaves differently every time the hardware load changes.min_hitstrades false tracks for late starts. Withmin_hits=3an object is invisible for its first three frames — if your counting line is near the frame edge, objects can cross before they exist. Either lower it or move the line.- IoU alone fails on fast motion. If an object moves more than its own width between frames, IoU is zero and association fails. Raise the frame rate, or add appearance features (below).
5. Counting: geometry, not thresholds
With stable IDs, counting becomes a crossing test on the track's own history. Use the signed side of the line and count on a sign change, so an object loitering on the line does not oscillate the count:
def side(p, a, b):
return np.sign((b[0] - a[0]) * (p[1] - a[1]) - (b[1] - a[1]) * (p[0] - a[0]))
LINE_A, LINE_B = np.array([100, 600]), np.array([1180, 600])
counts = {"in": 0, "out": 0}
last_side = {}
for t in active_tracks:
p = foot_point(t.box) # see below
s = side(p, LINE_A, LINE_B)
prev = last_side.get(t.id)
if prev is not None and s != 0 and prev != 0 and s != prev:
counts["in" if s > 0 else "out"] += 1
if s != 0:
last_side[t.id] = s
def foot_point(box):
"""Bottom-centre of the box ~= where the object touches the ground."""
x1, y1, x2, y2 = box
return np.array([(x1 + x2) / 2.0, y2], dtype=np.float32)
Three details that separate a demo from a deployable counter:
- Count the foot point, not the centroid. For people and vehicles the bottom-centre of the box approximates ground contact. Using the centroid makes tall objects cross early and short objects late, and the bias changes with distance from the camera.
- Require a minimum track length before counting. A track with 3 hits crossing a line is likely a detector artefact. Gate on
t.hits >= 8for anything you invoice against. - Prefer a zone over a line where you can. Two parallel lines (an entry band) or a polygon with
cv2.pointPolygonTestgives direction robustly and tolerates jitter far better than a single line at an oblique angle to travel.
If the camera is fixed and you have a floor plane, take one more step: homography. Calibrate four ground points with cv2.findHomography, project foot points into floor coordinates, and do all counting, speed and dwell computation in metres. Then a "loitering longer than 30 seconds within 2 m of the door" rule is a real spatial rule instead of a pixel heuristic that breaks when someone nudges the camera.
6. When IoU is not enough: appearance and re-identification
IoU-based association breaks down in three situations: crowded scenes with heavy occlusion, long occlusions (someone walks behind a pillar for two seconds), and multi-camera hand-off. The fix is an appearance embedding — a small ReID network producing a 128–512-D vector per crop, matched by cosine distance and fused with IoU:
cost = 0.7 * (1.0 - iou) + 0.3 * cosine_distance(track_features, det_features)
OpenCV's DNN module runs the ReID model fine; the cost is throughput, since you now infer once per detection per frame. Practical mitigations: batch all crops from a frame into one forward pass, only extract features for tracks that are unmatched or recently born, and keep a small gallery (say the last 10 embeddings) per track with a running average rather than a single snapshot.
Be candid with clients about what ReID can and cannot do. Same-camera re-association across a few seconds is reliable. Cross-camera identification with different lighting, angles and colour balance is a research-grade problem that degrades sharply in the field — and in many jurisdictions it also changes the privacy posture of the whole system, which is a legal conversation, not a technical one.
7. Measuring the thing you are actually selling
Detector mAP tells you almost nothing about count accuracy. Track the metrics that map to the deliverable:
- HOTA — the current standard for MOT, balancing detection and association quality in one number. Use TrackEval on annotated clips.
- IDF1 — identity-preserving F1; sensitive to exactly the ID switches that corrupt counts.
- ID switches per minute — the metric that most directly predicts over-counting.
- Count error against a human ground truth. This is the number the client cares about. Take three clips — quiet, busy, adversarial (a group entering together, someone stopping on the line) — count them by hand, and report percentage error per clip.
Report the adversarial clip. Systems fail at the tails, and a client who saw the failure mode in week two accepts it as a known limitation; a client who discovers it in month five treats it as a defect.
8. Throughput: detect less often than you track
Detection dominates the compute budget; association is nearly free. So do not detect every frame:
DETECT_EVERY = 3
if frame_idx % DETECT_EVERY == 0:
boxes, scores = detector(frame)
tracks = tracker.update(boxes, scores)
else:
tracks = [t for t in tracker.tracks if t.time_since_update == 0]
for t in tracks:
t.predict() # motion model carries IDs between detections
At 25 FPS with DETECT_EVERY = 3 you run detection ~8 times a second, which is enough for walking-speed targets and roughly triples throughput. Validate the interval against your fastest expected object: if a target crosses the counting zone in under DETECT_EVERY frames, you will miss it. Vehicles at 50 km/h need every frame; people in a lobby do not.
Other wins, in rough order of value: use FP16 or INT8 inference (see our Jetson Orin Nano OpenCV + TensorRT guide); resize once at the decoder rather than per-frame in NumPy; keep the pipeline in a fixed colour format end-to-end to avoid repeated cvtColor; and run capture, inference and counting in separate threads with bounded queues so a slow disk write never stalls the decoder.
Where this fits
Tracking is the layer where a computer vision demo turns into a system whose output someone will act on — a footfall report, an SLA credit, a safety alert. It is also the layer with the least publicly available engineering guidance, because it lives between the model and the business rule, and both sides assume the other owns it.
SentientSight's OpenCV consultants build and audit tracking, counting and video-analytics pipelines — ingest reliability, association tuning, ground-plane geometry, and the evaluation harness that proves the number is right. If you have a detector that works and a count that does not, get in touch.