Every few months a manufacturing or logistics client asks the same question in slightly different words: "can the cameras tell us when someone is in the wrong place, or lifting badly, or reaching into the machine?" Detection alone cannot answer it. A bounding box tells you a person is present; it does not tell you where their hands are, which way they are facing, or whether they crossed a line with a foot or a shadow.
That is what human pose estimation is for. This tutorial builds a full 2D pose pipeline on OpenCV 5 — top-down detection plus keypoints via ONNX, temporal smoothing, homography to floor coordinates, and zone/posture rules — and, just as importantly, sets out how to decide whether the events it emits are trustworthy enough to show an operations manager.
1. Top-down or bottom-up: pick before you write code
There are two families, and the choice drives your whole latency budget.
Top-down runs a person detector, crops each person, and runs a single-person keypoint model per crop. Accuracy is high and stable at small scales, but cost grows linearly with the number of people. RTMPose and the ViTPose family are the usual choices; a 256x192 RTMPose-m crop is roughly 2-4 ms on a modern discrete GPU and 10-20 ms on an edge accelerator.
Bottom-up (OpenPose-style, or the single-pass YOLO-pose heads) predicts all keypoints in one forward pass and groups them. Cost is constant regardless of crowd size, and the modern one-stage pose heads have closed much of the accuracy gap.
Our rule of thumb from deployments:
- Fewer than ~6 people in frame, accuracy matters, keypoints feed measurements: top-down.
- Crowded scenes, or a hard fixed frame budget: one-stage YOLO-pose.
- Never mix: two pose models in one product is two calibration problems, two retraining stories, and two sets of failure modes.
This tutorial uses the one-stage route for the main loop because it keeps the OpenCV 5 DNN code short, then shows the top-down variant where precision matters.
2. Export once, pin the graph
Whatever model you choose, export to ONNX with a fixed input size and static batch, then never regenerate it casually. Pin the opset, the exporter version, and the file hash in your repo.
from ultralytics import YOLO
m = YOLO("yolo11n-pose.pt") # or your fine-tuned checkpoint
m.export(format="onnx", imgsz=640, opset=17, simplify=True, dynamic=False)
Then sanity-check the graph before it ever reaches your pipeline:
import onnx
g = onnx.load("yolo11n-pose.onnx")
print([(i.name, [d.dim_value for d in i.type.tensor_type.shape.dim]) for i in g.graph.input])
print([(o.name, [d.dim_value for d in o.type.tensor_type.shape.dim]) for o in g.graph.output])
You expect 1x3x640x640 in and 1x56xN out for a 17-keypoint COCO pose head: 4 box values, 1 objectness score, then 17 x (x, y, confidence). If those numbers differ, your decoder will silently produce nonsense rather than crash, which is far worse.
Two migration notes for anyone coming from OpenCV 4.x, covered more fully in our 4.x to 5.0 migration checklist:
- OpenCV 5 ships a rewritten DNN engine with much better ONNX coverage. If your old workaround was "re-export with opset 11 because OpenCV chokes on the newer ops", retry the modern export first — most of those workarounds are now dead weight.
blobFromImagestill exists and still behaves the same way. The trap is unchanged too: it does not letterbox. Feed it a raw 16:9 frame withsize=(640,640)and every keypoint you compute is wrong in a way that looks almost right.
3. The inference loop
import cv2
import numpy as np
NET_SIZE = 640
KPTS = 17
net = cv2.dnn.readNetFromONNX("yolo11n-pose.onnx")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)
def letterbox(frame, size=NET_SIZE):
h, w = frame.shape[:2]
s = min(size / h, size / w)
nh, nw = int(round(h * s)), int(round(w * s))
resized = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_LINEAR)
canvas = np.full((size, size, 3), 114, dtype=np.uint8)
top, left = (size - nh) // 2, (size - nw) // 2
canvas[top:top + nh, left:left + nw] = resized
return canvas, s, left, top
def infer(frame, conf_thr=0.35, nms_thr=0.55):
canvas, s, dx, dy = letterbox(frame)
blob = cv2.dnn.blobFromImage(canvas, 1 / 255.0, (NET_SIZE, NET_SIZE), swapRB=True, crop=False)
net.setInput(blob)
out = net.forward()[0].T # (N, 56)
scores = out[:, 4]
keep = scores > conf_thr
out, scores = out[keep], scores[keep]
if len(out) == 0:
return []
cx, cy, bw, bh = out[:, 0], out[:, 1], out[:, 2], out[:, 3]
boxes = np.stack([cx - bw / 2, cy - bh / 2, bw, bh], axis=1)
idx = cv2.dnn.NMSBoxes(boxes.tolist(), scores.tolist(), conf_thr, nms_thr)
people = []
for i in np.array(idx).flatten():
k = out[i, 5:].reshape(KPTS, 3).copy()
k[:, 0] = (k[:, 0] - dx) / s # undo letterbox, not just the resize
k[:, 1] = (k[:, 1] - dy) / s
b = boxes[i].copy()
b[0] = (b[0] - dx) / s; b[1] = (b[1] - dy) / s
b[2] /= s; b[3] /= s
people.append({"box": b, "score": float(scores[i]), "kpts": k})
return people
The undo-the-letterbox step is where most home-grown pose pipelines quietly break. Draw the skeleton over the original frame on day one and keep that debug view for the life of the project.
4. Keypoint confidence is not a decoration
Each keypoint carries its own confidence. Treat anything below ~0.4 as missing, not as a coordinate. Occluded wrists, ankles behind a pallet, and heads cropped by the frame edge all come back with plausible-looking coordinates and low scores — and if you average them into an angle, you get a confident wrong answer.
KP_THR = 0.4
def valid(kpts, *names, index=None):
return all(kpts[index[n], 2] >= KP_THR for n in names)
Two follow-on habits:
- Every derived quantity (a joint angle, a reach distance, a stride) should declare which keypoints it needs, and return
Nonewhen any of them are missing. Never substitute a default. - Log the fraction of frames a rule was computable, alongside the events. A rule that fires three times a shift but was only evaluable in 40% of frames is not a measurement; it is a rumour.
5. Smooth in time, or drown in flicker
Raw per-frame keypoints jitter by several pixels even on a static subject. Feed that into an angle threshold and you get hundreds of spurious events per hour. A One-Euro filter is the right tool: it adapts its cutoff to speed, so slow motion is smoothed hard and fast motion stays responsive — unlike a fixed moving average, which adds lag exactly when you care about accuracy.
class OneEuro:
def __init__(self, freq=25.0, min_cutoff=1.0, beta=0.3, d_cutoff=1.0):
self.freq, self.min_cutoff, self.beta, self.d_cutoff = freq, min_cutoff, beta, d_cutoff
self.x_prev = None
self.dx_prev = 0.0
@staticmethod
def _alpha(cutoff, freq):
tau = 1.0 / (2 * np.pi * cutoff)
te = 1.0 / freq
return 1.0 / (1.0 + tau / te)
def __call__(self, x):
if self.x_prev is None:
self.x_prev = x
return x
dx = (x - self.x_prev) * self.freq
a_d = self._alpha(self.d_cutoff, self.freq)
self.dx_prev = a_d * dx + (1 - a_d) * self.dx_prev
cutoff = self.min_cutoff + self.beta * abs(self.dx_prev)
a = self._alpha(cutoff, self.freq)
x_hat = a * x + (1 - a) * self.x_prev
self.x_prev = x_hat
return x_hat
Run one filter per (track id, keypoint, axis). That means you need identity across frames: attach a tracker to the person boxes — the ByteTrack-style association described in our multi-object tracking and counting tutorial works unchanged here — and key your filters and rule state off the track id, never off a per-frame index.
6. Pixels to floor metres: the homography that makes zones honest
A "zone" drawn as a polygon in image coordinates is a polygon in a projective distortion of the floor. It behaves differently near and far, and a person's bounding box overlaps it long before their feet do. Fix it once with a ground-plane homography.
Measure four points on the floor that you can identify in the image — pillar bases, painted line corners, tape crosses on a tape-measured grid — and solve:
img_pts = np.array([[412, 690], [1290, 664], [1521, 953], [188, 998]], dtype=np.float32)
flr_pts = np.array([[0.0, 0.0], [6.0, 0.0], [6.0, 4.0], [0.0, 4.0]], dtype=np.float32) # metres
H, _ = cv2.findHomography(img_pts, flr_pts, cv2.RANSAC, 3.0)
def to_floor(pt):
p = np.array([[[pt[0], pt[1]]]], dtype=np.float32)
return cv2.perspectiveTransform(p, H)[0][0]
Then define the person's ground position from the ankle midpoint (falling back to the bottom-centre of the box only when both ankles are low-confidence), transform it, and test containment in floor-space polygons with cv2.pointPolygonTest. Undistort first if the lens is wide — see our camera calibration tutorial — because a homography fitted on distorted pixels absorbs the distortion into the plane fit and drifts at the edges.
Validate the homography before trusting it: walk a tape-measured path through the scene and check that the reconstructed floor coordinates match to within your tolerance. On typical 4-6 m installations we accept about ±0.15 m; if you cannot hit that, your points are too clustered or the lens is uncorrected.
7. Turning keypoints into events people act on
Now the rules. Keep them small, explicit, and hysteretic.
def angle(a, b, c):
"""Interior angle at b, degrees."""
v1, v2 = a[:2] - b[:2], c[:2] - b[:2]
cosang = np.dot(v1, v2) / (np.linalg.norm(v1) * np.linalg.norm(v2) + 1e-9)
return float(np.degrees(np.arccos(np.clip(cosang, -1.0, 1.0))))
Three rules that cover most of what clients actually ask for:
- Zone intrusion. Ankle midpoint inside a floor polygon for N consecutive frames.
- Reach-in. A wrist keypoint inside a machine-aperture polygon while the torso stays outside it — the case a bounding box can never distinguish.
- Bend/lift posture. Hip-shoulder-knee angle below a threshold while the wrists are below hip height, sustained for over a second.
Every rule needs three parameters, not one: an enter threshold, an exit threshold (lower, so a person hovering on the boundary does not chatter), and a minimum duration. A useful default is enter at 8 consecutive frames, exit after 15 frames clear, at 25 fps. Emit one event per entry with a start time, end time, track id, and the peak measured value — not one event per frame.
And a discipline point: an event is a cue for a human, not a verdict. Systems that count as evidence against individuals invite a fight about the 3% of frames where the keypoints were wrong. Systems that surface aggregate heatmaps and near-miss counts get adopted. This is also where governance lands — if your deployment falls under the EU AI Act's workplace provisions, keypoint analytics on identifiable staff needs a documented purpose, retention limit, and worker consultation before a single camera goes up. The redaction pipeline we published pairs naturally with this: blur faces on any clip that leaves the site.
8. Evaluating it honestly
Two numbers matter, and they are not the model's published mAP on COCO.
Keypoint accuracy on your scene. Label 200-300 frames sampled across shifts, camera angles, and clothing, and compute OKS-based PCK per keypoint. You will typically find head and shoulders near-perfect and ankles the weak point — which matters enormously, because ankles are what your floor projection depends on.
Event-level precision and recall. Take four hours of real footage, have someone mark true events, and score your pipeline's emitted events with a ±2 s matching window. Report both numbers with the thresholds that produced them. On a safety-cue system, we usually tune for high recall and accept precision around 0.7-0.8, then reduce alarm fatigue by aggregating rather than by raising thresholds until the recall quietly collapses.
Re-run both evaluations after every model, threshold, or camera change. Store the clips. That regression set is the single most valuable artefact of the project.
9. Frame budget on real hardware
A rough per-camera budget at 1080p/25 fps on an Orin-class device with the one-stage model in FP16:
| Stage | Typical cost |
|---|---|
| Hardware decode | 2-4 ms |
| Letterbox + blob | 1-2 ms |
| Pose inference (640) | 12-20 ms |
| Decode + NMS | 1-3 ms |
| Tracking + filtering | < 1 ms |
| Rules + event I/O | < 1 ms |
That is one camera near saturation. For four or eight cameras you batch crops (top-down) or drop to 10-12 fps of inference with tracking carrying the identities between inferred frames — pose is far more forgiving of frame skipping than tracking is. If those numbers do not add up on your hardware, our pipeline profiling tutorial shows how to find the real bottleneck before buying more silicon.
10. The short version
- Pick top-down or one-stage on people-count and frame budget, and commit.
- Letterbox correctly and undo it correctly; verify by drawing the skeleton on the source frame.
- Treat per-keypoint confidence as a gate; return
Nonerather than guessing. - Smooth with One-Euro per track id, not a global moving average.
- Do zone logic on the floor plane via a validated homography, from ankles — not on boxes in image space.
- Give every rule enter/exit thresholds and a minimum duration, and emit intervals, not frames.
- Measure event precision and recall on your own footage, and keep the clips.
Pose analytics is one of those problems where the model is the easy part and the geometry, thresholds, and governance are the project. If you are scoping a workplace-safety, ergonomics, or process-timing vision system and want senior OpenCV engineers who have shipped one, get in touch — we do short scoping engagements that end with a measured baseline on your footage and a build plan you can cost.