Most of the vision work we are asked to quote in 2026 that involves a robot arm is not a 2D problem pretending to be 3D. The customer has a bin of parts, a CAD model of each part, and a cycle time. They want a 6-DoF pose per pick, a confidence number, and a tolerance they can hand to the integrator.
OpenCV 4.x made this awkward: point clouds lived half in contrib, half in whatever library you bolted on. OpenCV 5 reorganised geometry into the new 3d module and brought point-cloud primitives — voxel grids, plane and sphere fitting via a modern RANSAC, and ICP — into the main build. This tutorial walks the whole chain: capture, clean, segment, register, verify, and quote.
It assumes you already have intrinsics and a hand-eye transform. If you do not, start with camera calibration in OpenCV 5 and hand-eye calibration and 6-DoF pick poses; every number below inherits their error.
1. Where 3D lives in OpenCV 5
#include <opencv2/3d.hpp> // voxelGrid, SACSegmentation-style fitting, ICP, RGBD utilities
#include <opencv2/calib.hpp> // calibrateCamera, stereoCalibrate
In Python the entry points sit on the flat namespace (cv2.ppf_match_3d, cv2.voxelGrid-family helpers and the RGBD/ODOMETRY types depending on build flags). Two practical warnings:
- Build flags matter. A pip wheel is not guaranteed to carry every 3d entry point. Check with
print([n for n in dir(cv2) if 'oxel' in n or 'ICP' in n or 'Plane' in n])before you design around a function. - OpenCV's point-cloud surface is deliberately smaller than a dedicated library's. It is excellent for the capture-to-pose path described here. If you need surface reconstruction, mesh booleans or large-scale SLAM maps, pair it with Open3D or PCL rather than reimplementing them.
2. Getting a cloud that is worth processing
Three capture routes dominate industrial bin picking:
- Structured light (projector + one or two cameras). Best accuracy on matte parts, slowest, hates ambient sunlight and specular metal.
- Active stereo (stereo pair + random IR pattern projector). The pragmatic default: pattern gives texture to blank surfaces, and it degrades gracefully. You reconstruct with the same
stereoRectify/StereoSGBM/reprojectImageTo3Dpath as passive stereo. - ToF / LiDAR-on-arm. Fast and robust to texture, but sparse and prone to multipath error in a deep metal bin.
From a rectified disparity map, the cloud is one call:
disp = matcher.compute(rect_l, rect_r).astype(np.float32) / 16.0
xyz = cv2.reprojectImageTo3D(disp, Q) # metres, camera frame
mask = (disp > matcher.getMinDisparity()) & np.isfinite(xyz).all(axis=2)
cloud = xyz[mask].reshape(-1, 3)
colors = rect_l[mask].reshape(-1, 3)
Two things sink more projects at this step than any algorithm choice. First, exposure: a saturated highlight on a machined face is a hole in the cloud, and a hole in the cloud is a missed pick. Bracket the exposure or run the input-quality gate from our camera control and HDR tutorial. Second, baseline versus bin depth: depth error grows with the square of range and inversely with baseline, so a 60 mm baseline sensor hovering 1.2 m above a bin is physically incapable of the 0.5 mm your customer assumed. Do that arithmetic before the kick-off call.
3. Clean and downsample before you do anything clever
A 2 MP active-stereo frame yields 1–2 million points. ICP on the raw cloud is both slow and worse, because dense flat regions dominate the cost function.
# 1. Crop to the known bin volume (camera frame, metres)
m = ((cloud[:,2] > 0.45) & (cloud[:,2] < 1.10) &
(np.abs(cloud[:,0]) < 0.35) & (np.abs(cloud[:,1]) < 0.25))
cloud = cloud[m]
# 2. Voxel downsample to a uniform density
cloud = voxel_downsample(cloud, leaf=0.002) # 2 mm leaf
# 3. Statistical outlier removal: drop points whose mean k-NN
# distance exceeds mu + 2*sigma (flying pixels at depth edges)
cloud = remove_statistical_outliers(cloud, k=20, std_ratio=2.0)
Pick the voxel leaf from the feature you must resolve, not from a benchmark: if the smallest discriminating feature on the part is a 6 mm boss, a 2 mm leaf keeps it and a 5 mm leaf erases it. Log the point count after each stage — a sudden drop in production is the earliest warning that the projector is failing or the bin has moved.
4. Remove the bin, keep the parts
Plane segmentation finds the bin floor and, run iteratively, the four walls:
remaining = cloud
for _ in range(5):
plane, inliers = fit_plane_ransac(remaining, dist_thresh=0.003, iters=1000)
if inliers.sum() < 0.05 * len(remaining):
break
remaining = remaining[~inliers]
parts = euclidean_cluster(remaining, tol=0.01, min_pts=300)
Set the RANSAC distance threshold from measured sensor noise, not from a round number: capture 50 frames of the empty bin, fit one plane, and take the standard deviation of the residuals. If that sigma is 1.4 mm, a 3 mm threshold is right; if it is 4 mm, your sensor cannot support the tolerance being asked for and the honest answer is a different sensor.
Euclidean clustering gives candidate part blobs. In a full bin, touching parts merge into one cluster — expected, and the reason the next step matches a model rather than trusting the cluster.
5. Coarse match, then ICP
ICP is a local optimiser. Hand it a bad initial guess and it converges confidently to the wrong pose, which is worse than failing. The reliable recipe is coarse-then-fine:
Coarse. Use PPF (point-pair features, cv2.ppf_match_3d) trained once on the CAD model, sampled to the same density as the scene cloud. It returns ranked pose hypotheses without any initial guess.
detector = cv2.ppf_match_3d_PPF3DDetector(0.025, 0.05)
detector.trainModel(model_pc) # Nx6: xyz + normals
poses = detector.match(scene_pc, 1.0/40.0, 0.05)
Fine. Refine the top hypotheses with point-to-plane ICP, which converges in far fewer iterations than point-to-point on planar industrial surfaces:
icp = cv2.ppf_match_3d_ICP(iterations=100, tolerance=0.0005,
rejectionScale=2.5, numLevels=6)
retval, residual, pose = icp.registerModelToScene(model_pc, scene_pc)
Four guards that turn a demo into a production cell:
- Reject on residual. Set a hard RMS ceiling (for example 1.5x the plane-fit sigma from step 4). Above it, no pick.
- Reject on coverage. Count model points with a scene correspondence inside the threshold. A pose matching 20% of the model is a symmetry artefact, not a part.
- Handle symmetry explicitly. Rotationally symmetric parts have infinitely many valid poses; collapse them to a canonical one, or the gripper will chase a yaw that does not matter.
- Verify in 2D. Project the posed CAD silhouette back into the left image and compare against the segmentation mask. It is cheap and catches the failures ICP scores as good.
6. From pose to a pick the robot will accept
The registered pose is in the camera frame. The robot wants base frame:
T_base_part = T_base_gripper · T_gripper_cam · T_cam_part
T_gripper_cam is your hand-eye result. Then apply the grasp offset defined on the CAD model, check reachability and collision against the bin walls, and rank the candidates by accessibility (topmost, most clearance) rather than by ICP score alone.
7. The error budget you actually quote
Quote this as a stack, and measure each term instead of guessing it:
| Term | Typical | How to measure |
|---|---|---|
| Sensor depth noise at working distance | 0.3–2 mm | sigma of residuals on a flat reference plate |
| Intrinsic/stereo calibration | 0.1–0.5 mm | reprojection error converted to depth at range |
| Hand-eye transform | 0.3–1.5 mm | move to the same taught point from 10 poses |
| ICP registration residual | 0.2–1 mm | reported RMS, per-pick |
| Robot repeatability | 0.02–0.1 mm | from the datasheet |
Sum in quadrature, then validate end-to-end: place a part at a known fixtured position, run 100 picks, and report the actual spread. That measured distribution — not the sum of brochure numbers — is what belongs in the acceptance test. If the customer needs 0.5 mm and the stack says 2.1 mm, you have discovered it during scoping instead of during install.
8. Cycle time
Budget per pick: capture 40–150 ms (structured light is at the top of that range), cleaning and segmentation 10–30 ms, PPF coarse match 50–400 ms depending on model sampling, ICP refine 10–50 ms per hypothesis. Two easy wins: run the capture for pick n+1 while the arm executes pick n, and re-use the previous pose as the ICP seed when the bin has not been disturbed, which skips PPF entirely.
9. Ship it
- Version the CAD-derived model cloud and its sampling density alongside the code; a re-exported STL silently changes your match rate.
- Record the raw cloud for every rejected pick. It is the only way to debug a cell that fails twice a shift.
- Add the golden-frame regression harness from our pipeline testing tutorial so a sensor firmware update cannot quietly change your poses.
- Re-run hand-eye after any collision. Crashes move cameras.
Need a bin-picking or 3D inspection cell scoped, prototyped, or rescued? Our OpenCV consultants have shipped these pipelines in production. Get in touch.