OpenCV 5 inside ROS 2: cv_bridge, zero-copy image transport, and a vision node that holds its deadline

Most of the robotics work that lands on our desk arrives in the same shape. There is a working OpenCV script — detection, pose, a measurement — and there is a ROS 2 robot. The script runs at 60 FPS on a bench. Dropped into the robot's node graph it runs at 12 FPS, the arm reacts a third of a second late, and nobody can say where the time went.

The pipeline is almost never the problem. The plumbing is. This tutorial walks the plumbing: how OpenCV 5 and ROS 2 (Jazzy and the newer Kilted release line) meet, where copies and serialisation get inserted behind your back, and how to end up with a latency number you can put in a statement of work.

1. The three places an image gets copied

Between the sensor and your cv::Mat there are three habitual copies:

1. Driver to DDS. The camera driver publishes sensor_msgs/msg/Image. If the driver and your node are separate processes, the middleware serialises the whole frame. A 1920x1080 BGR8 frame is 6.2 MB; at 30 FPS that is 186 MB/s through the transport for one camera.
2. Message to cv::Mat. cv_bridge::toCvCopy() does exactly what it says.
3. Result back out. Every annotated debug image you publish is another full frame on the wire — and debug topics are the single most common reason a robot that was fine in the lab misses deadlines at a customer site.

You can remove most of copy 1 and all of copy 2, and you can make copy 3 conditional. That is usually the whole performance story.

2. cv_bridge with OpenCV 5: the ABI question first

cv_bridge is compiled against the OpenCV that your ROS 2 distribution was built against. Jazzy binaries on Ubuntu 24.04 link the distro OpenCV 4.x. If you pip install opencv-python for OpenCV 5, or build OpenCV 5 into /usr/local, you now have two OpenCV runtimes in one process and the failure mode is not a clean error — it is a segfault inside cv::Mat's destructor, or a silent corruption of the image header, typically twenty minutes into a demo.

There are only three safe configurations:

• Stay on the distro OpenCV for the ROS node, and keep the OpenCV 5 work in a separate process that talks over a topic or a shared-memory buffer.
• Rebuild cv_bridge, image_transport, image_pipeline and vision_opencv from source against your OpenCV 5 install, in the same colcon workspace, with -DOpenCV_DIR=/opt/opencv5/lib/cmake/opencv5. This is the configuration we use most often on customer robots.
• Skip cv_bridge entirely and wrap the message buffer yourself, which is three lines and removes the dependency:

void on_image(const sensor_msgs::msg::Image::ConstSharedPtr & msg)
{
  // No copy: cv::Mat borrows the message's buffer. Valid only while msg is alive.
  const cv::Mat frame(msg->height, msg->width, CV_8UC3,
                      const_cast<uint8_t *>(msg->data.data()), msg->step);
  process(frame);   // must not store `frame` beyond this scope
}

That borrow is the real win. If you need to keep the frame — a tracker history, an async inference queue — clone it there, deliberately, rather than cloning every frame at the door.

In Python the same rule applies in a weaker form: cv_bridge.imgmsg_to_cv2(msg, desired_encoding='passthrough') avoids a conversion, but numpy.frombuffer(msg.data, ...).reshape(...) avoids the dependency altogether.

3. Intra-process composition: the setting that actually buys the frame rate

ROS 2 can pass a std::shared_ptr between nodes with no serialisation and no copy at all — but only if the publisher and subscriber are composed into the same process, both use use_intra_process_comms(true), and the subscription takes a ConstSharedPtr (taking the message by value or by unique pointer forces a copy).

rclcpp::NodeOptions opts;
opts.use_intra_process_comms(true);
auto camera = std::make_shared<CameraDriverNode>(opts);
auto vision = std::make_shared<VisionNode>(opts);

rclcpp::executors::MultiThreadedExecutor exec;
exec.add_node(camera);
exec.add_node(vision);
exec.spin();

Or, without writing a main, with a component container and a launch file using ComposableNodeContainer and extra_arguments=[{'use_intra_process_comms': True}].

Two caveats we hit on nearly every engagement:

• Intra-process transport bypasses image_transport plugins. Compressed republishing only exists on the inter-process path.
• If any subscriber on the topic is out of process, the message is serialised for that subscriber anyway. One rqt_image_view left running is enough to put the full frame rate back on the wire. Publish debug images on their own topic, and gate them:

if (debug_pub_->get_subscription_count() > 0) {
  debug_pub_->publish(*cv_bridge::CvImage(msg->header, "bgr8", annotated).toImageMsg());
}

For the cross-process case, ROS 2 loaned messages (borrow_loaned_message()) plus a shared-memory-capable RMW such as Iceoryx-backed configurations get you close to zero copy without composition — at the cost of a fixed-size message type, which sensor_msgs/Image is not. In practice, compose.

4. QoS: choose a policy that matches a vision pipeline

Default reliable QoS on a camera topic is usually wrong. A vision node that falls behind should drop frames, not queue them — a detection on a 400 ms old frame is worse than no detection.

auto qos = rclcpp::SensorDataQoS().keep_last(1);   // best effort, depth 1
sub_ = create_subscription<sensor_msgs::msg::Image>(
  "/camera/image_raw", qos,
  std::bind(&VisionNode::on_image, this, std::placeholders::_1));

Then make the timing contract explicit rather than hoping. A Deadline QoS policy on the publisher plus a deadline-missed callback turns "the robot felt laggy" into a logged, countable event:

rclcpp::PublisherOptions pub_opts;
pub_opts.event_callbacks.deadline_callback =
  [this](rclcpp::QOSDeadlineOfferedInfo & e) {
    RCLCPP_WARN(get_logger(), "detection deadline missed, total=%d", e.total_count);
  };
auto det_qos = rclcpp::QoS(1).deadline(std::chrono::milliseconds(50));

Note that deadline, liveliness and lifespan are QoS-compatibility-checked: if the subscriber requests a tighter deadline than the publisher offers, the match silently does not happen. ros2 topic info -v /detections shows you the offered and requested policies when a topic mysteriously has no subscribers.

5. Time, frames and the mistake that costs you accuracy

A vision result is only useful if the rest of the stack knows when and from where it was observed.

• Stamp from the sensor, not from now(). Copy msg->header.stamp straight onto every output message. Restamping with the current time hides your whole latency and makes TF2 lookups wrong by exactly the amount you care about.
• Look up the transform at the image stamp, with a timeout, and handle the exception. This is where tf2::ExtrapolationException turns into a robot that reaches for where the part used to be:

geometry_msgs::msg::TransformStamped tf;
try {
  tf = tf_buffer_->lookupTransform("base_link", msg->header.frame_id,
                                   msg->header.stamp, 50ms);
} catch (const tf2::TransformException & ex) {
  RCLCPP_WARN_THROTTLE(get_logger(), *get_clock(), 1000, "TF: %s", ex.what());
  return;
}

• Use message_filters::sync_policies::ApproximateTime for stereo pairs, RGB-D, or image-plus-CameraInfo, and log the residual time offset it accepted. A synchroniser that quietly pairs frames 30 ms apart on a moving robot is a calibration error you will chase for a week.
• Take intrinsics from CameraInfo, not from a YAML you copied into your node. Undistortion and any PnP or depth maths in OpenCV 5 should read K, D, R, P off the live topic — see our camera calibration walkthrough for how those matrices are produced and what P means after rectification.

6. Where the inference goes

Do not run a 40 ms network inside the subscription callback of a single-threaded executor. It blocks TF, parameters, services and every other callback in that node.

The pattern that has survived contact with real robots for us:

• Subscription callback in a MutuallyExclusiveCallbackGroup does capture, borrow, and a cheap pre-check only; it pushes at most one frame into a slot (overwriting an unprocessed one).
• A worker thread — or a Reentrant callback group on a MultiThreadedExecutor — runs the OpenCV 5 DNN or ONNX Runtime / TensorRT inference.
• Results publish from the worker, carrying the original stamp and frame_id.

On Jetson-class hardware, keep the whole thing in one process so that a GPU buffer never round-trips through a serialised message; our notes on Jetson Orin Nano with OpenCV and TensorRT cover the accelerator side, and profiling OpenCV 5 pipelines covers finding the 8 ms you did not know you were spending.

7. Measure the end-to-end latency, not the node

The number a customer cares about is shutter-to-actuation. Measure it like this:

• Record header.stamp at the driver and the wall clock at the moment the result is published; the difference is pipeline latency including transport.
• Use ros2 topic delay /detections for a quick read, and ros2 run tracetools_trace / LTTng with ros2_tracing when you need per-callback breakdown. ros2 topic hz alone will happily show 30 Hz on a pipeline that is half a second behind.
• Log the 99th percentile, not the mean. Deadline misses live in the tail.

A representative budget for a 1440x1080 mono camera on an Orin NX, everything composed in one process:

Stage	Typical
Exposure + driver to message	8-12 ms
Intra-process handoff	<0.5 ms
Undistort + resize (OpenCV 5, UMat)	2-4 ms
Detector inference (TensorRT FP16)	12-18 ms
Post-process, TF, publish	1-3 ms
Shutter to /detections	25-38 ms

The same graph with three out-of-process nodes, reliable QoS, depth 10, and an rqt_image_view attached measured 120-260 ms with periodic 400 ms spikes. Nothing about the vision code changed.

8. A short checklist before you ship

• cv_bridge and OpenCV come from one build, not two.
• Camera, vision and any consumer are composed; intra-process comms on; subscriptions take ConstSharedPtr.
• Sensor-data QoS, depth 1, on image topics; deadline QoS with a logged miss callback on result topics.
• Debug image publication gated on get_subscription_count().
• Output stamps copied from input; TF looked up at the image stamp with a timeout.
• Intrinsics read from CameraInfo.
• Inference off the executor's critical path.
• p99 shutter-to-result latency recorded in CI on a replayed rosbag, with a threshold that fails the build — the same discipline as our pipeline regression testing approach.

Need this done on your robot?

SentientSight places senior OpenCV and ROS 2 engineers on exactly this kind of integration work — taking a vision algorithm that already works and making it hold a deadline inside a real robot stack, with the latency budget written down and tested. If you have a perception node that is slower in the robot than it was on the bench, get in touch and tell us what the graph looks like.