Computer vision in the browser: OpenCV.js, WebAssembly SIMD, and ONNX Runtime Web

Every deployment target we cover on this blog so far puts the pixels somewhere you control: a Jetson on the factory floor, an Android handset, a Linux box next to the camera. There is a fourth target that clients keep asking us about and that almost nobody documents properly — the browser tab.

The ask is usually one of these:

• "Our users upload photos of documents/parts/skin/receipts. Legal will not let us send them to a server."
• "We want a live camera preview that guides the user to a good capture before we spend money on a cloud inference call."
• "We need a demo that works from a URL with no install, on a locked-down corporate laptop."

All three are solvable today with OpenCV.js compiled to WebAssembly, plus a small ONNX model running in ONNX Runtime Web. This tutorial builds that pipeline end to end: build or select the right OpenCV.js artefact, get frames off the camera without tanking the frame rate, do the classical CV in WASM, run a network on WebGPU, and keep the main thread responsive. It also covers the parts that go wrong in production — SIMD and thread gating, memory leaks, iOS Safari quirks, and how to decide when not to run in the browser at all.

1. What actually runs in the browser in 2026

Three separate runtimes are involved, and confusing them is the most common source of wasted weeks:

Layer	What it does	Technology
OpenCV.js	cv.Mat, colour conversion, warps, thresholding, contours, ArUco, feature matching	C++ compiled to WebAssembly via Emscripten
ONNX Runtime Web	Neural network inference	WASM (CPU) or WebGPU (GPU) backend
Browser media stack	Camera access, frame delivery, hardware decode	getUserMedia, VideoFrame, WebCodecs

OpenCV's own DNN module is compiled into some OpenCV.js builds, but you should not use it for anything real in the browser. It is single-backend, it does not touch the GPU, and the OpenCV 5 DNN engine's strength — the OpenVINO/CUDA backends we benchmarked in Running YOLO26 in OpenCV 5's new DNN engine — simply is not available in WASM. Use OpenCV.js for the geometry and pre/post-processing it is excellent at, and hand the tensor to ONNX Runtime Web.

The practical performance envelope, measured on a mid-range 2023 laptop with an integrated GPU:

• Classical pipelines (undistort → threshold → contours → perspective warp) on 1280×720: 2–8 ms per frame with SIMD enabled. Comfortably real time.
• A 320×320 detector (YOLO-class, ~4M params, INT8) on the WASM CPU backend: 60–160 ms. Usable for capture assistance at 6–10 fps, not for 30 fps tracking.
• The same model on the WebGPU backend: 8–25 ms. Real time, when WebGPU is available.
• Anything above ~640×640 with a >20M-parameter backbone: treat the browser as out of scope and go server-side.

Those numbers move by a factor of five between a desktop Chrome and a three-year-old iPhone on low-power mode. Budget accordingly, and always measure on the worst device in your user base, not on the developer's machine.

2. Getting the right OpenCV.js build

The prebuilt opencv.js on the OpenCV site is a general-purpose artefact: every module, no threading, and often no SIMD. It is typically 8–11 MB uncompressed. That is a bad first impression on a mobile connection and it leaves 2–4× of CPU performance on the table.

For anything beyond a prototype, build your own. The Emscripten build script accepts module filters:

# Emscripten SDK 3.1.x+ must be active in the shell
git clone --depth 1 --branch 5.x https://github.com/opencv/opencv.git
cd opencv

python3 ./platforms/js/build_js.py build_wasm \n  --build_wasm \n  --simd \n  --threads \n  --disable_single_file \n  --cmake_option="-DBUILD_LIST=core,imgproc,imgcodecs,calib,3d,objdetect,features2d,video"

Notes that matter:

• --simd emits WASM SIMD128. Supported by every current browser; this is the single biggest performance lever and is worth a separate build.
• --threads uses SharedArrayBuffer and pthreads. It is fast, but it requires cross-origin isolation (see §6) and it will silently fail to help if your headers are wrong.
• BUILD_LIST is where the size goes. Dropping dnn, photo, stitching, ml and the contrib modules typically takes a 10 MB artefact to 2.5–4 MB, which gzips to well under 1.5 MB.
• Serve the .wasm with Content-Type: application/wasm so the browser can use streaming compilation. Getting this wrong costs several hundred milliseconds of startup on every load and is one of the most common misconfigurations we find during audits.

Ship two builds — one SIMD+threads, one SIMD-only — and pick at load time:

async function pickOpenCvBuild() {
  const simd = await wasmFeatureDetect.simd();          // from the wasm-feature-detect package
  const threads = self.crossOriginIsolated && await wasmFeatureDetect.threads();
  if (simd && threads) return "/wasm/opencv_simd_threads.js";
  if (simd)            return "/wasm/opencv_simd.js";
  return "/wasm/opencv_base.js";
}

3. Frames without tears: OffscreenCanvas and a worker

The naive tutorial pattern — requestAnimationFrame, draw video to a canvas, getImageData, cv.matFromImageData — works and is the wrong thing to ship. It runs OpenCV on the main thread, so every frame competes with layout, and getImageData forces a GPU→CPU readback synchronously.

Do it in a Web Worker instead, and move pixels with VideoFrame + OffscreenCanvas:

// main.js
const stream = await navigator.mediaDevices.getUserMedia({
  video: { width: { ideal: 1280 }, height: { ideal: 720 }, facingMode: "environment" }
});
const [track] = stream.getVideoTracks();
const processor = new MediaStreamTrackProcessor({ track });   // WebCodecs
const worker = new Worker("/worker.js", { type: "module" });
worker.postMessage({ type: "init", readable: processor.readable }, [processor.readable]);

// worker.js
import cvReady from "./opencv_loader.js";

const cv = await cvReady();
let src, gray;

self.onmessage = async ({ data }) => {
  if (data.type !== "init") return;
  const reader = data.readable.getReader();
  const canvas = new OffscreenCanvas(1280, 720);
  const ctx = canvas.getContext("2d", { willReadFrequently: true });

  while (true) {
    const { value: frame, done } = await reader.read();
    if (done) break;

    // Drop frames if we are behind: latency beats completeness for live preview.
    if (frame.timestamp < lastProcessed - 100_000) { frame.close(); continue; }

    ctx.drawImage(frame, 0, 0);
    frame.close();                                   // ALWAYS. VideoFrame holds GPU memory.

    const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);
    src  = src  ?? new cv.Mat(canvas.height, canvas.width, cv.CV_8UC4);
    gray = gray ?? new cv.Mat();
    src.data.set(imageData.data);                    // reuse the Mat, do not allocate per frame

    cv.cvtColor(src, gray, cv.COLOR_RGBA2GRAY);
    // ... your pipeline ...

    self.postMessage({ type: "result", payload: summarise(gray) });
  }
};

Two rules in that snippet carry most of the performance:

1. Allocate Mats once and reuse them. Every new cv.Mat() is a malloc inside the WASM heap. Per-frame allocation fragments the heap and, if you forget .delete(), grows it until the tab dies.
2. Close every VideoFrame. WebCodecs frames are reference-counted GPU resources. Leaking them stalls the camera pipeline within a couple of seconds.

4. A real pipeline: document capture assistance

Here is the classical half of the most common browser-vision request: detect a rectangular document in the preview, tell the user to move closer or flatten the angle, and deliver a deskewed crop.

function findDocumentQuad(gray, cv) {
  const blurred = new cv.Mat(), edges = new cv.Mat();
  const contours = new cv.MatVector(), hierarchy = new cv.Mat();
  try {
    cv.GaussianBlur(gray, blurred, new cv.Size(5, 5), 0);
    cv.Canny(blurred, edges, 60, 180);
    cv.findContours(edges, contours, hierarchy, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE);

    let best = null, bestArea = gray.rows * gray.cols * 0.15;   // must fill 15% of frame
    for (let i = 0; i < contours.size(); i++) {
      const c = contours.get(i);
      const peri = cv.arcLength(c, true);
      const approx = new cv.Mat();
      cv.approxPolyDP(c, approx, 0.02 * peri, true);
      const area = Math.abs(cv.contourArea(approx));
      if (approx.rows === 4 && area > bestArea && cv.isContourConvex(approx)) {
        bestArea = area;
        best?.delete();
        best = approx;
      } else {
        approx.delete();
      }
      c.delete();
    }
    return best;                                  // caller owns it and must delete()
  } finally {
    blurred.delete(); edges.delete(); contours.delete(); hierarchy.delete();
  }
}

Then the deskew, which is where OpenCV.js earns its download size:

function deskew(src, quad, cv, outW = 1240, outH = 1754) {   // A4 at ~150 DPI
  const ordered = orderCorners(quad);            // TL, TR, BR, BL
  const srcTri = cv.matFromArray(4, 1, cv.CV_32FC2, ordered.flat());
  const dstTri = cv.matFromArray(4, 1, cv.CV_32FC2,
    [0, 0, outW, 0, outW, outH, 0, outH]);
  const M = cv.getPerspectiveTransform(srcTri, dstTri);
  const out = new cv.Mat();
  cv.warpPerspective(src, out, M, new cv.Size(outW, outH),
                     cv.INTER_LINEAR, cv.BORDER_CONSTANT, new cv.Scalar());
  srcTri.delete(); dstTri.delete(); M.delete();
  return out;
}

The capture-assistance logic on top is unglamorous and is what makes the feature work: refuse the shot if the quad area is below a threshold (too far), if the opposing-edge length ratio is worse than ~1.4 (too oblique), if the variance of the Laplacian is below a tuned value (blurred), or if the crop's mean luminance is outside a band (too dark, or blown out by glare). Rejecting a bad capture in the browser is worth more than any amount of server-side cleverness afterwards.

5. Adding a neural network with ONNX Runtime Web

Hand the pre-processed tensor from OpenCV.js straight into ONNX Runtime Web. Do the letterbox in OpenCV — it is faster and it matches what your training pipeline did.

import * as ort from "onnxruntime-web/webgpu";

const session = await ort.InferenceSession.create("/models/det_320_int8.onnx", {
  executionProviders: ["webgpu", "wasm"],   // graceful fallback, in order
  graphOptimizationLevel: "all"
});

function toNCHW(rgbMat, cv, size = 320) {
  const resized = new cv.Mat();
  cv.resize(rgbMat, resized, new cv.Size(size, size), 0, 0, cv.INTER_LINEAR);
  const data = new Float32Array(3 * size * size);
  const px = resized.data;                       // RGBA, uint8
  const plane = size * size;
  for (let i = 0; i < plane; i++) {
    data[i]             = px[i * 4]     / 255;
    data[i + plane]     = px[i * 4 + 1] / 255;
    data[i + 2 * plane] = px[i * 4 + 2] / 255;
  }
  resized.delete();
  return new ort.Tensor("float32", data, [1, 3, size, size]);
}

const output = await session.run({ images: toNCHW(rgb, cv) });

Practical constraints we have hit repeatedly:

• Quantise. INT8 or at minimum FP16 weights. On the WASM backend, INT8 is roughly 2–3× faster than FP32 and a quarter of the download. Use per-channel quantisation with a calibration set drawn from real user captures, not from a public dataset.
• Opset and operator coverage. ONNX Runtime Web supports a subset of operators per backend. An op that is missing on WebGPU falls back to CPU per node, which can make a "GPU" run slower than pure WASM. Check the profiler output rather than assuming.
• Warm up the session. The first inference includes shader compilation on WebGPU — often 300–800 ms. Run a dummy tensor at load time, behind your splash screen.
• Cache the weights. Put the .onnx in the Cache Storage API with a content-hashed URL and an immutable Cache-Control. Users should download the model once, ever.
• WebGPU is not universal. It is solid on current Chrome/Edge desktop and Chrome Android; Safari support has landed but is newer and less predictable. Your wasm fallback is not a formality, it is the path a large minority of your users will take.

6. Headers, isolation and the deployment checklist

Threading in WASM requires SharedArrayBuffer, which requires cross-origin isolation:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

The moment you set those, every third-party resource — analytics scripts, fonts, embedded videos, your marketing tag manager — must serve Cross-Origin-Resource-Policy: cross-origin or it will be blocked. On a marketing site with an embedded demo, this is frequently the deciding factor: teams disable threading rather than fight the tag stack. Measure the SIMD-only build first; if it is fast enough, skip the isolation headers entirely and save yourself the argument.

The rest of the checklist:

• getUserMedia requires a secure context (HTTPS or localhost). No exceptions.
• On iOS Safari, camera access inside a cross-origin iframe is restricted, and the first play() may require a user gesture. Test on a real device; the simulator lies.
• iOS also caps WASM memory more aggressively than desktop. Set -s INITIAL_MEMORY and -s MAXIMUM_MEMORY deliberately in the Emscripten build rather than relying on unbounded growth.
• Handle track ending (track.onended) — users revoke camera permission, unplug webcams, and background the tab.
• Add a visible privacy statement. The whole value proposition of browser-side vision is that pixels never leave the device; say so, and make sure it stays true, including in your error reporting (never attach frames to Sentry payloads).

7. When the browser is the wrong answer

Be honest with the trade-off. Move server-side, or to a native app, when:

• The model is larger than roughly 30–50 MB after quantisation. Download cost dominates.
• You need sustained 30 fps on arbitrary devices, including budget Android hardware.
• You need reproducible, auditable results — browser/GPU/driver variation means two users can get different outputs from the same image, which is a problem in regulated inspection or medical contexts.
• You need model confidentiality. Anything you ship to a browser is downloadable. Assume your weights are public.
• You need multi-camera sync, hardware triggers, or precise timestamps.

A very common and very good compromise: run the cheap, fast work in the browser (detect, crop, deskew, quality-gate, blur faces or redact PII) and send only the small validated crop to a server model. That cuts bandwidth, cuts cloud inference cost by an order of magnitude, keeps most raw imagery on the device, and gives the user instant feedback.

Where this fits

Browser-side vision is not a replacement for the embedded and server pipelines we cover in Real-time object detection on Jetson Orin Nano or OpenCV on Android in 2026 — it is the zero-install tier above them, and it is often the fastest way to get a vision feature in front of users without provisioning a single GPU.

If you are weighing a browser-side capture pipeline, a hybrid browser/server split, or you have an OpenCV.js prototype that runs at three frames per second and you need to know whether that is fixable, get in touch. Our OpenCV consultants have built and profiled these pipelines across document capture, retail, inspection and medical imaging, and can usually tell you in a short review whether the browser is the right deployment target for your workload.