Building a Zero-Copy Dual-Vision System on RK3576 with DMA-BUF

I. The Problem: Why a Traditional OpenCV Pipeline Falls Short

During the initial validation phase of our 360° surround view system, we relied on a straightforward, battle-tested stack: OpenCV handling everything from frame capture to display. Functionally, it worked — four-channel fisheye undistortion, perspective projection, and bird's-eye stitching all produced correct results. But the moment we shifted the question from "does it run?" to "is it actually usable?", a hard reality surfaced: end-to-end latency sat at roughly 300 ms, nearly eight times the 40 ms frame budget demanded by 25 fps.

A thorough profiling session revealed that the bottleneck had nothing to do with algorithmic complexity. The RK3576's quad Cortex-A72 cluster had plenty of headroom on raw compute. What was eating the budget was something far more mundane: relentless, unnecessary data movement at every stage of the pipeline. The typical data flow looked like this:

1.1 Trying GPU Acceleration — and Hitting a New Wall

The natural next step was to offload the compute-heavy stages — undistortion, projection, stitching — to the Mali-G52 GPU via OpenCL (cv::UMat). The speedup on the compute side was real and immediate: undistortion dropped from ~10 ms on the CPU to ~1 ms, and projection transformation fell from ~30 ms to ~8 ms. The catch? Explicitly moving data between cv::Mat and cv::UMat cost roughly 15 ms on the upload and another 10 ms on the download — nearly cancelling out every gain the GPU delivered.

1.2 The Core Insight: Compute Isn't the Bottleneck — Data Movement Is

The underlying problem was architectural, not algorithmic. The "allocate independently, copy at every step" paradigm turned data movement into an absolute ceiling. The path forward wasn't a faster algorithm; it was eliminating the copies altogether.

II. The Optimization Strategy: From "It Runs" to "It Ships"

With the root cause clearly identified, the roadmap wrote itself: replace memory copies with DMA-BUF file descriptors (fds), so that V4L2, RGA, the CPU, and DRM all operate on the same physical memory without ever duplicating it. On the display side, bypass the X11 protocol stack entirely and drive output directly through a DRM Overlay Plane — eliminating the last remaining copy on the render path. The goal was simple: every pixel written once, read once, and never touched again.

III. Inside the DMA-BUF Zero-Copy Pipeline

The concept behind DMA-BUF is straightforward: instead of handing data between modules by copying it, you hand them a file descriptor that points to the same underlying physical memory. Each module reads from or writes to that shared buffer in place — no duplication, no extra allocation.

The optimized pipeline collapses to a single DMA-BUF fd chain, with each module operating in place on the same physical memory:

V4L2 MMAP (camera DMA-writes directly to physical memory) → RGA NV12→BGR (hardware blit, source virAddr → target fd, ~3 ms/channel) → CPU undistortion (cv::Mat constructed on the DMA-BUF mmap pointer — zero extra allocation) → CPU stitching (9-grid bird's-eye layout + LUT weight-map blending + car-model overlay, ~8 ms) → RGA fd→fd scaling (hardware DMA, ~2 ms) → drmModeSetPlane (Overlay Plane hardware page-flip, ~0.1 ms)

IV. Replacing X11 with DRM Overlay Plane Direct Display

Dropping cv::imshow in favor of a direct drmModeSetPlane call binds the DMA-BUF fd straight to the Overlay Plane. The display controller scans the buffer on the next VSYNC with no intermediate copies, no X Server round-trips, and no GPU compositor involvement. The submission itself takes roughly 0.1 ms and is fully non-blocking.

V. Measured Performance and Comparative Analysis

All figures below were measured on the MYIR MYD-LR3576 development board under the following conditions: four 720P fisheye cameras, HDMI output at 2560×1440@60 Hz, DMA-BUF pipeline active.

5.1 End-to-End Latency Breakdown

5.2 Head-to-Head: Three Approaches Compared

VI. DMS — Driver Monitoring System

6.1 System Overview and Architecture

If the 360° surround view gives the vehicle eyes that look outward, the DMS gives it eyes that look inward. Surround view protects the environment around the car; DMS protects the person behind the wheel. Together, they form a complete closed loop for in-vehicle intelligent vision.

On the MYIR MYD-LR3576, the DMS is built on top of the Rockchip DMS SDK, running real-time AI inference on the RK3576's 6 TOPS NPU:

• Input: MIPI CSI RGB camera at 1920×1080, mounted in front of the driver's seat and aimed at the driver's face.

• Inference Engine: RK3576 on-chip NPU running the packaged DMS model (rkdms_3576.data, ~4.9 MB), which bundles face detection, facial landmark localization, and attribute analysis into a single model file.

• Detection Capabilities: Drowsy driving (eye-closure detection), yawning, phone-call detection, and smoking detection.

• Alert Output: Categorized local audio playback — fatigue.wav, yawning.wav, phone.wav, smoking.wav.

6.2 Core Detection Metrics

The DMS SDK surfaces a rich set of facial state indicators. Applications can layer any custom decision logic on top of these raw outputs:

VII. Bringing It Together: 360° Surround View + DMS on a Single Screen

7.1 Display Layout

In a real vehicle deployment, both systems need to be visible to the driver simultaneously. We took advantage of the full 2560×1440@60 Hz HDMI resolution and implemented a left-right split-screen layout using the DRM Overlay Plane mechanism:

7.2 Full System Resource Budget

Running both systems concurrently, the resource utilization across the RK3576's heterogeneous compute units breaks down as follows. The key takeaway is that each workload lands on the right type of processor — CPU+RGA for the data-movement-heavy surround view pipeline, NPU for the inference-heavy DMS — leaving the GPU almost entirely free:

VIII. Key Takeaways

• End-to-end latency cut from 300 ms to ~100 ms. Full-chain DMA-BUF zero-copy eliminated five memory moves between capture and display. Unlike the GPU approach, which introduced timing jitter, the DMA-BUF pipeline is deterministic and fully reproducible.

• Two vision systems, one screen, one board. The 360° surround view and DMS run as independent processes in a left-right split-screen layout on a single 2560×1440 display. Process isolation keeps the architecture modular and fault-tolerant.

• Heterogeneous compute working as designed. Surround view runs on CPU+RGA; DMS runs on NPU; GPU sits nearly idle. Each workload is matched to the right accelerator — exactly what the RK3576's heterogeneous architecture was built for.