WebCodecs API Interoperability #1378

Description

Focus Area Proposal: WebCodecs API Interoperability

Feature description

This is a proposal to include the WebCodecs API as a primary Focus Area.

The WebCodecs API provides web developers with low-level access to individual frames of a video or audio stream, as well as the browser's hardware media decoders and encoders. This provides finer control over media than the standard <video> element, allowing developers to decode frames natively and render them directly into WebGL/WebGPU contexts.

Rationale

Currently, rendering high-performance video on the web relies heavily on the HTML <video> element. While sufficient for document-driven web pages, this is a massive architectural bottleneck for complex, WebGL/WebGPU-driven applications, WebAssembly (WASM) game engines, and heavily utilized UI frameworks like Flutter Web.

The Developer Pain Point (Canvas & WASM):
Frameworks like Flutter Web render their entire UI using WASM-compiled engines (CanvasKit or Impeller) bound to a single HTML <canvas> element. Because they cannot currently reliably decode video streams internally across all browsers, they are forced to use "Platform Views"—injecting HTML <video> DOM elements over or under the canvas context.

This DOM interleaving creates severe performance issues:

1. Z-Index and Clipping Nightmares: Synchronizing the layout, scrolling, and clipping of an overlapping DOM node with a heavily animated WebGL canvas causes visual artifacts and lagging.
2. Context Loss: Moving the <video> element around the DOM tree to match canvas state changes often destroys the media context or causes rendering stutters.
3. WASM Friction: WebAssembly engines lack direct, synchronous access to decoded media frames from the HTML <video> tag without incurring severe performance penalties copying frame data.

With robust, interoperable WebCodecs support, WASM engines can fetch a video stream, decode the individual frames natively, and paint those VideoFrame objects directly onto the canvas as raw image textures. This completely eliminates the need for HTML DOM interleaving, solving one of the most highly requested capabilities for hybrid rendering frameworks.

The Mobile Urgency:
Apple shipped WasmGC support back in iOS 18.2 (late 2024), finally allowing true WebAssembly applications to run on iPhones and iPads. However, nearly two years later, mobile WebKit's implementation of WebCodecs still struggles with memory management and hardware decoding limits. Developers remain unable to reliably render video inside these mobile WASM/WebGL contexts without crashing. Stabilizing WebCodecs across all platforms is long overdue and remains the final missing piece for a unified, high-performance hybrid rendering ecosystem.

Specification

• WebCodecs W3C Specification

Tests

Tests for WebCodecs are well-established in the WPT repository, but pass rates currently fluctuate across engines due to discrepancies in supported codecs, hardware acceleration fallbacks, and frame output handling.

• WPT Directory: /webcodecs
• Note: Tests covering VideoFrame integration with OffscreenCanvas and WebGL/WebGPU textures should be prioritized within this focus area.

Current Implementer Status

While WebCodecs has launched in major browsers, implementations of specific decoding profiles, hardware acceleration integration, and frame-handling behaviors remain fragmented.

• Blink: Supported.
• WebKit: Supported (Added in 16.4). While desktop Safari performs reasonably well, the mobile WebKit implementation requires critical interop focus. Despite WasmGC being available on iOS for some time now, developers trying to leverage WebCodecs inside mobile WASM/WebGL contexts continue to frequently report crashes related to aggressive mobile hardware decoding limits, as well as persistent memory leaks when passing VideoFrame objects into WebGL or OffscreenCanvas.
• Gecko: Supported (Enabled recently in Firefox 130). Because this is a newly enabled feature in Firefox, cross-browser interop smoothing is highly timely and critical right now to ensure baseline consistency.

Specification

https://w3c.github.io/webcodecs/

web-feature

webcodecs

Test Links

https://wpt.fyi/results/webcodecs

Additional Signals

No response