Skip to content

feat(scene): HTML and CSS overlays drawn by Blitz on the program clock - #34

Merged
kixelated merged 7 commits into
mainfrom
quest/m0/overlays
Nov 4, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m0/overlays

Conversation

@kixelated

@kixelated kixelated commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem

quest/m0/overlays.md: titles, lower thirds, and tickers should be written as script-free HTML and CSS and render through Blitz into the program, with CSS animations on the media clock. #30 scoped this to the studio: the layout track, browser DOM rendering, and the headless compositor wait for the m1 composition epic.

Approach

  • Model: a new overlay source kind, { html, width, height }, beside browser. The HTML is inline, so the source is self-contained: no local path, nothing studio-only, and it travels through undo, the clipboard, and (in m1) the layout track as is.
  • Rendering, no readback: Blitz 0.3.0-beta.4 lays the document out, blitz_paint::paint_scene paints it into a vello::Scene through anyrender_vello::VelloScenePainter, and vello::Renderer::render_to_texture writes it on the studio's own wgpu 30 device into an Rgba8Unorm storage texture the compositor samples directly. Vello writes gamma-encoded straight alpha, which is what the compositor already reads.
  • Program clock: the compositor starts an overlay's CSS timeline at the render time it first sees it, and resolves the document at now - start each frame, so a frame shows the animation where the program time says. A document repaints only while Blitz reports it animating, and only while preview or program draws it (through clones and masks); one off air waits and lands in the right place when shown again.
  • Overlay thread: a Blitz BaseDocument is !Send, and the compositor sits in the engine's shared state, so documents live on one thread per compositor, which owns the Vello pipelines too. The compositor asks it to paint and waits for the reply before submitting its frame. A panic in Blitz fails the overlays ("the overlay thread stopped"; edit to restart) instead of the studio.
  • Untrusted input: the document gets no network or navigation provider and Blitz runs no scripts. On top, the source's driver checks the HTML when it opens and fails the source naming the first thing outside the subset: elements and attributes by allowlist; CSS parsed by Stylo with an error reporter (so colr: red or an invalid value is named too); rules other than style, nested, @media, and @keyframes refused (@import, @font-face, ...); properties checked against an allowlist by the names the block serializes under, so border: 1px solid passes as border rather than tripping on the border-image it resets; any url( or image-set( refused.
  • Subset: what Blitz lays out and paints through Vello's scene painter. Not filter and backdrop-filter (blitz-paint computes them, but VelloScenePainter::push_layer drops both), mix-blend-mode, text-shadow, mask, text-overflow, multi-column layout, or logical borders. A test checks every listed name is one Stylo parses.
  • Fonts: Noto Sans regular, bold, italic, and bold italic (the notosans crate) registered as every generic family, with system fonts off. Blitz synthesizes no bold, so one face would draw font-weight: bold light.
  • UI: the Add source form takes an overlay's HTML in a code editor and its size; it lands at its own size, like a page. Overlays publish no track (transparency over black in H.264), so Replay excludes them as it does pages.

Verified the claims the brief carried over, from crate sources and a probe on this Mac:

  • Stable blitz-paint 0.2.1 takes anyrender ^0.6, whose anyrender_vello 0.6 is on vello ^0.6 and wgpu ^26.
  • 0.3.0-beta.4 resolves to anyrender 0.14, anyrender_vello 0.15, vello 0.11, and wgpu 30.0.1, one copy, the same as moq-video's.
  • The path works as described, with one correction: paint_scene takes (painter, doc, scale, width, height, x_offset, y_offset).

Measurements

Apple M4, Metal, 1920x1080 canvas at 30 fps, a solid background plus a 1920x1080 lower third (gradient, rounded box, shadow, two lines of text), 120 frames after a warm-up, release build:

render() CPU render + GPU wait (mean) p95
no overlay 0.29 ms 2.24 ms 5.9 ms
static overlay 0.22 ms 2.01 ms 2.3 ms
animated overlay 1.06 ms 3.89 ms 6.0 ms
  • GPU copies: none. Vello writes the texture the compositor samples; nothing is read back or uploaded per frame.
  • A static overlay paints once, so it costs nothing per frame. An animated 1080p one adds about 0.8 ms of CPU (style, layout, and Vello encoding on the overlay thread, waited on) and about 1.7 ms of GPU per frame.
  • Vello's pipelines compile with the first overlay. Its macOS default of one compile thread took 1.0 s there; across threads, 22 ms. It uses threads.
  • Encoder load: unchanged. The overlay is composited before export, like any item.

Impact

  • Scene file: a new source kind, kind = "overlay", with html, width, and height (each 1 to 4096).
  • Sources panel: Add source offers Overlay, with an HTML editor and a size.
  • An overlay outside the supported subset fails to start, with the reason in the Sources panel.
  • README documents overlays and the supported subset.

Public API: moq_scene::Source::Overlay (a new variant; moq-studio is the only consumer) and moq_scene::Error::Overlay. Wire: none; overlays publish no track and the catalog is unchanged.

Alternatives

  • Stable Blitz 0.2 with a readback: a second wgpu 26 device, then a readback and upload of every animated frame (about 8 MB at 1080p). Passed over (see Decisions).
  • A path to an .html file, like image: easier to edit in an editor, but a local file the m1 layout cannot serialize, which relative-path pastes break, and outside undo.
  • Building the document on the compositor's thread: BaseDocument is !Send, and the compositor must be Send.
  • Checking the subset in Collection::validate, so an edit is refused outright: validation runs on every edit, drags included, and would re-parse every overlay each time. The driver's check matches how an unreadable image fails.
  • A denylist of what Blitz does not draw: shorter, but every Stylo property not listed would pass silently.

Choices made without asking

  • Inline html over a file path (above).
  • Noto Sans as the only face, about 830 KB in the binary, over the machine's fonts: the same pixels on every machine, and no fontconfig in the Linux build.
  • An overlay's timeline starts when it opens, and restarts only when it is edited.
  • Inert attributes like onclick are refused rather than ignored, so a pasted snippet says what it expected.

Decisions

  • ✅ Blitz pinned exactly to =0.3.0-beta.4 (blitz-dom, blitz-html, blitz-paint, blitz-traits), the first line whose Vello and wgpu match moq-video's, over stable 0.2 plus a readback or waiting for stable 0.3 (maintainer, 2026-11-04).

Follow-ups

  • Overlays carry their own fonts and images, for logos and brand faces, as data: URLs or files beside the scene: a NetProvider that serves only those, and the check widened to match.
  • An overlay's timeline restarts when it goes on air, so a lower third plays its entrance at each cut, as OBS's "refresh when the scene becomes active" does.
  • Edit an overlay's HTML in the studio after adding it, not only in the scene file.
  • Upstream to DioxusLabs/blitz (outside our orgs, so not filed): VelloScenePainter::push_layer ignores filter and backdrop-filter, and blitz-paint synthesizes no bold from parley's Synthesis::embolden.

Validation

  • just ci passes at the head, after merging origin/main (chore(moq): pin moq ffcb7cd for the presentation timeline #35's moq bump to ffcb7cd and feat(studio): microphones and system audio read stalled when they stop delivering #31). Cargo.lock is main's with Blitz's crates re-added by cargo; nothing in the moq bump broke the overlay code.
  • GPU tests (ignored like the other compositor tests; just render-sw runs them on Linux), on Metal:
    • compositor::tests::an_overlay_animates_on_the_program_clock: the overlay opens off air, is cut to, and shows the midpoint at 500 ms, three quarters at 750 ms, and blue at 2 s.
    • compositor::tests::a_stopped_overlay_thread_fails_every_overlay: with one overlay shown and one painted then off air, stopping the overlay thread fails both, and cutting to the off-air one shows no frozen paint.

Review

Codex (quest review), gpt-6.1-sol, medium effort, head 8de1b23: no findings. The head after it only merges origin/main (#31) without conflicts.

  • Round 1 (head 72e6644), medium: an overlay first drawn after it opened started its animations at the first draw, since Stylo starts an animation at the resolve that creates it. Fixed in e2360ff: the overlay thread resolves each document at timeline 0 when it opens. Reproduced first by the extended GPU test, which failed with 500 ms: [255, 0, 0, 255].
  • Round 2 (head e2360ff): no findings.
  • Online, OpenAI P2 at compositor.rs:392: when the overlay thread stopped, only the overlays in that paint lost their handle and picture, so an off-air one kept its last paint. Fixed in 8de1b23: a worker-wide failure clears and fails every overlay. Reproduced first by a_stopped_overlay_thread_fails_every_overlay, which failed with aside fails.
  • Round 3 (head 8de1b23): no findings.

(Written by Claude Opus 5.5)

kixelated and others added 2 commits November 4, 2026 03:52
Quest-Claim: 734fe25480786f0d
An `overlay` source holds script-free HTML and CSS, laid out by Blitz and
painted by Vello straight into a texture on the compositor's device, with
its CSS animations on the program clock. HTML or CSS outside what overlays
draw fails the source, naming it, and an overlay loads nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title HTML/CSS overlays feat(scene): HTML and CSS overlays drawn by Blitz on the program clock Nov 4, 2026
…st drawn

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 30 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7c88a64c-38ae-48a2-ba3d-96473413f62d
Commits

Reviewing files that changed from the base of the PR and between 3a4171a and f766e20.

Files ignored due to path filters: 1
  • Cargo.lock is excluded by !**/*.lock
Files selected for processing: 11
  • README.md
  • quest/m0/README.md
  • quest/m0/overlays.md
  • rs/moq-scene/Cargo.toml
  • rs/moq-scene/src/collection.rs
  • rs/moq-scene/src/compositor.rs
  • rs/moq-scene/src/lib.rs
  • rs/moq-scene/src/overlay.rs
  • rs/moq-scene/src/source.rs
  • rs/moq-studio/src/replay.rs
  • rs/moq-studio/src/ui.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: e2360ff

Overall direction: the inline source model and direct-to-texture rendering fit the studio-only scope. One recovery issue should be fixed: a failed overlay worker must invalidate all documents it owned, including off-air ones (inline finding).

Verification: static review of the diff, source integration, and relevant Blitz/Vello dependency paths via GitHub. No build, GPU tests, or performance measurements were run here; GitHub's Check workflow is still in progress.

Comment thread rs/moq-scene/src/compositor.rs Resolved

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

This adds script-free HTML/CSS overlays rendered by Blitz on the program clock.

  • Model and serialization: New Source::Overlay { html, width, height } with validation (1..4096px, reusing browser bounds), serialized as kind = "overlay", stored inline (undo/clipboard compatible). Overlays excluded from publishing and from Replay (like Browser). README and quest docs updated accordingly.
  • Rendering: rs/moq-scene/src/overlay.rs adds a dedicated overlay thread (Blitz BaseDocument is !Send) owning Vello pipelines, fonts (bundled Noto Sans as all generic families), and painting. Documents start at timeline 0 on open, animate on the program clock via resolve(now - start), and only repaint while animating and while visible/previewed. Vello renders to an Rgba8Unorm storage texture sampled directly by the compositor; no readback/upload per frame.
  • Safety/subset: Strict allowlists for elements/attributes/CSS rules/properties; CSS parsed by Stylo with error reporting (naming first violation). Blocks url(, image-set(, @import, @font-face, scripts, resources, and unsupported properties (filter, backdrop-filter, mix-blend-mode, text-shadow, mask, etc.). Font synthesis disabled; bold uses the bundled bold face. Panic on overlay thread fails overlays gracefully ("the overlay thread stopped") without crashing the studio.
  • Integration: Compositor tracks overlays via Overlays/Overlay, paints only those visible in preview/program/top/fade-out scenes (including through clones/masks), updates timeline on open, handles thread failure. Source driver validates HTML at startup (overlay::check) and surfaces failures in Sources panel. UI adds "Overlay" kind with multiline HTML editor and size controls; placement matches Browser (own size). Compositor GPU test added for program-clock animation.

All changes look consistent with existing patterns (threading model, error handling, validation flow). Tests added for subset enforcement, font weight, and animation timing.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

@kixelated kixelated left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 5f65927 against base c9d4185.

The prior off-air overlay finding is addressed: a detected worker-wide paint failure now clears every overlay handle and cached picture, including layers outside the current paint. The new regression tests an off-air layer after worker failure and checks that cutting back cannot show its frozen paint. No new actionable finding in the follow-up.

Verification limits: GitHub-only static review of the compositor and worker paths; I did not run GPU tests, failure injection, or performance measurements.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Re-reviewed the commits since the prior pullfrog review at e2360ff, which fix overlay-thread failure recovery and merge origin/main.

  • Failed every overlay when the thread stops: Compositor::paint now walks every overlay layer, off-air ones included, clearing overlay/picture and recording "overlay thread stopped" instead of only touching the layers in the current paint set, so an off-air overlay cannot show a frozen picture when cut to. Healthy paints still skip off-air layers.
  • Added a regression test: a_stopped_overlay_thread_fails_every_overlay paints an overlay, leaves it off air, stops the worker, asserts both overlays report the failure, and checks that cutting back renders no overlay.
  • Added a test-only worker panic hook: Request::Panic and Overlays::panic stop the thread the way a Blitz panic would.
  • Merged origin/main: picked up the moq pin bump (#35) and #31, touching only Cargo.lock and quest/m0/README.md.

Pullfrog  | View workflow run | Using big-pickle (free) | 𝕏

@kixelated
kixelated merged commit 501fbb5 into main Nov 4, 2026
4 checks passed
@kixelated
kixelated deleted the quest/m0/overlays branch November 4, 2026 17:08
kixelated added a commit that referenced this pull request Nov 4, 2026
Overlays (#34) follow the transition state: the scenes they paint for
include the one a transition leaves, and a move glides an overlay item
like any other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant