[kixelated]

Part of the moq-video VAAPI quest, moq-dev/moq#8013, which adds the H.265 path to moq-video's VAAPI encode backend.

Problem

The encoder writes H.264 only, so moq-video falls back to software or another backend for H.265 on Intel and AMD.

Approach

• Config::codec picks Codec::H264 (the default from Config::new) or Codec::H265, Main profile. The render node search, the entrypoint pick, and encode::probe all go by codec.
• cros-codecs has no stateless H.265 encoder to vendor, so src/encode/h265.rs writes the VPS, SPS, PPS, and slice headers itself, with the VUI carrying the color space and timing as the H.264 path does. They go to the driver packed, the parameter sets as one sequence header and each slice header on its own, which is how ffmpeg's hevc_vaapi feeds the same drivers. The packed parameter sets are required: VA's H.265 sequence parameters have no conformance window, so an SPS the driver wrote would lose the crop and the color, and a driver that takes no packed sequence header fails the open. A driver that takes no packed slice header writes its own.
• The coding tools follow VAConfigAttribEncHEVCFeatures and VAConfigAttribEncHEVCBlockSizes: a tool the driver supports or requires is on, and one it requires that this writer does not emit fails the open with its name. A driver that predicts only from non-empty lists in both directions (VA_PREDICTION_DIRECTION_BI_NOT_EMPTY, which iHD reports) gets each P picture as a generalized B picture whose list 1 repeats list 0.
• The coded size rounds the frame up to whole minimum coding blocks, as ffmpeg does, and also to the driver's VASurfaceAttribAlignmentSize. radeonsi encodes at its own 64 by 16 alignment and rewrites the VPS, SPS, and PPS at that size, keeping our conformance window. With only the minimum coding block, 1080p came out of the RX 9070 as 1920x1088, and 852x478 as 892x478.
• A DMA-BUF is encoded in place when it is NV12 at the coded size, the H.265 version of the whole-macroblock rule.
• The H.264 per-picture code moves into h264_buffers and packed_parameter_sets so both codecs share the picture loop. It writes the same buffers as before.

Impact

• Breaking: encode::probe takes the codec, and encode::Config has a new public field, codec, so a struct literal without .. no longer compiles.
• New: Codec, Config::codec.
• The encoder tests run for both codecs at 640x480, up from 320x240. radeonsi's H.265 minimum on VCN 5 is 384 wide.

Validation

• Intel Arrow Lake (iHD 26.1.2): cargo test --lib (35 tests, all on the hardware), clippy, and fmt. Each encoder test runs for both codecs: ffmpeg round trip above 35 dB at 640x480 and at 852x478, the VUI color, IDRs carrying the VPS, SPS, and PPS at the group length, in-place and scaled DMA-BUFs, RGB and YUV conversion, a 1076-row buffer blitted, and a mid-stream bitrate drop.
• AMD Radeon RX 9070 (radeonsi, Mesa 26.0.8), with the tests pointed at its node for the run only and H.265 alone: all pass except the two color conversion checks, where chroma lands about 7 code values off (the video processor, not the codec), and the bitrate drop, where radeonsi's rate control holds 250 kbps on smooth content but runs about 4x over on per-frame noise.
• Streams from both GPUs at 1920x1080, 1280x720, 852x478, 640x360, and 400x300 decode with no ffmpeg warnings at the requested size, at the level Table A.8 gives for the coded size. Luma PSNR against the source is about 55 dB on both.

Alternatives

• Letting the driver write the headers: we would lose the VUI and the level, and radeonsi takes its sequence settings from a packed SPS in any case.
• Aligning the coded size to the CTB everywhere: that would fix radeonsi too, but on iHD it would turn every 1080p DMA-BUF into a blit.

Follow-ups

• H.264 encode on the RX 9070 produces an undecodable stream; main does the same, so this PR leaves it alone.
• The chroma offset in radeonsi's video processor conversion.
• The tests always use the first render node. A way to point them at another node would let a multi-GPU machine cover every driver in one run.

Review

Codex (quest review), two rounds. Round 1 found one issue: a driver without packed sequence headers would have written its own SPS, silently dropping the crop and the VUI. Fixed in 91914d1 by refusing such a driver. Round 2: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 7e494905-f647-4d9d-8459-16646ffa4544

Commits

Reviewing files that changed from the base of the PR and between af121c7 and 91914d1.

Files selected for processing: 6

• Cargo.toml
• README.md
• src/codec.rs
• src/encode.rs
• src/encode/h265.rs
• src/lib.rs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn