feat(decode)!: H.265 Main decode beside H.264

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "moq-vaapi"
 version = "0.1.1"
-description = "(AI GENERATED) VA-API H.264/H.265 hardware encoder and H.264 decoder, derived from discord/cros-libva + discord/cros-codecs"
+description = "(AI GENERATED) VA-API H.264/H.265 hardware encoder and decoder, derived from discord/cros-libva + discord/cros-codecs"
 authors = ["The ChromiumOS Authors", "Discord", "Luke Curley <kixelated@gmail.com>"]
 repository = "https://github.com/moq-dev/vaapi"
 license = "BSD-3-Clause"

README.md

@@ -46,14 +46,21 @@ pulled in as a git dependency.
   through `VAEntrypointVideoProc`, scaling, converting, and re-tiling on the
   way. `Processor::resize` scales into NV12 surfaces it recycles per size, so a
   stream of resizes costs no allocation per frame.
-- **H.264 decode** over VA-API: one Annex-B access unit in, tightly-packed NV12
-  out, with in-stream parameter sets, a conformant DPB (reference marking,
-  reordering, frame_num gaps), and mid-stream resolution changes. Progressive
-  8-bit 4:2:0 only, with right/bottom cropping supported and left/top cropping
-  rejected. Verified bit-exact against ffmpeg's software decoder on Intel
+- **H.264 and H.265 decode** over VA-API: one Annex-B access unit in,
+  tightly-packed NV12 out, with in-stream parameter sets, a conformant DPB, and
+  mid-stream resolution changes. H.264 covers reference marking, reordering,
+  and frame_num gaps; H.265 covers reference picture sets, long-term
+  references, open-GOP CRA pictures, and dependent slice segments, and puts a
+  picture out as soon as the SPS's reorder limit allows, so a stream without
+  B-frames comes out with no delay. H.265 is the Main profile, plus Main 10 and
+  Main Still Picture streams that hold 8-bit samples. Progressive 8-bit 4:2:0
+  only, with right/bottom cropping supported and left/top cropping rejected.
+  H.264 was verified bit-exact against ffmpeg's software decoder on Intel
   Meteor Lake (iHD 26.1.5) for constrained baseline, main with B-frames, high at
-  720p, and a cropped non-macroblock-aligned size. The main-with-B-frames case
-  runs as a test wherever ffmpeg and a VA-API device are both present.
+  720p, and a cropped non-macroblock-aligned size. Both codecs, coded by x264
+  and x265 with B-frames at 320x240 and cropped at 852x478, run bit-exact as a
+  test wherever ffmpeg and a VA-API device are both present, and pass on Intel
+  Arrow Lake (iHD 26.1.2) and AMD Radeon RX 9070 (radeonsi, Mesa 26.0.8).
 - **Checked-in bindings from pinned headers.** libva's headers are vendored
   into [`libva/`](./libva) (see `just vendor`), and `just bindgen` generates
   [`src/bindings/libva.rs`](./src/bindings/libva.rs) from them. The output is
@@ -87,7 +94,7 @@ This run checked all five cros-libva commits and the one cros-codecs commit:
 | cros-libva [`89454d66`](https://github.com/chromeos/cros-libva/commit/89454d660411627a4a463116fdea00ac1ac888bc), explicit image lifetimes | Already present in `Image::create_from` and `Picture::{derive_image,create_image}`. |
 | cros-libva [`9f3c0d92`](https://github.com/chromeos/cros-libva/commit/9f3c0d9264ec04b4fa792764178b879341ce33b1), packed-header bindings | Already present in the `bindgen` allowlist, `buffer.rs`, and the H.264 encoder. |
 | cros-libva [`ed9c63b7`](https://github.com/chromeos/cros-libva/commit/ed9c63b743ce9490e1079a8b0c1a24b8e879324f), borrowed input buffers | Optional copy avoidance, not a correctness fix. Keep the existing owned-buffer API; no new public API or optimization without measurement. |
-| cros-libva [`59384456`](https://github.com/chromeos/cros-libva/commit/59384456ac2ae78c0c3e5515f41ef1efd9b802cf), libva 2.23 VP9 fields | Build compatibility already covered by `..Default::default()` in the retained VP9 wrapper against pinned 2.23 headers. The codec drivers support H.264 only, so no VP9 segmentation control is needed. |
+| cros-libva [`59384456`](https://github.com/chromeos/cros-libva/commit/59384456ac2ae78c0c3e5515f41ef1efd9b802cf), libva 2.23 VP9 fields | Build compatibility already covered by `..Default::default()` in the retained VP9 wrapper against pinned 2.23 headers. The codec drivers support only H.264 and H.265, so no VP9 segmentation control is needed. |
 | cros-codecs [`a050c6cd`](https://github.com/chromeos/cros-codecs/commit/a050c6cde530e563197ee9e3e72ab01f48a0c41f), release log filtering | Ported: remove `log/release_max_level_debug`, which conflicts with a consumer selecting another release level. `just ci` checks feature unification with `log/release_max_level_trace`. |
 
 GitHub's published security-advisory endpoints for
@@ -102,8 +109,8 @@ advisories checked, not a guarantee that no undisclosed vulnerabilities exist.
 ## Layout
 
 - `src/` — libva bindings (`bindings`, `display`, `surface`, `buffer`, ...), the
-  H.264 bitstream layer (`bitstream_utils`, `codec::h264`), the H.265 header
-  writer (`encode/h265.rs`), the thin encode
+  H.264 and H.265 bitstream layers (`bitstream_utils`, `codec::h264`,
+  `codec::h265`), the H.265 header writer (`encode/h265.rs`), the thin encode
   and decode drivers (`encode`, `decode`), DMA-BUF import (`dmabuf`), video
   post-processing (`vpp`), and the color space both share (`color`).
 - `libva/` — vendored libva headers (checked in; refreshed by `just vendor`).

src/codec.rs

@@ -1,5 +1,7 @@
-// Vendored from discord/cros-codecs (BSD-3-Clause), trimmed to H.264.
+// Vendored from discord/cros-codecs (BSD-3-Clause): H.264, and the H.265
+// parser and DPB.
 pub mod h264;
+pub mod h265;
 
 /// A video coding format.
 #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]

src/codec/h265.rs

@@ -0,0 +1,7 @@
+// Copyright 2023 The ChromiumOS Authors
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+pub mod dpb;
+pub mod parser;
+pub mod picture;

src/codec/h265/dpb.rs

@@ -0,0 +1,271 @@
+// Copyright 2023 The ChromiumOS Authors
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file.
+
+use std::cell::Ref;
+use std::cell::RefCell;
+use std::cell::RefMut;
+use std::rc::Rc;
+
+use crate::codec::h265::parser::Sps;
+use crate::codec::h265::picture::PictureData;
+use crate::codec::h265::picture::Reference;
+
+// Shortcut to refer to a DPB entry.
+//
+// The first member of the tuple is the `PictureData` for the frame.
+//
+// The second member is the backend handle of the frame.
+#[derive(Clone, Debug)]
+pub struct DpbEntry<T>(pub Rc<RefCell<PictureData>>, pub T);
+
+pub struct Dpb<T> {
+	/// List of `PictureData` and backend handles to decoded pictures.
+	entries: Vec<DpbEntry<T>>,
+	/// The maximum number of pictures that can be stored.
+	max_num_pics: usize,
+}
+
+impl<T: Clone> Dpb<T> {
+	/// Returns an iterator over the underlying H265 pictures stored in the
+	/// DPB.
+	pub fn pictures(&self) -> impl Iterator<Item = Ref<'_, PictureData>> {
+		self.entries.iter().map(|h| h.0.borrow())
+	}
+
+	/// Returns a mutable iterator over the underlying H265 pictures stored in
+	/// the DPB.
+	pub fn pictures_mut(&mut self) -> impl Iterator<Item = RefMut<'_, PictureData>> {
+		self.entries.iter().map(|h| h.0.borrow_mut())
+	}
+
+	/// Returns the length of the DPB.
+	pub fn len(&self) -> usize {
+		self.entries.len()
+	}
+
+	pub fn is_empty(&self) -> bool {
+		self.len() == 0
+	}
+
+	/// Get a reference to the whole DPB entries.
+	pub fn entries(&self) -> &Vec<DpbEntry<T>> {
+		&self.entries
+	}
+
+	/// Set the dpb's max num pics.
+	pub fn set_max_num_pics(&mut self, max_num_pics: usize) {
+		self.max_num_pics = max_num_pics;
+	}
+
+	/// Get a reference to the dpb's max num pics.
+	pub fn max_num_pics(&self) -> usize {
+		self.max_num_pics
+	}
+
+	/// Mark all pictures in the DPB as unused for reference.
+	pub fn mark_all_as_unused_for_ref(&mut self) {
+		for mut picture in self.pictures_mut() {
+			picture.set_reference(Reference::None);
+		}
+	}
+
+	/// Gets the position of `needle` in the DPB, if any.
+	fn get_position(&self, needle: &Rc<RefCell<PictureData>>) -> Option<usize> {
+		self.entries.iter().position(|handle| Rc::ptr_eq(&handle.0, needle))
+	}
+
+	/// Finds a reference picture in the DPB using `poc`.
+	pub fn find_ref_by_poc(&self, poc: i32) -> Option<DpbEntry<T>> {
+		let position = self.pictures().position(|p| p.is_ref() && p.pic_order_cnt_val == poc);
+
+		log::debug!("find_ref_by_poc: {}, found position {:?}", poc, position);
+		Some(self.entries[position?].clone())
+	}
+
+	/// Finds a reference picture in the DPB using `poc` and `mask`.
+	pub fn find_ref_by_poc_masked(&self, poc: i32, mask: i32) -> Option<DpbEntry<T>> {
+		let position = self
+			.pictures()
+			.position(|p| p.is_ref() && p.pic_order_cnt_val & mask == poc);
+
+		log::debug!("find_ref_by_poc: {}, found position {:?}", poc, position);
+		Some(self.entries[position?].clone())
+	}
+
+	/// Finds a short term reference picture in the DPB using `poc`.
+	pub fn find_short_term_ref_by_poc(&self, poc: i32) -> Option<DpbEntry<T>> {
+		let position = self
+			.pictures()
+			.position(|p| matches!(p.reference(), Reference::ShortTerm) && p.pic_order_cnt_val == poc);
+
+		log::debug!("find_short_term_ref_by_poc: {}, found position {:?}", poc, position);
+		Some(self.entries[position?].clone())
+	}
+
+	/// Drains the DPB by continuously invoking the bumping process.
+	pub fn drain(&mut self) -> Vec<DpbEntry<T>> {
+		log::debug!("Draining the DPB.");
+
+		let mut pics = vec![];
+		while let Some(pic) = self.bump(true) {
+			pics.push(pic);
+		}
+
+		pics
+	}
+
+	/// Whether the DPB needs bumping. See C.5.2.2.
+	pub fn needs_bumping(&mut self, sps: &Sps) -> bool {
+		let highest_tid = sps.max_sub_layers_minus1;
+		let max_dec_pic_buffering = usize::from(sps.max_dec_pic_buffering_minus1[usize::from(highest_tid)]) + 1;
+
+		self.output_overdue(sps) || self.entries().len() >= max_dec_pic_buffering
+	}
+
+	/// Whether more pictures wait for output than the reorder limit allows, or
+	/// one has waited past the latency limit, SpsMaxLatencyPictures (7-9).
+	fn output_overdue(&self, sps: &Sps) -> bool {
+		let highest_tid = usize::from(sps.max_sub_layers_minus1);
+		let max_num_reorder_pics = sps.max_num_reorder_pics[highest_tid];
+		let max_latency_increase_plus1 = sps.max_latency_increase_plus1[highest_tid];
+		let max_latency_pictures = i32::from(max_num_reorder_pics) + i32::from(max_latency_increase_plus1) - 1;
+
+		let num_needed_for_output = self.pictures().filter(|pic| pic.needed_for_output).count();
+		let pic_over_max_latency = self
+			.pictures()
+			.any(|pic| pic.needed_for_output && pic.pic_latency_cnt >= max_latency_pictures);
+
+		num_needed_for_output > max_num_reorder_pics.into() || (max_latency_increase_plus1 != 0 && pic_over_max_latency)
+	}
+
+	/// Find the lowest POC in the DPB that can be bumped.
+	fn find_lowest_poc_for_bumping(&self) -> Option<DpbEntry<T>> {
+		let lowest = self
+			.pictures()
+			.filter(|pic| pic.needed_for_output)
+			.min_by_key(|pic| pic.pic_order_cnt_val)?;
+
+		let position = self
+			.entries
+			.iter()
+			.position(|handle| handle.0.borrow().pic_order_cnt_val == lowest.pic_order_cnt_val)
+			.unwrap();
+
+		Some(self.entries[position].clone())
+	}
+
+	/// See C.5.2.4 "Bumping process".
+	pub fn bump(&mut self, flush: bool) -> Option<DpbEntry<T>> {
+		let handle = self.find_lowest_poc_for_bumping()?;
+		let mut pic = handle.0.borrow_mut();
+
+		pic.needed_for_output = false;
+		log::debug!("Bumping POC {} from the dpb", pic.pic_order_cnt_val);
+		log::trace!("{:#?}", pic);
+
+		if !pic.is_ref() || flush {
+			let index = self.get_position(&handle.0).unwrap();
+
+			log::debug!(
+				"Removed POC {} from the dpb: reference: {}, flush: {}",
+				pic.pic_order_cnt_val,
+				pic.is_ref(),
+				flush
+			);
+			log::trace!("{:#?}", pic);
+
+			self.entries.remove(index);
+		}
+
+		Some(handle.clone())
+	}
+
+	/// See C.5.2.3. Happens when we are done decoding the picture.
+	pub fn needs_additional_bumping(&mut self, sps: &Sps) -> bool {
+		self.output_overdue(sps)
+	}
+
+	/// Clears the DPB, dropping all the pictures.
+	pub fn clear(&mut self) {
+		log::debug!("Clearing the DPB");
+
+		let max_num_pics = self.max_num_pics;
+
+		*self = Default::default();
+		self.max_num_pics = max_num_pics;
+	}
+
+	/// Removes all pictures which are marked as "not needed for output" and
+	/// "unused for reference". See C.5.2.2
+	pub fn remove_unused(&mut self) {
+		log::debug!("Removing unused pictures from DPB.");
+		self.entries.retain(|e| {
+			let pic = e.0.borrow();
+			let retain = pic.needed_for_output || pic.is_ref();
+			log::debug!("Retaining pic POC: {}: {}", pic.pic_order_cnt_val, retain);
+			retain
+		})
+	}
+
+	/// Store a picture and its backend handle in the DPB.
+	pub fn store_picture(&mut self, picture: Rc<RefCell<PictureData>>, handle: T) -> Result<(), String> {
+		if self.entries.len() >= self.max_num_pics {
+			return Err("Can't add a picture to the DPB: DPB is full.".into());
+		}
+
+		let mut pic = picture.borrow_mut();
+		log::debug!(
+			"Stored picture POC {:?}, the DPB length is {:?}",
+			pic.pic_order_cnt_val,
+			self.entries.len()
+		);
+
+		if pic.pic_output_flag {
+			pic.needed_for_output = true;
+			pic.pic_latency_cnt = 0;
+		} else {
+			pic.needed_for_output = false;
+		}
+
+		// C.3.4.
+		// After all the slices of the current picture have been decoded, this
+		// picture is marked as "used for short-term reference".
+		pic.set_reference(Reference::ShortTerm);
+		drop(pic);
+
+		for mut pic in self.pictures_mut() {
+			pic.pic_latency_cnt += 1;
+		}
+
+		self.entries.push(DpbEntry(picture, handle));
+
+		Ok(())
+	}
+
+	/// Returns all the references in the DPB.
+	pub fn get_all_references(&self) -> Vec<DpbEntry<T>> {
+		self.entries.iter().filter(|e| e.0.borrow().is_ref()).cloned().collect()
+	}
+}
+
+impl<T: Clone> Default for Dpb<T> {
+	fn default() -> Self {
+		// See https://github.com/rust-lang/rust/issues/26925 on why this can't
+		// be derived.
+		Self {
+			entries: Default::default(),
+			max_num_pics: Default::default(),
+		}
+	}
+}
+
+impl<T: Clone> std::fmt::Debug for Dpb<T> {
+	fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+		let pics = self.entries.iter().map(|h| &h.0).enumerate().collect::<Vec<_>>();
+		f.debug_struct("Dpb")
+			.field("pictures", &pics)
+			.field("max_num_pics", &self.max_num_pics)
+			.finish()
+	}
+}