mirror of
https://github.com/storytold/filmcraft.git
synced 2026-10-14 19:26:59 -04:00
Merge pull request #162 from SaymonMaster14/feat/windows-vp9-av1-hwdecode
Windows: VP9 + AV1 hardware decode (Media Foundation / DXVA) # Conflicts: # ROADMAP.md # crates/platform/README.md
This commit is contained in:
Brandon Thomas committed
commit
71483312b6
20 files changed
+1268
-244
No files matched your search
+3
-3
@@ -36,7 +36,7 @@ before choosing work.
|
||||
| **Correctness in depth** | not tracked | **~60–70%** | A first-time contributor found four real bugs in core paths within hours (#5–#8: AIFF import failed, Slide left linked audio behind, transitions didn't follow ripple trims, export ignored start/end times). User reports: Color Matte always grey with no picker (#29); ALSA audio error on Linux (#23). Settings are "most wired", not all. |
|
||||
| **Codecs and media** | ~90% | **~75%** | Our decoders are bit-exact on conformance streams. Real camera and phone media (variable frame rate, damaged files, unusual containers) is far less tested. No camera RAW (RED, BRAW, ARRIRAW), no E-AC-3. |
|
||||
| **Export** | ~92% | **~70%** | Delivery: H.264 only (ProRes, DNxHR and image sequences cover mastering). No HEVC or AV1 export. AAF and OMF have never been validated in Avid Media Composer or Pro Tools. Export renders and encodes on the CPU only. |
|
||||
| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software) and Windows (Media Foundation / Direct3D 11 DXVA, H.264 and HEVC Main / Main 10, #30). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames), and H.265 (HEVC) export exists there only, through the same hardware encoder. Linux has no hardware path. 8K and AV1 aren't real-time. The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. |
|
||||
| **Performance and hardware** | ~62% | **~35–40%** | Premiere runs effects, decoding and encoding on the GPU and the hardware media engines. Here, hardware decode works on macOS (VideoToolbox, #33; active in the desktop app only since #41, so releases up to 0.2.1 decoded in software) and Windows (Media Foundation / Direct3D 11 DXVA, H.264, HEVC Main / Main 10, VP9 and AV1 4:2:0 8/10-bit, #30). The GPU does compositing, blend modes (#32) and 31 common effects (GPU2): colour adjustments, crop/flip/transform, blurs, sharpen. Lumetri, keys, masks, the remaining effects and export rendering are CPU; H.264 encoding can use VideoToolbox on macOS (opt-in per export, no B-frames), and H.265 (HEVC) export exists there only, through the same hardware encoder. Linux has no hardware path. 8K isn't real-time (AV1 is on Windows with a hardware decoder). The effect list shows Premiere's "GPU accelerated" badge, but only 31 effect ids run on the GPU here; the rest, Lumetri included, run on the CPU. |
|
||||
| **Stability** | not tracked | **improving, unproven** | The never-crash pass (no panics in product code, last-resort guards) landed on 2026-10-04. There's no field record from real users' projects yet. |
|
||||
| **Plugins and pro ecosystem** | not tracked | **~0–10%** | No audio plugin hosting (VST3 / Audio Units) and no third-party video effects (OpenFX). For many professional editors this alone rules FilmCraft out. Team Projects and Productions are out of scope by design. |
|
||||
| **AI features** | partial | **~25–35%** | Speech to text exists but is off in default builds (the `whisper` feature). Enhance Speech is a DSP chain, not a model. Auto Reframe is approximate. No Generative Extend, and no media-intelligence search or auto colour. |
|
||||
@@ -48,7 +48,7 @@ before choosing work.
|
||||
In priority order. Agents should prefer this work over adding more checklist items.
|
||||
|
||||
1. **Hardware acceleration** (#30), the most visible gap to users:
|
||||
- hardware decode on Linux (VA-API); VP9 / AV1 on Windows;
|
||||
- hardware decode on Linux (VA-API);
|
||||
- zero-copy decoded frames into wgpu;
|
||||
- the remaining effects on the GPU: Lumetri, keys, Vignette, Video Limiter, masks (31 common effects are done);
|
||||
- GPU export, then hardware encode: H.264 and H.265 on macOS are in (H.264 opt-in; H.265 has no other encoder); other systems remain.
|
||||
@@ -154,7 +154,7 @@ Status: ✅ done · 🟡 in progress · ⬜ not started. Estimates are remaining
|
||||
|
||||
## Running now
|
||||
|
||||
- **Hardware acceleration (#30):** landed: blend modes on the GPU (#32); VideoToolbox hardware decode + the `platform` FFI crate (#33); 31 common effects on the GPU (GPU2). Windows: Media Foundation / Direct3D 11 H.264 + HEVC decode (HW2). Next: VA-API (Linux) decode, zero-copy upload, Lumetri on the GPU.
|
||||
- **Hardware acceleration (#30):** landed: blend modes on the GPU (#32); VideoToolbox hardware decode + the `platform` FFI crate (#33); 31 common effects on the GPU (GPU2). Windows: Media Foundation / Direct3D 11 H.264 + HEVC decode (HW2) and VP9 + AV1 (HW3). Next: VA-API (Linux) decode, zero-copy upload, Lumetri on the GPU.
|
||||
- **Next:** [Where we are lacking](#where-we-are-lacking) items 1–5. GPU export waits for the export-crate work (#19 and the never-crash follow-up) to land.
|
||||
|
||||
## Log
|
||||
|
||||
@@ -12,6 +12,7 @@ use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
|
||||
use filmcraft_isobmff::{CodecConfig, SampleEntry};
|
||||
|
||||
pub use crate::hw_frame::{FrameCodec, FrameStreamInfo, PictureParams};
|
||||
use crate::video::{avcc_length_size, h264_disposable, hevc_disposable, hvcc_length_size_and_tid, sar_par, vui_color};
|
||||
use crate::{CodecError, Result};
|
||||
|
||||
@@ -133,6 +134,71 @@ pub struct NalStreamInfo {
|
||||
pub reorder: usize,
|
||||
}
|
||||
|
||||
/// Any stream a hardware decoder can take: what [`crate::hw`]'s hybrid decoder needs to know about
|
||||
/// it to find restart points and in-band parameter changes.
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum StreamInfo {
|
||||
Nal(NalStreamInfo),
|
||||
Frame(FrameStreamInfo),
|
||||
}
|
||||
|
||||
impl From<NalStreamInfo> for StreamInfo {
|
||||
fn from(i: NalStreamInfo) -> Self {
|
||||
Self::Nal(i)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<FrameStreamInfo> for StreamInfo {
|
||||
fn from(i: FrameStreamInfo) -> Self {
|
||||
Self::Frame(i)
|
||||
}
|
||||
}
|
||||
|
||||
impl StreamInfo {
|
||||
/// Whether decoding can restart at `sample` with no earlier state (IDR / IRAP, a key frame).
|
||||
pub fn is_irap(&self, sample: &[u8]) -> bool {
|
||||
match self {
|
||||
Self::Nal(i) => i.is_irap(sample),
|
||||
Self::Frame(i) => i.is_random_access(sample),
|
||||
}
|
||||
}
|
||||
|
||||
/// An HEVC CRA restarts nothing by itself (its RASL pictures need the previous restart point).
|
||||
pub fn keeps_previous_restart(&self, sample: &[u8]) -> bool {
|
||||
match self {
|
||||
Self::Nal(i) => i.codec == NalCodec::Hevc && i.nal_types(sample).contains(&21),
|
||||
Self::Frame(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `sample` carries parameters different from the ones the hardware decoder was set up
|
||||
/// with (parameter sets, a sequence header, a key frame of another format).
|
||||
pub fn parameters_changed(&self, sample: &[u8]) -> bool {
|
||||
match self {
|
||||
Self::Nal(i) => {
|
||||
i.nals(sample).into_iter().any(|n| n.first().is_some_and(|&h| i.is_parameter_set(i.nal_type(h))) && !i.parameter_sets.iter().any(|p| p == n))
|
||||
}
|
||||
Self::Frame(i) => i.parameters_changed(sample),
|
||||
}
|
||||
}
|
||||
|
||||
/// As the software decoder's [`crate::VideoDecoder::is_random_access`].
|
||||
pub fn is_random_access(&self, sample: &[u8]) -> Option<bool> {
|
||||
match self {
|
||||
Self::Nal(i) => i.is_random_access(sample),
|
||||
Self::Frame(i) => Some(i.is_random_access(sample)),
|
||||
}
|
||||
}
|
||||
|
||||
/// As the software decoder's [`crate::VideoDecoder::is_disposable`].
|
||||
pub fn is_disposable(&self, sample: &[u8]) -> bool {
|
||||
match self {
|
||||
Self::Nal(i) => i.is_disposable(sample),
|
||||
Self::Frame(_) => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Length-prefixed parameter-set entries (`u16` length + bytes) starting at `pos`.
|
||||
fn read_sets(rec: &[u8], pos: &mut usize, count: usize, out: &mut Vec<Vec<u8>>) -> Result<()> {
|
||||
for _ in 0..count {
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
//! What a hardware decoder needs to know about a VP9 or AV1 stream so that it behaves exactly like
|
||||
//! our software decoder for it (the frame-coded counterpart of [`crate::hw::NalStreamInfo`]):
|
||||
//! codec profile, bit depth and chroma format from the container's `vpcC` / `av1C`, the picture
|
||||
//! size and colour as the software decoder reports them (read from the bitstream the same way),
|
||||
//! random-access samples, and in-band changes of the stream's parameters.
|
||||
|
||||
use filmcraft_color::{ColorInfo, Primaries, Transfer};
|
||||
use filmcraft_isobmff::{CodecConfig, SampleEntry};
|
||||
|
||||
use crate::video::primaries_from_code;
|
||||
use crate::{CodecError, Result};
|
||||
|
||||
/// The two frame-coded codecs hardware decoders take besides H.264 / HEVC.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum FrameCodec {
|
||||
Vp9,
|
||||
Av1,
|
||||
}
|
||||
|
||||
/// Picture parameters read from a random-access sample.
|
||||
#[derive(Clone, Copy, Debug, PartialEq)]
|
||||
pub struct PictureParams {
|
||||
/// Picture size in luma samples (the decoded size).
|
||||
pub size: (u32, u32),
|
||||
pub bit_depth: u32,
|
||||
/// `subsampling_x`, `subsampling_y`.
|
||||
pub subsampling: (u8, u8),
|
||||
/// Colour of every output picture, as the software decoder reports it.
|
||||
pub color: ColorInfo,
|
||||
}
|
||||
|
||||
/// A VP9 or AV1 stream, from its sample entry.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct FrameStreamInfo {
|
||||
pub codec: FrameCodec,
|
||||
/// VP9 `profile` (0..=3) / AV1 `seq_profile` (0..=2).
|
||||
pub profile: u8,
|
||||
pub bit_depth: u32,
|
||||
pub subsampling: (u8, u8),
|
||||
pub mono: bool,
|
||||
/// Picture size: the sample entry's, checked against the bitstream's.
|
||||
pub size: (u32, u32),
|
||||
/// AV1: the `av1C` configuration OBUs (the sequence header the decoder is primed with).
|
||||
pub config_obus: Vec<u8>,
|
||||
/// AV1: the sequence header's payload; VP9: none.
|
||||
sequence_header: Option<Vec<u8>>,
|
||||
/// Colour of the stream when it is known before the first picture (AV1: from the sequence
|
||||
/// header).
|
||||
pub color: Option<ColorInfo>,
|
||||
/// VP9: transfer and primaries from the container (not in the bitstream).
|
||||
transfer: Option<Transfer>,
|
||||
primaries: Option<Primaries>,
|
||||
}
|
||||
|
||||
/// The OBUs of a low-overhead AV1 sample: `(type, payload)`, stopping at the first malformed one.
|
||||
fn obus(data: &[u8]) -> Vec<(u8, &[u8])> {
|
||||
let mut out = Vec::new();
|
||||
let mut pos = 0usize;
|
||||
while let Some(&h) = data.get(pos) {
|
||||
let (obu_type, ext, has_size) = ((h >> 3) & 0xf, h & 4 != 0, h & 2 != 0);
|
||||
let mut p = pos + 1 + usize::from(ext);
|
||||
let size = if has_size {
|
||||
let (mut v, mut shift, mut n) = (0usize, 0u32, 0usize);
|
||||
loop {
|
||||
let Some(&b) = data.get(p + n) else { return out };
|
||||
v |= usize::from(b & 0x7f).checked_shl(shift).unwrap_or(0);
|
||||
n += 1;
|
||||
if b & 0x80 == 0 || n >= 8 {
|
||||
break;
|
||||
}
|
||||
shift += 7;
|
||||
}
|
||||
p += n;
|
||||
v
|
||||
} else {
|
||||
data.len().saturating_sub(p)
|
||||
};
|
||||
let Some(payload) = data.get(p..p.saturating_add(size)) else { return out };
|
||||
out.push((obu_type, payload));
|
||||
pos = p + size;
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// VP9 colour as our decoder reports it: `color_space` and range from the bitstream, transfer and
|
||||
/// primaries from the container when it says. (Shared with `Vp9Decoder`.)
|
||||
pub(crate) fn vp9_color(width: u32, height: u32, color_space: u8, full_range: bool, transfer: Option<Transfer>, primaries: Option<Primaries>) -> ColorInfo {
|
||||
use filmcraft_color::{Matrix, Range};
|
||||
let mut color = ColorInfo { matrix: filmcraft_frame::default_matrix(width, height), ..ColorInfo::REC709 };
|
||||
// color_space (7.2.2): 1 BT.601, 2 BT.709, 3 SMPTE-170, 4 SMPTE-240, 5 BT.2020, 7 sRGB.
|
||||
match color_space {
|
||||
1 | 3 => color.matrix = Matrix::Bt601,
|
||||
2 | 4 => color.matrix = Matrix::Bt709,
|
||||
5 => {
|
||||
color.matrix = Matrix::Bt2020Ncl;
|
||||
color.primaries = Primaries::Bt2020;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
if let Some(t) = transfer {
|
||||
color.transfer = t;
|
||||
}
|
||||
if let Some(p) = primaries {
|
||||
color.primaries = p;
|
||||
}
|
||||
if full_range {
|
||||
color.range = Range::Full;
|
||||
}
|
||||
color
|
||||
}
|
||||
|
||||
/// AV1 colour as our decoder reports it. (Shared with `Av1Decoder`.)
|
||||
pub(crate) fn av1_color(width: u32, height: u32, matrix_coefficients: u8, transfer: u8, primaries: u8, full_range: bool) -> ColorInfo {
|
||||
let mut color = ColorInfo::REC709;
|
||||
color.matrix = filmcraft_color::Matrix::from_code(matrix_coefficients).unwrap_or_else(|| filmcraft_frame::default_matrix(width, height));
|
||||
if let Some(t) = Transfer::from_code(transfer) {
|
||||
color.transfer = t;
|
||||
}
|
||||
color.primaries = match primaries {
|
||||
9 => Primaries::Bt2020,
|
||||
12 => Primaries::P3D65,
|
||||
5 => Primaries::Bt601_625,
|
||||
6 => Primaries::Bt601_525,
|
||||
_ => Primaries::Bt709,
|
||||
};
|
||||
if full_range {
|
||||
color.range = filmcraft_color::Range::Full;
|
||||
}
|
||||
color
|
||||
}
|
||||
|
||||
impl FrameStreamInfo {
|
||||
/// The stream info of a `vpcC` / `av1C` sample entry (`None` for other codecs).
|
||||
pub fn from_entry(e: &SampleEntry) -> Option<Result<Self>> {
|
||||
let size = e.video.as_ref().map_or((0, 0), |v| (u32::from(v.width), u32::from(v.height)));
|
||||
match &e.codec {
|
||||
CodecConfig::Vp9(c) => {
|
||||
// vpcC chroma_subsampling: 0 / 1 4:2:0, 2 4:2:2, 3 4:4:4
|
||||
let subsampling = match c.chroma_subsampling {
|
||||
0 | 1 => (1, 1),
|
||||
2 => (1, 0),
|
||||
_ => (0, 0),
|
||||
};
|
||||
Some(Ok(Self {
|
||||
codec: FrameCodec::Vp9,
|
||||
profile: c.profile,
|
||||
bit_depth: u32::from(c.bit_depth),
|
||||
subsampling,
|
||||
mono: false,
|
||||
size,
|
||||
config_obus: Vec::new(),
|
||||
sequence_header: None,
|
||||
color: None,
|
||||
transfer: Transfer::from_code(c.transfer_characteristics),
|
||||
primaries: primaries_from_code(c.colour_primaries),
|
||||
}))
|
||||
}
|
||||
CodecConfig::Av1(c) => Some(Self::from_av1(c, size)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn from_av1(c: &filmcraft_isobmff::Av1Config, entry_size: (u32, u32)) -> Result<Self> {
|
||||
let seq =
|
||||
obus(&c.config_obus).into_iter().find(|(t, _)| *t == 1).map(|(_, p)| p).ok_or_else(|| CodecError::Decode("av1C has no sequence header".into()))?;
|
||||
let h = filmcraft_av1::SequenceHeader::parse(seq).map_err(|e| CodecError::Decode(e.to_string()))?;
|
||||
let size = (h.max_frame_width, h.max_frame_height);
|
||||
if entry_size != (0, 0) && entry_size != size {
|
||||
// the sample entry and the sequence header disagree: leave the stream to our decoder
|
||||
return Err(CodecError::Decode(format!("sample entry says {}x{}, the sequence header {}x{}", entry_size.0, entry_size.1, size.0, size.1)));
|
||||
}
|
||||
let cc = &h.color;
|
||||
Ok(Self {
|
||||
codec: FrameCodec::Av1,
|
||||
profile: h.profile,
|
||||
bit_depth: u32::from(cc.bit_depth),
|
||||
subsampling: (cc.subsampling_x, cc.subsampling_y),
|
||||
mono: cc.mono_chrome,
|
||||
size,
|
||||
config_obus: c.config_obus.clone(),
|
||||
sequence_header: Some(seq.to_vec()),
|
||||
color: Some(av1_color(size.0, size.1, cc.matrix_coefficients, cc.transfer_characteristics, cc.color_primaries, cc.color_range)),
|
||||
transfer: None,
|
||||
primaries: None,
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether `sample` has a sequence header OBU of its own (AV1).
|
||||
pub fn carries_sequence_header(&self, sample: &[u8]) -> bool {
|
||||
self.codec == FrameCodec::Av1 && obus(sample).iter().any(|(t, _)| *t == 1)
|
||||
}
|
||||
|
||||
/// A description made by hand (tests of what a backend declines).
|
||||
#[doc(hidden)]
|
||||
pub fn for_tests(codec: FrameCodec, profile: u8, bit_depth: u32, subsampling: (u8, u8), size: (u32, u32)) -> Self {
|
||||
Self {
|
||||
codec,
|
||||
profile,
|
||||
bit_depth,
|
||||
subsampling,
|
||||
mono: false,
|
||||
size,
|
||||
config_obus: Vec::new(),
|
||||
sequence_header: None,
|
||||
color: None,
|
||||
transfer: None,
|
||||
primaries: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether decoding can start at `sample` (a VP9 key frame / an AV1 temporal unit whose first
|
||||
/// frame is a shown key frame): the software decoders' own test.
|
||||
pub fn is_random_access(&self, sample: &[u8]) -> bool {
|
||||
match self.codec {
|
||||
FrameCodec::Vp9 => filmcraft_vp9::is_keyframe(sample),
|
||||
FrameCodec::Av1 => filmcraft_av1::is_key_frame_unit(sample),
|
||||
}
|
||||
}
|
||||
|
||||
/// The picture parameters a random-access `sample` declares (a VP9 key frame header; an AV1
|
||||
/// sample carrying a sequence header), or `None`.
|
||||
pub fn picture_params(&self, sample: &[u8]) -> Option<PictureParams> {
|
||||
match self.codec {
|
||||
FrameCodec::Vp9 => {
|
||||
let k = filmcraft_vp9::keyframe_info(sample)?;
|
||||
let color = vp9_color(k.width, k.height, k.color.color_space, k.color.full_range, self.transfer, self.primaries);
|
||||
Some(PictureParams {
|
||||
size: (k.width, k.height),
|
||||
bit_depth: k.bit_depth,
|
||||
subsampling: (u8::from(k.subsampling_x), u8::from(k.subsampling_y)),
|
||||
color,
|
||||
})
|
||||
}
|
||||
FrameCodec::Av1 => {
|
||||
let seq = obus(sample).into_iter().find(|(t, _)| *t == 1).map(|(_, p)| p)?;
|
||||
let h = filmcraft_av1::SequenceHeader::parse(seq).ok()?;
|
||||
let cc = &h.color;
|
||||
let size = (h.max_frame_width, h.max_frame_height);
|
||||
let color = av1_color(size.0, size.1, cc.matrix_coefficients, cc.transfer_characteristics, cc.color_primaries, cc.color_range);
|
||||
Some(PictureParams { size, bit_depth: u32::from(cc.bit_depth), subsampling: (cc.subsampling_x, cc.subsampling_y), color })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `sample` declares parameters the hardware session was not set up for: an AV1
|
||||
/// sequence header that differs from `av1C`'s, a VP9 key frame of another size, bit depth or
|
||||
/// chroma format. The decoder then hands the stream to our software decoder.
|
||||
pub fn parameters_changed(&self, sample: &[u8]) -> bool {
|
||||
match self.codec {
|
||||
FrameCodec::Av1 => obus(sample).into_iter().any(|(t, p)| t == 1 && self.sequence_header.as_deref() != Some(p)),
|
||||
FrameCodec::Vp9 => {
|
||||
self.picture_params(sample).is_some_and(|p| p.size != self.size || p.bit_depth != self.bit_depth || p.subsampling != self.subsampling)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,7 @@ pub mod apv;
|
||||
pub mod audio;
|
||||
pub mod gop;
|
||||
pub mod hw;
|
||||
mod hw_frame;
|
||||
pub mod mkv;
|
||||
pub mod mp4;
|
||||
pub mod mpeg;
|
||||
|
||||
@@ -412,32 +412,14 @@ impl Vp9Decoder {
|
||||
|
||||
/// `None` for a picture whose planes mix sample types (never produced by the decoder).
|
||||
fn convert(&self, p: filmcraft_vp9::Picture) -> Option<DecodedFrame> {
|
||||
use filmcraft_color::{Matrix, Primaries, Range};
|
||||
use filmcraft_color::Range;
|
||||
use filmcraft_frame::{Chroma, PixelData};
|
||||
use filmcraft_vp9::Plane;
|
||||
use std::sync::Arc;
|
||||
let (w, h) = (p.width as usize, p.height as usize);
|
||||
let (cw, ch) = (p.chroma_width as usize, p.chroma_height as usize);
|
||||
let mut color = filmcraft_color::ColorInfo { matrix: filmcraft_frame::default_matrix(p.width, p.height), ..filmcraft_color::ColorInfo::REC709 };
|
||||
// color_space (7.2.2): 1 BT.601, 2 BT.709, 3 SMPTE-170, 4 SMPTE-240, 5 BT.2020, 7 sRGB.
|
||||
match p.color.color_space {
|
||||
1 | 3 => color.matrix = Matrix::Bt601,
|
||||
2 | 4 => color.matrix = Matrix::Bt709,
|
||||
5 => {
|
||||
color.matrix = Matrix::Bt2020Ncl;
|
||||
color.primaries = Primaries::Bt2020;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
if let Some(t) = self.transfer {
|
||||
color.transfer = t;
|
||||
}
|
||||
if let Some(pr) = self.primaries {
|
||||
color.primaries = pr;
|
||||
}
|
||||
if p.color.full_range {
|
||||
color.range = Range::Full;
|
||||
}
|
||||
// (shared with the hardware decoders: `hw_frame::vp9_color`)
|
||||
let color = crate::hw_frame::vp9_color(p.width, p.height, p.color.color_space, p.color.full_range, self.transfer, self.primaries);
|
||||
let (pts, draft) = (p.pts, p.draft);
|
||||
if p.color.color_space == 7 {
|
||||
// RGB (profiles 1 / 3, 4:4:4): the planes carry G, B, R.
|
||||
@@ -536,25 +518,8 @@ impl Av1Decoder {
|
||||
let draft = p.draft;
|
||||
let w = p.width as usize;
|
||||
let h = p.height as usize;
|
||||
let mut color = filmcraft_color::ColorInfo::REC709;
|
||||
if let Some(m) = filmcraft_color::Matrix::from_code(p.matrix_coefficients) {
|
||||
color.matrix = m;
|
||||
} else {
|
||||
color.matrix = filmcraft_frame::default_matrix(p.width, p.height);
|
||||
}
|
||||
if let Some(t) = filmcraft_color::Transfer::from_code(p.transfer_characteristics) {
|
||||
color.transfer = t;
|
||||
}
|
||||
color.primaries = match p.color_primaries {
|
||||
9 => filmcraft_color::Primaries::Bt2020,
|
||||
12 => filmcraft_color::Primaries::P3D65,
|
||||
5 => filmcraft_color::Primaries::Bt601_625,
|
||||
6 => filmcraft_color::Primaries::Bt601_525,
|
||||
_ => filmcraft_color::Primaries::Bt709,
|
||||
};
|
||||
if p.full_range {
|
||||
color.range = filmcraft_color::Range::Full;
|
||||
}
|
||||
// (shared with the hardware decoders: `hw_frame::av1_color`)
|
||||
let color = crate::hw_frame::av1_color(p.width, p.height, p.matrix_coefficients, p.transfer_characteristics, p.color_primaries, p.full_range);
|
||||
let chroma = match (p.subsampling_x, p.subsampling_y) {
|
||||
(1, 1) => filmcraft_frame::Chroma::C420,
|
||||
(1, 0) => filmcraft_frame::Chroma::C422,
|
||||
|
||||
@@ -22,8 +22,9 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox")
|
||||
reorder buffer of the stream's own depth (`max_num_reorder_frames` /
|
||||
`sps_max_num_reorder_pics`) restores presentation order; a run starting at an HEVC CRA leaves
|
||||
out its RASL pictures, as our decoder does. Each seek (`reset`) starts a fresh session.
|
||||
- **Windows: Media Foundation + Direct3D 11 / DXVA, H.264 (`avcC`) and HEVC (`hvcC`)**, 8-bit 4:2:0
|
||||
(H.264 Baseline / Main / High, HEVC Main) and 10-bit 4:2:0 (HEVC Main 10)
|
||||
- **Windows: Media Foundation + Direct3D 11 / DXVA, H.264 (`avcC`), HEVC (`hvcC`), VP9 (`vpcC`) and
|
||||
AV1 (`av1C`)**, 8-bit 4:2:0 (H.264 Baseline / Main / High, HEVC Main, VP9 profile 0, AV1 main) and
|
||||
10-bit 4:2:0 (HEVC Main 10, VP9 profile 2, AV1 main 10)
|
||||
(`media_foundation/`). A Direct3D-aware decoder MFT (Microsoft's H.264 decoder, the HEVC Video
|
||||
Extensions' decoder, or a vendor's synchronous hardware MFT) is driven at the level of single
|
||||
access units, so there is no Source Reader and no second demuxer: the container samples FilmCraft
|
||||
@@ -43,9 +44,18 @@ let availability = filmcraft_platform::register(); // Available("VideoToolbox")
|
||||
Windows' own software decoding is never used in place of ours.
|
||||
- *Declined up front* (our decoder is used): field-coded H.264, H.264 profiles other than
|
||||
Baseline / Main / High, 10-bit H.264, 4:2:2 / 4:4:4 / monochrome, HEVC profiles other than
|
||||
Main / Main Still Picture / Main 10, larger than 8192×8192, no Direct3D 11 video device, no
|
||||
DXVA decoder for the stream on this GPU, no decoder MFT (HEVC needs the *HEVC Video Extensions*
|
||||
from the Microsoft Store, which the Windows "N" editions and some installs lack).
|
||||
Main / Main Still Picture / Main 10, VP9 profiles 1 / 3 (4:2:2 / 4:4:4 / RGB) and 12-bit, AV1
|
||||
profiles 1 / 2 and 12-bit, larger than 8192×8192, no Direct3D 11 video device, no DXVA decoder
|
||||
for the stream on this GPU, no decoder MFT (HEVC, VP9 and AV1 need the *HEVC Video Extensions*,
|
||||
*VP9 Video Extensions* and *AV1 Video Extension* from the Microsoft Store, which the Windows "N"
|
||||
editions and some installs lack). Several GPUs' DXVA lacks AV1 profiles 1 / 2 as well.
|
||||
- *VP9 and AV1* (`codecs::hw::FrameStreamInfo`, `media_foundation/stream.rs`): the container
|
||||
sample goes in as it is (a VP9 frame or superframe, an AV1 temporal unit; the `av1C` sequence
|
||||
header goes first after a seek). The MFT outputs only shown pictures, in presentation order, so
|
||||
hidden alt-ref frames and `show_existing_frame` need nothing special. Picture size and colour
|
||||
are read from the bitstream the way the software decoders do (`hw_frame::vp9_color` /
|
||||
`av1_color` are shared with them). A VP9 key frame of another size or format, or an AV1 sequence
|
||||
header unlike `av1C`'s, hands the stream to the software decoder (`HybridDecoder`).
|
||||
- `mfplat.dll` is loaded at run time (`mft.rs`), not linked: Windows "N" editions without the Media
|
||||
Feature Pack still start FilmCraft, which then decodes in software.
|
||||
- After a `flush` (which drains the MFT) the MFT only restarts at an IDR picture; the GOP cache
|
||||
@@ -165,5 +175,6 @@ playback with no dropped frames at Full, 1/2 and 1/4. Details in
|
||||
|
||||
## Not yet
|
||||
|
||||
Zero-copy upload of `CVPixelBuffer`s into wgpu textures; B-frames; 10-bit and HDR HEVC (Main 10);
|
||||
Media Foundation / D3D11 (Windows) and VA-API (Linux) decoders; field-coded H.264.
|
||||
Zero-copy upload of decoded pictures into wgpu textures (`CVPixelBuffer`s on macOS, Direct3D 11
|
||||
textures on Windows); B-frames and 10-bit / HDR HEVC (Main 10) in hardware encoding; VA-API (Linux)
|
||||
decoders; field-coded H.264; VP9 / AV1 4:4:4 and 12-bit on Windows.
|
||||
@@ -0,0 +1,7 @@
|
||||
//! Diagnostics: which decoder MFTs and DXVA profiles does this machine offer? `mfcaps`
|
||||
#[cfg(target_os = "windows")]
|
||||
fn main() {
|
||||
println!("{}", filmcraft_platform::media_foundation::capabilities());
|
||||
}
|
||||
#[cfg(not(target_os = "windows"))]
|
||||
fn main() {}
|
||||
@@ -20,8 +20,8 @@ fn main() {
|
||||
continue;
|
||||
};
|
||||
let Some(entry) = file.tracks[t].entries.first() else { continue };
|
||||
let Some(Ok(info)) = filmcraft_codecs::hw::NalStreamInfo::from_entry(entry) else {
|
||||
println!("{path}: not avcC/hvcC");
|
||||
let Some(info) = filmcraft_platform::media_foundation::stream_info(entry) else {
|
||||
println!("{path}: not an avcC / hvcC / vpcC / av1C stream this backend reads");
|
||||
continue;
|
||||
};
|
||||
let mut d = match filmcraft_platform::media_foundation::MfDecoder::new(info) {
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
use filmcraft_codecs::hw::NalStreamInfo;
|
||||
use filmcraft_codecs::hw::StreamInfo;
|
||||
use filmcraft_codecs::{CodecError, DecodedFrame, Result, VideoDecoder};
|
||||
use filmcraft_isobmff::SampleEntry;
|
||||
|
||||
@@ -27,7 +27,7 @@ pub struct HybridDecoder {
|
||||
hw: Option<Box<dyn VideoDecoder>>,
|
||||
sw: Option<Box<dyn VideoDecoder>>,
|
||||
entry: SampleEntry,
|
||||
info: NalStreamInfo,
|
||||
info: StreamInfo,
|
||||
draft: bool,
|
||||
/// Samples (and pts) since the last restart point; `None` once the bound was passed.
|
||||
log: Option<Vec<(Vec<u8>, i64)>>,
|
||||
@@ -42,7 +42,8 @@ pub struct HybridDecoder {
|
||||
|
||||
impl HybridDecoder {
|
||||
/// Wrap `hw`, a decoder for `entry` (whose stream is described by `info`).
|
||||
pub fn new(hw: Box<dyn VideoDecoder>, entry: SampleEntry, info: NalStreamInfo) -> Self {
|
||||
pub fn new(hw: Box<dyn VideoDecoder>, entry: SampleEntry, info: impl Into<StreamInfo>) -> Self {
|
||||
let info = info.into();
|
||||
filmcraft_codecs::hw::note_hw_session();
|
||||
Self {
|
||||
hw: Some(hw),
|
||||
@@ -67,7 +68,7 @@ impl HybridDecoder {
|
||||
if self.info.is_irap(sample) {
|
||||
// An HEVC CRA keeps the previous restart point: replaying from the CRA itself would
|
||||
// treat it as a first picture and drop its RASL pictures, which the hardware decodes.
|
||||
let cra = self.info.codec == filmcraft_codecs::hw::NalCodec::Hevc && self.info.nal_types(sample).contains(&21);
|
||||
let cra = self.info.keeps_previous_restart(sample);
|
||||
match self.log.as_mut() {
|
||||
Some(log) if cra => {
|
||||
log.drain(..self.last_irap.min(log.len()));
|
||||
@@ -98,13 +99,10 @@ impl HybridDecoder {
|
||||
}
|
||||
}
|
||||
|
||||
/// In-band parameter sets that differ from the sample entry's (the hardware session was set
|
||||
/// up from those).
|
||||
/// In-band parameter sets (sequence header, key frame format) that differ from the sample
|
||||
/// entry's (the hardware session was set up from those).
|
||||
fn parameter_sets_changed(&self, sample: &[u8]) -> bool {
|
||||
self.info
|
||||
.nals(sample)
|
||||
.into_iter()
|
||||
.any(|n| n.first().is_some_and(|&h| self.info.is_parameter_set(self.info.nal_type(h))) && !self.info.parameter_sets.iter().any(|p| p == n))
|
||||
self.info.parameters_changed(sample)
|
||||
}
|
||||
|
||||
/// Software output after a fallback: pictures already returned are dropped, carried hardware
|
||||
|
||||
@@ -102,7 +102,7 @@ pub fn hardware_decoder_for(entry: &filmcraft_isobmff::SampleEntry) -> bool {
|
||||
}
|
||||
#[cfg(target_os = "windows")]
|
||||
{
|
||||
filmcraft_codecs::hw::NalStreamInfo::from_entry(entry).and_then(|r| r.ok()).is_some_and(|info| media_foundation::MfDecoder::new(info).is_ok())
|
||||
media_foundation::stream_info(entry).is_some_and(|info| media_foundation::MfDecoder::new(info).is_ok())
|
||||
}
|
||||
#[cfg(not(any(target_os = "macos", target_os = "windows")))]
|
||||
{
|
||||
@@ -130,14 +130,14 @@ pub fn videotoolbox_factory(entry: &filmcraft_isobmff::SampleEntry) -> Option<fi
|
||||
}
|
||||
|
||||
/// The Media Foundation factory: a [`HybridDecoder`] around [`media_foundation::MfDecoder`] for
|
||||
/// `avcC` / `hvcC` streams a Direct3D-aware decoder MFT can decode with DXVA on this system's
|
||||
/// GPU, `None` otherwise.
|
||||
/// H.264 / HEVC / VP9 / AV1 streams a Direct3D-aware decoder MFT can decode with DXVA on this
|
||||
/// system's GPU, `None` otherwise.
|
||||
#[cfg(target_os = "windows")]
|
||||
pub fn media_foundation_factory(entry: &filmcraft_isobmff::SampleEntry) -> Option<filmcraft_codecs::Result<Box<dyn filmcraft_codecs::VideoDecoder>>> {
|
||||
if !filmcraft_codecs::hw::hardware_decoding() {
|
||||
return None;
|
||||
}
|
||||
let info = filmcraft_codecs::hw::NalStreamInfo::from_entry(entry)?.ok()?;
|
||||
let info = media_foundation::stream_info(entry)?;
|
||||
match media_foundation::MfDecoder::new(info.clone()) {
|
||||
Ok(mf) => Some(Ok(Box::new(HybridDecoder::new(Box::new(mf), entry.clone(), info)))),
|
||||
Err(why) => {
|
||||
|
||||
@@ -159,6 +159,15 @@ impl Gpu {
|
||||
}
|
||||
}
|
||||
|
||||
/// Every decoder profile the device's DXVA lists.
|
||||
pub fn profiles(&self) -> Vec<GUID> {
|
||||
// SAFETY: plain COM calls on a live interface; each index below the count is valid.
|
||||
unsafe {
|
||||
let n = self.video.GetVideoDecoderProfileCount();
|
||||
(0..n).filter_map(|i| self.video.GetVideoDecoderProfile(i).ok()).collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether DXVA on this device decodes `profile` into `format` at `size` (so a stream is
|
||||
/// declined up front instead of failing on its first sample).
|
||||
pub fn supports(&self, profile: GUID, format: SurfaceFormat, size: (u32, u32)) -> Result<(), String> {
|
||||
|
||||
@@ -12,17 +12,15 @@ use std::ffi::c_void;
|
||||
use std::mem::ManuallyDrop;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
use filmcraft_codecs::hw::NalCodec;
|
||||
use windows::Win32::Foundation::HMODULE;
|
||||
use windows::Win32::Graphics::Direct3D11::ID3D11Device;
|
||||
use windows::Win32::Media::MediaFoundation::{
|
||||
IMFActivate, IMFDXGIDeviceManager, IMFMediaBuffer, IMFMediaType, IMFSample, IMFTransform, MF_E_NO_MORE_TYPES, MF_E_NOTACCEPTING,
|
||||
MF_E_TRANSFORM_NEED_MORE_INPUT, MF_E_TRANSFORM_STREAM_CHANGE, MF_MT_FRAME_SIZE, MF_MT_INTERLACE_MODE, MF_MT_MAJOR_TYPE, MF_MT_MPEG2_PROFILE, MF_MT_SUBTYPE,
|
||||
MF_SA_D3D11_AWARE, MF_TRANSFORM_ASYNC, MF_VERSION, MFMediaType_Video, MFT_CATEGORY_VIDEO_DECODER, MFT_ENUM_FLAG, MFT_ENUM_FLAG_ASYNCMFT,
|
||||
MF_E_TRANSFORM_NEED_MORE_INPUT, MF_E_TRANSFORM_STREAM_CHANGE, MF_MT_FRAME_SIZE, MF_MT_INTERLACE_MODE, MF_MT_MAJOR_TYPE, MF_MT_MINIMUM_DISPLAY_APERTURE,
|
||||
MF_MT_SUBTYPE, MF_SA_D3D11_AWARE, MF_TRANSFORM_ASYNC, MF_VERSION, MFMediaType_Video, MFT_CATEGORY_VIDEO_DECODER, MFT_ENUM_FLAG, MFT_ENUM_FLAG_ASYNCMFT,
|
||||
MFT_ENUM_FLAG_HARDWARE, MFT_ENUM_FLAG_SORTANDFILTER, MFT_ENUM_FLAG_SYNCMFT, MFT_FRIENDLY_NAME_Attribute, MFT_MESSAGE_COMMAND_DRAIN,
|
||||
MFT_MESSAGE_COMMAND_FLUSH, MFT_MESSAGE_NOTIFY_BEGIN_STREAMING, MFT_MESSAGE_NOTIFY_START_OF_STREAM, MFT_MESSAGE_SET_D3D_MANAGER, MFT_OUTPUT_DATA_BUFFER,
|
||||
MFT_OUTPUT_STREAM_PROVIDES_SAMPLES, MFT_REGISTER_TYPE_INFO, MFVideoFormat_H264, MFVideoFormat_HEVC, MFVideoFormat_NV12, MFVideoFormat_P010,
|
||||
MFVideoInterlace_Progressive,
|
||||
MFT_OUTPUT_STREAM_PROVIDES_SAMPLES, MFT_REGISTER_TYPE_INFO, MFVideoFormat_NV12, MFVideoFormat_P010, MFVideoInterlace_Progressive,
|
||||
};
|
||||
use windows::Win32::System::Com::CoTaskMemFree;
|
||||
use windows::Win32::System::LibraryLoader::{GetProcAddress, LOAD_LIBRARY_SEARCH_SYSTEM32, LoadLibraryExW};
|
||||
@@ -152,15 +150,9 @@ impl MfApi {
|
||||
Ok(sample)
|
||||
}
|
||||
|
||||
/// The decoder MFTs for `codec`, hardware ones first, as activation objects.
|
||||
fn decoders(&self, codec: NalCodec) -> Result<Vec<IMFActivate>, String> {
|
||||
let input = MFT_REGISTER_TYPE_INFO {
|
||||
guidMajorType: MFMediaType_Video,
|
||||
guidSubtype: match codec {
|
||||
NalCodec::H264 => MFVideoFormat_H264,
|
||||
NalCodec::Hevc => MFVideoFormat_HEVC,
|
||||
},
|
||||
};
|
||||
/// The decoder MFTs taking input of `subtype`, hardware ones first, as activation objects.
|
||||
fn decoders(&self, subtype: GUID) -> Result<Vec<IMFActivate>, String> {
|
||||
let input = MFT_REGISTER_TYPE_INFO { guidMajorType: MFMediaType_Video, guidSubtype: subtype };
|
||||
let flags = MFT_ENUM_FLAG(MFT_ENUM_FLAG_SYNCMFT.0 | MFT_ENUM_FLAG_ASYNCMFT.0 | MFT_ENUM_FLAG_HARDWARE.0 | MFT_ENUM_FLAG_SORTANDFILTER.0);
|
||||
let (mut list, mut count) = (std::ptr::null_mut::<*mut c_void>(), 0u32);
|
||||
// SAFETY: the type info and the out-pointers refer to live locals.
|
||||
@@ -197,6 +189,27 @@ pub enum Poll {
|
||||
FormatChanged,
|
||||
}
|
||||
|
||||
/// An attribute of the decoder's input media type.
|
||||
pub enum Attr {
|
||||
U32(u32),
|
||||
Blob(Vec<u8>),
|
||||
}
|
||||
|
||||
/// What the decoder MFT is asked to decode.
|
||||
pub struct Spec {
|
||||
/// The input subtype (`MFVideoFormat_H264`, `_HEVC`, `_VP90`, `_AV1`).
|
||||
pub subtype: GUID,
|
||||
/// The codec's name and what is missing when no MFT takes it.
|
||||
pub label: &'static str,
|
||||
pub missing: &'static str,
|
||||
/// Picture size in luma samples.
|
||||
pub size: (u32, u32),
|
||||
/// Output format.
|
||||
pub format: SurfaceFormat,
|
||||
/// Extra input type attributes (profile, sequence header...).
|
||||
pub attrs: Vec<(GUID, Attr)>,
|
||||
}
|
||||
|
||||
/// A Direct3D-aware decoder MFT, configured for one stream.
|
||||
pub struct Mft {
|
||||
transform: IMFTransform,
|
||||
@@ -212,18 +225,15 @@ impl Mft {
|
||||
/// Create the decoder for `codec` with output in `format`, for pictures of `size`, decoding
|
||||
/// on the device of `gpu`. Errors (and so the stream is declined) when no Direct3D-aware
|
||||
/// synchronous decoder MFT takes the stream.
|
||||
pub fn new(api: &MfApi, gpu: &Gpu, codec: NalCodec, size: (u32, u32), format: SurfaceFormat) -> Result<Mft, String> {
|
||||
pub fn new(api: &MfApi, gpu: &Gpu, spec: &Spec) -> Result<Mft, String> {
|
||||
ensure_com()?;
|
||||
let candidates = api.decoders(codec)?;
|
||||
let candidates = api.decoders(spec.subtype)?;
|
||||
if candidates.is_empty() {
|
||||
return Err(match codec {
|
||||
NalCodec::H264 => "no H.264 decoder MFT".into(),
|
||||
NalCodec::Hevc => "no HEVC decoder MFT (the HEVC Video Extensions are not installed)".into(),
|
||||
});
|
||||
return Err(format!("no {} decoder MFT{}", spec.label, spec.missing));
|
||||
}
|
||||
let mut reasons = Vec::new();
|
||||
for a in candidates {
|
||||
match Self::configure(api, gpu, &a, codec, size, format) {
|
||||
match Self::configure(api, gpu, &a, spec) {
|
||||
Ok(m) => return Ok(m),
|
||||
Err(e) => reasons.push(e),
|
||||
}
|
||||
@@ -231,7 +241,8 @@ impl Mft {
|
||||
Err(reasons.join("; "))
|
||||
}
|
||||
|
||||
fn configure(api: &MfApi, gpu: &Gpu, activate: &IMFActivate, codec: NalCodec, size: (u32, u32), format: SurfaceFormat) -> Result<Mft, String> {
|
||||
fn configure(api: &MfApi, gpu: &Gpu, activate: &IMFActivate, spec: &Spec) -> Result<Mft, String> {
|
||||
let (size, format) = (spec.size, spec.format);
|
||||
let name = friendly_name(activate).unwrap_or_else(|| "a decoder MFT".into());
|
||||
// SAFETY: plain COM calls on live interfaces.
|
||||
let transform: IMFTransform = unsafe { activate.ActivateObject() }.map_err(|e| format!("{name}: cannot be created: {e}"))?;
|
||||
@@ -254,22 +265,15 @@ impl Mft {
|
||||
// SAFETY: plain COM calls on live interfaces.
|
||||
unsafe {
|
||||
input.SetGUID(&MF_MT_MAJOR_TYPE, &MFMediaType_Video).map_err(|e| e.to_string())?;
|
||||
input
|
||||
.SetGUID(
|
||||
&MF_MT_SUBTYPE,
|
||||
&match codec {
|
||||
NalCodec::H264 => MFVideoFormat_H264,
|
||||
NalCodec::Hevc => MFVideoFormat_HEVC,
|
||||
},
|
||||
)
|
||||
.map_err(|e| e.to_string())?;
|
||||
input.SetGUID(&MF_MT_SUBTYPE, &spec.subtype).map_err(|e| e.to_string())?;
|
||||
input.SetUINT64(&MF_MT_FRAME_SIZE, (u64::from(size.0) << 32) | u64::from(size.1)).map_err(|e| e.to_string())?;
|
||||
input.SetUINT32(&MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive.0 as u32).map_err(|e| e.to_string())?;
|
||||
if codec == NalCodec::Hevc {
|
||||
// the HEVC decoder lists P010 output only when told the stream is Main 10
|
||||
// (eAVEncH265VProfile_Main_420_8 = 1, eAVEncH265VProfile_Main_420_10 = 2)
|
||||
let profile = if format == SurfaceFormat::P010 { 2 } else { 1 };
|
||||
input.SetUINT32(&MF_MT_MPEG2_PROFILE, profile).map_err(|e| e.to_string())?;
|
||||
for (key, value) in &spec.attrs {
|
||||
match value {
|
||||
Attr::U32(v) => input.SetUINT32(key, *v),
|
||||
Attr::Blob(b) => input.SetBlob(key, b),
|
||||
}
|
||||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
mft.transform.SetInputType(0, &input, 0).map_err(|e| format!("{}: does not take this stream: {e}", mft.name))?;
|
||||
}
|
||||
@@ -309,6 +313,27 @@ impl Mft {
|
||||
Err(format!("{}: cannot output {:?}", self.name, self.format))
|
||||
}
|
||||
|
||||
/// The size of the pictures the decoder says it outputs: the display aperture of its output
|
||||
/// type when it has one (the surfaces are aligned: 640x368 for a 640x360 picture), else its
|
||||
/// frame size.
|
||||
pub fn output_size(&self) -> Option<(u32, u32)> {
|
||||
// SAFETY: plain COM calls on live interfaces; `GetBlob` writes at most the 16 bytes of the
|
||||
// buffer it is given (an `MFVideoArea`: two offsets of 4 bytes, then the area as two i32).
|
||||
unsafe {
|
||||
let t = self.transform.GetOutputCurrentType(0).ok()?;
|
||||
let mut area = [0u8; 16];
|
||||
let mut len = 0u32;
|
||||
if t.GetBlob(&MF_MT_MINIMUM_DISPLAY_APERTURE, &mut area, Some(&mut len)).is_ok() && len == 16 {
|
||||
let (cx, cy) = (i32::from_le_bytes([area[8], area[9], area[10], area[11]]), i32::from_le_bytes([area[12], area[13], area[14], area[15]]));
|
||||
if cx > 0 && cy > 0 {
|
||||
return Some((cx as u32, cy as u32));
|
||||
}
|
||||
}
|
||||
let packed = t.GetUINT64(&MF_MT_FRAME_SIZE).ok()?;
|
||||
Some(((packed >> 32) as u32, packed as u32))
|
||||
}
|
||||
}
|
||||
|
||||
/// The MFT's name, for logs.
|
||||
pub fn name(&self) -> &str {
|
||||
&self.name
|
||||
@@ -369,6 +394,25 @@ impl Mft {
|
||||
}
|
||||
}
|
||||
|
||||
/// One line per decoder MFT taking `subtype` (name, synchronous or not, Direct3D-aware or not):
|
||||
/// diagnostics.
|
||||
pub fn describe(api: &MfApi, subtype: GUID) -> Vec<String> {
|
||||
let Ok(list) = api.decoders(subtype) else { return vec!["enumeration failed".into()] };
|
||||
list.iter()
|
||||
.map(|a| {
|
||||
let name = friendly_name(a).unwrap_or_else(|| "?".into());
|
||||
// SAFETY: plain COM calls on live interfaces.
|
||||
let (asynchronous, d3d11) = unsafe {
|
||||
match a.ActivateObject::<IMFTransform>().and_then(|t| t.GetAttributes()) {
|
||||
Ok(attrs) => (attrs.GetUINT32(&MF_TRANSFORM_ASYNC).unwrap_or(0) != 0, attrs.GetUINT32(&MF_SA_D3D11_AWARE).unwrap_or(0) != 0),
|
||||
Err(_) => (false, false),
|
||||
}
|
||||
};
|
||||
format!("{name} (async {asynchronous}, Direct3D 11 aware {d3d11})")
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn friendly_name(a: &IMFActivate) -> Option<String> {
|
||||
// SAFETY: a plain COM call; `GetAllocatedString` returns a CoTaskMem string that the
|
||||
// returned `PWSTR` wrapper does not free, so it is freed here after copying.
|
||||
|
||||
@@ -30,58 +30,69 @@
|
||||
mod gpu;
|
||||
#[allow(unsafe_code)]
|
||||
mod mft;
|
||||
mod stream;
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use filmcraft_codecs::hw::{NalCodec, NalStreamInfo};
|
||||
use filmcraft_codecs::hw::{NalCodec, StreamInfo};
|
||||
use filmcraft_codecs::{CodecError, DecodedFrame, Result, VideoDecoder};
|
||||
use windows::Win32::Graphics::Direct3D11::{D3D11_DECODER_PROFILE_H264_VLD_NOFGT, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10};
|
||||
use windows::core::GUID;
|
||||
|
||||
use self::gpu::{Gpu, Readback, SurfaceFormat};
|
||||
use self::mft::{MfApi, Mft, Poll};
|
||||
use crate::annexb::to_annex_b;
|
||||
use self::stream::Stream;
|
||||
use crate::biplanar::{self, Geometry};
|
||||
|
||||
/// Largest picture the backend takes (as the VideoToolbox one).
|
||||
const MAX_SIDE: u32 = 8192;
|
||||
/// Pictures the decoder may hold back (a DPB is at most 16 frames; more means it lost track).
|
||||
const MAX_IN_FLIGHT: usize = 64;
|
||||
/// Upper bound on the outputs taken per call: a misbehaving MFT cannot make a decode call loop.
|
||||
const MAX_OUTPUTS_PER_CALL: usize = 4096;
|
||||
|
||||
/// The stream description of a sample entry this backend could take (`avcC` / `hvcC` / `vpcC` /
|
||||
/// `av1C`), or `None` for other codecs and unreadable configuration records.
|
||||
pub fn stream_info(entry: &filmcraft_isobmff::SampleEntry) -> Option<StreamInfo> {
|
||||
use filmcraft_codecs::hw::{FrameStreamInfo, NalStreamInfo};
|
||||
match NalStreamInfo::from_entry(entry) {
|
||||
Some(r) => r.ok().map(StreamInfo::Nal),
|
||||
None => FrameStreamInfo::from_entry(entry)?.ok().map(StreamInfo::Frame),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether Media Foundation and a Direct3D 11 video device exist on this system.
|
||||
pub fn available() -> bool {
|
||||
mft::api().and_then(Gpu::shared).is_ok()
|
||||
}
|
||||
|
||||
/// What the decoder needs for a stream, or why it is declined.
|
||||
fn plan(info: &NalStreamInfo) -> std::result::Result<(SurfaceFormat, GUID), String> {
|
||||
if info.interlaced {
|
||||
return Err("field-coded H.264".into());
|
||||
/// What each codec's decoders and the GPU's DXVA profiles offer on this machine (diagnostics:
|
||||
/// `examples/mfcaps.rs`).
|
||||
pub fn capabilities() -> String {
|
||||
use windows::Win32::Graphics::Direct3D11 as d3d;
|
||||
use windows::Win32::Media::MediaFoundation as mf;
|
||||
let (api, gpu) = match mft::api().and_then(|a| Gpu::shared(a).map(|g| (a, g))) {
|
||||
Ok(x) => x,
|
||||
Err(e) => return format!("no Media Foundation / Direct3D 11 video device: {e}"),
|
||||
};
|
||||
let mut out = format!("adapter: {}\n", gpu.name());
|
||||
for (name, subtype) in
|
||||
[("H.264", mf::MFVideoFormat_H264), ("HEVC", mf::MFVideoFormat_HEVC), ("VP9", mf::MFVideoFormat_VP90), ("AV1", mf::MFVideoFormat_AV1)]
|
||||
{
|
||||
let list = mft::describe(api, subtype);
|
||||
out += &format!("{name} decoder MFTs: {}\n", if list.is_empty() { "none".into() } else { list.join("; ") });
|
||||
}
|
||||
if info.chroma_format_idc != 1 {
|
||||
return Err(format!("chroma_format_idc {}", info.chroma_format_idc));
|
||||
}
|
||||
if info.bit_depth_luma != info.bit_depth_chroma || !matches!(info.bit_depth_luma, 8 | 10) {
|
||||
return Err(format!("{}-bit luma / {}-bit chroma", info.bit_depth_luma, info.bit_depth_chroma));
|
||||
}
|
||||
let (_, _, w, h) = info.crop;
|
||||
let (cx, cy) = (info.crop.0, info.crop.1);
|
||||
if w == 0 || h == 0 || w > MAX_SIDE || h > MAX_SIDE || cx.saturating_add(w) > info.coded.0 || cy.saturating_add(h) > info.coded.1 {
|
||||
return Err(format!("picture size {w}x{h}"));
|
||||
}
|
||||
let ten = info.bit_depth_luma == 10;
|
||||
match (info.codec, info.profile_idc, ten) {
|
||||
// Baseline, Main and High (the profiles DXVA H.264 decoders list)
|
||||
(NalCodec::H264, 66 | 77 | 100, false) => Ok((SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_H264_VLD_NOFGT)),
|
||||
(NalCodec::H264, p, _) => Err(format!("H.264 profile {p} at {} bits", info.bit_depth_luma)),
|
||||
// Main and Main Still Picture
|
||||
(NalCodec::Hevc, 1 | 3, false) => Ok((SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN)),
|
||||
(NalCodec::Hevc, 2, true) => Ok((SurfaceFormat::P010, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10)),
|
||||
(NalCodec::Hevc, p, _) => Err(format!("HEVC profile {p} at {} bits", info.bit_depth_luma)),
|
||||
let profiles = gpu.profiles();
|
||||
for (name, guid) in [
|
||||
("H.264 VLD", d3d::D3D11_DECODER_PROFILE_H264_VLD_NOFGT),
|
||||
("HEVC Main", d3d::D3D11_DECODER_PROFILE_HEVC_VLD_MAIN),
|
||||
("HEVC Main 10", d3d::D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10),
|
||||
("VP9 profile 0", d3d::D3D11_DECODER_PROFILE_VP9_VLD_PROFILE0),
|
||||
("VP9 profile 2 (10-bit)", d3d::D3D11_DECODER_PROFILE_VP9_VLD_10BIT_PROFILE2),
|
||||
("AV1 profile 0", d3d::D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0),
|
||||
("AV1 profile 1", d3d::D3D11_DECODER_PROFILE_AV1_VLD_PROFILE1),
|
||||
("AV1 profile 2", d3d::D3D11_DECODER_PROFILE_AV1_VLD_PROFILE2),
|
||||
] {
|
||||
out += &format!("DXVA {name}: {}\n", profiles.contains(&guid));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Where a decoder's time went (wall clock), for diagnostics (`examples/mfprobe.rs --time`).
|
||||
@@ -99,7 +110,9 @@ pub struct Timings {
|
||||
|
||||
/// The Media Foundation decoder for one `avcC` / `hvcC` stream.
|
||||
pub struct MfDecoder {
|
||||
info: NalStreamInfo,
|
||||
stream: Stream,
|
||||
/// The stream as the hybrid decoder's tests (random access, disposable) see it.
|
||||
info: StreamInfo,
|
||||
format: SurfaceFormat,
|
||||
geometry: Geometry,
|
||||
api: &'static MfApi,
|
||||
@@ -115,6 +128,9 @@ pub struct MfDecoder {
|
||||
/// out (as the software decoder does).
|
||||
skip_rasl: bool,
|
||||
first: bool,
|
||||
/// The size the MFT declares for its pictures was checked against the stream's (VP9 / AV1: the
|
||||
/// picture size is in the bitstream, and a decoder that disagrees would show padding or crop).
|
||||
size_checked: bool,
|
||||
/// [`VideoDecoder::flush`] drained the MFT, which then behaves as at the end of a stream: it
|
||||
/// only restarts at an IDR picture, the references of the run are gone.
|
||||
drained: bool,
|
||||
@@ -128,21 +144,21 @@ pub struct MfDecoder {
|
||||
impl MfDecoder {
|
||||
/// A decoder for the stream, or why Media Foundation does not take it (unsupported format, no
|
||||
/// DXVA decoder for it on the GPU, no Direct3D-aware decoder MFT).
|
||||
pub fn new(info: NalStreamInfo) -> std::result::Result<Self, String> {
|
||||
let (format, profile) = plan(&info)?;
|
||||
pub fn new(stream: impl Into<Stream>) -> std::result::Result<Self, String> {
|
||||
let stream = stream.into();
|
||||
let plan = stream.plan()?;
|
||||
let api = mft::api()?;
|
||||
let gpu = Gpu::shared(api)?;
|
||||
gpu.supports(profile, format, info.coded)?;
|
||||
let (_, _, w, h) = info.crop;
|
||||
let mft = Mft::new(api, &gpu, info.codec, (w, h), format)?;
|
||||
gpu.supports(plan.profile, plan.format, plan.size)?;
|
||||
let format = plan.format;
|
||||
let mft = Mft::new(api, &gpu, &stream.spec(format))?;
|
||||
log::info!("hardware decoding: {} on {}", mft.name(), gpu.name());
|
||||
let name = match info.codec {
|
||||
NalCodec::H264 => "Media Foundation H.264",
|
||||
NalCodec::Hevc => "Media Foundation HEVC",
|
||||
};
|
||||
let geometry = stream.first_geometry(format);
|
||||
Ok(Self {
|
||||
geometry: Geometry { crop: info.crop, bits: format.bits(), color: info.color, par: info.par },
|
||||
info,
|
||||
name: stream.name(),
|
||||
info: stream.info(),
|
||||
geometry,
|
||||
stream,
|
||||
format,
|
||||
api,
|
||||
gpu,
|
||||
@@ -152,10 +168,10 @@ impl MfDecoder {
|
||||
need_headers: true,
|
||||
skip_rasl: false,
|
||||
first: true,
|
||||
size_checked: false,
|
||||
drained: false,
|
||||
fail_after: None,
|
||||
fed: 0,
|
||||
name,
|
||||
timings: Timings::default(),
|
||||
})
|
||||
}
|
||||
@@ -187,12 +203,15 @@ impl MfDecoder {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `sample` holds an IDR picture (H.264 IDR slice; HEVC IDR_W_RADL / IDR_N_LP).
|
||||
fn is_idr(&self, sample: &[u8]) -> bool {
|
||||
self.info.nal_types(sample).iter().any(|&t| match self.info.codec {
|
||||
NalCodec::H264 => t == 5,
|
||||
NalCodec::Hevc => matches!(t, 19 | 20),
|
||||
})
|
||||
/// A VP9 / AV1 decoder must output the picture size the bitstream declares (H.264 / HEVC
|
||||
/// decoders report coded or cropped sizes, which the cropping handles).
|
||||
fn check_size(&mut self) -> std::result::Result<(), String> {
|
||||
self.size_checked = true;
|
||||
let (_, _, w, h) = self.geometry.crop;
|
||||
match (&self.stream, self.mft.as_ref().and_then(Mft::output_size)) {
|
||||
(Stream::Frame(_), Some(got)) if got != (w, h) => Err(format!("the decoder outputs {}x{} pictures, the stream says {w}x{h}", got.0, got.1)),
|
||||
_ => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Take every picture the decoder has ready (presentation order).
|
||||
@@ -201,8 +220,11 @@ impl MfDecoder {
|
||||
let Some(mft) = self.mft.as_ref() else { return Ok(()) };
|
||||
match mft.poll()? {
|
||||
Poll::NeedInput => return Ok(()),
|
||||
Poll::FormatChanged => {}
|
||||
Poll::FormatChanged => self.size_checked = false,
|
||||
Poll::Frame(sample, t) => {
|
||||
if !self.size_checked {
|
||||
self.check_size()?;
|
||||
}
|
||||
let geometry = self.geometry;
|
||||
let (t0, mut convert) = (Instant::now(), Duration::ZERO);
|
||||
let frame = self.readback.read(&self.gpu, &sample, self.format, |b| {
|
||||
@@ -257,14 +279,16 @@ impl VideoDecoder for MfDecoder {
|
||||
// A drained MFT cannot go on from the middle of a GOP (software decoders can). The
|
||||
// GOP cache seeks (`reset`) after every flush, so this is an error only for callers
|
||||
// that continue after one: the hybrid decoder then replays the run in software.
|
||||
if !self.is_idr(sample) {
|
||||
if !self.stream.is_restart(sample) {
|
||||
return Err(CodecError::Decode("the hardware decoder was drained and restarts only at an IDR picture".into()));
|
||||
}
|
||||
self.drained = false;
|
||||
self.first = true;
|
||||
}
|
||||
if self.info.codec == NalCodec::Hevc {
|
||||
let types = self.info.nal_types(sample);
|
||||
if let Stream::Nal(info) = &self.stream
|
||||
&& info.codec == NalCodec::Hevc
|
||||
{
|
||||
let types = info.nal_types(sample);
|
||||
if types.iter().any(|t| (16..=23).contains(t)) {
|
||||
// CRA / BLA starting a run: its RASL pictures reference pictures we never decoded.
|
||||
self.skip_rasl = self.first && types.iter().any(|t| (16..=21).contains(t) && !(19..=20).contains(t));
|
||||
@@ -274,11 +298,15 @@ impl VideoDecoder for MfDecoder {
|
||||
}
|
||||
}
|
||||
self.first = false;
|
||||
let annex_b = to_annex_b(&self.info, sample, self.need_headers).map_err(CodecError::Decode)?;
|
||||
// a VP9 key frame says the picture's size and colour (they are not in the container)
|
||||
if let Some(g) = self.stream.declared_picture(sample) {
|
||||
self.geometry = g;
|
||||
}
|
||||
let annex_b = self.stream.input(sample, self.need_headers).map_err(CodecError::Decode)?;
|
||||
if self.mft.is_none() {
|
||||
let (_, _, w, h) = self.info.crop;
|
||||
let mft = Mft::new(self.api, &self.gpu, self.info.codec, (w, h), self.format).map_err(CodecError::Decode)?;
|
||||
let mft = Mft::new(self.api, &self.gpu, &self.stream.spec(self.format)).map_err(CodecError::Decode)?;
|
||||
self.mft = Some(mft);
|
||||
self.size_checked = false;
|
||||
self.need_headers = true;
|
||||
}
|
||||
let (t0, spent) = (Instant::now(), self.timings.readback + self.timings.convert);
|
||||
@@ -316,6 +344,7 @@ impl VideoDecoder for MfDecoder {
|
||||
self.need_headers = true;
|
||||
self.skip_rasl = false;
|
||||
self.first = true;
|
||||
self.size_checked = false;
|
||||
self.drained = false;
|
||||
}
|
||||
|
||||
@@ -338,73 +367,3 @@ impl Drop for MfDecoder {
|
||||
let _ = gpu::ensure_com();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn info(codec: NalCodec, profile_idc: u8, bits: u32, chroma_format_idc: u32) -> NalStreamInfo {
|
||||
NalStreamInfo {
|
||||
codec,
|
||||
length_size: 4,
|
||||
highest_tid: None,
|
||||
parameter_sets: Vec::new(),
|
||||
coded: (1920, 1088),
|
||||
crop: (0, 0, 1920, 1080),
|
||||
chroma_format_idc,
|
||||
bit_depth_luma: bits,
|
||||
bit_depth_chroma: bits,
|
||||
interlaced: false,
|
||||
profile_idc,
|
||||
color: filmcraft_color::ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
reorder: 2,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn takes_h264_baseline_main_high_and_hevc_main_main10() {
|
||||
for p in [66, 77, 100] {
|
||||
assert_eq!(plan(&info(NalCodec::H264, p, 8, 1)).unwrap().0, SurfaceFormat::Nv12, "H.264 profile {p}");
|
||||
}
|
||||
assert_eq!(plan(&info(NalCodec::Hevc, 1, 8, 1)).unwrap(), (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN));
|
||||
assert_eq!(plan(&info(NalCodec::Hevc, 2, 10, 1)).unwrap(), (SurfaceFormat::P010, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn declines_what_dxva_does_not_decode() {
|
||||
// Hi10, High 4:2:2 / 4:4:4, Extended; 10-bit under an 8-bit profile; 4:2:2 / 4:4:4 / mono HEVC
|
||||
for (codec, p, bits, chroma) in [
|
||||
(NalCodec::H264, 110, 10, 1),
|
||||
(NalCodec::H264, 122, 10, 2),
|
||||
(NalCodec::H264, 244, 8, 3),
|
||||
(NalCodec::H264, 88, 8, 1),
|
||||
(NalCodec::H264, 100, 10, 1),
|
||||
(NalCodec::H264, 100, 8, 2),
|
||||
(NalCodec::Hevc, 4, 10, 2),
|
||||
(NalCodec::Hevc, 4, 12, 1),
|
||||
(NalCodec::Hevc, 1, 10, 1),
|
||||
(NalCodec::Hevc, 2, 8, 1),
|
||||
(NalCodec::Hevc, 1, 8, 0),
|
||||
(NalCodec::Hevc, 1, 8, 3),
|
||||
] {
|
||||
assert!(plan(&info(codec, p, bits, chroma)).is_err(), "{codec:?} profile {p} {bits}-bit chroma {chroma}");
|
||||
}
|
||||
let mut field_coded = info(NalCodec::H264, 100, 8, 1);
|
||||
field_coded.interlaced = true;
|
||||
assert!(plan(&field_coded).is_err());
|
||||
// luma and chroma depths that differ
|
||||
let mut mixed = info(NalCodec::Hevc, 2, 10, 1);
|
||||
mixed.bit_depth_chroma = 8;
|
||||
assert!(plan(&mixed).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn declines_absurd_pictures() {
|
||||
for crop in [(0, 0, 0, 1080), (0, 0, 1920, 0), (0, 0, 9000, 1080), (0, 0, 1080, 9000), (8, 0, 1920, 1080), (0, 16, 1920, 1080), (u32::MAX, 0, 16, 16)] {
|
||||
let mut i = info(NalCodec::H264, 100, 8, 1);
|
||||
i.crop = crop;
|
||||
assert!(plan(&i).is_err(), "{crop:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
//! What differs between the codecs the Windows decoder takes: which streams the GPU's DXVA decoders
|
||||
//! can do, how the decoder MFT is asked for them, and how a container sample becomes the MFT's
|
||||
//! input. H.264 / HEVC (`avcC` / `hvcC`) go in as Annex B; VP9 frames and AV1 temporal units go in
|
||||
//! as they are in the container. Safe code, with the unit tests of what is declined.
|
||||
|
||||
use filmcraft_codecs::hw::{FrameCodec, FrameStreamInfo, NalCodec, NalStreamInfo, StreamInfo};
|
||||
use windows::Win32::Graphics::Direct3D11::{
|
||||
D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0, D3D11_DECODER_PROFILE_H264_VLD_NOFGT, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10,
|
||||
D3D11_DECODER_PROFILE_VP9_VLD_10BIT_PROFILE2, D3D11_DECODER_PROFILE_VP9_VLD_PROFILE0,
|
||||
};
|
||||
use windows::Win32::Media::MediaFoundation::{MF_MT_MPEG2_PROFILE, MFVideoFormat_AV1, MFVideoFormat_H264, MFVideoFormat_HEVC, MFVideoFormat_VP90};
|
||||
use windows::core::GUID;
|
||||
|
||||
use super::gpu::SurfaceFormat;
|
||||
use super::mft::{Attr, Spec};
|
||||
use crate::annexb::to_annex_b;
|
||||
use crate::biplanar::Geometry;
|
||||
|
||||
/// Largest picture the backend takes (as the VideoToolbox one).
|
||||
pub const MAX_SIDE: u32 = 8192;
|
||||
|
||||
/// A stream the decoder takes.
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum Stream {
|
||||
/// H.264 / HEVC.
|
||||
Nal(NalStreamInfo),
|
||||
/// VP9 / AV1.
|
||||
Frame(FrameStreamInfo),
|
||||
}
|
||||
|
||||
/// What the GPU must decode for a stream.
|
||||
pub struct Plan {
|
||||
pub format: SurfaceFormat,
|
||||
pub profile: GUID,
|
||||
/// Size DXVA is asked about (the coded size).
|
||||
pub size: (u32, u32),
|
||||
}
|
||||
|
||||
impl From<StreamInfo> for Stream {
|
||||
fn from(i: StreamInfo) -> Self {
|
||||
match i {
|
||||
StreamInfo::Nal(n) => Self::Nal(n),
|
||||
StreamInfo::Frame(f) => Self::Frame(f),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<NalStreamInfo> for Stream {
|
||||
fn from(i: NalStreamInfo) -> Self {
|
||||
Self::Nal(i)
|
||||
}
|
||||
}
|
||||
|
||||
impl From<FrameStreamInfo> for Stream {
|
||||
fn from(i: FrameStreamInfo) -> Self {
|
||||
Self::Frame(i)
|
||||
}
|
||||
}
|
||||
|
||||
/// What the decoder needs for an `avcC` / `hvcC` stream, or why it is declined.
|
||||
pub fn plan_nal(info: &NalStreamInfo) -> Result<Plan, String> {
|
||||
if info.interlaced {
|
||||
return Err("field-coded H.264".into());
|
||||
}
|
||||
if info.chroma_format_idc != 1 {
|
||||
return Err(format!("chroma_format_idc {}", info.chroma_format_idc));
|
||||
}
|
||||
if info.bit_depth_luma != info.bit_depth_chroma || !matches!(info.bit_depth_luma, 8 | 10) {
|
||||
return Err(format!("{}-bit luma / {}-bit chroma", info.bit_depth_luma, info.bit_depth_chroma));
|
||||
}
|
||||
let (cx, cy, w, h) = info.crop;
|
||||
if w == 0 || h == 0 || w > MAX_SIDE || h > MAX_SIDE || cx.saturating_add(w) > info.coded.0 || cy.saturating_add(h) > info.coded.1 {
|
||||
return Err(format!("picture size {w}x{h}"));
|
||||
}
|
||||
let ten = info.bit_depth_luma == 10;
|
||||
let (format, profile) = match (info.codec, info.profile_idc, ten) {
|
||||
// Baseline, Main and High (the profiles DXVA H.264 decoders list)
|
||||
(NalCodec::H264, 66 | 77 | 100, false) => (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_H264_VLD_NOFGT),
|
||||
(NalCodec::H264, p, _) => return Err(format!("H.264 profile {p} at {} bits", info.bit_depth_luma)),
|
||||
// Main and Main Still Picture
|
||||
(NalCodec::Hevc, 1 | 3, false) => (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN),
|
||||
(NalCodec::Hevc, 2, true) => (SurfaceFormat::P010, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10),
|
||||
(NalCodec::Hevc, p, _) => return Err(format!("HEVC profile {p} at {} bits", info.bit_depth_luma)),
|
||||
};
|
||||
Ok(Plan { format, profile, size: info.coded })
|
||||
}
|
||||
|
||||
/// What the decoder needs for a VP9 / AV1 stream, or why it is declined: 4:2:0 only, 8-bit
|
||||
/// (NV12) or 10-bit (P010), the profiles DXVA lists (VP9 profile 0 and 2, AV1 main profile).
|
||||
pub fn plan_frame(info: &FrameStreamInfo) -> Result<Plan, String> {
|
||||
let (w, h) = info.size;
|
||||
if w == 0 || h == 0 || w > MAX_SIDE || h > MAX_SIDE {
|
||||
return Err(format!("picture size {w}x{h}"));
|
||||
}
|
||||
if info.mono || info.subsampling != (1, 1) {
|
||||
return Err(format!("chroma subsampling {:?}{}", info.subsampling, if info.mono { " (monochrome)" } else { "" }));
|
||||
}
|
||||
let (format, profile) = match (info.codec, info.profile, info.bit_depth) {
|
||||
(FrameCodec::Vp9, 0, 8) => (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_VP9_VLD_PROFILE0),
|
||||
(FrameCodec::Vp9, 2, 10) => (SurfaceFormat::P010, D3D11_DECODER_PROFILE_VP9_VLD_10BIT_PROFILE2),
|
||||
(FrameCodec::Vp9, p, b) => return Err(format!("VP9 profile {p} at {b} bits")),
|
||||
(FrameCodec::Av1, 0, 8) => (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0),
|
||||
(FrameCodec::Av1, 0, 10) => (SurfaceFormat::P010, D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0),
|
||||
(FrameCodec::Av1, p, b) => return Err(format!("AV1 profile {p} at {b} bits")),
|
||||
};
|
||||
Ok(Plan { format, profile, size: info.size })
|
||||
}
|
||||
|
||||
impl Stream {
|
||||
pub fn plan(&self) -> Result<Plan, String> {
|
||||
match self {
|
||||
Self::Nal(i) => plan_nal(i),
|
||||
Self::Frame(i) => plan_frame(i),
|
||||
}
|
||||
}
|
||||
|
||||
/// The hybrid decoder's view of the stream.
|
||||
pub fn info(&self) -> StreamInfo {
|
||||
match self {
|
||||
Self::Nal(i) => i.clone().into(),
|
||||
Self::Frame(i) => i.clone().into(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn name(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Nal(i) if i.codec == NalCodec::H264 => "Media Foundation H.264",
|
||||
Self::Nal(_) => "Media Foundation HEVC",
|
||||
Self::Frame(i) if i.codec == FrameCodec::Vp9 => "Media Foundation VP9",
|
||||
Self::Frame(_) => "Media Foundation AV1",
|
||||
}
|
||||
}
|
||||
|
||||
/// The decoder MFT request.
|
||||
pub fn spec(&self, format: SurfaceFormat) -> Spec {
|
||||
match self {
|
||||
Self::Nal(info) => {
|
||||
let (_, _, w, h) = info.crop;
|
||||
match info.codec {
|
||||
NalCodec::H264 => Spec { subtype: MFVideoFormat_H264, label: "H.264", missing: "", size: (w, h), format, attrs: Vec::new() },
|
||||
NalCodec::Hevc => Spec {
|
||||
subtype: MFVideoFormat_HEVC,
|
||||
label: "HEVC",
|
||||
missing: " (the HEVC Video Extensions are not installed)",
|
||||
size: (w, h),
|
||||
format,
|
||||
// the HEVC decoder lists P010 output only when told the stream is Main 10
|
||||
// (eAVEncH265VProfile_Main_420_8 = 1, eAVEncH265VProfile_Main_420_10 = 2)
|
||||
attrs: vec![(MF_MT_MPEG2_PROFILE, Attr::U32(if format == SurfaceFormat::P010 { 2 } else { 1 }))],
|
||||
},
|
||||
}
|
||||
}
|
||||
Self::Frame(info) => match info.codec {
|
||||
FrameCodec::Vp9 => Spec {
|
||||
subtype: MFVideoFormat_VP90,
|
||||
label: "VP9",
|
||||
missing: " (the VP9 Video Extensions are not installed)",
|
||||
size: info.size,
|
||||
format,
|
||||
attrs: Vec::new(),
|
||||
},
|
||||
FrameCodec::Av1 => Spec {
|
||||
subtype: MFVideoFormat_AV1,
|
||||
label: "AV1",
|
||||
missing: " (the AV1 Video Extension is not installed)",
|
||||
size: info.size,
|
||||
format,
|
||||
attrs: Vec::new(),
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// The geometry of the first picture: the stream's own (H.264 / HEVC), or the expected one
|
||||
/// until the first VP9 key frame says (colour and size come from its header).
|
||||
pub fn first_geometry(&self, format: SurfaceFormat) -> Geometry {
|
||||
match self {
|
||||
Self::Nal(i) => Geometry { crop: i.crop, bits: format.bits(), color: i.color, par: i.par },
|
||||
Self::Frame(i) => {
|
||||
Geometry { crop: (0, 0, i.size.0, i.size.1), bits: format.bits(), color: i.color.unwrap_or(filmcraft_color::ColorInfo::REC709), par: (1, 1) }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The even-rectangle test sharing a surface needs (see `MfDecoder::can_share`).
|
||||
pub fn picture_is_even(&self, g: &Geometry) -> bool {
|
||||
let (x, y, w, h) = g.crop;
|
||||
[x, y, w, h].iter().all(|v| v % 2 == 0)
|
||||
}
|
||||
|
||||
/// Whether the decoder can start (again) at `sample` after a drain: an IDR picture, a key frame.
|
||||
pub fn is_restart(&self, sample: &[u8]) -> bool {
|
||||
match self {
|
||||
Self::Nal(info) => info.nal_types(sample).iter().any(|&t| match info.codec {
|
||||
NalCodec::H264 => t == 5,
|
||||
NalCodec::Hevc => matches!(t, 19 | 20),
|
||||
}),
|
||||
Self::Frame(i) => i.is_random_access(sample),
|
||||
}
|
||||
}
|
||||
|
||||
/// `sample` as the MFT's input. With `headers` the stream's parameter sets / sequence header go
|
||||
/// first (unless the sample carries them): a decoder starting at this sample has what it needs.
|
||||
pub fn input(&self, sample: &[u8], headers: bool) -> Result<Vec<u8>, String> {
|
||||
match self {
|
||||
Self::Nal(info) => to_annex_b(info, sample, headers),
|
||||
Self::Frame(i) => {
|
||||
if sample.is_empty() {
|
||||
return Err("empty sample".into());
|
||||
}
|
||||
if headers && !i.config_obus.is_empty() && !i.carries_sequence_header(sample) {
|
||||
let mut v = Vec::with_capacity(i.config_obus.len() + sample.len());
|
||||
v.extend_from_slice(&i.config_obus);
|
||||
v.extend_from_slice(sample);
|
||||
return Ok(v);
|
||||
}
|
||||
Ok(sample.to_vec())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Picture format a VP9 key frame declares (size and colour); `None` for other samples and codecs.
|
||||
pub fn declared_picture(&self, sample: &[u8]) -> Option<Geometry> {
|
||||
let Self::Frame(i) = self else { return None };
|
||||
if i.codec != FrameCodec::Vp9 {
|
||||
return None;
|
||||
}
|
||||
let p = i.picture_params(sample)?;
|
||||
Some(Geometry { crop: (0, 0, p.size.0, p.size.1), bits: p.bit_depth, color: p.color, par: (1, 1) })
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn nal(codec: NalCodec, profile_idc: u8, bits: u32, chroma_format_idc: u32) -> NalStreamInfo {
|
||||
NalStreamInfo {
|
||||
codec,
|
||||
length_size: 4,
|
||||
highest_tid: None,
|
||||
parameter_sets: Vec::new(),
|
||||
coded: (1920, 1088),
|
||||
crop: (0, 0, 1920, 1080),
|
||||
chroma_format_idc,
|
||||
bit_depth_luma: bits,
|
||||
bit_depth_chroma: bits,
|
||||
interlaced: false,
|
||||
profile_idc,
|
||||
color: filmcraft_color::ColorInfo::REC709,
|
||||
par: (1, 1),
|
||||
reorder: 2,
|
||||
}
|
||||
}
|
||||
|
||||
fn frame(codec: FrameCodec, profile: u8, bits: u32, subsampling: (u8, u8)) -> FrameStreamInfo {
|
||||
FrameStreamInfo::for_tests(codec, profile, bits, subsampling, (1920, 1080))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn takes_h264_baseline_main_high_and_hevc_main_main10() {
|
||||
for p in [66, 77, 100] {
|
||||
assert_eq!(plan_nal(&nal(NalCodec::H264, p, 8, 1)).unwrap().format, SurfaceFormat::Nv12, "H.264 profile {p}");
|
||||
}
|
||||
let main = plan_nal(&nal(NalCodec::Hevc, 1, 8, 1)).unwrap();
|
||||
assert_eq!((main.format, main.profile), (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN));
|
||||
let main10 = plan_nal(&nal(NalCodec::Hevc, 2, 10, 1)).unwrap();
|
||||
assert_eq!((main10.format, main10.profile), (SurfaceFormat::P010, D3D11_DECODER_PROFILE_HEVC_VLD_MAIN10));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn declines_what_dxva_does_not_decode() {
|
||||
// Hi10, High 4:2:2 / 4:4:4, Extended; 10-bit under an 8-bit profile; 4:2:2 / 4:4:4 / mono HEVC
|
||||
for (codec, p, bits, chroma) in [
|
||||
(NalCodec::H264, 110, 10, 1),
|
||||
(NalCodec::H264, 122, 10, 2),
|
||||
(NalCodec::H264, 244, 8, 3),
|
||||
(NalCodec::H264, 88, 8, 1),
|
||||
(NalCodec::H264, 100, 10, 1),
|
||||
(NalCodec::H264, 100, 8, 2),
|
||||
(NalCodec::Hevc, 4, 10, 2),
|
||||
(NalCodec::Hevc, 4, 12, 1),
|
||||
(NalCodec::Hevc, 1, 10, 1),
|
||||
(NalCodec::Hevc, 2, 8, 1),
|
||||
(NalCodec::Hevc, 1, 8, 0),
|
||||
(NalCodec::Hevc, 1, 8, 3),
|
||||
] {
|
||||
assert!(plan_nal(&nal(codec, p, bits, chroma)).is_err(), "{codec:?} profile {p} {bits}-bit chroma {chroma}");
|
||||
}
|
||||
let mut field_coded = nal(NalCodec::H264, 100, 8, 1);
|
||||
field_coded.interlaced = true;
|
||||
assert!(plan_nal(&field_coded).is_err());
|
||||
// luma and chroma depths that differ
|
||||
let mut mixed = nal(NalCodec::Hevc, 2, 10, 1);
|
||||
mixed.bit_depth_chroma = 8;
|
||||
assert!(plan_nal(&mixed).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn declines_absurd_pictures() {
|
||||
for crop in [(0, 0, 0, 1080), (0, 0, 1920, 0), (0, 0, 9000, 1080), (0, 0, 1080, 9000), (8, 0, 1920, 1080), (0, 16, 1920, 1080), (u32::MAX, 0, 16, 16)] {
|
||||
let mut i = nal(NalCodec::H264, 100, 8, 1);
|
||||
i.crop = crop;
|
||||
assert!(plan_nal(&i).is_err(), "{crop:?}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn takes_vp9_profiles_0_and_2_and_av1_main_in_420() {
|
||||
let a = plan_frame(&frame(FrameCodec::Vp9, 0, 8, (1, 1))).unwrap();
|
||||
assert_eq!((a.format, a.profile), (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_VP9_VLD_PROFILE0));
|
||||
let b = plan_frame(&frame(FrameCodec::Vp9, 2, 10, (1, 1))).unwrap();
|
||||
assert_eq!((b.format, b.profile), (SurfaceFormat::P010, D3D11_DECODER_PROFILE_VP9_VLD_10BIT_PROFILE2));
|
||||
let c = plan_frame(&frame(FrameCodec::Av1, 0, 8, (1, 1))).unwrap();
|
||||
assert_eq!((c.format, c.profile), (SurfaceFormat::Nv12, D3D11_DECODER_PROFILE_AV1_VLD_PROFILE0));
|
||||
assert_eq!(plan_frame(&frame(FrameCodec::Av1, 0, 10, (1, 1))).unwrap().format, SurfaceFormat::P010);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn declines_the_rest_of_vp9_and_av1() {
|
||||
for (codec, p, bits, ss) in [
|
||||
(FrameCodec::Vp9, 1, 8, (0, 0)),
|
||||
(FrameCodec::Vp9, 1, 8, (1, 0)),
|
||||
(FrameCodec::Vp9, 3, 10, (0, 0)),
|
||||
(FrameCodec::Vp9, 2, 12, (1, 1)),
|
||||
(FrameCodec::Vp9, 0, 10, (1, 1)),
|
||||
(FrameCodec::Vp9, 2, 8, (1, 1)),
|
||||
(FrameCodec::Av1, 1, 8, (0, 0)),
|
||||
(FrameCodec::Av1, 2, 12, (1, 0)),
|
||||
(FrameCodec::Av1, 0, 12, (1, 1)),
|
||||
(FrameCodec::Av1, 0, 8, (1, 0)),
|
||||
] {
|
||||
assert!(plan_frame(&frame(codec, p, bits, ss)).is_err(), "{codec:?} profile {p} {bits}-bit {ss:?}");
|
||||
}
|
||||
let mut mono = frame(FrameCodec::Av1, 0, 8, (1, 1));
|
||||
mono.mono = true;
|
||||
assert!(plan_frame(&mono).is_err());
|
||||
for size in [(0, 1080), (1920, 0), (9000, 1080), (1920, 9000)] {
|
||||
let mut i = frame(FrameCodec::Vp9, 0, 8, (1, 1));
|
||||
i.size = size;
|
||||
assert!(plan_frame(&i).is_err(), "{size:?}");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -76,9 +76,11 @@ fn parity(name: &str, s: &Stream) {
|
||||
/// of a GOP the decoder reports an error (the GOP cache seeks after every flush, and the hybrid
|
||||
/// decoder would replay the run in software). After an IDR the pictures are the software decoder's.
|
||||
fn drained_decoder_restarts_only_at_idr(name: &str, s: &Stream) {
|
||||
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
|
||||
let is_idr = |i: usize| {
|
||||
info.nal_types(&s.samples[i].0).iter().any(|&t| if info.codec == filmcraft_codecs::hw::NalCodec::H264 { t == 5 } else { matches!(t, 19 | 20) })
|
||||
use filmcraft_codecs::hw::{NalCodec, StreamInfo};
|
||||
let info = filmcraft_platform::media_foundation::stream_info(&s.entry).unwrap();
|
||||
let is_idr = |i: usize| match &info {
|
||||
StreamInfo::Nal(n) => n.nal_types(&s.samples[i].0).iter().any(|&t| if n.codec == NalCodec::H264 { t == 5 } else { matches!(t, 19 | 20) }),
|
||||
StreamInfo::Frame(f) => f.is_random_access(&s.samples[i].0),
|
||||
};
|
||||
let Ok(mut mf) = MfDecoder::new(info.clone()) else { return };
|
||||
let mid = s.samples.len() / 2;
|
||||
@@ -101,12 +103,28 @@ fn drained_decoder_restarts_only_at_idr(name: &str, s: &Stream) {
|
||||
#[test]
|
||||
fn bit_exact_with_the_software_decoders() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
parity(name, &read_stream(&path));
|
||||
for (name, s) in small_streams(&ff) {
|
||||
parity(&name, &s);
|
||||
}
|
||||
}
|
||||
|
||||
/// The small parity fixtures: H.264 High, HEVC Main (open GOP) and Main 10, VP9 profiles 0 / 2 and
|
||||
/// AV1 8- / 10-bit (640x360, B-frames or hidden reference frames, two GOPs or more).
|
||||
fn small_streams(ff: &std::path::Path) -> Vec<(String, Stream)> {
|
||||
let mut out = Vec::new();
|
||||
for (name, _) in FIXTURES {
|
||||
if let Some(path) = named(ff, name) {
|
||||
out.push((name.to_string(), read_stream(&path)));
|
||||
}
|
||||
}
|
||||
for codec in ["vp9", "vp9p2", "av1", "av1p10"] {
|
||||
if let Some(path) = frame_fixture(ff, codec, "640x360") {
|
||||
out.push((format!("{codec} 640x360"), read_stream(&path)));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// 1920x1080 and 3840x2160 with B-frames and two GOPs: H.264 High, HEVC Main (open GOP) and
|
||||
/// HEVC Main 10.
|
||||
fn large_fixture(ff: &std::path::Path, codec: &str, size: &str) -> Option<std::path::PathBuf> {
|
||||
@@ -204,15 +222,15 @@ fn hevc_main10_1080p_and_2160p_are_bit_exact() {
|
||||
#[test]
|
||||
fn mid_stream_failure_continues_in_software() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
let s = read_stream(&path);
|
||||
let info = NalStreamInfo::from_entry(&s.entry).unwrap().unwrap();
|
||||
for (name, s) in small_streams(&ff) {
|
||||
let name = name.as_str();
|
||||
let info = filmcraft_platform::media_foundation::stream_info(&s.entry).unwrap();
|
||||
let Ok(_) = MfDecoder::new(info.clone()) else {
|
||||
eprintln!("SKIPPED: no Media Foundation hardware decoder for {name}");
|
||||
continue;
|
||||
};
|
||||
let reference = decode_all(software(&s).as_mut(), &s.samples);
|
||||
assert!(!reference.is_empty(), "{name}");
|
||||
// fail at the first sample, inside the first GOP, at a sync sample and just after one
|
||||
let k = (1..s.samples.len()).find(|&i| s.sync[i]).unwrap();
|
||||
for fail_at in [0, 5, k, k + 1, k + 9] {
|
||||
@@ -248,9 +266,8 @@ fn bounded(what: &str, limit: Duration, f: impl FnOnce() + Send + 'static) {
|
||||
#[test]
|
||||
fn damaged_samples_and_parameter_sets_never_crash_or_hang() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for (name, _) in FIXTURES {
|
||||
let Some(path) = named(&ff, name) else { continue };
|
||||
let s = read_stream(&path);
|
||||
for (name, s) in small_streams(&ff) {
|
||||
let name = name.as_str();
|
||||
if hardware(&s).is_none() {
|
||||
continue;
|
||||
}
|
||||
@@ -288,6 +305,10 @@ fn damaged_samples_and_parameter_sets_never_crash_or_hang() {
|
||||
let rec = match &s.entry.codec {
|
||||
filmcraft_isobmff::CodecConfig::Avc(a) => a.to_bytes(),
|
||||
filmcraft_isobmff::CodecConfig::Hevc(c) => c.to_bytes(),
|
||||
filmcraft_isobmff::CodecConfig::Av1(c) => {
|
||||
damaged_av1c(name, &s, c);
|
||||
continue;
|
||||
}
|
||||
_ => continue,
|
||||
};
|
||||
let hevc = matches!(s.entry.codec, filmcraft_isobmff::CodecConfig::Hevc(_));
|
||||
@@ -423,3 +444,210 @@ fn decoder_moves_between_threads() {
|
||||
let out = std::thread::spawn(move || decode_all(hw.as_mut(), &samples[..30])).join().unwrap();
|
||||
assert_same("second thread", &out, &reference);
|
||||
}
|
||||
|
||||
// ---- VP9 and AV1 (frame-coded streams: vpcC / av1C) ----
|
||||
|
||||
/// A VP9 / AV1 fixture with hidden reference frames (VP9 alt-ref superframes, AV1 alt-ref and
|
||||
/// show_existing_frame), two GOPs: `codec` is `vp9`, `vp9p2` (10-bit), `av1` or `av1p10`.
|
||||
fn frame_fixture(ff: &std::path::Path, codec: &str, size: &str) -> Option<std::path::PathBuf> {
|
||||
let (_, h) = size.split_once('x')?;
|
||||
let src = format!("testsrc2=s={size}:r=24:d=2,noise=alls=12:allf=t");
|
||||
let (name, args): (String, Vec<&str>) = match codec {
|
||||
"vp9" => (
|
||||
format!("vp9_p0_{h}.mp4"),
|
||||
vec![
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
&src,
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-profile:v",
|
||||
"0",
|
||||
"-b:v",
|
||||
"0",
|
||||
"-crf",
|
||||
"32",
|
||||
"-deadline",
|
||||
"good",
|
||||
"-cpu-used",
|
||||
"8",
|
||||
"-g",
|
||||
"24",
|
||||
"-auto-alt-ref",
|
||||
"1",
|
||||
"-lag-in-frames",
|
||||
"8",
|
||||
"-row-mt",
|
||||
"1",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
),
|
||||
"vp9p2" => (
|
||||
format!("vp9_p2_{h}.mp4"),
|
||||
vec![
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
&src,
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-profile:v",
|
||||
"2",
|
||||
"-b:v",
|
||||
"0",
|
||||
"-crf",
|
||||
"32",
|
||||
"-deadline",
|
||||
"good",
|
||||
"-cpu-used",
|
||||
"8",
|
||||
"-g",
|
||||
"24",
|
||||
"-auto-alt-ref",
|
||||
"1",
|
||||
"-lag-in-frames",
|
||||
"8",
|
||||
"-row-mt",
|
||||
"1",
|
||||
"-pix_fmt",
|
||||
"yuv420p10le",
|
||||
],
|
||||
),
|
||||
"av1" => (
|
||||
format!("av1_p0_{h}.mp4"),
|
||||
vec![
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
&src,
|
||||
"-c:v",
|
||||
"libaom-av1",
|
||||
"-crf",
|
||||
"34",
|
||||
"-b:v",
|
||||
"0",
|
||||
"-cpu-used",
|
||||
"8",
|
||||
"-row-mt",
|
||||
"1",
|
||||
"-g",
|
||||
"24",
|
||||
"-lag-in-frames",
|
||||
"8",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
),
|
||||
_ => (
|
||||
format!("av1_p0_10bit_{h}.mp4"),
|
||||
vec![
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
&src,
|
||||
"-c:v",
|
||||
"libaom-av1",
|
||||
"-crf",
|
||||
"34",
|
||||
"-b:v",
|
||||
"0",
|
||||
"-cpu-used",
|
||||
"8",
|
||||
"-row-mt",
|
||||
"1",
|
||||
"-g",
|
||||
"24",
|
||||
"-lag-in-frames",
|
||||
"8",
|
||||
"-pix_fmt",
|
||||
"yuv420p10le",
|
||||
],
|
||||
),
|
||||
};
|
||||
fixture(ff, &name, &args)
|
||||
}
|
||||
|
||||
fn frame_parity(codec: &str, sizes: &[&str]) {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for size in sizes {
|
||||
let Some(path) = frame_fixture(&ff, codec, size) else { continue };
|
||||
parity(&format!("{codec} {size}"), &read_stream(&path));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn vp9_profile_0_is_bit_exact() {
|
||||
frame_parity("vp9", &["640x360", "1920x1080", "3840x2160"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn vp9_profile_2_10bit_is_bit_exact() {
|
||||
frame_parity("vp9p2", &["640x360", "1920x1080", "3840x2160"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn av1_8bit_is_bit_exact() {
|
||||
frame_parity("av1", &["640x360", "1920x1080", "3840x2160"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn av1_10bit_is_bit_exact() {
|
||||
frame_parity("av1p10", &["640x360", "1920x1080", "3840x2160"]);
|
||||
}
|
||||
|
||||
/// Damaged `av1C` sequence headers (bit flips, truncation): the backend declines or the decoder
|
||||
/// errors / falls back, never crashes or hangs.
|
||||
fn damaged_av1c(name: &str, s: &Stream, c: &filmcraft_isobmff::Av1Config) {
|
||||
let mut seed = 0x9E37_79B9u64;
|
||||
for round in 0..24 {
|
||||
let mut cfg = c.clone();
|
||||
if round % 3 == 2 {
|
||||
cfg.config_obus.truncate(cfg.config_obus.len() * (round + 1) / 30);
|
||||
} else {
|
||||
for _ in 0..1 + round % 4 {
|
||||
let at = (xorshift(&mut seed) as usize) % cfg.config_obus.len().max(1);
|
||||
if let Some(b) = cfg.config_obus.get_mut(at) {
|
||||
*b ^= 1 << (xorshift(&mut seed) % 8);
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut entry = s.entry.clone();
|
||||
entry.codec = filmcraft_isobmff::CodecConfig::Av1(cfg);
|
||||
let samples: Vec<_> = s.samples.iter().take(12).cloned().collect();
|
||||
bounded(&format!("{name} av1C round {round}"), Duration::from_secs(60), move || {
|
||||
let Some(info) = filmcraft_platform::media_foundation::stream_info(&entry) else { return };
|
||||
let Ok(mut d) = MfDecoder::new(info) else { return };
|
||||
for (smp, pts) in &samples {
|
||||
if d.decode(smp, *pts).is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let _ = d.flush();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// A key frame of another size (VP9) / a sequence header unlike the `av1C` one (AV1) appearing in
|
||||
/// the middle of a stream is something the hardware session was not set up for: the hybrid decoder
|
||||
/// continues with the software decoder, whose output is the reference.
|
||||
#[test]
|
||||
fn in_band_format_changes_fall_back_to_software() {
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
for codec in ["vp9", "av1"] {
|
||||
let (Some(big), Some(small)) = (frame_fixture(&ff, codec, "1920x1080"), frame_fixture(&ff, codec, "640x360")) else { continue };
|
||||
let (a, b) = (read_stream(&big), read_stream(&small));
|
||||
let Some(mut hw) = hardware(&a) else { continue };
|
||||
let mut sw = software(&a);
|
||||
// (the second stream's pictures come later in time, as in a real stream)
|
||||
let samples: Vec<_> = a.samples[..8].iter().cloned().chain(b.samples[..8].iter().map(|(d, p)| (d.clone(), p + 100_000))).collect();
|
||||
let before = filmcraft_codecs::hw::hw_stats().fallbacks;
|
||||
let got = decode_all(hw.as_mut(), &samples);
|
||||
let want = decode_all(sw.as_mut(), &samples);
|
||||
assert!(filmcraft_codecs::hw::hw_stats().fallbacks > before, "{codec}: the change was counted as a fallback");
|
||||
assert!(!hw.name().starts_with("Media Foundation"), "{codec}: continued in {}", hw.name());
|
||||
assert_same(&format!("{codec} 1080p then 360p"), &got, &want);
|
||||
assert!(got.iter().any(|f| f.frame.width == 640) && got.iter().any(|f| f.frame.width == 1920), "{codec}: both sizes came out");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
//! VP9 and AV1 in WebM / Matroska through the media stack with Settings > Playback > Hardware
|
||||
//! decoding Auto and Off (Windows). Its own test binary: the switch is process-wide.
|
||||
#![cfg(target_os = "windows")]
|
||||
|
||||
mod common;
|
||||
|
||||
use common::*;
|
||||
|
||||
/// WebM / Matroska VP9 and AV1 through the media stack: the same pictures with hardware decoding
|
||||
/// Auto as with it Off, and the hardware decoder really ran.
|
||||
#[test]
|
||||
fn webm_and_mkv_match_software() {
|
||||
use filmcraft_media::FrameRequest;
|
||||
let ff = filmcraft_testkit::require_ffmpeg!();
|
||||
let src = "testsrc2=s=640x360:r=24:d=2,noise=alls=12:allf=t";
|
||||
let cases: [(&str, Vec<&str>); 2] = [
|
||||
(
|
||||
"vp9.webm",
|
||||
vec![
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
src,
|
||||
"-c:v",
|
||||
"libvpx-vp9",
|
||||
"-b:v",
|
||||
"0",
|
||||
"-crf",
|
||||
"32",
|
||||
"-deadline",
|
||||
"good",
|
||||
"-cpu-used",
|
||||
"8",
|
||||
"-g",
|
||||
"24",
|
||||
"-auto-alt-ref",
|
||||
"1",
|
||||
"-lag-in-frames",
|
||||
"8",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
),
|
||||
(
|
||||
"av1.mkv",
|
||||
vec![
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
src,
|
||||
"-c:v",
|
||||
"libaom-av1",
|
||||
"-crf",
|
||||
"34",
|
||||
"-b:v",
|
||||
"0",
|
||||
"-cpu-used",
|
||||
"8",
|
||||
"-g",
|
||||
"24",
|
||||
"-lag-in-frames",
|
||||
"8",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
],
|
||||
),
|
||||
];
|
||||
filmcraft_platform::register();
|
||||
for (name, args) in cases {
|
||||
let Some(path) = fixture(&ff, name, &args) else { continue };
|
||||
let bytes: std::sync::Arc<[u8]> = std::fs::read(&path).unwrap().into();
|
||||
let frames = |hw: bool| {
|
||||
filmcraft_codecs::hw::set_hardware_decoding(hw);
|
||||
let src = filmcraft_codecs::open_bytes(name, bytes.clone()).unwrap();
|
||||
let rate = src.info().frame_rate();
|
||||
(0..40).map(|i| src.video_frame(FrameRequest::full(rate.tick_of(i))).unwrap()).collect::<Vec<_>>()
|
||||
};
|
||||
let off = frames(false);
|
||||
let before = filmcraft_codecs::hw::hw_stats();
|
||||
let auto = frames(true);
|
||||
let after = filmcraft_codecs::hw::hw_stats();
|
||||
filmcraft_codecs::hw::set_hardware_decoding(true);
|
||||
if after.sessions == before.sessions {
|
||||
eprintln!("SKIPPED: no Media Foundation decoder for {name}");
|
||||
continue;
|
||||
}
|
||||
assert!(after.frames > before.frames, "{name}: hardware frames");
|
||||
for (i, (a, b)) in auto.iter().zip(&off).enumerate() {
|
||||
assert_eq!((a.width, a.height, a.color, a.par), (b.width, b.height, b.color, b.par), "{name} frame {i}");
|
||||
let (x, y) = (a.as_ref().clone(), b.as_ref().clone());
|
||||
assert_same(
|
||||
&format!("{name} frame {i}"),
|
||||
&[filmcraft_codecs::DecodedFrame { pts: 0, frame: x, draft: false }],
|
||||
&[filmcraft_codecs::DecodedFrame { pts: 0, frame: y, draft: false }],
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,8 @@ const X265_MAIN10: &[&str] = &[
|
||||
"-x265-params",
|
||||
"log-level=error",
|
||||
];
|
||||
/// AV1 where ffmpeg has libaom but not SVT-AV1 (the Windows builds): slower to encode, same role.
|
||||
const AV1_AOM: &[&str] = &["-c:v", "libaom-av1", "-crf", "35", "-b:v", "0", "-cpu-used", "8", "-row-mt", "1", "-g", "48", "-pix_fmt", "yuv420p"];
|
||||
const PRORES: &[&str] = &["-c:v", "prores_ks", "-profile:v", "3", "-pix_fmt", "yuv422p10le", "-vendor", "apl0"];
|
||||
|
||||
const SPECS: &[Spec] = &[
|
||||
@@ -78,6 +80,11 @@ const SPECS: &[Spec] = &[
|
||||
Spec { name: "clip360.mp4", src: "testsrc2=s=640x360:r=24000/1001:d=20", secs: 20, audio: true, codec: X264 },
|
||||
];
|
||||
|
||||
/// Whether this ffmpeg lists `encoder`.
|
||||
fn has_encoder(ffmpeg: &Path, encoder: &str) -> bool {
|
||||
Command::new(ffmpeg).args(["-hide_banner", "-encoders"]).output().is_ok_and(|o| String::from_utf8_lossy(&o.stdout).contains(encoder))
|
||||
}
|
||||
|
||||
/// The fixture `name`, generating it on first use (None when ffmpeg or its encoder is missing).
|
||||
pub fn fixture(name: &str) -> Option<PathBuf> {
|
||||
let dir = fixtures_dir();
|
||||
@@ -87,6 +94,7 @@ pub fn fixture(name: &str) -> Option<PathBuf> {
|
||||
}
|
||||
let ff = ffmpeg()?;
|
||||
let spec = SPECS.iter().find(|f| f.name == name)?;
|
||||
let codec = if std::ptr::eq(spec.codec, AV1) && !has_encoder(&ff, "libsvtav1") { AV1_AOM } else { spec.codec };
|
||||
std::fs::create_dir_all(&dir).ok()?;
|
||||
eprintln!("generating {}", path.display());
|
||||
// Write under a temporary name and rename, so a concurrent run never sees half a file.
|
||||
@@ -98,7 +106,7 @@ pub fn fixture(name: &str) -> Option<PathBuf> {
|
||||
if spec.audio {
|
||||
c.args(["-f", "lavfi", "-i", &format!("sine=f=440:d={}", spec.secs), "-c:a", "aac", "-shortest"]);
|
||||
}
|
||||
c.args(spec.codec);
|
||||
c.args(codec);
|
||||
if ext != "webm" {
|
||||
c.args(["-movflags", "+faststart"]);
|
||||
}
|
||||
|
||||
@@ -254,7 +254,8 @@ file ──► codecs (MP4/MOV, MKV, audio) demux + decode, GOP-aware see
|
||||
samples): the `avcC` / `hvcC` sample becomes Annex B, the MFT is given the process's Direct3D 11
|
||||
video device through an `IMFDXGIDeviceManager` so it decodes with DXVA, and its NV12 (8-bit) /
|
||||
P010 (10-bit) texture is read back through a staging texture (the one GPU to CPU copy) into
|
||||
planar `Yuv8` / `Yuv16`. H.264 Baseline / Main / High and HEVC Main / Main 10, 4:2:0, progressive;
|
||||
planar `Yuv8` / `Yuv16`. H.264 Baseline / Main / High, HEVC Main / Main 10, VP9 profiles 0 / 2 and AV1 main
|
||||
(8- and 10-bit), 4:2:0, progressive (VP9 / AV1 samples go in as they are, not as Annex B);
|
||||
everything else, and any stream the GPU's DXVA decoder does not list, is declined. A decoder that
|
||||
would hand back system-memory pictures (Microsoft's decoder MFTs do that when DXVA is not
|
||||
available) fails the stream, so the hybrid continues with our decoder and Windows' software
|
||||
|
||||
@@ -301,6 +301,34 @@ frame, about 8 cores busy); Auto uses a twentieth of that CPU but drops 2–3 of
|
||||
Draft playback (1/2 draft: 410–416 → 93–105 CPU ms/frame) is dominated by the draft plan's decimation,
|
||||
as on macOS, because the hardware ignores draft mode.
|
||||
|
||||
## Results (HW2 follow-up: Windows VP9 and AV1 hardware decoding, Off → Auto)
|
||||
|
||||
Same machine and method as the H.264 / HEVC results above (Xeon E5-2680 v4, RTX 5060 driver 617.14,
|
||||
idle, 2026-10-07, two alternating rounds of `--hw off` / `--hw auto`, the same binary). MFTs:
|
||||
`VP9VideoExtensionDecoder` and `AV1VideoExtension` (Microsoft Store codec extensions), DXVA on the
|
||||
GPU's NVDEC (profiles VP9 0 / 2, AV1 main). AV1 fixtures here are libaom (`libsvtav1` is not in this
|
||||
ffmpeg build; the bench falls back to it) and VP9 are libvpx-vp9, as in the bench's `dec_*` specs.
|
||||
`cargo xtask bench --sections decode --only dec_vp9 --repeat 3 --hw off|auto` (likewise `dec_av1`),
|
||||
`--sections playback --only "av1-2160 full" --hw off|auto`.
|
||||
|
||||
| codec | size | CPU ms/frame Off → **Auto** | fps Off → **Auto** (best of 3) |
|
||||
|---|---|---|---|
|
||||
| VP9 | 1080p | 38-43 → **3.5-3.6** | 60 → **281-283** |
|
||||
| VP9 | 2160p | 165-169 → **15-16** | 16 → **70-79** |
|
||||
| AV1 | 1080p | 55-56 → **3.3-3.4** | 22 → **373-374** |
|
||||
| AV1 | 2160p | 219-221 → **10.9-11.3** | 5.5 → **106-107** |
|
||||
|
||||
Every Auto row had `hw frames` = frames × 3 repeats, 3 sessions, 0 fallbacks, 0 declined.
|
||||
|
||||
| playback 8 s, Full | shown/dropped Off → **Auto** | CPU ms/frame Off → **Auto** |
|
||||
|---|---|---|
|
||||
| VP9 2160p | 11-12/180-181 → **191/1** | 117-124 → **24-27** |
|
||||
| AV1 2160p | 0/192 → **192/0** | 47.5-47.9 → **17-20** |
|
||||
|
||||
Software AV1 decodes 4K at 5.5 fps here, so it never plays in real time; with the hardware decoder it
|
||||
plays without a drop. As on the other codecs, the hardware ignores draft mode and what is left on the
|
||||
CPU is the readback and plane conversion.
|
||||
|
||||
## Results (GPU1: blend modes on the GPU compositor, #30, before → after)
|
||||
|
||||
Before = this change with the old whole-frame CPU fallback for non-Normal blend modes put back
|
||||
|
||||
+2
-1
@@ -68,7 +68,8 @@ are never linked or shipped ([AGENTS.md](../AGENTS.md) §2).
|
||||
H.264 and HEVC fixtures need ffmpeg built with libx264 and libx265. VideoToolbox fixtures are
|
||||
generated only on macOS. The Media Foundation parity tests (`crates/platform/tests/media_foundation.rs`,
|
||||
Windows only) make their 640x360, 1080p and 2160p H.264 / HEVC / Main 10 fixtures with ffmpeg and skip
|
||||
without a Direct3D 11 video device or the HEVC Video Extensions.
|
||||
without a Direct3D 11 video device or the HEVC / VP9 / AV1 codec extensions of the Microsoft Store. The VP9 and
|
||||
AV1 fixtures (libvpx-vp9, libaom-av1; 360p, 1080p, 2160p, hidden alt-ref frames, two GOPs) need an ffmpeg with those encoders.
|
||||
|
||||
### Pass criteria per codec
|
||||
|
||||
|
||||
Reference in new issue
Block a user