M: video and sound in interactive PDF (screen annotations, renditions)

Co-Authored-By: Claude Opus 5.5 <[email protected]>

ROADMAP.md

@@ -95,7 +95,7 @@ DesignCraft aims at full Adobe InDesign parity — and to be better: faster, ope
 - Liquid Layout (`liquid.pageRule/object`, `guide.liquid`, `layout.createAlternate`, Liquid Layout panel): scale / re-center / guide-based / object-based page rules applied on page resize and Document Setup; dashed liquid guides; alternate layouts as resized page copies in a named section. Text in an alternate layout stays linked to the original stories (Links panel shows them out of date; `story.updateLink`).
 - Split Window / New Window (`window.split`, `window.newWindow`, Window ▸ Arrange): two side-by-side views of the document with their own zoom, scroll and render cache. New Window puts the second view in its own OS window (an egui viewport; a floating window on the web).
 - Transparency flattener (`file.exportPdf {flatten: high|medium|low|ppi}`, Edit ▸ Transparency Flattener Presets): each object involving transparency becomes an opaque image of its area over what's beneath it, so everything else stays vector (spreads with transparent parent items are rasterised whole).
-- Video and sound (place .mp4/.mov/.webm/.mp3/.wav…, `media.options/get`, Media panel): media frames show a poster or placeholder on screen, in print and in PDF, and export as HTML5 `<video>`/`<audio>` in EPUB. No in-app playback or PDF rich media.
+- Video and sound (place .mp4/.mov/.webm/.mp3/.wav…, `media.options/get`, Media panel): media frames show a poster or placeholder on screen, in print and in PDF, and export as HTML5 `<video>`/`<audio>` in EPUB. Interactive PDF export (`file.exportPdf {media: true}`, File ▸ Export PDF (Interactive)) embeds each file and plays it in a Screen annotation through a rendition action (controls, loop, play on page load). No in-app playback.
 - Right-to-left text (World-Ready basics): right-to-left runs are shaped right to left, lines are reordered by the Unicode Bidirectional Algorithm (matched against the reference implementation), Paragraph Direction (`type.para {direction}`, Paragraph panel, IDML ParagraphDirection, EPUB `dir`). Digits (`digits`: Default, Arabic, Hindi, Farsi, Native by language; Character panel for right-to-left text; IDML DigitsType) draw 0–9 in another script. Kashidas (`kashidas` paragraph attribute, on by default): justified Arabic lines stretch at one join per word (after seen/sad when possible, never inside lam–alef) with a drawn tatweel before word spaces widen. Carets follow the drawing: the caret before a right-to-left glyph sits at its right edge, arrow keys move as drawn (by words in the line's direction), clicks find the nearest drawn caret position and selections highlight the selected glyphs. Binding: Right to Left (Document Setup, `layout.documentSetup {binding}`, IDML PageBinding) makes page 1 a left page and lays spreads out right to left.
 - Interface language (Edit ▸ Interface Language, `app.language`): German, French, Spanish and Japanese menu titles, common menu items and panel names (our own translations); dialogs and the rest of the UI stay English.
 - Page transitions (`page.transition/transitions`, Page Transitions panel): twelve transition types with speed and direction per spread, written into exported PDFs as `/Trans` entries through an incremental update.

crates/engine/src/cmd/export.rs

@@ -9,7 +9,7 @@ use crate::{EngineError, Result, Session};
 pub fn specs() -> Vec<CommandSpec> {
     vec![
         cmd!(noundo "file.exportPdf", "Export PDF…", ["File"], None,
-        "{path?, pages?: \"1-3,5\" | [1,3] (1-based positions; default all), flatten?: high|medium|low|ppi (Transparency Flattener: spreads with transparency are rasterised), spreads?: bool, fullScreen?: bool, bookmarksPanel?: bool, pageLayout?: single|continuous|twoUp|twoUpCover|twoUpContinuous, view?: fitPage|fitWidth|actual, advanceSeconds?: number (interactive PDF), bleed?: bool (document bleed), marks?: bool | {crop?, bleed?, pageInfo?, weight?, offset?}, standard?: \"none\"|\"x4\"|\"a2b\", compressImages?: bool, tagged?: bool (structure tree: stories as paragraphs, figures with alt text), title?, author?} → {path, bytes, pages, warnings} (no path: {base64, …})",
+        "{path?, pages?: \"1-3,5\" | [1,3] (1-based positions; default all), flatten?: high|medium|low|ppi (Transparency Flattener: spreads with transparency are rasterised), spreads?: bool, fullScreen?: bool, bookmarksPanel?: bool, pageLayout?: single|continuous|twoUp|twoUpCover|twoUpContinuous, view?: fitPage|fitWidth|actual, advanceSeconds?: number (interactive PDF), bleed?: bool (document bleed), marks?: bool | {crop?, bleed?, pageInfo?, weight?, offset?}, standard?: \"none\"|\"x4\"|\"a2b\", compressImages?: bool, tagged?: bool (structure tree: stories as paragraphs, figures with alt text), media?: bool (interactive: embed placed video and sound), title?, author?} → {path, bytes, pages, warnings} (no path: {base64, …})",
         has_doc, export_pdf),
         cmd!(noundo "file.exportEpub", "Export EPUB (Reflowable)…", ["File"], None,
         "{path?, title?, author?, language?: \"en\", cover?: bool (the first page as the cover image), fixedLayout?: bool (pre-paginated: each page as an image with its text)} → {path, bytes} (no path: {base64, bytes})",
@@ -219,6 +219,7 @@ pub(crate) fn options(p: &Value, page_count: usize) -> Result<PdfOptions> {
         title: str_param(p, "title").map(str::to_string),
         author: str_param(p, "author").map(str::to_string),
         tagged: p.get("tagged").and_then(Value::as_bool).unwrap_or(false),
+        media: p.get("media").and_then(Value::as_bool).unwrap_or(false),
         ..PdfOptions::default()
     })
 }

crates/engine/src/cmd/media.rs

@@ -170,8 +170,24 @@ mod tests {
         let img = rr.render_page(&d, &s.cache, 0, 1.0, false, &Default::default()).unwrap();
         let p = img.pixel(110, 110);
         assert!(p[0] > 200 && p[1] < 60, "poster: {p:?}");
-        // PDF export draws it; EPUB carries the video.
-        assert!(s.execute("file.exportPdf", &json!({})).is_ok());
+        // PDF export draws it; interactive PDF embeds the video and plays it in a screen annotation.
+        let pdf = |s: &mut Session, media: bool| {
+            let r = s.execute("file.exportPdf", &json!({"media": media})).unwrap();
+            super::super::file::base64_decode(r["base64"].as_str().unwrap())
+        };
+        let plain = String::from_utf8_lossy(&pdf(&mut s, false)).to_string();
+        assert!(!plain.contains("/Subtype/Screen"));
+        let rich = pdf(&mut s, true);
+        let t = String::from_utf8_lossy(&rich);
+        assert!(
+            t.contains("/Subtype/Screen") && t.contains("/S/Rendition") && t.contains("/CT(video/mp4)") && t.contains("/RC 0"),
+            "screen + rendition"
+        );
+        assert!(t.contains("/Type/EmbeddedFile/Subtype/video#2Fmp4") && t.contains("ftypmp42"), "the file is embedded");
+        assert!(!t.contains("/AcroForm"), "no form without fields");
+        // Still a readable PDF.
+        assert_eq!(hayro_syntax::Pdf::new(rich.clone()).expect("valid PDF").pages().len(), 1);
+        // EPUB carries the video.
         let e = s.execute("file.exportEpub", &json!({})).unwrap();
         let bytes = super::super::file::base64_decode(e["base64"].as_str().unwrap());
         let mut z = zip::ZipArchive::new(std::io::Cursor::new(bytes)).unwrap();

crates/pdf/src/export.rs

@@ -202,15 +202,16 @@ pub fn export_pdf_with_report(doc: &Document, cache: &Cache, opts: &PdfOptions)
     }
     let mut bytes = pdf.finish().map_err(|e| PdfError::Write(format!("{e:?}")))?;
     let mut warnings = ex.warnings;
-    // Form fields (Buttons and Forms), not in PDF/X.
+    // Form fields (Buttons and Forms) and, for interactive PDF, video and sound; not in PDF/X.
     let fields = crate::forms::collect(doc, &sheets);
-    if !fields.is_empty() {
+    let media = if opts.media { crate::forms::collect_media(doc, &sheets) } else { Vec::new() };
+    if !fields.is_empty() || !media.is_empty() {
         if opts.standard == Standard::PdfX4 {
-            warnings.push("PDF/X-4: form fields were left out".into());
+            warnings.push("PDF/X-4: form fields and media were left out".into());
         } else {
-            match crate::forms::add_fields(&bytes, &fields) {
+            match crate::forms::add_fields(&bytes, &fields, &media) {
                 Some(b) => bytes = b,
-                None => warnings.push("form fields couldn't be added to this PDF".into()),
+                None => warnings.push("form fields and media couldn't be added to this PDF".into()),
             }
         }
     }

crates/pdf/src/forms.rs

@@ -1,10 +1,11 @@
 //! Interactive PDF forms (Buttons and Forms: text fields, check boxes, combo and list boxes,
-//! signature fields). krilla doesn't write AcroForm fields, so they're added as an incremental
-//! update: widget annotations on their pages, the page `/Annots` and the catalog `/AcroForm`.
+//! signature fields) and media (placed video and sound). krilla writes neither, so they're added
+//! as an incremental update: widget and screen annotations on their pages, the page `/Annots`, the
+//! catalog `/AcroForm`, and the media files embedded for rendition actions.
 
 use std::fmt::Write as _;
 
-use designcraft_doc::{Document, FieldKind, FormField, Item, SpreadRef};
+use designcraft_doc::{Document, FieldKind, FormField, Item, MediaOptions, SpreadRef};
 use designcraft_geom::Affine;
 
 use crate::export::Sheet;
@@ -13,15 +14,17 @@ use crate::transitions::{int_after, object, rfind};
 /// A field placed on an output page: (page index, PDF rectangle, field).
 pub(crate) type Placed = (usize, [f64; 4], FormField);
 
-/// The form fields of every exported page, in PDF coordinates (origin bottom left).
-pub(crate) fn collect(doc: &Document, sheets: &[Sheet]) -> Vec<Placed> {
-    let mut out = Vec::new();
+/// A placed video or sound on an output page: (page index, PDF rectangle, file name, MIME type,
+/// bytes, options).
+pub(crate) type PlacedMedia = (usize, [f64; 4], String, String, std::sync::Arc<Vec<u8>>, MediaOptions);
+
+/// Every shown item of the exported pages with its PDF rectangle (origin bottom left).
+fn visit(doc: &Document, sheets: &[Sheet], mut f: impl FnMut(usize, [f64; 4], &Item)) {
     for (pi, sh) in sheets.iter().enumerate() {
         let Some(sp) = doc.spread(SpreadRef::Doc(sh.spread)) else { continue };
         let mut stack: Vec<(Affine, &std::sync::Arc<Item>)> = sp.items.iter().map(|it| (Affine::IDENTITY, it)).collect();
         while let Some((xf, it)) = stack.pop() {
             stack.extend(it.shown_children().map(|c| (xf * it.xf, c)));
-            let Some(f) = &it.form_field else { continue };
             if it.hidden {
                 continue;
             }
@@ -30,9 +33,32 @@ pub(crate) fn collect(doc: &Document, sheets: &[Sheet]) -> Vec<Placed> {
                 continue;
             }
             let h = sh.media.height();
-            out.push((pi, [r.x0 - sh.media.x0, h - (r.y1 - sh.media.y0), r.x1 - sh.media.x0, h - (r.y0 - sh.media.y0)], f.clone()));
+            f(pi, [r.x0 - sh.media.x0, h - (r.y1 - sh.media.y0), r.x1 - sh.media.x0, h - (r.y0 - sh.media.y0)], it);
         }
     }
+}
+
+/// The form fields of every exported page.
+pub(crate) fn collect(doc: &Document, sheets: &[Sheet]) -> Vec<Placed> {
+    let mut out = Vec::new();
+    visit(doc, sheets, |pi, r, it| {
+        if let Some(f) = &it.form_field {
+            out.push((pi, r, f.clone()));
+        }
+    });
+    out
+}
+
+/// The video and sound of every exported page (those whose file is in the document).
+pub(crate) fn collect_media(doc: &Document, sheets: &[Sheet]) -> Vec<PlacedMedia> {
+    let mut out = Vec::new();
+    visit(doc, sheets, |pi, r, it| {
+        if let (Some(m), designcraft_doc::Content::Graphic(g)) = (&it.media, &it.content)
+            && let Some(a) = doc.assets.get(&g.asset).filter(|a| !a.data.is_empty())
+        {
+            out.push((pi, r, a.name.clone(), a.mime.clone(), a.data.clone(), m.clone()));
+        }
+    });
     out
 }
 
@@ -57,10 +83,10 @@ fn pdf_string(s: &str) -> String {
     o
 }
 
-/// Add `fields` to `pdf` (page indices in output order). `None` when the file's structure isn't
-/// the simple kind this understands.
-pub(crate) fn add_fields(pdf: &[u8], fields: &[Placed]) -> Option<Vec<u8>> {
-    if fields.is_empty() {
+/// Add `fields` and `media` to `pdf` (page indices in output order). `None` when the file's
+/// structure isn't the simple kind this understands.
+pub(crate) fn add_fields(pdf: &[u8], fields: &[Placed], media: &[PlacedMedia]) -> Option<Vec<u8>> {
+    if fields.is_empty() && media.is_empty() {
         return Some(pdf.to_vec());
     }
     let t_at = rfind(pdf, b"trailer")?;
@@ -90,10 +116,53 @@ pub(crate) fn add_fields(pdf: &[u8], fields: &[Placed]) -> Option<Vec<u8>> {
         out.extend_from_slice(body);
         out.extend_from_slice(b"\nendobj\n");
     };
-    let font = next;
-    next += 1;
-    put(&mut out, font, b"<</Type/Font/Subtype/Type1/BaseFont/Helvetica/Encoding/WinAnsiEncoding>>");
     let mut per_page: Vec<Vec<usize>> = vec![Vec::new(); kids.len()];
+    // Media: the file embedded, a rendition of it, and a screen annotation that plays it (on
+    // click, and on page open when asked).
+    for (page, r, name, mime, data, m) in media {
+        let Some(&page_ref) = kids.get(*page) else { continue };
+        let (file, spec, rendition, screen) = (next, next + 1, next + 2, next + 3);
+        next += 4;
+        let mut body = format!("<</Type/EmbeddedFile/Subtype/{}/Length {}>>\nstream\n", mime.replace('/', "#2F"), data.len()).into_bytes();
+        body.extend_from_slice(data);
+        body.extend_from_slice(b"\nendstream");
+        put(&mut out, file, &body);
+        put(&mut out, spec, format!("<</Type/Filespec/F{}/UF{}/EF<</F {file} 0 R>>>>", pdf_string(name), pdf_string(name)).as_bytes());
+        // Play parameters: player controls, and a repeat count of 0 (forever) when looping.
+        let params = format!("<</BE<</C {}/RC {}>>>>", m.controls, if m.looping { 0 } else { 1 });
+        put(
+            &mut out,
+            rendition,
+            format!(
+                "<</Type/Rendition/S/MR/N{}/C<</Type/MediaClip/S/MCD/N{}/CT{}/D {spec} 0 R/P<</TF(TEMPACCESS)>>>>/P{params}>>",
+                pdf_string(name),
+                pdf_string(name),
+                pdf_string(mime)
+            )
+            .as_bytes(),
+        );
+        let play = format!("<</S/Rendition/OP 0/AN {screen} 0 R/R {rendition} 0 R>>");
+        let on_open = if m.play_on_page_load { format!("/AA<</PO{play}>>") } else { String::new() };
+        put(
+            &mut out,
+            screen,
+            format!(
+                "<</Type/Annot/Subtype/Screen/Rect[{:.2} {:.2} {:.2} {:.2}]/P {page_ref} 0 R/F 4/T{}/A{play}{on_open}>>",
+                r[0],
+                r[1],
+                r[2],
+                r[3],
+                pdf_string(name)
+            )
+            .as_bytes(),
+        );
+        per_page[*page].push(screen);
+    }
+    let font = next;
+    if !fields.is_empty() {
+        next += 1;
+        put(&mut out, font, b"<</Type/Font/Subtype/Type1/BaseFont/Helvetica/Encoding/WinAnsiEncoding>>");
+    }
     let mut all = Vec::new();
     for (page, r, f) in fields {
         let Some(&page_ref) = kids.get(*page) else { continue };
@@ -174,10 +243,12 @@ pub(crate) fn add_fields(pdf: &[u8], fields: &[Placed]) -> Option<Vec<u8>> {
         }
     }
     // The catalog's form.
-    let refs: Vec<String> = all.iter().map(|i| format!("{i} 0 R")).collect();
-    let close = cat.rfind(">>")?;
-    let acro = format!("/AcroForm<</Fields[{}]/NeedAppearances true/DA(/Helv 0 Tf 0 g)/DR<</Font<</Helv {font} 0 R>>>>>>", refs.join(" "));
-    put(&mut out, root, format!("{}{acro}{}", cat[..close].trim(), &cat[close..]).trim().as_bytes());
+    if !all.is_empty() {
+        let refs: Vec<String> = all.iter().map(|i| format!("{i} 0 R")).collect();
+        let close = cat.rfind(">>")?;
+        let acro = format!("/AcroForm<</Fields[{}]/NeedAppearances true/DA(/Helv 0 Tf 0 g)/DR<</Font<</Helv {font} 0 R>>>>>>", refs.join(" "));
+        put(&mut out, root, format!("{}{acro}{}", cat[..close].trim(), &cat[close..]).trim().as_bytes());
+    }
     let xref_at = out.len();
     entries.sort();
     let mut x = String::from("xref\n");

crates/pdf/src/lib.rs

@@ -115,6 +115,8 @@ pub struct PdfOptions {
     /// Tagged PDF: stories become paragraphs in reading order, graphics figures with their alt
     /// text, parent-page items and printer's marks artifacts.
     pub tagged: bool,
+    /// Interactive PDF: placed video and sound are embedded and play in Screen annotations.
+    pub media: bool,
 }
 
 impl Default for PdfOptions {
@@ -131,6 +133,7 @@ impl Default for PdfOptions {
             author: None,
             created: None,
             tagged: false,
+            media: false,
         }
     }
 }

crates/ui-egui/src/menus.rs

@@ -718,7 +718,7 @@ pub fn run_ui(app: &mut DesignApp, id: &str, p: &Value) -> Option<Result<Value,
         }
         "app.exportInteractivePdf" => {
             let mut q = if p.is_object() { p.clone() } else { json!({}) };
-            for (k, v) in [("pageLayout", json!("single")), ("view", json!("fitPage")), ("bookmarksPanel", json!(true))] {
+            for (k, v) in [("pageLayout", json!("single")), ("view", json!("fitPage")), ("bookmarksPanel", json!(true)), ("media", json!(true))] {
                 q.as_object_mut().map(|o| o.entry(k).or_insert(v));
             }
             export_pdf(app, &q)