Merge pull request #425 from storytold/fix/issue-419

Fix #419: PDF/.ai art past the layer scan's limit never shows a hidden layer

ROADMAP.md

@@ -4,7 +4,7 @@ VectorCraft is a clean-room, open-source, pure-Rust reimplementation of the Adob
 
 This file tracks **how far we are and what's left**. Time estimates are wall-clock hours of continuous Claude Opus 5.5 agent work (including builds and the CI gate), given both for **one agent** and for **4–6 parallel agents** on disjoint crates. They are counted from the remaining work (see [Parity estimate](#parity-estimate)), calibrated against measured throughput, and updated as work lands.
 
-_Last updated: 2026-10-08 (after M4.14–M4.98, M8.1–M8.20, M14.4–M14.7, the MCP prompts, resource templates, completions and logging, the 2026-10-07 issue fixes, the Layers panel rework and the 2026-10-08 community contributions: Pen and shape-tool modifiers, path editing, Layers, Artboards, Swatches/Brushes and dialogs, Japanese composition and PDF text, Hebrew/Arabic type, interface languages (Spanish among them), the font menu, saved selections, MCP and EPS, the missing-font export warning (#319), open paths in the Shape Builder (#374), PDF/`.ai` import keeping strokes live and hidden and nested layers (#372), per-corner Live Corners (#381), SVG type opening at the size it draws at (#396), the Wayland clipboard with copied-file paste (#398), ruler guides that can be selected, moved and deleted (#414), two batches of Preferences options that were stored but unused (#394), skeleton reads with `document.find` for large documents over MCP (#416) and anchors dragged with Direct Selection snapping to points, segments and smart guides, clipped SVG layers opening as clipping layers (#420); see [Honest assessment](#honest-assessment-2026-10-05))._
+_Last updated: 2026-10-08 (after M4.14–M4.98, M8.1–M8.20, M14.4–M14.7, the MCP prompts, resource templates, completions and logging, the 2026-10-07 issue fixes, the Layers panel rework and the 2026-10-08 community contributions: Pen and shape-tool modifiers, path editing, Layers, Artboards, Swatches/Brushes and dialogs, Japanese composition and PDF text, Hebrew/Arabic type, interface languages (Spanish among them), the font menu, saved selections, MCP and EPS, the missing-font export warning (#319), open paths in the Shape Builder (#374), PDF/`.ai` import keeping strokes live and hidden and nested layers (#372), per-corner Live Corners (#381), SVG type opening at the size it draws at (#396), the Wayland clipboard with copied-file paste (#398), ruler guides that can be selected, moved and deleted (#414), two batches of Preferences options that were stored but unused (#394), skeleton reads with `document.find` for large documents over MCP (#416) and anchors dragged with Direct Selection snapping to points, segments and smart guides, clipped SVG layers opening as clipping layers (#420), PDF/`.ai` layers read on pages of tens of millions of operators (#419); see [Honest assessment](#honest-assessment-2026-10-05))._
 
 ## Where we are
 
@@ -147,7 +147,7 @@ grade by behaviour against `plan/illustrator/`, not by whether a menu item exist
 - **Fonts:** a font is found by any of its names (family and style in every language of its name table, legacy family/style pairs, PostScript name), so documents from other apps open with the font they name; Find Font and `text.fonts` say for each font whether it was found exactly, by another name, substituted or missing, and how many glyphs are missing. PDF and `.ai` import brings Japanese text back as live type: CID fonts without a ToUnicode map read through the standard CMaps, vertical columns (WMode 1) as vertical type, letter spacing as tracking; a glyph is live type only when the installed font (in the closest style of its family) draws the same glyph, and text in a font that isn't installed comes in as live type shown in the fallback font until the font is installed (exports that draw it — PDF, EPS, EMF/WMF, raster, SVG outlines or embedded fonts — warn that they wrote the fallback font; `textAs: outlines` keeps the PDF's embedded glyphs).
 - **Files (M4):**
   - `.vectorcraft` (lossless JSON, compressed, atomic saves, save down to v1), Save As with a format chooser, Save a Copy, Revert, templates, Data Recovery after a crash (browser tabs hold their copies with Web Locks too, so a background tab's copies are never taken for a crash's, and a tab repairs copies another removed when it resumes: #367), background save and export.
-  - SVG/SVGZ in and out with SVG Options, Preserve Editing, symbols, filters, rich text and physical units (SVG type under a transform or `viewBox` scale opens at the size it draws at, the scale moved into its size, leading, baseline shift and strokes, a stretch, skew or rotation kept as its transform: #396; a top-level group with a clip path, such as a clipped Inkscape layer, opens as a layer with a clipping mask, its name kept: #420); PDF and PDF-compatible `.ai` in and out (layers, with hidden ones kept hidden with their art and nested ones as sublayers; strokes kept live, a fill and a stroke of one outline as one object; masks, editable text with type set along a curve coming back as type on a path, each artboard's art on its own page only, an `.ai`'s white page under its layers left out rather than coming in as art, spot colours, presets, security, marks and bleed, ICC output intents, raster effects, subset fonts, PDF/X, PDF layers, overprint, page thumbnails, fast web view, PDF 1.3). CMYK images keep their ink amounts from PDF import (and from placed CMYK TIFFs) to PDF export: a CMYK JPEG passes through unchanged, others stay CMYK (DeviceCMYK, or ICC-based CMYK in PDF/X-3/X-4) unless the Output settings convert them; the canvas shows them through the working CMYK profile (#320).
+  - SVG/SVGZ in and out with SVG Options, Preserve Editing, symbols, filters, rich text and physical units (SVG type under a transform or `viewBox` scale opens at the size it draws at, the scale moved into its size, leading, baseline shift and strokes, a stretch, skew or rotation kept as its transform: #396; a top-level group with a clip path, such as a clipped Inkscape layer, opens as a layer with a clipping mask, its name kept: #420); PDF and PDF-compatible `.ai` in and out (layers, with hidden ones kept hidden with their art and nested ones as sublayers, on pages of up to 64 million operators, art whose layer can't be told going to a hidden, non-printing layer rather than showing or printing: #419; strokes kept live, a fill and a stroke of one outline as one object; masks, editable text with type set along a curve coming back as type on a path, each artboard's art on its own page only, an `.ai`'s white page under its layers left out rather than coming in as art, spot colours, presets, security, marks and bleed, ICC output intents, raster effects, subset fonts, PDF/X, PDF layers, overprint, page thumbnails, fast web view, PDF 1.3). CMYK images keep their ink amounts from PDF import (and from placed CMYK TIFFs) to PDF export: a CMYK JPEG passes through unchanged, others stay CMYK (DeviceCMYK, or ICC-based CMYK in PDF/X-3/X-4) unless the Output settings convert them; the canvas shows them through the working CMYK profile (#320).
   - EPS and DXF in and out, EMF/WMF in and out; PNG (with PNG-8), JPEG, WebP, GIF, TIFF, BMP, Targa and layered PSD export. EPS import runs Illustrator's AGM/CoolType prologs (resource categories, `resourceforall`, `clipsave`/`cliprestore`, subarrays and substrings that share storage, the standard dictionaries in `systemdict`, `dictstack`, `internaldict`, executable filters, `cshow`, metadata skipped with `flushfile`), so Illustrator 30 EPS files open as vectors, and falls back to its palette TIFF preview. DXF TEXT justified Fit or Aligned spans its two points (stretched, or scaled as a whole).
   - Place and the Links panel, Package, File Info, Print with print presets, PostScript output and print tiling, slices and Save for Web, Export for Screens and Asset Export, CSS Properties, PNG/PDF/SVG/text clipboard flavours; files copied in a file manager paste as art (SVG, PDF, EMF/WMF, bitmaps) on every desktop platform, and Linux/Wayland reads the compositor's clipboard (#398).
 - **Performance:** 20k shapes + 1k texts render in 27 ms per full-retina frame (7.8 ms zoomed), 7× faster than the first version. The UI thread never blocks. The web build is 7.1 MB gzipped.

crates/pdf/src/import.rs

@@ -21,7 +21,7 @@ use vectorcraft_geom::{FillRule, PathData};
 
 use crate::import_color::{Colors, Native};
 use crate::import_mask::{MaskSpec, contains, is_rectangle, luminance, mask_spec, white_cover};
-use crate::import_scan::{MAX_NESTING, Ocgs, Scan, all_on, hides_forms, scan_page};
+use crate::import_scan::{MAX_NESTING, Ocgs, Scan, all_on, hides_forms, scan_page, tag_key};
 use crate::import_shading::{clipped, extend_clip, fold_stop_opacity, mesh_shading, shading_gradient};
 use crate::import_text::{Families, Look, Placement, TextLine, Upright};
 use crate::{CropTo, ImportOptions, ImportReport, PdfError, TextAs};
@@ -32,6 +32,9 @@ const TEXT_OUTLINES: &str = "<Text Outlines>";
 const MAX_NESTED: u32 = 8;
 /// A tiling pattern read this many times with the same art is taken to always draw it.
 const PATTERN_REUSE: u32 = 16;
+/// Stands for the group of art whose group couldn't be told (see [`Builder::unsure`]): it goes
+/// to a hidden, non-printing layer of its page.
+const UNSORTED: usize = usize::MAX;
 
 /// Import a PDF (or PDF-compatible `.ai`) with default options.
 pub fn import(bytes: &[u8]) -> Result<Document, PdfError> {
@@ -146,6 +149,9 @@ pub fn import_with_report(bytes: &[u8], opts: &ImportOptions) -> Result<ImportRe
     doc.layers.clear();
     let taken: Vec<String> = doc.swatches_iter().map(|s| s.name.clone()).chain(doc.swatch_groups.iter().map(|g| g.name.clone())).collect();
     let mut b = Builder::new(doc.peek_next_id(), Colors::new(&pdf, taken.clone()), opts.text_as, route);
+    // The art of a group that is off (drawn as `all_on` turned it on) or doesn't print mustn't
+    // show or print on its page's layer.
+    b.unsure = route && ocgs.list.iter().any(|g| (all_on && !g.on) || !g.print);
     b.taken = taken;
     b.warnings = notes;
     let cache = InterpreterCache::new();
@@ -198,19 +204,22 @@ pub fn import_with_report(bytes: &[u8], opts: &ImportOptions) -> Result<ImportRe
             show_center_mark: false,
             show_cross_hairs: false,
         });
-        let page_layer = |b: &mut Builder<'_>, art: Vec<Arc<Node>>| {
-            let mut layer = Node::layer(b.id(), &format!("Page {}", number + 1), LayerColor::Preset((i % 27) as u8));
-            if let NodeKind::Layer { children: c, .. } = &mut layer.kind {
-                *c = art;
+        let page_layer = |b: &mut Builder<'_>, art: Vec<Arc<Node>>, shown: bool| {
+            let name = if shown { format!("Page {}", number + 1) } else { format!("Page {} (unsorted)", number + 1) };
+            let mut layer = Node::layer(b.id(), &name, LayerColor::Preset((i % 27) as u8));
+            layer.visible = shown;
+            if let NodeKind::Layer { children: c, printable, .. } = &mut layer.kind {
+                (*c, *printable) = (art, shown);
             }
             Slot::Page(Box::new(layer))
         };
         if parts.is_empty() {
-            slots.push(page_layer(&mut b, vec![]));
+            slots.push(page_layer(&mut b, vec![], true));
         }
         for (key, art) in parts {
             match key {
-                None => slots.push(page_layer(&mut b, art)),
+                None => slots.push(page_layer(&mut b, art, true)),
+                Some(UNSORTED) => slots.push(page_layer(&mut b, art, false)),
                 Some(g) => {
                     let into = group_art.entry(g).or_insert_with(|| {
                         place_group(g, &ocgs, &mut slots, &mut nesting);
@@ -542,6 +551,9 @@ struct Builder<'p> {
     aligned: bool,
     /// Whether optional content groups become layers.
     route: bool,
+    /// Some groups are hidden or don't print: art in marked content whose group can't be told
+    /// (once the scan and the interpreter part) is [`UNSORTED`] rather than the page's.
+    unsure: bool,
     /// The marked-content sequences open: the group each marks.
     marked: Vec<Option<usize>>,
     /// A transparency group was just pushed: a form's (whose flags come next) or an image's.
@@ -678,6 +690,7 @@ impl<'p> Builder<'p> {
             group_at: 0,
             aligned: true,
             route,
+            unsure: false,
             marked: vec![],
             pending: false,
             nested: 0,
@@ -1521,13 +1534,15 @@ impl<'a> Device<'a> for Builder<'_> {
         let at = self.tag_at;
         self.tag_at += 1;
         let group = match self.scan.tags.get(at) {
-            Some((t, g)) if self.aligned && t.as_slice() == tag => *g,
+            Some((t, g)) if self.aligned && *t == tag_key(tag) => *g,
             _ => {
                 if self.route && self.aligned {
                     self.aligned = false;
-                    self.warn("some art couldn't be told apart by layer and went to its page's layer");
+                    let why = if self.scan.cut { "a page draws more than can be read for its layers: " } else { "" };
+                    let went = if self.unsure { "a hidden, non-printing layer of its page" } else { "its page's layer" };
+                    self.warn(&format!("{why}some art couldn't be told apart by layer and went to {went}"));
                 }
-                None
+                self.unsure.then_some(UNSORTED)
             }
         }
         .filter(|_| self.route);

crates/pdf/src/import_scan.rs

@@ -21,8 +21,14 @@ use hayro_syntax::page::{Page, Resources};
 
 /// The interpreter's limit on nested form XObjects.
 const MAX_DEPTH: u32 = 50;
-/// The most content operators walked per page: past it the rest of the page isn't told apart.
-const MAX_OPS: usize = 2_000_000;
+/// The most content operators walked per page (a guard against forms drawn within each other
+/// many times over): the art past it can't be told apart by group. A drawing of a million paths
+/// takes a few million; the walk reads about 15 million a second, twice as fast as the
+/// interpreter draws them.
+const MAX_OPS: usize = 64_000_000;
+/// The most marked-content sequences noted per page (24 bytes each): past it the rest of the
+/// page isn't told apart either.
+const MAX_TAGS: usize = 4_000_000;
 /// The most groups read.
 const MAX_GROUPS: usize = 10_000;
 /// The deepest layers nest in the configuration's `/Order`.
@@ -206,10 +212,35 @@ pub(crate) fn text_string(b: &[u8]) -> String {
 /// One page as the interpreter walks it.
 #[derive(Default)]
 pub(crate) struct Scan {
-    /// Each marked-content sequence: its tag and the group (index into [`Ocgs::list`]) it marks.
-    pub tags: Vec<(Vec<u8>, Option<usize>)>,
+    /// Each marked-content sequence: its tag ([`tag_key`]) and the group (index into
+    /// [`Ocgs::list`]) it marks.
+    pub tags: Vec<(u64, Option<usize>)>,
     /// Each form transparency group: isolated, knockout.
     pub groups: Vec<(bool, bool)>,
+    /// The walk stopped at [`MAX_OPS`] or [`MAX_TAGS`]: the rest of the page isn't told apart.
+    pub cut: bool,
+}
+
+/// A marked-content tag as [`Scan::tags`] keeps it: its FNV-1a hash (the tags are only compared,
+/// and a page can have millions).
+pub(crate) fn tag_key(tag: &[u8]) -> u64 {
+    tag.iter().fold(0xcbf2_9ce4_8422_2325, |h, &b| (h ^ u64::from(b)).wrapping_mul(0x0100_0000_01b3))
+}
+
+/// [`MAX_OPS`], lowered by tests that need a page past it.
+#[cfg(not(test))]
+fn max_ops() -> usize {
+    MAX_OPS
+}
+
+#[cfg(test)]
+thread_local! {
+    pub(crate) static TEST_MAX_OPS: std::cell::Cell<usize> = const { std::cell::Cell::new(MAX_OPS) };
+}
+
+#[cfg(test)]
+fn max_ops() -> usize {
+    TEST_MAX_OPS.with(std::cell::Cell::get)
 }
 
 struct Walker<'w, 'p> {
@@ -220,6 +251,7 @@ struct Walker<'w, 'p> {
     fonts: &'w mut HashMap<u128, String>,
     visible: Vec<bool>,
     ops: usize,
+    max_ops: usize,
     out: Scan,
     /// A form XObject hidden by its own optional content was met.
     hidden_form: bool,
@@ -273,7 +305,8 @@ impl Walker<'_, '_> {
         self.note_fonts(res);
         while let Some(op) = iter.next() {
             self.ops += 1;
-            if self.ops > MAX_OPS {
+            if self.ops > self.max_ops || self.out.tags.len() >= MAX_TAGS {
+                self.out.cut = true;
                 return;
             }
             match op {
@@ -296,11 +329,11 @@ impl Walker<'_, '_> {
                             None
                         }
                     };
-                    self.out.tags.push((bdc.0.to_vec(), group));
+                    self.out.tags.push((tag_key(bdc.0.as_ref()), group));
                 }
                 TypedInstruction::BeginMarkedContent(bmc) => {
                     self.begin(true);
-                    self.out.tags.push((bmc.0.to_vec(), None));
+                    self.out.tags.push((tag_key(bmc.0.as_ref()), None));
                 }
                 TypedInstruction::EndMarkedContent(_) => {
                     self.visible.pop();
@@ -347,7 +380,7 @@ pub(crate) fn scan_page(page: &Page<'_>, ocgs: &mut Ocgs, all_on: bool, fonts: &
 
 fn walk_page<'w, 'p>(page: &Page<'p>, ocgs: &'w mut Ocgs, all_on: bool, fonts: &'w mut HashMap<u128, String>) -> Walker<'w, 'p> {
     let off = if all_on { HashSet::new() } else { ocgs.off.clone() };
-    let mut w = Walker { ocgs, off, xref: page.xref(), fonts, visible: vec![], ops: 0, out: Scan::default(), hidden_form: false };
+    let mut w = Walker { ocgs, off, xref: page.xref(), fonts, visible: vec![], ops: 0, max_ops: max_ops(), out: Scan::default(), hidden_form: false };
     w.walk(page.typed_operations(), page.resources(), 0);
     w
 }

crates/pdf/src/tests_import_layers.rs

@@ -1,5 +1,6 @@
 //! Optional content groups become layers: name, visibility (the default configuration's or the
-//! view state's), print state, lock and nesting; art that is off comes in as a hidden layer.
+//! view state's), print state, lock and nesting; art that is off comes in as a hidden layer, and
+//! art past what is read of a huge page never shows or prints a hidden layer.
 
 use vectorcraft_color::Color;
 use vectorcraft_doc::{Document, Node, NodeKind};
@@ -298,3 +299,56 @@ fn thousands_of_layers_hidden_ones_included_all_come_in() {
     assert_eq!(d.layers.len(), N);
     assert!(d.layers.iter().enumerate().all(|(i, l)| l.visible == (i % 2 == 1) && l.children().is_some_and(|c| c.len() == 1)));
 }
+
+/// Import `bytes` with the scan's operator limit lowered to `n`.
+fn import_with_max_ops(bytes: &[u8], n: usize) -> ImportReport {
+    let old = crate::import_scan::TEST_MAX_OPS.replace(n);
+    let r = import_with_report(bytes, &ImportOptions::default());
+    crate::import_scan::TEST_MAX_OPS.set(old);
+    r.unwrap()
+}
+
+#[test]
+fn art_past_the_scan_limit_never_shows_a_hidden_layer() {
+    // "Shapes" draws 100 operators before its square, then "Hidden" (off, not printed) its own.
+    let (objs, catalog, ids) = groups(1, "/OFF [{B}]");
+    let filler = "q Q ".repeat(50);
+    let page = PdfPage {
+        resources: format!("/Properties << /MC0 {} 0 R /MC1 {} 0 R >>", ids[0], ids[1]),
+        ..PdfPage::new(
+            100.0,
+            100.0,
+            &format!("/OC /MC0 BDC {filler}1 0 0 rg 10 10 30 30 re f EMC /OC /MC1 BDC 0 0 1 rg 50 50 30 30 re f EMC 0 1 0 rg 0 0 5 5 re f"),
+        )
+    };
+    let bytes = pdf_with_catalog(&[page], &[&objs[0], &objs[1]], &catalog, None);
+    // Within the limit, every group is told apart.
+    assert_eq!(names(&import(&bytes).unwrap()), ["Shapes", "Hidden", "Page 1"]);
+    let r = import_with_max_ops(&bytes, 60);
+    let d = &r.document;
+    assert_eq!(names(d), ["Shapes", "Page 1 (unsorted)", "Page 1"], "{:?}", r.warnings);
+    // The art drawn past the limit in a group already open stays in it.
+    assert!(d.layers[0].visible);
+    assert_eq!(colors(&d.layers[0]), [Color::rgb(1.0, 0.0, 0.0)]);
+    // The hidden group's art, past it, can't be told apart: it neither shows nor prints.
+    let unsorted = &d.layers[1];
+    assert!(!unsorted.visible && !printable(unsorted));
+    assert_eq!(colors(unsorted), [Color::rgb(0.0, 0.0, 1.0)]);
+    // Art outside any group is the page's, as before.
+    assert!(d.layers[2].visible && printable(&d.layers[2]));
+    assert_eq!(colors(&d.layers[2]), [Color::rgb(0.0, 1.0, 0.0)]);
+    assert!(r.warnings.iter().any(|w| w.contains("more than can be read") && w.contains("hidden, non-printing")), "{:?}", r.warnings);
+}
+
+#[test]
+fn art_past_the_scan_limit_of_shown_layers_stays_shown() {
+    // Every group shows and prints: what can't be told apart goes to the page's layer, shown.
+    let bytes = three_groups([&ocg("A", ""), &ocg("B", ""), &ocg("C", "")], &[], "");
+    let r = import_with_max_ops(&bytes, 4);
+    let d = &r.document;
+    assert_eq!(names(d), ["A", "Page 1"], "{:?}", r.warnings);
+    assert_eq!(colors(&d.layers[0]), [Color::rgb(1.0, 0.0, 0.0)]);
+    assert!(d.layers[1].visible && printable(&d.layers[1]));
+    assert_eq!(colors(&d.layers[1]), [Color::rgb(0.0, 0.0, 1.0), Color::rgb(0.0, 1.0, 0.0)]);
+    assert!(r.warnings.iter().any(|w| w.contains("went to its page's layer")), "{:?}", r.warnings);
+}