Forms: encode appearances with the existing simple font (#1044)

* fix(forms): encode appearances with the chosen font

* docs: record form font encoding limits

* docs: log verified form encoding regressions

---------

Co-authored-by: Brandon Thomas <[email protected]>

ROADMAP.md

@@ -140,6 +140,8 @@ Newest first: the date, what moved, and the new overall estimate.
 - **2026-10-11 (M4, #569):** Combine files takes images and text files as well as PDFs. PNG, JPEG, TIFF, GIF, BMP, JPEG 2000 and .txt files become PDFs as they are added (Add files…, drops, folder scans), so each is an ordinary row with its pages, thumbnails and page range; a file that can't be read (a damaged image or PDF, a type that can't be converted) stays in the list with why in its Warnings, and Combine asks before leaving such files out (Cancel changes nothing); password-protected files still have to be unlocked first. `doc_combine` accepts the same types, and a converted file's bookmark drops its extension. A PNG whose header claims more than 2 GiB of pixels is now refused before its buffer is allocated (the png crate's limits don't cover it), as TIFF already was. Office documents and HTML are still not converted. Overall estimate unchanged.
 - **2026-10-11 (M3, Windows renderer startup):** Renderer-creation GPU errors return through the existing scoped handler so Auto startup can use the existing OpenGL fallback. A real-device invalid/valid pipeline regression covers error propagation and device reuse; the independent Windows hostile-argument fixture reports unavailable symbolic-link privileges. The underlying x86 DX12 compiler error remains unresolved; see `docs/windows-renderer-startup.md`. Milestone and effort estimates unchanged.
 
+- **2026-10-11 (M6):** Existing simple field-font encodings and Differences are respected when generating appearances; an unencodable value uses a separate Helvetica appearance font without changing the source font. Synthetic reopen regressions cover punctuation, subset fallback and multiline text. Exact font metrics/layout and broader Unicode fallback remain incomplete. Estimates unchanged.
+
 - **2026-10-10 (M14, interface size):** Preferences ▸ Interface size scales the whole interface (50–250 %, setting `ui_scale`, control option `ui-scale`); 50 % brings an interface the system draws at 200 % back to 100 % (#456). Auto, the default, follows the desktop's text scaling: on Linux and the BSDs the desktop app reads GNOME's `text-scaling-factor` once at launch, so with Large Text (1.25) on a 1× display the interface no longer draws 13 px labels that show their pixels next to the system's text. The size is egui's zoom factor, so `pixels_per_point` grows with it and page rasters stay one texel per screen pixel; egui's own ⌘+/⌘− interface zoom is turned off so those keys stay page zoom. Still missing: KDE's font DPI and Windows' "Make text bigger" aren't read (both scale only text there), and egui's text is still greyscale-antialiased without the system's gamma, so small text looks rougher than GTK's at 100 %.
 - **2026-10-10 (M14, #496):** Shortcut displays share a UI formatter across toolbar search, registry menus, command palette, Help and tooltips. Modifier and key names use a dedicated catalog context in every bundled translation; missing entries keep the English key name rather than a command translation. German shows Strg/Umschalt/Entf, macOS retains its symbols, and runtime language changes refresh the display. Regression coverage checks localized layout widths, English/German switching, platform conventions and unchanged command execution. Milestone and effort estimates unchanged.
 

crates/forms/src/appearance.rs

@@ -5,17 +5,19 @@
 //! marked-content block as viewers expect. Supported: single-line, multiline (wrapped), comb,
 //! password, quadding, auto font size (`0 Tf`), combo boxes and list boxes (selection shown).
 //!
-//! The `/DA` font is used when the form's `/DR` defines it as a simple font (text encoded in
-//! WinAnsi) or as a composite font with a predefined Unicode CMap such as `UniJIS-UTF16-H`, as
-//! Japanese forms use (text encoded in that CMap). Otherwise (other composite fonts, missing
-//! resources) Helvetica is used, so text is always visible. Widths use the approximate metrics
-//! of `pdfcraft-fonts` (Helvetica, or one em per full-width character in composite fonts).
+//! The `/DA` font is used when the form's `/DR` defines it as a simple font that can encode the
+//! displayed text (using its own encoding and Differences), or as a composite font with a
+//! predefined Unicode CMap such as `UniJIS-UTF16-H`, as Japanese forms use. Otherwise (unsupported
+//! characters, other composite fonts, missing resources) Helvetica is used. Widths use the
+//! approximate metrics of `pdfcraft-fonts` (Helvetica, or one em per full-width character in
+//! composite fonts).
 //!
 //! `/MK /R` (0, 90, 180, 270, counterclockwise) lays the content out in the upright box and sets
 //! the appearance `/Matrix` so that box lands on the widget rectangle. Any other value is 0, and
 //! rotation 0 leaves `/Matrix` off.
 
 use pdfcraft_cos::{Dict, Document, Object, Stream};
+use pdfcraft_fonts::pdf::Metrics;
 use pdfcraft_fonts::{UnicodeCMap, cjk_width, helvetica_width, literal, win_ansi, wrap_with};
 
 use crate::{Field, FieldKind, Widget, acroform, flags};
@@ -111,24 +113,35 @@ pub fn parse_da(da: &str) -> Da {
 }
 
 /// How a field font's text is encoded and measured.
-#[derive(Clone, Copy, Debug, PartialEq)]
+#[derive(Clone, Debug, PartialEq)]
 enum FontText {
     WinAnsi,
+    /// A reused simple font's encoding, including indirect dictionaries and Differences.
+    Simple(Box<Metrics>),
     /// A composite font with a predefined Unicode CMap.
     Unicode(UnicodeCMap),
 }
 
 impl FontText {
-    fn encode(self, s: &str) -> Vec<u8> {
+    fn encode(&self, s: &str) -> Vec<u8> {
         match self {
             Self::WinAnsi => win_ansi(s),
+            // Selection checks every displayed string first; wrapping only removes line breaks.
+            Self::Simple(metrics) => metrics.encode(&s.replace('\t', " ")).unwrap_or_default(),
             Self::Unicode(cmap) => cmap.encode(s),
         }
     }
 
-    fn width(self, s: &str, size: f64) -> f64 {
+    fn can_encode(&self, s: &str) -> bool {
         match self {
-            Self::WinAnsi => helvetica_width(s, size),
+            Self::Simple(metrics) => metrics.encode(&s.replace('\t', " ")).is_some(),
+            _ => true,
+        }
+    }
+
+    fn width(&self, s: &str, size: f64) -> f64 {
+        match self {
+            Self::WinAnsi | Self::Simple(_) => helvetica_width(s, size),
             Self::Unicode(_) => cjk_width(s, size),
         }
     }
@@ -154,7 +167,7 @@ fn dr_font(doc: &Document, name: &str) -> Option<(Object, FontText)> {
     if matches!(fd.name(b"BaseFont"), Some(b"ZapfDingbats" | b"Symbol")) {
         return None;
     }
-    Some((entry, FontText::WinAnsi))
+    Some((entry, FontText::Simple(Box::new(Metrics::from_dict(doc, fd)))))
 }
 
 fn helvetica() -> Object {
@@ -281,7 +294,22 @@ pub fn field_appearance_as(doc: &Document, f: &Field, w: &Widget, values: &[Stri
     let place = placement(mk_rotation(doc, &wd), page_w, page_h);
     let (width, height) = (place.layout_w, place.layout_h);
     let da = parse_da(wd.get(b"DA").and_then(|o| doc.resolve(o).as_string().map(|s| s.to_text())).as_deref().unwrap_or(&f.da));
-    let (font_name, font_obj, enc) = match dr_font(doc, &da.font) {
+    let text: String = if f.kind == FieldKind::Combo {
+        values.iter().map(|v| f.options.iter().find(|(e, _)| e == v).map_or(v.clone(), |(_, d)| d.clone())).collect::<Vec<_>>().join(", ")
+    } else {
+        values.first().cloned().unwrap_or_default()
+    };
+    let text = if f.has(flags::PASSWORD) { "*".repeat(text.chars().count()) } else { text };
+    let can_show = |enc: &FontText| {
+        if f.kind == FieldKind::List {
+            f.options.iter().all(|(_, display)| enc.can_encode(display))
+        } else if f.kind == FieldKind::Text && f.has(flags::MULTILINE) {
+            text.split(['\n', '\r']).all(|line| enc.can_encode(line))
+        } else {
+            enc.can_encode(&text)
+        }
+    };
+    let (font_name, font_obj, enc) = match dr_font(doc, &da.font).filter(|(_, enc)| can_show(enc)) {
         Some((o, enc)) => (da.font.clone(), o, enc),
         None => ("Helv".to_string(), helvetica(), FontText::WinAnsi),
     };
@@ -326,12 +354,6 @@ pub fn field_appearance_as(doc: &Document, f: &Field, w: &Widget, values: &[Stri
             }
         }
         _ => {
-            let text: String = if f.kind == FieldKind::Combo {
-                values.iter().map(|v| f.options.iter().find(|(e, _)| e == v).map_or(v.clone(), |(_, d)| d.clone())).collect::<Vec<_>>().join(", ")
-            } else {
-                values.first().cloned().unwrap_or_default()
-            };
-            let text = if f.has(flags::PASSWORD) { "*".repeat(text.chars().count()) } else { text };
             let comb = f.has(flags::COMB) && !f.has(flags::MULTILINE) && !f.has(flags::PASSWORD) && f.max_len.is_some_and(|m| m > 0);
             if f.kind == FieldKind::Text && f.has(flags::MULTILINE) {
                 if size == 0.0 {

crates/forms/src/tests.rs

@@ -73,6 +73,24 @@ fn ap(doc: &Document, w: &Widget) -> String {
     String::from_utf8_lossy(&s.decoded().unwrap()).into_owned()
 }
 
+/// Read the drawn strings together with the appearance's actual font after saving and reopening.
+fn appearance_text(doc: &Document, w: &Widget, font_name: &[u8]) -> (Object, Vec<Vec<u8>>) {
+    let widget = doc.get(w.obj);
+    let normal = widget.as_dict().unwrap().get(b"AP").unwrap().as_dict().unwrap().reference(b"N").unwrap();
+    let appearance = doc.get(normal);
+    let Object::Stream(stream) = appearance.as_ref() else { panic!("appearance stream") };
+    let resources = doc.resolve(stream.dict.get(b"Resources").unwrap());
+    let fonts = doc.resolve(resources.as_dict().unwrap().get(b"Font").unwrap());
+    let font = doc.resolve(fonts.as_dict().unwrap().get(font_name).unwrap());
+    let shown = pdfcraft_content::parse(&stream.decoded().unwrap())
+        .ops
+        .into_iter()
+        .filter(|op| op.is("Tj"))
+        .map(|op| op.operands[0].as_string().unwrap().bytes.clone())
+        .collect();
+    (font.as_ref().clone(), shown)
+}
+
 #[test]
 fn the_field_tree_is_read_with_inheritance() {
     let doc = fixture();
@@ -227,6 +245,93 @@ fn text_fields_get_new_appearances() {
     assert!(raw.windows(8).any(|x| x == b"(Z\xfcrich)"));
 }
 
+#[test]
+fn existing_field_font_encoding_keeps_italian_punctuation_visible() {
+    // ReportLab-style Helvetica: an indirect encoding dictionary with PDFDoc-like Differences.
+    // WinAnsi 0x92/0x97 would show trademark/Scaron in this font, not quoteright/emdash.
+    let objs: Vec<String> = vec![
+        "<< /Type /Catalog /Pages 2 0 R /AcroForm 4 0 R >>".into(),
+        "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(),
+        "<< /Type /Page /Parent 2 0 R /Annots [6 0 R] >>".into(),
+        "<< /Fields [6 0 R] /DA (/Helv 12 Tf 0 g) /DR << /Font << /Helv 5 0 R >> >> >>".into(),
+        "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding 7 0 R >>".into(),
+        "<< /Type /Annot /Subtype /Widget /FT /Tx /T (localita) /Rect [50 700 450 730] /P 3 0 R >>".into(),
+        "<< /Type /Encoding /Differences [39 /quotesingle 96 /grave 132 /emdash 144 /quoteright 146 /trademark 151 /Scaron 224 /agrave] >>".into(),
+    ];
+    let mut doc = document(&objs);
+    let original_font = doc.get(ObjRef::new(5, 0)).as_ref().clone();
+    let value = "Città Sant’Angelo — qualità";
+    set_value(&mut doc, "localita", &FieldValue::Text(value.into())).unwrap();
+    let doc = reopen(&doc);
+    let all = fields(&doc);
+    let f = field(&all, "localita");
+    assert_eq!(f.value, [value], "the stored value and the drawn text must agree");
+    let (font, lines) = appearance_text(&doc, &f.widgets[0], b"Helv");
+    assert_eq!(font, original_font, "keep the existing field's font and encoding");
+    let metrics = pdfcraft_fonts::pdf::Metrics::from_dict(&doc, font.as_dict().unwrap());
+    let shown = lines.concat();
+    assert_eq!(metrics.decode(&shown), value, "decode the appearance with its actual font");
+    assert_eq!(shown, b"Citt\xe0 Sant\x90Angelo \x84 qualit\xe0");
+}
+
+#[test]
+fn unavailable_glyph_uses_an_appearance_font_without_changing_the_source_font() {
+    // A subset font with only A cannot draw the newly entered accented text. Replacing its
+    // encoding would corrupt every existing use of that font, so only the appearance falls back.
+    let objs: Vec<String> = vec![
+        "<< /Type /Catalog /Pages 2 0 R /AcroForm 4 0 R >>".into(),
+        "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(),
+        "<< /Type /Page /Parent 2 0 R /Annots [6 0 R] >>".into(),
+        "<< /Fields [6 0 R] /DA (/Original 12 Tf 0 g) /DR << /Font << /Original 5 0 R >> >> >>".into(),
+        "<< /Type /Font /Subtype /Type1 /BaseFont /ABCDEF+Custom /FirstChar 65 /LastChar 65 /Widths [600] /Encoding /WinAnsiEncoding >>".into(),
+        "<< /Type /Annot /Subtype /Widget /FT /Tx /T (localita) /Rect [50 700 450 730] /P 3 0 R >>".into(),
+    ];
+    let mut doc = document(&objs);
+    let original_font = doc.get(ObjRef::new(5, 0)).as_ref().clone();
+    let value = "Città Sant’Angelo — qualità";
+    set_value(&mut doc, "localita", &FieldValue::Text(value.into())).unwrap();
+    let doc = reopen(&doc);
+    assert_eq!(doc.get(ObjRef::new(5, 0)).as_ref(), &original_font);
+    let all = fields(&doc);
+    let f = field(&all, "localita");
+    assert_eq!(f.value, [value]);
+    let (font, lines) = appearance_text(&doc, &f.widgets[0], b"Helv");
+    let font = font.as_dict().unwrap();
+    assert_eq!(font.name(b"BaseFont"), Some(&b"Helvetica"[..]));
+    assert_eq!(font.name(b"Encoding"), Some(&b"WinAnsiEncoding"[..]));
+    let metrics = pdfcraft_fonts::pdf::Metrics::from_dict(&doc, font);
+    let shown = lines.concat();
+    assert_eq!(metrics.decode(&shown), value);
+    assert_eq!(shown, b"Citt\xe0 Sant\x92Angelo \x97 qualit\xe0");
+}
+
+#[test]
+fn multiline_wrapping_keeps_encodable_text_without_requiring_ascii_space() {
+    let objs: Vec<String> = vec![
+        "<< /Type /Catalog /Pages 2 0 R /AcroForm 4 0 R >>".into(),
+        "<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 600 800] >>".into(),
+        "<< /Type /Page /Parent 2 0 R /Annots [6 0 R] >>".into(),
+        "<< /Fields [6 0 R] /DA (/Helv 12 Tf 0 g) /DR << /Font << /Helv 5 0 R >> >> >>".into(),
+        "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding 7 0 R >>".into(),
+        "<< /Type /Annot /Subtype /Widget /FT /Tx /T (localita) /Ff 4096 /Rect [50 500 130 550] /P 3 0 R >>".into(),
+        "<< /Type /Encoding /Differences [32 /.notdef 144 /quoteright 160 /nbspace 224 /agrave] >>".into(),
+    ];
+    let mut doc = document(&objs);
+    let original_font = doc.get(ObjRef::new(5, 0)).as_ref().clone();
+    // NBSP remains part of the word; wrapping splits it by character without adding spaces.
+    let value = "Città\u{a0}Sant’Angelo\nqualità";
+    set_value(&mut doc, "localita", &FieldValue::Text(value.into())).unwrap();
+    let doc = reopen(&doc);
+    let all = fields(&doc);
+    let f = field(&all, "localita");
+    assert_eq!(f.value, [value]);
+    let (font, lines) = appearance_text(&doc, &f.widgets[0], b"Helv");
+    assert_eq!(font, original_font);
+    assert!(lines.len() >= 3 && lines.iter().all(|line| !line.is_empty()));
+    let metrics = pdfcraft_fonts::pdf::Metrics::from_dict(&doc, font.as_dict().unwrap());
+    assert_eq!(metrics.decode(&lines.concat()), value.replace('\n', ""));
+}
+
 #[test]
 fn japanese_choices_use_the_unicode_cid_font() {
     // A choice field whose /DA names a non-embedded CID font with a predefined Unicode CMap

docs/roadmap.md

@@ -1,6 +1,6 @@
 # PdfCraft roadmap: milestones and what's next
 
-> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-10 · **Change:** minor (localized shortcut labels, #496; alpha gate added; convert-and-share re-judged with the cross-app rule → alpha) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
+> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (M6: respect existing form-font encodings; retain source fonts on appearance fallback; estimates unchanged) · **Target:** Adobe Acrobat Pro (Acrobat DC, continuous track 26.002.21931, macOS)
 
 Forward-looking: the milestones, the current focus and what comes next. The parity numbers behind
 it are in [target-app-parity.md](target-app-parity.md), the ranked work list in [gaps.md](gaps.md),
@@ -72,6 +72,8 @@ total matches the parity estimate.
 | M5 | Comments (all annotation types, XFDF) | 120–200 | 88% | 12–25 | Done: notes, highlight/underline/strikeout/squiggly, text boxes, ink, lines, arrows, rectangles, ovals, stamps (dynamic, Sign Here, Standard Business), with appearance streams; replies, status, checkmarks, locking, move/resize/restyle/delete; properties dialog; panel filter and sort; hide all; summaries (comments only); XFDF/FDF import and export; flatten; Fill & Sign; agent tools.<br>Done since: polygons, connected lines, clouds, callouts, inserted text, colour and checkmark filters; image signature pointer previews, selection handles and live image movement/resizing with proportional corners and independent edge handles. Missing: custom stamps, replace-text proposals, summary layouts with the page |
 | M6 | Forms + JavaScript | 160–320 | 50% | 40–75 | Done: filling all field types with regenerated appearances; Prepare a form (every field type, multi-select/shared properties, move/resize/delete, Field Properties General/Appearance/Position/Options/Format/Validate/Calculate); Acrobat's AF format/keystroke/validate/calculate functions and simplified field notation run natively in Acrobat's event order; tab order; push-button actions; flatten; agent tools. Form data exchange (FDF, XFDF, XML, CSV, text) done.<br>Done since: JavaScript engine (boa, sandboxed) running custom keystroke/validate/calculate/format and button scripts, document JavaScripts, console. Missing: Actions tab, wider object model (annotations, layers, dialogs), XFA |
 | M7 | Content editing (text, images, header/footer, watermark) | 250–500 | 20% | 100–180 | Done: header & footer, watermarks, backgrounds, Bates; added text and images that stay editable (move, resize, format, rotate, flip, crop, replace); links (Link tool, Link Properties, create from URLs, remove all). Missing: editing existing text and images in place (the longest pole), watermark print/screen visibility and repeat |
+| M6 | Forms + JavaScript | 160–320 | 50% | 40–75 | Done: filling all field types with regenerated appearances; Prepare a form (every field type, multi-select/shared properties, move/resize/delete, Field Properties General/Appearance/Position/Options/Format/Validate/Calculate); Acrobat's AF format/keystroke/validate/calculate functions and simplified field notation run natively in Acrobat's event order; tab order; push-button actions; flatten; agent tools. Form data exchange (FDF, XFDF, XML, CSV, text) done.<br>Done since: JavaScript engine (boa, sandboxed) running custom keystroke/validate/calculate/format and button scripts, document JavaScripts, console. Missing: Actions tab, wider object model (annotations, layers, dialogs), XFA<br>Done since: Existing simple field-font encodings and Differences are respected when generating appearances; an unencodable value uses a separate Helvetica appearance font without changing the source font. Synthetic reopen regressions cover punctuation, subset fallback and multiline text. Exact font metrics/layout and broader Unicode fallback remain incomplete. |
+| M7 | Content editing (text, images, header/footer, watermark) | 250–500 | 20% | 100–180 | Done: header & footer, watermarks, backgrounds, Bates; added text and images that stay editable (move, resize, format, rotate, flip, crop, replace); links (Link tool, Link Properties, create from URLs, remove all). Missing: editing existing text and images in place (the longest pole), image/PDF watermarks |
 | M8 | Security + redaction | 100–180 | 68% | 15–30 | Done: opening protected documents, permissions, Protect Using Password, Remove security; redaction (mark text/areas/pages, Search & Redact with patterns, apply removing glyphs, image pixels, vectors, XObject content, annotations and fields, verification, full rewrite on save); Remove hidden information and Sanitize.<br>Done since: redaction codes (U.S. FOIA and Privacy Act sets as overlay text, in Redaction properties and `redact_mark`). Missing: certificate security, custom redaction code sets and pattern locales, DCT re-encoding |
 | M9 | Signatures (PAdES, validation) | 160–280 | 60% | 40–70 | Done: new `sign` crate (DER, X.509, CMS, PKCS #12 on RustCrypto; aws-lc-rs for RSA private keys); PAdES B-B signing (visible, invisible, existing fields, certification with DocMDP); validation with trust store and changes-after-signing classification; self-signed digital IDs; Signatures panel, message bar, sign dialogs; agent tools. Checked with pdfsig and OpenSSL.<br>Done since: macOS Keychain and Windows Current User Personal store signing (software-backed CNG keys), certificate viewer, signed documents protected from rewrites. Missing: timestamps (B-T), LTV (DSS, OCSP, CRL), FieldMDP, smart cards, PKCS #11 |
 | M10 | OCR, create, export, print | 200–350 | 33% | 120–215 | Done: create from blank/text/PNG/JPEG/TIFF (multi-page)/GIF/BMP; export PNG/JPEG/TIFF and text; Print (Acrobat's sizing, n-up, booklet, poster, comments & forms, preview, the CUPS and Windows spoolers, print-ready PDF).<br>Done since: embedded/72/custom DPI choices for image imports; export all images; OCR (searchable image for pages, ranges and multiple files); single-sided cut-and-stack imposition with cut marks, through Print and doc_print; Create PDF from multiple files (PDFs, images and text, edited as pages in the grid). Missing: OCR languages beyond Latin, editable-text OCR output, Office export/import, web printing |
@@ -121,6 +123,7 @@ M2 (renderer), M7 (content editing), M10 (OCR and Office), M11 (Preflight and st
 
 | Date | Change | Summary |
 |---|---|---|
+| 2026-10-11 | minor | M6: respect existing form-font encodings; retain source fonts on appearance fallback. Estimates unchanged. |
 | 2026-10-10 | minor | M14: localized shortcut key names (#496), including runtime switching and native layout checks. Estimates unchanged. |
 | 2026-10-10 | minor | Re-judged convert and share with the cross-app gate rule (only a workflow that can't be completed at all blocks): partial, not blocking → stage alpha |
 | 2026-10-10 | minor | Added the alpha gate (six core workflows; convert and share fails → pre-alpha, 40–70 h to alpha) and rank 0 in the upcoming milestones |