Styles: overlays keep anti-aliased edges clean (#416)

Cause: the interior styles were blended over the layer body with the
content's coverage folded into their opacity. On an edge pixel at 50 %
coverage a Gradient or Color Overlay went over the half-transparent body
as a half-transparent layer, so the result kept a quarter of the layer's
own fill and its alpha rose to 0.75: a light rim (the white fill
showing through) on styled text.

Fix: blend_clipped composites the interior styles (Gradient Overlay,
Color Overlay, Satin, Inner Glow, Inner Shadow and the inside of
Stroke) onto the layer within its shape: the body is divided by the
content coverage, blended with the style at its own opacity, and
multiplied back, so the alpha is kept and an edge pixel mixes the style
with the layer's colour inside the shape only, as a clipped layer does.
Fill Opacity and Blend Interior Styles as Group behave as before.

Tests: tests_styles::overlays_leave_antialiased_edges_clean (a masked
square with half-covered edge columns; Color Overlay, Gradient Overlay
and an inside Stroke now match a plain layer of the overlay colour;
Color Overlay's edge was 0.41 red against 0.23 before).

Co-Authored-By: Claude Opus 5.5 <[email protected]>

crates/render/src/styles.rs

@@ -460,18 +460,41 @@ fn noisy(a: f32, noise: f32, x: usize, y: usize, seed: u32) -> f32 {
     if noise <= 0.0 { a } else { a * (1.0 - noise * hash_noise(x as u32, y as u32, seed)) }
 }
 
+/// `dst` with the straight colour `c` at coverage `a` blended onto it at pixel (`x`, `y`).
+#[inline]
+fn blend_style(mode: BlendMode, seed: u32, x: usize, y: usize, dst: [f32; 4], c: [f32; 3], a: f32) -> [f32; 4] {
+    let n = if mode == BlendMode::Dissolve { hash_noise(x as u32, y as u32, seed) } else { 0.5 };
+    blend_pixel(mode, dst, [c[0] * a, c[1] * a, c[2] * a, a], n)
+}
+
 /// Blend a straight colour with coverage `a` onto every pixel of `img` (`f(x, y)` → (colour, a)).
 fn blend_into(img: &mut Image, mode: BlendMode, seed: u32, f: impl Fn(usize, usize) -> ([f32; 3], f32) + Sync) {
     let w = img.width as usize;
     img.data.par_chunks_mut(w.max(1)).enumerate().for_each(|(y, row)| {
         for (x, d) in row.iter_mut().enumerate() {
             let (c, a) = f(x, y);
-            if a <= 0.0 {
-                continue;
+            if a > 0.0 {
+                *d = blend_style(mode, seed, x, y, *d, c, a);
+            }
+        }
+    });
+}
+
+/// Blend an interior style clipped to the content: `f(x, y)` → (straight colour, opacity) is
+/// blended onto the layer *inside* its shape, whose coverage is `clip`, so the body keeps its
+/// alpha and an anti-aliased edge pixel mixes the style with the layer's colour in the
+/// proportion the style covers it, instead of letting the layer's own colour show through
+/// around a blended edge (#416).
+fn blend_clipped(img: &mut Image, clip: &Plane, mode: BlendMode, seed: u32, f: impl Fn(usize, usize) -> ([f32; 3], f32) + Sync) {
+    let w = img.width as usize;
+    img.data.par_chunks_mut(w.max(1)).zip(clip.data.par_chunks(w.max(1))).enumerate().for_each(|(y, (row, cov))| {
+        for (x, (d, &k)) in row.iter_mut().zip(cov).enumerate() {
+            if k > 0.0
+                && let (c, a) = f(x, y)
+                && a > 0.0
+            {
+                *d = blend_style(mode, seed, x, y, d.map(|v| v / k), c, a).map(|v| v * k);
             }
-            let src = [c[0] * a, c[1] * a, c[2] * a, a];
-            let n = if mode == BlendMode::Dissolve { hash_noise(x as u32, y as u32, seed) } else { 0.5 };
-            *d = blend_pixel(mode, *d, src, n);
         }
     });
 }
@@ -568,13 +591,12 @@ pub fn render(ctx: &EvalCtx, layer: &Layer, content: &Buf, layer_size: [f64; 2])
     if !bl.interior_group && bl.fill < 1.0 {
         body.scale_alpha(bl.fill);
     }
-    let cov = |i: usize| alpha.data[i];
     if let Some(go) = &styles.gradient {
         let (cx, cy, hw, hh) = gradient_frame(go, &buf, &alpha, layer_size);
         let (ux, uy) = (go.angle.to_radians().cos(), -go.angle.to_radians().sin());
         let ext = ((hw * ux).abs() + (hh * uy).abs()).max(1e-3) * go.scale;
         let radius = hw.max(hh).max(1e-3) * go.scale;
-        blend_into(&mut body, go.mode, seed, |x, y| {
+        blend_clipped(&mut body, &alpha, go.mode, seed, |x, y| {
             let (px, py) = (x as f64 + 0.5 - cx, y as f64 + 0.5 - cy);
             let along = px * ux + py * uy;
             let across = -px * uy + py * ux;
@@ -592,35 +614,35 @@ pub fn render(ctx: &EvalCtx, layer: &Layer, content: &Buf, layer_size: [f64; 2])
                 t = 1.0 - t;
             }
             let c = go.gradient.sample(t.clamp(0.0, 1.0));
-            ([c[0], c[1], c[2]], c[3] * go.opacity * cov(y * w + x))
+            ([c[0], c[1], c[2]], c[3] * go.opacity)
         });
     }
     if let Some(co) = &styles.color {
-        blend_into(&mut body, co.mode, seed, |x, y| (co.color, co.opacity * cov(y * w + x)));
+        blend_clipped(&mut body, &alpha, co.mode, seed, |_, _| (co.color, co.opacity));
     }
     if let Some(sa) = &styles.satin {
         let b = gauss_plane(&alpha, sa.size * s / 2.0, sa.size * s / 2.0);
         let off = [sa.offset[0] * s, sa.offset[1] * s];
-        blend_into(&mut body, sa.mode, seed, |x, y| {
+        blend_clipped(&mut body, &alpha, sa.mode, seed, |x, y| {
             let v = (shifted(&b, x, y, off) - shifted(&b, x, y, [-off[0], -off[1]])).abs().min(1.0);
             let v = if sa.invert { 1.0 - v } else { v };
-            (sa.color, v * sa.opacity * cov(y * w + x))
+            (sa.color, v * sa.opacity)
         });
     }
     if let Some(gl) = &styles.inner_glow {
         let v = glow_intensity(&alpha, &d, gl, s, true);
-        blend_into(&mut body, gl.mode, seed, |x, y| {
+        blend_clipped(&mut body, &alpha, gl.mode, seed, |x, y| {
             let i = y * w + x;
             let e = if gl.center { 1.0 - v.data[i] } else { v.data[i] };
             let (c, a) = glow_color(gl, e, x, y);
-            (c, noisy(a * gl.opacity * cov(i), gl.noise, x, y, seed ^ 0x3c3c))
+            (c, noisy(a * gl.opacity, gl.noise, x, y, seed ^ 0x3c3c))
         });
     }
     if let Some(is) = &styles.inner_shadow {
         let shape = soft_shape(&alpha, &d, is.size * s, is.spread, true);
         let off = [is.offset[0] * s, is.offset[1] * s];
-        blend_into(&mut body, is.mode, seed, |x, y| {
-            let a = shifted(&shape, x, y, off).min(1.0) * is.opacity * cov(y * w + x);
+        blend_clipped(&mut body, &alpha, is.mode, seed, |x, y| {
+            let a = shifted(&shape, x, y, off).min(1.0) * is.opacity;
             (is.color, noisy(a, is.noise, x, y, seed ^ 0x1234))
         });
     }
@@ -635,11 +657,13 @@ pub fn render(ctx: &EvalCtx, layer: &Layer, content: &Buf, layer_size: [f64; 2])
             2 => (size / 2.0, size / 2.0),
             _ => (size, 0.0),
         };
-        // Inside part: blended over the body within the content.
+        // Inside part: blended over the body within the content, over the share of each
+        // pixel's coverage the ring takes.
         if r_in > 0.0 {
-            blend_into(&mut body, sk.mode, seed, |x, y| {
+            blend_clipped(&mut body, &alpha, sk.mode, seed, |x, y| {
                 let i = y * w + x;
-                let ring = (alpha.data[i] - grown(sd.data[i], -r_in)).max(0.0);
+                let a = alpha.data[i];
+                let ring = ((a - grown(sd.data[i], -r_in)) / a.max(1e-6)).clamp(0.0, 1.0);
                 (sk.color, ring * sk.opacity)
             });
         }

crates/render/src/tests_styles.rs

@@ -1,10 +1,10 @@
 //! Layer Styles pixel tests.
 
 use effectcraft_color::Label;
-use effectcraft_keyframe::{Gradient, Keyframe, Value};
+use effectcraft_keyframe::{Gradient, Keyframe, ShapePath, Value};
 use effectcraft_project::build::{self, Ids};
 use effectcraft_project::styles::{self as st, GROUP};
-use effectcraft_project::{Comp, ItemId, ItemKind, Layer, LayerSource, Project, Solid};
+use effectcraft_project::{Comp, ItemId, ItemKind, Layer, LayerSource, MaskMode, Project, Solid};
 use effectcraft_time::{FrameRate, Tick};
 
 use crate::{Image, NoFootage, RenderOpts, Renderer, render_frame};
@@ -182,6 +182,67 @@ fn gradient_overlay_follows_angle() {
     assert!(img.get(81, 50)[0] > 0.9 && img.get(118, 50)[0] < 0.1);
 }
 
+/// Overlays replace the layer's colour inside its alpha: an anti-aliased edge pixel mixes the
+/// background with the overlay colour only, exactly as a layer of that colour would, with no
+/// fringe of the layer's own colour (#416).
+#[test]
+fn overlays_leave_antialiased_edges_clean() {
+    let (blue, bg) = ([0.2, 0.533, 1.0], [0.25, 0.22, 0.45]);
+    // A mask half a pixel inside the square's left and right sides: in layer space, where the
+    // styles run, columns 80 and 119 are half covered.
+    let edge = |color: [f32; 3], f: &dyn Fn(&mut Project, &Comp, &mut Layer)| {
+        let (p, cid) = scene(color, Some(bg), |p, c, l| {
+            let mut next = p.next_id;
+            let pts = vec![[0.5, 0.0], [39.5, 0.0], [39.5, 40.0], [0.5, 40.0]];
+            let sp = ShapePath { in_tangents: vec![[0.0; 2]; 4], out_tangents: vec![[0.0; 2]; 4], vertices: pts, closed: true, feather: Vec::new() };
+            let m = build::mask(&mut Ids(&mut next), "Mask 1", sp, MaskMode::Add, [255, 255, 0]);
+            p.next_id = next;
+            l.props.group_mut("masks").unwrap().children.push(m.into());
+            f(p, c, l);
+        });
+        let img = render(&p, cid);
+        [img.get(80, 50), img.get(119, 50), img.get(100, 50)]
+    };
+    let plain = edge(blue, &|_, _, _| {});
+    assert!(near(plain[0][0], 0.5 * (blue[0] + bg[0]), 0.01), "half covered: {:?}", plain[0]);
+    let color = Value::Color([blue[0] as f64, blue[1] as f64, blue[2] as f64, 1.0]);
+    let flat =
+        Gradient { colors: vec![(0.0, [blue[0], blue[1], blue[2], 1.0]), (1.0, [blue[0], blue[1], blue[2], 1.0])], opacities: vec![(0.0, 1.0), (1.0, 1.0)] };
+    let styled: [(&str, Box<dyn Fn(&mut Project, &Comp, &mut Layer)>); 3] = [
+        (
+            "Color Overlay",
+            Box::new(|p, c, l| {
+                add(p, c, l, "colorOverlay");
+                set(l, "colorOverlay/color", color.clone());
+            }),
+        ),
+        (
+            "Gradient Overlay",
+            Box::new(|p, c, l| {
+                add(p, c, l, "gradientOverlay");
+                set(l, "gradientOverlay/colors", Value::Gradient(flat.clone()));
+            }),
+        ),
+        (
+            "Stroke (inside)",
+            Box::new(|p, c, l| {
+                add(p, c, l, "stroke");
+                set(l, "stroke/position", Value::Enum(1));
+                set(l, "stroke/size", Value::Scalar(25.0));
+                set(l, "stroke/color", color.clone());
+            }),
+        ),
+    ];
+    for (name, f) in &styled {
+        let px = edge([1.0; 3], f.as_ref());
+        for (a, b) in px.iter().zip(&plain) {
+            for ch in 0..4 {
+                assert!(near(a[ch], b[ch], 0.01), "{name}: {a:?} against the plain layer {b:?}");
+            }
+        }
+    }
+}
+
 fn bevel_scene(angle: f64) -> Image {
     let (p, cid) = scene([0.5; 3], None, |p, c, l| {
         add(p, c, l, "bevelEmboss");

docs/architecture.md

@@ -207,8 +207,9 @@ switches that don't change pixels (Audio, Lock, Shy, Hide Shy Layers: `Switches:
 **Layer styles** (Layer ▸ Layer Styles; `crates/render/src/styles.rs`) render in layer space and
 may grow the layer's bounds. Drop Shadow and Outer Glow become separate passes composited below the
 layer with their own blend modes; the interior styles, Stroke and Bevel and Emboss are baked into
-the layer body. Layer opacity fades the whole stack; Knockout and the R/G/B channel switches apply
-at composite time. Styled pixels are cached separately from the layer content, so editing a style
+the layer body. Interior styles are clipped to the layer: they blend onto its colour inside its
+shape and keep its alpha, so an anti-aliased edge mixes the style and the background only. Layer
+opacity fades the whole stack; Knockout and the R/G/B channel switches apply at composite time. Styled pixels are cached separately from the layer content, so editing a style
 reuses the cached source/masks/effects. Global Light is one setting per comp, mirrored into every
 layer's Blending Options and kept in step after each edit.
 

docs/parity.md

@@ -408,6 +408,10 @@ Fixes from measured reports:
 - Glyphs that touch or overlap ("ff") are drawn as one outline, so no light seam shows where
   their anti-aliased edges meet (#415). Characters of other colours, with Blur, or under an
   Inter-Character Blending mode are still composited one by one.
+- Interior layer styles (Gradient / Color Overlay, Satin, Inner Glow, Inner Shadow, the inside of
+  Stroke) blend onto the layer's colour inside its alpha, so an anti-aliased edge no longer shows
+  a rim of the layer's own fill (#416). Not yet: Inner Bevel still shades an anti-aliased edge
+  over the layer's own colour.
 
 ## Update: plug-ins and extensions as in After Effects; ease presets become a ScriptUI panel