#783: Object Matte fidelity plan; gaps and parity notes updated

Problem: Object Matte had no After Effects comparison, and none can be made
on this Linux machine (no After Effects). The gap notes still listed the
viewer tool, background job and disk cache as missing.

Fix: docs/fidelity/object-matte.md states plainly that nothing is measured
yet and pins down the comparison for when an After Effects 26.2+ reference
is available: four original or openly licensed clips (the synthetic moving
disk from the engine tests, the disk with a jump, two objects with a box
prompt, one CC0/CC BY real clip), the prompt and settings, where reference
mattes stay (gitignored), and the metrics: per-frame IoU against After
Effects (headline), IoU against ground truth, boundary distance, soft-edge
error, temporal stability, drift and time per frame, plus a proposed pass
bar left to the maintainers. Linked from docs/fidelity/README.md.
docs/gaps.md (#14) and docs/target-app-parity.md now list what landed and
leave the comparison as the one remaining item.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
vangel76andClaude Opus 5.5 committed 2026-10-11 14:07:22 +02:00
1 parent 3466538d0e
commit a8de199aca
4 files changed
+80 -4

No files matched your search

+3
View File
@@ -3,6 +3,9 @@
The [AE 2026 replacement acceptance gates](ae-2026-acceptance.md) define the broader
target across workflows, rendered output, work-project compatibility, performance and platforms.
[Object Matte](object-matte.md) has a comparison plan (clips, prompt, per-frame IoU and the other
metrics) but no measurement yet.
The first G1 comparison measures eight original scalar Rotation cases against a live After Effects
2026 session through AEsync 2.0.4: Easy Ease, overlapping influences, asymmetric ease, overshoot,
descending values, equal endpoints, 100%/100% influence and 0.1%/100% influence. Each case has
+73
View File
@@ -0,0 +1,73 @@
# Object Matte: comparison plan (not measured yet)
**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on
this page is a measurement. It lists what to compare once an After Effects reference session
is available, so the first run can be repeated and scored without guesswork.
EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte
tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` /
`.status`, and segmentations kept in the disk cache. The point (and an optional box) is
segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with
the box as the model's box prompt), otherwise the classical graph cut. The result is tracked
frame by frame with optical flow, and the model re-prompts each frame from the carried matte.
Record which engine was active, because the two are separate cases.
## Test clips
All original, generated or openly licensed; none of them goes into the repository.
1. **Synthetic disk**: the moving textured disk over a textured background from
`crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius
40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is
analytic (`gt(center(f))`).
2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This
checks how each tool handles losing the object.
3. **Two objects**: the disk plus a second still, differently coloured square that touches it
on some frames, prompted once with a point only and once with a box around the disk. This
checks the box prompt.
4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving
against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the
results, and keep the clip itself out of the repository.
## Prompt and settings
- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`).
The box, where used, is `[centre − 55, centre + 55]` on each axis.
- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0).
- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no
Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After
Effects) as a full-resolution PNG sequence.
- Record the After Effects build, the OS, the GPU, and the exact click position in After
Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom.
Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an
aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed.
## Metrics (per frame, then mean / worst)
- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over
union. This is the headline number. `effectcraft_track::roto::iou` computes it, and
`roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the
same for exported mattes.
- **IoU against ground truth** for the synthetic clips, for both applications. It tells
"differs from After Effects" apart from "worse than After Effects".
- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5
contours, which shows edge disagreement that IoU dilutes on large objects.
- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After
Effects contour.
- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the
object moves rigidly, so jitter shows.
- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2).
- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the
disk cache on and off.
Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects
≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers
are reported but not gated.
## Reproduce (to write)
The script does not exist yet. It should: generate the synthetic clips; run EffectCraft
headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then
render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with
the same prompt and render its alpha; then compute the metrics above into a JSON results file.