mirror of
https://github.com/storytold/effectcraft.git
synced 2026-10-14 19:27:05 -04:00
vangel76andClaude Opus 5.5
a8de199aca
Problem: Object Matte had no After Effects comparison, and none can be made on this Linux machine (no After Effects). The gap notes still listed the viewer tool, background job and disk cache as missing. Fix: docs/fidelity/object-matte.md states plainly that nothing is measured yet and pins down the comparison for when an After Effects 26.2+ reference is available: four original or openly licensed clips (the synthetic moving disk from the engine tests, the disk with a jump, two objects with a box prompt, one CC0/CC BY real clip), the prompt and settings, where reference mattes stay (gitignored), and the metrics: per-frame IoU against After Effects (headline), IoU against ground truth, boundary distance, soft-edge error, temporal stability, drift and time per frame, plus a proposed pass bar left to the maintainers. Linked from docs/fidelity/README.md. docs/gaps.md (#14) and docs/target-app-parity.md now list what landed and leave the comparison as the one remaining item. Co-Authored-By: Claude Opus 5.5 <[email protected]>
74 lines
4.5 KiB
Markdown
74 lines
4.5 KiB
Markdown
# Object Matte: comparison plan (not measured yet)
|
||
|
||
**Status: untested.** There is no After Effects 26.2+ comparison of Object Matte yet. Nothing on
|
||
this page is a measurement. It lists what to compare once an After Effects reference session
|
||
is available, so the first run can be repeated and scored without guesswork.
|
||
|
||
EffectCraft side, as of #783: the Object Matte effect (`ec.matte.objectmatte`), the Object Matte
|
||
tool in the Composition viewer, `objectMatte.apply` / `.propagate` / `.freeze` /
|
||
`.status`, and segmentations kept in the disk cache. The point (and an optional box) is
|
||
segmented by Roto Brush's engine: MobileSAM when it is installed (Settings ▸ Roto Brush, with
|
||
the box as the model's box prompt), otherwise the classical graph cut. The result is tracked
|
||
frame by frame with optical flow, and the model re-prompts each frame from the carried matte.
|
||
Record which engine was active, because the two are separate cases.
|
||
|
||
## Test clips
|
||
|
||
All original, generated or openly licensed; none of them goes into the repository.
|
||
|
||
1. **Synthetic disk**: the moving textured disk over a textured background from
|
||
`crates/engine/src/tests_roto.rs` (`draw(center(f))`, 320×240, 25 fps, 25 frames, radius
|
||
40 px, about 4 px a frame to the right). Export it as a PNG sequence. Its ground truth is
|
||
analytic (`gt(center(f))`).
|
||
2. **Synthetic disk with a jump**: the same disk with `jumping(f)` (it jumps at frame 8). This
|
||
checks how each tool handles losing the object.
|
||
3. **Two objects**: the disk plus a second still, differently coloured square that touches it
|
||
on some frames, prompted once with a point only and once with a box around the disk. This
|
||
checks the box prompt.
|
||
4. **Real footage**: one openly licensed clip (CC0 or CC BY) with a person or object moving
|
||
against clutter, at least 1080p and 5 s long. Record its URL, licence and SHA-256 next to the
|
||
results, and keep the clip itself out of the repository.
|
||
|
||
## Prompt and settings
|
||
|
||
- Reference frame 0. The point at the object's centre, in layer pixels (clip 1: `[90, 110]`).
|
||
The box, where used, is `[centre − 55, centre + 55]` on each axis.
|
||
- Track forward to the last frame (Frames After = clip length − 1, Frames Before = 0).
|
||
- Matte refinement at its defaults on both sides: Feather 0, Contrast 0, Shift Edge 0, no
|
||
Invert. Output the matte itself (View ▸ Matte in EffectCraft, the alpha channel in After
|
||
Effects) as a full-resolution PNG sequence.
|
||
- Record the After Effects build, the OS, the GPU, and the exact click position in After
|
||
Effects. It is a viewer click, so convert it to layer pixels with the comp's zoom.
|
||
|
||
Keep the After Effects mattes in gitignored `plan/aftereffects/ref/object-matte/`. Only an
|
||
aggregate results file (like `keyframe-ease.json`) and the comparison script may be committed.
|
||
|
||
## Metrics (per frame, then mean / worst)
|
||
|
||
- **IoU against After Effects**: binarize both mattes at alpha 0.5, then intersection over
|
||
union. This is the headline number. `effectcraft_track::roto::iou` computes it, and
|
||
`roto.status` already reports it for Roto Brush through `compareTo`; Object Matte needs the
|
||
same for exported mattes.
|
||
- **IoU against ground truth** for the synthetic clips, for both applications. It tells
|
||
"differs from After Effects" apart from "worse than After Effects".
|
||
- **Boundary error**: the mean and 95th-percentile distance (px) between the two mattes' 0.5
|
||
contours, which shows edge disagreement that IoU dilutes on large objects.
|
||
- **Soft-edge error**: the mean absolute alpha difference inside a 3 px band around the After
|
||
Effects contour.
|
||
- **Temporal stability**: the frame-to-frame IoU change of each matte on clip 1, where the
|
||
object moves rigidly, so jitter shows.
|
||
- **Drift**: the frame where IoU against ground truth first drops below 0.9 (clips 1 and 2).
|
||
- **Time**: seconds per frame to track the clip, warm and cold, on the same machine, with the
|
||
disk cache on and off.
|
||
|
||
Proposed pass bar (a maintainer decision, not yet agreed): mean IoU against After Effects
|
||
≥ 0.95 and worst ≥ 0.90 on clips 1 and 3, with MobileSAM. With the classical engine the numbers
|
||
are reported but not gated.
|
||
|
||
## Reproduce (to write)
|
||
|
||
The script does not exist yet. It should: generate the synthetic clips; run EffectCraft
|
||
headless (`effectcraft-cli run … objectMatte.apply … objectMatte.propagate {"wait":true}`, then
|
||
render the matte); drive After Effects through ExtendScript or AEsync to apply Object Matte with
|
||
the same prompt and render its alpha; then compute the metrics above into a JSON results file.
|