M13.36: face tracking trained model — MediaPipe Face Landmarker on a pure-Rust TFLite interpreter

- effectcraft-segment: a FaceModel interface (find a face in a region, follow it; a Topology
  maps the model's points to the tracker's landmarks and outline), a TensorFlow Lite reader and
  interpreter (float ops, float16 weights folded; matches ai-edge-litert to ~1e-6), and
  MediaPipe Face Landmarker (Google, Apache-2.0: BlazeFace + Face Mesh V2, 3.8 MB, pinned
  SHA-256). The registry gains a task and authors per model; installed weights get a NOTICE.
- effectcraft-track: FaceTracker::new_with uses the model, falls back to the classical fitter
  when it finds no face; chin and jaw come from the outline either way so measurements agree.
- Engine: roto_models becomes models (both tasks); face.models / face.model.* commands,
  Settings ▸ Face Tracking, `wait` on select/install/download for scripts; track.mask reports
  faceModel.
- Credits: ATTRIBUTION.md lists the optional models and the Apache-2.0 specs the code follows.
- Docs: README, ROADMAP (G7), gaps, parity (TRK-3 score unchanged pending G1), agents,
  preferences, architecture, segment README.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
bflatasticandClaude Opus 5.5 committed 2026-10-05 20:39:45 +07:00
1 parent 4624c00308
commit e43bf66775
33 files changed
+2298 -485

No files matched your search

+6 -2
View File
@@ -37,7 +37,7 @@ of them have been reconciled.
| Real-user experience | ≈ 70%, uneven by platform | Issues #41–#47 (all from the Linux AppImage, 5 October): panning the viewer snaps back, panels can't be resized or rearranged, drag-and-drop and double-click import don't work, layer rename gets stuck, the Layer Settings arrow does nothing, the Project panel clips, the Wayland window icon is generic. parity.md counts docking as done. Most checking happens on macOS. No localisation, no accessibility work |
| Performance | unknown against After Effects | Internal numbers only (e.g. Advanced 3D 290 ms/frame at 1080p on the GPU, an M4 Pro under load). Nothing benchmarked against After Effects; no large real projects (4K footage, hundreds of layers) tested |
| Media formats | ≈ 80% | H.264, ProRes, HEVC, AV1, image sequences and audio exist. The new HEVC / AV1 encoders have no B-frames, multi-reference or SAO / CDEF, so files are larger than from mature encoders. Camera formats (BRAW, R3D, ProRes RAW, variable-frame-rate phone video) are unverified |
| AI-assisted tools | ≈ 50% | Face tracking uses a classical shape-model fitter. Roto Brush 2.0 / 3.0 can use MobileSAM since M13.35 (pure-Rust inference, an optional download): on our synthetic moving-disc test it scores IoU 0.989 on the base frame and ≥ 0.980 over 20 propagated frames (classic: 0.973), but nothing compares it with After Effects yet |
| AI-assisted tools | ≈ 50% | Both tools can use trained models (pure-Rust inference, optional downloads), but nothing compares them with After Effects yet. Roto Brush 2.0 / 3.0 with MobileSAM (M13.35) scores IoU 0.989 on the base frame of our synthetic moving-disc test and ≥ 0.980 over 20 propagated frames (classic: 0.973). Face tracking with MediaPipe Face Landmarker (M13.36) matches Google's own pipeline to 0.85 px on average on a test portrait; on our synthetic clip the eyes and chin stay within 4% of the face height |
| Maturity | early | First commit 1 October 2026. ≈ 285,000 lines of almost entirely agent-written Rust, ≈ 2,050 tests, 8 external issue reports so far. After Effects has around 30 years of edge cases behind it |
## Where we're going: workstreams in priority order
@@ -110,7 +110,11 @@ The most important missing piece: it turns every other estimate here into a meas
open-source licences, MobileSAM under Apache-2.0), weights downloaded on demand and verified,
Settings ▸ Roto Brush to choose. Next: measure it against After Effects (G1); faster encoders
(GPU) for long shots.
- Face tracking: still an owner decision (a learned detector, or the classical fitter as final).
- Face tracking: decided the same way on 5 October 2026, done in M13.36: MediaPipe Face
Landmarker (Google, Apache-2.0: BlazeFace detector and the 478-point Face Mesh V2) on a pure-Rust
TensorFlow Lite interpreter, behind a `FaceModel` interface in the same crate, a 3.8 MB optional
download chosen in Settings ▸ Face Tracking. The classical fitter stays as the fallback. Next:
measure both against After Effects (G1).
### G8. Plug-in ecosystem