mirror of
https://github.com/storytold/effectcraft.git
synced 2026-10-14 19:27:05 -04:00
M13.36: face tracking trained model — MediaPipe Face Landmarker on a pure-Rust TFLite interpreter
- effectcraft-segment: a FaceModel interface (find a face in a region, follow it; a Topology maps the model's points to the tracker's landmarks and outline), a TensorFlow Lite reader and interpreter (float ops, float16 weights folded; matches ai-edge-litert to ~1e-6), and MediaPipe Face Landmarker (Google, Apache-2.0: BlazeFace + Face Mesh V2, 3.8 MB, pinned SHA-256). The registry gains a task and authors per model; installed weights get a NOTICE. - effectcraft-track: FaceTracker::new_with uses the model, falls back to the classical fitter when it finds no face; chin and jaw come from the outline either way so measurements agree. - Engine: roto_models becomes models (both tasks); face.models / face.model.* commands, Settings ▸ Face Tracking, `wait` on select/install/download for scripts; track.mask reports faceModel. - Credits: ATTRIBUTION.md lists the optional models and the Apache-2.0 specs the code follows. - Docs: README, ROADMAP (G7), gaps, parity (TRK-3 score unchanged pending G1), agents, preferences, architecture, segment README. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
4624c00308
commit
e43bf66775
33 files changed
+2298
-485
No files matched your search
+19
-4
@@ -207,6 +207,20 @@ orientation X/Y/Z, eye openness, eyebrow distance from eye, mouth openness, widt
|
||||
those keys on the keyframe clipboard and returns them per frame plus a tab-separated `clipboard`
|
||||
text.
|
||||
|
||||
Face tracking's trained model works like Roto Brush's (below): `face.models` lists the built-in
|
||||
tracker and MediaPipe Face Landmarker with its authors and licence; `face.model.download {id}` /
|
||||
`face.model.install {path, id?}` install it (verified); `face.model.select {id:
|
||||
classical|mediapipe-face}` chooses it; `face.model.remove {id}` deletes it. Downloads and loading
|
||||
run in the background; `"wait": true` on download, install or select returns once the model is in
|
||||
use (or with its error), which scripts need before tracking. `track.mask` with a
|
||||
face method returns `faceModel`, the engine it tries first (the classical one when the model finds
|
||||
no face in the mask).
|
||||
|
||||
```sh
|
||||
effectcraft-cli run clip.ecproj face.model.download '{"id":"mediapipe-face","wait":true}' face.model.select '{"id":"mediapipe-face","wait":true}' \
|
||||
track.mask '{"layer":"#1","mask":1,"method":"faceDetailed","direction":"forward","wait":true}' --save
|
||||
```
|
||||
|
||||
`mask.interpolate` (Window ▸ Mask Interpolation ▸ Apply) adds in-between Mask Path keys between
|
||||
each pair of selected Mask Path keys (or the existing keys at `times`, in seconds), giving both
|
||||
ends the same vertex count with a matched correspondence. Options: `keyframeRate` (number or
|
||||
@@ -296,17 +310,18 @@ reference. Matte settings are ordinary properties (`effects/#1/rotoBrushMatte/se
|
||||
`effects/#1/refineEdgeMatte/decontaminateEdgeColors`, …).
|
||||
|
||||
Trained model (Roto Brush 2.0 / 3.0):
|
||||
- `roto.models` lists the built-in engine and every registered model, with its licence, size,
|
||||
URL, SHA-256 and whether it is installed, selected and active.
|
||||
- `roto.models` lists the built-in engine and every registered model, with its authors,
|
||||
licence, size, URL, SHA-256 and whether it is installed, selected and active.
|
||||
- `roto.model.download {id}` fetches and verifies the official weights in the background;
|
||||
`roto.model.install {path, id?}` installs a file you already have.
|
||||
- `roto.model.select {id: classical|mobilesam}` chooses which one Roto Brush uses.
|
||||
- `roto.model.select {id: classical|mobilesam}` chooses which one Roto Brush uses. Loading runs
|
||||
in the background; `"wait": true` (also on download and install) returns once it is in use.
|
||||
- `roto.model.remove {id}` deletes the installed weights.
|
||||
- The `version` property (`effects/#1/version`: 0 = 1.0, 1 = 2.0, 2 = 3.0) picks between the
|
||||
classic engine (1.0) and the chosen model (2.0, 3.0).
|
||||
|
||||
```sh
|
||||
effectcraft-cli run clip.ecproj roto.model.install '{"path":"mobile_sam.pt"}' roto.model.select '{"id":"mobilesam"}' \
|
||||
effectcraft-cli run clip.ecproj roto.model.install '{"path":"mobile_sam.pt"}' roto.model.select '{"id":"mobilesam","wait":true}' \
|
||||
roto.stroke '{"layer":"#1","points":[[300,200],[360,230]],"radius":10}' roto.propagate '{"layer":"#1","wait":true}' --save
|
||||
```
|
||||
|
||||
|
||||
Reference in new issue
Block a user