0 Fork 0

mirror of https://github.com/storytold/effectcraft.git synced 2026-10-14 19:27:05 -04:00

Files

T

BFlatandClaude Opus 5.5 928117db25 Fix #368 : tflite rejects models whose scratch or work exceeds a budget

Model::read bounded each tensor but nothing else, so a tiny hostile
.tflite could make Model::run abort on a huge CONV_2D patch allocation,
or keep it busy for minutes. Read now works out from the static shapes,
with checked arithmetic, what one run takes and refuses a model over a
fixed budget: kernel side (32), one operator's working space (2^22
floats), multiply-adds (2^33), values written (2^30), floats alive at
once (2^29) and constant values decoded or laid out at load (2^27).

The shapes run() computes itself now have to match the declared ones
(PAD, CONCATENATION, MAX_POOL_2D channels), and a zero dimension no
longer lets the others grow without bound.

Reported by KazW, with the suggested load-time budget.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-10-09 16:33:32 +07:00

src

Fix #368 : tflite rejects models whose scratch or work exceeds a budget

2026-10-09 16:33:32 +07:00

Cargo.toml

M13.36: face tracking trained model — MediaPipe Face Landmarker on a pure-Rust TFLite interpreter

2026-10-05 20:39:45 +07:00

README.md

M13.36: face tracking trained model — MediaPipe Face Landmarker on a pure-Rust TFLite interpreter

2026-10-05 20:39:45 +07:00

README.md

effectcraft-segment

Trained models for Roto Brush and face tracking, each kind behind one swappable interface.

MaskModel: what Roto Brush asks of a model: given a frame and prompts (foreground / background points, a box, a prior mask), the foreground probability of every pixel. Roto Brush (effectcraft-track) only talks to this trait, so models can be added or swapped without touching it. The classical graph-cut segmenter stays built in as the fallback.
face::FaceModel: what face tracking asks of a model: find a face in a region of a frame, then follow it from frame to frame. A model reports its own points plus a Topology naming which are the tracker's landmarks and which trace the face outline, so the tracker never depends on one model's layout. The classical face tracker stays built in as the fallback.
MODELS: the registry. Every entry must be open source under a licence compatible with EffectCraft's MIT OR Apache-2.0 (ALLOWED_LICENCES), with its task, authors, homepage, licence URL, the official weights URL, size and SHA-256. load refuses anything else.
Weights are never bundled. They are downloaded on demand (or installed from a file) by the engine (roto.model.* / face.model.*, Settings ▸ Roto Brush / Face Tracking), verified against the registry's SHA-256 (sha256, dependency-free) and kept with a notice naming their authors, licence and source (notice).
pt: reads PyTorch checkpoints (torch.save zip archives and the pickle subset state dicts use), so official weights load exactly as published.
tflite: reads TensorFlow Lite models (the FlatBuffer laid out by TensorFlow Lite's schema.fbs) and runs the float operators small vision models use (convolutions, depthwise convolutions, pooling, PReLU, padding, concatenation; float16 weights dequantised at load).
mobilesam: MobileSAM in plain Rust: the TinyViT-5M image encoder and Segment Anything's prompt encoder and two-way-transformer mask decoder.
mediapipe: MediaPipe Face Landmarker: the BlazeFace short-range detector (SSD anchors, weighted non-maximum suppression) and Face Mesh V2 (478 points), with the rotated crops MediaPipe's face landmarker graph uses.

All of it runs on nn kernels (rayon, and ndarray's safe general_mat_mul, which picks AVX/FMA kernels at run time). No ML framework, no unsafe code.

Models

id	Task	Model	Authors	Licence	File	Size
`mobilesam`	Roto Brush	MobileSAM: Segment Anything with a 5M-parameter TinyViT encoder	Chaoning Zhang et al., building on Meta AI's Segment Anything and Microsoft's TinyViT	Apache-2.0	`mobile_sam.pt` from the official repository	40.7 MB
`mediapipe-face`	Face tracking	MediaPipe Face Landmarker: BlazeFace detector and Face Mesh V2	Google LLC	Apache-2.0 (model card)	`face_landmarker.task` (float16, version 1) from Google's model storage	3.8 MB

How they were checked:

MobileSAM: the implementation follows the published architectures (MobileSAM, TinyViT and Segment Anything papers and their Apache-2.0 reference code). Its output was checked against an independent implementation (Hugging Face candle's, MIT/Apache-2.0) on the same weights: embeddings within 0.3% RMS (exact vs. tanh GELU) and 99.9% of mask pixels equal.
MediaPipe Face Landmarker: the interpreter's outputs match TensorFlow Lite's own (ai-edge-litert 2.2.0) on both models to about one part in a million (a test pins them). The whole pipeline's 478 points on a test portrait are 0.85 px from MediaPipe's own Python pipeline on average (2.2 px at worst). Finding a face takes about 45 ms; following it, about 20 ms a frame (32-thread CPU).

Tests that need the weights run when EFFECTCRAFT_MOBILESAM / EFFECTCRAFT_FACE_LANDMARKER point at the files, and are skipped otherwise.

Adding a model

Pick a model whose code and weights are open source under an allowed licence. Check the weights' own terms: many models with MIT code ship weights trained on research-only data.
Add a ModelInfo to MODELS (its task and authors; pin the official file's SHA-256 and size), and credit it in the root ATTRIBUTION.md.
Implement MaskModel or FaceModel for it and add its loader to load.
Test it against a reference implementation; Settings, the roto.model.* / face.model.* commands and the tools pick it up from the registry.