lightcraft/docs/ai-masks.md

AI masks (Object and Describe)

LightCraft selects objects with SAM 3 (Segment Anything with Concepts, Meta 2025), run in pure Rust by crates/segment on candle: on the GPU through Metal on macOS, on the CPU elsewhere (for now).

The model is optional. It is not part of LightCraft, and nothing requires it: without it, Object and Describe offer to download it, every other feature (and every existing mask, see below) works, and no render, export or command ever waits for it.

Using them

In the Masking panel (M):

• Object: click the thing you want; every click refines the selection. ⌥-click (Alt) a part to leave it out. Clicks show as green (include) and red (leave out) dots.
• Describe: type what to select — sky, trees, the red car — and press Return. Every instance the model finds (score above 0.5) joins the mask; if nothing matches, nothing is created and you are told so.
• Several things at once: car, road (comma-separated) selects each and merges them.
• + / − under the selected mask (and the Add / Subtract / Intersect menus) combine selections: Describe car, click the mask, + ▸ Describe… road; − ▸ Describe… people takes the people out of a sky mask; − ▸ Object then clicks remove one object.
• Edge (per selection, −100…100): below 0 a crisper border, above 0 a feathered one (up to 2 % of the long edge, so previews and exports match).
• Detail: about a second after the last click (or right after a description), the photo around the selection is analyzed again, zoomed in, from a sharper source (~5 s on an M4 Pro, in the background); the result is kept as a high-resolution patch (at most 4 per selection), so small objects get 5–10× finer edges than one pass over the whole photo gives.

The first Object or Describe on a photo analyzes it (the image encoder runs once per photo and look: ~3 s on an Apple M4 Pro, plus a one-time ~6 s for compiling GPU kernels and loading the model after launch). That starts in the background as soon as you pick Object; after it, each click takes ~35 ms and each description ~0.6 s. Everything the model does runs on its own worker thread: the window stays responsive, a click shows its dot at once and the selection follows when it's computed ("Selecting…" / "Analyzing the photo…" under the mask tiles). The model is unloaded after 10 minutes without use (it takes several GB of memory) and loaded again when needed.

The result is stored with the mask: the model's 288 × 288 mask logits over the uncropped photo (quantized, compressed, ~10–30 KB). Renders and exports sample them at any resolution and never need the model, so masks survive moving the library to a machine without it, and the web build renders them. Editing the photo's look later does not recompute a mask (click again to update).

Remote Metal inference

Native builds with sam can keep the editor, RAW files, library, and exports local while a Mac runs SAM 3. Build both ends from the same revision. On the Mac, install the model as described below, then run:

cargo build --release --locked -p lightcraft-segment --bin lightcraft-sam3-worker
target/release/lightcraft-sam3-worker "$HOME/Library/Application Support/LightCraft/models/sam3" 127.0.0.1:8793

On the editing computer, start an SSH tunnel in one terminal and LightCraft in another:

ssh -N -T -o ExitOnForwardFailure=yes -L 127.0.0.1:8793:127.0.0.1:8793 user@your-mac
LIGHTCRAFT_SAM3_REMOTE=127.0.0.1:8793 lightcraft

Object clicks, Describe prompts, and detail refinement use the remote worker. Only the rendered model input (long edge at most 1008 pixels) and prompts cross the tunnel. The worker caches the current photo; detail crops do not replace that cache. Returned masks stay in the local library, so later adjustments and exports work without the connection.

The worker requires Metal and accepts one editing connection at a time. It loads weights on demand and releases them when the client disconnects or remains idle for ten minutes. Both endpoints require loopback addresses: SSH provides authentication and encryption. The listener has no independent authentication; do not expose it through a public proxy. Any process of any user on the worker Mac can connect to it, and on the editing computer a local process listening on the port while the tunnel is down receives the rendered model input, so use this only on machines whose local users you trust.

No local weights are required in remote mode. An unavailable worker produces an explicit error, not a silent CPU fallback. Restore the tunnel and retry. Unset LIGHTCRAFT_SAM3_REMOTE to use the local model again. This offloads segmentation only, not RAW development, denoising, or upscaling. The model licence below still applies.

Getting the model

In the app (recommended)

The first time you pick Object or Describe without the model, LightCraft asks:

Download the SAM 3 model? Object and Describe masks use SAM 3, Meta's segmentation model. It isn't part of LightCraft, and everything else works without it. Download it once (about 3.4 GB) · folder. Licence: SAM License (Meta) — Meta's terms, not LightCraft's. Downloading it means accepting them. [Read the SAM License] — Not Now / Download

Nothing is downloaded until you choose Download. The download runs in the background (close the dialog and keep working; the Masking panel shows its progress), can be cancelled, resumes where it stopped if interrupted (also after quitting), and tries the next download location when one fails or stalls. Each file is written as <name>.part and moved into the model folder only when complete and verified — model.safetensors must have the exact size and SHA-256 of Meta's official checkpoint; a file that doesn't match is deleted, never used. If no location works, the dialog says why and LightCraft carries on without the model.

Agents and scripts use the same commands (control channel, lightcraft-cli mcp --connect): segment.model.status, segment.model.download {"acknowledged": true} (refused without the explicit acknowledged: pass it only after the user agreed to the download and the licence), segment.model.cancel.

Download locations

The model is fetched from an ordered list of mirrors: base URLs where <base>/model.safetensors, <base>/vocab.json and <base>/merges.txt live. The list is, in order:

1. LIGHTCRAFT_SAM3_MIRRORS (environment variable; URLs separated by commas or spaces);
2. the file models/sam3-mirrors.txt in LightCraft's settings folder (one URL per line, # comments) — ~/Library/Application Support/LightCraft/ (macOS), %APPDATA%\LightCraft\ (Windows), ~/.config/lightcraft/ (Linux);
3. the built-in list, DEFAULT_MIRRORS in crates/segment/src/fetch/mod.rs.

Maintainers: the built-in list is empty until LightCraft's own CDN locations exist (see the TODO(maintainer) there): add them in order of preference, host the three files unchanged, and pin vocab.json / merges.txt (size + SHA-256) in SAM3_FILES at the same time. Until then the in-app download needs a user-configured mirror, and the dialog says so: it names the three files to put in the model folder by hand, with a button that shows that folder (created if needed) and a link to this guide.

Hugging Face's facebook/sam3 can't be a default: it is gated (each person must accept the licence there, wait for approval and download with their own token). Someone with access can use it as their mirror:

LIGHTCRAFT_SAM3_MIRRORS=https://huggingface.co/facebook/sam3/resolve/main \
LIGHTCRAFT_SAM3_TOKEN=hf_... lightcraft

(LIGHTCRAFT_SAM3_TOKEN is sent as a bearer token only over https, and only to the mirror's own host — not to the storage host it redirects to.)

The download is plain HTTP/1.1 over TLS in pure Rust (rustls with the RustCrypto provider and the Mozilla root certificates; no OpenSSL, ring or aws-lc, nothing compiled from C), with a 15 s connect timeout and a 30 s stall timeout; it follows redirects but doesn't use HTTP proxies.

By hand

Put model.safetensors, vocab.json and merges.txt from facebook/sam3 in the model folder: models/sam3/ in the settings folder above, or the folder LIGHTCRAFT_SAM3_DIR names. Developers can use the installer scripts, which download from Hugging Face with your token and verify the weights:

HF_TOKEN=hf_... tools/install-sam3.sh            # macOS, Linux
tools/install-sam3.sh --check                     # verify an installation

$env:HF_TOKEN = "hf_..."; .\tools\install-sam3.ps1   # Windows

Licence of the model

LightCraft's code is MIT OR Apache-2.0; crates/segment (a port of Apache-2.0 code) is Apache-2.0. The SAM 3 weights are neither: they are Meta's, released under the SAM License (https://github.com/facebookresearch/sam3/blob/main/LICENSE), a custom licence that is not an OSI open-source licence. It grants a non-exclusive, worldwide, non-transferable, royalty-free licence to use, reproduce, distribute, copy, modify and create derivative works of the "SAM Materials", subject to its terms, including: anyone redistributing them must provide a copy of the licence with them, and derivative works are distributed under its terms; uses subject to ITAR, military or warfare purposes, nuclear industries, espionage, guns or illegal weapons, and reverse engineering are prohibited, and trade controls (sanctions, export rules) apply; the materials come "as is", without warranty; Meta may terminate the licence on a breach (then the materials must be deleted); California law governs. Read the licence itself — this summary is not legal advice.

Therefore:

• The weights are never part of this repository, its releases or installers, and must not be committed.
• They reach a user's computer only when that user asks for them, after the dialog names the licence (or when they install them by hand).
• Whoever hosts a mirror redistributes the SAM Materials and must follow the licence (ship it next to the files).

Commands (control channel, MCP)

Command	Params
mask.add	{kind: "object", points?: [[x,y],…], exclude?: [[x,y],…], seg?}	an Object mask (empty until clicked)
mask.add	{kind: "prompt", text, seg?}	a Describe mask
mask.addComponent	{op: add|subtract|intersect, kind: "object"|"prompt", …}	the same as a component
mask.objectPoint	{x, y, exclude?: bool, id?}	one click on the selected mask's Object selection (≤ 64 clicks)
mask.refineDetail	{id?, component?}	the zoomed-in detail pass → {started}
segment.prepare	{}	load the model and analyze the active photo → {busy}
segment.model.status	{}	{available, installed, remote, dir, loaded, busy, analyzing, sizeBytes, license, licenseUrl, mirrors, download}
segment.model.download	{acknowledged: true}	start the download in the background → {started}
segment.model.cancel	{}	stop it (it resumes next time) → {cancelled}

remote reports the configured endpoint, not its health. installed describes local weights only.

Coordinates are normalized to the uncropped, oriented photo, like every mask shape. seg passes a stored segmentation (as mask.list / the develop settings hold it): the mask is made without the model. Without the model, the AI kinds fail with an error starting "The SAM 3 model is not installed" that says how to get it.

In the desktop app the commands don't wait for the model: mask.objectPoint returns {pending: true} and the selection is applied when it's computed (one undo step per settled selection); mask.add {kind: "prompt"} returns {pending: true} and the mask appears when something is found (or a message says nothing was). lightcraft-cli and the headless MCP server wait and return the result (when built with the sam feature; the default CLI has no AI masks).

Implementation notes

• crates/segment ports the Hugging Face transformers implementation (Apache-2.0; see NOTICE — the ported files say so and that they were modified): the 32-layer ViT backbone with 2-D RoPE and windowed attention, the feature pyramids, the SAM 2-style prompt encoder and two-way mask decoder for clicks, and the CLIP text encoder, DETR encoder/decoder (box relative-position bias, presence token) and pixel decoder for text. The CLIP tokenizer is a small BPE in tokenizer.rs. The downloader (original code) is the lightcraft-fetch crate, shared by any model that is downloaded; fetch/ here says what SAM 3 needs.
• candle is pinned at 0.9.2: later releases make candle-core depend on tokenizers with the Oniguruma C library, and the product is pure Rust. Weights are read with positional reads (no memory map: unsafe stays in lightcraft-sysmem), only the tensors a path needs, and every tensor's byte range is checked against the file's length when it is opened (a truncated or damaged file is an error, never a crash).
• The engine (crates/engine/src/segment/) owns one worker thread with the model and the last encoded photo; the session sends it jobs and receives results over channels, never holding a lock the model holds. Each job runs under catch_unwind: a panic inside candle becomes an error and the model is reloaded on the next request; busy counters are reset by drop guards.
• Hostile documents: a selection keeps at most 4 detail patches (when read and when rendered), each at most 1024² logits; more than 64 clicks are never sent to the model.
• Accuracy: crates/segment/tests/reference.rs compares every stage against the Python reference (tools/sam3_reference.py); on Metal the backbone matches to 7e-5 relative error and masks to ~5e-5. f16 was tried and rejected (overflow, ~7 % faster).
• The engine feature sam (on in the desktop app, off for web and CLI) gates the model; without it the commands report that the build has no AI masks.

Developers: the accuracy test against the Python reference needs torch and transformers (tools/requirements-sam3-reference.txt):

python3 -m venv .venv-sam3 && .venv-sam3/bin/pip install -r tools/requirements-sam3-reference.txt
.venv-sam3/bin/python tools/sam3_reference.py photo.jpg ref.safetensors
LIGHTCRAFT_SAM3_DIR=<model dir> LIGHTCRAFT_SAM3_REF=ref.safetensors cargo test -p lightcraft-segment --release -- --nocapture