M13.30: Content-Aware Fill in a job worker on its GPU

contentFill.generate sends its plan (FillPlan) to a browser job worker:
the layer's frames render in passes on the worker's WebGPU device, the PNG
sequence goes back to browser storage (Services::store_file), and the page
adds the fill layer when the plan returns (WorkerReply::Fill). The smoke
test checks it; docs updated.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Brandon ThomasandClaude Opus 5.5 committed 2026-10-04 03:15:54 -04:00
1 parent 91f4a603a9
commit 3921e6c680
10 files changed
+350 -71

No files matched your search

+5 -3
View File
@@ -151,7 +151,7 @@ the disk cache in the formats wave; the 3D Camera Tracker in M12.6.)
| Compositions | ≈ 81% | 3.3 | Mocha-style planar tracks for templates (CMP-7: Essential Graphics mirrored and linked properties landed in M13.11; Font and uniform Scale controls, Composition ▸ Open in Essential Graphics, Save Frame As ▸ Photoshop Layers / ProEXR and the VR Comp Editor landed in M13.7; the marker dialog, Composition Flowchart, Essential Graphics with master properties, `.ectemplate` templates and Responsive Design — Time landed: CMP-6, CMP-7) |
| Animation | 70% | 9.0 | puppet depth beyond pins, recording, rigging and follow-through (puppet pin recording with Record Options landed in M13.1; pin selection, rotate/scale handles, nulls for pins and Follow-Through in M13.14–M13.16), Wiggler/Smoother/Motion Sketch (motion-path handles and the graph editor transform box landed in M5.8; keyframe colour labels and Select Keyframe Label Group, Graph Editor snapping to markers / layer ends in M13.5) |
| Text | ≈ 96% | 0.3 | no extruded strokes (M13.12: the Variable Font Axes animator re-spaces the text — advances follow the animated axes; M13.6: variable font axes in the character style — `layer.setText variations`, the Character panel's Variable Font Axes fields — shape with HVAR / gvar advances and draw at that design-space position; OpenType features — stylistic sets, discretionary ligatures, contextual / stylistic alternates, swash, titling, ordinals, fractions, figure styles, true small caps / all small caps and superior / inferior glyphs with faux fallback — per character with the Character panel's OpenType popup and `text.fontFeatures` landed in M13.2; vertical Roman / Tate-Chu-Yoko, forced LTR paragraphs, caret on animated and path text, Variable Font Axes and Lottie style runs landed in M13.5; extruded, bevelled text in M7.6; per-character styles, paragraph settings, on-canvas editing and the `sourceText` style API in M9.9–M9.10) |
| Web | 98% | 0.1 | No shared-memory threads inside one engine instance (the decided design is one engine instance per worker; a threaded build needs nightly `build-std`); Content-Aware Fill runs on the page (M13.30: Render Queue and analyses render on each job worker's own WebGPU device in passes, particles and Advanced 3D read back under keys, layer buffers in the browser's disk cache; browser storage, Web Audio, Web Worker renders/analyses, WebGPU viewer and offline install landed in M15.2; viewer frames in frame workers fed by project diffs, Roto Brush propagation in a worker, non-blocking `wait: true` jobs and a browser Media Browser (File System Access folders, browser storage) in M13.10; WEB-1 in M13.24: GPU effects and the GPU compositor in the frame workers on their own WebGPU devices with deferred readbacks (frames render in passes; Backend Auto per comp), the disk cache in the Origin Private File System (written by the workers with sync access handles, LRU under the settings' limit, served after a reload), the storage manager (Settings ▸ Disk ▸ Browser Storage, `storage.info` / `storage.persist` / `storage.clear`), and a WGSL constant Chrome rejected (it disabled every GPU kernel in Chrome) fixed) |
| Web | 98% | 0.1 | No shared-memory threads inside one engine instance (the decided design is one engine instance per worker; a threaded build needs nightly `build-std`) (M13.30: Render Queue, analyses and Content-Aware Fill render on each job worker's own WebGPU device in passes, particles and Advanced 3D read back under keys, layer buffers in the browser's disk cache; browser storage, Web Audio, Web Worker renders/analyses, WebGPU viewer and offline install landed in M15.2; viewer frames in frame workers fed by project diffs, Roto Brush propagation in a worker, non-blocking `wait: true` jobs and a browser Media Browser (File System Access folders, browser storage) in M13.10; WEB-1 in M13.24: GPU effects and the GPU compositor in the frame workers on their own WebGPU devices with deferred readbacks (frames render in passes; Backend Auto per comp), the disk cache in the Origin Private File System (written by the workers with sync access handles, LRU under the settings' limit, served after a reload), the storage manager (Settings ▸ Disk ▸ Browser Storage, `storage.info` / `storage.persist` / `storage.clear`), and a WGSL constant Chrome rejected (it disabled every GPU kernel in Chrome) fixed) |
| 3D | 88% | 5.5 | multi-view layouts, the Extended Viewer for Advanced 3D comps (Classic 3D Extended Viewer landed in M13.5 UI completion); collapsed precomps of another size seen through the parent's camera render (fixed in M13.2); stereo rigs, orbit nulls, lights controlled by the camera, cameras/lights from glTF models, environment backgrounds, Advanced 3D motion blur, blend modes and track mattes landed in M7.7; Classic 3D iris-shaped bokeh with highlights, progressive depth of field on tilted layers and the focus-link commands landed in M13.5; Advanced 3D depth of field with the iris and highlight options, collapsed precomps as real Advanced 3D geometry and extruded text/shape strokes landed in M13.8; Advanced 3D (glTF/OBJ models, primitives, extruded text and shapes, PBR, image-based light, shadow maps, GPU rasteriser) in M7.4–M7.6; Advanced 3D end to end on the GPU (motion blur, iris depth of field, compositing) in M13.11 |
| Effects | ≈ 85% | 3.0 | GPU versions of the remaining effects (243 run on the GPU since M13.29, EFF-5: M13.29 added the particle effects (CC Particle World / Systems II, Particle Playground, CC Ball Action, CC Pixel Polly, CC Scatterize, Curl Noise) and removed the fallbacks of Shatter's wireframe views, Foam's extras, Timewarp's Matte Layer, strong Warp Fisheye / Twist bends and custom `.ocio` configs; M13.23 added the 3D Channel and Immersive Video families, Apply Color LUT, the OCIO effects, Color Profile Converter and the simulations' render passes (Card Wipe included); M13.22 added the CC light family, the CC transitions, Block Dissolve, Radial Shadow, CC Bender / Blobbylize / Cylinder / Sphere / Spotlight / Environment, 3D Glasses, Numbers, Timecode and the time effects joined the 166 of M13.13; audio-driven effects still render on the CPU) and the missing controls listed as partial in [effects.md](effects.md) (every After Effects effect exists since M9.11, M12.5 and M12.6; parameter names, order, twirl-downs, popups, units and defaults were aligned in M9.12) |
| Interface | 75% | 5.0 | more Learn tutorials and pixel-level fidelity of dialogs (the Home ▸ Templates gallery (eight original built-in templates, user templates from File ▸ Save as Template…) and View ▸ Simulate Output ▸ My Custom RGB… landed in M13.25; Timeline layer reordering by drag, a non-snapping viewer pan, a working rename field and twirl arrows, and the full set of property reveal shortcuts — double presses, Alt+Shift keyframes, Ctrl+` — landed in M13.17–M13.20; the Home ▸ Learn tab with interactive tutorials and a UI fidelity pass landed in M13.5 UI completion; visual editors for Lumetri RGB / hue-saturation curves, Colorama's output cycle wheel, Glow's colour map and Reshape's correspondence points (viewer handles), a scrolling Preview panel and AE-style Composition / Timeline tabs (close, label swatch, viewer lock) landed in M13.10; Timeline outline and Project panel columns scroll horizontally, the Layer Style dialog, ROI resize handles, Pan Behind snapping and 3D Reference Axes landed in M13.5; native macOS menu bar, Timeline columns/search/reveal-add, Home screen with recent projects and all AE workspaces landed; viewer rulers/snapping/channels/snapshots landed in M0.13) |
@@ -496,8 +496,10 @@ are pinned bit for bit by golden hashes (`crates/effects/tests/particle_golden.r
tracking and Roto Brush (prefetched before each step) use the GPU. GPU particles and
Advanced 3D (`raster_3d`, `render_3d`) read back under keys, so they run on the device in
frame and job workers too (a depth-of-field run still resolves on the CPU, its scenes on
the GPU). `effectcraft.info().workers` reports the job workers' adapter and each job's passes
and readbacks; the headless-Chrome smoke test measures them.
the GPU). Content-Aware Fill became a job-worker job too (its files go back to browser
storage; before, it ran on the page and could not write its folder). `effectcraft.info().workers`
reports the job workers' adapter and each job's passes and readbacks; the headless-Chrome
smoke test measures them.
- **Layer buffers in the browser's disk cache**: frame workers back their layer caches with a
`PrefetchStore`; the page plans which layer entries to read before a request from the
previous frames' misses (`LayerPrefetch` over the shared `DiskIndex`, one LRU order for
+13 -5
View File
@@ -52,7 +52,7 @@ device of their own), `?nosw` (don't register the service worker).
|---|---|
| frame render thread pool (`ui-egui/src/frames.rs`) | frame workers (below), each with its own WebGPU device: `Frames::pump` hands queued frames to them, most important first (the viewer's frame, then prefetch); frames without effects that the page's GPU compositor can keep on the GPU render on the page. Without workers, queued frames render on the UI thread after each egui frame within 40 ms (24 ms while playing) |
| Render Queue on a background thread | a Web Worker running a second engine instance with its own WebGPU device (below); progress streams back, the page never waits |
| tracker / mask tracker / Warp Stabilizer / 3D Camera Tracker / Roto Brush Freeze threads | the same workers; the analysed property group comes back as one undo step |
| tracker / mask tracker / Warp Stabilizer / 3D Camera Tracker / Roto Brush Freeze threads, Content-Aware Fill tasks | the same workers; the analysed property group (or the fill layer) comes back as one undo step |
| `rayon` parallel loops (raster, effects, export batches) | rayon's global pool falls back to the calling thread when threads are unavailable: same code, serial |
| `std::time::Instant` / `SystemTime` (panic on wasm32) | `web-time` (re-exports `std::time` on native) |
| `std::fs` project reads/writes (`FsServices`) | `files::WebServices`: the virtual file table, persisted (below); saving also downloads the `.ecproj` |
@@ -103,7 +103,8 @@ compiled twice) and its own memory (`crates/engine/src/offload.rs`, `src/worker.
1. The session's `Offload` serializes a `WorkerRequest`: the project JSON, the footage paths it
reads, and the job (`render` with the resolved queue items and output paths; `warp`, `camera`,
`track`, `maskTrack`, `rotoFreeze` with the target and analysis parameters).
`track`, `maskTrack`, `rotoFreeze`, `rotoPropagate` with the target and analysis parameters;
`contentFill` with its plan).
2. `js/host.js` takes an idle worker from its pool (or starts one), sends it the footage bytes it
doesn't have yet, then the request.
3. The worker runs the job on a plain engine session and posts `WorkerReply`s: render progress
@@ -185,8 +186,16 @@ the same futures with `passes::block_on`; they never wait there).
render on the device in frame workers and job workers alike.
`effectcraft.info().workers` reports `{running, idle, gpu: {adapter} | {cpu: why}, jobs:
[{kind, gpu, passes, readbacks, ms}]}` (the last jobs). Content-Aware Fill runs on the page
(it is not a worker job).
[{kind, gpu, passes, readbacks, ms}]}` (the last jobs).
**Content-Aware Fill in a job worker (M13.30).** `contentFill.generate` (the panel's Generate
Fill Layer) sends its plan (`content_fill::FillPlan`: layer, frames, method, folder) to a job
worker (`WorkerJob::ContentFill`, `jobs.list` id `contentFill`): the worker renders the
layer's frames in passes on its device, fills them and writes the PNG sequence through its
services, which send each file to the page's browser storage (`{type: "store"}` messages, under
`/Fill/<layer>_Fill_<n>/`); then the plan comes back (`WorkerReply::Fill`) and the page adds
the fill layer as one undo step. (Before, the fill ran on the page's thread and could not
write its folder.)
**Threading design: one engine instance per worker (decided, M13.24).** Shared-memory threads
inside one instance (SharedArrayBuffer + atomics, a `wasm-bindgen-rayon`-style pool) need the
@@ -434,7 +443,6 @@ layer buffers through the disk store and the prefetch plan (`remote.rs`
project replica and the footage it reads); see the threading design above.
- Browsers without WebGPU in workers render frames, Render Queue jobs and analyses on the
workers' CPU. A frame renders at most 24 passes on the GPU, then on the CPU.
- Content-Aware Fill runs on the page's thread (it is not offloaded to a job worker).
- A layer buffer is fetched from disk only after some worker missed it in a recent frame (the
first frame after a reload renders its layers).
- Cancelling an analysis drops its partial result (the worker is terminated).