mirror of
https://github.com/storytold/effectcraft.git
synced 2026-10-14 19:27:05 -04:00
M15.2: web app depth (WEB-1): browser storage, Web Audio, worker renders, WebGPU viewer, PWA
- Persistence: settings, shortcuts, recent projects, imported media, saved projects and auto-saves live in OPFS (IndexedDB fallback) through an in-memory mirror flushed in the background (apps/effectcraft-web/src/store.rs, persist.rs); ConfigStore gains FileOps so auto-save / recovery work in the browser store; the session snapshot restores the last project + editor state on reload; File > Open Recent works from storage. - Audio: Web Audio output (AudioWorklet, ScriptProcessor fallback) fed by AudioPlayback::pump on wasm32; A/V sync from AudioContext.currentTime; meters; unlock on first gesture. - Background jobs: engine::offload (WorkerRequest/WorkerReply protocol, Offload trait, run_request); Render Queue, tracker, mask tracker, Warp Stabilizer and Roto Freeze run in Web Workers (a second engine instance on the shared compiled module), progress streamed back; cancel terminates the worker. - GPU: WebGPU viewer frames drawn straight from the compositor's texture (gpu_display on wasm32 when the viewer draws plainly); async readback (Gpu::read_display_async) for pixels. - PWA: manifest, service worker precaching the build (xtask fills list/version), icons copied from assets/app-icon; COOP/COEP kept from cached responses; offline reload works. - window.effectcraft: saveToBrowser, listStored, removeStored, flush; info() reports storage, audio, gpu, workers. - Tests: native store + protocol + offload (render, track, cancel) tests; smoke.mjs extended (GPU path, worker render responsiveness, audio gesture + playback, reload persistence, Open Recent, offline reload). Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Brandon ThomasandClaude Opus 5.5 committed
1 parent
c406b71cba
commit
0bee1ab2da
41 files changed
+3230
-319
No files matched your search
+137
-30
@@ -2,9 +2,10 @@
|
||||
|
||||
`apps/effectcraft-web` runs the same engine and egui UI as the desktop app in the browser: the
|
||||
workspace compiled to `wasm32-unknown-unknown`, started by eframe's web runner on WebGPU (WebGL2
|
||||
fallback). It is a static site: one `.wasm`, its `wasm-bindgen` JavaScript glue, `index.html` and an
|
||||
icon. Nothing is uploaded anywhere; projects, media and renders stay in the page and on the user's
|
||||
machine.
|
||||
fallback). It is a static site: one `.wasm`, its `wasm-bindgen` JavaScript glue, `index.html`, a
|
||||
few small scripts (worker, audio worklet, service worker), a web manifest and icons. Nothing is
|
||||
uploaded anywhere: projects, media, settings and renders stay in the browser (its private storage)
|
||||
and on the user's machine.
|
||||
|
||||
## Build and run
|
||||
|
||||
@@ -15,48 +16,136 @@ machine.
|
||||
| wasm-opt (optional) | any recent binaryen | `brew install binaryen`; used by `cargo xtask web` when found |
|
||||
|
||||
```sh
|
||||
cargo xtask web # release build → <target>/web/dist (index.html, effectcraft_web.js, effectcraft_web_bg.wasm, favicon.svg)
|
||||
cargo xtask web # release build → <target>/web/dist
|
||||
cargo xtask web --dev # unoptimised build (faster to compile, slow to run)
|
||||
cargo xtask web --serve 8765 # build, then serve dist on http://127.0.0.1:8765/
|
||||
```
|
||||
|
||||
`<target>` is `$CARGO_TARGET_DIR` or `target`. `--serve` is a tiny localhost-only static server that
|
||||
sends `Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy`, so the page is cross-origin
|
||||
`<target>` is `$CARGO_TARGET_DIR` or `target`. `dist` holds `index.html`, `effectcraft_web.js` +
|
||||
`effectcraft_web_bg.wasm` (+ `snippets/`, the browser glue `js/host.js`), `worker.js`,
|
||||
`audio-worklet.js`, `sw.js` (its precache list and version filled in by the build),
|
||||
`manifest.webmanifest`, `favicon.svg` and `icon-256.png` / `icon-512.png` (copied from
|
||||
`assets/app-icon`). `--serve` is a tiny localhost-only static server that sends
|
||||
`Cross-Origin-Opener-Policy` / `Cross-Origin-Embedder-Policy`, so the page is cross-origin
|
||||
isolated like a production deployment should be. Any static server works
|
||||
(`python3 -m http.server -d target/web/dist 8765`); it must serve `.wasm` as `application/wasm`.
|
||||
Service workers need a secure context: `https://`, or `http://localhost` / `127.0.0.1`.
|
||||
|
||||
The release `.wasm` is about 31 MB before `wasm-opt` (20 MB code: the effects, codecs, the
|
||||
expression engine and the UI; 8.5 MB data, mostly the bundled fonts; 2 MB function names for
|
||||
readable panics). Served compressed it is a fraction of that.
|
||||
The release `.wasm` is about 38 MB before `wasm-opt` (code: the effects, codecs, the expression
|
||||
engine and the UI; data, mostly the bundled fonts; function names for readable panics). Served
|
||||
compressed it is a fraction of that, and the service worker caches it after the first visit.
|
||||
|
||||
The web crate is empty on non-wasm targets, so `cargo test --workspace` and clippy are unaffected;
|
||||
`cargo xtask wasm` (part of `cargo xtask ci`) checks it, with every L0–L4 crate and
|
||||
`effectcraft-ui-egui`, for `wasm32-unknown-unknown`.
|
||||
On other targets the web crate holds only its portable storage model (`store.rs`, unit-tested by
|
||||
`cargo test --workspace`); `cargo xtask wasm` (part of `cargo xtask ci`) checks the whole crate,
|
||||
with every L0–L4 crate and `effectcraft-ui-egui`, for `wasm32-unknown-unknown`.
|
||||
|
||||
URL flags: `?empty` (start without the demo project), `?home` (show the start screen).
|
||||
URL flags: `?empty` (a blank project; the session is not restored or recorded), `?demo` (the demo
|
||||
project instead of the last session), `?home` (show the start screen), `?storage=indexeddb` /
|
||||
`?storage=memory` (force a storage backend), `?noworkers` (render and analyse on the page's
|
||||
thread), `?nosw` (don't register the service worker).
|
||||
|
||||
## How the desktop pieces map to the browser
|
||||
|
||||
| Desktop | Web |
|
||||
|---|---|
|
||||
| frame render thread pool (`ui-egui/src/frames.rs`) | `Frames::pump`: queued frames render on the UI thread after each egui frame, most important first (the viewer's frame, then prefetch), within 40 ms (24 ms while playing) |
|
||||
| Render Queue on a background thread | `Session::start_render` always runs inline on wasm32: the page waits until the render is done |
|
||||
| Render Queue on a background thread | a Web Worker running a second engine instance (below); progress streams back, the page never waits |
|
||||
| tracker / mask tracker / Warp Stabilizer / Roto Brush Freeze threads | the same workers; the analysed property group comes back as one undo step |
|
||||
| `rayon` parallel loops (raster, effects, export batches) | rayon's global pool falls back to the calling thread when threads are unavailable: same code, serial |
|
||||
| `std::time::Instant` / `SystemTime` (panic on wasm32) | `web-time` (re-exports `std::time` on native) |
|
||||
| `std::fs` project reads/writes (`FsServices`) | `files::WebServices`: an in-memory file table; saving downloads the `.ecproj` |
|
||||
| `std::fs` project reads/writes (`FsServices`) | `files::WebServices`: the virtual file table, persisted (below); saving also downloads the `.ecproj` |
|
||||
| config directory: settings, shortcut presets, recent projects, the recovery sentinel | `store::WebConfig` in browser storage |
|
||||
| auto-save folder | `/EffectCraft Auto-Save/` in browser storage (`ConfigStore::files`) |
|
||||
| media reads (`MediaPool`, `probe`) | `files::WebImporter`: bytes from the file table, handed to `MediaPool::add_bytes` / `probe_bytes` |
|
||||
| export writes (`effectcraft-export`) | the job's `sink` (`effectcraft_host::FileExporter { sink }`): files land in the table and download after the render; several files (an image sequence) download as one stored `.zip` |
|
||||
| rfd file dialogs | `<input type=file>` for File ▸ Open / Import; drop files anywhere on the page (`.ecproj` opens, everything else imports); "Save As" / "Output To" pick a download name |
|
||||
| system fonts | not scanned; the bundled fonts (Inter, Noto Serif, JetBrains Mono) are always there |
|
||||
| TCP control channel / MCP | `window.effectcraft` (below) |
|
||||
| audio output | none yet (if `cpal` lands for desktop playback it must stay behind a cfg/feature off wasm32; a WebAudio backend would replace it) |
|
||||
| cpal audio output | Web Audio (below) |
|
||||
| GPU compositor on the desktop's wgpu device | the same compositor on eframe's WebGPU device (below) |
|
||||
|
||||
### Threads
|
||||
### Persistence
|
||||
|
||||
wasm threads need a cross-origin isolated page **and** a wasm build with atomics (a nightly
|
||||
`build-std` toolchain), so this build is single-threaded: frames, effects and exports run on the UI
|
||||
thread. A long Render Queue job freezes the page until it finishes; keep web renders short or
|
||||
small (resolution, time span).
|
||||
Everything that lives in files on the desktop lives in the browser's **Origin Private File
|
||||
System** (OPFS); where OPFS can't be written from the page (older Safari), **IndexedDB**; with
|
||||
neither (some private windows), memory only. `effectcraft.info().storage.backend` says which.
|
||||
The page asks for persistent storage (`navigator.storage.persist()`), which browsers grant to
|
||||
installed apps, so the store isn't evicted under storage pressure.
|
||||
|
||||
The engine's storage traits are synchronous and the browser's are not, so the app works on an
|
||||
in-memory mirror (`store.rs`): every stored entry is read before the app starts, reads and writes
|
||||
hit the mirror, and changes flush in the background (one write per changed key, in order; OPFS
|
||||
writes are atomic: a swap file replaced on close). Keys:
|
||||
|
||||
| Key | What |
|
||||
|---|---|
|
||||
| `config/prefs.json`, `config/shortcuts.json` | Settings (with File ▸ Open Recent's list) and keyboard shortcut presets |
|
||||
| `config/session.lock` | the crash-recovery sentinel |
|
||||
| `config/session.ecproj`, `config/session.json` | the session snapshot: the open project and editor state (active comp, time, selection), written at most once a second while they change |
|
||||
| `files/<path>` | the file table: imported media, projects saved with File ▸ Save / Save As or `saveToBrowser`, auto-saves |
|
||||
|
||||
On the next visit the app reopens the snapshot (unsaved changes stay unsaved: the project is
|
||||
dirty), registers every stored media file with the media pool so its footage decodes, and File ▸
|
||||
Open Recent reopens saved projects from storage. Without a snapshot (first visit) it opens the
|
||||
demo project. Render outputs are not stored: they download.
|
||||
|
||||
### Background jobs: Web Workers
|
||||
|
||||
wasm threads need shared memory, which needs a nightly `build-std` toolchain, so this build is
|
||||
single-threaded per instance. Long jobs run in **Web Workers** instead, each with its own instance
|
||||
of the same module (the page compiles it once and posts the `WebAssembly.Module`, so nothing is
|
||||
compiled twice) and its own memory (`crates/engine/src/offload.rs`, `src/worker.rs`):
|
||||
|
||||
1. The session's `Offload` serializes a `WorkerRequest`: the project JSON, the footage paths it
|
||||
reads, and the job (`render` with the resolved queue items and output paths; `warp`, `track`,
|
||||
`maskTrack`, `rotoFreeze` with the target and analysis parameters).
|
||||
2. `js/host.js` takes an idle worker from its pool (or starts one), sends it the footage bytes it
|
||||
doesn't have yet, then the request.
|
||||
3. The worker runs the job on a plain engine session, blocking, and posts `WorkerReply`s: render
|
||||
progress (at most every 100 ms), item started/finished, rendered files (transferred), analysis
|
||||
progress, the analysed property group, `failed`, `done`.
|
||||
4. The page applies them every frame: `Session::poll_render` updates the queue items and the
|
||||
progress bar, `Session::poll_offload` writes an analysis result as one undo step (the effect /
|
||||
tracker / mask group is replaced; other edits made meanwhile are kept) and the downloads start.
|
||||
|
||||
Stopping a job terminates its worker (a worker can't be interrupted mid-frame): the item being
|
||||
rendered becomes "User Stopped", an analysis writes nothing. `renderQueue.render {wait: true}` (the
|
||||
default for agents) and analysis commands with `wait: true` still run on the page's thread and
|
||||
return when done. Roto Brush propagation fills the session's segmentation cache rather than the
|
||||
project, so it runs on the page's thread too, as do viewer frames (`Frames::pump`).
|
||||
|
||||
### Audio
|
||||
|
||||
Audio preview goes through **Web Audio**: an AudioWorklet (`audio-worklet.js`; a
|
||||
ScriptProcessorNode where worklets are unavailable) plays interleaved stereo blocks. The mixdown
|
||||
is the desktop's (`AudioPlayback`, `mix_comp`): without a feeder thread, `AudioPlayback::pump`
|
||||
mixes ahead on every UI frame (≈ 350 ms queued) and the output device takes what keeps it
|
||||
≈ 150 ms ahead of the playhead. A/V sync follows `AudioContext.currentTime`: the playback clock
|
||||
is the frames played by the context clock minus its output latency, and the viewer shows the
|
||||
frame under the audible sample, as on the desktop. The Audio panel's meters read the same feed.
|
||||
|
||||
Browsers start an AudioContext suspended until the user interacts with the page; the app resumes
|
||||
it on the first pointer or key event, so audio plays from the first preview after any click or
|
||||
key press.
|
||||
|
||||
### GPU
|
||||
|
||||
eframe starts on WebGPU where the browser has it (WebGL2 otherwise), and the GPU compositor
|
||||
(`effectcraft-gpu`) runs on the same device: viewer frames are composited by compute shaders and
|
||||
drawn straight from their texture, with no readback. Steps that need the CPU mid-render (3D runs,
|
||||
adjustment layers, CPU-only effects) fall back to the CPU renderer for that frame, because the
|
||||
page's thread can't wait for a GPU readback. The Info panel's pixel readout and the eyedroppers
|
||||
read GPU frames back asynchronously (`Gpu::read_display_async`: the pixels arrive a frame
|
||||
later). On WebGL2, or without a usable adapter, everything renders on the CPU.
|
||||
`effectcraft.info().gpu` reports `{compositor, viewerOnGpu}`.
|
||||
|
||||
### Offline and install (PWA)
|
||||
|
||||
`manifest.webmanifest` (name, icons, standalone display) makes the app installable; `sw.js`
|
||||
precaches every file of the build into a cache named after the build's hash, serves them
|
||||
cache-first (the cached responses keep the server's COOP/COEP headers, so the page stays
|
||||
cross-origin isolated offline), and drops older caches when a new build activates. After the
|
||||
first visit the app loads without a network.
|
||||
|
||||
## `window.effectcraft`: the agent / test API
|
||||
|
||||
@@ -70,34 +159,52 @@ with the method's `result`, errors reject with the message.
|
||||
| `effectcraft.commands()` / `effectcraft.inspect()` | `engine.commands` / `ui.inspect` |
|
||||
| `effectcraft.renderFrame(params)` | `render.frame` with `base64: true`: `{comp, time, width, height, png}` |
|
||||
| `effectcraft.screenshot(params)` | `ui.screenshot`; without `path` the PNG comes back inline as `png` (base64) |
|
||||
| `effectcraft.addFile(fileOrUrl, name?)` | put a `File`/`Blob` (or fetched URL) into the file table and open (`.ecproj`) or import it; resolves with its path |
|
||||
| `effectcraft.addFile(fileOrUrl, name?)` | put a `File`/`Blob` (or fetched URL) into the file table (stored) and open (`.ecproj`) or import it; resolves with its path |
|
||||
| `effectcraft.files()` / `effectcraft.readFile(path)` | the file table `[{path, size}]` / a file's bytes (`Uint8Array`), e.g. a render |
|
||||
| `effectcraft.info()` | graphics backend, version, `crossOriginIsolated`, load timings |
|
||||
| `effectcraft.saveToBrowser(path?)` | save the project to browser storage without downloading it (default: its path, or `/<name>.ecproj`); resolves with `{path, bytes}` |
|
||||
| `effectcraft.listStored()` | `{backend, usage, quota, persisted, pending, files: [{path, size, modified}], config: [name]}` |
|
||||
| `effectcraft.removeStored(path)` | delete a stored file |
|
||||
| `effectcraft.flush()` | resolves once every change is written to browser storage |
|
||||
| `effectcraft.info()` | graphics backend, `gpu`, `storage`, `audio` (`{state, sampleRate, backend, posted, played, underruns}`), `workers`, `restored`, `webgpu`, `serviceWorker`, version, `crossOriginIsolated`, load timings |
|
||||
|
||||
`window.effectcraftLoad` holds `{wasmMs, readyMs}` (module fetch + compile, and until the app runs).
|
||||
|
||||
```js
|
||||
await effectcraft.execute("layer.newSolid", {color: "#ff8800"});
|
||||
await effectcraft.request("ui.menu.invoke", {id: "renderQueue.add", params: {format: "gif", resolution: 0.25}});
|
||||
await effectcraft.execute("renderQueue.render", {}); // the GIF downloads
|
||||
await effectcraft.execute("renderQueue.render", {wait: false}); // renders in a worker; the GIF downloads
|
||||
await effectcraft.saveToBrowser("/my-project.ecproj"); // File ▸ Open Recent has it after a reload
|
||||
```
|
||||
|
||||
## Browser test
|
||||
|
||||
`apps/effectcraft-web/tests/smoke.mjs` drives headless Chrome over the DevTools protocol (Node ≥ 22,
|
||||
no npm packages): load, the demo comp in the viewer, `render.frame`, Render Queue GIF and PNG
|
||||
sequence (`.zip`) downloads, Save As and re-opening the saved project. It writes screenshots and
|
||||
`report.json`:
|
||||
no npm packages): load, the GPU path (WebGPU → GPU compositor, viewer on the GPU), the demo comp
|
||||
in the viewer, `render.frame`, Render Queue GIF and PNG sequence (`.zip`) downloads, a background
|
||||
render in a worker while measuring the page's event-loop gaps (must stay under 400 ms) and its
|
||||
progress, the AudioContext starting on a (simulated) key press and a tone playing through it
|
||||
with the meters moving, Save As and re-opening the saved project, persistence across a reload
|
||||
(session snapshot, a stored project, imported media, a setting, Open Recent) and an offline
|
||||
reload through the service worker. It writes screenshots and `report.json`:
|
||||
|
||||
```sh
|
||||
cargo xtask web --serve 8765 &
|
||||
node apps/effectcraft-web/tests/smoke.mjs --url http://127.0.0.1:8765/ --out target/web/smoke
|
||||
```
|
||||
|
||||
Native unit tests cover the storage model (`apps/effectcraft-web/src/store.rs`: write
|
||||
coalescing, file table, `ConfigStore` / `FileOps`, auto-save and crash recovery through the
|
||||
browser store) and the worker protocol (`crates/engine/src/offload.rs`: serde round trips of
|
||||
every request and reply, a render through an in-process offload).
|
||||
|
||||
## Gaps
|
||||
|
||||
- Single-threaded (above); Render Queue renders block the page.
|
||||
- No audio playback.
|
||||
- Imported media live in memory for the session only (nothing is persisted; reload = fresh start).
|
||||
- Viewer frames, RAM preview and Roto Brush propagation still render on the page's thread (one
|
||||
frame at a time between UI frames).
|
||||
- GPU effects run only in the desktop GPU path: in the browser, effects run on the CPU (inside
|
||||
workers for renders) because the page's thread can't wait for mid-render readbacks.
|
||||
- Cancelling an analysis drops its partial result (the worker is terminated).
|
||||
- Stored media count against the origin's storage quota; `effectcraft.listStored()` shows usage,
|
||||
`removeStored` frees it (there is no storage manager in the UI yet).
|
||||
- `engine.execute` rejects the spread Render Settings / Output Module keys of `renderQueue.add`
|
||||
(`resolution`, `timeSpan`…) through its strict parameter check; use `ui.menu.invoke` as above.
|
||||
Reference in new issue
Block a user