From aa4656b54318408ae938405f93490ef28f4cedc6 Mon Sep 17 00:00:00 2001 From: Brandon Thomas Date: Mon, 5 Oct 2026 06:32:19 -0400 Subject: [PATCH] =?UTF-8?q?Docs:=20honest=20gap=20assessment=20(docs/gaps.?= =?UTF-8?q?md),=20roadmap=20workstreams=20G1=E2=80=93G9,=20README=20status?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit parity.md's ≈99% measures breadth only. docs/gaps.md assesses real-world readiness by dimension (fidelity, .aep interop, stability, platform reliability, performance, media, AI tools, maturity) with evidence, and sets prioritised workstreams with done criteria. ROADMAP, README, parity.md, AGENTS.md and CLAUDE.md point agents at it. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 1 + CLAUDE.md | 3 +- README.md | 24 ++++++--- ROADMAP.md | 40 ++++++++++++--- docs/gaps.md | 129 +++++++++++++++++++++++++++++++++++++++++++++++++ docs/parity.md | 8 +++ 6 files changed, 189 insertions(+), 16 deletions(-) create mode 100644 docs/gaps.md diff --git a/AGENTS.md b/AGENTS.md index e27a83fa..b96c7816 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,6 +107,7 @@ a command, everything is agent-drivable, and the quality gates (`cargo xtask ci` ## See also +- [docs/gaps.md](docs/gaps.md): where EffectCraft falls short of After Effects and the prioritised workstreams (G1–G9); read it before choosing work - [CONTRIBUTING.md](CONTRIBUTING.md): setup, gates, commits, how to add things - [docs/architecture.md](docs/architecture.md): layers, data model, commands, pipeline - [docs/testing.md](docs/testing.md): oracle tests, criteria, benchmarks diff --git a/CLAUDE.md b/CLAUDE.md index fa41b323..afa08d3d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,8 @@ EffectCraft is a clean-room, open-source, pure-Rust motion graphics and visual e `plan/` is maintainer-local (gitignored). Public equivalents: [`ROADMAP.md`](ROADMAP.md), [`docs/`](docs/). 1. Read `plan/STATUS.md` (current milestone, next task, running agents, blockers). 2. Read the task in `plan/execution-plan.md` §3, the relevant section of `plan/architecture.md`, and the crate README/docs you touch. Feature ids: `plan/aftereffects/feature-catalog.md`; effects: `plan/aftereffects/effects-list.md`; UI reference: `plan/aftereffects/README.md`. -3. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask. +3. Read [`docs/gaps.md`](docs/gaps.md): the honest assessment of where we fall short and the prioritised workstreams. Fidelity, reliability and stability (G1–G3) come before new features. +4. Follow the autonomous operation protocol (`plan/execution-plan.md` §7). Don't stop to ask. ## Non-negotiables **Read [`AGENTS.md`](AGENTS.md) first; its rules override everything here.** No Adobe assets, every asset licensed + attributed (`cargo xtask assets`). diff --git a/README.md b/README.md index fa1d6fc6..3d9280c1 100644 --- a/README.md +++ b/README.md @@ -195,13 +195,23 @@ tests, layering, asset attribution and the WebAssembly build). See [CONTRIBUTING ## Where it stands -EffectCraft is young. The core of After Effects is in place and gets deeper every week: what is -done and what comes next is in the [ROADMAP](ROADMAP.md). Still to come: learned models for -Roto Brush and face tracking (both use classical methods today), GPU versions of the remaining -CPU-only effects, the Advanced 3D Extended Viewer, and shared-memory threads in the web build. The -web build ([docs/web.md](docs/web.md)) runs the full app in the browser, with renders, analyses -and viewer frames in Web Workers. Projects are saved as `.ecproj`, readable versioned JSON; After -Effects `.aep` files cannot be opened. +EffectCraft is young: its first commit was on 1 October 2026. Nearly every After Effects feature +exists, including all of its effects, but existing is not the same as behaving exactly like After +Effects, and we haven't measured that yet. Today it's a good place to try things and to tell us +what breaks. It isn't yet a replacement for After Effects on client work. In particular: + +- **After Effects projects can't be opened.** Projects are saved as `.ecproj`, readable versioned + JSON, but EffectCraft can't open `.aep` / `.aepx` files, and After Effects plug-ins don't run. +- **Behaviour still differs from After Effects in places.** Nothing yet compares our renders with + After Effects automatically, so please report differences. +- **macOS is the most tested platform.** Linux and Windows users have hit basic interaction + bugs, which we're fixing first. +- **Roto Brush and face tracking use classical methods,** not learned models. + +The [ROADMAP](ROADMAP.md) has the plan and [docs/gaps.md](docs/gaps.md) the full, honest +assessment. The web build ([docs/web.md](docs/web.md)) runs the full app in the browser. +[Bug reports](https://github.com/storytold/effectcraft/issues) are the most useful thing you can +send us right now. ## How it's made diff --git a/ROADMAP.md b/ROADMAP.md index 5936aa9a..f31ab93f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -25,14 +25,38 @@ once. ## How far from full parity -Measured feature by feature in [docs/parity.md](docs/parity.md) (4 October 2026): +There are two honest answers, and they are far apart. The full assessment, with the evidence, is +in [docs/gaps.md](docs/gaps.md). -- **≈ 99% of After Effects' features, weighted by importance**, counting partial features as half - done (≈ 99.8% with per-feature fractions): 89 of 92 features done, 3 partial, none missing; every - essential (P0) feature done; all 306 effects implemented in full, 280 of them on the GPU. -- **≈ 8.5 agent-hours of work remain**: about **2–3 hours of wall-clock time** with five Claude - Opus 5.5 agents in parallel, ≈ 1 hour without learned models for Roto Brush and face tracking. -- What is left: GPU kernels for the remaining visual effects, and learned-model quality for Roto - Brush and face tracking ([parity.md](docs/parity.md#highest-value-gaps-in-order)). +- **Breadth: ≈ 99%.** Almost every After Effects feature exists: 89 of 92 catalogued features + done and 3 partial, all 306 effects implemented, 280 of them on the GPU + ([docs/parity.md](docs/parity.md)). This is our own checklist, graded by the agents that built + the features. +- **Real use: ≈ 30–50% (estimated, not yet measured).** Whether someone who uses After Effects for + a living can do client work in EffectCraft. What holds it back: + - **Fidelity is unmeasured.** No test compares our output with After Effects itself, and users + are still finding behaviour bugs in features marked done. + - **After Effects projects can't be opened.** EffectCraft can't read `.aep` / `.aepx` files, + and third-party After Effects plug-ins can't run. + - **Reliability is uneven across platforms.** Early Linux users hit basic problems: viewer + panning, panel docking, drag-and-drop import. + - **AI tools are behind.** Roto Brush and face tracking use classical methods, not learned + models. + +## Where we're going + +In priority order (details and "done" criteria in [docs/gaps.md](docs/gaps.md)): + +| | Workstream | Why | +|---|---|---| +| G1 | Measure fidelity against After Effects: a corpus of test projects rendered and sampled in both apps, scored per feature | Turns every estimate into a measurement; a feature that behaves differently is not done | +| G2 | Real-user reliability on macOS, Windows and Linux; fix every open user issue | Basic workflows must work everywhere before more features matter | +| G3 | Stability: a green gate on the current toolchain, fuzzing of every input | A crash loses people's work | +| G4 | Open After Effects projects (`.aep` / `.aepx`, pending an owner decision on clean-room scope); relinking moved footage | The biggest barrier to switching | +| G5 | Performance benchmarks at 1080p and 4K on real-world-sized projects | Unknown today | +| G6 | Media depth: encoder efficiency, camera and phone formats | Files are larger than from mature encoders; camera formats are unverified | +| G7 | Learned models for Roto Brush and face tracking (owner decision) | Quality is well below After Effects' | +| G8 | Plug-in ecosystem on our WebAssembly plug-in API | After Effects plug-ins can't run here | +| G9 | Localisation, accessibility, user documentation | Reach beyond English-speaking power users | Come tell us what matters most to you on [Discord](https://discord.gg/artcraft). diff --git a/docs/gaps.md b/docs/gaps.md new file mode 100644 index 00000000..442beb10 --- /dev/null +++ b/docs/gaps.md @@ -0,0 +1,129 @@ +# Where EffectCraft falls short + +An honest assessment of how far EffectCraft is from being a real replacement for After Effects, +and the work that closes the gap. This document is meant for contributors and agents choosing +what to work on. The [ROADMAP](../ROADMAP.md) summarises it; [parity.md](parity.md) is the +feature-by-feature checklist it builds on. + +*Assessed 5 October 2026. Estimates marked "≈" are judgements from the evidence listed, not +measurements. Update this file when the evidence changes.* + +## Two different questions + +[parity.md](parity.md) reports **≈ 99%**. That number answers one question: *does each After +Effects feature exist?* It scores a 92-item catalogue we wrote ourselves, graded by the same agents +that built the features, with partial features counting half. As a measure of breadth it is fair, +and the breadth is real: every one of After Effects' effects exists by name, along with the +menus, panels, 3D, tracking, expressions, scripting and export. + +It does not answer the question users care about: *can someone who uses After Effects for a living +do real work in EffectCraft?* Nobody has measured that yet. Our best estimate is **≈ 30–50%**, +limited mainly by four things: projects we can't open, behaviour nobody has checked against After +Effects, reliability on platforms other than macOS, and the third-party plug-ins professional +projects depend on. + +The checklist is also internally inconsistent: parity.md's per-area table still shows Interface +75%, Project 68%, Animation 70% and Paint 0% next to the 99% headline. Some rows are stale; none +of them have been reconciled. + +## By dimension + +| Dimension | Estimate | Evidence | +|---|---|---| +| Breadth of features | ≈ 95%+ | All 306 effects, the After Effects menus and panels, Classic and Advanced 3D, tracking, expressions, scripting, render queue (parity.md) | +| Behaves like After Effects | unmeasured, ≈ 60–80% | One outside contributor found four bugs in features marked done within a day (PRs #6–#10): Hold keyframes eased the motion into them, `keyInSpatialTangent` / `keyOutSpatialTangent` had the wrong names, a zero frame rate crashed, Find and Enter Full Screen shared Ctrl+F off macOS. There are about 10 After Effects reference captures in total. No test compares our renders or property values with After Effects itself; the tests check what the agents believed After Effects does | +| Opening existing After Effects work | ≈ 0% | `.aep` / `.aepx` projects cannot be opened. Third-party After Effects plug-ins cannot run. Expressions and the scripting object model are strong, so scripts and expressions carry over | +| Stability | improving | 23 never-crash PRs landed on 4 October; [AGENTS.md](../AGENTS.md) "Never crash" now binds every crate. On 5 October `cargo xtask ci` failed on main under Rust 1.99's clippy (a fix is in progress) | +| Real-user experience | ≈ 70%, uneven by platform | Issues #41–#47 (all from the Linux AppImage, 5 October): panning the viewer snaps back, panels can't be resized or rearranged, drag-and-drop and double-click import don't work, layer rename gets stuck, the Layer Settings arrow does nothing, the Project panel clips, the Wayland window icon is generic. parity.md counts docking as done. Most checking happens on macOS. No localisation, no accessibility work | +| Performance | unknown against After Effects | Internal numbers only (e.g. Advanced 3D 290 ms/frame at 1080p on the GPU, an M4 Pro under load). Nothing benchmarked against After Effects; no large real projects (4K footage, hundreds of layers) tested | +| Media formats | ≈ 80% | H.264, ProRes, HEVC, AV1, image sequences and audio exist. The new HEVC / AV1 encoders have no B-frames, multi-reference or SAO / CDEF, so files are larger than from mature encoders. Camera formats (BRAW, R3D, ProRes RAW, variable-frame-rate phone video) are unverified | +| AI-assisted tools | ≈ 50% | Roto Brush and face tracking use classical methods (graph cut + optical flow; a shape-model fitter); quality is well below After Effects' learned models | +| Maturity | early | First commit 1 October 2026. ≈ 285,000 lines of almost entirely agent-written Rust, ≈ 2,050 tests, 8 external issue reports so far. After Effects has around 30 years of edge cases behind it | + +## Where we're going: workstreams in priority order + +Each workstream lists what "done" means, so progress can be measured rather than self-graded. + +### G1. Measure fidelity against After Effects + +The most important missing piece: it turns every other estimate here into a measurement. + +- Build a corpus of original test projects (our own content, no Adobe assets), one or more per + feature id in `plan/aftereffects/feature-catalog.md`: keyframe interpolation of every kind, + blend modes, track mattes, each effect at default and at non-default settings, text animators, + expressions, 3D, motion blur, time remapping. +- Drive After Effects through ExtendScript (see CLAUDE.md) to record property values at sampled + times and render reference frames; render the same projects in EffectCraft headless. +- Compare values exactly and frames with a perceptual metric; publish a per-feature fidelity + score and a fidelity column in parity.md. +- Clean room: After Effects output stays local in `plan/aftereffects/ref/` (gitignored). Commit + only our projects, the harness and the scores, never After Effects frames. +- Done when: every P0 and P1 feature has at least one corpus project, scores are reproducible from + one command, and parity.md reports breadth and measured fidelity side by side. + +### G2. Real-user reliability on every platform + +- Fix the open user issues (#41–#47), each with a regression test. +- Run the headless snapshot and control-channel checks on Linux (X11 and Wayland) and Windows, not + only macOS. Interactions that only fail with real input (docking drags, viewer pan, drag-and-drop + import, inline rename) need scripted input tests. +- Done when: no open bug blocks a basic workflow (import, arrange, animate, preview, render) on + any of the three desktop platforms, and every bug users report gets triaged within a day. + +### G3. Stability and a green main + +- Keep `cargo xtask ci` green on the current stable toolchain. A toolchain release that breaks the + gate is a P0. +- Fuzz the inputs we don't control: project files, imported media and documents, expressions, + scripts, control-channel and MCP requests. Every crash found becomes a regression test + (AGENTS.md "Never crash"). +- Done when: the gate is green on stable and the fuzzers run regularly without new crashes. + +### G4. Open After Effects projects + +- `.aep` / `.aepx` import is the largest single barrier to switching. It needs an **owner + decision** on clean-room scope (whether and how the format may be studied), like the open + `.prproj` question in `plan/STATUS.md`. +- Also in this area: relative footage paths and relinking when a project moves (parity.md, Project). +- Done when: the decision is recorded and, if approved, a corpus of real-world-shaped projects + opens with its comps, layers, keyframes, effects and expressions intact. + +### G5. Performance at real-world scale + +- Benchmark projects at 1080p and 4K with many layers, heavy effects, long footage and nested + comps; record preview frame times, RAM preview fill and render times on reference machines. +- Compare with After Effects on the same machine where possible. +- Done when: benchmark numbers are tracked over time and regressions fail a check. + +### G6. Media depth + +- Encoder efficiency: B-frames and multi-reference for HEVC / AV1, SAO / CDEF / restoration, Opus + FEC / DTX. +- Verify decode of the camera and phone formats people actually bring, including variable frame + rate; document what is not supported. + +### G7. Learned models for Roto Brush and face tracking + +- Already an owner decision in `plan/STATUS.md`: licensed weights with pure-Rust inference, or + accept the classical methods as final. + +### G8. Plug-in ecosystem + +- After Effects SDK plug-ins cannot run in EffectCraft. Our own WebAssembly plug-in API exists + ([plugins.md](plugins.md)). Grow it: documentation, examples, and original effects that cover + what the most common third-party plug-ins are used for (particles, glows, 3D objects, sabers). + +### G9. Reach + +- Localisation of the interface, accessibility (keyboard navigation, screen readers, contrast), + user documentation and tutorials. + +## For agents choosing work + +1. Read `plan/STATUS.md` for owner blockers and running work, then this file. +2. Prefer G1–G3 over new features. A feature that exists but behaves differently from After + Effects is not done. +3. When you fix a behaviour bug in a feature parity.md marks as done, add it to the evidence + above. That is how this assessment stays honest. +4. Don't raise parity.md's numbers without evidence: a G1 score, a user-facing check on more than + one platform, or both. diff --git a/docs/parity.md b/docs/parity.md index 79711127..2191bea6 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -2,6 +2,14 @@ How close EffectCraft is to After Effects 2026, feature by feature, and how much work is left. +> **Read this as a measure of breadth, not of real-world readiness.** It checks whether each +> feature exists, using our own catalogue, graded by the agents that built it. It does not measure +> whether the feature behaves like After Effects (nothing compares the two yet), whether After +> Effects projects open (they don't), or how the app holds up for real users on every platform. +> Those gaps, and the work that closes them, are in [gaps.md](gaps.md). Don't raise the numbers +> here without the evidence gaps.md asks for. The per-area table further down has stale rows that +> have not been reconciled with the headline. + ## Current status (audit at commit `d39c0e8`, 4 October 2026; updated 5 October for M3.9–M4.12) | Measure | Value |