DX12: load the shader compiler from System32 or beside the exe, never from PATH (#712) (#817)

wgpu's default DX12 compiler (`Auto`) loads `dxcompiler.dll` by name, and the Windows DLL search
falls through to the current directory and PATH, so another program's DXC build (a Chrome or SDK
folder) was loaded and crashed shader compilation or failed device creation. The app now uses FXC
(d3dcompiler_47.dll, part of Windows) unless a dxcompiler.dll ships beside photocraft.exe, which is
loaded by its full path. WGPU_DX12_COMPILER still overrides.

Co-authored-by: Claude Opus 5.5 <[email protected]>

apps/photocraft/src/gpu_startup.rs

@@ -186,6 +186,22 @@ pub fn backends(plan: &Plan, os: Os) -> Option<wgpu::Backends> {
     })
 }
 
+/// The DX12 shader compiler (#712). wgpu's default loads `dxcompiler.dll` by name, and the
+/// Windows DLL search falls through to the current directory and `PATH`, so another program's DXC
+/// build (a browser's, an SDK's) got loaded and failed device creation or crashed the first shader
+/// compile. Use a `dxcompiler.dll` shipped beside the executable, by its full path; otherwise FXC
+/// (`d3dcompiler_47.dll`, part of Windows). `WGPU_DX12_COMPILER` still picks one explicitly.
+pub fn dx12_compiler(env: Option<&str>, exe_dir: Option<&Path>) -> wgpu::Dx12Compiler {
+    if let Some(c) = env.and_then(|v| v.trim().parse().ok()) {
+        return c;
+    }
+    let beside_exe = exe_dir.map(|d| d.join("dxcompiler.dll")).filter(|p| p.is_file());
+    match beside_exe.as_deref().and_then(Path::to_str) {
+        Some(p) => wgpu::Dx12Compiler::DynamicDxc { dxc_path: p.to_string() },
+        None => wgpu::Dx12Compiler::Fxc,
+    }
+}
+
 /// What adapter selection needs to know about an adapter.
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
 pub struct Candidate {
@@ -237,14 +253,19 @@ pub fn intel_dx12_applied(adapters: &[Candidate], chosen: usize, os: Os) -> bool
         && adapters.iter().any(|a| a.vendor == INTEL && a.backend == wgpu::Backend::Vulkan)
 }
 
-/// Configure eframe's wgpu setup for `plan`: instance backends, adapter selection (Intel on
-/// Windows, software adapters for `cpu`) and the marker update once the adapter is chosen.
-/// `note` receives a remark for System Info (e.g. the Intel default).
+/// Configure eframe's wgpu setup for `plan`: instance backends, the DX12 shader compiler, adapter
+/// selection (Intel on Windows, software adapters for `cpu`) and the marker update once the
+/// adapter is chosen. `note` receives a remark for System Info (e.g. the Intel default).
 pub fn configure(setup: &mut egui_wgpu::WgpuSetup, plan: &Plan, os: Os, sentinel: SharedSentinel, note: Arc<Mutex<Option<String>>>) {
     if let egui_wgpu::WgpuSetup::CreateNew(create) = setup {
         if let Some(b) = backends(plan, os) {
             create.instance_descriptor.backends = b;
         }
+        if os == Os::Windows {
+            let exe = std::env::current_exe().ok();
+            let env = std::env::var("WGPU_DX12_COMPILER").ok();
+            create.instance_descriptor.backend_options.dx12.shader_compiler = dx12_compiler(env.as_deref(), exe.as_deref().and_then(Path::parent));
+        }
         let select = plan.env.is_none() && (plan.backend == GpuBackend::Cpu || (plan.backend == GpuBackend::Auto && os == Os::Windows));
         if select {
             let backend = plan.backend;
@@ -606,4 +627,36 @@ mod tests {
         assert_eq!(read_rendering_prefs(Some(&p)), (Auto, RenderingMode::Auto));
         let _ = std::fs::remove_dir_all(&dir);
     }
+
+    /// #712: never a `dxcompiler.dll` found through the DLL search path (current directory, PATH).
+    #[test]
+    fn dx12_compiler_is_fxc_or_dxc_beside_the_exe() {
+        use wgpu::Dx12Compiler as C;
+        let dir = temp_dir("dxc");
+        std::fs::create_dir_all(&dir).unwrap();
+        assert!(matches!(dx12_compiler(None, None), C::Fxc));
+        assert!(matches!(dx12_compiler(None, Some(&dir)), C::Fxc));
+        std::fs::write(dir.join("dxcompiler.dll"), b"MZ").unwrap();
+        let expected = dir.join("dxcompiler.dll");
+        assert!(matches!(dx12_compiler(None, Some(&dir)), C::DynamicDxc { dxc_path } if Path::new(&dxc_path) == expected));
+        // An explicit WGPU_DX12_COMPILER wins; an unknown value is ignored.
+        assert!(matches!(dx12_compiler(Some(" FXC "), Some(&dir)), C::Fxc));
+        assert!(matches!(dx12_compiler(Some("dxc"), None), C::DynamicDxc { dxc_path } if dxc_path == "dxcompiler.dll"));
+        assert!(matches!(dx12_compiler(Some("quantum"), None), C::Fxc));
+        let _ = std::fs::remove_dir_all(&dir);
+
+        // The app's setup replaces wgpu's default (`Auto`: DXC by name from the search path).
+        let mut setup = egui_wgpu::WgpuSetup::without_display_handle();
+        let plan = plan(Auto, None, None, false, Os::Windows);
+        configure(&mut setup, &plan, Os::Windows, Arc::new(Mutex::new(None)), Arc::default());
+        let egui_wgpu::WgpuSetup::CreateNew(create) = &setup else { panic!("a new instance") };
+        if std::env::var_os("WGPU_DX12_COMPILER").is_none() {
+            let safe = match &create.instance_descriptor.backend_options.dx12.shader_compiler {
+                C::Fxc => true,
+                C::DynamicDxc { dxc_path } => Path::new(dxc_path).is_absolute(),
+                C::StaticDxc | C::Auto => false,
+            };
+            assert!(safe, "{:?}", create.instance_descriptor.backend_options.dx12.shader_compiler);
+        }
+    }
 }

docs/development.md

@@ -42,6 +42,8 @@ CRAFT_FONTS_DIR="$PWD/../craft-fonts" cargo test --workspace     # runs the Japa
 
 `--safe-gpu` starts with the CPU renderer for one launch (no GPU canvas; a software adapter for the window where the platform has one: WARP on Windows, llvmpipe over GL on Linux). Before creating the wgpu device the app writes and locks `gpu-starting.json` in the config directory; it clears it once the first frames have rendered. A launch that finds an unlocked marker knows the previous start died inside the graphics driver (#4) and uses the next safer backend (Windows: Vulkan → DX12 → CPU; Linux: Vulkan → GL → CPU; macOS: Metal → CPU), remembering it in `performance.gpuBackend` (Preferences › Performance › GPU Backend, with **Reset GPU Backend**). With `auto`, Intel adapters on Windows use DX12. Help › System Info shows the adapter, backend, driver and fallback state.
 
+On DX12 the shader compiler is FXC (`d3dcompiler_47.dll`, part of Windows), or a `dxcompiler.dll` placed beside `photocraft.exe`, loaded by its full path. wgpu's default looks `dxcompiler.dll` up by name, which reaches the current directory and `PATH` and loaded other programs' incompatible builds (#712).
+
 If the device is lost while running (#243), every GPU entry point checks the device's health flag first, the canvas switches to the CPU compositor for the rest of the session and a notice says "GPU device was lost; using the CPU renderer." `ui.gpu.simulateLoss` triggers this path from the control channel.
 
 ## Environment variables
@@ -55,6 +57,7 @@ If the device is lost while running (#243), every GPU entry point checks the dev
 | `PHOTOCRAFT_AUTOMATION_WRITE_ROOT` | Separate directory capability for automation writes; requests use relative paths |
 | `PHOTOCRAFT_CPU_CANVAS=1` | Force the CPU canvas path instead of the wgpu shader canvas |
 | `WGPU_BACKEND=dx12` | Pick the wgpu backend(s) (`vulkan`, `dx12`, `metal`, `gl`); overrides `performance.gpuBackend` and the startup fallback |
+| `WGPU_DX12_COMPILER=fxc` | DX12 shader compiler (`fxc`, `dxc`, `auto`); `dxc` and `auto` look `dxcompiler.dll` up through the DLL search path |
 | `PHOTOCRAFT_GPU_TILE=2048` | Force GPU canvas tiling (tests tile seams) |
 | `PHOTOCRAFT_FX_NOCACHE=1` | Bypass the CPU layer-effect map cache (`compose::effect_maps`) |
 | `PHOTOCRAFT_CPU_COMPOSE=1` | Keep the wgpu canvas but composite on the CPU (compare GPU vs CPU renders, e.g. with the snapshot example) |