mirror of
https://github.com/storytold/lightcraft.git
synced 2026-10-14 19:27:01 -04:00
LR-PREF-PERFORMANCE: GPU fallback reason per render; the export fallback test no longer reads process-wide state (#675)
`export_falls_back_to_the_cpu_when_gpu_work_is_lost` asserted on `lightcraft_gpu::last_fallback()`, one value for the whole process that every other engine test's GPU render overwrites between the export and the assertion (out of device memory under parallel load, once "a GPU render panicked"). It also switched the GPU off for the whole process through the injected fault and cleared everyone's state with `reset_failures()`. A render now reports its own fallback: `lightcraft_gpu::try_render` returns `Result<Rendered, Fallback>` (`NotTried` when the GPU was not used, `Failed(reason)` when it was tried and the CPU must render), and `render` is a thin `.ok()` over it. `media::develop_reporting` passes the reason on as `RenderResult::gpu_fallback`, an export carries it as `Exported::gpu_fallback`, and each file in an `app.export` batch result has `gpuFallback` when its render fell back. `last_fallback()` stays as the process-wide diagnostic for `ui.inspect` → `perf.gpuFallback`. A simulated failure (`inject_fault`) is thread-local, and so are its consequences now: a fatal injected fault stops GPU use on the injecting thread only (`available()` false, `unavailable_reason()` set there); `reset_failures()` clears that and nothing else, so neither reaches the other tests in the process. Real failures still stop the whole process. The engine test asserts on its own export's reason and checks that another thread still has the GPU; `crates/gpu/tests/fallback.rs` uses `try_render` and checks the same; a new batch test checks `gpuFallback` lands on the file whose render fell back. Docs: gpu-pipeline.md, control-protocol.md (app.export result), parity.md.
This commit is contained in:
1 parent
b70ae1741e
commit
47e9181f69
9 files changed
+243
-63
No files matched your search
@@ -291,6 +291,7 @@ impl Workers {
|
||||
level: b.level,
|
||||
key: b.key,
|
||||
rendered: Err(msg),
|
||||
gpu_fallback: None,
|
||||
loaded: None,
|
||||
quick: None,
|
||||
display: None,
|
||||
@@ -378,6 +379,7 @@ impl Workers {
|
||||
level: b.level,
|
||||
key: b.key,
|
||||
rendered,
|
||||
gpu_fallback: None,
|
||||
loaded: None,
|
||||
quick: None,
|
||||
display: None,
|
||||
@@ -431,6 +433,7 @@ impl RenderOffload for Workers {
|
||||
level: job.level,
|
||||
key: job.key,
|
||||
rendered: Ok(Rendered { image, histogram, deep: None }),
|
||||
gpu_fallback: None,
|
||||
loaded: None,
|
||||
quick: None,
|
||||
display: None,
|
||||
|
||||
@@ -1354,6 +1354,9 @@ pub struct Exported {
|
||||
/// Files written next to it, named like it with another extension: `(extension, bytes)` (the
|
||||
/// XMP sidecar of an `Original` export).
|
||||
pub sidecars: Vec<(&'static str, Vec<u8>)>,
|
||||
/// Set when the render was tried on the GPU and fell back to the CPU: why (see
|
||||
/// `docs/gpu-pipeline.md`). The file holds the CPU render either way.
|
||||
pub gpu_fallback: Option<String>,
|
||||
}
|
||||
|
||||
/// Output size of photo `p` under `o` (its cropped full size when `o.resize` is `None`).
|
||||
@@ -1488,7 +1491,9 @@ impl PreparedExport {
|
||||
match self.work {
|
||||
Work::Render(w) => {
|
||||
let RenderWork { job, hdr_job, meta, opts } = *w;
|
||||
let r = job.run().rendered?;
|
||||
let result = job.run();
|
||||
let r = result.rendered?;
|
||||
let gpu_fallback = result.gpu_fallback;
|
||||
let bytes = match hdr_job {
|
||||
Some(hj) => {
|
||||
let hr = hj.run().rendered?;
|
||||
@@ -1497,7 +1502,7 @@ impl PreparedExport {
|
||||
}
|
||||
None => encode_rendered(&r, &opts, meta.as_ref())?,
|
||||
};
|
||||
Ok(Exported { file_name, bytes, width: r.image.width, height: r.image.height, sidecars: Vec::new() })
|
||||
Ok(Exported { file_name, bytes, width: r.image.width, height: r.image.height, sidecars: Vec::new(), gpu_fallback })
|
||||
}
|
||||
Work::File { path, read, packet, dng, label, size } => {
|
||||
let bytes = match &read {
|
||||
@@ -1505,7 +1510,14 @@ impl PreparedExport {
|
||||
None => std::fs::read(&path).map_err(|e| format!("{path}: {e}"))?,
|
||||
};
|
||||
let Some(dng) = dng else {
|
||||
return Ok(Exported { file_name, bytes, width: size.0, height: size.1, sidecars: vec![("xmp", packet.into_bytes())] });
|
||||
return Ok(Exported {
|
||||
file_name,
|
||||
bytes,
|
||||
width: size.0,
|
||||
height: size.1,
|
||||
sidecars: vec![("xmp", packet.into_bytes())],
|
||||
gpu_fallback: None,
|
||||
});
|
||||
};
|
||||
if lightcraft_raw::probe(&bytes).is_none() {
|
||||
return Err(format!("{label}: DNG export needs a raw photo"));
|
||||
@@ -1519,7 +1531,7 @@ impl PreparedExport {
|
||||
};
|
||||
let dng = lightcraft_raw::write_dng(&raw, &lightcraft_raw::DngWriteOptions { xmp: Some(packet), compression, ..Default::default() })
|
||||
.map_err(|e| e.to_string())?;
|
||||
Ok(Exported { file_name, bytes: dng, width: raw.width, height: raw.height, sidecars: Vec::new() })
|
||||
Ok(Exported { file_name, bytes: dng, width: raw.width, height: raw.height, sidecars: Vec::new(), gpu_fallback: None })
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1868,7 +1880,11 @@ impl<'a> Placer<'a> {
|
||||
match written {
|
||||
Ok((path, files, sidecars)) => {
|
||||
self.taken.extend(files);
|
||||
self.out.push(json!({"path": path, "width": e.width, "height": e.height, "bytes": e.bytes.len(), "sidecars": sidecars}));
|
||||
let mut entry = json!({"path": path, "width": e.width, "height": e.height, "bytes": e.bytes.len(), "sidecars": sidecars});
|
||||
if let Some(why) = &e.gpu_fallback {
|
||||
entry["gpuFallback"] = json!(why);
|
||||
}
|
||||
self.out.push(entry);
|
||||
}
|
||||
Err(err) if self.stop_on_error => return Err(err),
|
||||
Err(err) => self.out.push(json!({"photo": photo.0, "file": file, "error": err})),
|
||||
@@ -2574,6 +2590,29 @@ mod tests {
|
||||
files.iter().map(|f| json!({"path": f["path"], "width": f["width"], "height": f["height"], "error": f["error"]})).collect()
|
||||
}
|
||||
|
||||
/// Issue #675: a batch reports a render that fell back from the GPU to the CPU on that file's
|
||||
/// entry (`gpuFallback`, the render's own reason) and says nothing on the others.
|
||||
#[test]
|
||||
fn a_batch_reports_a_gpu_fallback_on_the_file_it_hit() {
|
||||
if !lightcraft_gpu::available() {
|
||||
eprintln!("skipped: no GPU adapter ({:?})", lightcraft_gpu::unavailable_reason());
|
||||
return;
|
||||
}
|
||||
let mut s = crate::Session::with_demo();
|
||||
let ids: Vec<_> = s.visible().iter().copied().take(2).collect();
|
||||
let o = ExportOptions::from_json(&json!({"format": "png", "width": 64}));
|
||||
// one lane: both photos render on this thread, and the fault hits the first render here
|
||||
lightcraft_gpu::inject_fault(lightcraft_gpu::Fault::Limit(1 << 10));
|
||||
let (files, _, _) = lanes_batch(&mut s, &ids, &o, 1, true, usize::MAX);
|
||||
let files = files.unwrap();
|
||||
assert_eq!(files.len(), 2, "{files:?}");
|
||||
let why: Vec<_> = files.iter().filter_map(|f| f.get("gpuFallback").and_then(serde_json::Value::as_str)).collect();
|
||||
assert_eq!(why.len(), 1, "exactly one file fell back: {files:?}");
|
||||
assert!(why[0].contains("limit"), "{files:?}");
|
||||
assert!(files[1].get("gpuFallback").is_none(), "the second render was fine: {files:?}");
|
||||
assert!(lightcraft_gpu::available(), "a limit is no device failure");
|
||||
}
|
||||
|
||||
// Issue #496: photos rendered side by side give the same files, names, order and progress as
|
||||
// one at a time, and a cancel stops at the same photo
|
||||
#[test]
|
||||
|
||||
@@ -685,6 +685,9 @@ pub struct RenderResult {
|
||||
pub level: SourceLevel,
|
||||
pub key: u64,
|
||||
pub rendered: Result<Rendered, String>,
|
||||
/// Set when this job's render was tried on the GPU and fell back to the CPU: why
|
||||
/// ([`lightcraft_gpu::Fallback::Failed`]). The result is the CPU render either way.
|
||||
pub gpu_fallback: Option<String>,
|
||||
/// A source that was loaded by this job (to be inserted into the cache).
|
||||
pub loaded: Option<DecodedSource>,
|
||||
/// Set for a [`QuickJob`]'s stand-in: where the image came from.
|
||||
@@ -767,6 +770,7 @@ impl RenderJob {
|
||||
level: self.level,
|
||||
key: self.key,
|
||||
rendered: Ok(Rendered { image, histogram, deep: None }),
|
||||
gpu_fallback: None,
|
||||
loaded: None,
|
||||
quick: None,
|
||||
display: None,
|
||||
@@ -788,7 +792,7 @@ impl RenderJob {
|
||||
// Thumbnails (many small jobs side by side) stay on the CPU; views and exports use
|
||||
// the GPU when there is one.
|
||||
let gpu = self.cache.is_none();
|
||||
let mut rendered = develop(src, &info, &self.settings, &self.request, self.stages.as_deref(), gpu);
|
||||
let (mut rendered, gpu_fallback) = develop_reporting(src, &info, &self.settings, &self.request, self.stages.as_deref(), gpu);
|
||||
// for a display profile: the image goes to the display's values, the histogram
|
||||
// and the cached view preview stay sRGB
|
||||
let mut srgb = None;
|
||||
@@ -820,6 +824,7 @@ impl RenderJob {
|
||||
level: self.level,
|
||||
key: self.key,
|
||||
rendered: Ok(rendered),
|
||||
gpu_fallback,
|
||||
loaded: (!was_loaded).then_some(source),
|
||||
quick: None,
|
||||
display,
|
||||
@@ -832,6 +837,7 @@ impl RenderJob {
|
||||
level: self.level,
|
||||
key: self.key,
|
||||
rendered: Err(e),
|
||||
gpu_fallback: None,
|
||||
loaded: None,
|
||||
quick: None,
|
||||
display: None,
|
||||
@@ -844,19 +850,39 @@ impl RenderJob {
|
||||
/// the CPU, reusing `stages` either way. Both produce the same image within 1–3 LSB (see
|
||||
/// `docs/gpu-pipeline.md`); `LIGHTCRAFT_GPU=0` or [`lightcraft_gpu::set_enabled`] forces the CPU.
|
||||
pub fn develop(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &RenderRequest, stages: Option<&StageCache>, gpu: bool) -> Rendered {
|
||||
develop_reporting(src, info, s, req, stages, gpu).0
|
||||
}
|
||||
|
||||
/// [`develop`], also saying why the render fell back from the GPU to the CPU when it did (`None`:
|
||||
/// it rendered on the GPU, or the GPU was not tried). The reason is this render's own, not the
|
||||
/// process-wide [`lightcraft_gpu::last_fallback`].
|
||||
pub fn develop_reporting(
|
||||
src: &Arc<Rgb32f>,
|
||||
info: &SourceInfo,
|
||||
s: &DevelopSettings,
|
||||
req: &RenderRequest,
|
||||
stages: Option<&StageCache>,
|
||||
gpu: bool,
|
||||
) -> (Rendered, Option<String>) {
|
||||
// LUT profiles have no GPU stage: they render on the CPU
|
||||
if gpu
|
||||
let fallback = if gpu
|
||||
&& !lightcraft_pipeline::lut::is_lut_profile(&s.profile.id)
|
||||
&& !s.masks.iter().any(|m| m.visible && m.refine > 0.0)
|
||||
&& req.proof.is_none()
|
||||
&& let Some(r) = lightcraft_gpu::render(src, info, s, req, stages)
|
||||
{
|
||||
return r;
|
||||
}
|
||||
match stages {
|
||||
match lightcraft_gpu::try_render(src, info, s, req, stages) {
|
||||
Ok(r) => return (r, None),
|
||||
Err(lightcraft_gpu::Fallback::Failed(why)) => Some(why),
|
||||
Err(lightcraft_gpu::Fallback::NotTried) => None,
|
||||
}
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let rendered = match stages {
|
||||
Some(st) => lightcraft_pipeline::render_cached(src, info, s, req, st),
|
||||
None => lightcraft_pipeline::render(src, info, s, req),
|
||||
}
|
||||
};
|
||||
(rendered, fallback)
|
||||
}
|
||||
|
||||
/// Where a [`QuickJob`]'s stand-in image came from.
|
||||
@@ -897,6 +923,7 @@ impl QuickJob {
|
||||
level: SourceLevel::Thumb,
|
||||
key,
|
||||
rendered,
|
||||
gpu_fallback: None,
|
||||
loaded: None,
|
||||
quick,
|
||||
display: None,
|
||||
|
||||
@@ -177,7 +177,13 @@ fn prepared_exports_run_on_another_thread() {
|
||||
|
||||
/// Issue #78: a GPU render whose work never ran (a driver that drops a submission without an
|
||||
/// error) gave an all-black export. The export must come from the CPU instead — the same
|
||||
/// image — with the reason recorded.
|
||||
/// image — with the reason recorded on the export itself.
|
||||
///
|
||||
/// Issue #675: this test runs beside every other engine test, many of them rendering on the GPU.
|
||||
/// It asks only about its own export (`Exported::gpu_fallback`), never the process-wide
|
||||
/// `lightcraft_gpu::last_fallback` another thread's render overwrites, and the simulated failure
|
||||
/// switches the GPU off for this thread alone, so neither the fault nor `reset_failures` reaches
|
||||
/// the other tests.
|
||||
#[test]
|
||||
fn export_falls_back_to_the_cpu_when_gpu_work_is_lost() {
|
||||
let mut s = Session::with_demo();
|
||||
@@ -192,18 +198,26 @@ fn export_falls_back_to_the_cpu_when_gpu_work_is_lost() {
|
||||
let gpu = lightcraft_gpu::available();
|
||||
let healthy = mean(&export_photo(&mut s, id, &o, 1).unwrap().bytes);
|
||||
lightcraft_gpu::inject_fault(lightcraft_gpu::Fault::DropWork);
|
||||
let faulted = export_photo(&mut s, id, &o, 1).unwrap().bytes;
|
||||
let faulted = export_photo(&mut s, id, &o, 1).unwrap();
|
||||
if gpu {
|
||||
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
|
||||
let why = faulted.gpu_fallback.clone().unwrap_or_default();
|
||||
assert!(why.contains("incomplete"), "{why}");
|
||||
// the GPU is off for the process now: this export renders on the CPU
|
||||
// the GPU is off for this thread now: the next export is not even tried on it
|
||||
assert!(!lightcraft_gpu::available());
|
||||
assert!(lightcraft_gpu::unavailable_reason().unwrap_or_default().contains("incomplete"), "{:?}", lightcraft_gpu::unavailable_reason());
|
||||
// the thread's own `inject_fault` stopped only its own GPU use: the others still have it
|
||||
let elsewhere = std::thread::spawn(|| (lightcraft_gpu::available(), lightcraft_gpu::unavailable_reason())).join().unwrap();
|
||||
assert_eq!(elsewhere, (true, None), "a simulated failure must not switch the GPU off for the other threads");
|
||||
} else {
|
||||
assert_eq!(faulted.gpu_fallback, None);
|
||||
}
|
||||
let cpu = export_photo(&mut s, id, &o, 1).unwrap().bytes;
|
||||
let cpu = export_photo(&mut s, id, &o, 1).unwrap();
|
||||
assert_eq!(cpu.gpu_fallback, None, "not tried on the GPU: no fallback to report");
|
||||
lightcraft_gpu::reset_failures();
|
||||
assert_eq!(lightcraft_gpu::available(), gpu, "reset_failures gives this thread the GPU back");
|
||||
let px = |b: &[u8]| lightcraft_codecs::decode(b, Default::default()).unwrap().image.data;
|
||||
assert!(px(&faulted) == px(&cpu), "the fallback is the CPU render");
|
||||
let m = mean(&faulted);
|
||||
assert!(px(&faulted.bytes) == px(&cpu.bytes), "the fallback is the CPU render");
|
||||
let m = mean(&faulted.bytes);
|
||||
assert!(m > 0.02, "not black: mean {m}");
|
||||
assert!((m - healthy).abs() < 0.01, "GPU {healthy} vs CPU {m}");
|
||||
}
|
||||
|
||||
+96
-28
@@ -11,8 +11,10 @@
|
||||
//!
|
||||
//! Use [`render`]: it returns `None` when the GPU is unavailable, disabled (`LIGHTCRAFT_GPU=0`,
|
||||
//! `LIGHTCRAFT_GPU_BACKEND=off` or [`set_enabled`]), the render does not fit the device, or the device reported an error, ran out
|
||||
//! of memory or returned an incomplete image — callers then render on the CPU. [`unavailable_reason`]
|
||||
//! and [`last_fallback`] say why (`ui.inspect` → `perf.gpuReason` / `perf.gpuFallback`).
|
||||
//! of memory or returned an incomplete image — callers then render on the CPU. [`try_render`] is
|
||||
//! the same call reporting why *this* render fell back ([`Fallback`]); [`unavailable_reason`] and
|
||||
//! [`last_fallback`] (the latest fallback in the process) feed the status displays (`ui.inspect` →
|
||||
//! `perf.gpuReason` / `perf.gpuFallback`).
|
||||
//! The browser build has no GPU path yet (WebGPU device creation is asynchronous): everything here
|
||||
//! compiles to the CPU fallback on wasm32.
|
||||
//!
|
||||
@@ -88,6 +90,9 @@ pub fn unavailable_reason() -> Option<String> {
|
||||
let r = BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner()).clone().unwrap_or_else(|| "device error".into());
|
||||
return Some(format!("stopped after a GPU failure: {r}"));
|
||||
}
|
||||
if let Some(r) = broken_here() {
|
||||
return Some(format!("stopped after a GPU failure: {r}"));
|
||||
}
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
{
|
||||
match GPU.get() {
|
||||
@@ -101,12 +106,35 @@ pub fn unavailable_reason() -> Option<String> {
|
||||
}
|
||||
}
|
||||
|
||||
/// The latest render that fell back from the GPU to the CPU and why (device limit, out of memory,
|
||||
/// device error, incomplete or blank result), if any.
|
||||
/// The latest render in the process that fell back from the GPU to the CPU and why (device limit,
|
||||
/// out of memory, device error, incomplete or blank result), if any. A diagnostic for status
|
||||
/// displays (`ui.inspect` → `perf.gpuFallback`): renders on other threads overwrite it, so a caller
|
||||
/// that wants the reason for *its own* render uses [`try_render`], which returns it.
|
||||
pub fn last_fallback() -> Option<String> {
|
||||
LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()).clone()
|
||||
}
|
||||
|
||||
/// Why [`try_render`] made no GPU image: the caller renders on the CPU.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub enum Fallback {
|
||||
/// The GPU was not tried: it is unavailable, disabled or closing ([`unavailable_reason`]), or
|
||||
/// the request is one only the CPU renders (high bit depth, soft proof, HDR range overlay).
|
||||
NotTried,
|
||||
/// The render was tried on the GPU and failed, for this reason (the same text
|
||||
/// [`last_fallback`] reports). The CPU renders it instead.
|
||||
Failed(String),
|
||||
}
|
||||
|
||||
impl Fallback {
|
||||
/// The failure's reason; `None` when the GPU was not tried.
|
||||
pub fn reason(&self) -> Option<&str> {
|
||||
match self {
|
||||
Fallback::NotTried => None,
|
||||
Fallback::Failed(r) => Some(r),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A simulated GPU failure for the next [`render`] on this thread (tests of the CPU fallback).
|
||||
#[doc(hidden)]
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
@@ -122,9 +150,15 @@ pub enum Fault {
|
||||
|
||||
thread_local! {
|
||||
static FAULT: std::cell::Cell<Option<Fault>> = const { std::cell::Cell::new(None) };
|
||||
/// Why a simulated failure ([`inject_fault`]) stopped GPU use on this thread. A simulated
|
||||
/// fatal failure behaves like a real one for the thread that asked for it — [`render`] returns
|
||||
/// `None`, [`available`] is false, [`unavailable_reason`] says why — but leaves the rest of
|
||||
/// the process on the GPU: tests of the fallback run beside other tests rendering on it.
|
||||
static BROKEN_HERE: std::cell::RefCell<Option<String>> = const { std::cell::RefCell::new(None) };
|
||||
}
|
||||
|
||||
/// Make the next [`render`] on this thread fail with `f` (tests).
|
||||
/// Make the next [`render`] on this thread fail with `f` (tests). A fatal `f` then stops GPU use
|
||||
/// on this thread only, until [`reset_failures`].
|
||||
#[doc(hidden)]
|
||||
pub fn inject_fault(f: Fault) {
|
||||
FAULT.with(|c| c.set(Some(f)));
|
||||
@@ -135,12 +169,27 @@ fn take_fault() -> Option<Fault> {
|
||||
FAULT.with(|c| c.take())
|
||||
}
|
||||
|
||||
/// Use the GPU again after a failure (tests that inject faults).
|
||||
/// Stop using the GPU on this thread (a simulated fatal failure).
|
||||
#[cfg_attr(target_arch = "wasm32", allow(dead_code))]
|
||||
fn mark_broken_here(reason: &str) {
|
||||
let _ = BROKEN_HERE.try_with(|b| {
|
||||
let mut b = b.borrow_mut();
|
||||
if b.is_none() {
|
||||
*b = Some(reason.to_string());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Why a simulated failure stopped GPU use on this thread, if one did.
|
||||
fn broken_here() -> Option<String> {
|
||||
BROKEN_HERE.try_with(|b| b.borrow().clone()).ok().flatten()
|
||||
}
|
||||
|
||||
/// Use the GPU again on this thread after a simulated failure ([`inject_fault`]; tests). Real
|
||||
/// failures are not undone: the process stays on the CPU, and nothing another thread did changes.
|
||||
#[doc(hidden)]
|
||||
pub fn reset_failures() {
|
||||
BROKEN.store(false, Ordering::Relaxed);
|
||||
*BROKEN_REASON.lock().unwrap_or_else(|e| e.into_inner()) = None;
|
||||
*LAST_FALLBACK.lock().unwrap_or_else(|e| e.into_inner()) = None;
|
||||
let _ = BROKEN_HERE.try_with(|b| *b.borrow_mut() = None);
|
||||
}
|
||||
|
||||
/// Allow or forbid GPU rendering at runtime (a preference). `LIGHTCRAFT_GPU=0` forbids it for the
|
||||
@@ -149,9 +198,10 @@ pub fn set_enabled(on: bool) {
|
||||
ENABLED.store(on, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// Whether GPU rendering is allowed (environment, preference, no earlier failure).
|
||||
/// Whether GPU rendering is allowed (environment, preference, no earlier failure — real, or
|
||||
/// simulated on this thread).
|
||||
pub fn enabled() -> bool {
|
||||
ENABLED.load(Ordering::Relaxed) && !BROKEN.load(Ordering::Relaxed) && !env_disabled()
|
||||
ENABLED.load(Ordering::Relaxed) && !BROKEN.load(Ordering::Relaxed) && !env_disabled() && broken_here().is_none()
|
||||
}
|
||||
|
||||
/// The process is about to end: no GPU work starts from now on, on any thread. [`render`] returns
|
||||
@@ -378,8 +428,22 @@ pub fn stage_bytes(stages: &StageCache) -> usize {
|
||||
}
|
||||
|
||||
/// Render `src` with `s` on the GPU, reusing the device-resident stages kept with `stages` (the
|
||||
/// view's CPU stage cache). `None`: render on the CPU instead.
|
||||
/// view's CPU stage cache). `None`: render on the CPU instead. [`try_render`] also says why.
|
||||
pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &RenderRequest, stages: Option<&StageCache>) -> Option<Rendered> {
|
||||
try_render(src, info, s, req, stages).ok()
|
||||
}
|
||||
|
||||
/// [`render`], reporting why this render made no GPU image: [`Fallback::NotTried`] when the GPU
|
||||
/// was not used (callers can ask [`unavailable_reason`]), [`Fallback::Failed`] with the reason
|
||||
/// when the GPU render failed and the CPU must render it. The reason belongs to this call, unlike
|
||||
/// [`last_fallback`], which the latest render on any thread overwrites.
|
||||
pub fn try_render(
|
||||
src: &Arc<Rgb32f>,
|
||||
info: &SourceInfo,
|
||||
s: &DevelopSettings,
|
||||
req: &RenderRequest,
|
||||
stages: Option<&StageCache>,
|
||||
) -> Result<Rendered, Fallback> {
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
{
|
||||
// sections switched off with their eye render as if at their defaults (issue #316)
|
||||
@@ -392,14 +456,16 @@ pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &R
|
||||
|| req.proof.is_some()
|
||||
|| req.overlay == lightcraft_pipeline::Overlay::HdrRange
|
||||
{
|
||||
return None;
|
||||
return Err(Fallback::NotTried);
|
||||
}
|
||||
let s = &*lightcraft_pipeline::settings_for(s, req);
|
||||
// in flight until this returns; `None` once the process is ending (issue #620)
|
||||
let _work = exit::enter()?;
|
||||
let gpu = device()?;
|
||||
// in flight until this returns; not tried once the process is ending (issue #620)
|
||||
let _work = exit::enter().ok_or(Fallback::NotTried)?;
|
||||
let gpu = device().ok_or(Fallback::NotTried)?;
|
||||
let ext = stages.map(|c| c.extension::<GpuStages>());
|
||||
let fault = take_fault();
|
||||
// A simulated failure stops GPU use on this thread only (see `BROKEN_HERE`).
|
||||
let stop = |reason: &str| if fault.is_some() { mark_broken_here(reason) } else { mark_broken(reason) };
|
||||
let _ = ctx::take_failure(); // nothing left over from an earlier render on this thread
|
||||
let (r, scoped) = {
|
||||
let _scope = ctx::RenderScope::new(gpu);
|
||||
@@ -413,12 +479,11 @@ pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &R
|
||||
(Some(f), _) if f.kind == ctx::FailKind::Limit => Some(f),
|
||||
(f, scoped) => scoped.or(f),
|
||||
};
|
||||
match (r, failure) {
|
||||
let reason = match (r, failure) {
|
||||
(Err(_), _) => {
|
||||
log::error!("gpu: render failed; using the CPU pipeline from now on");
|
||||
mark_broken("a GPU render panicked");
|
||||
record_fallback("a GPU render panicked; the GPU is not used again".into());
|
||||
None
|
||||
stop("a GPU render panicked");
|
||||
"a GPU render panicked; the GPU is not used again".to_string()
|
||||
}
|
||||
(Ok(r), Some(f)) => {
|
||||
// the view's cached device stages may hold results of the failed passes
|
||||
@@ -431,7 +496,7 @@ pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &R
|
||||
};
|
||||
let reason = match f.kind {
|
||||
ctx::FailKind::Fatal => {
|
||||
mark_broken(&f.reason);
|
||||
stop(&f.reason);
|
||||
format!("{what}: {}; the GPU is not used again", f.reason)
|
||||
}
|
||||
ctx::FailKind::OutOfMemory => {
|
||||
@@ -444,20 +509,23 @@ pub fn render(src: &Arc<Rgb32f>, info: &SourceInfo, s: &DevelopSettings, req: &R
|
||||
if render::profiling() {
|
||||
eprintln!(" gpu fallback to the CPU: {reason}");
|
||||
}
|
||||
record_fallback(reason);
|
||||
None
|
||||
reason
|
||||
}
|
||||
// a device error on another thread or a lost device during the render
|
||||
(Ok(Some(_)), None) if BROKEN.load(Ordering::Relaxed) => {
|
||||
record_fallback(format!("device failure during the render: {}", unavailable_reason().unwrap_or_default()));
|
||||
None
|
||||
format!("device failure during the render: {}", unavailable_reason().unwrap_or_default())
|
||||
}
|
||||
(Ok(r), None) => r,
|
||||
}
|
||||
(Ok(Some(r)), None) => return Ok(r),
|
||||
// every path of `render::render` that gives up records a failure first; should one
|
||||
// not, the CPU renders it rather than nothing
|
||||
(Ok(None), None) => "GPU render: no image came back".to_string(),
|
||||
};
|
||||
record_fallback(reason.clone());
|
||||
Err(Fallback::Failed(reason))
|
||||
}
|
||||
#[cfg(target_arch = "wasm32")]
|
||||
{
|
||||
let _ = (src, info, s, req, stages);
|
||||
None
|
||||
Err(Fallback::NotTried)
|
||||
}
|
||||
}
|
||||
@@ -1,14 +1,15 @@
|
||||
//! Issue #78: a GPU render that cannot be trusted — over the device's buffer limits, a device
|
||||
//! error, work that silently did not run — must come back as `None` (the caller renders on the
|
||||
//! CPU) with the reason recorded, never as an image of zeros. One test, sequential: the fatal
|
||||
//! faults stop GPU use for the whole process until `reset_failures`.
|
||||
//! CPU) with the reason recorded, never as an image of zeros. One test, sequential: the simulated
|
||||
//! fatal faults stop GPU use on this thread until `reset_failures` (issue #675: on this thread
|
||||
//! only, so a test of the fallback can run beside tests that render on the GPU).
|
||||
//! Skips (passes with a note) when no GPU adapter exists.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use lightcraft_develop::{DevelopSettings, Mask, MaskComponent, MaskOp, MaskShape};
|
||||
use lightcraft_geom::Point;
|
||||
use lightcraft_gpu::Fault;
|
||||
use lightcraft_gpu::{Fallback, Fault};
|
||||
use lightcraft_pipeline::{RenderRequest, SourceInfo, render};
|
||||
use lightcraft_raster::Rgba8;
|
||||
|
||||
@@ -57,39 +58,51 @@ fn failed_gpu_renders_fall_back_with_a_reason() {
|
||||
assert!(d < 0.5, "{what}: GPU differs from the CPU by {d:.3} LSB");
|
||||
};
|
||||
ok("healthy");
|
||||
// the render that fell back says why itself; the process-wide diagnostic agrees
|
||||
let failed = |what: &str| -> String {
|
||||
match lightcraft_gpu::try_render(&src, &info, &s, &req, None) {
|
||||
Ok(_) => panic!("{what}: the GPU render must fall back"),
|
||||
Err(Fallback::NotTried) => panic!("{what}: the render must be tried on the GPU ({:?})", lightcraft_gpu::unavailable_reason()),
|
||||
Err(Fallback::Failed(why)) => {
|
||||
assert_eq!(lightcraft_gpu::last_fallback().as_deref(), Some(why.as_str()), "{what}");
|
||||
why
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// the output itself is over the limit: refused up front, the GPU stays in use
|
||||
lightcraft_gpu::inject_fault(Fault::Limit(1 << 20));
|
||||
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
|
||||
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
|
||||
let why = failed("output over the limit");
|
||||
assert!(why.contains("limit"), "{why}");
|
||||
assert!(lightcraft_gpu::available(), "a limit is no device failure");
|
||||
|
||||
// the output fits but the mask planes (4 masks: 4 planes in one buffer) don't: caught when the
|
||||
// buffer is created, without a device error, and the GPU stays in use
|
||||
lightcraft_gpu::inject_fault(Fault::Limit((w * h * 3 * 4 + 1024) as u64));
|
||||
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
|
||||
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
|
||||
let why = failed("mask planes over the limit");
|
||||
assert!(why.contains("exceeds"), "{why}");
|
||||
assert!(lightcraft_gpu::available(), "{:?}", lightcraft_gpu::unavailable_reason());
|
||||
ok("after the limit faults");
|
||||
|
||||
// a device (validation) error: captured by the render's error scope; the GPU is not used again
|
||||
// on this thread — other threads keep it (a simulated failure, not a real one)
|
||||
lightcraft_gpu::inject_fault(Fault::Validation);
|
||||
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none());
|
||||
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
|
||||
let why = failed("validation error");
|
||||
assert!(why.contains("device error"), "{why}");
|
||||
assert!(!lightcraft_gpu::available());
|
||||
assert!(lightcraft_gpu::unavailable_reason().is_some_and(|r| r.contains("device error")), "{:?}", lightcraft_gpu::unavailable_reason());
|
||||
assert_eq!(lightcraft_gpu::render(&src, &info, &s, &req, None).map(|_| ()), None, "no GPU render on this thread after a fatal fault");
|
||||
let elsewhere = std::thread::spawn(|| (lightcraft_gpu::available(), lightcraft_gpu::unavailable_reason())).join();
|
||||
assert_eq!(elsewhere.ok(), Some((true, None)), "a simulated failure switches the GPU off for its own thread only");
|
||||
lightcraft_gpu::reset_failures();
|
||||
assert_eq!(lightcraft_gpu::unavailable_reason(), None);
|
||||
ok("after a device error");
|
||||
|
||||
// the per-pixel work never runs (a driver dropping a submission without an error): before
|
||||
// the fix this came back as a fully black image
|
||||
lightcraft_gpu::inject_fault(Fault::DropWork);
|
||||
assert!(lightcraft_gpu::render(&src, &info, &s, &req, None).is_none(), "an unwritten image must not be returned");
|
||||
let why = lightcraft_gpu::last_fallback().unwrap_or_default();
|
||||
assert!(why.contains("incomplete"), "{why}");
|
||||
let why = failed("dropped work");
|
||||
assert!(why.contains("incomplete"), "an unwritten image must not be returned: {why}");
|
||||
assert!(!lightcraft_gpu::available());
|
||||
lightcraft_gpu::reset_failures();
|
||||
ok("after dropped work");
|
||||
@@ -97,7 +110,14 @@ fn failed_gpu_renders_fall_back_with_a_reason() {
|
||||
// a black photo is still rendered (on the CPU, to be sure)
|
||||
let black = Arc::new(lightcraft_raster::Rgb32f::from_fn(256, 256, |_, _| [0.0; 3]));
|
||||
let s0 = DevelopSettings::default();
|
||||
assert!(lightcraft_gpu::render(&black, &info, &s0, &RenderRequest::fit(256, 256), None).is_none());
|
||||
match lightcraft_gpu::try_render(&black, &info, &s0, &RenderRequest::fit(256, 256), None) {
|
||||
Err(Fallback::Failed(why)) => assert!(why.contains("black"), "{why}"),
|
||||
other => panic!("a black result is redone on the CPU: {:?}", other.map(|_| ())),
|
||||
}
|
||||
assert!(lightcraft_gpu::last_fallback().unwrap_or_default().contains("black"));
|
||||
assert!(lightcraft_gpu::available());
|
||||
|
||||
// a request the GPU never renders is not a fallback
|
||||
let deep = RenderRequest { depth: lightcraft_pipeline::OutputDepth::U16, ..RenderRequest::fit(256, 256) };
|
||||
assert_eq!(lightcraft_gpu::try_render(&src, &info, &s, &deep, None).err(), Some(Fallback::NotTried));
|
||||
}
|
||||
@@ -45,7 +45,7 @@ mode ([mcp.md](mcp.md)) is a thin layer over this channel. Implementation:
|
||||
| `ui.resize` | `{width, height}` | Resize the window |
|
||||
| `ui.zoomFactor` | `{factor}` | Set the interface scale (0.5–3; egui's zoom keys are off, Cmd+= / Cmd+- zoom the photo) |
|
||||
| `ui.screenshot` | `{path?, headless?}` | `{path, width, height}` once the frame (with finished renders) is captured. `headless: true` draws the UI on the CPU (no compositor needed); a windowed capture that gets no frame within 2 s falls back to headless automatically |
|
||||
| `engine.execute {command: "app.export", params}` | export params (see `docs/mcp.md`), plus `preset`, `dir` / `path`, `ids`, `background` | Writes the files and returns `{files}`; with `background: true` (what the Export dialog and menus use) it returns `{background: true, total}` at once and the batch runs on a worker thread — poll `ui.inspect` → `export` |
|
||||
| `engine.execute {command: "app.export", params}` | export params (see `docs/mcp.md`), plus `preset`, `dir` / `path`, `ids`, `background` | Writes the files and returns `{files}` (one `{path, width, height, bytes, sidecars}` per file, plus `gpuFallback` — why that render fell back from the GPU to the CPU — when it did; see `docs/gpu-pipeline.md`); with `background: true` (what the Export dialog and menus use) it returns `{background: true, total}` at once and the batch runs on a worker thread — poll `ui.inspect` → `export` |
|
||||
| `engine.execute {command: "activity.list"}` | — | The long-running tasks in flight (imports, exports, preview builds, downloads, the face scan…), oldest first: `{tasks: [{id, kind, label, done, total, unit, detail, cancellable, cancelling, ageMs}]}`; `total` 0 = not known yet; `unit` is `count`, `bytes` or `percent`; `label` is English (the app shows it translated) |
|
||||
| `engine.execute {command: "activity.cancel"}` | `{id}` or `{all: true}` | Asks the task (or every cancellable one) to stop at its next file or photo → `{cancelled: n}`; an unknown id (it may have just finished) or a task that can't be cancelled is an error |
|
||||
| `ui.render` | `{id?, size?, path?}` | Render a photo (PNG to `path`), `{width, height}` |
|
||||
|
||||
+10
-1
@@ -190,10 +190,19 @@ redone on the CPU and why, e.g. `"6000×4000: the GPU returned an incomplete ima
|
||||
24000000 pixels unwritten); the GPU is not used again"`). The same in `app.gpu` (`reason`,
|
||||
`lastFallback`), Help ▸ System Info and Settings ▸ Performance. `LIGHTCRAFT_PROFILE=1` prints the
|
||||
fallback with the stage timings; `RUST_LOG=warn` logs it.
|
||||
`perf.gpuFallback` is one value for the whole process, overwritten by the latest render on any
|
||||
thread. A render's *own* reason comes from `lightcraft_gpu::try_render` (`Err(Fallback::Failed(why))`;
|
||||
`Fallback::NotTried` when the GPU was not used at all), which `lightcraft_engine::media::develop_reporting`
|
||||
passes on as `RenderResult::gpu_fallback`; an export carries it as `Exported::gpu_fallback`, and
|
||||
each file of an `app.export` batch result has `gpuFallback` when its render fell back (issue #675).
|
||||
**Reproducing a smaller GPU:** `LIGHTCRAFT_GPU_LIMITS=webgpu` (or `downlevel`) creates the device
|
||||
with 128 MiB storage bindings / 256 MiB buffers, `LIGHTCRAFT_GPU_LIMITS=<n>` with n MiB / 2n MiB.
|
||||
Tests inject failures with `lightcraft_gpu::inject_fault` (`crates/gpu/tests/fallback.rs`,
|
||||
`export_falls_back_to_the_cpu_when_gpu_work_is_lost`).
|
||||
`export_falls_back_to_the_cpu_when_gpu_work_is_lost`). The fault is thread-local, and so are its
|
||||
consequences: a simulated fatal failure stops GPU use on the injecting thread only (`available()`
|
||||
false, `unavailable_reason()` set there), until `reset_failures()`, which undoes nothing else — a
|
||||
real failure still stops the whole process. Such a test asserts on its own render's reason, not on
|
||||
`last_fallback()`, so it can run beside other tests rendering on the GPU (issue #675).
|
||||
|
||||
## Correctness: CPU oracle and equivalence tests
|
||||
`crates/gpu/tests/equivalence.rs` renders the same settings on both and compares 8-bit sRGB:
|
||||
|
||||
+1
-1
@@ -521,7 +521,7 @@ under the UI command that wraps it (a no-op error); the UI command now wins.
|
||||
| LR-BEHAV-BG-TASKS | Background tasks | P0 | ✅ | `cmd:activity.list`, `cmd:activity.cancel`, `crates/engine/src/activity.rs`, `crates/ui-egui/src/panels/activity.rs`, `docs/background-tasks.md`, `crates/preview/src/lib.rs` (`JobPool`) | renders off the UI thread; exports run several photos of a batch side by side (`export_parallelism`, memory-gated; files, names and order as one at a time, #496); one activity stack in the top-left corner under the top bar, as in Lightroom Classic (issue #345): imports, folder scans, exports (and the contact sheet PDF), Synchronize Folder (its scan and its run), Build / Smart Previews, Lightroom catalog inspect / import, Photo Merge, model downloads (SAM 3, face, AI Denoise), the face scan, AI Denoise and Find Missing Photos each show a row (name, progress bar or a sweep while the amount isn't known, count and current item) once 0.5 s old, three rows then "+N more"; ✕ cancels the jobs that can stop ("Stopping…" until they do); quitting asks while a cancellable task runs; agents and `ui.inspect` (`activity`) see the same list. No pause (Lightroom can pause some tasks) |
|
||||
| LR-BEHAV-OFFLINE | Offline editing | P1 | ✅ | | local-first: everything works offline |
|
||||
| LR-BEHAV-WEB-SAFETY | Library safety in the browser build | P1 | 🟡 | `cmd:file.backupLibrary`, `cmd:file.restoreLibrary`, `apps/lightcraft-web/src/safety.rs`, `apps/lightcraft-web/src/backup.rs`, `apps/lightcraft-web/src/files.rs` | web only (experimental): Back Up Library (zip of the catalog and every stored original) and a non-destructive Restore; failed saves show as unsaved and are retried; a photo that can't be stored isn't added; one tab per library (Web Locks); notices for storage that isn't persistent and for a library that can't open; `?reset` asks first; a panic shows a message. Missing: `?safe` start, deleting originals of removed photos, zip64 (backups > 4 GB) |
|
||||
| LR-BEHAV-GPU | GPU acceleration | P0 | ✅ | `crates/gpu/src/render.rs`, `cmd:app.gpu`, `docs/gpu-pipeline.md` | CPU fallback per render on device limits, out of memory, device errors / loss, incomplete or black results (`perf.gpuReason`, `perf.gpuFallback`); short submissions for slow iGPUs; backend choice (DX12 only on Windows, `LIGHTCRAFT_GPU_BACKEND` / `WGPU_BACKEND`), device created after the window opens, crash sentinel (issue #136); quitting waits (up to 2 s) for renders in flight and starts no new GPU work, so none is inside the driver when the process exits (`crates/gpu/src/exit.rs`, issue #620; seen crashing only on Linux + NVIDIA, other drivers unverified); no tiling beyond the buffer limit (CPU) |
|
||||
| LR-BEHAV-GPU | GPU acceleration | P0 | ✅ | `crates/gpu/src/render.rs`, `cmd:app.gpu`, `docs/gpu-pipeline.md` | CPU fallback per render on device limits, out of memory, device errors / loss, incomplete or black results (`perf.gpuReason`, `perf.gpuFallback`; each render reports its own reason — `lightcraft_gpu::try_render`, `gpuFallback` per exported file — issue #675); short submissions for slow iGPUs; backend choice (DX12 only on Windows, `LIGHTCRAFT_GPU_BACKEND` / `WGPU_BACKEND`), device created after the window opens, crash sentinel (issue #136); quitting waits (up to 2 s) for renders in flight and starts no new GPU work, so none is inside the driver when the process exits (`crates/gpu/src/exit.rs`, issue #620; seen crashing only on Linux + NVIDIA, other drivers unverified); no tiling beyond the buffer limit (CPU) |
|
||||
| LR-BEHAV-RENDER-FIDELITY | Rendering matches Lightroom | P1 | 🟡 | `crates/pipeline/src/lib.rs`, `crates/pipeline/src/tone.rs` | every slider exists and works, but the character of the result (tone curve shape, highlight recovery, texture / clarity / dehaze, noise reduction, sharpening, default look) is tuned by eye; there is no systematic side-by-side comparison against Lightroom on the same CC0 raws. Needs a fidelity suite: Lightroom reference renders kept only in the local `plan/` (never committed), compared per slider and preset with a perceptual metric. Tuning that changes how existing settings render ships as a new process version (LRC-DEV-PROCESS, `docs/process-versions.md`) |
|
||||
| LR-BEHAV-DRAGDROP | Drag and drop | P1 | ✅ | `crates/ui-egui/src/lib.rs`, `crates/ui-egui/src/panels/grid.rs` (`drag_feedback`), `crates/ui-egui/src/panels/left.rs` (`drop_target`, `album_drag_over`, `top_level_drop_target`, `drag_auto_scroll`) | files → app (import); grid photos → an album row (adds the selection, with a count badge while dragging); sidebar album / folder rows → a folder (moves it; its name follows the pointer); a drag of albums or grid photos near the top or bottom edge of the sidebar scrolls it (faster nearer the edge, only while the pointer is over the sidebar) |
|
||||
| LR-BEHAV-TOAST | Toast notifications | P1 | ✅ | `crates/ui-egui/src/panels/mod.rs` | |
|
||||
|
||||
Reference in new issue
Block a user