LR-PROF-CAMERACOLOR: docs: clipped sensor pixels left out of the camera look fit, archive numbers

docs/camera-preview-colour.md

@@ -12,6 +12,8 @@ The JPEG's colourfulness also depends on brightness, as a camera's per-channel t
 
 Matrix-only and matrix + table looks are each completed (tone curve, then chroma curve) and compared on the held-out pixels in gamma-encoded display values: in linear values a slightly missed bright rock outweighed a clearly wrong dark shirt. The acceptance gates below still use linear values. Luminance is fitted separately, by quantile matching: knot *i* pairs the median of the *i*-th 1/32 of the sorted scene luminances with the median of the *i*-th 1/32 of the sorted JPEG luminances. A tone curve is monotone, so it maps each scene quantile to the same JPEG quantile; this needs no pixel-exact registration and is increasing by construction. The earlier fit (the JPEG's median per scene-luminance bin, then pooled-adjacent-violators isotonic regression) broke on lens-corrected, sharpened JPEGs of busy scenes: neighbouring bins disagreed by more than the gap between them, pooling turned them into flat steps (13 of 31 segments on a CC0 Z 6II forest shot) that rendered as posterised patches with hard edges, and regression toward the mean flattened both ends of the curve (issue #475, also #211, #454). On the 1,781 files of the CC0 archive that render, 384 of the 1,023 accepted fits had at least one flat segment (1,073 in total, up to 14 of 31); with quantile matching 1,017 of the 1,023 have none. The quantile fit is slightly worse by least squares than the conditional medians (median held-out RMS 0.0331 against 0.0330, worst +0.013), and on 7 files (compacts, bridge cameras, early Micro Four Thirds) the look cleared the acceptance gates only with the old curve. The search therefore runs once with the quantile fit (all pixels, away from edges, with and without the camera profile's colour) and, only when no look passes, once more with the earlier conditional-median fit, kept as `fit_tone_conditional`; the gates are the same, so no file loses its look. Those 7 files render exactly as before, and 6 of them still have flat segments (35 in total). The curve preserves black, cannot reverse, and extends beyond observed highlights with a bounded shoulder.
 
+**Clipped sensor pixels are left out of the fit.** A raw pixel with a channel at the clip level (0.99 of white, as in highlight reconstruction) no longer records the scene's colour: a white overcast sky with the green channel clipped and red and blue still rising comes out magenta or blue through the camera matrix, while the camera JPEG shows it white. As training pairs such pixels pulled the colour matrix toward grey and made the chroma curve remove colour at their display luminance. On the CC0 Z 6II forest sample (raw.pixls.us 4160) 24% of the proxy pixels are clipped; every chroma sample above display luminance 0.36 was one of them, and the curve cut colourfulness there to 13-19% (`[1.10, 0.94, 0.69, 0.19, 0.13, 0.19, 0.89, 1.0]`); the colours of the forest were muted by the matrix as well (spread of a\*/b\* on the proxy 63% of the JPEG's before the curve, 48% after). Over the 940 accepted looks with a chroma curve, 37% of the curve's downward pull came from clipped pixels whose JPEG pixel is near neutral, 53% in the 245 curves with a node below 0.4. The proxies therefore carry a clip mask (any sensor sample a proxy pixel covers, aligned with the CR3 framing), and the look is refitted without those pixels. The search on all pixels still decides whether a photo gets a look and which attempt (tone fit, profile or own colour, all pixels or away from edges) gives it; only that attempt is refitted, and when the refit misses the gates the look on all pixels is kept, so no photo loses its look or switches attempt. A photo without a look gets one when the search without the clipped pixels finds it. Camera-profile pairs (`calibrate`) leave clipped pixels out too. On the 1,781 files of the CC0 archive that render: 1,023 looks before, 1,027 after (none lost, 4 gained); 618 renders change (all with a look), 8 keep their earlier look through the fallback; median CIEDE2000 to the camera JPEG over the changed files 3.24 -> 3.07 (worse by more than 0.5 on 3, better by more than 0.5 on 59); median colourfulness (mean C\* of the render over the camera JPEG's) 0.973 -> 0.982 over the files with a look. The rest of the chroma curve's pull is mostly misregistration (a near-neutral JPEG pixel with a matching colour within one proxy pixel: 21%); a registration-free estimate of the curve (band means of colourfulness by each image's own luminance) brought the median colourfulness ratio to 0.999 but did not lower CIEDE2000, and is not used.
+
 The matrix is applied to scene-linear sensor pixels. The curve is carried in `SourceInfo` and applied in the pipeline finish stage, after exposure/WB/local edits. It is never baked into decoded luminance or clipped to JPEG precision. CPU and GPU use the same tone LUT. Contrast, whites and blacks adjust the starting look through the display-domain controls. JPEG neutral rendering and the existing DNG colour model are unchanged.
 
 One third of the pixels are held out from both fits. The combined model must reduce linear-display squared error by at least 30% and have per-channel RMS error ≤0.10 (0.055 rejected half of the public raw.pixls.us samples tried — ILCE-6400, -6700, -7M4 — whose fits were still 1.5–3× closer to the camera JPEG than the fallback and visibly better). When the fit on all pixels fails these gates, it is tried once more, with the same gates, on the pixels away from edges (where the 3×3 neighbourhood in either proxy spans a luminance ratio above 3). Many camera JPEGs are lens-corrected while the raw is not (Sony "Distortion Comp.: Auto", compacts and kit zooms), and on a detailed scene the misaligned edges alone fail the gate although the colours agree everywhere else (issue #232). The public ILCE-7RM2 samples (raw.pixls.us, one façade shot in seven compression/correction variants, 24–70 mm at 51 mm) are up to 15 px of 1440 off their JPEG; the six with distortion correction on missed the gate at held-out RMS 0.108–0.115 and opened in the grey neutral fallback, 13 L* too dark (ΔE76 16.6–16.9 against the JPEG after registering its geometry). Away from edges they pass at 0.049–0.052 and land at ΔE 3.7–3.8 with lightness within 0.6 L*, like the variant shot with distortion correction off (0.059 on all pixels, ΔE 3.9). The corpus GX80 kit-zoom sample (up to 63 px of 1440 off) passes too: 0.107 → 0.081, ΔE 19.1 → 7.9. Fits accepted on all pixels are unchanged. These are provisional acceptance gates, not a colour-accuracy certification. Monochrome, singular, unrelated, aspect-mismatched and poor-fitting references retain the documented fallback. Both fits use the same fixed proxy at every decode resolution.