No colour key at all: segmentation alone drives the trimap (mode forced
to 'seg', whose chroma terms degrade to no-ops on a zero background-
confidence map), ViTMatte still refines the unknown band at full
resolution, and pymatting still estimates edge foreground colour. Every
colour-keyed stage is bypassed: background auto-detect, the directional
hue split, chroma alpha suppression, despill, the cross-check veto (its
suspect zone is background-hued by definition; with reuse_as_seg the
second opinion still serves as the segmenter), and the unmix fallback.
Guards: requires segmentation.enabled and matting_method 'vitmatte'
(clear errors otherwise); despill/foreground handle model=None.
Validation: flat-background default path is bit-identical pre/post
(alpha and RGB |D|max = 0 on TestImage3). On a known-alpha subject
composited over a gradient+blotch background, recovered alpha scores
MAE 0.0018 / IoU@0.5 0.993 (0.995 with --cross-check-as-seg); the same
input correctly fails auto-detection in default mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three optimizations from profiling the cross-check-dominated pipeline
(9700X CPU, all pilot-validated on TestImage3/FixImage1):
- Reuse the cross-check HR-matting@2048 forward as the segmentation mask
(cross_check.reuse_as_seg, default ON; --no-cross-check-as-seg to opt
out). Skips the BiRefNet@1024 load+forward entirely: ~66s -> ~45s,
one less 0.9GB model. Trimap 99.8% identical, no structural change.
- --precision bf16 now fans out to all three models: ViTMatte keeps its
weight cast; both BiRefNets run their forward under autocast with a
dispatcher-level AutocastCPU fp32 shim for torchvision::deform_conv2d
(no bf16 CPU kernel, no autocast wrapper upstream). Shared hardware
gate in bgfilter/precision.py falls back to fp32 off native-bf16
hardware. TestImage3: 51.9s -> 33.1s; alpha diff max 0.15, none >0.25.
- MIMALLOC_PURGE_DELAY=0 (bgfilter/__init__.py, before torch loads):
Windows torch's bundled mimalloc lazily retains ~10GB of freed
BiRefNet activations, stacking under ViTMatte's attention peak.
2048x2048 bf16: peak 25.2 -> 21.4GB and slightly faster (73 -> 63s).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bf16 halves the matting model's activation memory -- its full-resolution
attention is the pipeline's memory peak -- with visually identical alpha
(measured: 0 px alpha deviation > 0.25 on samples; cross-check veto
behaviour unchanged, region IoU 0.984).
Guarded by a hardware check so it never lands on a slow emulation path:
CUDA requires torch.cuda.is_bf16_supported(); CPU requires the same
oneDNN native-bf16 gate PyTorch uses for matmul routing (AVX512-BF16/
AMX). Without support it warns and falls back to fp32 -- on a Zen2 EPYC
the fallback kernels measured 17-370x slower than fp32, so silent bf16
there would be a performance landmine.
Segmentation stays fp32: torchvision deform_conv2d (used by BiRefNet)
has no bf16 CPU kernel, and the segmenter is not the memory peak.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colour-drifted background trapped between hair strands defeats every
single-signal defence: the chroma key reads it as foreground (bgc ~0.07),
the segmenter backs it, ViTMatte rates it opaque, and post-hoc removal is
a proven dead end (it shreds the hair volume the same pixels belong to).
A second, trimap-free matting model (BiRefNet_HR-matting) is the only
tested model that separates this residue from the subject, so its opinion
is fused in as a veto: min-fusion that may only LOWER alpha, restricted to
the background-hued bright suspect zone (proj >= 3, L >= 45, feathered)
and gated by primary-alpha confidence (0.70 -> 0.95 ramp) so soft wisps
and dark hair are exempt by construction.
- settings/config/CLI: cross_check block, --cross-check/--no-cross-check
- alpha_post.cross_check_alpha after clean_alpha; second opinion reuses
BiRefNetSegmenter; saved to debug as cross_check_alpha.png
- chroma.bg_hue_projection extracted and shared with the trimap
- docs: methodology.md (new), hair_gap_artifacts.md (investigation log)
Verified: cross-check ON reproduces the visually-reviewed B1gate
prototype byte-for-byte on TestImage3; --no-cross-check reproduces the
previous baseline byte-for-byte; pink-bg FixImage1 face untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Segmentation backend now defaults to birefnet (ZhengPeng7/BiRefNet); anime-seg is
selected at runtime with --seg-backend anime-seg, which also swaps in the matching
weights (unknown backends raise). Config files stay for detailed tuning, not
backend selection.
- Default device is now cpu everywhere (dataclass defaults, default.yaml,
birefnet.yaml); pass --device cuda for GPU. Avoids a hard CUDA-required failure on
machines without a CUDA-enabled PyTorch build.
- Docs: README + workflow status note updated (backend via --seg-backend, CPU
default, examples no longer force --device cuda).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the two experimental bools (directional, bg_hued_to_bg) with a single
TrimapSettings.mode selector for the segmentation pipeline:
- "directional" (new default): chroma magnitude + seg + a Lab hue-direction
split. In the chroma-unknown zone a confidently-segmented pixel stays
foreground unless it is displaced toward the background hue, so a neutral
background-coloured garment (e.g. a white shirt) is kept while a background-
hued residual (blue between hair strands) is left unknown for ViTMatte /
chroma-suppress to clear.
- "seg": original fuse_trimap, unchanged.
- "directional-hard-bg": aggressive variant that hard-removes background-hued
pixels (can eat cool/shadowed white cloth).
Selectable via configs/default.yaml (trimap.mode) or CLI --trimap-mode; unknown
modes raise. Default CLI output verified byte-identical to the reviewed UNK
result on the pastel sample.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strategic pivot: the dropout-fill rescue, the anime-seg fill backend, the
BiRefNet+anime-seg intersection, and the recolor pass all existed to repair
source images whose hair was already green-contaminated or broken at generation
time. Fixing the source instead (clean pastel-background generation) makes the
matte high-contrast, so that whole compensation layer is unnecessary. Remove it.
- Two pipelines via segmentation.enabled:
false -> chroma-only (Chroma + ViTMatte)
true -> single segmenter (default anime-seg, switchable birefnet) -> trimap
then ViTMatte refines, pymatting estimates foreground, despill cleans spill.
- Generalise the green-hardcoded colour logic to a screen_color prior (hex, e.g.
"#CFEFFF"; default null = green auto-detect, byte-identical). chroma keys off
Lab/RGB distance to the colour; despill removes chroma along its Lab direction.
Added --screen-color CLI flag.
- Removed dead code: fill_seg_dropouts + seg_fill*/intersection params,
SegmentationSettings.fill_backend/fill_model_name/sharpen, unsharp_mask,
recolor.py + RecolorSettings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add bgfilter/recolor.py: after despill, propagate clean neighbour chroma into
pixels whose colour is still off (residual green, or the magenta a green unmix
overshoots into), keeping luminance partway so hair texture survives. It installs
a real observed colour rather than subtracting, so it cannot overshoot. Despill
already neutralises the matte edge (off ~ 0 there), so recolor only touches the
contaminated interior and the wisps the dropout-fill rescued.
Runs as a final colour pass on the despilled foreground; pymatting + despill keep
their stronger edge handling. On the samples the magenta fringe drops sharply
(TestImage vis-magenta 0.065->0.014, TestImage2 0.085->0.031) with edge green
unchanged (vis-green ~0.02). Debug output color_mask.png replaces the despill /
foreground-background / foreground-correction maps; smoke updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire BiRefNet into the pipeline (lazy-loaded) and build the trimap from the
semantic subject mask refined by the green key, used when segmentation is
enabled. generate_trimap (chroma-only) stays as the no-segmentation fallback.
Authority split: the mask decides subject topology (keeps green-tinted hair as
foreground, drops see-through holes), the bright-green key sharpens the screen
boundary, ViTMatte refines a small protective band. Background follows the mask
directly so interior holes survive; the band is small so it does not swallow
finger gaps.
End-to-end on the samples: green hair preserved, finger-gap green 99%
transparent (BiRefNet marks it background; was 100% foreground before), leg
edges smooth -- topology/colour/edge problems local cues could not solve, with
no hard alpha surgery.
- SegmentationSettings wired into PipelineSettings/config; --device propagates.
- TrimapSettings: seg_core_threshold, seg_loose_threshold.
- New debug output seg_mask.png.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lhk229
Codex