Three optimizations from profiling the cross-check-dominated pipeline
(9700X CPU, all pilot-validated on TestImage3/FixImage1):
- Reuse the cross-check HR-matting@2048 forward as the segmentation mask
(cross_check.reuse_as_seg, default ON; --no-cross-check-as-seg to opt
out). Skips the BiRefNet@1024 load+forward entirely: ~66s -> ~45s,
one less 0.9GB model. Trimap 99.8% identical, no structural change.
- --precision bf16 now fans out to all three models: ViTMatte keeps its
weight cast; both BiRefNets run their forward under autocast with a
dispatcher-level AutocastCPU fp32 shim for torchvision::deform_conv2d
(no bf16 CPU kernel, no autocast wrapper upstream). Shared hardware
gate in bgfilter/precision.py falls back to fp32 off native-bf16
hardware. TestImage3: 51.9s -> 33.1s; alpha diff max 0.15, none >0.25.
- MIMALLOC_PURGE_DELAY=0 (bgfilter/__init__.py, before torch loads):
Windows torch's bundled mimalloc lazily retains ~10GB of freed
BiRefNet activations, stacking under ViTMatte's attention peak.
2048x2048 bf16: peak 25.2 -> 21.4GB and slightly faster (73 -> 63s).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bf16 halves the matting model's activation memory -- its full-resolution
attention is the pipeline's memory peak -- with visually identical alpha
(measured: 0 px alpha deviation > 0.25 on samples; cross-check veto
behaviour unchanged, region IoU 0.984).
Guarded by a hardware check so it never lands on a slow emulation path:
CUDA requires torch.cuda.is_bf16_supported(); CPU requires the same
oneDNN native-bf16 gate PyTorch uses for matmul routing (AVX512-BF16/
AMX). Without support it warns and falls back to fp32 -- on a Zen2 EPYC
the fallback kernels measured 17-370x slower than fp32, so silent bf16
there would be a performance landmine.
Segmentation stays fp32: torchvision deform_conv2d (used by BiRefNet)
has no bf16 CPU kernel, and the segmenter is not the memory peak.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both call sites convert outputs to numpy immediately, so the stricter
inference-mode tensors are safe; saves autograd view/version tracking
overhead. Verified end-to-end on CUDA (alpha finite, no NaN).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colour-drifted background trapped between hair strands defeats every
single-signal defence: the chroma key reads it as foreground (bgc ~0.07),
the segmenter backs it, ViTMatte rates it opaque, and post-hoc removal is
a proven dead end (it shreds the hair volume the same pixels belong to).
A second, trimap-free matting model (BiRefNet_HR-matting) is the only
tested model that separates this residue from the subject, so its opinion
is fused in as a veto: min-fusion that may only LOWER alpha, restricted to
the background-hued bright suspect zone (proj >= 3, L >= 45, feathered)
and gated by primary-alpha confidence (0.70 -> 0.95 ramp) so soft wisps
and dark hair are exempt by construction.
- settings/config/CLI: cross_check block, --cross-check/--no-cross-check
- alpha_post.cross_check_alpha after clean_alpha; second opinion reuses
BiRefNetSegmenter; saved to debug as cross_check_alpha.png
- chroma.bg_hue_projection extracted and shared with the trimap
- docs: methodology.md (new), hair_gap_artifacts.md (investigation log)
Verified: cross-check ON reproduces the visually-reviewed B1gate
prototype byte-for-byte on TestImage3; --no-cross-check reproduces the
previous baseline byte-for-byte; pink-bg FixImage1 face untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
suppress_alpha_by_chroma was the one stage where colour evidence could veto the
matting model outright, and every observed failure had the same shape: ViTMatte
was right (raw ~0.97 on a white shirt over pastel blue, on blush over pink) and
the bgc-keyed suppression crushed it, because bgc magnitude cannot tell a
background-coloured subject from background residue. The matte's raw alpha can:
it rates enclosed background pockets low (~0.28 median) and subject texture high.
Suppression is now scaled by (1 - smoothstep(raw_alpha, suppress_raw_lo,
suppress_raw_hi)) (defaults 0.85/0.98): colour evidence only vetoes the matte
where the matte itself is unsure. The gate applies only when alpha really comes
from ViTMatte; chroma-seeded alpha passes raw_alpha=None and keeps old behaviour.
Set suppress_raw_lo: 1.0 to disable (verified byte-identical to the old output).
Validated on both screen colours: pink #F4D8E8 face restored (skin alpha 0.841 ->
0.993, holes 17.2% -> 0.6%), pastel blue #CFEFFF unchanged-to-better (shirt +0.01,
wisps identical, gap clearing 99.8%). Also fixes the long-deferred chroma-only
pipeline defects (white shirt 0.900, light skin 0.989). CLI output verified
byte-identical to the reviewed prototype on both images.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BiRefNet is the default backend and --seg-backend switches segmenters, so the
birefnet.yaml config (a copy of default.yaml with backend: birefnet) no longer had
a purpose. configs/ now holds just default.yaml, the editable parameter template.
Also dropped the now-redundant --config from the chroma-debug README example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ModelSettings.model_name dataclass default was still vitmatte-small (the
phase-1 "start small" choice) while configs/default.yaml used vitmatte-base, so a
bare run (no --config) silently differed from every documented example. Set the
dataclass default to base too: the built-in defaults now match default.yaml
field-for-field, so --config is only needed after editing that file to tune
parameters. README examples drop the now-redundant --config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- chroma: drop BackgroundModel.hsv_center/hue_sigma (unused since the green gating
went away), their estimation, _hue_distance, and the now-unused hsv channel from
convert_color_spaces (one fewer cvtColor per image).
- despill: delete the dead green-channel-algebra path (the pipeline always passes a
colour) and fold the remaining Lab-direction despill into a single despill() with
no screen_color arg (127 -> 60 lines). Renamed despill_green -> despill.
- settings: remove the now-dead ChromaSettings.hue_sigma_min and
DespillSettings.green_excess_margin (plus the matching config lines).
- docs/comments: generalise stale "green" wording to "background colour" (module
docstring, alpha_post/trimap/foreground/settings) and rename the trimap variable
green_bg -> screen_bg.
Kept: the unmix foreground fallback, --matting-method chroma, and green/blue as
illustrative examples. Verified: package compiles, both configs parse, and an
auto-detect end-to-end run is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Segmentation backend now defaults to birefnet (ZhengPeng7/BiRefNet); anime-seg is
selected at runtime with --seg-backend anime-seg, which also swaps in the matching
weights (unknown backends raise). Config files stay for detailed tuning, not
backend selection.
- Default device is now cpu everywhere (dataclass defaults, default.yaml,
birefnet.yaml); pass --device cuda for GPU. Avoids a hard CUDA-required failure on
machines without a CUDA-enabled PyTorch build.
- Docs: README + workflow status note updated (backend via --seg-backend, CPU
default, examples no longer force --device cuda).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hardcoded green-screen auto-detection with a general flat-colour
detector. When screen_color is null the background colour is now detected from the
image border (dominant colour of the border strip) and used directly as the chroma
model, with a failure gate that raises when the border is not one clean flat colour
(gradient / texture / subject filling the frame).
- chroma: add detect_background_color + _background_border_cluster; the null case of
estimate_background_model samples the detected cluster directly (no seed search);
compute_bg_confidence is now always perceptual (Lab/RGB distance). Removed the green
heuristic (initial_green_candidates), the green multiplicative gating, the now-unused
_smoothstep, and five green-only ChromaSettings fields.
- pipeline: reuse the detected colour for de-spill in the auto case.
- settings / default.yaml: add detect_* tuning fields; refresh screen_color docs.
- README: document auto-detection + the failure gate, trimap modes, birefnet config.
A supplied --screen-color hex still uses the seed-search refinement. Validated:
detects green / pastel / text samples correctly, raises on gradient / two-colour, CLI
exits 1 on failure, green and pastel mattes unchanged in quality.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A copy of default.yaml with segmentation.backend switched to birefnet
(ZhengPeng7/BiRefNet). BiRefNet is a general salient-object segmenter, so it
handles non-anime subjects (text, logos, real photos) that anime-seg returns an
empty mask for. Run with: --config configs/birefnet.yaml
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the two experimental bools (directional, bg_hued_to_bg) with a single
TrimapSettings.mode selector for the segmentation pipeline:
- "directional" (new default): chroma magnitude + seg + a Lab hue-direction
split. In the chroma-unknown zone a confidently-segmented pixel stays
foreground unless it is displaced toward the background hue, so a neutral
background-coloured garment (e.g. a white shirt) is kept while a background-
hued residual (blue between hair strands) is left unknown for ViTMatte /
chroma-suppress to clear.
- "seg": original fuse_trimap, unchanged.
- "directional-hard-bg": aggressive variant that hard-removes background-hued
pixels (can eat cool/shadowed white cloth).
Selectable via configs/default.yaml (trimap.mode) or CLI --trimap-mode; unknown
modes raise. Default CLI output verified byte-identical to the reviewed UNK
result on the pastel sample.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README rewritten for the current design: two pipelines (chroma-only vs single
segmenter, default anime-seg), the screen_color prior (--screen-color / config,
green default), correct debug-file list, and HF/hf-mirror notes for the seg models.
No rescue/recolour layer is documented (it was removed).
green_screen_matting_workflow.md keeps the phase-1 green-screen spec but gains a
status note pointing to the README for the evolved architecture, and its debug-file
list is corrected (foreground_rgb / color_mask / seg_mask, not the old
foreground_background / foreground_correction / despill_mask).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strategic pivot: the dropout-fill rescue, the anime-seg fill backend, the
BiRefNet+anime-seg intersection, and the recolor pass all existed to repair
source images whose hair was already green-contaminated or broken at generation
time. Fixing the source instead (clean pastel-background generation) makes the
matte high-contrast, so that whole compensation layer is unnecessary. Remove it.
- Two pipelines via segmentation.enabled:
false -> chroma-only (Chroma + ViTMatte)
true -> single segmenter (default anime-seg, switchable birefnet) -> trimap
then ViTMatte refines, pymatting estimates foreground, despill cleans spill.
- Generalise the green-hardcoded colour logic to a screen_color prior (hex, e.g.
"#CFEFFF"; default null = green auto-detect, byte-identical). chroma keys off
Lab/RGB distance to the colour; despill removes chroma along its Lab direction.
Added --screen-color CLI flag.
- Removed dead code: fill_seg_dropouts + seg_fill*/intersection params,
SegmentationSettings.fill_backend/fill_model_name/sharpen, unsharp_mask,
recolor.py + RecolorSettings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add seg_fill_intersect (default off) and seg_fill_gate_min to AlphaPostSettings, plus a gate_mask arg to fill_seg_dropouts. When enabled, the anime-seg dropout-fill is intersected with the reliable BiRefNet mask (>= gate_min): anime-seg proposes wisps at a low seg_fill_seg_min, but only rescues BiRefNet also endorses survive, trimming anime-seg's over-inclusive fringe while keeping its smooth value. Default off, so the shipped Option B (union) output is byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add SegmentationSettings.fill_backend / fill_model_name so the dropout-fill can
use a different (higher-recall) segmenter than the trimap. Default is now the
combination the investigation converged on: BiRefNet builds the trimap (reliable
topology -- finger gap removed, strands kept as sure_fg), anime-seg drives the
fill (more, more-coherent anime wisps). Because the fill only adds (max), a
higher-recall-but-less-reliable mask there is harmless.
On TestImage2 this eliminates the strand breakage a full anime-seg pipeline
caused (lost 699 -> 0) while keeping ~91% of its extra wisp recall (gained
1133 -> 1031). Strands stay continuous; the recovered wisps are anime-seg's
coherent curves rather than BiRefNet's grainy fill.
Also adds an optional unsharp-mask knob (segmentation.sharpen, default off).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add AnimeSegSegmenter (skytnt/anime-seg ISNet ONNX via onnxruntime, no remote
code) and a make_segmenter factory selected by SegmentationSettings.backend
("birefnet" | "anime-seg"). The ONNX output is already 0..1, so it slots into the
same soft-mask interface BiRefNetSegmenter uses.
On the anime samples anime-seg recovers more and more-coherent hair wisps than
BiRefNet (TestImage2 shoulder rescue: added px 1886 -> 3278, largest connected
component 147 -> 454), as expected from an anime-trained model. Default backend
stays birefnet.
Adds onnxruntime to requirements.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rescue floored alpha with BiRefNet's soft mask verbatim, which sprinkled
~800 isolated <10px specks (burrs) across the matte from the mask's scattered
low-confidence noise. Drop rescued connected components below a min area, keeping
only the connected wisp structures.
On TestImage2 the rescued components fall from 900 (818 isolated specks) to 91
and the added pixels from 5124 to 1886 -- the genuine wisp strands stay, the
burrs go. smoke still passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add bgfilter/recolor.py: after despill, propagate clean neighbour chroma into
pixels whose colour is still off (residual green, or the magenta a green unmix
overshoots into), keeping luminance partway so hair texture survives. It installs
a real observed colour rather than subtracting, so it cannot overshoot. Despill
already neutralises the matte edge (off ~ 0 there), so recolor only touches the
contaminated interior and the wisps the dropout-fill rescued.
Runs as a final colour pass on the despilled foreground; pymatting + despill keep
their stronger edge handling. On the samples the magenta fringe drops sharply
(TestImage vis-magenta 0.065->0.014, TestImage2 0.085->0.031) with edge green
unchanged (vis-green ~0.02). Debug output color_mask.png replaces the despill /
foreground-background / foreground-correction maps; smoke updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add fill_seg_dropouts: where ViTMatte's alpha is ~0 but BiRefNet confidently
asserts foreground, restore a faint alpha from the mask (strength * seg_mask).
This recovers green-contaminated wisps that ViTMatte cuts because they are
nearly the background colour, while leaving ViTMatte's sharper edges elsewhere
untouched (gated to true dropouts) and not refilling the finger gap (low mask).
On the TestImage2 shoulder, faint wisps ViTMatte cut drop from 72% to 33%; the
finger gap stays transparent. The rescued wisps are still green-tinted -- the
recolor pass (next) restores their colour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route faint hair (BiRefNet M 0.08-0.15) into the unknown band instead of sure
background, giving ViTMatte a chance at it. On its own this does not recover the
green-contaminated shoulder wisps: those are ~pure green (dominance +0.88, like
the screen) so ViTMatte cuts them regardless of trimap -- that is handled by the
upcoming segmentation dropout-fill. The finger gap (M~0.02) stays sure background.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire BiRefNet into the pipeline (lazy-loaded) and build the trimap from the
semantic subject mask refined by the green key, used when segmentation is
enabled. generate_trimap (chroma-only) stays as the no-segmentation fallback.
Authority split: the mask decides subject topology (keeps green-tinted hair as
foreground, drops see-through holes), the bright-green key sharpens the screen
boundary, ViTMatte refines a small protective band. Background follows the mask
directly so interior holes survive; the band is small so it does not swallow
finger gaps.
End-to-end on the samples: green hair preserved, finger-gap green 99%
transparent (BiRefNet marks it background; was 100% foreground before), leg
edges smooth -- topology/colour/edge problems local cues could not solve, with
no hard alpha surgery.
- SegmentationSettings wired into PipelineSettings/config; --device propagates.
- TrimapSettings: seg_core_threshold, seg_loose_threshold.
- New debug output seg_mask.png.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds bgfilter/segmentation.py: BiRefNetSegmenter produces a soft subject mask
that defines topology -- it includes colour-contaminated foreground (green-tinted
hair) and excludes see-through holes (finger gaps), the distinction chroma/colour
cues cannot make. Validated on both samples: green hair -> foreground, finger gap
-> background, both correct.
Foundation of the semantic-guided redesign; the mask will drive trimap
construction next, with ViTMatte still refining soft edges.
- SegmentationSettings (ZhengPeng7/BiRefNet, 1024 input, fp32).
- Deps: torchvision, timm, einops, kornia. Weights load from HuggingFace via
trust_remote_code; behind a firewall set HF_ENDPOINT=https://hf-mirror.com and
bypass the proxy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This reverts commit f7e12d3.
The carve set alpha to 0 on a colour-thresholded mask across the whole
foreground, which cannot distinguish background green from green the AI
generator bled onto the subject. It therefore (1) deleted green-tinted hair
strands and (2) hard-cut the green-spill silhouette edge, replacing ViTMatte's
anti-aliased edge with a jagged one. Local cues (surround, opacity, thinness,
connectivity, neighbour colour) could not reliably separate green hair from
green holes, so the mask is not fixable by tuning. Reverting restores intact
hair and smooth edges; the finger-gap green stays a minor residual.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The green in the finger gap survives keying because it is dark and desaturated
(brightness ~0.58, hue-shifted), so chroma confidence reads ~0 and the trimap
leaves it in the unknown band, where ViTMatte fills the pocket as foreground.
It is also connected to the exterior screen (an open notch), so neither
confidence suppression nor classic hole-filling removes it.
Add carve_background_holes (doc section 10): a pixel that is clearly
green-dominant in the input *and* mostly surrounded by genuine (non-green)
foreground is set to alpha 0. The surround cue separates such an intrusion
(finger gaps, inter-strand pockets) from the outer hair edge, which has
background on one side and is preserved. Runs after alpha cleanup.
On TestImage the finger-gap survivors (green & alpha>0.5) drop 170 -> 28 with
no visible loss of hair wisps; inter-strand green pockets clear too. Knobs in
AlphaPostSettings (hole_*, default on, surround_min 0.4).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull alpha toward 0 in the unknown band where bg_confidence is high, so green
that survives in hair gaps and hole pockets goes transparent (doc section 10).
Runs between trimap enforcement and alpha cleanup; sure-foreground pixels are
never touched, keyed by the bg_confidence we already compute.
On the two samples this clears the faint outer green halo (edge pixels
19.3k -> 16.0k) but has little *visible* effect, because their residual is no
longer green: it is magenta from the pymatting unmix on genuine semi-transparent
hair (vis-magenta p95 ~0.07, identical pre/post despill), which this pass
deliberately leaves alone. The magenta needs a separate colour fix.
Knobs live in AlphaPostSettings (chroma_suppress*, default on).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the full-foreground despill (5ddf0cd) the residual green was confined to
semi-transparent hair (alpha < 0.5): 85% of the worst pixels had alpha < 0.3,
where a pixel-wise unmix (divide by small alpha) is too noisy to trust.
Replace the hand-rolled unmix/local-blur estimator with pymatting's
estimate_foreground_ml (Germer et al. multi-level closed form), which propagates
reliable foreground colour from high-alpha neighbours into the fringe and
estimates the background, so green spill is unmixed rather than clamped. The
despill pass stays as a light cleanup on top.
- foreground.method selects "ml" (default) or "unmix" (legacy heuristic kept as
a fallback when pymatting is unavailable, matching the chroma fallback idiom).
- ForegroundEstimate now exposes rgb (F), background (B) and a correction map;
debug outputs become foreground_rgb / foreground_background / foreground_correction.
Controlled comparison (same ViTMatte alpha, only foreground method changed),
edge_green_excess:
TestImage p95 0.176->0.031, mean 0.061->0.005
TestImage2 p95 0.165->0.004, mean 0.052->0.002
Adds pymatting (pulls in numba) to requirements.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Green spill bleeds into pixels ViTMatte marks as solid foreground (alpha~1),
but despill was gated to the soft edge band (0.005<alpha<0.995) and weakened
exactly there: the alpha tent peaked at 0.5 and bg_confidence (low on the
character edge) further suppressed the correction.
- (1) Apply despill to the whole foreground (alpha > edge_low) instead of the
soft band only; neutral pixels stay untouched via the green-excess term.
- (2) Weight by alpha at full strength across the opaque range (alpha>=0.5),
easing off only where mostly transparent, instead of a tent peaking at 0.5.
- (3) Drop bg_confidence as a despill gate; localise purely by green excess.
Controlled comparison (same alpha, only despill changed), edge_green_excess:
ViTMatte TestImage p95 0.239->0.176, mean 0.131->0.061
ViTMatte TestImage2 p95 0.231->0.165, mean 0.115->0.052
Removes now-unused DespillSettings fields edge_high and bg_confidence_weight.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lhk229
Codex