Cross-check + reuse-as-seg are already the config/CLI default; align the HTTP
service with them instead of forcing cross-check off. The earlier off-by-default
was to avoid the extra inference, but reuse-as-seg now makes that forward double
as the seg mask (birefnet backend) -- it replaces the primary seg model rather
than adding to it, so the cost concern is gone.
- app.py: honour the config default (on); BGFILTER_CROSS_CHECK still forces
either way, unset = config.
- service.py preload: warm the cross-check HR-matting model, and skip the now-
redundant primary segmenter when reuse covers segmentation.
- README / DEPLOY(_ZH): document cross-check as default-on, BiRefNet_HR-matting
as required (not optional), BGFILTER_CROSS_CHECK=0 to disable.
Verified: service defaults cross_check on, env forces both ways, preload picks
cross-checker and skips the primary segmenter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adopts master as the standard for all overlapping work. Master's landed
optimizations supersede server-edition's own fp16 experiment:
- Unified --precision fp32|bf16 knob (bgfilter/precision.py) driving all
three models: ViTMatte weight cast + both BiRefNets via autocast, with a
hardware gate (falls back to fp32 off native-bf16 CPUs) and an
AutocastCPU fp32 shim for torchvision deform_conv2d.
- Query-chunked ViTMatte global attention (bgfilter/attn_chunk.py), exact
and bitwise-identical, caps the N^2 spike (~19 -> ~4 GB at 2048).
- Cross-check HR-matting forward reused as the seg mask (birefnet backend
only), skipping the primary seg model; MIMALLOC_PURGE_DELAY=0.
- inference_mode and the detect_background_color removal converge with
server-edition's earlier equivalents.
Conflict resolution (favoring master, preserving server-only features):
- vitmatte_infer/segmentation: dropped server's device-derived fp16 for
master's precision path, kept resolve_model_source (local weights).
- service.py: cross-check SegmentationSettings now passes precision so the
HTTP service honors bf16 like the CLI's _second_opinion does.
Verified on CPU: default fp32 pipeline loads 2 models (seg-reuse active)
and bf16 path runs (autocast + deform_conv2d shim) — both exit 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
reuse_as_seg swaps one BiRefNet-family model for another (the validated
equivalence); it must not silently override an explicitly chosen anime-seg
backend. With --seg-backend anime-seg the segmenter now keeps its own
forward and the cross-check veto runs independently on top -- both
combinable, no flag juggling. Default birefnet path is untouched
(verified byte-identical); anime-seg + cross-check verified to run both
models.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- chroma.py: remove detect_background_color — orphaned since auto-detection
goes through estimate_background_model -> _background_border_cluster;
nothing in the repo calls it.
- settings.py: fix stale "Segmentation always stays fp32" comment
(--precision now fans out to the BiRefNet models too).
- README: add a measured CPU performance section (9700X reference numbers,
memory ceiling explanation, Zen 2 fallback guidance).
- docs/hair_gap_artifacts.md: record that the cross-check cost note is
obsolete — reuse_as_seg returns the net model count to 2, bf16 and
chunked attention absorb the rest.
- docs/green_screen_matting_workflow.md: add the 2026-07 additions to the
architecture-evolution note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ViTMatte's VitDet backbone runs 4 global attention blocks that materialize
the full [heads x N x N] map: ~19 GB transient at 2048x2048 (16384 tokens),
the pipeline's memory peak. transformers has no SDPA path for this
architecture (the decomposed rel-pos bias is added to raw scores), so
compute the same attention in query-row chunks instead: the bias
factorizes over query rows, making the chunked form mathematically exact
-- output verified BITWISE-identical (unit: fp32/bf16 x 3 sizes; full
pipeline: TestImage3 fp32 and 2048x2048 bf16, all byte-equal).
model.attn_query_chunk (default 2048, 0 = stock one-shot) engages only on
blocks seeing more tokens than the chunk size, so window blocks keep the
original path. Measured @2048x2048 bf16 (9700X): ViTMatte spike
19.8 -> 4.0 GB for ~15% more ViTMatte time; whole pipeline peak
21.4 -> 11.4 GB, warm 51 -> 53 s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three optimizations from profiling the cross-check-dominated pipeline
(9700X CPU, all pilot-validated on TestImage3/FixImage1):
- Reuse the cross-check HR-matting@2048 forward as the segmentation mask
(cross_check.reuse_as_seg, default ON; --no-cross-check-as-seg to opt
out). Skips the BiRefNet@1024 load+forward entirely: ~66s -> ~45s,
one less 0.9GB model. Trimap 99.8% identical, no structural change.
- --precision bf16 now fans out to all three models: ViTMatte keeps its
weight cast; both BiRefNets run their forward under autocast with a
dispatcher-level AutocastCPU fp32 shim for torchvision::deform_conv2d
(no bf16 CPU kernel, no autocast wrapper upstream). Shared hardware
gate in bgfilter/precision.py falls back to fp32 off native-bf16
hardware. TestImage3: 51.9s -> 33.1s; alpha diff max 0.15, none >0.25.
- MIMALLOC_PURGE_DELAY=0 (bgfilter/__init__.py, before torch loads):
Windows torch's bundled mimalloc lazily retains ~10GB of freed
BiRefNet activations, stacking under ViTMatte's attention peak.
2048x2048 bf16: peak 25.2 -> 21.4GB and slightly faster (73 -> 63s).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bf16 halves the matting model's activation memory -- its full-resolution
attention is the pipeline's memory peak -- with visually identical alpha
(measured: 0 px alpha deviation > 0.25 on samples; cross-check veto
behaviour unchanged, region IoU 0.984).
Guarded by a hardware check so it never lands on a slow emulation path:
CUDA requires torch.cuda.is_bf16_supported(); CPU requires the same
oneDNN native-bf16 gate PyTorch uses for matmul routing (AVX512-BF16/
AMX). Without support it warns and falls back to fp32 -- on a Zen2 EPYC
the fallback kernels measured 17-370x slower than fp32, so silent bf16
there would be a performance landmine.
Segmentation stays fp32: torchvision deform_conv2d (used by BiRefNet)
has no bf16 CPU kernel, and the segmenter is not the memory peak.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /remove-background gains an optional cross_check form field
(1/0/true/false/yes/no/on/off; empty = service default). The effective
state is reported in the X-BGFilter-Cross-Check response header.
The cross-check model is now never preloaded at startup: the first
request that enables it pays the one-time load (measured ~8 s on top of
the ~50 s cc inference), after which the instance is cached and shared.
Verified with a functional test against PipelineManager: cc-off requests
never load the model, cc=1 lazy-loads then reuses the cached instance,
and omitting the field falls back to the service default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cross-model veto adds one extra large-model inference per request
(measured ~35 s/image on CPU: 14.4 s -> 49.1 s), which is too slow as a
service default. The service now disables it regardless of the config
default; BGFILTER_CROSS_CHECK=1 re-enables it (GPU or quality-first
deployments). CLI behaviour is unchanged (still on by default).
Startup log and /healthz report the effective state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings in the two algorithm optimisations from master:
- Gate chroma suppression by the matte's own confidence (639fdf0)
- Add cross-model veto of hair-gap background residue (adbbb9d)
Server-edition integration on top of the plain merge:
- PipelineManager caches the cross-check model (BiRefNet_HR-matting) and
injects it into MattingPipeline, so HTTP requests never reload it.
- MattingResult carries cross_check_alpha; debug output matches master.
- /healthz reports crossCheck; README lists models/BiRefNet_HR-matting
in the local-weights bundle.
Verified: CLI --no-cross-check baseline and full cross-check run both
pass on samples/TestImage3.png; cross_check_alpha.png written to debug.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Colour-drifted background trapped between hair strands defeats every
single-signal defence: the chroma key reads it as foreground (bgc ~0.07),
the segmenter backs it, ViTMatte rates it opaque, and post-hoc removal is
a proven dead end (it shreds the hair volume the same pixels belong to).
A second, trimap-free matting model (BiRefNet_HR-matting) is the only
tested model that separates this residue from the subject, so its opinion
is fused in as a veto: min-fusion that may only LOWER alpha, restricted to
the background-hued bright suspect zone (proj >= 3, L >= 45, feathered)
and gated by primary-alpha confidence (0.70 -> 0.95 ramp) so soft wisps
and dark hair are exempt by construction.
- settings/config/CLI: cross_check block, --cross-check/--no-cross-check
- alpha_post.cross_check_alpha after clean_alpha; second opinion reuses
BiRefNetSegmenter; saved to debug as cross_check_alpha.png
- chroma.bg_hue_projection extracted and shared with the trimap
- docs: methodology.md (new), hair_gap_artifacts.md (investigation log)
Verified: cross-check ON reproduces the visually-reviewed B1gate
prototype byte-for-byte on TestImage3; --no-cross-check reproduces the
previous baseline byte-for-byte; pink-bg FixImage1 face untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Document deploying the FastAPI service to a Linux server end to end: venv +
CPU-only torch (avoid the CUDA build), opencv-python-headless, the two weight
layouts (HF cache offline vs local models/ folders) with the mirror/proxy/Xet
download workaround, environment variables, uvicorn --workers 1, a systemd unit,
verification, remote exposure, capacity notes, and a troubleshooting table.
Link to it from README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add bgfilter/weights.py: resolve_model_source() maps a HuggingFace repo id
to a local folder under the weights dir (models/ by default, override with
BGFILTER_WEIGHTS_DIR) when one named after the repo basename exists; otherwise
the repo id is returned unchanged. Opt-in and backward compatible.
- vitmatte_infer.py / segmentation.py resolve model_name through it; anime-seg
reads <dir>/isnetis.onnx directly instead of hf_hub_download when local.
- .gitignore: models/, model-cache/
- README: document bundling weights as plain project folders.
Verified: anime-seg loads from a local models/anime-seg/ folder offline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BiRefNet is the default backend and --seg-backend switches segmenters, so the
birefnet.yaml config (a copy of default.yaml with backend: birefnet) no longer had
a purpose. configs/ now holds just default.yaml, the editable parameter template.
Also dropped the now-redundant --config from the chroma-debug README example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ModelSettings.model_name dataclass default was still vitmatte-small (the
phase-1 "start small" choice) while configs/default.yaml used vitmatte-base, so a
bare run (no --config) silently differed from every documented example. Set the
dataclass default to base too: the built-in defaults now match default.yaml
field-for-field, so --config is only needed after editing that file to tune
parameters. README examples drop the now-redundant --config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Segmentation backend now defaults to birefnet (ZhengPeng7/BiRefNet); anime-seg is
selected at runtime with --seg-backend anime-seg, which also swaps in the matching
weights (unknown backends raise). Config files stay for detailed tuning, not
backend selection.
- Default device is now cpu everywhere (dataclass defaults, default.yaml,
birefnet.yaml); pass --device cuda for GPU. Avoids a hard CUDA-required failure on
machines without a CUDA-enabled PyTorch build.
- Docs: README + workflow status note updated (backend via --seg-backend, CPU
default, examples no longer force --device cuda).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hardcoded green-screen auto-detection with a general flat-colour
detector. When screen_color is null the background colour is now detected from the
image border (dominant colour of the border strip) and used directly as the chroma
model, with a failure gate that raises when the border is not one clean flat colour
(gradient / texture / subject filling the frame).
- chroma: add detect_background_color + _background_border_cluster; the null case of
estimate_background_model samples the detected cluster directly (no seed search);
compute_bg_confidence is now always perceptual (Lab/RGB distance). Removed the green
heuristic (initial_green_candidates), the green multiplicative gating, the now-unused
_smoothstep, and five green-only ChromaSettings fields.
- pipeline: reuse the detected colour for de-spill in the auto case.
- settings / default.yaml: add detect_* tuning fields; refresh screen_color docs.
- README: document auto-detection + the failure gate, trimap modes, birefnet config.
A supplied --screen-color hex still uses the seed-search refinement. Validated:
detects green / pastel / text samples correctly, raises on gradient / two-colour, CLI
exits 1 on failure, green and pastel mattes unchanged in quality.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README rewritten for the current design: two pipelines (chroma-only vs single
segmenter, default anime-seg), the screen_color prior (--screen-color / config,
green default), correct debug-file list, and HF/hf-mirror notes for the seg models.
No rescue/recolour layer is documented (it was removed).
green_screen_matting_workflow.md keeps the phase-1 green-screen spec but gains a
status note pointing to the README for the evolved architecture, and its debug-file
list is corrected (foreground_rgb / color_mask / seg_mask, not the old
foreground_background / foreground_correction / despill_mask).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the full-foreground despill (5ddf0cd) the residual green was confined to
semi-transparent hair (alpha < 0.5): 85% of the worst pixels had alpha < 0.3,
where a pixel-wise unmix (divide by small alpha) is too noisy to trust.
Replace the hand-rolled unmix/local-blur estimator with pymatting's
estimate_foreground_ml (Germer et al. multi-level closed form), which propagates
reliable foreground colour from high-alpha neighbours into the fringe and
estimates the background, so green spill is unmixed rather than clamped. The
despill pass stays as a light cleanup on top.
- foreground.method selects "ml" (default) or "unmix" (legacy heuristic kept as
a fallback when pymatting is unavailable, matching the chroma fallback idiom).
- ForegroundEstimate now exposes rgb (F), background (B) and a correction map;
debug outputs become foreground_rgb / foreground_background / foreground_correction.
Controlled comparison (same ViTMatte alpha, only foreground method changed),
edge_green_excess:
TestImage p95 0.176->0.031, mean 0.061->0.005
TestImage2 p95 0.165->0.004, mean 0.052->0.002
Adds pymatting (pulls in numba) to requirements.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lhk229
Codex