# BgFilter

Offline character matting for AI-generated images on a flat-colour background.
The background colour is **auto-detected** from the image border (green, pastel, any
flat colour); pass `--screen-color` to set it explicitly.

```text
RGB input
  -> chroma bg-confidence (keyed to the auto-detected or given colour)
  -> [optional] semantic segmentation mask
  -> trimap -> ViTMatte -> alpha cleanup
  -> cross-model veto (second matting opinion on bg-hued residue)
  -> pymatting foreground -> despill -> RGBA PNG -> QA previews
```

## Pipelines

Two pipelines, selected by `segmentation.enabled` in the config:

- **Chroma-only** (`enabled: false`) — Chroma + ViTMatte, no segmentation model.
  Lightest / fastest; leans entirely on the colour key for topology.
- **Single segmenter** (`enabled: true`, default) — one segmentation model drives the
  trimap topology (holes, hair), ViTMatte then refines the soft edges. Backend is
  switchable: `birefnet` (default, general salient objects — text, logos, photos; needs
  `trust_remote_code`) or `anime-seg` (ONNX, tuned for anime characters). Switch at
  runtime with `--seg-backend anime-seg` (it also selects the matching weights).

Both share pymatting foreground estimation and a colour de-spill. There is **no**
green-contamination rescue / recolour layer — with clean source images it is
unnecessary, so it was removed.

Both also run a **cross-model veto** by default: a second, trimap-free matting
model (`ZhengPeng7/BiRefNet_HR-matting`) may only *lower* alpha, only on
background-hued bright pixels the primary result is confident about — this clears
colour-drifted background residue trapped between hair strands that the chroma
key, the segmenter and ViTMatte all read as foreground. Costs one extra model
download (~0.9 GB) and one inference pass per image; disable with
`--no-cross-check` (see the `cross_check` config section, and
`docs/hair_gap_artifacts.md` for the analysis behind it).

The segmentation trimap defaults to `directional` mode (chroma + seg + a hue-direction
split: it keeps a background-coloured garment such as a white shirt while dropping a
background-hued residual such as blue trapped between hair strands). Switch with
`--trimap-mode seg` (topology only, no hue split) or `directional-hard-bg` (aggressive
— hard-removes background-hued pixels; can eat cool/shadowed white cloth).

## Background colour

By default (`screen_color: null`) the background colour is **auto-detected** from the
image border: the dominant flat colour of the border strip becomes the key colour. If
the border is not one clean flat colour — a gradient, texture, or a subject filling the
frame — detection **fails with an error**; pass `--screen-color` explicitly in that case.

To set it yourself, give a hex prior:

```powershell
... --screen-color "#CFEFFF"
```

or `screen_color: "#CFEFFF"` in the config. Either way the chroma key scores pixels by
perceptual (Lab/RGB) distance to the colour, and de-spill removes chroma along that
colour's direction. (A supplied hex is refined against nearby border pixels; an
auto-detected colour is used directly.)

## Environment

Use the conda environment `lightML`.

```powershell
conda activate lightML
pip install -r requirements.txt
```

If the shell is not activated, call the environment Python directly:

```powershell
D:\MiniConda\envs\lightML\python.exe -m bgfilter.cli --help
```

## Single Image

```powershell
D:\MiniConda\envs\lightML\python.exe -m bgfilter.cli `
  --input Samples\TestImage.png `
  --output Outputs\TestImage_rgba.png `
  --debug-dir Outputs\TestImage_debug
```

No `--config` is needed — the built-in defaults are identical to
`configs/default.yaml`. Pass `--config configs\default.yaml` only after you edit that
file to tune the detailed parameters. Runtime choices stay on the command line:
`--device` (default **CPU**; use `--device cuda` for GPU), `--seg-backend` (default
`birefnet`; `anime-seg` for anime characters), `--screen-color` (default: auto-detect
the flat background), and `--trimap-mode`.

## Batch

```powershell
D:\MiniConda\envs\lightML\python.exe -m bgfilter.cli `
  --input-dir Samples `
  --output-dir Outputs `
  --debug-dir Outputs\debug
```

## Chroma-alpha debug mode

`--matting-method chroma` skips ViTMatte and uses chroma confidence directly as the
alpha seed. Useful for fast inspection of chroma confidence, trimap, and despill.
(Distinct from the chroma-only *pipeline* above, which still runs ViTMatte.)

```powershell
D:\MiniConda\envs\lightML\python.exe -m bgfilter.cli `
  --input-dir Samples `
  --output-dir Outputs\chroma `
  --debug-dir Outputs\chroma_debug `
  --matting-method chroma `
  --device cpu
```

## Outputs

For each processed image, the CLI writes an RGBA PNG and optional debug files:

```text
bg_confidence.png
trimap.png
seg_mask.png          # segmentation pipeline only
alpha.png
foreground_rgb.png    # despilled foreground colour
color_mask.png        # per-pixel despill weight
preview_black.png
preview_white.png
preview_gray.png
preview_red.png
preview_blue.png
qa_grid.png
metadata.json
```

## HTTP service

An HTTP wrapper (`app.py` + `bgfilter/service.py`) exposes the pipeline as a
long-running FastAPI service. Models load once and are reused across requests.
For deploying to a Linux server (dependencies, weights, systemd, offline mode), see
[DEPLOY.md](DEPLOY.md).

```powershell
D:\MiniConda\envs\lightML\python.exe -m uvicorn app:app `
  --host 127.0.0.1 --port 18083 --workers 1
```

Two endpoints:

- `GET /healthz` — liveness/config JSON.
- `POST /remove-background` — `multipart/form-data`, returns `image/png`.
  - `file` (required) — the source image. The field name is fixed as `file`.
  - `screen_color` (optional) — `#RRGGBB` prior; omit/empty for auto-detect.
  - `seg_model` (optional) — `birefnet` (default) or `anime-seg`.
  - `cross_check` (optional) — `1`/`0` (also `true`/`false`/`yes`/`no`/`on`/`off`);
    per-request override of the cross-model veto. Omit/empty to use the service
    default. The response reports the effective state in `X-BGFilter-Cross-Check`.

```bash
# default (birefnet + auto background colour)
curl -sS -F "file=@input.png" \
  http://127.0.0.1:18083/remove-background -o output.png

# explicit background colour + anime segmenter
curl -sS \
  -F "file=@input.png" \
  -F "screen_color=#CFEFFF" \
  -F "seg_model=anime-seg" \
  http://127.0.0.1:18083/remove-background -o output.png

# enable the cross-model veto for this request only (see below; slow on CPU)
curl -sS -F "file=@input.png" -F "cross_check=1" \
  http://127.0.0.1:18083/remove-background -o output.png
```

Config comes from `BGFILTER_CONFIG` (defaults to `configs/default.yaml` when present);
`BGFILTER_DEVICE` overrides both model and segmentation device; `BGFILTER_MAX_IMAGE_PIXELS`
caps input size (default ~4MP → `413`); `BGFILTER_PRELOAD=1` loads the default models at
startup. Run a single worker (`--workers 1`) — each worker loads its own copy of the models.

Unlike the CLI, the service defaults the **cross-model veto to off** — it adds one
extra large-model inference (~35 s/image on CPU, 14 s → 49 s total). Enable it
per request with the `cross_check` form field, or flip the service default with
`BGFILTER_CROSS_CHECK=1` (worth it on GPU or when hair-gap residue matters more
than latency); `/healthz` reports the default as `crossCheck`. The cross-check
model is lazy-loaded: it is never preloaded at startup, so the first request that
enables it also pays the one-time model load (~1 GB) on top of the inference.

## Quality Check

The quality checker measures alpha validity and edge spill on semi-transparent edge
pixels.

```powershell
D:\MiniConda\envs\lightML\python.exe -m bgfilter.quality_cli `
  Outputs\TestImage_rgba.png `
  --max-edge-green-excess-p95 0.30
```

Run the bundled sample smoke check:

```powershell
D:\MiniConda\envs\lightML\python.exe scripts\smoke_samples.py `
  --samples-dir Samples `
  --output-dir Outputs\smoke_samples `
  --config configs\default.yaml `
  --device cpu `
  --fallback-to-chroma-alpha `
  --max-edge-green-excess-p95 0.30
```

## Notes

- `Samples/` and `Outputs/` are ignored by Git.
- ViTMatte and segmentation weights load from Hugging Face on first use.
  `anime-seg` (`skytnt/anime-seg`) is a plain ONNX download; `birefnet`
  (`ZhengPeng7/BiRefNet`) ships custom modelling code so it needs
  `trust_remote_code=True` plus `timm` / `einops` / `kornia`.
- Downloading the weights behind a firewall — the reliable combination is the
  `hf-mirror.com` mirror **with the local proxy bypassed** and **Xet disabled**:
  ```powershell
  $env:HF_ENDPOINT      = "https://hf-mirror.com"  # domestic mirror
  $env:NO_PROXY         = "*"                       # bypass the proxy; the mirror is direct
  $env:HF_HUB_DISABLE_XET = "1"                     # these repos are Xet-backed; force classic HTTP
  ```
  Two gotchas this avoids: (1) mirror + an overseas proxy makes the mirror's `resolve`
  bounce back to `huggingface.co`, which recent `huggingface_hub` rejects with
  `FileMetadataError`; (2) with `hf-xet` installed the Xet download path fails instantly.
  Once the weights are cached, run offline with `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1`
  (as the service does in production).
- Bundling weights in the project — instead of the HF cache, drop each model into a
  plain folder named after the repo basename under `models/`:
  `models/vitmatte-base-composition-1k`, `models/BiRefNet`, `models/anime-seg`,
  `models/BiRefNet_HR-matting` (cross-check second opinion, default on). The loader
  prefers a matching local folder and falls back to the HF repo id / cache when absent, so
  it is opt-in. Populate them with e.g. `hf download ZhengPeng7/BiRefNet --local-dir
  models/BiRefNet`. Override the base directory with `BGFILTER_WEIGHTS_DIR`. `models/` is
  gitignored.
- Foreground colour estimation uses pymatting's `estimate_foreground_ml` to propagate
  clean foreground colour into semi-transparent edges before de-spill. Set
  `foreground.method: unmix` to fall back to the legacy heuristic.
- `docs/green_screen_matting_workflow.md` is the original phase-1 green-screen spec;
  this README reflects the current, generalised architecture.