# BgFilter

> 中文: [README_ZH.md](README_ZH.md)

Offline character matting for AI-generated images on a flat-colour background.
The background colour is **auto-detected** from the image border (green, pastel, any
flat colour); pass `--screen-color` to set it explicitly.

```text
RGB input
  -> chroma bg-confidence (keyed to the auto-detected or given colour)
  -> [optional] semantic segmentation mask
  -> trimap -> ViTMatte -> alpha cleanup
  -> cross-model veto (second matting opinion on bg-hued residue)
  -> pymatting foreground -> despill -> RGBA PNG -> QA previews
```

## Pipelines

Two pipelines, selected by `segmentation.enabled` in the config:

- **Chroma-only** (`enabled: false`) — Chroma + ViTMatte, no segmentation model.
  Lightest / fastest; leans entirely on the colour key for topology.
- **Single segmenter** (`enabled: true`, default) — one segmentation model drives the
  trimap topology (holes, hair), ViTMatte then refines the soft edges. Backend is
  switchable: `birefnet` (default, general salient objects — text, logos, photos; needs
  `trust_remote_code`) or `anime-seg` (ONNX, tuned for anime characters). Switch at
  runtime with `--seg-backend anime-seg` (it also selects the matching weights).

Both share pymatting foreground estimation and a colour de-spill. There is **no**
green-contamination rescue / recolour layer — with clean source images it is
unnecessary, so it was removed.

Both also run a **cross-model veto** by default: a second, trimap-free matting
model (`ZhengPeng7/BiRefNet_HR-matting`) may only *lower* alpha, only on
background-hued bright pixels the primary result is confident about — this clears
colour-drifted background residue trapped between hair strands that the chroma
key, the segmenter and ViTMatte all read as foreground. Costs one extra model
download (~0.9 GB) and one inference pass per image; disable with
`--no-cross-check` (see the `cross_check` config section, and
`docs/hair_gap_artifacts.md` for the analysis behind it).

When the cross-check is on, that same HR-matting forward can optionally be
**reused as the segmentation mask** (`--cross-check-as-seg`), skipping the
primary seg model entirely (one less model to load, ~20 s faster per image on
CPU, ~1 GB less VRAM on GPU). Pilot-validated (TestImage3 / FixImage1): trimap
99.8% identical, no structural change to fingers, hair wisps or thin lines.
**Off by default**: the dedicated segmenter keeps its own forward and the veto
stays an independent second signal. The reuse is a same-family swap, so it
applies to the `birefnet` backend only: with `--seg-backend anime-seg` the
anime segmenter always keeps its own forward.

The segmentation trimap defaults to `directional` mode (chroma + seg + a hue-direction
split: it keeps a background-coloured garment such as a white shirt while dropping a
background-hued residual such as blue trapped between hair strands). Switch with
`--trimap-mode seg` (topology only, no hue split) or `directional-hard-bg` (aggressive
— hard-removes background-hued pixels; can eat cool/shadowed white cloth).

## Background colour

By default (`screen_color: null`) the background colour is **auto-detected** from the
image border: the dominant flat colour of the border strip becomes the key colour. If
the border is not one clean flat colour — a gradient, texture, or a subject filling the
frame — detection **fails with an error**; pass `--screen-color` explicitly in that case.

To set it yourself, give a hex prior:

```bash
... --screen-color "#CFEFFF"
```

or `screen_color: "#CFEFFF"` in the config. Either way the chroma key scores pixels by
perceptual (Lab/RGB) distance to the colour, and de-spill removes chroma along that
colour's direction. (A supplied hex is refined against nearby border pixels; an
auto-detected colour is used directly.)

## Environment

Any Python 3.11–3.13 environment (venv or conda) works:

```bash
pip install -r requirements.txt
python -m bgfilter.cli --help
```

For a server install — CPU-wheel torch, offline weights, systemd — follow
[DEPLOY.md](DEPLOY.md) instead; it covers the same dependencies with the
production choices spelled out.

## Single Image

```bash
python -m bgfilter.cli \
  --input Samples/TestImage.png \
  --output Outputs/TestImage_rgba.png \
  --debug-dir Outputs/TestImage_debug
```

No `--config` is needed — the built-in defaults are identical to
`configs/default.yaml`. Pass `--config configs\default.yaml` only after you edit that
file to tune the detailed parameters. Runtime choices stay on the command line:
`--device` (default **CPU**; use `--device cuda` for GPU), `--precision` (default
`fp32`; `bf16` speeds up all three models and halves the matting model's activation
memory with visually identical alpha — needs bf16-capable hardware, falls back to
fp32 elsewhere; large inputs additionally get ViTMatte's global attention computed
in query chunks by default — exact, bitwise-identical, caps the memory spike at
~4 GB instead of ~19 GB at 2048x2048, see `model.attn_query_chunk`), `--seg-backend`
(default `birefnet`; `anime-seg` for anime characters), `--screen-color` (default:
auto-detect the flat background), and `--trimap-mode`.

## Batch

```bash
python -m bgfilter.cli \
  --input-dir Samples \
  --output-dir Outputs \
  --debug-dir Outputs/debug
```

## Performance

Rough orders of magnitude: **~33–53 s/image** on a fast desktop CPU (bf16,
cross-check on), **~1–2 s/image** on a modern GPU. All measured numbers — CPU
lanes, GPU VRAM footprints, memory ceilings and the knobs that move them — live
in [DEPLOY.md](DEPLOY.md) §11 (*Capacity & performance*).

## Chroma-alpha debug mode

`--matting-method chroma` skips ViTMatte and uses chroma confidence directly as the
alpha seed. Useful for fast inspection of chroma confidence, trimap, and despill.
(Distinct from the chroma-only *pipeline* above, which still runs ViTMatte.)

```bash
python -m bgfilter.cli \
  --input-dir Samples \
  --output-dir Outputs/chroma \
  --debug-dir Outputs/chroma_debug \
  --matting-method chroma \
  --device cpu
```

## Outputs

For each processed image, the CLI writes an RGBA PNG and optional debug files:

```text
bg_confidence.png
trimap.png
seg_mask.png          # segmentation pipeline only
alpha.png
foreground_rgb.png    # despilled foreground colour
color_mask.png        # per-pixel despill weight
preview_black.png
preview_white.png
preview_gray.png
preview_red.png
preview_blue.png
qa_grid.png
metadata.json
```

## HTTP service

An HTTP wrapper (`app.py` + `bgfilter/service.py`) exposes the pipeline as a
long-running FastAPI service. Models load once and are reused across requests.
For deploying to a Linux server (dependencies, weights, systemd, offline mode, auth),
see [DEPLOY.md](DEPLOY.md).

```bash
python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1
```

Run a single worker (`--workers 1`) — each worker loads its own copy of the models.

### Endpoints

- **`GET /healthz`** — liveness + config JSON, e.g.
  `{"ok":true,"service":"bgfilter","version":"0.1.0","defaultSegModel":"birefnet","device":"cpu","crossCheck":true}`.
  Always unauthenticated.
- **`POST /remove-background`** — `multipart/form-data` in, `image/png` (RGBA) out.

### Authentication

Optional, opt-in via the `BGFILTER_AUTH_TOKEN` env var. When set, `/remove-background`
requires header **`X-Genarrative-Image-Token: <token>`** (constant-time compare); a
missing or wrong token returns `401`. When unset the endpoint is open (a fronting proxy
is expected to authenticate). `/healthz` is always open. See [DEPLOY.md](DEPLOY.md) §10.

### Request fields (`POST /remove-background`, `multipart/form-data`)

| Field | Required | Values | Meaning |
| --- | --- | --- | --- |
| `file` | one of `file`/`image_url` | image file | Source image as an upload (field name fixed as `file`). |
| `image_url` | one of `file`/`image_url` | `http(s)://…` | Source image by URL — the service fetches it (bounded by size/time; follows ≤3 redirects). Provide exactly one of `file` or `image_url`. No SSRF filtering: the endpoint trusts authenticated callers, so keep auth on and callers trusted. |
| `background_mode` | no | `flat` (default) / `complex` | `flat` = solid-colour background (colour-keyed pipeline). `complex` = non-flat/scene background: no colour key, the segmenter alone drives the matte. `screen_color` is ignored in `complex`; expect flat backgrounds to stay stronger on hair detail. |
| `screen_color` | no | `#RRGGBB`, or omit/empty/`auto` | Background-colour prior; omit to auto-detect from the border. Ignored when `background_mode=complex`. |
| `seg_model` | no | `birefnet` (default) / `anime-seg` | Segmentation backend. |
| `cross_check` | no | `1`/`0` (also `true`/`false`/`yes`/`no`/`on`/`off`) | Per-request override of the cross-model veto; omit/empty = service default in `flat` mode, but **defaults off** in `complex` mode (set `on` to enable its hue-free gate). |

### Response headers (on `200`)

| Header | Example | Meaning |
| --- | --- | --- |
| `X-BGFilter-Elapsed-Ms` | `2372` | Server-side processing time (ms). |
| `X-BGFilter-Seg-Model` | `birefnet` | Segmenter actually used. |
| `X-BGFilter-Screen-Color` | `#CFEFFF` / `auto` | Background colour used. |
| `X-BGFilter-Cross-Check` | `on` / `off` | Effective cross-check state. |
| `X-BGFilter-Background-Mode` | `flat` / `complex` | Background mode actually used. |
| `X-Request-ID` | `42efe4e0d76f4b83` | Correlation id (echoes an inbound `X-Request-ID` if you send one, else generated). Quote it to correlate with server logs. Returned on every response, including errors. |

### Status codes

| Code | When |
| --- | --- |
| `200` | Success — body is the RGBA PNG. |
| `400` | Bad input: unreadable image, not exactly one of `file`/`image_url`, bad `image_url` / fetch failed, invalid `screen_color`/`seg_model`/`cross_check`/`background_mode`, or auto-detect failed (pass `screen_color`). |
| `401` | Auth enabled and the token is missing or wrong. |
| `413` | Image exceeds `BGFILTER_MAX_IMAGE_PIXELS` (~4 MP by default), or an `image_url` body exceeds `BGFILTER_FETCH_MAX_BYTES` (~25 MB). |
| `500` | Inference failed. |
| `503` | Service still loading models (not ready). |

### Examples

```bash
# default (birefnet + auto background colour); add the auth header if the service requires it
curl -sS -F "file=@input.png" \
  -H "X-Genarrative-Image-Token: <token>" \
  http://127.0.0.1:18083/remove-background -o output.png

# explicit background colour + anime segmenter, cross-check off for this request
curl -sS \
  -F "file=@input.png" \
  -F "screen_color=#CFEFFF" \
  -F "seg_model=anime-seg" \
  -F "cross_check=off" \
  http://127.0.0.1:18083/remove-background -o output.png

# non-flat / scene background (no colour key); cross-check defaults off here,
# add cross_check=on to enable the hue-free residue veto
curl -sS \
  -F "file=@scene.jpg" \
  -F "background_mode=complex" \
  http://127.0.0.1:18083/remove-background -o output.png

# by URL instead of an upload (service fetches it; use image_url OR file, not both)
curl -sS \
  -F "image_url=https://example.com/input.png" \
  -F "screen_color=#CFEFFF" \
  -H "X-Genarrative-Image-Token: <token>" \
  http://127.0.0.1:18083/remove-background -o output.png
```

Runtime configuration (config file, device, size caps, preload, auth token,
pipelining workers) is all via `BGFILTER_*` environment variables — the full table
is [DEPLOY.md](DEPLOY.md) §6.

Like the CLI, the service runs the **cross-model veto by default**; `/healthz`
reports the effective default as `crossCheck` and the per-request `cross_check`
field overrides it either way (service-wide kill switch: `BGFILTER_CROSS_CHECK=0`).
`cross_check.reuse_as_seg` (config-only, default off — see *Pipelines* above) has
no per-request switch; changing it means editing the config and restarting.

## Quality Check

The quality checker measures alpha validity and edge spill on semi-transparent edge
pixels.

```bash
python -m bgfilter.quality_cli \
  Outputs/TestImage_rgba.png \
  --max-edge-green-excess-p95 0.30
```

Run the bundled sample smoke check:

```bash
python scripts/smoke_samples.py \
  --samples-dir Samples \
  --output-dir Outputs/smoke_samples \
  --config configs/default.yaml \
  --device cpu \
  --fallback-to-chroma-alpha \
  --max-edge-green-excess-p95 0.30
```

## Notes

- `Samples/` and `Outputs/` are ignored by Git.
- ViTMatte and segmentation weights load from Hugging Face on first use.
  `anime-seg` (`skytnt/anime-seg`) is a plain ONNX download; `birefnet`
  (`ZhengPeng7/BiRefNet`) ships custom modelling code so it needs
  `trust_remote_code=True` plus `timm` / `einops` / `kornia`.
- **Provisioning weights** — offline HF cache, plain `models/` folders
  (`BGFILTER_WEIGHTS_DIR`), the `scripts/fetch_weights.py` one-shot downloader,
  and the CN-network mirror workaround — is covered in [DEPLOY.md](DEPLOY.md) §5.
- Foreground colour estimation uses pymatting's `estimate_foreground_ml` to propagate
  clean foreground colour into semi-transparent edges before de-spill. Set
  `foreground.method: unmix` to fall back to the legacy heuristic.
- `docs/green_screen_matting_workflow.md` is the original phase-1 green-screen spec;
  this README reflects the current, generalised architecture.