# Deploying BgFilter to a Linux server

> 中文版：[DEPLOY_ZH.md](DEPLOY_ZH.md)

This guide covers everything needed to run the BgFilter HTTP service
(`POST /remove-background`, `GET /healthz`) on a Linux server. It assumes a CPU
deployment (the default); GPU notes are called out where relevant.

The service entry point is `app.py` (`uvicorn app:app`). See [README.md](README.md)
for the API contract and algorithm details.

Sections 1–12 below are the **manual CPU deploy** (systemd + nginx, the default).
If you are deploying to an **AutoDL GPU box**, the one-click script in section 0 is
faster; the two are independent deployment paths.

---

## 0. Fast path: AutoDL one-click deploy (GPU)

On an AutoDL GPU instance, use the repo's bundled one-click script instead of the
manual dependency install / systemd setup:

```bash
# Assumes the repo is already checked out on the instance (the script does NOT
# git pull); use the server-edition branch.
cd /root/BGfilter-server            # your actual path
scripts/deploy_autodl.sh --token <TOKEN>
```

The script **idempotently** installs deps + fetches model weights + installs the
CuPy wheel matching the CUDA version (GPU foreground compositing) + starts a
**supervisord** instance (crash auto-restart) + adds an interactive-login
autostart to `~/.bashrc`, then launches the service and verifies `/healthz`.

Arguments:

| Argument | Default | Meaning |
| --- | --- | --- |
| `--token <TOKEN>` | none | Auth token; sets/updates the `X-Genarrative-Image-Token` requirement. **Omitting it reuses an existing token** (re-running to tweak another knob won't drop auth); only a first run with no token starts OPEN. |
| `--open` | — | Explicitly remove the token and run OPEN (no auth). |
| `--port <PORT>` | `6006` | In-container listen port (matches the AutoDL public port mapping). |
| `--config <cfg>` | `configs/gpu.yaml` | Pipeline config. |
| `--log-dir <dir>` | `~/autodl-tmp` | Log directory (AutoDL persistent data disk by default — survives container resets, unlike `/tmp`). Holds `bgfilter.log` (prose event log) and `bgfilter-access.jsonl` (structured per-request log, app-rotated). |

**Assumes**: repo already checked out (no `git pull`), root user, torch/CUDA
provided by the AutoDL image, python at `/root/miniconda3`.

**Two AutoDL-inherent gotchas** (the script can't fix these):

- The **public URL mapping** (e.g. `6006 → https://...:8443`) is set in the
  **AutoDL web console**, not by this script.
- **After a container restart** the service does not come back on its own — AutoDL
  has no systemd / boot hook, so the autostart lives in `~/.bashrc` and only fires
  when you open an **interactive shell (SSH / JupyterLab terminal)**.

**Managing the service** (supervisorctl lives next to python; socket
`~/bgfilter-supervisor.sock`, supervisord config `~/supervisord.conf`):

```bash
S="/root/miniconda3/bin/supervisorctl -c $HOME/supervisord.conf"
$S status bgfilter      # state + uptime
$S restart bgfilter     # restart (e.g. after editing configs/*.yaml)
$S stop bgfilter        # stop the service (supervisord keeps running)
$S start bgfilter       # start it again
$S shutdown             # stop everything, incl. supervisord itself

tail -f ~/autodl-tmp/bgfilter.log            # event log: startup/warnings/tracebacks (prose)
tail -f ~/autodl-tmp/bgfilter-access.jsonl   # access log: one JSON record per request
tail -f ~/supervisord.log                    # supervisord's own log (service won't start?)
```

If supervisord itself is not running — fresh container restart with no shell
opened yet, or a stale-socket error — start it the same way the `~/.bashrc`
autostart does:

```bash
rm -f ~/bgfilter-supervisor.sock ~/supervisord.pid
/root/miniconda3/bin/supervisord -c ~/supervisord.conf
```

Changes to `~/supervisord.conf` itself (port, log path, …) are applied by
re-running the deploy script — it regenerates the file and does
`reread + update + restart`.

> After editing `configs/*.yaml` (e.g. enabling `cross_check.reuse_as_seg`), run
> the `restart bgfilter` above so the service reloads config — config is read only
> at startup; there is no per-request switch for it.

---

## 1. Prerequisites

- Linux x86-64 (tested on Ubuntu; any modern distro works).
- **Python 3.11–3.13** recommended. Newer (3.14) can work, but some ML wheels may
  not be published for it yet — stick to a version with prebuilt wheels for
  `torch`, `onnxruntime`, `opencv`, `pymatting`.
- `git`, and the venv module (`python3-venv` on Debian/Ubuntu).
- ~3 GB free disk for the Python env + ~1 GB for model weights.
- Outbound network **once** to fetch dependencies and model weights (the running
  service can then run fully offline).

```bash
sudo apt update
sudo apt install -y python3-venv python3-pip git
```

## 2. Get the code

```bash
sudo mkdir -p /opt/genarrative-image-host/bgfilter-service
sudo chown "$USER" /opt/genarrative-image-host/bgfilter-service
cd /opt/genarrative-image-host/bgfilter-service
git clone <repo-url> .            # or copy the repo here; use the server-edition branch
```

## 3. Python environment

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip
```

## 4. Install dependencies (CPU)

Install **CPU-only PyTorch first** from the CPU wheel index — otherwise `pip` pulls
the default CUDA build and ~2–3 GB of `nvidia-*` packages you don't need on a CPU box:

```bash
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
```

`requirements.txt` already pins **`opencv-python-headless`** (the GUI build needs
`libGL.so.1` etc. that headless servers lack), so no change is needed. Then install
the rest — `torch`/`torchvision` are already satisfied, so this won't re-pull CUDA:

```bash
pip install -r requirements.txt
```

Verify the stack imports and torch is the CPU build (`cuda= None`):

```bash
python -c "import torch,cv2,fastapi,transformers,pymatting,onnxruntime,kornia,timm; \
print('torch', torch.__version__, 'cuda=', torch.version.cuda)"
```

> Behind the Great Firewall, add `-i https://pypi.tuna.tsinghua.edu.cn/simple` to the
> second `pip install` for speed.

## 5. Provide the model weights

Three models are used by default: `hustvl/vitmatte-base-composition-1k` (matting),
`ZhengPeng7/BiRefNet` (default segmenter), `skytnt/anime-seg` (optional segmenter).
A fourth model, `ZhengPeng7/BiRefNet_HR-matting` (~425 MB), is only needed if you
enable the **cross-check** veto — see the note at the end of this section.

**Quick path (recommended):** run the bundled script — no arguments — to download
every model into `models/` via the hf-mirror (proxy bypassed, Xet disabled,
already-present folders skipped). Afterwards the app has full functionality offline:

```bash
python scripts/fetch_weights.py
```

The two manual layouts below are for finer control (e.g. a shared HF cache, or
picking specific models).

### Option A — HuggingFace cache + offline mode (recommended)

Download once into a cache dir, then run offline:

```bash
export HF_HOME=/opt/genarrative-image-host/bgfilter-service/model-cache/hf
hf download hustvl/vitmatte-base-composition-1k
hf download ZhengPeng7/BiRefNet
hf download skytnt/anime-seg
hf download ZhengPeng7/BiRefNet_HR-matting   # only if using cross-check (see below)
```

At runtime set `HF_HOME` to the same path plus `HF_HUB_OFFLINE=1
TRANSFORMERS_OFFLINE=1` (see §6). You can also download on another machine and copy
the `hub/` folder over.

### Option B — plain folders inside the project

Drop each model into `models/<repo-basename>/` and the loader uses it directly (no
HF cache, no network). Absent folders fall back to the repo id, so this is opt-in.

**The folder name must match the repo basename exactly** — that is how the loader
finds it (`ZhengPeng7/BiRefNet` → `models/BiRefNet`). Required files per model:

```text
<project-root>/
└── models/
    ├── vitmatte-base-composition-1k/     # ViTMatte matting model
    │   ├── config.json
    │   ├── preprocessor_config.json
    │   └── pytorch_model.bin             # ~369 MB
    ├── BiRefNet/                         # default segmenter (trust_remote_code)
    │   ├── config.json
    │   ├── BiRefNet_config.py            # custom code — required by trust_remote_code
    │   ├── birefnet.py                   # custom code — required by trust_remote_code
    │   └── model.safetensors             # ~424 MB
    ├── anime-seg/                        # optional segmenter (seg_model=anime-seg)
    │   └── isnetis.onnx                  # ~168 MB
    └── BiRefNet_HR-matting/              # optional — only for the cross-check veto
        ├── config.json
        ├── BiRefNet_config.py            # custom code — required by trust_remote_code
        ├── birefnet.py                   # custom code — required by trust_remote_code
        └── model.safetensors             # ~425 MB
```

Notes:
- **ViTMatte**: needs `config.json`, `preprocessor_config.json`, and the weights
  (`pytorch_model.bin`; `model.safetensors` also works if present).
- **BiRefNet**: the two `.py` files are mandatory — `trust_remote_code` executes them
  to build the model. Keep `config.json` + `model.safetensors` alongside them. (The
  repo's `handler.py` / `requirements.txt` / `README.md` are not needed.)
- **anime-seg**: only `isnetis.onnx` is read; nothing else is required.
- `anime-seg` and `BiRefNet_HR-matting` are optional — omit their folders unless you
  use `seg_model=anime-seg` / the cross-check veto respectively.
- The **`cpu-fast` lane** (§6) loads only **ViTMatte + BiRefNet**; on an 8 GB box
  using that lane you can skip `anime-seg` and `BiRefNet_HR-matting` entirely.
  (`scripts/fetch_weights.py` still fetches all four for full functionality.)

The easiest way to populate the folders is to let `hf download` fetch them (it also
pulls a couple of unused files like `README.md`, which is harmless):

```bash
hf download hustvl/vitmatte-base-composition-1k --local-dir models/vitmatte-base-composition-1k
hf download ZhengPeng7/BiRefNet                 --local-dir models/BiRefNet
hf download skytnt/anime-seg                     --local-dir models/anime-seg
hf download ZhengPeng7/BiRefNet_HR-matting       --local-dir models/BiRefNet_HR-matting  # cross-check only
```

Override the base directory with `BGFILTER_WEIGHTS_DIR`. `models/` is gitignored.

### Cross-check veto (on by default)

The pipeline runs a second matting model (`ZhengPeng7/BiRefNet_HR-matting`,
~425 MB) as an independent "second opinion" — what it does and why is in
[README.md](README.md) → *Pipelines*. Deploy-side facts:

- It is **ON by default** (CLI and HTTP service), so **provision this model**
  (Option A or B above). ⚠️ On an offline server (`HF_HUB_OFFLINE=1`) it must be
  present before the first request — otherwise every default request fails.
- Turn it off service-wide with `BGFILTER_CROSS_CHECK=0`, or per request with the
  `cross_check` form field; the service preloads it at startup when enabled.
- `cross_check.reuse_as_seg: true` in the config (default **off**) makes that
  forward double as the segmentation mask: one less model loaded, ~20 s/image
  faster on CPU, ~1 GB less VRAM. Config-only — restart to change it.

### Background mode (flat vs complex)

`background_mode=complex` (non-flat / scene backgrounds) needs **no extra
weights** — it reuses the already-provisioned BiRefNet segmenter + ViTMatte and
bypasses the colour-keyed stages. The cross-check veto **defaults off** in
complex mode (pass `cross_check=on` to enable it). Field semantics and response
headers: [README.md](README.md) → *HTTP service*.

### Downloading behind a firewall (CN networks)

If direct HuggingFace access is blocked, use the mirror **with the local proxy
bypassed and Xet disabled** — the reliable combination:

```bash
export HF_ENDPOINT=https://hf-mirror.com   # domestic mirror
export NO_PROXY='*'                        # bypass the proxy; the mirror is direct
export HF_HUB_DISABLE_XET=1                 # repos are Xet-backed; force classic HTTP
```

This avoids two failure modes: (1) mirror + an overseas proxy makes the mirror
redirect back to `huggingface.co`, which recent `huggingface_hub` rejects with
`FileMetadataError`; (2) with `hf-xet` installed the Xet path fails instantly.

## 6. Configuration (environment variables)

| Variable | Default | Purpose |
| --- | --- | --- |
| `BGFILTER_CONFIG` | `configs/default.yaml` | Pipeline config file |
| `BGFILTER_DEVICE` | `cpu` | Overrides **both** model and segmentation device (`cuda` for GPU) |
| `BGFILTER_MAX_IMAGE_PIXELS` | `4194304` | Reject larger inputs with `413` (~4 MP) |
| `BGFILTER_FETCH_MAX_BYTES` | `26214400` | `image_url` mode: cap the downloaded body (~25 MB) → `413`. No SSRF filtering — keep auth on / callers trusted. |
| `BGFILTER_FETCH_TIMEOUT` | `15` | `image_url` mode: read timeout (s) for the fetch (connect fixed at 5 s). Outbound fetch concurrency is fixed at `2 × cpu_workers` (hard cap 64). |
| `BGFILTER_PRELOAD` | `1` | Load default models at startup (first request isn't cold) |
| `BGFILTER_ACCESS_LOG` | — | Path of the **structured access log** (JSONL, one record per request + one `startup` meta record; app-rotated at 50 MB × 5). Fields: `request_id`, params, sizes, stage timings (`t_queue/fetch/decode/process/total_ms`), status, truncated `exc` on failures; the id is also echoed as the `X-Request-ID` response header. Unset = records fall back to stderr (dev). The prose **event log** (startup/warnings/tracebacks) always goes to stderr for supervisord/journald to capture — per-request data never lands there. |
| `BGFILTER_LOG_LEVEL` | `INFO` | Root log level (event stream). |
| `BGFILTER_CROSS_CHECK` | config (`on`) | Force the cross-check veto on/off (`0` disables; needs `BiRefNet_HR-matting`) |
| `BGFILTER_CPU_WORKERS` | config `server.cpu_workers` (`1`) | Thread-pool size for request pipelining. `1` = strictly serial (one request at a time). `>1` lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). Set it in the config (`server:` section) or override per box with this env var. |
| `BGFILTER_GPU_CONCURRENCY` | config `server.gpu_concurrency` (`1`) | How many requests may be inside a GPU forward at once. `1` = single-stream (safe). `2` lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as `weights + N × peak_forward ≤ VRAM` (on a 32 GB card, `2` peaks ~19 GB at 2048 cross-check; keep `1` on ≤16 GB). Needs `cpu_workers>1` to matter. Config key `server.gpu_concurrency`; this env var overrides it. |
| `BGFILTER_AUTH_TOKEN` | — | If set, `/remove-background` requires header `X-Genarrative-Image-Token` to equal it (constant-time; `401` otherwise). Unset = **open** (auth left to a fronting proxy). `/healthz` stays open. See §10. |
| `BGFILTER_WEIGHTS_DIR` | `models` | Where local weight folders are looked up (Option B) |
| `HF_HOME` | — | HF cache location (Option A) |
| `HF_HUB_OFFLINE` / `TRANSFORMERS_OFFLINE` | — | Set to `1` in production once weights are cached |
| `OMP_NUM_THREADS` / `MKL_NUM_THREADS` | — | Cap CPU threads (e.g. `4`) |

### Config file: `default.yaml` vs the `cpu-fast` lane

`BGFILTER_CONFIG` selects the pipeline config. Two are shipped:

- **`configs/default.yaml`** — full quality. Cross-check veto **on** (loads the extra
  `BiRefNet_HR-matting` at 2048 as a second opinion), best edges. Heavier: on large
  inputs the peak working set runs well into double-digit GB, so it wants a roomy box
  (or a GPU).
- **`configs/cpu-fast.yaml`** — the lean CPU lane, tuned for an **8 GB box**. fp32,
  BiRefNet segmentation only, **no cross-check, no anime-seg**, chunked attention
  (`attn_query_chunk: 512`). Loads only **two** weights (base ViTMatte + BiRefNet).
  Measured peak working set: **~5.6 GB @ 2048×1152 (2.4 MP)**, ~7.1 GB @ 2048×2048
  (4.2 MP); ~14–15 s per 1.5 MP image.

> **On a ≤8 GB server, set `BGFILTER_CONFIG=configs/cpu-fast.yaml`.** The default lane
> plus a large input is exactly what drives ViTMatte's attention transient into an
> OOM — the worker is killed (SIGKILL) and a fronting nginx returns **502** while the
> app log shows `DefaultCPUAllocator: can't allocate memory`.

Two model knobs live in the **config file** (`model:` section), not env vars:

| Config key | Default | Purpose |
| --- | --- | --- |
| `model.precision` | `fp32` | `fp32` or `bf16`. bf16 is used **only** on hardware with native kernels (CUDA, or a CPU with AVX512-BF16/AMX); otherwise it warns and falls back to fp32. Also settable per model under `segmentation:` / `cross_check:`. |
| `model.attn_query_chunk` | `2048` | Query-row chunk for ViTMatte's global attention. Smaller (e.g. `512`) caps the O(N²) memory transient for a few % more compute; output is bit-for-bit identical. `0` disables chunking (stock one-shot attention). |

> **Despill is off by default** (all lanes, cpu-fast included). It pulls foreground
> chroma along the background-hue axis with no positional/semantic guard, so a subject
> sharing the background's hue (e.g. a blue suit on a blue backdrop) gets desaturated /
> hue-shifted (~ΔE 35). Re-enable per-config with `despill: {enabled: true}` only when
> edge-spill removal matters more than same-hue fidelity.

## 7. Run the service

```bash
BGFILTER_ACCESS_LOG=logs/bgfilter-access.jsonl \
python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1 --no-access-log
```

- **Keep `--workers 1`.** Each worker loads its own full copy of the models; more
  workers multiply memory (an 8 GB box will OOM). The service already serializes
  inference with a global lock.
- `--host 127.0.0.1` exposes it to the local host only (intended for a co-located
  caller). See §10 for remote access.
- `--no-access-log` silences uvicorn's prose per-request lines — they duplicate (a
  subset of) the app's structured access records (§6, `BGFILTER_ACCESS_LOG`), and
  per-request data does not belong in the prose stream.

## 8. Run as a systemd service

`/etc/systemd/system/genarrative-bgfilter.service`:

```ini
[Unit]
Description=Genarrative BgFilter background removal service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/opt/genarrative-image-host/bgfilter-service
Environment=HF_HOME=/opt/genarrative-image-host/bgfilter-service/model-cache/hf
Environment=HF_HUB_OFFLINE=1
Environment=TRANSFORMERS_OFFLINE=1
Environment=HF_HUB_DISABLE_XET=1
Environment=BGFILTER_CONFIG=configs/default.yaml
Environment=BGFILTER_DEVICE=cpu
Environment=BGFILTER_MAX_IMAGE_PIXELS=4194304
Environment=BGFILTER_PRELOAD=1
Environment=BGFILTER_ACCESS_LOG=/opt/genarrative-image-host/bgfilter-service/logs/bgfilter-access.jsonl
Environment=OMP_NUM_THREADS=4
Environment=MKL_NUM_THREADS=4
ExecStart=/opt/genarrative-image-host/bgfilter-service/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1 --no-access-log
Restart=on-failure
RestartSec=5
MemoryMax=8G
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/opt/genarrative-image-host

[Install]
WantedBy=multi-user.target
```

On a ≤8 GB box, change the config line to the lean lane:
`Environment=BGFILTER_CONFIG=configs/cpu-fast.yaml` (see §6) — the default lane can
OOM on large inputs.

If you use Option B (local `models/` folders) instead of the HF cache, drop the
`HF_HOME`/offline lines — those weights need no cache or network.

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now genarrative-bgfilter
sudo systemctl status genarrative-bgfilter
journalctl -u genarrative-bgfilter -f          # wait for "Application startup complete"
```

## 9. Verify

```bash
curl -s http://127.0.0.1:18083/healthz
# {"ok":true,"service":"bgfilter","version":"0.1.0","defaultSegModel":"birefnet","device":"cpu"}

curl -sS -F "file=@input.png" -F "screen_color=#CFEFFF" \
  http://127.0.0.1:18083/remove-background -o out.png

# non-flat / scene background (no colour key; cross-check defaults off here)
curl -sS -F "file=@scene.jpg" -F "background_mode=complex" \
  http://127.0.0.1:18083/remove-background -o out.png
```

## 10. Exposing the service

The unit binds `127.0.0.1` — intended to be called by a co-located client (e.g. the
Genarrative Rust BFF via `GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_BASE_URL`). To reach
it from other hosts:

- **Reverse proxy (recommended):** put nginx/caddy in front, terminate TLS, add
  rate-limiting, and forward to `127.0.0.1:18083`.
- **Direct:** change `--host` to `0.0.0.0` and restrict source IPs with a firewall.
  Do not expose it to the public internet without authentication (below).

### Authentication

App-level token auth is **opt-in** via `BGFILTER_AUTH_TOKEN`:

- **Set it** → `/remove-background` requires header `X-Genarrative-Image-Token: <token>`
  (constant-time compare); missing or wrong returns `401`. `/healthz` stays open. Use
  this whenever the app is reachable directly — `--host 0.0.0.0`, or a cloud port
  mapping with no proxy in front.
- **Leave it unset** → the app is **open** (fail-open) and logs a startup WARNING.
  Correct only when a fronting proxy already authenticates — e.g. the production nginx
  checks `X-Genarrative-Image-Token` itself, so the app behind it needs no token.

The header name matches the nginx layer, so a caller only swaps the base URL + token
value — no request-shape change:

```bash
curl -H "X-Genarrative-Image-Token: <token>" \
  -F "file=@input.png" \
  https://<host>/remove-background -o out.png
```

## 11. Capacity & performance

- CPU throughput: the **`cpu-fast` lane** does ~**14–15 s per 1.5 MP image** (base
  ViTMatte + one BiRefNet@1024, chunked attention). The **default lane** is slower —
  the cross-check adds a second HR-matting@2048 forward. Requests are serialized by
  the global lock, so plan for a handful of images/min per instance.
- **Default lane, workstation CPU reference (Ryzen 9700X, native bf16, 32 GB;
  cross-check on with `reuse_as_seg`, chunked attention):**

  | input | precision | warm / image | peak memory |
  |---|---|---|---|
  | 1024×1536 | fp32 | ~45 s | — |
  | 1024×1536 | bf16 | ~33 s | — |
  | 2048×2048 | bf16 | ~53 s | 11.4 GB (batch), 8.1 GB (single image) |

  Without `reuse_as_seg` add the primary BiRefNet@1024 forward (~+20 s/image).
  The memory ceiling is the cross-check HR forward (fixed `input_size` 2048
  regardless of image size). Before chunked attention the same 2048² bf16 run
  peaked at 25.2 GB; fully un-optimized fp32 would need an estimated 55–60 GB.
  CPUs without native bf16 (e.g. Zen 2) auto-fall back to fp32 — prefer
  `--no-cross-check` there if memory or latency is tight.
- For throughput: use a GPU (`BGFILTER_DEVICE=cuda`, install a CUDA torch build) —
  roughly an order of magnitude faster — and/or run multiple instances behind a load
  balancer (each with `--workers 1`).
- **GPU deployment (measured, RTX 5070 Ti 16 GB, full pipeline — cross-check on,
  bf16):** ~1–2 s/image (2 s at 4 MP), peak VRAM ~10.9 GB allocated (10.1 GB with
  `reuse_as_seg: true`). The number that must fit on the card is the allocator's
  *reserved* footprint. `bgfilter` defaults
  `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` on Linux/WSL2 (set before
  torch loads; native Windows does not support it, and an explicit env value
  always wins). Measured effect: reserved 15.3 GB → **12.0 GB** with reuse
  (13.3 GB without, the default) and ~10 % *faster* — the full pipeline fits a
  16 GB card with ~2–3.5 GB headroom even next to a desktop session. Without
  expandable segments (e.g. native Windows) reserved is ~15.3–16.2 GB:
  borderline-to-overflow on a 16 GB card. Precision `bf16` is what makes 16 GB
  viable at all — fp32 reserved ~17 GB overflows.
- Memory: chunked attention (`attn_query_chunk`) caps ViTMatte's O(N²) transient, so
  the peak is set by dense compute, not the attention map. On the `cpu-fast` lane the
  measured peak is **~5.6 GB @ 2.4 MP / ~7.1 GB @ 4.2 MP** — `MemoryMax=8G` with the
  default 4 MP input cap is a safe ceiling for a single CPU worker. The default lane
  (cross-check on) needs more headroom.
- **Keeping RSS flat (glibc).** After each image the service calls glibc
  `malloc_trim(0)` (`bgfilter/memtune.py`) to hand a large BiRefNet@2048 forward's
  freed activations back to the OS — without it, ptmalloc keeps them in the arena
  and RSS ratchets up across requests. For lower memory *and* better CPU
  throughput, preload **jemalloc** (PyTorch's recommendation for CPU inference)
  and let it decay dirty pages aggressively:
  ```ini
  # in the [Service] section of the systemd unit
  Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
  Environment=MALLOC_CONF=background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0
  ```
  (`apt install libjemalloc2`; tcmalloc via `libtcmalloc_minimal.so.4` works too.)
  jemalloc/tcmalloc manage their own release, so `malloc_trim` simply no-ops under
  them. If you cannot preload an allocator, `MALLOC_ARENA_MAX=2` and
  `MALLOC_TRIM_THRESHOLD_=0` in the unit environment curb ptmalloc retention.

## 12. Troubleshooting

| Symptom | Cause / fix |
| --- | --- |
| `pip` pulls `nvidia-*`/`cuda-*` (huge) | You skipped the CPU index in §4. Install torch from `--index-url https://download.pytorch.org/whl/cpu`. |
| `ImportError: libGL.so.1` on `import cv2` | Using `opencv-python` on a headless box. Switch to `opencv-python-headless` (§4). |
| `ensurepip is not available` on venv create | Install `python3-venv` (§1). |
| `FileMetadataError: ... not on huggingface.co` while downloading | hf-mirror + a proxy conflict. Bypass the proxy (`NO_PROXY='*'`) and set `HF_HUB_DISABLE_XET=1` (§5). |
| `LocalEntryNotFoundError`, 0 bytes | Xet download path failing; set `HF_HUB_DISABLE_XET=1`. |
| `503 service not ready` | Models still loading at startup; wait for "Application startup complete". |
| `500 inference failed` right after enabling `BGFILTER_CROSS_CHECK` (offline) | `BiRefNet_HR-matting` not provisioned; download it into the cache/`models/` (§5). |
| OOM / killed under load | More than one worker, or `MemoryMax` too low. Keep `--workers 1`. |
| Worker `killed` (SIGKILL) / `502` on a large image; log shows `DefaultCPUAllocator: can't allocate memory` | ViTMatte OOM on a big input. Use `BGFILTER_CONFIG=configs/cpu-fast.yaml` (§6), and/or lower `BGFILTER_MAX_IMAGE_PIXELS`. |
| `413`/large upload rejected at the proxy | nginx `client_max_body_size` (default 1 MB) — raise it in the server block (e.g. `client_max_body_size 12m;`); this is separate from the app's `BGFILTER_MAX_IMAGE_PIXELS`. |
| `400 could not auto-detect background colour` | Input has no clean flat border; pass `screen_color=#RRGGBB`. |