# Deploying BgFilter to a Linux server

> 中文版：[DEPLOY_ZH.md](DEPLOY_ZH.md)

This guide covers everything needed to run the BgFilter HTTP service
(`POST /remove-background`, `GET /healthz`) on a Linux server. It assumes a CPU
deployment (the default); GPU notes are called out where relevant.

The service entry point is `app.py` (`uvicorn app:app`). See [README.md](README.md)
for the API contract and algorithm details.

---

## 1. Prerequisites

- Linux x86-64 (tested on Ubuntu; any modern distro works).
- **Python 3.11–3.13** recommended. Newer (3.14) can work, but some ML wheels may
  not be published for it yet — stick to a version with prebuilt wheels for
  `torch`, `onnxruntime`, `opencv`, `pymatting`.
- `git`, and the venv module (`python3-venv` on Debian/Ubuntu).
- ~3 GB free disk for the Python env + ~1 GB for model weights.
- Outbound network **once** to fetch dependencies and model weights (the running
  service can then run fully offline).

```bash
sudo apt update
sudo apt install -y python3-venv python3-pip git
```

## 2. Get the code

```bash
sudo mkdir -p /opt/genarrative-image-host/bgfilter-service
sudo chown "$USER" /opt/genarrative-image-host/bgfilter-service
cd /opt/genarrative-image-host/bgfilter-service
git clone <repo-url> .            # or copy the repo here; use the server-edition branch
```

## 3. Python environment

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip
```

## 4. Install dependencies (CPU)

Install **CPU-only PyTorch first** from the CPU wheel index — otherwise `pip` pulls
the default CUDA build and ~2–3 GB of `nvidia-*` packages you don't need on a CPU box:

```bash
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
```

`requirements.txt` already pins **`opencv-python-headless`** (the GUI build needs
`libGL.so.1` etc. that headless servers lack), so no change is needed. Then install
the rest — `torch`/`torchvision` are already satisfied, so this won't re-pull CUDA:

```bash
pip install -r requirements.txt
```

Verify the stack imports and torch is the CPU build (`cuda= None`):

```bash
python -c "import torch,cv2,fastapi,transformers,pymatting,onnxruntime,kornia,timm; \
print('torch', torch.__version__, 'cuda=', torch.version.cuda)"
```

> Behind the Great Firewall, add `-i https://pypi.tuna.tsinghua.edu.cn/simple` to the
> second `pip install` for speed.

## 5. Provide the model weights

Three models are used by default: `hustvl/vitmatte-base-composition-1k` (matting),
`ZhengPeng7/BiRefNet` (default segmenter), `skytnt/anime-seg` (optional segmenter).
A fourth model, `ZhengPeng7/BiRefNet_HR-matting` (~425 MB), is only needed if you
enable the **cross-check** veto — see the note at the end of this section.

**Quick path (recommended):** run the bundled script — no arguments — to download
every model into `models/` via the hf-mirror (proxy bypassed, Xet disabled,
already-present folders skipped). Afterwards the app has full functionality offline:

```bash
python scripts/fetch_weights.py
```

The two manual layouts below are for finer control (e.g. a shared HF cache, or
picking specific models).

### Option A — HuggingFace cache + offline mode (recommended)

Download once into a cache dir, then run offline:

```bash
export HF_HOME=/opt/genarrative-image-host/bgfilter-service/model-cache/hf
hf download hustvl/vitmatte-base-composition-1k
hf download ZhengPeng7/BiRefNet
hf download skytnt/anime-seg
hf download ZhengPeng7/BiRefNet_HR-matting   # only if using cross-check (see below)
```

At runtime set `HF_HOME` to the same path plus `HF_HUB_OFFLINE=1
TRANSFORMERS_OFFLINE=1` (see §6). You can also download on another machine and copy
the `hub/` folder over.

### Option B — plain folders inside the project

Drop each model into `models/<repo-basename>/` and the loader uses it directly (no
HF cache, no network). Absent folders fall back to the repo id, so this is opt-in.

**The folder name must match the repo basename exactly** — that is how the loader
finds it (`ZhengPeng7/BiRefNet` → `models/BiRefNet`). Required files per model:

```text
<project-root>/
└── models/
    ├── vitmatte-base-composition-1k/     # ViTMatte matting model
    │   ├── config.json
    │   ├── preprocessor_config.json
    │   └── pytorch_model.bin             # ~369 MB
    ├── BiRefNet/                         # default segmenter (trust_remote_code)
    │   ├── config.json
    │   ├── BiRefNet_config.py            # custom code — required by trust_remote_code
    │   ├── birefnet.py                   # custom code — required by trust_remote_code
    │   └── model.safetensors             # ~424 MB
    ├── anime-seg/                        # optional segmenter (seg_model=anime-seg)
    │   └── isnetis.onnx                  # ~168 MB
    └── BiRefNet_HR-matting/              # optional — only for the cross-check veto
        ├── config.json
        ├── BiRefNet_config.py            # custom code — required by trust_remote_code
        ├── birefnet.py                   # custom code — required by trust_remote_code
        └── model.safetensors             # ~425 MB
```

Notes:
- **ViTMatte**: needs `config.json`, `preprocessor_config.json`, and the weights
  (`pytorch_model.bin`; `model.safetensors` also works if present).
- **BiRefNet**: the two `.py` files are mandatory — `trust_remote_code` executes them
  to build the model. Keep `config.json` + `model.safetensors` alongside them. (The
  repo's `handler.py` / `requirements.txt` / `README.md` are not needed.)
- **anime-seg**: only `isnetis.onnx` is read; nothing else is required.
- `anime-seg` and `BiRefNet_HR-matting` are optional — omit their folders unless you
  use `seg_model=anime-seg` / the cross-check veto respectively.
- The **`cpu-fast` lane** (§6) loads only **ViTMatte + BiRefNet**; on an 8 GB box
  using that lane you can skip `anime-seg` and `BiRefNet_HR-matting` entirely.
  (`scripts/fetch_weights.py` still fetches all four for full functionality.)

The easiest way to populate the folders is to let `hf download` fetch them (it also
pulls a couple of unused files like `README.md`, which is harmless):

```bash
hf download hustvl/vitmatte-base-composition-1k --local-dir models/vitmatte-base-composition-1k
hf download ZhengPeng7/BiRefNet                 --local-dir models/BiRefNet
hf download skytnt/anime-seg                     --local-dir models/anime-seg
hf download ZhengPeng7/BiRefNet_HR-matting       --local-dir models/BiRefNet_HR-matting  # cross-check only
```

Override the base directory with `BGFILTER_WEIGHTS_DIR`. `models/` is gitignored.

### Cross-check veto (on by default)

The pipeline runs a second, high-resolution matting model
(`ZhengPeng7/BiRefNet_HR-matting`, ~425 MB) as a "second opinion" that clears
background residue trapped between hair strands. It is **ON by default** (both CLI
and HTTP service) and, with reuse-as-seg, that same forward doubles as the
segmentation mask on the birefnet backend — so it replaces the primary seg model
rather than adding to it. **You must provision this model** (Option A or B above):

- ⚠️ **On an offline server (`HF_HUB_OFFLINE=1`) it must be present before the
  first request** — otherwise every default request fails, since it cannot
  download the model at runtime.
- Turn it off service-wide with `BGFILTER_CROSS_CHECK=0`, or override per request
  with the `cross_check` form field (`true`/`false`).
- The service preloads it at startup with the rest of the warm set; on the
  birefnet backend the redundant primary segmenter is skipped.

### Downloading behind a firewall (CN networks)

If direct HuggingFace access is blocked, use the mirror **with the local proxy
bypassed and Xet disabled** — the reliable combination:

```bash
export HF_ENDPOINT=https://hf-mirror.com   # domestic mirror
export NO_PROXY='*'                        # bypass the proxy; the mirror is direct
export HF_HUB_DISABLE_XET=1                 # repos are Xet-backed; force classic HTTP
```

This avoids two failure modes: (1) mirror + an overseas proxy makes the mirror
redirect back to `huggingface.co`, which recent `huggingface_hub` rejects with
`FileMetadataError`; (2) with `hf-xet` installed the Xet path fails instantly.

## 6. Configuration (environment variables)

| Variable | Default | Purpose |
| --- | --- | --- |
| `BGFILTER_CONFIG` | `configs/default.yaml` | Pipeline config file |
| `BGFILTER_DEVICE` | `cpu` | Overrides **both** model and segmentation device (`cuda` for GPU) |
| `BGFILTER_MAX_IMAGE_PIXELS` | `4194304` | Reject larger inputs with `413` (~4 MP) |
| `BGFILTER_PRELOAD` | `1` | Load default models at startup (first request isn't cold) |
| `BGFILTER_CROSS_CHECK` | config (`on`) | Force the cross-check veto on/off (`0` disables; needs `BiRefNet_HR-matting`) |
| `BGFILTER_WEIGHTS_DIR` | `models` | Where local weight folders are looked up (Option B) |
| `HF_HOME` | — | HF cache location (Option A) |
| `HF_HUB_OFFLINE` / `TRANSFORMERS_OFFLINE` | — | Set to `1` in production once weights are cached |
| `OMP_NUM_THREADS` / `MKL_NUM_THREADS` | — | Cap CPU threads (e.g. `4`) |

### Config file: `default.yaml` vs the `cpu-fast` lane

`BGFILTER_CONFIG` selects the pipeline config. Two are shipped:

- **`configs/default.yaml`** — full quality. Cross-check veto **on** (loads the extra
  `BiRefNet_HR-matting` at 2048 as a second opinion), best edges. Heavier: on large
  inputs the peak working set runs well into double-digit GB, so it wants a roomy box
  (or a GPU).
- **`configs/cpu-fast.yaml`** — the lean CPU lane, tuned for an **8 GB box**. fp32,
  BiRefNet segmentation only, **no cross-check, no anime-seg**, chunked attention
  (`attn_query_chunk: 512`). Loads only **two** weights (base ViTMatte + BiRefNet).
  Measured peak working set: **~5.6 GB @ 2048×1152 (2.4 MP)**, ~7.1 GB @ 2048×2048
  (4.2 MP); ~14–15 s per 1.5 MP image.

> **On a ≤8 GB server, set `BGFILTER_CONFIG=configs/cpu-fast.yaml`.** The default lane
> plus a large input is exactly what drives ViTMatte's attention transient into an
> OOM — the worker is killed (SIGKILL) and a fronting nginx returns **502** while the
> app log shows `DefaultCPUAllocator: can't allocate memory`.

Two model knobs live in the **config file** (`model:` section), not env vars:

| Config key | Default | Purpose |
| --- | --- | --- |
| `model.precision` | `fp32` | `fp32` or `bf16`. bf16 is used **only** on hardware with native kernels (CUDA, or a CPU with AVX512-BF16/AMX); otherwise it warns and falls back to fp32. Also settable per model under `segmentation:` / `cross_check:`. |
| `model.attn_query_chunk` | `2048` | Query-row chunk for ViTMatte's global attention. Smaller (e.g. `512`) caps the O(N²) memory transient for a few % more compute; output is bit-for-bit identical. `0` disables chunking (stock one-shot attention). |

## 7. Run the service

```bash
python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1
```

- **Keep `--workers 1`.** Each worker loads its own full copy of the models; more
  workers multiply memory (an 8 GB box will OOM). The service already serializes
  inference with a global lock.
- `--host 127.0.0.1` exposes it to the local host only (intended for a co-located
  caller). See §10 for remote access.

## 8. Run as a systemd service

`/etc/systemd/system/genarrative-bgfilter.service`:

```ini
[Unit]
Description=Genarrative BgFilter background removal service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/opt/genarrative-image-host/bgfilter-service
Environment=HF_HOME=/opt/genarrative-image-host/bgfilter-service/model-cache/hf
Environment=HF_HUB_OFFLINE=1
Environment=TRANSFORMERS_OFFLINE=1
Environment=HF_HUB_DISABLE_XET=1
Environment=BGFILTER_CONFIG=configs/default.yaml
Environment=BGFILTER_DEVICE=cpu
Environment=BGFILTER_MAX_IMAGE_PIXELS=4194304
Environment=BGFILTER_PRELOAD=1
Environment=OMP_NUM_THREADS=4
Environment=MKL_NUM_THREADS=4
ExecStart=/opt/genarrative-image-host/bgfilter-service/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1
Restart=on-failure
RestartSec=5
MemoryMax=8G
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/opt/genarrative-image-host

[Install]
WantedBy=multi-user.target
```

On a ≤8 GB box, change the config line to the lean lane:
`Environment=BGFILTER_CONFIG=configs/cpu-fast.yaml` (see §6) — the default lane can
OOM on large inputs.

If you use Option B (local `models/` folders) instead of the HF cache, drop the
`HF_HOME`/offline lines — those weights need no cache or network.

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now genarrative-bgfilter
sudo systemctl status genarrative-bgfilter
journalctl -u genarrative-bgfilter -f          # wait for "Application startup complete"
```

## 9. Verify

```bash
curl -s http://127.0.0.1:18083/healthz
# {"ok":true,"service":"bgfilter","version":"0.1.0","defaultSegModel":"birefnet","device":"cpu"}

curl -sS -F "file=@input.png" -F "screen_color=#CFEFFF" \
  http://127.0.0.1:18083/remove-background -o out.png
```

## 10. Exposing the service

The unit binds `127.0.0.1` — intended to be called by a co-located client (e.g. the
Genarrative Rust BFF via `GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_BASE_URL`). The
service has **no authentication**. To reach it from other hosts:

- **Reverse proxy (recommended):** put nginx/caddy in front, terminate TLS, add
  auth/rate-limiting, and forward to `127.0.0.1:18083`.
- **Direct:** change `--host` to `0.0.0.0` and restrict source IPs with a firewall.
  Do not expose it unauthenticated to the public internet.

## 11. Capacity & performance

- CPU throughput: the **`cpu-fast` lane** does ~**14–15 s per 1.5 MP image** (base
  ViTMatte + one BiRefNet@1024, chunked attention). The **default lane** is slower —
  the cross-check adds a second HR-matting@2048 forward. Requests are serialized by
  the global lock, so plan for a handful of images/min per instance.
- For throughput: use a GPU (`BGFILTER_DEVICE=cuda`, install a CUDA torch build) —
  roughly an order of magnitude faster — and/or run multiple instances behind a load
  balancer (each with `--workers 1`).
- Memory: chunked attention (`attn_query_chunk`) caps ViTMatte's O(N²) transient, so
  the peak is set by dense compute, not the attention map. On the `cpu-fast` lane the
  measured peak is **~5.6 GB @ 2.4 MP / ~7.1 GB @ 4.2 MP** — `MemoryMax=8G` with the
  default 4 MP input cap is a safe ceiling for a single CPU worker. The default lane
  (cross-check on) needs more headroom.
- **Keeping RSS flat (glibc).** After each image the service calls glibc
  `malloc_trim(0)` (`bgfilter/memtune.py`) to hand a large BiRefNet@2048 forward's
  freed activations back to the OS — without it, ptmalloc keeps them in the arena
  and RSS ratchets up across requests. For lower memory *and* better CPU
  throughput, preload **jemalloc** (PyTorch's recommendation for CPU inference)
  and let it decay dirty pages aggressively:
  ```ini
  # in the [Service] section of the systemd unit
  Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
  Environment=MALLOC_CONF=background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0
  ```
  (`apt install libjemalloc2`; tcmalloc via `libtcmalloc_minimal.so.4` works too.)
  jemalloc/tcmalloc manage their own release, so `malloc_trim` simply no-ops under
  them. If you cannot preload an allocator, `MALLOC_ARENA_MAX=2` and
  `MALLOC_TRIM_THRESHOLD_=0` in the unit environment curb ptmalloc retention.

## 12. Troubleshooting

| Symptom | Cause / fix |
| --- | --- |
| `pip` pulls `nvidia-*`/`cuda-*` (huge) | You skipped the CPU index in §4. Install torch from `--index-url https://download.pytorch.org/whl/cpu`. |
| `ImportError: libGL.so.1` on `import cv2` | Using `opencv-python` on a headless box. Switch to `opencv-python-headless` (§4). |
| `ensurepip is not available` on venv create | Install `python3-venv` (§1). |
| `FileMetadataError: ... not on huggingface.co` while downloading | hf-mirror + a proxy conflict. Bypass the proxy (`NO_PROXY='*'`) and set `HF_HUB_DISABLE_XET=1` (§5). |
| `LocalEntryNotFoundError`, 0 bytes | Xet download path failing; set `HF_HUB_DISABLE_XET=1`. |
| `503 service not ready` | Models still loading at startup; wait for "Application startup complete". |
| `500 inference failed` right after enabling `BGFILTER_CROSS_CHECK` (offline) | `BiRefNet_HR-matting` not provisioned; download it into the cache/`models/` (§5). |
| OOM / killed under load | More than one worker, or `MemoryMax` too low. Keep `--workers 1`. |
| Worker `killed` (SIGKILL) / `502` on a large image; log shows `DefaultCPUAllocator: can't allocate memory` | ViTMatte OOM on a big input. Use `BGFILTER_CONFIG=configs/cpu-fast.yaml` (§6), and/or lower `BGFILTER_MAX_IMAGE_PIXELS`. |
| `413`/large upload rejected at the proxy | nginx `client_max_body_size` (default 1 MB) — raise it in the server block (e.g. `client_max_body_size 12m;`); this is separate from the app's `BGFILTER_MAX_IMAGE_PIXELS`. |
| `400 could not auto-detect background colour` | Input has no clean flat border; pass `screen_color=#RRGGBB`. |