BGfilter/DEPLOY.md

Deploying BgFilter to a Linux server

中文版：DEPLOY_ZH.md

This guide covers everything needed to run the BgFilter HTTP service (POST /remove-background, GET /healthz) on a Linux server. It assumes a CPU deployment (the default); GPU notes are called out where relevant.

The service entry point is app.py (uvicorn app:app). See README.md for the API contract and algorithm details.

Sections 1–12 below are the manual CPU deploy (systemd + nginx, the default). If you are deploying to an AutoDL GPU box, the one-click script in section 0 is faster; the two are independent deployment paths.

---

0. Fast path: AutoDL one-click deploy (GPU)

On an AutoDL GPU instance, use the repo's bundled one-click script instead of the manual dependency install / systemd setup:

# Assumes the repo is already checked out on the instance (the script does NOT
# git pull); use the server-edition branch.
cd /root/BGfilter-server            # your actual path
scripts/deploy_autodl.sh --token <TOKEN>

The script idempotently installs deps + fetches model weights + installs the CuPy wheel matching the CUDA version (GPU foreground compositing) + starts a supervisord instance (crash auto-restart) + adds an interactive-login autostart to ~/.bashrc, then launches the service and verifies /healthz.

Arguments:

Argument	Default	Meaning
--token <TOKEN>	none	Auth token; sets/updates the X-Genarrative-Image-Token requirement. Omitting it reuses an existing token (re-running to tweak another knob won't drop auth); only a first run with no token starts OPEN.
--open	—	Explicitly remove the token and run OPEN (no auth).
--port <PORT>	6006	In-container listen port (matches the AutoDL public port mapping).
--config <cfg>	configs/gpu.yaml	Pipeline config.
--log-dir <dir>	~/autodl-tmp	Log directory (AutoDL persistent data disk by default — survives container resets, unlike /tmp). Holds bgfilter.log (prose event log) and bgfilter-access.jsonl (structured per-request log, app-rotated).

Assumes: repo already checked out (no git pull), root user, torch/CUDA provided by the AutoDL image, python at /root/miniconda3.

Two AutoDL-inherent gotchas (the script can't fix these):

• The public URL mapping (e.g. 6006 → https://...:8443) is set in the AutoDL web console, not by this script.
• After a container restart the service does not come back on its own — AutoDL has no systemd / boot hook, so the autostart lives in ~/.bashrc and only fires when you open an interactive shell (SSH / JupyterLab terminal).

Managing the service (supervisorctl lives next to python; socket ~/bgfilter-supervisor.sock, supervisord config ~/supervisord.conf):

S="/root/miniconda3/bin/supervisorctl -c $HOME/supervisord.conf"
$S status bgfilter      # state + uptime
$S restart bgfilter     # restart (e.g. after editing configs/*.yaml)
$S stop bgfilter        # stop the service (supervisord keeps running)
$S start bgfilter       # start it again
$S shutdown             # stop everything, incl. supervisord itself

tail -f ~/autodl-tmp/bgfilter.log            # event log: startup/warnings/tracebacks (prose)
tail -f ~/autodl-tmp/bgfilter-access.jsonl   # access log: one JSON record per request
tail -f ~/supervisord.log                    # supervisord's own log (service won't start?)

If supervisord itself is not running — fresh container restart with no shell opened yet, or a stale-socket error — start it the same way the ~/.bashrc autostart does:

rm -f ~/bgfilter-supervisor.sock ~/supervisord.pid
/root/miniconda3/bin/supervisord -c ~/supervisord.conf

Changes to ~/supervisord.conf itself (port, log path, …) are applied by re-running the deploy script — it regenerates the file and does reread + update + restart.

After editing configs/*.yaml (e.g. enabling cross_check.reuse_as_seg), run the restart bgfilter above so the service reloads config — config is read only at startup; there is no per-request switch for it.

---

1. Prerequisites

• Linux x86-64 (tested on Ubuntu; any modern distro works).
• Python 3.11–3.13 recommended. Newer (3.14) can work, but some ML wheels may not be published for it yet — stick to a version with prebuilt wheels for torch, onnxruntime, opencv, pymatting.
• git, and the venv module (python3-venv on Debian/Ubuntu).
• ~3 GB free disk for the Python env + ~1 GB for model weights.
• Outbound network once to fetch dependencies and model weights (the running service can then run fully offline).

sudo apt update
sudo apt install -y python3-venv python3-pip git

2. Get the code

sudo mkdir -p /opt/genarrative-image-host/bgfilter-service
sudo chown "$USER" /opt/genarrative-image-host/bgfilter-service
cd /opt/genarrative-image-host/bgfilter-service
git clone <repo-url> .            # or copy the repo here; use the server-edition branch

3. Python environment

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip

4. Install dependencies (CPU)

Install CPU-only PyTorch first from the CPU wheel index — otherwise pip pulls the default CUDA build and ~2–3 GB of nvidia-* packages you don't need on a CPU box:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

requirements.txt already pins opencv-python-headless (the GUI build needs libGL.so.1 etc. that headless servers lack), so no change is needed. Then install the rest — torch/torchvision are already satisfied, so this won't re-pull CUDA:

pip install -r requirements.txt

Verify the stack imports and torch is the CPU build (cuda= None):

python -c "import torch,cv2,fastapi,transformers,pymatting,onnxruntime,kornia,timm; \
print('torch', torch.__version__, 'cuda=', torch.version.cuda)"

Behind the Great Firewall, add -i https://pypi.tuna.tsinghua.edu.cn/simple to the second pip install for speed.

5. Provide the model weights

Three models are used by default: hustvl/vitmatte-base-composition-1k (matting), ZhengPeng7/BiRefNet (default segmenter), skytnt/anime-seg (optional segmenter). A fourth model, ZhengPeng7/BiRefNet_HR-matting (~425 MB), is only needed if you enable the cross-check veto — see the note at the end of this section.

Quick path (recommended): run the bundled script — no arguments — to download every model into models/ via the hf-mirror (proxy bypassed, Xet disabled, already-present folders skipped). Afterwards the app has full functionality offline:

python scripts/fetch_weights.py

The two manual layouts below are for finer control (e.g. a shared HF cache, or picking specific models).

Option A — HuggingFace cache + offline mode (recommended)

Download once into a cache dir, then run offline:

export HF_HOME=/opt/genarrative-image-host/bgfilter-service/model-cache/hf
hf download hustvl/vitmatte-base-composition-1k
hf download ZhengPeng7/BiRefNet
hf download skytnt/anime-seg
hf download ZhengPeng7/BiRefNet_HR-matting   # only if using cross-check (see below)

At runtime set HF_HOME to the same path plus HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 (see §6). You can also download on another machine and copy the hub/ folder over.

Option B — plain folders inside the project

Drop each model into models/<repo-basename>/ and the loader uses it directly (no HF cache, no network). Absent folders fall back to the repo id, so this is opt-in.

The folder name must match the repo basename exactly — that is how the loader finds it (ZhengPeng7/BiRefNet → models/BiRefNet). Required files per model:

<project-root>/
└── models/
    ├── vitmatte-base-composition-1k/     # ViTMatte matting model
    │   ├── config.json
    │   ├── preprocessor_config.json
    │   └── pytorch_model.bin             # ~369 MB
    ├── BiRefNet/                         # default segmenter (trust_remote_code)
    │   ├── config.json
    │   ├── BiRefNet_config.py            # custom code — required by trust_remote_code
    │   ├── birefnet.py                   # custom code — required by trust_remote_code
    │   └── model.safetensors             # ~424 MB
    ├── anime-seg/                        # optional segmenter (seg_model=anime-seg)
    │   └── isnetis.onnx                  # ~168 MB
    └── BiRefNet_HR-matting/              # optional — only for the cross-check veto
        ├── config.json
        ├── BiRefNet_config.py            # custom code — required by trust_remote_code
        ├── birefnet.py                   # custom code — required by trust_remote_code
        └── model.safetensors             # ~425 MB

Notes:

• ViTMatte: needs config.json, preprocessor_config.json, and the weights (pytorch_model.bin; model.safetensors also works if present).
• BiRefNet: the two .py files are mandatory — trust_remote_code executes them to build the model. Keep config.json + model.safetensors alongside them. (The repo's handler.py / requirements.txt / README.md are not needed.)
• anime-seg: only isnetis.onnx is read; nothing else is required.
• anime-seg and BiRefNet_HR-matting are optional — omit their folders unless you use seg_model=anime-seg / the cross-check veto respectively.
• The cpu-fast lane (§6) loads only ViTMatte + BiRefNet; on an 8 GB box using that lane you can skip anime-seg and BiRefNet_HR-matting entirely. (scripts/fetch_weights.py still fetches all four for full functionality.)

The easiest way to populate the folders is to let hf download fetch them (it also pulls a couple of unused files like README.md, which is harmless):

hf download hustvl/vitmatte-base-composition-1k --local-dir models/vitmatte-base-composition-1k
hf download ZhengPeng7/BiRefNet                 --local-dir models/BiRefNet
hf download skytnt/anime-seg                     --local-dir models/anime-seg
hf download ZhengPeng7/BiRefNet_HR-matting       --local-dir models/BiRefNet_HR-matting  # cross-check only

Override the base directory with BGFILTER_WEIGHTS_DIR. models/ is gitignored.

Cross-check veto (on by default)

The pipeline runs a second matting model (ZhengPeng7/BiRefNet_HR-matting, ~425 MB) as an independent "second opinion" — what it does and why is in README.md → Pipelines. Deploy-side facts:

• It is ON by default (CLI and HTTP service), so provision this model (Option A or B above). ⚠️ On an offline server (HF_HUB_OFFLINE=1) it must be present before the first request — otherwise every default request fails.
• Turn it off service-wide with BGFILTER_CROSS_CHECK=0, or per request with the cross_check form field; the service preloads it at startup when enabled.
• cross_check.reuse_as_seg: true in the config (default off) makes that forward double as the segmentation mask: one less model loaded, ~20 s/image faster on CPU, ~1 GB less VRAM. Config-only — restart to change it.

Background mode (flat vs complex)

background_mode=complex (non-flat / scene backgrounds) needs no extra weights — it reuses the already-provisioned BiRefNet segmenter + ViTMatte and bypasses the colour-keyed stages. The cross-check veto defaults off in complex mode (pass cross_check=on to enable it). Field semantics and response headers: README.md → HTTP service.

Downloading behind a firewall (CN networks)

If direct HuggingFace access is blocked, use the mirror with the local proxy bypassed and Xet disabled — the reliable combination:

export HF_ENDPOINT=https://hf-mirror.com   # domestic mirror
export NO_PROXY='*'                        # bypass the proxy; the mirror is direct
export HF_HUB_DISABLE_XET=1                 # repos are Xet-backed; force classic HTTP

This avoids two failure modes: (1) mirror + an overseas proxy makes the mirror redirect back to huggingface.co, which recent huggingface_hub rejects with FileMetadataError; (2) with hf-xet installed the Xet path fails instantly.

6. Configuration (environment variables)

Variable	Default	Purpose
BGFILTER_CONFIG	configs/default.yaml	Pipeline config file
BGFILTER_DEVICE	cpu	Overrides both model and segmentation device (cuda for GPU)
BGFILTER_MAX_IMAGE_PIXELS	4194304	Reject larger inputs with 413 (~4 MP)
BGFILTER_FETCH_MAX_BYTES	26214400	image_url mode: cap the downloaded body (~25 MB) → 413. No SSRF filtering — keep auth on / callers trusted.
BGFILTER_FETCH_TIMEOUT	15	image_url mode: read timeout (s) for the fetch (connect fixed at 5 s). Outbound fetch concurrency is fixed at 2 × cpu_workers (hard cap 64).
BGFILTER_PRELOAD	1	Load default models at startup (first request isn't cold)
BGFILTER_ACCESS_LOG	—	Path of the structured access log (JSONL, one record per request + one startup meta record; app-rotated at 50 MB × 5). Fields: request_id, params, sizes, stage timings (t_queue/fetch/decode/process/total_ms), status, truncated exc on failures; the id is also echoed as the X-Request-ID response header. Unset = records fall back to stderr (dev). The prose event log (startup/warnings/tracebacks) always goes to stderr for supervisord/journald to capture — per-request data never lands there.
BGFILTER_LOG_LEVEL	INFO	Root log level (event stream).
BGFILTER_CROSS_CHECK	config (on)	Force the cross-check veto on/off (0 disables; needs BiRefNet_HR-matting)
BGFILTER_CPU_WORKERS	config server.cpu_workers (1)	Thread-pool size for request pipelining. 1 = strictly serial (one request at a time). >1 lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). Set it in the config (server: section) or override per box with this env var.
BGFILTER_GPU_CONCURRENCY	config server.gpu_concurrency (1)	How many requests may be inside a GPU forward at once. 1 = single-stream (safe). 2 lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as weights + N × peak_forward ≤ VRAM (on a 32 GB card, 2 peaks ~19 GB at 2048 cross-check; keep 1 on ≤16 GB). Needs cpu_workers>1 to matter. Config key server.gpu_concurrency; this env var overrides it.
BGFILTER_AUTH_TOKEN	—	If set, /remove-background requires header X-Genarrative-Image-Token to equal it (constant-time; 401 otherwise). Unset = open (auth left to a fronting proxy). /healthz stays open. See §10.
BGFILTER_WEIGHTS_DIR	models	Where local weight folders are looked up (Option B)
HF_HOME	—	HF cache location (Option A)
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE	—	Set to 1 in production once weights are cached
OMP_NUM_THREADS / MKL_NUM_THREADS	—	Cap CPU threads (e.g. 4)

Config file: default.yaml vs the cpu-fast lane

BGFILTER_CONFIG selects the pipeline config. Two are shipped:

• configs/default.yaml — full quality. Cross-check veto on (loads the extra BiRefNet_HR-matting at 2048 as a second opinion), best edges. Heavier: on large inputs the peak working set runs well into double-digit GB, so it wants a roomy box (or a GPU).
• configs/cpu-fast.yaml — the lean CPU lane, tuned for an 8 GB box. fp32, BiRefNet segmentation only, no cross-check, no anime-seg, chunked attention (attn_query_chunk: 512). Loads only two weights (base ViTMatte + BiRefNet). Measured peak working set: ~5.6 GB @ 2048×1152 (2.4 MP), ~7.1 GB @ 2048×2048 (4.2 MP); ~14–15 s per 1.5 MP image.

On a ≤8 GB server, set BGFILTER_CONFIG=configs/cpu-fast.yaml. The default lane plus a large input is exactly what drives ViTMatte's attention transient into an OOM — the worker is killed (SIGKILL) and a fronting nginx returns 502 while the app log shows DefaultCPUAllocator: can't allocate memory.

Two model knobs live in the config file (model: section), not env vars:

Config key	Default	Purpose
model.precision	fp32	fp32 or bf16. bf16 is used only on hardware with native kernels (CUDA, or a CPU with AVX512-BF16/AMX); otherwise it warns and falls back to fp32. Also settable per model under segmentation: / cross_check:.
model.attn_query_chunk	2048	Query-row chunk for ViTMatte's global attention. Smaller (e.g. 512) caps the O(N²) memory transient for a few % more compute; output is bit-for-bit identical. 0 disables chunking (stock one-shot attention).

Despill is off by default (all lanes, cpu-fast included). It pulls foreground chroma along the background-hue axis with no positional/semantic guard, so a subject sharing the background's hue (e.g. a blue suit on a blue backdrop) gets desaturated / hue-shifted (~ΔE 35). Re-enable per-config with despill: {enabled: true} only when edge-spill removal matters more than same-hue fidelity.

7. Run the service

BGFILTER_ACCESS_LOG=logs/bgfilter-access.jsonl \
python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1 --no-access-log

• Keep --workers 1. Each worker loads its own full copy of the models; more workers multiply memory (an 8 GB box will OOM). The service already serializes inference with a global lock.
• --host 127.0.0.1 exposes it to the local host only (intended for a co-located caller). See §10 for remote access.
• --no-access-log silences uvicorn's prose per-request lines — they duplicate (a subset of) the app's structured access records (§6, BGFILTER_ACCESS_LOG), and per-request data does not belong in the prose stream.

8. Run as a systemd service

/etc/systemd/system/genarrative-bgfilter.service:

[Unit]
Description=Genarrative BgFilter background removal service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/opt/genarrative-image-host/bgfilter-service
Environment=HF_HOME=/opt/genarrative-image-host/bgfilter-service/model-cache/hf
Environment=HF_HUB_OFFLINE=1
Environment=TRANSFORMERS_OFFLINE=1
Environment=HF_HUB_DISABLE_XET=1
Environment=BGFILTER_CONFIG=configs/default.yaml
Environment=BGFILTER_DEVICE=cpu
Environment=BGFILTER_MAX_IMAGE_PIXELS=4194304
Environment=BGFILTER_PRELOAD=1
Environment=BGFILTER_ACCESS_LOG=/opt/genarrative-image-host/bgfilter-service/logs/bgfilter-access.jsonl
Environment=OMP_NUM_THREADS=4
Environment=MKL_NUM_THREADS=4
ExecStart=/opt/genarrative-image-host/bgfilter-service/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1 --no-access-log
Restart=on-failure
RestartSec=5
MemoryMax=8G
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/opt/genarrative-image-host

[Install]
WantedBy=multi-user.target

On a ≤8 GB box, change the config line to the lean lane: Environment=BGFILTER_CONFIG=configs/cpu-fast.yaml (see §6) — the default lane can OOM on large inputs.

If you use Option B (local models/ folders) instead of the HF cache, drop the HF_HOME/offline lines — those weights need no cache or network.

sudo systemctl daemon-reload
sudo systemctl enable --now genarrative-bgfilter
sudo systemctl status genarrative-bgfilter
journalctl -u genarrative-bgfilter -f          # wait for "Application startup complete"

9. Verify

curl -s http://127.0.0.1:18083/healthz
# {"ok":true,"service":"bgfilter","version":"0.1.0","defaultSegModel":"birefnet","device":"cpu"}

curl -sS -F "file=@input.png" -F "screen_color=#CFEFFF" \
  http://127.0.0.1:18083/remove-background -o out.png

# non-flat / scene background (no colour key; cross-check defaults off here)
curl -sS -F "file=@scene.jpg" -F "background_mode=complex" \
  http://127.0.0.1:18083/remove-background -o out.png

10. Exposing the service

The unit binds 127.0.0.1 — intended to be called by a co-located client (e.g. the Genarrative Rust BFF via GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_BASE_URL). To reach it from other hosts:

• Reverse proxy (recommended): put nginx/caddy in front, terminate TLS, add rate-limiting, and forward to 127.0.0.1:18083.
• Direct: change --host to 0.0.0.0 and restrict source IPs with a firewall. Do not expose it to the public internet without authentication (below).

Authentication

App-level token auth is opt-in via BGFILTER_AUTH_TOKEN:

• Set it → /remove-background requires header X-Genarrative-Image-Token: <token> (constant-time compare); missing or wrong returns 401. /healthz stays open. Use this whenever the app is reachable directly — --host 0.0.0.0, or a cloud port mapping with no proxy in front.
• Leave it unset → the app is open (fail-open) and logs a startup WARNING. Correct only when a fronting proxy already authenticates — e.g. the production nginx checks X-Genarrative-Image-Token itself, so the app behind it needs no token.

The header name matches the nginx layer, so a caller only swaps the base URL + token value — no request-shape change:

curl -H "X-Genarrative-Image-Token: <token>" \
  -F "file=@input.png" \
  https://<host>/remove-background -o out.png

11. Capacity & performance

• CPU throughput: the cpu-fast lane does ~14–15 s per 1.5 MP image (base ViTMatte + one BiRefNet@1024, chunked attention). The default lane is slower — the cross-check adds a second HR-matting@2048 forward. Requests are serialized by the global lock, so plan for a handful of images/min per instance.

• Default lane, workstation CPU reference (Ryzen 9700X, native bf16, 32 GB; cross-check on with reuse_as_seg, chunked attention):

  input	precision	warm / image	peak memory
  1024×1536	fp32	~45 s	—
  1024×1536	bf16	~33 s	—
  2048×2048	bf16	~53 s	11.4 GB (batch), 8.1 GB (single image)

  Without reuse_as_seg add the primary BiRefNet@1024 forward (~+20 s/image). The memory ceiling is the cross-check HR forward (fixed input_size 2048 regardless of image size). Before chunked attention the same 2048² bf16 run peaked at 25.2 GB; fully un-optimized fp32 would need an estimated 55–60 GB. CPUs without native bf16 (e.g. Zen 2) auto-fall back to fp32 — prefer --no-cross-check there if memory or latency is tight.

• For throughput: use a GPU (BGFILTER_DEVICE=cuda, install a CUDA torch build) — roughly an order of magnitude faster — and/or run multiple instances behind a load balancer (each with --workers 1).

• GPU deployment (measured, RTX 5070 Ti 16 GB, full pipeline — cross-check on, bf16): ~1–2 s/image (2 s at 4 MP), peak VRAM ~10.9 GB allocated (10.1 GB with reuse_as_seg: true). The number that must fit on the card is the allocator's reserved footprint. bgfilter defaults PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True on Linux/WSL2 (set before torch loads; native Windows does not support it, and an explicit env value always wins). Measured effect: reserved 15.3 GB → 12.0 GB with reuse (13.3 GB without, the default) and ~10 % faster — the full pipeline fits a 16 GB card with ~2–3.5 GB headroom even next to a desktop session. Without expandable segments (e.g. native Windows) reserved is ~15.3–16.2 GB: borderline-to-overflow on a 16 GB card. Precision bf16 is what makes 16 GB viable at all — fp32 reserved ~17 GB overflows.

• Memory: chunked attention (attn_query_chunk) caps ViTMatte's O(N²) transient, so the peak is set by dense compute, not the attention map. On the cpu-fast lane the measured peak is ~5.6 GB @ 2.4 MP / ~7.1 GB @ 4.2 MP — MemoryMax=8G with the default 4 MP input cap is a safe ceiling for a single CPU worker. The default lane (cross-check on) needs more headroom.

• Keeping RSS flat (glibc). After each image the service calls glibc malloc_trim(0) (bgfilter/memtune.py) to hand a large BiRefNet@2048 forward's freed activations back to the OS — without it, ptmalloc keeps them in the arena and RSS ratchets up across requests. For lower memory and better CPU throughput, preload jemalloc (PyTorch's recommendation for CPU inference) and let it decay dirty pages aggressively:

  # in the [Service] section of the systemd unit
  Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
  Environment=MALLOC_CONF=background_thread:true,dirty_decay_ms:0,muzzy_decay_ms:0
  

  (apt install libjemalloc2; tcmalloc via libtcmalloc_minimal.so.4 works too.) jemalloc/tcmalloc manage their own release, so malloc_trim simply no-ops under them. If you cannot preload an allocator, MALLOC_ARENA_MAX=2 and MALLOC_TRIM_THRESHOLD_=0 in the unit environment curb ptmalloc retention.

12. Troubleshooting

Symptom	Cause / fix
pip pulls nvidia-*/cuda-* (huge)	You skipped the CPU index in §4. Install torch from --index-url https://download.pytorch.org/whl/cpu.
ImportError: libGL.so.1 on import cv2	Using opencv-python on a headless box. Switch to opencv-python-headless (§4).
ensurepip is not available on venv create	Install python3-venv (§1).
FileMetadataError: ... not on huggingface.co while downloading	hf-mirror + a proxy conflict. Bypass the proxy (NO_PROXY='*') and set HF_HUB_DISABLE_XET=1 (§5).
LocalEntryNotFoundError, 0 bytes	Xet download path failing; set HF_HUB_DISABLE_XET=1.
503 service not ready	Models still loading at startup; wait for "Application startup complete".
500 inference failed right after enabling BGFILTER_CROSS_CHECK (offline)	BiRefNet_HR-matting not provisioned; download it into the cache/models/ (§5).
OOM / killed under load	More than one worker, or MemoryMax too low. Keep --workers 1.
Worker killed (SIGKILL) / 502 on a large image; log shows DefaultCPUAllocator: can't allocate memory	ViTMatte OOM on a big input. Use BGFILTER_CONFIG=configs/cpu-fast.yaml (§6), and/or lower BGFILTER_MAX_IMAGE_PIXELS.
413/large upload rejected at the proxy	nginx client_max_body_size (default 1 MB) — raise it in the server block (e.g. client_max_body_size 12m;); this is separate from the app's BGFILTER_MAX_IMAGE_PIXELS.
400 could not auto-detect background colour	Input has no clean flat border; pass screen_color=#RRGGBB.