Document the measured GPU deployment profile (bf16 + expandable_segments)

Measured on an RTX 5070 Ti 16 GB, full default pipeline (cross-check on, reused as seg, bf16): ~1-2 s/image, ~10.1 GB allocated. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (Linux/WSL2 only; silently unsupported on native Windows) cuts the allocator's reserved footprint 15.3 -> 12.0 GB and is ~10% faster, leaving ~3.5 GB headroom on a 16 GB card even beside a desktop session. bf16 is the precondition: fp32 reserved ~17 GB overflows. reuse-as-seg also saves ~0.9 GB VRAM / ~0.25 s per image on GPU (reuse off measured 13.3 GB reserved under expandable_segments). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 20:32:13 +08:00
parent a48d70d9f7
commit 6bf648cd64
2 changed files with 24 additions and 0 deletions
@@ -289,6 +289,19 @@ service has **no authentication**. To reach it from other hosts:
 - For throughput: use a GPU (`BGFILTER_DEVICE=cuda`, install a CUDA torch build) —
  roughly an order of magnitude faster — and/or run multiple instances behind a load
  balancer (each with `--workers 1`).
+- **GPU deployment (measured, RTX 5070 Ti 16 GB, full default pipeline —
+  cross-check on, reused as seg, bf16):** ~1–2 s/image (2 s at 4 MP), peak VRAM
+  ~10.1 GB allocated. The number that must fit on the card is the allocator's
+  *reserved* footprint; set this in the service environment on Linux:
+  ```ini
+  Environment=PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+  ```
+  Measured effect: reserved 15.3 GB → **12.0 GB** (fragmentation 5.2 → 1.9 GB) and
+  ~10 % *faster* — with it, the full pipeline fits a 16 GB card with ~3.5 GB
+  headroom even next to a desktop session. Without it (or on native Windows, where
+  the option is silently unsupported), reserved is ~15.3 GB: a headless 16 GB card
+  still fits, but a desktop-attached one is borderline. Precision `bf16` is what
+  makes 16 GB viable at all — fp32 reserved ~17 GB overflows.
 - Memory: one worker holds all three models (~1 GB weights + runtime). `MemoryMax=8G`
  is a safe ceiling for a single CPU worker.
 - **Keeping RSS flat (glibc).** After each image the service calls glibc