Make the pipelining knobs config-file settable (server: section)

CPU_WORKERS / GPU_CONCURRENCY were env-only. Add a `server:` config
section (server.cpu_workers, server.gpu_concurrency) so a lane's
concurrency lives in its yaml alongside device/precision, matching how
model.device already works. Precedence mirrors BGFILTER_DEVICE: config is
the default, the env var overrides per box.

The pool + admission semaphore now build in lifespan (after the config
loads) instead of at import, and PipelineManager reads gpu_concurrency
from the settings. configs/gpu.yaml ships 8 / 2 (its target is the 32 GB
GPU boxes) with an inline VRAM caveat: gpu_concurrency>1 must fit
weights + N × forward, so keep it 1 on ≤16 GB cards.

Verified on a 4080 SUPER: startup logs cpu_workers=8 gpu_concurrency=2
from gpu.yaml with no env set, request returns a valid RGBA.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

DEPLOY.md

@@ -219,8 +219,8 @@ redirect back to `huggingface.co`, which recent `huggingface_hub` rejects with
 | `BGFILTER_MAX_IMAGE_PIXELS` | `4194304` | Reject larger inputs with `413` (~4 MP) |
 | `BGFILTER_PRELOAD` | `1` | Load default models at startup (first request isn't cold) |
 | `BGFILTER_CROSS_CHECK` | config (`on`) | Force the cross-check veto on/off (`0` disables; needs `BiRefNet_HR-matting`) |
-| `BGFILTER_CPU_WORKERS` | `1` | Thread-pool size for request pipelining. `1` = strictly serial (one request at a time). `>1` lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). |
-| `BGFILTER_GPU_CONCURRENCY` | `1` | How many requests may be inside a GPU forward at once. `1` = single-stream (safe). `2` lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as `weights + N × peak_forward ≤ VRAM` (on a 32 GB card, `2` peaks ~19 GB at 2048 cross-check). Needs `BGFILTER_CPU_WORKERS>1` to matter. |
+| `BGFILTER_CPU_WORKERS` | config `server.cpu_workers` (`1`) | Thread-pool size for request pipelining. `1` = strictly serial (one request at a time). `>1` lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). Set it in the config (`server:` section) or override per box with this env var. |
+| `BGFILTER_GPU_CONCURRENCY` | config `server.gpu_concurrency` (`1`) | How many requests may be inside a GPU forward at once. `1` = single-stream (safe). `2` lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as `weights + N × peak_forward ≤ VRAM` (on a 32 GB card, `2` peaks ~19 GB at 2048 cross-check; keep `1` on ≤16 GB). Needs `cpu_workers>1` to matter. Config key `server.gpu_concurrency`; this env var overrides it. |
 | `BGFILTER_AUTH_TOKEN` | — | If set, `/remove-background` requires header `X-Genarrative-Image-Token` to equal it (constant-time; `401` otherwise). Unset = **open** (auth left to a fronting proxy). `/healthz` stays open. See §10. |
 | `BGFILTER_WEIGHTS_DIR` | `models` | Where local weight folders are looked up (Option B) |
 | `HF_HOME` | — | HF cache location (Option A) |

DEPLOY_ZH.md

@@ -202,8 +202,8 @@ export HF_HUB_DISABLE_XET=1                 # 这些 repo 是 Xet 存储；强
 | `BGFILTER_MAX_IMAGE_PIXELS` | `4194304` | 超过则返回 `413`（约 4 MP）|
 | `BGFILTER_PRELOAD` | `1` | 启动时预加载默认模型（首个请求不用冷加载）|
 | `BGFILTER_CROSS_CHECK` | 配置（`on`）| 强制开/关 cross-check 否决（`0` 关闭；需 `BiRefNet_HR-matting`）|
-| `BGFILTER_CPU_WORKERS` | `1` | 请求流水线的线程池大小。`1` = 严格串行（一次一个请求）。`>1` 允许这么多请求并发,重叠各自的 CPU 段,同时用一把 GPU 互斥锁串行化模型前向——让 GPU 别闲着。CPU 为瓶颈时(如关 cross-check)收益最大。 |
-| `BGFILTER_GPU_CONCURRENCY` | `1` | 允许几个请求同时进入 GPU 前向。`1` = 单流(安全)。`2` 让两次前向在 SM 上共同调度(batch-1 喂不满 SM),提升 GPU 瓶颈下的吞吐——但并发激活会成倍占显存,按 `权重 + N × 单次前向峰值 ≤ 显存` 来定(32GB 卡上 `2` 在 2048 cross-check 时峰值约 19GB)。需 `BGFILTER_CPU_WORKERS>1` 才有意义。 |
+| `BGFILTER_CPU_WORKERS` | 配置 `server.cpu_workers`（`1`）| 请求流水线的线程池大小。`1` = 严格串行（一次一个请求）。`>1` 允许这么多请求并发,重叠各自的 CPU 段,同时用一把 GPU 互斥锁串行化模型前向——让 GPU 别闲着。CPU 为瓶颈时(如关 cross-check)收益最大。可写进配置的 `server:` 段,或用此 env 变量按机器覆盖。 |
+| `BGFILTER_GPU_CONCURRENCY` | 配置 `server.gpu_concurrency`（`1`）| 允许几个请求同时进入 GPU 前向。`1` = 单流(安全)。`2` 让两次前向在 SM 上共同调度(batch-1 喂不满 SM),提升 GPU 瓶颈下的吞吐——但并发激活会成倍占显存,按 `权重 + N × 单次前向峰值 ≤ 显存` 来定(32GB 卡上 `2` 在 2048 cross-check 时峰值约 19GB;≤16GB 卡保持 `1`)。需 `cpu_workers>1` 才有意义。配置键 `server.gpu_concurrency`,此 env 变量覆盖它。 |
 | `BGFILTER_AUTH_TOKEN` | — | 若设置，`/remove-background` 要求 header `X-Genarrative-Image-Token` 等于它（常量时间比较；否则 `401`）。不设 = **开放**（鉴权交给前置代理）。`/healthz` 始终开放。见 §10。 |
 | `BGFILTER_WEIGHTS_DIR` | `models` | 本地权重文件夹的查找目录（方案 B）|
 | `HF_HOME` | — | HF 缓存位置（方案 A）|

app.py

@@ -74,12 +74,13 @@ _manager: PipelineManager | None = None
 # overlap, while a GPU mutex inside the pipeline keeps the model forwards mutually
 # exclusive on the single device -- i.e. when one request steps off the GPU to do
 # CPU work, another steps on, keeping the GPU fed instead of idle.
-# BGFILTER_CPU_WORKERS=1 (default) gives a 1-thread pool == the old strictly serial
-# behaviour. A semaphore bounds in-flight requests (hence decoded-image RAM) to the
-# pool size so a burst cannot pile unbounded decoded images before the pool drains.
-CPU_WORKERS = max(1, _int_env("BGFILTER_CPU_WORKERS", 1))
-_executor = ThreadPoolExecutor(max_workers=CPU_WORKERS, thread_name_prefix="bgf")
-_sem = asyncio.Semaphore(CPU_WORKERS)
+# The pool size defaults to config (server.cpu_workers, default 1 == strictly serial)
+# and is overridable by BGFILTER_CPU_WORKERS. Both are resolved in lifespan once the
+# config is loaded; a same-sized semaphore bounds in-flight requests (hence
+# decoded-image RAM) so a burst cannot pile unbounded decoded images before the pool
+# drains.
+_executor: ThreadPoolExecutor | None = None
+_sem: asyncio.Semaphore | None = None
 
 
 def _build_settings():
@@ -105,15 +106,22 @@ def _build_settings():
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):
-    global _manager
+    global _manager, _executor, _sem
     settings = _build_settings()
     preload = os.environ.get("BGFILTER_PRELOAD", "1").strip().lower() not in ("0", "false", "no", "")
+    # Pool size: config default (server.cpu_workers), env override wins.
+    cpu_workers = max(1, _int_env("BGFILTER_CPU_WORKERS", settings.server.cpu_workers))
+    _executor = ThreadPoolExecutor(max_workers=cpu_workers, thread_name_prefix="bgf")
+    _sem = asyncio.Semaphore(cpu_workers)
     logger.info(
-        "starting bgfilter service: device=%s default_seg=%s cross_check=%s preload=%s max_pixels=%d auth=%s",
+        "starting bgfilter service: device=%s default_seg=%s cross_check=%s preload=%s "
+        "cpu_workers=%d gpu_concurrency=%d max_pixels=%d auth=%s",
         settings.model.device,
         settings.segmentation.backend,
         settings.cross_check.enabled,
         preload,
+        cpu_workers,
+        max(1, _int_env("BGFILTER_GPU_CONCURRENCY", settings.server.gpu_concurrency)),
         MAX_IMAGE_PIXELS,
         "enabled" if AUTH_TOKEN else "DISABLED(open)",
     )

bgfilter/config.py

@@ -13,6 +13,7 @@ from .settings import (
     ModelSettings,
     PipelineSettings,
     SegmentationSettings,
+    ServerSettings,
     TrimapSettings,
 )
 
@@ -49,7 +50,7 @@ def _update_dataclass(instance: T, values: dict[str, Any] | None) -> T:
 def settings_from_dict(data: dict[str, Any]) -> PipelineSettings:
     data = dict(data)
     screen_color = data.pop("screen_color", None)
-    allowed_sections = {"chroma", "trimap", "alpha_post", "cross_check", "foreground", "despill", "model", "segmentation"}
+    allowed_sections = {"chroma", "trimap", "alpha_post", "cross_check", "foreground", "despill", "model", "segmentation", "server"}
     unknown_sections = sorted(set(data) - allowed_sections)
     if unknown_sections:
         raise ValueError(f"Unknown config section(s): {', '.join(unknown_sections)}")
@@ -63,6 +64,7 @@ def settings_from_dict(data: dict[str, Any]) -> PipelineSettings:
         despill=_update_dataclass(DespillSettings(), data.get("despill")),
         model=_update_dataclass(ModelSettings(), data.get("model")),
         segmentation=_update_dataclass(SegmentationSettings(), data.get("segmentation")),
+        server=_update_dataclass(ServerSettings(), data.get("server")),
         screen_color=screen_color,
     )
 
@@ -141,5 +143,6 @@ def override_settings(settings: PipelineSettings, **overrides: Any) -> PipelineS
         despill=_update_dataclass(despill, despill_updates),
         model=_update_dataclass(model, model_updates),
         segmentation=_update_dataclass(segmentation, seg_updates),
+        server=settings.server,
         screen_color=screen_color,
     )

bgfilter/service.py

@@ -43,8 +43,9 @@ class PipelineManager:
         # pipeline this manager builds. =1 (default) makes the device strictly
         # single-stream; >1 lets that many forwards overlap so batch-1 kernels that
         # underfill the SMs can co-schedule (higher GPU-bound throughput, but the
-        # concurrent activations multiply GPU memory). Uncontended when CPU_WORKERS=1.
-        gpu_conc = max(1, int(os.environ.get("BGFILTER_GPU_CONCURRENCY", "1")))
+        # concurrent activations multiply GPU memory). Config value
+        # (``server.gpu_concurrency``) is the default; the env var overrides it.
+        gpu_conc = max(1, int(os.environ.get("BGFILTER_GPU_CONCURRENCY", base_settings.server.gpu_concurrency)))
         self._gpu_lock = threading.Lock() if gpu_conc == 1 else threading.BoundedSemaphore(gpu_conc)
         if preload:
             if base_settings.model.matting_method == "vitmatte":

bgfilter/settings.py

@@ -196,6 +196,22 @@ class SegmentationSettings:
     precision: str = "fp32"
 
 
+@dataclass(frozen=True)
+class ServerSettings:
+    """HTTP-service request concurrency (ignored by the CLI).
+
+    ``cpu_workers`` is the thread-pool size for request pipelining: >1 lets that
+    many requests run concurrently, overlapping their CPU stages while the GPU
+    forwards stay serialized. ``gpu_concurrency`` caps how many forwards may
+    co-schedule on the device at once (>1 fills the SMs that batch-1 underuses,
+    but multiplies GPU activation memory -- size it to the VRAM). Both env vars
+    ``BGFILTER_CPU_WORKERS`` / ``BGFILTER_GPU_CONCURRENCY`` override these.
+    """
+
+    cpu_workers: int = 1
+    gpu_concurrency: int = 1
+
+
 @dataclass(frozen=True)
 class PipelineSettings:
     chroma: ChromaSettings = ChromaSettings()
@@ -206,6 +222,7 @@ class PipelineSettings:
     despill: DespillSettings = DespillSettings()
     model: ModelSettings = ModelSettings()
     segmentation: SegmentationSettings = SegmentationSettings()
+    server: ServerSettings = ServerSettings()
     # Background-colour prior. None = auto-detect the flat background colour from the
     # image border (raises if there is no clean flat background). A hex string like
     # "#CFEFFF" sets it explicitly.

configs/gpu.yaml

@@ -73,3 +73,15 @@ foreground:
 
 despill:
   enabled: false
+
+server:
+  # Request concurrency (HTTP service only). cpu_workers = thread-pool size: >1 runs
+  # several requests at once, overlapping their CPU stages while the GPU forwards
+  # stay serialized -- keeps the GPU fed. gpu_concurrency = how many forwards may
+  # co-schedule on the SMs at once: 2 lifts GPU-bound throughput (batch-1 underfills
+  # the SMs) but roughly DOUBLES GPU activation memory.
+  #   * On a >=24 GB card (e.g. 4080 SUPER 32 GB): gpu_concurrency: 2 peaks ~19 GB.
+  #   * On a <=16 GB card: keep gpu_concurrency: 1 or it will OOM at 2048 cross-check.
+  # Env vars BGFILTER_CPU_WORKERS / BGFILTER_GPU_CONCURRENCY override these per box.
+  cpu_workers: 8
+  gpu_concurrency: 2