Pipeline concurrent requests over a shared GPU with a narrowed lock

The single asyncio lock serialized the whole request (chroma -> seg ->
trimap -> ViTMatte -> cross-check -> foreground -> despill), so the GPU
sat idle ~70-80% of each request while the CPU stages ran. Replace it
with a thread pool that runs several requests concurrently, and move the
mutex down to just the GPU forwards: when one request steps off the GPU
to do CPU work, another steps on, keeping the device fed.

Two opt-in env knobs, both defaulting to the old strictly-serial
behaviour:
  * BGFILTER_CPU_WORKERS (pool size) attacks the CPU-bound regime -- with
    cross-check off, throughput went 0.91 -> 1.67 req/s (1.84x) at pool=8
    on a 4080S, vs a mere +6% from a second worker process, at half the
    VRAM (one model copy).
  * BGFILTER_GPU_CONCURRENCY caps how many forwards may co-schedule on the
    SMs. batch-1 underfills the GPU, so =2 recovers the SM overlap a lock
    forgoes: cross-check on went 0.83 -> 1.01 req/s, beating two worker
    processes (0.92) with one model copy (~19 GB vs 24 GB).

The CLI/single-thread path passes no lock and uses a nullcontext, so it
stays allocation- and contention-free. Concurrent output verified
identical (10x same-frame -> one hash). A per-worker admission semaphore
bounds in-flight decoded images to the pool size.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

bgfilter/pipeline.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import contextlib
 import json
 from dataclasses import asdict, dataclass
 from pathlib import Path
@@ -60,6 +61,7 @@ class MattingPipeline:
         runner: ViTMatteRunner | None = None,
         segmenter=None,
         cross_checker=None,
+        gpu_lock=None,
     ):
         self.settings = settings
         # Pre-loaded models can be injected so a long-lived service shares them
@@ -67,6 +69,10 @@ class MattingPipeline:
         self._runner: ViTMatteRunner | None = runner
         self._segmenter = segmenter
         self._cross_checker = cross_checker
+        # Held around every GPU model forward so concurrent requests never drive the
+        # device at once (models are shared, single GPU). None = no contention
+        # (CLI / single-thread), so a no-op context keeps that path lock-free.
+        self._gpu_lock = gpu_lock if gpu_lock is not None else contextlib.nullcontext()
 
     def _segment(self, rgb: np.ndarray) -> np.ndarray:
         if self._segmenter is None:
@@ -166,10 +172,12 @@ def _process_rgb(rgb: np.ndarray, pipeline: MattingPipeline) -> MattingResult:
         if reuse:
             # One HR-matting forward serves both the trimap topology and the
             # cross-check second opinion; the primary seg model is never loaded.
-            second_alpha = pipeline._second_opinion(rgb)
+            with pipeline._gpu_lock:
+                second_alpha = pipeline._second_opinion(rgb)
             seg_mask = second_alpha
         else:
-            seg_mask = pipeline._segment(rgb)
+            with pipeline._gpu_lock:
+                seg_mask = pipeline._segment(rgb)
         # The directional modes are colour tests against the key colour; without
         # one, the seg-topology trimap is the only meaningful choice.
         mode = settings.trimap.mode if settings.chroma.enabled else "seg"
@@ -188,7 +196,8 @@ def _process_rgb(rgb: np.ndarray, pipeline: MattingPipeline) -> MattingResult:
     else:
         seg_mask = None
         trimap, trimap_stats = generate_trimap(bg_confidence, settings.trimap)
-    raw_alpha, alpha_source = pipeline._predict_alpha(rgb, trimap, bg_confidence)
+    with pipeline._gpu_lock:
+        raw_alpha, alpha_source = pipeline._predict_alpha(rgb, trimap, bg_confidence)
 
     alpha = enforce_trimap(raw_alpha, trimap)
     if settings.chroma.enabled:
@@ -202,7 +211,8 @@ def _process_rgb(rgb: np.ndarray, pipeline: MattingPipeline) -> MattingResult:
     if settings.cross_check.enabled:
         lab = convert_color_spaces(rgb)[1]
         if second_alpha is None:
-            second_alpha = pipeline._second_opinion(rgb)
+            with pipeline._gpu_lock:
+                second_alpha = pipeline._second_opinion(rgb)
         alpha = cross_check_alpha(
             alpha,
             second_alpha,
@@ -213,9 +223,12 @@ def _process_rgb(rgb: np.ndarray, pipeline: MattingPipeline) -> MattingResult:
             trimap,
             settings.cross_check,
         )
-    foreground = estimate_foreground_rgb(
-        rgb, alpha, bg_confidence, model, settings.foreground
-    )
+    # Foreground colour estimation runs on the GPU (cupy) when enabled, so it shares
+    # the mutex; the CPU fallback path is short and simply runs while holding it.
+    with pipeline._gpu_lock:
+        foreground = estimate_foreground_rgb(
+            rgb, alpha, bg_confidence, model, settings.foreground
+        )
     corrected_rgb, color_mask = despill(foreground, alpha, model, settings.despill)
 
     metadata = {