The single asyncio lock serialized the whole request (chroma -> seg ->
trimap -> ViTMatte -> cross-check -> foreground -> despill), so the GPU
sat idle ~70-80% of each request while the CPU stages ran. Replace it
with a thread pool that runs several requests concurrently, and move the
mutex down to just the GPU forwards: when one request steps off the GPU
to do CPU work, another steps on, keeping the device fed.
Two opt-in env knobs, both defaulting to the old strictly-serial
behaviour:
* BGFILTER_CPU_WORKERS (pool size) attacks the CPU-bound regime -- with
cross-check off, throughput went 0.91 -> 1.67 req/s (1.84x) at pool=8
on a 4080S, vs a mere +6% from a second worker process, at half the
VRAM (one model copy).
* BGFILTER_GPU_CONCURRENCY caps how many forwards may co-schedule on the
SMs. batch-1 underfills the GPU, so =2 recovers the SM overlap a lock
forgoes: cross-check on went 0.83 -> 1.01 req/s, beating two worker
processes (0.92) with one model copy (~19 GB vs 24 GB).
The CLI/single-thread path passes no lock and uses a nullcontext, so it
stays allocation- and contention-free. Concurrent output verified
identical (10x same-frame -> one hash). A per-worker admission semaphore
bounds in-flight decoded images to the pool size.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lhk229
c748fc22a4