diff --git a/DEPLOY.md b/DEPLOY.md index cd44a4e..2462dfb 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -219,8 +219,8 @@ redirect back to `huggingface.co`, which recent `huggingface_hub` rejects with | `BGFILTER_MAX_IMAGE_PIXELS` | `4194304` | Reject larger inputs with `413` (~4 MP) | | `BGFILTER_PRELOAD` | `1` | Load default models at startup (first request isn't cold) | | `BGFILTER_CROSS_CHECK` | config (`on`) | Force the cross-check veto on/off (`0` disables; needs `BiRefNet_HR-matting`) | -| `BGFILTER_CPU_WORKERS` | `1` | Thread-pool size for request pipelining. `1` = strictly serial (one request at a time). `>1` lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). | -| `BGFILTER_GPU_CONCURRENCY` | `1` | How many requests may be inside a GPU forward at once. `1` = single-stream (safe). `2` lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as `weights + N × peak_forward ≤ VRAM` (on a 32 GB card, `2` peaks ~19 GB at 2048 cross-check). Needs `BGFILTER_CPU_WORKERS>1` to matter. | +| `BGFILTER_CPU_WORKERS` | config `server.cpu_workers` (`1`) | Thread-pool size for request pipelining. `1` = strictly serial (one request at a time). `>1` lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). Set it in the config (`server:` section) or override per box with this env var. | +| `BGFILTER_GPU_CONCURRENCY` | config `server.gpu_concurrency` (`1`) | How many requests may be inside a GPU forward at once. `1` = single-stream (safe). `2` lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as `weights + N × peak_forward ≤ VRAM` (on a 32 GB card, `2` peaks ~19 GB at 2048 cross-check; keep `1` on ≤16 GB). Needs `cpu_workers>1` to matter. Config key `server.gpu_concurrency`; this env var overrides it. | | `BGFILTER_AUTH_TOKEN` | — | If set, `/remove-background` requires header `X-Genarrative-Image-Token` to equal it (constant-time; `401` otherwise). Unset = **open** (auth left to a fronting proxy). `/healthz` stays open. See §10. | | `BGFILTER_WEIGHTS_DIR` | `models` | Where local weight folders are looked up (Option B) | | `HF_HOME` | — | HF cache location (Option A) | diff --git a/DEPLOY_ZH.md b/DEPLOY_ZH.md index 79a93e6..5d7cb49 100644 --- a/DEPLOY_ZH.md +++ b/DEPLOY_ZH.md @@ -202,8 +202,8 @@ export HF_HUB_DISABLE_XET=1 # 这些 repo 是 Xet 存储;强 | `BGFILTER_MAX_IMAGE_PIXELS` | `4194304` | 超过则返回 `413`(约 4 MP)| | `BGFILTER_PRELOAD` | `1` | 启动时预加载默认模型(首个请求不用冷加载)| | `BGFILTER_CROSS_CHECK` | 配置(`on`)| 强制开/关 cross-check 否决(`0` 关闭;需 `BiRefNet_HR-matting`)| -| `BGFILTER_CPU_WORKERS` | `1` | 请求流水线的线程池大小。`1` = 严格串行(一次一个请求)。`>1` 允许这么多请求并发,重叠各自的 CPU 段,同时用一把 GPU 互斥锁串行化模型前向——让 GPU 别闲着。CPU 为瓶颈时(如关 cross-check)收益最大。 | -| `BGFILTER_GPU_CONCURRENCY` | `1` | 允许几个请求同时进入 GPU 前向。`1` = 单流(安全)。`2` 让两次前向在 SM 上共同调度(batch-1 喂不满 SM),提升 GPU 瓶颈下的吞吐——但并发激活会成倍占显存,按 `权重 + N × 单次前向峰值 ≤ 显存` 来定(32GB 卡上 `2` 在 2048 cross-check 时峰值约 19GB)。需 `BGFILTER_CPU_WORKERS>1` 才有意义。 | +| `BGFILTER_CPU_WORKERS` | 配置 `server.cpu_workers`(`1`)| 请求流水线的线程池大小。`1` = 严格串行(一次一个请求)。`>1` 允许这么多请求并发,重叠各自的 CPU 段,同时用一把 GPU 互斥锁串行化模型前向——让 GPU 别闲着。CPU 为瓶颈时(如关 cross-check)收益最大。可写进配置的 `server:` 段,或用此 env 变量按机器覆盖。 | +| `BGFILTER_GPU_CONCURRENCY` | 配置 `server.gpu_concurrency`(`1`)| 允许几个请求同时进入 GPU 前向。`1` = 单流(安全)。`2` 让两次前向在 SM 上共同调度(batch-1 喂不满 SM),提升 GPU 瓶颈下的吞吐——但并发激活会成倍占显存,按 `权重 + N × 单次前向峰值 ≤ 显存` 来定(32GB 卡上 `2` 在 2048 cross-check 时峰值约 19GB;≤16GB 卡保持 `1`)。需 `cpu_workers>1` 才有意义。配置键 `server.gpu_concurrency`,此 env 变量覆盖它。 | | `BGFILTER_AUTH_TOKEN` | — | 若设置,`/remove-background` 要求 header `X-Genarrative-Image-Token` 等于它(常量时间比较;否则 `401`)。不设 = **开放**(鉴权交给前置代理)。`/healthz` 始终开放。见 §10。 | | `BGFILTER_WEIGHTS_DIR` | `models` | 本地权重文件夹的查找目录(方案 B)| | `HF_HOME` | — | HF 缓存位置(方案 A)| diff --git a/app.py b/app.py index 8c24c81..5e27064 100644 --- a/app.py +++ b/app.py @@ -74,12 +74,13 @@ _manager: PipelineManager | None = None # overlap, while a GPU mutex inside the pipeline keeps the model forwards mutually # exclusive on the single device -- i.e. when one request steps off the GPU to do # CPU work, another steps on, keeping the GPU fed instead of idle. -# BGFILTER_CPU_WORKERS=1 (default) gives a 1-thread pool == the old strictly serial -# behaviour. A semaphore bounds in-flight requests (hence decoded-image RAM) to the -# pool size so a burst cannot pile unbounded decoded images before the pool drains. -CPU_WORKERS = max(1, _int_env("BGFILTER_CPU_WORKERS", 1)) -_executor = ThreadPoolExecutor(max_workers=CPU_WORKERS, thread_name_prefix="bgf") -_sem = asyncio.Semaphore(CPU_WORKERS) +# The pool size defaults to config (server.cpu_workers, default 1 == strictly serial) +# and is overridable by BGFILTER_CPU_WORKERS. Both are resolved in lifespan once the +# config is loaded; a same-sized semaphore bounds in-flight requests (hence +# decoded-image RAM) so a burst cannot pile unbounded decoded images before the pool +# drains. +_executor: ThreadPoolExecutor | None = None +_sem: asyncio.Semaphore | None = None def _build_settings(): @@ -105,15 +106,22 @@ def _build_settings(): @asynccontextmanager async def lifespan(app: FastAPI): - global _manager + global _manager, _executor, _sem settings = _build_settings() preload = os.environ.get("BGFILTER_PRELOAD", "1").strip().lower() not in ("0", "false", "no", "") + # Pool size: config default (server.cpu_workers), env override wins. + cpu_workers = max(1, _int_env("BGFILTER_CPU_WORKERS", settings.server.cpu_workers)) + _executor = ThreadPoolExecutor(max_workers=cpu_workers, thread_name_prefix="bgf") + _sem = asyncio.Semaphore(cpu_workers) logger.info( - "starting bgfilter service: device=%s default_seg=%s cross_check=%s preload=%s max_pixels=%d auth=%s", + "starting bgfilter service: device=%s default_seg=%s cross_check=%s preload=%s " + "cpu_workers=%d gpu_concurrency=%d max_pixels=%d auth=%s", settings.model.device, settings.segmentation.backend, settings.cross_check.enabled, preload, + cpu_workers, + max(1, _int_env("BGFILTER_GPU_CONCURRENCY", settings.server.gpu_concurrency)), MAX_IMAGE_PIXELS, "enabled" if AUTH_TOKEN else "DISABLED(open)", ) diff --git a/bgfilter/config.py b/bgfilter/config.py index d58574f..e312cc2 100644 --- a/bgfilter/config.py +++ b/bgfilter/config.py @@ -13,6 +13,7 @@ from .settings import ( ModelSettings, PipelineSettings, SegmentationSettings, + ServerSettings, TrimapSettings, ) @@ -49,7 +50,7 @@ def _update_dataclass(instance: T, values: dict[str, Any] | None) -> T: def settings_from_dict(data: dict[str, Any]) -> PipelineSettings: data = dict(data) screen_color = data.pop("screen_color", None) - allowed_sections = {"chroma", "trimap", "alpha_post", "cross_check", "foreground", "despill", "model", "segmentation"} + allowed_sections = {"chroma", "trimap", "alpha_post", "cross_check", "foreground", "despill", "model", "segmentation", "server"} unknown_sections = sorted(set(data) - allowed_sections) if unknown_sections: raise ValueError(f"Unknown config section(s): {', '.join(unknown_sections)}") @@ -63,6 +64,7 @@ def settings_from_dict(data: dict[str, Any]) -> PipelineSettings: despill=_update_dataclass(DespillSettings(), data.get("despill")), model=_update_dataclass(ModelSettings(), data.get("model")), segmentation=_update_dataclass(SegmentationSettings(), data.get("segmentation")), + server=_update_dataclass(ServerSettings(), data.get("server")), screen_color=screen_color, ) @@ -141,5 +143,6 @@ def override_settings(settings: PipelineSettings, **overrides: Any) -> PipelineS despill=_update_dataclass(despill, despill_updates), model=_update_dataclass(model, model_updates), segmentation=_update_dataclass(segmentation, seg_updates), + server=settings.server, screen_color=screen_color, ) diff --git a/bgfilter/service.py b/bgfilter/service.py index ec1178a..f24aa7d 100644 --- a/bgfilter/service.py +++ b/bgfilter/service.py @@ -43,8 +43,9 @@ class PipelineManager: # pipeline this manager builds. =1 (default) makes the device strictly # single-stream; >1 lets that many forwards overlap so batch-1 kernels that # underfill the SMs can co-schedule (higher GPU-bound throughput, but the - # concurrent activations multiply GPU memory). Uncontended when CPU_WORKERS=1. - gpu_conc = max(1, int(os.environ.get("BGFILTER_GPU_CONCURRENCY", "1"))) + # concurrent activations multiply GPU memory). Config value + # (``server.gpu_concurrency``) is the default; the env var overrides it. + gpu_conc = max(1, int(os.environ.get("BGFILTER_GPU_CONCURRENCY", base_settings.server.gpu_concurrency))) self._gpu_lock = threading.Lock() if gpu_conc == 1 else threading.BoundedSemaphore(gpu_conc) if preload: if base_settings.model.matting_method == "vitmatte": diff --git a/bgfilter/settings.py b/bgfilter/settings.py index 6642f1d..068fde4 100644 --- a/bgfilter/settings.py +++ b/bgfilter/settings.py @@ -196,6 +196,22 @@ class SegmentationSettings: precision: str = "fp32" +@dataclass(frozen=True) +class ServerSettings: + """HTTP-service request concurrency (ignored by the CLI). + + ``cpu_workers`` is the thread-pool size for request pipelining: >1 lets that + many requests run concurrently, overlapping their CPU stages while the GPU + forwards stay serialized. ``gpu_concurrency`` caps how many forwards may + co-schedule on the device at once (>1 fills the SMs that batch-1 underuses, + but multiplies GPU activation memory -- size it to the VRAM). Both env vars + ``BGFILTER_CPU_WORKERS`` / ``BGFILTER_GPU_CONCURRENCY`` override these. + """ + + cpu_workers: int = 1 + gpu_concurrency: int = 1 + + @dataclass(frozen=True) class PipelineSettings: chroma: ChromaSettings = ChromaSettings() @@ -206,6 +222,7 @@ class PipelineSettings: despill: DespillSettings = DespillSettings() model: ModelSettings = ModelSettings() segmentation: SegmentationSettings = SegmentationSettings() + server: ServerSettings = ServerSettings() # Background-colour prior. None = auto-detect the flat background colour from the # image border (raises if there is no clean flat background). A hex string like # "#CFEFFF" sets it explicitly. diff --git a/configs/gpu.yaml b/configs/gpu.yaml index a01c319..617c3a3 100644 --- a/configs/gpu.yaml +++ b/configs/gpu.yaml @@ -73,3 +73,15 @@ foreground: despill: enabled: false + +server: + # Request concurrency (HTTP service only). cpu_workers = thread-pool size: >1 runs + # several requests at once, overlapping their CPU stages while the GPU forwards + # stay serialized -- keeps the GPU fed. gpu_concurrency = how many forwards may + # co-schedule on the SMs at once: 2 lifts GPU-bound throughput (batch-1 underfills + # the SMs) but roughly DOUBLES GPU activation memory. + # * On a >=24 GB card (e.g. 4080 SUPER 32 GB): gpu_concurrency: 2 peaks ~19 GB. + # * On a <=16 GB card: keep gpu_concurrency: 1 or it will OOM at 2048 cross-check. + # Env vars BGFILTER_CPU_WORKERS / BGFILTER_GPU_CONCURRENCY override these per box. + cpu_workers: 8 + gpu_concurrency: 2