Split logging: prose event stream vs structured JSONL access stream
Rule: bulk per-request data never lands in the prose log; everything with analysis value goes to a structured file. - Event stream (stderr, text): startup summary, warnings, tracebacks -- what supervisord/journald already captures. Human-readable, low volume. - Access stream: one JSONL record per request (plus a "startup" meta record with the instance config) written to BGFILTER_ACCESS_LOG, app-rotated 50MB x 5; falls back to stderr when unset (dev). Failed requests carry a truncated `exc` copy so the access file is self-contained; the full traceback stays in the event stream, joined by request_id. - BGFILTER_LOG_FORMAT retired: format is now a property of the stream, not a global switch. - uvicorn runs with --no-access-log everywhere (deploy script + unit examples): its prose per-request lines duplicated a subset of ours. - deploy_autodl.sh --log-dir now provisions both files; docs updated (EN+ZH: env table, AutoDL section, run/systemd examples). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -40,7 +40,7 @@ Arguments:
|
||||
| `--open` | — | Explicitly remove the token and run OPEN (no auth). |
|
||||
| `--port <PORT>` | `6006` | In-container listen port (matches the AutoDL public port mapping). |
|
||||
| `--config <cfg>` | `configs/gpu.yaml` | Pipeline config. |
|
||||
| `--log-dir <dir>` | `~/autodl-tmp` | Directory for the service log `bgfilter.log`. Defaults to the AutoDL persistent data disk `~/autodl-tmp` (survives container resets, unlike `/tmp`). |
|
||||
| `--log-dir <dir>` | `~/autodl-tmp` | Log directory (AutoDL persistent data disk by default — survives container resets, unlike `/tmp`). Holds `bgfilter.log` (prose event log) and `bgfilter-access.jsonl` (structured per-request log, app-rotated). |
|
||||
|
||||
**Assumes**: repo already checked out (no `git pull`), root user, torch/CUDA
|
||||
provided by the AutoDL image, python at `/root/miniconda3`.
|
||||
@@ -64,8 +64,9 @@ $S stop bgfilter # stop the service (supervisord keeps running)
|
||||
$S start bgfilter # start it again
|
||||
$S shutdown # stop everything, incl. supervisord itself
|
||||
|
||||
tail -f ~/autodl-tmp/bgfilter.log # service log (dir set by --log-dir)
|
||||
tail -f ~/supervisord.log # supervisord's own log (when the service won't start)
|
||||
tail -f ~/autodl-tmp/bgfilter.log # event log: startup/warnings/tracebacks (prose)
|
||||
tail -f ~/autodl-tmp/bgfilter-access.jsonl # access log: one JSON record per request
|
||||
tail -f ~/supervisord.log # supervisord's own log (service won't start?)
|
||||
```
|
||||
|
||||
If supervisord itself is not running — fresh container restart with no shell
|
||||
@@ -283,8 +284,8 @@ redirect back to `huggingface.co`, which recent `huggingface_hub` rejects with
|
||||
| `BGFILTER_FETCH_MAX_BYTES` | `26214400` | `image_url` mode: cap the downloaded body (~25 MB) → `413`. No SSRF filtering — keep auth on / callers trusted. |
|
||||
| `BGFILTER_FETCH_TIMEOUT` | `15` | `image_url` mode: read timeout (s) for the fetch (connect fixed at 5 s). Outbound fetch concurrency is fixed at `2 × cpu_workers` (hard cap 64). |
|
||||
| `BGFILTER_PRELOAD` | `1` | Load default models at startup (first request isn't cold) |
|
||||
| `BGFILTER_LOG_FORMAT` | `text` | `text` (human) or `json` (one structured object per line, for log shippers). Each request emits one access line with a `request_id`, params, and stage timings (`t_queue/fetch/decode/process/total_ms`); the id is also echoed as the `X-Request-ID` response header. |
|
||||
| `BGFILTER_LOG_LEVEL` | `INFO` | Root log level. |
|
||||
| `BGFILTER_ACCESS_LOG` | — | Path of the **structured access log** (JSONL, one record per request + one `startup` meta record; app-rotated at 50 MB × 5). Fields: `request_id`, params, sizes, stage timings (`t_queue/fetch/decode/process/total_ms`), status, truncated `exc` on failures; the id is also echoed as the `X-Request-ID` response header. Unset = records fall back to stderr (dev). The prose **event log** (startup/warnings/tracebacks) always goes to stderr for supervisord/journald to capture — per-request data never lands there. |
|
||||
| `BGFILTER_LOG_LEVEL` | `INFO` | Root log level (event stream). |
|
||||
| `BGFILTER_CROSS_CHECK` | config (`on`) | Force the cross-check veto on/off (`0` disables; needs `BiRefNet_HR-matting`) |
|
||||
| `BGFILTER_CPU_WORKERS` | config `server.cpu_workers` (`1`) | Thread-pool size for request pipelining. `1` = strictly serial (one request at a time). `>1` lets that many requests run concurrently, overlapping their CPU stages while a GPU mutex serializes the model forwards — keeps the GPU fed instead of idle. Biggest win when the pipeline is CPU-bound (e.g. cross-check off). Set it in the config (`server:` section) or override per box with this env var. |
|
||||
| `BGFILTER_GPU_CONCURRENCY` | config `server.gpu_concurrency` (`1`) | How many requests may be inside a GPU forward at once. `1` = single-stream (safe). `2` lets two forwards co-schedule on the SMs (batch-1 underfills them), raising GPU-bound throughput — but concurrent activations multiply VRAM, so size it as `weights + N × peak_forward ≤ VRAM` (on a 32 GB card, `2` peaks ~19 GB at 2048 cross-check; keep `1` on ≤16 GB). Needs `cpu_workers>1` to matter. Config key `server.gpu_concurrency`; this env var overrides it. |
|
||||
@@ -329,7 +330,8 @@ Two model knobs live in the **config file** (`model:` section), not env vars:
|
||||
## 7. Run the service
|
||||
|
||||
```bash
|
||||
python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1
|
||||
BGFILTER_ACCESS_LOG=logs/bgfilter-access.jsonl \
|
||||
python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1 --no-access-log
|
||||
```
|
||||
|
||||
- **Keep `--workers 1`.** Each worker loads its own full copy of the models; more
|
||||
@@ -337,6 +339,9 @@ python -m uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1
|
||||
inference with a global lock.
|
||||
- `--host 127.0.0.1` exposes it to the local host only (intended for a co-located
|
||||
caller). See §10 for remote access.
|
||||
- `--no-access-log` silences uvicorn's prose per-request lines — they duplicate (a
|
||||
subset of) the app's structured access records (§6, `BGFILTER_ACCESS_LOG`), and
|
||||
per-request data does not belong in the prose stream.
|
||||
|
||||
## 8. Run as a systemd service
|
||||
|
||||
@@ -359,9 +364,10 @@ Environment=BGFILTER_CONFIG=configs/default.yaml
|
||||
Environment=BGFILTER_DEVICE=cpu
|
||||
Environment=BGFILTER_MAX_IMAGE_PIXELS=4194304
|
||||
Environment=BGFILTER_PRELOAD=1
|
||||
Environment=BGFILTER_ACCESS_LOG=/opt/genarrative-image-host/bgfilter-service/logs/bgfilter-access.jsonl
|
||||
Environment=OMP_NUM_THREADS=4
|
||||
Environment=MKL_NUM_THREADS=4
|
||||
ExecStart=/opt/genarrative-image-host/bgfilter-service/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1
|
||||
ExecStart=/opt/genarrative-image-host/bgfilter-service/.venv/bin/uvicorn app:app --host 127.0.0.1 --port 18083 --workers 1 --no-access-log
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
MemoryMax=8G
|
||||
|
||||
Reference in New Issue
Block a user