lhk229/BGfilter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split logging: prose event stream vs structured JSONL access stream

Browse Source 
Rule: bulk per-request data never lands in the prose log; everything with
analysis value goes to a structured file.

- Event stream (stderr, text): startup summary, warnings, tracebacks --
  what supervisord/journald already captures. Human-readable, low volume.
- Access stream: one JSONL record per request (plus a "startup" meta
  record with the instance config) written to BGFILTER_ACCESS_LOG,
  app-rotated 50MB x 5; falls back to stderr when unset (dev).
  Failed requests carry a truncated `exc` copy so the access file is
  self-contained; the full traceback stays in the event stream, joined
  by request_id.
- BGFILTER_LOG_FORMAT retired: format is now a property of the stream,
  not a global switch.
- uvicorn runs with --no-access-log everywhere (deploy script + unit
  examples): its prose per-request lines duplicated a subset of ours.
- deploy_autodl.sh --log-dir now provisions both files; docs updated
  (EN+ZH: env table, AutoDL section, run/systemd examples).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
孔令弘
2026-07-17 13:40:33 +00:00
parent 3a8a1ef38a
commit 43797eedeb
4 changed files with 98 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy_autodl.sh
+9 -5
View File
@@ -15,9 +15,10 @@
# --open explicitly remove the token and run OPEN (no auth).
# --port public-mapped container port (default 6006).
# --config pipeline config (default configs/gpu.yaml).
# --log-dir directory for bgfilter.log (default ~/autodl-tmp, the AutoDL
# persistent data disk; unlike /tmp it survives container resets).
# File is <dir>/bgfilter.log.
# --log-dir log directory (default ~/autodl-tmp, the AutoDL persistent data
# disk; unlike /tmp it survives container resets). Holds
# bgfilter.log (prose event log, supervisord-captured) and
# bgfilter-access.jsonl (structured per-request log, app-rotated).
#
# Assumes: the repo is already checked out (this script does NOT git pull), root
# user, torch/CUDA provided by the AutoDL image, python at /root/miniconda3.
@@ -43,7 +44,7 @@ while [ $# -gt 0 ]; do
--port) PORT="${2:-}"; shift 2;;
--config) CONFIG="${2:-}"; shift 2;;
--log-dir) LOG_DIR="${2:-}"; shift 2;;
-h|--help) sed -n '2,30p' "$0" | sed 's/^# \?//'; exit 0;;
-h|--help) sed -n '2,31p' "$0" | sed 's/^# \?//'; exit 0;;
*) echo "unknown arg: $1" >&2; exit 1;;
esac
done
@@ -112,8 +113,11 @@ cat > "$RUN_WRAPPER" <<WRAP
#!/bin/bash
cd "$APP_DIR" || exit 1
export BGFILTER_CONFIG="$CONFIG"
export BGFILTER_ACCESS_LOG="$LOG_DIR/bgfilter-access.jsonl"
[ -f "$TOKEN_FILE" ] && export BGFILTER_AUTH_TOKEN="\$(cat "$TOKEN_FILE")"
exec "$PY" -m uvicorn app:app --host 0.0.0.0 --port $PORT --workers 1
# --no-access-log: uvicorn's prose per-request lines would duplicate (a subset of)
# the app's structured access records -- per-request data stays out of the prose log.
exec "$PY" -m uvicorn app:app --host 0.0.0.0 --port $PORT --workers 1 --no-access-log
WRAP
chmod +x "$RUN_WRAPPER"