添加gfilter专用worker (#103)
Co-authored-by: kdletters <kdletters@qq.com> Reviewed-on: https://git.genarrative.world/git/GenarrativeAI/Genarrative/pulls/103 Co-authored-by: Linghong <ink29535@proton.me> Co-committed-by: Linghong <ink29535@proton.me>
This commit was merged in pull request #103.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Genarrative 容器化压测、隔离部署与 CI Job 镜像
|
||||
|
||||
本目录同时保存两类互不替代的容器资产：本机或预发的容器化模拟压测，以及 Gitea Actions 使用的预构建 CI job 镜像。它们都不替换当前生产 `systemd + Nginx + Jenkins` 发布路径；生产服务器仍以 `deploy/systemd/`、`deploy/nginx/`、`scripts/jenkins-*.sh` 和 `scripts/deploy/production-api-deploy.sh` 为准。
|
||||
本目录同时保存两类互不替代的容器资产：本机或预发的容器化模拟压测，以及 Gitea Actions 使用的预构建 CI job 镜像。它们都不替换当前生产 `systemd + Nginx + Jenkins` 发布路径；生产服务器仍以 `deploy/systemd/`、`deploy/nginx/`、`scripts/jenkins-*.sh` 和 `scripts/deploy/production-api-deploy.sh` 为准。当前 compose 不包含独立 `bgfilter-worker`，因此不是完整 BgFilter 预发拓扑，也不覆盖会触发 BgFilter 的现役任务；这里只验证非 BgFilter 路径，或使用 unsupported job 检查队列 claim / fail 回写和 API / worker 进程隔离。
|
||||
|
||||
## 拓扑
|
||||
|
||||
@@ -15,7 +15,7 @@ Docker Compose
|
||||
|
||||
当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`external-generation-worker cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`。SpacetimeDB 同时设置 `--page_pool_max_size=402653184`，给 reducer、订阅与运行时保留更多非 page pool 内存。
|
||||
容器 `api-server` 默认 `GENARRATIVE_API_WORKER_THREADS=4`，用于让 Tokio 在 2 vCPU 配额内有更多 I/O 调度 worker；该值不会突破 compose 里的 `cpus=2.0` CPU 上限。
|
||||
容器默认 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`，用于验证 `api-server -> external_generation_job -> external-generation-worker` 链路；如只想本地同步排查 provider/OSS/SpacetimeDB 写回，可在本机 env 临时改为 `inline`，但该模式不会覆盖 worker 动态扩缩容验证。
|
||||
容器默认 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`，用于验证不经过 BgFilter 的 `api-server -> external_generation_job -> external-generation-worker` 链路；会触发 BgFilter 的任务不属于当前 compose 验收范围。如只想本地同步排查非 BgFilter provider / OSS / SpacetimeDB 写回，可在本机 env 临时改为 `inline`，但该模式不会覆盖 worker 动态扩缩容验证。
|
||||
Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。
|
||||
生产服务器若启用 Collector，则由 `deploy/systemd/otelcol-contrib.service` 和 `deploy/otelcol/genarrative-debug.yaml` 托管，不走容器镜像。
|
||||
|
||||
|
||||
@@ -30,9 +30,10 @@ GENARRATIVE_WALLET_REFUND_OUTBOX_DIR=/var/lib/genarrative/wallet-refund-outbox
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE=100
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS=1000
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES=67108864
|
||||
GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS=180000
|
||||
GENARRATIVE_BGFILTER_WORKER_CONCURRENCY=16
|
||||
GENARRATIVE_EDITOR_BGFILTER_SINGLE_IMAGE_ESTIMATE_MS=5000
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD=3
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS=300
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS=120
|
||||
# BgFilter 失败后的中间兜底：阿里云通用抠图（SegmentCommonImage）。AccessKey 留空则跳过该层，
|
||||
# BgFilter 失败直接本地 editor_green_screen 去背；填入后恢复 BgFilter→阿里云→本地三级兜底。
|
||||
# AccessKey 也可复用标准 SDK 命名 ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET。
|
||||
|
||||
Vendored
+10
-3
@@ -17,6 +17,10 @@ GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
|
||||
GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=600
|
||||
GENARRATIVE_EXTERNAL_GENERATION_WORKER_JOB_TIMEOUT_SECONDS=900
|
||||
GENARRATIVE_EXTERNAL_GENERATION_WORKER_LONG_JOB_TIMEOUT_SECONDS=1800
|
||||
# 父流程只访问同机 BgFilter worker；内部 Token 只通过受保护文件共享，不写明文 env。
|
||||
GENARRATIVE_BGFILTER_WORKER_BASE_URL=http://127.0.0.1:8083
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE=/etc/genarrative/secrets/bgfilter-worker.token
|
||||
GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS=2000
|
||||
GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
|
||||
GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16
|
||||
GENARRATIVE_API_SHUTDOWN_OUTBOX_FLUSH_TIMEOUT_MS=5000
|
||||
@@ -30,9 +34,12 @@ GENARRATIVE_WALLET_REFUND_OUTBOX_DIR=/var/lib/genarrative/wallet-refund-outbox
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE=100
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS=1000
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES=67108864
|
||||
GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS=180000
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD=3
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS=300
|
||||
# N 与单图估时：父子共同派生 attempt = N×est×2、callBudget = 2×attempt+1s 的公式输入；
|
||||
# 单一来源放共享 env，专属 worker env 不重复定义。旧固定 REQUEST_TIMEOUT_MS 已删除。
|
||||
GENARRATIVE_BGFILTER_WORKER_CONCURRENCY=16
|
||||
GENARRATIVE_EDITOR_BGFILTER_SINGLE_IMAGE_ESTIMATE_MS=5000
|
||||
GENARRATIVE_EDITOR_BGFILTER_BASE_URL=http://58.87.105.82/bgfilter
|
||||
GENARRATIVE_EDITOR_BGFILTER_TOKEN=
|
||||
# BgFilter 失败后的中间兜底：阿里云通用抠图（SegmentCommonImage）。AccessKey 留空则跳过该层，
|
||||
# BgFilter 失败直接本地 editor_green_screen 去背；填入后恢复 BgFilter→阿里云→本地三级兜底。
|
||||
# AccessKey 也可复用标准 SDK 命名 ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET。
|
||||
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
# 复制到 /etc/genarrative/bgfilter-worker.env；只放专用进程独占参数和可选日志覆盖。
|
||||
# provider、OSS、内部 Token 文件、N（CONCURRENCY）与单图估时统一来自先加载的 api-server.env，禁止在此重复定义。
|
||||
# systemd unit 会强制设置 GENARRATIVE_PROCESS_ROLE=bgfilter-worker。
|
||||
|
||||
GENARRATIVE_ENV=production
|
||||
GENARRATIVE_BGFILTER_WORKER_HOST=127.0.0.1
|
||||
GENARRATIVE_BGFILTER_WORKER_PORT=8083
|
||||
# Q：admission 保险丝，只防连接风暴；正常业务不应触达，显式配置时必须 >= N。
|
||||
GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS=2048
|
||||
# flat / complex 熔断只由本进程维护；两种模式共享参数，但状态互相独立。
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD=3
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS=120
|
||||
|
||||
GENARRATIVE_API_LOG=info,tower_http=info
|
||||
GENARRATIVE_OTEL_ENABLED=true
|
||||
OTEL_SERVICE_NAME=genarrative-bgfilter-worker
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=genarrative
|
||||
Vendored
+3
@@ -2,6 +2,9 @@
|
||||
# 默认不启用 Pingora shadow 巡检；只有同时配置 base URL 与 probe token 才会检查。
|
||||
|
||||
GENARRATIVE_HEALTH_PATROL_API_BASE_URL=http://127.0.0.1:8082
|
||||
# 巡检是独立进程，不读取 bgfilter-worker.env；若自定义了 GENARRATIVE_BGFILTER_WORKER_PORT，
|
||||
# 必须在此同步改为相同端口，否则巡检会持续误报 worker 不健康。
|
||||
GENARRATIVE_HEALTH_PATROL_BGFILTER_BASE_URL=http://127.0.0.1:8083
|
||||
GENARRATIVE_HEALTH_PATROL_SPACETIME_BASE_URL=http://127.0.0.1:3101
|
||||
GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL=http://127.0.0.1
|
||||
# 默认公网入口仍按 Nginx 巡检；Pingora 直连切换后改为 pingora-direct。
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
[Unit]
|
||||
Description=Genarrative BgFilter Worker
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=genarrative
|
||||
Group=genarrative
|
||||
WorkingDirectory=/opt/genarrative/current
|
||||
EnvironmentFile=/etc/genarrative/api-server.env
|
||||
EnvironmentFile=/etc/genarrative/bgfilter-worker.env
|
||||
Environment="LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib"
|
||||
ExecStart=/usr/bin/env GENARRATIVE_PROCESS_ROLE=bgfilter-worker OTEL_SERVICE_NAME=genarrative-bgfilter-worker /opt/genarrative/current/api-server
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
KillSignal=SIGINT
|
||||
# shutdown 会立即拒绝仍在排队的请求，只排空已取得 provider permit 的调用；默认 N=16、est=5000ms 时 callBudgetMs=321s，额外窗口用于响应发送和进程收口。
|
||||
TimeoutStopSec=900
|
||||
LimitNOFILE=65535
|
||||
TasksMax=2048
|
||||
|
||||
# 固定 loopback 地址与非模板 unit 共同保证首版同机只运行一个 BgFilter worker。
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths=/opt/genarrative /var/lib/genarrative
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user