落地 BgFilter 单实例受限资源 Worker
新增私有 BgFilter worker,提供内部鉴权、并发限流、超时、顺序重试、flat 熔断和图片校验。 父生成流程改为通过内部二进制 HTTP 原地等待,并保留 flat 降级与 complex 失败语义。 接入本地开发、systemd、生产部署、Provision、健康巡检和配置漂移门禁。 补充动画、部署与运维测试、Linux fixture 隔离以及对应架构文档。
This commit is contained in:
@@ -143,6 +143,17 @@ ALIYUN_OSS_POST_EXPIRE_SECONDS="600"
|
||||
ALIYUN_OSS_POST_MAX_SIZE_BYTES="20971520"
|
||||
ALIYUN_OSS_SUCCESS_ACTION_STATUS="200"
|
||||
|
||||
# BgFilter 受限资源 worker。父 api-server / external-generation-worker 与唯一的
|
||||
# `GENARRATIVE_PROCESS_ROLE=bgfilter-worker` 进程必须使用同一个内部 Token。
|
||||
# 本地需要真实联调 BgFilter 时,在第二个终端启动专用进程;不要让 `all` 角色兼任它。
|
||||
GENARRATIVE_BGFILTER_WORKER_HOST="127.0.0.1"
|
||||
GENARRATIVE_BGFILTER_WORKER_PORT="8083"
|
||||
GENARRATIVE_BGFILTER_WORKER_BASE_URL="http://127.0.0.1:8083"
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN="CHANGE_ME_FOR_LOCAL"
|
||||
GENARRATIVE_BGFILTER_WORKER_CONCURRENCY="4"
|
||||
GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS="128"
|
||||
GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS="2000"
|
||||
|
||||
# SpacetimeDB 数据目录备份到 OSS。备份 bucket 可与资源 bucket 分离;未设置时脚本回退使用 ALIYUN_OSS_BUCKET。
|
||||
GENARRATIVE_DATABASE_BACKUP_DATA_DIR=""
|
||||
GENARRATIVE_DATABASE_BACKUP_WORK_DIR=""
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
name: genarrative-dev-stack-port-routing
|
||||
short_description: 修改 Genarrative 本地 dev 启动端口、代理目标、端口冲突处理时使用。
|
||||
description: 在 Genarrative 中修改 npm run dev / dev:spacetime / dev:api-server / dev:web / dev:admin-web 的本地启动端口、端口可用性探测、端口漂移、SpacetimeDB publish server、api-server 环境变量、Vite 代理目标和后台 admin-web 启动串联时使用。
|
||||
version: 1.0.0
|
||||
description: 在 Genarrative 中修改 npm run dev / dev:spacetime / dev:api-server / dev:bgfilter-worker / dev:web / dev:admin-web 的本地启动端口、端口可用性探测、端口漂移、SpacetimeDB publish server、Rust 进程环境变量、Vite 代理目标和后台 admin-web 启动串联时使用。
|
||||
version: 1.1.0
|
||||
author: Hermes Agent
|
||||
license: MIT
|
||||
metadata:
|
||||
@@ -13,7 +13,7 @@ metadata:
|
||||
|
||||
# Genarrative 本地 dev 启动端口与代理目标串联流程
|
||||
|
||||
用于维护 Genarrative 本地开发栈启动脚本,重点覆盖 `npm run dev` 与四个 `dev:*` 单模块命令的端口检查、端口漂移和后续流程目标传递。
|
||||
用于维护 Genarrative 本地开发栈启动脚本,重点覆盖 `npm run dev` 与五个 `dev:*` 单模块命令的端口检查、端口漂移和后续流程目标传递。
|
||||
|
||||
## 适用场景
|
||||
|
||||
@@ -31,40 +31,44 @@ metadata:
|
||||
2. Rust `api-server`:`8082`,健康检查为 `http://127.0.0.1:<api-port>/healthz`。
|
||||
3. SpacetimeDB standalone:`3101`,健康检查为 `http://127.0.0.1:<spacetime-port>/v1/ping`。
|
||||
4. 后台 Vite:`3102`,后台地址为 `http://127.0.0.1:<admin-web-port>/admin/`。
|
||||
5. 独立 BgFilter worker:`8083`,就绪检查为 `http://127.0.0.1:<bgfilter-worker-port>/readyz`。
|
||||
|
||||
端口不可用时,脚本会从优先端口开始向后寻找可用端口。后续流程必须以解析后的实际端口为准,不能继续使用默认端口。
|
||||
|
||||
Linux 多用户并发开发时,`GENARRATIVE_DEV_PORT_RANGE` 或 `--port-range` 会先向系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json` 申请一个端口段,再把该段映射为 `web = start`、`api = start + 1`、`spacetime = start + 2`、`adminWeb = start + 3`。注册表锁文件是 `/var/tmp/genarrative-dev-port-ranges/registry.lock`,可通过 `GENARRATIVE_DEV_PORT_RANGE_REGISTRY_DIR` 覆盖目录。自动分配从 `10000-10099` 起,每次占用 100 个端口块,后续块按 `10100-10199`、`10200-10299` 递增;当前口径是“一个用户固定占用一个段,后续启动继续复用这段并在段内漂移”;该注册表只在 Linux 上生效;Windows 继续沿用原有端口探测、漂移和复用逻辑,不读系统级注册表。
|
||||
Linux 多用户并发开发时,`GENARRATIVE_DEV_PORT_RANGE` 或 `--port-range` 会先向系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json` 申请一个端口段,再把该段映射为 `web = start`、`api = start + 1`、`spacetime = start + 2`、`adminWeb = start + 3`、`bgfilterWorker = start + 4`。注册表锁文件是 `/var/tmp/genarrative-dev-port-ranges/registry.lock`,可通过 `GENARRATIVE_DEV_PORT_RANGE_REGISTRY_DIR` 覆盖目录。自动分配从 `10000-10099` 起,每次占用 100 个端口块,后续块按 `10100-10199`、`10200-10299` 递增;当前口径是“一个用户固定占用一个段,后续启动继续复用这段并在段内漂移”;该注册表只在 Linux 上生效;Windows 继续沿用原有统一端口探测和漂移逻辑,不读系统级注册表。
|
||||
|
||||
## 实现入口
|
||||
|
||||
- `package.json`
|
||||
- `dev`:执行 `node scripts/dev.mjs`,启动完整四模块。
|
||||
- `dev:spacetime` / `dev:api-server` / `dev:web` / `dev:admin-web`:执行 `node scripts/dev.mjs <module>`。
|
||||
- `dev`:执行 `node scripts/dev.mjs`,启动完整五服务。
|
||||
- `dev:spacetime` / `dev:api-server` / `dev:bgfilter-worker` / `dev:web` / `dev:admin-web`:执行 `node scripts/dev.mjs <module>`;`dev:api-server` 会安全带起其依赖的 BgFilter worker。
|
||||
- `scripts/dev-stack-port-utils.mjs`
|
||||
- `isPortAvailable(...)`:探测端口是否可监听。
|
||||
- `findAvailablePort(...)`:从优先端口向后寻找可用端口,`0` 表示申请临时端口。
|
||||
- `resolveDevStackPorts(...)`:一次性解析 SpacetimeDB、api-server、主站 Vite、后台 Vite 端口,并避免本次解析结果互相冲突。
|
||||
- `resolveDevStackPorts(...)`:一次性解析 SpacetimeDB、api-server、主站 Vite、后台 Vite、BgFilter worker 端口,并避免本次解析结果互相冲突。
|
||||
- Linux 注册表分配:`reserveLinuxDevPortRange(...)` / `releaseLinuxDevPortRange(...)`,仅在 Linux 上启用系统级端口段登记与用户段复用,自动分配从 `10000-10099` 起。
|
||||
- CLI 模式:`node scripts/dev-stack-port-utils.mjs resolve-dev-stack spacetime:127.0.0.1:3101 api:127.0.0.1:8082 web:0.0.0.0:3000 adminWeb:127.0.0.1:3102`。
|
||||
- CLI 模式:`node scripts/dev-stack-port-utils.mjs resolve-dev-stack spacetime:127.0.0.1:3101 api:127.0.0.1:8082 web:0.0.0.0:3000 adminWeb:127.0.0.1:3102 bgfilterWorker:127.0.0.1:8083`。
|
||||
- `scripts/dev.mjs`
|
||||
- 解析 CLI 参数后统一计算 client host、端口、`SPACETIME_SERVER`、`RUST_SERVER_TARGET`。
|
||||
- 完整栈按 SpacetimeDB、publish、api-server、主站 Vite、后台 Vite 顺序启动。
|
||||
- Linux 下会先申请系统级端口段并把它映射成四个 dev 端口;自动分配从 `10000-10099` 起,Windows 则直接沿用原有参数解析与端口漂移逻辑。
|
||||
- 完整栈按 SpacetimeDB、publish、BgFilter worker readiness、api-server readiness、主站 Vite、后台 Vite 顺序启动。
|
||||
- Linux 下会先申请系统级端口段并把它映射成五个 dev 端口;自动分配从 `10000-10099` 起,Windows 则把第五个服务纳入原有统一参数解析与端口漂移逻辑。
|
||||
- 完整栈和 `dev:api-server` 把两个 Rust 进程作为同一重启单元,先全部停止,再先启动 BgFilter worker、后启动 api-server;不要为同一份 Rust 源码创建两个并发 `cargo` watcher。
|
||||
- 单模块命令复用同一套参数和 env 解析。
|
||||
|
||||
## 必须保持的传递链路
|
||||
|
||||
`npm run dev` 和四个 `dev:*` 单模块命令中端口解析后,必须同步到以下位置:
|
||||
`npm run dev` 和五个 `dev:*` 单模块命令中端口解析后,必须同步到以下位置:
|
||||
|
||||
1. SpacetimeDB 启动:`spacetime start --listen-addr "${SPACETIME_HOST}:${SPACETIME_PORT}"`。
|
||||
2. SpacetimeDB 发布:`spacetime publish ... --server "${SPACETIME_SERVER}"`。
|
||||
3. Rust api-server:`GENARRATIVE_API_HOST`、`GENARRATIVE_API_PORT`、`GENARRATIVE_SPACETIME_SERVER_URL`、`GENARRATIVE_SPACETIME_DATABASE`。
|
||||
4. api-server 健康检查:`wait_for_api_server "${RUST_SERVER_TARGET}/healthz" ...`。
|
||||
5. 主站 Vite:`RUST_SERVER_TARGET`、`GENARRATIVE_RUNTIME_SERVER_TARGET`、`ADMIN_WEB_TARGET`、`ADMIN_WEB_PORT`、`--port=${WEB_PORT}`、`--host=${WEB_HOST}`。
|
||||
6. 后台 Vite:`ADMIN_API_TARGET`、`GENARRATIVE_API_TARGET`、`GENARRATIVE_API_PORT`、`--port=${ADMIN_WEB_PORT}`。
|
||||
7. 控制台日志:`[dev:ports]` 和 `[dev] web/admin web/api-server/spacetime` 必须显示最终实际地址。
|
||||
8. Linux 端口段注册:`[dev] port-range:` 与 `[dev] port-range-registry:` 只在 Linux 输出,Windows 不应依赖系统级注册表。
|
||||
5. BgFilter worker:`GENARRATIVE_PROCESS_ROLE=bgfilter-worker`、解析后的 `HOST / PORT`、与父 API 相同的 `GENARRATIVE_BGFILTER_WORKER_BASE_URL` / `GENARRATIVE_BGFILTER_INTERNAL_TOKEN`,以及显式有效的 `N / Q`。
|
||||
6. BgFilter worker readiness:父 API 启动前检查解析后地址的 `/readyz`。
|
||||
7. 主站 Vite:`RUST_SERVER_TARGET`、`GENARRATIVE_RUNTIME_SERVER_TARGET`、`ADMIN_WEB_TARGET`、`ADMIN_WEB_PORT`、`--port=${WEB_PORT}`、`--host=${WEB_HOST}`。
|
||||
8. 后台 Vite:`ADMIN_API_TARGET`、`GENARRATIVE_API_TARGET`、`GENARRATIVE_API_PORT`、`--port=${ADMIN_WEB_PORT}`。
|
||||
9. 控制台日志:`[dev:ports]` 和 `[dev] web/admin web/api-server/bgfilter-worker/spacetime` 必须显示最终实际地址。
|
||||
10. Linux 端口段注册:`[dev] port-range:` 与 `[dev] port-range-registry:` 只在 Linux 输出,Windows 不应依赖系统级注册表。
|
||||
|
||||
如果只改了其中一段,通常会出现:浏览器打开的前端可用,但 `/api/*` 代理到旧端口;后台页面可用但后台 API 失败;SpacetimeDB 启动在新端口但 publish 仍发往旧端口。
|
||||
|
||||
@@ -91,14 +95,14 @@ Linux 多用户并发开发时,`GENARRATIVE_DEV_PORT_RANGE` 或 `--port-range`
|
||||
node --check scripts/dev.mjs
|
||||
npm run test -- scripts/dev-stack-port-utils.test.ts
|
||||
npm run check:encoding
|
||||
node scripts/dev-stack-port-utils.mjs resolve-dev-stack spacetime:127.0.0.1:0 api:127.0.0.1:0 web:0.0.0.0:0 adminWeb:127.0.0.1:0
|
||||
node scripts/dev-stack-port-utils.mjs resolve-dev-stack spacetime:127.0.0.1:0 api:127.0.0.1:0 web:0.0.0.0:0 adminWeb:127.0.0.1:0 bgfilterWorker:127.0.0.1:0
|
||||
```
|
||||
|
||||
端口冲突回归测试建议:
|
||||
|
||||
1. 用测试或临时 Node server 占用某个优先端口。
|
||||
2. 调用 `findAvailablePort`,断言结果大于被占用端口。
|
||||
3. 调用 `resolveDevStackPorts`,断言四个结果互不相同。
|
||||
3. 调用 `resolveDevStackPorts`,断言五个结果互不相同。
|
||||
4. 如果实际启动完整栈,观察控制台:
|
||||
- `[dev:ports] ... 不可用,改用 ...`
|
||||
- `[dev] api-server: http://...:<actual-api-port>`
|
||||
@@ -122,6 +126,7 @@ node scripts/dev-stack-port-utils.mjs resolve-dev-stack spacetime:127.0.0.1:0 ap
|
||||
- [ ] Linux 注册表分配、同用户复用固定段并继续漂移、自动分配从 `10000-10099` 起、Windows bypass 都有测试覆盖。
|
||||
- [ ] `scripts/dev.mjs` 通过 `node --check`。
|
||||
- [ ] `npm run dev` 的 SpacetimeDB、publish、api-server、主站 Vite、后台 Vite 都使用实际端口。
|
||||
- [ ] BgFilter worker 在 api-server 前 ready,父子共享实际 base URL / Token,Rust watch 只触发一次组合重启。
|
||||
- [ ] `npm run dev:web` 在主站端口不可用时能切换到可用端口。
|
||||
- [ ] 文档同步更新 `docs/technical/RUST_LOCAL_AND_REMOTE_DEPLOYMENT_SCRIPTS_2026-04-22.md`。
|
||||
- [ ] 长期踩坑同步更新 `docs/project-memory/shared-memory/pitfalls.md`。
|
||||
|
||||
@@ -44,10 +44,10 @@ npm run dev
|
||||
|
||||
补充说明:
|
||||
|
||||
- `npm run dev` 会启动 SpacetimeDB standalone、Rust `api-server`、主站 Vite 与后台 Vite,适合完整联调。
|
||||
- `npm run dev` 会启动 SpacetimeDB standalone、独立 `bgfilter-worker`、Rust `api-server`、主站 Vite 与后台 Vite,适合完整联调;内部 worker ready 后才启动 API。
|
||||
- 主站默认地址是 `http://127.0.0.1:3000`,后台可从 `http://127.0.0.1:3000/admin/` 进入,也可直连 `http://127.0.0.1:3102`。
|
||||
- 四个模块可独立启动:`npm run dev:spacetime`、`npm run dev:api-server`、`npm run dev:web`、`npm run dev:admin-web`。
|
||||
- 如需自动刷新后端模块,使用 `npm run dev -- --watch`;其中 `spacetime-module` 改动后只会重新发布模块,不会重启 standalone,`api-server` 改动后会重启 Rust 进程。主站和后台前端源码变化交给 Vite 自身 HMR,不由外层 watcher 重启。非 watch 模式下可在 `npm run dev` 终端输入 `rs api-server`、`rs web`、`rs admin-web`、`rs spacetime` 或 `rs all`,其中 `rs spacetime` 也是只重新发布模块。
|
||||
- 五个模块可独立启动:`npm run dev:spacetime`、`npm run dev:api-server`、`npm run dev:bgfilter-worker`、`npm run dev:web`、`npm run dev:admin-web`;其中 `dev:api-server` 会安全带起同 runner 的 BgFilter worker 依赖。
|
||||
- 如需自动刷新后端模块,使用 `npm run dev -- --watch`;其中 `spacetime-module` 改动后只会重新发布模块,不会重启 standalone,Rust 源码改动会把 `api-server` 与 `bgfilter-worker` 作为一个组合单元重启。主站和后台前端源码变化交给 Vite 自身 HMR,不由外层 watcher 重启。非 watch 模式下可在 `npm run dev` 终端输入 `rs api-server`、`rs bgfilter-worker`、`rs web`、`rs admin-web`、`rs spacetime` 或 `rs all`,其中 `rs spacetime` 也是只重新发布模块。
|
||||
|
||||
构建生产包:
|
||||
|
||||
|
||||
Vendored
+7
-2
@@ -17,6 +17,10 @@ GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
|
||||
GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=600
|
||||
GENARRATIVE_EXTERNAL_GENERATION_WORKER_JOB_TIMEOUT_SECONDS=900
|
||||
GENARRATIVE_EXTERNAL_GENERATION_WORKER_LONG_JOB_TIMEOUT_SECONDS=1800
|
||||
# 父流程只访问同机 BgFilter worker;内部 Token 只通过受保护文件共享,不写明文 env。
|
||||
GENARRATIVE_BGFILTER_WORKER_BASE_URL=http://127.0.0.1:8083
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE=/etc/genarrative/secrets/bgfilter-worker.token
|
||||
GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS=2000
|
||||
GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
|
||||
GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=16
|
||||
GENARRATIVE_API_SHUTDOWN_OUTBOX_FLUSH_TIMEOUT_MS=5000
|
||||
@@ -30,9 +34,10 @@ GENARRATIVE_WALLET_REFUND_OUTBOX_DIR=/var/lib/genarrative/wallet-refund-outbox
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE=100
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS=1000
|
||||
GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES=67108864
|
||||
# 共享的单次 provider attempt 超时上限,父侧据此计算两次逻辑预算;专属 worker env 不重复定义。
|
||||
GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS=180000
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD=3
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS=300
|
||||
GENARRATIVE_EDITOR_BGFILTER_BASE_URL=http://58.87.105.82/bgfilter
|
||||
GENARRATIVE_EDITOR_BGFILTER_TOKEN=
|
||||
# BgFilter 失败后的中间兜底:阿里云通用抠图(SegmentCommonImage)。AccessKey 留空则跳过该层,
|
||||
# BgFilter 失败直接本地 editor_green_screen 去背;填入后恢复 BgFilter→阿里云→本地三级兜底。
|
||||
# AccessKey 也可复用标准 SDK 命名 ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET。
|
||||
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
# 复制到 /etc/genarrative/bgfilter-worker.env;只放专用进程独占参数和可选日志覆盖。
|
||||
# provider、OSS、内部 Token 文件和请求 timeout 统一来自先加载的 api-server.env,禁止在此重复定义。
|
||||
# systemd unit 会强制设置 GENARRATIVE_PROCESS_ROLE=bgfilter-worker。
|
||||
|
||||
GENARRATIVE_ENV=production
|
||||
GENARRATIVE_BGFILTER_WORKER_HOST=127.0.0.1
|
||||
GENARRATIVE_BGFILTER_WORKER_PORT=8083
|
||||
GENARRATIVE_BGFILTER_WORKER_CONCURRENCY=4
|
||||
GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS=128
|
||||
# flat 熔断只由本进程维护;complex 不读写熔断状态。
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD=3
|
||||
GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS=300
|
||||
|
||||
GENARRATIVE_API_LOG=info,tower_http=info
|
||||
GENARRATIVE_OTEL_ENABLED=true
|
||||
OTEL_SERVICE_NAME=genarrative-bgfilter-worker
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=genarrative
|
||||
Vendored
+1
@@ -2,6 +2,7 @@
|
||||
# 默认不启用 Pingora shadow 巡检;只有同时配置 base URL 与 probe token 才会检查。
|
||||
|
||||
GENARRATIVE_HEALTH_PATROL_API_BASE_URL=http://127.0.0.1:8082
|
||||
GENARRATIVE_HEALTH_PATROL_BGFILTER_BASE_URL=http://127.0.0.1:8083
|
||||
GENARRATIVE_HEALTH_PATROL_SPACETIME_BASE_URL=http://127.0.0.1:3101
|
||||
GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL=http://127.0.0.1
|
||||
# 默认公网入口仍按 Nginx 巡检;Pingora 直连切换后改为 pingora-direct。
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
[Unit]
|
||||
Description=Genarrative BgFilter Worker
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=genarrative
|
||||
Group=genarrative
|
||||
WorkingDirectory=/opt/genarrative/current
|
||||
EnvironmentFile=/etc/genarrative/api-server.env
|
||||
EnvironmentFile=/etc/genarrative/bgfilter-worker.env
|
||||
Environment="LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib"
|
||||
ExecStart=/usr/bin/env GENARRATIVE_PROCESS_ROLE=bgfilter-worker OTEL_SERVICE_NAME=genarrative-bgfilter-worker /opt/genarrative/current/api-server
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
KillSignal=SIGINT
|
||||
# 内部 requestBudgetMs 协议上限为 600s;额外窗口用于响应发送和进程收口。
|
||||
TimeoutStopSec=900
|
||||
LimitNOFILE=65535
|
||||
TasksMax=2048
|
||||
|
||||
# 固定 loopback 地址与非模板 unit 共同保证首版同机只运行一个 BgFilter worker。
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=full
|
||||
ReadWritePaths=/opt/genarrative /var/lib/genarrative
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -20,8 +20,9 @@
|
||||
|
||||
- 背景:角色动画在单个 `external_generation_job` 内通过 `buffer_unordered(frame_count)` 可并发发射最多 `48` 次 BgFilter 请求;限制父 worker 并发不能限制单个父 job 内的实际 BgFilter 并发。父 job checkpoint / continuation 和 SpacetimeDB 持久子任务都会扩大父状态机、attempt、计费、恢复和清理改动,而当前 BgFilter 成功结果本来就是 HTTP 图片二进制。
|
||||
- 决策:父 future 保持原调用栈、lease 和 attempt,等待期间继续占用通用 worker 槽并由现有 heartbeat 续租;所有调用统一同步请求唯一 `bgfilter-worker` 的内部 loopback HTTP。输入只传 OSS object key、参数和剩余预算,成功直接返回经过校验的图片二进制。子 worker 使用有界 admission `Q` 和进程内 `Semaphore(N)`,负责最多两次顺序 provider attempt、flat 进程级熔断和失败审计;父流程继续负责 flat 降级、complex 失败、Alpha / 尺寸恢复、动画 finalizer、最终 OSS、业务写回、计费和父终态。
|
||||
- 超时边界:父 job 总预算仍为普通 `900s` / 长任务 `1800s`,并保留现有 `60s` 终态写回窗口;内部 RPC 总预算包含等待 semaphore、两次 attempt、结果校验和二进制返回,子 worker 的单次 provider timeout 默认 `180s`。动画删除旧的 `2000ms × frame_count` 单次 timeout 增量。父侧不得在内部 timeout / 断连后重试整次 RPC。
|
||||
- 超时边界:父 job 总预算仍为普通 `900s` / 长任务 `1800s`,并保留现有 `60s` 终态写回窗口;`180s` 仅是子 worker 单次真实 provider attempt 的默认上限,父侧据此派生 `2 × attempt timeout + 1s response window` 的逻辑调用上限,再按父绝对 deadline 和 flat fallback reserve 截短。内部 RPC 总预算包含 admission 后的 semaphore 等待、最多两次 attempt、结果校验和二进制返回。动画删除旧的 `2000ms × frame_count` 单次 timeout 增量,父侧不得在内部 timeout / 断连后重试整次 RPC。
|
||||
- 故障边界:首版不新增 `bgfilter_task_group`、`bgfilter_request_task`、raw OSS、checkpoint、continuation、数据库 capacity slot、共享熔断或 QPS token bucket。父或子进程崩溃、RPC 丢失时不查询、不恢复结果;父 job 沿用现有 lease / `max_attempts=1` 失败退款语义。动画首版保持所有已提交帧 collect / drain,不增加跨帧取消组。
|
||||
- 部署边界:专用进程首版仍复用完整 `AppState`,因此 systemd unit 先加载共享 `/etc/genarrative/api-server.env`,再加载 `/etc/genarrative/bgfilter-worker.env` 覆盖 worker 独占参数;共享 `GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS` 必须在父子进程保持同一有效值,flat 熔断 threshold / cooldown 只归子 worker。发布切换前校验父子使用同一个非空、非符号链接、`root:genarrative 0440` 的内部 Token 文件;拒绝 `external-generation-worker.env` 覆盖出不同的内部 URL、Token 文件、connect timeout、provider timeout、OSS bucket 或 endpoint(同 bucket 的独立 AK 允许),并要求父 base URL、子 `HOST / PORT` 与 readiness URL 指向同一 loopback endpoint。worker unit 使用 `TimeoutStopSec=900` 覆盖最大 `600s` 请求预算的优雅排空,运行期巡检同时检查唯一 worker unit active 与 loopback readiness。本地 `npm run dev` 同样启动独立子进程并解析第五个 dev 端口,`ProcessRole::All` 不内嵌 listener。
|
||||
- 影响范围:后续实现涉及 `api-server` 内部 HTTP client、专用 `bgfilter-worker` listener / process role、并发与超时配置、部署和运维观测;不修改 SpacetimeDB schema、父 job schema、用户任务 DTO、任务列表或收费归属。
|
||||
- 验证方式:`48` 帧并发进入父 future 时,健康唯一子 worker 进程持有的 BgFilter HTTP future 峰值不得超过生产显式配置的 `N`,且 `queued + running` 不超过 `Q`;覆盖 flat / complex 降级矩阵、内部 deadline、断连后已启动请求排空、动画全帧 drain、External v1 / inline 旁路扫描、二进制大小 / MIME / 尺寸门禁、单实例部署、DDD、编码和 diff 门禁。
|
||||
- 关联文档:`docs/technical/【后端架构】BgFilter受限资源调度方案-2026-07-21.md`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`、`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`。
|
||||
@@ -1730,7 +1731,7 @@
|
||||
## 2026-05-30 Linux 本地 dev 端口段按系统级注册表分配
|
||||
|
||||
- 背景:同一台 Linux 开发机上有多个用户同时跑 `npm run dev` 时,单纯靠各自 `GENARRATIVE_DEV_PORT_RANGE` 容易撞段,且同一用户并发起两个 dev 会话时也会把相同端口段重复拿走。
|
||||
- 决策:Linux 上的本地 dev 端口段分配统一收口到系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json`,锁文件为 `/var/tmp/genarrative-dev-port-ranges/registry.lock`,可通过 `GENARRATIVE_DEV_PORT_RANGE_REGISTRY_DIR` 覆盖目录。未手动指定时自动从 `10000-10099` 开始按 100 端口块分配,后续块按 `10100-10199`、`10200-10299` 递增;端口段映射固定为 `web = start`、`api = start + 1`、`spacetime = start + 2`、`admin-web = start + 3`;注册表会拒绝不同用户的相同或重叠段,并让同一用户后续启动继续复用自己已占用的固定段。`GENARRATIVE_DEV_PORT_RANGE` 与 `--port-range` 仍可手动指定端口段,但只在 Linux 生效,Windows 继续沿用原有端口探测与漂移逻辑,不读注册表。
|
||||
- 决策:Linux 上的本地 dev 端口段分配统一收口到系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json`,锁文件为 `/var/tmp/genarrative-dev-port-ranges/registry.lock`,可通过 `GENARRATIVE_DEV_PORT_RANGE_REGISTRY_DIR` 覆盖目录。未手动指定时自动从 `10000-10099` 开始按 100 端口块分配,后续块按 `10100-10199`、`10200-10299` 递增;端口段最初映射为 `web = start`、`api = start + 1`、`spacetime = start + 2`、`admin-web = start + 3`,2026-07-21 按顶部 BgFilter 决策扩展 `bgfilter-worker = start + 4`;注册表会拒绝不同用户的相同或重叠段,并让同一用户后续启动继续复用自己已占用的固定段。`GENARRATIVE_DEV_PORT_RANGE` 与 `--port-range` 仍可手动指定端口段,但只在 Linux 生效,Windows 继续沿用统一端口探测与漂移逻辑,不读注册表。
|
||||
- 影响范围:`scripts/dev-stack-port-utils.mjs`、`scripts/dev.mjs`、`scripts/dev-stack-port-utils.test.ts`、`scripts/dev.test.ts`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`、本条决策记录、`development-workflow.md`。
|
||||
- 验证方式:`node --check scripts/dev-stack-port-utils.mjs`、`node --check scripts/dev.mjs`、`node node_modules/vitest/vitest.mjs run scripts/dev-stack-port-utils.test.ts scripts/dev.test.ts` 通过;Linux 下能看到 `[dev] port-range:` 与 `registry.json` 路径日志,自动分配从 `10000-10099` 起步,Windows 不出现注册表分配日志。
|
||||
- 关联文档:`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
|
||||
|
||||
@@ -51,18 +51,19 @@ npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Linux 多用户共享同一台机器开发时,本地 dev 脚本会为当前 Linux 用户分配一个固定端口段并写入系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json`,自动分配从 `10000-10099` 开始,每段 100 个端口,四个 dev 服务依次使用 `start` 到 `start + 3`。可用 `GENARRATIVE_DEV_PORT_RANGE` 或 `npm run dev -- --port-range` 手动指定端口段用于特殊场景;注册表会阻止不同用户使用相同或重叠段,并让同一用户后续启动继续复用自己已占用的固定段。该机制只在 Linux 生效,Windows 仍沿用原有端口探测与漂移逻辑。
|
||||
Linux 多用户共享同一台机器开发时,本地 dev 脚本会为当前 Linux 用户分配一个固定端口段并写入系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json`,自动分配从 `10000-10099` 开始,每段 100 个端口,五个 dev 服务依次使用 `start` 到 `start + 4`,其中 BgFilter worker 固定为 `start + 4`。可用 `GENARRATIVE_DEV_PORT_RANGE` 或 `npm run dev -- --port-range` 手动指定端口段用于特殊场景;注册表会阻止不同用户使用相同或重叠段,并让同一用户后续启动继续复用自己已占用的固定段。该机制只在 Linux 生效,Windows 把第五个服务纳入原有统一端口探测与漂移逻辑。
|
||||
|
||||
本地 `npm run dev`、`npm run dev:spacetime` 和 `npm run dev:api-server` 会在 Rust 子进程环境中绕过项目默认 `sccache` wrapper,避免损坏的本机 cache daemon 阻断 `spacetime publish` 或 `api-server` 启动;显式设置的非 sccache 自定义 wrapper 会被保留。生产 / Jenkins 构建仍按流水线自身的 sccache 策略执行。
|
||||
本地 `npm run dev`、`npm run dev:spacetime`、`npm run dev:api-server` 和 `npm run dev:bgfilter-worker` 会在 Rust 子进程环境中绕过项目默认 `sccache` wrapper,避免损坏的本机 cache daemon 阻断 `spacetime publish` 或 Rust 服务启动;显式设置的非 sccache 自定义 wrapper 会被保留。生产 / Jenkins 构建仍按流水线自身的 sccache 策略执行。
|
||||
|
||||
该命令会启动:
|
||||
|
||||
- SpacetimeDB standalone
|
||||
- 独立 `bgfilter-worker`
|
||||
- Rust `api-server`
|
||||
- 主站 Vite
|
||||
- 后台 Vite
|
||||
|
||||
`npm run dev` 和单模块 `dev:*` 命令会更新根目录 `.app/dev-stack.json`,记录四个本地服务的 pid、端口、URL、启动状态和当前命令。该目录只作本机运行态观测,不提交 Git。
|
||||
`npm run dev` 和单模块 `dev:*` 命令会更新根目录 `.app/dev-stack.json`,记录五个本地服务的 pid、端口、URL、启动状态和当前命令。该目录只作本机运行态观测,不提交 Git。
|
||||
|
||||
开启自动刷新:
|
||||
|
||||
@@ -70,9 +71,9 @@ Linux 多用户共享同一台机器开发时,本地 dev 脚本会为当前 Li
|
||||
npm run dev -- --watch
|
||||
```
|
||||
|
||||
watch 模式只由外层调度器自动处理后端侧刷新:`spacetime-module` 改动后重新发布模块但不重启 standalone 宿主,`api-server` 改动后重启 Rust 进程。主站 Vite 与后台 Vite 的源码变化交给 Vite 自身 HMR,避免外层 watcher 监听到依赖缓存或临时文件后循环重启。
|
||||
watch 模式只由外层调度器自动处理后端侧刷新:`spacetime-module` 改动后重新发布模块但不重启 standalone 宿主;完整栈和 `dev:api-server` 只创建一套 Rust watcher,改动后先停止 API 与 BgFilter worker,再先启动并验活 worker、最后启动并验活 API。主站 Vite 与后台 Vite 的源码变化交给 Vite 自身 HMR,避免外层 watcher 监听到依赖缓存或临时文件后循环重启。
|
||||
|
||||
非 watch 模式下,`npm run dev` 终端支持输入 `rs spacetime`、`rs api-server`、`rs web`、`rs admin-web` 或 `rs all`。其中 `rs spacetime` 只会重新发布 `spacetime-module`,不会重启 standalone 宿主;其他模块仍按进程重启。
|
||||
非 watch 模式下,`npm run dev` 终端支持输入 `rs spacetime`、`rs api-server`、`rs bgfilter-worker`、`rs web`、`rs admin-web` 或 `rs all`。其中 `rs spacetime` 只会重新发布 `spacetime-module`,不会重启 standalone 宿主;重启任一 Rust 角色都会走 API / BgFilter worker 组合重启。
|
||||
|
||||
单独启动 SpacetimeDB:
|
||||
|
||||
@@ -86,6 +87,12 @@ npm run dev:spacetime
|
||||
npm run dev:api-server
|
||||
```
|
||||
|
||||
该命令会由同一 runner 自动带起独立 BgFilter worker,确保共享实际内部 base URL 和 Token。只单独启动内部 worker 时使用:
|
||||
|
||||
```bash
|
||||
npm run dev:bgfilter-worker
|
||||
```
|
||||
|
||||
单独启动前端:
|
||||
|
||||
```bash
|
||||
@@ -107,7 +114,7 @@ npm run server-manager:panel
|
||||
该命令启动 `server-rs/crates/server-manager-panel` 的 egui 桌面工具,从本机 `~/.ssh/config` 读取可用 `Host` alias,支持多服务器健康巡检、可折叠侧边栏和受控 systemd 服务启停。服务操作通过远端 `sudo -n systemctl start|stop|restart <unit>` 执行,目标服务器需要提前配置对应 unit 的免交互 sudo 权限。
|
||||
面板启动时会自动注入本机中文字体;如开发机中文仍显示为方块,可设置 `GENARRATIVE_SERVER_PANEL_CJK_FONT=/path/to/font.ttc|index` 指向本机 CJK 字体。
|
||||
|
||||
`npm run dev:api-server` 会保留终端实时输出,并把同一份输出持久化到 `logs/api-server/api-server-<timestamp>.log`。完整联调入口 `npm run dev` 启动的 Rust `api-server` 使用同一套日志规则。如需改写路径,可设置 `GENARRATIVE_API_SERVER_LOG_FILE`;如只改目录,可设置 `GENARRATIVE_API_SERVER_LOG_DIR`。
|
||||
`npm run dev:api-server` 会保留终端实时输出,并把 API 输出持久化到 `logs/api-server/api-server-<timestamp>.log`、BgFilter worker 输出持久化到 `logs/bgfilter-worker/bgfilter-worker-<timestamp>.log`。完整联调入口 `npm run dev` 使用同一套日志规则。API 日志可通过 `GENARRATIVE_API_SERVER_LOG_FILE` / `GENARRATIVE_API_SERVER_LOG_DIR` 改写,worker 日志可通过 `GENARRATIVE_BGFILTER_WORKER_LOG_FILE` / `GENARRATIVE_BGFILTER_WORKER_LOG_DIR` 改写。
|
||||
|
||||
开发态 `npm run dev` / `npm run dev:api-server` 默认打开 `GENARRATIVE_DEV_PASSWORD_ENTRY_AUTO_REGISTER_ENABLED=true`,密码入口可以直接注册未知手机号账号;生产默认仍关闭该开关。
|
||||
|
||||
|
||||
@@ -3254,3 +3254,11 @@
|
||||
- 现象:VectorEngine 单次请求超时大于 worker job 执行预算时,worker 已停止续租,provider 才超时或开始重试;最终 lease 过期、任务失败并退款,上游却可能继续消耗资源或迟到成功。
|
||||
- 原因:单 attempt timeout、重试退避、图片下载与 worker / lease 分别使用独立的相对计时,没有共享同一绝对 deadline;只抬高 worker timeout 或单独压低 provider timeout 都无法保证留出终态写回窗口。
|
||||
- 处理:实际调用 VectorEngine 的四类图片 job 使用 `1800s` long 预算;从 job 开始的同一起点派生 provider deadline,常规提前 `60s`、短预算提前一半。每次 attempt、退避、下一次 attempt 和图片下载都必须在该 deadline 内;普通 HTTP / `inline` 不伪造 worker deadline。修复时不改动 lease fencing、迟到写回仲裁和原子退款语义。
|
||||
|
||||
## 同一 Rust 二进制的本地双进程不能各自并发 watch 重启(2026-07-21)
|
||||
|
||||
- 现象:本地把 `api-server` 与独立 `bgfilter-worker` 都用 `cargo run -p api-server` 启动后,一次 Rust 源码变更触发两套 watcher 并发停止、编译和链接;Windows 常因另一个实例仍占用 `api-server.exe` 而链接失败,或出现 API 已恢复但内部 worker 尚未 ready 的半更新状态。
|
||||
- 原因:两个进程角色共享同一 crate、target 和可执行文件,却被错误地当成两个互不相关的 dev service。更危险的是先启动 `GENARRATIVE_PROCESS_ROLE=all` 的 API:它会立即消费外部生成队列,可能在内部 BgFilter worker 尚未 ready 时领取任务。
|
||||
- 处理:`npm run dev` 与 `npm run dev:api-server` 只创建一套 Rust watcher,并把两个进程作为组合重启单元:先停止 API 与 BgFilter worker,再只让 worker 的 `cargo run` 完成必要构建,等待 worker `/readyz`,最后启动并验活 API。交互 `rs api-server`、`rs bgfilter-worker` 在完整栈内也必须走同一组合重启。`ProcessRole::All` 永远不内嵌 BgFilter listener;父子进程共享解析后的内部 base URL / Token,Linux 第五端口固定为端口段 `start + 4`,Windows 把第五端口纳入统一探测和漂移。
|
||||
- 验证:定向测试断言组合重启顺序为“stop API → stop worker → start/ready worker → start/ready API”,`dev:api-server` 自动带起同 runner worker,端口解析得到五个互不冲突的端口;再运行 `node --check scripts/dev.mjs`、dev-stack 定向测试和编码检查。
|
||||
- 关联:`scripts/dev.mjs`、`scripts/dev-stack-port-utils.mjs`、`.hermes/skills/genarrative-dev-stack-port-routing/SKILL.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
更新时间:`2026-07-21`
|
||||
|
||||
状态:`待实施`
|
||||
状态:`已实施,待生产压测`
|
||||
|
||||
> 本文替代此前讨论的“SpacetimeDB 持久子任务 + raw 结果 OSS”以及更早的“父 job checkpoint / continuation”方案。首版改为父流程在原调用栈内同步等待唯一 `bgfilter-worker` 的内部 HTTP 响应。当前代码仍由各调用方直接请求 BgFilter,本文描述目标实现。
|
||||
> 本文替代此前讨论的“SpacetimeDB 持久子任务 + raw 结果 OSS”以及更早的“父 job checkpoint / continuation”方案。首版改为父流程在原调用栈内同步等待唯一 `bgfilter-worker` 的内部 HTTP 响应。代码与部署接线已经实施;完成本文生产压测和验收门禁前,不视为可上线实现。
|
||||
|
||||
## 1. 决策摘要
|
||||
|
||||
@@ -13,8 +13,8 @@
|
||||
| 调度单位 | 一次逻辑 BgFilter 调用;角色动画为单帧 |
|
||||
| 父流程 | 保持原 future、调用栈、lease 和 `attempt`,同步等待内部 HTTP |
|
||||
| 通用 worker 槽 | 等待期间继续占用;父 heartbeat 继续运行 |
|
||||
| 输入 | 只传私有 OSS `objectKey`、BgFilter 参数、剩余预算和有界审计关联;不传源图字节或签名 URL |
|
||||
| 成功输出 | 内部 HTTP body 直接返回图片二进制;不使用 Base64,不先写 raw OSS |
|
||||
| 请求输入(父 → 子) | 只传私有 OSS `objectKey`、BgFilter 参数、剩余预算和有界审计关联;不重复传源图字节,也不传签名 URL |
|
||||
| 成功输出(子 → 父) | 内部 HTTP body 直接传回 BgFilter 结果图片的原始字节;不使用 Base64、不返回结果 object key、不先写 raw OSS |
|
||||
| BgFilter worker | 首版只运行一个内部 HTTP worker 实例 |
|
||||
| 并发 | 进程内 `Semaphore(N)`,并增加有界 admission 上限 `Q` |
|
||||
| 重试 | 子 worker 对一次逻辑调用最多做两次顺序 provider attempt;父侧不重试整次内部 RPC |
|
||||
@@ -24,6 +24,7 @@
|
||||
| 动画失败 | 首版保持当前“所有已提交帧都等待并排空”语义,不新增跨帧取消组 |
|
||||
| 崩溃恢复 | 不查询、不恢复 BgFilter 结果;父 job 沿用现有 lease、失败和退款语义 |
|
||||
| 数据模型 | 不新增 SpacetimeDB 表,不修改 `external_generation_job` schema |
|
||||
| 配置加载 | 子 worker 先加载 API 基础环境,再加载 worker 专属环境覆盖;共享超时保持单一来源 |
|
||||
|
||||
首版明确不实现:
|
||||
|
||||
@@ -42,6 +43,8 @@
|
||||
|
||||
编辑器 queue job 固定 `max_attempts = 1`。普通 requeue 会改变 attempt、失败和退款语义;checkpoint / continuation 又会扩大父状态机和计费恢复改动。父流程既然可以接受继续占用 worker 槽,首版无需为 BgFilter 建第二套持久任务系统。
|
||||
|
||||
checkpoint / continuation 不是当前已有能力,而是旧版方案需要新增的恢复状态机;同步原地等待版不新增它们。旧版 `bgfilter_task_group` 用于聚合动画帧,`bgfilter_request_task` 用于持久调度一次逻辑调用(动画时为单帧);本版继续由父 future 聚合帧结果,由内部 HTTP handler、`Q` admission 和 `Semaphore(N)` 调度单次调用,因此两张表都不再需要。
|
||||
|
||||
当前 BgFilter 成功结果本来就是 HTTP 图片二进制,调用方读取后再由父流程做最终处理和 OSS 持久化。因此让专用 worker 通过内部 HTTP 直接返回二进制,最接近现有数据流。
|
||||
|
||||
### 2.2 目标
|
||||
@@ -91,6 +94,26 @@ flowchart LR
|
||||
|
||||
`bgfilter-worker` 从实现形态看是只监听内部地址的同步 worker service,不是队列 consumer。父 worker 调另一个 worker 在这里是允许的:父进程明确选择保留调用栈和槽位,因此同步内部 HTTP 正是首版的最小交接方式。
|
||||
|
||||
这里的“同步等待”是控制流上的 request / response `await`:不会阻塞 OS 执行线程或整个父进程,但父 job future 仍留在通用 worker 的并发集合中,占用一个父 worker 槽,并由现有 heartbeat 继续续租。
|
||||
|
||||
首版进程角色仍复用现有完整 `AppState` 构造路径,以获得 OSS、BgFilter provider、SpacetimeDB 审计、HTTP client 和可观测性依赖;进程角色只阻止它挂载公共路由、claim 外部生成 job 或启动其它后台循环,并不等于它只需要 `N / Q` 几个环境变量。因此生产 unit 必须先加载 `/etc/genarrative/api-server.env`,再加载 `/etc/genarrative/bgfilter-worker.env` 覆盖监听地址、`N / Q` 和 worker 独占参数。后续若拆出轻量专用 state,可再缩小共享配置依赖,首版不能假设该拆分已经存在。
|
||||
|
||||
### 3.1 图片数据流口径
|
||||
|
||||
请求和响应采用不同口径,不能把“请求不传源图字节”理解成“响应也不能传图片字节”:
|
||||
|
||||
| 阶段 | 传递内容 | 是否新增持久化 |
|
||||
| --- | --- | --- |
|
||||
| 父流程 → `bgfilter-worker` | JSON:源图 `objectKey`、参数和预算 | 否 |
|
||||
| `bgfilter-worker` → BgFilter provider | 子 worker 现场签发的源图短期 URL | 否 |
|
||||
| BgFilter provider → `bgfilter-worker` | 结果图片字节 | 否,只在子 worker 有界内存中读取和校验 |
|
||||
| `bgfilter-worker` → 父流程 | `2xx` HTTP body 中的原始结果图片字节 | 否,父侧直接读入有界字节缓冲 |
|
||||
| 父流程 → OSS / 业务写回 | 现有后处理后的最终图片 | 是,仍只走父流程现有最终持久化路径 |
|
||||
|
||||
因此,本方案所说的“直接返回二进制”就是直接传图片字节:父侧内部 client 的成功结果是 `Bytes` / `Vec<u8>` 一类有界内存缓冲及可信的图片类型,而不是 Base64 字符串、临时 object key 或子任务结果记录。这里不是把 provider 响应边读边透明转发;子 worker 要先完整读取并校验结果,确认本次 attempt 成功后,再把同一份图片内容作为内部 HTTP body 返回,以保留第二次顺序尝试和无效图片拦截能力。
|
||||
|
||||
输入与输出采用非对称传输是有意设计:源图在调用前已经持久化到私有 OSS,传 `objectKey` 可避免重复上传和跨进程复制大块输入;输出则是父流程马上消费的短生命周期结果,直接用内部 HTTP 二进制 body 返回最小,不需要先制造一份 raw OSS 资产。
|
||||
|
||||
## 4. 内部 HTTP 契约
|
||||
|
||||
### 4.1 请求
|
||||
@@ -128,7 +151,7 @@ Authorization: Bearer <internal-token>
|
||||
- 如果未来确实支持多个 bucket,新增字段也必须由服务端 allowlist 校验;不能接受调用方提供任意下载 URL。
|
||||
- `backgroundMode` 只允许 `flat / complex`;`segModel` 继续沿用当前 `birefnet / anime-seg` allowlist;complex 固定使用当前参数组合。
|
||||
- `screenColor` 只对 flat 必填;complex 不得误接 flat 参数或熔断。
|
||||
- `requestBudgetMs` 是从子 worker 收到请求开始计算的相对预算,不是跨机器绝对时间。
|
||||
- `requestBudgetMs` 是相对预算,不是跨机器绝对时间。当前实现从内部鉴权通过并取得 `Q` admission permit 的时刻起算;`Q` 满时立即返回 `overloaded`,成功 admission 后的 JSON 解析、等待 `N` permit、provider attempt、结果校验和响应构造都消耗该预算。
|
||||
- JSON body 设置很小的固定上限;源图字节不进入该 JSON。
|
||||
|
||||
签名 URL 必须在取得 provider permit 后、每次 attempt 前生成,避免排队期间过期。签名 URL 只存在于子 worker 内存和发往 BgFilter 的请求中。
|
||||
@@ -146,7 +169,7 @@ Content-Type: image/png
|
||||
|
||||
`image/png` 是常见响应示例。为保持当前行为,首版可以返回实际受支持的 `image/png`、`image/webp` 或 `image/jpeg`,但必须保证响应头与实际解码类型一致;不使用 JSON、Data URL 或 Base64 包装,也不为传输先落 raw OSS。
|
||||
|
||||
子 worker 必须完整读取并校验 provider body 后才向父侧返回成功,这样 provider body 中途断开时仍可在预算内执行第二次 attempt。父侧收到二进制后继续执行一次独立校验,不能只信任内部响应头。
|
||||
子 worker 必须完整读取并校验 provider body 后才向父侧返回成功,这样 provider body 中途断开时仍可在预算内执行第二次 attempt。父侧内部 client 对 `2xx` 响应读取有界二进制 body,并把字节直接交回现有 Alpha / 尺寸恢复与 finalizer;父侧仍执行一次独立校验,不能只信任内部响应头。任何一侧都不得把成功 body 转成 Base64、JSON 数组或临时 OSS 引用。
|
||||
|
||||
### 4.3 错误响应
|
||||
|
||||
@@ -189,19 +212,21 @@ HTTP status 只作粗粒度传输分类,父侧以稳定 `error.code` 映射业
|
||||
|
||||
queue job 的总预算从父 job 开始执行时起算,不从开始申请 BgFilter 时重新计时。现有父 worker 还会把 provider deadline 设在 job deadline 前 `60s`,为最终写回和终态保留时间。同步 RPC 实现必须显式读取父侧剩余 provider budget 并传入 `requestBudgetMs`,不能像当前 BgFilter helper 一样忽略 `RequestContext` deadline。
|
||||
|
||||
本版没有 BgFilter 子任务等待 claim 的阶段。几个起算点必须区分:父 job 在数据库中尚未被 claim 的等待不消耗 job 执行预算;父 job 开始实际执行后,生图及 BgFilter 之前的耗时都会消耗父总预算;父内部 HTTP client timeout 从开始发送请求起覆盖 loopback 传输、worker admission、等待 `N`、provider 和回包;单次 `180s` attempt timer 只在子 worker 真正开始一次 BgFilter provider HTTP 时启动。父侧不是放弃超时,而是不再直接执行 provider 单次 attempt 的计时器。
|
||||
|
||||
父总 deadline 到达时,现有 worker 会停止续租、释放 JoinSet 槽并把 work 交给 lease fencing 仲裁,不是立刻杀死所有内部工作。正常情况下内部 RPC 自身应在更早的 provider deadline 内结束,避免进入这条脱管路径。
|
||||
|
||||
### 5.2 子 worker attempt timeout
|
||||
|
||||
父侧与子 worker 的预算必须满足:
|
||||
父侧先按 `requestBudgetMs + 2s` 计算期望的内部 client timeout,再受父绝对 deadline 截断:
|
||||
|
||||
```text
|
||||
requestBudgetMs + 2s parent transport window
|
||||
<= parent client timeout
|
||||
<= 父侧剩余绝对预算
|
||||
parent client timeout
|
||||
= min(requestBudgetMs + 2s parent transport window,
|
||||
父侧当前剩余绝对预算)
|
||||
```
|
||||
|
||||
parent client timeout 从父侧开始发请求时起算,`requestBudgetMs` 从子 worker 收到请求时起算,因此两者不能设成同一个值。额外 `2s` 用于 loopback 传输、调度抖动和父侧读取类型化错误,不增加 provider 可执行时间。
|
||||
`requestBudgetMs` 派生时会预扣这段 transport reserve,因此正常路径仍为子 worker 保留约 `2s` 的 loopback 传输、调度抖动和父侧读取类型化错误时间;父绝对 deadline 始终是硬上限。额外 `2s` 不增加 provider 可执行时间,也不能在父预算已经不足时强行延长 client timeout。
|
||||
|
||||
子 worker 收到请求后使用本机单调时钟计算 RPC deadline。每次 attempt 的 timeout 为:
|
||||
|
||||
@@ -213,6 +238,8 @@ worker 内部 `1s` 和 parent transport `2s` 都只是固定的小型进程 /
|
||||
|
||||
flat 调用还要由父侧从可分配给 BgFilter 的预算中保留当前阿里云 request timeout(默认 `30s`)和少量本地处理余量,避免 BgFilter 排队吃完全部 provider budget 后名义上有 fallback、实际上已无时间执行。complex 没有 flat fallback,不使用该预留。
|
||||
|
||||
`GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS` 是单次真实 provider attempt 的基础上限,默认 `180s`。父侧据此派生一次逻辑调用的上限:`2 × attempt timeout + 1s worker response window`,再按父绝对 deadline 和 flat fallback reserve 截短为 `requestBudgetMs`;parent client timeout 则取前述 `min(requestBudgetMs + 最多 2s transport window, 父绝对 deadline 剩余)`。该配置必须由父子进程使用同一个有效值:首版把它放在共享 API 基础环境中作为单一来源,worker 专属环境不得悄悄覆盖成另一个值。
|
||||
|
||||
inline / External v1 没有 queue job deadline 时,内部 RPC 仍必须有界;默认总上限按“两次现有 BgFilter attempt 上限 + 内部响应窗口”计算,排队时间同样包含在内。
|
||||
|
||||
### 5.3 动画旧增量
|
||||
@@ -271,17 +298,19 @@ inline / External v1 没有 queue job deadline 时,内部 RPC 仍必须有界
|
||||
|
||||
### 6.3 熔断
|
||||
|
||||
熔断是故障保护:flat 连续多次请求失败后,在 cooldown 内暂时不再请求 BgFilter,而是快速返回 `circuit_open`,由父流程进入“阿里云 → 本地”fallback,避免故障 provider 持续占满并发和超时。
|
||||
熔断是故障保护:flat 的真实 provider attempt 连续失败达到阈值后,在 cooldown 内暂时不再请求 BgFilter,而是快速返回 `circuit_open`,由父流程进入“阿里云 → 本地”fallback,避免故障 provider 持续占满并发和超时。
|
||||
|
||||
保持当前语义:
|
||||
|
||||
- 只有 flat 读取和更新熔断;complex 完全不读写。
|
||||
- flat 在取得 permit、即将发送第一次 provider HTTP 前重新检查熔断,避免 48 个排队请求在熔断打开前全部通过旧检查。
|
||||
- 已经获准执行的逻辑调用,即使第一次失败使熔断打开,也仍允许在预算内完成自己的第二次顺序 attempt;后续请求快速返回 `circuit_open`。
|
||||
- 每个真实失败 attempt 计一次失败,保持当前计数口径;成功重置。
|
||||
- 每个真实失败 attempt 计一次失败,保持当前计数口径;flat 任一真实 attempt 成功后重置。
|
||||
- 只有拿到完整配置 attempt 上限(默认 `180s`)后发生的 provider timeout,以及真实传输失败、非 2xx 和无效 / 超限图片计入。因 `requestBudgetMs` 剩余不足而被截短的 timeout 返回 `deadline_exceeded`,不更新熔断;排队满、排队超时、客户端取消、鉴权和本地配置错误同样不计入。
|
||||
- 进程重启后熔断状态清零是首版接受行为。
|
||||
|
||||
flat 熔断的 `GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD` 与 `GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS` 属于 `bgfilter-worker` 运行参数;父 API / external-generation worker 不再读取或更新熔断。生产示例必须把这两个值放进 worker 专属环境,避免运维人员在父侧修改了一个实际不生效的配置。
|
||||
|
||||
首版不增加 QPS 限制。若 provider 以后要求 QPS,需要另加 token bucket;不能把并发 semaphore 当作 QPS。
|
||||
|
||||
## 7. 断连、崩溃与动画语义
|
||||
@@ -335,6 +364,7 @@ BgFilter 成功二进制不是一份新的业务资产:
|
||||
|
||||
- 首版限定父 worker 与子 worker 同机部署,内部 listener 只绑定 loopback 固定端口,不挂公共 Axum router、Nginx、BFF 或 OpenAPI。
|
||||
- 使用独立内部 Token;缺失时生产 fail-closed。Token 不复用 BgFilter provider Token。
|
||||
- 父、子进程只从同一个 `GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE` 读取内部 Token。发布脚本必须在切换 `current` 链接前确认该路径是非符号链接的普通非空文件,owner / group / mode 符合 `root:genarrative 0440`;不能先切换版本、再等 readiness 暴露首次未 provision 或权限错误。
|
||||
- 只接受配置 bucket 下的规范化 object key;禁止 `http://`、`https://`、`data:`、`blob:` 和路径逃逸。
|
||||
- 不在日志、trace、metrics、错误 JSON 或 SpacetimeDB 审计中写签名 URL、Token、图片字节或 Base64。
|
||||
|
||||
@@ -347,7 +377,9 @@ BgFilter 成功二进制不是一份新的业务资产:
|
||||
- MIME、魔数和实际解码结果必须一致;
|
||||
- 空 body、截断 body 和超限图片按 `invalid_result` 处理。
|
||||
|
||||
二进制跨进程传输期间,子 worker 和父 worker 可能同时持有同一张图片。前述 `N / Q` permit 必须由 response-body guard 持有到发送完成或 body drop,才能把子 worker 中的完整成功 body 控制在 `N` 级;在此前提下,极端内存至少按 `2 × N × 32 MiB` 再加解码缓冲评估。若实现没有该 guard,完整 body 可能积累到 `Q` 级,本文的内存模型即不成立,不能上线。生产 `N / Q` 必须结合主机内存压测,而不是只看 BgFilter 吞吐。
|
||||
二进制跨进程传输期间,子 worker 和父 worker 可能同时持有同一张图片。子侧 provider permit、成功 body guard 和图片校验槽均与 `N` 对齐;guard 持有到内部响应发送完成或 body drop,完整成功 body 不会积累到 `Q` 级。父侧另有固定 `P = 4` 个成功图片读取 / 解码槽,必须在开始读取 `2xx` body 前取得,并覆盖有界 body 读取与 `spawn_blocking` 校验。极端完整 body 内存按 `(N + P) × 32 MiB` 再加父子解码缓冲、provider 读取缓冲和运行时开销评估;生产 `N / Q` 必须结合主机内存压测,而不是只看 BgFilter 吞吐。
|
||||
|
||||
图片解码运行在不可强制取消的 blocking task 中。父子两侧等待校验结果都必须受各自 deadline 约束;deadline 到达后请求可按类型化超时收口。子 worker 已启动但尚未结束的校验 task 继续持有图片字节和校验槽,并由 shutdown tracker 等待真实结束;provider `N` 只覆盖真实 provider 调用及内部响应发送,不因后台 CPU 校验延长而虚假占用。父侧超时后的 blocking task 继续持有父侧校验槽直到真实结束,防止后续大图无界叠加,但它没有外部副作用,不纳入子 worker 的 shutdown tracker。`TimeoutStopSec=900` 覆盖的是子 worker 的排空边界。
|
||||
|
||||
### 9.3 指标与日志
|
||||
|
||||
@@ -372,26 +404,36 @@ BgFilter 成功二进制不是一份新的业务资产:
|
||||
1. 在现有 Rust 后端增加 `bgfilter-worker` 进程角色和独立 loopback Axum listener;它不启动用户 HTTP router,也不 claim `external_generation_job`。
|
||||
2. 增加内部 request / binary response / typed error 契约、Token 校验、JSON body 上限、object key allowlist 和健康检查;listener 在 body 解析前接入连接 / request concurrency limit、固定 backlog 和 load shedding。
|
||||
3. 增加 admission `Q`、`Semaphore(N)`、两次顺序 attempt、预算检查、结果限长 / 解码校验、response-body permit guard 和 flat 进程级熔断。
|
||||
4. 增加父侧共享内部 HTTP client。该 client 不自动重试,把父剩余预算显式转换为较短的 `requestBudgetMs` 和略长的 client timeout,并保证两者都早于父绝对 deadline。
|
||||
4. 增加父侧共享内部 HTTP client。该 client 不自动重试;对 `2xx` 读取并返回受限图片字节,对非 `2xx` 只解析有界类型化 JSON 错误;把父剩余预算显式转换为较短的 `requestBudgetMs` 和略长的 client timeout,并保证两者都早于父绝对 deadline。
|
||||
5. 用内部 client 替换两个集中调用边界:
|
||||
- flat:`remove_editor_generated_screen_background_with_bgfilter_with_request_timeout`;
|
||||
- complex:`request_editor_background_removal_image_with_retry`。
|
||||
6. 从父侧移除 BgFilter provider retry 和 flat 熔断实现;保留 flat fallback、complex 失败、Alpha / 尺寸恢复和所有最终持久化。
|
||||
7. 删除动画 `2000ms × frame_count` 单 attempt timeout 增量;保留现有所有帧 collect / drain。
|
||||
8. 扫描 queue、inline、External v1 的角色、图标、UI、动画和手动 complex 路径,确认没有 direct BgFilter HTTP 旁路。
|
||||
9. 增加单实例 systemd unit、内部地址 / Token、`N / Q` 配置、readiness 和指标;真实压测 `32 / 40 / 48` 帧后启用。
|
||||
9. 增加单实例 systemd unit、内部地址 / Token、`N / Q` 配置、readiness 和指标;unit 按“API 基础环境 → worker 专属环境”加载,部署 preflight 校验共享超时和 Token;真实压测 `32 / 40 / 48` 帧后启用。
|
||||
10. 把独立子进程纳入本地 dev 调度器;`ProcessRole::All` 保持不内嵌 BgFilter listener,避免本地与生产形成两套调用实现。
|
||||
|
||||
本计划不产生 SpacetimeDB schema、migration、bindings 或表目录改动。
|
||||
|
||||
### 10.2 部署与回滚
|
||||
|
||||
1. 先部署并启动唯一 `bgfilter-worker`,确认 loopback health、鉴权和 provider smoke。
|
||||
2. 再发布使用内部 client 的 API / external-generation-worker。
|
||||
3. 确认所有父进程只访问内部 endpoint,BgFilter provider 日志中不再出现父进程直连。
|
||||
4. 发布子 worker 时执行 `stop old -> 等待排空/退出 -> start new`,不得滚动重叠。
|
||||
1. 在切换发布目录前完成 preflight:共享 API env 与两类 worker env 均存在;`external-generation-worker.env` 和 `bgfilter-worker.env` 不得把共享 BgFilter 配置覆盖为不同有效值;父 base URL、子 `HOST / PORT` 与 readiness URL 指向同一 loopback endpoint;内部 Token 文件存在、非空、非符号链接且权限正确;`N / Q` 为正整数且 `Q >= N`。
|
||||
2. 安装非模板单实例 unit;它先加载 `/etc/genarrative/api-server.env`,再加载 `/etc/genarrative/bgfilter-worker.env`。
|
||||
3. 执行 `stop old -> 等待排空/退出 -> start new`,确认唯一 `bgfilter-worker` 的 loopback readiness、鉴权和 provider smoke,不得滚动重叠。
|
||||
4. 再重启使用内部 client 的 API / external-generation-worker / controller。
|
||||
5. 确认所有父进程只访问内部 endpoint,BgFilter provider 日志中不再出现父进程直连。
|
||||
|
||||
内部 worker 不可用时禁止自动 direct fallback。flat 仍可走业务已有阿里云 / 本地 fallback;complex 明确失败。需要整体回滚时回滚父、子进程版本和配置,不在运行中混用两种 BgFilter 调度方式。
|
||||
|
||||
`bgfilter-worker` 收到停止信号后先停止接收新请求,再等待已 admission 的 handler 和已启动 provider attempt 排空。内部协议允许的 `requestBudgetMs` 最大为 `600s`,因此 systemd unit 固定使用 `TimeoutStopSec=900`;部署脚本的同步 `systemctl stop` 必须允许该窗口完成,不能沿用 systemd 常见的约 `90s` 默认值强杀在途调用。
|
||||
|
||||
### 10.3 本地开发
|
||||
|
||||
`npm run dev` 必须自动启动独立 `bgfilter-worker` 子进程,并在启动父 API / external-generation 路径前完成 loopback readiness。开发端口解析新增第五个 `bgfilter` 职责:Linux 多用户端口段使用 `start + 4`,Windows 沿用现有端口探测与漂移;解析后的实际 host / port 注入子进程,实际 base URL 注入父进程,不能继续硬编码 `8083`。父、子使用同一个仅存在于本地进程环境的内部 Token。
|
||||
|
||||
`GENARRATIVE_PROCESS_ROLE=all` 仍不直接运行 BgFilter listener。单模块联调需要提供明确的独立 worker 启动入口,并在 `dev:api-server` 没有可用 worker 时自动带起或 fail-fast 给出该入口,不能让开发者等到一次图片生成才看到连接拒绝。dev 状态文件、watch 重启、端口日志和退出清理都要把第五个子进程纳入,防止遗留进程占端口。
|
||||
|
||||
## 11. 验收门禁
|
||||
|
||||
必须覆盖:
|
||||
@@ -399,18 +441,28 @@ BgFilter 成功二进制不是一份新的业务资产:
|
||||
- `48` 帧同时进入时,唯一子 worker 观测到的客户端 BgFilter HTTP future 峰值不超过 `N`。
|
||||
- listener 在 body 解析前执行 concurrency limit / load shedding;`queued + running + egress` 达到 `Q` 后新请求立即返回 `overloaded`,handler 数不超过 `Q`,内核 socket backlog 按独立固定值验证。
|
||||
- 排队时间计入 `requestBudgetMs`;deadline 到达后不开始新的 provider attempt。
|
||||
- `requestBudgetMs + parent transport window <= parent client timeout <= 父剩余绝对预算`,worker 有时间把类型化错误交回父侧。
|
||||
- 不存在 BgFilter 子任务 claim 状态;成功 admission 后等待 `N` 的时间同时计入子 `requestBudgetMs` 和父 client timeout。
|
||||
- parent client timeout 取 `min(requestBudgetMs + parent transport window, 父剩余绝对预算)`;预算派生正常预留响应窗口,结果校验等待也必须 deadline-aware,不能只在校验完成后事后判超时。
|
||||
- 第一次失败后预算不足时不开始第二次;父侧从不重试整次内部 RPC。
|
||||
- 父业务预算仍有效时,flat 两次失败、熔断、overload、内部 RPC deadline 或断连仍走“阿里云 → 本地”;complex 任意失败直接失败且不读写熔断。
|
||||
- flat 熔断按真实失败 attempt 计数;由剩余业务预算截短的 timeout 不计入。已获准调用可完成第二次,后续排队请求快速 `circuit_open`。
|
||||
- `cancelled`、父 cancellation / 绝对 deadline、`invalid_request` 和 `unauthorized` 不启动 flat fallback;其它 flat 错误只在父业务预算仍有效时进入 fallback。
|
||||
- 客户端断连时,等待 permit 的请求最终由 deadline 收口;已开始 provider attempt 持有 permit 并排空。明确 cancellation 已被观察到后不再开始第二次,单纯 TCP 断连只作 best-effort 测试,不作为硬保证。
|
||||
- 动画全部已提交帧继续 collect / drain,根因按现有稳定帧序号收口;首版不存在未实现的 group cancellation 承诺。
|
||||
- 成功 body 为原始二进制而非 Base64;父、子两侧都拒绝空 body、MIME / 魔数不一致、chunked 超 `32 MiB` 和超过 `8192 × 8192` 的图片。
|
||||
- 成功 body 为原始图片字节而非 Base64、JSON 或结果 object key;父侧 client 把该有界字节缓冲直接交给现有后处理,子 worker 不执行 raw OSS PUT。父、子两侧都拒绝空 body、MIME / 魔数不一致、chunked 超 `32 MiB` 和超过 `8192 × 8192` 的图片。
|
||||
- `N / Q` response-body guard 在发送完成或 body drop 前不释放,慢读 / 断连时完整成功 body 不积累到 `Q` 级。
|
||||
- 子侧图片校验槽与 `N` 对齐;校验等待受 child deadline 约束,超时后的 blocking 校验 task 继续持有图片字节与校验槽,并被 shutdown tracker 排空;真实 provider 调用已经结束后不继续占用 `N`。
|
||||
- 父侧固定最多 `4` 个成功图片读取 / 解码槽,permit 在读取 `2xx` body 前取得并覆盖 `spawn_blocking` 校验;内部响应并发再高也不能无界累积父侧完整 body 或解码任务。
|
||||
- worker 重启 / RPC 丢失不查询、不恢复结果;父 job 的 heartbeat、lease、失败退款和 fencing 保持现状。
|
||||
- External v1 / inline 不再直连 BgFilter;公共 router、BFF、账单和任务列表中没有内部 endpoint 或内部调用记录。
|
||||
- 生产不存在两个同时运行的 `bgfilter-worker`,配置缺失或 `N / Q = 0` 时 fail-closed。
|
||||
- worker unit 先加载共享 API env、再加载 worker 专属 env;父子有效 `GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS` 完全一致,flat 熔断参数只由子 worker 配置和执行。
|
||||
- `external-generation-worker.env` 后加载时不得把内部 base URL、Token / Token 文件、connect timeout、provider attempt timeout、OSS bucket 或 endpoint 覆盖为与共享 API env 不同的有效值;父侧必须把源对象写到子 worker 将要签名读取的同一 OSS 位置。外部生成 worker 可使用同 bucket 下权限等价或更小的独立 AK,不要求凭据文本相同。
|
||||
- 父进程 `GENARRATIVE_BGFILTER_WORKER_BASE_URL`、子 worker `HOST / PORT` 和部署 readiness URL 必须指向同一个 `127.0.0.1:<port>` endpoint;旧非空配置不能因为“无需补默认值”而绕过一致性检查。
|
||||
- `genarrative-bgfilter-worker.service` 必须保持 `TimeoutStopSec=900`,覆盖最大 `600s` 内部请求预算和停止收口余量。
|
||||
- 发布目录切换前拒绝缺失、空、符号链接或权限错误的内部 Token 文件;父、子有效 Token 文件路径必须相同。
|
||||
- 生产运行期巡检同时检查 `genarrative-bgfilter-worker.service` 为 active 且 `127.0.0.1:8083/readyz` 成功,不能只依赖 systemd 自动重启。
|
||||
- `npm run dev` 启动独立 BgFilter 子进程并使用解析后的第五个端口;`all` 角色不内嵌 listener,单模块入口、watch、状态文件和退出清理没有遗留进程或硬编码端口。
|
||||
|
||||
实现后按范围运行:
|
||||
|
||||
@@ -435,3 +487,5 @@ git diff --check
|
||||
- 生产 `GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS = Q`。若要求一个满帧动画不因自身 admission 被拒绝,`Q` 至少为 `48`;候选 `128` 可容纳两个满帧动画并留少量余量,但明确不能覆盖 controller 理论最大 `768` 次同时提交,超出部分会按 mode fallback 或失败。最终值以可接受的 overload 行为和内存压测为准。
|
||||
|
||||
其余首版固定边界:provider 单 attempt 默认 `180s`,内部响应最大 `32 MiB / 8192 × 8192`,内部 listener 只绑定 loopback 且必须鉴权。若要多实例、严格服务端全局并发、QPS 或动画 peer cancellation,应先升级本文,不在编码中临时扩 scope。
|
||||
|
||||
配置归属同时冻结:`GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS=180000` 来自父子共同加载的 API 基础环境;`GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD`、`GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS`、`N` 与 `Q` 由 worker 专属环境管理。若部署脚本发现共享值被 worker 环境重复定义且不同,必须在启动前失败。
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -7,6 +7,7 @@
|
||||
"dev": "node scripts/dev.mjs",
|
||||
"dev:spacetime": "node scripts/dev.mjs spacetime",
|
||||
"dev:api-server": "node scripts/dev.mjs api-server",
|
||||
"dev:bgfilter-worker": "node scripts/dev.mjs bgfilter-worker",
|
||||
"dev:web": "node scripts/dev.mjs web",
|
||||
"dev:admin-web": "node scripts/dev.mjs admin-web",
|
||||
"server-manager:panel": "cargo run -p server-manager-panel --manifest-path server-rs/Cargo.toml",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -258,6 +258,42 @@ function assertApiReleaseContainsPingoraDirectDependencies() {
|
||||
),
|
||||
'API release 必须包含外部生成 worker controller systemd 单元。',
|
||||
);
|
||||
assertFileExists(
|
||||
path.join(
|
||||
releaseDir,
|
||||
'deploy/systemd/genarrative-bgfilter-worker.service',
|
||||
),
|
||||
'API release 必须包含唯一 BgFilter worker systemd 单元。',
|
||||
);
|
||||
assertFileExists(
|
||||
path.join(releaseDir, 'deploy/env/bgfilter-worker.env.example'),
|
||||
'API release 必须包含 BgFilter worker env 示例。',
|
||||
);
|
||||
const bgfilterUnit = readFileSync(
|
||||
path.join(
|
||||
releaseDir,
|
||||
'deploy/systemd/genarrative-bgfilter-worker.service',
|
||||
),
|
||||
'utf8',
|
||||
);
|
||||
const sharedEnvIndex = bgfilterUnit.indexOf(
|
||||
'EnvironmentFile=/etc/genarrative/api-server.env',
|
||||
);
|
||||
const dedicatedEnvIndex = bgfilterUnit.indexOf(
|
||||
'EnvironmentFile=/etc/genarrative/bgfilter-worker.env',
|
||||
);
|
||||
if (
|
||||
sharedEnvIndex < 0 ||
|
||||
dedicatedEnvIndex < 0 ||
|
||||
sharedEnvIndex > dedicatedEnvIndex
|
||||
) {
|
||||
failures.push('API release 的 BgFilter unit 必须按共享 env → 专属 env 加载。');
|
||||
}
|
||||
assertIncludes(
|
||||
bgfilterUnit,
|
||||
'TimeoutStopSec=900',
|
||||
'API release 的 BgFilter unit 必须给最多 600s 的内部请求预算留足优雅排空时间。',
|
||||
);
|
||||
assertFileExists(
|
||||
path.join(releaseDir, 'deploy/pingora/pingora-gateway.env.example'),
|
||||
'API release 必须包含 Pingora env 示例。',
|
||||
|
||||
@@ -54,6 +54,11 @@ function assertPublicBaseUrlDefaultsToGatewayEntry() {
|
||||
"process.env.GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL ||\n 'http://127.0.0.1'",
|
||||
'publicBaseUrl 默认必须指向本机网关入口,不能回落到 API 直连端口。',
|
||||
);
|
||||
assertIncludes(
|
||||
script,
|
||||
"process.env.GENARRATIVE_HEALTH_PATROL_BGFILTER_BASE_URL ||\n 'http://127.0.0.1:8083'",
|
||||
'BgFilter worker 巡检默认必须指向唯一实例的 loopback 端口。',
|
||||
);
|
||||
if (
|
||||
script.includes(
|
||||
'process.env.GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL ||\n process.env.GENARRATIVE_HEALTH_PATROL_API_BASE_URL',
|
||||
@@ -85,6 +90,14 @@ async function assertNginxModeChecksNginxService() {
|
||||
'systemctl is-active nginx.service',
|
||||
'nginx gateway mode 必须检查 nginx.service。',
|
||||
);
|
||||
assertIncludes(
|
||||
commandsLog,
|
||||
'systemctl is-active genarrative-bgfilter-worker.service',
|
||||
'生产巡检必须检查唯一 BgFilter worker service。',
|
||||
);
|
||||
if (!payload.checks.some((check) => check.name === 'bgfilter:/readyz')) {
|
||||
failures.push('生产巡检必须探测 BgFilter worker /readyz。');
|
||||
}
|
||||
if (commandsLog.includes('genarrative-pingora-gateway.service')) {
|
||||
failures.push(
|
||||
'nginx gateway mode 不应要求 Pingora gateway service active。',
|
||||
@@ -369,6 +382,8 @@ async function runPatrol(fixture, args) {
|
||||
'scripts/ops/production-health-patrol.mjs',
|
||||
'--api-base-url',
|
||||
fixture.baseUrl,
|
||||
'--bgfilter-base-url',
|
||||
fixture.baseUrl,
|
||||
'--spacetime-base-url',
|
||||
fixture.baseUrl,
|
||||
'--public-base-url',
|
||||
|
||||
@@ -1647,6 +1647,50 @@ const checks = [
|
||||
includes: 'genarrative-external-generation-worker@1.service',
|
||||
reason: 'Server-Provision 必须启用外部生成保底 worker 实例。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-bgfilter-worker.service',
|
||||
includes: 'TimeoutStopSec=900',
|
||||
reason:
|
||||
'BgFilter worker 必须给最多 600s 的内部请求预算留足优雅排空时间,不能沿用 systemd 默认停止窗口。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'validate_no_bgfilter_internal_token_plaintext',
|
||||
reason:
|
||||
'Server-Provision 必须拒绝 API 或 BgFilter worker env 保存内部 Token 明文。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes:
|
||||
'for env_file in "${API_ENV_FILE}" "${WORKER_ENV_FILE}" "${BGFILTER_WORKER_ENV_FILE}"; do',
|
||||
reason:
|
||||
'Server-Provision 必须同时拒绝 external-generation-worker.env 保存 BgFilter 内部 Token 明文。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes:
|
||||
'validate_bgfilter_env_file_alignment "${WORKER_ENV_FILE}" "外部生成 worker env" "false"',
|
||||
reason:
|
||||
'Server-Provision 启动外部生成 worker 前必须拒绝 BgFilter URL、Token 文件、timeout 与 OSS 位置漂移。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'validate_bgfilter_loopback_endpoint_alignment',
|
||||
reason:
|
||||
'Server-Provision 启动 BgFilter worker 前必须确认父 base URL 与子 listener 指向同一 loopback endpoint。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'BgFilter 内部 Token 文件不得为空或只包含空白字符',
|
||||
reason:
|
||||
'Server-Provision 必须拒绝仅含空白字符的 BgFilter 内部 Token 文件。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: "root:genarrative:440",
|
||||
reason:
|
||||
'Server-Provision 必须复核 BgFilter 内部 Token 文件的 owner、group 与 0440 权限。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/deploy/production-api-deploy.sh',
|
||||
includes: 'ensure_default_worker_service',
|
||||
|
||||
@@ -5,11 +5,11 @@ set -euo pipefail
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
用法:
|
||||
./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode]
|
||||
./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url http://127.0.0.1:8083/readyz] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode]
|
||||
|
||||
说明:
|
||||
进入维护模式,校验并发布 api-server 单文件,更新 current 链接,重启 systemd 服务并执行 readiness 检查。
|
||||
默认同时重启外部生成 worker controller 和已加载的 worker 实例;未启用 worker 单元时会自动跳过。
|
||||
默认先停止、启动并验活唯一 BgFilter worker,再重启 API、外部生成 worker controller 和已加载的 worker 实例。
|
||||
若传入 --database,会在重启前把 GENARRATIVE_SPACETIME_DATABASE 写入 api-server 环境文件,避免服务继续读取旧库。
|
||||
若发布包包含 pingora-gateway,或传入 --require-pingora-gateway,部署脚本会要求 release manifest、二进制与 checksum 一致,再在 current 链接切换后先复核 systemd/env 仍是本机高端口 shadow 配置,启动或重启 Pingora 影子服务并复核 active。
|
||||
默认在 readiness 通过后退出维护模式;传入 --keep-maintenance-mode 时保留维护文件,供人工验收后再恢复公网。
|
||||
@@ -175,6 +175,81 @@ if matched_value is not None:
|
||||
fi
|
||||
}
|
||||
|
||||
env_contains_nonempty_assignment() {
|
||||
local file_path="$1"
|
||||
local key="$2"
|
||||
|
||||
if [[ ! -f "${file_path}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
local python_script='
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
path = Path(sys.argv[1])
|
||||
key = sys.argv[2]
|
||||
for raw_line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
current_key, value = line.split("=", 1)
|
||||
if current_key.strip() != key:
|
||||
continue
|
||||
value = value.strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in ("\"", chr(39)):
|
||||
value = value[1:-1]
|
||||
if value.strip():
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(1)
|
||||
'
|
||||
|
||||
if [[ -r "${file_path}" ]]; then
|
||||
python3 -c "${python_script}" "${file_path}" "${key}"
|
||||
else
|
||||
if ! sudo -n true >/dev/null 2>&1; then
|
||||
echo "[production-api-deploy] 当前用户无权读取 ${file_path},且 sudo -n 不可用;无法检查运行态环境变量。" >&2
|
||||
exit 1
|
||||
fi
|
||||
sudo -n python3 -c "${python_script}" "${file_path}" "${key}"
|
||||
fi
|
||||
}
|
||||
|
||||
env_has_assignment() {
|
||||
local file_path="$1"
|
||||
local key="$2"
|
||||
|
||||
if [[ ! -f "${file_path}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
local python_script='
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
path = Path(sys.argv[1])
|
||||
key = sys.argv[2]
|
||||
for raw_line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
current_key, _ = line.split("=", 1)
|
||||
if current_key.strip() == key:
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(1)
|
||||
'
|
||||
|
||||
if [[ -r "${file_path}" ]]; then
|
||||
python3 -c "${python_script}" "${file_path}" "${key}"
|
||||
else
|
||||
if ! sudo -n true >/dev/null 2>&1; then
|
||||
echo "[production-api-deploy] 当前用户无权读取 ${file_path},且 sudo -n 不可用;无法检查运行态环境变量。" >&2
|
||||
exit 1
|
||||
fi
|
||||
sudo -n python3 -c "${python_script}" "${file_path}" "${key}"
|
||||
fi
|
||||
}
|
||||
|
||||
ensure_env_value() {
|
||||
local file_path="$1"
|
||||
local key="$2"
|
||||
@@ -303,10 +378,11 @@ ensure_runtime_env_and_dirs() {
|
||||
ensure_env_value_migrates_old_default "${api_env_file}" "GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS" "3600" "600"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_EXTERNAL_GENERATION_WORKER_JOB_TIMEOUT_SECONDS" "900"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_EXTERNAL_GENERATION_WORKER_LONG_JOB_TIMEOUT_SECONDS" "1800"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_BGFILTER_WORKER_BASE_URL" "http://127.0.0.1:8083"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE" "/etc/genarrative/secrets/bgfilter-worker.token"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS" "2000"
|
||||
ensure_runtime_bootstrap_secret_file_env "${api_env_file}"
|
||||
ensure_env_value_migrates_old_default "${api_env_file}" "GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS" "45000" "180000"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD" "3"
|
||||
ensure_env_value "${api_env_file}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS" "300"
|
||||
|
||||
tracking_enabled="$(read_env_value "${api_env_file}" "GENARRATIVE_TRACKING_OUTBOX_ENABLED")"
|
||||
tracking_outbox_dir="$(read_env_value "${api_env_file}" "GENARRATIVE_TRACKING_OUTBOX_DIR")"
|
||||
@@ -351,6 +427,186 @@ ensure_worker_runtime_env_defaults() {
|
||||
ensure_runtime_bootstrap_secret_file_env "${worker_env_file}"
|
||||
}
|
||||
|
||||
ensure_bgfilter_worker_runtime_env_defaults() {
|
||||
local bgfilter_env_file="$1"
|
||||
|
||||
if [[ -z "${bgfilter_env_file}" ]]; then
|
||||
return
|
||||
fi
|
||||
if [[ ! -f "${bgfilter_env_file}" ]]; then
|
||||
echo "[production-api-deploy] BgFilter worker 环境文件不存在: ${bgfilter_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
ensure_env_value "${bgfilter_env_file}" "GENARRATIVE_BGFILTER_WORKER_HOST" "127.0.0.1"
|
||||
ensure_env_value "${bgfilter_env_file}" "GENARRATIVE_BGFILTER_WORKER_PORT" "8083"
|
||||
ensure_env_value "${bgfilter_env_file}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD" "3"
|
||||
ensure_env_value "${bgfilter_env_file}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS" "300"
|
||||
}
|
||||
|
||||
validate_bgfilter_shared_runtime_env() {
|
||||
local api_env_file="$1"
|
||||
local request_timeout_ms connect_timeout_ms
|
||||
|
||||
request_timeout_ms="$(read_env_value "${api_env_file}" "GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS")"
|
||||
if [[ ! "${request_timeout_ms}" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "[production-api-deploy] GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS 必须在共享 API env 中配置为正整数毫秒: ${api_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
connect_timeout_ms="$(read_env_value "${api_env_file}" "GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS")"
|
||||
if [[ ! "${connect_timeout_ms}" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "[production-api-deploy] GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS 必须在共享 API env 中配置为正整数毫秒: ${api_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
validate_bgfilter_worker_capacity() {
|
||||
local bgfilter_env_file="$1"
|
||||
local concurrency max_requests
|
||||
|
||||
concurrency="$(read_env_value "${bgfilter_env_file}" "GENARRATIVE_BGFILTER_WORKER_CONCURRENCY")"
|
||||
max_requests="$(read_env_value "${bgfilter_env_file}" "GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS")"
|
||||
if [[ ! "${concurrency}" =~ ^[1-9][0-9]{0,8}$ ]]; then
|
||||
echo "[production-api-deploy] GENARRATIVE_BGFILTER_WORKER_CONCURRENCY 必须是正整数: ${bgfilter_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! "${max_requests}" =~ ^[1-9][0-9]{0,8}$ ]]; then
|
||||
echo "[production-api-deploy] GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS 必须是正整数: ${bgfilter_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
if (( 10#${max_requests} < 10#${concurrency} )); then
|
||||
echo "[production-api-deploy] GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS 必须大于或等于 CONCURRENCY: ${bgfilter_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
validate_bgfilter_worker_shared_env_alignment() {
|
||||
local api_env_file="$1"
|
||||
local bgfilter_env_file="$2"
|
||||
local key shared_value dedicated_value
|
||||
|
||||
for key in \
|
||||
GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS \
|
||||
GENARRATIVE_EDITOR_BGFILTER_BASE_URL \
|
||||
GENARRATIVE_EDITOR_BGFILTER_TOKEN \
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE \
|
||||
ALIYUN_OSS_BUCKET \
|
||||
ALIYUN_OSS_ENDPOINT \
|
||||
ALIYUN_OSS_ACCESS_KEY_ID \
|
||||
ALIYUN_OSS_ACCESS_KEY_SECRET \
|
||||
ALIYUN_OSS_READ_EXPIRE_SECONDS; do
|
||||
if ! env_has_assignment "${bgfilter_env_file}" "${key}"; then
|
||||
continue
|
||||
fi
|
||||
dedicated_value="$(read_env_value "${bgfilter_env_file}" "${key}")"
|
||||
shared_value="$(read_env_value "${api_env_file}" "${key}")"
|
||||
if [[ "${dedicated_value}" != "${shared_value}" ]]; then
|
||||
echo "[production-api-deploy] BgFilter 专属 env 中的共享配置与 API env 不一致: ${key};请迁移到 ${api_env_file} 并从 ${bgfilter_env_file} 删除重复项。" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
validate_external_generation_worker_bgfilter_env_alignment() {
|
||||
local api_env_file="$1"
|
||||
local worker_env_file="$2"
|
||||
local key shared_value worker_value
|
||||
|
||||
if [[ -z "${worker_env_file}" || ! -f "${worker_env_file}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
for key in \
|
||||
GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS \
|
||||
GENARRATIVE_BGFILTER_WORKER_BASE_URL \
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE \
|
||||
GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS \
|
||||
ALIYUN_OSS_BUCKET \
|
||||
ALIYUN_OSS_ENDPOINT; do
|
||||
if ! env_has_assignment "${worker_env_file}" "${key}"; then
|
||||
continue
|
||||
fi
|
||||
worker_value="$(read_env_value "${worker_env_file}" "${key}")"
|
||||
shared_value="$(read_env_value "${api_env_file}" "${key}")"
|
||||
if [[ "${worker_value}" != "${shared_value}" ]]; then
|
||||
echo "[production-api-deploy] 外部生成 worker env 中的 BgFilter 共享配置与 API env 不一致: ${key};${worker_env_file} 会在 systemd 中后加载并覆盖父侧有效值。" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
validate_bgfilter_loopback_endpoint_alignment() {
|
||||
local api_env_file="$1"
|
||||
local bgfilter_env_file="$2"
|
||||
local health_url="$3"
|
||||
local base_url host port expected_base_url expected_health_url
|
||||
|
||||
base_url="$(read_env_value "${api_env_file}" "GENARRATIVE_BGFILTER_WORKER_BASE_URL")"
|
||||
host="$(read_env_value "${bgfilter_env_file}" "GENARRATIVE_BGFILTER_WORKER_HOST")"
|
||||
port="$(read_env_value "${bgfilter_env_file}" "GENARRATIVE_BGFILTER_WORKER_PORT")"
|
||||
|
||||
if [[ "${host}" != "127.0.0.1" ]]; then
|
||||
echo "[production-api-deploy] BgFilter worker 首版必须监听 127.0.0.1,当前 GENARRATIVE_BGFILTER_WORKER_HOST=${host:-<empty>}: ${bgfilter_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! "${port}" =~ ^[1-9][0-9]{0,4}$ ]] || (( 10#${port} > 65535 )); then
|
||||
echo "[production-api-deploy] GENARRATIVE_BGFILTER_WORKER_PORT 必须是 1-65535 的有效端口: ${bgfilter_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
expected_base_url="http://${host}:${port}"
|
||||
if [[ "${base_url%/}" != "${expected_base_url}" ]]; then
|
||||
echo "[production-api-deploy] 父进程 GENARRATIVE_BGFILTER_WORKER_BASE_URL 必须与 BgFilter worker 有效监听地址一致: expected=${expected_base_url}, actual=${base_url:-<empty>}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
expected_health_url="${expected_base_url}/readyz"
|
||||
if [[ "${health_url}" != "${expected_health_url}" ]]; then
|
||||
echo "[production-api-deploy] --bgfilter-worker-health-url 必须与父进程 base URL 和子 worker listener 指向同一 loopback endpoint: expected=${expected_health_url}, actual=${health_url:-<empty>}" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
validate_no_bgfilter_internal_token_plaintext() {
|
||||
local env_file
|
||||
|
||||
for env_file in "$@"; do
|
||||
if [[ -z "${env_file}" ]]; then
|
||||
continue
|
||||
fi
|
||||
if env_contains_nonempty_assignment "${env_file}" "GENARRATIVE_BGFILTER_INTERNAL_TOKEN"; then
|
||||
echo "[production-api-deploy] ${env_file} 不得保存 GENARRATIVE_BGFILTER_INTERNAL_TOKEN 明文;生产环境只允许使用 GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE。" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
validate_bgfilter_internal_token_file() {
|
||||
local api_env_file="$1"
|
||||
local token_file token_metadata
|
||||
|
||||
token_file="$(read_env_value "${api_env_file}" "GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE")"
|
||||
if [[ -z "${token_file}" || "${token_file}" != /* ]]; then
|
||||
echo "[production-api-deploy] GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE 必须指向绝对路径: ${api_env_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ -L "${token_file}" || ! -f "${token_file}" || ! -s "${token_file}" ]]; then
|
||||
echo "[production-api-deploy] BgFilter 内部 Token 必须是非空普通文件且不能是符号链接: ${token_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! run_privileged grep -q '[^[:space:]]' -- "${token_file}"; then
|
||||
echo "[production-api-deploy] BgFilter 内部 Token 文件必须至少包含一个非空白字符: ${token_file}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
token_metadata="$(run_privileged stat -c '%U:%G:%a' -- "${token_file}")"
|
||||
if [[ "${token_metadata}" != "root:genarrative:440" ]]; then
|
||||
echo "[production-api-deploy] BgFilter 内部 Token 权限必须为 root:genarrative 0440,且必须是普通文件: ${token_file} (${token_metadata})" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
extract_pingora_env_files_from_unit() {
|
||||
local service_name="$1"
|
||||
local unit_content
|
||||
@@ -572,8 +828,17 @@ install_worker_systemd_units() {
|
||||
local release_dir="$1"
|
||||
local pattern="$2"
|
||||
local controller_service="$3"
|
||||
local bgfilter_service="$4"
|
||||
local installed_any=0
|
||||
|
||||
if [[ "${bgfilter_service}" == "genarrative-bgfilter-worker.service" ]]; then
|
||||
install_release_systemd_unit \
|
||||
"${release_dir}/deploy/systemd/genarrative-bgfilter-worker.service" \
|
||||
"genarrative-bgfilter-worker.service" \
|
||||
"BgFilter worker systemd 单元"
|
||||
installed_any=1
|
||||
fi
|
||||
|
||||
if [[ "${pattern}" == "genarrative-external-generation-worker@*.service" ]]; then
|
||||
install_release_systemd_unit \
|
||||
"${release_dir}/deploy/systemd/genarrative-external-generation-worker@.service" \
|
||||
@@ -688,6 +953,38 @@ wait_for_worker_controller_service() {
|
||||
return 1
|
||||
}
|
||||
|
||||
restart_and_wait_for_bgfilter_worker() {
|
||||
local service="$1"
|
||||
local health_url="$2"
|
||||
|
||||
if [[ -z "${service}" ]]; then
|
||||
echo "[production-api-deploy] 跳过 BgFilter worker 启动。"
|
||||
return 0
|
||||
fi
|
||||
if ! systemctl cat "${service}" >/dev/null 2>&1; then
|
||||
echo "[production-api-deploy] 缺少 BgFilter worker systemd 单元: ${service}" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "[production-api-deploy] 停止旧 BgFilter worker 并等待在途请求排空: ${service}"
|
||||
systemctl stop "${service}"
|
||||
systemctl enable "${service}"
|
||||
echo "[production-api-deploy] 启动唯一 BgFilter worker: ${service}"
|
||||
systemctl start "${service}"
|
||||
|
||||
for _ in {1..30}; do
|
||||
if systemctl is-active --quiet "${service}" && curl -fsS --max-time 2 "${health_url}" >/dev/null; then
|
||||
echo "[production-api-deploy] BgFilter worker readiness 通过: ${health_url}"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
systemctl --no-pager --full status "${service}" || true
|
||||
echo "[production-api-deploy] BgFilter worker readiness 检查超时: ${health_url}" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SOURCE_DIR=""
|
||||
VERSION=""
|
||||
@@ -697,6 +994,9 @@ SERVICE_NAME="genarrative-api.service"
|
||||
PINGORA_SERVICE_NAME="genarrative-pingora-gateway.service"
|
||||
WORKER_SERVICE_PATTERN="genarrative-external-generation-worker@*.service"
|
||||
WORKER_CONTROLLER_SERVICE="genarrative-external-generation-controller.service"
|
||||
BGFILTER_WORKER_SERVICE="genarrative-bgfilter-worker.service"
|
||||
BGFILTER_WORKER_HEALTH_URL="http://127.0.0.1:8083/readyz"
|
||||
BGFILTER_WORKER_ENV_FILE="/etc/genarrative/bgfilter-worker.env"
|
||||
HEALTH_URL="http://127.0.0.1:8082/readyz"
|
||||
API_ENV_FILE="/etc/genarrative/api-server.env"
|
||||
WORKER_ENV_FILE="/etc/genarrative/external-generation-worker.env"
|
||||
@@ -766,6 +1066,23 @@ while [[ $# -gt 0 ]]; do
|
||||
WORKER_CONTROLLER_SERVICE=""
|
||||
shift
|
||||
;;
|
||||
--bgfilter-worker-service)
|
||||
BGFILTER_WORKER_SERVICE="${2:?缺少 --bgfilter-worker-service 的值}"
|
||||
shift 2
|
||||
;;
|
||||
--bgfilter-worker-health-url)
|
||||
BGFILTER_WORKER_HEALTH_URL="${2:?缺少 --bgfilter-worker-health-url 的值}"
|
||||
shift 2
|
||||
;;
|
||||
--bgfilter-worker-env-file)
|
||||
BGFILTER_WORKER_ENV_FILE="${2:?缺少 --bgfilter-worker-env-file 的值}"
|
||||
shift 2
|
||||
;;
|
||||
--no-bgfilter-worker)
|
||||
BGFILTER_WORKER_SERVICE=""
|
||||
BGFILTER_WORKER_ENV_FILE=""
|
||||
shift
|
||||
;;
|
||||
--health-url)
|
||||
HEALTH_URL="${2:?缺少 --health-url 的值}"
|
||||
shift 2
|
||||
@@ -801,6 +1118,9 @@ require_absolute_path "${API_ENV_FILE}" "--api-env-file"
|
||||
if [[ -n "${WORKER_ENV_FILE}" ]]; then
|
||||
require_absolute_path "${WORKER_ENV_FILE}" "--worker-env-file"
|
||||
fi
|
||||
if [[ -n "${BGFILTER_WORKER_ENV_FILE}" ]]; then
|
||||
require_absolute_path "${BGFILTER_WORKER_ENV_FILE}" "--bgfilter-worker-env-file"
|
||||
fi
|
||||
|
||||
if [[ -n "${DATABASE}" ]]; then
|
||||
validate_spacetime_database_name "${DATABASE}"
|
||||
@@ -1127,8 +1447,20 @@ if [[ -n "${SPACETIME_SERVER_URL}" ]]; then
|
||||
fi
|
||||
|
||||
ensure_runtime_env_and_dirs "${API_ENV_FILE}"
|
||||
validate_bgfilter_shared_runtime_env "${API_ENV_FILE}"
|
||||
validate_real_wechat_pay_refund_reconciliation "${API_ENV_FILE}"
|
||||
ensure_worker_runtime_env_defaults "${WORKER_ENV_FILE}"
|
||||
ensure_bgfilter_worker_runtime_env_defaults "${BGFILTER_WORKER_ENV_FILE}"
|
||||
validate_external_generation_worker_bgfilter_env_alignment "${API_ENV_FILE}" "${WORKER_ENV_FILE}"
|
||||
validate_no_bgfilter_internal_token_plaintext "${API_ENV_FILE}" "${WORKER_ENV_FILE}" "${BGFILTER_WORKER_ENV_FILE}"
|
||||
if [[ -n "${BGFILTER_WORKER_SERVICE}" ]]; then
|
||||
validate_bgfilter_internal_token_file "${API_ENV_FILE}"
|
||||
validate_bgfilter_loopback_endpoint_alignment "${API_ENV_FILE}" "${BGFILTER_WORKER_ENV_FILE}" "${BGFILTER_WORKER_HEALTH_URL}"
|
||||
fi
|
||||
if [[ -n "${BGFILTER_WORKER_ENV_FILE}" ]]; then
|
||||
validate_bgfilter_worker_capacity "${BGFILTER_WORKER_ENV_FILE}"
|
||||
validate_bgfilter_worker_shared_env_alignment "${API_ENV_FILE}" "${BGFILTER_WORKER_ENV_FILE}"
|
||||
fi
|
||||
migrate_legacy_editor_generation_pricing_override "${CURRENT_LINK}"
|
||||
|
||||
if [[ "${PINGORA_INCLUDED}" -eq 1 ]]; then
|
||||
@@ -1149,7 +1481,9 @@ if [[ "${PINGORA_INCLUDED}" -eq 1 ]]; then
|
||||
ensure_pingora_shadow_service "${PINGORA_SERVICE_NAME}" "${PINGORA_SHADOW_ENV_FILE}"
|
||||
fi
|
||||
|
||||
install_worker_systemd_units "${RELEASE_DIR}" "${WORKER_SERVICE_PATTERN}" "${WORKER_CONTROLLER_SERVICE}"
|
||||
install_worker_systemd_units "${RELEASE_DIR}" "${WORKER_SERVICE_PATTERN}" "${WORKER_CONTROLLER_SERVICE}" "${BGFILTER_WORKER_SERVICE}"
|
||||
|
||||
restart_and_wait_for_bgfilter_worker "${BGFILTER_WORKER_SERVICE}" "${BGFILTER_WORKER_HEALTH_URL}"
|
||||
|
||||
echo "[production-api-deploy] 重启服务: ${SERVICE_NAME}"
|
||||
systemctl restart "${SERVICE_NAME}"
|
||||
|
||||
@@ -55,8 +55,8 @@ export function parsePortRangeSpec(value) {
|
||||
throw new Error(`端口段无效: ${spec},端口必须在 1024-65535 且起始不大于结束`);
|
||||
}
|
||||
|
||||
if (end - start + 1 < 4) {
|
||||
throw new Error(`端口段至少需要 4 个端口: ${spec}`);
|
||||
if (end - start + 1 < 5) {
|
||||
throw new Error(`端口段至少需要 5 个端口: ${spec}`);
|
||||
}
|
||||
|
||||
return {start, end, label: `${start}-${end}`};
|
||||
@@ -118,6 +118,7 @@ export function mapDevPortsToPortRange(portRange) {
|
||||
apiPort: normalizedRange.start + 1,
|
||||
spacetimePort: normalizedRange.start + 2,
|
||||
adminWebPort: normalizedRange.start + 3,
|
||||
bgfilterWorkerPort: normalizedRange.start + 4,
|
||||
range: normalizedRange,
|
||||
};
|
||||
}
|
||||
@@ -569,6 +570,7 @@ export async function resolveDevStackPorts(config) {
|
||||
['api', config.api],
|
||||
['web', config.web],
|
||||
['adminWeb', config.adminWeb],
|
||||
['bgfilterWorker', config.bgfilterWorker],
|
||||
].filter(([, portConfig]) => Boolean(portConfig));
|
||||
const result = {};
|
||||
|
||||
|
||||
@@ -47,7 +47,7 @@ async function reserveConsecutivePorts() {
|
||||
}
|
||||
|
||||
describe('dev stack port utils', () => {
|
||||
it('解析端口段并映射到四个 dev 端口', () => {
|
||||
it('解析端口段并映射到五个 dev 端口', () => {
|
||||
expect(parsePortRangeSpec('10000-10099')).toEqual({
|
||||
start: 10000,
|
||||
end: 10099,
|
||||
@@ -58,7 +58,11 @@ describe('dev stack port utils', () => {
|
||||
apiPort: 10001,
|
||||
spacetimePort: 10002,
|
||||
adminWebPort: 10003,
|
||||
bgfilterWorkerPort: 10004,
|
||||
});
|
||||
expect(() => parsePortRangeSpec('10000-10003')).toThrow(
|
||||
'端口段至少需要 5 个端口',
|
||||
);
|
||||
});
|
||||
|
||||
it('使用端口可用性检查为被占用端口寻找后续可用端口', async () => {
|
||||
@@ -112,9 +116,10 @@ describe('dev stack port utils', () => {
|
||||
api: {host: '127.0.0.1', preferredPort: 0},
|
||||
web: {host: '127.0.0.1', preferredPort: 0},
|
||||
adminWeb: {host: '127.0.0.1', preferredPort: 0},
|
||||
bgfilterWorker: {host: '127.0.0.1', preferredPort: 0},
|
||||
});
|
||||
|
||||
expect(new Set(Object.values(resolvedPorts)).size).toBe(4);
|
||||
expect(new Set(Object.values(resolvedPorts)).size).toBe(5);
|
||||
});
|
||||
|
||||
it('端口段内会一直漂移到段尾,不会被默认 200 次尝试截断', async () => {
|
||||
|
||||
+335
-23
File diff suppressed because it is too large
Load Diff
+172
-1
@@ -19,6 +19,7 @@ import {
|
||||
assertReusableSpacetimeProcessVersionMatchesWorkspace,
|
||||
assertSpacetimeToolVersionMatchesWorkspace,
|
||||
buildApiServerProcessEnv,
|
||||
buildBgfilterWorkerProcessEnv,
|
||||
buildDevStackSnapshot,
|
||||
buildFrontendProcessEnv,
|
||||
buildLocalRustProcessEnv,
|
||||
@@ -85,6 +86,26 @@ describe('dev scheduler argument routing', () => {
|
||||
expect(runner.resolveFrontendApiTarget()).toBe('http://127.0.0.1:8090');
|
||||
});
|
||||
|
||||
test('独立 BgFilter worker 命令解析内部监听地址', () => {
|
||||
const { command, explicitOptions, options } = parseArgs(
|
||||
[
|
||||
'bgfilter-worker',
|
||||
'--bgfilter-worker-host',
|
||||
'127.0.0.2',
|
||||
'--bgfilter-worker-port',
|
||||
'18083',
|
||||
],
|
||||
{},
|
||||
);
|
||||
|
||||
expect(command).toBe('bgfilter-worker');
|
||||
expect(explicitOptions).toEqual(
|
||||
new Set(['bgfilterWorkerHost', 'bgfilterWorkerPort']),
|
||||
);
|
||||
expect(options.bgfilterWorkerHost).toBe('127.0.0.2');
|
||||
expect(options.bgfilterWorkerPort).toBe(18083);
|
||||
});
|
||||
|
||||
test('单独 dev:web 未显式指定 api 参数时沿用已有 Rust target', () => {
|
||||
const testEnv = {
|
||||
RUST_SERVER_TARGET: 'http://127.0.0.1:3100',
|
||||
@@ -129,7 +150,7 @@ describe('dev scheduler argument routing', () => {
|
||||
);
|
||||
});
|
||||
|
||||
linuxTest('Linux 启动时按系统级端口段映射四个 dev 端口', async () => {
|
||||
linuxTest('Linux 启动时按系统级端口段映射五个 dev 端口', async () => {
|
||||
const tempDir = mkdtempSync(join(tmpdir(), 'genarrative-dev-port-range-'));
|
||||
try {
|
||||
const { command, explicitOptions, options } = parseArgs([], {
|
||||
@@ -156,7 +177,9 @@ describe('dev scheduler argument routing', () => {
|
||||
expect(runner.options.apiPort).toBe(22001);
|
||||
expect(runner.options.spacetimePort).toBe(22002);
|
||||
expect(runner.options.adminWebPort).toBe(22003);
|
||||
expect(runner.options.bgfilterWorkerPort).toBe(22004);
|
||||
expect(runner.state.apiTarget).toBe('http://127.0.0.1:22001');
|
||||
expect(runner.state.bgfilterWorkerTarget).toBe('http://127.0.0.1:22004');
|
||||
expect(runner.state.spacetimeServer).toBe('http://127.0.0.1:22002');
|
||||
} finally {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
@@ -196,6 +219,7 @@ describe('dev scheduler argument routing', () => {
|
||||
expect(runner.options.apiPort).toBe(22001);
|
||||
expect(runner.options.spacetimePort).toBe(22002);
|
||||
expect(runner.options.adminWebPort).toBe(22003);
|
||||
expect(runner.options.bgfilterWorkerPort).toBe(22004);
|
||||
} finally {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
@@ -233,6 +257,7 @@ describe('dev scheduler argument routing', () => {
|
||||
expect(runner.options.apiPort).toBe(8082);
|
||||
expect(runner.options.spacetimePort).toBe(3101);
|
||||
expect(runner.options.adminWebPort).toBe(3102);
|
||||
expect(runner.options.bgfilterWorkerPort).toBe(8083);
|
||||
} finally {
|
||||
if (originalPlatform) {
|
||||
Object.defineProperty(process, 'platform', originalPlatform);
|
||||
@@ -282,6 +307,45 @@ describe('dev scheduler api-server env', () => {
|
||||
expect(env.GENARRATIVE_PROCESS_ROLE).toBe('api');
|
||||
});
|
||||
|
||||
test('父 API 与独立 BgFilter worker 共享实际 URL 和内部 token', () => {
|
||||
const { options } = parseArgs([], {});
|
||||
options.bgfilterWorkerPort = 18083;
|
||||
const state = {
|
||||
spacetimeServer: 'http://127.0.0.1:3199',
|
||||
bgfilterWorkerTarget: 'http://127.0.0.1:18083',
|
||||
};
|
||||
const internalToken = 'local-bgfilter-token';
|
||||
|
||||
const apiEnv = buildApiServerProcessEnv({
|
||||
baseEnv: {},
|
||||
options,
|
||||
state,
|
||||
bgfilterInternalToken: internalToken,
|
||||
processRole: 'all',
|
||||
});
|
||||
const workerEnv = buildBgfilterWorkerProcessEnv({
|
||||
baseEnv: {},
|
||||
options,
|
||||
state,
|
||||
bgfilterInternalToken: internalToken,
|
||||
});
|
||||
|
||||
expect(apiEnv.GENARRATIVE_PROCESS_ROLE).toBe('all');
|
||||
expect(workerEnv.GENARRATIVE_PROCESS_ROLE).toBe('bgfilter-worker');
|
||||
expect(apiEnv.GENARRATIVE_BGFILTER_WORKER_BASE_URL).toBe(
|
||||
state.bgfilterWorkerTarget,
|
||||
);
|
||||
expect(workerEnv.GENARRATIVE_BGFILTER_WORKER_BASE_URL).toBe(
|
||||
state.bgfilterWorkerTarget,
|
||||
);
|
||||
expect(apiEnv.GENARRATIVE_BGFILTER_INTERNAL_TOKEN).toBe(internalToken);
|
||||
expect(workerEnv.GENARRATIVE_BGFILTER_INTERNAL_TOKEN).toBe(internalToken);
|
||||
expect(workerEnv.GENARRATIVE_BGFILTER_WORKER_HOST).toBe('127.0.0.1');
|
||||
expect(workerEnv.GENARRATIVE_BGFILTER_WORKER_PORT).toBe('18083');
|
||||
expect(workerEnv.GENARRATIVE_BGFILTER_WORKER_CONCURRENCY).toBe('4');
|
||||
expect(workerEnv.GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS).toBe('128');
|
||||
});
|
||||
|
||||
test('Windows 本地 dev 自动注入已安装的 FFmpeg 路径', () => {
|
||||
const tempDir = mkdtempSync(join(tmpdir(), 'genarrative-ffmpeg-'));
|
||||
try {
|
||||
@@ -339,6 +403,100 @@ describe('dev scheduler api-server env', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('dev scheduler Rust service orchestration', () => {
|
||||
test('Rust 双进程重启时先全部停止,再先 ready BgFilter、后 ready API', async () => {
|
||||
const { explicitOptions, options } = parseArgs([], {});
|
||||
const runner = new DevRunner(options, {}, explicitOptions);
|
||||
const events: string[] = [];
|
||||
runner.command = 'all';
|
||||
runner.windowsApiServerCleanupCompleted = true;
|
||||
runner.services = new Map([
|
||||
[
|
||||
'api-server',
|
||||
{
|
||||
stop: async () => events.push('stop-api'),
|
||||
start: async () => events.push('start-api'),
|
||||
},
|
||||
],
|
||||
[
|
||||
'bgfilter-worker',
|
||||
{
|
||||
stop: async () => events.push('stop-bgfilter'),
|
||||
start: async () => events.push('start-bgfilter'),
|
||||
},
|
||||
],
|
||||
]);
|
||||
vi.spyOn(runner, 'waitForBgfilterWorker').mockImplementation(async () => {
|
||||
events.push('ready-bgfilter');
|
||||
});
|
||||
vi.spyOn(runner, 'waitForApiServer').mockImplementation(async () => {
|
||||
events.push('ready-api');
|
||||
});
|
||||
|
||||
await runner.restartRustServicePair();
|
||||
|
||||
expect(events).toEqual([
|
||||
'stop-api',
|
||||
'stop-bgfilter',
|
||||
'start-bgfilter',
|
||||
'ready-bgfilter',
|
||||
'start-api',
|
||||
'ready-api',
|
||||
]);
|
||||
});
|
||||
|
||||
test('dev:api-server 安全自动带起同 runner 的 BgFilter worker', async () => {
|
||||
const { explicitOptions, options } = parseArgs(['api-server'], {});
|
||||
const runner = new DevRunner(options, {}, explicitOptions);
|
||||
const startPair = vi
|
||||
.spyOn(runner, 'startRustServicePair')
|
||||
.mockResolvedValue(undefined);
|
||||
const startWatchers = vi
|
||||
.spyOn(runner, 'startWatchers')
|
||||
.mockImplementation(() => {});
|
||||
|
||||
await runner.startCommand('api-server');
|
||||
|
||||
expect(startPair).toHaveBeenCalledOnce();
|
||||
expect(startWatchers).toHaveBeenCalledWith([
|
||||
'api-server',
|
||||
'bgfilter-worker',
|
||||
]);
|
||||
});
|
||||
|
||||
test('完整栈只为两个 Rust 角色创建一套组合 watcher', () => {
|
||||
const { explicitOptions, options } = parseArgs(['--watch'], {});
|
||||
const runner = new DevRunner(options, {}, explicitOptions);
|
||||
runner.command = 'all';
|
||||
runner.registerServices();
|
||||
|
||||
try {
|
||||
runner.startWatchers(['api-server', 'bgfilter-worker']);
|
||||
expect(runner.watchers).toHaveLength(1);
|
||||
} finally {
|
||||
for (const watcher of runner.watchers) {
|
||||
watcher.close();
|
||||
}
|
||||
runner.watchers = [];
|
||||
}
|
||||
});
|
||||
|
||||
test('BgFilter worker 在 readiness 前退出时立即失败', async () => {
|
||||
const { explicitOptions, options } = parseArgs([], {});
|
||||
const runner = new DevRunner(options, {}, explicitOptions);
|
||||
runner.services = new Map([
|
||||
['bgfilter-worker', { runtime: { status: 'failed' } }],
|
||||
]);
|
||||
globalThis.fetch = vi.fn(async () => ({
|
||||
status: 503,
|
||||
})) as unknown as typeof fetch;
|
||||
|
||||
await expect(runner.waitForBgfilterWorker()).rejects.toThrow(
|
||||
'bgfilter-worker 在 readiness 前退出',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('dev scheduler local worker cleanup', () => {
|
||||
const expected = {
|
||||
expectedDatabase: 'xushi-p4wfr',
|
||||
@@ -471,6 +629,8 @@ describe('dev scheduler stack state file', () => {
|
||||
options: {
|
||||
apiHost: '127.0.0.1',
|
||||
apiPort: 8090,
|
||||
bgfilterWorkerHost: '127.0.0.1',
|
||||
bgfilterWorkerPort: 8091,
|
||||
webHost: '0.0.0.0',
|
||||
webPort: 3010,
|
||||
adminWebHost: '127.0.0.1',
|
||||
@@ -483,6 +643,7 @@ describe('dev scheduler stack state file', () => {
|
||||
},
|
||||
state: {
|
||||
apiTarget: 'http://127.0.0.1:8090',
|
||||
bgfilterWorkerTarget: 'http://127.0.0.1:8091',
|
||||
adminWebTargetHost: '127.0.0.1',
|
||||
spacetimeServer: 'http://127.0.0.1:3120',
|
||||
},
|
||||
@@ -527,6 +688,12 @@ describe('dev scheduler stack state file', () => {
|
||||
port: 8090,
|
||||
url: 'http://127.0.0.1:8090',
|
||||
});
|
||||
expect(snapshot.services['bgfilter-worker']).toMatchObject({
|
||||
status: 'idle',
|
||||
pid: null,
|
||||
port: 8091,
|
||||
url: 'http://127.0.0.1:8091',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1258,6 +1425,8 @@ spacetimedb tool version 2.6.0; spacetimedb-lib version 2.6.0;
|
||||
'migration-secret-hash',
|
||||
GENARRATIVE_SPACETIME_RUNTIME_SERVICE_BOOTSTRAP_SECRET:
|
||||
'runtime-secret',
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN: 'bgfilter-token',
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE: 'bgfilter-token-file',
|
||||
SAFE_VALUE: 'kept',
|
||||
},
|
||||
{ RUST_SERVER_TARGET: 'http://127.0.0.1:8082' },
|
||||
@@ -1273,6 +1442,8 @@ spacetimedb tool version 2.6.0; spacetimedb-lib version 2.6.0;
|
||||
expect(env).not.toHaveProperty(
|
||||
'GENARRATIVE_SPACETIME_RUNTIME_SERVICE_BOOTSTRAP_SECRET',
|
||||
);
|
||||
expect(env).not.toHaveProperty('GENARRATIVE_BGFILTER_INTERNAL_TOKEN');
|
||||
expect(env).not.toHaveProperty('GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE');
|
||||
expect(env.SAFE_VALUE).toBe('kept');
|
||||
});
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ SPACETIME_BIN_SOURCE="${SPACETIME_BIN_SOURCE:-${PROVISION_TOOLS_DIR}/spacetime/s
|
||||
OTELCOL_BIN_SOURCE="${OTELCOL_BIN_SOURCE:-${PROVISION_TOOLS_DIR}/otelcol-contrib}"
|
||||
WORKER_ENV_FILE="${WORKER_ENV_FILE:-/etc/genarrative/external-generation-worker.env}"
|
||||
CONTROLLER_ENV_FILE="${CONTROLLER_ENV_FILE:-/etc/genarrative/external-generation-controller.env}"
|
||||
BGFILTER_WORKER_ENV_FILE="${BGFILTER_WORKER_ENV_FILE:-/etc/genarrative/bgfilter-worker.env}"
|
||||
GENARRATIVE_OPENSSL_VERSION="${GENARRATIVE_OPENSSL_VERSION:-3.2.0}"
|
||||
GENARRATIVE_OPENSSL_PREFIX="${GENARRATIVE_OPENSSL_PREFIX:-/opt/genarrative/openssl-3.2.0}"
|
||||
GENARRATIVE_OPENSSL_SOURCE_URL="${GENARRATIVE_OPENSSL_SOURCE_URL:-https://github.com/openssl/openssl/releases/download/openssl-${GENARRATIVE_OPENSSL_VERSION}/openssl-${GENARRATIVE_OPENSSL_VERSION}.tar.gz}"
|
||||
@@ -408,6 +409,101 @@ read_env_value() {
|
||||
done <"${file}"
|
||||
}
|
||||
|
||||
env_contains_nonempty_assignment() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
local line value first_char last_char
|
||||
|
||||
if [[ ! -f "${file}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
while IFS= read -r line || [[ -n "${line}" ]]; do
|
||||
line="${line#"${line%%[![:space:]]*}"}"
|
||||
if [[ -z "${line}" || "${line}" == \#* || "${line}" != "${key}="* ]]; then
|
||||
continue
|
||||
fi
|
||||
value="${line#*=}"
|
||||
value="${value%$'\r'}"
|
||||
value="${value#"${value%%[![:space:]]*}"}"
|
||||
value="${value%"${value##*[![:space:]]}"}"
|
||||
if [[ ${#value} -ge 2 ]]; then
|
||||
first_char="${value:0:1}"
|
||||
last_char="${value: -1}"
|
||||
if [[ "${first_char}" == "${last_char}" && ( "${first_char}" == '"' || "${first_char}" == "'" ) ]]; then
|
||||
value="${value:1:${#value}-2}"
|
||||
fi
|
||||
fi
|
||||
if [[ "${value}" =~ [^[:space:]] ]]; then
|
||||
return 0
|
||||
fi
|
||||
done <"${file}"
|
||||
return 1
|
||||
}
|
||||
|
||||
env_has_assignment() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
local line current_key
|
||||
|
||||
if [[ ! -f "${file}" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
while IFS= read -r line || [[ -n "${line}" ]]; do
|
||||
line="${line%$'\r'}"
|
||||
line="${line#"${line%%[![:space:]]*}"}"
|
||||
if [[ -z "${line}" || "${line}" == \#* || "${line}" != *"="* ]]; then
|
||||
continue
|
||||
fi
|
||||
current_key="${line%%=*}"
|
||||
current_key="${current_key%"${current_key##*[![:space:]]}"}"
|
||||
if [[ "${current_key}" == "${key}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
done <"${file}"
|
||||
return 1
|
||||
}
|
||||
|
||||
read_effective_env_value() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
local line current_key value first_char last_char matched_value="" found="false"
|
||||
|
||||
if [[ ! -f "${file}" ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
while IFS= read -r line || [[ -n "${line}" ]]; do
|
||||
line="${line%$'\r'}"
|
||||
line="${line#"${line%%[![:space:]]*}"}"
|
||||
if [[ -z "${line}" || "${line}" == \#* || "${line}" != *"="* ]]; then
|
||||
continue
|
||||
fi
|
||||
current_key="${line%%=*}"
|
||||
current_key="${current_key%"${current_key##*[![:space:]]}"}"
|
||||
if [[ "${current_key}" != "${key}" ]]; then
|
||||
continue
|
||||
fi
|
||||
value="${line#*=}"
|
||||
value="${value#"${value%%[![:space:]]*}"}"
|
||||
value="${value%"${value##*[![:space:]]}"}"
|
||||
if [[ ${#value} -ge 2 ]]; then
|
||||
first_char="${value:0:1}"
|
||||
last_char="${value: -1}"
|
||||
if [[ "${first_char}" == "${last_char}" && ( "${first_char}" == '"' || "${first_char}" == "'" ) ]]; then
|
||||
value="${value:1:${#value}-2}"
|
||||
fi
|
||||
fi
|
||||
matched_value="${value}"
|
||||
found="true"
|
||||
done <"${file}"
|
||||
|
||||
if [[ "${found}" == "true" ]]; then
|
||||
printf "%s" "${matched_value}"
|
||||
fi
|
||||
}
|
||||
|
||||
write_env_value() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
@@ -525,10 +621,120 @@ ensure_api_runtime_env_defaults() {
|
||||
ensure_env_value_migrates_old_default "${API_ENV_FILE}" "GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS" "3600" "600"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_EXTERNAL_GENERATION_WORKER_JOB_TIMEOUT_SECONDS" "900"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_EXTERNAL_GENERATION_WORKER_LONG_JOB_TIMEOUT_SECONDS" "1800"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_BASE_URL" "http://127.0.0.1:8083"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE" "/etc/genarrative/secrets/bgfilter-worker.token"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS" "2000"
|
||||
ensure_runtime_bootstrap_secret_file_env "${API_ENV_FILE}"
|
||||
ensure_env_value_migrates_old_default "${API_ENV_FILE}" "GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS" "45000" "180000"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD" "3"
|
||||
ensure_env_value "${API_ENV_FILE}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS" "300"
|
||||
}
|
||||
|
||||
validate_bgfilter_shared_runtime_env() {
|
||||
local request_timeout_ms connect_timeout_ms
|
||||
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ validate shared BgFilter provider attempt timeout in ${API_ENV_FILE}"
|
||||
return
|
||||
fi
|
||||
|
||||
request_timeout_ms="$(read_effective_env_value "${API_ENV_FILE}" "GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS")"
|
||||
if [[ ! "${request_timeout_ms}" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "[server-provision] GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS 必须在共享 API env 中配置为正整数毫秒: ${API_ENV_FILE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
connect_timeout_ms="$(read_effective_env_value "${API_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS")"
|
||||
if [[ ! "${connect_timeout_ms}" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "[server-provision] GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS 必须在共享 API env 中配置为正整数毫秒: ${API_ENV_FILE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
validate_no_bgfilter_internal_token_plaintext() {
|
||||
local env_file
|
||||
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ reject non-empty GENARRATIVE_BGFILTER_INTERNAL_TOKEN in ${API_ENV_FILE}, ${WORKER_ENV_FILE}, and ${BGFILTER_WORKER_ENV_FILE}"
|
||||
return
|
||||
fi
|
||||
|
||||
for env_file in "${API_ENV_FILE}" "${WORKER_ENV_FILE}" "${BGFILTER_WORKER_ENV_FILE}"; do
|
||||
if env_contains_nonempty_assignment "${env_file}" "GENARRATIVE_BGFILTER_INTERNAL_TOKEN"; then
|
||||
echo "[server-provision] ${env_file} 不得保存 GENARRATIVE_BGFILTER_INTERNAL_TOKEN 明文;生产环境只允许使用 GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE。" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
validate_bgfilter_env_file_alignment() {
|
||||
local env_file="$1"
|
||||
local label="$2"
|
||||
local include_provider_credentials="$3"
|
||||
local key shared_value dedicated_value
|
||||
local -a shared_keys=(
|
||||
GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS
|
||||
GENARRATIVE_BGFILTER_WORKER_BASE_URL
|
||||
GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE
|
||||
GENARRATIVE_BGFILTER_WORKER_CONNECT_TIMEOUT_MS
|
||||
ALIYUN_OSS_BUCKET
|
||||
ALIYUN_OSS_ENDPOINT
|
||||
)
|
||||
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ validate ${label} BgFilter shared configuration alignment with ${API_ENV_FILE}"
|
||||
return
|
||||
fi
|
||||
if [[ ! -f "${env_file}" ]]; then
|
||||
echo "[server-provision] ${label} 不存在,无法检查 BgFilter 共享配置: ${env_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${include_provider_credentials}" == "true" ]]; then
|
||||
shared_keys+=(
|
||||
GENARRATIVE_EDITOR_BGFILTER_BASE_URL
|
||||
GENARRATIVE_EDITOR_BGFILTER_TOKEN
|
||||
ALIYUN_OSS_ACCESS_KEY_ID
|
||||
ALIYUN_OSS_ACCESS_KEY_SECRET
|
||||
ALIYUN_OSS_READ_EXPIRE_SECONDS
|
||||
)
|
||||
fi
|
||||
|
||||
for key in "${shared_keys[@]}"; do
|
||||
if ! env_has_assignment "${env_file}" "${key}"; then
|
||||
continue
|
||||
fi
|
||||
dedicated_value="$(read_effective_env_value "${env_file}" "${key}")"
|
||||
shared_value="$(read_effective_env_value "${API_ENV_FILE}" "${key}")"
|
||||
if [[ "${dedicated_value}" != "${shared_value}" ]]; then
|
||||
echo "[server-provision] ${label} 中的 BgFilter 共享配置与 API env 不一致: ${key};后加载 env 会覆盖进程有效值。" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
validate_bgfilter_loopback_endpoint_alignment() {
|
||||
local base_url host port expected_base_url
|
||||
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ validate BgFilter parent base URL and child listener alignment"
|
||||
return
|
||||
fi
|
||||
|
||||
base_url="$(read_effective_env_value "${API_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_BASE_URL")"
|
||||
host="$(read_effective_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_HOST")"
|
||||
port="$(read_effective_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_PORT")"
|
||||
if [[ "${host}" != "127.0.0.1" ]]; then
|
||||
echo "[server-provision] BgFilter worker 首版必须监听 127.0.0.1,当前 GENARRATIVE_BGFILTER_WORKER_HOST=${host:-<empty>}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! "${port}" =~ ^[1-9][0-9]{0,4}$ ]] || (( 10#${port} > 65535 )); then
|
||||
echo "[server-provision] GENARRATIVE_BGFILTER_WORKER_PORT 必须是 1-65535 的有效端口: ${BGFILTER_WORKER_ENV_FILE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
expected_base_url="http://${host}:${port}"
|
||||
if [[ "${base_url%/}" != "${expected_base_url}" ]]; then
|
||||
echo "[server-provision] 父进程 GENARRATIVE_BGFILTER_WORKER_BASE_URL 必须与 BgFilter worker 有效监听地址一致: expected=${expected_base_url}, actual=${base_url:-<empty>}" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
ensure_worker_runtime_env_defaults() {
|
||||
@@ -547,6 +753,80 @@ ensure_worker_runtime_env_defaults() {
|
||||
ensure_runtime_bootstrap_secret_file_env "${WORKER_ENV_FILE}"
|
||||
}
|
||||
|
||||
ensure_bgfilter_worker_runtime_env_defaults() {
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ ensure BgFilter worker runtime env defaults in ${BGFILTER_WORKER_ENV_FILE}"
|
||||
return
|
||||
fi
|
||||
if [[ ! -f "${BGFILTER_WORKER_ENV_FILE}" ]]; then
|
||||
echo "[server-provision] BgFilter worker 环境文件不存在,无法补齐运行态变量: ${BGFILTER_WORKER_ENV_FILE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ensure_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_HOST" "127.0.0.1"
|
||||
ensure_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_PORT" "8083"
|
||||
ensure_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_CONCURRENCY" "4"
|
||||
ensure_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_BGFILTER_WORKER_MAX_REQUESTS" "128"
|
||||
ensure_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD" "3"
|
||||
ensure_env_value "${BGFILTER_WORKER_ENV_FILE}" "GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS" "300"
|
||||
}
|
||||
|
||||
ensure_bgfilter_internal_token_file() {
|
||||
local token_file="/etc/genarrative/secrets/bgfilter-worker.token"
|
||||
local token_dir="/etc/genarrative/secrets"
|
||||
local openssl_bin="${GENARRATIVE_OPENSSL_PREFIX}/bin/openssl"
|
||||
local configured_token_file temporary_file token_metadata
|
||||
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ ensure shared BgFilter internal token file ${token_file} (root:genarrative 0440)"
|
||||
return
|
||||
fi
|
||||
configured_token_file="$(read_effective_env_value "${API_ENV_FILE}" "GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE")"
|
||||
if [[ "${configured_token_file}" != "${token_file}" ]]; then
|
||||
echo "[server-provision] GENARRATIVE_BGFILTER_INTERNAL_TOKEN_FILE 必须与 Provision 管理路径一致: ${token_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -L "${token_dir}" || -L "${token_file}" ]]; then
|
||||
echo "[server-provision] BgFilter 内部 Token 目录和文件不能是符号链接: ${token_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -e "${token_file}" && ! -f "${token_file}" ]]; then
|
||||
echo "[server-provision] BgFilter 内部 Token 必须是普通文件: ${token_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
install -d -o root -g genarrative -m 0750 "${token_dir}"
|
||||
if [[ ! -f "${token_file}" ]]; then
|
||||
temporary_file="$(mktemp "${token_dir}/.bgfilter-worker.token.XXXXXX")"
|
||||
if ! "${openssl_bin}" rand -hex 32 >"${temporary_file}"; then
|
||||
rm -f "${temporary_file}"
|
||||
echo "[server-provision] 生成 BgFilter 内部 Token 失败。" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! grep -q '[^[:space:]]' -- "${temporary_file}"; then
|
||||
rm -f "${temporary_file}"
|
||||
echo "[server-provision] 生成的 BgFilter 内部 Token 不得为空或只包含空白字符。" >&2
|
||||
exit 1
|
||||
fi
|
||||
chown root:genarrative "${temporary_file}"
|
||||
chmod 0440 "${temporary_file}"
|
||||
mv -T "${temporary_file}" "${token_file}"
|
||||
echo "[server-provision] 已生成 BgFilter 内部 Token 文件: ${token_file}"
|
||||
else
|
||||
if ! grep -q '[^[:space:]]' -- "${token_file}"; then
|
||||
echo "[server-provision] BgFilter 内部 Token 文件不得为空或只包含空白字符: ${token_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
chown root:genarrative "${token_file}"
|
||||
chmod 0440 "${token_file}"
|
||||
echo "[server-provision] BgFilter 内部 Token 文件已存在,保留内容并收紧权限。"
|
||||
fi
|
||||
token_metadata="$(stat -c '%U:%G:%a' -- "${token_file}")"
|
||||
if [[ "${token_metadata}" != "root:genarrative:440" ]]; then
|
||||
echo "[server-provision] BgFilter 内部 Token 权限必须为 root:genarrative 0440: ${token_file} (${token_metadata})" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
parse_json_string_field() {
|
||||
local json="$1"
|
||||
local key="$2"
|
||||
@@ -684,6 +964,10 @@ render_external_generation_controller_env_example() {
|
||||
cat deploy/env/external-generation-controller.env.example
|
||||
}
|
||||
|
||||
render_bgfilter_worker_env_example() {
|
||||
cat deploy/env/bgfilter-worker.env.example
|
||||
}
|
||||
|
||||
render_otelcol_service() {
|
||||
cat deploy/systemd/otelcol-contrib.service
|
||||
}
|
||||
@@ -927,6 +1211,35 @@ render_external_generation_controller_service() {
|
||||
deploy/systemd/genarrative-external-generation-controller.service
|
||||
}
|
||||
|
||||
render_bgfilter_worker_service() {
|
||||
local current_escaped api_env_escaped bgfilter_env_escaped
|
||||
current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
|
||||
api_env_escaped="$(escape_sed_replacement "${API_ENV_FILE}")"
|
||||
bgfilter_env_escaped="$(escape_sed_replacement "${BGFILTER_WORKER_ENV_FILE}")"
|
||||
sed \
|
||||
-e "s|/opt/genarrative/current|${current_escaped}|g" \
|
||||
-e "s|/etc/genarrative/api-server.env|${api_env_escaped}|g" \
|
||||
-e "s|/etc/genarrative/bgfilter-worker.env|${bgfilter_env_escaped}|g" \
|
||||
deploy/systemd/genarrative-bgfilter-worker.service
|
||||
}
|
||||
|
||||
wait_for_bgfilter_worker_service() {
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ curl -fsS --max-time 2 http://127.0.0.1:8083/readyz"
|
||||
return
|
||||
fi
|
||||
echo "[server-provision] 等待 BgFilter worker readiness。"
|
||||
for _ in {1..30}; do
|
||||
if systemctl is-active --quiet genarrative-bgfilter-worker.service && curl -fsS --max-time 2 http://127.0.0.1:8083/readyz >/dev/null; then
|
||||
return
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
systemctl --no-pager --full status genarrative-bgfilter-worker.service || true
|
||||
echo "[server-provision] BgFilter worker 未在超时时间内通过 readiness。" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
render_database_backup_service() {
|
||||
local current_escaped env_escaped
|
||||
current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
|
||||
@@ -995,6 +1308,7 @@ require_path deploy/systemd/spacetimedb.service
|
||||
require_path deploy/systemd/genarrative-api.service
|
||||
require_path deploy/systemd/genarrative-external-generation-worker@.service
|
||||
require_path deploy/systemd/genarrative-external-generation-controller.service
|
||||
require_path deploy/systemd/genarrative-bgfilter-worker.service
|
||||
require_path deploy/systemd/genarrative-database-backup.service
|
||||
require_path deploy/systemd/genarrative-database-backup-files-history.conf
|
||||
require_path deploy/systemd/genarrative-database-backup.timer
|
||||
@@ -1013,6 +1327,7 @@ require_path deploy/logrotate/genarrative-pingora-gateway
|
||||
require_path deploy/env/api-server.env.example
|
||||
require_path deploy/env/external-generation-worker.env.example
|
||||
require_path deploy/env/external-generation-controller.env.example
|
||||
require_path deploy/env/bgfilter-worker.env.example
|
||||
require_path scripts/deploy/maintenance-on.sh
|
||||
require_path scripts/deploy/maintenance-off.sh
|
||||
require_path scripts/deploy/maintenance-status.sh
|
||||
@@ -1026,7 +1341,7 @@ echo "[server-provision] target=${DEPLOY_TARGET}, dry_run=${DRY_RUN}, nginx_conf
|
||||
run_cmd id
|
||||
require_root_for_real_provision
|
||||
install_nginx_brotli_modules
|
||||
run_cmd mkdir -p "${SPACETIME_ROOT}" "${RELEASE_ROOT}" "$(dirname "${CURRENT_LINK}")" "$(dirname "${WEB_LINK}")" /etc/genarrative /etc/genarrative/pingora /var/lib/genarrative/maintenance /var/lib/genarrative/auth /var/lib/genarrative/tracking-outbox /var/lib/genarrative/wallet-refund-outbox /var/lib/genarrative/editor-generation-pricing /var/lib/genarrative/spacetime /var/lib/genarrative/database-backups /var/lib/genarrative/health-patrol /var/log/genarrative
|
||||
run_cmd mkdir -p "${SPACETIME_ROOT}" "${RELEASE_ROOT}" "$(dirname "${CURRENT_LINK}")" "$(dirname "${WEB_LINK}")" /etc/genarrative /etc/genarrative/pingora /etc/genarrative/secrets /var/lib/genarrative/maintenance /var/lib/genarrative/auth /var/lib/genarrative/tracking-outbox /var/lib/genarrative/wallet-refund-outbox /var/lib/genarrative/editor-generation-pricing /var/lib/genarrative/spacetime /var/lib/genarrative/database-backups /var/lib/genarrative/health-patrol /var/log/genarrative
|
||||
|
||||
if ! id spacetimedb >/dev/null 2>&1; then
|
||||
run_cmd useradd --system --home-dir "${SPACETIME_ROOT}" --shell /usr/sbin/nologin spacetimedb
|
||||
@@ -1069,18 +1384,21 @@ spacetimedb_service="$(mktemp)"
|
||||
api_service="$(mktemp)"
|
||||
external_generation_worker_service="$(mktemp)"
|
||||
external_generation_controller_service="$(mktemp)"
|
||||
bgfilter_worker_service="$(mktemp)"
|
||||
database_backup_service="$(mktemp)"
|
||||
health_patrol_service="$(mktemp)"
|
||||
render_spacetimedb_service >"${spacetimedb_service}"
|
||||
render_api_service >"${api_service}"
|
||||
render_external_generation_worker_service >"${external_generation_worker_service}"
|
||||
render_external_generation_controller_service >"${external_generation_controller_service}"
|
||||
render_bgfilter_worker_service >"${bgfilter_worker_service}"
|
||||
render_database_backup_service >"${database_backup_service}"
|
||||
render_health_patrol_service >"${health_patrol_service}"
|
||||
install_file "${spacetimedb_service}" /etc/systemd/system/spacetimedb.service 0644
|
||||
install_file "${api_service}" /etc/systemd/system/genarrative-api.service 0644
|
||||
install_file "${external_generation_worker_service}" /etc/systemd/system/genarrative-external-generation-worker@.service 0644
|
||||
install_file "${external_generation_controller_service}" /etc/systemd/system/genarrative-external-generation-controller.service 0644
|
||||
install_file "${bgfilter_worker_service}" /etc/systemd/system/genarrative-bgfilter-worker.service 0644
|
||||
install_file "${database_backup_service}" /etc/systemd/system/genarrative-database-backup.service 0644
|
||||
install_file deploy/systemd/genarrative-database-backup.timer /etc/systemd/system/genarrative-database-backup.timer 0644
|
||||
install_file "${health_patrol_service}" /etc/systemd/system/genarrative-health-patrol.service 0644
|
||||
@@ -1088,7 +1406,7 @@ install_file deploy/systemd/genarrative-health-patrol.timer /etc/systemd/system/
|
||||
install_file deploy/systemd/genarrative-pingora-gateway.service /etc/systemd/system/genarrative-pingora-gateway.service 0644
|
||||
install_file deploy/systemd/genarrative-pingora-gateway-direct-entry.conf /etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf 0644
|
||||
install_file deploy/logrotate/genarrative-pingora-gateway /etc/logrotate.d/genarrative-pingora-gateway 0644
|
||||
rm -f "${spacetimedb_service}" "${api_service}" "${external_generation_worker_service}" "${external_generation_controller_service}" "${database_backup_service}" "${health_patrol_service}"
|
||||
rm -f "${spacetimedb_service}" "${api_service}" "${external_generation_worker_service}" "${external_generation_controller_service}" "${bgfilter_worker_service}" "${database_backup_service}" "${health_patrol_service}"
|
||||
|
||||
if [[ ! -f "${API_ENV_FILE}" ]]; then
|
||||
echo "+ create ${API_ENV_FILE} from example"
|
||||
@@ -1101,6 +1419,7 @@ else
|
||||
echo "[server-provision] 已存在环境文件,保留不覆盖: ${API_ENV_FILE}"
|
||||
fi
|
||||
ensure_api_runtime_env_defaults
|
||||
validate_bgfilter_shared_runtime_env
|
||||
configure_database_backup_profile
|
||||
|
||||
if [[ ! -f "${WORKER_ENV_FILE}" ]]; then
|
||||
@@ -1115,6 +1434,23 @@ else
|
||||
fi
|
||||
ensure_worker_runtime_env_defaults
|
||||
|
||||
if [[ ! -f "${BGFILTER_WORKER_ENV_FILE}" ]]; then
|
||||
echo "+ create ${BGFILTER_WORKER_ENV_FILE} from example"
|
||||
if [[ "${DRY_RUN}" != "true" ]]; then
|
||||
render_bgfilter_worker_env_example >"${BGFILTER_WORKER_ENV_FILE}"
|
||||
chmod 0600 "${BGFILTER_WORKER_ENV_FILE}"
|
||||
chown root:root "${BGFILTER_WORKER_ENV_FILE}"
|
||||
fi
|
||||
else
|
||||
echo "[server-provision] 已存在 BgFilter worker 环境文件,保留不覆盖: ${BGFILTER_WORKER_ENV_FILE}"
|
||||
fi
|
||||
ensure_bgfilter_worker_runtime_env_defaults
|
||||
validate_bgfilter_env_file_alignment "${WORKER_ENV_FILE}" "外部生成 worker env" "false"
|
||||
validate_bgfilter_env_file_alignment "${BGFILTER_WORKER_ENV_FILE}" "BgFilter 专属 env" "true"
|
||||
validate_bgfilter_loopback_endpoint_alignment
|
||||
validate_no_bgfilter_internal_token_plaintext
|
||||
ensure_bgfilter_internal_token_file
|
||||
|
||||
if [[ ! -f "${CONTROLLER_ENV_FILE}" ]]; then
|
||||
echo "+ create ${CONTROLLER_ENV_FILE} from example"
|
||||
if [[ "${DRY_RUN}" != "true" ]]; then
|
||||
@@ -1149,7 +1485,7 @@ if [[ "${ENABLE_SERVICES}" == "true" ]]; then
|
||||
run_cmd systemctl enable otelcol-contrib.service
|
||||
fi
|
||||
stamp_database_backup_timer_now
|
||||
run_cmd systemctl enable spacetimedb.service genarrative-api.service genarrative-database-backup.timer genarrative-external-generation-worker@1.service genarrative-external-generation-controller.service genarrative-health-patrol.timer
|
||||
run_cmd systemctl enable spacetimedb.service genarrative-bgfilter-worker.service genarrative-api.service genarrative-database-backup.timer genarrative-external-generation-worker@1.service genarrative-external-generation-controller.service genarrative-health-patrol.timer
|
||||
run_cmd systemctl start genarrative-database-backup.timer
|
||||
if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
|
||||
run_cmd systemctl restart otelcol-contrib.service
|
||||
@@ -1158,13 +1494,17 @@ if [[ "${ENABLE_SERVICES}" == "true" ]]; then
|
||||
wait_for_spacetimedb_service
|
||||
ensure_spacetime_owner_client_token
|
||||
if [[ -x "${CURRENT_LINK}/api-server" ]]; then
|
||||
run_cmd systemctl enable genarrative-bgfilter-worker.service
|
||||
run_cmd systemctl stop genarrative-bgfilter-worker.service
|
||||
run_cmd systemctl start genarrative-bgfilter-worker.service
|
||||
wait_for_bgfilter_worker_service
|
||||
run_cmd systemctl restart genarrative-api.service
|
||||
run_cmd systemctl enable --now genarrative-external-generation-worker@1.service
|
||||
run_cmd systemctl restart genarrative-external-generation-worker@1.service
|
||||
run_cmd systemctl enable --now genarrative-external-generation-controller.service
|
||||
run_cmd systemctl restart genarrative-external-generation-controller.service
|
||||
else
|
||||
echo "[server-provision] 尚未发现 ${CURRENT_LINK}/api-server,跳过 api-server、外部生成 worker 和 controller 首次启动。后续 API deploy 会启用并启动默认 worker 与 controller。"
|
||||
echo "[server-provision] 尚未发现 ${CURRENT_LINK}/api-server,跳过 BgFilter worker、api-server、外部生成 worker 和 controller 首次启动。后续 API deploy 会按顺序启动。"
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
@@ -18,12 +18,14 @@ const DEFAULT_PUBLIC_PATHS = [
|
||||
|
||||
const DEFAULT_SERVICES = [
|
||||
'genarrative-api.service',
|
||||
'genarrative-bgfilter-worker.service',
|
||||
'genarrative-external-generation-controller.service',
|
||||
'spacetimedb.service',
|
||||
'nginx.service',
|
||||
];
|
||||
const PINGORA_DIRECT_SERVICES = [
|
||||
'genarrative-api.service',
|
||||
'genarrative-bgfilter-worker.service',
|
||||
'genarrative-external-generation-controller.service',
|
||||
'spacetimedb.service',
|
||||
'genarrative-pingora-gateway.service',
|
||||
@@ -37,6 +39,7 @@ function usage() {
|
||||
|
||||
Options:
|
||||
--api-base-url <url> API direct base URL, default http://127.0.0.1:8082
|
||||
--bgfilter-base-url <url> BgFilter worker base URL, default http://127.0.0.1:8083
|
||||
--spacetime-base-url <url> SpacetimeDB base URL, default http://127.0.0.1:3101
|
||||
--public-base-url <url> Nginx/public base URL, default http://127.0.0.1
|
||||
--public-host <host> Optional public Host header, useful when probing 127.0.0.1
|
||||
@@ -88,6 +91,9 @@ function parseArgs(argv) {
|
||||
apiBaseUrl:
|
||||
process.env.GENARRATIVE_HEALTH_PATROL_API_BASE_URL ||
|
||||
'http://127.0.0.1:8082',
|
||||
bgfilterBaseUrl:
|
||||
process.env.GENARRATIVE_HEALTH_PATROL_BGFILTER_BASE_URL ||
|
||||
'http://127.0.0.1:8083',
|
||||
spacetimeBaseUrl:
|
||||
process.env.GENARRATIVE_HEALTH_PATROL_SPACETIME_BASE_URL ||
|
||||
'http://127.0.0.1:3101',
|
||||
@@ -131,6 +137,9 @@ function parseArgs(argv) {
|
||||
case '--api-base-url':
|
||||
config.apiBaseUrl = requireValue(argv, ++index, arg);
|
||||
break;
|
||||
case '--bgfilter-base-url':
|
||||
config.bgfilterBaseUrl = requireValue(argv, ++index, arg);
|
||||
break;
|
||||
case '--spacetime-base-url':
|
||||
config.spacetimeBaseUrl = requireValue(argv, ++index, arg);
|
||||
break;
|
||||
@@ -693,6 +702,13 @@ async function main() {
|
||||
config,
|
||||
),
|
||||
);
|
||||
checks.push(
|
||||
await checkHttp(
|
||||
'bgfilter:/readyz',
|
||||
joinUrl(config.bgfilterBaseUrl, '/readyz'),
|
||||
config,
|
||||
),
|
||||
);
|
||||
checks.push(
|
||||
await checkHttp(
|
||||
'spacetimedb:/v1/ping',
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user