diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 989acc572..fa952afd5 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -4426,3 +4426,10 @@ - 平台差异(Windows 开发环境):连接已关闭的 loopback 端口不回 RST 而是挂到 connect timeout,错误呈现为 `deadline_exceeded` 且 `is_connect` 为真;重连判定只看 connect 分类,不看错误码。生产 Linux 即时拒绝,呈现 `internal_error`。 - 安全不变量测试:除配额 / 跨窗 / deadline 地板路径外,专项覆盖「TCP 已 accept、未回任何 HTTP 字节即断开 → 不得发起第二次连接」,以 mock listener 的 accept 计数证明父侧未重连。 - 影响范围:`server-rs/crates/api-server/src/bgfilter_worker.rs`、`state.rs`、`editor_project.rs`、调度方案 §5.1/§7/§9.3/§11、数据契约 247-248 行、运维文档及各编辑器专题文档的旧禁令措辞统一改为「不重试已被 worker 接收的内部 RPC」。 + +## 2026-07-23 生产 API 发布按实际路径渲染 worker systemd unit + +- 背景:Server-Provision 支持自定义 current link、API env 和角色 env,并在首次安装时渲染三个 worker unit;API deploy 为下发随 release 更新的 unit 又原样覆盖目标机配置,导致自定义路径在下一次发布时退回模板默认值。 +- 决策:`production-api-deploy.sh` 继续随 release 安装默认命名的 BgFilter、external-generation worker 和 controller unit,但安装前必须用本次部署参数渲染临时文件;新增 `--controller-env-file` 补齐 controller 专属 env 输入。release 内模板保持默认路径,供 provision 和 deploy 共同作为单一模板来源;自定义服务名仍由目标机自行管理,不强制覆盖。 +- 影响范围:`scripts/deploy/production-api-deploy.sh`、`scripts/check-production-api-deploy.mjs`、`jenkins/Jenkinsfile.production-api-deploy`、`jenkins/Jenkinsfile.production-full-build-and-deploy`、`scripts/check-production-ops-guardrails.mjs`、生产运维文档和 worker systemd 发布契约。 +- 验证方式:`bash -n scripts/deploy/production-api-deploy.sh`、`node --check scripts/check-production-api-deploy.mjs`、`npm run check:production-api-deploy`、`npm run check:production-ops`、`npm run check:encoding`、`git diff --check`。 diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index e30f3e187..d81032d37 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -3312,3 +3312,11 @@ - 验证与回滚:重启后先跑真实 PR 的四个 job,再清理旧镜像。失败时先把 workflow `runs-on` 改回 `ubuntu-latest`,再恢复 runner config 备份并重启;不在 Git、共享文档或日志中记录 config 备份路径、注册信息或 token。 - 重启边界:`docker restart --timeout 660` 只设置容器停止宽限,不能替代 Runner drain。rootless DinD supervisor 可能与 runner 同时停止内层 dockerd,使仍在收尾的 job 因连接关闭被标记失败;切换前必须同时确认 Gitea 没有 `in_progress` run 且内层 `docker ps` 为空。误触发时只重跑受影响的失败 job,不重跑已成功项。 - 关联:`deploy/container/README.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`、`docs/project-memory/shared-memory/development-workflow.md`。 + +## 生产 API 发布重装 worker unit 不能丢失自定义路径(2026-07-23) + +- 现象:Server-Provision 已按自定义 current link 和 env 路径安装 worker systemd unit,但下一次 API 发布后,worker 可能重新读取 `/opt/genarrative/current` 与 `/etc/genarrative/*.env`;默认路径仍有旧 release 时,服务 active 和部署成功都不能证明新二进制已运行。 +- 原因:发布包中的 BgFilter、external-generation worker 和 controller unit 是带默认路径的模板;deploy 若直接 `install` 原文件,会覆盖 provision 已渲染的目标机 unit。external-generation 专属 env 还是可选加载,错误路径可能不会阻止服务进入 active。 +- 处理:API deploy 安装三个 unit 前必须按本次 current、API env 和各角色 env 参数渲染临时文件,安装后保留 release 内原始模板不变;controller 自定义 env 由 `--controller-env-file` 显式传入。API Deploy 与 Full Job 必须同步暴露并透传 controller/BgFilter env,不能让流水线回退默认路径。自定义服务名表示沿用目标机自管 unit,不进入默认 unit 安装分支。 +- 验证:部署 guard 使用临时自定义绝对路径,直接读取实际安装目录中的三个 unit,核对 `WorkingDirectory`、`ExecStart`、共享 API env 与角色 env,不能只用 fake `systemctl is-active` 判绿。 +- 关联:`scripts/deploy/production-api-deploy.sh`、`scripts/check-production-api-deploy.mjs`、`scripts/jenkins-server-provision.sh`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index c6754e144..f2079082e 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -68,7 +68,7 @@ Windows 本地如果已在 `%LOCALAPPDATA%\Genarrative\ffmpeg\bin` 安装 FFmpeg HTTP 角色的 `GENARRATIVE_SPACETIME_POOL_SIZE` 只表示 procedure / reducer 调用池大小;池连接不订阅 read model。HTTP 角色会额外创建 1 条共享缓存读连接,当前只保留可选的 `user_account` 读取;配置为 `8` 时基础连接拓扑是 8 条调用连接加 1 条缓存读连接。`/readyz` 同时检查调用池与缓存读连接,缓存连接未准备好时不能放量。 -生产拆分角色时,`external-generation-worker` 和 `external-generation-controller` 的专属 env 示例会把 `GENARRATIVE_SPACETIME_POOL_SIZE` 覆盖为 `1`;非 HTTP 角色不创建 API 缓存读连接,只保留 `external_generation_job` 队列窄订阅作为响应式唤醒信号,实际抢占和扩缩容判断仍走 SpacetimeDB procedure。worker / controller 不执行模型定价 seed,启动时先调用受 runtime writer 鉴权的 queue-stats procedure 做只读预检,身份不匹配时 fail-fast;当前正式 systemd unit 通过共同加载 `/etc/genarrative/api-server.env` 继承同一 `GENARRATIVE_SPACETIME_TOKEN`,专属角色 env 示例不重复配置该 token。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 与 controller poll interval 只作为订阅失效、漏事件和 lease 过期这类时间条件的兜底,不作为正常领取任务的主路径。 +生产拆分角色时,`external-generation-worker` 和 `external-generation-controller` 的专属 env 示例会把 `GENARRATIVE_SPACETIME_POOL_SIZE` 覆盖为 `1`;非 HTTP 角色不创建 API 缓存读连接,只保留 `external_generation_job` 队列窄订阅作为响应式唤醒信号,实际抢占和扩缩容判断仍走 SpacetimeDB procedure。worker / controller 不执行模型定价 seed,启动时先调用受 runtime writer 鉴权的 queue-stats procedure 做只读预检,身份不匹配时 fail-fast;当前正式 systemd unit 通过共同加载 API env 继承同一 `GENARRATIVE_SPACETIME_TOKEN`,默认路径为 `/etc/genarrative/api-server.env`,自定义部署由 provision 和 API deploy 按实际参数渲染,专属角色 env 示例不重复配置该 token。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 与 controller poll interval 只作为订阅失效、漏事件和 lease 过期这类时间条件的兜底,不作为正常领取任务的主路径。 生产 worker 默认 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=600`,只覆盖 worker 心跳抖动和短暂断连窗口,不再把 lease 当成完整任务时长;默认 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_JOB_TIMEOUT_SECONDS=900`。`editor_image_generation`、`editor_image_edit`、`editor_icon_spritesheet_generation`、`editor_ui_design_asset_extraction` 四类 VectorEngine 图片任务与角色动画 / 视频类长任务使用 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_LONG_JOB_TIMEOUT_SECONDS=1800`,手动去背景、音效和背景音乐继续使用普通预算。worker 在单次尝试超过执行预算后会停止续租并释放 worker 槽位,但不会取消已启动的业务 future 或主动写入失败 / 重试状态;在途执行由 lease fencing 仲裁,有效租约内写回仍可完成,租约过期后任务才可重新领取,attempt 耗尽时由认领事务标记失败并结算退款。生产部署和 provision 脚本会给 `/etc/genarrative/api-server.env` 与 `/etc/genarrative/external-generation-worker.env` 补齐这些变量;已有自定义值不覆盖,只会把历史旧默认 `3600` 迁移为 `600`。 @@ -609,7 +609,7 @@ BgFilter 受限资源调度使用非模板单实例 `genarrative-bgfilter-worker 生产发布必须按 `共享配置 / endpoint / Token / N / est / Q 预检 → stop 旧 BgFilter worker → 等待 systemd 排空 → start 唯一实例 → 检查 http://127.0.0.1:8083/readyz → 重启 API → 重启 external-generation worker / controller` 的顺序执行。Token 预检要求 API env 指向非空、非符号链接的普通文件,权限固定为 `root:genarrative 0440`;endpoint 预检要求父进程 `GENARRATIVE_BGFILTER_WORKER_BASE_URL`、子 worker `HOST / PORT` 和 readiness URL 指向同一个 `127.0.0.1:`;预算预检要求共享 `N / est` 均为正整数、父子有效值一致,当前模板默认和压测前冻结值为 `N=16 / est=5000ms`,后续允许按真实压测校准 `est`,不由 deploy 脚本写死;`Q` 缺省为 `2048`,显式值不得小于 `N`,历史模板默认 `128` 会在 deploy / Provision 时定向迁移为 `2048`;熔断 cooldown 的历史模板默认 `300s` 同样会定向迁移为 `120s`;其它显式定制值均保留。任何预检失败都发生在 `current` 切换和停止现役 worker 之前。非模板 unit、固定 loopback 端口和显式 stop/start 共同避免新旧 BgFilter worker 重叠;第二实例会因固定端口绑定失败。内部请求分别携带 `maxQueueWaitMs` 与公式化 `callBudgetMs`;worker 停机时立即让尚未取得 provider permit 的排队请求失败,只排空已经取得 permit 的调用。默认 `callBudgetMs=321s`,unit 使用 `TimeoutStopSec=900` 给最多两次 attempt、响应发送和进程收口留足余量,禁止沿用约 `90s` 的默认停止窗口。默认 API deploy 会安装、enable、启动并验活该 unit;只有明确回滚或应急排障时才使用 `--no-bgfilter-worker` 跳过,且不得让父进程偷偷恢复为直连 BgFilter。 -`api-server` 进程角色由 `GENARRATIVE_PROCESS_ROLE` 控制:`api` 只监听 HTTP,`external-generation-worker` 只消费外部生成队列,`external-generation-controller` 只管理 worker systemd 实例,`all` 仅用于本地或临时 smoke,不隐式启动 controller。外部生成策略由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制;生产和容器压测默认保持 `queue`,本地 `npm run dev` / `npm run dev:api-server` 默认由 dev 脚本注入 `GENARRATIVE_PROCESS_ROLE=all`,如果外部生成策略为 `queue` 会由同一进程消费队列。`inline` 只用于本地或低并发同步排查,HTTP handler 会直接复用 worker executor,完成后返回 `completed`,但不会落 `external_generation_job`,也不能通过增加 worker 进程扩吞吐。外部生成 worker 使用同一发布包和同一套 SpacetimeDB 配置,按实例数和 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩;生产默认由 `genarrative-external-generation-controller.service` 读取 `get_external_generation_queue_stats_and_return`,按 `claimable_pending + running_active + expired_running` 计算目标 worker 数,并对 `genarrative-external-generation-worker@N.service` 精确执行 `systemctl start/stop`。controller 参数模板是 `deploy/env/external-generation-controller.env.example`:默认保底 `MIN_WORKERS=1`、上限 `MAX_WORKERS=8`、每 worker 目标 `TARGET_JOBS_PER_WORKER=2`、`POLL_INTERVAL_MS=10000`、连续 `SCALE_DOWN_IDLE_ROUNDS=6` 轮完全空闲才缩容;缩容每轮只停止最高编号的一个实例,且不主动停止 `@1`。worker 收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成;若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`,未完成任务才会在 lease 过期后由其它 worker 重领。每个 worker 实例应设置唯一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`,默认会用主机名和 pid 兜底;systemd 生产模板 `deploy/systemd/genarrative-external-generation-worker@.service` 会用 `%H-%i` 生成实例 ID,并把 tracking outbox 隔离到 `/var/lib/genarrative/tracking-outbox/%H-%i`。`Genarrative-Server-Provision` 会安装 worker 模板、controller unit 和两份专属 env 模板,默认 enable 首个 `genarrative-external-generation-worker@1.service` 与 `genarrative-external-generation-controller.service`;首次 API deploy 会在默认 worker pattern 下自动 `enable --now genarrative-external-generation-worker@1.service` 并等待 worker active,同时重启并验活 controller。手动兜底扩容仍可用 `systemctl start genarrative-external-generation-worker@2.service` / `@3.service`,缩容用 `systemctl stop genarrative-external-generation-worker@N.service`;controller 下轮会按队列压力修正到目标实例数。worker 专属参数模板是 `deploy/env/external-generation-worker.env.example`,密钥与 SpacetimeDB 连接仍复用 `/etc/genarrative/api-server.env`。API 发布脚本默认会重启并验活 `genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service`;若本次只发 HTTP 且不希望滚动 worker,可传 `--no-worker-services`,若不希望重启 controller 可传 `--no-worker-controller`。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 控制空队列轮询间隔,`GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS` 控制单次 lease,worker 会约每三分之一 lease、最长 30 秒续租;该值应覆盖一次心跳网络抖动窗口,不需要大于完整外部生成链路耗时。SpacetimeDB 使用自身事务时间计算 claim/renew/complete/fail,完成和失败回写还会校验 `lease_token` 与未过期 lease,避免同一 job 被过期 worker 覆盖。首版 worker 粒度是单动作单 job,不拆阶段 job;当前外部生成动作覆盖拼图、跳一跳、拼消消、敲木鱼和图片画布编辑器生成 / 抠图入口,纯元信息保存、发布、试玩启动、运行态动作和公开读取继续 inline。图片画布 worker 成功后由后端保存 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`,前端只轮询 job 并重新读取项目快照。当前生成业务失败只做用户重新触发,不做自动业务重试,避免 worker 退款和重试成功之间产生钱包账本漂移。 +`api-server` 进程角色由 `GENARRATIVE_PROCESS_ROLE` 控制:`api` 只监听 HTTP,`external-generation-worker` 只消费外部生成队列,`external-generation-controller` 只管理 worker systemd 实例,`all` 仅用于本地或临时 smoke,不隐式启动 controller。外部生成策略由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制;生产和容器压测默认保持 `queue`,本地 `npm run dev` / `npm run dev:api-server` 默认由 dev 脚本注入 `GENARRATIVE_PROCESS_ROLE=all`,如果外部生成策略为 `queue` 会由同一进程消费队列。`inline` 只用于本地或低并发同步排查,HTTP handler 会直接复用 worker executor,完成后返回 `completed`,但不会落 `external_generation_job`,也不能通过增加 worker 进程扩吞吐。外部生成 worker 使用同一发布包和同一套 SpacetimeDB 配置,按实例数和 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩;生产默认由 `genarrative-external-generation-controller.service` 读取 `get_external_generation_queue_stats_and_return`,按 `claimable_pending + running_active + expired_running` 计算目标 worker 数,并对 `genarrative-external-generation-worker@N.service` 精确执行 `systemctl start/stop`。controller 参数模板是 `deploy/env/external-generation-controller.env.example`:默认保底 `MIN_WORKERS=1`、上限 `MAX_WORKERS=8`、每 worker 目标 `TARGET_JOBS_PER_WORKER=2`、`POLL_INTERVAL_MS=10000`、连续 `SCALE_DOWN_IDLE_ROUNDS=6` 轮完全空闲才缩容;缩容每轮只停止最高编号的一个实例,且不主动停止 `@1`。worker 收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成;若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`,未完成任务才会在 lease 过期后由其它 worker 重领。每个 worker 实例应设置唯一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`,默认会用主机名和 pid 兜底;systemd 生产模板 `deploy/systemd/genarrative-external-generation-worker@.service` 会用 `%H-%i` 生成实例 ID,并把 tracking outbox 隔离到 `/var/lib/genarrative/tracking-outbox/%H-%i`。`Genarrative-Server-Provision` 会安装 worker 模板、controller unit 和两份专属 env 模板,默认 enable 首个 `genarrative-external-generation-worker@1.service` 与 `genarrative-external-generation-controller.service`;首次 API deploy 会在默认 worker pattern 下自动 `enable --now genarrative-external-generation-worker@1.service` 并等待 worker active,同时重启并验活 controller。手动兜底扩容仍可用 `systemctl start genarrative-external-generation-worker@2.service` / `@3.service`,缩容用 `systemctl stop genarrative-external-generation-worker@N.service`;controller 下轮会按队列压力修正到目标实例数。worker 专属参数模板是 `deploy/env/external-generation-worker.env.example`,密钥与 SpacetimeDB 连接默认复用 `/etc/genarrative/api-server.env`。API 发布脚本默认会重启并验活 `genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service`;安装随包 BgFilter、external-generation worker 和 controller unit 前,必须按本次 `--current-link`、`--api-env-file`、`--worker-env-file`、`--controller-env-file` 与 `--bgfilter-worker-env-file` 渲染路径,不能用原始模板覆盖 Server-Provision 已安装的自定义路径。`Genarrative-Api-Deploy` 和 `Genarrative-Full-Build-And-Deploy` 必须同时暴露并透传这三类角色 env 参数,避免独立发布脚本修复后又被流水线默认值截断。若本次只发 HTTP 且不希望滚动 worker,可传 `--no-worker-services`,若不希望重启 controller 可传 `--no-worker-controller`。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 控制空队列轮询间隔,`GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS` 控制单次 lease,worker 会约每三分之一 lease、最长 30 秒续租;该值应覆盖一次心跳网络抖动窗口,不需要大于完整外部生成链路耗时。SpacetimeDB 使用自身事务时间计算 claim/renew/complete/fail,完成和失败回写还会校验 `lease_token` 与未过期 lease,避免同一 job 被过期 worker 覆盖。首版 worker 粒度是单动作单 job,不拆阶段 job;当前外部生成动作覆盖拼图、跳一跳、拼消消、敲木鱼和图片画布编辑器生成 / 抠图入口,纯元信息保存、发布、试玩启动、运行态动作和公开读取继续 inline。图片画布 worker 成功后由后端保存 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`,前端只轮询 job 并重新读取项目快照。当前生成业务失败只做用户重新触发,不做自动业务重试,避免 worker 退款和重试成功之间产生钱包账本漂移。 worker 被硬杀或断电后,lease 过期任务只有尚未耗尽 `max_attempts` 才由其它 worker 重领;最终 attempt 的过期任务由 claim transaction 直接失败并结算,不会继续执行外部 provider。当前业务入队点默认 `max_attempts=1`,因此一次已领取任务若以 lease 过期结束,后续 poll 负责终态与退款收口,而不是发起第二次 provider 请求。 diff --git a/jenkins/Jenkinsfile.production-api-deploy b/jenkins/Jenkinsfile.production-api-deploy index aef01f183..1c72797a0 100644 --- a/jenkins/Jenkinsfile.production-api-deploy +++ b/jenkins/Jenkinsfile.production-api-deploy @@ -24,6 +24,8 @@ pipeline { string(name: 'HEALTH_URL', defaultValue: 'http://127.0.0.1:8082/readyz', description: '本机 readiness 检查地址') string(name: 'API_ENV_FILE', defaultValue: '/etc/genarrative/api-server.env', description: 'api-server 环境文件') string(name: 'WORKER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-worker.env', description: '外部生成 worker 环境文件') + string(name: 'CONTROLLER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-controller.env', description: '外部生成 controller 环境文件') + string(name: 'BGFILTER_WORKER_ENV_FILE', defaultValue: '/etc/genarrative/bgfilter-worker.env', description: 'BgFilter worker 环境文件') string(name: 'DATABASE', defaultValue: 'genarrative-prod', description: 'api-server 连接的 SpacetimeDB database') string(name: 'SPACETIME_SERVER_URL', defaultValue: 'http://127.0.0.1:3101', description: 'api-server 连接的 SpacetimeDB server URL') } @@ -53,7 +55,7 @@ pipeline { if (!(params.DATABASE.trim() ==~ /^[a-z0-9]+(-[a-z0-9]+)*$/)) { error("DATABASE 必须匹配 ^[a-z0-9]+(-[a-z0-9]+)*\$: ${params.DATABASE}") } - [API_ENV_FILE: params.API_ENV_FILE, WORKER_ENV_FILE: params.WORKER_ENV_FILE].each { name, value -> + [API_ENV_FILE: params.API_ENV_FILE, WORKER_ENV_FILE: params.WORKER_ENV_FILE, CONTROLLER_ENV_FILE: params.CONTROLLER_ENV_FILE, BGFILTER_WORKER_ENV_FILE: params.BGFILTER_WORKER_ENV_FILE].each { name, value -> def envFile = value?.trim() if (!envFile || envFile == '/' || !(envFile ==~ /^\/(?!.*\.\.)[A-Za-z0-9._\/-]+$/)) { error("${name} 必须是 Linux 绝对路径且不能包含 ..: ${envFile}") @@ -117,6 +119,8 @@ pipeline { --health-url "${HEALTH_URL}" \ --api-env-file "${API_ENV_FILE:-/etc/genarrative/api-server.env}" \ --worker-env-file "${WORKER_ENV_FILE:-/etc/genarrative/external-generation-worker.env}" \ + --controller-env-file "${CONTROLLER_ENV_FILE:-/etc/genarrative/external-generation-controller.env}" \ + --bgfilter-worker-env-file "${BGFILTER_WORKER_ENV_FILE:-/etc/genarrative/bgfilter-worker.env}" \ --database "${DATABASE}" \ --spacetime-server-url "${SPACETIME_SERVER_URL:-http://127.0.0.1:3101}" ' diff --git a/jenkins/Jenkinsfile.production-full-build-and-deploy b/jenkins/Jenkinsfile.production-full-build-and-deploy index aa1968f20..64b032762 100644 --- a/jenkins/Jenkinsfile.production-full-build-and-deploy +++ b/jenkins/Jenkinsfile.production-full-build-and-deploy @@ -41,6 +41,8 @@ pipeline { string(name: 'SPACETIME_RUN_AS_USER', defaultValue: 'spacetimedb', description: 'Stdb 发布使用的本机用户') string(name: 'API_ENV_FILE', defaultValue: '/etc/genarrative/api-server.env', description: 'API 与 Stdb publish 共用的 api-server 环境文件') string(name: 'WORKER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-worker.env', description: 'API 与 Stdb publish 共用的 worker 环境文件') + string(name: 'CONTROLLER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-controller.env', description: 'API deploy 使用的外部生成 controller 环境文件') + string(name: 'BGFILTER_WORKER_ENV_FILE', defaultValue: '/etc/genarrative/bgfilter-worker.env', description: 'API deploy 使用的 BgFilter worker 环境文件') choice(name: 'STDB_API_ROLLOUT_MODE', choices: ['normal', 'pause-after-stdb'], description: '定时任务默认 normal 完整发布 dev;人工维护窗口才选择 pause-after-stdb') string(name: 'STDB_API_ROLLOUT_APPROVERS', defaultValue: '', description: 'pause-after-stdb 必填:允许放行 API 的 Jenkins 用户/组,多个值用逗号分隔') booleanParam(name: 'EXIT_MAINTENANCE_MODE_AFTER_COMPLETION', defaultValue: true, description: '完整发布成功后是否退出维护模式;取消勾选会保留维护页,便于内网验收后人工恢复公网') @@ -233,6 +235,8 @@ pipeline { booleanParam(name: 'KEEP_MAINTENANCE_MODE', value: true), string(name: 'API_ENV_FILE', value: params.API_ENV_FILE ?: '/etc/genarrative/api-server.env'), string(name: 'WORKER_ENV_FILE', value: params.WORKER_ENV_FILE ?: '/etc/genarrative/external-generation-worker.env'), + string(name: 'CONTROLLER_ENV_FILE', value: params.CONTROLLER_ENV_FILE ?: '/etc/genarrative/external-generation-controller.env'), + string(name: 'BGFILTER_WORKER_ENV_FILE', value: params.BGFILTER_WORKER_ENV_FILE ?: '/etc/genarrative/bgfilter-worker.env'), string(name: 'DATABASE', value: params.DATABASE), string(name: 'SPACETIME_SERVER_URL', value: params.SPACETIME_SERVER_URL ?: ''), ] diff --git a/scripts/check-production-api-deploy.mjs b/scripts/check-production-api-deploy.mjs index d6ccd35d3..b13be4b2d 100644 --- a/scripts/check-production-api-deploy.mjs +++ b/scripts/check-production-api-deploy.mjs @@ -352,63 +352,87 @@ function assertDeployCopiesPingoraDirectReleaseDependencies() { ), 'API deploy 必须把随包 BgFilter worker 单元安装到 systemd unit 目录。', ); - // 安装的 unit 必须按本次部署参数渲染(fixture 全程使用自定义路径): - // 含 fixture 的 current 链接与 env 路径,不得残留模板默认字面量—— - // 否则自定义 --current-link/env 部署会让服务启动旧二进制或加载默认配置。 - const renderedUnits = [ - ['genarrative-bgfilter-worker.service', fixture.bgfilterWorkerEnvFile], - [ + const externalGenerationWorkerUnit = readFileSync( + path.join( + fixture.systemdUnitDir, 'genarrative-external-generation-worker@.service', - fixture.externalGenerationWorkerEnvFile, - ], - [ + ), + 'utf8', + ); + const externalGenerationControllerUnit = readFileSync( + path.join( + fixture.systemdUnitDir, 'genarrative-external-generation-controller.service', - fixture.controllerEnvFile, - ], - ]; - for (const [unitName, roleEnvFile] of renderedUnits) { - const unitContent = readFileSync( - path.join(fixture.systemdUnitDir, unitName), - 'utf8', - ); - assertIncludes( - unitContent, - `WorkingDirectory=${fixture.currentLink}`, - `${unitName} 的工作目录必须渲染为 --current-link。`, - ); - assertIncludes( - unitContent, - `${fixture.currentLink}/api-server`, - `${unitName} 的 ExecStart 必须指向 --current-link 下的二进制。`, - ); - assertIncludes( - unitContent, - `EnvironmentFile=${fixture.apiEnvFile}`, - `${unitName} 必须加载 --api-env-file 指定的共享 env。`, - ); - assertIncludes( - unitContent, - roleEnvFile, - `${unitName} 必须加载本角色 env 参数指定的文件。`, - ); - for (const templateDefault of [ - '/opt/genarrative/current', - '/etc/genarrative/api-server.env', - '/etc/genarrative/bgfilter-worker.env', - '/etc/genarrative/external-generation-worker.env', - '/etc/genarrative/external-generation-controller.env', - ]) { - if (unitContent.includes(templateDefault)) { - failures.push( - `${unitName} 安装后不得残留模板默认路径 ${templateDefault}:unit 必须按部署参数渲染。`, - ); - } - } - } + ), + 'utf8', + ); const bgfilterUnit = readFileSync( path.join(fixture.systemdUnitDir, 'genarrative-bgfilter-worker.service'), 'utf8', ); + + for (const [label, unit] of [ + ['外部生成 worker unit', externalGenerationWorkerUnit], + ['外部生成 controller unit', externalGenerationControllerUnit], + ['BgFilter worker unit', bgfilterUnit], + ]) { + assertIncludes( + unit, + `WorkingDirectory=${fixture.currentLink}`, + `${label} 必须使用本次部署的自定义 current link。`, + ); + assertIncludes( + unit, + `${fixture.currentLink}/api-server`, + `${label} 必须启动本次部署的自定义 current link 下的二进制。`, + ); + assertIncludes( + unit, + `EnvironmentFile=${fixture.apiEnvFile}`, + `${label} 必须加载本次部署指定的 API env。`, + ); + assertNotIncludes( + unit, + '/opt/genarrative/current', + `${label} 安装后不得继续引用模板默认 current link。`, + ); + assertNotIncludes( + unit, + '/etc/genarrative/api-server.env', + `${label} 安装后不得继续引用模板默认 API env。`, + ); + } + + assertIncludes( + externalGenerationWorkerUnit, + `EnvironmentFile=-${fixture.externalGenerationWorkerEnvFile}`, + '外部生成 worker unit 必须加载本次部署指定的专属 env。', + ); + assertNotIncludes( + externalGenerationWorkerUnit, + '/etc/genarrative/external-generation-worker.env', + '外部生成 worker unit 安装后不得继续引用模板默认专属 env。', + ); + assertIncludes( + externalGenerationControllerUnit, + `EnvironmentFile=-${fixture.externalGenerationControllerEnvFile}`, + '外部生成 controller unit 必须加载本次部署指定的专属 env。', + ); + assertNotIncludes( + externalGenerationControllerUnit, + '/etc/genarrative/external-generation-controller.env', + '外部生成 controller unit 安装后不得继续引用模板默认专属 env。', + ); + assertIncludes( + bgfilterUnit, + `EnvironmentFile=${fixture.bgfilterWorkerEnvFile}`, + 'BgFilter unit 必须加载本次部署指定的专属 env。', + ); + assertNotIncludes( + bgfilterUnit, + '/etc/genarrative/bgfilter-worker.env', + 'BgFilter unit 安装后不得继续引用模板默认专属 env。', + ); const sharedEnvIndex = bgfilterUnit.indexOf( `EnvironmentFile=${fixture.apiEnvFile}`, ); @@ -2046,16 +2070,16 @@ function prepareFixture(name) { 'etc', 'external-generation-worker.env', ); + const externalGenerationControllerEnvFile = path.join( + root, + 'etc', + 'external-generation-controller.env', + ); const bgfilterWorkerEnvFile = path.join( root, 'etc', 'bgfilter-worker.env', ); - const controllerEnvFile = path.join( - root, - 'etc', - 'external-generation-controller.env', - ); const bgfilterTokenFile = path.join(root, 'etc', 'bgfilter-worker.token'); const pingoraEnvFile = path.join(root, 'etc', 'pingora-gateway.env'); const maintenanceFile = path.join(root, 'maintenance', 'enabled'); @@ -2102,8 +2126,8 @@ function prepareFixture(name) { 'utf8', ); writeFileSync( - controllerEnvFile, - 'GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_PLACEHOLDER=1\n', + externalGenerationControllerEnvFile, + 'GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MAX_WORKERS=4\n', 'utf8', ); writeFileSync( @@ -2461,8 +2485,8 @@ function prepareFixture(name) { currentLink, apiEnvFile, externalGenerationWorkerEnvFile, + externalGenerationControllerEnvFile, bgfilterWorkerEnvFile, - controllerEnvFile, bgfilterTokenFile, pingoraEnvFile, maintenanceFile, @@ -2538,10 +2562,10 @@ function runDeploy(fixture, options = {}) { options.apiEnvFile ?? fixture.apiEnvFile, '--worker-env-file', fixture.externalGenerationWorkerEnvFile, + '--controller-env-file', + fixture.externalGenerationControllerEnvFile, '--bgfilter-worker-env-file', fixture.bgfilterWorkerEnvFile, - '--controller-env-file', - options.controllerEnvFile ?? fixture.controllerEnvFile, // 缺省不传 --bgfilter-worker-health-url,与真实 Jenkins 调用一致: // 让预检从已校验的 worker env HOST/PORT 派生 readiness URL。 ...(options.bgfilterWorkerHealthUrl === undefined @@ -2663,6 +2687,12 @@ function assertIncludes(content, needle, reason) { } } +function assertNotIncludes(content, needle, reason) { + if (content.includes(needle)) { + failures.push(`${reason} 不应包含: ${needle}`); + } +} + function assertPingoraEnvProductionDefaults(filePath, label) { if (!existsSync(filePath)) { return; diff --git a/scripts/check-production-ops-guardrails.mjs b/scripts/check-production-ops-guardrails.mjs index 15705c668..00d92b4b3 100644 --- a/scripts/check-production-ops-guardrails.mjs +++ b/scripts/check-production-ops-guardrails.mjs @@ -233,6 +233,12 @@ const checks = [ reason: 'API readiness 单次请求必须有超时,避免端口已建立但服务尚未响应时绕过重试上限无限挂起。', }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: + 'on_exit() {\n local exit_code=$?\n cleanup_rendered_systemd_unit', + reason: 'API deploy 被中断时必须清理尚未安装完成的临时 systemd unit。', + }, { file: 'jenkins/Jenkinsfile.production-api-deploy', includes: @@ -244,6 +250,54 @@ const checks = [ includes: 'maintenance_deploy_args+=(--keep-maintenance-mode)', reason: 'API Deploy Job 必须把保持维护参数传给发布产物内的部署脚本。', }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + "string(name: 'CONTROLLER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-controller.env'", + reason: 'API Deploy Job 必须暴露外部生成 controller env 路径。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + "string(name: 'BGFILTER_WORKER_ENV_FILE', defaultValue: '/etc/genarrative/bgfilter-worker.env'", + reason: 'API Deploy Job 必须暴露 BgFilter worker env 路径。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + '--controller-env-file "${CONTROLLER_ENV_FILE:-/etc/genarrative/external-generation-controller.env}"', + reason: 'API Deploy Job 必须把 controller env 路径传给发布脚本。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + '--bgfilter-worker-env-file "${BGFILTER_WORKER_ENV_FILE:-/etc/genarrative/bgfilter-worker.env}"', + reason: 'API Deploy Job 必须把 BgFilter worker env 路径传给发布脚本。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + "string(name: 'CONTROLLER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-controller.env'", + reason: 'Full Job 必须暴露外部生成 controller env 路径。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + "string(name: 'BGFILTER_WORKER_ENV_FILE', defaultValue: '/etc/genarrative/bgfilter-worker.env'", + reason: 'Full Job 必须暴露 BgFilter worker env 路径。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + "string(name: 'CONTROLLER_ENV_FILE', value: params.CONTROLLER_ENV_FILE ?: '/etc/genarrative/external-generation-controller.env')", + reason: 'Full Job 必须把 controller env 路径传给 API Deploy Job。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + "string(name: 'BGFILTER_WORKER_ENV_FILE', value: params.BGFILTER_WORKER_ENV_FILE ?: '/etc/genarrative/bgfilter-worker.env')", + reason: 'Full Job 必须把 BgFilter worker env 路径传给 API Deploy Job。', + }, { file: 'scripts/jenkins-server-provision.sh', includes: 'ensure_runtime_bootstrap_secret_file_env', diff --git a/scripts/deploy/production-api-deploy.sh b/scripts/deploy/production-api-deploy.sh index b374cd534..b3713bbab 100644 --- a/scripts/deploy/production-api-deploy.sh +++ b/scripts/deploy/production-api-deploy.sh @@ -5,7 +5,7 @@ set -euo pipefail usage() { cat <<'EOF' 用法: - ./scripts/deploy/production-api-deploy.sh --source-dir build/ [--version ] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url ] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--controller-env-file /etc/genarrative/external-generation-controller.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode] + ./scripts/deploy/production-api-deploy.sh --source-dir build/ [--version ] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--bgfilter-worker-service genarrative-bgfilter-worker.service] [--bgfilter-worker-health-url ] [--bgfilter-worker-env-file /etc/genarrative/bgfilter-worker.env] [--no-bgfilter-worker] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--controller-env-file /etc/genarrative/external-generation-controller.env] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode] 说明: 进入维护模式,校验并发布 api-server 单文件,更新 current 链接,重启 systemd 服务并执行 readiness 检查。 @@ -37,6 +37,10 @@ require_absolute_path() { echo "[production-api-deploy] ${label} 必须使用绝对路径: ${value}" >&2 exit 1 fi + if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then + echo "[production-api-deploy] ${label} 不能包含换行符。" >&2 + exit 1 + fi } validate_spacetime_database_name() { @@ -897,21 +901,11 @@ ensure_default_worker_service() { systemctl enable --now "${default_service}" } -escape_sed_replacement() { - printf "%s" "$1" | sed "s/[&|]/\\\\&/g" -} - -# 与 provision 的 render_*_service 同语义:unit 文件在仓库里是模板,安装前必须把 -# 默认 current 链接与 env 路径渲染为本次部署参数,自定义 --current-link/env 时 -# 不得让服务启动旧二进制或加载默认 env(默认参数下渲染输出与模板逐字节一致)。 install_release_systemd_unit() { local source_path="$1" local unit_name="$2" local label="$3" - shift 3 local unit_dir="${GENARRATIVE_SYSTEMD_UNIT_DIR:-/etc/systemd/system}" - local sed_args=() - local template_default replacement rendered if [[ ! -f "${source_path}" ]]; then echo "[production-api-deploy] 发布产物缺少${label}: ${source_path}" >&2 @@ -921,27 +915,61 @@ install_release_systemd_unit() { echo "[production-api-deploy] systemd unit 目录必须使用绝对路径: ${unit_dir}" >&2 return 1 fi - while [[ $# -gt 0 ]]; do - template_default="$1" - replacement="${2:?unit 渲染替换必须成对提供模板默认值与实际值}" - shift 2 - if [[ -z "${replacement}" ]]; then - echo "[production-api-deploy] ${label} 渲染值不能为空: ${template_default}" >&2 - return 1 - fi - sed_args+=(-e "s|${template_default}|$(escape_sed_replacement "${replacement}")|g") - done - rendered="$(mktemp)" - if [[ "${#sed_args[@]}" -gt 0 ]]; then - sed "${sed_args[@]}" "${source_path}" > "${rendered}" - else - cat "${source_path}" > "${rendered}" - fi echo "[production-api-deploy] 安装${label}: ${unit_name}" - run_privileged install -d -m 0755 "${unit_dir}" - run_privileged install -m 0644 "${rendered}" "${unit_dir}/${unit_name}" - rm -f "${rendered}" + if ! run_privileged install -d -m 0755 "${unit_dir}"; then + return 1 + fi + run_privileged install -m 0644 "${source_path}" "${unit_dir}/${unit_name}" +} + +render_and_install_release_systemd_unit() { + local source_path="$1" + local unit_name="$2" + local label="$3" + local rendered_path + local install_status=0 + shift 3 + + if [[ ! -f "${source_path}" ]]; then + echo "[production-api-deploy] 发布产物缺少${label}: ${source_path}" >&2 + return 1 + fi + if [[ "$#" -eq 0 || $(( $# % 2 )) -ne 0 ]]; then + echo "[production-api-deploy] ${label} 缺少成对的模板路径和实际路径。" >&2 + return 1 + fi + + RENDERED_SYSTEMD_UNIT_FILE="$(mktemp)" + rendered_path="${RENDERED_SYSTEMD_UNIT_FILE}" + if ! python3 - "${source_path}" "${rendered_path}" "$@" <<'PY' +import re +import sys +from pathlib import Path + +source_path = Path(sys.argv[1]) +rendered_path = Path(sys.argv[2]) +replacements = sys.argv[3:] +content = source_path.read_text(encoding="utf-8") +replacement_map = dict(zip(replacements[::2], replacements[1::2])) + +for template_path in replacement_map: + if template_path not in content: + raise SystemExit(f"systemd unit 模板缺少占位路径: {template_path}") + +pattern = re.compile("|".join(re.escape(path) for path in replacement_map)) +content = pattern.sub(lambda match: replacement_map[match.group(0)], content) + +rendered_path.write_text(content, encoding="utf-8") +PY + then + cleanup_rendered_systemd_unit + return 1 + fi + + install_release_systemd_unit "${rendered_path}" "${unit_name}" "${label}" || install_status=$? + cleanup_rendered_systemd_unit + return "${install_status}" } install_worker_systemd_units() { @@ -949,38 +977,43 @@ install_worker_systemd_units() { local pattern="$2" local controller_service="$3" local bgfilter_service="$4" + local current_link="$5" + local api_env_file="$6" + local worker_env_file="$7" + local controller_env_file="$8" + local bgfilter_env_file="$9" local installed_any=0 if [[ "${bgfilter_service}" == "genarrative-bgfilter-worker.service" ]]; then - install_release_systemd_unit \ + render_and_install_release_systemd_unit \ "${release_dir}/deploy/systemd/genarrative-bgfilter-worker.service" \ "genarrative-bgfilter-worker.service" \ "BgFilter worker systemd 单元" \ - "/opt/genarrative/current" "${CURRENT_LINK}" \ - "/etc/genarrative/api-server.env" "${API_ENV_FILE}" \ - "/etc/genarrative/bgfilter-worker.env" "${BGFILTER_WORKER_ENV_FILE}" + "/opt/genarrative/current" "${current_link}" \ + "/etc/genarrative/api-server.env" "${api_env_file}" \ + "/etc/genarrative/bgfilter-worker.env" "${bgfilter_env_file}" installed_any=1 fi if [[ "${pattern}" == "genarrative-external-generation-worker@*.service" ]]; then - install_release_systemd_unit \ + render_and_install_release_systemd_unit \ "${release_dir}/deploy/systemd/genarrative-external-generation-worker@.service" \ "genarrative-external-generation-worker@.service" \ "外部生成 worker systemd 模板" \ - "/opt/genarrative/current" "${CURRENT_LINK}" \ - "/etc/genarrative/api-server.env" "${API_ENV_FILE}" \ - "/etc/genarrative/external-generation-worker.env" "${WORKER_ENV_FILE}" + "/opt/genarrative/current" "${current_link}" \ + "/etc/genarrative/api-server.env" "${api_env_file}" \ + "/etc/genarrative/external-generation-worker.env" "${worker_env_file}" installed_any=1 fi if [[ "${controller_service}" == "genarrative-external-generation-controller.service" ]]; then - install_release_systemd_unit \ + render_and_install_release_systemd_unit \ "${release_dir}/deploy/systemd/genarrative-external-generation-controller.service" \ "genarrative-external-generation-controller.service" \ "外部生成 worker controller systemd 单元" \ - "/opt/genarrative/current" "${CURRENT_LINK}" \ - "/etc/genarrative/api-server.env" "${API_ENV_FILE}" \ - "/etc/genarrative/external-generation-controller.env" "${CONTROLLER_ENV_FILE}" + "/opt/genarrative/current" "${current_link}" \ + "/etc/genarrative/api-server.env" "${api_env_file}" \ + "/etc/genarrative/external-generation-controller.env" "${controller_env_file}" installed_any=1 fi @@ -1143,6 +1176,7 @@ MAINTENANCE_FILE="${GENARRATIVE_MAINTENANCE_FILE:-/var/lib/genarrative/maintenan CURRENT_LINK_SWITCHED=0 RELEASE_DIR="" STAGING_RELEASE_DIR="" +RENDERED_SYSTEMD_UNIT_FILE="" while [[ $# -gt 0 ]]; do case "$1" in @@ -1194,6 +1228,10 @@ while [[ $# -gt 0 ]]; do WORKER_CONTROLLER_SERVICE="${2:?缺少 --worker-controller-service 的值}" shift 2 ;; + --controller-env-file) + CONTROLLER_ENV_FILE="${2:?缺少 --controller-env-file 的值}" + shift 2 + ;; --no-worker-controller) WORKER_CONTROLLER_SERVICE="" shift @@ -1227,10 +1265,6 @@ while [[ $# -gt 0 ]]; do WORKER_ENV_FILE="${2:?缺少 --worker-env-file 的值}" shift 2 ;; - --controller-env-file) - CONTROLLER_ENV_FILE="${2:?缺少 --controller-env-file 的值}" - shift 2 - ;; --database) DATABASE="${2:?缺少 --database 的值}" shift 2 @@ -1254,12 +1288,12 @@ require_absolute_path "${API_ENV_FILE}" "--api-env-file" if [[ -n "${WORKER_ENV_FILE}" ]]; then require_absolute_path "${WORKER_ENV_FILE}" "--worker-env-file" fi -if [[ -n "${BGFILTER_WORKER_ENV_FILE}" ]]; then - require_absolute_path "${BGFILTER_WORKER_ENV_FILE}" "--bgfilter-worker-env-file" -fi if [[ -n "${CONTROLLER_ENV_FILE}" ]]; then require_absolute_path "${CONTROLLER_ENV_FILE}" "--controller-env-file" fi +if [[ -n "${BGFILTER_WORKER_ENV_FILE}" ]]; then + require_absolute_path "${BGFILTER_WORKER_ENV_FILE}" "--bgfilter-worker-env-file" +fi if [[ -n "${DATABASE}" ]]; then validate_spacetime_database_name "${DATABASE}" @@ -1312,8 +1346,16 @@ cleanup_staging_release() { fi } +cleanup_rendered_systemd_unit() { + if [[ -n "${RENDERED_SYSTEMD_UNIT_FILE:-}" && -f "${RENDERED_SYSTEMD_UNIT_FILE}" ]]; then + rm -f "${RENDERED_SYSTEMD_UNIT_FILE}" + fi + RENDERED_SYSTEMD_UNIT_FILE="" +} + on_exit() { local exit_code=$? + cleanup_rendered_systemd_unit if [[ "${exit_code}" -ne 0 && "${DEPLOY_COMPLETED}" -ne 1 ]]; then cleanup_staging_release if [[ "${MAINTENANCE_ENABLED_BY_DEPLOY}" -eq 1 && "${CURRENT_LINK_SWITCHED}" -ne 1 ]]; then @@ -1620,7 +1662,16 @@ if [[ "${PINGORA_INCLUDED}" -eq 1 ]]; then ensure_pingora_shadow_service "${PINGORA_SERVICE_NAME}" "${PINGORA_SHADOW_ENV_FILE}" fi -install_worker_systemd_units "${RELEASE_DIR}" "${WORKER_SERVICE_PATTERN}" "${WORKER_CONTROLLER_SERVICE}" "${BGFILTER_WORKER_SERVICE}" +install_worker_systemd_units \ + "${RELEASE_DIR}" \ + "${WORKER_SERVICE_PATTERN}" \ + "${WORKER_CONTROLLER_SERVICE}" \ + "${BGFILTER_WORKER_SERVICE}" \ + "${CURRENT_LINK}" \ + "${API_ENV_FILE}" \ + "${WORKER_ENV_FILE}" \ + "${CONTROLLER_ENV_FILE}" \ + "${BGFILTER_WORKER_ENV_FILE}" restart_and_wait_for_bgfilter_worker "${BGFILTER_WORKER_SERVICE}" "${BGFILTER_WORKER_HEALTH_URL}"