diff --git a/deploy/nginx/README.md b/deploy/nginx/README.md index 4000e7481..7f8e0f8d0 100644 --- a/deploy/nginx/README.md +++ b/deploy/nginx/README.md @@ -46,7 +46,8 @@ - Pingora 网关默认仍只监听本机 shadow 端口，不绑定公网 `80/443`。 - 需要评估直连入口时，显式配置 `GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN`、`GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` 和 `GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE`；网关只读取现有证书文件，不负责 Certbot / ACME 自动续期。 - 需要 HTTP 到 HTTPS 跳转时，再显式配置 `GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN`；该入口除 `/.well-known/acme-challenge/*` 外统一 301 到 HTTPS，ACME challenge 仍从 `GENARRATIVE_PINGORA_GATEWAY_ACME_ROOT` 精确读取。 -- API release 必须把 `api-server.sha256`、可选 `pingora-gateway.sha256`、`scripts/check-pingora-release-readiness.mjs`、`scripts/check-pingora-canary-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-production-health-patrol-env.mjs`、`scripts/deploy/pingora-health-patrol-env-switch.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/` 一起复制到 `/opt/genarrative/current`；current release 自审、直连启用、health patrol env 切换、canary live、access log 对账、cutover 状态快照和证据包脚本都从 current release 推导依赖路径，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。 +- API release 必须把 `api-server.sha256`、可选 `pingora-gateway.sha256`、`scripts/check-pingora-release-readiness.mjs`、`scripts/check-pingora-canary-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-production-health-patrol-env.mjs`、`scripts/deploy/pingora-health-patrol-env-switch.mjs`、`scripts/deploy/pingora-tls-cert-sync.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/` 一起复制到 `/opt/genarrative/current`；current release 自审、直连启用、health patrol env 切换、TLS 证书同步、canary live、access log 对账、cutover 状态快照和证据包脚本都从 current release 推导依赖路径，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。 +- 目标机如果使用 Certbot / Let’s Encrypt 证书，不要直接放宽 `/etc/letsencrypt/live` 或 `archive` 的目录 / 私钥权限。先用随包脚本把 root 可读的 live symlink 解析并复制到 Pingora 私有目录，例如 `sudo -n node -- /opt/genarrative/current/scripts/deploy/pingora-tls-cert-sync.mjs --apply --source-cert-file /etc/letsencrypt/live/<域名>/fullchain.pem --source-key-file /etc/letsencrypt/live/<域名>/privkey.pem --target-dir /etc/genarrative/pingora-tls/<域名>`；脚本默认 dry-run，`--apply` 才原子写入 `fullchain.pem` / `privkey.pem`，目标目录默认 `root:genarrative 0750`，文件默认 `root:genarrative 0640`，并复核 `genarrative` 服务用户可读。随后把 `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` / `TLS_KEY_FILE` 指向 `/etc/genarrative/pingora-tls/<域名>/` 下的副本。 - 默认 systemd service 以 `genarrative` 非 root 用户运行，不具备绑定 `80/443` 的能力。Server-Provision 只安装 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 作为人工启用模板；切直连低端口时必须先确认 `genarrative` 可读证书文件、Nginx 已释放 `80/443`，再执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --apply --preflight-env-file /etc/genarrative/pingora-gateway.env --preflight-check-cert-readable --preflight-check-service-env-file --preflight-check-service-user-cert-readable --preflight-check-service-binary-executable --preflight-check-ports-free --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；脚本会先执行随包 current release 自审，确认 `pingora-gateway` 可执行且 systemd `ExecStart` 指向 current release，失败时不会安装 drop-in。自审通过后才安装 drop-in、`systemctl daemon-reload` 并重启 Pingora；脚本会用 `systemctl cat` 确认 `CAP_NET_BIND_SERVICE` drop-in 已进入 systemd 最终配置，用 `systemctl show ... ExecStart` 确认最终 service 指向随包主 service 模板中的 current release `pingora-gateway`，用 `systemctl is-active` 确认 Pingora active，并执行 direct live smoke，强制覆盖 HTTPS / HTTP redirect / ACME / WSS 101 和 Pingora access log request_id 落盘。 - `npm run check:pingora-gateway-smoke` 会用临时自签证书覆盖 TLS 直连、HTTP redirect 和 WSS subscribe 行为；正式公网切换前仍必须先走 Nginx canary 和 release readiness 门禁。 - 目标机直连入口启用后先运行 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`，再运行 `npm run check:pingora-direct-live`，检查 HTTPS 根路径、HTTP/2 ALPN、代表性 API、SpacetimeDB identity、WSS subscribe 握手、generated 拒绝、公开 `/healthz` 拒绝、可选 shadow probe，以及 HTTP 301 / ACME challenge；真实 SpacetimeDB 对 `GET /v1/identity` 返回 `405 Method Not Allowed` 属于可接受语义，direct live 只把它作为路径转发代表，不要求该 GET 创建 identity；release readiness 可用 `--require-direct --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log --direct-health-patrol-env-file /etc/genarrative/health-patrol.env --direct-preflight-env-file /etc/genarrative/pingora-gateway.env --direct-preflight-systemd --direct-preflight-check-cert-readable --direct-preflight-check-service-env-file --direct-preflight-check-service-user-cert-readable --direct-preflight-check-service-binary-executable --direct-preflight-check-ports-free` 强制纳入同一条门禁，漏掉 direct HTTP base URL、正式域名 Host/SNI、redirect Location host、目标 SpacetimeDB 库、Pingora access log request_id 落盘、health patrol env direct 模式复核、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户 / systemd 服务用户证书可读检查、service 二进制可执行检查或 80/443 端口释放检查都会直接失败。 diff --git a/deploy/pingora/pingora-gateway.env.example b/deploy/pingora/pingora-gateway.env.example index a60b9d456..1c69ad8cf 100644 --- a/deploy/pingora/pingora-gateway.env.example +++ b/deploy/pingora/pingora-gateway.env.example @@ -4,11 +4,13 @@ GENARRATIVE_PINGORA_GATEWAY_LISTEN=127.0.0.1:18081 # 可选：直连 HTTPS 入口。默认关闭；启用时必须同时配置 cert/key，并把 FORWARDED_PROTO 设为 https。 -# cert/key 必须允许 genarrative 用户读取；80/443 必须先从 Nginx 或其它进程释放。 +# cert/key 必须允许 genarrative 用户读取；不要直接 chmod Let's Encrypt live/archive 原始目录。 +# 可先用随包 scripts/deploy/pingora-tls-cert-sync.mjs 把证书同步到 /etc/genarrative/pingora-tls//。 +# 80/443 必须先从 Nginx 或其它进程释放；高端口 loopback 演练可先使用 127.0.0.1:18443 / 127.0.0.1:18080。 # 如果要绑定 443/80，必须另行人工启用 systemd direct-entry drop-in 授予 CAP_NET_BIND_SERVICE。 # GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN=0.0.0.0:18443 -# GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE=/etc/letsencrypt/live/genarrative.world/fullchain.pem -# GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE=/etc/letsencrypt/live/genarrative.world/privkey.pem +# GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE=/etc/genarrative/pingora-tls/genarrative.world/fullchain.pem +# GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE=/etc/genarrative/pingora-tls/genarrative.world/privkey.pem # 可选：直连 HTTP 只做 301 到 HTTPS；启用时必须已经配置 TLS_LISTEN，ACME challenge 仍从 ACME_ROOT 静态读取。 # GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN=0.0.0.0:18080 # GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_TARGET_SCHEME=https diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index a65fc8c9e..55d6e0d82 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -20,6 +20,7 @@ - 现象：`/etc/genarrative/pingora-gateway.env` 已把 `GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN` / `HTTP_REDIRECT_LISTEN` 改到 `0.0.0.0:443` / `0.0.0.0:80`，但 `genarrative-pingora-gateway.service` 启动失败，日志出现低端口绑定权限错误。 - 原因：默认 service 用非 root `genarrative` 用户运行，并且主模板为了保持 shadow 安全边界不带 `CAP_NET_BIND_SERVICE`。低端口直连必须通过显式 systemd drop-in 单独授予 capability；同时 Certbot 私钥默认未必允许 `genarrative` 读取，Nginx 也可能仍占用 `80/443`。另一个常见误区是 API release 只带 `pingora-direct-enable.sh` / rollback 壳脚本，却漏带 `pingora-current-release-audit.mjs`、`check-pingora-direct-preflight.mjs`、`check-pingora-direct-live.mjs`、`deploy/systemd/`、`deploy/env/` 或 `deploy/pingora/`，导致从 `/opt/genarrative/current` 启用时依赖 Jenkins 工作区、源码 checkout 或 `/etc` 里某份参考模板；或者 release 已经包含新版 `pingora-gateway`，但已运行的 shadow / canary / direct service 没有随 `current` 链接切换重启，仍在跑旧二进制。Server-Provision 安装到 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 的 drop-in 只用于人工审阅和显式覆盖；直连启用脚本默认必须读取 current release 随包 `deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`，否则旧 `/etc` 模板会掩盖发布包缺失。API deploy 脚本本身也不能继续用部署工作区根部的 `scripts/deploy/production-api-deploy.sh`，否则 Jenkins workspace 里的脚本会掩盖 `build/` 发布包缺少 deploy / maintenance 同目录脚本的问题；备份脚本、健康巡检脚本和 env 示例目录同样不能从部署工作区兜底，切换命令证据脚本也不能从部署工作区兜底，否则 current release 会和上游构建归档漂移。Pingora 直连依赖、备份脚本、巡检脚本、env 示例目录和 API deploy 执行入口都必须来自上游发布产物；随包 `api-server.sha256` 和可选 `pingora-gateway.sha256` 也必须复制进 current release，供随包 current release 自审校验二进制；随包 `deploy/pingora/pingora-gateway.env.example` 也不能只检查存在，还要保持 gzip-only、不信任 XFF、前置代理确认关闭、接流保护开启和空 probe token 这些生产安全默认值；`production-api-deploy.sh` 发现缺失时应 fail-fast 并保持维护模式，不应从部署工作区兜底补齐；所有 API 发布包都必须携带 `release-manifest.json` 且登记 `api-server` artifact，发布包包含 Pingora 时还必须登记 `pingora-gateway` artifact，否则 deploy 应在切换 current 前失败；deploy 必须要求 release root、current link 和 api env file 都是绝对路径，release version 以数字或字母开头并拒绝点目录，再先写 staging release，全部复制完成后用非合并语义提升为正式 release，失败时清理 staging 且不留下正式 release，同版本 release 已存在、提升前竞态出现或 current 路径不是符号链接时拒绝覆盖 / 合并，避免旧文件混入 current；发布包包含 Pingora 且服务已 active 时，deploy 必须在 current 链接切换后 `try-restart` 并复核 active，服务未运行时只跳过，不主动拉起。 - 处理：确认真实 TLS 证书和 redirect env 已写入 `/etc/genarrative/pingora-gateway.env`、service 模板和 `systemctl cat` 最终配置读取的 `EnvironmentFile=` 都包含这份 env、当前执行用户和 `genarrative-pingora-gateway.service` 的 `User=` 服务用户都能读取证书链 / 私钥、current release 的 `pingora-gateway` 已存在且可执行、Nginx 或其它进程已释放 `80/443` 后，先用 `npm run plan:pingora-direct-cutover -- --require-direct ...` 生成只读 JSON runbook，并逐条审阅 Host 与回退巡检入口确认、current release 自包含自审、current release preflight、启用前基础 readiness、direct enable dry-run、direct enable apply、启用后 `--require-direct` 复核、rollback dry-run、rollback apply、回退后 health patrol 切回 Nginx 并恢复切换前 public base URL / Host、回退后 health patrol env 复核；runbook 只用于审阅，不修改系统。正式 runbook 中 `--direct-redirect-host`、`--rollback-nginx-smoke-host` 和 `--direct-host` 必须使用同一 hostname，只允许端口不同，避免 redirect 或回退 smoke 各自验证到不同入口；同时必须提供 `--rollback-health-patrol-public-base-url <切换前Nginx巡检入口>`，若切换前 Nginx 巡检需要 Host 覆盖，再追加 `--rollback-health-patrol-public-host <切换前Host>`，确认步骤会展示回退后要恢复的 public base URL / Host，避免回退 runbook 把现场巡检入口覆盖成仓库默认值；如需把回退后 Pingora shadow 探针复核纳入 runbook，追加 `--rollback-pingora-shadow-probe-url` / `--rollback-pingora-shadow-probe-token`，JSON 输出会隐藏 token 原文。随后先执行 `/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs --release-root /opt/genarrative/current --require-pingora-gateway --systemd-show`，再 dry-run `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --no-status`，最后执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --apply --preflight-env-file /etc/genarrative/pingora-gateway.env --preflight-check-cert-readable --preflight-check-service-env-file --preflight-check-service-user-cert-readable --preflight-check-service-binary-executable --preflight-check-ports-free --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`，由脚本先跑 direct preflight，再安装 drop-in、reload systemd、重启 Pingora，并用 `systemctl cat` 核验 capability 和 `EnvironmentFile=/etc/genarrative/pingora-gateway.env` 已生效、用 `systemctl show ... ExecStart` 核验最终 service 仍指向随包主 service 模板里的 current release `pingora-gateway`、用 `systemctl is-active` 确认服务 active，再以 JSON 模式执行 direct live smoke，验证 HTTPS / HTTP redirect / ACME / WSS 101 和 Pingora access log request_id 落盘，并要求 `direct-access-log` 结构化结果 `matchedCount == checked`、`missingCount=0`、`mismatchCount=0`；如果 direct live 退出 0 但缺少该结构化证据，也必须视为启用失败。直连启用后同步调整 `/etc/genarrative/health-patrol.env`：设置 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=pingora-direct`，本机打 `127.0.0.1` 时设置 `GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST=<域名>`，否则巡检会继续按 Nginx 模式误报。验证失败时执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-rollback.sh --apply --reload-nginx --nginx-smoke-url http://127.0.0.1/healthz --nginx-smoke-host <域名>` 或 `npm run deploy:pingora-direct-rollback -- --apply --reload-nginx --nginx-smoke-url http://127.0.0.1/healthz --nginx-smoke-host <域名>`；回退脚本先跑 `nginx -t`，通过后才移除 drop-in、reload systemd、重启 Pingora，并用 `systemctl cat` 核验 capability 已移除、用 `systemctl show ... ExecStart` 核验最终 service 仍指向随包主 service 模板里的 current release `pingora-gateway`，随后 reload Nginx、确认 Nginx service 仍为 active，并用 curl smoke URL 证明公网入口已回到 Nginx；回退脚本 `--apply` 不允许省略 `--reload-nginx` 或 `--nginx-smoke-url`，当 smoke URL 指向本机地址时必须同时提供 `--nginx-smoke-host <域名>`，且 host 值不能包含 URL、路径或查询；回退后把 health patrol gateway mode 改回 `nginx`，恢复切换前 public base URL / Host，并用 `node -- /opt/genarrative/current/scripts/check-production-health-patrol-env.mjs --env-file /etc/genarrative/health-patrol.env --expected-gateway-mode nginx --expected-public-base-url <切换前Nginx巡检入口> --require-empty-public-host` 复核；若切换前 Nginx 巡检需要 Host 覆盖，则把 `--require-empty-public-host` 换成 `--expected-public-host <切换前Host>`。若 env 已在回退命令前切回 Nginx，也可给 rollback 脚本追加 `--health-patrol-env-file /etc/genarrative/health-patrol.env --health-patrol-expected-public-base-url <切换前Nginx巡检入口> --health-patrol-require-empty-public-host` 让它在 Nginx smoke 后自动复核；切换前 Nginx 巡检需要 Host 覆盖时把最后一项换成 `--health-patrol-expected-public-host <切换前Host>`。若要同时证明 Pingora shadow 高端口仍活着，追加 `--pingora-shadow-probe-url http://127.0.0.1:18081/__genarrative_pingora/healthz --pingora-shadow-probe-token `，脚本会隐藏 token 并要求响应为 `gateway=pingora-shadow`。 +- 处理补充：不要直接 chmod `/etc/letsencrypt/live` 或 `archive` 来让 Pingora 读取证书；Certbot live 路径通常是 symlink，即使 `stat -L` 看起来是普通文件，父目录权限也会让非 root `genarrative` 用户不可达。先用随包 `node -- /opt/genarrative/current/scripts/deploy/pingora-tls-cert-sync.mjs --apply --source-cert-file /etc/letsencrypt/live/<域名>/fullchain.pem --source-key-file /etc/letsencrypt/live/<域名>/privkey.pem --target-dir /etc/genarrative/pingora-tls/<域名>` 把证书同步到 Pingora 私有目录，再让 `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` / `TLS_KEY_FILE` 指向 `/etc/genarrative/pingora-tls/<域名>/fullchain.pem` 和 `privkey.pem`。脚本默认 dry-run，`--apply` 才写入，目标目录默认 `root:genarrative 0750`，文件默认 `root:genarrative 0640`，并拒绝符号链接目标目录或目标文件。 - 处理补充：不要在切换窗口手工编辑 `/etc/genarrative/health-patrol.env` 的三项网关变量；使用 `node -- /opt/genarrative/current/scripts/deploy/pingora-health-patrol-env-switch.mjs --apply --env-file /etc/genarrative/health-patrol.env --gateway-mode pingora-direct --public-base-url <直连HTTPS入口> --public-host <域名>` 切到直连，回退前用同一脚本传 `--gateway-mode nginx --public-base-url <切换前Nginx巡检入口>` 并按切换前记录选择 `--clear-public-host` 或 `--public-host <切换前Host>`。脚本只改 gateway mode / public base URL / public Host，并立即复用随包 env 复核脚本，减少空 Host 和旧值残留；生产巡检、env 复核和 env 切换脚本读取的布尔 env 都必须是明确布尔值，非法值直接失败，不能把拼写错误当成 false；env 复核脚本的 `--env-file` 与 env 切换脚本的 `--env-file` / `--check-script` 必须是绝对路径且不能是文件系统根目录，也不能包含换行或 NUL；env 切换脚本写入的 public base URL / Host 同样不能包含换行或 NUL。Node 22 已内置 `--env-file` 启动参数，直接用 `node script.mjs --env-file ...` 或 shebang 执行 `.mjs --env-file ...` 都可能让 Node 抢走业务参数；所有这类命令都必须写成 `node -- script.mjs --env-file ...`，或通过已内置 `node --` 的 npm script 执行。 - 踩坑补充：health patrol env 切换脚本必须先复核权限固定为 `0600` 的临时目标 env 再写真实文件，真实 env 原子替换时保持原文件权限和 owner/group；如果随包 env 复核脚本失败，`--apply` 应失败且真实 env 保持原样，避免“切换脚本失败但巡检配置已半改”的状态。`--apply` 的 `--env-file` 必须直接指向真实普通文件，不能传符号链接；如果 `/etc/genarrative/health-patrol.env` 是链接，先确认真实目标路径后再传给脚本，避免替换链接本身或写入非预期目标。 - 踩坑补充：直连启用脚本的 `--preflight-script`、`--direct-live-script`、`--current-release-audit-script`、`--template-path`、`--service-unit-path`、`--dropin-path` 和 env 文件参数都必须使用绝对路径；不要在切换窗口传相对脚本路径，否则会把 current release、Jenkins 工作区或现场 cwd 混在一起。`--apply` 会在安装 direct-entry drop-in 前确认 current release 自审、direct preflight 和 direct live smoke 脚本存在，缺脚本时应先修发布包或复制链路，不要手工改成工作区相对路径绕过。启用脚本还会在任何自审、preflight、drop-in 写入或 systemctl 前拒绝 service、路径、URL、Host、probe token、access log、数据库名、tail 行数和 timeout 参数中的换行或 NUL 字符；遇到这类失败先修 runbook 参数来源或现场 env，不要手工绕过脚本。启用脚本还会拒绝符号链接形式的 drop-in 目录或 drop-in 目标文件，以及已存在但不是普通文件的目标；如果现场 systemd 目录被软链改写，应先修正真实路径，不要让脚本把低端口 capability 写入非预期位置。回退脚本 `--apply` 同样会在 `nginx -t` 和删除 drop-in 前拒绝 service、路径、Nginx smoke URL / Host / 响应片段、health patrol 复核参数、shadow probe URL / token 和二进制 override 中的换行或 NUL 字符，并拒绝符号链接 drop-in 目录 / 目标以及非普通 drop-in 目标；如果现场路径或参数异常，应先修正 systemd 路径、runbook 参数或现场 env，不要手工删 drop-in、绕过 `nginx -t` 或把删除 symlink 当成已回退真实低端口能力。回退脚本覆盖 `--nginx-binary` 或 `--curl-binary` 时也不要传 `./nginx`、`tools/curl` 这类相对路径；裸命令名可以走 `PATH`，路径形式必须使用绝对路径。`--nginx-smoke-url` 必须带 `http://` 或 `https://`，不要只写 host/path，否则脚本会在移除 drop-in 前失败。 @@ -58,7 +59,7 @@ - 踩坑补充：同一天多次演练或切换时，只靠“最新证据”和 24 小时窗口仍可能把两轮证据拼在一起。正式 runbook 会生成或接受 `--cutover-run-id `，并把同一 `manifest.cutoverRunId` 写入三阶段证据包、enable / rollback apply 命令证据和最终总审计；最终审计必须带 `--require-cutover-run-id <本次cutoverRunId>`，缺少该字段或 ID 不一致时必须失败。即使人工临时总审计忘记带 `--require-cutover-run-id`，标准五段时间线里只要任一证据声明了 `manifest.cutoverRunId`，五段也必须全部声明同一个值，否则总审计失败。 - 验证：先运行 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`，确认 env、drop-in、service EnvironmentFile 一致性、当前用户证书权限、服务用户证书权限、service 二进制可执行性和 80/443 已释放；`systemctl cat genarrative-pingora-gateway.service` 必须显示 `AmbientCapabilities=CAP_NET_BIND_SERVICE`、`CapabilityBoundingSet=CAP_NET_BIND_SERVICE` 和 `EnvironmentFile=/etc/genarrative/pingora-gateway.env`；启用脚本 apply 必须先通过 current release 自审，失败时不安装 direct-entry drop-in；还必须带 direct HTTPS / HTTP / Host / redirect host / SpacetimeDB database / Pingora access log 参数，并在重启后直接完成 direct live smoke 和 direct-access-log JSON 证据校验；也可用 release readiness `--require-direct --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log --direct-health-patrol-env-file /etc/genarrative/health-patrol.env --direct-preflight-env-file /etc/genarrative/pingora-gateway.env --direct-preflight-systemd --direct-preflight-check-cert-readable --direct-preflight-check-service-env-file --direct-preflight-check-service-user-cert-readable --direct-preflight-check-service-binary-executable --direct-preflight-check-ports-free` 把 HTTPS、HTTP redirect / ACME、正式域名 Host/SNI、redirect Location host、Pingora access log request_id 落盘、env 预检、systemd drop-in、service EnvironmentFile 一致性、当前用户和服务用户证书可读、service 二进制可执行、端口释放、显式目标库和 WSS 101 一起纳入硬门禁，并拒绝 `--direct-skip-wss`，避免 TLS 证书只按 `127.0.0.1` 误测、HTTP redirect Location 指错域名、Nginx 仍占用 80/443、service 实际读取另一份 env、root / deploy 用户可读但 systemd 服务用户不可读、current release 缺少可执行 `pingora-gateway`，或 WSS subscribe 隐式打到默认 SpacetimeDB 库。`check-pingora-release-readiness.mjs --help` 的正式直连和只生成 runbook 示例也必须带 `--direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`，不要让值班人员复制示例后才被 `--require-direct` 拦截。current release 自审、状态快照和证据包的布尔 env 必须是明确布尔值，非法值会失败，不得把拼错的 run / require / fail 开关当成 false。`npm run plan:pingora-direct-cutover -- --require-direct ...` 输出必须包含 Host 与回退巡检入口确认、current release preflight、启用前不带 `--require-direct` 的基础 readiness、direct enable dry-run/apply、启用后带 `--require-direct` 的复核、rollback dry-run/apply、回退后 health patrol 切回 Nginx 并恢复切换前 public base URL / Host、回退后 health patrol env 复核；缺少 `--require-direct`、缺少 `--rollback-health-patrol-public-base-url`、缺少 `--direct-pingora-access-log`、redirect Host 漂移或 rollback smoke Host 漂移时必须失败，避免生成缺少正式直连硬门禁或验证不同入口的切换计划。Host 与回退巡检入口确认步骤必须展示回退后要恢复的 health patrol public base URL / Host。直连后 `genarrative-health-patrol.service` 应使用 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=pingora-direct`，状态 JSON 中 `gatewayMode` 应为 `pingora-direct`，并检查 `genarrative-pingora-gateway.service` 而不是 `nginx.service`；public probe 走 `127.0.0.1` 时应带 `GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST=<域名>`。回退后 `nginx -t` 必须先通过，`systemctl cat genarrative-pingora-gateway.service` 不应再显示这两条 capability，`systemctl show genarrative-pingora-gateway.service --property=ExecStart --value --no-pager` 必须仍指向 current release 的 `pingora-gateway`，`systemctl is-active nginx.service` 应为 `active`，`curl --fail --max-time 5` 访问 `--nginx-smoke-url` 应成功；若 smoke URL 为本机地址必须带 `--nginx-smoke-host <域名>`，证明正式 vhost 已回到 Nginx；随后用 `node -- /opt/genarrative/current/scripts/check-production-health-patrol-env.mjs ...` 复核 health patrol env，必须显示 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=nginx` 且 public base URL / Host 与切换前记录一致，shadow probe 可选复核必须返回 `gateway=pingora-shadow`。本机提交前还要运行 `npm run check:pingora-direct-enable`、`npm run check:pingora-direct-rollback`、`npm run check:production-health-patrol`、`npm run check:production-api-release`、`npm run check:pingora-production-release-build` 和 `npm run check:production-api-deploy`，确保脚本默认 dry-run 不会安装或删除 drop-in、current release 自审失败时启用脚本不会安装 drop-in、direct live 退出 0 但缺少 `direct-access-log` 结构化证据时启用失败，API release 布局自包含，真实 Pingora release 二进制能构建并进入发布包，API deploy 从发布产物内执行后 current release 自包含；缺少数据库备份脚本、健康巡检脚本、健康巡检 env 复核脚本、切换命令证据脚本、env 示例目录或 direct live smoke 脚本的发布包都必须部署失败并保持维护模式。正式直连 readiness 必须带 `--direct-health-patrol-env-file /etc/genarrative/health-patrol.env`，并用 `scripts/check-production-health-patrol-env.mjs` 阻断 health patrol 仍停在 Nginx 模式或本机 direct probe 缺少正式 Host；发布包包含 `pingora-gateway` 时，`npm run check:production-api-deploy` 必须覆盖服务 active 时执行 `systemctl try-restart genarrative-pingora-gateway.service` 并复核 active，以及服务 inactive 时跳过且不主动拉起。 - 顺序补充：正式 runbook 必须先通过 health patrol env 切换脚本预置回 Nginx 和切换前 public base URL / Host，再执行 `rollback apply`；回退脚本内置 env 复核和独立 env 复核都会阻断 public base URL / Host 漂移。 -- 关联：`deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`、`deploy/env/health-patrol.env.example`、`deploy/env/pingora-direct-live.env.example`、`deploy/env/pingora-canary-live.env.example`、`scripts/deploy/pingora-direct-enable.sh`、`scripts/deploy/pingora-direct-rollback.sh`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/ops/pingora-cutover-command-evidence.mjs`、`scripts/ops/pingora-cutover-evidence-verify.mjs`、`scripts/ops/pingora-cutover-evidence-audit.mjs`、`scripts/jenkins-server-provision.sh`、`scripts/build-production-release.sh`、`scripts/deploy/production-api-deploy.sh`、`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`。 +- 关联：`deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`、`deploy/env/health-patrol.env.example`、`deploy/env/pingora-direct-live.env.example`、`deploy/env/pingora-canary-live.env.example`、`scripts/deploy/pingora-direct-enable.sh`、`scripts/deploy/pingora-direct-rollback.sh`、`scripts/deploy/pingora-tls-cert-sync.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/ops/pingora-cutover-command-evidence.mjs`、`scripts/ops/pingora-cutover-evidence-verify.mjs`、`scripts/ops/pingora-cutover-evidence-audit.mjs`、`scripts/jenkins-server-provision.sh`、`scripts/build-production-release.sh`、`scripts/deploy/production-api-deploy.sh`、`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`。 ## 外部生成 worker 业务失败重试会撞上钱包扣退费幂等 diff --git a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md index 0832bf487..17aab3968 100644 --- a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md +++ b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md @@ -221,7 +221,7 @@ npm run check:pingora-direct-live npm run build:production-release -- --component api-server --name --include-pingora-gateway ``` -Jenkins `Genarrative-Api-Build` 对应参数是 `INCLUDE_PINGORA_GATEWAY`，默认关闭；勾选后才会归档 `pingora-gateway` 与 `pingora-gateway.sha256`，触发后续 `Genarrative-Api-Deploy` 时也会传递该布尔参数并复制这两个可选产物。无论是否打包 Pingora 二进制，API release 都必须携带 `build//scripts/deploy/production-api-deploy.sh`、同目录的 `maintenance-on.sh` / `maintenance-off.sh`、`/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh`、`pingora-direct-rollback.sh`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/ops/pingora-cutover-command-evidence.mjs`、`scripts/ops/pingora-cutover-evidence-verify.mjs`、`scripts/ops/pingora-cutover-evidence-audit.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/check-production-health-patrol-env.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/`。`Genarrative-Api-Deploy` 只能从上游构建归档复制并执行 `build//scripts/deploy/production-api-deploy.sh`，不能继续执行部署工作区根部脚本；维护脚本必须与 deploy 脚本来自同一发布包同一目录，避免 Jenkins 工作区里的旧脚本掩盖 release 包布局缺陷。直连启用脚本从 current release 执行时默认读取 `/opt/genarrative/current/deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`，必须能自包含完成 preflight、systemd 模板读取和 direct live smoke，不依赖 Jenkins 工作区、目标机源码 checkout 或 `/etc` 参考模板；current release 自审、health patrol env 复核、canary access log 对账、canary live、direct live、状态快照、证据包、命令证据、证据验真和证据根目录审计也必须来自 current release 随包脚本与 `deploy/env/`。`/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 只作为 Server-Provision 安装的人工审阅 / 手动覆盖模板；确需使用时显式传 `--template-path` 或 `GENARRATIVE_PINGORA_DIRECT_TEMPLATE_PATH`。`npm run check:production-api-release` 会用临时 `CARGO_TARGET_DIR` 和假 `api-server` / `pingora-gateway` release binary 跑 `build-production-release.sh --component api-server --skip-api-build`，动态验证默认 API release 不登记 Pingora，显式 `--include-pingora-gateway --skip-pingora-gateway-build` 时则必须包含 `pingora-gateway`、`pingora-gateway.sha256` 并写入 manifest，同时验证 Pingora 直连依赖、current release 自审脚本、状态快照脚本、证据包脚本、命令证据脚本、证据验真脚本、证据根目录审计脚本、canary access log 对账脚本、健康巡检 env 复核脚本、env 示例目录、API deploy 执行入口自包含和 `deploy/pingora/pingora-gateway.env.example` 的生产安全默认值：gzip-only、`TRUST_X_FORWARDED_FOR=false`、`TRUSTED_FRONT_PROXY_CONFIRMED=false`、接流保护默认开启、probe token 为空；该检查还会读取发布包 README，并直接运行发布包内 `scripts/check-pingora-release-readiness.mjs --dry-run-cutover`，确认最终证据根目录总审计步骤仍带两条 `--require-command-executable ... /opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh` / `pingora-direct-rollback.sh` current release 脚本身份要求。`npm run check:pingora-production-release-build` 会用假 `api-server`、临时 `CARGO_TARGET_DIR` 和真实 `cargo build -p pingora-gateway --release --target x86_64-unknown-linux-gnu` 验证显式 include 路径能构出可执行 `pingora-gateway`、checksum 和 manifest 登记，避免只靠假二进制布局 smoke。`production-api-deploy.sh` 对这些 Pingora 直连依赖采取 fail-fast：上游发布产物缺失时保持维护模式并停止部署，不再从部署工作区兜底复制；`release-manifest.json` 必须存在且登记 `api-server` artifact，若发布包包含 `pingora-gateway`，manifest 也必须登记 `pingora-gateway` artifact，否则部署会在切换 current 前失败。API deploy 要求 `--release-root`、`--current-link`、`--api-env-file` 使用绝对路径，`--version` 必须以数字或字母开头并拒绝点目录，会先写 `${RELEASE_ROOT}/.${VERSION}.staging.$`，全部校验和复制完成后用非合并语义提升为 `${RELEASE_ROOT}/${VERSION}`，再用固定替换语义切换 current 符号链接；同版本正式 release 已存在、提升前竞态出现或 current 路径不是符号链接时都会拒绝合并 / 覆盖，失败时清理 staging 并保持维护模式。本机用 `npm run check:production-api-deploy` 通过临时 release、fake `systemctl` / `curl` 动态验证从发布产物内执行 deploy 脚本后 current release 自洽，并覆盖缺少 current release 自审脚本、状态快照脚本、证据包脚本、证据验真脚本、证据根目录审计脚本、canary access log 对账脚本、健康巡检 env 复核脚本、`deploy/env/`、release manifest、api-server manifest artifact、Pingora manifest artifact、相对 release root / current link / api env file、点目录或点开头 version、同版本 release 目录已存在、current 路径不是符号链接或提升前 release 目录竞态出现时必须失败，同时复核 current release 内 Pingora env 示例仍保持同一组生产安全默认值。 +Jenkins `Genarrative-Api-Build` 对应参数是 `INCLUDE_PINGORA_GATEWAY`，默认关闭；勾选后才会归档 `pingora-gateway` 与 `pingora-gateway.sha256`，触发后续 `Genarrative-Api-Deploy` 时也会传递该布尔参数并复制这两个可选产物。无论是否打包 Pingora 二进制，API release 都必须携带 `build//scripts/deploy/production-api-deploy.sh`、同目录的 `maintenance-on.sh` / `maintenance-off.sh`、`/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh`、`pingora-direct-rollback.sh`、`pingora-tls-cert-sync.mjs`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/ops/pingora-cutover-command-evidence.mjs`、`scripts/ops/pingora-cutover-evidence-verify.mjs`、`scripts/ops/pingora-cutover-evidence-audit.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/check-production-health-patrol-env.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/`。`Genarrative-Api-Deploy` 只能从上游构建归档复制并执行 `build//scripts/deploy/production-api-deploy.sh`，不能继续执行部署工作区根部脚本；维护脚本必须与 deploy 脚本来自同一发布包同一目录，避免 Jenkins 工作区里的旧脚本掩盖 release 包布局缺陷。直连启用脚本从 current release 执行时默认读取 `/opt/genarrative/current/deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`，必须能自包含完成 preflight、systemd 模板读取和 direct live smoke，不依赖 Jenkins 工作区、目标机源码 checkout 或 `/etc` 参考模板；current release 自审、TLS 证书同步、health patrol env 复核、canary access log 对账、canary live、direct live、状态快照、证据包、命令证据、证据验真和证据根目录审计也必须来自 current release 随包脚本与 `deploy/env/`。`/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 只作为 Server-Provision 安装的人工审阅 / 手动覆盖模板；确需使用时显式传 `--template-path` 或 `GENARRATIVE_PINGORA_DIRECT_TEMPLATE_PATH`。`npm run check:production-api-release` 会用临时 `CARGO_TARGET_DIR` 和假 `api-server` / `pingora-gateway` release binary 跑 `build-production-release.sh --component api-server --skip-api-build`，动态验证默认 API release 不登记 Pingora，显式 `--include-pingora-gateway --skip-pingora-gateway-build` 时则必须包含 `pingora-gateway`、`pingora-gateway.sha256` 并写入 manifest，同时验证 Pingora 直连依赖、TLS 证书同步脚本、current release 自审脚本、状态快照脚本、证据包脚本、命令证据脚本、证据验真脚本、证据根目录审计脚本、canary access log 对账脚本、健康巡检 env 复核脚本、env 示例目录、API deploy 执行入口自包含和 `deploy/pingora/pingora-gateway.env.example` 的生产安全默认值：gzip-only、`TRUST_X_FORWARDED_FOR=false`、`TRUSTED_FRONT_PROXY_CONFIRMED=false`、接流保护默认开启、probe token 为空；该检查还会读取发布包 README，并直接运行发布包内 `scripts/check-pingora-release-readiness.mjs --dry-run-cutover`，确认最终证据根目录总审计步骤仍带两条 `--require-command-executable ... /opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh` / `pingora-direct-rollback.sh` current release 脚本身份要求。`npm run check:pingora-production-release-build` 会用假 `api-server`、临时 `CARGO_TARGET_DIR` 和真实 `cargo build -p pingora-gateway --release --target x86_64-unknown-linux-gnu` 验证显式 include 路径能构出可执行 `pingora-gateway`、checksum 和 manifest 登记，避免只靠假二进制布局 smoke。`production-api-deploy.sh` 对这些 Pingora 直连依赖采取 fail-fast：上游发布产物缺失时保持维护模式并停止部署，不再从部署工作区兜底复制；`release-manifest.json` 必须存在且登记 `api-server` artifact，若发布包包含 `pingora-gateway`，manifest 也必须登记 `pingora-gateway` artifact，否则部署会在切换 current 前失败。API deploy 要求 `--release-root`、`--current-link`、`--api-env-file` 使用绝对路径，`--version` 必须以数字或字母开头并拒绝点目录，会先写 `${RELEASE_ROOT}/.${VERSION}.staging.$`，全部校验和复制完成后用非合并语义提升为 `${RELEASE_ROOT}/${VERSION}`，再用固定替换语义切换 current 符号链接；同版本正式 release 已存在、提升前竞态出现或 current 路径不是符号链接时都会拒绝合并 / 覆盖，失败时清理 staging 并保持维护模式。本机用 `npm run check:production-api-deploy` 通过临时 release、fake `systemctl` / `curl` 动态验证从发布产物内执行 deploy 脚本后 current release 自洽，并覆盖缺少 TLS 证书同步脚本、current release 自审脚本、状态快照脚本、证据包脚本、证据验真脚本、证据根目录审计脚本、canary access log 对账脚本、健康巡检 env 复核脚本、`deploy/env/`、release manifest、api-server manifest artifact、Pingora manifest artifact、相对 release root / current link / api env file、点目录或点开头 version、同版本 release 目录已存在、current 路径不是符号链接或提升前 release 目录竞态出现时必须失败，同时复核 current release 内 Pingora env 示例仍保持同一组生产安全默认值。 API release 还必须携带 `scripts/check-pingora-release-readiness.mjs` 与 `scripts/check-pingora-canary-live.mjs`。前者支撑 current release 的 `--release-runtime-only` 聚合复核，后者支撑目标 Nginx canary live smoke；缺少任一脚本时 `check:production-api-release`、`check:production-api-deploy` 和生产运维护栏都必须失败。 @@ -229,7 +229,17 @@ API release 还必须携带 `scripts/check-pingora-release-readiness.mjs` 与 `s 也可以复制 `deploy/pingora/pingora-gateway.env.example` 到部署环境的非 Git 配置文件，由 systemd 或容器注入。仓库提供 `deploy/systemd/genarrative-pingora-gateway.service` 作为影子服务模板，默认读取 `/etc/genarrative/pingora-gateway.env`，仍只应监听本机高端口，再由 Nginx 或本机 smoke 主动访问。Server-Provision 会把主 service 安装到 `/etc/systemd/system/genarrative-pingora-gateway.service`，并把直连低端口 drop-in 模板安装到 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 作为参考和手动覆盖来源；该模板不会默认生效，`/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh` 默认使用随 current release 发布的 `deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`。`deploy/nginx/snippets/genarrative-pingora-canary.conf` 是可选的 Nginx -> Pingora 前缀 canary 模板，Server-Provision 会安装到 `/etc/nginx/snippets/`，但主站配置默认不 include；启用前必须把 `__GENARRATIVE_PINGORA_PROBE_TOKEN__` 替换为真实 token，并确认 `allow/deny` 来源边界符合当次验证窗口。 -直连公网入口切换窗口如果需要让非 root 的 `genarrative` 用户绑定 `80/443`，先确认 `/etc/genarrative/pingora-gateway.env` 已显式设置 TLS / redirect 入口和证书路径，并确认 `genarrative` 用户可读取证书链和私钥；生产 `80/443` 还必须先从 Nginx 或其它进程释放，非切换窗口建议先用 `18443/18080` 这类高端口验证。随后先 dry-run 直连启用脚本： +直连公网入口切换窗口如果需要让非 root 的 `genarrative` 用户绑定 `80/443`，先确认 `/etc/genarrative/pingora-gateway.env` 已显式设置 TLS / redirect 入口和证书路径，并确认 `genarrative` 用户可读取证书链和私钥；生产 `80/443` 还必须先从 Nginx 或其它进程释放，非切换窗口建议先用 `18443/18080` 这类高端口验证。若证书来自 Certbot / Let’s Encrypt，不要放宽 `/etc/letsencrypt/live` 或 `archive` 的目录 / 私钥权限；先用随包 `pingora-tls-cert-sync.mjs` 把 live symlink 解析后的真实证书复制到 `/etc/genarrative/pingora-tls/<域名>/`，再把 env 中 `TLS_CERT_FILE` / `TLS_KEY_FILE` 指向该私有副本： + +```bash +sudo -n node -- /opt/genarrative/current/scripts/deploy/pingora-tls-cert-sync.mjs \ + --apply \ + --source-cert-file /etc/letsencrypt/live/<域名>/fullchain.pem \ + --source-key-file /etc/letsencrypt/live/<域名>/privkey.pem \ + --target-dir /etc/genarrative/pingora-tls/<域名> +``` + +该脚本默认 dry-run，`--apply` 才原子写入 `fullchain.pem` / `privkey.pem`，目标目录默认 `root:genarrative 0750`，文件默认 `root:genarrative 0640`，并复核 `genarrative` 服务用户可读；目标目录和目标文件不能是符号链接。随后先 dry-run 直连启用脚本： ```bash npm run plan:pingora-direct-cutover -- --require-direct --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log --direct-health-patrol-env-file /etc/genarrative/health-patrol.env --direct-preflight-env-file /etc/genarrative/pingora-gateway.env --direct-preflight-systemd --direct-preflight-check-cert-readable --direct-preflight-check-service-env-file --direct-preflight-check-service-user-cert-readable --direct-preflight-check-service-binary-executable --direct-preflight-check-ports-free --rollback-nginx-smoke-url http://127.0.0.1/healthz --rollback-health-patrol-public-base-url <切换前Nginx巡检入口> @@ -383,8 +393,8 @@ dev 根盘空间在安装后曾接近满盘；2026-06-17 进入 canary 前已清 | ------------------------------------------------------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------------- | | `GENARRATIVE_PINGORA_GATEWAY_LISTEN` | `127.0.0.1:18081` | Pingora 监听地址。 | | `GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN` | 空 | 可选 HTTPS 监听地址；启用时必须同时设置 `TLS_CERT_FILE` 和 `TLS_KEY_FILE`。 | -| `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` | 空 | 可选 HTTPS 证书链文件；必须是网关运行用户可读取的文件。 | -| `GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE` | 空 | 可选 HTTPS 私钥文件；必须是网关运行用户可读取的文件。 | +| `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` | 空 | 可选 HTTPS 证书链文件；必须是网关运行用户可读取的文件。Certbot 证书建议先同步到 `/etc/genarrative/pingora-tls/<域名>/fullchain.pem`。 | +| `GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE` | 空 | 可选 HTTPS 私钥文件；必须是网关运行用户可读取的文件。Certbot 私钥建议先同步到 `/etc/genarrative/pingora-tls/<域名>/privkey.pem`。 | | `GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN` | 空 | 可选 HTTP 重定向监听地址；启用时必须已配置 TLS 入口，ACME challenge 仍静态读取。 | | `GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_TARGET_SCHEME` | `https` | HTTP 重定向目标 scheme，当前只允许 `https`。 | | `GENARRATIVE_PINGORA_GATEWAY_API_UPSTREAM` | `127.0.0.1:8082` | `api-server` 上游地址。 | @@ -462,7 +472,7 @@ dev 根盘空间在安装后曾接近满盘；2026-06-17 进入 canary 前已清 4. 使用 `deploy/nginx/snippets/genarrative-pingora-canary.conf` 做 Nginx 前缀 canary；启用前先跑 `npm run check:nginx-pingora-canary` 和 `npm run check:pingora-canary-docker`，有 Nginx 或 Docker 的目标环境分别强制跑 `node scripts/check-nginx-pingora-canary.mjs --require-nginx` 与 `node scripts/check-pingora-canary-docker.mjs --require-docker --pull`，其中 Docker handoff 会自动对账临时 Nginx 与 Pingora access log。启用后跑 `npm run check:pingora-canary-live`，再用 current release 随包 access log parity 脚本对账目标机 Nginx 与 Pingora access log。canary live 的 base URL、prefix、Host、额外 path 和 timeout 不能包含换行或 NUL；canary live timeout 和 access log `since-lines` 必须是正整数，非法值直接失败。 5. 前缀 canary 通过后，再使用 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 做真实路径 canary；它必须作为独立本机 `server` include 到 Nginx `http` 上下文，不能 include 到生产 `443` server 内。启用后跑 `node scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再用 current release 随包 access log parity 脚本传 `--realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log` 对账 `/api/creation-entry/config`、`/v1/identity` 和 `/assets/app.js` 等真实路径。 6. 目标机 canary include 后必须跑正式切换聚合门禁：源码 checkout / CI / 构建环境执行默认全量命令 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；目标机 current release 执行 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。启用真实路径 canary 时，两种命令都追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。缺少 Host 会直接失败，避免 live canary 误测默认 vhost；live smoke 后还会按 `request_id` 对账 Nginx 与 Pingora access log，缺少同一请求的 Pingora 日志、状态码、方法或 path 漂移都会失败。 -7. 如需评估 Pingora 直连公网入口，必须显式配置 `TLS_LISTEN`、证书、私钥和 `HTTP_REDIRECT_LISTEN`；绑定 `80/443` 时还必须人工启用 `genarrative-pingora-gateway-direct-entry.conf` drop-in 授予 `CAP_NET_BIND_SERVICE`。随后用 `npm run check:pingora-gateway-smoke` 覆盖 TLS / HTTP/2 ALPN / redirect / WSS subscribe；目标机必须先跑 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`，再跑 `npm run check:pingora-direct-live` 或 release readiness 的 `--require-direct`，且 `--require-direct` 必须带 direct HTTPS base URL、direct HTTP base URL、正式域名 Host/SNI、redirect Location host、Pingora access log 文件、health patrol env 文件、direct preflight env 文件、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户和 systemd 服务用户证书可读检查、service 二进制可执行检查、端口释放检查和显式 SpacetimeDB 数据库名，并会拒绝 `--skip-wss`。direct live 会用生成的 `request_id` 反查 Pingora access log；缺少对应日志、method 漂移、path 漂移或 status 漂移都算直连门禁失败。direct preflight 会拒绝开启网关保护但未确认共享保护层的 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT>1` 配置；`--env-file`、`--systemd-service`、服务用户和 env 中的 listen / cert / key 值都不能包含换行或 NUL，执行 `systemctl cat` 或 `sudo -u test -r ` 前还会复核子命令参数，避免污染参数进入目标机预检命令；direct live 的 URL、Host、probe token、额外 path、数据库名、access log 路径、timeout 和布尔 env 也不能包含换行或 NUL，且会在发起请求前失败；direct live timeout 必须是正整数，直连相关布尔 env 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。证书申请与续期仍由 Certbot / 外部自动化承担，网关只读取现有文件。 +7. 如需评估 Pingora 直连公网入口，必须显式配置 `TLS_LISTEN`、证书、私钥和 `HTTP_REDIRECT_LISTEN`；Certbot 证书先用随包 `scripts/deploy/pingora-tls-cert-sync.mjs` 同步到 `/etc/genarrative/pingora-tls/<域名>/`，不要直接 chmod Let’s Encrypt live/archive 原路径；绑定 `80/443` 时还必须人工启用 `genarrative-pingora-gateway-direct-entry.conf` drop-in 授予 `CAP_NET_BIND_SERVICE`。随后用 `npm run check:pingora-gateway-smoke` 覆盖 TLS / HTTP/2 ALPN / redirect / WSS subscribe；目标机必须先跑 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`，再跑 `npm run check:pingora-direct-live` 或 release readiness 的 `--require-direct`，且 `--require-direct` 必须带 direct HTTPS base URL、direct HTTP base URL、正式域名 Host/SNI、redirect Location host、Pingora access log 文件、health patrol env 文件、direct preflight env 文件、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户和 systemd 服务用户证书可读检查、service 二进制可执行检查、端口释放检查和显式 SpacetimeDB 数据库名，并会拒绝 `--skip-wss`。direct live 会用生成的 `request_id` 反查 Pingora access log；缺少对应日志、method 漂移、path 漂移或 status 漂移都算直连门禁失败。direct preflight 会拒绝开启网关保护但未确认共享保护层的 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT>1` 配置；`--env-file`、`--systemd-service`、服务用户和 env 中的 listen / cert / key 值都不能包含换行或 NUL，执行 `systemctl cat` 或 `sudo -u test -r ` 前还会复核子命令参数，避免污染参数进入目标机预检命令；direct live 的 URL、Host、probe token、额外 path、数据库名、access log 路径、timeout 和布尔 env 也不能包含换行或 NUL，且会在发起请求前失败；direct live timeout 必须是正整数，直连相关布尔 env 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。证书申请与续期仍由 Certbot / 外部自动化承担，网关只读取现有文件。 8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs` 和 `deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release；缺失时部署应 fail-fast，切换窗口不能依赖源码 checkout 或 Jenkins workspace。 9. direct preflight / direct live timeout 必须是正整数；直连相关布尔 env 和 Pingora gateway env 里的 `TRUST_X_FORWARDED_FOR` / `TRUSTED_FRONT_PROXY_CONFIRMED` 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。 10. Pingora 正式化口径固定为 gzip-only；Brotli 继续由 Nginx / 前置代理承担，直连 Pingora 不以 Brotli parity 作为切换门禁。如需多实例，开启网关接流保护时必须先引入共享限流 / 共享并发保护层并设置 `GENARRATIVE_PINGORA_GATEWAY_SHARED_PROTECTION_CONFIRMED=true`，否则保持 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT=1`；关闭网关保护的多实例方案必须明确由前置 Nginx / LB 承担全局限流。 diff --git a/package.json b/package.json index 82a573da4..8fe9d55e0 100644 --- a/package.json +++ b/package.json @@ -44,6 +44,7 @@ "check:pingora-direct-enable": "node scripts/check-pingora-direct-enable.mjs", "check:pingora-direct-rollback": "node scripts/check-pingora-direct-rollback.mjs", "check:pingora-health-patrol-env-switch": "node scripts/check-pingora-health-patrol-env-switch.mjs", + "check:pingora-tls-cert-sync": "node scripts/check-pingora-tls-cert-sync.mjs", "check:pingora-current-release-audit": "node scripts/check-pingora-current-release-audit.mjs", "check:pingora-cutover-status-snapshot": "node scripts/check-pingora-cutover-status-snapshot.mjs", "check:pingora-cutover-evidence-bundle": "node scripts/check-pingora-cutover-evidence-bundle.mjs", diff --git a/scripts/build-production-release.sh b/scripts/build-production-release.sh index de2103e1f..d31b556e0 100644 --- a/scripts/build-production-release.sh +++ b/scripts/build-production-release.sh @@ -521,6 +521,7 @@ cp "${SCRIPT_DIR}/deploy/maintenance-off.sh" "${TARGET_DIR}/scripts/deploy/maint cp "${SCRIPT_DIR}/deploy/pingora-direct-enable.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh" cp "${SCRIPT_DIR}/deploy/pingora-direct-rollback.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh" cp "${SCRIPT_DIR}/deploy/pingora-health-patrol-env-switch.mjs" "${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" +cp "${SCRIPT_DIR}/deploy/pingora-tls-cert-sync.mjs" "${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" cp "${SCRIPT_DIR}/deploy/jenkins-inbound-agent-start.sh" "${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh" cp "${SCRIPT_DIR}/deploy/install-jenkins-inbound-agent.sh" "${TARGET_DIR}/scripts/install-jenkins-inbound-agent.sh" cp "${SCRIPT_DIR}/deploy/jenkins-agent-reverse-tunnel.ps1" "${TARGET_DIR}/scripts/jenkins-agent-reverse-tunnel.ps1" @@ -535,6 +536,7 @@ chmod +x \ "${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh" \ "${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh" \ "${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" \ + "${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" \ "${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh" \ "${TARGET_DIR}/scripts/install-jenkins-inbound-agent.sh" @@ -578,7 +580,7 @@ cat >"${TARGET_DIR}/README.md" < artifact.path === 'scripts/deploy/pingora-tls-cert-sync.mjs', + ); + if (!tlsCertSyncArtifact || tlsCertSyncArtifact.status !== 'OK') { + failures.push('完整 release 自审必须确认 TLS 证书同步脚本存在且可执行。'); + } } function assertChecksumMismatchFails() { @@ -453,6 +464,7 @@ function prepareReleaseRoot(releaseRoot, options) { 'scripts/deploy/pingora-direct-enable.sh', 'scripts/deploy/pingora-direct-rollback.sh', 'scripts/deploy/pingora-health-patrol-env-switch.mjs', + 'scripts/deploy/pingora-tls-cert-sync.mjs', 'deploy/systemd/genarrative-pingora-gateway.service', 'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf', 'deploy/nginx/snippets/genarrative-pingora-canary.conf', @@ -476,6 +488,7 @@ function prepareReleaseRoot(releaseRoot, options) { 'scripts/deploy/pingora-direct-enable.sh', 'scripts/deploy/pingora-direct-rollback.sh', 'scripts/deploy/pingora-health-patrol-env-switch.mjs', + 'scripts/deploy/pingora-tls-cert-sync.mjs', ]) { chmodExecutable(path.join(releaseRoot, file)); } diff --git a/scripts/check-pingora-tls-cert-sync.mjs b/scripts/check-pingora-tls-cert-sync.mjs new file mode 100644 index 000000000..fb02d670b --- /dev/null +++ b/scripts/check-pingora-tls-cert-sync.mjs @@ -0,0 +1,281 @@ +#!/usr/bin/env node + +import { spawnSync } from 'node:child_process'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + statSync, + symlinkSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; + +const SCRIPT = 'scripts/deploy/pingora-tls-cert-sync.mjs'; +const failures = []; +const tmpRoot = mkdtempSync( + path.join(tmpdir(), 'genarrative-pingora-tls-cert-sync-'), +); + +try { + main(); +} finally { + rmSync(tmpRoot, { recursive: true, force: true }); +} + +if (failures.length > 0) { + console.error('[check:pingora-tls-cert-sync] FAILED'); + for (const failure of failures) { + console.error(`- ${failure}`); + } + process.exit(1); +} + +console.log('[check:pingora-tls-cert-sync] OK'); + +function main() { + assertDryRunAcceptsCertbotLiveSymlinks(); + assertApplyCopiesToPrivateTarget(); + assertRejectsRelativeAndRootPaths(); + assertRejectsControlCharactersBeforeApply(); + assertRejectsSymlinkTargetDirectory(); + assertRejectsSymlinkTargetFiles(); + assertRejectsWorldReadableModes(); +} + +function assertDryRunAcceptsCertbotLiveSymlinks() { + const fixture = prepareFixture('dry-run-live-symlink'); + const result = runSync(fixture, []); + assertStatus(result, 0, 'dry-run 应接受 Certbot live symlink 源。'); + if (existsSync(fixture.targetCert) || existsSync(fixture.targetKey)) { + failures.push('dry-run 不应写入目标证书文件。'); + } + const payload = parseJson(result.stdout, 'dry-run JSON 输出'); + assertEqual(payload.mode, 'dry-run', 'dry-run JSON mode 应为 dry-run。'); + assertEqual( + payload.files?.[0]?.source?.sourceIsSymlink, + true, + 'dry-run JSON 应标记源证书来自 symlink。', + ); +} + +function assertApplyCopiesToPrivateTarget() { + const fixture = prepareFixture('apply-copy', { symlinkSources: false }); + const result = runSync(fixture, ['--apply'], { + serviceUser: `#${process.getuid?.() ?? 0}`, + serviceGroup: `#${process.getgid?.() ?? 0}`, + }); + assertStatus(result, 0, 'apply 应复制证书到目标目录。'); + assertFileContent(fixture.targetCert, 'fixture fullchain\n'); + assertFileContent(fixture.targetKey, 'fixture private key\n'); + assertMode(fixture.targetDir, 0o750, '目标目录权限应为 0750。'); + assertMode(fixture.targetCert, 0o640, '目标证书权限应为 0640。'); + assertMode(fixture.targetKey, 0o640, '目标私钥权限应为 0640。'); +} + +function assertRejectsRelativeAndRootPaths() { + const fixture = prepareFixture('reject-paths'); + const relative = runRaw([ + '--source-cert-file', + 'relative/fullchain.pem', + '--source-key-file', + fixture.sourceKey, + '--target-dir', + fixture.targetDir, + ]); + assertNonZero(relative, '相对源证书路径必须失败。'); + assertIncludes( + `${relative.stdout}\n${relative.stderr}`, + '--source-cert-file 必须是绝对路径', + '相对路径必须给出明确错误。', + ); + + const root = runRaw([ + '--source-cert-file', + fixture.sourceCert, + '--source-key-file', + fixture.sourceKey, + '--target-dir', + '/', + ]); + assertNonZero(root, 'target-dir 指向文件系统根目录必须失败。'); + assertIncludes( + `${root.stdout}\n${root.stderr}`, + '--target-dir 不能是文件系统根目录', + '根目录目标必须给出明确错误。', + ); +} + +function assertRejectsControlCharactersBeforeApply() { + const fixture = prepareFixture('reject-control'); + const result = runRaw([ + '--source-cert-file', + `${fixture.sourceCert}\n--apply`, + '--source-key-file', + fixture.sourceKey, + '--target-dir', + fixture.targetDir, + ]); + assertNonZero(result, '带换行的源证书路径必须失败。'); + assertIncludes( + `${result.stdout}\n${result.stderr}`, + '--source-cert-file 不能包含换行或 NUL 字符', + '控制字符路径必须给出明确错误。', + ); + if (existsSync(fixture.targetCert) || existsSync(fixture.targetKey)) { + failures.push('控制字符参数失败时不应写入目标证书。'); + } +} + +function assertRejectsSymlinkTargetDirectory() { + const fixture = prepareFixture('reject-target-dir-symlink'); + const realTarget = path.join(fixture.root, 'real-target'); + mkdirSync(realTarget, { recursive: true }); + rmSync(fixture.targetDir, { recursive: true, force: true }); + symlinkSync(realTarget, fixture.targetDir); + const result = runSync(fixture, []); + assertNonZero(result, 'target-dir 是 symlink 时必须失败。'); + assertIncludes( + `${result.stdout}\n${result.stderr}`, + '--target-dir 已存在路径不能包含符号链接', + 'target-dir symlink 必须给出明确错误。', + ); +} + +function assertRejectsSymlinkTargetFiles() { + const fixture = prepareFixture('reject-target-file-symlink'); + mkdirSync(fixture.targetDir, { recursive: true }); + const outside = path.join(fixture.root, 'outside-fullchain.pem'); + writeFileSync(outside, 'outside\n', 'utf8'); + symlinkSync(outside, fixture.targetCert); + const result = runSync(fixture, []); + assertNonZero(result, '目标证书是 symlink 时必须失败。'); + assertIncludes( + `${result.stdout}\n${result.stderr}`, + '目标 fullchain.pem 不能是符号链接', + '目标文件 symlink 必须给出明确错误。', + ); +} + +function assertRejectsWorldReadableModes() { + const fixture = prepareFixture('reject-world-readable-mode'); + const result = runSync(fixture, ['--file-mode', '0644']); + assertNonZero(result, 'world-readable file mode 必须失败。'); + assertIncludes( + `${result.stdout}\n${result.stderr}`, + '--file-mode 不能允许 other 读取证书或私钥', + 'world-readable file mode 必须给出明确错误。', + ); +} + +function prepareFixture(name, options = {}) { + const root = path.join(tmpRoot, name); + const archiveDir = path.join(root, 'archive'); + const liveDir = path.join(root, 'live'); + const targetDir = path.join(root, 'target'); + mkdirSync(archiveDir, { recursive: true }); + mkdirSync(liveDir, { recursive: true }); + writeFileSync(path.join(archiveDir, 'fullchain1.pem'), 'fixture fullchain\n', 'utf8'); + writeFileSync(path.join(archiveDir, 'privkey1.pem'), 'fixture private key\n', 'utf8'); + + const symlinkSources = options.symlinkSources !== false; + const sourceCert = symlinkSources + ? path.join(liveDir, 'fullchain.pem') + : path.join(archiveDir, 'fullchain1.pem'); + const sourceKey = symlinkSources + ? path.join(liveDir, 'privkey.pem') + : path.join(archiveDir, 'privkey1.pem'); + if (symlinkSources) { + symlinkSync('../archive/fullchain1.pem', sourceCert); + symlinkSync('../archive/privkey1.pem', sourceKey); + } + + return { + root, + sourceCert, + sourceKey, + targetDir, + targetCert: path.join(targetDir, 'fullchain.pem'), + targetKey: path.join(targetDir, 'privkey.pem'), + }; +} + +function runSync(fixture, args = [], options = {}) { + return runRaw([ + '--source-cert-file', + fixture.sourceCert, + '--source-key-file', + fixture.sourceKey, + '--target-dir', + fixture.targetDir, + '--service-user', + options.serviceUser || `#${process.getuid?.() ?? 0}`, + '--service-group', + options.serviceGroup || `#${process.getgid?.() ?? 0}`, + '--json', + ...args, + ]); +} + +function runRaw(args) { + return spawnSync('node', [SCRIPT, ...args], { + cwd: process.cwd(), + encoding: 'utf8', + }); +} + +function parseJson(text, label) { + try { + return JSON.parse(text); + } catch (error) { + failures.push(`${label} 不是合法 JSON: ${error.message}\n${text}`); + return {}; + } +} + +function assertStatus(result, expected, reason) { + if ((result.status ?? 0) !== expected) { + failures.push( + `${reason} 实际退出码 ${result.status}。\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, + ); + } +} + +function assertNonZero(result, reason) { + if ((result.status ?? 0) === 0) { + failures.push(`${reason}\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`); + } +} + +function assertIncludes(value, expected, reason) { + if (!String(value).includes(expected)) { + failures.push(`${reason} 缺少: ${expected}`); + } +} + +function assertEqual(actual, expected, reason) { + if (actual !== expected) { + failures.push(`${reason} 实际 ${JSON.stringify(actual)}，预期 ${JSON.stringify(expected)}`); + } +} + +function assertFileContent(filePath, expected) { + if (!existsSync(filePath)) { + failures.push(`缺少文件: ${filePath}`); + return; + } + const actual = readFileSync(filePath, 'utf8'); + if (actual !== expected) { + failures.push(`${filePath} 内容不符合预期。`); + } +} + +function assertMode(filePath, expected, reason) { + const mode = statSync(filePath).mode & 0o777; + if (mode !== expected) { + failures.push(`${reason} 实际 0${mode.toString(8)}，预期 0${expected.toString(8)}。`); + } +} diff --git a/scripts/check-production-api-deploy.mjs b/scripts/check-production-api-deploy.mjs index 21843139c..8920d7035 100644 --- a/scripts/check-production-api-deploy.mjs +++ b/scripts/check-production-api-deploy.mjs @@ -62,6 +62,7 @@ function main() { assertMissingHealthPatrolEnvCheckFails(); assertMissingPingoraReleaseReadinessFails(); assertMissingPingoraHealthPatrolEnvSwitchFails(); + assertMissingPingoraTlsCertSyncFails(); assertMissingEnvExamplesFails(); assertMissingPingoraDirectCheckFails(); assertMissingPingoraCanaryLiveFails(); @@ -105,6 +106,10 @@ function assertDeployCopiesPingoraDirectReleaseDependencies() { ), 'current release 必须包含 Pingora health patrol env 切换脚本。', ); + assertFileExists( + path.join(releaseDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'), + 'current release 必须包含 Pingora TLS 证书同步脚本。', + ); assertFileExists( path.join(releaseDir, 'scripts/check-pingora-direct-preflight.mjs'), 'current release 必须包含 Pingora 直连预检脚本。', @@ -885,6 +890,29 @@ function assertMissingPingoraHealthPatrolEnvSwitchFails() { } } +function assertMissingPingoraTlsCertSyncFails() { + const fixture = prepareFixture('missing-pingora-tls-cert-sync'); + rmSync( + path.join( + fixture.sourceDir, + 'scripts/deploy/pingora-tls-cert-sync.mjs', + ), + ); + const result = runDeploy(fixture); + + if (result.status === 0) { + failures.push('发布产物缺少 Pingora TLS 证书同步脚本时部署必须失败。'); + } + assertIncludes( + result.stderr, + '发布产物缺少 Pingora TLS 证书同步脚本', + '缺少 Pingora TLS 证书同步脚本时必须给出明确错误。', + ); + if (!existsSync(fixture.maintenanceFile)) { + failures.push('部署失败时必须保持维护模式。'); + } +} + function assertMissingEnvExamplesFails() { const fixture = prepareFixture('missing-env-examples'); rmSync(path.join(fixture.sourceDir, 'deploy/env'), { @@ -1020,6 +1048,10 @@ function prepareFixture(name) { 'scripts/deploy/pingora-health-patrol-env-switch.mjs', path.join(sourceDir, 'scripts/deploy/pingora-health-patrol-env-switch.mjs'), ); + copyFile( + 'scripts/deploy/pingora-tls-cert-sync.mjs', + path.join(sourceDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'), + ); copyFile( 'scripts/check-production-health-patrol-env.mjs', path.join(sourceDir, 'scripts/check-production-health-patrol-env.mjs'), diff --git a/scripts/check-production-api-release.mjs b/scripts/check-production-api-release.mjs index 27647404c..151ccd3f3 100644 --- a/scripts/check-production-api-release.mjs +++ b/scripts/check-production-api-release.mjs @@ -116,6 +116,14 @@ function assertApiReleaseContainsPingoraDirectDependencies() { ), 'Pingora health patrol env 切换脚本必须可执行。', ); + assertFileExists( + path.join(releaseDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'), + 'API release 必须包含 Pingora TLS 证书同步脚本。', + ); + assertExecutable( + path.join(releaseDir, 'scripts/deploy/pingora-tls-cert-sync.mjs'), + 'Pingora TLS 证书同步脚本必须可执行。', + ); assertFileExists( path.join(releaseDir, 'scripts/check-production-health-patrol-env.mjs'), 'API release 必须包含健康巡检 env 复核脚本。', @@ -246,7 +254,7 @@ function assertApiReleaseContainsPingoraDirectDependencies() { ); assertIncludes( releaseReadme, - 'Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本', + 'Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本', 'API release README 必须说明随包携带 Pingora 直连脚本。', ); assertIncludes( diff --git a/scripts/deploy/pingora-tls-cert-sync.mjs b/scripts/deploy/pingora-tls-cert-sync.mjs new file mode 100644 index 000000000..e1f7beb6f --- /dev/null +++ b/scripts/deploy/pingora-tls-cert-sync.mjs @@ -0,0 +1,470 @@ +#!/usr/bin/env node + +import { randomBytes } from 'node:crypto'; +import { + accessSync, + chmodSync, + chownSync, + constants, + copyFileSync, + existsSync, + lstatSync, + mkdirSync, + realpathSync, + renameSync, + rmSync, + statSync, +} from 'node:fs'; +import { execFileSync, spawnSync } from 'node:child_process'; +import path from 'node:path'; +import { userInfo } from 'node:os'; + +const DEFAULT_SERVICE_USER = 'genarrative'; +const DEFAULT_SERVICE_GROUP = 'genarrative'; +const DEFAULT_DIRECTORY_MODE = '0750'; +const DEFAULT_FILE_MODE = '0640'; + +const config = parseArgs(process.argv.slice(2)); +const result = run(); + +if (config.json) { + console.log(`${JSON.stringify(result, null, 2)}\n`); +} else { + const mode = config.apply ? 'apply' : 'dry-run'; + console.log(`[pingora-tls-cert-sync] ${mode} OK`); + console.log(`[pingora-tls-cert-sync] target=${result.target.directory}`); + for (const item of result.files) { + console.log( + `[pingora-tls-cert-sync] ${item.kind}: ${item.source.path} -> ${item.target.path}`, + ); + } +} + +function usage() { + console.log(`Usage: + node scripts/deploy/pingora-tls-cert-sync.mjs --source-cert-file --source-key-file --target-dir [options] + +Options: + --source-cert-file 已存在的证书链文件，通常是 Let's Encrypt live/fullchain.pem。 + --source-key-file 已存在的私钥文件，通常是 Let's Encrypt live/privkey.pem。 + --target-dir Pingora 私有 TLS 目录，例如 /etc/genarrative/pingora-tls/。 + --service-user 目标服务用户，默认 genarrative。 + --service-group + 目标服务组，默认 genarrative。 + --directory-mode 目标目录权限，默认 0750。 + --file-mode 证书和私钥权限，默认 0640。 + --apply 执行复制；默认只做 dry-run 校验。 + --json 输出 JSON。 + +该脚本只把现有证书复制到 Pingora 私有目录，不修改 Let's Encrypt / Certbot / Nginx 原始文件。 +源路径可以是 Certbot live symlink，但解析后的目标必须是普通文件；目标目录和目标文件不允许是 symlink。 +`); +} + +function parseArgs(argv) { + const result = { + sourceCertFile: + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SOURCE_CERT_FILE || '', + sourceKeyFile: + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SOURCE_KEY_FILE || '', + targetDir: process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_TARGET_DIR || '', + serviceUser: + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SERVICE_USER || + DEFAULT_SERVICE_USER, + serviceGroup: + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_SERVICE_GROUP || + DEFAULT_SERVICE_GROUP, + directoryMode: + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_DIRECTORY_MODE || + DEFAULT_DIRECTORY_MODE, + fileMode: + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_FILE_MODE || + DEFAULT_FILE_MODE, + apply: readBoolEnv( + process.env.GENARRATIVE_PINGORA_TLS_CERT_SYNC_APPLY, + false, + 'GENARRATIVE_PINGORA_TLS_CERT_SYNC_APPLY', + ), + json: false, + }; + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + switch (arg) { + case '-h': + case '--help': + usage(); + process.exit(0); + break; + case '--source-cert-file': + result.sourceCertFile = requireValue(argv, ++index, arg); + break; + case '--source-key-file': + result.sourceKeyFile = requireValue(argv, ++index, arg); + break; + case '--target-dir': + result.targetDir = requireValue(argv, ++index, arg); + break; + case '--service-user': + result.serviceUser = requireValue(argv, ++index, arg); + break; + case '--service-group': + result.serviceGroup = requireValue(argv, ++index, arg); + break; + case '--directory-mode': + result.directoryMode = requireValue(argv, ++index, arg); + break; + case '--file-mode': + result.fileMode = requireValue(argv, ++index, arg); + break; + case '--apply': + result.apply = true; + break; + case '--json': + result.json = true; + break; + default: + throw new Error(`未知参数: ${arg}`); + } + } + + validateSafeAbsoluteFilePath( + result.sourceCertFile, + '--source-cert-file', + ); + validateSafeAbsoluteFilePath(result.sourceKeyFile, '--source-key-file'); + validateSafeAbsoluteDirectoryPath(result.targetDir, '--target-dir'); + validateIdentity(result.serviceUser, '--service-user'); + validateIdentity(result.serviceGroup, '--service-group'); + result.directoryModeNumber = parseMode( + result.directoryMode, + '--directory-mode', + ); + result.fileModeNumber = parseMode(result.fileMode, '--file-mode'); + if ((result.fileModeNumber & 0o007) !== 0) { + throw new Error('--file-mode 不能允许 other 读取证书或私钥。'); + } + if ((result.directoryModeNumber & 0o007) !== 0) { + throw new Error('--directory-mode 不能允许 other 进入 TLS 私有目录。'); + } + return result; +} + +function requireValue(argv, index, flag) { + const value = argv[index]; + if (!value || value.startsWith('--')) { + throw new Error(`${flag} 缺少参数值`); + } + return value; +} + +function readBoolEnv(raw, fallback, label) { + if (raw === undefined || raw === null || String(raw).trim() === '') { + return fallback; + } + validateNoControlCharacters(raw, label); + const normalized = String(raw).trim().toLowerCase(); + if (['1', 'true', 'yes', 'on'].includes(normalized)) { + return true; + } + if (['0', 'false', 'no', 'off'].includes(normalized)) { + return false; + } + throw new Error(`${label} 必须是布尔值 true/false 或 1/0。`); +} + +function validateSafeAbsoluteFilePath(value, label) { + validateNoControlCharacters(value, label); + if (!path.isAbsolute(value)) { + throw new Error(`${label} 必须是绝对路径。`); + } + if (isFilesystemRootPath(value)) { + throw new Error(`${label} 不能是文件系统根目录。`); + } +} + +function validateSafeAbsoluteDirectoryPath(value, label) { + validateSafeAbsoluteFilePath(value, label); + if (path.basename(path.resolve(value)) === '..') { + throw new Error(`${label} 不能以 .. 结尾。`); + } +} + +function validateNoControlCharacters(value, label) { + if (/[\0\r\n]/u.test(String(value))) { + throw new Error(`${label} 不能包含换行或 NUL 字符。`); + } +} + +function validateIdentity(value, label) { + validateNoControlCharacters(value, label); + if (!/^[A-Za-z0-9_.#-]+$/u.test(String(value))) { + throw new Error(`${label} 只能包含字母、数字、点、下划线、短横线或 #。`); + } +} + +function parseMode(value, label) { + validateNoControlCharacters(value, label); + const text = String(value).trim(); + if (!/^0?[0-7]{3}$/u.test(text)) { + throw new Error(`${label} 必须是三位八进制权限，例如 0640。`); + } + return Number.parseInt(text, 8); +} + +function isFilesystemRootPath(value) { + const resolved = path.resolve(String(value)); + return resolved === path.parse(resolved).root; +} + +function run() { + const certSource = inspectSourceFile( + config.sourceCertFile, + '--source-cert-file', + ); + const keySource = inspectSourceFile(config.sourceKeyFile, '--source-key-file'); + const certTarget = path.join(config.targetDir, 'fullchain.pem'); + const keyTarget = path.join(config.targetDir, 'privkey.pem'); + const owner = config.apply + ? resolveOwner(config.serviceUser, config.serviceGroup) + : { uid: null, gid: null }; + + assertNoSymlinkAncestors(config.targetDir, '--target-dir'); + assertTargetPath(certTarget, '目标 fullchain.pem'); + assertTargetPath(keyTarget, '目标 privkey.pem'); + if (existsSync(config.targetDir)) { + assertTargetDirectory(config.targetDir); + } + + const planned = { + mode: config.apply ? 'apply' : 'dry-run', + target: { + directory: config.targetDir, + user: config.serviceUser, + group: config.serviceGroup, + uid: owner.uid, + gid: owner.gid, + directoryMode: formatMode(config.directoryModeNumber), + fileMode: formatMode(config.fileModeNumber), + }, + files: [ + { + kind: 'cert', + source: certSource, + target: { path: certTarget }, + }, + { + kind: 'key', + source: keySource, + target: { path: keyTarget }, + }, + ], + serviceUserReadable: false, + }; + + if (!config.apply) { + return planned; + } + + mkdirSync(config.targetDir, { recursive: true }); + assertTargetDirectory(config.targetDir); + chmodSync(config.targetDir, config.directoryModeNumber); + applyOwner(config.targetDir, owner); + + copyAtomic(certSource.resolvedPath, certTarget, owner); + copyAtomic(keySource.resolvedPath, keyTarget, owner); + + planned.serviceUserReadable = assertServiceUserReadable( + config.serviceUser, + [certTarget, keyTarget], + ); + return planned; +} + +function inspectSourceFile(filePath, label) { + const linkStat = lstatSync(filePath); + const sourceIsSymlink = linkStat.isSymbolicLink(); + const resolvedPath = realpathSync(filePath); + const fileStat = statSync(resolvedPath); + if (!fileStat.isFile()) { + throw new Error(`${label} 解析后必须是普通文件: ${resolvedPath}`); + } + return { + path: filePath, + resolvedPath, + sourceIsSymlink, + sizeBytes: fileStat.size, + }; +} + +function assertTargetDirectory(directory) { + const linkStat = lstatSync(directory); + if (linkStat.isSymbolicLink()) { + throw new Error(`--target-dir 不能是符号链接: ${directory}`); + } + if (!linkStat.isDirectory()) { + throw new Error(`--target-dir 已存在但不是目录: ${directory}`); + } +} + +function assertNoSymlinkAncestors(targetDir, label) { + const resolved = path.resolve(targetDir); + const parsed = path.parse(resolved); + let current = parsed.root; + for (const part of resolved.slice(parsed.root.length).split(path.sep)) { + if (!part) { + continue; + } + current = path.join(current, part); + if (!existsSync(current)) { + break; + } + const linkStat = lstatSync(current); + if (linkStat.isSymbolicLink()) { + throw new Error(`${label} 已存在路径不能包含符号链接: ${current}`); + } + if (!linkStat.isDirectory()) { + throw new Error(`${label} 已存在父路径不是目录: ${current}`); + } + } +} + +function assertTargetPath(filePath, label) { + if (!existsSync(filePath)) { + return; + } + const linkStat = lstatSync(filePath); + if (linkStat.isSymbolicLink()) { + throw new Error(`${label} 不能是符号链接: ${filePath}`); + } + if (!linkStat.isFile()) { + throw new Error(`${label} 已存在但不是普通文件: ${filePath}`); + } +} + +function resolveOwner(user, group) { + const current = currentIdentity(); + const uid = resolveUserId(user, current); + const gid = resolveGroupId(group, current, user); + if (!isCurrentUserRoot()) { + if (uid !== current.uid || gid !== current.gid) { + throw new Error( + `当前用户不是 root，不能把证书授权给 ${user}:${group}；请用 sudo -n 执行 --apply。`, + ); + } + } + return { uid, gid }; +} + +function resolveUserId(user, current) { + if (/^#?\d+$/u.test(user)) { + return Number.parseInt(user.replace(/^#/u, ''), 10); + } + if (user === current.username) { + return current.uid; + } + return Number.parseInt( + execFileSync('id', ['-u', user], { encoding: 'utf8' }).trim(), + 10, + ); +} + +function resolveGroupId(group, current, user) { + if (/^#?\d+$/u.test(group)) { + return Number.parseInt(group.replace(/^#/u, ''), 10); + } + if (group === current.groupname || group === String(current.gid)) { + return current.gid; + } + const getent = spawnSync('getent', ['group', group], { encoding: 'utf8' }); + if (getent.status === 0 && getent.stdout.trim()) { + const parts = getent.stdout.trim().split(':'); + return Number.parseInt(parts[2], 10); + } + if (group === user) { + return Number.parseInt( + execFileSync('id', ['-g', user], { encoding: 'utf8' }).trim(), + 10, + ); + } + throw new Error(`无法解析服务组: ${group}`); +} + +function currentIdentity() { + const uid = typeof process.getuid === 'function' ? process.getuid() : -1; + const gid = typeof process.getgid === 'function' ? process.getgid() : -1; + let username = ''; + try { + username = userInfo().username; + } catch { + username = process.env.USER || process.env.LOGNAME || ''; + } + return { + uid, + gid, + username, + groupname: '', + }; +} + +function isCurrentUserRoot() { + return typeof process.getuid === 'function' && process.getuid() === 0; +} + +function applyOwner(filePath, owner) { + const current = currentIdentity(); + if (!isCurrentUserRoot() && owner.uid === current.uid && owner.gid === current.gid) { + return; + } + chownSync(filePath, owner.uid, owner.gid); +} + +function copyAtomic(sourcePath, targetPath, owner) { + const tempPath = path.join( + path.dirname(targetPath), + `.${path.basename(targetPath)}.tmp-${process.pid}-${randomBytes(4).toString('hex')}`, + ); + try { + copyFileSync(sourcePath, tempPath, constants.COPYFILE_EXCL); + chmodSync(tempPath, config.fileModeNumber); + applyOwner(tempPath, owner); + renameSync(tempPath, targetPath); + chmodSync(targetPath, config.fileModeNumber); + applyOwner(targetPath, owner); + } catch (error) { + rmSync(tempPath, { force: true }); + throw error; + } +} + +function assertServiceUserReadable(serviceUser, files) { + if (/^#?\d+$/u.test(serviceUser)) { + for (const file of files) { + accessSync(file, constants.R_OK); + } + return true; + } + const current = currentIdentity(); + if (serviceUser === current.username && !isCurrentUserRoot()) { + for (const file of files) { + accessSync(file, constants.R_OK); + } + return true; + } + for (const file of files) { + const check = spawnSync('sudo', ['-n', '-u', serviceUser, 'test', '-r', file], { + encoding: 'utf8', + }); + if (check.status !== 0) { + const detail = (check.stderr || check.stdout || '').trim(); + throw new Error( + `服务用户 ${serviceUser} 不可读: ${file}${detail ? ` (${detail})` : ''}`, + ); + } + } + return true; +} + +function formatMode(mode) { + return `0${mode.toString(8).padStart(3, '0')}`; +} diff --git a/scripts/deploy/production-api-deploy.sh b/scripts/deploy/production-api-deploy.sh index 7a017686b..142d32e45 100644 --- a/scripts/deploy/production-api-deploy.sh +++ b/scripts/deploy/production-api-deploy.sh @@ -620,6 +620,7 @@ PINGORA_RELEASE_READINESS_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-rel PINGORA_ENABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-enable.sh" PINGORA_ROLLBACK_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-rollback.sh" PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" +PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" PINGORA_PREFLIGHT_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-direct-preflight.mjs" PINGORA_LIVE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-direct-live.mjs" PINGORA_CANARY_LIVE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-canary-live.mjs" @@ -707,6 +708,12 @@ if [[ ! -f "${PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE}" ]]; then fi cp "${PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" +if [[ ! -f "${PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE}" ]]; then + echo "[production-api-deploy] 发布产物缺少 Pingora TLS 证书同步脚本: ${SOURCE_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" >&2 + exit 1 +fi +cp "${PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" +chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" if [[ ! -f "${PINGORA_PREFLIGHT_SCRIPT_SOURCE}" ]]; then echo "[production-api-deploy] 发布产物缺少 Pingora 直连预检脚本: ${SOURCE_DIR}/scripts/check-pingora-direct-preflight.mjs" >&2 exit 1 diff --git a/scripts/ops/pingora-current-release-audit.mjs b/scripts/ops/pingora-current-release-audit.mjs index 1a2c7b968..e73da3f64 100644 --- a/scripts/ops/pingora-current-release-audit.mjs +++ b/scripts/ops/pingora-current-release-audit.mjs @@ -22,6 +22,7 @@ const REQUIRED_ARTIFACTS = [ { path: 'scripts/deploy/pingora-direct-enable.sh', executable: true }, { path: 'scripts/deploy/pingora-direct-rollback.sh', executable: true }, { path: 'scripts/deploy/pingora-health-patrol-env-switch.mjs', executable: true }, + { path: 'scripts/deploy/pingora-tls-cert-sync.mjs', executable: true }, { path: 'deploy/systemd/genarrative-pingora-gateway.service' }, { path: 'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf' }, { path: 'deploy/nginx/snippets/genarrative-pingora-canary.conf' },