diff --git a/deploy/nginx/README.md b/deploy/nginx/README.md index ac9a3f253..4ab54ceb0 100644 --- a/deploy/nginx/README.md +++ b/deploy/nginx/README.md @@ -68,7 +68,8 @@ - 启用 Nginx canary 前，目标机必须已经有持久 `genarrative-pingora-gateway.service` 运行在 `127.0.0.1:18081`，且带真实 `api-server`、SpacetimeDB、静态目录和 `/var/log/genarrative/pingora-gateway.access.log` 完成本机 shadow 验收；不要从一次性 `/tmp` 网关进程直接切到 Nginx handoff。 - 启用前必须运行 `npm run check:nginx-pingora-canary`；目标 agent 有 Nginx 时运行 `node scripts/check-nginx-pingora-canary.mjs --require-nginx`，同时做两个 snippet 的静态护栏和真实 `nginx -t`。本机或 CI 可运行 `npm run check:pingora-canary-docker`，用 Docker Nginx、真实 Pingora 和 mock 上游复现前缀 canary 与真实路径 canary 两条 handoff 链路。 - 前缀 canary reload 后运行 `GENARRATIVE_PINGORA_CANARY_BASE_URL=http://127.0.0.1 GENARRATIVE_PINGORA_CANARY_HOST=<域名> npm run check:pingora-canary-live`，再用 `scripts/check-pingora-canary-access-log-parity.mjs` 对账 `/var/log/nginx/genarrative.access.log` 与 `/var/log/genarrative/pingora-gateway.access.log`。 -- 真实路径 canary reload 后运行 `node scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再运行 `node scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js`。真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致。 +- 真实路径 canary 不再手工编辑 `/etc/nginx/conf.d/`。目标机从 current release 执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083`；脚本会渲染到 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，保持晚于主站 `log_format genarrative_upstream` 加载，执行 `nginx -t`、reload Nginx，并默认运行 realpath live smoke。`nginx -t`、reload 或 live smoke 任一失败都会恢复写入前配置。关闭时执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`，脚本会删除该 `zz-...` 文件，执行 `nginx -t` 和 reload，失败时恢复删除前配置。 +- 真实路径 canary reload 后可再次运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js`。真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致。 - 正式切换前在源码 checkout / CI / 构建环境跑 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；启用真实路径 canary 时追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。 - 目标机 `/opt/genarrative/current` 只能跑随包 runtime-only 门禁：`/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；验证真实路径 canary 或 Pingora 直连入口时分别追加 `--require-realpath-live ...` 或 `--require-direct ...` 参数。生成正式直连切换 runbook 时，`--direct-redirect-host` 和回退 smoke Host 必须与 `--direct-host` 同 hostname，只允许端口不同。 diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index b87df7a6a..8fd688ad1 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -129,8 +129,8 @@ - 现象：目标机把 `genarrative-pingora-realpath-canary.conf` 放进 `/etc/nginx/conf.d/` 后，`nginx -t` 失败并报 `unknown log format "genarrative_upstream"`。 - 原因：真实路径 canary 是独立 `server` 片段，并使用 `access_log /var/log/nginx/genarrative-pingora-realpath-canary.access.log genarrative_upstream;`。Nginx 会按文件名顺序加载 `conf.d`；如果 canary 文件名早于定义 `log_format genarrative_upstream` 的主站配置，access log 行会先被解析而找不到格式。 -- 处理：真实路径 canary 要么 include 在已经定义 `log_format genarrative_upstream` 之后，要么放入晚于主站配置加载的文件，例如 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`；另一种长期做法是把 `log_format` 放到所有 `conf.d` server 之前的全局 Nginx 配置。启用前必须先跑 `nginx -t`，失败时先移除临时 canary 文件再 reload，避免保留坏配置；检查配置时不要把 probe token 原文写入记录。 -- 验证：`nginx -t` 通过后 reload Nginx，再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`、`node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log ...` 和 `node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...`。若只启用了真实路径 canary，不要同时传 `--require-live`，否则前缀 canary 未启用时会按正式 Nginx HTTP 入口返回 301。 +- 处理：真实路径 canary 启停统一用 current release 随包脚本，不再手工写 `/etc/nginx/conf.d/`。启用执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083`，脚本固定写入晚于主站配置加载的 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，并在 `nginx -t`、reload 或 live smoke 失败时恢复写入前配置。关闭执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`，脚本在 `nginx -t` 或 reload 失败时恢复删除前配置。另一种长期做法是把 `log_format` 放到所有 `conf.d` server 之前的全局 Nginx 配置。检查配置时不要把 probe token 原文写入记录。 +- 验证：启用脚本通过后，再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`、`node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log ...` 和 `node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...`。若只启用了真实路径 canary，不要同时传 `--require-live`，否则前缀 canary 未启用时会按正式 Nginx HTTP 入口返回 301。 - 关联：`deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf`、`deploy/nginx/README.md`、`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`、`scripts/check-pingora-release-readiness.mjs`。 ## Pingora release readiness 脚本不能只存在于源码 checkout diff --git a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md index fdecfb491..7ef9331dd 100644 --- a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md +++ b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md @@ -375,6 +375,7 @@ dev 根盘空间在安装后曾接近满盘；2026-06-17 进入 canary 前已清 - `node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath ... --json` 对账 `12/12 matched`，`missingCount=0`，`mismatchCount=0`。 - `node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...` 通过 current release 自包含自审、realpath live smoke 和 realpath access log 对账，最终对账 `18/18 matched`。 - 本阶段没有启用前缀 canary，因此不能同时传 `--require-live`；否则正式 HTTP 入口会按 Nginx 策略返回 `301`，并缺少 `X-Genarrative-Nginx-Handoff: pingora-canary`。当前 public `80/443` 仍由 Nginx 承接，realpath canary 只保留在 `127.0.0.1:18083` 用于后续 loopback 复核。 +- 后续 realpath canary 启停统一走 current release 随包脚本：`/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083` 和 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`。启用脚本固定写入 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，并在 `nginx -t`、reload 或 live smoke 失败时回滚；关闭脚本在 `nginx -t` 或 reload 失败时恢复删除前配置。 ## dev API release 正式路径验收记录 @@ -477,10 +478,10 @@ dev 根盘空间在安装后曾接近满盘；2026-06-17 进入 canary 前已清 2. 涉及 Nginx 模板、Pingora 路由、限流分组或路由文档时，同步更新 `deploy/pingora/nginx-route-parity.matrix.json`，并运行 `npm run check:pingora-route-parity` 与 `cargo test -p pingora-gateway --manifest-path server-rs/Cargo.toml matches_nginx_route_parity_matrix`。 3. 容器内使用同一份 Web 产物、同一组真实上游地址跑 Pingora smoke，并继续对照 `deploy/nginx/genarrative.conf` 扩展真实上游路由 parity 自动测试。 4. 使用 `deploy/nginx/snippets/genarrative-pingora-canary.conf` 做 Nginx 前缀 canary；启用前先跑 `npm run check:nginx-pingora-canary` 和 `npm run check:pingora-canary-docker`，有 Nginx 或 Docker 的目标环境分别强制跑 `node scripts/check-nginx-pingora-canary.mjs --require-nginx` 与 `node scripts/check-pingora-canary-docker.mjs --require-docker --pull`，其中 Docker handoff 会自动对账临时 Nginx 与 Pingora access log。启用后跑 `npm run check:pingora-canary-live`，再用 current release 随包 access log parity 脚本对账目标机 Nginx 与 Pingora access log。canary live 的 base URL、prefix、Host、额外 path 和 timeout 不能包含换行或 NUL；canary live timeout 和 access log `since-lines` 必须是正整数，非法值直接失败。 -5. 前缀 canary 通过后，再使用 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 做真实路径 canary；它必须作为独立本机 `server` include 到 Nginx `http` 上下文，不能 include 到生产 `443` server 内。该片段使用 `access_log ... genarrative_upstream`，如果放入 `/etc/nginx/conf.d/`，文件名必须保证晚于定义 `log_format genarrative_upstream` 的主站配置加载，例如 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`；否则 `nginx -t` 会报 `unknown log format "genarrative_upstream"`。启用后跑 `node scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再用 current release 随包 access log parity 脚本传 `--realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log` 对账 `/api/creation-entry/config`、`/v1/identity` 和 `/assets/app.js` 等真实路径。 +5. 前缀 canary 通过后，再使用 current release 随包 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083` 启用真实路径 canary；它会把 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 渲染成独立本机 `server`，写入 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，不能 include 到生产 `443` server 内。该片段使用 `access_log ... genarrative_upstream`，文件名必须保证晚于定义 `log_format genarrative_upstream` 的主站配置加载；否则 `nginx -t` 会报 `unknown log format "genarrative_upstream"`。启用脚本会先执行 `nginx -t`、reload Nginx，再默认运行 realpath live smoke，任一阶段失败都会恢复写入前配置。关闭时执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`，脚本会在 `nginx -t` 或 reload 失败时恢复删除前配置。启用后跑 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再用 current release 随包 access log parity 脚本传 `--realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log` 对账 `/api/creation-entry/config`、`/v1/identity` 和 `/assets/app.js` 等真实路径。 6. 目标机 canary include 后必须跑正式切换聚合门禁：源码 checkout / CI / 构建环境执行默认全量命令 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；目标机 current release 执行 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。启用真实路径 canary 时，两种命令都追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。缺少 Host 会直接失败，避免 live canary 误测默认 vhost；live smoke 后还会按 `request_id` 对账 Nginx 与 Pingora access log，缺少同一请求的 Pingora 日志、状态码、方法或 path 漂移都会失败。 7. 如需评估 Pingora 直连公网入口，必须显式配置 `TLS_LISTEN`、证书、私钥和 `HTTP_REDIRECT_LISTEN`；Certbot 证书先用随包 `scripts/deploy/pingora-tls-cert-sync.mjs` 同步到 `/etc/genarrative/pingora-tls/<域名>/`，不要直接 chmod Let’s Encrypt live/archive 原路径；绑定 `80/443` 时还必须人工启用 `genarrative-pingora-gateway-direct-entry.conf` drop-in 授予 `CAP_NET_BIND_SERVICE`。随后用 `npm run check:pingora-gateway-smoke` 覆盖 TLS / HTTP/2 ALPN / redirect / WSS subscribe；目标机必须先跑 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`，再跑 `npm run check:pingora-direct-live` 或 release readiness 的 `--require-direct`，且 `--require-direct` 必须带 direct HTTPS base URL、direct HTTP base URL、正式域名 Host/SNI、redirect Location host、Pingora access log 文件、health patrol env 文件、direct preflight env 文件、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户和 systemd 服务用户证书可读检查、service 二进制可执行检查、端口释放检查和显式 SpacetimeDB 数据库名，并会拒绝 `--skip-wss`。高端口 rehearsal 使用 `https://127.0.0.1:<高端口>` 打入但期望 HTTP redirect Location 指向正式域名默认 HTTPS 入口时，额外传 `--direct-redirect-base-url https://<域名>`；`--direct-redirect-host` 仍必须保留，用于 runbook Host 一致性约束。direct live 会用生成的 `request_id` 反查 Pingora access log；缺少对应日志、method 漂移、path 漂移或 status 漂移都算直连门禁失败。direct preflight 会拒绝开启网关保护但未确认共享保护层的 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT>1` 配置；`--env-file`、`--systemd-service`、服务用户和 env 中的 listen / cert / key 值都不能包含换行或 NUL，执行 `systemctl cat` 或 `sudo -u test -r ` 前还会复核子命令参数，避免污染参数进入目标机预检命令；direct live 的 URL、Host、redirect base URL、probe token、额外 path、数据库名、access log 路径、timeout 和布尔 env 也不能包含换行或 NUL，且会在发起请求前失败；direct live timeout 必须是正整数，直连相关布尔 env 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。证书申请与续期仍由 Certbot / 外部自动化承担，网关只读取现有文件。 -8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs` 和 `deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release；缺失时部署应 fail-fast，切换窗口不能依赖源码 checkout 或 Jenkins workspace。 +8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs`、realpath canary 启停脚本和 `deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release；缺失时部署应 fail-fast，切换窗口不能依赖源码 checkout 或 Jenkins workspace。 9. direct preflight / direct live timeout 必须是正整数；直连相关布尔 env 和 Pingora gateway env 里的 `TRUST_X_FORWARDED_FOR` / `TRUSTED_FRONT_PROXY_CONFIRMED` 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。 10. Pingora 正式化口径固定为 gzip-only；Brotli 继续由 Nginx / 前置代理承担，直连 Pingora 不以 Brotli parity 作为切换门禁。如需多实例，开启网关接流保护时必须先引入共享限流 / 共享并发保护层并设置 `GENARRATIVE_PINGORA_GATEWAY_SHARED_PROTECTION_CONFIRMED=true`，否则保持 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT=1`；关闭网关保护的多实例方案必须明确由前置 Nginx / LB 承担全局限流。 10. 前缀 canary 稳定后，再评估是否做真实路径 canary；真实路径 canary 稳定后，再评估是否让 Pingora 直接承接公网入口。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index c115880e5..98cd0ff95 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -322,7 +322,7 @@ current release 随包执行的 release readiness 必须追加 `--release-runtim 涉及 Nginx 模板、Pingora 路由、限流分组或路由文档时，必须同步更新 `deploy/pingora/nginx-route-parity.matrix.json`，并运行 `npm run check:pingora-route-parity`；Rust 侧 `cargo test -p pingora-gateway --manifest-path server-rs/Cargo.toml matches_nginx_route_parity_matrix` 会读取同一份矩阵验证 `classify_path` 的路由结果、body limit 和接流保护分组。 -Pingora canary 入口默认由 Server-Provision 安装但不接入主站配置。`deploy/nginx/snippets/genarrative-pingora-canary.conf` 是前缀 canary，人工 include 后用 `/__genarrative_pingora_canary/` rewrite 到 Pingora shadow；`deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 是前缀 canary 之后、direct 直连之前的真实路径 canary，只能在 Nginx `http` 上下文 include，默认监听 `127.0.0.1:18083` 并写独立 `/var/log/nginx/genarrative-pingora-realpath-canary.access.log`，不要 include 到生产 `443` server 内覆盖正式 location。启用前必须运行 `npm run check:nginx-pingora-canary`；目标 agent 有 Nginx 时运行 `node scripts/check-nginx-pingora-canary.mjs --require-nginx`，强制把前缀 location 片段和真实路径 server 片段一起执行 `nginx -t`。本机或 CI 可用 `npm run check:pingora-canary-docker` 启动 Docker Nginx、真实 Pingora 和 mock 上游复现两条 Nginx -> Pingora handoff，并复用 `scripts/check-pingora-canary-access-log-parity.mjs` 按 `request_id` 对账。需要证明 Docker Nginx -> Pingora -> 真实本地服务时，先启动真实 api-server 与 SpacetimeDB，再运行 `node scripts/check-pingora-canary-docker.mjs --require-docker --real-upstreams --api-upstream 127.0.0.1: --spacetime-upstream 127.0.0.1: --web-root dist`；该模式不会启动内置 mock 上游，会先检查真实 `/healthz` 和 `/v1/ping`，随后复用同一组 live smoke 与 access log 对账。真实 SpacetimeDB 对 `GET /v1/identity` 返回 `405 Method Not Allowed` 属于可接受语义，canary live 只把它作为路径路由代表，不要求该 GET 创建 identity。前缀 canary reload 后运行 `GENARRATIVE_PINGORA_CANARY_BASE_URL=http://127.0.0.1 GENARRATIVE_PINGORA_CANARY_HOST=<域名> npm run check:pingora-canary-live`；真实路径 canary reload 后运行 `node scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，随后用 `node scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js` 对账；真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致。canary live 的 base URL、prefix、Host、额外 path、`--timeout-ms` / `GENARRATIVE_PINGORA_CANARY_TIMEOUT_MS` 不能包含换行或 NUL，脚本会在发起请求前失败；canary access log 对账脚本的日志路径、prefix、必需路径和 `--since-lines` / `GENARRATIVE_PINGORA_CANARY_ACCESS_LOG_SINCE_LINES` 也不能包含换行或 NUL，日志行里解析出的 URI / path 含控制字符时必须失败；timeout 和 access log 对账的 `--since-lines` / 对应 env 必须是正整数，非法值直接失败，不静默回默认值。 +Pingora canary 入口默认由 Server-Provision 安装但不接入主站配置。`deploy/nginx/snippets/genarrative-pingora-canary.conf` 是前缀 canary，人工 include 后用 `/__genarrative_pingora_canary/` rewrite 到 Pingora shadow；`deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 是前缀 canary 之后、direct 直连之前的真实路径 canary，只能在 Nginx `http` 上下文 include，默认监听 `127.0.0.1:18083` 并写独立 `/var/log/nginx/genarrative-pingora-realpath-canary.access.log`，不要 include 到生产 `443` server 内覆盖正式 location。启用前必须运行 `npm run check:nginx-pingora-canary`；目标 agent 有 Nginx 时运行 `node scripts/check-nginx-pingora-canary.mjs --require-nginx`，强制把前缀 location 片段和真实路径 server 片段一起执行 `nginx -t`。本机或 CI 可用 `npm run check:pingora-canary-docker` 启动 Docker Nginx、真实 Pingora 和 mock 上游复现两条 Nginx -> Pingora handoff，并复用 `scripts/check-pingora-canary-access-log-parity.mjs` 按 `request_id` 对账。需要证明 Docker Nginx -> Pingora -> 真实本地服务时，先启动真实 api-server 与 SpacetimeDB，再运行 `node scripts/check-pingora-canary-docker.mjs --require-docker --real-upstreams --api-upstream 127.0.0.1: --spacetime-upstream 127.0.0.1: --web-root dist`；该模式不会启动内置 mock 上游，会先检查真实 `/healthz` 和 `/v1/ping`，随后复用同一组 live smoke 与 access log 对账。真实 SpacetimeDB 对 `GET /v1/identity` 返回 `405 Method Not Allowed` 属于可接受语义，canary live 只把它作为路径路由代表，不要求该 GET 创建 identity。前缀 canary reload 后运行 `GENARRATIVE_PINGORA_CANARY_BASE_URL=http://127.0.0.1 GENARRATIVE_PINGORA_CANARY_HOST=<域名> npm run check:pingora-canary-live`；真实路径 canary 启停统一使用 current release 随包脚本：`/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083` 写入 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf` 并执行 `nginx -t`、reload 和 realpath live smoke，`/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply` 删除该配置并复核 Nginx；两者失败都会恢复操作前状态。真实路径 canary reload 后运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，随后用 `node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js` 对账；真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致。canary live 的 base URL、prefix、Host、额外 path、`--timeout-ms` / `GENARRATIVE_PINGORA_CANARY_TIMEOUT_MS` 不能包含换行或 NUL，脚本会在发起请求前失败；canary access log 对账脚本的日志路径、prefix、必需路径和 `--since-lines` / `GENARRATIVE_PINGORA_CANARY_ACCESS_LOG_SINCE_LINES` 也不能包含换行或 NUL，日志行里解析出的 URI / path 含控制字符时必须失败；timeout 和 access log 对账的 `--since-lines` / 对应 env 必须是正整数，非法值直接失败，不静默回默认值。 Pingora 正式切换前必须额外运行聚合门禁：`node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。该门禁串起 Rust 单测、mock smoke、路由 parity、Nginx snippet 校验、Docker Nginx handoff、canary access log 对账烟测、直连入口静态预检、直连启用 / 回退 dry-run 行为检查、health patrol env 切换脚本烟测、current release 自审烟测、release readiness 计划自检、生产运维护栏、API release build 烟测、Pingora production release 真实构建烟测、API deploy release 烟测、目标机 live canary 和目标机真实 access log 对账；其中 `--require-live` 只能在目标 Nginx 已人工 include canary snippet 并 reload 后执行，并会强制要求 `--live-host`，避免只访问 `127.0.0.1` 命中 Nginx 默认 vhost。live smoke 成功后会立即读取 Nginx 与 Pingora access log 尾部记录，按 `request_id` 验证 `/__genarrative_pingora_canary/healthz` 和 `/__genarrative_pingora_canary/api/creation-entry/config` 已在两边落盘且 method/status/path 没有漂移。启用真实路径 canary 时追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`，它会用独立 Nginx access log 对账真实 `/api`、`/v1` 与 `/assets` 路径。live canary、direct live 和 access log 对账的显式毫秒超时或尾部行数参数都必须是正整数，非法值应先修参数再重跑，不能把默认值兜底后的结果当成切换证据；direct live 与 release readiness 读取的直连布尔 env 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，拼写错误会直接失败，避免 `REQUIRE_WSS_UPGRADE`、preflight 开关或 `SKIP_WSS` 被悄悄当成 false。若验证 Pingora 直连公网入口，还必须追加 direct preflight 参数检查目标机 env、systemd drop-in、service EnvironmentFile 一致性、当前用户证书权限、服务用户证书权限、service 二进制可执行性和端口释放：`--require-direct --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log --direct-health-patrol-env-file /etc/genarrative/health-patrol.env --direct-preflight-env-file /etc/genarrative/pingora-gateway.env --direct-preflight-systemd --direct-preflight-check-cert-readable --direct-preflight-check-service-env-file --direct-preflight-check-service-user-cert-readable --direct-preflight-check-service-binary-executable --direct-preflight-check-ports-free`；该模式缺少 `--direct-http-base-url`、`--direct-host`、`--direct-redirect-host`、`--direct-pingora-access-log`、`--direct-preflight-systemd`、`--direct-preflight-check-cert-readable`、`--direct-preflight-check-service-env-file`、`--direct-preflight-check-service-user-cert-readable`、`--direct-preflight-check-service-binary-executable`、`--direct-preflight-check-ports-free` 、缺少 `--direct-health-patrol-env-file` 或缺少 `--direct-spacetime-database` 会直接失败，且会拒绝 `--direct-skip-wss`，并默认要求 WSS subscribe 返回 101，证明 HTTP redirect / ACME、正式域名 Host/SNI、redirect Location host、Pingora access log request_id + path/status 对账、health patrol direct 模式、systemd drop-in、service EnvironmentFile 一致性、当前用户证书权限、服务用户证书权限、service 二进制可执行性、80/443 端口释放与目标 SpacetimeDB 长连接都能从直连入口透传。`check-pingora-release-readiness.mjs --help` 中的正式直连和只生成 runbook 示例也必须显式带 `--direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`，避免值班人员复制示例后被 `--require-direct` 自身拦住。普通本机提交前可先跑 `npm run check:pingora-release-readiness`，但该默认模式不能替代目标机切换窗口的强制门禁。直连切换前还应先运行 `npm run plan:pingora-direct-cutover -- --require-direct --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log --direct-health-patrol-env-file /etc/genarrative/health-patrol.env --direct-preflight-env-file /etc/genarrative/pingora-gateway.env --direct-preflight-systemd --direct-preflight-check-cert-readable --direct-preflight-check-service-env-file --direct-preflight-check-service-user-cert-readable --direct-preflight-check-service-binary-executable --direct-preflight-check-ports-free --rollback-nginx-smoke-url http://127.0.0.1/healthz --rollback-health-patrol-public-base-url <切换前Nginx巡检入口>` 生成只读 JSON runbook；若切换参数提供 `--direct-probe-token`，启用后复核会继续透传 direct probe token 检查内部探针，runbook JSON 只显示 ``。runbook 会列出 Host 与回退巡检入口确认、current release 自包含自审、current release preflight、启用前基础门禁、direct enable dry-run、direct enable apply、用 `/opt/genarrative/current/scripts/deploy/pingora-health-patrol-env-switch.mjs --apply` 切到 `pingora-direct`、启用后 health patrol env 直连复核、启用后 `--require-direct` 复核、rollback dry-run、回退前用同一脚本预置回 `nginx` 并恢复切换前 public base URL / Host、rollback apply、回退后 health patrol env Nginx 模式复核，供当班人员逐条审阅。direct enable / rollback `--apply` 都会拒绝符号链接形式的 systemd drop-in 目录或目标文件，并拒绝已存在但不是普通文件的目标，避免把低端口 capability 写入非预期位置，或在回退窗口误删 / 误判非预期 systemd 位置；health patrol env 切换脚本 `--apply` 会保留原文件权限和 owner/group，此时 `--env-file` 必须直接指向真实普通文件，不能是符号链接。如果现场 env 是链接，先确认真实目标路径后再传给脚本，避免替换链接本身或写入非预期目标。 diff --git a/jenkins/Jenkinsfile.production-api-build b/jenkins/Jenkinsfile.production-api-build index 919ffcb7a..ba7a9cf2e 100644 --- a/jenkins/Jenkinsfile.production-api-build +++ b/jenkins/Jenkinsfile.production-api-build @@ -123,7 +123,7 @@ pipeline { stage('Archive') { steps { - archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/api-server,build/${env.EFFECTIVE_BUILD_VERSION}/api-server.sha256,build/${env.EFFECTIVE_BUILD_VERSION}/release-manifest.json,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/systemd/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/nginx/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/pingora/**", fingerprint: true + archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/api-server,build/${env.EFFECTIVE_BUILD_VERSION}/api-server.sha256,build/${env.EFFECTIVE_BUILD_VERSION}/release-manifest.json,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/systemd/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/nginx/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/pingora/**", fingerprint: true script { if (params.INCLUDE_PINGORA_GATEWAY) { archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/pingora-gateway,build/${env.EFFECTIVE_BUILD_VERSION}/pingora-gateway.sha256", fingerprint: true diff --git a/jenkins/Jenkinsfile.production-api-deploy b/jenkins/Jenkinsfile.production-api-deploy index 657d5fefa..50bee6574 100644 --- a/jenkins/Jenkinsfile.production-api-deploy +++ b/jenkins/Jenkinsfile.production-api-deploy @@ -66,7 +66,7 @@ pipeline { copyArtifacts( projectName: params.BUILD_JOB_NAME, selector: specific(params.BUILD_NUMBER_TO_DEPLOY), - filter: "build/${params.BUILD_VERSION}/api-server,build/${params.BUILD_VERSION}/api-server.sha256,build/${params.BUILD_VERSION}/release-manifest.json,build/${params.BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${params.BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${params.BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${params.BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${params.BUILD_VERSION}/deploy/systemd/**,build/${params.BUILD_VERSION}/deploy/nginx/**,build/${params.BUILD_VERSION}/deploy/env/**,build/${params.BUILD_VERSION}/deploy/pingora/**", + filter: "build/${params.BUILD_VERSION}/api-server,build/${params.BUILD_VERSION}/api-server.sha256,build/${params.BUILD_VERSION}/release-manifest.json,build/${params.BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${params.BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${params.BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${params.BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${params.BUILD_VERSION}/deploy/systemd/**,build/${params.BUILD_VERSION}/deploy/nginx/**,build/${params.BUILD_VERSION}/deploy/env/**,build/${params.BUILD_VERSION}/deploy/pingora/**", target: '.', fingerprintArtifacts: true ) @@ -92,7 +92,7 @@ pipeline { sh ''' bash -lc ' set -euo pipefail - chmod +x "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-on.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-off.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs" + chmod +x "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-on.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-off.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs" "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" \ --source-dir "build/${BUILD_VERSION}" \ --version "${BUILD_VERSION}" \ diff --git a/package.json b/package.json index 8fe9d55e0..2e715da20 100644 --- a/package.json +++ b/package.json @@ -38,6 +38,7 @@ "check:pingora-route-parity": "node scripts/check-pingora-route-parity.mjs", "check:pingora-canary-live": "node scripts/check-pingora-canary-live.mjs", "check:pingora-canary-live-guard": "node scripts/check-pingora-canary-live-guard.mjs", + "check:pingora-realpath-canary-toggle": "node scripts/check-pingora-realpath-canary-toggle.mjs", "check:pingora-direct-preflight": "node -- scripts/check-pingora-direct-preflight.mjs && node scripts/check-pingora-direct-preflight-guard.mjs", "check:pingora-direct-live": "node scripts/check-pingora-direct-live.mjs", "check:pingora-direct-live-guard": "node scripts/check-pingora-direct-live-guard.mjs", @@ -57,6 +58,8 @@ "check:pingora-release-readiness-plan": "node scripts/check-pingora-release-readiness-plan.mjs", "check:pingora-release-readiness": "node scripts/check-pingora-release-readiness.mjs", "plan:pingora-direct-cutover": "node scripts/check-pingora-release-readiness.mjs --dry-run-cutover", + "deploy:pingora-realpath-canary-enable": "bash scripts/deploy/pingora-realpath-canary-enable.sh", + "deploy:pingora-realpath-canary-disable": "bash scripts/deploy/pingora-realpath-canary-disable.sh", "deploy:pingora-direct-enable": "bash scripts/deploy/pingora-direct-enable.sh", "deploy:pingora-direct-rollback": "bash scripts/deploy/pingora-direct-rollback.sh", "assets:child-motion-demo": "node scripts/generate-child-motion-demo-assets.mjs", diff --git a/scripts/build-production-release.sh b/scripts/build-production-release.sh index d31b556e0..b5a007db9 100644 --- a/scripts/build-production-release.sh +++ b/scripts/build-production-release.sh @@ -520,6 +520,8 @@ cp "${SCRIPT_DIR}/deploy/maintenance-on.sh" "${TARGET_DIR}/scripts/deploy/mainte cp "${SCRIPT_DIR}/deploy/maintenance-off.sh" "${TARGET_DIR}/scripts/deploy/maintenance-off.sh" cp "${SCRIPT_DIR}/deploy/pingora-direct-enable.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh" cp "${SCRIPT_DIR}/deploy/pingora-direct-rollback.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh" +cp "${SCRIPT_DIR}/deploy/pingora-realpath-canary-enable.sh" "${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" +cp "${SCRIPT_DIR}/deploy/pingora-realpath-canary-disable.sh" "${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" cp "${SCRIPT_DIR}/deploy/pingora-health-patrol-env-switch.mjs" "${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" cp "${SCRIPT_DIR}/deploy/pingora-tls-cert-sync.mjs" "${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" cp "${SCRIPT_DIR}/deploy/jenkins-inbound-agent-start.sh" "${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh" @@ -535,6 +537,8 @@ chmod +x \ "${TARGET_DIR}/scripts/deploy/maintenance-off.sh" \ "${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh" \ "${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh" \ + "${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" \ + "${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" \ "${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" \ "${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" \ "${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh" \ @@ -580,7 +584,7 @@ cat >"${TARGET_DIR}/README.md" < 0) { + console.error('[check:pingora-realpath-canary-toggle] FAILED'); + for (const failure of failures) { + console.error(`- ${failure}`); + } + process.exit(1); +} + +console.log('[check:pingora-realpath-canary-toggle] OK'); + +function main() { + assertScriptShape(); + assertDryRunDoesNotWrite(); + assertApplyWritesRenderedConfigAndRunsLive(); + assertEnableRollsBackWhenLiveFails(); + assertDisableRemovesConfig(); + assertDisableRollsBackWhenReloadFails(); + assertRejectsUnsafeInputs(); +} + +function assertScriptShape() { + for (const script of [ENABLE_SCRIPT, DISABLE_SCRIPT]) { + const content = readFileSync(script, 'utf8'); + assertIncludes(content, '--apply', `${script} 必须默认 dry-run 并要求显式 --apply。`); + assertIncludes(content, 'nginx -t', `${script} 帮助文案必须说明会先做 nginx -t。`); + assertIncludes( + content, + 'zz-genarrative-pingora-realpath-canary.conf', + `${script} 必须默认使用晚于主站配置加载的 conf.d 文件名。`, + ); + assertIncludes( + content, + '不能包含换行或 NUL 字符', + `${script} 必须拒绝控制字符参数。`, + ); + assertIncludes(content, '不能是文件系统根目录', `${script} 必须拒绝根路径。`); + } + + const enableContent = readFileSync(ENABLE_SCRIPT, 'utf8'); + assertIncludes( + enableContent, + 'check-pingora-canary-live.mjs', + 'enable 脚本必须默认运行 realpath live smoke。', + ); + assertIncludes( + enableContent, + '恢复写入前配置', + 'enable 脚本失败时必须恢复写入前配置。', + ); + + const disableContent = readFileSync(DISABLE_SCRIPT, 'utf8'); + assertIncludes( + disableContent, + '恢复删除前配置', + 'disable 脚本失败时必须恢复删除前配置。', + ); +} + +function assertDryRunDoesNotWrite() { + const fixture = createFixture('dry-run'); + const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf'); + + const result = runScript(ENABLE_SCRIPT, [ + '--probe-token', + 'dry-run-token', + '--host', + 'dev.genarrative.world', + '--template-path', + fixture.templatePath, + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--live-script', + fixture.liveScript, + ]); + + assertStatus(result, 0, 'enable dry-run 应成功。'); + if (existsSync(targetPath)) { + failures.push('enable dry-run 不应写入目标 Nginx 配置。'); + } + assertIncludes(result.stdout, 'dry-run', 'enable dry-run 应输出 dry-run 提示。'); +} + +function assertApplyWritesRenderedConfigAndRunsLive() { + const fixture = createFixture('apply-ok'); + const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf'); + const result = runScript(ENABLE_SCRIPT, [ + '--apply', + '--probe-token', + 'apply-real-token', + '--host', + 'dev.genarrative.world', + '--base-url', + 'http://127.0.0.1:18083', + '--template-path', + fixture.templatePath, + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--live-script', + fixture.liveScript, + '--no-status', + ]); + + assertStatus(result, 0, 'enable apply 应成功。'); + const rendered = readFileSync(targetPath, 'utf8'); + assertIncludes(rendered, '"apply-real-token"', 'enable apply 应替换 probe token。'); + if (rendered.includes('__GENARRATIVE_PINGORA_PROBE_TOKEN__')) { + failures.push('enable apply 后目标配置不应保留 probe token 占位符。'); + } + const calls = readFileSync(fixture.callsPath, 'utf8'); + assertIncludes(calls, 'nginx -t', 'enable apply 必须执行 nginx -t。'); + assertIncludes(calls, 'systemctl reload nginx.service', 'enable apply 必须 reload nginx。'); + assertIncludes( + calls, + 'live --realpath --base-url http://127.0.0.1:18083 --host dev.genarrative.world', + 'enable apply 必须执行 realpath live smoke。', + ); +} + +function assertEnableRollsBackWhenLiveFails() { + const fixture = createFixture('enable-live-fails', { liveExitCode: 7 }); + const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf'); + writeFileSync(targetPath, 'previous-config\n', 'utf8'); + + const result = runScript( + ENABLE_SCRIPT, + [ + '--apply', + '--probe-token', + 'rollback-token', + '--host', + 'dev.genarrative.world', + '--template-path', + fixture.templatePath, + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--live-script', + fixture.liveScript, + '--no-status', + ], + fixture.env, + ); + + if (result.status === 0) { + failures.push('realpath live smoke 失败时 enable apply 必须失败。'); + } + const restored = readFileSync(targetPath, 'utf8'); + if (restored !== 'previous-config\n') { + failures.push('realpath live smoke 失败时 enable apply 必须恢复旧配置。'); + } + assertIncludes( + result.stderr, + '恢复写入前配置', + 'enable live 失败时必须说明已恢复写入前配置。', + ); +} + +function assertDisableRemovesConfig() { + const fixture = createFixture('disable-ok'); + const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf'); + writeFileSync(targetPath, 'enabled-config\n', 'utf8'); + + const result = runScript( + DISABLE_SCRIPT, + [ + '--apply', + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--no-status', + ], + fixture.env, + ); + + assertStatus(result, 0, 'disable apply 应成功。'); + if (existsSync(targetPath)) { + failures.push('disable apply 成功后必须删除 realpath canary 配置。'); + } + const calls = readFileSync(fixture.callsPath, 'utf8'); + assertIncludes(calls, 'nginx -t', 'disable apply 必须执行 nginx -t。'); + assertIncludes(calls, 'systemctl reload nginx.service', 'disable apply 必须 reload nginx。'); +} + +function assertDisableRollsBackWhenReloadFails() { + const fixture = createFixture('disable-reload-fails', { reloadExitCode: 9 }); + const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf'); + writeFileSync(targetPath, 'enabled-config\n', 'utf8'); + + const result = runScript(DISABLE_SCRIPT, [ + '--apply', + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--no-status', + ]); + + if (result.status === 0) { + failures.push('reload 失败时 disable apply 必须失败。'); + } + const restored = readFileSync(targetPath, 'utf8'); + if (restored !== 'enabled-config\n') { + failures.push('reload 失败时 disable apply 必须恢复删除前配置。'); + } + assertIncludes( + result.stderr, + '恢复删除前配置', + 'disable reload 失败时必须说明已恢复删除前配置。', + ); +} + +function assertRejectsUnsafeInputs() { + const fixture = createFixture('unsafe-inputs'); + const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf'); + const cases = [ + { + name: 'missing token', + script: ENABLE_SCRIPT, + args: [ + '--apply', + '--host', + 'dev.genarrative.world', + '--template-path', + fixture.templatePath, + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--live-script', + fixture.liveScript, + ], + expected: '--apply 必须提供 --probe-token', + }, + { + name: 'url host', + script: ENABLE_SCRIPT, + args: [ + '--probe-token', + 'unsafe-token', + '--host', + 'https://dev.genarrative.world', + '--template-path', + fixture.templatePath, + '--target-path', + targetPath, + '--nginx-binary', + fixture.nginxBinary, + '--systemctl-binary', + fixture.systemctlBinary, + '--live-script', + fixture.liveScript, + ], + expected: '不能是 URL', + }, + { + name: 'relative target', + script: DISABLE_SCRIPT, + args: ['--target-path', 'relative.conf'], + expected: '--target-path 必须是绝对路径', + }, + { + name: 'root target', + script: DISABLE_SCRIPT, + args: ['--target-path', '/'], + expected: '--target-path 不能是文件系统根目录', + }, + ]; + + for (const item of cases) { + const result = runScript(item.script, item.args); + if (result.status === 0) { + failures.push(`${item.name}: 不安全参数必须失败。`); + } + assertIncludes( + `${result.stderr}\n${result.stdout}`, + item.expected, + `${item.name}: 应输出明确错误。`, + ); + } +} + +function createFixture(name, options = {}) { + const root = path.join(tmpRoot, name); + const nginxDir = path.join(root, 'nginx-conf'); + mkdirSync(nginxDir, { recursive: true }); + const callsPath = path.join(root, 'calls.log'); + const templatePath = path.join(root, 'genarrative-pingora-realpath-canary.conf'); + const nginxBinary = path.join(root, 'nginx'); + const systemctlBinary = path.join(root, 'systemctl'); + const liveScript = path.join(root, 'live.mjs'); + + writeFileSync( + templatePath, + [ + 'server {', + ' listen 127.0.0.1:18083;', + ' access_log /var/log/nginx/genarrative-pingora-realpath-canary.access.log genarrative_upstream;', + ' location = /__genarrative_pingora_realpath_canary/healthz {', + ' proxy_set_header X-Genarrative-Pingora-Probe "__GENARRATIVE_PINGORA_PROBE_TOKEN__";', + ' proxy_pass http://127.0.0.1:18081/__genarrative_pingora/healthz;', + ' }', + '}', + '', + ].join('\n'), + 'utf8', + ); + writeFileSync( + nginxBinary, + [ + '#!/usr/bin/env bash', + `echo "nginx $*" >> ${shellQuote(callsPath)}`, + 'if [[ "${GENARRATIVE_FAKE_NGINX_T_EXIT:-0}" != "0" ]]; then exit "${GENARRATIVE_FAKE_NGINX_T_EXIT}"; fi', + 'exit 0', + '', + ].join('\n'), + 'utf8', + ); + writeFileSync( + systemctlBinary, + [ + '#!/usr/bin/env bash', + `echo "systemctl $*" >> ${shellQuote(callsPath)}`, + `reload_exit="\${GENARRATIVE_FAKE_RELOAD_EXIT:-${Number(options.reloadExitCode || 0)}}"`, + 'if [[ "$1" == "reload" && "${reload_exit}" != "0" ]]; then exit "${reload_exit}"; fi', + 'exit 0', + '', + ].join('\n'), + 'utf8', + ); + writeFileSync( + liveScript, + [ + '#!/usr/bin/env node', + "import { appendFileSync } from 'node:fs';", + `appendFileSync(${JSON.stringify(callsPath)}, 'live ' + process.argv.slice(2).join(' ') + '\\n');`, + `process.exit(Number(process.env.GENARRATIVE_FAKE_LIVE_EXIT || ${Number(options.liveExitCode || 0)}));`, + '', + ].join('\n'), + 'utf8', + ); + chmodExecutable(nginxBinary); + chmodExecutable(systemctlBinary); + chmodExecutable(liveScript); + + return { + root, + nginxDir, + callsPath, + templatePath, + nginxBinary, + systemctlBinary, + liveScript, + env: { + GENARRATIVE_FAKE_RELOAD_EXIT: String(options.reloadExitCode || 0), + GENARRATIVE_FAKE_LIVE_EXIT: String(options.liveExitCode || 0), + }, + }; +} + +function runScript(script, args, env = {}) { + return spawnSync('bash', [script, ...args], { + cwd: process.cwd(), + env: { ...process.env, ...env }, + encoding: 'utf8', + }); +} + +function chmodExecutable(file) { + chmodSync(file, 0o755); +} + +function shellQuote(value) { + return `'${String(value).replaceAll("'", "'\\''")}'`; +} + +function assertStatus(result, expected, message) { + if (result.status !== expected) { + failures.push( + `${message} 实际退出码 ${result.status}。\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`, + ); + } +} + +function assertIncludes(content, needle, message) { + if (!content.includes(needle)) { + failures.push(`${message} 缺少 ${needle}`); + } +} diff --git a/scripts/check-production-api-deploy.mjs b/scripts/check-production-api-deploy.mjs index 8920d7035..2fb6ea2e5 100644 --- a/scripts/check-production-api-deploy.mjs +++ b/scripts/check-production-api-deploy.mjs @@ -61,6 +61,8 @@ function main() { assertMissingPingoraCutoverEvidenceAuditFails(); assertMissingHealthPatrolEnvCheckFails(); assertMissingPingoraReleaseReadinessFails(); + assertMissingPingoraRealpathCanaryEnableFails(); + assertMissingPingoraRealpathCanaryDisableFails(); assertMissingPingoraHealthPatrolEnvSwitchFails(); assertMissingPingoraTlsCertSyncFails(); assertMissingEnvExamplesFails(); @@ -99,6 +101,14 @@ function assertDeployCopiesPingoraDirectReleaseDependencies() { path.join(releaseDir, 'scripts/deploy/pingora-direct-rollback.sh'), 'current release 必须包含 Pingora 直连回退脚本。', ); + assertFileExists( + path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'), + 'current release 必须包含 Pingora realpath canary 启用脚本。', + ); + assertFileExists( + path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'), + 'current release 必须包含 Pingora realpath canary 关闭脚本。', + ); assertFileExists( path.join( releaseDir, @@ -890,6 +900,52 @@ function assertMissingPingoraHealthPatrolEnvSwitchFails() { } } +function assertMissingPingoraRealpathCanaryEnableFails() { + const fixture = prepareFixture('missing-pingora-realpath-canary-enable'); + rmSync( + path.join( + fixture.sourceDir, + 'scripts/deploy/pingora-realpath-canary-enable.sh', + ), + ); + const result = runDeploy(fixture); + + if (result.status === 0) { + failures.push('发布产物缺少 Pingora realpath canary 启用脚本时部署必须失败。'); + } + assertIncludes( + result.stderr, + '发布产物缺少 Pingora realpath canary 启用脚本', + '缺少 Pingora realpath canary 启用脚本时必须给出明确错误。', + ); + if (!existsSync(fixture.maintenanceFile)) { + failures.push('部署失败时必须保持维护模式。'); + } +} + +function assertMissingPingoraRealpathCanaryDisableFails() { + const fixture = prepareFixture('missing-pingora-realpath-canary-disable'); + rmSync( + path.join( + fixture.sourceDir, + 'scripts/deploy/pingora-realpath-canary-disable.sh', + ), + ); + const result = runDeploy(fixture); + + if (result.status === 0) { + failures.push('发布产物缺少 Pingora realpath canary 关闭脚本时部署必须失败。'); + } + assertIncludes( + result.stderr, + '发布产物缺少 Pingora realpath canary 关闭脚本', + '缺少 Pingora realpath canary 关闭脚本时必须给出明确错误。', + ); + if (!existsSync(fixture.maintenanceFile)) { + failures.push('部署失败时必须保持维护模式。'); + } +} + function assertMissingPingoraTlsCertSyncFails() { const fixture = prepareFixture('missing-pingora-tls-cert-sync'); rmSync( @@ -1044,6 +1100,14 @@ function prepareFixture(name) { 'scripts/deploy/pingora-direct-rollback.sh', path.join(sourceDir, 'scripts/deploy/pingora-direct-rollback.sh'), ); + copyFile( + 'scripts/deploy/pingora-realpath-canary-enable.sh', + path.join(sourceDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'), + ); + copyFile( + 'scripts/deploy/pingora-realpath-canary-disable.sh', + path.join(sourceDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'), + ); copyFile( 'scripts/deploy/pingora-health-patrol-env-switch.mjs', path.join(sourceDir, 'scripts/deploy/pingora-health-patrol-env-switch.mjs'), diff --git a/scripts/check-production-api-release.mjs b/scripts/check-production-api-release.mjs index 151ccd3f3..a0a3dc73a 100644 --- a/scripts/check-production-api-release.mjs +++ b/scripts/check-production-api-release.mjs @@ -102,6 +102,22 @@ function assertApiReleaseContainsPingoraDirectDependencies() { path.join(releaseDir, 'scripts/deploy/pingora-direct-rollback.sh'), 'Pingora 直连回退脚本必须可执行。', ); + assertFileExists( + path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'), + 'API release 必须包含 Pingora realpath canary 启用脚本。', + ); + assertExecutable( + path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'), + 'Pingora realpath canary 启用脚本必须可执行。', + ); + assertFileExists( + path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'), + 'API release 必须包含 Pingora realpath canary 关闭脚本。', + ); + assertExecutable( + path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'), + 'Pingora realpath canary 关闭脚本必须可执行。', + ); assertFileExists( path.join( releaseDir, @@ -254,9 +270,19 @@ function assertApiReleaseContainsPingoraDirectDependencies() { ); assertIncludes( releaseReadme, - 'Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本', + 'Pingora release readiness 聚合门禁、直连启用 / 回退', 'API release README 必须说明随包携带 Pingora 直连脚本。', ); + assertIncludes( + releaseReadme, + 'realpath canary 启用 / 关闭', + 'API release README 必须说明随包携带 Pingora realpath canary 启停脚本。', + ); + assertIncludes( + releaseReadme, + 'health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审', + 'API release README 必须说明随包携带 Pingora 运行态复核脚本。', + ); assertIncludes( releaseReadme, '状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本', diff --git a/scripts/check-production-ops-guardrails.mjs b/scripts/check-production-ops-guardrails.mjs index 0206acc8b..c7dc68928 100644 --- a/scripts/check-production-ops-guardrails.mjs +++ b/scripts/check-production-ops-guardrails.mjs @@ -71,6 +71,16 @@ const checks = [ includes: 'scripts/check-pingora-canary-live.mjs', reason: 'Pingora current release 自审必须确认 canary live smoke 脚本已随包发布。', }, + { + file: 'scripts/ops/pingora-current-release-audit.mjs', + includes: 'scripts/deploy/pingora-realpath-canary-enable.sh', + reason: 'Pingora current release 自审必须确认 realpath canary 启用脚本已随包发布。', + }, + { + file: 'scripts/ops/pingora-current-release-audit.mjs', + includes: 'scripts/deploy/pingora-realpath-canary-disable.sh', + reason: 'Pingora current release 自审必须确认 realpath canary 关闭脚本已随包发布。', + }, { file: 'scripts/ops/pingora-current-release-audit.mjs', includes: '--systemd-show', @@ -1013,6 +1023,16 @@ const checks = [ includes: 'scripts/deploy/pingora-direct-rollback.sh', reason: '生产 API release 必须携带 Pingora 直连回退脚本。', }, + { + file: 'scripts/build-production-release.sh', + includes: 'scripts/deploy/pingora-realpath-canary-enable.sh', + reason: '生产 API release 必须携带 Pingora realpath canary 启用脚本。', + }, + { + file: 'scripts/build-production-release.sh', + includes: 'scripts/deploy/pingora-realpath-canary-disable.sh', + reason: '生产 API release 必须携带 Pingora realpath canary 关闭脚本。', + }, { file: 'scripts/build-production-release.sh', includes: 'scripts/deploy/pingora-health-patrol-env-switch.mjs', @@ -1069,6 +1089,26 @@ const checks = [ includes: 'PINGORA_CANARY_LIVE_SCRIPT_SOURCE', reason: 'API deploy 必须从发布产物复制 Pingora canary live smoke 脚本到 current release。', }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: 'PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE', + reason: 'API deploy 必须从发布产物复制 Pingora realpath canary 启用脚本到 current release。', + }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: 'PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE', + reason: 'API deploy 必须从发布产物复制 Pingora realpath canary 关闭脚本到 current release。', + }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: '发布产物缺少 Pingora realpath canary 启用脚本', + reason: 'API deploy 缺少 realpath canary 启用脚本时必须 fail-fast。', + }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: '发布产物缺少 Pingora realpath canary 关闭脚本', + reason: 'API deploy 缺少 realpath canary 关闭脚本时必须 fail-fast。', + }, { file: 'scripts/deploy/production-api-deploy.sh', includes: 'PINGORA_CURRENT_RELEASE_AUDIT_SCRIPT_SOURCE', @@ -1119,6 +1159,16 @@ const checks = [ includes: 'scripts/check-production-health-patrol-env.mjs', reason: 'API release 动态烟测必须断言健康巡检 env 复核脚本随包发布。', }, + { + file: 'scripts/check-production-api-release.mjs', + includes: 'scripts/deploy/pingora-realpath-canary-enable.sh', + reason: 'API release 动态烟测必须断言 Pingora realpath canary 启用脚本随包发布。', + }, + { + file: 'scripts/check-production-api-release.mjs', + includes: 'scripts/deploy/pingora-realpath-canary-disable.sh', + reason: 'API release 动态烟测必须断言 Pingora realpath canary 关闭脚本随包发布。', + }, { file: 'scripts/check-production-api-deploy.mjs', includes: 'deploy/env/health-patrol.env.example', @@ -1149,6 +1199,30 @@ const checks = [ reason: 'API deploy 动态烟测必须断言 Pingora 切换命令证据脚本复制到 current release。', }, + { + file: 'scripts/check-production-api-deploy.mjs', + includes: 'scripts/deploy/pingora-realpath-canary-enable.sh', + reason: + 'API deploy 动态烟测必须断言 Pingora realpath canary 启用脚本复制到 current release。', + }, + { + file: 'scripts/check-production-api-deploy.mjs', + includes: 'scripts/deploy/pingora-realpath-canary-disable.sh', + reason: + 'API deploy 动态烟测必须断言 Pingora realpath canary 关闭脚本复制到 current release。', + }, + { + file: 'scripts/check-production-api-deploy.mjs', + includes: 'assertMissingPingoraRealpathCanaryEnableFails', + reason: + 'API deploy 动态烟测必须覆盖缺少 Pingora realpath canary 启用脚本时 fail-fast。', + }, + { + file: 'scripts/check-production-api-deploy.mjs', + includes: 'assertMissingPingoraRealpathCanaryDisableFails', + reason: + 'API deploy 动态烟测必须覆盖缺少 Pingora realpath canary 关闭脚本时 fail-fast。', + }, { file: 'scripts/check-production-api-deploy.mjs', includes: 'assertMissingPingoraCutoverCommandEvidenceFails', @@ -1171,6 +1245,21 @@ const checks = [ includes: 'deploy:pingora-direct-enable', reason: 'Pingora 直连启用必须保留可重复执行的 npm 入口。', }, + { + file: 'package.json', + includes: 'check:pingora-realpath-canary-toggle', + reason: 'Pingora realpath canary 启停脚本必须有本地自测入口。', + }, + { + file: 'package.json', + includes: 'deploy:pingora-realpath-canary-enable', + reason: 'Pingora realpath canary 启用必须保留可重复执行的 npm 入口。', + }, + { + file: 'package.json', + includes: 'deploy:pingora-realpath-canary-disable', + reason: 'Pingora realpath canary 关闭必须保留可重复执行的 npm 入口。', + }, { file: 'package.json', includes: 'deploy:pingora-direct-rollback', @@ -5545,6 +5634,18 @@ const checks = [ 'build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs', reason: 'API Build 归档必须包含 Pingora health patrol env 切换脚本。', }, + { + file: 'jenkins/Jenkinsfile.production-api-build', + includes: + 'build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh', + reason: 'API Build 归档必须包含 Pingora realpath canary 启用脚本。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-build', + includes: + 'build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh', + reason: 'API Build 归档必须包含 Pingora realpath canary 关闭脚本。', + }, { file: 'jenkins/Jenkinsfile.production-api-build', includes: 'build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**', @@ -5640,6 +5741,18 @@ const checks = [ 'build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs', reason: 'API Deploy 复制上游产物时必须包含 Pingora canary access log 对账脚本。', }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + 'build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh', + reason: 'API Deploy 复制上游产物时必须包含 Pingora realpath canary 启用脚本。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + 'build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh', + reason: 'API Deploy 复制上游产物时必须包含 Pingora realpath canary 关闭脚本。', + }, { file: 'jenkins/Jenkinsfile.production-api-deploy', includes: diff --git a/scripts/deploy/pingora-realpath-canary-disable.sh b/scripts/deploy/pingora-realpath-canary-disable.sh new file mode 100644 index 000000000..379048b8d --- /dev/null +++ b/scripts/deploy/pingora-realpath-canary-disable.sh @@ -0,0 +1,197 @@ +#!/usr/bin/env bash + +set -euo pipefail + +TARGET_PATH="${GENARRATIVE_PINGORA_REALPATH_CANARY_TARGET_PATH:-/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf}" +NGINX_BINARY="${GENARRATIVE_NGINX_BINARY:-nginx}" +NGINX_SERVICE="${GENARRATIVE_NGINX_SERVICE:-nginx.service}" +SYSTEMCTL_BINARY="${GENARRATIVE_SYSTEMCTL_BINARY:-systemctl}" +APPLY="false" +STATUS_AFTER="true" + +usage() { + cat <<'EOF' +用法: + scripts/deploy/pingora-realpath-canary-disable.sh [--apply] [--target-path ] [--nginx-binary ] [--nginx-service ] [--systemctl-binary ] [--no-status] + +说明: + 关闭 Nginx -> Pingora 真实路径 canary：删除 + /etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf，执行 nginx -t， + 再 reload nginx。默认 dry-run，必须显式传 --apply 才会修改系统。 + 如果 nginx -t 或 reload 失败，脚本会恢复删除前的目标文件。 +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) + usage + exit 0 + ;; + --apply) + APPLY="true" + shift + ;; + --target-path) + TARGET_PATH="${2:-}" + if [[ -z "${TARGET_PATH}" ]]; then + echo "[pingora-realpath-canary-disable] --target-path 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --nginx-binary) + NGINX_BINARY="${2:-}" + if [[ -z "${NGINX_BINARY}" ]]; then + echo "[pingora-realpath-canary-disable] --nginx-binary 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --nginx-service) + NGINX_SERVICE="${2:-}" + if [[ -z "${NGINX_SERVICE}" ]]; then + echo "[pingora-realpath-canary-disable] --nginx-service 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --systemctl-binary) + SYSTEMCTL_BINARY="${2:-}" + if [[ -z "${SYSTEMCTL_BINARY}" ]]; then + echo "[pingora-realpath-canary-disable] --systemctl-binary 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --no-status) + STATUS_AFTER="false" + shift + ;; + *) + echo "[pingora-realpath-canary-disable] 未知参数: $1" >&2 + usage >&2 + exit 1 + ;; + esac +done + +reject_control_characters() { + local label="$1" + local value="$2" + if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then + echo "[pingora-realpath-canary-disable] ${label} 不能包含换行或 NUL 字符。" >&2 + exit 1 + fi +} + +is_filesystem_root_path() { + local value="$1" + local without_slashes="${value//\//}" + [[ -n "${value}" && -z "${without_slashes}" ]] +} + +require_absolute_path() { + local label="$1" + local value="$2" + if [[ "${value}" != /* ]]; then + echo "[pingora-realpath-canary-disable] ${label} 必须是绝对路径。" >&2 + exit 1 + fi + if is_filesystem_root_path "${value}"; then + echo "[pingora-realpath-canary-disable] ${label} 不能是文件系统根目录。" >&2 + exit 1 + fi +} + +validate_command_or_absolute_path() { + local label="$1" + local value="$2" + if [[ "${value}" == */* ]]; then + require_absolute_path "${label}" "${value}" + fi +} + +print_command() { + printf '+' + for arg in "$@"; do + printf ' %q' "${arg}" + done + printf '\n' +} + +run_or_print() { + if [[ "${APPLY}" == "true" ]]; then + "$@" + else + print_command "$@" + fi +} + +restore_previous() { + local backup_path="$1" + cp -p -- "${backup_path}" "${TARGET_PATH}" +} + +rollback_after_failure() { + local backup_path="$1" + local reason="$2" + echo "[pingora-realpath-canary-disable] ${reason}，恢复删除前配置。" >&2 + restore_previous "${backup_path}" + "${NGINX_BINARY}" -t >/dev/null 2>&1 || true + "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}" >/dev/null 2>&1 || true +} + +reject_control_characters "--target-path" "${TARGET_PATH}" +reject_control_characters "--nginx-binary" "${NGINX_BINARY}" +reject_control_characters "--nginx-service" "${NGINX_SERVICE}" +reject_control_characters "--systemctl-binary" "${SYSTEMCTL_BINARY}" +require_absolute_path "--target-path" "${TARGET_PATH}" +validate_command_or_absolute_path "--nginx-binary" "${NGINX_BINARY}" +validate_command_or_absolute_path "--systemctl-binary" "${SYSTEMCTL_BINARY}" + +TARGET_DIR="$(dirname -- "${TARGET_PATH}")" +if [[ "${APPLY}" == "true" && ( -L "${TARGET_DIR}" || -L "${TARGET_PATH}" ) ]]; then + echo "[pingora-realpath-canary-disable] 目标目录或目标文件不能是符号链接。" >&2 + exit 1 +fi + +if [[ "${APPLY}" != "true" ]]; then + echo "[pingora-realpath-canary-disable] dry-run：将删除 realpath canary ${TARGET_PATH}。" + print_command rm -f -- "${TARGET_PATH}" + run_or_print "${NGINX_BINARY}" -t + run_or_print "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}" + exit 0 +fi + +if [[ ! -e "${TARGET_PATH}" ]]; then + echo "[pingora-realpath-canary-disable] 目标文件不存在，视为已关闭: ${TARGET_PATH}" + if [[ "${STATUS_AFTER}" == "true" ]]; then + "${SYSTEMCTL_BINARY}" is-active "${NGINX_SERVICE}" + fi + exit 0 +fi +if [[ ! -f "${TARGET_PATH}" ]]; then + echo "[pingora-realpath-canary-disable] 目标存在但不是普通文件: ${TARGET_PATH}" >&2 + exit 1 +fi + +backup_path="$(mktemp "${TARGET_DIR}/.genarrative-pingora-realpath-canary.disable.backup.XXXXXX")" +trap 'rm -f -- "${backup_path}"' EXIT +cp -p -- "${TARGET_PATH}" "${backup_path}" +rm -f -- "${TARGET_PATH}" + +if ! "${NGINX_BINARY}" -t; then + rollback_after_failure "${backup_path}" "nginx -t 失败" + exit 1 +fi +if ! "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}"; then + rollback_after_failure "${backup_path}" "reload ${NGINX_SERVICE} 失败" + exit 1 +fi + +if [[ "${STATUS_AFTER}" == "true" ]]; then + "${SYSTEMCTL_BINARY}" is-active "${NGINX_SERVICE}" +fi + +echo "[pingora-realpath-canary-disable] 已关闭 realpath canary: ${TARGET_PATH}" diff --git a/scripts/deploy/pingora-realpath-canary-enable.sh b/scripts/deploy/pingora-realpath-canary-enable.sh new file mode 100644 index 000000000..dd22528f4 --- /dev/null +++ b/scripts/deploy/pingora-realpath-canary-enable.sh @@ -0,0 +1,350 @@ +#!/usr/bin/env bash + +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)" +REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." >/dev/null 2>&1 && pwd)" + +TEMPLATE_PATH="${GENARRATIVE_PINGORA_REALPATH_CANARY_TEMPLATE_PATH:-${REPO_ROOT}/deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf}" +TARGET_PATH="${GENARRATIVE_PINGORA_REALPATH_CANARY_TARGET_PATH:-/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf}" +PROBE_TOKEN="${GENARRATIVE_PINGORA_REALPATH_CANARY_PROBE_TOKEN:-${GENARRATIVE_PINGORA_GATEWAY_PROBE_TOKEN:-}}" +BASE_URL="${GENARRATIVE_PINGORA_REALPATH_CANARY_BASE_URL:-http://127.0.0.1:18083}" +HOST_HEADER="${GENARRATIVE_PINGORA_REALPATH_CANARY_HOST:-}" +NGINX_BINARY="${GENARRATIVE_NGINX_BINARY:-nginx}" +NGINX_SERVICE="${GENARRATIVE_NGINX_SERVICE:-nginx.service}" +SYSTEMCTL_BINARY="${GENARRATIVE_SYSTEMCTL_BINARY:-systemctl}" +LIVE_SCRIPT="${GENARRATIVE_PINGORA_REALPATH_CANARY_LIVE_SCRIPT:-${REPO_ROOT}/scripts/check-pingora-canary-live.mjs}" +APPLY="false" +RUN_LIVE="true" +STATUS_AFTER="true" + +PLACEHOLDER="__GENARRATIVE_PINGORA_PROBE_TOKEN__" + +usage() { + cat <<'EOF' +用法: + scripts/deploy/pingora-realpath-canary-enable.sh [--apply] [--probe-token ] [--host ] [--base-url ] [--template-path ] [--target-path ] [--nginx-binary ] [--nginx-service ] [--systemctl-binary ] [--live-script ] [--skip-live] [--no-status] + +说明: + 启用 Nginx -> Pingora 真实路径 canary。脚本会把随包 + deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf 渲染到 + /etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf。 + +默认是 dry-run，只打印计划；必须显式传 --apply 才会写 Nginx 配置。 +--apply 会替换 probe token、写入目标文件、执行 nginx -t、reload nginx， +然后运行 realpath canary live smoke。任一阶段失败都会恢复写入前的目标文件状态。 +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) + usage + exit 0 + ;; + --apply) + APPLY="true" + shift + ;; + --probe-token) + PROBE_TOKEN="${2:-}" + if [[ -z "${PROBE_TOKEN}" ]]; then + echo "[pingora-realpath-canary-enable] --probe-token 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --host) + HOST_HEADER="${2:-}" + if [[ -z "${HOST_HEADER}" ]]; then + echo "[pingora-realpath-canary-enable] --host 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --base-url) + BASE_URL="${2:-}" + if [[ -z "${BASE_URL}" ]]; then + echo "[pingora-realpath-canary-enable] --base-url 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --template-path) + TEMPLATE_PATH="${2:-}" + if [[ -z "${TEMPLATE_PATH}" ]]; then + echo "[pingora-realpath-canary-enable] --template-path 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --target-path) + TARGET_PATH="${2:-}" + if [[ -z "${TARGET_PATH}" ]]; then + echo "[pingora-realpath-canary-enable] --target-path 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --nginx-binary) + NGINX_BINARY="${2:-}" + if [[ -z "${NGINX_BINARY}" ]]; then + echo "[pingora-realpath-canary-enable] --nginx-binary 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --nginx-service) + NGINX_SERVICE="${2:-}" + if [[ -z "${NGINX_SERVICE}" ]]; then + echo "[pingora-realpath-canary-enable] --nginx-service 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --systemctl-binary) + SYSTEMCTL_BINARY="${2:-}" + if [[ -z "${SYSTEMCTL_BINARY}" ]]; then + echo "[pingora-realpath-canary-enable] --systemctl-binary 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --live-script) + LIVE_SCRIPT="${2:-}" + if [[ -z "${LIVE_SCRIPT}" ]]; then + echo "[pingora-realpath-canary-enable] --live-script 缺少参数" >&2 + exit 1 + fi + shift 2 + ;; + --skip-live) + RUN_LIVE="false" + shift + ;; + --no-status) + STATUS_AFTER="false" + shift + ;; + *) + echo "[pingora-realpath-canary-enable] 未知参数: $1" >&2 + usage >&2 + exit 1 + ;; + esac +done + +reject_control_characters() { + local label="$1" + local value="$2" + if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then + echo "[pingora-realpath-canary-enable] ${label} 不能包含换行或 NUL 字符。" >&2 + exit 1 + fi +} + +is_filesystem_root_path() { + local value="$1" + local without_slashes="${value//\//}" + [[ -n "${value}" && -z "${without_slashes}" ]] +} + +require_absolute_path() { + local label="$1" + local value="$2" + if [[ "${value}" != /* ]]; then + echo "[pingora-realpath-canary-enable] ${label} 必须是绝对路径。" >&2 + exit 1 + fi + if is_filesystem_root_path "${value}"; then + echo "[pingora-realpath-canary-enable] ${label} 不能是文件系统根目录。" >&2 + exit 1 + fi +} + +validate_command_or_absolute_path() { + local label="$1" + local value="$2" + if [[ "${value}" == */* ]]; then + require_absolute_path "${label}" "${value}" + fi +} + +validate_host() { + local label="$1" + local value="$2" + if [[ -z "${value}" ]]; then + echo "[pingora-realpath-canary-enable] ${label} 必须提供。" >&2 + exit 1 + fi + if [[ "${value}" == http://* || "${value}" == https://* || "${value}" == */* || "${value}" == *\?* || "${value}" =~ [[:space:]] ]]; then + echo "[pingora-realpath-canary-enable] ${label} 必须是 Host 或 host:port，不能是 URL、路径、查询或包含空白。" >&2 + exit 1 + fi +} + +validate_base_url() { + if [[ "${BASE_URL}" != http://* && "${BASE_URL}" != https://* ]]; then + echo "[pingora-realpath-canary-enable] --base-url 必须以 http:// 或 https:// 开头。" >&2 + exit 1 + fi +} + +validate_probe_token() { + if [[ "${APPLY}" != "true" ]]; then + return + fi + if [[ -z "${PROBE_TOKEN}" ]]; then + echo "[pingora-realpath-canary-enable] --apply 必须提供 --probe-token 或 GENARRATIVE_PINGORA_REALPATH_CANARY_PROBE_TOKEN。" >&2 + exit 1 + fi + if [[ "${PROBE_TOKEN}" == "${PLACEHOLDER}" || "${#PROBE_TOKEN}" -lt 8 ]]; then + echo "[pingora-realpath-canary-enable] probe token 不能是占位值或过短。" >&2 + exit 1 + fi +} + +escape_nginx_double_quoted() { + local value="$1" + value="${value//\\/\\\\}" + value="${value//\"/\\\"}" + printf '%s' "${value}" +} + +render_template_to() { + local output_path="$1" + local content escaped_token + content="$(<"${TEMPLATE_PATH}")" + escaped_token="$(escape_nginx_double_quoted "${PROBE_TOKEN}")" + printf '%s' "${content//${PLACEHOLDER}/${escaped_token}}" >"${output_path}" +} + +print_command() { + printf '+' + for arg in "$@"; do + printf ' %q' "${arg}" + done + printf '\n' +} + +run_or_print() { + if [[ "${APPLY}" == "true" ]]; then + "$@" + else + print_command "$@" + fi +} + +restore_previous() { + local backup_path="$1" + local had_target="$2" + if [[ "${had_target}" == "true" ]]; then + cp -p -- "${backup_path}" "${TARGET_PATH}" + else + rm -f -- "${TARGET_PATH}" + fi +} + +rollback_after_failure() { + local backup_path="$1" + local had_target="$2" + local reason="$3" + echo "[pingora-realpath-canary-enable] ${reason}，恢复写入前配置。" >&2 + restore_previous "${backup_path}" "${had_target}" + "${NGINX_BINARY}" -t >/dev/null 2>&1 || true + "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}" >/dev/null 2>&1 || true +} + +reject_control_characters "--template-path" "${TEMPLATE_PATH}" +reject_control_characters "--target-path" "${TARGET_PATH}" +reject_control_characters "--probe-token" "${PROBE_TOKEN}" +reject_control_characters "--base-url" "${BASE_URL}" +reject_control_characters "--host" "${HOST_HEADER}" +reject_control_characters "--nginx-binary" "${NGINX_BINARY}" +reject_control_characters "--nginx-service" "${NGINX_SERVICE}" +reject_control_characters "--systemctl-binary" "${SYSTEMCTL_BINARY}" +reject_control_characters "--live-script" "${LIVE_SCRIPT}" + +require_absolute_path "--template-path" "${TEMPLATE_PATH}" +require_absolute_path "--target-path" "${TARGET_PATH}" +if [[ "${RUN_LIVE}" == "true" ]]; then + require_absolute_path "--live-script" "${LIVE_SCRIPT}" +fi +validate_command_or_absolute_path "--nginx-binary" "${NGINX_BINARY}" +validate_command_or_absolute_path "--systemctl-binary" "${SYSTEMCTL_BINARY}" +validate_host "--host" "${HOST_HEADER}" +validate_base_url +validate_probe_token + +if [[ ! -f "${TEMPLATE_PATH}" ]]; then + echo "[pingora-realpath-canary-enable] 未找到 realpath canary 模板: ${TEMPLATE_PATH}" >&2 + exit 1 +fi +if [[ "${RUN_LIVE}" == "true" && ! -f "${LIVE_SCRIPT}" ]]; then + echo "[pingora-realpath-canary-enable] 未找到 realpath canary live smoke 脚本: ${LIVE_SCRIPT}" >&2 + exit 1 +fi + +TARGET_DIR="$(dirname -- "${TARGET_PATH}")" +if [[ "${APPLY}" == "true" ]]; then + if [[ ! -d "${TARGET_DIR}" ]]; then + echo "[pingora-realpath-canary-enable] 目标目录不存在: ${TARGET_DIR}" >&2 + exit 1 + fi + if [[ -L "${TARGET_DIR}" || -L "${TARGET_PATH}" ]]; then + echo "[pingora-realpath-canary-enable] 目标目录或目标文件不能是符号链接。" >&2 + exit 1 + fi +fi + +if [[ "${APPLY}" != "true" ]]; then + echo "[pingora-realpath-canary-enable] dry-run：将渲染 realpath canary 到 ${TARGET_PATH}，probe token 输出已隐藏。" + print_command install -m 0644 "" "${TARGET_PATH}" + run_or_print "${NGINX_BINARY}" -t + run_or_print "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}" + if [[ "${RUN_LIVE}" == "true" ]]; then + run_or_print node -- "${LIVE_SCRIPT}" --realpath --base-url "${BASE_URL}" --host "${HOST_HEADER}" + fi + exit 0 +fi + +tmp_path="$(mktemp "${TARGET_DIR}/.genarrative-pingora-realpath-canary.XXXXXX")" +backup_path="$(mktemp "${TARGET_DIR}/.genarrative-pingora-realpath-canary.backup.XXXXXX")" +had_target="false" +cleanup_paths=("${tmp_path}" "${backup_path}") +trap 'rm -f -- "${cleanup_paths[@]}"' EXIT + +render_template_to "${tmp_path}" +chmod 0644 "${tmp_path}" + +if [[ -e "${TARGET_PATH}" ]]; then + if [[ ! -f "${TARGET_PATH}" ]]; then + echo "[pingora-realpath-canary-enable] 目标已存在但不是普通文件: ${TARGET_PATH}" >&2 + exit 1 + fi + cp -p -- "${TARGET_PATH}" "${backup_path}" + had_target="true" +fi + +install -m 0644 "${tmp_path}" "${TARGET_PATH}" +if ! "${NGINX_BINARY}" -t; then + rollback_after_failure "${backup_path}" "${had_target}" "nginx -t 失败" + exit 1 +fi +if ! "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}"; then + rollback_after_failure "${backup_path}" "${had_target}" "reload ${NGINX_SERVICE} 失败" + exit 1 +fi + +if [[ "${RUN_LIVE}" == "true" ]]; then + if ! node -- "${LIVE_SCRIPT}" --realpath --base-url "${BASE_URL}" --host "${HOST_HEADER}"; then + rollback_after_failure "${backup_path}" "${had_target}" "realpath canary live smoke 失败" + exit 1 + fi +fi + +if [[ "${STATUS_AFTER}" == "true" ]]; then + "${SYSTEMCTL_BINARY}" is-active "${NGINX_SERVICE}" +fi + +echo "[pingora-realpath-canary-enable] 已启用 realpath canary: ${TARGET_PATH}" diff --git a/scripts/deploy/production-api-deploy.sh b/scripts/deploy/production-api-deploy.sh index 142d32e45..e3ee985f2 100644 --- a/scripts/deploy/production-api-deploy.sh +++ b/scripts/deploy/production-api-deploy.sh @@ -619,6 +619,8 @@ HEALTH_PATROL_ENV_CHECK_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-production-he PINGORA_RELEASE_READINESS_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-release-readiness.mjs" PINGORA_ENABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-enable.sh" PINGORA_ROLLBACK_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-rollback.sh" +PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" +PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" PINGORA_PREFLIGHT_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-direct-preflight.mjs" @@ -702,6 +704,18 @@ if [[ ! -f "${PINGORA_ROLLBACK_SCRIPT_SOURCE}" ]]; then fi cp "${PINGORA_ROLLBACK_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-direct-rollback.sh" chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-direct-rollback.sh" +if [[ ! -f "${PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE}" ]]; then + echo "[production-api-deploy] 发布产物缺少 Pingora realpath canary 启用脚本: ${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" >&2 + exit 1 +fi +cp "${PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" +chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" +if [[ ! -f "${PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE}" ]]; then + echo "[production-api-deploy] 发布产物缺少 Pingora realpath canary 关闭脚本: ${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" >&2 + exit 1 +fi +cp "${PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" +chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" if [[ ! -f "${PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE}" ]]; then echo "[production-api-deploy] 发布产物缺少 Pingora health patrol env 切换脚本: ${SOURCE_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" >&2 exit 1 diff --git a/scripts/ops/pingora-current-release-audit.mjs b/scripts/ops/pingora-current-release-audit.mjs index e73da3f64..0f84584a3 100644 --- a/scripts/ops/pingora-current-release-audit.mjs +++ b/scripts/ops/pingora-current-release-audit.mjs @@ -21,6 +21,8 @@ const REQUIRED_ARTIFACTS = [ { path: 'scripts/check-pingora-canary-access-log-parity.mjs' }, { path: 'scripts/deploy/pingora-direct-enable.sh', executable: true }, { path: 'scripts/deploy/pingora-direct-rollback.sh', executable: true }, + { path: 'scripts/deploy/pingora-realpath-canary-enable.sh', executable: true }, + { path: 'scripts/deploy/pingora-realpath-canary-disable.sh', executable: true }, { path: 'scripts/deploy/pingora-health-patrol-env-switch.mjs', executable: true }, { path: 'scripts/deploy/pingora-tls-cert-sync.mjs', executable: true }, { path: 'deploy/systemd/genarrative-pingora-gateway.service' }, diff --git a/scripts/ops/pingora-cutover-status-snapshot.mjs b/scripts/ops/pingora-cutover-status-snapshot.mjs index c37ee8d69..34e001217 100644 --- a/scripts/ops/pingora-cutover-status-snapshot.mjs +++ b/scripts/ops/pingora-cutover-status-snapshot.mjs @@ -34,6 +34,8 @@ const RELEASE_ARTIFACTS = [ 'scripts/check-pingora-canary-access-log-parity.mjs', 'scripts/deploy/pingora-direct-enable.sh', 'scripts/deploy/pingora-direct-rollback.sh', + 'scripts/deploy/pingora-realpath-canary-enable.sh', + 'scripts/deploy/pingora-realpath-canary-disable.sh', 'scripts/deploy/pingora-health-patrol-env-switch.mjs', 'deploy/systemd/genarrative-pingora-gateway.service', 'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf',