脚本化 Pingora realpath canary 启停

新增 realpath canary 启用和关闭脚本

将脚本纳入 release、deploy、Jenkins 和 current release 自审

补充脚本化 canary 文档和生产运维护栏

增加 realpath canary 启停自测和发布部署护栏
This commit is contained in:
2026-06-17 23:45:58 +08:00
parent 93f085f95c
commit 123d4ada01
19 changed files with 1230 additions and 11 deletions
+2 -1
View File
@@ -68,7 +68,8 @@
- 启用 Nginx canary 前,目标机必须已经有持久 `genarrative-pingora-gateway.service` 运行在 `127.0.0.1:18081`,且带真实 `api-server`、SpacetimeDB、静态目录和 `/var/log/genarrative/pingora-gateway.access.log` 完成本机 shadow 验收;不要从一次性 `/tmp` 网关进程直接切到 Nginx handoff。
- 启用前必须运行 `npm run check:nginx-pingora-canary`;目标 agent 有 Nginx 时运行 `node scripts/check-nginx-pingora-canary.mjs --require-nginx`,同时做两个 snippet 的静态护栏和真实 `nginx -t`。本机或 CI 可运行 `npm run check:pingora-canary-docker`,用 Docker Nginx、真实 Pingora 和 mock 上游复现前缀 canary 与真实路径 canary 两条 handoff 链路。
- 前缀 canary reload 后运行 `GENARRATIVE_PINGORA_CANARY_BASE_URL=http://127.0.0.1 GENARRATIVE_PINGORA_CANARY_HOST=<域名> npm run check:pingora-canary-live`,再用 `scripts/check-pingora-canary-access-log-parity.mjs` 对账 `/var/log/nginx/genarrative.access.log``/var/log/genarrative/pingora-gateway.access.log`
- 真实路径 canary reload 后运行 `node scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`,再运行 `node scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js`。真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致
- 真实路径 canary 不再手工编辑 `/etc/nginx/conf.d/`。目标机从 current release 执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token <token> --host <域名> --base-url http://127.0.0.1:18083`;脚本会渲染到 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`,保持晚于主站 `log_format genarrative_upstream` 加载,执行 `nginx -t`、reload Nginx,并默认运行 realpath live smoke。`nginx -t`、reload 或 live smoke 任一失败都会恢复写入前配置。关闭时执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`,脚本会删除该 `zz-...` 文件,执行 `nginx -t` 和 reload,失败时恢复删除前配置
- 真实路径 canary reload 后可再次运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`,再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js`。真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致。
- 正式切换前在源码 checkout / CI / 构建环境跑 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`;启用真实路径 canary 时追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`
- 目标机 `/opt/genarrative/current` 只能跑随包 runtime-only 门禁:`/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`;验证真实路径 canary 或 Pingora 直连入口时分别追加 `--require-realpath-live ...``--require-direct ...` 参数。生成正式直连切换 runbook 时,`--direct-redirect-host` 和回退 smoke Host 必须与 `--direct-host` 同 hostname,只允许端口不同。
@@ -129,8 +129,8 @@
- 现象:目标机把 `genarrative-pingora-realpath-canary.conf` 放进 `/etc/nginx/conf.d/` 后,`nginx -t` 失败并报 `unknown log format "genarrative_upstream"`
- 原因:真实路径 canary 是独立 `server` 片段,并使用 `access_log /var/log/nginx/genarrative-pingora-realpath-canary.access.log genarrative_upstream;`。Nginx 会按文件名顺序加载 `conf.d`;如果 canary 文件名早于定义 `log_format genarrative_upstream` 的主站配置,access log 行会先被解析而找不到格式。
- 处理:真实路径 canary 要么 include 在已经定义 `log_format genarrative_upstream` 之后,要么放入晚于主站配置加载的文件,例如 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`另一种长期做法是把 `log_format` 放到所有 `conf.d` server 之前的全局 Nginx 配置。启用前必须先跑 `nginx -t`,失败时先移除临时 canary 文件再 reload,避免保留坏配置;检查配置时不要把 probe token 原文写入记录。
- 验证:`nginx -t` 通过后 reload Nginx,再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>``node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log ...``node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...`。若只启用了真实路径 canary,不要同时传 `--require-live`,否则前缀 canary 未启用时会按正式 Nginx HTTP 入口返回 301。
- 处理:真实路径 canary 启停统一用 current release 随包脚本,不再手工写 `/etc/nginx/conf.d/`。启用执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token <token> --host <域名> --base-url http://127.0.0.1:18083`,脚本固定写入晚于主站配置加载的 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`,并在 `nginx -t`、reload 或 live smoke 失败时恢复写入前配置。关闭执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`,脚本在 `nginx -t` 或 reload 失败时恢复删除前配置。另一种长期做法是把 `log_format` 放到所有 `conf.d` server 之前的全局 Nginx 配置。检查配置时不要把 probe token 原文写入记录。
- 验证:启用脚本通过后,再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>``node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log ...``node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...`。若只启用了真实路径 canary,不要同时传 `--require-live`,否则前缀 canary 未启用时会按正式 Nginx HTTP 入口返回 301。
- 关联:`deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf``deploy/nginx/README.md``docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md``scripts/check-pingora-release-readiness.mjs`
## Pingora release readiness 脚本不能只存在于源码 checkout
@@ -375,6 +375,7 @@ dev 根盘空间在安装后曾接近满盘;2026-06-17 进入 canary 前已清
- `node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath ... --json` 对账 `12/12 matched``missingCount=0``mismatchCount=0`
- `node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...` 通过 current release 自包含自审、realpath live smoke 和 realpath access log 对账,最终对账 `18/18 matched`
- 本阶段没有启用前缀 canary,因此不能同时传 `--require-live`;否则正式 HTTP 入口会按 Nginx 策略返回 `301`,并缺少 `X-Genarrative-Nginx-Handoff: pingora-canary`。当前 public `80/443` 仍由 Nginx 承接,realpath canary 只保留在 `127.0.0.1:18083` 用于后续 loopback 复核。
- 后续 realpath canary 启停统一走 current release 随包脚本:`/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token <token> --host <域名> --base-url http://127.0.0.1:18083``/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`。启用脚本固定写入 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`,并在 `nginx -t`、reload 或 live smoke 失败时回滚;关闭脚本在 `nginx -t` 或 reload 失败时恢复删除前配置。
## dev API release 正式路径验收记录
@@ -477,10 +478,10 @@ dev 根盘空间在安装后曾接近满盘;2026-06-17 进入 canary 前已清
2. 涉及 Nginx 模板、Pingora 路由、限流分组或路由文档时,同步更新 `deploy/pingora/nginx-route-parity.matrix.json`,并运行 `npm run check:pingora-route-parity``cargo test -p pingora-gateway --manifest-path server-rs/Cargo.toml matches_nginx_route_parity_matrix`
3. 容器内使用同一份 Web 产物、同一组真实上游地址跑 Pingora smoke,并继续对照 `deploy/nginx/genarrative.conf` 扩展真实上游路由 parity 自动测试。
4. 使用 `deploy/nginx/snippets/genarrative-pingora-canary.conf` 做 Nginx 前缀 canary;启用前先跑 `npm run check:nginx-pingora-canary``npm run check:pingora-canary-docker`,有 Nginx 或 Docker 的目标环境分别强制跑 `node scripts/check-nginx-pingora-canary.mjs --require-nginx``node scripts/check-pingora-canary-docker.mjs --require-docker --pull`,其中 Docker handoff 会自动对账临时 Nginx 与 Pingora access log。启用后跑 `npm run check:pingora-canary-live`,再用 current release 随包 access log parity 脚本对账目标机 Nginx 与 Pingora access log。canary live 的 base URL、prefix、Host、额外 path 和 timeout 不能包含换行或 NULcanary live timeout 和 access log `since-lines` 必须是正整数,非法值直接失败。
5. 前缀 canary 通过后,再使用 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 做真实路径 canary;它必须作为独立本机 `server` include 到 Nginx `http` 上下文,不能 include 到生产 `443` server 内。该片段使用 `access_log ... genarrative_upstream`如果放入 `/etc/nginx/conf.d/`文件名必须保证晚于定义 `log_format genarrative_upstream` 的主站配置加载,例如 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`;否则 `nginx -t` 会报 `unknown log format "genarrative_upstream"`。启用后跑 `node scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`,再用 current release 随包 access log parity 脚本传 `--realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log` 对账 `/api/creation-entry/config``/v1/identity``/assets/app.js` 等真实路径。
5. 前缀 canary 通过后,再使用 current release 随包 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token <token> --host <域名> --base-url http://127.0.0.1:18083` 启用真实路径 canary;它会把 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 渲染成独立本机 `server`,写入 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`,不能 include 到生产 `443` server 内。该片段使用 `access_log ... genarrative_upstream`,文件名必须保证晚于定义 `log_format genarrative_upstream` 的主站配置加载;否则 `nginx -t` 会报 `unknown log format "genarrative_upstream"`。启用脚本会先执行 `nginx -t`、reload Nginx,再默认运行 realpath live smoke,任一阶段失败都会恢复写入前配置。关闭时执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`,脚本会在 `nginx -t` 或 reload 失败时恢复删除前配置。启用后跑 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`,再用 current release 随包 access log parity 脚本传 `--realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log` 对账 `/api/creation-entry/config``/v1/identity``/assets/app.js` 等真实路径。
6. 目标机 canary include 后必须跑正式切换聚合门禁:源码 checkout / CI / 构建环境执行默认全量命令 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`;目标机 current release 执行 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。启用真实路径 canary 时,两种命令都追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。缺少 Host 会直接失败,避免 live canary 误测默认 vhostlive smoke 后还会按 `request_id` 对账 Nginx 与 Pingora access log,缺少同一请求的 Pingora 日志、状态码、方法或 path 漂移都会失败。
7. 如需评估 Pingora 直连公网入口,必须显式配置 `TLS_LISTEN`、证书、私钥和 `HTTP_REDIRECT_LISTEN`Certbot 证书先用随包 `scripts/deploy/pingora-tls-cert-sync.mjs` 同步到 `/etc/genarrative/pingora-tls/<域名>/`,不要直接 chmod Lets Encrypt live/archive 原路径;绑定 `80/443` 时还必须人工启用 `genarrative-pingora-gateway-direct-entry.conf` drop-in 授予 `CAP_NET_BIND_SERVICE`。随后用 `npm run check:pingora-gateway-smoke` 覆盖 TLS / HTTP/2 ALPN / redirect / WSS subscribe;目标机必须先跑 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`,再跑 `npm run check:pingora-direct-live` 或 release readiness 的 `--require-direct`,且 `--require-direct` 必须带 direct HTTPS base URL、direct HTTP base URL、正式域名 Host/SNI、redirect Location host、Pingora access log 文件、health patrol env 文件、direct preflight env 文件、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户和 systemd 服务用户证书可读检查、service 二进制可执行检查、端口释放检查和显式 SpacetimeDB 数据库名,并会拒绝 `--skip-wss`。高端口 rehearsal 使用 `https://127.0.0.1:<高端口>` 打入但期望 HTTP redirect Location 指向正式域名默认 HTTPS 入口时,额外传 `--direct-redirect-base-url https://<域名>``--direct-redirect-host` 仍必须保留,用于 runbook Host 一致性约束。direct live 会用生成的 `request_id` 反查 Pingora access log;缺少对应日志、method 漂移、path 漂移或 status 漂移都算直连门禁失败。direct preflight 会拒绝开启网关保护但未确认共享保护层的 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT>1` 配置;`--env-file``--systemd-service`、服务用户和 env 中的 listen / cert / key 值都不能包含换行或 NUL,执行 `systemctl cat``sudo -u <serviceUser> test -r <file>` 前还会复核子命令参数,避免污染参数进入目标机预检命令;direct live 的 URL、Host、redirect base URL、probe token、额外 path、数据库名、access log 路径、timeout 和布尔 env 也不能包含换行或 NUL,且会在发起请求前失败;direct live timeout 必须是正整数,直连相关布尔 env 只接受 `true/false``1/0``yes/no``on/off` 或空值,非法值直接失败。证书申请与续期仍由 Certbot / 外部自动化承担,网关只读取现有文件。
8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs` `deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release;缺失时部署应 fail-fast,切换窗口不能依赖源码 checkout 或 Jenkins workspace。
8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs`、realpath canary 启停脚本`deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release;缺失时部署应 fail-fast,切换窗口不能依赖源码 checkout 或 Jenkins workspace。
9. direct preflight / direct live timeout 必须是正整数;直连相关布尔 env 和 Pingora gateway env 里的 `TRUST_X_FORWARDED_FOR` / `TRUSTED_FRONT_PROXY_CONFIRMED` 只接受 `true/false``1/0``yes/no``on/off` 或空值,非法值直接失败。
10. Pingora 正式化口径固定为 gzip-onlyBrotli 继续由 Nginx / 前置代理承担,直连 Pingora 不以 Brotli parity 作为切换门禁。如需多实例,开启网关接流保护时必须先引入共享限流 / 共享并发保护层并设置 `GENARRATIVE_PINGORA_GATEWAY_SHARED_PROTECTION_CONFIRMED=true`,否则保持 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT=1`;关闭网关保护的多实例方案必须明确由前置 Nginx / LB 承担全局限流。
10. 前缀 canary 稳定后,再评估是否做真实路径 canary;真实路径 canary 稳定后,再评估是否让 Pingora 直接承接公网入口。
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -123,7 +123,7 @@ pipeline {
stage('Archive') {
steps {
archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/api-server,build/${env.EFFECTIVE_BUILD_VERSION}/api-server.sha256,build/${env.EFFECTIVE_BUILD_VERSION}/release-manifest.json,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/systemd/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/nginx/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/pingora/**", fingerprint: true
archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/api-server,build/${env.EFFECTIVE_BUILD_VERSION}/api-server.sha256,build/${env.EFFECTIVE_BUILD_VERSION}/release-manifest.json,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/systemd/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/nginx/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/pingora/**", fingerprint: true
script {
if (params.INCLUDE_PINGORA_GATEWAY) {
archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/pingora-gateway,build/${env.EFFECTIVE_BUILD_VERSION}/pingora-gateway.sha256", fingerprint: true
+2 -2
View File
@@ -66,7 +66,7 @@ pipeline {
copyArtifacts(
projectName: params.BUILD_JOB_NAME,
selector: specific(params.BUILD_NUMBER_TO_DEPLOY),
filter: "build/${params.BUILD_VERSION}/api-server,build/${params.BUILD_VERSION}/api-server.sha256,build/${params.BUILD_VERSION}/release-manifest.json,build/${params.BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${params.BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${params.BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${params.BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${params.BUILD_VERSION}/deploy/systemd/**,build/${params.BUILD_VERSION}/deploy/nginx/**,build/${params.BUILD_VERSION}/deploy/env/**,build/${params.BUILD_VERSION}/deploy/pingora/**",
filter: "build/${params.BUILD_VERSION}/api-server,build/${params.BUILD_VERSION}/api-server.sha256,build/${params.BUILD_VERSION}/release-manifest.json,build/${params.BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${params.BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${params.BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${params.BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${params.BUILD_VERSION}/deploy/systemd/**,build/${params.BUILD_VERSION}/deploy/nginx/**,build/${params.BUILD_VERSION}/deploy/env/**,build/${params.BUILD_VERSION}/deploy/pingora/**",
target: '.',
fingerprintArtifacts: true
)
@@ -92,7 +92,7 @@ pipeline {
sh '''
bash -lc '
set -euo pipefail
chmod +x "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-on.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-off.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
chmod +x "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-on.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-off.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
"build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" \
--source-dir "build/${BUILD_VERSION}" \
--version "${BUILD_VERSION}" \
+3
View File
@@ -38,6 +38,7 @@
"check:pingora-route-parity": "node scripts/check-pingora-route-parity.mjs",
"check:pingora-canary-live": "node scripts/check-pingora-canary-live.mjs",
"check:pingora-canary-live-guard": "node scripts/check-pingora-canary-live-guard.mjs",
"check:pingora-realpath-canary-toggle": "node scripts/check-pingora-realpath-canary-toggle.mjs",
"check:pingora-direct-preflight": "node -- scripts/check-pingora-direct-preflight.mjs && node scripts/check-pingora-direct-preflight-guard.mjs",
"check:pingora-direct-live": "node scripts/check-pingora-direct-live.mjs",
"check:pingora-direct-live-guard": "node scripts/check-pingora-direct-live-guard.mjs",
@@ -57,6 +58,8 @@
"check:pingora-release-readiness-plan": "node scripts/check-pingora-release-readiness-plan.mjs",
"check:pingora-release-readiness": "node scripts/check-pingora-release-readiness.mjs",
"plan:pingora-direct-cutover": "node scripts/check-pingora-release-readiness.mjs --dry-run-cutover",
"deploy:pingora-realpath-canary-enable": "bash scripts/deploy/pingora-realpath-canary-enable.sh",
"deploy:pingora-realpath-canary-disable": "bash scripts/deploy/pingora-realpath-canary-disable.sh",
"deploy:pingora-direct-enable": "bash scripts/deploy/pingora-direct-enable.sh",
"deploy:pingora-direct-rollback": "bash scripts/deploy/pingora-direct-rollback.sh",
"assets:child-motion-demo": "node scripts/generate-child-motion-demo-assets.mjs",
+5 -1
View File
@@ -520,6 +520,8 @@ cp "${SCRIPT_DIR}/deploy/maintenance-on.sh" "${TARGET_DIR}/scripts/deploy/mainte
cp "${SCRIPT_DIR}/deploy/maintenance-off.sh" "${TARGET_DIR}/scripts/deploy/maintenance-off.sh"
cp "${SCRIPT_DIR}/deploy/pingora-direct-enable.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh"
cp "${SCRIPT_DIR}/deploy/pingora-direct-rollback.sh" "${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh"
cp "${SCRIPT_DIR}/deploy/pingora-realpath-canary-enable.sh" "${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh"
cp "${SCRIPT_DIR}/deploy/pingora-realpath-canary-disable.sh" "${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh"
cp "${SCRIPT_DIR}/deploy/pingora-health-patrol-env-switch.mjs" "${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
cp "${SCRIPT_DIR}/deploy/pingora-tls-cert-sync.mjs" "${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs"
cp "${SCRIPT_DIR}/deploy/jenkins-inbound-agent-start.sh" "${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh"
@@ -535,6 +537,8 @@ chmod +x \
"${TARGET_DIR}/scripts/deploy/maintenance-off.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-direct-enable.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-direct-rollback.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" \
"${TARGET_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" \
"${TARGET_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs" \
"${TARGET_DIR}/scripts/jenkins-inbound-agent-start.sh" \
@@ -580,7 +584,7 @@ cat >"${TARGET_DIR}/README.md" <<EOF
- \`migration-bootstrap-secret.txt\`:构建 \`spacetime_module.wasm\` 时注入的迁移引导密钥,仅用于创建首个迁移操作员;请作为敏感文件保存到 Jenkins Secret Text,授权完成后不要长期留在公开归档中。
- \`*.sha256\`:发布产物 checksum,用于部署前校验。
- \`release-manifest.json\`:发布版本、源码 commit 与产物清单。
- \`scripts/\`:维护模式脚本、数据库导入导出脚本、数据库 OSS 备份脚本、生产健康巡检脚本、Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本、迁移授权脚本和 Jenkins inbound agent systemd 安装脚本。
- \`scripts/\`:维护模式脚本、数据库导入导出脚本、数据库 OSS 备份脚本、生产健康巡检脚本、Pingora release readiness 聚合门禁、直连启用 / 回退、realpath canary 启用 / 关闭、health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本、迁移授权脚本和 Jenkins inbound agent systemd 安装脚本。
- \`scripts/deploy/production-api-deploy.sh\`API Deploy 执行入口;同目录的 \`maintenance-on.sh\` / \`maintenance-off.sh\` 必须来自同一发布包。
- \`deploy/\`systemd、Nginx 和生产环境变量示例;\`deploy/nginx/genarrative-dev-http.conf\` 仅供无域名开发服初始化使用。
@@ -463,6 +463,8 @@ function prepareReleaseRoot(releaseRoot, options) {
'scripts/check-pingora-canary-access-log-parity.mjs',
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-realpath-canary-enable.sh',
'scripts/deploy/pingora-realpath-canary-disable.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'scripts/deploy/pingora-tls-cert-sync.mjs',
'deploy/systemd/genarrative-pingora-gateway.service',
@@ -487,6 +489,8 @@ function prepareReleaseRoot(releaseRoot, options) {
for (const file of [
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-realpath-canary-enable.sh',
'scripts/deploy/pingora-realpath-canary-disable.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'scripts/deploy/pingora-tls-cert-sync.mjs',
]) {
@@ -738,7 +738,10 @@ function prepareReleaseRoot(releaseRoot) {
'scripts/check-pingora-canary-access-log-parity.mjs',
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-realpath-canary-enable.sh',
'scripts/deploy/pingora-realpath-canary-disable.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'scripts/deploy/pingora-tls-cert-sync.mjs',
'deploy/systemd/genarrative-pingora-gateway.service',
'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf',
'deploy/nginx/snippets/genarrative-pingora-canary.conf',
@@ -760,7 +763,10 @@ function prepareReleaseRoot(releaseRoot) {
for (const file of [
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-realpath-canary-enable.sh',
'scripts/deploy/pingora-realpath-canary-disable.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'scripts/deploy/pingora-tls-cert-sync.mjs',
]) {
chmodExecutable(path.join(releaseRoot, file));
}
@@ -0,0 +1,432 @@
#!/usr/bin/env node
import { spawnSync } from 'node:child_process';
import {
chmodSync,
existsSync,
mkdtempSync,
mkdirSync,
readFileSync,
rmSync,
writeFileSync,
} from 'node:fs';
import { tmpdir } from 'node:os';
import path from 'node:path';
const ENABLE_SCRIPT = 'scripts/deploy/pingora-realpath-canary-enable.sh';
const DISABLE_SCRIPT = 'scripts/deploy/pingora-realpath-canary-disable.sh';
const failures = [];
const tmpRoot = mkdtempSync(path.join(tmpdir(), 'genarrative-realpath-canary-'));
try {
main();
} finally {
rmSync(tmpRoot, { recursive: true, force: true });
}
if (failures.length > 0) {
console.error('[check:pingora-realpath-canary-toggle] FAILED');
for (const failure of failures) {
console.error(`- ${failure}`);
}
process.exit(1);
}
console.log('[check:pingora-realpath-canary-toggle] OK');
function main() {
assertScriptShape();
assertDryRunDoesNotWrite();
assertApplyWritesRenderedConfigAndRunsLive();
assertEnableRollsBackWhenLiveFails();
assertDisableRemovesConfig();
assertDisableRollsBackWhenReloadFails();
assertRejectsUnsafeInputs();
}
function assertScriptShape() {
for (const script of [ENABLE_SCRIPT, DISABLE_SCRIPT]) {
const content = readFileSync(script, 'utf8');
assertIncludes(content, '--apply', `${script} 必须默认 dry-run 并要求显式 --apply。`);
assertIncludes(content, 'nginx -t', `${script} 帮助文案必须说明会先做 nginx -t。`);
assertIncludes(
content,
'zz-genarrative-pingora-realpath-canary.conf',
`${script} 必须默认使用晚于主站配置加载的 conf.d 文件名。`,
);
assertIncludes(
content,
'不能包含换行或 NUL 字符',
`${script} 必须拒绝控制字符参数。`,
);
assertIncludes(content, '不能是文件系统根目录', `${script} 必须拒绝根路径。`);
}
const enableContent = readFileSync(ENABLE_SCRIPT, 'utf8');
assertIncludes(
enableContent,
'check-pingora-canary-live.mjs',
'enable 脚本必须默认运行 realpath live smoke。',
);
assertIncludes(
enableContent,
'恢复写入前配置',
'enable 脚本失败时必须恢复写入前配置。',
);
const disableContent = readFileSync(DISABLE_SCRIPT, 'utf8');
assertIncludes(
disableContent,
'恢复删除前配置',
'disable 脚本失败时必须恢复删除前配置。',
);
}
function assertDryRunDoesNotWrite() {
const fixture = createFixture('dry-run');
const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf');
const result = runScript(ENABLE_SCRIPT, [
'--probe-token',
'dry-run-token',
'--host',
'dev.genarrative.world',
'--template-path',
fixture.templatePath,
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--live-script',
fixture.liveScript,
]);
assertStatus(result, 0, 'enable dry-run 应成功。');
if (existsSync(targetPath)) {
failures.push('enable dry-run 不应写入目标 Nginx 配置。');
}
assertIncludes(result.stdout, 'dry-run', 'enable dry-run 应输出 dry-run 提示。');
}
function assertApplyWritesRenderedConfigAndRunsLive() {
const fixture = createFixture('apply-ok');
const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf');
const result = runScript(ENABLE_SCRIPT, [
'--apply',
'--probe-token',
'apply-real-token',
'--host',
'dev.genarrative.world',
'--base-url',
'http://127.0.0.1:18083',
'--template-path',
fixture.templatePath,
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--live-script',
fixture.liveScript,
'--no-status',
]);
assertStatus(result, 0, 'enable apply 应成功。');
const rendered = readFileSync(targetPath, 'utf8');
assertIncludes(rendered, '"apply-real-token"', 'enable apply 应替换 probe token。');
if (rendered.includes('__GENARRATIVE_PINGORA_PROBE_TOKEN__')) {
failures.push('enable apply 后目标配置不应保留 probe token 占位符。');
}
const calls = readFileSync(fixture.callsPath, 'utf8');
assertIncludes(calls, 'nginx -t', 'enable apply 必须执行 nginx -t。');
assertIncludes(calls, 'systemctl reload nginx.service', 'enable apply 必须 reload nginx。');
assertIncludes(
calls,
'live --realpath --base-url http://127.0.0.1:18083 --host dev.genarrative.world',
'enable apply 必须执行 realpath live smoke。',
);
}
function assertEnableRollsBackWhenLiveFails() {
const fixture = createFixture('enable-live-fails', { liveExitCode: 7 });
const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf');
writeFileSync(targetPath, 'previous-config\n', 'utf8');
const result = runScript(
ENABLE_SCRIPT,
[
'--apply',
'--probe-token',
'rollback-token',
'--host',
'dev.genarrative.world',
'--template-path',
fixture.templatePath,
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--live-script',
fixture.liveScript,
'--no-status',
],
fixture.env,
);
if (result.status === 0) {
failures.push('realpath live smoke 失败时 enable apply 必须失败。');
}
const restored = readFileSync(targetPath, 'utf8');
if (restored !== 'previous-config\n') {
failures.push('realpath live smoke 失败时 enable apply 必须恢复旧配置。');
}
assertIncludes(
result.stderr,
'恢复写入前配置',
'enable live 失败时必须说明已恢复写入前配置。',
);
}
function assertDisableRemovesConfig() {
const fixture = createFixture('disable-ok');
const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf');
writeFileSync(targetPath, 'enabled-config\n', 'utf8');
const result = runScript(
DISABLE_SCRIPT,
[
'--apply',
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--no-status',
],
fixture.env,
);
assertStatus(result, 0, 'disable apply 应成功。');
if (existsSync(targetPath)) {
failures.push('disable apply 成功后必须删除 realpath canary 配置。');
}
const calls = readFileSync(fixture.callsPath, 'utf8');
assertIncludes(calls, 'nginx -t', 'disable apply 必须执行 nginx -t。');
assertIncludes(calls, 'systemctl reload nginx.service', 'disable apply 必须 reload nginx。');
}
function assertDisableRollsBackWhenReloadFails() {
const fixture = createFixture('disable-reload-fails', { reloadExitCode: 9 });
const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf');
writeFileSync(targetPath, 'enabled-config\n', 'utf8');
const result = runScript(DISABLE_SCRIPT, [
'--apply',
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--no-status',
]);
if (result.status === 0) {
failures.push('reload 失败时 disable apply 必须失败。');
}
const restored = readFileSync(targetPath, 'utf8');
if (restored !== 'enabled-config\n') {
failures.push('reload 失败时 disable apply 必须恢复删除前配置。');
}
assertIncludes(
result.stderr,
'恢复删除前配置',
'disable reload 失败时必须说明已恢复删除前配置。',
);
}
function assertRejectsUnsafeInputs() {
const fixture = createFixture('unsafe-inputs');
const targetPath = path.join(fixture.nginxDir, 'zz-genarrative-pingora-realpath-canary.conf');
const cases = [
{
name: 'missing token',
script: ENABLE_SCRIPT,
args: [
'--apply',
'--host',
'dev.genarrative.world',
'--template-path',
fixture.templatePath,
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--live-script',
fixture.liveScript,
],
expected: '--apply 必须提供 --probe-token',
},
{
name: 'url host',
script: ENABLE_SCRIPT,
args: [
'--probe-token',
'unsafe-token',
'--host',
'https://dev.genarrative.world',
'--template-path',
fixture.templatePath,
'--target-path',
targetPath,
'--nginx-binary',
fixture.nginxBinary,
'--systemctl-binary',
fixture.systemctlBinary,
'--live-script',
fixture.liveScript,
],
expected: '不能是 URL',
},
{
name: 'relative target',
script: DISABLE_SCRIPT,
args: ['--target-path', 'relative.conf'],
expected: '--target-path 必须是绝对路径',
},
{
name: 'root target',
script: DISABLE_SCRIPT,
args: ['--target-path', '/'],
expected: '--target-path 不能是文件系统根目录',
},
];
for (const item of cases) {
const result = runScript(item.script, item.args);
if (result.status === 0) {
failures.push(`${item.name}: 不安全参数必须失败。`);
}
assertIncludes(
`${result.stderr}\n${result.stdout}`,
item.expected,
`${item.name}: 应输出明确错误。`,
);
}
}
function createFixture(name, options = {}) {
const root = path.join(tmpRoot, name);
const nginxDir = path.join(root, 'nginx-conf');
mkdirSync(nginxDir, { recursive: true });
const callsPath = path.join(root, 'calls.log');
const templatePath = path.join(root, 'genarrative-pingora-realpath-canary.conf');
const nginxBinary = path.join(root, 'nginx');
const systemctlBinary = path.join(root, 'systemctl');
const liveScript = path.join(root, 'live.mjs');
writeFileSync(
templatePath,
[
'server {',
' listen 127.0.0.1:18083;',
' access_log /var/log/nginx/genarrative-pingora-realpath-canary.access.log genarrative_upstream;',
' location = /__genarrative_pingora_realpath_canary/healthz {',
' proxy_set_header X-Genarrative-Pingora-Probe "__GENARRATIVE_PINGORA_PROBE_TOKEN__";',
' proxy_pass http://127.0.0.1:18081/__genarrative_pingora/healthz;',
' }',
'}',
'',
].join('\n'),
'utf8',
);
writeFileSync(
nginxBinary,
[
'#!/usr/bin/env bash',
`echo "nginx $*" >> ${shellQuote(callsPath)}`,
'if [[ "${GENARRATIVE_FAKE_NGINX_T_EXIT:-0}" != "0" ]]; then exit "${GENARRATIVE_FAKE_NGINX_T_EXIT}"; fi',
'exit 0',
'',
].join('\n'),
'utf8',
);
writeFileSync(
systemctlBinary,
[
'#!/usr/bin/env bash',
`echo "systemctl $*" >> ${shellQuote(callsPath)}`,
`reload_exit="\${GENARRATIVE_FAKE_RELOAD_EXIT:-${Number(options.reloadExitCode || 0)}}"`,
'if [[ "$1" == "reload" && "${reload_exit}" != "0" ]]; then exit "${reload_exit}"; fi',
'exit 0',
'',
].join('\n'),
'utf8',
);
writeFileSync(
liveScript,
[
'#!/usr/bin/env node',
"import { appendFileSync } from 'node:fs';",
`appendFileSync(${JSON.stringify(callsPath)}, 'live ' + process.argv.slice(2).join(' ') + '\\n');`,
`process.exit(Number(process.env.GENARRATIVE_FAKE_LIVE_EXIT || ${Number(options.liveExitCode || 0)}));`,
'',
].join('\n'),
'utf8',
);
chmodExecutable(nginxBinary);
chmodExecutable(systemctlBinary);
chmodExecutable(liveScript);
return {
root,
nginxDir,
callsPath,
templatePath,
nginxBinary,
systemctlBinary,
liveScript,
env: {
GENARRATIVE_FAKE_RELOAD_EXIT: String(options.reloadExitCode || 0),
GENARRATIVE_FAKE_LIVE_EXIT: String(options.liveExitCode || 0),
},
};
}
function runScript(script, args, env = {}) {
return spawnSync('bash', [script, ...args], {
cwd: process.cwd(),
env: { ...process.env, ...env },
encoding: 'utf8',
});
}
function chmodExecutable(file) {
chmodSync(file, 0o755);
}
function shellQuote(value) {
return `'${String(value).replaceAll("'", "'\\''")}'`;
}
function assertStatus(result, expected, message) {
if (result.status !== expected) {
failures.push(
`${message} 实际退出码 ${result.status}\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`,
);
}
}
function assertIncludes(content, needle, message) {
if (!content.includes(needle)) {
failures.push(`${message} 缺少 ${needle}`);
}
}
+64
View File
@@ -61,6 +61,8 @@ function main() {
assertMissingPingoraCutoverEvidenceAuditFails();
assertMissingHealthPatrolEnvCheckFails();
assertMissingPingoraReleaseReadinessFails();
assertMissingPingoraRealpathCanaryEnableFails();
assertMissingPingoraRealpathCanaryDisableFails();
assertMissingPingoraHealthPatrolEnvSwitchFails();
assertMissingPingoraTlsCertSyncFails();
assertMissingEnvExamplesFails();
@@ -99,6 +101,14 @@ function assertDeployCopiesPingoraDirectReleaseDependencies() {
path.join(releaseDir, 'scripts/deploy/pingora-direct-rollback.sh'),
'current release 必须包含 Pingora 直连回退脚本。',
);
assertFileExists(
path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'),
'current release 必须包含 Pingora realpath canary 启用脚本。',
);
assertFileExists(
path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'),
'current release 必须包含 Pingora realpath canary 关闭脚本。',
);
assertFileExists(
path.join(
releaseDir,
@@ -890,6 +900,52 @@ function assertMissingPingoraHealthPatrolEnvSwitchFails() {
}
}
function assertMissingPingoraRealpathCanaryEnableFails() {
const fixture = prepareFixture('missing-pingora-realpath-canary-enable');
rmSync(
path.join(
fixture.sourceDir,
'scripts/deploy/pingora-realpath-canary-enable.sh',
),
);
const result = runDeploy(fixture);
if (result.status === 0) {
failures.push('发布产物缺少 Pingora realpath canary 启用脚本时部署必须失败。');
}
assertIncludes(
result.stderr,
'发布产物缺少 Pingora realpath canary 启用脚本',
'缺少 Pingora realpath canary 启用脚本时必须给出明确错误。',
);
if (!existsSync(fixture.maintenanceFile)) {
failures.push('部署失败时必须保持维护模式。');
}
}
function assertMissingPingoraRealpathCanaryDisableFails() {
const fixture = prepareFixture('missing-pingora-realpath-canary-disable');
rmSync(
path.join(
fixture.sourceDir,
'scripts/deploy/pingora-realpath-canary-disable.sh',
),
);
const result = runDeploy(fixture);
if (result.status === 0) {
failures.push('发布产物缺少 Pingora realpath canary 关闭脚本时部署必须失败。');
}
assertIncludes(
result.stderr,
'发布产物缺少 Pingora realpath canary 关闭脚本',
'缺少 Pingora realpath canary 关闭脚本时必须给出明确错误。',
);
if (!existsSync(fixture.maintenanceFile)) {
failures.push('部署失败时必须保持维护模式。');
}
}
function assertMissingPingoraTlsCertSyncFails() {
const fixture = prepareFixture('missing-pingora-tls-cert-sync');
rmSync(
@@ -1044,6 +1100,14 @@ function prepareFixture(name) {
'scripts/deploy/pingora-direct-rollback.sh',
path.join(sourceDir, 'scripts/deploy/pingora-direct-rollback.sh'),
);
copyFile(
'scripts/deploy/pingora-realpath-canary-enable.sh',
path.join(sourceDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'),
);
copyFile(
'scripts/deploy/pingora-realpath-canary-disable.sh',
path.join(sourceDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'),
);
copyFile(
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
path.join(sourceDir, 'scripts/deploy/pingora-health-patrol-env-switch.mjs'),
+27 -1
View File
@@ -102,6 +102,22 @@ function assertApiReleaseContainsPingoraDirectDependencies() {
path.join(releaseDir, 'scripts/deploy/pingora-direct-rollback.sh'),
'Pingora 直连回退脚本必须可执行。',
);
assertFileExists(
path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'),
'API release 必须包含 Pingora realpath canary 启用脚本。',
);
assertExecutable(
path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-enable.sh'),
'Pingora realpath canary 启用脚本必须可执行。',
);
assertFileExists(
path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'),
'API release 必须包含 Pingora realpath canary 关闭脚本。',
);
assertExecutable(
path.join(releaseDir, 'scripts/deploy/pingora-realpath-canary-disable.sh'),
'Pingora realpath canary 关闭脚本必须可执行。',
);
assertFileExists(
path.join(
releaseDir,
@@ -254,9 +270,19 @@ function assertApiReleaseContainsPingoraDirectDependencies() {
);
assertIncludes(
releaseReadme,
'Pingora release readiness 聚合门禁、直连启用 / 回退 / health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审 / 状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本',
'Pingora release readiness 聚合门禁、直连启用 / 回退',
'API release README 必须说明随包携带 Pingora 直连脚本。',
);
assertIncludes(
releaseReadme,
'realpath canary 启用 / 关闭',
'API release README 必须说明随包携带 Pingora realpath canary 启停脚本。',
);
assertIncludes(
releaseReadme,
'health patrol env 切换 / TLS 证书同步 / 预检 / direct live smoke / canary live smoke / canary access log 对账 / current release 自审',
'API release README 必须说明随包携带 Pingora 运行态复核脚本。',
);
assertIncludes(
releaseReadme,
'状态快照 / 证据包 / 命令证据 / 证据验真 / 证据根目录审计脚本',
+113
View File
@@ -71,6 +71,16 @@ const checks = [
includes: 'scripts/check-pingora-canary-live.mjs',
reason: 'Pingora current release 自审必须确认 canary live smoke 脚本已随包发布。',
},
{
file: 'scripts/ops/pingora-current-release-audit.mjs',
includes: 'scripts/deploy/pingora-realpath-canary-enable.sh',
reason: 'Pingora current release 自审必须确认 realpath canary 启用脚本已随包发布。',
},
{
file: 'scripts/ops/pingora-current-release-audit.mjs',
includes: 'scripts/deploy/pingora-realpath-canary-disable.sh',
reason: 'Pingora current release 自审必须确认 realpath canary 关闭脚本已随包发布。',
},
{
file: 'scripts/ops/pingora-current-release-audit.mjs',
includes: '--systemd-show',
@@ -1013,6 +1023,16 @@ const checks = [
includes: 'scripts/deploy/pingora-direct-rollback.sh',
reason: '生产 API release 必须携带 Pingora 直连回退脚本。',
},
{
file: 'scripts/build-production-release.sh',
includes: 'scripts/deploy/pingora-realpath-canary-enable.sh',
reason: '生产 API release 必须携带 Pingora realpath canary 启用脚本。',
},
{
file: 'scripts/build-production-release.sh',
includes: 'scripts/deploy/pingora-realpath-canary-disable.sh',
reason: '生产 API release 必须携带 Pingora realpath canary 关闭脚本。',
},
{
file: 'scripts/build-production-release.sh',
includes: 'scripts/deploy/pingora-health-patrol-env-switch.mjs',
@@ -1069,6 +1089,26 @@ const checks = [
includes: 'PINGORA_CANARY_LIVE_SCRIPT_SOURCE',
reason: 'API deploy 必须从发布产物复制 Pingora canary live smoke 脚本到 current release。',
},
{
file: 'scripts/deploy/production-api-deploy.sh',
includes: 'PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE',
reason: 'API deploy 必须从发布产物复制 Pingora realpath canary 启用脚本到 current release。',
},
{
file: 'scripts/deploy/production-api-deploy.sh',
includes: 'PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE',
reason: 'API deploy 必须从发布产物复制 Pingora realpath canary 关闭脚本到 current release。',
},
{
file: 'scripts/deploy/production-api-deploy.sh',
includes: '发布产物缺少 Pingora realpath canary 启用脚本',
reason: 'API deploy 缺少 realpath canary 启用脚本时必须 fail-fast。',
},
{
file: 'scripts/deploy/production-api-deploy.sh',
includes: '发布产物缺少 Pingora realpath canary 关闭脚本',
reason: 'API deploy 缺少 realpath canary 关闭脚本时必须 fail-fast。',
},
{
file: 'scripts/deploy/production-api-deploy.sh',
includes: 'PINGORA_CURRENT_RELEASE_AUDIT_SCRIPT_SOURCE',
@@ -1119,6 +1159,16 @@ const checks = [
includes: 'scripts/check-production-health-patrol-env.mjs',
reason: 'API release 动态烟测必须断言健康巡检 env 复核脚本随包发布。',
},
{
file: 'scripts/check-production-api-release.mjs',
includes: 'scripts/deploy/pingora-realpath-canary-enable.sh',
reason: 'API release 动态烟测必须断言 Pingora realpath canary 启用脚本随包发布。',
},
{
file: 'scripts/check-production-api-release.mjs',
includes: 'scripts/deploy/pingora-realpath-canary-disable.sh',
reason: 'API release 动态烟测必须断言 Pingora realpath canary 关闭脚本随包发布。',
},
{
file: 'scripts/check-production-api-deploy.mjs',
includes: 'deploy/env/health-patrol.env.example',
@@ -1149,6 +1199,30 @@ const checks = [
reason:
'API deploy 动态烟测必须断言 Pingora 切换命令证据脚本复制到 current release。',
},
{
file: 'scripts/check-production-api-deploy.mjs',
includes: 'scripts/deploy/pingora-realpath-canary-enable.sh',
reason:
'API deploy 动态烟测必须断言 Pingora realpath canary 启用脚本复制到 current release。',
},
{
file: 'scripts/check-production-api-deploy.mjs',
includes: 'scripts/deploy/pingora-realpath-canary-disable.sh',
reason:
'API deploy 动态烟测必须断言 Pingora realpath canary 关闭脚本复制到 current release。',
},
{
file: 'scripts/check-production-api-deploy.mjs',
includes: 'assertMissingPingoraRealpathCanaryEnableFails',
reason:
'API deploy 动态烟测必须覆盖缺少 Pingora realpath canary 启用脚本时 fail-fast。',
},
{
file: 'scripts/check-production-api-deploy.mjs',
includes: 'assertMissingPingoraRealpathCanaryDisableFails',
reason:
'API deploy 动态烟测必须覆盖缺少 Pingora realpath canary 关闭脚本时 fail-fast。',
},
{
file: 'scripts/check-production-api-deploy.mjs',
includes: 'assertMissingPingoraCutoverCommandEvidenceFails',
@@ -1171,6 +1245,21 @@ const checks = [
includes: 'deploy:pingora-direct-enable',
reason: 'Pingora 直连启用必须保留可重复执行的 npm 入口。',
},
{
file: 'package.json',
includes: 'check:pingora-realpath-canary-toggle',
reason: 'Pingora realpath canary 启停脚本必须有本地自测入口。',
},
{
file: 'package.json',
includes: 'deploy:pingora-realpath-canary-enable',
reason: 'Pingora realpath canary 启用必须保留可重复执行的 npm 入口。',
},
{
file: 'package.json',
includes: 'deploy:pingora-realpath-canary-disable',
reason: 'Pingora realpath canary 关闭必须保留可重复执行的 npm 入口。',
},
{
file: 'package.json',
includes: 'deploy:pingora-direct-rollback',
@@ -5545,6 +5634,18 @@ const checks = [
'build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs',
reason: 'API Build 归档必须包含 Pingora health patrol env 切换脚本。',
},
{
file: 'jenkins/Jenkinsfile.production-api-build',
includes:
'build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh',
reason: 'API Build 归档必须包含 Pingora realpath canary 启用脚本。',
},
{
file: 'jenkins/Jenkinsfile.production-api-build',
includes:
'build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh',
reason: 'API Build 归档必须包含 Pingora realpath canary 关闭脚本。',
},
{
file: 'jenkins/Jenkinsfile.production-api-build',
includes: 'build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**',
@@ -5640,6 +5741,18 @@ const checks = [
'build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs',
reason: 'API Deploy 复制上游产物时必须包含 Pingora canary access log 对账脚本。',
},
{
file: 'jenkins/Jenkinsfile.production-api-deploy',
includes:
'build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh',
reason: 'API Deploy 复制上游产物时必须包含 Pingora realpath canary 启用脚本。',
},
{
file: 'jenkins/Jenkinsfile.production-api-deploy',
includes:
'build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh',
reason: 'API Deploy 复制上游产物时必须包含 Pingora realpath canary 关闭脚本。',
},
{
file: 'jenkins/Jenkinsfile.production-api-deploy',
includes:
@@ -0,0 +1,197 @@
#!/usr/bin/env bash
set -euo pipefail
TARGET_PATH="${GENARRATIVE_PINGORA_REALPATH_CANARY_TARGET_PATH:-/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf}"
NGINX_BINARY="${GENARRATIVE_NGINX_BINARY:-nginx}"
NGINX_SERVICE="${GENARRATIVE_NGINX_SERVICE:-nginx.service}"
SYSTEMCTL_BINARY="${GENARRATIVE_SYSTEMCTL_BINARY:-systemctl}"
APPLY="false"
STATUS_AFTER="true"
usage() {
cat <<'EOF'
用法:
scripts/deploy/pingora-realpath-canary-disable.sh [--apply] [--target-path <path>] [--nginx-binary <path|name>] [--nginx-service <unit>] [--systemctl-binary <path|name>] [--no-status]
说明:
关闭 Nginx -> Pingora 真实路径 canary:删除
/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf,执行 nginx -t
再 reload nginx。默认 dry-run,必须显式传 --apply 才会修改系统。
如果 nginx -t 或 reload 失败,脚本会恢复删除前的目标文件。
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help)
usage
exit 0
;;
--apply)
APPLY="true"
shift
;;
--target-path)
TARGET_PATH="${2:-}"
if [[ -z "${TARGET_PATH}" ]]; then
echo "[pingora-realpath-canary-disable] --target-path 缺少参数" >&2
exit 1
fi
shift 2
;;
--nginx-binary)
NGINX_BINARY="${2:-}"
if [[ -z "${NGINX_BINARY}" ]]; then
echo "[pingora-realpath-canary-disable] --nginx-binary 缺少参数" >&2
exit 1
fi
shift 2
;;
--nginx-service)
NGINX_SERVICE="${2:-}"
if [[ -z "${NGINX_SERVICE}" ]]; then
echo "[pingora-realpath-canary-disable] --nginx-service 缺少参数" >&2
exit 1
fi
shift 2
;;
--systemctl-binary)
SYSTEMCTL_BINARY="${2:-}"
if [[ -z "${SYSTEMCTL_BINARY}" ]]; then
echo "[pingora-realpath-canary-disable] --systemctl-binary 缺少参数" >&2
exit 1
fi
shift 2
;;
--no-status)
STATUS_AFTER="false"
shift
;;
*)
echo "[pingora-realpath-canary-disable] 未知参数: $1" >&2
usage >&2
exit 1
;;
esac
done
reject_control_characters() {
local label="$1"
local value="$2"
if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then
echo "[pingora-realpath-canary-disable] ${label} 不能包含换行或 NUL 字符。" >&2
exit 1
fi
}
is_filesystem_root_path() {
local value="$1"
local without_slashes="${value//\//}"
[[ -n "${value}" && -z "${without_slashes}" ]]
}
require_absolute_path() {
local label="$1"
local value="$2"
if [[ "${value}" != /* ]]; then
echo "[pingora-realpath-canary-disable] ${label} 必须是绝对路径。" >&2
exit 1
fi
if is_filesystem_root_path "${value}"; then
echo "[pingora-realpath-canary-disable] ${label} 不能是文件系统根目录。" >&2
exit 1
fi
}
validate_command_or_absolute_path() {
local label="$1"
local value="$2"
if [[ "${value}" == */* ]]; then
require_absolute_path "${label}" "${value}"
fi
}
print_command() {
printf '+'
for arg in "$@"; do
printf ' %q' "${arg}"
done
printf '\n'
}
run_or_print() {
if [[ "${APPLY}" == "true" ]]; then
"$@"
else
print_command "$@"
fi
}
restore_previous() {
local backup_path="$1"
cp -p -- "${backup_path}" "${TARGET_PATH}"
}
rollback_after_failure() {
local backup_path="$1"
local reason="$2"
echo "[pingora-realpath-canary-disable] ${reason},恢复删除前配置。" >&2
restore_previous "${backup_path}"
"${NGINX_BINARY}" -t >/dev/null 2>&1 || true
"${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}" >/dev/null 2>&1 || true
}
reject_control_characters "--target-path" "${TARGET_PATH}"
reject_control_characters "--nginx-binary" "${NGINX_BINARY}"
reject_control_characters "--nginx-service" "${NGINX_SERVICE}"
reject_control_characters "--systemctl-binary" "${SYSTEMCTL_BINARY}"
require_absolute_path "--target-path" "${TARGET_PATH}"
validate_command_or_absolute_path "--nginx-binary" "${NGINX_BINARY}"
validate_command_or_absolute_path "--systemctl-binary" "${SYSTEMCTL_BINARY}"
TARGET_DIR="$(dirname -- "${TARGET_PATH}")"
if [[ "${APPLY}" == "true" && ( -L "${TARGET_DIR}" || -L "${TARGET_PATH}" ) ]]; then
echo "[pingora-realpath-canary-disable] 目标目录或目标文件不能是符号链接。" >&2
exit 1
fi
if [[ "${APPLY}" != "true" ]]; then
echo "[pingora-realpath-canary-disable] dry-run:将删除 realpath canary ${TARGET_PATH}"
print_command rm -f -- "${TARGET_PATH}"
run_or_print "${NGINX_BINARY}" -t
run_or_print "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}"
exit 0
fi
if [[ ! -e "${TARGET_PATH}" ]]; then
echo "[pingora-realpath-canary-disable] 目标文件不存在,视为已关闭: ${TARGET_PATH}"
if [[ "${STATUS_AFTER}" == "true" ]]; then
"${SYSTEMCTL_BINARY}" is-active "${NGINX_SERVICE}"
fi
exit 0
fi
if [[ ! -f "${TARGET_PATH}" ]]; then
echo "[pingora-realpath-canary-disable] 目标存在但不是普通文件: ${TARGET_PATH}" >&2
exit 1
fi
backup_path="$(mktemp "${TARGET_DIR}/.genarrative-pingora-realpath-canary.disable.backup.XXXXXX")"
trap 'rm -f -- "${backup_path}"' EXIT
cp -p -- "${TARGET_PATH}" "${backup_path}"
rm -f -- "${TARGET_PATH}"
if ! "${NGINX_BINARY}" -t; then
rollback_after_failure "${backup_path}" "nginx -t 失败"
exit 1
fi
if ! "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}"; then
rollback_after_failure "${backup_path}" "reload ${NGINX_SERVICE} 失败"
exit 1
fi
if [[ "${STATUS_AFTER}" == "true" ]]; then
"${SYSTEMCTL_BINARY}" is-active "${NGINX_SERVICE}"
fi
echo "[pingora-realpath-canary-disable] 已关闭 realpath canary: ${TARGET_PATH}"
@@ -0,0 +1,350 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
REPO_ROOT="$(cd -- "${SCRIPT_DIR}/../.." >/dev/null 2>&1 && pwd)"
TEMPLATE_PATH="${GENARRATIVE_PINGORA_REALPATH_CANARY_TEMPLATE_PATH:-${REPO_ROOT}/deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf}"
TARGET_PATH="${GENARRATIVE_PINGORA_REALPATH_CANARY_TARGET_PATH:-/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf}"
PROBE_TOKEN="${GENARRATIVE_PINGORA_REALPATH_CANARY_PROBE_TOKEN:-${GENARRATIVE_PINGORA_GATEWAY_PROBE_TOKEN:-}}"
BASE_URL="${GENARRATIVE_PINGORA_REALPATH_CANARY_BASE_URL:-http://127.0.0.1:18083}"
HOST_HEADER="${GENARRATIVE_PINGORA_REALPATH_CANARY_HOST:-}"
NGINX_BINARY="${GENARRATIVE_NGINX_BINARY:-nginx}"
NGINX_SERVICE="${GENARRATIVE_NGINX_SERVICE:-nginx.service}"
SYSTEMCTL_BINARY="${GENARRATIVE_SYSTEMCTL_BINARY:-systemctl}"
LIVE_SCRIPT="${GENARRATIVE_PINGORA_REALPATH_CANARY_LIVE_SCRIPT:-${REPO_ROOT}/scripts/check-pingora-canary-live.mjs}"
APPLY="false"
RUN_LIVE="true"
STATUS_AFTER="true"
PLACEHOLDER="__GENARRATIVE_PINGORA_PROBE_TOKEN__"
usage() {
cat <<'EOF'
用法:
scripts/deploy/pingora-realpath-canary-enable.sh [--apply] [--probe-token <token>] [--host <host>] [--base-url <url>] [--template-path <path>] [--target-path <path>] [--nginx-binary <path|name>] [--nginx-service <unit>] [--systemctl-binary <path|name>] [--live-script <path>] [--skip-live] [--no-status]
说明:
启用 Nginx -> Pingora 真实路径 canary。脚本会把随包
deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf 渲染到
/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf。
默认是 dry-run,只打印计划;必须显式传 --apply 才会写 Nginx 配置。
--apply 会替换 probe token、写入目标文件、执行 nginx -t、reload nginx
然后运行 realpath canary live smoke。任一阶段失败都会恢复写入前的目标文件状态。
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help)
usage
exit 0
;;
--apply)
APPLY="true"
shift
;;
--probe-token)
PROBE_TOKEN="${2:-}"
if [[ -z "${PROBE_TOKEN}" ]]; then
echo "[pingora-realpath-canary-enable] --probe-token 缺少参数" >&2
exit 1
fi
shift 2
;;
--host)
HOST_HEADER="${2:-}"
if [[ -z "${HOST_HEADER}" ]]; then
echo "[pingora-realpath-canary-enable] --host 缺少参数" >&2
exit 1
fi
shift 2
;;
--base-url)
BASE_URL="${2:-}"
if [[ -z "${BASE_URL}" ]]; then
echo "[pingora-realpath-canary-enable] --base-url 缺少参数" >&2
exit 1
fi
shift 2
;;
--template-path)
TEMPLATE_PATH="${2:-}"
if [[ -z "${TEMPLATE_PATH}" ]]; then
echo "[pingora-realpath-canary-enable] --template-path 缺少参数" >&2
exit 1
fi
shift 2
;;
--target-path)
TARGET_PATH="${2:-}"
if [[ -z "${TARGET_PATH}" ]]; then
echo "[pingora-realpath-canary-enable] --target-path 缺少参数" >&2
exit 1
fi
shift 2
;;
--nginx-binary)
NGINX_BINARY="${2:-}"
if [[ -z "${NGINX_BINARY}" ]]; then
echo "[pingora-realpath-canary-enable] --nginx-binary 缺少参数" >&2
exit 1
fi
shift 2
;;
--nginx-service)
NGINX_SERVICE="${2:-}"
if [[ -z "${NGINX_SERVICE}" ]]; then
echo "[pingora-realpath-canary-enable] --nginx-service 缺少参数" >&2
exit 1
fi
shift 2
;;
--systemctl-binary)
SYSTEMCTL_BINARY="${2:-}"
if [[ -z "${SYSTEMCTL_BINARY}" ]]; then
echo "[pingora-realpath-canary-enable] --systemctl-binary 缺少参数" >&2
exit 1
fi
shift 2
;;
--live-script)
LIVE_SCRIPT="${2:-}"
if [[ -z "${LIVE_SCRIPT}" ]]; then
echo "[pingora-realpath-canary-enable] --live-script 缺少参数" >&2
exit 1
fi
shift 2
;;
--skip-live)
RUN_LIVE="false"
shift
;;
--no-status)
STATUS_AFTER="false"
shift
;;
*)
echo "[pingora-realpath-canary-enable] 未知参数: $1" >&2
usage >&2
exit 1
;;
esac
done
reject_control_characters() {
local label="$1"
local value="$2"
if [[ "${value}" == *$'\n'* || "${value}" == *$'\r'* ]]; then
echo "[pingora-realpath-canary-enable] ${label} 不能包含换行或 NUL 字符。" >&2
exit 1
fi
}
is_filesystem_root_path() {
local value="$1"
local without_slashes="${value//\//}"
[[ -n "${value}" && -z "${without_slashes}" ]]
}
require_absolute_path() {
local label="$1"
local value="$2"
if [[ "${value}" != /* ]]; then
echo "[pingora-realpath-canary-enable] ${label} 必须是绝对路径。" >&2
exit 1
fi
if is_filesystem_root_path "${value}"; then
echo "[pingora-realpath-canary-enable] ${label} 不能是文件系统根目录。" >&2
exit 1
fi
}
validate_command_or_absolute_path() {
local label="$1"
local value="$2"
if [[ "${value}" == */* ]]; then
require_absolute_path "${label}" "${value}"
fi
}
validate_host() {
local label="$1"
local value="$2"
if [[ -z "${value}" ]]; then
echo "[pingora-realpath-canary-enable] ${label} 必须提供。" >&2
exit 1
fi
if [[ "${value}" == http://* || "${value}" == https://* || "${value}" == */* || "${value}" == *\?* || "${value}" =~ [[:space:]] ]]; then
echo "[pingora-realpath-canary-enable] ${label} 必须是 Host 或 host:port,不能是 URL、路径、查询或包含空白。" >&2
exit 1
fi
}
validate_base_url() {
if [[ "${BASE_URL}" != http://* && "${BASE_URL}" != https://* ]]; then
echo "[pingora-realpath-canary-enable] --base-url 必须以 http:// 或 https:// 开头。" >&2
exit 1
fi
}
validate_probe_token() {
if [[ "${APPLY}" != "true" ]]; then
return
fi
if [[ -z "${PROBE_TOKEN}" ]]; then
echo "[pingora-realpath-canary-enable] --apply 必须提供 --probe-token 或 GENARRATIVE_PINGORA_REALPATH_CANARY_PROBE_TOKEN。" >&2
exit 1
fi
if [[ "${PROBE_TOKEN}" == "${PLACEHOLDER}" || "${#PROBE_TOKEN}" -lt 8 ]]; then
echo "[pingora-realpath-canary-enable] probe token 不能是占位值或过短。" >&2
exit 1
fi
}
escape_nginx_double_quoted() {
local value="$1"
value="${value//\\/\\\\}"
value="${value//\"/\\\"}"
printf '%s' "${value}"
}
render_template_to() {
local output_path="$1"
local content escaped_token
content="$(<"${TEMPLATE_PATH}")"
escaped_token="$(escape_nginx_double_quoted "${PROBE_TOKEN}")"
printf '%s' "${content//${PLACEHOLDER}/${escaped_token}}" >"${output_path}"
}
print_command() {
printf '+'
for arg in "$@"; do
printf ' %q' "${arg}"
done
printf '\n'
}
run_or_print() {
if [[ "${APPLY}" == "true" ]]; then
"$@"
else
print_command "$@"
fi
}
restore_previous() {
local backup_path="$1"
local had_target="$2"
if [[ "${had_target}" == "true" ]]; then
cp -p -- "${backup_path}" "${TARGET_PATH}"
else
rm -f -- "${TARGET_PATH}"
fi
}
rollback_after_failure() {
local backup_path="$1"
local had_target="$2"
local reason="$3"
echo "[pingora-realpath-canary-enable] ${reason},恢复写入前配置。" >&2
restore_previous "${backup_path}" "${had_target}"
"${NGINX_BINARY}" -t >/dev/null 2>&1 || true
"${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}" >/dev/null 2>&1 || true
}
reject_control_characters "--template-path" "${TEMPLATE_PATH}"
reject_control_characters "--target-path" "${TARGET_PATH}"
reject_control_characters "--probe-token" "${PROBE_TOKEN}"
reject_control_characters "--base-url" "${BASE_URL}"
reject_control_characters "--host" "${HOST_HEADER}"
reject_control_characters "--nginx-binary" "${NGINX_BINARY}"
reject_control_characters "--nginx-service" "${NGINX_SERVICE}"
reject_control_characters "--systemctl-binary" "${SYSTEMCTL_BINARY}"
reject_control_characters "--live-script" "${LIVE_SCRIPT}"
require_absolute_path "--template-path" "${TEMPLATE_PATH}"
require_absolute_path "--target-path" "${TARGET_PATH}"
if [[ "${RUN_LIVE}" == "true" ]]; then
require_absolute_path "--live-script" "${LIVE_SCRIPT}"
fi
validate_command_or_absolute_path "--nginx-binary" "${NGINX_BINARY}"
validate_command_or_absolute_path "--systemctl-binary" "${SYSTEMCTL_BINARY}"
validate_host "--host" "${HOST_HEADER}"
validate_base_url
validate_probe_token
if [[ ! -f "${TEMPLATE_PATH}" ]]; then
echo "[pingora-realpath-canary-enable] 未找到 realpath canary 模板: ${TEMPLATE_PATH}" >&2
exit 1
fi
if [[ "${RUN_LIVE}" == "true" && ! -f "${LIVE_SCRIPT}" ]]; then
echo "[pingora-realpath-canary-enable] 未找到 realpath canary live smoke 脚本: ${LIVE_SCRIPT}" >&2
exit 1
fi
TARGET_DIR="$(dirname -- "${TARGET_PATH}")"
if [[ "${APPLY}" == "true" ]]; then
if [[ ! -d "${TARGET_DIR}" ]]; then
echo "[pingora-realpath-canary-enable] 目标目录不存在: ${TARGET_DIR}" >&2
exit 1
fi
if [[ -L "${TARGET_DIR}" || -L "${TARGET_PATH}" ]]; then
echo "[pingora-realpath-canary-enable] 目标目录或目标文件不能是符号链接。" >&2
exit 1
fi
fi
if [[ "${APPLY}" != "true" ]]; then
echo "[pingora-realpath-canary-enable] dry-run:将渲染 realpath canary 到 ${TARGET_PATH}probe token 输出已隐藏。"
print_command install -m 0644 "<rendered-realpath-canary>" "${TARGET_PATH}"
run_or_print "${NGINX_BINARY}" -t
run_or_print "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}"
if [[ "${RUN_LIVE}" == "true" ]]; then
run_or_print node -- "${LIVE_SCRIPT}" --realpath --base-url "${BASE_URL}" --host "${HOST_HEADER}"
fi
exit 0
fi
tmp_path="$(mktemp "${TARGET_DIR}/.genarrative-pingora-realpath-canary.XXXXXX")"
backup_path="$(mktemp "${TARGET_DIR}/.genarrative-pingora-realpath-canary.backup.XXXXXX")"
had_target="false"
cleanup_paths=("${tmp_path}" "${backup_path}")
trap 'rm -f -- "${cleanup_paths[@]}"' EXIT
render_template_to "${tmp_path}"
chmod 0644 "${tmp_path}"
if [[ -e "${TARGET_PATH}" ]]; then
if [[ ! -f "${TARGET_PATH}" ]]; then
echo "[pingora-realpath-canary-enable] 目标已存在但不是普通文件: ${TARGET_PATH}" >&2
exit 1
fi
cp -p -- "${TARGET_PATH}" "${backup_path}"
had_target="true"
fi
install -m 0644 "${tmp_path}" "${TARGET_PATH}"
if ! "${NGINX_BINARY}" -t; then
rollback_after_failure "${backup_path}" "${had_target}" "nginx -t 失败"
exit 1
fi
if ! "${SYSTEMCTL_BINARY}" reload "${NGINX_SERVICE}"; then
rollback_after_failure "${backup_path}" "${had_target}" "reload ${NGINX_SERVICE} 失败"
exit 1
fi
if [[ "${RUN_LIVE}" == "true" ]]; then
if ! node -- "${LIVE_SCRIPT}" --realpath --base-url "${BASE_URL}" --host "${HOST_HEADER}"; then
rollback_after_failure "${backup_path}" "${had_target}" "realpath canary live smoke 失败"
exit 1
fi
fi
if [[ "${STATUS_AFTER}" == "true" ]]; then
"${SYSTEMCTL_BINARY}" is-active "${NGINX_SERVICE}"
fi
echo "[pingora-realpath-canary-enable] 已启用 realpath canary: ${TARGET_PATH}"
+14
View File
@@ -619,6 +619,8 @@ HEALTH_PATROL_ENV_CHECK_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-production-he
PINGORA_RELEASE_READINESS_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-release-readiness.mjs"
PINGORA_ENABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-enable.sh"
PINGORA_ROLLBACK_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-direct-rollback.sh"
PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh"
PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh"
PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs"
PINGORA_TLS_CERT_SYNC_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/deploy/pingora-tls-cert-sync.mjs"
PINGORA_PREFLIGHT_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/check-pingora-direct-preflight.mjs"
@@ -702,6 +704,18 @@ if [[ ! -f "${PINGORA_ROLLBACK_SCRIPT_SOURCE}" ]]; then
fi
cp "${PINGORA_ROLLBACK_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-direct-rollback.sh"
chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-direct-rollback.sh"
if [[ ! -f "${PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE}" ]]; then
echo "[production-api-deploy] 发布产物缺少 Pingora realpath canary 启用脚本: ${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh" >&2
exit 1
fi
cp "${PINGORA_REALPATH_CANARY_ENABLE_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh"
chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-enable.sh"
if [[ ! -f "${PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE}" ]]; then
echo "[production-api-deploy] 发布产物缺少 Pingora realpath canary 关闭脚本: ${SOURCE_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh" >&2
exit 1
fi
cp "${PINGORA_REALPATH_CANARY_DISABLE_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh"
chmod 0755 "${RELEASE_CONTENT_DIR}/scripts/deploy/pingora-realpath-canary-disable.sh"
if [[ ! -f "${PINGORA_HEALTH_PATROL_ENV_SWITCH_SCRIPT_SOURCE}" ]]; then
echo "[production-api-deploy] 发布产物缺少 Pingora health patrol env 切换脚本: ${SOURCE_DIR}/scripts/deploy/pingora-health-patrol-env-switch.mjs" >&2
exit 1
@@ -21,6 +21,8 @@ const REQUIRED_ARTIFACTS = [
{ path: 'scripts/check-pingora-canary-access-log-parity.mjs' },
{ path: 'scripts/deploy/pingora-direct-enable.sh', executable: true },
{ path: 'scripts/deploy/pingora-direct-rollback.sh', executable: true },
{ path: 'scripts/deploy/pingora-realpath-canary-enable.sh', executable: true },
{ path: 'scripts/deploy/pingora-realpath-canary-disable.sh', executable: true },
{ path: 'scripts/deploy/pingora-health-patrol-env-switch.mjs', executable: true },
{ path: 'scripts/deploy/pingora-tls-cert-sync.mjs', executable: true },
{ path: 'deploy/systemd/genarrative-pingora-gateway.service' },
@@ -34,6 +34,8 @@ const RELEASE_ARTIFACTS = [
'scripts/check-pingora-canary-access-log-parity.mjs',
'scripts/deploy/pingora-direct-enable.sh',
'scripts/deploy/pingora-direct-rollback.sh',
'scripts/deploy/pingora-realpath-canary-enable.sh',
'scripts/deploy/pingora-realpath-canary-disable.sh',
'scripts/deploy/pingora-health-patrol-env-switch.mjs',
'deploy/systemd/genarrative-pingora-gateway.service',
'deploy/systemd/genarrative-pingora-gateway-direct-entry.conf',