diff --git a/deploy/nginx/README.md b/deploy/nginx/README.md index 55b572897..4a1bd5700 100644 --- a/deploy/nginx/README.md +++ b/deploy/nginx/README.md @@ -69,6 +69,7 @@ - 启用前必须运行 `npm run check:nginx-pingora-canary`；目标 agent 有 Nginx 时运行 `node scripts/check-nginx-pingora-canary.mjs --require-nginx`，同时做两个 snippet 的静态护栏和真实 `nginx -t`。本机或 CI 可运行 `npm run check:pingora-canary-docker`，用 Docker Nginx、真实 Pingora 和 mock 上游复现前缀 canary 与真实路径 canary 两条 handoff 链路。 - 前缀 canary reload 后运行 `GENARRATIVE_PINGORA_CANARY_BASE_URL=http://127.0.0.1 GENARRATIVE_PINGORA_CANARY_HOST=<域名> npm run check:pingora-canary-live`，再用 `scripts/check-pingora-canary-access-log-parity.mjs` 对账 `/var/log/nginx/genarrative.access.log` 与 `/var/log/genarrative/pingora-gateway.access.log`。 - 真实路径 canary 不再手工编辑 `/etc/nginx/conf.d/`。目标机从 current release 执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083`；脚本会渲染到 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，保持晚于主站 `log_format genarrative_upstream` 加载，执行 `nginx -t`、reload Nginx，并默认运行 realpath live smoke。`nginx -t`、reload 或 live smoke 任一失败都会恢复写入前配置。关闭时执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`，脚本会删除该 `zz-...` 文件，执行 `nginx -t` 和 reload，失败时恢复删除前配置。 +- `npm run check:pingora-release-readiness` 默认包含 `npm run check:pingora-realpath-canary-toggle`，用临时 Nginx 目录和 fake `nginx` / `systemctl` / live smoke 验证 realpath canary 启停脚本默认 dry-run、apply 写入、失败回滚和 disable 恢复逻辑；不再把这条脚本护栏留成单独可选项。 - 真实路径 canary reload 后可再次运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log --path /__genarrative_pingora_realpath_canary/healthz --path /api/creation-entry/config --path /v1/identity --path /assets/app.js`。真实路径模式除 healthz 探针外要求 Nginx path 与 Pingora path 完全一致。 - 正式切换前在源码 checkout / CI / 构建环境跑 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；启用真实路径 canary 时追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。 - 目标机 `/opt/genarrative/current` 只能跑随包 runtime-only 门禁：`/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；验证真实路径 canary 或 Pingora 直连入口时分别追加 `--require-realpath-live ...` 或 `--require-direct ...` 参数。未带 `--require-direct` 的 runtime-only 基础门禁会自动执行直连彩排状态复核，确认公网 `80/443` 仍由 Nginx 接流、Pingora shadow `127.0.0.1:18081`、realpath canary `127.0.0.1:18083` 和 current release 自审均通过；启用后 `--require-direct` 复核不再要求 Nginx 接公网彩排状态。生成正式直连切换 runbook 时，`--direct-redirect-host` 和回退 smoke Host 必须与 `--direct-host` 同 hostname，只允许端口不同。 diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 35f2461da..4591282bc 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -92,9 +92,9 @@ - 背景：Pingora 影子网关已覆盖核心 Nginx 路由 parity，但只靠静态 snippet 检查和目标机手工 live canary，无法在本机 / CI 中复现真实 Nginx -> Pingora handoff 链路。 - 决策：新增 `npm run check:pingora-canary-docker` 作为正式切换前的本机容器验收：脚本启动 Docker Nginx、真实 `pingora-gateway`、mock `api-server` 和 mock SpacetimeDB，渲染同一份 `deploy/nginx/snippets/genarrative-pingora-canary.conf` 后复用 live canary 断言。Docker Nginx 必须写入生产同口径 access log，并在 live smoke 后复用 `scripts/check-pingora-canary-access-log-parity.mjs` 按同一 `request_id` 对账 canary healthz、代表性 API、SpacetimeDB identity 和静态资源路径，避免本机 / CI 只验证 handoff 响应头。默认 Docker 或镜像缺失时跳过，CI / 目标 agent 用 `node scripts/check-pingora-canary-docker.mjs --require-docker --pull` 强制执行；目标机人工 include 后仍必须跑 `npm run check:pingora-canary-live`。同时新增 `npm run check:pingora-release-readiness` 作为正式切换聚合门禁，默认适合本机提交前检查；切换窗口必须用 `--require-docker --pull-docker --require-nginx --require-live` 强制 Docker handoff、目标机 `nginx -t` 和 live canary 全部通过，且 `--require-live` 必须显式提供 `--live-host`，避免只打到 Nginx 默认 vhost。 -- 真实路径补充：前缀 canary 通过后、Pingora direct 直连前，使用 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 做真实路径 canary。该 snippet 必须作为独立本机 `server` include 到 Nginx `http` 上下文，默认监听 `127.0.0.1:18083` 并写独立 `genarrative-pingora-realpath-canary.access.log`，不能 include 到生产 `443` server 内覆盖正式 location。`check-pingora-canary-live.mjs --realpath` 和 `check-pingora-canary-access-log-parity.mjs --realpath` 负责验证真实 `/api`、`/v1`、`/assets` 路径；release readiness 用 `--require-realpath-live` 把该阶段纳入目标机 runtime-only 门禁。 +- 真实路径补充：前缀 canary 通过后、Pingora direct 直连前，使用 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 做真实路径 canary。该 snippet 必须作为独立本机 `server` include 到 Nginx `http` 上下文，默认监听 `127.0.0.1:18083` 并写独立 `genarrative-pingora-realpath-canary.access.log`，不能 include 到生产 `443` server 内覆盖正式 location。`check-pingora-canary-live.mjs --realpath` 和 `check-pingora-canary-access-log-parity.mjs --realpath` 负责验证真实 `/api`、`/v1`、`/assets` 路径；release readiness 默认执行 `check:pingora-realpath-canary-toggle`，验证 realpath canary 启停脚本 dry-run、apply、失败回滚和 disable 恢复逻辑，目标机 runtime-only 用 `--require-realpath-live` 把已启用真实路径 canary 纳入门禁。 - 影响范围：`scripts/check-pingora-canary-docker.mjs`、`scripts/check-pingora-release-readiness.mjs`、`package.json`、生产运维护栏、Nginx README、Pingora 试点文档和生产运维文档。 -- 验证方式：`node --check scripts/check-pingora-canary-docker.mjs scripts/check-pingora-release-readiness.mjs`、`npm run check:pingora-canary-docker`、`npm run check:pingora-release-readiness`、`npm run check:nginx-pingora-canary`、`npm run check:production-ops`；有 Docker 镜像或允许拉取时追加 `node scripts/check-pingora-canary-docker.mjs --require-docker --pull`，目标机切换窗口追加 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名>`。 +- 验证方式：`node --check scripts/check-pingora-canary-docker.mjs scripts/check-pingora-release-readiness.mjs scripts/check-pingora-release-readiness-plan.mjs scripts/check-pingora-realpath-canary-toggle.mjs`、`npm run check:pingora-realpath-canary-toggle`、`npm run check:pingora-canary-docker`、`npm run check:pingora-release-readiness`、`npm run check:nginx-pingora-canary`、`npm run check:production-ops`；有 Docker 镜像或允许拉取时追加 `node scripts/check-pingora-canary-docker.mjs --require-docker --pull`，目标机切换窗口追加 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名>`。 - 关联文档：`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`、`deploy/nginx/README.md`。 ## 2026-06-15 SpacetimeDB 本地 skills 只保留 CLI / Concepts / Rust diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index df4a1823d..f6d061cc4 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -131,7 +131,7 @@ - 现象：目标机把 `genarrative-pingora-realpath-canary.conf` 放进 `/etc/nginx/conf.d/` 后，`nginx -t` 失败并报 `unknown log format "genarrative_upstream"`。 - 原因：真实路径 canary 是独立 `server` 片段，并使用 `access_log /var/log/nginx/genarrative-pingora-realpath-canary.access.log genarrative_upstream;`。Nginx 会按文件名顺序加载 `conf.d`；如果 canary 文件名早于定义 `log_format genarrative_upstream` 的主站配置，access log 行会先被解析而找不到格式。 - 处理：真实路径 canary 启停统一用 current release 随包脚本，不再手工写 `/etc/nginx/conf.d/`。启用执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083`，脚本固定写入晚于主站配置加载的 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，并在 `nginx -t`、reload 或 live smoke 失败时恢复写入前配置。关闭执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`，脚本在 `nginx -t` 或 reload 失败时恢复删除前配置。另一种长期做法是把 `log_format` 放到所有 `conf.d` server 之前的全局 Nginx 配置。检查配置时不要把 probe token 原文写入记录。 -- 验证：启用脚本通过后，再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`、`node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log ...` 和 `node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...`。若只启用了真实路径 canary，不要同时传 `--require-live`，否则前缀 canary 未启用时会按正式 Nginx HTTP 入口返回 301。 +- 验证：提交前运行 `npm run check:pingora-realpath-canary-toggle` 或默认聚合门禁 `npm run check:pingora-release-readiness`，确认启停脚本的 dry-run、apply、失败回滚和 disable 恢复逻辑仍被覆盖。启用脚本通过后，再运行 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`、`node -- /opt/genarrative/current/scripts/check-pingora-canary-access-log-parity.mjs --realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log --pingora-log-file /var/log/genarrative/pingora-gateway.access.log ...` 和 `node -- /opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-realpath-live ...`。若只启用了真实路径 canary，不要同时传 `--require-live`，否则前缀 canary 未启用时会按正式 Nginx HTTP 入口返回 301。 - 关联：`deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf`、`deploy/nginx/README.md`、`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`、`scripts/check-pingora-release-readiness.mjs`。 ## Pingora release readiness 脚本不能只存在于源码 checkout diff --git a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md index 8e8476b0a..e4c8d7940 100644 --- a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md +++ b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md @@ -117,7 +117,7 @@ node -- /opt/genarrative/current/scripts/ops/pingora-direct-rehearsal-status.mjs `--warn-only`、`--allow-extra-files` 和 `--allow-extra-root-entries` 只允许值班人员在正式 runbook 之外作为独立排障命令使用；`plan:pingora-direct-cutover` 生成的正式切换计划不得携带这些放行参数。若需要放行额外证据条目或忽略 `CRITICAL`，应先在证据根目录外保存人工说明、修正现场或重新生成正式证据，而不是把排障开关写进切换 runbook。 -`check:pingora-release-readiness` 是正式切换前的聚合门禁，默认串行执行 `cargo test -p pingora-gateway`、mock 上游 smoke、路由矩阵 parity、Nginx canary snippet 校验、Docker handoff smoke、canary access log 对账烟测、直连入口静态预检、直连启用 / 回退 dry-run 行为检查、current release 自审烟测、直连彩排状态烟测、release readiness 计划自检、生产运维护栏、Pingora cutover 状态快照烟测、Pingora cutover 证据包烟测、Pingora cutover 命令证据烟测、Pingora cutover 证据 manifest 验真烟测、Pingora cutover 证据根目录审计烟测、API release build 烟测、Pingora production release 真实构建烟测和 API deploy release 烟测。普通本机执行时 Docker / Nginx 能力仍按子脚本默认口径跳过；切换窗口或 CI 必须执行： +`check:pingora-release-readiness` 是正式切换前的聚合门禁，默认串行执行 `cargo test -p pingora-gateway`、mock 上游 smoke、路由矩阵 parity、Nginx canary snippet 校验、Docker handoff smoke、canary access log 对账烟测、realpath canary 启停烟测、直连入口静态预检、直连启用 / 回退 dry-run 行为检查、current release 自审烟测、直连彩排状态烟测、release readiness 计划自检、生产运维护栏、Pingora cutover 状态快照烟测、Pingora cutover 证据包烟测、Pingora cutover 命令证据烟测、Pingora cutover 证据 manifest 验真烟测、Pingora cutover 证据根目录审计烟测、API release build 烟测、Pingora production release 真实构建烟测和 API deploy release 烟测。普通本机执行时 Docker / Nginx 能力仍按子脚本默认口径跳过；切换窗口或 CI 必须执行： 目标机 current release 上的启用前基础门禁和启用后 `--require-direct` 复核使用同一个聚合脚本，但必须追加 `--release-runtime-only`。该模式只执行发布包内可自包含的运行时复核：current release 自审、启用前直连彩排状态复核、live canary、真实 access log 对账、direct preflight、health patrol env 复核和 direct live smoke；不会运行 Cargo、npm、Docker 或 Nginx 源码 / 构建环境门禁，并会拒绝 `--require-docker`、`--pull-docker` 与 `--require-nginx`。未带 `--require-direct` 的启用前基础门禁会自动执行 `scripts/ops/pingora-direct-rehearsal-status.mjs --expect-public-gateway nginx --require-pingora-shadow --require-realpath-canary --require-current-release-gateway --fail-on-critical`，确认公网 `80/443` 仍由 Nginx 接流、Pingora shadow 与 realpath canary 高端口在线；启用后 `--require-direct` 复核不再要求 Nginx 接公网的彩排状态，改为检查 direct preflight、health patrol 直连模式和 direct live smoke。 diff --git a/scripts/check-pingora-release-readiness-plan.mjs b/scripts/check-pingora-release-readiness-plan.mjs index b7e434af2..adc5241f0 100644 --- a/scripts/check-pingora-release-readiness-plan.mjs +++ b/scripts/check-pingora-release-readiness-plan.mjs @@ -94,6 +94,7 @@ function main() { assertDefaultPlanIncludesPingoraCutoverEvidenceVerifySmoke(); assertDefaultPlanIncludesPingoraCutoverEvidenceAuditSmoke(); assertDefaultPlanIncludesCanaryAccessLogParitySmoke(); + assertDefaultPlanIncludesRealpathCanaryToggleSmoke(); assertDefaultPlanIncludesPingoraProductionReleaseBuildSmoke(); assertReleaseRuntimeOnlyPlanUsesCurrentReleaseScripts(); assertReleaseRuntimeOnlyPlanIncludesDirectRehearsalStatus(); @@ -366,6 +367,22 @@ function assertDefaultPlanIncludesCanaryAccessLogParitySmoke() { ); } +function assertDefaultPlanIncludesRealpathCanaryToggleSmoke() { + const plan = readPlan(['--dry-run-plan']); + const step = findStep(plan, 'Pingora realpath canary 启停烟测'); + if (!step) { + failures.push( + '默认 release readiness 计划必须包含 Pingora realpath canary 启停烟测。', + ); + return; + } + assertIncludes( + step.args, + 'check:pingora-realpath-canary-toggle', + 'Pingora realpath canary 启停烟测必须调用 check:pingora-realpath-canary-toggle。', + ); +} + function assertDefaultPlanIncludesProductionApiReleaseSmoke() { const plan = readPlan(['--dry-run-plan']); const step = findStep(plan, 'API release build 烟测'); diff --git a/scripts/check-pingora-release-readiness.mjs b/scripts/check-pingora-release-readiness.mjs index 126e95a18..28c7abc95 100644 --- a/scripts/check-pingora-release-readiness.mjs +++ b/scripts/check-pingora-release-readiness.mjs @@ -1601,6 +1601,11 @@ function buildSteps(config) { command: 'npm', args: ['run', 'check:pingora-canary-access-log-parity'], }, + { + name: 'Pingora realpath canary 启停烟测', + command: 'npm', + args: ['run', 'check:pingora-realpath-canary-toggle'], + }, { name: 'Pingora canary live 参数护栏', command: 'node', diff --git a/scripts/check-production-health-patrol.mjs b/scripts/check-production-health-patrol.mjs index abc2fe718..b61f12da5 100644 --- a/scripts/check-production-health-patrol.mjs +++ b/scripts/check-production-health-patrol.mjs @@ -306,6 +306,10 @@ async function prepareFixture(name) { [ '#!/usr/bin/env bash', `printf 'systemctl %s\\n' "$*" >> ${shellQuote(commandsLog)}`, + 'if [[ "$1" == "list-units" && "$2" == "genarrative-external-generation-worker@*.service" ]]; then', + ' echo "genarrative-external-generation-worker@1.service loaded active running Genarrative external generation worker 1"', + ' exit 0', + 'fi', 'echo active', 'exit 0', '',