diff --git a/deploy/nginx/README.md b/deploy/nginx/README.md index 4ab54ceb0..2e400add7 100644 --- a/deploy/nginx/README.md +++ b/deploy/nginx/README.md @@ -46,7 +46,7 @@ - Pingora 网关默认仍只监听本机 shadow 端口，不绑定公网 `80/443`。 - 需要评估直连入口时，显式配置 `GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN`、`GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` 和 `GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE`；网关只读取现有证书文件，不负责 Certbot / ACME 自动续期。 - 需要 HTTP 到 HTTPS 跳转时，再显式配置 `GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN`；该入口除 `/.well-known/acme-challenge/*` 外统一 301 到 HTTPS，ACME challenge 仍从 `GENARRATIVE_PINGORA_GATEWAY_ACME_ROOT` 精确读取。 -- API release 必须把 `api-server.sha256`、可选 `pingora-gateway.sha256`、`scripts/check-pingora-release-readiness.mjs`、`scripts/check-pingora-canary-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-production-health-patrol-env.mjs`、`scripts/deploy/pingora-health-patrol-env-switch.mjs`、`scripts/deploy/pingora-tls-cert-sync.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/` 一起复制到 `/opt/genarrative/current`；current release 自审、直连启用、health patrol env 切换、TLS 证书同步、canary live、access log 对账、cutover 状态快照和证据包脚本都从 current release 推导依赖路径，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。 +- API release 必须把 `api-server.sha256`、可选 `pingora-gateway.sha256`、`scripts/check-pingora-release-readiness.mjs`、`scripts/check-pingora-canary-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-direct-rehearsal-status.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-production-health-patrol-env.mjs`、`scripts/deploy/pingora-health-patrol-env-switch.mjs`、`scripts/deploy/pingora-tls-cert-sync.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/` 一起复制到 `/opt/genarrative/current`；current release 自审、直连彩排状态、直连启用、health patrol env 切换、TLS 证书同步、canary live、access log 对账、cutover 状态快照和证据包脚本都从 current release 推导依赖路径，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。 - 目标机如果使用 Certbot / Let’s Encrypt 证书，不要直接放宽 `/etc/letsencrypt/live` 或 `archive` 的目录 / 私钥权限。先用随包脚本把 root 可读的 live symlink 解析并复制到 Pingora 私有目录，例如 `sudo -n node -- /opt/genarrative/current/scripts/deploy/pingora-tls-cert-sync.mjs --apply --source-cert-file /etc/letsencrypt/live/<域名>/fullchain.pem --source-key-file /etc/letsencrypt/live/<域名>/privkey.pem --target-dir /etc/genarrative/pingora-tls/<域名>`；脚本默认 dry-run，`--apply` 才原子写入 `fullchain.pem` / `privkey.pem`，目标目录默认 `root:genarrative 0750`，文件默认 `root:genarrative 0640`，并复核 `genarrative` 服务用户可读。随后把 `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` / `TLS_KEY_FILE` 指向 `/etc/genarrative/pingora-tls/<域名>/` 下的副本。 - 默认 systemd service 以 `genarrative` 非 root 用户运行，不具备绑定 `80/443` 的能力。Server-Provision 只安装 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 作为人工启用模板；切直连低端口时必须先确认 `genarrative` 可读证书文件、Nginx 已释放 `80/443`，再执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --apply --preflight-env-file /etc/genarrative/pingora-gateway.env --preflight-check-cert-readable --preflight-check-service-env-file --preflight-check-service-user-cert-readable --preflight-check-service-binary-executable --preflight-check-ports-free --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；脚本会先执行随包 current release 自审，确认 `pingora-gateway` 可执行且 systemd `ExecStart` 指向 current release，失败时不会安装 drop-in。自审通过后才安装 drop-in、`systemctl daemon-reload` 并重启 Pingora；脚本会用 `systemctl cat` 确认 `CAP_NET_BIND_SERVICE` drop-in 已进入 systemd 最终配置，用 `systemctl show ... ExecStart` 确认最终 service 指向随包主 service 模板中的 current release `pingora-gateway`，用 `systemctl is-active` 确认 Pingora active，并执行 direct live smoke，强制覆盖 HTTPS / HTTP redirect / ACME / WSS 101 和 Pingora access log request_id 落盘。 - `npm run check:pingora-gateway-smoke` 会用临时自签证书覆盖 TLS 直连、HTTP redirect 和 WSS subscribe 行为；正式公网切换前仍必须先走 Nginx canary 和 release readiness 门禁。 diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index c62b18dbd..0a6cf7756 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -2331,8 +2331,9 @@ - 决策补充：生产健康巡检的公网入口模式必须显式区分 `nginx` 和 `pingora-direct`。默认 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=nginx` 检查 API、SpacetimeDB 和 Nginx；Pingora 直连接管公网后切到 `pingora-direct`，改为检查 API、SpacetimeDB 和 `genarrative-pingora-gateway.service`，不再要求 `nginx.service` active。目标机本机探测 `127.0.0.1` 时用 `GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST=<域名>` 保留正式 Host / vhost 语义。 - 决策补充：Pingora 影子网关产物不进入默认 API release；只有显式传 `--include-pingora-gateway` 或在 `Genarrative-Api-Build` 勾选 `INCLUDE_PINGORA_GATEWAY` 时，才构建并打包 `pingora-gateway` / `pingora-gateway.sha256`。真实构建 Pingora 前必须先检查 `cmake`、C 编译器和 C++ 编译器；Jenkins 勾选 `INCLUDE_PINGORA_GATEWAY` 时也要先 fail-fast 检查这些工具，避免进入 Cargo 后才因 `libz-ng-sys` 构建依赖缺失失败。`production-api-deploy.sh` 仅在两者同时存在时校验并复制到 current release，避免现有 API 流水线被 Pingora 构建依赖影响。若发布包包含 Pingora 且 `genarrative-pingora-gateway.service` 已经处于 `active`，API deploy 在 `current` 链接切换后执行 `systemctl try-restart genarrative-pingora-gateway.service` 并复核仍为 `active`，让已启用的 shadow / canary / direct 机器加载同一份 current release 网关二进制；若服务未运行则只跳过，不主动拉起 Pingora。`npm run check:production-api-release` 必须同时验证默认 API release 不登记 Pingora，以及显式 `--include-pingora-gateway --skip-pingora-gateway-build` 时发布包包含 `pingora-gateway`、`pingora-gateway.sha256` 并写入 manifest。 - 决策补充：生产健康巡检的显式 `--timeout-ms`、`--slow-ms`、`GENARRATIVE_HEALTH_PATROL_TIMEOUT_MS` 和 `GENARRATIVE_HEALTH_PATROL_SLOW_MS` 必须是正整数，非法值直接失败，不静默回退默认 `5000ms` / `3000ms`。Pingora canary live 的 `--timeout-ms` / `GENARRATIVE_PINGORA_CANARY_TIMEOUT_MS`、direct live 的 `--timeout-ms` / `GENARRATIVE_PINGORA_DIRECT_TIMEOUT_MS`、canary access log 对账的 `--since-lines` / `GENARRATIVE_PINGORA_CANARY_ACCESS_LOG_SINCE_LINES` 也必须正整数。Pingora direct live 和 release readiness 读取的直连布尔 env 必须严格解析，只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败，避免 `REQUIRE_WSS_UPGRADE`、preflight 开关或 `SKIP_WSS` 因拼写错误被当成 false。canary live 的 base URL、prefix、Host、额外 path 和 timeout 不能包含换行或 NUL；脚本必须在发起 canary 请求前失败，避免污染参数进入 URL、Host header 或 JSON 输出。canary access log 对账的日志路径、prefix、必需路径、tail 行数以及日志行中解析出的 URI / path 也不能包含换行或 NUL；脚本必须失败并给出对应参数或日志行诊断，不能把污染值写入 JSON 对账输出。direct live 的 HTTPS / HTTP base URL、Host、redirect Host、probe token、额外 path、SpacetimeDB 数据库名、access log 路径、timeout 和布尔 env 都不能包含换行或 NUL；脚本必须在发起 HTTPS / HTTP / WSS 请求前失败，避免污染参数进入请求头、URL、日志对账或 JSON 证据。Pingora 切换窗口调整巡检、live smoke、日志对账阈值或直连布尔开关时，把参数解析失败视为配置错误，而不是继续执行检查。 -- 决策补充：即使不打包 Pingora 二进制，API release 也必须随包携带 Pingora release readiness 聚合门禁、直连启用 / 回退 / preflight / live smoke / current release 自审脚本，以及 `deploy/systemd/`、`deploy/pingora/` 支撑配置；`pingora-direct-enable.sh` 和正式 cutover runbook 默认从 `/opt/genarrative/current` 推导这些路径，启用前 release readiness 基础门禁和启用后 `--require-direct` 复核也必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs`，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。API release 还必须携带 `build//scripts/deploy/production-api-deploy.sh` 和同目录 `maintenance-on.sh` / `maintenance-off.sh`；`Genarrative-Api-Deploy` 只能复制并执行 build 产物内的 deploy 脚本，禁止继续执行部署工作区根部脚本，避免 workspace 中的旧脚本掩盖发布包布局缺陷。`production-api-deploy.sh` 对数据库备份脚本、健康巡检脚本和 Pingora 直连依赖都执行 fail-fast，发布产物缺失时保持维护模式并停止部署，不再从部署工作区兜底复制；API deploy 必须要求 `--release-root`、`--current-link`、`--api-env-file` 使用绝对路径，且 `--version` 必须以数字或字母开头并拒绝点目录，再先写 `${RELEASE_ROOT}/.${VERSION}.staging.$`，全部复制完成后再用非合并语义提升为 `${RELEASE_ROOT}/${VERSION}`，并用固定替换语义切换 current 符号链接，同版本 release 已存在、提升前竞态出现或 current 路径不是符号链接时拒绝覆盖 / 合并，失败时清理 staging 且不留下正式 release；`npm run check:production-api-release` 与 `npm run check:production-api-deploy` 必须进入 `check:pingora-release-readiness` 聚合门禁，前者用临时 `CARGO_TARGET_DIR` 和假 `api-server` release binary 验证 `build-production-release.sh --component api-server --skip-api-build` 产物自包含，后者用临时 release 和 fake `systemctl` / `curl` 验证从发布产物内执行 deploy 脚本后 current release 自包含，并覆盖缺少备份脚本、健康巡检脚本、release readiness 聚合门禁脚本、current release 自审脚本、direct live smoke 脚本、相对 release root / current link / api env file、点目录或点开头 version、既有 release 目录、目录型 current 或提升前 release 目录竞态时的失败维护模式。 +- 决策补充：即使不打包 Pingora 二进制，API release 也必须随包携带 Pingora release readiness 聚合门禁、直连启用 / 回退 / preflight / live smoke / current release 自审脚本、直连彩排状态脚本，以及 `deploy/systemd/`、`deploy/pingora/` 支撑配置；`pingora-direct-enable.sh` 和正式 cutover runbook 默认从 `/opt/genarrative/current` 推导这些路径，启用前 release readiness 基础门禁和启用后 `--require-direct` 复核也必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs`，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。API release 还必须携带 `build//scripts/deploy/production-api-deploy.sh` 和同目录 `maintenance-on.sh` / `maintenance-off.sh`；`Genarrative-Api-Deploy` 只能复制并执行 build 产物内的 deploy 脚本，禁止继续执行部署工作区根部脚本，避免 workspace 中的旧脚本掩盖发布包布局缺陷。`production-api-deploy.sh` 对数据库备份脚本、健康巡检脚本和 Pingora 直连依赖都执行 fail-fast，发布产物缺失时保持维护模式并停止部署，不再从部署工作区兜底复制；API deploy 必须要求 `--release-root`、`--current-link`、`--api-env-file` 使用绝对路径，且 `--version` 必须以数字或字母开头并拒绝点目录，再先写 `${RELEASE_ROOT}/.${VERSION}.staging.$`，全部复制完成后再用非合并语义提升为 `${RELEASE_ROOT}/${VERSION}`，并用固定替换语义切换 current 符号链接，同版本 release 已存在、提升前竞态出现或 current 路径不是符号链接时拒绝覆盖 / 合并，失败时清理 staging 且不留下正式 release；`npm run check:production-api-release` 与 `npm run check:production-api-deploy` 必须进入 `check:pingora-release-readiness` 聚合门禁，前者用临时 `CARGO_TARGET_DIR` 和假 `api-server` release binary 验证 `build-production-release.sh --component api-server --skip-api-build` 产物自包含，后者用临时 release 和 fake `systemctl` / `curl` 验证从发布产物内执行 deploy 脚本后 current release 自包含，并覆盖缺少备份脚本、健康巡检脚本、release readiness 聚合门禁脚本、current release 自审脚本、直连彩排状态脚本、direct live smoke 脚本、相对 release root / current link / api env file、点目录或点开头 version、既有 release 目录、目录型 current 或提升前 release 目录竞态时的失败维护模式。 - 决策补充：正式直连 runbook 在采集状态快照前必须先执行 current release 自审：`/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs --release-root /opt/genarrative/current --require-pingora-gateway --systemd-show`。该脚本只读检查发布包自包含、`pingora-gateway` 可执行，以及 systemd `ExecStart` 是否指向 current release 网关二进制；失败时应先修发布包、Jenkins 归档过滤、deploy 复制或 systemd 指向，再继续切换。 +- 决策补充：直连前的 dev / release 彩排状态使用 `/opt/genarrative/current/scripts/ops/pingora-direct-rehearsal-status.mjs --release-root /opt/genarrative/current --expect-public-gateway nginx --require-pingora-shadow --require-realpath-canary --require-current-release-gateway --fail-on-critical`。该脚本只读读取 health patrol env、Pingora env、`systemctl`、`ss -H -ltnp`、realpath canary 配置和 current release 自审结果；`nginx` 期望模式要求公网 `80/443` 仍由 Nginx 监听，Pingora 只在 `127.0.0.1:18081` shadow，realpath canary 在 `127.0.0.1:18083`，不会写 `/etc`、reload systemd 或修改 Nginx / Pingora。 - 决策补充：Pingora 切换证据链正式纳入 `npm run check:pingora-current-release-audit`、`npm run check:pingora-cutover-status-snapshot`、`npm run check:pingora-cutover-evidence-bundle`、`npm run check:pingora-cutover-command-evidence`、`npm run check:pingora-cutover-evidence-verify`、`/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs`、`/opt/genarrative/current/scripts/ops/pingora-cutover-status-snapshot.mjs`、`/opt/genarrative/current/scripts/ops/pingora-cutover-evidence-bundle.mjs` 与 `/opt/genarrative/current/scripts/ops/pingora-cutover-evidence-verify.mjs`。状态快照按 `pre-cutover`、`post-enable`、`post-rollback` 三个阶段输出只读 JSON evidence，收录 `summary`、`healthPatrolEnv`、`pingoraEnv`、`releaseArtifacts`、`systemd` 和 `checks`；直连 runbook 的三个证据包阶段都显式透传绝对路径 `--output-root` 和 `--require-pingora-gateway`，让 `checks.current-release-audit.details` 同步归档 Pingora 二进制、sha256、release manifest 和 systemd `ExecStart` 自审结果；证据包脚本把快照 JSON、stdout、stderr、命令记录和 manifest 写入 `--output-root` 下的新证据目录，manifest 对已生成 snapshot、direct live、stdout / stderr、命令记录和 parse-error 文件记录 `path`、`sizeBytes` 与 `sha256`；每个阶段证据目录生成、复制或归档后，都必须用随包 verifier 按 `manifest.files` 只读复核文件存在、大小和 sha256，路径逃逸、符号链接证据目录、非目录证据路径、缺文件、大小漂移或 sha256 漂移都应失败。命令证据脚本把 direct enable apply / rollback apply 的真实 stdout、stderr、退出码、脱敏命令记录和 manifest 写入同一证据根目录，命令记录同时保留脱敏后的可读命令和结构化 `executable` / `args[]`，manifest 对 `command.stdout.txt`、`command.stderr.txt` 和 `command-record.json` 同样记录 `path`、`sizeBytes` 与 `sha256`，便于切换窗口后复核归档文件未漂移；runbook 必须在命令证据生成后立即用随包 verifier 验真 `` / ``，再继续 health patrol 切换、回退后 env 复核或最终总审计，且 `--phase` 只允许 ASCII 字母、数字、点、下划线和短横线，非法阶段名直接失败，不做隐式清洗。自审和快照只读采集，证据包只写归档目录且不覆盖既有文件，证据验真脚本只读 manifest 和证据文件，命令证据脚本只执行 `--` 后面的真实命令并归档输出，证据目录权限固定为 `0750`，证据文件权限固定为 `0640`，这些脚本都不写 `/etc`、不 reload systemd，也不修改 Nginx 或 Pingora；probe token 和其他 env 敏感值只允许以是否存在或 `` 的形式进入证据链，聚合门禁真实执行日志、dry-run plan、cutover runbook、直连启用脚本 direct live 命令日志、直连回退脚本 shadow probe 命令日志、gateway smoke 命令日志和证据包命令记录都不得输出 token 原文，状态快照在收录健康巡检、current release 自审等子检查 stdout / stderr 前必须按 env 敏感值脱敏，状态快照、证据包、命令证据和证据验真自测必须确认 token 原文不会进入 stdout、snapshot、manifest、命令记录或子检查输出；正式 runbook 应在 `--fail-on-critical` 下把自审、快照、证据包和 manifest 验真当成阻断证据，避免把发布包 checksum / manifest 漂移、env 漂移、systemd capability 残留、巡检失败或归档文件损坏带入后续阶段。 - 决策补充：正式直连 runbook 的即时证据 verifier 和最终证据根目录总审计内部复用 verifier 时，都必须使用 `--require-summary-ok`；总审计不能退化成只验 `manifest.files` hash，还要把 `manifest.summary.status=OK` 作为底层 verifier 严格模式的一部分。 - 决策补充：Pingora 切换命令证据生成端必须在执行前约束真实命令身份：`-- ` 本身必须是绝对路径，不能是文件系统根目录；真实命令和每个真实命令参数都不能包含换行或 NUL 字符。正式 runbook 因此直接执行 current release 随包 enable / rollback 脚本绝对路径，禁止用 `node`、`bash`、脚本名或其它 PATH 裸命令名包装。 diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index 8fd688ad1..01b2673b0 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -18,13 +18,14 @@ ## Pingora 直连 80/443 不能只改 env - 现象：`/etc/genarrative/pingora-gateway.env` 已把 `GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN` / `HTTP_REDIRECT_LISTEN` 改到 `0.0.0.0:443` / `0.0.0.0:80`，但 `genarrative-pingora-gateway.service` 启动失败，日志出现低端口绑定权限错误。 -- 原因：默认 service 用非 root `genarrative` 用户运行，并且主模板为了保持 shadow 安全边界不带 `CAP_NET_BIND_SERVICE`。低端口直连必须通过显式 systemd drop-in 单独授予 capability；同时 Certbot 私钥默认未必允许 `genarrative` 读取，Nginx 也可能仍占用 `80/443`。另一个常见误区是 API release 只带 `pingora-direct-enable.sh` / rollback 壳脚本，却漏带 `pingora-current-release-audit.mjs`、`check-pingora-direct-preflight.mjs`、`check-pingora-direct-live.mjs`、`deploy/systemd/`、`deploy/env/` 或 `deploy/pingora/`，导致从 `/opt/genarrative/current` 启用时依赖 Jenkins 工作区、源码 checkout 或 `/etc` 里某份参考模板；或者 release 已经包含新版 `pingora-gateway`，但已运行的 shadow / canary / direct service 没有随 `current` 链接切换重启，仍在跑旧二进制。Server-Provision 安装到 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 的 drop-in 只用于人工审阅和显式覆盖；直连启用脚本默认必须读取 current release 随包 `deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`，否则旧 `/etc` 模板会掩盖发布包缺失。API deploy 脚本本身也不能继续用部署工作区根部的 `scripts/deploy/production-api-deploy.sh`，否则 Jenkins workspace 里的脚本会掩盖 `build/` 发布包缺少 deploy / maintenance 同目录脚本的问题；备份脚本、健康巡检脚本和 env 示例目录同样不能从部署工作区兜底，切换命令证据脚本也不能从部署工作区兜底，否则 current release 会和上游构建归档漂移。Pingora 直连依赖、备份脚本、巡检脚本、env 示例目录和 API deploy 执行入口都必须来自上游发布产物；随包 `api-server.sha256` 和可选 `pingora-gateway.sha256` 也必须复制进 current release，供随包 current release 自审校验二进制；随包 `deploy/pingora/pingora-gateway.env.example` 也不能只检查存在，还要保持 gzip-only、不信任 XFF、前置代理确认关闭、接流保护开启和空 probe token 这些生产安全默认值；`production-api-deploy.sh` 发现缺失时应 fail-fast 并保持维护模式，不应从部署工作区兜底补齐；所有 API 发布包都必须携带 `release-manifest.json` 且登记 `api-server` artifact，发布包包含 Pingora 时还必须登记 `pingora-gateway` artifact，否则 deploy 应在切换 current 前失败；deploy 必须要求 release root、current link 和 api env file 都是绝对路径，release version 以数字或字母开头并拒绝点目录，再先写 staging release，全部复制完成后用非合并语义提升为正式 release，失败时清理 staging 且不留下正式 release，同版本 release 已存在、提升前竞态出现或 current 路径不是符号链接时拒绝覆盖 / 合并，避免旧文件混入 current；发布包包含 Pingora 且服务已 active 时，deploy 必须在 current 链接切换后 `try-restart` 并复核 active，服务未运行时只跳过，不主动拉起。 +- 原因：默认 service 用非 root `genarrative` 用户运行，并且主模板为了保持 shadow 安全边界不带 `CAP_NET_BIND_SERVICE`。低端口直连必须通过显式 systemd drop-in 单独授予 capability；同时 Certbot 私钥默认未必允许 `genarrative` 读取，Nginx 也可能仍占用 `80/443`。另一个常见误区是 API release 只带 `pingora-direct-enable.sh` / rollback 壳脚本，却漏带 `pingora-current-release-audit.mjs`、`pingora-direct-rehearsal-status.mjs`、`check-pingora-direct-preflight.mjs`、`check-pingora-direct-live.mjs`、`deploy/systemd/`、`deploy/env/` 或 `deploy/pingora/`，导致从 `/opt/genarrative/current` 启用时依赖 Jenkins 工作区、源码 checkout 或 `/etc` 里某份参考模板；或者 release 已经包含新版 `pingora-gateway`，但已运行的 shadow / canary / direct service 没有随 `current` 链接切换重启，仍在跑旧二进制。Server-Provision 安装到 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 的 drop-in 只用于人工审阅和显式覆盖；直连启用脚本默认必须读取 current release 随包 `deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`，否则旧 `/etc` 模板会掩盖发布包缺失。API deploy 脚本本身也不能继续用部署工作区根部的 `scripts/deploy/production-api-deploy.sh`，否则 Jenkins workspace 里的脚本会掩盖 `build/` 发布包缺少 deploy / maintenance 同目录脚本的问题；备份脚本、健康巡检脚本和 env 示例目录同样不能从部署工作区兜底，切换命令证据脚本也不能从部署工作区兜底，否则 current release 会和上游构建归档漂移。Pingora 直连依赖、备份脚本、巡检脚本、env 示例目录和 API deploy 执行入口都必须来自上游发布产物；随包 `api-server.sha256` 和可选 `pingora-gateway.sha256` 也必须复制进 current release，供随包 current release 自审校验二进制；随包 `deploy/pingora/pingora-gateway.env.example` 也不能只检查存在，还要保持 gzip-only、不信任 XFF、前置代理确认关闭、接流保护开启和空 probe token 这些生产安全默认值；`production-api-deploy.sh` 发现缺失时应 fail-fast 并保持维护模式，不应从部署工作区兜底补齐；所有 API 发布包都必须携带 `release-manifest.json` 且登记 `api-server` artifact，发布包包含 Pingora 时还必须登记 `pingora-gateway` artifact，否则 deploy 应在切换 current 前失败；deploy 必须要求 release root、current link 和 api env file 都是绝对路径，release version 以数字或字母开头并拒绝点目录，再先写 staging release，全部复制完成后用非合并语义提升为正式 release，失败时清理 staging 且不留下正式 release，同版本 release 已存在、提升前竞态出现或 current 路径不是符号链接时拒绝覆盖 / 合并，避免旧文件混入 current；发布包包含 Pingora 且服务已 active 时，deploy 必须在 current 链接切换后 `try-restart` 并复核 active，服务未运行时只跳过，不主动拉起。 - 处理：确认真实 TLS 证书和 redirect env 已写入 `/etc/genarrative/pingora-gateway.env`、service 模板和 `systemctl cat` 最终配置读取的 `EnvironmentFile=` 都包含这份 env、当前执行用户和 `genarrative-pingora-gateway.service` 的 `User=` 服务用户都能读取证书链 / 私钥、current release 的 `pingora-gateway` 已存在且可执行、Nginx 或其它进程已释放 `80/443` 后，先用 `npm run plan:pingora-direct-cutover -- --require-direct ...` 生成只读 JSON runbook，并逐条审阅 Host 与回退巡检入口确认、current release 自包含自审、current release preflight、启用前基础 readiness、direct enable dry-run、direct enable apply、启用后 `--require-direct` 复核、rollback dry-run、rollback apply、回退后 health patrol 切回 Nginx 并恢复切换前 public base URL / Host、回退后 health patrol env 复核；runbook 只用于审阅，不修改系统。正式 runbook 中 `--direct-redirect-host`、`--rollback-nginx-smoke-host` 和 `--direct-host` 必须使用同一 hostname，只允许端口不同，避免 redirect 或回退 smoke 各自验证到不同入口；同时必须提供 `--rollback-health-patrol-public-base-url <切换前Nginx巡检入口>`，若切换前 Nginx 巡检需要 Host 覆盖，再追加 `--rollback-health-patrol-public-host <切换前Host>`，确认步骤会展示回退后要恢复的 public base URL / Host，避免回退 runbook 把现场巡检入口覆盖成仓库默认值；如需把回退后 Pingora shadow 探针复核纳入 runbook，追加 `--rollback-pingora-shadow-probe-url` / `--rollback-pingora-shadow-probe-token`，JSON 输出会隐藏 token 原文。随后先执行 `/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs --release-root /opt/genarrative/current --require-pingora-gateway --systemd-show`，再 dry-run `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --no-status`，最后执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --apply --preflight-env-file /etc/genarrative/pingora-gateway.env --preflight-check-cert-readable --preflight-check-service-env-file --preflight-check-service-user-cert-readable --preflight-check-service-binary-executable --preflight-check-ports-free --direct-https-base-url https://127.0.0.1 --direct-http-base-url http://127.0.0.1 --direct-host <域名> --direct-redirect-host <域名或host:port> --direct-spacetime-database <库名> --direct-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`，由脚本先跑 direct preflight，再安装 drop-in、reload systemd、重启 Pingora，并用 `systemctl cat` 核验 capability 和 `EnvironmentFile=/etc/genarrative/pingora-gateway.env` 已生效、用 `systemctl show ... ExecStart` 核验最终 service 仍指向随包主 service 模板里的 current release `pingora-gateway`、用 `systemctl is-active` 确认服务 active，再以 JSON 模式执行 direct live smoke，验证 HTTPS / HTTP redirect / ACME / WSS 101 和 Pingora access log request_id 落盘，并要求 `direct-access-log` 结构化结果 `matchedCount == checked`、`missingCount=0`、`mismatchCount=0`；如果 direct live 退出 0 但缺少该结构化证据，也必须视为启用失败。直连启用后同步调整 `/etc/genarrative/health-patrol.env`：设置 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=pingora-direct`，本机打 `127.0.0.1` 时设置 `GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST=<域名>`，否则巡检会继续按 Nginx 模式误报。验证失败时执行 `/opt/genarrative/current/scripts/deploy/pingora-direct-rollback.sh --apply --reload-nginx --nginx-smoke-url http://127.0.0.1/healthz --nginx-smoke-host <域名>` 或 `npm run deploy:pingora-direct-rollback -- --apply --reload-nginx --nginx-smoke-url http://127.0.0.1/healthz --nginx-smoke-host <域名>`；回退脚本先跑 `nginx -t`，通过后才移除 drop-in、reload systemd、重启 Pingora，并用 `systemctl cat` 核验 capability 已移除、用 `systemctl show ... ExecStart` 核验最终 service 仍指向随包主 service 模板里的 current release `pingora-gateway`，随后 reload Nginx、确认 Nginx service 仍为 active，并用 curl smoke URL 证明公网入口已回到 Nginx；回退脚本 `--apply` 不允许省略 `--reload-nginx` 或 `--nginx-smoke-url`，当 smoke URL 指向本机地址时必须同时提供 `--nginx-smoke-host <域名>`，且 host 值不能包含 URL、路径或查询；回退后把 health patrol gateway mode 改回 `nginx`，恢复切换前 public base URL / Host，并用 `node -- /opt/genarrative/current/scripts/check-production-health-patrol-env.mjs --env-file /etc/genarrative/health-patrol.env --expected-gateway-mode nginx --expected-public-base-url <切换前Nginx巡检入口> --require-empty-public-host` 复核；若切换前 Nginx 巡检需要 Host 覆盖，则把 `--require-empty-public-host` 换成 `--expected-public-host <切换前Host>`。若 env 已在回退命令前切回 Nginx，也可给 rollback 脚本追加 `--health-patrol-env-file /etc/genarrative/health-patrol.env --health-patrol-expected-public-base-url <切换前Nginx巡检入口> --health-patrol-require-empty-public-host` 让它在 Nginx smoke 后自动复核；切换前 Nginx 巡检需要 Host 覆盖时把最后一项换成 `--health-patrol-expected-public-host <切换前Host>`。若要同时证明 Pingora shadow 高端口仍活着，追加 `--pingora-shadow-probe-url http://127.0.0.1:18081/__genarrative_pingora/healthz --pingora-shadow-probe-token `，脚本会隐藏 token 并要求响应为 `gateway=pingora-shadow`。 - 处理补充：不要直接 chmod `/etc/letsencrypt/live` 或 `archive` 来让 Pingora 读取证书；Certbot live 路径通常是 symlink，即使 `stat -L` 看起来是普通文件，父目录权限也会让非 root `genarrative` 用户不可达。先用随包 `node -- /opt/genarrative/current/scripts/deploy/pingora-tls-cert-sync.mjs --apply --source-cert-file /etc/letsencrypt/live/<域名>/fullchain.pem --source-key-file /etc/letsencrypt/live/<域名>/privkey.pem --target-dir /etc/genarrative/pingora-tls/<域名>` 把证书同步到 Pingora 私有目录，再让 `GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE` / `TLS_KEY_FILE` 指向 `/etc/genarrative/pingora-tls/<域名>/fullchain.pem` 和 `privkey.pem`。脚本默认 dry-run，`--apply` 才写入，目标目录默认 `root:genarrative 0750`，文件默认 `root:genarrative 0640`，并拒绝符号链接目标目录或目标文件。 - 处理补充：不要在切换窗口手工编辑 `/etc/genarrative/health-patrol.env` 的三项网关变量；使用 `node -- /opt/genarrative/current/scripts/deploy/pingora-health-patrol-env-switch.mjs --apply --env-file /etc/genarrative/health-patrol.env --gateway-mode pingora-direct --public-base-url <直连HTTPS入口> --public-host <域名>` 切到直连，回退前用同一脚本传 `--gateway-mode nginx --public-base-url <切换前Nginx巡检入口>` 并按切换前记录选择 `--clear-public-host` 或 `--public-host <切换前Host>`。脚本只改 gateway mode / public base URL / public Host，并立即复用随包 env 复核脚本，减少空 Host 和旧值残留；生产巡检、env 复核和 env 切换脚本读取的布尔 env 都必须是明确布尔值，非法值直接失败，不能把拼写错误当成 false；env 复核脚本的 `--env-file` 与 env 切换脚本的 `--env-file` / `--check-script` 必须是绝对路径且不能是文件系统根目录，也不能包含换行或 NUL；env 切换脚本写入的 public base URL / Host 同样不能包含换行或 NUL。Node 22 已内置 `--env-file` 启动参数，直接用 `node script.mjs --env-file ...` 或 shebang 执行 `.mjs --env-file ...` 都可能让 Node 抢走业务参数；所有这类命令都必须写成 `node -- script.mjs --env-file ...`，或通过已内置 `node --` 的 npm script 执行。 - 踩坑补充：health patrol env 切换脚本必须先复核权限固定为 `0600` 的临时目标 env 再写真实文件，真实 env 原子替换时保持原文件权限和 owner/group；如果随包 env 复核脚本失败，`--apply` 应失败且真实 env 保持原样，避免“切换脚本失败但巡检配置已半改”的状态。`--apply` 的 `--env-file` 必须直接指向真实普通文件，不能传符号链接；如果 `/etc/genarrative/health-patrol.env` 是链接，先确认真实目标路径后再传给脚本，避免替换链接本身或写入非预期目标。 - 踩坑补充：直连启用脚本的 `--preflight-script`、`--direct-live-script`、`--current-release-audit-script`、`--template-path`、`--service-unit-path`、`--dropin-path` 和 env 文件参数都必须使用绝对路径；不要在切换窗口传相对脚本路径，否则会把 current release、Jenkins 工作区或现场 cwd 混在一起。`--apply` 会在安装 direct-entry drop-in 前确认 current release 自审、direct preflight 和 direct live smoke 脚本存在，缺脚本时应先修发布包或复制链路，不要手工改成工作区相对路径绕过。启用脚本还会在任何自审、preflight、drop-in 写入或 systemctl 前拒绝 service、路径、URL、Host、probe token、access log、数据库名、tail 行数和 timeout 参数中的换行或 NUL 字符；遇到这类失败先修 runbook 参数来源或现场 env，不要手工绕过脚本。启用脚本还会拒绝符号链接形式的 drop-in 目录或 drop-in 目标文件，以及已存在但不是普通文件的目标；如果现场 systemd 目录被软链改写，应先修正真实路径，不要让脚本把低端口 capability 写入非预期位置。回退脚本 `--apply` 同样会在 `nginx -t` 和删除 drop-in 前拒绝 service、路径、Nginx smoke URL / Host / 响应片段、health patrol 复核参数、shadow probe URL / token 和二进制 override 中的换行或 NUL 字符，并拒绝符号链接 drop-in 目录 / 目标以及非普通 drop-in 目标；如果现场路径或参数异常，应先修正 systemd 路径、runbook 参数或现场 env，不要手工删 drop-in、绕过 `nginx -t` 或把删除 symlink 当成已回退真实低端口能力。回退脚本覆盖 `--nginx-binary` 或 `--curl-binary` 时也不要传 `./nginx`、`tools/curl` 这类相对路径；裸命令名可以走 `PATH`，路径形式必须使用绝对路径。`--nginx-smoke-url` 必须带 `http://` 或 `https://`，不要只写 host/path，否则脚本会在移除 drop-in 前失败。 - 踩坑补充：回退到 Nginx 后不要只把 `curl --fail` / HTTP 200 当作 Nginx 已接回的证据；正式 runbook 默认给 rollback dry-run / apply 传 `--nginx-smoke-expect-body '"ok":true'`，要求 `/healthz` 响应体包含预期片段。若现场把 `--rollback-nginx-smoke-url` 改成其它端点，必须同步调整 `--rollback-nginx-smoke-expect-body`，否则要么误卡回退，要么只验证到了错误入口。 +- 踩坑补充：直连彩排状态脚本不是修复动作。`node -- /opt/genarrative/current/scripts/ops/pingora-direct-rehearsal-status.mjs --release-root /opt/genarrative/current --expect-public-gateway nginx --require-pingora-shadow --require-realpath-canary --require-current-release-gateway --fail-on-critical` 只读检查公网端口归属、health patrol 模式、Pingora shadow、realpath canary、systemd 和 current release 自审；如果它报 `CRITICAL`，应先修发布包、端口归属、canary 配置、health patrol env 或 systemd 指向，不要把它当成会自动启用 canary、停止 Nginx 或修复 current release 的脚本。 - 踩坑补充：Pingora current release 自审和切换证据链都不是修复动作，`npm run check:pingora-current-release-audit` / `scripts/ops/pingora-current-release-audit.mjs` 只负责只读确认发布包自包含、`api-server.sha256` / `pingora-gateway.sha256` 匹配、release manifest 登记了当前要接流的 Pingora 产物、`pingora-gateway` 可执行和 systemd `ExecStart` 指向；`npm run check:pingora-cutover-status-snapshot` / `scripts/ops/pingora-cutover-status-snapshot.mjs` 只负责输出 `pre-cutover`、`post-enable`、`post-rollback` 三阶段只读 JSON evidence，并在直连 runbook 中通过 `--require-pingora-gateway` 把上述自审结果收录到 `checks.current-release-audit.details`；快照还必须确认 `systemctl cat genarrative-pingora-gateway.service` 的 `EnvironmentFile=` 精确包含本次 `--pingora-env-file`，否则 `systemd.pingoraUnit.environmentFileMatchesPingoraEnvFile=false` 且标记 `CRITICAL`，避免证据包读到一份 env、真实服务读另一份 env。正式切换窗口用 `scripts/ops/pingora-cutover-evidence-bundle.mjs` 把快照 JSON、stdout、stderr、命令记录和 manifest 写入 `--output-root` 下的新证据目录，证据包 manifest 必须记录已生成 snapshot、direct live、stdout / stderr、命令记录和 parse-error 文件的 `path`、`sizeBytes` 与 `sha256`，便于归档后复核；证据目录生成、复制或归档后必须用随包 `scripts/ops/pingora-cutover-evidence-verify.mjs --bundle-dir ` 做只读验真，确认 `manifest.files` 登记的文件未缺失、大小未漂移、sha256 未漂移，且证据目录不是符号链接或非目录；三阶段证据分别验真后，还必须用随包 `scripts/ops/pingora-cutover-evidence-audit.mjs --evidence-root <证据根目录> --require-phase pre-cutover --require-phase post-enable --require-phase post-rollback` 做只读总审计，自动选择每个阶段最新 bundle 并复用 verifier，缺阶段、最新证据损坏、坏 manifest 或符号链接条目都应失败。direct enable apply / rollback apply 必须通过 `scripts/ops/pingora-cutover-command-evidence.mjs` 包装真实脚本，单独保存命令 stdout、stderr、退出码、脱敏命令记录和 manifest，runbook 必须显式传绝对路径 `--output-root`，且该路径不能是文件系统根目录、符号链接或包含换行 / NUL 字符；命令记录必须同时保留脱敏后的可读命令和结构化 `executable` / `args[]`，命令证据 manifest 也必须记录 `command.stdout.txt`、`command.stderr.txt` 和 `command-record.json` 的 `sizeBytes` 与 `sha256`；命令证据生成后也要立即把 stdout 中的 `bundleDir` 填入随包 verifier 的 `` 或 `` 占位符做只读验真，不能只等最终根目录总审计才发现 command-record 或 stdout/stderr 归档漂移。启用后证据包必须额外运行随包 direct live smoke，并写入 `direct-live.json`、`direct-live.stdout.txt`、`direct-live.stderr.txt`、`direct-live-command.json` 和 manifest summary 的 `directLiveStatus`，让 Pingora access log `request_id` 反查结果可复盘；`direct-live.json` 的 `direct-access-log` 结果必须保留扫描行数、匹配数量、缺失明细以及 method/path/status 漂移明细，不要只保留 count 或依赖 stderr。证据包 manifest summary 还必须包含 `directLiveAccessLog` 摘要；如果 direct live JSON 缺少 `direct-access-log` 结构化结果，整包应记为 `CRITICAL`。若 snapshot 或 direct live stdout 解析失败，必须保留 `snapshot-parse-error.txt` 或 `direct-live-parse-error.txt` 并在 manifest / 最终 stdout 中给出路径；不要只截图或复制 `pingora-direct-enable.sh` / release readiness 的终端输出当作直连证据。证据阶段名只能使用 ASCII 字母、数字、点、下划线和短横线，非法 `--phase` 会直接失败，不会被清洗后继续落盘；自审、状态快照和证据包的 `--release-root` 都不能是文件系统根目录，状态快照的 `--health-patrol-env-file` / `--pingora-env-file` 以及证据包所有显式路径参数也不能是文件系统根目录，状态快照自身还必须拒绝带换行或 NUL 的 release/env 路径，并在执行 systemctl、current release 自审、health patrol env 复核或生产巡检子命令前复核子命令参数，证据包执行状态快照或 direct live 子命令前也必须拒绝任何带换行或 NUL 字符的子命令参数，避免污染后的结构化 `args[]` 先进入正式证据再等总审计兜底，`--output-root` 及其已存在上级路径不能是符号链接，已存在的 `--output-root` 必须是真实目录，路径异常时会在执行状态快照前失败，避免把证据写入非预期软链目标；current release 自审、状态快照和证据包的显式 `--timeout-ms` 及对应 env 必须是正整数，生产健康巡检的 `--timeout-ms`、`--slow-ms`、`GENARRATIVE_HEALTH_PATROL_TIMEOUT_MS` 和 `GENARRATIVE_HEALTH_PATROL_SLOW_MS`，canary / direct live smoke 的 `--timeout-ms` 及对应 env，canary access log 对账的 `--since-lines` 及对应 env 也必须是正整数，直连 live / release readiness 的布尔 env 也必须是明确布尔值，非法值都会失败，不再静默回退默认值或 false。自审、快照、证据包、证据验真和证据总审计脚本都不修改 `/etc`、systemd、Nginx 或 Pingora；命令证据脚本只执行 `--` 后面的真实命令并归档输出，不自行理解 systemd / Nginx；证据包和命令证据目录必须是 `0750`，证据文件必须是 `0640`，且不能覆盖既有文件；probe token 和其他 env 敏感值只能记录是否存在或显示 ``，不能把 env 原文写入终端执行日志、gateway smoke / direct live / direct rollback shadow probe 命令日志、stdout、snapshot、manifest、命令记录或子检查 stdout / stderr；如果自审、快照、direct live 或总审计证据里出现 `CRITICAL`，应先修发布包、Jenkins 归档过滤、deploy 复制、env、systemd capability、直连入口、巡检状态或证据归档，再继续下一阶段，不要把自审、快照或证据包当成可自动修复的烟测。 - 踩坑补充：直连 Pingora 后不要让静态缓存头继续依赖框架默认值。HTML、目录 index 和 SPA fallback 必须保持 `Cache-Control: no-cache`，否则旧入口页可能长期引用已经切换的 chunk；带 Vite 指纹的 `/assets/*` 和 `/admin/assets/*` 才能使用 `public, max-age=31536000, immutable`；普通非指纹静态和 ACME challenge 继续保守 `no-cache`。如果需要临时覆盖 `GENARRATIVE_PINGORA_GATEWAY_*_CACHE_CONTROL`，值不能包含换行或 NUL，修改后必须跑 `npm run check:pingora-gateway-smoke` 确认 HTML、普通静态和指纹资源三类响应头没有漂移。 - 踩坑补充：直连 Pingora 后也不能只验证整文件静态读取。浏览器、媒体探测和线上签名 URL 排障都可能使用 `Range: bytes=`；Pingora 静态文件必须支持单段 range 的 `206 + Content-Range` 和越界 range 的 `416 + Content-Range: bytes */`，同时给静态响应写入 `Accept-Ranges: bytes`。`If-Range` 不能被忽略：日期匹配才继续给局部内容，旧日期或弱 ETag 校验器应回完整 `200`，避免客户端拿旧校验器拼接错误文件片段。`206`、`304`、`416` 不应被 gzip 压缩，否则 `Content-Range` 指向的字节区间会和实际响应体不一致。多段 range 暂按完整文件处理，不要在切换窗口临时拼 multipart 响应。 @@ -2357,6 +2358,14 @@ - 验证：`npm run check:pingora-gateway-smoke` 必须覆盖静态 `HEAD`、`If-None-Match` 304、`If-Modified-Since` 304，并用 access log method/path/status 对账证明本地静态边界进入日志证据链；`cargo test -p pingora-gateway --manifest-path server-rs/Cargo.toml` 必须覆盖 ETag 构造和匹配 helper。 - 关联：`server-rs/crates/pingora-gateway/src/main.rs`、`scripts/check-pingora-gateway-smoke.mjs`、`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`。 +## dev health patrol 不能缺少公网 HTTPS 入口配置 + +- 现象：dev 上 `genarrative-health-patrol.timer` 正常 active，但 `genarrative-health-patrol.service` 最近一次运行失败；Pingora 直连彩排状态脚本只因 `/etc/genarrative/health-patrol.env` 缺失或 public probe 命中 `http://127.0.0.1` 后被 Nginx 301 而报 `CRITICAL`。 +- 原因：health patrol systemd unit 的 `EnvironmentFile=-/etc/genarrative/health-patrol.env` 允许文件缺失，脚本会退回默认 public base URL `http://127.0.0.1`；dev / release 的 Nginx 公开入口会把 HTTP 跳到 HTTPS，巡检按非 2xx 判失败。 +- 处理：目标机应创建 `/etc/genarrative/health-patrol.env`，保持 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=nginx`，把 `GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL` 指向真实 HTTPS 域名，例如 `https://dev.genarrative.world`；Pingora shadow 巡检同时配置 `GENARRATIVE_HEALTH_PATROL_PINGORA_BASE_URL=http://127.0.0.1:18081` 和与 `/etc/genarrative/pingora-gateway.env` 一致的 probe token。不要为了让彩排状态变绿把缺 env 降级成 warning。 +- 验证：先运行随包 `node -- /opt/genarrative/current/scripts/check-production-health-patrol-env.mjs --env-file /etc/genarrative/health-patrol.env --expected-gateway-mode nginx --expected-public-base-url https://dev.genarrative.world --require-empty-public-host`，再 `systemctl start genarrative-health-patrol.service`；最后运行 `node -- /opt/genarrative/current/scripts/ops/pingora-direct-rehearsal-status.mjs --release-root /opt/genarrative/current --expect-public-gateway nginx --require-pingora-shadow --require-realpath-canary --require-current-release-gateway --fail-on-critical`。 +- 关联：`deploy/env/health-patrol.env.example`、`scripts/ops/production-health-patrol.mjs`、`scripts/ops/pingora-direct-rehearsal-status.mjs`、`docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md`。 + ## SpacetimeDB 连接池租约必须有 Drop 兜底，acquire 不允许无界自旋 - 现象：release 上 api-server 周期性出现全量 `spacetime_stage="pool_acquire" elapsed_ms=45000` 业务超时，`/readyz` 503（`reason=spacetime_unhealthy, stage=pool_acquire`），`/healthz` 仍 200，只有重启能恢复，过若干小时复发。 diff --git a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md index 7ef9331dd..001662fe1 100644 --- a/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md +++ b/docs/technical/【开发运维】Pingora独立网关试点-2026-06-11.md @@ -50,6 +50,7 @@ npm run check:pingora-direct-live npm run check:pingora-direct-enable npm run check:pingora-direct-rollback npm run check:pingora-current-release-audit +npm run check:pingora-direct-rehearsal-status npm run check:pingora-cutover-status-snapshot npm run check:pingora-cutover-evidence-bundle npm run check:pingora-cutover-evidence-verify @@ -87,6 +88,20 @@ npm run check:pingora-direct-preflight -- \ `check:pingora-current-release-audit` 会烟测 `scripts/ops/pingora-current-release-audit.mjs` 的只读边界、current release 必备脚本 / 配置目录、可选 `pingora-gateway` 可执行性和 `--systemd-show` 下的 `ExecStart` 指向。正式直连 runbook 会先执行 `/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs --release-root /opt/genarrative/current --require-pingora-gateway --systemd-show`，在采集状态快照前确认目标机 current release 本身自包含；显式 `--timeout-ms` 和 `GENARRATIVE_PINGORA_CURRENT_RELEASE_TIMEOUT_MS` 必须是正整数，非法值直接失败，不静默回退默认 `5000`。`GENARRATIVE_PINGORA_CURRENT_RELEASE_REQUIRE_GATEWAY`、`GENARRATIVE_PINGORA_CURRENT_RELEASE_SYSTEMD_SHOW` 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败，不再按 false 继续执行；`--release-root` 必须是绝对路径且不能是文件系统根目录，也不能包含换行或 NUL 字符，`--systemd-service` 同样不能包含换行或 NUL 字符；启用 `--systemd-show` 时，脚本会在执行 `systemctl show` 前复核子命令可执行文件和参数不含换行或 NUL，避免把污染参数带进只读审计命令。 +`check:pingora-direct-rehearsal-status` 会烟测 `scripts/ops/pingora-direct-rehearsal-status.mjs` 的只读彩排状态采集。目标机切换前可执行： + +```bash +node -- /opt/genarrative/current/scripts/ops/pingora-direct-rehearsal-status.mjs \ + --release-root /opt/genarrative/current \ + --expect-public-gateway nginx \ + --require-pingora-shadow \ + --require-realpath-canary \ + --require-current-release-gateway \ + --fail-on-critical +``` + +该脚本只读读取 health patrol env、Pingora env、`systemctl`、`ss -H -ltnp`、realpath canary 配置和 current release 自审结果，输出 JSON 总览；不会写 `/etc`、不会 reload systemd，也不会修改 Nginx 或 Pingora。`--expect-public-gateway nginx` 会要求公网 `80/443` 仍由 Nginx 监听，`--require-pingora-shadow` 会要求 `127.0.0.1:18081` 由 Pingora shadow 监听，`--require-realpath-canary` 会要求 `127.0.0.1:18083` 有 Nginx realpath canary，`--require-current-release-gateway` 会复用 current release 自审确认 `pingora-gateway`、checksum、manifest 和 systemd `ExecStart`。该脚本用于直连前确认现场状态已经进入“可彩排但未切公网”的安全边界，不替代后续证据包，也不做任何启用或回退动作。 + `check:pingora-cutover-status-snapshot` 会烟测 `scripts/ops/pingora-cutover-status-snapshot.mjs` 的只读边界、release artifact 采集、health patrol env 复核、systemd capability / EnvironmentFile 判断和 `--fail-on-critical` 行为。正式切换窗口由 `/opt/genarrative/current/scripts/ops/pingora-cutover-evidence-bundle.mjs` 调用随包状态快照脚本，按 `--phase pre-cutover`、`--phase post-enable`、`--phase post-rollback` 生成时间戳证据目录，保存 `manifest.json`、`snapshot.json`、`snapshot.stdout.txt`、`snapshot.stderr.txt` 和 `snapshot-command.json`；状态快照 stdout 解析失败时保存 `snapshot-parse-error.txt` 并在 manifest / 最终 stdout 中给出路径，启用后 direct live stdout 解析失败时保存 `direct-live-parse-error.txt` 并同样索引。状态快照会把 `--pingora-env-file` 与 `systemctl cat genarrative-pingora-gateway.service` 的 `EnvironmentFile=` 做精确匹配，支持 `EnvironmentFile=-/path` 和一行多个文件，但不接受路径前缀误判；未包含本次 pingora env 时 `systemd.pingoraUnit.environmentFileMatchesPingoraEnvFile=false` 并标记 `CRITICAL`。启用后 direct live 结果中的 `direct-access-log` 必须保留 `scannedLineCount`、`matchedCount`、`missing[]`、`mismatches[]` 以及每个 `request_id` 的预期 / 实际 method、path 与 status，避免证据包只留下计数或终端 stderr；静态 GET / HEAD / 304 / Range 结果还必须在 `direct-live.json` 中保留白名单响应头，字段限于 `cache-control`、`etag`、`last-modified`、`accept-ranges`、`content-range`、`content-length` 和 `content-encoding`，便于复盘缓存分档、校验器和 Range 语义，同时避免把 API / WSS 原始响应头落进正式证据。证据包还会把 `directLiveAccessLog` 和 `directLiveStaticHeaders` 摘要提升到 `manifest.summary`，前者用于快速确认 request_id 对账，后者用于快速确认普通静态 / Vite 指纹静态的 `Cache-Control`、`ETag`、`Last-Modified`、`Content-Length`、Range `Content-Range` 与 304 状态证据；缺少 `direct-access-log` 结构化结果、缺少可判定的静态头摘要，或静态摘要缺少缓存头、校验头、Range `206 + Content-Range`、ETag 304 / Last-Modified 304 证据时，证据包都会直接记为 `CRITICAL`。维护模式、非 HTML 或首页确实没有构建资产引用时，direct live 会把静态检查标记为 skipped，此时允许证据包继续记录跳过原因。`snapshot-command.json` 与 manifest 的 `commands[]` 同时保留脱敏后的可读命令和结构化 `executable` / `args[]`，便于路径或参数包含空格时复盘；证据包在执行状态快照或 direct live 子命令前也会拒绝任何带换行或 NUL 字符的子命令参数，避免污染后的 `args[]` 先落盘再等总审计兜底。`--run-direct-live` 使用的 HTTPS / HTTP base URL、Host、redirect Host、probe token、SpacetimeDB 数据库名、Pingora access log 路径和 access log tail 行数也会在证据包配置层先拒绝换行或 NUL，且 `--direct-pingora-access-log` 必须是绝对路径并且不能是文件系统根目录，避免进入 snapshot 或 direct live 子命令后才暴露参数污染。`--phase` 只允许 ASCII 字母、数字、点、下划线和短横线，非法阶段名直接失败，不做隐式清洗，避免目录名和 manifest 阶段漂移；状态快照和证据包的显式 `--timeout-ms` 以及 `GENARRATIVE_PINGORA_CUTOVER_SNAPSHOT_TIMEOUT_MS` 必须是正整数，非法值直接失败，不再静默回退默认 `5000`。`GENARRATIVE_HEALTH_PATROL_REQUIRE_EMPTY_PUBLIC_HOST`、`GENARRATIVE_PINGORA_CUTOVER_SNAPSHOT_RUN_HEALTH_PATROL`、`GENARRATIVE_PINGORA_CUTOVER_SNAPSHOT_REQUIRE_PINGORA_GATEWAY`、`GENARRATIVE_PINGORA_CUTOVER_SNAPSHOT_FAIL_ON_CRITICAL` 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败，避免 run / require / fail 开关拼写错误后静默按 false 采集证据。状态快照脚本只读采集 `summary`、`healthPatrolEnv`、`pingoraEnv`、`releaseArtifacts`、`systemd` 和 `checks`；probe token 和其他 env 敏感值只记录是否存在，不输出原文，健康巡检、current release 自审等子检查的 stdout / stderr 在进入快照 JSON 前也必须按 env 敏感值脱敏，证据包里的 manifest、snapshot、stdout 和命令记录都必须通过自测确认不落盘 token 原文；状态快照脚本的 `--release-root`、`--health-patrol-env-file` 和 `--pingora-env-file` 都必须是绝对路径、不能是文件系统根目录，也不能包含换行或 NUL 字符，且执行 systemctl、current release 自审、health patrol env 复核或生产巡检子命令前还会复核子命令参数不含换行或 NUL；证据包脚本只写 `--output-root` 下的新目录，且证据包所有显式路径参数也不能是文件系统根目录，`--output-root` 及其已存在上级路径不能是符号链接，已存在的 `--output-root` 必须是真实目录；路径异常时会在执行状态快照前失败，不写入软链目标、不覆盖既有文件，证据目录权限固定为 `0750`，证据文件权限固定为 `0640`，不写 `/etc`、不 reload systemd，也不修改 Nginx 或 Pingora。 直连 runbook 的三个证据包阶段都会透传 `--require-pingora-gateway`，让 `checks.current-release-audit.details` 同步归档 Pingora 二进制、sha256、release manifest 和 systemd `ExecStart` 自审结果，避免证据包只有巡检状态而缺少发布物可信度证明。每个证据包生成后都要用 `/opt/genarrative/current/scripts/ops/pingora-cutover-evidence-verify.mjs --bundle-dir <本阶段bundleDir>` 做只读验真；该脚本只接受 `schemaVersion=1` 的 manifest，并把 `manifest.files` 视为闭集，只读取 `manifest.files` 中的 `{ path, sizeBytes, sha256 }`，拒绝路径逃逸、符号链接证据目录、非目录证据目录、非元数据对象文件条目，以及任何未登记的额外普通文件、目录或符号链接，发现缺文件、大小漂移或 sha256 漂移时退出失败；`--allow-extra-files` 只用于人工排障时显式放行额外条目，正式切换归档不应使用。验真脚本不修改证据目录、不 reload systemd，也不访问 Nginx 或 Pingora。enable apply / rollback apply 的命令证据生成后也要立即用同一个随包 verifier 验真，将上一步 stdout 的 `bundleDir` 分别填入 `` / ``，确认 `command.stdout.txt`、`command.stderr.txt`、`command-record.json` 与 manifest 元数据一致，再继续后续 health patrol 或回退复核。三阶段证据包都生成并分别验真、enable / rollback apply 命令证据也即时验真后，还要执行 `/opt/genarrative/current/scripts/ops/pingora-cutover-evidence-audit.mjs --evidence-root <证据根目录> --require-phase pre-cutover --require-phase post-enable --require-phase post-rollback --require-phase-direct-live-access-log post-enable --require-phase-direct-live-static-headers post-enable --require-command enable-apply:pingora-direct-enable-apply --require-command rollback-apply:pingora-direct-rollback-apply --require-command-executable enable-apply:pingora-direct-enable-apply:/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh --require-command-executable rollback-apply:pingora-direct-rollback-apply:/opt/genarrative/current/scripts/deploy/pingora-direct-rollback.sh --require-command-arg enable-apply:pingora-direct-enable-apply:--apply --require-command-arg rollback-apply:pingora-direct-rollback-apply:--apply --require-cutover-run-id <本次cutoverRunId> --timeline-max-span-ms 86400000`，由脚本按 `manifest.phase` 找到每个阶段最新证据目录、按 `manifest.phase + manifest.commandName` 找到 enable / rollback apply 最新命令证据，并复核两条真实切换命令都来自 current release 随包脚本；所有候选证据都必须带合法 `manifest.generatedAt`，格式必须是 `new Date().toISOString()` 产出的 UTC 毫秒格式 `YYYY-MM-DDTHH:mm:ss.sssZ`，最新选择和后续时间线证明只使用该字段，不依赖目录 mtime；随后复用随包 verifier 的 `--require-summary-ok` 严格模式再次只读验真，输出可归档的阶段与命令总表；阶段证据必须是不带 `manifest.commandName` 的状态快照证据包，命令证据不能冒充同名 phase 的阶段证据。总审计输出会把阶段 manifest 中的 `directLiveAccessLog` 与 `directLiveStaticHeaders` 一并带到对应 `phases[]` 项，启用后阶段可直接看到 request_id 对账数量、普通静态和指纹静态的缓存头、校验头、Range `Content-Range` 与 304 状态摘要，不必再逐个打开 bundle manifest；正式 runbook 还会通过 `--require-phase-direct-live-access-log post-enable` 和 `--require-phase-direct-live-static-headers post-enable` 把缺少 request_id 对账摘要、缺少静态头摘要、摘要被跳过或摘要缺少缓存头 / 校验头 / Range / 304 证据的启用后证据判为失败，避免旧 post-enable 证据包混入最终归档。总审计的证据根目录默认只能包含带 `manifest.json` 的证据目录，夹带普通文件、无 manifest 子目录或符号链接都会失败，`--allow-extra-root-entries` 只用于人工排障显式放行，正式切换归档不应使用。总审计不只验文件 hash，还要求所有候选证据 `manifest.schemaVersion=1`，阶段证据 `manifest.summary.status=OK`，命令证据 `manifest.summary.status=OK`、`manifest.summary.exitCode=0` 且无 signal；命令证据的顶层 `manifest.commandName` 和内嵌 `manifest.command.name` 只要存在就必须各自是安全非空命令名，且两者同时存在时必须一致，否则按坏 manifest 处理；并在标准五段证据都被要求时校验每段审计状态都是 `OK`，以及 `pre-cutover -> enable-apply -> post-enable -> rollback-apply -> post-rollback` 的 `manifest.generatedAt` 顺序和默认 24 小时最大跨度，防止从不同切换窗口拼出一组看似完整的最新证据；缺失、非法或非规范格式的 `manifest.generatedAt` 会直接失败，不能靠目录 mtime 兜底；最终总审计还会在 `--require-command-arg` 下要求 enable / rollback 命令证据的 `manifest.command.args` 和独立 `command-record.json.args` 都包含 `--apply`，并在 `--require-cutover-run-id` 下要求所有阶段和命令证据的 `manifest.cutoverRunId` 与本次 runbook 一致；确需跨更长维护窗口时，只能在生成 runbook 时显式传 `--cutover-evidence-timeline-max-span-ms `，让最终总审计 JSON 可见本次放宽值；证据根目录、verifier 路径、阶段名、命令名或 cutover run id 不安全时直接失败。 @@ -102,7 +117,7 @@ npm run check:pingora-direct-preflight -- \ `--warn-only`、`--allow-extra-files` 和 `--allow-extra-root-entries` 只允许值班人员在正式 runbook 之外作为独立排障命令使用；`plan:pingora-direct-cutover` 生成的正式切换计划不得携带这些放行参数。若需要放行额外证据条目或忽略 `CRITICAL`，应先在证据根目录外保存人工说明、修正现场或重新生成正式证据，而不是把排障开关写进切换 runbook。 -`check:pingora-release-readiness` 是正式切换前的聚合门禁，默认串行执行 `cargo test -p pingora-gateway`、mock 上游 smoke、路由矩阵 parity、Nginx canary snippet 校验、Docker handoff smoke、canary access log 对账烟测、直连入口静态预检、直连启用 / 回退 dry-run 行为检查、current release 自审烟测、release readiness 计划自检、生产运维护栏、Pingora cutover 状态快照烟测、Pingora cutover 证据包烟测、Pingora cutover 命令证据烟测、Pingora cutover 证据 manifest 验真烟测、Pingora cutover 证据根目录审计烟测、API release build 烟测、Pingora production release 真实构建烟测和 API deploy release 烟测。普通本机执行时 Docker / Nginx 能力仍按子脚本默认口径跳过；切换窗口或 CI 必须执行： +`check:pingora-release-readiness` 是正式切换前的聚合门禁，默认串行执行 `cargo test -p pingora-gateway`、mock 上游 smoke、路由矩阵 parity、Nginx canary snippet 校验、Docker handoff smoke、canary access log 对账烟测、直连入口静态预检、直连启用 / 回退 dry-run 行为检查、current release 自审烟测、直连彩排状态烟测、release readiness 计划自检、生产运维护栏、Pingora cutover 状态快照烟测、Pingora cutover 证据包烟测、Pingora cutover 命令证据烟测、Pingora cutover 证据 manifest 验真烟测、Pingora cutover 证据根目录审计烟测、API release build 烟测、Pingora production release 真实构建烟测和 API deploy release 烟测。普通本机执行时 Docker / Nginx 能力仍按子脚本默认口径跳过；切换窗口或 CI 必须执行： 目标机 current release 上的启用前基础门禁和启用后 `--require-direct` 复核使用同一个聚合脚本，但必须追加 `--release-runtime-only`。该模式只执行发布包内可自包含的运行时复核：current release 自审、live canary、真实 access log 对账、direct preflight、health patrol env 复核和 direct live smoke；不会运行 Cargo、npm、Docker 或 Nginx 源码 / 构建环境门禁，并会拒绝 `--require-docker`、`--pull-docker` 与 `--require-nginx`。 @@ -223,7 +238,7 @@ npm run build:production-release -- --component api-server --name --in Jenkins `Genarrative-Api-Build` 对应参数是 `INCLUDE_PINGORA_GATEWAY`，默认关闭；勾选后才会归档 `pingora-gateway` 与 `pingora-gateway.sha256`，触发后续 `Genarrative-Api-Deploy` 时也会传递该布尔参数并复制这两个可选产物。无论是否打包 Pingora 二进制，API release 都必须携带 `build//scripts/deploy/production-api-deploy.sh`、同目录的 `maintenance-on.sh` / `maintenance-off.sh`、`/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh`、`pingora-direct-rollback.sh`、`pingora-tls-cert-sync.mjs`、`scripts/ops/pingora-current-release-audit.mjs`、`scripts/ops/pingora-cutover-status-snapshot.mjs`、`scripts/ops/pingora-cutover-evidence-bundle.mjs`、`scripts/ops/pingora-cutover-command-evidence.mjs`、`scripts/ops/pingora-cutover-evidence-verify.mjs`、`scripts/ops/pingora-cutover-evidence-audit.mjs`、`scripts/check-pingora-direct-preflight.mjs`、`scripts/check-pingora-direct-live.mjs`、`scripts/check-pingora-canary-access-log-parity.mjs`、`scripts/check-production-health-patrol-env.mjs`、`deploy/systemd/`、`deploy/env/` 和 `deploy/pingora/`。`Genarrative-Api-Deploy` 只能从上游构建归档复制并执行 `build//scripts/deploy/production-api-deploy.sh`，不能继续执行部署工作区根部脚本；维护脚本必须与 deploy 脚本来自同一发布包同一目录，避免 Jenkins 工作区里的旧脚本掩盖 release 包布局缺陷。直连启用脚本从 current release 执行时默认读取 `/opt/genarrative/current/deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`，必须能自包含完成 preflight、systemd 模板读取和 direct live smoke，不依赖 Jenkins 工作区、目标机源码 checkout 或 `/etc` 参考模板；current release 自审、TLS 证书同步、health patrol env 复核、canary access log 对账、canary live、direct live、状态快照、证据包、命令证据、证据验真和证据根目录审计也必须来自 current release 随包脚本与 `deploy/env/`。`/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 只作为 Server-Provision 安装的人工审阅 / 手动覆盖模板；确需使用时显式传 `--template-path` 或 `GENARRATIVE_PINGORA_DIRECT_TEMPLATE_PATH`。`npm run check:production-api-release` 会用临时 `CARGO_TARGET_DIR` 和假 `api-server` / `pingora-gateway` release binary 跑 `build-production-release.sh --component api-server --skip-api-build`，动态验证默认 API release 不登记 Pingora，显式 `--include-pingora-gateway --skip-pingora-gateway-build` 时则必须包含 `pingora-gateway`、`pingora-gateway.sha256` 并写入 manifest，同时验证 Pingora 直连依赖、TLS 证书同步脚本、current release 自审脚本、状态快照脚本、证据包脚本、命令证据脚本、证据验真脚本、证据根目录审计脚本、canary access log 对账脚本、健康巡检 env 复核脚本、env 示例目录、API deploy 执行入口自包含和 `deploy/pingora/pingora-gateway.env.example` 的生产安全默认值：gzip-only、`TRUST_X_FORWARDED_FOR=false`、`TRUSTED_FRONT_PROXY_CONFIRMED=false`、接流保护默认开启、probe token 为空；该检查还会读取发布包 README，并直接运行发布包内 `scripts/check-pingora-release-readiness.mjs --dry-run-cutover`，确认最终证据根目录总审计步骤仍带两条 `--require-command-executable ... /opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh` / `pingora-direct-rollback.sh` current release 脚本身份要求。`npm run check:pingora-production-release-build` 会用假 `api-server`、临时 `CARGO_TARGET_DIR` 和真实 `cargo build -p pingora-gateway --release --target x86_64-unknown-linux-gnu` 验证显式 include 路径能构出可执行 `pingora-gateway`、checksum 和 manifest 登记，避免只靠假二进制布局 smoke。`production-api-deploy.sh` 对这些 Pingora 直连依赖采取 fail-fast：上游发布产物缺失时保持维护模式并停止部署，不再从部署工作区兜底复制；`release-manifest.json` 必须存在且登记 `api-server` artifact，若发布包包含 `pingora-gateway`，manifest 也必须登记 `pingora-gateway` artifact，否则部署会在切换 current 前失败。API deploy 要求 `--release-root`、`--current-link`、`--api-env-file` 使用绝对路径，`--version` 必须以数字或字母开头并拒绝点目录，会先写 `${RELEASE_ROOT}/.${VERSION}.staging.$`，全部校验和复制完成后用非合并语义提升为 `${RELEASE_ROOT}/${VERSION}`，再用固定替换语义切换 current 符号链接；同版本正式 release 已存在、提升前竞态出现或 current 路径不是符号链接时都会拒绝合并 / 覆盖，失败时清理 staging 并保持维护模式。本机用 `npm run check:production-api-deploy` 通过临时 release、fake `systemctl` / `curl` 动态验证从发布产物内执行 deploy 脚本后 current release 自洽，并覆盖缺少 TLS 证书同步脚本、current release 自审脚本、状态快照脚本、证据包脚本、证据验真脚本、证据根目录审计脚本、canary access log 对账脚本、健康巡检 env 复核脚本、`deploy/env/`、release manifest、api-server manifest artifact、Pingora manifest artifact、相对 release root / current link / api env file、点目录或点开头 version、同版本 release 目录已存在、current 路径不是符号链接或提升前 release 目录竞态出现时必须失败，同时复核 current release 内 Pingora env 示例仍保持同一组生产安全默认值。 -API release 还必须携带 `scripts/check-pingora-release-readiness.mjs` 与 `scripts/check-pingora-canary-live.mjs`。前者支撑 current release 的 `--release-runtime-only` 聚合复核，后者支撑目标 Nginx canary live smoke；缺少任一脚本时 `check:production-api-release`、`check:production-api-deploy` 和生产运维护栏都必须失败。 +API release 还必须携带 `scripts/check-pingora-release-readiness.mjs`、`scripts/check-pingora-canary-live.mjs` 与 `scripts/ops/pingora-direct-rehearsal-status.mjs`。前者支撑 current release 的 `--release-runtime-only` 聚合复核，canary live 脚本支撑目标 Nginx canary live smoke，直连彩排状态脚本支撑目标机切换前只读确认 Nginx 仍接公网、Pingora shadow / realpath canary 高端口和 current release 自审均可用；缺少任一脚本时 `check:production-api-release`、`check:production-api-deploy` 和生产运维护栏都必须失败。 发布包会额外包含 `pingora-gateway` 与 `pingora-gateway.sha256`。`production-api-deploy.sh` 看到这两个文件时会校验并把 `pingora-gateway` / `pingora-gateway.sha256` 一起复制到 `/opt/genarrative/current`，供 shadow systemd 模板和 current release 自审使用；`api-server.sha256` 也必须随 `api-server` 一起进入 current release。没有这两个 Pingora 文件时现有 API 发布行为不变。API 发布仍只重启 `genarrative-api.service`；如果 `genarrative-pingora-gateway.service` 已经处于 `active`，部署脚本会在 `current` 链接切换后执行 `systemctl try-restart genarrative-pingora-gateway.service` 并复核仍为 `active`，让已启用的 shadow / canary / direct 机器加载同一份 current release 网关二进制。若 Pingora 服务未运行，部署脚本只打印跳过信息，不主动拉起影子服务，也不改变默认接流边界。 @@ -481,7 +496,7 @@ dev 根盘空间在安装后曾接近满盘；2026-06-17 进入 canary 前已清 5. 前缀 canary 通过后，再使用 current release 随包 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-enable.sh --apply --probe-token --host <域名> --base-url http://127.0.0.1:18083` 启用真实路径 canary；它会把 `deploy/nginx/snippets/genarrative-pingora-realpath-canary.conf` 渲染成独立本机 `server`，写入 `/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf`，不能 include 到生产 `443` server 内。该片段使用 `access_log ... genarrative_upstream`，文件名必须保证晚于定义 `log_format genarrative_upstream` 的主站配置加载；否则 `nginx -t` 会报 `unknown log format "genarrative_upstream"`。启用脚本会先执行 `nginx -t`、reload Nginx，再默认运行 realpath live smoke，任一阶段失败都会恢复写入前配置。关闭时执行 `/opt/genarrative/current/scripts/deploy/pingora-realpath-canary-disable.sh --apply`，脚本会在 `nginx -t` 或 reload 失败时恢复删除前配置。启用后跑 `node -- /opt/genarrative/current/scripts/check-pingora-canary-live.mjs --realpath --base-url http://127.0.0.1:18083 --host <域名>`，再用 current release 随包 access log parity 脚本传 `--realpath --nginx-log-file /var/log/nginx/genarrative-pingora-realpath-canary.access.log` 对账 `/api/creation-entry/config`、`/v1/identity` 和 `/assets/app.js` 等真实路径。 6. 目标机 canary include 后必须跑正式切换聚合门禁：源码 checkout / CI / 构建环境执行默认全量命令 `node scripts/check-pingora-release-readiness.mjs --require-docker --pull-docker --require-nginx --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`；目标机 current release 执行 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only --require-live --live-base-url http://127.0.0.1 --live-host <域名> --live-nginx-access-log /var/log/nginx/genarrative.access.log --live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。启用真实路径 canary 时，两种命令都追加 `--require-realpath-live --realpath-live-base-url http://127.0.0.1:18083 --realpath-live-host <域名> --realpath-live-nginx-access-log /var/log/nginx/genarrative-pingora-realpath-canary.access.log --realpath-live-pingora-access-log /var/log/genarrative/pingora-gateway.access.log`。缺少 Host 会直接失败，避免 live canary 误测默认 vhost；live smoke 后还会按 `request_id` 对账 Nginx 与 Pingora access log，缺少同一请求的 Pingora 日志、状态码、方法或 path 漂移都会失败。 7. 如需评估 Pingora 直连公网入口，必须显式配置 `TLS_LISTEN`、证书、私钥和 `HTTP_REDIRECT_LISTEN`；Certbot 证书先用随包 `scripts/deploy/pingora-tls-cert-sync.mjs` 同步到 `/etc/genarrative/pingora-tls/<域名>/`，不要直接 chmod Let’s Encrypt live/archive 原路径；绑定 `80/443` 时还必须人工启用 `genarrative-pingora-gateway-direct-entry.conf` drop-in 授予 `CAP_NET_BIND_SERVICE`。随后用 `npm run check:pingora-gateway-smoke` 覆盖 TLS / HTTP/2 ALPN / redirect / WSS subscribe；目标机必须先跑 `npm run check:pingora-direct-preflight -- --env-file /etc/genarrative/pingora-gateway.env --require-live-env --systemd-cat --check-cert-readable --check-service-env-file --check-service-user-cert-readable --check-service-binary-executable --check-ports-free`，再跑 `npm run check:pingora-direct-live` 或 release readiness 的 `--require-direct`，且 `--require-direct` 必须带 direct HTTPS base URL、direct HTTP base URL、正式域名 Host/SNI、redirect Location host、Pingora access log 文件、health patrol env 文件、direct preflight env 文件、systemd drop-in 生效检查、service EnvironmentFile 一致性检查、当前用户和 systemd 服务用户证书可读检查、service 二进制可执行检查、端口释放检查和显式 SpacetimeDB 数据库名，并会拒绝 `--skip-wss`。高端口 rehearsal 使用 `https://127.0.0.1:<高端口>` 打入但期望 HTTP redirect Location 指向正式域名默认 HTTPS 入口时，额外传 `--direct-redirect-base-url https://<域名>`；`--direct-redirect-host` 仍必须保留，用于 runbook Host 一致性约束。direct live 会用生成的 `request_id` 反查 Pingora access log；缺少对应日志、method 漂移、path 漂移或 status 漂移都算直连门禁失败。direct preflight 会拒绝开启网关保护但未确认共享保护层的 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT>1` 配置；`--env-file`、`--systemd-service`、服务用户和 env 中的 listen / cert / key 值都不能包含换行或 NUL，执行 `systemctl cat` 或 `sudo -u test -r ` 前还会复核子命令参数，避免污染参数进入目标机预检命令；direct live 的 URL、Host、redirect base URL、probe token、额外 path、数据库名、access log 路径、timeout 和布尔 env 也不能包含换行或 NUL，且会在发起请求前失败；direct live timeout 必须是正整数，直连相关布尔 env 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。证书申请与续期仍由 Certbot / 外部自动化承担，网关只读取现有文件。 -8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs`、realpath canary 启停脚本和 `deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release；缺失时部署应 fail-fast，切换窗口不能依赖源码 checkout 或 Jenkins workspace。 +8. 正式直连 runbook 的启用前基础门禁和启用后 `--require-direct` 复核必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only`。该脚本、`scripts/check-pingora-canary-live.mjs`、`scripts/ops/pingora-direct-rehearsal-status.mjs`、realpath canary 启停脚本和 `deploy/nginx/` 必须进入生产 API release、Jenkins API Build 归档、Jenkins API Deploy 复制清单和目标机 current release；缺失时部署应 fail-fast，切换窗口不能依赖源码 checkout 或 Jenkins workspace。 9. direct preflight / direct live timeout 必须是正整数；直连相关布尔 env 和 Pingora gateway env 里的 `TRUST_X_FORWARDED_FOR` / `TRUSTED_FRONT_PROXY_CONFIRMED` 只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败。 10. Pingora 正式化口径固定为 gzip-only；Brotli 继续由 Nginx / 前置代理承担，直连 Pingora 不以 Brotli parity 作为切换门禁。如需多实例，开启网关接流保护时必须先引入共享限流 / 共享并发保护层并设置 `GENARRATIVE_PINGORA_GATEWAY_SHARED_PROTECTION_CONFIRMED=true`，否则保持 `GENARRATIVE_PINGORA_GATEWAY_INSTANCE_COUNT=1`；关闭网关保护的多实例方案必须明确由前置 Nginx / LB 承担全局限流。 10. 前缀 canary 稳定后，再评估是否做真实路径 canary；真实路径 canary 稳定后，再评估是否让 Pingora 直接承接公网入口。 diff --git a/jenkins/Jenkinsfile.production-api-build b/jenkins/Jenkinsfile.production-api-build index ba7a9cf2e..ace9ca71a 100644 --- a/jenkins/Jenkinsfile.production-api-build +++ b/jenkins/Jenkinsfile.production-api-build @@ -123,7 +123,7 @@ pipeline { stage('Archive') { steps { - archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/api-server,build/${env.EFFECTIVE_BUILD_VERSION}/api-server.sha256,build/${env.EFFECTIVE_BUILD_VERSION}/release-manifest.json,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/systemd/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/nginx/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/pingora/**", fingerprint: true + archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/api-server,build/${env.EFFECTIVE_BUILD_VERSION}/api-server.sha256,build/${env.EFFECTIVE_BUILD_VERSION}/release-manifest.json,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-direct-rehearsal-status.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${env.EFFECTIVE_BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/systemd/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/nginx/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/env/**,build/${env.EFFECTIVE_BUILD_VERSION}/deploy/pingora/**", fingerprint: true script { if (params.INCLUDE_PINGORA_GATEWAY) { archiveArtifacts artifacts: "build/${env.EFFECTIVE_BUILD_VERSION}/pingora-gateway,build/${env.EFFECTIVE_BUILD_VERSION}/pingora-gateway.sha256", fingerprint: true diff --git a/jenkins/Jenkinsfile.production-api-deploy b/jenkins/Jenkinsfile.production-api-deploy index 50bee6574..7f60e03a9 100644 --- a/jenkins/Jenkinsfile.production-api-deploy +++ b/jenkins/Jenkinsfile.production-api-deploy @@ -66,7 +66,7 @@ pipeline { copyArtifacts( projectName: params.BUILD_JOB_NAME, selector: specific(params.BUILD_NUMBER_TO_DEPLOY), - filter: "build/${params.BUILD_VERSION}/api-server,build/${params.BUILD_VERSION}/api-server.sha256,build/${params.BUILD_VERSION}/release-manifest.json,build/${params.BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${params.BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${params.BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${params.BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${params.BUILD_VERSION}/deploy/systemd/**,build/${params.BUILD_VERSION}/deploy/nginx/**,build/${params.BUILD_VERSION}/deploy/env/**,build/${params.BUILD_VERSION}/deploy/pingora/**", + filter: "build/${params.BUILD_VERSION}/api-server,build/${params.BUILD_VERSION}/api-server.sha256,build/${params.BUILD_VERSION}/release-manifest.json,build/${params.BUILD_VERSION}/scripts/database-backup-to-oss.mjs,build/${params.BUILD_VERSION}/scripts/ops/production-health-patrol.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-current-release-audit.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-direct-rehearsal-status.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-status-snapshot.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-bundle.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-command-evidence.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-verify.mjs,build/${params.BUILD_VERSION}/scripts/ops/pingora-cutover-evidence-audit.mjs,build/${params.BUILD_VERSION}/scripts/check-production-health-patrol-env.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-release-readiness.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-preflight.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-direct-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-live.mjs,build/${params.BUILD_VERSION}/scripts/check-pingora-canary-access-log-parity.mjs,build/${params.BUILD_VERSION}/scripts/deploy/production-api-deploy.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-on.sh,build/${params.BUILD_VERSION}/scripts/deploy/maintenance-off.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh,build/${params.BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs,build/${params.BUILD_VERSION}/deploy/systemd/**,build/${params.BUILD_VERSION}/deploy/nginx/**,build/${params.BUILD_VERSION}/deploy/env/**,build/${params.BUILD_VERSION}/deploy/pingora/**", target: '.', fingerprintArtifacts: true ) diff --git a/package.json b/package.json index 2e715da20..f8e01769f 100644 --- a/package.json +++ b/package.json @@ -47,6 +47,7 @@ "check:pingora-health-patrol-env-switch": "node scripts/check-pingora-health-patrol-env-switch.mjs", "check:pingora-tls-cert-sync": "node scripts/check-pingora-tls-cert-sync.mjs", "check:pingora-current-release-audit": "node scripts/check-pingora-current-release-audit.mjs", + "check:pingora-direct-rehearsal-status": "node scripts/check-pingora-direct-rehearsal-status.mjs", "check:pingora-cutover-status-snapshot": "node scripts/check-pingora-cutover-status-snapshot.mjs", "check:pingora-cutover-evidence-bundle": "node scripts/check-pingora-cutover-evidence-bundle.mjs", "check:pingora-cutover-command-evidence": "node scripts/check-pingora-cutover-command-evidence.mjs", diff --git a/scripts/build-production-release.sh b/scripts/build-production-release.sh index b5a007db9..017af9a8b 100644 --- a/scripts/build-production-release.sh +++ b/scripts/build-production-release.sh @@ -552,6 +552,7 @@ copy_required_file "${SCRIPT_DIR}/spacetime-revoke-migration-operator.mjs" "${TA copy_required_file "${SCRIPT_DIR}/database-backup-to-oss.mjs" "${TARGET_DIR}/scripts/database-backup-to-oss.mjs" "数据库 OSS 备份脚本" copy_required_file "${SCRIPT_DIR}/ops/production-health-patrol.mjs" "${TARGET_DIR}/scripts/ops/production-health-patrol.mjs" "生产健康巡检脚本" copy_required_file "${SCRIPT_DIR}/ops/pingora-current-release-audit.mjs" "${TARGET_DIR}/scripts/ops/pingora-current-release-audit.mjs" "Pingora current release 自审脚本" +copy_required_file "${SCRIPT_DIR}/ops/pingora-direct-rehearsal-status.mjs" "${TARGET_DIR}/scripts/ops/pingora-direct-rehearsal-status.mjs" "Pingora 直连切换彩排状态脚本" copy_required_file "${SCRIPT_DIR}/ops/pingora-cutover-status-snapshot.mjs" "${TARGET_DIR}/scripts/ops/pingora-cutover-status-snapshot.mjs" "Pingora 直连切换状态快照脚本" copy_required_file "${SCRIPT_DIR}/ops/pingora-cutover-evidence-bundle.mjs" "${TARGET_DIR}/scripts/ops/pingora-cutover-evidence-bundle.mjs" "Pingora 直连切换证据包脚本" copy_required_file "${SCRIPT_DIR}/ops/pingora-cutover-command-evidence.mjs" "${TARGET_DIR}/scripts/ops/pingora-cutover-command-evidence.mjs" "Pingora 直连切换命令证据脚本" @@ -584,7 +585,7 @@ cat >"${TARGET_DIR}/README.md" < item.path === 'scripts/ops/pingora-direct-rehearsal-status.mjs', + ); + if (!rehearsalStatusArtifact || rehearsalStatusArtifact.status !== 'OK') { + failures.push('快照必须确认 current release 已包含直连彩排状态脚本。'); + } const evidenceAuditArtifact = payload.releaseArtifacts.artifacts.find( (item) => item.path === 'scripts/ops/pingora-cutover-evidence-audit.mjs', ); @@ -727,6 +738,7 @@ function prepareReleaseRoot(releaseRoot) { for (const file of [ 'scripts/database-backup-to-oss.mjs', 'scripts/ops/pingora-current-release-audit.mjs', + 'scripts/ops/pingora-direct-rehearsal-status.mjs', 'scripts/ops/pingora-cutover-evidence-audit.mjs', 'scripts/ops/pingora-cutover-evidence-bundle.mjs', 'scripts/ops/pingora-cutover-command-evidence.mjs', diff --git a/scripts/check-pingora-direct-rehearsal-status.mjs b/scripts/check-pingora-direct-rehearsal-status.mjs new file mode 100644 index 000000000..0cbf8bbda --- /dev/null +++ b/scripts/check-pingora-direct-rehearsal-status.mjs @@ -0,0 +1,566 @@ +#!/usr/bin/env node + +import { spawnSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { + chmodSync, + copyFileSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; + +const STATUS_SCRIPT = 'scripts/ops/pingora-direct-rehearsal-status.mjs'; +const failures = []; +const tmpRoot = mkdtempSync( + path.join(tmpdir(), 'genarrative-pingora-direct-rehearsal-'), +); + +try { + main(); +} finally { + rmSync(tmpRoot, { recursive: true, force: true }); +} + +if (failures.length > 0) { + console.error('[check:pingora-direct-rehearsal-status] FAILED'); + for (const failure of failures) { + console.error(`- ${failure}`); + } + process.exit(1); +} + +console.log('[check:pingora-direct-rehearsal-status] OK'); + +function main() { + assertScriptShape(); + assertNginxPublicRealpathRehearsalPasses(); + assertFailsWhenPublicPortsAlreadyOwnedByPingora(); + assertFailsWhenRealpathCanaryMissing(); + assertRejectsUnsafeArgs(); + assertRejectsInvalidBoolEnv(); +} + +function assertScriptShape() { + const content = readFileSync(STATUS_SCRIPT, 'utf8'); + assertIncludes( + content, + '该脚本只读采集', + 'rehearsal 状态脚本 usage 必须说明只读边界。', + ); + assertIncludes( + content, + 'systemctl', + 'rehearsal 状态脚本必须读取 systemd 状态。', + ); + assertIncludes( + content, + 'ss', + 'rehearsal 状态脚本必须读取端口监听状态。', + ); + assertIncludes( + content, + 'pingora-current-release-audit.mjs', + 'rehearsal 状态脚本必须复用 current release 自审。', + ); + if ( + content.includes('writeFile') || + content.includes('rmSync(') || + content.includes('systemctl reload') || + content.includes('daemon-reload') + ) { + failures.push('rehearsal 状态脚本不应写文件、删除文件或 reload systemd。'); + } +} + +function assertNginxPublicRealpathRehearsalPasses() { + const fixture = prepareFixture('nginx-public-realpath', { + ssMode: 'nginx-public-realpath', + }); + const result = runStatus(fixture, [ + '--expect-public-gateway', + 'nginx', + '--require-realpath-canary', + '--require-pingora-shadow', + '--require-current-release-gateway', + '--fail-on-critical', + ]); + + assertStatus(result, 0, 'Nginx 接公网 + Pingora shadow + realpath canary 应通过。'); + if (result.status !== 0) { + return; + } + const payload = parseJson(result.stdout, 'rehearsal 状态输出'); + assertEqual(payload.summary.status, 'OK', 'rehearsal summary 应为 OK。'); + assertEqual( + payload.publicBoundary.status, + 'OK', + 'Nginx 接公网时 public boundary 应通过。', + ); + assertEqual( + payload.ports.byPort['80'].hasNginx, + true, + '80 端口必须识别为 Nginx。', + ); + assertEqual( + payload.ports.byPort['443'].hasNginx, + true, + '443 端口必须识别为 Nginx。', + ); + assertEqual( + payload.ports.byPort['18081'].hasPingora, + true, + '18081 必须识别为 Pingora shadow。', + ); + assertEqual( + payload.ports.byPort['18083'].hasNginx, + true, + '18083 必须识别为 Nginx realpath canary。', + ); + assertEqual( + payload.realpathCanary.templateLooksValid, + true, + 'realpath canary 配置必须通过关键片段检查。', + ); + assertEqual( + payload.systemd.hasAmbientCapability, + false, + 'Nginx 接公网彩排阶段不得有 Pingora 低端口 capability。', + ); + assertEqual( + payload.healthPatrolEnv.values.gatewayMode, + 'nginx', + 'Nginx 接公网彩排阶段 health patrol gateway mode 应为 nginx。', + ); + const commandsLog = readFileSync(fixture.commandsLog, 'utf8'); + assertIncludes( + commandsLog, + 'systemctl is-active nginx.service', + 'rehearsal 状态必须读取 Nginx service 状态。', + ); + assertIncludes( + commandsLog, + 'ss -H -ltnp', + 'rehearsal 状态必须读取端口监听。', + ); +} + +function assertFailsWhenPublicPortsAlreadyOwnedByPingora() { + const fixture = prepareFixture('pingora-public', { + ssMode: 'pingora-public', + }); + const result = runStatus(fixture, [ + '--expect-public-gateway', + 'nginx', + '--require-pingora-shadow', + '--fail-on-critical', + ]); + + assertStatus(result, 1, '期望 Nginx 接公网但 80/443 由 Pingora 监听时必须失败。'); + const payload = parseJson(result.stdout, 'Pingora 抢占公网端口状态输出'); + assertEqual( + payload.publicBoundary.status, + 'CRITICAL', + '公网端口归属漂移时 public boundary 必须 CRITICAL。', + ); + assertIncludes( + payload.publicBoundary.diagnostics.join('\n'), + '已被 Pingora 监听', + '公网端口归属漂移必须给出明确诊断。', + ); +} + +function assertFailsWhenRealpathCanaryMissing() { + const fixture = prepareFixture('missing-realpath', { + ssMode: 'nginx-public-shadow-only', + writeRealpathConfig: false, + }); + const result = runStatus(fixture, [ + '--expect-public-gateway', + 'nginx', + '--require-realpath-canary', + '--fail-on-critical', + ]); + + assertStatus(result, 1, '要求 realpath canary 但配置或端口缺失时必须失败。'); + const payload = parseJson(result.stdout, '缺少 realpath canary 状态输出'); + assertEqual( + payload.realpathCanary.status, + 'CRITICAL', + '缺少 realpath canary 时必须 CRITICAL。', + ); + assertIncludes( + payload.realpathCanary.diagnostics.join('\n'), + 'realpath canary 配置不存在', + '缺少 realpath canary 配置时必须给出明确诊断。', + ); +} + +function assertRejectsUnsafeArgs() { + const fixture = prepareFixture('unsafe'); + const cases = [ + { + args: ['--release-root', 'relative'], + expected: '--release-root 必须是绝对路径', + reason: 'release root 相对路径必须失败。', + }, + { + args: ['--release-root', '/'], + expected: '--release-root 不能是文件系统根目录', + reason: 'release root 为文件系统根目录必须失败。', + }, + { + args: ['--expect-public-gateway', 'apache'], + expected: '--expect-public-gateway 只支持 none / nginx / pingora-direct', + reason: '未知 public gateway 必须失败。', + }, + { + args: ['--timeout-ms', '0'], + expected: '--timeout-ms 必须是正整数', + reason: 'timeout 非正整数必须失败。', + }, + ]; + + for (const testCase of cases) { + const result = runStatus(fixture, testCase.args); + if ((result.status ?? 0) === 0) { + failures.push(testCase.reason); + } + assertIncludes( + `${result.stdout}\n${result.stderr}`, + testCase.expected, + `${testCase.reason} 必须输出明确错误。`, + ); + } +} + +function assertRejectsInvalidBoolEnv() { + const fixture = prepareFixture('invalid-bool'); + const result = runStatus(fixture, [], { + GENARRATIVE_PINGORA_REHEARSAL_REQUIRE_REALPATH_CANARY: 'maybe', + }); + if ((result.status ?? 0) === 0) { + failures.push('非法布尔 env 必须失败。'); + } + assertIncludes( + `${result.stdout}\n${result.stderr}`, + 'GENARRATIVE_PINGORA_REHEARSAL_REQUIRE_REALPATH_CANARY 必须是布尔值', + '非法布尔 env 必须输出明确错误。', + ); +} + +function prepareFixture(name, options = {}) { + const root = path.join(tmpRoot, name); + const releaseRoot = path.join(root, 'current'); + const fakeBin = path.join(root, 'bin'); + const commandsLog = path.join(root, 'commands.log'); + const healthPatrolEnvFile = path.join(root, 'etc', 'health-patrol.env'); + const pingoraEnvFile = path.join(root, 'etc', 'pingora-gateway.env'); + const realpathConfigFile = path.join( + root, + 'nginx', + 'zz-genarrative-pingora-realpath-canary.conf', + ); + + mkdirSync(releaseRoot, { recursive: true }); + mkdirSync(fakeBin, { recursive: true }); + mkdirSync(path.dirname(healthPatrolEnvFile), { recursive: true }); + mkdirSync(path.dirname(realpathConfigFile), { recursive: true }); + + prepareReleaseRoot(releaseRoot); + writeFileSync( + healthPatrolEnvFile, + [ + 'GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=nginx', + 'GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL=http://127.0.0.1', + 'GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST=', + 'GENARRATIVE_HEALTH_PATROL_PINGORA_PROBE_TOKEN=health-secret-token', + '', + ].join('\n'), + 'utf8', + ); + writeFileSync( + pingoraEnvFile, + [ + 'GENARRATIVE_PINGORA_GATEWAY_LISTEN=127.0.0.1:18081', + 'GENARRATIVE_PINGORA_GATEWAY_PROBE_TOKEN=pingora-secret-token', + 'GENARRATIVE_PINGORA_GATEWAY_ACCESS_LOG_FILE=/var/log/genarrative/pingora-gateway.access.log', + 'GENARRATIVE_PINGORA_GATEWAY_COMPRESSION_ALGORITHMS=gzip', + 'GENARRATIVE_PINGORA_GATEWAY_TRUST_X_FORWARDED_FOR=false', + '', + ].join('\n'), + 'utf8', + ); + if (options.writeRealpathConfig !== false) { + writeFileSync( + realpathConfigFile, + [ + 'server {', + ' listen 127.0.0.1:18083;', + ' access_log /var/log/nginx/genarrative-pingora-realpath-canary.access.log genarrative_upstream;', + ' add_header X-Genarrative-Nginx-Handoff pingora-realpath-canary always;', + '}', + '', + ].join('\n'), + 'utf8', + ); + } + + writeFileSync( + path.join(fakeBin, 'systemctl'), + [ + '#!/usr/bin/env bash', + 'set -euo pipefail', + `printf 'systemctl %s\\n' "$*" >> ${shellQuote(commandsLog)}`, + 'if [[ "$1" == "is-active" ]]; then', + ' printf "active\\n"', + ' exit 0', + 'fi', + 'if [[ "$1" == "cat" && "${2:-}" == "genarrative-pingora-gateway.service" ]]; then', + ' cat <> ${shellQuote(commandsLog)}`, + `mode="${options.ssMode || 'nginx-public-realpath'}"`, + 'case "${mode}" in', + ' nginx-public-realpath)', + ' cat <&2 + exit 1 +fi +cp "${PINGORA_DIRECT_REHEARSAL_STATUS_SCRIPT_SOURCE}" "${RELEASE_CONTENT_DIR}/scripts/ops/pingora-direct-rehearsal-status.mjs" +chmod 0644 "${RELEASE_CONTENT_DIR}/scripts/ops/pingora-direct-rehearsal-status.mjs" if [[ ! -f "${PINGORA_CUTOVER_STATUS_SNAPSHOT_SCRIPT_SOURCE}" ]]; then echo "[production-api-deploy] 发布产物缺少 Pingora 直连切换状态快照脚本: ${SOURCE_DIR}/scripts/ops/pingora-cutover-status-snapshot.mjs" >&2 exit 1 diff --git a/scripts/ops/pingora-current-release-audit.mjs b/scripts/ops/pingora-current-release-audit.mjs index 0f84584a3..88f9a1e55 100644 --- a/scripts/ops/pingora-current-release-audit.mjs +++ b/scripts/ops/pingora-current-release-audit.mjs @@ -11,6 +11,7 @@ const REQUIRED_ARTIFACTS = [ { path: 'scripts/database-backup-to-oss.mjs' }, { path: 'scripts/ops/production-health-patrol.mjs' }, { path: 'scripts/ops/pingora-current-release-audit.mjs' }, + { path: 'scripts/ops/pingora-direct-rehearsal-status.mjs' }, { path: 'scripts/ops/pingora-cutover-status-snapshot.mjs' }, { path: 'scripts/ops/pingora-cutover-evidence-bundle.mjs' }, { path: 'scripts/check-production-health-patrol-env.mjs' }, diff --git a/scripts/ops/pingora-cutover-status-snapshot.mjs b/scripts/ops/pingora-cutover-status-snapshot.mjs index 34e001217..2ae7bb3d9 100644 --- a/scripts/ops/pingora-cutover-status-snapshot.mjs +++ b/scripts/ops/pingora-cutover-status-snapshot.mjs @@ -21,6 +21,7 @@ const SERVICES = [ const RELEASE_ARTIFACTS = [ 'scripts/ops/pingora-cutover-status-snapshot.mjs', 'scripts/ops/pingora-current-release-audit.mjs', + 'scripts/ops/pingora-direct-rehearsal-status.mjs', 'scripts/ops/pingora-cutover-evidence-audit.mjs', 'scripts/ops/pingora-cutover-evidence-bundle.mjs', 'scripts/ops/pingora-cutover-command-evidence.mjs', diff --git a/scripts/ops/pingora-direct-rehearsal-status.mjs b/scripts/ops/pingora-direct-rehearsal-status.mjs new file mode 100644 index 000000000..91dfe13bd --- /dev/null +++ b/scripts/ops/pingora-direct-rehearsal-status.mjs @@ -0,0 +1,966 @@ +#!/usr/bin/env node + +import { execFile } from 'node:child_process'; +import { constants as fsConstants } from 'node:fs'; +import { access, readFile, stat } from 'node:fs/promises'; +import path from 'node:path'; + +const STATUS_RANK = { + OK: 0, + WARNING: 1, + CRITICAL: 2, +}; + +const PUBLIC_GATEWAYS = new Set(['none', 'nginx', 'pingora-direct']); +const GATEWAY_MODES = new Set(['nginx', 'pingora-direct']); +const DEFAULT_PORTS = [80, 443, 18081, 18083]; +const SECRET_ENV_KEY_PATTERN = + /(?:TOKEN|SECRET|PASSWORD|PRIVATE_KEY|ACCESS_KEY|API_KEY|AUTH|CREDENTIAL)/iu; + +const config = parseArgs(process.argv.slice(2)); +const status = await buildStatus(config); + +console.log(`${JSON.stringify(status, null, 2)}\n`); + +if (config.failOnCritical && status.summary.status === 'CRITICAL') { + process.exit(1); +} + +function parseArgs(argv) { + const result = { + releaseRoot: + process.env.GENARRATIVE_PINGORA_REHEARSAL_RELEASE_ROOT || + '/opt/genarrative/current', + healthPatrolEnvFile: + process.env.GENARRATIVE_HEALTH_PATROL_ENV_FILE || + '/etc/genarrative/health-patrol.env', + pingoraEnvFile: + process.env.GENARRATIVE_PINGORA_DIRECT_PREFLIGHT_ENV_FILE || + '/etc/genarrative/pingora-gateway.env', + realpathConfigFile: + process.env.GENARRATIVE_PINGORA_REALPATH_CANARY_CONFIG_FILE || + '/etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf', + expectedPublicGateway: + process.env.GENARRATIVE_PINGORA_REHEARSAL_EXPECT_PUBLIC_GATEWAY || + 'none', + expectedHealthPatrolGatewayMode: + process.env.GENARRATIVE_HEALTH_PATROL_EXPECTED_GATEWAY_MODE || '', + requireRealpathCanary: readBoolEnv( + 'GENARRATIVE_PINGORA_REHEARSAL_REQUIRE_REALPATH_CANARY', + ), + requirePingoraShadow: readBoolEnv( + 'GENARRATIVE_PINGORA_REHEARSAL_REQUIRE_PINGORA_SHADOW', + ), + requireCurrentReleaseGateway: readBoolEnv( + 'GENARRATIVE_PINGORA_REHEARSAL_REQUIRE_CURRENT_RELEASE_GATEWAY', + ), + timeoutMs: parseOptionalPositiveInt( + process.env.GENARRATIVE_PINGORA_REHEARSAL_TIMEOUT_MS, + 5000, + 'GENARRATIVE_PINGORA_REHEARSAL_TIMEOUT_MS', + ), + failOnCritical: readBoolEnv( + 'GENARRATIVE_PINGORA_REHEARSAL_FAIL_ON_CRITICAL', + ), + }; + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + switch (arg) { + case '-h': + case '--help': + printUsage(); + process.exit(0); + break; + case '--release-root': + result.releaseRoot = requireValue(argv, ++index, arg); + break; + case '--health-patrol-env-file': + result.healthPatrolEnvFile = requireValue(argv, ++index, arg); + break; + case '--pingora-env-file': + result.pingoraEnvFile = requireValue(argv, ++index, arg); + break; + case '--realpath-config-file': + result.realpathConfigFile = requireValue(argv, ++index, arg); + break; + case '--expect-public-gateway': + result.expectedPublicGateway = requireValue(argv, ++index, arg); + break; + case '--expected-health-patrol-gateway-mode': + result.expectedHealthPatrolGatewayMode = requireValue( + argv, + ++index, + arg, + ); + break; + case '--require-realpath-canary': + result.requireRealpathCanary = true; + break; + case '--require-pingora-shadow': + result.requirePingoraShadow = true; + break; + case '--require-current-release-gateway': + result.requireCurrentReleaseGateway = true; + break; + case '--timeout-ms': + result.timeoutMs = parseRequiredPositiveInt( + requireValue(argv, ++index, arg), + arg, + ); + break; + case '--fail-on-critical': + result.failOnCritical = true; + break; + default: + throw new Error(`未知参数: ${arg}`); + } + } + + validateConfig(result); + return result; +} + +function printUsage() { + console.log(`Usage: + node scripts/ops/pingora-direct-rehearsal-status.mjs [options] + +Options: + --release-root current release 根目录，默认 /opt/genarrative/current。 + --health-patrol-env-file health-patrol env 文件，默认 /etc/genarrative/health-patrol.env。 + --pingora-env-file pingora-gateway env 文件，默认 /etc/genarrative/pingora-gateway.env。 + --realpath-config-file realpath canary Nginx 配置，默认 /etc/nginx/conf.d/zz-genarrative-pingora-realpath-canary.conf。 + --expect-public-gateway 可选，none / nginx / pingora-direct；nginx 模式会要求 80/443 仍由 Nginx 接流。 + --expected-health-patrol-gateway-mode + 可选，nginx 或 pingora-direct；不传时随 --expect-public-gateway 推导。 + --require-realpath-canary 要求 realpath canary 配置存在，且 127.0.0.1:18083 由 Nginx 监听。 + --require-pingora-shadow 要求 Pingora shadow 127.0.0.1:18081 正在监听。 + --require-current-release-gateway 要求 current release 自审确认 pingora-gateway、checksum 和 manifest。 + --timeout-ms systemctl / ss / 子检查超时，默认 5000。 + --fail-on-critical 如果状态中出现 CRITICAL，则以退出码 1 结束。 + +该脚本只读采集 dev / release 上的 Pingora 直连切换彩排状态，不写 /etc、不 reload systemd、不修改 Nginx 或 Pingora。 +`); +} + +function validateConfig(input) { + for (const [label, value] of [ + ['--release-root', input.releaseRoot], + ['--health-patrol-env-file', input.healthPatrolEnvFile], + ['--pingora-env-file', input.pingoraEnvFile], + ['--realpath-config-file', input.realpathConfigFile], + ]) { + validateSafeAbsoluteFilePath(value, label); + } + + if (!PUBLIC_GATEWAYS.has(input.expectedPublicGateway)) { + throw new Error( + `--expect-public-gateway 只支持 none / nginx / pingora-direct: ${input.expectedPublicGateway}`, + ); + } + if ( + input.expectedHealthPatrolGatewayMode && + !GATEWAY_MODES.has(input.expectedHealthPatrolGatewayMode) + ) { + throw new Error( + `--expected-health-patrol-gateway-mode 只支持 nginx 或 pingora-direct: ${input.expectedHealthPatrolGatewayMode}`, + ); + } +} + +function requireValue(argv, index, flag) { + const value = argv[index]; + if (value === undefined || value.startsWith('--')) { + throw new Error(`${flag} 缺少参数值`); + } + return value; +} + +function readBoolEnv(name) { + const value = process.env[name]; + if (value === undefined || value === null || String(value).trim() === '') { + return false; + } + const normalized = String(value).trim().toLowerCase(); + if (['1', 'true', 'yes', 'on'].includes(normalized)) { + return true; + } + if (['0', 'false', 'no', 'off'].includes(normalized)) { + return false; + } + throw new Error(`${name} 必须是布尔值 true/false 或 1/0。`); +} + +function parseOptionalPositiveInt(raw, fallback, label) { + if (raw === undefined || raw === null || String(raw).trim() === '') { + return fallback; + } + return parseRequiredPositiveInt(raw, label); +} + +function parseRequiredPositiveInt(raw, label) { + const text = String(raw ?? '').trim(); + if (!/^[1-9][0-9]*$/u.test(text)) { + throw new Error(`${label} 必须是正整数。`); + } + return Number.parseInt(text, 10); +} + +function validateSafeAbsoluteFilePath(value, label) { + validateNoControlCharacters(value, label); + if (!path.isAbsolute(value)) { + throw new Error(`${label} 必须是绝对路径。`); + } + if (isFilesystemRootPath(value)) { + throw new Error(`${label} 不能是文件系统根目录。`); + } +} + +function validateNoControlCharacters(value, label) { + if (/[\0\r\n]/u.test(String(value))) { + throw new Error(`${label} 不能包含换行或 NUL 字符。`); + } +} + +function isFilesystemRootPath(value) { + const resolved = path.resolve(String(value)); + return resolved === path.parse(resolved).root; +} + +async function buildStatus(input) { + const healthPatrolEnv = await inspectHealthPatrolEnv(input); + const pingoraEnv = await inspectPingoraEnv(input); + const secrets = collectSecretValues(healthPatrolEnv, pingoraEnv); + delete healthPatrolEnv.secretValues; + delete pingoraEnv.secretValues; + const services = await inspectServices(input); + const systemd = await inspectPingoraSystemd(input); + const ports = await inspectPorts(input); + const realpathCanary = await inspectRealpathCanary(input, ports); + const currentReleaseAudit = await inspectCurrentRelease(input, secrets); + const publicBoundary = inspectPublicBoundary(input, ports, systemd); + const checks = [ + healthPatrolEnv.status, + pingoraEnv.status, + services.status, + systemd.status, + ports.status, + realpathCanary.status, + currentReleaseAudit.status, + publicBoundary.status, + ]; + + return { + schemaVersion: 1, + generatedAt: new Date().toISOString(), + summary: summarize(checks), + releaseRoot: input.releaseRoot, + expectations: { + publicGateway: input.expectedPublicGateway, + healthPatrolGatewayMode: expectedHealthPatrolGatewayMode(input), + requireRealpathCanary: input.requireRealpathCanary, + requirePingoraShadow: input.requirePingoraShadow, + requireCurrentReleaseGateway: input.requireCurrentReleaseGateway, + }, + publicBoundary, + ports, + services, + systemd, + healthPatrolEnv, + pingoraEnv, + realpathCanary, + currentReleaseAudit, + }; +} + +async function inspectHealthPatrolEnv(input) { + const parsed = await readEnvFile(input.healthPatrolEnvFile); + if (parsed.status === 'CRITICAL') { + return parsed; + } + + const values = parsed.values; + const expectedMode = expectedHealthPatrolGatewayMode(input); + const gatewayMode = values.GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE || ''; + const diagnostics = [...parsed.diagnostics]; + let status = parsed.status; + + if (expectedMode && gatewayMode !== expectedMode) { + diagnostics.push( + `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE 应为 ${expectedMode}，实际 ${gatewayMode || '(空)'}`, + ); + status = maxStatus(status, 'CRITICAL'); + } + + return { + path: input.healthPatrolEnvFile, + status, + secretValues: collectSecretValuesFromEnv(values), + values: { + gatewayMode, + publicBaseUrl: + values.GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL || '', + publicHost: values.GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST || '', + apiBaseUrl: values.GENARRATIVE_HEALTH_PATROL_API_BASE_URL || '', + pingoraBaseUrl: + values.GENARRATIVE_HEALTH_PATROL_PINGORA_BASE_URL || '', + hasPingoraProbeToken: Boolean( + values.GENARRATIVE_HEALTH_PATROL_PINGORA_PROBE_TOKEN || + values.GENARRATIVE_PINGORA_GATEWAY_PROBE_TOKEN, + ), + }, + diagnostics, + }; +} + +function expectedHealthPatrolGatewayMode(input) { + if (input.expectedHealthPatrolGatewayMode) { + return input.expectedHealthPatrolGatewayMode; + } + if (input.expectedPublicGateway === 'nginx') { + return 'nginx'; + } + if (input.expectedPublicGateway === 'pingora-direct') { + return 'pingora-direct'; + } + return ''; +} + +async function inspectPingoraEnv(input) { + const parsed = await readEnvFile(input.pingoraEnvFile); + if (parsed.status === 'CRITICAL') { + return { + ...parsed, + values: {}, + }; + } + const values = parsed.values; + return { + path: input.pingoraEnvFile, + status: parsed.status, + secretValues: collectSecretValuesFromEnv(values), + values: { + listen: values.GENARRATIVE_PINGORA_GATEWAY_LISTEN || '', + tlsListen: values.GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN || '', + httpRedirectListen: + values.GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN || '', + tlsCertFile: values.GENARRATIVE_PINGORA_GATEWAY_TLS_CERT_FILE || '', + hasTlsKeyFile: Boolean( + values.GENARRATIVE_PINGORA_GATEWAY_TLS_KEY_FILE, + ), + hasProbeToken: Boolean(values.GENARRATIVE_PINGORA_GATEWAY_PROBE_TOKEN), + accessLogFile: + values.GENARRATIVE_PINGORA_GATEWAY_ACCESS_LOG_FILE || '', + compressionAlgorithms: + values.GENARRATIVE_PINGORA_GATEWAY_COMPRESSION_ALGORITHMS || '', + trustXForwardedFor: + values.GENARRATIVE_PINGORA_GATEWAY_TRUST_X_FORWARDED_FOR || '', + }, + diagnostics: parsed.diagnostics, + }; +} + +async function readEnvFile(filePath) { + let text; + try { + text = await readFile(filePath, 'utf8'); + } catch (error) { + return { + path: filePath, + status: 'CRITICAL', + values: {}, + diagnostics: [`无法读取 env 文件: ${error.message}`], + }; + } + + const values = {}; + const diagnostics = []; + for (const [index, rawLine] of text.split(/\r?\n/u).entries()) { + let line = rawLine.trim(); + if (!line || line.startsWith('#')) { + continue; + } + if (line.startsWith('export ')) { + line = line.slice('export '.length).trim(); + } + const equalsIndex = line.indexOf('='); + if (equalsIndex <= 0) { + diagnostics.push(`第 ${index + 1} 行不是 KEY=VALUE 格式`); + continue; + } + const key = line.slice(0, equalsIndex).trim(); + values[key] = stripQuotes(line.slice(equalsIndex + 1).trim()); + } + + return { + path: filePath, + status: diagnostics.length > 0 ? 'WARNING' : 'OK', + values, + diagnostics, + }; +} + +function stripQuotes(value) { + if (value.length >= 2) { + const first = value[0]; + const last = value[value.length - 1]; + if ((first === '"' && last === '"') || (first === "'" && last === "'")) { + return value.slice(1, -1); + } + } + return value; +} + +function collectSecretValuesFromEnv(values) { + const secrets = []; + for (const [key, value] of Object.entries(values || {})) { + if (!SECRET_ENV_KEY_PATTERN.test(key) || typeof value !== 'string') { + continue; + } + const trimmed = value.trim(); + if (trimmed.length > 0) { + secrets.push(trimmed); + } + } + return secrets; +} + +function collectSecretValues(...envSnapshots) { + const secrets = new Set(); + for (const snapshot of envSnapshots) { + for (const value of snapshot?.secretValues || []) { + if (typeof value === 'string' && value.trim()) { + secrets.add(value.trim()); + } + } + } + return [...secrets].sort((left, right) => right.length - left.length); +} + +async function inspectServices(input) { + const serviceNames = [ + 'nginx.service', + 'genarrative-pingora-gateway.service', + 'genarrative-api.service', + 'spacetimedb.service', + ]; + const services = []; + for (const service of serviceNames) { + services.push(await inspectService(service, input)); + } + return { + status: summarize(services.map((service) => service.status)).status, + services, + }; +} + +async function inspectService(service, input) { + const result = await runCommand('systemctl', ['is-active', service], input); + const state = result.stdout.trim() || result.stderr.trim() || result.error; + let status = 'OK'; + if (service === 'nginx.service' && input.expectedPublicGateway === 'nginx') { + status = result.code === 0 && state === 'active' ? 'OK' : 'CRITICAL'; + } else if ( + service === 'genarrative-pingora-gateway.service' && + (input.requirePingoraShadow || + input.expectedPublicGateway === 'pingora-direct') + ) { + status = result.code === 0 && state === 'active' ? 'OK' : 'CRITICAL'; + } else if ( + (service === 'genarrative-api.service' || + service === 'spacetimedb.service') && + result.code !== 0 + ) { + status = 'WARNING'; + } + return { + name: service, + activeState: state || 'unknown', + status, + command: result.command, + }; +} + +async function inspectPingoraSystemd(input) { + const catResult = await runCommand( + 'systemctl', + ['cat', 'genarrative-pingora-gateway.service'], + input, + ); + const showResult = await runCommand( + 'systemctl', + [ + 'show', + 'genarrative-pingora-gateway.service', + '--property=FragmentPath', + '--property=DropInPaths', + '--property=User', + '--property=ExecStart', + '--no-pager', + ], + input, + ); + const unitText = catResult.stdout; + const hasAmbientCapability = unitText.includes( + 'AmbientCapabilities=CAP_NET_BIND_SERVICE', + ); + const hasCapabilityBoundingSet = unitText.includes( + 'CapabilityBoundingSet=CAP_NET_BIND_SERVICE', + ); + const environmentFiles = [ + ...unitText.matchAll(/^\s*EnvironmentFile=(.+)$/gmu), + ].map((match) => match[1].trim()); + const environmentFileMatchesPingoraEnvFile = environmentFilesInclude( + environmentFiles, + input.pingoraEnvFile, + ); + const show = parseSystemctlShow(showResult.stdout); + const diagnostics = []; + let status = 'OK'; + + if (catResult.code !== 0) { + diagnostics.push(`systemctl cat 失败: ${catResult.stderr || catResult.error}`); + status = 'CRITICAL'; + } + if (showResult.code !== 0) { + diagnostics.push( + `systemctl show 失败: ${showResult.stderr || showResult.error}`, + ); + status = 'CRITICAL'; + } + if (catResult.code === 0 && !environmentFileMatchesPingoraEnvFile) { + diagnostics.push( + `systemctl cat genarrative-pingora-gateway.service EnvironmentFile 未包含本次 --pingora-env-file: ${input.pingoraEnvFile}`, + ); + status = 'CRITICAL'; + } + if ( + input.expectedPublicGateway === 'nginx' && + (hasAmbientCapability || hasCapabilityBoundingSet) + ) { + diagnostics.push( + 'Nginx 接公网阶段不应残留 Pingora 低端口 CAP_NET_BIND_SERVICE。', + ); + status = 'CRITICAL'; + } + if ( + input.expectedPublicGateway === 'pingora-direct' && + (!hasAmbientCapability || !hasCapabilityBoundingSet) + ) { + diagnostics.push( + 'pingora-direct 接公网阶段必须在 systemd 最终配置中包含 CAP_NET_BIND_SERVICE。', + ); + status = 'CRITICAL'; + } + + return { + status, + hasAmbientCapability, + hasCapabilityBoundingSet, + environmentFiles, + environmentFileMatchesPingoraEnvFile, + fragmentPath: show.FragmentPath || '', + dropInPaths: show.DropInPaths || '', + user: show.User || '', + execStart: show.ExecStart || '', + diagnostics, + commands: [catResult.command, showResult.command], + }; +} + +function environmentFilesInclude(environmentFiles, expectedPath) { + return environmentFiles.some((entry) => + splitSystemdEnvironmentFileEntry(entry).includes(expectedPath), + ); +} + +function splitSystemdEnvironmentFileEntry(entry) { + const files = []; + for (const word of String(entry || '').split(/\s+/u)) { + if (!word) { + continue; + } + let normalized = word.trim(); + if (normalized.startsWith('-')) { + normalized = normalized.slice(1); + } + if ( + (normalized.startsWith('"') && normalized.endsWith('"')) || + (normalized.startsWith("'") && normalized.endsWith("'")) + ) { + normalized = normalized.slice(1, -1); + } + if (normalized) { + files.push(normalized); + } + } + return files; +} + +function parseSystemctlShow(text) { + const result = {}; + for (const line of text.split(/\r?\n/u)) { + const equalsIndex = line.indexOf('='); + if (equalsIndex <= 0) { + continue; + } + result[line.slice(0, equalsIndex)] = line.slice(equalsIndex + 1); + } + return result; +} + +async function inspectPorts(input) { + const result = await runCommand('ss', ['-H', '-ltnp'], input); + const listeners = result.code === 0 ? parseSsOutput(result.stdout) : []; + const byPort = {}; + const statuses = []; + for (const port of DEFAULT_PORTS) { + const portListeners = listeners.filter((item) => item.port === port); + const entry = inspectPort(port, portListeners, input); + byPort[String(port)] = entry; + statuses.push(entry.status); + } + if (result.code !== 0) { + statuses.push('CRITICAL'); + } + return { + status: summarize(statuses).status, + command: result.command, + error: result.code === 0 ? '' : trimForJson(result.stderr || result.error), + byPort, + }; +} + +function parseSsOutput(stdout) { + const listeners = []; + for (const line of String(stdout || '').split(/\r?\n/u)) { + const trimmed = line.trim(); + if (!trimmed) { + continue; + } + const parts = trimmed.split(/\s+/u); + const local = parts[3] || ''; + const port = parsePortFromLocalAddress(local); + if (!port) { + continue; + } + const processText = parts.slice(5).join(' '); + const processNames = [ + ...new Set([...processText.matchAll(/"([^"]+)"/gu)].map((match) => match[1])), + ]; + listeners.push({ + localAddress: local, + port, + loopback: isLoopbackLocalAddress(local), + processNames, + }); + } + return listeners; +} + +function parsePortFromLocalAddress(value) { + const bracketMatch = String(value).match(/^\[[^\]]+\]:(\d+)$/u); + const plainMatch = String(value).match(/:(\d+)$/u); + const portText = bracketMatch?.[1] || plainMatch?.[1] || ''; + const port = Number.parseInt(portText, 10); + return Number.isInteger(port) && port > 0 ? port : null; +} + +function isLoopbackLocalAddress(value) { + const text = String(value || '').toLowerCase(); + return ( + text.startsWith('127.') || + text.startsWith('[::1]') || + text.startsWith('::1') || + text.startsWith('localhost:') + ); +} + +function inspectPort(port, listeners, input) { + const hasNginx = listeners.some((listener) => + listener.processNames.some((name) => name.toLowerCase().includes('nginx')), + ); + const hasPingora = listeners.some((listener) => + listener.processNames.some((name) => name.toLowerCase().includes('pingora')), + ); + const hasUnknownProcess = + listeners.length > 0 && + listeners.some((listener) => listener.processNames.length === 0); + const loopbackOnly = + listeners.length > 0 && listeners.every((listener) => listener.loopback); + const diagnostics = []; + let status = 'OK'; + + if ([80, 443].includes(port)) { + if (input.expectedPublicGateway === 'nginx') { + if (listeners.length === 0) { + diagnostics.push(`${port} 未监听，Nginx 公网入口未就绪。`); + status = 'CRITICAL'; + } + if (!hasNginx) { + diagnostics.push(`${port} 未看到 Nginx 监听。`); + status = 'CRITICAL'; + } + if (hasPingora) { + diagnostics.push(`${port} 已被 Pingora 监听，不能作为未切公网彩排状态。`); + status = 'CRITICAL'; + } + } + if (input.expectedPublicGateway === 'pingora-direct') { + if (!hasPingora) { + diagnostics.push(`${port} 未看到 Pingora 监听。`); + status = 'CRITICAL'; + } + if (hasNginx) { + diagnostics.push(`${port} 仍由 Nginx 监听。`); + status = 'CRITICAL'; + } + } + } + + if (port === 18081 && input.requirePingoraShadow) { + if (!hasPingora) { + diagnostics.push('18081 未看到 Pingora shadow 监听。'); + status = 'CRITICAL'; + } + if (!loopbackOnly) { + diagnostics.push('Pingora shadow 18081 必须只监听 loopback。'); + status = 'CRITICAL'; + } + } + + if (port === 18083 && input.requireRealpathCanary) { + if (!hasNginx) { + diagnostics.push('18083 未看到 Nginx realpath canary 监听。'); + status = 'CRITICAL'; + } + if (!loopbackOnly) { + diagnostics.push('Nginx realpath canary 18083 必须只监听 loopback。'); + status = 'CRITICAL'; + } + } + + if ( + hasUnknownProcess && + ([80, 443].includes(port) || + (port === 18081 && input.requirePingoraShadow) || + (port === 18083 && input.requireRealpathCanary)) + ) { + diagnostics.push('ss 未返回进程名，无法确认监听归属。'); + status = maxStatus(status, 'WARNING'); + } + + return { + port, + status, + listening: listeners.length > 0, + loopbackOnly, + hasNginx, + hasPingora, + hasUnknownProcess, + listeners, + diagnostics, + }; +} + +async function inspectRealpathCanary(input, ports) { + const diagnostics = []; + let status = 'OK'; + let exists = false; + let templateLooksValid = false; + try { + const fileStat = await stat(input.realpathConfigFile); + exists = fileStat.isFile(); + } catch { + exists = false; + } + + if (!exists) { + diagnostics.push(`realpath canary 配置不存在: ${input.realpathConfigFile}`); + status = input.requireRealpathCanary ? 'CRITICAL' : 'WARNING'; + } else { + const content = await readFile(input.realpathConfigFile, 'utf8'); + const requiredSnippets = [ + 'listen 127.0.0.1:18083', + 'genarrative-pingora-realpath-canary.access.log', + 'X-Genarrative-Nginx-Handoff pingora-realpath-canary', + ]; + const missing = requiredSnippets.filter((snippet) => !content.includes(snippet)); + templateLooksValid = missing.length === 0; + if (missing.length > 0) { + diagnostics.push(`realpath canary 配置缺少关键片段: ${missing.join(', ')}`); + status = 'CRITICAL'; + } + } + + const portStatus = ports.byPort['18083']; + if (input.requireRealpathCanary && portStatus?.status === 'CRITICAL') { + status = 'CRITICAL'; + } + + return { + path: input.realpathConfigFile, + status, + exists, + templateLooksValid, + port: portStatus, + diagnostics, + }; +} + +async function inspectCurrentRelease(input, secrets) { + const script = path.join( + input.releaseRoot, + 'scripts/ops/pingora-current-release-audit.mjs', + ); + try { + await access(script, fsConstants.R_OK); + } catch (error) { + return { + status: 'CRITICAL', + script, + code: null, + diagnostics: [`无法读取 current release 自审脚本: ${error.message}`], + }; + } + + const args = [ + '--', + script, + '--release-root', + input.releaseRoot, + '--systemd-show', + '--timeout-ms', + String(input.timeoutMs), + ]; + if (input.requireCurrentReleaseGateway) { + args.push('--require-pingora-gateway'); + } + const result = await runCommand('node', args, input); + const parsed = parseJsonObject(result.stdout); + let status = result.code === 0 ? 'OK' : 'CRITICAL'; + const diagnostics = []; + if (!parsed.ok) { + diagnostics.push(`无法解析 current release 自审 JSON: ${parsed.error}`); + status = 'CRITICAL'; + } + return { + status, + script, + code: result.code, + command: result.command, + summary: parsed.ok ? parsed.value?.summary || null : null, + pingoraGateway: parsed.ok ? parsed.value?.pingoraGateway || null : null, + releaseManifest: parsed.ok ? parsed.value?.releaseManifest || null : null, + stdout: trimForJson(redactSecrets(result.stdout, secrets)), + stderr: trimForJson(redactSecrets(result.stderr || result.error, secrets)), + diagnostics, + }; +} + +function inspectPublicBoundary(input, ports, systemd) { + const diagnostics = []; + let status = 'OK'; + if (input.expectedPublicGateway === 'none') { + return { + status, + expectedPublicGateway: input.expectedPublicGateway, + diagnostics, + }; + } + + for (const port of [80, 443]) { + const portStatus = ports.byPort[String(port)]; + if (portStatus?.status === 'CRITICAL') { + status = 'CRITICAL'; + diagnostics.push(...portStatus.diagnostics); + } + } + if (systemd.status === 'CRITICAL') { + status = 'CRITICAL'; + diagnostics.push(...systemd.diagnostics); + } + + return { + status, + expectedPublicGateway: input.expectedPublicGateway, + diagnostics, + }; +} + +function runCommand(command, args, input, env = process.env) { + validateNoControlCharacters(command, '子命令可执行文件'); + for (const arg of args) { + validateNoControlCharacters(arg, '子命令参数'); + } + return new Promise((resolve) => { + execFile( + command, + args, + { + env, + timeout: input.timeoutMs, + windowsHide: true, + maxBuffer: 1024 * 1024, + }, + (error, stdout, stderr) => { + resolve({ + command: formatCommand(command, args), + code: typeof error?.code === 'number' ? error.code : error ? 1 : 0, + stdout: String(stdout || ''), + stderr: String(stderr || ''), + timedOut: Boolean(error?.killed), + error: error ? error.message : '', + }); + }, + ); + }); +} + +function formatCommand(command, args) { + return [command, ...args].join(' '); +} + +function redactSecrets(value, secrets) { + let text = String(value || ''); + for (const secret of secrets) { + if (!secret) { + continue; + } + text = text.split(secret).join(''); + } + return text; +} + +function trimForJson(value) { + const text = String(value || '').trim(); + if (text.length <= 1000) { + return text; + } + return `${text.slice(0, 1000)}...`; +} + +function parseJsonObject(value) { + try { + return { ok: true, value: JSON.parse(value) }; + } catch (error) { + return { ok: false, error: error.message }; + } +} + +function summarize(statuses) { + const normalized = statuses.filter(Boolean); + const status = normalized.reduce( + (current, item) => maxStatus(current, item), + 'OK', + ); + return { + status, + criticalCount: normalized.filter((item) => item === 'CRITICAL').length, + warningCount: normalized.filter((item) => item === 'WARNING').length, + }; +} + +function maxStatus(left, right) { + return STATUS_RANK[right] > STATUS_RANK[left] ? right : left; +}