自动拉起 Pingora shadow 发布服务
让 API deploy 在发布包包含 Pingora 时先校验 shadow systemd 和 env 安全边界 在 shadow 配置安全时自动启动或重启 genarrative-pingora-gateway.service 拒绝 direct-entry capability、公网监听 env，并在 current 切换前失败 补充 production-api-deploy 动态烟测、生产运维护栏和 Pingora 运维文档 在 dev 服务器真实验证 shadow inactive 场景可由发布脚本自动拉起
This commit is contained in:
@@ -2329,7 +2329,7 @@
|
||||
- 决策补充：`/api` 通用路由必须同时检查 `Content-Length` 与实际流式请求体累计字节数；缺少长度头时超过上限也返回统一 `PAYLOAD_TOO_LARGE` JSON。影子部署模板使用 `deploy/systemd/genarrative-pingora-gateway.service`，默认读取 `/etc/genarrative/pingora-gateway.env`，仍只监听本机高端口；`/__genarrative_pingora/healthz` 只在配置并匹配 `X-Genarrative-Pingora-Probe` token 时返回 shadow JSON。
|
||||
- 决策补充：生产 `genarrative-health-patrol.service` 只在显式配置 `GENARRATIVE_HEALTH_PATROL_PINGORA_BASE_URL` 与 `GENARRATIVE_HEALTH_PATROL_PINGORA_PROBE_TOKEN` 时检查 Pingora shadow probe；未配置时巡检口径不变。Pingora shadow 日志必须保留 request/route/upstream/body 字段，方便和 Nginx access log 做 canary 对照。
|
||||
- 决策补充：生产健康巡检的公网入口模式必须显式区分 `nginx` 和 `pingora-direct`。默认 `GENARRATIVE_HEALTH_PATROL_GATEWAY_MODE=nginx` 检查 API、SpacetimeDB 和 Nginx；Pingora 直连接管公网后切到 `pingora-direct`，改为检查 API、SpacetimeDB 和 `genarrative-pingora-gateway.service`，不再要求 `nginx.service` active。目标机本机探测 `127.0.0.1` 时用 `GENARRATIVE_HEALTH_PATROL_PUBLIC_HOST=<域名>` 保留正式 Host / vhost 语义。
|
||||
- 决策补充：Pingora 影子网关产物不进入默认 API release；只有显式传 `--include-pingora-gateway` 或在 `Genarrative-Api-Build` 勾选 `INCLUDE_PINGORA_GATEWAY` 时，才构建并打包 `pingora-gateway` / `pingora-gateway.sha256`。真实构建 Pingora 前必须先检查 `cmake`、C 编译器和 C++ 编译器；Jenkins 勾选 `INCLUDE_PINGORA_GATEWAY` 时也要先 fail-fast 检查这些工具，避免进入 Cargo 后才因 `libz-ng-sys` 构建依赖缺失失败。`production-api-deploy.sh` 仅在两者同时存在时校验并复制到 current release，避免现有 API 流水线被 Pingora 构建依赖影响。若发布包包含 Pingora 且 `genarrative-pingora-gateway.service` 已经处于 `active`，API deploy 在 `current` 链接切换后执行 `systemctl try-restart genarrative-pingora-gateway.service` 并复核仍为 `active`，让已启用的 shadow / canary / direct 机器加载同一份 current release 网关二进制；若服务未运行则只跳过，不主动拉起 Pingora。`npm run check:production-api-release` 必须同时验证默认 API release 不登记 Pingora，以及显式 `--include-pingora-gateway --skip-pingora-gateway-build` 时发布包包含 `pingora-gateway`、`pingora-gateway.sha256` 并写入 manifest。
|
||||
- 决策补充：Pingora 影子网关产物不进入默认 API release；只有显式传 `--include-pingora-gateway` 或在 `Genarrative-Api-Build` 勾选 `INCLUDE_PINGORA_GATEWAY` 时，才构建并打包 `pingora-gateway` / `pingora-gateway.sha256`。真实构建 Pingora 前必须先检查 `cmake`、C 编译器和 C++ 编译器；Jenkins 勾选 `INCLUDE_PINGORA_GATEWAY` 时也要先 fail-fast 检查这些工具，避免进入 Cargo 后才因 `libz-ng-sys` 构建依赖缺失失败。`production-api-deploy.sh` 仅在两者同时存在时校验并复制到 current release，避免现有 API 流水线被 Pingora 构建依赖影响。发布包包含 Pingora 时，API deploy 会在提升 release 前读取 `systemctl cat genarrative-pingora-gateway.service` 和其 `EnvironmentFile`，拒绝 direct-entry `CAP_NET_BIND_SERVICE`、拒绝非 `127.0.0.1:18081` 的 shadow listen、拒绝 `TLS_LISTEN` / `HTTP_REDIRECT_LISTEN`，确认仍是本机 shadow 高端口后才切换 current；切换后执行 `systemctl restart genarrative-pingora-gateway.service` 并复核 active，让 shadow / canary 机器加载同一份 current release 网关二进制。该自动拉起不会启用公网 `80/443` 直连入口；已经进入 direct-entry 状态的机器应走正式直连 runbook 或先回退到 shadow。`npm run check:production-api-release` 必须同时验证默认 API release 不登记 Pingora，以及显式 `--include-pingora-gateway --skip-pingora-gateway-build` 时发布包包含 `pingora-gateway`、`pingora-gateway.sha256` 并写入 manifest。
|
||||
- 决策补充：生产健康巡检的显式 `--timeout-ms`、`--slow-ms`、`GENARRATIVE_HEALTH_PATROL_TIMEOUT_MS` 和 `GENARRATIVE_HEALTH_PATROL_SLOW_MS` 必须是正整数，非法值直接失败，不静默回退默认 `5000ms` / `3000ms`。Pingora canary live 的 `--timeout-ms` / `GENARRATIVE_PINGORA_CANARY_TIMEOUT_MS`、direct live 的 `--timeout-ms` / `GENARRATIVE_PINGORA_DIRECT_TIMEOUT_MS`、canary access log 对账的 `--since-lines` / `GENARRATIVE_PINGORA_CANARY_ACCESS_LOG_SINCE_LINES` 也必须正整数。Pingora direct live 和 release readiness 读取的直连布尔 env 必须严格解析，只接受 `true/false`、`1/0`、`yes/no`、`on/off` 或空值，非法值直接失败，避免 `REQUIRE_WSS_UPGRADE`、preflight 开关或 `SKIP_WSS` 因拼写错误被当成 false。canary live 的 base URL、prefix、Host、额外 path 和 timeout 不能包含换行或 NUL；脚本必须在发起 canary 请求前失败，避免污染参数进入 URL、Host header 或 JSON 输出。canary access log 对账的日志路径、prefix、必需路径、tail 行数以及日志行中解析出的 URI / path 也不能包含换行或 NUL；脚本必须失败并给出对应参数或日志行诊断，不能把污染值写入 JSON 对账输出。direct live 的 HTTPS / HTTP base URL、Host、redirect Host、probe token、额外 path、SpacetimeDB 数据库名、access log 路径、timeout 和布尔 env 都不能包含换行或 NUL；脚本必须在发起 HTTPS / HTTP / WSS 请求前失败，避免污染参数进入请求头、URL、日志对账或 JSON 证据。Pingora 切换窗口调整巡检、live smoke、日志对账阈值或直连布尔开关时，把参数解析失败视为配置错误，而不是继续执行检查。
|
||||
- 决策补充：即使不打包 Pingora 二进制，API release 也必须随包携带 Pingora release readiness 聚合门禁、直连启用 / 回退 / preflight / live smoke / current release 自审脚本、直连彩排状态脚本，以及 `deploy/systemd/`、`deploy/pingora/` 支撑配置；`pingora-direct-enable.sh` 和正式 cutover runbook 默认从 `/opt/genarrative/current` 推导这些路径，启用前 release readiness 基础门禁和启用后 `--require-direct` 复核也必须调用 `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs`，切换窗口不得依赖 Jenkins 工作区或目标机源码 checkout。API release 还必须携带 `build/<version>/scripts/deploy/production-api-deploy.sh` 和同目录 `maintenance-on.sh` / `maintenance-off.sh`；`Genarrative-Api-Deploy` 只能复制并执行 build 产物内的 deploy 脚本，禁止继续执行部署工作区根部脚本，避免 workspace 中的旧脚本掩盖发布包布局缺陷。`production-api-deploy.sh` 对数据库备份脚本、健康巡检脚本和 Pingora 直连依赖都执行 fail-fast，发布产物缺失时保持维护模式并停止部署，不再从部署工作区兜底复制；API deploy 必须要求 `--release-root`、`--current-link`、`--api-env-file` 使用绝对路径，且 `--version` 必须以数字或字母开头并拒绝点目录，再先写 `${RELEASE_ROOT}/.${VERSION}.staging.$`，全部复制完成后再用非合并语义提升为 `${RELEASE_ROOT}/${VERSION}`，并用固定替换语义切换 current 符号链接，同版本 release 已存在、提升前竞态出现或 current 路径不是符号链接时拒绝覆盖 / 合并，失败时清理 staging 且不留下正式 release；`npm run check:production-api-release` 与 `npm run check:production-api-deploy` 必须进入 `check:pingora-release-readiness` 聚合门禁，前者用临时 `CARGO_TARGET_DIR` 和假 `api-server` release binary 验证 `build-production-release.sh --component api-server --skip-api-build` 产物自包含，后者用临时 release 和 fake `systemctl` / `curl` 验证从发布产物内执行 deploy 脚本后 current release 自包含，并覆盖缺少备份脚本、健康巡检脚本、release readiness 聚合门禁脚本、current release 自审脚本、直连彩排状态脚本、direct live smoke 脚本、相对 release root / current link / api env file、点目录或点开头 version、既有 release 目录、目录型 current 或提升前 release 目录竞态时的失败维护模式。
|
||||
- 决策补充：正式直连 runbook 在采集状态快照前必须先执行 current release 自审：`/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs --release-root /opt/genarrative/current --require-pingora-gateway --systemd-show`。该脚本只读检查发布包自包含、`pingora-gateway` 可执行，以及 systemd `ExecStart` 是否指向 current release 网关二进制；失败时应先修发布包、Jenkins 归档过滤、deploy 复制或 systemd 指向，再继续切换。
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -240,7 +240,7 @@ Jenkins `Genarrative-Api-Build` 对应参数是 `INCLUDE_PINGORA_GATEWAY`，默
|
||||
|
||||
API release 还必须携带 `scripts/check-pingora-release-readiness.mjs`、`scripts/check-pingora-canary-live.mjs` 与 `scripts/ops/pingora-direct-rehearsal-status.mjs`。前者支撑 current release 的 `--release-runtime-only` 聚合复核，canary live 脚本支撑目标 Nginx canary live smoke，直连彩排状态脚本支撑目标机切换前只读确认 Nginx 仍接公网、Pingora shadow / realpath canary 高端口和 current release 自审均可用；缺少任一脚本时 `check:production-api-release`、`check:production-api-deploy` 和生产运维护栏都必须失败。
|
||||
|
||||
发布包会额外包含 `pingora-gateway` 与 `pingora-gateway.sha256`。`production-api-deploy.sh` 看到这两个文件时会校验并把 `pingora-gateway` / `pingora-gateway.sha256` 一起复制到 `/opt/genarrative/current`，供 shadow systemd 模板和 current release 自审使用；`api-server.sha256` 也必须随 `api-server` 一起进入 current release。没有这两个 Pingora 文件时现有 API 发布行为不变。API 发布仍只重启 `genarrative-api.service`；如果 `genarrative-pingora-gateway.service` 已经处于 `active`，部署脚本会在 `current` 链接切换后执行 `systemctl try-restart genarrative-pingora-gateway.service` 并复核仍为 `active`，让已启用的 shadow / canary / direct 机器加载同一份 current release 网关二进制。若 Pingora 服务未运行，部署脚本只打印跳过信息，不主动拉起影子服务，也不改变默认接流边界。
|
||||
发布包会额外包含 `pingora-gateway` 与 `pingora-gateway.sha256`。`production-api-deploy.sh` 看到这两个文件时会校验并把 `pingora-gateway` / `pingora-gateway.sha256` 一起复制到 `/opt/genarrative/current`，供 shadow systemd 模板和 current release 自审使用；`api-server.sha256` 也必须随 `api-server` 一起进入 current release。没有这两个 Pingora 文件时现有 API 发布行为不变。API 发布仍只重启 `genarrative-api.service`；包含 Pingora 时，部署脚本会在提升 release 和切换 `current` 前读取 `systemctl cat genarrative-pingora-gateway.service` 与其 `EnvironmentFile`，拒绝已出现 `CAP_NET_BIND_SERVICE` direct-entry capability、拒绝 `GENARRATIVE_PINGORA_GATEWAY_LISTEN` 不是 `127.0.0.1:18081`、拒绝已配置 `TLS_LISTEN` 或 `HTTP_REDIRECT_LISTEN`，确认仍是本机 shadow 高端口后才切换 current，并执行 `systemctl restart genarrative-pingora-gateway.service` 后复核 active。该自动拉起只覆盖 shadow 服务，不会启用公网 `80/443` 直连入口；若现场已经处于 direct-entry 状态，应走正式直连 runbook 或先回退到 shadow 后再执行 API deploy。
|
||||
|
||||
也可以复制 `deploy/pingora/pingora-gateway.env.example` 到部署环境的非 Git 配置文件，由 systemd 或容器注入。仓库提供 `deploy/systemd/genarrative-pingora-gateway.service` 作为影子服务模板，默认读取 `/etc/genarrative/pingora-gateway.env`，仍只应监听本机高端口，再由 Nginx 或本机 smoke 主动访问。Server-Provision 会把主 service 安装到 `/etc/systemd/system/genarrative-pingora-gateway.service`，并把直连低端口 drop-in 模板安装到 `/etc/genarrative/pingora/genarrative-pingora-gateway-direct-entry.conf` 作为参考和手动覆盖来源；该模板不会默认生效，`/opt/genarrative/current/scripts/deploy/pingora-direct-enable.sh` 默认使用随 current release 发布的 `deploy/systemd/genarrative-pingora-gateway-direct-entry.conf`。`deploy/nginx/snippets/genarrative-pingora-canary.conf` 是可选的 Nginx -> Pingora 前缀 canary 模板，Server-Provision 会安装到 `/etc/nginx/snippets/`，但主站配置默认不 include；启用前必须把 `__GENARRATIVE_PINGORA_PROBE_TOKEN__` 替换为真实 token，并确认 `allow/deny` 来源边界符合当次验证窗口。
|
||||
|
||||
@@ -399,7 +399,7 @@ dev 根盘空间在安装后曾接近满盘；2026-06-17 进入 canary 前已清
|
||||
本阶段验收只证明 API release 的正式打包、复制、current 切换、服务重启和 shadow Pingora 随 current 重启可用，不切 `80/443` 到 Pingora。验收结果：
|
||||
|
||||
- `production-api-deploy.sh` 进入维护模式后完成 `api-server.sha256` 和 `pingora-gateway.sha256` 校验，复制二进制、checksum、manifest 和 Pingora 直连依赖到新 release，再把 `/opt/genarrative/current` 切到 `/opt/genarrative/releases/dev-pingora-api-20260617140915`，最后退出维护模式。
|
||||
- 部署时 `genarrative-pingora-gateway.service` 已为 `active`，脚本在 current 切换后执行 `systemctl try-restart genarrative-pingora-gateway.service` 并复核 active；`genarrative-api.service`、外部生成 worker 和 worker controller 也完成重启和 active 等待。
|
||||
- 部署时 `genarrative-pingora-gateway.service` 仍是 shadow 配置，脚本在 current 切换前拒绝 direct-entry capability 和公网监听 env，切换后执行 `systemctl restart genarrative-pingora-gateway.service` 并复核 active；`genarrative-api.service`、外部生成 worker 和 worker controller 也完成重启和 active 等待。
|
||||
- `/opt/genarrative/current/release-manifest.api-server.json` 记录 `api-server` 和 `pingora-gateway` 两个 artifact，`cd /opt/genarrative/current && sha256sum -c api-server.sha256 && sha256sum -c pingora-gateway.sha256` 均为 `OK`。
|
||||
- `/opt/genarrative/current/scripts/ops/pingora-current-release-audit.mjs --release-root /opt/genarrative/current --require-pingora-gateway --systemd-show` 返回 `summary.status=OK`，确认 current release 自包含、checksum 匹配、`pingora-gateway` 可执行，且 systemd `ExecStart` 指向 `/opt/genarrative/current/pingora-gateway`。
|
||||
- `/opt/genarrative/current/scripts/check-pingora-release-readiness.mjs --release-runtime-only` 通过，证明目标机可以只依赖 current release 运行 runtime-only 发布自审，不需要源码 checkout 或 Jenkins 工作区。
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -37,7 +37,9 @@ console.log('[check:production-api-deploy] OK');
|
||||
function main() {
|
||||
assertDeployCopiesPingoraDirectReleaseDependencies();
|
||||
assertDeployRestartsActivePingoraWhenArtifactIncluded();
|
||||
assertDeploySkipsInactivePingoraWhenArtifactIncluded();
|
||||
assertDeployStartsInactivePingoraWhenArtifactIncluded();
|
||||
assertDeployRejectsPingoraDirectEntryWhenArtifactIncluded();
|
||||
assertDeployRejectsPingoraPublicListenWhenArtifactIncluded();
|
||||
assertDeployRejectsPingoraArtifactMissingManifestEntry();
|
||||
assertMissingReleaseManifestFails();
|
||||
assertReleaseManifestMissingApiArtifactFails();
|
||||
@@ -286,22 +288,22 @@ function assertDeployRestartsActivePingoraWhenArtifactIncluded() {
|
||||
const commandsLog = readFileSync(fixture.commandsLog, 'utf8');
|
||||
assertIncludes(
|
||||
commandsLog,
|
||||
'systemctl is-active --quiet genarrative-pingora-gateway.service',
|
||||
'部署脚本看到 Pingora 产物后必须先检查 shadow service 是否 active。',
|
||||
'systemctl cat genarrative-pingora-gateway.service',
|
||||
'部署脚本看到 Pingora 产物后必须先读取 systemd 最终配置。',
|
||||
);
|
||||
assertIncludes(
|
||||
commandsLog,
|
||||
'systemctl try-restart genarrative-pingora-gateway.service',
|
||||
'Pingora shadow service 已 active 时必须随 current release 切换 try-restart。',
|
||||
'systemctl restart genarrative-pingora-gateway.service',
|
||||
'Pingora shadow service 已 active 时必须随 current release 切换 restart。',
|
||||
);
|
||||
assertIncludes(
|
||||
result.stdout,
|
||||
'发布包包含 Pingora，重启已运行的影子服务',
|
||||
'发布包包含 Pingora，启动或重启 shadow 影子服务',
|
||||
'Pingora shadow service 自动重启时必须输出明确提示。',
|
||||
);
|
||||
}
|
||||
|
||||
function assertDeploySkipsInactivePingoraWhenArtifactIncluded() {
|
||||
function assertDeployStartsInactivePingoraWhenArtifactIncluded() {
|
||||
const fixture = prepareFixture('with-inactive-pingora-artifact');
|
||||
addPingoraGatewayArtifact(fixture);
|
||||
const result = runDeploy(fixture, { pingoraActive: false });
|
||||
@@ -309,7 +311,7 @@ function assertDeploySkipsInactivePingoraWhenArtifactIncluded() {
|
||||
assertStatus(
|
||||
result,
|
||||
0,
|
||||
'包含 Pingora 但 shadow service inactive 时应部署成功。',
|
||||
'包含 Pingora 且 shadow service inactive 但仍是安全 shadow 配置时应部署成功。',
|
||||
);
|
||||
if (result.status !== 0) {
|
||||
return;
|
||||
@@ -318,20 +320,86 @@ function assertDeploySkipsInactivePingoraWhenArtifactIncluded() {
|
||||
const commandsLog = readFileSync(fixture.commandsLog, 'utf8');
|
||||
assertIncludes(
|
||||
commandsLog,
|
||||
'systemctl is-active --quiet genarrative-pingora-gateway.service',
|
||||
'部署脚本看到 Pingora 产物后必须检查 shadow service 是否 active。',
|
||||
'systemctl cat genarrative-pingora-gateway.service',
|
||||
'部署脚本看到 Pingora 产物后必须先读取 systemd 最终配置。',
|
||||
);
|
||||
assertIncludes(
|
||||
commandsLog,
|
||||
'systemctl restart genarrative-pingora-gateway.service',
|
||||
'Pingora shadow service inactive 且 shadow 配置安全时必须随 current release 启动。',
|
||||
);
|
||||
assertIncludes(
|
||||
commandsLog,
|
||||
'systemctl is-active --quiet genarrative-pingora-gateway.service',
|
||||
'启动或重启 Pingora shadow 后必须复核 active。',
|
||||
);
|
||||
if (
|
||||
commandsLog.includes(
|
||||
'systemctl try-restart genarrative-pingora-gateway.service',
|
||||
)
|
||||
) {
|
||||
failures.push('Pingora shadow service inactive 时不能主动 try-restart。');
|
||||
}
|
||||
assertIncludes(
|
||||
result.stdout,
|
||||
'Pingora 影子服务未处于 active，跳过自动重启',
|
||||
'Pingora shadow service inactive 时必须明确说明不会主动拉起。',
|
||||
'发布包包含 Pingora，启动或重启 shadow 影子服务',
|
||||
'Pingora shadow service 自动启动时必须输出明确提示。',
|
||||
);
|
||||
}
|
||||
|
||||
function assertDeployRejectsPingoraDirectEntryWhenArtifactIncluded() {
|
||||
const fixture = prepareFixture('with-direct-entry-pingora-artifact');
|
||||
addPingoraGatewayArtifact(fixture);
|
||||
const result = runDeploy(fixture, { pingoraDirectEntry: true });
|
||||
|
||||
if (result.status === 0) {
|
||||
failures.push('包含 Pingora 但 systemd 已启用 direct-entry capability 时部署必须失败。');
|
||||
}
|
||||
assertIncludes(
|
||||
result.stderr,
|
||||
'Pingora systemd 已包含 CAP_NET_BIND_SERVICE',
|
||||
'direct-entry capability 存在时必须给出明确错误。',
|
||||
);
|
||||
const commandsLog = readFileSync(fixture.commandsLog, 'utf8');
|
||||
if (
|
||||
commandsLog.includes(
|
||||
'systemctl restart genarrative-pingora-gateway.service',
|
||||
)
|
||||
) {
|
||||
failures.push('direct-entry capability 存在时不能自动 restart Pingora。');
|
||||
}
|
||||
if (!existsSync(fixture.maintenanceFile)) {
|
||||
failures.push('direct-entry capability 导致部署失败时必须保持维护模式。');
|
||||
}
|
||||
assertNoReleasePromoted(
|
||||
fixture,
|
||||
'direct-entry capability 导致部署失败时不能提升正式 release。',
|
||||
);
|
||||
}
|
||||
|
||||
function assertDeployRejectsPingoraPublicListenWhenArtifactIncluded() {
|
||||
const fixture = prepareFixture('with-public-listen-pingora-artifact');
|
||||
addPingoraGatewayArtifact(fixture);
|
||||
writePingoraEnv(fixture, {
|
||||
listen: '0.0.0.0:443',
|
||||
});
|
||||
const result = runDeploy(fixture);
|
||||
|
||||
if (result.status === 0) {
|
||||
failures.push('包含 Pingora 但 env 已配置公网监听时部署必须失败。');
|
||||
}
|
||||
assertIncludes(
|
||||
result.stderr,
|
||||
'Pingora 自动启动只允许 shadow 监听 127.0.0.1:18081',
|
||||
'公网监听 env 存在时必须给出明确错误。',
|
||||
);
|
||||
const commandsLog = readFileSync(fixture.commandsLog, 'utf8');
|
||||
if (
|
||||
commandsLog.includes(
|
||||
'systemctl restart genarrative-pingora-gateway.service',
|
||||
)
|
||||
) {
|
||||
failures.push('公网监听 env 存在时不能自动 restart Pingora。');
|
||||
}
|
||||
if (!existsSync(fixture.maintenanceFile)) {
|
||||
failures.push('公网监听 env 导致部署失败时必须保持维护模式。');
|
||||
}
|
||||
assertNoReleasePromoted(
|
||||
fixture,
|
||||
'公网监听 env 导致部署失败时不能提升正式 release。',
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1024,10 +1092,12 @@ function prepareFixture(name) {
|
||||
const releaseRoot = path.join(root, 'releases');
|
||||
const currentLink = path.join(root, 'current');
|
||||
const apiEnvFile = path.join(root, 'etc', 'api-server.env');
|
||||
const pingoraEnvFile = path.join(root, 'etc', 'pingora-gateway.env');
|
||||
const maintenanceFile = path.join(root, 'maintenance', 'enabled');
|
||||
const fakeBin = path.join(root, 'bin');
|
||||
const commandsLog = path.join(root, 'commands.log');
|
||||
const workerStateFile = path.join(root, 'worker-service-enabled');
|
||||
const pingoraStateFile = path.join(root, 'pingora-service-active');
|
||||
const version = `20260614-${name}`;
|
||||
|
||||
mkdirSync(sourceDir, { recursive: true });
|
||||
@@ -1054,6 +1124,7 @@ function prepareFixture(name) {
|
||||
].join('\n'),
|
||||
'utf8',
|
||||
);
|
||||
writePingoraEnv({ pingoraEnvFile });
|
||||
chmodExecutable(path.join(sourceDir, 'api-server'));
|
||||
writeSha256(sourceDir, 'api-server');
|
||||
writeFileSync(
|
||||
@@ -1226,6 +1297,17 @@ function prepareFixture(name) {
|
||||
'set -euo pipefail',
|
||||
`printf 'systemctl %s\\n' "$*" >> ${shellQuote(commandsLog)}`,
|
||||
'worker_state_file="${FAKE_WORKER_STATE_FILE}"',
|
||||
'pingora_state_file="${FAKE_PINGORA_STATE_FILE}"',
|
||||
'if [[ "$1" == "cat" && "${2:-}" == "genarrative-pingora-gateway.service" ]]; then',
|
||||
' printf "[Service]\\n"',
|
||||
' printf "EnvironmentFile=%s\\n" "${FAKE_PINGORA_ENV_FILE}"',
|
||||
' printf "ExecStart=/opt/genarrative/current/pingora-gateway\\n"',
|
||||
' if [[ "${FAKE_PINGORA_DIRECT_ENTRY:-false}" == "true" ]]; then',
|
||||
' printf "AmbientCapabilities=CAP_NET_BIND_SERVICE\\n"',
|
||||
' printf "CapabilityBoundingSet=CAP_NET_BIND_SERVICE\\n"',
|
||||
' fi',
|
||||
' exit 0',
|
||||
'fi',
|
||||
'if [[ "$1" == "list-units" ]]; then',
|
||||
' pattern="${@: -1}"',
|
||||
' if [[ "${pattern}" == "genarrative-external-generation-worker@*.service" && -f "${worker_state_file}" ]]; then',
|
||||
@@ -1237,9 +1319,16 @@ function prepareFixture(name) {
|
||||
' printf "enabled\\n" > "${worker_state_file}"',
|
||||
' exit 0',
|
||||
'fi',
|
||||
'if [[ "$1 $2 ${3:-}" == "is-active --quiet genarrative-pingora-gateway.service" && "${FAKE_PINGORA_ACTIVE:-true}" == "false" ]]; then',
|
||||
'if [[ "$1 $2 ${3:-}" == "is-active --quiet genarrative-pingora-gateway.service" ]]; then',
|
||||
' if [[ "${FAKE_PINGORA_ACTIVE:-true}" == "true" || -f "${pingora_state_file}" ]]; then',
|
||||
' exit 0',
|
||||
' fi',
|
||||
' exit 3',
|
||||
'fi',
|
||||
'if [[ "$1" == "restart" && "${2:-}" == "genarrative-pingora-gateway.service" ]]; then',
|
||||
' printf "active\\n" > "${pingora_state_file}"',
|
||||
' exit 0',
|
||||
'fi',
|
||||
'exit 0',
|
||||
'',
|
||||
].join('\n'),
|
||||
@@ -1283,14 +1372,32 @@ function prepareFixture(name) {
|
||||
releaseRoot,
|
||||
currentLink,
|
||||
apiEnvFile,
|
||||
pingoraEnvFile,
|
||||
maintenanceFile,
|
||||
fakeBin,
|
||||
commandsLog,
|
||||
workerStateFile,
|
||||
pingoraStateFile,
|
||||
version,
|
||||
};
|
||||
}
|
||||
|
||||
function writePingoraEnv(fixture, options = {}) {
|
||||
const filePath = fixture.pingoraEnvFile;
|
||||
const lines = [
|
||||
`GENARRATIVE_PINGORA_GATEWAY_LISTEN=${options.listen ?? '127.0.0.1:18081'}`,
|
||||
];
|
||||
if (options.tlsListen) {
|
||||
lines.push(`GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN=${options.tlsListen}`);
|
||||
}
|
||||
if (options.redirectListen) {
|
||||
lines.push(
|
||||
`GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN=${options.redirectListen}`,
|
||||
);
|
||||
}
|
||||
writeFileSync(filePath, `${lines.join('\n')}\n`, 'utf8');
|
||||
}
|
||||
|
||||
function addPingoraGatewayArtifact(fixture, options = {}) {
|
||||
writeFileSync(
|
||||
path.join(fixture.sourceDir, 'pingora-gateway'),
|
||||
@@ -1351,6 +1458,10 @@ function runDeploy(fixture, options = {}) {
|
||||
PATH: `${fixture.fakeBin}:${process.env.PATH || ''}`,
|
||||
GENARRATIVE_MAINTENANCE_FILE: fixture.maintenanceFile,
|
||||
FAKE_PINGORA_ACTIVE: options.pingoraActive === false ? 'false' : 'true',
|
||||
FAKE_PINGORA_DIRECT_ENTRY:
|
||||
options.pingoraDirectEntry === true ? 'true' : 'false',
|
||||
FAKE_PINGORA_ENV_FILE: fixture.pingoraEnvFile,
|
||||
FAKE_PINGORA_STATE_FILE: fixture.pingoraStateFile,
|
||||
FAKE_CREATE_RELEASE_DURING_COPY:
|
||||
options.createReleaseDuringCopy === true ? 'true' : 'false',
|
||||
FAKE_RELEASE_ROOT: fixture.releaseRoot,
|
||||
@@ -1397,6 +1508,16 @@ function assertFileExists(filePath, reason) {
|
||||
}
|
||||
}
|
||||
|
||||
function assertNoReleasePromoted(fixture, reason) {
|
||||
const releaseDir = path.join(fixture.releaseRoot, fixture.version);
|
||||
if (existsSync(releaseDir)) {
|
||||
failures.push(`${reason} 已存在: ${releaseDir}`);
|
||||
}
|
||||
if (existsSync(fixture.currentLink)) {
|
||||
failures.push(`${reason} 不应切换 current: ${fixture.currentLink}`);
|
||||
}
|
||||
}
|
||||
|
||||
function readDirNames(directory) {
|
||||
return spawnSync('find', [directory, '-maxdepth', '1', '-mindepth', '1', '-printf', '%f\n'], {
|
||||
cwd: process.cwd(),
|
||||
|
||||
@@ -5317,21 +5317,33 @@ const checks = [
|
||||
},
|
||||
{
|
||||
file: 'scripts/deploy/production-api-deploy.sh',
|
||||
includes: 'systemctl is-active --quiet "${service_name}"',
|
||||
includes: 'ensure_pingora_shadow_service',
|
||||
reason:
|
||||
'API deploy 看到 Pingora 影子网关产物时必须先确认 shadow service 已 active。',
|
||||
'API deploy 看到 Pingora 影子网关产物时必须先进入受控 shadow service 启动流程。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/deploy/production-api-deploy.sh',
|
||||
includes: 'systemctl try-restart "${service_name}"',
|
||||
includes: 'systemctl cat "${service_name}"',
|
||||
reason:
|
||||
'API deploy 看到 Pingora 影子网关产物且 shadow service 已 active 时必须随 current release 切换重启。',
|
||||
'API deploy 自动启动或重启 Pingora 前必须读取 systemd 最终配置。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/deploy/production-api-deploy.sh',
|
||||
includes: 'Pingora 影子服务未处于 active，跳过自动重启',
|
||||
includes: 'CAP_NET_BIND_SERVICE',
|
||||
reason:
|
||||
'API deploy 不能在 Pingora shadow service 未运行时主动拉起新公网入口候选服务。',
|
||||
'API deploy 自动启动或重启 Pingora 前必须拒绝疑似 direct-entry 低端口 capability。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/deploy/production-api-deploy.sh',
|
||||
includes: 'Pingora 自动启动只允许 shadow 监听 127.0.0.1:18081',
|
||||
reason:
|
||||
'API deploy 自动启动 Pingora 时必须限制为本机 shadow 高端口，避免误拉起公网直连入口。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/deploy/production-api-deploy.sh',
|
||||
includes: 'systemctl restart "${service_name}"',
|
||||
reason:
|
||||
'API deploy 看到 Pingora 影子网关产物且 shadow 配置安全时必须随 current release 启动或重启。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/check-production-api-deploy.mjs',
|
||||
@@ -5341,9 +5353,21 @@ const checks = [
|
||||
},
|
||||
{
|
||||
file: 'scripts/check-production-api-deploy.mjs',
|
||||
includes: 'assertDeploySkipsInactivePingoraWhenArtifactIncluded',
|
||||
includes: 'assertDeployStartsInactivePingoraWhenArtifactIncluded',
|
||||
reason:
|
||||
'API deploy 动态烟测必须覆盖 Pingora shadow service inactive 时不会被主动拉起。',
|
||||
'API deploy 动态烟测必须覆盖 Pingora shadow service inactive 但配置安全时会被主动拉起。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/check-production-api-deploy.mjs',
|
||||
includes: 'assertDeployRejectsPingoraDirectEntryWhenArtifactIncluded',
|
||||
reason:
|
||||
'API deploy 动态烟测必须覆盖 direct-entry capability 下不会自动拉起 Pingora。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/check-production-api-deploy.mjs',
|
||||
includes: 'assertDeployRejectsPingoraPublicListenWhenArtifactIncluded',
|
||||
reason:
|
||||
'API deploy 动态烟测必须覆盖公网监听 env 下不会自动拉起 Pingora。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/check-production-api-release.mjs',
|
||||
|
||||
@@ -11,7 +11,7 @@ usage() {
|
||||
进入维护模式，校验并发布 api-server 单文件，更新 current 链接，重启 systemd 服务并执行 readiness 检查。
|
||||
默认同时重启外部生成 worker controller 和已加载的 worker 实例；未启用 worker 单元时会自动跳过。
|
||||
若传入 --database，会在重启前把 GENARRATIVE_SPACETIME_DATABASE 写入 api-server 环境文件，避免服务继续读取旧库。
|
||||
若发布包包含 pingora-gateway，部署脚本会在 current 链接切换后仅对已 active 的 Pingora 影子服务执行 try-restart 并复核 active；未运行时不会主动拉起。
|
||||
若发布包包含 pingora-gateway，部署脚本会在 current 链接切换后先复核 systemd/env 仍是本机高端口 shadow 配置，再启动或重启 Pingora 影子服务并复核 active。
|
||||
失败时保留维护模式。
|
||||
EOF
|
||||
}
|
||||
@@ -235,18 +235,101 @@ ensure_runtime_env_and_dirs() {
|
||||
fi
|
||||
}
|
||||
|
||||
restart_pingora_if_active() {
|
||||
extract_pingora_env_files_from_unit() {
|
||||
local service_name="$1"
|
||||
local unit_content
|
||||
|
||||
if ! systemctl is-active --quiet "${service_name}"; then
|
||||
echo "[production-api-deploy] Pingora 影子服务未处于 active，跳过自动重启: ${service_name}"
|
||||
return
|
||||
if ! unit_content="$(systemctl cat "${service_name}")"; then
|
||||
echo "[production-api-deploy] 无法读取 Pingora systemd 最终配置: ${service_name}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[production-api-deploy] 发布包包含 Pingora，重启已运行的影子服务: ${service_name}"
|
||||
systemctl try-restart "${service_name}"
|
||||
if printf "%s\n" "${unit_content}" | grep -Eq '^[[:space:]]*(AmbientCapabilities|CapabilityBoundingSet)=.*CAP_NET_BIND_SERVICE'; then
|
||||
echo "[production-api-deploy] Pingora systemd 已包含 CAP_NET_BIND_SERVICE，疑似直连入口配置；API deploy 不会自动启动或重启直连服务: ${service_name}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf "%s\n" "${unit_content}" | while IFS= read -r raw_line; do
|
||||
local line value token env_file
|
||||
line="${raw_line#"${raw_line%%[![:space:]]*}"}"
|
||||
[[ "${line}" == EnvironmentFile=* ]] || continue
|
||||
value="${line#EnvironmentFile=}"
|
||||
for token in ${value}; do
|
||||
env_file="${token#-}"
|
||||
env_file="${env_file%\"}"
|
||||
env_file="${env_file#\"}"
|
||||
env_file="${env_file%\'}"
|
||||
env_file="${env_file#\'}"
|
||||
[[ -n "${env_file}" ]] && printf "%s\n" "${env_file}"
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
find_pingora_gateway_env_file() {
|
||||
local service_name="$1"
|
||||
local env_file listen
|
||||
|
||||
while IFS= read -r env_file; do
|
||||
if [[ "${env_file}" != /* ]]; then
|
||||
echo "[production-api-deploy] Pingora EnvironmentFile 必须使用绝对路径: ${env_file}" >&2
|
||||
exit 1
|
||||
fi
|
||||
listen="$(read_env_value "${env_file}" "GENARRATIVE_PINGORA_GATEWAY_LISTEN")"
|
||||
if [[ -n "${listen}" ]]; then
|
||||
printf "%s\n" "${env_file}"
|
||||
return
|
||||
fi
|
||||
done < <(extract_pingora_env_files_from_unit "${service_name}")
|
||||
|
||||
echo "[production-api-deploy] Pingora systemd 配置缺少包含 GENARRATIVE_PINGORA_GATEWAY_LISTEN 的 EnvironmentFile: ${service_name}" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
require_pingora_shadow_env() {
|
||||
local env_file="$1"
|
||||
local listen tls_listen redirect_listen
|
||||
|
||||
listen="$(read_env_value "${env_file}" "GENARRATIVE_PINGORA_GATEWAY_LISTEN")"
|
||||
tls_listen="$(read_env_value "${env_file}" "GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN")"
|
||||
redirect_listen="$(read_env_value "${env_file}" "GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN")"
|
||||
|
||||
if [[ "${listen}" != "127.0.0.1:18081" ]]; then
|
||||
echo "[production-api-deploy] Pingora 自动启动只允许 shadow 监听 127.0.0.1:18081，当前 GENARRATIVE_PINGORA_GATEWAY_LISTEN=${listen:-<empty>}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "${tls_listen}" ]]; then
|
||||
echo "[production-api-deploy] Pingora 自动启动不允许启用 TLS_LISTEN，当前 GENARRATIVE_PINGORA_GATEWAY_TLS_LISTEN=${tls_listen}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "${redirect_listen}" ]]; then
|
||||
echo "[production-api-deploy] Pingora 自动启动不允许启用 HTTP_REDIRECT_LISTEN，当前 GENARRATIVE_PINGORA_GATEWAY_HTTP_REDIRECT_LISTEN=${redirect_listen}" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
check_pingora_shadow_service_config() {
|
||||
local service_name="$1"
|
||||
local env_file
|
||||
|
||||
env_file="$(find_pingora_gateway_env_file "${service_name}")"
|
||||
require_pingora_shadow_env "${env_file}"
|
||||
printf "%s\n" "${env_file}"
|
||||
}
|
||||
|
||||
ensure_pingora_shadow_service() {
|
||||
local service_name="$1"
|
||||
local env_file="${2:-}"
|
||||
|
||||
if [[ -z "${env_file}" ]]; then
|
||||
env_file="$(check_pingora_shadow_service_config "${service_name}")"
|
||||
else
|
||||
require_pingora_shadow_env "${env_file}"
|
||||
fi
|
||||
|
||||
echo "[production-api-deploy] 发布包包含 Pingora，启动或重启 shadow 影子服务: ${service_name} (${env_file})"
|
||||
systemctl restart "${service_name}"
|
||||
if ! systemctl is-active --quiet "${service_name}"; then
|
||||
echo "[production-api-deploy] Pingora 影子服务重启后不是 active: ${service_name}" >&2
|
||||
echo "[production-api-deploy] Pingora shadow 影子服务启动或重启后不是 active: ${service_name}" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
@@ -604,7 +687,7 @@ if [[ -f "${SOURCE_DIR}/pingora-gateway" ]]; then
|
||||
cp "${SOURCE_DIR}/pingora-gateway" "${RELEASE_CONTENT_DIR}/pingora-gateway"
|
||||
cp "${SOURCE_DIR}/pingora-gateway.sha256" "${RELEASE_CONTENT_DIR}/pingora-gateway.sha256"
|
||||
chmod +x "${RELEASE_CONTENT_DIR}/pingora-gateway"
|
||||
echo "[production-api-deploy] 已复制 Pingora 影子网关；current 链接切换后将复核并重启已运行的 ${PINGORA_SERVICE_NAME}"
|
||||
echo "[production-api-deploy] 已复制 Pingora 影子网关；current 链接切换后将复核 shadow 配置并启动或重启 ${PINGORA_SERVICE_NAME}"
|
||||
fi
|
||||
|
||||
BACKUP_SCRIPT_SOURCE="${SOURCE_DIR}/scripts/database-backup-to-oss.mjs"
|
||||
@@ -786,6 +869,8 @@ cp -R "${ENV_DEPLOY_DIR_SOURCE}" "${RELEASE_CONTENT_DIR}/deploy/env"
|
||||
|
||||
cp "${SOURCE_DIR}/release-manifest.json" "${RELEASE_CONTENT_DIR}/release-manifest.api-server.json"
|
||||
|
||||
PINGORA_SHADOW_ENV_FILE=""
|
||||
|
||||
if [[ -n "${DATABASE}" ]]; then
|
||||
echo "[production-api-deploy] 写入 api-server SpacetimeDB database: ${DATABASE} -> ${API_ENV_FILE}"
|
||||
write_env_value "${API_ENV_FILE}" "GENARRATIVE_SPACETIME_DATABASE" "${DATABASE}"
|
||||
@@ -798,6 +883,10 @@ fi
|
||||
|
||||
ensure_runtime_env_and_dirs "${API_ENV_FILE}"
|
||||
|
||||
if [[ "${PINGORA_INCLUDED}" -eq 1 ]]; then
|
||||
PINGORA_SHADOW_ENV_FILE="$(check_pingora_shadow_service_config "${PINGORA_SERVICE_NAME}")"
|
||||
fi
|
||||
|
||||
mkdir -p "$(dirname "${CURRENT_LINK}")"
|
||||
if [[ -e "${RELEASE_DIR}" ]]; then
|
||||
echo "[production-api-deploy] 目标 release 在发布过程中出现，拒绝合并 staging: ${RELEASE_DIR}" >&2
|
||||
@@ -808,7 +897,7 @@ STAGING_RELEASE_DIR=""
|
||||
ln -sfnT "${RELEASE_DIR}" "${CURRENT_LINK}"
|
||||
|
||||
if [[ "${PINGORA_INCLUDED}" -eq 1 ]]; then
|
||||
restart_pingora_if_active "${PINGORA_SERVICE_NAME}"
|
||||
ensure_pingora_shadow_service "${PINGORA_SERVICE_NAME}" "${PINGORA_SHADOW_ENV_FILE}"
|
||||
fi
|
||||
|
||||
echo "[production-api-deploy] 重启服务: ${SERVICE_NAME}"
|
||||
|
||||
Reference in New Issue
Block a user