diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 6be33af5a..bd998f442 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -3989,3 +3989,11 @@ - 限制：网关放行不等于后端存活；`pause-after-stdb` 停止 api-server 时，内网普通 API 和后台 API 仍不可用。 - 影响范围：生产 / dev Nginx 模板、维护 snippet、Pingora maintenance gate、Nginx 静态门禁与 Pingora smoke。 - 验证方式：`npm run check:nginx-spa-routes`、`npm run check:pingora-route-parity`、`cargo test -p pingora-gateway --manifest-path server-rs/Cargo.toml`、`npm run check:pingora-gateway-smoke`、`npm run check:production-ops`、`npm run check:encoding`、`git diff --check`。 + +## 2026-07-12 Full 发布显式控制成功后维护状态 + +- 背景：Full Job 只能用 `STDB_API_ROLLOUT_MODE` 控制 Stdb 与 API 之间是否暂停，但 API Deploy 在 readiness 成功后固定执行 `maintenance-off.sh`，因此无法选择完整流水线结束后继续保留维护页。 +- 决策：Full Job 新增默认勾选的 `EXIT_MAINTENANCE_MODE_AFTER_COMPLETION`，并让 Stdb Publish、API Deploy 两个下游阶段固定保持维护；Web Deploy 成功后才由独立 `Exit Maintenance` 阶段按该参数决定是否调用 current release 随包 `maintenance-off.sh`。API Deploy Job 单独使用 `KEEP_MAINTENANCE_MODE`，将其转换为随包 `production-api-deploy.sh --keep-maintenance-mode`；默认仍退出维护，失败路径继续沿用 current 切换前后既有安全语义。 +- 参数刷新：Jenkinsfile 是参数事实源。推送后必须让 Full 与 API Deploy live Job 安全加载一次新 Jenkinsfile，再只读确认两个参数已进入 `config.xml`；只在 Jenkins UI 手工加参数不是持久修复。 +- 影响范围：Full / API Deploy Jenkinsfile、API 发布脚本、生产 API deploy fixture、生产运维门禁与 live Job 参数 schema。 +- 验证方式：`bash -n scripts/deploy/production-api-deploy.sh`、`npm run check:production-api-deploy`、`npm run check:production-ops`、`npm run check:encoding`、`git diff --check`。 diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index 7cbc7e606..352e56d07 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -2968,3 +2968,10 @@ - 处理：Nginx 按 TCP `$remote_addr` 判断内网；Pingora 按 TCP peer 判断，只有 peer 为 loopback 的同机 Nginx 时才接受 Nginx 强制覆盖的 `X-Real-IP`。可信内网来源绕过整站维护响应，公网应用主站、普通 API、后台和 SpacetimeDB 路由仍保持维护响应；绝不能用 `X-Forwarded-For` 做放行判断。 - 验证：Pingora smoke 同时覆盖公网主站、普通 API、后台为 503，以及内网对应路由为 200；Rust 单测覆盖 IPv4 / IPv6 内网、公网和空来源；Nginx 静态门禁反查两份模板的内网来源定义与全局维护变量清零逻辑。 - 限制：如果发布门禁已经停止 api-server，网关放行后普通 API 和后台 API 仍会失败；需要调用后端时应确保对应服务仍运行，不能把维护页绕过误当作服务可用性保证。 + +## Full 结束后保持维护不能只加一个 UI 参数 + +- 现象：Full Job 参数页没有“完整发布成功后是否退出维护”选项，或者补了选项后 API readiness 一通过仍自动撤掉维护。 +- 原因：维护退出发生在随 API artifact 发布的 `production-api-deploy.sh` 内；Full、API Deploy Job 和脚本任一层没有透传，最终都会回到固定执行 `maintenance-off.sh`。Declarative Pipeline 参数还要等 live Job 加载新版 Jenkinsfile 后才会刷新。 +- 处理：Full 使用 `EXIT_MAINTENANCE_MODE_AFTER_COMPLETION` 表达产品选择，Stdb Publish 和 API Deploy 全程固定保持维护，Web Deploy 成功后才进入独立最终退出阶段；API Deploy 的独立 `KEEP_MAINTENANCE_MODE` 再转换为脚本 `--keep-maintenance-mode`。默认值仍在 Full 结束时退出维护，避免定时 dev 发布行为变化。 +- 验证：API deploy fixture 必须覆盖成功发布并保留 marker；生产运维静态门禁同时反查 Full 参数、下游透传、API Deploy 参数和脚本 flag。推送后用 fail-closed 首阶段运行刷新 live Job 参数，再核对 `config.xml`，不能只看仓库文件。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index 2bf6d98b8..5a640a933 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -71,6 +71,8 @@ lease 过期后不代表任务一定再次执行：claim transaction 只有在 ` 自 2026-07-11 起，`Genarrative-Full-Build-And-Deploy` 的每日 04:00 timer 默认以 `DEPLOY_TARGET=development`、`STDB_API_ROLLOUT_MODE=normal` 对仅供开发使用的 dev 服务器执行 Stdb → API → Web 完整发布，不进入人工 rollout gate。三个下游 Build 都由 Full Job 显式传 `PUBLISH_AFTER_BUILD=false`，不得依赖下游 Job 默认值或提前各自发布；统一 Build 完成后仍由 Full Job 按固定顺序发布。人工维护窗口才选择 `pause-after-stdb`，且必须配置 `STDB_API_ROLLOUT_APPROVERS`。上文“定时构建缺少审批人时失败”的旧口径不再作为当前 dev 定时发布行为。 +Full Job 通过 `EXIT_MAINTENANCE_MODE_AFTER_COMPLETION` 明确选择完整发布成功后是否退出维护，默认勾选以保持历史行为。Full 对 Stdb Publish 和 API Deploy 两个下游阶段都固定传 `KEEP_MAINTENANCE_MODE=true`，让 maintenance marker 持续覆盖 Stdb → API → Web 整段发布；Web Deploy 成功后才进入独立 `Exit Maintenance` 阶段。取消勾选时跳过最终退出阶段，便于内网验收完成后人工恢复公网。`Genarrative-Api-Deploy` 也单独暴露 `KEEP_MAINTENANCE_MODE` 参数，并转换为随发布包脚本的 `--keep-maintenance-mode`；失败路径仍按既有 current 切换边界保留或退出维护，不受成功态选项覆盖。 + 需要验证“更新 API 不停 worker”和“worker 是否持续消费队列”时，优先使用隔离容器 smoke：`npm run container:worker-smoke -- smoke`。该脚本生成 gitignored 的 `deploy/container/worker-smoke/api-server.env`，启动独立 compose project 与独立 SpacetimeDB，发布当前 `spacetime-module` 后写入 `worker_smoke_unsupported` 测试 job；预期 worker claim 后执行 unsupported 失败分支，再执行 API-only recreate 并确认 worker 容器 ID 不变，最后再次入队验证 API 更新后队列仍可消费。`external_generation_job` 是 private table，脚本通过 worker 日志确认 job_id 被消费，不用 CLI SQL 查询私表。该 smoke 不读取 `.env.local`，也不依赖真实 VectorEngine / OSS 密钥；真实生图链路联调再在本地私有 env 中补齐 provider 配置。worker-smoke 默认把本机 `spacetime` CLI 打成轻量 SpacetimeDB 镜像，避免本机首次 smoke 依赖官方大镜像下载。若容器内 Cargo 拉取 crates.io 依赖不稳定，可用 `npm run container:worker-smoke -- smoke --local-binary` 让容器内 Cargo 复用本机 Cargo 缓存构建当前二进制，再打入 Debian bookworm smoke runtime 临时镜像；可用 `GENARRATIVE_WORKER_SMOKE_LOCAL_BASE_IMAGE` 覆盖运行时基础镜像；若隔离端口或库数据需要重建，追加 `--force`。完成 queue 链路验证时，还要用队列概览 BFF 和单 job 状态接口确认 job 从 queued/running 收敛，并用对应玩法 session/detail 接口确认业务状态同步完成。 本地只做账号/UI smoke 且需要短信登录时，`SMS_AUTH_PROVIDER` 应显式设为 `mock`，并把 `SMS_AUTH_MOCK_VERIFY_CODE` 设为固定值（当前常用 `123456`），再重启 `npm run dev` 或 `npm run dev:api-server`。如果 `.env.local` 还保留 `SMS_AUTH_PROVIDER=aliyun`，`POST /api/auth/phone/login` 用 mock 验证码会稳定报“验证码错误”，不是前端表单问题。真实短信联调再切回 `aliyun` 并重启。 @@ -409,6 +411,8 @@ Nginx 与 Pingora 在维护 marker 存在时对内网来源绕过整站维护闸 该规则只绕过网关维护响应，不会自动拉起 api-server、SpacetimeDB 或其它已停止的服务。人工执行 `maintenance-on.sh` 且后端仍运行时，可以从内网继续访问整站和修改后台数据；`pause-after-stdb` 会停止旧 API/controller/worker，在 API 被停期间静态页面可能仍可加载，但普通 API 与 `/admin/api/**` 仍不可用。验证使用 `npm run check:nginx-spa-routes`、`npm run check:pingora-route-parity`、`cargo test -p pingora-gateway --manifest-path server-rs/Cargo.toml`、`npm run check:pingora-gateway-smoke` 和 `npm run check:production-ops`，不要在 live 机器上为测试临时创建维护 marker。 +维护页源码固定为 `public/maintenance.html`，正常 Web 构建由 Vite 复制到发布包根目录的 `web/maintenance.html`。计划内停服需要临时更新公告时，先更新该源码，避免后续 Web Deploy 把现场公告覆盖回旧内容；现场紧急替换必须原子写入当前 `/srv/genarrative/web/maintenance.html`，并同时用 `genarrative.world` 与 `www.genarrative.world` 的真实 HTTPS 响应校验 `503` 和公告正文。 + 生产 Jenkins 的 `Pipeline script from SCM` 由 Jenkins controller 读取 Jenkinsfile。`Genarrative-Server-Provision` 是服务器初始化流水线，Job 配置里的 SCM URL 必须使用 controller 本机可访问的仓库路径或内网 Gitea 地址，不能使用 `https://git.genarrative.world/...`；否则日志一开始的 `Checking out git ... to read jenkins/Jenkinsfile.production-server-provision` 就会先从公网拉 Jenkinsfile。构建类流水线和 `Genarrative-Server-Provision` 的 Jenkinsfile 内部源码准备阶段统一使用 `ssh://git@192.168.35.82:2222/GenarrativeAI/Genarrative.git`，并显式传入 Jenkins SSH 凭据 `genarrative-local-gitea-ssh`；不再配置 `https://git.genarrative.world/...` 公网 fallback，也不再默认使用 `http://genarrative-station/git/GenarrativeAI/Genarrative.git`。所有 `GitSCM checkout` 都必须保留单分支 refspec、`shallow=true`、`depth=1`、`noTags=true` 与 `honorRefspec=true`。API / Web / Stdb 发布类流水线不在目标机器 checkout Git，统一执行上游构建归档里的部署脚本，避免产物 commit 与部署脚本 commit 漂移；Server-Provision 也不在目标 dev / release agent checkout Git，而是由 Jenkins 构建节点先准备 provision 脚本与配置并上传给目标 agent。 当前 Jenkins / 本机内网 Git 入口固定为 `ssh://git@192.168.35.82:2222/GenarrativeAI/Genarrative.git`，用于 controller、构建节点和本机 Agent 直接拉取仓库，避免绕公网 `git.genarrative.world`。验证时在具备对应 SSH key 和 known_hosts 的环境执行 `git ls-remote ssh://git@192.168.35.82:2222/GenarrativeAI/Genarrative.git HEAD`，应能返回 HEAD。若机器仍保留旧的 `http://genarrative-station/git/GenarrativeAI/Genarrative.git` 或 `http://10.2.0.10/GenarrativeAI/Genarrative.git` 内网入口，只作为历史兼容和排障参考，新流水线不再默认使用。 diff --git a/jenkins/Jenkinsfile.production-api-deploy b/jenkins/Jenkinsfile.production-api-deploy index 8c21fe16f..aef01f183 100644 --- a/jenkins/Jenkinsfile.production-api-deploy +++ b/jenkins/Jenkinsfile.production-api-deploy @@ -17,6 +17,7 @@ pipeline { string(name: 'BUILD_JOB_NAME', defaultValue: 'Genarrative-Api-Build', description: 'API 构建流水线作业名') string(name: 'BUILD_NUMBER_TO_DEPLOY', defaultValue: '', description: '要复制归档产物的上游构建号') booleanParam(name: 'INCLUDE_PINGORA_GATEWAY', defaultValue: true, description: '上游构建是否包含 Pingora 影子网关产物；release 发布默认必须包含') + booleanParam(name: 'KEEP_MAINTENANCE_MODE', defaultValue: false, description: '发布成功且 readiness 通过后是否继续保持维护模式；默认退出维护') string(name: 'RELEASE_ROOT', defaultValue: '/opt/genarrative/releases', description: '生产 release 根目录') string(name: 'CURRENT_LINK', defaultValue: '/opt/genarrative/current', description: '当前版本软链接') string(name: 'SERVICE_NAME', defaultValue: 'genarrative-api.service', description: 'systemd 服务名') @@ -98,9 +99,13 @@ pipeline { set -euo pipefail chmod +x "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-on.sh" "build/${BUILD_VERSION}/scripts/deploy/maintenance-off.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-direct-rollback.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-enable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-realpath-canary-disable.sh" "build/${BUILD_VERSION}/scripts/deploy/pingora-health-patrol-env-switch.mjs" "build/${BUILD_VERSION}/scripts/deploy/pingora-gateway-env-shadow-switch.mjs" "build/${BUILD_VERSION}/scripts/deploy/pingora-tls-cert-sync.mjs" pingora_deploy_args=() + maintenance_deploy_args=() if [[ "${INCLUDE_PINGORA_GATEWAY:-false}" == "true" ]]; then pingora_deploy_args+=(--require-pingora-gateway) fi + if [[ "${KEEP_MAINTENANCE_MODE:-false}" == "true" ]]; then + maintenance_deploy_args+=(--keep-maintenance-mode) + fi "build/${BUILD_VERSION}/scripts/deploy/production-api-deploy.sh" \ --source-dir "build/${BUILD_VERSION}" \ --version "${BUILD_VERSION}" \ @@ -108,6 +113,7 @@ pipeline { --current-link "${CURRENT_LINK}" \ --service "${SERVICE_NAME}" \ "${pingora_deploy_args[@]}" \ + "${maintenance_deploy_args[@]}" \ --health-url "${HEALTH_URL}" \ --api-env-file "${API_ENV_FILE:-/etc/genarrative/api-server.env}" \ --worker-env-file "${WORKER_ENV_FILE:-/etc/genarrative/external-generation-worker.env}" \ @@ -133,6 +139,7 @@ pipeline { string(name: 'DEPLOY_TARGET', value: params.DEPLOY_TARGET ?: ''), string(name: 'DATABASE', value: params.DATABASE ?: ''), string(name: 'INCLUDE_PINGORA_GATEWAY', value: String.valueOf(params.INCLUDE_PINGORA_GATEWAY)), + string(name: 'KEEP_MAINTENANCE_MODE', value: String.valueOf(params.KEEP_MAINTENANCE_MODE)), string(name: 'SUMMARY', value: 'API 发布流水线结束'), ] def notificationRecipients = params.NOTIFICATION_EMAILS?.trim() diff --git a/jenkins/Jenkinsfile.production-full-build-and-deploy b/jenkins/Jenkinsfile.production-full-build-and-deploy index 96d61d361..dd6bc36fd 100644 --- a/jenkins/Jenkinsfile.production-full-build-and-deploy +++ b/jenkins/Jenkinsfile.production-full-build-and-deploy @@ -42,6 +42,7 @@ pipeline { string(name: 'WORKER_ENV_FILE', defaultValue: '/etc/genarrative/external-generation-worker.env', description: 'API 与 Stdb publish 共用的 worker 环境文件') choice(name: 'STDB_API_ROLLOUT_MODE', choices: ['normal', 'pause-after-stdb'], description: '定时任务默认 normal 完整发布 dev；人工维护窗口才选择 pause-after-stdb') string(name: 'STDB_API_ROLLOUT_APPROVERS', defaultValue: '', description: 'pause-after-stdb 必填：允许放行 API 的 Jenkins 用户/组，多个值用逗号分隔') + booleanParam(name: 'EXIT_MAINTENANCE_MODE_AFTER_COMPLETION', defaultValue: true, description: '完整发布成功后是否退出维护模式；取消勾选会保留维护页，便于内网验收后人工恢复公网') } stages { @@ -192,7 +193,7 @@ pipeline { string(name: 'SPACETIME_RUN_AS_USER', value: params.SPACETIME_RUN_AS_USER ?: 'spacetimedb'), string(name: 'API_ENV_FILE', value: params.API_ENV_FILE ?: '/etc/genarrative/api-server.env'), string(name: 'WORKER_ENV_FILE', value: params.WORKER_ENV_FILE ?: '/etc/genarrative/external-generation-worker.env'), - booleanParam(name: 'KEEP_MAINTENANCE_MODE', value: params.STDB_API_ROLLOUT_MODE == 'pause-after-stdb'), + booleanParam(name: 'KEEP_MAINTENANCE_MODE', value: true), string(name: 'DEPLOY_TARGET', value: params.DEPLOY_TARGET), booleanParam(name: 'CONFIRM_RELEASE_DEPLOY_AGENT', value: params.CONFIRM_RELEASE_DEPLOY_AGENT), string(name: 'BUILD_JOB_NAME', value: params.STDB_BUILD_JOB_NAME), @@ -227,6 +228,7 @@ pipeline { string(name: 'BUILD_JOB_NAME', value: params.API_BUILD_JOB_NAME), string(name: 'BUILD_NUMBER_TO_DEPLOY', value: env.API_BUILD_NUMBER), booleanParam(name: 'INCLUDE_PINGORA_GATEWAY', value: params.INCLUDE_PINGORA_GATEWAY), + booleanParam(name: 'KEEP_MAINTENANCE_MODE', value: true), string(name: 'API_ENV_FILE', value: params.API_ENV_FILE ?: '/etc/genarrative/api-server.env'), string(name: 'WORKER_ENV_FILE', value: params.WORKER_ENV_FILE ?: '/etc/genarrative/external-generation-worker.env'), string(name: 'DATABASE', value: params.DATABASE), @@ -252,6 +254,28 @@ pipeline { ] } } + + stage('Exit Maintenance') { + when { + expression { return params.EXIT_MAINTENANCE_MODE_AFTER_COMPLETION != false } + } + agent { + label "${params.DEPLOY_TARGET == 'development' ? 'linux && genarrative-dev-deploy' : 'linux && genarrative-release-deploy'}" + } + steps { + sh ''' + bash -lc ' + set -euo pipefail + maintenance_script="/opt/genarrative/current/scripts/deploy/maintenance-off.sh" + if [[ ! -f "${maintenance_script}" ]]; then + echo "Full 发布完成但 current release 缺少维护退出脚本: ${maintenance_script}" >&2 + exit 1 + fi + bash "${maintenance_script}" + ' + ''' + } + } } post { @@ -268,6 +292,7 @@ pipeline { string(name: 'DEPLOY_TARGET', value: params.DEPLOY_TARGET ?: ''), string(name: 'DATABASE', value: params.DATABASE ?: ''), string(name: 'INCLUDE_PINGORA_GATEWAY', value: String.valueOf(params.INCLUDE_PINGORA_GATEWAY)), + string(name: 'EXIT_MAINTENANCE_MODE_AFTER_COMPLETION', value: String.valueOf(params.EXIT_MAINTENANCE_MODE_AFTER_COMPLETION)), string(name: 'SUMMARY', value: '全量构建发布编排结束'), ] def notificationRecipients = params.NOTIFICATION_EMAILS?.trim() diff --git a/public/maintenance.html b/public/maintenance.html new file mode 100644 index 000000000..75799f377 --- /dev/null +++ b/public/maintenance.html @@ -0,0 +1,187 @@ + + + + + + + 陶泥儿停服升级公告 + + + +
+
+ + 服务维护中 +
+ +

陶泥儿美术工具停服升级

+

+ 各位陶泥儿创作者，美术工具官网将在今天晚上 21:00~22:00 停服升级。
+ 注意保存好自己的项目和下载好素材。 +

+ +
+ 维护时间 + 今天晚上 21:00~22:00 +
+ +
+

本次更新修复内容

+
    +
  1. 修复多个影响画布使用的 BUG。
    2. +
  2. + 修复了 PC 端不能充值的恶性 BUG（bushi），PC + 网页端支持微信二维码充值。 +
    3. +
+
+ +
+

新增功能

+

+ 抠图与背景色功能升级：使用更强、更稳定的抠图技术，解决角色扣残问题，角色动作的逐帧抠图同步升级。 +

+
+
+ + diff --git a/scripts/check-production-api-deploy.mjs b/scripts/check-production-api-deploy.mjs index f46e6d961..f08715dfd 100644 --- a/scripts/check-production-api-deploy.mjs +++ b/scripts/check-production-api-deploy.mjs @@ -36,6 +36,7 @@ console.log('[check:production-api-deploy] OK'); function main() { assertDeployCopiesPingoraDirectReleaseDependencies(); + assertSuccessfulDeployCanKeepMaintenance(); assertDeployRestartsActivePingoraWhenArtifactIncluded(); assertDeployStartsInactivePingoraWhenArtifactIncluded(); assertDeployRejectsPingoraDirectEntryWhenArtifactIncluded(); @@ -79,6 +80,22 @@ function main() { assertMissingPingoraCanaryAccessLogParityFails(); } +function assertSuccessfulDeployCanKeepMaintenance() { + const fixture = prepareFixture('keep-maintenance-after-success'); + const result = runDeploy(fixture, { keepMaintenance: true }); + + assertStatus(result, 0, '显式保持维护时完整 fixture 应部署成功。'); + if (result.status !== 0) { + return; + } + assertMaintenanceKept(fixture, '显式要求成功部署后保持维护'); + assertIncludes( + result.stdout, + '按参数保持维护模式', + '成功部署并保持维护时必须输出明确状态。', + ); +} + function readOptionalCommandsLog(fixture) { if (!existsSync(fixture.commandsLog)) { return ''; @@ -1666,6 +1683,9 @@ function runDeploy(fixture, options = {}) { if (options.requirePingoraGateway) { args.push('--require-pingora-gateway'); } + if (options.keepMaintenance) { + args.push('--keep-maintenance-mode'); + } return spawnSync( 'bash', args, diff --git a/scripts/check-production-ops-guardrails.mjs b/scripts/check-production-ops-guardrails.mjs index 09e3b359e..be546af1b 100644 --- a/scripts/check-production-ops-guardrails.mjs +++ b/scripts/check-production-ops-guardrails.mjs @@ -157,6 +157,27 @@ const checks = [ includes: '继承已有维护模式；部署失败时不得误退出上游维护窗口', reason: 'API deploy 必须区分自己打开的维护模式与 Stdb gate 继承的维护模式。', }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: '--keep-maintenance-mode)', + reason: 'API deploy 必须允许成功发布后按显式参数保留维护模式。', + }, + { + file: 'scripts/deploy/production-api-deploy.sh', + includes: 'readiness 通过，按参数保持维护模式', + reason: 'API deploy 保留维护模式时必须在 readiness 通过后给出明确状态。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: + "booleanParam(name: 'KEEP_MAINTENANCE_MODE', defaultValue: false", + reason: 'API Deploy Job 必须向独立发布和 Full 编排暴露成功后保持维护的参数。', + }, + { + file: 'jenkins/Jenkinsfile.production-api-deploy', + includes: 'maintenance_deploy_args+=(--keep-maintenance-mode)', + reason: 'API Deploy Job 必须把保持维护参数传给发布产物内的部署脚本。', + }, { file: 'scripts/jenkins-server-provision.sh', includes: 'ensure_runtime_bootstrap_secret_file_env', @@ -353,8 +374,32 @@ const checks = [ { file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', includes: - "booleanParam(name: 'KEEP_MAINTENANCE_MODE', value: params.STDB_API_ROLLOUT_MODE == 'pause-after-stdb')", - reason: 'Full Build 暂停时必须要求 Stdb Publish 保持维护模式和旧运行时停止状态。', + "booleanParam(name: 'KEEP_MAINTENANCE_MODE', value: true)", + reason: 'Full Build 必须让 Stdb 与 API Deploy 全程保持维护，直到 Web Deploy 结束。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + "booleanParam(name: 'EXIT_MAINTENANCE_MODE_AFTER_COMPLETION', defaultValue: true", + reason: 'Full Build 必须显式提供完整发布成功后是否退出维护模式的选项。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + "stage('Exit Maintenance')", + reason: 'Full Build 必须在 Web Deploy 之后用独立阶段决定是否退出维护。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + 'expression { return params.EXIT_MAINTENANCE_MODE_AFTER_COMPLETION != false }', + reason: 'Full Build 只有在显式允许时才执行最终维护退出阶段。', + }, + { + file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', + includes: + 'maintenance_script="/opt/genarrative/current/scripts/deploy/maintenance-off.sh"', + reason: 'Full Build 必须使用本次 current release 随包维护脚本退出维护。', }, { file: 'jenkins/Jenkinsfile.production-full-build-and-deploy', @@ -6779,6 +6824,16 @@ if ((forcedBuildOnlyCalls?.length ?? 0) !== 3) { ); } +const fullPipelineMaintenanceHoldCalls = fullPipelineContent.match( + /booleanParam\(name: 'KEEP_MAINTENANCE_MODE', value: true\)/gu, +); +if ((fullPipelineMaintenanceHoldCalls?.length ?? 0) !== 2) { + failed = true; + console.error( + '[check:production-ops] Full Build 必须让 Stdb Publish 与 API Deploy 两个下游阶段都保持维护模式。', + ); +} + for (const file of nodeEnvFileCommandFiles) { const content = readFileSync(file, 'utf8'); const commandText = content.replace(/\\\r?\n\s*/g, ' '); diff --git a/scripts/deploy/production-api-deploy.sh b/scripts/deploy/production-api-deploy.sh index 90d1af69b..c7a7e80c6 100644 --- a/scripts/deploy/production-api-deploy.sh +++ b/scripts/deploy/production-api-deploy.sh @@ -5,13 +5,14 @@ set -euo pipefail usage() { cat <<'EOF' 用法: - ./scripts/deploy/production-api-deploy.sh --source-dir build/ [--version ] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] + ./scripts/deploy/production-api-deploy.sh --source-dir build/ [--version ] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--pingora-service genarrative-pingora-gateway.service] [--require-pingora-gateway] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--worker-env-file /etc/genarrative/external-generation-worker.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101] [--keep-maintenance-mode] 说明: 进入维护模式，校验并发布 api-server 单文件，更新 current 链接，重启 systemd 服务并执行 readiness 检查。 默认同时重启外部生成 worker controller 和已加载的 worker 实例；未启用 worker 单元时会自动跳过。 若传入 --database，会在重启前把 GENARRATIVE_SPACETIME_DATABASE 写入 api-server 环境文件，避免服务继续读取旧库。 若发布包包含 pingora-gateway，或传入 --require-pingora-gateway，部署脚本会要求 release manifest、二进制与 checksum 一致，再在 current 链接切换后先复核 systemd/env 仍是本机高端口 shadow 配置，启动或重启 Pingora 影子服务并复核 active。 + 默认在 readiness 通过后退出维护模式；传入 --keep-maintenance-mode 时保留维护文件，供人工验收后再恢复公网。 current 链接切换前失败时会退出本次打开的维护模式；current 链接切换后失败时保留维护模式，避免暴露半发布版本。 EOF } @@ -687,6 +688,7 @@ SPACETIME_SERVER_URL="" DEPLOY_COMPLETED=0 PINGORA_INCLUDED=0 REQUIRE_PINGORA_GATEWAY=0 +KEEP_MAINTENANCE_MODE=0 MAINTENANCE_ENABLED_BY_DEPLOY=0 MAINTENANCE_FILE="${GENARRATIVE_MAINTENANCE_FILE:-/var/lib/genarrative/maintenance/enabled}" CURRENT_LINK_SWITCHED=0 @@ -727,6 +729,10 @@ while [[ $# -gt 0 ]]; do REQUIRE_PINGORA_GATEWAY=1 shift ;; + --keep-maintenance-mode) + KEEP_MAINTENANCE_MODE=1 + shift + ;; --worker-service-pattern) WORKER_SERVICE_PATTERN="${2:?缺少 --worker-service-pattern 的值}" shift 2 @@ -1121,7 +1127,11 @@ wait_for_worker_controller_service "${WORKER_CONTROLLER_SERVICE}" echo "[production-api-deploy] 等待 readiness: ${HEALTH_URL}" for _ in {1..30}; do if curl -fsS "${HEALTH_URL}" >/dev/null; then - bash "${SCRIPT_DIR}/maintenance-off.sh" + if [[ "${KEEP_MAINTENANCE_MODE}" -eq 1 ]]; then + echo "[production-api-deploy] readiness 通过，按参数保持维护模式: ${MAINTENANCE_FILE}" + else + bash "${SCRIPT_DIR}/maintenance-off.sh" + fi DEPLOY_COMPLETED=1 echo "[production-api-deploy] 完成: ${RELEASE_DIR}/api-server" exit 0