完善SpacetimeDB逐文件增量备份
新增data-dir逐文件CAS基线、history归档与安全清理 发布并校验OSS latest pointer,支持异机自动恢复 接入dev和release可选的files-history定时备份profile 补齐备份测试、生产门禁、环境示例与运维文档
This commit is contained in:
Vendored
+6
@@ -157,8 +157,14 @@ GENARRATIVE_DATABASE_BACKUP_OSS_BUCKET=
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_ENDPOINT=oss-cn-shanghai.aliyuncs.com
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_PREFIX=database-backups
|
||||
GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL=false
|
||||
# 可选:files 为逐文件 CAS + catalog,不生成 tar.gz;archive 保留旧全量压缩包兼容行为。
|
||||
GENARRATIVE_DATABASE_BACKUP_STORAGE_FORMAT=archive
|
||||
# 可选:显式要求备份工作目录所在文件系统至少保留的可用空间;为空时按数据目录大小 + 安全余量估算。
|
||||
GENARRATIVE_DATABASE_BACKUP_MIN_FREE_BYTES=
|
||||
# archive history 模式持久化已验真 full baseline 与追加批次;files 模式改用 work-dir 下的 files state。
|
||||
GENARRATIVE_DATABASE_BACKUP_BASELINE_STATE=/var/lib/genarrative/database-backups/genarrative-prod-history-state.json
|
||||
# 仅 archive history 首次从 uploadStatus=uploaded 的 full manifest 初始化 state 时设置;初始化后可留空。
|
||||
GENARRATIVE_DATABASE_BACKUP_BASELINE_MANIFEST=
|
||||
# 可选:定时 / publish 前备份使用独立最小权限 AccessKey;为空时回退 ALIYUN_OSS_ACCESS_KEY_*。
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_ID=
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_SECRET=
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/bin/node -- /opt/genarrative/current/scripts/database-backup-to-oss.mjs --env-file /etc/genarrative/api-server.env --storage-format files --mode history --work-dir /var/lib/genarrative/database-backups/files-history
|
||||
@@ -16,6 +16,19 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-16 SpacetimeDB 备份采用逐文件基线、CAS 增量与安全历史清理
|
||||
|
||||
- 背景:SpacetimeDB standalone 2.6.0 不自动删除已被 snapshot 覆盖的历史 commitlog 与旧 snapshot;反复压缩整个 `/stdb` 会重复占用磁盘、停机和 OSS 带宽。上游 issue #5542 的 contributor 明确说明,不触碰最新 snapshot 与重启所需 commitlog suffix 时,可在运行中移动或删除这些历史文件。
|
||||
- 决策:统一脚本新增 `--storage-format files`,完整基线递归保留目录与文件相对路径,普通文件按 SHA-256 上传为不可变 CAS 对象,catalog 记录目录、路径、长度、SHA 与对象 key;相同内容不重复 PUT,后续 full 扫描只上传新增或变化内容,不再生成 tar.gz。旧 `archive` 路径保留兼容。full 必须从停库目录或已验证的冻结副本生成,不能把在线跨文件扫描称为一致时点备份。
|
||||
- history 继续按 replica 计算安全边界:只接受完整、未锁定且含同 offset `.snapshot_bsatn` 的 snapshot,保留跨越最新 snapshot 的边界 segment 及全部后缀。旧 segment 对和旧 snapshot 被递归映射为单文件 CAS 对象;对象、history catalog、full baseline catalog、候选 fingerprint 与当前边界全部验真后才删除源文件。同库执行用 work-dir PID lock 互斥。
|
||||
- OSS 固定恢复入口为 `<prefix>/<database>/latest.json`。CAS 文件和 full/history catalog 保持不可变;latest pointer 只保存最新 full catalog 与已发布 history catalog 的 object key、长度和 SHA,不包含主机绝对路径或文件内容。每次 state 变化先验真全部引用 catalog,再覆盖上传并 HEAD 验真 latest pointer,成功后才落本地 state;history 还必须在 pointer 成功后才允许删除源文件。全新机器可仅凭 bucket、database、prefix 与 OSS 凭据自动下载 pointer 和 full catalog。
|
||||
- dev 带宽不足时,允许把已冻结的 dev 基线经 `10.2.0.10 -> 10.2.4.16` 内网 rsync 到 release 独立 staging,再用 release 出口上传 dev bucket;staging 不得指向 release `/stdb`,不得停止或修改 release 服务,传输凭据必须临时创建并在演练后移除。catalog 不记录 staging 绝对路径,files state 可回传 dev 继续 history。
|
||||
- 恢复边界:恢复时默认从 OSS `latest.json` 自动定位 full catalog,创建目录并按相对路径下载每个对象、逐文件校验长度与 SHA;本地 state 只用于备份续跑,不再是异机恢复前置条件。远程 dev 已完成真实 OSS、清理、重启和异机隔离恢复演练;release timer 与 publish 前备份继续保持原行为。
|
||||
- systemd 接线:主 service 保持 `archive-full`。Server-Provision 新增默认值为 `archive-full` 的 `DATABASE_BACKUP_PROFILE`;dev 或 release 显式选择 `files-history` 时,必须为各自主机指定独立 work-dir,并先用 current release 脚本执行 history dry-run,确认已有 full state 后才安装仓库托管 drop-in,并删除现场手写旧 drop-in。切回默认 profile 必须删除所有 history 覆盖。
|
||||
- 影响范围:`scripts/database-backup-to-oss.mjs`、备份门禁、生产 env 示例、systemd 模板、Server-Provision、SpacetimeDB 运维与恢复流程;release timer 可在独立 baseline 验证后显式选择 profile,publish 前备份是否切换仍需单独决策。
|
||||
- 验证方式:`npm run check:database-backup`、`npm run check:production-ops`、`npm run check:encoding`、`git diff --check`;dev 现场必须完成逐文件 full catalog、重复 full 零 PUT、history dry-run、上传后清理、STDB 重启和按 catalog 隔离恢复 roundtrip。
|
||||
- 关联:<https://github.com/clockworklabs/SpacetimeDB/issues/5542#issuecomment-4981566448>。
|
||||
|
||||
## 2026-07-14 后台账号采用 owner 引导账号与一级 Tab 实时授权
|
||||
|
||||
- 背景:后台此前只支持一组环境变量管理员,所有 `/admin/api/*` 共用统一 admin 门禁,无法给运营、审核等人员分配独立账号和页面范围。
|
||||
|
||||
@@ -3117,3 +3117,11 @@
|
||||
- 原因:`.admin-info-list div` 会命中列表内所有后代 `div`,把字段值内部的 `.admin-inline-identity` 和昵称容器也覆盖成双列 grid;陶泥号又允许任意位置换行,最终只剩单字符宽度。素材详情布局若始终固定为 `220px + 信息列`,移动端也没有足够空间。
|
||||
- 处理:信息列表的行布局只使用直接子选择器 `.admin-info-list > div`;作者昵称与陶泥号在身份组件内分行,陶泥号保持单行并在真正不足时省略。`560px` 以下的素材详情改为单列,缩略图居中;素材查询与精选审核共用该规则。
|
||||
- 验证:在桌面、560px、390px 和 320px 浏览器宽度打开素材详情,确认 `.admin-inline-identity` 的 computed `display` 为 `flex`、陶泥号横向显示、详情字段不溢出页面。
|
||||
|
||||
## SpacetimeDB 历史归档不能按文件名小于 snapshot 就全部删除
|
||||
|
||||
- 现象:看到最新 `N.snapshot_dir` 后,把所有起始 offset 小于 `N` 的 `.stdb.log` 删除,或者只把旧日志上传 OSS 就宣称已有完整增量灾备。
|
||||
- 原因:segment 文件名只表示该段最早事务;起始 offset 小于等于最新 snapshot 的最后一个 segment 可能跨越 snapshot 边界,重启仍需要它。历史归档也不会及时覆盖 control-db、program bytes、最新 snapshot 和 active segment。
|
||||
- 处理:latest snapshot 必须是未锁定且存在同 offset `.snapshot_bsatn` 的完整目录,空目录或同名 `.lock` 存在时忽略。每个 replica 独立保留 `max(segment_start <= latest_snapshot)` 及全部后缀,只处理更早 segment 对;旧 snapshot 只保留最新一个。`--storage-format files` 必须先发布完整 full catalog;history 对每个候选文件 CAS 对象、history catalog 和 full catalog 执行 HEAD 长度/SHA 验真,再复算边界与 stat fingerprint,最后发布并验真固定 `latest.json`;pointer 失败时不得推进 state 或删除源文件。不要把在线逐文件 full 扫描当成跨文件一致备份,基线必须来自停库目录或已验证冻结副本。SSH 或工具超时后先检查 work-dir PID lock 与原进程,不要直接并发重跑;不要在 history 模式传 `--stop-service`。定时任务通过 Server-Provision 的显式 profile 和仓库 drop-in 管理,启用前 dry-run 验证 baseline,切回 archive 时同时移除托管与现场遗留 drop-in;不要在 `/etc/systemd/system` 长期保留手写覆盖,release 必须建立和验证自己的 full baseline 与 work-dir,不能直接复用 dev 的本地 state。
|
||||
- 验证:dry-run 输出 replica 的 `latestSnapshot`、`boundarySegment` 和候选清单;从另一台机器仅凭 OSS `latest.json` 自动定位 full catalog,创建目录、下载文件并逐项校验长度/SHA,启动隔离 data-dir 验证 `/v1/ping`、snapshot restore、commitlog replay、module launch、代表性 SQL 与 reducer。
|
||||
- 关联:`scripts/database-backup-to-oss.mjs`、`scripts/check-database-backup-to-oss.mjs`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
|
||||
|
||||
@@ -303,7 +303,7 @@ UI 相关修改要重点验证:
|
||||
|
||||
### SpacetimeDB 数据目录 OSS 备份
|
||||
|
||||
数据库备份不放进 `spacetime-module` reducer / procedure:备份属于文件系统与 OSS 外部副作用,必须由运维脚本在 SpacetimeDB 宿主外执行。当前统一脚本为 `scripts/database-backup-to-oss.mjs`(npm 命令 `npm run database:backup:oss`);生产 provision 还会安装 `genarrative-database-backup.timer`,每天 `03:20` 左右自动执行一次 OSS 冷备份:
|
||||
数据库备份不放进 `spacetime-module` reducer / procedure:备份属于文件系统与 OSS 外部副作用,必须由运维脚本在 SpacetimeDB 宿主外执行。当前统一脚本为 `scripts/database-backup-to-oss.mjs`(npm 命令 `npm run database:backup:oss`)。默认 `--storage-format archive --mode full` 保持原有全量压缩包冷备行为;`--storage-format files` 不生成 tar.gz,而是把目录树映射成逐文件 CAS 对象与 catalog,full 重跑只上传新增或内容变化的文件,history 只处理已被最新 snapshot 完全覆盖的历史 commitlog 与旧 snapshot。`Genarrative-Server-Provision` 的 `DATABASE_BACKUP_PROFILE` 默认是 `archive-full`,继续安装每天 `03:20` 左右执行的全量冷备主 service;development 和 release 都可以显式选择 `files-history`,但指定 work-dir 必须已经有与本机 database/bucket 匹配且已发布的 full baseline state:
|
||||
|
||||
```bash
|
||||
npm run database:backup:oss -- --data-dir /stdb --stop-service spacetimedb.service --restart-service-after genarrative-api.service --restart-service-after genarrative-external-generation-worker@1.service --restart-service-after genarrative-external-generation-controller.service
|
||||
@@ -326,13 +326,67 @@ GENARRATIVE_DATABASE_BACKUP_OSS_BUCKET=
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_ENDPOINT=oss-cn-shanghai.aliyuncs.com
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_PREFIX=database-backups
|
||||
GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL=false
|
||||
GENARRATIVE_DATABASE_BACKUP_STORAGE_FORMAT=archive
|
||||
GENARRATIVE_DATABASE_BACKUP_MIN_FREE_BYTES=
|
||||
GENARRATIVE_DATABASE_BACKUP_BASELINE_STATE=/var/lib/genarrative/database-backups/genarrative-prod-history-state.json
|
||||
# 仅 archive history 首次从一份 uploadStatus=uploaded 的全量 manifest 初始化 state 时设置或传 --baseline-manifest。
|
||||
GENARRATIVE_DATABASE_BACKUP_BASELINE_MANIFEST=
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_ID=
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_SECRET=
|
||||
```
|
||||
|
||||
`GENARRATIVE_DATABASE_BACKUP_OSS_BUCKET` 为空时会回退 `ALIYUN_OSS_BUCKET`;AccessKey 默认复用 `ALIYUN_OSS_ACCESS_KEY_ID` / `ALIYUN_OSS_ACCESS_KEY_SECRET`,也可用 `GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_ID` / `GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_SECRET` 为备份 bucket 单独配置最小权限账号。冷备脚本会在停止 SpacetimeDB 前检查 `GENARRATIVE_DATABASE_BACKUP_WORK_DIR` 所在文件系统剩余空间;未设置 `GENARRATIVE_DATABASE_BACKUP_MIN_FREE_BYTES` 时,按数据目录大小加安全余量估算,空间不足会在停库前失败,避免写满根分区。即使打包或上传前步骤失败,只要脚本已经停过 SpacetimeDB,也会先恢复 SpacetimeDB 并执行 `--restart-service-after` 指定的 API / worker / controller,再带着原始备份错误退出。`Genarrative-Server-Provision` 会创建 `/var/lib/genarrative/database-backups` 并归属 `genarrative:genarrative`,同时安装并启用 `genarrative-database-backup.timer`。手动检查定时器:`systemctl list-timers genarrative-database-backup.timer`;手动触发一次:`systemctl start genarrative-database-backup.service`。如果 timer 显示 `enabled` 但 `inactive/dead` 且 `NEXT` / `Trigger` 为空,先写入当前 stamp 避免 `Persistent=true` 在白天立刻补跑冷备份:`touch /var/lib/systemd/timers/stamp-genarrative-database-backup.timer && systemctl daemon-reload && systemctl start genarrative-database-backup.timer`,随后确认下一次触发时间约为次日 `03:20`。
|
||||
|
||||
`files-history` 使用仓库模板 `deploy/systemd/genarrative-database-backup-files-history.conf` 覆盖主 service 的 `ExecStart`,从 `/etc/genarrative/api-server.env` 读取 data-dir、database、bucket、prefix 与 OSS 凭据,不在 unit 写死环境目标,也不传 `--stop-service`。Server-Provision 在改动 drop-in 前,先用 current release 的同一脚本、同一 env 和 `DATABASE_BACKUP_FILES_HISTORY_WORK_DIR` 执行一次 history `--dry-run`;缺少已发布 full catalog 的 files state、current 脚本过旧或配置不匹配都会在安装 drop-in 和 `daemon-reload` 前失败。选择 `archive-full` 会主动删除仓库托管的 `10-files-history.conf` 与 dev 试点遗留的 `10-dev-files.conf`,防止 systemd 继续合并旧覆盖。dev 可继续指定已有 `/var/lib/genarrative/database-backups/dev-files`,release 建议先在 `/var/lib/genarrative/database-backups/release-files` 建立自己的 full baseline;两台机器不得复用或互传本地 state 目录冒充本机基线。启用时通过 Server-Provision Job 选择目标、`DATABASE_BACKUP_PROFILE=files-history` 和对应 work-dir,先保持 `DRY_RUN=true` 核对,再以同参数正式 provision。不要直接在 `/etc/systemd/system` 手写第二份 drop-in。
|
||||
|
||||
files full 会递归扫描 data-dir,保留空目录和每个普通文件的相对路径;文件按 SHA-256 上传到不可变对象 key,catalog 记录目录、路径、长度、SHA 和对象 key,不写 staging 主机的绝对路径。相同 catalog 重跑不重复 PUT;新增或变化文件先 HEAD CAS 对象,存在且长度/SHA 元数据一致就复用,否则上传。full 基线必须来自停库后的 data-dir 或已通过恢复验证的冻结副本;源文件上传前后 stat 虽会复核,但在线扫描不能保证 2083 个文件属于同一跨文件一致时点。catalog 验真后,脚本把最新 full/history 引用发布到固定 `<prefix>/<database>/latest.json`,全新机器不需要本地 state 即可自动发现恢复入口。
|
||||
|
||||
history 的安全边界按每个 replica 独立计算。设最新完整且未锁定的 snapshot offset 为 `S`;数字更大但缺少同 offset `.snapshot_bsatn`、仍存在同名 `.lock` 的目录不能参与边界计算。脚本必须保留起始 offset 小于等于 `S` 的最后一个 commitlog segment,以及它之后的全部 segment;只处理更早的 `.stdb.log` / `.stdb.ofs`,snapshot 只处理最新目录之前的旧目录。files history 会递归展开候选目录,逐对象复用或上传,随后依次验真候选对象、history catalog 与 full baseline catalog,再重新扫描边界和 stat fingerprint,最后覆盖发布并验真 `latest.json`;任何一步失败都不推进 state 或删除源文件。脚本在 work-dir 使用 PID lock 拒绝同库并发上传,SSH 超时后必须先检查原进程,不能直接重跑。
|
||||
|
||||
```bash
|
||||
# 从停库目录或已验证冻结副本建立逐文件完整基线;相同 work-dir 重跑只传变化内容。
|
||||
node -- scripts/database-backup-to-oss.mjs \
|
||||
--storage-format files \
|
||||
--mode full \
|
||||
--data-dir /path/to/frozen/stdb \
|
||||
--work-dir /var/lib/genarrative/database-backups/dev-files \
|
||||
--database genarrative-prod \
|
||||
--env-file /etc/genarrative/api-server.env
|
||||
|
||||
# 把完整 work-dir/state 放回 dev 后,先只读查看可清理历史候选。
|
||||
node -- scripts/database-backup-to-oss.mjs \
|
||||
--storage-format files \
|
||||
--mode history \
|
||||
--data-dir /stdb \
|
||||
--work-dir /var/lib/genarrative/database-backups/dev-files \
|
||||
--database genarrative-prod \
|
||||
--env-file /etc/genarrative/api-server.env \
|
||||
--dry-run \
|
||||
--result-file /var/lib/genarrative/database-backups/history-dry-run.json
|
||||
|
||||
# 核对 dry-run 后执行真实归档与清理;history 在线处理不可变历史文件,不传 --stop-service。
|
||||
node -- scripts/database-backup-to-oss.mjs \
|
||||
--storage-format files \
|
||||
--mode history \
|
||||
--data-dir /stdb \
|
||||
--work-dir /var/lib/genarrative/database-backups/dev-files \
|
||||
--database genarrative-prod \
|
||||
--env-file /etc/genarrative/api-server.env
|
||||
```
|
||||
|
||||
dev 出口过慢时,可以把冻结基线经内网 rsync 到 release 独立 staging,再由 release 上传 dev bucket。staging 必须位于 `/var/lib/genarrative/dev-database-backup-staging/` 一类隔离目录,命令显式传 staging `--data-dir`、独立 `--work-dir`、dev `--bucket`,且不得传 `--stop-service`;禁止指向或修改 release `/stdb`。中转 key 只为本次传输临时授权,结束后从 dev 私钥和 release `authorized_keys` 同时移除。上传完成后把整个 files work-dir/state 回传 dev,history 才能延续同一 baseline catalog。
|
||||
|
||||
完整恢复默认从 OSS 固定 `latest.json` 读取最新 full catalog:先创建 `directories`,再把每个 `files[].objectKey` 下载到 `<restore-root>/<files[].path>` 并逐项核对 `sizeBytes` / `sha256`;history catalog 用于证明已清理历史仍有 OSS 对象,不需要把已被 full baseline 覆盖的旧文件叠回当前恢复目录。本地 state 仍可作为兼容入口,但不再是异机恢复的前置条件。随后用隔离 data-dir 启动同版本 standalone,验证 `/v1/ping`、日志中的 snapshot restore / commitlog replay / module launch、代表性 SQL 和 reducer。dev 已完成这轮 OSS-only 异机恢复与重启演练;当前 live release 仍保持 `archive-full`,需要切换时先为 release 建立并恢复验证独立 full baseline,再通过 Server-Provision 显式选择 `files-history`,无需修改代码或解除额外硬门禁。
|
||||
|
||||
```bash
|
||||
node -- scripts/database-backup-to-oss.mjs \
|
||||
--env-file /etc/genarrative/api-server.env \
|
||||
--database genarrative-prod \
|
||||
--restore-files-latest \
|
||||
--restore-dir /var/lib/genarrative/database-backup-restore/stdb \
|
||||
--result-file /var/lib/genarrative/database-backup-restore/restore-result.json
|
||||
```
|
||||
|
||||
冷备份后必须做一次只读验收,不要只看 `genarrative-database-backup.service` 是否成功退出:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -33,6 +33,8 @@ pipeline {
|
||||
string(name: 'WEB_LINK', defaultValue: '/srv/genarrative/web', description: 'Nginx 静态站点目录或软链接')
|
||||
string(name: 'API_ENV_FILE', defaultValue: '/etc/genarrative/api-server.env', description: 'api-server 环境文件')
|
||||
string(name: 'API_PORT', defaultValue: '8082', description: 'api-server 本机监听端口')
|
||||
choice(name: 'DATABASE_BACKUP_PROFILE', choices: ['archive-full', 'files-history'], description: '数据库定时备份 profile;默认 archive-full,files-history 仅在指定 work-dir 已有完整 full baseline 后启用')
|
||||
string(name: 'DATABASE_BACKUP_FILES_HISTORY_WORK_DIR', defaultValue: '/var/lib/genarrative/database-backups/files-history', description: 'files-history 的本地 state/catalog 目录;dev/release 必须使用各自已建立 full baseline 的独立目录')
|
||||
choice(name: 'NGINX_CONFIG_MODE', choices: ['none', 'production-https', 'development-http'], description: 'Nginx 配置模式;开发服无域名时选 development-http,release 正式入口选 production-https')
|
||||
booleanParam(name: 'ENABLE_SERVICES', defaultValue: true, description: '启用并启动 spacetimedb 与 api-server systemd 服务')
|
||||
booleanParam(name: 'ENABLE_OTELCOL', defaultValue: true, description: '安装并启用本机 OpenTelemetry Collector;api-server 模板默认开启 OTLP,如需关闭请在 API_ENV_FILE 中将 GENARRATIVE_OTEL_ENABLED 改为 false')
|
||||
@@ -103,6 +105,14 @@ pipeline {
|
||||
if (params.DEPLOY_TARGET == 'release' && nginxMode == 'development-http') {
|
||||
error('release 目标禁止安装 development-http Nginx 配置;无证书初始化请使用 NGINX_CONFIG_MODE=none。')
|
||||
}
|
||||
def databaseBackupProfile = params.DATABASE_BACKUP_PROFILE?.trim()
|
||||
if (!(databaseBackupProfile in ['archive-full', 'files-history'])) {
|
||||
error("DATABASE_BACKUP_PROFILE 只能是 archive-full 或 files-history,当前值: ${params.DATABASE_BACKUP_PROFILE}")
|
||||
}
|
||||
def databaseBackupFilesHistoryWorkDir = params.DATABASE_BACKUP_FILES_HISTORY_WORK_DIR?.trim()
|
||||
if (!(databaseBackupFilesHistoryWorkDir ==~ /^\/var\/lib\/genarrative\/database-backups\/[A-Za-z0-9._\/-]+$/) || databaseBackupFilesHistoryWorkDir.contains('..')) {
|
||||
error("DATABASE_BACKUP_FILES_HISTORY_WORK_DIR 必须是 /var/lib/genarrative/database-backups/ 下不含连续点号的绝对路径,当前值: ${params.DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}")
|
||||
}
|
||||
if (!params.DRY_RUN && nginxMode == 'production-https' && params.SERVER_NAME?.trim() == 'genarrative.example.com') {
|
||||
error('真实初始化安装 Nginx 配置时必须把 SERVER_NAME 改成真实域名,不能使用 genarrative.example.com 占位值。证书未准备好时请先保持 NGINX_CONFIG_MODE=none。')
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -514,6 +514,76 @@ const checks = [
|
||||
reason:
|
||||
'生产冷备份 service 必须用 node -- 分隔脚本参数,避免 Node 22 抢占业务 --env-file。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-database-backup.service',
|
||||
excludes: '--storage-format files',
|
||||
reason: '生产数据库备份主 service 必须继续保持 archive full 默认行为。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-database-backup-files-history.conf',
|
||||
includes: 'ExecStart=\nExecStart=/usr/bin/node -- /opt/genarrative/current/scripts/database-backup-to-oss.mjs --env-file',
|
||||
reason: 'files-history drop-in 必须先清空主 service 的 ExecStart,并使用 current release 脚本。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-database-backup-files-history.conf',
|
||||
includes: '--storage-format files --mode history --work-dir /var/lib/genarrative/database-backups/files-history',
|
||||
reason: 'files-history drop-in 必须只执行逐文件历史归档,并复用已建立基线的独立 work-dir。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-database-backup-files-history.conf',
|
||||
excludes: '--stop-service',
|
||||
reason: 'files-history 在线归档不可停止 SpacetimeDB。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-database-backup-files-history.conf',
|
||||
excludes: '--database',
|
||||
reason: 'files-history drop-in 不得写死数据库名,必须从环境文件读取。',
|
||||
},
|
||||
{
|
||||
file: 'deploy/systemd/genarrative-database-backup-files-history.conf',
|
||||
excludes: '--bucket',
|
||||
reason: 'files-history drop-in 不得写死 OSS bucket,必须从环境文件读取。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'DATABASE_BACKUP_PROFILE="${DATABASE_BACKUP_PROFILE:-archive-full}"',
|
||||
reason: 'Server-Provision 的数据库备份 profile 必须默认保持 archive-full。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'archive-full|files-history)',
|
||||
reason: 'Server-Provision 必须拒绝未知数据库备份 profile。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: '--storage-format files --mode history --work-dir',
|
||||
reason: 'Server-Provision 启用 files-history 前必须用真实 current release 脚本执行只读 baseline 预检。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'run_cmd rm -f "${DATABASE_BACKUP_FILES_HISTORY_DROP_IN}" "${DATABASE_BACKUP_LEGACY_DEV_DROP_IN}"',
|
||||
reason: 'Server-Provision 切回 archive-full 时必须移除托管及现场遗留的 history drop-in。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/jenkins-server-provision.sh',
|
||||
includes: 'install_file "${rendered_drop_in}" "${DATABASE_BACKUP_FILES_HISTORY_DROP_IN}" 0644',
|
||||
reason: 'Server-Provision 必须从仓库模板安装托管的 files-history drop-in。',
|
||||
},
|
||||
{
|
||||
file: 'jenkins/Jenkinsfile.production-server-provision',
|
||||
includes: "choice(name: 'DATABASE_BACKUP_PROFILE', choices: ['archive-full', 'files-history']",
|
||||
reason: 'Server-Provision Job 必须显式暴露 archive-first 的数据库备份 profile。',
|
||||
},
|
||||
{
|
||||
file: 'jenkins/Jenkinsfile.production-server-provision',
|
||||
includes: "string(name: 'DATABASE_BACKUP_FILES_HISTORY_WORK_DIR', defaultValue: '/var/lib/genarrative/database-backups/files-history'",
|
||||
reason: 'Server-Provision Job 必须允许 dev/release 为 files-history 选择各自的 baseline state 目录。',
|
||||
},
|
||||
{
|
||||
file: 'jenkins/Jenkinsfile.production-server-provision',
|
||||
excludes: "params.DEPLOY_TARGET == 'release' && databaseBackupProfile == 'files-history'",
|
||||
reason: 'release 必须能在显式选择 profile 且 baseline 预检通过后启用 files-history。',
|
||||
},
|
||||
{
|
||||
file: 'scripts/database-backup-to-oss.mjs',
|
||||
includes: 'assertSufficientWorkDirSpace({dataDir, workDir, args, env})',
|
||||
@@ -6837,6 +6907,7 @@ const checks = [
|
||||
const nodeEnvFileCommandFiles = [
|
||||
'package.json',
|
||||
'deploy/systemd/genarrative-database-backup.service',
|
||||
'deploy/systemd/genarrative-database-backup-files-history.conf',
|
||||
'scripts/deploy/production-stdb-publish.sh',
|
||||
'scripts/deploy/pingora-direct-enable.sh',
|
||||
'scripts/deploy/pingora-direct-rollback.sh',
|
||||
|
||||
+1926
-66
File diff suppressed because it is too large
Load Diff
@@ -10,6 +10,11 @@ GENARRATIVE_OPENSSL_VERSION="${GENARRATIVE_OPENSSL_VERSION:-3.2.0}"
|
||||
GENARRATIVE_OPENSSL_PREFIX="${GENARRATIVE_OPENSSL_PREFIX:-/opt/genarrative/openssl-3.2.0}"
|
||||
GENARRATIVE_OPENSSL_SOURCE_URL="${GENARRATIVE_OPENSSL_SOURCE_URL:-https://github.com/openssl/openssl/releases/download/openssl-${GENARRATIVE_OPENSSL_VERSION}/openssl-${GENARRATIVE_OPENSSL_VERSION}.tar.gz}"
|
||||
GENARRATIVE_OPENSSL_SOURCE_SHA256="${GENARRATIVE_OPENSSL_SOURCE_SHA256:-14c826f07c7e433706fb5c69fa9e25dab95684844b4c962a2cf1bf183eb4690e}"
|
||||
DATABASE_BACKUP_PROFILE="${DATABASE_BACKUP_PROFILE:-archive-full}"
|
||||
DATABASE_BACKUP_FILES_HISTORY_WORK_DIR="${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR:-/var/lib/genarrative/database-backups/files-history}"
|
||||
DATABASE_BACKUP_FILES_HISTORY_DROP_IN_DIR="/etc/systemd/system/genarrative-database-backup.service.d"
|
||||
DATABASE_BACKUP_FILES_HISTORY_DROP_IN="${DATABASE_BACKUP_FILES_HISTORY_DROP_IN_DIR}/10-files-history.conf"
|
||||
DATABASE_BACKUP_LEGACY_DEV_DROP_IN="${DATABASE_BACKUP_FILES_HISTORY_DROP_IN_DIR}/10-dev-files.conf"
|
||||
|
||||
require_non_root_relative_path() {
|
||||
local label="$1"
|
||||
@@ -63,6 +68,21 @@ validate_server_names() {
|
||||
done
|
||||
}
|
||||
|
||||
validate_database_backup_profile() {
|
||||
case "${DATABASE_BACKUP_PROFILE}" in
|
||||
archive-full|files-history)
|
||||
;;
|
||||
*)
|
||||
echo "[server-provision] DATABASE_BACKUP_PROFILE 只能是 archive-full 或 files-history,当前值: ${DATABASE_BACKUP_PROFILE}" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
if [[ ! "${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}" =~ ^/var/lib/genarrative/database-backups/[A-Za-z0-9._/-]+$ || "${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}" == *..* ]]; then
|
||||
echo "[server-provision] DATABASE_BACKUP_FILES_HISTORY_WORK_DIR 必须是 /var/lib/genarrative/database-backups/ 下不含连续点号的绝对路径,当前值: ${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_cmd() {
|
||||
echo "+ $*"
|
||||
if [[ "${DRY_RUN}" != "true" ]]; then
|
||||
@@ -917,6 +937,52 @@ render_database_backup_service() {
|
||||
deploy/systemd/genarrative-database-backup.service
|
||||
}
|
||||
|
||||
render_database_backup_files_history_drop_in() {
|
||||
local current_escaped env_escaped work_dir_escaped
|
||||
current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
|
||||
env_escaped="$(escape_sed_replacement "${API_ENV_FILE}")"
|
||||
work_dir_escaped="$(escape_sed_replacement "${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}")"
|
||||
sed \
|
||||
-e "s|/opt/genarrative/current|${current_escaped}|g" \
|
||||
-e "s|/etc/genarrative/api-server.env|${env_escaped}|g" \
|
||||
-e "s|/var/lib/genarrative/database-backups/files-history|${work_dir_escaped}|g" \
|
||||
deploy/systemd/genarrative-database-backup-files-history.conf
|
||||
}
|
||||
|
||||
configure_database_backup_profile() {
|
||||
local rendered_drop_in
|
||||
|
||||
if [[ "${DATABASE_BACKUP_PROFILE}" == "archive-full" ]]; then
|
||||
echo "[server-provision] 数据库备份 profile=archive-full,保留主 service 的全量冷备行为。"
|
||||
run_cmd rm -f "${DATABASE_BACKUP_FILES_HISTORY_DROP_IN}" "${DATABASE_BACKUP_LEGACY_DEV_DROP_IN}"
|
||||
return
|
||||
fi
|
||||
|
||||
echo "[server-provision] 数据库备份 profile=files-history,先只读验证 full baseline state。"
|
||||
if [[ "${DRY_RUN}" == "true" ]]; then
|
||||
echo "+ /usr/bin/node -- ${CURRENT_LINK}/scripts/database-backup-to-oss.mjs --env-file ${API_ENV_FILE} --storage-format files --mode history --work-dir ${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR} --dry-run"
|
||||
else
|
||||
if [[ ! -f "${CURRENT_LINK}/scripts/database-backup-to-oss.mjs" ]]; then
|
||||
echo "[server-provision] current release 缺少数据库备份脚本: ${CURRENT_LINK}/scripts/database-backup-to-oss.mjs" >&2
|
||||
exit 1
|
||||
fi
|
||||
/usr/bin/node -- "${CURRENT_LINK}/scripts/database-backup-to-oss.mjs" \
|
||||
--env-file "${API_ENV_FILE}" \
|
||||
--storage-format files \
|
||||
--mode history \
|
||||
--work-dir "${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}" \
|
||||
--dry-run
|
||||
fi
|
||||
|
||||
run_cmd install -d -o genarrative -g genarrative -m 0750 "${DATABASE_BACKUP_FILES_HISTORY_WORK_DIR}"
|
||||
run_cmd install -d -o root -g root -m 0755 "${DATABASE_BACKUP_FILES_HISTORY_DROP_IN_DIR}"
|
||||
run_cmd rm -f "${DATABASE_BACKUP_LEGACY_DEV_DROP_IN}"
|
||||
rendered_drop_in="$(mktemp)"
|
||||
render_database_backup_files_history_drop_in >"${rendered_drop_in}"
|
||||
install_file "${rendered_drop_in}" "${DATABASE_BACKUP_FILES_HISTORY_DROP_IN}" 0644
|
||||
rm -f "${rendered_drop_in}"
|
||||
}
|
||||
|
||||
render_health_patrol_service() {
|
||||
local current_escaped
|
||||
current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
|
||||
@@ -930,6 +996,7 @@ require_path deploy/systemd/genarrative-api.service
|
||||
require_path deploy/systemd/genarrative-external-generation-worker@.service
|
||||
require_path deploy/systemd/genarrative-external-generation-controller.service
|
||||
require_path deploy/systemd/genarrative-database-backup.service
|
||||
require_path deploy/systemd/genarrative-database-backup-files-history.conf
|
||||
require_path deploy/systemd/genarrative-database-backup.timer
|
||||
require_path deploy/systemd/genarrative-health-patrol.service
|
||||
require_path deploy/systemd/genarrative-health-patrol.timer
|
||||
@@ -951,9 +1018,10 @@ require_path scripts/deploy/maintenance-off.sh
|
||||
require_path scripts/deploy/maintenance-status.sh
|
||||
|
||||
validate_server_names
|
||||
validate_database_backup_profile
|
||||
require_non_root_relative_path "PROVISION_TOOLS_DIR" "${PROVISION_TOOLS_DIR}"
|
||||
|
||||
echo "[server-provision] target=${DEPLOY_TARGET}, dry_run=${DRY_RUN}, nginx_config_mode=${NGINX_CONFIG_MODE}, source_commit=$(cat .jenkins-source-commit)"
|
||||
echo "[server-provision] target=${DEPLOY_TARGET}, dry_run=${DRY_RUN}, nginx_config_mode=${NGINX_CONFIG_MODE}, database_backup_profile=${DATABASE_BACKUP_PROFILE}, source_commit=$(cat .jenkins-source-commit)"
|
||||
|
||||
run_cmd id
|
||||
require_root_for_real_provision
|
||||
@@ -1033,6 +1101,7 @@ else
|
||||
echo "[server-provision] 已存在环境文件,保留不覆盖: ${API_ENV_FILE}"
|
||||
fi
|
||||
ensure_api_runtime_env_defaults
|
||||
configure_database_backup_profile
|
||||
|
||||
if [[ ! -f "${WORKER_ENV_FILE}" ]]; then
|
||||
echo "+ create ${WORKER_ENV_FILE} from example"
|
||||
|
||||
Reference in New Issue
Block a user