From 0386a5df7ca45502cda56b7aa3a992131dd96b86 Mon Sep 17 00:00:00 2001 From: kdletters Date: Fri, 31 Jul 2026 12:08:45 +0800 Subject: [PATCH] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E6=95=B0=E6=8D=AE=E5=BA=93?= =?UTF-8?q?=E5=A4=87=E4=BB=BD=E5=BC=82=E6=AD=A5=E4=B8=8A=E4=BC=A0=E6=94=B6?= =?UTF-8?q?=E5=B0=BE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 使用独立systemd服务承接Jenkins退出后的OSS上传 串行补偿延期归档并仅在完整验真后清理 补充备份回归门禁与生产运维约定 --- .../shared-memory/decision-log.md | 11 ++ docs/project-memory/shared-memory/pitfalls.md | 8 ++ ...发运维】本地开发验证与生产运维-2026-05-15.md | 2 +- scripts/check-database-backup-to-oss.mjs | 83 +++++++++++++++ scripts/check-production-ops-guardrails.mjs | 66 ++++++++++++ scripts/database-backup-to-oss.mjs | 100 ++++++++++++++++++ scripts/deploy/production-stdb-publish.sh | 65 ++++++++++-- 7 files changed, 325 insertions(+), 10 deletions(-) diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index bab05cac6..0006d8813 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -16,6 +16,17 @@ --- +## 2026-07-31 发布前延期冷备份由独立 systemd 上传并补偿扫描 + +- 背景：Jenkins Stdb Publish 的 async 备份先生成 `uploadStatus=deferred` 的本地 tar.gz，再从 EXIT trap 用 `nohup` 启动上传。后台进程仍继承 Jenkins Cookie，作业结束时可被清理；旧 deferred manifest 也没有后续补偿扫描，导致 dev 的本地冷备份持续占满根盘。 +- 决策：`production-stdb-publish.sh` 只能用具名、`Type=exec`、`--collect` 的 `systemd-run` transient service 启动异步上传，禁止回退 `nohup`。独立服务执行 `database-backup-to-oss.mjs --upload-deferred-dir `，在同一备份锁内按文件名串行补传同库 `deferred/pending` 归档；目录外路径或 manifest/归档不匹配时失败关闭，缺失归档的历史 manifest 只报告不删除。 +- 清理边界：只有 archive 上传与 HEAD 验真、manifest sidecar 上传验真、baseline state 写入全部成功后，才按 `GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL` 删除精确的 archive 与 manifest。transient unit 未启动或上传失败时保留归档，由后续 publish 继续补偿；`files-history` timer 仍不负责清理这些 tar.gz。 +- 影响范围：`scripts/deploy/production-stdb-publish.sh`、`scripts/database-backup-to-oss.mjs`、生产运维门禁和本文档。 +- 验证方式：`npm run check:database-backup`、`npm run check:production-ops`、`npm run check:encoding`、`git diff --check`；dev 现场还必须确认 transient unit 不在 Jenkins session scope，旧 deferred 归档逐份变为 OSS 已验真对象后被删除，备份锁清空，核心服务与公开接口健康。 +- 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。 + +--- + ## 2026-07-29 图集切片必须受前置容量和有界 CPU 保护 - 背景：图标与 UI 图集的 alpha 连通域识别会在 async handler 上同步执行；原始连通域合并采用全量两两比较，`64` 个输出限制又晚于排序、裁剪和 PNG 编码。碎块或噪点图会放大 CPU 与内存成本，手动拆分、图标自动拆分和 UI 提取都受影响。另一方面，图标与 UI 的 Alpha 尺寸恢复、provider 原图回读或透明图解码失败此前只记日志，仍会把不可信透明图持久化并拆分。 diff --git a/docs/project-memory/shared-memory/pitfalls.md b/docs/project-memory/shared-memory/pitfalls.md index 47df14d11..1a1a7372f 100644 --- a/docs/project-memory/shared-memory/pitfalls.md +++ b/docs/project-memory/shared-memory/pitfalls.md @@ -14,6 +14,14 @@ - 关联：相关文件、文档、提交或 Issue ``` +## Jenkins 异步备份不能用 nohup 脱离作业 + +- 现象：Stdb Publish 成功，上传日志只留下“已获取进程锁 / 上传已有备份 / 目标对象”，没有成功或可捕获错误；本地 tar.gz 和 `uploadStatus=deferred` manifest 每次发布后继续增长。 +- 原因：`nohup` 只忽略终端 HUP，不会移除 Jenkins/Hudson 进程 Cookie；Job 收尾可清理后台 uploader。原链路只上传当次归档，旧 deferred manifest 没有扫描重试，而 `files-history` timer 只处理 `/stdb` 历史文件。 +- 处理：发布退出时用独立 `systemd-run --collect --service-type=exec` transient unit 执行 `--upload-deferred-dir`，串行处理同库 deferred/pending 归档。启动前拒绝符号链接和非绝对路径；unit 启动失败必须保留 status、archive 和 manifest。补偿扫描不删除上传未验真的文件，也不扫描目录外路径。 +- 验证：门禁必须禁止 `nohup`，要求命名 transient unit、`--collect`、`Type=exec` 与失败后保留 status；备份测试覆盖稳定顺序、同库过滤、已上传但未清理的归档收敛、归档缺失报告与路径逃逸拒绝。现场最终核对 backup lock、manifest、transient unit/result、根盘、SpacetimeDB/API/worker/controller/Nginx 和公开端点。 +- 关联：`scripts/deploy/production-stdb-publish.sh`、`scripts/database-backup-to-oss.mjs`、`scripts/check-production-ops-guardrails.mjs`、`scripts/check-database-backup-to-oss.mjs`。 + ## 图集切片上限必须早于合并、裁剪和编码 - 现象：透明图集含大量独立碎块或噪点时，接口长时间占用 async worker；最终即使报“超过 64 个切片”，此前仍已完成全量两两合并、裁剪和 PNG 编码。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index ba2539a15..513500932 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -374,7 +374,7 @@ UI 相关修改要重点验证： npm run database:backup:oss -- --data-dir /stdb --stop-service spacetimedb.service --restart-service-after genarrative-api.service --restart-service-after genarrative-external-generation-worker@1.service --restart-service-after genarrative-external-generation-controller.service ``` -脚本会将数据目录打包成 `tar.gz`，上传到 `oss://///-.tar.gz`。生产建议做冷备份：传入 `--stop-service spacetimedb.service`，脚本会在打包前停止服务、打包后恢复服务，再上传 OSS；因 `genarrative-api.service`、`genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service` 都依赖 `spacetimedb.service`，生产定时冷备份还必须传入对应的 `--restart-service-after`，确保备份后 API、保底 worker 和 controller 随数据库一起恢复。`2026-06-10` release 故障就是现场 unit 漏掉 API 重启参数，`03:20` 冷备份停止 SpacetimeDB 后 API 被依赖关系一并停止，备份脚本只恢复了 SpacetimeDB，API 直到人工重启前都不可用；`2026-06-24` release 又出现同类依赖停机后只恢复 API、未恢复外部生成 worker/controller，导致图片画布生成任务长期停留在队列中。后续现场变更、provision 模板和 Jenkins 归档都必须通过 `npm run check:production-ops` 防止回退。由于 OSS 上传可能受服务器带宽限制，`Genarrative-Stdb-Module-Publish` 默认使用 `DATABASE_BACKUP_MODE=async`：先在 publish 前用 `--defer-upload` 生成本地冷备份和 `.manifest.json`，随后继续执行 publish；发布脚本退出前会用后台 `node -- ... --upload-archive ` 上传同一份发布前备份，不等待上传完成。`Genarrative-Full-Build-And-Deploy` 必须显式暴露并透传同一个 `DATABASE_BACKUP_MODE`，不得静默使用下游 `async`；release 已有验真冷备且明确禁止再上传时，Full 必须选择 `skip`。发布脚本在校验 wasm 后、执行 `spacetime publish` 前会等待显式 `SPACETIME_SERVER_URL` 的 `/v1/ping` 就绪，默认最多等待 `60` 秒；如生产机器冷备份恢复 `spacetimedb.service` 较慢，可临时设置 `GENARRATIVE_STDB_PUBLISH_READY_TIMEOUT_SECONDS` 调整等待时间。需要强一致发布闸门时改用 `DATABASE_BACKUP_MODE=sync`（等价脚本参数 `--backup-mode sync`），备份会在 publish 前同步打包并上传，失败会阻断 publish；确认已有其他备份窗口时才使用 `DATABASE_BACKUP_MODE=skip`（兼容脚本参数 `--skip-backup`）。若业务不能接受停机窗口，应先规划 SpacetimeDB 原生快照或主备策略，不要直接在写入中的数据目录上做热拷贝并当作强一致备份。 +脚本会将数据目录打包成 `tar.gz`，上传到 `oss://///-.tar.gz`。生产建议做冷备份：传入 `--stop-service spacetimedb.service`，脚本会在打包前停止服务、打包后恢复服务，再上传 OSS；因 `genarrative-api.service`、`genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service` 都依赖 `spacetimedb.service`，生产定时冷备份还必须传入对应的 `--restart-service-after`，确保备份后 API、保底 worker 和 controller 随数据库一起恢复。`2026-06-10` release 故障就是现场 unit 漏掉 API 重启参数，`03:20` 冷备份停止 SpacetimeDB 后 API 被依赖关系一并停止，备份脚本只恢复了 SpacetimeDB，API 直到人工重启前都不可用；`2026-06-24` release 又出现同类依赖停机后只恢复 API、未恢复外部生成 worker/controller，导致图片画布生成任务长期停留在队列中。后续现场变更、provision 模板和 Jenkins 归档都必须通过 `npm run check:production-ops` 防止回退。由于 OSS 上传可能受服务器带宽限制，`Genarrative-Stdb-Module-Publish` 默认使用 `DATABASE_BACKUP_MODE=async`：先在 publish 前用 `--defer-upload` 生成本地冷备份和 `.manifest.json`，随后继续执行 publish；发布脚本退出前会用独立 `systemd-run` transient service 执行 `--upload-deferred-dir `，串行补传该目录内同库的 `deferred/pending` 归档，不依赖 Jenkins 作业进程树存活。任一归档只有在 OSS archive、manifest 和 baseline state 全部上传并验真后，才按 `keep-local` 规则删除；失败归档保留原 manifest，由下次 publish 重试。`Genarrative-Full-Build-And-Deploy` 必须显式暴露并透传同一个 `DATABASE_BACKUP_MODE`，不得静默使用下游 `async`；release 已有验真冷备且明确禁止再上传时，Full 必须选择 `skip`。发布脚本在校验 wasm 后、执行 `spacetime publish` 前会等待显式 `SPACETIME_SERVER_URL` 的 `/v1/ping` 就绪，默认最多等待 `60` 秒；如生产机器冷备份恢复 `spacetimedb.service` 较慢，可临时设置 `GENARRATIVE_STDB_PUBLISH_READY_TIMEOUT_SECONDS` 调整等待时间。需要强一致发布闸门时改用 `DATABASE_BACKUP_MODE=sync`（等价脚本参数 `--backup-mode sync`），备份会在 publish 前同步打包并上传，失败会阻断 publish；确认已有其他备份窗口时才使用 `DATABASE_BACKUP_MODE=skip`（兼容脚本参数 `--skip-backup`）。若业务不能接受停机窗口，应先规划 SpacetimeDB 原生快照或主备策略，不要直接在写入中的数据目录上做热拷贝并当作强一致备份。 生产环境变量模板在 `deploy/env/api-server.env.example`： diff --git a/scripts/check-database-backup-to-oss.mjs b/scripts/check-database-backup-to-oss.mjs index 9bae19c58..57bc09368 100644 --- a/scripts/check-database-backup-to-oss.mjs +++ b/scripts/check-database-backup-to-oss.mjs @@ -14,6 +14,7 @@ import { cleanupHistoryCandidates, collectDirectFileEntries, createUploadBandwidthLimiter, + discoverDeferredArchiveUploads, discoverHistoryPlan, restoreDirectFilesBackup, restoreDirectFilesLatest, @@ -46,6 +47,7 @@ if (failures.length > 0) { console.log('[check:database-backup] OK'); async function main() { + assertDeferredArchiveDiscoveryIsBoundedAndDeterministic(); assertCanonicalQueryAndAuthorizationIncludeMultipartParameters(); assertInsufficientSpaceStopsBeforeServiceChanges(); assertArchiveFailureStillRestoresDependentServices(); @@ -73,6 +75,87 @@ async function main() { await assertDirectFilesRestoreDownloadsCatalogAndObjects(); } +function assertDeferredArchiveDiscoveryIsBoundedAndDeterministic() { + const root = path.join(tmpRoot, 'deferred-archive-discovery'); + mkdirSync(root, {recursive: true}); + const createCandidate = ({name, status, database = 'test-db', withArchive = true}) => { + const archivePath = path.join(root, `${name}.tar.gz`); + const manifestPath = `${archivePath}.manifest.json`; + if (withArchive) { + writeFileSync(archivePath, name); + } + writeFileSync(manifestPath, `${JSON.stringify({ + backupKind: 'spacetimedb-data-dir', + database, + archivePath, + uploadStatus: status, + })}\n`); + return {archivePath, manifestPath}; + }; + const later = createCandidate({name: 'test-db-20260731T020000Z', status: 'pending'}); + const earlier = createCandidate({name: 'test-db-20260731T010000Z', status: 'deferred'}); + const uploaded = createCandidate({name: 'test-db-20260731T000000Z', status: 'uploaded'}); + createCandidate({name: 'other-db-20260731T000000Z', status: 'deferred', database: 'other-db'}); + const missing = createCandidate({name: 'test-db-20260730T230000Z', status: 'deferred', withArchive: false}); + + const result = discoverDeferredArchiveUploads({workDir: root, database: 'test-db'}); + assertEqual( + result.archives.map(({archivePath}) => archivePath).join(','), + [earlier.archivePath, later.archivePath].join(','), + 'deferred/pending 扫描必须只返回同库现存归档，并按文件名稳定排序。', + ); + assertEqual(result.missingArchives.length, 1, '缺失归档的 deferred 清单必须单独报告。'); + assertEqual(result.missingArchives[0].manifestPath, missing.manifestPath, '缺失归档报告必须保留精确 manifest。'); + const cleanupResult = discoverDeferredArchiveUploads({workDir: root, database: 'test-db', includeUploaded: true}); + assertEqual( + cleanupResult.archives.map(({archivePath}) => archivePath).join(','), + [uploaded.archivePath, earlier.archivePath, later.archivePath].join(','), + '未要求保留本地归档时，补偿扫描必须同时收敛上传后未清理的本地归档。', + ); + const cliDryRun = spawnSync(process.execPath, [ + BACKUP_SCRIPT, + '--upload-deferred-dir', root, + '--database', 'test-db', + '--bucket', 'test-bucket', + '--endpoint', 'oss-cn-shanghai.aliyuncs.com', + '--access-key-id', 'test-id', + '--access-key-secret', 'test-secret', + '--keep-local', + '--dry-run', + ], {encoding: 'utf8'}); + assertStatus(cliDryRun, 0, 'deferred 补偿扫描 dry-run 必须可通过统一 CLI 入口执行。'); + assertIncludes(cliDryRun.stdout, 'count=2', 'deferred 补偿扫描 CLI 必须报告待处理归档数量。'); + assertTrue(existsSync(earlier.archivePath) && existsSync(later.archivePath), 'dry-run 不得删除 deferred 本地归档。'); + + const unsafeRoot = path.join(tmpRoot, 'deferred-archive-unsafe'); + mkdirSync(unsafeRoot, {recursive: true}); + const escapedArchive = path.join(tmpRoot, 'outside.tar.gz'); + writeFileSync(escapedArchive, 'outside'); + writeFileSync( + path.join(unsafeRoot, 'test-db-unsafe.tar.gz.manifest.json'), + `${JSON.stringify({database: 'test-db', archivePath: escapedArchive, uploadStatus: 'deferred'})}\n`, + ); + assertThrows( + () => discoverDeferredArchiveUploads({workDir: unsafeRoot, database: 'test-db'}), + '路径与清单不匹配', + 'deferred 扫描必须拒绝目录外归档或 manifest 名不匹配。', + ); + + const symlinkRoot = path.join(tmpRoot, 'deferred-archive-symlink'); + mkdirSync(symlinkRoot, {recursive: true}); + const symlinkArchive = path.join(symlinkRoot, 'test-db-symlink.tar.gz'); + symlinkSync(escapedArchive, symlinkArchive); + writeFileSync( + `${symlinkArchive}.manifest.json`, + `${JSON.stringify({database: 'test-db', archivePath: symlinkArchive, uploadStatus: 'deferred'})}\n`, + ); + assertThrows( + () => discoverDeferredArchiveUploads({workDir: symlinkRoot, database: 'test-db'}), + '非符号链接的普通文件', + 'deferred 扫描必须拒绝符号链接归档。', + ); +} + function readGzipJson(filePath) { return JSON.parse(gunzipSync(readFileSync(filePath)).toString('utf8')); } diff --git a/scripts/check-production-ops-guardrails.mjs b/scripts/check-production-ops-guardrails.mjs index 00d92b4b3..9b4e2f706 100644 --- a/scripts/check-production-ops-guardrails.mjs +++ b/scripts/check-production-ops-guardrails.mjs @@ -206,6 +206,40 @@ const checks = [ includes: '按参数保持维护模式和旧运行时服务停止状态', reason: '受控维护发布成功后不得自动重启旧运行时或退出维护模式。', }, + { + file: 'scripts/deploy/production-stdb-publish.sh', + includes: 'systemd-run', + reason: + '生产 Stdb publish 的异步 OSS 上传必须交给 systemd transient service，避免 Jenkins 结束时清理上传进程。', + }, + { + file: 'scripts/deploy/production-stdb-publish.sh', + includes: '--unit="${unit_name}"', + reason: '生产 Stdb publish 必须为异步 OSS 上传创建独立、可追踪的 transient unit。', + }, + { + file: 'scripts/deploy/production-stdb-publish.sh', + includes: '--collect', + reason: '异步 OSS 上传的 transient unit 结束后必须允许 systemd 回收。', + }, + { + file: 'scripts/deploy/production-stdb-publish.sh', + includes: '--service-type=exec', + reason: + '异步 OSS 上传必须等待 systemd 确认上传进程 exec 成功，不能把启动失败误判为已接管。', + }, + { + file: 'scripts/deploy/production-stdb-publish.sh', + includes: '--upload-deferred-dir "${log_dir}"', + reason: + '独立 OSS 上传服务必须补偿扫描历史 deferred/pending 归档，不能只处理当次归档。', + }, + { + file: 'scripts/deploy/production-stdb-publish.sh', + excludes: 'nohup ', + reason: + '生产 Stdb publish 不得恢复会继承 Jenkins 进程生命周期的 nohup 异步上传。', + }, { file: 'scripts/deploy/production-api-deploy.sh', includes: 'ensure_runtime_bootstrap_secret_file_env', @@ -7148,6 +7182,38 @@ for (const check of checks) { } } +const stdbPublishContent = readFileSync( + 'scripts/deploy/production-stdb-publish.sh', + 'utf8', +); +const asyncBackupUploadStart = stdbPublishContent.indexOf( + 'start_async_backup_upload() {', +); +const asyncBackupUploadEnd = stdbPublishContent.indexOf( + '\nwait_for_spacetime_ready() {', + asyncBackupUploadStart, +); +const asyncBackupUploadFunction = + asyncBackupUploadStart >= 0 && asyncBackupUploadEnd > asyncBackupUploadStart + ? stdbPublishContent.slice(asyncBackupUploadStart, asyncBackupUploadEnd) + : ''; +const systemdRunFailureGuard = asyncBackupUploadFunction.match( + /if\s+!\s+(?:run_privileged\s+)?systemd-run\b[\s\S]*?\bthen\b(?:(?!\n\s*fi\b)[\s\S])*?\breturn\s+[1-9][0-9]*\b(?:(?!\n\s*fi\b)[\s\S])*?\n\s*fi\b/u, +); +const asyncBackupStatusClearOffset = asyncBackupUploadFunction.indexOf( + 'rm -f "${ASYNC_BACKUP_STATUS_FILE}"', +); +if ( + !systemdRunFailureGuard || + asyncBackupStatusClearOffset < + (systemdRunFailureGuard.index ?? 0) + systemdRunFailureGuard[0].length +) { + failed = true; + console.error( + '[check:production-ops] production-stdb-publish 的 systemd-run 启动失败分支必须先返回非零，并且只能在 transient unit 启动成功后清理异步备份 status 文件。', + ); +} + for (const file of jenkinsSourceCheckoutFiles) { const content = readFileSync(file, 'utf8'); if (!content.includes(jenkinsLoopbackGitRemote)) { diff --git a/scripts/database-backup-to-oss.mjs b/scripts/database-backup-to-oss.mjs index 95e6b54f9..d9a1ff9eb 100644 --- a/scripts/database-backup-to-oss.mjs +++ b/scripts/database-backup-to-oss.mjs @@ -64,6 +64,7 @@ function usage() { npm run database:backup:oss -- [--mode full|history] [--storage-format archive|files] [--data-dir ] [--work-dir ] [--bucket ] [--object-prefix ] [--keep-local] node -- scripts/database-backup-to-oss.mjs [--stop-service spacetimedb.service] [--restart-service-after genarrative-api.service] [--defer-upload] node -- scripts/database-backup-to-oss.mjs --upload-archive + node -- scripts/database-backup-to-oss.mjs --upload-deferred-dir node -- scripts/database-backup-to-oss.mjs --publish-manifest node -- scripts/database-backup-to-oss.mjs --restore-files-state --restore-dir node -- scripts/database-backup-to-oss.mjs --restore-files-latest --restore-dir [--dry-run] @@ -74,6 +75,7 @@ function usage() { --storage-format files 不打包：按原相对路径建立 catalog，文件内容以 SHA-256 不可变对象上传；重复运行只上传新增或变化内容。 archive history 必须有已验真的 full baseline state；files history 必须复用同一 work-dir 中已发布的 full catalog state。 --defer-upload 只生成本地冷备份和 manifest，不上传；后续用 --upload-archive 异步上传。 + --upload-deferred-dir 串行收敛目录内 deferred/pending 及已上传未清理归档；只有 OSS 上传与验真完成后才按 keep-local 规则删除。 默认读取 .env / .env.local / .env.secrets.local；生产服务可传 --env-file /etc/genarrative/api-server.env。 shell 环境变量优先级最高，不会被 env 文件覆盖。 @@ -157,6 +159,7 @@ function parseArgs(argv) { dryRun: false, deferUpload: false, uploadArchive: '', + uploadDeferredDir: '', manifestFile: '', objectKey: '', resultFile: '', @@ -237,6 +240,9 @@ function parseArgs(argv) { case '--upload-archive': options.uploadArchive = readValue(); break; + case '--upload-deferred-dir': + options.uploadDeferredDir = readValue(); + break; case '--manifest-file': options.manifestFile = readValue(); break; @@ -2811,6 +2817,50 @@ export async function uploadHistoryArchiveWithCleanup({ return {result, uploadedManifest, cleanup, state}; } +export function discoverDeferredArchiveUploads({workDir, database, includeUploaded = false}) { + const resolvedWorkDir = resolvePath(workDir); + if (!existsSync(resolvedWorkDir)) { + return {archives: [], missingArchives: []}; + } + + const archives = []; + const missingArchives = []; + const manifestSuffix = '.tar.gz.manifest.json'; + const expectedDatabase = String(database || '').trim(); + const entries = readdirSync(resolvedWorkDir, {withFileTypes: true}) + .filter((candidate) => candidate.isFile() && candidate.name.endsWith(manifestSuffix)) + .sort((left, right) => left.name.localeCompare(right.name, 'en')); + for (const entry of entries) { + const manifestPath = join(resolvedWorkDir, entry.name); + const manifest = readManifest(manifestPath); + const uploadStatus = String(manifest.uploadStatus || '').trim(); + if (!['deferred', 'pending'].includes(uploadStatus) && !(includeUploaded && uploadStatus === 'uploaded')) { + continue; + } + if (expectedDatabase && String(manifest.database || '').trim() !== expectedDatabase) { + continue; + } + if (!manifest.archivePath) { + throw new Error(`deferred 备份清单缺少 archivePath: ${manifestPath}`); + } + const archivePath = resolvePath(manifest.archivePath); + if (dirname(archivePath) !== resolvedWorkDir || manifestPath !== `${archivePath}.manifest.json`) { + throw new Error(`deferred 备份路径与清单不匹配: ${manifestPath}`); + } + const candidate = {archivePath, manifestPath, manifest}; + if (existsSync(archivePath)) { + const archiveStat = lstatSync(archivePath); + if (!archiveStat.isFile() || archiveStat.isSymbolicLink()) { + throw new Error(`deferred 备份归档必须是非符号链接的普通文件: ${archivePath}`); + } + archives.push(candidate); + } else { + missingArchives.push(candidate); + } + } + return {archives, missingArchives}; +} + async function uploadExistingArchive({args, env, bucket, endpoint, accessKeyId, accessKeySecret, objectPrefix, bandwidthLimiter}) { const archivePath = resolvePath(args.uploadArchive); if (!existsSync(archivePath)) { @@ -2897,6 +2947,37 @@ async function uploadExistingArchive({args, env, bucket, endpoint, accessKeyId, } } +async function uploadDeferredArchives({args, env, bucket, endpoint, accessKeyId, accessKeySecret, objectPrefix, database, bandwidthLimiter}) { + const workDir = resolvePath(args.uploadDeferredDir); + const keepLocal = args.keepLocal || String(env.GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL ?? '').trim().toLowerCase() === 'true'; + const {archives, missingArchives} = discoverDeferredArchiveUploads({ + workDir, + database, + includeUploaded: !keepLocal, + }); + for (const {manifestPath} of missingArchives) { + console.warn(`[database-backup] deferred 清单对应的本地归档不存在，跳过: ${manifestPath}`); + } + if (archives.length === 0) { + console.log(`[database-backup] 没有可补偿的本地归档: ${workDir}`); + return; + } + console.log(`[database-backup] 开始串行上传待补偿本地归档: count=${archives.length}`); + for (const {archivePath, manifestPath} of archives) { + await uploadExistingArchive({ + args: {...args, uploadArchive: archivePath, manifestFile: manifestPath}, + env, + bucket, + endpoint, + accessKeyId, + accessKeySecret, + objectPrefix, + bandwidthLimiter, + }); + } + console.log(`[database-backup] 待补偿本地归档上传完成: count=${archives.length}`); +} + async function publishExistingManifest({args, bucket, endpoint, accessKeyId, accessKeySecret, bandwidthLimiter}) { const manifestPath = resolvePath(args.publishManifest); const manifest = readManifest(manifestPath); @@ -3108,6 +3189,7 @@ async function main() { )); const workDir = resolvePath(firstNonEmpty( args.workDir, + args.uploadDeferredDir, env.GENARRATIVE_DATABASE_BACKUP_WORK_DIR, isProductionLike ? DEFAULT_PRODUCTION_WORK_DIR : DEFAULT_LOCAL_WORK_DIR, )); @@ -3171,6 +3253,9 @@ async function main() { if (args.restoreDir) { throw new Error('--restore-dir 只能与 --restore-files-state 或 --restore-files-latest 一起使用。'); } + if (args.uploadArchive && args.uploadDeferredDir) { + throw new Error('--upload-archive 与 --upload-deferred-dir 不能同时使用。'); + } if (!args.dryRun) { const lockPath = acquireBackupLock({workDir, database}); @@ -3196,6 +3281,21 @@ async function main() { return; } + if (args.uploadDeferredDir) { + await uploadDeferredArchives({ + args, + env, + bucket, + endpoint, + accessKeyId, + accessKeySecret, + objectPrefix, + database, + bandwidthLimiter: uploadBandwidthLimiter, + }); + return; + } + if (storageFormat === 'files') { if (args.deferUpload) { throw new Error('files 模式无需本地归档且不支持 --defer-upload；失败后使用同一 work-dir 重跑即可续传。'); diff --git a/scripts/deploy/production-stdb-publish.sh b/scripts/deploy/production-stdb-publish.sh index 96817420b..f34492476 100644 --- a/scripts/deploy/production-stdb-publish.sh +++ b/scripts/deploy/production-stdb-publish.sh @@ -532,6 +532,11 @@ prepare_async_backup() { } start_async_backup_upload() { + local log_dir="" + local node_binary="" + local unit_name="" + local unit_suffix="" + if [[ -z "${ASYNC_BACKUP_STATUS_FILE}" || ! -f "${ASYNC_BACKUP_STATUS_FILE}" ]]; then echo "[production-stdb-publish] 警告：未找到可上传的本地备份状态文件，跳过异步上传" >&2 return 0 @@ -543,16 +548,58 @@ start_async_backup_upload() { echo "[production-stdb-publish] 警告：备份状态文件缺少 archivePath 或 manifestPath，跳过异步上传" >&2 return 0 fi + if [[ "${ASYNC_BACKUP_ARCHIVE}" != /* || ! -f "${ASYNC_BACKUP_ARCHIVE}" || -L "${ASYNC_BACKUP_ARCHIVE}" ]]; then + echo "[production-stdb-publish] 警告：异步上传归档必须是现存、非符号链接的普通绝对路径文件，保留状态文件等待处理: ${ASYNC_BACKUP_ARCHIVE}" >&2 + return 1 + fi + if [[ "${ASYNC_BACKUP_MANIFEST}" != /* || ! -f "${ASYNC_BACKUP_MANIFEST}" || -L "${ASYNC_BACKUP_MANIFEST}" ]]; then + echo "[production-stdb-publish] 警告：异步上传 manifest 必须是现存、非符号链接的普通绝对路径文件，保留状态文件等待处理: ${ASYNC_BACKUP_MANIFEST}" >&2 + return 1 + fi + if ! command -v systemd-run >/dev/null 2>&1; then + echo "[production-stdb-publish] 警告：systemd-run 不可用，无法启动独立上传服务；保留状态文件等待处理" >&2 + return 1 + fi - mkdir -p "$(dirname "${ASYNC_BACKUP_ARCHIVE}")" - ASYNC_BACKUP_LOG="$(dirname "${ASYNC_BACKUP_ARCHIVE}")/${DATABASE}-upload.log" - echo "[production-stdb-publish] 后台上传本地备份到 OSS: ${ASYNC_BACKUP_ARCHIVE}" - nohup node -- "${ASYNC_BACKUP_SCRIPT}" \ - --env-file /etc/genarrative/api-server.env \ - --upload-archive "${ASYNC_BACKUP_ARCHIVE}" \ - --manifest-file "${ASYNC_BACKUP_MANIFEST}" \ - >"${ASYNC_BACKUP_LOG}" 2>&1 & - echo "[production-stdb-publish] OSS 后台上传日志: ${ASYNC_BACKUP_LOG}" + node_binary="$(command -v node || true)" + if [[ "${node_binary}" != /* || ! -x "${node_binary}" ]]; then + echo "[production-stdb-publish] 警告：未找到可供 systemd 服务执行的绝对 node 路径；保留状态文件等待处理" >&2 + return 1 + fi + + log_dir="$(dirname "${ASYNC_BACKUP_ARCHIVE}")" + unit_suffix="$(date -u +%Y%m%dT%H%M%S%N)-$$-${RANDOM}" + unit_name="genarrative-stdb-backup-upload-${unit_suffix}.service" + if ! ASYNC_BACKUP_LOG="$(mktemp "${log_dir}/${DATABASE}-upload-${unit_suffix}.XXXXXX.log")"; then + echo "[production-stdb-publish] 警告：无法创建独立 OSS 上传日志，保留状态文件和本地归档等待处理" >&2 + return 1 + fi + if ! chmod 0600 "${ASYNC_BACKUP_LOG}"; then + echo "[production-stdb-publish] 警告：无法收紧独立 OSS 上传日志权限，保留状态文件和本地归档等待处理: ${ASYNC_BACKUP_LOG}" >&2 + return 1 + fi + + echo "[production-stdb-publish] 通过独立 systemd 服务串行上传 deferred/pending 本地备份到 OSS: ${log_dir}" + if ! run_privileged systemd-run \ + --no-ask-password \ + --unit="${unit_name}" \ + --description="Genarrative SpacetimeDB backup upload ${DATABASE}" \ + --collect \ + --service-type=exec \ + --property="Restart=no" \ + --property="UMask=0077" \ + --property="StandardOutput=append:${ASYNC_BACKUP_LOG}" \ + --property="StandardError=append:${ASYNC_BACKUP_LOG}" \ + -- "${node_binary}" -- "${ASYNC_BACKUP_SCRIPT}" \ + --env-file /etc/genarrative/api-server.env \ + --database "${DATABASE}" \ + --upload-deferred-dir "${log_dir}"; then + echo "[production-stdb-publish] 警告：独立 OSS 上传服务启动失败，保留状态文件和本地归档等待处理；启动日志: ${ASYNC_BACKUP_LOG}" >&2 + return 1 + fi + + echo "[production-stdb-publish] OSS 上传服务已启动: ${unit_name}" + echo "[production-stdb-publish] OSS 上传日志: ${ASYNC_BACKUP_LOG}" rm -f "${ASYNC_BACKUP_STATUS_FILE}" ASYNC_BACKUP_STATUS_FILE="" }