修复数据库备份异步上传收尾

使用独立systemd服务承接Jenkins退出后的OSS上传
串行补偿延期归档并仅在完整验真后清理
补充备份回归门禁与生产运维约定

docs/project-memory/shared-memory/decision-log.md

@@ -16,6 +16,17 @@
 
 ---
 
+## 2026-07-31 发布前延期冷备份由独立 systemd 上传并补偿扫描
+
+- 背景：Jenkins Stdb Publish 的 async 备份先生成 `uploadStatus=deferred` 的本地 tar.gz，再从 EXIT trap 用 `nohup` 启动上传。后台进程仍继承 Jenkins Cookie，作业结束时可被清理；旧 deferred manifest 也没有后续补偿扫描，导致 dev 的本地冷备份持续占满根盘。
+- 决策：`production-stdb-publish.sh` 只能用具名、`Type=exec`、`--collect` 的 `systemd-run` transient service 启动异步上传，禁止回退 `nohup`。独立服务执行 `database-backup-to-oss.mjs --upload-deferred-dir <backup-dir>`，在同一备份锁内按文件名串行补传同库 `deferred/pending` 归档；目录外路径或 manifest/归档不匹配时失败关闭，缺失归档的历史 manifest 只报告不删除。
+- 清理边界：只有 archive 上传与 HEAD 验真、manifest sidecar 上传验真、baseline state 写入全部成功后，才按 `GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL` 删除精确的 archive 与 manifest。transient unit 未启动或上传失败时保留归档，由后续 publish 继续补偿；`files-history` timer 仍不负责清理这些 tar.gz。
+- 影响范围：`scripts/deploy/production-stdb-publish.sh`、`scripts/database-backup-to-oss.mjs`、生产运维门禁和本文档。
+- 验证方式：`npm run check:database-backup`、`npm run check:production-ops`、`npm run check:encoding`、`git diff --check`；dev 现场还必须确认 transient unit 不在 Jenkins session scope，旧 deferred 归档逐份变为 OSS 已验真对象后被删除，备份锁清空，核心服务与公开接口健康。
+- 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
+
+---
+
 ## 2026-07-29 图集切片必须受前置容量和有界 CPU 保护
 
 - 背景：图标与 UI 图集的 alpha 连通域识别会在 async handler 上同步执行；原始连通域合并采用全量两两比较，`64` 个输出限制又晚于排序、裁剪和 PNG 编码。碎块或噪点图会放大 CPU 与内存成本，手动拆分、图标自动拆分和 UI 提取都受影响。另一方面，图标与 UI 的 Alpha 尺寸恢复、provider 原图回读或透明图解码失败此前只记日志，仍会把不可信透明图持久化并拆分。

docs/project-memory/shared-memory/pitfalls.md

@@ -14,6 +14,14 @@
 - 关联：相关文件、文档、提交或 Issue
 ```
 
+## Jenkins 异步备份不能用 nohup 脱离作业
+
+- 现象：Stdb Publish 成功，上传日志只留下“已获取进程锁 / 上传已有备份 / 目标对象”，没有成功或可捕获错误；本地 tar.gz 和 `uploadStatus=deferred` manifest 每次发布后继续增长。
+- 原因：`nohup` 只忽略终端 HUP，不会移除 Jenkins/Hudson 进程 Cookie；Job 收尾可清理后台 uploader。原链路只上传当次归档，旧 deferred manifest 没有扫描重试，而 `files-history` timer 只处理 `/stdb` 历史文件。
+- 处理：发布退出时用独立 `systemd-run --collect --service-type=exec` transient unit 执行 `--upload-deferred-dir`，串行处理同库 deferred/pending 归档。启动前拒绝符号链接和非绝对路径；unit 启动失败必须保留 status、archive 和 manifest。补偿扫描不删除上传未验真的文件，也不扫描目录外路径。
+- 验证：门禁必须禁止 `nohup`，要求命名 transient unit、`--collect`、`Type=exec` 与失败后保留 status；备份测试覆盖稳定顺序、同库过滤、已上传但未清理的归档收敛、归档缺失报告与路径逃逸拒绝。现场最终核对 backup lock、manifest、transient unit/result、根盘、SpacetimeDB/API/worker/controller/Nginx 和公开端点。
+- 关联：`scripts/deploy/production-stdb-publish.sh`、`scripts/database-backup-to-oss.mjs`、`scripts/check-production-ops-guardrails.mjs`、`scripts/check-database-backup-to-oss.mjs`。
+
 ## 图集切片上限必须早于合并、裁剪和编码
 
 - 现象：透明图集含大量独立碎块或噪点时，接口长时间占用 async worker；最终即使报“超过 64 个切片”，此前仍已完成全量两两合并、裁剪和 PNG 编码。

docs/【开发运维】本地开发验证与生产运维-2026-05-15.md

@@ -374,7 +374,7 @@ UI 相关修改要重点验证：
 npm run database:backup:oss -- --data-dir /stdb --stop-service spacetimedb.service --restart-service-after genarrative-api.service --restart-service-after genarrative-external-generation-worker@1.service --restart-service-after genarrative-external-generation-controller.service
 ```
 
-脚本会将数据目录打包成 `tar.gz`，上传到 `oss://<bucket>/<prefix>/<database>/<database>-<UTC时间>.tar.gz`。生产建议做冷备份：传入 `--stop-service spacetimedb.service`，脚本会在打包前停止服务、打包后恢复服务，再上传 OSS；因 `genarrative-api.service`、`genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service` 都依赖 `spacetimedb.service`，生产定时冷备份还必须传入对应的 `--restart-service-after`，确保备份后 API、保底 worker 和 controller 随数据库一起恢复。`2026-06-10` release 故障就是现场 unit 漏掉 API 重启参数，`03:20` 冷备份停止 SpacetimeDB 后 API 被依赖关系一并停止，备份脚本只恢复了 SpacetimeDB，API 直到人工重启前都不可用；`2026-06-24` release 又出现同类依赖停机后只恢复 API、未恢复外部生成 worker/controller，导致图片画布生成任务长期停留在队列中。后续现场变更、provision 模板和 Jenkins 归档都必须通过 `npm run check:production-ops` 防止回退。由于 OSS 上传可能受服务器带宽限制，`Genarrative-Stdb-Module-Publish` 默认使用 `DATABASE_BACKUP_MODE=async`：先在 publish 前用 `--defer-upload` 生成本地冷备份和 `.manifest.json`，随后继续执行 publish；发布脚本退出前会用后台 `node -- ... --upload-archive <tar.gz>` 上传同一份发布前备份，不等待上传完成。`Genarrative-Full-Build-And-Deploy` 必须显式暴露并透传同一个 `DATABASE_BACKUP_MODE`，不得静默使用下游 `async`；release 已有验真冷备且明确禁止再上传时，Full 必须选择 `skip`。发布脚本在校验 wasm 后、执行 `spacetime publish` 前会等待显式 `SPACETIME_SERVER_URL` 的 `/v1/ping` 就绪，默认最多等待 `60` 秒；如生产机器冷备份恢复 `spacetimedb.service` 较慢，可临时设置 `GENARRATIVE_STDB_PUBLISH_READY_TIMEOUT_SECONDS` 调整等待时间。需要强一致发布闸门时改用 `DATABASE_BACKUP_MODE=sync`（等价脚本参数 `--backup-mode sync`），备份会在 publish 前同步打包并上传，失败会阻断 publish；确认已有其他备份窗口时才使用 `DATABASE_BACKUP_MODE=skip`（兼容脚本参数 `--skip-backup`）。若业务不能接受停机窗口，应先规划 SpacetimeDB 原生快照或主备策略，不要直接在写入中的数据目录上做热拷贝并当作强一致备份。
+脚本会将数据目录打包成 `tar.gz`，上传到 `oss://<bucket>/<prefix>/<database>/<database>-<UTC时间>.tar.gz`。生产建议做冷备份：传入 `--stop-service spacetimedb.service`，脚本会在打包前停止服务、打包后恢复服务，再上传 OSS；因 `genarrative-api.service`、`genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service` 都依赖 `spacetimedb.service`，生产定时冷备份还必须传入对应的 `--restart-service-after`，确保备份后 API、保底 worker 和 controller 随数据库一起恢复。`2026-06-10` release 故障就是现场 unit 漏掉 API 重启参数，`03:20` 冷备份停止 SpacetimeDB 后 API 被依赖关系一并停止，备份脚本只恢复了 SpacetimeDB，API 直到人工重启前都不可用；`2026-06-24` release 又出现同类依赖停机后只恢复 API、未恢复外部生成 worker/controller，导致图片画布生成任务长期停留在队列中。后续现场变更、provision 模板和 Jenkins 归档都必须通过 `npm run check:production-ops` 防止回退。由于 OSS 上传可能受服务器带宽限制，`Genarrative-Stdb-Module-Publish` 默认使用 `DATABASE_BACKUP_MODE=async`：先在 publish 前用 `--defer-upload` 生成本地冷备份和 `.manifest.json`，随后继续执行 publish；发布脚本退出前会用独立 `systemd-run` transient service 执行 `--upload-deferred-dir <backup-dir>`，串行补传该目录内同库的 `deferred/pending` 归档，不依赖 Jenkins 作业进程树存活。任一归档只有在 OSS archive、manifest 和 baseline state 全部上传并验真后，才按 `keep-local` 规则删除；失败归档保留原 manifest，由下次 publish 重试。`Genarrative-Full-Build-And-Deploy` 必须显式暴露并透传同一个 `DATABASE_BACKUP_MODE`，不得静默使用下游 `async`；release 已有验真冷备且明确禁止再上传时，Full 必须选择 `skip`。发布脚本在校验 wasm 后、执行 `spacetime publish` 前会等待显式 `SPACETIME_SERVER_URL` 的 `/v1/ping` 就绪，默认最多等待 `60` 秒；如生产机器冷备份恢复 `spacetimedb.service` 较慢，可临时设置 `GENARRATIVE_STDB_PUBLISH_READY_TIMEOUT_SECONDS` 调整等待时间。需要强一致发布闸门时改用 `DATABASE_BACKUP_MODE=sync`（等价脚本参数 `--backup-mode sync`），备份会在 publish 前同步打包并上传，失败会阻断 publish；确认已有其他备份窗口时才使用 `DATABASE_BACKUP_MODE=skip`（兼容脚本参数 `--skip-backup`）。若业务不能接受停机窗口，应先规划 SpacetimeDB 原生快照或主备策略，不要直接在写入中的数据目录上做热拷贝并当作强一致备份。
 
 生产环境变量模板在 `deploy/env/api-server.env.example`：
 

scripts/check-database-backup-to-oss.mjs

@@ -14,6 +14,7 @@ import {
   cleanupHistoryCandidates,
   collectDirectFileEntries,
   createUploadBandwidthLimiter,
+  discoverDeferredArchiveUploads,
   discoverHistoryPlan,
   restoreDirectFilesBackup,
   restoreDirectFilesLatest,
@@ -46,6 +47,7 @@ if (failures.length > 0) {
 console.log('[check:database-backup] OK');
 
 async function main() {
+  assertDeferredArchiveDiscoveryIsBoundedAndDeterministic();
   assertCanonicalQueryAndAuthorizationIncludeMultipartParameters();
   assertInsufficientSpaceStopsBeforeServiceChanges();
   assertArchiveFailureStillRestoresDependentServices();
@@ -73,6 +75,87 @@ async function main() {
   await assertDirectFilesRestoreDownloadsCatalogAndObjects();
 }
 
+function assertDeferredArchiveDiscoveryIsBoundedAndDeterministic() {
+  const root = path.join(tmpRoot, 'deferred-archive-discovery');
+  mkdirSync(root, {recursive: true});
+  const createCandidate = ({name, status, database = 'test-db', withArchive = true}) => {
+    const archivePath = path.join(root, `${name}.tar.gz`);
+    const manifestPath = `${archivePath}.manifest.json`;
+    if (withArchive) {
+      writeFileSync(archivePath, name);
+    }
+    writeFileSync(manifestPath, `${JSON.stringify({
+      backupKind: 'spacetimedb-data-dir',
+      database,
+      archivePath,
+      uploadStatus: status,
+    })}\n`);
+    return {archivePath, manifestPath};
+  };
+  const later = createCandidate({name: 'test-db-20260731T020000Z', status: 'pending'});
+  const earlier = createCandidate({name: 'test-db-20260731T010000Z', status: 'deferred'});
+  const uploaded = createCandidate({name: 'test-db-20260731T000000Z', status: 'uploaded'});
+  createCandidate({name: 'other-db-20260731T000000Z', status: 'deferred', database: 'other-db'});
+  const missing = createCandidate({name: 'test-db-20260730T230000Z', status: 'deferred', withArchive: false});
+
+  const result = discoverDeferredArchiveUploads({workDir: root, database: 'test-db'});
+  assertEqual(
+    result.archives.map(({archivePath}) => archivePath).join(','),
+    [earlier.archivePath, later.archivePath].join(','),
+    'deferred/pending 扫描必须只返回同库现存归档，并按文件名稳定排序。',
+  );
+  assertEqual(result.missingArchives.length, 1, '缺失归档的 deferred 清单必须单独报告。');
+  assertEqual(result.missingArchives[0].manifestPath, missing.manifestPath, '缺失归档报告必须保留精确 manifest。');
+  const cleanupResult = discoverDeferredArchiveUploads({workDir: root, database: 'test-db', includeUploaded: true});
+  assertEqual(
+    cleanupResult.archives.map(({archivePath}) => archivePath).join(','),
+    [uploaded.archivePath, earlier.archivePath, later.archivePath].join(','),
+    '未要求保留本地归档时，补偿扫描必须同时收敛上传后未清理的本地归档。',
+  );
+  const cliDryRun = spawnSync(process.execPath, [
+    BACKUP_SCRIPT,
+    '--upload-deferred-dir', root,
+    '--database', 'test-db',
+    '--bucket', 'test-bucket',
+    '--endpoint', 'oss-cn-shanghai.aliyuncs.com',
+    '--access-key-id', 'test-id',
+    '--access-key-secret', 'test-secret',
+    '--keep-local',
+    '--dry-run',
+  ], {encoding: 'utf8'});
+  assertStatus(cliDryRun, 0, 'deferred 补偿扫描 dry-run 必须可通过统一 CLI 入口执行。');
+  assertIncludes(cliDryRun.stdout, 'count=2', 'deferred 补偿扫描 CLI 必须报告待处理归档数量。');
+  assertTrue(existsSync(earlier.archivePath) && existsSync(later.archivePath), 'dry-run 不得删除 deferred 本地归档。');
+
+  const unsafeRoot = path.join(tmpRoot, 'deferred-archive-unsafe');
+  mkdirSync(unsafeRoot, {recursive: true});
+  const escapedArchive = path.join(tmpRoot, 'outside.tar.gz');
+  writeFileSync(escapedArchive, 'outside');
+  writeFileSync(
+    path.join(unsafeRoot, 'test-db-unsafe.tar.gz.manifest.json'),
+    `${JSON.stringify({database: 'test-db', archivePath: escapedArchive, uploadStatus: 'deferred'})}\n`,
+  );
+  assertThrows(
+    () => discoverDeferredArchiveUploads({workDir: unsafeRoot, database: 'test-db'}),
+    '路径与清单不匹配',
+    'deferred 扫描必须拒绝目录外归档或 manifest 名不匹配。',
+  );
+
+  const symlinkRoot = path.join(tmpRoot, 'deferred-archive-symlink');
+  mkdirSync(symlinkRoot, {recursive: true});
+  const symlinkArchive = path.join(symlinkRoot, 'test-db-symlink.tar.gz');
+  symlinkSync(escapedArchive, symlinkArchive);
+  writeFileSync(
+    `${symlinkArchive}.manifest.json`,
+    `${JSON.stringify({database: 'test-db', archivePath: symlinkArchive, uploadStatus: 'deferred'})}\n`,
+  );
+  assertThrows(
+    () => discoverDeferredArchiveUploads({workDir: symlinkRoot, database: 'test-db'}),
+    '非符号链接的普通文件',
+    'deferred 扫描必须拒绝符号链接归档。',
+  );
+}
+
 function readGzipJson(filePath) {
   return JSON.parse(gunzipSync(readFileSync(filePath)).toString('utf8'));
 }

scripts/check-production-ops-guardrails.mjs

@@ -206,6 +206,40 @@ const checks = [
     includes: '按参数保持维护模式和旧运行时服务停止状态',
     reason: '受控维护发布成功后不得自动重启旧运行时或退出维护模式。',
   },
+  {
+    file: 'scripts/deploy/production-stdb-publish.sh',
+    includes: 'systemd-run',
+    reason:
+      '生产 Stdb publish 的异步 OSS 上传必须交给 systemd transient service，避免 Jenkins 结束时清理上传进程。',
+  },
+  {
+    file: 'scripts/deploy/production-stdb-publish.sh',
+    includes: '--unit="${unit_name}"',
+    reason: '生产 Stdb publish 必须为异步 OSS 上传创建独立、可追踪的 transient unit。',
+  },
+  {
+    file: 'scripts/deploy/production-stdb-publish.sh',
+    includes: '--collect',
+    reason: '异步 OSS 上传的 transient unit 结束后必须允许 systemd 回收。',
+  },
+  {
+    file: 'scripts/deploy/production-stdb-publish.sh',
+    includes: '--service-type=exec',
+    reason:
+      '异步 OSS 上传必须等待 systemd 确认上传进程 exec 成功，不能把启动失败误判为已接管。',
+  },
+  {
+    file: 'scripts/deploy/production-stdb-publish.sh',
+    includes: '--upload-deferred-dir "${log_dir}"',
+    reason:
+      '独立 OSS 上传服务必须补偿扫描历史 deferred/pending 归档，不能只处理当次归档。',
+  },
+  {
+    file: 'scripts/deploy/production-stdb-publish.sh',
+    excludes: 'nohup ',
+    reason:
+      '生产 Stdb publish 不得恢复会继承 Jenkins 进程生命周期的 nohup 异步上传。',
+  },
   {
     file: 'scripts/deploy/production-api-deploy.sh',
     includes: 'ensure_runtime_bootstrap_secret_file_env',
@@ -7148,6 +7182,38 @@ for (const check of checks) {
   }
 }
 
+const stdbPublishContent = readFileSync(
+  'scripts/deploy/production-stdb-publish.sh',
+  'utf8',
+);
+const asyncBackupUploadStart = stdbPublishContent.indexOf(
+  'start_async_backup_upload() {',
+);
+const asyncBackupUploadEnd = stdbPublishContent.indexOf(
+  '\nwait_for_spacetime_ready() {',
+  asyncBackupUploadStart,
+);
+const asyncBackupUploadFunction =
+  asyncBackupUploadStart >= 0 && asyncBackupUploadEnd > asyncBackupUploadStart
+    ? stdbPublishContent.slice(asyncBackupUploadStart, asyncBackupUploadEnd)
+    : '';
+const systemdRunFailureGuard = asyncBackupUploadFunction.match(
+  /if\s+!\s+(?:run_privileged\s+)?systemd-run\b[\s\S]*?\bthen\b(?:(?!\n\s*fi\b)[\s\S])*?\breturn\s+[1-9][0-9]*\b(?:(?!\n\s*fi\b)[\s\S])*?\n\s*fi\b/u,
+);
+const asyncBackupStatusClearOffset = asyncBackupUploadFunction.indexOf(
+  'rm -f "${ASYNC_BACKUP_STATUS_FILE}"',
+);
+if (
+  !systemdRunFailureGuard ||
+  asyncBackupStatusClearOffset <
+    (systemdRunFailureGuard.index ?? 0) + systemdRunFailureGuard[0].length
+) {
+  failed = true;
+  console.error(
+    '[check:production-ops] production-stdb-publish 的 systemd-run 启动失败分支必须先返回非零，并且只能在 transient unit 启动成功后清理异步备份 status 文件。',
+  );
+}
+
 for (const file of jenkinsSourceCheckoutFiles) {
   const content = readFileSync(file, 'utf8');
   if (!content.includes(jenkinsLoopbackGitRemote)) {

scripts/database-backup-to-oss.mjs

@@ -64,6 +64,7 @@ function usage() {
   npm run database:backup:oss -- [--mode full|history] [--storage-format archive|files] [--data-dir <path>] [--work-dir <path>] [--bucket <bucket>] [--object-prefix <prefix>] [--keep-local]
   node -- scripts/database-backup-to-oss.mjs [--stop-service spacetimedb.service] [--restart-service-after genarrative-api.service] [--defer-upload]
   node -- scripts/database-backup-to-oss.mjs --upload-archive <path>
+  node -- scripts/database-backup-to-oss.mjs --upload-deferred-dir <path>
   node -- scripts/database-backup-to-oss.mjs --publish-manifest <path>
   node -- scripts/database-backup-to-oss.mjs --restore-files-state <path> --restore-dir <path>
   node -- scripts/database-backup-to-oss.mjs --restore-files-latest --restore-dir <path> [--dry-run]
@@ -74,6 +75,7 @@ function usage() {
   --storage-format files 不打包：按原相对路径建立 catalog，文件内容以 SHA-256 不可变对象上传；重复运行只上传新增或变化内容。
   archive history 必须有已验真的 full baseline state；files history 必须复用同一 work-dir 中已发布的 full catalog state。
   --defer-upload 只生成本地冷备份和 manifest，不上传；后续用 --upload-archive 异步上传。
+  --upload-deferred-dir 串行收敛目录内 deferred/pending 及已上传未清理归档；只有 OSS 上传与验真完成后才按 keep-local 规则删除。
   默认读取 .env / .env.local / .env.secrets.local；生产服务可传 --env-file /etc/genarrative/api-server.env。
   shell 环境变量优先级最高，不会被 env 文件覆盖。
 
@@ -157,6 +159,7 @@ function parseArgs(argv) {
     dryRun: false,
     deferUpload: false,
     uploadArchive: '',
+    uploadDeferredDir: '',
     manifestFile: '',
     objectKey: '',
     resultFile: '',
@@ -237,6 +240,9 @@ function parseArgs(argv) {
       case '--upload-archive':
         options.uploadArchive = readValue();
         break;
+      case '--upload-deferred-dir':
+        options.uploadDeferredDir = readValue();
+        break;
       case '--manifest-file':
         options.manifestFile = readValue();
         break;
@@ -2811,6 +2817,50 @@ export async function uploadHistoryArchiveWithCleanup({
   return {result, uploadedManifest, cleanup, state};
 }
 
+export function discoverDeferredArchiveUploads({workDir, database, includeUploaded = false}) {
+  const resolvedWorkDir = resolvePath(workDir);
+  if (!existsSync(resolvedWorkDir)) {
+    return {archives: [], missingArchives: []};
+  }
+
+  const archives = [];
+  const missingArchives = [];
+  const manifestSuffix = '.tar.gz.manifest.json';
+  const expectedDatabase = String(database || '').trim();
+  const entries = readdirSync(resolvedWorkDir, {withFileTypes: true})
+    .filter((candidate) => candidate.isFile() && candidate.name.endsWith(manifestSuffix))
+    .sort((left, right) => left.name.localeCompare(right.name, 'en'));
+  for (const entry of entries) {
+    const manifestPath = join(resolvedWorkDir, entry.name);
+    const manifest = readManifest(manifestPath);
+    const uploadStatus = String(manifest.uploadStatus || '').trim();
+    if (!['deferred', 'pending'].includes(uploadStatus) && !(includeUploaded && uploadStatus === 'uploaded')) {
+      continue;
+    }
+    if (expectedDatabase && String(manifest.database || '').trim() !== expectedDatabase) {
+      continue;
+    }
+    if (!manifest.archivePath) {
+      throw new Error(`deferred 备份清单缺少 archivePath: ${manifestPath}`);
+    }
+    const archivePath = resolvePath(manifest.archivePath);
+    if (dirname(archivePath) !== resolvedWorkDir || manifestPath !== `${archivePath}.manifest.json`) {
+      throw new Error(`deferred 备份路径与清单不匹配: ${manifestPath}`);
+    }
+    const candidate = {archivePath, manifestPath, manifest};
+    if (existsSync(archivePath)) {
+      const archiveStat = lstatSync(archivePath);
+      if (!archiveStat.isFile() || archiveStat.isSymbolicLink()) {
+        throw new Error(`deferred 备份归档必须是非符号链接的普通文件: ${archivePath}`);
+      }
+      archives.push(candidate);
+    } else {
+      missingArchives.push(candidate);
+    }
+  }
+  return {archives, missingArchives};
+}
+
 async function uploadExistingArchive({args, env, bucket, endpoint, accessKeyId, accessKeySecret, objectPrefix, bandwidthLimiter}) {
   const archivePath = resolvePath(args.uploadArchive);
   if (!existsSync(archivePath)) {
@@ -2897,6 +2947,37 @@ async function uploadExistingArchive({args, env, bucket, endpoint, accessKeyId,
   }
 }
 
+async function uploadDeferredArchives({args, env, bucket, endpoint, accessKeyId, accessKeySecret, objectPrefix, database, bandwidthLimiter}) {
+  const workDir = resolvePath(args.uploadDeferredDir);
+  const keepLocal = args.keepLocal || String(env.GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL ?? '').trim().toLowerCase() === 'true';
+  const {archives, missingArchives} = discoverDeferredArchiveUploads({
+    workDir,
+    database,
+    includeUploaded: !keepLocal,
+  });
+  for (const {manifestPath} of missingArchives) {
+    console.warn(`[database-backup] deferred 清单对应的本地归档不存在，跳过: ${manifestPath}`);
+  }
+  if (archives.length === 0) {
+    console.log(`[database-backup] 没有可补偿的本地归档: ${workDir}`);
+    return;
+  }
+  console.log(`[database-backup] 开始串行上传待补偿本地归档: count=${archives.length}`);
+  for (const {archivePath, manifestPath} of archives) {
+    await uploadExistingArchive({
+      args: {...args, uploadArchive: archivePath, manifestFile: manifestPath},
+      env,
+      bucket,
+      endpoint,
+      accessKeyId,
+      accessKeySecret,
+      objectPrefix,
+      bandwidthLimiter,
+    });
+  }
+  console.log(`[database-backup] 待补偿本地归档上传完成: count=${archives.length}`);
+}
+
 async function publishExistingManifest({args, bucket, endpoint, accessKeyId, accessKeySecret, bandwidthLimiter}) {
   const manifestPath = resolvePath(args.publishManifest);
   const manifest = readManifest(manifestPath);
@@ -3108,6 +3189,7 @@ async function main() {
   ));
   const workDir = resolvePath(firstNonEmpty(
     args.workDir,
+    args.uploadDeferredDir,
     env.GENARRATIVE_DATABASE_BACKUP_WORK_DIR,
     isProductionLike ? DEFAULT_PRODUCTION_WORK_DIR : DEFAULT_LOCAL_WORK_DIR,
   ));
@@ -3171,6 +3253,9 @@ async function main() {
   if (args.restoreDir) {
     throw new Error('--restore-dir 只能与 --restore-files-state 或 --restore-files-latest 一起使用。');
   }
+  if (args.uploadArchive && args.uploadDeferredDir) {
+    throw new Error('--upload-archive 与 --upload-deferred-dir 不能同时使用。');
+  }
 
   if (!args.dryRun) {
     const lockPath = acquireBackupLock({workDir, database});
@@ -3196,6 +3281,21 @@ async function main() {
     return;
   }
 
+  if (args.uploadDeferredDir) {
+    await uploadDeferredArchives({
+      args,
+      env,
+      bucket,
+      endpoint,
+      accessKeyId,
+      accessKeySecret,
+      objectPrefix,
+      database,
+      bandwidthLimiter: uploadBandwidthLimiter,
+    });
+    return;
+  }
+
   if (storageFormat === 'files') {
     if (args.deferUpload) {
       throw new Error('files 模式无需本地归档且不支持 --defer-upload；失败后使用同一 work-dir 重跑即可续传。');

scripts/deploy/production-stdb-publish.sh

@@ -532,6 +532,11 @@ prepare_async_backup() {
 }
 
 start_async_backup_upload() {
+  local log_dir=""
+  local node_binary=""
+  local unit_name=""
+  local unit_suffix=""
+
   if [[ -z "${ASYNC_BACKUP_STATUS_FILE}" || ! -f "${ASYNC_BACKUP_STATUS_FILE}" ]]; then
     echo "[production-stdb-publish] 警告：未找到可上传的本地备份状态文件，跳过异步上传" >&2
     return 0
@@ -543,16 +548,58 @@ start_async_backup_upload() {
     echo "[production-stdb-publish] 警告：备份状态文件缺少 archivePath 或 manifestPath，跳过异步上传" >&2
     return 0
   fi
+  if [[ "${ASYNC_BACKUP_ARCHIVE}" != /* || ! -f "${ASYNC_BACKUP_ARCHIVE}" || -L "${ASYNC_BACKUP_ARCHIVE}" ]]; then
+    echo "[production-stdb-publish] 警告：异步上传归档必须是现存、非符号链接的普通绝对路径文件，保留状态文件等待处理: ${ASYNC_BACKUP_ARCHIVE}" >&2
+    return 1
+  fi
+  if [[ "${ASYNC_BACKUP_MANIFEST}" != /* || ! -f "${ASYNC_BACKUP_MANIFEST}" || -L "${ASYNC_BACKUP_MANIFEST}" ]]; then
+    echo "[production-stdb-publish] 警告：异步上传 manifest 必须是现存、非符号链接的普通绝对路径文件，保留状态文件等待处理: ${ASYNC_BACKUP_MANIFEST}" >&2
+    return 1
+  fi
+  if ! command -v systemd-run >/dev/null 2>&1; then
+    echo "[production-stdb-publish] 警告：systemd-run 不可用，无法启动独立上传服务；保留状态文件等待处理" >&2
+    return 1
+  fi
 
-  mkdir -p "$(dirname "${ASYNC_BACKUP_ARCHIVE}")"
-  ASYNC_BACKUP_LOG="$(dirname "${ASYNC_BACKUP_ARCHIVE}")/${DATABASE}-upload.log"
-  echo "[production-stdb-publish] 后台上传本地备份到 OSS: ${ASYNC_BACKUP_ARCHIVE}"
-  nohup node -- "${ASYNC_BACKUP_SCRIPT}" \
-    --env-file /etc/genarrative/api-server.env \
-    --upload-archive "${ASYNC_BACKUP_ARCHIVE}" \
-    --manifest-file "${ASYNC_BACKUP_MANIFEST}" \
-    >"${ASYNC_BACKUP_LOG}" 2>&1 &
-  echo "[production-stdb-publish] OSS 后台上传日志: ${ASYNC_BACKUP_LOG}"
+  node_binary="$(command -v node || true)"
+  if [[ "${node_binary}" != /* || ! -x "${node_binary}" ]]; then
+    echo "[production-stdb-publish] 警告：未找到可供 systemd 服务执行的绝对 node 路径；保留状态文件等待处理" >&2
+    return 1
+  fi
+
+  log_dir="$(dirname "${ASYNC_BACKUP_ARCHIVE}")"
+  unit_suffix="$(date -u +%Y%m%dT%H%M%S%N)-$$-${RANDOM}"
+  unit_name="genarrative-stdb-backup-upload-${unit_suffix}.service"
+  if ! ASYNC_BACKUP_LOG="$(mktemp "${log_dir}/${DATABASE}-upload-${unit_suffix}.XXXXXX.log")"; then
+    echo "[production-stdb-publish] 警告：无法创建独立 OSS 上传日志，保留状态文件和本地归档等待处理" >&2
+    return 1
+  fi
+  if ! chmod 0600 "${ASYNC_BACKUP_LOG}"; then
+    echo "[production-stdb-publish] 警告：无法收紧独立 OSS 上传日志权限，保留状态文件和本地归档等待处理: ${ASYNC_BACKUP_LOG}" >&2
+    return 1
+  fi
+
+  echo "[production-stdb-publish] 通过独立 systemd 服务串行上传 deferred/pending 本地备份到 OSS: ${log_dir}"
+  if ! run_privileged systemd-run \
+    --no-ask-password \
+    --unit="${unit_name}" \
+    --description="Genarrative SpacetimeDB backup upload ${DATABASE}" \
+    --collect \
+    --service-type=exec \
+    --property="Restart=no" \
+    --property="UMask=0077" \
+    --property="StandardOutput=append:${ASYNC_BACKUP_LOG}" \
+    --property="StandardError=append:${ASYNC_BACKUP_LOG}" \
+    -- "${node_binary}" -- "${ASYNC_BACKUP_SCRIPT}" \
+      --env-file /etc/genarrative/api-server.env \
+      --database "${DATABASE}" \
+      --upload-deferred-dir "${log_dir}"; then
+    echo "[production-stdb-publish] 警告：独立 OSS 上传服务启动失败，保留状态文件和本地归档等待处理；启动日志: ${ASYNC_BACKUP_LOG}" >&2
+    return 1
+  fi
+
+  echo "[production-stdb-publish] OSS 上传服务已启动: ${unit_name}"
+  echo "[production-stdb-publish] OSS 上传日志: ${ASYNC_BACKUP_LOG}"
   rm -f "${ASYNC_BACKUP_STATUS_FILE}"
   ASYNC_BACKUP_STATUS_FILE=""
 }