合并最新master并同步备份优化
- 合并逐文件备份并发上传与符号链接恢复支持。 - 同步备份环境示例、运维文档与验证脚本。
This commit is contained in:
Vendored
+2
@@ -159,6 +159,8 @@ GENARRATIVE_DATABASE_BACKUP_OSS_PREFIX=database-backups
|
||||
GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL=false
|
||||
# 可选:files 为逐文件 CAS + catalog,不生成 tar.gz;archive 保留旧全量压缩包兼容行为。
|
||||
GENARRATIVE_DATABASE_BACKUP_STORAGE_FORMAT=archive
|
||||
# files 模式并行 PUT/HEAD 数量,必须为 1-64;默认 16。
|
||||
GENARRATIVE_DATABASE_BACKUP_FILES_CONCURRENCY=16
|
||||
# 可选:显式要求备份工作目录所在文件系统至少保留的可用空间;为空时按数据目录大小 + 安全余量估算。
|
||||
GENARRATIVE_DATABASE_BACKUP_MIN_FREE_BYTES=
|
||||
# archive history 模式持久化已验真 full baseline 与追加批次;files 模式改用 work-dir 下的 files state。
|
||||
|
||||
@@ -66,7 +66,7 @@
|
||||
## 2026-07-16 SpacetimeDB 备份采用逐文件基线、CAS 增量与安全历史清理
|
||||
|
||||
- 背景:SpacetimeDB standalone 2.6.0 不自动删除已被 snapshot 覆盖的历史 commitlog 与旧 snapshot;反复压缩整个 `/stdb` 会重复占用磁盘、停机和 OSS 带宽。上游 issue #5542 的 contributor 明确说明,不触碰最新 snapshot 与重启所需 commitlog suffix 时,可在运行中移动或删除这些历史文件。
|
||||
- 决策:统一脚本新增 `--storage-format files`,完整基线递归保留目录与文件相对路径,普通文件按 SHA-256 上传为不可变 CAS 对象,catalog 记录目录、路径、长度、SHA 与对象 key;相同内容不重复 PUT,后续 full 扫描只上传新增或变化内容,不再生成 tar.gz。旧 `archive` 路径保留兼容。full 必须从停库目录或已验证的冻结副本生成,不能把在线跨文件扫描称为一致时点备份。
|
||||
- 决策:统一脚本新增 `--storage-format files`,完整基线递归保留目录、文件和 data-dir 内部相对符号链接,普通文件按 SHA-256 上传为不可变 CAS 对象,catalog 记录目录、路径、长度、SHA、对象 key 与相对链接目标;绝对或越界链接拒绝备份。相同内容不重复 PUT,后续 full 扫描只上传新增或变化内容,不再生成 tar.gz。旧 `archive` 路径保留兼容。full 必须从停库目录或已验证的冻结副本生成,不能把在线跨文件扫描称为一致时点备份。
|
||||
- history 继续按 replica 计算安全边界:只接受完整、未锁定且含同 offset `.snapshot_bsatn` 的 snapshot,保留跨越最新 snapshot 的边界 segment 及全部后缀。旧 segment 对和旧 snapshot 被递归映射为单文件 CAS 对象;对象、history catalog、full baseline catalog、候选 fingerprint 与当前边界全部验真后才删除源文件。同库执行用 work-dir PID lock 互斥。
|
||||
- OSS 固定恢复入口为 `<prefix>/<database>/latest.json`。CAS 文件和 full/history catalog 保持不可变;latest pointer 只保存最新 full catalog 与已发布 history catalog 的 object key、长度和 SHA,不包含主机绝对路径或文件内容。每次 state 变化先验真全部引用 catalog,再覆盖上传并 HEAD 验真 latest pointer,成功后才落本地 state;history 还必须在 pointer 成功后才允许删除源文件。全新机器可仅凭 bucket、database、prefix 与 OSS 凭据自动下载 pointer 和 full catalog。
|
||||
- dev 带宽不足时,允许把已冻结的 dev 基线经 `10.2.0.10 -> 10.2.4.16` 内网 rsync 到 release 独立 staging,再用 release 出口上传 dev bucket;staging 不得指向 release `/stdb`,不得停止或修改 release 服务,传输凭据必须临时创建并在演练后移除。catalog 不记录 staging 绝对路径,files state 可回传 dev 继续 history。
|
||||
|
||||
@@ -330,6 +330,7 @@ GENARRATIVE_DATABASE_BACKUP_OSS_ENDPOINT=oss-cn-shanghai.aliyuncs.com
|
||||
GENARRATIVE_DATABASE_BACKUP_OSS_PREFIX=database-backups
|
||||
GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL=false
|
||||
GENARRATIVE_DATABASE_BACKUP_STORAGE_FORMAT=archive
|
||||
GENARRATIVE_DATABASE_BACKUP_FILES_CONCURRENCY=16
|
||||
GENARRATIVE_DATABASE_BACKUP_MIN_FREE_BYTES=
|
||||
GENARRATIVE_DATABASE_BACKUP_BASELINE_STATE=/var/lib/genarrative/database-backups/genarrative-prod-history-state.json
|
||||
# 仅 archive history 首次从一份 uploadStatus=uploaded 的全量 manifest 初始化 state 时设置或传 --baseline-manifest。
|
||||
@@ -342,7 +343,7 @@ GENARRATIVE_DATABASE_BACKUP_OSS_ACCESS_KEY_SECRET=
|
||||
|
||||
`files-history` 使用仓库模板 `deploy/systemd/genarrative-database-backup-files-history.conf` 覆盖主 service 的 `ExecStart`,从 `/etc/genarrative/api-server.env` 读取 data-dir、database、bucket、prefix 与 OSS 凭据,不在 unit 写死环境目标,也不传 `--stop-service`。Server-Provision 在改动 drop-in 前,先用 current release 的同一脚本、同一 env 和 `DATABASE_BACKUP_FILES_HISTORY_WORK_DIR` 执行一次 history `--dry-run`;缺少已发布 full catalog 的 files state、current 脚本过旧或配置不匹配都会在安装 drop-in 和 `daemon-reload` 前失败。选择 `archive-full` 会主动删除仓库托管的 `10-files-history.conf` 与 dev 试点遗留的 `10-dev-files.conf`,防止 systemd 继续合并旧覆盖。dev 可继续指定已有 `/var/lib/genarrative/database-backups/dev-files`,release 建议先在 `/var/lib/genarrative/database-backups/release-files` 建立自己的 full baseline;两台机器不得复用或互传本地 state 目录冒充本机基线。启用时通过 Server-Provision Job 选择目标、`DATABASE_BACKUP_PROFILE=files-history` 和对应 work-dir,先保持 `DRY_RUN=true` 核对,再以同参数正式 provision。不要直接在 `/etc/systemd/system` 手写第二份 drop-in。
|
||||
|
||||
files full 会递归扫描 data-dir,保留空目录和每个普通文件的相对路径;文件按 SHA-256 上传到不可变对象 key,catalog 记录目录、路径、长度、SHA 和对象 key,不写 staging 主机的绝对路径。相同 catalog 重跑不重复 PUT;新增或变化文件先 HEAD CAS 对象,存在且长度/SHA 元数据一致就复用,否则上传。full 基线必须来自停库后的 data-dir 或已通过恢复验证的冻结副本;源文件上传前后 stat 虽会复核,但在线扫描不能保证 2083 个文件属于同一跨文件一致时点。catalog 验真后,脚本把最新 full/history 引用发布到固定 `<prefix>/<database>/latest.json`,全新机器不需要本地 state 即可自动发现恢复入口。
|
||||
files full 会递归扫描 data-dir,保留空目录、每个普通文件的相对路径,以及目标仍位于 data-dir 内部的相对符号链接;绝对链接或解析后越界的链接直接拒绝。文件按 SHA-256 上传到不可变对象 key,catalog 记录目录、路径、长度、SHA、对象 key 和相对链接目标,不写 staging 主机的绝对路径。相同 catalog 重跑不重复 PUT;新增或变化文件先 HEAD CAS 对象,存在且长度/SHA 元数据一致就复用,否则上传。对象 PUT/HEAD 默认以 16 路并行执行,可用 `GENARRATIVE_DATABASE_BACKUP_FILES_CONCURRENCY=1..64` 调整;并发只缩短传输与验真时间,不改变“全部对象、catalog 与 latest pointer 成功后才推进 state/清理”的顺序。full 基线必须来自停库后的 data-dir 或已通过恢复验证的冻结副本;源文件上传前后 stat 虽会复核,但在线扫描不能保证大量文件属于同一跨文件一致时点。catalog 验真后,脚本把最新 full/history 引用发布到固定 `<prefix>/<database>/latest.json`,全新机器不需要本地 state 即可自动发现恢复入口。
|
||||
|
||||
history 的安全边界按每个 replica 独立计算。设最新完整且未锁定的 snapshot offset 为 `S`;数字更大但缺少同 offset `.snapshot_bsatn`、仍存在同名 `.lock` 的目录不能参与边界计算。脚本必须保留起始 offset 小于等于 `S` 的最后一个 commitlog segment,以及它之后的全部 segment;只处理更早的 `.stdb.log` / `.stdb.ofs`,snapshot 只处理最新目录之前的旧目录。files history 会递归展开候选目录,逐对象复用或上传,随后依次验真候选对象、history catalog 与 full baseline catalog,再重新扫描边界和 stat fingerprint,最后覆盖发布并验真 `latest.json`;任何一步失败都不推进 state 或删除源文件。脚本在 work-dir 使用 PID lock 拒绝同库并发上传,SSH 超时后必须先检查原进程,不能直接重跑。
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
import {spawnSync} from 'node:child_process';
|
||||
import {createHash} from 'node:crypto';
|
||||
import {chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync} from 'node:fs';
|
||||
import {chmodSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readlinkSync, rmSync, statSync, symlinkSync, writeFileSync} from 'node:fs';
|
||||
import {tmpdir} from 'node:os';
|
||||
import path from 'node:path';
|
||||
|
||||
@@ -60,6 +60,7 @@ async function main() {
|
||||
await assertHistorySuccessfulUploadCleansAndIsIdempotent();
|
||||
await assertHistoryResumeReverifiesArchiveAndManifest();
|
||||
await assertDirectFilesPreservePathsAndIncrementWithoutDuplicateUpload();
|
||||
await assertDirectFilesConcurrencyIsBounded();
|
||||
await assertDirectHistoryPublishesCatalogBeforeCleanup();
|
||||
await assertDirectHistoryWithoutCandidatesPublishesLatest();
|
||||
await assertDirectFilesRestoreDownloadsCatalogAndObjects();
|
||||
@@ -106,6 +107,8 @@ async function assertDirectFilesPreservePathsAndIncrementWithoutDuplicateUpload(
|
||||
const workDir = path.join(root, 'work');
|
||||
mkdirSync(path.join(dataDir, 'replicas', '1', 'snapshots', '00000000000000000010.snapshot_dir', 'objects'), {recursive: true});
|
||||
mkdirSync(path.join(dataDir, 'empty-directory'), {recursive: true});
|
||||
mkdirSync(path.join(dataDir, 'bin', '2.6.0'), {recursive: true});
|
||||
symlinkSync('2.6.0', path.join(dataDir, 'bin', 'current'));
|
||||
writeFileSync(path.join(dataDir, 'control-db'), 'control');
|
||||
writeFileSync(
|
||||
path.join(dataDir, 'replicas', '1', 'snapshots', '00000000000000000010.snapshot_dir', 'objects', 'object.bin'),
|
||||
@@ -122,6 +125,10 @@ async function assertDirectFilesPreservePathsAndIncrementWithoutDuplicateUpload(
|
||||
'files catalog 必须原样保留 snapshot 内文件的相对路径。',
|
||||
);
|
||||
assertTrue(collected.directories.includes('empty-directory'), 'files catalog 必须保留空目录。');
|
||||
assertTrue(
|
||||
collected.symlinks.some(({path: symlinkPath, target}) => symlinkPath === 'bin/current' && target === '2.6.0'),
|
||||
'files catalog 必须保留指向 data-dir 内部的相对符号链接。',
|
||||
);
|
||||
|
||||
const first = await runDirectFilesBackup(options);
|
||||
assertEqual(first.uploadedCount, 2, '首次 files full 应上传全部普通文件。');
|
||||
@@ -147,6 +154,44 @@ async function assertDirectFilesPreservePathsAndIncrementWithoutDuplicateUpload(
|
||||
assertEqual(incremental.reusedCount, 1, '增量 files full 应复用未变化 snapshot 文件。');
|
||||
}
|
||||
|
||||
async function assertDirectFilesConcurrencyIsBounded() {
|
||||
const root = path.join(tmpRoot, 'direct-files-concurrency');
|
||||
const dataDir = path.join(root, 'stdb');
|
||||
const workDir = path.join(root, 'work');
|
||||
mkdirSync(dataDir, {recursive: true});
|
||||
for (let index = 0; index < 12; index += 1) {
|
||||
writeFileSync(path.join(dataDir, `file-${index}.bin`), `content-${index}`);
|
||||
}
|
||||
const harness = createDirectOssHarness();
|
||||
let activeUploads = 0;
|
||||
let maxActiveUploads = 0;
|
||||
const uploadFn = async (options) => {
|
||||
activeUploads += 1;
|
||||
maxActiveUploads = Math.max(maxActiveUploads, activeUploads);
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
try {
|
||||
return await harness.uploadFn(options);
|
||||
} finally {
|
||||
activeUploads -= 1;
|
||||
}
|
||||
};
|
||||
await runDirectFilesBackup({
|
||||
mode: 'full',
|
||||
dataDir,
|
||||
workDir,
|
||||
database: 'test-db',
|
||||
bucket: 'backup-bucket',
|
||||
objectPrefix: 'database-backups',
|
||||
uploadOptions: {},
|
||||
uploadFn,
|
||||
uploadManifestFn: harness.uploadManifestFn,
|
||||
verifyFn: harness.verifyFn,
|
||||
concurrency: 3,
|
||||
});
|
||||
assertTrue(maxActiveUploads > 1, 'files 备份应按配置并发处理多个对象。');
|
||||
assertTrue(maxActiveUploads <= 3, 'files 备份对象并发数不得超过配置上限。');
|
||||
}
|
||||
|
||||
async function assertDirectHistoryPublishesCatalogBeforeCleanup() {
|
||||
const fixture = createHistoryFixture('direct-files-history-cleanup', {nestedData: false});
|
||||
createReplicaHistory(fixture.replicasDir, '1', {snapshots: [0, 10], segments: [0, 1, 11]});
|
||||
@@ -258,6 +303,8 @@ async function assertDirectFilesRestoreDownloadsCatalogAndObjects() {
|
||||
const restoreDir = path.join(root, 'restore');
|
||||
mkdirSync(path.join(dataDir, 'empty-directory'), {recursive: true});
|
||||
mkdirSync(path.join(dataDir, 'config'), {recursive: true});
|
||||
mkdirSync(path.join(dataDir, 'bin', '2.6.0'), {recursive: true});
|
||||
symlinkSync('2.6.0', path.join(dataDir, 'bin', 'current'));
|
||||
const keyPath = path.join(dataDir, 'config', 'id_ecdsa');
|
||||
writeFileSync(keyPath, 'private key fixture');
|
||||
chmodSync(keyPath, 0o640);
|
||||
@@ -291,6 +338,8 @@ async function assertDirectFilesRestoreDownloadsCatalogAndObjects() {
|
||||
assertEqual(readFileSync(path.join(restoreDir, 'config', 'id_ecdsa'), 'utf8'), 'updated private key fixture', 'files restore 必须按最新 full catalog 的原相对路径恢复内容。');
|
||||
assertTrue(existsSync(path.join(restoreDir, 'empty-directory')), 'files restore 必须重建空目录。');
|
||||
assertEqual(statSync(path.join(restoreDir, 'config', 'id_ecdsa')).mode & 0o7777, 0o640, 'files restore 必须恢复文件权限。');
|
||||
assertTrue(lstatSync(path.join(restoreDir, 'bin', 'current')).isSymbolicLink(), 'files restore 必须重建符号链接。');
|
||||
assertEqual(readlinkSync(path.join(restoreDir, 'bin', 'current'), 'utf8'), '2.6.0', 'files restore 必须保留符号链接目标。');
|
||||
|
||||
rmSync(restoreDir, {recursive: true, force: true});
|
||||
const downloadBufferFn = async ({objectKey}) => {
|
||||
@@ -322,6 +371,7 @@ async function assertDirectFilesRestoreDownloadsCatalogAndObjects() {
|
||||
});
|
||||
assertEqual(dryRun.catalogId, latestFull.catalogId, 'OSS-only dry-run 必须选择 latestFullCatalog。');
|
||||
assertEqual(dryRun.fileCount, 1, 'OSS-only dry-run 应返回 full catalog 文件数。');
|
||||
assertEqual(dryRun.symlinkCount, 1, 'OSS-only dry-run 应返回 full catalog 符号链接数。');
|
||||
assertEqual(dryRun.totalSizeBytes, String(Buffer.byteLength('updated private key fixture')), 'OSS-only dry-run 应返回总字节数。');
|
||||
assertEqual(objectDownloadCount, 0, 'OSS-only dry-run 不得下载数据对象。');
|
||||
assertTrue(!existsSync(restoreDir), 'OSS-only dry-run 不得创建恢复目录。');
|
||||
@@ -339,6 +389,7 @@ async function assertDirectFilesRestoreDownloadsCatalogAndObjects() {
|
||||
assertEqual(latestRestored.catalogId, latestFull.catalogId, 'OSS-only restore 必须选择 latestFullCatalog。');
|
||||
assertEqual(latestRestored.downloadedCount, 1, 'OSS-only restore 应下载 latest full catalog 的数据对象。');
|
||||
assertEqual(readFileSync(path.join(restoreDir, 'config', 'id_ecdsa'), 'utf8'), 'updated private key fixture', 'OSS-only restore 应还原最新 full 内容。');
|
||||
assertEqual(readlinkSync(path.join(restoreDir, 'bin', 'current'), 'utf8'), '2.6.0', 'OSS-only restore 应还原符号链接。');
|
||||
}
|
||||
|
||||
function assertCanonicalQueryAndAuthorizationIncludeMultipartParameters() {
|
||||
|
||||
@@ -12,11 +12,13 @@ import {
|
||||
openSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
realpathSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
statfsSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from 'node:fs';
|
||||
import {basename, dirname, isAbsolute, join, relative, resolve, sep} from 'node:path';
|
||||
@@ -45,6 +47,8 @@ const OSS_MAX_MULTIPART_PARTS = 10_000;
|
||||
const DEFAULT_OSS_REQUEST_MAX_ATTEMPTS = 5;
|
||||
const DEFAULT_OSS_RETRY_BASE_DELAY_MS = 1_000;
|
||||
const DEFAULT_OSS_RETRY_MAX_DELAY_MS = 30_000;
|
||||
const DEFAULT_DIRECT_FILES_CONCURRENCY = 16;
|
||||
const MAX_DIRECT_FILES_CONCURRENCY = 64;
|
||||
const RETRYABLE_OSS_HTTP_STATUSES = new Set([408, 429, 500, 502, 503, 504]);
|
||||
const HISTORY_STATE_SCHEMA_VERSION = 1;
|
||||
const HISTORY_MANIFEST_SCHEMA_VERSION = 1;
|
||||
@@ -275,6 +279,14 @@ function firstNonEmpty(...values) {
|
||||
return values.map((value) => String(value ?? '').trim()).find(Boolean) ?? '';
|
||||
}
|
||||
|
||||
function parseDirectFilesConcurrency(rawValue) {
|
||||
const value = Number(String(rawValue ?? DEFAULT_DIRECT_FILES_CONCURRENCY).trim());
|
||||
if (!Number.isSafeInteger(value) || value < 1 || value > MAX_DIRECT_FILES_CONCURRENCY) {
|
||||
throw new Error(`GENARRATIVE_DATABASE_BACKUP_FILES_CONCURRENCY 必须是 1-${MAX_DIRECT_FILES_CONCURRENCY} 的整数,实际: ${rawValue}`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function resolvePath(value) {
|
||||
return isAbsolute(value) ? value : resolve(REPO_ROOT, value);
|
||||
}
|
||||
@@ -1058,6 +1070,7 @@ export async function collectDirectFileEntries({dataDir, candidates = null, obje
|
||||
throw new Error(`files 数据目录不存在或不是目录: ${resolvedDataDir}`);
|
||||
}
|
||||
const files = new Map();
|
||||
const symlinks = new Map();
|
||||
const directories = new Set(['.']);
|
||||
const roots = candidates === null
|
||||
? [{absolutePath: resolvedDataDir, relativePath: '.'}]
|
||||
@@ -1069,7 +1082,13 @@ export async function collectDirectFileEntries({dataDir, candidates = null, obje
|
||||
const visit = async (absolutePath, relativePath) => {
|
||||
const stat = lstatSync(absolutePath);
|
||||
if (stat.isSymbolicLink()) {
|
||||
throw new Error(`files 模式拒绝符号链接: ${absolutePath}`);
|
||||
const target = readlinkSync(absolutePath, 'utf8');
|
||||
if (!target || isAbsolute(target)) {
|
||||
throw new Error(`files 模式只允许 data-dir 内部的相对符号链接: ${absolutePath} -> ${target}`);
|
||||
}
|
||||
assertSafeRelativePath(resolvedDataDir, resolve(dirname(absolutePath), target));
|
||||
symlinks.set(relativePath, {path: relativePath, target});
|
||||
return;
|
||||
}
|
||||
if (stat.isDirectory()) {
|
||||
directories.add(relativePath);
|
||||
@@ -1108,16 +1127,18 @@ export async function collectDirectFileEntries({dataDir, candidates = null, obje
|
||||
return {
|
||||
directories: [...directories].sort(),
|
||||
files: [...files.values()].sort((left, right) => left.path.localeCompare(right.path)),
|
||||
symlinks: [...symlinks.values()].sort((left, right) => left.path.localeCompare(right.path)),
|
||||
};
|
||||
}
|
||||
|
||||
function directCatalogIdentity({mode, baselineCatalogId, rootName, directories, files}) {
|
||||
function directCatalogIdentity({mode, baselineCatalogId, rootName, directories, files, symlinks}) {
|
||||
return sha256Hex(JSON.stringify({
|
||||
mode,
|
||||
baselineCatalogId: baselineCatalogId || '',
|
||||
rootName,
|
||||
directories,
|
||||
files: files.map(({path, sizeBytes, sha256, mode, objectKey}) => ({path, sizeBytes, sha256, mode, objectKey})),
|
||||
symlinks,
|
||||
}));
|
||||
}
|
||||
|
||||
@@ -1330,6 +1351,7 @@ export async function runDirectFilesBackup({
|
||||
uploadFn = uploadArchive,
|
||||
uploadManifestFn = uploadManifestFile,
|
||||
verifyFn = verifyOssObject,
|
||||
concurrency = 1,
|
||||
}) {
|
||||
mkdirSync(workDir, {recursive: true});
|
||||
const statePath = directFilesStatePath({workDir, database});
|
||||
@@ -1362,6 +1384,7 @@ export async function runDirectFilesBackup({
|
||||
rootName,
|
||||
directories: collected.directories,
|
||||
files: collected.files.map(({sourceStat: _sourceStat, ...file}) => file),
|
||||
symlinks: collected.symlinks,
|
||||
};
|
||||
writeManifest({manifestPath: catalogPath, payload: catalog});
|
||||
const summary = {
|
||||
@@ -1370,13 +1393,14 @@ export async function runDirectFilesBackup({
|
||||
catalogObjectKey,
|
||||
catalogId,
|
||||
fileCount: collected.files.length,
|
||||
symlinkCount: collected.symlinks.length,
|
||||
totalSizeBytes: collected.files.reduce((sum, file) => sum + BigInt(file.sizeBytes), 0n).toString(),
|
||||
candidateCount: plan?.candidates.length ?? 0,
|
||||
};
|
||||
if (resultFile) {
|
||||
atomicWriteJson(resolvePath(resultFile), {...summary, dryRun});
|
||||
}
|
||||
console.log(`[database-backup] files ${mode}: files=${summary.fileCount}, size=${formatBytes(summary.totalSizeBytes)}, catalog=${catalogId}`);
|
||||
console.log(`[database-backup] files ${mode}: files=${summary.fileCount}, symlinks=${summary.symlinkCount}, size=${formatBytes(summary.totalSizeBytes)}, catalog=${catalogId}`);
|
||||
if (dryRun) {
|
||||
console.log('[database-backup] files dry-run,仅扫描并生成本地 catalog,不上传或删除。');
|
||||
return {...summary, catalog, uploadedCount: 0, reusedCount: 0};
|
||||
@@ -1436,23 +1460,35 @@ export async function runDirectFilesBackup({
|
||||
const previousFiles = new Map((state?.latestCatalog?.files ?? []).map((file) => [file.path, file]));
|
||||
let uploadedCount = 0;
|
||||
let reusedCount = 0;
|
||||
for (const [index, file] of collected.files.entries()) {
|
||||
const result = await ensureDirectObject({
|
||||
file,
|
||||
dataDir,
|
||||
uploadOptions,
|
||||
previousFile: previousFiles.get(file.path),
|
||||
verifyCatalogReuse: mode === 'history',
|
||||
uploadFn,
|
||||
verifyFn,
|
||||
});
|
||||
if (result.status === 'uploaded') {
|
||||
uploadedCount += 1;
|
||||
} else {
|
||||
reusedCount += 1;
|
||||
let nextIndex = 0;
|
||||
let completedCount = 0;
|
||||
const workerCount = Math.min(concurrency, collected.files.length);
|
||||
const workers = Array.from({length: workerCount}, async () => {
|
||||
while (nextIndex < collected.files.length) {
|
||||
const index = nextIndex;
|
||||
nextIndex += 1;
|
||||
const file = collected.files[index];
|
||||
const result = await ensureDirectObject({
|
||||
file,
|
||||
dataDir,
|
||||
uploadOptions,
|
||||
previousFile: previousFiles.get(file.path),
|
||||
verifyCatalogReuse: mode === 'history',
|
||||
uploadFn,
|
||||
verifyFn,
|
||||
});
|
||||
if (result.status === 'uploaded') {
|
||||
uploadedCount += 1;
|
||||
} else {
|
||||
reusedCount += 1;
|
||||
}
|
||||
completedCount += 1;
|
||||
if (collected.files.length <= 100 || completedCount % 1000 === 0 || completedCount === collected.files.length) {
|
||||
console.log(`[database-backup] files 进度: ${completedCount}/${collected.files.length} (${result.status}) ${file.path}`);
|
||||
}
|
||||
}
|
||||
console.log(`[database-backup] files 进度: ${index + 1}/${collected.files.length} (${result.status}) ${file.path}`);
|
||||
}
|
||||
});
|
||||
await Promise.all(workers);
|
||||
const catalogUpload = await ensureDirectManifest({
|
||||
manifestPath: catalogPath,
|
||||
objectKey: catalogObjectKey,
|
||||
@@ -1478,6 +1514,7 @@ export async function runDirectFilesBackup({
|
||||
sha256: catalogUpload.archiveSha256,
|
||||
verifiedAt: catalogUpload.verifiedAt,
|
||||
files: catalog.files,
|
||||
symlinks: catalog.symlinks,
|
||||
};
|
||||
const nextState = {
|
||||
schemaVersion: DIRECT_FILES_STATE_SCHEMA_VERSION,
|
||||
@@ -1563,7 +1600,7 @@ async function loadDirectFilesCatalog({catalogRef, database, bucket, uploadOptio
|
||||
) {
|
||||
throw new Error(`files restore catalog 契约无效: ${catalogRef.objectKey}`);
|
||||
}
|
||||
return catalog;
|
||||
return {...catalog, symlinks: catalog.symlinks ?? []};
|
||||
}
|
||||
|
||||
function assertDirectCatalogFile(file, index) {
|
||||
@@ -1581,6 +1618,22 @@ function assertDirectCatalogFile(file, index) {
|
||||
}
|
||||
}
|
||||
|
||||
function assertDirectCatalogSymlink(symlink, index, restoreDir) {
|
||||
if (
|
||||
!symlink
|
||||
|| typeof symlink.path !== 'string'
|
||||
|| !symlink.path
|
||||
|| typeof symlink.target !== 'string'
|
||||
|| !symlink.target
|
||||
|| isAbsolute(symlink.target)
|
||||
) {
|
||||
throw new Error(`files restore catalog 符号链接项无效: index=${index}`);
|
||||
}
|
||||
const destinationPath = resolve(restoreDir, symlink.path);
|
||||
assertSafeRelativePath(restoreDir, destinationPath);
|
||||
assertSafeRelativePath(restoreDir, resolve(dirname(destinationPath), symlink.target));
|
||||
}
|
||||
|
||||
async function restoreDirectFilesCatalog({
|
||||
catalog,
|
||||
restoreDir,
|
||||
@@ -1591,12 +1644,14 @@ async function restoreDirectFilesCatalog({
|
||||
}) {
|
||||
const resolvedRestoreDir = resolvePath(restoreDir);
|
||||
catalog.files.forEach(assertDirectCatalogFile);
|
||||
catalog.symlinks.forEach((symlink, index) => assertDirectCatalogSymlink(symlink, index, resolvedRestoreDir));
|
||||
const totalSizeBytes = catalog.files.reduce((sum, file) => sum + BigInt(file.sizeBytes), 0n).toString();
|
||||
if (dryRun) {
|
||||
const result = {
|
||||
restoreDir: resolvedRestoreDir,
|
||||
catalogId: catalog.catalogId,
|
||||
fileCount: catalog.files.length,
|
||||
symlinkCount: catalog.symlinks.length,
|
||||
totalSizeBytes,
|
||||
downloadedCount: 0,
|
||||
reusedCount: 0,
|
||||
@@ -1644,10 +1699,19 @@ async function restoreDirectFilesCatalog({
|
||||
chmodSync(destinationPath, file.mode & 0o7777);
|
||||
console.log(`[database-backup] files restore: ${index + 1}/${catalog.files.length} (${reusable ? 'reused' : 'downloaded'}) ${file.path}`);
|
||||
}
|
||||
for (const symlink of catalog.symlinks) {
|
||||
const destinationPath = resolve(resolvedRestoreDir, symlink.path);
|
||||
assertSafeRelativePath(resolvedRestoreDir, destinationPath);
|
||||
mkdirSync(dirname(destinationPath), {recursive: true});
|
||||
rmSync(destinationPath, {recursive: true, force: true});
|
||||
symlinkSync(symlink.target, destinationPath);
|
||||
console.log(`[database-backup] files restore: symlink ${symlink.path} -> ${symlink.target}`);
|
||||
}
|
||||
const result = {
|
||||
restoreDir: resolvedRestoreDir,
|
||||
catalogId: catalog.catalogId,
|
||||
fileCount: catalog.files.length,
|
||||
symlinkCount: catalog.symlinks.length,
|
||||
totalSizeBytes,
|
||||
downloadedCount,
|
||||
reusedCount,
|
||||
@@ -2663,6 +2727,7 @@ async function main() {
|
||||
const database = firstNonEmpty(args.database, env.GENARRATIVE_SPACETIME_DATABASE, basename(dataDir));
|
||||
const keepLocal = args.keepLocal || String(env.GENARRATIVE_DATABASE_BACKUP_KEEP_LOCAL ?? '').trim().toLowerCase() === 'true';
|
||||
const storageFormat = firstNonEmpty(args.storageFormat, env.GENARRATIVE_DATABASE_BACKUP_STORAGE_FORMAT, 'archive');
|
||||
const directFilesConcurrency = parseDirectFilesConcurrency(env.GENARRATIVE_DATABASE_BACKUP_FILES_CONCURRENCY);
|
||||
|
||||
if (!['full', 'history'].includes(args.mode)) {
|
||||
throw new Error(`--mode 只能是 full 或 history,实际: ${args.mode}`);
|
||||
@@ -2752,6 +2817,7 @@ async function main() {
|
||||
dryRun: args.dryRun,
|
||||
resultFile: args.resultFile,
|
||||
uploadOptions: {bucket, endpoint, accessKeyId, accessKeySecret},
|
||||
concurrency: directFilesConcurrency,
|
||||
});
|
||||
} catch (error) {
|
||||
backupError = error;
|
||||
|
||||
Reference in New Issue
Block a user