Compare commits

...

1 Commits

Author SHA1 Message Date
kdletters a8b44f5100 修复画布历史资源空来源修复
允许资源 remap 精确匹配显式 null 的 sourceResourceId
修正 SpacetimeDB CLI procedure 的 SHA 字段与 Option wire 编码
补充 Rust 与 Node 回归测试并重新生成 Rust bindings
更新画布迁移方案和共享排障约定
2026-07-27 22:28:00 +08:00
6 changed files with 202 additions and 26 deletions
@@ -50,7 +50,7 @@
- 现象:工程、素材、图层和元数据都已禁止 Data URL 后,服务器仍在生成高峰出现 SpacetimeDB / api-server 内存急剧膨胀甚至 OOM;读取少量正式生成任务也会造成远大于响应体的瞬时内存增长。
- 原因:同步接口 worker 化时把原请求整体序列化到 `external_generation_job.request_payload_json`,而前端又把已有 `objectKey` 下载成 Data URL 提交。任务表也是正式持久化边界;列表 procedure 若先收集完整任务行再截断,还会把 request/result 大字段在 SpacetimeDB、SDK mapper 和 BFF 多次持有。
- 处理:先在事故涉及的编辑器持久任务 JSON 上由 api-server 与 SpacetimeDB 两层递归拒绝 `data:` / `blob:` 并限制字节数;已有媒体传 `objectKey` / `resourceId` / `assetId`,本地派生图先用强唯一 key 上传。其它玩法若仍以 Data URL 作为正式请求契约,必须先资源化,不能直接扩大门禁造成玩法回归。列表、详情和 acknowledge 只走无 payload 的摘要投影,ack 不能为了同步旧字段重写大任务行;摘要错误文本也必须清除内联媒体并设硬上限,列表只能维护有界 top-N,不能先收集 owner 全量历史再截断。历史只通过迁移操作员的 dry-run + B-tree cursor 分批 procedure 压缩 `editor-canvas` 终态任务,cursor 选择读取量必须受 limit 约束,绝不全表扫描、绝不处理 pending / runningdry-run 后 apply 同一批时保持输入 cursor 不变,最后一批即使 `has_more=false` 只要仍有命中也必须 apply,只有 apply 成功后才推进到返回 cursor。SpacetimeDB CLI 2.5 的 `Option<T>` 非空参数必须使用 SATS sum 编码;维护脚本要统一编码 `cursor_job_id``owner_user_id``completed_before_micros`,否则首批空 cursor 可运行,但第二批或带截止时间的调用会在写入前被拒绝。
- 处理:先在事故涉及的编辑器持久任务 JSON 上由 api-server 与 SpacetimeDB 两层递归拒绝 `data:` / `blob:` 并限制字节数;已有媒体传 `objectKey` / `resourceId` / `assetId`,本地派生图先用强唯一 key 上传。其它玩法若仍以 Data URL 作为正式请求契约,必须先资源化,不能直接扩大门禁造成玩法回归。列表、详情和 acknowledge 只走无 payload 的摘要投影,ack 不能为了同步旧字段重写大任务行;摘要错误文本也必须清除内联媒体并设硬上限,列表只能维护有界 top-N,不能先收集 owner 全量历史再截断。历史只通过迁移操作员的 dry-run + B-tree cursor 分批 procedure 压缩 `editor-canvas` 终态任务,cursor 选择读取量必须受 limit 约束,绝不全表扫描、绝不处理 pending / runningdry-run 后 apply 同一批时保持输入 cursor 不变,最后一批即使 `has_more=false` 只要仍有命中也必须 apply,只有 apply 成功后才推进到返回 cursor。SpacetimeDB CLI 2.5 的 `Option<T>` 非空参数必须使用 SATS sum 编码;维护脚本要统一编码 `cursor_job_id``owner_user_id``completed_before_micros`,否则首批空 cursor 可运行,但第二批或带截止时间的调用会在写入前被拒绝。运维 plan 可以保留人工契约字段,但组装 CLI procedure input 时必须按生成 schema 的 wire 名称映射;例如 Rust 字段 `expected_canvas_layout_sha256` 的 CLI wire 名为 `expected_canvas_layout_sha_256`,不能直接透传 plan key。
- 发布门禁:生产发布入口必须固定 `--delete-data=never` 与 scoped `--yes=migrate,break-clients`,普通 Jenkins 参数不得暴露清库开关;需要删数据的 schema 冲突必须直接阻断并重新检查 artifact/schema,不能靠裸 `--yes` 放行。
- 验证:构造嵌套 Data URL、Blob URL 和超限 JSON 确认入队失败;检查正式 UI procedure / client record 不含 request/result payload;用 dry-run 和 apply 测试确认活动任务不变、终态普通提示词保留且内联媒体被替换;至少带一次非空 `--cursor-job-id``--completed-before-micros` 验证 CLI Option 编码,而不是只测首批空 cursor。
- 关联:`server-rs/crates/api-server/src/editor_generation_queue.rs``server-rs/crates/spacetime-module/src/external_generation.rs``server-rs/crates/api-server/src/external_generation.rs``src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts``docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`
@@ -73,7 +73,7 @@ worker 完成生成任务时,本次先用读取时 revision 调用 CAS 保存
若全量 backfill 审计发现普通图层缺少 `editor_project_resource`,先使用定向资源修复入口:`npm run spacetime:editor-canvas-resources:repair -- --database <db> --server <server> --plan-file </absolute/outside-repo/repair-plan.json>`。plan 必须是仓库外、当前用户持有、权限严格为 `0600` 的普通文件;每个 canvas 绑定 owner / project、expected revision、`editor_canvas.layers_json` 与 legacy `editor_project.layers_json` 两份原始字节 SHA-256,以及精确 layer/resource/sourceResourceId 或 asset_object 修复动作。默认逐 canvas dry-runapply 必须追加 `--apply --confirm-plan-sha256 <dry-run 输出>`,成功后脚本自动以同一 plan 再 dry-run,并要求全部返回 `already_repaired`
`repair_editor_canvas_resources_and_return` 只允许 migration operator 调用,并且只修尚无迁移记录的 legacy canvas。图片动作仅在缺失旧 resourceId、精确旧 sourceResourceId、同工程唯一现存资源、layout 资源元数据与 private asset_object owner/objectKey/task 谱系全部一致时,替换 `resourceId` 并删除重复 `sourceResourceId`。音频动作仅在原 resourceId 全局不存在、图层与 private asset_object 的 owner/objectKey/source job/entity/content type/长度全部匹配时,恢复 `420x120``sound-effect` / `background-music` 项目资源行;资源 `asset_kind` 分别映射 asset_object 的 `editor_sound_effect` / `editor_background_music`。apply 在一个事务内插入资源、同步两份 legacy layout、递增一次 canvas revision;任一 guard 失败整画布回滚。plan、脚本输出、测试和文档均不得包含生产真实 ID。
`repair_editor_canvas_resources_and_return` 只允许 migration operator 调用,并且只修尚无迁移记录的 legacy canvas。图片动作仅在缺失旧 resourceId、精确旧 sourceResourceId、同工程唯一现存资源、layout 资源元数据与 private asset_object owner/objectKey/task 谱系全部一致时,替换 `resourceId` 并删除重复 `sourceResourceId`plan 中 `expected_layout_source_resource_id` 为字符串时必须与 layout 原值完全一致,显式为 `null` 时则要求 layout 中该字段明确存在且值为 JSON `null`,不得把字段缺失与 `null` 混同。音频动作仅在原 resourceId 全局不存在、图层与 private asset_object 的 owner/objectKey/source job/entity/content type/长度全部匹配时,恢复 `420x120``sound-effect` / `background-music` 项目资源行;资源 `asset_kind` 分别映射 asset_object 的 `editor_sound_effect` / `editor_background_music`。apply 在一个事务内插入资源、同步两份 legacy layout、递增一次 canvas revision;任一 guard 失败整画布回滚。plan、脚本输出、测试和文档均不得包含生产真实 ID。
## 5. 兼容与回滚
@@ -7,6 +7,7 @@ import { fileURLToPath } from 'node:url';
import {
callSpacetimeProcedureViaCli,
encodeSpacetimeCliOption,
} from './spacetime-migration-common.mjs';
const PROCEDURE_NAME = 'repair_editor_canvas_resources_and_return';
@@ -304,11 +305,6 @@ function validateRepairAction(action, label) {
`${label}.replacement_resource_id 为复用资源动作的必填项。`,
);
}
if (!normalized.expected_layout_source_resource_id) {
throw new Error(
`${label}.expected_layout_source_resource_id 为复用资源动作的必填项。`,
);
}
if (
normalized.replacement_resource_id === normalized.expected_resource_id
) {
@@ -327,7 +323,7 @@ function validateRepairAction(action, label) {
`${label} 的补资源动作不能携带 replacement_resource_id。`,
);
}
if (normalized.expected_layout_source_resource_id) {
if (normalized.expected_layout_source_resource_id !== null) {
throw new Error(
`${label} 的补资源动作不能携带 expected_layout_source_resource_id。`,
);
@@ -415,16 +411,17 @@ export function buildProcedureInput(canvas, updatedAtMicros, dryRun) {
project_id: canvas.project_id,
owner_user_id: canvas.owner_user_id,
expected_revision: canvas.expected_revision,
expected_canvas_layout_sha256: canvas.expected_canvas_layout_sha256,
expected_project_layout_sha256: canvas.expected_project_layout_sha256,
expected_canvas_layout_sha_256: canvas.expected_canvas_layout_sha256,
expected_project_layout_sha_256: canvas.expected_project_layout_sha256,
remaps: canvas.actions
.filter((action) => action.action_kind === 'reuse_project_resource')
.map((action) => ({
layer_id: action.layer_id,
expected_missing_resource_id: action.expected_resource_id,
replacement_resource_id: action.replacement_resource_id,
expected_layout_source_resource_id:
expected_layout_source_resource_id: encodeSpacetimeCliOption(
action.expected_layout_source_resource_id,
),
})),
restores: canvas.actions
.filter((action) => action.action_kind === 'restore_project_resource')
@@ -166,6 +166,39 @@ describe('spacetime editor canvas resource repair plan', () => {
]);
});
it('accepts an explicit null layout source only for reuse actions', () => {
const plan = validateRepairPlan(
planValue([
canvas({
actions: [
action({ expected_layout_source_resource_id: null }),
],
}),
]),
);
expect(
plan.canvases[0].actions[0].expected_layout_source_resource_id,
).toBeNull();
expect(() =>
validateRepairPlan(
planValue([
canvas({
actions: [
action({
action_kind: 'restore_project_resource',
replacement_resource_id: null,
expected_layout_source_resource_id: 'unexpected-source',
asset_object_id: 'private-asset-object-id',
asset_kind: 'sound-effect',
}),
],
}),
]),
),
).toThrow('不能携带 expected_layout_source_resource_id');
});
it('rejects count mismatches, duplicate layers, and invalid kind-specific fields', () => {
expect(() =>
validateRepairPlan({ ...planValue(), expected_action_count: 2 }),
@@ -263,7 +296,10 @@ describe('spacetime editor canvas resource repair execution', () => {
layer_id: 'private-layer-id',
expected_missing_resource_id: 'missing-private-resource-id',
replacement_resource_id: 'replacement-private-resource-id',
expected_layout_source_resource_id: 'source-private-resource-id',
expected_layout_source_resource_id: [
0,
'source-private-resource-id',
],
},
]);
expect(input.restores).toEqual([
@@ -275,10 +311,37 @@ describe('spacetime editor canvas resource repair execution', () => {
},
]);
expect(input.canvas_id).toBe('private-canvas-id');
expect(input.expected_canvas_layout_sha_256).toBe(SHA_A);
expect(input.expected_project_layout_sha_256).toBe(SHA_B);
expect(input).not.toHaveProperty('expected_canvas_layout_sha256');
expect(input).not.toHaveProperty('expected_project_layout_sha256');
expect(input.updated_at_micros).toBe(123);
expect(input.dry_run).toBe(true);
});
it('encodes an explicit null expected layout source as a CLI Option null', () => {
const plan = validateRepairPlan(
planValue([
canvas({
actions: [
action({ expected_layout_source_resource_id: null }),
],
}),
]),
);
const input = buildProcedureInput(plan.canvases[0], 123, true);
expect(input.remaps).toEqual([
{
layer_id: 'private-layer-id',
expected_missing_resource_id: 'missing-private-resource-id',
replacement_resource_id: 'replacement-private-resource-id',
expected_layout_source_resource_id: null,
},
]);
});
it('calls one procedure per canvas and emits only anonymous aggregate output', async () => {
const plan = validateRepairPlan(
planValue([
@@ -10,7 +10,7 @@ pub struct EditorCanvasResourceRemapInput {
pub layer_id: String,
pub expected_missing_resource_id: String,
pub replacement_resource_id: String,
pub expected_layout_source_resource_id: String,
pub expected_layout_source_resource_id: Option<String>,
}
impl __sdk::InModule for EditorCanvasResourceRemapInput {
@@ -514,7 +514,7 @@ pub struct EditorCanvasResourceRemapInput {
pub layer_id: String,
pub expected_missing_resource_id: String,
pub replacement_resource_id: String,
pub expected_layout_source_resource_id: String,
pub expected_layout_source_resource_id: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, SpacetimeType)]
@@ -6669,10 +6669,13 @@ fn validate_editor_canvas_resource_repair_actions(
&action.replacement_resource_id,
"remap.replacement_resource_id",
)?;
normalize_required(
&action.expected_layout_source_resource_id,
"remap.expected_layout_source_resource_id",
)?;
if let Some(expected_layout_source_resource_id) = &action.expected_layout_source_resource_id
{
normalize_required(
expected_layout_source_resource_id,
"remap.expected_layout_source_resource_id",
)?;
}
if missing == replacement {
return Err("资源 remap 的缺失 ID 与替换 ID 不能相同".to_string());
}
@@ -6761,10 +6764,21 @@ fn patch_editor_canvas_resource_layout(
if current_resource_id != action.expected_missing_resource_id {
return Err("图片图层的原 resourceId 与修复计划不一致".to_string());
}
let source_resource_id =
json_required_string(object.get("sourceResourceId"), "sourceResourceId")?;
if source_resource_id != action.expected_layout_source_resource_id {
return Err("图片图层的 sourceResourceId 与修复计划不一致".to_string());
match &action.expected_layout_source_resource_id {
Some(expected_source_resource_id) => {
let source_resource_id =
json_required_string(object.get("sourceResourceId"), "sourceResourceId")?;
if source_resource_id != expected_source_resource_id.as_str() {
return Err("图片图层的 sourceResourceId 与修复计划不一致".to_string());
}
}
None => {
if object.get("sourceResourceId") != Some(&JsonValue::Null) {
return Err(
"图片图层的 sourceResourceId 必须明确为 null 且与修复计划一致".to_string(),
);
}
}
}
object.insert(
"resourceId".to_string(),
@@ -8040,7 +8054,7 @@ mod tests {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: "resource-source-old".to_string(),
expected_layout_source_resource_id: Some("resource-source-old".to_string()),
}];
let patch = patch_editor_canvas_resource_layout(
@@ -8074,7 +8088,7 @@ mod tests {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: "expected-source".to_string(),
expected_layout_source_resource_id: Some("expected-source".to_string()),
}];
let error = patch_editor_canvas_resource_layout(
@@ -8088,6 +8102,93 @@ mod tests {
assert!(error.contains("sourceResourceId"));
}
#[test]
fn editor_canvas_resource_repair_accepts_explicit_null_source_resource_id() {
let layout = json!([{
"layerId": "layer-target",
"resourceId": "resource-missing",
"sourceResourceId": null,
"prompt": "保留提示词"
}]);
let remaps = vec![EditorCanvasResourceRemapInput {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: None,
}];
let patch = patch_editor_canvas_resource_layout(
layout.to_string().as_str(),
remaps.as_slice(),
&[],
false,
)
.expect("explicit null source resource should match null repair plan");
let repaired: JsonValue =
serde_json::from_str(&patch.layers_json).expect("repaired layout json");
assert_eq!(repaired[0]["resourceId"], json!("resource-existing"));
assert!(repaired[0].get("sourceResourceId").is_none());
assert_eq!(repaired[0]["prompt"], json!("保留提示词"));
assert_eq!(patch.remapped_layer_count, 1);
assert_eq!(patch.removed_source_resource_id_count, 1);
}
#[test]
fn editor_canvas_resource_repair_null_plan_rejects_missing_or_string_source_resource_id() {
let remaps = vec![EditorCanvasResourceRemapInput {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: None,
}];
let missing_field = json!([{
"layerId": "layer-target",
"resourceId": "resource-missing"
}]);
let string_field = json!([{
"layerId": "layer-target",
"resourceId": "resource-missing",
"sourceResourceId": "source-old"
}]);
for layout in [missing_field, string_field] {
let error = patch_editor_canvas_resource_layout(
layout.to_string().as_str(),
remaps.as_slice(),
&[],
false,
)
.expect_err("null plan must only match an explicitly null sourceResourceId");
assert!(error.contains("明确为 null"));
}
}
#[test]
fn editor_canvas_resource_repair_string_plan_rejects_null_source_resource_id() {
let layout = json!([{
"layerId": "layer-target",
"resourceId": "resource-missing",
"sourceResourceId": null
}]);
let remaps = vec![EditorCanvasResourceRemapInput {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: Some("source-old".to_string()),
}];
let error = patch_editor_canvas_resource_layout(
layout.to_string().as_str(),
remaps.as_slice(),
&[],
false,
)
.expect_err("string plan must reject a null sourceResourceId");
assert!(error.contains("sourceResourceId"));
}
#[test]
fn editor_canvas_resource_repair_recognizes_only_exact_applied_state() {
let repaired = json!([{
@@ -8099,7 +8200,7 @@ mod tests {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: "resource-source-old".to_string(),
expected_layout_source_resource_id: Some("resource-source-old".to_string()),
}];
let patch = patch_editor_canvas_resource_layout(
@@ -8126,6 +8227,21 @@ mod tests {
)
.is_err()
);
let unsafe_null_state = json!([{
"layerId": "layer-target",
"resourceId": "resource-existing",
"sourceResourceId": null
}]);
assert!(
patch_editor_canvas_resource_layout(
unsafe_null_state.to_string().as_str(),
remaps.as_slice(),
&[],
true,
)
.is_err()
);
}
#[test]
@@ -8169,7 +8285,7 @@ mod tests {
layer_id: "layer-target".to_string(),
expected_missing_resource_id: "resource-missing".to_string(),
replacement_resource_id: "resource-existing".to_string(),
expected_layout_source_resource_id: "source-old".to_string(),
expected_layout_source_resource_id: Some("source-old".to_string()),
}];
let error = patch_editor_canvas_resource_layout(