Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a8b44f5100 |
@@ -50,7 +50,7 @@
|
||||
|
||||
- 现象:工程、素材、图层和元数据都已禁止 Data URL 后,服务器仍在生成高峰出现 SpacetimeDB / api-server 内存急剧膨胀甚至 OOM;读取少量正式生成任务也会造成远大于响应体的瞬时内存增长。
|
||||
- 原因:同步接口 worker 化时把原请求整体序列化到 `external_generation_job.request_payload_json`,而前端又把已有 `objectKey` 下载成 Data URL 提交。任务表也是正式持久化边界;列表 procedure 若先收集完整任务行再截断,还会把 request/result 大字段在 SpacetimeDB、SDK mapper 和 BFF 多次持有。
|
||||
- 处理:先在事故涉及的编辑器持久任务 JSON 上由 api-server 与 SpacetimeDB 两层递归拒绝 `data:` / `blob:` 并限制字节数;已有媒体传 `objectKey` / `resourceId` / `assetId`,本地派生图先用强唯一 key 上传。其它玩法若仍以 Data URL 作为正式请求契约,必须先资源化,不能直接扩大门禁造成玩法回归。列表、详情和 acknowledge 只走无 payload 的摘要投影,ack 不能为了同步旧字段重写大任务行;摘要错误文本也必须清除内联媒体并设硬上限,列表只能维护有界 top-N,不能先收集 owner 全量历史再截断。历史只通过迁移操作员的 dry-run + B-tree cursor 分批 procedure 压缩 `editor-canvas` 终态任务,cursor 选择读取量必须受 limit 约束,绝不全表扫描、绝不处理 pending / running;dry-run 后 apply 同一批时保持输入 cursor 不变,最后一批即使 `has_more=false` 只要仍有命中也必须 apply,只有 apply 成功后才推进到返回 cursor。SpacetimeDB CLI 2.5 的 `Option<T>` 非空参数必须使用 SATS sum 编码;维护脚本要统一编码 `cursor_job_id`、`owner_user_id` 和 `completed_before_micros`,否则首批空 cursor 可运行,但第二批或带截止时间的调用会在写入前被拒绝。
|
||||
- 处理:先在事故涉及的编辑器持久任务 JSON 上由 api-server 与 SpacetimeDB 两层递归拒绝 `data:` / `blob:` 并限制字节数;已有媒体传 `objectKey` / `resourceId` / `assetId`,本地派生图先用强唯一 key 上传。其它玩法若仍以 Data URL 作为正式请求契约,必须先资源化,不能直接扩大门禁造成玩法回归。列表、详情和 acknowledge 只走无 payload 的摘要投影,ack 不能为了同步旧字段重写大任务行;摘要错误文本也必须清除内联媒体并设硬上限,列表只能维护有界 top-N,不能先收集 owner 全量历史再截断。历史只通过迁移操作员的 dry-run + B-tree cursor 分批 procedure 压缩 `editor-canvas` 终态任务,cursor 选择读取量必须受 limit 约束,绝不全表扫描、绝不处理 pending / running;dry-run 后 apply 同一批时保持输入 cursor 不变,最后一批即使 `has_more=false` 只要仍有命中也必须 apply,只有 apply 成功后才推进到返回 cursor。SpacetimeDB CLI 2.5 的 `Option<T>` 非空参数必须使用 SATS sum 编码;维护脚本要统一编码 `cursor_job_id`、`owner_user_id` 和 `completed_before_micros`,否则首批空 cursor 可运行,但第二批或带截止时间的调用会在写入前被拒绝。运维 plan 可以保留人工契约字段,但组装 CLI procedure input 时必须按生成 schema 的 wire 名称映射;例如 Rust 字段 `expected_canvas_layout_sha256` 的 CLI wire 名为 `expected_canvas_layout_sha_256`,不能直接透传 plan key。
|
||||
- 发布门禁:生产发布入口必须固定 `--delete-data=never` 与 scoped `--yes=migrate,break-clients`,普通 Jenkins 参数不得暴露清库开关;需要删数据的 schema 冲突必须直接阻断并重新检查 artifact/schema,不能靠裸 `--yes` 放行。
|
||||
- 验证:构造嵌套 Data URL、Blob URL 和超限 JSON 确认入队失败;检查正式 UI procedure / client record 不含 request/result payload;用 dry-run 和 apply 测试确认活动任务不变、终态普通提示词保留且内联媒体被替换;至少带一次非空 `--cursor-job-id` 与 `--completed-before-micros` 验证 CLI Option 编码,而不是只测首批空 cursor。
|
||||
- 关联:`server-rs/crates/api-server/src/editor_generation_queue.rs`、`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/api-server/src/external_generation.rs`、`src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts`、`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`。
|
||||
|
||||
@@ -73,7 +73,7 @@ worker 完成生成任务时,本次先用读取时 revision 调用 CAS 保存
|
||||
|
||||
若全量 backfill 审计发现普通图层缺少 `editor_project_resource`,先使用定向资源修复入口:`npm run spacetime:editor-canvas-resources:repair -- --database <db> --server <server> --plan-file </absolute/outside-repo/repair-plan.json>`。plan 必须是仓库外、当前用户持有、权限严格为 `0600` 的普通文件;每个 canvas 绑定 owner / project、expected revision、`editor_canvas.layers_json` 与 legacy `editor_project.layers_json` 两份原始字节 SHA-256,以及精确 layer/resource/sourceResourceId 或 asset_object 修复动作。默认逐 canvas dry-run;apply 必须追加 `--apply --confirm-plan-sha256 <dry-run 输出>`,成功后脚本自动以同一 plan 再 dry-run,并要求全部返回 `already_repaired`。
|
||||
|
||||
`repair_editor_canvas_resources_and_return` 只允许 migration operator 调用,并且只修尚无迁移记录的 legacy canvas。图片动作仅在缺失旧 resourceId、精确旧 sourceResourceId、同工程唯一现存资源、layout 资源元数据与 private asset_object owner/objectKey/task 谱系全部一致时,替换 `resourceId` 并删除重复 `sourceResourceId`。音频动作仅在原 resourceId 全局不存在、图层与 private asset_object 的 owner/objectKey/source job/entity/content type/长度全部匹配时,恢复 `420x120` 的 `sound-effect` / `background-music` 项目资源行;资源 `asset_kind` 分别映射 asset_object 的 `editor_sound_effect` / `editor_background_music`。apply 在一个事务内插入资源、同步两份 legacy layout、递增一次 canvas revision;任一 guard 失败整画布回滚。plan、脚本输出、测试和文档均不得包含生产真实 ID。
|
||||
`repair_editor_canvas_resources_and_return` 只允许 migration operator 调用,并且只修尚无迁移记录的 legacy canvas。图片动作仅在缺失旧 resourceId、精确旧 sourceResourceId、同工程唯一现存资源、layout 资源元数据与 private asset_object owner/objectKey/task 谱系全部一致时,替换 `resourceId` 并删除重复 `sourceResourceId`;plan 中 `expected_layout_source_resource_id` 为字符串时必须与 layout 原值完全一致,显式为 `null` 时则要求 layout 中该字段明确存在且值为 JSON `null`,不得把字段缺失与 `null` 混同。音频动作仅在原 resourceId 全局不存在、图层与 private asset_object 的 owner/objectKey/source job/entity/content type/长度全部匹配时,恢复 `420x120` 的 `sound-effect` / `background-music` 项目资源行;资源 `asset_kind` 分别映射 asset_object 的 `editor_sound_effect` / `editor_background_music`。apply 在一个事务内插入资源、同步两份 legacy layout、递增一次 canvas revision;任一 guard 失败整画布回滚。plan、脚本输出、测试和文档均不得包含生产真实 ID。
|
||||
|
||||
## 5. 兼容与回滚
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import { fileURLToPath } from 'node:url';
|
||||
|
||||
import {
|
||||
callSpacetimeProcedureViaCli,
|
||||
encodeSpacetimeCliOption,
|
||||
} from './spacetime-migration-common.mjs';
|
||||
|
||||
const PROCEDURE_NAME = 'repair_editor_canvas_resources_and_return';
|
||||
@@ -304,11 +305,6 @@ function validateRepairAction(action, label) {
|
||||
`${label}.replacement_resource_id 为复用资源动作的必填项。`,
|
||||
);
|
||||
}
|
||||
if (!normalized.expected_layout_source_resource_id) {
|
||||
throw new Error(
|
||||
`${label}.expected_layout_source_resource_id 为复用资源动作的必填项。`,
|
||||
);
|
||||
}
|
||||
if (
|
||||
normalized.replacement_resource_id === normalized.expected_resource_id
|
||||
) {
|
||||
@@ -327,7 +323,7 @@ function validateRepairAction(action, label) {
|
||||
`${label} 的补资源动作不能携带 replacement_resource_id。`,
|
||||
);
|
||||
}
|
||||
if (normalized.expected_layout_source_resource_id) {
|
||||
if (normalized.expected_layout_source_resource_id !== null) {
|
||||
throw new Error(
|
||||
`${label} 的补资源动作不能携带 expected_layout_source_resource_id。`,
|
||||
);
|
||||
@@ -415,16 +411,17 @@ export function buildProcedureInput(canvas, updatedAtMicros, dryRun) {
|
||||
project_id: canvas.project_id,
|
||||
owner_user_id: canvas.owner_user_id,
|
||||
expected_revision: canvas.expected_revision,
|
||||
expected_canvas_layout_sha256: canvas.expected_canvas_layout_sha256,
|
||||
expected_project_layout_sha256: canvas.expected_project_layout_sha256,
|
||||
expected_canvas_layout_sha_256: canvas.expected_canvas_layout_sha256,
|
||||
expected_project_layout_sha_256: canvas.expected_project_layout_sha256,
|
||||
remaps: canvas.actions
|
||||
.filter((action) => action.action_kind === 'reuse_project_resource')
|
||||
.map((action) => ({
|
||||
layer_id: action.layer_id,
|
||||
expected_missing_resource_id: action.expected_resource_id,
|
||||
replacement_resource_id: action.replacement_resource_id,
|
||||
expected_layout_source_resource_id:
|
||||
expected_layout_source_resource_id: encodeSpacetimeCliOption(
|
||||
action.expected_layout_source_resource_id,
|
||||
),
|
||||
})),
|
||||
restores: canvas.actions
|
||||
.filter((action) => action.action_kind === 'restore_project_resource')
|
||||
|
||||
@@ -166,6 +166,39 @@ describe('spacetime editor canvas resource repair plan', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it('accepts an explicit null layout source only for reuse actions', () => {
|
||||
const plan = validateRepairPlan(
|
||||
planValue([
|
||||
canvas({
|
||||
actions: [
|
||||
action({ expected_layout_source_resource_id: null }),
|
||||
],
|
||||
}),
|
||||
]),
|
||||
);
|
||||
|
||||
expect(
|
||||
plan.canvases[0].actions[0].expected_layout_source_resource_id,
|
||||
).toBeNull();
|
||||
expect(() =>
|
||||
validateRepairPlan(
|
||||
planValue([
|
||||
canvas({
|
||||
actions: [
|
||||
action({
|
||||
action_kind: 'restore_project_resource',
|
||||
replacement_resource_id: null,
|
||||
expected_layout_source_resource_id: 'unexpected-source',
|
||||
asset_object_id: 'private-asset-object-id',
|
||||
asset_kind: 'sound-effect',
|
||||
}),
|
||||
],
|
||||
}),
|
||||
]),
|
||||
),
|
||||
).toThrow('不能携带 expected_layout_source_resource_id');
|
||||
});
|
||||
|
||||
it('rejects count mismatches, duplicate layers, and invalid kind-specific fields', () => {
|
||||
expect(() =>
|
||||
validateRepairPlan({ ...planValue(), expected_action_count: 2 }),
|
||||
@@ -263,7 +296,10 @@ describe('spacetime editor canvas resource repair execution', () => {
|
||||
layer_id: 'private-layer-id',
|
||||
expected_missing_resource_id: 'missing-private-resource-id',
|
||||
replacement_resource_id: 'replacement-private-resource-id',
|
||||
expected_layout_source_resource_id: 'source-private-resource-id',
|
||||
expected_layout_source_resource_id: [
|
||||
0,
|
||||
'source-private-resource-id',
|
||||
],
|
||||
},
|
||||
]);
|
||||
expect(input.restores).toEqual([
|
||||
@@ -275,10 +311,37 @@ describe('spacetime editor canvas resource repair execution', () => {
|
||||
},
|
||||
]);
|
||||
expect(input.canvas_id).toBe('private-canvas-id');
|
||||
expect(input.expected_canvas_layout_sha_256).toBe(SHA_A);
|
||||
expect(input.expected_project_layout_sha_256).toBe(SHA_B);
|
||||
expect(input).not.toHaveProperty('expected_canvas_layout_sha256');
|
||||
expect(input).not.toHaveProperty('expected_project_layout_sha256');
|
||||
expect(input.updated_at_micros).toBe(123);
|
||||
expect(input.dry_run).toBe(true);
|
||||
});
|
||||
|
||||
it('encodes an explicit null expected layout source as a CLI Option null', () => {
|
||||
const plan = validateRepairPlan(
|
||||
planValue([
|
||||
canvas({
|
||||
actions: [
|
||||
action({ expected_layout_source_resource_id: null }),
|
||||
],
|
||||
}),
|
||||
]),
|
||||
);
|
||||
|
||||
const input = buildProcedureInput(plan.canvases[0], 123, true);
|
||||
|
||||
expect(input.remaps).toEqual([
|
||||
{
|
||||
layer_id: 'private-layer-id',
|
||||
expected_missing_resource_id: 'missing-private-resource-id',
|
||||
replacement_resource_id: 'replacement-private-resource-id',
|
||||
expected_layout_source_resource_id: null,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it('calls one procedure per canvas and emits only anonymous aggregate output', async () => {
|
||||
const plan = validateRepairPlan(
|
||||
planValue([
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ pub struct EditorCanvasResourceRemapInput {
|
||||
pub layer_id: String,
|
||||
pub expected_missing_resource_id: String,
|
||||
pub replacement_resource_id: String,
|
||||
pub expected_layout_source_resource_id: String,
|
||||
pub expected_layout_source_resource_id: Option<String>,
|
||||
}
|
||||
|
||||
impl __sdk::InModule for EditorCanvasResourceRemapInput {
|
||||
|
||||
@@ -514,7 +514,7 @@ pub struct EditorCanvasResourceRemapInput {
|
||||
pub layer_id: String,
|
||||
pub expected_missing_resource_id: String,
|
||||
pub replacement_resource_id: String,
|
||||
pub expected_layout_source_resource_id: String,
|
||||
pub expected_layout_source_resource_id: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, SpacetimeType)]
|
||||
@@ -6669,10 +6669,13 @@ fn validate_editor_canvas_resource_repair_actions(
|
||||
&action.replacement_resource_id,
|
||||
"remap.replacement_resource_id",
|
||||
)?;
|
||||
normalize_required(
|
||||
&action.expected_layout_source_resource_id,
|
||||
"remap.expected_layout_source_resource_id",
|
||||
)?;
|
||||
if let Some(expected_layout_source_resource_id) = &action.expected_layout_source_resource_id
|
||||
{
|
||||
normalize_required(
|
||||
expected_layout_source_resource_id,
|
||||
"remap.expected_layout_source_resource_id",
|
||||
)?;
|
||||
}
|
||||
if missing == replacement {
|
||||
return Err("资源 remap 的缺失 ID 与替换 ID 不能相同".to_string());
|
||||
}
|
||||
@@ -6761,10 +6764,21 @@ fn patch_editor_canvas_resource_layout(
|
||||
if current_resource_id != action.expected_missing_resource_id {
|
||||
return Err("图片图层的原 resourceId 与修复计划不一致".to_string());
|
||||
}
|
||||
let source_resource_id =
|
||||
json_required_string(object.get("sourceResourceId"), "sourceResourceId")?;
|
||||
if source_resource_id != action.expected_layout_source_resource_id {
|
||||
return Err("图片图层的 sourceResourceId 与修复计划不一致".to_string());
|
||||
match &action.expected_layout_source_resource_id {
|
||||
Some(expected_source_resource_id) => {
|
||||
let source_resource_id =
|
||||
json_required_string(object.get("sourceResourceId"), "sourceResourceId")?;
|
||||
if source_resource_id != expected_source_resource_id.as_str() {
|
||||
return Err("图片图层的 sourceResourceId 与修复计划不一致".to_string());
|
||||
}
|
||||
}
|
||||
None => {
|
||||
if object.get("sourceResourceId") != Some(&JsonValue::Null) {
|
||||
return Err(
|
||||
"图片图层的 sourceResourceId 必须明确为 null 且与修复计划一致".to_string(),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
object.insert(
|
||||
"resourceId".to_string(),
|
||||
@@ -8040,7 +8054,7 @@ mod tests {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: "resource-source-old".to_string(),
|
||||
expected_layout_source_resource_id: Some("resource-source-old".to_string()),
|
||||
}];
|
||||
|
||||
let patch = patch_editor_canvas_resource_layout(
|
||||
@@ -8074,7 +8088,7 @@ mod tests {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: "expected-source".to_string(),
|
||||
expected_layout_source_resource_id: Some("expected-source".to_string()),
|
||||
}];
|
||||
|
||||
let error = patch_editor_canvas_resource_layout(
|
||||
@@ -8088,6 +8102,93 @@ mod tests {
|
||||
assert!(error.contains("sourceResourceId"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_canvas_resource_repair_accepts_explicit_null_source_resource_id() {
|
||||
let layout = json!([{
|
||||
"layerId": "layer-target",
|
||||
"resourceId": "resource-missing",
|
||||
"sourceResourceId": null,
|
||||
"prompt": "保留提示词"
|
||||
}]);
|
||||
let remaps = vec![EditorCanvasResourceRemapInput {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: None,
|
||||
}];
|
||||
|
||||
let patch = patch_editor_canvas_resource_layout(
|
||||
layout.to_string().as_str(),
|
||||
remaps.as_slice(),
|
||||
&[],
|
||||
false,
|
||||
)
|
||||
.expect("explicit null source resource should match null repair plan");
|
||||
let repaired: JsonValue =
|
||||
serde_json::from_str(&patch.layers_json).expect("repaired layout json");
|
||||
|
||||
assert_eq!(repaired[0]["resourceId"], json!("resource-existing"));
|
||||
assert!(repaired[0].get("sourceResourceId").is_none());
|
||||
assert_eq!(repaired[0]["prompt"], json!("保留提示词"));
|
||||
assert_eq!(patch.remapped_layer_count, 1);
|
||||
assert_eq!(patch.removed_source_resource_id_count, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_canvas_resource_repair_null_plan_rejects_missing_or_string_source_resource_id() {
|
||||
let remaps = vec![EditorCanvasResourceRemapInput {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: None,
|
||||
}];
|
||||
let missing_field = json!([{
|
||||
"layerId": "layer-target",
|
||||
"resourceId": "resource-missing"
|
||||
}]);
|
||||
let string_field = json!([{
|
||||
"layerId": "layer-target",
|
||||
"resourceId": "resource-missing",
|
||||
"sourceResourceId": "source-old"
|
||||
}]);
|
||||
|
||||
for layout in [missing_field, string_field] {
|
||||
let error = patch_editor_canvas_resource_layout(
|
||||
layout.to_string().as_str(),
|
||||
remaps.as_slice(),
|
||||
&[],
|
||||
false,
|
||||
)
|
||||
.expect_err("null plan must only match an explicitly null sourceResourceId");
|
||||
assert!(error.contains("明确为 null"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_canvas_resource_repair_string_plan_rejects_null_source_resource_id() {
|
||||
let layout = json!([{
|
||||
"layerId": "layer-target",
|
||||
"resourceId": "resource-missing",
|
||||
"sourceResourceId": null
|
||||
}]);
|
||||
let remaps = vec![EditorCanvasResourceRemapInput {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: Some("source-old".to_string()),
|
||||
}];
|
||||
|
||||
let error = patch_editor_canvas_resource_layout(
|
||||
layout.to_string().as_str(),
|
||||
remaps.as_slice(),
|
||||
&[],
|
||||
false,
|
||||
)
|
||||
.expect_err("string plan must reject a null sourceResourceId");
|
||||
|
||||
assert!(error.contains("sourceResourceId"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_canvas_resource_repair_recognizes_only_exact_applied_state() {
|
||||
let repaired = json!([{
|
||||
@@ -8099,7 +8200,7 @@ mod tests {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: "resource-source-old".to_string(),
|
||||
expected_layout_source_resource_id: Some("resource-source-old".to_string()),
|
||||
}];
|
||||
|
||||
let patch = patch_editor_canvas_resource_layout(
|
||||
@@ -8126,6 +8227,21 @@ mod tests {
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
|
||||
let unsafe_null_state = json!([{
|
||||
"layerId": "layer-target",
|
||||
"resourceId": "resource-existing",
|
||||
"sourceResourceId": null
|
||||
}]);
|
||||
assert!(
|
||||
patch_editor_canvas_resource_layout(
|
||||
unsafe_null_state.to_string().as_str(),
|
||||
remaps.as_slice(),
|
||||
&[],
|
||||
true,
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -8169,7 +8285,7 @@ mod tests {
|
||||
layer_id: "layer-target".to_string(),
|
||||
expected_missing_resource_id: "resource-missing".to_string(),
|
||||
replacement_resource_id: "resource-existing".to_string(),
|
||||
expected_layout_source_resource_id: "source-old".to_string(),
|
||||
expected_layout_source_resource_id: Some("source-old".to_string()),
|
||||
}];
|
||||
|
||||
let error = patch_editor_canvas_resource_layout(
|
||||
|
||||
Reference in New Issue
Block a user