保存抠图实际模型元数据
Project CI / Repository checks (pull_request) Failing after 10s
Project CI / Backend tests (pull_request) Failing after 10s
Project CI / Frontend tests (pull_request) Successful in 1m32s
Project CI / Native shell tests (pull_request) Successful in 1m55s

在 generationInputs 顶层记录实际抠图供应商与模型

覆盖 BgFilter、阿里云和本地键色及三条派生资产持久化链路

保持源生图模型与图片信息展示契约不变

补充后端前端测试并同步数据契约和决策文档
This commit is contained in:
2026-07-30 12:10:40 +00:00
parent 0c4775f5ab
commit 4670632919
5 changed files with 206 additions and 29 deletions
@@ -16,6 +16,16 @@
---
## 2026-07-30 抠图实际后端作为 generationInputs 顶层内部元数据保存
- 背景:角色、图标图集和 UI 图集抠图派生资产需要保留最终实际执行的处理后端,供后台诊断 BgFilter、阿里云通用抠图和本地键色的降级结果;把抠图模型写成 `generationInputs.fields` 的“处理模型”会进入图片信息,与用户可见输入快照语义冲突,而覆盖正式资产 `model` 又会丢失源生图模型。
- 决策:继续使用现有 `generation_inputs_json` JSON 包络,不修改 SpacetimeDB schema。`fields` / `references` 只保存用户可见生成输入;仿照顶层 `screenColorHex`,抠图派生资产在顶层写入 `mattingProvider` / `mattingModel`。BgFilter 记录本次实际 `seg_model`;阿里云记录 `Aliyun Matting / segment-common-image`;本地键色记录 `Genarrative Local / screen-color-keying`。三条链路的正式资产 `model` 继续继承源生图模型,图片信息不读取顶层内部字段;这些字段不是保密字段,原始管理接口仍可按既有权限读取。角色动作逐帧可能混用多个 fallback,本次不把单帧结果提升为整组动画模型。
- 影响范围:`server-rs/crates/api-server/src/editor_project.rs` 的抠图结果和角色 / 图标 / UI 派生资产持久化、图片信息兼容测试、后端数据契约与图片画布 MVP 文档;不修改前端生产展示逻辑、请求 DTO、SpacetimeDB 表、迁移或生成绑定。
- 验证方式:后端单测覆盖三种实际结果映射、顶层元数据不改写 `fields`,结构断言覆盖三条派生资产持久化链路仍保留源生图模型;前端测试覆盖顶层字段不在图片信息出现。运行 `cargo test -p api-server editor_project::tests --manifest-path server-rs/Cargo.toml`、图片信息定向前端测试、`cargo check -p api-server --manifest-path server-rs/Cargo.toml``npm run typecheck``npm run check:encoding``git diff --check`
- 关联文档:`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md``docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md`
---
## 2026-07-29 图集切片必须受前置容量和有界 CPU 保护
- 背景:图标与 UI 图集的 alpha 连通域识别会在 async handler 上同步执行;原始连通域合并采用全量两两比较,`64` 个输出限制又晚于排序、裁剪和 PNG 编码。碎块或噪点图会放大 CPU 与内存成本,手动拆分、图标自动拆分和 UI 提取都受影响。另一方面,图标与 UI 的 Alpha 尺寸恢复、provider 原图回读或透明图解码失败此前只记日志,仍会把不可信透明图持久化并拆分。
@@ -172,6 +172,7 @@
- “我的”页项目入口能进入 `/project`;项目页能列出工程、重命名 / 删除单个工程、批量选择和批量删除;点击工程后进入 `/editor/canvas?projectid=<projectId>` 并按 query 加载该工程。
- 用户可见的 `generationInputs.fields` 不得写入或展示“处理模型”;角色抠图、图标 / UI 图集抠图及自动拆分形成的派生资产,其 `Model` 继续继承并展示源生图模型,不得改写为 `birefnet``anime-seg``connected-components`、阿里云抠图或本地键色等内部后处理模型。前端同时过滤历史项目中已持久化的“处理模型”字段;历史资源若只剩内部处理模型而无法恢复源生图模型,保留 `Model` 行并显示 `-`。画布 ZIP 的用户可见导出元数据使用同一过滤口径且保持既有结构。
- `generationInputs` 整体是可扩展 JSON 包络,不等同于全量用户可见快照;图片信息只消费 `fields` / `references`。后端可仿照 `screenColorHex` 在顶层保存 `mattingProvider` / `mattingModel`,记录角色、图标图集和 UI 图集抠图实际成功的 BgFilter、阿里云或本地键色结果;前端规范化和图片信息展示不得把这些顶层内部字段映射成可见字段,正式 `Model` 仍取源生图模型。
## 后续扩展点
@@ -576,6 +576,7 @@ npm run check:server-rs-ddd
- Rust 结构体:`EditorProjectResource`
- 源码:`server-rs/crates/spacetime-module/src/editor_project_storage.rs`
- 说明:图片画布工程资源元数据表,保存已经放入某个 project 画布的上传 / 生成图片资源快照、OSS 引用、尺寸、来源类型、prompt、provider、task、源资源关系、`asset_kind``generation_inputs_json` 和历史 `public_showcase_enabled``asset_kind` 标记角色、图标、UI 设计图、视频、音频等素材类别;`generation_inputs_json` 保存用户可见生成输入快照,供图片信息页刷新后恢复。`public_showcase_enabled` 只保留旧接口兼容,不再作为 `/creation``陶泥儿精选` 事实源;精选公开改由账号级生成素材提交 `editor_showcase_asset` 审核决定。图片 / 图标 / UI 提取等生成 BFF 在请求携带 `project_id` 时负责创建该表记录并把 resource 快照返回前端;前端只保存稳定 `resource_id` 布局引用,不能把同一生成结果再次作为正式业务真相写入。项目封面快照也落在该表,使用 `asset_kind = project-cover-snapshot``source_type = uploaded` 和私有 OSS / asset object 引用,代表画布当前视口栅格化后的静态封面;项目列表和创作主页最近项目只读取最新封面快照资源,不在列表页根据 layout 临时拼画布。从账号级素材库把同一生成素材拖回同一项目画布时,后端优先复用同项目内同源同媒体资源,避免每个图层实例都插入新的资源行。账号级素材删除不级联删除该表,避免历史画布丢图。结构化 canvas 的几何、层级、分组和资源引用以 `editor_canvas_layer` 为权威,生成器对象以 `editor_canvas_generation_dialog` 为权威;legacy canvas 才在 2 MiB 上限内从 `editor_canvas.layers_json` 兼容读取。新写入不再把素材生成输入快照作为图层布局真相保存。历史普通图层缺资源只能由 migration operator 调用 `repair_editor_canvas_resources_and_return` 定向修复:procedure 每次只处理一个尚无迁移记录的 legacy canvas,校验 owner/project、revision、canvas/project 两份 raw layout SHA-256、精确 layer/resource/sourceResourceId、同工程替换资源与 private asset_object 谱系;图片只替换引用,音频只恢复经核验的 `420x120` 项目资源行。运维入口 `npm run spacetime:editor-canvas-resources:repair` 默认 dry-runapply 必须绑定 plan SHA-256 并在成功后自动复核 already-repaired,禁止手工 SQL 绕过事务 guard。
- `generation_inputs_json` 包络契约:`fields` / `references` 是图片信息读取的用户可见生成输入快照;顶层允许保存后端内部结果扩展。现有 `screenColorHex` 保存实际背景色,角色、图标图集和 UI 图集抠图派生资产使用 `mattingProvider` / `mattingModel` 保存实际成功的处理后端与模型。BgFilter 保存本次 `seg_model`,阿里云通用抠图保存 `Aliyun Matting / segment-common-image`,本地键色保存 `Genarrative Local / screen-color-keying`。这些顶层字段不写入 `fields`、不在图片信息展示,也不具有保密边界,原始管理接口按既有权限仍可读取。
- 索引:`by_editor_project_resource_project_id``by_editor_project_resource_owner_user_id`
### `editor_asset_folder`
+182 -22
View File
@@ -152,6 +152,11 @@ const EDITOR_LEGACY_GREEN_SCREEN_SOURCE_ASSET_KIND: &str = "editor_green_screen_
const EDITOR_PROVIDER_SOURCE_SLOT: &str = "provider_source";
pub(crate) const EDITOR_BGFILTER_DEFAULT_SEG_MODEL: &str = "birefnet";
const EDITOR_BGFILTER_SEG_MODEL_ANIME_SEG: &str = "anime-seg";
const EDITOR_BGFILTER_MATTING_PROVIDER: &str = "BgFilter";
const EDITOR_ALIYUN_MATTING_PROVIDER: &str = "Aliyun Matting";
const EDITOR_ALIYUN_MATTING_MODEL: &str = "segment-common-image";
const EDITOR_LOCAL_MATTING_PROVIDER: &str = "Genarrative Local";
const EDITOR_LOCAL_MATTING_MODEL: &str = "screen-color-keying";
const EDITOR_MATTING_SOURCE_URL_EXPIRE_SECONDS: u64 = 600;
pub(crate) const EDITOR_BGFILTER_CROSS_CHECK_ENABLED: bool = true;
pub(crate) const EDITOR_BGFILTER_CROSS_CHECK_DISABLED: bool = false;
@@ -2030,6 +2035,7 @@ pub(crate) async fn generate_editor_image_for_owner(
}
};
let removal_provider = removal.provider;
let removal_model = removal.model.clone();
let (restored_removal_image, postprocess_dimension_error, pixel_art_error) = if image_style
== EditorImageGenerationStyle::PixelArt
{
@@ -2115,8 +2121,11 @@ pub(crate) async fn generate_editor_image_for_owner(
output_prompt = "去除纯色背景".to_string();
output_actual_prompt = None;
output_provider = removal_provider.to_string();
output_generation_inputs =
build_editor_derived_asset_generation_inputs("角色抠图", &source_record);
output_generation_inputs = apply_editor_matting_metadata_to_generation_inputs(
build_editor_derived_asset_generation_inputs("角色抠图", &source_record),
removal_provider,
removal_model.as_str(),
);
Some(source_record)
} else {
None
@@ -2269,6 +2278,23 @@ pub(crate) fn apply_editor_screen_background_decision_to_generation_inputs(
Some(value)
}
fn apply_editor_matting_metadata_to_generation_inputs(
generation_inputs: Option<Value>,
provider: &str,
model: &str,
) -> Option<Value> {
let mut value = match generation_inputs {
Some(value) if value.is_object() => value,
Some(other) => return Some(other),
None => json!({ "fields": [], "references": [] }),
};
// 抠图供应商和模型是内部实际执行结果,不属于用户可见的生成输入快照;
// 仿照 screenColorHex 只写顶层扩展,图片信息仍只读取 fields / references。
value["mattingProvider"] = Value::String(provider.to_string());
value["mattingModel"] = Value::String(model.to_string());
Some(value)
}
fn build_editor_derived_asset_generation_inputs(
stage: &str,
source: &EditorGeneratedAssetRecord,
@@ -4165,6 +4191,33 @@ async fn request_editor_background_removal_image_with_bgfilter_worker(
pub(crate) struct EditorScreenBackgroundRemovalOutput {
pub(crate) image: DownloadedOpenAiImage,
pub(crate) provider: &'static str,
pub(crate) model: String,
}
impl EditorScreenBackgroundRemovalOutput {
fn bgfilter(image: DownloadedOpenAiImage, seg_model: &str) -> Self {
Self {
image,
provider: EDITOR_BGFILTER_MATTING_PROVIDER,
model: seg_model.to_string(),
}
}
fn aliyun(image: DownloadedOpenAiImage) -> Self {
Self {
image,
provider: EDITOR_ALIYUN_MATTING_PROVIDER,
model: EDITOR_ALIYUN_MATTING_MODEL.to_string(),
}
}
fn local(image: DownloadedOpenAiImage) -> Self {
Self {
image,
provider: EDITOR_LOCAL_MATTING_PROVIDER,
model: EDITOR_LOCAL_MATTING_MODEL.to_string(),
}
}
}
pub(crate) async fn remove_editor_generated_screen_background_with_bgfilter(
@@ -4192,14 +4245,14 @@ pub(crate) async fn remove_editor_generated_screen_background_with_bgfilter(
.await
{
Ok(image) => {
return Ok(EditorScreenBackgroundRemovalOutput {
image: DownloadedOpenAiImage {
return Ok(EditorScreenBackgroundRemovalOutput::bgfilter(
DownloadedOpenAiImage {
bytes: image.bytes,
extension: image.extension,
mime_type: image.mime_type,
},
provider: "BgFilter",
});
seg_model,
));
}
Err(error) if !error.allows_flat_fallback() => {
return Err(error.into_app_error());
@@ -4250,10 +4303,7 @@ async fn fallback_editor_screen_background_removal(
Err(error) => Err(error),
};
match aliyun_result {
Ok(image) => Ok(EditorScreenBackgroundRemovalOutput {
image,
provider: "Aliyun Matting",
}),
Ok(image) => Ok(EditorScreenBackgroundRemovalOutput::aliyun(image)),
Err(error) => {
tracing::warn!(
provider = "aliyun-matting",
@@ -4287,10 +4337,7 @@ async fn fallback_editor_screen_background_removal(
let output =
remove_editor_generated_green_screen_background(&source_image, screen_color);
drop(source_image);
output.map(|image| EditorScreenBackgroundRemovalOutput {
image,
provider: "Genarrative Local",
})
output.map(EditorScreenBackgroundRemovalOutput::local)
}
}
}
@@ -4781,6 +4828,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner(
}
};
let removal_provider = removal.provider;
let removal_model = removal.model.clone();
let (image, postprocess_dimension_error, pixel_art_error) = if image_style
== EditorImageGenerationStyle::PixelArt
{
@@ -4858,8 +4906,11 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner(
Some(editor_pixel_art_fallback_warning()),
);
}
let matting_generation_inputs =
build_editor_derived_asset_generation_inputs("图标图集抠图", &source_record);
let matting_generation_inputs = apply_editor_matting_metadata_to_generation_inputs(
build_editor_derived_asset_generation_inputs("图标图集抠图", &source_record),
removal_provider,
removal_model.as_str(),
);
let (spritesheet_width, spritesheet_height) = decode_editor_provider_source_dimensions(
&image,
"editor-icon-spritesheet",
@@ -4877,7 +4928,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner(
"icon-spritesheets",
"spritesheet",
"spritesheet",
removal.provider,
removal_provider,
)
.await?;
let spritesheet_image_src =
@@ -4897,7 +4948,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner(
prompt: "去除纯色背景".to_string(),
actual_prompt: None,
model: generation_options.model.to_string(),
provider: removal.provider.to_string(),
provider: removal_provider.to_string(),
task_id: generated.task_id.clone(),
group_task_id: None,
group_task_expected_asset_count: None,
@@ -5703,6 +5754,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
}
};
let removal_provider = removal.provider;
let removal_model = removal.model.clone();
let (image, postprocess_dimension_error) =
apply_editor_postprocessed_alpha_from_persisted_provider_source_or_original(
state,
@@ -5753,8 +5805,11 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
},
));
}
let matting_generation_inputs =
build_editor_derived_asset_generation_inputs("UI图集抠图", &source_record);
let matting_generation_inputs = apply_editor_matting_metadata_to_generation_inputs(
build_editor_derived_asset_generation_inputs("UI图集抠图", &source_record),
removal_provider,
removal_model.as_str(),
);
let (spritesheet_width, spritesheet_height) = decode_editor_provider_source_dimensions(
&image,
"editor-ui-design-asset-extraction",
@@ -5772,7 +5827,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
"ui-design-spritesheets",
"spritesheet",
"spritesheet",
removal.provider,
removal_provider,
)
.await?;
let spritesheet_image_src =
@@ -5792,7 +5847,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
prompt: "去除纯色背景".to_string(),
actual_prompt: None,
model: generation_options.model.to_string(),
provider: removal.provider.to_string(),
provider: removal_provider.to_string(),
task_id: generated.task_id.clone(),
group_task_id: None,
group_task_expected_asset_count: None,
@@ -8763,6 +8818,80 @@ mod tests {
assert_eq!(updated["screenColorHex"], json!("#CFEFFF"));
}
#[test]
fn matting_metadata_records_internal_result_without_touching_visible_fields() {
let updated = apply_editor_matting_metadata_to_generation_inputs(
Some(json!({
"fields": [{ "title": "处理阶段", "value": "角色抠图" }],
"references": [],
"screenColorHex": "#CFEFFF",
})),
EDITOR_BGFILTER_MATTING_PROVIDER,
EDITOR_BGFILTER_SEG_MODEL_ANIME_SEG,
)
.expect("matting metadata should be recorded");
assert_eq!(
updated["fields"],
json!([{ "title": "处理阶段", "value": "角色抠图" }])
);
assert_eq!(updated["references"], json!([]));
assert_eq!(updated["screenColorHex"], json!("#CFEFFF"));
assert_eq!(updated["mattingProvider"], json!("BgFilter"));
assert_eq!(updated["mattingModel"], json!("anime-seg"));
assert!(
updated["fields"]
.as_array()
.expect("fields should remain an array")
.iter()
.all(|field| field["title"] != "处理模型")
);
let created = apply_editor_matting_metadata_to_generation_inputs(
None,
EDITOR_LOCAL_MATTING_PROVIDER,
EDITOR_LOCAL_MATTING_MODEL,
)
.expect("missing inputs should receive an internal metadata envelope");
assert_eq!(
created,
json!({
"fields": [],
"references": [],
"mattingProvider": "Genarrative Local",
"mattingModel": "screen-color-keying",
})
);
}
fn test_screen_background_removal_image() -> DownloadedOpenAiImage {
DownloadedOpenAiImage {
bytes: vec![1, 2, 3],
extension: "png".to_string(),
mime_type: "image/png".to_string(),
}
}
#[test]
fn screen_background_removal_outputs_record_each_actual_matting_backend() {
let bgfilter = EditorScreenBackgroundRemovalOutput::bgfilter(
test_screen_background_removal_image(),
EDITOR_BGFILTER_SEG_MODEL_ANIME_SEG,
);
assert_eq!(bgfilter.provider, "BgFilter");
assert_eq!(bgfilter.model, "anime-seg");
let aliyun =
EditorScreenBackgroundRemovalOutput::aliyun(test_screen_background_removal_image());
assert_eq!(aliyun.provider, "Aliyun Matting");
assert_eq!(aliyun.model, "segment-common-image");
let local =
EditorScreenBackgroundRemovalOutput::local(test_screen_background_removal_image());
assert_eq!(local.provider, "Genarrative Local");
assert_eq!(local.model, "screen-color-keying");
}
fn apply_editor_canvas_generation_completion(
layers: Value,
completion: &EditorCanvasGenerationCompletionRequest,
@@ -12157,6 +12286,7 @@ mod tests {
assert_eq!(removed.image.mime_type, "image/png");
assert_eq!(removed.image.extension, "png");
assert_eq!(removed.provider, "BgFilter");
assert_eq!(removed.model, "birefnet");
}
#[tokio::test]
@@ -12279,8 +12409,17 @@ mod tests {
"build_multi_asset_canvas_layer_items",
"if is_character_generation",
"let output_model = generation_options.model.to_string();",
"apply_editor_matting_metadata_to_generation_inputs",
"build_editor_derived_asset_generation_inputs(\"角色抠图\"",
"generation_inputs: output_generation_inputs",
],
);
assert_function_not_contains(
source,
"pub(crate) async fn generate_editor_image_for_owner",
"fn normalize_editor_image_generation_size",
&["{ \"title\": \"处理模型\""],
);
assert_function_contains_in_order(
source,
"pub(crate) async fn generate_editor_image_for_owner",
@@ -12305,8 +12444,17 @@ mod tests {
"slice_editor_icon_spritesheet_all",
"persist_editor_spritesheet_slices",
"model: generation_options.model.to_string(),",
"apply_editor_matting_metadata_to_generation_inputs",
"build_editor_derived_asset_generation_inputs(\"图标图集抠图\"",
"generation_inputs: matting_generation_inputs",
],
);
assert_function_not_contains(
source,
"pub(crate) async fn generate_editor_icon_spritesheet_for_owner",
"pub async fn extract_editor_ui_design_assets",
&["{ \"title\": \"处理模型\""],
);
assert_function_contains_in_order(
source,
"pub(crate) async fn generate_editor_icon_spritesheet_for_owner",
@@ -12341,8 +12489,17 @@ mod tests {
"persist_editor_provider_source_resource",
"slice_editor_icon_spritesheet_all",
"model: generation_options.model.to_string(),",
"apply_editor_matting_metadata_to_generation_inputs",
"build_editor_derived_asset_generation_inputs(\"UI图集抠图\"",
"generation_inputs: matting_generation_inputs",
],
);
assert_function_not_contains(
source,
"pub(crate) async fn extract_editor_ui_design_assets_for_owner",
"pub(crate) fn editor_project_payload_from_record",
&["{ \"title\": \"处理模型\""],
);
assert_function_contains_in_order(
source,
"pub(crate) async fn extract_editor_ui_design_assets_for_owner",
@@ -12363,6 +12520,7 @@ mod tests {
"crate::bgfilter_worker::BgfilterBackgroundMode::Flat",
"editor_bgfilter_flat_deadline_reserve",
"state.config.aliyun_matting_request_timeout_ms",
"EditorScreenBackgroundRemovalOutput::bgfilter",
"error.allows_flat_fallback()",
"fallback_editor_screen_background_removal",
],
@@ -12385,9 +12543,11 @@ mod tests {
"fn sign_editor_private_object_read_url",
&[
"segment_image_url_with_aliyun_matting",
"EditorScreenBackgroundRemovalOutput::aliyun",
"download_editor_persisted_image_object",
"remove_editor_generated_green_screen_background",
"drop(source_image)",
"output.map(EditorScreenBackgroundRemovalOutput::local)",
],
);
assert_function_contains(
@@ -68,17 +68,20 @@ describe('ImageCanvasMetadataModalView', () => {
});
it('keeps the generation model while hiding internal processing models', () => {
const generationInputsWithInternalMattingMetadata = {
fields: [
{ title: '处理阶段', value: '角色抠图' },
{ title: '处理模型', value: 'birefnet' },
],
references: [],
mattingProvider: 'Aliyun Matting',
mattingModel: 'segment-common-image',
};
const { rerender } = render(
<ImageCanvasMetadataModalView
layer={createLayer({
model: 'gpt-image-2',
generationInputs: {
fields: [
{ title: '处理阶段', value: '角色抠图' },
{ title: '处理模型', value: 'birefnet' },
],
references: [],
},
generationInputs: generationInputsWithInternalMattingMetadata,
})}
onClose={vi.fn()}
/>,
@@ -92,6 +95,8 @@ describe('ImageCanvasMetadataModalView', () => {
expect(within(dialog).getByText('Model')).toBeTruthy();
expect(within(dialog).getByText('gpt-image-2')).toBeTruthy();
expect(within(dialog).queryByText('birefnet')).toBeNull();
expect(within(dialog).queryByText('Aliyun Matting')).toBeNull();
expect(within(dialog).queryByText('segment-common-image')).toBeNull();
rerender(
<ImageCanvasMetadataModalView