Add UI-design asset extraction & spritesheet support

Introduce a new UI design asset extraction flow: add POST /api/editor/ui-designs/assets/extractions handler that always uses gpt-image-2 with the fixed prompt "提取画面中的所有独立并整理成spritesheet"; parse Data URL references, call OpenAI image edits, return a spritesheet and sliced icon assets. Increase request body limit for image Data URLs and wire the route with bearer auth. Add platform-image slicing: slice_generated_icon_spritesheet_all_by_connected_components (with auto-naming "素材 1..N", foreground estimation and heuristics) and export it. Add Rust tests for the handler, prompt constant, and slicing behavior. Update frontend client and integration tests to call the new extractEditorUiDesignAssets API and verify spritesheet + icon layers are added to the canvas and persisted. Update multiple docs to describe the extraction UI, toolbar placement, and related audio/playback pitfalls. Minor related updates: toolbar button ordering and audio duration / signed read-url notes.
This commit is contained in:
2026-06-19 10:09:19 +08:00
parent ea29887d9c
commit 097170836d
36 changed files with 1092 additions and 110 deletions
@@ -16,6 +16,14 @@
---
## 2026-06-18 图片画布 UI 设计图提取素材保留图集
- 背景:UI 设计图需要从成图中继续抽取可复用独立素材;原图标素材生成只把拆分后的图标放入画布,spritesheet 原图没有保留,后续追溯和二次切图不方便。
- 决策:`assetKind="ui-design"` 图层浮动工具栏新增 `提取素材`,固定调用 `/api/editor/ui-designs/assets/extractions`,后端固定 `gpt-image-2` 和提示词 `提取画面中的所有独立并整理成spritesheet`,返回结构复用图标 spritesheet 响应。图标生成与 UI 提取都必须把 spritesheet 图集作为 `assetKind="icon-spritesheet"` 图层放到画布,再放拆分后的 `assetKind="icon"` 素材。
- 影响范围:图片画布浮动工具栏、编辑器图片生成 BFF、`platform-image` 图集连通域拆分、画布图层类型和编辑器文档。
- 验证方式:运行图片画布工具栏 / 图集落层 / 生成提交相关前端测试,`cargo test -p platform-image generated_asset_sheets --manifest-path server-rs/Cargo.toml`,以及 `cargo test -p api-server editor_ui_design_asset_extraction_prompt_is_fixed --manifest-path server-rs/Cargo.toml`
- 关联文档:`docs/【编辑器】画板UI设计图生成入口设计-2026-06-17.md``docs/【编辑器】画板图标素材生成入口设计-2026-06-15.md`
## 2026-06-18 图片画布 Seedance 2.0 参考媒体提交边界
- 背景:`/editor/canvas` 生成视频需要严格对齐火山 Seedance 2.0 多模态参考输入;参考视频若继续走 Base64 / `data:video` 会超过请求体并被上游拒绝,参考音频单独输入和非 Seedance 模型携带参考字段也会违反文档契约。
@@ -23,6 +23,14 @@
- 验证:`cargo test -p platform-audio --manifest-path server-rs/Cargo.toml``cargo test -p api-server vector_engine_audio_generation --manifest-path server-rs/Cargo.toml`
- 关联:`server-rs/crates/platform-audio/src/client.rs``server-rs/crates/platform-audio/src/response.rs``docs/【编辑器】画板音乐生成入口设计-2026-06-18.md`
## 图片画布音频卡播放条 0:00 要优先查签名 URL 和嵌套交互
- 现象:画板音效或背景音乐已经生成成功,但卡片里的播放条显示 `0:00`,点击无法预览。
- 原因:generated 音频资源通常是私有 OSS 路径,直接把 `/generated-*` 或 generated OSS 地址交给 `<audio>` 会无鉴权读取失败;如果音频控件嵌在 `<button>` 图层里,浏览器还可能因嵌套交互元素阻断 controls 行为。
- 处理:音频图层使用非嵌套交互容器承接画布选择语义,内部 `<audio controls preload="metadata">` 单独阻止 pointer / click 冒泡;generated 音频播放前统一通过 `useResolvedAssetReadUrl` / `/api/assets/read-url` 换签。卡片和角标展示 `时长`,后端没返回时长时可用 `loadedmetadata.duration` 兜底。
- 验证:`npx vitest run src/components/image-editor/ImageCanvasWorldView.test.tsx src/components/image-editor/ImageCanvasMetadataModalView.test.tsx src/components/image-editor/ImageCanvasGenerationLayerModel.test.ts --reporter verbose`,并在浏览器确认 generated 音频控件可播放。
- 关联:`src/components/image-editor/ImageCanvasWorldView.tsx``src/components/image-editor/ImageCanvasMediaModel.ts``docs/【编辑器】画板音乐生成入口设计-2026-06-18.md`
## 图片编辑器底部生成按钮不要复用单一画布生成状态
- 现象:图片画布里先新建一个“生成规范”占位,再点击“生成角色形象”或其它底部生成入口,前一个规范占位和面板状态被销毁。
@@ -67,6 +67,7 @@
- `DELETE /api/editor/assets/{assetId}`:删除素材。已放入画布的 project resource 不被级联删除,避免旧画布丢图。
- `POST /api/editor/images/generations`:按提示词调用 VectorEngine 生成图片;角色生成可携带 `model``aspectRatio``imageSize``referenceImageSrcs``nanobanana2` 参考图作为 `inline_data` 进入 `generateContent``gpt-image-2` 参考图进入 edits。携带参考图的快速编辑也走该接口,前端必须把参考图源预读成图片 Data URL 后放入 `referenceImageSrcs`;接口返回 data URL、尺寸、prompt、model、provider 和 taskId。
- `POST /api/editor/icon-spritesheets/generations`:按图标规范图和素材描述数组生成 spritesheet,再由后端切分为独立透明图标。请求支持 `model``aspectRatio``imageSize``priceMudPoints``priceMudPoints` 必须来自编辑器生成计费配置的 `icon` 档位(首版 12 泥点),后端用 `editor_generation_config` 校验后才调用上游;`nanobanana2` 走原生 `generateContent` 并写入 `generationConfig.imageConfig.aspectRatio/imageSize``0.5K``"512"``gpt-image-2``/v1/images/edits`,后端把 UI 尺寸归一为文档支持的 `1024x1024``1024x1536``1536x1024``2048x2048``2048x1152``size` 字符串。
- `POST /api/editor/ui-designs/assets/extractions`:以 UI 设计图 Data URL 作为参考图,固定 `gpt-image-2` 和提示词 `提取画面中的所有独立并整理成spritesheet` 生成素材 spritesheet,再按连通域自动拆分为 `素材 1..N`,返回结构复用图标 spritesheet 响应。前端必须把 spritesheet 原图与拆分素材都加入画布。
- `POST /api/editor/images/edits`:按提示词和当前图片 Data URL 调用 VectorEngine edits,返回新的生成图片元数据。
- `POST /api/editor/videos/generations`:按视频描述、模型、比例、时长、分辨率、模式、声音和泥点价格生成视频。前端可选模型为 `seedance2.0-fast``seedance2.0``kling3.0``kling3.0-omni`,默认 `seedance2.0-fast`;后端必须将 `seedance2.0-fast` 映射到 `doubao-seedance-2-0-fast-260128`,将 `seedance2.0` 映射到 `doubao-seedance-2-0-260128`,两者不得混用。后端复用 Ark / VectorEngine content generation task 轮询链路,下载最终视频并持久化到 OSS,返回 `videoSrc`、尺寸、prompt、model、provider、taskId、durationSeconds、resolution 和 `priceMudPoints`
@@ -95,7 +96,7 @@
- 快速编辑面板底部必须以禁用态参数按钮显示原图尺寸和原图模型,视觉对齐可编辑面板参数控件;参考图顺序为额外参考图在前、原图在最后。
- 点击生成、生成规范、生成角色形象或生成图标素材后创建的占位图可继续保留;点击画布空白区域让当前图片或占位图失焦时,关闭当前生成面板并移除图片选中样式,但不删除占位图本身。
- 生成资源显示元数据按钮,元数据窗口展示来源、生成输入快照、model、provider、task、Resolution 和 OSS 引用;生成输入快照只包含用户面板输入和参考图,不包含后端拼接 Prompt,不再展示独立 Size 字段。
- 图片选中后的浮动工具栏按钮顺序固定为:快速编辑、分割线、裁扩按钮、去除背景按钮、角色图专属生成动画、分割线、重绘、下载按钮。裁扩通过画布边界拖拉完成,不再展示四边数值输入;默认自由比例,选择固定比例后拖拉边界保持对应比例,完成后在原素材旁边新增裁扩结果图层,扩展区域透明填充。去除背景使用前端轻量去背并回写为新的工程资源快照。
- 图片选中后的浮动工具栏按钮顺序固定为:快速编辑、分割线、裁扩按钮、去除背景按钮、UI设计图专属提取素材、角色图专属生成动画、分割线、重绘、下载按钮。裁扩通过画布边界拖拉完成,不再展示四边数值输入;默认自由比例,选择固定比例后拖拉边界保持对应比例,完成后在原素材旁边新增裁扩结果图层,扩展区域透明填充。去除背景使用前端轻量去背并回写为新的工程资源快照。UI设计图的提取素材固定走 `gpt-image-2` 和提示词 `提取画面中的所有独立并整理成spritesheet`,生成的 spritesheet 原图和拆分后的独立素材都作为画布图层保留。
- 重绘生成资源后,右侧出现新生成结果图层,并自动 fit 原图 + 新图,且重绘面板保持打开。
- 快速编辑 / 重绘站内 public 示例图、历史 generated 图或 OSS generated 图时,前端先读取成 `data:image/*;base64,...` 再提交,后端不得再收到 `/creation-type-references/*``/generated-*` 或 OSS URL 作为 `referenceImageSrcs/sourceImageSrc`
- 快速编辑的额外参考图同样必须在前端读取成图片 Data URL 后提交;后端 `referenceImageSrcs` 上限为 9 张,承载 8 张额外参考图加 1 张原图。
@@ -120,7 +120,7 @@
## 生成视频模型与接口
- 底部 `生成视频` 面板提交到 `POST /api/editor/videos/generations`
- 固定 `16:9`、静音、标准模式;结果作为 `mediaType="video"``assetKind="video"` 的视频图层加入画布。
- 固定 `16:9`、静音、标准模式;结果作为 `mediaType="video"``assetKind="video"` 的视频图层加入画布,并在画布中使用原生 `<video>` 播放控件渲染,不再走图片元素
- 支持模型:
- `seedance2.0-fast`
- `seedance2.0`
@@ -163,6 +163,7 @@
- 快速编辑生成中的占位图同样只支持键盘 `Delete` / `Backspace` 删除,不新增 UI 删除按钮;删除后异步结果不得再落回画布。
- 生成视频 / 角色形象 / 角色动作 / 音效 / 背景音乐新建后,画布占位空白样式和右上角标签均与对应生成类型一致,不再统一使用图片占位 icon。
- 点击角色图只选中图层并显示工具栏,不自动弹出重绘、快速编辑或角色动画面板;点击工具栏或右键菜单中的 `生成动画` 才创建角色动作占位和面板。
- 点击 UI 设计图只选中图层并显示工具栏;工具栏在 `去除背景按钮` 后显示 `提取素材`,点击后固定用 `gpt-image-2` 和提示词 `提取画面中的所有独立并整理成spritesheet` 生成 spritesheet,再复用图标素材拆分流程,把 spritesheet 图集和拆分素材都放到画布。
- 生成游戏音效面板底部不显示 `type` / `tempo` 字段标题,只有一个音效参数按钮;type 只支持 `one-shot` / `loop`BPM 为 `1-300` 的数字或 `null`,选择后按钮显示 `单次·120BPM` 这类组合值。
- 生成视频结果以视频图层加入画布,画布媒体元素标记为 `画布视频:生成视频 N`
- 生成器输入、参数、参考图和占位框在刷新后仍存在;已生成对象的生成器面板继续跟随成品图层。
@@ -7,6 +7,7 @@
- 底部 AI 画布工具栏新增 `生成UI设计图`
- 点击后立即在画布中新建 `UI设计图生成占位图`,不复用普通新建图片的空白样式。
- 占位图默认 16:9 展示,生成成功后替换为 `assetKind: "ui-design"` 的画布图层。
- 用户点击 `assetKind: "ui-design"` 图层后,图片上方浮动工具栏在 `去除背景按钮` 后显示 `提取素材` 按钮;普通图片、角色图和图标图层不显示该入口。
## 生成面板
@@ -44,6 +45,18 @@
生成请求固定使用 `gpt-image-2`。有参考图时走图片编辑请求;无参考图时走图片生成请求。
## 提取素材
- 点击浮动工具栏 `提取素材` 后,前端把当前 UI 设计图读取为图片 Data URL,提交到 `POST /api/editor/ui-designs/assets/extractions`
- 后端固定使用 `gpt-image-2` 图片编辑链路,并固定提示词:
```text
提取画面中的所有独立并整理成spritesheet
```
- 后端收到 spritesheet 后复用图标素材的连通域拆分能力;未知素材数量时按从上到下、从左到右自动命名为 `素材 1``素材 2`
- 前端先把 spritesheet 原图作为 `assetKind: "icon-spritesheet"` 图集图层放在 UI 设计图右侧,再把拆分出的独立素材作为 `assetKind: "icon"` 图标图层继续放到画布。
## 验收点
- 点击 `生成UI设计图` 后出现 UI 设计占位图和独立生成面板。
@@ -51,3 +64,4 @@
- 从画布选择时只能绑定图标规范图片。
- 请求参数包含 `kind: "ui-design"``model: "gpt-image-2"`、比例、大小与可选参考图。
- 生成图层信息面板展示 `用户输入``图标规范`
- 选中 UI 设计图时浮动工具栏显示 `提取素材`;点击后调用固定 `gpt-image-2` 提取接口,画布同时出现 spritesheet 图集和拆分后的独立素材。
@@ -11,7 +11,7 @@
- 底部 AI 画布工具栏新增 `生成图标素材` 按钮。
- 点击后立即在画布中心创建图标素材占位图,不复用普通“单张空白图片”图标;占位图表现为一叠空白素材图标卡片。
- 图标素材面板锚定在占位图下方,和现有生成输入框同一层级展示。
- 生成完成后删除占位态,把拆分出的每个独立图标素材作为画布图片图层铺开,图之间不重叠,并保留少量间距。
- 生成完成后删除占位态,先把后端返回的 spritesheet 原图作为 `assetKind: "icon-spritesheet"` 的图集图层放到画布,再把拆分出的每个独立图标素材作为画布图片图层铺开,图集与图标之间、图标彼此之间不重叠,并保留少量间距。
- 图标素材图层写入 `assetKind: "icon"`;图标规范图写入 `assetKind: "icon-spec"`,用于刷新后保留标签和限制点选来源。
## 面板结构
@@ -64,10 +64,10 @@
## 前端铺放规则
- 第一张图标放在原占位图左上附近。
- 后续图标按行铺开,图标之间保留约 `24px` 世界坐标间距。
- spritesheet 图集放在原占位图左上附近。
- 第一张拆分图标放在 spritesheet 图集右侧,后续图标按行铺开,图标之间保留约 `24px` 世界坐标间距。
- 每个图标图层标题使用对应素材描述文本。
- 生成成功后关闭图标素材面板,选中第一张图标素材,并打开图层面板。
- 生成成功后关闭图标素材面板,选中 spritesheet 图集,并打开图层面板。
## 验收
@@ -76,5 +76,5 @@
- 默认 6 个素材描述会进入 prompt;用户在单个文本框中继续输入时最多解析 100 个素材描述。
- 默认打开图标素材面板时选中 `nanobanana2 / 1:1 / 1K`;模型切换后,角色和图标素材面板之间沿用上次选择的模型。
- 图标素材生成请求必须带 `model``aspectRatio``imageSize``nanobanana2` 请求体必须包含 `generationConfig.imageConfig.aspectRatio/imageSize``gpt-image-2` 请求必须包含文档映射后的 `size`
- 生成成功后画布出现按描述命名的多个透明图标素材图层,图层之间不重叠。
- 生成成功后画布出现 spritesheet 图集和按描述命名的多个透明图标素材图层,图层之间不重叠。
- 生成图标素材提交体包含 `priceMudPoints: 12`;若前端传入与后端计费配置不一致的值,后端返回 `priceMudPoints` 校验错误,不继续调用上游生成。
@@ -44,7 +44,11 @@
- `assetKind="sound-effect"`
- `assetKind="background-music"`
- 音频结果以小型音频卡加入画布,卡片内使用 `<audio controls>` 播放;音频结果当前只保存为画布资源,不进入图片素材库。
- 元数据弹窗按音频显示 `音频信息` / `音频类型`,生成输入快照只展示用户面板字段
- 音频结果卡片显示用户提示词和 `时长 mm:ss`;如果上游暂未返回时长,先显示 `时长 --:--`,并在 `<audio>` 元数据加载后用真实 duration 兜底
- generated 私有音频资源播放前必须通过 `/api/assets/read-url` 换签;画布卡片不得直接把 `/generated-*` 或 generated OSS 私有地址交给 `<audio>` 裸请求。
- 音频卡片、音频生成占位和元数据弹窗都使用 `时长`,不使用图片 / 视频的分辨率语义。
- 音频图层右上角标签显示在信息按钮左侧,和其他素材卡右上角信息区保持一致。
- 元数据弹窗按音频显示 `音频信息` / `音频类型` / `时长`,生成输入快照只展示用户面板字段。
## 前端提交契约
@@ -84,7 +88,8 @@ POST /api/editor/audios/background-music/generations
provider: string,
taskId: string,
priceMudPoints: number,
audioKind: "sound-effect" | "background-music"
audioKind: "sound-effect" | "background-music",
durationSeconds?: number | null
}
```
@@ -113,5 +118,7 @@ POST /api/editor/audios/background-music/generations
- 点击 `生成游戏背景音乐` 后出现背景音乐面板,字段为 `gpt_description_prompt`,不展示 `make_instrumental`
- 音效提交到 `/api/editor/audios/sound-effects/generations`,背景音乐提交到 `/api/editor/audios/background-music/generations`
- 成功后画布新增音频卡,能通过 `<audio controls>` 播放。
- 成功后的音频卡展示提示词;卡片、悬停角标、信息弹窗和待生成占位均显示时长。
- 私有 generated 音频能先换签再预览播放,不出现播放条一直为 `0:00` 的裸路径失败状态。
- 刷新后 layout 能恢复音频生成器和音频图层。
+43
View File
@@ -1516,6 +1516,49 @@ mod tests {
assert!(!body_text.contains("length limit exceeded"));
}
#[tokio::test]
async fn editor_ui_design_asset_extraction_accepts_image_body_above_default_limit() {
let state = AppState::new(AppConfig::default()).expect("state should build");
let seed_user = seed_phone_user_with_password(&state, "13800138188", TEST_PASSWORD).await;
let token = sign_test_user_token(&state, &seed_user, "sess_editor_ui_extract_body");
let app = build_router(state);
let source_image_src = format!("data:image/png;base64,{}", "A".repeat(3 * 1024 * 1024));
let request_body = serde_json::json!({
"sourceImageSrc": source_image_src,
})
.to_string();
assert!(request_body.len() > 2 * 1024 * 1024);
let response = app
.oneshot(
Request::builder()
.method("POST")
.uri("/api/editor/ui-designs/assets/extractions")
.header("authorization", format!("Bearer {token}"))
.header("content-type", "application/json")
.body(Body::from(request_body))
.expect("request should build"),
)
.await
.expect("request should succeed");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
let body = response
.into_body()
.collect()
.await
.expect("response body should collect")
.to_bytes();
let body_text = String::from_utf8_lossy(&body);
assert!(
body_text.contains("VECTOR_ENGINE")
|| body_text.contains("OPENAI")
|| body_text.contains("配置"),
"handler should parse the oversized UI design image before checking image config: {body_text}"
);
assert!(!body_text.contains("length limit exceeded"));
}
#[tokio::test]
async fn password_entry_rejects_unknown_phone_without_registration() {
let app = build_router(AppState::new(AppConfig::default()).expect("state should build"));
@@ -13,7 +13,9 @@ use module_assets::{
use platform_image::{
DownloadedImage,
generated_asset_sheets::{
GeneratedAssetSheetConnectedIcon, slice_generated_icon_spritesheet_by_connected_components,
GeneratedAssetSheetConnectedIcon,
slice_generated_icon_spritesheet_all_by_connected_components,
slice_generated_icon_spritesheet_by_connected_components,
},
};
use platform_oss::{LegacyAssetPrefix, OssHeadObjectRequest, OssObjectAccess};
@@ -60,6 +62,7 @@ const EDITOR_PROJECT_DEFAULT_TITLE: &str = "未命名画布";
const EDITOR_IMAGE_GENERATION_SIZE: &str = "1024x1024";
const EDITOR_IMAGE_MODEL_NANOBANANA2: &str = "gemini-3.1-flash-image-preview";
const EDITOR_ICON_DESCRIPTION_LIMIT: usize = 100;
const EDITOR_UI_DESIGN_ASSET_EXTRACTION_PROMPT: &str = "提取画面中的所有独立并整理成spritesheet";
const EDITOR_CHARACTER_IMAGE_ASSET_KIND: &str = "editor_character_image";
const EDITOR_CHARACTER_IMAGE_ENTITY_KIND: &str = "editor_project";
const EDITOR_CHARACTER_IMAGE_SLOT: &str = "character";
@@ -177,6 +180,12 @@ pub struct EditorIconSpritesheetGenerationRequest {
price_mud_points: u32,
}
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct EditorUiDesignAssetExtractionRequest {
source_image_src: String,
}
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct EditorProjectResponse {
@@ -1225,6 +1234,85 @@ pub async fn generate_editor_icon_spritesheet(
))
}
pub async fn extract_editor_ui_design_assets(
State(state): State<AppState>,
Extension(request_context): Extension<RequestContext>,
Extension(authenticated): Extension<AuthenticatedAccessToken>,
Json(payload): Json<EditorUiDesignAssetExtractionRequest>,
) -> Result<Json<Value>, AppError> {
let reference_image =
parse_editor_reference_image(payload.source_image_src.as_str()).map_err(|error| {
error.with_details(json!({
"provider": "editor-ui-design-asset-extraction",
"field": "sourceImageSrc",
"message": "UI设计图必须是图片 Data URL。",
}))
})?;
let settings = require_openai_image_settings(&state)?.with_external_api_audit_context(
&request_context,
Some(authenticated.claims().user_id().to_string()),
None,
);
let http_client = build_openai_image_http_client(&settings)?;
let generated = create_openai_image_edit_with_references_and_model(
&http_client,
&settings,
GPT_IMAGE_2_MODEL,
EDITOR_UI_DESIGN_ASSET_EXTRACTION_PROMPT,
None,
EDITOR_IMAGE_GENERATION_SIZE,
1,
&[reference_image],
"图片画布提取UI设计图素材 spritesheet",
)
.await?;
let image = generated.images.into_iter().next().ok_or_else(|| {
AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
"provider": "vector-engine",
"message": "VectorEngine 未返回 UI 设计图素材 spritesheet",
}))
})?;
let (spritesheet_width, spritesheet_height) = image::load_from_memory(image.bytes.as_slice())
.map(|image| (image.width(), image.height()))
.unwrap_or((512, 512));
let source = DownloadedImage {
bytes: image.bytes.clone(),
mime_type: image.mime_type.clone(),
extension: image.extension.clone(),
};
let icon_slices = slice_generated_icon_spritesheet_all_by_connected_components(&source)
.map_err(|error| {
AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
"provider": "editor-ui-design-asset-extraction",
"message": error.to_string(),
}))
})?;
let spritesheet_image_src =
data_url_from_image_bytes(image.mime_type.as_str(), image.bytes.as_slice());
let icon_image_srcs = icon_slices
.into_iter()
.map(editor_icon_response_from_slice)
.collect();
Ok(json_success_body(
Some(&request_context),
EditorIconSpritesheetGenerationResponse {
spritesheet_image_src,
spritesheet_width,
spritesheet_height,
icon_image_srcs,
prompt: EDITOR_UI_DESIGN_ASSET_EXTRACTION_PROMPT.to_string(),
actual_prompt: generated.actual_prompt,
model: GPT_IMAGE_2_MODEL.to_string(),
provider: "VectorEngine",
task_id: generated.task_id,
price_mud_points: crate::editor_generation_config::editor_image_generation_mud_points(
Some("icon"),
),
},
))
}
fn editor_project_payload_from_record(record: EditorProjectRecord) -> EditorProjectPayload {
let canvas = editor_canvas_payload_from_record(record.canvas);
EditorProjectPayload {
@@ -1898,6 +1986,14 @@ mod tests {
assert!(prompt.contains("返回按钮、设置按钮、下一关按钮"));
}
#[test]
fn editor_ui_design_asset_extraction_prompt_is_fixed() {
assert_eq!(
EDITOR_UI_DESIGN_ASSET_EXTRACTION_PROMPT,
"提取画面中的所有独立并整理成spritesheet"
);
}
#[test]
fn editor_icon_description_validation_filters_empty_and_rejects_more_than_limit() {
let descriptions = normalize_icon_descriptions(vec![
@@ -1,5 +1,7 @@
use axum::{
Router, middleware,
Router,
extract::DefaultBodyLimit,
middleware,
routing::{get, patch, post},
};
@@ -8,14 +10,17 @@ use crate::{
editor_project::{
create_editor_asset, create_editor_asset_folder, create_editor_project,
create_editor_project_resource, delete_editor_asset, delete_editor_asset_folder,
delete_editor_project, edit_editor_image, generate_editor_icon_spritesheet,
generate_editor_image, get_editor_asset_library, get_editor_project, list_editor_projects,
load_recent_editor_project, rename_editor_project, save_editor_project_layout,
update_editor_asset, update_editor_asset_folder,
delete_editor_project, edit_editor_image, extract_editor_ui_design_assets,
generate_editor_icon_spritesheet, generate_editor_image, get_editor_asset_library,
get_editor_project, list_editor_projects, load_recent_editor_project,
rename_editor_project, save_editor_project_layout, update_editor_asset,
update_editor_asset_folder,
},
state::AppState,
};
const EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES: usize = 12 * 1024 * 1024;
pub fn router(state: AppState) -> Router<AppState> {
Router::new()
.route(
@@ -113,9 +118,26 @@ pub fn router(state: AppState) -> Router<AppState> {
)
.route(
"/api/editor/icon-spritesheets/generations",
post(generate_editor_icon_spritesheet).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(generate_editor_icon_spritesheet)
// 中文注释:图标规范参考图会以 Data URL 提交,需避免默认 2MB body limit 在 handler 前拦截。
.layer(DefaultBodyLimit::max(
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/ui-designs/assets/extractions",
post(extract_editor_ui_design_assets)
// 中文注释:UI 设计图提取素材复用图片 Data URL 入参,需兼容较大的画板生成图。
.layer(DefaultBodyLimit::max(
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
}
@@ -20,5 +20,6 @@ pub use sheet::{
crop_generated_asset_sheet_view_edge_matte,
crop_generated_asset_sheet_view_edge_matte_with_options, slice_generated_asset_sheet,
slice_generated_asset_sheet_two_items_per_row,
slice_generated_icon_spritesheet_all_by_connected_components,
slice_generated_icon_spritesheet_by_connected_components,
};
@@ -12,6 +12,7 @@ use super::color::{
};
use super::error::GeneratedAssetSheetError;
use image::{GenericImageView, ImageFormat};
use std::collections::BTreeMap;
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct GeneratedAssetSheetSliceImage {
@@ -148,7 +149,17 @@ pub fn slice_generated_icon_spritesheet_by_connected_components(
GeneratedAssetSheetError::decode_image(format!("图标 spritesheet 解码失败:{error}"))
})?;
let source = apply_generated_asset_sheet_green_screen_alpha(source);
slice_generated_icon_spritesheet_rgba_by_connected_components(source, icon_names)
slice_generated_icon_spritesheet_rgba_by_connected_components(source, icon_names, false)
}
pub fn slice_generated_icon_spritesheet_all_by_connected_components(
image: &crate::DownloadedImage,
) -> Result<Vec<GeneratedAssetSheetConnectedIcon>, GeneratedAssetSheetError> {
let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| {
GeneratedAssetSheetError::decode_image(format!("图标 spritesheet 解码失败:{error}"))
})?;
let source = apply_generated_asset_sheet_green_screen_alpha(source);
slice_generated_icon_spritesheet_rgba_by_connected_components(source, &[], true)
}
pub fn crop_generated_asset_sheet_view_edge_matte(
@@ -163,8 +174,9 @@ pub fn crop_generated_asset_sheet_view_edge_matte(
fn slice_generated_icon_spritesheet_rgba_by_connected_components(
source: image::DynamicImage,
icon_names: &[String],
auto_name_all_components: bool,
) -> Result<Vec<GeneratedAssetSheetConnectedIcon>, GeneratedAssetSheetError> {
let image = source.to_rgba8();
let mut image = source.to_rgba8();
let (width, height) = image.dimensions();
let pixel_count = (width as usize).saturating_mul(height as usize);
if pixel_count == 0 {
@@ -173,28 +185,31 @@ fn slice_generated_icon_spritesheet_rgba_by_connected_components(
));
}
let mut visited = vec![false; pixel_count];
let mut components = Vec::<GeneratedAssetSheetCellBounds>::new();
for y in 0..height {
for x in 0..width {
let pixel_index = (y as usize)
.saturating_mul(width as usize)
.saturating_add(x as usize);
if visited[pixel_index] || image.get_pixel(x, y).0[3] == 0 {
continue;
}
components.push(flood_fill_generated_icon_component(
&image,
&mut visited,
width,
height,
x,
y,
));
let mut components = detect_generated_icon_components_by_alpha(&image, width, height);
if components.len() < icon_names.len()
|| (auto_name_all_components && generated_icon_alpha_fill_ratio(&image) > 0.92)
{
let foreground_image =
build_generated_icon_spritesheet_foreground_image(&image, width, height);
let foreground_components =
detect_generated_icon_components_by_alpha(&foreground_image, width, height);
if !foreground_components.is_empty()
&& (foreground_components.len() >= icon_names.len()
|| auto_name_all_components && foreground_components.len() >= components.len())
{
image = foreground_image;
components = foreground_components;
}
}
components.sort_by_key(|bounds| (bounds.y0, bounds.x0));
let icon_names = if auto_name_all_components {
(1..=components.len())
.map(|index| format!("素材 {index}"))
.collect::<Vec<_>>()
} else {
icon_names.to_vec()
};
if components.len() < icon_names.len() {
return Err(GeneratedAssetSheetError::invalid_request(format!(
"图标 spritesheet 连通域数量不足:需要 {} 个,实际 {} 个。",
@@ -235,6 +250,101 @@ fn slice_generated_icon_spritesheet_rgba_by_connected_components(
Ok(icons)
}
fn generated_icon_alpha_fill_ratio(image: &image::RgbaImage) -> f32 {
let (width, height) = image.dimensions();
let total = (width as usize).saturating_mul(height as usize).max(1);
let visible = image.pixels().filter(|pixel| pixel.0[3] > 0).count();
visible as f32 / total as f32
}
fn detect_generated_icon_components_by_alpha(
image: &image::RgbaImage,
width: u32,
height: u32,
) -> Vec<GeneratedAssetSheetCellBounds> {
let pixel_count = (width as usize).saturating_mul(height as usize);
let mut visited = vec![false; pixel_count];
let mut components = Vec::<GeneratedAssetSheetCellBounds>::new();
for y in 0..height {
for x in 0..width {
let pixel_index = (y as usize)
.saturating_mul(width as usize)
.saturating_add(x as usize);
if visited[pixel_index] || image.get_pixel(x, y).0[3] == 0 {
continue;
}
let bounds =
flood_fill_generated_icon_component(image, &mut visited, width, height, x, y);
if bounds.area() >= 16 {
components.push(bounds);
}
}
}
components
}
fn build_generated_icon_spritesheet_foreground_image(
image: &image::RgbaImage,
width: u32,
height: u32,
) -> image::RgbaImage {
let background = estimate_generated_icon_spritesheet_background_pixel(image, width, height);
let mut foreground = image.clone();
for pixel in foreground.pixels_mut() {
if !is_generated_asset_sheet_foreground_pixel(pixel.0, background) {
pixel.0[3] = 0;
}
}
foreground
}
fn estimate_generated_icon_spritesheet_background_pixel(
image: &image::RgbaImage,
width: u32,
height: u32,
) -> [u8; 4] {
let mut buckets = BTreeMap::<u32, (u32, [u32; 4])>::new();
let mut push_pixel = |pixel: [u8; 4]| {
let key = ((pixel[0] as u32 / 16) << 20)
| ((pixel[1] as u32 / 16) << 12)
| ((pixel[2] as u32 / 16) << 4)
| (pixel[3] as u32 / 64);
let entry = buckets.entry(key).or_insert((0, [0, 0, 0, 0]));
entry.0 = entry.0.saturating_add(1);
entry.1[0] = entry.1[0].saturating_add(pixel[0] as u32);
entry.1[1] = entry.1[1].saturating_add(pixel[1] as u32);
entry.1[2] = entry.1[2].saturating_add(pixel[2] as u32);
entry.1[3] = entry.1[3].saturating_add(pixel[3] as u32);
};
for x in 0..width {
push_pixel(image.get_pixel(x, 0).0);
if height > 1 {
push_pixel(image.get_pixel(x, height - 1).0);
}
}
for y in 1..height.saturating_sub(1) {
push_pixel(image.get_pixel(0, y).0);
if width > 1 {
push_pixel(image.get_pixel(width - 1, y).0);
}
}
buckets
.into_values()
.max_by_key(|(count, _)| *count)
.map(|(count, sums)| {
let count = count.max(1);
[
(sums[0] / count) as u8,
(sums[1] / count) as u8,
(sums[2] / count) as u8,
(sums[3] / count) as u8,
]
})
.unwrap_or([255, 255, 255, 255])
}
fn flood_fill_generated_icon_component(
image: &image::RgbaImage,
visited: &mut [bool],
@@ -352,6 +462,38 @@ mod tests {
assert!(error.to_string().contains("连通域数量不足"));
}
#[test]
fn slices_all_icon_spritesheet_components_with_auto_names() {
let mut sheet: image::RgbaImage =
ImageBuffer::from_pixel(96, 64, Rgba([255, 255, 255, 255]));
for y in 8..24 {
for x in 10..28 {
sheet.put_pixel(x, y, Rgba([240, 80, 80, 255]));
}
}
for y in 34..50 {
for x in 56..76 {
sheet.put_pixel(x, y, Rgba([80, 120, 240, 255]));
}
}
let source = crate::DownloadedImage {
bytes: encode_png(sheet),
mime_type: "image/png".to_string(),
extension: "png".to_string(),
};
let icons = slice_generated_icon_spritesheet_all_by_connected_components(&source)
.expect("all icons should slice");
assert_eq!(icons.len(), 2);
assert_eq!(icons[0].name, "素材 1");
assert_eq!(icons[1].name, "素材 2");
let decoded = image::load_from_memory(icons[0].bytes.as_slice())
.expect("icon should decode")
.to_rgba8();
assert_eq!(decoded.get_pixel(0, 0).0[3], 0);
}
}
pub fn crop_generated_asset_sheet_view_edge_matte_with_options(
@@ -20,6 +20,7 @@ import {
const generateEditorImageMock = vi.hoisted(() => vi.fn());
const generateEditorIconSpritesheetMock = vi.hoisted(() => vi.fn());
const extractEditorUiDesignAssetsMock = vi.hoisted(() => vi.fn());
const generateEditorCharacterAnimationMock = vi.hoisted(() => vi.fn());
const editEditorImageMock = vi.hoisted(() => vi.fn());
const createEditorAssetMock = vi.hoisted(() => vi.fn());
@@ -42,6 +43,7 @@ vi.mock('../../services/image-editor/editorProjectClient', async () => {
return {
...actual,
editEditorImage: editEditorImageMock,
extractEditorUiDesignAssets: extractEditorUiDesignAssetsMock,
createEditorAsset: createEditorAssetMock,
createEditorAssetFolder: createEditorAssetFolderMock,
createEditorProjectResource: createEditorProjectResourceMock,
@@ -64,6 +66,7 @@ describe('ImageCanvasEditorView generation integration', () => {
setupImageCanvasEditorViewTestLifecycle({
generateEditorImageMock,
generateEditorIconSpritesheetMock,
extractEditorUiDesignAssetsMock,
generateEditorCharacterAnimationMock,
editEditorImageMock,
createEditorAssetMock,
@@ -2264,6 +2267,102 @@ describe('ImageCanvasEditorView generation integration', () => {
expect(screen.getByText('动作')).toBeTruthy();
});
it('extracts UI design assets from the floating toolbar and keeps the spritesheet on canvas', async () => {
loadOrCreateRecentEditorProjectMock.mockResolvedValueOnce({
projectId: 'editor-project-ui-extract',
title: 'UI素材提取画布',
viewport: { x: 0, y: 0, scale: 1 },
layers: [
{
layerId: 'layer-ui-design',
resourceId: 'resource-ui-design',
title: '战斗UI设计图',
src: 'data:image/png;base64,dWktZGVzaWdu',
x: 120,
y: 140,
width: 560,
height: 315,
originalWidth: 2048,
originalHeight: 1152,
zIndex: 2,
sourceType: 'generated',
prompt: '战斗UI',
actualPrompt: '战斗UI',
model: 'gpt-image-2',
provider: 'VectorEngine',
taskId: 'ui-design-task-1',
assetKind: 'ui-design',
},
],
resources: [],
updatedAt: '2026-06-18T00:00:00.000Z',
});
extractEditorUiDesignAssetsMock.mockResolvedValueOnce({
spritesheetImageSrc: 'data:image/png;base64,dWktc2hlZXQ=',
spritesheetWidth: 512,
spritesheetHeight: 512,
iconImageSrcs: [
{
name: '素材 1',
imageSrc: 'data:image/png;base64,YXNzZXQtMQ==',
width: 96,
height: 96,
},
],
prompt: '提取画面中的所有独立并整理成spritesheet',
actualPrompt: '提取画面中的所有独立并整理成spritesheet',
model: 'gpt-image-2',
provider: 'VectorEngine',
taskId: 'ui-extract-task-1',
priceMudPoints: 12,
});
render(<ImageCanvasEditorView />);
const uiDesignImage = await screen.findByAltText(
'画布图片:战斗UI设计图',
);
fireEvent.pointerDown(uiDesignImage.closest('button')!, {
button: 0,
pointerId: 156,
clientX: 180,
clientY: 180,
});
fireEvent.pointerUp(screen.getByLabelText('画布工作区'), {
pointerId: 156,
clientX: 180,
clientY: 180,
});
fireEvent.click(screen.getByRole('button', { name: '提取素材' }));
await waitFor(() => {
expect(extractEditorUiDesignAssetsMock).toHaveBeenCalledWith({
sourceImageSrc: 'data:image/png;base64,dWktZGVzaWdu',
});
});
expect(
await screen.findByAltText('画布图片:战斗UI设计图 素材图集'),
).toBeTruthy();
expect(screen.getByAltText('画布图片:素材 1')).toBeTruthy();
expect(screen.getByText('图集')).toBeTruthy();
await waitFor(() => {
expect(saveEditorProjectLayoutMock).toHaveBeenCalledWith(
'editor-project-ui-extract',
expect.objectContaining({
layers: expect.arrayContaining([
expect.objectContaining({
title: '战斗UI设计图 素材图集',
assetKind: 'icon-spritesheet',
}),
expect.objectContaining({
title: '素材 1',
assetKind: 'icon',
}),
]),
}),
);
});
});
it('opens quick edit from the floating toolbar, keeps the source as the last reference, and generates from a new frame', async () => {
loadOrCreateRecentEditorProjectMock.mockResolvedValueOnce({
projectId: 'editor-project-quick-edit',
@@ -141,25 +141,30 @@ describe('ImageCanvasEditorModel', () => {
originalWidth: 420,
originalHeight: 120,
zIndex: 5,
sourceType: 'generated',
mediaType: 'audio',
assetKind: 'sound-effect',
},
new Map([
[
'resource-audio',
{ imageSrc: '/generated-character-drafts/editor-audios/sfx.mp3' },
],
]),
);
sourceType: 'generated',
mediaType: 'audio',
assetKind: 'sound-effect',
durationSeconds: 2.4,
},
new Map([
[
'resource-audio',
{
imageSrc: '/generated-character-drafts/editor-audios/sfx.mp3',
durationSeconds: 2.4,
},
],
]),
);
expect(hydrated).toMatchObject({
id: 'layer-audio',
src: '/generated-character-drafts/editor-audios/sfx.mp3',
mediaType: 'audio',
assetKind: 'sound-effect',
});
src: '/generated-character-drafts/editor-audios/sfx.mp3',
mediaType: 'audio',
assetKind: 'sound-effect',
durationSeconds: 2.4,
});
});
it('restores audio generation dialogs from saved layout', () => {
const dialog: CanvasGenerationDialogState = {
@@ -135,6 +135,7 @@ export function createLayerFromAsset(
taskId: asset.taskId,
objectKey: asset.objectKey,
assetObjectId: asset.assetObjectId,
durationSeconds: asset.durationSeconds,
sourceAssetId: asset.id,
} satisfies CanvasLayer;
}
@@ -162,6 +163,7 @@ export function serializeLayer(
taskId: layer.taskId,
objectKey: layer.objectKey,
assetObjectId: layer.assetObjectId,
durationSeconds: layer.durationSeconds,
sourceResourceId: layer.sourceResourceId,
sourceAssetId: layer.sourceAssetId,
groupId: layer.groupId,
@@ -344,6 +346,7 @@ export function hydrateCanvasGenerationDialog(
typeof snapshot.makeInstrumental === 'boolean'
? snapshot.makeInstrumental
: undefined,
audioDurationSeconds: audioDurationOrNull(snapshot.audioDurationSeconds),
aspectRatio: stringOrUndefined(snapshot.aspectRatio),
imageSize: stringOrUndefined(snapshot.imageSize),
errorMessage: stringOrUndefined(snapshot.errorMessage),
@@ -400,6 +403,13 @@ export function hydrateLayer(
taskId: stringOrNull(snapshot.taskId),
objectKey: stringOrNull(snapshot.objectKey),
assetObjectId: stringOrNull(snapshot.assetObjectId),
durationSeconds:
audioDurationOrNull(snapshot.durationSeconds) ??
audioDurationOrNull(
resourcesById.get(resourceId) as
| ({ imageSrc: string } & { durationSeconds?: unknown })
| undefined,
),
sourceResourceId: stringOrNull(snapshot.sourceResourceId),
sourceAssetId: stringOrNull(snapshot.sourceAssetId),
groupId: stringOrNull(snapshot.groupId),
@@ -443,6 +453,7 @@ export function mapAssetLibrarySnapshot(
taskId: asset.taskId ?? undefined,
objectKey: asset.objectKey ?? undefined,
assetObjectId: asset.assetObjectId ?? undefined,
durationSeconds: asset.durationSeconds ?? undefined,
})),
};
}
@@ -478,6 +489,17 @@ export function numberFromSnapshot(value: unknown, fallback: number) {
return typeof value === 'number' && Number.isFinite(value) ? value : fallback;
}
function audioDurationOrNull(value: unknown) {
if (value && typeof value === 'object' && 'durationSeconds' in value) {
return audioDurationOrNull(
(value as { durationSeconds?: unknown }).durationSeconds,
);
}
return typeof value === 'number' && Number.isFinite(value) && value > 0
? value
: undefined;
}
export function bpmOrNull(value: unknown) {
if (value === null) {
return null;
@@ -631,6 +653,7 @@ export function canvasAssetKindOrNull(value: unknown): CanvasAssetKind | null {
value === 'character' ||
value === 'character-animation' ||
value === 'icon' ||
value === 'icon-spritesheet' ||
value === 'icon-spec' ||
value === 'ui-design' ||
value === 'video' ||
@@ -165,6 +165,7 @@ function createStageProps(): ImageCanvasStageViewProps {
onOpenRedrawPanel: vi.fn(),
onOpenCropExpandPanel: vi.fn(),
onRemoveBackground: vi.fn(),
onExtractUiDesignAssets: vi.fn(),
onOpenCharacterAnimationPanel: vi.fn(),
onDownloadLayer: vi.fn(),
onPasteCanvasClipboard: vi.fn(),
@@ -13,6 +13,7 @@ export type CanvasAssetKind =
| 'character'
| 'character-animation'
| 'icon'
| 'icon-spritesheet'
| 'icon-spec'
| 'ui-design'
| 'video'
@@ -188,6 +189,7 @@ export type GenerateDialogState = {
// 中文注释:音效 tempo 改为可空 BPM,null 表示用户不指定节奏。
soundTempo?: number | null;
makeInstrumental?: boolean;
audioDurationSeconds?: number | null;
aspectRatio?: string;
imageSize?: string;
errorMessage?: string;
@@ -16,6 +16,7 @@ export type AuthValue = NonNullable<
export type ImageCanvasEditorViewServiceMocks = {
generateEditorImageMock: Mock;
generateEditorIconSpritesheetMock: Mock;
extractEditorUiDesignAssetsMock?: Mock;
generateEditorCharacterAnimationMock: Mock;
editEditorImageMock: Mock;
createEditorAssetMock: Mock;
@@ -214,6 +215,7 @@ export async function readZipText(zip: JSZip, path: string) {
export function setupImageCanvasEditorViewTestLifecycle({
generateEditorImageMock,
generateEditorIconSpritesheetMock,
extractEditorUiDesignAssetsMock,
generateEditorCharacterAnimationMock,
editEditorImageMock,
createEditorAssetMock,
@@ -323,6 +325,7 @@ export function setupImageCanvasEditorViewTestLifecycle({
vi.restoreAllMocks();
generateEditorImageMock.mockReset();
generateEditorIconSpritesheetMock.mockReset();
extractEditorUiDesignAssetsMock?.mockReset();
generateEditorCharacterAnimationMock.mockReset();
editEditorImageMock.mockReset();
createEditorAssetMock.mockReset();
@@ -561,6 +561,7 @@ export function ImageCanvasEditorView() {
openRedrawPanel,
openCropExpandPanel,
removeSelectedLayerBackground,
extractUiDesignAssets,
pickCharacterSpecFromLayer,
pickGenerationReferenceFromLayer,
pickQuickEditReferenceFromLayer,
@@ -923,6 +924,7 @@ export function ImageCanvasEditorView() {
onOpenRedrawPanel: openRedrawPanel,
onOpenCropExpandPanel: openCropExpandPanel,
onRemoveBackground: removeSelectedLayerBackground,
onExtractUiDesignAssets: extractUiDesignAssets,
onOpenCharacterAnimationPanel: openCharacterAnimationPanel,
onDownloadLayer: exportLayerImage,
onPasteCanvasClipboard: pasteCanvasClipboard,
@@ -220,31 +220,42 @@ describe('ImageCanvasGenerationLayerModel', () => {
generationInputs: createGenerationInputs(),
});
expect(layers).toHaveLength(3);
expect(layers).toHaveLength(4);
expect(layers[0]).toMatchObject({
id: 'layer-icon-11',
title: '返回按钮',
id: 'layer-icon-spritesheet-11',
title: '图标素材图集 11',
x: 200,
y: 100,
width: 512,
height: 512,
assetKind: 'icon-spritesheet',
prompt: '图标 prompt',
actualPrompt: '图标 actual prompt',
});
expect(layers[1]).toMatchObject({
id: 'layer-icon-12',
title: '返回按钮',
x: 744,
y: 100,
width: 256,
height: 256,
assetKind: 'icon',
prompt: '图标 prompt',
actualPrompt: '图标 actual prompt',
});
expect(layers[1]).toMatchObject({
id: 'layer-icon-12',
expect(layers[2]).toMatchObject({
id: 'layer-icon-13',
title: '设置按钮',
x: 200,
x: 744,
y: 380,
width: 512,
height: 256,
assetKind: 'icon',
});
expect(layers[2]).toMatchObject({
id: 'layer-icon-13',
expect(layers[3]).toMatchObject({
id: 'layer-icon-14',
title: '提示按钮',
x: 200,
x: 744,
y: 660,
width: 128,
height: 128,
@@ -266,6 +277,7 @@ describe('ImageCanvasGenerationLayerModel', () => {
taskId: 'sound-task-1',
priceMudPoints: 10,
audioKind: 'sound-effect',
durationSeconds: 75,
},
generatedIndex: 14,
title: '游戏音效 14',
@@ -303,6 +315,7 @@ describe('ImageCanvasGenerationLayerModel', () => {
model: 'audio1.0',
provider: 'VectorEngine',
taskId: 'sound-task-1',
durationSeconds: 75,
});
});
@@ -13,7 +13,10 @@ import type {
CanvasViewport,
GenerateDialogState,
} from './ImageCanvasEditorTypes';
import { ICON_FRAME_DISPLAY_SIZE } from './ImageCanvasGenerationModel';
import {
ICON_FRAME_DISPLAY_SIZE,
ICON_FRAME_ORIGINAL_SIZE,
} from './ImageCanvasGenerationModel';
type CanvasSize = { width: number; height: number };
@@ -46,6 +49,7 @@ type IconSpritesheetResultLayerOptions = {
viewport: CanvasViewport;
generationInputs: CanvasGenerationInputs;
frame?: GenerateDialogState['placeholder'];
spritesheetTitle?: string;
};
type VideoResultLayerOptions = {
@@ -226,6 +230,7 @@ export function createIconSpritesheetResultLayers({
viewport,
generationInputs,
frame,
spritesheetTitle,
}: IconSpritesheetResultLayerOptions): CanvasLayer[] {
const worldCenter = getViewportWorldCenter({ canvasSize, viewport });
const startX =
@@ -234,11 +239,42 @@ export function createIconSpritesheetResultLayers({
frame?.y ?? worldCenter.y - ICON_FRAME_DISPLAY_SIZE.height / 2;
const spacing = 24;
const maxRowWidth = 560;
let cursorX = startX;
const spritesheetOriginalWidth =
generated.spritesheetWidth || ICON_FRAME_ORIGINAL_SIZE.width;
const spritesheetOriginalHeight =
generated.spritesheetHeight || ICON_FRAME_ORIGINAL_SIZE.height;
const spritesheetDisplaySize = resolveLayerResolutionSize(
spritesheetOriginalWidth,
spritesheetOriginalHeight,
ICON_FRAME_DISPLAY_SIZE,
);
const spritesheetLayer: CanvasLayer = {
id: `layer-icon-spritesheet-${startIndex}`,
resourceId: `local-resource-icon-spritesheet-${startIndex}`,
title: spritesheetTitle ?? `图标素材图集 ${startIndex}`,
src: generated.spritesheetImageSrc,
x: startX,
y: startY,
width: spritesheetDisplaySize.width,
height: spritesheetDisplaySize.height,
originalWidth: spritesheetOriginalWidth,
originalHeight: spritesheetOriginalHeight,
zIndex: startIndex + 10,
sourceType: 'generated',
prompt: generated.prompt,
actualPrompt: generated.actualPrompt ?? generated.prompt,
model: generated.model,
provider: generated.provider,
taskId: generated.taskId,
assetKind: 'icon-spritesheet',
generationInputs,
};
const iconStartX = startX + spritesheetDisplaySize.width + 32;
let cursorX = iconStartX;
let cursorY = startY;
let rowHeight = 0;
return iconResults.map((icon, index) => {
const iconLayers = iconResults.map((icon, index) => {
const originalWidth = icon.width || 128;
const originalHeight = icon.height || 128;
const { width, height } = resolveLayerResolutionSize(
@@ -246,13 +282,13 @@ export function createIconSpritesheetResultLayers({
originalHeight,
{ width: 128, height: 128 },
);
if (cursorX > startX && cursorX + width - startX > maxRowWidth) {
cursorX = startX;
if (cursorX > iconStartX && cursorX + width - iconStartX > maxRowWidth) {
cursorX = iconStartX;
cursorY += rowHeight + spacing;
rowHeight = 0;
}
const generatedIndex = startIndex + index;
const generatedIndex = startIndex + index + 1;
const layer: CanvasLayer = {
id: `layer-icon-${generatedIndex}`,
resourceId: `local-resource-icon-${generatedIndex}`,
@@ -279,6 +315,8 @@ export function createIconSpritesheetResultLayers({
rowHeight = Math.max(rowHeight, height);
return layer;
});
return [spritesheetLayer, ...iconLayers];
}
export function createVideoResultLayer({
@@ -377,6 +415,7 @@ export function createAudioResultLayer({
model: generated.model,
provider: generated.provider,
taskId: generated.taskId,
durationSeconds: generated.durationSeconds ?? undefined,
generationInputs,
};
}
@@ -66,6 +66,8 @@ export const DEFAULT_ICON_DESCRIPTIONS = [
'原图按钮',
'冻结按钮',
];
export const UI_DESIGN_ASSET_EXTRACTION_PROMPT =
'提取画面中的所有独立并整理成spritesheet';
export const QUICK_EDIT_SIZE_PRESETS = [
'1024x1024',
'1536x1024',
@@ -266,6 +268,9 @@ export function getLayerKindLabel(layer: CanvasLayer) {
if (layer.assetKind === 'icon') {
return '图标';
}
if (layer.assetKind === 'icon-spritesheet') {
return '图集';
}
if (layer.assetKind === 'icon-spec') {
return '图标规范';
}
@@ -316,6 +321,9 @@ export function formatLayerImageType(layer: CanvasLayer) {
if (layer.assetKind === 'icon') {
return '图标素材图片';
}
if (layer.assetKind === 'icon-spritesheet') {
return '图标素材图集';
}
if (layer.assetKind === 'icon-spec') {
return '图标规范图片';
}
@@ -578,6 +586,24 @@ export function buildIconGenerationInputs(
};
}
export function buildUiDesignAssetExtractionGenerationInputs(
sourceLayer: CanvasLayer,
): CanvasGenerationInputs {
return {
fields: createGenerationInputField(
'提取提示词',
UI_DESIGN_ASSET_EXTRACTION_PROMPT,
),
references: [
{
title: 'UI设计图',
label: sourceLayer.title,
src: sourceLayer.objectKey?.trim() || sourceLayer.src,
},
],
};
}
export function buildEditGenerationInputs(
title: '修改要求' | '快速编辑提示词' | '重绘提示词',
prompt: string,
@@ -131,6 +131,40 @@ describe('ImageCanvasMetadataModalView', () => {
expect(onClose).toHaveBeenCalledTimes(1);
});
it('renders generated audio metadata with duration instead of resolution', () => {
render(
<ImageCanvasMetadataModalView
layer={createLayer({
title: '游戏背景音乐',
src: '/generated-character-drafts/editor-audios/music.wav',
mediaType: 'audio',
assetKind: 'background-music',
originalWidth: 420,
originalHeight: 120,
model: 'suno',
taskId: 'audio-task-75',
durationSeconds: 75,
generationInputs: {
fields: [{ title: '音乐描述', value: '紧张的地下城循环音乐' }],
references: [],
},
})}
onClose={vi.fn()}
/>,
);
const dialog = screen.getByRole('dialog', { name: '音频信息' });
expect(within(dialog).getByText('音频类型')).toBeTruthy();
expect(within(dialog).getByText('游戏背景音乐')).toBeTruthy();
expect(within(dialog).getByText('音乐描述')).toBeTruthy();
expect(within(dialog).getByText('紧张的地下城循环音乐')).toBeTruthy();
expect(within(dialog).getByText('时长')).toBeTruthy();
expect(within(dialog).getByText('1:15')).toBeTruthy();
expect(within(dialog).queryByText('Resolution')).toBeNull();
expect(within(dialog).queryByText('420 x 120 px')).toBeNull();
});
it('does not render a dialog when no layer is selected', () => {
render(<ImageCanvasMetadataModalView layer={null} onClose={vi.fn()} />);
@@ -1,6 +1,7 @@
import { UnifiedModal } from '../common/UnifiedModal';
import { formatLayerImageType } from './ImageCanvasGenerationModel';
import type { CanvasLayer } from './ImageCanvasEditorTypes';
import { formatCanvasDurationMetric } from './ImageCanvasMediaModel';
export type ImageCanvasMetadataModalViewProps = {
layer: CanvasLayer | null;
@@ -77,10 +78,24 @@ export function ImageCanvasMetadataModalView({
</dd>
<dt>Model</dt>
<dd>{layer.model ?? '-'}</dd>
<dt>Resolution</dt>
<dd>
{layer.originalWidth} x {layer.originalHeight} px
</dd>
{layer.mediaType === 'audio' ? (
<>
<dt></dt>
<dd>
{formatCanvasDurationMetric(layer.durationSeconds).replace(
/^\s*/u,
'',
)}
</dd>
</>
) : (
<>
<dt>Resolution</dt>
<dd>
{layer.originalWidth} x {layer.originalHeight} px
</dd>
</>
)}
<dt>Task</dt>
<dd>{formatTaskIdForDisplay(layer.taskId)}</dd>
<dt>Object</dt>
@@ -40,6 +40,7 @@ function renderSelectedToolbar(
onOpenRedrawPanel: vi.fn(),
onOpenCropExpandPanel: vi.fn(),
onRemoveBackground: vi.fn(),
onExtractUiDesignAssets: vi.fn(),
onOpenCharacterAnimationPanel: vi.fn(),
onDownloadLayer: vi.fn(),
...overrides,
@@ -88,6 +89,32 @@ describe('ImageCanvasSelectedLayerToolbarView', () => {
expect(
within(toolbar).queryByRole('button', { name: /.*/u }),
).toBeNull();
expect(
within(toolbar).queryByRole('button', { name: '提取素材' }),
).toBeNull();
});
it('renders UI design asset extraction after remove background', () => {
const layer = createLayer({
sourceType: 'generated',
assetKind: 'ui-design',
});
const props = renderSelectedToolbar({ selectedLayer: layer });
const toolbar = screen.getByRole('toolbar', { name: '图片工具栏' });
const buttons = within(toolbar).getAllByRole('button');
expect(buttons.map((button) => button.getAttribute('aria-label'))).toEqual([
'快速编辑',
'裁扩按钮',
'去除背景按钮',
'提取素材',
'重绘',
'下载按钮',
]);
fireEvent.click(within(toolbar).getByRole('button', { name: '提取素材' }));
expect(props.onExtractUiDesignAssets).toHaveBeenCalledWith(layer);
});
it('renders character animation only for character layers', () => {
@@ -115,6 +142,7 @@ describe('ImageCanvasSelectedLayerToolbarView', () => {
onOpenRedrawPanel={vi.fn()}
onOpenCropExpandPanel={vi.fn()}
onRemoveBackground={vi.fn()}
onExtractUiDesignAssets={vi.fn()}
onOpenCharacterAnimationPanel={vi.fn()}
onDownloadLayer={vi.fn()}
/>,
@@ -130,6 +158,7 @@ describe('ImageCanvasSelectedLayerToolbarView', () => {
onOpenRedrawPanel={vi.fn()}
onOpenCropExpandPanel={vi.fn()}
onRemoveBackground={vi.fn()}
onExtractUiDesignAssets={vi.fn()}
onOpenCharacterAnimationPanel={vi.fn()}
onDownloadLayer={vi.fn()}
/>,

Some files were not shown because too many files have changed in this diff Show More