完善画板生成音乐与生成面板交互
新增画板底部生成音乐入口，支持游戏音效和游戏背景音乐生成面板。 接入编辑器音频生成前后端契约、VectorEngine 请求与 BFF 路由。 补齐音频结果图层、元数据、画布恢复和客户端提交流程。 统一生成类面板 Lovart 式参数浮层、参考素材入口与相关测试。 更新编辑器音乐生成设计文档和项目长期记忆。
This commit is contained in:
@@ -16,6 +16,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-18 图片画布生成音乐入口作为音频图层接入
|
||||
|
||||
- 背景：图片画布底部生成工具需要补齐游戏音效和游戏背景音乐生成，既要复用现有 Lovart 式画布生成器快照、占位避让和持久化，又不能把音频能力并入图片素材库或视觉小说专用音频开关。
|
||||
- 决策：`/editor/canvas` 新增底部 `生成音乐` 入口，点击后先弹出“生成游戏音效 / 生成游戏背景音乐”选项框，再分别创建 `audio-sound-effect` 或 `audio-background-music` 生成器；生成结果作为 `mediaType="audio"` 的画布音频卡保存，`assetKind` 分别为 `sound-effect` / `background-music`。音效请求字段固定映射 `sound/type/tempo`，背景音乐请求字段固定映射 `gpt_description_prompt` 且 `make_instrumental=true`。
|
||||
- 影响范围：图片画布生成工作流、前端 editorProjectClient、`shared-contracts`、`platform-audio`、`api-server` 编辑器音频 BFF、图片画布技术方案和音乐生成入口设计文档。
|
||||
- 验证方式：运行编辑器生成入口 / 提交 / 音频图层相关前端测试，`platform-audio` 请求体测试，`shared-contracts` editor audio 序列化测试，`api-server` editor audio 归一化测试，并执行 `npm run typecheck`、`npm run check:encoding`、`git diff --check`。
|
||||
- 关联文档：`docs/【编辑器】画板音乐生成入口设计-2026-06-18.md`、`docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md`、`docs/【编辑器】生成类面板Lovart统一改造方案-2026-06-17.md`。
|
||||
|
||||
## 2026-06-17 图片画布生成占位统一避让落点
|
||||
|
||||
- 背景：图片画布的普通图片、规范、角色、图标、视频和 UI 设计图生成入口都会先在画布中新建“即将生成”的占位图；若各入口直接使用当前视口中心，容易压住已有图片或已有生成占位，Lovart 式连续创作体验不稳定。
|
||||
|
||||
@@ -50,10 +50,10 @@
|
||||
## 图片编辑器生成类菜单要挂到页面级 portal
|
||||
|
||||
- 现象：底部 `生成规范` 菜单、角色面板里的 `角色形象规范` 来源菜单点击后像没有弹出来，实际被按钮所在的局部滚动容器挡住了。
|
||||
- 原因：菜单仍然渲染在底部工具栏或参考图横向滚动行内部，父容器带 `overflow`，弹层无法越出边界。
|
||||
- 处理：这类轻量菜单统一用页面级 fixed portal 挂到 `document.body`，位置根据触发按钮的 `getBoundingClientRect()` 计算；底部 AI 工具栏在生成面板打开时仍保持可见，不要整栏隐藏。
|
||||
- 验证：测试断言菜单不包含在底部工具栏 / 参考图行里，并且生成面板打开时底部 `AI画布工具栏` 仍存在。
|
||||
- 关联：`src/components/image-editor/ImageCanvasEditorView.tsx`、`src/components/image-editor/ImageCanvasEditorView.test.tsx`。
|
||||
- 原因：菜单仍然渲染在底部工具栏或参考图横向滚动行内部，父容器带 `overflow`，弹层无法越出边界；即便挂到 portal，如果菜单根节点的 `pointerdown` 继续冒泡到画布视口，也会先触发画布失焦并卸载面板，导致菜单项 `click` 前消失。
|
||||
- 处理：这类轻量菜单统一用页面级 fixed portal 挂到 `document.body`，位置根据触发按钮的 `getBoundingClientRect()` 计算；`PlatformFloatingMenu` 根节点必须阻止 `pointerdown` 冒泡，避免画布清空当前生成面板；底部 AI 工具栏在生成面板打开时仍保持可见，不要整栏隐藏。
|
||||
- 验证：测试断言菜单不包含在底部工具栏 / 参考图行里，并且生成面板打开时底部 `AI画布工具栏` 仍存在；规范参考图来源菜单应能通过 portal 点击“从画布中选择 / 上传图片”并写回规范参考图。
|
||||
- 关联：`src/components/common/PlatformFloatingMenu.tsx`、`src/components/image-editor/ImageCanvasEditorView.tsx`、`src/components/image-editor/ImageCanvasEditorGenerationIntegration.test.tsx`。
|
||||
|
||||
## 图片编辑器规范图片面板不要脱离统一生成 shell
|
||||
|
||||
|
||||
@@ -88,7 +88,7 @@
|
||||
- 拖拽图片接近其它图片边缘或中心时显示吸附线，并保存吸附后的最终布局。
|
||||
- 生成工具点击后显示画布内 `Image Generator` 占位框和跟随占位框的生成输入框，生成失败保留占位和输入状态，生成成功后在占位位置创建真实图层，并让输入框继续跟随该生成图。
|
||||
- 生成器快照刷新后必须恢复；待生成、生成中、失败和已生成后跟随成品图层的生成器都不能因为刷新丢失输入、参数、参考图或占位框位置。
|
||||
- 生成类入口打开画布内面板时，底部 AI 工具栏必须保持可见；`生成规范`、角色 / 图标规范来源、角色常规参考图来源这类轻量菜单通过页面级 fixed portal 渲染，不能留在底部工具栏或参考图横向滚动容器内部，避免被局部 `overflow` 裁切。角色形象规范和常规参考图来源菜单必须向上弹出；常规参考图点击后先选择“从画布中选择”或“上传图片”，从画布取图时只绑定参考图，不触发普通画布图层选中、聚焦、面板隐藏或拖拽逻辑，绑定后退出画布选择状态。
|
||||
- 生成类入口打开画布内面板时，底部 AI 工具栏必须保持可见；`生成规范`、角色 / 图标规范来源、角色常规参考图来源这类轻量菜单通过页面级 fixed portal 渲染，不能留在底部工具栏或参考图横向滚动容器内部，避免被局部 `overflow` 裁切。角色形象规范和常规参考图来源菜单必须向上弹出；常规参考图点击后先选择“从画布中选择”或“上传图片”，从画布取图时只绑定参考图，不触发普通画布图层选中、聚焦、面板隐藏或拖拽逻辑，绑定后退出画布选择状态。所有生成面板参考图槽位统一为方形图标组件；角色形象规范槽位只显示规范 logo 和 `角色规范` 四字，绑定来源标题只保留给可访问名称、悬浮 title 和图片信息。已有参考图槽位只有在 hover / focus 时显示右上角 `×`，点击后只解绑对应参考图。
|
||||
- 点击生成、生成规范、生成角色形象或生成图标素材后创建的占位图可继续保留；点击画布空白区域让当前图片或占位图失焦时，关闭当前生成面板并移除图片选中样式，但不删除占位图本身。
|
||||
- 生成资源显示元数据按钮，元数据窗口展示来源、生成输入快照、model、provider、task、Resolution 和 OSS 引用；生成输入快照只包含用户面板输入和参考图，不包含后端拼接 Prompt，不再展示独立 Size 字段。
|
||||
- 修改生成资源后，右侧出现新生成结果图层，并自动 fit 原图 + 新图。
|
||||
|
||||
@@ -18,21 +18,23 @@
|
||||
## 统一布局
|
||||
|
||||
1. 参考图区域永远位于面板第一行。
|
||||
2. 参考图使用统一参考图卡片：
|
||||
2. 参考图使用统一方形参考图图标组件，不再使用横向长卡：
|
||||
- 普通参考图：灰蓝色图片图标。
|
||||
- 规范参考图：紫色剪贴板图标。
|
||||
- 规范参考图：紫色规范 logo。
|
||||
- 图标素材规范：绿色图标图标。
|
||||
- UI 图标规范：琥珀色图标图标。
|
||||
- 视频参考图：深色视频图标。
|
||||
3. 参考图卡片尽量少文字；必要文字写在图标块内或短标签内，不写规则说明。
|
||||
4. 普通参考图支持连续追加：已有图缩略图后始终保留一个 `+` 入口。点击入口只弹出“从画布中选择 / 上传图片”来源选项，不再直接打开系统文件选择器；生成图片、生成视频、角色常规参考图等同类参考图入口都遵循同一交互。
|
||||
5. 单文本输入面板不显示文本框标题，用问题式 placeholder：
|
||||
3. 角色形象规范参考图组件只展示一个方形参考图图标；图标内上方是规范 logo，下方固定短标 `角色规范`，不再展示绑定状态、来源说明或长标题。已有绑定图片的原始标题只作为可访问名称、悬浮 title 和图片信息回看使用。
|
||||
4. 参考图图标尽量少文字；必要文字写在图标块内或短标签内，不写规则说明。
|
||||
5. 已有参考图在鼠标悬停或键盘聚焦到对应参考图槽位时，右上角显示一个 `×` 删除按钮；鼠标不在槽位上时不显示。点击 `×` 只移除该参考图绑定，不触发来源菜单、不删除画布图片。
|
||||
6. 普通参考图支持连续追加：已有图缩略图后始终保留一个 `+` 入口。点击入口只弹出“从画布中选择 / 上传图片”来源选项，不再直接打开系统文件选择器；生成图片、生成视频、角色常规参考图等同类参考图入口都遵循同一交互。
|
||||
7. 单文本输入面板不显示文本框标题，用问题式 placeholder：
|
||||
- 生成图片：`今天想生成什么画面？`
|
||||
- 生成角色：`你希望角色如何设计？`
|
||||
- 生成 UI：`你希望这个 UI 长什么样？`
|
||||
- 生成视频：`你希望生成什么视频？`
|
||||
6. 多输入框面板必须保留每个字段标题和输入框边界，例如生成规范、图标素材多描述。
|
||||
7. 生成规范下的角色形象规范、UI素材规范、图标素材规范和自定义规范都使用同一生成类 shell：首行参考图区域、中央字段区、底部生成按钮区，不再出现缺首行参考区或单独 footer 样式。
|
||||
8. 多输入框面板必须保留每个字段标题和输入框边界，例如生成规范、图标素材多描述。
|
||||
9. 生成规范下的角色形象规范、UI素材规范、图标素材规范和自定义规范都使用同一生成类 shell：首行参考图区域、中央字段区、底部生成按钮区，不再出现缺首行参考区或单独 footer 样式。
|
||||
|
||||
## 参数交互
|
||||
|
||||
@@ -47,6 +49,7 @@
|
||||
- 模型选项框和生成按钮位于右下角。
|
||||
- 点击后以页面级浮层向上弹出独立选项面板；父级面板隐藏或销毁时，选项面板同步销毁。
|
||||
- 点击选项只更新字段和选中样式，不收起选项面板；再次点击当前参数按钮可收起。
|
||||
- 选项面板打开后，点击父级生成面板内任意非选项框、非触发按钮区域必须收起；点击选项面板自身仍保持打开，便于连续修改。
|
||||
- 弹出面板内可分组展示字段，点击某个选项只更新当前字段和选中样式，方便连续修改。
|
||||
- 模型子面板每行固定一个模型，不用方框包裹模型名；模型名不换行，前置对应模型类型图标，选中项在模型名后用对号标记。
|
||||
- 比例选项卡片内展示对应比例的线框。
|
||||
@@ -115,6 +118,7 @@
|
||||
## 验收
|
||||
|
||||
- 所有生成类面板首行都是参考图区域。
|
||||
- 参考图槽位统一是方形图标；角色形象规范槽位显示规范 logo 和 `角色规范` 四字，悬停已有参考图时才显示右上角删除 `×`。
|
||||
- 比例 / 尺寸 / 模型不再平铺全部选项；比例与尺寸合并为左下角当前值按钮，模型与生成按钮位于右下角。
|
||||
- 单文本输入面板不显示字段标题，placeholder 是问题式文案。
|
||||
- 多文本输入面板字段标题和边界仍清晰。
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
# 画板音乐生成入口设计
|
||||
|
||||
日期：`2026-06-18`
|
||||
|
||||
## 范围
|
||||
|
||||
本次只在 `/editor/canvas` 图片画布编辑器内新增底部 `生成音乐` 入口，用于生成完整游戏音效或游戏背景音乐。该入口属于画板生成类工具，不新增平台玩法入口、不进入作品发布链路，也不修改现有视觉小说音频生成开关。
|
||||
|
||||
## 入口与交互
|
||||
|
||||
1. 底部 AI 画布工具栏新增 `生成音乐`。
|
||||
2. 点击 `生成音乐` 后先弹出页面级 fixed 选项框，选项为：
|
||||
- `生成游戏音效`
|
||||
- `生成游戏背景音乐`
|
||||
3. 选择某一项后创建独立 `generation-dialog` 画布生成对象，并通过现有 placement 模型避让已有图层和占位。
|
||||
4. 面板 UI 复用 `生成角色形象` 的紧凑结构：上方为字段区，底部为参数 / 生成按钮区，不写规则说明类文案。
|
||||
5. 生成中隐藏设置面板，只保留画布中的音频生成占位；失败后恢复面板并展示短错误。
|
||||
|
||||
## 面板字段
|
||||
|
||||
### 生成游戏音效
|
||||
|
||||
- `sound`：用户输入的音效提示词。
|
||||
- `type`：合成选项，第一版以前端下拉 / 子面板保存字符串。
|
||||
- `tempo`：合成选项，第一版以前端下拉 / 子面板保存字符串。
|
||||
- 提交时映射到 VectorEngine 游戏音效接口字段：`sound`、`type`、`tempo`。
|
||||
|
||||
### 生成游戏背景音乐
|
||||
|
||||
- `gpt_description_prompt`：用户输入的背景音乐提示词。
|
||||
- `make_instrumental`：固定传 `true`，不在 UI 中展示为可改字段。
|
||||
- 提交时映射到 VectorEngine 游戏背景音乐接口字段：`gpt_description_prompt`、`make_instrumental: true`。
|
||||
|
||||
## 画布数据
|
||||
|
||||
- 生成器继续保存到画布 layout JSON 的 `itemType: "generation-dialog"`，不新增表。
|
||||
- 新增生成器模式：
|
||||
- `audio-sound-effect`
|
||||
- `audio-background-music`
|
||||
- 新增结果图层媒体类型 `mediaType="audio"`。
|
||||
- 新增素材类型：
|
||||
- `assetKind="sound-effect"`
|
||||
- `assetKind="background-music"`
|
||||
- 音频结果以小型音频卡加入画布，卡片内使用 `<audio controls>` 播放；音频结果当前只保存为画布资源，不进入图片素材库。
|
||||
- 元数据弹窗按音频显示 `音频信息` / `音频类型`，生成输入快照只展示用户面板字段。
|
||||
|
||||
## 前端提交契约
|
||||
|
||||
前端新增两个 BFF client：
|
||||
|
||||
```ts
|
||||
POST /api/editor/audios/sound-effects/generations
|
||||
{
|
||||
sound: string,
|
||||
type: string,
|
||||
tempo: string,
|
||||
priceMudPoints: 10
|
||||
}
|
||||
```
|
||||
|
||||
```ts
|
||||
POST /api/editor/audios/background-music/generations
|
||||
{
|
||||
gptDescriptionPrompt: string,
|
||||
makeInstrumental: true,
|
||||
priceMudPoints: 5
|
||||
}
|
||||
```
|
||||
|
||||
统一响应：
|
||||
|
||||
```ts
|
||||
{
|
||||
ok: true,
|
||||
audioSrc: string,
|
||||
width: 420,
|
||||
height: 120,
|
||||
sourceType: "generated",
|
||||
prompt: string,
|
||||
actualPrompt?: string | null,
|
||||
model: string,
|
||||
provider: string,
|
||||
taskId: string,
|
||||
priceMudPoints: number,
|
||||
audioKind: "sound-effect" | "background-music"
|
||||
}
|
||||
```
|
||||
|
||||
## 后端实现
|
||||
|
||||
- 在 `shared-contracts/src/assets.rs` 增加编辑器音频请求 / 响应 DTO。
|
||||
- 在 `platform-audio` 增加编辑器专用 body builder 和 submit 函数：
|
||||
- 背景音乐 body 使用 `gpt_description_prompt`、`make_instrumental`。
|
||||
- 音效 body 使用 `sound`、`type`、`tempo`。
|
||||
- 在 `api-server` 增加编辑器音频 BFF：
|
||||
- `/api/editor/audios/sound-effects/generations`
|
||||
- `/api/editor/audios/background-music/generations`
|
||||
- BFF 复用现有 `vector_engine_audio_generation` 的任务轮询、下载、OSS 持久化和计费包装；音效 10 泥点，背景音乐 5 泥点。
|
||||
|
||||
## 验收
|
||||
|
||||
- 底部工具栏显示 `生成音乐`。
|
||||
- 点击 `生成音乐` 只出现选项框，不立刻创建占位。
|
||||
- 点击 `生成游戏音效` 后出现音效面板，字段为 `sound`、`type`、`tempo`。
|
||||
- 点击 `生成游戏背景音乐` 后出现背景音乐面板，字段为 `gpt_description_prompt`，不展示 `make_instrumental`。
|
||||
- 音效提交到 `/api/editor/audios/sound-effects/generations`，背景音乐提交到 `/api/editor/audios/background-music/generations`。
|
||||
- 成功后画布新增音频卡，能通过 `<audio controls>` 播放。
|
||||
- 刷新后 layout 能恢复音频生成器和音频图层。
|
||||
|
||||
@@ -98,6 +98,11 @@ const EDITOR_VIDEO_MODEL_KLING_3: &str = "kling3.0";
|
||||
const EDITOR_VIDEO_MODEL_KLING_3_OMNI: &str = "kling3.0-omni";
|
||||
const EDITOR_VIDEO_MODEL_VEO_3_1: &str = "veo3.1";
|
||||
const EDITOR_VIDEO_MODEL_VEO_3_1_FAST: &str = "veo3.1-fast";
|
||||
const EDITOR_VIDEO_REFERENCE_IMAGE_LIMIT: usize = 9;
|
||||
const EDITOR_VIDEO_REFERENCE_VIDEO_LIMIT: usize = 3;
|
||||
const EDITOR_VIDEO_REFERENCE_AUDIO_LIMIT: usize = 3;
|
||||
const EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES: usize = 64 * 1024 * 1024;
|
||||
const EDITOR_VIDEO_REFERENCE_READ_EXPIRE_SECONDS: u64 = 60 * 60;
|
||||
|
||||
const BUILT_IN_MOTION_TEMPLATES: [MotionTemplate; 4] = [
|
||||
MotionTemplate {
|
||||
@@ -1448,7 +1453,7 @@ async fn request_editor_video_preview(
|
||||
request: &NormalizedEditorVideoRequest,
|
||||
) -> Result<GeneratedAnimationPreview, AppError> {
|
||||
let upstream_task_id =
|
||||
create_editor_text_to_video_task(http_client, settings, request).await?;
|
||||
create_editor_text_to_video_task(state, http_client, settings, request).await?;
|
||||
let video_url =
|
||||
wait_for_ark_content_generation_task(http_client, &settings.ark, upstream_task_id.as_str())
|
||||
.await?;
|
||||
@@ -1466,10 +1471,12 @@ async fn request_editor_video_preview(
|
||||
}
|
||||
|
||||
async fn create_editor_text_to_video_task(
|
||||
state: &AppState,
|
||||
http_client: &reqwest::Client,
|
||||
settings: &EditorVideoSettings,
|
||||
request: &NormalizedEditorVideoRequest,
|
||||
) -> Result<String, AppError> {
|
||||
let content = build_editor_video_ark_content(state, request)?;
|
||||
let response = http_client
|
||||
.post(format!("{}/contents/generations/tasks", settings.ark.base_url))
|
||||
.header(
|
||||
@@ -1479,16 +1486,11 @@ async fn create_editor_text_to_video_task(
|
||||
.header(reqwest::header::CONTENT_TYPE, "application/json")
|
||||
.json(&json!({
|
||||
"model": request.provider_model,
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": request.prompt,
|
||||
}
|
||||
],
|
||||
"content": content,
|
||||
"resolution": request.resolution,
|
||||
"ratio": request.aspect_ratio,
|
||||
"duration": request.duration_seconds,
|
||||
"mode": "std",
|
||||
"generate_audio": false,
|
||||
"watermark": false,
|
||||
}))
|
||||
.send()
|
||||
@@ -2417,6 +2419,12 @@ fn normalize_editor_video_request(
|
||||
"priceMudPoints 与分辨率和时长不一致，应为 {expected_price}。"
|
||||
)));
|
||||
}
|
||||
let references = normalize_editor_video_references(
|
||||
model,
|
||||
payload.reference_image_srcs,
|
||||
payload.reference_video_srcs,
|
||||
payload.reference_audio_srcs,
|
||||
)?;
|
||||
let (width, height) = resolve_editor_video_size(aspect_ratio, resolution);
|
||||
|
||||
Ok(NormalizedEditorVideoRequest {
|
||||
@@ -2429,9 +2437,228 @@ fn normalize_editor_video_request(
|
||||
price_mud_points: expected_price,
|
||||
width,
|
||||
height,
|
||||
reference_image_srcs: references.reference_image_srcs,
|
||||
reference_video_srcs: references.reference_video_srcs,
|
||||
reference_audio_srcs: references.reference_audio_srcs,
|
||||
})
|
||||
}
|
||||
|
||||
fn build_editor_video_ark_content(
|
||||
state: &AppState,
|
||||
request: &NormalizedEditorVideoRequest,
|
||||
) -> Result<Vec<Value>, AppError> {
|
||||
let mut content = Vec::new();
|
||||
if !request.prompt.trim().is_empty() {
|
||||
content.push(json!({
|
||||
"type": "text",
|
||||
"text": request.prompt,
|
||||
}));
|
||||
}
|
||||
let reference_image_urls = resolve_editor_video_reference_urls(
|
||||
state,
|
||||
request.reference_image_srcs.as_slice(),
|
||||
"参考图片",
|
||||
)?;
|
||||
let reference_video_urls = resolve_editor_video_reference_urls(
|
||||
state,
|
||||
request.reference_video_srcs.as_slice(),
|
||||
"参考视频",
|
||||
)?;
|
||||
let reference_audio_urls = resolve_editor_video_reference_urls(
|
||||
state,
|
||||
request.reference_audio_srcs.as_slice(),
|
||||
"参考音频",
|
||||
)?;
|
||||
content.extend(reference_image_urls.iter().map(|url| {
|
||||
json!({
|
||||
"type": "image_url",
|
||||
"image_url": { "url": url },
|
||||
"role": "reference_image",
|
||||
})
|
||||
}));
|
||||
content.extend(reference_video_urls.iter().map(|url| {
|
||||
json!({
|
||||
"type": "video_url",
|
||||
"video_url": { "url": url },
|
||||
"role": "reference_video",
|
||||
})
|
||||
}));
|
||||
content.extend(reference_audio_urls.iter().map(|url| {
|
||||
json!({
|
||||
"type": "audio_url",
|
||||
"audio_url": { "url": url },
|
||||
"role": "reference_audio",
|
||||
})
|
||||
}));
|
||||
Ok(content)
|
||||
}
|
||||
|
||||
fn resolve_editor_video_reference_urls(
|
||||
state: &AppState,
|
||||
sources: &[String],
|
||||
label: &str,
|
||||
) -> Result<Vec<String>, AppError> {
|
||||
sources
|
||||
.iter()
|
||||
.map(|source| resolve_editor_video_reference_url(state, source.as_str(), label))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn resolve_editor_video_reference_url(
|
||||
state: &AppState,
|
||||
source: &str,
|
||||
label: &str,
|
||||
) -> Result<String, AppError> {
|
||||
let value = source.trim();
|
||||
if value.starts_with("data:") || value.starts_with("asset://") {
|
||||
return Ok(value.to_string());
|
||||
}
|
||||
if value.starts_with("http://") || value.starts_with("https://") {
|
||||
return Ok(value.to_string());
|
||||
}
|
||||
|
||||
let object_key = value.trim_start_matches('/');
|
||||
if LegacyAssetPrefix::from_object_key(object_key).is_none() {
|
||||
return Err(editor_video_bad_request(format!(
|
||||
"{label}必须是官方支持的公网 URL、素材 ID、data URL 或画板资源路径。"
|
||||
)));
|
||||
}
|
||||
let oss_client = state.oss_client().ok_or_else(|| {
|
||||
AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({
|
||||
"provider": "aliyun-oss",
|
||||
"reason": "OSS 未完成环境变量配置，无法为画板参考素材生成可访问 URL。",
|
||||
}))
|
||||
})?;
|
||||
let signed = oss_client
|
||||
.sign_get_object_url(OssSignedGetObjectUrlRequest {
|
||||
object_key: object_key.to_string(),
|
||||
expire_seconds: Some(EDITOR_VIDEO_REFERENCE_READ_EXPIRE_SECONDS),
|
||||
})
|
||||
.map_err(|error| map_oss_error(error, "aliyun-oss"))?;
|
||||
Ok(signed.signed_url)
|
||||
}
|
||||
|
||||
struct NormalizedEditorVideoReferences {
|
||||
reference_image_srcs: Vec<String>,
|
||||
reference_video_srcs: Vec<String>,
|
||||
reference_audio_srcs: Vec<String>,
|
||||
}
|
||||
|
||||
fn normalize_editor_video_references(
|
||||
model: &str,
|
||||
image_srcs: Vec<String>,
|
||||
video_srcs: Vec<String>,
|
||||
audio_srcs: Vec<String>,
|
||||
) -> Result<NormalizedEditorVideoReferences, AppError> {
|
||||
let is_seedance =
|
||||
model == EDITOR_VIDEO_MODEL_SEEDANCE_2 || model == EDITOR_VIDEO_MODEL_SEEDANCE_2_FAST;
|
||||
let image_srcs = normalize_reference_srcs(image_srcs);
|
||||
let video_srcs = normalize_reference_srcs(video_srcs);
|
||||
let audio_srcs = normalize_reference_srcs(audio_srcs);
|
||||
|
||||
if !is_seedance {
|
||||
if image_srcs.is_empty() && video_srcs.is_empty() && audio_srcs.is_empty() {
|
||||
return Ok(NormalizedEditorVideoReferences {
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: Vec::new(),
|
||||
});
|
||||
}
|
||||
return Err(editor_video_bad_request(
|
||||
"参考图片、参考视频和参考音频仅支持 Seedance 2.0 系列模型。",
|
||||
));
|
||||
}
|
||||
if image_srcs.len() > EDITOR_VIDEO_REFERENCE_IMAGE_LIMIT {
|
||||
return Err(editor_video_bad_request("参考图片最多 9 张。"));
|
||||
}
|
||||
if video_srcs.len() > EDITOR_VIDEO_REFERENCE_VIDEO_LIMIT {
|
||||
return Err(editor_video_bad_request("参考视频最多 3 个。"));
|
||||
}
|
||||
if audio_srcs.len() > EDITOR_VIDEO_REFERENCE_AUDIO_LIMIT {
|
||||
return Err(editor_video_bad_request("参考音频最多 3 个。"));
|
||||
}
|
||||
if !audio_srcs.is_empty() && image_srcs.is_empty() && video_srcs.is_empty() {
|
||||
return Err(editor_video_bad_request(
|
||||
"参考音频必须搭配参考图片或参考视频。",
|
||||
));
|
||||
}
|
||||
|
||||
for src in &image_srcs {
|
||||
validate_editor_video_reference_src(
|
||||
src,
|
||||
"参考图片",
|
||||
&["data:image/", "http://", "https://", "asset://"],
|
||||
true,
|
||||
"对应媒体 data URL",
|
||||
)?;
|
||||
}
|
||||
for src in &video_srcs {
|
||||
validate_editor_video_reference_src(
|
||||
src,
|
||||
"参考视频",
|
||||
&["http://", "https://", "asset://"],
|
||||
true,
|
||||
"URL、素材 ID 或画板资源路径",
|
||||
)?;
|
||||
}
|
||||
for src in &audio_srcs {
|
||||
validate_editor_video_reference_src(
|
||||
src,
|
||||
"参考音频",
|
||||
&["data:audio/", "http://", "https://", "asset://"],
|
||||
true,
|
||||
"对应媒体 data URL",
|
||||
)?;
|
||||
}
|
||||
|
||||
let request_bytes = image_srcs
|
||||
.iter()
|
||||
.chain(video_srcs.iter())
|
||||
.chain(audio_srcs.iter())
|
||||
.map(String::len)
|
||||
.sum::<usize>();
|
||||
if request_bytes > EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES {
|
||||
return Err(editor_video_bad_request("参考文件总请求体不能超过 64MB。"));
|
||||
}
|
||||
|
||||
Ok(NormalizedEditorVideoReferences {
|
||||
reference_image_srcs: image_srcs,
|
||||
reference_video_srcs: video_srcs,
|
||||
reference_audio_srcs: audio_srcs,
|
||||
})
|
||||
}
|
||||
|
||||
fn normalize_reference_srcs(values: Vec<String>) -> Vec<String> {
|
||||
values
|
||||
.into_iter()
|
||||
.map(|value| value.trim().to_string())
|
||||
.filter(|value| !value.is_empty())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn validate_editor_video_reference_src(
|
||||
value: &str,
|
||||
label: &str,
|
||||
allowed_prefixes: &[&str],
|
||||
allow_generated_object_key: bool,
|
||||
format_hint: &str,
|
||||
) -> Result<(), AppError> {
|
||||
if allowed_prefixes
|
||||
.iter()
|
||||
.any(|prefix| value.starts_with(prefix))
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
if allow_generated_object_key
|
||||
&& LegacyAssetPrefix::from_object_key(value.trim_start_matches('/')).is_some()
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
Err(editor_video_bad_request(format!(
|
||||
"{label}格式不支持，请使用官方支持的{format_hint}。"
|
||||
)))
|
||||
}
|
||||
|
||||
fn normalize_editor_video_model(value: &str) -> Result<&'static str, AppError> {
|
||||
match value.trim() {
|
||||
EDITOR_VIDEO_MODEL_SEEDANCE_2 => Ok(EDITOR_VIDEO_MODEL_SEEDANCE_2),
|
||||
@@ -4185,6 +4412,9 @@ struct NormalizedEditorVideoRequest {
|
||||
price_mud_points: u32,
|
||||
width: u32,
|
||||
height: u32,
|
||||
reference_image_srcs: Vec<String>,
|
||||
reference_video_srcs: Vec<String>,
|
||||
reference_audio_srcs: Vec<String>,
|
||||
}
|
||||
|
||||
struct GeneratedAnimationPreview {
|
||||
@@ -4501,6 +4731,9 @@ mod tests {
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 50,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: Vec::new(),
|
||||
})
|
||||
.expect("editor video request should normalize");
|
||||
|
||||
@@ -4523,6 +4756,9 @@ mod tests {
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 80,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: Vec::new(),
|
||||
})
|
||||
.expect("seedance fast request should normalize");
|
||||
|
||||
@@ -4542,6 +4778,9 @@ mod tests {
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 80,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: Vec::new(),
|
||||
})
|
||||
.expect("seedance standard request should normalize");
|
||||
|
||||
@@ -4550,6 +4789,74 @@ mod tests {
|
||||
assert_ne!(normalized.provider_model, CHARACTER_ANIMATION_MODEL);
|
||||
}
|
||||
|
||||
|
||||
#[test]
|
||||
fn editor_video_normalizes_seedance_multimodal_references() {
|
||||
let normalized = normalize_editor_video_request(EditorVideoGenerateRequest {
|
||||
prompt: "参考素材生成视频".to_string(),
|
||||
model: EDITOR_VIDEO_MODEL_SEEDANCE_2_FAST.to_string(),
|
||||
aspect_ratio: "16:9".to_string(),
|
||||
duration_seconds: 5,
|
||||
resolution: "720p".to_string(),
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 100,
|
||||
reference_image_srcs: vec!["data:image/png;base64,image".to_string()],
|
||||
reference_video_srcs: vec![
|
||||
"generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(),
|
||||
],
|
||||
reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()],
|
||||
})
|
||||
.expect("seedance multimodal request should normalize");
|
||||
|
||||
assert_eq!(normalized.reference_image_srcs, vec!["data:image/png;base64,image"]);
|
||||
assert_eq!(
|
||||
normalized.reference_video_srcs,
|
||||
vec!["generated-character-drafts/editor/seedance-references/video/demo.mp4"]
|
||||
);
|
||||
assert_eq!(normalized.reference_audio_srcs, vec!["data:audio/mpeg;base64,audio"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_video_rejects_seedance_video_data_urls() {
|
||||
let error = normalize_editor_video_request(EditorVideoGenerateRequest {
|
||||
prompt: "参考视频生成视频".to_string(),
|
||||
model: EDITOR_VIDEO_MODEL_SEEDANCE_2_FAST.to_string(),
|
||||
aspect_ratio: "16:9".to_string(),
|
||||
duration_seconds: 4,
|
||||
resolution: "480p".to_string(),
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 40,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: vec!["data:video/mp4;base64,video".to_string()],
|
||||
reference_audio_srcs: Vec::new(),
|
||||
})
|
||||
.expect_err("Seedance reference videos must not accept Base64 data URLs");
|
||||
|
||||
assert!(error.body_text().contains("参考视频格式不支持"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_video_rejects_audio_only_seedance_references() {
|
||||
let error = normalize_editor_video_request(EditorVideoGenerateRequest {
|
||||
prompt: "根据旁白生成视频".to_string(),
|
||||
model: EDITOR_VIDEO_MODEL_SEEDANCE_2.to_string(),
|
||||
aspect_ratio: "16:9".to_string(),
|
||||
duration_seconds: 4,
|
||||
resolution: "480p".to_string(),
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 40,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: vec!["data:audio/wav;base64,audio".to_string()],
|
||||
})
|
||||
.expect_err("audio-only references should fail");
|
||||
|
||||
assert!(error.body_text().contains("参考音频必须搭配参考图片或参考视频"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_video_rejects_price_mismatch() {
|
||||
let error = normalize_editor_video_request(EditorVideoGenerateRequest {
|
||||
@@ -4561,6 +4868,9 @@ mod tests {
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 40,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: Vec::new(),
|
||||
})
|
||||
.expect_err("wrong price should fail");
|
||||
|
||||
|
||||
@@ -17,8 +17,8 @@ use crate::{
|
||||
auth::require_bearer_auth,
|
||||
character_animation_assets::{
|
||||
generate_character_animation, generate_editor_character_animation, generate_editor_video,
|
||||
get_character_animation_job, get_character_workflow_cache, import_character_animation_video,
|
||||
list_character_animation_templates,
|
||||
get_character_animation_job, get_character_workflow_cache,
|
||||
import_character_animation_video, list_character_animation_templates,
|
||||
publish_character_animation, put_role_asset_workflow, resolve_role_asset_workflow,
|
||||
save_character_workflow_cache,
|
||||
},
|
||||
@@ -46,10 +46,14 @@ use crate::{
|
||||
runtime_save::{list_profile_save_archives, resume_profile_save_archive},
|
||||
runtime_settings::{get_runtime_settings, put_runtime_settings},
|
||||
state::AppState,
|
||||
vector_engine_audio_generation::{
|
||||
generate_editor_background_music, generate_editor_sound_effect,
|
||||
},
|
||||
};
|
||||
|
||||
const HYPER3D_IMAGE_TO_MODEL_BODY_LIMIT_BYTES: usize = 56 * 1024 * 1024;
|
||||
const EDITOR_CHARACTER_ANIMATION_BODY_LIMIT_BYTES: usize = 12 * 1024 * 1024;
|
||||
const EDITOR_VIDEO_BODY_LIMIT_BYTES: usize = 64 * 1024 * 1024;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) struct PlayFlowDomainAdapter {
|
||||
@@ -463,7 +467,23 @@ fn play_flow_support_router(state: AppState) -> Router<AppState> {
|
||||
)
|
||||
.route(
|
||||
"/api/editor/videos/generations",
|
||||
post(generate_editor_video).route_layer(middleware::from_fn_with_state(
|
||||
post(generate_editor_video)
|
||||
.layer(DefaultBodyLimit::max(EDITOR_VIDEO_BODY_LIMIT_BYTES))
|
||||
.route_layer(middleware::from_fn_with_state(
|
||||
state.clone(),
|
||||
require_bearer_auth,
|
||||
)),
|
||||
)
|
||||
.route(
|
||||
"/api/editor/audios/sound-effects/generations",
|
||||
post(generate_editor_sound_effect).route_layer(middleware::from_fn_with_state(
|
||||
state.clone(),
|
||||
require_bearer_auth,
|
||||
)),
|
||||
)
|
||||
.route(
|
||||
"/api/editor/audios/background-music/generations",
|
||||
post(generate_editor_background_music).route_layer(middleware::from_fn_with_state(
|
||||
state.clone(),
|
||||
require_bearer_auth,
|
||||
)),
|
||||
|
||||
@@ -11,6 +11,7 @@ mod tasks;
|
||||
mod tests;
|
||||
mod types;
|
||||
|
||||
pub use generation::{generate_editor_background_music, generate_editor_sound_effect};
|
||||
pub use handlers::{
|
||||
create_background_music_task, create_sound_effect_task,
|
||||
create_visual_novel_background_music_task, create_visual_novel_sound_effect_task,
|
||||
|
||||
@@ -1,18 +1,229 @@
|
||||
use axum::Extension;
|
||||
use axum::http::StatusCode;
|
||||
use axum::{
|
||||
Json,
|
||||
extract::{State, rejection::JsonRejection},
|
||||
response::Response,
|
||||
};
|
||||
use platform_oss::LegacyAssetPrefix;
|
||||
use serde_json::Value;
|
||||
use serde_json::json;
|
||||
use shared_contracts::creation_audio;
|
||||
use shared_contracts::{assets, creation_audio};
|
||||
|
||||
use crate::{
|
||||
http_error::AppError, state::AppState, tracking::record_external_generation_run_after_success,
|
||||
api_response::json_success_body, auth::AuthenticatedAccessToken, http_error::AppError,
|
||||
request_context::RequestContext, state::AppState,
|
||||
tracking::record_external_generation_run_after_success,
|
||||
};
|
||||
|
||||
use super::{
|
||||
clock::{current_utc_iso_text, current_utc_micros},
|
||||
errors::{map_platform_audio_error, vector_engine_bad_gateway},
|
||||
errors::{map_platform_audio_error, parse_json_payload, vector_engine_bad_gateway},
|
||||
publish::wait_for_generated_audio_asset,
|
||||
settings::require_vector_engine_audio_settings,
|
||||
tasks::create_sound_effect_task_response,
|
||||
types::{AudioAssetBindingTarget, AudioAssetSlot, GeneratedCreationAudioTarget},
|
||||
};
|
||||
|
||||
const EDITOR_SOUND_EFFECT_POINTS_COST: u32 = 10;
|
||||
const EDITOR_BACKGROUND_MUSIC_POINTS_COST: u32 = 5;
|
||||
const EDITOR_AUDIO_WIDTH: u32 = 420;
|
||||
const EDITOR_AUDIO_HEIGHT: u32 = 120;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(super) struct NormalizedEditorSoundEffectRequest {
|
||||
pub(super) sound: String,
|
||||
pub(super) sound_type: String,
|
||||
pub(super) tempo: String,
|
||||
pub(super) price_mud_points: u32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(super) struct NormalizedEditorBackgroundMusicRequest {
|
||||
pub(super) gpt_description_prompt: String,
|
||||
pub(super) make_instrumental: bool,
|
||||
pub(super) price_mud_points: u32,
|
||||
}
|
||||
|
||||
pub(super) fn normalize_editor_sound_effect_request(
|
||||
payload: assets::EditorSoundEffectGenerateRequest,
|
||||
) -> Result<NormalizedEditorSoundEffectRequest, AppError> {
|
||||
if payload.price_mud_points != EDITOR_SOUND_EFFECT_POINTS_COST {
|
||||
return Err(editor_audio_bad_request("音效生成泥点消耗不匹配"));
|
||||
}
|
||||
Ok(NormalizedEditorSoundEffectRequest {
|
||||
sound: platform_audio::normalize_limited_text(
|
||||
&payload.sound,
|
||||
"sound",
|
||||
platform_audio::VIDU_PROMPT_MAX_CHARS,
|
||||
)
|
||||
.map_err(map_platform_audio_error)?,
|
||||
sound_type: platform_audio::normalize_limited_text(&payload.sound_type, "type", 80)
|
||||
.map_err(map_platform_audio_error)?,
|
||||
tempo: platform_audio::normalize_limited_text(&payload.tempo, "tempo", 80)
|
||||
.map_err(map_platform_audio_error)?,
|
||||
price_mud_points: payload.price_mud_points,
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn normalize_editor_background_music_request(
|
||||
payload: assets::EditorBackgroundMusicGenerateRequest,
|
||||
) -> Result<NormalizedEditorBackgroundMusicRequest, AppError> {
|
||||
if payload.price_mud_points != EDITOR_BACKGROUND_MUSIC_POINTS_COST {
|
||||
return Err(editor_audio_bad_request("背景音乐生成泥点消耗不匹配"));
|
||||
}
|
||||
Ok(NormalizedEditorBackgroundMusicRequest {
|
||||
gpt_description_prompt: platform_audio::normalize_limited_text(
|
||||
&payload.gpt_description_prompt,
|
||||
"gpt_description_prompt",
|
||||
platform_audio::SUNO_PROMPT_MAX_CHARS,
|
||||
)
|
||||
.map_err(map_platform_audio_error)?,
|
||||
make_instrumental: true,
|
||||
price_mud_points: payload.price_mud_points,
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn generate_editor_sound_effect(
|
||||
State(state): State<AppState>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
payload: Result<Json<assets::EditorSoundEffectGenerateRequest>, JsonRejection>,
|
||||
) -> Result<Json<Value>, Response> {
|
||||
let Json(payload) = parse_json_payload(&request_context, payload)?;
|
||||
let normalized = normalize_editor_sound_effect_request(payload)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let settings = require_vector_engine_audio_settings(&state)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let http_client = platform_audio::build_vector_engine_audio_http_client(&settings)
|
||||
.map_err(map_platform_audio_error)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let task = platform_audio::submit_editor_sound_effect_task(
|
||||
&http_client,
|
||||
&settings,
|
||||
platform_audio::EditorSoundEffectTaskRequest {
|
||||
sound: normalized.sound.clone(),
|
||||
sound_type: normalized.sound_type.clone(),
|
||||
tempo: normalized.tempo.clone(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(map_platform_audio_error)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let generated = wait_for_generated_audio_asset(
|
||||
&state,
|
||||
authenticated.claims().user_id(),
|
||||
task.task_id.clone(),
|
||||
AudioAssetSlot::SoundEffect,
|
||||
build_editor_audio_target(&task.task_id, "sound_effect", "editor_sound_effect"),
|
||||
)
|
||||
.await
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let audio_src = generated
|
||||
.audio_src
|
||||
.ok_or_else(|| vector_engine_bad_gateway("音效生成完成但缺少播放地址"))
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
|
||||
Ok(json_success_body(
|
||||
Some(&request_context),
|
||||
assets::EditorAudioGenerateResponse {
|
||||
ok: true,
|
||||
audio_src,
|
||||
width: EDITOR_AUDIO_WIDTH,
|
||||
height: EDITOR_AUDIO_HEIGHT,
|
||||
source_type: "generated".to_string(),
|
||||
prompt: normalized.sound.clone(),
|
||||
actual_prompt: Some(normalized.sound),
|
||||
model: platform_audio::VIDU_AUDIO_MODEL.to_string(),
|
||||
provider: generated.provider,
|
||||
task_id: generated.task_id,
|
||||
price_mud_points: normalized.price_mud_points,
|
||||
audio_kind: "sound-effect".to_string(),
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn generate_editor_background_music(
|
||||
State(state): State<AppState>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
payload: Result<Json<assets::EditorBackgroundMusicGenerateRequest>, JsonRejection>,
|
||||
) -> Result<Json<Value>, Response> {
|
||||
let Json(payload) = parse_json_payload(&request_context, payload)?;
|
||||
let normalized = normalize_editor_background_music_request(payload)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let settings = require_vector_engine_audio_settings(&state)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let http_client = platform_audio::build_vector_engine_audio_http_client(&settings)
|
||||
.map_err(map_platform_audio_error)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let task = platform_audio::submit_editor_background_music_task(
|
||||
&http_client,
|
||||
&settings,
|
||||
platform_audio::EditorBackgroundMusicTaskRequest {
|
||||
gpt_description_prompt: normalized.gpt_description_prompt.clone(),
|
||||
make_instrumental: normalized.make_instrumental,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(map_platform_audio_error)
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let generated = wait_for_generated_audio_asset(
|
||||
&state,
|
||||
authenticated.claims().user_id(),
|
||||
task.task_id.clone(),
|
||||
AudioAssetSlot::BackgroundMusic,
|
||||
build_editor_audio_target(&task.task_id, "background_music", "editor_background_music"),
|
||||
)
|
||||
.await
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
let audio_src = generated
|
||||
.audio_src
|
||||
.ok_or_else(|| vector_engine_bad_gateway("背景音乐生成完成但缺少播放地址"))
|
||||
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
|
||||
|
||||
Ok(json_success_body(
|
||||
Some(&request_context),
|
||||
assets::EditorAudioGenerateResponse {
|
||||
ok: true,
|
||||
audio_src,
|
||||
width: EDITOR_AUDIO_WIDTH,
|
||||
height: EDITOR_AUDIO_HEIGHT,
|
||||
source_type: "generated".to_string(),
|
||||
prompt: normalized.gpt_description_prompt.clone(),
|
||||
actual_prompt: Some(normalized.gpt_description_prompt),
|
||||
model: platform_audio::SUNO_DEFAULT_MODEL.to_string(),
|
||||
provider: generated.provider,
|
||||
task_id: generated.task_id,
|
||||
price_mud_points: normalized.price_mud_points,
|
||||
audio_kind: "background-music".to_string(),
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
fn build_editor_audio_target(
|
||||
task_id: &str,
|
||||
slot: &'static str,
|
||||
asset_kind: &'static str,
|
||||
) -> AudioAssetBindingTarget {
|
||||
AudioAssetBindingTarget {
|
||||
entity_kind: "editor_audio".to_string(),
|
||||
entity_id: task_id.to_string(),
|
||||
slot: slot.to_string(),
|
||||
asset_kind: asset_kind.to_string(),
|
||||
profile_id: None,
|
||||
storage_prefix: LegacyAssetPrefix::CharacterDrafts,
|
||||
storage_scope: "editor_audio".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn editor_audio_bad_request(message: impl Into<String>) -> AppError {
|
||||
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
|
||||
"provider": platform_audio::VECTOR_ENGINE_PROVIDER,
|
||||
"message": message.into(),
|
||||
}))
|
||||
}
|
||||
|
||||
pub(crate) async fn generate_sound_effect_asset_for_creation(
|
||||
state: &AppState,
|
||||
owner_user_id: &str,
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
use axum::http::StatusCode;
|
||||
use platform_oss::LegacyAssetPrefix;
|
||||
use shared_contracts::creation_audio;
|
||||
use shared_contracts::{assets, creation_audio};
|
||||
|
||||
use super::{
|
||||
generation::{
|
||||
normalize_editor_background_music_request, normalize_editor_sound_effect_request,
|
||||
},
|
||||
publish::resolve_creation_audio_points_cost,
|
||||
targets::{build_creation_audio_target, creation_audio_generation_disabled_error_for_target},
|
||||
types::{AudioAssetBindingTarget, AudioAssetSlot},
|
||||
@@ -77,3 +80,48 @@ fn disabled_creation_audio_targets_return_gone_including_wooden_fish_sound_effec
|
||||
.expect_err("wooden fish hit sound target should be disabled");
|
||||
assert_eq!(error.status_code(), StatusCode::GONE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_sound_effect_request_normalizes_sound_type_tempo_and_price() {
|
||||
let normalized =
|
||||
normalize_editor_sound_effect_request(assets::EditorSoundEffectGenerateRequest {
|
||||
sound: " 金币掉落叮当声 ".to_string(),
|
||||
sound_type: " ui ".to_string(),
|
||||
tempo: " fast ".to_string(),
|
||||
price_mud_points: 10,
|
||||
})
|
||||
.expect("editor sound effect request should normalize");
|
||||
|
||||
assert_eq!(normalized.sound, "金币掉落叮当声");
|
||||
assert_eq!(normalized.sound_type, "ui");
|
||||
assert_eq!(normalized.tempo, "fast");
|
||||
assert_eq!(normalized.price_mud_points, 10);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_background_music_request_forces_instrumental_and_price() {
|
||||
let normalized =
|
||||
normalize_editor_background_music_request(assets::EditorBackgroundMusicGenerateRequest {
|
||||
gpt_description_prompt: " 森林冒险背景音乐 ".to_string(),
|
||||
make_instrumental: false,
|
||||
price_mud_points: 5,
|
||||
})
|
||||
.expect("editor background music request should normalize");
|
||||
|
||||
assert_eq!(normalized.gpt_description_prompt, "森林冒险背景音乐");
|
||||
assert!(normalized.make_instrumental);
|
||||
assert_eq!(normalized.price_mud_points, 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_audio_rejects_price_mismatch() {
|
||||
let error = normalize_editor_sound_effect_request(assets::EditorSoundEffectGenerateRequest {
|
||||
sound: "点击按钮".to_string(),
|
||||
sound_type: "ui".to_string(),
|
||||
tempo: "medium".to_string(),
|
||||
price_mud_points: 5,
|
||||
})
|
||||
.expect_err("wrong sound effect price should fail");
|
||||
|
||||
assert!(error.to_string().contains("泥点"));
|
||||
}
|
||||
|
||||
@@ -8,7 +8,9 @@ use crate::response::{
|
||||
};
|
||||
use crate::{
|
||||
AudioError, AudioTaskKind, AudioTaskResponse, BackgroundMusicTaskRequest,
|
||||
SoundEffectTaskRequest, VectorEngineAudioSettings, build_background_music_task_body,
|
||||
EditorBackgroundMusicTaskRequest, EditorSoundEffectTaskRequest, SoundEffectTaskRequest,
|
||||
VectorEngineAudioSettings, build_background_music_task_body,
|
||||
build_editor_background_music_task_body, build_editor_sound_effect_task_body,
|
||||
build_sound_effect_task_body,
|
||||
};
|
||||
|
||||
@@ -83,6 +85,62 @@ pub async fn submit_sound_effect_task(
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn submit_editor_background_music_task(
|
||||
http_client: &reqwest::Client,
|
||||
settings: &VectorEngineAudioSettings,
|
||||
request: EditorBackgroundMusicTaskRequest,
|
||||
) -> Result<AudioTaskResponse, AudioError> {
|
||||
let body = build_editor_background_music_task_body(request)?;
|
||||
let response = post_vector_engine_json(
|
||||
http_client,
|
||||
settings,
|
||||
AudioTaskKind::BackgroundMusic.submit_path(),
|
||||
body,
|
||||
"提交编辑器背景音乐任务失败",
|
||||
)
|
||||
.await?;
|
||||
let task_id = extract_string_by_path(&response, &["data"])
|
||||
.or_else(|| find_first_string_by_key(&response, "task_id"))
|
||||
.or_else(|| find_first_string_by_key(&response, "taskId"))
|
||||
.ok_or_else(|| {
|
||||
AudioError::missing_audio("提交编辑器背景音乐任务失败：上游未返回任务 ID")
|
||||
})?;
|
||||
|
||||
Ok(AudioTaskResponse {
|
||||
kind: AudioTaskKind::BackgroundMusic,
|
||||
task_id,
|
||||
provider: AudioTaskKind::BackgroundMusic.provider().to_string(),
|
||||
status: "submitted".to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
pub async fn submit_editor_sound_effect_task(
|
||||
http_client: &reqwest::Client,
|
||||
settings: &VectorEngineAudioSettings,
|
||||
request: EditorSoundEffectTaskRequest,
|
||||
) -> Result<AudioTaskResponse, AudioError> {
|
||||
let body = build_editor_sound_effect_task_body(request)?;
|
||||
let response = post_vector_engine_json(
|
||||
http_client,
|
||||
settings,
|
||||
AudioTaskKind::SoundEffect.submit_path(),
|
||||
body,
|
||||
"提交编辑器音效任务失败",
|
||||
)
|
||||
.await?;
|
||||
let task_id = find_first_string_by_key(&response, "task_id")
|
||||
.or_else(|| find_first_string_by_key(&response, "taskId"))
|
||||
.ok_or_else(|| AudioError::missing_audio("提交编辑器音效任务失败：上游未返回任务 ID"))?;
|
||||
let status = find_first_string_by_key(&response, "state").unwrap_or_else(|| "created".into());
|
||||
|
||||
Ok(AudioTaskResponse {
|
||||
kind: AudioTaskKind::SoundEffect,
|
||||
task_id,
|
||||
provider: AudioTaskKind::SoundEffect.provider().to_string(),
|
||||
status,
|
||||
})
|
||||
}
|
||||
|
||||
async fn fetch_audio_task_payload(
|
||||
http_client: &reqwest::Client,
|
||||
settings: &VectorEngineAudioSettings,
|
||||
|
||||
@@ -8,7 +8,8 @@ mod types;
|
||||
|
||||
pub use client::{
|
||||
build_vector_engine_audio_http_client, resolve_audio_task_download_urls,
|
||||
submit_background_music_task, submit_sound_effect_task,
|
||||
submit_background_music_task, submit_editor_background_music_task,
|
||||
submit_editor_sound_effect_task, submit_sound_effect_task,
|
||||
};
|
||||
pub use download::{audio_mime_to_extension, download_generated_audio, normalize_audio_mime_type};
|
||||
pub use error::{AudioError, AudioStatusHint};
|
||||
@@ -16,7 +17,8 @@ pub use persist::{
|
||||
GeneratedAudioPersistInput, GeneratedAudioPersistTarget, prepare_generated_audio_put_request,
|
||||
};
|
||||
pub use request::{
|
||||
build_background_music_task_body, build_sound_effect_task_body, normalize_limited_text,
|
||||
build_background_music_task_body, build_editor_background_music_task_body,
|
||||
build_editor_sound_effect_task_body, build_sound_effect_task_body, normalize_limited_text,
|
||||
normalize_limited_text_allow_empty, normalize_optional_text,
|
||||
};
|
||||
pub use response::{
|
||||
@@ -24,9 +26,9 @@ pub use response::{
|
||||
};
|
||||
pub use types::{
|
||||
AudioTaskKind, AudioTaskResponse, BackgroundMusicTaskRequest,
|
||||
DEFAULT_SOUND_EFFECT_DURATION_SECONDS, DownloadedAudio, MAX_GENERATED_AUDIO_BYTES,
|
||||
SUNO_DEFAULT_MODEL, SUNO_PROMPT_MAX_CHARS, SUNO_TAGS_MAX_CHARS, SUNO_TITLE_MAX_CHARS,
|
||||
SoundEffectTaskRequest, VECTOR_ENGINE_PROVIDER, VECTOR_ENGINE_SUNO_PROVIDER,
|
||||
VECTOR_ENGINE_VIDU_PROVIDER, VIDU_AUDIO_MODEL, VIDU_PROMPT_MAX_CHARS,
|
||||
VectorEngineAudioSettings,
|
||||
DEFAULT_SOUND_EFFECT_DURATION_SECONDS, DownloadedAudio, EditorBackgroundMusicTaskRequest,
|
||||
EditorSoundEffectTaskRequest, MAX_GENERATED_AUDIO_BYTES, SUNO_DEFAULT_MODEL,
|
||||
SUNO_PROMPT_MAX_CHARS, SUNO_TAGS_MAX_CHARS, SUNO_TITLE_MAX_CHARS, SoundEffectTaskRequest,
|
||||
VECTOR_ENGINE_PROVIDER, VECTOR_ENGINE_SUNO_PROVIDER, VECTOR_ENGINE_VIDU_PROVIDER,
|
||||
VIDU_AUDIO_MODEL, VIDU_PROMPT_MAX_CHARS, VectorEngineAudioSettings,
|
||||
};
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
use serde_json::{Map, Value, json};
|
||||
|
||||
use crate::{
|
||||
AudioError, BackgroundMusicTaskRequest, SUNO_DEFAULT_MODEL, SUNO_PROMPT_MAX_CHARS,
|
||||
SUNO_TAGS_MAX_CHARS, SUNO_TITLE_MAX_CHARS, SoundEffectTaskRequest, VIDU_AUDIO_MODEL,
|
||||
VIDU_PROMPT_MAX_CHARS,
|
||||
AudioError, BackgroundMusicTaskRequest, EditorBackgroundMusicTaskRequest,
|
||||
EditorSoundEffectTaskRequest, SUNO_DEFAULT_MODEL, SUNO_PROMPT_MAX_CHARS, SUNO_TAGS_MAX_CHARS,
|
||||
SUNO_TITLE_MAX_CHARS, SoundEffectTaskRequest, VIDU_AUDIO_MODEL, VIDU_PROMPT_MAX_CHARS,
|
||||
};
|
||||
|
||||
pub fn build_background_music_task_body(
|
||||
@@ -53,6 +53,33 @@ pub fn build_sound_effect_task_body(request: SoundEffectTaskRequest) -> Result<V
|
||||
Ok(Value::Object(body))
|
||||
}
|
||||
|
||||
pub fn build_editor_background_music_task_body(
|
||||
request: EditorBackgroundMusicTaskRequest,
|
||||
) -> Result<Value, AudioError> {
|
||||
let prompt = normalize_limited_text(
|
||||
&request.gpt_description_prompt,
|
||||
"gpt_description_prompt",
|
||||
SUNO_PROMPT_MAX_CHARS,
|
||||
)?;
|
||||
Ok(json!({
|
||||
"gpt_description_prompt": prompt,
|
||||
"make_instrumental": request.make_instrumental,
|
||||
}))
|
||||
}
|
||||
|
||||
pub fn build_editor_sound_effect_task_body(
|
||||
request: EditorSoundEffectTaskRequest,
|
||||
) -> Result<Value, AudioError> {
|
||||
let sound = normalize_limited_text(&request.sound, "sound", VIDU_PROMPT_MAX_CHARS)?;
|
||||
let sound_type = normalize_limited_text(&request.sound_type, "type", 80)?;
|
||||
let tempo = normalize_limited_text(&request.tempo, "tempo", 80)?;
|
||||
Ok(json!({
|
||||
"sound": sound,
|
||||
"type": sound_type,
|
||||
"tempo": tempo,
|
||||
}))
|
||||
}
|
||||
|
||||
pub fn normalize_limited_text(
|
||||
value: &str,
|
||||
field: &'static str,
|
||||
|
||||
@@ -52,6 +52,19 @@ pub struct SoundEffectTaskRequest {
|
||||
pub seed: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct EditorBackgroundMusicTaskRequest {
|
||||
pub gpt_description_prompt: String,
|
||||
pub make_instrumental: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct EditorSoundEffectTaskRequest {
|
||||
pub sound: String,
|
||||
pub sound_type: String,
|
||||
pub tempo: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct AudioTaskResponse {
|
||||
pub kind: AudioTaskKind,
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
use platform_audio::{
|
||||
AudioTaskKind, BackgroundMusicTaskRequest, SUNO_DEFAULT_MODEL, VIDU_PROMPT_MAX_CHARS,
|
||||
audio_mime_to_extension, build_background_music_task_body, build_sound_effect_task_body,
|
||||
extract_audio_urls, is_failed_task_status, is_pending_task_status, normalize_audio_mime_type,
|
||||
normalize_task_status,
|
||||
AudioTaskKind, BackgroundMusicTaskRequest, EditorBackgroundMusicTaskRequest,
|
||||
EditorSoundEffectTaskRequest, SUNO_DEFAULT_MODEL, VIDU_PROMPT_MAX_CHARS,
|
||||
audio_mime_to_extension, build_background_music_task_body,
|
||||
build_editor_background_music_task_body, build_editor_sound_effect_task_body,
|
||||
build_sound_effect_task_body, extract_audio_urls, is_failed_task_status,
|
||||
is_pending_task_status, normalize_audio_mime_type, normalize_task_status,
|
||||
};
|
||||
use serde_json::json;
|
||||
|
||||
@@ -82,3 +84,32 @@ fn sound_effect_request_rejects_overlong_prompt() {
|
||||
|
||||
assert!(error.message().contains("prompt 超过"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_background_music_request_body_uses_gpt_description_prompt_and_instrumental() {
|
||||
let body =
|
||||
build_editor_background_music_task_body(EditorBackgroundMusicTaskRequest {
|
||||
gpt_description_prompt: " 轻快森林冒险背景音乐 ".to_string(),
|
||||
make_instrumental: true,
|
||||
})
|
||||
.expect("editor background music body should be valid");
|
||||
|
||||
assert_eq!(body["gpt_description_prompt"], "轻快森林冒险背景音乐");
|
||||
assert_eq!(body["make_instrumental"], true);
|
||||
assert!(body.get("prompt").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_sound_effect_request_body_uses_sound_type_and_tempo() {
|
||||
let body = build_editor_sound_effect_task_body(EditorSoundEffectTaskRequest {
|
||||
sound: " 金币掉落叮当声 ".to_string(),
|
||||
sound_type: " ui ".to_string(),
|
||||
tempo: " fast ".to_string(),
|
||||
})
|
||||
.expect("editor sound effect body should be valid");
|
||||
|
||||
assert_eq!(body["sound"], "金币掉落叮当声");
|
||||
assert_eq!(body["type"], "ui");
|
||||
assert_eq!(body["tempo"], "fast");
|
||||
assert!(body.get("prompt").is_none());
|
||||
}
|
||||
|
||||
@@ -357,6 +357,12 @@ pub struct EditorVideoGenerateRequest {
|
||||
pub mode: String,
|
||||
pub sound: String,
|
||||
pub price_mud_points: u32,
|
||||
#[serde(default)]
|
||||
pub reference_image_srcs: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub reference_video_srcs: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub reference_audio_srcs: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
@@ -378,6 +384,42 @@ pub struct EditorVideoGenerateResponse {
|
||||
pub price_mud_points: u32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct EditorSoundEffectGenerateRequest {
|
||||
pub sound: String,
|
||||
#[serde(rename = "type")]
|
||||
pub sound_type: String,
|
||||
pub tempo: String,
|
||||
pub price_mud_points: u32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct EditorBackgroundMusicGenerateRequest {
|
||||
pub gpt_description_prompt: String,
|
||||
pub make_instrumental: bool,
|
||||
pub price_mud_points: u32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct EditorAudioGenerateResponse {
|
||||
pub ok: bool,
|
||||
pub audio_src: String,
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub source_type: String,
|
||||
pub prompt: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub actual_prompt: Option<String>,
|
||||
pub model: String,
|
||||
pub provider: String,
|
||||
pub task_id: String,
|
||||
pub price_mud_points: u32,
|
||||
pub audio_kind: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct CharacterAnimationDraftPayload {
|
||||
@@ -907,7 +949,10 @@ mod tests {
|
||||
.expect("request should serialize");
|
||||
|
||||
assert_eq!(payload["sourceLayerId"], json!("layer-1"));
|
||||
assert_eq!(payload["sourceImageSrc"], json!("/generated-characters/hero/master.png"));
|
||||
assert_eq!(
|
||||
payload["sourceImageSrc"],
|
||||
json!("/generated-characters/hero/master.png")
|
||||
);
|
||||
assert_eq!(payload["priceMudPoints"], json!(120));
|
||||
assert_eq!(payload["model"], json!("seedance2.0-fast"));
|
||||
}
|
||||
@@ -935,8 +980,14 @@ mod tests {
|
||||
})
|
||||
.expect("response should serialize");
|
||||
|
||||
assert_eq!(payload["previewVideoPath"], json!("/generated-character-drafts/editor/layer/preview.mp4"));
|
||||
assert_eq!(payload["frames"][0]["imageSrc"], json!("/generated-animations/editor/layer/frame01.png"));
|
||||
assert_eq!(
|
||||
payload["previewVideoPath"],
|
||||
json!("/generated-character-drafts/editor/layer/preview.mp4")
|
||||
);
|
||||
assert_eq!(
|
||||
payload["frames"][0]["imageSrc"],
|
||||
json!("/generated-animations/editor/layer/frame01.png")
|
||||
);
|
||||
assert_eq!(payload["fps"], json!(8));
|
||||
}
|
||||
|
||||
@@ -951,6 +1002,9 @@ mod tests {
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 50,
|
||||
reference_image_srcs: Vec::new(),
|
||||
reference_video_srcs: Vec::new(),
|
||||
reference_audio_srcs: Vec::new(),
|
||||
})
|
||||
.expect("request should serialize");
|
||||
|
||||
@@ -960,6 +1014,37 @@ mod tests {
|
||||
assert_eq!(payload["model"], json!("kling3.0-omni"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_video_request_supports_seedance_multimodal_references() {
|
||||
let payload = serde_json::to_value(EditorVideoGenerateRequest {
|
||||
prompt: "参考素材生成视频".to_string(),
|
||||
model: "seedance2.0-fast".to_string(),
|
||||
aspect_ratio: "16:9".to_string(),
|
||||
duration_seconds: 5,
|
||||
resolution: "720p".to_string(),
|
||||
mode: "std".to_string(),
|
||||
sound: "off".to_string(),
|
||||
price_mud_points: 100,
|
||||
reference_image_srcs: vec!["data:image/png;base64,image".to_string()],
|
||||
reference_video_srcs: vec!["data:video/mp4;base64,video".to_string()],
|
||||
reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()],
|
||||
})
|
||||
.expect("request should serialize");
|
||||
|
||||
assert_eq!(
|
||||
payload["referenceImageSrcs"],
|
||||
json!(["data:image/png;base64,image"])
|
||||
);
|
||||
assert_eq!(
|
||||
payload["referenceVideoSrcs"],
|
||||
json!(["data:video/mp4;base64,video"])
|
||||
);
|
||||
assert_eq!(
|
||||
payload["referenceAudioSrcs"],
|
||||
json!(["data:audio/mpeg;base64,audio"])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_video_response_uses_canvas_video_shape() {
|
||||
let payload = serde_json::to_value(EditorVideoGenerateResponse {
|
||||
@@ -987,6 +1072,54 @@ mod tests {
|
||||
assert_eq!(payload["durationSeconds"], json!(5));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn editor_audio_requests_and_response_use_canvas_audio_shape() {
|
||||
let sound_payload = serde_json::to_value(EditorSoundEffectGenerateRequest {
|
||||
sound: "金币掉落叮当声".to_string(),
|
||||
sound_type: "ui".to_string(),
|
||||
tempo: "fast".to_string(),
|
||||
price_mud_points: 10,
|
||||
})
|
||||
.expect("sound request should serialize");
|
||||
assert_eq!(sound_payload["sound"], json!("金币掉落叮当声"));
|
||||
assert_eq!(sound_payload["type"], json!("ui"));
|
||||
assert_eq!(sound_payload["tempo"], json!("fast"));
|
||||
assert_eq!(sound_payload["priceMudPoints"], json!(10));
|
||||
|
||||
let music_payload = serde_json::to_value(EditorBackgroundMusicGenerateRequest {
|
||||
gpt_description_prompt: "森林冒险背景音乐".to_string(),
|
||||
make_instrumental: true,
|
||||
price_mud_points: 5,
|
||||
})
|
||||
.expect("background music request should serialize");
|
||||
assert_eq!(
|
||||
music_payload["gptDescriptionPrompt"],
|
||||
json!("森林冒险背景音乐")
|
||||
);
|
||||
assert_eq!(music_payload["makeInstrumental"], json!(true));
|
||||
|
||||
let response_payload = serde_json::to_value(EditorAudioGenerateResponse {
|
||||
ok: true,
|
||||
audio_src: "/generated-character-drafts/editor-audios/sfx.mp3".to_string(),
|
||||
width: 420,
|
||||
height: 120,
|
||||
source_type: "generated".to_string(),
|
||||
prompt: "金币掉落叮当声".to_string(),
|
||||
actual_prompt: Some("金币掉落叮当声".to_string()),
|
||||
model: "audio1.0".to_string(),
|
||||
provider: "VectorEngine".to_string(),
|
||||
task_id: "sound-task-1".to_string(),
|
||||
price_mud_points: 10,
|
||||
audio_kind: "sound-effect".to_string(),
|
||||
})
|
||||
.expect("audio response should serialize");
|
||||
assert_eq!(
|
||||
response_payload["audioSrc"],
|
||||
json!("/generated-character-drafts/editor-audios/sfx.mp3")
|
||||
);
|
||||
assert_eq!(response_payload["audioKind"], json!("sound-effect"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn character_workflow_cache_response_keeps_legacy_shape() {
|
||||
let payload = serde_json::to_value(CharacterWorkflowCacheSaveResponse {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/* @vitest-environment jsdom */
|
||||
|
||||
import { render, screen } from '@testing-library/react';
|
||||
import { fireEvent, render, screen } from '@testing-library/react';
|
||||
import userEvent from '@testing-library/user-event';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
@@ -28,4 +28,26 @@ describe('PlatformFloatingMenu', () => {
|
||||
|
||||
expect(onRename).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it('keeps pointer events from leaking to canvas-style parents', () => {
|
||||
const onParentPointerDown = vi.fn();
|
||||
const onMenuPointerDown = vi.fn();
|
||||
|
||||
render(
|
||||
<div onPointerDown={onParentPointerDown}>
|
||||
<PlatformFloatingMenu
|
||||
label="项目菜单"
|
||||
placement="bottom-end"
|
||||
onPointerDown={onMenuPointerDown}
|
||||
>
|
||||
<PlatformFloatingMenuItem>重命名</PlatformFloatingMenuItem>
|
||||
</PlatformFloatingMenu>
|
||||
</div>,
|
||||
);
|
||||
|
||||
fireEvent.pointerDown(screen.getByRole('menu', { name: '项目菜单' }));
|
||||
|
||||
expect(onMenuPointerDown).toHaveBeenCalledOnce();
|
||||
expect(onParentPointerDown).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -26,6 +26,7 @@ export function PlatformFloatingMenu({
|
||||
label,
|
||||
placement = 'top-end',
|
||||
style,
|
||||
onPointerDown,
|
||||
...divProps
|
||||
}: PlatformFloatingMenuProps) {
|
||||
return (
|
||||
@@ -41,6 +42,10 @@ export function PlatformFloatingMenu({
|
||||
role="menu"
|
||||
aria-label={label}
|
||||
style={style}
|
||||
onPointerDown={(event) => {
|
||||
event.stopPropagation();
|
||||
onPointerDown?.(event);
|
||||
}}
|
||||
>
|
||||
{children}
|
||||
</div>
|
||||
|
||||
@@ -134,6 +134,26 @@ describe('ImageCanvasBasicGenerationComposerView', () => {
|
||||
).toBeTruthy();
|
||||
});
|
||||
|
||||
it('removes an existing reference from its hover delete button', () => {
|
||||
render(
|
||||
<BasicGenerationHarness
|
||||
initialDialog={createDialog({
|
||||
generationReferences: [
|
||||
{
|
||||
id: 'ref-a',
|
||||
label: '参考图A',
|
||||
src: 'data:image/png;base64,cmVmQQ==',
|
||||
},
|
||||
],
|
||||
})}
|
||||
/>,
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: '删除参考图A' }));
|
||||
|
||||
expect(screen.queryByText('参考图A')).toBeNull();
|
||||
});
|
||||
|
||||
it('shows only current image options and opens upward option panels', () => {
|
||||
render(<BasicGenerationHarness />);
|
||||
|
||||
@@ -166,6 +186,26 @@ describe('ImageCanvasBasicGenerationComposerView', () => {
|
||||
expect(submitButton.textContent).toBe('生成12泥点');
|
||||
});
|
||||
|
||||
it('clicks the parent generation panel to collapse an opened option panel', () => {
|
||||
render(<BasicGenerationHarness />);
|
||||
|
||||
const panel = screen.getByRole('dialog', { name: '生成图片' });
|
||||
fireEvent.click(
|
||||
within(panel).getByRole('button', {
|
||||
name: '生成图片尺寸 16:9 · 1K',
|
||||
}),
|
||||
);
|
||||
expect(
|
||||
screen.getByRole('menu', { name: '生成图片尺寸选项' }),
|
||||
).toBeTruthy();
|
||||
|
||||
fireEvent.click(screen.getByRole('textbox', { name: '生成提示词' }));
|
||||
|
||||
expect(
|
||||
screen.queryByRole('menu', { name: '生成图片尺寸选项' }),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it('closes through its interface', () => {
|
||||
const closeComposer = vi.fn();
|
||||
render(<BasicGenerationHarness onClose={closeComposer} />);
|
||||
|
||||
@@ -14,14 +14,16 @@ import {
|
||||
import { PlatformStatusMessage } from '../common/PlatformStatusMessage';
|
||||
import { PlatformTextField } from '../common/PlatformTextField';
|
||||
import { EditorIconButton } from './ImageCanvasEditorPrimitives';
|
||||
import {
|
||||
EDITOR_GENERATION_MUD_POINT_CONFIG,
|
||||
} from './ImageCanvasGenerationModel';
|
||||
import { ImageCanvasGenerationImageOptionsView } from './ImageCanvasGenerationImageOptionsView';
|
||||
import type {
|
||||
CharacterReferenceImage,
|
||||
GenerateDialogState,
|
||||
} from './ImageCanvasEditorTypes';
|
||||
import { ImageCanvasGenerationImageOptionsView } from './ImageCanvasGenerationImageOptionsView';
|
||||
import { ImageCanvasReferenceSlot } from './ImageCanvasReferenceSlot';
|
||||
import {
|
||||
EDITOR_GENERATION_MUD_POINT_CONFIG,
|
||||
} from './ImageCanvasGenerationModel';
|
||||
import { useImageCanvasFloatingOptionDismiss } from './useImageCanvasFloatingOptionDismiss';
|
||||
|
||||
type ImageCanvasBasicGenerationComposerViewProps = {
|
||||
dialog: GenerateDialogState;
|
||||
@@ -53,22 +55,24 @@ function resetFailedDialogStatus(dialog: GenerateDialogState) {
|
||||
function ReferenceChip({
|
||||
reference,
|
||||
index,
|
||||
onRemove,
|
||||
}: {
|
||||
reference: CharacterReferenceImage;
|
||||
index: number;
|
||||
onRemove: () => void;
|
||||
}) {
|
||||
const label = reference.label || `参考图${index + 1}`;
|
||||
return (
|
||||
<span
|
||||
className="image-canvas-editor__reference-chip"
|
||||
<ImageCanvasReferenceSlot
|
||||
tone="default"
|
||||
icon={<ImageIcon className="h-4 w-4" aria-hidden="true" />}
|
||||
imageSrc={reference.src}
|
||||
label={label}
|
||||
ariaLabel={label}
|
||||
title={reference.label}
|
||||
>
|
||||
<span className="image-canvas-editor__reference-chip-icon">
|
||||
<img src={reference.src} alt="" aria-hidden="true" />
|
||||
</span>
|
||||
<span className="image-canvas-editor__reference-chip-label">
|
||||
{reference.label || `参考图${index + 1}`}
|
||||
</span>
|
||||
</span>
|
||||
onRemove={onRemove}
|
||||
removeLabel={`删除${label}`}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -89,6 +93,12 @@ export function ImageCanvasBasicGenerationComposerView({
|
||||
onClose,
|
||||
}: ImageCanvasBasicGenerationComposerViewProps) {
|
||||
const references = dialog.generationReferences ?? [];
|
||||
useImageCanvasFloatingOptionDismiss({
|
||||
isOpen: isGenerationReferenceMenuOpen,
|
||||
boundaryRefs: [generationReferenceButtonRef],
|
||||
onDismiss: () => setIsGenerationReferenceMenuOpen?.(false),
|
||||
});
|
||||
|
||||
return (
|
||||
<>
|
||||
<form
|
||||
@@ -110,23 +120,30 @@ export function ImageCanvasBasicGenerationComposerView({
|
||||
key={reference.id}
|
||||
reference={reference}
|
||||
index={index}
|
||||
onRemove={() =>
|
||||
setGenerateDialog((currentDialog) =>
|
||||
currentDialog
|
||||
? {
|
||||
...resetFailedDialogStatus(currentDialog),
|
||||
generationReferences: (
|
||||
currentDialog.generationReferences ?? []
|
||||
).filter((item) => item.id !== reference.id),
|
||||
}
|
||||
: currentDialog,
|
||||
)
|
||||
}
|
||||
/>
|
||||
))}
|
||||
<button
|
||||
ref={generationReferenceButtonRef}
|
||||
type="button"
|
||||
className="image-canvas-editor__reference-chip image-canvas-editor__reference-chip--upload"
|
||||
<ImageCanvasReferenceSlot
|
||||
buttonRef={generationReferenceButtonRef}
|
||||
tone="default"
|
||||
icon={<ImageIcon className="h-4 w-4" aria-hidden="true" />}
|
||||
label="参考图"
|
||||
ariaLabel="添加参考图"
|
||||
disabled={dialog.status === 'generating'}
|
||||
aria-label="添加参考图"
|
||||
isAdd
|
||||
onClick={() => onToggleReferenceMenu?.()}
|
||||
>
|
||||
<span className="image-canvas-editor__reference-chip-icon">
|
||||
<ImageIcon className="h-4 w-4" aria-hidden="true" />
|
||||
</span>
|
||||
<span className="image-canvas-editor__reference-chip-label">
|
||||
参考图
|
||||
</span>
|
||||
</button>
|
||||
/>
|
||||
</div>
|
||||
<PlatformTextField
|
||||
variant="textarea"
|
||||
|
||||
@@ -36,12 +36,14 @@ describe('ImageCanvasBottomToolbarView', () => {
|
||||
fireEvent.click(
|
||||
within(toolbar).getByRole('button', { name: '生成UI设计图' }),
|
||||
);
|
||||
fireEvent.click(within(toolbar).getByRole('button', { name: '生成音乐' }));
|
||||
fireEvent.click(within(toolbar).getByRole('button', { name: '文字工具' }));
|
||||
|
||||
expect(switchTool).toHaveBeenNthCalledWith(1, 'hand');
|
||||
expect(switchTool).toHaveBeenNthCalledWith(2, 'video');
|
||||
expect(switchTool).toHaveBeenNthCalledWith(3, 'spec');
|
||||
expect(switchTool).toHaveBeenNthCalledWith(4, 'ui-design');
|
||||
expect(switchTool).toHaveBeenNthCalledWith(5, 'text');
|
||||
expect(switchTool).toHaveBeenNthCalledWith(5, 'music');
|
||||
expect(switchTool).toHaveBeenNthCalledWith(6, 'text');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -7,6 +7,7 @@ import {
|
||||
ImagePlus,
|
||||
LayoutTemplate,
|
||||
MousePointer2,
|
||||
Music,
|
||||
Shapes,
|
||||
Sparkles,
|
||||
Type,
|
||||
@@ -33,6 +34,7 @@ const canvasTools: Array<{
|
||||
{ id: 'upload', label: '上传工具', icon: ImagePlus },
|
||||
{ id: 'generate', label: '生成工具', icon: WandSparkles },
|
||||
{ id: 'video', label: '生成视频', icon: Clapperboard },
|
||||
{ id: 'music', label: '生成音乐', icon: Music },
|
||||
{ id: 'spec', label: '生成规范', icon: ClipboardList },
|
||||
{ id: 'character', label: '生成角色形象', icon: Sparkles },
|
||||
{ id: 'icon', label: '生成图标素材', icon: ImageIcon },
|
||||
|
||||
@@ -132,6 +132,21 @@ describe('ImageCanvasCharacterAnimationPanelView', () => {
|
||||
expect(screen.getByLabelText('当前错误').textContent).toBe('-');
|
||||
});
|
||||
|
||||
it('clicks the parent animation panel to collapse the parameter menu', () => {
|
||||
render(
|
||||
<CharacterAnimationPanelHarness initialPanel={createPanel()} />,
|
||||
);
|
||||
|
||||
fireEvent.click(
|
||||
screen.getByRole('button', { name: '动画参数 同图尺寸 · 4秒 · 480p' }),
|
||||
);
|
||||
expect(screen.getByRole('menu', { name: '动画参数选项' })).toBeTruthy();
|
||||
|
||||
fireEvent.click(screen.getByRole('textbox', { name: '动画描述' }));
|
||||
|
||||
expect(screen.queryByRole('menu', { name: '动画参数选项' })).toBeNull();
|
||||
});
|
||||
|
||||
it('applies preset prompt and duration updates', () => {
|
||||
const updateDuration = vi.fn();
|
||||
render(
|
||||
|
||||
@@ -1,33 +1,35 @@
|
||||
import { ChevronDown, X } from 'lucide-react';
|
||||
import {
|
||||
type CSSProperties,
|
||||
type Dispatch,
|
||||
type ReactNode,
|
||||
type SetStateAction,
|
||||
useCallback,
|
||||
useRef,
|
||||
useState,
|
||||
} from 'react';
|
||||
import { createPortal } from 'react-dom';
|
||||
import { ChevronDown, X } from 'lucide-react';
|
||||
|
||||
import type {
|
||||
EditorCharacterAnimationRatio,
|
||||
EditorCharacterAnimationResolution,
|
||||
} from '../../services/image-editor/editorProjectClient';
|
||||
import { PlatformActionButton } from '../common/PlatformActionButton';
|
||||
import { PlatformFloatingMenu } from '../common/PlatformFloatingMenu';
|
||||
import { PlatformInlineOptionButton } from '../common/PlatformInlineOptionButton';
|
||||
import { PlatformStatusMessage } from '../common/PlatformStatusMessage';
|
||||
import { PlatformTextField } from '../common/PlatformTextField';
|
||||
import { EditorIconButton } from './ImageCanvasEditorPrimitives';
|
||||
import type {
|
||||
CanvasLayer,
|
||||
CharacterAnimationPanelState,
|
||||
} from './ImageCanvasEditorTypes';
|
||||
import {
|
||||
CHARACTER_ANIMATION_ACTION_PROMPTS,
|
||||
CHARACTER_ANIMATION_DURATION_OPTIONS,
|
||||
CHARACTER_ANIMATION_RATIO_OPTIONS,
|
||||
} from './ImageCanvasGenerationModel';
|
||||
import type {
|
||||
EditorCharacterAnimationRatio,
|
||||
EditorCharacterAnimationResolution,
|
||||
} from '../../services/image-editor/editorProjectClient';
|
||||
import type {
|
||||
CanvasLayer,
|
||||
CharacterAnimationPanelState,
|
||||
} from './ImageCanvasEditorTypes';
|
||||
import { useImageCanvasFloatingOptionDismiss } from './useImageCanvasFloatingOptionDismiss';
|
||||
|
||||
type ImageCanvasCharacterAnimationPanelViewProps = {
|
||||
panel: CharacterAnimationPanelState;
|
||||
@@ -131,6 +133,15 @@ export function ImageCanvasCharacterAnimationPanelView({
|
||||
const parameterButtonRef = useRef<HTMLButtonElement | null>(null);
|
||||
const isGenerating = panel.status === 'generating';
|
||||
const ratioLabel = getRatioLabel(panel.ratio);
|
||||
const dismissParameterMenu = useCallback(() => {
|
||||
setIsParameterMenuOpen(false);
|
||||
}, []);
|
||||
|
||||
useImageCanvasFloatingOptionDismiss({
|
||||
isOpen: isParameterMenuOpen,
|
||||
boundaryRefs: [parameterButtonRef],
|
||||
onDismiss: dismissParameterMenu,
|
||||
});
|
||||
|
||||
const updatePanel = (patch: Partial<CharacterAnimationPanelState>) => {
|
||||
setCharacterAnimationPanel((currentPanel) =>
|
||||
|
||||
@@ -166,4 +166,32 @@ describe('ImageCanvasCharacterGenerationComposerView', () => {
|
||||
|
||||
expect(screen.getByLabelText('选择常规参考图').textContent).toBe('true');
|
||||
});
|
||||
|
||||
it('removes bound character spec and regular references from their delete buttons', () => {
|
||||
render(<CharacterGenerationHarness />);
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: '删除角色形象规范' }));
|
||||
expect(screen.getByRole('button', { name: '角色形象规范' })).toBeTruthy();
|
||||
expect(screen.queryByRole('button', { name: '角色规范A' })).toBeNull();
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: '删除参考图A' }));
|
||||
expect(screen.queryByText('参考图A')).toBeNull();
|
||||
});
|
||||
|
||||
it('clicks the parent character panel to collapse reference menus', () => {
|
||||
render(
|
||||
<CharacterGenerationHarness
|
||||
initialDialog={createDialog()}
|
||||
/>,
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole('button', { name: '角色规范A' }));
|
||||
expect(screen.getByRole('menu', { name: '角色形象规范来源' })).toBeTruthy();
|
||||
|
||||
fireEvent.click(screen.getByRole('textbox', { name: '角色设定' }));
|
||||
|
||||
expect(
|
||||
screen.queryByRole('menu', { name: '角色形象规范来源' }),
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user