Merge branch 'codex/editor-image-library-bugfix' into codex/editor-asset-library

This commit is contained in:
2026-06-19 16:24:21 +08:00
31 changed files with 1058 additions and 111 deletions
@@ -2411,3 +2411,10 @@
- 决策:`/editor/canvas``生成游戏音效` 入口继续保留,但默认且暂时唯一可用模型为 Vidu `audio1.0`,前端请求固定发送 `prompt``model: "audio1.0"``duration``priceMudPoints`,面板只显示 `Vidu``2-10` 秒时长选项,默认 `5` 秒;不再展示 `type``tempo`、BPM 或 Suno 文生音效入口。后端 `/api/editor/audios/sound-effects/generations` 只接受空模型或 `audio1.0`,拒绝 Suno / `chirp-*`;提交到 VectorEngine 时对内 `prompt` 同步映射为上游 body 的 `prompt``sound`,兼容 Apifox 文档和线上网关实际 `missing field sound` 校验;提交和轮询改走 Vidu `/ent/v2/text2audio``/ent/v2/tasks/{taskId}/creations`。背景音乐仍保留 Suno `/suno/submit/music``/suno/fetch/{taskId}` 和 wav clip 兜底逻辑。
- 影响范围:`server-rs/crates/platform-audio``server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs``server-rs/crates/shared-contracts/src/assets.rs``src/services/image-editor/editorProjectClient.ts``src/components/image-editor/ImageCanvasGeneration*`
- 验证方式:`cargo test -p platform-audio --manifest-path server-rs/Cargo.toml --test vector_engine_audio``cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_sound_effect``cargo test -p shared-contracts --manifest-path server-rs/Cargo.toml editor_audio_requests_and_response_use_canvas_audio_shape``npx vitest run src/services/image-editor/editorProjectClient.test.ts src/components/image-editor/ImageCanvasGenerationSubmissionModel.test.ts src/components/image-editor/ImageCanvasGenerationComposerView.test.tsx src/components/image-editor/ImageCanvasGenerationModel.test.ts --reporter=dot`
## 2026-06-19 角色主图抠图补充内部镂空检测
- 背景:编辑器角色形象和 Big Fish 正式图复用 `character_visual_assets::try_apply_background_alpha_to_png` 的角色主图透明背景后处理;原算法主要从画布四边扩散清理绿幕 / 近白背景,对主体包围的内部绿幕或近白镂空不稳定。
- 决策:角色主图抠图继续保留在 `api-server``character_visual_assets` 口径内,不切换到 `platform-image::generated_asset_sheets`。在边缘连通背景 BFS 之后、软边扩展和边缘去污染之前,新增内部背景连通域检测:只清理不触画布边界、像素数达到阈值、且为高置信绿幕 / 近白背景的内部连通域,避免误伤普通角色纹理。
- 影响范围:`server-rs/crates/api-server/src/character_visual_assets.rs``docs/【编辑器】画板角色形象生成入口设计-2026-06-15.md`
- 验证方式:`cargo test -p api-server --manifest-path server-rs/Cargo.toml character_background_alpha_removes_internal_green_holes``cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_character_image_postprocess``cargo check -p api-server --manifest-path server-rs/Cargo.toml``npm run check:encoding``git diff --check`
@@ -111,7 +111,7 @@
- 生成占位图和生成器对话框不是临时浮层,必须作为画布布局数据保存。
- 保存时在现有画布布局数组中追加 `itemType: "generation-dialog"` 项,记录生成器 ID、模式、提示词、参数、参考图、状态、占位框位置和 `generatedLayerId`
- 生成成功后仍保留生成器快照;画布渲染优先用 `generatedLayerId` 锚定到成品图层,不再重复显示灰色占位框。
- 图片生成结果要写入账号级素材库;视频结果先只作为画布资源和视频图层保存
- 图片、视频和音频生成结果要写入账号级素材库;视频 / 音频结果由后端持久化到 OSS 并回传 `objectKey` / `assetObjectId`,前端保存素材库时一并记录,后续预览和再次加入画布走统一换签链路
- 刷新项目后,画布需要同时恢复图层、生成器快照和生成输入框跟随关系。
## 第一版计费配置
@@ -64,7 +64,7 @@
角色设定:<用户输入的角色设定>
```
- 角色图生成完成后,后端必须先对返回图片执行绿幕 / 近白背景去背,并统一输出透明背景 PNG;随后写入 OSS 私有对象,并确认 `asset_object`。接口回包仍返回透明 PNG Data URL 供画板立即显示,同时返回 `objectKey` / `assetObjectId`,前端创建图层和画板资源记录时必须保存这两个字段。
- 角色图生成完成后,后端必须先对返回图片执行绿幕 / 近白背景去背,连同角色主体包围的内部绿幕 / 近白镂空区域一起转为透明,并统一输出透明背景 PNG;随后写入 OSS 私有对象,并确认 `asset_object`。接口回包仍返回透明 PNG Data URL 供画板立即显示,同时返回 `objectKey` / `assetObjectId`,前端创建图层和画板资源记录时必须保存这两个字段。
## 生成规范参考图
@@ -111,7 +111,7 @@
- 画布生成类入口已统一接入新建占位图落点避让:优先使用当前屏幕中心对应画板位置,重叠时自动选择最近的不重叠位置,并将视口中心移动到新占位图。
- 角色生成提交统一走 `/api/editor/images/generations`,按 `角色规范 -> 常规参考图` 顺序传 `referenceImageSrcs`,并写入 `assetKind: "character"`
- 角色和图标素材生成已接入 `nanobanana2` / `gpt-image-2` 模型切换、上次模型记忆,以及按模型归一的比例 / 大小尺寸;`nanobanana2` 使用原生 `generateContent``imageConfig.aspectRatio/imageSize``gpt-image-2` 使用文档列出的 `size` 字符串。
- 角色生成后端已按固定 prompt 骨架补入 `角色设定`,并在生成成功后自动执行绿幕去背、写入 `generated-character-drafts/editor/character-images/<taskId>/image.png` 路径下的 OSS 私有对象,返回的 `objectKey` / `assetObjectId` 会随画板资源记录保存。
- 角色生成后端已按固定 prompt 骨架补入 `角色设定`,并在生成成功后自动执行绿幕 / 近白背景去背和内部镂空清理、写入 `generated-character-drafts/editor/character-images/<taskId>/image.png` 路径下的 OSS 私有对象,返回的 `objectKey` / `assetObjectId` 会随画板资源记录保存。
- `Esc` 只退出角色规范画布点选状态,不关闭角色生成面板。
- 已补充回归测试覆盖角色形象生成、点选退出、角色动画入口隔离和快速编辑入口。
- 本次验证命令:
@@ -42,9 +42,10 @@
- 新增素材类型:
- `assetKind="sound-effect"`
- `assetKind="background-music"`
- 音频结果以小型音频卡加入画布,卡片底部使用融入卡片的自定义播放器组件承载进度、时间和音量,播放 / 暂停只收口到卡片中央图标按钮;音频结果当前只保存为画布资源,不进入图片素材库
- 音频结果卡片底部显示用户提示词与播放器辅助控件,不在卡片左下角或悬停左上角展示时长
- 音频结果卡片的提示词固定显示在卡片左上角;底部播放器辅助控件仅在鼠标悬停音频卡片时从下方滑入显示,移出后向下滑出收起
- 音频结果以小型音频卡加入画布,卡片底部使用融入卡片的自定义播放器组件承载进度、时间和音量,播放 / 暂停只收口到卡片中央图标按钮;生成成功后同时保存为 OSS 私有对象、画布资源和账号级素材库素材,响应携带 `objectKey` / `assetObjectId` 供后续换签和复用
- 普通素材上传入口首版支持图片、MP3 和 MP4;MP3 / MP4 先走 OSS 直传和 asset object confirm,再以素材库素材保存,其他音视频格式暂不开放
- 音频结果卡片底部显示播放器辅助控件,不在卡片左下角或悬停左上角展示时长;提示词固定显示在卡片左上角
- 音频结果卡片底部播放器辅助控件仅在鼠标悬停音频卡片时从下方滑入显示,移出后向下滑出收起。
- 音频卡片中央播放区按状态切换:未悬停且未播放时显示音效 / 背景音乐图标,悬停且未播放时显示播放按钮,播放中始终显示暂停按钮。
- generated 私有音频资源播放前必须通过 `/api/assets/read-url` 换签;画布卡片不得直接把 `/generated-*` 或 generated OSS 私有地址交给 `<audio>` 裸请求。
- 音频元数据弹窗使用 `时长`,不使用图片 / 视频的分辨率语义;音频生成占位不显示分辨率或时长角标。
@@ -81,6 +82,8 @@ POST /api/editor/audios/background-music/generations
{
ok: true,
audioSrc: string,
objectKey?: string | null,
assetObjectId?: string | null,
width: 420,
height: 120,
sourceType: "generated",
@@ -109,6 +112,7 @@ POST /api/editor/audios/background-music/generations
- `/api/editor/audios/sound-effects/generations`
- `/api/editor/audios/background-music/generations`
- BFF 复用现有 `vector_engine_audio_generation` 的任务轮询、下载、OSS 持久化和计费包装;音效 10 泥点,背景音乐 5 泥点。
- 生成音频持久化后返回 OSS `objectKey``assetObjectId`;前端保存素材库时继续使用 `audioSrc` 作为兼容路径,并把 OSS 身份写入素材记录。
## 验收
@@ -120,6 +124,8 @@ POST /api/editor/audios/background-music/generations
- 点击 `生成游戏背景音乐` 后出现背景音乐面板,字段为 `gpt_description_prompt`,右侧固定显示 `Suno` 模型胶囊,不展示 `make_instrumental`
- 音效提交到 `/api/editor/audios/sound-effects/generations`,背景音乐提交到 `/api/editor/audios/background-music/generations`
- 成功后画布新增音频卡,能通过卡片中央播放按钮播放,底部进度、时间和音量控件可操作。
- 成功后音频素材自动出现在账号级素材库;从素材库再次添加到画布时仍恢复为 `mediaType="audio"` 音频图层。
- 普通素材上传 MP3 / MP4 会写入 OSS 并进入素材库;MP3 添加到画布为音频图层,MP4 添加到画布为视频图层。
- 成功后的音频卡展示提示词;卡片中央按未悬停图标、悬停播放、播放中暂停切换,不在底部重复显示播放 / 暂停按钮,播放器辅助控件直接融入卡片底部。
- 鼠标未悬停音频卡时底部播放器辅助控件不可见,悬停时从底部滑动显现;提示词位于卡片左上角。
- 音频卡片、悬停角标和待生成占位不显示时长;信息弹窗显示时长。
@@ -135,6 +135,12 @@ const BUILT_IN_MOTION_TEMPLATES: [MotionTemplate; 4] = [
},
];
struct PersistedEditorVideo {
legacy_public_path: String,
object_key: String,
asset_object_id: String,
}
pub async fn list_character_animation_templates(
Extension(request_context): Extension<RequestContext>,
) -> Json<Value> {
@@ -630,6 +636,8 @@ pub async fn generate_editor_video(
EditorVideoGenerateResponse {
ok: true,
video_src: generated.preview_video_path,
object_key: generated.object_key,
asset_object_id: generated.asset_object_id,
width: normalized.width,
height: normalized.height,
source_type: "generated".to_string(),
@@ -1146,6 +1154,8 @@ async fn request_image_to_video_preview(
Ok(GeneratedAnimationPreview {
preview_video_path,
object_key: None,
asset_object_id: None,
upstream_task_id,
submitted_prompt,
moderation_fallback_applied,
@@ -1438,6 +1448,8 @@ async fn request_editor_character_animation_preview(
Ok(GeneratedAnimationPreview {
preview_video_path,
object_key: None,
asset_object_id: None,
upstream_task_id,
submitted_prompt: prompt.to_string(),
moderation_fallback_applied: false,
@@ -1459,11 +1471,13 @@ async fn request_editor_video_preview(
.await?;
let preview_payload =
download_generated_video(http_client, video_url.as_str(), "下载画板生成视频失败。").await?;
let preview_video_path =
let persisted =
put_generated_editor_video(state, owner_user_id, task_id, request, preview_payload).await?;
Ok(GeneratedAnimationPreview {
preview_video_path,
preview_video_path: persisted.legacy_public_path,
object_key: Some(persisted.object_key),
asset_object_id: Some(persisted.asset_object_id),
upstream_task_id,
submitted_prompt: request.prompt.clone(),
moderation_fallback_applied: false,
@@ -1519,7 +1533,8 @@ async fn put_generated_editor_video(
task_id: &str,
request: &NormalizedEditorVideoRequest,
preview_payload: MediaPayload,
) -> Result<String, AppError> {
) -> Result<PersistedEditorVideo, AppError> {
let content_type = preview_payload.mime_type.clone();
let put_result = put_character_animation_object(
state,
LegacyAssetPrefix::CharacterDrafts,
@@ -1541,7 +1556,19 @@ async fn put_generated_editor_video(
),
)
.await?;
Ok(put_result.legacy_public_path)
let confirmed = confirm_editor_video_asset_object(
state,
owner_user_id,
task_id,
put_result.object_key.clone(),
content_type,
)
.await?;
Ok(PersistedEditorVideo {
legacy_public_path: put_result.legacy_public_path,
object_key: put_result.object_key,
asset_object_id: confirmed.record.asset_object_id,
})
}
async fn create_editor_ark_image_to_video_task(
@@ -1978,6 +2005,45 @@ async fn confirm_character_animation_asset_object(
Ok(module_assets::ConfirmAssetObjectResult { record })
}
async fn confirm_editor_video_asset_object(
state: &AppState,
owner_user_id: &str,
task_id: &str,
object_key: String,
content_type: String,
) -> Result<module_assets::ConfirmAssetObjectResult, AppError> {
let oss_client = require_oss_client(state)?;
let head = oss_client
.head_object(&reqwest::Client::new(), OssHeadObjectRequest { object_key })
.await
.map_err(map_character_animation_oss_error)?;
let now_micros = current_utc_micros();
let record = state
.spacetime_client()
.confirm_asset_object(
build_asset_object_upsert_input(
generate_asset_object_id(now_micros),
head.bucket,
head.object_key,
AssetObjectAccessPolicy::Private,
head.content_type.or(Some(content_type)),
head.content_length,
head.etag,
EDITOR_VIDEO_ASSET_KIND.to_string(),
Some(task_id.to_string()),
Some(owner_user_id.to_string()),
None,
Some(task_id.to_string()),
now_micros,
)
.map_err(map_asset_object_prepare_error)?,
)
.await
.map_err(map_character_animation_spacetime_error)?;
Ok(module_assets::ConfirmAssetObjectResult { record })
}
async fn bind_character_animation_asset(
state: &AppState,
owner_user_id: &str,
@@ -4477,6 +4543,8 @@ struct NormalizedEditorVideoRequest {
struct GeneratedAnimationPreview {
preview_video_path: String,
object_key: Option<String>,
asset_object_id: Option<String>,
upstream_task_id: String,
submitted_prompt: String,
moderation_fallback_applied: bool,
@@ -1372,6 +1372,7 @@ fn collect_foreground_neighbor_color(
}
pub(crate) fn remove_background_from_rgba(pixels: &mut [u8], width: usize, height: usize) -> bool {
const INTERNAL_BACKGROUND_HOLE_MIN_PIXELS: usize = 16;
const SOFT_EDGE_ALPHA_THRESHOLD: u8 = 224;
const FOREGROUND_NEIGHBOR_ALPHA_THRESHOLD: u8 = 96;
@@ -1477,6 +1478,18 @@ pub(crate) fn remove_background_from_rgba(pixels: &mut [u8], width: usize, heigh
}
}
if mark_internal_background_holes(
pixels,
width,
height,
&mut background_mask,
&green_scores,
&white_scores,
INTERNAL_BACKGROUND_HOLE_MIN_PIXELS,
) {
changed = true;
}
for _ in 0..2 {
let mut expanded_mask = background_mask.clone();
for y in 0..height {
@@ -1712,6 +1725,138 @@ pub(crate) fn remove_background_from_rgba(pixels: &mut [u8], width: usize, heigh
changed
}
fn mark_internal_background_holes(
pixels: &[u8],
width: usize,
height: usize,
background_mask: &mut [u8],
green_scores: &[f32],
white_scores: &[f32],
min_pixels: usize,
) -> bool {
let pixel_count = width.saturating_mul(height);
if pixel_count == 0
|| min_pixels == 0
|| pixels.len() < pixel_count.saturating_mul(4)
|| background_mask.len() < pixel_count
|| green_scores.len() < pixel_count
|| white_scores.len() < pixel_count
{
return false;
}
let mut visited = vec![false; pixel_count];
let mut queue = Vec::<usize>::new();
let mut component = Vec::<usize>::new();
let mut changed = false;
for start_index in 0..pixel_count {
if background_mask[start_index] != 0
|| visited[start_index]
|| !is_internal_background_hole_candidate(
pixels,
start_index,
green_scores[start_index],
white_scores[start_index],
)
{
continue;
}
queue.clear();
component.clear();
queue.push(start_index);
visited[start_index] = true;
let mut queue_index = 0usize;
let mut touches_border = false;
while queue_index < queue.len() {
let pixel_index = queue[queue_index];
queue_index += 1;
component.push(pixel_index);
let x = pixel_index % width;
let y = pixel_index / width;
if x == 0 || x == width.saturating_sub(1) || y == 0 || y == height.saturating_sub(1) {
touches_border = true;
}
let neighbors = [
if x > 0 { Some(pixel_index - 1) } else { None },
if x + 1 < width {
Some(pixel_index + 1)
} else {
None
},
if y > 0 {
Some(pixel_index - width)
} else {
None
},
if y + 1 < height {
Some(pixel_index + width)
} else {
None
},
];
for next_index in neighbors.into_iter().flatten() {
if background_mask[next_index] != 0 || visited[next_index] {
continue;
}
if !is_internal_background_hole_candidate(
pixels,
next_index,
green_scores[next_index],
white_scores[next_index],
) {
continue;
}
visited[next_index] = true;
queue.push(next_index);
}
}
if touches_border || component.len() < min_pixels {
continue;
}
for pixel_index in component.drain(..) {
background_mask[pixel_index] = 1;
}
changed = true;
}
changed
}
fn is_internal_background_hole_candidate(
pixels: &[u8],
pixel_index: usize,
green_score: f32,
white_score: f32,
) -> bool {
let offset = pixel_index * 4;
let Some(pixel) = pixels.get(offset..offset + 4) else {
return false;
};
let red = pixel[0];
let green = pixel[1];
let blue = pixel[2];
let alpha = pixel[3];
if alpha < 40 {
return true;
}
let hard_green_key =
green_score >= 0.82 && green >= 180 && red <= 72 && blue <= 96 && green > red.max(blue);
let hard_white_key = white_score >= 0.62
&& red.min(green).min(blue) >= 188
&& red.max(green).max(blue) - red.min(green).min(blue) <= 42;
hard_green_key || hard_white_key
}
fn character_visual_error_response(request_context: &RequestContext, error: AppError) -> Response {
error.into_response_with_context(Some(request_context))
}
@@ -1792,6 +1937,33 @@ mod tests {
assert!(is_image_test_moderation_error(&error));
}
#[test]
fn character_background_alpha_removes_internal_green_holes() {
let width = 16;
let height = 16;
let mut image = image::RgbaImage::from_pixel(width, height, image::Rgba([0, 255, 0, 255]));
for y in 3..13 {
for x in 3..13 {
image.put_pixel(x, y, image::Rgba([188, 82, 45, 255]));
}
}
for y in 6..10 {
for x in 6..10 {
image.put_pixel(x, y, image::Rgba([0, 255, 0, 255]));
}
}
assert!(remove_background_from_rgba(
image.as_mut(),
width as usize,
height as usize
));
assert_eq!(image.get_pixel(0, 0).0[3], 0);
assert_eq!(image.get_pixel(7, 7).0[3], 0);
assert_eq!(image.get_pixel(4, 4).0[3], 255);
}
#[test]
fn sanitize_storage_segment_keeps_legacy_safe_shape() {
assert_eq!(
@@ -163,6 +163,8 @@ pub async fn generate_editor_sound_effect(
assets::EditorAudioGenerateResponse {
ok: true,
audio_src,
object_key: generated.object_key,
asset_object_id: generated.asset_object_id,
width: EDITOR_AUDIO_WIDTH,
height: EDITOR_AUDIO_HEIGHT,
source_type: "generated".to_string(),
@@ -222,6 +224,8 @@ pub async fn generate_editor_background_music(
assets::EditorAudioGenerateResponse {
ok: true,
audio_src,
object_key: generated.object_key,
asset_object_id: generated.asset_object_id,
width: EDITOR_AUDIO_WIDTH,
height: EDITOR_AUDIO_HEIGHT,
source_type: "generated".to_string(),
@@ -17,6 +17,7 @@ use super::{
#[derive(Clone, Debug)]
pub(super) struct PersistedAudioAsset {
pub(super) asset_object_id: String,
pub(super) object_key: String,
pub(super) audio_src: String,
}
@@ -111,6 +112,7 @@ pub(super) async fn persist_generated_audio_asset(
Ok(PersistedAudioAsset {
asset_object_id: asset_object.asset_object_id,
object_key: put_result.object_key,
audio_src: put_result.legacy_public_path,
})
}
@@ -65,6 +65,7 @@ pub(super) async fn publish_generated_audio_asset_with_task_kind(
provider: task_kind.provider().to_string(),
status: status.clone(),
asset_object_id: None,
object_key: None,
asset_kind: None,
audio_src: None,
});
@@ -118,6 +119,7 @@ pub(super) async fn publish_generated_audio_asset_with_task_kind(
provider: task_kind.provider().to_string(),
status: "completed".to_string(),
asset_object_id: Some(persisted.asset_object_id),
object_key: Some(persisted.object_key),
asset_kind: Some(target.asset_kind),
audio_src: Some(persisted.audio_src),
})
@@ -370,6 +370,10 @@ pub struct EditorVideoGenerateRequest {
pub struct EditorVideoGenerateResponse {
pub ok: bool,
pub video_src: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub object_key: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub asset_object_id: Option<String>,
pub width: u32,
pub height: u32,
pub source_type: String,
@@ -446,6 +450,10 @@ pub struct EditorBackgroundMusicGenerateRequest {
pub struct EditorAudioGenerateResponse {
pub ok: bool,
pub audio_src: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub object_key: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub asset_object_id: Option<String>,
pub width: u32,
pub height: u32,
pub source_type: String,
@@ -1091,6 +1099,8 @@ mod tests {
let payload = serde_json::to_value(EditorVideoGenerateResponse {
ok: true,
video_src: "/generated-editor-videos/task-1/preview.mp4".to_string(),
object_key: Some("generated-character-drafts/editor-videos/task-1/preview.mp4".to_string()),
asset_object_id: Some("assetobj_video_1".to_string()),
width: 1280,
height: 720,
source_type: "generated".to_string(),
@@ -1109,6 +1119,11 @@ mod tests {
payload["videoSrc"],
json!("/generated-editor-videos/task-1/preview.mp4")
);
assert_eq!(
payload["objectKey"],
json!("generated-character-drafts/editor-videos/task-1/preview.mp4")
);
assert_eq!(payload["assetObjectId"], json!("assetobj_video_1"));
assert_eq!(payload["sourceType"], json!("generated"));
assert_eq!(payload["durationSeconds"], json!(5));
}
@@ -1205,6 +1220,8 @@ mod tests {
let response_payload = serde_json::to_value(EditorAudioGenerateResponse {
ok: true,
audio_src: "/generated-character-drafts/editor-audios/sfx.mp3".to_string(),
object_key: Some("generated-character-drafts/editor-audios/sfx.mp3".to_string()),
asset_object_id: Some("assetobj_audio_1".to_string()),
width: 420,
height: 120,
source_type: "generated".to_string(),
@@ -1221,6 +1238,11 @@ mod tests {
response_payload["audioSrc"],
json!("/generated-character-drafts/editor-audios/sfx.mp3")
);
assert_eq!(
response_payload["objectKey"],
json!("generated-character-drafts/editor-audios/sfx.mp3")
);
assert_eq!(response_payload["assetObjectId"], json!("assetobj_audio_1"));
assert_eq!(response_payload["audioKind"], json!("sound-effect"));
}
@@ -88,6 +88,8 @@ pub struct GeneratedAudioAssetResponse {
#[serde(default)]
pub asset_object_id: Option<String>,
#[serde(default)]
pub object_key: Option<String>,
#[serde(default)]
pub asset_kind: Option<String>,
#[serde(default)]
pub audio_src: Option<String>,
@@ -128,4 +128,20 @@ describe('ImageCanvasAssetRowView', () => {
screen.getByLabelText('素材账号素材A上传进度').getAttribute('value'),
).toBe('42');
});
it('uses icon thumbnails for audio assets instead of rendering media as images', () => {
renderAssetRow({
asset: createAsset({
label: '胜利音效.mp3',
src: '/generated-character-drafts/editor/asset-library/audio/胜利音效.mp3',
mediaType: 'audio',
}),
});
expect(screen.getByText('音频')).toBeTruthy();
expect(screen.queryByRole('img', { name: '素材:胜利音效.mp3' })).toBeNull();
expect(
document.querySelector('.image-canvas-editor__asset-media-overlay'),
).toBeTruthy();
});
});
@@ -1,4 +1,4 @@
import { Check, Pencil, Trash2, X } from 'lucide-react';
import { Check, Music2, Pencil, Trash2, Video, X } from 'lucide-react';
import type { Dispatch, SetStateAction } from 'react';
import { PlatformTextField } from '../common/PlatformTextField';
@@ -65,6 +65,22 @@ export function ImageCanvasAssetRowView({
const isUploadingAsset = asset.uploadStatus === 'uploading';
const isFailedUpload = asset.uploadStatus === 'failed';
const uploadProgress = clamp(asset.uploadProgress ?? 0, 0, 100);
const mediaDetail =
asset.mediaType === 'audio'
? '音频'
: asset.mediaType === 'video'
? '视频'
: `${asset.width} x ${asset.height}`;
const mediaOverlay =
asset.mediaType === 'audio' ? (
<div className="image-canvas-editor__asset-media-overlay">
<Music2 className="h-5 w-5" />
</div>
) : asset.mediaType === 'video' ? (
<div className="image-canvas-editor__asset-media-overlay">
<Video className="h-5 w-5" />
</div>
) : undefined;
const titleNode = isRenaming ? (
<PlatformTextField
aria-label={`重命名素材${asset.label}`}
@@ -134,8 +150,8 @@ export function ImageCanvasAssetRowView({
<div data-asset-id={asset.id}>
<SidebarMediaItem
title={asset.label}
detail={`${asset.width} x ${asset.height}`}
imageSrc={asset.src}
detail={mediaDetail}
imageSrc={asset.mediaType === 'image' ? asset.src : ''}
imageAlt={`素材:${asset.label}`}
primaryLabel={
isUploadingAsset
@@ -182,7 +198,9 @@ export function ImageCanvasAssetRowView({
<div className="image-canvas-editor__asset-upload-overlay">
<span></span>
</div>
) : undefined
) : (
mediaOverlay
)
}
footerNode={
isUploadingAsset || isFailedUpload ? (
@@ -9,7 +9,7 @@ import {
within,
} from '@testing-library/react';
import userEvent from '@testing-library/user-event';
import { describe, expect, it, vi } from 'vitest';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import {
ApiClientError,
@@ -38,6 +38,7 @@ const loadEditorProjectMock = vi.hoisted(() => vi.fn());
const loadOrCreateRecentEditorProjectMock = vi.hoisted(() => vi.fn());
const renameEditorProjectMock = vi.hoisted(() => vi.fn());
const saveEditorProjectLayoutMock = vi.hoisted(() => vi.fn());
const uploadEditorMediaAssetFileMock = vi.hoisted(() => vi.fn());
vi.mock('../../services/image-editor/editorProjectClient', async () => {
const actual = await vi.importActual<
@@ -64,6 +65,10 @@ vi.mock('../../services/image-editor/editorProjectClient', async () => {
};
});
vi.mock('../../services/image-editor/editorMediaAssetUploadClient', () => ({
uploadEditorMediaAssetFile: uploadEditorMediaAssetFileMock,
}));
describe('ImageCanvasEditorView asset library integration', () => {
setupImageCanvasEditorViewTestLifecycle({
generateEditorImageMock,
@@ -84,6 +89,17 @@ describe('ImageCanvasEditorView asset library integration', () => {
saveEditorProjectLayoutMock,
});
beforeEach(() => {
uploadEditorMediaAssetFileMock.mockImplementation(
async (file: File, mediaType: 'video' | 'audio') => ({
src: `https://signed.example.com/editor/${mediaType}/${file.name}`,
legacyPublicPath: `/generated-character-drafts/editor/asset-library/${mediaType}/${file.name}`,
objectKey: `generated-character-drafts/editor/asset-library/${mediaType}/${file.name}`,
assetObjectId: `assetobj-editor-${mediaType}`,
}),
);
});
it('keeps only one default asset folder when the persisted library returns duplicated defaults', async () => {
loadEditorAssetLibraryMock.mockResolvedValueOnce({
folders: [
@@ -202,7 +218,7 @@ describe('ImageCanvasEditorView asset library integration', () => {
await user.type(folderNameInput, '角色上传');
await user.click(screen.getByRole('button', { name: '保存素材文件夹' }));
const uploadInput = screen.getByLabelText('上传图片文件');
const uploadInput = screen.getByLabelText('上传媒体文件');
await user.click(screen.getByRole('button', { name: '上传到角色上传' }));
await userEvent.upload(
uploadInput,
@@ -346,7 +362,7 @@ describe('ImageCanvasEditorView asset library integration', () => {
it('uploads multiple files as account-level assets without adding canvas layers', async () => {
render(<ImageCanvasEditorView />);
await userEvent.upload(screen.getByLabelText('上传图片文件'), [
await userEvent.upload(screen.getByLabelText('上传媒体文件'), [
new File(['image-a'], '第一张.png', { type: 'image/png' }),
new File(['image-b'], '第二张.png', { type: 'image/png' }),
]);
@@ -364,6 +380,79 @@ describe('ImageCanvasEditorView asset library integration', () => {
expect(screen.queryByAltText('画布图片:第二张.png')).toBeNull();
});
it('uploads MP3 and MP4 files through OSS before adding them to the asset library', async () => {
const createdMediaElements: HTMLMediaElement[] = [];
const originalCreateElement = document.createElement.bind(document);
vi.spyOn(document, 'createElement').mockImplementation((tagName, options) => {
const element = originalCreateElement(tagName, options);
if (tagName === 'video' || tagName === 'audio') {
createdMediaElements.push(element as HTMLMediaElement);
}
return element;
});
Object.defineProperty(URL, 'createObjectURL', {
configurable: true,
value: vi.fn(() => 'blob:editor-media'),
});
Object.defineProperty(URL, 'revokeObjectURL', {
configurable: true,
value: vi.fn(),
});
Object.defineProperty(HTMLMediaElement.prototype, 'duration', {
configurable: true,
value: 6,
});
vi.spyOn(HTMLMediaElement.prototype, 'load').mockImplementation(
() => undefined,
);
render(<ImageCanvasEditorView />);
await userEvent.upload(screen.getByLabelText('上传媒体文件'), [
new File(['audio'], '胜利音效.mp3', { type: 'audio/mpeg' }),
new File(['video'], '开场动画.mp4', { type: 'video/mp4' }),
]);
act(() => {
createdMediaElements.forEach((element) =>
element.onloadedmetadata?.(new Event('loadedmetadata')),
);
});
await waitFor(() => {
expect(uploadEditorMediaAssetFileMock).toHaveBeenCalledWith(
expect.objectContaining({ name: '胜利音效.mp3' }),
'audio',
);
expect(uploadEditorMediaAssetFileMock).toHaveBeenCalledWith(
expect.objectContaining({ name: '开场动画.mp4' }),
'video',
);
});
await waitFor(() => {
expect(screen.getByRole('button', { name: '添加胜利音效.mp3' })).toBeTruthy();
expect(screen.getByRole('button', { name: '添加开场动画.mp4' })).toBeTruthy();
});
expect(createEditorAssetMock).toHaveBeenCalledWith(
expect.objectContaining({
label: '胜利音效.mp3',
imageSrc:
'/generated-character-drafts/editor/asset-library/audio/胜利音效.mp3',
objectKey:
'generated-character-drafts/editor/asset-library/audio/胜利音效.mp3',
assetObjectId: 'assetobj-editor-audio',
}),
);
expect(createEditorAssetMock).toHaveBeenCalledWith(
expect.objectContaining({
label: '开场动画.mp4',
imageSrc:
'/generated-character-drafts/editor/asset-library/video/开场动画.mp4',
objectKey:
'generated-character-drafts/editor/asset-library/video/开场动画.mp4',
assetObjectId: 'assetobj-editor-video',
}),
);
});
it('opens login before uploading assets while logged out and resumes after login', async () => {
const openLoginModal = vi.fn();
const authValue = createAuthValue({ openLoginModal });
@@ -374,7 +463,7 @@ describe('ImageCanvasEditorView asset library integration', () => {
</AuthUiContext.Provider>,
);
await userEvent.upload(screen.getByLabelText('上传图片文件'), [
await userEvent.upload(screen.getByLabelText('上传媒体文件'), [
new File(['image'], '登录后上传.png', { type: 'image/png' }),
]);
@@ -429,7 +518,7 @@ describe('ImageCanvasEditorView asset library integration', () => {
createEditorAssetMock.mockReturnValueOnce(deferredAsset.promise);
render(<ImageCanvasEditorView />);
await userEvent.upload(screen.getByLabelText('上传图片文件'), [
await userEvent.upload(screen.getByLabelText('上传媒体文件'), [
new File(['image'], '素材上传进度.png', { type: 'image/png' }),
]);
@@ -489,7 +578,7 @@ describe('ImageCanvasEditorView asset library integration', () => {
</AuthUiContext.Provider>,
);
await userEvent.upload(screen.getByLabelText('上传图片文件'), [
await userEvent.upload(screen.getByLabelText('上传媒体文件'), [
new File(['image'], '过期登录.png', { type: 'image/png' }),
]);
@@ -1199,7 +1199,7 @@ describe('ImageCanvasEditorView generation integration', () => {
);
fireEvent.click(screen.getByRole('menuitem', { name: '上传图片' }));
await userEvent.upload(
screen.getByLabelText('上传图片文件'),
screen.getByLabelText('上传媒体文件'),
new File(['icon-spec'], '图标规范.png', { type: 'image/png' }),
);
await waitFor(() => {
@@ -1689,7 +1689,7 @@ describe('ImageCanvasEditorView generation integration', () => {
),
);
await userEvent.upload(
screen.getByLabelText('上传图片文件'),
screen.getByLabelText('上传媒体文件'),
new File(['spec-reference'], '规范上传.png', { type: 'image/png' }),
);
@@ -1794,7 +1794,7 @@ describe('ImageCanvasEditorView generation integration', () => {
);
fireEvent.click(screen.getByRole('menuitem', { name: '上传图片' }));
await userEvent.upload(
screen.getByLabelText('上传图片文件'),
screen.getByLabelText('上传媒体文件'),
new File(['reference'], '常规参考.png', { type: 'image/png' }),
);
await waitFor(() => {
@@ -50,6 +50,58 @@ describe('ImageCanvasEditorModel', () => {
expect(library.folders[0]?.id).toBe('project');
});
it('infers MP3 and MP4 asset media types from persisted paths', () => {
const library = normalizeAssetLibrary({
folders: [
{
folderId: 'project',
label: '项目素材',
sortOrder: 0,
collapsed: false,
systemDefault: true,
},
],
assets: [
{
assetId: 'asset-audio',
folderId: 'project',
label: '胜利音效.mp3',
imageSrc: '/generated-character-drafts/editor/asset-library/audio/胜利音效.mp3',
width: 420,
height: 120,
sourceType: 'uploaded',
objectKey:
'generated-character-drafts/editor/asset-library/audio/胜利音效.mp3',
},
{
assetId: 'asset-video',
folderId: 'project',
label: '开场动画.mp4',
imageSrc: '/generated-character-drafts/editor/asset-library/video/开场动画.mp4',
width: 560,
height: 315,
sourceType: 'uploaded',
objectKey:
'generated-character-drafts/editor/asset-library/video/开场动画.mp4',
},
],
});
expect(library.assets[0]).toMatchObject({ mediaType: 'audio' });
expect(library.assets[1]).toMatchObject({ mediaType: 'video' });
expect(
createLayerFromAsset(
library.assets[0]!,
1,
{ x: 0, y: 0, scale: 1 },
{ x: 300, y: 200 },
),
).toMatchObject({
mediaType: 'audio',
assetKind: 'sound-effect',
});
});
it('creates a layer from an account asset at the requested screen point', () => {
const asset: EditorAsset = {
id: 'asset-1',
@@ -166,30 +218,30 @@ describe('ImageCanvasEditorModel', () => {
originalWidth: 420,
originalHeight: 120,
zIndex: 5,
sourceType: 'generated',
mediaType: 'audio',
assetKind: 'sound-effect',
durationSeconds: 2.4,
},
new Map([
[
'resource-audio',
{
imageSrc: '/generated-character-drafts/editor-audios/sfx.mp3',
durationSeconds: 2.4,
},
],
]),
);
sourceType: 'generated',
mediaType: 'audio',
assetKind: 'sound-effect',
durationSeconds: 2.4,
},
new Map([
[
'resource-audio',
{
imageSrc: '/generated-character-drafts/editor-audios/sfx.mp3',
durationSeconds: 2.4,
},
],
]),
);
expect(hydrated).toMatchObject({
id: 'layer-audio',
src: '/generated-character-drafts/editor-audios/sfx.mp3',
mediaType: 'audio',
assetKind: 'sound-effect',
durationSeconds: 2.4,
src: '/generated-character-drafts/editor-audios/sfx.mp3',
mediaType: 'audio',
assetKind: 'sound-effect',
durationSeconds: 2.4,
});
});
});
it('restores audio generation dialogs from saved layout', () => {
const dialog: CanvasGenerationDialogState = {
@@ -114,12 +114,19 @@ export function createLayerFromAsset(
const worldCenterX = (safeScreenCenter.x - viewport.x) / safeScale;
const worldCenterY = (safeScreenCenter.y - viewport.y) / safeScale;
const offset = index * 34;
const assetKind: CanvasAssetKind | undefined =
asset.mediaType === 'video'
? 'video'
: asset.mediaType === 'audio'
? inferAudioAssetKindFromLabel(asset.label)
: undefined;
return {
id: `layer-${asset.id}-${index}`,
resourceId: `local-resource-${asset.id}-${index}`,
title: asset.label,
src: asset.src,
mediaType: asset.mediaType,
x: worldCenterX - width / 2 + offset,
y: worldCenterY - height / 2 + offset,
width,
@@ -137,6 +144,7 @@ export function createLayerFromAsset(
assetObjectId: asset.assetObjectId,
durationSeconds: asset.durationSeconds,
sourceAssetId: asset.id,
assetKind,
} satisfies CanvasLayer;
}
@@ -436,28 +444,56 @@ export function mapAssetLibrarySnapshot(
systemDefault: folder.systemDefault,
persisted: true,
})),
assets: library.assets.map((asset) => ({
id: asset.assetId,
label: asset.label,
src: asset.imageSrc,
width: asset.width,
height: asset.height,
folderId: asset.folderId,
sourceKind: 'uploaded',
sourceType: asset.sourceType,
persisted: true,
prompt: asset.prompt ?? undefined,
actualPrompt: asset.actualPrompt ?? undefined,
model: asset.model ?? undefined,
provider: asset.provider ?? undefined,
taskId: asset.taskId ?? undefined,
objectKey: asset.objectKey ?? undefined,
assetObjectId: asset.assetObjectId ?? undefined,
durationSeconds: asset.durationSeconds ?? undefined,
})),
assets: library.assets.map((asset) => {
const mediaType = inferEditorAssetMediaType(
asset.imageSrc,
asset.objectKey ?? undefined,
);
return {
id: asset.assetId,
label: asset.label,
src: asset.imageSrc,
mediaType,
width: asset.width,
height: asset.height,
folderId: asset.folderId,
sourceKind: 'uploaded',
sourceType: asset.sourceType,
persisted: true,
prompt: asset.prompt ?? undefined,
actualPrompt: asset.actualPrompt ?? undefined,
model: asset.model ?? undefined,
provider: asset.provider ?? undefined,
taskId: asset.taskId ?? undefined,
objectKey: asset.objectKey ?? undefined,
assetObjectId: asset.assetObjectId ?? undefined,
durationSeconds: asset.durationSeconds ?? undefined,
};
}),
};
}
export function inferEditorAssetMediaType(
imageSrc: string,
objectKey?: string | null,
): CanvasMediaType {
const source = `${objectKey ?? ''} ${imageSrc}`.toLowerCase();
const sourceWithoutQuery = source.split(/[?#]/u)[0] ?? source;
if (sourceWithoutQuery.includes('.mp4')) {
return 'video';
}
if (sourceWithoutQuery.includes('.mp3')) {
return 'audio';
}
return 'image';
}
function inferAudioAssetKindFromLabel(label: string): CanvasAssetKind {
return label.includes('背景音乐') || label.toLowerCase().includes('music')
? 'background-music'
: 'sound-effect';
}
export function normalizeAssetLibrary(library: EditorAssetLibrarySnapshot) {
const mapped = mapAssetLibrarySnapshot(library);
let hasDefaultFolder = false;
@@ -261,9 +261,12 @@ export function setupImageCanvasEditorViewTestLifecycle({
folderId: input.folderId,
label: input.label,
imageSrc: input.imageSrc,
objectKey: input.objectKey,
assetObjectId: input.assetObjectId,
width: input.width,
height: input.height,
sourceType: input.sourceType,
durationSeconds: input.durationSeconds,
}));
createEditorAssetFolderMock.mockResolvedValue({
folderId: 'folder-role-persisted',
@@ -342,6 +342,7 @@ export function ImageCanvasEditorView() {
model: layer.model,
provider: layer.provider,
taskId: layer.taskId,
durationSeconds: layer.durationSeconds,
})
.then((asset) => {
setAssets((currentAssets) => [
@@ -352,6 +353,7 @@ export function ImageCanvasEditorView() {
id: asset.assetId,
label: asset.label,
src: asset.imageSrc,
mediaType: layer.mediaType,
width: asset.width,
height: asset.height,
folderId: asset.folderId,
@@ -365,6 +367,7 @@ export function ImageCanvasEditorView() {
taskId: asset.taskId ?? undefined,
objectKey: asset.objectKey ?? undefined,
assetObjectId: asset.assetObjectId ?? undefined,
durationSeconds: asset.durationSeconds ?? layer.durationSeconds,
},
]);
setAssetFolders((currentFolders) =>
@@ -2,6 +2,9 @@ export function isImageFile(file: File) {
return file.type.startsWith('image/');
}
export const EDITOR_ASSET_UPLOAD_ACCEPT =
'image/*,.mp3,.mp4,audio/mpeg,audio/mp3,video/mp4';
export const SEEDANCE_REFERENCE_ACCEPT = {
image:
'.jpg,.jpeg,.png,.webp,.bmp,.tiff,.tif,.gif,.heic,.heif,image/jpeg,image/png,image/webp,image/bmp,image/tiff,image/gif,image/heic,image/heif',
@@ -19,7 +22,7 @@ export function getEditorUploadAccept(target: string) {
if (target === 'video-reference-image') {
return SEEDANCE_REFERENCE_ACCEPT.image;
}
return 'image/*';
return EDITOR_ASSET_UPLOAD_ACCEPT;
}
export const SEEDANCE_REFERENCE_FILE_LIMITS = {
@@ -45,6 +48,8 @@ const SEEDANCE_IMAGE_EXTENSIONS = new Set([
]);
const SEEDANCE_VIDEO_EXTENSIONS = new Set(['mp4', 'mov']);
const SEEDANCE_AUDIO_EXTENSIONS = new Set(['wav', 'mp3']);
const EDITOR_ASSET_VIDEO_EXTENSIONS = new Set(['mp4']);
const EDITOR_ASSET_AUDIO_EXTENSIONS = new Set(['mp3']);
function getFileExtension(file: File) {
return file.name.split('.').pop()?.toLowerCase() ?? '';
@@ -77,10 +82,37 @@ export function isSeedanceReferenceAudioFile(file: File) {
file.type === 'audio/mpeg' ||
file.type === 'audio/mp3' ||
file.type === '') &&
hasAllowedExtension(file, SEEDANCE_AUDIO_EXTENSIONS)
hasAllowedExtension(file, SEEDANCE_AUDIO_EXTENSIONS)
);
}
export function isEditorAssetVideoFile(file: File) {
return (
(file.type === 'video/mp4' || file.type === '') &&
hasAllowedExtension(file, EDITOR_ASSET_VIDEO_EXTENSIONS)
);
}
export function isEditorAssetAudioFile(file: File) {
return (
(file.type === 'audio/mpeg' || file.type === 'audio/mp3' || file.type === '') &&
hasAllowedExtension(file, EDITOR_ASSET_AUDIO_EXTENSIONS)
);
}
export function resolveEditorAssetUploadMediaType(file: File) {
if (isEditorAssetVideoFile(file)) {
return 'video' as const;
}
if (isEditorAssetAudioFile(file)) {
return 'audio' as const;
}
if (isImageFile(file)) {
return 'image' as const;
}
return null;
}
export function readFileAsDataUrl(file: File) {
return new Promise<string>((resolve, reject) => {
const reader = new FileReader();
@@ -10,6 +10,7 @@ import {
createGeneratedResultLayer,
createIconSpritesheetResultLayers,
createQuickEditResultLayer,
createVideoResultLayer,
} from './ImageCanvasGenerationLayerModel';
function createGenerated(overrides = {}) {
@@ -267,6 +268,8 @@ describe('ImageCanvasGenerationLayerModel', () => {
const layer = createAudioResultLayer({
generated: {
audioSrc: '/generated-character-drafts/editor-audios/sfx.mp3',
objectKey: 'generated-character-drafts/editor-audios/sfx.mp3',
assetObjectId: 'assetobj-audio-1',
width: 420,
height: 120,
sourceType: 'generated',
@@ -315,10 +318,45 @@ describe('ImageCanvasGenerationLayerModel', () => {
model: 'audio1.0',
provider: 'VectorEngine',
taskId: 'sound-task-1',
objectKey: 'generated-character-drafts/editor-audios/sfx.mp3',
assetObjectId: 'assetobj-audio-1',
durationSeconds: 75,
});
});
it('creates video result layers with OSS asset identity', () => {
const layer = createVideoResultLayer({
generated: {
videoSrc: '/generated-character-drafts/editor-videos/video.mp4',
objectKey: 'generated-character-drafts/editor-videos/video.mp4',
assetObjectId: 'assetobj-video-1',
width: 1280,
height: 720,
sourceType: 'generated',
prompt: '开场动画',
actualPrompt: '开场动画',
model: 'seedance2.0-fast',
provider: 'VectorEngine',
taskId: 'video-task-1',
priceMudPoints: 40,
},
generatedIndex: 15,
title: '生成视频 15',
canvasSize: { width: 900, height: 640 },
viewport: { x: 10, y: 20, scale: 2 },
generationInputs: createGenerationInputs(),
});
expect(layer).toMatchObject({
id: 'layer-video-15',
src: '/generated-character-drafts/editor-videos/video.mp4',
mediaType: 'video',
assetKind: 'video',
objectKey: 'generated-character-drafts/editor-videos/video.mp4',
assetObjectId: 'assetobj-video-1',
});
});
it('creates character animation result layers with action metadata', () => {
const layer = createCharacterAnimationResultLayer({
generated: {
@@ -365,6 +365,8 @@ export function createVideoResultLayer({
model: generated.model,
provider: generated.provider,
taskId: generated.taskId,
objectKey: generated.objectKey,
assetObjectId: generated.assetObjectId,
generationInputs,
};
}
@@ -415,6 +417,8 @@ export function createAudioResultLayer({
model: generated.model,
provider: generated.provider,
taskId: generated.taskId,
objectKey: generated.objectKey,
assetObjectId: generated.assetObjectId,
durationSeconds: generated.durationSeconds ?? undefined,
generationInputs,
};
@@ -18,10 +18,12 @@ type PersistedUploadAsset = {
folderId: string;
label: string;
imageSrc: string;
mediaType?: CanvasMediaType;
width: number;
height: number;
objectKey?: string | null;
assetObjectId?: string | null;
durationSeconds?: number | null;
};
type GenerationReferenceUploadTarget =
| 'character-reference'
@@ -46,6 +48,24 @@ export const UPLOAD_LAYER_FALLBACK_SIZE = {
width: 420,
height: 315,
} as const;
export const UPLOAD_VIDEO_FALLBACK_SIZE = {
width: 560,
height: 315,
} as const;
export const UPLOAD_AUDIO_FALLBACK_SIZE = {
width: 420,
height: 120,
} as const;
export function resolveUploadFallbackSize(mediaType: CanvasMediaType) {
if (mediaType === 'video') {
return UPLOAD_VIDEO_FALLBACK_SIZE;
}
if (mediaType === 'audio') {
return UPLOAD_AUDIO_FALLBACK_SIZE;
}
return UPLOAD_LAYER_FALLBACK_SIZE;
}
export function resolveUploadFolderId({
assetFolders,
@@ -259,21 +279,28 @@ export function createUploadingAssetPlaceholder({
uploadIndex,
fileName,
folderId,
mediaType = 'image',
durationSeconds,
}: {
uploadIndex: number;
fileName?: string;
folderId: string;
mediaType?: CanvasMediaType;
durationSeconds?: number;
}): EditorAsset {
const fallbackSize = resolveUploadFallbackSize(mediaType);
return {
id: `upload-${uploadIndex}`,
label: fileName || '上传图片',
label: fileName || (mediaType === 'audio' ? '上传音频' : mediaType === 'video' ? '上传视频' : '上传图片'),
src: '',
width: UPLOAD_LAYER_FALLBACK_SIZE.width,
height: UPLOAD_LAYER_FALLBACK_SIZE.height,
mediaType,
width: fallbackSize.width,
height: fallbackSize.height,
folderId,
sourceKind: 'uploaded',
sourceType: 'uploaded',
persisted: false,
durationSeconds,
uploadStatus: 'uploading',
uploadProgress: 8,
uploadMessage: '准备上传',
@@ -312,7 +339,7 @@ export function applyUploadAssetReadSuccess({
...asset,
src: imageSrc,
uploadProgress: 42,
uploadMessage: '读取图片',
uploadMessage: '读取素材',
}
: asset,
);
@@ -372,10 +399,12 @@ export function applyPersistedUploadAsset({
folderId: persistedAsset.folderId,
label: persistedAsset.label,
src: persistedAsset.imageSrc,
mediaType: persistedAsset.mediaType ?? asset.mediaType,
width: persistedAsset.width,
height: persistedAsset.height,
objectKey: persistedAsset.objectKey ?? undefined,
assetObjectId: persistedAsset.assetObjectId ?? undefined,
durationSeconds: persistedAsset.durationSeconds ?? asset.durationSeconds,
persisted: true,
uploadStatus: undefined,
uploadProgress: undefined,
@@ -442,8 +471,11 @@ export function bindUploadLayerToPersistedAsset({
? {
...layer,
sourceAssetId: persistedAsset.assetId,
mediaType: persistedAsset.mediaType ?? layer.mediaType,
objectKey: persistedAsset.objectKey ?? layer.objectKey,
assetObjectId: persistedAsset.assetObjectId ?? layer.assetObjectId,
durationSeconds:
persistedAsset.durationSeconds ?? layer.durationSeconds,
}
: layer,
);
@@ -471,6 +503,8 @@ export function createUploadCanvasLayer({
uploadIndex,
fileName,
imageSrc,
mediaType = 'image',
durationSeconds,
canvasPoint,
canvasSize,
viewport,
@@ -478,10 +512,13 @@ export function createUploadCanvasLayer({
uploadIndex: number;
fileName?: string;
imageSrc: string;
mediaType?: CanvasMediaType;
durationSeconds?: number;
canvasPoint?: CanvasPoint;
canvasSize: CanvasSize;
viewport: CanvasViewport;
}): CanvasLayer {
const fallbackSize = resolveUploadFallbackSize(mediaType);
const screenPoint = normalizeUploadScreenPoint({ canvasPoint, canvasSize });
const safeScale = viewport.scale > 0 ? viewport.scale : 1;
const worldCenterX = (screenPoint.x - viewport.x) / safeScale;
@@ -489,17 +526,25 @@ export function createUploadCanvasLayer({
return {
id: `layer-upload-${uploadIndex}`,
resourceId: `local-resource-upload-${uploadIndex}`,
title: fileName || '上传图片',
title: fileName || (mediaType === 'audio' ? '上传音频' : mediaType === 'video' ? '上传视频' : '上传图片'),
src: imageSrc,
x: worldCenterX - UPLOAD_LAYER_FALLBACK_SIZE.width / 2,
y: worldCenterY - UPLOAD_LAYER_FALLBACK_SIZE.height / 2,
width: UPLOAD_LAYER_FALLBACK_SIZE.width,
height: UPLOAD_LAYER_FALLBACK_SIZE.height,
originalWidth: UPLOAD_LAYER_FALLBACK_SIZE.width,
originalHeight: UPLOAD_LAYER_FALLBACK_SIZE.height,
mediaType,
assetKind:
mediaType === 'video'
? 'video'
: mediaType === 'audio'
? 'sound-effect'
: undefined,
x: worldCenterX - fallbackSize.width / 2,
y: worldCenterY - fallbackSize.height / 2,
width: fallbackSize.width,
height: fallbackSize.height,
originalWidth: fallbackSize.width,
originalHeight: fallbackSize.height,
zIndex: uploadIndex + 10,
sourceType: 'uploaded',
sourceAssetId: `upload-${uploadIndex}`,
durationSeconds,
};
}
@@ -518,6 +518,32 @@ function ImageCanvasAudioLayerCard({
);
}
function ImageCanvasVideoLayer({
layer,
mediaTransform,
}: {
layer: CanvasLayer;
mediaTransform?: string;
}) {
const { resolvedUrl } = useResolvedAssetReadUrl(layer.src, {
refreshKey: layer.taskId ?? layer.resourceId,
});
return (
<video
className="image-canvas-editor__layer-video"
src={resolvedUrl ?? layer.src}
controls
playsInline
preload="metadata"
aria-label={`画布视频:${layer.title}`}
style={{ transform: mediaTransform }}
onClick={(event) => event.stopPropagation()}
onPointerDown={(event) => event.stopPropagation()}
/>
);
}
export function ImageCanvasWorldView({
viewport,
snapGuide,
@@ -630,16 +656,9 @@ export function ImageCanvasWorldView({
isHovered={isHovered}
/>
) : layer.mediaType === 'video' ? (
<video
className="image-canvas-editor__layer-video"
src={layer.src}
controls
playsInline
preload="metadata"
aria-label={`画布视频:${layer.title}`}
style={{ transform: mediaTransform }}
onClick={(event) => event.stopPropagation()}
onPointerDown={(event) => event.stopPropagation()}
<ImageCanvasVideoLayer
layer={layer}
mediaTransform={mediaTransform}
/>
) : (
<img
@@ -648,23 +667,25 @@ export function ImageCanvasWorldView({
style={{ transform: mediaTransform }}
/>
)}
<span
role="button"
tabIndex={0}
aria-label={`素材标签:${kindLabel}`}
className={`image-canvas-editor__kind-badge image-canvas-editor__kind-badge--${kindTagMeta.className} image-canvas-editor__kind-badge--beside-info image-canvas-editor__kind-badge--editable`}
style={inverseScaleStyle}
onPointerDown={(event) => event.stopPropagation()}
onClick={(event) => {
event.stopPropagation();
setOpenLayerKindMenuId((currentId) =>
currentId === layer.id ? null : layer.id,
);
}}
onKeyDown={handleLayerTagKeyboardActivation}
>
{kindLabel}
</span>
{kindLabel ? (
<span
role="button"
tabIndex={0}
aria-label={`素材标签:${kindLabel}`}
className={`image-canvas-editor__kind-badge image-canvas-editor__kind-badge--${kindTagMeta.className} image-canvas-editor__kind-badge--beside-info image-canvas-editor__kind-badge--editable`}
style={inverseScaleStyle}
onPointerDown={(event) => event.stopPropagation()}
onClick={(event) => {
event.stopPropagation();
setOpenLayerKindMenuId((currentId) =>
currentId === layer.id ? null : layer.id,
);
}}
onKeyDown={handleLayerTagKeyboardActivation}
>
{kindLabel}
</span>
) : null}
<PlatformIconButton
asChild="spanButton"
variant="darkMini"
@@ -147,11 +147,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({
const addGeneratedLayersToCanvas = useCallback(
(nextLayers: CanvasLayer[]) => {
appendCanvasLayersWithResources(nextLayers);
nextLayers
.filter(
(layer) => layer.mediaType !== 'video' && layer.mediaType !== 'audio',
)
.forEach((layer) => persistGeneratedAsset?.(layer));
nextLayers.forEach((layer) => persistGeneratedAsset?.(layer));
},
[appendCanvasLayersWithResources, persistGeneratedAsset],
);

Some files were not shown because too many files have changed in this diff Show More