合并编辑器素材修复分支

合并 codex/240e-editor-asset-fixes 到 codex/editor-asset-library

保留视频生成资源与素材返回

保留视频缩略图与生成输入元数据
This commit is contained in:
2026-06-29 11:36:49 +08:00
24 changed files with 748 additions and 80 deletions
@@ -110,7 +110,7 @@
- 快速编辑 / 重绘站内 public 示例图、历史 generated 图或 OSS generated 图时,前端先读取成 `data:image/*;base64,...` 再提交,后端不得再收到 `/creation-type-references/*``/generated-*` 或 OSS URL 作为 `referenceImageSrcs/sourceImageSrc`
- 快速编辑的额外参考图既进入输入快照,也作为 `/api/editor/images/edits``referenceImageSrcs` 提交给后端;原图仍只作为 `sourceImageSrc`,不占用额外参考图额度。
- 素材文件夹可以新建、折叠、重命名和删除;删除普通文件夹后,其素材移动到“项目素材”。普通上传默认落入“上传素材”文件夹;素材库缺少该文件夹时,前端在首次普通上传前创建一次并复用,拖到指定文件夹或点击指定文件夹上传时仍进入目标文件夹。
- 上传按钮和拖拽上传都支持多文件;底部工具栏的上传入口选择文件后直接进入“上传素材”并在当前画布视口中心创建画布图层,素材栏文件夹内的上传入口只写入对应素材文件夹、不自动入画布;拖到文件夹或该文件夹内素材时进入目标文件夹;拖到画布时进入“上传素材”并在投放点创建画布图层。上传图片必须在创建占位素材、画布图层和账号级素材记录前先读取原图 Resolution,图层宽高、`originalWidth/originalHeight` 和素材库 `width/height` 都使用图片本身尺寸;仅在无法解析尺寸时才使用上传兜底尺寸。
- 上传按钮和拖拽上传都支持多文件;底部工具栏的上传入口选择文件后直接进入“上传素材”并在当前画布视口中心创建画布图层,素材栏文件夹内的上传入口只写入对应素材文件夹、不自动入画布;拖到文件夹或该文件夹内素材时进入目标文件夹;拖到画布时进入“上传素材”并在投放点创建画布图层。上传图片必须在创建占位素材、画布图层和账号级素材记录前先读取原图 Resolution,图层宽高、`originalWidth/originalHeight` 和素材库 `width/height` 都使用图片本身尺寸;上传视频同样在创建素材和图层前读取视频 metadata 宽高,保证单层下载或 ZIP 导出的真实视频文件重新导入后仍按文件自身尺寸入画布;仅在无法解析尺寸时才使用对应媒体兜底尺寸。
- 音频 / 视频素材卡和画布媒体图层必须提供稳定的非文字视觉预览:优先使用 `thumbnailSrc` / 视频 `poster`,没有真实首帧或音频封面时使用由媒体类型、素材名和地址派生的确定性视觉底图。视频图层使用原生 `<video controls preload="metadata" playsInline>` 播放,外层图层仍承接选择和拖拽语义;音频图层播放前继续通过 `/api/assets/read-url` 换签。画布素材导出按 `mediaType` 保留真实媒体格式:图片进入 `images/`,音频 / 视频进入 `media/`,文件扩展名从响应 MIME、`objectKey` 或源 URL 推断,不得把音频 / 视频导出成 PNG。
- 画布素材 ZIP 的 `metadata.json` 只保存前端信息弹窗和导出文件列表可见的展示快照:项目标题、导出时间、图层标题、文件路径、类型、生成输入、模型显示名、Task 短 ID、Object 显示值、Resolution / 时长和导出错误。导出的生成输入只保留用户实际填写或选择的内容;系统默认兜底提示词、固定工作流提示词、内置图标描述、UI 提取素材固定提示词等内置提示词即使存在于历史 `generationInputs`,也不得写入导出元数据。不得把 `projectId``layerId``resourceId``sourceAssetId``sourceResourceId`、原始 `prompt` / `actualPrompt` / `provider` 或画布坐标、锁定、隐藏等布局状态写入导出元数据;Object 字段仅沿用信息弹窗当前可见值。
- `生成角色动作` 的完成结果同样按视频素材处理:图层主 `src` 使用后端返回的 `previewVideoPath``mediaType` 固定为 `video``assetKind` 固定为 `character-animation`,首个透明帧只写入 `thumbnailSrc` 供 poster 和后续动作源使用。下载和 ZIP 导出必须因此得到 mp4/webm/mov 等真实视频文件,不能回退为首帧 PNG;后端抽帧、绿幕去背和帧素材落盘流程仍完整保留。
@@ -61,9 +61,9 @@ use crate::{
editor_generation_source_entity_id, enqueue_editor_generation_job,
},
editor_project::{
EditorCanvasGeneratedLayerInput, PersistEditorGeneratedAssetInput,
EditorCanvasGeneratedLayerInput, PersistEditorGeneratedAssetRequest,
build_editor_canvas_generated_layer_item, complete_editor_canvas_generation_with_items,
persist_editor_generated_asset,
persist_editor_generated_media_asset,
},
http_error::AppError,
platform_errors::map_oss_error,
@@ -830,7 +830,10 @@ pub(crate) async fn generate_editor_video_for_owner(
let canvas_completion = payload.canvas_completion.clone();
let generation_inputs = payload.generation_inputs.clone();
let source_resource_id = payload.source_resource_id.clone();
let asset_kind = payload.asset_kind.clone();
let asset_kind = payload
.asset_kind
.clone()
.unwrap_or_else(|| "video".to_string());
let asset_folder_id = payload.asset_folder_id.clone();
let asset_label = payload.asset_label.clone();
@@ -869,10 +872,9 @@ pub(crate) async fn generate_editor_video_for_owner(
.await
.map_err(|error| editor_video_error_response(&request_context, error))?;
let image_src = generated.preview_video_path.clone();
let effective_asset_kind = asset_kind.clone().unwrap_or_else(|| "video".to_string());
let generated_asset = persist_editor_generated_asset(
let (resource, asset) = persist_editor_generated_media_asset(
&state,
PersistEditorGeneratedAssetInput {
PersistEditorGeneratedAssetRequest {
project_id: project_id.clone(),
owner_user_id: owner_user_id.clone(),
folder_id: asset_folder_id,
@@ -893,7 +895,7 @@ pub(crate) async fn generate_editor_video_for_owner(
provider: "VectorEngine".to_string(),
task_id: task_id.clone(),
source_resource_id: source_resource_id.clone(),
asset_kind: Some(effective_asset_kind.clone()),
asset_kind: Some(asset_kind.clone()),
generation_inputs: generation_inputs.clone(),
thumbnail_src: generated.thumbnail_src.clone(),
},
@@ -902,15 +904,15 @@ pub(crate) async fn generate_editor_video_for_owner(
.map_err(|error| editor_video_error_response(&request_context, error))?;
let completed_project = if let Some(completion) = canvas_completion.as_ref() {
let layer_id = format!("layer-editor-video-{task_id}");
let resource_id = resource
.as_ref()
.map(|resource| resource.resource_id.clone())
.unwrap_or_else(|| format!("local-resource-editor-video-{task_id}"));
let item = build_editor_canvas_generated_layer_item(
completion,
EditorCanvasGeneratedLayerInput {
layer_id: layer_id.clone(),
resource_id: generated_asset
.resource
.as_ref()
.map(|resource| resource.resource_id.clone())
.unwrap_or_else(|| format!("local-resource-editor-video-{task_id}")),
resource_id,
title: completion.title.trim().to_string(),
src: image_src.clone(),
media_type: Some("video".to_string()),
@@ -929,7 +931,7 @@ pub(crate) async fn generate_editor_video_for_owner(
asset_object_id: generated.asset_object_id.clone(),
duration_seconds: Some(normalized.duration_seconds),
source_resource_id,
asset_kind: Some(effective_asset_kind.clone()),
asset_kind: Some(asset_kind.clone()),
generation_inputs,
image_sequence_frames: None,
preview_video_path: None,
@@ -950,9 +952,8 @@ pub(crate) async fn generate_editor_video_for_owner(
};
let completed_project =
completed_project.and_then(|project| serde_json::to_value(project).ok());
let generated_asset = generated_asset
.asset
.and_then(|asset| serde_json::to_value(asset).ok());
let resource = resource.and_then(|resource| serde_json::to_value(resource).ok());
let asset = asset.and_then(|asset| serde_json::to_value(asset).ok());
Ok(json_success_body(
Some(&request_context),
@@ -974,7 +975,8 @@ pub(crate) async fn generate_editor_video_for_owner(
resolution: normalized.resolution,
price_mud_points: normalized.price_mud_points,
project: completed_project,
asset: generated_asset,
resource,
asset,
queue_state: None,
},
))
+231 -16
View File
@@ -375,6 +375,27 @@ pub(crate) struct PersistEditorGeneratedAssetInput {
pub(crate) thumbnail_src: Option<String>,
}
pub(crate) struct PersistEditorGeneratedAssetRequest {
pub(crate) project_id: Option<String>,
pub(crate) owner_user_id: String,
pub(crate) folder_id: Option<String>,
pub(crate) label: String,
pub(crate) image_src: String,
pub(crate) object_key: Option<String>,
pub(crate) asset_object_id: Option<String>,
pub(crate) width: u32,
pub(crate) height: u32,
pub(crate) prompt: String,
pub(crate) actual_prompt: Option<String>,
pub(crate) model: String,
pub(crate) provider: String,
pub(crate) task_id: String,
pub(crate) source_resource_id: Option<String>,
pub(crate) asset_kind: Option<String>,
pub(crate) generation_inputs: Option<Value>,
pub(crate) thumbnail_src: Option<String>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct EditorGeneratedImageStorageProfile {
asset_kind: &'static str,
@@ -1813,7 +1834,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner(
error.with_details(json!({
"provider": "editor-icon-spritesheet",
"field": "referenceImageSrc",
"message": "图标素材规范必须是图片 Data URLobjectKey。",
"message": "图标素材规范必须是图片 Data URLobjectKey、资源或素材引用",
}))
})?,
);
@@ -1827,7 +1848,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner(
error.with_details(json!({
"provider": "editor-icon-spritesheet",
"field": "referenceImageSrcs",
"message": "图标素材参考图必须是图片 Data URLobjectKey。",
"message": "图标素材参考图必须是图片 Data URLobjectKey、资源或素材引用",
}))
})?,
);
@@ -2042,7 +2063,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
error.with_details(json!({
"provider": "editor-ui-design-asset-extraction",
"field": "sourceImageSrc",
"message": "UI设计图必须是图片 Data URLobjectKey。",
"message": "UI设计图必须是图片 Data URLobjectKey、资源或素材引用",
}))
})?;
let mut reference_images = vec![reference_image];
@@ -2057,7 +2078,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
error.with_details(json!({
"provider": "editor-ui-design-asset-extraction",
"field": "referenceImageSrcs",
"message": "UI素材参考图必须是图片 Data URLobjectKey。",
"message": "UI素材参考图必须是图片 Data URLobjectKey、资源或素材引用",
}))
})?,
);
@@ -2865,6 +2886,43 @@ pub(crate) async fn persist_editor_generated_asset(
Ok(EditorGeneratedAssetRecord { resource, asset })
}
pub(crate) async fn persist_editor_generated_media_asset(
state: &AppState,
input: PersistEditorGeneratedAssetRequest,
) -> Result<
(
Option<EditorProjectResourcePayload>,
Option<EditorAssetPayload>,
),
AppError,
> {
let persisted = persist_editor_generated_asset(
state,
PersistEditorGeneratedAssetInput {
project_id: input.project_id,
owner_user_id: input.owner_user_id,
folder_id: input.folder_id,
label: input.label,
image_src: input.image_src,
object_key: input.object_key,
asset_object_id: input.asset_object_id,
width: input.width,
height: input.height,
prompt: input.prompt,
actual_prompt: input.actual_prompt,
model: input.model,
provider: input.provider,
task_id: input.task_id,
source_resource_id: input.source_resource_id,
asset_kind: input.asset_kind,
generation_inputs: input.generation_inputs,
thumbnail_src: input.thumbnail_src,
},
)
.await?;
Ok((persisted.resource, persisted.asset))
}
pub(crate) async fn complete_editor_canvas_generation(
state: &AppState,
owner_user_id: &str,
@@ -3839,7 +3897,7 @@ pub(crate) async fn parse_editor_reference_image(
if trimmed.starts_with("data:") {
return parse_editor_reference_image_data_url(trimmed);
}
let object_key = normalize_editor_reference_object_key(trimmed)?;
let object_key = resolve_editor_reference_object_key(state, owner_user_id, trimmed).await?;
ensure_editor_reference_object_key_owned(state, owner_user_id, object_key.as_str()).await?;
read_editor_reference_image_object(state, object_key.as_str()).await
}
@@ -3862,13 +3920,93 @@ fn normalize_editor_reference_object_key(source: &str) -> Result<String, AppErro
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
"provider": "editor-reference-image",
"field": "sourceImageSrc",
"message": "图片参考图必须是图片 Data URL已登记的 generated objectKey。",
"message": "图片参考图必须是图片 Data URL已登记的 generated objectKey、项目资源或素材引用",
})),
);
}
Ok(object_key)
}
async fn resolve_editor_reference_object_key(
state: &AppState,
owner_user_id: &str,
source: &str,
) -> Result<String, AppError> {
match normalize_editor_reference_object_key(source) {
Ok(object_key) => Ok(object_key),
Err(error) => {
find_editor_reference_object_key_by_registered_id(state, owner_user_id, source.trim())
.await?
.ok_or(error)
}
}
}
async fn find_editor_reference_object_key_by_registered_id(
state: &AppState,
owner_user_id: &str,
source: &str,
) -> Result<Option<String>, AppError> {
if source.is_empty() {
return Ok(None);
}
let projects = state
.spacetime_client()
.list_editor_projects(owner_user_id.to_string())
.await
.map_err(map_editor_project_error)?;
if let Some(object_key) = projects
.iter()
.flat_map(|project| project.resources.iter())
.find_map(|resource| editor_reference_resource_object_key(resource, source))
{
return Ok(Some(object_key));
}
let library = state
.spacetime_client()
.get_editor_asset_library(owner_user_id.to_string(), current_utc_micros())
.await
.map_err(map_editor_project_error)?;
Ok(library
.assets
.iter()
.find_map(|asset| editor_reference_asset_object_key(asset, source)))
}
fn editor_reference_resource_object_key(
resource: &EditorProjectResourceRecord,
source: &str,
) -> Option<String> {
(resource.resource_id.trim() == source)
.then(|| {
normalize_editor_record_object_key(
resource.object_key.as_deref(),
resource.image_src.as_str(),
)
})
.flatten()
}
fn editor_reference_asset_object_key(asset: &EditorAssetRecord, source: &str) -> Option<String> {
(asset.asset_id.trim() == source)
.then(|| {
normalize_editor_record_object_key(
asset.object_key.as_deref(),
asset.image_src.as_str(),
)
})
.flatten()
}
fn normalize_editor_record_object_key(object_key: Option<&str>, image_src: &str) -> Option<String> {
object_key
.map(str::trim)
.map(|value| value.trim_start_matches('/'))
.filter(|value| !value.is_empty())
.map(ToOwned::to_owned)
.or_else(|| normalize_editor_reference_object_key(image_src).ok())
}
async fn ensure_editor_reference_object_key_owned(
state: &AppState,
owner_user_id: &str,
@@ -3885,7 +4023,11 @@ async fn ensure_editor_reference_object_key_owned(
.map_err(map_editor_project_error)?;
if projects.iter().any(|project| {
project.resources.iter().any(|resource| {
editor_record_object_key_matches(resource.object_key.as_deref(), object_key)
editor_record_object_key_matches(
resource.object_key.as_deref(),
resource.image_src.as_str(),
object_key,
)
})
}) {
return Ok(());
@@ -3895,11 +4037,13 @@ async fn ensure_editor_reference_object_key_owned(
.get_editor_asset_library(owner_user_id.to_string(), current_utc_micros())
.await
.map_err(map_editor_project_error)?;
if library
.assets
.iter()
.any(|asset| editor_record_object_key_matches(asset.object_key.as_deref(), object_key))
{
if library.assets.iter().any(|asset| {
editor_record_object_key_matches(
asset.object_key.as_deref(),
asset.image_src.as_str(),
object_key,
)
}) {
return Ok(());
}
if let Some(asset_object) = state
@@ -3937,10 +4081,12 @@ fn validate_editor_reference_asset_object(
}
}
fn editor_record_object_key_matches(record_object_key: Option<&str>, object_key: &str) -> bool {
record_object_key
.map(str::trim)
.map(|value| value.trim_start_matches('/'))
fn editor_record_object_key_matches(
record_object_key: Option<&str>,
image_src: &str,
object_key: &str,
) -> bool {
normalize_editor_record_object_key(record_object_key, image_src)
.is_some_and(|value| value == object_key)
}
@@ -5410,6 +5556,73 @@ mod tests {
assert_eq!(parsed.bytes, vec![0, 0, 0]);
}
#[test]
fn editor_reference_resource_id_resolves_registered_object_key() {
let resource = EditorProjectResourceRecord {
resource_id: "editor-resource-icon-spec".to_string(),
project_id: "project-1".to_string(),
owner_user_id: "user-1".to_string(),
asset_object_id: Some("asset-object-1".to_string()),
image_src: "/generated-character-drafts/editor/spec.png".to_string(),
object_key: None,
width: 512,
height: 512,
source_type: "generated".to_string(),
prompt: None,
actual_prompt: None,
model: None,
provider: None,
task_id: None,
source_resource_id: None,
asset_kind: Some("icon-spec".to_string()),
generation_inputs: None,
public_showcase_enabled: true,
created_at: "2026-06-23T00:00:00.000Z".to_string(),
updated_at: "2026-06-23T00:00:00.000Z".to_string(),
};
assert_eq!(
editor_reference_resource_object_key(&resource, "editor-resource-icon-spec"),
Some("generated-character-drafts/editor/spec.png".to_string())
);
assert_eq!(
editor_reference_resource_object_key(&resource, "missing"),
None
);
}
#[test]
fn editor_reference_asset_id_resolves_registered_object_key() {
let asset = EditorAssetRecord {
asset_id: "editor-asset-icon-spec".to_string(),
folder_id: "project".to_string(),
label: "图标规范".to_string(),
asset_object_id: Some("asset-object-1".to_string()),
image_src: "/generated-character-drafts/editor/spec.png".to_string(),
object_key: None,
width: 512,
height: 512,
source_type: "generated".to_string(),
prompt: None,
actual_prompt: None,
model: None,
provider: None,
task_id: None,
asset_kind: Some("icon-spec".to_string()),
generation_inputs: None,
source_resource_id: None,
public_showcase_enabled: Some(true),
created_at: "2026-06-23T00:00:00.000Z".to_string(),
updated_at: "2026-06-23T00:00:00.000Z".to_string(),
};
assert_eq!(
editor_reference_asset_object_key(&asset, "editor-asset-icon-spec"),
Some("generated-character-drafts/editor/spec.png".to_string())
);
assert_eq!(editor_reference_asset_object_key(&asset, "missing"), None);
}
#[test]
fn editor_reference_image_sources_are_trimmed_and_limited() {
let sources = vec![
@@ -5442,10 +5655,12 @@ mod tests {
fn editor_reference_object_key_matches_registered_media() {
assert!(editor_record_object_key_matches(
Some("/generated-character-drafts/editor/ref.png"),
"",
"generated-character-drafts/editor/ref.png",
));
assert!(!editor_record_object_key_matches(
Some("generated-character-drafts/editor/other.png"),
"",
"generated-character-drafts/editor/ref.png",
));
}
@@ -818,18 +818,32 @@ async fn response_error_message(response: axum::response::Response) -> String {
return format!("外部生成任务失败:{status}");
}
if let Ok(body_json) = serde_json::from_str::<serde_json::Value>(&body_text)
&& let Some(message) = body_json
.get("error")
.and_then(|error| error.get("message"))
.and_then(serde_json::Value::as_str)
.map(str::trim)
.filter(|message| !message.is_empty())
&& let Some(message) = extract_api_error_display_message(&body_json)
{
return message.to_string();
return message;
}
body_text
}
fn extract_api_error_display_message(body_json: &serde_json::Value) -> Option<String> {
let error = body_json.get("error")?;
error
.get("details")
.and_then(|details| {
read_trimmed_json_string(details.get("reason"))
.or_else(|| read_trimmed_json_string(details.get("message")))
})
.or_else(|| read_trimmed_json_string(error.get("message")))
}
fn read_trimmed_json_string(value: Option<&serde_json::Value>) -> Option<String> {
value
.and_then(serde_json::Value::as_str)
.map(str::trim)
.filter(|message| !message.is_empty())
.map(ToOwned::to_owned)
}
async fn fail_queue_job_after_worker_error(
state: &AppState,
worker_id: &str,
@@ -1009,6 +1023,34 @@ mod tests {
assert!(error.contains("缺少 lease token"));
}
#[tokio::test]
async fn worker_response_error_message_prefers_api_error_details() {
let response = axum::http::Response::builder()
.status(axum::http::StatusCode::BAD_GATEWAY)
.body(axum::body::Body::from(
json!({
"ok": false,
"data": null,
"error": {
"code": "UPSTREAM_ERROR",
"message": "上游服务请求失败",
"details": {
"provider": "vector-engine",
"reason": "VECTOR_ENGINE_API_KEY 未配置",
"message": "提交编辑器音效任务失败:missing field sound"
}
},
"meta": {}
})
.to_string(),
))
.expect("response should build");
let message = response_error_message(response).await;
assert_eq!(message, "VECTOR_ENGINE_API_KEY 未配置");
}
fn external_generation_job_record_fixture(
lease_token: Option<&str>,
) -> ExternalGenerationJobRecord {
@@ -423,6 +423,8 @@ pub struct EditorVideoGenerateResponse {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub project: Option<Value>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub resource: Option<Value>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub asset: Option<Value>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub queue_state: Option<ExternalGenerationJobStatusRecord>,
@@ -1225,6 +1227,7 @@ mod tests {
resolution: "480p".to_string(),
price_mud_points: 100,
project: None,
resource: None,
asset: None,
queue_state: None,
})
@@ -2,11 +2,17 @@
import { fireEvent, render, screen } from '@testing-library/react';
import type { ReactNode } from 'react';
import { describe, expect, it, vi } from 'vitest';
import { beforeEach, describe, expect, it, vi } from 'vitest';
import type { EditorAsset } from './ImageCanvasEditorTypes';
import { ImageCanvasAssetRowView } from './ImageCanvasAssetRowView';
const useResolvedAssetReadUrlMock = vi.hoisted(() => vi.fn());
vi.mock('../../hooks/useResolvedAssetReadUrl', () => ({
useResolvedAssetReadUrl: useResolvedAssetReadUrlMock,
}));
vi.mock('../common/PlatformMediaFrame', () => ({
PlatformMediaFrame: ({
src,
@@ -32,6 +38,18 @@ vi.mock('../common/PlatformMediaFrame', () => ({
),
}));
beforeEach(() => {
useResolvedAssetReadUrlMock.mockImplementation(
(source: string | null | undefined, options?: { objectKey?: string | null }) => ({
resolvedUrl: options?.objectKey
? `https://signed.example.com/${options.objectKey}`
: (source ?? ''),
isResolving: false,
shouldResolve: Boolean(options?.objectKey),
}),
);
});
function createAsset(overrides: Partial<EditorAsset> = {}): EditorAsset {
return {
id: 'asset-1',
@@ -187,16 +205,19 @@ describe('ImageCanvasAssetRowView', () => {
asset: createAsset({
label: '生成视频.mp4',
src: '/generated-character-drafts/editor/asset-library/video/demo.mp4',
objectKey: 'generated-character-drafts/editor/asset-library/video/demo.mp4',
mediaType: 'video',
}),
});
expect(screen.getByText('视频')).toBeTruthy();
expect(screen.queryByRole('img', { name: '素材:生成视频.mp4' })).toBeNull();
expect(
document.querySelector('.image-canvas-editor__media-preview--video')
?.className,
).toContain('image-canvas-editor__media-preview--variant-');
const video = screen.getByLabelText(
'视频素材预览:生成视频.mp4',
) as HTMLVideoElement;
expect(video.getAttribute('src')).toBe(
'https://signed.example.com/generated-character-drafts/editor/asset-library/video/demo.mp4',
);
});
it('passes objectKey to image thumbnails in the asset library', () => {
@@ -2,7 +2,9 @@ import { Check, Music2, Pencil, Video, X } from 'lucide-react';
import { useCallback, useEffect, useRef, useState } from 'react';
import type { Dispatch, SetStateAction } from 'react';
import { useResolvedAssetReadUrl } from '../../hooks/useResolvedAssetReadUrl';
import { PlatformTextField } from '../common/PlatformTextField';
import type { UploadFilesOptions } from './ImageCanvasAssetLibraryPanelView';
import {
EditorIconButton,
SidebarMediaItem,
@@ -21,7 +23,6 @@ import {
getCanvasMediaPreviewClassName,
getCanvasMediaPreviewMarker,
} from './ImageCanvasMediaModel';
import type { UploadFilesOptions } from './ImageCanvasAssetLibraryPanelView';
export type ImageCanvasAssetRowViewProps = {
asset: EditorAsset;
@@ -116,6 +117,25 @@ export function ImageCanvasAssetRowView({
const mediaPreviewMarker = getCanvasMediaPreviewMarker(asset);
const mediaPreviewClassName =
getCanvasMediaPreviewClassName(mediaPreviewMarker);
const { resolvedUrl: resolvedVideoPreviewUrl } = useResolvedAssetReadUrl(
asset.mediaType === 'video' ? asset.src : '',
{
enabled: asset.mediaType === 'video' && !asset.thumbnailSrc?.trim(),
objectKey: asset.mediaType === 'video' ? asset.objectKey : undefined,
refreshKey: asset.id,
},
);
const videoPreview =
asset.mediaType === 'video' && !asset.thumbnailSrc?.trim() ? (
<video
className="image-canvas-editor__asset-video-preview"
src={resolvedVideoPreviewUrl || undefined}
muted
playsInline
preload="metadata"
aria-label={`视频素材预览:${asset.label}`}
/>
) : undefined;
const mediaDetail =
asset.mediaType === 'audio'
? '音频'
@@ -251,6 +271,7 @@ export function ImageCanvasAssetRowView({
: (asset.thumbnailSrc ?? '')
}
objectKey={asset.mediaType === 'image' ? asset.objectKey : undefined}
previewNode={videoPreview}
titleNode={
isUploadingAsset || isFailedUpload ? <span>{asset.label}</span> : titleNode
}
@@ -94,7 +94,6 @@ export function ImageCanvasContextMenusView({
<button
type="button"
role="menuitem"
disabled={!canvasClipboard?.layers.length}
onClick={() => onPasteCanvasClipboard(contextMenu.canvasPoint)}
>
@@ -149,7 +148,6 @@ export function ImageCanvasContextMenusView({
<button
type="button"
role="menuitem"
disabled={!canvasClipboard?.layers.length}
onClick={() => onPasteCanvasClipboard(contextMenu.canvasPoint)}
>
@@ -1142,6 +1142,50 @@ describe('ImageCanvasEditorView asset library integration', () => {
).toBe(true);
});
it('pastes a system clipboard image from the canvas context menu', async () => {
const originalClipboard = navigator.clipboard;
const clipboardBlob = new Blob(['image'], { type: 'image/png' });
Object.defineProperty(navigator, 'clipboard', {
configurable: true,
value: {
read: vi.fn(async () => [
{
types: ['image/png'],
getType: vi.fn(async () => clipboardBlob),
},
]),
},
});
try {
render(<ImageCanvasEditorView />);
await waitFor(() => {
expect(loadOrCreateRecentEditorProjectMock).toHaveBeenCalled();
});
fireEvent.contextMenu(screen.getByLabelText('画布工作区'), {
clientX: 360,
clientY: 240,
});
fireEvent.click(screen.getByRole('menuitem', { name: '粘贴' }));
await waitFor(() => {
expect(screen.getByAltText('画布图片:剪贴板图片.png')).toBeTruthy();
});
expect(uploadEditorMediaAssetFileMock).toHaveBeenCalledWith(
expect.objectContaining({
name: '剪贴板图片.png',
type: 'image/png',
}),
'image',
);
} finally {
Object.defineProperty(navigator, 'clipboard', {
configurable: true,
value: originalClipboard,
});
}
});
it('keeps canvas clipboard paste ahead of system clipboard images', async () => {
render(<ImageCanvasEditorView />);
await waitFor(() => {
@@ -692,6 +692,7 @@ export function mapAssetLibrarySnapshot(
const mediaType = inferEditorAssetMediaType(
asset.imageSrc,
asset.objectKey ?? undefined,
asset.assetKind,
);
return {
id: asset.assetId,
@@ -725,7 +726,14 @@ export function mapAssetLibrarySnapshot(
export function inferEditorAssetMediaType(
imageSrc: string,
objectKey?: string | null,
assetKind?: string | null,
): CanvasMediaType {
if (assetKind === 'video') {
return 'video';
}
if (assetKind === 'sound-effect' || assetKind === 'background-music') {
return 'audio';
}
const source = `${objectKey ?? ''} ${imageSrc}`.toLowerCase();
const sourceWithoutQuery = source.split(/[?#]/u)[0] ?? source;
if (sourceWithoutQuery.includes('.mp4')) {
@@ -70,6 +70,7 @@ export type SidebarMediaItemProps = {
primaryClassName?: string;
actions?: ReactNode;
titleNode?: ReactNode;
previewNode?: ReactNode;
previewOverlay?: ReactNode;
footerNode?: ReactNode;
draggable?: boolean;
@@ -97,6 +98,7 @@ export function SidebarMediaItem({
primaryClassName,
actions,
titleNode,
previewNode,
previewOverlay,
footerNode,
draggable,
@@ -129,16 +131,18 @@ export function SidebarMediaItem({
onDragStart={onDragStart}
onDragEnd={onDragEnd}
>
<PlatformMediaFrame
src={imageSrc}
objectKey={objectKey}
alt={imageAlt}
fallbackLabel={title}
aspect="square"
surface="none"
className={thumbnailClassName}
previewOverlay={previewOverlay}
/>
{previewNode ?? (
<PlatformMediaFrame
src={imageSrc}
objectKey={objectKey}
alt={imageAlt}
fallbackLabel={title}
aspect="square"
surface="none"
className={thumbnailClassName}
previewOverlay={previewOverlay}
/>
)}
</button>
<div className={metaClassName}>
{titleNode ?? <span>{title}</span>}
@@ -842,7 +842,7 @@ describe('ImageCanvasEditorView', () => {
expect(contextMenuEvent.defaultPrevented).toBe(true);
});
it('shows the blank canvas context menu with paste disabled, zoom, and fit all', () => {
it('shows the blank canvas context menu with paste, zoom, and fit all', () => {
render(<ImageCanvasEditorView />);
const viewport = screen.getByLabelText('画布工作区');
@@ -858,7 +858,7 @@ describe('ImageCanvasEditorView', () => {
name: '粘贴',
}) as HTMLButtonElement
).disabled,
).toBe(true);
).toBe(false);
expect(within(menu).getByRole('menuitem', { name: '放大' })).toBeTruthy();
expect(
within(menu).getByRole('menuitem', { name: '显示画布所有元素' }),
@@ -114,6 +114,28 @@ function getClipboardImageFiles(clipboardData: DataTransfer | null) {
.filter((file): file is File => file !== null && isImageFile(file));
}
async function readSystemClipboardImageFiles() {
const read = globalThis.navigator?.clipboard?.read;
if (!read) {
return [];
}
const items = await read();
const imageFiles: File[] = [];
for (const item of items) {
const imageType = item.types.find((type) => type.startsWith('image/'));
if (!imageType) {
continue;
}
const blob = await item.getType(imageType);
imageFiles.push(
new File([blob], `.${imageType.split('/')[1] ?? 'png'}`, {
type: imageType,
}),
);
}
return imageFiles;
}
export function ImageCanvasEditorView() {
const authUi = useAuthUi();
const [, setGenerationPricingVersion] = useState(0);
@@ -1279,6 +1301,24 @@ export function ImageCanvasEditorView() {
setShiftPressed,
});
const pasteCanvasOrSystemClipboard = useCallback(
(canvasPoint?: { x: number; y: number }) => {
if (pasteCanvasClipboard(canvasPoint)) {
return;
}
void readSystemClipboardImageFiles()
.then((imageFiles) => {
if (imageFiles.length) {
addUploadedFiles(imageFiles, { addToCanvas: true, canvasPoint });
}
})
.catch(() => {
// 系统剪贴板读取受浏览器权限限制,失败时保留原生粘贴路径。
});
},
[addUploadedFiles, pasteCanvasClipboard],
);
useEffect(() => {
const handleClipboardPaste = (event: ClipboardEvent) => {
if (isEditablePasteTarget(event.target)) {
@@ -1527,7 +1567,7 @@ export function ImageCanvasEditorView() {
onQuickEditSelectionPointerEnd: endQuickEditSelectionPointer,
onOpenCharacterAnimationPanel: openCharacterAnimationPanel,
onDownloadLayer: exportLayerImage,
onPasteCanvasClipboard: pasteCanvasClipboard,
onPasteCanvasClipboard: pasteCanvasOrSystemClipboard,
onCopyContextLayers: copyContextLayers,
onDuplicateContextLayers: duplicateContextLayers,
onMoveContextLayers: moveContextLayers,
@@ -426,9 +426,25 @@ export function probeMediaDurationSeconds(
file: File,
mediaType: 'video' | 'audio',
) {
return new Promise<number>((resolve, reject) => {
if (typeof document === 'undefined' || typeof URL === 'undefined') {
resolve(Number.NaN);
return probeMediaFileMetadata(file, mediaType).then(
(metadata) => metadata.durationSeconds,
);
}
export function probeMediaFileMetadata(
file: File,
mediaType: 'video' | 'audio',
) {
return new Promise<{
durationSeconds: number;
dimensions: ImageFileDimensions | null;
}>((resolve, reject) => {
if (
typeof document === 'undefined' ||
typeof URL === 'undefined' ||
typeof URL.createObjectURL !== 'function'
) {
resolve({ durationSeconds: Number.NaN, dimensions: null });
return;
}
const element =
@@ -443,13 +459,20 @@ export function probeMediaDurationSeconds(
};
element.preload = 'metadata';
element.onloadedmetadata = () => {
const duration = element.duration;
const dimensions =
mediaType === 'video'
? normalizeImageFileDimensions({
width: (element as HTMLVideoElement).videoWidth,
height: (element as HTMLVideoElement).videoHeight,
})
: null;
const durationSeconds = element.duration;
cleanup();
resolve(duration);
resolve({ durationSeconds, dimensions });
};
element.onerror = () => {
cleanup();
reject(new Error('读取媒体时长失败'));
reject(new Error('读取媒体元数据失败'));
};
element.src = objectUrl;
});
@@ -415,14 +415,16 @@ export function createVideoResultLayer({
frame && frame.height > 0
? frame.y + frame.height / 2 - height / 2
: undefined;
const resource = generated.resource;
const asset = generated.asset;
return {
id: `layer-video-${generatedIndex}`,
resourceId: `local-resource-video-${generatedIndex}`,
resourceId: resource?.resourceId ?? `local-resource-video-${generatedIndex}`,
title,
src: generated.videoSrc,
mediaType: 'video',
assetKind,
assetKind: (resource?.assetKind as CanvasLayer['assetKind'] | undefined) ?? assetKind,
x: frameX ?? worldCenter.x - width / 2,
y: frameY ?? worldCenter.y - height / 2,
width,
@@ -436,14 +438,15 @@ export function createVideoResultLayer({
model: generated.model,
provider: generated.provider,
taskId: generated.taskId,
objectKey: generated.objectKey,
assetObjectId: generated.assetObjectId,
objectKey: resource?.objectKey ?? generated.objectKey,
assetObjectId: resource?.assetObjectId ?? generated.assetObjectId,
thumbnailSrc: generated.thumbnailSrc ?? generated.asset?.thumbnailSrc ?? undefined,
sourceResourceId: sourceLayer?.resourceId,
sourceAssetId: asset?.assetId,
groupId: sourceLayer?.groupId,
generationInputs,
generationInputs: resource?.generationInputs ?? generationInputs,
durationSeconds: generated.durationSeconds,
generatedAssetSnapshot: generated.asset ?? undefined,
generatedAssetSnapshot: asset ?? generated.asset ?? undefined,
};
}
@@ -72,6 +72,9 @@ export function resolvePointerSelection({
isMultiSelectGesture: boolean;
}) {
if (!isMultiSelectGesture) {
if (selectedIds.includes(targetId)) {
return selectedIds;
}
return [targetId];
}
if (!selectedIds.includes(targetId)) {
@@ -181,6 +181,58 @@ describe('ImageCanvasStageInteractionModel', () => {
});
});
it('keeps the existing mixed selection when dragging an already selected layer', () => {
const selectedGenerationId = getCanvasGenerationSelectionId('dialog-1');
const layers = [
createLayer({ id: 'a', x: 40, y: 50 }),
createLayer({ id: 'b', x: 120, y: 150 }),
];
const generationDialogs: CanvasGenerationDialogState[] = [
{
id: 'dialog-1',
mode: 'generate',
prompt: '',
status: 'idle',
placeholder: {
x: 400,
y: 180,
width: 320,
height: 320,
originalWidth: 1024,
originalHeight: 1024,
},
},
];
expect(
createLayerDragStart({
layer: layers[1]!,
layers,
generationDialogs,
selectedLayerIds: ['a', 'b', selectedGenerationId],
isMultiSelectGesture: false,
pointerId: 4,
pointer: { x: 300, y: 260 },
viewportScale: 1.25,
}),
).toEqual(
expect.objectContaining({
selectedLayerIds: ['a', 'b', selectedGenerationId],
selectedLayerId: 'b',
dragState: expect.objectContaining({
layerId: 'b',
layerIds: ['a', 'b'],
generationDialogIds: ['dialog-1'],
startLayers: [
{ id: 'a', x: 40, y: 50 },
{ id: 'b', x: 120, y: 150 },
],
startGenerationFrames: [{ id: 'dialog-1', x: 400, y: 180 }],
}),
}),
);
});
it('closes canvas generation composers when clicking existing layers', () => {
const generatedDialog: GenerateDialogState = {
id: 'dialog-1',
@@ -365,4 +417,66 @@ describe('ImageCanvasStageInteractionModel', () => {
},
});
});
it('keeps the existing mixed selection when dragging an already selected generation frame', () => {
const dialog: CanvasGenerationDialogState = {
id: 'dialog-1',
mode: 'generate',
prompt: '',
status: 'idle',
placeholder: {
x: 200,
y: 160,
width: 420,
height: 420,
originalWidth: 2048,
originalHeight: 2048,
},
};
const anotherDialog: CanvasGenerationDialogState = {
...dialog,
id: 'dialog-2',
placeholder: {
...dialog.placeholder!,
x: 680,
y: 160,
},
};
const layers = [createLayer({ id: 'layer-a', x: 40, y: 50 })];
expect(
createGenerationFrameSelectionStart({
dialog: anotherDialog,
layers,
generationDialogs: [dialog, anotherDialog],
selectedLayerIds: [
'layer-a',
getCanvasGenerationSelectionId('dialog-1'),
getCanvasGenerationSelectionId('dialog-2'),
],
isMultiSelectGesture: false,
pointerId: 12,
pointer: { x: 360, y: 260 },
viewportScale: 1,
}),
).toEqual(
expect.objectContaining({
selectedLayerIds: [
'layer-a',
getCanvasGenerationSelectionId('dialog-1'),
getCanvasGenerationSelectionId('dialog-2'),
],
dragState: expect.objectContaining({
dialogId: 'dialog-2',
dialogIds: ['dialog-1', 'dialog-2'],
layerIds: ['layer-a'],
startFrames: [
{ id: 'dialog-1', x: 200, y: 160 },
{ id: 'dialog-2', x: 680, y: 160 },
],
startLayers: [{ id: 'layer-a', x: 40, y: 50 }],
}),
}),
);
});
});
@@ -154,6 +154,19 @@ describe('ImageCanvasUploadModel', () => {
width: 1536,
height: 1024,
});
expect(
createUploadingAssetPlaceholder({
uploadIndex: 9,
fileName: 'intro.mp4',
folderId: 'characters',
mediaType: 'video',
imageDimensions: { width: 1280, height: 720 },
}),
).toMatchObject({
id: 'upload-9',
width: 1280,
height: 720,
});
});
it('opens the target upload folder without changing other folders', () => {
@@ -520,6 +533,31 @@ describe('ImageCanvasUploadModel', () => {
});
});
it('creates upload video layers with measured video dimensions', () => {
const layer = createUploadCanvasLayer({
uploadIndex: 1,
fileName: '画布视频.mp4',
imageSrc: '/generated/video.mp4',
mediaType: 'video',
imageDimensions: { width: 1280, height: 720 },
canvasPoint: { x: 110, y: 120 },
canvasSize: { width: 900, height: 640 },
viewport: { x: 10, y: 20, scale: 2 },
});
expect(layer).toMatchObject({
title: '画布视频.mp4',
mediaType: 'video',
assetKind: 'video',
x: -590,
y: -310,
width: 1280,
height: 720,
originalWidth: 1280,
originalHeight: 720,
});
});
it('uses viewport center fallback for invalid upload points', () => {
const layer = createUploadCanvasLayer({
uploadIndex: 2,
@@ -310,7 +310,7 @@ export function createUploadingAssetPlaceholder({
}): EditorAsset {
const fallbackSize = resolveUploadFallbackSize(mediaType);
const size =
mediaType === 'image' && imageDimensions
mediaType !== 'audio' && imageDimensions
? resolveLayerResolutionSize(
imageDimensions.width,
imageDimensions.height,
@@ -562,7 +562,7 @@ export function createUploadCanvasLayer({
}): CanvasLayer {
const fallbackSize = resolveUploadFallbackSize(mediaType);
const layerSize =
mediaType === 'image' && imageDimensions
mediaType !== 'audio' && imageDimensions
? resolveLayerResolutionSize(
imageDimensions.width,
imageDimensions.height,
@@ -1489,6 +1489,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({
generationInputs: submissionPlan.result.generationInputs,
assetFolderId,
assetLabel: submissionPlan.result.title,
assetKind: 'video',
...(projectId && canvasCompletionPlaceholder
? {
canvasCompletion: {
@@ -102,6 +102,21 @@ function createSizedPngFile(name: string, width: number, height: number) {
return new File([bytes], name, { type: 'image/png' });
}
function setVideoElementSize(
element: HTMLMediaElement | undefined,
width: number,
height: number,
) {
Object.defineProperty(element, 'videoWidth', {
configurable: true,
value: width,
});
Object.defineProperty(element, 'videoHeight', {
configurable: true,
value: height,
});
}
function createDeferred<T>() {
let resolve!: (value: T) => void;
let reject!: (reason?: unknown) => void;
@@ -248,6 +263,20 @@ function UploadWorkflowHarness({
>
</button>
<button
type="button"
onClick={() =>
workflow.addUploadedFiles(
[new File(['video'], '画布视频.mp4', { type: 'video/mp4' })],
{
addToCanvas: true,
canvasPoint: { x: 110, y: 120 },
},
)
}
>
</button>
<button
type="button"
onClick={() =>
@@ -587,6 +616,44 @@ describe('useImageCanvasUploadWorkflow', () => {
);
});
it('uses the uploaded video resolution before creating the canvas layer and persisted asset', async () => {
render(<UploadWorkflowHarness />);
fireEvent.click(screen.getByRole('button', { name: '上传视频到画布' }));
await waitFor(() => {
expect(
createdMediaElements.some(
(element) => element.tagName.toLowerCase() === 'video',
),
).toBe(true);
});
createdMediaElements
.filter((element) => element.tagName.toLowerCase() === 'video')
.forEach((element) => setVideoElementSize(element, 1280, 720));
act(() => {
createdMediaElements.forEach((element) =>
element.onloadedmetadata?.(new Event('loadedmetadata')),
);
});
await waitFor(() => {
expect(createEditorAssetMock).toHaveBeenCalledWith(
expect.objectContaining({
label: '画布视频.mp4',
width: 1280,
height: 720,
durationSeconds: 4,
}),
);
});
expect(screen.getByTestId('assets').textContent).toContain(
'persisted-画布视频.mp4:画布视频.mp4:folder-uploaded:ready:-:1280x720',
);
expect(screen.getByTestId('layers').textContent).toContain(
'layer-upload-1:画布视频.mp4:persisted-画布视频.mp4:-590:-310:1280x720',
);
});
it('marks upload cards as failed and reopens login on auth errors returned by asset creation', async () => {
const openEditorLoginModal = vi.fn();
createEditorAssetMock.mockRejectedValueOnce(
@@ -40,6 +40,7 @@ import {
isSeedanceReferenceImageFile,
isSeedanceReferenceVideoFile,
probeImageFileDimensions,
probeMediaFileMetadata,
probeMediaDurationSeconds,
readFileAsDataUrl,
readImageFileAsDataUrl,
@@ -944,10 +945,18 @@ export function useImageCanvasUploadWorkflow({
}
const uploadIndex = options.uploadIndex;
let imageDimensions: { width: number; height: number } | null = null;
let mediaMetadata: Awaited<
ReturnType<typeof probeMediaFileMetadata>
> | null = null;
if (mediaType === 'image') {
imageDimensions = await probeImageFileDimensions(file).catch(
() => null,
);
} else if (mediaType === 'video') {
mediaMetadata = await probeMediaFileMetadata(file, 'video').catch(
() => null,
);
imageDimensions = mediaMetadata?.dimensions ?? null;
}
const uploadedAsset = createUploadingAssetPlaceholder({
uploadIndex,
@@ -970,10 +979,11 @@ export function useImageCanvasUploadWorkflow({
let durationSeconds: number | undefined;
try {
if (mediaType !== 'image') {
durationSeconds = await probeMediaDurationSeconds(
file,
mediaType,
).catch(() => undefined);
durationSeconds =
mediaMetadata?.durationSeconds ??
(await probeMediaDurationSeconds(file, mediaType).catch(
() => undefined,
));
}
const uploadedMedia = await uploadEditorMediaAssetFile(file, mediaType);
imageSrc = uploadedMedia.legacyPublicPath;
+9
View File
@@ -4745,6 +4745,15 @@ html[data-mobile-keyboard-open='true'] .platform-mobile-bottom-dock {
color: #f8fafc;
}
.image-canvas-editor__asset-video-preview {
display: block;
width: 100%;
aspect-ratio: 1;
border-radius: 0.35rem;
background: #0f172a;
object-fit: cover;
}
.image-canvas-editor__media-preview {
position: relative;
isolation: isolate;
@@ -350,6 +350,8 @@ export type EditorVideoGenerationResult = {
taskId: string;
durationSeconds: number;
priceMudPoints?: number;
resource?: EditorProjectResourceSnapshot | null;
asset?: EditorAssetSnapshot | null;
project?: EditorProjectSnapshot | null;
asset?: EditorAssetSnapshot | null;
queueState?: ExternalGenerationJobStatusRecord | null;