收紧编辑器媒体引用契约

禁止当前画布与外部编辑器接口接收 Data URL 和 Blob URL
生成请求统一先上传 OSS 并使用稳定资源引用
角色动画和视频签名前校验资源归属并支持资源与素材 ID
同步前后端 256KB 引用限制及相关文档测试
This commit is contained in:
2026-07-17 16:44:01 +08:00
parent 93b27a2307
commit 18af49c27f
18 changed files with 697 additions and 393 deletions
+15 -15
View File
@@ -240,21 +240,21 @@
- 验证:`npm run test -- src/components/image-editor/ImageCanvasEditorModel.test.ts src/components/image-editor/ImageCanvasGenerationComposerView.test.tsx src/components/image-editor/ImageCanvasEditorGenerationIntegration.test.tsx --reporter verbose`
- 关联:`src/components/image-editor/ImageCanvasEditorModel.ts``src/components/image-editor/ImageCanvasPublicationMaterialsDemoPanelView.tsx``src/components/image-editor/ImageCanvasEditorGenerationIntegration.test.tsx`
## 图片画布快速编辑不要直接提交普通图片 URL
## 图片画布生成请求不要直接提交临时媒体源
- 现象:图片画布快速编辑站内示例图、历史 generated 图或 OSS generated 图时,后端返回 `修改图片参考图必须是图片 Data URL。`
- 原因:快速编辑直接把图层 `src` 塞进 `/api/editor/images/generations``referenceImageSrcs`;默认示例图和部分持久化图层的 `src``/creation-type-references/*.webp``/generated-*` 或 OSS URL,而 `api-server` 的编辑参考图解析只接收 `data:image/*;base64,...`
- 处理:前端统一通过 `resolveEditorImageReferenceDataUrl(...)` 在提交前读取图片字节并转成图片 Data URL;Data URL 原样透传,`/generated-*` 和 generated OSS URL 先走 `/api/assets/read-url` 换签后由浏览器直读 OSS,直读失败时才 fallback 到 `/api/assets/read-bytes`,普通 public 路径直接 fetch
- 验证:`npm run test -- src/services/image-editor/editorImageReference.test.ts src/components/image-editor/ImageCanvasEditorView.test.tsx -t "editorImageReference|converts non-data-url quick edit source images before submitting references"`
- 关联:`src/services/image-editor/editorImageReference.ts``src/components/image-editor/ImageCanvasEditorView.tsx``docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md`
- 现象:图片画布快速编辑、参考生成、去背景或角色动画提交 Data URL / Blob URL 时,前端或后端返回“必须先上传 OSS”
- 原因:Data URL / Blob URL 体积大且不能作为持久引用;如果写入外部生成队列,请求体会膨胀,worker 也无法稳定复用浏览器临时资源
- 处理:提交前优先复用图层已有 `objectKey`、项目资源 ID 或素材 ID;未登记的本地图片和普通 public 图片路径先通过 `resolveEditorGenerationMediaReference(...)` 读取并上传 OSS,再把稳定引用交给生成接口。Data URL 只允许用于浏览器内压缩、标注等临时处理,不能进入 API 请求、队列载荷或项目持久化
- 验证:`npm run test -- src/services/image-editor/editorProjectClient.test.ts src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx``cargo test -p api-server inline_data_url --manifest-path server-rs/Cargo.toml`
- 关联:`src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts``src/services/image-editor/editorProjectClient.ts``server-rs/crates/api-server/src/editor_generation_queue.rs``docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md`
## 图片编辑器角色动画不要默认提交大图 Data URL
## 图片编辑器角色动画必须提交稳定图片引用
- 现象:图片编辑器里对角色图点击 `生成动画` 后,后端返回 `Failed to buffer the request body: length limit exceeded`,请求还没进入角色动画 handler
- 原因:角色动画生成请求曾把角色图片 `src` 原样作为 `sourceImageSrc` 放进 JSON;角色图如果是较大的 Data URL,会超过 Axum 默认 `2MB` body limit,在 `Json` 提取器阶段被拦截
- 处理:前端在角色图已持久化时优先提交 `objectKey`,只把 Data URL 作为未持久化本地临时图兜底;后端 `/api/editor/character-animations/generations` 单独配置 `12MB` body limit 兼容旧请求,但新链路不应依赖传大图 JSON
- 验证:`npm run test -- src/components/image-editor/ImageCanvasEditorView.test.tsx -t "only exposes character animation"``cargo test -p api-server editor_character_animation_accepts_character_image_body_above_default_limit --manifest-path server-rs/Cargo.toml`
- 关联:`src/components/image-editor/ImageCanvasEditorView.tsx``server-rs/crates/api-server/src/modules/play_flow.rs``server-rs/crates/api-server/src/app.rs``docs/【编辑器】画板角色形象生成入口设计-2026-06-15.md`
- 现象:图片编辑器里对尚未上传的角色图点击 `生成动画` 后,前端或后端返回 `sourceImageSrc 必须先上传 OSS`
- 原因:角色动画会进入外部生成队列,浏览器 Data URL / Blob URL 既不适合持久任务,也会放大 JSON 请求体
- 处理:前端统一通过 `resolveEditorGenerationMediaReference(...)` 取得 `objectKey` 或画板资源引用;本地临时角色图必须先上传 OSS。后端在入队前同步拒绝内联媒体,不再通过放宽 body limit 兼容 Data URL
- 验证:`npm run test -- src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx``cargo test -p api-server editor_character_animation --manifest-path server-rs/Cargo.toml`
- 关联:`src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts``server-rs/crates/api-server/src/character_animation_assets.rs``server-rs/crates/api-server/src/app.rs``docs/【编辑器】画板角色形象生成入口设计-2026-06-15.md`
## 图片编辑器角色动画抽帧不要采到视频尾点
@@ -328,11 +328,11 @@
- 验证:`npm run test -- src/services/assetReadUrlService.test.ts --reporter verbose` 应覆盖大量不同 objectKey 同时换签时首批限量放行、后续按间隔派发;发布现场同类页面刷新时,Nginx `GET /api/assets/read-url` 429 应从 `upstream_status=-` / `genarrative_api_rps` 收敛。
- 关联:`src/services/assetReadUrlService.ts``src/hooks/useResolvedAssetReadUrl.ts``src/components/ResolvedAssetImage.tsx``docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md`
## 图片编辑器 Seedance 2.0 参考媒体不要提交视频 Data URL
## 图片编辑器 Seedance 2.0 参考媒体只提交稳定引用
- 现象:画板生成视频选择 Seedance 2.0 并上传参考视频后,请求体暴涨、可能返回 `413` 或上游拒绝 `video_url.url`;文档示例或测试如果写 `data:video/mp4;base64,...`,后续实现很容易照抄。
- 原因:火山 Seedance 2.0 参考视频只支持公网 URL 或 `asset://` 素材 ID,项目内画板资源应以 `objectKey` 由后端换签;视频不支持 Base64 / `data:video`,且 50MB 视频转 Base64 后会逼近或超过 64MB 请求体上限。参考音频虽然支持 Base64,但也不能单独输入,且大文件同样不应塞进 JSON
- 处理:参考视频 / 音频上传先走 `/api/assets/direct-upload-tickets` 直传 OSS,再 `/api/assets/objects/confirm` 确认;前端保留 signed URL 做预览,提交生成时优先使用 `objectKey`。后端归一化必须拒绝 `data:video/*`,非 Seedance 模型携带参考字段也必须拒绝;Ark body 显式带 `generate_audio:false`
- 原因:画板生成会进入持久队列,Base64 / Data URL / Blob URL 会放大请求和任务 JSON;直接签名客户端给出的 objectKey 又会绕过跨账号素材归属校验。项目资源 ID、素材 ID 和 objectKey 必须先解析到当前 owner 的正式对象,公网 URL 与 `asset://` 才能按供应商契约直接透传
- 处理:本地参考图片 / 视频 / 音频先走 `/api/assets/direct-upload-tickets` 直传 OSS,再 `/api/assets/objects/confirm` 确认;前端保留 signed URL 做预览,提交生成时使用 `objectKey`、项目资源 ID 或素材 ID。后端归一化拒绝全部 `data:*` / `blob:*`,签名 OSS URL 前统一校验 owner;稳定引用字段总长度限制为 `256KB`,非 Seedance 模型携带参考字段也必须拒绝;Ark body 显式带 `generate_audio:false`
- 验证:`npx vitest run src/components/image-editor/useImageCanvasUploadWorkflow.test.tsx src/components/image-editor/ImageCanvasGenerationSubmissionModel.test.ts src/services/image-editor/editorReferenceUploadClient.test.ts --reporter verbose``cargo test -p api-server editor_video --manifest-path server-rs/Cargo.toml``cargo test -p shared-contracts editor_video_request_supports_seedance_multimodal_references --manifest-path server-rs/Cargo.toml`
- 关联:`src/services/image-editor/editorReferenceUploadClient.ts``src/components/image-editor/useImageCanvasUploadWorkflow.ts``src/components/image-editor/ImageCanvasGenerationSubmissionModel.ts``server-rs/crates/api-server/src/character_animation_assets.rs``docs/【编辑器】生成类面板Lovart统一改造方案-2026-06-17.md`
File diff suppressed because one or more lines are too long
@@ -160,11 +160,11 @@
- 参考媒体只对 `seedance2.0-fast` / `seedance2.0` 开放;切换到 Kling 等非 Seedance 模型时,前端不提交 `referenceImageSrcs` / `referenceVideoSrcs` / `referenceAudioSrcs`,后端收到非 Seedance 参考字段必须拒绝。
- 多模态参考按火山 Seedance 2.0 文档限制:参考图片 0~9 张,参考视频 0~3 个,参考音频 0~3 段;音频不可单独输入,必须至少搭配 1 张参考图片或 1 个参考视频。
- 参考图片支持 URL / Base64 / `asset://` / 画板资源路径;格式为 jpeg、png、webp、bmp、tiff、gif、heic、heif,单张小于 30MB请求体总大小不超过 64MB
- 参考视频支持 URL / `asset://` / 画板资源路径,不支持 Base64 / `data:video/*`;格式为 mp4、mov,单个文件不超过 50MB,单个时长 [2, 15] 秒,最多 3 个且总时长不超过 15 秒。
- 参考音频支持 URL / Base64 / `asset://` / 画板资源路径;格式为 wav、mp3,单个文件不超过 15MB,单个时长 [2, 15] 秒,最多 3 段且总时长不超过 15 秒。
- 前端上传参考视频 / 参考音频必须走 `/api/assets/direct-upload-tickets` 直传 OSS,再 `/api/assets/objects/confirm` 入库;前端状态保存 signed URL 供预览,同时保存 `objectKey` / `assetObjectId`,提交生成时优先使用 `objectKey`
- 后端接收画板资源 `objectKey` 后统一重新签名为 Ark 可读 URL按文档构造 `content``image_url` + `role=reference_image``video_url` + `role=reference_video``audio_url` + `role=reference_audio`
- 参考图片支持公网 URL`asset://`、当前账号的 `objectKey`、项目资源 ID 或素材 ID;格式为 jpeg、png、webp、bmp、tiff、gif、heic、heif,单张小于 30MB不接受 Base64 / Data URL / Blob URL
- 参考视频支持公网 URL`asset://`、当前账号的 `objectKey`、项目资源 ID 或素材 ID;格式为 mp4、mov,单个文件不超过 50MB,单个时长 [2, 15] 秒,最多 3 个且总时长不超过 15 秒,不接受 Base64 / Data URL / Blob URL
- 参考音频支持公网 URL`asset://`、当前账号的 `objectKey`、项目资源 ID 或素材 ID;格式为 wav、mp3,单个文件不超过 15MB,单个时长 [2, 15] 秒,最多 3 段且总时长不超过 15 秒,不接受 Base64 / Data URL / Blob URL。三类参考媒体最终提交的稳定引用字段总长度不得超过 `256KB`
- 前端本地参考媒体必须走 `/api/assets/direct-upload-tickets` 直传 OSS,再 `/api/assets/objects/confirm` 入库;前端状态保存 signed URL 供预览,同时保存 `objectKey` / `assetObjectId`,提交生成时使用稳定引用
- 后端接收 `objectKey`、项目资源 ID 或素材 ID 后,先解析并校验对象属于当前 owner,再重新签名为 Ark 可读 URL按文档构造 `content``image_url` + `role=reference_image``video_url` + `role=reference_video``audio_url` + `role=reference_audio`
- 画板生成视频按静音 toggle 映射 Ark `generate_audio``sound=on` 时传 `true``sound=off` 时传 `false`,不能只依赖上游默认值。
- Seedance 2.0 文档提示不支持直接上传含真人人脸的参考图 / 视频;当前画板尚未做真人脸授权证明、来源声明或服务端人脸拦截,后续开放真人素材前必须补授权/来源确认链路。
@@ -147,9 +147,9 @@
### Prompt 与生成契约
- 前端提交到 `POST /api/editor/character-animations/generations`
- 请求必须带上角色图片来源、原始尺寸、动画描述、分辨率、画面比例、帧数和时长,并可通过 `assetFolderId` 指定中间视频素材落点。角色图片已经持久化到 OSS 时,`sourceImageSrc` 必须优先传 `objectKey`;只有未持久化的本地临时图片才允许传 Data URL
- 请求必须带上角色图片来源、原始尺寸、动画描述、分辨率、画面比例、帧数和时长,并可通过 `assetFolderId` 指定中间视频素材落点。`sourceImageSrc` 只允许传当前账号的 `objectKey`、项目资源 ID 或素材 ID;未持久化的本地临时图片必须先上传 OSS,不能提交 Data URL 或 Blob URL。后端解析稳定引用后必须校验对象属于当前 owner,再读取 OSS 内容
- 后端使用角色图片作为首帧和尾帧参考,模型固定映射到 `doubao-seedance-2-0-fast-260128`
- 后端路由兼容旧 Data URL 请求并单独放宽 JSON body limit 到 `12MB`,但该限额只作为兼容兜底,不作为新链路默认传大图的方式
- 后端在内联执行和队列入队前都会拒绝 Data URL / Blob URL;路由使用默认 JSON body limit,不再为内联图片请求单独放宽
- 后端 prompt 使用以下固定骨架,并把面板输入追加到 `动作描述:` 后:
```text
+24 -38
View File
@@ -1725,19 +1725,18 @@ mod tests {
}
#[tokio::test]
async fn editor_character_animation_accepts_character_image_body_above_default_limit() {
async fn editor_character_animation_rejects_inline_data_url_before_queueing() {
let state = AppState::new(AppConfig {
external_generation_mode: ExternalGenerationMode::Inline,
external_generation_mode: ExternalGenerationMode::Queue,
..AppConfig::default()
})
.expect("state should build");
let seed_user = seed_phone_user_with_password(&state, "13800138190", TEST_PASSWORD).await;
let token = sign_test_user_token(&state, &seed_user, "sess_editor_character_body");
let app = build_router(state);
let source_image_src = format!("data:image/png;base64,{}", "A".repeat(3 * 1024 * 1024));
let request_body = serde_json::json!({
"sourceLayerId": "layer-character-large",
"sourceImageSrc": source_image_src,
"sourceImageSrc": "data:image/png;base64,AAAA",
"sourceWidth": 1024,
"sourceHeight": 1024,
"promptText": "待机呼吸循环。",
@@ -1748,7 +1747,6 @@ mod tests {
"model": "seedance2.0-fast"
})
.to_string();
assert!(request_body.len() > 2 * 1024 * 1024);
let response = app
.oneshot(
@@ -1763,7 +1761,7 @@ mod tests {
.await
.expect("request should succeed");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let body = response
.into_body()
.collect()
@@ -1772,10 +1770,9 @@ mod tests {
.to_bytes();
let body_text = String::from_utf8_lossy(&body);
assert!(
body_text.contains("ARK_CHARACTER_VIDEO_BASE_URL"),
"handler should parse the oversized character source image before checking Ark config: {body_text}"
body_text.contains("先上传 OSS"),
"handler should reject inline character sources: {body_text}"
);
assert!(!body_text.contains("length limit exceeded"));
}
#[tokio::test]
@@ -1811,23 +1808,21 @@ mod tests {
}
#[tokio::test]
async fn editor_ui_design_asset_extraction_accepts_image_body_above_default_limit() {
async fn editor_ui_design_asset_extraction_rejects_inline_data_url_before_queueing() {
let state = AppState::new(AppConfig {
external_generation_mode: ExternalGenerationMode::Inline,
external_generation_mode: ExternalGenerationMode::Queue,
..AppConfig::default()
})
.expect("state should build");
let seed_user = seed_phone_user_with_password(&state, "13800138188", TEST_PASSWORD).await;
let token = sign_test_user_token(&state, &seed_user, "sess_editor_ui_extract_body");
let app = build_router(state);
let source_image_src = format!("data:image/png;base64,{}", "A".repeat(3 * 1024 * 1024));
let request_body = serde_json::json!({
"sourceImageSrc": source_image_src,
"sourceImageSrc": "data:image/png;base64,AAAA",
"aspectRatio": "1:1",
"imageSize": "1K",
})
.to_string();
assert!(request_body.len() > 2 * 1024 * 1024);
let response = app
.oneshot(
@@ -1842,7 +1837,7 @@ mod tests {
.await
.expect("request should succeed");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let body = response
.into_body()
.collect()
@@ -1851,33 +1846,28 @@ mod tests {
.to_bytes();
let body_text = String::from_utf8_lossy(&body);
assert!(
body_text.contains("VECTOR_ENGINE")
|| body_text.contains("OPENAI")
|| body_text.contains("配置"),
"handler should parse the oversized UI design image before checking image config: {body_text}"
body_text.contains("先上传 OSS"),
"handler should reject inline UI design sources: {body_text}"
);
assert!(!body_text.contains("length limit exceeded"));
}
#[tokio::test]
async fn editor_image_generation_accepts_reference_body_above_default_limit() {
async fn editor_image_generation_rejects_inline_data_url_before_queueing() {
let state = AppState::new(AppConfig {
external_generation_mode: ExternalGenerationMode::Inline,
external_generation_mode: ExternalGenerationMode::Queue,
..AppConfig::default()
})
.expect("state should build");
let seed_user = seed_phone_user_with_password(&state, "13800138028", TEST_PASSWORD).await;
let token = sign_test_user_token(&state, &seed_user, "sess_editor_image_reference_body");
let app = build_router(state);
let reference_image_src = format!("data:image/png;base64,{}", "A".repeat(3 * 1024 * 1024));
let request_body = serde_json::json!({
"prompt": "生成宣发素材",
"kind": "publication-material",
"size": "720x540",
"referenceImageSrcs": [reference_image_src],
"referenceImageSrcs": ["data:image/png;base64,AAAA"],
})
.to_string();
assert!(request_body.len() > 2 * 1024 * 1024);
let response = app
.oneshot(
@@ -1892,7 +1882,7 @@ mod tests {
.await
.expect("request should succeed");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let body = response
.into_body()
.collect()
@@ -1901,31 +1891,28 @@ mod tests {
.to_bytes();
let body_text = String::from_utf8_lossy(&body);
assert!(
body_text.contains("VECTOR_ENGINE_BASE_URL"),
"handler should parse the oversized editor reference before checking VectorEngine config: {body_text}"
body_text.contains("先上传 OSS"),
"handler should reject inline editor references: {body_text}"
);
assert!(!body_text.contains("length limit exceeded"));
}
#[tokio::test]
async fn editor_image_edit_accepts_reference_body_above_default_limit() {
async fn editor_image_edit_rejects_inline_data_url_before_queueing() {
let state = AppState::new(AppConfig {
external_generation_mode: ExternalGenerationMode::Inline,
external_generation_mode: ExternalGenerationMode::Queue,
..AppConfig::default()
})
.expect("state should build");
let seed_user = seed_phone_user_with_password(&state, "13800138029", TEST_PASSWORD).await;
let token = sign_test_user_token(&state, &seed_user, "sess_editor_image_edit_body");
let app = build_router(state);
let source_image_src = format!("data:image/png;base64,{}", "A".repeat(3 * 1024 * 1024));
let request_body = serde_json::json!({
"prompt": "快速编辑图片",
"sourceImageSrc": source_image_src,
"sourceImageSrc": "data:image/png;base64,AAAA",
"size": "1024x1024",
"model": "gpt-image-2"
})
.to_string();
assert!(request_body.len() > 2 * 1024 * 1024);
let response = app
.oneshot(
@@ -1940,7 +1927,7 @@ mod tests {
.await
.expect("request should succeed");
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
let body = response
.into_body()
.collect()
@@ -1949,10 +1936,9 @@ mod tests {
.to_bytes();
let body_text = String::from_utf8_lossy(&body);
assert!(
body_text.contains("VECTOR_ENGINE_BASE_URL"),
"handler should parse the oversized editor edit image before checking VectorEngine config: {body_text}"
body_text.contains("先上传 OSS"),
"handler should reject inline editor edit sources: {body_text}"
);
assert!(!body_text.contains("length limit exceeded"));
}
#[tokio::test]
@@ -64,13 +64,15 @@ use crate::{
EditorScreenBackgroundColor, editor_green_screen_character_prompt_clause,
},
editor_project::{
EDITOR_BGFILTER_CROSS_CHECK_ENABLED, EDITOR_BGFILTER_DEFAULT_SEG_MODEL,
EDITOR_ASSET_ID_PREFIX, EDITOR_BGFILTER_CROSS_CHECK_ENABLED,
EDITOR_BGFILTER_DEFAULT_SEG_MODEL, EDITOR_RESOURCE_ID_PREFIX,
EditorCanvasGeneratedLayerInput, EditorGenerationPhaseReporter,
PersistEditorGeneratedAssetRequest,
apply_editor_screen_background_decision_to_generation_inputs,
build_editor_canvas_generated_layer_item, complete_editor_canvas_generation_with_items,
persist_editor_generated_media_asset,
remove_editor_generated_screen_background_with_bgfilter_with_request_timeout,
resolve_editor_reference_object_key_for_owner,
},
editor_screen_background_decision::{
EditorScreenBackgroundDecisionInput, EditorScreenBackgroundDecisionKind,
@@ -130,7 +132,7 @@ const EDITOR_VIDEO_MODEL_VEO_3_1_FAST: &str = "veo3.1-fast";
const EDITOR_VIDEO_REFERENCE_IMAGE_LIMIT: usize = 9;
const EDITOR_VIDEO_REFERENCE_VIDEO_LIMIT: usize = 3;
const EDITOR_VIDEO_REFERENCE_AUDIO_LIMIT: usize = 3;
const EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES: usize = 64 * 1024 * 1024;
const EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES: usize = 256 * 1024;
const EDITOR_VIDEO_REFERENCE_READ_EXPIRE_SECONDS: u64 = 60 * 60;
const BUILT_IN_MOTION_TEMPLATES: [MotionTemplate; 4] = [
@@ -568,6 +570,14 @@ pub async fn generate_editor_character_animation(
)
})?;
let owner_user_id = authenticated.claims().user_id().to_string();
if matches_inline_media_source(payload.source_image_src.as_str()) {
return Err(character_animation_error_response(
&request_context,
editor_character_animation_bad_request(
"sourceImageSrc 必须先上传 OSS,并使用 objectKey 或画板资源引用。",
),
));
}
if !state.config.external_generation_mode.is_inline() {
let pricing = state.editor_generation_pricing().await.map_err(|error| {
character_animation_error_response(
@@ -641,6 +651,14 @@ pub(crate) async fn generate_editor_character_animation_for_owner(
let asset_folder_id = payload.asset_folder_id.clone();
let asset_label =
resolve_editor_character_animation_asset_label(payload.asset_label.as_deref());
if matches_inline_media_source(payload.source_image_src.as_str()) {
return Err(character_animation_error_response(
&request_context,
editor_character_animation_bad_request(
"sourceImageSrc 必须先上传 OSS,并使用 objectKey 或画板资源引用。",
),
));
}
let pricing = state.editor_generation_pricing().await.map_err(|error| {
character_animation_error_response(
@@ -654,10 +672,17 @@ pub(crate) async fn generate_editor_character_animation_for_owner(
// 先解析源角色图:图生视频的主体颜色完全由源图决定,背景色决策必须看到这张图。
let source_resolve_client = build_upstream_http_client(SOURCE_IMAGE_RESOLVE_TIMEOUT_MS)
.map_err(|error| character_animation_error_response(&request_context, error))?;
let source_object_key = resolve_editor_reference_object_key_for_owner(
&state,
owner_user_id.as_str(),
payload.source_image_src.trim(),
)
.await
.map_err(|error| character_animation_error_response(&request_context, error))?;
let source_data_url = resolve_media_source_as_data_url(
&state,
&source_resolve_client,
payload.source_image_src.trim(),
source_object_key.as_str(),
"sourceImageSrc",
)
.await
@@ -2014,7 +2039,8 @@ async fn request_editor_video_preview(
request: &NormalizedEditorVideoRequest,
) -> Result<GeneratedAnimationPreview, AppError> {
let upstream_task_id =
create_editor_text_to_video_task(state, http_client, settings, request).await?;
create_editor_text_to_video_task(state, http_client, settings, owner_user_id, request)
.await?;
let video_url =
wait_for_ark_content_generation_task(http_client, &settings.ark, upstream_task_id.as_str())
.await?;
@@ -2038,9 +2064,10 @@ async fn create_editor_text_to_video_task(
state: &AppState,
http_client: &reqwest::Client,
settings: &EditorVideoSettings,
owner_user_id: &str,
request: &NormalizedEditorVideoRequest,
) -> Result<String, AppError> {
let request_body = build_editor_video_ark_request_body(state, request)?;
let request_body = build_editor_video_ark_request_body(state, owner_user_id, request).await?;
let response = http_client
.post(format!(
"{}/contents/generations/tasks",
@@ -3402,11 +3429,12 @@ fn normalize_editor_video_request_with_pricing(
})
}
fn build_editor_video_ark_request_body(
async fn build_editor_video_ark_request_body(
state: &AppState,
owner_user_id: &str,
request: &NormalizedEditorVideoRequest,
) -> Result<Value, AppError> {
let content = build_editor_video_ark_content(state, request)?;
let content = build_editor_video_ark_content(state, owner_user_id, request).await?;
Ok(json!({
"model": request.provider_model,
"content": content,
@@ -3418,8 +3446,9 @@ fn build_editor_video_ark_request_body(
}))
}
fn build_editor_video_ark_content(
async fn build_editor_video_ark_content(
state: &AppState,
owner_user_id: &str,
request: &NormalizedEditorVideoRequest,
) -> Result<Vec<Value>, AppError> {
let mut content = Vec::new();
@@ -3431,22 +3460,25 @@ fn build_editor_video_ark_content(
}
let reference_image_urls = resolve_editor_video_reference_urls(
state,
owner_user_id,
request.reference_image_srcs.as_slice(),
"参考图片",
true,
)?;
)
.await?;
let reference_video_urls = resolve_editor_video_reference_urls(
state,
owner_user_id,
request.reference_video_srcs.as_slice(),
"参考视频",
false,
)?;
)
.await?;
let reference_audio_urls = resolve_editor_video_reference_urls(
state,
owner_user_id,
request.reference_audio_srcs.as_slice(),
"参考音频",
true,
)?;
)
.await?;
content.extend(reference_image_urls.iter().map(|url| {
json!({
"type": "image_url",
@@ -3471,31 +3503,30 @@ fn build_editor_video_ark_content(
Ok(content)
}
fn resolve_editor_video_reference_urls(
async fn resolve_editor_video_reference_urls(
state: &AppState,
owner_user_id: &str,
sources: &[String],
label: &str,
allow_data_url: bool,
) -> Result<Vec<String>, AppError> {
sources
.iter()
.map(|source| {
resolve_editor_video_reference_url(state, source.as_str(), label, allow_data_url)
})
.collect()
let mut resolved = Vec::with_capacity(sources.len());
for source in sources {
resolved.push(
resolve_editor_video_reference_url(state, owner_user_id, source.as_str(), label)
.await?,
);
}
Ok(resolved)
}
fn resolve_editor_video_reference_url(
async fn resolve_editor_video_reference_url(
state: &AppState,
owner_user_id: &str,
source: &str,
label: &str,
allow_data_url: bool,
) -> Result<String, AppError> {
let value = source.trim();
if value.starts_with("data:") {
if allow_data_url {
return Ok(value.to_string());
}
return Err(editor_video_bad_request(format!(
"{label}格式不支持,请使用官方支持的 URL、素材 ID 或画板资源路径。"
)));
@@ -3507,17 +3538,24 @@ fn resolve_editor_video_reference_url(
return Ok(value.to_string());
}
let object_key = value.trim_start_matches('/');
if LegacyAssetPrefix::from_object_key(object_key).is_none() {
let format_hint = if allow_data_url {
"公网 URL、素材 ID、data URL 或画板资源路径"
} else {
"公网 URL、素材 ID 或画板资源路径"
};
return Err(editor_video_bad_request(format!(
"{label}必须是官方支持的{format_hint}"
)));
}
let object_key = resolve_editor_reference_object_key_for_owner(state, owner_user_id, value)
.await
.map_err(|error| {
error.with_details(json!({
"provider": "editor-video",
"field": label,
"message": format!(
"{label}必须是当前账号的 objectKey、项目资源 ID、素材 ID,或官方支持的公网 URL。"
),
}))
})?;
sign_editor_video_object_key_url(state, object_key.as_str())
}
fn sign_editor_video_object_key_url(
state: &AppState,
object_key: &str,
) -> Result<String, AppError> {
let oss_client = state.oss_client().ok_or_else(|| {
AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({
"provider": "aliyun-oss",
@@ -3582,9 +3620,9 @@ fn normalize_editor_video_references(
validate_editor_video_reference_src(
src,
"参考图片",
&["data:image/", "http://", "https://", "asset://"],
&["http://", "https://", "asset://"],
true,
"对应媒体 data URL",
"URL、素材 ID 或画板资源路径",
)?;
}
for src in &video_srcs {
@@ -3600,9 +3638,9 @@ fn normalize_editor_video_references(
validate_editor_video_reference_src(
src,
"参考音频",
&["data:audio/", "http://", "https://", "asset://"],
&["http://", "https://", "asset://"],
true,
"对应媒体 data URL",
"URL、素材 ID 或画板资源路径",
)?;
}
@@ -3613,7 +3651,9 @@ fn normalize_editor_video_references(
.map(String::len)
.sum::<usize>();
if request_bytes > EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES {
return Err(editor_video_bad_request("参考文件总请求体不能超过 64MB。"));
return Err(editor_video_bad_request(
"参考素材引用字段总长度不能超过 256KB。",
));
}
Ok(NormalizedEditorVideoReferences {
@@ -3631,6 +3671,11 @@ fn normalize_reference_srcs(values: Vec<String>) -> Vec<String> {
.collect()
}
fn matches_inline_media_source(value: &str) -> bool {
let value = value.trim_start().to_ascii_lowercase();
value.starts_with("data:") || value.starts_with("blob:")
}
fn validate_editor_video_reference_src(
value: &str,
label: &str,
@@ -3649,6 +3694,9 @@ fn validate_editor_video_reference_src(
{
return Ok(());
}
if value.starts_with(EDITOR_RESOURCE_ID_PREFIX) || value.starts_with(EDITOR_ASSET_ID_PREFIX) {
return Ok(());
}
Err(editor_video_bad_request(format!(
"{label}格式不支持,请使用官方支持的{format_hint}"
)))
@@ -6072,6 +6120,34 @@ mod tests {
);
}
#[test]
fn editor_character_animation_rejects_inline_media_before_queueing() {
let source = include_str!("character_animation_assets.rs");
assert_function_contains_in_order(
source,
"pub async fn generate_editor_character_animation",
"pub(crate) async fn generate_editor_character_animation_for_owner",
&[
"matches_inline_media_source",
"enqueue_editor_generation_job",
],
);
assert_function_contains_in_order(
source,
"pub(crate) async fn generate_editor_character_animation_for_owner",
"pub async fn generate_editor_video",
&[
"resolve_editor_reference_object_key_for_owner",
"resolve_media_source_as_data_url",
],
);
assert!(matches_inline_media_source("data:image/png;base64,AAAA"));
assert!(matches_inline_media_source(" BLOB:local-preview"));
assert!(!matches_inline_media_source(
"/generated-images/editor/character.png"
));
}
#[test]
fn editor_character_animation_reports_processing_after_video_generation() {
let source = include_str!("character_animation_assets.rs");
@@ -6532,11 +6608,15 @@ mod tests {
mode: "std".to_string(),
sound: "off".to_string(),
web_search_enabled: true,
reference_image_srcs: vec!["data:image/png;base64,image".to_string()],
reference_image_srcs: vec![
"generated-character-drafts/editor/seedance-references/image/demo.png".to_string(),
],
reference_video_srcs: vec![
"generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(),
],
reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()],
reference_audio_srcs: vec![
"generated-character-drafts/editor/seedance-references/audio/demo.mp3".to_string(),
],
project_id: None,
canvas_completion: None,
generation_inputs: None,
@@ -6549,7 +6629,7 @@ mod tests {
assert_eq!(
normalized.reference_image_srcs,
vec!["data:image/png;base64,image"]
vec!["generated-character-drafts/editor/seedance-references/image/demo.png"]
);
assert_eq!(
normalized.reference_video_srcs,
@@ -6557,7 +6637,7 @@ mod tests {
);
assert_eq!(
normalized.reference_audio_srcs,
vec!["data:audio/mpeg;base64,audio"]
vec!["generated-character-drafts/editor/seedance-references/audio/demo.mp3"]
);
}
@@ -6589,7 +6669,46 @@ mod tests {
}
#[test]
fn editor_video_ark_body_sets_generate_audio_false_without_mode() {
fn editor_video_rejects_seedance_image_and_audio_data_urls() {
for (image_srcs, audio_srcs, expected_message) in [
(
vec!["data:image/png;base64,image".to_string()],
Vec::new(),
"参考图片格式不支持",
),
(
vec!["generated-character-drafts/editor/reference.png".to_string()],
vec!["data:audio/mpeg;base64,audio".to_string()],
"参考音频格式不支持",
),
] {
let error = normalize_editor_video_request(EditorVideoGenerateRequest {
prompt: "参考素材生成视频".to_string(),
model: EDITOR_VIDEO_MODEL_SEEDANCE_2_FAST.to_string(),
aspect_ratio: "16:9".to_string(),
duration_seconds: 4,
resolution: "480p".to_string(),
mode: "std".to_string(),
sound: "off".to_string(),
web_search_enabled: true,
reference_image_srcs: image_srcs,
reference_video_srcs: Vec::new(),
reference_audio_srcs: audio_srcs,
project_id: None,
canvas_completion: None,
generation_inputs: None,
source_resource_id: None,
asset_kind: None,
asset_folder_id: None,
asset_label: None,
})
.expect_err("Seedance references must not accept data URLs");
assert!(error.body_text().contains(expected_message));
}
}
#[tokio::test]
async fn editor_video_ark_body_sets_generate_audio_false_without_mode() {
let state = AppState::new(AppConfig::default()).expect("state should build");
let request = normalize_editor_video_request(EditorVideoGenerateRequest {
prompt: "参考素材生成视频".to_string(),
@@ -6600,9 +6719,9 @@ mod tests {
mode: "std".to_string(),
sound: "off".to_string(),
web_search_enabled: true,
reference_image_srcs: vec!["data:image/png;base64,image".to_string()],
reference_image_srcs: vec!["https://assets.example.com/reference.png".to_string()],
reference_video_srcs: vec!["https://assets.example.com/reference.mp4".to_string()],
reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()],
reference_audio_srcs: vec!["https://assets.example.com/reference.mp3".to_string()],
project_id: None,
canvas_completion: None,
generation_inputs: None,
@@ -6613,8 +6732,9 @@ mod tests {
})
.expect("seedance multimodal request should normalize");
let body =
build_editor_video_ark_request_body(&state, &request).expect("ark body should build");
let body = build_editor_video_ark_request_body(&state, "video-owner", &request)
.await
.expect("ark body should build");
assert_eq!(body["generate_audio"], json!(false));
assert!(body.get("mode").is_none());
@@ -6628,7 +6748,7 @@ mod tests {
}
#[test]
fn editor_video_ark_content_signs_generated_object_keys() {
fn editor_video_reference_resolution_checks_owner_before_signing() {
let state = AppState::new(AppConfig {
oss_bucket: Some("genarrative-assets".to_string()),
oss_endpoint: Some("oss-cn-shanghai.aliyuncs.com".to_string()),
@@ -6637,7 +6757,14 @@ mod tests {
..AppConfig::default()
})
.expect("state should build with OSS");
let request = normalize_editor_video_request(EditorVideoGenerateRequest {
let object_key = "generated-character-drafts/editor/seedance-references/video/demo.mp4";
let signed_url = sign_editor_video_object_key_url(&state, object_key)
.expect("validated object keys should be signable");
assert!(signed_url.starts_with("https://genarrative-assets.oss-cn-shanghai.aliyuncs.com/"));
assert!(signed_url.contains(object_key));
assert!(signed_url.contains("x-oss-signature="));
let registered_reference = normalize_editor_video_request(EditorVideoGenerateRequest {
prompt: "参考视频生成视频".to_string(),
model: EDITOR_VIDEO_MODEL_SEEDANCE_2_FAST.to_string(),
aspect_ratio: "16:9".to_string(),
@@ -6647,9 +6774,7 @@ mod tests {
sound: "off".to_string(),
web_search_enabled: true,
reference_image_srcs: Vec::new(),
reference_video_srcs: vec![
"generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(),
],
reference_video_srcs: vec!["editor-resource-video-reference".to_string()],
reference_audio_srcs: Vec::new(),
project_id: None,
canvas_completion: None,
@@ -6659,23 +6784,21 @@ mod tests {
asset_folder_id: None,
asset_label: None,
})
.expect("seedance objectKey request should normalize");
let content = build_editor_video_ark_content(&state, &request)
.expect("ark content should sign object keys");
let signed_url = content[1]["video_url"]["url"]
.as_str()
.expect("video url should be string");
assert!(signed_url.starts_with("https://genarrative-assets.oss-cn-shanghai.aliyuncs.com/"));
assert!(
signed_url
.contains("generated-character-drafts/editor/seedance-references/video/demo.mp4")
.expect("registered editor resource IDs should normalize");
assert_eq!(
registered_reference.reference_video_srcs,
vec!["editor-resource-video-reference"]
);
assert!(signed_url.contains("x-oss-signature="));
assert_ne!(
signed_url,
"generated-character-drafts/editor/seedance-references/video/demo.mp4"
let source = include_str!("character_animation_assets.rs");
assert_function_contains_in_order(
source,
"async fn resolve_editor_video_reference_url",
"fn sign_editor_video_object_key_url",
&[
"resolve_editor_reference_object_key_for_owner",
"sign_editor_video_object_key_url",
],
);
}
File diff suppressed because it is too large Load Diff
@@ -27,7 +27,6 @@ use crate::{
state::AppState,
};
const EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES: usize = 12 * 1024 * 1024;
const EDITOR_AGENT_MESSAGE_BODY_LIMIT_BYTES: usize = 2 * 1024 * 1024;
pub fn router(state: AppState) -> Router<AppState> {
@@ -168,49 +167,31 @@ pub fn router(state: AppState) -> Router<AppState> {
)
.route(
"/api/editor/images/generations",
post(generate_editor_image)
.layer(DefaultBodyLimit::max(
// 中文注释:图片画布生图仍兼容参考图 Data URL 入参,宣发素材提交前会压缩,后端保留有限兜底。
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(generate_editor_image).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/images/edits",
post(edit_editor_image)
.layer(DefaultBodyLimit::max(
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(edit_editor_image).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/images/background-removals",
post(remove_editor_image_background)
.layer(DefaultBodyLimit::max(
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(remove_editor_image_background).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/icon-spritesheets/generations",
post(generate_editor_icon_spritesheet)
// 中文注释:图标规范参考图会以 Data URL 提交,需避免默认 2MB body limit 在 handler 前拦截。
.layer(DefaultBodyLimit::max(
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(generate_editor_icon_spritesheet).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/icon-spritesheets/slices",
@@ -221,14 +202,9 @@ pub fn router(state: AppState) -> Router<AppState> {
)
.route(
"/api/editor/ui-designs/assets/extractions",
post(extract_editor_ui_design_assets)
// 中文注释:UI 设计图提取素材复用图片 Data URL 入参,需兼容较大的画板生成图。
.layer(DefaultBodyLimit::max(
EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(extract_editor_ui_design_assets).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
}
@@ -1,7 +1,5 @@
use axum::{
Router,
extract::DefaultBodyLimit,
middleware,
Router, middleware,
routing::{get, patch, post},
};
@@ -28,9 +26,6 @@ use crate::{
state::AppState,
};
const EXTERNAL_EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES: usize = 12 * 1024 * 1024;
const EXTERNAL_EDITOR_VIDEO_BODY_LIMIT_BYTES: usize = 64 * 1024 * 1024;
pub fn router(state: AppState) -> Router<AppState> {
Router::new()
.route("/api/external/v1/openapi.json", get(openapi_json))
@@ -141,69 +136,42 @@ pub fn router(state: AppState) -> Router<AppState> {
)
.route(
"/api/external/v1/editor/images/generations",
post(generate_external_editor_image)
.layer(DefaultBodyLimit::max(
EXTERNAL_EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
post(generate_external_editor_image).route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
)
.route(
"/api/external/v1/editor/images/edits",
post(edit_external_editor_image)
.layer(DefaultBodyLimit::max(
EXTERNAL_EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
post(edit_external_editor_image).route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
)
.route(
"/api/external/v1/editor/icon-spritesheets/generations",
post(generate_external_editor_icon_spritesheet)
.layer(DefaultBodyLimit::max(
EXTERNAL_EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
post(generate_external_editor_icon_spritesheet).route_layer(
middleware::from_fn_with_state(state.clone(), require_external_api_key),
),
)
.route(
"/api/external/v1/editor/ui-designs/assets/extractions",
post(extract_external_editor_ui_design_assets)
.layer(DefaultBodyLimit::max(
EXTERNAL_EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
post(extract_external_editor_ui_design_assets).route_layer(
middleware::from_fn_with_state(state.clone(), require_external_api_key),
),
)
.route(
"/api/external/v1/editor/character-animations/generations",
post(generate_external_editor_character_animation)
.layer(DefaultBodyLimit::max(
EXTERNAL_EDITOR_IMAGE_REFERENCE_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
post(generate_external_editor_character_animation).route_layer(
middleware::from_fn_with_state(state.clone(), require_external_api_key),
),
)
.route(
"/api/external/v1/editor/videos/generations",
post(generate_external_editor_video)
.layer(DefaultBodyLimit::max(
EXTERNAL_EDITOR_VIDEO_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
post(generate_external_editor_video).route_layer(middleware::from_fn_with_state(
state.clone(),
require_external_api_key,
)),
)
.route(
"/api/external/v1/editor/audios/sound-effects/generations",
@@ -52,8 +52,6 @@ use crate::{
};
const HYPER3D_IMAGE_TO_MODEL_BODY_LIMIT_BYTES: usize = 56 * 1024 * 1024;
const EDITOR_CHARACTER_ANIMATION_BODY_LIMIT_BYTES: usize = 12 * 1024 * 1024;
const EDITOR_VIDEO_BODY_LIMIT_BYTES: usize = 64 * 1024 * 1024;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct PlayFlowDomainAdapter {
@@ -460,24 +458,17 @@ fn play_flow_support_router(state: AppState) -> Router<AppState> {
)
.route(
"/api/editor/character-animations/generations",
post(generate_editor_character_animation)
.layer(DefaultBodyLimit::max(
// 中文注释:画板角色动画首版仍兼容角色图 Data URL 入参,避免大于 Axum 默认 2MB 的角色图在 handler 前被拦截。
EDITOR_CHARACTER_ANIMATION_BODY_LIMIT_BYTES,
))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(generate_editor_character_animation).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/videos/generations",
post(generate_editor_video)
.layer(DefaultBodyLimit::max(EDITOR_VIDEO_BODY_LIMIT_BYTES))
.route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
post(generate_editor_video).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/editor/audios/sound-effects/generations",
+14 -10
View File
@@ -1201,7 +1201,7 @@ mod tests {
}
#[test]
fn editor_video_request_supports_seedance_multimodal_references() {
fn editor_video_request_supports_stable_seedance_multimodal_references() {
let payload = serde_json::to_value(EditorVideoGenerateRequest {
prompt: "参考素材生成视频".to_string(),
model: "seedance2.0-fast".to_string(),
@@ -1211,11 +1211,15 @@ mod tests {
mode: "std".to_string(),
sound: "off".to_string(),
web_search_enabled: true,
reference_image_srcs: vec!["data:image/png;base64,image".to_string()],
reference_image_srcs: vec![
"generated-character-drafts/editor/seedance-references/image/demo.png".to_string(),
],
reference_video_srcs: vec![
"generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(),
],
reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()],
reference_audio_srcs: vec![
"generated-character-drafts/editor/seedance-references/audio/demo.mp3".to_string(),
],
project_id: None,
canvas_completion: None,
generation_inputs: None,
@@ -1228,7 +1232,7 @@ mod tests {
assert_eq!(
payload["referenceImageSrcs"],
json!(["data:image/png;base64,image"])
json!(["generated-character-drafts/editor/seedance-references/image/demo.png"])
);
assert_eq!(
payload["referenceVideoSrcs"],
@@ -1236,7 +1240,7 @@ mod tests {
);
assert_eq!(
payload["referenceAudioSrcs"],
json!(["data:audio/mpeg;base64,audio"])
json!(["generated-character-drafts/editor/seedance-references/audio/demo.mp3"])
);
}
@@ -1288,8 +1292,8 @@ mod tests {
#[test]
fn editor_icon_spritesheet_request_omits_price_and_response_includes_price() {
let request_payload = serde_json::to_value(EditorIconSpritesheetGenerateRequest {
reference_image_src: "data:image/png;base64,spec".to_string(),
reference_image_srcs: vec!["data:image/png;base64,style".to_string()],
reference_image_src: "generated-images/editor/spec.png".to_string(),
reference_image_srcs: vec!["generated-images/editor/style.png".to_string()],
icon_descriptions: vec!["返回按钮".to_string(), "设置按钮".to_string()],
model: Some("gemini-3.1-flash-image-preview".to_string()),
aspect_ratio: Some("1:1".to_string()),
@@ -1299,16 +1303,16 @@ mod tests {
assert_eq!(
request_payload["referenceImageSrc"],
json!("data:image/png;base64,spec")
json!("generated-images/editor/spec.png")
);
assert_eq!(
request_payload["referenceImageSrcs"],
json!(["data:image/png;base64,style"])
json!(["generated-images/editor/style.png"])
);
assert!(request_payload.get("priceMudPoints").is_none());
let response_payload = serde_json::to_value(EditorIconSpritesheetGenerateResponse {
spritesheet_image_src: "data:image/png;base64,sheet".to_string(),
spritesheet_image_src: "/generated-images/editor/sheet.png".to_string(),
spritesheet_width: 512,
spritesheet_height: 512,
icon_image_srcs: Vec::new(),
@@ -411,7 +411,6 @@ export const SEEDANCE_VIDEO_REFERENCE_LIMITS = {
image: 9,
video: 3,
audio: 3,
requestBytes: 64 * 1024 * 1024,
} as const;
export const QUICK_EDIT_REFERENCE_LIMIT = 8;
export const CHARACTER_ANIMATION_ACTION_PROMPTS = [
@@ -91,11 +91,6 @@ export function resolveImageReferenceSubmissionSource(reference: {
return reference.objectKey?.trim() || reference.src;
}
function estimateReferenceRequestBytes(references: Array<{ src: string }>) {
// 中文注释:JSON 包装字段还有少量额外字节;这里按 src 字符串长度估算,前后端都会再兜底校验 64MB。
return references.reduce((sum, reference) => sum + reference.src.length, 0);
}
function buildSeedanceVideoReferenceInput(
references: NonNullable<GenerateDialogState['generationReferences']>,
) {
@@ -129,13 +124,6 @@ function buildSeedanceVideoReferenceInput(
) {
throw new Error('参考音频必须搭配参考图片或参考视频');
}
if (
estimateReferenceRequestBytes(references) >
SEEDANCE_VIDEO_REFERENCE_LIMITS.requestBytes
) {
throw new Error('参考文件总请求体不能超过 64MB');
}
return {
referenceImageSrcs: imageReferences.map(resolveReferenceSource),
referenceVideoSrcs: videoReferences.map(resolveReferenceSource),
@@ -537,6 +537,31 @@ describe('useImageCanvasGenerationSubmissionWorkflow', () => {
expect(firstOptions.pathSegments).not.toEqual(secondOptions.pathSegments);
});
it('uploads image references without an object reference before generation', async () => {
resolveEditorImageReferenceDataUrlMock.mockResolvedValueOnce(
'data:image/png;base64,ZXhhbXBsZQ==',
);
const result = await resolveEditorGenerationMediaReference(
{ src: '/creation-type-references/example.webp' },
'image',
'project-1',
);
expect(resolveEditorImageReferenceDataUrlMock).toHaveBeenCalledWith(
'/creation-type-references/example.webp',
);
expect(uploadEditorMediaAssetFileMock).toHaveBeenCalledWith(
expect.any(File),
'image',
expect.objectContaining({
assetKind: 'editor_generation_reference_image',
}),
);
expect(result).toBe(
'generated-character-drafts/editor/generation-references/reference.png',
);
});
it('submits quick edits and updates the source layer directly', async () => {
editEditorImageMock.mockResolvedValueOnce(
createGenerated({
@@ -218,20 +218,27 @@ export async function resolveEditorGenerationMediaReference(
projectId?: string | null,
options: EditorGenerationMediaReferenceOptions = {},
) {
const resourceId = reference.resourceId?.trim();
const hasRegisteredReference =
options.allowRegisteredIds !== false &&
((resourceId &&
!resourceId.startsWith('local-resource-') &&
!resourceId.startsWith('generation-dialog:')) ||
reference.sourceAssetId?.trim());
const source = resolveEditorGenerationMediaReferenceSource(
reference,
options.allowRegisteredIds !== false,
);
const inlineSource = /^(?:data|blob):/iu.test(source);
const remoteImageSource =
options.requireImageObjectReference === true &&
const imageSourceRequiresUpload =
options.requireImageObjectReference !== false &&
mediaType === 'image' &&
!reference.objectKey?.trim() &&
/^(?:https?:|\/api\/)/iu.test(source);
if (!inlineSource && !remoteImageSource) {
!hasRegisteredReference;
if (!inlineSource && !imageSourceRequiresUpload) {
return source;
}
const uploadSource = remoteImageSource
const uploadSource = imageSourceRequiresUpload && !inlineSource
? await resolveEditorImageReferenceDataUrl(source)
: source;
return uploadEditorGenerationInlineMediaSource(
@@ -28,9 +28,8 @@ function encodeBytesAsBase64(bytes: Uint8Array) {
}
/**
* 图片画布的生成编辑接口只接收图片 Data URL
* 画布里既有上传 / 生成图,也有内置 public 参考图和历史 generated 路径;
* 提交给后端前统一读取成 Data URL,避免后端误收到普通 URL 后进入 Data URL 校验错误。
* 将画布图片读取为浏览器临时 Data URL,供压缩、标注等本地处理使用
* 生成请求提交前必须再上传 OSS,并把 objectKey 传给后端;Data URL 不属于接口契约。
*/
export async function resolveEditorImageReferenceDataUrl(
source: string,
@@ -121,7 +120,7 @@ export async function compressEditorImageReferenceDataUrlForGeneration(
return dataUrl;
}
// 中文注释:参考图只用于模型理解视觉方向,提交前压缩可避免原图 Data URL 撑爆 JSON 请求体
// 中文注释:参考图只用于模型理解视觉方向,上传 OSS 前压缩可减少传输与存储体积
if (mimeType === 'image/jpeg') {
context.fillStyle = '#fff';
context.fillRect(0, 0, targetWidth, targetHeight);
@@ -18,8 +18,8 @@ import {
generateEditorVideo,
listEditorProjects,
listPublicEditorProjectResources,
loadEditorGenerationPricing,
loadEditorAssetLibrary,
loadEditorGenerationPricing,
loadEditorProject,
loadOrCreateRecentEditorProject,
removeEditorImageBackground,
@@ -774,6 +774,47 @@ describe('editorProjectClient', () => {
);
});
it('rejects inline media before submitting editor generation requests', async () => {
await expect(
generateEditorImage({
prompt: '一张画布图片',
referenceImageSrcs: ['data:image/png;base64,inline'],
}),
).rejects.toThrow('必须先上传 OSS');
await expect(
generateEditorVideo({
prompt: '生成视频',
model: 'seedance2.0-fast',
aspectRatio: '16:9',
durationSeconds: 5,
resolution: '720p',
mode: 'std',
sound: 'off',
referenceVideoSrcs: ['blob:local-video'],
}),
).rejects.toThrow('必须先上传 OSS');
expect(requestJsonMock).not.toHaveBeenCalled();
});
it('rejects oversized stable video reference fields before submission', async () => {
await expect(
generateEditorVideo({
prompt: '生成视频',
model: 'seedance2.0-fast',
aspectRatio: '16:9',
durationSeconds: 5,
resolution: '720p',
mode: 'std',
sound: 'off',
referenceVideoSrcs: [`https://assets.example.test/${'a'.repeat(256 * 1024)}`],
}),
).rejects.toThrow('稳定引用字段总长度不能超过 256KB');
expect(requestJsonMock).not.toHaveBeenCalled();
});
it('passes image model options to editor image generation', async () => {
requestJsonMock.mockResolvedValueOnce({
imageSrc: 'data:image/png;base64,character',
@@ -919,8 +960,8 @@ describe('editorProjectClient', () => {
});
const result = await generateEditorIconSpritesheet({
referenceImageSrc: 'data:image/png;base64,spec',
referenceImageSrcs: ['data:image/png;base64,icon-ref'],
referenceImageSrc: '/generated-images/editor/spec.png',
referenceImageSrcs: ['/generated-images/editor/icon-ref.png'],
iconDescriptions: ['返回按钮', '设置按钮'],
assetLabel: '冒险游戏图标',
});
@@ -936,8 +977,8 @@ describe('editorProjectClient', () => {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
referenceImageSrc: 'data:image/png;base64,spec',
referenceImageSrcs: ['data:image/png;base64,icon-ref'],
referenceImageSrc: '/generated-images/editor/spec.png',
referenceImageSrcs: ['/generated-images/editor/icon-ref.png'],
iconDescriptions: ['返回按钮', '设置按钮'],
model: 'gemini-3.1-flash-image-preview',
assetLabel: '冒险游戏图标',
@@ -965,7 +1006,7 @@ describe('editorProjectClient', () => {
});
await generateEditorIconSpritesheet({
referenceImageSrc: 'data:image/png;base64,spec',
referenceImageSrc: '/generated-images/editor/spec.png',
iconDescriptions: ['返回按钮'],
model: 'gpt-image-2',
screenColor: '#E6D8FF',
@@ -986,7 +1027,7 @@ describe('editorProjectClient', () => {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
referenceImageSrc: 'data:image/png;base64,spec',
referenceImageSrc: '/generated-images/editor/spec.png',
iconDescriptions: ['返回按钮'],
model: 'gpt-image-2',
screenColor: '#E6D8FF',
@@ -1105,11 +1146,11 @@ describe('editorProjectClient', () => {
});
const result = await extractEditorUiDesignAssets({
sourceImageSrc: 'data:image/png;base64,ui-design',
sourceImageSrc: '/generated-images/editor/ui-design.png',
model: 'gpt-image-2',
screenColor: '#7FB3FF',
segModel: 'anime-seg',
referenceImageSrcs: ['data:image/png;base64,ui-ref'],
referenceImageSrcs: ['/generated-images/editor/ui-ref.png'],
projectId: 'editor-project-1',
assetFolderId: 'project',
spritesheetLabel: '战斗UI设计图 素材图集',
@@ -1128,11 +1169,11 @@ describe('editorProjectClient', () => {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
sourceImageSrc: 'data:image/png;base64,ui-design',
sourceImageSrc: '/generated-images/editor/ui-design.png',
model: 'gpt-image-2',
screenColor: '#7FB3FF',
segModel: 'anime-seg',
referenceImageSrcs: ['data:image/png;base64,ui-ref'],
referenceImageSrcs: ['/generated-images/editor/ui-ref.png'],
projectId: 'editor-project-1',
generationInputs: {
fields: [{ title: '来源', value: '战斗UI设计图' }],
@@ -1212,7 +1253,7 @@ describe('editorProjectClient', () => {
kind: 'publication-material',
aspectRatio: '9:16',
imageSize: '1K',
referenceImageSrcs: ['data:image/png;base64,ref'],
referenceImageSrcs: ['/generated-images/editor/publication-ref.png'],
});
expect(result.height).toBe(1280);
@@ -1227,7 +1268,7 @@ describe('editorProjectClient', () => {
kind: 'publication-material',
aspectRatio: '9:16',
imageSize: '1K',
referenceImageSrcs: ['data:image/png;base64,ref'],
referenceImageSrcs: ['/generated-images/editor/publication-ref.png'],
}),
}),
'生成图片失败',
@@ -1259,7 +1300,7 @@ describe('editorProjectClient', () => {
const result = await generateEditorCharacterAnimation({
sourceLayerId: 'layer-character',
sourceImageSrc: 'data:image/png;base64,character',
sourceImageSrc: '/generated-images/editor/character.png',
sourceWidth: 1024,
sourceHeight: 1024,
promptText: '待机',
@@ -1279,7 +1320,7 @@ describe('editorProjectClient', () => {
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
sourceLayerId: 'layer-character',
sourceImageSrc: 'data:image/png;base64,character',
sourceImageSrc: '/generated-images/editor/character.png',
sourceWidth: 1024,
sourceHeight: 1024,
promptText: '待机',
@@ -1395,7 +1436,7 @@ describe('editorProjectClient', () => {
resolution: '720p',
mode: 'std',
sound: 'off',
referenceImageSrcs: ['data:image/png;base64,image'],
referenceImageSrcs: ['/generated-images/editor/video-reference.png'],
referenceVideoSrcs: [
'generated-character-drafts/editor/seedance-references/video/video.mp4',
],
@@ -1418,7 +1459,7 @@ describe('editorProjectClient', () => {
mode: 'std',
sound: 'off',
webSearchEnabled: true,
referenceImageSrcs: ['data:image/png;base64,image'],
referenceImageSrcs: ['/generated-images/editor/video-reference.png'],
referenceVideoSrcs: [
'generated-character-drafts/editor/seedance-references/video/video.mp4',
],
@@ -1578,12 +1619,12 @@ describe('editorProjectClient', () => {
const result = await editEditorImage({
prompt: '把画面改成黄昏光线',
sourceImageSrc: 'data:image/png;base64,source',
sourceImageSrc: '/generated-images/editor/source.png',
size: '1024x1024',
model: 'gpt-image-2',
aspectRatio: '1:1',
imageSize: '1K',
referenceImageSrcs: ['data:image/png;base64,style'],
referenceImageSrcs: ['/generated-images/editor/style.png'],
projectId: 'editor-project-1',
assetKind: 'character',
assetFolderId: 'project',
@@ -1604,12 +1645,12 @@ describe('editorProjectClient', () => {
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
prompt: '把画面改成黄昏光线',
sourceImageSrc: 'data:image/png;base64,source',
sourceImageSrc: '/generated-images/editor/source.png',
size: '1024x1024',
model: 'gpt-image-2',
aspectRatio: '1:1',
imageSize: '1K',
referenceImageSrcs: ['data:image/png;base64,style'],
referenceImageSrcs: ['/generated-images/editor/style.png'],
projectId: 'editor-project-1',
assetKind: 'character',
generationInputs: {
@@ -1643,7 +1684,7 @@ describe('editorProjectClient', () => {
});
const response = await removeEditorImageBackground({
sourceImageSrc: 'data:image/png;base64,source',
sourceImageSrc: '/generated-images/editor/source.png',
projectId: 'editor-project-1',
targetLayerId: 'layer-source',
canvasCompletion: {
@@ -1668,7 +1709,7 @@ describe('editorProjectClient', () => {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
sourceImageSrc: 'data:image/png;base64,source',
sourceImageSrc: '/generated-images/editor/source.png',
projectId: 'editor-project-1',
targetLayerId: 'layer-source',
canvasCompletion: {
@@ -1,5 +1,5 @@
import { requestJson } from '../apiClient';
import type { ExternalGenerationJobStatusRecord } from '../../../packages/shared/src/contracts/externalGeneration';
import { requestJson } from '../apiClient';
import { EDITOR_REQUEST_RETRY_OPTIONS } from './editorRetryOptions';
const EDITOR_PROJECT_API_BASE = '/api/editor/projects';
@@ -25,8 +25,38 @@ const EDITOR_BACKGROUND_MUSIC_GENERATION_API =
'/api/editor/audios/background-music/generations';
const EDITOR_GENERATION_PRICING_API = '/api/editor/generation-pricing';
const EDITOR_IMAGE_MODEL_NANOBANANA2 = 'gemini-3.1-flash-image-preview';
const EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES = 256 * 1024;
const DEFAULT_PROJECT_TITLE = '未命名画布';
function assertStableEditorMediaReference(value: string, fieldLabel: string) {
const normalized = value.trimStart().toLowerCase();
if (normalized.startsWith('data:') || normalized.startsWith('blob:')) {
throw new Error(
`${fieldLabel}必须先上传 OSS,并使用 objectKey、项目资源、素材 ID 或支持的 URL 引用。`,
);
}
}
function assertStableEditorMediaReferences(
values: readonly string[] | undefined,
fieldLabel: string,
) {
values?.forEach((value) => assertStableEditorMediaReference(value, fieldLabel));
}
function assertStableEditorVideoReferences(input: EditorVideoGenerationInput) {
const references = [
...(input.referenceImageSrcs ?? []),
...(input.referenceVideoSrcs ?? []),
...(input.referenceAudioSrcs ?? []),
];
assertStableEditorMediaReferences(references, '视频参考素材');
const requestBytes = new TextEncoder().encode(references.join('')).byteLength;
if (requestBytes > EDITOR_VIDEO_REFERENCE_REQUEST_LIMIT_BYTES) {
throw new Error('视频参考素材稳定引用字段总长度不能超过 256KB');
}
}
export type EditorCanvasViewport = {
x: number;
y: number;
@@ -874,6 +904,7 @@ export async function toggleEditorShowcaseAssetLike(
}
export async function generateEditorImage(input: EditorImageGenerationInput) {
assertStableEditorMediaReferences(input.referenceImageSrcs, '生成参考图');
return requestJson<EditorImageGenerationResponse>(
EDITOR_IMAGE_GENERATION_API,
jsonRequest('POST', {
@@ -913,6 +944,11 @@ export async function generateEditorImage(input: EditorImageGenerationInput) {
export async function generateEditorIconSpritesheet(
input: EditorIconSpritesheetGenerationInput,
) {
assertStableEditorMediaReference(input.referenceImageSrc, '图标素材规范');
assertStableEditorMediaReferences(
input.referenceImageSrcs,
'图标素材参考图',
);
return requestJson<EditorIconSpritesheetGenerationResponse>(
EDITOR_ICON_SPRITESHEET_GENERATION_API,
jsonRequest('POST', {
@@ -962,6 +998,8 @@ export async function splitEditorIconSpritesheet(
export async function extractEditorUiDesignAssets(
input: EditorUiDesignAssetExtractionInput,
) {
assertStableEditorMediaReference(input.sourceImageSrc, 'UI设计图');
assertStableEditorMediaReferences(input.referenceImageSrcs, 'UI素材参考图');
return requestJson<EditorIconSpritesheetGenerationResponse>(
EDITOR_UI_DESIGN_ASSET_EXTRACTION_API,
jsonRequest('POST', {
@@ -995,6 +1033,8 @@ export async function extractEditorUiDesignAssets(
}
export async function editEditorImage(input: EditorImageEditInput) {
assertStableEditorMediaReference(input.sourceImageSrc, '待修改图片');
assertStableEditorMediaReferences(input.referenceImageSrcs, '修改参考图');
return requestJson<EditorImageGenerationResponse>(
EDITOR_IMAGE_EDIT_API,
jsonRequest('POST', {
@@ -1033,6 +1073,7 @@ export async function editEditorImage(input: EditorImageEditInput) {
export async function removeEditorImageBackground(
input: EditorBackgroundRemovalInput,
) {
assertStableEditorMediaReference(input.sourceImageSrc, '待去除背景图片');
return requestJson<EditorBackgroundRemovalResponse>(
EDITOR_BACKGROUND_REMOVAL_API,
jsonRequest('POST', {
@@ -1064,6 +1105,7 @@ export async function removeEditorImageBackground(
export async function generateEditorCharacterAnimation(
input: EditorCharacterAnimationGenerationInput,
) {
assertStableEditorMediaReference(input.sourceImageSrc, '角色动画源图');
return requestJson<EditorCharacterAnimationGenerationResult>(
EDITOR_CHARACTER_ANIMATION_GENERATION_API,
jsonRequest('POST', {
@@ -1088,6 +1130,7 @@ export async function generateEditorCharacterAnimation(
}
export async function generateEditorVideo(input: EditorVideoGenerationInput) {
assertStableEditorVideoReferences(input);
return requestJson<EditorVideoGenerationResponse>(
EDITOR_VIDEO_GENERATION_API,
jsonRequest('POST', {