diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index b354820bb..a97df3eb8 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -16,6 +16,16 @@ --- +## 2026-07-23 画布 Agent 工具生命周期统一经 object-safe trait 分派 + +- 背景：画布 Agent 八类工具的参数规范化、确认展示、计价与 worker payload、完成结果格式化和媒体投影分别在 `tool_args.rs`、`display_args.rs`、`api.rs`、`reconcile.rs` 重复按工具名分派；新增或调整工具时容易漏改其中一处。 +- 决策：api-server 以 object-safe `EditorAgentTool: ToolDyn` 取代仅承载计价的 `EditorAgentPricedTool`。trait 的所有动态方法统一接收 `serde_json::Value`；每个具体工具实现自行反序列化为真实 Args / 结果，`validate_args` 与 `format_execute_message` 显式转发到 `platform-editor-agent` 已有强类型实现，再把规范 Args、展示投影、job payload、完成文本或媒体引用擦除回公共类型。`editor_agent_tool(toolName, context)` 绑定当前 `EditorToolContext` 并作为唯一八分支工具名分派；规划、确认和回填不得再维护平行 switch。LLM builder 的工具注册列表保持独立显式维护。 +- 边界：不改变工具名、LLM schema、OSS 消息文档、`displayArgs`、模型定价、job kind / payload、dedupe key、worker、计费、完成消息或图片 / 视频 / 音频引用契约，不涉及前端、SpacetimeDB schema 或迁移。 +- 影响范围：`server-rs/crates/api-server/src/editor_agent` 的工具 trait、参数规范化、确认入队与终态回填，以及画布 Agent 专题文档。 +- 验证方式：覆盖八类 factory 与 dyn validation / pricing / display / job / formatter / media projection 的 api-server 定向测试，运行 `cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_agent`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`npm run check:rustfmt`、`npm run check:encoding` 和 `git diff --check`。 + +--- + ## 2026-07-20 角色动作抠图前禁止透明 padding - 背景：图片画布角色动作此前在 BgFilter 前复用最终帧 finalizer，把 FFmpeg 抽帧先转成目标尺寸 RGBA 画布并用透明黑像素补边；透明区域进入 BgFilter、阿里云和本地键色共同读取的 OSS 源帧后，会干扰主体边缘判断并降低抠图质量。 diff --git a/docs/【编辑器】画布Agent对话面板-2026-07-03.md b/docs/【编辑器】画布Agent对话面板-2026-07-03.md index 16ce2f33c..7bb88fc6f 100644 --- a/docs/【编辑器】画布Agent对话面板-2026-07-03.md +++ b/docs/【编辑器】画布Agent对话面板-2026-07-03.md @@ -85,12 +85,13 @@ - 确认接口必须先把工具参数转换为既有编辑器 worker payload，再使用 `editor-agent:{conversationId}:{messageId}:{toolName}` 稳定 dedupe key 入队；同一确认的请求重试只能得到同一个 external job。入队成功后把返回的 job id 写回同一条 OSS 工具消息，不新增 Agent 工具执行关联表。 - 前端根据 `externalJobId` 查询通用 external-generation job 状态；worker 继续通过 `canvasCompletion` 把生成结果写回工程与素材库。浏览器断线、刷新或 api-server 重启不得导致确认接口重新扣费或重新提交 provider。 - `GET /conversation` 会在同一个 conversation lock 内扫描 `status=not_completed` 且已有 `externalJobId` 的工具消息：只对这些消息按 job id 定向读取主任务；任务完成后复用对应工具的 `format_execute_message` 替换 system text、回填轻量图片 / 视频 / 音频引用并写为 `completed`，任务失败则回填 `error` 并写为 `failed`。任务结果读取或 completed payload 解析 / formatter 回填失败时，必须在同一次 GET 内完成首次尝试及最多 3 次重试，三次重试各间隔 100ms 并重新读取任务结果；仍失败才把该工具消息写为 `failed` 并保存最后错误。该重试不依赖前端再次刷新。排队和执行中都保持 `not_completed`，整轮扫描结果一次性写回 OSS。 -- `EditorAgentToolCall.args` 的正式持久化契约是**校验后的规范参数 JSON**，不是 LLM 返回的原始 JSON。api-server 收到工具调用后，必须先按已注册的 ToolArgs 反序列化、补齐字段默认值、删除未进入 ToolArgs 的未知 / 退役字段、执行工具参数校验，再重新序列化并写入 `args`；校验失败的调用不得持久化为待确认消息。LLM 原始参数只作为本次规范化的瞬时输入，不作为执行或审计真相；确认、取消、任务回填与后续上下文统一读取同一条消息中的规范 `args`。图片参数继续只保存由真实 data key 计算出的 opaque SHA-256 `imageId`；不得为了前端预览把 `args` 中的图片 ID 改写成 `objectKey`、URL 或展示对象，也不得由前端重组或回传一份新的执行参数。 +- `EditorAgentToolCall.args` 的正式持久化契约是**校验后的规范参数 JSON**，不是 LLM 返回的原始 JSON。api-server 收到工具调用后，必须先按已注册的 ToolArgs 反序列化、补齐字段默认值、删除未进入 ToolArgs 的未知 / 退役字段、执行工具参数校验，再重新序列化并写入 `args`；校验失败的调用不得持久化为待确认消息。所有有明确默认值的工具标量参数在强类型 ToolArgs 中必须使用非 `Option` 字段：调用方省略时由 Serde 补默认值，显式 `null` 视为非法参数，后续计价、确认展示和 job payload 不得再次使用 `unwrap_or` 补同一默认值。LLM 原始参数只作为本次规范化的瞬时输入，不作为执行或审计真相；确认、取消、任务回填与后续上下文统一读取同一条消息中的规范 `args`。图片参数继续只保存由真实 data key 计算出的 opaque SHA-256 `imageId`；不得为了前端预览把 `args` 中的图片 ID 改写成 `objectKey`、URL 或展示对象，也不得由前端重组或回传一份新的执行参数。 +- api-server 内画布 Agent 工具统一实现 object-safe `EditorAgentTool: ToolDyn`。`validate_args`、计价、确认展示、worker job 构建、`format_execute_message` 和结果媒体投影都使用统一 JSON 边界；每个具体工具实现负责把 JSON 反序列化为自己的强类型 Args / 结果，并把校验与完成消息格式化转发到 `platform-editor-agent` 中既有的 typed `validate_args` / `format_execute_message`，不得在调用方复制工具规则。`editor_agent_tool(toolName, context)` 是唯一按工具名分派的位置，规划、确认和任务回填只调用返回的 dyn tool；新增工具必须补齐同一个 trait 实现和该工厂分支。LLM builder 的 `.tool(...)` 注册列表仍是独立显式清单，不属于本次动态分派。framework runner 必须在 `ToolCallOutput` 中保留工具返回的结构化 output；runner 写入 LLM memory 与 api-server 使用规范参数持久化 system text 时统一调用公开的 `format_tool_call_message`，不得丢弃 `TOOL_CALL_PENDING_MESSAGE` 后自行拼另一套“等待确认”输出。 - `EditorAgentToolCall.displayArgs` 是必填、只读的用户确认展示投影，与 `args` 分离： - `stringArgs` 保存提示词、比例、清晰度、模型、时长等可展示参数的稳定名称、用户可见标题和值；前端渲染模型字段时复用图片编辑器公共展示名映射，`gemini-3.1-flash-image-preview` 显示为 `nanobanana2`、`audio1.0` 显示为 `Vidu`、`chirp-v5` 显示为 `Suno`，视频模型显示现有产品标签，不得改写后端参数真相； - `imageArgs` 按“目标图片 / 参考图片”等参数分组，每个 `refs` 项包含与规范参数对应的 `imageId`，以及后端从已校验会话上下文解析出的 `objectKey`、`imageSrc`、可选 `thumbnailSrc` / `label` / `width` / `height`。 - `extras.priceMudPoints` 保存创建待确认消息时按后端运行时模型定价快照计算的预计泥点消耗；前端统一展示为“预计消耗 N泥点”，不自行计算价格。 -- `displayArgs` 只能由 api-server 按已注册 tool 白名单，基于已经通过 ToolArgs 校验的 `args` 和当前 OSS 会话文档中的附件 / 历史生成结果构建；不能信任 LLM 自报的展示地址、标题或素材元数据。展示投影不参与确认执行，确认接口仍只读取同一条持久化 tool call 的 `args`，避免“看到的素材”和“实际执行的素材”分叉。 +- `displayArgs` 只能由 api-server 按已注册 tool 白名单，基于已经通过 ToolArgs 校验的 `args` 和当前请求开始时从 OSS 会话文档一次性构建的 `EditorToolContext` 生成；该 context 必须按 opaque `ImageId` 同时保存执行所需的 `dataKey` 与展示所需的图片地址、Object Key、缩略图、label、宽高，参数校验、确认展示和 job payload 统一查同一份 context。不能信任 LLM 自报的展示地址、标题或素材元数据。展示投影不参与确认执行，确认接口仍只读取同一条持久化 tool call 的 `args`，避免“看到的素材”和“实际执行的素材”分叉。 - `extras.priceMudPoints` 同样只属于展示投影，不作为扣费输入；确认后仍由既有生成 BFF 按后端运行时定价执行预扣费，因此该字段表达用户确认时看到的价格快照，而不是前端可提交或覆盖的计费真相。 - `EditorAgentToolCall.summary` 只是 `args` 的重复字符串且没有稳定语义，当前契约删除该字段，不再作为展示或执行输入。 - 前端待确认卡只消费必填 `displayArgs`，不解析各 tool 私有的 snake_case / camelCase schema，也不把 `sha256:*` ID 当标题或图片地址。图片统一通过 `ResolvedAssetImage` 使用 `objectKey` 换签后显示，签名 URL 不进入消息文档。模块尚未上线，不保留缺少 `displayArgs` 时读取 raw `args` 的旧消息降级路径。 diff --git a/server-rs/crates/api-server/src/editor_agent/api.rs b/server-rs/crates/api-server/src/editor_agent/api.rs index 655435a79..7b448bd0a 100644 --- a/server-rs/crates/api-server/src/editor_agent/api.rs +++ b/server-rs/crates/api-server/src/editor_agent/api.rs @@ -10,16 +10,11 @@ use module_editor_agent::{ use platform_editor_agent::framework::agent_builder::AgentBuilder; use platform_editor_agent::framework::error::PromptError; use platform_editor_agent::framework::memory::VecMemory; -use platform_editor_agent::framework::run::PromptOutput; -use platform_editor_agent::framework::tool::Tool; +use platform_editor_agent::framework::run::{PromptOutput, format_tool_call_message}; use platform_llm::LlmMessage; use serde::Serialize; use serde_json::{Value, json}; use sha2::{Digest, Sha256}; -use shared_contracts::assets::{ - EditorBackgroundMusicGenerateRequest, EditorSoundEffectGenerateRequest, - EditorVideoGenerateRequest, -}; use shared_contracts::editor_agent::{ CreateEditorAgentConversationRequest, EDITOR_AGENT_ERROR_MESSAGE_PREFIX, EditorAgentConversationListResponse, EditorAgentConversationMessagesDocument, @@ -35,52 +30,32 @@ use spacetime_client::{ use crate::api_response::json_success_body; use crate::auth::AuthenticatedAccessToken; -use crate::editor_agent::pricing::EditorAgentPricedTool; +use crate::editor_agent::tool::{ + EditorAgentPrepareJobContext, EditorAgentToolError, editor_agent_tool, +}; use crate::editor_agent::utils::{ - IntoImageId, build_editor_agent_canvas_completion, conversation_detail_from_record, - conversation_summary_from_record, editor_agent_bad_request, empty_messages_document, - ensure_editor_project_access, normalize_editor_agent_attachments, now_rfc3339, - read_messages_document, require_editor_agent_sidebar_enabled, write_messages_document, + IntoImageId, conversation_detail_from_record, conversation_summary_from_record, + editor_agent_bad_request, empty_messages_document, ensure_editor_project_access, + normalize_editor_agent_attachments, now_rfc3339, read_messages_document, + require_editor_agent_sidebar_enabled, write_messages_document, }; -use crate::editor_agent::{context, display_args, reconcile, tool_args}; +use crate::editor_agent::{context, reconcile}; use crate::editor_generation_config::EditorGenerationPricingConfig; -use crate::editor_generation_queue::{ - EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, - EDITOR_IMAGE_EDIT_JOB_KIND, EDITOR_IMAGE_GENERATION_JOB_KIND, - EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, EDITOR_VIDEO_GENERATION_JOB_KIND, - enqueue_editor_generation_job_with_identity, -}; -use crate::editor_project::{ - EditorIconSpritesheetGenerationRequest, EditorImageEditRequest, EditorImageGenerationRequest, -}; +use crate::editor_generation_queue::enqueue_editor_generation_job_with_identity; use crate::editor_project::{current_utc_micros, map_editor_project_error}; use crate::http_error::AppError; use crate::request_context::RequestContext; use crate::state::AppState; use platform_editor_agent::agent::agent::LlmChatAgentBuilder; use platform_editor_agent::agent::tools::context::EditorToolContext; -use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; -use platform_editor_agent::agent::tools::generate_background_music::{ - GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, -}; -use platform_editor_agent::agent::tools::generate_character::{ - GenerateCharacterTool, GenerateCharacterToolArgs, -}; -use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ - GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, -}; -use platform_editor_agent::agent::tools::generate_image::{ - GenerateImageTool, GenerateImageToolArgs, -}; -use platform_editor_agent::agent::tools::generate_sound_effect::{ - GenerateSoundEffectTool, GenerateSoundEffectToolArgs, -}; -use platform_editor_agent::agent::tools::generate_ui_design::{ - GenerateUiDesignTool, GenerateUiDesignToolArgs, -}; -use platform_editor_agent::agent::tools::generate_video::{ - GenerateVideoTool, GenerateVideoToolArgs, -}; +use platform_editor_agent::agent::tools::edit_image::EditImageTool; +use platform_editor_agent::agent::tools::generate_background_music::GenerateBackgroundMusicTool; +use platform_editor_agent::agent::tools::generate_character::GenerateCharacterTool; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::GenerateIconSpritesheetTool; +use platform_editor_agent::agent::tools::generate_image::GenerateImageTool; +use platform_editor_agent::agent::tools::generate_sound_effect::GenerateSoundEffectTool; +use platform_editor_agent::agent::tools::generate_ui_design::GenerateUiDesignTool; +use platform_editor_agent::agent::tools::generate_video::GenerateVideoTool; use shared_kernel::{build_prefixed_uuid_id, normalize_optional_string, normalize_required_string}; use tokio::time::{Instant, timeout}; @@ -309,7 +284,6 @@ pub async fn editor_agent_message( agent_result, &assistant_now, document.messages.len(), - &document, &tool_context, &pricing, ) { @@ -455,6 +429,8 @@ fn editor_agent_attachment_requests_match( #[cfg(test)] mod tests { use super::*; + use platform_editor_agent::framework::run::ToolCallOutput; + use platform_editor_agent::framework::tool::{Tool, ToolCall}; use shared_contracts::editor_agent::{EditorAgentAttachmentRef, EditorAgentAttachmentSource}; fn attachment(reference_id: impl Into) -> EditorAgentAttachmentRef { @@ -581,31 +557,39 @@ mod tests { } #[test] - fn legacy_confirm_args_use_current_defaults_and_ignore_retired_fields() { - let music: GenerateBackgroundMusicToolArgs = parse_confirm_tool_args(&json!({ - "prompt": "温暖舒缓的钢琴背景音乐", - "make_instrumental": false - })) - .expect("旧背景音乐字段不应阻断确认"); - assert_eq!(music.model, GenerateBackgroundMusicTool::DEFAULT_MODEL); + fn pending_tool_message_reuses_the_runner_output_and_shared_formatter() { + let tool_name = GenerateBackgroundMusicTool::NAME; + let output = json!({ "message": "runner pending output" }); + let pricing = + crate::editor_generation_config::load_editor_generation_pricing_from_paths(None) + .expect("default editor pricing should load"); + let messages = build_delta_messages( + Ok(vec![PromptOutput::Tool(ToolCallOutput { + tool_call: ToolCall { + id: "tool-call-1".to_string(), + name: tool_name.to_string(), + args: json!({ "prompt": "轻快冒险音乐" }), + }, + output: output.clone(), + })]), + "2026-07-23T00:00:00Z", + 0, + &EditorToolContext::default(), + &pricing, + ) + .expect("pending tool message should build"); - let ui: GenerateUiDesignToolArgs = parse_confirm_tool_args(&json!({ - "prompt": "生成游戏主界面" - })) - .expect("旧 UI 消息缺少 model 时应使用当前固定模型"); - assert_eq!(ui.model, platform_image::GPT_IMAGE_2_MODEL); - - let video: GenerateVideoToolArgs = parse_confirm_tool_args(&json!({ - "prompt": "镜头向前推进" - })) - .expect("旧视频消息缺少可选字段时应使用当前默认值"); - assert_eq!(video.sound.as_deref(), Some("on")); - - let sound: GenerateSoundEffectToolArgs = parse_confirm_tool_args(&json!({ - "prompt": "按钮点击声" - })) - .expect("旧音效消息缺少时长时应使用当前默认值"); - assert_eq!(sound.duration, Some(5)); + let tool_call = messages[0] + .tool_call + .as_ref() + .expect("pending message should retain its tool call"); + assert_eq!( + messages[0].text, + format_tool_call_message(tool_name, &tool_call.args, &output) + .expect("shared formatter should produce the persisted text") + ); + assert!(messages[0].text.contains("runner pending output")); + assert!(!messages[0].text.contains("等待用户确认")); } #[tokio::test] @@ -650,7 +634,6 @@ fn build_delta_messages( result: Result, PromptError>, created_at: &str, messages_offset: usize, - document: &EditorAgentConversationMessagesDocument, tool_context: &EditorToolContext, pricing: &EditorGenerationPricingConfig, ) -> Result, PromptError> { @@ -673,21 +656,21 @@ fn build_delta_messages( } PromptOutput::Tool(tco) => { let tool_name = tco.tool_call.name; - let normalized_args = tool_args::normalize_tool_args( - tool_name.as_str(), - &tco.tool_call.args, - tool_context, - )?; - let display_args = display_args::build_tool_call_display_args( - tool_name.as_str(), - &normalized_args, - document, - tool_context, - pricing, - )?; - // TODO duplicated with prompt inside tools + let tool = + editor_agent_tool(tool_name.as_str(), tool_context).ok_or_else(|| { + PromptError::ToolError(format!( + "unsupported editor agent tool: {tool_name}" + )) + })?; + let normalized_args = tool + .validate_args(&tco.tool_call.args) + .map_err(|error| error.into_prompt_error(tool_name.as_str()))?; + let display_args = tool + .build_display_args(&normalized_args, pricing) + .map_err(|error| error.into_prompt_error(tool_name.as_str()))?; + let text = - format!("[tool_call:{tool_name}] args: {normalized_args} output: 等待用户确认"); + format_tool_call_message(tool_name.as_str(), &normalized_args, &tco.output)?; messages.push(EditorAgentMessage { id: absolute_idx, client_message_id: None, @@ -979,385 +962,26 @@ pub async fn confirm_editor_agent_tool_call( })?; let project = load_editor_agent_project(&state, &conversation).await?; let context = context::build_tool_context(&document); - let generation_inputs = |title: &str, value: &str| { - Some(json!({ - // TODO extract const - "source": "editor-agent", - "conversationId": conversation.conversation_id, - "toolCallMessageId": message_id, - "fields": [{ "title": title, "value": value }], - })) - }; - - let (job_kind, request_label, price_mud_points, payload) = match tool_name.as_str() { - GenerateImageTool::NAME => { - let args: GenerateImageToolArgs = parse_confirm_tool_args(&tool_args)?; - let tool = GenerateImageTool { - context: context.clone(), - }; - tool.validate_args(&args).map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let price = tool.pricing(&pricing, &args); - let title = args.prompt.clone(); - let reference_image_srcs = args - .reference_image_ids - .iter() - .map(|image_id| { - context - .image_data_key(image_id) - .map(ToOwned::to_owned) - .ok_or_else(|| { - editor_agent_bad_request(format!("image {image_id} not found")) - }) - }) - .collect::, _>>()?; - let payload = EditorImageGenerationRequest { - prompt: args.prompt, - size: None, - kind: None, - model: Some(args.model), - screen_color: None, - seg_model: None, - aspect_ratio: args.aspect_ratio, - image_size: args.image_size, - reference_image_srcs: Some(reference_image_srcs), - project_id: Some(conversation.project_id.clone()), - asset_kind: Some("editor_agent_generated_image".to_string()), - generation_inputs: generation_inputs("用户指令", &title), - asset_folder_id: Some("project".to_string()), - asset_label: Some(title.clone()), - source_resource_id: None, - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, &tool_name, &title, - )), - }; - ( - EDITOR_IMAGE_GENERATION_JOB_KIND, - "画布 Agent 生成图片", - price, - serde_json::to_value(payload), - ) - } - GenerateCharacterTool::NAME => { - let args: GenerateCharacterToolArgs = parse_confirm_tool_args(&tool_args)?; - let tool = GenerateCharacterTool { - context: context.clone(), - }; - tool.validate_args(&args).map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let price = tool.pricing(&pricing, &args); - let title = args.prompt.clone(); - let reference_image_srcs = - resolve_editor_agent_image_ids(args.reference_image_ids.as_slice(), &context)?; - let payload = EditorImageGenerationRequest { - prompt: args.prompt, - size: None, - kind: Some("character".to_string()), - model: Some(args.model), - screen_color: Some("auto".to_string()), - seg_model: Some("birefnet".to_string()), - aspect_ratio: args.aspect_ratio, - image_size: args.image_size, - reference_image_srcs: Some(reference_image_srcs), - project_id: Some(conversation.project_id.clone()), - asset_kind: Some("character".to_string()), - generation_inputs: generation_inputs("用户指令", &title), - asset_folder_id: Some("project".to_string()), - asset_label: Some(title.clone()), - source_resource_id: None, - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, &tool_name, &title, - )), - }; - ( - EDITOR_IMAGE_GENERATION_JOB_KIND, - "画布 Agent 生成图片", - price, - serde_json::to_value(payload), - ) - } - GenerateUiDesignTool::NAME => { - let args: GenerateUiDesignToolArgs = parse_confirm_tool_args(&tool_args)?; - let tool = GenerateUiDesignTool { - context: context.clone(), - }; - tool.validate_args(&args).map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let price = tool.pricing(&pricing, &args); - let title = args.prompt.clone(); - let reference_image_srcs = - resolve_editor_agent_image_ids(args.reference_image_ids.as_slice(), &context)?; - let payload = EditorImageGenerationRequest { - prompt: args.prompt, - size: None, - kind: Some("ui-design".to_string()), - model: Some(args.model), - screen_color: None, - seg_model: None, - aspect_ratio: args.aspect_ratio, - image_size: args.image_size, - reference_image_srcs: Some(reference_image_srcs), - project_id: Some(conversation.project_id.clone()), - asset_kind: Some("ui-design".to_string()), - generation_inputs: generation_inputs("用户指令", &title), - asset_folder_id: Some("project".to_string()), - asset_label: Some(title.clone()), - source_resource_id: None, - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, &tool_name, &title, - )), - }; - ( - EDITOR_IMAGE_GENERATION_JOB_KIND, - "画布 Agent 生成图片", - price, - serde_json::to_value(payload), - ) - } - EditImageTool::NAME => { - let args: EditImageToolArgs = parse_confirm_tool_args(&tool_args)?; - let tool = EditImageTool { - context: context.clone(), - }; - if let Some(error) = tool.validate_args(&args) { - return Err(editor_agent_bad_request(format!( - "invalid tool call args: {error}" - ))); - } - let source_image_src = context - .image_data_key(&args.object_image_id) - .ok_or_else(|| editor_agent_bad_request("object image not found"))? - .to_string(); - let reference_image_srcs = args - .reference_image_ids - .iter() - .map(|image_id| { - context - .image_data_key(image_id) - .map(ToOwned::to_owned) - .ok_or_else(|| { - editor_agent_bad_request(format!("image {image_id} not found")) - }) - }) - .collect::, _>>()?; - let title = args.prompt.clone(); - let price = tool.pricing(&pricing, &args); - let payload = EditorImageEditRequest { - prompt: args.prompt, - source_image_src, - size: None, - model: Some(args.model), - aspect_ratio: None, - image_size: None, - reference_image_srcs: Some(reference_image_srcs), - project_id: Some(conversation.project_id.clone()), - asset_kind: Some("editor_agent_edit_image".to_string()), - generation_inputs: generation_inputs("用户指令", &title), - asset_folder_id: Some("project".to_string()), - asset_label: Some(title.clone()), - source_resource_id: None, - target_layer_id: None, - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, - EditImageTool::NAME, - &title, - )), - }; - ( - EDITOR_IMAGE_EDIT_JOB_KIND, - "画布 Agent 修改图片", - price, - serde_json::to_value(payload), - ) - } - GenerateIconSpritesheetTool::NAME => { - let args: GenerateIconSpritesheetToolArgs = parse_confirm_tool_args(&tool_args)?; - let tool = GenerateIconSpritesheetTool { - context: context.clone(), - }; - tool.validate_args(&args).map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let price = tool.pricing(&pricing, &args); - let reference_image_src = context - .image_data_key(&args.reference_image_id) - .ok_or_else(|| editor_agent_bad_request("reference image not found"))? - .to_string(); - let reference_image_srcs = args - .reference_image_ids - .iter() - .map(|image_id| { - context - .image_data_key(image_id) - .map(ToOwned::to_owned) - .ok_or_else(|| { - editor_agent_bad_request(format!("image {image_id} not found")) - }) - }) - .collect::, _>>()?; - let payload = EditorIconSpritesheetGenerationRequest { - reference_image_src, - reference_image_srcs: Some(reference_image_srcs), - icon_descriptions: args.icon_descriptions.clone(), - model: Some(args.model), - screen_color: Some("auto".to_string()), - seg_model: Some("birefnet".to_string()), - aspect_ratio: args.aspect_ratio, - image_size: args.image_size, - project_id: Some(conversation.project_id.clone()), - generation_inputs: Some( - json!({"source":"editor-agent","conversationId":conversation.conversation_id,"toolCallMessageId":message_id}), - ), - asset_folder_id: Some("project".to_string()), - asset_label: Some("图标素材图集".to_string()), - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, - GenerateIconSpritesheetTool::NAME, - "图标素材图集", - )), - }; - ( - EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, - "画布 Agent 生成图标素材", - price, - serde_json::to_value(payload), - ) - } - GenerateVideoTool::NAME => { - let args: GenerateVideoToolArgs = parse_confirm_tool_args(&tool_args)?; - let tool = GenerateVideoTool { - context: context.clone(), - }; - tool.validate_args(&args).map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let reference_image_srcs = args - .reference_image_ids - .iter() - .map(|image_id| { - context - .image_data_key(image_id) - .map(ToOwned::to_owned) - .ok_or_else(|| { - editor_agent_bad_request(format!("image {image_id} not found")) - }) - }) - .collect::, _>>()?; - let title = args.prompt.clone(); - let price = tool.pricing(&pricing, &args); - let payload = EditorVideoGenerateRequest { - prompt: args.prompt, - model: args.model, - aspect_ratio: args - .aspect_ratio - .unwrap_or_else(|| GenerateVideoTool::DEFAULT_VIDEO_ASPECT_RATIO.to_string()), - duration_seconds: args - .duration_seconds - .unwrap_or(GenerateVideoTool::DEFAULT_VIDEO_DURATION_SECONDS), - resolution: args - .resolution - .unwrap_or_else(|| GenerateVideoTool::DEFAULT_VIDEO_RESOLUTION.to_string()), - mode: "std".to_string(), - sound: args - .sound - .unwrap_or_else(|| GenerateVideoTool::DEFAULT_VIDEO_SOUND.to_string()), - web_search_enabled: false, - reference_image_srcs, - reference_video_srcs: Vec::new(), - reference_audio_srcs: Vec::new(), - project_id: Some(conversation.project_id.clone()), - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, - GenerateVideoTool::NAME, - &title, - )), - generation_inputs: generation_inputs("用户指令", &title), - source_resource_id: None, - asset_kind: Some("video".to_string()), - asset_folder_id: Some("project".to_string()), - asset_label: Some(title), - }; - ( - EDITOR_VIDEO_GENERATION_JOB_KIND, - "画布 Agent 生成视频", - price, - serde_json::to_value(payload), - ) - } - GenerateSoundEffectTool::NAME => { - let args: GenerateSoundEffectToolArgs = parse_confirm_tool_args(&tool_args)?; - GenerateSoundEffectTool - .validate_args(&args) - .map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let price = GenerateSoundEffectTool.pricing(&pricing, &args); - let title = args.prompt.clone(); - let payload = EditorSoundEffectGenerateRequest { - prompt: args.prompt, - model: Some(args.model), - duration: args - .duration - .unwrap_or(GenerateSoundEffectTool::DEFAULT_DURATION), - project_id: Some(conversation.project_id.clone()), - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, - GenerateSoundEffectTool::NAME, - &title, - )), - generation_inputs: generation_inputs("用户指令", &title), - asset_folder_id: Some("project".to_string()), - asset_label: Some(title), - }; - ( - EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, - "画布 Agent 生成音效", - price, - serde_json::to_value(payload), - ) - } - GenerateBackgroundMusicTool::NAME => { - let args: GenerateBackgroundMusicToolArgs = parse_confirm_tool_args(&tool_args)?; - GenerateBackgroundMusicTool - .validate_args(&args) - .map_err(|error| { - editor_agent_bad_request(format!("invalid tool call args: {error}")) - })?; - let price = GenerateBackgroundMusicTool.pricing(&pricing, &args); - let title = args.prompt.clone(); - let payload = EditorBackgroundMusicGenerateRequest { - gpt_description_prompt: args.prompt, - make_instrumental: true, - project_id: Some(conversation.project_id.clone()), - canvas_completion: Some(build_editor_agent_canvas_completion( - &project, - GenerateBackgroundMusicTool::NAME, - "生成背景音乐", - )), - generation_inputs: generation_inputs("用户指令", &title), - asset_folder_id: Some("project".to_string()), - asset_label: Some("生成背景音乐".to_string()), - }; - ( - EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, - "画布 Agent 生成背景音乐", - price, - serde_json::to_value(payload), - ) - } - _ => { - return Err(editor_agent_bad_request(format!( - "unsupported tool: {tool_name}" - ))); - } - }; - let payload = payload.map_err(|error| { - AppError::from_status(axum::http::StatusCode::INTERNAL_SERVER_ERROR) - .with_details(json!({"message": format!("failed to serialize tool payload: {error}")})) - })?; + let tool = editor_agent_tool(tool_name.as_str(), &context) + .ok_or_else(|| editor_agent_bad_request(format!("unsupported tool: {tool_name}")))?; + let normalized_args = tool + .validate_args(&tool_args) + .map_err(map_editor_agent_tool_app_error)?; + let prepared_job = tool + .prepare_job( + &normalized_args, + &EditorAgentPrepareJobContext { + conversation: &conversation, + project: &project, + message_id, + pricing: &pricing, + }, + ) + .map_err(map_editor_agent_tool_app_error)?; + let job_kind = prepared_job.job_kind; + let request_label = prepared_job.request_label; + let price_mud_points = prepared_job.price_mud_points; + let payload = prepared_job.payload; let (job_id, dedupe_key) = editor_agent_tool_job_identity( conversation.conversation_id.as_str(), message_id, @@ -1389,24 +1013,12 @@ pub async fn confirm_editor_agent_tool_call( )) } -fn parse_confirm_tool_args(value: &Value) -> Result { - serde_json::from_value(value.clone()) - .map_err(|error| editor_agent_bad_request(format!("invalid tool call args: {error}"))) -} - -fn resolve_editor_agent_image_ids( - image_ids: &[platform_editor_agent::agent::asset::ImageId], - context: &EditorToolContext, -) -> Result, AppError> { - image_ids - .iter() - .map(|image_id| { - context - .image_data_key(image_id) - .map(ToOwned::to_owned) - .ok_or_else(|| editor_agent_bad_request(format!("image {image_id} not found"))) - }) - .collect() +fn map_editor_agent_tool_app_error(error: EditorAgentToolError) -> AppError { + if error.is_invalid_args() { + return editor_agent_bad_request(format!("invalid tool call args: {error}")); + } + AppError::from_status(axum::http::StatusCode::INTERNAL_SERVER_ERROR) + .with_details(json!({ "message": error.to_string() })) } fn editor_agent_tool_job_identity( diff --git a/server-rs/crates/api-server/src/editor_agent/context.rs b/server-rs/crates/api-server/src/editor_agent/context.rs index d5a1f1e6b..8b86d0cb9 100644 --- a/server-rs/crates/api-server/src/editor_agent/context.rs +++ b/server-rs/crates/api-server/src/editor_agent/context.rs @@ -8,30 +8,92 @@ pub fn build_tool_context(document: &EditorAgentConversationMessagesDocument) -> let mut images: HashMap = HashMap::new(); for msg in document.messages.iter().rev() { - // tool gen + for a in &msg.attachments { + let data_key = a.clone().into_data_key(); + let image_id = ImageId::from_data_key(&data_key); + let metadata = ImageMetadata { + data_key, + image_src: a.image_src.clone(), + object_key: a.object_key.clone(), + thumbnail_src: a.thumbnail_src.clone(), + label: a.label.clone(), + width: a.width, + height: a.height, + }; + images.entry(image_id).or_insert(metadata); + } + if let Some(tc) = &msg.tool_call { for img in &tc.images { let data_key = img.clone().into_data_key(); let image_id = ImageId::from_data_key(&data_key); let metadata = ImageMetadata { - tag: String::new(), data_key, + image_src: img.image_src.clone(), + object_key: img.object_key.clone(), + thumbnail_src: img.thumbnail_src.clone(), + label: None, + width: img.width, + height: img.height, }; - images.insert(image_id, metadata); + images.entry(image_id).or_insert(metadata); } } - - // user pointed - for a in &msg.attachments { - let data_key = a.clone().into_data_key(); - let image_id = ImageId::from_data_key(&data_key); - let metadata = ImageMetadata { - tag: String::new(), - data_key, - }; - images.insert(image_id, metadata); - } } EditorToolContext { images } } + +#[cfg(test)] +mod tests { + use super::*; + use shared_contracts::editor_agent::{ + EditorAgentAttachmentRef, EditorAgentAttachmentSource, EditorAgentMessage, + EditorAgentMessageRole, + }; + + fn message(id: usize, label: &str, image_src: &str) -> EditorAgentMessage { + EditorAgentMessage { + id, + client_message_id: Some(format!("message-{id}")), + role: EditorAgentMessageRole::User, + text: String::new(), + attachments: vec![EditorAgentAttachmentRef { + source: EditorAgentAttachmentSource::CanvasResource, + reference_id: "resource-1".to_string(), + object_key: Some("generated/reference.png".to_string()), + image_src: image_src.to_string(), + thumbnail_src: Some(format!("{image_src}?thumbnail=1")), + label: Some(label.to_string()), + width: Some(640), + height: Some(480), + }], + tool_call: None, + created_at: "2026-07-23T00:00:00Z".to_string(), + } + } + + #[test] + fn tool_context_keeps_complete_metadata_from_the_latest_image_reference() { + let document = EditorAgentConversationMessagesDocument { + version: 2, + conversation_id: "conversation-1".to_string(), + messages: vec![ + message(0, "旧名称", "/api/assets/read/old.png"), + message(1, "最新名称", "/api/assets/read/current.png"), + ], + }; + + let context = build_tool_context(&document); + let image_id = ImageId::from_data_key("generated/reference.png"); + let metadata = context + .image_metadata(&image_id) + .expect("latest image metadata should be present"); + + assert_eq!(metadata.data_key, "generated/reference.png"); + assert_eq!(metadata.image_src, "/api/assets/read/current.png"); + assert_eq!(metadata.label.as_deref(), Some("最新名称")); + assert_eq!(metadata.width, Some(640)); + assert_eq!(metadata.height, Some(480)); + } +} diff --git a/server-rs/crates/api-server/src/editor_agent/display_args.rs b/server-rs/crates/api-server/src/editor_agent/display_args.rs index cbf2de4de..8f0b08d02 100644 --- a/server-rs/crates/api-server/src/editor_agent/display_args.rs +++ b/server-rs/crates/api-server/src/editor_agent/display_args.rs @@ -1,271 +1,34 @@ -use crate::editor_agent::pricing::EditorAgentPricedTool; -use crate::editor_agent::utils::{IntoDataKey, IntoImageId}; -use crate::editor_generation_config::EditorGenerationPricingConfig; use platform_editor_agent::agent::asset::ImageId; use platform_editor_agent::agent::tools::context::EditorToolContext; -use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; -use platform_editor_agent::agent::tools::generate_background_music::{ - GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, -}; -use platform_editor_agent::agent::tools::generate_character::{ - GenerateCharacterTool, GenerateCharacterToolArgs, -}; -use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ - GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, -}; -use platform_editor_agent::agent::tools::generate_image::{ - GenerateImageTool, GenerateImageToolArgs, -}; -use platform_editor_agent::agent::tools::generate_sound_effect::{ - GenerateSoundEffectTool, GenerateSoundEffectToolArgs, -}; -use platform_editor_agent::agent::tools::generate_ui_design::{ - GenerateUiDesignTool, GenerateUiDesignToolArgs, -}; -use platform_editor_agent::agent::tools::generate_video::{ - GenerateVideoTool, GenerateVideoToolArgs, -}; use platform_editor_agent::framework::error::PromptError; -use platform_editor_agent::framework::tool::Tool; -use serde_json::Value; use shared_contracts::editor_agent::{ - EditorAgentConversationMessagesDocument, EditorAgentToolCallDisplayArgs, - EditorAgentToolCallDisplayExtras, EditorAgentToolCallImageArg, EditorAgentToolCallImageRef, + EditorAgentToolCallDisplayArgs, EditorAgentToolCallImageArg, EditorAgentToolCallImageRef, EditorAgentToolCallStringArg, }; -pub fn build_tool_call_display_args( - tool_name: &str, - args: &Value, - document: &EditorAgentConversationMessagesDocument, - tool_context: &EditorToolContext, - pricing: &EditorGenerationPricingConfig, -) -> Result { - let mut display_args = EditorAgentToolCallDisplayArgs::default(); - - let price_mud_points = match tool_name { - EditImageTool::NAME => { - let args: EditImageToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = EditImageTool { - context: tool_context.clone(), - } - .pricing(pricing, &args); - push_string_display_arg(&mut display_args, "prompt", "修改要求", args.prompt); - push_string_display_arg(&mut display_args, "model", "模型", args.model); - push_image_display_arg( - &mut display_args, - document, - "object_image_id", - "目标图片", - std::slice::from_ref(&args.object_image_id), - )?; - push_image_display_arg( - &mut display_args, - document, - "reference_image_ids", - "参考图片", - args.reference_image_ids.as_slice(), - )?; - price_mud_points - } - GenerateImageTool::NAME => { - let args: GenerateImageToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateImageTool { - context: tool_context.clone(), - } - .pricing(pricing, &args); - push_image_generation_display_args( - &mut display_args, - document, - args.prompt, - args.model, - args.aspect_ratio, - args.image_size, - args.reference_image_ids, - )?; - price_mud_points - } - GenerateCharacterTool::NAME => { - let args: GenerateCharacterToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateCharacterTool { - context: tool_context.clone(), - } - .pricing(pricing, &args); - push_image_generation_display_args( - &mut display_args, - document, - args.prompt, - args.model, - args.aspect_ratio, - args.image_size, - args.reference_image_ids, - )?; - price_mud_points - } - GenerateUiDesignTool::NAME => { - let args: GenerateUiDesignToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateUiDesignTool { - context: tool_context.clone(), - } - .pricing(pricing, &args); - push_image_generation_display_args( - &mut display_args, - document, - args.prompt, - args.model, - args.aspect_ratio, - args.image_size, - args.reference_image_ids, - )?; - price_mud_points - } - GenerateIconSpritesheetTool::NAME => { - let args: GenerateIconSpritesheetToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateIconSpritesheetTool { - context: tool_context.clone(), - } - .pricing(pricing, &args); - push_string_display_arg( - &mut display_args, - "icon_descriptions", - "图标描述", - args.icon_descriptions.join("\n"), - ); - push_string_display_arg(&mut display_args, "model", "模型", args.model); - push_optional_string_display_arg( - &mut display_args, - "aspect_ratio", - "画面比例", - args.aspect_ratio, - ); - push_optional_string_display_arg( - &mut display_args, - "image_size", - "图片尺寸", - args.image_size, - ); - push_image_display_arg( - &mut display_args, - document, - "reference_image_id", - "规范参考图", - std::slice::from_ref(&args.reference_image_id), - )?; - push_image_display_arg( - &mut display_args, - document, - "reference_image_ids", - "额外参考图", - args.reference_image_ids.as_slice(), - )?; - price_mud_points - } - GenerateVideoTool::NAME => { - let args: GenerateVideoToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateVideoTool { - context: tool_context.clone(), - } - .pricing(pricing, &args); - push_string_display_arg(&mut display_args, "prompt", "视频描述", args.prompt); - push_optional_string_display_arg( - &mut display_args, - "aspect_ratio", - "画面比例", - args.aspect_ratio, - ); - if let Some(duration_seconds) = args.duration_seconds { - push_string_display_arg( - &mut display_args, - "duration_seconds", - "时长（秒）", - duration_seconds.to_string(), - ); - } - push_string_display_arg(&mut display_args, "model", "模型", args.model); - push_optional_string_display_arg( - &mut display_args, - "resolution", - "分辨率", - args.resolution, - ); - push_optional_string_display_arg(&mut display_args, "sound", "声音", args.sound); - push_image_display_arg( - &mut display_args, - document, - "reference_image_ids", - "参考图片", - args.reference_image_ids.as_slice(), - )?; - price_mud_points - } - GenerateSoundEffectTool::NAME => { - let args: GenerateSoundEffectToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateSoundEffectTool.pricing(pricing, &args); - push_string_display_arg(&mut display_args, "prompt", "音效描述", args.prompt); - push_string_display_arg(&mut display_args, "model", "模型", args.model); - if let Some(duration) = args.duration { - push_string_display_arg( - &mut display_args, - "duration", - "时长（秒）", - duration.to_string(), - ); - } - price_mud_points - } - GenerateBackgroundMusicTool::NAME => { - let args: GenerateBackgroundMusicToolArgs = parse_display_tool_args(tool_name, args)?; - let price_mud_points = GenerateBackgroundMusicTool.pricing(pricing, &args); - push_string_display_arg(&mut display_args, "prompt", "音乐描述", args.prompt); - push_string_display_arg(&mut display_args, "model", "模型", args.model); - price_mud_points - } - _ => { - return Err(PromptError::InternalError(format!( - "missing display argument mapping for tool {tool_name}" - ))); - } - }; - - display_args.extras = EditorAgentToolCallDisplayExtras { price_mud_points }; - - Ok(display_args) -} - -fn parse_display_tool_args(tool_name: &str, args: &Value) -> Result -where - T: serde::de::DeserializeOwned, -{ - serde_json::from_value(args.clone()).map_err(|error| { - PromptError::InternalError(format!( - "failed to build display arguments for {tool_name}: {error}" - )) - }) -} - -fn push_image_generation_display_args( +pub(crate) fn push_image_generation_display_args( display_args: &mut EditorAgentToolCallDisplayArgs, - document: &EditorAgentConversationMessagesDocument, + context: &EditorToolContext, prompt: String, model: String, - aspect_ratio: Option, - image_size: Option, + aspect_ratio: String, + image_size: String, reference_image_ids: Vec, ) -> Result<(), PromptError> { push_string_display_arg(display_args, "prompt", "提示词", prompt); push_string_display_arg(display_args, "model", "模型", model); - push_optional_string_display_arg(display_args, "aspect_ratio", "画面比例", aspect_ratio); - push_optional_string_display_arg(display_args, "image_size", "图片尺寸", image_size); + push_string_display_arg(display_args, "aspect_ratio", "画面比例", aspect_ratio); + push_string_display_arg(display_args, "image_size", "图片尺寸", image_size); push_image_display_arg( display_args, - document, + context, "reference_image_ids", "参考图片", reference_image_ids.as_slice(), ) } -fn push_string_display_arg( +pub(crate) fn push_string_display_arg( display_args: &mut EditorAgentToolCallDisplayArgs, name: &str, label: &str, @@ -282,20 +45,9 @@ fn push_string_display_arg( }); } -fn push_optional_string_display_arg( +pub(crate) fn push_image_display_arg( display_args: &mut EditorAgentToolCallDisplayArgs, - name: &str, - label: &str, - value: Option, -) { - if let Some(value) = value { - push_string_display_arg(display_args, name, label, value); - } -} - -fn push_image_display_arg( - display_args: &mut EditorAgentToolCallDisplayArgs, - document: &EditorAgentConversationMessagesDocument, + context: &EditorToolContext, name: &str, label: &str, image_ids: &[ImageId], @@ -307,9 +59,9 @@ fn push_image_display_arg( let refs = image_ids .iter() .map(|image_id| { - resolve_tool_call_image_ref(document, image_id).ok_or_else(|| { + resolve_tool_call_image_ref(context, image_id).ok_or_else(|| { PromptError::InternalError(format!( - "display image reference {} is missing from conversation context", + "display image reference {} is missing from tool context", image_id.id )) }) @@ -324,42 +76,19 @@ fn push_image_display_arg( Ok(()) } -pub fn resolve_tool_call_image_ref( - document: &EditorAgentConversationMessagesDocument, +fn resolve_tool_call_image_ref( + context: &EditorToolContext, image_id: &ImageId, ) -> Option { - for message in document.messages.iter().rev() { - for attachment in message.attachments.iter().rev() { - if attachment.clone().into_image_id() == *image_id { - return Some(EditorAgentToolCallImageRef { - image_id: image_id.id.clone(), - image_src: attachment.image_src.clone(), - object_key: attachment.object_key.clone(), - thumbnail_src: attachment.thumbnail_src.clone(), - label: attachment.label.clone(), - width: attachment.width, - height: attachment.height, - }); - } - } - - if let Some(tool_call) = &message.tool_call { - for image in tool_call.images.iter().rev() { - let data_key = image.clone().into_data_key(); - if ImageId::from_data_key(data_key) == *image_id { - return Some(EditorAgentToolCallImageRef { - image_id: image_id.id.clone(), - image_src: image.image_src.clone(), - object_key: image.object_key.clone(), - thumbnail_src: image.thumbnail_src.clone(), - label: None, - width: image.width, - height: image.height, - }); - } - } - } - } - - None + context + .image_metadata(image_id) + .map(|metadata| EditorAgentToolCallImageRef { + image_id: image_id.id.clone(), + image_src: metadata.image_src.clone(), + object_key: metadata.object_key.clone(), + thumbnail_src: metadata.thumbnail_src.clone(), + label: metadata.label.clone(), + width: metadata.width, + height: metadata.height, + }) } diff --git a/server-rs/crates/api-server/src/editor_agent/mod.rs b/server-rs/crates/api-server/src/editor_agent/mod.rs index c73b28bb9..55a9b2d17 100644 --- a/server-rs/crates/api-server/src/editor_agent/mod.rs +++ b/server-rs/crates/api-server/src/editor_agent/mod.rs @@ -1,10 +1,9 @@ pub mod api; mod context; mod display_args; -pub mod pricing; mod reconcile; mod resp_to_asset; -mod tool_args; +mod tool; mod utils; pub use api::{ diff --git a/server-rs/crates/api-server/src/editor_agent/pricing.rs b/server-rs/crates/api-server/src/editor_agent/pricing.rs deleted file mode 100644 index 75438ebeb..000000000 --- a/server-rs/crates/api-server/src/editor_agent/pricing.rs +++ /dev/null @@ -1,354 +0,0 @@ -use crate::editor_generation_config::EditorGenerationPricingConfig; -#[cfg(test)] -use crate::editor_generation_config::load_editor_generation_pricing_from_paths; -#[cfg(test)] -use platform_editor_agent::agent::asset::ImageId; -#[cfg(test)] -use platform_editor_agent::agent::tools::context::EditorToolContext; -use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; -use platform_editor_agent::agent::tools::generate_background_music::{ - GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, -}; -use platform_editor_agent::agent::tools::generate_character::{ - GenerateCharacterTool, GenerateCharacterToolArgs, -}; -use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ - GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, -}; -use platform_editor_agent::agent::tools::generate_image::{ - GenerateImageTool, GenerateImageToolArgs, -}; -use platform_editor_agent::agent::tools::generate_sound_effect::{ - GenerateSoundEffectTool, GenerateSoundEffectToolArgs, -}; -use platform_editor_agent::agent::tools::generate_ui_design::{ - GenerateUiDesignTool, GenerateUiDesignToolArgs, -}; -use platform_editor_agent::agent::tools::generate_video::{ - GenerateVideoTool, GenerateVideoToolArgs, -}; -use platform_editor_agent::framework::tool::Tool; -use platform_image::GPT_IMAGE_2_MODEL; -#[cfg(test)] -use std::collections::HashMap; - -#[cfg(test)] -fn context() -> EditorToolContext { - EditorToolContext { - images: HashMap::new(), - } -} - -#[cfg(test)] -fn image_args(image_size: Option<&str>) -> GenerateImageToolArgs { - GenerateImageToolArgs { - prompt: "生成图片".to_string(), - model: platform_image::NANOBANANA_2_MODEL.to_string(), - reference_image_ids: Vec::new(), - aspect_ratio: Some("1:1".to_string()), - image_size: image_size.map(ToOwned::to_owned), - } -} - -#[cfg(test)] -fn character_args(image_size: Option<&str>) -> GenerateCharacterToolArgs { - GenerateCharacterToolArgs { - prompt: "生成角色".to_string(), - model: platform_image::NANOBANANA_2_MODEL.to_string(), - reference_image_ids: Vec::new(), - aspect_ratio: Some("1:1".to_string()), - image_size: image_size.map(ToOwned::to_owned), - } -} - -#[test] -fn every_editor_agent_tool_exposes_argument_based_pricing() { - let pricing = load_editor_generation_pricing_from_paths(None) - .expect("default editor pricing should load"); - let context = context(); - - assert_eq!( - EditImageTool { - context: context.clone(), - } - .pricing( - &pricing, - &EditImageToolArgs { - object_image_id: ImageId { - id: "image-1".to_string(), - }, - reference_image_ids: Vec::new(), - prompt: "改成蓝色".to_string(), - model: GPT_IMAGE_2_MODEL.to_string(), - }, - ), - 3 - ); - assert_eq!( - GenerateImageTool { - context: context.clone(), - } - .pricing(&pricing, &image_args(Some("2K"))), - 24 - ); - assert_eq!( - GenerateCharacterTool { - context: context.clone(), - } - .pricing(&pricing, &character_args(None)), - 12 - ); - assert_eq!( - GenerateUiDesignTool { - context: context.clone(), - } - .pricing( - &pricing, - &GenerateUiDesignToolArgs { - prompt: "生成游戏主界面".to_string(), - model: GPT_IMAGE_2_MODEL.to_string(), - reference_image_ids: Vec::new(), - aspect_ratio: Some("1:1".to_string()), - image_size: Some("2K".to_string()), - } - ), - 5 - ); - assert_eq!( - GenerateIconSpritesheetTool { - context: context.clone(), - } - .pricing( - &pricing, - &GenerateIconSpritesheetToolArgs { - reference_image_id: ImageId { - id: "image-1".to_string(), - }, - model: platform_image::NANOBANANA_2_MODEL.to_string(), - reference_image_ids: Vec::new(), - icon_descriptions: vec!["背包".to_string(), "地图".to_string()], - aspect_ratio: Some("1:1".to_string()), - image_size: Some("2K".to_string()), - }, - ), - 24 - ); - assert_eq!( - GenerateVideoTool { - context: context.clone(), - } - .pricing( - &pricing, - &GenerateVideoToolArgs { - prompt: "镜头缓慢推进".to_string(), - reference_image_ids: Vec::new(), - aspect_ratio: None, - duration_seconds: Some(6), - model: "seedance2.0".to_string(), - resolution: Some("720p".to_string()), - sound: None, - }, - ), - 144 - ); - assert_eq!( - GenerateSoundEffectTool.pricing( - &pricing, - &GenerateSoundEffectToolArgs { - prompt: "按钮点击声".to_string(), - model: platform_audio::VIDU_AUDIO_MODEL.to_string(), - duration: None, - }, - ), - 5 - ); - assert_eq!( - GenerateBackgroundMusicTool.pricing( - &pricing, - &GenerateBackgroundMusicToolArgs { - prompt: "轻松背景音乐".to_string(), - model: platform_audio::SUNO_DEFAULT_MODEL.to_string(), - }, - ), - 12 - ); -} - -#[test] -fn pricing_uses_the_supplied_runtime_snapshot() { - let mut pricing = load_editor_generation_pricing_from_paths(None) - .expect("default editor pricing should load"); - pricing - .models - .get_mut("gpt-image-2") - .expect("gpt image pricing should exist") - .prices - .insert("2K".to_string(), 37); - pricing - .models - .get_mut("seedance2.0-fast") - .expect("video pricing should exist") - .prices - .insert("720p".to_string(), 7); - pricing - .models - .get_mut("audio1.0") - .expect("sound pricing should exist") - .price = Some(19); - - let mut image2_args = image_args(Some("2K")); - image2_args.model = GPT_IMAGE_2_MODEL.to_string(); - assert_eq!( - GenerateImageTool { context: context() }.pricing(&pricing, &image2_args), - 37 - ); - assert_eq!( - GenerateVideoTool { context: context() }.pricing( - &pricing, - &GenerateVideoToolArgs { - prompt: "four seconds".to_string(), - reference_image_ids: Vec::new(), - aspect_ratio: None, - duration_seconds: None, - model: GenerateVideoTool::DEFAULT_VIDEO_MODEL.to_string(), - resolution: None, - sound: None, - }, - ), - 28 - ); - assert_eq!( - GenerateSoundEffectTool.pricing( - &pricing, - &GenerateSoundEffectToolArgs { - prompt: "sound".to_string(), - model: platform_audio::VIDU_AUDIO_MODEL.to_string(), - duration: None, - }, - ), - 19 - ); -} - -impl EditorAgentPricedTool for GenerateVideoTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateVideoToolArgs, - ) -> u32 { - let model = args.model.as_str(); - let resolution = args - .resolution - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty()) - .unwrap_or(Self::DEFAULT_VIDEO_RESOLUTION); - let duration_seconds = args - .duration_seconds - .unwrap_or(Self::DEFAULT_VIDEO_DURATION_SECONDS); - pricing.video_model_mud_points(Some(model), resolution, duration_seconds) - } -} - -impl EditorAgentPricedTool for GenerateUiDesignTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateUiDesignToolArgs, - ) -> u32 { - editor_agent_image_mud_points( - pricing, - Some("ui-design"), - args.model.as_str(), - args.image_size.as_deref(), - ) - } -} - -impl EditorAgentPricedTool for GenerateSoundEffectTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateSoundEffectToolArgs, - ) -> u32 { - pricing.sound_effect_model_mud_points(Some(args.model.as_str())) - } -} - -impl EditorAgentPricedTool for GenerateImageTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateImageToolArgs, - ) -> u32 { - editor_agent_image_mud_points( - pricing, - None, - args.model.as_str(), - args.image_size.as_deref(), - ) - } -} - -impl EditorAgentPricedTool for GenerateIconSpritesheetTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateIconSpritesheetToolArgs, - ) -> u32 { - editor_agent_image_mud_points( - pricing, - Some("icon"), - args.model.as_str(), - args.image_size.as_deref(), - ) - } -} - -impl EditorAgentPricedTool for GenerateCharacterTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateCharacterToolArgs, - ) -> u32 { - editor_agent_image_mud_points( - pricing, - Some("character"), - args.model.as_str(), - args.image_size.as_deref(), - ) - } -} - -/// api-server 侧的画布 Agent 工具计价扩展。 -/// -/// 通用 `Tool` 仍只负责参数校验；价格依赖 api-server 的运行时配置，不能下沉到 -/// `module-editor-agent`。实际执行和扣费仍由既有生成 BFF 负责。 -pub(crate) trait EditorAgentPricedTool: Tool { - fn pricing(&self, pricing: &EditorGenerationPricingConfig, args: &::Args) -> u32; -} - -pub(crate) fn editor_agent_image_mud_points( - pricing: &EditorGenerationPricingConfig, - kind: Option<&str>, - model: &str, - image_size: Option<&str>, -) -> u32 { - pricing.image_generation_mud_points(kind, Some(model), image_size) -} - -impl EditorAgentPricedTool for GenerateBackgroundMusicTool { - fn pricing( - &self, - pricing: &EditorGenerationPricingConfig, - args: &GenerateBackgroundMusicToolArgs, - ) -> u32 { - pricing.background_music_model_mud_points(Some(args.model.as_str())) - } -} - -impl EditorAgentPricedTool for EditImageTool { - fn pricing(&self, pricing: &EditorGenerationPricingConfig, _args: &EditImageToolArgs) -> u32 { - editor_agent_image_mud_points(pricing, Some("quick-edit"), GPT_IMAGE_2_MODEL, Some("1K")) - } -} diff --git a/server-rs/crates/api-server/src/editor_agent/reconcile.rs b/server-rs/crates/api-server/src/editor_agent/reconcile.rs index 8351b2429..dbc2f9667 100644 --- a/server-rs/crates/api-server/src/editor_agent/reconcile.rs +++ b/server-rs/crates/api-server/src/editor_agent/reconcile.rs @@ -1,34 +1,7 @@ -use crate::editor_agent::resp_to_asset; use crate::http_error::AppError; use crate::state::AppState; use platform_editor_agent::agent::tools::context::EditorToolContext; -use platform_editor_agent::agent::tools::edit_image::{ - EditImageTool, EditImageToolArgs, EditorImageEditResult, -}; -use platform_editor_agent::agent::tools::generate_background_music::{ - GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, -}; -use platform_editor_agent::agent::tools::generate_character::{ - GenerateCharacterTool, GenerateCharacterToolArgs, -}; -use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ - EditorIconSpritesheetResult, GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, -}; -use platform_editor_agent::agent::tools::generate_image::{ - EditorImageGenerationResult, GenerateImageTool, GenerateImageToolArgs, -}; -use platform_editor_agent::agent::tools::generate_sound_effect::{ - GenerateSoundEffectTool, GenerateSoundEffectToolArgs, -}; -use platform_editor_agent::agent::tools::generate_ui_design::{ - GenerateUiDesignTool, GenerateUiDesignToolArgs, -}; -use platform_editor_agent::agent::tools::generate_video::{ - GenerateVideoTool, GenerateVideoToolArgs, -}; -use platform_editor_agent::framework::tool::Tool; use serde_json::Value; -use shared_contracts::assets::{EditorAudioGenerateResponse, EditorVideoGenerateResponse}; use shared_contracts::editor_agent::{ EditorAgentConversationMessagesDocument, EditorAgentMessage, EditorAgentToolCallStatus, }; @@ -209,88 +182,31 @@ fn reconcile_completed_editor_agent_tool_call( .ok_or_else(|| ResultReconcileFailure::Fatal("消息缺少工具调用".to_string()))?; let tool_name = tool_call.tool_name.clone(); - match tool_name.as_str() { - GenerateImageTool::NAME => { - let args: GenerateImageToolArgs = parse_reconciled_value(&tool_call.args)?; - let result: EditorImageGenerationResult = parse_reconciled_value(&response)?; - message.text = GenerateImageTool { - context: EditorToolContext::default(), - } - .format_execute_message(&args, &result); - tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)]; - } - GenerateCharacterTool::NAME => { - let args: GenerateCharacterToolArgs = parse_reconciled_value(&tool_call.args)?; - let result: EditorImageGenerationResult = parse_reconciled_value(&response)?; - message.text = GenerateCharacterTool { - context: EditorToolContext::default(), - } - .format_execute_message(&args, &result); - tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)]; - } - GenerateUiDesignTool::NAME => { - let args: GenerateUiDesignToolArgs = parse_reconciled_value(&tool_call.args)?; - let result: EditorImageGenerationResult = parse_reconciled_value(&response)?; - message.text = GenerateUiDesignTool { - context: EditorToolContext::default(), - } - .format_execute_message(&args, &result); - tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)]; - } - EditImageTool::NAME => { - let args: EditImageToolArgs = parse_reconciled_value(&tool_call.args)?; - let result: EditorImageEditResult = parse_reconciled_value(&response)?; - tool_call.images = vec![resp_to_asset::editor_agent_generated_edited_image(&result)]; - message.text = EditImageTool { - context: EditorToolContext::default(), - } - .format_execute_message(args, result); - } - GenerateIconSpritesheetTool::NAME => { - let args: GenerateIconSpritesheetToolArgs = parse_reconciled_value(&tool_call.args)?; - let result: EditorIconSpritesheetResult = parse_reconciled_value(&response)?; - message.text = GenerateIconSpritesheetTool { - context: EditorToolContext::default(), - } - .format_execute_message(&args, &result); - tool_call.images = resp_to_asset::editor_agent_generated_spritesheet_images(&result); - } - GenerateVideoTool::NAME => { - let args: GenerateVideoToolArgs = parse_reconciled_value(&tool_call.args)?; - let result: EditorVideoGenerateResponse = parse_reconciled_value(&response)?; - message.text = GenerateVideoTool { - context: EditorToolContext::default(), - } - .format_execute_message(&args, &result); - tool_call.videos = vec![resp_to_asset::editor_agent_generated_video(&result)]; - } - GenerateSoundEffectTool::NAME | GenerateBackgroundMusicTool::NAME => { - let result: EditorAudioGenerateResponse = parse_reconciled_value(&response)?; - message.text = if tool_name == GenerateSoundEffectTool::NAME { - let args: GenerateSoundEffectToolArgs = parse_reconciled_value(&tool_call.args)?; - GenerateSoundEffectTool.format_execute_message(&args, &result) - } else { - let args: GenerateBackgroundMusicToolArgs = - parse_reconciled_value(&tool_call.args)?; - GenerateBackgroundMusicTool.format_execute_message(&args, &result) - }; - tool_call.audios = vec![resp_to_asset::editor_agent_generated_audio(&result)]; - } - _ => { - return Err(ResultReconcileFailure::Fatal(format!( - "不支持回填工具 {tool_name}" - ))); - } + let tool = crate::editor_agent::tool::editor_agent_tool( + tool_name.as_str(), + &EditorToolContext::default(), + ) + .ok_or_else(|| ResultReconcileFailure::Fatal(format!("不支持回填工具 {tool_name}")))?; + message.text = tool + .format_execute_message(&tool_call.args, &response) + .map_err(|error| ResultReconcileFailure::Fatal(error.to_string()))?; + let assets = tool + .result_assets(&response) + .map_err(|error| ResultReconcileFailure::Fatal(error.to_string()))?; + if let Some(images) = assets.images { + tool_call.images = images; + } + if let Some(videos) = assets.videos { + tool_call.videos = videos; + } + if let Some(audios) = assets.audios { + tool_call.audios = audios; } tool_call.error = None; tool_call.status = EditorAgentToolCallStatus::Completed; Ok(()) } -fn parse_reconciled_value(value: &Value) -> Result { - serde_json::from_value(value.clone()).map_err(|error| error.to_string()) -} - #[cfg(test)] mod tests { use super::*; diff --git a/server-rs/crates/api-server/src/editor_agent/tool.rs b/server-rs/crates/api-server/src/editor_agent/tool.rs new file mode 100644 index 000000000..58fd32ef3 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/tool.rs @@ -0,0 +1,1502 @@ +use std::fmt::{Display, Formatter}; + +use platform_editor_agent::agent::asset::ImageId; +use platform_editor_agent::agent::tools::context::EditorToolContext; +use platform_editor_agent::agent::tools::edit_image::{ + EditImageTool, EditImageToolArgs, EditorImageEditResult, +}; +use platform_editor_agent::agent::tools::generate_background_music::{ + GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, +}; +use platform_editor_agent::agent::tools::generate_character::{ + GenerateCharacterTool, GenerateCharacterToolArgs, +}; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ + EditorIconSpritesheetResult, GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, +}; +use platform_editor_agent::agent::tools::generate_image::{ + EditorImageGenerationResult, GenerateImageTool, GenerateImageToolArgs, +}; +use platform_editor_agent::agent::tools::generate_sound_effect::{ + GenerateSoundEffectTool, GenerateSoundEffectToolArgs, +}; +use platform_editor_agent::agent::tools::generate_ui_design::{ + GenerateUiDesignTool, GenerateUiDesignToolArgs, +}; +use platform_editor_agent::agent::tools::generate_video::{ + GenerateVideoTool, GenerateVideoToolArgs, +}; +use platform_editor_agent::framework::error::PromptError; +use platform_editor_agent::framework::tool::{Tool, ToolDyn}; +use platform_image::GPT_IMAGE_2_MODEL; +use serde::Serialize; +use serde::de::DeserializeOwned; +use serde_json::{Value, json}; +use shared_contracts::assets::{ + EditorAudioGenerateResponse, EditorBackgroundMusicGenerateRequest, + EditorSoundEffectGenerateRequest, EditorVideoGenerateRequest, EditorVideoGenerateResponse, +}; +use shared_contracts::editor_agent::{ + EditorAgentGeneratedAudio, EditorAgentGeneratedImage, EditorAgentGeneratedVideo, + EditorAgentToolCallDisplayArgs, EditorAgentToolCallDisplayExtras, +}; +use spacetime_client::{EditorAgentConversationRecord, EditorProjectRecord}; + +use crate::editor_agent::display_args::{ + push_image_display_arg, push_image_generation_display_args, push_string_display_arg, +}; +use crate::editor_agent::resp_to_asset; +use crate::editor_agent::utils::build_editor_agent_canvas_completion; +use crate::editor_generation_config::EditorGenerationPricingConfig; +use crate::editor_generation_queue::{ + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, + EDITOR_IMAGE_EDIT_JOB_KIND, EDITOR_IMAGE_GENERATION_JOB_KIND, + EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, EDITOR_VIDEO_GENERATION_JOB_KIND, +}; +use crate::editor_project::{ + EditorIconSpritesheetGenerationRequest, EditorImageEditRequest, EditorImageGenerationRequest, +}; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum EditorAgentToolError { + InvalidArgs(String), + Internal(String), +} + +impl EditorAgentToolError { + fn invalid_args(message: impl Into) -> Self { + Self::InvalidArgs(message.into()) + } + + fn internal(message: impl Into) -> Self { + Self::Internal(message.into()) + } + + pub(crate) fn into_prompt_error(self, tool_name: &str) -> PromptError { + match self { + Self::InvalidArgs(message) => { + PromptError::ToolError(format!("invalid args for tool {tool_name}: {message}")) + } + Self::Internal(message) => PromptError::InternalError(message), + } + } + + pub(crate) fn is_invalid_args(&self) -> bool { + matches!(self, Self::InvalidArgs(_)) + } +} + +impl Display for EditorAgentToolError { + fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result { + match self { + Self::InvalidArgs(message) | Self::Internal(message) => formatter.write_str(message), + } + } +} + +impl std::error::Error for EditorAgentToolError {} + +pub(crate) struct EditorAgentPrepareJobContext<'a> { + pub(crate) conversation: &'a EditorAgentConversationRecord, + pub(crate) project: &'a EditorProjectRecord, + pub(crate) message_id: usize, + pub(crate) pricing: &'a EditorGenerationPricingConfig, +} + +impl EditorAgentPrepareJobContext<'_> { + fn generation_inputs(&self, title: &str, value: &str) -> Option { + Some(json!({ + "source": "editor-agent", + "conversationId": self.conversation.conversation_id, + "toolCallMessageId": self.message_id, + "fields": [{ "title": title, "value": value }], + })) + } +} + +pub(crate) struct PreparedEditorAgentJob { + pub(crate) job_kind: &'static str, + pub(crate) request_label: &'static str, + pub(crate) price_mud_points: u32, + pub(crate) payload: Value, +} + +#[derive(Default)] +pub(crate) struct EditorAgentResultAssets { + pub(crate) images: Option>, + pub(crate) videos: Option>, + pub(crate) audios: Option>, +} + +/// api-server 侧统一的画布 Agent 工具能力。 +/// +/// 所有方法都使用统一 JSON 边界以保持 trait object-safe；每个具体实现负责把 JSON +/// 反序列化为真实 Tool Args / 执行结果，并转发到具体工具已有的强类型行为。 +pub(crate) trait EditorAgentTool: ToolDyn { + fn validate_args(&self, args: &Value) -> Result; + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result; + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result; + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result; + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result; + + fn result_assets( + &self, + result: &Value, + ) -> Result; +} + +pub(crate) fn editor_agent_tool( + tool_name: &str, + context: &EditorToolContext, +) -> Option> { + match tool_name { + GenerateImageTool::NAME => Some(Box::new(GenerateImageTool { + context: context.clone(), + })), + GenerateCharacterTool::NAME => Some(Box::new(GenerateCharacterTool { + context: context.clone(), + })), + GenerateUiDesignTool::NAME => Some(Box::new(GenerateUiDesignTool { + context: context.clone(), + })), + EditImageTool::NAME => Some(Box::new(EditImageTool { + context: context.clone(), + })), + GenerateIconSpritesheetTool::NAME => Some(Box::new(GenerateIconSpritesheetTool { + context: context.clone(), + })), + GenerateVideoTool::NAME => Some(Box::new(GenerateVideoTool { + context: context.clone(), + })), + GenerateSoundEffectTool::NAME => Some(Box::new(GenerateSoundEffectTool)), + GenerateBackgroundMusicTool::NAME => Some(Box::new(GenerateBackgroundMusicTool)), + _ => None, + } +} + +fn parse_invalid_args( + tool_name: &str, + value: &Value, +) -> Result { + serde_json::from_value(value.clone()).map_err(|error| { + EditorAgentToolError::invalid_args(format!( + "failed to deserialize {tool_name} args: {error}" + )) + }) +} + +fn parse_internal( + label: &str, + value: &Value, +) -> Result { + serde_json::from_value(value.clone()).map_err(|error| { + EditorAgentToolError::internal(format!("failed to deserialize {label}: {error}")) + }) +} + +fn serialize_normalized_args( + tool_name: &str, + args: &T, +) -> Result { + serde_json::to_value(args).map_err(|error| { + EditorAgentToolError::internal(format!( + "failed to normalize args for tool {tool_name}: {error}" + )) + }) +} + +fn serialize_job_payload( + tool_name: &str, + payload: T, +) -> Result { + serde_json::to_value(payload).map_err(|error| { + EditorAgentToolError::internal(format!( + "failed to serialize tool payload for {tool_name}: {error}" + )) + }) +} + +fn finish_display_args( + mut display_args: EditorAgentToolCallDisplayArgs, + price_mud_points: u32, +) -> EditorAgentToolCallDisplayArgs { + display_args.extras = EditorAgentToolCallDisplayExtras { price_mud_points }; + display_args +} + +fn map_display_error(error: PromptError) -> EditorAgentToolError { + EditorAgentToolError::internal(error.to_string()) +} + +fn resolve_image_ids( + image_ids: &[ImageId], + context: &EditorToolContext, +) -> Result, EditorAgentToolError> { + image_ids + .iter() + .map(|image_id| { + context + .image_data_key(image_id) + .map(ToOwned::to_owned) + .ok_or_else(|| { + EditorAgentToolError::invalid_args(format!("image {image_id} not found")) + }) + }) + .collect() +} + +fn editor_agent_image_mud_points( + pricing: &EditorGenerationPricingConfig, + kind: Option<&str>, + model: &str, + image_size: Option<&str>, +) -> u32 { + pricing.image_generation_mud_points(kind, Some(model), image_size) +} + +impl EditorAgentTool for GenerateImageTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateImageToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateImageTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateImageToolArgs = parse_internal("generate image args", args)?; + Ok(editor_agent_image_mud_points( + pricing, + None, + args.model.as_str(), + Some(args.image_size.as_str()), + )) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateImageToolArgs = parse_internal("generate image display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_image_generation_display_args( + &mut display_args, + &self.context, + args.prompt, + args.model, + args.aspect_ratio, + args.image_size, + args.reference_image_ids, + ) + .map_err(map_display_error)?; + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateImageToolArgs = parse_invalid_args(Self::NAME, args)?; + let title = args.prompt.clone(); + let reference_image_srcs = resolve_image_ids(&args.reference_image_ids, &self.context)?; + let payload = EditorImageGenerationRequest { + prompt: args.prompt, + size: None, + kind: None, + model: Some(args.model), + screen_color: None, + seg_model: None, + aspect_ratio: Some(args.aspect_ratio), + image_size: Some(args.image_size), + reference_image_srcs: Some(reference_image_srcs), + project_id: Some(context.conversation.project_id.clone()), + asset_kind: Some("editor_agent_generated_image".to_string()), + generation_inputs: context.generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title.clone()), + source_resource_id: None, + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + &title, + )), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_IMAGE_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成图片", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateImageToolArgs = parse_internal("generate image args", args)?; + let result: EditorImageGenerationResult = parse_internal("generate image result", result)?; + Ok(GenerateImageTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorImageGenerationResult = parse_internal("generate image result", result)?; + Ok(EditorAgentResultAssets { + images: Some(vec![resp_to_asset::editor_agent_generated_image(&result)]), + ..Default::default() + }) + } +} + +impl EditorAgentTool for GenerateCharacterTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateCharacterToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateCharacterTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateCharacterToolArgs = parse_internal("generate character args", args)?; + Ok(editor_agent_image_mud_points( + pricing, + Some("character"), + args.model.as_str(), + Some(args.image_size.as_str()), + )) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateCharacterToolArgs = + parse_internal("generate character display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_image_generation_display_args( + &mut display_args, + &self.context, + args.prompt, + args.model, + args.aspect_ratio, + args.image_size, + args.reference_image_ids, + ) + .map_err(map_display_error)?; + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateCharacterToolArgs = parse_invalid_args(Self::NAME, args)?; + let title = args.prompt.clone(); + let reference_image_srcs = resolve_image_ids(&args.reference_image_ids, &self.context)?; + let payload = EditorImageGenerationRequest { + prompt: args.prompt, + size: None, + kind: Some("character".to_string()), + model: Some(args.model), + screen_color: Some("auto".to_string()), + seg_model: Some("birefnet".to_string()), + aspect_ratio: Some(args.aspect_ratio), + image_size: Some(args.image_size), + reference_image_srcs: Some(reference_image_srcs), + project_id: Some(context.conversation.project_id.clone()), + asset_kind: Some("character".to_string()), + generation_inputs: context.generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title.clone()), + source_resource_id: None, + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + &title, + )), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_IMAGE_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成图片", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateCharacterToolArgs = parse_internal("generate character args", args)?; + let result: EditorImageGenerationResult = + parse_internal("generate character result", result)?; + Ok(GenerateCharacterTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorImageGenerationResult = + parse_internal("generate character result", result)?; + Ok(EditorAgentResultAssets { + images: Some(vec![resp_to_asset::editor_agent_generated_image(&result)]), + ..Default::default() + }) + } +} + +impl EditorAgentTool for GenerateUiDesignTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateUiDesignToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateUiDesignTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateUiDesignToolArgs = parse_internal("generate UI design args", args)?; + Ok(editor_agent_image_mud_points( + pricing, + Some("ui-design"), + args.model.as_str(), + Some(args.image_size.as_str()), + )) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateUiDesignToolArgs = + parse_internal("generate UI design display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_image_generation_display_args( + &mut display_args, + &self.context, + args.prompt, + args.model, + args.aspect_ratio, + args.image_size, + args.reference_image_ids, + ) + .map_err(map_display_error)?; + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateUiDesignToolArgs = parse_invalid_args(Self::NAME, args)?; + let title = args.prompt.clone(); + let reference_image_srcs = resolve_image_ids(&args.reference_image_ids, &self.context)?; + let payload = EditorImageGenerationRequest { + prompt: args.prompt, + size: None, + kind: Some("ui-design".to_string()), + model: Some(args.model), + screen_color: None, + seg_model: None, + aspect_ratio: Some(args.aspect_ratio), + image_size: Some(args.image_size), + reference_image_srcs: Some(reference_image_srcs), + project_id: Some(context.conversation.project_id.clone()), + asset_kind: Some("ui-design".to_string()), + generation_inputs: context.generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title.clone()), + source_resource_id: None, + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + &title, + )), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_IMAGE_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成图片", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateUiDesignToolArgs = parse_internal("generate UI design args", args)?; + let result: EditorImageGenerationResult = + parse_internal("generate UI design result", result)?; + Ok(GenerateUiDesignTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorImageGenerationResult = + parse_internal("generate UI design result", result)?; + Ok(EditorAgentResultAssets { + images: Some(vec![resp_to_asset::editor_agent_generated_image(&result)]), + ..Default::default() + }) + } +} + +impl EditorAgentTool for EditImageTool { + fn validate_args(&self, args: &Value) -> Result { + let args: EditImageToolArgs = parse_invalid_args(Self::NAME, args)?; + if let Some(error) = EditImageTool::validate_args(self, &args) { + return Err(EditorAgentToolError::invalid_args(error.to_string())); + } + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let _: EditImageToolArgs = parse_internal("edit image args", args)?; + Ok(editor_agent_image_mud_points( + pricing, + Some("quick-edit"), + GPT_IMAGE_2_MODEL, + Some("1K"), + )) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: EditImageToolArgs = parse_internal("edit image display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_string_display_arg(&mut display_args, "prompt", "修改要求", args.prompt); + push_string_display_arg(&mut display_args, "model", "模型", args.model); + push_image_display_arg( + &mut display_args, + &self.context, + "object_image_id", + "目标图片", + std::slice::from_ref(&args.object_image_id), + ) + .map_err(map_display_error)?; + push_image_display_arg( + &mut display_args, + &self.context, + "reference_image_ids", + "参考图片", + args.reference_image_ids.as_slice(), + ) + .map_err(map_display_error)?; + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: EditImageToolArgs = parse_invalid_args(Self::NAME, args)?; + let source_image_src = self + .context + .image_data_key(&args.object_image_id) + .ok_or_else(|| EditorAgentToolError::invalid_args("object image not found"))? + .to_string(); + let reference_image_srcs = resolve_image_ids(&args.reference_image_ids, &self.context)?; + let title = args.prompt.clone(); + let payload = EditorImageEditRequest { + prompt: args.prompt, + source_image_src, + size: None, + model: Some(args.model), + aspect_ratio: None, + image_size: None, + reference_image_srcs: Some(reference_image_srcs), + project_id: Some(context.conversation.project_id.clone()), + asset_kind: Some("editor_agent_edit_image".to_string()), + generation_inputs: context.generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title.clone()), + source_resource_id: None, + target_layer_id: None, + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + &title, + )), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_IMAGE_EDIT_JOB_KIND, + request_label: "画布 Agent 修改图片", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: EditImageToolArgs = parse_internal("edit image args", args)?; + let result: EditorImageEditResult = parse_internal("edit image result", result)?; + Ok(EditImageTool::format_execute_message(self, args, result)) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorImageEditResult = parse_internal("edit image result", result)?; + Ok(EditorAgentResultAssets { + images: Some(vec![resp_to_asset::editor_agent_generated_edited_image( + &result, + )]), + ..Default::default() + }) + } +} + +impl EditorAgentTool for GenerateIconSpritesheetTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateIconSpritesheetToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateIconSpritesheetTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateIconSpritesheetToolArgs = + parse_internal("generate icon spritesheet args", args)?; + Ok(editor_agent_image_mud_points( + pricing, + Some("icon"), + args.model.as_str(), + Some(args.image_size.as_str()), + )) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateIconSpritesheetToolArgs = + parse_internal("generate icon spritesheet display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_string_display_arg( + &mut display_args, + "icon_descriptions", + "图标描述", + args.icon_descriptions.join("\n"), + ); + push_string_display_arg(&mut display_args, "model", "模型", args.model); + push_string_display_arg( + &mut display_args, + "aspect_ratio", + "画面比例", + args.aspect_ratio, + ); + push_string_display_arg(&mut display_args, "image_size", "图片尺寸", args.image_size); + push_image_display_arg( + &mut display_args, + &self.context, + "reference_image_id", + "规范参考图", + std::slice::from_ref(&args.reference_image_id), + ) + .map_err(map_display_error)?; + push_image_display_arg( + &mut display_args, + &self.context, + "reference_image_ids", + "额外参考图", + args.reference_image_ids.as_slice(), + ) + .map_err(map_display_error)?; + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateIconSpritesheetToolArgs = parse_invalid_args(Self::NAME, args)?; + let reference_image_src = self + .context + .image_data_key(&args.reference_image_id) + .ok_or_else(|| EditorAgentToolError::invalid_args("reference image not found"))? + .to_string(); + let reference_image_srcs = resolve_image_ids(&args.reference_image_ids, &self.context)?; + let payload = EditorIconSpritesheetGenerationRequest { + reference_image_src, + reference_image_srcs: Some(reference_image_srcs), + icon_descriptions: args.icon_descriptions, + model: Some(args.model), + screen_color: Some("auto".to_string()), + seg_model: Some("birefnet".to_string()), + aspect_ratio: Some(args.aspect_ratio), + image_size: Some(args.image_size), + project_id: Some(context.conversation.project_id.clone()), + generation_inputs: Some(json!({ + "source": "editor-agent", + "conversationId": context.conversation.conversation_id, + "toolCallMessageId": context.message_id, + })), + asset_folder_id: Some("project".to_string()), + asset_label: Some("图标素材图集".to_string()), + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + "图标素材图集", + )), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成图标素材", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateIconSpritesheetToolArgs = + parse_internal("generate icon spritesheet args", args)?; + let result: EditorIconSpritesheetResult = + parse_internal("generate icon spritesheet result", result)?; + Ok(GenerateIconSpritesheetTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorIconSpritesheetResult = + parse_internal("generate icon spritesheet result", result)?; + Ok(EditorAgentResultAssets { + images: Some(resp_to_asset::editor_agent_generated_spritesheet_images( + &result, + )), + ..Default::default() + }) + } +} + +impl EditorAgentTool for GenerateVideoTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateVideoToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateVideoTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateVideoToolArgs = parse_internal("generate video args", args)?; + Ok(pricing.video_model_mud_points( + Some(args.model.as_str()), + args.resolution.as_str(), + args.duration_seconds, + )) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateVideoToolArgs = parse_internal("generate video display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_string_display_arg(&mut display_args, "prompt", "视频描述", args.prompt); + push_string_display_arg( + &mut display_args, + "aspect_ratio", + "画面比例", + args.aspect_ratio, + ); + push_string_display_arg( + &mut display_args, + "duration_seconds", + "时长（秒）", + args.duration_seconds.to_string(), + ); + push_string_display_arg(&mut display_args, "model", "模型", args.model); + push_string_display_arg(&mut display_args, "resolution", "分辨率", args.resolution); + push_string_display_arg(&mut display_args, "sound", "声音", args.sound); + push_image_display_arg( + &mut display_args, + &self.context, + "reference_image_ids", + "参考图片", + args.reference_image_ids.as_slice(), + ) + .map_err(map_display_error)?; + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateVideoToolArgs = parse_invalid_args(Self::NAME, args)?; + let reference_image_srcs = resolve_image_ids(&args.reference_image_ids, &self.context)?; + let title = args.prompt.clone(); + let payload = EditorVideoGenerateRequest { + prompt: args.prompt, + model: args.model, + aspect_ratio: args.aspect_ratio, + duration_seconds: args.duration_seconds, + resolution: args.resolution, + mode: "std".to_string(), + sound: args.sound, + web_search_enabled: false, + reference_image_srcs, + reference_video_srcs: Vec::new(), + reference_audio_srcs: Vec::new(), + project_id: Some(context.conversation.project_id.clone()), + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + &title, + )), + generation_inputs: context.generation_inputs("用户指令", &title), + source_resource_id: None, + asset_kind: Some("video".to_string()), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_VIDEO_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成视频", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateVideoToolArgs = parse_internal("generate video args", args)?; + let result: EditorVideoGenerateResponse = parse_internal("generate video result", result)?; + Ok(GenerateVideoTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorVideoGenerateResponse = parse_internal("generate video result", result)?; + Ok(EditorAgentResultAssets { + videos: Some(vec![resp_to_asset::editor_agent_generated_video(&result)]), + ..Default::default() + }) + } +} + +impl EditorAgentTool for GenerateSoundEffectTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateSoundEffectToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateSoundEffectTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateSoundEffectToolArgs = parse_internal("generate sound effect args", args)?; + Ok(pricing.sound_effect_model_mud_points(Some(args.model.as_str()))) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateSoundEffectToolArgs = + parse_internal("generate sound effect display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_string_display_arg(&mut display_args, "prompt", "音效描述", args.prompt); + push_string_display_arg(&mut display_args, "model", "模型", args.model); + push_string_display_arg( + &mut display_args, + "duration", + "时长（秒）", + args.duration.to_string(), + ); + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateSoundEffectToolArgs = parse_invalid_args(Self::NAME, args)?; + let title = args.prompt.clone(); + let payload = EditorSoundEffectGenerateRequest { + prompt: args.prompt, + model: Some(args.model), + duration: args.duration, + project_id: Some(context.conversation.project_id.clone()), + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + &title, + )), + generation_inputs: context.generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成音效", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateSoundEffectToolArgs = parse_internal("generate sound effect args", args)?; + let result: EditorAudioGenerateResponse = + parse_internal("generate sound effect result", result)?; + Ok(GenerateSoundEffectTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorAudioGenerateResponse = + parse_internal("generate sound effect result", result)?; + Ok(EditorAgentResultAssets { + audios: Some(vec![resp_to_asset::editor_agent_generated_audio(&result)]), + ..Default::default() + }) + } +} + +impl EditorAgentTool for GenerateBackgroundMusicTool { + fn validate_args(&self, args: &Value) -> Result { + let args: GenerateBackgroundMusicToolArgs = parse_invalid_args(Self::NAME, args)?; + GenerateBackgroundMusicTool::validate_args(self, &args) + .map_err(|error| EditorAgentToolError::invalid_args(error.to_string()))?; + serialize_normalized_args(Self::NAME, &args) + } + + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &Value, + ) -> Result { + let args: GenerateBackgroundMusicToolArgs = + parse_internal("generate background music args", args)?; + Ok(pricing.background_music_model_mud_points(Some(args.model.as_str()))) + } + + fn build_display_args( + &self, + args: &Value, + pricing: &EditorGenerationPricingConfig, + ) -> Result { + let price_mud_points = self.pricing(pricing, args)?; + let args: GenerateBackgroundMusicToolArgs = + parse_internal("generate background music display args", args)?; + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + push_string_display_arg(&mut display_args, "prompt", "音乐描述", args.prompt); + push_string_display_arg(&mut display_args, "model", "模型", args.model); + Ok(finish_display_args(display_args, price_mud_points)) + } + + fn prepare_job( + &self, + args: &Value, + context: &EditorAgentPrepareJobContext<'_>, + ) -> Result { + let price_mud_points = self.pricing(context.pricing, args)?; + let args: GenerateBackgroundMusicToolArgs = parse_invalid_args(Self::NAME, args)?; + let title = args.prompt.clone(); + let payload = EditorBackgroundMusicGenerateRequest { + gpt_description_prompt: args.prompt, + make_instrumental: true, + project_id: Some(context.conversation.project_id.clone()), + canvas_completion: Some(build_editor_agent_canvas_completion( + context.project, + Self::NAME, + "生成背景音乐", + )), + generation_inputs: context.generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some("生成背景音乐".to_string()), + }; + Ok(PreparedEditorAgentJob { + job_kind: EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, + request_label: "画布 Agent 生成背景音乐", + price_mud_points, + payload: serialize_job_payload(Self::NAME, payload)?, + }) + } + + fn format_execute_message( + &self, + args: &Value, + result: &Value, + ) -> Result { + let args: GenerateBackgroundMusicToolArgs = + parse_internal("generate background music args", args)?; + let result: EditorAudioGenerateResponse = + parse_internal("generate background music result", result)?; + Ok(GenerateBackgroundMusicTool::format_execute_message( + self, &args, &result, + )) + } + + fn result_assets( + &self, + result: &Value, + ) -> Result { + let result: EditorAudioGenerateResponse = + parse_internal("generate background music result", result)?; + Ok(EditorAgentResultAssets { + audios: Some(vec![resp_to_asset::editor_agent_generated_audio(&result)]), + ..Default::default() + }) + } +} + +#[cfg(test)] +mod tests { + use std::collections::HashMap; + + use platform_editor_agent::agent::asset::ImageMetadata; + use serde_json::json; + use spacetime_client::{EditorCanvasRecord, EditorCanvasViewportRecord}; + + use super::*; + use crate::editor_generation_config::load_editor_generation_pricing_from_paths; + + fn pricing() -> EditorGenerationPricingConfig { + load_editor_generation_pricing_from_paths(None).expect("default editor pricing should load") + } + + fn context_with_image(image_id: &str) -> EditorToolContext { + EditorToolContext { + images: HashMap::from([( + ImageId { + id: image_id.to_string(), + }, + ImageMetadata { + data_key: format!("generated/{image_id}.png"), + image_src: format!("/api/assets/read/generated/{image_id}.png"), + object_key: Some(format!("generated/{image_id}.png")), + thumbnail_src: Some(format!("/api/assets/read/generated/{image_id}-thumb.png")), + label: Some("测试图片".to_string()), + width: Some(640), + height: Some(480), + }, + )]), + } + } + + fn conversation() -> EditorAgentConversationRecord { + EditorAgentConversationRecord { + conversation_id: "conversation-1".to_string(), + project_id: "project-1".to_string(), + owner_user_id: "user-1".to_string(), + title: "测试会话".to_string(), + messages_object_key: "editor-agent/conversation-1.json".to_string(), + deleted: false, + created_at: "2026-07-23T00:00:00Z".to_string(), + updated_at: "2026-07-23T00:00:00Z".to_string(), + updated_at_micros: 0, + } + } + + fn project() -> EditorProjectRecord { + let viewport = EditorCanvasViewportRecord { + x: 0.0, + y: 0.0, + scale: 1.0, + }; + EditorProjectRecord { + project_id: "project-1".to_string(), + owner_user_id: "user-1".to_string(), + title: "测试工程".to_string(), + canvas: EditorCanvasRecord { + canvas_id: "canvas-1".to_string(), + project_id: "project-1".to_string(), + title: "测试画布".to_string(), + viewport: viewport.clone(), + layers: json!([]), + revision: 0, + layout_storage_version: 2, + background_color: None, + created_at: "2026-07-23T00:00:00Z".to_string(), + updated_at: "2026-07-23T00:00:00Z".to_string(), + }, + viewport, + layers: json!([]), + resources: Vec::new(), + created_at: "2026-07-23T00:00:00Z".to_string(), + updated_at: "2026-07-23T00:00:00Z".to_string(), + } + } + + #[test] + fn factory_resolves_every_editor_agent_tool_as_one_dyn_contract() { + let context = EditorToolContext::default(); + for name in [ + GenerateImageTool::NAME, + GenerateCharacterTool::NAME, + GenerateUiDesignTool::NAME, + EditImageTool::NAME, + GenerateIconSpritesheetTool::NAME, + GenerateVideoTool::NAME, + GenerateSoundEffectTool::NAME, + GenerateBackgroundMusicTool::NAME, + ] { + let tool = editor_agent_tool(name, &context).expect("registered tool should resolve"); + assert_eq!(tool.tool_name(), name); + } + assert!(editor_agent_tool("unknown-tool", &context).is_none()); + } + + #[test] + fn dyn_validation_deserializes_forwards_and_reserializes_typed_args() { + let ui = editor_agent_tool(GenerateUiDesignTool::NAME, &EditorToolContext::default()) + .expect("UI tool should resolve"); + let normalized = ui + .validate_args(&json!({ + "prompt": "生成游戏主界面", + "unknown": "drop-me" + })) + .expect("typed UI validation should pass"); + assert_eq!(normalized["model"], GPT_IMAGE_2_MODEL); + assert_eq!(normalized["aspect_ratio"], "1:1"); + assert_eq!(normalized["image_size"], "1K"); + assert!(normalized.get("unknown").is_none()); + + let music = editor_agent_tool( + GenerateBackgroundMusicTool::NAME, + &EditorToolContext::default(), + ) + .expect("music tool should resolve"); + let normalized = music + .validate_args(&json!({ + "prompt": "温暖舒缓的钢琴背景音乐", + "make_instrumental": false + })) + .expect("legacy music args should normalize"); + assert_eq!( + normalized["model"], + GenerateBackgroundMusicTool::DEFAULT_MODEL + ); + assert!(normalized.get("make_instrumental").is_none()); + + let sound = editor_agent_tool(GenerateSoundEffectTool::NAME, &EditorToolContext::default()) + .expect("sound tool should resolve"); + assert!( + sound + .validate_args(&json!({ "prompt": "按钮点击声", "duration": 11 })) + .is_err() + ); + } + + #[test] + fn dyn_validation_uses_the_context_bound_to_the_concrete_tool() { + let tool = editor_agent_tool(EditImageTool::NAME, &context_with_image("image-1")) + .expect("edit tool should resolve"); + let normalized = tool + .validate_args(&json!({ + "object_image_id": "image-1", + "prompt": "改成蓝色" + })) + .expect("bound image should pass typed edit validation"); + assert_eq!(normalized["object_image_id"], "image-1"); + + let missing = editor_agent_tool(EditImageTool::NAME, &EditorToolContext::default()) + .expect("edit tool should resolve"); + assert!(missing.validate_args(&normalized).is_err()); + } + + #[test] + fn dyn_pricing_keeps_all_existing_tool_formulas() { + let pricing = pricing(); + let context = context_with_image("image-1"); + let cases = [ + ( + GenerateImageTool::NAME, + json!({ "prompt": "图片", "model": platform_image::NANOBANANA_2_MODEL, "image_size": "2K" }), + 24, + ), + ( + GenerateCharacterTool::NAME, + json!({ "prompt": "角色", "model": platform_image::NANOBANANA_2_MODEL }), + 12, + ), + ( + GenerateUiDesignTool::NAME, + json!({ "prompt": "界面", "model": platform_image::GPT_IMAGE_2_MODEL, "image_size": "2K" }), + 5, + ), + ( + EditImageTool::NAME, + json!({ "object_image_id": "image-1", "prompt": "编辑" }), + 3, + ), + ( + GenerateIconSpritesheetTool::NAME, + json!({ "reference_image_id": "image-1", "icon_descriptions": ["背包"], "model": platform_image::NANOBANANA_2_MODEL, "image_size": "2K" }), + 24, + ), + ( + GenerateVideoTool::NAME, + json!({ "prompt": "视频", "model": "seedance2.0", "duration_seconds": 6, "resolution": "720p" }), + 144, + ), + ( + GenerateSoundEffectTool::NAME, + json!({ "prompt": "音效" }), + 5, + ), + ( + GenerateBackgroundMusicTool::NAME, + json!({ "prompt": "音乐" }), + 12, + ), + ]; + for (name, raw_args, expected) in cases { + let tool = editor_agent_tool(name, &context).expect("tool should resolve"); + let args = tool.validate_args(&raw_args).expect("args should validate"); + assert_eq!( + tool.pricing(&pricing, &args).expect("pricing should work"), + expected + ); + } + } + + #[test] + fn dyn_pricing_uses_the_supplied_runtime_snapshot() { + let mut pricing = pricing(); + pricing + .models + .get_mut(GPT_IMAGE_2_MODEL) + .expect("gpt image pricing should exist") + .prices + .insert("2K".to_string(), 37); + let tool = editor_agent_tool(GenerateImageTool::NAME, &EditorToolContext::default()) + .expect("image tool should resolve"); + let args = tool + .validate_args(&json!({ + "prompt": "生成图片", + "model": GPT_IMAGE_2_MODEL, + "image_size": "2K" + })) + .expect("image args should validate"); + assert_eq!( + tool.pricing(&pricing, &args).expect("pricing should work"), + 37 + ); + } + + #[test] + fn dyn_display_projection_uses_the_same_validated_json_and_pricing() { + let tool = editor_agent_tool(GenerateSoundEffectTool::NAME, &EditorToolContext::default()) + .expect("sound tool should resolve"); + let args = tool + .validate_args(&json!({ "prompt": "按钮点击声" })) + .expect("sound args should validate"); + let display = tool + .build_display_args(&args, &pricing()) + .expect("display projection should build"); + assert_eq!(display.extras.price_mud_points, 5); + assert_eq!(display.string_args[0].label, "音效描述"); + assert_eq!(display.string_args[0].value, "按钮点击声"); + assert_eq!(display.string_args[2].value, "5"); + } + + #[test] + fn dyn_display_projection_resolves_images_from_bound_context() { + let tool = editor_agent_tool(GenerateImageTool::NAME, &context_with_image("image-1")) + .expect("image tool should resolve"); + let args = tool + .validate_args(&json!({ + "prompt": "生成图片", + "reference_image_ids": ["image-1"] + })) + .expect("image args should validate"); + let display = tool + .build_display_args(&args, &pricing()) + .expect("display projection should resolve context image"); + + let image_ref = &display.image_args[0].refs[0]; + assert_eq!(image_ref.image_id, "image-1"); + assert_eq!( + image_ref.object_key.as_deref(), + Some("generated/image-1.png") + ); + assert_eq!(image_ref.label.as_deref(), Some("测试图片")); + assert_eq!(image_ref.width, Some(640)); + assert_eq!(image_ref.height, Some(480)); + } + + #[test] + fn dyn_formatter_and_asset_projection_forward_to_typed_audio_behavior() { + let tool = editor_agent_tool(GenerateSoundEffectTool::NAME, &EditorToolContext::default()) + .expect("sound tool should resolve"); + let args = tool + .validate_args(&json!({ "prompt": "按钮点击声" })) + .expect("sound args should validate"); + let result = json!({ + "ok": true, + "audioSrc": "/generated/click.mp3", + "objectKey": "generated/click.mp3", + "assetObjectId": "asset-1", + "width": 0, + "height": 0, + "sourceType": "generated", + "prompt": "按钮点击声", + "model": "audio1.0", + "provider": "vectorengine", + "taskId": "task-1", + "priceMudPoints": 5, + "audioKind": "sound-effect" + }); + let message = tool + .format_execute_message(&args, &result) + .expect("typed formatter should run"); + assert!(message.contains("generated audio saved as: /generated/click.mp3")); + let assets = tool + .result_assets(&result) + .expect("audio asset should project"); + assert_eq!( + assets.audios.expect("audio projection")[0].audio_src, + "/generated/click.mp3" + ); + } + + #[test] + fn dyn_job_preparation_keeps_queue_contract_and_defaults() { + let pricing = pricing(); + let conversation = conversation(); + let project = project(); + let tool = editor_agent_tool(GenerateSoundEffectTool::NAME, &EditorToolContext::default()) + .expect("sound tool should resolve"); + let args = tool + .validate_args(&json!({ "prompt": "按钮点击声" })) + .expect("sound args should validate"); + let job = tool + .prepare_job( + &args, + &EditorAgentPrepareJobContext { + conversation: &conversation, + project: &project, + message_id: 7, + pricing: &pricing, + }, + ) + .expect("sound job should prepare"); + assert_eq!(job.job_kind, EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND); + assert_eq!(job.request_label, "画布 Agent 生成音效"); + assert_eq!(job.price_mud_points, 5); + assert_eq!( + job.payload["duration"], + GenerateSoundEffectTool::DEFAULT_DURATION + ); + assert_eq!(job.payload["projectId"], "project-1"); + assert_eq!(job.payload["generationInputs"]["toolCallMessageId"], 7); + } +} diff --git a/server-rs/crates/api-server/src/editor_agent/tool_args.rs b/server-rs/crates/api-server/src/editor_agent/tool_args.rs deleted file mode 100644 index 7a2784dc9..000000000 --- a/server-rs/crates/api-server/src/editor_agent/tool_args.rs +++ /dev/null @@ -1,184 +0,0 @@ -use platform_editor_agent::agent::tools::context::EditorToolContext; -use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; -use platform_editor_agent::agent::tools::generate_background_music::{ - GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, -}; -use platform_editor_agent::agent::tools::generate_character::{ - GenerateCharacterTool, GenerateCharacterToolArgs, -}; -use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ - GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, -}; -use platform_editor_agent::agent::tools::generate_image::{ - GenerateImageTool, GenerateImageToolArgs, -}; -use platform_editor_agent::agent::tools::generate_sound_effect::{ - GenerateSoundEffectTool, GenerateSoundEffectToolArgs, -}; -use platform_editor_agent::agent::tools::generate_ui_design::{ - GenerateUiDesignTool, GenerateUiDesignToolArgs, -}; -use platform_editor_agent::agent::tools::generate_video::{ - GenerateVideoTool, GenerateVideoToolArgs, -}; -use platform_editor_agent::framework::error::PromptError; -use platform_editor_agent::framework::tool::Tool; -use serde::Serialize; -use serde::de::DeserializeOwned; -use serde_json::Value; - -pub fn normalize_tool_args( - tool_name: &str, - args: &Value, - context: &EditorToolContext, -) -> Result { - match tool_name { - GenerateImageTool::NAME => { - let args: GenerateImageToolArgs = parse_args(tool_name, args)?; - let tool = GenerateImageTool { - context: context.clone(), - }; - tool.validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - GenerateCharacterTool::NAME => { - let args: GenerateCharacterToolArgs = parse_args(tool_name, args)?; - let tool = GenerateCharacterTool { - context: context.clone(), - }; - tool.validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - GenerateUiDesignTool::NAME => { - let args: GenerateUiDesignToolArgs = parse_args(tool_name, args)?; - let tool = GenerateUiDesignTool { - context: context.clone(), - }; - tool.validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - EditImageTool::NAME => { - let args: EditImageToolArgs = parse_args(tool_name, args)?; - let tool = EditImageTool { - context: context.clone(), - }; - if let Some(error) = tool.validate_args(&args) { - return Err(invalid_args(tool_name, error)); - } - serialize_args(tool_name, args) - } - GenerateIconSpritesheetTool::NAME => { - let args: GenerateIconSpritesheetToolArgs = parse_args(tool_name, args)?; - let tool = GenerateIconSpritesheetTool { - context: context.clone(), - }; - tool.validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - GenerateVideoTool::NAME => { - let args: GenerateVideoToolArgs = parse_args(tool_name, args)?; - let tool = GenerateVideoTool { - context: context.clone(), - }; - tool.validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - GenerateSoundEffectTool::NAME => { - let args: GenerateSoundEffectToolArgs = parse_args(tool_name, args)?; - GenerateSoundEffectTool - .validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - GenerateBackgroundMusicTool::NAME => { - let args: GenerateBackgroundMusicToolArgs = parse_args(tool_name, args)?; - GenerateBackgroundMusicTool - .validate_args(&args) - .map_err(|error| invalid_args(tool_name, error))?; - serialize_args(tool_name, args) - } - _ => Err(PromptError::ToolError(format!( - "unsupported editor agent tool: {tool_name}" - ))), - } -} - -fn parse_args(tool_name: &str, args: &Value) -> Result { - serde_json::from_value(args.clone()).map_err(|error| { - PromptError::ToolError(format!("invalid args for tool {tool_name}: {error}")) - }) -} - -fn serialize_args(tool_name: &str, args: T) -> Result { - serde_json::to_value(args).map_err(|error| { - PromptError::InternalError(format!( - "failed to normalize args for tool {tool_name}: {error}" - )) - }) -} - -fn invalid_args(tool_name: &str, error: impl std::fmt::Display) -> PromptError { - PromptError::ToolError(format!("invalid args for tool {tool_name}: {error}")) -} - -#[cfg(test)] -mod tests { - use super::*; - use platform_audio::SUNO_DEFAULT_MODEL; - use platform_image::GPT_IMAGE_2_MODEL; - use serde_json::json; - - #[test] - fn normalizes_defaults_and_drops_unknown_fields_before_persistence() { - let normalized = normalize_tool_args( - GenerateUiDesignTool::NAME, - &json!({ - "prompt": "生成游戏主界面", - "unknown": "drop-me" - }), - &EditorToolContext::default(), - ) - .expect("UI 参数应使用工具默认模型并完成规范化"); - - assert_eq!(normalized["model"], GPT_IMAGE_2_MODEL); - assert_eq!(normalized["aspect_ratio"], "1:1"); - assert_eq!(normalized["image_size"], "1K"); - assert!(normalized.get("unknown").is_none()); - } - - #[test] - fn drops_legacy_background_music_make_instrumental_field() { - let normalized = normalize_tool_args( - GenerateBackgroundMusicTool::NAME, - &json!({ - "prompt": "温暖舒缓的钢琴背景音乐", - "model": SUNO_DEFAULT_MODEL, - "make_instrumental": false - }), - &EditorToolContext::default(), - ) - .expect("旧背景音乐字段应被安全忽略"); - - assert!(normalized.get("make_instrumental").is_none()); - } - - #[test] - fn rejects_invalid_sound_duration_before_persistence() { - let error = normalize_tool_args( - GenerateSoundEffectTool::NAME, - &json!({ - "prompt": "按钮点击声", - "duration": 11 - }), - &EditorToolContext::default(), - ) - .expect_err("非法音效时长必须在待确认消息持久化前失败"); - - assert!(error.to_string().contains("11")); - } -} diff --git a/server-rs/crates/platform-editor-agent/src/agent/asset.rs b/server-rs/crates/platform-editor-agent/src/agent/asset.rs index 0f9bced59..f35c6c69f 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/asset.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/asset.rs @@ -38,6 +38,11 @@ impl Display for ImageId { #[derive(Debug, Clone, Serialize, Deserialize)] pub struct ImageMetadata { - pub tag: String, pub data_key: String, + pub image_src: String, + pub object_key: Option, + pub thumbnail_src: Option, + pub label: Option, + pub width: Option, + pub height: Option, } diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs index 2cd2aebfd..ef1a6d443 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs @@ -10,10 +10,13 @@ impl EditorToolContext { } pub fn image_data_key(&self, image_id: &ImageId) -> Option<&str> { - self.images - .get(image_id) + self.image_metadata(image_id) .map(|metadata| metadata.data_key.as_str()) } + + pub fn image_metadata(&self, image_id: &ImageId) -> Option<&ImageMetadata> { + self.images.get(image_id) + } } #[derive(Debug, Clone, Default, Serialize, Deserialize)] diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs index e3fa88810..801c2816c 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs @@ -26,9 +26,9 @@ pub struct GenerateCharacterToolArgs { #[serde(default)] pub reference_image_ids: Vec, #[serde(default = "default_image_aspect_ratio")] - pub aspect_ratio: Option, + pub aspect_ratio: String, #[serde(default = "default_image_size")] - pub image_size: Option, + pub image_size: String, } impl Tool for GenerateCharacterTool { @@ -90,8 +90,8 @@ impl GenerateCharacterTool { ) -> Result<(), GenerateImageError> { validate_image_generation_options( args.model.as_str(), - args.aspect_ratio.as_deref(), - args.image_size.as_deref(), + args.aspect_ratio.as_str(), + args.image_size.as_str(), )?; if args.prompt.trim().is_empty() { return Err(GenerateImageError::PromptNotProvided); diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs index fde5ba420..180def213 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs @@ -78,9 +78,9 @@ pub struct GenerateIconSpritesheetToolArgs { pub reference_image_ids: Vec, pub icon_descriptions: Vec, #[serde(default = "default_image_aspect_ratio")] - pub aspect_ratio: Option, + pub aspect_ratio: String, #[serde(default = "default_image_size")] - pub image_size: Option, + pub image_size: String, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -179,8 +179,8 @@ impl GenerateIconSpritesheetTool { ) -> Result<(), GenerateIconSpritesheetError> { validate_image_generation_options( args.model.as_str(), - args.aspect_ratio.as_deref(), - args.image_size.as_deref(), + args.aspect_ratio.as_str(), + args.image_size.as_str(), )?; if args.reference_image_id.id.trim().is_empty() { return Err(GenerateIconSpritesheetError::ReferenceNotProvided); @@ -252,8 +252,13 @@ mod tests { images: HashMap::from([( reference_image_id.clone(), ImageMetadata { - tag: "image".to_string(), data_key: "asset://reference-image".to_string(), + image_src: "asset://reference-image".to_string(), + object_key: None, + thumbnail_src: None, + label: None, + width: None, + height: None, }, )]), }, @@ -263,8 +268,8 @@ mod tests { model: NANOBANANA_2_MODEL.to_string(), reference_image_ids: Vec::new(), icon_descriptions, - aspect_ratio: Some("1:1".to_string()), - image_size: Some("1K".to_string()), + aspect_ratio: "1:1".to_string(), + image_size: "1K".to_string(), }; (tool, args) } diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs index 2e3e44386..e2700b17a 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs @@ -74,9 +74,9 @@ pub struct GenerateImageToolArgs { #[serde(default)] pub reference_image_ids: Vec, #[serde(default = "default_image_aspect_ratio")] - pub aspect_ratio: Option, + pub aspect_ratio: String, #[serde(default = "default_image_size")] - pub image_size: Option, + pub image_size: String, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -170,8 +170,8 @@ impl GenerateImageTool { pub fn validate_args(&self, args: &GenerateImageToolArgs) -> Result<(), GenerateImageError> { validate_image_generation_options( args.model.as_str(), - args.aspect_ratio.as_deref(), - args.image_size.as_deref(), + args.aspect_ratio.as_str(), + args.image_size.as_str(), )?; if args.prompt.trim().is_empty() { return Err(GenerateImageError::PromptNotProvided); diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs index 9496003bd..6c9c22a53 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs @@ -45,15 +45,15 @@ pub struct GenerateSoundEffectToolArgs { #[serde(default = "default_sound_effect_model")] pub model: String, #[serde(default = "default_sound_effect_duration")] - pub duration: Option, + pub duration: u8, } fn default_sound_effect_model() -> String { VIDU_AUDIO_MODEL.to_string() } -fn default_sound_effect_duration() -> Option { - Some(GenerateSoundEffectTool::DEFAULT_DURATION) +fn default_sound_effect_duration() -> u8 { + GenerateSoundEffectTool::DEFAULT_DURATION } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -120,10 +120,8 @@ impl GenerateSoundEffectTool { if args.model != VIDU_AUDIO_MODEL { return Err(GenerateSoundEffectError::InvalidModel(args.model.clone())); } - if let Some(duration) = args.duration - && !Self::SUPPORTED_DURATIONS.contains(&duration) - { - return Err(GenerateSoundEffectError::InvalidDuration(duration)); + if !Self::SUPPORTED_DURATIONS.contains(&args.duration) { + return Err(GenerateSoundEffectError::InvalidDuration(args.duration)); } Ok(()) } @@ -146,7 +144,7 @@ impl GenerateSoundEffectTool { mod tests { use super::*; - fn args(duration: Option) -> GenerateSoundEffectToolArgs { + fn args(duration: u8) -> GenerateSoundEffectToolArgs { GenerateSoundEffectToolArgs { prompt: "按钮点击声".to_string(), model: GenerateSoundEffectTool::DEFAULT_MODEL.to_string(), @@ -157,11 +155,10 @@ mod tests { #[test] fn validates_schema_durations_and_default() { assert_eq!(GenerateSoundEffectTool::DEFAULT_DURATION, 5); - assert!(GenerateSoundEffectTool.validate_args(&args(None)).is_ok()); for duration in GenerateSoundEffectTool::SUPPORTED_DURATIONS { assert!( GenerateSoundEffectTool - .validate_args(&args(Some(*duration))) + .validate_args(&args(*duration)) .is_ok() ); } @@ -171,7 +168,7 @@ mod tests { async fn planning_rejects_durations_outside_schema() { for duration in [1, 11, u8::MAX] { assert!(matches!( - GenerateSoundEffectTool.call(args(Some(duration))).await, + GenerateSoundEffectTool.call(args(duration)).await, Err(GenerateSoundEffectError::InvalidDuration(value)) if value == duration )); } diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs index 152d9a99a..bf12b3192 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs @@ -26,9 +26,9 @@ pub struct GenerateUiDesignToolArgs { #[serde(default)] pub reference_image_ids: Vec, #[serde(default = "default_image_aspect_ratio")] - pub aspect_ratio: Option, + pub aspect_ratio: String, #[serde(default = "default_image_size")] - pub image_size: Option, + pub image_size: String, } fn default_ui_design_model() -> String { @@ -100,8 +100,8 @@ impl GenerateUiDesignTool { } validate_image_generation_options( args.model.as_str(), - args.aspect_ratio.as_deref(), - args.image_size.as_deref(), + args.aspect_ratio.as_str(), + args.image_size.as_str(), )?; if args.prompt.trim().is_empty() { return Err(GenerateImageError::PromptNotProvided); @@ -151,8 +151,8 @@ mod tests { prompt: "生成游戏主界面".to_string(), model: model.to_string(), reference_image_ids: Vec::new(), - aspect_ratio: Some("16:9".to_string()), - image_size: Some("1K".to_string()), + aspect_ratio: "16:9".to_string(), + image_size: "1K".to_string(), } } diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs index 21935959e..d3e800839 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs @@ -67,35 +67,35 @@ pub struct GenerateVideoToolArgs { #[serde(default)] pub reference_image_ids: Vec, #[serde(default = "default_video_aspect_ratio")] - pub aspect_ratio: Option, + pub aspect_ratio: String, #[serde(default = "default_video_duration_seconds")] - pub duration_seconds: Option, + pub duration_seconds: u32, #[serde(default = "default_video_model")] pub model: String, #[serde(default = "default_video_resolution")] - pub resolution: Option, + pub resolution: String, #[serde(default = "default_video_sound")] - pub sound: Option, + pub sound: String, } fn default_video_model() -> String { GenerateVideoTool::DEFAULT_VIDEO_MODEL.to_string() } -fn default_video_aspect_ratio() -> Option { - Some(GenerateVideoTool::DEFAULT_VIDEO_ASPECT_RATIO.to_string()) +fn default_video_aspect_ratio() -> String { + GenerateVideoTool::DEFAULT_VIDEO_ASPECT_RATIO.to_string() } -fn default_video_duration_seconds() -> Option { - Some(GenerateVideoTool::DEFAULT_VIDEO_DURATION_SECONDS) +fn default_video_duration_seconds() -> u32 { + GenerateVideoTool::DEFAULT_VIDEO_DURATION_SECONDS } -fn default_video_resolution() -> Option { - Some(GenerateVideoTool::DEFAULT_VIDEO_RESOLUTION.to_string()) +fn default_video_resolution() -> String { + GenerateVideoTool::DEFAULT_VIDEO_RESOLUTION.to_string() } -fn default_video_sound() -> Option { - Some(GenerateVideoTool::DEFAULT_VIDEO_SOUND.to_string()) +fn default_video_sound() -> String { + GenerateVideoTool::DEFAULT_VIDEO_SOUND.to_string() } #[derive(Debug, Clone, Serialize, Deserialize)] pub struct GenerateVideoToolOutput { @@ -202,31 +202,23 @@ impl GenerateVideoTool { if !Self::SUPPORTED_VIDEO_MODELS.contains(&args.model.as_str()) { return Err(GenerateVideoError::InvalidModel(args.model.clone())); } - let aspect_ratio = args - .aspect_ratio - .as_deref() - .unwrap_or(Self::DEFAULT_VIDEO_ASPECT_RATIO); + let aspect_ratio = args.aspect_ratio.as_str(); if !Self::SUPPORTED_ASPECT_RATIOS.contains(&aspect_ratio) { return Err(GenerateVideoError::InvalidAspectRatio( aspect_ratio.to_string(), )); } - let duration_seconds = args - .duration_seconds - .unwrap_or(Self::DEFAULT_VIDEO_DURATION_SECONDS); + let duration_seconds = args.duration_seconds; if !Self::SUPPORTED_DURATION_SECONDS.contains(&duration_seconds) { return Err(GenerateVideoError::InvalidDurationSeconds(duration_seconds)); } - let resolution = args - .resolution - .as_deref() - .unwrap_or(Self::DEFAULT_VIDEO_RESOLUTION); + let resolution = args.resolution.as_str(); if !Self::SUPPORTED_RESOLUTIONS.contains(&resolution) { return Err(GenerateVideoError::InvalidResolution( resolution.to_string(), )); } - let sound = args.sound.as_deref().unwrap_or(Self::DEFAULT_VIDEO_SOUND); + let sound = args.sound.as_str(); if !Self::SUPPORTED_SOUND_OPTIONS.contains(&sound) { return Err(GenerateVideoError::InvalidSound(sound.to_string())); } @@ -281,11 +273,11 @@ mod tests { GenerateVideoToolArgs { prompt: "镜头向前推进".to_string(), reference_image_ids: Vec::new(), - aspect_ratio: Some("16:9".to_string()), - duration_seconds: Some(4), + aspect_ratio: "16:9".to_string(), + duration_seconds: 4, model: GenerateVideoTool::SEEDANCE_2_FAST_MODEL.to_string(), - resolution: Some("720p".to_string()), - sound: Some("on".to_string()), + resolution: "720p".to_string(), + sound: "on".to_string(), } } @@ -298,28 +290,28 @@ mod tests { #[test] fn validates_each_video_option() { let mut invalid_aspect_ratio = args(); - invalid_aspect_ratio.aspect_ratio = Some("2:1".to_string()); + invalid_aspect_ratio.aspect_ratio = "2:1".to_string(); assert!(matches!( tool().validate_args(&invalid_aspect_ratio), Err(GenerateVideoError::InvalidAspectRatio(value)) if value == "2:1" )); let mut invalid_duration = args(); - invalid_duration.duration_seconds = Some(16); + invalid_duration.duration_seconds = 16; assert!(matches!( tool().validate_args(&invalid_duration), Err(GenerateVideoError::InvalidDurationSeconds(16)) )); let mut invalid_resolution = args(); - invalid_resolution.resolution = Some("4K".to_string()); + invalid_resolution.resolution = "4K".to_string(); assert!(matches!( tool().validate_args(&invalid_resolution), Err(GenerateVideoError::InvalidResolution(value)) if value == "4K" )); let mut invalid_sound = args(); - invalid_sound.sound = Some("auto".to_string()); + invalid_sound.sound = "auto".to_string(); assert!(matches!( tool().validate_args(&invalid_sound), Err(GenerateVideoError::InvalidSound(value)) if value == "auto" @@ -329,7 +321,7 @@ mod tests { #[test] fn rejects_invalid_model_combinations_and_reference_count() { let mut invalid = args(); - invalid.resolution = Some("1080p".to_string()); + invalid.resolution = "1080p".to_string(); assert!(matches!( tool().validate_args(&invalid), Err(GenerateVideoError::UnsupportedModelResolution { model, resolution }) diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/image_generation_options.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/image_generation_options.rs index 8cf725664..a76b2e3d2 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/image_generation_options.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/image_generation_options.rs @@ -43,28 +43,26 @@ pub fn default_image_model() -> String { NANOBANANA_2_MODEL.to_string() } -pub fn default_image_aspect_ratio() -> Option { - Some(DEFAULT_IMAGE_ASPECT_RATIO.to_string()) +pub fn default_image_aspect_ratio() -> String { + DEFAULT_IMAGE_ASPECT_RATIO.to_string() } -pub fn default_image_size() -> Option { - Some(DEFAULT_IMAGE_SIZE.to_string()) +pub fn default_image_size() -> String { + DEFAULT_IMAGE_SIZE.to_string() } pub fn validate_image_generation_options( model: &str, - aspect_ratio: Option<&str>, - image_size: Option<&str>, + aspect_ratio: &str, + image_size: &str, ) -> Result<(), ImageGenerationOptionsError> { let supported_sizes = supported_image_sizes(model) .ok_or_else(|| ImageGenerationOptionsError::InvalidModel(model.to_string()))?; - let aspect_ratio = aspect_ratio.unwrap_or(DEFAULT_IMAGE_ASPECT_RATIO); if !SUPPORTED_IMAGE_ASPECT_RATIOS.contains(&aspect_ratio) { return Err(ImageGenerationOptionsError::InvalidAspectRatio( aspect_ratio.to_string(), )); } - let image_size = image_size.unwrap_or(DEFAULT_IMAGE_SIZE); if !supported_sizes.contains(&image_size) { return Err(ImageGenerationOptionsError::InvalidImageSize { model: model.to_string(), @@ -120,30 +118,26 @@ mod tests { assert!( validate_image_generation_options( NANOBANANA_2_MODEL, - Some(aspect_ratio), - Some(image_size), + aspect_ratio, + image_size, ) .is_ok() ); } for image_size in GPT_IMAGE_2_IMAGE_SIZES { assert!( - validate_image_generation_options( - GPT_IMAGE_2_MODEL, - Some(aspect_ratio), - Some(image_size), - ) - .is_ok() + validate_image_generation_options(GPT_IMAGE_2_MODEL, aspect_ratio, image_size,) + .is_ok() ); } } assert!(matches!( - validate_image_generation_options(GPT_IMAGE_2_MODEL, Some("1:1"), Some("0.5K")), + validate_image_generation_options(GPT_IMAGE_2_MODEL, "1:1", "0.5K"), Err(ImageGenerationOptionsError::InvalidImageSize { .. }) )); assert!(matches!( - validate_image_generation_options(NANOBANANA_2_MODEL, Some("21:9"), Some("1K")), + validate_image_generation_options(NANOBANANA_2_MODEL, "21:9", "1K"), Err(ImageGenerationOptionsError::InvalidAspectRatio(_)) )); } diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs index 5aa5a3537..5c0436723 100644 --- a/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs @@ -66,19 +66,53 @@ mod tests { .expect("music args should deserialize"); assert_eq!(image.model, NANOBANANA_2_MODEL); - assert_eq!(image.aspect_ratio.as_deref(), Some("1:1")); - assert_eq!(image.image_size.as_deref(), Some("1K")); + assert_eq!(image.aspect_ratio, "1:1"); + assert_eq!(image.image_size, "1K"); assert_eq!(edit.model, GPT_IMAGE_2_MODEL); assert_eq!(character.model, NANOBANANA_2_MODEL); + assert_eq!(character.aspect_ratio, "1:1"); + assert_eq!(character.image_size, "1K"); assert_eq!(ui_design.model, GPT_IMAGE_2_MODEL); + assert_eq!(ui_design.aspect_ratio, "1:1"); + assert_eq!(ui_design.image_size, "1K"); assert_eq!(icon.model, NANOBANANA_2_MODEL); + assert_eq!(icon.aspect_ratio, "1:1"); + assert_eq!(icon.image_size, "1K"); assert_eq!(video.model, GenerateVideoTool::DEFAULT_VIDEO_MODEL); - assert_eq!(video.sound.as_deref(), Some("on")); + assert_eq!(video.aspect_ratio, "16:9"); + assert_eq!(video.duration_seconds, 4); + assert_eq!(video.resolution, "720p"); + assert_eq!(video.sound, "on"); assert_eq!(sound.model, VIDU_AUDIO_MODEL); - assert_eq!(sound.duration, Some(5)); + assert_eq!(sound.duration, 5); assert_eq!(music.model, SUNO_DEFAULT_MODEL); } + #[test] + fn default_backed_tool_args_reject_null() { + assert!( + serde_json::from_value::(json!({ + "prompt": "生成森林场景", + "aspect_ratio": null + })) + .is_err() + ); + assert!( + serde_json::from_value::(json!({ + "prompt": "镜头向前推进", + "duration_seconds": null + })) + .is_err() + ); + assert!( + serde_json::from_value::(json!({ + "prompt": "按钮点击声", + "duration": null + })) + .is_err() + ); + } + #[test] fn generate_image_accepts_nanobanana_and_image2_only() { let tool = GenerateImageTool { @@ -88,8 +122,8 @@ mod tests { prompt: "生成森林场景".to_string(), model: model.to_string(), reference_image_ids: Vec::new(), - aspect_ratio: None, - image_size: None, + aspect_ratio: "1:1".to_string(), + image_size: "1K".to_string(), }; assert!(tool.validate_args(&args(NANOBANANA_2_MODEL)).is_ok()); @@ -113,8 +147,8 @@ mod tests { prompt: "生成角色".to_string(), model: NANOBANANA_2_MODEL.to_string(), reference_image_ids: vec![missing_image.clone()], - aspect_ratio: Some("2:3".to_string()), - image_size: Some("1K".to_string()), + aspect_ratio: "2:3".to_string(), + image_size: "1K".to_string(), }; assert!(matches!( @@ -128,8 +162,8 @@ mod tests { prompt: "生成游戏主界面".to_string(), model: GPT_IMAGE_2_MODEL.to_string(), reference_image_ids: vec![missing_image.clone()], - aspect_ratio: Some("16:9".to_string()), - image_size: Some("1K".to_string()), + aspect_ratio: "16:9".to_string(), + image_size: "1K".to_string(), }; assert!(matches!( GenerateUiDesignTool { diff --git a/server-rs/crates/platform-editor-agent/src/framework/run.rs b/server-rs/crates/platform-editor-agent/src/framework/run.rs index 7dd7b036e..2acd2dc6e 100644 --- a/server-rs/crates/platform-editor-agent/src/framework/run.rs +++ b/server-rs/crates/platform-editor-agent/src/framework/run.rs @@ -14,7 +14,7 @@ pub type TextOutput = String; #[derive(Debug, Clone)] pub struct ToolCallOutput { pub tool_call: ToolCall, - pub message: String, + pub output: Value, } #[derive(Debug, Clone)] pub enum PromptOutput { @@ -35,6 +35,20 @@ pub enum ToolCallFlow { Stop, } +pub fn format_tool_call_message( + tool_call_id: impl std::fmt::Display, + args: &Value, + output: &Value, +) -> Result { + let arg_json = serde_json::to_string(args) + .map_err(|error| PromptError::InternalError(error.to_string()))?; + let output_json = serde_json::to_string(output) + .map_err(|error| PromptError::InternalError(error.to_string()))?; + Ok(format!( + "[tool_call:{tool_call_id}] args: {arg_json} output: {output_json}" + )) +} + pub struct PromptRequest<'a, M: LlmApiAdaptor + 'a, Message: 'a> { agent: &'a mut Agent, message: Message, @@ -190,7 +204,7 @@ where match hook.after_tool_call(&tc.name, &mut json_output) { ToolCallFlow::Stop => { return Err(PromptError::ToolError( - "tool call output caused this _turn stopped by hook".to_string(), + "tool call output caused this turn stopped by hook".to_string(), )); } ToolCallFlow::Skip => { @@ -201,20 +215,14 @@ where ToolCallFlow::Continue => {} } } - let arg_json = serde_json::to_string(&tc.args) - .map_err(|e| PromptError::InternalError(e.to_string()))?; - let output_json = serde_json::to_string(&json_output) - .map_err(|e| PromptError::InternalError(e.to_string()))?; - - let overall_message = format!( - "[tool_call:{tc_id}] args: {arg_json} output: {output_json}" - ); + let overall_message = + format_tool_call_message(tc_id, &tc.args, &json_output)?; let msg = agent.model.tool_result_message(&tc.name, &overall_message); memory.append_message(msg); prompt_result.push(Tool(ToolCallOutput { tool_call: tc.clone(), - message: overall_message.clone(), + output: json_output, })) } ToolOutcome::InternalError(failure) if failure.fatal => { @@ -381,7 +389,22 @@ mod tests { assert_eq!(completion_count.load(Ordering::SeqCst), 1); assert_eq!(outputs.len(), 2); assert!(matches!(outputs[0], PromptOutput::Text(_))); - assert!(matches!(outputs[1], PromptOutput::Tool(_))); + let PromptOutput::Tool(tool_output) = &outputs[1] else { + panic!("second output should be a tool call"); + }; + assert_eq!( + tool_output.output, + json!({ "message": "pending user confirmation" }) + ); + assert_eq!( + format_tool_call_message( + "test-tool", + &tool_output.tool_call.args, + &tool_output.output, + ) + .expect("tool call message should format"), + "[tool_call:test-tool] args: {\"prompt\":\"生成一张图\"} output: {\"message\":\"pending user confirmation\"}" + ); } #[tokio::test]