GenarrativeAI/Genarrative
11
0
Fork 0
Code Issues 18 Pull Requests 4 Actions Packages Projects 3 Releases Wiki Activity

refactor: remove duplicated switch

Browse Source
This commit is contained in:
王德宇
2026-07-23 12:23:38 +08:00
parent c1378ff833
commit abd98e7375
21 changed files with 1890 additions and 1544 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/project-memory/shared-memory/decision-log.md
+10
View File
@@ -16,6 +16,16 @@
---
## 2026-07-23 画布 Agent 工具生命周期统一经 object-safe trait 分派
- 背景：画布 Agent 八类工具的参数规范化、确认展示、计价与 worker payload、完成结果格式化和媒体投影分别在 `tool_args.rs`、`display_args.rs`、`api.rs`、`reconcile.rs` 重复按工具名分派；新增或调整工具时容易漏改其中一处。
- 决策：api-server 以 object-safe `EditorAgentTool: ToolDyn` 取代仅承载计价的 `EditorAgentPricedTool`。trait 的所有动态方法统一接收 `serde_json::Value`；每个具体工具实现自行反序列化为真实 Args / 结果，`validate_args` 与 `format_execute_message` 显式转发到 `platform-editor-agent` 已有强类型实现，再把规范 Args、展示投影、job payload、完成文本或媒体引用擦除回公共类型。`editor_agent_tool(toolName, context)` 绑定当前 `EditorToolContext` 并作为唯一八分支工具名分派；规划、确认和回填不得再维护平行 switch。LLM builder 的工具注册列表保持独立显式维护。
- 边界：不改变工具名、LLM schema、OSS 消息文档、`displayArgs`、模型定价、job kind / payload、dedupe key、worker、计费、完成消息或图片 / 视频 / 音频引用契约，不涉及前端、SpacetimeDB schema 或迁移。
- 影响范围：`server-rs/crates/api-server/src/editor_agent` 的工具 trait、参数规范化、确认入队与终态回填，以及画布 Agent 专题文档。
- 验证方式：覆盖八类 factory 与 dyn validation / pricing / display / job / formatter / media projection 的 api-server 定向测试，运行 `cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_agent`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`npm run check:rustfmt`、`npm run check:encoding` 和 `git diff --check`。
---
## 2026-07-20 角色动作抠图前禁止透明 padding
- 背景：图片画布角色动作此前在 BgFilter 前复用最终帧 finalizer，把 FFmpeg 抽帧先转成目标尺寸 RGBA 画布并用透明黑像素补边；透明区域进入 BgFilter、阿里云和本地键色共同读取的 OSS 源帧后，会干扰主体边缘判断并降低抠图质量。
docs/【编辑器】画布Agent对话面板-2026-07-03.md
+3 -2
View File
@@ -85,12 +85,13 @@
- 确认接口必须先把工具参数转换为既有编辑器 worker payload，再使用 `editor-agent:{conversationId}:{messageId}:{toolName}` 稳定 dedupe key 入队；同一确认的请求重试只能得到同一个 external job。入队成功后把返回的 job id 写回同一条 OSS 工具消息，不新增 Agent 工具执行关联表。
- 前端根据 `externalJobId` 查询通用 external-generation job 状态；worker 继续通过 `canvasCompletion` 把生成结果写回工程与素材库。浏览器断线、刷新或 api-server 重启不得导致确认接口重新扣费或重新提交 provider。
- `GET /conversation` 会在同一个 conversation lock 内扫描 `status=not_completed` 且已有 `externalJobId` 的工具消息：只对这些消息按 job id 定向读取主任务；任务完成后复用对应工具的 `format_execute_message` 替换 system text、回填轻量图片 / 视频 / 音频引用并写为 `completed`，任务失败则回填 `error` 并写为 `failed`。任务结果读取或 completed payload 解析 / formatter 回填失败时，必须在同一次 GET 内完成首次尝试及最多 3 次重试，三次重试各间隔 100ms 并重新读取任务结果；仍失败才把该工具消息写为 `failed` 并保存最后错误。该重试不依赖前端再次刷新。排队和执行中都保持 `not_completed`，整轮扫描结果一次性写回 OSS。
- `EditorAgentToolCall.args` 的正式持久化契约是**校验后的规范参数 JSON**，不是 LLM 返回的原始 JSON。api-server 收到工具调用后，必须先按已注册的 ToolArgs 反序列化、补齐字段默认值、删除未进入 ToolArgs 的未知 / 退役字段、执行工具参数校验，再重新序列化并写入 `args`；校验失败的调用不得持久化为待确认消息。LLM 原始参数只作为本次规范化的瞬时输入，不作为执行或审计真相；确认、取消、任务回填与后续上下文统一读取同一条消息中的规范 `args`。图片参数继续只保存由真实 data key 计算出的 opaque SHA-256 `imageId`；不得为了前端预览把 `args` 中的图片 ID 改写成 `objectKey`、URL 或展示对象，也不得由前端重组或回传一份新的执行参数。
- `EditorAgentToolCall.args` 的正式持久化契约是**校验后的规范参数 JSON**，不是 LLM 返回的原始 JSON。api-server 收到工具调用后，必须先按已注册的 ToolArgs 反序列化、补齐字段默认值、删除未进入 ToolArgs 的未知 / 退役字段、执行工具参数校验，再重新序列化并写入 `args`；校验失败的调用不得持久化为待确认消息。所有有明确默认值的工具标量参数在强类型 ToolArgs 中必须使用非 `Option` 字段：调用方省略时由 Serde 补默认值，显式 `null` 视为非法参数，后续计价、确认展示和 job payload 不得再次使用 `unwrap_or` 补同一默认值。LLM 原始参数只作为本次规范化的瞬时输入，不作为执行或审计真相；确认、取消、任务回填与后续上下文统一读取同一条消息中的规范 `args`。图片参数继续只保存由真实 data key 计算出的 opaque SHA-256 `imageId`；不得为了前端预览把 `args` 中的图片 ID 改写成 `objectKey`、URL 或展示对象，也不得由前端重组或回传一份新的执行参数。
- api-server 内画布 Agent 工具统一实现 object-safe `EditorAgentTool: ToolDyn`。`validate_args`、计价、确认展示、worker job 构建、`format_execute_message` 和结果媒体投影都使用统一 JSON 边界；每个具体工具实现负责把 JSON 反序列化为自己的强类型 Args / 结果，并把校验与完成消息格式化转发到 `platform-editor-agent` 中既有的 typed `validate_args` / `format_execute_message`，不得在调用方复制工具规则。`editor_agent_tool(toolName, context)` 是唯一按工具名分派的位置，规划、确认和任务回填只调用返回的 dyn tool；新增工具必须补齐同一个 trait 实现和该工厂分支。LLM builder 的 `.tool(...)` 注册列表仍是独立显式清单，不属于本次动态分派。framework runner 必须在 `ToolCallOutput` 中保留工具返回的结构化 output；runner 写入 LLM memory 与 api-server 使用规范参数持久化 system text 时统一调用公开的 `format_tool_call_message`，不得丢弃 `TOOL_CALL_PENDING_MESSAGE` 后自行拼另一套“等待确认”输出。
- `EditorAgentToolCall.displayArgs` 是必填、只读的用户确认展示投影，与 `args` 分离：
- `stringArgs` 保存提示词、比例、清晰度、模型、时长等可展示参数的稳定名称、用户可见标题和值；前端渲染模型字段时复用图片编辑器公共展示名映射，`gemini-3.1-flash-image-preview` 显示为 `nanobanana2`、`audio1.0` 显示为 `Vidu`、`chirp-v5` 显示为 `Suno`，视频模型显示现有产品标签，不得改写后端参数真相；
- `imageArgs` 按“目标图片 / 参考图片”等参数分组，每个 `refs` 项包含与规范参数对应的 `imageId`，以及后端从已校验会话上下文解析出的 `objectKey`、`imageSrc`、可选 `thumbnailSrc` / `label` / `width` / `height`。
- `extras.priceMudPoints` 保存创建待确认消息时按后端运行时模型定价快照计算的预计泥点消耗；前端统一展示为“预计消耗 N泥点”，不自行计算价格。
- `displayArgs` 只能由 api-server 按已注册 tool 白名单，基于已经通过 ToolArgs 校验的 `args` 和当前 OSS 会话文档中的附件 / 历史生成结果构建；不能信任 LLM 自报的展示地址、标题或素材元数据。展示投影不参与确认执行，确认接口仍只读取同一条持久化 tool call 的 `args`，避免“看到的素材”和“实际执行的素材”分叉。
- `displayArgs` 只能由 api-server 按已注册 tool 白名单，基于已经通过 ToolArgs 校验的 `args` 和当前请求开始时从 OSS 会话文档一次性构建的 `EditorToolContext` 生成；该 context 必须按 opaque `ImageId` 同时保存执行所需的 `dataKey` 与展示所需的图片地址、Object Key、缩略图、label、宽高，参数校验、确认展示和 job payload 统一查同一份 context。不能信任 LLM 自报的展示地址、标题或素材元数据。展示投影不参与确认执行，确认接口仍只读取同一条持久化 tool call 的 `args`，避免“看到的素材”和“实际执行的素材”分叉。
- `extras.priceMudPoints` 同样只属于展示投影，不作为扣费输入；确认后仍由既有生成 BFF 按后端运行时定价执行预扣费，因此该字段表达用户确认时看到的价格快照，而不是前端可提交或覆盖的计费真相。
- `EditorAgentToolCall.summary` 只是 `args` 的重复字符串且没有稳定语义，当前契约删除该字段，不再作为展示或执行输入。
- 前端待确认卡只消费必填 `displayArgs`，不解析各 tool 私有的 snake_case / camelCase schema，也不把 `sha256:*` ID 当标题或图片地址。图片统一通过 `ResolvedAssetImage` 使用 `objectKey` 换签后显示，签名 URL 不进入消息文档。模块尚未上线，不保留缺少 `displayArgs` 时读取 raw `args` 的旧消息降级路径。
server-rs/crates/api-server/src/editor_agent/api.rs
+92 -480
View File
File diff suppressed because it is too large Load Diff
server-rs/crates/api-server/src/editor_agent/context.rs
+76 -14
View File
@@ -8,30 +8,92 @@ pub fn build_tool_context(document: &EditorAgentConversationMessagesDocument) ->
let mut images: HashMap<ImageId, ImageMetadata> = HashMap::new();
for msg in document.messages.iter().rev() {
// tool gen
for a in &msg.attachments {
let data_key = a.clone().into_data_key();
let image_id = ImageId::from_data_key(&data_key);
let metadata = ImageMetadata {
data_key,
image_src: a.image_src.clone(),
object_key: a.object_key.clone(),
thumbnail_src: a.thumbnail_src.clone(),
label: a.label.clone(),
width: a.width,
height: a.height,
};
images.entry(image_id).or_insert(metadata);
}
if let Some(tc) = &msg.tool_call {
for img in &tc.images {
let data_key = img.clone().into_data_key();
let image_id = ImageId::from_data_key(&data_key);
let metadata = ImageMetadata {
tag: String::new(),
data_key,
image_src: img.image_src.clone(),
object_key: img.object_key.clone(),
thumbnail_src: img.thumbnail_src.clone(),
label: None,
width: img.width,
height: img.height,
};
images.insert(image_id, metadata);
images.entry(image_id).or_insert(metadata);
}
}
// user pointed
for a in &msg.attachments {
let data_key = a.clone().into_data_key();
let image_id = ImageId::from_data_key(&data_key);
let metadata = ImageMetadata {
tag: String::new(),
data_key,
};
images.insert(image_id, metadata);
}
}
EditorToolContext { images }
}
#[cfg(test)]
mod tests {
use super::*;
use shared_contracts::editor_agent::{
EditorAgentAttachmentRef, EditorAgentAttachmentSource, EditorAgentMessage,
EditorAgentMessageRole,
};
fn message(id: usize, label: &str, image_src: &str) -> EditorAgentMessage {
EditorAgentMessage {
id,
client_message_id: Some(format!("message-{id}")),
role: EditorAgentMessageRole::User,
text: String::new(),
attachments: vec![EditorAgentAttachmentRef {
source: EditorAgentAttachmentSource::CanvasResource,
reference_id: "resource-1".to_string(),
object_key: Some("generated/reference.png".to_string()),
image_src: image_src.to_string(),
thumbnail_src: Some(format!("{image_src}?thumbnail=1")),
label: Some(label.to_string()),
width: Some(640),
height: Some(480),
}],
tool_call: None,
created_at: "2026-07-23T00:00:00Z".to_string(),
}
}
#[test]
fn tool_context_keeps_complete_metadata_from_the_latest_image_reference() {
let document = EditorAgentConversationMessagesDocument {
version: 2,
conversation_id: "conversation-1".to_string(),
messages: vec![
message(0, "旧名称", "/api/assets/read/old.png"),
message(1, "最新名称", "/api/assets/read/current.png"),
],
};
let context = build_tool_context(&document);
let image_id = ImageId::from_data_key("generated/reference.png");
let metadata = context
.image_metadata(&image_id)
.expect("latest image metadata should be present");
assert_eq!(metadata.data_key, "generated/reference.png");
assert_eq!(metadata.image_src, "/api/assets/read/current.png");
assert_eq!(metadata.label.as_deref(), Some("最新名称"));
assert_eq!(metadata.width, Some(640));
assert_eq!(metadata.height, Some(480));
}
}
server-rs/crates/api-server/src/editor_agent/display_args.rs
+26 -297
View File
@@ -1,271 +1,34 @@
use crate::editor_agent::pricing::EditorAgentPricedTool;
use crate::editor_agent::utils::{IntoDataKey, IntoImageId};
use crate::editor_generation_config::EditorGenerationPricingConfig;
use platform_editor_agent::agent::asset::ImageId;
use platform_editor_agent::agent::tools::context::EditorToolContext;
use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs};
use platform_editor_agent::agent::tools::generate_background_music::{
GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs,
};
use platform_editor_agent::agent::tools::generate_character::{
GenerateCharacterTool, GenerateCharacterToolArgs,
};
use platform_editor_agent::agent::tools::generate_icon_spritesheet::{
GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs,
};
use platform_editor_agent::agent::tools::generate_image::{
GenerateImageTool, GenerateImageToolArgs,
};
use platform_editor_agent::agent::tools::generate_sound_effect::{
GenerateSoundEffectTool, GenerateSoundEffectToolArgs,
};
use platform_editor_agent::agent::tools::generate_ui_design::{
GenerateUiDesignTool, GenerateUiDesignToolArgs,
};
use platform_editor_agent::agent::tools::generate_video::{
GenerateVideoTool, GenerateVideoToolArgs,
};
use platform_editor_agent::framework::error::PromptError;
use platform_editor_agent::framework::tool::Tool;
use serde_json::Value;
use shared_contracts::editor_agent::{
EditorAgentConversationMessagesDocument, EditorAgentToolCallDisplayArgs,
EditorAgentToolCallDisplayExtras, EditorAgentToolCallImageArg, EditorAgentToolCallImageRef,
EditorAgentToolCallDisplayArgs, EditorAgentToolCallImageArg, EditorAgentToolCallImageRef,
EditorAgentToolCallStringArg,
};
pub fn build_tool_call_display_args(
tool_name: &str,
args: &Value,
document: &EditorAgentConversationMessagesDocument,
tool_context: &EditorToolContext,
pricing: &EditorGenerationPricingConfig,
) -> Result<EditorAgentToolCallDisplayArgs, PromptError> {
let mut display_args = EditorAgentToolCallDisplayArgs::default();
let price_mud_points = match tool_name {
EditImageTool::NAME => {
let args: EditImageToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = EditImageTool {
context: tool_context.clone(),
}
.pricing(pricing, &args);
push_string_display_arg(&mut display_args, "prompt", "修改要求", args.prompt);
push_string_display_arg(&mut display_args, "model", "模型", args.model);
push_image_display_arg(
&mut display_args,
document,
"object_image_id",
"目标图片",
std::slice::from_ref(&args.object_image_id),
)?;
push_image_display_arg(
&mut display_args,
document,
"reference_image_ids",
"参考图片",
args.reference_image_ids.as_slice(),
)?;
price_mud_points
}
GenerateImageTool::NAME => {
let args: GenerateImageToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateImageTool {
context: tool_context.clone(),
}
.pricing(pricing, &args);
push_image_generation_display_args(
&mut display_args,
document,
args.prompt,
args.model,
args.aspect_ratio,
args.image_size,
args.reference_image_ids,
)?;
price_mud_points
}
GenerateCharacterTool::NAME => {
let args: GenerateCharacterToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateCharacterTool {
context: tool_context.clone(),
}
.pricing(pricing, &args);
push_image_generation_display_args(
&mut display_args,
document,
args.prompt,
args.model,
args.aspect_ratio,
args.image_size,
args.reference_image_ids,
)?;
price_mud_points
}
GenerateUiDesignTool::NAME => {
let args: GenerateUiDesignToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateUiDesignTool {
context: tool_context.clone(),
}
.pricing(pricing, &args);
push_image_generation_display_args(
&mut display_args,
document,
args.prompt,
args.model,
args.aspect_ratio,
args.image_size,
args.reference_image_ids,
)?;
price_mud_points
}
GenerateIconSpritesheetTool::NAME => {
let args: GenerateIconSpritesheetToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateIconSpritesheetTool {
context: tool_context.clone(),
}
.pricing(pricing, &args);
push_string_display_arg(
&mut display_args,
"icon_descriptions",
"图标描述",
args.icon_descriptions.join("\n"),
);
push_string_display_arg(&mut display_args, "model", "模型", args.model);
push_optional_string_display_arg(
&mut display_args,
"aspect_ratio",
"画面比例",
args.aspect_ratio,
);
push_optional_string_display_arg(
&mut display_args,
"image_size",
"图片尺寸",
args.image_size,
);
push_image_display_arg(
&mut display_args,
document,
"reference_image_id",
"规范参考图",
std::slice::from_ref(&args.reference_image_id),
)?;
push_image_display_arg(
&mut display_args,
document,
"reference_image_ids",
"额外参考图",
args.reference_image_ids.as_slice(),
)?;
price_mud_points
}
GenerateVideoTool::NAME => {
let args: GenerateVideoToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateVideoTool {
context: tool_context.clone(),
}
.pricing(pricing, &args);
push_string_display_arg(&mut display_args, "prompt", "视频描述", args.prompt);
push_optional_string_display_arg(
&mut display_args,
"aspect_ratio",
"画面比例",
args.aspect_ratio,
);
if let Some(duration_seconds) = args.duration_seconds {
push_string_display_arg(
&mut display_args,
"duration_seconds",
"时长（秒）",
duration_seconds.to_string(),
);
}
push_string_display_arg(&mut display_args, "model", "模型", args.model);
push_optional_string_display_arg(
&mut display_args,
"resolution",
"分辨率",
args.resolution,
);
push_optional_string_display_arg(&mut display_args, "sound", "声音", args.sound);
push_image_display_arg(
&mut display_args,
document,
"reference_image_ids",
"参考图片",
args.reference_image_ids.as_slice(),
)?;
price_mud_points
}
GenerateSoundEffectTool::NAME => {
let args: GenerateSoundEffectToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateSoundEffectTool.pricing(pricing, &args);
push_string_display_arg(&mut display_args, "prompt", "音效描述", args.prompt);
push_string_display_arg(&mut display_args, "model", "模型", args.model);
if let Some(duration) = args.duration {
push_string_display_arg(
&mut display_args,
"duration",
"时长（秒）",
duration.to_string(),
);
}
price_mud_points
}
GenerateBackgroundMusicTool::NAME => {
let args: GenerateBackgroundMusicToolArgs = parse_display_tool_args(tool_name, args)?;
let price_mud_points = GenerateBackgroundMusicTool.pricing(pricing, &args);
push_string_display_arg(&mut display_args, "prompt", "音乐描述", args.prompt);
push_string_display_arg(&mut display_args, "model", "模型", args.model);
price_mud_points
}
_ => {
return Err(PromptError::InternalError(format!(
"missing display argument mapping for tool {tool_name}"
)));
}
};
display_args.extras = EditorAgentToolCallDisplayExtras { price_mud_points };
Ok(display_args)
}
fn parse_display_tool_args<T>(tool_name: &str, args: &Value) -> Result<T, PromptError>
where
T: serde::de::DeserializeOwned,
{
serde_json::from_value(args.clone()).map_err(|error| {
PromptError::InternalError(format!(
"failed to build display arguments for {tool_name}: {error}"
))
})
}
fn push_image_generation_display_args(
pub(crate) fn push_image_generation_display_args(
display_args: &mut EditorAgentToolCallDisplayArgs,
document: &EditorAgentConversationMessagesDocument,
context: &EditorToolContext,
prompt: String,
model: String,
aspect_ratio: Option<String>,
image_size: Option<String>,
aspect_ratio: String,
image_size: String,
reference_image_ids: Vec<ImageId>,
) -> Result<(), PromptError> {
push_string_display_arg(display_args, "prompt", "提示词", prompt);
push_string_display_arg(display_args, "model", "模型", model);
push_optional_string_display_arg(display_args, "aspect_ratio", "画面比例", aspect_ratio);
push_optional_string_display_arg(display_args, "image_size", "图片尺寸", image_size);
push_string_display_arg(display_args, "aspect_ratio", "画面比例", aspect_ratio);
push_string_display_arg(display_args, "image_size", "图片尺寸", image_size);
push_image_display_arg(
display_args,
document,
context,
"reference_image_ids",
"参考图片",
reference_image_ids.as_slice(),
)
}
fn push_string_display_arg(
pub(crate) fn push_string_display_arg(
display_args: &mut EditorAgentToolCallDisplayArgs,
name: &str,
label: &str,
@@ -282,20 +45,9 @@ fn push_string_display_arg(
});
}
fn push_optional_string_display_arg(
pub(crate) fn push_image_display_arg(
display_args: &mut EditorAgentToolCallDisplayArgs,
name: &str,
label: &str,
value: Option<String>,
) {
if let Some(value) = value {
push_string_display_arg(display_args, name, label, value);
}
}
fn push_image_display_arg(
display_args: &mut EditorAgentToolCallDisplayArgs,
document: &EditorAgentConversationMessagesDocument,
context: &EditorToolContext,
name: &str,
label: &str,
image_ids: &[ImageId],
@@ -307,9 +59,9 @@ fn push_image_display_arg(
let refs = image_ids
.iter()
.map(|image_id| {
resolve_tool_call_image_ref(document, image_id).ok_or_else(|| {
resolve_tool_call_image_ref(context, image_id).ok_or_else(|| {
PromptError::InternalError(format!(
"display image reference {} is missing from conversation context",
"display image reference {} is missing from tool context",
image_id.id
))
})
@@ -324,42 +76,19 @@ fn push_image_display_arg(
Ok(())
}
pub fn resolve_tool_call_image_ref(
document: &EditorAgentConversationMessagesDocument,
fn resolve_tool_call_image_ref(
context: &EditorToolContext,
image_id: &ImageId,
) -> Option<EditorAgentToolCallImageRef> {
for message in document.messages.iter().rev() {
for attachment in message.attachments.iter().rev() {
if attachment.clone().into_image_id() == *image_id {
return Some(EditorAgentToolCallImageRef {
image_id: image_id.id.clone(),
image_src: attachment.image_src.clone(),
object_key: attachment.object_key.clone(),
thumbnail_src: attachment.thumbnail_src.clone(),
label: attachment.label.clone(),
width: attachment.width,
height: attachment.height,
});
}
}
if let Some(tool_call) = &message.tool_call {
for image in tool_call.images.iter().rev() {
let data_key = image.clone().into_data_key();
if ImageId::from_data_key(data_key) == *image_id {
return Some(EditorAgentToolCallImageRef {
image_id: image_id.id.clone(),
image_src: image.image_src.clone(),
object_key: image.object_key.clone(),
thumbnail_src: image.thumbnail_src.clone(),
label: None,
width: image.width,
height: image.height,
});
}
}
}
}
None
context
.image_metadata(image_id)
.map(|metadata| EditorAgentToolCallImageRef {
image_id: image_id.id.clone(),
image_src: metadata.image_src.clone(),
object_key: metadata.object_key.clone(),
thumbnail_src: metadata.thumbnail_src.clone(),
label: metadata.label.clone(),
width: metadata.width,
height: metadata.height,
})
}
server-rs/crates/api-server/src/editor_agent/mod.rs
+1 -2
View File
@@ -1,10 +1,9 @@
pub mod api;
mod context;
mod display_args;
pub mod pricing;
mod reconcile;
mod resp_to_asset;
mod tool_args;
mod tool;
mod utils;
pub use api::{
server-rs/crates/api-server/src/editor_agent/pricing.rs
-354
View File
@@ -1,354 +0,0 @@
use crate::editor_generation_config::EditorGenerationPricingConfig;
#[cfg(test)]
use crate::editor_generation_config::load_editor_generation_pricing_from_paths;
#[cfg(test)]
use platform_editor_agent::agent::asset::ImageId;
#[cfg(test)]
use platform_editor_agent::agent::tools::context::EditorToolContext;
use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs};
use platform_editor_agent::agent::tools::generate_background_music::{
GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs,
};
use platform_editor_agent::agent::tools::generate_character::{
GenerateCharacterTool, GenerateCharacterToolArgs,
};
use platform_editor_agent::agent::tools::generate_icon_spritesheet::{
GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs,
};
use platform_editor_agent::agent::tools::generate_image::{
GenerateImageTool, GenerateImageToolArgs,
};
use platform_editor_agent::agent::tools::generate_sound_effect::{
GenerateSoundEffectTool, GenerateSoundEffectToolArgs,
};
use platform_editor_agent::agent::tools::generate_ui_design::{
GenerateUiDesignTool, GenerateUiDesignToolArgs,
};
use platform_editor_agent::agent::tools::generate_video::{
GenerateVideoTool, GenerateVideoToolArgs,
};
use platform_editor_agent::framework::tool::Tool;
use platform_image::GPT_IMAGE_2_MODEL;
#[cfg(test)]
use std::collections::HashMap;
#[cfg(test)]
fn context() -> EditorToolContext {
EditorToolContext {
images: HashMap::new(),
}
}
#[cfg(test)]
fn image_args(image_size: Option<&str>) -> GenerateImageToolArgs {
GenerateImageToolArgs {
prompt: "生成图片".to_string(),
model: platform_image::NANOBANANA_2_MODEL.to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: Some("1:1".to_string()),
image_size: image_size.map(ToOwned::to_owned),
}
}
#[cfg(test)]
fn character_args(image_size: Option<&str>) -> GenerateCharacterToolArgs {
GenerateCharacterToolArgs {
prompt: "生成角色".to_string(),
model: platform_image::NANOBANANA_2_MODEL.to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: Some("1:1".to_string()),
image_size: image_size.map(ToOwned::to_owned),
}
}
#[test]
fn every_editor_agent_tool_exposes_argument_based_pricing() {
let pricing = load_editor_generation_pricing_from_paths(None)
.expect("default editor pricing should load");
let context = context();
assert_eq!(
EditImageTool {
context: context.clone(),
}
.pricing(
&pricing,
&EditImageToolArgs {
object_image_id: ImageId {
id: "image-1".to_string(),
},
reference_image_ids: Vec::new(),
prompt: "改成蓝色".to_string(),
model: GPT_IMAGE_2_MODEL.to_string(),
},
),
3
);
assert_eq!(
GenerateImageTool {
context: context.clone(),
}
.pricing(&pricing, &image_args(Some("2K"))),
24
);
assert_eq!(
GenerateCharacterTool {
context: context.clone(),
}
.pricing(&pricing, &character_args(None)),
12
);
assert_eq!(
GenerateUiDesignTool {
context: context.clone(),
}
.pricing(
&pricing,
&GenerateUiDesignToolArgs {
prompt: "生成游戏主界面".to_string(),
model: GPT_IMAGE_2_MODEL.to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: Some("1:1".to_string()),
image_size: Some("2K".to_string()),
}
),
5
);
assert_eq!(
GenerateIconSpritesheetTool {
context: context.clone(),
}
.pricing(
&pricing,
&GenerateIconSpritesheetToolArgs {
reference_image_id: ImageId {
id: "image-1".to_string(),
},
model: platform_image::NANOBANANA_2_MODEL.to_string(),
reference_image_ids: Vec::new(),
icon_descriptions: vec!["背包".to_string(), "地图".to_string()],
aspect_ratio: Some("1:1".to_string()),
image_size: Some("2K".to_string()),
},
),
24
);
assert_eq!(
GenerateVideoTool {
context: context.clone(),
}
.pricing(
&pricing,
&GenerateVideoToolArgs {
prompt: "镜头缓慢推进".to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: None,
duration_seconds: Some(6),
model: "seedance2.0".to_string(),
resolution: Some("720p".to_string()),
sound: None,
},
),
144
);
assert_eq!(
GenerateSoundEffectTool.pricing(
&pricing,
&GenerateSoundEffectToolArgs {
prompt: "按钮点击声".to_string(),
model: platform_audio::VIDU_AUDIO_MODEL.to_string(),
duration: None,
},
),
5
);
assert_eq!(
GenerateBackgroundMusicTool.pricing(
&pricing,
&GenerateBackgroundMusicToolArgs {
prompt: "轻松背景音乐".to_string(),
model: platform_audio::SUNO_DEFAULT_MODEL.to_string(),
},
),
12
);
}
#[test]
fn pricing_uses_the_supplied_runtime_snapshot() {
let mut pricing = load_editor_generation_pricing_from_paths(None)
.expect("default editor pricing should load");
pricing
.models
.get_mut("gpt-image-2")
.expect("gpt image pricing should exist")
.prices
.insert("2K".to_string(), 37);
pricing
.models
.get_mut("seedance2.0-fast")
.expect("video pricing should exist")
.prices
.insert("720p".to_string(), 7);
pricing
.models
.get_mut("audio1.0")
.expect("sound pricing should exist")
.price = Some(19);
let mut image2_args = image_args(Some("2K"));
image2_args.model = GPT_IMAGE_2_MODEL.to_string();
assert_eq!(
GenerateImageTool { context: context() }.pricing(&pricing, &image2_args),
37
);
assert_eq!(
GenerateVideoTool { context: context() }.pricing(
&pricing,
&GenerateVideoToolArgs {
prompt: "four seconds".to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: None,
duration_seconds: None,
model: GenerateVideoTool::DEFAULT_VIDEO_MODEL.to_string(),
resolution: None,
sound: None,
},
),
28
);
assert_eq!(
GenerateSoundEffectTool.pricing(
&pricing,
&GenerateSoundEffectToolArgs {
prompt: "sound".to_string(),
model: platform_audio::VIDU_AUDIO_MODEL.to_string(),
duration: None,
},
),
19
);
}
impl EditorAgentPricedTool for GenerateVideoTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateVideoToolArgs,
) -> u32 {
let model = args.model.as_str();
let resolution = args
.resolution
.as_deref()
.map(str::trim)
.filter(|value| !value.is_empty())
.unwrap_or(Self::DEFAULT_VIDEO_RESOLUTION);
let duration_seconds = args
.duration_seconds
.unwrap_or(Self::DEFAULT_VIDEO_DURATION_SECONDS);
pricing.video_model_mud_points(Some(model), resolution, duration_seconds)
}
}
impl EditorAgentPricedTool for GenerateUiDesignTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateUiDesignToolArgs,
) -> u32 {
editor_agent_image_mud_points(
pricing,
Some("ui-design"),
args.model.as_str(),
args.image_size.as_deref(),
)
}
}
impl EditorAgentPricedTool for GenerateSoundEffectTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateSoundEffectToolArgs,
) -> u32 {
pricing.sound_effect_model_mud_points(Some(args.model.as_str()))
}
}
impl EditorAgentPricedTool for GenerateImageTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateImageToolArgs,
) -> u32 {
editor_agent_image_mud_points(
pricing,
None,
args.model.as_str(),
args.image_size.as_deref(),
)
}
}
impl EditorAgentPricedTool for GenerateIconSpritesheetTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateIconSpritesheetToolArgs,
) -> u32 {
editor_agent_image_mud_points(
pricing,
Some("icon"),
args.model.as_str(),
args.image_size.as_deref(),
)
}
}
impl EditorAgentPricedTool for GenerateCharacterTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateCharacterToolArgs,
) -> u32 {
editor_agent_image_mud_points(
pricing,
Some("character"),
args.model.as_str(),
args.image_size.as_deref(),
)
}
}
/// api-server 侧的画布 Agent 工具计价扩展。
///
/// 通用 `Tool` 仍只负责参数校验；价格依赖 api-server 的运行时配置，不能下沉到
/// `module-editor-agent`。实际执行和扣费仍由既有生成 BFF 负责。
pub(crate) trait EditorAgentPricedTool: Tool {
fn pricing(&self, pricing: &EditorGenerationPricingConfig, args: &<Self as Tool>::Args) -> u32;
}
pub(crate) fn editor_agent_image_mud_points(
pricing: &EditorGenerationPricingConfig,
kind: Option<&str>,
model: &str,
image_size: Option<&str>,
) -> u32 {
pricing.image_generation_mud_points(kind, Some(model), image_size)
}
impl EditorAgentPricedTool for GenerateBackgroundMusicTool {
fn pricing(
&self,
pricing: &EditorGenerationPricingConfig,
args: &GenerateBackgroundMusicToolArgs,
) -> u32 {
pricing.background_music_model_mud_points(Some(args.model.as_str()))
}
}
impl EditorAgentPricedTool for EditImageTool {
fn pricing(&self, pricing: &EditorGenerationPricingConfig, _args: &EditImageToolArgs) -> u32 {
editor_agent_image_mud_points(pricing, Some("quick-edit"), GPT_IMAGE_2_MODEL, Some("1K"))
}
}
server-rs/crates/api-server/src/editor_agent/reconcile.rs
+19 -103
View File
@@ -1,34 +1,7 @@
use crate::editor_agent::resp_to_asset;
use crate::http_error::AppError;
use crate::state::AppState;
use platform_editor_agent::agent::tools::context::EditorToolContext;
use platform_editor_agent::agent::tools::edit_image::{
EditImageTool, EditImageToolArgs, EditorImageEditResult,
};
use platform_editor_agent::agent::tools::generate_background_music::{
GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs,
};
use platform_editor_agent::agent::tools::generate_character::{
GenerateCharacterTool, GenerateCharacterToolArgs,
};
use platform_editor_agent::agent::tools::generate_icon_spritesheet::{
EditorIconSpritesheetResult, GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs,
};
use platform_editor_agent::agent::tools::generate_image::{
EditorImageGenerationResult, GenerateImageTool, GenerateImageToolArgs,
};
use platform_editor_agent::agent::tools::generate_sound_effect::{
GenerateSoundEffectTool, GenerateSoundEffectToolArgs,
};
use platform_editor_agent::agent::tools::generate_ui_design::{
GenerateUiDesignTool, GenerateUiDesignToolArgs,
};
use platform_editor_agent::agent::tools::generate_video::{
GenerateVideoTool, GenerateVideoToolArgs,
};
use platform_editor_agent::framework::tool::Tool;
use serde_json::Value;
use shared_contracts::assets::{EditorAudioGenerateResponse, EditorVideoGenerateResponse};
use shared_contracts::editor_agent::{
EditorAgentConversationMessagesDocument, EditorAgentMessage, EditorAgentToolCallStatus,
};
@@ -209,88 +182,31 @@ fn reconcile_completed_editor_agent_tool_call(
.ok_or_else(|| ResultReconcileFailure::Fatal("消息缺少工具调用".to_string()))?;
let tool_name = tool_call.tool_name.clone();
match tool_name.as_str() {
GenerateImageTool::NAME => {
let args: GenerateImageToolArgs = parse_reconciled_value(&tool_call.args)?;
let result: EditorImageGenerationResult = parse_reconciled_value(&response)?;
message.text = GenerateImageTool {
context: EditorToolContext::default(),
}
.format_execute_message(&args, &result);
tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)];
}
GenerateCharacterTool::NAME => {
let args: GenerateCharacterToolArgs = parse_reconciled_value(&tool_call.args)?;
let result: EditorImageGenerationResult = parse_reconciled_value(&response)?;
message.text = GenerateCharacterTool {
context: EditorToolContext::default(),
}
.format_execute_message(&args, &result);
tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)];
}
GenerateUiDesignTool::NAME => {
let args: GenerateUiDesignToolArgs = parse_reconciled_value(&tool_call.args)?;
let result: EditorImageGenerationResult = parse_reconciled_value(&response)?;
message.text = GenerateUiDesignTool {
context: EditorToolContext::default(),
}
.format_execute_message(&args, &result);
tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)];
}
EditImageTool::NAME => {
let args: EditImageToolArgs = parse_reconciled_value(&tool_call.args)?;
let result: EditorImageEditResult = parse_reconciled_value(&response)?;
tool_call.images = vec![resp_to_asset::editor_agent_generated_edited_image(&result)];
message.text = EditImageTool {
context: EditorToolContext::default(),
}
.format_execute_message(args, result);
}
GenerateIconSpritesheetTool::NAME => {
let args: GenerateIconSpritesheetToolArgs = parse_reconciled_value(&tool_call.args)?;
let result: EditorIconSpritesheetResult = parse_reconciled_value(&response)?;
message.text = GenerateIconSpritesheetTool {
context: EditorToolContext::default(),
}
.format_execute_message(&args, &result);
tool_call.images = resp_to_asset::editor_agent_generated_spritesheet_images(&result);
}
GenerateVideoTool::NAME => {
let args: GenerateVideoToolArgs = parse_reconciled_value(&tool_call.args)?;
let result: EditorVideoGenerateResponse = parse_reconciled_value(&response)?;
message.text = GenerateVideoTool {
context: EditorToolContext::default(),
}
.format_execute_message(&args, &result);
tool_call.videos = vec![resp_to_asset::editor_agent_generated_video(&result)];
}
GenerateSoundEffectTool::NAME | GenerateBackgroundMusicTool::NAME => {
let result: EditorAudioGenerateResponse = parse_reconciled_value(&response)?;
message.text = if tool_name == GenerateSoundEffectTool::NAME {
let args: GenerateSoundEffectToolArgs = parse_reconciled_value(&tool_call.args)?;
GenerateSoundEffectTool.format_execute_message(&args, &result)
} else {
let args: GenerateBackgroundMusicToolArgs =
parse_reconciled_value(&tool_call.args)?;
GenerateBackgroundMusicTool.format_execute_message(&args, &result)
};
tool_call.audios = vec![resp_to_asset::editor_agent_generated_audio(&result)];
}
_ => {
return Err(ResultReconcileFailure::Fatal(format!(
"不支持回填工具 {tool_name}"
)));
}
let tool = crate::editor_agent::tool::editor_agent_tool(
tool_name.as_str(),
&EditorToolContext::default(),
)
.ok_or_else(|| ResultReconcileFailure::Fatal(format!("不支持回填工具 {tool_name}")))?;
message.text = tool
.format_execute_message(&tool_call.args, &response)
.map_err(|error| ResultReconcileFailure::Fatal(error.to_string()))?;
let assets = tool
.result_assets(&response)
.map_err(|error| ResultReconcileFailure::Fatal(error.to_string()))?;
if let Some(images) = assets.images {
tool_call.images = images;
}
if let Some(videos) = assets.videos {
tool_call.videos = videos;
}
if let Some(audios) = assets.audios {
tool_call.audios = audios;
}
tool_call.error = None;
tool_call.status = EditorAgentToolCallStatus::Completed;
Ok(())
}
fn parse_reconciled_value<T: serde::de::DeserializeOwned>(value: &Value) -> Result<T, String> {
serde_json::from_value(value.clone()).map_err(|error| error.to_string())
}
#[cfg(test)]
mod tests {
use super::*;
server-rs/crates/api-server/src/editor_agent/tool.rs
+1502
View File
File diff suppressed because it is too large Load Diff
server-rs/crates/api-server/src/editor_agent/tool_args.rs
-184
View File
@@ -1,184 +0,0 @@
use platform_editor_agent::agent::tools::context::EditorToolContext;
use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs};
use platform_editor_agent::agent::tools::generate_background_music::{
GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs,
};
use platform_editor_agent::agent::tools::generate_character::{
GenerateCharacterTool, GenerateCharacterToolArgs,
};
use platform_editor_agent::agent::tools::generate_icon_spritesheet::{
GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs,
};
use platform_editor_agent::agent::tools::generate_image::{
GenerateImageTool, GenerateImageToolArgs,
};
use platform_editor_agent::agent::tools::generate_sound_effect::{
GenerateSoundEffectTool, GenerateSoundEffectToolArgs,
};
use platform_editor_agent::agent::tools::generate_ui_design::{
GenerateUiDesignTool, GenerateUiDesignToolArgs,
};
use platform_editor_agent::agent::tools::generate_video::{
GenerateVideoTool, GenerateVideoToolArgs,
};
use platform_editor_agent::framework::error::PromptError;
use platform_editor_agent::framework::tool::Tool;
use serde::Serialize;
use serde::de::DeserializeOwned;
use serde_json::Value;
pub fn normalize_tool_args(
tool_name: &str,
args: &Value,
context: &EditorToolContext,
) -> Result<Value, PromptError> {
match tool_name {
GenerateImageTool::NAME => {
let args: GenerateImageToolArgs = parse_args(tool_name, args)?;
let tool = GenerateImageTool {
context: context.clone(),
};
tool.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
GenerateCharacterTool::NAME => {
let args: GenerateCharacterToolArgs = parse_args(tool_name, args)?;
let tool = GenerateCharacterTool {
context: context.clone(),
};
tool.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
GenerateUiDesignTool::NAME => {
let args: GenerateUiDesignToolArgs = parse_args(tool_name, args)?;
let tool = GenerateUiDesignTool {
context: context.clone(),
};
tool.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
EditImageTool::NAME => {
let args: EditImageToolArgs = parse_args(tool_name, args)?;
let tool = EditImageTool {
context: context.clone(),
};
if let Some(error) = tool.validate_args(&args) {
return Err(invalid_args(tool_name, error));
}
serialize_args(tool_name, args)
}
GenerateIconSpritesheetTool::NAME => {
let args: GenerateIconSpritesheetToolArgs = parse_args(tool_name, args)?;
let tool = GenerateIconSpritesheetTool {
context: context.clone(),
};
tool.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
GenerateVideoTool::NAME => {
let args: GenerateVideoToolArgs = parse_args(tool_name, args)?;
let tool = GenerateVideoTool {
context: context.clone(),
};
tool.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
GenerateSoundEffectTool::NAME => {
let args: GenerateSoundEffectToolArgs = parse_args(tool_name, args)?;
GenerateSoundEffectTool
.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
GenerateBackgroundMusicTool::NAME => {
let args: GenerateBackgroundMusicToolArgs = parse_args(tool_name, args)?;
GenerateBackgroundMusicTool
.validate_args(&args)
.map_err(|error| invalid_args(tool_name, error))?;
serialize_args(tool_name, args)
}
_ => Err(PromptError::ToolError(format!(
"unsupported editor agent tool: {tool_name}"
))),
}
}
fn parse_args<T: DeserializeOwned>(tool_name: &str, args: &Value) -> Result<T, PromptError> {
serde_json::from_value(args.clone()).map_err(|error| {
PromptError::ToolError(format!("invalid args for tool {tool_name}: {error}"))
})
}
fn serialize_args<T: Serialize>(tool_name: &str, args: T) -> Result<Value, PromptError> {
serde_json::to_value(args).map_err(|error| {
PromptError::InternalError(format!(
"failed to normalize args for tool {tool_name}: {error}"
))
})
}
fn invalid_args(tool_name: &str, error: impl std::fmt::Display) -> PromptError {
PromptError::ToolError(format!("invalid args for tool {tool_name}: {error}"))
}
#[cfg(test)]
mod tests {
use super::*;
use platform_audio::SUNO_DEFAULT_MODEL;
use platform_image::GPT_IMAGE_2_MODEL;
use serde_json::json;
#[test]
fn normalizes_defaults_and_drops_unknown_fields_before_persistence() {
let normalized = normalize_tool_args(
GenerateUiDesignTool::NAME,
&json!({
"prompt": "生成游戏主界面",
"unknown": "drop-me"
}),
&EditorToolContext::default(),
)
.expect("UI 参数应使用工具默认模型并完成规范化");
assert_eq!(normalized["model"], GPT_IMAGE_2_MODEL);
assert_eq!(normalized["aspect_ratio"], "1:1");
assert_eq!(normalized["image_size"], "1K");
assert!(normalized.get("unknown").is_none());
}
#[test]
fn drops_legacy_background_music_make_instrumental_field() {
let normalized = normalize_tool_args(
GenerateBackgroundMusicTool::NAME,
&json!({
"prompt": "温暖舒缓的钢琴背景音乐",
"model": SUNO_DEFAULT_MODEL,
"make_instrumental": false
}),
&EditorToolContext::default(),
)
.expect("旧背景音乐字段应被安全忽略");
assert!(normalized.get("make_instrumental").is_none());
}
#[test]
fn rejects_invalid_sound_duration_before_persistence() {
let error = normalize_tool_args(
GenerateSoundEffectTool::NAME,
&json!({
"prompt": "按钮点击声",
"duration": 11
}),
&EditorToolContext::default(),
)
.expect_err("非法音效时长必须在待确认消息持久化前失败");
assert!(error.to_string().contains("11"));
}
}
server-rs/crates/platform-editor-agent/src/agent/asset.rs
+6 -1
View File
@@ -38,6 +38,11 @@ impl Display for ImageId {
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ImageMetadata {
pub tag: String,
pub data_key: String,
pub image_src: String,
pub object_key: Option<String>,
pub thumbnail_src: Option<String>,
pub label: Option<String>,
pub width: Option<u32>,
pub height: Option<u32>,
}
server-rs/crates/platform-editor-agent/src/agent/tools/context.rs
+5 -2
View File
@@ -10,10 +10,13 @@ impl EditorToolContext {
}
pub fn image_data_key(&self, image_id: &ImageId) -> Option<&str> {
self.images
.get(image_id)
self.image_metadata(image_id)
.map(|metadata| metadata.data_key.as_str())
}
pub fn image_metadata(&self, image_id: &ImageId) -> Option<&ImageMetadata> {
self.images.get(image_id)
}
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs
+4 -4
View File
@@ -26,9 +26,9 @@ pub struct GenerateCharacterToolArgs {
#[serde(default)]
pub reference_image_ids: Vec<ImageId>,
#[serde(default = "default_image_aspect_ratio")]
pub aspect_ratio: Option<String>,
pub aspect_ratio: String,
#[serde(default = "default_image_size")]
pub image_size: Option<String>,
pub image_size: String,
}
impl Tool for GenerateCharacterTool {
@@ -90,8 +90,8 @@ impl GenerateCharacterTool {
) -> Result<(), GenerateImageError> {
validate_image_generation_options(
args.model.as_str(),
args.aspect_ratio.as_deref(),
args.image_size.as_deref(),
args.aspect_ratio.as_str(),
args.image_size.as_str(),
)?;
if args.prompt.trim().is_empty() {
return Err(GenerateImageError::PromptNotProvided);
server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs
+12 -7
View File
@@ -78,9 +78,9 @@ pub struct GenerateIconSpritesheetToolArgs {
pub reference_image_ids: Vec<ImageId>,
pub icon_descriptions: Vec<String>,
#[serde(default = "default_image_aspect_ratio")]
pub aspect_ratio: Option<String>,
pub aspect_ratio: String,
#[serde(default = "default_image_size")]
pub image_size: Option<String>,
pub image_size: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -179,8 +179,8 @@ impl GenerateIconSpritesheetTool {
) -> Result<(), GenerateIconSpritesheetError> {
validate_image_generation_options(
args.model.as_str(),
args.aspect_ratio.as_deref(),
args.image_size.as_deref(),
args.aspect_ratio.as_str(),
args.image_size.as_str(),
)?;
if args.reference_image_id.id.trim().is_empty() {
return Err(GenerateIconSpritesheetError::ReferenceNotProvided);
@@ -252,8 +252,13 @@ mod tests {
images: HashMap::from([(
reference_image_id.clone(),
ImageMetadata {
tag: "image".to_string(),
data_key: "asset://reference-image".to_string(),
image_src: "asset://reference-image".to_string(),
object_key: None,
thumbnail_src: None,
label: None,
width: None,
height: None,
},
)]),
},
@@ -263,8 +268,8 @@ mod tests {
model: NANOBANANA_2_MODEL.to_string(),
reference_image_ids: Vec::new(),
icon_descriptions,
aspect_ratio: Some("1:1".to_string()),
image_size: Some("1K".to_string()),
aspect_ratio: "1:1".to_string(),
image_size: "1K".to_string(),
};
(tool, args)
}
server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs
+4 -4
View File
@@ -74,9 +74,9 @@ pub struct GenerateImageToolArgs {
#[serde(default)]
pub reference_image_ids: Vec<ImageId>,
#[serde(default = "default_image_aspect_ratio")]
pub aspect_ratio: Option<String>,
pub aspect_ratio: String,
#[serde(default = "default_image_size")]
pub image_size: Option<String>,
pub image_size: String,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -170,8 +170,8 @@ impl GenerateImageTool {
pub fn validate_args(&self, args: &GenerateImageToolArgs) -> Result<(), GenerateImageError> {
validate_image_generation_options(
args.model.as_str(),
args.aspect_ratio.as_deref(),
args.image_size.as_deref(),
args.aspect_ratio.as_str(),
args.image_size.as_str(),
)?;
if args.prompt.trim().is_empty() {
return Err(GenerateImageError::PromptNotProvided);
server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs
+8 -11
View File
@@ -45,15 +45,15 @@ pub struct GenerateSoundEffectToolArgs {
#[serde(default = "default_sound_effect_model")]
pub model: String,
#[serde(default = "default_sound_effect_duration")]
pub duration: Option<u8>,
pub duration: u8,
}
fn default_sound_effect_model() -> String {
VIDU_AUDIO_MODEL.to_string()
}
fn default_sound_effect_duration() -> Option<u8> {
Some(GenerateSoundEffectTool::DEFAULT_DURATION)
fn default_sound_effect_duration() -> u8 {
GenerateSoundEffectTool::DEFAULT_DURATION
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -120,10 +120,8 @@ impl GenerateSoundEffectTool {
if args.model != VIDU_AUDIO_MODEL {
return Err(GenerateSoundEffectError::InvalidModel(args.model.clone()));
}
if let Some(duration) = args.duration
&& !Self::SUPPORTED_DURATIONS.contains(&duration)
{
return Err(GenerateSoundEffectError::InvalidDuration(duration));
if !Self::SUPPORTED_DURATIONS.contains(&args.duration) {
return Err(GenerateSoundEffectError::InvalidDuration(args.duration));
}
Ok(())
}
@@ -146,7 +144,7 @@ impl GenerateSoundEffectTool {
mod tests {
use super::*;
fn args(duration: Option<u8>) -> GenerateSoundEffectToolArgs {
fn args(duration: u8) -> GenerateSoundEffectToolArgs {
GenerateSoundEffectToolArgs {
prompt: "按钮点击声".to_string(),
model: GenerateSoundEffectTool::DEFAULT_MODEL.to_string(),
@@ -157,11 +155,10 @@ mod tests {
#[test]
fn validates_schema_durations_and_default() {
assert_eq!(GenerateSoundEffectTool::DEFAULT_DURATION, 5);
assert!(GenerateSoundEffectTool.validate_args(&args(None)).is_ok());
for duration in GenerateSoundEffectTool::SUPPORTED_DURATIONS {
assert!(
GenerateSoundEffectTool
.validate_args(&args(Some(*duration)))
.validate_args(&args(*duration))
.is_ok()
);
}
@@ -171,7 +168,7 @@ mod tests {
async fn planning_rejects_durations_outside_schema() {
for duration in [1, 11, u8::MAX] {
assert!(matches!(
GenerateSoundEffectTool.call(args(Some(duration))).await,
GenerateSoundEffectTool.call(args(duration)).await,
Err(GenerateSoundEffectError::InvalidDuration(value)) if value == duration
));
}
server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs
+6 -6
View File
@@ -26,9 +26,9 @@ pub struct GenerateUiDesignToolArgs {
#[serde(default)]
pub reference_image_ids: Vec<ImageId>,
#[serde(default = "default_image_aspect_ratio")]
pub aspect_ratio: Option<String>,
pub aspect_ratio: String,
#[serde(default = "default_image_size")]
pub image_size: Option<String>,
pub image_size: String,
}
fn default_ui_design_model() -> String {
@@ -100,8 +100,8 @@ impl GenerateUiDesignTool {
}
validate_image_generation_options(
args.model.as_str(),
args.aspect_ratio.as_deref(),
args.image_size.as_deref(),
args.aspect_ratio.as_str(),
args.image_size.as_str(),
)?;
if args.prompt.trim().is_empty() {
return Err(GenerateImageError::PromptNotProvided);
@@ -151,8 +151,8 @@ mod tests {
prompt: "生成游戏主界面".to_string(),
model: model.to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: Some("16:9".to_string()),
image_size: Some("1K".to_string()),
aspect_ratio: "16:9".to_string(),
image_size: "1K".to_string(),
}
}
server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs
+25 -33
View File
@@ -67,35 +67,35 @@ pub struct GenerateVideoToolArgs {
#[serde(default)]
pub reference_image_ids: Vec<ImageId>,
#[serde(default = "default_video_aspect_ratio")]
pub aspect_ratio: Option<String>,
pub aspect_ratio: String,
#[serde(default = "default_video_duration_seconds")]
pub duration_seconds: Option<u32>,
pub duration_seconds: u32,
#[serde(default = "default_video_model")]
pub model: String,
#[serde(default = "default_video_resolution")]
pub resolution: Option<String>,
pub resolution: String,
#[serde(default = "default_video_sound")]
pub sound: Option<String>,
pub sound: String,
}
fn default_video_model() -> String {
GenerateVideoTool::DEFAULT_VIDEO_MODEL.to_string()
}
fn default_video_aspect_ratio() -> Option<String> {
Some(GenerateVideoTool::DEFAULT_VIDEO_ASPECT_RATIO.to_string())
fn default_video_aspect_ratio() -> String {
GenerateVideoTool::DEFAULT_VIDEO_ASPECT_RATIO.to_string()
}
fn default_video_duration_seconds() -> Option<u32> {
Some(GenerateVideoTool::DEFAULT_VIDEO_DURATION_SECONDS)
fn default_video_duration_seconds() -> u32 {
GenerateVideoTool::DEFAULT_VIDEO_DURATION_SECONDS
}
fn default_video_resolution() -> Option<String> {
Some(GenerateVideoTool::DEFAULT_VIDEO_RESOLUTION.to_string())
fn default_video_resolution() -> String {
GenerateVideoTool::DEFAULT_VIDEO_RESOLUTION.to_string()
}
fn default_video_sound() -> Option<String> {
Some(GenerateVideoTool::DEFAULT_VIDEO_SOUND.to_string())
fn default_video_sound() -> String {
GenerateVideoTool::DEFAULT_VIDEO_SOUND.to_string()
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct GenerateVideoToolOutput {
@@ -202,31 +202,23 @@ impl GenerateVideoTool {
if !Self::SUPPORTED_VIDEO_MODELS.contains(&args.model.as_str()) {
return Err(GenerateVideoError::InvalidModel(args.model.clone()));
}
let aspect_ratio = args
.aspect_ratio
.as_deref()
.unwrap_or(Self::DEFAULT_VIDEO_ASPECT_RATIO);
let aspect_ratio = args.aspect_ratio.as_str();
if !Self::SUPPORTED_ASPECT_RATIOS.contains(&aspect_ratio) {
return Err(GenerateVideoError::InvalidAspectRatio(
aspect_ratio.to_string(),
));
}
let duration_seconds = args
.duration_seconds
.unwrap_or(Self::DEFAULT_VIDEO_DURATION_SECONDS);
let duration_seconds = args.duration_seconds;
if !Self::SUPPORTED_DURATION_SECONDS.contains(&duration_seconds) {
return Err(GenerateVideoError::InvalidDurationSeconds(duration_seconds));
}
let resolution = args
.resolution
.as_deref()
.unwrap_or(Self::DEFAULT_VIDEO_RESOLUTION);
let resolution = args.resolution.as_str();
if !Self::SUPPORTED_RESOLUTIONS.contains(&resolution) {
return Err(GenerateVideoError::InvalidResolution(
resolution.to_string(),
));
}
let sound = args.sound.as_deref().unwrap_or(Self::DEFAULT_VIDEO_SOUND);
let sound = args.sound.as_str();
if !Self::SUPPORTED_SOUND_OPTIONS.contains(&sound) {
return Err(GenerateVideoError::InvalidSound(sound.to_string()));
}
@@ -281,11 +273,11 @@ mod tests {
GenerateVideoToolArgs {
prompt: "镜头向前推进".to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: Some("16:9".to_string()),
duration_seconds: Some(4),
aspect_ratio: "16:9".to_string(),
duration_seconds: 4,
model: GenerateVideoTool::SEEDANCE_2_FAST_MODEL.to_string(),
resolution: Some("720p".to_string()),
sound: Some("on".to_string()),
resolution: "720p".to_string(),
sound: "on".to_string(),
}
}
@@ -298,28 +290,28 @@ mod tests {
#[test]
fn validates_each_video_option() {
let mut invalid_aspect_ratio = args();
invalid_aspect_ratio.aspect_ratio = Some("2:1".to_string());
invalid_aspect_ratio.aspect_ratio = "2:1".to_string();
assert!(matches!(
tool().validate_args(&invalid_aspect_ratio),
Err(GenerateVideoError::InvalidAspectRatio(value)) if value == "2:1"
));
let mut invalid_duration = args();
invalid_duration.duration_seconds = Some(16);
invalid_duration.duration_seconds = 16;
assert!(matches!(
tool().validate_args(&invalid_duration),
Err(GenerateVideoError::InvalidDurationSeconds(16))
));
let mut invalid_resolution = args();
invalid_resolution.resolution = Some("4K".to_string());
invalid_resolution.resolution = "4K".to_string();
assert!(matches!(
tool().validate_args(&invalid_resolution),
Err(GenerateVideoError::InvalidResolution(value)) if value == "4K"
));
let mut invalid_sound = args();
invalid_sound.sound = Some("auto".to_string());
invalid_sound.sound = "auto".to_string();
assert!(matches!(
tool().validate_args(&invalid_sound),
Err(GenerateVideoError::InvalidSound(value)) if value == "auto"
@@ -329,7 +321,7 @@ mod tests {
#[test]
fn rejects_invalid_model_combinations_and_reference_count() {
let mut invalid = args();
invalid.resolution = Some("1080p".to_string());
invalid.resolution = "1080p".to_string();
assert!(matches!(
tool().validate_args(&invalid),
Err(GenerateVideoError::UnsupportedModelResolution { model, resolution })
server-rs/crates/platform-editor-agent/src/agent/tools/image_generation_options.rs
+12 -18
View File
@@ -43,28 +43,26 @@ pub fn default_image_model() -> String {
NANOBANANA_2_MODEL.to_string()
}
pub fn default_image_aspect_ratio() -> Option<String> {
Some(DEFAULT_IMAGE_ASPECT_RATIO.to_string())
pub fn default_image_aspect_ratio() -> String {
DEFAULT_IMAGE_ASPECT_RATIO.to_string()
}
pub fn default_image_size() -> Option<String> {
Some(DEFAULT_IMAGE_SIZE.to_string())
pub fn default_image_size() -> String {
DEFAULT_IMAGE_SIZE.to_string()
}
pub fn validate_image_generation_options(
model: &str,
aspect_ratio: Option<&str>,
image_size: Option<&str>,
aspect_ratio: &str,
image_size: &str,
) -> Result<(), ImageGenerationOptionsError> {
let supported_sizes = supported_image_sizes(model)
.ok_or_else(|| ImageGenerationOptionsError::InvalidModel(model.to_string()))?;
let aspect_ratio = aspect_ratio.unwrap_or(DEFAULT_IMAGE_ASPECT_RATIO);
if !SUPPORTED_IMAGE_ASPECT_RATIOS.contains(&aspect_ratio) {
return Err(ImageGenerationOptionsError::InvalidAspectRatio(
aspect_ratio.to_string(),
));
}
let image_size = image_size.unwrap_or(DEFAULT_IMAGE_SIZE);
if !supported_sizes.contains(&image_size) {
return Err(ImageGenerationOptionsError::InvalidImageSize {
model: model.to_string(),
@@ -120,30 +118,26 @@ mod tests {
assert!(
validate_image_generation_options(
NANOBANANA_2_MODEL,
Some(aspect_ratio),
Some(image_size),
aspect_ratio,
image_size,
)
.is_ok()
);
}
for image_size in GPT_IMAGE_2_IMAGE_SIZES {
assert!(
validate_image_generation_options(
GPT_IMAGE_2_MODEL,
Some(aspect_ratio),
Some(image_size),
)
.is_ok()
validate_image_generation_options(GPT_IMAGE_2_MODEL, aspect_ratio, image_size,)
.is_ok()
);
}
}
assert!(matches!(
validate_image_generation_options(GPT_IMAGE_2_MODEL, Some("1:1"), Some("0.5K")),
validate_image_generation_options(GPT_IMAGE_2_MODEL, "1:1", "0.5K"),
Err(ImageGenerationOptionsError::InvalidImageSize { .. })
));
assert!(matches!(
validate_image_generation_options(NANOBANANA_2_MODEL, Some("21:9"), Some("1K")),
validate_image_generation_options(NANOBANANA_2_MODEL, "21:9", "1K"),
Err(ImageGenerationOptionsError::InvalidAspectRatio(_))
));
}
server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs
+44 -10
View File
@@ -66,19 +66,53 @@ mod tests {
.expect("music args should deserialize");
assert_eq!(image.model, NANOBANANA_2_MODEL);
assert_eq!(image.aspect_ratio.as_deref(), Some("1:1"));
assert_eq!(image.image_size.as_deref(), Some("1K"));
assert_eq!(image.aspect_ratio, "1:1");
assert_eq!(image.image_size, "1K");
assert_eq!(edit.model, GPT_IMAGE_2_MODEL);
assert_eq!(character.model, NANOBANANA_2_MODEL);
assert_eq!(character.aspect_ratio, "1:1");
assert_eq!(character.image_size, "1K");
assert_eq!(ui_design.model, GPT_IMAGE_2_MODEL);
assert_eq!(ui_design.aspect_ratio, "1:1");
assert_eq!(ui_design.image_size, "1K");
assert_eq!(icon.model, NANOBANANA_2_MODEL);
assert_eq!(icon.aspect_ratio, "1:1");
assert_eq!(icon.image_size, "1K");
assert_eq!(video.model, GenerateVideoTool::DEFAULT_VIDEO_MODEL);
assert_eq!(video.sound.as_deref(), Some("on"));
assert_eq!(video.aspect_ratio, "16:9");
assert_eq!(video.duration_seconds, 4);
assert_eq!(video.resolution, "720p");
assert_eq!(video.sound, "on");
assert_eq!(sound.model, VIDU_AUDIO_MODEL);
assert_eq!(sound.duration, Some(5));
assert_eq!(sound.duration, 5);
assert_eq!(music.model, SUNO_DEFAULT_MODEL);
}
#[test]
fn default_backed_tool_args_reject_null() {
assert!(
serde_json::from_value::<GenerateImageToolArgs>(json!({
"prompt": "生成森林场景",
"aspect_ratio": null
}))
.is_err()
);
assert!(
serde_json::from_value::<GenerateVideoToolArgs>(json!({
"prompt": "镜头向前推进",
"duration_seconds": null
}))
.is_err()
);
assert!(
serde_json::from_value::<GenerateSoundEffectToolArgs>(json!({
"prompt": "按钮点击声",
"duration": null
}))
.is_err()
);
}
#[test]
fn generate_image_accepts_nanobanana_and_image2_only() {
let tool = GenerateImageTool {
@@ -88,8 +122,8 @@ mod tests {
prompt: "生成森林场景".to_string(),
model: model.to_string(),
reference_image_ids: Vec::new(),
aspect_ratio: None,
image_size: None,
aspect_ratio: "1:1".to_string(),
image_size: "1K".to_string(),
};
assert!(tool.validate_args(&args(NANOBANANA_2_MODEL)).is_ok());
@@ -113,8 +147,8 @@ mod tests {
prompt: "生成角色".to_string(),
model: NANOBANANA_2_MODEL.to_string(),
reference_image_ids: vec![missing_image.clone()],
aspect_ratio: Some("2:3".to_string()),
image_size: Some("1K".to_string()),
aspect_ratio: "2:3".to_string(),
image_size: "1K".to_string(),
};
assert!(matches!(
@@ -128,8 +162,8 @@ mod tests {
prompt: "生成游戏主界面".to_string(),
model: GPT_IMAGE_2_MODEL.to_string(),
reference_image_ids: vec![missing_image.clone()],
aspect_ratio: Some("16:9".to_string()),
image_size: Some("1K".to_string()),
aspect_ratio: "16:9".to_string(),
image_size: "1K".to_string(),
};
assert!(matches!(
GenerateUiDesignTool {
server-rs/crates/platform-editor-agent/src/framework/run.rs
+35 -12
View File
@@ -14,7 +14,7 @@ pub type TextOutput = String;
#[derive(Debug, Clone)]
pub struct ToolCallOutput {
pub tool_call: ToolCall,
pub message: String,
pub output: Value,
}
#[derive(Debug, Clone)]
pub enum PromptOutput {
@@ -35,6 +35,20 @@ pub enum ToolCallFlow {
Stop,
}
pub fn format_tool_call_message(
tool_call_id: impl std::fmt::Display,
args: &Value,
output: &Value,
) -> Result<String, PromptError> {
let arg_json = serde_json::to_string(args)
.map_err(|error| PromptError::InternalError(error.to_string()))?;
let output_json = serde_json::to_string(output)
.map_err(|error| PromptError::InternalError(error.to_string()))?;
Ok(format!(
"[tool_call:{tool_call_id}] args: {arg_json} output: {output_json}"
))
}
pub struct PromptRequest<'a, M: LlmApiAdaptor<Message> + 'a, Message: 'a> {
agent: &'a mut Agent<M, Message>,
message: Message,
@@ -190,7 +204,7 @@ where
match hook.after_tool_call(&tc.name, &mut json_output) {
ToolCallFlow::Stop => {
return Err(PromptError::ToolError(
"tool call output caused this _turn stopped by hook".to_string(),
"tool call output caused this turn stopped by hook".to_string(),
));
}
ToolCallFlow::Skip => {
@@ -201,20 +215,14 @@ where
ToolCallFlow::Continue => {}
}
}
let arg_json = serde_json::to_string(&tc.args)
.map_err(|e| PromptError::InternalError(e.to_string()))?;
let output_json = serde_json::to_string(&json_output)
.map_err(|e| PromptError::InternalError(e.to_string()))?;
let overall_message = format!(
"[tool_call:{tc_id}] args: {arg_json} output: {output_json}"
);
let overall_message =
format_tool_call_message(tc_id, &tc.args, &json_output)?;
let msg =
agent.model.tool_result_message(&tc.name, &overall_message);
memory.append_message(msg);
prompt_result.push(Tool(ToolCallOutput {
tool_call: tc.clone(),
message: overall_message.clone(),
output: json_output,
}))
}
ToolOutcome::InternalError(failure) if failure.fatal => {
@@ -381,7 +389,22 @@ mod tests {
assert_eq!(completion_count.load(Ordering::SeqCst), 1);
assert_eq!(outputs.len(), 2);
assert!(matches!(outputs[0], PromptOutput::Text(_)));
assert!(matches!(outputs[1], PromptOutput::Tool(_)));
let PromptOutput::Tool(tool_output) = &outputs[1] else {
panic!("second output should be a tool call");
};
assert_eq!(
tool_output.output,
json!({ "message": "pending user confirmation" })
);
assert_eq!(
format_tool_call_message(
"test-tool",
&tool_output.tool_call.args,
&tool_output.output,
)
.expect("tool call message should format"),
"[tool_call:test-tool] args: {\"prompt\":\"生成一张图\"} output: {\"message\":\"pending user confirmation\"}"
);
}
#[tokio::test]