diff --git a/docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md b/docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md index 194a23ea0..c1c881dbe 100644 --- a/docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md +++ b/docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md @@ -4,7 +4,7 @@ ## 背景 -画布Agent对话需要保存历史记录并支持新开会话。消息正文是流式追加、体量随对话增长的 JSON 数据；SpacetimeDB 是后端唯一结构化真相存储，但表行不适合承载不断增长的长文本；画布工程快照已有独立保存链路，且对话流式更新频率远高于布局保存。 +画布Agent对话需要保存历史记录并支持新开会话。消息正文会随对话和工具结果持续增长；SpacetimeDB 是后端唯一结构化真相存储，但表行不适合承载不断增长的长文本；画布工程快照已有独立保存链路，对话消息也不应与布局保存互相竞争。 ## 决策 @@ -14,7 +14,7 @@ - 浏览器禁止直接上传、覆盖或签名写入 `editor-agent/` 对象；前端只通过 `api-server` 的会话接口创建会话、发送消息、读取历史，OSS 读写由服务端完成。 - 消息文档序列化后的读写上限为 2 MiB；超过上限时后端拒绝继续读写该会话消息文档，并返回 payload too large 语义错误。 - 同一会话内的消息追加采用 `conversationId` 级串行锁，避免同一会话的“读-改-写”整对象过程互相覆盖。 -- Agent 工具调用失败时，仍保留 assistant 消息和 `failed` generation record，包括 tool call、模型、错误信息等排障字段；失败记录是会话历史的一部分，不因 SSE 已返回错误而丢弃。 +- Agent 规划失败使用 `role=system`、正文以 `ERROR ` 开头的消息持久化；首次响应和同 `clientMessageId` 重放都通过 `deltaMessages` 返回该消息，`errorMessage` 不重复携带。前端隐藏前缀并显示红色错误气泡，后端构建后续 LLM memory 时仍保留该消息，让 Agent 获取上一轮失败上下文。工具调用失败继续保留 `status=failed` 工具记录、模型和错误信息；失败记录都是会话历史的一部分。 - 不把消息明细写入 SpacetimeDB 表，不把对话混入画布工程快照，不在 api-server 内存中保存会话真相。 ## 备选方案与取舍 @@ -31,5 +31,5 @@ - 会话表仍只保存元数据和 `messagesObjectKey`；API 可以在读取时把 OSS 消息文档拼装为会话详情返回，但完整消息正文的持久化真相仍是 OSS JSON 文档。 - 消息文档最大 2 MiB；该限制用于阻止单个会话无限增长。后续如果需要更长历史，应引入归档、分页对象或摘要压缩，不应把正文回填进 SpacetimeDB 表。 - 会话软删只打表标记，OSS 对象保留，便于恢复与审计。 -- 工具生成失败也会写入消息文档：assistant 消息可标记为失败 / 错误类型，`generations` 中保留 `failed` record、模型和错误信息，便于用户回看失败原因和后续排障。 +- 规划或工具生成失败也必须写入消息文档：规划失败保存 `ERROR ` system 消息，工具失败保存失败状态、模型和错误信息，便于用户回看失败原因和后续排障。 - 若未来出现跨会话消息检索需求，需另建投影或索引，不回退为消息入表。 diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 74aa4db36..7af62ba89 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -15,6 +15,21 @@ ``` --- +## 2026-07-17 画布 Agent 普通消息不提供客户端停止 + +- 背景：普通消息进入 LLM 前，后端已经把用户消息写入 OSS；前端中断 fetch 只能停止本地等待，不能保证后端停止规划，且会保留无法与后端消息对齐的 optimistic message。 +- 决策：移除画布 Agent 普通消息的“停止”按钮和 `stopCurrentTurn`，发送期间保持按钮禁用并等待后端响应。待确认工具调用的“取消”仍保留，不受本决策影响。 +- 影响范围：画布 Agent 对话 hook、发送区交互、前端测试和专题文档。 +- 验证方式：运行画布 Agent hook / 面板定向测试、`npm run typecheck`、`npm run check:encoding` 和 `git diff --check`。 +- 关联文档：`docs/【编辑器】画布Agent对话面板-2026-07-03.md`。 + +## 2026-07-10 画布 Agent 工具确认分离执行参数与展示投影 + +- 背景：画布 Agent 已在实际生成前进入 `pending_confirmation`，但 `EditorAgentToolCall.args` 只保存工具私有 JSON，其中图片参数是保护真实 data key 的 SHA-256 opaque ID。前端直接解析 raw args 只能显示内部哈希或图片数量，无法向用户准确展示即将使用的目标图、参考图和完整参数；若直接把图片 URL 或对象塞回 raw args，又会破坏确认执行反序列化和 LLM 不可见真实 data key 的安全边界。 +- 决策：`EditorAgentToolCall.args` 继续作为确认执行唯一真相，不允许前端改写或回传替代参数；新增必填 `displayArgs` 只读展示投影，内含 `stringArgs`、`imageArgs` 和 `extras.priceMudPoints`。`stringArgs` 承载提示词与规格等用户可见字段，`imageArgs.refs` 承载 `imageId` 及后端解析出的 `objectKey`、`imageSrc`、可选缩略图、标签和尺寸；`extras.priceMudPoints` 由 api-server 在创建待确认消息时使用后端运行时模型定价快照计算，前端只显示“预计消耗 N泥点”，不自行计算或回传价格。api-server 必须按已注册 tool 白名单，从已校验 args 与 OSS 会话文档的附件 / 历史生成结果构建该投影；前端只渲染投影，以 `ResolvedAssetImage` 换签显示图片，不解析 tool 私有 schema、不展示 SHA-256 ID。展示价格不参与确认执行或实际扣费，确认后仍由既有生成 BFF 按后端运行时定价预扣费。删除只重复 `args` 且没有稳定语义的 `EditorAgentToolCall.summary`。模块尚未上线，不保留缺少 `displayArgs` 时读取 raw `args` 的旧消息降级路径。 +- 影响范围：`shared-contracts` / `packages/shared` 的 `editorAgent` DTO、`api-server/src/editor_agent/api.rs` 的待确认消息构建、画布 Agent 待确认卡、OSS 会话消息文档与相关测试。 +- 验证方式：`cargo test -p shared-contracts --manifest-path server-rs/Cargo.toml editor_agent`、`cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_agent`、`npm run test -- src/components/image-editor/EditorAgentConversation/EditorAgentConversationPanelView.test.tsx src/components/image-editor/EditorAgentConversation/useEditorAgentConversation.test.tsx src/services/image-editor/editorAgentClient.test.ts`、`npm run typecheck`、`npm run check:encoding`、`git diff --check`。 +- 关联文档：`docs/【编辑器】画布Agent对话面板-2026-07-03.md`、`docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md`。 ## 2026-07-15 角色动作 BgFilter 请求超时按帧数扩展 @@ -275,10 +290,10 @@ ## 2026-07-03 画布Agent会话元数据入 SpacetimeDB、消息正文存 OSS -- 背景：图片画布工程需要对话式编辑历史，但消息正文随流式输出增长，不适合放入表行或画布布局快照；同时画布 Agent 只属于编辑器画布域，不能复用拼图 `creative-agent` 内存会话。 -- 决策：新增 `module-editor-agent` 承载纯领域规则，`editor_agent_conversation` 只保存会话元数据，完整消息以 `editor-agent/{conversationId}.json` 会话粒度存 OSS；`api-server` 负责编排 LLM、SSE、OSS 读写和既有生成工具调用。 +- 背景：图片画布工程需要对话式编辑历史，但消息正文随对话和工具结果增长，不适合放入表行或画布布局快照；同时画布 Agent 只属于编辑器画布域，不能复用拼图 `creative-agent` 内存会话。 +- 决策：`module-editor-agent` 只承载可供 SpacetimeDB WASM 使用的纯领域规则；Agent runner、工具实现和资产 DTO 迁入原生 `platform-editor-agent`，仅由 `api-server` 依赖。`editor_agent_conversation` 只保存会话元数据，完整消息以 `editor-agent/{conversationId}.json` 会话粒度存 OSS；`api-server` 负责编排 LLM、普通 JSON 消息、OSS 读写和既有生成工具调用。用户消息以独立 `clientMessageId` 在会话锁内幂等，数字 `message.id` 只作后端定位；旧 OSS 消息允许缺失幂等键，早期用户消息字符串 `id` 在读取时迁入 `clientMessageId`。画布 Agent 只与任务侧栏互斥，不与左侧素材 / 图层栏互斥。 - 影响范围：图片画布右侧 Agent 面板、`shared-contracts` / `packages/shared` 的 `editorAgent` 契约、`spacetime-module` / `spacetime-client`、`platform-oss` 内部读签名边界、画布生成落板规则。 -- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`cargo test -p module-editor-agent --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_agent`、前端 Agent 面板与 SSE client 定向测试、`npm run check:encoding`、`git diff --check`。 +- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`cargo test -p module-editor-agent --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_agent`、前端 Agent 面板与 JSON client 定向测试、`npm run check:encoding`、`git diff --check`。 - 关联文档：`docs/【编辑器】画布Agent对话面板-2026-07-03.md`、`docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md`、`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`。 ## 2026-07-01 认证工作集只经 typed projection 同步正式表 @@ -3233,6 +3248,13 @@ - 影响范围：`server-rs/crates/api-server/src/editor_generation_queue.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/editor_project.rs`、`server-rs/crates/api-server/src/character_animation_assets.rs`、`server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs`、`src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts`、`src/services/image-editor/editorProjectClient.ts`。 - 验证方式：`cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server editor_canvas_generation --manifest-path server-rs/Cargo.toml`、`cargo test -p shared-contracts --manifest-path server-rs/Cargo.toml`、`npm run test -- src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx src/services/image-editor/editorProjectClient.test.ts`、`npm run typecheck`、`npm run check:encoding`、`git diff --check`。 +## 2026-07-13 画布 Agent 工具执行状态复用外部生成任务 + +- 决策：画布 Agent 的 OSS 工具消息使用 `status=not_completed|completed|failed|cancelled` 和可选 `externalJobId`；不使用 `cancelledAt`，不新增关联表。`external_generation_job` 是排队、执行、lease 与计费结算的唯一真相；OSS status 只表达该消息回填结果，不复制 queued / running。确认接口按 `conversationId + messageId + toolName` 稳定去重并复用既有编辑器 worker job kind。 +- 懒回填：`GET /conversation` 会在持有 conversation lock 后扫描 `status=not_completed` 且有 `externalJobId` 的工具消息，按 job id 定向读取主任务；完成时复用原工具 formatter 更新 system text、写入轻量媒体引用并标记 `completed`，任务本身失败时写入 `error` 并标记 `failed`。任务结果读取或 completed payload 解析 / formatter 首次失败后，在同一次 GET 内最多重试 3 次，每次等待 100ms 并重新读取主任务；读取失败或 completed 任务暂缺 `result_payload_json` 时，本次重试耗尽后保留 `not_completed + externalJobId` 供下次 GET 继续 reconcile，确定性的 payload 损坏、结构不兼容或 formatter 错误才在重试耗尽后写为 `failed`，避免致命错误永久循环。排队 / 执行保持 `not_completed`。worker 的 `result_payload_json` 只保留 formatter 与媒体引用所需的轻量生成回包，不向通用 summary 状态接口投影。 +- 影响范围：画布 Agent 共享契约、确认/取消接口、编辑器生成入队 helper、对话状态展示与恢复。 +- 验证方式：`cargo check -p api-server -p shared-contracts --manifest-path server-rs/Cargo.toml`、画布 Agent 定向前端测试、`npm run typecheck`、`npm run check:encoding`、`git diff --check`。 + ## 2026-06-18 原生壳 WebView 刷新能力只保留受控当前页刷新 - 背景：Expo 移动壳和 Tauri 桌面壳都需要一个真实的宿主级刷新入口，供 H5 在检测到资源、登录态或运行态需要重新载入时请求宿主刷新当前容器；该能力不能演变成任意 URL 导航或原生 WebView ref 透传。 @@ -4055,7 +4077,7 @@ - 背景：VectorEngine Apifox `api-349239079` 暴露 OpenAI-compatible `POST /v1/chat/completions`；创意 Agent 和通用 LLM 代理需要统一到 VectorEngine 文本服务，并将默认文本模型切换为 `gpt-5.4-mini`。 - 决策：创意 Agent 的 `CREATIVE_AGENT_GPT5_MODEL` 固定为 `gpt-5.4-mini`，协议切到 Chat Completions，不再携带旧 APIMart `official_fallback` 字段；画布 Agent 侧边栏聊天规划请求也复用该模型和 Chat Completions 协议，不再显式使用 `gpt-4o` / Responses。通用 `/api/llm/chat/completions` 代理使用 `GENARRATIVE_LLM_PROVIDER=openai-compatible`、`GENARRATIVE_LLM_BASE_URL=https://api.vectorengine.cn/v1`、`GENARRATIVE_LLM_MODEL=gpt-5.4-mini`。未单独配置 `GENARRATIVE_LLM_API_KEY` 时，api-server 可复用 `VECTOR_ENGINE_API_KEY`；前端 LLM 客户端必须兼容 OpenAI `choices`、api-server raw `{content}` 和项目 envelope `{ok,data:{content}}` 三种非流式响应，以及 OpenAI SSE delta 和 api-server `event: delta` 两种流式响应。 -- 决策补充：画布 Agent 的 planning prompt 必须自动注入上一条已完成生成结果的 `latestGeneratedImage`，来源为上一轮 generation 的 `summary` / `toolName` / `resourceId` / `objectKey` 等轻量摘要。用户用「这张」「刚才那个」「上一张」「把衣服换成……」等方式指代上一张图或继续编辑时，规划默认调用 `edit_image` 并引用该结果；不能因为本轮没有手动附件而退回 `generate_image`。 +- 决策补充：画布 Agent 的 planning prompt 必须自动注入上一条已完成生成结果的 `latestGeneratedImage`，来源为上一轮 generation 的 `toolName` / `resourceId` / `objectKey` 等轻量元数据。用户用「这张」「刚才那个」「上一张」「把衣服换成……」等方式指代上一张图或继续编辑时，规划默认调用 `edit_image` 并引用该结果；不能因为本轮没有手动附件而退回 `generate_image`。 - 决策补充：画布 Agent 侧边栏的“规范图 / 视觉规范图 / 风格规范图 / 素材规范展板”是 Agent 规划 prompt 和 function-calling 工具选择约束，不是侧边栏 UI 说明文案。此类请求默认走 `generate_image`，prompt 必须要求规范展板包含统一视角、线条粗细、色卡、材质、阴影、圆角、状态层级、尺寸标注等视觉规范元素；角色规范图若是规范展板也走 `generate_image`，只有实际角色立绘才走 `generate_character`，多个图标素材 / 图集才走 `generate_icon_spritesheet`。 - 影响范围：`server-rs/crates/platform-agent`、`server-rs/crates/api-server/src/config.rs`、`src/services/llmClient.ts`、`.env.example`、`deploy/env/api-server.env.example`、`scripts/test-ve-llm.mjs`。 - 验证方式：`npm run test -- src/services/llmClient.test.ts`、`cargo test -p api-server --manifest-path server-rs/Cargo.toml from_env_reads_non_public_models_and_urls app_state_builds_creative_agent_gpt5_client_from_vector_engine_settings llm_chat_completions editor_agent_llm_request_uses_vector_engine_chat_model`、`cargo test -p platform-agent --manifest-path server-rs/Cargo.toml`、`npm run check:encoding`、`git diff --check`。 diff --git a/docs/project-memory/shared-memory/document-map.md b/docs/project-memory/shared-memory/document-map.md index 974e0d77c..8aa4b638f 100644 --- a/docs/project-memory/shared-memory/document-map.md +++ b/docs/project-memory/shared-memory/document-map.md @@ -1,6 +1,6 @@ # 文档地图与阅读索引 -更新时间：`2026-07-03` +更新时间：`2026-07-16` ## 当前文档入口 @@ -18,7 +18,7 @@ | 微信小程序虚拟支付 | `docs/【技术方案】微信虚拟支付接入-2026-05-26.md` | | UI 像素资产与 9-slice 规范 | `UI_CODING_STANDARD.md` | | 图片画布生成面板与模型定价 | `docs/【编辑器】生成类面板Lovart统一改造方案-2026-06-17.md`、`docs/【编辑器】模型定价配置管理方案-2026-06-22.md` | -| 图片画布右侧 Agent 对话、会话消息 OSS 持久化、SSE 事件契约、与任务侧栏 / 左侧栏互斥规则 | `docs/【编辑器】画布Agent对话面板-2026-07-03.md`、`docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md` | +| 图片画布右侧 Agent 对话、会话消息 OSS 持久化、普通 JSON 消息、工具任务懒回填及侧栏并存规则 | `docs/【编辑器】画布Agent对话面板-2026-07-03.md`、`docs/adr/【ADR】画布Agent会话消息存OSS-2026-07-03.md` | ## 阅读顺序 diff --git a/docs/technical/【前端架构】SSE客户端传输层收口约定-2026-06-03.md b/docs/technical/【前端架构】SSE客户端传输层收口约定-2026-06-03.md index 567120384..e6198b20e 100644 --- a/docs/technical/【前端架构】SSE客户端传输层收口约定-2026-06-03.md +++ b/docs/technical/【前端架构】SSE客户端传输层收口约定-2026-06-03.md @@ -1,6 +1,6 @@ # SSE 客户端传输层收口约定 -更新时间：`2026-06-03` +更新时间：`2026-07-16` ## 背景 @@ -13,7 +13,6 @@ - `readSseStream` 负责读取 `Response.body`、识别 `\n\n` 与 `\r\n\r\n` 事件边界、合并多行 `data:`、flush `TextDecoder` 尾部缓冲，并支持事件处理函数返回 `false` 后取消 reader。 - `readSseJsonStream` 只在传输事件基础上解析 JSON object，空 data 与异常 JSON 继续按旧口径静默跳过。 - 各业务 client 只保留领域事件归一化、最终结果聚合和中文错误文案，不再重复实现 SSE 边界扫描、reader 循环或 UTF-8 flush。 -- `src/services/image-editor/editorAgentSse.ts` 消费画布 Agent 的 JSON SSE，只校验并归一 `stage`、`message_delta`、`tool_started`、`tool_completed`、`generation_result`、`error`、`done` 事件，不持有 reader 循环或事件边界解析。 - OpenAI 兼容流、`[DONE]` 哨兵或其它非 JSON SSE 可直接使用 `readSseStream`；业务 JSON 事件优先使用 `readSseJsonStream`。 ## 落地范围 @@ -26,27 +25,15 @@ - `src/services/visual-novel-runtime/visualNovelRuntimeSse.ts` - `src/services/rpg-entry/rpgProfileClient.ts` - `src/services/llmClient.ts` -- `src/services/image-editor/editorAgentSse.ts` 后续新增 SSE client 时不得复制 `findSseEventBoundary`、`parseSseEventBlock` 或手写 reader 循环；若确实需要特殊 framing，应先扩展 `sseStream.ts` 的传输能力，再在业务 client 中处理领域语义。 -## 画布 Agent SSE client 口径 - -`src/services/image-editor/editorAgentSse.ts` 是画布 Agent 的业务事件归一层，只允许做以下事情： - -- 调用 `readSseJsonStream(response, handler)` 读取 JSON SSE。 -- 白名单识别 `stage`、`message_delta`、`tool_started`、`tool_completed`、`generation_result`、`error`、`done`。 -- 支持两种兼容输入形态：优先使用 SSE `event:` 字段作为事件名；若 `data` 内含 `{ "event": "...", "data": {...} }`，则归一为同一 `EditorAgentSseEvent`。 -- 对未知事件名、缺失 `data` 或非对象 JSON 静默跳过，保持业务 client 的容错口径。 - -`editorAgentSse.ts` 不得实现 `Response.body.getReader()`、`TextDecoder`、SSE 边界扫描、JSON 字符串拼接、fetch、鉴权、错误文案、会话状态聚合或画布聚焦逻辑；这些职责分别留在 `sseStream.ts`、`editorAgentClient.ts`、`useEditorAgentConversation.ts` 和画布视图层。 +画布 Agent 已改为 `POST /api/editor/agent-conversations/{conversationId}/messages` 普通 JSON 请求，不属于本 SSE 传输层的落地范围；其客户端只通过 `requestJson` 读取 `EditorAgentMessageResponse`，不得为了恢复旧文档口径重新增加私有 SSE parser。 ## 验收 - `src/services/sseStream.test.ts` 覆盖 CRLF / LF 边界、UTF-8 尾部 flush、异常 JSON 跳过和提前停止取消 reader。 - `src/services/llmClient.test.ts` 覆盖 OpenAI 兼容文本流、异常 JSON 跳过和 `[DONE]` 后提前停止。 -- `src/services/image-editor/editorAgentSse.test.ts` 覆盖画布 Agent 的 `stage`、`generation_result`、`done` 事件解析，以及 `{ event, data }` fallback 形态归一。 -- `src/services/image-editor/editorAgentClient.test.ts` 覆盖会话 CRUD 路由和 `/messages/stream` 发起后能收到类型化 SSE 事件。 -- 后续修改 `editorAgentSse.ts` 时，若出现手写 reader、`TextDecoder`、SSE 边界扫描或 fetch 逻辑，应视为违反本收口约定。 +- `src/services/image-editor/editorAgentClient.test.ts` 覆盖会话 CRUD 和 `/messages` 普通 JSON 路由；画布 Agent 不纳入 SSE parser 验收。 - 已有 OpenAI 兼容文本流、NPC 聊天流、创作 Agent、创意互动 Agent、视觉小说运行态和充值订单状态测试继续通过。 - `npm run typecheck` 不产生新的类型错误。 diff --git a/docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md b/docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md index 84c136ca2..0a33c8375 100644 --- a/docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md +++ b/docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md @@ -28,7 +28,7 @@ - 图片快速编辑面板只保留一个提示词输入框和模型选择，不展示额外参考图或比例 / 尺寸控件；原图 / 原素材作为 `/api/editor/images/edits` 的 `sourceImageSrc` 直接提交，不作为 `referenceImageSrcs`。完整图标图集 `icon-spritesheet` 支持快速编辑，拆分后的单个 `icon` 不提供该入口，前后端必须使用同一素材类型规则。打开快速编辑时画布必须自动平移缩放，让原素材完整落在可视区上半部分，底部面板固定出现在素材下方且不遮挡内容，竖屏 UI 素材也必须完整展示。快速编辑右侧显示矩形、椭圆、画笔框选工具，但进入时不默认启用；点击工具后显示选中态，再点同一工具取消启用。完成框选后，画布红色细框显示连续序号，提示词可按这些编号填写每个区域怎么改。点击 `修改` 后仍停留在当前快速编辑面板显示修改中，不创建独立 `Quick Edit Generator` 画布占位；生成成功后直接用结果覆盖原图图层，失败时保留当前面板并在错误红框中显示具体错误文案。 - 底部生成类按钮每次点击都必须创建独立的画布生成对象；新建规范、角色形象或图标素材时，只切换当前编辑面板，不得销毁此前尚未生成或已生成后的其它生成对象状态。归档为非当前编辑对象的生成占位仍可拖动、删除和等待异步完成，完成 / 失败回写必须按生成对象 ID 读取最新占位状态，不能使用提交瞬间的旧快照。 - 画布右上角提供自动隐藏任务侧栏。列表为空且侧栏关闭时只保留图标开关；生成或去背景任务进入时默认打开；用户可手动切换开关状态。进行中阶段只使用外部生成 BFF 返回的 `phaseDetail`：调用或等待图片 / 视频生成服务时显示“正在生成”，进入 BgFilter、逐帧抠图或手动去背景时显示“正在处理”；前端不得按耗时或任务类型猜测阶段。 -- 画布底部工具栏 / 面板 Dock 提供“画布 Agent”入口。点击后打开右侧独立 Agent 对话面板；桌面端为右侧窄面板，移动端占满可用宽度。该面板与素材侧栏、图层侧栏、右上角任务侧栏互斥，打开 Agent 时必须收起其它侧栏，打开其它侧栏或任务侧栏时也必须收起 Agent。Agent 面板不得在当前画布内容下方追加内联内容，也不默认展示大段功能说明文案。 +- 画布底部工具栏 / 面板 Dock 提供“画布 Agent”入口。点击后打开右侧独立 Agent 对话面板；桌面端为右侧窄面板，移动端占满可用宽度。该面板只与右上角任务侧栏互斥；素材 / 图层侧栏允许与 Agent 同时展开，切换左侧栏不得关闭 Agent。Agent 面板不得在当前画布内容下方追加内联内容，也不默认展示大段功能说明文案。 - 所有会新建画布生成占位的入口必须先创建 draft，再统一经过 `ImageCanvasGenerationPlacementModel` 计算落点，禁止各入口自行使用当前视口中心裸坐标或原图右侧固定偏移。当前覆盖入口包括 `生成图片`、`生成规范`、`生成角色形象`、`生成图标素材`、`生成视频`、`生成UI设计图` 和 `生成角色动作`。placement 模型的避让对象为所有未隐藏画布图层，以及当前 active / inactive generation dialogs 中仍存在的 placeholder；每个避让矩形按 32px 画布世界坐标间距外扩。候选落点以当前视口世界中心为距离目标，优先选择离视口中心最近且不重叠的占位位置；若中心被占用，会按上下左右和环形候选继续寻找。打开生成面板时必须把避让后的 placeholder 写入 `openCanvasGenerationDialog(...)`，并立即调用 `centerViewportOnPlacement(...)` 居中到新占位中心，保持原 viewport scale 不变；图片快速编辑不属于新建占位入口，提交后覆盖源图。 ## 交互规则 @@ -59,7 +59,7 @@ - 资源表保存资源和素材级元数据；图层位置、层级、分组选中所需 ID 和 groupId 保存在 `editor_canvas` 的布局 JSON。布局 JSON 是混合数组：普通图层按 `layerId/resourceId` 保存，生成器占位和生成器对话框按 `itemType: "generation-dialog"` 保存，不新增单独表。普通图层的新保存不再把 `assetKind/generationInputs` 写入布局 JSON；刷新时优先从 `editor_project_resource` 恢复，旧布局中的同名字段只作为兼容兜底。生成器快照必须包含生成器 ID、模式、提示词、参数、参考图、状态、占位框位置和可选 `generatedLayerId`；角色、图标等纯色抠图生成器的前端用户路径不保存或恢复 `screenColor` / `segModel`，同源重绘也不再从 `generationInputs.fields` 恢复 `抠图背景色` 或 `抠图模型`；宣发素材生成器还必须保存并恢复 `publicationWorkflowId`、`publicationGameInfo` 和 `publicationReferences`，避免刷新后生成卡片字段或参考图丢失。生成器快照中的参考图同样只保存 `resourceId/sourceAssetId` 行引用和展示所需 label，不保存图片 Data URL、signed URL 或 `objectKey`；刷新时用 `editor_project_resource` / `editor_asset` 行恢复临时生成请求所需图片源。生成成功后仍保存该快照，只是渲染时由 `generatedLayerId` 锚定到成品图层而不重复显示灰色占位框。`generationInputs.references` 是用户可见输入快照中的行级索引，只允许保存 `{ title, label, refType, refId }`；生成接口只接收提交前临时状态解析出的 `objectKey` 或资源 ID；Data URL、Blob URL 和 signed URL 不进入请求体，不进入资源 / 素材元数据。图层展示尺寸不再作为独立 `Size` 真相保存，刷新与新建图层均按 `Resolution`（`originalWidth/originalHeight`）原分辨率显示。图层组第一版是画布内布局语义，不单独建表。 - 图片类、生成视频和音频结果除作为 `editor_project_resource` 和画布图层保存外，还要写入账号级 `editor_asset` 素材库；该写入由生成 BFF 在请求携带 `assetFolderId` 时完成。角色、图标图集、UI 提取和角色动作等多产物任务把实际产生的 provider 原始输出及后处理结果分别入库：所有条目沿用 `character`、`icon-spritesheet`、`character-animation` 等真实类型，provider 原始输出承载任务模型成本，后处理派生产物阶段成本为 0。后台素材查询以最终产物为父行、每个中间产物为可展开的独立子行，分页只计算父任务；手动重拆图集保留独立 `taskId` 用于存储隔离和日志排障，通过私有 provenance 从服务端生成账号素材的 source resource、asset object 或 Object Key 取得可信来源任务，并把它写入 `groupTaskId`，不信任客户端可提交的 resource `taskId/assetKind`；跨项目复用后仍可通过稳定媒体引用找回来源。没有可信来源的新拆分显式归到自身任务，不走历史资源链回溯。每个手动切片同时写入 `groupTaskExpectedAssetCount`，全部切片落库后写独立 cohort 完成事实；后台 read model 只让同一根任务的一个已完成拆分批次并入原图集父项，用户后来删除单片不会让批次脱组，部分失败批次和后续重复拆分批次按各自真实任务分页，避免残缺批次抢占根任务、单组无限增长或素材丢失。历史行在项目资源仍存在时兼容回溯，删除项目资源前只固化直接受影响行的真实来源字段，有界展示 ID 不反写数据库。`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 下，画布图片、改图、图标素材、UI 素材提取、角色动作、视频、音效和背景音乐生成都先返回 `queueState`，前端轮询 `/api/runtime/external-generation/jobs/{jobId}` 到完成后重新读取项目快照；`inline` 或无项目上下文时才使用响应中的 resource / asset 快照做本地落画布兜底，不再把同一生成结果二次调用素材创建接口。生成请求失败、inline 完成或 queue 任务终态完成 / 失败后，右上角泥点 chip 必须通过 `/profile/dashboard` 回读余额，不做本地乐观扣减。生成视频会单独抽取首帧封面写入 `thumbnailSrc`，素材栏和拖回画布时沿用该封面作为 poster。 - 生成面板不展示资源名称输入，默认使用原有自动编号；提示词输入保持统一可见边框。内部命名契约仍使用可选 `assetLabel`，最大 80 字符并在提交时 trim；历史状态或内部调用携带非空名称时，同一个名称必须贯穿 `assetLabel`、`canvasCompletion.title`、项目资源、账号素材和本地兜底图层，刷新后不得退回模板名。图标图集与角色动作请求同样兼容该字段，中间原图使用主名称加固定后缀，拆分素材继续按素材描述命名。 -- 画布 Agent 会话按“SpacetimeDB 元数据 + OSS 消息正文”存储：`editor_agent_conversation` 只保存 `conversationId/projectId/ownerUserId/title/messagesObjectKey/deleted/createdAt/updatedAt` 等会话元数据；消息正文整体保存为私有 OSS JSON 文档 `editor-agent/{conversationId}.json`。消息文档单对象上限为 2 MiB，同一会话的消息追加和 SSE 最终写回由 api-server 按 `conversationId` 串行化，避免“读 OSS → 改消息 → 写 OSS”并发覆盖。前端只通过 api-server BFF 读取和发送会话，不直接读写 SpacetimeDB，也不直接读写 OSS。 +- 画布 Agent 会话按“SpacetimeDB 元数据 + OSS 消息正文”存储：`editor_agent_conversation` 只保存 `conversationId/projectId/ownerUserId/title/messagesObjectKey/deleted/createdAt/updatedAt` 等会话元数据；消息正文整体保存为私有 OSS JSON 文档 `editor-agent/{conversationId}.json`。消息文档单对象上限为 2 MiB，同一会话的消息追加和工具结果回填由 api-server 按 `conversationId` 串行化，避免“读 OSS → 改消息 → 写 OSS”并发覆盖。前端只通过 api-server BFF 读取和发送会话，不直接读写 SpacetimeDB，也不直接读写 OSS。 - Agent 消息附件只允许引用当前工程画布资源或账号素材库图片，来源类型为 `canvas_resource` / `library_asset`，最多 9 张。附件请求可携带展示用 `imageSrc/thumbnailSrc/objectKey/width/height/label`，但持久化真相仍以后端校验后的 resource / asset 行和 OSS 对象为准；不得把 Data URL、signed URL 或 blob URL 当作会话长期事实。 - 前端不直接订阅 SpacetimeDB，统一通过 api-server 的 `/api/editor/projects*` BFF 读写。 - 工程刷新恢复可先应用 session 级轻量项目快照缓存，让画布和素材 chrome 尽快显示；缓存快照必须排除 `data:*` / `blob:` 内联媒体，且在后端项目快照返回前不得触发自动保存。后端快照回来后覆盖本地缓存显示并恢复正常保存队列。 @@ -79,7 +79,7 @@ - `POST /api/editor/projects/{projectId}/agent-conversations`：在当前工程下创建画布 Agent 会话；可选传入标题，默认标题为“新对话”。 - `GET /api/editor/agent-conversations/{conversationId}`：读取指定画布 Agent 会话详情，返回会话摘要和 OSS 消息正文中的消息列表。 - `DELETE /api/editor/agent-conversations/{conversationId}`：软删除指定画布 Agent 会话，并返回删除后的会话摘要。 -- `POST /api/editor/agent-conversations/{conversationId}/messages/stream`：发送画布 Agent 消息并返回 JSON SSE。请求体包含 `clientMessageId`、`text` 和可选 `attachments`；文本与附件不可同时为空。SSE 事件名固定为 `stage`、`message_delta`、`tool_started`、`tool_completed`、`generation_result`、`error`、`done`。生成结果中的图片通过后端写入 project resource / 账号素材后返回轻量快照，前端只消费返回的资源引用并刷新 / 聚焦画布。工具失败时也必须保留失败 generation record，不得只返回一次性错误。 +- `POST /api/editor/agent-conversations/{conversationId}/messages`：发送画布 Agent 消息并返回普通 JSON `EditorAgentMessageResponse`。请求体包含 `clientMessageId`、`text` 和可选 `attachments`；文本与附件不可同时为空，同一会话重复 `clientMessageId` 必须幂等返回或拒绝重复追加。响应包含权威会话摘要、`deltaMessages` 和可选 `errorMessage`。LLM / 规划失败写入 `role=system`、正文以 `ERROR ` 开头的 OSS 消息并放入 `deltaMessages`，不再重复设置 `errorMessage`；前端隐藏前缀后显示红色错误气泡。工具失败继续保存工具状态和错误信息。 - `GET /api/editor/assets/library`：读取当前账号的素材文件夹和素材。首次读取时自动创建“项目素材”默认文件夹。 - `POST /api/editor/assets/folders`：新建素材文件夹。 - `PATCH /api/editor/assets/folders/{folderId}`：重命名、折叠 / 展开素材文件夹。 @@ -103,8 +103,7 @@ - 示例素材可继续复用 `public/creation-type-references/` 下的站内图片；用户上传和后续生成资源必须通过资源记录表达。 - 不把 hover、dragging、临时吸附线、Space 临时抓手等瞬时 UI 状态写入后端。 - 不在 UI 中加入大段功能说明，编辑器界面只展示必要的工具、素材和状态信息。 -- 画布 Agent 前端分层固定为：`editorAgentClient.ts` 负责 BFF 请求、鉴权错误和 stream 发起；`editorAgentSse.ts` 只负责 SSE JSON 事件归一；`useEditorAgentConversation.ts` 负责会话列表、当前会话、消息流、阶段和停止状态；`EditorAgentConversationPanelView.tsx` 只负责右侧面板展示与交互。 -- `editorAgentSse.ts` 必须复用 `src/services/sseStream.ts` 的 `readSseJsonStream`，不得复制 SSE reader、`TextDecoder`、事件边界扫描或 UTF-8 flush 逻辑；该文件不得承接 fetch、鉴权、会话聚合、画布聚焦或 UI 错误文案。 +- 画布 Agent 前端分层固定为：`editorAgentClient.ts` 负责普通 JSON BFF 请求、鉴权和错误映射；`useEditorAgentConversation.ts` 负责会话列表、当前会话、消息请求等待态、工具任务状态和客户端取消等待；`EditorAgentConversationPanelView.tsx` 只负责右侧面板展示与交互。画布 Agent 不新增私有 SSE parser。 - 不复用或改写 `CreativeImageInputPanel` 的单图资产编辑语义；`/editor/canvas` 是独立图片画布工程的画布入口。 ## 验收用例 @@ -124,11 +123,11 @@ - 快速编辑面板底部只显示模型选择和 `修改` 按钮；打开时视口聚焦必须预留底部面板空间，面板位于素材下方，不得遮挡原素材，且素材在当前屏幕内完整可见。快速编辑请求只把原图或红框序号标注图作为 `sourceImageSrc` 直接提交，信息面板输入快照只展示用户填写的快速编辑提示词。 - 点击生成、生成规范、生成角色形象或生成图标素材后创建的占位图可继续保留；点击画布空白区域让当前图片或占位图失焦时，关闭当前生成面板并移除图片选中样式，但不删除占位图本身。 - 生成资源显示元数据按钮，元数据窗口展示来源、生成输入快照、model、task、Resolution 和 OSS 引用；生成输入快照只包含用户面板输入和参考图行引用，不包含后端拼接 Prompt，不再展示独立 Size 字段，也不渲染参考图 Data URL 缩略图。 -- 点击底部 Dock 的“画布 Agent”后，右侧独立 Agent 面板打开；素材 / 图层侧栏和任务侧栏被收起。再次点击或点击面板关闭按钮后收起 Agent。打开素材 / 图层侧栏或任务侧栏时，Agent 面板同步关闭。 +- 点击底部 Dock 的“画布 Agent”后，右侧独立 Agent 面板打开，任务侧栏被收起；素材 / 图层侧栏保持当前状态并可继续切换。再次点击或点击面板关闭按钮后收起 Agent；打开任务侧栏时 Agent 面板同步关闭。 - Agent 面板能读取当前工程会话列表；无历史会话时发送第一条消息会先创建“新对话”。支持新建会话、切换会话和删除当前会话；删除必须通过独立确认弹窗完成，不能在面板下方追加确认内容。 - Agent 输入支持文本消息、附件消息和纯附件消息；附件选择弹窗可在“画布 / 素材库”之间切换，只展示图片类资源，最多选择 9 张。 -- 发送消息后，面板展示用户消息、Agent 阶段状态和 SSE 增量回复；`stage/message_delta/tool_started/tool_completed/generation_result/error/done` 都能被正确渲染。流式响应中点击“停止”会中断当前请求，并把仍在 streaming / generating 的消息标记为停止态。 -- Agent 返回生成结果缩略图后，点击缩略图应优先聚焦当前画布中已有 `resourceId` 对应图层；如果当前内存布局尚未包含该资源，则重新读取工程快照，应用后再聚焦新图层。对话入口触发生成时不创建“即将生成”画布占位；生成中状态只显示在消息流，生成完成后通过后端 `canvasCompletion` 落新图层。工具失败时消息内必须保留失败 generation record 和错误气泡，不能只弹一次性 toast。 +- 发送消息后，面板先展示本地用户消息和请求等待态，再应用普通 JSON 响应中的 `deltaMessages`；客户端取消等待只终止本次 transport 等待，不把已经确认入队的外部生成任务改成停止态。 +- Agent 工具任务完成并懒回填后，消息内缩略图只作纯预览，不显示名称也不点击聚焦图层；前端同时重新读取工程快照和素材库。对话入口触发生成时不创建“即将生成”画布占位，生成完成后由后端 `canvasCompletion` 落新图层。规划或工具失败时消息内必须保留可回读的失败状态和错误气泡，不能只弹一次性 toast 或返回瞬时 `errorMessage`。 - 画布 Agent 会话刷新后能从后端恢复会话标题、消息、附件和生成记录；前端不得根据本地临时状态伪造会话持久化结果。 - 图片选中后的浮动工具栏按钮顺序固定为：快速编辑、分割线、裁扩按钮、去除背景按钮、UI设计图专属提取素材、角色图专属生成动画、分割线、重绘、下载按钮。裁扩通过画布边界拖拉完成，不再展示四边数值输入；默认自由比例，选择固定比例后拖拉边界保持对应比例，完成后在原素材旁边新增裁扩结果图层，扩展区域透明填充。去除背景调用同源 BFF `POST /api/editor/images/background-removals`，由 api-server 通过共享 BgFilter `background_mode=complex` 链路去背景并持久化结果；有项目上下文时先在画布创建关闭面板的去背景生成占位，完成后由后端通过 `canvasCompletion` 把新 project resource 写入该占位并返回快照，无占位上下文时才用新的 project resource 引用替换当前图层。画布任务侧栏按“排队/生成中”和“已完成”分页，生成中排在排队前，生成中耗时从任务开始时间戳实时计算，排队中不计时；进行中任务只显示阶段文本和已用时，不显示百分比；完成态生成任务副标题显示用户提示词并单行截断；点击任务只聚焦对应画布内容，不激活生成面板或改变任务顺序，聚焦时必须预留图片上方工具栏、底部工具栏和可见生成对话框空间。UI设计图的提取素材必须先进入红框素材框选状态，默认启用矩形框选，右侧框选工具与快速编辑统一且可再次点击取消启用态，当前启用工具按钮必须保持高亮。素材提取面板必须在素材下方，使用与生成新素材一致的面板宽度和底部模型 / 按钮样式，提示语显示 `使用框选工具框选你希望从画面中提取的素材`，并展示按原图坐标准确裁剪的框选区域截图预览、固定模型 `gpt-image-2`、左下角计划规格 `1:1·1K/2K` 和 `提取 · N泥点` 按钮，不显示额外取消按钮；点击素材和面板以外的画布区域即退出 UI 素材提取。至少框选一个区域后才可提交，前端把红色轮廓绘入原图后固定走 `gpt-image-2` 和自动决策纯色背景素材提取提示词。透明处理及拆分正常完成时，透明 spritesheet 和拆分素材都按后端快照保留为画布图层；透明处理失败时仅原图作为主结果，既不要求透明图也不要求切片；透明图成功但拆分失败时保留整张透明图并展示拆分告警。三种完成结果都以后端项目快照为准。 - 重绘生成资源后，右侧出现新生成结果图层，并自动 fit 原图 + 新图，且重绘面板保持打开。 diff --git a/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md b/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md index 18b59ea78..7428ce850 100644 --- a/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md +++ b/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md @@ -31,6 +31,7 @@ - `acknowledge_external_generation_job_summaries_and_return`：按当前账号确认已终态任务的完成 / 失败提示，写入摘要投影的 `notification_acknowledged_at` 并追加审计事件。 - `get_external_generation_queue_stats_and_return`：controller 读取队列积压、运行中任务和过期 lease 数量，用于计算 worker 目标实例数；该 procedure 只读 `external_generation_job`，不直接操作 systemd。 - `get_external_generation_job_summary_and_return`：按 `job_id` 从轻量摘要投影读取单个任务状态，给 BFF 和生成页展示使用；必须只返回调用者有权读取的任务，不能暴露其它用户的 payload、错误详情或 worker 内部字段。 +- `get_external_generation_job_result_and_return`：仅供后端内部回填异步编辑器 Agent 工具调用；按 `job_id + owner_user_id` 返回 `status`、`last_error_message` 和已持久化的 `result_payload_json`，不返回请求 payload、lease 或其它 worker 字段。该 procedure 不替代摘要状态读取接口，也不经 BFF 暴露给前端。 不带 `summary / summaries` 的旧 `get / list / acknowledge_external_generation_job*` procedure 只保留给受控内部兼容，不是 BFF 正式读取入口。 diff --git a/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md b/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md index 6c1dc704f..a5bc0abfc 100644 --- a/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md +++ b/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md @@ -78,12 +78,15 @@ npm run check:server-rs-ddd ### 图片画布 Agent 对话 -- `/api/editor/projects/{projectId}/agent-conversations` 负责当前工程会话列表和新建；`/api/editor/agent-conversations/{conversationId}` 负责详情读取和软删；`/api/editor/agent-conversations/{conversationId}/messages/stream` 负责发送消息并返回 SSE。 +- `/api/editor/projects/{projectId}/agent-conversations` 负责当前工程会话列表和新建；`/api/editor/agent-conversations/{conversationId}` 负责详情读取、终态工具消息懒回填和软删；`POST /api/editor/agent-conversations/{conversationId}/messages` 负责发送消息并返回普通 JSON `EditorAgentMessageResponse`，画布 Agent 不提供 `/messages/stream` SSE 路由。消息请求必须携带最长 128 字符的 `clientMessageId`；前端对该 POST 显式启用 1 次瞬时 transport 重试，并复用同一个序列化 body、`clientMessageId` 和 `x-request-id`。同一会话在锁内按该键幂等，重复键同内容返回已有回合或从已保存用户消息继续，异内容返回 `409`。数字 `EditorAgentMessage.id` 仍只作为工具确认 / 取消的后端消息定位符，不能复用为客户端幂等键。 - `module-editor-agent` 只承载纯领域校验：标题派生、附件上限、消息输入规则和会话软删访问规则；不直接依赖 Axum、SpacetimeDB、OSS、LLM 或 Tokio。 - `spacetime-module` 的 `editor_agent_conversation` 只保存元数据；创建、列表、读取、更新时间和软删通过 `create_editor_agent_conversation_and_return`、`list_editor_agent_conversations_and_return`、`get_editor_agent_conversation_and_return`、`touch_editor_agent_conversation_and_return`、`delete_editor_agent_conversation_and_return` procedure 完成，`api-server` 只能经 `spacetime-client` facade 访问。 -- 完整消息文档存 OSS `editor-agent/{conversationId}.json`，由 `api-server` 负责 2 MiB 上限、会话内串行锁、读改写、SSE 事件持久化和 `touch` 元数据更新时间；该 JSON 不进入 `editor_canvas.layers_json`，也不作为画布布局真相。 +- 完整消息文档存 OSS `editor-agent/{conversationId}.json`，由 `api-server` 负责 2 MiB 上限、会话内串行锁、读改写、消息与工具结果持久化和 `touch` 元数据更新时间；该 JSON 不进入 `editor_canvas.layers_json`，也不作为画布布局真相。LLM / 规划失败必须写入 `role=system`、正文以 `ERROR ` 开头的消息，并通过 `deltaMessages` 返回，`errorMessage` 保持为空；前端隐藏前缀并显示红色错误气泡，后端仍把该 system 消息注入后续 LLM memory，使 Agent 能读取失败上下文。工具失败同样必须形成可回读记录，不能只返回瞬时错误。 - 对话附件只允许引用当前工程 `editor_project_resource` 或当前账号 `editor_asset` 的图片；前端可提交展示用 `imageSrc` / `thumbnailSrc`，后端必须按 `resourceId` / `assetId` 重新归一、校验 owner / project 和 `objectKey`，再给 LLM 或生成工具使用。 - 画布 Agent 工具复用既有编辑器图片生成 / 修改 / 图标 spritesheet BFF，并继续使用后端模型定价和 `execute_billable_asset_operation_with_cost`；前端不提交 `priceMudPoints`。 +- `/messages/{messageId}/confirm` 与 `/messages/{messageId}/cancel` 只返回成功确认；前端成功后立即重新读取整个会话，以会话详情中的权威消息状态和 `externalJobId` 驱动气泡展示与任务轮询。 +- 会话详情的终态懒回填必须在单次 GET 和同一 conversation lock 内完成有界重试：任务结果读取、completed payload 解析或工具 formatter 首次失败后最多重试 3 次，每次等待 100ms 并重新读取主任务。任务读取失败或 completed 任务暂缺 `result_payload_json` 时，本次重试耗尽后仍保留 OSS 工具消息的 `not_completed + externalJobId`，由下次会话读取继续 reconcile；JSON 损坏、结果结构不兼容或 formatter 失败等确定性致命错误在重试耗尽后原子写为 `failed`，保存“重试 3 次后仍失败”的最后错误，避免永久循环。 +- 画布 Agent 是“正式任务 payload 不进入通用用户 read model”规则的窄例外消费者：`GET /api/editor/agent-conversations/{conversationId}` 只按会话中已有的 `externalJobId` 定向读取主任务，完成后由对应工具 formatter 从 `result_payload_json` 提取并归一有界的图片 / 视频 / 音频引用，写入 OSS 工具消息后返回。前端仍不得通过通用任务列表 / 状态接口读取或解析 `request_payload_json` / `result_payload_json`；OSS 轻量媒体引用只是会话展示与后续 Agent 上下文，不替代 `editor_project_resource`、`editor_asset`、`editor_canvas.layers_json` 或 `external_generation_job` 的业务真相。 ### 创作 / 游玩统一流程主干 @@ -282,8 +285,8 @@ npm run check:server-rs-ddd - Rust 结构体：`ExternalGenerationJob` - 源码：`server-rs/crates/spacetime-module/src/external_generation.rs` - 用途：外部生成 worker 的内部持久任务队列；`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时，`api-server` HTTP 角色只入队，`external-generation-worker` 角色通过 claim lease 领取、续租、执行，并用 `lease_token` 栅栏回写阶段、完成 / 失败。队列行继续保存 worker 执行、计费与滚动发布兼容所需字段，末尾可选 `phase` 只取 `generating / processing`；claim 写 `generating`，真实进入抠图处理时由受 `job_id + worker_id + lease_token` 保护的 procedure 写 `processing`。phase procedure 以结构化结果区分 `LeaseFencingRejected` 与 `OtherRejected`；`LeaseFencingRejected` 立即终止，`OtherRejected` 以及 SDK 的 `Procedure` / `Runtime` 错误不重试，只有 `Build` / `ConnectDropped` / `Timeout` 在同一个 job attempt 内重试一次。该重试只重新上报 phase，不把任务写回 `pending`，也不重新调用 provider；编辑器 job 入队固定 `max_attempts=1`，第二次传输失败后任务进入 `failed`，不会回到 `pending` 或从 provider 生成起点重跑。用户可见任务列表、价格、状态、阶段、未确认终态数量和通知确认时间的正式读取事实源已经迁到 `external_generation_job_summary`；BFF 不得再为列表 / 详情 / acknowledge 读取该大表。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft`、`generate_puzzle_images` 与 `generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，避免过期 worker 写 session / work profile；图片画布编辑器的 `editor_image_generation`、`editor_image_edit`、`editor_background_removal`、`editor_icon_spritesheet_generation`、`editor_ui_design_asset_extraction`、`editor_character_animation_generation`、`editor_video_generation`、`editor_sound_effect_generation` 和 `editor_background_music_generation` 复用同一队列表，worker 成功后经 `api-server` facade 写入 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`，前端只通过 BFF job 状态轮询和项目快照读取恢复完成态。`GENARRATIVE_EXTERNAL_GENERATION_MODE=inline` 时不创建该队列行，三个 external generation guard 字段必须同时为空才允许 api-server 受控同步写回，半空 guard 仍会拒绝。worker 成功写回业务事实后才能 complete job；业务失败态写回成功后才能 fail job，失败态未写回时保留租约等待后续重领。 -- 载荷约束：本次先对 `source_module = editor-canvas` 的 `request_payload_json` / `result_payload_json` 实施有限大小合法 JSON、任意层级禁止 `data:` / `blob:` 的双层门禁，只保存 worker 执行必需的普通参数和已登记媒体引用；其它玩法在完成各自参考图资源化之前不由本次门禁静默改变既有请求契约。该主表只供 worker claim / 执行和受控维护读取；正式用户任务列表、单任务状态、队列概览与 acknowledge 不得再返回或解析这两个 payload。 -- 非阻断告警：角色形象、图标图集和 UI 素材提取已保存 provider 原图、但透明背景处理最终失败时，以原图唯一主图完成任务；透明图和切片不写入画布。这个 source-only 降级只包住透明背景处理的最终失败，phase 上报、provider 原图持久化、透明处理图持久化或画布写回失败仍按任务错误传播。图标 / UI 透明图集成功但自动拆分降级时仍保留透明图集；通用 `warning` 与 `sliceWarning` 互斥。两类成功降级都以既有 `completed` 状态收口，不新增状态值：source-only 的 inline / external v1 响应使用结构化 `warning.code/reason`，仅拆分失败的 inline / external v1 响应继续使用既有 `sliceWarning.code/reason`，其 `reason` 保留原始诊断；queue worker 才把两者归一为有界的 `result_payload_json.warning`，且通用 `warning` 优先并原样保留完整 `reason`，只有 `sliceWarning.reason` 由 worker 添加“图集已生成，但自动拆分未完成：”前缀。队列结果不保存图片、切片列表或媒体 URL。 +- 载荷约束：本次先对 `source_module = editor-canvas` 的 `request_payload_json` / `result_payload_json` 实施有限大小合法 JSON、任意层级禁止 `data:` / `blob:` 的双层门禁，只保存 worker 执行必需的普通参数和已登记媒体引用。画布 Agent 来源的任务可在 `result_payload_json.editor-agent-tool-call-result` 中保存有界的轻量结果和已登记媒体引用，供后端按已有 `externalJobId + owner_user_id` 定向懒回填；其它编辑器任务保持元数据结果，并可保存有界的 `warning.code/reason`。其它玩法在完成各自参考图资源化之前不由本次门禁静默改变既有请求契约。该主表只供 worker claim / 执行、受控维护以及画布 Agent 的定向结果回填读取；正式用户任务列表、单任务状态、队列概览与 acknowledge 不得返回或解析这两个 payload。画布 Agent 懒回填必须经对应工具 formatter 归一为有界轻量媒体引用后写入 OSS 会话，不能把原始 payload 直接透传前端。 +- 非阻断告警：角色形象、图标图集和 UI 素材提取已保存 provider 原图、但透明背景处理最终失败时，以原图唯一主图完成任务；透明图和切片不写入画布。这个 source-only 降级只包住透明背景处理的最终失败，phase 上报、provider 原图持久化、透明处理图持久化或画布写回失败仍按任务错误传播。图标 / UI 透明图集成功但自动拆分降级时仍保留透明图集；通用 `warning` 与 `sliceWarning` 互斥。两类成功降级都以既有 `completed` 状态收口，不新增状态值：source-only 的 inline / external v1 响应使用结构化 `warning.code/reason`，仅拆分失败的 inline / external v1 响应继续使用既有 `sliceWarning.code/reason`，其 `reason` 保留原始诊断；queue worker 才把两者归一为有界的 `result_payload_json.warning`，且通用 `warning` 优先并原样保留完整 `reason`，只有 `sliceWarning.reason` 由 worker 添加“图集已生成，但自动拆分未完成：”前缀。除上述画布 Agent 定向回填的轻量结果外，队列结果不保存图片、切片列表或媒体 URL。 ### `external_generation_job_summary` diff --git a/docs/【编辑器】画布Agent对话面板-2026-07-03.md b/docs/【编辑器】画布Agent对话面板-2026-07-03.md index 632598beb..bbb9f3286 100644 --- a/docs/【编辑器】画布Agent对话面板-2026-07-03.md +++ b/docs/【编辑器】画布Agent对话面板-2026-07-03.md @@ -1,6 +1,6 @@ # 画布Agent对话面板 -日期：`2026-07-03` +日期：`2026-07-16` ## 定位与边界 @@ -8,9 +8,9 @@ - 它是画布域工具，**不承接玩法创作**、不产出玩法作品或模板，与 `CONTEXT.md` 中「表单/图片输入创作工作台」的 Avoid 边界不冲突。 - 独立于拼图专用的 `/api/runtime/creative-agent/sessions`（该会话为 api-server 内存态、拼图领域专用，不复用）。 -## 能力范围（第一期） +## 能力范围 -对话 Agent 可通过 function-calling 触发以下五类工具，全部复用既有计费收口接口： +对话 Agent 可通过 function-calling 触发以下八类工具，全部复用既有计费收口接口： | 工具 | 后端接口 | | --- | --- | @@ -19,19 +19,22 @@ | 生成角色形象 | 既有角色形象生成入口对应接口 | | 生成图标素材 | `POST /api/editor/icon-spritesheets/generations` | | 生成 UI 设计图 | 既有 UI 设计图生成入口对应接口 | +| 生成视频 | `POST /api/editor/videos/generations` | +| 生成游戏音效 | `POST /api/editor/audios/sound-effects/generations` | +| 生成背景音乐 | `POST /api/editor/audios/background-music/generations` | - 意图解析与工具编排在后端 api-server，前端只渲染状态，不承接业务规则。 - 下面的工具选择口径属于 Agent 规划 prompt / function-calling 约束，不是侧边栏 UI 说明文案；侧边栏面板不展示这些规则解释。 - 用户要求“规范图 / 视觉规范图 / 风格规范图 / 素材规范展板”时，规划默认选择 `generate_image`，并在 prompt 中明确要求生成规范展板，包含统一视角、线条粗细、色卡、材质、阴影、圆角、状态层级、尺寸标注等可落地的视觉规范元素。 - 用户要求“角色规范图”且语义是角色的规范展板、风格展板或设定板时，仍走 `generate_image`，不要误分流到 `generate_character`；只有实际生成角色立绘、角色主形象或角色视觉资产时才走 `generate_character`。用户要求多个图标素材、图集或 spritesheet 时才走 `generate_icon_spritesheet`。 - 所有生成必须走 `execute_billable_asset_operation_with_cost` 与模型定价配置，禁止绕过定价收口。 -- 视频 / 音频 / 图层操作等其余画板功能第一期不进入对话工具面，仍走现有面板。 +- 图层操作及其他未注册的画板功能第一期不进入对话工具面，仍走现有面板。 ## 当前分支落地状态 -- 已落地：会话元数据、OSS 消息文档、会话 CRUD、SSE 消息流、后端 LLM 工具规划、右侧对话面板、会话历史、新建 / 软删会话、停止当前 SSE 回合、附件从画布资源 / 账号素材库选择，以及五类图片工具对既有生成入口的复用。 -- 已落地：`tool_started` / `tool_completed` 事件携带 `status`；工具失败时也会写入失败 generation record，并随后发送 `stage=failed` 与 `error`，前端应保留消息内失败条目。 -- 未落地：附件弹窗末尾上传格、跨刷新异步生成恢复、external generation task 轮询回填。未落地前，对话消息状态只表示本次 SSE 回合记录，不作为后台任务队列真相。 +- 已落地：会话元数据、OSS 消息文档、会话 CRUD、带 `clientMessageId` 幂等键的普通 JSON 消息请求、后端 LLM 工具规划、右侧对话面板、会话历史、新建 / 软删会话、附件从画布资源 / 账号素材库选择，以及八类图片 / 音视频工具对既有生成入口的复用。 +- 已落地：工具确认 / 取消、external generation task 轮询与会话懒回填。LLM 未配置、请求失败或规划结果解析失败时，后端把 `role=system`、正文以 `ERROR ` 开头的消息写入 OSS，并通过 `deltaMessages` 返回，`errorMessage` 保持为空；前端隐藏 wire 前缀并以红色错误气泡展示。工具执行失败继续保存 `status=failed`、模型和错误信息，不能只返回瞬时错误。 +- 未落地：附件弹窗末尾上传格。`external_generation_job` 继续作为后台任务队列真相，对话消息只保存确认、回填状态和轻量媒体结果引用。 ## 会话与持久化 @@ -39,24 +42,27 @@ - SpacetimeDB 新表 `editor_agent_conversation` 只存会话元数据：会话 ID、projectId、ownerUserId、标题、软删标记、聊天记录 OSS 对象引用、创建/更新时间。 - 完整消息内容存 OSS：`editor-agent/{conversationId}.json`，**会话粒度整体读写**（追加消息=重写对象），不按消息拆对象。 - 不把对话塞进画布工程快照 payload，不在 api-server 内存中保存会话真相。 -- 会话标题：新会话默认「新对话」，首条用户消息发出后自动截取前 N 字作为标题；第一期不做手动重命名。 +- 会话标题：新会话默认「新对话」，首条含文本的用户消息发出后自动截取前 N 字作为标题；列表摘要、详情和消息回包均携带同一必填标题，前端只展示该标题，不以会话 ID 或本地推导兜底。标题写入失败会使该消息请求失败，不能静默继续。 - 会话删除：列表项 hover 出删除按钮 + 确认；软删（表打 deleted 标记，OSS 对象保留）。 +- 每次用户主动发送生成一个最长 128 字符的 `clientMessageId`；`editorAgentClient` 对网络错误和通用瞬时状态码显式启用 1 次 POST transport 重试，重试复用同一个已序列化 body、`clientMessageId` 和 `x-request-id`。该字段独立于数字 `message.id` 并随用户消息写入 OSS。旧消息缺失时按 `None` 兼容；早期 SSE 文档若把客户端键存成用户消息字符串 `id`，读取时将其迁入 `clientMessageId`，同时重建数字定位符。后端在会话锁内检查重复键：内容一致时返回已持久化的同一回合结果，尚无结果时复用原用户消息继续规划；文本或附件身份不同则返回 `409`，不得再次追加用户消息或调用 LLM。 ## 生成结果落画板（对现有占位规则的例外） -- 对话入口触发的生成**不创建"即将生成"画布占位**（区别于其余生成面板）；生成中状态由对话消息流承载。 -- 生成完成后：结果图按统一 placement 避让模型（视口中心就近、避开现有图层、32px 间距）落画板为新图层，同时登记到默认项目素材库，并在对话消息内显示纯缩略图；前端收到 `generation_result` 后立即刷新工程快照与素材库，缩略图本身不显示名称也不承担图层跳转。 +- 对话入口触发的生成**不创建"即将生成"画布占位**（区别于其余生成面板）；生成中状态由工具消息和外部任务状态承载。 +- 生成完成后：结果图按统一 placement 避让模型（视口中心就近、避开现有图层、32px 间距）落画板为新图层，同时登记到默认项目素材库；前端轮询到任务终态并重新读取会话后，以回填的轻量媒体引用显示纯缩略图并刷新工程快照与素材库，缩略图本身不显示名称也不承担图层跳转。 - 消息内生成结果缩略图必须携带并优先使用 `objectKey` / `assetObjectId`，前端通过 `ResolvedAssetImage` / `/api/assets/read-url` 换签后渲染，不能把裸 `/generated-*` 私有路径直接交给 ``。 -- 当前第一阶段通过既有编辑器生成 BFF 的 `canvasCompletion` 写回工程快照；刷新后异步任务恢复和轮询回填属于后续能力，不在本阶段声明为已完成。 +- 既有编辑器 worker 通过 `canvasCompletion` 写回工程快照；刷新后由 external generation task 状态和会话懒回填恢复结果。 - 该例外已同步登记在《生成类面板Lovart统一改造方案-2026-06-17》「画布占位落点」节。 ## 右侧布局 - 对话框与既有任务侧栏（`ImageCanvasTaskSidebarView`）**互斥展开**：展开一个自动收起另一个；各自收起后保留入口按钮。 -- 对话框与左侧素材 / 图层侧栏也互斥：打开画布 Agent 时收起左侧栏；再次打开素材、图层或任务侧栏时收起 Agent 面板。 +- 对话框与左侧素材 / 图层侧栏**不互斥**，允许同时展开，便于在对话中选取和核对画布素材；左侧栏切换不改变 Agent 面板开关状态。 - 桌面端对话框固定宽约 360–400px；移动端抽屉式全宽覆盖；收起态为胶囊/圆形入口按钮。 - 会话管理入口在对话框头部：当前会话标题 + 历史会话下拉（按更新时间倒序）+ 新建对话按钮，全部包在对话框内。 -- 收起对话框只是隐藏面板，不卸载当前会话 hook；流式回复、`生成中` 阶段和停止按钮状态必须在收起 / 重新打开之间保持一致。 +- 快速切换会话或会话轮询刷新产生并发详情请求时，前端只允许最后发起的请求更新当前会话、消息、错误和加载态；旧响应不得覆盖用户最新选择。 +- 普通 JSON 消息请求的回包必须绑定发送时的会话：用户在等待期间切换到其他会话后，只更新原会话的列表摘要，不得把原会话的 `deltaMessages` 、错误或画布刷新副作用应用到当前面板。 +- 收起对话框只是隐藏面板，不卸载当前会话 hook；普通 JSON 消息请求的等待态和外部生成任务状态必须在收起 / 重新打开之间保持一致。 ## 附件 @@ -68,11 +74,29 @@ - 应用后附件以胶囊 chip 挂在输入框上方；发出的消息内附件渲染为纯文本胶囊 chip（名称 + 小图标），**默认无缩略图，鼠标悬浮才浮出缩略图预览**。 - 附件领域形状：统一为画布资源 / 素材库对象引用（`resourceId` / `assetId` + 可选 `objectKey`），不存在只属于对话的第三种图；单条消息上限 9 张（前后端共同校验）。前端可携带展示用 `imageSrc` / `thumbnailSrc`，后端必须按当前工程和当前账号重新归一、校验归属与 `objectKey`。 +## 工具调用确认展示契约 + +- Agent 规划出生成或编辑工具后，先把 `status=not_completed` 且没有 `externalJobId` 的工具消息持久化为待确认记录；确认卡必须在真正调用生成 provider 前展示本次提示词、规格参数、目标图和参考图缩略图，用户确认后才执行，取消后保留同一条 `status=cancelled` 记录。内部 system 文本和图片哈希 ID 不直接展示给用户。 +- 工具消息只保存 `status`（`not_completed` / `completed` / `failed` / `cancelled`）和可选 `externalJobId`。`external_generation_job` 仍是队列、执行、lease 与计费结算真相；OSS status 仅表示该条对话消息是否已经回填完成结果或失败，不复制 queued / running。 +- 确认接口必须先把工具参数转换为既有编辑器 worker payload，再使用 `editor-agent:{conversationId}:{messageId}:{toolName}` 稳定 dedupe key 入队；同一确认的请求重试只能得到同一个 external job。入队成功后把返回的 job id 写回同一条 OSS 工具消息，不新增 Agent 工具执行关联表。 +- 前端根据 `externalJobId` 查询通用 external-generation job 状态；worker 继续通过 `canvasCompletion` 把生成结果写回工程与素材库。浏览器断线、刷新或 api-server 重启不得导致确认接口重新扣费或重新提交 provider。 +- `GET /conversation` 会在同一个 conversation lock 内扫描 `status=not_completed` 且已有 `externalJobId` 的工具消息：只对这些消息按 job id 定向读取主任务；任务完成后复用对应工具的 `format_execute_message` 替换 system text、回填轻量图片 / 视频 / 音频引用并写为 `completed`，任务失败则回填 `error` 并写为 `failed`。任务结果读取或 completed payload 解析 / formatter 回填失败时，必须在同一次 GET 内完成首次尝试及最多 3 次重试，三次重试各间隔 100ms 并重新读取任务结果；仍失败才把该工具消息写为 `failed` 并保存最后错误。该重试不依赖前端再次刷新。排队和执行中都保持 `not_completed`，整轮扫描结果一次性写回 OSS。 +- `EditorAgentToolCall.args` 保留为工具返回的原始 JSON，是确认接口重新反序列化并执行工具的唯一参数真相。图片参数继续只保存由真实 data key 计算出的 opaque SHA-256 `imageId`；不得为了前端预览把 `args` 中的图片 ID 改写成 `objectKey`、URL 或展示对象，也不得由前端重组或回传一份新的执行参数。 +- `EditorAgentToolCall.displayArgs` 是必填、只读的用户确认展示投影，与 `args` 分离： + - `stringArgs` 保存提示词、比例、清晰度、模型、时长等可展示参数的稳定名称、用户可见标题和值； + - `imageArgs` 按“目标图片 / 参考图片”等参数分组，每个 `refs` 项包含与原始参数对应的 `imageId`，以及后端从已校验会话上下文解析出的 `objectKey`、`imageSrc`、可选 `thumbnailSrc` / `label` / `width` / `height`。 + - `extras.priceMudPoints` 保存创建待确认消息时按后端运行时模型定价快照计算的预计泥点消耗；前端统一展示为“预计消耗 N泥点”，不自行计算价格。 +- `displayArgs` 只能由 api-server 按已注册 tool 白名单，基于已经通过 ToolArgs 校验的 `args` 和当前 OSS 会话文档中的附件 / 历史生成结果构建；不能信任 LLM 自报的展示地址、标题或素材元数据。展示投影不参与确认执行，确认接口仍只读取同一条持久化 tool call 的 `args`，避免“看到的素材”和“实际执行的素材”分叉。 +- `extras.priceMudPoints` 同样只属于展示投影，不作为扣费输入；确认后仍由既有生成 BFF 按后端运行时定价执行预扣费，因此该字段表达用户确认时看到的价格快照，而不是前端可提交或覆盖的计费真相。 +- `EditorAgentToolCall.summary` 只是 `args` 的重复字符串且没有稳定语义，当前契约删除该字段，不再作为展示或执行输入。 +- 前端待确认卡只消费必填 `displayArgs`，不解析各 tool 私有的 snake_case / camelCase schema，也不把 `sha256:*` ID 当标题或图片地址。图片统一通过 `ResolvedAssetImage` 使用 `objectKey` 换签后显示，签名 URL 不进入消息文档。模块尚未上线，不保留缺少 `displayArgs` 时读取 raw `args` 的旧消息降级路径。 + ## LLM 与计费 -- 编排复用 `creative_agent_gpt5_client` 的 LLM 接入配置（同 provider/env，独立用途标识），画布 Agent 规划请求固定使用 VectorEngine `gpt-5.4-mini` Chat Completions；function-calling 注册五类工具。 -- 每个用户回合必须由 LLM 返回结构化计划；LLM 未配置、请求失败或返回格式不可解析时，后端写入明确错误消息，不使用本地关键词或“收到：...”回显兜底。 -- 规划 prompt 必须自动带入上一条已完成生成结果的 `latestGeneratedImage` 引用，内容只包含上一轮 generation 的 `summary` / `toolName` / `resourceId` / `objectKey` / `assetObjectId` 等轻量元数据，不把私有签名 URL 或大图内容塞进 prompt。 +- 编排复用 `creative_agent_gpt5_client` 的 LLM 接入配置（同 provider/env，独立用途标识），画布 Agent 规划请求固定使用 VectorEngine `gpt-5.4-mini` Chat Completions；function-calling 注册八类工具。 +- 每个用户回合必须由 LLM 返回结构化计划；LLM 未配置、请求失败或返回格式不可解析时，后端写入正文为 `ERROR <错误内容>` 的 system 消息，不使用本地关键词或“收到：...”回显兜底。该错误消息与其它 system 消息一样进入后续 LLM memory，使 Agent 能看到上一轮失败上下文。 +- 规划 prompt 必须自动带入上一条已完成生成结果的 `latestGeneratedImage` 引用，内容只包含上一轮 generation 的 `toolName` / `resourceId` / `objectKey` / `assetObjectId` 等轻量元数据，不把私有签名 URL 或大图内容塞进 prompt。 +- 工具参数中的图片 ID 是由真实 object key 或图片地址计算的稳定 SHA-256 标识；真实 data key 仅存于 api-server 的工具上下文映射，所有图片工具在执行时查表恢复，不能把 object key 或图片地址作为 LLM 可见的工具 ID。 - 用户使用「这张」「刚才那个」「上一张」「把衣服换成……」等方式指代或编辑上一张结果图时，LLM 默认选择 `edit_image` 并引用 `latestGeneratedImage` 作为源图；除非用户明确要求全新生成，否则不能因为本轮没有重新上传附件而降级为 `generate_image`。 - 规划 prompt 必须显式区分“规范展板”和“实际素材产出”：规范图、视觉规范图、风格规范图、素材规范展板、角色规范图等规范展板请求走 `generate_image`，并补齐统一视角、线条粗细、色卡、材质、阴影、圆角、状态层级、尺寸标注等要求；实际角色立绘才走 `generate_character`，多个图标素材 / 图集才走 `generate_icon_spritesheet`。 - 画布 Agent 规划请求使用 Chat Completions、1024 `max_tokens` 和 60 秒 Agent 专用请求超时；生成图片/编辑图片仍走对应生成工具和模型计费。 @@ -83,13 +107,13 @@ 做（第一期）： -1. 助手文本 SSE 流式输出； -2. 阶段提示行（思考中 → 思考完成 → 生成中 → 完成/失败）； +1. 助手文本随普通 JSON 消息响应一次性返回； +2. 消息请求等待态，以及工具任务的待确认、生成中、完成 / 失败状态； 3. 工具/模型标注行（生成时显示模型名 + 图标）； 4. 消息内生成结果缩略图（纯预览，不显示名称，不点击聚焦图层）； 5. 生成中的进行中动画； 6. 错误气泡（失败/余额不足，带原因）； -7. 发送中断：进行中时发送按钮变「停止」，可中断当前回合（已提交的生成任务不追回，照常落画板）。 +7. 普通消息请求等待期间禁用发送按钮，不提供客户端停止操作；前端持续等待后端响应，避免后端已持久化消息但前端中断请求后产生会话状态错位。 不做（明确排除，防止后人补齐）： @@ -113,22 +137,22 @@ - `api-server`： - `GET/POST /api/editor/projects/{projectId}/agent-conversations`（列表/新建）； - `GET/DELETE /api/editor/agent-conversations/{conversationId}`（详情/软删）； - - `POST /api/editor/agent-conversations/{conversationId}/messages/stream`（SSE）； + - `POST /api/editor/agent-conversations/{conversationId}/messages`（JSON）； - Agent 编排（function-calling 循环、工具内部调既有生成执行链路）放 api-server 编排层，独立文件，不复用 `creative_agent.rs` 内存会话。 -- `shared-contracts` + `packages/shared`：新 `editorAgent` DTO 与 SSE 事件契约（`stage`、`message_delta`、`tool_started`、`tool_completed`、`generation_result`、`error`、`done`）。 +- `shared-contracts` + `packages/shared`：`editorAgent` 会话、消息、工具确认展示与轻量媒体结果 DTO；消息响应返回 `conversation`、`deltaMessages` 和可选 `errorMessage`。 ## 实施顺序 1. 契约与领域规则（shared-contracts / packages/shared + module-editor-agent）； 2. 存储层（spacetime-module 表 + procedure + migration + spacetime-client + schema check）； -3. api-server 会话 CRUD + 消息 OSS 读写 + SSE 回显桩（不接 LLM，先保证会话链路端到端真实落库）； +3. api-server 会话 CRUD + 消息 OSS 读写 + JSON 回显桩（不接 LLM，先保证会话链路端到端真实落库）； 4. 前端最小纵切（对话框、会话管理、消息流、附件弹窗、侧边栏/小地图默认值）——可与 3 并行：3 只碰 `server-rs/`，4 只碰 `src/` 且先以契约 mock 客户端联调，汇合点在 4 末接真实 API； -5. LLM 编排 + 五类工具接入 + 生成落画板 + 停止/错误态（替换回显桩这一个点）； +5. LLM 编排 + 八类工具接入 + 生成落画板 + 请求等待 / 错误态（替换回显桩这一个点）； 6. 验证与文档：定向测试、类型检查、`npm run check:encoding`、`git diff --check`、`npm run check:spacetime-schema`、api-server smoke `/healthz`；同步 `CONTEXT.md` 与相关文档。 ## 第一阶段验收补充 -- 打开画布 Agent 后，任务侧栏和左侧素材 / 图层面板应关闭；再次打开任务侧栏或素材 / 图层面板时，Agent 面板应关闭。 -- 发送消息时先本地追加用户消息，再消费 SSE 增量；停止按钮只中断当前 SSE 回合，不追回已经提交的生成工具调用。 -- Agent 消息内生成结果缩略图只用于预览，不显示名称，也不点击跳转图层；收到 `generation_result` 时统一刷新工程快照和素材库。 +- 打开画布 Agent 后任务侧栏应关闭，再次打开任务侧栏时 Agent 面板应关闭；素材 / 图层面板与 Agent 可同时展开，互不改写开关状态。 +- 发送消息时先本地追加用户消息，再应用 JSON 响应中的 `deltaMessages`；请求等待期间发送按钮保持禁用，前端不主动中断当前回合。 +- Agent 消息内生成结果缩略图只用于预览，不显示名称，也不点击跳转图层；轮询到任务终态并完成会话懒回填后统一刷新工程快照和素材库。 - 对话内容可被用户选中复制；用户从输入框或对话内容点击回画布图层 / 生成器时，焦点应回到画布对象，Backspace / Delete 等画布快捷键继续生效。 diff --git a/packages/shared/src/contracts/editorAgent.ts b/packages/shared/src/contracts/editorAgent.ts index a7daaeb33..b85143eae 100644 --- a/packages/shared/src/contracts/editorAgent.ts +++ b/packages/shared/src/contracts/editorAgent.ts @@ -1,38 +1,18 @@ // 画布Agent对话契约：会话元数据存 SpacetimeDB，消息正文整体存 OSS（editor-agent/{conversationId}.json）。 export const EDITOR_AGENT_MAX_ATTACHMENTS = 9; -export const EDITOR_AGENT_TITLE_MAX_CHARS = 20; -export const EDITOR_AGENT_DEFAULT_CONVERSATION_TITLE = '新对话'; -export const EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION = 1; +export const EDITOR_AGENT_ERROR_MESSAGE_PREFIX = 'ERROR '; -export type EditorAgentStage = - | 'idle' - | 'thinking' - | 'responding' - | 'generating' - | 'completed' - | 'failed'; +export type EditorAgentMessageRole = 'user' | 'assistant' | 'system'; -export type EditorAgentMessageRole = 'user' | 'assistant'; - -export type EditorAgentMessageKind = 'chat' | 'stage' | 'error'; - -export type EditorAgentMessageStatus = - | 'streaming' - | 'generating' +export type EditorAgentToolCallStatus = + | 'not_completed' | 'completed' | 'failed' - | 'stopped'; + | 'cancelled'; export type EditorAgentAttachmentSource = 'canvas_resource' | 'library_asset'; -export type EditorAgentToolName = - | 'generate_image' - | 'edit_image' - | 'generate_character' - | 'generate_icon_spritesheet' - | 'generate_ui_design'; - export interface EditorAgentAttachmentRef { source: EditorAgentAttachmentSource; referenceId: string; @@ -45,36 +25,87 @@ export interface EditorAgentAttachmentRef { } export interface EditorAgentGeneratedImage { - resourceId: string | null; + resourceId?: string | null; objectKey?: string | null; assetObjectId?: string | null; imageSrc: string; - thumbnailSrc: string | null; - width: number | null; - height: number | null; + thumbnailSrc?: string | null; + width?: number | null; + height?: number | null; } -export type EditorAgentGenerationStatus = 'generating' | 'completed' | 'failed'; +export interface EditorAgentGeneratedVideo { + resourceId?: string | null; + objectKey?: string | null; + assetObjectId?: string | null; + videoSrc: string; + thumbnailSrc?: string | null; + width?: number | null; + height?: number | null; +} -export interface EditorAgentGenerationRecord { - toolCallId: string; - toolName: EditorAgentToolName; - summary?: string | null; - taskId: string | null; - status: EditorAgentGenerationStatus; - model: string | null; +export interface EditorAgentGeneratedAudio { + resourceId?: string | null; + objectKey?: string | null; + assetObjectId?: string | null; + audioSrc: string; +} + +export interface EditorAgentToolCallStringArg { + name: string; + label: string; + value: string; +} + +export interface EditorAgentToolCallImageRef { + imageId: string; + imageSrc: string; + objectKey?: string | null; + thumbnailSrc?: string | null; + label?: string | null; + width?: number | null; + height?: number | null; +} + +export interface EditorAgentToolCallImageArg { + name: string; + label: string; + refs: EditorAgentToolCallImageRef[]; +} + +export interface EditorAgentToolCallDisplayExtras { + priceMudPoints: number; +} + +export interface EditorAgentToolCallDisplayArgs { + stringArgs: EditorAgentToolCallStringArg[]; + imageArgs: EditorAgentToolCallImageArg[]; + extras: EditorAgentToolCallDisplayExtras; +} + +export interface EditorAgentToolCall { + toolName: string; + status: EditorAgentToolCallStatus; + args: unknown; + displayArgs: EditorAgentToolCallDisplayArgs; + externalJobId?: string | null; images: EditorAgentGeneratedImage[]; + // Older persisted conversation documents do not contain these media fields. + videos?: EditorAgentGeneratedVideo[]; + audios?: EditorAgentGeneratedAudio[]; error?: string | null; } export interface EditorAgentMessage { - id: string; + // Frontend must not use this to organize messages. + // It is an opaque backend locator for pending tool-call operations. + id: number; + // Present only on user messages created from a client send request. + clientMessageId?: string | null; role: EditorAgentMessageRole; - kind: EditorAgentMessageKind; text: string; attachments: EditorAgentAttachmentRef[]; - generations: EditorAgentGenerationRecord[]; - status: EditorAgentMessageStatus; + toolCall: EditorAgentToolCall | null; createdAt: string; } @@ -82,12 +113,15 @@ export interface EditorAgentConversationSummary { conversationId: string; projectId: string; title: string; - createdAt: string; updatedAt: string; } -export interface EditorAgentConversationDetail - extends EditorAgentConversationSummary { +export interface EditorAgentConversationDetail { + conversationId: string; + projectId: string; + title: string; + createdAt: string; + updatedAt: string; messages: EditorAgentMessage[]; } @@ -109,63 +143,14 @@ export interface EditorAgentConversationResponse { conversation: EditorAgentConversationDetail; } -export interface StreamEditorAgentMessageRequest { +export interface EditorAgentMessageRequest { clientMessageId: string; text: string; attachments?: EditorAgentAttachmentRef[]; } -export interface EditorAgentStageEvent { - conversationId: string; - stage: EditorAgentStage; +export interface EditorAgentMessageResponse { + conversation: EditorAgentConversationSummary; + deltaMessages: EditorAgentMessage[]; + errorMessage: string | null; } - -export interface EditorAgentMessageDeltaEvent { - conversationId: string; - messageId: string; - role: EditorAgentMessageRole; - kind: EditorAgentMessageKind; - textDelta: string; -} - -export interface EditorAgentToolEvent { - conversationId: string; - messageId: string; - toolCallId: string; - toolName: EditorAgentToolName; - summary?: string | null; - taskId?: string | null; - model?: string | null; - status?: EditorAgentGenerationStatus | null; - error?: string | null; -} - -export interface EditorAgentGenerationResultEvent { - conversationId: string; - messageId: string; - toolCallId: string; - toolName: EditorAgentToolName; - model: string | null; - images: EditorAgentGeneratedImage[]; -} - -export interface EditorAgentErrorEvent { - conversationId: string | null; - code: string; - message: string; - recoverable: boolean; -} - -export interface EditorAgentDoneEvent { - conversationId: string; - title: string | null; -} - -export type EditorAgentSseEvent = - | { event: 'stage'; data: EditorAgentStageEvent } - | { event: 'message_delta'; data: EditorAgentMessageDeltaEvent } - | { event: 'tool_started'; data: EditorAgentToolEvent } - | { event: 'tool_completed'; data: EditorAgentToolEvent } - | { event: 'generation_result'; data: EditorAgentGenerationResultEvent } - | { event: 'error'; data: EditorAgentErrorEvent } - | { event: 'done'; data: EditorAgentDoneEvent }; diff --git a/scripts/dev.mjs b/scripts/dev.mjs index ebb73c5f0..5bdf33f72 100644 --- a/scripts/dev.mjs +++ b/scripts/dev.mjs @@ -921,7 +921,8 @@ function readLinuxApiServerProcessSnapshot(pid) { if ( error?.code === 'ENOENT' || error?.code === 'EACCES' || - error?.code === 'EPERM' + error?.code === 'EPERM' || + error?.code === 'ESRCH' ) { return null; } diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index d1d376428..d3058ee33 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -238,6 +238,7 @@ dependencies = [ "platform-agent", "platform-audio", "platform-auth", + "platform-editor-agent", "platform-hyper3d", "platform-image", "platform-llm", @@ -4441,6 +4442,19 @@ dependencies = [ "urlencoding", ] +[[package]] +name = "platform-editor-agent" +version = "0.1.0" +dependencies = [ + "hmac", + "platform-agent", + "platform-llm", + "serde", + "serde_json", + "sha2", + "shared-contracts", +] + [[package]] name = "platform-hyper3d" version = "0.1.0" diff --git a/server-rs/Cargo.toml b/server-rs/Cargo.toml index 60814bfc5..61d585844 100644 --- a/server-rs/Cargo.toml +++ b/server-rs/Cargo.toml @@ -42,6 +42,7 @@ members = [ "crates/platform-wechat", "crates/platform-speech", "crates/platform-agent", + "crates/platform-editor-agent", "crates/pingora-gateway", "crates/server-manager-panel", "crates/shared-contracts", @@ -84,6 +85,7 @@ module-square-hole = { path = "crates/module-square-hole", default-features = fa module-story = { path = "crates/module-story", default-features = false } module-visual-novel = { path = "crates/module-visual-novel", default-features = false } platform-agent = { path = "crates/platform-agent", default-features = false } +platform-editor-agent = { path = "crates/platform-editor-agent", default-features = false } platform-auth = { path = "crates/platform-auth", default-features = false } platform-audio = { path = "crates/platform-audio", default-features = false } platform-hyper3d = { path = "crates/platform-hyper3d", default-features = false } diff --git a/server-rs/crates/api-server/Cargo.toml b/server-rs/crates/api-server/Cargo.toml index 8949fdca5..29a238b70 100644 --- a/server-rs/crates/api-server/Cargo.toml +++ b/server-rs/crates/api-server/Cargo.toml @@ -38,6 +38,7 @@ module-square-hole = { workspace = true } module-story = { workspace = true } module-visual-novel = { workspace = true } platform-agent = { workspace = true } +platform-editor-agent = { workspace = true } platform-audio = { workspace = true } platform-auth = { workspace = true } platform-hyper3d = { workspace = true } diff --git a/server-rs/crates/api-server/src/editor_agent.rs b/server-rs/crates/api-server/src/editor_agent.rs deleted file mode 100644 index 5d79a37a8..000000000 --- a/server-rs/crates/api-server/src/editor_agent.rs +++ /dev/null @@ -1,2813 +0,0 @@ -use std::{ - collections::BTreeMap, - convert::Infallible, - sync::{Arc, Mutex, OnceLock}, -}; - -use axum::{ - Json, - extract::{Extension, Path, State}, - http::StatusCode, - response::{ - IntoResponse, Response, - sse::{Event, Sse}, - }, -}; -use module_editor_agent::{ - EDITOR_AGENT_CONVERSATION_ID_PREFIX, EDITOR_AGENT_DEFAULT_CONVERSATION_TITLE, - EDITOR_AGENT_MESSAGE_ID_PREFIX, derive_conversation_title, editor_agent_messages_object_key, - validate_user_message, -}; -use platform_llm::{LlmError, LlmErrorKind, LlmMessage, LlmMessageContentPart, LlmTextRequest}; -use platform_oss::{ - LegacyAssetPrefix, OssObjectAccess, OssPutObjectRequest, OssSignedGetObjectUrlRequest, -}; -use serde::{Deserialize, Serialize}; -use serde_json::{Value, json}; -use shared_contracts::assets::{ - EditorCanvasGenerationCompletionPayload, EditorCanvasGenerationPlaceholderPayload, -}; -use shared_contracts::editor_agent::{ - CreateEditorAgentConversationRequest, EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION, - EditorAgentAttachmentRef, EditorAgentAttachmentSource, EditorAgentConversationDetail, - EditorAgentConversationListResponse, EditorAgentConversationMessagesDocument, - EditorAgentConversationResponse, EditorAgentConversationSummary, EditorAgentDoneEvent, - EditorAgentErrorEvent, EditorAgentGeneratedImage, EditorAgentGenerationRecord, - EditorAgentGenerationResultEvent, EditorAgentGenerationStatus, EditorAgentMessage, - EditorAgentMessageDeltaEvent, EditorAgentMessageKind, EditorAgentMessageRole, - EditorAgentMessageStatus, EditorAgentStage, EditorAgentStageEvent, EditorAgentToolEvent, - EditorAgentToolName, StreamEditorAgentMessageRequest, -}; -use shared_kernel::{build_prefixed_uuid_id, normalize_optional_string, normalize_required_string}; -use spacetime_client::{ - EditorAgentConversationCreateRecordInput, EditorAgentConversationDeleteRecordInput, - EditorAgentConversationRecord, EditorAgentConversationTouchRecordInput, - EditorAssetLibraryRecord, EditorAssetRecord, EditorProjectGetRecordInput, EditorProjectRecord, - EditorProjectResourceRecord, -}; - -use crate::{ - api_response::json_success_body, - auth::AuthenticatedAccessToken, - editor_project::{ - EditorGenerationCaller, EditorIconSpritesheetGenerationRequest, EditorImageEditRequest, - EditorImageGenerationRequest, current_utc_micros, edit_editor_image_for_owner, - generate_editor_icon_spritesheet_for_owner, generate_editor_image_for_owner, - map_editor_project_error, - }, - http_error::AppError, - platform_errors::map_oss_error, - request_context::RequestContext, - state::AppState, -}; - -const EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES: usize = 2 * 1024 * 1024; -const EDITOR_AGENT_MESSAGES_READ_EXPIRE_SECONDS: u64 = 60; -const EDITOR_AGENT_LLM_ATTACHMENT_URL_EXPIRE_SECONDS: u64 = 300; -const EDITOR_AGENT_LLM_MAX_HISTORY_MESSAGES: usize = 12; -const EDITOR_AGENT_LLM_PLANNING_MODEL: &str = platform_agent::CREATIVE_AGENT_GPT5_MODEL; -const EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS: u32 = 1024; -const EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS: u64 = 60_000; -const EDITOR_AGENT_TOOL_CALL_ID_PREFIX: &str = "editor-agent-tool"; -const EDITOR_AGENT_CANVAS_RESULT_GAP: f64 = 32.0; -type EditorAgentConversationLockMap = Mutex>>>; -static EDITOR_AGENT_CONVERSATION_LOCKS: OnceLock = OnceLock::new(); - -#[derive(Debug, Serialize)] -#[serde(rename_all = "camelCase")] -struct EditorAgentConversationDeleteResponse { - deleted_conversation_id: String, - conversation: EditorAgentConversationSummary, -} - -pub async fn list_editor_agent_conversations( - State(state): State, - Path(project_id): Path, - Extension(request_context): Extension, - Extension(authenticated): Extension, -) -> Result, AppError> { - let owner_user_id = authenticated.claims().user_id().to_string(); - require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; - ensure_editor_project_access(&state, project_id.as_str(), owner_user_id.as_str()).await?; - let conversations = state - .spacetime_client() - .list_editor_agent_conversations(project_id, owner_user_id) - .await - .map_err(map_editor_project_error)? - .into_iter() - .map(conversation_summary_from_record) - .collect(); - - Ok(json_success_body( - Some(&request_context), - EditorAgentConversationListResponse { conversations }, - )) -} - -pub async fn create_editor_agent_conversation( - State(state): State, - Path(project_id): Path, - Extension(request_context): Extension, - Extension(authenticated): Extension, - Json(payload): Json, -) -> Result, AppError> { - let owner_user_id = authenticated.claims().user_id().to_string(); - require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; - ensure_editor_project_access(&state, project_id.as_str(), owner_user_id.as_str()).await?; - - let conversation_id = build_prefixed_uuid_id(EDITOR_AGENT_CONVERSATION_ID_PREFIX); - let messages_object_key = editor_agent_messages_object_key(conversation_id.as_str()); - let title = normalize_optional_string(payload.title) - .unwrap_or_else(|| EDITOR_AGENT_DEFAULT_CONVERSATION_TITLE.to_string()); - let now_micros = current_utc_micros(); - let seed_record = EditorAgentConversationRecord { - conversation_id: conversation_id.clone(), - project_id: project_id.clone(), - owner_user_id: owner_user_id.clone(), - title: title.clone(), - messages_object_key: messages_object_key.clone(), - deleted: false, - created_at: now_rfc3339(), - updated_at: now_rfc3339(), - updated_at_micros: now_micros, - }; - write_messages_document( - &state, - &seed_record, - &empty_messages_document(conversation_id.as_str()), - ) - .await?; - - let conversation = state - .spacetime_client() - .create_editor_agent_conversation(EditorAgentConversationCreateRecordInput { - conversation_id, - project_id, - owner_user_id, - title, - messages_object_key, - created_at_micros: now_micros, - }) - .await - .map_err(map_editor_project_error)?; - let document = read_messages_document(&state, &conversation).await?; - - Ok(json_success_body( - Some(&request_context), - EditorAgentConversationResponse { - conversation: conversation_detail_from_record(conversation, document.messages), - }, - )) -} - -pub async fn get_editor_agent_conversation( - State(state): State, - Path(conversation_id): Path, - Extension(request_context): Extension, - Extension(authenticated): Extension, -) -> Result, AppError> { - let owner_user_id = authenticated.claims().user_id().to_string(); - require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; - let conversation = state - .spacetime_client() - .get_editor_agent_conversation(conversation_id, owner_user_id) - .await - .map_err(map_editor_project_error)?; - let document = read_messages_document(&state, &conversation).await?; - - Ok(json_success_body( - Some(&request_context), - EditorAgentConversationResponse { - conversation: conversation_detail_from_record(conversation, document.messages), - }, - )) -} - -pub async fn delete_editor_agent_conversation( - State(state): State, - Path(conversation_id): Path, - Extension(request_context): Extension, - Extension(authenticated): Extension, -) -> Result, AppError> { - let owner_user_id = authenticated.claims().user_id().to_string(); - require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; - let conversation = state - .spacetime_client() - .delete_editor_agent_conversation(EditorAgentConversationDeleteRecordInput { - conversation_id, - owner_user_id, - updated_at_micros: current_utc_micros(), - }) - .await - .map_err(map_editor_project_error)?; - - Ok(json_success_body( - Some(&request_context), - EditorAgentConversationDeleteResponse { - deleted_conversation_id: conversation.conversation_id.clone(), - conversation: conversation_summary_from_record(conversation), - }, - )) -} - -pub async fn stream_editor_agent_message( - State(state): State, - Path(conversation_id): Path, - Extension(request_context): Extension, - Extension(authenticated): Extension, - Json(payload): Json, -) -> Result { - let owner_user_id = authenticated.claims().user_id().to_string(); - require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; - let client_message_id = normalize_required_string(payload.client_message_id.as_str()) - .ok_or_else(|| editor_agent_bad_request("clientMessageId is required"))?; - let attachment_reference_ids = payload - .attachments - .iter() - .map(|attachment| attachment.reference_id.clone()) - .collect::>(); - validate_user_message(payload.text.as_str(), &attachment_reference_ids) - .map_err(|error| editor_agent_bad_request(error.to_string()))?; - - let conversation = state - .spacetime_client() - .get_editor_agent_conversation(conversation_id, owner_user_id.clone()) - .await - .map_err(map_editor_project_error)?; - let attachments = - normalize_editor_agent_attachments(&state, &conversation, payload.attachments.as_slice()) - .await?; - let conversation_lock = editor_agent_conversation_lock(conversation.conversation_id.as_str()); - let conversation_lock_guard = conversation_lock.lock_owned().await; - let mut document = read_messages_document(&state, &conversation).await?; - if document - .messages - .iter() - .any(|message| message.id == client_message_id) - { - return Err( - AppError::from_status(StatusCode::CONFLICT).with_details(json!({ - "provider": "editor-agent", - "field": "clientMessageId", - "message": "clientMessageId already exists in this conversation", - })), - ); - } - - let was_empty = document.messages.is_empty(); - let user_message = EditorAgentMessage { - id: client_message_id, - role: EditorAgentMessageRole::User, - kind: EditorAgentMessageKind::Chat, - text: payload.text.trim().to_string(), - attachments, - generations: Vec::new(), - status: EditorAgentMessageStatus::Completed, - created_at: now_rfc3339(), - }; - document.messages.push(user_message.clone()); - write_messages_document(&state, &conversation, &document).await?; - let latest_generation_reference = latest_editor_agent_generated_image_reference(&document); - - let assistant_message_id = build_prefixed_uuid_id(EDITOR_AGENT_MESSAGE_ID_PREFIX); - let stream_state = state.clone(); - let stream_conversation = conversation.clone(); - let stream_title = was_empty.then(|| derive_conversation_title(user_message.text.as_str())); - let stream_assistant_message_id = assistant_message_id.clone(); - let stream_user_message = user_message.clone(); - let stream_latest_generation_reference = latest_generation_reference.clone(); - let stream_request_context = request_context.clone(); - let stream_conversation_lock_guard = conversation_lock_guard; - - let stream = async_stream::stream! { - let _conversation_lock_guard = stream_conversation_lock_guard; - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Thinking, - }, - )); - - let mut assistant_status = EditorAgentMessageStatus::Completed; - let mut generation_records = Vec::new(); - let mut planning_failed = false; - - let turn_plan = match plan_editor_agent_turn( - &stream_state, - &stream_conversation, - &document, - &stream_user_message, - stream_latest_generation_reference.as_ref(), - ) - .await { - Ok(turn_plan) => turn_plan, - Err(error) => { - let app_error = error.into_app_error(); - let error_message = app_error.body_text(); - assistant_status = EditorAgentMessageStatus::Failed; - planning_failed = true; - EditorAgentTurnPlan { - reply_text: error_message, - tool_call: None, - } - } - }; - let assistant_text = turn_plan.reply_text; - let assistant_kind = if assistant_status == EditorAgentMessageStatus::Failed { - EditorAgentMessageKind::Error - } else { - EditorAgentMessageKind::Chat - }; - - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Responding, - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "message_delta", - EditorAgentMessageDeltaEvent { - conversation_id: stream_conversation.conversation_id.clone(), - message_id: stream_assistant_message_id.clone(), - role: EditorAgentMessageRole::Assistant, - kind: assistant_kind, - text_delta: assistant_text.clone(), - }, - )); - if planning_failed { - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Failed, - }, - )); - } - - if let Some(tool_call) = turn_plan.tool_call { - let tool_call_id = build_prefixed_uuid_id(EDITOR_AGENT_TOOL_CALL_ID_PREFIX); - let tool_summary = tool_call - .summary - .clone() - .or_else(|| Some(editor_agent_tool_default_summary(tool_call.tool_name))); - yield Ok::(editor_agent_sse_json_event_or_error( - "tool_started", - EditorAgentToolEvent { - conversation_id: stream_conversation.conversation_id.clone(), - message_id: stream_assistant_message_id.clone(), - tool_call_id: tool_call_id.clone(), - tool_name: tool_call.tool_name, - summary: tool_summary.clone(), - task_id: None, - model: tool_call.model.clone(), - status: Some(EditorAgentGenerationStatus::Generating), - error: None, - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Generating, - }, - )); - - match execute_editor_agent_tool_call( - &stream_state, - &stream_request_context, - &stream_conversation, - &stream_user_message, - &tool_call_id, - stream_latest_generation_reference.as_ref(), - tool_call, - ) - .await - { - Ok(execution) => { - generation_records.push(execution.record.clone()); - yield Ok::(editor_agent_sse_json_event_or_error( - "generation_result", - EditorAgentGenerationResultEvent { - conversation_id: stream_conversation.conversation_id.clone(), - message_id: stream_assistant_message_id.clone(), - tool_call_id: tool_call_id.clone(), - tool_name: execution.record.tool_name, - model: execution.record.model.clone(), - images: execution.record.images.clone(), - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "tool_completed", - EditorAgentToolEvent { - conversation_id: stream_conversation.conversation_id.clone(), - message_id: stream_assistant_message_id.clone(), - tool_call_id, - tool_name: execution.record.tool_name, - summary: Some(execution.summary), - task_id: execution.record.task_id.clone(), - model: execution.record.model.clone(), - status: Some(EditorAgentGenerationStatus::Completed), - error: None, - }, - )); - } - Err(error) => { - let error_code = error.error.code().to_string(); - let error_message = error.error.body_text(); - let error_model = error.model.clone(); - assistant_status = EditorAgentMessageStatus::Failed; - generation_records.push(EditorAgentGenerationRecord { - tool_call_id: tool_call_id.clone(), - tool_name: error.tool_name, - summary: tool_summary.clone(), - task_id: None, - status: EditorAgentGenerationStatus::Failed, - model: error_model.clone(), - images: Vec::new(), - error: Some(error_message.clone()), - }); - yield Ok::(editor_agent_sse_json_event_or_error( - "tool_completed", - EditorAgentToolEvent { - conversation_id: stream_conversation.conversation_id.clone(), - message_id: stream_assistant_message_id.clone(), - tool_call_id, - tool_name: error.tool_name, - summary: tool_summary, - task_id: None, - model: error_model, - status: Some(EditorAgentGenerationStatus::Failed), - error: Some(error_message.clone()), - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Failed, - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "error", - EditorAgentErrorEvent { - conversation_id: Some(stream_conversation.conversation_id.clone()), - code: error_code, - message: error_message, - recoverable: true, - }, - )); - } - } - } - - let write_result = async { - let mut next_document = read_messages_document(&stream_state, &stream_conversation).await?; - if !next_document - .messages - .iter() - .any(|message| message.id == stream_assistant_message_id) - { - next_document.messages.push(EditorAgentMessage { - id: stream_assistant_message_id.clone(), - role: EditorAgentMessageRole::Assistant, - kind: if assistant_status == EditorAgentMessageStatus::Failed { - EditorAgentMessageKind::Error - } else { - EditorAgentMessageKind::Chat - }, - text: assistant_text, - attachments: Vec::new(), - generations: generation_records, - status: assistant_status, - created_at: now_rfc3339(), - }); - } - write_messages_document(&stream_state, &stream_conversation, &next_document).await - } - .await; - match write_result { - Ok(()) => { - let mut done_title = None; - if let Some(title) = stream_title { - match stream_state.spacetime_client().touch_editor_agent_conversation( - EditorAgentConversationTouchRecordInput { - conversation_id: stream_conversation.conversation_id.clone(), - owner_user_id: stream_conversation.owner_user_id.clone(), - title: Some(title.clone()), - updated_at_micros: current_utc_micros(), - }, - ).await { - Ok(updated) => done_title = Some(updated.title), - Err(error) => { - yield Ok::(editor_agent_sse_json_event_or_error( - "error", - EditorAgentErrorEvent { - conversation_id: Some(stream_conversation.conversation_id.clone()), - code: "SPACETIME_UPDATE_FAILED".to_string(), - message: error.to_string(), - recoverable: true, - }, - )); - } - } - } else { - match stream_state.spacetime_client().touch_editor_agent_conversation( - EditorAgentConversationTouchRecordInput { - conversation_id: stream_conversation.conversation_id.clone(), - owner_user_id: stream_conversation.owner_user_id.clone(), - title: None, - updated_at_micros: current_utc_micros(), - }, - ).await { - Ok(_) => {} - Err(error) => { - yield Ok::(editor_agent_sse_json_event_or_error( - "error", - EditorAgentErrorEvent { - conversation_id: Some(stream_conversation.conversation_id.clone()), - code: "SPACETIME_UPDATE_FAILED".to_string(), - message: error.to_string(), - recoverable: true, - }, - )); - } - } - } - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Completed, - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "done", - EditorAgentDoneEvent { - conversation_id: stream_conversation.conversation_id.clone(), - title: done_title, - }, - )); - } - Err(error) => { - yield Ok::(editor_agent_sse_json_event_or_error( - "stage", - EditorAgentStageEvent { - conversation_id: stream_conversation.conversation_id.clone(), - stage: EditorAgentStage::Failed, - }, - )); - yield Ok::(editor_agent_sse_json_event_or_error( - "error", - EditorAgentErrorEvent { - conversation_id: Some(stream_conversation.conversation_id.clone()), - code: error.code().to_string(), - message: error.body_text(), - recoverable: true, - }, - )); - } - } - }; - - Ok(Sse::new(stream).into_response()) -} - -fn editor_agent_conversation_lock(conversation_id: &str) -> Arc> { - let locks = EDITOR_AGENT_CONVERSATION_LOCKS.get_or_init(|| Mutex::new(BTreeMap::new())); - let mut locks = locks - .lock() - .unwrap_or_else(|poisoned| poisoned.into_inner()); - locks - .entry(conversation_id.to_string()) - .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(()))) - .clone() -} - -async fn ensure_editor_project_access( - state: &AppState, - project_id: &str, - owner_user_id: &str, -) -> Result<(), AppError> { - state - .spacetime_client() - .get_editor_project(EditorProjectGetRecordInput { - project_id: project_id.to_string(), - owner_user_id: owner_user_id.to_string(), - }) - .await - .map(|_| ()) - .map_err(map_editor_project_error) -} - -async fn normalize_editor_agent_attachments( - state: &AppState, - conversation: &EditorAgentConversationRecord, - attachments: &[EditorAgentAttachmentRef], -) -> Result, AppError> { - if attachments.is_empty() { - return Ok(Vec::new()); - } - - let needs_canvas_resources = attachments - .iter() - .any(|attachment| attachment.source == EditorAgentAttachmentSource::CanvasResource); - let needs_library_assets = attachments - .iter() - .any(|attachment| attachment.source == EditorAgentAttachmentSource::LibraryAsset); - - let project = if needs_canvas_resources { - Some( - state - .spacetime_client() - .get_editor_project(EditorProjectGetRecordInput { - project_id: conversation.project_id.clone(), - owner_user_id: conversation.owner_user_id.clone(), - }) - .await - .map_err(map_editor_project_error)?, - ) - } else { - None - }; - let library = if needs_library_assets { - Some( - state - .spacetime_client() - .get_editor_asset_library(conversation.owner_user_id.clone(), current_utc_micros()) - .await - .map_err(map_editor_project_error)?, - ) - } else { - None - }; - - attachments - .iter() - .map(|attachment| { - normalize_editor_agent_attachment( - conversation, - project.as_ref(), - library.as_ref(), - attachment, - ) - }) - .collect() -} - -fn normalize_editor_agent_attachment( - conversation: &EditorAgentConversationRecord, - project: Option<&EditorProjectRecord>, - library: Option<&EditorAssetLibraryRecord>, - attachment: &EditorAgentAttachmentRef, -) -> Result { - let reference_id = normalize_required_string(attachment.reference_id.as_str()) - .ok_or_else(|| editor_agent_bad_request("attachment.referenceId is required"))?; - match attachment.source { - EditorAgentAttachmentSource::CanvasResource => { - let project = project.ok_or_else(|| { - editor_agent_bad_request("canvas resource attachment project context missing") - })?; - let resource = project - .resources - .iter() - .find(|resource| resource.resource_id == reference_id) - .ok_or_else(|| { - editor_agent_bad_request(format!( - "canvas resource attachment not found in current project: {reference_id}" - )) - })?; - normalize_canvas_resource_attachment(conversation, attachment, resource) - } - EditorAgentAttachmentSource::LibraryAsset => { - let library = library.ok_or_else(|| { - editor_agent_bad_request("library asset attachment context missing") - })?; - let asset = library - .assets - .iter() - .find(|asset| asset.asset_id == reference_id) - .ok_or_else(|| { - editor_agent_bad_request(format!( - "library asset attachment not found for current user: {reference_id}" - )) - })?; - normalize_library_asset_attachment(attachment, asset) - } - } -} - -fn normalize_canvas_resource_attachment( - conversation: &EditorAgentConversationRecord, - attachment: &EditorAgentAttachmentRef, - resource: &EditorProjectResourceRecord, -) -> Result { - if resource.project_id != conversation.project_id - || resource.owner_user_id != conversation.owner_user_id - { - return Err(editor_agent_bad_request( - "canvas resource attachment does not belong to this conversation project", - )); - } - validate_attachment_object_key( - attachment.object_key.as_deref(), - resource.object_key.as_deref(), - resource.resource_id.as_str(), - )?; - - Ok(EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::CanvasResource, - reference_id: resource.resource_id.clone(), - object_key: resource.object_key.clone(), - image_src: resource.image_src.clone(), - thumbnail_src: None, - label: normalize_optional_string(attachment.label.clone()), - width: Some(resource.width), - height: Some(resource.height), - }) -} - -fn normalize_library_asset_attachment( - attachment: &EditorAgentAttachmentRef, - asset: &EditorAssetRecord, -) -> Result { - validate_attachment_object_key( - attachment.object_key.as_deref(), - asset.object_key.as_deref(), - asset.asset_id.as_str(), - )?; - - Ok(EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::LibraryAsset, - reference_id: asset.asset_id.clone(), - object_key: asset.object_key.clone(), - image_src: asset.image_src.clone(), - thumbnail_src: asset.thumbnail_src.clone(), - label: normalize_optional_string(attachment.label.clone()) - .or_else(|| Some(asset.label.clone())), - width: Some(asset.width), - height: Some(asset.height), - }) -} - -fn validate_attachment_object_key( - submitted_object_key: Option<&str>, - stored_object_key: Option<&str>, - reference_id: &str, -) -> Result<(), AppError> { - let Some(submitted_object_key) = submitted_object_key.and_then(normalize_required_string) - else { - return Ok(()); - }; - let Some(stored_object_key) = stored_object_key.and_then(normalize_required_string) else { - return Err(editor_agent_bad_request(format!( - "attachment objectKey is not available for reference: {reference_id}" - ))); - }; - if submitted_object_key != stored_object_key { - return Err(editor_agent_bad_request(format!( - "attachment objectKey does not match reference: {reference_id}" - ))); - } - Ok(()) -} - -fn conversation_summary_from_record( - conversation: EditorAgentConversationRecord, -) -> EditorAgentConversationSummary { - EditorAgentConversationSummary { - conversation_id: conversation.conversation_id, - project_id: conversation.project_id, - title: conversation.title, - created_at: conversation.created_at, - updated_at: conversation.updated_at, - } -} - -fn conversation_detail_from_record( - conversation: EditorAgentConversationRecord, - messages: Vec, -) -> EditorAgentConversationDetail { - EditorAgentConversationDetail { - conversation_id: conversation.conversation_id, - project_id: conversation.project_id, - title: conversation.title, - created_at: conversation.created_at, - updated_at: conversation.updated_at, - messages, - } -} - -fn empty_messages_document(conversation_id: &str) -> EditorAgentConversationMessagesDocument { - EditorAgentConversationMessagesDocument { - version: EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION, - conversation_id: conversation_id.to_string(), - messages: Vec::new(), - } -} - -async fn read_messages_document( - state: &AppState, - conversation: &EditorAgentConversationRecord, -) -> Result { - let oss_client = state - .oss_client() - .ok_or_else(editor_agent_oss_unavailable)?; - let signed = oss_client - .sign_internal_get_object_url(OssSignedGetObjectUrlRequest { - object_key: conversation.messages_object_key.clone(), - expire_seconds: Some(EDITOR_AGENT_MESSAGES_READ_EXPIRE_SECONDS), - }) - .map_err(|error| map_oss_error(error, "aliyun-oss"))?; - let response = reqwest::Client::new() - .get(signed.signed_url.as_str()) - .send() - .await - .map_err(|error| editor_agent_oss_read_error(error.to_string()))?; - if response.status() == reqwest::StatusCode::NOT_FOUND { - return Ok(empty_messages_document( - conversation.conversation_id.as_str(), - )); - } - if !response.status().is_success() { - return Err(editor_agent_oss_read_error(format!( - "OSS returned non-success status {}", - response.status().as_u16() - ))); - } - if response - .content_length() - .is_some_and(|size| size > EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES as u64) - { - return Err(editor_agent_messages_document_too_large()); - } - let bytes = response - .bytes() - .await - .map_err(|error| editor_agent_oss_read_error(error.to_string()))?; - if bytes.is_empty() { - return Ok(empty_messages_document( - conversation.conversation_id.as_str(), - )); - } - if bytes.len() > EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES { - return Err(editor_agent_messages_document_too_large()); - } - let document: EditorAgentConversationMessagesDocument = serde_json::from_slice(&bytes) - .map_err(|error| { - editor_agent_oss_read_error(format!("message document JSON invalid: {error}")) - })?; - if document.conversation_id != conversation.conversation_id { - return Err( - AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({ - "provider": "editor-agent", - "message": "message document conversationId does not match metadata", - "conversationId": conversation.conversation_id, - "documentConversationId": document.conversation_id, - })), - ); - } - Ok(document) -} - -async fn write_messages_document( - state: &AppState, - conversation: &EditorAgentConversationRecord, - document: &EditorAgentConversationMessagesDocument, -) -> Result<(), AppError> { - if document.conversation_id != conversation.conversation_id { - return Err( - AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ - "provider": "editor-agent", - "message": "message document conversationId does not match metadata", - })), - ); - } - let body = serde_json::to_vec(document).map_err(|error| { - AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ - "provider": "editor-agent", - "message": format!("failed to serialize message document: {error}"), - })) - })?; - if body.len() > EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES { - return Err(editor_agent_messages_document_too_large()); - } - - let oss_client = state - .oss_client() - .ok_or_else(editor_agent_oss_unavailable)?; - let put_result = oss_client - .put_object( - &reqwest::Client::new(), - OssPutObjectRequest { - prefix: LegacyAssetPrefix::EditorAgent, - path_segments: Vec::new(), - file_name: format!("{}.json", conversation.conversation_id), - content_type: Some("application/json; charset=utf-8".to_string()), - access: OssObjectAccess::Private, - metadata: BTreeMap::from([ - ( - "conversation-id".to_string(), - conversation.conversation_id.clone(), - ), - ("project-id".to_string(), conversation.project_id.clone()), - ( - "owner-user-id".to_string(), - conversation.owner_user_id.clone(), - ), - ]), - body, - }, - ) - .await - .map_err(|error| map_oss_error(error, "aliyun-oss"))?; - if put_result.object_key != conversation.messages_object_key { - return Err( - AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ - "provider": "editor-agent", - "message": "OSS object key mismatch while writing message document", - "expectedObjectKey": conversation.messages_object_key, - "actualObjectKey": put_result.object_key, - })), - ); - } - Ok(()) -} - -#[derive(Clone, Debug, PartialEq)] -struct EditorAgentTurnPlan { - reply_text: String, - tool_call: Option, -} - -#[derive(Clone, Debug, PartialEq)] -struct EditorAgentToolCallPlan { - tool_name: EditorAgentToolName, - prompt: String, - summary: Option, - model: Option, - aspect_ratio: Option, - image_size: Option, - icon_descriptions: Vec, -} - -#[derive(Debug, Deserialize)] -#[serde(rename_all = "camelCase")] -struct EditorAgentRawTurnPlan { - reply_text: Option, - tool_call: Option, -} - -#[derive(Debug, Deserialize)] -#[serde(rename_all = "camelCase")] -struct EditorAgentRawToolCallPlan { - tool_name: EditorAgentToolName, - prompt: Option, - summary: Option, - model: Option, - aspect_ratio: Option, - image_size: Option, - icon_descriptions: Option>, -} - -#[derive(Debug)] -struct EditorAgentToolExecution { - record: EditorAgentGenerationRecord, - summary: String, -} - -#[derive(Clone, Debug)] -struct EditorAgentGeneratedImageReference { - source_message_id: String, - tool_call_id: String, - tool_name: EditorAgentToolName, - summary: Option, - model: Option, - resource_id: Option, - object_key: Option, - asset_object_id: Option, - image_src: String, - thumbnail_src: Option, - width: Option, - height: Option, -} - -#[derive(Debug)] -struct EditorAgentToolExecutionError { - tool_name: EditorAgentToolName, - model: Option, - error: AppError, -} - -#[derive(Debug)] -enum EditorAgentPlanningError { - LlmUnavailable, - LlmRequestFailed(LlmError), - InvalidLlmResponse, -} - -impl EditorAgentPlanningError { - fn into_app_error(self) -> AppError { - match self { - Self::LlmUnavailable => AppError::from_status(StatusCode::SERVICE_UNAVAILABLE) - .with_details(json!({ - "provider": "editor-agent-llm", - "message": "画布 Agent 的 LLM 未配置，无法处理这句话。", - })), - Self::LlmRequestFailed(error) => { - let status = match error.kind() { - LlmErrorKind::Timeout => StatusCode::GATEWAY_TIMEOUT, - LlmErrorKind::InvalidConfig | LlmErrorKind::InvalidRequest => { - StatusCode::SERVICE_UNAVAILABLE - } - LlmErrorKind::Connectivity - | LlmErrorKind::Upstream - | LlmErrorKind::StreamUnavailable - | LlmErrorKind::EmptyResponse - | LlmErrorKind::Transport - | LlmErrorKind::Deserialize => StatusCode::BAD_GATEWAY, - }; - AppError::from_status(status).with_details(json!({ - "provider": "editor-agent-llm", - "message": format!("画布 Agent 调用 LLM 失败：{error}"), - })) - } - Self::InvalidLlmResponse => AppError::from_status(StatusCode::BAD_GATEWAY) - .with_details(json!({ - "provider": "editor-agent-llm", - "message": "画布 Agent 的 LLM 返回格式错误，未能解析回复。", - })), - } - } -} - -async fn plan_editor_agent_turn( - state: &AppState, - conversation: &EditorAgentConversationRecord, - document: &EditorAgentConversationMessagesDocument, - user_message: &EditorAgentMessage, - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, -) -> Result { - let Some(llm_client) = state.creative_agent_gpt5_client() else { - return Err(EditorAgentPlanningError::LlmUnavailable); - }; - - let request = build_editor_agent_llm_request( - state, - conversation, - document, - user_message, - latest_generation_reference, - ) - .await; - let response = llm_client - .request_text(request) - .await - .map_err(EditorAgentPlanningError::LlmRequestFailed)?; - parse_editor_agent_turn_plan(response.content.as_str()) - .map(|plan| { - apply_previous_generation_edit_default(plan, user_message, latest_generation_reference) - }) - .ok_or(EditorAgentPlanningError::InvalidLlmResponse) -} - -async fn build_editor_agent_llm_request( - state: &AppState, - conversation: &EditorAgentConversationRecord, - document: &EditorAgentConversationMessagesDocument, - user_message: &EditorAgentMessage, - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, -) -> LlmTextRequest { - let mut user_parts = vec![LlmMessageContentPart::InputText { - text: build_editor_agent_llm_user_prompt( - conversation, - document, - user_message, - latest_generation_reference, - ), - }]; - for attachment in &user_message.attachments { - if let Some(image_url) = sign_editor_agent_attachment_image_url(state, attachment) { - user_parts.push(LlmMessageContentPart::InputImage { image_url }); - } - } - if user_message.attachments.is_empty() - && editor_agent_text_mentions_previous_generation(user_message.text.as_str()) - { - if let Some(reference) = latest_generation_reference { - if let Some(image_url) = sign_editor_agent_generated_image_url(state, reference) { - user_parts.push(LlmMessageContentPart::InputImage { image_url }); - } - } - } - - LlmTextRequest::new(vec![ - LlmMessage::system(editor_agent_llm_system_prompt()), - LlmMessage::user_multimodal(user_parts), - ]) - .with_model(EDITOR_AGENT_LLM_PLANNING_MODEL) - .with_max_tokens(EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS) - .with_request_timeout_ms(EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS) -} - -fn editor_agent_llm_system_prompt() -> String { - [ - "你是 Genarrative 图片画布 Agent，只负责帮助用户理解、规划和触发画布生成工具。", - "必须只输出一个 JSON 对象，不要输出 Markdown、解释或额外文本。", - "JSON 结构：{\"replyText\":\"给用户看的中文回复\",\"toolCall\":null 或 {\"toolName\":\"generate_image|edit_image|generate_character|generate_icon_spritesheet|generate_ui_design\",\"prompt\":\"生成或修改提示词\",\"summary\":\"短动作摘要\",\"model\":null,\"aspectRatio\":null,\"imageSize\":\"1K\",\"iconDescriptions\":[\"图标描述\"]}}。", - "只有用户明确要求生成、重绘、修改、画 UI、出角色或图标时才给 toolCall；普通咨询、评价、解释时 toolCall 为 null。", - "工具说明：", - "generate_image：从文字生成一张全新图片，适合新场景、新物体、新插画、新背景、规范图/视觉规范图/风格规范图/素材规范展板；不要用于修改上一张图或附件图。prompt 必须写完整画面、主体、风格、构图和背景。", - "edit_image：修改已有图片，适合换衣服、改颜色、替换背景、局部重绘、保持主体/构图/姿势不变的编辑请求；必须使用 latestAttachments 或 latestGeneratedImage 作为源图。", - "generate_character：生成新的角色形象、人物立绘或角色设定图；如果用户是在改上一张角色图的服装、颜色、表情、姿势或背景，应改用 edit_image。", - "generate_icon_spritesheet：生成一组图标素材或图标图集，适合用户明确要多个 icon / 图标 / spritesheet；必须有 latestAttachments 作为图标规范或风格参考，并填写 iconDescriptions。", - "generate_ui_design：生成一张完整 UI 设计图或界面稿，适合 HUD、弹窗、面板、按钮组合和整页界面；不要用于提取图标、拆素材或修改上一张图。", - "规范图要求：用户要求生成规范图/视觉规范图/风格规范图/素材规范展板时，当前使用 generate_image；prompt 必须明确这是规范展板，并写入统一视角、线条粗细、描边、填充风格、材质、阴影、圆角、状态层级、色卡/色号、尺寸标注和排版层级。", - "角色规范图/角色美术视觉规范设定图属于规范展板时使用 generate_image，prompt 要包含头身比例、标准立绘、动作帧样例、服饰配饰分层和专属角色色卡；只有用户要生成单个全新角色形象/立绘/普通角色设定图时才使用 generate_character。", - "图标规范图/图标视觉规范展板属于规范展板时使用 generate_image；只有用户明确要生成多个图标成品、图标素材图集或 spritesheet，并提供图标规范/风格参考附件时才使用 generate_icon_spritesheet。", - "UI 规范图/组件规范展板如果是规范展板而非完整可用界面稿，也使用 generate_image；完整界面稿/HUD/弹窗/面板才使用 generate_ui_design。", - "工具选择优先级：明确修改/指代已有图 => edit_image；规范图/视觉规范图/风格规范图/素材规范展板 => generate_image；明确新角色且不是规范展板或修改已有图 => generate_character；多个图标成品/图标图集 => generate_icon_spritesheet；完整界面稿 => generate_ui_design；其他全新图片 => generate_image。", - "edit_image 必须依赖 latestAttachments 或 latestGeneratedImage；generate_icon_spritesheet 必须依赖 latestAttachments。", - "当 latestGeneratedImage 存在，且用户说“这张/刚才那个/上一张/把衣服换成/改成/换成/修改上一张图”等指代或修改上一轮结果的话，必须选择 edit_image，prompt 保留用户修改要求，不要要求用户重新选择参考图，也不要降级成 generate_image。", - "没有 latestAttachments 且没有 latestGeneratedImage 时，不要调用 edit_image 或 generate_icon_spritesheet，应在 replyText 中提示先选择参考图。", - "generate_ui_design 表示生成一张 UI 设计图，不是提取 UI 素材。", - "对话回复要简短，不能承诺免费生成；生成工具由后端按模型定价扣泥点。", - ] - .join("\n") -} - -fn build_editor_agent_llm_user_prompt( - conversation: &EditorAgentConversationRecord, - document: &EditorAgentConversationMessagesDocument, - user_message: &EditorAgentMessage, - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, -) -> String { - let history = document - .messages - .iter() - .rev() - .take(EDITOR_AGENT_LLM_MAX_HISTORY_MESSAGES) - .collect::>() - .into_iter() - .rev() - .map(|message| { - json!({ - "role": message.role, - "kind": message.kind, - "text": message.text, - "attachmentCount": message.attachments.len(), - "generationCount": message.generations.len(), - "generations": message - .generations - .iter() - .map(editor_agent_generation_context_json) - .collect::>(), - }) - }) - .collect::>(); - let attachments = user_message - .attachments - .iter() - .map(|attachment| { - json!({ - "source": attachment.source, - "referenceId": attachment.reference_id, - "label": attachment.label, - "width": attachment.width, - "height": attachment.height, - "hasObjectKey": attachment.object_key.as_ref().is_some_and(|value| !value.trim().is_empty()), - }) - }) - .collect::>(); - - json!({ - "conversationId": conversation.conversation_id, - "projectId": conversation.project_id, - "latestUserText": user_message.text, - "latestAttachments": attachments, - "latestGeneratedImage": latest_generation_reference - .map(editor_agent_generated_image_reference_context_json), - "recentMessages": history, - }) - .to_string() -} - -fn editor_agent_generation_context_json(generation: &EditorAgentGenerationRecord) -> Value { - json!({ - "toolCallId": generation.tool_call_id, - "toolName": generation.tool_name, - "summary": generation.summary, - "status": generation.status, - "model": generation.model, - "images": generation.images.iter().take(4).map(|image| json!({ - "resourceId": image.resource_id, - "objectKey": image.object_key, - "assetObjectId": image.asset_object_id, - "imageSrc": image.image_src, - "thumbnailSrc": image.thumbnail_src, - "width": image.width, - "height": image.height, - "hasObjectKey": image.object_key.as_ref().is_some_and(|value| !value.trim().is_empty()), - })).collect::>(), - }) -} - -fn editor_agent_generated_image_reference_context_json( - reference: &EditorAgentGeneratedImageReference, -) -> Value { - json!({ - "sourceMessageId": reference.source_message_id, - "toolCallId": reference.tool_call_id, - "toolName": reference.tool_name, - "summary": reference.summary, - "model": reference.model, - "resourceId": reference.resource_id, - "objectKey": reference.object_key, - "assetObjectId": reference.asset_object_id, - "imageSrc": reference.image_src, - "thumbnailSrc": reference.thumbnail_src, - "width": reference.width, - "height": reference.height, - "hasObjectKey": reference.object_key.as_ref().is_some_and(|value| !value.trim().is_empty()), - }) -} - -fn sign_editor_agent_attachment_image_url( - state: &AppState, - attachment: &EditorAgentAttachmentRef, -) -> Option { - let object_key = attachment - .object_key - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty())?; - state - .oss_client()? - .sign_get_object_url(OssSignedGetObjectUrlRequest { - object_key: object_key.trim_start_matches('/').to_string(), - expire_seconds: Some(EDITOR_AGENT_LLM_ATTACHMENT_URL_EXPIRE_SECONDS), - }) - .ok() - .map(|signed| signed.signed_url) -} - -fn sign_editor_agent_generated_image_url( - state: &AppState, - reference: &EditorAgentGeneratedImageReference, -) -> Option { - let object_key = reference - .object_key - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty())?; - state - .oss_client()? - .sign_get_object_url(OssSignedGetObjectUrlRequest { - object_key: object_key.trim_start_matches('/').to_string(), - expire_seconds: Some(EDITOR_AGENT_LLM_ATTACHMENT_URL_EXPIRE_SECONDS), - }) - .ok() - .map(|signed| signed.signed_url) -} - -fn latest_editor_agent_generated_image_reference( - document: &EditorAgentConversationMessagesDocument, -) -> Option { - document.messages.iter().rev().find_map(|message| { - message - .generations - .iter() - .rev() - .filter(|generation| generation.status == EditorAgentGenerationStatus::Completed) - .find_map(|generation| { - generation.images.iter().rev().find_map(|image| { - let has_reference_source = image - .object_key - .as_deref() - .is_some_and(|value| !value.trim().is_empty()) - || image - .resource_id - .as_deref() - .is_some_and(|value| !value.trim().is_empty()) - || !image.image_src.trim().is_empty(); - has_reference_source.then(|| EditorAgentGeneratedImageReference { - source_message_id: message.id.clone(), - tool_call_id: generation.tool_call_id.clone(), - tool_name: generation.tool_name, - summary: generation.summary.clone(), - model: generation.model.clone(), - resource_id: image.resource_id.clone(), - object_key: image.object_key.clone(), - asset_object_id: image.asset_object_id.clone(), - image_src: image.image_src.clone(), - thumbnail_src: image.thumbnail_src.clone(), - width: image.width, - height: image.height, - }) - }) - }) - }) -} - -fn editor_agent_generated_reference_source( - reference: &EditorAgentGeneratedImageReference, -) -> Option { - reference - .object_key - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(|value| value.trim_start_matches('/').to_string()) - .or_else(|| { - reference - .resource_id - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(ToOwned::to_owned) - }) - .or_else(|| { - reference - .image_src - .trim() - .strip_prefix('/') - .map(str::to_string) - .filter(|value| !value.is_empty()) - }) -} - -fn editor_agent_text_mentions_previous_generation(text: &str) -> bool { - let normalized = text.trim().to_lowercase(); - contains_any( - normalized.as_str(), - &[ - "这张", - "这个图", - "这幅", - "这张图", - "刚才", - "刚刚", - "刚才那个", - "刚才那张", - "上一张", - "上张", - "上一幅", - "上一个", - "前一张", - "上次生成", - "刚生成", - "latest image", - "previous image", - "last image", - ], - ) -} - -fn editor_agent_text_requests_previous_generation_edit(text: &str) -> bool { - let normalized = text.trim().to_lowercase(); - if editor_agent_text_mentions_previous_generation(normalized.as_str()) - && contains_any( - normalized.as_str(), - &[ - "修改", "改成", "换成", "替换", "重绘", "编辑", "优化", "去掉", "加上", "增加", - "变成", "调整", "换一", "换掉", "remove", "change", "edit", "redraw", - ], - ) - { - return true; - } - - let has_visual_target = contains_any( - normalized.as_str(), - &[ - "衣服", "服装", "发型", "头发", "背景", "颜色", "色调", "表情", "姿势", "主体", "风格", - "脸", "眼睛", "帽子", "鞋", "裙子", "外套", - ], - ); - let has_edit_verb = contains_any( - normalized.as_str(), - &[ - "换", "改", "变", "加", "去", "删", "替换", "调整", "change", "edit", "remove", - ], - ); - has_visual_target && has_edit_verb -} - -fn apply_previous_generation_edit_default( - mut plan: EditorAgentTurnPlan, - user_message: &EditorAgentMessage, - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, -) -> EditorAgentTurnPlan { - if latest_generation_reference.is_none() - || !user_message.attachments.is_empty() - || !editor_agent_text_requests_previous_generation_edit(user_message.text.as_str()) - { - return plan; - } - - let should_force_edit = plan - .tool_call - .as_ref() - .is_none_or(|tool_call| tool_call.tool_name != EditorAgentToolName::EditImage); - if !should_force_edit { - return plan; - } - - let original_tool = plan.tool_call.take(); - let prompt = original_tool - .as_ref() - .map(|tool_call| tool_call.prompt.clone()) - .unwrap_or_else(|| user_message.text.trim().to_string()); - let summary = original_tool - .as_ref() - .and_then(|tool_call| tool_call.summary.clone()) - .or_else(|| Some("修改上一张图".to_string())); - let model = original_tool - .as_ref() - .and_then(|tool_call| tool_call.model.clone()); - let image_size = original_tool - .as_ref() - .and_then(|tool_call| tool_call.image_size.clone()) - .or_else(|| Some("1K".to_string())); - - plan.reply_text = "我会基于上一张生成图按你的要求修改。".to_string(); - plan.tool_call = Some(EditorAgentToolCallPlan { - tool_name: EditorAgentToolName::EditImage, - prompt, - summary, - model, - aspect_ratio: None, - image_size, - icon_descriptions: Vec::new(), - }); - plan -} - -fn parse_editor_agent_turn_plan(raw_text: &str) -> Option { - let json_text = extract_editor_agent_json_object(raw_text)?; - let raw: EditorAgentRawTurnPlan = serde_json::from_str(json_text).ok()?; - let tool_call = match raw.tool_call { - Some(tool) => { - let prompt = normalize_optional_string(tool.prompt)?; - Some(EditorAgentToolCallPlan { - tool_name: tool.tool_name, - prompt, - summary: normalize_optional_string(tool.summary), - model: normalize_optional_string(tool.model), - aspect_ratio: normalize_optional_string(tool.aspect_ratio), - image_size: normalize_optional_string(tool.image_size), - icon_descriptions: tool - .icon_descriptions - .unwrap_or_default() - .into_iter() - .filter_map(|description| normalize_optional_string(Some(description))) - .collect(), - }) - } - None => None, - }; - let reply_text = normalize_editor_agent_reply_text(raw.reply_text, tool_call.as_ref())?; - - Some(EditorAgentTurnPlan { - reply_text, - tool_call, - }) -} - -fn extract_editor_agent_json_object(raw_text: &str) -> Option<&str> { - let trimmed = raw_text.trim(); - let without_fence = if let Some(rest) = trimmed.strip_prefix("```json") { - rest.strip_suffix("```").unwrap_or(rest).trim() - } else if let Some(rest) = trimmed.strip_prefix("```") { - rest.strip_suffix("```").unwrap_or(rest).trim() - } else { - trimmed - }; - let start = without_fence.find('{')?; - let end = without_fence.rfind('}')?; - (start <= end).then_some(&without_fence[start..=end]) -} - -fn normalize_editor_agent_reply_text( - reply_text: Option, - tool_call: Option<&EditorAgentToolCallPlan>, -) -> Option { - let reply_text = normalize_optional_string(reply_text)?; - if !editor_agent_reply_text_looks_structural(reply_text.as_str()) { - return Some(reply_text); - } - tool_call.map(editor_agent_tool_call_reply_text) -} - -fn editor_agent_reply_text_looks_structural(text: &str) -> bool { - let trimmed = text.trim(); - if trimmed.is_empty() { - return true; - } - let lower = trimmed.to_ascii_lowercase(); - if matches!(lower.as_str(), "null" | "undefined") { - return true; - } - trimmed.len() <= 2 - && trimmed - .chars() - .all(|character| matches!(character, '{' | '}' | '[' | ']' | '"' | ':' | ',')) -} - -fn editor_agent_tool_call_reply_text(tool_call: &EditorAgentToolCallPlan) -> String { - match tool_call.tool_name { - EditorAgentToolName::GenerateImage => "我来生成图片。".to_string(), - EditorAgentToolName::EditImage => "我来修改图片。".to_string(), - EditorAgentToolName::GenerateCharacter => "我来生成角色形象。".to_string(), - EditorAgentToolName::GenerateIconSpritesheet => "我来生成图标素材。".to_string(), - EditorAgentToolName::GenerateUiDesign => "我来生成 UI 设计图。".to_string(), - } -} - -#[cfg(test)] -fn heuristic_editor_agent_turn_plan( - text: &str, - attachments: &[EditorAgentAttachmentRef], -) -> EditorAgentTurnPlan { - let trimmed = text.trim(); - let normalized = trimmed.to_lowercase(); - let wants_generation = contains_any( - normalized.as_str(), - &[ - "生成", "出图", "绘制", "设计", "画一", "画个", "画张", "generate", "create", - ], - ); - let wants_edit = contains_any( - normalized.as_str(), - &[ - "修改", "改成", "重绘", "换成", "编辑", "优化", "edit", "redraw", - ], - ); - let wants_character = contains_any( - normalized.as_str(), - &["角色", "人物", "立绘", "形象", "character"], - ); - let wants_icons = contains_any( - normalized.as_str(), - &["图标", "icon", "spritesheet", "素材图集"], - ); - let wants_ui = contains_any( - normalized.as_str(), - &["ui", "界面", "hud", "按钮", "面板", "弹窗"], - ); - - let tool_name = if wants_icons && !attachments.is_empty() { - Some(EditorAgentToolName::GenerateIconSpritesheet) - } else if wants_edit && !attachments.is_empty() { - Some(EditorAgentToolName::EditImage) - } else if wants_character && wants_generation { - Some(EditorAgentToolName::GenerateCharacter) - } else if wants_ui && wants_generation { - Some(EditorAgentToolName::GenerateUiDesign) - } else if wants_generation { - Some(EditorAgentToolName::GenerateImage) - } else { - None - }; - - let reply_text = match tool_name { - Some(EditorAgentToolName::EditImage) => "我来按你的要求修改这张图。".to_string(), - Some(EditorAgentToolName::GenerateCharacter) => "我来生成角色形象。".to_string(), - Some(EditorAgentToolName::GenerateIconSpritesheet) => "我来生成图标素材图集。".to_string(), - Some(EditorAgentToolName::GenerateUiDesign) => "我来生成 UI 设计图。".to_string(), - Some(EditorAgentToolName::GenerateImage) => "我来生成图片。".to_string(), - None => build_echo_assistant_text(trimmed, attachments.len()), - }; - let tool_call = tool_name.map(|tool_name| EditorAgentToolCallPlan { - tool_name, - prompt: if trimmed.is_empty() { - "根据附件生成适合当前画布的图片".to_string() - } else { - trimmed.to_string() - }, - summary: Some(editor_agent_tool_default_summary(tool_name)), - model: None, - aspect_ratio: None, - image_size: Some("1K".to_string()), - icon_descriptions: derive_icon_descriptions(trimmed), - }); - - EditorAgentTurnPlan { - reply_text, - tool_call, - } -} - -fn contains_any(text: &str, needles: &[&str]) -> bool { - needles.iter().any(|needle| text.contains(needle)) -} - -fn derive_icon_descriptions(prompt: &str) -> Vec { - let normalized = prompt.trim(); - if normalized.is_empty() { - return vec![ - "主按钮图标".to_string(), - "奖励图标".to_string(), - "关闭图标".to_string(), - ]; - } - normalized - .split(['、', ',', '，', '\n', ';', '；']) - .map(str::trim) - .filter(|item| !item.is_empty()) - .take(8) - .map(ToOwned::to_owned) - .collect::>() - .into_iter() - .filter(|item| item.chars().count() <= 80) - .collect::>() - .pipe_non_empty_or_else(|| vec![normalized.to_string()]) -} - -trait VecPipeNonEmpty { - fn pipe_non_empty_or_else(self, fallback: F) -> Self - where - F: FnOnce() -> Self; -} - -impl VecPipeNonEmpty for Vec { - fn pipe_non_empty_or_else(self, fallback: F) -> Self - where - F: FnOnce() -> Self, - { - if self.is_empty() { fallback() } else { self } - } -} - -async fn execute_editor_agent_tool_call( - state: &AppState, - request_context: &RequestContext, - conversation: &EditorAgentConversationRecord, - user_message: &EditorAgentMessage, - tool_call_id: &str, - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, - tool_call: EditorAgentToolCallPlan, -) -> Result { - let project = state - .spacetime_client() - .get_editor_project(EditorProjectGetRecordInput { - project_id: conversation.project_id.clone(), - owner_user_id: conversation.owner_user_id.clone(), - }) - .await - .map_err(map_editor_project_error) - .map_err(|error| EditorAgentToolExecutionError { - tool_name: tool_call.tool_name, - model: tool_call.model.clone(), - error, - })?; - let completion = - build_editor_agent_canvas_completion(&project, tool_call.tool_name, &tool_call.prompt); - let caller = EditorGenerationCaller { - owner_user_id: conversation.owner_user_id.clone(), - audit_subject_user_id: Some(conversation.owner_user_id.clone()), - audit_project_id: Some(conversation.project_id.clone()), - phase_reporter: None, - }; - let tool_request_context = editor_agent_tool_request_context(request_context, tool_call_id); - let attachment_sources = editor_agent_attachment_sources(user_message.attachments.as_slice()); - let previous_generation_source = (tool_call.tool_name == EditorAgentToolName::EditImage - && user_message.attachments.is_empty()) - .then(|| latest_generation_reference.and_then(editor_agent_generated_reference_source)) - .flatten(); - let tool_reference_sources = previous_generation_source - .clone() - .map(|source| vec![source]) - .unwrap_or_else(|| attachment_sources.clone()); - let generation_reference_context = editor_agent_tool_generation_reference_context( - user_message.attachments.as_slice(), - latest_generation_reference, - previous_generation_source.as_deref(), - ); - let generation_inputs = Some(json!({ - "source": "editor-agent", - "conversationId": conversation.conversation_id, - "messageId": user_message.id, - "toolCallId": tool_call_id, - "fields": [{ "title": "用户指令", "value": tool_call.prompt }], - "references": generation_reference_context, - })); - - let tool_name = tool_call.tool_name; - let tool_model = tool_call.model.clone(); - let response = match tool_name { - EditorAgentToolName::GenerateImage - | EditorAgentToolName::GenerateCharacter - | EditorAgentToolName::GenerateUiDesign => { - let kind = match tool_name { - EditorAgentToolName::GenerateCharacter => Some("character".to_string()), - EditorAgentToolName::GenerateUiDesign => Some("ui-design".to_string()), - _ => None, - }; - generate_editor_image_for_owner( - state, - &tool_request_context, - caller, - EditorImageGenerationRequest { - prompt: tool_call.prompt.clone(), - size: None, - kind, - model: tool_call.model.clone(), - screen_color: Some("auto".to_string()), - seg_model: Some("birefnet".to_string()), - aspect_ratio: tool_call.aspect_ratio.clone(), - image_size: tool_call.image_size.clone(), - reference_image_srcs: Some(tool_reference_sources.clone()), - project_id: Some(conversation.project_id.clone()), - asset_kind: Some(editor_agent_tool_asset_kind(tool_name).to_string()), - generation_inputs, - asset_folder_id: Some(editor_agent_default_asset_folder_id()), - asset_label: tool_call.summary.clone(), - source_resource_id: editor_agent_tool_source_resource_id( - user_message.attachments.as_slice(), - latest_generation_reference, - previous_generation_source.as_deref(), - ), - canvas_completion: Some(completion), - }, - ) - .await - } - EditorAgentToolName::EditImage => { - let (source_image_src, reference_image_srcs) = split_editor_agent_edit_sources( - tool_reference_sources.clone(), - ) - .ok_or_else(|| EditorAgentToolExecutionError { - tool_name, - model: tool_model.clone(), - error: editor_agent_bad_request("修改图片需要先选择一张画布或素材库图片附件"), - })?; - edit_editor_image_for_owner( - state, - &tool_request_context, - caller, - EditorImageEditRequest { - prompt: tool_call.prompt.clone(), - source_image_src, - size: None, - model: tool_call.model.clone(), - aspect_ratio: None, - image_size: None, - reference_image_srcs: Some(reference_image_srcs), - project_id: Some(conversation.project_id.clone()), - asset_kind: Some(editor_agent_tool_asset_kind(tool_name).to_string()), - generation_inputs, - asset_folder_id: Some(editor_agent_default_asset_folder_id()), - asset_label: tool_call.summary.clone(), - source_resource_id: editor_agent_tool_source_resource_id( - user_message.attachments.as_slice(), - latest_generation_reference, - previous_generation_source.as_deref(), - ), - target_layer_id: None, - canvas_completion: Some(completion), - }, - ) - .await - } - EditorAgentToolName::GenerateIconSpritesheet => { - let (reference_image_src, extra_references) = - split_editor_agent_edit_sources(attachment_sources).ok_or_else(|| { - EditorAgentToolExecutionError { - tool_name, - model: tool_model.clone(), - error: editor_agent_bad_request("生成图标素材需要先选择一张图标规范参考图"), - } - })?; - let icon_descriptions = if tool_call.icon_descriptions.is_empty() { - derive_icon_descriptions(tool_call.prompt.as_str()) - } else { - tool_call.icon_descriptions.clone() - }; - generate_editor_icon_spritesheet_for_owner( - state, - &tool_request_context, - caller, - EditorIconSpritesheetGenerationRequest { - reference_image_src, - reference_image_srcs: Some(extra_references), - icon_descriptions, - model: tool_call.model.clone(), - screen_color: Some("auto".to_string()), - seg_model: Some("birefnet".to_string()), - aspect_ratio: tool_call.aspect_ratio.clone(), - image_size: tool_call.image_size.clone(), - project_id: Some(conversation.project_id.clone()), - generation_inputs, - asset_folder_id: Some(editor_agent_default_asset_folder_id()), - asset_label: None, - canvas_completion: Some(completion), - }, - ) - .await - } - } - .map_err(|error| EditorAgentToolExecutionError { - tool_name, - model: tool_model.clone(), - error, - })?; - - Ok(build_editor_agent_tool_execution( - tool_call_id, - tool_name, - tool_call.summary, - response.0, - )) -} - -fn editor_agent_tool_request_context(base: &RequestContext, tool_call_id: &str) -> RequestContext { - RequestContext::new( - format!("{}:{tool_call_id}", base.request_id()), - format!("{} editor-agent-tool", base.operation()), - std::time::Duration::ZERO, - false, - ) -} - -fn build_editor_agent_tool_execution( - tool_call_id: &str, - tool_name: EditorAgentToolName, - summary: Option, - response_body: Value, -) -> EditorAgentToolExecution { - let data = response_body - .get("data") - .filter(|_| response_body.get("ok").and_then(Value::as_bool) == Some(true)) - .unwrap_or(&response_body); - let images = editor_agent_generated_images_from_response(tool_name, data); - let task_id = data - .get("taskId") - .and_then(Value::as_str) - .map(str::to_string) - .or_else(|| { - images - .iter() - .find_map(|image| image.resource_id.as_ref().map(|_| ())) - .and_then(|_| { - data.get("taskId") - .and_then(Value::as_str) - .map(str::to_string) - }) - }); - let model = data - .get("model") - .and_then(Value::as_str) - .map(str::to_string); - let resolved_summary = summary.unwrap_or_else(|| editor_agent_tool_default_summary(tool_name)); - - EditorAgentToolExecution { - record: EditorAgentGenerationRecord { - tool_call_id: tool_call_id.to_string(), - tool_name, - summary: Some(resolved_summary.clone()), - task_id, - status: EditorAgentGenerationStatus::Completed, - model, - images, - error: None, - }, - summary: resolved_summary, - } -} - -fn editor_agent_generated_images_from_response( - tool_name: EditorAgentToolName, - data: &Value, -) -> Vec { - if tool_name == EditorAgentToolName::GenerateIconSpritesheet { - let mut images = Vec::new(); - if let Some(resource) = data.get("spritesheetResource") { - images.push(editor_agent_generated_image_from_resource( - resource, - data.get("spritesheetImageSrc").and_then(Value::as_str), - data.get("spritesheetWidth").and_then(Value::as_u64), - data.get("spritesheetHeight").and_then(Value::as_u64), - )); - } - if let Some(icon_images) = data.get("iconImageSrcs").and_then(Value::as_array) { - for icon in icon_images { - if let Some(resource) = icon.get("resource") { - images.push(editor_agent_generated_image_from_resource( - resource, - icon.get("imageSrc").and_then(Value::as_str), - icon.get("width").and_then(Value::as_u64), - icon.get("height").and_then(Value::as_u64), - )); - } - } - } - return images; - } - - data.get("resource") - .map(|resource| { - vec![editor_agent_generated_image_from_resource( - resource, - data.get("imageSrc").and_then(Value::as_str), - data.get("width").and_then(Value::as_u64), - data.get("height").and_then(Value::as_u64), - )] - }) - .unwrap_or_else(|| { - data.get("imageSrc") - .and_then(Value::as_str) - .map(|image_src| { - vec![EditorAgentGeneratedImage { - resource_id: None, - object_key: None, - asset_object_id: None, - image_src: image_src.to_string(), - thumbnail_src: None, - width: data - .get("width") - .and_then(Value::as_u64) - .and_then(|value| u32::try_from(value).ok()), - height: data - .get("height") - .and_then(Value::as_u64) - .and_then(|value| u32::try_from(value).ok()), - }] - }) - .unwrap_or_default() - }) -} - -fn editor_agent_generated_image_from_resource( - resource: &Value, - fallback_image_src: Option<&str>, - fallback_width: Option, - fallback_height: Option, -) -> EditorAgentGeneratedImage { - EditorAgentGeneratedImage { - resource_id: resource - .get("resourceId") - .and_then(Value::as_str) - .map(str::to_string), - object_key: resource - .get("objectKey") - .and_then(Value::as_str) - .map(str::to_string), - asset_object_id: resource - .get("assetObjectId") - .and_then(Value::as_str) - .map(str::to_string), - image_src: resource - .get("imageSrc") - .and_then(Value::as_str) - .or(fallback_image_src) - .unwrap_or("") - .to_string(), - thumbnail_src: resource - .get("thumbnailSrc") - .and_then(Value::as_str) - .map(str::to_string), - width: resource - .get("width") - .and_then(Value::as_u64) - .or(fallback_width) - .and_then(|value| u32::try_from(value).ok()), - height: resource - .get("height") - .and_then(Value::as_u64) - .or(fallback_height) - .and_then(|value| u32::try_from(value).ok()), - } -} - -fn editor_agent_attachment_sources(attachments: &[EditorAgentAttachmentRef]) -> Vec { - attachments - .iter() - .filter_map(|attachment| { - attachment - .object_key - .as_deref() - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(|value| value.trim_start_matches('/').to_string()) - .or_else(|| normalize_required_string(attachment.reference_id.as_str())) - }) - .collect() -} - -fn editor_agent_tool_generation_reference_context( - attachments: &[EditorAgentAttachmentRef], - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, - previous_generation_source: Option<&str>, -) -> Vec { - if previous_generation_source.is_some() { - return latest_generation_reference - .map(|reference| { - vec![json!({ - "title": reference - .summary - .clone() - .unwrap_or_else(|| "上一张生成图".to_string()), - "label": reference.summary, - "refType": "previous-generation", - "refId": reference.resource_id, - "resourceId": reference.resource_id, - "objectKey": reference.object_key, - "assetObjectId": reference.asset_object_id, - "toolCallId": reference.tool_call_id, - "toolName": reference.tool_name, - "sourceMessageId": reference.source_message_id, - "source": previous_generation_source, - "implicit": true, - })] - }) - .unwrap_or_default(); - } - - attachments - .iter() - .map(|attachment| { - json!({ - "title": attachment.label.clone().unwrap_or_else(|| "对话附件".to_string()), - "label": attachment.label, - "refType": match attachment.source { - EditorAgentAttachmentSource::CanvasResource => "project-resource", - EditorAgentAttachmentSource::LibraryAsset => "asset", - }, - "refId": attachment.reference_id, - "resourceId": (attachment.source == EditorAgentAttachmentSource::CanvasResource) - .then(|| attachment.reference_id.clone()), - "objectKey": attachment.object_key, - }) - }) - .collect() -} - -fn editor_agent_tool_source_resource_id( - attachments: &[EditorAgentAttachmentRef], - latest_generation_reference: Option<&EditorAgentGeneratedImageReference>, - previous_generation_source: Option<&str>, -) -> Option { - if previous_generation_source.is_some() { - return latest_generation_reference.and_then(|reference| reference.resource_id.clone()); - } - attachments - .first() - .map(|attachment| attachment.reference_id.clone()) -} - -fn split_editor_agent_edit_sources(sources: Vec) -> Option<(String, Vec)> { - let mut iter = sources.into_iter(); - let first = iter.next()?; - Some((first, iter.collect())) -} - -fn editor_agent_tool_asset_kind(tool_name: EditorAgentToolName) -> &'static str { - match tool_name { - EditorAgentToolName::GenerateCharacter => "character", - EditorAgentToolName::GenerateUiDesign => "ui-design", - EditorAgentToolName::EditImage => "editor_agent_edit_image", - EditorAgentToolName::GenerateIconSpritesheet => "icon-spritesheet", - EditorAgentToolName::GenerateImage => "editor_agent_generated_image", - } -} - -fn editor_agent_default_asset_folder_id() -> String { - "project".to_string() -} - -fn editor_agent_tool_default_summary(tool_name: EditorAgentToolName) -> String { - match tool_name { - EditorAgentToolName::GenerateImage => "生成图片", - EditorAgentToolName::EditImage => "修改图片", - EditorAgentToolName::GenerateCharacter => "生成角色形象", - EditorAgentToolName::GenerateIconSpritesheet => "生成图标素材", - EditorAgentToolName::GenerateUiDesign => "生成 UI 设计图", - } - .to_string() -} - -fn build_editor_agent_canvas_completion( - project: &EditorProjectRecord, - tool_name: EditorAgentToolName, - title: &str, -) -> EditorCanvasGenerationCompletionPayload { - let (width, height) = editor_agent_tool_display_size(tool_name); - let (x, y) = next_editor_agent_canvas_position(project.layers.clone(), width, height); - EditorCanvasGenerationCompletionPayload { - dialog_id: None, - title: normalize_optional_string(Some(title.to_string())) - .unwrap_or_else(|| editor_agent_tool_default_summary(tool_name)), - placeholder: EditorCanvasGenerationPlaceholderPayload { - x, - y, - width, - height, - original_width: width, - original_height: height, - }, - } -} - -fn editor_agent_tool_display_size(tool_name: EditorAgentToolName) -> (f64, f64) { - match tool_name { - EditorAgentToolName::GenerateUiDesign => (640.0, 360.0), - EditorAgentToolName::GenerateCharacter => (512.0, 768.0), - _ => (512.0, 512.0), - } -} - -fn next_editor_agent_canvas_position(layers: Value, _width: f64, _height: f64) -> (f64, f64) { - let mut max_right: Option = None; - let mut min_y: Option = None; - if let Value::Array(items) = layers { - for item in items { - if item.get("itemType").and_then(Value::as_str) == Some("generation-dialog") { - continue; - } - let Some(x) = item.get("x").and_then(Value::as_f64) else { - continue; - }; - let Some(y) = item.get("y").and_then(Value::as_f64) else { - continue; - }; - let width = item.get("width").and_then(Value::as_f64).unwrap_or(0.0); - if !x.is_finite() || !y.is_finite() || !width.is_finite() { - continue; - } - max_right = Some(max_right.map_or(x + width, |value| value.max(x + width))); - min_y = Some(min_y.map_or(y, |value| value.min(y))); - } - } - ( - max_right.map_or(0.0, |right| right + EDITOR_AGENT_CANVAS_RESULT_GAP), - min_y.unwrap_or(0.0), - ) -} - -#[cfg(test)] -fn build_echo_assistant_text(text: &str, attachment_count: usize) -> String { - let trimmed = text.trim(); - if trimmed.is_empty() { - return format!("已收到 {attachment_count} 个附件。"); - } - format!("收到：{trimmed}") -} - -fn now_rfc3339() -> String { - shared_kernel::format_rfc3339(time::OffsetDateTime::now_utc()) - .unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_string()) -} - -async fn require_editor_agent_sidebar_enabled( - state: &AppState, - owner_user_id: &str, -) -> Result<(), AppError> { - match state - .is_image_editor_agent_sidebar_enabled_for_user(Some(owner_user_id)) - .await - { - Ok(true) => Ok(()), - Ok(false) => Err(editor_agent_sidebar_unavailable()), - Err(error) => Err(AppError::from_status(StatusCode::BAD_GATEWAY) - .with_message("读取画布 Agent 灰度配置失败") - .with_details(json!({ - "provider": "spacetimedb", - "message": error.to_string(), - }))), - } -} - -fn editor_agent_sidebar_unavailable() -> AppError { - AppError::from_status(StatusCode::SERVICE_UNAVAILABLE) - .with_message("画布 Agent 暂不可用") - .with_details(json!({ - "provider": "editor-agent", - "reason": "image_editor_agent_sidebar_disabled", - "gateKey": module_runtime::IMAGE_EDITOR_AGENT_SIDEBAR_GATE_KEY, - })) -} - -fn editor_agent_bad_request(message: impl Into) -> AppError { - AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ - "provider": "editor-agent", - "message": message.into(), - })) -} - -fn editor_agent_oss_unavailable() -> AppError { - AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({ - "provider": "aliyun-oss", - "reason": "OSS is not configured for editor agent conversations", - })) -} - -fn editor_agent_oss_read_error(message: impl Into) -> AppError { - AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({ - "provider": "aliyun-oss", - "message": message.into(), - })) -} - -fn editor_agent_messages_document_too_large() -> AppError { - AppError::from_status(StatusCode::PAYLOAD_TOO_LARGE).with_details(json!({ - "provider": "editor-agent", - "message": "message document is too large", - "maxBytes": EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES, - })) -} - -fn editor_agent_sse_json_event_or_error(event_name: &str, payload: T) -> Event -where - T: Serialize, -{ - match serde_json::to_string(&payload) { - Ok(data) => Event::default().event(event_name).data(data), - Err(error) => Event::default().event("error").data( - serde_json::to_string(&EditorAgentErrorEvent { - conversation_id: None, - code: "SERIALIZE_EVENT_FAILED".to_string(), - message: error.to_string(), - recoverable: false, - }) - .unwrap_or_else(|_| "{}".to_string()), - ), - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::AppConfig; - use platform_llm::LlmTextProtocol; - - #[test] - fn echo_assistant_text_uses_attachment_fallback() { - assert_eq!( - build_echo_assistant_text(" 画一棵树 ", 0), - "收到：画一棵树" - ); - assert_eq!(build_echo_assistant_text(" ", 2), "已收到 2 个附件。"); - } - - #[test] - fn empty_document_uses_contract_version() { - let document = empty_messages_document("editor-agent-conv-1"); - assert_eq!(document.version, EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION); - assert_eq!(document.conversation_id, "editor-agent-conv-1"); - assert!(document.messages.is_empty()); - } - - #[test] - fn canvas_attachment_normalization_uses_project_resource_truth() { - let conversation = test_conversation_record(); - let attachment = EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::CanvasResource, - reference_id: " resource-1 ".to_string(), - object_key: Some("generated-editor-assets/resource-1.png".to_string()), - image_src: "https://client.example/stale.png".to_string(), - thumbnail_src: Some("https://client.example/thumb.png".to_string()), - label: Some(" 参考图 ".to_string()), - width: Some(1), - height: Some(1), - }; - let resource = test_project_resource_record(); - - let normalized = - normalize_canvas_resource_attachment(&conversation, &attachment, &resource) - .expect("attachment should normalize"); - - assert_eq!(normalized.reference_id, "resource-1"); - assert_eq!( - normalized.object_key.as_deref(), - Some("generated-editor-assets/resource-1.png") - ); - assert_eq!(normalized.image_src, "/api/assets/read/resource-1.png"); - assert_eq!(normalized.thumbnail_src, None); - assert_eq!(normalized.label.as_deref(), Some("参考图")); - assert_eq!(normalized.width, Some(512)); - assert_eq!(normalized.height, Some(256)); - } - - #[test] - fn attachment_normalization_rejects_mismatched_object_key() { - let conversation = test_conversation_record(); - let attachment = EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::CanvasResource, - reference_id: "resource-1".to_string(), - object_key: Some("generated-editor-assets/other.png".to_string()), - image_src: "/api/assets/read/resource-1.png".to_string(), - thumbnail_src: None, - label: None, - width: None, - height: None, - }; - let resource = test_project_resource_record(); - - assert!( - normalize_canvas_resource_attachment(&conversation, &attachment, &resource).is_err() - ); - } - - #[test] - fn library_attachment_normalization_uses_asset_label_fallback() { - let attachment = EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::LibraryAsset, - reference_id: "asset-1".to_string(), - object_key: None, - image_src: "https://client.example/stale.png".to_string(), - thumbnail_src: None, - label: None, - width: None, - height: None, - }; - let asset = EditorAssetRecord { - asset_id: "asset-1".to_string(), - folder_id: "project".to_string(), - label: "素材库图".to_string(), - asset_object_id: Some("asset-object-1".to_string()), - image_src: "/api/assets/read/asset-1.png".to_string(), - object_key: Some("generated-editor-assets/asset-1.png".to_string()), - width: 128, - height: 128, - source_type: "uploaded".to_string(), - prompt: None, - actual_prompt: None, - model: None, - provider: None, - task_id: None, - asset_kind: None, - generation_inputs: None, - source_resource_id: None, - public_showcase_enabled: None, - thumbnail_src: Some("/api/assets/read/asset-1-thumb.png".to_string()), - generation_cost_mud_points: 0, - showcase_id: None, - showcase_review_status: None, - showcase_display_enabled: None, - showcase_like_count: None, - created_at: "2026-07-03T00:00:00Z".to_string(), - updated_at: "2026-07-03T00:00:00Z".to_string(), - }; - - let normalized = normalize_library_asset_attachment(&attachment, &asset) - .expect("library attachment should normalize"); - - assert_eq!(normalized.label.as_deref(), Some("素材库图")); - assert_eq!( - normalized.thumbnail_src.as_deref(), - Some("/api/assets/read/asset-1-thumb.png") - ); - assert_eq!(normalized.width, Some(128)); - assert_eq!(normalized.height, Some(128)); - } - - #[test] - fn generated_image_from_resource_keeps_private_asset_keys() { - let image = editor_agent_generated_image_from_resource( - &json!({ - "resourceId": "resource-1", - "imageSrc": "/generated-editor-assets/result.png", - "thumbnailSrc": "/generated-editor-assets/result-thumb.png", - "objectKey": "generated-editor-assets/result.png", - "assetObjectId": "asset-object-result", - "width": 1024, - "height": 768 - }), - None, - None, - None, - ); - - assert_eq!(image.resource_id.as_deref(), Some("resource-1")); - assert_eq!( - image.object_key.as_deref(), - Some("generated-editor-assets/result.png") - ); - assert_eq!( - image.asset_object_id.as_deref(), - Some("asset-object-result") - ); - assert_eq!(image.width, Some(1024)); - assert_eq!(image.height, Some(768)); - } - - #[test] - fn editor_agent_generations_target_default_asset_library_folder() { - assert_eq!(editor_agent_default_asset_folder_id(), "project"); - } - - #[test] - fn editor_agent_turn_plan_parses_fenced_json_tool_call() { - let plan = parse_editor_agent_turn_plan( - r#"```json -{ - "replyText": "我来生成一张像素风角色图。", - "toolCall": { - "toolName": "generate_character", - "prompt": "像素风银发游侠", - "summary": "生成像素角色", - "imageSize": "1K" - } -} -```"#, - ) - .expect("fenced JSON should parse"); - - assert_eq!(plan.reply_text, "我来生成一张像素风角色图。"); - let tool_call = plan.tool_call.expect("tool call should exist"); - assert_eq!(tool_call.tool_name, EditorAgentToolName::GenerateCharacter); - assert_eq!(tool_call.prompt, "像素风银发游侠"); - assert_eq!(tool_call.summary.as_deref(), Some("生成像素角色")); - assert_eq!(tool_call.image_size.as_deref(), Some("1K")); - } - - #[test] - fn editor_agent_turn_plan_rejects_invalid_llm_response() { - assert!(parse_editor_agent_turn_plan("收到：画一棵树").is_none()); - assert!( - parse_editor_agent_turn_plan( - r#"{"toolCall":{"toolName":"generate_image","prompt":"画一棵树"}}"#, - ) - .is_none() - ); - assert!( - parse_editor_agent_turn_plan( - r#"{"replyText":"我来生成图片。","toolCall":{"toolName":"generate_image"}}"#, - ) - .is_none() - ); - } - - #[test] - fn editor_agent_turn_plan_replaces_structural_reply_fragment() { - let plan = parse_editor_agent_turn_plan( - r#"{"replyText":"{","toolCall":{"toolName":"generate_image","prompt":"生成一张角色规范图","summary":"生成角色规范图"}}"#, - ) - .expect("valid tool call should parse"); - - assert_eq!(plan.reply_text, "我来生成图片。"); - assert_eq!( - plan.tool_call.as_ref().map(|tool_call| tool_call.tool_name), - Some(EditorAgentToolName::GenerateImage) - ); - } - - #[test] - fn editor_agent_llm_system_prompt_describes_each_tool() { - let prompt = editor_agent_llm_system_prompt(); - - for expected in [ - "generate_image：从文字生成一张全新图片", - "edit_image：修改已有图片", - "generate_character：生成新的角色形象", - "generate_icon_spritesheet：生成一组图标素材或图标图集", - "generate_ui_design：生成一张完整 UI 设计图或界面稿", - "规范图/视觉规范图/风格规范图/素材规范展板", - "当前使用 generate_image", - "角色规范图/角色美术视觉规范设定图", - "图标规范图/图标视觉规范展板", - "统一视角、线条粗细", - "色卡/色号", - "工具选择优先级", - "明确修改/指代已有图 => edit_image", - "规范图/视觉规范图/风格规范图/素材规范展板 => generate_image", - ] { - assert!( - prompt.contains(expected), - "system prompt should contain {expected}" - ); - } - } - - #[test] - fn latest_generation_reference_enters_llm_prompt() { - let conversation = test_conversation_record(); - let user_message = test_user_message("把刚才那个衣服换成蓝色"); - let mut document = empty_messages_document(conversation.conversation_id.as_str()); - document.messages.push(EditorAgentMessage { - id: "editor-agent-message-assistant-1".to_string(), - role: EditorAgentMessageRole::Assistant, - kind: EditorAgentMessageKind::Chat, - text: "已生成角色图。".to_string(), - attachments: Vec::new(), - generations: vec![test_completed_generation_record()], - status: EditorAgentMessageStatus::Completed, - created_at: "2026-07-05T00:00:00Z".to_string(), - }); - document.messages.push(user_message.clone()); - - let reference = latest_editor_agent_generated_image_reference(&document) - .expect("latest generated image should be available"); - let prompt = build_editor_agent_llm_user_prompt( - &conversation, - &document, - &user_message, - Some(&reference), - ); - let prompt_json: Value = serde_json::from_str(prompt.as_str()).expect("prompt is JSON"); - - assert_eq!( - prompt_json["latestGeneratedImage"]["toolName"], - json!("generate_image") - ); - assert_eq!( - prompt_json["latestGeneratedImage"]["summary"], - json!("生成红衣角色") - ); - assert_eq!( - prompt_json["latestGeneratedImage"]["resourceId"], - json!("resource-generated-1") - ); - assert_eq!( - prompt_json["latestGeneratedImage"]["objectKey"], - json!("generated-editor-assets/result.png") - ); - assert_eq!( - prompt_json["recentMessages"][0]["generations"][0]["images"][0]["objectKey"], - json!("generated-editor-assets/result.png") - ); - } - - #[test] - fn previous_generation_edit_default_coerces_generate_image_to_edit_image() { - let user_message = test_user_message("把衣服换成蓝色"); - let reference = test_generated_image_reference(); - let plan = EditorAgentTurnPlan { - reply_text: "我来生成图片。".to_string(), - tool_call: Some(EditorAgentToolCallPlan { - tool_name: EditorAgentToolName::GenerateImage, - prompt: "把衣服换成蓝色".to_string(), - summary: Some("生成蓝色衣服".to_string()), - model: None, - aspect_ratio: Some("1:1".to_string()), - image_size: Some("1K".to_string()), - icon_descriptions: Vec::new(), - }), - }; - - let adjusted = - apply_previous_generation_edit_default(plan, &user_message, Some(&reference)); - let tool_call = adjusted.tool_call.expect("tool should be forced"); - - assert_eq!(tool_call.tool_name, EditorAgentToolName::EditImage); - assert_eq!(tool_call.prompt, "把衣服换成蓝色"); - assert_eq!(adjusted.reply_text, "我会基于上一张生成图按你的要求修改。"); - } - - #[test] - fn previous_generation_reference_becomes_implicit_edit_source() { - let reference = test_generated_image_reference(); - - let sources = editor_agent_generated_reference_source(&reference); - let context = editor_agent_tool_generation_reference_context( - &[], - Some(&reference), - sources.as_deref(), - ); - - assert_eq!( - sources.as_deref(), - Some("generated-editor-assets/result.png") - ); - assert_eq!(context[0]["refType"], json!("previous-generation")); - assert_eq!(context[0]["resourceId"], json!("resource-generated-1")); - assert_eq!( - context[0]["objectKey"], - json!("generated-editor-assets/result.png") - ); - assert_eq!(context[0]["implicit"], json!(true)); - } - - #[test] - fn editor_agent_tool_execution_persists_summary() { - let execution = build_editor_agent_tool_execution( - "editor-agent-tool-1", - EditorAgentToolName::EditImage, - Some("换蓝色衣服".to_string()), - json!({ - "ok": true, - "data": { - "model": "gpt-image-2", - "resource": { - "resourceId": "resource-generated-2", - "imageSrc": "/generated-editor-assets/blue.png", - "objectKey": "generated-editor-assets/blue.png", - "assetObjectId": "asset-object-blue", - "width": 1024, - "height": 1024 - } - } - }), - ); - - assert_eq!(execution.summary, "换蓝色衣服"); - assert_eq!(execution.record.summary.as_deref(), Some("换蓝色衣服")); - } - - #[tokio::test] - async fn editor_agent_llm_request_uses_vector_engine_chat_model() { - let conversation = test_conversation_record(); - let document = empty_messages_document(conversation.conversation_id.as_str()); - let user_message = test_user_message("帮我看看这张图还能怎么改"); - let state = AppState::new(AppConfig::default()).expect("state should build"); - - let request = - build_editor_agent_llm_request(&state, &conversation, &document, &user_message, None) - .await; - - assert_eq!(request.protocol, LlmTextProtocol::ChatCompletions); - assert_eq!( - request.model.as_deref(), - Some(platform_agent::CREATIVE_AGENT_GPT5_MODEL) - ); - assert_eq!(request.max_tokens, Some(EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS)); - assert_eq!( - request.request_timeout_ms, - Some(EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS) - ); - } - - #[test] - fn editor_agent_heuristic_routes_tools_from_user_text() { - let character_plan = - heuristic_editor_agent_turn_plan("帮我生成一个角色形象：猫耳骑士", &[]); - assert_eq!( - character_plan - .tool_call - .as_ref() - .expect("character tool expected") - .tool_name, - EditorAgentToolName::GenerateCharacter - ); - - let edit_plan = heuristic_editor_agent_turn_plan( - "把附件这张图改成赛博朋克风格", - &[EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::CanvasResource, - reference_id: "resource-1".to_string(), - object_key: Some("generated-editor-assets/resource-1.png".to_string()), - image_src: "/generated-editor-assets/resource-1.png".to_string(), - thumbnail_src: None, - label: None, - width: Some(512), - height: Some(512), - }], - ); - assert_eq!( - edit_plan - .tool_call - .as_ref() - .expect("edit tool expected") - .tool_name, - EditorAgentToolName::EditImage - ); - - let chat_plan = heuristic_editor_agent_turn_plan("这张图适合做头像吗？", &[]); - assert!(chat_plan.tool_call.is_none()); - } - - #[test] - fn editor_agent_canvas_completion_places_result_after_existing_layers() { - let project = EditorProjectRecord { - project_id: "project-1".to_string(), - owner_user_id: "user-1".to_string(), - title: "画布".to_string(), - canvas: spacetime_client::EditorCanvasRecord { - canvas_id: "canvas-1".to_string(), - project_id: "project-1".to_string(), - title: "画布".to_string(), - viewport: spacetime_client::EditorCanvasViewportRecord { - x: 0.0, - y: 0.0, - scale: 1.0, - }, - layers: json!([]), - created_at: "2026-07-03T00:00:00Z".to_string(), - updated_at: "2026-07-03T00:00:00Z".to_string(), - }, - viewport: spacetime_client::EditorCanvasViewportRecord { - x: 0.0, - y: 0.0, - scale: 1.0, - }, - layers: json!([ - { - "layerId": "layer-1", - "resourceId": "resource-1", - "x": 10.0, - "y": 20.0, - "width": 180.0, - "height": 120.0 - } - ]), - resources: Vec::new(), - created_at: "2026-07-03T00:00:00Z".to_string(), - updated_at: "2026-07-03T00:00:00Z".to_string(), - }; - - let completion = build_editor_agent_canvas_completion( - &project, - EditorAgentToolName::GenerateUiDesign, - "UI 设计图", - ); - - assert_eq!(completion.dialog_id, None); - assert_eq!(completion.title, "UI 设计图"); - assert_eq!(completion.placeholder.x, 222.0); - assert_eq!(completion.placeholder.y, 20.0); - assert_eq!(completion.placeholder.width, 640.0); - assert_eq!(completion.placeholder.height, 360.0); - } - - fn test_conversation_record() -> EditorAgentConversationRecord { - EditorAgentConversationRecord { - conversation_id: "editor-agent-conv-1".to_string(), - project_id: "project-1".to_string(), - owner_user_id: "user-1".to_string(), - title: "新对话".to_string(), - messages_object_key: "editor-agent/editor-agent-conv-1.json".to_string(), - deleted: false, - created_at: "2026-07-03T00:00:00Z".to_string(), - updated_at: "2026-07-03T00:00:00Z".to_string(), - updated_at_micros: 1, - } - } - - fn test_project_resource_record() -> EditorProjectResourceRecord { - EditorProjectResourceRecord { - resource_id: "resource-1".to_string(), - project_id: "project-1".to_string(), - owner_user_id: "user-1".to_string(), - asset_object_id: Some("asset-object-1".to_string()), - image_src: "/api/assets/read/resource-1.png".to_string(), - object_key: Some("generated-editor-assets/resource-1.png".to_string()), - width: 512, - height: 256, - source_type: "generated".to_string(), - prompt: Some("森林".to_string()), - actual_prompt: None, - model: Some("gpt-image-2".to_string()), - provider: Some("VectorEngine".to_string()), - task_id: Some("task-1".to_string()), - source_resource_id: None, - asset_kind: Some("editor_generated_image".to_string()), - generation_inputs: None, - public_showcase_enabled: false, - created_at: "2026-07-03T00:00:00Z".to_string(), - updated_at: "2026-07-03T00:00:00Z".to_string(), - } - } - - fn test_user_message(text: &str) -> EditorAgentMessage { - EditorAgentMessage { - id: "editor-agent-message-user-1".to_string(), - role: EditorAgentMessageRole::User, - kind: EditorAgentMessageKind::Chat, - text: text.to_string(), - attachments: Vec::new(), - generations: Vec::new(), - status: EditorAgentMessageStatus::Completed, - created_at: "2026-07-05T00:00:00Z".to_string(), - } - } - - fn test_completed_generation_record() -> EditorAgentGenerationRecord { - EditorAgentGenerationRecord { - tool_call_id: "editor-agent-tool-generated-1".to_string(), - tool_name: EditorAgentToolName::GenerateImage, - summary: Some("生成红衣角色".to_string()), - task_id: Some("task-generated-1".to_string()), - status: EditorAgentGenerationStatus::Completed, - model: Some("gpt-image-2".to_string()), - images: vec![EditorAgentGeneratedImage { - resource_id: Some("resource-generated-1".to_string()), - object_key: Some("generated-editor-assets/result.png".to_string()), - asset_object_id: Some("asset-object-result".to_string()), - image_src: "/generated-editor-assets/result.png".to_string(), - thumbnail_src: Some("/generated-editor-assets/result-thumb.png".to_string()), - width: Some(1024), - height: Some(1024), - }], - error: None, - } - } - - fn test_generated_image_reference() -> EditorAgentGeneratedImageReference { - let mut document = empty_messages_document("editor-agent-conv-1"); - document.messages.push(EditorAgentMessage { - id: "editor-agent-message-assistant-1".to_string(), - role: EditorAgentMessageRole::Assistant, - kind: EditorAgentMessageKind::Chat, - text: "已生成角色图。".to_string(), - attachments: Vec::new(), - generations: vec![test_completed_generation_record()], - status: EditorAgentMessageStatus::Completed, - created_at: "2026-07-05T00:00:00Z".to_string(), - }); - latest_editor_agent_generated_image_reference(&document) - .expect("latest generated image should be available") - } -} diff --git a/server-rs/crates/api-server/src/editor_agent/api.rs b/server-rs/crates/api-server/src/editor_agent/api.rs new file mode 100644 index 000000000..04a7e33a2 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/api.rs @@ -0,0 +1,1196 @@ +use axum::extract::{Path, State}; +use axum::{Extension, Json}; +use module_editor_agent::{ + EDITOR_AGENT_CONVERSATION_ID_PREFIX, EDITOR_AGENT_DEFAULT_CONVERSATION_TITLE, + derive_conversation_title, editor_agent_messages_object_key, validate_user_message, +}; +use platform_editor_agent::framework::agent_builder::AgentBuilder; +use platform_editor_agent::framework::error::PromptError; +use platform_editor_agent::framework::memory::VecMemory; +use platform_editor_agent::framework::run::PromptOutput; +use platform_editor_agent::framework::tool::Tool; +use platform_llm::LlmMessage; +use serde::Serialize; +use serde_json::{Value, json}; +use sha2::{Digest, Sha256}; +use shared_contracts::assets::{ + EditorBackgroundMusicGenerateRequest, EditorSoundEffectGenerateRequest, + EditorVideoGenerateRequest, +}; +use shared_contracts::editor_agent::{ + CreateEditorAgentConversationRequest, EDITOR_AGENT_ERROR_MESSAGE_PREFIX, + EditorAgentConversationListResponse, EditorAgentConversationMessagesDocument, + EditorAgentConversationResponse, EditorAgentConversationSummary, EditorAgentMessage, + EditorAgentMessageRequest, EditorAgentMessageResponse, EditorAgentMessageRole, + EditorAgentToolCall, EditorAgentToolCallStatus, +}; +use spacetime_client::{ + EditorAgentConversationCreateRecordInput, EditorAgentConversationDeleteRecordInput, + EditorAgentConversationRecord, EditorAgentConversationTouchRecordInput, + EditorProjectGetRecordInput, +}; + +use crate::api_response::json_success_body; +use crate::auth::AuthenticatedAccessToken; +use crate::editor_agent::pricing::EditorAgentPricedTool; +use crate::editor_agent::utils::{ + IntoImageId, build_editor_agent_canvas_completion, conversation_detail_from_record, + conversation_summary_from_record, editor_agent_bad_request, empty_messages_document, + ensure_editor_project_access, normalize_editor_agent_attachments, now_rfc3339, + read_messages_document, require_editor_agent_sidebar_enabled, write_messages_document, +}; +use crate::editor_agent::{context, display_args, reconcile}; +use crate::editor_generation_config::EditorGenerationPricingConfig; +use crate::editor_generation_queue::{ + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, + EDITOR_IMAGE_EDIT_JOB_KIND, EDITOR_IMAGE_GENERATION_JOB_KIND, + EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, EDITOR_VIDEO_GENERATION_JOB_KIND, + enqueue_editor_generation_job_with_identity, +}; +use crate::editor_project::{ + EditorIconSpritesheetGenerationRequest, EditorImageEditRequest, EditorImageGenerationRequest, +}; +use crate::editor_project::{current_utc_micros, map_editor_project_error}; +use crate::http_error::AppError; +use crate::request_context::RequestContext; +use crate::state::AppState; +use platform_editor_agent::agent::agent::LlmChatAgentBuilder; +use platform_editor_agent::agent::tools::context::EditorToolContext; +use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; +use platform_editor_agent::agent::tools::generate_background_music::{ + GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, +}; +use platform_editor_agent::agent::tools::generate_character::GenerateCharacterTool; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ + GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, +}; +use platform_editor_agent::agent::tools::generate_image::{ + GenerateImageTool, GenerateImageToolArgs, +}; +use platform_editor_agent::agent::tools::generate_sound_effect::{ + GenerateSoundEffectTool, GenerateSoundEffectToolArgs, +}; +use platform_editor_agent::agent::tools::generate_ui_design::GenerateUiDesignTool; +use platform_editor_agent::agent::tools::generate_video::{ + GenerateVideoTool, GenerateVideoToolArgs, +}; +use shared_kernel::{build_prefixed_uuid_id, normalize_optional_string, normalize_required_string}; + +const EDITOR_AGENT_CLIENT_MESSAGE_ID_MAX_CHARS: usize = 128; + +pub async fn editor_agent_message( + State(state): State, + Path(conversation_id): Path, + Extension(_request_context): Extension, + Extension(authenticated): Extension, + Json(payload): Json, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + let client_message_id = validate_editor_agent_message_request(&payload)?; + let normalized_text = payload.text.trim().to_string(); + // Load conversation & attachments + let conversation = state + .spacetime_client() + .get_editor_agent_conversation(conversation_id.clone(), owner_user_id.clone()) + .await + .map_err(|e| { + AppError::from_status(axum::http::StatusCode::NOT_FOUND) + .with_details(json!({ "message": format!("conversation not found: {e}") })) + })?; + + let attachments = + normalize_editor_agent_attachments(&state, &conversation, payload.attachments.as_slice()) + .await?; + + let conversation_lock = crate::editor_agent::utils::editor_agent_conversation_lock( + conversation.conversation_id.as_str(), + ); + let _conversation_lock_guard = conversation_lock.lock_owned().await; + + let mut document: EditorAgentConversationMessagesDocument = + read_messages_document(&state, &conversation).await?; + + let existing_user_index = find_idempotent_editor_agent_user_message( + &document, + client_message_id.as_str(), + normalized_text.as_str(), + attachments.as_slice(), + )?; + + let (user_message, history_end, conversation_summary) = + if let Some(user_index) = existing_user_index { + let delta_messages = document.messages[user_index + 1..] + .iter() + .take_while(|message| message.role != EditorAgentMessageRole::User) + .cloned() + .collect::>(); + if !delta_messages.is_empty() { + return Ok(Json(EditorAgentMessageResponse { + conversation: conversation_summary_from_record(conversation), + delta_messages, + error_message: None, + })); + } + + ( + document.messages[user_index].clone(), + user_index, + conversation_summary_from_record(conversation.clone()), + ) + } else { + // Determine initialization before attachment bookkeeping adds a system message. + let was_empty = document.messages.is_empty(); + let now = now_rfc3339(); + if !attachments.is_empty() { + let mut attachment_info = String::new(); + attachment_info.push_str("user has just uploaded attachments of the order: "); + for attachment in &attachments { + attachment_info.push_str(&format!("{} ,", attachment.clone().into_image_id())); + } + document.messages.push(EditorAgentMessage { + id: document.messages.len(), + client_message_id: None, + role: EditorAgentMessageRole::System, + text: attachment_info, + attachments: Vec::new(), + tool_call: None, + created_at: now.clone(), + }); + } + + let history_end = document.messages.len(); + let user_message = EditorAgentMessage { + id: document.messages.len(), + client_message_id: Some(client_message_id), + role: EditorAgentMessageRole::User, + text: normalized_text, + attachments, + tool_call: None, + created_at: now, + }; + document.messages.push(user_message.clone()); + write_messages_document(&state, &conversation, &document).await?; + + // Persist and return the authoritative summary for every turn. Initialization sets the + // title from the first user prompt; a metadata write failure must fail the request. + let updated_conversation = state + .spacetime_client() + .touch_editor_agent_conversation(EditorAgentConversationTouchRecordInput { + conversation_id: conversation.conversation_id.clone(), + owner_user_id: conversation.owner_user_id.clone(), + title: was_empty.then(|| derive_conversation_title(user_message.text.as_str())), + updated_at_micros: current_utc_micros(), + }) + .await + .map_err(map_editor_project_error)?; + + ( + user_message, + history_end, + conversation_summary_from_record(updated_conversation), + ) + }; + + // The current user message is passed separately to prompt(), so memory stops before it. + let previous_messages: Vec = document.messages[..history_end] + .iter() + .map(|message| match message.role { + EditorAgentMessageRole::User => LlmMessage::user(&message.text), + EditorAgentMessageRole::Assistant => LlmMessage::assistant(&message.text), + EditorAgentMessageRole::System => LlmMessage::system(&message.text), + }) + // Tool calls and attachment bookkeeping are separate system messages. + .rev() + .take(18) + .rev() + .collect(); + + // Build tool context from document + let tool_context = context::build_tool_context(&document); + + // Build and run agent + let Some(llm_client) = state.creative_agent_gpt5_client() else { + return persist_editor_agent_planning_error( + &state, + &conversation, + &mut document, + conversation_summary, + "Creative Agent GPT-5 client not configured", + ) + .await; + }; + let llm_client = llm_client.clone(); + let pricing = match state.editor_generation_pricing().await { + Ok(pricing) => pricing, + Err(error) => { + return persist_editor_agent_planning_error( + &state, + &conversation, + &mut document, + conversation_summary, + format!("failed to load editor generation pricing: {error}"), + ) + .await; + } + }; + + let memory = VecMemory::new(previous_messages); + + let mut agent = LlmChatAgentBuilder::new() + .with_client(llm_client) + .system_prompt(editor_agent_system_prompt()) + .tool(EditImageTool { + context: tool_context.clone(), + }) + .tool(GenerateImageTool { + context: tool_context.clone(), + }) + .tool(GenerateCharacterTool { + context: tool_context.clone(), + }) + .tool(GenerateIconSpritesheetTool { + context: tool_context.clone(), + }) + .tool(GenerateSoundEffectTool) + .tool(GenerateBackgroundMusicTool) + .tool(GenerateVideoTool { + context: tool_context.clone(), + }) + .tool(GenerateUiDesignTool { + context: tool_context.clone(), + }) + .max_turns(3) + .memory(memory) + .build(); + + let agent_result = agent + .prompt(LlmMessage::user(user_message.text.clone())) + .await; + + let assistant_now = now_rfc3339(); + + match build_delta_messages( + agent_result, + &assistant_now, + document.messages.len(), + &document, + &tool_context, + &pricing, + ) { + Err(error) => { + persist_editor_agent_planning_error( + &state, + &conversation, + &mut document, + conversation_summary, + error.to_string(), + ) + .await + } + Ok(delta_messages) => { + for msg in &delta_messages { + document.messages.push(msg.clone()); + } + write_messages_document(&state, &conversation, &document).await?; + + Ok(Json(EditorAgentMessageResponse { + conversation: conversation_summary, + delta_messages, + error_message: None, + })) + } + } +} + +fn build_editor_agent_error_message( + message_id: usize, + error: impl std::fmt::Display, +) -> EditorAgentMessage { + EditorAgentMessage { + id: message_id, + client_message_id: None, + role: EditorAgentMessageRole::System, + text: format!("{EDITOR_AGENT_ERROR_MESSAGE_PREFIX}{error}"), + attachments: Vec::new(), + tool_call: None, + created_at: now_rfc3339(), + } +} + +async fn persist_editor_agent_planning_error( + state: &AppState, + conversation: &EditorAgentConversationRecord, + document: &mut EditorAgentConversationMessagesDocument, + conversation_summary: EditorAgentConversationSummary, + error: impl std::fmt::Display, +) -> Result, AppError> { + let error_message = build_editor_agent_error_message(document.messages.len(), error); + document.messages.push(error_message.clone()); + write_messages_document(state, conversation, document).await?; + + Ok(Json(EditorAgentMessageResponse { + conversation: conversation_summary, + delta_messages: vec![error_message], + error_message: None, + })) +} + +fn validate_editor_agent_message_request( + payload: &EditorAgentMessageRequest, +) -> Result { + let client_message_id = normalize_required_string(payload.client_message_id.as_str()) + .ok_or_else(|| editor_agent_bad_request("clientMessageId is required"))?; + if client_message_id.chars().count() > EDITOR_AGENT_CLIENT_MESSAGE_ID_MAX_CHARS { + return Err(editor_agent_bad_request(format!( + "clientMessageId must not exceed {EDITOR_AGENT_CLIENT_MESSAGE_ID_MAX_CHARS} characters" + ))); + } + let attachment_reference_ids = payload + .attachments + .iter() + .map(|attachment| attachment.reference_id.clone()) + .collect::>(); + validate_user_message(payload.text.as_str(), attachment_reference_ids.as_slice()) + .map_err(|error| editor_agent_bad_request(error.to_string()))?; + Ok(client_message_id) +} + +fn find_idempotent_editor_agent_user_message( + document: &EditorAgentConversationMessagesDocument, + client_message_id: &str, + text: &str, + attachments: &[shared_contracts::editor_agent::EditorAgentAttachmentRef], +) -> Result, AppError> { + let Some((index, message)) = document + .messages + .iter() + .enumerate() + .find(|(_, message)| message.client_message_id.as_deref() == Some(client_message_id)) + else { + return Ok(None); + }; + + if message.role != EditorAgentMessageRole::User + || message.text != text + || !editor_agent_attachment_requests_match(&message.attachments, attachments) + { + return Err( + AppError::from_status(axum::http::StatusCode::CONFLICT).with_details(json!({ + "provider": "editor-agent", + "field": "clientMessageId", + "message": "clientMessageId already exists with different message content", + })), + ); + } + + Ok(Some(index)) +} + +fn editor_agent_attachment_requests_match( + stored: &[shared_contracts::editor_agent::EditorAgentAttachmentRef], + submitted: &[shared_contracts::editor_agent::EditorAgentAttachmentRef], +) -> bool { + stored.len() == submitted.len() + && stored.iter().zip(submitted).all(|(left, right)| { + left.source == right.source && left.reference_id == right.reference_id + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use shared_contracts::editor_agent::{EditorAgentAttachmentRef, EditorAgentAttachmentSource}; + + fn attachment(reference_id: impl Into) -> EditorAgentAttachmentRef { + EditorAgentAttachmentRef { + source: EditorAgentAttachmentSource::CanvasResource, + reference_id: reference_id.into(), + object_key: None, + image_src: "/generated/test.png".to_string(), + thumbnail_src: None, + label: None, + width: None, + height: None, + } + } + + #[test] + fn validates_editor_agent_message_before_normalizing_attachments() { + let empty_payload = EditorAgentMessageRequest { + client_message_id: "client-message-empty".to_string(), + text: " ".to_string(), + attachments: Vec::new(), + }; + assert!(validate_editor_agent_message_request(&empty_payload).is_err()); + + let too_many_payload = EditorAgentMessageRequest { + client_message_id: "client-message-many".to_string(), + text: "生成一张图".to_string(), + attachments: (0..10) + .map(|index| attachment(format!("res-{index}"))) + .collect(), + }; + assert!(validate_editor_agent_message_request(&too_many_payload).is_err()); + + let attachment_only_payload = EditorAgentMessageRequest { + client_message_id: "client-message-attachment".to_string(), + text: String::new(), + attachments: vec![attachment("res-1")], + }; + assert!(validate_editor_agent_message_request(&attachment_only_payload).is_ok()); + + let missing_client_message_id = EditorAgentMessageRequest { + client_message_id: " ".to_string(), + text: "生成一张图".to_string(), + attachments: Vec::new(), + }; + assert!(validate_editor_agent_message_request(&missing_client_message_id).is_err()); + + let oversized_client_message_id = EditorAgentMessageRequest { + client_message_id: "x".repeat(EDITOR_AGENT_CLIENT_MESSAGE_ID_MAX_CHARS + 1), + text: "生成一张图".to_string(), + attachments: Vec::new(), + }; + assert!(validate_editor_agent_message_request(&oversized_client_message_id).is_err()); + } + + #[test] + fn detects_idempotent_message_replays_and_content_conflicts() { + let stored_attachment = attachment("res-1"); + let document = EditorAgentConversationMessagesDocument { + version: 2, + conversation_id: "conversation-1".to_string(), + messages: vec![EditorAgentMessage { + id: 0, + client_message_id: Some("client-message-1".to_string()), + role: EditorAgentMessageRole::User, + text: "生成一张图".to_string(), + attachments: vec![stored_attachment.clone()], + tool_call: None, + created_at: "2026-07-16T00:00:00Z".to_string(), + }], + }; + + assert_eq!( + find_idempotent_editor_agent_user_message( + &document, + "client-message-1", + "生成一张图", + &[stored_attachment.clone()], + ) + .expect("same request should be an idempotent replay"), + Some(0), + ); + assert!( + find_idempotent_editor_agent_user_message( + &document, + "client-message-1", + "生成另一张图", + &[stored_attachment], + ) + .is_err() + ); + assert_eq!( + find_idempotent_editor_agent_user_message( + &document, + "client-message-2", + "生成一张图", + &[], + ) + .expect("new request should not match"), + None, + ); + } + + #[test] + fn builds_system_error_message_with_wire_prefix() { + let message = build_editor_agent_error_message(3, "planning failed"); + + assert_eq!(message.id, 3); + assert_eq!(message.role, EditorAgentMessageRole::System); + assert_eq!(message.text, "ERROR planning failed"); + assert!(message.tool_call.is_none()); + } +} +fn editor_agent_system_prompt() -> &'static str { + r#" +你是 Genarrative 图片画布 Agent，只负责帮助用户理解、规划和触发画布生成工具。 +对话回复要简短，不能承诺免费生成；实际生成工具由后端按模型定价扣泥点。 +"# +} + +fn build_delta_messages( + result: Result, PromptError>, + created_at: &str, + messages_offset: usize, + document: &EditorAgentConversationMessagesDocument, + tool_context: &EditorToolContext, + pricing: &EditorGenerationPricingConfig, +) -> Result, PromptError> { + let outputs = result?; + let mut messages = Vec::with_capacity(outputs.len()); + + for (i, out) in outputs.into_iter().enumerate() { + let absolute_idx = messages_offset + i; + match out { + PromptOutput::Text(text) => { + messages.push(EditorAgentMessage { + id: absolute_idx, + client_message_id: None, + role: EditorAgentMessageRole::Assistant, + text, + attachments: Vec::new(), + tool_call: None, + created_at: created_at.to_string(), + }); + } + PromptOutput::Tool(tco) => { + let display_args = display_args::build_tool_call_display_args( + tco.tool_call.name.as_str(), + &tco.tool_call.args, + document, + tool_context, + pricing, + )?; + messages.push(EditorAgentMessage { + id: absolute_idx, + client_message_id: None, + role: EditorAgentMessageRole::System, + text: tco.message, + attachments: Vec::new(), + tool_call: Some(EditorAgentToolCall { + tool_name: tco.tool_call.name, + status: EditorAgentToolCallStatus::NotCompleted, + args: tco.tool_call.args, + display_args, + external_job_id: None, + images: Vec::new(), + videos: Vec::new(), + audios: Vec::new(), + error: None, + }), + created_at: created_at.to_string(), + }); + } + } + } + + Ok(messages) +} + +#[derive(Debug, Serialize)] +#[serde(rename_all = "camelCase")] +struct EditorAgentConversationDeleteResponse { + deleted_conversation_id: String, + conversation: EditorAgentConversationSummary, +} + +pub async fn list_editor_agent_conversations( + State(state): State, + Path(project_id): Path, + Extension(request_context): Extension, + Extension(authenticated): Extension, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + ensure_editor_project_access(&state, project_id.as_str(), owner_user_id.as_str()).await?; + let conversations = state + .spacetime_client() + .list_editor_agent_conversations(project_id, owner_user_id) + .await + .map_err(map_editor_project_error)? + .into_iter() + .map(conversation_summary_from_record) + .collect(); + + Ok(json_success_body( + Some(&request_context), + EditorAgentConversationListResponse { conversations }, + )) +} + +pub async fn create_editor_agent_conversation( + State(state): State, + Path(project_id): Path, + Extension(request_context): Extension, + Extension(authenticated): Extension, + Json(payload): Json, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + ensure_editor_project_access(&state, project_id.as_str(), owner_user_id.as_str()).await?; + + let conversation_id = build_prefixed_uuid_id(EDITOR_AGENT_CONVERSATION_ID_PREFIX); + let messages_object_key = editor_agent_messages_object_key(conversation_id.as_str()); + let title = normalize_optional_string(payload.title) + .unwrap_or_else(|| EDITOR_AGENT_DEFAULT_CONVERSATION_TITLE.to_string()); + let now_micros = current_utc_micros(); + let seed_record = EditorAgentConversationRecord { + conversation_id: conversation_id.clone(), + project_id: project_id.clone(), + owner_user_id: owner_user_id.clone(), + title: title.clone(), + messages_object_key: messages_object_key.clone(), + deleted: false, + created_at: now_rfc3339(), + updated_at: now_rfc3339(), + updated_at_micros: now_micros, + }; + write_messages_document( + &state, + &seed_record, + &empty_messages_document(conversation_id.as_str()), + ) + .await?; + + let conversation = state + .spacetime_client() + .create_editor_agent_conversation(EditorAgentConversationCreateRecordInput { + conversation_id, + project_id, + owner_user_id, + title, + messages_object_key, + created_at_micros: now_micros, + }) + .await + .map_err(map_editor_project_error)?; + let document = read_messages_document(&state, &conversation).await?; + + Ok(json_success_body( + Some(&request_context), + EditorAgentConversationResponse { + conversation: conversation_detail_from_record(conversation, document.messages), + }, + )) +} + +pub async fn get_editor_agent_conversation( + State(state): State, + Path(conversation_id): Path, + Extension(request_context): Extension, + Extension(authenticated): Extension, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + let conversation = state + .spacetime_client() + .get_editor_agent_conversation(conversation_id, owner_user_id) + .await + .map_err(map_editor_project_error)?; + let conversation_lock = crate::editor_agent::utils::editor_agent_conversation_lock( + conversation.conversation_id.as_str(), + ); + let _conversation_lock_guard = conversation_lock.lock_owned().await; + let mut document = read_messages_document(&state, &conversation).await?; + let reconciled_messages = + reconcile::reconcile_editor_agent_tool_calls(&state, &conversation, &mut document).await?; + if !reconciled_messages.is_empty() { + write_messages_document(&state, &conversation, &document).await?; + } + + Ok(json_success_body( + Some(&request_context), + EditorAgentConversationResponse { + conversation: conversation_detail_from_record(conversation, document.messages), + }, + )) +} + +pub async fn delete_editor_agent_conversation( + State(state): State, + Path(conversation_id): Path, + Extension(request_context): Extension, + Extension(authenticated): Extension, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + let conversation = state + .spacetime_client() + .delete_editor_agent_conversation(EditorAgentConversationDeleteRecordInput { + conversation_id, + owner_user_id, + updated_at_micros: current_utc_micros(), + }) + .await + .map_err(map_editor_project_error)?; + + Ok(json_success_body( + Some(&request_context), + EditorAgentConversationDeleteResponse { + deleted_conversation_id: conversation.conversation_id.clone(), + conversation: conversation_summary_from_record(conversation), + }, + )) +} + +pub async fn cancel_editor_agent_tool_call( + State(state): State, + Path((conversation_id, message_id)): Path<(String, usize)>, + Extension(request_context): Extension, + Extension(authenticated): Extension, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + + let conversation = state + .spacetime_client() + .get_editor_agent_conversation(conversation_id, owner_user_id) + .await + .map_err(|e| { + AppError::from_status(axum::http::StatusCode::NOT_FOUND) + .with_details(json!({ "message": format!("conversation not found: {e}") })) + })?; + + let conversation_lock = crate::editor_agent::utils::editor_agent_conversation_lock( + conversation.conversation_id.as_str(), + ); + let _conversation_lock_guard = conversation_lock.lock_owned().await; + + let mut document: EditorAgentConversationMessagesDocument = + read_messages_document(&state, &conversation).await?; + + // Validate message index + if message_id >= document.messages.len() { + return Err(AppError::from_status(axum::http::StatusCode::NOT_FOUND) + .with_details(json!({ "message": "message not found" }))); + } + + let msg = &mut document.messages[message_id]; + + // Validate role and tool_call + if msg.role != EditorAgentMessageRole::System { + return Err(editor_agent_bad_request("message is not a system message")); + } + let tc = msg + .tool_call + .as_mut() + .ok_or_else(|| editor_agent_bad_request("message has no tool call"))?; + if tc.status != EditorAgentToolCallStatus::NotCompleted || tc.external_job_id.is_some() { + return Err(editor_agent_bad_request( + "tool call is no longer pending confirmation", + )); + } + + tc.status = EditorAgentToolCallStatus::Cancelled; + let arg_json = tc.args.to_string(); + msg.text = format!( + "[tool_call:{tool_name}] args: {arg_json} output: 用户已取消该操作", + tool_name = tc.tool_name, + arg_json = arg_json, + ); + + write_messages_document(&state, &conversation, &document).await?; + + Ok(json_success_body( + Some(&request_context), + json!({ "ok": true }), + )) +} + +pub async fn confirm_editor_agent_tool_call( + State(state): State, + Path((conversation_id, message_id)): Path<(String, usize)>, + Extension(request_context): Extension, + Extension(authenticated): Extension, +) -> Result, AppError> { + let owner_user_id = authenticated.claims().user_id().to_string(); + require_editor_agent_sidebar_enabled(&state, owner_user_id.as_str()).await?; + let conversation = state + .spacetime_client() + .get_editor_agent_conversation(conversation_id, owner_user_id) + .await + .map_err(|error| { + AppError::from_status(axum::http::StatusCode::NOT_FOUND).with_details(json!({ + "message": format!("conversation not found: {error}"), + })) + })?; + let conversation_lock = crate::editor_agent::utils::editor_agent_conversation_lock( + conversation.conversation_id.as_str(), + ); + let _conversation_lock_guard = conversation_lock.lock_owned().await; + let mut document = read_messages_document(&state, &conversation).await?; + let message = document + .messages + .get(message_id) + .ok_or_else(|| AppError::from_status(axum::http::StatusCode::NOT_FOUND))?; + if message.role != EditorAgentMessageRole::System { + return Err(editor_agent_bad_request("message is not a system message")); + } + let tool_call = message + .tool_call + .as_ref() + .ok_or_else(|| editor_agent_bad_request("message has no tool call"))?; + if tool_call.status == EditorAgentToolCallStatus::Cancelled { + return Err(editor_agent_bad_request("tool call was cancelled")); + } + if tool_call.status != EditorAgentToolCallStatus::NotCompleted + || tool_call.external_job_id.is_some() + { + return Ok(json_success_body( + Some(&request_context), + json!({ "ok": true }), + )); + } + + let tool_name = tool_call.tool_name.clone(); + let tool_args = tool_call.args.clone(); + let pricing = state.editor_generation_pricing().await.map_err(|error| { + AppError::from_status(axum::http::StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ + "provider": "editor-generation-pricing", + "message": error.to_string(), + })) + })?; + let project = load_editor_agent_project(&state, &conversation).await?; + let context = context::build_tool_context(&document); + let generation_inputs = |title: &str, value: &str| { + Some(json!({ + // TODO extract const + "source": "editor-agent", + "conversationId": conversation.conversation_id, + "toolCallMessageId": message_id, + "fields": [{ "title": title, "value": value }], + })) + }; + + let (job_kind, request_label, price_mud_points, payload) = match tool_name.as_str() { + GenerateImageTool::NAME | GenerateCharacterTool::NAME | GenerateUiDesignTool::NAME => { + let args: GenerateImageToolArgs = parse_confirm_tool_args(&tool_args)?; + let tool = GenerateImageTool { + context: context.clone(), + }; + tool.validate_args(&args).map_err(|error| { + editor_agent_bad_request(format!("invalid tool call args: {error}")) + })?; + let kind = match tool_name.as_str() { + GenerateCharacterTool::NAME => Some("character"), + GenerateUiDesignTool::NAME => Some("ui-design"), + _ => None, + }; + let price = match kind { + Some("character") => GenerateCharacterTool { + context: context.clone(), + } + .pricing(&pricing, &args), + Some("ui-design") => GenerateUiDesignTool { + context: context.clone(), + } + .pricing(&pricing, &args), + _ => tool.pricing(&pricing, &args), + }; + let title = args.prompt.clone(); + let reference_image_srcs = args + .reference_image_ids + .iter() + .map(|image_id| { + context + .image_data_key(image_id) + .map(ToOwned::to_owned) + .ok_or_else(|| { + editor_agent_bad_request(format!("image {image_id} not found")) + }) + }) + .collect::, _>>()?; + let payload = EditorImageGenerationRequest { + prompt: args.prompt, + size: None, + kind: kind.map(ToOwned::to_owned), + model: None, + screen_color: (kind == Some("character")).then(|| "auto".to_string()), + seg_model: (kind == Some("character")).then(|| "birefnet".to_string()), + aspect_ratio: args.aspect_ratio, + image_size: args.image_size, + reference_image_srcs: Some(reference_image_srcs), + project_id: Some(conversation.project_id.clone()), + asset_kind: Some(kind.unwrap_or("editor_agent_generated_image").to_string()), + generation_inputs: generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title.clone()), + source_resource_id: None, + canvas_completion: Some(build_editor_agent_canvas_completion( + &project, &tool_name, &title, + )), + }; + ( + EDITOR_IMAGE_GENERATION_JOB_KIND, + "画布 Agent 生成图片", + price, + serde_json::to_value(payload), + ) + } + EditImageTool::NAME => { + let args: EditImageToolArgs = parse_confirm_tool_args(&tool_args)?; + if args.prompt.trim().is_empty() { + return Err(editor_agent_bad_request("prompt not provided")); + } + let source_image_src = context + .image_data_key(&args.object_image_id) + .ok_or_else(|| editor_agent_bad_request("object image not found"))? + .to_string(); + let reference_image_srcs = args + .reference_image_ids + .iter() + .map(|image_id| { + context + .image_data_key(image_id) + .map(ToOwned::to_owned) + .ok_or_else(|| { + editor_agent_bad_request(format!("image {image_id} not found")) + }) + }) + .collect::, _>>()?; + let title = args.prompt.clone(); + let price = EditImageTool { + context: context.clone(), + } + .pricing(&pricing, &args); + let payload = EditorImageEditRequest { + prompt: args.prompt, + source_image_src, + size: None, + model: None, + aspect_ratio: None, + image_size: None, + reference_image_srcs: Some(reference_image_srcs), + project_id: Some(conversation.project_id.clone()), + asset_kind: Some("editor_agent_edit_image".to_string()), + generation_inputs: generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title.clone()), + source_resource_id: None, + target_layer_id: None, + canvas_completion: Some(build_editor_agent_canvas_completion( + &project, + EditImageTool::NAME, + &title, + )), + }; + ( + EDITOR_IMAGE_EDIT_JOB_KIND, + "画布 Agent 修改图片", + price, + serde_json::to_value(payload), + ) + } + GenerateIconSpritesheetTool::NAME => { + let args: GenerateIconSpritesheetToolArgs = parse_confirm_tool_args(&tool_args)?; + let tool = GenerateIconSpritesheetTool { + context: context.clone(), + }; + let price = tool.pricing(&pricing, &args); + let reference_image_src = context + .image_data_key(&args.reference_image_id) + .ok_or_else(|| editor_agent_bad_request("reference image not found"))? + .to_string(); + let reference_image_srcs = args + .reference_image_ids + .iter() + .map(|image_id| { + context + .image_data_key(image_id) + .map(ToOwned::to_owned) + .ok_or_else(|| { + editor_agent_bad_request(format!("image {image_id} not found")) + }) + }) + .collect::, _>>()?; + let payload = EditorIconSpritesheetGenerationRequest { + reference_image_src, + reference_image_srcs: Some(reference_image_srcs), + icon_descriptions: args.icon_descriptions.clone(), + model: None, + screen_color: Some("auto".to_string()), + seg_model: Some("birefnet".to_string()), + aspect_ratio: args.aspect_ratio, + image_size: args.image_size, + project_id: Some(conversation.project_id.clone()), + generation_inputs: Some( + json!({"source":"editor-agent","conversationId":conversation.conversation_id,"toolCallMessageId":message_id}), + ), + asset_folder_id: Some("project".to_string()), + asset_label: Some("图标素材图集".to_string()), + canvas_completion: Some(build_editor_agent_canvas_completion( + &project, + GenerateIconSpritesheetTool::NAME, + "图标素材图集", + )), + }; + ( + EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, + "画布 Agent 生成图标素材", + price, + serde_json::to_value(payload), + ) + } + GenerateVideoTool::NAME => { + let args: GenerateVideoToolArgs = parse_confirm_tool_args(&tool_args)?; + let reference_image_srcs = args + .reference_image_ids + .iter() + .map(|image_id| { + context + .image_data_key(image_id) + .map(ToOwned::to_owned) + .ok_or_else(|| { + editor_agent_bad_request(format!("image {image_id} not found")) + }) + }) + .collect::, _>>()?; + let title = args.prompt.clone(); + let price = GenerateVideoTool { + context: context.clone(), + } + .pricing(&pricing, &args); + let payload = EditorVideoGenerateRequest { + prompt: args.prompt, + model: args.model.unwrap_or_else(|| "seedance2.0-fast".to_string()), + aspect_ratio: args.aspect_ratio.unwrap_or_else(|| "16:9".to_string()), + duration_seconds: args.duration_seconds.unwrap_or(4), + resolution: args.resolution.unwrap_or_else(|| "720p".to_string()), + mode: "std".to_string(), + sound: args.sound.unwrap_or_else(|| "off".to_string()), + web_search_enabled: false, + reference_image_srcs, + reference_video_srcs: Vec::new(), + reference_audio_srcs: Vec::new(), + project_id: Some(conversation.project_id.clone()), + canvas_completion: Some(build_editor_agent_canvas_completion( + &project, + GenerateVideoTool::NAME, + &title, + )), + generation_inputs: generation_inputs("用户指令", &title), + source_resource_id: None, + asset_kind: Some("video".to_string()), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title), + }; + ( + EDITOR_VIDEO_GENERATION_JOB_KIND, + "画布 Agent 生成视频", + price, + serde_json::to_value(payload), + ) + } + GenerateSoundEffectTool::NAME => { + let args: GenerateSoundEffectToolArgs = parse_confirm_tool_args(&tool_args)?; + let price = GenerateSoundEffectTool.pricing(&pricing, &args); + let title = args.prompt.clone(); + let payload = EditorSoundEffectGenerateRequest { + prompt: args.prompt, + model: None, + duration: args.duration.unwrap_or(3), + project_id: Some(conversation.project_id.clone()), + canvas_completion: Some(build_editor_agent_canvas_completion( + &project, + GenerateSoundEffectTool::NAME, + &title, + )), + generation_inputs: generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some(title), + }; + ( + EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, + "画布 Agent 生成音效", + price, + serde_json::to_value(payload), + ) + } + GenerateBackgroundMusicTool::NAME => { + let args: GenerateBackgroundMusicToolArgs = parse_confirm_tool_args(&tool_args)?; + let price = GenerateBackgroundMusicTool.pricing(&pricing, &args); + let title = args.prompt.clone(); + let payload = EditorBackgroundMusicGenerateRequest { + gpt_description_prompt: args.prompt, + make_instrumental: args.make_instrumental, + project_id: Some(conversation.project_id.clone()), + canvas_completion: Some(build_editor_agent_canvas_completion( + &project, + GenerateBackgroundMusicTool::NAME, + "生成背景音乐", + )), + generation_inputs: generation_inputs("用户指令", &title), + asset_folder_id: Some("project".to_string()), + asset_label: Some("生成背景音乐".to_string()), + }; + ( + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, + "画布 Agent 生成背景音乐", + price, + serde_json::to_value(payload), + ) + } + _ => { + return Err(editor_agent_bad_request(format!( + "unsupported tool: {tool_name}" + ))); + } + }; + let payload = payload.map_err(|error| { + AppError::from_status(axum::http::StatusCode::INTERNAL_SERVER_ERROR) + .with_details(json!({"message": format!("failed to serialize tool payload: {error}")})) + })?; + let (job_id, dedupe_key) = editor_agent_tool_job_identity( + conversation.conversation_id.as_str(), + message_id, + tool_name.as_str(), + ); + let job = enqueue_editor_generation_job_with_identity( + &state, + conversation.owner_user_id.as_str(), + job_kind, + conversation.project_id.clone(), + request_label, + u64::from(price_mud_points), + &payload, + job_id, + dedupe_key, + ) + .await?; + let message = &mut document.messages[message_id]; + let tool_call = message + .tool_call + .as_mut() + .ok_or_else(|| editor_agent_bad_request("message has no tool call"))?; + tool_call.external_job_id = Some(job.job_id); + tool_call.status = EditorAgentToolCallStatus::NotCompleted; + write_messages_document(&state, &conversation, &document).await?; + Ok(json_success_body( + Some(&request_context), + json!({ "ok": true }), + )) +} + +fn parse_confirm_tool_args(value: &Value) -> Result { + serde_json::from_value(value.clone()) + .map_err(|error| editor_agent_bad_request(format!("invalid tool call args: {error}"))) +} + +fn editor_agent_tool_job_identity( + conversation_id: &str, + message_id: usize, + tool_name: &str, +) -> (String, String) { + let dedupe_key = format!("editor-agent:{conversation_id}:{message_id}:{tool_name}"); + let digest = Sha256::digest(dedupe_key.as_bytes()); + (format!("task-editor-agent-{digest:x}"), dedupe_key) +} + +async fn load_editor_agent_project( + state: &AppState, + conversation: &EditorAgentConversationRecord, +) -> Result { + state + .spacetime_client() + .get_editor_project(EditorProjectGetRecordInput { + project_id: conversation.project_id.clone(), + owner_user_id: conversation.owner_user_id.clone(), + }) + .await + .map_err(|error| { + AppError::from_status(axum::http::StatusCode::NOT_FOUND) + .with_details(json!({ "message": format!("project not found: {error}") })) + }) +} diff --git a/server-rs/crates/api-server/src/editor_agent/context.rs b/server-rs/crates/api-server/src/editor_agent/context.rs new file mode 100644 index 000000000..40d194bdc --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/context.rs @@ -0,0 +1,37 @@ +use platform_editor_agent::agent::tools::context::EditorToolContext; +use crate::editor_agent::utils::IntoDataKey; +use shared_contracts::editor_agent::EditorAgentConversationMessagesDocument; +use std::collections::HashMap; +use platform_editor_agent::agent::asset::{ImageId, ImageMetadata}; + +pub fn build_tool_context(document: &EditorAgentConversationMessagesDocument) -> EditorToolContext { + let mut images: HashMap = HashMap::new(); + + for msg in document.messages.iter().rev() { + // tool gen + if let Some(tc) = &msg.tool_call { + for img in &tc.images { + let data_key = img.clone().into_data_key(); + let image_id = ImageId::from_data_key(&data_key); + let metadata = ImageMetadata { + tag: String::new(), + data_key, + }; + images.insert(image_id, metadata); + } + } + + // user pointed + for a in &msg.attachments { + let data_key = a.clone().into_data_key(); + let image_id = ImageId::from_data_key(&data_key); + let metadata = ImageMetadata { + tag: String::new(), + data_key, + }; + images.insert(image_id, metadata); + } + } + + EditorToolContext { images } +} diff --git a/server-rs/crates/api-server/src/editor_agent/display_args.rs b/server-rs/crates/api-server/src/editor_agent/display_args.rs new file mode 100644 index 000000000..23b51a40b --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/display_args.rs @@ -0,0 +1,315 @@ +use serde_json::Value; +use platform_editor_agent::framework::error::PromptError; +use platform_editor_agent::framework::tool::Tool; +use shared_contracts::editor_agent::{EditorAgentConversationMessagesDocument, EditorAgentToolCallDisplayArgs, EditorAgentToolCallDisplayExtras, EditorAgentToolCallImageArg, EditorAgentToolCallImageRef, EditorAgentToolCallStringArg}; +use crate::editor_agent::pricing::EditorAgentPricedTool; +use platform_editor_agent::agent::tools::context::EditorToolContext; +use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; +use platform_editor_agent::agent::tools::generate_background_music::{GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs}; +use platform_editor_agent::agent::tools::generate_character::GenerateCharacterTool; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::{GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs}; +use platform_editor_agent::agent::tools::generate_image::{GenerateImageTool, GenerateImageToolArgs}; +use platform_editor_agent::agent::tools::generate_sound_effect::{GenerateSoundEffectTool, GenerateSoundEffectToolArgs}; +use platform_editor_agent::agent::tools::generate_ui_design::GenerateUiDesignTool; +use platform_editor_agent::agent::tools::generate_video::{GenerateVideoTool, GenerateVideoToolArgs}; +use platform_editor_agent::agent::asset::ImageId; +use crate::editor_agent::utils::{IntoDataKey, IntoImageId}; +use crate::editor_generation_config::EditorGenerationPricingConfig; + +pub fn build_tool_call_display_args( + tool_name: &str, + args: &Value, + document: &EditorAgentConversationMessagesDocument, + tool_context: &EditorToolContext, + pricing: &EditorGenerationPricingConfig, +) -> Result { + let mut display_args = EditorAgentToolCallDisplayArgs::default(); + + let price_mud_points = match tool_name { + EditImageTool::NAME => { + let args: EditImageToolArgs = parse_display_tool_args(tool_name, args)?; + let price_mud_points = EditImageTool { + context: tool_context.clone(), + } + .pricing(pricing, &args); + push_string_display_arg(&mut display_args, "prompt", "修改要求", args.prompt); + push_image_display_arg( + &mut display_args, + document, + "object_image_id", + "目标图片", + std::slice::from_ref(&args.object_image_id), + )?; + push_image_display_arg( + &mut display_args, + document, + "reference_image_ids", + "参考图片", + args.reference_image_ids.as_slice(), + )?; + price_mud_points + } + GenerateImageTool::NAME | GenerateCharacterTool::NAME | GenerateUiDesignTool::NAME => { + let args: GenerateImageToolArgs = parse_display_tool_args(tool_name, args)?; + let price_mud_points = match tool_name { + GenerateImageTool::NAME => GenerateImageTool { + context: tool_context.clone(), + } + .pricing(pricing, &args), + GenerateCharacterTool::NAME => GenerateCharacterTool { + context: tool_context.clone(), + } + .pricing(pricing, &args), + GenerateUiDesignTool::NAME => GenerateUiDesignTool { + context: tool_context.clone(), + } + .pricing(pricing, &args), + _ => unreachable!("tool name was matched above"), + }; + push_string_display_arg(&mut display_args, "prompt", "提示词", args.prompt); + push_optional_string_display_arg( + &mut display_args, + "aspect_ratio", + "画面比例", + args.aspect_ratio, + ); + push_optional_string_display_arg( + &mut display_args, + "image_size", + "图片尺寸", + args.image_size, + ); + push_image_display_arg( + &mut display_args, + document, + "reference_image_ids", + "参考图片", + args.reference_image_ids.as_slice(), + )?; + price_mud_points + } + GenerateIconSpritesheetTool::NAME => { + let args: GenerateIconSpritesheetToolArgs = parse_display_tool_args(tool_name, args)?; + let price_mud_points = GenerateIconSpritesheetTool { + context: tool_context.clone(), + } + .pricing(pricing, &args); + push_string_display_arg( + &mut display_args, + "icon_descriptions", + "图标描述", + args.icon_descriptions.join("\n"), + ); + push_optional_string_display_arg( + &mut display_args, + "aspect_ratio", + "图集比例", + args.aspect_ratio, + ); + push_optional_string_display_arg( + &mut display_args, + "image_size", + "图片尺寸", + args.image_size, + ); + push_image_display_arg( + &mut display_args, + document, + "reference_image_id", + "规范参考图", + std::slice::from_ref(&args.reference_image_id), + )?; + push_image_display_arg( + &mut display_args, + document, + "reference_image_ids", + "额外参考图", + args.reference_image_ids.as_slice(), + )?; + price_mud_points + } + GenerateVideoTool::NAME => { + let args: GenerateVideoToolArgs = parse_display_tool_args(tool_name, args)?; + let price_mud_points = GenerateVideoTool { + context: tool_context.clone(), + } + .pricing(pricing, &args); + push_string_display_arg(&mut display_args, "prompt", "视频描述", args.prompt); + push_optional_string_display_arg( + &mut display_args, + "aspect_ratio", + "画面比例", + args.aspect_ratio, + ); + if let Some(duration_seconds) = args.duration_seconds { + push_string_display_arg( + &mut display_args, + "duration_seconds", + "时长（秒）", + duration_seconds.to_string(), + ); + } + push_optional_string_display_arg(&mut display_args, "model", "模型", args.model); + push_optional_string_display_arg( + &mut display_args, + "resolution", + "分辨率", + args.resolution, + ); + push_optional_string_display_arg(&mut display_args, "sound", "声音", args.sound); + push_image_display_arg( + &mut display_args, + document, + "reference_image_ids", + "参考图片", + args.reference_image_ids.as_slice(), + )?; + price_mud_points + } + GenerateSoundEffectTool::NAME => { + let args: GenerateSoundEffectToolArgs = parse_display_tool_args(tool_name, args)?; + let price_mud_points = GenerateSoundEffectTool.pricing(pricing, &args); + push_string_display_arg(&mut display_args, "prompt", "音效描述", args.prompt); + if let Some(duration) = args.duration { + push_string_display_arg( + &mut display_args, + "duration", + "时长（秒）", + duration.to_string(), + ); + } + price_mud_points + } + GenerateBackgroundMusicTool::NAME => { + let args: GenerateBackgroundMusicToolArgs = parse_display_tool_args(tool_name, args)?; + let price_mud_points = GenerateBackgroundMusicTool.pricing(pricing, &args); + push_string_display_arg(&mut display_args, "prompt", "音乐描述", args.prompt); + push_string_display_arg( + &mut display_args, + "make_instrumental", + "纯音乐", + if args.make_instrumental { "是" } else { "否" }, + ); + price_mud_points + } + _ => { + return Err(PromptError::InternalError(format!( + "missing display argument mapping for tool {tool_name}" + ))); + } + }; + + display_args.extras = EditorAgentToolCallDisplayExtras { price_mud_points }; + + Ok(display_args) +} + +fn parse_display_tool_args(tool_name: &str, args: &Value) -> Result +where + T: serde::de::DeserializeOwned, +{ + serde_json::from_value(args.clone()).map_err(|error| { + PromptError::InternalError(format!( + "failed to build display arguments for {tool_name}: {error}" + )) + }) +} + +fn push_string_display_arg( + display_args: &mut EditorAgentToolCallDisplayArgs, + name: &str, + label: &str, + value: impl Into, +) { + let value = value.into(); + if value.trim().is_empty() { + return; + } + display_args.string_args.push(EditorAgentToolCallStringArg { + name: name.to_string(), + label: label.to_string(), + value, + }); +} + +fn push_optional_string_display_arg( + display_args: &mut EditorAgentToolCallDisplayArgs, + name: &str, + label: &str, + value: Option, +) { + if let Some(value) = value { + push_string_display_arg(display_args, name, label, value); + } +} + +fn push_image_display_arg( + display_args: &mut EditorAgentToolCallDisplayArgs, + document: &EditorAgentConversationMessagesDocument, + name: &str, + label: &str, + image_ids: &[ImageId], +) -> Result<(), PromptError> { + if image_ids.is_empty() { + return Ok(()); + } + + let refs = image_ids + .iter() + .map(|image_id| { + resolve_tool_call_image_ref(document, image_id).ok_or_else(|| { + PromptError::InternalError(format!( + "display image reference {} is missing from conversation context", + image_id.id + )) + }) + }) + .collect::, _>>()?; + + display_args.image_args.push(EditorAgentToolCallImageArg { + name: name.to_string(), + label: label.to_string(), + refs, + }); + Ok(()) +} + +pub fn resolve_tool_call_image_ref( + document: &EditorAgentConversationMessagesDocument, + image_id: &ImageId, +) -> Option { + for message in document.messages.iter().rev() { + for attachment in message.attachments.iter().rev() { + if attachment.clone().into_image_id() == *image_id { + return Some(EditorAgentToolCallImageRef { + image_id: image_id.id.clone(), + image_src: attachment.image_src.clone(), + object_key: attachment.object_key.clone(), + thumbnail_src: attachment.thumbnail_src.clone(), + label: attachment.label.clone(), + width: attachment.width, + height: attachment.height, + }); + } + } + + if let Some(tool_call) = &message.tool_call { + for image in tool_call.images.iter().rev() { + let data_key = image.clone().into_data_key(); + if ImageId::from_data_key(data_key) == *image_id { + return Some(EditorAgentToolCallImageRef { + image_id: image_id.id.clone(), + image_src: image.image_src.clone(), + object_key: image.object_key.clone(), + thumbnail_src: image.thumbnail_src.clone(), + label: None, + width: image.width, + height: image.height, + }); + } + } + } + } + + None +} diff --git a/server-rs/crates/api-server/src/editor_agent/mod.rs b/server-rs/crates/api-server/src/editor_agent/mod.rs new file mode 100644 index 000000000..ecbc880e9 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/mod.rs @@ -0,0 +1,13 @@ +mod utils; +pub mod api; +mod display_args; +mod context; +mod resp_to_asset; +mod reconcile; +pub mod pricing; + +pub use api::{ + cancel_editor_agent_tool_call, confirm_editor_agent_tool_call, + create_editor_agent_conversation, delete_editor_agent_conversation, + get_editor_agent_conversation, list_editor_agent_conversations, +}; diff --git a/server-rs/crates/api-server/src/editor_agent/pricing.rs b/server-rs/crates/api-server/src/editor_agent/pricing.rs new file mode 100644 index 000000000..0ddd88796 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/pricing.rs @@ -0,0 +1,315 @@ +use crate::editor_generation_config::{ + EditorGenerationPricingConfig, +}; +#[cfg(test)] +use crate::editor_generation_config::load_editor_generation_pricing_from_paths; +#[cfg(test)] +use platform_editor_agent::agent::asset::ImageId; +#[cfg(test)] +use platform_editor_agent::agent::tools::context::EditorToolContext; +use platform_editor_agent::agent::tools::edit_image::{EditImageTool, EditImageToolArgs}; +use platform_editor_agent::agent::tools::generate_background_music::{ + GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, +}; +use platform_editor_agent::agent::tools::generate_character::GenerateCharacterTool; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ + GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, +}; +use platform_editor_agent::agent::tools::generate_image::{ + GenerateImageTool, GenerateImageToolArgs, +}; +use platform_editor_agent::agent::tools::generate_sound_effect::{ + GenerateSoundEffectTool, GenerateSoundEffectToolArgs, +}; +use platform_editor_agent::agent::tools::generate_ui_design::GenerateUiDesignTool; +use platform_editor_agent::agent::tools::generate_video::{ + GenerateVideoTool, GenerateVideoToolArgs, +}; +use platform_editor_agent::framework::tool::Tool; +use platform_image::GPT_IMAGE_2_MODEL; +#[cfg(test)] +use std::collections::HashMap; + +#[cfg(test)] +fn context() -> EditorToolContext { + EditorToolContext { + images: HashMap::new(), + } +} + +#[cfg(test)] +fn image_args(image_size: Option<&str>) -> GenerateImageToolArgs { + GenerateImageToolArgs { + prompt: "生成图片".to_string(), + reference_image_ids: Vec::new(), + aspect_ratio: Some("1:1".to_string()), + image_size: image_size.map(ToOwned::to_owned), + } +} + +#[test] +fn every_editor_agent_tool_exposes_argument_based_pricing() { + let pricing = load_editor_generation_pricing_from_paths(None) + .expect("default editor pricing should load"); + let context = context(); + + assert_eq!( + EditImageTool { + context: context.clone(), + } + .pricing( + &pricing, + &EditImageToolArgs { + object_image_id: ImageId { + id: "image-1".to_string(), + }, + reference_image_ids: Vec::new(), + prompt: "改成蓝色".to_string(), + }, + ), + 3 + ); + assert_eq!( + GenerateImageTool { + context: context.clone(), + } + .pricing(&pricing, &image_args(Some("2K"))), + 5 + ); + assert_eq!( + GenerateCharacterTool { + context: context.clone(), + } + .pricing(&pricing, &image_args(None)), + 3 + ); + assert_eq!( + GenerateUiDesignTool { + context: context.clone(), + } + .pricing(&pricing, &image_args(Some("2K"))), + 5 + ); + assert_eq!( + GenerateIconSpritesheetTool { + context: context.clone(), + } + .pricing( + &pricing, + &GenerateIconSpritesheetToolArgs { + reference_image_id: ImageId { + id: "image-1".to_string(), + }, + reference_image_ids: Vec::new(), + icon_descriptions: vec!["背包".to_string(), "地图".to_string()], + aspect_ratio: Some("1:1".to_string()), + image_size: Some("2K".to_string()), + }, + ), + 5 + ); + assert_eq!( + GenerateVideoTool { + context: context.clone(), + } + .pricing( + &pricing, + &GenerateVideoToolArgs { + prompt: "镜头缓慢推进".to_string(), + reference_image_ids: Vec::new(), + aspect_ratio: None, + duration_seconds: Some(6), + model: Some("seedance2.0".to_string()), + resolution: Some("720p".to_string()), + sound: None, + }, + ), + 144 + ); + assert_eq!( + GenerateSoundEffectTool.pricing( + &pricing, + &GenerateSoundEffectToolArgs { + prompt: "按钮点击声".to_string(), + duration: None, + }, + ), + 5 + ); + assert_eq!( + GenerateBackgroundMusicTool.pricing( + &pricing, + &GenerateBackgroundMusicToolArgs { + prompt: "轻松背景音乐".to_string(), + make_instrumental: true, + }, + ), + 12 + ); +} + +#[test] +fn pricing_uses_the_supplied_runtime_snapshot() { + let mut pricing = load_editor_generation_pricing_from_paths(None) + .expect("default editor pricing should load"); + pricing + .models + .get_mut("gpt-image-2") + .expect("gpt image pricing should exist") + .prices + .insert("2K".to_string(), 37); + pricing + .models + .get_mut("seedance2.0-fast") + .expect("video pricing should exist") + .prices + .insert("720p".to_string(), 7); + pricing + .models + .get_mut("audio1.0") + .expect("sound pricing should exist") + .price = Some(19); + + assert_eq!( + GenerateImageTool { context: context() }.pricing(&pricing, &image_args(Some("2K"))), + 37 + ); + assert_eq!( + GenerateVideoTool { context: context() }.pricing( + &pricing, + &GenerateVideoToolArgs { + prompt: "four seconds".to_string(), + reference_image_ids: Vec::new(), + aspect_ratio: None, + duration_seconds: None, + model: None, + resolution: None, + sound: None, + }, + ), + 28 + ); + assert_eq!( + GenerateSoundEffectTool.pricing( + &pricing, + &GenerateSoundEffectToolArgs { + prompt: "sound".to_string(), + duration: None, + }, + ), + 19 + ); +} + +impl EditorAgentPricedTool for GenerateVideoTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &GenerateVideoToolArgs, + ) -> u32 { + let model = args + .model + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .unwrap_or(Self::DEFAULT_VIDEO_MODEL); + let resolution = args + .resolution + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .unwrap_or(Self::DEFAULT_VIDEO_RESOLUTION); + let duration_seconds = args + .duration_seconds + .unwrap_or(Self::DEFAULT_VIDEO_DURATION_SECONDS); + pricing.video_model_mud_points(Some(model), resolution, duration_seconds) + } +} + +impl EditorAgentPricedTool for GenerateUiDesignTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &GenerateImageToolArgs, + ) -> u32 { + editor_agent_image_mud_points(pricing, Some("ui-design"), args.image_size.as_deref()) + } +} + +impl EditorAgentPricedTool for GenerateSoundEffectTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + _args: &GenerateSoundEffectToolArgs, + ) -> u32 { + pricing.sound_effect_model_mud_points(None) + } +} + +impl EditorAgentPricedTool for GenerateImageTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &GenerateImageToolArgs, + ) -> u32 { + editor_agent_image_mud_points(pricing, None, args.image_size.as_deref()) + } +} + +impl EditorAgentPricedTool for GenerateIconSpritesheetTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &GenerateIconSpritesheetToolArgs, + ) -> u32 { + editor_agent_image_mud_points(pricing, Some("icon"), args.image_size.as_deref()) + } +} + +impl EditorAgentPricedTool for GenerateCharacterTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + args: &GenerateImageToolArgs, + ) -> u32 { + editor_agent_image_mud_points(pricing, Some("character"), args.image_size.as_deref()) + } +} + +/// api-server 侧的画布 Agent 工具计价扩展。 +/// +/// 通用 `Tool` 仍只负责参数校验；价格依赖 api-server 的运行时配置，不能下沉到 +/// `module-editor-agent`。实际执行和扣费仍由既有生成 BFF 负责。 +pub(crate) trait EditorAgentPricedTool: Tool { + fn pricing(&self, pricing: &EditorGenerationPricingConfig, args: &::Args) -> u32; +} + +pub(crate) fn editor_agent_image_mud_points( + pricing: &EditorGenerationPricingConfig, + kind: Option<&str>, + image_size: Option<&str>, +) -> u32 { + // 这些 Agent 工具当前向既有 BFF 传 model=None；BFF 会先归一为 gpt-image-2。 + // 尺寸同样只把精确的 2K 识别为 2K，其余值回落到 1K。 + let normalized_image_size = match image_size.map(str::trim) { + Some("2K") => "2K", + _ => "1K", + }; + pricing.image_generation_mud_points(kind, Some(GPT_IMAGE_2_MODEL), Some(normalized_image_size)) +} + +impl EditorAgentPricedTool for GenerateBackgroundMusicTool { + fn pricing( + &self, + pricing: &EditorGenerationPricingConfig, + _args: &GenerateBackgroundMusicToolArgs, + ) -> u32 { + pricing.background_music_model_mud_points(None) + } +} + +impl EditorAgentPricedTool for EditImageTool { + fn pricing(&self, pricing: &EditorGenerationPricingConfig, _args: &EditImageToolArgs) -> u32 { + editor_agent_image_mud_points(pricing, Some("quick-edit"), Some("1K")) + } +} diff --git a/server-rs/crates/api-server/src/editor_agent/reconcile.rs b/server-rs/crates/api-server/src/editor_agent/reconcile.rs new file mode 100644 index 000000000..20d21a02f --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/reconcile.rs @@ -0,0 +1,369 @@ +use crate::editor_agent::resp_to_asset; +use crate::http_error::AppError; +use crate::state::AppState; +use platform_editor_agent::agent::tools::context::EditorToolContext; +use platform_editor_agent::agent::tools::edit_image::{ + EditImageTool, EditImageToolArgs, EditorImageEditResult, +}; +use platform_editor_agent::agent::tools::generate_background_music::{ + GenerateBackgroundMusicTool, GenerateBackgroundMusicToolArgs, +}; +use platform_editor_agent::agent::tools::generate_character::GenerateCharacterTool; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::{ + EditorIconSpritesheetResult, GenerateIconSpritesheetTool, GenerateIconSpritesheetToolArgs, +}; +use platform_editor_agent::agent::tools::generate_image::{ + EditorImageGenerationResult, GenerateImageTool, GenerateImageToolArgs, +}; +use platform_editor_agent::agent::tools::generate_sound_effect::{ + GenerateSoundEffectTool, GenerateSoundEffectToolArgs, +}; +use platform_editor_agent::agent::tools::generate_ui_design::GenerateUiDesignTool; +use platform_editor_agent::agent::tools::generate_video::{ + GenerateVideoTool, GenerateVideoToolArgs, +}; +use platform_editor_agent::framework::tool::Tool; +use serde_json::Value; +use shared_contracts::assets::{EditorAudioGenerateResponse, EditorVideoGenerateResponse}; +use shared_contracts::editor_agent::{ + EditorAgentConversationMessagesDocument, EditorAgentMessage, EditorAgentToolCallStatus, +}; +use spacetime_client::{EditorAgentConversationRecord, ExternalGenerationJobGetRecordInput}; +use tokio::time::{Duration, sleep}; + +pub(crate) const EDITOR_AGENT_RESULT_RECONCILE_MAX_RETRIES: u8 = 3; +const EDITOR_AGENT_RESULT_RECONCILE_RETRY_DELAY: Duration = Duration::from_millis(100); + +#[derive(Debug, PartialEq)] +enum ResultReconcileFailure { + RetryLater(String), + Fatal(String), +} + +impl From for ResultReconcileFailure { + fn from(error: String) -> Self { + Self::Fatal(error) + } +} + +pub async fn reconcile_editor_agent_tool_calls( + state: &AppState, + conversation: &EditorAgentConversationRecord, + document: &mut EditorAgentConversationMessagesDocument, +) -> Result, AppError> { + let candidates = document + .messages + .iter() + .enumerate() + .filter_map(|(message_index, message)| { + let tool_call = message.tool_call.as_ref()?; + if tool_call.status != EditorAgentToolCallStatus::NotCompleted { + return None; + } + Some((message_index, tool_call.external_job_id.clone()?)) + }) + .collect::>(); + let mut reconciled = Vec::new(); + + for (message_index, job_id) in candidates { + let mut last_reconcile_error = None; + + for retry_count in 0..=EDITOR_AGENT_RESULT_RECONCILE_MAX_RETRIES { + let job = match state + .spacetime_client() + .get_external_generation_job_generated_artifacts( + ExternalGenerationJobGetRecordInput { + job_id: job_id.clone(), + owner_user_id: conversation.owner_user_id.clone(), + }, + ) + .await + { + Ok(job) => job, + Err(error) => { + last_reconcile_error = Some(ResultReconcileFailure::RetryLater(format!( + "读取生成任务结果失败：{error}" + ))); + if should_retry_result_reconcile(retry_count) { + sleep(EDITOR_AGENT_RESULT_RECONCILE_RETRY_DELAY).await; + } + continue; + } + }; + + match job.status.as_str() { + "completed" => { + let original_message = document.messages[message_index].clone(); + match reconcile_completed_editor_agent_tool_call( + &mut document.messages[message_index], + job.result_payload_json.as_deref(), + ) { + Ok(()) => { + last_reconcile_error = None; + reconciled.push(document.messages[message_index].clone()); + break; + } + Err(error) => { + document.messages[message_index] = original_message; + last_reconcile_error = Some(error); + } + } + } + "failed" => { + mark_job_failed( + &mut document.messages[message_index], + job.last_error_message + .unwrap_or_else(|| "生成失败".to_string()), + ); + last_reconcile_error = None; + reconciled.push(document.messages[message_index].clone()); + break; + } + _ => { + last_reconcile_error = None; + break; + } + } + + if should_retry_result_reconcile(retry_count) { + sleep(EDITOR_AGENT_RESULT_RECONCILE_RETRY_DELAY).await; + } + } + + if let Some(error) = last_reconcile_error + && apply_exhausted_result_reconcile_failure( + &mut document.messages[message_index], + error, + ) + { + reconciled.push(document.messages[message_index].clone()); + } + } + + Ok(reconciled) +} + +fn should_retry_result_reconcile(retry_count: u8) -> bool { + retry_count < EDITOR_AGENT_RESULT_RECONCILE_MAX_RETRIES +} + +fn mark_job_failed(message: &mut EditorAgentMessage, error: String) { + let tool_call = message + .tool_call + .as_mut() + .expect("reconcile candidate contains a tool call"); + tool_call.error = Some(error.clone()); + tool_call.status = EditorAgentToolCallStatus::Failed; + message.text = format!("[tool_call:{}] output: {error}", tool_call.tool_name); +} + +fn mark_fatal_result_reconcile_failed(message: &mut EditorAgentMessage, error: String) { + mark_job_failed( + message, + format!( + "任务终态结果回填重试 {EDITOR_AGENT_RESULT_RECONCILE_MAX_RETRIES} 次后仍失败：{error}" + ), + ); +} + +fn apply_exhausted_result_reconcile_failure( + message: &mut EditorAgentMessage, + failure: ResultReconcileFailure, +) -> bool { + match failure { + ResultReconcileFailure::RetryLater(_) => false, + ResultReconcileFailure::Fatal(error) => { + mark_fatal_result_reconcile_failed(message, error); + true + } + } +} + +fn reconcile_completed_editor_agent_tool_call( + message: &mut EditorAgentMessage, + result_payload_json: Option<&str>, +) -> Result<(), ResultReconcileFailure> { + let response = result_payload_json + .ok_or_else(|| ResultReconcileFailure::RetryLater("生成任务缺少结果数据".to_string())) + .and_then(|payload| { + serde_json::from_str::(payload) + .map_err(|error| ResultReconcileFailure::Fatal(error.to_string())) + })? + // TODO extract const + .get("editor-agent-tool-call-result") + .cloned() + .ok_or_else(|| { + ResultReconcileFailure::Fatal("生成任务缺少 Agent 工具调用结果".to_string()) + })?; + let tool_call = message + .tool_call + .as_mut() + .ok_or_else(|| ResultReconcileFailure::Fatal("消息缺少工具调用".to_string()))?; + let tool_name = tool_call.tool_name.clone(); + + match tool_name.as_str() { + GenerateImageTool::NAME | GenerateCharacterTool::NAME | GenerateUiDesignTool::NAME => { + let args: GenerateImageToolArgs = parse_reconciled_value(&tool_call.args)?; + let result: EditorImageGenerationResult = parse_reconciled_value(&response)?; + message.text = match tool_name.as_str() { + GenerateCharacterTool::NAME => GenerateCharacterTool { + context: EditorToolContext::default(), + } + .format_execute_message(&args, &result), + GenerateUiDesignTool::NAME => GenerateUiDesignTool { + context: EditorToolContext::default(), + } + .format_execute_message(&args, &result), + _ => GenerateImageTool { + context: EditorToolContext::default(), + } + .format_execute_message(&args, &result), + }; + tool_call.images = vec![resp_to_asset::editor_agent_generated_image(&result)]; + } + EditImageTool::NAME => { + let args: EditImageToolArgs = parse_reconciled_value(&tool_call.args)?; + let result: EditorImageEditResult = parse_reconciled_value(&response)?; + tool_call.images = vec![resp_to_asset::editor_agent_generated_edited_image(&result)]; + message.text = EditImageTool { + context: EditorToolContext::default(), + } + .format_execute_message(args, result); + } + GenerateIconSpritesheetTool::NAME => { + let args: GenerateIconSpritesheetToolArgs = parse_reconciled_value(&tool_call.args)?; + let result: EditorIconSpritesheetResult = parse_reconciled_value(&response)?; + message.text = GenerateIconSpritesheetTool { + context: EditorToolContext::default(), + } + .format_execute_message(&args, &result); + tool_call.images = resp_to_asset::editor_agent_generated_spritesheet_images(&result); + } + GenerateVideoTool::NAME => { + let args: GenerateVideoToolArgs = parse_reconciled_value(&tool_call.args)?; + let result: EditorVideoGenerateResponse = parse_reconciled_value(&response)?; + message.text = GenerateVideoTool { + context: EditorToolContext::default(), + } + .format_execute_message(&args, &result); + tool_call.videos = vec![resp_to_asset::editor_agent_generated_video(&result)]; + } + GenerateSoundEffectTool::NAME | GenerateBackgroundMusicTool::NAME => { + let result: EditorAudioGenerateResponse = parse_reconciled_value(&response)?; + message.text = if tool_name == GenerateSoundEffectTool::NAME { + let args: GenerateSoundEffectToolArgs = parse_reconciled_value(&tool_call.args)?; + GenerateSoundEffectTool.format_execute_message(&args, &result) + } else { + let args: GenerateBackgroundMusicToolArgs = + parse_reconciled_value(&tool_call.args)?; + GenerateBackgroundMusicTool.format_execute_message(&args, &result) + }; + tool_call.audios = vec![resp_to_asset::editor_agent_generated_audio(&result)]; + } + _ => { + return Err(ResultReconcileFailure::Fatal(format!( + "不支持回填工具 {tool_name}" + ))); + } + } + tool_call.error = None; + tool_call.status = EditorAgentToolCallStatus::Completed; + Ok(()) +} + +fn parse_reconciled_value(value: &Value) -> Result { + serde_json::from_value(value.clone()).map_err(|error| error.to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn pending_tool_message() -> EditorAgentMessage { + serde_json::from_value(json!({ + "id": 1, + "role": "system", + "text": "waiting", + "attachments": [], + "toolCall": { + "toolName": "generate_image", + "status": "not_completed", + "args": {}, + "displayArgs": { + "stringArgs": [], + "imageArgs": [], + "extras": { "priceMudPoints": 0 } + }, + "externalJobId": "job-1", + "images": [] + }, + "createdAt": "2026-07-16T00:00:00Z" + })) + .expect("pending tool message should deserialize") + } + + #[test] + fn terminal_result_reconcile_retries_three_times_after_the_initial_attempt() { + assert!(should_retry_result_reconcile(0)); + assert!(should_retry_result_reconcile(1)); + assert!(should_retry_result_reconcile(2)); + assert!(!should_retry_result_reconcile(3)); + } + + #[test] + fn exhausted_fatal_result_reconcile_marks_the_tool_call_failed() { + let mut message = pending_tool_message(); + + let changed = apply_exhausted_result_reconcile_failure( + &mut message, + ResultReconcileFailure::Fatal("invalid result payload".to_string()), + ); + + assert!(changed); + let tool_call = message.tool_call.expect("tool call should remain present"); + assert_eq!(tool_call.status, EditorAgentToolCallStatus::Failed); + assert_eq!( + tool_call.error.as_deref(), + Some("任务终态结果回填重试 3 次后仍失败：invalid result payload") + ); + assert!(message.text.contains("重试 3 次后仍失败")); + } + + #[test] + fn missing_result_payload_is_retryable_and_preserves_pending_job_reference() { + let mut message = pending_tool_message(); + + let error = reconcile_completed_editor_agent_tool_call(&mut message, None) + .expect_err("missing result payload should be retried later"); + + assert_eq!( + error, + ResultReconcileFailure::RetryLater("生成任务缺少结果数据".to_string()) + ); + assert!(!apply_exhausted_result_reconcile_failure( + &mut message, + error, + )); + let tool_call = message.tool_call.expect("tool call should remain present"); + assert_eq!(tool_call.status, EditorAgentToolCallStatus::NotCompleted); + assert_eq!(tool_call.external_job_id.as_deref(), Some("job-1")); + assert_eq!(tool_call.error, None); + } + + #[test] + fn read_failure_is_retryable_without_terminal_message_mutation() { + let mut message = pending_tool_message(); + let error = ResultReconcileFailure::RetryLater( + "读取生成任务结果失败：temporarily unavailable".to_string(), + ); + + assert!(!apply_exhausted_result_reconcile_failure( + &mut message, + error, + )); + let tool_call = message.tool_call.expect("tool call should remain present"); + assert_eq!(tool_call.status, EditorAgentToolCallStatus::NotCompleted); + assert_eq!(tool_call.external_job_id.as_deref(), Some("job-1")); + } +} diff --git a/server-rs/crates/api-server/src/editor_agent/resp_to_asset.rs b/server-rs/crates/api-server/src/editor_agent/resp_to_asset.rs new file mode 100644 index 000000000..014a7d159 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/resp_to_asset.rs @@ -0,0 +1,105 @@ +use platform_editor_agent::agent::tools::edit_image::EditorImageEditResult; +use platform_editor_agent::agent::tools::generate_icon_spritesheet::EditorIconSpritesheetResult; +use platform_editor_agent::agent::tools::generate_image::EditorImageGenerationResult; +use serde_json::Value; +use shared_contracts::assets::{EditorAudioGenerateResponse, EditorVideoGenerateResponse}; +use shared_contracts::editor_agent::{ + EditorAgentGeneratedAudio, EditorAgentGeneratedImage, EditorAgentGeneratedVideo, +}; + +fn value_string(value: Option<&Value>, field: &str) -> Option { + value + .and_then(|resource| resource.get(field)) + .and_then(Value::as_str) + .map(ToOwned::to_owned) +} + +pub fn editor_agent_generated_image( + result: &EditorImageGenerationResult, +) -> EditorAgentGeneratedImage { + EditorAgentGeneratedImage { + resource_id: value_string(result.resource.as_ref(), "resourceId"), + object_key: result.object_key.clone(), + asset_object_id: result.asset_object_id.clone(), + image_src: result.image_src.clone(), + thumbnail_src: None, + width: Some(result.width), + height: Some(result.height), + } +} + +pub fn editor_agent_generated_edited_image( + result: &EditorImageEditResult, +) -> EditorAgentGeneratedImage { + EditorAgentGeneratedImage { + resource_id: value_string(result.resource.as_ref(), "resourceId"), + object_key: result.object_key.clone(), + asset_object_id: result.asset_object_id.clone(), + image_src: result.image_src.clone(), + thumbnail_src: None, + width: Some(result.width), + height: Some(result.height), + } +} + +pub fn editor_agent_generated_spritesheet( + result: &EditorIconSpritesheetResult, +) -> EditorAgentGeneratedImage { + EditorAgentGeneratedImage { + resource_id: value_string(result.spritesheet_resource.as_ref(), "resourceId"), + object_key: value_string(result.spritesheet_resource.as_ref(), "objectKey"), + asset_object_id: value_string(result.spritesheet_resource.as_ref(), "assetObjectId"), + image_src: result.spritesheet_image_src.clone(), + thumbnail_src: None, + width: Some(result.spritesheet_width), + height: Some(result.spritesheet_height), + } +} + +pub fn editor_agent_generated_spritesheet_images( + result: &EditorIconSpritesheetResult, +) -> Vec { + let images = result + .icon_image_srcs + .iter() + .map(|image| EditorAgentGeneratedImage { + resource_id: value_string(image.resource.as_ref(), "resourceId"), + object_key: value_string(image.resource.as_ref(), "objectKey"), + asset_object_id: value_string(image.resource.as_ref(), "assetObjectId"), + image_src: image.image_src.clone(), + thumbnail_src: None, + width: Some(image.width), + height: Some(image.height), + }) + .collect::>(); + if images.is_empty() { + vec![editor_agent_generated_spritesheet(result)] + } else { + images + } +} + +pub fn editor_agent_generated_video( + result: &EditorVideoGenerateResponse, +) -> EditorAgentGeneratedVideo { + EditorAgentGeneratedVideo { + resource_id: value_string(result.resource.as_ref(), "resourceId"), + object_key: result.object_key.clone(), + asset_object_id: result.asset_object_id.clone(), + video_src: result.video_src.clone(), + thumbnail_src: result.thumbnail_src.clone(), + width: Some(result.width), + height: Some(result.height), + } +} + +pub fn editor_agent_generated_audio( + result: &EditorAudioGenerateResponse, +) -> EditorAgentGeneratedAudio { + EditorAgentGeneratedAudio { + resource_id: value_string(result.resource.as_ref(), "resourceId"), + object_key: result.object_key.clone(), + asset_object_id: result.asset_object_id.clone(), + audio_src: result.audio_src.clone(), + } +} diff --git a/server-rs/crates/api-server/src/editor_agent/utils.rs b/server-rs/crates/api-server/src/editor_agent/utils.rs new file mode 100644 index 000000000..8fbcacff9 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_agent/utils.rs @@ -0,0 +1,613 @@ +use crate::editor_project::{current_utc_micros, map_editor_project_error}; +use crate::http_error::AppError; +use crate::platform_errors::map_oss_error; +use crate::state::AppState; +use axum::http::StatusCode; +use platform_oss::{ + LegacyAssetPrefix, OssObjectAccess, OssPutObjectRequest, OssSignedGetObjectUrlRequest, +}; +use serde_json::{json, Value}; +use shared_contracts::assets::{ + EditorCanvasGenerationCompletionPayload, EditorCanvasGenerationPlaceholderPayload, +}; +use shared_contracts::editor_agent::{ + EditorAgentAttachmentRef, EditorAgentAttachmentSource, EditorAgentConversationDetail, + EditorAgentConversationMessagesDocument, EditorAgentConversationSummary, EditorAgentGeneratedImage, + EditorAgentMessage, EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION, +}; +use shared_kernel::{normalize_optional_string, normalize_required_string}; +use spacetime_client::{ + EditorAgentConversationRecord, EditorAssetLibraryRecord, EditorAssetRecord, + EditorProjectGetRecordInput, EditorProjectRecord, EditorProjectResourceRecord, +}; +use std::collections::BTreeMap; +use std::sync::{Arc, Mutex, OnceLock}; +use platform_editor_agent::agent::asset::ImageId; + +pub trait IntoDataKey { + fn into_data_key(self) -> String; +} +impl IntoDataKey for EditorAgentAttachmentRef { + fn into_data_key(self) -> String { + self.object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + // .or_else(|| normalize_required_string(self.reference_id.as_str())) + .unwrap_or_else(|| self.image_src.clone()) + } +} +impl IntoDataKey for EditorAgentGeneratedImage { + fn into_data_key(self) -> String { + self.object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or(self.image_src) + } +} +impl IntoDataKey for EditorProjectResourceRecord { + fn into_data_key(self) -> String { + self.object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| self.image_src.clone()) + } +} + +impl IntoDataKey for EditorAssetRecord { + fn into_data_key(self) -> String { + self.object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| self.image_src.clone()) + } +} + +pub trait IntoImageId { + fn into_image_id(self) -> ImageId; +} +impl IntoImageId for EditorAgentAttachmentRef { + fn into_image_id(self) -> ImageId { + ImageId::from_data_key(self.into_data_key()) + } +} +impl IntoImageId for EditorProjectResourceRecord { + fn into_image_id(self) -> ImageId { + ImageId::from_data_key(self.into_data_key()) + } +} +impl IntoImageId for EditorAssetRecord { + fn into_image_id(self) -> ImageId { + ImageId::from_data_key(self.into_data_key()) + } +} + +// TODO resource id is not traced by now +// pub trait IntoResourceId { +// fn into_resource_id(self) -> String; +// } +// +// impl IntoResourceId for EditorProjectResourceRecord { +// fn into_resource_id(self) -> String { +// self.resource_id +// } +// } +// +// impl IntoResourceId for EditorAssetRecord { +// fn into_resource_id(self) -> String { +// self.asset_id +// } +// } + +#[cfg(test)] +mod image_id_tests { + use platform_editor_agent::agent::asset::ImageId; + + #[test] + fn image_id_is_a_stable_hash_of_the_data_key() { + let data_key = "editor-projects/proj-1/image.png"; + let image_id = ImageId::from_data_key(data_key); + + assert_eq!(image_id, ImageId::from_data_key(data_key)); + assert_ne!(image_id, ImageId::from_data_key("another-image.png")); + assert!(image_id.id.starts_with("sha256:")); + assert!(!image_id.id.contains(data_key)); + } +} + +type EditorAgentConversationLockMap = Mutex>>>; +static EDITOR_AGENT_CONVERSATION_LOCKS: OnceLock = OnceLock::new(); + +pub fn editor_agent_conversation_lock(conversation_id: &str) -> Arc> { + let locks = EDITOR_AGENT_CONVERSATION_LOCKS.get_or_init(|| Mutex::new(BTreeMap::new())); + let mut locks = locks + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + locks + .entry(conversation_id.to_string()) + .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(()))) + .clone() +} +pub fn now_rfc3339() -> String { + shared_kernel::format_rfc3339(time::OffsetDateTime::now_utc()) + .unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_string()) +} + +const EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES: usize = 2 * 1024 * 1024; +pub async fn require_editor_agent_sidebar_enabled( + state: &AppState, + owner_user_id: &str, +) -> Result<(), AppError> { + match state + .is_image_editor_agent_sidebar_enabled_for_user(Some(owner_user_id)) + .await + { + Ok(true) => Ok(()), + Ok(false) => Err(editor_agent_sidebar_unavailable()), + Err(error) => Err(AppError::from_status(StatusCode::BAD_GATEWAY) + .with_message("读取画布 Agent 灰度配置失败") + .with_details(json!({ + "provider": "spacetimedb", + "message": error.to_string(), + }))), + } +} + +fn editor_agent_sidebar_unavailable() -> AppError { + AppError::from_status(StatusCode::SERVICE_UNAVAILABLE) + .with_message("画布 Agent 暂不可用") + .with_details(json!({ + "provider": "editor-agent", + "reason": "image_editor_agent_sidebar_disabled", + "gateKey": module_runtime::IMAGE_EDITOR_AGENT_SIDEBAR_GATE_KEY, + })) +} +// const EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES: usize = 2 * 1024 * 1024; +pub fn editor_agent_bad_request(message: impl Into) -> AppError { + AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ + "provider": "editor-agent", + "message": message.into(), + })) +} + +fn editor_agent_oss_unavailable() -> AppError { + AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({ + "provider": "aliyun-oss", + "reason": "OSS is not configured for editor agent conversations", + })) +} + +fn editor_agent_oss_read_error(message: impl Into) -> AppError { + AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({ + "provider": "aliyun-oss", + "message": message.into(), + })) +} + +fn editor_agent_messages_document_too_large() -> AppError { + AppError::from_status(StatusCode::PAYLOAD_TOO_LARGE).with_details(json!({ + "provider": "editor-agent", + "message": "message document is too large", + "maxBytes": EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES, + })) +} +pub async fn write_messages_document( + state: &AppState, + conversation: &EditorAgentConversationRecord, + document: &EditorAgentConversationMessagesDocument, +) -> Result<(), AppError> { + if document.conversation_id != conversation.conversation_id { + return Err( + AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ + "provider": "editor-agent", + "message": "message document conversationId does not match metadata", + })), + ); + } + let body = serde_json::to_vec(document).map_err(|error| { + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ + "provider": "editor-agent", + "message": format!("failed to serialize message document: {error}"), + })) + })?; + if body.len() > EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES { + return Err(editor_agent_messages_document_too_large()); + } + + let oss_client = state + .oss_client() + .ok_or_else(editor_agent_oss_unavailable)?; + let put_result = oss_client + .put_object( + &reqwest::Client::new(), + OssPutObjectRequest { + prefix: LegacyAssetPrefix::EditorAgent, + path_segments: Vec::new(), + file_name: format!("{}.json", conversation.conversation_id), + content_type: Some("application/json; charset=utf-8".to_string()), + access: OssObjectAccess::Private, + metadata: BTreeMap::from([ + ( + "conversation-id".to_string(), + conversation.conversation_id.clone(), + ), + ("project-id".to_string(), conversation.project_id.clone()), + ( + "owner-user-id".to_string(), + conversation.owner_user_id.clone(), + ), + ]), + body, + }, + ) + .await + .map_err(|error| map_oss_error(error, "aliyun-oss"))?; + if put_result.object_key != conversation.messages_object_key { + return Err( + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ + "provider": "editor-agent", + "message": "OSS object key mismatch while writing message document", + "expectedObjectKey": conversation.messages_object_key, + "actualObjectKey": put_result.object_key, + })), + ); + } + Ok(()) +} + +const EDITOR_AGENT_MESSAGES_READ_EXPIRE_SECONDS: u64 = 60; + +pub async fn read_messages_document( + state: &AppState, + conversation: &EditorAgentConversationRecord, +) -> Result { + let oss_client = state + .oss_client() + .ok_or_else(editor_agent_oss_unavailable)?; + let signed = oss_client + .sign_internal_get_object_url(OssSignedGetObjectUrlRequest { + object_key: conversation.messages_object_key.clone(), + expire_seconds: Some(EDITOR_AGENT_MESSAGES_READ_EXPIRE_SECONDS), + }) + .map_err(|error| map_oss_error(error, "aliyun-oss"))?; + let response = reqwest::Client::new() + .get(signed.signed_url.as_str()) + .send() + .await + .map_err(|error| editor_agent_oss_read_error(error.to_string()))?; + if response.status() == StatusCode::NOT_FOUND { + return Ok(empty_messages_document( + conversation.conversation_id.as_str(), + )); + } + if !response.status().is_success() { + return Err(editor_agent_oss_read_error(format!( + "OSS returned non-success status {}", + response.status().as_u16() + ))); + } + if response + .content_length() + .is_some_and(|size| size > EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES as u64) + { + return Err(editor_agent_messages_document_too_large()); + } + let bytes = response + .bytes() + .await + .map_err(|error| editor_agent_oss_read_error(error.to_string()))?; + if bytes.is_empty() { + return Ok(empty_messages_document( + conversation.conversation_id.as_str(), + )); + } + if bytes.len() > EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES { + return Err(editor_agent_messages_document_too_large()); + } + let document: EditorAgentConversationMessagesDocument = serde_json::from_slice(&bytes) + .map_err(|error| { + editor_agent_oss_read_error(format!("message document JSON invalid: {error}")) + })?; + if document.conversation_id != conversation.conversation_id { + return Err( + AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({ + "provider": "editor-agent", + "message": "message document conversationId does not match metadata", + "conversationId": conversation.conversation_id, + "documentConversationId": document.conversation_id, + })), + ); + } + Ok(document) +} + +pub async fn ensure_editor_project_access( + state: &AppState, + project_id: &str, + owner_user_id: &str, +) -> Result<(), AppError> { + state + .spacetime_client() + .get_editor_project(EditorProjectGetRecordInput { + project_id: project_id.to_string(), + owner_user_id: owner_user_id.to_string(), + }) + .await + .map(|_| ()) + .map_err(map_editor_project_error) +} + +pub async fn normalize_editor_agent_attachments( + state: &AppState, + conversation: &EditorAgentConversationRecord, + attachments: &[EditorAgentAttachmentRef], +) -> Result, AppError> { + if attachments.is_empty() { + return Ok(Vec::new()); + } + + let needs_canvas_resources = attachments + .iter() + .any(|attachment| attachment.source == EditorAgentAttachmentSource::CanvasResource); + let needs_library_assets = attachments + .iter() + .any(|attachment| attachment.source == EditorAgentAttachmentSource::LibraryAsset); + + let project = if needs_canvas_resources { + Some( + state + .spacetime_client() + .get_editor_project(EditorProjectGetRecordInput { + project_id: conversation.project_id.clone(), + owner_user_id: conversation.owner_user_id.clone(), + }) + .await + .map_err(map_editor_project_error)?, + ) + } else { + None + }; + let library = if needs_library_assets { + Some( + state + .spacetime_client() + .get_editor_asset_library(conversation.owner_user_id.clone(), current_utc_micros()) + .await + .map_err(map_editor_project_error)?, + ) + } else { + None + }; + + attachments + .iter() + .map(|attachment| { + normalize_editor_agent_attachment( + conversation, + project.as_ref(), + library.as_ref(), + attachment, + ) + }) + .collect() +} + +fn normalize_editor_agent_attachment( + conversation: &EditorAgentConversationRecord, + project: Option<&EditorProjectRecord>, + library: Option<&EditorAssetLibraryRecord>, + attachment: &EditorAgentAttachmentRef, +) -> Result { + let reference_id = normalize_required_string(attachment.reference_id.as_str()) + .ok_or_else(|| editor_agent_bad_request("attachment.referenceId is required"))?; + match attachment.source { + EditorAgentAttachmentSource::CanvasResource => { + let project = project.ok_or_else(|| { + editor_agent_bad_request("canvas resource attachment project context missing") + })?; + let resource = project + .resources + .iter() + .find(|resource| resource.resource_id == reference_id) + .ok_or_else(|| { + editor_agent_bad_request(format!( + "canvas resource attachment not found in current project: {reference_id}" + )) + })?; + normalize_canvas_resource_attachment(conversation, attachment, resource) + } + EditorAgentAttachmentSource::LibraryAsset => { + let library = library.ok_or_else(|| { + editor_agent_bad_request("library asset attachment context missing") + })?; + let asset = library + .assets + .iter() + .find(|asset| asset.asset_id == reference_id) + .ok_or_else(|| { + editor_agent_bad_request(format!( + "library asset attachment not found for current user: {reference_id}" + )) + })?; + normalize_library_asset_attachment(attachment, asset) + } + } +} + +pub fn normalize_canvas_resource_attachment( + conversation: &EditorAgentConversationRecord, + attachment: &EditorAgentAttachmentRef, + resource: &EditorProjectResourceRecord, +) -> Result { + if resource.project_id != conversation.project_id + || resource.owner_user_id != conversation.owner_user_id + { + return Err(editor_agent_bad_request( + "canvas resource attachment does not belong to this conversation project", + )); + } + validate_attachment_object_key( + attachment.object_key.as_deref(), + resource.object_key.as_deref(), + resource.resource_id.as_str(), + )?; + + Ok(EditorAgentAttachmentRef { + source: EditorAgentAttachmentSource::CanvasResource, + reference_id: resource.resource_id.clone(), + object_key: resource.object_key.clone(), + image_src: resource.image_src.clone(), + thumbnail_src: None, + label: normalize_optional_string(attachment.label.clone()), + width: Some(resource.width), + height: Some(resource.height), + }) +} + +pub fn normalize_library_asset_attachment( + attachment: &EditorAgentAttachmentRef, + asset: &EditorAssetRecord, +) -> Result { + validate_attachment_object_key( + attachment.object_key.as_deref(), + asset.object_key.as_deref(), + asset.asset_id.as_str(), + )?; + + Ok(EditorAgentAttachmentRef { + source: EditorAgentAttachmentSource::LibraryAsset, + reference_id: asset.asset_id.clone(), + object_key: asset.object_key.clone(), + image_src: asset.image_src.clone(), + thumbnail_src: asset.thumbnail_src.clone(), + label: normalize_optional_string(attachment.label.clone()) + .or_else(|| Some(asset.label.clone())), + width: Some(asset.width), + height: Some(asset.height), + }) +} + +fn validate_attachment_object_key( + submitted_object_key: Option<&str>, + stored_object_key: Option<&str>, + reference_id: &str, +) -> Result<(), AppError> { + let Some(submitted_object_key) = submitted_object_key.and_then(normalize_required_string) + else { + return Ok(()); + }; + let Some(stored_object_key) = stored_object_key.and_then(normalize_required_string) else { + return Err(editor_agent_bad_request(format!( + "attachment objectKey is not available for reference: {reference_id}" + ))); + }; + if submitted_object_key != stored_object_key { + return Err(editor_agent_bad_request(format!( + "attachment objectKey does not match reference: {reference_id}" + ))); + } + Ok(()) +} + +pub fn conversation_summary_from_record( + conversation: EditorAgentConversationRecord, +) -> EditorAgentConversationSummary { + EditorAgentConversationSummary { + conversation_id: conversation.conversation_id, + project_id: conversation.project_id, + title: conversation.title, + updated_at: conversation.updated_at, + } +} + +pub fn conversation_detail_from_record( + conversation: EditorAgentConversationRecord, + messages: Vec, +) -> EditorAgentConversationDetail { + EditorAgentConversationDetail { + conversation_id: conversation.conversation_id, + project_id: conversation.project_id, + title: conversation.title, + created_at: conversation.created_at, + updated_at: conversation.updated_at, + messages, + } +} + +pub fn build_editor_agent_canvas_completion( + project: &EditorProjectRecord, + tool_name: &str, + title: &str, +) -> EditorCanvasGenerationCompletionPayload { + let (width, height) = editor_agent_tool_display_size(tool_name); + let (x, y) = next_editor_agent_canvas_position(project.layers.clone(), width, height); + EditorCanvasGenerationCompletionPayload { + dialog_id: None, + title: title.to_string(), + placeholder: EditorCanvasGenerationPlaceholderPayload { + x, + y, + width, + height, + original_width: width, + original_height: height, + }, + } +} + +fn editor_agent_tool_display_size(tool_name: &str) -> (f64, f64) { + match tool_name { + "ui-design" | "generate-ui-design" => (640.0, 360.0), + "generate-character" => (512.0, 768.0), + _ => (512.0, 512.0), + } +} + +const EDITOR_AGENT_CANVAS_RESULT_GAP: f64 = 32.0; + +fn next_editor_agent_canvas_position( + layers: serde_json::Value, + _width: f64, + _height: f64, +) -> (f64, f64) { + let mut max_right: Option = None; + let mut min_y: Option = None; + if let Value::Array(items) = layers { + for item in items { + if item.get("itemType").and_then(Value::as_str) == Some("generation-dialog") { + continue; + } + let Some(x) = item.get("x").and_then(Value::as_f64) else { + continue; + }; + let Some(y) = item.get("y").and_then(Value::as_f64) else { + continue; + }; + let width = item.get("width").and_then(Value::as_f64).unwrap_or(0.0); + if !x.is_finite() || !y.is_finite() || !width.is_finite() { + continue; + } + max_right = Some(max_right.map_or(x + width, |value| value.max(x + width))); + min_y = Some(min_y.map_or(y, |value| value.min(y))); + } + } + ( + max_right.map_or(0.0, |right| right + EDITOR_AGENT_CANVAS_RESULT_GAP), + min_y.unwrap_or(0.0), + ) +} + +pub fn empty_messages_document(conversation_id: &str) -> EditorAgentConversationMessagesDocument { + EditorAgentConversationMessagesDocument { + version: EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION, + conversation_id: conversation_id.to_string(), + messages: Vec::new(), + } +} diff --git a/server-rs/crates/api-server/src/editor_generation_queue.rs b/server-rs/crates/api-server/src/editor_generation_queue.rs index e58ea8dc9..640aa6dc9 100644 --- a/server-rs/crates/api-server/src/editor_generation_queue.rs +++ b/server-rs/crates/api-server/src/editor_generation_queue.rs @@ -47,12 +47,41 @@ where T: Serialize, { let job_id = build_prefixed_uuid_id("task-"); + enqueue_editor_generation_job_with_identity( + state, + owner_user_id, + job_kind, + source_entity_id, + request_label, + price_mud_points, + payload, + job_id.clone(), + format!("editor-canvas:{job_kind}:{job_id}"), + ) + .await +} + +#[allow(clippy::too_many_arguments)] +pub(crate) async fn enqueue_editor_generation_job_with_identity( + state: &AppState, + owner_user_id: &str, + job_kind: &str, + source_entity_id: impl Into, + request_label: impl Into, + price_mud_points: u64, + payload: &T, + job_id: String, + dedupe_key: String, +) -> Result +where + T: Serialize, +{ let request_payload_json = serialize_editor_generation_job_payload(payload)?; let now_micros = current_utc_micros(); state .spacetime_client() .enqueue_external_generation_job(ExternalGenerationJobEnqueueRecordInput { - dedupe_key: format!("editor-canvas:{job_kind}:{job_id}"), + dedupe_key, job_id, job_kind: job_kind.to_string(), owner_user_id: owner_user_id.to_string(), diff --git a/server-rs/crates/api-server/src/external_generation_worker.rs b/server-rs/crates/api-server/src/external_generation_worker.rs index 2bedd96a3..da03bc87a 100644 --- a/server-rs/crates/api-server/src/external_generation_worker.rs +++ b/server-rs/crates/api-server/src/external_generation_worker.rs @@ -829,7 +829,9 @@ async fn process_external_generation_job_once( ) .await { - Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Ok(result) => { + complete_editor_generation_job(&state, &worker_id, &job, result.0).await + } Err(error) => { let message = error.body_text(); fail_job(&state, &worker_id, &job, message.clone()).await?; @@ -858,7 +860,9 @@ async fn process_external_generation_job_once( ) .await { - Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Ok(result) => { + complete_editor_generation_job(&state, &worker_id, &job, result.0).await + } Err(error) => { let message = error.body_text(); fail_job(&state, &worker_id, &job, message.clone()).await?; @@ -960,7 +964,9 @@ async fn process_external_generation_job_once( ) .await { - Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Ok(result) => { + complete_editor_generation_job(&state, &worker_id, &job, result.0).await + } Err(response) => { let message = response_error_message(response).await; fail_job(&state, &worker_id, &job, message.clone()).await?; @@ -989,7 +995,9 @@ async fn process_external_generation_job_once( ) .await { - Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Ok(result) => { + complete_editor_generation_job(&state, &worker_id, &job, result.0).await + } Err(response) => { let message = response_error_message(response).await; fail_job(&state, &worker_id, &job, message.clone()).await?; @@ -1018,7 +1026,9 @@ async fn process_external_generation_job_once( ) .await { - Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Ok(result) => { + complete_editor_generation_job(&state, &worker_id, &job, result.0).await + } Err(response) => { let message = response_error_message(response).await; fail_job(&state, &worker_id, &job, message.clone()).await?; @@ -1047,7 +1057,9 @@ async fn process_external_generation_job_once( ) .await { - Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Ok(result) => { + complete_editor_generation_job(&state, &worker_id, &job, result.0).await + } Err(response) => { let message = response_error_message(response).await; fail_job(&state, &worker_id, &job, message.clone()).await?; @@ -1167,18 +1179,13 @@ async fn complete_editor_generation_job( state: &AppState, worker_id: &str, job: &ExternalGenerationJobRecord, + response: Value, ) -> Result<(), String> { complete_job( state, worker_id, job, - Some( - json!({ - "sourceModule": job.source_module.clone(), - "sourceEntityId": job.source_entity_id.clone(), - }) - .to_string(), - ), + Some(editor_generation_result_payload_json(job, &response)), ) .await } @@ -1206,6 +1213,17 @@ fn editor_generation_result_payload_json( "sourceModule": job.source_module.clone(), "sourceEntityId": job.source_entity_id.clone(), }); + if is_editor_agent_generation_job(job) + && let Some(object) = payload.as_object_mut() + { + // The Agent needs this compact result to restore its tool-call card. Other jobs keep + // master's metadata-only completion payload to avoid turning the queue into an asset API. + object.insert( + // TODO extract const + "editor-agent-tool-call-result".to_string(), + compact_editor_generation_result(response.clone()), + ); + } if let Some(warning) = extract_editor_generation_warning(response) && let Some(object) = payload.as_object_mut() { @@ -1214,6 +1232,56 @@ fn editor_generation_result_payload_json( payload.to_string() } +fn is_editor_agent_generation_job(job: &ExternalGenerationJobRecord) -> bool { + serde_json::from_str::(job.request_payload_json.as_str()) + .ok() + .is_some_and(|payload| { + payload + .pointer("/generationInputs/source") + .and_then(Value::as_str) + // TODO extract const + .is_some_and(|source| source.trim() == "editor-agent") + }) +} + +fn compact_editor_generation_result(mut result: Value) -> Value { + let Some(object) = result.as_object_mut() else { + return result; + }; + object.remove("project"); + object.remove("asset"); + object.remove("spritesheetAsset"); + for field in ["resource", "spritesheetResource"] { + let Some(resource) = object.get_mut(field).and_then(Value::as_object_mut) else { + continue; + }; + resource + .retain(|key, _| matches!(key.as_str(), "resourceId" | "objectKey" | "assetObjectId")); + } + if let Some(icon_image_srcs) = object + .get_mut("iconImageSrcs") + .and_then(Value::as_array_mut) + { + for icon in icon_image_srcs { + let Some(icon) = icon.as_object_mut() else { + continue; + }; + if let Some(resource) = icon.get_mut("resource").and_then(Value::as_object_mut) { + resource.retain(|key, _| { + matches!(key.as_str(), "resourceId" | "objectKey" | "assetObjectId") + }); + } + icon.retain(|key, _| { + matches!( + key.as_str(), + "name" | "imageSrc" | "width" | "height" | "resource" + ) + }); + } + } + result +} + fn extract_editor_generation_warning(response: &Value) -> Option { let data = response.get("data").unwrap_or(response); let (warning, is_slice_warning) = match data.get("warning") { @@ -1629,6 +1697,128 @@ mod tests { ); assert!(payload.get("spritesheetImageSrc").is_none()); assert!(payload.get("iconImageSrcs").is_none()); + assert!(payload.get("editor-agent-tool-call-result").is_none()); + } + + #[test] + fn editor_agent_result_payload_keeps_compact_response() { + let mut job = external_generation_job_record_fixture(Some("lease-1")); + job.request_payload_json = json!({ + "generationInputs": { "source": "editor-agent" }, + }) + .to_string(); + let response = json!({ + "imageSrc": "/api/assets/object/generated.png", + "objectKey": "users/user-1/generated.png", + "assetObjectId": "asset-object-1", + "width": 1024, + "height": 1024, + "sourceType": "generated", + "prompt": "castle", + "actualPrompt": null, + "model": "gpt-image-2", + "provider": "VectorEngine", + "taskId": "provider-task-1", + "resource": { + "resourceId": "resource-1", + "objectKey": "users/user-1/generated.png", + "assetObjectId": "asset-object-1", + "imageSrc": "data:image/png;base64,SHOULD_NOT_PERSIST", + }, + "asset": { "assetId": "asset-1" }, + "project": { "projectId": "project-1" }, + }); + + let payload: Value = + serde_json::from_str(&editor_generation_result_payload_json(&job, &response)) + .expect("worker result should be valid JSON"); + + assert_eq!( + payload["editor-agent-tool-call-result"]["imageSrc"], + json!("/api/assets/object/generated.png") + ); + assert_eq!( + payload["editor-agent-tool-call-result"]["resource"], + json!({ + "resourceId": "resource-1", + "objectKey": "users/user-1/generated.png", + "assetObjectId": "asset-object-1", + }) + ); + assert!(payload["editor-agent-tool-call-result"].get("asset").is_none()); + assert!(payload["editor-agent-tool-call-result"].get("project").is_none()); + assert!(!payload.to_string().contains("data:image")); + } + + #[test] + fn editor_agent_spritesheet_result_keeps_all_persisted_slices() { + let mut job = external_generation_job_record_fixture(Some("lease-1")); + job.request_payload_json = json!({ + "generationInputs": { "source": "editor-agent" }, + }) + .to_string(); + let response = json!({ + "spritesheetImageSrc": "/api/assets/object/sheet.png", + "spritesheetWidth": 512, + "spritesheetHeight": 512, + "taskId": "provider-task-1", + "spritesheetResource": { + "resourceId": "sheet-resource", + "objectKey": "users/user-1/sheet.png", + "assetObjectId": "sheet-object", + }, + "iconImageSrcs": [ + { + "name": "backpack", + "imageSrc": "/api/assets/object/backpack.png", + "width": 64, + "height": 64, + "resource": { + "resourceId": "icon-resource-1", + "objectKey": "users/user-1/backpack.png", + "assetObjectId": "icon-object-1", + "imageSrc": "data:image/png;base64,SHOULD_NOT_PERSIST", + }, + "asset": { "assetId": "icon-asset-1" }, + }, + { + "name": "map", + "imageSrc": "/api/assets/object/map.png", + "width": 64, + "height": 64, + "resource": { + "resourceId": "icon-resource-2", + "objectKey": "users/user-1/map.png", + "assetObjectId": "icon-object-2", + }, + }, + ], + }); + + let payload: Value = + serde_json::from_str(&editor_generation_result_payload_json(&job, &response)) + .expect("worker result should be valid JSON"); + + assert_eq!( + payload["editor-agent-tool-call-result"]["iconImageSrcs"] + .as_array() + .map(Vec::len), + Some(2) + ); + assert_eq!( + payload["editor-agent-tool-call-result"]["iconImageSrcs"][0]["resource"], + json!({ + "resourceId": "icon-resource-1", + "objectKey": "users/user-1/backpack.png", + "assetObjectId": "icon-object-1", + }) + ); + assert!( + payload["editor-agent-tool-call-result"]["iconImageSrcs"][0] + .get("asset") + .is_none() + ); + assert!(!payload.to_string().contains("data:image")); } #[test] diff --git a/server-rs/crates/api-server/src/main.rs b/server-rs/crates/api-server/src/main.rs index 6ce33e2ce..cd0aea64e 100644 --- a/server-rs/crates/api-server/src/main.rs +++ b/server-rs/crates/api-server/src/main.rs @@ -39,7 +39,6 @@ mod custom_world_asset_prompts; mod custom_world_foundation_draft; mod custom_world_result_prompts; mod custom_world_rpg_draft_prompts; -mod editor_agent; mod editor_generation_config; mod editor_generation_queue; mod editor_green_screen; @@ -115,6 +114,7 @@ mod wechat; mod wooden_fish; mod work_author; mod work_play_tracking; +mod editor_agent; use shared_logging::{OtelConfig, init_tracing}; use socket2::{Domain, Protocol, Socket, Type}; diff --git a/server-rs/crates/api-server/src/modules/editor_project.rs b/server-rs/crates/api-server/src/modules/editor_project.rs index c5c086b61..547443534 100644 --- a/server-rs/crates/api-server/src/modules/editor_project.rs +++ b/server-rs/crates/api-server/src/modules/editor_project.rs @@ -7,10 +7,11 @@ use axum::{ use crate::{ auth::require_bearer_auth, + editor_agent::api::editor_agent_message, editor_agent::{ + cancel_editor_agent_tool_call, confirm_editor_agent_tool_call, create_editor_agent_conversation, delete_editor_agent_conversation, get_editor_agent_conversation, list_editor_agent_conversations, - stream_editor_agent_message, }, editor_project::{ create_editor_asset, create_editor_asset_folder, create_editor_project, @@ -98,14 +99,28 @@ pub fn router(state: AppState) -> Router { )), ) .route( - "/api/editor/agent-conversations/{conversation_id}/messages/stream", - post(stream_editor_agent_message) + "/api/editor/agent-conversations/{conversation_id}/messages", + post(editor_agent_message) .layer(DefaultBodyLimit::max(EDITOR_AGENT_MESSAGE_BODY_LIMIT_BYTES)) .route_layer(middleware::from_fn_with_state( state.clone(), require_bearer_auth, )), ) + .route( + "/api/editor/agent-conversations/{conversation_id}/messages/{message_id}/confirm", + post(confirm_editor_agent_tool_call).route_layer(middleware::from_fn_with_state( + state.clone(), + require_bearer_auth, + )), + ) + .route( + "/api/editor/agent-conversations/{conversation_id}/messages/{message_id}/cancel", + post(cancel_editor_agent_tool_call).route_layer(middleware::from_fn_with_state( + state.clone(), + require_bearer_auth, + )), + ) .route( "/api/editor/project-resources/{resource_id}/showcase", patch(update_editor_project_resource_showcase).route_layer( diff --git a/server-rs/crates/module-editor-agent/Cargo.toml b/server-rs/crates/module-editor-agent/Cargo.toml index e54246781..188b483f0 100644 --- a/server-rs/crates/module-editor-agent/Cargo.toml +++ b/server-rs/crates/module-editor-agent/Cargo.toml @@ -12,4 +12,4 @@ spacetime-types = ["dep:spacetimedb"] serde = { workspace = true } serde_json = { workspace = true } shared-kernel = { workspace = true } -spacetimedb = { workspace = true, optional = true } \ No newline at end of file +spacetimedb = { workspace = true, optional = true } diff --git a/server-rs/crates/platform-editor-agent/Cargo.toml b/server-rs/crates/platform-editor-agent/Cargo.toml new file mode 100644 index 000000000..a38849a97 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/Cargo.toml @@ -0,0 +1,14 @@ +[package] +name = "platform-editor-agent" +edition.workspace = true +version.workspace = true +license.workspace = true + +[dependencies] +hmac = { workspace = true } +platform-agent = { workspace = true } +platform-llm = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +sha2 = { workspace = true } +shared-contracts = { workspace = true, features = ["oss-contracts"] } diff --git a/server-rs/crates/platform-editor-agent/src/agent/agent.rs b/server-rs/crates/platform-editor-agent/src/agent/agent.rs new file mode 100644 index 000000000..5f353b646 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/agent.rs @@ -0,0 +1,198 @@ +use crate::framework::agent::{Agent, LlmApiAdaptor}; +use crate::framework::agent_builder::AgentBuilder; +use crate::framework::error::PromptError; +use crate::framework::hook::Hook; +use crate::framework::memory::AgentMemory; +use crate::framework::tool::{Tool, ToolDyn}; +use platform_agent::CREATIVE_AGENT_GPT5_MODEL; +use platform_llm::{LlmClient, LlmMessage, LlmTextRequest}; +use serde_json::Value; + +const EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS: u32 = 1024; +const EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS: u64 = 60_000; + +pub struct LlmCompletionModel { + client: LlmClient, +} + +impl LlmApiAdaptor for LlmCompletionModel { + async fn complete<'a>( + &self, + messages: impl Iterator + Send, + ) -> Result { + let request = build_editor_agent_llm_request(messages.cloned().collect()); + let response = self + .client + .request_text(request) + .await + .map_err(|e| PromptError::CompletionError(e.to_string()))?; + Ok(response.content) + } + + fn tool_result_message(&self, tool_name: &str, output: &str) -> LlmMessage { + LlmMessage::system(format!("Tool '{tool_name}' returned: {output}")) + } + + fn build_assistant_message(&self, text: &str) -> LlmMessage { + LlmMessage::assistant(text) + } +} + +fn build_editor_agent_llm_request(messages: Vec) -> LlmTextRequest { + LlmTextRequest::new(messages) + .with_model(CREATIVE_AGENT_GPT5_MODEL) + .with_max_tokens(EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS) + .with_request_timeout_ms(EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS) +} + +pub struct LlmChatAgentBuilder { + client: Option, + system_prompt_parts: Vec, + tools: Vec>, + hooks: Vec>, + max_turns: usize, + memory_data: Option>>, +} + +impl AgentBuilder for LlmChatAgentBuilder { + type Client = LlmClient; + + fn new() -> Self { + Self { + client: None, + system_prompt_parts: Vec::new(), + tools: Vec::new(), + hooks: Vec::new(), + max_turns: 10, + memory_data: None, + } + } + + fn with_client(mut self, client: LlmClient) -> Self { + self.client = Some(client); + self + } + + fn system_prompt(mut self, system_prompt: impl Into) -> Self { + self.system_prompt_parts.push(system_prompt.into()); + self + } + + fn tool(mut self, tool: impl Tool + Send + Sync + 'static) -> Self { + self.tools.push(Box::new(tool)); + self + } + + fn add_hook(mut self, hook: impl Hook + 'static) -> Self { + self.hooks.push(Box::new(hook)); + self + } + + fn max_turns(mut self, n: usize) -> Self { + self.max_turns = n; + self + } + + fn memory(mut self, memory: impl AgentMemory + 'static) -> Self { + self.memory_data = Some(Box::new(memory)); + self + } + + fn build(self) -> Agent { + let model = LlmCompletionModel { + client: self.client.expect("call .with_client() first"), + }; + let mut agent = Agent::new(model); + let tool_specs = self + .tools + .iter() + .map(|tool| ToolPromptSpec { + name: tool.tool_name().to_string(), + description: tool.description(), + parameters: tool.parameters(), + }) + .collect::>(); + let base_prompt = self.system_prompt_parts.join("\n\n"); + let system_prompt = build_tools_system_prompt(&base_prompt, &tool_specs); + agent.tools = self.tools; + agent.hooks = self.hooks; + agent.default_max_turns = self.max_turns; + agent.memory = self.memory_data; + agent.system_prompt = Some(LlmMessage::system(&system_prompt)); + agent + } +} + +struct ToolPromptSpec { + name: String, + description: String, + parameters: Value, +} + +fn build_tools_system_prompt(base_prompt: &str, tool_specs: &[ToolPromptSpec]) -> String { + let mut prompt = String::new(); + prompt.push_str(base_prompt); + prompt.push_str("\n\nYou have access to the following tools.\n\n"); + + if tool_specs.is_empty() { + prompt.push_str("(No tools available.)\n"); + } else { + prompt.push_str("## JSON Response Format\n"); + prompt.push_str("respond with valid JSON only (no markdown fences):\n"); + prompt.push_str("{\n"); + prompt.push_str(" \"reply_text\": \"your message to the user\",\n"); + prompt.push_str(" \"tool_calls\": [\n {\n"); + prompt.push_str(" \"tool_name\": \"tool_name_here\",\n"); + prompt.push_str(" \"args\": { \"argument_name\": \"argument_value\" }\n"); + prompt.push_str(" }\n ]\n"); + prompt.push_str("}\n\n"); + prompt.push_str("If you don't need to use a tool, respond with:\n"); + prompt.push_str("{\n"); + prompt.push_str(" \"reply_text\": \"your message\",\n"); + prompt.push_str(" \"tool_calls\": []\n"); + prompt.push_str("}\n\n"); + prompt.push_str("## Available Tools\n\n"); + + for tool in tool_specs { + prompt.push_str(&format!("- {}\n", tool.name)); + prompt.push_str(&format!(" Description: {}\n", tool.description)); + prompt.push_str(" Arguments JSON Schema:\n"); + let parameters = serde_json::to_string_pretty(&tool.parameters) + .unwrap_or_else(|_| tool.parameters.to_string()); + prompt.push_str(¶meters); + prompt.push_str("\n"); + } + + prompt.push_str( + "as you see, tool_calls is an array, several tools calls can be executed in one turn concurrently. ", + ); + prompt.push_str("your valid tool call will be recorded as system message"); + prompt.push_str( + "some tools calls needs user's confirmation, you should not retry the same tool call in this case.\ + And if all tool calls are pending confirmation, you should just end the turn, as you cant do more before user's action" + ); + } + + prompt +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn editor_agent_llm_request_keeps_gpt5_contract() { + let request = build_editor_agent_llm_request(vec![ + LlmMessage::system("系统提示"), + LlmMessage::user("用户请求"), + ]); + + assert_eq!( + request.model.as_deref(), + Some(platform_agent::CREATIVE_AGENT_GPT5_MODEL) + ); + assert_eq!(request.max_tokens, Some(1024)); + assert_eq!(request.request_timeout_ms, Some(60_000)); + assert_eq!(request.messages.len(), 2); + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/asset.rs b/server-rs/crates/platform-editor-agent/src/agent/asset.rs new file mode 100644 index 000000000..027ac007d --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/asset.rs @@ -0,0 +1,43 @@ +use std::fmt::Display; +use sha2::Sha256; +use serde::{Deserialize, Serialize}; +use hmac::digest::Digest; + +#[derive(Debug, Clone, PartialEq, Eq, Hash)] +pub struct ImageId { + pub id: String, +} + +impl ImageId { + /// Keep data keys server-side while giving the model a stable image reference. + pub fn from_data_key(data_key: impl AsRef) -> Self { + let digest = Sha256::digest(data_key.as_ref().as_bytes()); + Self { + id: format!("sha256:{digest:x}"), + } + } +} + +impl Serialize for ImageId { + fn serialize(&self, serializer: S) -> Result { + self.id.serialize(serializer) + } +} + +impl<'de> Deserialize<'de> for ImageId { + fn deserialize>(deserializer: D) -> Result { + String::deserialize(deserializer).map(|id| ImageId { id }) + } +} + +impl Display for ImageId { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.id) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ImageMetadata { + pub tag: String, + pub data_key: String, +} \ No newline at end of file diff --git a/server-rs/crates/platform-editor-agent/src/agent/mod.rs b/server-rs/crates/platform-editor-agent/src/agent/mod.rs new file mode 100644 index 000000000..e7c0893c8 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/mod.rs @@ -0,0 +1,3 @@ +pub mod tools; +pub mod asset; +pub mod agent; diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs new file mode 100644 index 000000000..9705c7867 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/context.rs @@ -0,0 +1,22 @@ +use std::collections::HashMap; +use serde::{Deserialize, Serialize}; +use crate::agent::asset::ImageId; +use crate::agent::asset::ImageMetadata; + +impl EditorToolContext { + /// Check if an image with the given ID exists in the context. + pub fn contains_image(&self, image_id: &ImageId) -> bool { + self.images.contains_key(image_id) + } + + pub fn image_data_key(&self, image_id: &ImageId) -> Option<&str> { + self.images + .get(image_id) + .map(|metadata| metadata.data_key.as_str()) + } +} + +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct EditorToolContext { + pub images: HashMap, +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/edit_image.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/edit_image.rs new file mode 100644 index 000000000..a93d4ff6f --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/edit_image.rs @@ -0,0 +1,175 @@ +use crate::agent::tools::context::EditorToolContext; +use crate::agent::asset::ImageId; +use crate::framework::tool::{Tool, ToolFailure, ToolFailureKind}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use std::error::Error; +use std::fmt::Display; +pub struct EditImageTool { + pub context: EditorToolContext, +} + +#[derive(Debug, Clone)] +pub enum EditImageError { + ObjectImageNotProvided, + PromptNotProvided, + AssetNotFound(ImageId), +} + +impl Display for EditImageError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + EditImageError::ObjectImageNotProvided => { + write!(f, "object image not provided") + } + EditImageError::PromptNotProvided => write!(f, "prompt not provided"), + EditImageError::AssetNotFound(image_id) => { + write!(f, "asset {image_id} not found in context") + } + } + } +} + +impl Error for EditImageError {} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct EditImageToolArgs { + pub object_image_id: ImageId, + #[serde(default)] + pub reference_image_ids: Vec, + pub prompt: String, + // #[serde(default)] + // pub tag: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct EditImageToolOutput { + pub message: String, +} + +impl Tool for EditImageTool { + const NAME: &'static str = "edit-image"; + type Error = EditImageError; + type Args = EditImageToolArgs; + type Output = EditImageToolOutput; + + fn description(&self) -> String { + "仅用于修改已有图片：换衣服、改颜色、替换背景、局部重绘，或保持主体、构图、姿势不变的编辑。用户指代“这张”“刚才那个”“上一张”“改成”“换成”时优先使用本工具。必须使用已有图片上下文；没有参考图时不要调用，应提示用户先选择参考图。" + .to_string() + } + + fn parameters(&self) -> Value { + json!({ + "type": "object", + "properties": { + "object_image_id": { + "type": "string", + "description": "要修改的目标图片。" + }, + "reference_image_ids": { + "type": "array", + "items": { "type": "string" }, + "description": "修改参考图 ID 列表（可选的，用于提供风格或元素参考）。" + }, + "prompt": { + "type": "string", + "description": "编辑提示词，描述希望如何修改图片。例如「把背景换成红色」、「把人物改成坐着」。" + }, + // "tag": { + // "type": "string", + // "description": "为新生成的图片添加标签，用于后续在上下文中引用。" + // } + }, + "required": ["object_image_id", "prompt"], + "additionalProperties": false + }) + } + + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + if let Some(error) = self.validate_args(&args) { + return Err(error); + } + Ok(EditImageToolOutput { + message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string(), + }) + } + } + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + match error { + EditImageError::AssetNotFound(_) => { + ToolFailure::new(ToolFailureKind::NotFound, error.to_string()) + } + EditImageError::ObjectImageNotProvided | EditImageError::PromptNotProvided => { + ToolFailure::invalid_args(error.to_string()) + } + } + } +} + +/// Typed result deserialized from `edit_editor_image_for_owner` response. +/// +/// Mirrors `EditorImageGenerationResponse` but uses `String` instead of `&'static str` +/// and `Value` for nested payload types so that `#[derive(Deserialize)]` works. +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct EditorImageEditResult { + pub image_src: String, + pub object_key: Option, + pub asset_object_id: Option, + pub width: u32, + pub height: u32, + pub source_type: String, + pub prompt: String, + pub actual_prompt: Option, + pub model: String, + pub provider: String, + pub task_id: String, + pub resource: Option, + pub asset: Option, + pub project: Option, +} + +impl EditImageTool { + /// Validate the semantic correctness of the arguments. + fn validate_args(&self, args: &EditImageToolArgs) -> Option { + if args.prompt.trim().is_empty() { + return Some(EditImageError::PromptNotProvided); + } + if !self.context.contains_image(&args.object_image_id) { + return Some(EditImageError::AssetNotFound(args.object_image_id.clone())); + } + + for ref_id in &args.reference_image_ids { + if !self.context.contains_image(ref_id) { + return Some(EditImageError::AssetNotFound(ref_id.clone())); + } + } + None + } + + pub fn format_execute_message( + &self, + args: ::Args, + result: EditorImageEditResult, + ) -> String { + let name = Self::NAME.to_string(); + let arg_json = serde_json::to_string(&args).unwrap_or_default(); + + let data_key = result + .object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| result.image_src.clone()); + let image_id = ImageId::from_data_key(data_key); + format!( + "[tool_call:{name}] args: {arg_json} output: edited result saved as image: {image_id}" + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_background_music.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_background_music.rs new file mode 100644 index 000000000..19c78ac7c --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_background_music.rs @@ -0,0 +1,84 @@ +use crate::framework::tool::{Tool, ToolFailure}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use shared_contracts::assets::EditorAudioGenerateResponse; +use std::error::Error; +use std::fmt::Display; + +pub struct GenerateBackgroundMusicTool; + +#[derive(Debug, Clone)] +pub enum GenerateBackgroundMusicError { + PromptNotProvided, +} +impl Display for GenerateBackgroundMusicError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "background music prompt not provided") + } +} +impl Error for GenerateBackgroundMusicError {} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateBackgroundMusicToolArgs { + pub prompt: String, + #[serde(default = "default_instrumental")] + pub make_instrumental: bool, +} +fn default_instrumental() -> bool { + true +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateBackgroundMusicToolOutput { + pub message: String, +} + +impl Tool for GenerateBackgroundMusicTool { + const NAME: &'static str = "generate-background-music"; + type Error = GenerateBackgroundMusicError; + type Args = GenerateBackgroundMusicToolArgs; + type Output = GenerateBackgroundMusicToolOutput; + fn description(&self) -> String { + "根据文字描述生成背景音乐。默认生成纯音乐，除非明确要求歌词或人声。".to_string() + } + fn parameters(&self) -> Value { + json!({ + "type": "object", "properties": { + "prompt": { "type": "string", "description": "音乐风格、情绪、乐器和节奏描述。" }, + "make_instrumental": { "type": "boolean", "description": "是否生成纯音乐，默认 true。" } + }, "required": ["prompt"], "additionalProperties": false + }) + } + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + if args.prompt.trim().is_empty() { + return Err(GenerateBackgroundMusicError::PromptNotProvided); + } + Ok(GenerateBackgroundMusicToolOutput { + message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string(), + }) + } + } + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + ToolFailure::invalid_args(error.to_string()) + } +} + +impl GenerateBackgroundMusicTool { + + pub fn format_execute_message( + &self, + args: &GenerateBackgroundMusicToolArgs, + result: &EditorAudioGenerateResponse, + ) -> String { + format!( + "[tool_call:{}] args: {} output: generated audio saved as: {}", + Self::NAME, + serde_json::to_string(args).unwrap_or_default(), + result.audio_src + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs new file mode 100644 index 000000000..9394d762b --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_character.rs @@ -0,0 +1,76 @@ +use crate::framework::tool::{Tool, ToolFailure}; +use serde_json::{json, Value}; +use crate::agent::tools::context::EditorToolContext; +use crate::agent::tools::generate_image::{EditorImageGenerationResult, GenerateImageError, GenerateImageTool, GenerateImageToolArgs, GenerateImageToolOutput}; + +pub struct GenerateCharacterTool { + pub context: EditorToolContext, +} + +impl Tool for GenerateCharacterTool { + const NAME: &'static str = "generate-character"; + type Error = GenerateImageError; + type Args = GenerateImageToolArgs; + type Output = GenerateImageToolOutput; + + fn description(&self) -> String { + "仅用于生成新的角色形象、人物立绘或普通角色设定图。已有角色图的服装、颜色、表情、姿势或背景修改必须选择 edit-image。角色规范图、角色美术视觉规范设定图或规范展板必须选择 generate-image。" + .to_string() + } + + fn parameters(&self) -> Value { + json!({ + "type": "object", + "properties": { + "prompt": { "type": "string", "description": "角色外貌、服装、姿势、画风和构图的完整描述。" }, + "reference_image_ids": { "type": "array", "items": { "type": "string" }, "description": "可选角色或画风参考图 ID。" }, + "aspect_ratio": { "type": "string", "description": "可选比例，例如 2:3 或 9:16。" }, + "image_size": { "type": "string", "description": "可选清晰度，例如 1K 或 2K。" } + }, + "required": ["prompt"], + "additionalProperties": false + }) + } + + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + GenerateImageTool { + context: self.context.clone(), + } + .validate_args(&args)?; + Ok(GenerateImageToolOutput { message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string() }) + } + } + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + GenerateImageTool { + context: self.context.clone(), + } + .classify_error(error) + } +} + +impl GenerateCharacterTool { + pub fn format_execute_message( + &self, + args: &GenerateImageToolArgs, + result: &EditorImageGenerationResult, + ) -> String { + let tool_name = Self::NAME; + let args = serde_json::to_string(args).unwrap_or_default(); + let data_key = result + .object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| result.image_src.clone()); + let image_id = crate::agent::asset::ImageId::from_data_key(data_key); + format!( + "[tool_call:{tool_name}] args: {args} output: generated result saved as image: {image_id}" + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs new file mode 100644 index 000000000..d1892b901 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_icon_spritesheet.rs @@ -0,0 +1,172 @@ +use crate::agent::asset::ImageId; +use crate::agent::tools::context::EditorToolContext; +use crate::framework::tool::{Tool, ToolFailure, ToolFailureKind}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use std::error::Error; +use std::fmt::Display; + +pub struct GenerateIconSpritesheetTool { + pub context: EditorToolContext, +} + +#[derive(Debug, Clone)] +pub enum GenerateIconSpritesheetError { + ReferenceNotProvided, + DescriptionsNotProvided, + AssetNotFound(ImageId), +} + +impl Display for GenerateIconSpritesheetError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::ReferenceNotProvided => write!(f, "reference image not provided"), + Self::DescriptionsNotProvided => write!(f, "icon descriptions not provided"), + Self::AssetNotFound(image_id) => write!(f, "asset {image_id} not found in context"), + } + } +} + +impl Error for GenerateIconSpritesheetError {} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateIconSpritesheetToolArgs { + pub reference_image_id: ImageId, + #[serde(default)] + pub reference_image_ids: Vec, + pub icon_descriptions: Vec, + #[serde(default)] + pub aspect_ratio: Option, + #[serde(default)] + pub image_size: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateIconSpritesheetToolOutput { + pub message: String, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct EditorIconSpritesheetResult { + pub spritesheet_image_src: String, + pub spritesheet_width: u32, + pub spritesheet_height: u32, + pub task_id: String, + pub spritesheet_resource: Option, + pub spritesheet_asset: Option, + pub project: Option, + #[serde(default)] + pub icon_image_srcs: Vec, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct EditorIconSpritesheetImage { + pub image_src: String, + pub width: u32, + pub height: u32, + pub resource: Option, +} + +impl Tool for GenerateIconSpritesheetTool { + const NAME: &'static str = "generate-icon-spritesheet"; + type Error = GenerateIconSpritesheetError; + type Args = GenerateIconSpritesheetToolArgs; + type Output = GenerateIconSpritesheetToolOutput; + + fn description(&self) -> String { + "仅用于生成多个图标成品、图标素材图集或 spritesheet。必须提供图标规范或风格参考图，并填写多个 icon_descriptions；没有参考图时不要调用，应提示用户先选择参考图。图标规范图或图标视觉规范展板不是本工具，应使用 generate-image。".to_string() + } + + fn parameters(&self) -> Value { + json!({ + "type": "object", + "properties": { + "reference_image_id": { "type": "string", "description": "必填的图标规范或风格参考图 ID。" }, + "reference_image_ids": { "type": "array", "items": { "type": "string" }, "description": "可选的额外参考图 ID。" }, + "icon_descriptions": { "type": "array", "items": { "type": "string" }, "description": "要生成的多个图标描述。" }, + "aspect_ratio": { "type": "string", "description": "可选图集比例。" }, + "image_size": { "type": "string", "description": "可选清晰度，例如 1K 或 2K。" } + }, + "required": ["reference_image_id", "icon_descriptions"], + "additionalProperties": false + }) + } + + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + self.validate_args(&args)?; + Ok(GenerateIconSpritesheetToolOutput { + message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string(), + }) + } + } + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + match error { + GenerateIconSpritesheetError::AssetNotFound(_) => { + ToolFailure::new(ToolFailureKind::NotFound, error.to_string()) + } + GenerateIconSpritesheetError::ReferenceNotProvided + | GenerateIconSpritesheetError::DescriptionsNotProvided => { + ToolFailure::invalid_args(error.to_string()) + } + } + } +} + +impl GenerateIconSpritesheetTool { + fn validate_args( + &self, + args: &GenerateIconSpritesheetToolArgs, + ) -> Result<(), GenerateIconSpritesheetError> { + if args.reference_image_id.id.trim().is_empty() { + return Err(GenerateIconSpritesheetError::ReferenceNotProvided); + } + if !self.context.contains_image(&args.reference_image_id) { + return Err(GenerateIconSpritesheetError::AssetNotFound( + args.reference_image_id.clone(), + )); + } + if args + .icon_descriptions + .iter() + .all(|description| description.trim().is_empty()) + { + return Err(GenerateIconSpritesheetError::DescriptionsNotProvided); + } + for image_id in &args.reference_image_ids { + if !self.context.contains_image(image_id) { + return Err(GenerateIconSpritesheetError::AssetNotFound( + image_id.clone(), + )); + } + } + Ok(()) + } + pub fn format_execute_message( + &self, + args: &GenerateIconSpritesheetToolArgs, + result: &EditorIconSpritesheetResult, + ) -> String { + let tool_name = Self::NAME; + let args = serde_json::to_string(args).unwrap_or_default(); + let data_key = result + .spritesheet_resource + .as_ref() + .and_then(|resource| resource.get("objectKey")) + .and_then(Value::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| result.spritesheet_image_src.clone()); + let image_id = ImageId::from_data_key(data_key); + format!( + "[tool_call:{tool_name}] args: {args} output: generated spritesheet saved as image: {image_id}", + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs new file mode 100644 index 000000000..1c13df998 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_image.rs @@ -0,0 +1,161 @@ +use crate::agent::tools::context::EditorToolContext; +use crate::agent::asset::ImageId; +use crate::framework::tool::{Tool, ToolFailure, ToolFailureKind}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use std::error::Error; +use std::fmt::Display; + +pub struct GenerateImageTool { + pub context: EditorToolContext, +} + +#[derive(Debug, Clone)] +pub enum GenerateImageError { + PromptNotProvided, + AssetNotFound(ImageId), +} + +impl Display for GenerateImageError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::PromptNotProvided => write!(f, "prompt not provided"), + Self::AssetNotFound(image_id) => write!(f, "asset {image_id} not found in context"), + } + } +} + +impl Error for GenerateImageError {} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateImageToolArgs { + pub prompt: String, + #[serde(default)] + pub reference_image_ids: Vec, + #[serde(default)] + // TODO restrict to a set of possible values + pub aspect_ratio: Option, + #[serde(default)] + pub image_size: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateImageToolOutput { + pub message: String, +} + +impl Tool for GenerateImageTool { + const NAME: &'static str = "generate-image"; + type Error = GenerateImageError; + type Args = GenerateImageToolArgs; + type Output = GenerateImageToolOutput; + + fn description(&self) -> String { + "用于从文字生成全新图片：新场景、新物体、新插画或新背景；也专用于规范图、视觉规范图、风格规范图、素材规范展板。不要用来修改已有图，已有图修改应选 edit-image。规范展板 prompt 必须写明统一视角、线条粗细、描边、填充风格、材质、阴影、圆角、状态层级、色卡或色号、尺寸标注和排版层级；角色规范展板还要含头身比例、标准立绘、动作帧样例、服饰配饰分层和专属色卡。完整 prompt 必须包含画面、主体、风格、构图和背景。" + .to_string() + } + + fn parameters(&self) -> Value { + json!({ + "type": "object", + "properties": { + "prompt": { + "type": "string", + "description": "完整的生图提示词，包含主体、场景、风格、构图和背景。" + }, + "reference_image_ids": { + "type": "array", + "items": { "type": "string" }, + "description": "可选参考图 ID 列表，用于提供画风或元素参考。" + }, + "aspect_ratio": { + "type": "string", + "description": "可选画面比例，例如 1:1、16:9、9:16。" + }, + "image_size": { + "type": "string", + "description": "可选图片清晰度，例如 1K、2K、4K。" + } + }, + "required": ["prompt"], + "additionalProperties": false + }) + } + + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + self.validate_args(&args)?; + Ok(GenerateImageToolOutput { + message: "this tool call is pending user confirmation.".to_string(), + }) + } + } + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + match error { + GenerateImageError::AssetNotFound(_) => { + ToolFailure::new(ToolFailureKind::NotFound, error.to_string()) + } + GenerateImageError::PromptNotProvided => ToolFailure::invalid_args(error.to_string()), + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct EditorImageGenerationResult { + pub image_src: String, + pub object_key: Option, + pub asset_object_id: Option, + pub width: u32, + pub height: u32, + pub source_type: String, + pub prompt: String, + pub actual_prompt: Option, + pub model: String, + pub provider: String, + pub task_id: String, + pub resource: Option, + pub asset: Option, + pub project: Option, +} + +impl GenerateImageTool { + pub fn validate_args( + &self, + args: &GenerateImageToolArgs, + ) -> Result<(), GenerateImageError> { + if args.prompt.trim().is_empty() { + return Err(GenerateImageError::PromptNotProvided); + } + for image_id in &args.reference_image_ids { + if !self.context.contains_image(image_id) { + return Err(GenerateImageError::AssetNotFound(image_id.clone())); + } + } + Ok(()) + } + + pub fn format_execute_message( + &self, + args: &GenerateImageToolArgs, + result: &EditorImageGenerationResult, + ) -> String { + let args = serde_json::to_string(args).unwrap_or_default(); + let data_key = result + .object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| result.image_src.clone()); + let image_id = ImageId::from_data_key(data_key); + format!( + "[tool_call:{}] args: {args} output: generated result saved as image: {image_id}", + Self::NAME, + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs new file mode 100644 index 000000000..1eaef84af --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_sound_effect.rs @@ -0,0 +1,92 @@ +use crate::framework::tool::{Tool, ToolFailure}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use shared_contracts::assets::EditorAudioGenerateResponse; +use std::error::Error; +use std::fmt::Display; + +pub struct GenerateSoundEffectTool; + +#[derive(Debug, Clone)] +pub enum GenerateSoundEffectError { + PromptNotProvided, +} + +impl Display for GenerateSoundEffectError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "sound effect prompt not provided") + } +} + +impl Error for GenerateSoundEffectError {} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateSoundEffectToolArgs { + pub prompt: String, + #[serde(default)] + pub duration: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateSoundEffectToolOutput { + pub message: String, +} + +impl Tool for GenerateSoundEffectTool { + const NAME: &'static str = "generate-sound-effect"; + type Error = GenerateSoundEffectError; + type Args = GenerateSoundEffectToolArgs; + type Output = GenerateSoundEffectToolOutput; + + fn description(&self) -> String { + "根据文字描述生成一段音效，例如脚步声、按钮点击声或环境音。".to_string() + } + + fn parameters(&self) -> Value { + json!({ + "type": "object", + "properties": { + "prompt": { "type": "string", "description": "音效内容、材质、节奏和情绪描述。" }, + "duration": { "type": "integer", "description": "可选时长（秒）。" }, + // "model": { "type": "string", "description": "可选音效模型。" } + }, + "required": ["prompt"], + "additionalProperties": false + }) + } + + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + if args.prompt.trim().is_empty() { + return Err(GenerateSoundEffectError::PromptNotProvided); + } + Ok(GenerateSoundEffectToolOutput { + message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string(), + }) + } + } + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + ToolFailure::invalid_args(error.to_string()) + } +} + +impl GenerateSoundEffectTool { + // pub const DEFAULT_MODEL: &'static str = "audio1.0"; + + pub fn format_execute_message( + &self, + args: &GenerateSoundEffectToolArgs, + result: &EditorAudioGenerateResponse, + ) -> String { + format!( + "[tool_call:{}] args: {} output: generated audio saved as: {}", + Self::NAME, + serde_json::to_string(args).unwrap_or_default(), + result.audio_src + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs new file mode 100644 index 000000000..203e85b3f --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_ui_design.rs @@ -0,0 +1,78 @@ +use crate::agent::tools::generate_image::{ + EditorImageGenerationResult, GenerateImageError, GenerateImageTool, GenerateImageToolArgs, + GenerateImageToolOutput, +}; +use crate::framework::tool::{Tool, ToolFailure}; +use serde_json::{json, Value}; +use crate::agent::tools::context::EditorToolContext; + +pub struct GenerateUiDesignTool { + pub context: EditorToolContext, +} + +impl Tool for GenerateUiDesignTool { + const NAME: &'static str = "generate-ui-design"; + type Error = GenerateImageError; + type Args = GenerateImageToolArgs; + type Output = GenerateImageToolOutput; + + fn description(&self) -> String { + "仅用于生成完整可用的 UI 设计图或界面稿，包括 HUD、弹窗、面板、按钮组合和整页界面。不要用于提取图标、拆素材或修改已有图；UI 规范图、组件规范展板或视觉规范展板应使用 generate-image。".to_string() + } + + fn parameters(&self) -> Value { + json!({ + "type": "object", + "properties": { + "prompt": { "type": "string", "description": "完整 UI 画面、信息层级、视觉风格和构图描述。" }, + "reference_image_ids": { "type": "array", "items": { "type": "string" }, "description": "可选 UI 风格或布局参考图 ID。" }, + "aspect_ratio": { "type": "string", "description": "可选画面比例。" }, + "image_size": { "type": "string", "description": "可选清晰度，例如 1K 或 2K。" } + }, + "required": ["prompt"], + "additionalProperties": false + }) + } + + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + GenerateImageTool { + context: self.context.clone(), + } + .validate_args(&args)?; + Ok(GenerateImageToolOutput { message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string() }) + } + } + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + GenerateImageTool { + context: self.context.clone(), + } + .classify_error(error) + } +} + +impl GenerateUiDesignTool { + pub fn format_execute_message( + &self, + args: &GenerateImageToolArgs, + result: &EditorImageGenerationResult, + ) -> String { + let tool_name = Self::NAME; + let args = serde_json::to_string(args).unwrap_or_default(); + let data_key = result + .object_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| value.trim_start_matches('/').to_string()) + .unwrap_or_else(|| result.image_src.clone()); + let image_id = crate::agent::asset::ImageId::from_data_key(data_key); + format!( + "[tool_call:{tool_name}] args: {args} output: generated result saved as image: {image_id}" + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs new file mode 100644 index 000000000..882f776bb --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/generate_video.rs @@ -0,0 +1,118 @@ +use crate::agent::asset::ImageId; +use crate::framework::tool::{Tool, ToolFailure, ToolFailureKind}; +use serde::{Deserialize, Serialize}; +use serde_json::{json, Value}; +use shared_contracts::assets::EditorVideoGenerateResponse; +use std::error::Error; +use std::fmt::Display; +use crate::agent::tools::context::EditorToolContext; + +pub struct GenerateVideoTool { + pub context: EditorToolContext, +} + + +#[derive(Debug, Clone)] +pub enum GenerateVideoError { + PromptNotProvided, + AssetNotFound(ImageId), +} +impl Display for GenerateVideoError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::PromptNotProvided => write!(f, "video prompt not provided"), + Self::AssetNotFound(image_id) => write!(f, "asset {image_id} not found in context"), + } + } +} +impl Error for GenerateVideoError {} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateVideoToolArgs { + pub prompt: String, + #[serde(default)] + pub reference_image_ids: Vec, + #[serde(default)] + pub aspect_ratio: Option, + #[serde(default)] + pub duration_seconds: Option, + #[serde(default)] + pub model: Option, + #[serde(default)] + pub resolution: Option, + #[serde(default)] + pub sound: Option, +} +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct GenerateVideoToolOutput { + pub message: String, +} + +impl Tool for GenerateVideoTool { + const NAME: &'static str = "generate-video"; + type Error = GenerateVideoError; + type Args = GenerateVideoToolArgs; + type Output = GenerateVideoToolOutput; + fn description(&self) -> String { + "根据文字描述生成一段短视频；可选图片参考图用于镜头风格或主体参考。".to_string() + } + fn parameters(&self) -> Value { + json!({ + "type": "object", "properties": { + "prompt": { "type": "string", "description": "镜头、主体动作、场景和风格描述。" }, + "reference_image_ids": { "type": "array", "items": { "type": "string" }, "description": "可选图片参考图 ID。" }, + "aspect_ratio": { "type": "string", "description": "可选比例，默认 16:9。" }, + "duration_seconds": { "type": "integer", "description": "可选时长，默认 4 秒。" }, + "model": { "type": "string", "description": "可选视频模型。" }, + "resolution": { "type": "string", "description": "可选清晰度，默认 720p。" }, + "sound": { "type": "string", "description": "是否生成声音，默认 off。" } + }, "required": ["prompt"], "additionalProperties": false + }) + } + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send { + async move { + if args.prompt.trim().is_empty() { + return Err(GenerateVideoError::PromptNotProvided); + } + for id in &args.reference_image_ids { + if !self.context.contains_image(id) { + return Err(GenerateVideoError::AssetNotFound(id.clone())); + } + } + Ok(GenerateVideoToolOutput { + message: "this tool call is pending user confirmation. if all is pending, just end this turn".to_string(), + }) + } + } + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + match error { + GenerateVideoError::AssetNotFound(_) => { + ToolFailure::new(ToolFailureKind::NotFound, error.to_string()) + } + GenerateVideoError::PromptNotProvided => ToolFailure::invalid_args(error.to_string()), + } + } +} + +impl GenerateVideoTool { + pub const DEFAULT_VIDEO_MODEL: &'static str = "seedance2.0-fast"; + pub const DEFAULT_VIDEO_RESOLUTION: &'static str = "720p"; + pub const DEFAULT_VIDEO_DURATION_SECONDS: u32 = 4; + + pub fn format_execute_message( + &self, + args: &GenerateVideoToolArgs, + result: &EditorVideoGenerateResponse, + ) -> String { + let video_id = ImageId::from_data_key(result.video_src.as_str()); + format!( + "[tool_call:{}] args: {} output: generated video saved as: {}", + Self::NAME, + serde_json::to_string(args).unwrap_or_default(), + video_id + ) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs b/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs new file mode 100644 index 000000000..afee4c893 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/agent/tools/mod.rs @@ -0,0 +1,9 @@ +pub mod edit_image; +pub mod generate_background_music; +pub mod generate_character; +pub mod generate_icon_spritesheet; +pub mod generate_image; +pub mod generate_sound_effect; +pub mod generate_ui_design; +pub mod generate_video; +pub mod context; \ No newline at end of file diff --git a/server-rs/crates/platform-editor-agent/src/framework/agent.rs b/server-rs/crates/platform-editor-agent/src/framework/agent.rs new file mode 100644 index 000000000..7aac0adf5 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/agent.rs @@ -0,0 +1,80 @@ +use crate::framework::error::PromptError; +use crate::framework::hook::Hook; +use crate::framework::memory::AgentMemory; +use crate::framework::run::PromptRequest; +use crate::framework::tool::{Tool, ToolDyn}; + +pub struct Agent, Message> { + pub model: M, + pub tools: Vec>, + pub hooks: Vec>, + pub default_max_turns: usize, + pub system_prompt: Option, + pub memory: Option>>, +} + +impl Agent +where + M: LlmApiAdaptor + 'static, + Message: Send + 'static, +{ + pub fn new(model: M) -> Self { + Self { + model, + tools: Vec::new(), + hooks: Vec::new(), + default_max_turns: 10, + system_prompt: None, + memory: None, + } + } + + pub fn tool(mut self, tool: impl Tool + Send + Sync + 'static) -> Self { + self.tools.push(Box::new(tool)); + self + } + + pub fn system_prompt(mut self, msg: Message) -> Self { + self.system_prompt = Some(msg); + self + } + + pub fn memory(mut self, mem: impl AgentMemory + 'static) -> Self { + self.memory = Some(Box::new(mem)); + self + } + + pub fn hook(mut self, hook: impl Hook + 'static) -> Self { + self.hooks.push(Box::new(hook)); + self + } + + pub fn max_turns(mut self, n: usize) -> Self { + self.default_max_turns = n; + self + } + + pub fn tools(&self) -> &[Box] { + &self.tools + } + + + pub fn prompt(&mut self, message: impl Into + Send) -> PromptRequest<'_, M, Message> + where + Message: 'static, + { + PromptRequest::new(self, message.into()) + } +} + +pub trait LlmApiAdaptor: Send + Sync { + fn complete<'a>( + &self, + messages: impl Iterator + Send, + ) -> impl Future> + Send + where + Message: 'a; + fn tool_result_message(&self, tool_name: &str, output: &str) -> Message; + + fn build_assistant_message(&self, text: &str) -> Message; +} diff --git a/server-rs/crates/platform-editor-agent/src/framework/agent_builder.rs b/server-rs/crates/platform-editor-agent/src/framework/agent_builder.rs new file mode 100644 index 000000000..f2f90d8cc --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/agent_builder.rs @@ -0,0 +1,18 @@ +use crate::framework::agent::{Agent, LlmApiAdaptor}; +use crate::framework::hook::Hook; +use crate::framework::memory::AgentMemory; +use crate::framework::tool::Tool; + +pub trait AgentBuilder> { + type Client; + + fn new() -> Self; + fn with_client(self, client: Self::Client) -> Self; + fn system_prompt(self, system_prompt: impl Into) -> Self; + fn tool(self, tool: impl Tool + Send + Sync + 'static) -> Self; + fn add_hook(self, hook: impl Hook + 'static) -> Self; + fn max_turns(self, n: usize) -> Self; + fn memory(self, memory: impl AgentMemory + 'static) -> Self; + + fn build(self) -> Agent; +} diff --git a/server-rs/crates/platform-editor-agent/src/framework/error.rs b/server-rs/crates/platform-editor-agent/src/framework/error.rs new file mode 100644 index 000000000..bc830dbac --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/error.rs @@ -0,0 +1,22 @@ +#[derive(Debug, Clone)] +pub enum PromptError { + CompletionError(String), + ToolError(String), + InternalError(String), + MaxTurnsReached { max_turns: usize }, +} + +impl std::fmt::Display for PromptError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::CompletionError(msg) => write!(f, "completion error: {msg}"), + Self::ToolError(msg) => write!(f, "tool error: {msg}"), + Self::InternalError(msg) => write!(f, "internal error: {msg}"), + Self::MaxTurnsReached { max_turns } => { + write!(f, "max turns reached: {max_turns}") + } + } + } +} + +impl std::error::Error for PromptError {} diff --git a/server-rs/crates/platform-editor-agent/src/framework/hook.rs b/server-rs/crates/platform-editor-agent/src/framework/hook.rs new file mode 100644 index 000000000..c5fd255a3 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/hook.rs @@ -0,0 +1,14 @@ +use crate::framework::run::{TextFlow, ToolCallFlow}; +use crate::framework::tool::ToolCall; + +pub trait Hook: Send + Sync { + fn on_text_reply(&self, _text: &str) -> TextFlow { + TextFlow::Continue + } + fn before_tool_call(&self, _tool_call: &ToolCall) -> ToolCallFlow { + ToolCallFlow::Continue + } + fn after_tool_call(&self, _tool_name: &str, _output: &mut serde_json::Value) -> ToolCallFlow { + ToolCallFlow::Continue + } +} diff --git a/server-rs/crates/platform-editor-agent/src/framework/memory.rs b/server-rs/crates/platform-editor-agent/src/framework/memory.rs new file mode 100644 index 000000000..3157d7387 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/memory.rs @@ -0,0 +1,31 @@ +use serde::{Deserialize, Serialize}; + +pub trait AgentMemory: Send + Sync { + fn get_memory(&self) -> &[Message]; + fn append_message(&mut self, message: Message); +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VecMemory { + messages: Vec, +} + +impl VecMemory { + pub fn new(messages: Vec) -> Self { + Self { messages } + } + + pub fn into_inner(self) -> Vec { + self.messages + } +} + +impl AgentMemory for VecMemory { + fn get_memory(&self) -> &[Message] { + &self.messages + } + + fn append_message(&mut self, message: Message) { + self.messages.push(message); + } +} diff --git a/server-rs/crates/platform-editor-agent/src/framework/mod.rs b/server-rs/crates/platform-editor-agent/src/framework/mod.rs new file mode 100644 index 000000000..0f84bfb15 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/mod.rs @@ -0,0 +1,7 @@ +pub mod agent; +pub mod agent_builder; +pub mod error; +pub mod hook; +pub mod memory; +pub mod run; +pub mod tool; diff --git a/server-rs/crates/platform-editor-agent/src/framework/run.rs b/server-rs/crates/platform-editor-agent/src/framework/run.rs new file mode 100644 index 000000000..93327d10f --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/run.rs @@ -0,0 +1,284 @@ +use crate::framework::agent::Agent; +use crate::framework::agent::LlmApiAdaptor; +use crate::framework::error::PromptError; +use crate::framework::hook::Hook; +use crate::framework::memory::VecMemory; +use crate::framework::run::PromptOutput::{Text, Tool}; +use crate::framework::tool::{ToolCall, ToolDyn, ToolExecutionResult, ToolFailure, ToolOutcome}; +use serde::Deserialize; +use serde_json::Value; +use std::pin::Pin; + +pub type TextOutput = String; + +#[derive(Debug, Clone)] +pub struct ToolCallOutput { + pub tool_call: ToolCall, + pub message: String, +} +#[derive(Debug, Clone)] +pub enum PromptOutput { + Text(TextOutput), + Tool(ToolCallOutput), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TextFlow { + Continue, + Stop, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ToolCallFlow { + Continue, + Skip, + Stop, +} + +pub struct PromptRequest<'a, M: LlmApiAdaptor + 'a, Message: 'a> { + agent: &'a mut Agent, + message: Message, + system_prompt: Option, + hooks: Vec>, + max_turns: usize, +} + +impl<'a, M, Message> PromptRequest<'a, M, Message> +where + M: LlmApiAdaptor + 'a, + Message: 'a, +{ + pub fn new(agent: &'a mut Agent, message: Message) -> Self { + let max_turns = agent.default_max_turns; + Self { + agent, + message, + system_prompt: None, + hooks: Vec::new(), + max_turns, + } + } + + pub fn system_prompt(mut self, msg: Message) -> Self { + self.system_prompt = Some(msg); + self + } + + pub fn add_hook(mut self, hook: impl Hook + 'static) -> Self { + self.hooks.push(Box::new(hook)); + self + } + + pub fn max_turns(mut self, n: usize) -> Self { + self.max_turns = n; + self + } +} + +impl<'a, M, Message> IntoFuture for PromptRequest<'a, M, Message> +where + M: LlmApiAdaptor + Send + Sync + 'a, + Message: Send + Sync + Clone + 'a + 'static, +{ + type Output = Result, PromptError>; + type IntoFuture = Pin + Send + 'a>>; + + fn into_future(self) -> Self::IntoFuture { + let agent = self.agent; + let message = self.message; + let extra_hooks = self.hooks; + let max_turns = self.max_turns; + + Box::pin(async move { + let memory = agent + .memory + .get_or_insert_with(|| Box::new(VecMemory::new(Vec::new()))); + memory.append_message(message); + let mut prompt_result: Vec = Vec::new(); + + for _ in 0..max_turns { + let text = { + let messages = agent + .system_prompt + .iter() + .chain(memory.get_memory().iter()); + agent.model.complete(messages).await? + }; + + // Try to parse the LLM reply as JSON (handle Markdown fences) + let cleaned = clean_json_response(&text); + match serde_json::from_str::(&cleaned) { + Ok(json_resp) => { + let clean_text = json_resp + .reply_text + .trim() + .to_string(); + + // Run on_text_reply hooks + for hook in agent.hooks.iter().chain(extra_hooks.iter()) { + match hook.on_text_reply(&clean_text) { + TextFlow::Stop => { + return Err(PromptError::ToolError( + "text reply rejected by hook".to_string(), + )); + } + TextFlow::Continue => {} + } + } + memory.append_message(agent.model.build_assistant_message(&clean_text)); + prompt_result.push(Text(clean_text.clone())); + + let tool_calls: Vec = json_resp + .tool_calls + .into_iter() + .enumerate() + .map(|(idx, tc)| ToolCall { + id: format!("{idx}"), + name: tc.tool_name, + args: tc.args, + }) + .collect(); + + // no tool call, turn terminate. + if tool_calls.is_empty() { + return Ok(prompt_result); + } + + for (tc_id, tc) in tool_calls.iter().enumerate() { + // inline run_hooks: before_tool_call hook + let mut should_skip = false; + for hook in agent.hooks.iter().chain(extra_hooks.iter()) { + match hook.before_tool_call(tc) { + ToolCallFlow::Stop => { + return Err(PromptError::ToolError( + "tool call rejected by hook".to_string(), + )); + } + ToolCallFlow::Skip => { + let msg = agent + .model + .tool_result_message(&tc.name, "(skipped by hook)"); + memory.append_message(msg); + should_skip = true; + break; + } + ToolCallFlow::Continue => {} + } + } + if should_skip { + continue; + } + + let result = { + let tools: Vec<&Box> = agent.tools.iter().collect(); + let name = tc.name.clone(); + let args = tc.args.clone(); + let fut = async move { + match tools.iter().find(|tool| tool.tool_name() == name) { + Some(tool) => tool.call(args).await, + None => ToolExecutionResult::failed( + Value::Null, + ToolFailure::invalid_args(format!( + "unknown tool: {name}" + )), + ), + } + }; + fut.await + }; + + match result.outcome { + ToolOutcome::InternalOk => { + let mut json_output = result.output; + // Run after_tool_call hooks to allow output modification + for hook in agent.hooks.iter().chain(extra_hooks.iter()) { + match hook.after_tool_call(&tc.name, &mut json_output) { + ToolCallFlow::Stop => { + return Err(PromptError::ToolError( + "tool call output caused this _turn stopped by hook".to_string(), + )); + } + ToolCallFlow::Skip => { + json_output = serde_json::json!({"message":"tool call is ignored by hook"}); + break; + } + ToolCallFlow::Continue => {} + } + } + let arg_json = serde_json::to_string(&tc.args) + .map_err(|e| PromptError::InternalError(e.to_string()))?; + let output_json = serde_json::to_string(&json_output) + .map_err(|e| PromptError::InternalError(e.to_string()))?; + + let overall_message = format!( + "[tool_call:{tc_id}] args: {arg_json} output: {output_json}" + ); + let msg = + agent.model.tool_result_message(&tc.name, &overall_message); + memory.append_message(msg); + prompt_result.push(Tool(ToolCallOutput { + tool_call: tc.clone(), + message: overall_message.clone(), + })) + } + ToolOutcome::InternalError(failure) if failure.fatal => { + return Err(PromptError::ToolError(failure.message)); + } + ToolOutcome::InternalError(failure) => { + let msg = agent.model.tool_result_message( + &tc.name, + &format!("error: {}", failure.message), + ); + memory.append_message(msg); + } + } + } + } + Err(_) => { + // Not valid JSON — push as assistant message and continue to next _turn + memory.append_message(agent.model.build_assistant_message(&text)); + continue; + } + } + } + + Err(PromptError::MaxTurnsReached { max_turns }) + }) + } +} + +#[derive(Deserialize)] +struct LlmJsonResponse { + reply_text: String, + #[serde(default)] + tool_calls: Vec, +} + +#[derive(Deserialize)] +struct LlmToolCallRequest { + tool_name: String, + #[serde(default)] + args: Value, +} + +pub fn clean_json_response(text: &str) -> String { + let text = text.trim(); + if text.starts_with("```") { + let lines: Vec<&str> = text.lines().collect(); + let mut cleaned = Vec::new(); + let mut in_code = false; + for line in lines { + if line.trim().starts_with("```") { + in_code = !in_code; + continue; + } + if in_code { + cleaned.push(line); + } + } + if !cleaned.is_empty() { + return cleaned.join("\n").trim().to_string(); + } + } + text.to_string() +} diff --git a/server-rs/crates/platform-editor-agent/src/framework/tool.rs b/server-rs/crates/platform-editor-agent/src/framework/tool.rs new file mode 100644 index 000000000..386a9ed36 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/framework/tool.rs @@ -0,0 +1,206 @@ +use serde::{Deserialize, Serialize}; +use std::pin::Pin; + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ToolCall { + pub id: String, + pub name: String, + pub args: serde_json::Value, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ToolCallResult { + pub tool_name: String, + pub output: serde_json::Value, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum ToolFailureKind { + InvalidArgs, + Timeout, + Cancelled, + NotFound, + PermissionDenied, + RateLimited, + Provider, + Network, + Internal, + Other, +} + +impl ToolFailureKind { + pub fn default_retryable(self) -> bool { + matches!( + self, + Self::Timeout | Self::RateLimited | Self::Provider | Self::Network + ) + } + + pub fn default_fatal(self) -> bool { + matches!(self, Self::Internal) + } +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ToolFailure { + pub kind: ToolFailureKind, + pub message: String, + pub retryable: bool, + pub fatal: bool, +} + +impl ToolFailure { + pub fn new(kind: ToolFailureKind, message: impl Into) -> Self { + Self { + kind, + message: message.into(), + retryable: kind.default_retryable(), + fatal: kind.default_fatal(), + } + } + + pub fn invalid_args(message: impl Into) -> Self { + Self::new(ToolFailureKind::InvalidArgs, message) + } + + pub fn internal(message: impl Into) -> Self { + Self::new(ToolFailureKind::Internal, message) + } + + pub fn other(message: impl Into) -> Self { + Self::new(ToolFailureKind::Other, message) + } + + pub fn with_retryable(mut self, retryable: bool) -> Self { + self.retryable = retryable; + self + } + + pub fn with_fatal(mut self, fatal: bool) -> Self { + self.fatal = fatal; + self + } +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub enum ToolOutcome { + InternalOk, + InternalError(ToolFailure), +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct ToolExecutionResult { + pub output: serde_json::Value, + pub outcome: ToolOutcome, +} + +impl ToolExecutionResult { + pub fn success(output: serde_json::Value) -> Self { + Self { + output, + outcome: ToolOutcome::InternalOk, + } + } + + pub fn failed(output: serde_json::Value, failure: ToolFailure) -> Self { + Self { + output, + outcome: ToolOutcome::InternalError(failure), + } + } + + pub fn failure(&self) -> Option<&ToolFailure> { + match &self.outcome { + ToolOutcome::InternalOk => None, + ToolOutcome::InternalError(failure) => Some(failure), + } + } + + pub fn is_fatal(&self) -> bool { + self.failure().is_some_and(|failure| failure.fatal) + } +} + +pub trait Tool: Sized { + const NAME: &'static str; + type Error: std::error::Error + 'static; + type Args: for<'a> Deserialize<'a>; + type Output: Serialize; + + fn tool_name(&self) -> &'static str { + Self::NAME + } + + /// Human-readable description of what the tool does. + fn description(&self) -> String; + /// JSON Schema describing the tool's parameters. + fn parameters(&self) -> serde_json::Value; + fn call( + &self, + args: Self::Args, + ) -> impl Future> + Send; + + fn classify_error(&self, error: &Self::Error) -> ToolFailure { + ToolFailure::other(error.to_string()) + } +} + +/// Wrapper trait to allow for dynamic dispatch of simple tools. +pub trait ToolDyn: Send + Sync { + fn tool_name(&self) -> &'static str; + fn description(&self) -> String; + fn parameters(&self) -> serde_json::Value; + fn call( + &self, + args: serde_json::Value, + ) -> Pin + Send + '_>>; +} + +impl ToolDyn for T { + fn tool_name(&self) -> &'static str { + T::NAME + } + + fn description(&self) -> String { + self.description() + } + + fn parameters(&self) -> serde_json::Value { + self.parameters() + } + + fn call( + &self, + args: serde_json::Value, + ) -> Pin + Send + '_>> { + Box::pin(async move { + let parsed: T::Args = match serde_json::from_value(args) { + Ok(parsed) => parsed, + Err(error) => { + return ToolExecutionResult::failed( + serde_json::Value::Null, + ToolFailure::invalid_args(format!("bad args for {}: {error}", T::NAME)), + ); + } + }; + + let output = match self.call(parsed).await { + Ok(output) => output, + Err(error) => { + return ToolExecutionResult::failed( + serde_json::Value::Null, + self.classify_error(&error), + ); + } + }; + + match serde_json::to_value(&output) { + Ok(output) => ToolExecutionResult::success(output), + Err(error) => ToolExecutionResult::failed( + serde_json::Value::Null, + ToolFailure::internal(error.to_string()), + ), + } + }) + } +} diff --git a/server-rs/crates/platform-editor-agent/src/lib.rs b/server-rs/crates/platform-editor-agent/src/lib.rs new file mode 100644 index 000000000..9bfb25278 --- /dev/null +++ b/server-rs/crates/platform-editor-agent/src/lib.rs @@ -0,0 +1,2 @@ +pub mod agent; +pub mod framework; diff --git a/server-rs/crates/shared-contracts/src/editor_agent.rs b/server-rs/crates/shared-contracts/src/editor_agent.rs index c3790a276..924e885e6 100644 --- a/server-rs/crates/shared-contracts/src/editor_agent.rs +++ b/server-rs/crates/shared-contracts/src/editor_agent.rs @@ -1,47 +1,21 @@ //! 画布Agent对话契约：与 packages/shared/src/contracts/editorAgent.ts 保持同形。 //! 会话元数据存 SpacetimeDB，消息正文整体存 OSS（editor-agent/{conversationId}.json）。 -use serde::{Deserialize, Serialize}; +use serde::{Deserialize, Deserializer, Serialize}; +use serde_json::json; pub const EDITOR_AGENT_MAX_ATTACHMENTS: usize = 9; +pub const EDITOR_AGENT_ERROR_MESSAGE_PREFIX: &str = "ERROR "; pub const EDITOR_AGENT_TITLE_MAX_CHARS: usize = 20; pub const EDITOR_AGENT_DEFAULT_CONVERSATION_TITLE: &str = "新对话"; -pub const EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION: u32 = 1; - -#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum EditorAgentStage { - Idle, - Thinking, - Responding, - Generating, - Completed, - Failed, -} +pub const EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION: u32 = 2; #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "lowercase")] pub enum EditorAgentMessageRole { User, Assistant, -} - -#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum EditorAgentMessageKind { - Chat, - Stage, - Error, -} - -#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum EditorAgentMessageStatus { - Streaming, - Generating, - Completed, - Failed, - Stopped, + System, } #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] @@ -53,20 +27,16 @@ pub enum EditorAgentAttachmentSource { #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] -pub enum EditorAgentToolName { - GenerateImage, - EditImage, - GenerateCharacter, - GenerateIconSpritesheet, - GenerateUiDesign, -} - -#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "snake_case")] -pub enum EditorAgentGenerationStatus { - Generating, +pub enum EditorAgentToolCallStatus { + #[serde(alias = "pending_confirmation", alias = "executing")] + NotCompleted, Completed, Failed, + Cancelled, +} + +fn default_editor_agent_tool_call_status() -> EditorAgentToolCallStatus { + EditorAgentToolCallStatus::NotCompleted } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] @@ -107,44 +77,165 @@ pub struct EditorAgentGeneratedImage { #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] -pub struct EditorAgentGenerationRecord { - pub tool_call_id: String, - pub tool_name: EditorAgentToolName, +pub struct EditorAgentGeneratedVideo { #[serde(default)] - pub summary: Option, + pub resource_id: Option, #[serde(default)] - pub task_id: Option, - pub status: EditorAgentGenerationStatus, + pub object_key: Option, #[serde(default)] - pub model: Option, + pub asset_object_id: Option, + pub video_src: String, #[serde(default)] - pub images: Vec, + pub thumbnail_src: Option, #[serde(default)] - pub error: Option, + pub width: Option, + #[serde(default)] + pub height: Option, } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] +pub struct EditorAgentGeneratedAudio { + #[serde(default)] + pub resource_id: Option, + #[serde(default)] + pub object_key: Option, + #[serde(default)] + pub asset_object_id: Option, + pub audio_src: String, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorAgentToolCallDisplayArgs { + pub string_args: Vec, + pub image_args: Vec, + pub extras: EditorAgentToolCallDisplayExtras, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorAgentToolCallDisplayExtras { + pub price_mud_points: u32, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorAgentToolCallStringArg { + pub name: String, + pub label: String, + pub value: String, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorAgentToolCallImageArg { + pub name: String, + pub label: String, + pub refs: Vec, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorAgentToolCallImageRef { + pub image_id: String, + pub image_src: String, + #[serde(default)] + pub object_key: Option, + #[serde(default)] + pub thumbnail_src: Option, + #[serde(default)] + pub label: Option, + #[serde(default)] + pub width: Option, + #[serde(default)] + pub height: Option, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorAgentToolCall { + pub tool_name: String, + #[serde(default = "default_editor_agent_tool_call_status")] + pub status: EditorAgentToolCallStatus, + pub args: serde_json::Value, + pub display_args: EditorAgentToolCallDisplayArgs, + #[serde(default)] + pub external_job_id: Option, + #[serde(default)] + pub images: Vec, + #[serde(default)] + pub videos: Vec, + #[serde(default)] + pub audios: Vec, + #[serde(default)] + pub error: Option, +} +#[derive(Clone, Debug, Deserialize)] +#[serde(untagged)] +enum RawEditorAgentMessageId { + Number(usize), + String(String), + #[allow(dead_code)] + Other(serde_json::Value), +} + +impl Default for RawEditorAgentMessageId { + fn default() -> Self { + Self::Number(0) + } +} + +impl RawEditorAgentMessageId { + fn numeric_id(&self) -> usize { + match self { + Self::Number(id) => *id, + Self::String(_) | Self::Other(_) => 0, + } + } + + fn legacy_client_message_id(&self, role: EditorAgentMessageRole) -> Option { + if role != EditorAgentMessageRole::User { + return None; + } + match self { + Self::String(id) if !id.trim().is_empty() => Some(id.trim().to_string()), + Self::Number(_) | Self::String(_) | Self::Other(_) => None, + } + } +} + +#[derive(Clone, Debug, Serialize, PartialEq)] +#[serde(rename_all = "camelCase")] pub struct EditorAgentMessage { - pub id: String, + // to compatible for legacy version + pub id: usize, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub client_message_id: Option, pub role: EditorAgentMessageRole, - pub kind: EditorAgentMessageKind, pub text: String, #[serde(default)] pub attachments: Vec, - #[serde(default)] - pub generations: Vec, - pub status: EditorAgentMessageStatus, + // one tool call is inside one system message. + pub tool_call: Option, pub created_at: String, } +impl<'de> Deserialize<'de> for EditorAgentMessage { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + Ok(RawEditorAgentMessage::deserialize(deserializer)?.into_single_message()) + } +} + #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorAgentConversationSummary { pub conversation_id: String, pub project_id: String, pub title: String, - pub created_at: String, pub updated_at: String, } @@ -159,7 +250,7 @@ pub struct EditorAgentConversationDetail { pub messages: Vec, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[derive(Clone, Debug, Serialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorAgentConversationMessagesDocument { pub version: u32, @@ -167,6 +258,235 @@ pub struct EditorAgentConversationMessagesDocument { pub messages: Vec, } +// for legacy migration start +impl<'de> Deserialize<'de> for EditorAgentConversationMessagesDocument { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let raw = RawEditorAgentConversationMessagesDocument::deserialize(deserializer)?; + let mut messages = Vec::new(); + for raw_message in raw.messages { + for mut message in raw_message.into_document_messages() { + message.id = messages.len(); + messages.push(message); + } + } + Ok(Self { + version: EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION, + conversation_id: raw.conversation_id, + messages, + }) + } +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct RawEditorAgentConversationMessagesDocument { + #[allow(dead_code)] + #[serde(default)] + version: u32, + conversation_id: String, + #[serde(default)] + messages: Vec, +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct RawEditorAgentMessage { + #[serde(default)] + id: RawEditorAgentMessageId, + #[serde(default)] + client_message_id: Option, + role: EditorAgentMessageRole, + #[serde(default)] + text: String, + #[serde(default)] + attachments: Vec, + #[serde(default)] + tool_call: Option, + #[serde(default)] + generations: Vec, + #[serde(default)] + created_at: String, +} + +impl RawEditorAgentMessage { + fn into_single_message(self) -> EditorAgentMessage { + let id = self.id.numeric_id(); + let client_message_id = self + .client_message_id + .or_else(|| self.id.legacy_client_message_id(self.role)); + let tool_call = self + .tool_call + .or_else(|| legacy_generations_to_tool_call(self.generations)); + EditorAgentMessage { + id, + client_message_id, + role: self.role, + text: self.text, + attachments: self.attachments, + tool_call, + created_at: self.created_at, + } + } + + fn into_document_messages(self) -> Vec { + if self.tool_call.is_some() || self.generations.is_empty() { + return vec![self.into_single_message()]; + } + + let id = self.id.numeric_id(); + let client_message_id = self + .client_message_id + .clone() + .or_else(|| self.id.legacy_client_message_id(self.role)); + let mut messages = Vec::new(); + if self.role != EditorAgentMessageRole::Assistant + || !self.text.trim().is_empty() + || !self.attachments.is_empty() + { + messages.push(EditorAgentMessage { + id, + client_message_id, + role: self.role, + text: self.text.clone(), + attachments: self.attachments, + tool_call: None, + created_at: self.created_at.clone(), + }); + } + + messages.extend(self.generations.into_iter().map(|generation| { + let tool_call_message = + legacy_generation_to_tool_call_message(generation, self.created_at.as_str()); + EditorAgentMessage { + id, + client_message_id: None, + role: EditorAgentMessageRole::System, + text: tool_call_message.text, + attachments: Vec::new(), + tool_call: Some(tool_call_message.tool_call), + created_at: self.created_at.clone(), + } + })); + messages + } +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase")] +struct LegacyEditorAgentGenerationRecord { + #[serde(default)] + tool_call_id: String, + tool_name: String, + #[serde(default)] + summary: Option, + #[serde(default)] + task_id: Option, + status: LegacyEditorAgentGenerationStatus, + #[serde(default)] + model: Option, + #[serde(default)] + images: Vec, + #[serde(default)] + error: Option, +} + +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +enum LegacyEditorAgentGenerationStatus { + Generating, + Completed, + Failed, +} + +struct LegacyToolCallMessage { + text: String, + tool_call: EditorAgentToolCall, +} + +fn legacy_generations_to_tool_call( + generations: Vec, +) -> Option { + let mut generations = generations.into_iter(); + let first = generations.next()?; + let mut tool_call_message = legacy_generation_to_tool_call_message(first, ""); + for generation in generations { + tool_call_message.tool_call.images.extend(generation.images); + if tool_call_message.tool_call.error.is_none() { + tool_call_message.tool_call.error = generation.error; + } + if tool_call_message.tool_call.status == EditorAgentToolCallStatus::Completed + && generation.status != LegacyEditorAgentGenerationStatus::Completed + { + tool_call_message.tool_call.status = legacy_generation_status(generation.status); + } + } + Some(tool_call_message.tool_call) +} + +fn legacy_generation_to_tool_call_message( + generation: LegacyEditorAgentGenerationRecord, + created_at: &str, +) -> LegacyToolCallMessage { + let tool_name = normalize_legacy_tool_name(generation.tool_name.as_str()); + let status = legacy_generation_status(generation.status); + let summary = generation.summary.clone(); + let error = generation.error.clone(); + let text = legacy_tool_call_text(tool_name.as_str(), summary.as_deref(), error.as_deref()); + LegacyToolCallMessage { + text, + tool_call: EditorAgentToolCall { + tool_name, + status, + args: json!({ + "legacyToolCallId": generation.tool_call_id, + "legacySummary": summary, + "legacyModel": generation.model, + "legacyCreatedAt": created_at, + }), + display_args: EditorAgentToolCallDisplayArgs::default(), + external_job_id: generation.task_id, + images: generation.images, + videos: Vec::new(), + audios: Vec::new(), + error, + }, + } +} + +fn legacy_generation_status( + status: LegacyEditorAgentGenerationStatus, +) -> EditorAgentToolCallStatus { + match status { + LegacyEditorAgentGenerationStatus::Generating => EditorAgentToolCallStatus::NotCompleted, + LegacyEditorAgentGenerationStatus::Completed => EditorAgentToolCallStatus::Completed, + LegacyEditorAgentGenerationStatus::Failed => EditorAgentToolCallStatus::Failed, + } +} + +fn normalize_legacy_tool_name(tool_name: &str) -> String { + let normalized = tool_name.trim().replace('_', "-"); + if normalized.is_empty() { + "generate-image".to_string() + } else { + normalized + } +} + +fn legacy_tool_call_text(tool_name: &str, summary: Option<&str>, error: Option<&str>) -> String { + if let Some(error) = error.map(str::trim).filter(|value| !value.is_empty()) { + return format!("[tool_call:{tool_name}] output: {error}"); + } + let output = summary + .map(str::trim) + .filter(|value| !value.is_empty()) + .unwrap_or("legacy generation restored"); + format!("[tool_call:{tool_name}] output: {output}") +} +// for legacy migration end + #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct CreateEditorAgentConversationRequest { @@ -188,96 +508,19 @@ pub struct EditorAgentConversationResponse { #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] -pub struct StreamEditorAgentMessageRequest { +pub struct EditorAgentMessageRequest { pub client_message_id: String, pub text: String, #[serde(default)] pub attachments: Vec, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "snake_case")] -pub enum EditorAgentSseEventType { - Stage, - MessageDelta, - ToolStarted, - ToolCompleted, - GenerationResult, - Error, - Done, -} - #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] -pub struct EditorAgentSseEnvelope { - pub event: EditorAgentSseEventType, - pub data: serde_json::Value, -} - -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct EditorAgentStageEvent { - pub conversation_id: String, - pub stage: EditorAgentStage, -} - -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct EditorAgentMessageDeltaEvent { - pub conversation_id: String, - pub message_id: String, - pub role: EditorAgentMessageRole, - pub kind: EditorAgentMessageKind, - pub text_delta: String, -} - -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct EditorAgentToolEvent { - pub conversation_id: String, - pub message_id: String, - pub tool_call_id: String, - pub tool_name: EditorAgentToolName, - #[serde(default)] - pub summary: Option, - #[serde(default)] - pub task_id: Option, - #[serde(default)] - pub model: Option, - #[serde(default)] - pub status: Option, - #[serde(default)] - pub error: Option, -} - -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct EditorAgentGenerationResultEvent { - pub conversation_id: String, - pub message_id: String, - pub tool_call_id: String, - pub tool_name: EditorAgentToolName, - #[serde(default)] - pub model: Option, - pub images: Vec, -} - -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct EditorAgentErrorEvent { - #[serde(default)] - pub conversation_id: Option, - pub code: String, - pub message: String, - pub recoverable: bool, -} - -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] -#[serde(rename_all = "camelCase")] -pub struct EditorAgentDoneEvent { - pub conversation_id: String, - #[serde(default)] - pub title: Option, +pub struct EditorAgentMessageResponse { + pub conversation: EditorAgentConversationSummary, + pub delta_messages: Vec, + pub error_message: Option, } #[cfg(test)] @@ -286,111 +529,220 @@ mod tests { use serde_json::json; #[test] - fn editor_agent_conversation_detail_uses_camel_case() { - let detail = EditorAgentConversationDetail { - conversation_id: "editor-agent-conv-1".to_string(), - project_id: "project-1".to_string(), - title: "新对话".to_string(), - created_at: "2026-07-03T00:00:00Z".to_string(), - updated_at: "2026-07-03T00:00:01Z".to_string(), - messages: vec![EditorAgentMessage { - id: "message-1".to_string(), - role: EditorAgentMessageRole::User, - kind: EditorAgentMessageKind::Chat, - text: "帮我生成一张森林背景".to_string(), - attachments: vec![EditorAgentAttachmentRef { - source: EditorAgentAttachmentSource::CanvasResource, - reference_id: "resource-1".to_string(), - object_key: Some("editor/resource-1.png".to_string()), - image_src: "https://example.test/resource-1.png".to_string(), - thumbnail_src: None, - label: Some("森林参考".to_string()), - width: Some(1024), - height: Some(768), - }], - generations: vec![], - status: EditorAgentMessageStatus::Completed, - created_at: "2026-07-03T00:00:00Z".to_string(), - }], - }; + fn message_request_requires_client_message_id_and_user_message_preserves_it() { + let missing_client_message_id = + serde_json::from_value::(json!({ + "text": "生成一张图", + "attachments": [] + })); + assert!(missing_client_message_id.is_err()); - let payload = serde_json::to_value(&detail).expect("detail should serialize"); - assert_eq!(payload["conversationId"], json!("editor-agent-conv-1")); - assert_eq!(payload["messages"][0]["role"], json!("user")); - assert_eq!( - payload["messages"][0]["attachments"][0]["source"], - json!("canvas_resource") - ); - assert_eq!( - payload["messages"][0]["attachments"][0]["referenceId"], - json!("resource-1") - ); - assert_eq!(payload["messages"][0]["status"], json!("completed")); - - let decoded: EditorAgentConversationDetail = - serde_json::from_value(payload).expect("detail should deserialize"); - assert_eq!(decoded, detail); - } - - #[test] - fn editor_agent_sse_events_serialize_event_names() { - let envelope = EditorAgentSseEnvelope { - event: EditorAgentSseEventType::GenerationResult, - data: serde_json::to_value(EditorAgentGenerationResultEvent { - conversation_id: "editor-agent-conv-1".to_string(), - message_id: "message-2".to_string(), - tool_call_id: "tool-call-1".to_string(), - tool_name: EditorAgentToolName::GenerateImage, - model: Some("z-image".to_string()), - images: vec![EditorAgentGeneratedImage { - resource_id: Some("resource-2".to_string()), - object_key: Some("generated/result.png".to_string()), - asset_object_id: Some("asset-object-2".to_string()), - image_src: "https://example.test/generated.png".to_string(), - thumbnail_src: None, - width: Some(1024), - height: Some(1024), - }], - }) - .expect("event data should serialize"), - }; - - let payload = serde_json::to_value(envelope).expect("envelope should serialize"); - assert_eq!(payload["event"], json!("generation_result")); - assert_eq!(payload["data"]["toolName"], json!("generate_image")); - assert_eq!( - payload["data"]["images"][0]["resourceId"], - json!("resource-2") - ); - assert_eq!( - payload["data"]["images"][0]["objectKey"], - json!("generated/result.png") - ); - - let error_envelope = EditorAgentSseEnvelope { - event: EditorAgentSseEventType::Error, - data: serde_json::to_value(EditorAgentErrorEvent { - conversation_id: Some("editor-agent-conv-1".to_string()), - code: "INSUFFICIENT_BALANCE".to_string(), - message: "泥点余额不足，本次生成未执行".to_string(), - recoverable: true, - }) - .expect("event data should serialize"), - }; - let error_payload = - serde_json::to_value(error_envelope).expect("envelope should serialize"); - assert_eq!(error_payload["event"], json!("error")); - assert_eq!(error_payload["data"]["recoverable"], json!(true)); - } - - #[test] - fn stream_request_defaults_attachments_to_empty() { - let payload = json!({ + let request = serde_json::from_value::(json!({ "clientMessageId": "client-message-1", - "text": "随便聊聊" - }); - let decoded: StreamEditorAgentMessageRequest = - serde_json::from_value(payload).expect("request should deserialize"); - assert!(decoded.attachments.is_empty()); + "text": "生成一张图", + "attachments": [] + })) + .expect("clientMessageId should deserialize"); + assert_eq!(request.client_message_id, "client-message-1"); + + let message = EditorAgentMessage { + id: 0, + client_message_id: Some(request.client_message_id), + role: EditorAgentMessageRole::User, + text: request.text, + attachments: request.attachments, + tool_call: None, + created_at: "2026-07-16T00:00:00Z".to_string(), + }; + let payload = serde_json::to_value(message).expect("message should serialize"); + assert_eq!(payload["id"], 0); + assert_eq!(payload["clientMessageId"], "client-message-1"); + } + + #[test] + fn tool_call_display_args_and_media_use_camel_case() { + let missing_display_args = serde_json::from_value::(json!({ + "toolName": "generate-video", + "args": {}, + "images": [] + })); + assert!(missing_display_args.is_err()); + + let tool_call = EditorAgentToolCall { + tool_name: "generate-video".to_string(), + status: EditorAgentToolCallStatus::Completed, + args: json!({}), + display_args: EditorAgentToolCallDisplayArgs { + string_args: vec![EditorAgentToolCallStringArg { + name: "prompt".to_string(), + label: "提示词".to_string(), + value: "生成像素风角色".to_string(), + }], + image_args: vec![EditorAgentToolCallImageArg { + name: "reference_image_ids".to_string(), + label: "参考图片".to_string(), + refs: vec![EditorAgentToolCallImageRef { + image_id: "sha256:reference-1".to_string(), + image_src: "/reference.png".to_string(), + object_key: Some("generated/reference.png".to_string()), + thumbnail_src: Some("/reference-thumbnail.png".to_string()), + label: Some("角色参考".to_string()), + width: Some(512), + height: Some(512), + }], + }], + extras: EditorAgentToolCallDisplayExtras { + price_mud_points: 5, + }, + }, + external_job_id: Some("task-editor-agent-1".to_string()), + images: Vec::new(), + videos: vec![EditorAgentGeneratedVideo { + resource_id: Some("resource-video-1".to_string()), + object_key: Some("generated/video.mp4".to_string()), + asset_object_id: Some("asset-video-1".to_string()), + video_src: "/generated-video.mp4".to_string(), + thumbnail_src: Some("/generated-video.png".to_string()), + width: Some(1280), + height: Some(720), + }], + audios: vec![EditorAgentGeneratedAudio { + resource_id: Some("resource-audio-1".to_string()), + object_key: Some("generated/audio.mp3".to_string()), + asset_object_id: Some("asset-audio-1".to_string()), + audio_src: "/generated-audio.mp3".to_string(), + }], + error: None, + }; + let payload = serde_json::to_value(tool_call).expect("tool call should serialize"); + assert_eq!(payload["videos"][0]["videoSrc"], "/generated-video.mp4"); + assert_eq!(payload["externalJobId"], "task-editor-agent-1"); + assert_eq!(payload["status"], "completed"); + assert_eq!(payload["audios"][0]["audioSrc"], "/generated-audio.mp3"); + assert_eq!( + payload["displayArgs"]["stringArgs"][0]["value"], + "生成像素风角色" + ); + assert_eq!( + payload["displayArgs"]["imageArgs"][0]["refs"][0]["imageId"], + "sha256:reference-1" + ); + assert_eq!( + payload["displayArgs"]["imageArgs"][0]["refs"][0]["objectKey"], + "generated/reference.png" + ); + assert_eq!(payload["displayArgs"]["extras"]["priceMudPoints"], 5); + } + + #[test] + fn legacy_tool_call_status_maps_to_not_completed_on_read() { + let tool_call = serde_json::from_value::(json!({ + "toolName": "generate-image", + "status": "executing", + "args": {}, + "displayArgs": { + "stringArgs": [], + "imageArgs": [], + "extras": { "priceMudPoints": 3 } + }, + "images": [] + })) + .expect("legacy status should not break persisted OSS documents"); + + let payload = serde_json::to_value(tool_call).expect("tool call should serialize"); + assert_eq!(payload["status"], "not_completed"); + } + + #[test] + fn legacy_message_generations_migrate_to_tool_call_media() { + let document = serde_json::from_value::(json!({ + "version": 1, + "conversationId": "editor-agent-conv-legacy", + "messages": [ + { + "id": "message-user-1", + "role": "user", + "kind": "chat", + "text": "帮我生成一张森林背景", + "attachments": [], + "generations": [], + "status": "completed", + "createdAt": "2026-07-03T00:00:00Z" + }, + { + "id": "message-assistant-1", + "role": "assistant", + "kind": "chat", + "text": "已为你生成森林背景。", + "attachments": [], + "generations": [ + { + "toolCallId": "tool-call-legacy-1", + "toolName": "generate_image", + "summary": "森林背景", + "taskId": "task-legacy-1", + "status": "completed", + "model": "z-image", + "images": [ + { + "resourceId": "resource-legacy-1", + "objectKey": "generated/legacy-forest.png", + "assetObjectId": "asset-legacy-1", + "imageSrc": "https://example.test/legacy-forest.png", + "thumbnailSrc": "https://example.test/legacy-forest-thumb.png", + "width": 1024, + "height": 1024 + } + ] + } + ], + "status": "completed", + "createdAt": "2026-07-03T00:00:01Z" + } + ] + })) + .expect("legacy document should deserialize"); + + assert_eq!(document.version, EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION); + assert_eq!(document.messages.len(), 3); + assert_eq!(document.messages[0].id, 0); + assert_eq!( + document.messages[0].client_message_id.as_deref(), + Some("message-user-1") + ); + assert_eq!(document.messages[1].id, 1); + assert!(document.messages[1].client_message_id.is_none()); + assert_eq!(document.messages[1].role, EditorAgentMessageRole::Assistant); + assert_eq!(document.messages[1].text, "已为你生成森林背景。"); + assert!(document.messages[1].tool_call.is_none()); + assert_eq!(document.messages[2].id, 2); + assert_eq!(document.messages[2].role, EditorAgentMessageRole::System); + + let tool_call = document.messages[2] + .tool_call + .as_ref() + .expect("legacy generation should become a tool call"); + assert_eq!(tool_call.tool_name, "generate-image"); + assert_eq!(tool_call.status, EditorAgentToolCallStatus::Completed); + assert_eq!(tool_call.external_job_id.as_deref(), Some("task-legacy-1")); + assert_eq!(tool_call.images.len(), 1); + assert_eq!( + tool_call.images[0].object_key.as_deref(), + Some("generated/legacy-forest.png") + ); + assert_eq!( + tool_call.images[0].image_src, + "https://example.test/legacy-forest.png" + ); + + let payload = serde_json::to_value(&document).expect("document should serialize"); + assert_eq!(payload["version"], EDITOR_AGENT_MESSAGES_DOCUMENT_VERSION); + assert!(payload["messages"][1].get("generations").is_none()); + assert!(payload["messages"][1].get("kind").is_none()); + assert!(payload["messages"][2].get("generations").is_none()); + assert_eq!( + payload["messages"][2]["toolCall"]["images"][0]["objectKey"], + "generated/legacy-forest.png" + ); } } diff --git a/server-rs/crates/spacetime-client/src/external_generation.rs b/server-rs/crates/spacetime-client/src/external_generation.rs index 0d00c249a..c7a0bedce 100644 --- a/server-rs/crates/spacetime-client/src/external_generation.rs +++ b/server-rs/crates/spacetime-client/src/external_generation.rs @@ -385,6 +385,32 @@ impl SpacetimeClient { .await } + // TODO icons gen's result query is not prepared for now. + pub async fn get_external_generation_job_generated_artifacts( + &self, + input: ExternalGenerationJobGetRecordInput, + ) -> Result { + let procedure_input = input.into(); + + self.call_after_connect( + "get_external_generation_job_result_and_return", + move |connection, sender| { + connection + .procedures() + .get_external_generation_job_result_and_return_then( + procedure_input, + move |_, result| { + let mapped = result + .map_err(SpacetimeClientError::from_sdk_error) + .and_then(map_external_generation_job_result_procedure_generated_artifacts); + send_once(&sender, mapped); + }, + ); + }, + ) + .await + } + pub async fn get_external_generation_job_summary( &self, input: ExternalGenerationJobGetRecordInput, diff --git a/server-rs/crates/spacetime-client/src/lib.rs b/server-rs/crates/spacetime-client/src/lib.rs index 80e5ad3c1..614943d65 100644 --- a/server-rs/crates/spacetime-client/src/lib.rs +++ b/server-rs/crates/spacetime-client/src/lib.rs @@ -6,132 +6,133 @@ mod mapper; mod telemetry; use mapper::*; pub use mapper::{ - AdminAccountCreateRecordInput, AdminAccountCredentialRecord, AdminAccountRecord, - AdminAccountUpdateRecordInput, AdminDashboardBreakdownRecord, AdminDashboardDailyCountRecord, - AdminDashboardPaymentConversionRecord, AdminDashboardRetentionRecord, - AdminDashboardStatsRecord, AdminDashboardStatsRecordInput, AdminEditorAssetListRecordInput, - AdminEditorAssetRecord, AdminWorkVisibilityRecord, AiResultReferenceRecord, - AiTaskMutationRecord, AiTaskRecord, AiTaskStageRecord, AiTextChunkRecord, - BarkBattleDraftConfigRecord, BarkBattleRunRecord, BarkBattleRuntimeConfigRecord, - BattleStateRecord, BigFishAgentMessageRecord, BigFishAnchorItemRecord, BigFishAnchorPackRecord, - BigFishAssetCoverageRecord, BigFishAssetGenerateRecordInput, BigFishAssetSlotRecord, - BigFishBackgroundBlueprintRecord, BigFishDraftCompileRecordInput, BigFishGameDraftRecord, - BigFishInputSubmitRecordInput, BigFishLevelBlueprintRecord, BigFishLikeReportRecordInput, - BigFishMessageFinalizeRecordInput, BigFishMessageSubmitRecordInput, - BigFishPlayReportRecordInput, BigFishRunStartRecordInput, BigFishRuntimeEntityRecord, - BigFishRuntimeParamsRecord, BigFishRuntimeRunRecord, BigFishSessionCreateRecordInput, - BigFishSessionRecord, BigFishVector2Record, BigFishWorkRemixRecordInput, - BigFishWorkSummaryRecord, CreationEntryConfigRecord, CustomWorldAgentActionExecuteRecord, - CustomWorldAgentActionExecuteRecordInput, CustomWorldAgentCheckpointRecord, - CustomWorldAgentMessageFinalizeRecordInput, CustomWorldAgentMessageRecord, - CustomWorldAgentMessageSubmitRecordInput, CustomWorldAgentOperationProgressRecordInput, - CustomWorldAgentOperationRecord, CustomWorldAgentSessionCreateRecordInput, - CustomWorldAgentSessionRecord, CustomWorldCheckpointRecord, CustomWorldDraftCardDetailRecord, - CustomWorldDraftCardDetailSectionRecord, CustomWorldDraftCardRecord, - CustomWorldGalleryEntryRecord, CustomWorldLibraryEntryRecord, CustomWorldLibraryMutationRecord, - CustomWorldProfileLikeReportRecordInput, CustomWorldProfilePlayReportRecordInput, - CustomWorldProfileRemixRecordInput, CustomWorldProfileUpsertRecordInput, - CustomWorldPublishGateRecord, CustomWorldPublishWorldRecord, - CustomWorldPublishWorldRecordInput, CustomWorldPublishedProfileCompileRecord, - CustomWorldResultPreviewBlockerRecord, CustomWorldSupportedActionRecord, - CustomWorldWorkSummaryRecord, EditorAgentConversationCreateRecordInput, - EditorAgentConversationDeleteRecordInput, EditorAgentConversationRecord, - EditorAgentConversationTouchRecordInput, EditorAssetCreateRecordInput, - EditorAssetDeleteRecordInput, EditorAssetFolderCreateRecordInput, - EditorAssetFolderDeleteRecordInput, EditorAssetFolderRecord, - EditorAssetFolderUpdateRecordInput, EditorAssetGroupCohortCompleteRecordInput, - EditorAssetGroupSourceLookupRecordInput, EditorAssetLibraryRecord, - EditorAssetMediaRepairRecordInput, EditorAssetRecord, EditorAssetUpdateRecordInput, - EditorCanvasRecord, EditorCanvasViewportRecord, EditorGenerationModelPricingRecord, - EditorGenerationPricingConfigRecord, EditorGenerationPricingConfigUpsertRecordInput, - EditorGenerationPricingTierRecord, EditorProjectCreateRecordInput, - EditorProjectDeleteRecordInput, EditorProjectGetRecordInput, EditorProjectLayoutSaveAckRecord, - EditorProjectLayoutSaveRecordInput, EditorProjectRecord, EditorProjectRenameRecordInput, - EditorProjectResourceCreateRecordInput, EditorProjectResourceMediaRepairRecordInput, - EditorProjectResourcePublicShowcaseListRecordInput, EditorProjectResourceRecord, - EditorProjectResourceShowcaseUpdateRecordInput, EditorShowcaseAssetAdminListRecordInput, - EditorShowcaseAssetAdminReviewRecordInput, EditorShowcaseAssetDisplayUpdateRecordInput, - EditorShowcaseAssetLikeToggleRecordInput, EditorShowcaseAssetPublicListRecordInput, - EditorShowcaseAssetRecord, EditorShowcaseAssetRefundMarkRecordInput, - EditorShowcaseAssetSubmitRecordInput, EditorShowcaseCampaignConfigGetRecordInput, - EditorShowcaseCampaignConfigRecord, EditorShowcaseCampaignConfigUpsertRecordInput, - ExternalApiKeyAuthenticateRecordInput, ExternalApiKeyCreateRecordInput, ExternalApiKeyRecord, - ExternalApiKeyRevokeRecordInput, ExternalGenerationJobAcknowledgeRecordInput, - ExternalGenerationJobClaimRecordInput, ExternalGenerationJobCompleteRecordInput, - ExternalGenerationJobEnqueueRecordInput, ExternalGenerationJobFailRecordInput, - ExternalGenerationJobGetRecordInput, ExternalGenerationJobListRecord, - ExternalGenerationJobListRecordInput, ExternalGenerationJobPhaseUpdateRecordInput, - ExternalGenerationJobRecord, ExternalGenerationJobRenewLeaseRecordInput, - ExternalGenerationJobSummaryListRecord, ExternalGenerationJobSummaryRecord, - ExternalGenerationQueueStatsRecord, FeatureGateConfigRecord, JumpHopActionRequest, - JumpHopActionResponse, JumpHopActionType, JumpHopCharacterAsset, JumpHopDifficulty, - JumpHopDraftResponse, JumpHopGalleryCardResponse, JumpHopGalleryDetailResponse, - JumpHopGalleryResponse, JumpHopGenerationStatus, JumpHopJumpRequest, JumpHopJumpResponse, - JumpHopJumpResult, JumpHopLastJump, JumpHopPath, JumpHopPlatform, JumpHopRestartRunRequest, - JumpHopRunResponse, JumpHopRunStatus, JumpHopRuntimeRunSnapshotResponse, JumpHopScoring, - JumpHopSessionResponse, JumpHopSessionSnapshotResponse, JumpHopStartRunRequest, - JumpHopStylePreset, JumpHopTileAsset, JumpHopTileType, JumpHopWorkDetailResponse, - JumpHopWorkMutationResponse, JumpHopWorkProfileResponse, JumpHopWorkSummaryResponse, - JumpHopWorksResponse, JumpHopWorkspaceCreateRequest, Match3DAgentMessageFinalizeRecordInput, - Match3DAgentMessageRecord, Match3DAgentMessageSubmitRecordInput, - Match3DAgentSessionCreateRecordInput, Match3DAgentSessionRecord, Match3DAnchorItemRecord, - Match3DAnchorPackRecord, Match3DClickConfirmationRecord, Match3DCompileDraftRecordInput, - Match3DCreatorConfigRecord, Match3DItemSnapshotRecord, Match3DResultDraftRecord, - Match3DRunClickRecordInput, Match3DRunRecord, Match3DRunRestartRecordInput, - Match3DRunStartRecordInput, Match3DRunStopRecordInput, Match3DRunTimeUpRecordInput, - Match3DTraySlotRecord, Match3DWorkProfileRecord, Match3DWorkUpdateRecordInput, - NpcBattleInteractionRecord, NpcInteractionRecord, NpcStateRecord, PublicWorkDetailEntryRecord, - PublicWorkGalleryEntryRecord, PuzzleAgentMessageFinalizeRecordInput, PuzzleAgentMessageRecord, - PuzzleAgentMessageSubmitRecordInput, PuzzleAgentSessionCreateRecordInput, - PuzzleAgentSessionRecord, PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord, - PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBackgroundCompileTaskClaimRecordInput, - PuzzleBackgroundCompileTaskReleaseRecordInput, PuzzleBoardRecord, PuzzleCellPositionRecord, - PuzzleClearActionRequest, PuzzleClearActionResponse, PuzzleClearActionType, - PuzzleClearBoardCell, PuzzleClearBoardSnapshot, PuzzleClearCardAsset, PuzzleClearDraftResponse, - PuzzleClearGenerationStatus, PuzzleClearImageAsset, PuzzleClearNextLevelRequest, - PuzzleClearPatternGroup, PuzzleClearRetryLevelRequest, PuzzleClearRunResponse, - PuzzleClearRunStatus, PuzzleClearRuntimeSnapshotResponse, PuzzleClearSessionResponse, - PuzzleClearSessionSnapshotResponse, PuzzleClearStartRunRequest, PuzzleClearSwapRequest, - PuzzleClearTimeUpRequest, PuzzleClearWorkDetailResponse, PuzzleClearWorkMutationResponse, - PuzzleClearWorkProfileResponse, PuzzleClearWorkSummaryResponse, PuzzleClearWorksResponse, - PuzzleClearWorkspaceCreateRequest, PuzzleCreatorIntentRecord, - PuzzleDraftCompileFailureRecordInput, PuzzleDraftLevelRecord, PuzzleFormDraftRecord, - PuzzleFormDraftSaveRecordInput, PuzzleGalleryCardRecord, PuzzleGeneratedImageCandidateRecord, - PuzzleGeneratedImagesSaveRecordInput, PuzzleLeaderboardEntryRecord, - PuzzleLeaderboardSubmitRecordInput, PuzzleLevelGenerationFailureRecordInput, - PuzzleMergedGroupRecord, PuzzlePieceStateRecord, PuzzlePublishRecordInput, - PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord, PuzzleResultPreviewBlockerRecord, - PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord, PuzzleRunDragRecordInput, - PuzzleRunNextLevelRecordInput, PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput, - PuzzleRunRecord, PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput, PuzzleRuntimeLevelRecord, - PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput, - PuzzleWorkLikeReportRecordInput, PuzzleWorkPointIncentiveClaimRecordInput, - PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput, PuzzleWorkUpsertRecordInput, - ResolveCombatActionRecord, ResolveNpcBattleInteractionInput, - SquareHoleAgentMessageFinalizeRecordInput, SquareHoleAgentMessageRecord, - SquareHoleAgentMessageSubmitRecordInput, SquareHoleAgentSessionCreateRecordInput, - SquareHoleAgentSessionRecord, SquareHoleAnchorItemRecord, SquareHoleAnchorPackRecord, - SquareHoleCompileDraftRecordInput, SquareHoleCreatorConfigRecord, - SquareHoleDropConfirmationRecord, SquareHoleDropFeedbackRecord, SquareHoleHoleOptionRecord, - SquareHoleHoleSnapshotRecord, SquareHoleResultDraftRecord, SquareHoleRunDropRecordInput, - SquareHoleRunRecord, SquareHoleRunRestartRecordInput, SquareHoleRunStartRecordInput, - SquareHoleRunStopRecordInput, SquareHoleRunTimeUpRecordInput, SquareHoleShapeOptionRecord, - SquareHoleShapeSnapshotRecord, SquareHoleWorkProfileRecord, SquareHoleWorkUpdateRecordInput, - VisualNovelAgentMessageFinalizeRecordInput, VisualNovelAgentMessageRecord, - VisualNovelAgentMessageSubmitRecordInput, VisualNovelAgentSessionCreateRecordInput, - VisualNovelAgentSessionRecord, VisualNovelHistoryEntryRecord, - VisualNovelHistoryEntryRecordInput, VisualNovelRunRecord, VisualNovelRunSnapshotRecordInput, - VisualNovelRunStartRecordInput, VisualNovelRuntimeEventRecord, - VisualNovelRuntimeEventRecordInput, VisualNovelWorkCompileRecordInput, - VisualNovelWorkProfileRecord, VisualNovelWorkUpdateRecordInput, WoodenFishActionRequest, - WoodenFishActionResponse, WoodenFishActionType, WoodenFishAudioAsset, - WoodenFishCheckpointRunRequest, WoodenFishDraftResponse, WoodenFishFinishRunRequest, - WoodenFishGalleryCardResponse, WoodenFishGalleryDetailResponse, WoodenFishGalleryResponse, - WoodenFishGenerationStatus, WoodenFishImageAsset, WoodenFishRunResponse, WoodenFishRunStatus, - WoodenFishRuntimeRunSnapshotResponse, WoodenFishSessionResponse, - WoodenFishSessionSnapshotResponse, WoodenFishStartRunRequest, WoodenFishWordCounter, - WoodenFishWorkDetailResponse, WoodenFishWorkMutationResponse, WoodenFishWorkProfileResponse, - WoodenFishWorkSummaryResponse, WoodenFishWorkspaceCreateRequest, + AdminAccountCreateRecordInput, AdminAccountCredentialRecord, AdminAccountRecord, + AdminAccountUpdateRecordInput, AdminDashboardBreakdownRecord, AdminDashboardDailyCountRecord, + AdminDashboardPaymentConversionRecord, AdminDashboardRetentionRecord, + AdminDashboardStatsRecord, AdminDashboardStatsRecordInput, AdminEditorAssetListRecordInput, + AdminEditorAssetRecord, AdminWorkVisibilityRecord, AiResultReferenceRecord, + AiTaskMutationRecord, AiTaskRecord, AiTaskStageRecord, AiTextChunkRecord, + BarkBattleDraftConfigRecord, BarkBattleRunRecord, BarkBattleRuntimeConfigRecord, + BattleStateRecord, BigFishAgentMessageRecord, BigFishAnchorItemRecord, BigFishAnchorPackRecord, + BigFishAssetCoverageRecord, BigFishAssetGenerateRecordInput, BigFishAssetSlotRecord, + BigFishBackgroundBlueprintRecord, BigFishDraftCompileRecordInput, BigFishGameDraftRecord, + BigFishInputSubmitRecordInput, BigFishLevelBlueprintRecord, BigFishLikeReportRecordInput, + BigFishMessageFinalizeRecordInput, BigFishMessageSubmitRecordInput, + BigFishPlayReportRecordInput, BigFishRunStartRecordInput, BigFishRuntimeEntityRecord, + BigFishRuntimeParamsRecord, BigFishRuntimeRunRecord, BigFishSessionCreateRecordInput, + BigFishSessionRecord, BigFishVector2Record, BigFishWorkRemixRecordInput, + BigFishWorkSummaryRecord, CreationEntryConfigRecord, CustomWorldAgentActionExecuteRecord, + CustomWorldAgentActionExecuteRecordInput, CustomWorldAgentCheckpointRecord, + CustomWorldAgentMessageFinalizeRecordInput, CustomWorldAgentMessageRecord, + CustomWorldAgentMessageSubmitRecordInput, CustomWorldAgentOperationProgressRecordInput, + CustomWorldAgentOperationRecord, CustomWorldAgentSessionCreateRecordInput, + CustomWorldAgentSessionRecord, CustomWorldCheckpointRecord, CustomWorldDraftCardDetailRecord, + CustomWorldDraftCardDetailSectionRecord, CustomWorldDraftCardRecord, + CustomWorldGalleryEntryRecord, CustomWorldLibraryEntryRecord, CustomWorldLibraryMutationRecord, + CustomWorldProfileLikeReportRecordInput, CustomWorldProfilePlayReportRecordInput, + CustomWorldProfileRemixRecordInput, CustomWorldProfileUpsertRecordInput, + CustomWorldPublishGateRecord, CustomWorldPublishWorldRecord, + CustomWorldPublishWorldRecordInput, CustomWorldPublishedProfileCompileRecord, + CustomWorldResultPreviewBlockerRecord, CustomWorldSupportedActionRecord, + CustomWorldWorkSummaryRecord, EditorAgentConversationCreateRecordInput, + EditorAgentConversationDeleteRecordInput, EditorAgentConversationRecord, + EditorAgentConversationTouchRecordInput, EditorAssetCreateRecordInput, + EditorAssetDeleteRecordInput, EditorAssetFolderCreateRecordInput, + EditorAssetFolderDeleteRecordInput, EditorAssetFolderRecord, + EditorAssetFolderUpdateRecordInput, EditorAssetGroupCohortCompleteRecordInput, + EditorAssetGroupSourceLookupRecordInput, EditorAssetLibraryRecord, + EditorAssetMediaRepairRecordInput, EditorAssetRecord, EditorAssetUpdateRecordInput, + EditorCanvasRecord, EditorCanvasViewportRecord, EditorGenerationModelPricingRecord, + EditorGenerationPricingConfigRecord, EditorGenerationPricingConfigUpsertRecordInput, + EditorGenerationPricingTierRecord, EditorProjectCreateRecordInput, + EditorProjectDeleteRecordInput, EditorProjectGetRecordInput, EditorProjectLayoutSaveAckRecord, + EditorProjectLayoutSaveRecordInput, EditorProjectRecord, EditorProjectRenameRecordInput, + EditorProjectResourceCreateRecordInput, EditorProjectResourceMediaRepairRecordInput, + EditorProjectResourcePublicShowcaseListRecordInput, EditorProjectResourceRecord, + EditorProjectResourceShowcaseUpdateRecordInput, EditorShowcaseAssetAdminListRecordInput, + EditorShowcaseAssetAdminReviewRecordInput, EditorShowcaseAssetDisplayUpdateRecordInput, + EditorShowcaseAssetLikeToggleRecordInput, EditorShowcaseAssetPublicListRecordInput, + EditorShowcaseAssetRecord, EditorShowcaseAssetRefundMarkRecordInput, + EditorShowcaseAssetSubmitRecordInput, EditorShowcaseCampaignConfigGetRecordInput, + EditorShowcaseCampaignConfigRecord, EditorShowcaseCampaignConfigUpsertRecordInput, + ExternalApiKeyAuthenticateRecordInput, ExternalApiKeyCreateRecordInput, ExternalApiKeyRecord, + ExternalApiKeyRevokeRecordInput, ExternalGenerationJobAcknowledgeRecordInput, + ExternalGenerationJobClaimRecordInput, ExternalGenerationJobCompleteRecordInput, + ExternalGenerationJobEnqueueRecordInput, ExternalGenerationJobFailRecordInput, + ExternalGenerationJobGetRecordInput, ExternalGenerationJobListRecord, + ExternalGenerationJobListRecordInput, ExternalGenerationJobPhaseUpdateRecordInput, + ExternalGenerationJobRecord, ExternalGenerationJobRenewLeaseRecordInput, + ExternalGenerationJobArtifactsRecord, ExternalGenerationJobSummaryListRecord, + ExternalGenerationJobSummaryRecord, ExternalGenerationQueueStatsRecord, + FeatureGateConfigRecord, JumpHopActionRequest, JumpHopActionResponse, JumpHopActionType, + JumpHopCharacterAsset, JumpHopDifficulty, JumpHopDraftResponse, JumpHopGalleryCardResponse, + JumpHopGalleryDetailResponse, JumpHopGalleryResponse, JumpHopGenerationStatus, + JumpHopJumpRequest, JumpHopJumpResponse, JumpHopJumpResult, JumpHopLastJump, JumpHopPath, + JumpHopPlatform, JumpHopRestartRunRequest, JumpHopRunResponse, JumpHopRunStatus, + JumpHopRuntimeRunSnapshotResponse, JumpHopScoring, JumpHopSessionResponse, + JumpHopSessionSnapshotResponse, JumpHopStartRunRequest, JumpHopStylePreset, JumpHopTileAsset, + JumpHopTileType, JumpHopWorkDetailResponse, JumpHopWorkMutationResponse, + JumpHopWorkProfileResponse, JumpHopWorkSummaryResponse, JumpHopWorksResponse, + JumpHopWorkspaceCreateRequest, Match3DAgentMessageFinalizeRecordInput, + Match3DAgentMessageRecord, Match3DAgentMessageSubmitRecordInput, + Match3DAgentSessionCreateRecordInput, Match3DAgentSessionRecord, Match3DAnchorItemRecord, + Match3DAnchorPackRecord, Match3DClickConfirmationRecord, Match3DCompileDraftRecordInput, + Match3DCreatorConfigRecord, Match3DItemSnapshotRecord, Match3DResultDraftRecord, + Match3DRunClickRecordInput, Match3DRunRecord, Match3DRunRestartRecordInput, + Match3DRunStartRecordInput, Match3DRunStopRecordInput, Match3DRunTimeUpRecordInput, + Match3DTraySlotRecord, Match3DWorkProfileRecord, Match3DWorkUpdateRecordInput, + NpcBattleInteractionRecord, NpcInteractionRecord, NpcStateRecord, PublicWorkDetailEntryRecord, + PublicWorkGalleryEntryRecord, PuzzleAgentMessageFinalizeRecordInput, PuzzleAgentMessageRecord, + PuzzleAgentMessageSubmitRecordInput, PuzzleAgentSessionCreateRecordInput, + PuzzleAgentSessionRecord, PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord, + PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBackgroundCompileTaskClaimRecordInput, + PuzzleBackgroundCompileTaskReleaseRecordInput, PuzzleBoardRecord, PuzzleCellPositionRecord, + PuzzleClearActionRequest, PuzzleClearActionResponse, PuzzleClearActionType, + PuzzleClearBoardCell, PuzzleClearBoardSnapshot, PuzzleClearCardAsset, PuzzleClearDraftResponse, + PuzzleClearGenerationStatus, PuzzleClearImageAsset, PuzzleClearNextLevelRequest, + PuzzleClearPatternGroup, PuzzleClearRetryLevelRequest, PuzzleClearRunResponse, + PuzzleClearRunStatus, PuzzleClearRuntimeSnapshotResponse, PuzzleClearSessionResponse, + PuzzleClearSessionSnapshotResponse, PuzzleClearStartRunRequest, PuzzleClearSwapRequest, + PuzzleClearTimeUpRequest, PuzzleClearWorkDetailResponse, PuzzleClearWorkMutationResponse, + PuzzleClearWorkProfileResponse, PuzzleClearWorkSummaryResponse, PuzzleClearWorksResponse, + PuzzleClearWorkspaceCreateRequest, PuzzleCreatorIntentRecord, + PuzzleDraftCompileFailureRecordInput, PuzzleDraftLevelRecord, PuzzleFormDraftRecord, + PuzzleFormDraftSaveRecordInput, PuzzleGalleryCardRecord, PuzzleGeneratedImageCandidateRecord, + PuzzleGeneratedImagesSaveRecordInput, PuzzleLeaderboardEntryRecord, + PuzzleLeaderboardSubmitRecordInput, PuzzleLevelGenerationFailureRecordInput, + PuzzleMergedGroupRecord, PuzzlePieceStateRecord, PuzzlePublishRecordInput, + PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord, PuzzleResultPreviewBlockerRecord, + PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord, PuzzleRunDragRecordInput, + PuzzleRunNextLevelRecordInput, PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput, + PuzzleRunRecord, PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput, PuzzleRuntimeLevelRecord, + PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput, + PuzzleWorkLikeReportRecordInput, PuzzleWorkPointIncentiveClaimRecordInput, + PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput, PuzzleWorkUpsertRecordInput, + ResolveCombatActionRecord, ResolveNpcBattleInteractionInput, + SquareHoleAgentMessageFinalizeRecordInput, SquareHoleAgentMessageRecord, + SquareHoleAgentMessageSubmitRecordInput, SquareHoleAgentSessionCreateRecordInput, + SquareHoleAgentSessionRecord, SquareHoleAnchorItemRecord, SquareHoleAnchorPackRecord, + SquareHoleCompileDraftRecordInput, SquareHoleCreatorConfigRecord, + SquareHoleDropConfirmationRecord, SquareHoleDropFeedbackRecord, SquareHoleHoleOptionRecord, + SquareHoleHoleSnapshotRecord, SquareHoleResultDraftRecord, SquareHoleRunDropRecordInput, + SquareHoleRunRecord, SquareHoleRunRestartRecordInput, SquareHoleRunStartRecordInput, + SquareHoleRunStopRecordInput, SquareHoleRunTimeUpRecordInput, SquareHoleShapeOptionRecord, + SquareHoleShapeSnapshotRecord, SquareHoleWorkProfileRecord, SquareHoleWorkUpdateRecordInput, + VisualNovelAgentMessageFinalizeRecordInput, VisualNovelAgentMessageRecord, + VisualNovelAgentMessageSubmitRecordInput, VisualNovelAgentSessionCreateRecordInput, + VisualNovelAgentSessionRecord, VisualNovelHistoryEntryRecord, + VisualNovelHistoryEntryRecordInput, VisualNovelRunRecord, VisualNovelRunSnapshotRecordInput, + VisualNovelRunStartRecordInput, VisualNovelRuntimeEventRecord, + VisualNovelRuntimeEventRecordInput, VisualNovelWorkCompileRecordInput, + VisualNovelWorkProfileRecord, VisualNovelWorkUpdateRecordInput, WoodenFishActionRequest, + WoodenFishActionResponse, WoodenFishActionType, WoodenFishAudioAsset, + WoodenFishCheckpointRunRequest, WoodenFishDraftResponse, WoodenFishFinishRunRequest, + WoodenFishGalleryCardResponse, WoodenFishGalleryDetailResponse, WoodenFishGalleryResponse, + WoodenFishGenerationStatus, WoodenFishImageAsset, WoodenFishRunResponse, WoodenFishRunStatus, + WoodenFishRuntimeRunSnapshotResponse, WoodenFishSessionResponse, + WoodenFishSessionSnapshotResponse, WoodenFishStartRunRequest, WoodenFishWordCounter, + WoodenFishWorkDetailResponse, WoodenFishWorkMutationResponse, WoodenFishWorkProfileResponse, + WoodenFishWorkSummaryResponse, WoodenFishWorkspaceCreateRequest, }; pub mod admin_account; diff --git a/server-rs/crates/spacetime-client/src/mapper.rs b/server-rs/crates/spacetime-client/src/mapper.rs index 9a3614adc..3b4ee8408 100644 --- a/server-rs/crates/spacetime-client/src/mapper.rs +++ b/server-rs/crates/spacetime-client/src/mapper.rs @@ -114,13 +114,14 @@ pub use self::external_api_key::{ ExternalApiKeyRevokeRecordInput, }; pub use self::external_generation::{ - ExternalGenerationJobAcknowledgeRecordInput, ExternalGenerationJobClaimRecordInput, - ExternalGenerationJobCompleteRecordInput, ExternalGenerationJobEnqueueRecordInput, - ExternalGenerationJobFailRecordInput, ExternalGenerationJobGetRecordInput, - ExternalGenerationJobListRecord, ExternalGenerationJobListRecordInput, - ExternalGenerationJobPhaseUpdateRecordInput, ExternalGenerationJobRecord, - ExternalGenerationJobRenewLeaseRecordInput, ExternalGenerationJobSummaryListRecord, - ExternalGenerationJobSummaryRecord, ExternalGenerationQueueStatsRecord, + ExternalGenerationJobAcknowledgeRecordInput, ExternalGenerationJobClaimRecordInput, + ExternalGenerationJobCompleteRecordInput, ExternalGenerationJobEnqueueRecordInput, + ExternalGenerationJobFailRecordInput, ExternalGenerationJobGetRecordInput, + ExternalGenerationJobListRecord, ExternalGenerationJobListRecordInput, + ExternalGenerationJobPhaseUpdateRecordInput, ExternalGenerationJobRecord, + ExternalGenerationJobRenewLeaseRecordInput, ExternalGenerationJobArtifactsRecord, + ExternalGenerationJobSummaryListRecord, ExternalGenerationJobSummaryRecord, + ExternalGenerationQueueStatsRecord, }; pub(crate) use self::external_generation::{ ExternalGenerationJobPhaseUpdateProcedureOutcome, @@ -267,10 +268,12 @@ pub(crate) use self::external_api_key::{ map_external_api_key_list_procedure_result, map_external_api_key_single_procedure_result, }; pub(crate) use self::external_generation::{ - map_external_generation_job_claim_result, map_external_generation_job_list_result, - map_external_generation_job_procedure_result, map_external_generation_job_summary_list_result, - map_external_generation_job_summary_procedure_result, - map_external_generation_queue_stats_result, + map_external_generation_job_claim_result, map_external_generation_job_list_result, + map_external_generation_job_procedure_result, + map_external_generation_job_result_procedure_generated_artifacts, + map_external_generation_job_summary_list_result, + map_external_generation_job_summary_procedure_result, + map_external_generation_queue_stats_result, }; pub(crate) use self::inventory::{ map_runtime_inventory_state_procedure_result, map_runtime_item_reward_item_snapshot, diff --git a/server-rs/crates/spacetime-client/src/mapper/external_generation.rs b/server-rs/crates/spacetime-client/src/mapper/external_generation.rs index 25cd092e2..cf9643c32 100644 --- a/server-rs/crates/spacetime-client/src/mapper/external_generation.rs +++ b/server-rs/crates/spacetime-client/src/mapper/external_generation.rs @@ -123,6 +123,24 @@ pub(crate) fn map_external_generation_job_procedure_result( Ok(map_external_generation_job_snapshot(job)) } +pub(crate) fn map_external_generation_job_result_procedure_generated_artifacts( + result: ExternalGenerationJobResultProcedureResult, +) -> Result { + if !result.ok { + return Err(SpacetimeClientError::procedure_failed(result.error_message)); + } + + let result = result.result.ok_or_else(|| { + SpacetimeClientError::missing_snapshot("external_generation_job 结果快照") + })?; + Ok(ExternalGenerationJobArtifactsRecord { + job_id: result.job_id, + status: result.status, + last_error_message: result.last_error_message, + result_payload_json: result.result_payload_json, + }) +} + pub(crate) enum ExternalGenerationJobPhaseUpdateProcedureOutcome { Updated(ExternalGenerationJobRecord), Rejected { @@ -427,6 +445,14 @@ pub struct ExternalGenerationJobRecord { pub phase: Option, } +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ExternalGenerationJobArtifactsRecord { + pub job_id: String, + pub status: String, + pub last_error_message: Option, + pub result_payload_json: Option, +} + #[derive(Clone, Debug, PartialEq, Eq)] pub struct ExternalGenerationJobListRecord { pub jobs: Vec, diff --git a/server-rs/crates/spacetime-client/src/module_bindings.rs b/server-rs/crates/spacetime-client/src/module_bindings.rs index 875fe740e..13162e635 100644 --- a/server-rs/crates/spacetime-client/src/module_bindings.rs +++ b/server-rs/crates/spacetime-client/src/module_bindings.rs @@ -520,8 +520,10 @@ pub mod external_generation_job_phase_update_failure_kind_type; pub mod external_generation_job_phase_update_input_type; pub mod external_generation_job_phase_update_procedure_result_type; pub mod external_generation_job_procedure_result_type; +pub mod external_generation_job_result_procedure_result_type; pub mod external_generation_job_renew_lease_input_type; pub mod external_generation_job_snapshot_type; +pub mod external_generation_job_result_snapshot_type; pub mod external_generation_job_summary_backfill_input_type; pub mod external_generation_job_summary_backfill_procedure_result_type; pub mod external_generation_job_summary_procedure_result_type; @@ -576,6 +578,7 @@ pub mod get_editor_generation_pricing_config_and_return_procedure; pub mod get_editor_project_and_return_procedure; pub mod get_editor_showcase_campaign_config_and_return_procedure; pub mod get_external_generation_job_and_return_procedure; +pub mod get_external_generation_job_result_and_return_procedure; pub mod get_external_generation_job_summary_and_return_procedure; pub mod get_external_generation_queue_stats_and_return_procedure; pub mod get_feature_gate_config_procedure; @@ -1956,8 +1959,10 @@ pub use external_generation_job_phase_update_failure_kind_type::ExternalGenerati pub use external_generation_job_phase_update_input_type::ExternalGenerationJobPhaseUpdateInput; pub use external_generation_job_phase_update_procedure_result_type::ExternalGenerationJobPhaseUpdateProcedureResult; pub use external_generation_job_procedure_result_type::ExternalGenerationJobProcedureResult; +pub use external_generation_job_result_procedure_result_type::ExternalGenerationJobResultProcedureResult; pub use external_generation_job_renew_lease_input_type::ExternalGenerationJobRenewLeaseInput; pub use external_generation_job_snapshot_type::ExternalGenerationJobSnapshot; +pub use external_generation_job_result_snapshot_type::ExternalGenerationJobResultSnapshot; pub use external_generation_job_summary_backfill_input_type::ExternalGenerationJobSummaryBackfillInput; pub use external_generation_job_summary_backfill_procedure_result_type::ExternalGenerationJobSummaryBackfillProcedureResult; pub use external_generation_job_summary_procedure_result_type::ExternalGenerationJobSummaryProcedureResult; @@ -2012,6 +2017,7 @@ pub use get_editor_generation_pricing_config_and_return_procedure::get_editor_ge pub use get_editor_project_and_return_procedure::get_editor_project_and_return; pub use get_editor_showcase_campaign_config_and_return_procedure::get_editor_showcase_campaign_config_and_return; pub use get_external_generation_job_and_return_procedure::get_external_generation_job_and_return; +pub use get_external_generation_job_result_and_return_procedure::get_external_generation_job_result_and_return; pub use get_external_generation_job_summary_and_return_procedure::get_external_generation_job_summary_and_return; pub use get_external_generation_queue_stats_and_return_procedure::get_external_generation_queue_stats_and_return; pub use get_feature_gate_config_procedure::get_feature_gate_config; diff --git a/server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_result_procedure_result_type.rs b/server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_result_procedure_result_type.rs new file mode 100644 index 000000000..4b07dda7d --- /dev/null +++ b/server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_result_procedure_result_type.rs @@ -0,0 +1,19 @@ +// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE +// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD. + +#![allow(unused, clippy::all)] +use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws}; + +use super::external_generation_job_result_snapshot_type::ExternalGenerationJobResultSnapshot; + +#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)] +#[sats(crate = __lib)] +pub struct ExternalGenerationJobResultProcedureResult { + pub ok: bool, + pub result: Option, + pub error_message: Option, +} + +impl __sdk::InModule for ExternalGenerationJobResultProcedureResult { + type Module = super::RemoteModule; +} diff --git a/server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_result_snapshot_type.rs b/server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_result_snapshot_type.rs new file mode 100644 index 000000000..881d0d786 --- /dev/null +++ b/server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_result_snapshot_type.rs @@ -0,0 +1,18 @@ +// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE +// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD. + +#![allow(unused, clippy::all)] +use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws}; + +#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)] +#[sats(crate = __lib)] +pub struct ExternalGenerationJobResultSnapshot { + pub job_id: String, + pub status: String, + pub last_error_message: Option, + pub result_payload_json: Option, +} + +impl __sdk::InModule for ExternalGenerationJobResultSnapshot { + type Module = super::RemoteModule; +} diff --git a/server-rs/crates/spacetime-client/src/module_bindings/get_external_generation_job_result_and_return_procedure.rs b/server-rs/crates/spacetime-client/src/module_bindings/get_external_generation_job_result_and_return_procedure.rs new file mode 100644 index 000000000..d0ff920cd --- /dev/null +++ b/server-rs/crates/spacetime-client/src/module_bindings/get_external_generation_job_result_and_return_procedure.rs @@ -0,0 +1,59 @@ +// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE +// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD. + +#![allow(unused, clippy::all)] +use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws}; + +use super::external_generation_job_get_input_type::ExternalGenerationJobGetInput; +use super::external_generation_job_result_procedure_result_type::ExternalGenerationJobResultProcedureResult; + +#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)] +#[sats(crate = __lib)] +struct GetExternalGenerationJobResultAndReturnArgs { + pub input: ExternalGenerationJobGetInput, +} + +impl __sdk::InModule for GetExternalGenerationJobResultAndReturnArgs { + type Module = super::RemoteModule; +} + +#[allow(non_camel_case_types)] +/// Extension trait for access to the procedure `get_external_generation_job_result_and_return`. +/// +/// Implemented for [`super::RemoteProcedures`]. +pub trait get_external_generation_job_result_and_return { + fn get_external_generation_job_result_and_return(&self, input: ExternalGenerationJobGetInput) { + self.get_external_generation_job_result_and_return_then(input, |_, _| {}); + } + + fn get_external_generation_job_result_and_return_then( + &self, + input: ExternalGenerationJobGetInput, + + __callback: impl FnOnce( + &super::ProcedureEventContext, + Result, + ) + Send + + 'static, + ); +} + +impl get_external_generation_job_result_and_return for super::RemoteProcedures { + fn get_external_generation_job_result_and_return_then( + &self, + input: ExternalGenerationJobGetInput, + + __callback: impl FnOnce( + &super::ProcedureEventContext, + Result, + ) + Send + + 'static, + ) { + self.imp + .invoke_procedure_with_callback::<_, ExternalGenerationJobResultProcedureResult>( + "get_external_generation_job_result_and_return", + GetExternalGenerationJobResultAndReturnArgs { input }, + __callback, + ); + } +} diff --git a/server-rs/crates/spacetime-module/src/external_generation.rs b/server-rs/crates/spacetime-module/src/external_generation.rs index 9df6e960c..ce1a19396 100644 --- a/server-rs/crates/spacetime-module/src/external_generation.rs +++ b/server-rs/crates/spacetime-module/src/external_generation.rs @@ -292,6 +292,22 @@ pub struct ExternalGenerationJobProcedureResult { pub error_message: Option, } +// Private backend read for reconciliation; it intentionally excludes job request and lease data. +#[derive(Clone, Debug, PartialEq, Eq, SpacetimeType)] +pub struct ExternalGenerationJobResultSnapshot { + pub job_id: String, + pub status: String, + pub last_error_message: Option, + pub result_payload_json: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq, SpacetimeType)] +pub struct ExternalGenerationJobResultProcedureResult { + pub ok: bool, + pub result: Option, + pub error_message: Option, +} + #[derive(Clone, Debug, PartialEq, Eq, SpacetimeType)] pub struct ExternalGenerationJobPhaseUpdateProcedureResult { pub ok: bool, @@ -522,6 +538,23 @@ pub fn get_external_generation_job_and_return( } } +#[spacetimedb::procedure] +pub fn get_external_generation_job_result_and_return( + ctx: &mut ProcedureContext, + input: ExternalGenerationJobGetInput, +) -> ExternalGenerationJobResultProcedureResult { + let caller = ctx.sender(); + match ctx.try_with_tx(|tx| { + crate::editor_project_storage::require_editor_generation_runtime_service_identity( + tx, caller, + )?; + get_external_generation_job_result_tx(tx, input.clone()) + }) { + Ok(result) => single_external_generation_job_result_read_result(result), + Err(message) => failed_external_generation_job_result_read_result(message), + } +} + #[spacetimedb::procedure] pub fn list_external_generation_jobs_and_return( ctx: &mut ProcedureContext, @@ -933,6 +966,34 @@ fn get_external_generation_job_tx( .map(map_external_generation_job_summary_to_compat_snapshot) } +fn get_external_generation_job_result_tx( + ctx: &ReducerContext, + input: ExternalGenerationJobGetInput, +) -> Result { + validate_required("external_generation_job.job_id", &input.job_id)?; + validate_required( + "external_generation_job.owner_user_id", + &input.owner_user_id, + )?; + let job_id = input.job_id.trim().to_string(); + let owner_user_id = input.owner_user_id.trim(); + let row = ctx + .db + .external_generation_job() + .job_id() + .find(&job_id) + .ok_or_else(|| "external_generation_job 不存在".to_string())?; + if row.owner_user_id.trim() != owner_user_id { + return Err("external_generation_job 不存在".to_string()); + } + Ok(ExternalGenerationJobResultSnapshot { + job_id: row.job_id, + status: row.status, + last_error_message: row.last_error_message, + result_payload_json: row.result_payload_json, + }) +} + fn list_external_generation_jobs_tx( ctx: &ReducerContext, input: ExternalGenerationJobListInput, @@ -2275,6 +2336,26 @@ fn failed_external_generation_job_result(message: String) -> ExternalGenerationJ } } +fn single_external_generation_job_result_read_result( + result: ExternalGenerationJobResultSnapshot, +) -> ExternalGenerationJobResultProcedureResult { + ExternalGenerationJobResultProcedureResult { + ok: true, + result: Some(result), + error_message: None, + } +} + +fn failed_external_generation_job_result_read_result( + message: String, +) -> ExternalGenerationJobResultProcedureResult { + ExternalGenerationJobResultProcedureResult { + ok: false, + result: None, + error_message: Some(message), + } +} + fn single_external_generation_job_summary_result( job: ExternalGenerationJobSummarySnapshot, ) -> ExternalGenerationJobSummaryProcedureResult { diff --git a/src/components/ResolvedAssetAudio.tsx b/src/components/ResolvedAssetAudio.tsx new file mode 100644 index 000000000..1cac1c542 --- /dev/null +++ b/src/components/ResolvedAssetAudio.tsx @@ -0,0 +1,33 @@ +import type { AudioHTMLAttributes } from 'react'; + +import { useResolvedAssetReadUrl } from '../hooks/useResolvedAssetReadUrl'; + +type ResolvedAssetAudioProps = Omit< + AudioHTMLAttributes, + 'src' +> & { + src?: string | null; + objectKey?: string | null; + fallbackSrc?: string | null; + refreshKey?: string | number | null; +}; + +export function ResolvedAssetAudio({ + src, + objectKey, + fallbackSrc, + refreshKey, + ...rest +}: ResolvedAssetAudioProps) { + const { resolvedUrl } = useResolvedAssetReadUrl(src, { + objectKey, + refreshKey, + }); + const finalSrc = resolvedUrl || fallbackSrc?.trim() || ''; + + if (!finalSrc) { + return null; + } + + return