Editor agent refactored #76

2026-07-10T19:48:54+08:00

k88936 commented

2026-07-10 19:48:54 +08:00

重构了editor agent.
使得它能进行多轮工具调用，把工具调用的结果嵌入到上下文里。
对于上下文的图片，使用哈希后的id用来引用，不暴露细节信息to llm。
当前实现，状态直接维护在json doc里，需要对整个doc加锁，任务目前只能串行。
把SSE改成一个简单请求（因为生图工具耗时需要二次确认，不需要再实时展示给用户进度）。
增加确认/取消生图操作。
把tool的参数和错误情况做了反馈。

TODO：工具调用的规范/prompt还可以改进。

改用external job来做素材生成，针对来自editor agent 的生成会把生成资产的引用加入到结果json里，
客户端轮询 external job 确定生成状态，发现结束了或者失败了就重新get 会话，后端重新提供会话的时候把结果插入回会话历史里，用来让 ai 引用以及显示

重构了editor agent. 使得它能进行多轮工具调用，把工具调用的结果嵌入到上下文里。对于上下文的图片，使用哈希后的id用来引用，不暴露细节信息to llm。当前实现，状态直接维护在json doc里，需要对整个doc加锁，任务目前只能串行。把SSE改成一个简单请求（因为生图工具耗时需要二次确认，不需要再实时展示给用户进度）。增加确认/取消生图操作。把tool的参数和错误情况做了反馈。 TODO：工具调用的规范/prompt还可以改进。改用external job来做素材生成，针对来自editor agent 的生成会把生成资产的引用加入到结果json里，客户端轮询 external job 确定生成状态，发现结束了或者失败了就重新get 会话，后端重新提供会话的时候把结果插入回会话历史里，用来让 ai 引用以及显示

k88936 added 41 commits 2026-07-10 19:48:55 +08:00

at least it runs 964e7e040a

agent module d639f0207f

tool out put include id 6d13239c1c

continue and retry instead of crash encounter invalid json output d0a4581683

improve error handling for tools: add classification and handling for recoverable and fatal tool failures b1851c0ab8

refactor editor agent structures ae4538a59a

refactor: enhance system prompt handling and tool integration in LlmChatAgent c3493d5458

refactor: simplify memory trait requirements and remove unused tool hooks d3c4b3fe0c

add "System" role to EditorAgentMessageRole enum 4149d35e2d

improve tool call logging and error messaging 71a00056bf

refactor: rename flow enums and improve tool interaction logic 476a555f22

fix return value 36ac392fdc

refactor: simplify agent memory handling and improve message flow logic 73d2b07d52

remove llm_chat_agent example and unused dependencies from module-editor-agent 5499af9ac1

change EditorAgentMessage.id type from String to usize 97d340ad4c

reimpl the agent db2e8cca8d

migrate the api usage a2babca6a6

confirm / cancel api c2b25e9ca0

remove old editor_agent.rs 37b4329fee

clean data structure 2709563c60

update prompt logic in editor_agent to clarify tool call behavior and user interaction d7d888a20d

clean code 19e6da8ff3

Refactor: split EditorAgent conversation components 5fb7ec03d1

Refactor: reuse existing ResolvedAssetVideo eeaa0f4766

Refactor: move editor agent hook 48f165c16c

provide thumbnail for audio and adjust style 084214e2ed

remove SSE-based implementation for editor agent 365ada84e3

replace onEditorAgentGenerationResult with onEditorAgentCanvasRefreshRequested and update related logic 74361af6bd

impl confirm cancel flow e5eee1dd4e

move context to tool own. fix memory not update. 337d358ede

fix terminate cond 2b7f4ffe19

include attachments c077767e0d

fix re enter problem e1ae49f114

impl image gen tool 10d7f09cbf

adjust toolLabel 79b21bc073

fix title display mistake 5444b5cb39

impl image gen tools e652563c23

inject prompt 0911c825d4

extend media display a3f7332dfe

fix mut borrow 5411965762

fix context logic 532b6434bd

k88936 added 2 commits 2026-07-10 19:49:28 +08:00

centralize data key query instead of using obj key. c55a90d23e

update video component to use poster attribute 7077557d8c

k88936 added 4 commits 2026-07-10 20:46:58 +08:00

price compute c623ffed9c

合并画布 Agent 工具计价实现 ffde522b1e

prepare info display to user for review 88368cd900

log and doc 83acd0e7c1

k88936 added 1 commit 2026-07-10 20:52:29 +08:00

Merge branch 'master' into editor-agent-refactored 9d28d7f2b9

# Conflicts:
#	docs/project-memory/shared-memory/decision-log.md

kdletters commented

2026-07-14 11:58:59 +08:00

[P1] 旧 OSS 会话文档会在升级后无法读取。[editor_agent.rs (line 9)](/tmp/genarrative-pr76-review/server-rs/crates/shared-contracts/src/editor_agent.rs:9) 仍将文档版本固定为 1，但 [editor_agent.rs (line 168)](/tmp/genarrative-pr76-review/server-rs/crates/shared-contracts/src/editor_agent.rs:168) 已不兼容地修改持久化消息结构，包括 id: String -> usize、删除旧字段并新增必填形状。读取路径在 [utils.rs (line 354)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/utils.rs:354) 直接按新结构反序列化，没有版本迁移或兼容解析。已有会话将报 message document JSON invalid，无法打开或继续。建议升级文档版本并实现 v1→v2 转换，补旧版 fixture 回归测试。
[P1] 大部分确认接口没有在执行生成前持久化 executing，无法兑现防重复执行契约。[api.rs (line 989)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/api.rs:989) 明确依赖持久化状态阻止重复确认，但视频、音频和多数图片生成分支直接调用 provider，直到完成后才由 [api.rs (line 1560)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/api.rs:1560) 写成 completed；只有编辑图片分支在 [api.rs (line 1348)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/api.rs:1348) 提前写入 executing。如果请求在扣费或任务提交后断线/进程重启，消息仍是 pending_confirmation，用户重试会再次扣费和生成。应在所有工具的外部副作用前统一持久化 executing，并明确崩溃后的恢复策略。

[P1] 旧 OSS 会话文档会在升级后无法读取。[[editor_agent.rs](http://editor_agent.rs/) (line 9)](/tmp/genarrative-pr76-review/server-rs/crates/shared-contracts/src/[editor_agent.rs:9](http://editor_agent.rs:9/)) 仍将文档版本固定为 1，但 [[editor_agent.rs](http://editor_agent.rs/) (line 168)](/tmp/genarrative-pr76-review/server-rs/crates/shared-contracts/src/[editor_agent.rs:168](http://editor_agent.rs:168/)) 已不兼容地修改持久化消息结构，包括 id: String -> usize、删除旧字段并新增必填形状。读取路径在 [[utils.rs](http://utils.rs/) (line 354)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/[utils.rs:354](http://utils.rs:354/)) 直接按新结构反序列化，没有版本迁移或兼容解析。已有会话将报 message document JSON invalid，无法打开或继续。建议升级文档版本并实现 v1→v2 转换，补旧版 fixture 回归测试。 [P1] 大部分确认接口没有在执行生成前持久化 executing，无法兑现防重复执行契约。[[api.rs](http://api.rs/) (line 989)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/[api.rs:989](http://api.rs:989/)) 明确依赖持久化状态阻止重复确认，但视频、音频和多数图片生成分支直接调用 provider，直到完成后才由 [[api.rs](http://api.rs/) (line 1560)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/[api.rs:1560](http://api.rs:1560/)) 写成 completed；只有编辑图片分支在 [[api.rs](http://api.rs/) (line 1348)](/tmp/genarrative-pr76-review/server-rs/crates/api-server/src/editor_agent/[api.rs:1348](http://api.rs:1348/)) 提前写入 executing。如果请求在扣费或任务提交后断线/进程重启，消息仍是 pending_confirmation，用户重试会再次扣费和生成。应在所有工具的外部副作用前统一持久化 executing，并明确崩溃后的恢复策略。

kdletters requested review from kdletters 2026-07-14 13:09:45 +08:00

k88936 added 26 commits 2026-07-14 20:36:58 +08:00

preset executing status for all in tool calls 4f64e945e3

扩展外部生成任务结果与稳定入队 04654bf760

支持调用方提供任务 ID 和去重键，复用既有外部任务队列。

保存并投影编辑器生成结果，供任务消费者恢复展示。

画布Agent关联外部生成任务 f774a791e8

工具消息改为保存外部任务 ID 与取消事实，不再复制执行状态。

确认请求以稳定身份提交外部任务，并由前端查询任务状态和结果。

同步更新画布 Agent 测试与执行契约文档。

更新编辑器代理消息文档版本为 2 6beaab0454

兼容旧版本，新增字段 id 的反序列化逻辑 6edb826324

Merge remote-tracking branch 'origin/master' into editor-agent-refactored 7f6d5722be

# Conflicts:
#	server-rs/crates/api-server/src/editor_generation_queue.rs
#	src/components/image-editor/ImageCanvasEditorView.test.tsx
#	src/components/image-editor/ImageCanvasEditorView.tsx

修正外部任务摘要状态读取边界 4e0c0c8cd9

移除从轻量任务摘要读取主任务结果载荷的错误实现。

画布 Agent 继续轮询摘要状态，并通过画布完成回写刷新工程结果。

clean code af6b8021ac

inline 3f215d352a

remove unused test 97a1e1663e

split code 2128bb109c

add status field for persistence result in chat doc 5c67bc7714

extend external generate job to get return value abc6e7c800

fix status mismatch afe32d0166

mirror backend change 16943476ea

task list call back d0451f8226

forward call back

log and doc 21c39ee8db

impl simple display 18381ec283

moved code 696d6935b4

only poll on not complete 4ea9a1bd39

reconcile on get req d25d9a2935

refresh on gen job done 3796445b3b

合并 master 的画布资源命名更新 f0cbadcea1

合并 origin/master 的资源命名、图集拆分告警与相关后端契约更新

保留编辑 Agent 完成任务后的任务列表和画布刷新流程

保留编辑生成结果的轻量化持久化处理

fix compile err db2ea496ad

fix no assets ref in resp 54e662ca24

Merge remote-tracking branch 'origin/master' into editor-agent-refactored 1d5f8c2cbd

kdletters added 1 commit 2026-07-14 20:59:03 +08:00

Merge branch 'master' into editor-agent-refactored 8fa0ce4101

k88936 added 7 commits 2026-07-15 14:50:52 +08:00

split code 9ff8e0a297

回退外部生成任务分支改动 9a6bf8d86d

恢复外部生成 worker 的 master 行为

恢复画布生成任务默认入队标识

移除 Agent 自定义任务标识入口

新增外部生成任务结果查询 d55f4ba71c

新增仅返回任务结果的 SpacetimeDB procedure

编辑器 Agent 回填改用结果查询接口

补齐客户端绑定与接口文档

恢复编辑器 Agent 稳定任务标识 ddb508e729

保留外部生成任务稳定 job id 与去重键

供异步结果查询按固定任务标识回填

disable icons gen since conflict with master's change 4484a9caec

Merge remote-tracking branch 'refs/remotes/origin/master' into editor-agent-refactored 7c58cb11f2

# Conflicts:
#	vitest.config.ts

move code 7b31e736d1

k88936 added 5 commits 2026-07-15 16:23:04 +08:00

add test dependency for ImageId fda0931716

storage generated assets metadata in external job payload (only for editor agent source) e39d5fb278

update data structure for icon gen tool 29a3fd9b84

fix field name mismatch 79bef5a238

clean code 50eec83cdb

k88936 added 1 commit 2026-07-15 16:29:49 +08:00

split components 0e8f437b25

k88936 added 1 commit 2026-07-15 16:33:10 +08:00

Merge remote-tracking branch 'origin/master' into editor-agent-refactored f5c4e9ed15

k88936 marked the pull request as work in progress 2026-07-15 16:37:23 +08:00

k88936 commented

2026-07-15 16:38:18 +08:00

改用external job来做素材生成，针对来自editor agent 的生成会把生成资产的引用加入到结果json里，
客户端轮询 external job 确定生成状态，发现结束了或者失败了就重新get 会话，后端重新提供会话的时候把结果插入回会话历史里，用来让 ai 引用以及显示

![Screenshot_15-7-2026_162412_localhost.jpeg](/attachments/f193f275-6967-4d5b-87e5-e0a81e433e41) ![Screenshot_15-7-2026_16250_localhost.jpeg](/attachments/768e9bcd-efd3-4997-9a2c-36c63ad45e46) ![Screenshot_15-7-2026_163141_localhost.jpeg](/attachments/ff153b3c-567a-4a5f-ba3f-01e6295b8cd1) ![Screenshot_15-7-2026_163159_localhost.jpeg](/attachments/fefdd6c6-2b04-41d7-adfa-5007f33b7a27) 改用external job来做素材生成，针对来自editor agent 的生成会把生成资产的引用加入到结果json里，客户端轮询 external job 确定生成状态，发现结束了或者失败了就重新get 会话，后端重新提供会话的时候把结果插入回会话历史里，用来让 ai 引用以及显示

Screenshot_15-7-2026_162412_localhost.jpeg

118 KiB

Screenshot_15-7-2026_163159_localhost.jpeg

114 KiB

Screenshot_15-7-2026_163141_localhost.jpeg

126 KiB

Screenshot_15-7-2026_16250_localhost.jpeg

125 KiB

k88936 marked the pull request as ready for review 2026-07-15 16:39:39 +08:00

k88936 marked the pull request as work in progress 2026-07-15 16:45:11 +08:00

k88936 added 2 commits 2026-07-15 17:40:49 +08:00

revert wrong merge acb8149a12

frontend message dont depend on message.id 2113c33789

k88936 added 6 commits 2026-07-15 20:30:02 +08:00

restore context and image_args functions as test utils a2926674dd

validate editor agent message request b021ffbc08

fix source mismatch in test code c6bfdafc43

fix message id mismatch in frontend 9856acb8c3

rollback optimistic message on send failure 3a30198fef

simplify confirm and cancel logic b8004a04e7

k88936 marked the pull request as ready for review 2026-07-16 11:44:48 +08:00

k88936 added 5 commits 2026-07-16 11:45:03 +08:00

reload after failed task 2f344ed107

legacy doc migration 701b2338c7

resolve warning bee9e6ee8a

not show loading on reload to prevent scollable jumpy a45ae07767

Merge remote-tracking branch 'origin/master' into editor-agent-refactored 862038b955

k88936 added 1 commit 2026-07-16 13:05:27 +08:00

follow master change, use gpt-5.4-mini max_tokens=1024 0866878d9e

kdletters commented

2026-07-16 14:59:32 +08:00

问题	归因	依据
消息发送丢失 clientMessageId 幂等键	PR 引入	base 有客户端幂等键和重复检查；PR 删除后改用数组下标
SSE 流式回复被改成普通 JSON	PR 主动改动	base 有 /messages/stream 和 SSE client；PR 明确删除。若产品已决定取消 SSE，则应降为文档债务，不算缺陷
LLM/规划错误不再持久化	PR 引入	base 会把失败 assistant 消息写入 OSS；PR 只返回瞬时 errorMessage
完整历史无上限发送给 LLM	PR 引入	base 限制最近 12 条并单独注入最新生成图；PR 改为遍历全部消息
任务终态回填失败后不重试	PR 新链路缺陷	外部任务确认、轮询和懒回填是本 PR 新增，base 没有这条链路
后端正式数据契约未同步 Agent 结果载荷例外	PR 引入的文档不一致	base 的代码和文档一致；PR 新增轻量媒体结果持久化，只更新了 decision log，未修正正式架构文档
会话快速切换响应乱序	原本就有	base 的 loadConversation 同样无 request token，并无条件应用响应
Agent 与素材/图层栏不互斥	排除	不是 PR 改动，而且按你的确认属于现行设计；是旧文档未更新

k88936 added 6 commits 2026-07-16 17:51:14 +08:00

update old doc 0f4ec0ddd5

limit llm message sliding window size 2a84e6d89b

impl client message id and client retry 9dc65e1d47

persist error message in oss doc ca4bf6db5e

add retry in result reconciliation logic ce2806dea8

fix: 会话快速切换响应乱序 e7da2fe788

kdletters commented

2026-07-16 18:00:50 +08:00

发送消息期间切换会话，JSON 响应会落到当前会话面板里。
PR 把原来的 SSE 改成普通 JSON POST 后，useEditorAgentConversation.ts 在 sendMessage 返回后直接 applyDeltaMessages(response.deltaMessages)，没有再确认 activeConversationIdRef.current === conversationId。applyDeltaMessages 又是直接 append 当前 messages。所以用户在会话 A 发送消息、等待期间切到会话 B，A 的 assistant/tool card 会插进 B 的 UI，并可能触发错误的画布刷新。base 旧 SSE 路径有 eventConversationId !== activeConversationIdRef.current 的过滤，这个是 PR 引入的回归。

发送消息期间切换会话，JSON 响应会落到当前会话面板里。 PR 把原来的 SSE 改成普通 JSON POST 后，useEditorAgentConversation.ts 在 sendMessage 返回后直接 applyDeltaMessages(response.deltaMessages)，没有再确认 activeConversationIdRef.current === conversationId。applyDeltaMessages 又是直接 append 当前 messages。所以用户在会话 A 发送消息、等待期间切到会话 B，A 的 assistant/tool card 会插进 B 的 UI，并可能触发错误的画布刷新。base 旧 SSE 路径有 eventConversationId !== activeConversationIdRef.current 的过滤，这个是 PR 引入的回归。

k88936 added 1 commit 2026-07-16 18:51:00 +08:00

fix: 会话快速切换pollute message 621022ea4f

k88936 added 1 commit 2026-07-16 19:33:47 +08:00

Merge remote-tracking branch 'origin/master' into editor-agent-refactored c99769736c

# Conflicts:
#	docs/technical/【前端架构】图片画布编辑器MVP接入方案-2026-06-11.md
#	docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md
#	docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md
#	server-rs/crates/api-server/src/editor_agent.rs
#	server-rs/crates/api-server/src/external_generation_worker.rs
#	server-rs/crates/spacetime-client/src/lib.rs
#	server-rs/crates/spacetime-client/src/mapper.rs
#	server-rs/crates/spacetime-client/src/mapper/external_generation.rs

kdletters added 1 commit 2026-07-16 19:35:55 +08:00

Merge branch 'master' into editor-agent-refactored 6f8f01a004

k88936 added 1 commit 2026-07-16 20:58:58 +08:00

adjust system prompt to adapt to gpt 5.4 mini 8519d29283

k88936 added 1 commit 2026-07-17 10:02:31 +08:00

Merge remote-tracking branch 'origin/master' into editor-agent-refactored 2103d88836

# Conflicts:
#	server-rs/crates/api-server/src/editor_agent.rs

k88936 added 2 commits 2026-07-17 13:20:27 +08:00

fix: add missing params 908d51c92b

rename new proc c43836d881

k88936 added 1 commit 2026-07-17 16:09:04 +08:00

删除停止（思考） 9b42c64300

解释： 停止是原来sse留下来的， 现在思考时间相对调用各种美术工具时间短， 且美术工具调用不阻塞对话，调用前已有确认取消

k88936 added 1 commit 2026-07-17 16:53:55 +08:00

Merge remote-tracking branch 'origin/master' into editor-agent-refactored 0d22ee5d71

k88936 added 1 commit 2026-07-17 17:22:17 +08:00

comment: prevent infinite loop in result reconciliation on fatal errors 2829344d3d

k88936 added 1 commit 2026-07-17 17:39:41 +08:00

Merge remote-tracking branch 'origin/master' into editor-agent-refactored 9c75ef6af9

# Conflicts:
#	docs/README.md
#	src/components/image-editor/EditorAgentConversation/EditorAgentConversationPanelView.tsx
#	src/components/image-editor/EditorAgentConversation/useEditorAgentConversation.ts
#	src/components/image-editor/EditorAgentConversationPanelView.test.tsx
#	src/components/image-editor/ImageCanvasStageView.tsx

k88936 added 2 commits 2026-07-17 19:05:40 +08:00

remove logs in repo 15c8a2f8b1

refactor: improve result reconciliation by distinguishing retryable errors and fatal failures 0f3c63a6c6

kdletters approved these changes 2026-07-17 21:02:53 +08:00

kdletters merged commit f5368c825f into master

2026-07-17 21:03:15 +08:00

kdletters deleted branch editor-agent-refactored

2026-07-17 21:03:15 +08:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: GenarrativeAI/Genarrative#76