修复画布Agent聊天生成体验

让画布 Agent 消息统一走后端规划与 LLM 回复
修复生成图片在聊天、画布和素材库中的展示与恢复
补齐右键删除、画布焦点、聊天滚轮和面板避让交互
同步 Agent 事件契约、文档和定向回归测试
This commit is contained in:
2026-07-04 16:41:33 +08:00
parent 8dad52f040
commit 1671d35995
29 changed files with 909 additions and 173 deletions
@@ -26,7 +26,7 @@
## 当前分支落地状态
- 已落地:会话元数据、OSS 消息文档、会话 CRUD、SSE 消息流、后端 LLM / 启发式工具规划、右侧对话面板、会话历史、新建 / 软删会话、停止当前 SSE 回合、附件从画布资源 / 账号素材库选择,以及五类图片工具对既有生成入口的复用。
- 已落地:会话元数据、OSS 消息文档、会话 CRUD、SSE 消息流、后端 LLM 工具规划、右侧对话面板、会话历史、新建 / 软删会话、停止当前 SSE 回合、附件从画布资源 / 账号素材库选择,以及五类图片工具对既有生成入口的复用。
- 已落地:`tool_started` / `tool_completed` 事件携带 `status`;工具失败时也会写入失败 generation record,并随后发送 `stage=failed``error`,前端应保留消息内失败条目。
- 未落地:附件弹窗末尾上传格、跨刷新异步生成恢复、external generation task 轮询回填。未落地前,对话消息状态只表示本次 SSE 回合记录,不作为后台任务队列真相。
@@ -42,7 +42,8 @@
## 生成结果落画板(对现有占位规则的例外)
- 对话入口触发的生成**不创建"即将生成"画布占位**(区别于其余生成面板);生成中状态由对话消息流承载。
- 生成完成后:结果图按统一 placement 避让模型(视口中心就近、避开现有图层、32px 间距)落画板为新图层,同时在对话消息内显示缩略图(点击可让画布聚焦到对应图层;如果当前画布尚无该资源图层,前端先刷新工程快照再聚焦)
- 生成完成后:结果图按统一 placement 避让模型(视口中心就近、避开现有图层、32px 间距)落画板为新图层,同时登记到默认项目素材库,并在对话消息内显示缩略图;前端收到 `generation_result` 后立即刷新工程快照与素材库,缩略图本身不显示名称也不承担图层跳转
- 消息内生成结果缩略图必须携带并优先使用 `objectKey` / `assetObjectId`,前端通过 `ResolvedAssetImage` / `/api/assets/read-url` 换签后渲染,不能把裸 `/generated-*` 私有路径直接交给 `<img>`
- 当前第一阶段通过既有编辑器生成 BFF 的 `canvasCompletion` 写回工程快照;刷新后异步任务恢复和轮询回填属于后续能力,不在本阶段声明为已完成。
- 该例外已同步登记在《生成类面板Lovart统一改造方案-2026-06-17》「画布占位落点」节。
@@ -52,6 +53,7 @@
- 对话框与左侧素材 / 图层侧栏也互斥:打开画布 Agent 时收起左侧栏;再次打开素材、图层或任务侧栏时收起 Agent 面板。
- 桌面端对话框固定宽约 360–400px;移动端抽屉式全宽覆盖;收起态为胶囊/圆形入口按钮。
- 会话管理入口在对话框头部:当前会话标题 + 历史会话下拉(按更新时间倒序)+ 新建对话按钮,全部包在对话框内。
- 收起对话框只是隐藏面板,不卸载当前会话 hook;流式回复、`生成中` 阶段和停止按钮状态必须在收起 / 重新打开之间保持一致。
## 附件
@@ -65,7 +67,9 @@
## LLM 与计费
- 编排复用 `creative_agent_gpt5_client` 的 LLM 接入配置(同 provider/env,独立用途标识),function-calling 注册五类工具。
- 编排复用 `creative_agent_gpt5_client` 的 LLM 接入配置(同 provider/env,独立用途标识),但画布 Agent 的轻量规划请求显式使用 `gpt-4o`,避免把简单 JSON 意图识别交给容易长 reasoning 的 gpt-5function-calling 注册五类工具。
- 每个用户回合必须由 LLM 返回结构化计划;LLM 未配置、请求失败或返回格式不可解析时,后端写入明确错误消息,不使用本地关键词或“收到:...”回显兜底。
- 画布 Agent 规划请求使用 Responses API、1024 `max_output_tokens` 和 60 秒 Agent 专用请求超时;生成图片/编辑图片仍走对应生成工具和模型计费。
- **对话回合免费**(聊天、分析回复不扣泥点),仅 Agent 实际触发生成工具时按对应模型定价扣泥点。
- 工具调用前后端校验泥点余额;不足时该次生成失败并在对话中以明确错误气泡告知,对话本身可继续。
@@ -76,7 +80,7 @@
1. 助手文本 SSE 流式输出;
2. 阶段提示行(思考中 → 思考完成 → 生成中 → 完成/失败);
3. 工具/模型标注行(生成时显示模型名 + 图标);
4. 消息内生成结果缩略图(点击 → 画布聚焦对应图层);
4. 消息内生成结果缩略图(纯预览,不显示名称,不点击聚焦图层);
5. 生成中的进行中动画;
6. 错误气泡(失败/余额不足,带原因);
7. 发送中断:进行中时发送按钮变「停止」,可中断当前回合(已提交的生成任务不追回,照常落画板)。
@@ -120,4 +124,5 @@
- 打开画布 Agent 后,任务侧栏和左侧素材 / 图层面板应关闭;再次打开任务侧栏或素材 / 图层面板时,Agent 面板应关闭。
- 发送消息时先本地追加用户消息,再消费 SSE 增量;停止按钮只中断当前 SSE 回合,不追回已经提交的生成工具调用。
- 点击 Agent 消息内生成结果缩略图时,若当前画布已有该资源图层则直接聚焦;若尚无该资源图层,应先刷新工程快照,再聚焦新图层
- Agent 消息内生成结果缩略图只用于预览,不显示名称,也不点击跳转图层;收到 `generation_result` 时统一刷新工程快照和素材库
- 对话内容可被用户选中复制;用户从输入框或对话内容点击回画布图层 / 生成器时,焦点应回到画布对象,Backspace / Delete 等画布快捷键继续生效。
@@ -46,6 +46,8 @@ export interface EditorAgentAttachmentRef {
export interface EditorAgentGeneratedImage {
resourceId: string | null;
objectKey?: string | null;
assetObjectId?: string | null;
imageSrc: string;
thumbnailSrc: string | null;
width: number | null;
+185 -61
View File
@@ -18,7 +18,7 @@ use module_editor_agent::{
EDITOR_AGENT_MESSAGE_ID_PREFIX, derive_conversation_title, editor_agent_messages_object_key,
validate_user_message,
};
use platform_llm::{LlmMessage, LlmMessageContentPart, LlmTextRequest};
use platform_llm::{LlmError, LlmErrorKind, LlmMessage, LlmMessageContentPart, LlmTextRequest};
use platform_oss::{
LegacyAssetPrefix, OssObjectAccess, OssPutObjectRequest, OssSignedGetObjectUrlRequest,
};
@@ -65,6 +65,9 @@ const EDITOR_AGENT_MESSAGES_DOCUMENT_MAX_BYTES: usize = 2 * 1024 * 1024;
const EDITOR_AGENT_MESSAGES_READ_EXPIRE_SECONDS: u64 = 60;
const EDITOR_AGENT_LLM_ATTACHMENT_URL_EXPIRE_SECONDS: u64 = 300;
const EDITOR_AGENT_LLM_MAX_HISTORY_MESSAGES: usize = 12;
const EDITOR_AGENT_LLM_PLANNING_MODEL: &str = "gpt-4o";
const EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS: u32 = 1024;
const EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS: u64 = 60_000;
const EDITOR_AGENT_TOOL_CALL_ID_PREFIX: &str = "editor-agent-tool";
const EDITOR_AGENT_CANVAS_RESULT_GAP: f64 = 32.0;
type EditorAgentConversationLockMap = Mutex<BTreeMap<String, Arc<tokio::sync::Mutex<()>>>>;
@@ -279,23 +282,35 @@ pub async fn stream_editor_agent_message(
},
));
let turn_plan = plan_editor_agent_turn(
let mut assistant_status = EditorAgentMessageStatus::Completed;
let mut generation_records = Vec::new();
let mut planning_failed = false;
let turn_plan = match plan_editor_agent_turn(
&stream_state,
&stream_conversation,
&document,
&stream_user_message,
)
.await;
let mut assistant_text = turn_plan.reply_text;
if assistant_text.trim().is_empty() {
assistant_text = heuristic_editor_agent_turn_plan(
stream_user_message.text.as_str(),
stream_user_message.attachments.as_slice(),
)
.reply_text;
}
let mut assistant_status = EditorAgentMessageStatus::Completed;
let mut generation_records = Vec::new();
.await {
Ok(turn_plan) => turn_plan,
Err(error) => {
let app_error = error.into_app_error();
let error_message = app_error.body_text();
assistant_status = EditorAgentMessageStatus::Failed;
planning_failed = true;
EditorAgentTurnPlan {
reply_text: error_message,
tool_call: None,
}
}
};
let assistant_text = turn_plan.reply_text;
let assistant_kind = if assistant_status == EditorAgentMessageStatus::Failed {
EditorAgentMessageKind::Error
} else {
EditorAgentMessageKind::Chat
};
yield Ok::<Event, Infallible>(editor_agent_sse_json_event_or_error(
"stage",
@@ -310,10 +325,19 @@ pub async fn stream_editor_agent_message(
conversation_id: stream_conversation.conversation_id.clone(),
message_id: stream_assistant_message_id.clone(),
role: EditorAgentMessageRole::Assistant,
kind: EditorAgentMessageKind::Chat,
kind: assistant_kind,
text_delta: assistant_text.clone(),
},
));
if planning_failed {
yield Ok::<Event, Infallible>(editor_agent_sse_json_event_or_error(
"stage",
EditorAgentStageEvent {
conversation_id: stream_conversation.conversation_id.clone(),
stage: EditorAgentStage::Failed,
},
));
}
if let Some(tool_call) = turn_plan.tool_call {
let tool_call_id = build_prefixed_uuid_id(EDITOR_AGENT_TOOL_CALL_ID_PREFIX);
@@ -951,30 +975,65 @@ struct EditorAgentToolExecutionError {
error: AppError,
}
#[derive(Debug)]
enum EditorAgentPlanningError {
LlmUnavailable,
LlmRequestFailed(LlmError),
InvalidLlmResponse,
}
impl EditorAgentPlanningError {
fn into_app_error(self) -> AppError {
match self {
Self::LlmUnavailable => AppError::from_status(StatusCode::SERVICE_UNAVAILABLE)
.with_details(json!({
"provider": "editor-agent-llm",
"message": "画布 Agent 的 LLM 未配置,无法处理这句话。",
})),
Self::LlmRequestFailed(error) => {
let status = match error.kind() {
LlmErrorKind::Timeout => StatusCode::GATEWAY_TIMEOUT,
LlmErrorKind::InvalidConfig | LlmErrorKind::InvalidRequest => {
StatusCode::SERVICE_UNAVAILABLE
}
LlmErrorKind::Connectivity
| LlmErrorKind::Upstream
| LlmErrorKind::StreamUnavailable
| LlmErrorKind::EmptyResponse
| LlmErrorKind::Transport
| LlmErrorKind::Deserialize => StatusCode::BAD_GATEWAY,
};
AppError::from_status(status).with_details(json!({
"provider": "editor-agent-llm",
"message": format!("画布 Agent 调用 LLM 失败:{error}"),
}))
}
Self::InvalidLlmResponse => AppError::from_status(StatusCode::BAD_GATEWAY)
.with_details(json!({
"provider": "editor-agent-llm",
"message": "画布 Agent 的 LLM 返回格式错误,未能解析回复。",
})),
}
}
}
async fn plan_editor_agent_turn(
state: &AppState,
conversation: &EditorAgentConversationRecord,
document: &EditorAgentConversationMessagesDocument,
user_message: &EditorAgentMessage,
) -> EditorAgentTurnPlan {
let fallback = heuristic_editor_agent_turn_plan(
user_message.text.as_str(),
user_message.attachments.as_slice(),
);
) -> Result<EditorAgentTurnPlan, EditorAgentPlanningError> {
let Some(llm_client) = state.creative_agent_gpt5_client() else {
return fallback;
return Err(EditorAgentPlanningError::LlmUnavailable);
};
let request = build_editor_agent_llm_request(state, conversation, document, user_message).await;
match llm_client.request_text(request).await {
Ok(response) => parse_editor_agent_turn_plan(
response.content.as_str(),
fallback.reply_text.as_str(),
user_message.text.as_str(),
)
.unwrap_or(fallback),
Err(_) => fallback,
}
let response = llm_client
.request_text(request)
.await
.map_err(EditorAgentPlanningError::LlmRequestFailed)?;
parse_editor_agent_turn_plan(response.content.as_str())
.ok_or(EditorAgentPlanningError::InvalidLlmResponse)
}
async fn build_editor_agent_llm_request(
@@ -997,7 +1056,9 @@ async fn build_editor_agent_llm_request(
LlmMessage::user_multimodal(user_parts),
])
.with_responses_api()
.with_max_tokens(900)
.with_model(EDITOR_AGENT_LLM_PLANNING_MODEL)
.with_max_tokens(EDITOR_AGENT_LLM_MAX_OUTPUT_TOKENS)
.with_request_timeout_ms(EDITOR_AGENT_LLM_REQUEST_TIMEOUT_MS)
}
fn editor_agent_llm_system_prompt() -> String {
@@ -1080,34 +1141,30 @@ fn sign_editor_agent_attachment_image_url(
.map(|signed| signed.signed_url)
}
fn parse_editor_agent_turn_plan(
raw_text: &str,
fallback_reply_text: &str,
fallback_prompt: &str,
) -> Option<EditorAgentTurnPlan> {
fn parse_editor_agent_turn_plan(raw_text: &str) -> Option<EditorAgentTurnPlan> {
let json_text = extract_editor_agent_json_object(raw_text)?;
let raw: EditorAgentRawTurnPlan = serde_json::from_str(json_text).ok()?;
let reply_text = normalize_optional_string(raw.reply_text)
.or_else(|| normalize_optional_string(Some(fallback_reply_text.to_string())))
.unwrap_or_else(|| build_echo_assistant_text(fallback_prompt, 0));
let tool_call = raw.tool_call.and_then(|tool| {
let prompt = normalize_optional_string(tool.prompt)
.or_else(|| normalize_optional_string(Some(fallback_prompt.to_string())))?;
Some(EditorAgentToolCallPlan {
tool_name: tool.tool_name,
prompt,
summary: normalize_optional_string(tool.summary),
model: normalize_optional_string(tool.model),
aspect_ratio: normalize_optional_string(tool.aspect_ratio),
image_size: normalize_optional_string(tool.image_size),
icon_descriptions: tool
.icon_descriptions
.unwrap_or_default()
.into_iter()
.filter_map(|description| normalize_optional_string(Some(description)))
.collect(),
})
});
let reply_text = normalize_optional_string(raw.reply_text)?;
let tool_call = match raw.tool_call {
Some(tool) => {
let prompt = normalize_optional_string(tool.prompt)?;
Some(EditorAgentToolCallPlan {
tool_name: tool.tool_name,
prompt,
summary: normalize_optional_string(tool.summary),
model: normalize_optional_string(tool.model),
aspect_ratio: normalize_optional_string(tool.aspect_ratio),
image_size: normalize_optional_string(tool.image_size),
icon_descriptions: tool
.icon_descriptions
.unwrap_or_default()
.into_iter()
.filter_map(|description| normalize_optional_string(Some(description)))
.collect(),
})
}
None => None,
};
Some(EditorAgentTurnPlan {
reply_text,
@@ -1129,6 +1186,7 @@ fn extract_editor_agent_json_object(raw_text: &str) -> Option<&str> {
(start <= end).then_some(&without_fence[start..=end])
}
#[cfg(test)]
fn heuristic_editor_agent_turn_plan(
text: &str,
attachments: &[EditorAgentAttachmentRef],
@@ -1202,6 +1260,7 @@ fn heuristic_editor_agent_turn_plan(
}
}
#[cfg(test)]
fn contains_any(text: &str, needles: &[&str]) -> bool {
needles.iter().any(|needle| text.contains(needle))
}
@@ -1316,7 +1375,7 @@ async fn execute_editor_agent_tool_call(
project_id: Some(conversation.project_id.clone()),
asset_kind: Some(editor_agent_tool_asset_kind(tool_name).to_string()),
generation_inputs,
asset_folder_id: None,
asset_folder_id: Some(editor_agent_default_asset_folder_id()),
asset_label: tool_call.summary.clone(),
source_resource_id: user_message
.attachments
@@ -1351,7 +1410,7 @@ async fn execute_editor_agent_tool_call(
project_id: Some(conversation.project_id.clone()),
asset_kind: Some(editor_agent_tool_asset_kind(tool_name).to_string()),
generation_inputs,
asset_folder_id: None,
asset_folder_id: Some(editor_agent_default_asset_folder_id()),
asset_label: tool_call.summary.clone(),
source_resource_id: user_message
.attachments
@@ -1390,7 +1449,7 @@ async fn execute_editor_agent_tool_call(
image_size: tool_call.image_size.clone(),
project_id: Some(conversation.project_id.clone()),
generation_inputs,
asset_folder_id: None,
asset_folder_id: Some(editor_agent_default_asset_folder_id()),
canvas_completion: Some(completion),
},
)
@@ -1508,6 +1567,8 @@ fn editor_agent_generated_images_from_response(
.map(|image_src| {
vec![EditorAgentGeneratedImage {
resource_id: None,
object_key: None,
asset_object_id: None,
image_src: image_src.to_string(),
thumbnail_src: None,
width: data
@@ -1535,6 +1596,14 @@ fn editor_agent_generated_image_from_resource(
.get("resourceId")
.and_then(Value::as_str)
.map(str::to_string),
object_key: resource
.get("objectKey")
.and_then(Value::as_str)
.map(str::to_string),
asset_object_id: resource
.get("assetObjectId")
.and_then(Value::as_str)
.map(str::to_string),
image_src: resource
.get("imageSrc")
.and_then(Value::as_str)
@@ -1589,6 +1658,10 @@ fn editor_agent_tool_asset_kind(tool_name: EditorAgentToolName) -> &'static str
}
}
fn editor_agent_default_asset_folder_id() -> String {
"project".to_string()
}
fn editor_agent_tool_default_summary(tool_name: EditorAgentToolName) -> String {
match tool_name {
EditorAgentToolName::GenerateImage => "生成图片",
@@ -1658,6 +1731,7 @@ fn next_editor_agent_canvas_position(layers: Value, _width: f64, _height: f64) -
)
}
#[cfg(test)]
fn build_echo_assistant_text(text: &str, attachment_count: usize) -> String {
let trimmed = text.trim();
if trimmed.is_empty() {
@@ -1838,6 +1912,41 @@ mod tests {
assert_eq!(normalized.height, Some(128));
}
#[test]
fn generated_image_from_resource_keeps_private_asset_keys() {
let image = editor_agent_generated_image_from_resource(
&json!({
"resourceId": "resource-1",
"imageSrc": "/generated-editor-assets/result.png",
"thumbnailSrc": "/generated-editor-assets/result-thumb.png",
"objectKey": "generated-editor-assets/result.png",
"assetObjectId": "asset-object-result",
"width": 1024,
"height": 768
}),
None,
None,
None,
);
assert_eq!(image.resource_id.as_deref(), Some("resource-1"));
assert_eq!(
image.object_key.as_deref(),
Some("generated-editor-assets/result.png")
);
assert_eq!(
image.asset_object_id.as_deref(),
Some("asset-object-result")
);
assert_eq!(image.width, Some(1024));
assert_eq!(image.height, Some(768));
}
#[test]
fn editor_agent_generations_target_default_asset_library_folder() {
assert_eq!(editor_agent_default_asset_folder_id(), "project");
}
#[test]
fn editor_agent_turn_plan_parses_fenced_json_tool_call() {
let plan = parse_editor_agent_turn_plan(
@@ -1852,8 +1961,6 @@ mod tests {
}
}
```"#,
"fallback",
"像素风银发游侠",
)
.expect("fenced JSON should parse");
@@ -1865,6 +1972,23 @@ mod tests {
assert_eq!(tool_call.image_size.as_deref(), Some("1K"));
}
#[test]
fn editor_agent_turn_plan_rejects_invalid_llm_response() {
assert!(parse_editor_agent_turn_plan("收到:画一棵树").is_none());
assert!(
parse_editor_agent_turn_plan(
r#"{"toolCall":{"toolName":"generate_image","prompt":"画一棵树"}}"#,
)
.is_none()
);
assert!(
parse_editor_agent_turn_plan(
r#"{"replyText":"我来生成图片。","toolCall":{"toolName":"generate_image"}}"#,
)
.is_none()
);
}
#[test]
fn editor_agent_heuristic_routes_tools_from_user_text() {
let character_plan =
+2
View File
@@ -44,6 +44,8 @@ pub async fn proxy_llm_chat_completions(
max_tokens: None,
enable_web_search: false,
request_timeout_ms: None,
response_reasoning_effort: None,
response_text_verbosity: None,
};
if payload.stream {
@@ -55,5 +55,7 @@ pub fn build_gpt5_multimodal_request(
request_timeout_ms: None,
enable_web_search: false,
protocol: LlmTextProtocol::Responses,
response_reasoning_effort: None,
response_text_verbosity: None,
}
}
+82
View File
@@ -81,6 +81,8 @@ pub struct LlmTextRequest {
pub enable_web_search: bool,
pub protocol: LlmTextProtocol,
pub request_timeout_ms: Option<u64>,
pub response_reasoning_effort: Option<LlmResponseReasoningEffort>,
pub response_text_verbosity: Option<LlmResponseTextVerbosity>,
}
// 文本协议必须由业务请求显式选择,避免全局默认模型把不同场景混到同一上游形态。
@@ -90,6 +92,40 @@ pub enum LlmTextProtocol {
Responses,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum LlmResponseReasoningEffort {
Low,
Medium,
High,
}
impl LlmResponseReasoningEffort {
fn as_str(self) -> &'static str {
match self {
Self::Low => "low",
Self::Medium => "medium",
Self::High => "high",
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum LlmResponseTextVerbosity {
Low,
Medium,
High,
}
impl LlmResponseTextVerbosity {
fn as_str(self) -> &'static str {
match self {
Self::Low => "low",
Self::Medium => "medium",
Self::High => "high",
}
}
}
// 上层在流式消费时拿到的是“累计文本 + 当前增量”,避免每层重新自己拼接。
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct LlmStreamDelta {
@@ -213,6 +249,10 @@ struct ResponsesRequestBody {
max_output_tokens: Option<u32>,
#[serde(skip_serializing_if = "Option::is_none")]
tools: Option<Vec<ResponsesWebSearchTool>>,
#[serde(skip_serializing_if = "Option::is_none")]
reasoning: Option<ResponsesReasoningOptions>,
#[serde(skip_serializing_if = "Option::is_none")]
text: Option<ResponsesTextOptions>,
}
#[derive(Serialize)]
@@ -235,6 +275,16 @@ struct ResponsesWebSearchTool {
max_keyword: u8,
}
#[derive(Serialize)]
struct ResponsesReasoningOptions {
effort: &'static str,
}
#[derive(Serialize)]
struct ResponsesTextOptions {
verbosity: &'static str,
}
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct LlmRawFailureInputLog<'a> {
@@ -519,6 +569,8 @@ impl LlmTextRequest {
enable_web_search: false,
protocol: LlmTextProtocol::ChatCompletions,
request_timeout_ms: None,
response_reasoning_effort: None,
response_text_verbosity: None,
}
}
@@ -549,6 +601,16 @@ impl LlmTextRequest {
self
}
pub fn with_response_reasoning_effort(mut self, effort: LlmResponseReasoningEffort) -> Self {
self.response_reasoning_effort = Some(effort);
self
}
pub fn with_response_text_verbosity(mut self, verbosity: LlmResponseTextVerbosity) -> Self {
self.response_text_verbosity = Some(verbosity);
self
}
pub fn with_request_timeout_ms(mut self, request_timeout_ms: u64) -> Self {
self.request_timeout_ms = Some(request_timeout_ms);
self
@@ -1178,6 +1240,16 @@ fn build_request_body(
max_keyword: 3,
}]
}),
reasoning: request
.response_reasoning_effort
.map(|effort| ResponsesReasoningOptions {
effort: effort.as_str(),
}),
text: request
.response_text_verbosity
.map(|verbosity| ResponsesTextOptions {
verbosity: verbosity.as_str(),
}),
}),
}
}
@@ -2040,6 +2112,8 @@ mod tests {
.with_model("deepseek-v3-2-251201")
.with_responses_api()
.with_web_search(true)
.with_response_reasoning_effort(LlmResponseReasoningEffort::Low)
.with_response_text_verbosity(LlmResponseTextVerbosity::Low)
.with_max_tokens(128),
)
.await
@@ -2074,6 +2148,14 @@ mod tests {
request_json["tools"],
serde_json::json!([{ "type": "web_search", "max_keyword": 3 }])
);
assert_eq!(
request_json["reasoning"],
serde_json::json!({ "effort": "low" })
);
assert_eq!(
request_json["text"],
serde_json::json!({ "verbosity": "low" })
);
assert!(request_json.get("official_fallback").is_none());
assert_eq!(
request_json["input"][0]["content"][0],
@@ -92,6 +92,10 @@ pub struct EditorAgentAttachmentRef {
pub struct EditorAgentGeneratedImage {
#[serde(default)]
pub resource_id: Option<String>,
#[serde(default)]
pub object_key: Option<String>,
#[serde(default)]
pub asset_object_id: Option<String>,
pub image_src: String,
#[serde(default)]
pub thumbnail_src: Option<String>,
@@ -338,6 +342,8 @@ mod tests {
model: Some("z-image".to_string()),
images: vec![EditorAgentGeneratedImage {
resource_id: Some("resource-2".to_string()),
object_key: Some("generated/result.png".to_string()),
asset_object_id: Some("asset-object-2".to_string()),
image_src: "https://example.test/generated.png".to_string(),
thumbnail_src: None,
width: Some(1024),
@@ -354,6 +360,10 @@ mod tests {
payload["data"]["images"][0]["resourceId"],
json!("resource-2")
);
assert_eq!(
payload["data"]["images"][0]["objectKey"],
json!("generated/result.png")
);
let error_envelope = EditorAgentSseEnvelope {
event: EditorAgentSseEventType::Error,
@@ -1,6 +1,7 @@
/* @vitest-environment jsdom */
import {
act,
fireEvent,
render,
screen,
@@ -253,6 +254,135 @@ describe('EditorAgentConversationPanelView', () => {
});
});
it('keeps streamed reply and generating state when the panel is collapsed and reopened', async () => {
const client = createClient();
let finishStream = () => {};
vi.mocked(client.streamMessage).mockImplementation(
async (conversationId, _payload, options) => {
options.onEvent?.({
event: 'message_delta',
data: {
conversationId,
messageId: 'assistant-stream',
role: 'assistant',
kind: 'chat',
textDelta: '我来生成图片。',
},
});
options.onEvent?.({
event: 'stage',
data: { conversationId, stage: 'generating' },
});
options.onEvent?.({
event: 'tool_started',
data: {
conversationId,
messageId: 'assistant-stream',
toolCallId: 'tool-call-generating',
toolName: 'generate_image',
taskId: 'task-generating',
model: 'gpt-image-2',
},
});
await new Promise<void>((resolve) => {
finishStream = resolve;
});
},
);
const { rerender } = render(
<EditorAgentConversationPanelView
projectId="project-1"
open
onToggleOpen={vi.fn()}
client={client}
/>,
);
await waitFor(() => {
expect(screen.getByText('已经看到画布内容')).toBeTruthy();
});
fireEvent.change(screen.getByLabelText('发送给画布 Agent'), {
target: { value: '生成一张图片' },
});
fireEvent.click(screen.getByRole('button', { name: '发送' }));
await waitFor(() => {
expect(screen.getByText('我来生成图片。')).toBeTruthy();
});
expect(screen.getByText('生成中')).toBeTruthy();
expect(screen.getByRole('button', { name: '停止' })).toBeTruthy();
rerender(
<EditorAgentConversationPanelView
projectId="project-1"
open={false}
onToggleOpen={vi.fn()}
client={client}
/>,
);
expect(screen.getByRole('button', { name: '打开画布 Agent' })).toBeTruthy();
rerender(
<EditorAgentConversationPanelView
projectId="project-1"
open
onToggleOpen={vi.fn()}
client={client}
/>,
);
expect(screen.getByText('我来生成图片。')).toBeTruthy();
expect(screen.getByText('生成中')).toBeTruthy();
expect(screen.getByRole('button', { name: '停止' })).toBeTruthy();
await act(async () => {
finishStream();
});
});
it('keeps wheel scrolling inside the message history and input', async () => {
const client = createClient();
const parentWheel = vi.fn();
render(
<div onWheel={parentWheel}>
<EditorAgentConversationPanelView
projectId="project-1"
open
onToggleOpen={vi.fn()}
client={client}
/>
</div>,
);
await waitFor(() => {
expect(screen.getByText('已经看到画布内容')).toBeTruthy();
});
const messageWheel = new WheelEvent('wheel', {
bubbles: true,
cancelable: true,
deltaY: 120,
});
fireEvent(
screen.getByRole('log', { name: '画布 Agent 消息流' }),
messageWheel,
);
expect(parentWheel).not.toHaveBeenCalled();
expect(messageWheel.defaultPrevented).toBe(false);
const inputWheel = new WheelEvent('wheel', {
bubbles: true,
cancelable: true,
deltaY: 120,
});
fireEvent(screen.getByLabelText('发送给画布 Agent'), inputWheel);
expect(parentWheel).not.toHaveBeenCalled();
expect(inputWheel.defaultPrevented).toBe(false);
});
it('confirms conversation deletion from an independent dialog', async () => {
const client = createClient();
render(
@@ -10,11 +10,18 @@ import {
Trash2,
X,
} from 'lucide-react';
import { type FormEvent, useMemo, useState } from 'react';
import {
type FormEvent,
type WheelEvent as ReactWheelEvent,
useEffect,
useMemo,
useState,
} from 'react';
import {
EDITOR_AGENT_MAX_ATTACHMENTS,
type EditorAgentAttachmentRef,
type EditorAgentGenerationResultEvent,
type EditorAgentGenerationRecord,
type EditorAgentMessage,
type EditorAgentStage,
@@ -22,6 +29,7 @@ import {
import { PlatformActionButton } from '../common/PlatformActionButton';
import { PlatformDangerConfirmDialog } from '../common/PlatformDangerConfirmDialog';
import { UnifiedModal } from '../common/UnifiedModal';
import { ResolvedAssetImage } from '../ResolvedAssetImage';
import type { CanvasLayer, EditorAsset } from './ImageCanvasEditorTypes';
import {
type EditorAgentConversationClient,
@@ -42,10 +50,14 @@ type EditorAgentConversationPanelViewProps = {
onToggleOpen: () => void;
layers?: CanvasLayer[];
assets?: EditorAsset[];
onFocusResource?: (resourceId: string) => void;
onGenerationResult?: (event: EditorAgentGenerationResultEvent) => void;
client?: EditorAgentConversationClient;
};
function stopAgentPanelWheel(event: ReactWheelEvent<HTMLElement>) {
event.stopPropagation();
}
function attachmentKey(attachment: EditorAgentAttachmentRef) {
return `${attachment.source}:${attachment.referenceId}`;
}
@@ -169,8 +181,10 @@ function AttachmentChip({
) : null}
{attachment.thumbnailSrc || attachment.imageSrc ? (
<span className="pointer-events-none absolute bottom-[calc(100%+0.4rem)] left-0 hidden rounded-2xl border border-white bg-white p-1 shadow-xl group-hover:block">
<img
<ResolvedAssetImage
src={attachment.thumbnailSrc ?? attachment.imageSrc}
objectKey={attachment.objectKey}
refreshKey={attachment.referenceId}
alt=""
className="h-24 w-24 rounded-xl object-cover"
/>
@@ -182,10 +196,8 @@ function AttachmentChip({
function GenerationRecordsView({
generations,
onFocusResource,
}: {
generations: EditorAgentGenerationRecord[];
onFocusResource?: (resourceId: string) => void;
}) {
if (!generations.length) {
return null;
@@ -197,42 +209,42 @@ function GenerationRecordsView({
key={generation.toolCallId}
className="rounded-2xl border border-slate-200 bg-white/80 p-2 text-xs text-slate-600"
>
<div className="flex items-center gap-2">
{generation.status === 'generating' ? (
<Loader2
className="h-3.5 w-3.5 animate-spin"
aria-hidden="true"
/>
) : (
<ImageIcon className="h-3.5 w-3.5" aria-hidden="true" />
)}
<span>{toolLabel(generation.toolName)}</span>
{generation.model ? <span>{generation.model}</span> : null}
</div>
{generation.status === 'generating' || generation.error ? (
<div className="flex items-center gap-2">
{generation.status === 'generating' ? (
<Loader2
className="h-3.5 w-3.5 animate-spin"
aria-hidden="true"
/>
) : (
<ImageIcon className="h-3.5 w-3.5" aria-hidden="true" />
)}
<span>{toolLabel(generation.toolName)}</span>
{generation.model ? <span>{generation.model}</span> : null}
</div>
) : null}
{generation.error ? (
<div className="mt-1 text-red-600">{generation.error}</div>
) : null}
{generation.images.length ? (
<div className="mt-2 grid grid-cols-3 gap-2">
{generation.images.map((image, index) => (
<button
<div
key={`${generation.toolCallId}-${image.resourceId ?? index}`}
type="button"
className="overflow-hidden rounded-xl border border-slate-200 bg-slate-100"
title="生成结果"
disabled={!image.resourceId || !onFocusResource}
onClick={() => {
if (image.resourceId) {
onFocusResource?.(image.resourceId);
}
}}
>
<img
<ResolvedAssetImage
src={image.thumbnailSrc ?? image.imageSrc}
objectKey={image.objectKey}
refreshKey={
image.resourceId ??
generation.taskId ??
generation.toolCallId
}
alt=""
className="h-20 w-full object-cover"
/>
</button>
</div>
))}
</div>
) : null}
@@ -244,10 +256,8 @@ function GenerationRecordsView({
function MessageBubble({
message,
onFocusResource,
}: {
message: EditorAgentMessage;
onFocusResource?: (resourceId: string) => void;
}) {
const isUser = message.role === 'user';
return (
@@ -279,7 +289,6 @@ function MessageBubble({
) : null}
<GenerationRecordsView
generations={message.generations}
onFocusResource={onFocusResource}
/>
</div>
</article>
@@ -367,10 +376,12 @@ function AttachmentPickerModal({
key={option.key}
className="flex cursor-pointer flex-col gap-2 rounded-2xl border border-slate-200 bg-white p-2 text-sm text-slate-700 shadow-sm"
>
<img
<ResolvedAssetImage
src={
option.attachment.thumbnailSrc ?? option.attachment.imageSrc
}
objectKey={option.attachment.objectKey}
refreshKey={option.attachment.referenceId}
alt=""
className="aspect-square rounded-xl bg-slate-100 object-cover"
/>
@@ -398,10 +409,16 @@ export function EditorAgentConversationPanelView({
onToggleOpen,
layers = [],
assets = [],
onFocusResource,
onGenerationResult,
client,
}: EditorAgentConversationPanelViewProps) {
const effectiveProjectId = open ? projectId : null;
const [hasConversationMounted, setHasConversationMounted] = useState(open);
useEffect(() => {
if (open) {
setHasConversationMounted(true);
}
}, [open]);
const effectiveProjectId = hasConversationMounted ? projectId : null;
const {
conversations,
activeConversationId,
@@ -418,7 +435,11 @@ export function EditorAgentConversationPanelView({
sendMessage,
stopCurrentTurn,
deleteActiveConversation,
} = useEditorAgentConversation({ projectId: effectiveProjectId, client });
} = useEditorAgentConversation({
projectId: effectiveProjectId,
client,
onGenerationResult,
});
const [draftText, setDraftText] = useState('');
const [attachments, setAttachments] = useState<EditorAgentAttachmentRef[]>(
[],
@@ -517,7 +538,7 @@ export function EditorAgentConversationPanelView({
return (
<button
type="button"
className="absolute right-3 top-20 z-40 inline-flex items-center gap-2 rounded-full border border-white/70 bg-white/95 px-3 py-2 text-sm font-semibold text-slate-700 shadow-lg backdrop-blur hover:bg-white"
className="absolute bottom-24 right-4 z-40 inline-flex items-center gap-2 rounded-full border border-white/70 bg-white/95 px-3 py-2 text-sm font-semibold text-slate-700 shadow-lg backdrop-blur hover:bg-white"
aria-label="打开画布 Agent"
onClick={onToggleOpen}
>
@@ -533,6 +554,8 @@ export function EditorAgentConversationPanelView({
className="absolute inset-y-0 right-0 z-50 flex w-full flex-col border-l border-slate-200 bg-slate-50/95 shadow-2xl backdrop-blur sm:inset-y-3 sm:right-3 sm:w-[390px] sm:overflow-hidden sm:rounded-3xl sm:border"
aria-label="画布 Agent 对话"
onPointerDown={(event) => event.stopPropagation()}
onWheel={stopAgentPanelWheel}
onWheelCapture={stopAgentPanelWheel}
>
<header className="flex items-center gap-2 border-b border-slate-200 bg-white/90 px-3 py-3">
<Bot className="h-5 w-5 text-slate-700" aria-hidden="true" />
@@ -604,7 +627,7 @@ export function EditorAgentConversationPanelView({
</div>
) : null}
<div
className="min-h-0 flex-1 space-y-3 overflow-y-auto px-3 py-4"
className="min-h-0 flex-1 space-y-3 overflow-y-auto overscroll-contain px-3 py-4"
role="log"
aria-label="画布 Agent 消息流"
>
@@ -618,7 +641,6 @@ export function EditorAgentConversationPanelView({
<MessageBubble
key={message.id}
message={message}
onFocusResource={onFocusResource}
/>
))
) : (
@@ -657,7 +679,7 @@ export function EditorAgentConversationPanelView({
<Paperclip className="h-4 w-4" aria-hidden="true" />
</button>
<textarea
className="max-h-32 min-h-10 flex-1 resize-none rounded-3xl border border-slate-200 bg-slate-50 px-3 py-2 text-sm text-slate-800 outline-none focus:border-slate-400"
className="max-h-32 min-h-10 flex-1 resize-none overflow-y-auto overscroll-contain rounded-3xl border border-slate-200 bg-slate-50 px-3 py-2 text-sm text-slate-800 outline-none focus:border-slate-400"
aria-label="发送给画布 Agent"
value={draftText}
rows={1}
@@ -100,7 +100,11 @@ export function ImageCanvasContextMenusView({
? [
'context',
contextMenu.kind,
contextMenu.kind === 'layer' ? contextMenu.layerId : 'blank',
contextMenu.kind === 'layer'
? contextMenu.layerId
: contextMenu.kind === 'generation-dialog'
? contextMenu.dialogId
: 'blank',
contextMenu.x,
contextMenu.y,
isImageSequenceLayer ? 'sequence' : 'single',
@@ -259,7 +263,11 @@ export function ImageCanvasContextMenusView({
className={contextMenuClassName}
role="menu"
aria-label={
contextMenu.kind === 'blank' ? '画布右键菜单' : '图片功能面板'
contextMenu.kind === 'blank'
? '画布右键菜单'
: contextMenu.kind === 'generation-dialog'
? '生成器右键菜单'
: '图片功能面板'
}
style={{
left: currentMeasuredMenuLayout?.x ?? contextMenu.x,
@@ -308,6 +316,15 @@ export function ImageCanvasContextMenusView({
</button>
</>
) : contextMenu.kind === 'generation-dialog' ? (
<button
type="button"
role="menuitem"
className="image-canvas-editor__context-menu-danger"
onClick={onDeleteContextLayers}
>
</button>
) : (
<>
<button
@@ -48,6 +48,7 @@ export const CONTEXT_MENU_VIEWPORT_MARGIN = 8;
export const CONTEXT_MENU_SIZE = {
blank: { width: 188, height: 176 },
layer: { width: 188, height: 492 },
'generation-dialog': { width: 188, height: 64 },
} as const;
export const CANVAS_BACKGROUND_OPTIONS = [
{ label: '白色', value: '#ffffff' },
@@ -180,6 +180,7 @@ function createStageProps(): ImageCanvasStageViewProps {
onOpenLayerMetadata: vi.fn(),
onUpdateLayerAssetKind: vi.fn(),
onGenerationFramePointerDown: vi.fn(),
onGenerationFrameContextMenu: vi.fn(),
onActivateGenerationDialog: vi.fn(),
onFocusExternalTask: vi.fn(),
onToggleTaskSidebar: vi.fn(),
@@ -285,6 +285,13 @@ export type CanvasContextMenuState =
y: number;
layerId: string;
canvasPoint: { x: number; y: number };
}
| {
kind: 'generation-dialog';
x: number;
y: number;
dialogId: string;
canvasPoint: { x: number; y: number };
};
export type QuickEditPanelState = {
@@ -16,6 +16,7 @@ import {
ApiClientError,
AuthUiContext,
createAuthValue,
defaultEditorAssetLibraryAssets,
defaultEditorProjectLayers,
defaultEditorProjectResources,
dispatchPointerEvent,
@@ -937,6 +938,33 @@ describe('ImageCanvasEditorView', () => {
expect(screen.getByAltText('画布图片:大鱼素材')).toBeTruthy();
});
it('moves focus from the Agent input back to the canvas layer before handling shortcuts', async () => {
render(<ImageCanvasEditorView />);
fireEvent.click(screen.getByRole('button', { name: '画布 Agent' }));
const agentInput = await screen.findByLabelText('发送给画布 Agent');
agentInput.focus();
expect(document.activeElement).toBe(agentInput);
const layerButton = screen
.getByAltText('画布图片:拼图素材')
.closest('button')!;
fireEvent.pointerDown(layerButton, {
button: 0,
pointerId: 53,
clientX: 120,
clientY: 120,
});
expect(document.activeElement).toBe(layerButton);
await act(async () => {
fireEvent.keyDown(window, { key: 'Backspace', code: 'Backspace' });
});
expect(screen.queryByAltText('画布图片:拼图素材')).toBeNull();
expect(screen.getByAltText('画布图片:大鱼素材')).toBeTruthy();
});
it('opens the matching audio generation panel when remodeling an existing sound effect layer', async () => {
loadOrCreateRecentEditorProjectMock.mockResolvedValueOnce({
projectId: 'editor-project-audio-remodel',
@@ -1123,6 +1151,23 @@ describe('ImageCanvasEditorView', () => {
expect(screen.getByRole('menuitem', { name: '创建副本' })).toBeTruthy();
});
it('shows delete when right-clicking a generation placeholder', async () => {
render(<ImageCanvasEditorView />);
fireEvent.click(screen.getByRole('button', { name: '生成图片' }));
const frame = await screen.findByLabelText('图像生成占位图');
fireEvent.contextMenu(frame, {
clientX: 510,
clientY: 330,
});
const menu = screen.getByRole('menu', { name: '生成器右键菜单' });
fireEvent.click(within(menu).getByRole('menuitem', { name: '删除' }));
expect(screen.queryByLabelText('图像生成占位图')).toBeNull();
});
it('copies, cuts, and pastes layers from the context menus', () => {
render(<ImageCanvasEditorView />);
@@ -1511,7 +1556,7 @@ describe('ImageCanvasEditorView', () => {
expect(screen.queryByRole('button', { name: '画布小地图' })).toBeNull();
});
it('focuses an existing canvas layer from an Agent generation thumbnail', async () => {
it('renders Agent generation thumbnails as passive previews', async () => {
const detail = createEditorAgentDetailWithGeneration('resource-puzzle');
listEditorAgentConversationsMock.mockResolvedValueOnce([
createEditorAgentConversationSummary(),
@@ -1521,22 +1566,90 @@ describe('ImageCanvasEditorView', () => {
render(<ImageCanvasEditorView />);
fireEvent.click(screen.getByRole('button', { name: '画布 Agent' }));
fireEvent.click(await screen.findByRole('button', { name: '生成结果' }));
await waitFor(() => {
expect(
screen.getByAltText('画布图片:拼图素材').closest('button')?.className,
).toContain('image-canvas-editor__layer--selected');
const messageLog = await screen.findByRole('log', {
name: '画布 Agent 消息流',
});
expect(
within(messageLog).queryByRole('button', { name: '生成结果' }),
).toBeNull();
expect(within(messageLog).queryByText('gpt-image-2')).toBeNull();
expect(loadEditorProjectMock).not.toHaveBeenCalled();
});
it('refreshes the project and focuses a newly added Agent result layer', async () => {
const detail = createEditorAgentDetailWithGeneration('resource-agent-new');
it('refreshes canvas and asset library when Agent generation finishes', async () => {
let showGeneratedAsset = false;
loadEditorAssetLibraryMock.mockImplementation(async () => ({
folders: [
{
folderId: 'project',
label: '项目素材',
sortOrder: 0,
collapsed: false,
systemDefault: true,
},
],
assets: showGeneratedAsset
? [
...defaultEditorAssetLibraryAssets,
{
assetId: 'asset-agent-stream',
folderId: 'project',
label: 'Agent结果素材',
imageSrc: '/agent-stream.png',
width: 512,
height: 512,
sourceType: 'generated',
assetKind: 'editor_agent_generated_image',
},
]
: defaultEditorAssetLibraryAssets,
}));
listEditorAgentConversationsMock.mockResolvedValueOnce([
createEditorAgentConversationSummary(),
]);
getEditorAgentConversationMock.mockResolvedValueOnce(detail);
getEditorAgentConversationMock.mockResolvedValueOnce({
...createEditorAgentConversationSummary(),
messages: [],
});
streamEditorAgentMessageMock.mockImplementation(
async (conversationId, _payload, options) => {
options.onEvent?.({
event: 'message_delta',
data: {
conversationId,
messageId: 'assistant-agent-stream',
role: 'assistant',
kind: 'chat',
textDelta: '我来生成图片。',
},
});
showGeneratedAsset = true;
options.onEvent?.({
event: 'generation_result',
data: {
conversationId,
messageId: 'assistant-agent-stream',
toolCallId: 'tool-call-agent-stream',
toolName: 'generate_image',
model: 'gpt-image-2',
images: [
{
resourceId: 'resource-agent-stream',
imageSrc: '/agent-stream.png',
thumbnailSrc: null,
width: 512,
height: 512,
},
],
},
});
options.onEvent?.({
event: 'done',
data: { conversationId, title: null },
});
},
);
loadEditorProjectMock.mockResolvedValueOnce({
projectId: 'editor-project-default',
title: '默认项目',
@@ -1544,9 +1657,10 @@ describe('ImageCanvasEditorView', () => {
layers: [
...defaultEditorProjectLayers,
{
layerId: 'layer-agent-new',
resourceId: 'resource-agent-new',
layerId: 'layer-agent-stream',
resourceId: 'resource-agent-stream',
title: 'Agent结果图',
src: '/agent-stream.png',
x: 1500,
y: 300,
width: 512,
@@ -1560,38 +1674,44 @@ describe('ImageCanvasEditorView', () => {
resources: [
...defaultEditorProjectResources,
{
resourceId: 'resource-agent-new',
resourceId: 'resource-agent-stream',
projectId: 'editor-project-default',
imageSrc: '/agent-result.png',
imageSrc: '/agent-stream.png',
width: 512,
height: 512,
sourceType: 'generated',
prompt: 'Agent生成结果',
model: 'gpt-image-2',
provider: 'VectorEngine',
taskId: 'task-agent-1',
taskId: 'task-agent-stream',
assetKind: 'editor_agent_generated_image',
},
],
updatedAt: '2026-07-03T00:00:00.000Z',
updatedAt: '2026-07-03T00:02:00.000Z',
});
render(<ImageCanvasEditorView />);
fireEvent.click(screen.getByRole('button', { name: '画布 Agent' }));
fireEvent.click(await screen.findByRole('button', { name: '生成结果' }));
await waitFor(() => {
expect(getEditorAgentConversationMock).toHaveBeenCalledWith(
'editor-agent-conv-test',
);
});
fireEvent.change(await screen.findByLabelText('发送给画布 Agent'), {
target: { value: '生成一张图' },
});
fireEvent.click(screen.getByRole('button', { name: '发送' }));
await waitFor(() => {
expect(loadEditorProjectMock).toHaveBeenCalledWith(
'editor-project-default',
);
});
await waitFor(() => {
expect(
screen.getByAltText('画布图片:Agent结果图').closest('button')
?.className,
).toContain('image-canvas-editor__layer--selected');
});
expect(await screen.findByAltText('画布图片:Agent结果图')).toBeTruthy();
fireEvent.click(screen.getByRole('button', { name: '打开素材' }));
expect(await screen.findByText('Agent结果素材')).toBeTruthy();
});
it('resets the canvas view without forwarding the click event to fit layers', () => {
@@ -10,6 +10,7 @@ import {
} from 'react';
import type { ExternalGenerationTaskRecord } from '../../../packages/shared/src/contracts/externalGeneration';
import type { EditorAgentGenerationResultEvent } from '../../../packages/shared/src/contracts/editorAgent';
import {
createEditorAsset,
createEditorProjectResource,
@@ -1008,40 +1009,19 @@ export function ImageCanvasEditorView() {
},
[applyProjectSnapshot, refreshAssetLibrary],
);
const focusCanvasLayerByResourceId = useCallback(
(resourceId: string) => {
const targetLayer = layersRef.current.find(
(layer) => layer.resourceId === resourceId || layer.id === resourceId,
const handleEditorAgentGenerationResult = useCallback(
(event: EditorAgentGenerationResultEvent) => {
const hasGeneratedResource = event.images.some((image) =>
image.resourceId?.trim(),
);
if (targetLayer) {
focusCanvasLayerById(targetLayer.id);
return;
}
if (!projectId) {
if (!projectId || !hasGeneratedResource) {
return;
}
void loadEditorProject(projectId)
.then((project) => {
applyGeneratedProjectSnapshot(project);
const targetLayerItem = project.layers.find(
(layer) =>
layer.resourceId === resourceId || layer.layerId === resourceId,
);
const targetLayerId = targetLayerItem?.layerId;
if (!targetLayerId) {
return;
}
window.setTimeout(() => {
focusCanvasLayerById(targetLayerId);
}, 0);
})
.then(applyGeneratedProjectSnapshot)
.catch(() => undefined);
},
[
applyGeneratedProjectSnapshot,
focusCanvasLayerById,
projectId,
],
[applyGeneratedProjectSnapshot, projectId],
);
const persistUpdatedLayerResource = useCallback(
(layer: CanvasLayer) => {
@@ -1318,11 +1298,14 @@ export function ImageCanvasEditorView() {
clearCanvasFocus,
handleCanvasContextMenu,
handleLayerContextMenu,
handleGenerationFrameContextMenu,
} = useImageCanvasStageController({
layers,
canvasGenerationDialogs,
selectedLayerId,
selectedLayerIds,
setSelectedLayerId,
setSelectedLayerIds,
imageContextMenu,
setImageContextMenu,
contextMenu,
@@ -1979,9 +1962,10 @@ export function ImageCanvasEditorView() {
},
onUpdateLayerAssetKind: updateLayerAssetKind,
onGenerationFramePointerDown: handleGenerationFramePointerDown,
onGenerationFrameContextMenu: handleGenerationFrameContextMenu,
onActivateGenerationDialog: activateCanvasGenerationDialog,
onFocusExternalTask: focusExternalGenerationTask,
onFocusEditorAgentResource: focusCanvasLayerByResourceId,
onEditorAgentGenerationResult: handleEditorAgentGenerationResult,
onToggleTaskSidebar: toggleTaskSidebar,
onToggleAgentConversation: toggleAgentConversation,
onCropExpandHandlePointerDown: generationSurface.startCropExpandFrameResize,
@@ -3,6 +3,7 @@ import type {
CanvasContextMenuState,
CanvasLayer,
} from './ImageCanvasEditorTypes';
import { getCanvasGenerationSelectionId } from './ImageCanvasSelectionModel';
export type CanvasLayerMoveMode = 'up' | 'down' | 'top' | 'bottom';
@@ -31,12 +32,14 @@ export function resolveContextTargetLayerIds(
menu: CanvasContextMenuState | null,
selectedLayerIds: string[],
) {
if (menu?.kind !== 'layer') {
if (!menu || menu.kind === 'blank') {
return [];
}
return selectedLayerIds.includes(menu.layerId)
? [...selectedLayerIds]
: [menu.layerId];
const targetId =
menu.kind === 'generation-dialog'
? getCanvasGenerationSelectionId(menu.dialogId)
: menu.layerId;
return selectedLayerIds.includes(targetId) ? [...selectedLayerIds] : [targetId];
}
export function getCanvasLayersByIds(
@@ -57,6 +57,9 @@ describe('ImageCanvasPanelDockView', () => {
const toolbar = screen.getByRole('toolbar', { name: '画布面板入口' });
expect(toolbar.className).toContain(
'image-canvas-editor__panel-dock--sidebar-open',
);
expect(
within(toolbar)
.getByRole('button', { name: '打开素材' })
@@ -90,7 +90,14 @@ export function ImageCanvasPanelDockView({
/>
<div
className="image-canvas-editor__panel-dock"
className={[
'image-canvas-editor__panel-dock',
activeSidebarPanel
? 'image-canvas-editor__panel-dock--sidebar-open'
: '',
]
.filter(Boolean)
.join(' ')}
role="toolbar"
aria-label="画布面板入口"
onPointerDown={(event) => event.stopPropagation()}
@@ -135,3 +135,22 @@ export function createLayerCanvasContextMenus({
},
};
}
export function createGenerationDialogCanvasContextMenu({
clientX,
clientY,
dialogId,
canvasPoint,
}: {
clientX: number;
clientY: number;
dialogId: string;
canvasPoint: { x: number; y: number };
}): CanvasContextMenuState {
return {
kind: 'generation-dialog',
dialogId,
...resolveContextMenuPosition(clientX, clientY, 'generation-dialog'),
canvasPoint,
};
}
@@ -8,6 +8,7 @@ import type {
} from 'react';
import type { ExternalGenerationTaskRecord } from '../../../packages/shared/src/contracts/externalGeneration';
import type { EditorAgentGenerationResultEvent } from '../../../packages/shared/src/contracts/editorAgent';
import { EditorAgentConversationPanelView } from './EditorAgentConversationPanelView';
import { ImageCanvasBottomToolbarView } from './ImageCanvasBottomToolbarView';
import { ImageCanvasContextMenusView } from './ImageCanvasContextMenusView';
@@ -118,9 +119,15 @@ export type ImageCanvasStageViewProps = {
event: ReactPointerEvent<HTMLDivElement>,
dialog: CanvasGenerationDialogState,
) => void;
onGenerationFrameContextMenu: (
event: ReactMouseEvent<HTMLDivElement>,
dialog: CanvasGenerationDialogState,
) => void;
onActivateGenerationDialog: (dialog: CanvasGenerationDialogState) => void;
onFocusExternalTask: (task: ExternalGenerationTaskRecord) => void;
onFocusEditorAgentResource?: (resourceId: string) => void;
onEditorAgentGenerationResult?: (
event: EditorAgentGenerationResultEvent,
) => void;
onToggleTaskSidebar: () => void;
onToggleAgentConversation: () => void;
onCropExpandHandlePointerDown: (
@@ -247,9 +254,10 @@ export function ImageCanvasStageView({
onOpenLayerMetadata,
onUpdateLayerAssetKind,
onGenerationFramePointerDown,
onGenerationFrameContextMenu,
onActivateGenerationDialog,
onFocusExternalTask,
onFocusEditorAgentResource,
onEditorAgentGenerationResult,
onToggleTaskSidebar,
onToggleAgentConversation,
onCropExpandHandlePointerDown,
@@ -347,6 +355,7 @@ export function ImageCanvasStageView({
onOpenLayerMetadata={onOpenLayerMetadata}
onUpdateLayerAssetKind={onUpdateLayerAssetKind}
onGenerationFramePointerDown={onGenerationFramePointerDown}
onGenerationFrameContextMenu={onGenerationFrameContextMenu}
onActivateGenerationDialog={onActivateGenerationDialog}
onCropExpandHandlePointerDown={onCropExpandHandlePointerDown}
/>
@@ -464,7 +473,7 @@ export function ImageCanvasStageView({
onToggleOpen={onToggleAgentConversation}
layers={layers}
assets={editorAgentAssets}
onFocusResource={onFocusEditorAgentResource}
onGenerationResult={onEditorAgentGenerationResult}
/>
{isToolbarGuideVisible ? (
@@ -85,6 +85,7 @@ function renderWorldView(
onOpenLayerMetadata: vi.fn(),
onUpdateLayerAssetKind: vi.fn(),
onGenerationFramePointerDown: vi.fn(),
onGenerationFrameContextMenu: vi.fn(),
onActivateGenerationDialog: vi.fn(),
onCropExpandHandlePointerDown: vi.fn(),
...overrides,
@@ -274,6 +274,10 @@ export type ImageCanvasWorldViewProps = {
event: ReactPointerEvent<HTMLDivElement>,
dialog: CanvasGenerationDialogState,
) => void;
onGenerationFrameContextMenu: (
event: ReactMouseEvent<HTMLDivElement>,
dialog: CanvasGenerationDialogState,
) => void;
onActivateGenerationDialog: (dialog: CanvasGenerationDialogState) => void;
onCropExpandHandlePointerDown: (
event: ReactPointerEvent<HTMLButtonElement>,
@@ -921,6 +925,7 @@ export function ImageCanvasWorldView({
onOpenLayerMetadata,
onUpdateLayerAssetKind,
onGenerationFramePointerDown,
onGenerationFrameContextMenu,
onActivateGenerationDialog,
onCropExpandHandlePointerDown,
}: ImageCanvasWorldViewProps) {
@@ -1259,6 +1264,9 @@ export function ImageCanvasWorldView({
onPointerDown={(event) =>
onGenerationFramePointerDown(event, dialog)
}
onContextMenu={(event) =>
onGenerationFrameContextMenu(event, dialog)
}
onDoubleClick={() => onActivateGenerationDialog(dialog)}
>
{showFocusedChrome ? (
@@ -111,8 +111,13 @@ describe('useEditorAgentConversation', () => {
it('loads conversations and applies message stream events', async () => {
const client = createClient();
const onGenerationResult = vi.fn();
const { result } = renderHook(() =>
useEditorAgentConversation({ projectId: 'project-1', client }),
useEditorAgentConversation({
projectId: 'project-1',
client,
onGenerationResult,
}),
);
await waitFor(() => {
@@ -138,6 +143,16 @@ describe('useEditorAgentConversation', () => {
);
expect(result.current.stage).toBe('completed');
expect(result.current.isStreaming).toBe(false);
expect(onGenerationResult).toHaveBeenCalledWith(
expect.objectContaining({
toolCallId: 'tool-call-1',
images: [
expect.objectContaining({
resourceId: 'resource-result-1',
}),
],
}),
);
expect(result.current.messages.map((message) => message.text)).toEqual([
'把这个角色改成像素风',
'我来处理',
@@ -221,6 +236,65 @@ describe('useEditorAgentConversation', () => {
);
});
it('keeps LLM planning failures as one failed assistant message', async () => {
const client = createClient();
vi.mocked(client.streamMessage).mockImplementation(
async (_conversationId, _payload, options) => {
options.onEvent?.({
event: 'message_delta',
data: {
conversationId: 'conversation-1',
messageId: 'assistant-planning-error',
role: 'assistant',
kind: 'error',
textDelta: '画布 Agent 的 LLM 未配置,无法处理这句话。',
},
});
options.onEvent?.({
event: 'stage',
data: {
conversationId: 'conversation-1',
stage: 'failed',
},
});
options.onEvent?.({
event: 'done',
data: {
conversationId: 'conversation-1',
title: null,
},
});
},
);
const { result } = renderHook(() =>
useEditorAgentConversation({ projectId: 'project-1', client }),
);
await waitFor(() => {
expect(result.current.activeConversation?.conversationId).toBe(
'conversation-1',
);
});
await act(async () => {
await result.current.sendMessage('这是美术素材');
});
expect(result.current.stage).toBe('failed');
expect(result.current.errorMessage).toBe(
'画布 Agent 的 LLM 未配置,无法处理这句话。',
);
expect(
result.current.messages.filter((message) => message.kind === 'error'),
).toEqual([
expect.objectContaining({
id: 'assistant-planning-error',
text: '画布 Agent 的 LLM 未配置,无法处理这句话。',
status: 'failed',
}),
]);
});
it('shows a failed tool completion as a generation record', async () => {
const client = createClient();
vi.mocked(client.streamMessage).mockImplementation(
@@ -5,6 +5,7 @@ import type {
EditorAgentAttachmentRef,
EditorAgentConversationDetail,
EditorAgentConversationSummary,
EditorAgentGenerationResultEvent,
EditorAgentGenerationRecord,
EditorAgentMessage,
EditorAgentSseEvent,
@@ -44,6 +45,7 @@ export type EditorAgentConversationClient = {
type UseEditorAgentConversationOptions = {
projectId?: string | null;
client?: EditorAgentConversationClient;
onGenerationResult?: (event: EditorAgentGenerationResultEvent) => void;
};
const defaultEditorAgentConversationClient: EditorAgentConversationClient = {
@@ -208,6 +210,7 @@ function upsertConversationSummary(
export function useEditorAgentConversation({
projectId,
client = defaultEditorAgentConversationClient,
onGenerationResult,
}: UseEditorAgentConversationOptions) {
const normalizedProjectId = projectId?.trim() ?? '';
const [conversations, setConversations] = useState<
@@ -377,6 +380,12 @@ export function useEditorAgentConversation({
}
if (event.event === 'message_delta') {
if (event.data.kind === 'error') {
const nextErrorMessage = event.data.textDelta.trim();
if (nextErrorMessage) {
setErrorMessage(nextErrorMessage);
}
}
setMessages((currentMessages) =>
upsertMessage(
currentMessages,
@@ -437,6 +446,7 @@ export function useEditorAgentConversation({
}
if (event.event === 'generation_result') {
onGenerationResult?.(event.data);
const nextRecord: EditorAgentGenerationRecord = {
toolCallId: event.data.toolCallId,
toolName: event.data.toolName,
@@ -485,7 +495,9 @@ export function useEditorAgentConversation({
}
if (event.event === 'done') {
setStage('completed');
setStage((currentStage) =>
currentStage === 'failed' ? 'failed' : 'completed',
);
setMessages((currentMessages) =>
markStreamingMessages(currentMessages, 'completed'),
);
@@ -500,7 +512,7 @@ export function useEditorAgentConversation({
);
}
}
}, []);
}, [onGenerationResult]);
const ensureConversationForSend = useCallback(async () => {
if (activeConversationId) {
@@ -496,21 +496,41 @@ export function useImageCanvasLayerCommands({
const deleteContextLayers = useCallback(() => {
const targetIds = getContextTargetLayerIds();
if (!targetIds.length) {
const targetLayerIds = getSelectedLayerIds(targetIds);
const targetDialogIds = targetIds
.map(getCanvasGenerationDialogIdFromSelectionId)
.filter((dialogId): dialogId is string => Boolean(dialogId))
.filter((dialogId) =>
canvasGenerationDialogs.some((dialog) => dialog.id === dialogId),
);
if (!targetLayerIds.length && !targetDialogIds.length) {
return;
}
captureCanvasHistory();
setLayers((currentLayers) => removeCanvasLayers(currentLayers, targetIds));
if (targetLayerIds.length) {
setLayers((currentLayers) =>
removeCanvasLayers(currentLayers, targetLayerIds),
);
}
selectSingleLayer(null);
setHoveredLayerId(null);
setMetadataLayer((currentLayer) =>
currentLayer && targetIds.includes(currentLayer.id) ? null : currentLayer,
currentLayer && targetLayerIds.includes(currentLayer.id)
? null
: currentLayer,
);
targetLayerIds.forEach((targetId) => onDeleteLayerSideEffects(targetId));
targetDialogIds.forEach((targetId) =>
onDeleteGenerationDialogSideEffects?.(targetId),
);
closeContextMenus();
}, [
captureCanvasHistory,
canvasGenerationDialogs,
closeContextMenus,
getContextTargetLayerIds,
onDeleteGenerationDialogSideEffects,
onDeleteLayerSideEffects,
selectSingleLayer,
setHoveredLayerId,
setLayers,

Some files were not shown because too many files have changed in this diff Show More