From 302b4addae5123f6c54ba85dd4a9438ca86d6d77 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Mon, 7 Sep 2026 16:08:25 +0800 Subject: [PATCH 001/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20GPT=20Image?= =?UTF-8?q?=202=20=E4=BB=A3=E7=90=86=E6=96=B9=E6=A1=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 明确 JSON 请求与 data 数组响应 记录预检查和钱包事务边界 拆分 api-server 与 platform-image 文件职责 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md new file mode 100644 index 000000000..ae006ffa7 --- /dev/null +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -0,0 +1,103 @@ +# Raw GPT Image 2 图片编辑代理 + +更新时间：`2026-09-07` + +## 目标 + +提供一个由主站客户端调用的独立同步图片编辑代理： + +```text +POST /api/raw/v1/images/edit +``` + +该入口使用登录态 Bearer access token，不进入 External v1 / MCP OpenAPI，不读取或写入画布、项目资源、素材库、OSS 结果或 `external_generation_job`。 + +## 请求合同 + +请求使用 `application/json`。图片字段只使用原始图片的 base64 数据和 MIME 类型，不接受 object key、URL、Data URL 或 Blob URL。 + +```json +{ + "images": [ + { + "data": "", + "mimeType": "image/png" + } + ], + "mask": { + "data": "", + "mimeType": "image/png" + }, + "prompt": "修改图片", + "quality": "auto", + "background": "auto", + "output_format": "png", + "width": 1536, + "height": 1024 +} +``` + +`images` 是必填数组，数组成员结构固定为 `{ data, mimeType }`；`mask` 可选并使用相同结构。输入格式由成员的 MIME 类型和解码后的图片字节共同确定，服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。`width`、`height` 为整数，组成发送给 provider 的输出尺寸；不把尺寸改写成业务字符串字段。 + +服务端发送给 `platform-image` 时固定注入： + +```text +model = gpt-image-2 +n = 1 +``` + +请求不暴露 `model`、`n`、`response_format`、`style`、`user` 或 `output_compression`。 + +## 成功响应合同 + +响应始终为 JSON，响应只保留 `data` 字段，图片内容只以 base64 返回： + +```json +{ + "data": [ + { + "b64_json": "" + } + ] +} +``` + +`data` 保持数组形状，即使服务端固定 `n=1`。响应不重复返回请求参数，不返回 URL、资源 ID、任务 ID、provider 原始 JSON 或 editor 字段。 + +## 预检查与计费事务 + +所有请求、JSON、base64、图片结构和 provider 参数检查必须在扣费前完成。预检查失败直接返回 4xx，不产生钱包流水，也不调用 provider。 + +检查通过后，api-server 调用一个 raw 图片操作的 SpacetimeDB 事务 procedure，在同一事务内完成： + +1. 以认证后的用户和请求 ID 建立 raw 操作幂等事实； +2. 按现有图片编辑算法解析价格：GPT Image 2 长边不超过 1536 使用 1K 价格，否则使用 2K 价格；当前默认价格为 3 / 5 泥点； +3. 原子扣除用户泥点并写入 `asset_operation_consume` 流水； +4. 持久化操作状态，供 provider 返回后成功或失败收口。 + +provider 调用在 SpacetimeDB 事务之外执行。成功后调用同一 raw 操作的完成 procedure；失败后调用失败 procedure，由数据库事务写入退款 outbox / settlement 事实。进程崩溃时不依赖 Rust future `Drop` 才能发现需要退款；恢复处理根据持久化的 raw 操作状态完成退款。 + +raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-edit`，不能复用编辑器资源 ID、编辑器任务 ID 或 `external_generation_job`。 + +## Provider 边界 + +`platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、JSON DTO、base64 解码、预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 + +provider 返回的原始 `size` 必须沿 `GeneratedImages` 结果传回 api-server；raw handler 从该字段解析响应尺寸，再编码 `data[].b64_json`。 + +## 代码拆分 + +- `server-rs/crates/api-server/src/raw_image.rs`：独立路由 handler、请求/响应 DTO、base64 输入校验、预检查和 raw billing 编排。 +- `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、provider 请求映射和原始响应解码；现有编辑器调用通过默认选项复用，不在业务 handler 复制 provider 协议。 +- `server-rs/crates/api-server/src/modules/raw.rs`：只注册 `/api/raw/v1/images/edit` 并挂载 Bearer middleware。 + +不修改 External v1 OpenAPI；不在 `external_editor_api.rs`、编辑器项目模块或外部生成 worker 中增加 raw 分支。 + +## 验收 + +- 未认证请求被 Bearer middleware 拒绝。 +- 预检查失败时钱包无扣费、provider 无请求。 +- 成功响应严格只包含 `data[].b64_json`。 +- provider 失败时 raw 操作失败事务产生可恢复退款事实。 +- raw 请求不创建 `external_generation_job`，不写 editor project/resource/asset/OSS。 +- 运行 api-server 与 platform-image 定向测试、`npm run check:encoding` 和 `git diff --check`。 From 4e1316e2726ba9acd3ef86ff377400554c3a9751 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 10:36:48 +0800 Subject: [PATCH 002/248] =?UTF-8?q?=E5=AE=9E=E7=8E=B0=20Raw=20GPT=20Image?= =?UTF-8?q?=202=20=E5=9B=BE=E7=89=87=E7=BC=96=E8=BE=91=E4=BB=A3=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增受 Bearer 保护的 /api/raw/v1/images/edit JSON 路由 按单图 image 与可选 mask 转发 GPT Image 2 参数 固定 model 与 n 并返回仅含 data[].b64_json 的响应 复用钱包计费与退款边界并补充单元测试 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 27 ++- server-rs/crates/api-server/src/app.rs | 1 + server-rs/crates/api-server/src/main.rs | 1 + server-rs/crates/api-server/src/modules.rs | 1 + .../crates/api-server/src/modules/raw.rs | 11 + .../api-server/src/openai_image_generation.rs | 2 +- server-rs/crates/api-server/src/raw_image.rs | 219 ++++++++++++++++++ server-rs/crates/platform-image/src/lib.rs | 11 +- .../src/vector_engine/client.rs | 75 ++++-- .../src/vector_engine/curl_transport.rs | 60 ++++- .../platform-image/src/vector_engine/mod.rs | 7 +- .../src/vector_engine/raw_edit.rs | 33 +++ .../platform-image/src/vector_engine/types.rs | 12 + 13 files changed, 423 insertions(+), 37 deletions(-) create mode 100644 server-rs/crates/api-server/src/modules/raw.rs create mode 100644 server-rs/crates/api-server/src/raw_image.rs create mode 100644 server-rs/crates/platform-image/src/vector_engine/raw_edit.rs diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index ae006ffa7..759fe959d 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -18,12 +18,10 @@ POST /api/raw/v1/images/edit ```json { - "images": [ - { - "data": "", - "mimeType": "image/png" - } - ], + "image": { + "data": "", + "mimeType": "image/png" + }, "mask": { "data": "", "mimeType": "image/png" @@ -37,7 +35,7 @@ POST /api/raw/v1/images/edit } ``` -`images` 是必填数组，数组成员结构固定为 `{ data, mimeType }`；`mask` 可选并使用相同结构。输入格式由成员的 MIME 类型和解码后的图片字节共同确定，服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。`width`、`height` 为整数，组成发送给 provider 的输出尺寸；不把尺寸改写成业务字符串字段。 +`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。输入格式由 MIME 类型和解码后的图片字节共同确定，服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。`width`、`height` 为整数，组成发送给 provider 的输出尺寸；不把尺寸改写成业务字符串字段。 服务端发送给 `platform-image` 时固定注入： @@ -68,14 +66,15 @@ n = 1 所有请求、JSON、base64、图片结构和 provider 参数检查必须在扣费前完成。预检查失败直接返回 4xx，不产生钱包流水，也不调用 provider。 -检查通过后，api-server 调用一个 raw 图片操作的 SpacetimeDB 事务 procedure，在同一事务内完成： +检查通过后，api-server 进入现有资产操作计费边界，通过 SpacetimeDB 钱包事务 procedure 原子完成： -1. 以认证后的用户和请求 ID 建立 raw 操作幂等事实； -2. 按现有图片编辑算法解析价格：GPT Image 2 长边不超过 1536 使用 1K 价格，否则使用 2K 价格；当前默认价格为 3 / 5 泥点； -3. 原子扣除用户泥点并写入 `asset_operation_consume` 流水； -4. 持久化操作状态，供 provider 返回后成功或失败收口。 +1. 按现有图片编辑算法解析价格：GPT Image 2 长边不超过 1536 使用 1K 价格，否则使用 2K 价格；当前默认价格为 3 / 5 泥点； +2. 以认证后的用户、`raw-image-edit` 命名空间和请求 ID 组成幂等扣费流水 ID； +3. 原子扣除用户泥点并写入 `asset_operation_consume` 流水。 -provider 调用在 SpacetimeDB 事务之外执行。成功后调用同一 raw 操作的完成 procedure；失败后调用失败 procedure，由数据库事务写入退款 outbox / settlement 事实。进程崩溃时不依赖 Rust future `Drop` 才能发现需要退款；恢复处理根据持久化的 raw 操作状态完成退款。 +provider 调用在 SpacetimeDB 事务之外执行。失败时由现有计费边界把幂等退款事实写入 SpacetimeDB refund outbox，再由 worker 完成退款。 + +TODO：新增 raw 操作持久化状态，将“创建 raw 操作事实 + 扣费”收入同一事务，并由恢复 worker 对“已扣费但未收口”状态自动退款，填补进程在扣费后、写入 refund outbox 前崩溃的窗口。 raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-edit`，不能复用编辑器资源 ID、编辑器任务 ID 或 `external_generation_job`。 @@ -83,7 +82,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、JSON DTO、base64 解码、预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 返回的原始 `size` 必须沿 `GeneratedImages` 结果传回 api-server；raw handler 从该字段解析响应尺寸，再编码 `data[].b64_json`。 +provider 结果统一解码为图片字节；raw handler 只将这些字节编码到 `data[].b64_json`。 ## 代码拆分 diff --git a/server-rs/crates/api-server/src/app.rs b/server-rs/crates/api-server/src/app.rs index d898bdbf9..a3a761b3f 100644 --- a/server-rs/crates/api-server/src/app.rs +++ b/server-rs/crates/api-server/src/app.rs @@ -50,6 +50,7 @@ pub fn build_router(state: AppState) -> Router { .merge(modules::platform::router(state.clone())) .merge(modules::external_generation::router(state.clone())) .merge(modules::platform_support::router(state.clone())) + .merge(modules::raw::router(state.clone())) .merge(crate::error_reports::router(state.clone())) .route( "/api/profile/recharge/wechat/notify", diff --git a/server-rs/crates/api-server/src/main.rs b/server-rs/crates/api-server/src/main.rs index f40e6309e..25e42cd28 100644 --- a/server-rs/crates/api-server/src/main.rs +++ b/server-rs/crates/api-server/src/main.rs @@ -67,6 +67,7 @@ mod profile_identity; mod profile_recharge_expiration_listener; mod profile_recharge_refund_reconciliation; mod prompt; +mod raw_image; mod refresh_session; mod registration_reward; mod request_context; diff --git a/server-rs/crates/api-server/src/modules.rs b/server-rs/crates/api-server/src/modules.rs index 558c37c2b..76f5958b7 100644 --- a/server-rs/crates/api-server/src/modules.rs +++ b/server-rs/crates/api-server/src/modules.rs @@ -10,3 +10,4 @@ pub mod internal; pub mod platform; pub mod platform_support; pub mod profile; +pub mod raw; diff --git a/server-rs/crates/api-server/src/modules/raw.rs b/server-rs/crates/api-server/src/modules/raw.rs new file mode 100644 index 000000000..905c16584 --- /dev/null +++ b/server-rs/crates/api-server/src/modules/raw.rs @@ -0,0 +1,11 @@ +use axum::{Router, middleware, routing::post}; + +use crate::{auth::require_bearer_auth, raw_image::edit_raw_image, state::AppState}; + +pub fn router(state: AppState) -> Router { + Router::new().route( + "/api/raw/v1/images/edit", + post(edit_raw_image) + .route_layer(middleware::from_fn_with_state(state, require_bearer_auth)), + ) +} diff --git a/server-rs/crates/api-server/src/openai_image_generation.rs b/server-rs/crates/api-server/src/openai_image_generation.rs index e3e381eb5..f0fa7591c 100644 --- a/server-rs/crates/api-server/src/openai_image_generation.rs +++ b/server-rs/crates/api-server/src/openai_image_generation.rs @@ -414,7 +414,7 @@ impl OpenAiImageSettings { self } - fn provider_settings(&self) -> VectorEngineImageSettings { + pub(crate) fn provider_settings(&self) -> VectorEngineImageSettings { VectorEngineImageSettings { base_url: self.base_url.clone(), api_key: self.api_key.clone(), diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs new file mode 100644 index 000000000..15ab2c765 --- /dev/null +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -0,0 +1,219 @@ +use axum::{ + Json, + extract::{Extension, State}, + http::StatusCode, +}; +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; +use platform_image::{RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; + +use crate::{ + asset_billing::{ + execute_billable_asset_operation_with_cost, with_editor_generation_durable_billing_boundary, + }, + auth::AuthenticatedAccessToken, + http_error::AppError, + openai_image_generation::{ + build_openai_image_http_client, map_platform_image_error, require_openai_image_settings, + }, + request_context::RequestContext, + state::AppState, +}; + +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub(crate) struct RawImageData { + pub(crate) data: String, + pub(crate) mime_type: String, +} + +#[derive(Clone, Debug, Deserialize)] +#[serde(rename_all = "snake_case", deny_unknown_fields)] +pub(crate) struct RawImageEditRequest { + pub(crate) image: RawImageData, + pub(crate) mask: Option, + pub(crate) prompt: String, + pub(crate) quality: Option, + pub(crate) background: Option, + pub(crate) output_format: Option, + pub(crate) width: u32, + pub(crate) height: u32, +} + +#[derive(Debug, Serialize)] +pub(crate) struct RawImageEditItem { + pub(crate) b64_json: String, +} + +#[derive(Debug, Serialize)] +pub(crate) struct RawImageEditResponse { + pub(crate) data: Vec, +} + +pub(crate) async fn edit_raw_image( + State(state): State, + Extension(request_context): Extension, + Extension(authenticated): Extension, + Json(payload): Json, +) -> Result, AppError> { + let prepared = prepare_request(payload)?; + let settings = require_openai_image_settings(&state)?.with_external_api_audit_context( + &request_context, + Some(authenticated.claims().user_id().to_string()), + None, + ); + let http_client = build_openai_image_http_client(&settings)?; + let provider_settings = settings.provider_settings(); + let user_id = authenticated.claims().user_id().to_string(); + let request_id = request_context.request_id().to_string(); + let points_cost = raw_image_edit_price(&state, prepared.width, prepared.height).await?; + let operation = async move { + let generated = create_vector_engine_raw_image_edit( + &http_client, + &provider_settings, + prepared.prompt.as_str(), + &prepared.image, + prepared.options, + "raw_image_edit", + ) + .await + .map_err(map_platform_image_error)?; + let data = generated + .images + .into_iter() + .map(|image| RawImageEditItem { + b64_json: BASE64_STANDARD.encode(image.bytes), + }) + .collect(); + Ok::<_, AppError>(RawImageEditResponse { data }) + }; + let result = with_editor_generation_durable_billing_boundary( + execute_billable_asset_operation_with_cost( + &state, + user_id.as_str(), + "raw-image-edit", + request_id.as_str(), + u64::from(points_cost), + operation, + ), + ) + .await?; + Ok(Json(serde_json::to_value(result).map_err(|error| { + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_message(error.to_string()) + })?)) +} + +struct PreparedRawImageEdit { + image: ReferenceImage, + prompt: String, + options: RawImageEditOptions, + width: u32, + height: u32, +} + +fn prepare_request(payload: RawImageEditRequest) -> Result { + if payload.prompt.trim().is_empty() { + return Err(bad_request("prompt 不能为空")); + } + if payload.width == 0 || payload.height == 0 { + return Err(bad_request("width 和 height 必须为正整数")); + } + let image = decode_image(payload.image, "image")?; + let mask = payload + .mask + .map(|value| decode_image(value, "mask")) + .transpose()?; + Ok(PreparedRawImageEdit { + image, + prompt: payload.prompt, + options: RawImageEditOptions { + quality: payload.quality, + background: payload.background, + output_format: payload.output_format, + width: payload.width, + height: payload.height, + mask, + }, + width: payload.width, + height: payload.height, + }) +} + +fn decode_image(value: RawImageData, field: &str) -> Result { + let mime_type = value.mime_type.trim().to_string(); + if mime_type.is_empty() { + return Err(bad_request(format!("{field}.mimeType 不能为空"))); + } + let bytes = BASE64_STANDARD + .decode(value.data.trim()) + .map_err(|error| bad_request(format!("{field}.data 必须是有效 base64：{error}")))?; + Ok(ReferenceImage { + bytes, + file_name: format!("{field}.png"), + mime_type, + }) +} + +async fn raw_image_edit_price(state: &AppState, width: u32, height: u32) -> Result { + let tier = if width.max(height) > 1536 { "2K" } else { "1K" }; + state + .editor_generation_pricing() + .await + .map(|pricing| { + pricing.image_generation_mud_points(Some("quick-edit"), Some("gpt-image-2"), Some(tier)) + }) + .map_err(|error| { + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ + "provider": "editor-generation-pricing", + "message": error.to_string(), + })) + }) +} + +fn bad_request(message: impl Into) -> AppError { + AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ + "provider": "raw-image-edit", + "message": message.into(), + })) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn request_uses_one_image_object_and_rejects_images_array() { + let payload = serde_json::json!({ + "image": {"data": "aGVsbG8=", "mimeType": "image/png"}, + "prompt": "edit", + "width": 1024, + "height": 1024 + }); + let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("image object"); + let prepared = prepare_request(parsed).expect("request should prepare"); + assert_eq!(prepared.image.bytes, b"hello"); + + let array_payload = serde_json::json!({ + "images": [{"data": "aGVsbG8=", "mimeType": "image/png"}], + "prompt": "edit", + "width": 1024, + "height": 1024 + }); + assert!(serde_json::from_value::(array_payload).is_err()); + } + + #[test] + fn response_contains_only_data_b64_json() { + let response = serde_json::to_value(RawImageEditResponse { + data: vec![RawImageEditItem { + b64_json: "aGVsbG8=".to_string(), + }], + }) + .expect("response should serialize"); + assert_eq!( + response, + serde_json::json!({"data": [{"b64_json": "aGVsbG8="}]}) + ); + } +} diff --git a/server-rs/crates/platform-image/src/lib.rs b/server-rs/crates/platform-image/src/lib.rs index 8db6a9c97..1e6e757bd 100644 --- a/server-rs/crates/platform-image/src/lib.rs +++ b/server-rs/crates/platform-image/src/lib.rs @@ -10,14 +10,15 @@ pub use pixel_art_snapper::{ }; pub use vector_engine::{ DownloadedImage, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, GeneratedImages, NANOBANANA_2_MODEL, - PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, ReferenceImage, - VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, VectorEngineImageSettings, - build_vector_engine_image_http_client, build_vector_engine_image_request_body, + PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, RawImageEditOptions, + ReferenceImage, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, + VectorEngineImageSettings, build_vector_engine_image_http_client, + build_vector_engine_image_request_body, build_vector_engine_nanobanana_generate_content_request_body, create_vector_engine_image_edit, create_vector_engine_image_edit_with_references, create_vector_engine_image_edit_with_references_and_model, create_vector_engine_image_generation, create_vector_engine_image_generation_with_model, - create_vector_engine_nanobanana_generate_content, download_remote_image, - vector_engine_images_edit_url, vector_engine_images_generation_url, + create_vector_engine_nanobanana_generate_content, create_vector_engine_raw_image_edit, + download_remote_image, vector_engine_images_edit_url, vector_engine_images_generation_url, vector_engine_nanobanana_generate_content_url, }; diff --git a/server-rs/crates/platform-image/src/vector_engine/client.rs b/server-rs/crates/platform-image/src/vector_engine/client.rs index 24b10196a..5e1abfe1c 100644 --- a/server-rs/crates/platform-image/src/vector_engine/client.rs +++ b/server-rs/crates/platform-image/src/vector_engine/client.rs @@ -16,6 +16,7 @@ use super::{ curl_transport::{ map_curl_error, send_vector_engine_json_request_with_curl, send_vector_engine_multipart_edit_request_with_curl, + send_vector_engine_multipart_edit_request_with_curl_options, }, error::PlatformImageError, image_source::resolve_reference_images, @@ -28,7 +29,7 @@ use super::{ vector_engine_nanobanana_generate_content_url, }, response::handle_vector_engine_response, - types::{GeneratedImages, ReferenceImage, VectorEngineImageSettings}, + types::{GeneratedImages, RawImageEditOptions, ReferenceImage, VectorEngineImageSettings}, util::truncate_raw, }; @@ -537,6 +538,34 @@ pub async fn create_vector_engine_image_edit_with_references_and_model( candidate_count: u32, reference_images: &[ReferenceImage], failure_context: &str, +) -> Result { + create_vector_engine_image_edit_with_references_and_model_and_options( + http_client, + settings, + model, + prompt, + negative_prompt, + size, + candidate_count, + reference_images, + None, + failure_context, + ) + .await +} + +#[allow(clippy::too_many_arguments)] +pub async fn create_vector_engine_image_edit_with_references_and_model_and_options( + http_client: &reqwest::Client, + settings: &VectorEngineImageSettings, + model: &str, + prompt: &str, + negative_prompt: Option<&str>, + size: &str, + candidate_count: u32, + reference_images: &[ReferenceImage], + options: Option<&RawImageEditOptions>, + failure_context: &str, ) -> Result { let requested_model = normalize_vector_engine_image_model(model); if reference_images.is_empty() { @@ -611,19 +640,37 @@ pub async fn create_vector_engine_image_edit_with_references_and_model( &mut recovered_failure_audits, )); }; - let response = match send_vector_engine_multipart_edit_request_with_curl( - request_url.as_str(), - settings.api_key.as_str(), - upstream_model, - prompt, - negative_prompt, - normalized_size.as_str(), - candidate_count, - reference_images, - attempt_timeout_ms, - ) - .await - { + let response = match match options { + Some(options) => { + send_vector_engine_multipart_edit_request_with_curl_options( + request_url.as_str(), + settings.api_key.as_str(), + upstream_model, + prompt, + negative_prompt, + normalized_size.as_str(), + candidate_count, + reference_images, + Some(options), + attempt_timeout_ms, + ) + .await + } + None => { + send_vector_engine_multipart_edit_request_with_curl( + request_url.as_str(), + settings.api_key.as_str(), + upstream_model, + prompt, + negative_prompt, + normalized_size.as_str(), + candidate_count, + reference_images, + attempt_timeout_ms, + ) + .await + } + } { Ok(response) => { if should_retry_vector_engine_upstream_response( response.status, diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index fbe94e1b1..6e653356c 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -7,8 +7,11 @@ use curl::{ use serde_json::Value; use super::{ - audit::build_failure_audit, constants::VECTOR_ENGINE_PROVIDER, error::PlatformImageError, - request::build_prompt_with_negative, types::ReferenceImage, + audit::build_failure_audit, + constants::VECTOR_ENGINE_PROVIDER, + error::PlatformImageError, + request::build_prompt_with_negative, + types::{RawImageEditOptions, ReferenceImage}, }; #[derive(Debug)] @@ -119,6 +122,34 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl( candidate_count: u32, reference_images: &[ReferenceImage], timeout_ms: u64, +) -> Result { + send_vector_engine_multipart_edit_request_with_curl_options( + request_url, + api_key, + model, + prompt, + negative_prompt, + normalized_size, + candidate_count, + reference_images, + None, + timeout_ms, + ) + .await +} + +#[allow(clippy::too_many_arguments)] +pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( + request_url: &str, + api_key: &str, + model: &str, + prompt: &str, + negative_prompt: Option<&str>, + normalized_size: &str, + candidate_count: u32, + reference_images: &[ReferenceImage], + options: Option<&RawImageEditOptions>, + timeout_ms: u64, ) -> Result { let request_url = request_url.to_string(); let api_key = api_key.to_string(); @@ -127,6 +158,7 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl( let negative_prompt = negative_prompt.map(str::to_string); let normalized_size = normalized_size.to_string(); let reference_images = reference_images.to_vec(); + let options = options.cloned(); tokio::task::spawn_blocking(move || { send_multipart_edit_request_with_curl_blocking( request_url.as_str(), @@ -137,6 +169,7 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl( normalized_size.as_str(), candidate_count, reference_images.as_slice(), + options.as_ref(), timeout_ms, ) }) @@ -239,6 +272,7 @@ fn send_multipart_edit_request_with_curl_blocking( normalized_size: &str, candidate_count: u32, reference_images: &[ReferenceImage], + options: Option<&RawImageEditOptions>, timeout_ms: u64, ) -> Result { let mut form = Form::new(); @@ -253,6 +287,28 @@ fn send_multipart_edit_request_with_curl_blocking( .contents(normalized_size.as_bytes()) .add()?; + if let Some(options) = options { + if let Some(quality) = options.quality.as_deref() { + form.part("quality").contents(quality.as_bytes()).add()?; + } + if let Some(background) = options.background.as_deref() { + form.part("background") + .contents(background.as_bytes()) + .add()?; + } + if let Some(output_format) = options.output_format.as_deref() { + form.part("output_format") + .contents(output_format.as_bytes()) + .add()?; + } + if let Some(mask) = options.mask.as_ref() { + form.part("mask") + .buffer(mask.file_name.as_str(), mask.bytes.clone()) + .content_type(mask.mime_type.as_str()) + .add()?; + } + } + for reference_image in reference_images { form.part("image") .buffer( diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index f64cba54a..a20f76189 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -6,6 +6,7 @@ mod curl_transport; mod error; mod image_source; mod payload; +mod raw_edit; mod request; mod response; mod transport; @@ -25,6 +26,7 @@ pub use constants::{ }; pub use error::{PlatformImageError, PlatformImageStatusHint}; pub use image_source::download_remote_image; +pub use raw_edit::create_vector_engine_raw_image_edit; pub use request::{ build_vector_engine_image_request_body, build_vector_engine_image_request_body_with_model, build_vector_engine_nanobanana_generate_content_request_body, normalize_image_size_for_model, @@ -32,4 +34,7 @@ pub use request::{ vector_engine_nanobanana_generate_content_url, }; pub use transport::build_vector_engine_image_http_client; -pub use types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}; +pub use types::{ + DownloadedImage, GeneratedImages, RawImageEditOptions, ReferenceImage, + VectorEngineImageSettings, +}; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs new file mode 100644 index 000000000..b7d6f49dd --- /dev/null +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -0,0 +1,33 @@ +use super::{ + client::create_vector_engine_image_edit_with_references_and_model_and_options, + constants::GPT_IMAGE_2_MODEL, + error::PlatformImageError, + types::{GeneratedImages, RawImageEditOptions, ReferenceImage, VectorEngineImageSettings}, +}; + +/// Sends the raw GPT Image 2 edit contract while keeping VectorEngine's +/// multipart transport inside this crate. +#[allow(clippy::too_many_arguments)] +pub async fn create_vector_engine_raw_image_edit( + http_client: &reqwest::Client, + settings: &VectorEngineImageSettings, + prompt: &str, + image: &ReferenceImage, + options: RawImageEditOptions, + failure_context: &str, +) -> Result { + let size = format!("{}x{}", options.width, options.height); + create_vector_engine_image_edit_with_references_and_model_and_options( + http_client, + settings, + GPT_IMAGE_2_MODEL, + prompt, + None, + size.as_str(), + 1, + std::slice::from_ref(image), + Some(&options), + failure_context, + ) + .await +} diff --git a/server-rs/crates/platform-image/src/vector_engine/types.rs b/server-rs/crates/platform-image/src/vector_engine/types.rs index 77fbd19f9..818cafe60 100644 --- a/server-rs/crates/platform-image/src/vector_engine/types.rs +++ b/server-rs/crates/platform-image/src/vector_engine/types.rs @@ -29,3 +29,15 @@ pub struct ReferenceImage { pub mime_type: String, pub file_name: String, } + +/// Raw GPT Image 2 edit options. The API layer owns validation; this type only +/// carries values that must be forwarded to VectorEngine. +#[derive(Clone, Debug, Default)] +pub struct RawImageEditOptions { + pub quality: Option, + pub background: Option, + pub output_format: Option, + pub width: u32, + pub height: u32, + pub mask: Option, +} From 18d60feb0481144660f07e6ad12478daa49a89d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 10:40:09 +0800 Subject: [PATCH 003/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=8F=82=E6=95=B0=E9=A2=84=E6=A3=80=E6=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在计费前校验质量、背景和输出格式 规范化可选参数并保持单图请求契约 --- server-rs/crates/api-server/src/raw_image.rs | 44 ++++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 15ab2c765..9b4cde5ac 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -119,6 +119,24 @@ fn prepare_request(payload: RawImageEditRequest) -> Result Result Result) -> Option { + value + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) +} + +fn validate_optional_value( + value: Option<&str>, + field: &str, + allowed: [&str; N], +) -> Result<(), AppError> { + let Some(value) = value.map(str::trim).filter(|value| !value.is_empty()) else { + return Ok(()); + }; + if allowed.contains(&value) { + return Ok(()); + } + Err(bad_request(format!("{field} 值无效"))) +} + fn decode_image(value: RawImageData, field: &str) -> Result { let mime_type = value.mime_type.trim().to_string(); if mime_type.is_empty() { From 5836aaec26feff0bdc7e974c12a5d12b7061d3c7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 12:10:56 +0800 Subject: [PATCH 004/248] =?UTF-8?q?=E7=AE=80=E5=8C=96=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E7=BC=96=E8=BE=91=20multipart=20=E8=AF=B7=E6=B1=82=E5=88=86?= =?UTF-8?q?=E6=94=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 统一通过带选项的 multipart 传输函数发送请求 移除冗余包装函数调用与未使用导入 --- .../src/vector_engine/client.rs | 46 ++++++------------- 1 file changed, 14 insertions(+), 32 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/client.rs b/server-rs/crates/platform-image/src/vector_engine/client.rs index 5e1abfe1c..f81a4613e 100644 --- a/server-rs/crates/platform-image/src/vector_engine/client.rs +++ b/server-rs/crates/platform-image/src/vector_engine/client.rs @@ -15,7 +15,6 @@ use super::{ }, curl_transport::{ map_curl_error, send_vector_engine_json_request_with_curl, - send_vector_engine_multipart_edit_request_with_curl, send_vector_engine_multipart_edit_request_with_curl_options, }, error::PlatformImageError, @@ -640,37 +639,20 @@ pub async fn create_vector_engine_image_edit_with_references_and_model_and_optio &mut recovered_failure_audits, )); }; - let response = match match options { - Some(options) => { - send_vector_engine_multipart_edit_request_with_curl_options( - request_url.as_str(), - settings.api_key.as_str(), - upstream_model, - prompt, - negative_prompt, - normalized_size.as_str(), - candidate_count, - reference_images, - Some(options), - attempt_timeout_ms, - ) - .await - } - None => { - send_vector_engine_multipart_edit_request_with_curl( - request_url.as_str(), - settings.api_key.as_str(), - upstream_model, - prompt, - negative_prompt, - normalized_size.as_str(), - candidate_count, - reference_images, - attempt_timeout_ms, - ) - .await - } - } { + let response = send_vector_engine_multipart_edit_request_with_curl_options( + request_url.as_str(), + settings.api_key.as_str(), + upstream_model, + prompt, + negative_prompt, + normalized_size.as_str(), + candidate_count, + reference_images, + options, + attempt_timeout_ms, + ) + .await; + let response = match response { Ok(response) => { if should_retry_vector_engine_upstream_response( response.status, From d1f9f4e1efe88a0866a8af7d9aa0213b5f8efdf0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 12:16:00 +0800 Subject: [PATCH 005/248] =?UTF-8?q?=E5=87=8F=E5=B0=91=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E7=BC=96=E8=BE=91=20multipart=20=E5=AD=97=E8=8A=82=E5=A4=8D?= =?UTF-8?q?=E5=88=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将已克隆的选项按值传入阻塞传输函数 构造 mask 与参考图表单时直接移动字节缓冲 --- .../src/vector_engine/curl_transport.rs | 44 ++++++++++++------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index 6e653356c..fdc35a71d 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -168,8 +168,8 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( negative_prompt.as_deref(), normalized_size.as_str(), candidate_count, - reference_images.as_slice(), - options.as_ref(), + reference_images, + options, timeout_ms, ) }) @@ -271,8 +271,8 @@ fn send_multipart_edit_request_with_curl_blocking( negative_prompt: Option<&str>, normalized_size: &str, candidate_count: u32, - reference_images: &[ReferenceImage], - options: Option<&RawImageEditOptions>, + reference_images: Vec, + options: Option, timeout_ms: u64, ) -> Result { let mut form = Form::new(); @@ -288,34 +288,48 @@ fn send_multipart_edit_request_with_curl_blocking( .add()?; if let Some(options) = options { - if let Some(quality) = options.quality.as_deref() { + let RawImageEditOptions { + quality, + background, + output_format, + mask, + .. + } = options; + if let Some(quality) = quality { form.part("quality").contents(quality.as_bytes()).add()?; } - if let Some(background) = options.background.as_deref() { + if let Some(background) = background { form.part("background") .contents(background.as_bytes()) .add()?; } - if let Some(output_format) = options.output_format.as_deref() { + if let Some(output_format) = output_format { form.part("output_format") .contents(output_format.as_bytes()) .add()?; } - if let Some(mask) = options.mask.as_ref() { + if let Some(mask) = mask { + let ReferenceImage { + bytes, + mime_type, + file_name, + } = mask; form.part("mask") - .buffer(mask.file_name.as_str(), mask.bytes.clone()) - .content_type(mask.mime_type.as_str()) + .buffer(file_name.as_str(), bytes) + .content_type(mime_type.as_str()) .add()?; } } for reference_image in reference_images { + let ReferenceImage { + bytes, + mime_type, + file_name, + } = reference_image; form.part("image") - .buffer( - reference_image.file_name.as_str(), - reference_image.bytes.clone(), - ) - .content_type(reference_image.mime_type.as_str()) + .buffer(file_name.as_str(), bytes) + .content_type(mime_type.as_str()) .add()?; } From 4dcf38715bacf1ac75ba6d322c011ecb2909f34e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 12:21:11 +0800 Subject: [PATCH 006/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E7=BC=96=E8=BE=91=E9=80=89=E9=A1=B9=20multipart=20=E6=B5=8B?= =?UTF-8?q?=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 覆盖质量、背景、输出格式与 mask 文件字段 断言文件名、MIME 类型和字节内容均正确发送 --- .../src/vector_engine/curl_transport.rs | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index fdc35a71d..42832b6ed 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -429,6 +429,53 @@ mod tests { server.abort(); } + #[tokio::test] + async fn vector_engine_curl_transport_posts_raw_edit_options_and_mask() { + let (base_url, server, request_rx) = start_single_response_server().await; + let response = send_vector_engine_multipart_edit_request_with_curl_options( + format!("{base_url}/v1/images/edits").as_str(), + "test-key", + GPT_IMAGE_2_MODEL, + "测试提示词", + None, + "1536x1024", + 1, + &[ReferenceImage { + bytes: b"reference".to_vec(), + mime_type: "image/webp".to_string(), + file_name: "reference.webp".to_string(), + }], + Some(&RawImageEditOptions { + quality: Some("high".to_string()), + background: Some("transparent".to_string()), + output_format: Some("png".to_string()), + width: 1536, + height: 1024, + mask: Some(ReferenceImage { + bytes: b"mask-bytes".to_vec(), + mime_type: "image/png".to_string(), + file_name: "mask.png".to_string(), + }), + }), + 1_000, + ) + .await + .expect("curl multipart raw edit request should succeed"); + + assert_eq!(response.status, 200); + let request = request_rx + .await + .expect("mock server should capture request"); + let request_text = String::from_utf8_lossy(request.as_slice()); + assert!(request_text.contains("name=\"quality\"\r\n\r\nhigh")); + assert!(request_text.contains("name=\"background\"\r\n\r\ntransparent")); + assert!(request_text.contains("name=\"output_format\"\r\n\r\npng")); + assert!(request_text.contains("name=\"mask\"; filename=\"mask.png\"")); + assert!(request_text.contains("Content-Type: image/png")); + assert!(request_text.contains("mask-bytes")); + server.abort(); + } + async fn start_single_response_server() -> ( String, tokio::task::JoinHandle<()>, From a576fdcc37f70a2145fbbb0030bf7dc1b69f3f47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 12:29:59 +0800 Subject: [PATCH 007/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=E6=9C=AA=E4=BD=BF?= =?UTF-8?q?=E7=94=A8=E7=9A=84=20multipart=20=E8=AF=B7=E6=B1=82=E5=8C=85?= =?UTF-8?q?=E8=A3=85=E5=87=BD=E6=95=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 无选项测试直接调用统一传输入口 避免生产构建保留未使用函数告警 --- .../src/vector_engine/curl_transport.rs | 30 ++----------------- 1 file changed, 2 insertions(+), 28 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index 42832b6ed..59ab22df2 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -111,33 +111,6 @@ pub(crate) async fn send_vector_engine_json_request_with_curl( .map_err(VectorEngineCurlError::WorkerJoin)? } -#[allow(clippy::too_many_arguments)] -pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl( - request_url: &str, - api_key: &str, - model: &str, - prompt: &str, - negative_prompt: Option<&str>, - normalized_size: &str, - candidate_count: u32, - reference_images: &[ReferenceImage], - timeout_ms: u64, -) -> Result { - send_vector_engine_multipart_edit_request_with_curl_options( - request_url, - api_key, - model, - prompt, - negative_prompt, - normalized_size, - candidate_count, - reference_images, - None, - timeout_ms, - ) - .await -} - #[allow(clippy::too_many_arguments)] pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( request_url: &str, @@ -399,7 +372,7 @@ mod tests { #[tokio::test] async fn vector_engine_curl_transport_posts_multipart_request() { let (base_url, server, request_rx) = start_single_response_server().await; - let response = send_vector_engine_multipart_edit_request_with_curl( + let response = send_vector_engine_multipart_edit_request_with_curl_options( format!("{base_url}/v1/images/edits").as_str(), "test-key", GPT_IMAGE_2_MODEL, @@ -412,6 +385,7 @@ mod tests { mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }], + None, 1_000, ) .await From 7fc1b841be5369f1082bcca2232f08ab8d7238f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 12:30:55 +0800 Subject: [PATCH 008/248] =?UTF-8?q?=E6=94=B6=E5=8F=A3=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=20base64=20=E6=A0=A1=E9=AA=8C=E9=94=99=E8=AF=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 客户端仅返回稳定的 base64 格式错误提示 补充测试避免泄露解码器内部诊断细节 --- server-rs/crates/api-server/src/raw_image.rs | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 9b4cde5ac..f26294861 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -185,7 +185,7 @@ fn decode_image(value: RawImageData, field: &str) -> Result panic!("invalid base64 should fail"), + Err(error) => error, + }; + let rendered = format!("{error:?}"); + assert!(rendered.contains("image.data 必须是有效 base64")); + assert!(!rendered.contains("InvalidByte")); + } } From f087ba3e2bb5600858abd8efc5b886aaa9efbbbc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 13:54:31 +0800 Subject: [PATCH 009/248] =?UTF-8?q?=E9=9A=94=E7=A6=BB=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E4=BB=A3=E7=90=86=E5=AE=9E=E7=8E=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 移除对现有图片编辑 client 与 curl transport 的复用 将 Raw GPT Image 2 请求、multipart 构造和响应解析收敛到独立文件 保持现有 editor 图片链路代码不变 --- server-rs/crates/api-server/src/raw_image.rs | 6 +- .../src/vector_engine/client.rs | 39 +--- .../src/vector_engine/curl_transport.rs | 113 ++--------- .../platform-image/src/vector_engine/mod.rs | 7 +- .../src/vector_engine/raw_edit.rs | 181 ++++++++++++++++-- .../platform-image/src/vector_engine/types.rs | 12 -- 6 files changed, 179 insertions(+), 179 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index f26294861..c083a7fd7 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -14,9 +14,7 @@ use crate::{ }, auth::AuthenticatedAccessToken, http_error::AppError, - openai_image_generation::{ - build_openai_image_http_client, map_platform_image_error, require_openai_image_settings, - }, + openai_image_generation::{map_platform_image_error, require_openai_image_settings}, request_context::RequestContext, state::AppState, }; @@ -63,14 +61,12 @@ pub(crate) async fn edit_raw_image( Some(authenticated.claims().user_id().to_string()), None, ); - let http_client = build_openai_image_http_client(&settings)?; let provider_settings = settings.provider_settings(); let user_id = authenticated.claims().user_id().to_string(); let request_id = request_context.request_id().to_string(); let points_cost = raw_image_edit_price(&state, prepared.width, prepared.height).await?; let operation = async move { let generated = create_vector_engine_raw_image_edit( - &http_client, &provider_settings, prepared.prompt.as_str(), &prepared.image, diff --git a/server-rs/crates/platform-image/src/vector_engine/client.rs b/server-rs/crates/platform-image/src/vector_engine/client.rs index f81a4613e..24b10196a 100644 --- a/server-rs/crates/platform-image/src/vector_engine/client.rs +++ b/server-rs/crates/platform-image/src/vector_engine/client.rs @@ -15,7 +15,7 @@ use super::{ }, curl_transport::{ map_curl_error, send_vector_engine_json_request_with_curl, - send_vector_engine_multipart_edit_request_with_curl_options, + send_vector_engine_multipart_edit_request_with_curl, }, error::PlatformImageError, image_source::resolve_reference_images, @@ -28,7 +28,7 @@ use super::{ vector_engine_nanobanana_generate_content_url, }, response::handle_vector_engine_response, - types::{GeneratedImages, RawImageEditOptions, ReferenceImage, VectorEngineImageSettings}, + types::{GeneratedImages, ReferenceImage, VectorEngineImageSettings}, util::truncate_raw, }; @@ -537,34 +537,6 @@ pub async fn create_vector_engine_image_edit_with_references_and_model( candidate_count: u32, reference_images: &[ReferenceImage], failure_context: &str, -) -> Result { - create_vector_engine_image_edit_with_references_and_model_and_options( - http_client, - settings, - model, - prompt, - negative_prompt, - size, - candidate_count, - reference_images, - None, - failure_context, - ) - .await -} - -#[allow(clippy::too_many_arguments)] -pub async fn create_vector_engine_image_edit_with_references_and_model_and_options( - http_client: &reqwest::Client, - settings: &VectorEngineImageSettings, - model: &str, - prompt: &str, - negative_prompt: Option<&str>, - size: &str, - candidate_count: u32, - reference_images: &[ReferenceImage], - options: Option<&RawImageEditOptions>, - failure_context: &str, ) -> Result { let requested_model = normalize_vector_engine_image_model(model); if reference_images.is_empty() { @@ -639,7 +611,7 @@ pub async fn create_vector_engine_image_edit_with_references_and_model_and_optio &mut recovered_failure_audits, )); }; - let response = send_vector_engine_multipart_edit_request_with_curl_options( + let response = match send_vector_engine_multipart_edit_request_with_curl( request_url.as_str(), settings.api_key.as_str(), upstream_model, @@ -648,11 +620,10 @@ pub async fn create_vector_engine_image_edit_with_references_and_model_and_optio normalized_size.as_str(), candidate_count, reference_images, - options, attempt_timeout_ms, ) - .await; - let response = match response { + .await + { Ok(response) => { if should_retry_vector_engine_upstream_response( response.status, diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index 59ab22df2..fbe94e1b1 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -7,11 +7,8 @@ use curl::{ use serde_json::Value; use super::{ - audit::build_failure_audit, - constants::VECTOR_ENGINE_PROVIDER, - error::PlatformImageError, - request::build_prompt_with_negative, - types::{RawImageEditOptions, ReferenceImage}, + audit::build_failure_audit, constants::VECTOR_ENGINE_PROVIDER, error::PlatformImageError, + request::build_prompt_with_negative, types::ReferenceImage, }; #[derive(Debug)] @@ -112,7 +109,7 @@ pub(crate) async fn send_vector_engine_json_request_with_curl( } #[allow(clippy::too_many_arguments)] -pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( +pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl( request_url: &str, api_key: &str, model: &str, @@ -121,7 +118,6 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( normalized_size: &str, candidate_count: u32, reference_images: &[ReferenceImage], - options: Option<&RawImageEditOptions>, timeout_ms: u64, ) -> Result { let request_url = request_url.to_string(); @@ -131,7 +127,6 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( let negative_prompt = negative_prompt.map(str::to_string); let normalized_size = normalized_size.to_string(); let reference_images = reference_images.to_vec(); - let options = options.cloned(); tokio::task::spawn_blocking(move || { send_multipart_edit_request_with_curl_blocking( request_url.as_str(), @@ -141,8 +136,7 @@ pub(crate) async fn send_vector_engine_multipart_edit_request_with_curl_options( negative_prompt.as_deref(), normalized_size.as_str(), candidate_count, - reference_images, - options, + reference_images.as_slice(), timeout_ms, ) }) @@ -244,8 +238,7 @@ fn send_multipart_edit_request_with_curl_blocking( negative_prompt: Option<&str>, normalized_size: &str, candidate_count: u32, - reference_images: Vec, - options: Option, + reference_images: &[ReferenceImage], timeout_ms: u64, ) -> Result { let mut form = Form::new(); @@ -260,49 +253,13 @@ fn send_multipart_edit_request_with_curl_blocking( .contents(normalized_size.as_bytes()) .add()?; - if let Some(options) = options { - let RawImageEditOptions { - quality, - background, - output_format, - mask, - .. - } = options; - if let Some(quality) = quality { - form.part("quality").contents(quality.as_bytes()).add()?; - } - if let Some(background) = background { - form.part("background") - .contents(background.as_bytes()) - .add()?; - } - if let Some(output_format) = output_format { - form.part("output_format") - .contents(output_format.as_bytes()) - .add()?; - } - if let Some(mask) = mask { - let ReferenceImage { - bytes, - mime_type, - file_name, - } = mask; - form.part("mask") - .buffer(file_name.as_str(), bytes) - .content_type(mime_type.as_str()) - .add()?; - } - } - for reference_image in reference_images { - let ReferenceImage { - bytes, - mime_type, - file_name, - } = reference_image; form.part("image") - .buffer(file_name.as_str(), bytes) - .content_type(mime_type.as_str()) + .buffer( + reference_image.file_name.as_str(), + reference_image.bytes.clone(), + ) + .content_type(reference_image.mime_type.as_str()) .add()?; } @@ -372,7 +329,7 @@ mod tests { #[tokio::test] async fn vector_engine_curl_transport_posts_multipart_request() { let (base_url, server, request_rx) = start_single_response_server().await; - let response = send_vector_engine_multipart_edit_request_with_curl_options( + let response = send_vector_engine_multipart_edit_request_with_curl( format!("{base_url}/v1/images/edits").as_str(), "test-key", GPT_IMAGE_2_MODEL, @@ -385,7 +342,6 @@ mod tests { mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }], - None, 1_000, ) .await @@ -403,53 +359,6 @@ mod tests { server.abort(); } - #[tokio::test] - async fn vector_engine_curl_transport_posts_raw_edit_options_and_mask() { - let (base_url, server, request_rx) = start_single_response_server().await; - let response = send_vector_engine_multipart_edit_request_with_curl_options( - format!("{base_url}/v1/images/edits").as_str(), - "test-key", - GPT_IMAGE_2_MODEL, - "测试提示词", - None, - "1536x1024", - 1, - &[ReferenceImage { - bytes: b"reference".to_vec(), - mime_type: "image/webp".to_string(), - file_name: "reference.webp".to_string(), - }], - Some(&RawImageEditOptions { - quality: Some("high".to_string()), - background: Some("transparent".to_string()), - output_format: Some("png".to_string()), - width: 1536, - height: 1024, - mask: Some(ReferenceImage { - bytes: b"mask-bytes".to_vec(), - mime_type: "image/png".to_string(), - file_name: "mask.png".to_string(), - }), - }), - 1_000, - ) - .await - .expect("curl multipart raw edit request should succeed"); - - assert_eq!(response.status, 200); - let request = request_rx - .await - .expect("mock server should capture request"); - let request_text = String::from_utf8_lossy(request.as_slice()); - assert!(request_text.contains("name=\"quality\"\r\n\r\nhigh")); - assert!(request_text.contains("name=\"background\"\r\n\r\ntransparent")); - assert!(request_text.contains("name=\"output_format\"\r\n\r\npng")); - assert!(request_text.contains("name=\"mask\"; filename=\"mask.png\"")); - assert!(request_text.contains("Content-Type: image/png")); - assert!(request_text.contains("mask-bytes")); - server.abort(); - } - async fn start_single_response_server() -> ( String, tokio::task::JoinHandle<()>, diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index a20f76189..aacb3256d 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -26,7 +26,7 @@ pub use constants::{ }; pub use error::{PlatformImageError, PlatformImageStatusHint}; pub use image_source::download_remote_image; -pub use raw_edit::create_vector_engine_raw_image_edit; +pub use raw_edit::{RawImageEditOptions, create_vector_engine_raw_image_edit}; pub use request::{ build_vector_engine_image_request_body, build_vector_engine_image_request_body_with_model, build_vector_engine_nanobanana_generate_content_request_body, normalize_image_size_for_model, @@ -34,7 +34,4 @@ pub use request::{ vector_engine_nanobanana_generate_content_url, }; pub use transport::build_vector_engine_image_http_client; -pub use types::{ - DownloadedImage, GeneratedImages, RawImageEditOptions, ReferenceImage, - VectorEngineImageSettings, -}; +pub use types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index b7d6f49dd..435a9e1f3 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -1,33 +1,172 @@ +use std::time::Duration; + +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; +use reqwest::multipart::{Form, Part}; +use serde_json::Value; + use super::{ - client::create_vector_engine_image_edit_with_references_and_model_and_options, - constants::GPT_IMAGE_2_MODEL, + constants::{GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER}, error::PlatformImageError, - types::{GeneratedImages, RawImageEditOptions, ReferenceImage, VectorEngineImageSettings}, + request::vector_engine_images_edit_url, + types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}, }; -/// Sends the raw GPT Image 2 edit contract while keeping VectorEngine's -/// multipart transport inside this crate. -#[allow(clippy::too_many_arguments)] +#[derive(Clone, Debug, Default)] +pub struct RawImageEditOptions { + pub quality: Option, + pub background: Option, + pub output_format: Option, + pub width: u32, + pub height: u32, + pub mask: Option, +} + +/// Independent raw GPT Image 2 proxy; it does not call the editor image-edit client. pub async fn create_vector_engine_raw_image_edit( - http_client: &reqwest::Client, settings: &VectorEngineImageSettings, prompt: &str, image: &ReferenceImage, options: RawImageEditOptions, failure_context: &str, ) -> Result { - let size = format!("{}x{}", options.width, options.height); - create_vector_engine_image_edit_with_references_and_model_and_options( - http_client, - settings, - GPT_IMAGE_2_MODEL, - prompt, - None, - size.as_str(), - 1, - std::slice::from_ref(image), - Some(&options), - failure_context, - ) - .await + let url = vector_engine_images_edit_url(settings); + let mut form = Form::new() + .text("model", GPT_IMAGE_2_MODEL.to_string()) + .text("n", "1".to_string()) + .text("prompt", prompt.to_string()) + .text("size", format!("{}x{}", options.width, options.height)) + .part( + "image", + Part::bytes(image.bytes.clone()) + .file_name(image.file_name.clone()) + .mime_str(image.mime_type.as_str()) + .map_err(|error| invalid_request(failure_context, error.to_string()))?, + ); + if let Some(value) = options.quality.clone() { + form = form.text("quality", value); + } + if let Some(value) = options.background.clone() { + form = form.text("background", value); + } + if let Some(value) = options.output_format.clone() { + form = form.text("output_format", value); + } + if let Some(mask) = options.mask.clone() { + form = form.part( + "mask", + Part::bytes(mask.bytes) + .file_name(mask.file_name) + .mime_str(mask.mime_type.as_str()) + .map_err(|error| invalid_request(failure_context, error.to_string()))?, + ); + } + + let client = reqwest::Client::builder() + .timeout(Duration::from_millis(settings.request_timeout_ms.max(1))) + .http1_only() + .build() + .map_err(|error| invalid_config(error.to_string()))?; + let response = client + .post(url.as_str()) + .bearer_auth(settings.api_key.as_str()) + .multipart(form) + .send() + .await + .map_err(|error| request_error(&url, failure_context, error))?; + let status = response.status(); + let body = response + .text() + .await + .map_err(|error| request_error(&url, failure_context, error))?; + if !status.is_success() { + return Err(PlatformImageError::Upstream { + provider: VECTOR_ENGINE_PROVIDER, + message: format!( + "{failure_context}：上游图片编辑失败（HTTP {}）", + status.as_u16() + ), + upstream_status: status.as_u16(), + raw_excerpt: body.chars().take(2_000).collect(), + audit: None, + }); + } + let payload: Value = + serde_json::from_str(body.as_str()).map_err(|error| PlatformImageError::ResponseParse { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{failure_context}：上游响应不是 JSON：{error}"), + raw_excerpt: body.chars().take(2_000).collect(), + audit: None, + })?; + let mut images = Vec::new(); + if let Some(entries) = payload.get("data").and_then(Value::as_array) { + for entry in entries { + let Some(value) = entry.get("b64_json").and_then(Value::as_str) else { + continue; + }; + let bytes = BASE64_STANDARD.decode(value).map_err(|error| { + PlatformImageError::ResponseParse { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{failure_context}：上游 b64_json 解码失败：{error}"), + raw_excerpt: body.chars().take(2_000).collect(), + audit: None, + } + })?; + let (mime_type, extension) = match options.output_format.as_deref() { + Some("jpeg") => ("image/jpeg", "jpg"), + Some("webp") => ("image/webp", "webp"), + _ => ("image/png", "png"), + }; + images.push(DownloadedImage { + bytes, + mime_type: mime_type.to_string(), + extension: extension.to_string(), + }); + } + } + if images.is_empty() { + return Err(PlatformImageError::MissingImage { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{failure_context}：上游未返回 b64_json 图片"), + audit: None, + }); + } + Ok(GeneratedImages { + task_id: payload + .get("id") + .and_then(Value::as_str) + .unwrap_or("raw-image-edit") + .to_string(), + actual_prompt: None, + images, + recovered_failure_audits: Vec::new(), + }) +} + +fn invalid_request(context: &str, message: String) -> PlatformImageError { + PlatformImageError::InvalidRequest { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{context}：构造上游请求失败：{message}"), + } +} + +fn invalid_config(message: String) -> PlatformImageError { + PlatformImageError::InvalidConfig { + provider: VECTOR_ENGINE_PROVIDER, + message, + } +} + +fn request_error(url: &str, context: &str, error: E) -> PlatformImageError { + PlatformImageError::Request { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{context}：上游请求失败：{error}"), + endpoint: Some(url.to_string()), + timeout: false, + connect: false, + request: true, + body: false, + status_code: None, + source: Some(error.to_string()), + audit: None, + } } diff --git a/server-rs/crates/platform-image/src/vector_engine/types.rs b/server-rs/crates/platform-image/src/vector_engine/types.rs index 818cafe60..77fbd19f9 100644 --- a/server-rs/crates/platform-image/src/vector_engine/types.rs +++ b/server-rs/crates/platform-image/src/vector_engine/types.rs @@ -29,15 +29,3 @@ pub struct ReferenceImage { pub mime_type: String, pub file_name: String, } - -/// Raw GPT Image 2 edit options. The API layer owns validation; this type only -/// carries values that must be forwarded to VectorEngine. -#[derive(Clone, Debug, Default)] -pub struct RawImageEditOptions { - pub quality: Option, - pub background: Option, - pub output_format: Option, - pub width: u32, - pub height: u32, - pub mask: Option, -} From 35d0db6377a16b632354dd31ac88c6b1e6d11c99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 16:19:29 +0800 Subject: [PATCH 010/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=A4=B1=E8=B4=A5=E5=AE=A1=E8=AE=A1=E4=B8=8E=E5=93=8D?= =?UTF-8?q?=E5=BA=94=E6=A0=BC=E5=BC=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 Raw provider 失败分支补齐结构化 failure audit 并接入 api-server 记录链 使用 VectorEngine 响应 output_format 生成返回图片 MIME 与扩展名 补充响应格式和审计字段定向测试并同步技术方案 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- server-rs/crates/api-server/src/raw_image.rs | 16 +- .../src/vector_engine/raw_edit.rs | 324 ++++++++++++++++-- 3 files changed, 300 insertions(+), 42 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 759fe959d..397defd40 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -82,7 +82,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、JSON DTO、base64 解码、预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 结果统一解码为图片字节；raw handler 只将这些字节编码到 `data[].b64_json`。 +provider 结果统一解码为图片字节；每项结果的 MIME 与扩展名以 VectorEngine 响应中的真实 `output_format` 为准，不得从请求参数反推。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链。raw handler 只将结果字节编码到 `data[].b64_json`。 ## 代码拆分 diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index c083a7fd7..ef4a22b33 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -14,7 +14,10 @@ use crate::{ }, auth::AuthenticatedAccessToken, http_error::AppError, - openai_image_generation::{map_platform_image_error, require_openai_image_settings}, + openai_image_generation::{ + map_platform_image_error, record_openai_image_failure_if_configured, + require_openai_image_settings, + }, request_context::RequestContext, state::AppState, }; @@ -65,8 +68,9 @@ pub(crate) async fn edit_raw_image( let user_id = authenticated.claims().user_id().to_string(); let request_id = request_context.request_id().to_string(); let points_cost = raw_image_edit_price(&state, prepared.width, prepared.height).await?; + let audit_settings = settings.clone(); let operation = async move { - let generated = create_vector_engine_raw_image_edit( + let generated = match create_vector_engine_raw_image_edit( &provider_settings, prepared.prompt.as_str(), &prepared.image, @@ -74,7 +78,13 @@ pub(crate) async fn edit_raw_image( "raw_image_edit", ) .await - .map_err(map_platform_image_error)?; + { + Ok(generated) => generated, + Err(error) => { + record_openai_image_failure_if_configured(&audit_settings, &error).await; + return Err(map_platform_image_error(error)); + } + }; let data = generated .images .into_iter() diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 435a9e1f3..a6c53ff0d 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -1,14 +1,16 @@ -use std::time::Duration; +use std::time::{Duration, Instant}; use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use reqwest::multipart::{Form, Part}; use serde_json::Value; use super::{ + audit::build_failure_audit, constants::{GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER}, error::PlatformImageError, request::vector_engine_images_edit_url, types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}, + util::truncate_raw, }; #[derive(Clone, Debug, Default)] @@ -30,6 +32,9 @@ pub async fn create_vector_engine_raw_image_edit( failure_context: &str, ) -> Result { let url = vector_engine_images_edit_url(settings); + let started_at = Instant::now(); + let prompt_chars = Some(prompt.chars().count()); + let reference_image_count = Some(1_usize + usize::from(options.mask.is_some())); let mut form = Form::new() .text("model", GPT_IMAGE_2_MODEL.to_string()) .text("n", "1".to_string()) @@ -72,50 +77,134 @@ pub async fn create_vector_engine_raw_image_edit( .multipart(form) .send() .await - .map_err(|error| request_error(&url, failure_context, error))?; + .map_err(|error| { + request_error( + &url, + failure_context, + "request_send", + error, + started_at, + prompt_chars, + reference_image_count, + ) + })?; let status = response.status(); - let body = response - .text() - .await - .map_err(|error| request_error(&url, failure_context, error))?; + let body = response.text().await.map_err(|error| { + request_error( + &url, + failure_context, + "response_read", + error, + started_at, + prompt_chars, + reference_image_count, + ) + })?; if !status.is_success() { + let message = format!( + "{failure_context}：上游图片编辑失败（HTTP {}）", + status.as_u16() + ); + let raw_excerpt = truncate_raw(body.as_str()); + let audit = build_failure_audit( + url.as_str(), + failure_context, + "upstream_status", + Some(status.as_u16()), + Some(status_class(status.as_u16())), + false, + false, + message.as_str(), + None, + Some(raw_excerpt.clone()), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + Some(GPT_IMAGE_2_MODEL), + ); return Err(PlatformImageError::Upstream { provider: VECTOR_ENGINE_PROVIDER, - message: format!( - "{failure_context}：上游图片编辑失败（HTTP {}）", - status.as_u16() - ), + message, upstream_status: status.as_u16(), - raw_excerpt: body.chars().take(2_000).collect(), - audit: None, + raw_excerpt, + audit: Some(audit), }); } - let payload: Value = - serde_json::from_str(body.as_str()).map_err(|error| PlatformImageError::ResponseParse { - provider: VECTOR_ENGINE_PROVIDER, - message: format!("{failure_context}：上游响应不是 JSON：{error}"), - raw_excerpt: body.chars().take(2_000).collect(), - audit: None, - })?; + let payload: Value = match serde_json::from_str(body.as_str()) { + Ok(payload) => payload, + Err(error) => { + let message = format!("{failure_context}：上游响应不是 JSON：{error}"); + let audit = build_failure_audit( + url.as_str(), + failure_context, + "response_parse", + Some(status.as_u16()), + Some(status_class(status.as_u16())), + false, + false, + message.as_str(), + Some(error.to_string()), + Some(truncate_raw(body.as_str())), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + Some(GPT_IMAGE_2_MODEL), + ); + return Err(PlatformImageError::ResponseParse { + provider: VECTOR_ENGINE_PROVIDER, + message, + raw_excerpt: truncate_raw(body.as_str()), + audit: Some(audit), + }); + } + }; let mut images = Vec::new(); if let Some(entries) = payload.get("data").and_then(Value::as_array) { for entry in entries { let Some(value) = entry.get("b64_json").and_then(Value::as_str) else { continue; }; - let bytes = BASE64_STANDARD.decode(value).map_err(|error| { - PlatformImageError::ResponseParse { - provider: VECTOR_ENGINE_PROVIDER, - message: format!("{failure_context}：上游 b64_json 解码失败：{error}"), - raw_excerpt: body.chars().take(2_000).collect(), - audit: None, + let bytes = match BASE64_STANDARD.decode(value) { + Ok(bytes) => bytes, + Err(error) => { + let message = format!("{failure_context}：上游 b64_json 解码失败：{error}"); + let audit = build_failure_audit( + url.as_str(), + failure_context, + "response_parse", + Some(status.as_u16()), + Some(status_class(status.as_u16())), + false, + false, + message.as_str(), + Some(error.to_string()), + Some(truncate_raw(body.as_str())), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + Some(GPT_IMAGE_2_MODEL), + ); + return Err(PlatformImageError::ResponseParse { + provider: VECTOR_ENGINE_PROVIDER, + message, + raw_excerpt: truncate_raw(body.as_str()), + audit: Some(audit), + }); } - })?; - let (mime_type, extension) = match options.output_format.as_deref() { - Some("jpeg") => ("image/jpeg", "jpg"), - Some("webp") => ("image/webp", "webp"), - _ => ("image/png", "png"), }; + let (mime_type, extension) = + response_image_format(&payload, entry).map_err(|message| { + response_parse_error( + &url, + failure_context, + message, + status.as_u16(), + started_at, + prompt_chars, + reference_image_count, + &body, + ) + })?; images.push(DownloadedImage { bytes, mime_type: mime_type.to_string(), @@ -124,10 +213,27 @@ pub async fn create_vector_engine_raw_image_edit( } } if images.is_empty() { + let message = format!("{failure_context}：上游未返回 b64_json 图片"); + let audit = build_failure_audit( + url.as_str(), + failure_context, + "missing_image", + Some(status.as_u16()), + Some(status_class(status.as_u16())), + false, + false, + message.as_str(), + None, + Some(truncate_raw(body.as_str())), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + Some(GPT_IMAGE_2_MODEL), + ); return Err(PlatformImageError::MissingImage { provider: VECTOR_ENGINE_PROVIDER, - message: format!("{failure_context}：上游未返回 b64_json 图片"), - audit: None, + message, + audit: Some(audit), }); } Ok(GeneratedImages { @@ -156,17 +262,159 @@ fn invalid_config(message: String) -> PlatformImageError { } } -fn request_error(url: &str, context: &str, error: E) -> PlatformImageError { +fn response_parse_error( + url: &str, + context: &str, + message: &str, + status: u16, + started_at: Instant, + prompt_chars: Option, + reference_image_count: Option, + body: &str, +) -> PlatformImageError { + let audit = build_failure_audit( + url, + context, + "response_parse", + Some(status), + Some(status_class(status)), + false, + false, + message, + None, + Some(truncate_raw(body)), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + Some(GPT_IMAGE_2_MODEL), + ); + PlatformImageError::ResponseParse { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{context}：{message}"), + raw_excerpt: truncate_raw(body), + audit: Some(audit), + } +} + +fn response_image_format( + payload: &Value, + entry: &Value, +) -> Result<(&'static str, &'static str), &'static str> { + let Some(value) = entry + .get("output_format") + .or_else(|| payload.get("output_format")) + .and_then(Value::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + else { + return Err("上游响应缺少 output_format"); + }; + match value.to_ascii_lowercase().as_str() { + "png" => Ok(("image/png", "png")), + "jpeg" | "jpg" => Ok(("image/jpeg", "jpg")), + "webp" => Ok(("image/webp", "webp")), + "gif" => Ok(("image/gif", "gif")), + _ => Err("上游响应包含不支持的 output_format"), + } +} + +fn request_error( + url: &str, + context: &str, + failure_stage: &'static str, + error: reqwest::Error, + started_at: Instant, + prompt_chars: Option, + reference_image_count: Option, +) -> PlatformImageError { + let timeout = error.is_timeout(); + let connect = error.is_connect(); + let source = error.to_string(); + let message = format!("{context}：上游请求失败：{source}"); + let audit = build_failure_audit( + url, + context, + failure_stage, + None, + Some("transport"), + timeout, + connect, + message.as_str(), + Some(source.clone()), + None, + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + Some(GPT_IMAGE_2_MODEL), + ); PlatformImageError::Request { provider: VECTOR_ENGINE_PROVIDER, - message: format!("{context}：上游请求失败：{error}"), + message, endpoint: Some(url.to_string()), - timeout: false, - connect: false, + timeout, + connect, request: true, body: false, status_code: None, - source: Some(error.to_string()), - audit: None, + source: Some(source), + audit: Some(audit), + } +} + +fn status_class(status: u16) -> &'static str { + match status { + 100..=199 => "1xx", + 200..=299 => "2xx", + 300..=399 => "3xx", + 400..=499 => "4xx", + _ => "5xx", + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn response_format_uses_vector_engine_output_format() { + let payload = json!({"output_format": "png"}); + assert_eq!( + response_image_format(&payload, &json!({"output_format": "webp"})), + Ok(("image/webp", "webp")) + ); + assert_eq!( + response_image_format(&payload, &json!({})), + Ok(("image/png", "png")) + ); + assert_eq!( + response_image_format(&json!({}), &json!({"output_format": "jpeg"})), + Ok(("image/jpeg", "jpg")) + ); + assert!(response_image_format(&json!({}), &json!({})).is_err()); + assert!( + response_image_format(&json!({}), &json!({"output_format": "bmp"})).is_err() + ); + } + + #[test] + fn response_parse_error_contains_structured_audit() { + let error = response_parse_error( + "https://vector.example/v1/images/edits", + "raw_image_edit", + "上游响应缺少 output_format", + 200, + Instant::now(), + Some(12), + Some(2), + "{\"data\":[]}", + ); + let audit = error.audit().expect("response error should carry audit"); + assert_eq!(audit.failure_stage, "response_parse"); + assert_eq!(audit.status_code, Some(200)); + assert_eq!(audit.status_class, Some("2xx")); + assert_eq!(audit.prompt_chars, Some(12)); + assert_eq!(audit.reference_image_count, Some(2)); + assert_eq!(audit.image_model, Some(GPT_IMAGE_2_MODEL)); } } From b22eae272612204c460d1ad0e2751c9b55c8d54e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 18:18:58 +0800 Subject: [PATCH 011/248] =?UTF-8?q?=E6=94=B6=E7=B4=A7=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E8=BE=93=E5=85=A5=E5=A5=91=E7=BA=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增 3840 边长、16 像素对齐、3:1 比例和总像素范围校验 要求 image 与 mask 为可完整解码的 PNG 并在扣费前拒绝非法输入 将 Raw 路由 JSON body limit 提升至 64 MiB 同步 Raw 图片编辑技术方案和边界测试 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 17 +++- .../crates/api-server/src/modules/raw.rs | 7 +- server-rs/crates/api-server/src/raw_image.rs | 77 +++++++++++++-- server-rs/crates/platform-image/src/lib.rs | 13 +-- .../platform-image/src/vector_engine/mod.rs | 6 +- .../src/vector_engine/raw_edit.rs | 96 ++++++++++++++++++- 6 files changed, 191 insertions(+), 25 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 397defd40..0783cbc83 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -1,6 +1,6 @@ # Raw GPT Image 2 图片编辑代理 -更新时间：`2026-09-07` +更新时间：`2026-09-08` ## 目标 @@ -35,7 +35,18 @@ POST /api/raw/v1/images/edit } ``` -`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。输入格式由 MIME 类型和解码后的图片字节共同确定，服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。`width`、`height` 为整数，组成发送给 provider 的输出尺寸；不把尺寸改写成业务字符串字段。 +`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件；空数据、非 PNG 字节或 MIME 不匹配均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 + +`width`、`height` 使用严格输出尺寸规则，均在扣费前校验： + +1. 单边最大值为 `3840px`； +2. 宽、高均为 `16px` 的倍数； +3. 长边 / 短边不超过 `3:1`； +4. 总像素范围为 `655360` 至 `8294400`（含边界）。 + +校验通过后按整数尺寸发送给 provider，不静默 clamp 或改写调用者尺寸。 + +Raw 路由的 JSON body limit 为 `64 MiB`，为 base64 编码膨胀和可选 mask 留出空间；同时必须在 base64 解码后拒绝空 PNG，并保留图片格式校验，避免仅依赖 HTTP body limit。 服务端发送给 `platform-image` 时固定注入： @@ -87,7 +98,7 @@ provider 结果统一解码为图片字节；每项结果的 MIME 与扩展名 ## 代码拆分 - `server-rs/crates/api-server/src/raw_image.rs`：独立路由 handler、请求/响应 DTO、base64 输入校验、预检查和 raw billing 编排。 -- `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、provider 请求映射和原始响应解码；现有编辑器调用通过默认选项复用，不在业务 handler 复制 provider 协议。 +- `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、严格尺寸校验、独立 provider 请求映射和原始响应解码；不复用现有 editor 图片编辑 client 或其 multipart transport。 - `server-rs/crates/api-server/src/modules/raw.rs`：只注册 `/api/raw/v1/images/edit` 并挂载 Bearer middleware。 不修改 External v1 OpenAPI；不在 `external_editor_api.rs`、编辑器项目模块或外部生成 worker 中增加 raw 分支。 diff --git a/server-rs/crates/api-server/src/modules/raw.rs b/server-rs/crates/api-server/src/modules/raw.rs index 905c16584..8307da70c 100644 --- a/server-rs/crates/api-server/src/modules/raw.rs +++ b/server-rs/crates/api-server/src/modules/raw.rs @@ -1,11 +1,14 @@ -use axum::{Router, middleware, routing::post}; +use axum::{Router, extract::DefaultBodyLimit, middleware, routing::post}; use crate::{auth::require_bearer_auth, raw_image::edit_raw_image, state::AppState}; +const RAW_IMAGE_EDIT_BODY_LIMIT_BYTES: usize = 64 * 1024 * 1024; + pub fn router(state: AppState) -> Router { Router::new().route( "/api/raw/v1/images/edit", post(edit_raw_image) - .route_layer(middleware::from_fn_with_state(state, require_bearer_auth)), + .route_layer(middleware::from_fn_with_state(state, require_bearer_auth)) + .layer(DefaultBodyLimit::max(RAW_IMAGE_EDIT_BODY_LIMIT_BYTES)), ) } diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index ef4a22b33..455ed49b8 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -4,9 +4,14 @@ use axum::{ http::StatusCode, }; use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; -use platform_image::{RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit}; +use image::{ImageFormat, ImageReader}; +use platform_image::{ + RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, + validate_raw_image_edit_dimensions, +}; use serde::{Deserialize, Serialize}; use serde_json::{Value, json}; +use std::io::Cursor; use crate::{ asset_billing::{ @@ -122,9 +127,8 @@ fn prepare_request(payload: RawImageEditRequest) -> Result( fn decode_image(value: RawImageData, field: &str) -> Result { let mime_type = value.mime_type.trim().to_string(); - if mime_type.is_empty() { - return Err(bad_request(format!("{field}.mimeType 不能为空"))); + if !mime_type.eq_ignore_ascii_case("image/png") { + return Err(bad_request(format!("{field}.mimeType 必须为 image/png"))); } let bytes = BASE64_STANDARD .decode(value.data.trim()) .map_err(|_| bad_request(format!("{field}.data 必须是有效 base64")))?; + if bytes.is_empty() { + return Err(bad_request(format!("{field}.data 不能为空"))); + } + let reader = ImageReader::new(Cursor::new(bytes.as_slice())) + .with_guessed_format() + .map_err(|_| bad_request(format!("{field}.data 必须是有效 PNG 文件")))?; + if reader.format() != Some(ImageFormat::Png) { + return Err(bad_request(format!("{field}.data 必须是有效 PNG 文件"))); + } + reader + .decode() + .map_err(|_| bad_request(format!("{field}.data 必须是有效 PNG 文件")))?; Ok(ReferenceImage { bytes, file_name: format!("{field}.png"), - mime_type, + mime_type: "image/png".to_string(), }) } @@ -225,18 +241,29 @@ fn bad_request(message: impl Into) -> AppError { #[cfg(test)] mod tests { use super::*; + use image::{ImageFormat, Rgba, RgbaImage}; + use std::io::Cursor; + + fn encoded_png(width: u32, height: u32) -> String { + let image = RgbaImage::from_pixel(width, height, Rgba([255, 0, 0, 255])); + let mut bytes = Vec::new(); + image + .write_to(&mut Cursor::new(&mut bytes), ImageFormat::Png) + .expect("test PNG should encode"); + BASE64_STANDARD.encode(bytes) + } #[test] fn request_uses_one_image_object_and_rejects_images_array() { let payload = serde_json::json!({ - "image": {"data": "aGVsbG8=", "mimeType": "image/png"}, + "image": {"data": encoded_png(1, 1), "mimeType": "image/png"}, "prompt": "edit", "width": 1024, "height": 1024 }); let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("image object"); let prepared = prepare_request(parsed).expect("request should prepare"); - assert_eq!(prepared.image.bytes, b"hello"); + assert!(prepared.image.bytes.starts_with(b"\x89PNG\r\n\x1a\n")); let array_payload = serde_json::json!({ "images": [{"data": "aGVsbG8=", "mimeType": "image/png"}], @@ -278,4 +305,36 @@ mod tests { assert!(rendered.contains("image.data 必须是有效 base64")); assert!(!rendered.contains("InvalidByte")); } + + #[test] + fn dimensions_follow_strict_raw_image_contract() { + assert!(validate_raw_image_edit_dimensions(1024, 1024).is_ok()); + assert!(validate_raw_image_edit_dimensions(3840, 1280).is_ok()); + assert!(validate_raw_image_edit_dimensions(3839, 1280).is_err()); + assert!(validate_raw_image_edit_dimensions(3840, 1264).is_err()); + assert!(validate_raw_image_edit_dimensions(1024, 1000).is_err()); + assert!(validate_raw_image_edit_dimensions(16, 16).is_err()); + assert!(validate_raw_image_edit_dimensions(3840, 3840).is_err()); + } + + #[test] + fn input_requires_decodable_png_and_png_mime() { + let valid = serde_json::json!({ + "image": {"data": encoded_png(1, 1), "mimeType": "IMAGE/PNG"}, + "prompt": "edit", + "width": 1024, + "height": 1024 + }); + assert!(prepare_request(serde_json::from_value(valid).expect("valid request")).is_ok()); + + for (data, mime_type) in [("aGVsbG8=", "image/png"), ("aGVsbG8=", "image/jpeg")] { + let payload = serde_json::json!({ + "image": {"data": data, "mimeType": mime_type}, + "prompt": "edit", + "width": 1024, + "height": 1024 + }); + assert!(prepare_request(serde_json::from_value(payload).expect("request")).is_err()); + } + } } diff --git a/server-rs/crates/platform-image/src/lib.rs b/server-rs/crates/platform-image/src/lib.rs index 1e6e757bd..7140aec48 100644 --- a/server-rs/crates/platform-image/src/lib.rs +++ b/server-rs/crates/platform-image/src/lib.rs @@ -10,15 +10,16 @@ pub use pixel_art_snapper::{ }; pub use vector_engine::{ DownloadedImage, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, GeneratedImages, NANOBANANA_2_MODEL, - PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, RawImageEditOptions, - ReferenceImage, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, - VectorEngineImageSettings, build_vector_engine_image_http_client, - build_vector_engine_image_request_body, + PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, + RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, + RawImageEditDimensionError, RawImageEditOptions, ReferenceImage, + VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, VectorEngineImageSettings, + build_vector_engine_image_http_client, build_vector_engine_image_request_body, build_vector_engine_nanobanana_generate_content_request_body, create_vector_engine_image_edit, create_vector_engine_image_edit_with_references, create_vector_engine_image_edit_with_references_and_model, create_vector_engine_image_generation, create_vector_engine_image_generation_with_model, create_vector_engine_nanobanana_generate_content, create_vector_engine_raw_image_edit, - download_remote_image, vector_engine_images_edit_url, vector_engine_images_generation_url, - vector_engine_nanobanana_generate_content_url, + download_remote_image, validate_raw_image_edit_dimensions, vector_engine_images_edit_url, + vector_engine_images_generation_url, vector_engine_nanobanana_generate_content_url, }; diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index aacb3256d..b94d22221 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -26,7 +26,11 @@ pub use constants::{ }; pub use error::{PlatformImageError, PlatformImageStatusHint}; pub use image_source::download_remote_image; -pub use raw_edit::{RawImageEditOptions, create_vector_engine_raw_image_edit}; +pub use raw_edit::{ + RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, + RawImageEditDimensionError, RawImageEditOptions, create_vector_engine_raw_image_edit, + validate_raw_image_edit_dimensions, +}; pub use request::{ build_vector_engine_image_request_body, build_vector_engine_image_request_body_with_model, build_vector_engine_nanobanana_generate_content_request_body, normalize_image_size_for_model, diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index a6c53ff0d..c46d46037 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -13,7 +13,7 @@ use super::{ util::truncate_raw, }; -#[derive(Clone, Debug, Default)] +#[derive(Clone, Debug)] pub struct RawImageEditOptions { pub quality: Option, pub background: Option, @@ -23,6 +23,68 @@ pub struct RawImageEditOptions { pub mask: Option, } +pub const RAW_IMAGE_MAX_EDGE: u32 = 3_840; +pub const RAW_IMAGE_DIMENSION_ALIGNMENT: u32 = 16; +pub const RAW_IMAGE_MIN_PIXELS: u64 = 655_360; +pub const RAW_IMAGE_MAX_PIXELS: u64 = 8_294_400; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum RawImageEditDimensionError { + Zero, + MaxEdge, + Alignment, + AspectRatio, + PixelCount, +} + +impl std::fmt::Display for RawImageEditDimensionError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Zero => formatter.write_str("width 和 height 必须为正整数"), + Self::MaxEdge => write!( + formatter, + "width 和 height 的单边最大值为 {RAW_IMAGE_MAX_EDGE}px" + ), + Self::Alignment => write!( + formatter, + "width 和 height 必须是 {RAW_IMAGE_DIMENSION_ALIGNMENT}px 的倍数" + ), + Self::AspectRatio => formatter.write_str("长边与短边的比例不能超过 3:1"), + Self::PixelCount => write!( + formatter, + "总像素必须在 {RAW_IMAGE_MIN_PIXELS} 至 {RAW_IMAGE_MAX_PIXELS} 之间" + ), + } + } +} + +pub fn validate_raw_image_edit_dimensions( + width: u32, + height: u32, +) -> Result<(), RawImageEditDimensionError> { + if width == 0 || height == 0 { + return Err(RawImageEditDimensionError::Zero); + } + if width > RAW_IMAGE_MAX_EDGE || height > RAW_IMAGE_MAX_EDGE { + return Err(RawImageEditDimensionError::MaxEdge); + } + if !width.is_multiple_of(RAW_IMAGE_DIMENSION_ALIGNMENT) + || !height.is_multiple_of(RAW_IMAGE_DIMENSION_ALIGNMENT) + { + return Err(RawImageEditDimensionError::Alignment); + } + let long_edge = u64::from(width.max(height)); + let short_edge = u64::from(width.min(height)); + if long_edge > short_edge.saturating_mul(3) { + return Err(RawImageEditDimensionError::AspectRatio); + } + let pixels = u64::from(width) * u64::from(height); + if !(RAW_IMAGE_MIN_PIXELS..=RAW_IMAGE_MAX_PIXELS).contains(&pixels) { + return Err(RawImageEditDimensionError::PixelCount); + } + Ok(()) +} + /// Independent raw GPT Image 2 proxy; it does not call the editor image-edit client. pub async fn create_vector_engine_raw_image_edit( settings: &VectorEngineImageSettings, @@ -31,6 +93,8 @@ pub async fn create_vector_engine_raw_image_edit( options: RawImageEditOptions, failure_context: &str, ) -> Result { + validate_raw_image_edit_dimensions(options.width, options.height) + .map_err(|error| invalid_request(failure_context, error.to_string()))?; let url = vector_engine_images_edit_url(settings); let started_at = Instant::now(); let prompt_chars = Some(prompt.chars().count()); @@ -392,9 +456,7 @@ mod tests { Ok(("image/jpeg", "jpg")) ); assert!(response_image_format(&json!({}), &json!({})).is_err()); - assert!( - response_image_format(&json!({}), &json!({"output_format": "bmp"})).is_err() - ); + assert!(response_image_format(&json!({}), &json!({"output_format": "bmp"})).is_err()); } #[test] @@ -417,4 +479,30 @@ mod tests { assert_eq!(audit.reference_image_count, Some(2)); assert_eq!(audit.image_model, Some(GPT_IMAGE_2_MODEL)); } + + #[test] + fn raw_image_edit_dimensions_enforce_strict_contract() { + assert!(validate_raw_image_edit_dimensions(1024, 640).is_ok()); + assert!(validate_raw_image_edit_dimensions(3840, 2160).is_ok()); + assert_eq!( + validate_raw_image_edit_dimensions(3856, 2160), + Err(RawImageEditDimensionError::MaxEdge) + ); + assert_eq!( + validate_raw_image_edit_dimensions(1024, 1000), + Err(RawImageEditDimensionError::Alignment) + ); + assert_eq!( + validate_raw_image_edit_dimensions(1936, 640), + Err(RawImageEditDimensionError::AspectRatio) + ); + assert_eq!( + validate_raw_image_edit_dimensions(1024, 624), + Err(RawImageEditDimensionError::PixelCount) + ); + assert_eq!( + validate_raw_image_edit_dimensions(3840, 2176), + Err(RawImageEditDimensionError::PixelCount) + ); + } } From a86a2ee4d2d69a164cb21aa4e7235a9734785d84 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 19:32:55 +0800 Subject: [PATCH 012/248] =?UTF-8?q?=E7=AE=80=E5=8C=96=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E5=93=8D=E5=BA=94=E7=B1=BB=E5=9E=8B?= =?UTF-8?q?=20=E7=A7=BB=E9=99=A4=E6=97=A0=E5=BF=85=E8=A6=81=E7=9A=84=20JSO?= =?UTF-8?q?N=20Value=20=E8=BD=AC=E6=8D=A2=20=E4=BF=9D=E6=8C=81=E5=93=8D?= =?UTF-8?q?=E5=BA=94=E7=BB=93=E6=9E=84=E7=94=B1=E7=B1=BB=E5=9E=8B=E7=B3=BB?= =?UTF-8?q?=E7=BB=9F=E6=A0=A1=E9=AA=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/api-server/src/raw_image.rs | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 455ed49b8..879179b57 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -10,7 +10,7 @@ use platform_image::{ validate_raw_image_edit_dimensions, }; use serde::{Deserialize, Serialize}; -use serde_json::{Value, json}; +use serde_json::json; use std::io::Cursor; use crate::{ @@ -62,7 +62,7 @@ pub(crate) async fn edit_raw_image( Extension(request_context): Extension, Extension(authenticated): Extension, Json(payload): Json, -) -> Result, AppError> { +) -> Result, AppError> { let prepared = prepare_request(payload)?; let settings = require_openai_image_settings(&state)?.with_external_api_audit_context( &request_context, @@ -110,9 +110,7 @@ pub(crate) async fn edit_raw_image( ), ) .await?; - Ok(Json(serde_json::to_value(result).map_err(|error| { - AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_message(error.to_string()) - })?)) + Ok(Json(result)) } struct PreparedRawImageEdit { From 22785b4d7b589979fcd44cb9bdac177b5f478169 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 19:45:02 +0800 Subject: [PATCH 013/248] =?UTF-8?q?=E9=99=90=E5=88=B6=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E8=A7=A3=E7=A0=81=E8=B5=84=E6=BA=90=20=E4=B8=BA=20PNG?= =?UTF-8?q?=20=E8=A7=A3=E7=A0=81=E8=AE=BE=E7=BD=AE=E8=BE=B9=E9=95=BF?= =?UTF-8?q?=E5=92=8C=E5=88=86=E9=85=8D=E4=B8=8A=E9=99=90=20=E9=81=BF?= =?UTF-8?q?=E5=85=8D=E5=8E=8B=E7=BC=A9=E5=9B=BE=E7=89=87=E8=86=A8=E8=83=80?= =?UTF-8?q?=E5=AF=BC=E8=87=B4=E5=86=85=E5=AD=98=E8=80=97=E5=B0=BD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/api-server/src/raw_image.rs | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 879179b57..250dd071d 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -6,8 +6,8 @@ use axum::{ use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use image::{ImageFormat, ImageReader}; use platform_image::{ - RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, - validate_raw_image_edit_dimensions, + RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RawImageEditOptions, ReferenceImage, + create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, }; use serde::{Deserialize, Serialize}; use serde_json::json; @@ -197,9 +197,14 @@ fn decode_image(value: RawImageData, field: &str) -> Result Date: Tue, 8 Sep 2026 19:48:58 +0800 Subject: [PATCH 014/248] =?UTF-8?q?=E7=A7=BB=E5=87=BA=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=90=8C=E6=AD=A5=E8=A7=A3=E7=A0=81=20=E5=B0=86?= =?UTF-8?q?=E8=AF=B7=E6=B1=82=E9=A2=84=E6=A0=A1=E9=AA=8C=E6=94=BE=E5=85=A5?= =?UTF-8?q?=E9=98=BB=E5=A1=9E=E7=BA=BF=E7=A8=8B=20=E9=81=BF=E5=85=8D?= =?UTF-8?q?=E5=8D=A0=E7=94=A8=20Tokio=20=E5=BC=82=E6=AD=A5=E5=B7=A5?= =?UTF-8?q?=E4=BD=9C=E7=BA=BF=E7=A8=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/api-server/src/raw_image.rs | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 250dd071d..a41bcfd73 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -63,7 +63,12 @@ pub(crate) async fn edit_raw_image( Extension(authenticated): Extension, Json(payload): Json, ) -> Result, AppError> { - let prepared = prepare_request(payload)?; + let prepared = tokio::task::spawn_blocking(move || prepare_request(payload)) + .await + .map_err(|error| { + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR) + .with_message(error.to_string()) + })??; let settings = require_openai_image_settings(&state)?.with_external_api_audit_context( &request_context, Some(authenticated.claims().user_id().to_string()), From cea8bf77fa465ebc10a190004b587824af0e6c1d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 19:52:13 +0800 Subject: [PATCH 015/248] =?UTF-8?q?=E5=87=8F=E5=B0=91=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E8=AF=B7=E6=B1=82=E6=8B=B7=E8=B4=9D?= =?UTF-8?q?=20=E7=A7=BB=E5=8A=A8=20multipart=20=E9=80=89=E9=A1=B9=E5=AD=97?= =?UTF-8?q?=E6=AE=B5=E8=80=8C=E9=9D=9E=E5=85=8B=E9=9A=86=20=E9=99=8D?= =?UTF-8?q?=E4=BD=8E=E6=8E=A9=E7=A0=81=E5=92=8C=E5=AD=97=E7=AC=A6=E4=B8=B2?= =?UTF-8?q?=E7=9A=84=E5=B3=B0=E5=80=BC=E5=86=85=E5=AD=98?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../crates/platform-image/src/vector_engine/raw_edit.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index c46d46037..64ae2887c 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -111,16 +111,16 @@ pub async fn create_vector_engine_raw_image_edit( .mime_str(image.mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, ); - if let Some(value) = options.quality.clone() { + if let Some(value) = options.quality { form = form.text("quality", value); } - if let Some(value) = options.background.clone() { + if let Some(value) = options.background { form = form.text("background", value); } - if let Some(value) = options.output_format.clone() { + if let Some(value) = options.output_format { form = form.text("output_format", value); } - if let Some(mask) = options.mask.clone() { + if let Some(mask) = options.mask { form = form.part( "mask", Part::bytes(mask.bytes) From 35dcfea3efe259e0d5e000ee505783ecd5c2c5d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 19:56:41 +0800 Subject: [PATCH 016/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E8=BE=93=E5=87=BA=E6=A0=BC=E5=BC=8F=E5=9B=9E=E9=80=80?= =?UTF-8?q?=20=E5=BF=BD=E7=95=A5=E7=A9=BA=E5=80=BC=E5=90=8E=E5=86=8D?= =?UTF-8?q?=E4=BD=BF=E7=94=A8=E8=BD=BD=E8=8D=B7=E7=BA=A7=E6=A0=BC=E5=BC=8F?= =?UTF-8?q?=20=E8=A1=A5=E5=85=85=E7=A9=BA=E5=AD=97=E7=AC=A6=E4=B8=B2?= =?UTF-8?q?=E5=92=8C=20null=20=E5=9B=9E=E5=BD=92=E6=B5=8B=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/vector_engine/raw_edit.rs | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 64ae2887c..9c1205fb2 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -366,10 +366,16 @@ fn response_image_format( ) -> Result<(&'static str, &'static str), &'static str> { let Some(value) = entry .get("output_format") - .or_else(|| payload.get("output_format")) .and_then(Value::as_str) .map(str::trim) .filter(|value| !value.is_empty()) + .or_else(|| { + payload + .get("output_format") + .and_then(Value::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + }) else { return Err("上游响应缺少 output_format"); }; @@ -455,6 +461,20 @@ mod tests { response_image_format(&json!({}), &json!({"output_format": "jpeg"})), Ok(("image/jpeg", "jpg")) ); + assert_eq!( + response_image_format( + &json!({"output_format": "png"}), + &json!({"output_format": ""}) + ), + Ok(("image/png", "png")) + ); + assert_eq!( + response_image_format( + &json!({"output_format": "webp"}), + &json!({"output_format": null}) + ), + Ok(("image/webp", "webp")) + ); assert!(response_image_format(&json!({}), &json!({})).is_err()); assert!(response_image_format(&json!({}), &json!({"output_format": "bmp"})).is_err()); } From e7f66ce950d1551e71294b836d2a651266adebd6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 20:03:13 +0800 Subject: [PATCH 017/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E4=BC=A0=E8=BE=93=E5=A4=B1=E8=B4=A5=E9=98=B6=E6=AE=B5?= =?UTF-8?q?=E6=A0=87=E8=AE=B0=20=E5=8C=BA=E5=88=86=E8=AF=B7=E6=B1=82?= =?UTF-8?q?=E5=8F=91=E9=80=81=E5=92=8C=E5=93=8D=E5=BA=94=E8=AF=BB=E5=8F=96?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=20=E8=AE=A9=E9=94=99=E8=AF=AF=E8=AF=A6?= =?UTF-8?q?=E6=83=85=E5=87=86=E7=A1=AE=E5=8F=8D=E6=98=A0=E6=95=85=E9=9A=9C?= =?UTF-8?q?=E9=98=B6=E6=AE=B5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/platform-image/src/vector_engine/raw_edit.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 9c1205fb2..23ca9ac41 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -423,8 +423,8 @@ fn request_error( endpoint: Some(url.to_string()), timeout, connect, - request: true, - body: false, + request: failure_stage == "request_send", + body: failure_stage == "response_read", status_code: None, source: Some(source), audit: Some(audit), From 55d2a393d14595755df5cccc5693e224edd9cd47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 21:52:15 +0800 Subject: [PATCH 018/248] =?UTF-8?q?=E5=AE=8C=E5=96=84=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=B0=BA=E5=AF=B8=E9=94=99=E8=AF=AF=E7=B1=BB=E5=9E=8B?= =?UTF-8?q?=20=E4=B8=BA=E5=85=AC=E5=BC=80=E9=94=99=E8=AF=AF=E5=AE=9E?= =?UTF-8?q?=E7=8E=B0=20Error=20trait=20=E6=94=AF=E6=8C=81=E9=80=9A?= =?UTF-8?q?=E7=94=A8=E9=94=99=E8=AF=AF=E9=93=BE=E5=A4=84=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/platform-image/src/vector_engine/raw_edit.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 23ca9ac41..00ff1b9a0 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -58,6 +58,8 @@ impl std::fmt::Display for RawImageEditDimensionError { } } +impl std::error::Error for RawImageEditDimensionError {} + pub fn validate_raw_image_edit_dimensions( width: u32, height: u32, From ad7c05960382b89a6e21d2e76dd1bb7b03a7c66f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 21:54:22 +0800 Subject: [PATCH 019/248] =?UTF-8?q?=E9=81=B5=E5=AE=88=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E8=AF=B7=E6=B1=82=E6=88=AA=E6=AD=A2=E6=97=B6=E9=97=B4?= =?UTF-8?q?=20=E5=8F=91=E9=80=81=E5=89=8D=E8=A3=81=E5=89=AA=E6=9C=89?= =?UTF-8?q?=E6=95=88=E8=B6=85=E6=97=B6=E9=A2=84=E7=AE=97=20=E9=A2=84?= =?UTF-8?q?=E7=AE=97=E8=80=97=E5=B0=BD=E6=97=B6=E6=8F=90=E5=89=8D=E8=BF=94?= =?UTF-8?q?=E5=9B=9E=E7=BB=93=E6=9E=84=E5=8C=96=E9=94=99=E8=AF=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../platform-image/src/vector_engine/raw_edit.rs | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 00ff1b9a0..17f882636 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -6,6 +6,7 @@ use serde_json::Value; use super::{ audit::build_failure_audit, + budget::{effective_request_timeout_ms, request_budget_exhausted_error}, constants::{GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER}, error::PlatformImageError, request::vector_engine_images_edit_url, @@ -132,8 +133,20 @@ pub async fn create_vector_engine_raw_image_edit( ); } + let Some(request_timeout_ms) = + effective_request_timeout_ms(settings.request_timeout_ms, settings.request_deadline) + else { + return Err(request_budget_exhausted_error( + url.as_str(), + failure_context, + Some(GPT_IMAGE_2_MODEL), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + )); + }; let client = reqwest::Client::builder() - .timeout(Duration::from_millis(settings.request_timeout_ms.max(1))) + .timeout(Duration::from_millis(request_timeout_ms)) .http1_only() .build() .map_err(|error| invalid_config(error.to_string()))?; From 0e244c64ea8b37817de01f1346de90208abaf5a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 22:11:35 +0800 Subject: [PATCH 020/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=20GPT=20Image=202=20?= =?UTF-8?q?=E5=B0=BA=E5=AF=B8=E5=B8=B8=E9=87=8F=20=E9=9B=86=E4=B8=AD?= =?UTF-8?q?=E7=BB=B4=E6=8A=A4=E5=83=8F=E7=B4=A0=E8=BE=B9=E7=95=8C=E5=92=8C?= =?UTF-8?q?=E5=AF=B9=E9=BD=90=E8=A7=84=E5=88=99=20=E4=BE=9B=E7=94=9F?= =?UTF-8?q?=E6=88=90=E4=B8=8E=E7=BC=96=E8=BE=91=E8=B7=AF=E5=BE=84=E5=85=B1?= =?UTF-8?q?=E5=90=8C=E5=BC=95=E7=94=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../platform-image/src/vector_engine/constants.rs | 5 +++++ .../platform-image/src/vector_engine/raw_edit.rs | 13 ++++++++----- .../platform-image/src/vector_engine/request.rs | 13 ++++++++----- 3 files changed, 21 insertions(+), 10 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/constants.rs b/server-rs/crates/platform-image/src/vector_engine/constants.rs index 6480fba73..dfb70f0f5 100644 --- a/server-rs/crates/platform-image/src/vector_engine/constants.rs +++ b/server-rs/crates/platform-image/src/vector_engine/constants.rs @@ -5,3 +5,8 @@ pub const VECTOR_ENGINE_GPT_IMAGE_2_MODEL: &str = GPT_IMAGE_2_MODEL; pub const VECTOR_ENGINE_PROVIDER: &str = "vector-engine"; pub const VECTOR_ENGINE_IMAGE_EDIT_MAX_REFERENCE_IMAGES: usize = 5; pub const VECTOR_ENGINE_NANOBANANA_MAX_REFERENCE_IMAGES: usize = 14; + +pub(crate) const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; +pub(crate) const GPT_IMAGE_2_MAX_PIXELS: u64 = 8_294_400; +pub(crate) const GPT_IMAGE_2_MAX_EDGE: u32 = 3_840; +pub(crate) const GPT_IMAGE_2_DIMENSION_ALIGNMENT: u32 = 16; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 17f882636..c8f6ae249 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -7,7 +7,10 @@ use serde_json::Value; use super::{ audit::build_failure_audit, budget::{effective_request_timeout_ms, request_budget_exhausted_error}, - constants::{GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER}, + constants::{ + GPT_IMAGE_2_DIMENSION_ALIGNMENT, GPT_IMAGE_2_MAX_EDGE, GPT_IMAGE_2_MAX_PIXELS, + GPT_IMAGE_2_MIN_PIXELS, GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, + }, error::PlatformImageError, request::vector_engine_images_edit_url, types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}, @@ -24,10 +27,10 @@ pub struct RawImageEditOptions { pub mask: Option, } -pub const RAW_IMAGE_MAX_EDGE: u32 = 3_840; -pub const RAW_IMAGE_DIMENSION_ALIGNMENT: u32 = 16; -pub const RAW_IMAGE_MIN_PIXELS: u64 = 655_360; -pub const RAW_IMAGE_MAX_PIXELS: u64 = 8_294_400; +pub const RAW_IMAGE_MAX_EDGE: u32 = GPT_IMAGE_2_MAX_EDGE; +pub const RAW_IMAGE_DIMENSION_ALIGNMENT: u32 = GPT_IMAGE_2_DIMENSION_ALIGNMENT; +pub const RAW_IMAGE_MIN_PIXELS: u64 = GPT_IMAGE_2_MIN_PIXELS; +pub const RAW_IMAGE_MAX_PIXELS: u64 = GPT_IMAGE_2_MAX_PIXELS; #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum RawImageEditDimensionError { diff --git a/server-rs/crates/platform-image/src/vector_engine/request.rs b/server-rs/crates/platform-image/src/vector_engine/request.rs index af232dbc8..4dc53443e 100644 --- a/server-rs/crates/platform-image/src/vector_engine/request.rs +++ b/server-rs/crates/platform-image/src/vector_engine/request.rs @@ -1,7 +1,10 @@ use serde_json::{Map, Value, json}; use super::{ - constants::{GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL}, + constants::{ + GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_DIMENSION_ALIGNMENT, GPT_IMAGE_2_MAX_EDGE, + GPT_IMAGE_2_MAX_PIXELS, GPT_IMAGE_2_MIN_PIXELS, GPT_IMAGE_2_MODEL, + }, types::{ReferenceImage, VectorEngineImageSettings}, }; @@ -129,10 +132,10 @@ fn normalize_explicit_pixel_size(value: &str) -> String { } fn clamp_gpt_image_2_pixel_size(size: &str) -> String { - const MIN_PIXELS: u64 = 655_360; - const MAX_PIXELS: u64 = 8_294_400; - const MAX_EDGE: u32 = 3_840; - const DIMENSION_ALIGNMENT: u32 = 16; + const MIN_PIXELS: u64 = GPT_IMAGE_2_MIN_PIXELS; + const MAX_PIXELS: u64 = GPT_IMAGE_2_MAX_PIXELS; + const MAX_EDGE: u32 = GPT_IMAGE_2_MAX_EDGE; + const DIMENSION_ALIGNMENT: u32 = GPT_IMAGE_2_DIMENSION_ALIGNMENT; const MAX_ASPECT_RATIO: f64 = 3.0; // 中文注释：这里是 VectorEngine 的共享发送边界，只处理 gpt-image-2 的显式像素尺寸。 From 6068eeda83ff68142f162562c4aae751a131c3e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 23:22:56 +0800 Subject: [PATCH 021/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E8=AF=B7=E6=B1=82=E6=97=A5=E5=BF=97?= =?UTF-8?q?=20=E8=AE=B0=E5=BD=95=E4=B8=8A=E6=B8=B8=E5=93=8D=E5=BA=94?= =?UTF-8?q?=E7=8A=B6=E6=80=81=E4=B8=8E=E8=80=97=E6=97=B6=20=E8=AE=B0?= =?UTF-8?q?=E5=BD=95=E4=BC=A0=E8=BE=93=E5=A4=B1=E8=B4=A5=E9=98=B6=E6=AE=B5?= =?UTF-8?q?=E5=92=8C=E7=BB=93=E6=9E=84=E5=8C=96=E4=B8=8A=E4=B8=8B=E6=96=87?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/vector_engine/raw_edit.rs | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index c8f6ae249..be96e8515 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -171,6 +171,16 @@ pub async fn create_vector_engine_raw_image_edit( ) })?; let status = response.status(); + tracing::info!( + provider = VECTOR_ENGINE_PROVIDER, + endpoint = %url, + status, + prompt_chars, + reference_image_count, + elapsed_ms = started_at.elapsed().as_millis() as u64, + failure_context, + "VectorEngine Raw 图片编辑 HTTP 返回" + ); let body = response.text().await.map_err(|error| { request_error( &url, @@ -419,6 +429,19 @@ fn request_error( let connect = error.is_connect(); let source = error.to_string(); let message = format!("{context}：上游请求失败：{source}"); + tracing::warn!( + provider = VECTOR_ENGINE_PROVIDER, + endpoint = %url, + failure_stage, + timeout, + connect, + prompt_chars, + reference_image_count, + elapsed_ms = started_at.elapsed().as_millis() as u64, + failure_context = context, + error = %source, + "VectorEngine Raw 图片编辑请求失败" + ); let audit = build_failure_audit( url, context, From 65e73eee0227123e532510d86216407da5ccf342 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 23:25:47 +0800 Subject: [PATCH 022/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=94=9F=E6=88=90=E6=88=90=E5=8A=9F=E8=BF=BD=E8=B8=AA?= =?UTF-8?q?=20=E8=AE=B0=E5=BD=95=E6=88=90=E5=8A=9F=E4=BB=BB=E5=8A=A1?= =?UTF-8?q?=E4=B8=8E=E8=BE=93=E5=87=BA=E6=95=B0=E9=87=8F=20=E5=AE=8C?= =?UTF-8?q?=E5=96=84=E4=B8=8A=E6=B8=B8=E5=93=8D=E5=BA=94=E5=92=8C=E5=A4=B1?= =?UTF-8?q?=E8=B4=A5=E9=98=B6=E6=AE=B5=E6=97=A5=E5=BF=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/api-server/src/raw_image.rs | 32 +++++++++++++++++-- .../src/vector_engine/raw_edit.rs | 2 +- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index a41bcfd73..2ef7a9be1 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -25,7 +25,9 @@ use crate::{ }, request_context::RequestContext, state::AppState, + tracking::record_external_generation_run_after_success, }; +use time::OffsetDateTime; #[derive(Clone, Debug, Deserialize)] #[serde(rename_all = "camelCase", deny_unknown_fields)] @@ -66,8 +68,7 @@ pub(crate) async fn edit_raw_image( let prepared = tokio::task::spawn_blocking(move || prepare_request(payload)) .await .map_err(|error| { - AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR) - .with_message(error.to_string()) + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_message(error.to_string()) })??; let settings = require_openai_image_settings(&state)?.with_external_api_audit_context( &request_context, @@ -79,6 +80,17 @@ pub(crate) async fn edit_raw_image( let request_id = request_context.request_id().to_string(); let points_cost = raw_image_edit_price(&state, prepared.width, prepared.height).await?; let audit_settings = settings.clone(); + let tracking_state = audit_settings.external_api_audit_state.clone(); + let tracking_payload = json!({ + "width": prepared.width, + "height": prepared.height, + "promptChars": prepared.prompt.chars().count(), + "hasMask": prepared.options.mask.is_some(), + "quality": prepared.options.quality.as_deref(), + "background": prepared.options.background.as_deref(), + "outputFormat": prepared.options.output_format.as_deref(), + }); + let started_at_micros = (OffsetDateTime::now_utc().unix_timestamp_nanos() / 1_000) as i64; let operation = async move { let generated = match create_vector_engine_raw_image_edit( &provider_settings, @@ -95,6 +107,7 @@ pub(crate) async fn edit_raw_image( return Err(map_platform_image_error(error)); } }; + let task_id = generated.task_id.clone(); let data = generated .images .into_iter() @@ -102,6 +115,21 @@ pub(crate) async fn edit_raw_image( b64_json: BASE64_STANDARD.encode(image.bytes), }) .collect(); + if let Some(state) = tracking_state.as_ref() { + record_external_generation_run_after_success( + state, + platform_image::VECTOR_ENGINE_PROVIDER, + "raw_image_edit", + "raw_image_edit", + tracking_payload, + started_at_micros, + true, + None, + Some(task_id), + Some(json!({ "imageCount": data.len() })), + ) + .await; + } Ok::<_, AppError>(RawImageEditResponse { data }) }; let result = with_editor_generation_durable_billing_boundary( diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index be96e8515..faa042e43 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -174,7 +174,7 @@ pub async fn create_vector_engine_raw_image_edit( tracing::info!( provider = VECTOR_ENGINE_PROVIDER, endpoint = %url, - status, + status = status.as_u16(), prompt_chars, reference_image_count, elapsed_ms = started_at.elapsed().as_millis() as u64, From ed86963de2a29432190bb082c0003e33fd0dbea6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 23:38:25 +0800 Subject: [PATCH 023/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E8=BF=BD=E8=B8=AA=E7=BB=93=E6=9E=9C=E7=B1=BB=E5=9E=8B?= =?UTF-8?q?=20=E6=98=8E=E7=A1=AE=E8=BE=93=E5=87=BA=E6=95=B0=E6=8D=AE?= =?UTF-8?q?=E9=9B=86=E5=90=88=E7=B1=BB=E5=9E=8B=20=E4=BF=9D=E6=8C=81=20API?= =?UTF-8?q?=20=E7=BC=96=E8=AF=91=E6=A3=80=E6=9F=A5=E9=80=9A=E8=BF=87?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- server-rs/crates/api-server/src/raw_image.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 2ef7a9be1..2023ecec8 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -108,7 +108,7 @@ pub(crate) async fn edit_raw_image( } }; let task_id = generated.task_id.clone(); - let data = generated + let data: Vec = generated .images .into_iter() .map(|image| RawImageEditItem { From 3a95715990fc36b2fc311e5405b054e2ad514430 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 23:41:27 +0800 Subject: [PATCH 024/248] =?UTF-8?q?=E5=90=8C=E6=AD=A5=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E6=8A=80=E6=9C=AF=E6=96=B9=E6=A1=88?= =?UTF-8?q?=20=E8=AE=B0=E5=BD=95=E8=A7=A3=E7=A0=81=E8=B5=84=E6=BA=90?= =?UTF-8?q?=E4=B8=8A=E9=99=90=E5=92=8C=E9=98=BB=E5=A1=9E=E7=BA=BF=E7=A8=8B?= =?UTF-8?q?=E7=AD=96=E7=95=A5=20=E8=A1=A5=E5=85=85=E8=BE=93=E5=87=BA?= =?UTF-8?q?=E6=A0=BC=E5=BC=8F=E5=9B=9E=E9=80=80=E4=B8=8E=E6=88=90=E5=8A=9F?= =?UTF-8?q?=E8=BF=BD=E8=B8=AA=E7=BA=A6=E5=AE=9A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 0783cbc83..9bd8cfd4f 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -46,7 +46,7 @@ POST /api/raw/v1/images/edit 校验通过后按整数尺寸发送给 provider，不静默 clamp 或改写调用者尺寸。 -Raw 路由的 JSON body limit 为 `64 MiB`，为 base64 编码膨胀和可选 mask 留出空间；同时必须在 base64 解码后拒绝空 PNG，并保留图片格式校验，避免仅依赖 HTTP body limit。 +Raw 路由的 JSON body limit 为 `64 MiB`，为 base64 编码膨胀和可选 mask 留出空间；同时必须在 base64 解码后拒绝空 PNG，并保留图片格式校验，避免仅依赖 HTTP body limit。PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；base64 与 PNG 解码在阻塞线程执行，不占用 Tokio 异步 worker。 服务端发送给 `platform-image` 时固定注入： @@ -93,7 +93,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、JSON DTO、base64 解码、预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 结果统一解码为图片字节；每项结果的 MIME 与扩展名以 VectorEngine 响应中的真实 `output_format` 为准，不得从请求参数反推。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链。raw handler 只将结果字节编码到 `data[].b64_json`。 +provider 结果统一解码为图片字节；每项结果的 MIME 与扩展名以 VectorEngine 响应中的真实 `output_format` 为准，不得从请求参数反推；entry 级格式为空或非字符串时回退 payload 级格式。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将结果字节编码到 `data[].b64_json`。 ## 代码拆分 From 5ee2d940c1c56d52cf7a294c34bf780a12f026a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 10:55:36 +0800 Subject: [PATCH 025/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=B0=BA=E5=AF=B8=E6=A0=A1=E9=AA=8C=E9=94=99=E8=AF=AF?= =?UTF-8?q?=E8=AF=AD=E4=B9=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 尺寸校验失败改用请求参数无效语义 补充错误文案回归测试 --- .../platform-image/src/vector_engine/raw_edit.rs | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index faa042e43..05b88b156 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -100,7 +100,7 @@ pub async fn create_vector_engine_raw_image_edit( failure_context: &str, ) -> Result { validate_raw_image_edit_dimensions(options.width, options.height) - .map_err(|error| invalid_request(failure_context, error.to_string()))?; + .map_err(|error| invalid_input(failure_context, error.to_string()))?; let url = vector_engine_images_edit_url(settings); let started_at = Instant::now(); let prompt_chars = Some(prompt.chars().count()); @@ -347,6 +347,13 @@ fn invalid_request(context: &str, message: String) -> PlatformImageError { } } +fn invalid_input(context: &str, message: String) -> PlatformImageError { + PlatformImageError::InvalidRequest { + provider: VECTOR_ENGINE_PROVIDER, + message: format!("{context}：请求参数无效：{message}"), + } +} + fn invalid_config(message: String) -> PlatformImageError { PlatformImageError::InvalidConfig { provider: VECTOR_ENGINE_PROVIDER, @@ -566,4 +573,11 @@ mod tests { Err(RawImageEditDimensionError::PixelCount) ); } + + #[test] + fn dimension_validation_error_identifies_client_input() { + let error = invalid_input("raw_image_edit", "尺寸无效".to_string()); + + assert_eq!(error.to_string(), "raw_image_edit：请求参数无效：尺寸无效"); + } } From f7cccd8592eccd97e747972a965868fe0cb1df7d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 10:57:04 +0800 Subject: [PATCH 026/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=AE=A2=E6=88=B7=E7=AB=AF=E9=94=99=E8=AF=AF=E4=B8=8A?= =?UTF-8?q?=E4=B8=8B=E6=96=87?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 HTTP 客户端构造失败统一保留操作上下文 补充配置错误文案回归测试 --- .../platform-image/src/vector_engine/raw_edit.rs | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 05b88b156..a428752ce 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -152,7 +152,7 @@ pub async fn create_vector_engine_raw_image_edit( .timeout(Duration::from_millis(request_timeout_ms)) .http1_only() .build() - .map_err(|error| invalid_config(error.to_string()))?; + .map_err(|error| invalid_config(failure_context, error.to_string()))?; let response = client .post(url.as_str()) .bearer_auth(settings.api_key.as_str()) @@ -354,10 +354,10 @@ fn invalid_input(context: &str, message: String) -> PlatformImageError { } } -fn invalid_config(message: String) -> PlatformImageError { +fn invalid_config(context: &str, message: String) -> PlatformImageError { PlatformImageError::InvalidConfig { provider: VECTOR_ENGINE_PROVIDER, - message, + message: format!("{context}：构造请求客户端失败：{message}"), } } @@ -580,4 +580,14 @@ mod tests { assert_eq!(error.to_string(), "raw_image_edit：请求参数无效：尺寸无效"); } + + #[test] + fn invalid_config_error_keeps_operation_context() { + let error = invalid_config("raw_image_edit", "builder failed".to_string()); + + assert_eq!( + error.to_string(), + "raw_image_edit：构造请求客户端失败：builder failed" + ); + } } From f60771dd007bb7184a6b866cad7270a06a0db7d2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 10:58:48 +0800 Subject: [PATCH 027/248] =?UTF-8?q?=E6=8F=90=E5=89=8D=E6=A3=80=E6=9F=A5=20?= =?UTF-8?q?Raw=20=E5=9B=BE=E7=89=87=E8=AF=B7=E6=B1=82=E9=A2=84=E7=AE=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在构造 multipart 表单前校验请求 deadline 避免预算耗尽时复制输入图片并缩短耗时统计范围 --- .../src/vector_engine/raw_edit.rs | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index a428752ce..1373d8545 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -105,6 +105,18 @@ pub async fn create_vector_engine_raw_image_edit( let started_at = Instant::now(); let prompt_chars = Some(prompt.chars().count()); let reference_image_count = Some(1_usize + usize::from(options.mask.is_some())); + let Some(request_timeout_ms) = + effective_request_timeout_ms(settings.request_timeout_ms, settings.request_deadline) + else { + return Err(request_budget_exhausted_error( + url.as_str(), + failure_context, + Some(GPT_IMAGE_2_MODEL), + Some(started_at.elapsed().as_millis() as u64), + prompt_chars, + reference_image_count, + )); + }; let mut form = Form::new() .text("model", GPT_IMAGE_2_MODEL.to_string()) .text("n", "1".to_string()) @@ -136,18 +148,6 @@ pub async fn create_vector_engine_raw_image_edit( ); } - let Some(request_timeout_ms) = - effective_request_timeout_ms(settings.request_timeout_ms, settings.request_deadline) - else { - return Err(request_budget_exhausted_error( - url.as_str(), - failure_context, - Some(GPT_IMAGE_2_MODEL), - Some(started_at.elapsed().as_millis() as u64), - prompt_chars, - reference_image_count, - )); - }; let client = reqwest::Client::builder() .timeout(Duration::from_millis(request_timeout_ms)) .http1_only() From f7edc9d0eed88632f03b1cadbe770950b5e8692f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:04:44 +0800 Subject: [PATCH 028/248] =?UTF-8?q?=E5=8C=BA=E5=88=86=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=20PNG=20=E8=B5=84=E6=BA=90=E9=99=90=E5=88=B6=E9=94=99?= =?UTF-8?q?=E8=AF=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将有效但超出解码限制的 PNG 映射为明确的 400 提示 补充超大合法 PNG 的回归测试 --- server-rs/crates/api-server/src/raw_image.rs | 32 +++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 2023ecec8..cea19316c 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -243,7 +243,7 @@ fn decode_image(value: RawImageData, field: &str) -> Result Result AppError { + let message = match error { + image::ImageError::Limits(_) => { + format!("{field}.data 超出 PNG 尺寸或解码资源上限（单边不超过 {RAW_IMAGE_MAX_EDGE}px）") + } + _ => format!("{field}.data 必须是有效 PNG 文件"), + }; + bad_request(message) +} + async fn raw_image_edit_price(state: &AppState, width: u32, height: u32) -> Result { let tier = if width.max(height) > 1536 { "2K" } else { "1K" }; state @@ -373,4 +383,24 @@ mod tests { assert!(prepare_request(serde_json::from_value(payload).expect("request")).is_err()); } } + + #[test] + fn oversized_valid_png_reports_resource_limit() { + let payload = serde_json::json!({ + "image": { + "data": encoded_png(RAW_IMAGE_MAX_EDGE + 1, 1), + "mimeType": "image/png" + }, + "prompt": "edit", + "width": 1024, + "height": 1024 + }); + let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); + let error = match prepare_request(parsed) { + Ok(_) => panic!("oversized PNG should fail"), + Err(error) => error, + }; + + assert!(format!("{error:?}").contains("超出 PNG 尺寸或解码资源上限")); + } } From b3693765ef5693ed1edcafc605b101bd7a200756 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:34:41 +0800 Subject: [PATCH 029/248] =?UTF-8?q?=E9=80=8F=E4=BC=A0=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BB=93=E6=9E=9C=E9=81=BF=E5=85=8D=E8=A7=A3=E7=A0=81?= =?UTF-8?q?=E5=9B=9E=E6=98=BE=E5=AD=97=E6=AE=B5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 仅转发上游 data.b64_json 并忽略 output_format 同步 Raw 技术方案并补充不解码回归测试 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 4 +- server-rs/crates/api-server/src/raw_image.rs | 6 +- server-rs/crates/platform-image/src/lib.rs | 2 +- .../platform-image/src/vector_engine/mod.rs | 4 +- .../src/vector_engine/raw_edit.rs | 197 +++--------------- .../platform-image/src/vector_engine/types.rs | 7 + 6 files changed, 40 insertions(+), 180 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 9bd8cfd4f..12b1f0331 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -93,12 +93,12 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、JSON DTO、base64 解码、预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 结果统一解码为图片字节；每项结果的 MIME 与扩展名以 VectorEngine 响应中的真实 `output_format` 为准，不得从请求参数反推；entry 级格式为空或非字符串时回退 payload 级格式。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将结果字节编码到 `data[].b64_json`。 +provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 ## 代码拆分 - `server-rs/crates/api-server/src/raw_image.rs`：独立路由 handler、请求/响应 DTO、base64 输入校验、预检查和 raw billing 编排。 -- `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、严格尺寸校验、独立 provider 请求映射和原始响应解码；不复用现有 editor 图片编辑 client 或其 multipart transport。 +- `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、严格尺寸校验、独立 provider 请求映射和 `b64_json` 响应透传；不复用现有 editor 图片编辑 client 或其 multipart transport。 - `server-rs/crates/api-server/src/modules/raw.rs`：只注册 `/api/raw/v1/images/edit` 并挂载 Bearer middleware。 不修改 External v1 OpenAPI；不在 `external_editor_api.rs`、编辑器项目模块或外部生成 worker 中增加 raw 分支。 diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index cea19316c..456dcca69 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -109,11 +109,9 @@ pub(crate) async fn edit_raw_image( }; let task_id = generated.task_id.clone(); let data: Vec = generated - .images + .b64_images .into_iter() - .map(|image| RawImageEditItem { - b64_json: BASE64_STANDARD.encode(image.bytes), - }) + .map(|b64_json| RawImageEditItem { b64_json }) .collect(); if let Some(state) = tracking_state.as_ref() { record_external_generation_run_after_success( diff --git a/server-rs/crates/platform-image/src/lib.rs b/server-rs/crates/platform-image/src/lib.rs index 7140aec48..c527d4c7d 100644 --- a/server-rs/crates/platform-image/src/lib.rs +++ b/server-rs/crates/platform-image/src/lib.rs @@ -12,7 +12,7 @@ pub use vector_engine::{ DownloadedImage, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, GeneratedImages, NANOBANANA_2_MODEL, PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, - RawImageEditDimensionError, RawImageEditOptions, ReferenceImage, + RawImageEditDimensionError, RawImageEditOptions, RawImageEditResult, ReferenceImage, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, VectorEngineImageSettings, build_vector_engine_image_http_client, build_vector_engine_image_request_body, build_vector_engine_nanobanana_generate_content_request_body, create_vector_engine_image_edit, diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index b94d22221..351244650 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -38,4 +38,6 @@ pub use request::{ vector_engine_nanobanana_generate_content_url, }; pub use transport::build_vector_engine_image_http_client; -pub use types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}; +pub use types::{ + DownloadedImage, GeneratedImages, RawImageEditResult, ReferenceImage, VectorEngineImageSettings, +}; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 1373d8545..c0d78f1e8 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -1,6 +1,5 @@ use std::time::{Duration, Instant}; -use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use reqwest::multipart::{Form, Part}; use serde_json::Value; @@ -13,7 +12,7 @@ use super::{ }, error::PlatformImageError, request::vector_engine_images_edit_url, - types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}, + types::{RawImageEditResult, ReferenceImage, VectorEngineImageSettings}, util::truncate_raw, }; @@ -98,7 +97,7 @@ pub async fn create_vector_engine_raw_image_edit( image: &ReferenceImage, options: RawImageEditOptions, failure_context: &str, -) -> Result { +) -> Result { validate_raw_image_edit_dimensions(options.width, options.height) .map_err(|error| invalid_input(failure_context, error.to_string()))?; let url = vector_engine_images_edit_url(settings); @@ -250,61 +249,8 @@ pub async fn create_vector_engine_raw_image_edit( }); } }; - let mut images = Vec::new(); - if let Some(entries) = payload.get("data").and_then(Value::as_array) { - for entry in entries { - let Some(value) = entry.get("b64_json").and_then(Value::as_str) else { - continue; - }; - let bytes = match BASE64_STANDARD.decode(value) { - Ok(bytes) => bytes, - Err(error) => { - let message = format!("{failure_context}：上游 b64_json 解码失败：{error}"); - let audit = build_failure_audit( - url.as_str(), - failure_context, - "response_parse", - Some(status.as_u16()), - Some(status_class(status.as_u16())), - false, - false, - message.as_str(), - Some(error.to_string()), - Some(truncate_raw(body.as_str())), - Some(started_at.elapsed().as_millis() as u64), - prompt_chars, - reference_image_count, - Some(GPT_IMAGE_2_MODEL), - ); - return Err(PlatformImageError::ResponseParse { - provider: VECTOR_ENGINE_PROVIDER, - message, - raw_excerpt: truncate_raw(body.as_str()), - audit: Some(audit), - }); - } - }; - let (mime_type, extension) = - response_image_format(&payload, entry).map_err(|message| { - response_parse_error( - &url, - failure_context, - message, - status.as_u16(), - started_at, - prompt_chars, - reference_image_count, - &body, - ) - })?; - images.push(DownloadedImage { - bytes, - mime_type: mime_type.to_string(), - extension: extension.to_string(), - }); - } - } - if images.is_empty() { + let b64_images = extract_b64_images(&payload); + if b64_images.is_empty() { let message = format!("{failure_context}：上游未返回 b64_json 图片"); let audit = build_failure_audit( url.as_str(), @@ -328,18 +274,28 @@ pub async fn create_vector_engine_raw_image_edit( audit: Some(audit), }); } - Ok(GeneratedImages { + Ok(RawImageEditResult { task_id: payload .get("id") .and_then(Value::as_str) .unwrap_or("raw-image-edit") .to_string(), - actual_prompt: None, - images, + b64_images, recovered_failure_audits: Vec::new(), }) } +fn extract_b64_images(payload: &Value) -> Vec { + payload + .get("data") + .and_then(Value::as_array) + .into_iter() + .flatten() + .filter_map(|entry| entry.get("b64_json").and_then(Value::as_str)) + .map(str::to_string) + .collect() +} + fn invalid_request(context: &str, message: String) -> PlatformImageError { PlatformImageError::InvalidRequest { provider: VECTOR_ENGINE_PROVIDER, @@ -361,68 +317,6 @@ fn invalid_config(context: &str, message: String) -> PlatformImageError { } } -fn response_parse_error( - url: &str, - context: &str, - message: &str, - status: u16, - started_at: Instant, - prompt_chars: Option, - reference_image_count: Option, - body: &str, -) -> PlatformImageError { - let audit = build_failure_audit( - url, - context, - "response_parse", - Some(status), - Some(status_class(status)), - false, - false, - message, - None, - Some(truncate_raw(body)), - Some(started_at.elapsed().as_millis() as u64), - prompt_chars, - reference_image_count, - Some(GPT_IMAGE_2_MODEL), - ); - PlatformImageError::ResponseParse { - provider: VECTOR_ENGINE_PROVIDER, - message: format!("{context}：{message}"), - raw_excerpt: truncate_raw(body), - audit: Some(audit), - } -} - -fn response_image_format( - payload: &Value, - entry: &Value, -) -> Result<(&'static str, &'static str), &'static str> { - let Some(value) = entry - .get("output_format") - .and_then(Value::as_str) - .map(str::trim) - .filter(|value| !value.is_empty()) - .or_else(|| { - payload - .get("output_format") - .and_then(Value::as_str) - .map(str::trim) - .filter(|value| !value.is_empty()) - }) - else { - return Err("上游响应缺少 output_format"); - }; - match value.to_ascii_lowercase().as_str() { - "png" => Ok(("image/png", "png")), - "jpeg" | "jpg" => Ok(("image/jpeg", "jpg")), - "webp" => Ok(("image/webp", "webp")), - "gif" => Ok(("image/gif", "gif")), - _ => Err("上游响应包含不支持的 output_format"), - } -} - fn request_error( url: &str, context: &str, @@ -495,57 +389,16 @@ mod tests { use serde_json::json; #[test] - fn response_format_uses_vector_engine_output_format() { - let payload = json!({"output_format": "png"}); - assert_eq!( - response_image_format(&payload, &json!({"output_format": "webp"})), - Ok(("image/webp", "webp")) - ); - assert_eq!( - response_image_format(&payload, &json!({})), - Ok(("image/png", "png")) - ); - assert_eq!( - response_image_format(&json!({}), &json!({"output_format": "jpeg"})), - Ok(("image/jpeg", "jpg")) - ); - assert_eq!( - response_image_format( - &json!({"output_format": "png"}), - &json!({"output_format": ""}) - ), - Ok(("image/png", "png")) - ); - assert_eq!( - response_image_format( - &json!({"output_format": "webp"}), - &json!({"output_format": null}) - ), - Ok(("image/webp", "webp")) - ); - assert!(response_image_format(&json!({}), &json!({})).is_err()); - assert!(response_image_format(&json!({}), &json!({"output_format": "bmp"})).is_err()); - } + fn raw_result_forwards_b64_without_decoding_or_using_output_format() { + let payload = json!({ + "output_format": "png", + "data": [{"b64_json": "not-base64-but-forwarded", "output_format": "jpeg"}] + }); - #[test] - fn response_parse_error_contains_structured_audit() { - let error = response_parse_error( - "https://vector.example/v1/images/edits", - "raw_image_edit", - "上游响应缺少 output_format", - 200, - Instant::now(), - Some(12), - Some(2), - "{\"data\":[]}", + assert_eq!( + extract_b64_images(&payload), + vec!["not-base64-but-forwarded".to_string()] ); - let audit = error.audit().expect("response error should carry audit"); - assert_eq!(audit.failure_stage, "response_parse"); - assert_eq!(audit.status_code, Some(200)); - assert_eq!(audit.status_class, Some("2xx")); - assert_eq!(audit.prompt_chars, Some(12)); - assert_eq!(audit.reference_image_count, Some(2)); - assert_eq!(audit.image_model, Some(GPT_IMAGE_2_MODEL)); } #[test] diff --git a/server-rs/crates/platform-image/src/vector_engine/types.rs b/server-rs/crates/platform-image/src/vector_engine/types.rs index 77fbd19f9..82a69bb4a 100644 --- a/server-rs/crates/platform-image/src/vector_engine/types.rs +++ b/server-rs/crates/platform-image/src/vector_engine/types.rs @@ -16,6 +16,13 @@ pub struct GeneratedImages { pub recovered_failure_audits: Vec, } +#[derive(Clone, Debug)] +pub struct RawImageEditResult { + pub task_id: String, + pub b64_images: Vec, + pub recovered_failure_audits: Vec, +} + #[derive(Clone, Debug)] pub struct DownloadedImage { pub bytes: Vec, From 6c77de4a36f7fb97dcd60afee129c1db46024142 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:44:31 +0800 Subject: [PATCH 030/248] =?UTF-8?q?=E4=B8=BA=20Raw=20=E6=88=90=E5=8A=9F?= =?UTF-8?q?=E5=93=8D=E5=BA=94=E6=94=B6=E7=B4=A7=E7=B1=BB=E5=9E=8B=E9=80=8F?= =?UTF-8?q?=E4=BC=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 使用 typed JSON DTO 提取并原样转发 b64_json 不解码图片内容且忽略 output_format 回显 --- .../src/vector_engine/raw_edit.rs | 52 +++++++++++-------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index c0d78f1e8..13f41e7e6 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -1,7 +1,7 @@ use std::time::{Duration, Instant}; use reqwest::multipart::{Form, Part}; -use serde_json::Value; +use serde::Deserialize; use super::{ audit::build_failure_audit, @@ -26,6 +26,17 @@ pub struct RawImageEditOptions { pub mask: Option, } +#[derive(Debug, Deserialize)] +struct RawImageEditResponsePayload { + id: Option, + data: Option>, +} + +#[derive(Debug, Deserialize)] +struct RawImageEditResponseEntry { + b64_json: Option, +} + pub const RAW_IMAGE_MAX_EDGE: u32 = GPT_IMAGE_2_MAX_EDGE; pub const RAW_IMAGE_DIMENSION_ALIGNMENT: u32 = GPT_IMAGE_2_DIMENSION_ALIGNMENT; pub const RAW_IMAGE_MIN_PIXELS: u64 = GPT_IMAGE_2_MIN_PIXELS; @@ -221,7 +232,7 @@ pub async fn create_vector_engine_raw_image_edit( audit: Some(audit), }); } - let payload: Value = match serde_json::from_str(body.as_str()) { + let payload: RawImageEditResponsePayload = match serde_json::from_str(body.as_str()) { Ok(payload) => payload, Err(error) => { let message = format!("{failure_context}：上游响应不是 JSON：{error}"); @@ -249,7 +260,12 @@ pub async fn create_vector_engine_raw_image_edit( }); } }; - let b64_images = extract_b64_images(&payload); + let b64_images = payload + .data + .unwrap_or_default() + .into_iter() + .filter_map(|entry| entry.b64_json) + .collect::>(); if b64_images.is_empty() { let message = format!("{failure_context}：上游未返回 b64_json 图片"); let audit = build_failure_audit( @@ -275,27 +291,12 @@ pub async fn create_vector_engine_raw_image_edit( }); } Ok(RawImageEditResult { - task_id: payload - .get("id") - .and_then(Value::as_str) - .unwrap_or("raw-image-edit") - .to_string(), + task_id: payload.id.unwrap_or_else(|| "raw-image-edit".to_string()), b64_images, recovered_failure_audits: Vec::new(), }) } -fn extract_b64_images(payload: &Value) -> Vec { - payload - .get("data") - .and_then(Value::as_array) - .into_iter() - .flatten() - .filter_map(|entry| entry.get("b64_json").and_then(Value::as_str)) - .map(str::to_string) - .collect() -} - fn invalid_request(context: &str, message: String) -> PlatformImageError { PlatformImageError::InvalidRequest { provider: VECTOR_ENGINE_PROVIDER, @@ -390,13 +391,20 @@ mod tests { #[test] fn raw_result_forwards_b64_without_decoding_or_using_output_format() { - let payload = json!({ + let payload = r#"{ "output_format": "png", "data": [{"b64_json": "not-base64-but-forwarded", "output_format": "jpeg"}] - }); + }"#; + let payload: RawImageEditResponsePayload = + serde_json::from_str(payload).expect("response envelope"); assert_eq!( - extract_b64_images(&payload), + payload + .data + .unwrap_or_default() + .into_iter() + .filter_map(|entry| entry.b64_json) + .collect::>(), vec!["not-base64-but-forwarded".to_string()] ); } From 40316d82c4b1e8ade506115a4665de0b2ffd7855 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:45:44 +0800 Subject: [PATCH 031/248] =?UTF-8?q?=E5=9C=A8=E6=89=A3=E8=B4=B9=E5=89=8D?= =?UTF-8?q?=E6=A0=A1=E9=AA=8C=20Raw=20mask=20=E5=B0=BA=E5=AF=B8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 解码时保留 image 与 mask 宽高并拒绝不一致输入 同步 Raw 图片编辑请求合同与回归测试 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- server-rs/crates/api-server/src/raw_image.rs | 49 +++++++++++++++---- 2 files changed, 40 insertions(+), 11 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 12b1f0331..c5dd38c37 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -35,7 +35,7 @@ POST /api/raw/v1/images/edit } ``` -`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件；空数据、非 PNG 字节或 MIME 不匹配均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 +`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件，提供 mask 时其宽高必须与 image 完全一致；空数据、非 PNG 字节、MIME 不匹配或尺寸不一致均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 `width`、`height` 使用严格输出尺寸规则，均在扣费前校验： diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 456dcca69..e0e88a876 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -4,7 +4,7 @@ use axum::{ http::StatusCode, }; use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; -use image::{ImageFormat, ImageReader}; +use image::{GenericImageView, ImageFormat, ImageReader}; use platform_image::{ RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, @@ -176,10 +176,16 @@ fn prepare_request(payload: RawImageEditRequest) -> Result( Err(bad_request(format!("{field} 值无效"))) } -fn decode_image(value: RawImageData, field: &str) -> Result { +fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32, u32), AppError> { let mime_type = value.mime_type.trim().to_string(); if !mime_type.eq_ignore_ascii_case("image/png") { return Err(bad_request(format!("{field}.mimeType 必须为 image/png"))); @@ -239,14 +245,19 @@ fn decode_image(value: RawImageData, field: &str) -> Result AppError { @@ -401,4 +412,22 @@ mod tests { assert!(format!("{error:?}").contains("超出 PNG 尺寸或解码资源上限")); } + + #[test] + fn mask_must_match_source_image_dimensions() { + let payload = serde_json::json!({ + "image": {"data": encoded_png(2, 1), "mimeType": "image/png"}, + "mask": {"data": encoded_png(1, 1), "mimeType": "image/png"}, + "prompt": "edit", + "width": 1024, + "height": 1024 + }); + let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); + let error = match prepare_request(parsed) { + Ok(_) => panic!("mismatched mask should fail before billing"), + Err(error) => error, + }; + + assert!(format!("{error:?}").contains("mask 尺寸必须与 image 一致")); + } } From 5bbedd9a5ca77efc46c14801c06859596d620d61 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:50:30 +0800 Subject: [PATCH 032/248] =?UTF-8?q?=E9=99=90=E5=88=B6=20Raw=20prompt=20?= =?UTF-8?q?=E5=8E=9F=E5=A7=8B=E5=AD=97=E8=8A=82=E5=A4=A7=E5=B0=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 prompt UTF-8 原始字节限制为 16 KiB 补充超限回归测试并同步请求合同 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- server-rs/crates/api-server/src/raw_image.rs | 24 +++++++++++++++++++ server-rs/crates/platform-image/Cargo.toml | 1 + 3 files changed, 26 insertions(+), 1 deletion(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index c5dd38c37..4c319b003 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -35,7 +35,7 @@ POST /api/raw/v1/images/edit } ``` -`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件，提供 mask 时其宽高必须与 image 完全一致；空数据、非 PNG 字节、MIME 不匹配或尺寸不一致均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 +`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件，提供 mask 时其宽高必须与 image 完全一致；空数据、非 PNG 字节、MIME 不匹配或尺寸不一致均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填，UTF-8 原始字节长度不得超过 `16 KiB`；超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 `width`、`height` 使用严格输出尺寸规则，均在扣费前校验： diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index e0e88a876..1c46e7fbc 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -59,6 +59,8 @@ pub(crate) struct RawImageEditResponse { pub(crate) data: Vec, } +const RAW_IMAGE_MAX_PROMPT_BYTES: usize = 16 * 1024; + pub(crate) async fn edit_raw_image( State(state): State, Extension(request_context): Extension, @@ -156,6 +158,11 @@ fn prepare_request(payload: RawImageEditRequest) -> Result RAW_IMAGE_MAX_PROMPT_BYTES { + return Err(bad_request(format!( + "prompt 不能超过 {RAW_IMAGE_MAX_PROMPT_BYTES} 字节" + ))); + } validate_raw_image_edit_dimensions(payload.width, payload.height) .map_err(|error| bad_request(error.to_string()))?; validate_optional_value( @@ -430,4 +437,21 @@ mod tests { assert!(format!("{error:?}").contains("mask 尺寸必须与 image 一致")); } + + #[test] + fn prompt_uses_raw_utf8_byte_limit() { + let payload = serde_json::json!({ + "image": {"data": encoded_png(1, 1), "mimeType": "image/png"}, + "prompt": "a".repeat(RAW_IMAGE_MAX_PROMPT_BYTES + 1), + "width": 1024, + "height": 1024 + }); + let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); + let error = match prepare_request(parsed) { + Ok(_) => panic!("oversized prompt should fail before image decode and billing"), + Err(error) => error, + }; + + assert!(format!("{error:?}").contains("prompt 不能超过 16384 字节")); + } } diff --git a/server-rs/crates/platform-image/Cargo.toml b/server-rs/crates/platform-image/Cargo.toml index 9da088343..2030dab06 100644 --- a/server-rs/crates/platform-image/Cargo.toml +++ b/server-rs/crates/platform-image/Cargo.toml @@ -9,6 +9,7 @@ base64 = { workspace = true } curl = { workspace = true } image = { workspace = true, features = ["jpeg", "png", "webp"] } reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls"] } +serde = { workspace = true } serde_json = { workspace = true } tokio = { workspace = true, features = ["io-util", "macros", "net", "time"] } tracing = { workspace = true } From acfcf0c158fa754867ac34dbc8998ba14e7ef2a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:51:25 +0800 Subject: [PATCH 033/248] =?UTF-8?q?=E5=90=8C=E6=AD=A5=20platform-image=20?= =?UTF-8?q?=E7=9A=84=20serde=20=E9=94=81=E5=AE=9A=E4=BE=9D=E8=B5=96?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 Raw typed 响应记录 serde workspace 依赖 --- server-rs/Cargo.lock | 1 + 1 file changed, 1 insertion(+) diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index 1acba75f0..6581b2ef0 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -4074,6 +4074,7 @@ dependencies = [ "image", "platform-oss", "reqwest", + "serde", "serde_json", "tokio", "tracing", From c049d68b8eeaf98e407deecfb77bd2f012b3ee50 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 12:03:03 +0800 Subject: [PATCH 034/248] =?UTF-8?q?=E6=94=B6=E7=B4=A7=20Raw=20=E6=88=90?= =?UTF-8?q?=E5=8A=9F=E5=93=8D=E5=BA=94=E5=AD=97=E6=AE=B5=E7=B1=BB=E5=9E=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 要求 data 与 b64_json 必填并移除无用 provider id prompt 上限调整为 4 KiB 原始字节并同步文档 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- server-rs/crates/api-server/src/raw_image.rs | 7 +++---- .../src/vector_engine/raw_edit.rs | 19 ++++++++++--------- .../platform-image/src/vector_engine/types.rs | 1 - 4 files changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 4c319b003..f8696987b 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -35,7 +35,7 @@ POST /api/raw/v1/images/edit } ``` -`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件，提供 mask 时其宽高必须与 image 完全一致；空数据、非 PNG 字节、MIME 不匹配或尺寸不一致均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填，UTF-8 原始字节长度不得超过 `16 KiB`；超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 +`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件，提供 mask 时其宽高必须与 image 完全一致；空数据、非 PNG 字节、MIME 不匹配或尺寸不一致均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填，UTF-8 原始字节长度不得超过 `4 KiB`；超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 `width`、`height` 使用严格输出尺寸规则，均在扣费前校验： diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 1c46e7fbc..9dc066ae1 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -59,7 +59,7 @@ pub(crate) struct RawImageEditResponse { pub(crate) data: Vec, } -const RAW_IMAGE_MAX_PROMPT_BYTES: usize = 16 * 1024; +const RAW_IMAGE_MAX_PROMPT_BYTES: usize = 4 * 1024; pub(crate) async fn edit_raw_image( State(state): State, @@ -109,7 +109,6 @@ pub(crate) async fn edit_raw_image( return Err(map_platform_image_error(error)); } }; - let task_id = generated.task_id.clone(); let data: Vec = generated .b64_images .into_iter() @@ -125,7 +124,7 @@ pub(crate) async fn edit_raw_image( started_at_micros, true, None, - Some(task_id), + Some("raw-image-edit".to_string()), Some(json!({ "imageCount": data.len() })), ) .await; @@ -452,6 +451,6 @@ mod tests { Err(error) => error, }; - assert!(format!("{error:?}").contains("prompt 不能超过 16384 字节")); + assert!(format!("{error:?}").contains("prompt 不能超过 4096 字节")); } } diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 13f41e7e6..77c386322 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -28,13 +28,12 @@ pub struct RawImageEditOptions { #[derive(Debug, Deserialize)] struct RawImageEditResponsePayload { - id: Option, - data: Option>, + data: Vec, } #[derive(Debug, Deserialize)] struct RawImageEditResponseEntry { - b64_json: Option, + b64_json: String, } pub const RAW_IMAGE_MAX_EDGE: u32 = GPT_IMAGE_2_MAX_EDGE; @@ -262,9 +261,8 @@ pub async fn create_vector_engine_raw_image_edit( }; let b64_images = payload .data - .unwrap_or_default() .into_iter() - .filter_map(|entry| entry.b64_json) + .map(|entry| entry.b64_json) .collect::>(); if b64_images.is_empty() { let message = format!("{failure_context}：上游未返回 b64_json 图片"); @@ -291,7 +289,6 @@ pub async fn create_vector_engine_raw_image_edit( }); } Ok(RawImageEditResult { - task_id: payload.id.unwrap_or_else(|| "raw-image-edit".to_string()), b64_images, recovered_failure_audits: Vec::new(), }) @@ -387,7 +384,6 @@ fn status_class(status: u16) -> &'static str { #[cfg(test)] mod tests { use super::*; - use serde_json::json; #[test] fn raw_result_forwards_b64_without_decoding_or_using_output_format() { @@ -401,14 +397,19 @@ mod tests { assert_eq!( payload .data - .unwrap_or_default() .into_iter() - .filter_map(|entry| entry.b64_json) + .map(|entry| entry.b64_json) .collect::>(), vec!["not-base64-but-forwarded".to_string()] ); } + #[test] + fn raw_success_response_requires_typed_data_and_b64_json_fields() { + assert!(serde_json::from_str::(r#"{}"#).is_err()); + assert!(serde_json::from_str::(r#"{"data":[{}]}"#).is_err()); + } + #[test] fn raw_image_edit_dimensions_enforce_strict_contract() { assert!(validate_raw_image_edit_dimensions(1024, 640).is_ok()); diff --git a/server-rs/crates/platform-image/src/vector_engine/types.rs b/server-rs/crates/platform-image/src/vector_engine/types.rs index 82a69bb4a..d25ad5c12 100644 --- a/server-rs/crates/platform-image/src/vector_engine/types.rs +++ b/server-rs/crates/platform-image/src/vector_engine/types.rs @@ -18,7 +18,6 @@ pub struct GeneratedImages { #[derive(Clone, Debug)] pub struct RawImageEditResult { - pub task_id: String, pub b64_images: Vec, pub recovered_failure_audits: Vec, } From 0400103a1985f0333dd353585d680521aeb8c71f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 13:06:22 +0800 Subject: [PATCH 035/248] =?UTF-8?q?=E5=B0=86=20Raw=20=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E7=BC=96=E8=BE=91=E5=85=A5=E7=AB=99=E6=94=B9=E4=B8=BA=20multip?= =?UTF-8?q?art?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 移除 JSON/base64 入站兼容，仅接受 multipart 图片与参数字段 复用扣费前 PNG、尺寸、mask 和 prompt 校验并补充 multipart 测试 优化上游图片字节转发，避免再次克隆 image bytes 同步 Raw 图片编辑技术方案与 Axum multipart 依赖 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 38 +- server-rs/Cargo.lock | 24 ++ server-rs/crates/api-server/Cargo.toml | 2 +- server-rs/crates/api-server/src/raw_image.rs | 325 ++++++++++++------ .../src/vector_engine/raw_edit.rs | 13 +- 5 files changed, 270 insertions(+), 132 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index f8696987b..ed130ed2a 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -14,28 +14,20 @@ POST /api/raw/v1/images/edit ## 请求合同 -请求使用 `application/json`。图片字段只使用原始图片的 base64 数据和 MIME 类型，不接受 object key、URL、Data URL 或 Blob URL。 +请求使用 `multipart/form-data`，不再接受 JSON/base64 入站格式。图片直接作为文件字段上传，避免 base64 膨胀和入站解码；服务端仍在扣费前完成 PNG 完整解码与资源限制校验。 -```json -{ - "image": { - "data": "", - "mimeType": "image/png" - }, - "mask": { - "data": "", - "mimeType": "image/png" - }, - "prompt": "修改图片", - "quality": "auto", - "background": "auto", - "output_format": "png", - "width": 1536, - "height": 1024 -} +```text +image: +mask: +prompt: 修改图片 +quality: auto +background: auto +output_format: png +width: 1536 +height: 1024 ``` -`image` 是必填的单图结构 `{ data, mimeType }`；`mask` 可选并使用相同结构。`image` 和 `mask` 的 `mimeType` 必须为 `image/png`，base64 解码后必须是可完整解码的有效 PNG 文件，提供 mask 时其宽高必须与 image 完全一致；空数据、非 PNG 字节、MIME 不匹配或尺寸不一致均在扣费前返回 400。服务端不把输入格式另建成请求参数。`prompt` 必填，UTF-8 原始字节长度不得超过 `4 KiB`；超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。 +`image` 和 `mask` 必须是 `image/png` 文件字段；服务端不信任客户端文件名，转发时使用固定文件名。空文件、非 PNG 字节、MIME 不匹配或 mask 与 image 尺寸不一致均在扣费前返回 400。`prompt` 必填，UTF-8 原始字节长度不得超过 `4 KiB`；超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。字段不能重复，未知字段拒绝；缺失的必填字段拒绝。 `width`、`height` 使用严格输出尺寸规则，均在扣费前校验： @@ -46,7 +38,7 @@ POST /api/raw/v1/images/edit 校验通过后按整数尺寸发送给 provider，不静默 clamp 或改写调用者尺寸。 -Raw 路由的 JSON body limit 为 `64 MiB`，为 base64 编码膨胀和可选 mask 留出空间；同时必须在 base64 解码后拒绝空 PNG，并保留图片格式校验，避免仅依赖 HTTP body limit。PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；base64 与 PNG 解码在阻塞线程执行，不占用 Tokio 异步 worker。 +Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；文件字段由 multipart 解析器直接收集为字节，随后在阻塞线程中完成 PNG 解码。PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 服务端发送给 `platform-image` 时固定注入： @@ -75,7 +67,7 @@ n = 1 ## 预检查与计费事务 -所有请求、JSON、base64、图片结构和 provider 参数检查必须在扣费前完成。预检查失败直接返回 4xx，不产生钱包流水，也不调用 provider。 +所有 multipart 字段、图片结构和 provider 参数检查必须在扣费前完成。预检查失败直接返回 4xx，不产生钱包流水，也不调用 provider。 检查通过后，api-server 进入现有资产操作计费边界，通过 SpacetimeDB 钱包事务 procedure 原子完成： @@ -91,13 +83,13 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed ## Provider 边界 -`platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、JSON DTO、base64 解码、预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 +`platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、multipart 字段解析、PNG 预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 ## 代码拆分 -- `server-rs/crates/api-server/src/raw_image.rs`：独立路由 handler、请求/响应 DTO、base64 输入校验、预检查和 raw billing 编排。 +- `server-rs/crates/api-server/src/raw_image.rs`：独立路由 handler、multipart 字段解析、请求/响应 DTO、PNG 输入校验、预检查和 raw billing 编排。 - `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、严格尺寸校验、独立 provider 请求映射和 `b64_json` 响应透传；不复用现有 editor 图片编辑 client 或其 multipart transport。 - `server-rs/crates/api-server/src/modules/raw.rs`：只注册 `/api/raw/v1/images/edit` 并挂载 Bearer middleware。 diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index 6581b2ef0..b77676b63 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -470,6 +470,7 @@ dependencies = [ "matchit", "memchr", "mime", + "multer", "percent-encoding", "pin-project-lite", "serde_core", @@ -2924,6 +2925,23 @@ dependencies = [ "pxfm", ] +[[package]] +name = "multer" +version = "3.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83e87776546dc87511aa5ee218730c92b666d7264ab6ed41f9d215af9cd5224b" +dependencies = [ + "bytes", + "encoding_rs", + "futures-util", + "http", + "httparse", + "memchr", + "mime", + "spin", + "version_check", +] + [[package]] name = "naga" version = "27.0.3" @@ -5629,6 +5647,12 @@ dependencies = [ "tokio-tungstenite 0.27.0", ] +[[package]] +name = "spin" +version = "0.9.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" + [[package]] name = "sse-stream" version = "0.2.5" diff --git a/server-rs/crates/api-server/Cargo.toml b/server-rs/crates/api-server/Cargo.toml index 78cfb8bab..0a166cd5f 100644 --- a/server-rs/crates/api-server/Cargo.toml +++ b/server-rs/crates/api-server/Cargo.toml @@ -7,7 +7,7 @@ license.workspace = true [dependencies] aes = { workspace = true } async-stream = { workspace = true } -axum = { workspace = true, features = ["ws"] } +axum = { workspace = true, features = ["ws", "multipart"] } base64 = { workspace = true } cbc = { workspace = true } bytes = { workspace = true } diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 9dc066ae1..fc0d33a6d 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -1,15 +1,14 @@ use axum::{ Json, - extract::{Extension, State}, + extract::{Extension, Multipart, State}, http::StatusCode, }; -use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use image::{GenericImageView, ImageFormat, ImageReader}; use platform_image::{ RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, }; -use serde::{Deserialize, Serialize}; +use serde::Serialize; use serde_json::json; use std::io::Cursor; @@ -29,16 +28,15 @@ use crate::{ }; use time::OffsetDateTime; -#[derive(Clone, Debug, Deserialize)] -#[serde(rename_all = "camelCase", deny_unknown_fields)] -pub(crate) struct RawImageData { - pub(crate) data: String, +#[derive(Debug)] +struct RawImageData { + pub(crate) bytes: Vec, pub(crate) mime_type: String, + pub(crate) file_name: String, } -#[derive(Clone, Debug, Deserialize)] -#[serde(rename_all = "snake_case", deny_unknown_fields)] -pub(crate) struct RawImageEditRequest { +#[derive(Debug)] +struct RawImageEditRequest { pub(crate) image: RawImageData, pub(crate) mask: Option, pub(crate) prompt: String, @@ -65,8 +63,9 @@ pub(crate) async fn edit_raw_image( State(state): State, Extension(request_context): Extension, Extension(authenticated): Extension, - Json(payload): Json, + multipart: Multipart, ) -> Result, AppError> { + let payload = parse_multipart_request(multipart).await?; let prepared = tokio::task::spawn_blocking(move || prepare_request(payload)) .await .map_err(|error| { @@ -97,7 +96,7 @@ pub(crate) async fn edit_raw_image( let generated = match create_vector_engine_raw_image_edit( &provider_settings, prepared.prompt.as_str(), - &prepared.image, + prepared.image, prepared.options, "raw_image_edit", ) @@ -153,6 +152,114 @@ struct PreparedRawImageEdit { height: u32, } +async fn parse_multipart_request( + mut multipart: Multipart, +) -> Result { + let mut image = None; + let mut mask = None; + let mut prompt = None; + let mut quality = None; + let mut background = None; + let mut output_format = None; + let mut width = None; + let mut height = None; + + while let Some(field) = multipart + .next_field() + .await + .map_err(|error| bad_request(format!("multipart 字段读取失败：{error}")))? + { + let name = field + .name() + .ok_or_else(|| bad_request("multipart 字段缺少名称"))? + .to_string(); + match name.as_str() { + "image" => { + if image.is_some() { + return Err(bad_request("image 字段不能重复")); + } + image = Some(read_multipart_image(field, "image").await?); + } + "mask" => { + if mask.is_some() { + return Err(bad_request("mask 字段不能重复")); + } + mask = Some(read_multipart_image(field, "mask").await?); + } + "prompt" => set_text_field(&mut prompt, field, "prompt").await?, + "quality" => set_text_field(&mut quality, field, "quality").await?, + "background" => set_text_field(&mut background, field, "background").await?, + "output_format" => set_text_field(&mut output_format, field, "output_format").await?, + "width" => set_text_field(&mut width, field, "width").await?, + "height" => set_text_field(&mut height, field, "height").await?, + _ => return Err(bad_request(format!("不支持的 multipart 字段：{name}"))), + } + } + + let image = image.ok_or_else(|| bad_request("image 字段不能为空"))?; + let prompt = prompt.ok_or_else(|| bad_request("prompt 字段不能为空"))?; + let width = parse_multipart_u32(width, "width")?; + let height = parse_multipart_u32(height, "height")?; + + Ok(RawImageEditRequest { + image, + mask, + prompt, + quality, + background, + output_format, + width, + height, + }) +} + +async fn read_multipart_image( + field: axum::extract::multipart::Field<'_>, + name: &str, +) -> Result { + let mime_type = field.content_type().unwrap_or_default().to_string(); + if !mime_type.eq_ignore_ascii_case("image/png") { + return Err(bad_request(format!("{name} 必须为 image/png"))); + } + let bytes = field + .bytes() + .await + .map_err(|error| bad_request(format!("{name} 文件读取失败：{error}")))?; + if bytes.is_empty() { + return Err(bad_request(format!("{name} 文件不能为空"))); + } + Ok(RawImageData { + bytes: bytes.to_vec(), + mime_type: "image/png".to_string(), + file_name: format!("{name}.png"), + }) +} + +async fn set_text_field( + target: &mut Option, + field: axum::extract::multipart::Field<'_>, + name: &str, +) -> Result<(), AppError> { + if target.is_some() { + return Err(bad_request(format!("{name} 字段不能重复"))); + } + *target = Some( + field + .text() + .await + .map_err(|error| bad_request(format!("{name} 字段读取失败：{error}")))?, + ); + Ok(()) +} + +fn parse_multipart_u32(value: Option, field: &str) -> Result { + let value = value.ok_or_else(|| bad_request(format!("{field} 字段不能为空")))?; + value + .trim() + .parse::() + .map_err(|_| bad_request(format!("{field} 必须为有效整数"))) +} + fn prepare_request(payload: RawImageEditRequest) -> Result { if payload.prompt.trim().is_empty() { return Err(bad_request("prompt 不能为空")); @@ -232,24 +339,24 @@ fn validate_optional_value( fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32, u32), AppError> { let mime_type = value.mime_type.trim().to_string(); if !mime_type.eq_ignore_ascii_case("image/png") { - return Err(bad_request(format!("{field}.mimeType 必须为 image/png"))); + return Err(bad_request(format!( + "{field} Content-Type 必须为 image/png" + ))); } - let bytes = BASE64_STANDARD - .decode(value.data.trim()) - .map_err(|_| bad_request(format!("{field}.data 必须是有效 base64")))?; + let bytes = value.bytes; if bytes.is_empty() { - return Err(bad_request(format!("{field}.data 不能为空"))); + return Err(bad_request(format!("{field} 文件不能为空"))); } let mut reader = ImageReader::new(Cursor::new(bytes.as_slice())) .with_guessed_format() - .map_err(|_| bad_request(format!("{field}.data 必须是有效 PNG 文件")))?; + .map_err(|_| bad_request(format!("{field} 文件必须是有效 PNG 文件")))?; let mut limits = image::Limits::default(); limits.max_image_width = Some(RAW_IMAGE_MAX_EDGE); limits.max_image_height = Some(RAW_IMAGE_MAX_EDGE); limits.max_alloc = Some(RAW_IMAGE_MAX_PIXELS.saturating_mul(4)); reader.limits(limits); if reader.format() != Some(ImageFormat::Png) { - return Err(bad_request(format!("{field}.data 必须是有效 PNG 文件"))); + return Err(bad_request(format!("{field} 文件必须是有效 PNG 文件"))); } let decoded = reader .decode() @@ -258,8 +365,8 @@ fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32 Ok(( ReferenceImage { bytes, - file_name: format!("{field}.png"), - mime_type: "image/png".to_string(), + file_name: value.file_name, + mime_type, }, width, height, @@ -269,9 +376,9 @@ fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32 fn map_decode_image_error(field: &str, error: image::ImageError) -> AppError { let message = match error { image::ImageError::Limits(_) => { - format!("{field}.data 超出 PNG 尺寸或解码资源上限（单边不超过 {RAW_IMAGE_MAX_EDGE}px）") + format!("{field} 文件超出 PNG 尺寸或解码资源上限（单边不超过 {RAW_IMAGE_MAX_EDGE}px）") } - _ => format!("{field}.data 必须是有效 PNG 文件"), + _ => format!("{field} 文件必须是有效 PNG 文件"), }; bad_request(message) } @@ -302,37 +409,84 @@ fn bad_request(message: impl Into) -> AppError { #[cfg(test)] mod tests { use super::*; + use axum::{body::Body, extract::FromRequest, http::Request}; use image::{ImageFormat, Rgba, RgbaImage}; use std::io::Cursor; - fn encoded_png(width: u32, height: u32) -> String { + fn png_bytes(width: u32, height: u32) -> Vec { let image = RgbaImage::from_pixel(width, height, Rgba([255, 0, 0, 255])); let mut bytes = Vec::new(); image .write_to(&mut Cursor::new(&mut bytes), ImageFormat::Png) .expect("test PNG should encode"); - BASE64_STANDARD.encode(bytes) + bytes } - #[test] - fn request_uses_one_image_object_and_rejects_images_array() { - let payload = serde_json::json!({ - "image": {"data": encoded_png(1, 1), "mimeType": "image/png"}, - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("image object"); - let prepared = prepare_request(parsed).expect("request should prepare"); - assert!(prepared.image.bytes.starts_with(b"\x89PNG\r\n\x1a\n")); + fn request(image: Vec, mask: Option>) -> RawImageEditRequest { + RawImageEditRequest { + image: RawImageData { + bytes: image, + mime_type: "image/png".to_string(), + file_name: "image.png".to_string(), + }, + mask: mask.map(|bytes| RawImageData { + bytes, + mime_type: "image/png".to_string(), + file_name: "mask.png".to_string(), + }), + prompt: "edit".to_string(), + width: 1024, + height: 1024, + quality: None, + background: None, + output_format: None, + } + } - let array_payload = serde_json::json!({ - "images": [{"data": "aGVsbG8=", "mimeType": "image/png"}], - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - assert!(serde_json::from_value::(array_payload).is_err()); + fn multipart_body(boundary: &str, image: &[u8]) -> Vec { + let mut body = Vec::new(); + let add_text = |body: &mut Vec, name: &str, value: &str| { + body.extend_from_slice(format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"{name}\"\r\n\r\n{value}\r\n" + ).as_bytes()); + }; + add_text(&mut body, "prompt", "edit"); + add_text(&mut body, "width", "1024"); + add_text(&mut body, "height", "1024"); + body.extend_from_slice( + format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"image\"; filename=\"ignored.png\"\r\nContent-Type: image/png\r\n\r\n" + ) + .as_bytes(), + ); + body.extend_from_slice(image); + body.extend_from_slice(format!("\r\n--{boundary}--\r\n").as_bytes()); + body + } + + #[tokio::test] + async fn multipart_parser_accepts_binary_image_and_text_fields() { + let boundary = "raw-test-boundary"; + let image = png_bytes(1, 1); + let body = multipart_body(boundary, &image); + let request = Request::builder() + .header( + "content-type", + format!("multipart/form-data; boundary={boundary}"), + ) + .body(Body::from(body)) + .expect("multipart request"); + let multipart = Multipart::from_request(request, &()) + .await + .expect("multipart"); + let parsed = parse_multipart_request(multipart) + .await + .expect("multipart fields should parse"); + + assert_eq!(parsed.prompt, "edit"); + assert_eq!(parsed.width, 1024); + assert_eq!(parsed.height, 1024); + assert!(parsed.image.bytes.starts_with(b"\x89PNG\r\n\x1a\n")); } #[test] @@ -349,24 +503,6 @@ mod tests { ); } - #[test] - fn invalid_base64_uses_generic_client_message() { - let payload = serde_json::json!({ - "image": {"data": "not base64!", "mimeType": "image/png"}, - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); - let error = match prepare_request(parsed) { - Ok(_) => panic!("invalid base64 should fail"), - Err(error) => error, - }; - let rendered = format!("{error:?}"); - assert!(rendered.contains("image.data 必须是有效 base64")); - assert!(!rendered.contains("InvalidByte")); - } - #[test] fn dimensions_follow_strict_raw_image_contract() { assert!(validate_raw_image_edit_dimensions(1024, 1024).is_ok()); @@ -380,38 +516,32 @@ mod tests { #[test] fn input_requires_decodable_png_and_png_mime() { - let valid = serde_json::json!({ - "image": {"data": encoded_png(1, 1), "mimeType": "IMAGE/PNG"}, - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - assert!(prepare_request(serde_json::from_value(valid).expect("valid request")).is_ok()); + assert!(prepare_request(request(png_bytes(1, 1), None)).is_ok()); - for (data, mime_type) in [("aGVsbG8=", "image/png"), ("aGVsbG8=", "image/jpeg")] { - let payload = serde_json::json!({ - "image": {"data": data, "mimeType": mime_type}, - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - assert!(prepare_request(serde_json::from_value(payload).expect("request")).is_err()); - } + let invalid_bytes = RawImageEditRequest { + image: RawImageData { + bytes: b"hello".to_vec(), + mime_type: "image/png".to_string(), + file_name: "image.png".to_string(), + }, + ..request(png_bytes(1, 1), None) + }; + assert!(prepare_request(invalid_bytes).is_err()); + + let invalid_mime = RawImageEditRequest { + image: RawImageData { + bytes: png_bytes(1, 1), + mime_type: "image/jpeg".to_string(), + file_name: "image.jpg".to_string(), + }, + ..request(png_bytes(1, 1), None) + }; + assert!(prepare_request(invalid_mime).is_err()); } #[test] fn oversized_valid_png_reports_resource_limit() { - let payload = serde_json::json!({ - "image": { - "data": encoded_png(RAW_IMAGE_MAX_EDGE + 1, 1), - "mimeType": "image/png" - }, - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); - let error = match prepare_request(parsed) { + let error = match prepare_request(request(png_bytes(RAW_IMAGE_MAX_EDGE + 1, 1), None)) { Ok(_) => panic!("oversized PNG should fail"), Err(error) => error, }; @@ -421,15 +551,7 @@ mod tests { #[test] fn mask_must_match_source_image_dimensions() { - let payload = serde_json::json!({ - "image": {"data": encoded_png(2, 1), "mimeType": "image/png"}, - "mask": {"data": encoded_png(1, 1), "mimeType": "image/png"}, - "prompt": "edit", - "width": 1024, - "height": 1024 - }); - let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); - let error = match prepare_request(parsed) { + let error = match prepare_request(request(png_bytes(2, 1), Some(png_bytes(1, 1)))) { Ok(_) => panic!("mismatched mask should fail before billing"), Err(error) => error, }; @@ -439,13 +561,8 @@ mod tests { #[test] fn prompt_uses_raw_utf8_byte_limit() { - let payload = serde_json::json!({ - "image": {"data": encoded_png(1, 1), "mimeType": "image/png"}, - "prompt": "a".repeat(RAW_IMAGE_MAX_PROMPT_BYTES + 1), - "width": 1024, - "height": 1024 - }); - let parsed: RawImageEditRequest = serde_json::from_value(payload).expect("request"); + let mut parsed = request(png_bytes(1, 1), None); + parsed.prompt = "a".repeat(RAW_IMAGE_MAX_PROMPT_BYTES + 1); let error = match prepare_request(parsed) { Ok(_) => panic!("oversized prompt should fail before image decode and billing"), Err(error) => error, diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 77c386322..d215adbbc 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -104,7 +104,7 @@ pub fn validate_raw_image_edit_dimensions( pub async fn create_vector_engine_raw_image_edit( settings: &VectorEngineImageSettings, prompt: &str, - image: &ReferenceImage, + image: ReferenceImage, options: RawImageEditOptions, failure_context: &str, ) -> Result { @@ -126,6 +126,11 @@ pub async fn create_vector_engine_raw_image_edit( reference_image_count, )); }; + let ReferenceImage { + bytes: image_bytes, + file_name: image_file_name, + mime_type: image_mime_type, + } = image; let mut form = Form::new() .text("model", GPT_IMAGE_2_MODEL.to_string()) .text("n", "1".to_string()) @@ -133,9 +138,9 @@ pub async fn create_vector_engine_raw_image_edit( .text("size", format!("{}x{}", options.width, options.height)) .part( "image", - Part::bytes(image.bytes.clone()) - .file_name(image.file_name.clone()) - .mime_str(image.mime_type.as_str()) + Part::bytes(image_bytes) + .file_name(image_file_name) + .mime_str(image_mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, ); if let Some(value) = options.quality { From b2d4690f9407a90a6c62f22cece6a08e50589b12 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 15:12:56 +0800 Subject: [PATCH 036/248] =?UTF-8?q?=E8=A1=A5=E5=85=85UI=E7=BC=96=E8=BE=91?= =?UTF-8?q?=E5=99=A8=E8=87=AA=E5=8A=A8=E5=88=86=E7=A6=BB=E5=B7=A5=E4=BD=9C?= =?UTF-8?q?=E6=B5=81=E5=90=88=E5=90=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增自动分离、临时 sidecar 与视觉绑定设计合同 明确前端资产登记职责与恢复 TODO --- ...术方案】UI编辑器自动分离工作流-2026-09-08.md | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md new file mode 100644 index 000000000..c21e40e68 --- /dev/null +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -0,0 +1,75 @@ +# UI 编辑器自动分离工作流 + +更新时间：`2026-09-08` + +## 目标 + +将 UI 编辑器现有“用户先提供独立图片/图标，再执行组件绑定”的入口替换为自动分离：结构识别阶段直接返回可渲染组件草稿，分离阶段按整页叶节点批次调用图片编辑模型，再由视觉模型确认处理图中的区域与目标节点。 + +## 识别结果 + +- `recognize` 返回完整 `Node.components` 草稿，不再要求用户先导入独立素材。 +- `components` 为空表示纯节点。 +- `ImageComponent.target_graphic = None` 表示图片组件等待分离结果回填；它不是“明确没有图片”。 +- 当前约束：需要分离的节点最多包含一个 `ImageComponent`，回填暂使用该节点的第一个图片组件。 +- 组件容器“一种组件类型最多一个”的正式重构列为 TODO；当前 `Vec` 仅按上述约束使用。 +- 组件草稿直接保存在正式 UI Node 中；临时 separation tree 不复制组件。 + +## Separation tree + +- recognition 完成后由 UI tree 构造临时 separation tree。 +- 纯节点、纯 Text 节点和不需要切图的节点在构造时过滤；被过滤节点的可处理 children 向上透传。 +- separation tree 只保留真实待处理节点。 +- 一个 batch 是整页当前所有互不重叠叶节点。 +- 一个 batch 的最小处理单元是：一次 image-edit + 一次 visual binding。 +- batch 成功后从 pending tree 移除对应叶节点，并把结果放入 bound 容器；失败节点移入 problematic 容器，流程继续消费剩余树。 +- 不额外维护节点状态枚举；节点是否仍在 pending tree、`rework_count` 和 problematic 容器共同表达状态。 + +## 图片编辑与视觉绑定 + +- image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。 +- 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 +- 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 +- `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 +- `NeedRework` 携带短问题描述。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 +- 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。 +- 父节点背景重建由 image-edit 模型完成，不由 Rust 硬编码重建算法完成。 + +## 临时 sidecar + +- separation 状态不写入 UI JSON，也不进入 manifest。 +- sidecar 目录按 UI manifest `asset_id` 生成，复用 `generated_file_stem(asset_id)` 的安全字符替换和 SHA-256 摘要规则，位于项目 `ui/` 下。 +- 目录只保存一份当前 separation state，而不是每 batch 一个状态文件。 +- state 文件只保留 `schema_version`、pending tree、bound 结果和 problematic 节点，不重复保存 `projectId / assetId / uiStateRevision`。 +- sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 +- 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 +- 临时图片可跨重启保留。raw image-edit 返回图、绿色/紫色标记图、处理图和 cut 图片当前都保留用于 debug；理论上只应在内存中，清理/归档策略列 TODO。 + +## bound 与 problematic + +- bound 结果仅保存 `NodeId + cut_image_path`，不保存 `BindingArea` 或 component kind。 +- problematic 记录原始 NodeId、问题描述和 `rework_count`；原始 UI Node 保留不变。 +- `SeparationDTO` 不返回计数字段，只返回 `bound_nodes` 与 `problematic_nodes`。 +- separation Rust 流程不自动登记项目级 SpriteAsset。 +- 前端调用方消费 `SeparationDTO.bound_nodes`，复制/登记 cut 图片为项目级 SpriteAsset，再回填对应 Node 的第一个 Image component。 +- 每次重做产生新的 SpriteAssetId，不假设 NodeId 到 SpriteAssetId 的稳定映射。 +- sidecar 中的图片保留，正式 SpriteAsset 的最终清理策略列 TODO。 + +## 重启与 Raw GPT Image 2 + +- 已保存的 separation state 是跨重启继续工作的最小单位；重启后从上一个已保存 batch 的状态继续。 +- 当前执行中的 batch 是否持久化、以及如何避免 image-edit 成功后在 patch 前崩溃导致重复调用，列为 TODO。 +- Raw endpoint 每次 HTTP 调用都是一次新操作；客户端不保存或复用 raw operation ID，不实现第二套本地幂等账本。 +- 后端 raw operation 的持久状态与扣费后崩溃恢复窗口，遵循 Raw GPT Image 2 方案中的独立 TODO。 + +## TODO + +- `Vec` 重构为一种组件类型最多一个的容器。 +- 当前第一个 Image component 回填规则的正式替代方案。 +- 正在执行 batch 的持久化和恢复。 +- 前端复制、登记 SpriteAsset、回填 State 的精确 IPC/提交合同。 +- 临时图片清理/归档策略。 +- 手动抠图能力。 +- problematic 对更高层 workflow 完成门禁的最终定义。 +- separation workflow 与 manifest/stage 的接入。 +- Raw GPT Image 2 后端 raw operation 持久状态及恢复 worker。 From 4cc7f4dd6442e70b248ee1125f0387193b90f87f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 18:07:27 +0800 Subject: [PATCH 037/248] =?UTF-8?q?=E5=AE=9E=E7=8E=B0UI=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E5=91=BD=E4=BB=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增分离树、批次处理与sidecar状态 接入Raw GPT Image 2与视觉绑定重试 注册Tauri命令并补充识别组件草稿 --- .../src-tauri/src/main.rs | 10 + .../src-tauri/src/ui_editor/commands/mod.rs | 3 + .../src/ui_editor/commands/recognition.rs | 35 +- .../src/ui_editor/commands/separation.rs | 861 ++++++++++++++++++ .../src-tauri/src/ui_editor/persistence.rs | 2 +- 5 files changed, 906 insertions(+), 5 deletions(-) create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs diff --git a/apps/ai-game-creator-shell/src-tauri/src/main.rs b/apps/ai-game-creator-shell/src-tauri/src/main.rs index e163cad21..3f4293087 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/main.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/main.rs @@ -328,6 +328,15 @@ async fn recognize_ui( ui_editor::commands::recognize_ui_impl(project_path, state).await } +#[tauri::command] +async fn separate_ui( + project_path: String, + asset_id: String, + state: ui_editor::state::State, +) -> Result { + ui_editor::commands::separate_ui_impl(project_path, asset_id, state).await +} + #[tauri::command] async fn merge_ui(state: ui_editor::state::State) -> Result { ui_editor::commands::merge_ui_impl(state).await @@ -2559,6 +2568,7 @@ fn main() { check_ui_editor_font_glyph_coverage, suggest_ui_design_semantic, recognize_ui, + separate_ui, merge_ui, bind_components, load_ui_design_state, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs index e2e1a1bc5..181931f2d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs @@ -1,6 +1,7 @@ pub mod binding; pub mod merge; pub mod recognition; +pub mod separation; pub mod ui_design_suggestion; pub mod utils; @@ -10,5 +11,7 @@ pub use merge::MergeDTO; pub(crate) use merge::{merge_ui_impl, merge_ui_impl_with_provider}; pub use recognition::RecognitionDTO; pub(crate) use recognition::{recognize_ui_impl, recognize_ui_impl_with_provider}; +pub(crate) use separation::separate_ui_impl; +pub use separation::SeparationDTO; pub(crate) use ui_design_suggestion::suggest_ui_design_semantic_impl; pub use ui_design_suggestion::UIDesignSuggestionTreeNode; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index 081db0bb1..ceae0d7ea 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -4,6 +4,7 @@ use crate::ui_editor::commands::utils::{ parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, strict_json_schema, }; +use crate::ui_editor::component::Component; use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; use crate::ui_editor::layout::control_layout::ControlLayout; use crate::ui_editor::layout::dimension::UIRect; @@ -46,6 +47,8 @@ const SYSTEM_PROMPT: &str = r#" * 由于每个截图未必是完整的, 可能是局部的, 每棵树描述清楚每个截图上UI的层次结构即可 * 不同树的共用框架/层次/...请使用使用相同的名称描述. 不同状态/变体名称使用相同的前缀, 用后缀区别 * 粒度要求: 尽可能细致, 最小单元举例: 进度条的底槽、填充和外框; slider的底槽, dragger等 +* 为每个节点直接返回完整 components。纯容器返回空数组；需要从设计图自动分离图片的 Image component 必须令 target_graphic 为 null。文本内容和组件类型完全由视觉判断，不调用或依赖 OCR。 +* 每个节点当前最多返回一个 Image component 和一个 Text component。 "#; @@ -109,6 +112,7 @@ struct RecognitionNode { description: String, children: Vec, confidence: Confidence, + components: Vec, } #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] @@ -312,10 +316,7 @@ fn convert_node( allow_llm_edit_component: true, source: NodeSource::Llm, }, - // V1 有意把识别结果限定为“结构草稿”：组件绑定属于后续独立阶段。 - // 因此空组件不是丢失数据，而是等待 visual-binding 阶段补齐 Image/Text。 - // 约定见 docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md。 - components: Vec::new(), + components: source.components.clone(), children_display_mode: ChildrenDisplayMode::Stack, children, }) @@ -323,6 +324,30 @@ fn convert_node( fn validate_confidence(nodes: &[RecognitionNode]) -> Result<(), String> { for node in nodes { + let image_count = node + .components + .iter() + .filter(|component| matches!(component, Component::Image(_))) + .count(); + let text_count = node + .components + .iter() + .filter(|component| matches!(component, Component::Text(_))) + .count(); + if image_count > 1 || text_count > 1 { + return Err("单个节点当前最多包含一个 Image 和一个 Text component".to_string()); + } + if node.components.iter().any(|component| { + matches!( + component, + Component::Image(crate::ui_editor::component::image::ImageComponent { + target_graphic: Some(_), + .. + }) + ) + }) { + return Err("识别阶段不能返回已绑定的 SpriteAssetId".to_string()); + } if let Confidence::UnSure(reason) = &node.confidence { if reason.trim().is_empty() { return Err("UnSure 必须包含审阅原因".to_string()); @@ -387,6 +412,7 @@ mod tests { description: String::new(), children: Vec::new(), confidence: Confidence::Confident, + components: Vec::new(), } } @@ -490,6 +516,7 @@ mod tests { description: String::new(), children: Vec::new(), confidence: Confidence::UnSure(String::new()), + components: Vec::new(), }; assert!(validate_confidence(&[node]).is_err()); } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs new file mode 100644 index 000000000..d1217aefa --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs @@ -0,0 +1,861 @@ +use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; +use crate::platform_session::current_platform_session; +use crate::ui_editor::commands::utils::{ + parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, + strict_json_schema, +}; +use crate::ui_editor::component::image::ImageComponent; +use crate::ui_editor::component::Component; +use crate::ui_editor::layout::node::Node; +use crate::ui_editor::state::{State, UITree}; +use crate::ui_editor::utils::{NodeId, UIDesignImageId}; +use base64::Engine as _; +use image::ImageFormat; +use platform_llm::{ + LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice, +}; +use schemars::JsonSchema; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::fs; +use std::path::{Path, PathBuf}; +use ts_rs::TS; + +pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; +pub const MAX_REWORK_COUNT: u32 = 3; + +#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationNote { + pub description: String, +} + +impl SeparationNote { + pub fn as_prompt(&self) -> String { + format!("- {}", self.description.trim()) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationNode { + pub id: NodeId, + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, + pub note: SeparationNote, + pub children: Vec, + pub rework_count: u32, +} + +impl SeparationNode { + pub fn as_prompt(&self) -> String { + format!( + "node_id={} area=({}, {}, {}, {}) {}", + self.id.as_str(), + self.global_pos_x_px, + self.global_pos_y_px, + self.width_px, + self.height_px, + self.note.as_prompt() + ) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationTree { + pub src_ui_design: UIDesignImageId, + pub root: SeparationNode, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct BoundNode { + pub node_id: NodeId, + pub cut_image_path: String, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct ProblematicNode { + pub node_id: NodeId, + pub problem_description: String, + pub rework_count: u32, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationState { + pub schema_version: String, + pub unprocessed_trees: Vec, + pub bound: Vec, + pub problematic_nodes: Vec, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationDTO { + pub bound_nodes: Vec, + pub problematic_nodes: Vec, +} + +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub struct BindingArea { + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, +} + +impl BindingArea { + pub fn validate_in(&self, image_width: u32, image_height: u32) -> Result<(), String> { + if self.width_px == 0 || self.height_px == 0 { + return Err("BindingArea 宽度和高度必须大于 0".to_string()); + } + let max_x = self + .global_pos_x_px + .checked_add(self.width_px) + .ok_or_else(|| "BindingArea 横向范围溢出".to_string())?; + let max_y = self + .global_pos_y_px + .checked_add(self.height_px) + .ok_or_else(|| "BindingArea 纵向范围溢出".to_string())?; + if max_x > image_width || max_y > image_height { + return Err("BindingArea 超出处理图边界".to_string()); + } + Ok(()) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub enum BindingDecision { + Ok { + separated_image_area: BindingArea, + to_node: NodeId, + }, + NeedRework { + problem_description: String, + to_node: NodeId, + }, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub struct BindingResp { + pub decisions: Vec, +} + +const SHARED_SEPARATION_REQ: &str = r#" +MUST hard edges; preserve no glow/blur beyond the exact visible shape. +NEVER keep its parent's background with it. +UI elements marked with GREEN line frames are extraction marks only; never include the frame. +PURPLE filled areas represent removed elements; reconstruct the background under them. +"#; + +pub fn gen_extract_prompt(separation_notes: &[SeparationNote]) -> String { + let mut result = format!("This is a UI design image, not a normal photo/illustration.\nExtract distinct UI elements as independent layers with clean edges and full transparency outside each element.\nKeep every element at its original position on a transparent canvas.\n{}\nElements to extract:\n", SHARED_SEPARATION_REQ); + for note in separation_notes { + result.push_str(¬e.as_prompt()); + result.push('\n'); + } + result +} + +pub fn gen_binding_prompt(nodes: &[&SeparationNode]) -> String { + let mut result = format!("You are reviewing a UI elements separation result.\n{}\nThe source and processed images use top-left pixel coordinates.\nReturn one decision for every requested node. For a failed node, provide a short repair description.\nNodes:\n", SHARED_SEPARATION_REQ); + for node in nodes { + result.push_str(&node.as_prompt()); + result.push('\n'); + } + result +} + +fn is_unbound_image(node: &Node) -> bool { + node.components.iter().any(|component| { + matches!( + component, + Component::Image(ImageComponent { + target_graphic: None, + .. + }) + ) + }) +} + +fn node_pixel_rect( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, +) -> (u32, u32, u32, u32) { + let rect = node.layout.transform.resolve(parent); + let x = (rect.min.x * ppu).max(0.0).round() as u32; + let y = (rect.min.y * ppu).max(0.0).round() as u32; + let w = (rect.size.x * ppu).max(0.0).round() as u32; + let h = (rect.size.y * ppu).max(0.0).round() as u32; + (x, y, w, h) +} + +fn node_description(node: &Node) -> String { + let name = node.metadata.name.trim(); + let description = node.metadata.description.trim(); + match (name.is_empty(), description.is_empty()) { + (true, true) => "未命名 UI 图片元素".to_string(), + (false, true) => name.to_string(), + (true, false) => description.to_string(), + (false, false) => format!("{name}：{description}"), + } +} + +fn collect_todo_nodes( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, + output: &mut Vec, +) { + let mut children = Vec::new(); + let rect = node.layout.transform.resolve(parent); + for child in &node.children { + collect_todo_nodes(child, &rect, ppu, &mut children); + } + if is_unbound_image(node) { + let (x, y, w, h) = node_pixel_rect(node, parent, ppu); + output.push(SeparationNode { + id: node.id.clone(), + global_pos_x_px: x, + global_pos_y_px: y, + width_px: w, + height_px: h, + note: SeparationNote { + description: node_description(node), + }, + children, + rework_count: 0, + }); + } else { + output.extend(children); + } +} + +pub fn construct_separation_state(state: &State) -> SeparationState { + let unprocessed_trees = state + .ui_trees + .iter() + .filter_map(|tree| { + let image = state.ui_design_images.get(&tree.src_ui_design)?; + let ppu = image.pixels_per_unit.get(); + let size = image.pixel_size / ppu; + let root_rect = + crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); + let mut children = Vec::new(); + collect_todo_nodes(&tree.root, &root_rect, ppu, &mut children); + (!children.is_empty()).then(|| SeparationTree { + src_ui_design: tree.src_ui_design.clone(), + root: SeparationNode { + id: tree.root.id.clone(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: image.pixel_size.x.max(0.0).round() as u32, + height_px: image.pixel_size.y.max(0.0).round() as u32, + note: SeparationNote::default(), + children, + rework_count: 0, + }, + }) + }) + .collect(); + SeparationState { + schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), + unprocessed_trees, + bound: Vec::new(), + problematic_nodes: Vec::new(), + } +} + +pub fn next_leaf_batch(tree: &SeparationTree) -> Vec<&SeparationNode> { + fn leaves<'a>(node: &'a SeparationNode, output: &mut Vec<&'a SeparationNode>) { + if node.children.is_empty() { + output.push(node); + } else { + for child in &node.children { + leaves(child, output); + } + } + } + let mut output = Vec::new(); + leaves(&tree.root, &mut output); + output +} + +pub fn validate_binding_response( + response: &BindingResp, + batch: &[&SeparationNode], +) -> Result<(), String> { + let expected = batch + .iter() + .map(|node| node.id.clone()) + .collect::>(); + let mut seen = std::collections::HashSet::new(); + for decision in &response.decisions { + let node_id = match decision { + BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => { + to_node + } + }; + if !expected.contains(node_id) { + return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); + } + if !seen.insert(node_id.clone()) { + return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); + } + if let BindingDecision::NeedRework { + problem_description, + .. + } = decision + { + if problem_description.trim().is_empty() { + return Err("NeedRework 必须包含问题描述".to_string()); + } + } + } + if seen.len() != expected.len() { + return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); + } + Ok(()) +} + +pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result { + if asset_id.trim().is_empty() || asset_id.trim() != asset_id { + return Err("UI 资源 ID 无效".to_string()); + } + let mut stem = String::new(); + for character in asset_id.chars() { + if character.is_ascii_alphanumeric() || matches!(character, '_' | '-') { + stem.push(character); + } else { + stem.push('_'); + } + } + let digest = format!("{:x}", Sha256::digest(asset_id.as_bytes())); + let dir = root + .join("ui") + .join(format!(".{stem}-{}-separation", &digest[..16])); + if !dir.starts_with(root) { + return Err("separation sidecar 路径越界".to_string()); + } + Ok(dir) +} + +pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<(), String> { + if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + return Err("不支持的 separation state schema".to_string()); + } + let bytes = serde_json::to_vec_pretty(state) + .map_err(|error| format!("序列化 separation state 失败：{error}"))?; + let parent = path + .parent() + .ok_or_else(|| "separation state 路径缺少父目录".to_string())?; + fs::create_dir_all(parent).map_err(|error| format!("创建 separation sidecar 失败：{error}"))?; + let temporary = path.with_extension("json.tmp"); + fs::write(&temporary, bytes).map_err(|error| format!("写入 separation state 失败：{error}"))?; + fs::rename(&temporary, path).map_err(|error| format!("安装 separation state 失败：{error}")) +} + +pub fn read_separation_state(path: &Path) -> Result { + let bytes = fs::read(path).map_err(|error| format!("读取 separation state 失败：{error}"))?; + let state: SeparationState = serde_json::from_slice(&bytes) + .map_err(|error| format!("解析 separation state 失败：{error}"))?; + if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + return Err("不支持的 separation state schema".to_string()); + } + Ok(state) +} + +pub fn separation_dto(state: &SeparationState) -> SeparationDTO { + SeparationDTO { + bound_nodes: state.bound.clone(), + problematic_nodes: state.problematic_nodes.clone(), + } +} + +pub fn apply_batch_patch( + state: &mut SeparationState, + tree_index: usize, + decisions: &[BindingDecision], + cut_paths: &std::collections::HashMap, +) -> Result<(), String> { + let tree = state + .unprocessed_trees + .get_mut(tree_index) + .ok_or_else(|| "separation tree 索引无效".to_string())?; + let batch = next_leaf_batch(tree); + validate_binding_response( + &BindingResp { + decisions: decisions.to_vec(), + }, + &batch, + )?; + let rework_counts = batch + .iter() + .map(|node| (node.id.clone(), node.rework_count)) + .collect::>(); + let mut ids = std::collections::HashSet::new(); + for decision in decisions { + match decision { + BindingDecision::Ok { to_node, .. } => { + let path = cut_paths + .get(to_node) + .ok_or_else(|| format!("缺少节点 {} 的 cut 图片", to_node.as_str()))?; + state.bound.push(BoundNode { + node_id: to_node.clone(), + cut_image_path: path.clone(), + }); + ids.insert(to_node.clone()); + } + BindingDecision::NeedRework { + to_node, + problem_description, + } => { + let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; + if count >= MAX_REWORK_COUNT { + state.problematic_nodes.push(ProblematicNode { + node_id: to_node.clone(), + problem_description: problem_description.clone(), + rework_count: count, + }); + ids.insert(to_node.clone()); + } else { + increment_rework_count(&mut tree.root, to_node, count); + } + } + } + } + remove_ids(&mut tree.root, &ids); + Ok(()) +} + +fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { + if node.id == *id { + node.rework_count = count; + return; + } + for child in &mut node.children { + increment_rework_count(child, id, count); + } +} + +#[derive(Deserialize)] +struct RawEditResponse { + data: Vec, +} +#[derive(Deserialize)] +struct RawEditItem { + b64_json: String, +} + +async fn raw_image_edit( + session: &crate::platform_session::PlatformSessionSnapshot, + image_data_url: &str, + prompt: &str, + width: u32, + height: u32, +) -> Result { + let (mime, data) = image_data_url + .split_once(",") + .ok_or_else(|| "界面图 data URL 无效".to_string())?; + let mime = mime + .strip_prefix("data:") + .and_then(|v| v.strip_suffix(";base64")) + .unwrap_or("image/png"); + let client = crate::http_client::agc_main_site_client_builder() + .build() + .map_err(|e| format!("创建图片编辑客户端失败：{e}"))?; + let url = format!( + "{}/api/raw/v1/images/edit", + session.api_base_url.trim_end_matches('/') + ); + let body = serde_json::json!({ + "image": {"data": data, "mimeType": mime}, + "prompt": prompt, + "width": width, + "height": height, + "output_format": "png", + "background": "transparent" + }); + let response = crate::http_client::with_agc_main_site_marker( + client + .post(url) + .bearer_auth(&session.access_token) + .json(&body), + ) + .send() + .await + .map_err(|e| format!("图片分离请求失败：{e}"))?; + if !response.status().is_success() { + return Err(format!("图片分离请求失败（HTTP {}）", response.status())); + } + let payload = response + .json::() + .await + .map_err(|e| format!("解析图片分离响应失败：{e}"))?; + payload + .data + .into_iter() + .next() + .map(|item| format!("data:image/png;base64,{}", item.b64_json)) + .ok_or_else(|| "图片分离响应没有图像".to_string()) +} + +fn build_marked_image( + source_url: &str, + nodes: &[&SeparationNode], + target: &Path, +) -> Result { + let encoded = source_url + .split_once(',') + .map(|(_, d)| d) + .ok_or_else(|| "源图 data URL 无效".to_string())?; + let bytes = base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|e| format!("解码源图失败：{e}"))?; + let mut image = image::load_from_memory(&bytes) + .map_err(|e| format!("读取源图失败：{e}"))? + .to_rgba8(); + let width = image.width(); + let height = image.height(); + for node in nodes { + let x0 = node.global_pos_x_px.min(width.saturating_sub(1)); + let y0 = node.global_pos_y_px.min(height.saturating_sub(1)); + let x1 = node + .global_pos_x_px + .saturating_add(node.width_px) + .min(width) + .saturating_sub(1); + let y1 = node + .global_pos_y_px + .saturating_add(node.height_px) + .min(height) + .saturating_sub(1); + if x0 >= x1 || y0 >= y1 { + continue; + } + for x in x0..=x1 { + image.put_pixel(x, y0, image::Rgba([0, 255, 0, 255])); + image.put_pixel(x, y1, image::Rgba([0, 255, 0, 255])); + } + for y in y0..=y1 { + image.put_pixel(x0, y, image::Rgba([0, 255, 0, 255])); + image.put_pixel(x1, y, image::Rgba([0, 255, 0, 255])); + } + for y in y0..=y1 { + for x in x0..=x1 { + if x > x0 && x < x1 && y > y0 && y < y1 { + image.put_pixel(x, y, image::Rgba([180, 0, 180, 120])); + } + } + } + } + image::DynamicImage::ImageRgba8(image.clone()) + .save_with_format(target, image::ImageFormat::Png) + .map_err(|e| format!("写入标记图失败：{e}"))?; + let mut png = Vec::new(); + image::DynamicImage::ImageRgba8(image) + .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) + .map_err(|e| format!("编码标记图失败：{e}"))?; + Ok(format!( + "data:image/png;base64,{}", + base64::engine::general_purpose::STANDARD.encode(png) + )) +} + +async fn visual_binding( + source_url: String, + processed_url: String, + nodes: &[&SeparationNode], +) -> Result { + let llm_config = load_game_creator_app_config() + .map_err(|e| e.to_string())? + .llm; + let client = build_game_creator_llm_client_from_llm_config(&llm_config, "llm") + .map_err(|e| e.to_string())?; + let schema = strict_json_schema::()?; + let tool = LlmFunctionTool::new( + "bind_ui_elements", + "确认处理图中的区域对应哪些 UI 节点", + schema, + ) + .with_strict(true); + let base_prompt = gen_binding_prompt(nodes); + let mut repair = None; + for attempt in 0..2 { + let prompt = repair.as_ref().map_or_else( + || base_prompt.clone(), + |error: &String| format!("{base_prompt}\n上一次输出错误：{error}\n请修正并完整返回。"), + ); + let request = LlmRunRequest::new(vec![ + LlmMessage::system("你是 UI 图片视觉绑定器。只根据图像判断区域，不做 OCR。"), + LlmMessage::user_multimodal(vec![ + LlmMessageContentPart::InputText { text: prompt }, + LlmMessageContentPart::InputImage { + image_url: source_url.clone(), + }, + LlmMessageContentPart::InputImage { + image_url: processed_url.clone(), + }, + ]), + ]) + .with_function_tools(vec![tool.clone()]) + .with_tool_choice(LlmToolChoice::Required); + let result = request_ui_editor_llm(&client, &llm_config, request) + .await + .map_err(|e| e.to_string()) + .and_then(|response| { + response + .tool_calls + .into_iter() + .find(|call| call.name == "bind_ui_elements") + .map(|call| call.arguments) + .ok_or_else(|| "视觉绑定模型未返回工具调用".to_string()) + }) + .and_then(|arguments| parse_limited_llm_tool_arguments(&arguments)) + .and_then(|args| { + serde_json::from_value::(args) + .map_err(|e| format!("视觉绑定结果无效：{e}")) + }) + .and_then(|parsed| validate_binding_response(&parsed, nodes).map(|_| parsed)); + match result { + Ok(value) => return Ok(value), + Err(error) if attempt == 0 => repair = Some(error), + Err(error) => return Err(error), + } + } + Err("视觉绑定失败".to_string()) +} + +pub(crate) async fn separate_ui_impl( + project_path: String, + asset_id: String, + state: State, +) -> Result { + let session = current_platform_session().ok_or_else(|| "请先登录平台账号".to_string())?; + let root = Path::new(project_path.trim()); + let sidecar = separation_sidecar_dir(root, &asset_id)?; + fs::create_dir_all(&sidecar).map_err(|e| format!("创建 separation sidecar 失败：{e}"))?; + let state_path = sidecar.join("state.json"); + let mut separation = if state_path.exists() { + read_separation_state(&state_path)? + } else { + construct_separation_state(&state) + }; + for (tree_index, tree) in separation.unprocessed_trees.clone().iter().enumerate() { + let image = state + .ui_design_images + .get(&tree.src_ui_design) + .ok_or_else(|| "缺少源界面图".to_string())?; + let source_path = crate::project::resolve_local_project_path(root, &image.path)?; + let source_url = read_ui_reference_image_data_url(source_path).await?; + write_separation_state(&state_path, &separation)?; + loop { + let Some(current_tree) = separation.unprocessed_trees.get(tree_index) else { + break; + }; + let batch = next_leaf_batch(current_tree); + if batch.is_empty() { + break; + } + let prompt = + gen_extract_prompt(&batch.iter().map(|n| n.note.clone()).collect::>()); + let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); + let marked_url = build_marked_image(&source_url, &batch, &marker_path)?; + let processed_url = raw_image_edit( + &session, + &marked_url, + &prompt, + image.pixel_size.x as u32, + image.pixel_size.y as u32, + ) + .await?; + let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); + let processed_bytes = base64::engine::general_purpose::STANDARD + .decode( + processed_url + .split_once(',') + .map(|(_, d)| d) + .unwrap_or_default(), + ) + .map_err(|e| e.to_string())?; + fs::write(&processed_path, processed_bytes).map_err(|e| e.to_string())?; + let binding = visual_binding(source_url.clone(), processed_url, &batch).await?; + let mut cut_paths = std::collections::HashMap::new(); + for decision in &binding.decisions { + if let BindingDecision::Ok { + to_node, + separated_image_area, + } = decision + { + let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); + cut_processed_image(&processed_path, separated_image_area, &cut_path)?; + cut_paths.insert(to_node.clone(), cut_path.to_string_lossy().to_string()); + } + } + apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; + write_separation_state(&state_path, &separation)?; + } + } + let _ = fs::remove_file(state_path); + Ok(separation_dto(&separation)) +} + +fn cut_processed_image(source: &Path, area: &BindingArea, target: &Path) -> Result<(), String> { + let image = image::open(source).map_err(|e| format!("读取处理图失败：{e}"))?; + area.validate_in(image.width(), image.height())?; + let cropped = image.crop_imm( + area.global_pos_x_px, + area.global_pos_y_px, + area.width_px, + area.height_px, + ); + cropped + .save_with_format(target, ImageFormat::Png) + .map_err(|e| format!("写入 cut 图片失败：{e}")) +} + +fn remove_ids(node: &mut SeparationNode, ids: &std::collections::HashSet) { + node.children.retain(|child| !ids.contains(&child.id)); + for child in &mut node.children { + remove_ids(child, ids); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ui_editor::component::image::{ImageComponent, ImageType}; + use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; + use crate::ui_editor::layout::control_layout::ControlLayout; + use crate::ui_editor::layout::node::{NodeMetadata, NodeSource, StageStatus}; + use crate::ui_editor::resource::ui_design_image::UIDesignImage; + use nalgebra::Vector2; + use std::collections::HashMap; + use typed_floats::tf32::StrictlyPositiveFinite; + + fn node(id: &str, components: Vec, children: Vec) -> Node { + Node { + id: NodeId::new(id).unwrap(), + layout: ControlLayout::default(), + metadata: NodeMetadata { + name: id.to_string(), + description: String::new(), + layout_status: StageStatus::NoProblem, + components_status: StageStatus::NoProblem, + allow_llm_edit_layout: true, + allow_llm_edit_component: true, + source: NodeSource::Llm, + }, + components, + children_display_mode: ChildrenDisplayMode::Stack, + children, + } + } + fn state(root: Node) -> State { + let image_id = UIDesignImageId::new("page").unwrap(); + State { + ui_trees: vec![UITree { + src_ui_design: image_id.clone(), + root, + }], + ui_design_images: HashMap::from([( + image_id, + UIDesignImage { + metadata: crate::ui_editor::resource::ui_design_image::UIDesignImageMetadata { + name: "page".to_string(), + description: String::new(), + role: None, + slave_to: None, + }, + path: "page.png".to_string(), + pixel_size: Vector2::new(100.0, 100.0), + pixels_per_unit: StrictlyPositiveFinite::new(1.0).unwrap(), + }, + )]), + sprite_assets: HashMap::new(), + font_assets: HashMap::new(), + } + } + #[test] + fn construction_filters_pure_nodes_and_passes_children_through() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let root = node( + "root", + vec![], + vec![node( + "container", + vec![], + vec![node("image", vec![image], vec![])], + )], + ); + let result = construct_separation_state(&state(root)); + assert_eq!( + result.unprocessed_trees[0].root.children[0].id.as_str(), + "image" + ); + } + #[test] + fn binding_validation_requires_exact_batch_coverage() { + let node = SeparationNode { + id: NodeId::new("image").unwrap(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + note: SeparationNote { + description: "image".to_string(), + }, + children: vec![], + rework_count: 0, + }; + assert!(validate_binding_response(&BindingResp { decisions: vec![] }, &[&node]).is_err()); + } + #[test] + fn sidecar_name_uses_asset_id_digest() { + let dir = separation_sidecar_dir(Path::new("/tmp/project"), "ui:1").unwrap(); + assert!(dir.to_string_lossy().contains("ui_1-")); + assert!(dir.to_string_lossy().ends_with("-separation")); + } + #[test] + fn patch_collects_bound_and_removes_leaf() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let mut state = construct_separation_state(&state(node( + "root", + vec![], + vec![node("image", vec![image], vec![])], + ))); + let id = NodeId::new("image").unwrap(); + let decisions = vec![BindingDecision::Ok { + to_node: id.clone(), + separated_image_area: BindingArea { + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + }, + }]; + let paths = HashMap::from([(id.clone(), "ui/.sidecar/cut.png".to_string())]); + apply_batch_patch(&mut state, 0, &decisions, &paths).unwrap(); + assert_eq!(state.bound[0].node_id, id); + assert!(state.unprocessed_trees[0].root.children.is_empty()); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs index 04b72c0b3..2384476b0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs @@ -196,7 +196,7 @@ pub(crate) fn generate_ui_design_code_at( }) } -fn generated_file_stem(asset_id: &str) -> String { +pub(crate) fn generated_file_stem(asset_id: &str) -> String { let mut stem = String::new(); for character in asset_id.chars() { if character.is_ascii_alphanumeric() || matches!(character, '_' | '-') { From f796aef7e23f00e2821ce22eb421862199e1024e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 18:32:36 +0800 Subject: [PATCH 038/248] =?UTF-8?q?=E5=AE=8C=E5=96=84UI=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=89=B9=E6=AC=A1=E5=A4=84=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复用资源ID安全路径并修正像素坐标 加入标记图、Raw编辑、视觉绑定修复与问题节点继续处理 --- .../src/ui_editor/commands/separation.rs | 77 ++++++++++++++----- 1 file changed, 58 insertions(+), 19 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs index d1217aefa..fb65a02c5 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs @@ -16,7 +16,6 @@ use platform_llm::{ }; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; -use sha2::{Digest, Sha256}; use std::fs; use std::path::{Path, PathBuf}; use ts_rs::TS; @@ -331,18 +330,10 @@ pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result>(); + for node in batch { + state.problematic_nodes.push(ProblematicNode { + node_id: node.id.clone(), + problem_description: error.clone(), + rework_count: MAX_REWORK_COUNT, + }); + } + if let Some(tree) = state.unprocessed_trees.get_mut(tree_index) { + remove_ids(&mut tree.root, &ids); + } +} + fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { if node.id == *id { node.rework_count = count; @@ -662,22 +675,41 @@ pub(crate) async fn separate_ui_impl( let Some(current_tree) = separation.unprocessed_trees.get(tree_index) else { break; }; - let batch = next_leaf_batch(current_tree); - if batch.is_empty() { + let batch_nodes = next_leaf_batch(current_tree) + .into_iter() + .cloned() + .collect::>(); + if batch_nodes.is_empty() { break; } + let batch = batch_nodes.iter().collect::>(); let prompt = gen_extract_prompt(&batch.iter().map(|n| n.note.clone()).collect::>()); let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); - let marked_url = build_marked_image(&source_url, &batch, &marker_path)?; - let processed_url = raw_image_edit( + let marked_url = match build_marked_image(&source_url, &batch, &marker_path) { + Ok(value) => value, + Err(error) => { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } + }; + let processed_url = match raw_image_edit( &session, &marked_url, &prompt, image.pixel_size.x as u32, image.pixel_size.y as u32, ) - .await?; + .await + { + Ok(value) => value, + Err(error) => { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } + }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); let processed_bytes = base64::engine::general_purpose::STANDARD .decode( @@ -688,7 +720,14 @@ pub(crate) async fn separate_ui_impl( ) .map_err(|e| e.to_string())?; fs::write(&processed_path, processed_bytes).map_err(|e| e.to_string())?; - let binding = visual_binding(source_url.clone(), processed_url, &batch).await?; + let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { + Ok(value) => value, + Err(error) => { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } + }; let mut cut_paths = std::collections::HashMap::new(); for decision in &binding.decisions { if let BindingDecision::Ok { From 109cc81740b33cfe01db35068a7d506e097e1904 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 23:44:04 +0800 Subject: [PATCH 039/248] =?UTF-8?q?=E6=8B=86=E5=88=86UI=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E5=91=BD=E4=BB=A4=E6=A8=A1=E5=9D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 保留原始图片分离与绑定提示词 提取模型、树、持久化和工作流子模块 新增可配置 more_turn 反馈重试 harness --- .../src/ui_editor/commands/separation.rs | 900 ------------------ .../src/ui_editor/commands/separation/mod.rs | 140 +++ .../ui_editor/commands/separation/model.rs | 123 +++ .../commands/separation/persistence.rs | 49 + .../ui_editor/commands/separation/prompt.rs | 52 + .../src/ui_editor/commands/separation/tree.rs | 158 +++ .../ui_editor/commands/separation/workflow.rs | 420 ++++++++ .../src-tauri/src/ui_editor/commands/utils.rs | 49 + 8 files changed, 991 insertions(+), 900 deletions(-) delete mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs deleted file mode 100644 index fb65a02c5..000000000 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation.rs +++ /dev/null @@ -1,900 +0,0 @@ -use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; -use crate::platform_session::current_platform_session; -use crate::ui_editor::commands::utils::{ - parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, - strict_json_schema, -}; -use crate::ui_editor::component::image::ImageComponent; -use crate::ui_editor::component::Component; -use crate::ui_editor::layout::node::Node; -use crate::ui_editor::state::{State, UITree}; -use crate::ui_editor::utils::{NodeId, UIDesignImageId}; -use base64::Engine as _; -use image::ImageFormat; -use platform_llm::{ - LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice, -}; -use schemars::JsonSchema; -use serde::{Deserialize, Serialize}; -use std::fs; -use std::path::{Path, PathBuf}; -use ts_rs::TS; - -pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; -pub const MAX_REWORK_COUNT: u32 = 3; - -#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationNote { - pub description: String, -} - -impl SeparationNote { - pub fn as_prompt(&self) -> String { - format!("- {}", self.description.trim()) - } -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationNode { - pub id: NodeId, - pub global_pos_x_px: u32, - pub global_pos_y_px: u32, - pub width_px: u32, - pub height_px: u32, - pub note: SeparationNote, - pub children: Vec, - pub rework_count: u32, -} - -impl SeparationNode { - pub fn as_prompt(&self) -> String { - format!( - "node_id={} area=({}, {}, {}, {}) {}", - self.id.as_str(), - self.global_pos_x_px, - self.global_pos_y_px, - self.width_px, - self.height_px, - self.note.as_prompt() - ) - } -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationTree { - pub src_ui_design: UIDesignImageId, - pub root: SeparationNode, -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct BoundNode { - pub node_id: NodeId, - pub cut_image_path: String, -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct ProblematicNode { - pub node_id: NodeId, - pub problem_description: String, - pub rework_count: u32, -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationState { - pub schema_version: String, - pub unprocessed_trees: Vec, - pub bound: Vec, - pub problematic_nodes: Vec, -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationDTO { - pub bound_nodes: Vec, - pub problematic_nodes: Vec, -} - -#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] -#[schemars(deny_unknown_fields)] -pub struct BindingArea { - pub global_pos_x_px: u32, - pub global_pos_y_px: u32, - pub width_px: u32, - pub height_px: u32, -} - -impl BindingArea { - pub fn validate_in(&self, image_width: u32, image_height: u32) -> Result<(), String> { - if self.width_px == 0 || self.height_px == 0 { - return Err("BindingArea 宽度和高度必须大于 0".to_string()); - } - let max_x = self - .global_pos_x_px - .checked_add(self.width_px) - .ok_or_else(|| "BindingArea 横向范围溢出".to_string())?; - let max_y = self - .global_pos_y_px - .checked_add(self.height_px) - .ok_or_else(|| "BindingArea 纵向范围溢出".to_string())?; - if max_x > image_width || max_y > image_height { - return Err("BindingArea 超出处理图边界".to_string()); - } - Ok(()) - } -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] -#[schemars(deny_unknown_fields)] -pub enum BindingDecision { - Ok { - separated_image_area: BindingArea, - to_node: NodeId, - }, - NeedRework { - problem_description: String, - to_node: NodeId, - }, -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] -#[schemars(deny_unknown_fields)] -pub struct BindingResp { - pub decisions: Vec, -} - -const SHARED_SEPARATION_REQ: &str = r#" -MUST hard edges; preserve no glow/blur beyond the exact visible shape. -NEVER keep its parent's background with it. -UI elements marked with GREEN line frames are extraction marks only; never include the frame. -PURPLE filled areas represent removed elements; reconstruct the background under them. -"#; - -pub fn gen_extract_prompt(separation_notes: &[SeparationNote]) -> String { - let mut result = format!("This is a UI design image, not a normal photo/illustration.\nExtract distinct UI elements as independent layers with clean edges and full transparency outside each element.\nKeep every element at its original position on a transparent canvas.\n{}\nElements to extract:\n", SHARED_SEPARATION_REQ); - for note in separation_notes { - result.push_str(¬e.as_prompt()); - result.push('\n'); - } - result -} - -pub fn gen_binding_prompt(nodes: &[&SeparationNode]) -> String { - let mut result = format!("You are reviewing a UI elements separation result.\n{}\nThe source and processed images use top-left pixel coordinates.\nReturn one decision for every requested node. For a failed node, provide a short repair description.\nNodes:\n", SHARED_SEPARATION_REQ); - for node in nodes { - result.push_str(&node.as_prompt()); - result.push('\n'); - } - result -} - -fn is_unbound_image(node: &Node) -> bool { - node.components.iter().any(|component| { - matches!( - component, - Component::Image(ImageComponent { - target_graphic: None, - .. - }) - ) - }) -} - -fn node_pixel_rect( - node: &Node, - parent: &crate::ui_editor::layout::dimension::UIRect, - ppu: f32, -) -> (u32, u32, u32, u32) { - let rect = node.layout.transform.resolve(parent); - let x = (rect.min.x * ppu).max(0.0).round() as u32; - let y = (rect.min.y * ppu).max(0.0).round() as u32; - let w = (rect.size.x * ppu).max(0.0).round() as u32; - let h = (rect.size.y * ppu).max(0.0).round() as u32; - (x, y, w, h) -} - -fn node_description(node: &Node) -> String { - let name = node.metadata.name.trim(); - let description = node.metadata.description.trim(); - match (name.is_empty(), description.is_empty()) { - (true, true) => "未命名 UI 图片元素".to_string(), - (false, true) => name.to_string(), - (true, false) => description.to_string(), - (false, false) => format!("{name}：{description}"), - } -} - -fn collect_todo_nodes( - node: &Node, - parent: &crate::ui_editor::layout::dimension::UIRect, - ppu: f32, - output: &mut Vec, -) { - let mut children = Vec::new(); - let rect = node.layout.transform.resolve(parent); - for child in &node.children { - collect_todo_nodes(child, &rect, ppu, &mut children); - } - if is_unbound_image(node) { - let (x, y, w, h) = node_pixel_rect(node, parent, ppu); - output.push(SeparationNode { - id: node.id.clone(), - global_pos_x_px: x, - global_pos_y_px: y, - width_px: w, - height_px: h, - note: SeparationNote { - description: node_description(node), - }, - children, - rework_count: 0, - }); - } else { - output.extend(children); - } -} - -pub fn construct_separation_state(state: &State) -> SeparationState { - let unprocessed_trees = state - .ui_trees - .iter() - .filter_map(|tree| { - let image = state.ui_design_images.get(&tree.src_ui_design)?; - let ppu = image.pixels_per_unit.get(); - let size = image.pixel_size / ppu; - let root_rect = - crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); - let mut children = Vec::new(); - collect_todo_nodes(&tree.root, &root_rect, ppu, &mut children); - (!children.is_empty()).then(|| SeparationTree { - src_ui_design: tree.src_ui_design.clone(), - root: SeparationNode { - id: tree.root.id.clone(), - global_pos_x_px: 0, - global_pos_y_px: 0, - width_px: image.pixel_size.x.max(0.0).round() as u32, - height_px: image.pixel_size.y.max(0.0).round() as u32, - note: SeparationNote::default(), - children, - rework_count: 0, - }, - }) - }) - .collect(); - SeparationState { - schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), - unprocessed_trees, - bound: Vec::new(), - problematic_nodes: Vec::new(), - } -} - -pub fn next_leaf_batch(tree: &SeparationTree) -> Vec<&SeparationNode> { - fn leaves<'a>(node: &'a SeparationNode, output: &mut Vec<&'a SeparationNode>) { - if node.children.is_empty() { - output.push(node); - } else { - for child in &node.children { - leaves(child, output); - } - } - } - let mut output = Vec::new(); - leaves(&tree.root, &mut output); - output -} - -pub fn validate_binding_response( - response: &BindingResp, - batch: &[&SeparationNode], -) -> Result<(), String> { - let expected = batch - .iter() - .map(|node| node.id.clone()) - .collect::>(); - let mut seen = std::collections::HashSet::new(); - for decision in &response.decisions { - let node_id = match decision { - BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => { - to_node - } - }; - if !expected.contains(node_id) { - return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); - } - if !seen.insert(node_id.clone()) { - return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); - } - if let BindingDecision::NeedRework { - problem_description, - .. - } = decision - { - if problem_description.trim().is_empty() { - return Err("NeedRework 必须包含问题描述".to_string()); - } - } - } - if seen.len() != expected.len() { - return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); - } - Ok(()) -} - -pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result { - if asset_id.trim().is_empty() || asset_id.trim() != asset_id { - return Err("UI 资源 ID 无效".to_string()); - } - let dir = root.join("ui").join(format!( - ".{}-separation", - crate::ui_editor::persistence::generated_file_stem(asset_id) - )); - if !dir.starts_with(root) { - return Err("separation sidecar 路径越界".to_string()); - } - Ok(dir) -} - -pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<(), String> { - if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { - return Err("不支持的 separation state schema".to_string()); - } - let bytes = serde_json::to_vec_pretty(state) - .map_err(|error| format!("序列化 separation state 失败：{error}"))?; - let parent = path - .parent() - .ok_or_else(|| "separation state 路径缺少父目录".to_string())?; - fs::create_dir_all(parent).map_err(|error| format!("创建 separation sidecar 失败：{error}"))?; - let temporary = path.with_extension("json.tmp"); - fs::write(&temporary, bytes).map_err(|error| format!("写入 separation state 失败：{error}"))?; - fs::rename(&temporary, path).map_err(|error| format!("安装 separation state 失败：{error}")) -} - -pub fn read_separation_state(path: &Path) -> Result { - let bytes = fs::read(path).map_err(|error| format!("读取 separation state 失败：{error}"))?; - let state: SeparationState = serde_json::from_slice(&bytes) - .map_err(|error| format!("解析 separation state 失败：{error}"))?; - if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { - return Err("不支持的 separation state schema".to_string()); - } - Ok(state) -} - -pub fn separation_dto(state: &SeparationState) -> SeparationDTO { - SeparationDTO { - bound_nodes: state.bound.clone(), - problematic_nodes: state.problematic_nodes.clone(), - } -} - -pub fn apply_batch_patch( - state: &mut SeparationState, - tree_index: usize, - decisions: &[BindingDecision], - cut_paths: &std::collections::HashMap, -) -> Result<(), String> { - let tree = state - .unprocessed_trees - .get_mut(tree_index) - .ok_or_else(|| "separation tree 索引无效".to_string())?; - let batch = next_leaf_batch(tree); - validate_binding_response( - &BindingResp { - decisions: decisions.to_vec(), - }, - &batch, - )?; - let rework_counts = batch - .iter() - .map(|node| (node.id.clone(), node.rework_count)) - .collect::>(); - let mut ids = std::collections::HashSet::new(); - for decision in decisions { - match decision { - BindingDecision::Ok { to_node, .. } => { - let path = cut_paths - .get(to_node) - .ok_or_else(|| format!("缺少节点 {} 的 cut 图片", to_node.as_str()))?; - state.bound.push(BoundNode { - node_id: to_node.clone(), - cut_image_path: path.clone(), - }); - ids.insert(to_node.clone()); - } - BindingDecision::NeedRework { - to_node, - problem_description, - } => { - let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; - if count >= MAX_REWORK_COUNT { - state.problematic_nodes.push(ProblematicNode { - node_id: to_node.clone(), - problem_description: problem_description.clone(), - rework_count: count, - }); - ids.insert(to_node.clone()); - } else { - increment_rework_count(&mut tree.root, to_node, count); - } - } - } - } - remove_ids(&mut tree.root, &ids); - Ok(()) -} - -fn mark_batch_problematic( - state: &mut SeparationState, - tree_index: usize, - batch: &[&SeparationNode], - error: String, -) { - let ids = batch - .iter() - .map(|node| node.id.clone()) - .collect::>(); - for node in batch { - state.problematic_nodes.push(ProblematicNode { - node_id: node.id.clone(), - problem_description: error.clone(), - rework_count: MAX_REWORK_COUNT, - }); - } - if let Some(tree) = state.unprocessed_trees.get_mut(tree_index) { - remove_ids(&mut tree.root, &ids); - } -} - -fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { - if node.id == *id { - node.rework_count = count; - return; - } - for child in &mut node.children { - increment_rework_count(child, id, count); - } -} - -#[derive(Deserialize)] -struct RawEditResponse { - data: Vec, -} -#[derive(Deserialize)] -struct RawEditItem { - b64_json: String, -} - -async fn raw_image_edit( - session: &crate::platform_session::PlatformSessionSnapshot, - image_data_url: &str, - prompt: &str, - width: u32, - height: u32, -) -> Result { - let (mime, data) = image_data_url - .split_once(",") - .ok_or_else(|| "界面图 data URL 无效".to_string())?; - let mime = mime - .strip_prefix("data:") - .and_then(|v| v.strip_suffix(";base64")) - .unwrap_or("image/png"); - let client = crate::http_client::agc_main_site_client_builder() - .build() - .map_err(|e| format!("创建图片编辑客户端失败：{e}"))?; - let url = format!( - "{}/api/raw/v1/images/edit", - session.api_base_url.trim_end_matches('/') - ); - let body = serde_json::json!({ - "image": {"data": data, "mimeType": mime}, - "prompt": prompt, - "width": width, - "height": height, - "output_format": "png", - "background": "transparent" - }); - let response = crate::http_client::with_agc_main_site_marker( - client - .post(url) - .bearer_auth(&session.access_token) - .json(&body), - ) - .send() - .await - .map_err(|e| format!("图片分离请求失败：{e}"))?; - if !response.status().is_success() { - return Err(format!("图片分离请求失败（HTTP {}）", response.status())); - } - let payload = response - .json::() - .await - .map_err(|e| format!("解析图片分离响应失败：{e}"))?; - payload - .data - .into_iter() - .next() - .map(|item| format!("data:image/png;base64,{}", item.b64_json)) - .ok_or_else(|| "图片分离响应没有图像".to_string()) -} - -fn build_marked_image( - source_url: &str, - nodes: &[&SeparationNode], - target: &Path, -) -> Result { - let encoded = source_url - .split_once(',') - .map(|(_, d)| d) - .ok_or_else(|| "源图 data URL 无效".to_string())?; - let bytes = base64::engine::general_purpose::STANDARD - .decode(encoded) - .map_err(|e| format!("解码源图失败：{e}"))?; - let mut image = image::load_from_memory(&bytes) - .map_err(|e| format!("读取源图失败：{e}"))? - .to_rgba8(); - let width = image.width(); - let height = image.height(); - for node in nodes { - let x0 = node.global_pos_x_px.min(width.saturating_sub(1)); - let y0 = node.global_pos_y_px.min(height.saturating_sub(1)); - let x1 = node - .global_pos_x_px - .saturating_add(node.width_px) - .min(width) - .saturating_sub(1); - let y1 = node - .global_pos_y_px - .saturating_add(node.height_px) - .min(height) - .saturating_sub(1); - if x0 >= x1 || y0 >= y1 { - continue; - } - for x in x0..=x1 { - image.put_pixel(x, y0, image::Rgba([0, 255, 0, 255])); - image.put_pixel(x, y1, image::Rgba([0, 255, 0, 255])); - } - for y in y0..=y1 { - image.put_pixel(x0, y, image::Rgba([0, 255, 0, 255])); - image.put_pixel(x1, y, image::Rgba([0, 255, 0, 255])); - } - for y in y0..=y1 { - for x in x0..=x1 { - if x > x0 && x < x1 && y > y0 && y < y1 { - image.put_pixel(x, y, image::Rgba([180, 0, 180, 120])); - } - } - } - } - image::DynamicImage::ImageRgba8(image.clone()) - .save_with_format(target, image::ImageFormat::Png) - .map_err(|e| format!("写入标记图失败：{e}"))?; - let mut png = Vec::new(); - image::DynamicImage::ImageRgba8(image) - .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) - .map_err(|e| format!("编码标记图失败：{e}"))?; - Ok(format!( - "data:image/png;base64,{}", - base64::engine::general_purpose::STANDARD.encode(png) - )) -} - -async fn visual_binding( - source_url: String, - processed_url: String, - nodes: &[&SeparationNode], -) -> Result { - let llm_config = load_game_creator_app_config() - .map_err(|e| e.to_string())? - .llm; - let client = build_game_creator_llm_client_from_llm_config(&llm_config, "llm") - .map_err(|e| e.to_string())?; - let schema = strict_json_schema::()?; - let tool = LlmFunctionTool::new( - "bind_ui_elements", - "确认处理图中的区域对应哪些 UI 节点", - schema, - ) - .with_strict(true); - let base_prompt = gen_binding_prompt(nodes); - let mut repair = None; - for attempt in 0..2 { - let prompt = repair.as_ref().map_or_else( - || base_prompt.clone(), - |error: &String| format!("{base_prompt}\n上一次输出错误：{error}\n请修正并完整返回。"), - ); - let request = LlmRunRequest::new(vec![ - LlmMessage::system("你是 UI 图片视觉绑定器。只根据图像判断区域，不做 OCR。"), - LlmMessage::user_multimodal(vec![ - LlmMessageContentPart::InputText { text: prompt }, - LlmMessageContentPart::InputImage { - image_url: source_url.clone(), - }, - LlmMessageContentPart::InputImage { - image_url: processed_url.clone(), - }, - ]), - ]) - .with_function_tools(vec![tool.clone()]) - .with_tool_choice(LlmToolChoice::Required); - let result = request_ui_editor_llm(&client, &llm_config, request) - .await - .map_err(|e| e.to_string()) - .and_then(|response| { - response - .tool_calls - .into_iter() - .find(|call| call.name == "bind_ui_elements") - .map(|call| call.arguments) - .ok_or_else(|| "视觉绑定模型未返回工具调用".to_string()) - }) - .and_then(|arguments| parse_limited_llm_tool_arguments(&arguments)) - .and_then(|args| { - serde_json::from_value::(args) - .map_err(|e| format!("视觉绑定结果无效：{e}")) - }) - .and_then(|parsed| validate_binding_response(&parsed, nodes).map(|_| parsed)); - match result { - Ok(value) => return Ok(value), - Err(error) if attempt == 0 => repair = Some(error), - Err(error) => return Err(error), - } - } - Err("视觉绑定失败".to_string()) -} - -pub(crate) async fn separate_ui_impl( - project_path: String, - asset_id: String, - state: State, -) -> Result { - let session = current_platform_session().ok_or_else(|| "请先登录平台账号".to_string())?; - let root = Path::new(project_path.trim()); - let sidecar = separation_sidecar_dir(root, &asset_id)?; - fs::create_dir_all(&sidecar).map_err(|e| format!("创建 separation sidecar 失败：{e}"))?; - let state_path = sidecar.join("state.json"); - let mut separation = if state_path.exists() { - read_separation_state(&state_path)? - } else { - construct_separation_state(&state) - }; - for (tree_index, tree) in separation.unprocessed_trees.clone().iter().enumerate() { - let image = state - .ui_design_images - .get(&tree.src_ui_design) - .ok_or_else(|| "缺少源界面图".to_string())?; - let source_path = crate::project::resolve_local_project_path(root, &image.path)?; - let source_url = read_ui_reference_image_data_url(source_path).await?; - write_separation_state(&state_path, &separation)?; - loop { - let Some(current_tree) = separation.unprocessed_trees.get(tree_index) else { - break; - }; - let batch_nodes = next_leaf_batch(current_tree) - .into_iter() - .cloned() - .collect::>(); - if batch_nodes.is_empty() { - break; - } - let batch = batch_nodes.iter().collect::>(); - let prompt = - gen_extract_prompt(&batch.iter().map(|n| n.note.clone()).collect::>()); - let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); - let marked_url = match build_marked_image(&source_url, &batch, &marker_path) { - Ok(value) => value, - Err(error) => { - mark_batch_problematic(&mut separation, tree_index, &batch, error); - write_separation_state(&state_path, &separation)?; - continue; - } - }; - let processed_url = match raw_image_edit( - &session, - &marked_url, - &prompt, - image.pixel_size.x as u32, - image.pixel_size.y as u32, - ) - .await - { - Ok(value) => value, - Err(error) => { - mark_batch_problematic(&mut separation, tree_index, &batch, error); - write_separation_state(&state_path, &separation)?; - continue; - } - }; - let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); - let processed_bytes = base64::engine::general_purpose::STANDARD - .decode( - processed_url - .split_once(',') - .map(|(_, d)| d) - .unwrap_or_default(), - ) - .map_err(|e| e.to_string())?; - fs::write(&processed_path, processed_bytes).map_err(|e| e.to_string())?; - let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { - Ok(value) => value, - Err(error) => { - mark_batch_problematic(&mut separation, tree_index, &batch, error); - write_separation_state(&state_path, &separation)?; - continue; - } - }; - let mut cut_paths = std::collections::HashMap::new(); - for decision in &binding.decisions { - if let BindingDecision::Ok { - to_node, - separated_image_area, - } = decision - { - let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); - cut_processed_image(&processed_path, separated_image_area, &cut_path)?; - cut_paths.insert(to_node.clone(), cut_path.to_string_lossy().to_string()); - } - } - apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; - write_separation_state(&state_path, &separation)?; - } - } - let _ = fs::remove_file(state_path); - Ok(separation_dto(&separation)) -} - -fn cut_processed_image(source: &Path, area: &BindingArea, target: &Path) -> Result<(), String> { - let image = image::open(source).map_err(|e| format!("读取处理图失败：{e}"))?; - area.validate_in(image.width(), image.height())?; - let cropped = image.crop_imm( - area.global_pos_x_px, - area.global_pos_y_px, - area.width_px, - area.height_px, - ); - cropped - .save_with_format(target, ImageFormat::Png) - .map_err(|e| format!("写入 cut 图片失败：{e}")) -} - -fn remove_ids(node: &mut SeparationNode, ids: &std::collections::HashSet) { - node.children.retain(|child| !ids.contains(&child.id)); - for child in &mut node.children { - remove_ids(child, ids); - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::ui_editor::component::image::{ImageComponent, ImageType}; - use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; - use crate::ui_editor::layout::control_layout::ControlLayout; - use crate::ui_editor::layout::node::{NodeMetadata, NodeSource, StageStatus}; - use crate::ui_editor::resource::ui_design_image::UIDesignImage; - use nalgebra::Vector2; - use std::collections::HashMap; - use typed_floats::tf32::StrictlyPositiveFinite; - - fn node(id: &str, components: Vec, children: Vec) -> Node { - Node { - id: NodeId::new(id).unwrap(), - layout: ControlLayout::default(), - metadata: NodeMetadata { - name: id.to_string(), - description: String::new(), - layout_status: StageStatus::NoProblem, - components_status: StageStatus::NoProblem, - allow_llm_edit_layout: true, - allow_llm_edit_component: true, - source: NodeSource::Llm, - }, - components, - children_display_mode: ChildrenDisplayMode::Stack, - children, - } - } - fn state(root: Node) -> State { - let image_id = UIDesignImageId::new("page").unwrap(); - State { - ui_trees: vec![UITree { - src_ui_design: image_id.clone(), - root, - }], - ui_design_images: HashMap::from([( - image_id, - UIDesignImage { - metadata: crate::ui_editor::resource::ui_design_image::UIDesignImageMetadata { - name: "page".to_string(), - description: String::new(), - role: None, - slave_to: None, - }, - path: "page.png".to_string(), - pixel_size: Vector2::new(100.0, 100.0), - pixels_per_unit: StrictlyPositiveFinite::new(1.0).unwrap(), - }, - )]), - sprite_assets: HashMap::new(), - font_assets: HashMap::new(), - } - } - #[test] - fn construction_filters_pure_nodes_and_passes_children_through() { - let image = Component::Image(ImageComponent { - target_graphic: None, - image_type: ImageType::Simple { - preserve_aspect: false, - }, - }); - let root = node( - "root", - vec![], - vec![node( - "container", - vec![], - vec![node("image", vec![image], vec![])], - )], - ); - let result = construct_separation_state(&state(root)); - assert_eq!( - result.unprocessed_trees[0].root.children[0].id.as_str(), - "image" - ); - } - #[test] - fn binding_validation_requires_exact_batch_coverage() { - let node = SeparationNode { - id: NodeId::new("image").unwrap(), - global_pos_x_px: 0, - global_pos_y_px: 0, - width_px: 1, - height_px: 1, - note: SeparationNote { - description: "image".to_string(), - }, - children: vec![], - rework_count: 0, - }; - assert!(validate_binding_response(&BindingResp { decisions: vec![] }, &[&node]).is_err()); - } - #[test] - fn sidecar_name_uses_asset_id_digest() { - let dir = separation_sidecar_dir(Path::new("/tmp/project"), "ui:1").unwrap(); - assert!(dir.to_string_lossy().contains("ui_1-")); - assert!(dir.to_string_lossy().ends_with("-separation")); - } - #[test] - fn patch_collects_bound_and_removes_leaf() { - let image = Component::Image(ImageComponent { - target_graphic: None, - image_type: ImageType::Simple { - preserve_aspect: false, - }, - }); - let mut state = construct_separation_state(&state(node( - "root", - vec![], - vec![node("image", vec![image], vec![])], - ))); - let id = NodeId::new("image").unwrap(); - let decisions = vec![BindingDecision::Ok { - to_node: id.clone(), - separated_image_area: BindingArea { - global_pos_x_px: 0, - global_pos_y_px: 0, - width_px: 1, - height_px: 1, - }, - }]; - let paths = HashMap::from([(id.clone(), "ui/.sidecar/cut.png".to_string())]); - apply_batch_patch(&mut state, 0, &decisions, &paths).unwrap(); - assert_eq!(state.bound[0].node_id, id); - assert!(state.unprocessed_trees[0].root.children.is_empty()); - } -} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs new file mode 100644 index 000000000..d103da6e1 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -0,0 +1,140 @@ +mod model; +mod persistence; +mod prompt; +mod tree; +mod workflow; + +pub use model::*; +pub use persistence::*; +pub use tree::*; +pub(crate) use workflow::separate_ui_impl; +#[cfg(test)] +mod tests { + use super::*; + use crate::ui_editor::component::image::{ImageComponent, ImageType}; + use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; + use crate::ui_editor::layout::control_layout::ControlLayout; + use crate::ui_editor::layout::node::{NodeMetadata, NodeSource, StageStatus}; + use crate::ui_editor::resource::ui_design_image::UIDesignImage; + use nalgebra::Vector2; + use std::collections::HashMap; + use typed_floats::tf32::StrictlyPositiveFinite; + + fn node(id: &str, components: Vec, children: Vec) -> Node { + Node { + id: NodeId::new(id).unwrap(), + layout: ControlLayout::default(), + metadata: NodeMetadata { + name: id.to_string(), + description: String::new(), + layout_status: StageStatus::NoProblem, + components_status: StageStatus::NoProblem, + allow_llm_edit_layout: true, + allow_llm_edit_component: true, + source: NodeSource::Llm, + }, + components, + children_display_mode: ChildrenDisplayMode::Stack, + children, + } + } + fn state(root: Node) -> State { + let image_id = UIDesignImageId::new("page").unwrap(); + State { + ui_trees: vec![UITree { + src_ui_design: image_id.clone(), + root, + }], + ui_design_images: HashMap::from([( + image_id, + UIDesignImage { + metadata: crate::ui_editor::resource::ui_design_image::UIDesignImageMetadata { + name: "page".to_string(), + description: String::new(), + role: None, + slave_to: None, + }, + path: "page.png".to_string(), + pixel_size: Vector2::new(100.0, 100.0), + pixels_per_unit: StrictlyPositiveFinite::new(1.0).unwrap(), + }, + )]), + sprite_assets: HashMap::new(), + font_assets: HashMap::new(), + } + } + #[test] + fn construction_filters_pure_nodes_and_passes_children_through() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let root = node( + "root", + vec![], + vec![node( + "container", + vec![], + vec![node("image", vec![image], vec![])], + )], + ); + let result = construct_separation_state(&state(root)); + assert_eq!( + result.unprocessed_trees[0].root.children[0].id.as_str(), + "image" + ); + } + #[test] + fn binding_validation_requires_exact_batch_coverage() { + let node = SeparationNode { + id: NodeId::new("image").unwrap(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + note: SeparationNote { + description: "image".to_string(), + text_note: String::new(), + }, + children: vec![], + rework_count: 0, + }; + assert!(validate_binding_response(&BindingResp { decisions: vec![] }, &[&node]).is_err()); + } + #[test] + fn sidecar_name_uses_asset_id_digest() { + let dir = separation_sidecar_dir(Path::new("/tmp/project"), "ui:1").unwrap(); + assert!(dir.to_string_lossy().contains("ui_1-")); + assert!(dir.to_string_lossy().ends_with("-separation")); + } + #[test] + fn patch_collects_bound_and_removes_leaf() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let mut state = construct_separation_state(&state(node( + "root", + vec![], + vec![node("image", vec![image], vec![])], + ))); + let id = NodeId::new("image").unwrap(); + let decisions = vec![BindingDecision::Ok { + to_node: id.clone(), + separated_image_area: BindingArea { + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + }, + }]; + let paths = HashMap::from([(id.clone(), "ui/.sidecar/cut.png".to_string())]); + apply_batch_patch(&mut state, 0, &decisions, &paths).unwrap(); + assert_eq!(state.bound[0].node_id, id); + assert!(state.unprocessed_trees[0].root.children.is_empty()); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs new file mode 100644 index 000000000..87ffc1832 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs @@ -0,0 +1,123 @@ +use crate::ui_editor::utils::{NodeId, UIDesignImageId}; +use schemars::JsonSchema; +use serde::{Deserialize, Serialize}; +use ts_rs::TS; + +pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; +pub const MAX_REWORK_COUNT: u32 = 3; + +#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationNote { + pub description: String, + pub text_note: String, +} +impl SeparationNote { + pub fn as_prompt(&self) -> String { + format!("desc: {} {}", self.description, self.text_note) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationNode { + pub id: NodeId, + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, + pub note: SeparationNote, + pub children: Vec, + pub rework_count: u32, +} +impl SeparationNode { + pub fn as_prompt(&self) -> String { + format!( + "node_id={} area=({}, {}, {}, {}) {}", + self.id.as_str(), + self.global_pos_x_px, + self.global_pos_y_px, + self.width_px, + self.height_px, + self.note.as_prompt() + ) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationTree { + pub src_ui_design: UIDesignImageId, + pub root: SeparationNode, +} +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct BoundNode { + pub node_id: NodeId, + pub cut_image_path: String, +} +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct ProblematicNode { + pub node_id: NodeId, + pub problem_description: String, + pub rework_count: u32, +} +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationState { + pub schema_version: String, + pub unprocessed_trees: Vec, + pub bound: Vec, + pub problematic_nodes: Vec, +} +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationDTO { + pub bound_nodes: Vec, + pub problematic_nodes: Vec, +} + +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub struct BindingArea { + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, +} +impl BindingArea { + pub fn validate_in(&self, w: u32, h: u32) -> Result<(), String> { + if self.width_px == 0 || self.height_px == 0 { + return Err("BindingArea 宽度和高度必须大于 0".into()); + } + if self + .global_pos_x_px + .checked_add(self.width_px) + .is_none_or(|v| v > w) + || self + .global_pos_y_px + .checked_add(self.height_px) + .is_none_or(|v| v > h) + { + return Err("BindingArea 超出处理图边界".into()); + } + Ok(()) + } +} +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub enum BindingDecision { + Ok { + separated_image_area: BindingArea, + to_node: NodeId, + }, + NeedRework { + problem_description: String, + to_node: NodeId, + }, +} +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +pub struct BindingResp { + pub decisions: Vec, +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs new file mode 100644 index 000000000..5d391e2e4 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -0,0 +1,49 @@ +use super::model::*; +use crate::ui_editor::commands::separation::*; +use std::fs; +use std::path::{Path, PathBuf}; +pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result { + if asset_id.trim().is_empty() || asset_id.trim() != asset_id { + return Err("UI 资源 ID 无效".to_string()); + } + let dir = root.join("ui").join(format!( + ".{}-separation", + crate::ui_editor::persistence::generated_file_stem(asset_id) + )); + if !dir.starts_with(root) { + return Err("separation sidecar 路径越界".to_string()); + } + Ok(dir) +} + +pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<(), String> { + if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + return Err("不支持的 separation state schema".to_string()); + } + let bytes = serde_json::to_vec_pretty(state) + .map_err(|error| format!("序列化 separation state 失败：{error}"))?; + let parent = path + .parent() + .ok_or_else(|| "separation state 路径缺少父目录".to_string())?; + fs::create_dir_all(parent).map_err(|error| format!("创建 separation sidecar 失败：{error}"))?; + let temporary = path.with_extension("json.tmp"); + fs::write(&temporary, bytes).map_err(|error| format!("写入 separation state 失败：{error}"))?; + fs::rename(&temporary, path).map_err(|error| format!("安装 separation state 失败：{error}")) +} + +pub fn read_separation_state(path: &Path) -> Result { + let bytes = fs::read(path).map_err(|error| format!("读取 separation state 失败：{error}"))?; + let state: SeparationState = serde_json::from_slice(&bytes) + .map_err(|error| format!("解析 separation state 失败：{error}"))?; + if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + return Err("不支持的 separation state schema".to_string()); + } + Ok(state) +} + +pub fn separation_dto(state: &SeparationState) -> SeparationDTO { + SeparationDTO { + bound_nodes: state.bound.clone(), + problematic_nodes: state.problematic_nodes.clone(), + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs new file mode 100644 index 000000000..3343ae1dc --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs @@ -0,0 +1,52 @@ +use crate::ui_editor::commands::separation::{SeparationNode, SeparationNote}; + +const SHARED_SEPARATION_REQ: &str = r#" + + MUST hard edges; preserve no glow/blur beyond the exact visible shape. + NEVER keep its parent's background with it. + + UI elements that needs to extract has been marked with GREEN line frames (only for mark purpose, NEVER wrap a frame in your extraction). + On some UI elements, there is some PURPLE filled area, they were removed UI elements, reconstruct the background under where they were. +"#; +pub(super) fn gen_extract_prompt(separation_notes: Vec) -> String { + let extract_system_prompt = format!( + r#" + This is a UI design image, not a normal photo/illustration. Extract it strictly as UI elements/layers, not as a generic foreground/background extraction. + Treat distinct UI element as its own layer with hard, clean, pixel-accurate edges and full transparency outside the element. + + MUST keep each element at its original position on a transparent canvas. + {SHARED_SEPARATION_REQ} + here are UI elements to extract: + + "# + ); + let mut result = extract_system_prompt; + result.reserve(512); + for elem in separation_notes { + result.push_str(&elem.as_prompt()); + result.push('\n'); + } + result +} +pub(super) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { + let binding_system_prompt = format!( + r#" + You are working under a UI elements separation workflow. + You will be given a src UI design image and a processed image, where some ui elements are separated. + Here were the separation requirements: + ``` + {SHARED_SEPARATION_REQ} + ``` + You need to recognize and review the separation: + + these node need handle: + "# + ); + let mut result = binding_system_prompt; + result.reserve(512); + for elem in nodes { + result.push_str(&elem.as_prompt()); + result.push('\n'); + } + result +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs new file mode 100644 index 000000000..25b7cf6f3 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -0,0 +1,158 @@ +use super::model::*; +use crate::ui_editor::component::{image::ImageComponent, Component}; +use crate::ui_editor::layout::node::Node; +use crate::ui_editor::state::{State, UITree}; +use crate::ui_editor::utils::NodeId; +fn is_unbound_image(node: &Node) -> bool { + node.components.iter().any(|component| { + matches!( + component, + Component::Image(ImageComponent { + target_graphic: None, + .. + }) + ) + }) +} + +fn node_pixel_rect( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, +) -> (u32, u32, u32, u32) { + let rect = node.layout.transform.resolve(parent); + let x = (rect.min.x * ppu).max(0.0).round() as u32; + let y = (rect.min.y * ppu).max(0.0).round() as u32; + let w = (rect.size.x * ppu).max(0.0).round() as u32; + let h = (rect.size.y * ppu).max(0.0).round() as u32; + (x, y, w, h) +} + +fn node_description(node: &Node) -> String { + let name = node.metadata.name.trim(); + let description = node.metadata.description.trim(); + match (name.is_empty(), description.is_empty()) { + (true, true) => "未命名 UI 图片元素".to_string(), + (false, true) => name.to_string(), + (true, false) => description.to_string(), + (false, false) => format!("{name}：{description}"), + } +} + +fn collect_todo_nodes( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, + output: &mut Vec, +) { + let mut children = Vec::new(); + let rect = node.layout.transform.resolve(parent); + for child in &node.children { + collect_todo_nodes(child, &rect, ppu, &mut children); + } + if is_unbound_image(node) { + let (x, y, w, h) = node_pixel_rect(node, parent, ppu); + output.push(SeparationNode { + id: node.id.clone(), + global_pos_x_px: x, + global_pos_y_px: y, + width_px: w, + height_px: h, + note: SeparationNote { + description: node_description(node), + text_note: String::new(), + }, + children, + rework_count: 0, + }); + } else { + output.extend(children); + } +} + +pub fn construct_separation_state(state: &State) -> SeparationState { + let unprocessed_trees = state + .ui_trees + .iter() + .filter_map(|tree| { + let image = state.ui_design_images.get(&tree.src_ui_design)?; + let ppu = image.pixels_per_unit.get(); + let size = image.pixel_size / ppu; + let root_rect = + crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); + let mut children = Vec::new(); + collect_todo_nodes(&tree.root, &root_rect, ppu, &mut children); + (!children.is_empty()).then(|| SeparationTree { + src_ui_design: tree.src_ui_design.clone(), + root: SeparationNode { + id: tree.root.id.clone(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: image.pixel_size.x.max(0.0).round() as u32, + height_px: image.pixel_size.y.max(0.0).round() as u32, + note: SeparationNote::default(), + children, + rework_count: 0, + }, + }) + }) + .collect(); + SeparationState { + schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), + unprocessed_trees, + bound: Vec::new(), + problematic_nodes: Vec::new(), + } +} + +pub fn next_leaf_batch(tree: &SeparationTree) -> Vec<&SeparationNode> { + fn leaves<'a>(node: &'a SeparationNode, output: &mut Vec<&'a SeparationNode>) { + if node.children.is_empty() { + output.push(node); + } else { + for child in &node.children { + leaves(child, output); + } + } + } + let mut output = Vec::new(); + leaves(&tree.root, &mut output); + output +} + +pub fn validate_binding_response( + response: &BindingResp, + batch: &[&SeparationNode], +) -> Result<(), String> { + let expected = batch + .iter() + .map(|node| node.id.clone()) + .collect::>(); + let mut seen = std::collections::HashSet::new(); + for decision in &response.decisions { + let node_id = match decision { + BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => { + to_node + } + }; + if !expected.contains(node_id) { + return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); + } + if !seen.insert(node_id.clone()) { + return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); + } + if let BindingDecision::NeedRework { + problem_description, + .. + } = decision + { + if problem_description.trim().is_empty() { + return Err("NeedRework 必须包含问题描述".to_string()); + } + } + } + if seen.len() != expected.len() { + return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); + } + Ok(()) +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs new file mode 100644 index 000000000..3ca8fc937 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -0,0 +1,420 @@ +use super::model::*; +use super::prompt::{gen_binding_prompt, gen_extract_prompt}; +use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; +use crate::platform_session::current_platform_session; +use crate::ui_editor::commands::separation::*; +use crate::ui_editor::commands::utils::{ + parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, + request_with_feedback, strict_json_schema, +}; +use crate::ui_editor::state::State; +use crate::ui_editor::utils::NodeId; +use base64::Engine as _; +use image::ImageFormat; +use platform_llm::{ + LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice, +}; +use serde::Deserialize; +use std::fs; +use std::path::Path; +pub fn apply_batch_patch( + state: &mut SeparationState, + tree_index: usize, + decisions: &[BindingDecision], + cut_paths: &std::collections::HashMap, +) -> Result<(), String> { + let tree = state + .unprocessed_trees + .get_mut(tree_index) + .ok_or_else(|| "separation tree 索引无效".to_string())?; + let batch = next_leaf_batch(tree); + validate_binding_response( + &BindingResp { + decisions: decisions.to_vec(), + }, + &batch, + )?; + let rework_counts = batch + .iter() + .map(|node| (node.id.clone(), node.rework_count)) + .collect::>(); + let mut ids = std::collections::HashSet::new(); + for decision in decisions { + match decision { + BindingDecision::Ok { to_node, .. } => { + let path = cut_paths + .get(to_node) + .ok_or_else(|| format!("缺少节点 {} 的 cut 图片", to_node.as_str()))?; + state.bound.push(BoundNode { + node_id: to_node.clone(), + cut_image_path: path.clone(), + }); + ids.insert(to_node.clone()); + } + BindingDecision::NeedRework { + to_node, + problem_description, + } => { + let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; + if count >= MAX_REWORK_COUNT { + state.problematic_nodes.push(ProblematicNode { + node_id: to_node.clone(), + problem_description: problem_description.clone(), + rework_count: count, + }); + ids.insert(to_node.clone()); + } else { + increment_rework_count(&mut tree.root, to_node, count); + } + } + } + } + remove_ids(&mut tree.root, &ids); + Ok(()) +} + +fn mark_batch_problematic( + state: &mut SeparationState, + tree_index: usize, + batch: &[&SeparationNode], + error: String, +) { + let ids = batch + .iter() + .map(|node| node.id.clone()) + .collect::>(); + for node in batch { + state.problematic_nodes.push(ProblematicNode { + node_id: node.id.clone(), + problem_description: error.clone(), + rework_count: MAX_REWORK_COUNT, + }); + } + if let Some(tree) = state.unprocessed_trees.get_mut(tree_index) { + remove_ids(&mut tree.root, &ids); + } +} + +fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { + if node.id == *id { + node.rework_count = count; + return; + } + for child in &mut node.children { + increment_rework_count(child, id, count); + } +} + +#[derive(Deserialize)] +struct RawEditResponse { + data: Vec, +} +#[derive(Deserialize)] +struct RawEditItem { + b64_json: String, +} + +async fn raw_image_edit( + session: &crate::platform_session::PlatformSessionSnapshot, + image_data_url: &str, + prompt: &str, + width: u32, + height: u32, +) -> Result { + let (mime, data) = image_data_url + .split_once(",") + .ok_or_else(|| "界面图 data URL 无效".to_string())?; + let mime = mime + .strip_prefix("data:") + .and_then(|v| v.strip_suffix(";base64")) + .unwrap_or("image/png"); + let client = crate::http_client::agc_main_site_client_builder() + .build() + .map_err(|e| format!("创建图片编辑客户端失败：{e}"))?; + let url = format!( + "{}/api/raw/v1/images/edit", + session.api_base_url.trim_end_matches('/') + ); + let body = serde_json::json!({ + "image": {"data": data, "mimeType": mime}, + "prompt": prompt, + "width": width, + "height": height, + "output_format": "png", + "background": "transparent" + }); + let response = crate::http_client::with_agc_main_site_marker( + client + .post(url) + .bearer_auth(&session.access_token) + .json(&body), + ) + .send() + .await + .map_err(|e| format!("图片分离请求失败：{e}"))?; + if !response.status().is_success() { + return Err(format!("图片分离请求失败（HTTP {}）", response.status())); + } + let payload = response + .json::() + .await + .map_err(|e| format!("解析图片分离响应失败：{e}"))?; + payload + .data + .into_iter() + .next() + .map(|item| format!("data:image/png;base64,{}", item.b64_json)) + .ok_or_else(|| "图片分离响应没有图像".to_string()) +} + +fn build_marked_image( + source_url: &str, + nodes: &[&SeparationNode], + target: &Path, +) -> Result { + let encoded = source_url + .split_once(',') + .map(|(_, d)| d) + .ok_or_else(|| "源图 data URL 无效".to_string())?; + let bytes = base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|e| format!("解码源图失败：{e}"))?; + let mut image = image::load_from_memory(&bytes) + .map_err(|e| format!("读取源图失败：{e}"))? + .to_rgba8(); + let width = image.width(); + let height = image.height(); + for node in nodes { + let x0 = node.global_pos_x_px.min(width.saturating_sub(1)); + let y0 = node.global_pos_y_px.min(height.saturating_sub(1)); + let x1 = node + .global_pos_x_px + .saturating_add(node.width_px) + .min(width) + .saturating_sub(1); + let y1 = node + .global_pos_y_px + .saturating_add(node.height_px) + .min(height) + .saturating_sub(1); + if x0 >= x1 || y0 >= y1 { + continue; + } + for x in x0..=x1 { + image.put_pixel(x, y0, image::Rgba([0, 255, 0, 255])); + image.put_pixel(x, y1, image::Rgba([0, 255, 0, 255])); + } + for y in y0..=y1 { + image.put_pixel(x0, y, image::Rgba([0, 255, 0, 255])); + image.put_pixel(x1, y, image::Rgba([0, 255, 0, 255])); + } + for y in y0..=y1 { + for x in x0..=x1 { + if x > x0 && x < x1 && y > y0 && y < y1 { + image.put_pixel(x, y, image::Rgba([180, 0, 180, 120])); + } + } + } + } + image::DynamicImage::ImageRgba8(image.clone()) + .save_with_format(target, image::ImageFormat::Png) + .map_err(|e| format!("写入标记图失败：{e}"))?; + let mut png = Vec::new(); + image::DynamicImage::ImageRgba8(image) + .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) + .map_err(|e| format!("编码标记图失败：{e}"))?; + Ok(format!( + "data:image/png;base64,{}", + base64::engine::general_purpose::STANDARD.encode(png) + )) +} + +async fn visual_binding( + source_url: String, + processed_url: String, + nodes: &[&SeparationNode], +) -> Result { + let llm_config = load_game_creator_app_config() + .map_err(|e| e.to_string())? + .llm; + let client = build_game_creator_llm_client_from_llm_config(&llm_config, "llm") + .map_err(|e| e.to_string())?; + let schema = strict_json_schema::()?; + let tool = LlmFunctionTool::new( + "bind_ui_elements", + "确认处理图中的区域对应哪些 UI 节点", + schema, + ) + .with_strict(true); + let base_prompt = gen_binding_prompt(nodes.to_vec()); + request_with_feedback( + 2, + |feedback| { + let prompt = feedback.map_or_else( + || base_prompt.clone(), + |error| format!("{base_prompt}\n上一次输出错误：{error}\n请修正并完整返回。"), + ); + let source_url = source_url.clone(); + let processed_url = processed_url.clone(); + let tool = tool.clone(); + let client = client.clone(); + let llm_config = llm_config.clone(); + async move { + let request = LlmRunRequest::new(vec![ + LlmMessage::system("你是 UI 图片视觉绑定器。只根据图像判断区域，不做 OCR。"), + LlmMessage::user_multimodal(vec![ + LlmMessageContentPart::InputText { text: prompt }, + LlmMessageContentPart::InputImage { + image_url: source_url, + }, + LlmMessageContentPart::InputImage { + image_url: processed_url, + }, + ]), + ]) + .with_function_tools(vec![tool.clone()]) + .with_tool_choice(LlmToolChoice::Required); + request_ui_editor_llm(&client, &llm_config, request) + .await + .map_err(|e| e.to_string()) + .and_then(|response| { + response + .tool_calls + .into_iter() + .find(|call| call.name == "bind_ui_elements") + .map(|call| call.arguments) + .ok_or_else(|| "视觉绑定模型未返回工具调用".to_string()) + }) + .and_then(|arguments| parse_limited_llm_tool_arguments(&arguments)) + .and_then(|args| { + serde_json::from_value::(args) + .map_err(|e| format!("视觉绑定结果无效：{e}")) + }) + .and_then(|parsed| Ok(parsed)) + } + }, + |value: &BindingResp| validate_binding_response(value, nodes), + ) + .await +} + +pub(crate) async fn separate_ui_impl( + project_path: String, + asset_id: String, + state: State, +) -> Result { + let session = current_platform_session().ok_or_else(|| "请先登录平台账号".to_string())?; + let root = Path::new(project_path.trim()); + let sidecar = separation_sidecar_dir(root, &asset_id)?; + fs::create_dir_all(&sidecar).map_err(|e| format!("创建 separation sidecar 失败：{e}"))?; + let state_path = sidecar.join("state.json"); + let mut separation = if state_path.exists() { + read_separation_state(&state_path)? + } else { + construct_separation_state(&state) + }; + for (tree_index, tree) in separation.unprocessed_trees.clone().iter().enumerate() { + let image = state + .ui_design_images + .get(&tree.src_ui_design) + .ok_or_else(|| "缺少源界面图".to_string())?; + let source_path = crate::project::resolve_local_project_path(root, &image.path)?; + let source_url = read_ui_reference_image_data_url(source_path).await?; + write_separation_state(&state_path, &separation)?; + loop { + let Some(current_tree) = separation.unprocessed_trees.get(tree_index) else { + break; + }; + let batch_nodes = next_leaf_batch(current_tree) + .into_iter() + .cloned() + .collect::>(); + if batch_nodes.is_empty() { + break; + } + let batch = batch_nodes.iter().collect::>(); + let prompt = + gen_extract_prompt(batch.iter().map(|n| n.note.clone()).collect::>()); + let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); + let marked_url = match build_marked_image(&source_url, &batch, &marker_path) { + Ok(value) => value, + Err(error) => { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } + }; + let processed_url = match raw_image_edit( + &session, + &marked_url, + &prompt, + image.pixel_size.x as u32, + image.pixel_size.y as u32, + ) + .await + { + Ok(value) => value, + Err(error) => { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } + }; + let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); + let processed_bytes = base64::engine::general_purpose::STANDARD + .decode( + processed_url + .split_once(',') + .map(|(_, d)| d) + .unwrap_or_default(), + ) + .map_err(|e| e.to_string())?; + fs::write(&processed_path, processed_bytes).map_err(|e| e.to_string())?; + let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { + Ok(value) => value, + Err(error) => { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } + }; + let mut cut_paths = std::collections::HashMap::new(); + for decision in &binding.decisions { + if let BindingDecision::Ok { + to_node, + separated_image_area, + } = decision + { + let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); + cut_processed_image(&processed_path, separated_image_area, &cut_path)?; + cut_paths.insert(to_node.clone(), cut_path.to_string_lossy().to_string()); + } + } + apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; + write_separation_state(&state_path, &separation)?; + } + } + let _ = fs::remove_file(state_path); + Ok(separation_dto(&separation)) +} + +fn cut_processed_image(source: &Path, area: &BindingArea, target: &Path) -> Result<(), String> { + let image = image::open(source).map_err(|e| format!("读取处理图失败：{e}"))?; + area.validate_in(image.width(), image.height())?; + let cropped = image.crop_imm( + area.global_pos_x_px, + area.global_pos_y_px, + area.width_px, + area.height_px, + ); + cropped + .save_with_format(target, ImageFormat::Png) + .map_err(|e| format!("写入 cut 图片失败：{e}")) +} + +fn remove_ids(node: &mut SeparationNode, ids: &std::collections::HashSet) { + node.children.retain(|child| !ids.contains(&child.id)); + for child in &mut node.children { + remove_ids(child, ids); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index 4faa40579..8aec25302 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -4,6 +4,7 @@ use base64::Engine as _; use platform_llm::{LlmClient, LlmError, LlmRunRequest, LlmRunResponse}; use schemars::JsonSchema; use std::fs::File; +use std::future::Future; use std::io::Read; use std::path::{Path, PathBuf}; @@ -25,6 +26,33 @@ pub(crate) async fn request_ui_editor_llm( request_game_creator_llm_text(client, llm, request).await } +/// 结构化 LLM 请求的小型 repair harness：第一次请求或校验失败后， +/// 将错误反馈给模型并只额外重试一次。网络/模型调用本身的错误也会 +/// 进入第二次请求的反馈文本；调用方负责在第二次失败后决定业务状态。 +pub(crate) async fn request_with_feedback( + more_turn: usize, + request: Request, + validate: Validate, +) -> Result +where + Request: Fn(Option) -> Fut, + Fut: Future>, + Validate: Fn(&T) -> Result<(), String>, +{ + let mut feedback = None; + for attempt in 0..=more_turn { + let result = request(feedback.clone()) + .await + .and_then(|value| validate(&value).map(|_| value)); + match result { + Ok(value) => return Ok(value), + Err(error) if attempt < more_turn => feedback = Some(error), + Err(error) => return Err(error), + } + } + unreachable!("repair harness always returns within requested turns") +} + pub(crate) fn parse_limited_llm_tool_arguments( arguments: &str, ) -> Result { @@ -144,6 +172,27 @@ mod tests { ); } + #[tokio::test] + async fn feedback_harness_zero_more_turn_calls_once_without_feedback() { + let calls = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let seen = calls.clone(); + let result = request_with_feedback( + 0, + move |feedback| { + let seen = seen.clone(); + async move { + seen.lock().unwrap().push(feedback); + Ok::<_, String>(serde_json::json!({"ok": true})) + } + }, + |_| Ok(()), + ) + .await + .expect("single turn should succeed"); + assert_eq!(result, serde_json::json!({"ok": true})); + assert_eq!(calls.lock().unwrap().as_slice(), &[None]); + } + #[test] fn reference_image_rejects_file_over_five_mib_before_reading() { let directory = tempfile::tempdir().expect("reference image fixture"); From 187faa607de59b78b31ede855fc52439c8906d9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Tue, 8 Sep 2026 23:49:31 +0800 Subject: [PATCH 040/248] =?UTF-8?q?=E4=BF=9D=E7=95=99=E5=8E=9F=E5=A7=8B?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=8F=90=E7=A4=BA=E5=B9=B6=E6=8B=86=E5=88=86?= =?UTF-8?q?=E6=A8=A1=E5=9D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按模型、树、持久化、提示词和工作流拆分 separation 提取支持 more_turn 的反馈重试 harness 删除旧 separation.rs 文件 --- .../src-tauri/src/ui_editor/commands/separation/mod.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index d103da6e1..edef9453e 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -7,10 +7,16 @@ mod workflow; pub use model::*; pub use persistence::*; pub use tree::*; +pub use workflow::apply_batch_patch; pub(crate) use workflow::separate_ui_impl; #[cfg(test)] mod tests { use super::*; + use crate::ui_editor::component::Component; + use crate::ui_editor::layout::node::Node; + use crate::ui_editor::state::{State, UITree}; + use crate::ui_editor::utils::{NodeId, UIDesignImageId}; + use std::path::Path; use crate::ui_editor::component::image::{ImageComponent, ImageType}; use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; use crate::ui_editor::layout::control_layout::ControlLayout; From aff40f794d58fdbaab5052728c5a2349e59b71c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 00:38:55 +0800 Subject: [PATCH 041/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=91=E5=90=88=E6=88=90=E6=A0=B9=E8=8A=82=E7=82=B9=20ID=20?= =?UTF-8?q?=E5=86=B2=E7=AA=81=20=E9=81=BF=E5=85=8D=E5=8D=95=E5=9B=BE?= =?UTF-8?q?=E6=A0=B9=E8=8A=82=E7=82=B9=E9=87=8D=E5=A4=8D=E8=BF=9B=E5=85=A5?= =?UTF-8?q?=E9=87=8D=E5=81=9A=E8=AE=A1=E6=95=B0=20=E8=A1=A5=E5=85=85?= =?UTF-8?q?=E6=A0=B9=E5=9B=BE=E7=89=87=E6=9E=84=E9=80=A0=E5=9B=9E=E5=BD=92?= =?UTF-8?q?=E6=B5=8B=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/separation/mod.rs | 15 +++++++++++++++ .../src/ui_editor/commands/separation/tree.rs | 9 ++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index edef9453e..708d3c191 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -92,6 +92,21 @@ mod tests { "image" ); } + + #[test] + fn construction_uses_distinct_root_id_for_root_image() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let root = node("root-image", vec![image], vec![]); + let result = construct_separation_state(&state(root)); + let tree = &result.unprocessed_trees[0]; + assert_ne!(tree.root.id, tree.root.children[0].id); + assert_eq!(tree.root.children[0].id.as_str(), "root-image"); + } #[test] fn binding_validation_requires_exact_batch_coverage() { let node = SeparationNode { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 25b7cf6f3..45c93dbec 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -85,7 +85,14 @@ pub fn construct_separation_state(state: &State) -> SeparationState { (!children.is_empty()).then(|| SeparationTree { src_ui_design: tree.src_ui_design.clone(), root: SeparationNode { - id: tree.root.id.clone(), + // The synthetic root must never share an ID with a real + // UI node. A single-image design may use the original + // tree root as an eligible separation leaf. + id: NodeId::new(format!( + "separation-root-{}", + uuid::Uuid::new_v4().simple() + )) + .expect("synthetic separation root id is valid"), global_pos_x_px: 0, global_pos_y_px: 0, width_px: image.pixel_size.x.max(0.0).round() as u32, From dda6b3946a4331d8a26866598c7d45de15b3be9a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 00:41:35 +0800 Subject: [PATCH 042/248] =?UTF-8?q?=E9=98=BB=E6=AD=A2=E8=AF=86=E5=88=AB?= =?UTF-8?q?=E7=BB=93=E6=9E=9C=E5=BC=95=E7=94=A8=E6=9C=AA=E6=8E=88=E6=9D=83?= =?UTF-8?q?=E5=AD=97=E4=BD=93=E7=B4=A0=E6=9D=90=20=E8=A1=A5=E9=BD=90?= =?UTF-8?q?=E8=AF=86=E5=88=AB=E9=98=B6=E6=AE=B5=E5=AD=97=E4=BD=93=E5=BC=95?= =?UTF-8?q?=E7=94=A8=E6=A0=A1=E9=AA=8C=20=E5=A2=9E=E5=8A=A0=E4=BC=AA?= =?UTF-8?q?=E9=80=A0=E5=AD=97=E4=BD=93=E7=BB=91=E5=AE=9A=E5=9B=9E=E5=BD=92?= =?UTF-8?q?=E6=B5=8B=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/recognition.rs | 67 +++++++++++++++---- 1 file changed, 54 insertions(+), 13 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index ceae0d7ea..bc578daf8 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -348,6 +348,17 @@ fn validate_confidence(nodes: &[RecognitionNode]) -> Result<(), String> { }) { return Err("识别阶段不能返回已绑定的 SpriteAssetId".to_string()); } + if node.components.iter().any(|component| { + matches!( + component, + Component::Text(crate::ui_editor::component::text::TextComponent { + font: crate::ui_editor::component::text::FontSource::Bound(_), + .. + }) + ) + }) { + return Err("识别阶段不能返回已绑定的字体素材".to_string()); + } if let Confidence::UnSure(reason) = &node.confidence { if reason.trim().is_empty() { return Err("UnSure 必须包含审阅原因".to_string()); @@ -464,19 +475,23 @@ mod tests { let oversized = (0..=MAX_RECOGNITION_TREE_NODES) .map(|_| serde_json::json!({"children": []})) .collect::>(); - assert!(validate_recognition_response_shape(&serde_json::json!({ - "trees": [{"children": oversized}] - })) - .is_err()); + assert!( + validate_recognition_response_shape(&serde_json::json!({ + "trees": [{"children": oversized}] + })) + .is_err() + ); let mut nested = serde_json::json!({"children": []}); for _ in 0..MAX_RECOGNITION_TREE_DEPTH { nested = serde_json::json!({"children": [nested]}); } - assert!(validate_recognition_response_shape(&serde_json::json!({ - "trees": [{"children": [nested]}] - })) - .is_err()); + assert!( + validate_recognition_response_shape(&serde_json::json!({ + "trees": [{"children": [nested]}] + })) + .is_err() + ); } #[test] @@ -521,6 +536,30 @@ mod tests { assert!(validate_confidence(&[node]).is_err()); } + #[test] + fn recognition_rejects_bound_font_references() { + let mut text = crate::ui_editor::component::text::TextComponent::default(); + text.font = crate::ui_editor::component::text::FontSource::Bound( + crate::ui_editor::utils::FontAssetId::new("font").expect("valid font id"), + ); + let node = RecognitionNode { + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + local_anchor: Anchor::Preset(PresetAnchor { + horizontal: HorizontalAnchor::Left, + vertical: VerticalAnchor::Top, + }), + name: "文本".to_string(), + description: String::new(), + children: Vec::new(), + confidence: Confidence::Confident, + components: vec![Component::Text(text)], + }; + assert!(validate_confidence(&[node]).is_err()); + } + #[test] fn conversion_stays_in_the_tree_image_coordinate_system() { let image_id = UIDesignImageId::new("slave").expect("valid image id"); @@ -572,11 +611,13 @@ mod tests { children: Vec::new(), }; - assert!(validate_tree_image_ids( - &[tree(page.clone()), tree(slave.clone())], - &[page.clone(), slave.clone()], - ) - .is_ok()); + assert!( + validate_tree_image_ids( + &[tree(page.clone()), tree(slave.clone())], + &[page.clone(), slave.clone()], + ) + .is_ok() + ); assert!( validate_tree_image_ids(&[tree(page.clone())], &[page.clone(), slave.clone()]).is_err() ); From 3c1e7bd5996d71062b1183db1833404d3a8d208a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 00:41:57 +0800 Subject: [PATCH 043/248] =?UTF-8?q?=E4=BF=9D=E7=95=99=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E9=97=AE=E9=A2=98=E8=8A=82=E7=82=B9=E7=9A=84=E7=9C=9F=E5=AE=9E?= =?UTF-8?q?=E9=87=8D=E5=81=9A=E6=AC=A1=E6=95=B0=20=E9=81=BF=E5=85=8D?= =?UTF-8?q?=E5=9F=BA=E7=A1=80=E8=AE=BE=E6=96=BD=E5=A4=B1=E8=B4=A5=E4=BC=AA?= =?UTF-8?q?=E9=80=A0=E8=BE=BE=E5=88=B0=E4=B8=8A=E9=99=90=20=E4=BF=9D?= =?UTF-8?q?=E6=8C=81=20problematic=20=E8=8A=82=E7=82=B9=E8=AF=8A=E6=96=AD?= =?UTF-8?q?=E4=BF=A1=E6=81=AF=E5=87=86=E7=A1=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/workflow.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 3ca8fc937..2d3cbb5ff 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -87,7 +87,7 @@ fn mark_batch_problematic( state.problematic_nodes.push(ProblematicNode { node_id: node.id.clone(), problem_description: error.clone(), - rework_count: MAX_REWORK_COUNT, + rework_count: node.rework_count, }); } if let Some(tree) = state.unprocessed_trees.get_mut(tree_index) { @@ -313,7 +313,8 @@ pub(crate) async fn separate_ui_impl( } else { construct_separation_state(&state) }; - for (tree_index, tree) in separation.unprocessed_trees.clone().iter().enumerate() { + for tree_index in 0..separation.unprocessed_trees.len() { + let tree = &separation.unprocessed_trees[tree_index]; let image = state .ui_design_images .get(&tree.src_ui_design) From 19f3d62d784068d935bdc3b10af90d561266c8ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 00:42:51 +0800 Subject: [PATCH 044/248] =?UTF-8?q?=E9=9A=94=E7=A6=BB=E9=9D=9E=E6=B3=95?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E5=8C=BA=E5=9F=9F=E5=AF=BC=E8=87=B4=E7=9A=84?= =?UTF-8?q?=E6=89=B9=E6=AC=A1=E5=A4=B1=E8=B4=A5=20=E5=B0=86=E8=A3=81?= =?UTF-8?q?=E5=88=87=E9=94=99=E8=AF=AF=E8=AE=B0=E5=BD=95=E4=B8=BA=20proble?= =?UTF-8?q?matic=20=E5=B9=B6=E7=BB=A7=E7=BB=AD=E6=B5=81=E7=A8=8B=20?= =?UTF-8?q?=E9=81=BF=E5=85=8D=E5=8D=95=E4=B8=AA=E6=A8=A1=E5=9E=8B=E5=8C=BA?= =?UTF-8?q?=E5=9F=9F=E7=BB=88=E6=AD=A2=E6=95=B4=E9=A1=B5=E5=88=86=E7=A6=BB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../ui_editor/commands/separation/workflow.rs | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 2d3cbb5ff..a17c07926 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -380,6 +380,7 @@ pub(crate) async fn separate_ui_impl( } }; let mut cut_paths = std::collections::HashMap::new(); + let mut cut_error = None; for decision in &binding.decisions { if let BindingDecision::Ok { to_node, @@ -387,10 +388,24 @@ pub(crate) async fn separate_ui_impl( } = decision { let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); - cut_processed_image(&processed_path, separated_image_area, &cut_path)?; - cut_paths.insert(to_node.clone(), cut_path.to_string_lossy().to_string()); + match cut_processed_image(&processed_path, separated_image_area, &cut_path) { + Ok(()) => { + cut_paths + .insert(to_node.clone(), cut_path.to_string_lossy().to_string()); + } + Err(error) => { + cut_error = + Some(format!("节点 {} 的分离区域无效：{error}", to_node.as_str())); + break; + } + } } } + if let Some(error) = cut_error { + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + continue; + } apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; write_separation_state(&state_path, &separation)?; } From 207b0820ddf8e7f5386df76c05395c463e8c2f8d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 00:52:48 +0800 Subject: [PATCH 045/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=91=E6=96=87=E4=BB=B6=E6=A0=BC=E5=BC=8F=20=E6=8C=89?= =?UTF-8?q?=E4=BB=93=E5=BA=93=20rustfmt=20=E8=A7=84=E8=8C=83=E6=95=B4?= =?UTF-8?q?=E7=90=86=20synthetic=20root=20=E5=AE=9E=E7=8E=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/tree.rs | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 45c93dbec..b600c48e1 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -1,5 +1,5 @@ use super::model::*; -use crate::ui_editor::component::{image::ImageComponent, Component}; +use crate::ui_editor::component::{Component, image::ImageComponent}; use crate::ui_editor::layout::node::Node; use crate::ui_editor::state::{State, UITree}; use crate::ui_editor::utils::NodeId; @@ -88,11 +88,8 @@ pub fn construct_separation_state(state: &State) -> SeparationState { // The synthetic root must never share an ID with a real // UI node. A single-image design may use the original // tree root as an eligible separation leaf. - id: NodeId::new(format!( - "separation-root-{}", - uuid::Uuid::new_v4().simple() - )) - .expect("synthetic separation root id is valid"), + id: NodeId::new(format!("separation-root-{}", uuid::Uuid::new_v4().simple())) + .expect("synthetic separation root id is valid"), global_pos_x_px: 0, global_pos_y_px: 0, width_px: image.pixel_size.x.max(0.0).round() as u32, From 884c5ec5eef5822588581725e2b2176bbde6650b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 10:43:38 +0800 Subject: [PATCH 046/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=20sidecar=20=E5=B9=B6=E5=8F=91=E8=BE=B9?= =?UTF-8?q?=E7=95=8C=E7=BA=A6=E6=9D=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录前端单次运行与 runWithStateLocked 的并发前提 明确 sidecar 不参与正式资产写入和项目 revision 补充未来多窗口多进程场景的进程级锁 TODO 记录图像本地处理使用 spawn_blocking 的执行边界 --- .../【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index c21e40e68..e967abc21 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -29,6 +29,8 @@ - image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 +- 标记图构建、处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 + `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 - `NeedRework` 携带短问题描述。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 @@ -44,6 +46,10 @@ - sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 - 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 - 临时图片可跨重启保留。raw image-edit 返回图、绿色/紫色标记图、处理图和 cut 图片当前都保留用于 debug；理论上只应在内存中，清理/归档策略列 TODO。 +- 并发边界：当前由前端 `isSeparating` 与 `runWithStateLocked` 保证同一 UI 编辑会话 + 同时只有一次 separation。sidecar 是临时恢复状态，不是正式 UI 资产真相，不参与 + manifest 或项目 revision，因此当前不额外持有项目写锁；若未来支持多窗口/多进程并发， + 再增加按 UI asset 的 sidecar 进程级锁。 ## bound 与 problematic From 56fff8325b3c6e1a154d9302f4fdf890cb04a05a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 10:45:11 +0800 Subject: [PATCH 047/248] =?UTF-8?q?=E5=B0=86=E8=87=AA=E5=8A=A8=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E5=9B=BE=E5=83=8F=E5=A4=84=E7=90=86=E7=A7=BB=E5=87=BA?= =?UTF-8?q?=E5=BC=82=E6=AD=A5=E6=89=A7=E8=A1=8C=E5=99=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在 spawn_blocking 中构建标记图并编码写入处理图 在 spawn_blocking 中完成处理图裁切与文件输出 保留 image-edit 和视觉绑定网络请求的 async 调度边界 --- .../ui_editor/commands/separation/workflow.rs | 104 ++++++++++++++---- 1 file changed, 80 insertions(+), 24 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index a17c07926..19e1ee531 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -16,7 +16,7 @@ use platform_llm::{ }; use serde::Deserialize; use std::fs; -use std::path::Path; +use std::path::{Path, PathBuf}; pub fn apply_batch_patch( state: &mut SeparationState, tree_index: usize, @@ -167,9 +167,32 @@ async fn raw_image_edit( .ok_or_else(|| "图片分离响应没有图像".to_string()) } -fn build_marked_image( +async fn build_marked_image( + source_url: String, + nodes: Vec, + target: PathBuf, +) -> Result { + tokio::task::spawn_blocking(move || { + let areas = nodes + .iter() + .map(|node| { + ( + node.global_pos_x_px, + node.global_pos_y_px, + node.width_px, + node.height_px, + ) + }) + .collect::>(); + build_marked_image_blocking(&source_url, &areas, &target) + }) + .await + .map_err(|error| format!("构建标记图任务失败：{error}"))? +} + +fn build_marked_image_blocking( source_url: &str, - nodes: &[&SeparationNode], + nodes: &[(u32, u32, u32, u32)], target: &Path, ) -> Result { let encoded = source_url @@ -184,17 +207,15 @@ fn build_marked_image( .to_rgba8(); let width = image.width(); let height = image.height(); - for node in nodes { - let x0 = node.global_pos_x_px.min(width.saturating_sub(1)); - let y0 = node.global_pos_y_px.min(height.saturating_sub(1)); - let x1 = node - .global_pos_x_px - .saturating_add(node.width_px) + for &(global_pos_x_px, global_pos_y_px, node_width_px, node_height_px) in nodes { + let x0 = global_pos_x_px.min(width.saturating_sub(1)); + let y0 = global_pos_y_px.min(height.saturating_sub(1)); + let x1 = global_pos_x_px + .saturating_add(node_width_px) .min(width) .saturating_sub(1); - let y1 = node - .global_pos_y_px - .saturating_add(node.height_px) + let y1 = global_pos_y_px + .saturating_add(node_height_px) .min(height) .saturating_sub(1); if x0 >= x1 || y0 >= y1 { @@ -229,6 +250,23 @@ fn build_marked_image( )) } +async fn write_processed_image(processed_url: String, target: PathBuf) -> Result<(), String> { + tokio::task::spawn_blocking(move || { + let processed_bytes = base64::engine::general_purpose::STANDARD + .decode( + processed_url + .split_once(',') + .map(|(_, data)| data) + .unwrap_or_default(), + ) + .map_err(|error| format!("解析处理图失败：{error}"))?; + fs::write(&target, processed_bytes) + .map_err(|error| format!("写入处理图失败：{}: {error}", target.display())) + }) + .await + .map_err(|error| format!("写入处理图任务失败：{error}"))? +} + async fn visual_binding( source_url: String, processed_url: String, @@ -337,7 +375,13 @@ pub(crate) async fn separate_ui_impl( let prompt = gen_extract_prompt(batch.iter().map(|n| n.note.clone()).collect::>()); let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); - let marked_url = match build_marked_image(&source_url, &batch, &marker_path) { + let marked_url = match build_marked_image( + source_url.clone(), + batch_nodes.clone(), + marker_path, + ) + .await + { Ok(value) => value, Err(error) => { mark_batch_problematic(&mut separation, tree_index, &batch, error); @@ -362,15 +406,7 @@ pub(crate) async fn separate_ui_impl( } }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); - let processed_bytes = base64::engine::general_purpose::STANDARD - .decode( - processed_url - .split_once(',') - .map(|(_, d)| d) - .unwrap_or_default(), - ) - .map_err(|e| e.to_string())?; - fs::write(&processed_path, processed_bytes).map_err(|e| e.to_string())?; + write_processed_image(processed_url.clone(), processed_path.clone()).await?; let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { Ok(value) => value, Err(error) => { @@ -388,7 +424,13 @@ pub(crate) async fn separate_ui_impl( } = decision { let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); - match cut_processed_image(&processed_path, separated_image_area, &cut_path) { + match cut_processed_image( + processed_path.clone(), + *separated_image_area, + cut_path.clone(), + ) + .await + { Ok(()) => { cut_paths .insert(to_node.clone(), cut_path.to_string_lossy().to_string()); @@ -414,7 +456,21 @@ pub(crate) async fn separate_ui_impl( Ok(separation_dto(&separation)) } -fn cut_processed_image(source: &Path, area: &BindingArea, target: &Path) -> Result<(), String> { +async fn cut_processed_image( + source: PathBuf, + area: BindingArea, + target: PathBuf, +) -> Result<(), String> { + tokio::task::spawn_blocking(move || cut_processed_image_blocking(&source, &area, &target)) + .await + .map_err(|error| format!("裁切处理图任务失败：{error}"))? +} + +fn cut_processed_image_blocking( + source: &Path, + area: &BindingArea, + target: &Path, +) -> Result<(), String> { let image = image::open(source).map_err(|e| format!("读取处理图失败：{e}"))?; area.validate_in(image.width(), image.height())?; let cropped = image.crop_imm( From 15e30acb0f9c6cc73b1d68345a9491c307e05df4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 11:38:52 +0800 Subject: [PATCH 048/248] =?UTF-8?q?=E8=B0=83=E6=95=B4UI=E8=AF=86=E5=88=AB?= =?UTF-8?q?=E6=8C=87=E4=BB=A4=E7=BB=86=E8=8A=82=EF=BC=8C=E6=98=8E=E7=A1=AE?= =?UTF-8?q?=E6=96=87=E5=AD=97=E7=BB=84=E4=BB=B6=E8=AF=86=E5=88=AB=E8=A7=84?= =?UTF-8?q?=E5=88=99=E4=B8=8E=E7=B2=92=E5=BA=A6=E6=A0=87=E5=87=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/recognition.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index bc578daf8..6671e9c83 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -46,9 +46,10 @@ const SYSTEM_PROMPT: &str = r#" * 面向用户的字段如名称描述等请用中文 * 由于每个截图未必是完整的, 可能是局部的, 每棵树描述清楚每个截图上UI的层次结构即可 * 不同树的共用框架/层次/...请使用使用相同的名称描述. 不同状态/变体名称使用相同的前缀, 用后缀区别 -* 粒度要求: 尽可能细致, 最小单元举例: 进度条的底槽、填充和外框; slider的底槽, dragger等 -* 为每个节点直接返回完整 components。纯容器返回空数组；需要从设计图自动分离图片的 Image component 必须令 target_graphic 为 null。文本内容和组件类型完全由视觉判断，不调用或依赖 OCR。 +* 粒度要求: 以可交互,方便程序化控制的最小单位为准: 进度条的底槽、填充和外框; slider的底槽, dragger等. +* 为每个节点直接返回完整 components。纯容器返回空数组；目前我们只做识别, 不要求图片字体参数 * 每个节点当前最多返回一个 Image component 和一个 Text component。 +* 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. "#; From 99c19e3ed92715e244a6398aff2033879adc7c4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 12:06:45 +0800 Subject: [PATCH 049/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=B5=81=E7=A8=8B=E6=8E=92=E9=9A=9C=E6=97=A5=E5=BF=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录 separation sidecar 状态读写与恢复阶段 记录树构造、批次选择、视觉绑定、裁切和 patch 结果 避免输出 base64、完整提示词和敏感凭据 --- .../commands/separation/persistence.rs | 91 ++++- .../src/ui_editor/commands/separation/tree.rs | 71 +++- .../ui_editor/commands/separation/workflow.rs | 333 ++++++++++++++++-- 3 files changed, 454 insertions(+), 41 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 5d391e2e4..91ea757f0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -4,6 +4,7 @@ use std::fs; use std::path::{Path, PathBuf}; pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result { if asset_id.trim().is_empty() || asset_id.trim() != asset_id { + app_log!("ui_separation.error stage=sidecar_dir reason=invalid_asset_id"); return Err("UI 资源 ID 无效".to_string()); } let dir = root.join("ui").join(format!( @@ -11,37 +12,105 @@ pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result") + ); Ok(dir) } pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<(), String> { if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + app_log!("ui_separation.error stage=state_write reason=schema_mismatch"); return Err("不支持的 separation state schema".to_string()); } - let bytes = serde_json::to_vec_pretty(state) - .map_err(|error| format!("序列化 separation state 失败：{error}"))?; - let parent = path - .parent() - .ok_or_else(|| "separation state 路径缺少父目录".to_string())?; - fs::create_dir_all(parent).map_err(|error| format!("创建 separation sidecar 失败：{error}"))?; + app_log!( + "ui_separation.state_write.start file={} trees={} bound={} problematic={}", + path.file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + state.unprocessed_trees.len(), + state.bound.len(), + state.problematic_nodes.len() + ); + let bytes = serde_json::to_vec_pretty(state).map_err(|error| { + app_log!("ui_separation.error stage=state_write reason=serialize error={error}"); + format!("序列化 separation state 失败：{error}") + })?; + let parent = path.parent().ok_or_else(|| { + app_log!("ui_separation.error stage=state_write reason=missing_parent"); + "separation state 路径缺少父目录".to_string() + })?; + fs::create_dir_all(parent).map_err(|error| { + app_log!("ui_separation.error stage=state_write reason=create_parent error={error}"); + format!("创建 separation sidecar 失败：{error}") + })?; let temporary = path.with_extension("json.tmp"); - fs::write(&temporary, bytes).map_err(|error| format!("写入 separation state 失败：{error}"))?; - fs::rename(&temporary, path).map_err(|error| format!("安装 separation state 失败：{error}")) + fs::write(&temporary, bytes).map_err(|error| { + app_log!("ui_separation.error stage=state_write reason=write_temp error={error}"); + format!("写入 separation state 失败：{error}") + })?; + fs::rename(&temporary, path).map_err(|error| { + app_log!("ui_separation.error stage=state_write reason=install error={error}"); + format!("安装 separation state 失败：{error}") + })?; + app_log!( + "ui_separation.state_write.completed file={} bytes={} trees={} bound={} problematic={}", + path.file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + fs::metadata(path) + .map(|metadata| metadata.len()) + .unwrap_or(0), + state.unprocessed_trees.len(), + state.bound.len(), + state.problematic_nodes.len() + ); + Ok(()) } pub fn read_separation_state(path: &Path) -> Result { - let bytes = fs::read(path).map_err(|error| format!("读取 separation state 失败：{error}"))?; - let state: SeparationState = serde_json::from_slice(&bytes) - .map_err(|error| format!("解析 separation state 失败：{error}"))?; + app_log!( + "ui_separation.state_read.start file={}", + path.file_name() + .and_then(|name| name.to_str()) + .unwrap_or("") + ); + let bytes = fs::read(path).map_err(|error| { + app_log!("ui_separation.error stage=state_read reason=read error={error}"); + format!("读取 separation state 失败：{error}") + })?; + let state: SeparationState = serde_json::from_slice(&bytes).map_err(|error| { + app_log!("ui_separation.error stage=state_read reason=parse error={error}"); + format!("解析 separation state 失败：{error}") + })?; if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + app_log!("ui_separation.error stage=state_read reason=schema_mismatch"); return Err("不支持的 separation state schema".to_string()); } + app_log!( + "ui_separation.state_read.completed bytes={} trees={} bound={} problematic={}", + bytes.len(), + state.unprocessed_trees.len(), + state.bound.len(), + state.problematic_nodes.len() + ); Ok(state) } pub fn separation_dto(state: &SeparationState) -> SeparationDTO { + app_log!( + "ui_separation.dto bound_nodes={} problematic_nodes={} remaining_trees={}", + state.bound.len(), + state.problematic_nodes.len(), + state.unprocessed_trees.len() + ); SeparationDTO { bound_nodes: state.bound.clone(), problematic_nodes: state.problematic_nodes.clone(), diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index b600c48e1..fe8414e7f 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -1,5 +1,5 @@ use super::model::*; -use crate::ui_editor::component::{Component, image::ImageComponent}; +use crate::ui_editor::component::{image::ImageComponent, Component}; use crate::ui_editor::layout::node::Node; use crate::ui_editor::state::{State, UITree}; use crate::ui_editor::utils::NodeId; @@ -71,17 +71,35 @@ fn collect_todo_nodes( } pub fn construct_separation_state(state: &State) -> SeparationState { + app_log!( + "ui_separation.tree_construct.start ui_trees={} ui_images={}", + state.ui_trees.len(), + state.ui_design_images.len() + ); let unprocessed_trees = state .ui_trees .iter() .filter_map(|tree| { - let image = state.ui_design_images.get(&tree.src_ui_design)?; + let Some(image) = state.ui_design_images.get(&tree.src_ui_design) else { + app_log!( + "ui_separation.error stage=tree_construct reason=missing_ui_image image_id={}", + tree.src_ui_design.as_str() + ); + return None; + }; let ppu = image.pixels_per_unit.get(); let size = image.pixel_size / ppu; let root_rect = crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); let mut children = Vec::new(); collect_todo_nodes(&tree.root, &root_rect, ppu, &mut children); + app_log!( + "ui_separation.tree_construct.tree image_id={} todo_nodes={} pixel_width={} pixel_height={}", + tree.src_ui_design.as_str(), + count_nodes(&children), + image.pixel_size.x.round() as u32, + image.pixel_size.y.round() as u32 + ); (!children.is_empty()).then(|| SeparationTree { src_ui_design: tree.src_ui_design.clone(), root: SeparationNode { @@ -100,13 +118,25 @@ pub fn construct_separation_state(state: &State) -> SeparationState { }, }) }) - .collect(); - SeparationState { + .collect::>(); + let result = SeparationState { schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), unprocessed_trees, bound: Vec::new(), problematic_nodes: Vec::new(), - } + }; + app_log!( + "ui_separation.tree_construct.completed trees={}", + result.unprocessed_trees.len() + ); + result +} + +fn count_nodes(nodes: &[SeparationNode]) -> usize { + nodes + .iter() + .map(|node| 1 + count_nodes(&node.children)) + .sum() } pub fn next_leaf_batch(tree: &SeparationTree) -> Vec<&SeparationNode> { @@ -121,6 +151,11 @@ pub fn next_leaf_batch(tree: &SeparationTree) -> Vec<&SeparationNode> { } let mut output = Vec::new(); leaves(&tree.root, &mut output); + app_log!( + "ui_separation.batch_selected image_id={} leaf_nodes={}", + tree.src_ui_design.as_str(), + output.len() + ); output } @@ -128,6 +163,11 @@ pub fn validate_binding_response( response: &BindingResp, batch: &[&SeparationNode], ) -> Result<(), String> { + app_log!( + "ui_separation.binding_validate.start expected_nodes={} decisions={}", + batch.len(), + response.decisions.len() + ); let expected = batch .iter() .map(|node| node.id.clone()) @@ -140,9 +180,17 @@ pub fn validate_binding_response( } }; if !expected.contains(node_id) { + app_log!( + "ui_separation.error stage=binding_validate reason=unknown_node node_id={}", + node_id.as_str() + ); return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); } if !seen.insert(node_id.clone()) { + app_log!( + "ui_separation.error stage=binding_validate reason=duplicate_node node_id={}", + node_id.as_str() + ); return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); } if let BindingDecision::NeedRework { @@ -151,12 +199,25 @@ pub fn validate_binding_response( } = decision { if problem_description.trim().is_empty() { + app_log!( + "ui_separation.error stage=binding_validate reason=empty_problem_description node_id={}", + node_id.as_str() + ); return Err("NeedRework 必须包含问题描述".to_string()); } } } if seen.len() != expected.len() { + app_log!( + "ui_separation.error stage=binding_validate reason=incomplete_coverage expected={} seen={}", + expected.len(), + seen.len() + ); return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); } + app_log!( + "ui_separation.binding_validate.completed covered_nodes={}", + seen.len() + ); Ok(()) } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 19e1ee531..dabc5f1a7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -23,6 +23,12 @@ pub fn apply_batch_patch( decisions: &[BindingDecision], cut_paths: &std::collections::HashMap, ) -> Result<(), String> { + app_log!( + "ui_separation.batch_patch.start tree_index={} decisions={} cut_paths={}", + tree_index, + decisions.len(), + cut_paths.len() + ); let tree = state .unprocessed_trees .get_mut(tree_index) @@ -70,6 +76,14 @@ pub fn apply_batch_patch( } } remove_ids(&mut tree.root, &ids); + app_log!( + "ui_separation.batch_patch.completed tree_index={} removed_nodes={} bound={} problematic={} pending_root_children={}", + tree_index, + ids.len(), + state.bound.len(), + state.problematic_nodes.len(), + tree.root.children.len() + ); Ok(()) } @@ -79,6 +93,12 @@ fn mark_batch_problematic( batch: &[&SeparationNode], error: String, ) { + app_log!( + "ui_separation.batch_problematic tree_index={} nodes={} error={}", + tree_index, + batch.len(), + error + ); let ids = batch .iter() .map(|node| node.id.clone()) @@ -121,6 +141,12 @@ async fn raw_image_edit( width: u32, height: u32, ) -> Result { + app_log!( + "ui_separation.image_edit.start width={} height={} prompt_chars={}", + width, + height, + prompt.chars().count() + ); let (mime, data) = image_data_url .split_once(",") .ok_or_else(|| "界面图 data URL 无效".to_string())?; @@ -151,20 +177,37 @@ async fn raw_image_edit( ) .send() .await - .map_err(|e| format!("图片分离请求失败：{e}"))?; + .map_err(|e| { + app_log!("ui_separation.error stage=image_edit reason=send error={e}"); + format!("图片分离请求失败：{e}") + })?; if !response.status().is_success() { + app_log!( + "ui_separation.error stage=image_edit reason=http_status status={}", + response.status() + ); return Err(format!("图片分离请求失败（HTTP {}）", response.status())); } - let payload = response - .json::() - .await - .map_err(|e| format!("解析图片分离响应失败：{e}"))?; - payload + let payload = response.json::().await.map_err(|e| { + app_log!("ui_separation.error stage=image_edit reason=parse_response error={e}"); + format!("解析图片分离响应失败：{e}") + })?; + let result = payload .data .into_iter() .next() .map(|item| format!("data:image/png;base64,{}", item.b64_json)) - .ok_or_else(|| "图片分离响应没有图像".to_string()) + .ok_or_else(|| "图片分离响应没有图像".to_string()); + match &result { + Ok(value) => app_log!( + "ui_separation.image_edit.completed data_url_chars={}", + value.chars().count() + ), + Err(error) => { + app_log!("ui_separation.error stage=image_edit reason=empty_result error={error}") + } + } + result } async fn build_marked_image( @@ -172,6 +215,14 @@ async fn build_marked_image( nodes: Vec, target: PathBuf, ) -> Result { + app_log!( + "ui_separation.mark_image.start nodes={} target_file={}", + nodes.len(), + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or("") + ); tokio::task::spawn_blocking(move || { let areas = nodes .iter() @@ -207,6 +258,12 @@ fn build_marked_image_blocking( .to_rgba8(); let width = image.width(); let height = image.height(); + app_log!( + "ui_separation.mark_image.decoded nodes={} width={} height={}", + nodes.len(), + width, + height + ); for &(global_pos_x_px, global_pos_y_px, node_width_px, node_height_px) in nodes { let x0 = global_pos_x_px.min(width.saturating_sub(1)); let y0 = global_pos_y_px.min(height.saturating_sub(1)); @@ -244,13 +301,26 @@ fn build_marked_image_blocking( image::DynamicImage::ImageRgba8(image) .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) .map_err(|e| format!("编码标记图失败：{e}"))?; - Ok(format!( + let result = format!( "data:image/png;base64,{}", base64::engine::general_purpose::STANDARD.encode(png) - )) + ); + app_log!( + "ui_separation.mark_image.completed data_url_chars={}", + result.len() + ); + Ok(result) } async fn write_processed_image(processed_url: String, target: PathBuf) -> Result<(), String> { + app_log!( + "ui_separation.processed_image.write.start target_file={} data_url_chars={}", + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + processed_url.chars().count() + ); tokio::task::spawn_blocking(move || { let processed_bytes = base64::engine::general_purpose::STANDARD .decode( @@ -260,8 +330,19 @@ async fn write_processed_image(processed_url: String, target: PathBuf) -> Result .unwrap_or_default(), ) .map_err(|error| format!("解析处理图失败：{error}"))?; + let byte_len = processed_bytes.len(); fs::write(&target, processed_bytes) .map_err(|error| format!("写入处理图失败：{}: {error}", target.display())) + .map(|_| { + app_log!( + "ui_separation.processed_image.write.completed target_file={} bytes={}", + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + byte_len + ); + }) }) .await .map_err(|error| format!("写入处理图任务失败：{error}"))? @@ -272,12 +353,27 @@ async fn visual_binding( processed_url: String, nodes: &[&SeparationNode], ) -> Result { + app_log!( + "ui_separation.visual_binding.start nodes={} source_url_chars={} processed_url_chars={}", + nodes.len(), + source_url.chars().count(), + processed_url.chars().count() + ); let llm_config = load_game_creator_app_config() - .map_err(|e| e.to_string())? + .map_err(|e| { + app_log!("ui_separation.error stage=visual_binding reason=load_config error={e}"); + e.to_string() + })? .llm; - let client = build_game_creator_llm_client_from_llm_config(&llm_config, "llm") - .map_err(|e| e.to_string())?; - let schema = strict_json_schema::()?; + let client = + build_game_creator_llm_client_from_llm_config(&llm_config, "llm").map_err(|e| { + app_log!("ui_separation.error stage=visual_binding reason=build_client error={e}"); + e.to_string() + })?; + let schema = strict_json_schema::().map_err(|error| { + app_log!("ui_separation.error stage=visual_binding reason=build_schema error={error}"); + error + })?; let tool = LlmFunctionTool::new( "bind_ui_elements", "确认处理图中的区域对应哪些 UI 节点", @@ -285,7 +381,7 @@ async fn visual_binding( ) .with_strict(true); let base_prompt = gen_binding_prompt(nodes.to_vec()); - request_with_feedback( + let result = request_with_feedback( 2, |feedback| { let prompt = feedback.map_or_else( @@ -333,7 +429,19 @@ async fn visual_binding( }, |value: &BindingResp| validate_binding_response(value, nodes), ) - .await + .await; + match &result { + Ok(value) => app_log!( + "ui_separation.visual_binding.completed nodes={} decisions={}", + nodes.len(), + value.decisions.len() + ), + Err(error) => app_log!( + "ui_separation.error stage=visual_binding reason=failed nodes={} error={error}", + nodes.len() + ), + } + result } pub(crate) async fn separate_ui_impl( @@ -341,25 +449,85 @@ pub(crate) async fn separate_ui_impl( asset_id: String, state: State, ) -> Result { + app_log!( + "ui_separation.start asset_id={} ui_trees={} ui_images={} sprites={}", + asset_id, + state.ui_trees.len(), + state.ui_design_images.len(), + state.sprite_assets.len() + ); let session = current_platform_session().ok_or_else(|| "请先登录平台账号".to_string())?; let root = Path::new(project_path.trim()); - let sidecar = separation_sidecar_dir(root, &asset_id)?; - fs::create_dir_all(&sidecar).map_err(|e| format!("创建 separation sidecar 失败：{e}"))?; + let sidecar = separation_sidecar_dir(root, &asset_id).map_err(|error| { + app_log!( + "ui_separation.error stage=sidecar_dir asset_id={} error={error}", + asset_id + ); + error + })?; + fs::create_dir_all(&sidecar).map_err(|e| { + app_log!( + "ui_separation.error stage=sidecar_create asset_id={} error={e}", + asset_id + ); + format!("创建 separation sidecar 失败：{e}") + })?; let state_path = sidecar.join("state.json"); - let mut separation = if state_path.exists() { - read_separation_state(&state_path)? + let restored = state_path.exists(); + let mut separation = if restored { + app_log!("ui_separation.state_restore.start asset_id={}", asset_id); + read_separation_state(&state_path).map_err(|error| { + app_log!( + "ui_separation.error stage=state_restore asset_id={} error={error}", + asset_id + ); + error + })? } else { + app_log!("ui_separation.state_construct.start asset_id={}", asset_id); construct_separation_state(&state) }; + app_log!( + "ui_separation.state_ready asset_id={} restored={} trees={} bound={} problematic={}", + asset_id, + restored, + separation.unprocessed_trees.len(), + separation.bound.len(), + separation.problematic_nodes.len() + ); for tree_index in 0..separation.unprocessed_trees.len() { let tree = &separation.unprocessed_trees[tree_index]; + let image_id = tree.src_ui_design.clone(); let image = state .ui_design_images - .get(&tree.src_ui_design) + .get(&image_id) .ok_or_else(|| "缺少源界面图".to_string())?; let source_path = crate::project::resolve_local_project_path(root, &image.path)?; - let source_url = read_ui_reference_image_data_url(source_path).await?; - write_separation_state(&state_path, &separation)?; + let source_url = read_ui_reference_image_data_url(source_path) + .await + .map_err(|error| { + app_log!( + "ui_separation.error stage=read_source tree_index={} image_id={} error={error}", + tree_index, + image_id.as_str() + ); + error + })?; + app_log!( + "ui_separation.tree_start tree_index={} image_id={} width={} height={}", + tree_index, + image_id.as_str(), + image.pixel_size.x.round() as u32, + image.pixel_size.y.round() as u32 + ); + write_separation_state(&state_path, &separation).map_err(|error| { + app_log!( + "ui_separation.error stage=state_checkpoint tree_index={} error={error}", + tree_index + ); + error + })?; + let mut batch_index = 0usize; loop { let Some(current_tree) = separation.unprocessed_trees.get(tree_index) else { break; @@ -369,11 +537,26 @@ pub(crate) async fn separate_ui_impl( .cloned() .collect::>(); if batch_nodes.is_empty() { + app_log!( + "ui_separation.tree_completed tree_index={} image_id={} bound={} problematic={}", + tree_index, + image_id.as_str(), + separation.bound.len(), + separation.problematic_nodes.len() + ); break; } let batch = batch_nodes.iter().collect::>(); let prompt = gen_extract_prompt(batch.iter().map(|n| n.note.clone()).collect::>()); + app_log!( + "ui_separation.batch_start tree_index={} batch_index={} nodes={} prompt_chars={} rework_total={}", + tree_index, + batch_index, + batch.len(), + prompt.chars().count(), + batch.iter().map(|node| node.rework_count).sum::() + ); let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); let marked_url = match build_marked_image( source_url.clone(), @@ -384,8 +567,14 @@ pub(crate) async fn separate_ui_impl( { Ok(value) => value, Err(error) => { + app_log!( + "ui_separation.error stage=mark_image tree_index={} batch_index={} error={error}", + tree_index, + batch_index + ); mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; + batch_index += 1; continue; } }; @@ -400,21 +589,51 @@ pub(crate) async fn separate_ui_impl( { Ok(value) => value, Err(error) => { + app_log!( + "ui_separation.error stage=image_edit tree_index={} batch_index={} error={error}", + tree_index, + batch_index + ); mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; + batch_index += 1; continue; } }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); - write_processed_image(processed_url.clone(), processed_path.clone()).await?; + if let Err(error) = + write_processed_image(processed_url.clone(), processed_path.clone()).await + { + app_log!( + "ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", + tree_index, + batch_index + ); + mark_batch_problematic(&mut separation, tree_index, &batch, error); + write_separation_state(&state_path, &separation)?; + batch_index += 1; + continue; + } let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { Ok(value) => value, Err(error) => { + app_log!( + "ui_separation.error stage=visual_binding tree_index={} batch_index={} error={error}", + tree_index, + batch_index + ); mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; + batch_index += 1; continue; } }; + app_log!( + "ui_separation.binding_decisions tree_index={} batch_index={} decisions={}", + tree_index, + batch_index, + binding.decisions.len() + ); let mut cut_paths = std::collections::HashMap::new(); let mut cut_error = None; for decision in &binding.decisions { @@ -436,6 +655,12 @@ pub(crate) async fn separate_ui_impl( .insert(to_node.clone(), cut_path.to_string_lossy().to_string()); } Err(error) => { + app_log!( + "ui_separation.error stage=cut_image tree_index={} batch_index={} node_id={} error={error}", + tree_index, + batch_index, + to_node.as_str() + ); cut_error = Some(format!("节点 {} 的分离区域无效：{error}", to_node.as_str())); break; @@ -444,15 +669,48 @@ pub(crate) async fn separate_ui_impl( } } if let Some(error) = cut_error { + app_log!( + "ui_separation.error stage=cut_batch tree_index={} batch_index={} error={error}", + tree_index, + batch_index + ); mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; + batch_index += 1; continue; } apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; write_separation_state(&state_path, &separation)?; + app_log!( + "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={}", + tree_index, + batch_index, + cut_paths.len(), + separation.bound.len(), + separation.problematic_nodes.len() + ); + batch_index += 1; } } - let _ = fs::remove_file(state_path); + match fs::remove_file(&state_path) { + Ok(()) => app_log!("ui_separation.state_removed asset_id={}", asset_id), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + app_log!( + "ui_separation.state_remove_skipped asset_id={} reason=not_found", + asset_id + ) + } + Err(error) => app_log!( + "ui_separation.error stage=state_remove asset_id={} error={error}", + asset_id + ), + } + app_log!( + "ui_separation.completed asset_id={} bound_nodes={} problematic_nodes={}", + asset_id, + separation.bound.len(), + separation.problematic_nodes.len() + ); Ok(separation_dto(&separation)) } @@ -461,6 +719,21 @@ async fn cut_processed_image( area: BindingArea, target: PathBuf, ) -> Result<(), String> { + app_log!( + "ui_separation.cut_image.start source_file={} target_file={} area=({}, {}, {}, {})", + source + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + area.global_pos_x_px, + area.global_pos_y_px, + area.width_px, + area.height_px + ); tokio::task::spawn_blocking(move || cut_processed_image_blocking(&source, &area, &target)) .await .map_err(|error| format!("裁切处理图任务失败：{error}"))? @@ -481,7 +754,17 @@ fn cut_processed_image_blocking( ); cropped .save_with_format(target, ImageFormat::Png) - .map_err(|e| format!("写入 cut 图片失败：{e}")) + .map_err(|e| format!("写入 cut 图片失败：{e}"))?; + app_log!( + "ui_separation.cut_image.completed target_file={} width={} height={}", + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + area.width_px, + area.height_px + ); + Ok(()) } fn remove_ids(node: &mut SeparationNode, ids: &std::collections::HashSet) { From 592b6b4ee72d510631f9f4613a17f1fa81549e91 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 12:36:16 +0800 Subject: [PATCH 050/248] =?UTF-8?q?=E7=AE=80=E5=8C=96=E8=AF=86=E5=88=AB?= =?UTF-8?q?=E6=A0=91=E9=AA=8C=E8=AF=81=E6=B5=8B=E8=AF=95=E4=B8=AD=E7=9A=84?= =?UTF-8?q?=E6=96=AD=E8=A8=80=E6=A0=BC=E5=BC=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/recognition.rs | 32 ++++++++----------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index 6671e9c83..613be637d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -476,23 +476,19 @@ mod tests { let oversized = (0..=MAX_RECOGNITION_TREE_NODES) .map(|_| serde_json::json!({"children": []})) .collect::>(); - assert!( - validate_recognition_response_shape(&serde_json::json!({ - "trees": [{"children": oversized}] - })) - .is_err() - ); + assert!(validate_recognition_response_shape(&serde_json::json!({ + "trees": [{"children": oversized}] + })) + .is_err()); let mut nested = serde_json::json!({"children": []}); for _ in 0..MAX_RECOGNITION_TREE_DEPTH { nested = serde_json::json!({"children": [nested]}); } - assert!( - validate_recognition_response_shape(&serde_json::json!({ - "trees": [{"children": [nested]}] - })) - .is_err() - ); + assert!(validate_recognition_response_shape(&serde_json::json!({ + "trees": [{"children": [nested]}] + })) + .is_err()); } #[test] @@ -612,13 +608,11 @@ mod tests { children: Vec::new(), }; - assert!( - validate_tree_image_ids( - &[tree(page.clone()), tree(slave.clone())], - &[page.clone(), slave.clone()], - ) - .is_ok() - ); + assert!(validate_tree_image_ids( + &[tree(page.clone()), tree(slave.clone())], + &[page.clone(), slave.clone()], + ) + .is_ok()); assert!( validate_tree_image_ids(&[tree(page.clone())], &[page.clone(), slave.clone()]).is_err() ); From 418d5732deda308f16f3c83030c18ba473678470 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 12:52:39 +0800 Subject: [PATCH 051/248] =?UTF-8?q?=E8=B0=83=E6=95=B4UI=E8=AF=86=E5=88=AB?= =?UTF-8?q?=E6=8C=87=E4=BB=A4=EF=BC=8C=E7=BB=86=E5=8C=96=E7=B2=92=E5=BA=A6?= =?UTF-8?q?=E8=A6=81=E6=B1=82=E5=B9=B6=E4=BC=98=E5=8C=96=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E8=BF=94=E5=9B=9E=E8=A7=84=E5=88=99?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/recognition.rs | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index 613be637d..ccfb53ac3 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -28,16 +28,13 @@ const MAX_RECOGNITION_TREE_NODES: usize = 512; const MAX_RECOGNITION_TREE_DEPTH: usize = 32; const SYSTEM_PROMPT: &str = r#" -角色： -你是游戏 UI 多图结构识别器。 - 任务： 同时分析同一 UI 系统的全部参考图，建立UI树 用户会给你一些UI截图(它们从属于同一个UI系统)和对应的元数据, 请用给定的工具描述UI结构 识别规则： -* 只识别 UI，不识别场景人物、地形、建筑、光影和背景装饰。 +* 只识别 UI元素. 要区分动态内容, 不要白费力气识别应该由程序生成/绘制的内容.(此类内容应该用一个整体节点+自然语言描述) 除此之外必须完整包含所有元素,结构. * 无法确定类型、层级、关系时，在 UnSure 中写明原因。 * 返回的 trees 必须与输入图片一一对应，每张输入图片只能有一棵树，不能合并多张图片的树。 每棵树的 src_ui_design_image_id 必须等于对应输入图片标注的 id。 * 每棵树必须使用自己的输入图片原始像素坐标系(0,0 as left top)输出 @@ -46,10 +43,13 @@ const SYSTEM_PROMPT: &str = r#" * 面向用户的字段如名称描述等请用中文 * 由于每个截图未必是完整的, 可能是局部的, 每棵树描述清楚每个截图上UI的层次结构即可 * 不同树的共用框架/层次/...请使用使用相同的名称描述. 不同状态/变体名称使用相同的前缀, 用后缀区别 -* 粒度要求: 以可交互,方便程序化控制的最小单位为准: 进度条的底槽、填充和外框; slider的底槽, dragger等. -* 为每个节点直接返回完整 components。纯容器返回空数组；目前我们只做识别, 不要求图片字体参数 -* 每个节点当前最多返回一个 Image component 和一个 Text component。 -* 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. +* 粒度要求: 尽可能细致, 以可交互,方便程序化控制的最小单位为准. 包括不限于: icon, 进度条的底槽、填充和外框; slider的底槽, dragger等. +* 为每个节点直接返回完整 components. + 无背景的逻辑容器返回空数组. + 有背景的容器推荐使用Simple+不锁定宽高比的Image component. + 目前我们只做识别, 不要求图片字体参数. + 每个节点当前最多返回一个 Image component 和一个 Text component。 + 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. "#; From 11d4d3d930393e53ca61581b4de3a7127bb3b790 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 13:17:36 +0800 Subject: [PATCH 052/248] =?UTF-8?q?=E9=80=82=E9=85=8D=20UI=20=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E5=9B=BE=E7=89=87=E7=BC=96=E8=BE=91=20multipart=20?= =?UTF-8?q?=E6=8E=A5=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将标记图 data URL 解码为 PNG 文件部件。 按 Raw GPT Image 2 新合同发送 prompt、尺寸和输出参数。 保留响应 data[].b64_json 解析与 separation 流程不变。 --- .../ui_editor/commands/separation/workflow.rs | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index dabc5f1a7..cf03003a7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -154,6 +154,15 @@ async fn raw_image_edit( .strip_prefix("data:") .and_then(|v| v.strip_suffix(";base64")) .unwrap_or("image/png"); + if !mime.eq_ignore_ascii_case("image/png") { + return Err("图片分离请求只支持 PNG 标记图".to_string()); + } + let image_bytes = base64::engine::general_purpose::STANDARD + .decode(data.trim()) + .map_err(|error| format!("解码标记图失败：{error}"))?; + if image_bytes.is_empty() { + return Err("标记图不能为空".to_string()); + } let client = crate::http_client::agc_main_site_client_builder() .build() .map_err(|e| format!("创建图片编辑客户端失败：{e}"))?; @@ -161,19 +170,22 @@ async fn raw_image_edit( "{}/api/raw/v1/images/edit", session.api_base_url.trim_end_matches('/') ); - let body = serde_json::json!({ - "image": {"data": data, "mimeType": mime}, - "prompt": prompt, - "width": width, - "height": height, - "output_format": "png", - "background": "transparent" - }); + let image_part = reqwest::multipart::Part::bytes(image_bytes) + .file_name("image.png") + .mime_str("image/png") + .map_err(|error| format!("构造图片编辑文件部件失败：{error}"))?; + let body = reqwest::multipart::Form::new() + .part("image", image_part) + .text("prompt", prompt.to_string()) + .text("width", width.to_string()) + .text("height", height.to_string()) + .text("output_format", "png") + .text("background", "transparent"); let response = crate::http_client::with_agc_main_site_marker( client .post(url) .bearer_auth(&session.access_token) - .json(&body), + .multipart(body), ) .send() .await From bc5398894ea0d9dc9a58e410beeb6bf2fbb1cdd1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 14:29:38 +0800 Subject: [PATCH 053/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=91=E4=B8=8E=E6=89=B9=E6=AC=A1=E7=AE=97=E6=B3=95=E7=BA=A6?= =?UTF-8?q?=E5=AE=9A=20=E6=98=8E=E7=A1=AE=E7=9C=9F=E5=AE=9E=E6=A0=B9?= =?UTF-8?q?=E8=8A=82=E7=82=B9=E3=80=81=E9=80=BB=E8=BE=91=E5=8F=B6=E5=92=8C?= =?UTF-8?q?=E9=9D=9E=E9=87=8D=E5=8F=A0=E8=B4=AA=E5=BF=83=E9=80=89=E6=8B=A9?= =?UTF-8?q?=20=E8=A1=A5=E5=85=85=E7=88=B6=E8=8A=82=E7=82=B9=E7=B4=AB?= =?UTF-8?q?=E8=89=B2=E9=87=8D=E5=BB=BA=E4=B8=8E=E9=94=99=E8=AF=AF=E6=81=A2?= =?UTF-8?q?=E5=A4=8D=E8=BE=B9=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...术方案】UI编辑器自动分离工作流-2026-09-08.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index e967abc21..e4533dd58 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -17,24 +17,26 @@ ## Separation tree -- recognition 完成后由 UI tree 构造临时 separation tree。 -- 纯节点、纯 Text 节点和不需要切图的节点在构造时过滤；被过滤节点的可处理 children 向上透传。 -- separation tree 只保留真实待处理节点。 -- 一个 batch 是整页当前所有互不重叠叶节点。 +- recognition 完成后由 UI tree 构造临时 separation tree；每棵树保留原 UI tree 的 `src_ui_design` 与真实 `root`，不生成 synthetic root。 +- 非 root 的纯容器、纯 Text 节点和不需要切图的节点在构造时过滤，被过滤节点的 children 向上透传。真实 root 始终保留；`root_extractable` 表示 root 是否含未绑定图片组件并可作为候选。 +- 节点的 `children` 在整个 workflow 中始终保留，不能因处理成功或失败而从树上删除。节点终态由 `bound`、`problematic_nodes` 反查；两者均不存在时仍待处理，`rework_count` 仅记录视觉模型返工次数。 +- 逻辑叶必须是未终止、可处理且所有 children 都已终止的节点；root 在 `root_extractable=true` 时按普通节点参与，否则只递归其 children。 +- 候选按 DFS 和 children 原顺序遍历，使用简单贪心选择与已选矩形无正面积交集的节点组成 batch；边或角接触不算重叠，不做面积或偏差排序检查。正常树结构下有候选时至少选中一个。 +- 一个 batch 是当前树中整批互不重叠的逻辑叶节点。 - 一个 batch 的最小处理单元是：一次 image-edit + 一次 visual binding。 -- batch 成功后从 pending tree 移除对应叶节点，并把结果放入 bound 容器；失败节点移入 problematic 容器，流程继续消费剩余树。 +- batch 成功后只把结果追加到 bound 容器，失败节点在达到返工上限后追加到 problematic 容器；树拓扑不变，流程继续消费剩余树。 - 不额外维护节点状态枚举；节点是否仍在 pending tree、`rework_count` 和 problematic 容器共同表达状态。 ## 图片编辑与视觉绑定 -- image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。 +- image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。处理父节点时，紫色填充其 children 的矩形区域（包括已 problematic 的 children），再在父节点自身外围绘制绿色框；绿色框覆盖在紫色之上。叶节点只绘制绿色框，不填充自身。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 - 标记图构建、处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 - `NeedRework` 携带短问题描述。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 -- 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。 +- 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 - 父节点背景重建由 image-edit 模型完成，不由 Rust 硬编码重建算法完成。 ## 临时 sidecar @@ -42,7 +44,7 @@ - separation 状态不写入 UI JSON，也不进入 manifest。 - sidecar 目录按 UI manifest `asset_id` 生成，复用 `generated_file_stem(asset_id)` 的安全字符替换和 SHA-256 摘要规则，位于项目 `ui/` 下。 - 目录只保存一份当前 separation state，而不是每 batch 一个状态文件。 -- state 文件只保留 `schema_version`、pending tree、bound 结果和 problematic 节点，不重复保存 `projectId / assetId / uiStateRevision`。 +- state 文件只保留 `schema_version`、separation trees、bound 结果和 problematic 节点，不重复保存 `projectId / assetId / uiStateRevision`。 - sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 - 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 - 临时图片可跨重启保留。raw image-edit 返回图、绿色/紫色标记图、处理图和 cut 图片当前都保留用于 debug；理论上只应在内存中，清理/归档策略列 TODO。 From 26b9ad7fad0eabb89b890785fa8bc72b9cbe224f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 14:48:18 +0800 Subject: [PATCH 054/248] =?UTF-8?q?=E9=87=8D=E6=9E=84=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=91=E4=B8=8E=E9=9D=9E=E9=87=8D=E5=8F=A0=E6=89=B9=E6=AC=A1?= =?UTF-8?q?=E9=80=89=E6=8B=A9=20=E4=BF=9D=E7=95=99=E7=9C=9F=E5=AE=9E?= =?UTF-8?q?=E6=A0=B9=E8=8A=82=E7=82=B9=E5=B9=B6=E8=AE=B0=E5=BD=95=20root?= =?UTF-8?q?=5Fextractable=20=E6=8C=89=E7=BB=88=E6=80=81=E5=8F=8D=E6=9F=A5?= =?UTF-8?q?=E5=92=8C=20DFS=20=E8=B4=AA=E5=BF=83=E7=AD=9B=E9=80=89=E9=80=BB?= =?UTF-8?q?=E8=BE=91=E5=8F=B6=20=E6=8B=86=E5=87=BA=E7=88=B6=E5=AD=90?= =?UTF-8?q?=E5=8C=BA=E5=9F=9F=E6=A0=87=E8=AE=B0=E5=9B=BE=E7=94=9F=E6=88=90?= =?UTF-8?q?=E6=A8=A1=E5=9D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../ui_editor/commands/separation/marker.rs | 52 ++++ .../src/ui_editor/commands/separation/mod.rs | 16 +- .../ui_editor/commands/separation/model.rs | 3 +- .../commands/separation/persistence.rs | 8 +- .../src/ui_editor/commands/separation/tree.rs | 256 ++++++------------ .../ui_editor/commands/separation/workflow.rs | 193 ++----------- 6 files changed, 174 insertions(+), 354 deletions(-) create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs new file mode 100644 index 000000000..f39015389 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -0,0 +1,52 @@ +use super::model::SeparationNode; +use base64::Engine as _; +use image::GenericImage; +use std::path::Path; +use std::path::PathBuf; + +pub async fn build_marked_image(source_url: String, nodes: Vec, target: PathBuf) -> Result { + tokio::task::spawn_blocking(move || build_marked_image_blocking(&source_url, &nodes, &target)) + .await + .map_err(|error| format!("构建标记图任务失败：{error}"))? +} + +fn build_marked_image_blocking(source_url: &str, nodes: &[SeparationNode], target: &Path) -> Result { + let encoded = source_url.split_once(',').map(|(_, d)| d).ok_or_else(|| "源图 data URL 无效".to_string())?; + let bytes = base64::engine::general_purpose::STANDARD.decode(encoded).map_err(|e| format!("解码源图失败：{e}"))?; + let mut image = image::load_from_memory(&bytes).map_err(|e| format!("读取源图失败：{e}"))?.to_rgba8(); + let width = image.width(); + let height = image.height(); + // Purple reconstruction comes first so the parent green frame remains visible on top. + for node in nodes { + for child in &node.children { + fill_rect(&mut image, child.global_pos_x_px, child.global_pos_y_px, child.width_px, child.height_px, image::Rgba([180, 0, 180, 120]), width, height); + } + } + for node in nodes { + draw_frame(&mut image, node.global_pos_x_px, node.global_pos_y_px, node.width_px, node.height_px, width, height); + } + image::DynamicImage::ImageRgba8(image.clone()).save_with_format(target, image::ImageFormat::Png).map_err(|e| format!("写入标记图失败：{e}"))?; + let mut png = Vec::new(); + image::DynamicImage::ImageRgba8(image).write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png).map_err(|e| format!("编码标记图失败：{e}"))?; + Ok(format!("data:image/png;base64,{}", base64::engine::general_purpose::STANDARD.encode(png))) +} + +fn clipped_rect(x: u32, y: u32, w: u32, h: u32, width: u32, height: u32) -> Option<(u32, u32, u32, u32)> { + if width == 0 || height == 0 || w == 0 || h == 0 { return None; } + let x0 = x.min(width - 1); let y0 = y.min(height - 1); + let x1 = x.saturating_add(w).min(width).saturating_sub(1); + let y1 = y.saturating_add(h).min(height).saturating_sub(1); + (x0 <= x1 && y0 <= y1).then_some((x0, y0, x1, y1)) +} + +fn fill_rect(image: &mut image::RgbaImage, x: u32, y: u32, w: u32, h: u32, color: image::Rgba, width: u32, height: u32) { + let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { return; }; + for yy in y0..=y1 { for xx in x0..=x1 { image.put_pixel(xx, yy, color); } } +} + +fn draw_frame(image: &mut image::RgbaImage, x: u32, y: u32, w: u32, h: u32, width: u32, height: u32) { + let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { return; }; + let green = image::Rgba([0, 255, 0, 255]); + for xx in x0..=x1 { image.put_pixel(xx, y0, green); image.put_pixel(xx, y1, green); } + for yy in y0..=y1 { image.put_pixel(x0, yy, green); image.put_pixel(x1, yy, green); } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 708d3c191..41f24538a 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -1,4 +1,5 @@ mod model; +mod marker; mod persistence; mod prompt; mod tree; @@ -9,6 +10,7 @@ pub use persistence::*; pub use tree::*; pub use workflow::apply_batch_patch; pub(crate) use workflow::separate_ui_impl; +pub(crate) use marker::build_marked_image; #[cfg(test)] mod tests { use super::*; @@ -88,13 +90,13 @@ mod tests { ); let result = construct_separation_state(&state(root)); assert_eq!( - result.unprocessed_trees[0].root.children[0].id.as_str(), + result.trees[0].root.children[0].id.as_str(), "image" ); } #[test] - fn construction_uses_distinct_root_id_for_root_image() { + fn construction_keeps_real_root_for_root_image() { let image = Component::Image(ImageComponent { target_graphic: None, image_type: ImageType::Simple { @@ -103,9 +105,9 @@ mod tests { }); let root = node("root-image", vec![image], vec![]); let result = construct_separation_state(&state(root)); - let tree = &result.unprocessed_trees[0]; - assert_ne!(tree.root.id, tree.root.children[0].id); - assert_eq!(tree.root.children[0].id.as_str(), "root-image"); + let tree = &result.trees[0]; + assert_eq!(tree.root.id.as_str(), "root-image"); + assert!(tree.root_extractable); } #[test] fn binding_validation_requires_exact_batch_coverage() { @@ -131,7 +133,7 @@ mod tests { assert!(dir.to_string_lossy().ends_with("-separation")); } #[test] - fn patch_collects_bound_and_removes_leaf() { + fn patch_collects_bound_and_keeps_tree_topology() { let image = Component::Image(ImageComponent { target_graphic: None, image_type: ImageType::Simple { @@ -156,6 +158,6 @@ mod tests { let paths = HashMap::from([(id.clone(), "ui/.sidecar/cut.png".to_string())]); apply_batch_patch(&mut state, 0, &decisions, &paths).unwrap(); assert_eq!(state.bound[0].node_id, id); - assert!(state.unprocessed_trees[0].root.children.is_empty()); + assert_eq!(state.trees[0].root.children.len(), 1); } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs index 87ffc1832..a9a026b6a 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs @@ -49,6 +49,7 @@ impl SeparationNode { pub struct SeparationTree { pub src_ui_design: UIDesignImageId, pub root: SeparationNode, + pub root_extractable: bool, } #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] @@ -67,7 +68,7 @@ pub struct ProblematicNode { #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] pub struct SeparationState { pub schema_version: String, - pub unprocessed_trees: Vec, + pub trees: Vec, pub bound: Vec, pub problematic_nodes: Vec, } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 91ea757f0..e9d7c9296 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -35,7 +35,7 @@ pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<() path.file_name() .and_then(|name| name.to_str()) .unwrap_or(""), - state.unprocessed_trees.len(), + state.trees.len(), state.bound.len(), state.problematic_nodes.len() ); @@ -68,7 +68,7 @@ pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<() fs::metadata(path) .map(|metadata| metadata.len()) .unwrap_or(0), - state.unprocessed_trees.len(), + state.trees.len(), state.bound.len(), state.problematic_nodes.len() ); @@ -97,7 +97,7 @@ pub fn read_separation_state(path: &Path) -> Result { app_log!( "ui_separation.state_read.completed bytes={} trees={} bound={} problematic={}", bytes.len(), - state.unprocessed_trees.len(), + state.trees.len(), state.bound.len(), state.problematic_nodes.len() ); @@ -109,7 +109,7 @@ pub fn separation_dto(state: &SeparationState) -> SeparationDTO { "ui_separation.dto bound_nodes={} problematic_nodes={} remaining_trees={}", state.bound.len(), state.problematic_nodes.len(), - state.unprocessed_trees.len() + state.trees.len() ); SeparationDTO { bound_nodes: state.bound.clone(), diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index fe8414e7f..5c84af6db 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -1,31 +1,24 @@ use super::model::*; use crate::ui_editor::component::{image::ImageComponent, Component}; use crate::ui_editor::layout::node::Node; -use crate::ui_editor::state::{State, UITree}; +use crate::ui_editor::state::State; use crate::ui_editor::utils::NodeId; +use std::collections::HashSet; + fn is_unbound_image(node: &Node) -> bool { node.components.iter().any(|component| { - matches!( - component, - Component::Image(ImageComponent { - target_graphic: None, - .. - }) - ) + matches!(component, Component::Image(ImageComponent { target_graphic: None, .. })) }) } -fn node_pixel_rect( - node: &Node, - parent: &crate::ui_editor::layout::dimension::UIRect, - ppu: f32, -) -> (u32, u32, u32, u32) { +fn node_pixel_rect(node: &Node, parent: &crate::ui_editor::layout::dimension::UIRect, ppu: f32) -> (u32, u32, u32, u32) { let rect = node.layout.transform.resolve(parent); - let x = (rect.min.x * ppu).max(0.0).round() as u32; - let y = (rect.min.y * ppu).max(0.0).round() as u32; - let w = (rect.size.x * ppu).max(0.0).round() as u32; - let h = (rect.size.y * ppu).max(0.0).round() as u32; - (x, y, w, h) + ( + (rect.min.x * ppu).max(0.0).round() as u32, + (rect.min.y * ppu).max(0.0).round() as u32, + (rect.size.x * ppu).max(0.0).round() as u32, + (rect.size.y * ppu).max(0.0).round() as u32, + ) } fn node_description(node: &Node) -> String { @@ -39,31 +32,19 @@ fn node_description(node: &Node) -> String { } } -fn collect_todo_nodes( - node: &Node, - parent: &crate::ui_editor::layout::dimension::UIRect, - ppu: f32, - output: &mut Vec, -) { - let mut children = Vec::new(); +fn collect_todo_nodes(node: &Node, parent: &crate::ui_editor::layout::dimension::UIRect, ppu: f32, output: &mut Vec) { let rect = node.layout.transform.resolve(parent); + let mut children = Vec::new(); for child in &node.children { collect_todo_nodes(child, &rect, ppu, &mut children); } if is_unbound_image(node) { let (x, y, w, h) = node_pixel_rect(node, parent, ppu); output.push(SeparationNode { - id: node.id.clone(), - global_pos_x_px: x, - global_pos_y_px: y, - width_px: w, - height_px: h, - note: SeparationNote { - description: node_description(node), - text_note: String::new(), - }, - children, - rework_count: 0, + id: node.id.clone(), global_pos_x_px: x, global_pos_y_px: y, + width_px: w, height_px: h, + note: SeparationNote { description: node_description(node), text_note: String::new() }, + children, rework_count: 0, }); } else { output.extend(children); @@ -71,153 +52,82 @@ fn collect_todo_nodes( } pub fn construct_separation_state(state: &State) -> SeparationState { - app_log!( - "ui_separation.tree_construct.start ui_trees={} ui_images={}", - state.ui_trees.len(), - state.ui_design_images.len() - ); - let unprocessed_trees = state - .ui_trees - .iter() - .filter_map(|tree| { - let Some(image) = state.ui_design_images.get(&tree.src_ui_design) else { - app_log!( - "ui_separation.error stage=tree_construct reason=missing_ui_image image_id={}", - tree.src_ui_design.as_str() - ); - return None; - }; - let ppu = image.pixels_per_unit.get(); - let size = image.pixel_size / ppu; - let root_rect = - crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); - let mut children = Vec::new(); - collect_todo_nodes(&tree.root, &root_rect, ppu, &mut children); - app_log!( - "ui_separation.tree_construct.tree image_id={} todo_nodes={} pixel_width={} pixel_height={}", - tree.src_ui_design.as_str(), - count_nodes(&children), - image.pixel_size.x.round() as u32, - image.pixel_size.y.round() as u32 - ); - (!children.is_empty()).then(|| SeparationTree { - src_ui_design: tree.src_ui_design.clone(), - root: SeparationNode { - // The synthetic root must never share an ID with a real - // UI node. A single-image design may use the original - // tree root as an eligible separation leaf. - id: NodeId::new(format!("separation-root-{}", uuid::Uuid::new_v4().simple())) - .expect("synthetic separation root id is valid"), - global_pos_x_px: 0, - global_pos_y_px: 0, - width_px: image.pixel_size.x.max(0.0).round() as u32, - height_px: image.pixel_size.y.max(0.0).round() as u32, - note: SeparationNote::default(), - children, - rework_count: 0, - }, - }) + let trees = state.ui_trees.iter().filter_map(|tree| { + let image = state.ui_design_images.get(&tree.src_ui_design)?; + let ppu = image.pixels_per_unit.get(); + let size = image.pixel_size / ppu; + let root_rect = crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); + let mut children = Vec::new(); + for child in &tree.root.children { + collect_todo_nodes(child, &root_rect, ppu, &mut children); + } + let root_extractable = is_unbound_image(&tree.root); + if !root_extractable && children.is_empty() { return None; } + let (x, y, w, h) = node_pixel_rect(&tree.root, &root_rect, ppu); + Some(SeparationTree { + src_ui_design: tree.src_ui_design.clone(), + root: SeparationNode { + id: tree.root.id.clone(), global_pos_x_px: x, global_pos_y_px: y, + width_px: w, height_px: h, + note: SeparationNote { description: node_description(&tree.root), text_note: String::new() }, + children, rework_count: 0, + }, + root_extractable, }) - .collect::>(); - let result = SeparationState { - schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), - unprocessed_trees, - bound: Vec::new(), - problematic_nodes: Vec::new(), - }; - app_log!( - "ui_separation.tree_construct.completed trees={}", - result.unprocessed_trees.len() - ); - result + }).collect(); + SeparationState { schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), trees, bound: Vec::new(), problematic_nodes: Vec::new() } } -fn count_nodes(nodes: &[SeparationNode]) -> usize { - nodes - .iter() - .map(|node| 1 + count_nodes(&node.children)) - .sum() +fn terminal_ids(state: &SeparationState) -> HashSet { + state.bound.iter().map(|n| n.node_id.clone()) + .chain(state.problematic_nodes.iter().map(|n| n.node_id.clone())).collect() } -pub fn next_leaf_batch(tree: &SeparationTree) -> Vec<&SeparationNode> { - fn leaves<'a>(node: &'a SeparationNode, output: &mut Vec<&'a SeparationNode>) { - if node.children.is_empty() { - output.push(node); - } else { - for child in &node.children { - leaves(child, output); - } - } +fn logical_leaves<'a>(node: &'a SeparationNode, extractable: bool, terminal: &HashSet, output: &mut Vec<&'a SeparationNode>) { + let is_terminal = terminal.contains(&node.id); + let children_terminal = node.children.iter().all(|child| terminal.contains(&child.id)); + if extractable && !is_terminal && children_terminal { + output.push(node); + return; + } + for child in &node.children { + logical_leaves(child, true, terminal, output); } - let mut output = Vec::new(); - leaves(&tree.root, &mut output); - app_log!( - "ui_separation.batch_selected image_id={} leaf_nodes={}", - tree.src_ui_design.as_str(), - output.len() - ); - output } -pub fn validate_binding_response( - response: &BindingResp, - batch: &[&SeparationNode], -) -> Result<(), String> { - app_log!( - "ui_separation.binding_validate.start expected_nodes={} decisions={}", - batch.len(), - response.decisions.len() - ); - let expected = batch - .iter() - .map(|node| node.id.clone()) - .collect::>(); - let mut seen = std::collections::HashSet::new(); +fn overlaps(a: &SeparationNode, b: &SeparationNode) -> bool { + let ax1 = a.global_pos_x_px as u64 + a.width_px as u64; + let ay1 = a.global_pos_y_px as u64 + a.height_px as u64; + let bx1 = b.global_pos_x_px as u64 + b.width_px as u64; + let by1 = b.global_pos_y_px as u64 + b.height_px as u64; + let width = ax1.min(bx1).saturating_sub(a.global_pos_x_px.max(b.global_pos_x_px) as u64); + let height = ay1.min(by1).saturating_sub(a.global_pos_y_px.max(b.global_pos_y_px) as u64); + width > 0 && height > 0 +} + +pub fn next_leaf_batch<'a>(state: &SeparationState, tree: &'a SeparationTree) -> Vec<&'a SeparationNode> { + let terminal = terminal_ids(state); + let mut candidates = Vec::new(); + logical_leaves(&tree.root, tree.root_extractable, &terminal, &mut candidates); + let mut selected: Vec<&'a SeparationNode> = Vec::new(); + for candidate in candidates { + if selected.iter().all(|other| !overlaps(candidate, other)) { selected.push(candidate); } + } + app_log!("ui_separation.batch_selected image_id={} leaf_nodes={}", tree.src_ui_design.as_str(), selected.len()); + selected +} + +pub fn validate_binding_response(response: &BindingResp, batch: &[&SeparationNode]) -> Result<(), String> { + let expected = batch.iter().map(|node| node.id.clone()).collect::>(); + let mut seen = HashSet::new(); for decision in &response.decisions { - let node_id = match decision { - BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => { - to_node - } - }; - if !expected.contains(node_id) { - app_log!( - "ui_separation.error stage=binding_validate reason=unknown_node node_id={}", - node_id.as_str() - ); - return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); - } - if !seen.insert(node_id.clone()) { - app_log!( - "ui_separation.error stage=binding_validate reason=duplicate_node node_id={}", - node_id.as_str() - ); - return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); - } - if let BindingDecision::NeedRework { - problem_description, - .. - } = decision - { - if problem_description.trim().is_empty() { - app_log!( - "ui_separation.error stage=binding_validate reason=empty_problem_description node_id={}", - node_id.as_str() - ); - return Err("NeedRework 必须包含问题描述".to_string()); - } + let node_id = match decision { BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => to_node }; + if !expected.contains(node_id) { return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); } + if !seen.insert(node_id.clone()) { return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); } + if let BindingDecision::NeedRework { problem_description, .. } = decision { + if problem_description.trim().is_empty() { return Err("NeedRework 必须包含问题描述".to_string()); } } } - if seen.len() != expected.len() { - app_log!( - "ui_separation.error stage=binding_validate reason=incomplete_coverage expected={} seen={}", - expected.len(), - seen.len() - ); - return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); - } - app_log!( - "ui_separation.binding_validate.completed covered_nodes={}", - seen.len() - ); + if seen.len() != expected.len() { return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); } Ok(()) } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index cf03003a7..07221b0e4 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -29,22 +29,28 @@ pub fn apply_batch_patch( decisions.len(), cut_paths.len() ); - let tree = state - .unprocessed_trees - .get_mut(tree_index) - .ok_or_else(|| "separation tree 索引无效".to_string())?; - let batch = next_leaf_batch(tree); + let batch_nodes = { + let tree = state + .trees + .get(tree_index) + .ok_or_else(|| "separation tree 索引无效".to_string())?; + next_leaf_batch(state, tree) + .into_iter() + .cloned() + .collect::>() + }; + let batch = batch_nodes.iter().collect::>(); validate_binding_response( &BindingResp { decisions: decisions.to_vec(), }, &batch, )?; - let rework_counts = batch + let rework_counts = batch_nodes .iter() .map(|node| (node.id.clone(), node.rework_count)) .collect::>(); - let mut ids = std::collections::HashSet::new(); + let tree = state.trees.get_mut(tree_index).expect("tree index checked"); for decision in decisions { match decision { BindingDecision::Ok { to_node, .. } => { @@ -55,7 +61,6 @@ pub fn apply_batch_patch( node_id: to_node.clone(), cut_image_path: path.clone(), }); - ids.insert(to_node.clone()); } BindingDecision::NeedRework { to_node, @@ -68,18 +73,15 @@ pub fn apply_batch_patch( problem_description: problem_description.clone(), rework_count: count, }); - ids.insert(to_node.clone()); } else { increment_rework_count(&mut tree.root, to_node, count); } } } } - remove_ids(&mut tree.root, &ids); app_log!( - "ui_separation.batch_patch.completed tree_index={} removed_nodes={} bound={} problematic={} pending_root_children={}", + "ui_separation.batch_patch.completed tree_index={} bound={} problematic={} pending_root_children={}", tree_index, - ids.len(), state.bound.len(), state.problematic_nodes.len(), tree.root.children.len() @@ -87,34 +89,6 @@ pub fn apply_batch_patch( Ok(()) } -fn mark_batch_problematic( - state: &mut SeparationState, - tree_index: usize, - batch: &[&SeparationNode], - error: String, -) { - app_log!( - "ui_separation.batch_problematic tree_index={} nodes={} error={}", - tree_index, - batch.len(), - error - ); - let ids = batch - .iter() - .map(|node| node.id.clone()) - .collect::>(); - for node in batch { - state.problematic_nodes.push(ProblematicNode { - node_id: node.id.clone(), - problem_description: error.clone(), - rework_count: node.rework_count, - }); - } - if let Some(tree) = state.unprocessed_trees.get_mut(tree_index) { - remove_ids(&mut tree.root, &ids); - } -} - fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { if node.id == *id { node.rework_count = count; @@ -222,108 +196,6 @@ async fn raw_image_edit( result } -async fn build_marked_image( - source_url: String, - nodes: Vec, - target: PathBuf, -) -> Result { - app_log!( - "ui_separation.mark_image.start nodes={} target_file={}", - nodes.len(), - target - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or("") - ); - tokio::task::spawn_blocking(move || { - let areas = nodes - .iter() - .map(|node| { - ( - node.global_pos_x_px, - node.global_pos_y_px, - node.width_px, - node.height_px, - ) - }) - .collect::>(); - build_marked_image_blocking(&source_url, &areas, &target) - }) - .await - .map_err(|error| format!("构建标记图任务失败：{error}"))? -} - -fn build_marked_image_blocking( - source_url: &str, - nodes: &[(u32, u32, u32, u32)], - target: &Path, -) -> Result { - let encoded = source_url - .split_once(',') - .map(|(_, d)| d) - .ok_or_else(|| "源图 data URL 无效".to_string())?; - let bytes = base64::engine::general_purpose::STANDARD - .decode(encoded) - .map_err(|e| format!("解码源图失败：{e}"))?; - let mut image = image::load_from_memory(&bytes) - .map_err(|e| format!("读取源图失败：{e}"))? - .to_rgba8(); - let width = image.width(); - let height = image.height(); - app_log!( - "ui_separation.mark_image.decoded nodes={} width={} height={}", - nodes.len(), - width, - height - ); - for &(global_pos_x_px, global_pos_y_px, node_width_px, node_height_px) in nodes { - let x0 = global_pos_x_px.min(width.saturating_sub(1)); - let y0 = global_pos_y_px.min(height.saturating_sub(1)); - let x1 = global_pos_x_px - .saturating_add(node_width_px) - .min(width) - .saturating_sub(1); - let y1 = global_pos_y_px - .saturating_add(node_height_px) - .min(height) - .saturating_sub(1); - if x0 >= x1 || y0 >= y1 { - continue; - } - for x in x0..=x1 { - image.put_pixel(x, y0, image::Rgba([0, 255, 0, 255])); - image.put_pixel(x, y1, image::Rgba([0, 255, 0, 255])); - } - for y in y0..=y1 { - image.put_pixel(x0, y, image::Rgba([0, 255, 0, 255])); - image.put_pixel(x1, y, image::Rgba([0, 255, 0, 255])); - } - for y in y0..=y1 { - for x in x0..=x1 { - if x > x0 && x < x1 && y > y0 && y < y1 { - image.put_pixel(x, y, image::Rgba([180, 0, 180, 120])); - } - } - } - } - image::DynamicImage::ImageRgba8(image.clone()) - .save_with_format(target, image::ImageFormat::Png) - .map_err(|e| format!("写入标记图失败：{e}"))?; - let mut png = Vec::new(); - image::DynamicImage::ImageRgba8(image) - .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) - .map_err(|e| format!("编码标记图失败：{e}"))?; - let result = format!( - "data:image/png;base64,{}", - base64::engine::general_purpose::STANDARD.encode(png) - ); - app_log!( - "ui_separation.mark_image.completed data_url_chars={}", - result.len() - ); - Ok(result) -} - async fn write_processed_image(processed_url: String, target: PathBuf) -> Result<(), String> { app_log!( "ui_separation.processed_image.write.start target_file={} data_url_chars={}", @@ -503,12 +375,12 @@ pub(crate) async fn separate_ui_impl( "ui_separation.state_ready asset_id={} restored={} trees={} bound={} problematic={}", asset_id, restored, - separation.unprocessed_trees.len(), + separation.trees.len(), separation.bound.len(), separation.problematic_nodes.len() ); - for tree_index in 0..separation.unprocessed_trees.len() { - let tree = &separation.unprocessed_trees[tree_index]; + for tree_index in 0..separation.trees.len() { + let tree = &separation.trees[tree_index]; let image_id = tree.src_ui_design.clone(); let image = state .ui_design_images @@ -541,10 +413,10 @@ pub(crate) async fn separate_ui_impl( })?; let mut batch_index = 0usize; loop { - let Some(current_tree) = separation.unprocessed_trees.get(tree_index) else { + let Some(current_tree) = separation.trees.get(tree_index) else { break; }; - let batch_nodes = next_leaf_batch(current_tree) + let batch_nodes = next_leaf_batch(&separation, current_tree) .into_iter() .cloned() .collect::>(); @@ -584,10 +456,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } }; let processed_url = match raw_image_edit( @@ -606,10 +476,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); @@ -621,10 +489,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { Ok(value) => value, @@ -634,10 +500,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } }; app_log!( @@ -686,10 +550,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; write_separation_state(&state_path, &separation)?; @@ -778,10 +640,3 @@ fn cut_processed_image_blocking( ); Ok(()) } - -fn remove_ids(node: &mut SeparationNode, ids: &std::collections::HashSet) { - node.children.retain(|child| !ids.contains(&child.id)); - for child in &mut node.children { - remove_ids(child, ids); - } -} From d30bc462c59f733ef7fe5180bcf62e6ae1d3f536 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 14:50:19 +0800 Subject: [PATCH 055/248] =?UTF-8?q?=E6=A0=BC=E5=BC=8F=E5=8C=96=E8=87=AA?= =?UTF-8?q?=E5=8A=A8=E5=88=86=E7=A6=BB=20Rust=20=E6=A8=A1=E5=9D=97=20?= =?UTF-8?q?=E7=BB=9F=E4=B8=80=E6=A0=91=E9=80=89=E6=8B=A9=E3=80=81=E6=A0=87?= =?UTF-8?q?=E8=AE=B0=E5=9B=BE=E5=92=8C=E6=89=B9=E6=AC=A1=E6=B5=81=E7=A8=8B?= =?UTF-8?q?=E4=BB=A3=E7=A0=81=E9=A3=8E=E6=A0=BC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../ui_editor/commands/separation/marker.rs | 120 +++++++++-- .../src/ui_editor/commands/separation/mod.rs | 19 +- .../src/ui_editor/commands/separation/tree.rs | 189 +++++++++++++----- .../ui_editor/commands/separation/workflow.rs | 42 +++- 4 files changed, 285 insertions(+), 85 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index f39015389..ffd972792 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -4,49 +4,129 @@ use image::GenericImage; use std::path::Path; use std::path::PathBuf; -pub async fn build_marked_image(source_url: String, nodes: Vec, target: PathBuf) -> Result { +pub async fn build_marked_image( + source_url: String, + nodes: Vec, + target: PathBuf, +) -> Result { tokio::task::spawn_blocking(move || build_marked_image_blocking(&source_url, &nodes, &target)) .await .map_err(|error| format!("构建标记图任务失败：{error}"))? } -fn build_marked_image_blocking(source_url: &str, nodes: &[SeparationNode], target: &Path) -> Result { - let encoded = source_url.split_once(',').map(|(_, d)| d).ok_or_else(|| "源图 data URL 无效".to_string())?; - let bytes = base64::engine::general_purpose::STANDARD.decode(encoded).map_err(|e| format!("解码源图失败：{e}"))?; - let mut image = image::load_from_memory(&bytes).map_err(|e| format!("读取源图失败：{e}"))?.to_rgba8(); +fn build_marked_image_blocking( + source_url: &str, + nodes: &[SeparationNode], + target: &Path, +) -> Result { + let encoded = source_url + .split_once(',') + .map(|(_, d)| d) + .ok_or_else(|| "源图 data URL 无效".to_string())?; + let bytes = base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|e| format!("解码源图失败：{e}"))?; + let mut image = image::load_from_memory(&bytes) + .map_err(|e| format!("读取源图失败：{e}"))? + .to_rgba8(); let width = image.width(); let height = image.height(); // Purple reconstruction comes first so the parent green frame remains visible on top. for node in nodes { for child in &node.children { - fill_rect(&mut image, child.global_pos_x_px, child.global_pos_y_px, child.width_px, child.height_px, image::Rgba([180, 0, 180, 120]), width, height); + fill_rect( + &mut image, + child.global_pos_x_px, + child.global_pos_y_px, + child.width_px, + child.height_px, + image::Rgba([180, 0, 180, 120]), + width, + height, + ); } } for node in nodes { - draw_frame(&mut image, node.global_pos_x_px, node.global_pos_y_px, node.width_px, node.height_px, width, height); + draw_frame( + &mut image, + node.global_pos_x_px, + node.global_pos_y_px, + node.width_px, + node.height_px, + width, + height, + ); } - image::DynamicImage::ImageRgba8(image.clone()).save_with_format(target, image::ImageFormat::Png).map_err(|e| format!("写入标记图失败：{e}"))?; + image::DynamicImage::ImageRgba8(image.clone()) + .save_with_format(target, image::ImageFormat::Png) + .map_err(|e| format!("写入标记图失败：{e}"))?; let mut png = Vec::new(); - image::DynamicImage::ImageRgba8(image).write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png).map_err(|e| format!("编码标记图失败：{e}"))?; - Ok(format!("data:image/png;base64,{}", base64::engine::general_purpose::STANDARD.encode(png))) + image::DynamicImage::ImageRgba8(image) + .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) + .map_err(|e| format!("编码标记图失败：{e}"))?; + Ok(format!( + "data:image/png;base64,{}", + base64::engine::general_purpose::STANDARD.encode(png) + )) } -fn clipped_rect(x: u32, y: u32, w: u32, h: u32, width: u32, height: u32) -> Option<(u32, u32, u32, u32)> { - if width == 0 || height == 0 || w == 0 || h == 0 { return None; } - let x0 = x.min(width - 1); let y0 = y.min(height - 1); +fn clipped_rect( + x: u32, + y: u32, + w: u32, + h: u32, + width: u32, + height: u32, +) -> Option<(u32, u32, u32, u32)> { + if width == 0 || height == 0 || w == 0 || h == 0 { + return None; + } + let x0 = x.min(width - 1); + let y0 = y.min(height - 1); let x1 = x.saturating_add(w).min(width).saturating_sub(1); let y1 = y.saturating_add(h).min(height).saturating_sub(1); (x0 <= x1 && y0 <= y1).then_some((x0, y0, x1, y1)) } -fn fill_rect(image: &mut image::RgbaImage, x: u32, y: u32, w: u32, h: u32, color: image::Rgba, width: u32, height: u32) { - let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { return; }; - for yy in y0..=y1 { for xx in x0..=x1 { image.put_pixel(xx, yy, color); } } +fn fill_rect( + image: &mut image::RgbaImage, + x: u32, + y: u32, + w: u32, + h: u32, + color: image::Rgba, + width: u32, + height: u32, +) { + let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { + return; + }; + for yy in y0..=y1 { + for xx in x0..=x1 { + image.put_pixel(xx, yy, color); + } + } } -fn draw_frame(image: &mut image::RgbaImage, x: u32, y: u32, w: u32, h: u32, width: u32, height: u32) { - let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { return; }; +fn draw_frame( + image: &mut image::RgbaImage, + x: u32, + y: u32, + w: u32, + h: u32, + width: u32, + height: u32, +) { + let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { + return; + }; let green = image::Rgba([0, 255, 0, 255]); - for xx in x0..=x1 { image.put_pixel(xx, y0, green); image.put_pixel(xx, y1, green); } - for yy in y0..=y1 { image.put_pixel(x0, yy, green); image.put_pixel(x1, yy, green); } + for xx in x0..=x1 { + image.put_pixel(xx, y0, green); + image.put_pixel(xx, y1, green); + } + for yy in y0..=y1 { + image.put_pixel(x0, yy, green); + image.put_pixel(x1, yy, green); + } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 41f24538a..83a984c2e 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -1,31 +1,31 @@ -mod model; mod marker; +mod model; mod persistence; mod prompt; mod tree; mod workflow; +pub(crate) use marker::build_marked_image; pub use model::*; pub use persistence::*; pub use tree::*; pub use workflow::apply_batch_patch; pub(crate) use workflow::separate_ui_impl; -pub(crate) use marker::build_marked_image; #[cfg(test)] mod tests { use super::*; - use crate::ui_editor::component::Component; - use crate::ui_editor::layout::node::Node; - use crate::ui_editor::state::{State, UITree}; - use crate::ui_editor::utils::{NodeId, UIDesignImageId}; - use std::path::Path; use crate::ui_editor::component::image::{ImageComponent, ImageType}; + use crate::ui_editor::component::Component; use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; use crate::ui_editor::layout::control_layout::ControlLayout; + use crate::ui_editor::layout::node::Node; use crate::ui_editor::layout::node::{NodeMetadata, NodeSource, StageStatus}; use crate::ui_editor::resource::ui_design_image::UIDesignImage; + use crate::ui_editor::state::{State, UITree}; + use crate::ui_editor::utils::{NodeId, UIDesignImageId}; use nalgebra::Vector2; use std::collections::HashMap; + use std::path::Path; use typed_floats::tf32::StrictlyPositiveFinite; fn node(id: &str, components: Vec, children: Vec) -> Node { @@ -89,10 +89,7 @@ mod tests { )], ); let result = construct_separation_state(&state(root)); - assert_eq!( - result.trees[0].root.children[0].id.as_str(), - "image" - ); + assert_eq!(result.trees[0].root.children[0].id.as_str(), "image"); } #[test] diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 5c84af6db..dcca538e7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -7,11 +7,21 @@ use std::collections::HashSet; fn is_unbound_image(node: &Node) -> bool { node.components.iter().any(|component| { - matches!(component, Component::Image(ImageComponent { target_graphic: None, .. })) + matches!( + component, + Component::Image(ImageComponent { + target_graphic: None, + .. + }) + ) }) } -fn node_pixel_rect(node: &Node, parent: &crate::ui_editor::layout::dimension::UIRect, ppu: f32) -> (u32, u32, u32, u32) { +fn node_pixel_rect( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, +) -> (u32, u32, u32, u32) { let rect = node.layout.transform.resolve(parent); ( (rect.min.x * ppu).max(0.0).round() as u32, @@ -32,7 +42,12 @@ fn node_description(node: &Node) -> String { } } -fn collect_todo_nodes(node: &Node, parent: &crate::ui_editor::layout::dimension::UIRect, ppu: f32, output: &mut Vec) { +fn collect_todo_nodes( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, + output: &mut Vec, +) { let rect = node.layout.transform.resolve(parent); let mut children = Vec::new(); for child in &node.children { @@ -41,10 +56,17 @@ fn collect_todo_nodes(node: &Node, parent: &crate::ui_editor::layout::dimension: if is_unbound_image(node) { let (x, y, w, h) = node_pixel_rect(node, parent, ppu); output.push(SeparationNode { - id: node.id.clone(), global_pos_x_px: x, global_pos_y_px: y, - width_px: w, height_px: h, - note: SeparationNote { description: node_description(node), text_note: String::new() }, - children, rework_count: 0, + id: node.id.clone(), + global_pos_x_px: x, + global_pos_y_px: y, + width_px: w, + height_px: h, + note: SeparationNote { + description: node_description(node), + text_note: String::new(), + }, + children, + rework_count: 0, }); } else { output.extend(children); @@ -52,40 +74,71 @@ fn collect_todo_nodes(node: &Node, parent: &crate::ui_editor::layout::dimension: } pub fn construct_separation_state(state: &State) -> SeparationState { - let trees = state.ui_trees.iter().filter_map(|tree| { - let image = state.ui_design_images.get(&tree.src_ui_design)?; - let ppu = image.pixels_per_unit.get(); - let size = image.pixel_size / ppu; - let root_rect = crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); - let mut children = Vec::new(); - for child in &tree.root.children { - collect_todo_nodes(child, &root_rect, ppu, &mut children); - } - let root_extractable = is_unbound_image(&tree.root); - if !root_extractable && children.is_empty() { return None; } - let (x, y, w, h) = node_pixel_rect(&tree.root, &root_rect, ppu); - Some(SeparationTree { - src_ui_design: tree.src_ui_design.clone(), - root: SeparationNode { - id: tree.root.id.clone(), global_pos_x_px: x, global_pos_y_px: y, - width_px: w, height_px: h, - note: SeparationNote { description: node_description(&tree.root), text_note: String::new() }, - children, rework_count: 0, - }, - root_extractable, + let trees = state + .ui_trees + .iter() + .filter_map(|tree| { + let image = state.ui_design_images.get(&tree.src_ui_design)?; + let ppu = image.pixels_per_unit.get(); + let size = image.pixel_size / ppu; + let root_rect = + crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); + let mut children = Vec::new(); + for child in &tree.root.children { + collect_todo_nodes(child, &root_rect, ppu, &mut children); + } + let root_extractable = is_unbound_image(&tree.root); + if !root_extractable && children.is_empty() { + return None; + } + let (x, y, w, h) = node_pixel_rect(&tree.root, &root_rect, ppu); + Some(SeparationTree { + src_ui_design: tree.src_ui_design.clone(), + root: SeparationNode { + id: tree.root.id.clone(), + global_pos_x_px: x, + global_pos_y_px: y, + width_px: w, + height_px: h, + note: SeparationNote { + description: node_description(&tree.root), + text_note: String::new(), + }, + children, + rework_count: 0, + }, + root_extractable, + }) }) - }).collect(); - SeparationState { schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), trees, bound: Vec::new(), problematic_nodes: Vec::new() } + .collect(); + SeparationState { + schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), + trees, + bound: Vec::new(), + problematic_nodes: Vec::new(), + } } fn terminal_ids(state: &SeparationState) -> HashSet { - state.bound.iter().map(|n| n.node_id.clone()) - .chain(state.problematic_nodes.iter().map(|n| n.node_id.clone())).collect() + state + .bound + .iter() + .map(|n| n.node_id.clone()) + .chain(state.problematic_nodes.iter().map(|n| n.node_id.clone())) + .collect() } -fn logical_leaves<'a>(node: &'a SeparationNode, extractable: bool, terminal: &HashSet, output: &mut Vec<&'a SeparationNode>) { +fn logical_leaves<'a>( + node: &'a SeparationNode, + extractable: bool, + terminal: &HashSet, + output: &mut Vec<&'a SeparationNode>, +) { let is_terminal = terminal.contains(&node.id); - let children_terminal = node.children.iter().all(|child| terminal.contains(&child.id)); + let children_terminal = node + .children + .iter() + .all(|child| terminal.contains(&child.id)); if extractable && !is_terminal && children_terminal { output.push(node); return; @@ -100,34 +153,74 @@ fn overlaps(a: &SeparationNode, b: &SeparationNode) -> bool { let ay1 = a.global_pos_y_px as u64 + a.height_px as u64; let bx1 = b.global_pos_x_px as u64 + b.width_px as u64; let by1 = b.global_pos_y_px as u64 + b.height_px as u64; - let width = ax1.min(bx1).saturating_sub(a.global_pos_x_px.max(b.global_pos_x_px) as u64); - let height = ay1.min(by1).saturating_sub(a.global_pos_y_px.max(b.global_pos_y_px) as u64); + let width = ax1 + .min(bx1) + .saturating_sub(a.global_pos_x_px.max(b.global_pos_x_px) as u64); + let height = ay1 + .min(by1) + .saturating_sub(a.global_pos_y_px.max(b.global_pos_y_px) as u64); width > 0 && height > 0 } -pub fn next_leaf_batch<'a>(state: &SeparationState, tree: &'a SeparationTree) -> Vec<&'a SeparationNode> { +pub fn next_leaf_batch<'a>( + state: &SeparationState, + tree: &'a SeparationTree, +) -> Vec<&'a SeparationNode> { let terminal = terminal_ids(state); let mut candidates = Vec::new(); - logical_leaves(&tree.root, tree.root_extractable, &terminal, &mut candidates); + logical_leaves( + &tree.root, + tree.root_extractable, + &terminal, + &mut candidates, + ); let mut selected: Vec<&'a SeparationNode> = Vec::new(); for candidate in candidates { - if selected.iter().all(|other| !overlaps(candidate, other)) { selected.push(candidate); } + if selected.iter().all(|other| !overlaps(candidate, other)) { + selected.push(candidate); + } } - app_log!("ui_separation.batch_selected image_id={} leaf_nodes={}", tree.src_ui_design.as_str(), selected.len()); + app_log!( + "ui_separation.batch_selected image_id={} leaf_nodes={}", + tree.src_ui_design.as_str(), + selected.len() + ); selected } -pub fn validate_binding_response(response: &BindingResp, batch: &[&SeparationNode]) -> Result<(), String> { - let expected = batch.iter().map(|node| node.id.clone()).collect::>(); +pub fn validate_binding_response( + response: &BindingResp, + batch: &[&SeparationNode], +) -> Result<(), String> { + let expected = batch + .iter() + .map(|node| node.id.clone()) + .collect::>(); let mut seen = HashSet::new(); for decision in &response.decisions { - let node_id = match decision { BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => to_node }; - if !expected.contains(node_id) { return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); } - if !seen.insert(node_id.clone()) { return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); } - if let BindingDecision::NeedRework { problem_description, .. } = decision { - if problem_description.trim().is_empty() { return Err("NeedRework 必须包含问题描述".to_string()); } + let node_id = match decision { + BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => { + to_node + } + }; + if !expected.contains(node_id) { + return Err(format!("视觉绑定返回了未知节点：{}", node_id.as_str())); + } + if !seen.insert(node_id.clone()) { + return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); + } + if let BindingDecision::NeedRework { + problem_description, + .. + } = decision + { + if problem_description.trim().is_empty() { + return Err("NeedRework 必须包含问题描述".to_string()); + } } } - if seen.len() != expected.len() { return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); } + if seen.len() != expected.len() { + return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string()); + } Ok(()) } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 07221b0e4..521754270 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -89,6 +89,27 @@ pub fn apply_batch_patch( Ok(()) } +fn mark_batch_problematic( + state: &mut SeparationState, + tree_index: usize, + batch: &[&SeparationNode], + error: String, +) { + app_log!( + "ui_separation.batch_problematic tree_index={} nodes={} error={}", + tree_index, + batch.len(), + error + ); + for node in batch { + state.problematic_nodes.push(ProblematicNode { + node_id: node.id.clone(), + problem_description: error.clone(), + rework_count: node.rework_count, + }); + } +} + fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { if node.id == *id { node.rework_count = count; @@ -279,7 +300,6 @@ async fn visual_binding( let llm_config = llm_config.clone(); async move { let request = LlmRunRequest::new(vec![ - LlmMessage::system("你是 UI 图片视觉绑定器。只根据图像判断区域，不做 OCR。"), LlmMessage::user_multimodal(vec![ LlmMessageContentPart::InputText { text: prompt }, LlmMessageContentPart::InputImage { @@ -456,8 +476,10 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); + mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - return Err(error); + batch_index += 1; + continue; } }; let processed_url = match raw_image_edit( @@ -476,8 +498,10 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); + mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - return Err(error); + batch_index += 1; + continue; } }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); @@ -489,8 +513,10 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); + mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - return Err(error); + batch_index += 1; + continue; } let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { Ok(value) => value, @@ -500,8 +526,10 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); + mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - return Err(error); + batch_index += 1; + continue; } }; app_log!( @@ -550,8 +578,10 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); + mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - return Err(error); + batch_index += 1; + continue; } apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; write_separation_state(&state_path, &separation)?; From 9b4924c73ebb12e4ede4fd50be379c9e3baec65e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 14:54:37 +0800 Subject: [PATCH 056/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=89=B9=E6=AC=A1=E9=87=8D=E5=8F=A0=E7=AD=9B=E9=80=89=E6=B5=8B?= =?UTF-8?q?=E8=AF=95=20=E9=AA=8C=E8=AF=81=20DFS=20=E9=A1=BA=E5=BA=8F?= =?UTF-8?q?=E4=B8=8B=E8=B4=AA=E5=BF=83=E8=B7=B3=E8=BF=87=E9=87=8D=E5=8F=A0?= =?UTF-8?q?=E5=8F=B6=E8=8A=82=E7=82=B9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/separation/mod.rs | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 83a984c2e..179dfdd6f 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -157,4 +157,66 @@ mod tests { assert_eq!(state.bound[0].node_id, id); assert_eq!(state.trees[0].root.children.len(), 1); } + + #[test] + fn batch_selection_greedily_skips_overlapping_leaves() { + let a = SeparationNode { + id: NodeId::new("a").unwrap(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 10, + height_px: 10, + note: SeparationNote::default(), + children: vec![], + rework_count: 0, + }; + let b = SeparationNode { + id: NodeId::new("b").unwrap(), + global_pos_x_px: 5, + global_pos_y_px: 5, + width_px: 10, + height_px: 10, + note: SeparationNote::default(), + children: vec![], + rework_count: 0, + }; + let c = SeparationNode { + id: NodeId::new("c").unwrap(), + global_pos_x_px: 20, + global_pos_y_px: 0, + width_px: 5, + height_px: 5, + note: SeparationNote::default(), + children: vec![], + rework_count: 0, + }; + let tree = SeparationTree { + src_ui_design: UIDesignImageId::new("page").unwrap(), + root: SeparationNode { + id: NodeId::new("root").unwrap(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 100, + height_px: 100, + note: SeparationNote::default(), + children: vec![a, b, c], + rework_count: 0, + }, + root_extractable: false, + }; + let state = SeparationState { + schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), + trees: vec![tree.clone()], + bound: vec![], + problematic_nodes: vec![], + }; + let batch = next_leaf_batch(&state, &tree); + assert_eq!( + batch + .iter() + .map(|node| node.id.as_str()) + .collect::>(), + vec!["a", "c"] + ); + } } From df3a172bc95a471c918d30520e48bd2670840684 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 14:57:55 +0800 Subject: [PATCH 057/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20GPT=20Image?= =?UTF-8?q?=202=20multipart=20=E5=90=88=E5=90=8C=20=E6=98=8E=E7=A1=AE?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E5=9B=BE=E7=89=87=E7=BC=96=E8=BE=91=E8=AF=B7?= =?UTF-8?q?=E6=B1=82=E5=AD=97=E6=AE=B5=E4=B8=8E=E6=97=A7=E5=8D=8F=E8=AE=AE?= =?UTF-8?q?=E8=BE=B9=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index e4533dd58..67b8a66a6 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -68,6 +68,7 @@ - 已保存的 separation state 是跨重启继续工作的最小单位；重启后从上一个已保存 batch 的状态继续。 - 当前执行中的 batch 是否持久化、以及如何避免 image-edit 成功后在 patch 前崩溃导致重复调用，列为 TODO。 - Raw endpoint 每次 HTTP 调用都是一次新操作；客户端不保存或复用 raw operation ID，不实现第二套本地幂等账本。 +- 图片编辑调用当前 Raw GPT Image 2 multipart 合同：`image`（PNG 文件）、`prompt`、`width`、`height`、`output_format=png`、`background=transparent`；不再发送旧 JSON/base64 请求体。 - 后端 raw operation 的持久状态与扣费后崩溃恢复窗口，遵循 Raw GPT Image 2 方案中的独立 TODO。 ## TODO From f349350d6031fbefaef14fcdb0c9df2e4a23dfb8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 15:03:48 +0800 Subject: [PATCH 058/248] =?UTF-8?q?=E5=90=8C=E6=AD=A5=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E7=8A=B6=E6=80=81=20TypeScript=20=E7=B1=BB=E5=9E=8B=20?= =?UTF-8?q?=E5=8A=A0=E5=85=A5=20root=5Fextractable=20=E5=B9=B6=E5=B0=86?= =?UTF-8?q?=E6=A0=91=E5=AD=97=E6=AE=B5=E7=BB=9F=E4=B8=80=E4=B8=BA=20trees?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/marker.rs | 1 - .../src/features/ui-editor/types/SeparationState.ts | 6 ++++++ .../src/features/ui-editor/types/SeparationTree.ts | 5 +++++ 3 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationState.ts create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationTree.ts diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index ffd972792..2a5cf93e5 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -1,6 +1,5 @@ use super::model::SeparationNode; use base64::Engine as _; -use image::GenericImage; use std::path::Path; use std::path::PathBuf; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationState.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationState.ts new file mode 100644 index 000000000..fa180b51e --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationState.ts @@ -0,0 +1,6 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { BoundNode } from "./BoundNode"; +import type { ProblematicNode } from "./ProblematicNode"; +import type { SeparationTree } from "./SeparationTree"; + +export type SeparationState = { schema_version: string, trees: Array, bound: Array, problematic_nodes: Array, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationTree.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationTree.ts new file mode 100644 index 000000000..2196bac3b --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationTree.ts @@ -0,0 +1,5 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { SeparationNode } from "./SeparationNode"; +import type { UIDesignImageId } from "./UIDesignImageId"; + +export type SeparationTree = { src_ui_design: UIDesignImageId, root: SeparationNode, root_extractable: boolean, }; From feb6bc0ad4c0d909d9460a5f6a829f1b46346d11 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 15:07:49 +0800 Subject: [PATCH 059/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E6=81=A2=E5=A4=8D=E8=BE=B9=E7=95=8C=20?= =?UTF-8?q?=E5=9B=BE=E7=89=87=E7=BC=96=E8=BE=91=E3=80=81=E5=86=99=E5=85=A5?= =?UTF-8?q?=E5=92=8C=E8=A3=81=E5=88=87=E5=A4=B1=E8=B4=A5=E4=BF=9D=E7=95=99?= =?UTF-8?q?=E7=8A=B6=E6=80=81=E5=B9=B6=E8=BF=94=E5=9B=9E=E9=94=99=E8=AF=AF?= =?UTF-8?q?=20=E4=BB=85=E8=A7=86=E8=A7=89=E6=A8=A1=E5=9E=8B=E8=BF=94?= =?UTF-8?q?=E5=B7=A5=E7=BB=93=E6=9E=9C=E8=BF=9B=E5=85=A5=20problematic?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../ui_editor/commands/separation/workflow.rs | 61 +++++-------------- 1 file changed, 14 insertions(+), 47 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 521754270..e6ff0cbfc 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -89,27 +89,6 @@ pub fn apply_batch_patch( Ok(()) } -fn mark_batch_problematic( - state: &mut SeparationState, - tree_index: usize, - batch: &[&SeparationNode], - error: String, -) { - app_log!( - "ui_separation.batch_problematic tree_index={} nodes={} error={}", - tree_index, - batch.len(), - error - ); - for node in batch { - state.problematic_nodes.push(ProblematicNode { - node_id: node.id.clone(), - problem_description: error.clone(), - rework_count: node.rework_count, - }); - } -} - fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { if node.id == *id { node.rework_count = count; @@ -299,17 +278,15 @@ async fn visual_binding( let client = client.clone(); let llm_config = llm_config.clone(); async move { - let request = LlmRunRequest::new(vec![ - LlmMessage::user_multimodal(vec![ - LlmMessageContentPart::InputText { text: prompt }, - LlmMessageContentPart::InputImage { - image_url: source_url, - }, - LlmMessageContentPart::InputImage { - image_url: processed_url, - }, - ]), - ]) + let request = LlmRunRequest::new(vec![LlmMessage::user_multimodal(vec![ + LlmMessageContentPart::InputText { text: prompt }, + LlmMessageContentPart::InputImage { + image_url: source_url, + }, + LlmMessageContentPart::InputImage { + image_url: processed_url, + }, + ])]) .with_function_tools(vec![tool.clone()]) .with_tool_choice(LlmToolChoice::Required); request_ui_editor_llm(&client, &llm_config, request) @@ -476,10 +453,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } }; let processed_url = match raw_image_edit( @@ -498,10 +473,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); @@ -513,10 +486,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { Ok(value) => value, @@ -526,10 +497,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } }; app_log!( @@ -578,10 +547,8 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index ); - mark_batch_problematic(&mut separation, tree_index, &batch, error); write_separation_state(&state_path, &separation)?; - batch_index += 1; - continue; + return Err(error); } apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; write_separation_state(&state_path, &separation)?; From 7aa96e8438da223c7eb3ed8f59f4f1b1b8b22b99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 15:16:39 +0800 Subject: [PATCH 060/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=87=E8=AE=B0=E5=9B=BE=E4=B8=AD=E6=96=87=E6=B3=A8=E9=87=8A?= =?UTF-8?q?=20=E8=AF=B4=E6=98=8E=E7=B4=AB=E8=89=B2=E9=87=8D=E5=BB=BA?= =?UTF-8?q?=E4=B8=8E=E7=BB=BF=E8=89=B2=E6=A1=86=E7=BB=98=E5=88=B6=E5=B1=82?= =?UTF-8?q?=E7=BA=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/marker.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index 2a5cf93e5..292725416 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -30,7 +30,7 @@ fn build_marked_image_blocking( .to_rgba8(); let width = image.width(); let height = image.height(); - // Purple reconstruction comes first so the parent green frame remains visible on top. + // 先填充紫色重建区域，随后绘制绿色框，确保绿色框位于最上层。 for node in nodes { for child in &node.children { fill_rect( From aa2a78f7e08300a5bca5c71da7defc872527de13 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 15:48:32 +0800 Subject: [PATCH 061/248] =?UTF-8?q?=E9=87=8D=E5=91=BD=E5=90=8D=20separated?= =?UTF-8?q?=5Fimage=5Farea=20=E4=B8=BA=20extracted=5Farea=EF=BC=8C?= =?UTF-8?q?=E4=BC=98=E5=8C=96=E5=91=BD=E5=90=8D=E4=B8=80=E8=87=B4=E6=80=A7?= =?UTF-8?q?=E5=B9=B6=E6=98=8E=E7=A1=AE=E5=8C=BA=E5=9F=9F=E5=90=AB=E4=B9=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/mod.rs | 2 +- .../src-tauri/src/ui_editor/commands/separation/model.rs | 2 +- .../src-tauri/src/ui_editor/commands/separation/prompt.rs | 4 +++- .../src-tauri/src/ui_editor/commands/separation/workflow.rs | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 179dfdd6f..7f86ff6f6 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -145,7 +145,7 @@ mod tests { let id = NodeId::new("image").unwrap(); let decisions = vec![BindingDecision::Ok { to_node: id.clone(), - separated_image_area: BindingArea { + extracted_area: BindingArea { global_pos_x_px: 0, global_pos_y_px: 0, width_px: 1, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs index a9a026b6a..85773a411 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs @@ -110,7 +110,7 @@ impl BindingArea { #[schemars(deny_unknown_fields)] pub enum BindingDecision { Ok { - separated_image_area: BindingArea, + extracted_area: BindingArea, to_node: NodeId, }, NeedRework { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs index 3343ae1dc..7781b6bf0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs @@ -37,7 +37,9 @@ pub(super) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { ``` {SHARED_SEPARATION_REQ} ``` - You need to recognize and review the separation: + You need to recognize and review the separation using the given tool. + field notes: + * extracted_area MUST be the recognized area from the processed image, INSTEAD OF from the src image. these node need handle: "# diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index e6ff0cbfc..506f291af 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -512,7 +512,7 @@ pub(crate) async fn separate_ui_impl( for decision in &binding.decisions { if let BindingDecision::Ok { to_node, - separated_image_area, + extracted_area: separated_image_area, } = decision { let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); From 8a0dcf8be7f83441b3cd8f53dac18c518202851d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 16:21:16 +0800 Subject: [PATCH 062/248] =?UTF-8?q?=E5=A2=9E=E5=BC=BA=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=87=E8=AE=B0=E6=A1=86=E7=9A=84=E5=8F=AF=E8=A7=86=E8=BE=A8?= =?UTF-8?q?=E8=AF=86=E5=BA=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为分离标记框增加两条绿色角到角交叉线。 新增 MARKER_LINE_WIDTH 常量统一配置标记线宽，并确保绘制结果裁剪在矩形内。 补充标记框像素级回归测试并同步更新自动分离专题文档。 --- .../ui_editor/commands/separation/marker.rs | 148 +++++++++++++++++- .../ui_editor/commands/separation/prompt.rs | 4 +- ...术方案】UI编辑器自动分离工作流-2026-09-08.md | 2 +- 3 files changed, 145 insertions(+), 9 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index 292725416..9c8cfd120 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -3,6 +3,8 @@ use base64::Engine as _; use std::path::Path; use std::path::PathBuf; +const MARKER_LINE_WIDTH: u32 = 2; + pub async fn build_marked_image( source_url: String, nodes: Vec, @@ -120,12 +122,144 @@ fn draw_frame( return; }; let green = image::Rgba([0, 255, 0, 255]); - for xx in x0..=x1 { - image.put_pixel(xx, y0, green); - image.put_pixel(xx, y1, green); - } - for yy in y0..=y1 { - image.put_pixel(x0, yy, green); - image.put_pixel(x1, yy, green); + draw_line( + image, + (x0, y0), + (x1, y0), + green, + MARKER_LINE_WIDTH, + (x0, y0, x1, y1), + ); + draw_line( + image, + (x0, y1), + (x1, y1), + green, + MARKER_LINE_WIDTH, + (x0, y0, x1, y1), + ); + draw_line( + image, + (x0, y0), + (x0, y1), + green, + MARKER_LINE_WIDTH, + (x0, y0, x1, y1), + ); + draw_line( + image, + (x1, y0), + (x1, y1), + green, + MARKER_LINE_WIDTH, + (x0, y0, x1, y1), + ); + draw_line( + image, + (x0, y0), + (x1, y1), + green, + MARKER_LINE_WIDTH, + (x0, y0, x1, y1), + ); + draw_line( + image, + (x1, y0), + (x0, y1), + green, + MARKER_LINE_WIDTH, + (x0, y0, x1, y1), + ); +} + +fn draw_line( + image: &mut image::RgbaImage, + start: (u32, u32), + end: (u32, u32), + color: image::Rgba, + line_width: u32, + bounds: (u32, u32, u32, u32), +) { + let mut x = start.0 as i64; + let mut y = start.1 as i64; + let target_x = end.0 as i64; + let target_y = end.1 as i64; + let dx = (target_x - x).abs(); + let sx = if x < target_x { 1 } else { -1 }; + let dy = -(target_y - y).abs(); + let sy = if y < target_y { 1 } else { -1 }; + let mut error = dx + dy; + + loop { + draw_brush(image, x, y, color, line_width, bounds); + if x == target_x && y == target_y { + break; + } + let twice_error = error * 2; + if twice_error >= dy { + error += dy; + x += sx; + } + if twice_error <= dx { + error += dx; + y += sy; + } + } +} + +fn draw_brush( + image: &mut image::RgbaImage, + x: i64, + y: i64, + color: image::Rgba, + line_width: u32, + bounds: (u32, u32, u32, u32), +) { + let (x0, y0, x1, y1) = bounds; + let line_width = line_width.max(1) as i64; + let before = (line_width - 1) / 2; + let after = line_width / 2; + let min_x = (x - before).max(x0 as i64); + let max_x = (x + after).min(x1 as i64); + let min_y = (y - before).max(y0 as i64); + let max_y = (y + after).min(y1 as i64); + for yy in min_y..=max_y { + for xx in min_x..=max_x { + image.put_pixel(xx as u32, yy as u32, color); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn draw_frame_adds_green_cross_corner_lines() { + let mut image = image::RgbaImage::from_pixel(8, 6, image::Rgba([1, 2, 3, 255])); + draw_frame(&mut image, 1, 1, 5, 3, 8, 6); + let green = image::Rgba([0, 255, 0, 255]); + + for &(x, y) in &[(1, 1), (5, 1), (1, 3), (5, 3), (3, 2)] { + assert_eq!(*image.get_pixel(x, y), green, "pixel ({x}, {y})"); + } + assert_eq!(*image.get_pixel(3, 1), green); + assert_eq!(*image.get_pixel(3, 3), green); + assert_eq!(*image.get_pixel(2, 2), green); + assert_eq!(*image.get_pixel(4, 2), green); + assert_eq!(*image.get_pixel(0, 0), image::Rgba([1, 2, 3, 255])); + } + + #[test] + fn draw_frame_keeps_cross_inside_clipped_rect() { + let mut image = image::RgbaImage::from_pixel(4, 4, image::Rgba([1, 2, 3, 255])); + draw_frame(&mut image, 2, 2, 4, 4, 4, 4); + let green = image::Rgba([0, 255, 0, 255]); + for y in 2..4 { + for x in 2..4 { + assert_eq!(*image.get_pixel(x, y), green); + } + } + assert_eq!(*image.get_pixel(1, 1), image::Rgba([1, 2, 3, 255])); } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs index 7781b6bf0..d6df93e9f 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs @@ -5,8 +5,10 @@ const SHARED_SEPARATION_REQ: &str = r#" MUST hard edges; preserve no glow/blur beyond the exact visible shape. NEVER keep its parent's background with it. - UI elements that needs to extract has been marked with GREEN line frames (only for mark purpose, NEVER wrap a frame in your extraction). + UI elements that needs to extract has been marked with GREEN line frames box with crossline inside. (only for mark purpose, NEVER wrap a frame in your extraction). On some UI elements, there is some PURPLE filled area, they were removed UI elements, reconstruct the background under where they were. + + MUST extract exactly these marked UI elements area. "#; pub(super) fn gen_extract_prompt(separation_notes: Vec) -> String { let extract_system_prompt = format!( diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index 67b8a66a6..0c87b1cdc 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -29,7 +29,7 @@ ## 图片编辑与视觉绑定 -- image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。处理父节点时，紫色填充其 children 的矩形区域（包括已 problematic 的 children），再在父节点自身外围绘制绿色框；绿色框覆盖在紫色之上。叶节点只绘制绿色框，不填充自身。 +- image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。处理父节点时，紫色填充其 children 的矩形区域（包括已 problematic 的 children），再在父节点自身外围绘制绿色框和角到角的绿色交叉线；绿色标记覆盖在紫色之上。叶节点只绘制绿色框和角到角的绿色交叉线，不填充自身。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 - 标记图构建、处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 From 6e4f9a6f1feb5003a51de81e70d3e8175b310be3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 16:29:54 +0800 Subject: [PATCH 063/248] =?UTF-8?q?=E5=AE=8C=E5=96=84=20UI=20=E8=87=AA?= =?UTF-8?q?=E5=8A=A8=E5=88=86=E7=A6=BB=E8=BF=94=E5=B7=A5=E6=84=8F=E8=A7=81?= =?UTF-8?q?=E9=93=BE=E8=B7=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录视觉模型 NeedRework 意见并注入后续提取提示 同步返工次数、problematic 终态和 512 字符校验 补充 separation DTO 类型、回归测试与技术方案 --- .../src/ui_editor/commands/separation/mod.rs | 98 ++++++++++++++++++- .../ui_editor/commands/separation/model.rs | 13 ++- .../src/ui_editor/commands/separation/tree.rs | 9 +- .../ui_editor/commands/separation/workflow.rs | 16 ++- .../src/features/ui-editor/types/BoundNode.ts | 4 + .../ui-editor/types/ProblematicNode.ts | 4 + .../features/ui-editor/types/SeparationDTO.ts | 5 + .../ui-editor/types/SeparationNode.ts | 5 + .../ui-editor/types/SeparationNote.ts | 3 + ...术方案】UI编辑器自动分离工作流-2026-09-08.md | 3 +- 10 files changed, 151 insertions(+), 9 deletions(-) create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/BoundNode.ts create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/ProblematicNode.ts create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationDTO.ts create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNote.ts diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 7f86ff6f6..bf51348a7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -116,13 +116,109 @@ mod tests { height_px: 1, note: SeparationNote { description: "image".to_string(), - text_note: String::new(), + rework_notes: Vec::new(), }, children: vec![], rework_count: 0, }; assert!(validate_binding_response(&BindingResp { decisions: vec![] }, &[&node]).is_err()); } + + #[test] + fn separation_note_prompt_keeps_rework_notes_in_order() { + let without_notes = SeparationNote { + description: "按钮".to_string(), + rework_notes: Vec::new(), + }; + assert_eq!(without_notes.as_prompt(), "desc: 按钮"); + + let with_notes = SeparationNote { + description: "按钮".to_string(), + rework_notes: vec!["保留圆角".to_string(), "去掉阴影".to_string()], + }; + assert_eq!( + with_notes.as_prompt(), + "desc: 按钮\nprevious rework notes:\n- 保留圆角\n- 去掉阴影" + ); + } + + #[test] + fn need_rework_appends_note_and_final_attempt_becomes_problematic() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let mut separation = construct_separation_state(&state(node( + "root", + vec![], + vec![node("image", vec![image], vec![])], + ))); + let id = NodeId::new("image").unwrap(); + let paths = HashMap::new(); + + for note in ["第一次意见", "第二次意见", "最后一次意见"] { + apply_batch_patch( + &mut separation, + 0, + &[BindingDecision::NeedRework { + to_node: id.clone(), + problem_description: note.to_string(), + }], + &paths, + ) + .unwrap(); + } + + let node = &separation.trees[0].root.children[0]; + assert_eq!( + node.note.rework_notes, + ["第一次意见", "第二次意见", "最后一次意见"] + ); + assert_eq!(separation.problematic_nodes.len(), 1); + assert_eq!( + separation.problematic_nodes[0].rework_count, + MAX_REWORK_COUNT + ); + assert_eq!(node.rework_count, MAX_REWORK_COUNT); + assert!(next_leaf_batch(&separation, &separation.trees[0]).is_empty()); + } + + #[test] + fn next_extract_prompt_contains_previous_rework_notes() { + let note = SeparationNote { + description: "图标".to_string(), + rework_notes: vec!["不要带父背景".to_string()], + }; + let prompt = super::prompt::gen_extract_prompt(vec![note]); + assert!(prompt.contains("previous rework notes:\n- 不要带父背景")); + } + + #[test] + fn binding_validation_rejects_overlong_rework_note() { + let node = SeparationNode { + id: NodeId::new("image").unwrap(), + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + note: SeparationNote::default(), + children: vec![], + rework_count: 0, + }; + let decision = BindingDecision::NeedRework { + to_node: node.id.clone(), + problem_description: "x".repeat(MAX_REWORK_NOTE_CHARS + 1), + }; + assert!(validate_binding_response( + &BindingResp { + decisions: vec![decision] + }, + &[&node] + ) + .is_err()); + } #[test] fn sidecar_name_uses_asset_id_digest() { let dir = separation_sidecar_dir(Path::new("/tmp/project"), "ui:1").unwrap(); diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs index 85773a411..16b617403 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs @@ -5,16 +5,25 @@ use ts_rs::TS; pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; pub const MAX_REWORK_COUNT: u32 = 3; +pub const MAX_REWORK_NOTE_CHARS: usize = 512; #[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] pub struct SeparationNote { pub description: String, - pub text_note: String, + pub rework_notes: Vec, } impl SeparationNote { pub fn as_prompt(&self) -> String { - format!("desc: {} {}", self.description, self.text_note) + let mut prompt = format!("desc: {}", self.description); + if !self.rework_notes.is_empty() { + prompt.push_str("\nprevious rework notes:"); + for note in &self.rework_notes { + prompt.push_str("\n- "); + prompt.push_str(note); + } + } + prompt } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index dcca538e7..42e09f701 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -63,7 +63,7 @@ fn collect_todo_nodes( height_px: h, note: SeparationNote { description: node_description(node), - text_note: String::new(), + rework_notes: Vec::new(), }, children, rework_count: 0, @@ -102,7 +102,7 @@ pub fn construct_separation_state(state: &State) -> SeparationState { height_px: h, note: SeparationNote { description: node_description(&tree.root), - text_note: String::new(), + rework_notes: Vec::new(), }, children, rework_count: 0, @@ -217,6 +217,11 @@ pub fn validate_binding_response( if problem_description.trim().is_empty() { return Err("NeedRework 必须包含问题描述".to_string()); } + if problem_description.chars().count() > MAX_REWORK_NOTE_CHARS { + return Err(format!( + "NeedRework 问题描述不能超过 {MAX_REWORK_NOTE_CHARS} 个字符" + )); + } } } if seen.len() != expected.len() { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 506f291af..6078abf04 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -66,15 +66,15 @@ pub fn apply_batch_patch( to_node, problem_description, } => { + append_rework_note(&mut tree.root, to_node, problem_description); let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; + increment_rework_count(&mut tree.root, to_node, count); if count >= MAX_REWORK_COUNT { state.problematic_nodes.push(ProblematicNode { node_id: to_node.clone(), problem_description: problem_description.clone(), rework_count: count, }); - } else { - increment_rework_count(&mut tree.root, to_node, count); } } } @@ -89,6 +89,16 @@ pub fn apply_batch_patch( Ok(()) } +fn append_rework_note(node: &mut SeparationNode, id: &NodeId, note: &str) -> bool { + if node.id == *id { + node.note.rework_notes.push(note.to_string()); + return true; + } + node.children + .iter_mut() + .any(|child| append_rework_note(child, id, note)) +} + fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { if node.id == *id { node.rework_count = count; @@ -512,7 +522,7 @@ pub(crate) async fn separate_ui_impl( for decision in &binding.decisions { if let BindingDecision::Ok { to_node, - extracted_area: separated_image_area, + extracted_area: separated_image_area, } = decision { let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/BoundNode.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/BoundNode.ts new file mode 100644 index 000000000..b27bee935 --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/BoundNode.ts @@ -0,0 +1,4 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { NodeId } from "./NodeId"; + +export type BoundNode = { node_id: NodeId, cut_image_path: string, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/ProblematicNode.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/ProblematicNode.ts new file mode 100644 index 000000000..c77e0a49b --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/ProblematicNode.ts @@ -0,0 +1,4 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { NodeId } from "./NodeId"; + +export type ProblematicNode = { node_id: NodeId, problem_description: string, rework_count: number, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationDTO.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationDTO.ts new file mode 100644 index 000000000..7408ed0d0 --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationDTO.ts @@ -0,0 +1,5 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { BoundNode } from "./BoundNode"; +import type { ProblematicNode } from "./ProblematicNode"; + +export type SeparationDTO = { bound_nodes: Array, problematic_nodes: Array, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts new file mode 100644 index 000000000..7aaa94d6f --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts @@ -0,0 +1,5 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { NodeId } from "./NodeId"; +import type { SeparationNote } from "./SeparationNote"; + +export type SeparationNode = { id: NodeId, global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, note: SeparationNote, children: Array, rework_count: number, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNote.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNote.ts new file mode 100644 index 000000000..0cb01f4ee --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNote.ts @@ -0,0 +1,3 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SeparationNote = { description: string, rework_notes: Array, }; diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index 0c87b1cdc..2ec3ae1b1 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -35,7 +35,8 @@ `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 -- `NeedRework` 携带短问题描述。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 +- `NeedRework` 携带短问题描述（最多 512 个 Unicode 字符）；通过校验后按产生顺序追加到目标 `SeparationNode.note.rework_notes`，下一次该节点进入 image-edit 时全部意见会注入提取 prompt。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 +- 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 - 父节点背景重建由 image-edit 模型完成，不由 Rust 硬编码重建算法完成。 From d4b5f9fc14cb8317f210d1e929c04bba3d8cec8b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 16:45:55 +0800 Subject: [PATCH 064/248] =?UTF-8?q?=E9=87=8D=E6=9E=84=20UI=20Editor=20LLM?= =?UTF-8?q?=20repair=20history?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 request_with_feedback 改为 run_with_repair_history 并统一 max_retries 与 validater 命名。 让 requester 接收 append-only LlmMessage history，业务校验失败追加序列化响应和错误反馈。 迁移 visual_binding 调用方并补充 retry-only 与业务反馈 history 测试。 同步 UI Editor 结构化请求 repair history 决策文档。 --- .../ui_editor/commands/separation/workflow.rs | 36 ++--- .../src-tauri/src/ui_editor/commands/utils.rs | 137 +++++++++++++++--- .../shared-memory/decision-log.md | 6 + 3 files changed, 135 insertions(+), 44 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 6078abf04..1081ff2fc 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -5,7 +5,7 @@ use crate::platform_session::current_platform_session; use crate::ui_editor::commands::separation::*; use crate::ui_editor::commands::utils::{ parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, - request_with_feedback, strict_json_schema, + run_with_repair_history, strict_json_schema, }; use crate::ui_editor::state::State; use crate::ui_editor::utils::NodeId; @@ -275,30 +275,26 @@ async fn visual_binding( ) .with_strict(true); let base_prompt = gen_binding_prompt(nodes.to_vec()); - let result = request_with_feedback( + let initial_history = vec![LlmMessage::user_multimodal(vec![ + LlmMessageContentPart::InputText { text: base_prompt }, + LlmMessageContentPart::InputImage { + image_url: source_url.clone(), + }, + LlmMessageContentPart::InputImage { + image_url: processed_url.clone(), + }, + ])]; + let result = run_with_repair_history( 2, - |feedback| { - let prompt = feedback.map_or_else( - || base_prompt.clone(), - |error| format!("{base_prompt}\n上一次输出错误：{error}\n请修正并完整返回。"), - ); - let source_url = source_url.clone(); - let processed_url = processed_url.clone(); + initial_history, + |history| { let tool = tool.clone(); let client = client.clone(); let llm_config = llm_config.clone(); async move { - let request = LlmRunRequest::new(vec![LlmMessage::user_multimodal(vec![ - LlmMessageContentPart::InputText { text: prompt }, - LlmMessageContentPart::InputImage { - image_url: source_url, - }, - LlmMessageContentPart::InputImage { - image_url: processed_url, - }, - ])]) - .with_function_tools(vec![tool.clone()]) - .with_tool_choice(LlmToolChoice::Required); + let request = LlmRunRequest::new(history) + .with_function_tools(vec![tool.clone()]) + .with_tool_choice(LlmToolChoice::Required); request_ui_editor_llm(&client, &llm_config, request) .await .map_err(|e| e.to_string()) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index 8aec25302..4f849d227 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -1,8 +1,9 @@ use crate::agent::request_game_creator_llm_text; use crate::config::{apply_game_creator_llm_reasoning_effort, parse_game_creator_llm_api_kind}; use base64::Engine as _; -use platform_llm::{LlmClient, LlmError, LlmRunRequest, LlmRunResponse}; +use platform_llm::{LlmClient, LlmError, LlmMessage, LlmRunRequest, LlmRunResponse}; use schemars::JsonSchema; +use serde::Serialize; use std::fs::File; use std::future::Future; use std::io::Read; @@ -26,31 +27,39 @@ pub(crate) async fn request_ui_editor_llm( request_game_creator_llm_text(client, llm, request).await } -/// 结构化 LLM 请求的小型 repair harness：第一次请求或校验失败后， -/// 将错误反馈给模型并只额外重试一次。网络/模型调用本身的错误也会 -/// 进入第二次请求的反馈文本；调用方负责在第二次失败后决定业务状态。 -pub(crate) async fn request_with_feedback( - more_turn: usize, - request: Request, - validate: Validate, +/// 按 append-only history 重试结构化 LLM 请求；仅业务校验失败会追加反馈消息。 +pub(crate) async fn run_with_repair_history( + max_retries: usize, + initial_history: Vec, + requester: Requester, + validater: Validater, ) -> Result where - Request: Fn(Option) -> Fut, + T: Serialize, + Requester: Fn(Vec) -> Fut, Fut: Future>, - Validate: Fn(&T) -> Result<(), String>, + Validater: Fn(&T) -> Result<(), String>, { - let mut feedback = None; - for attempt in 0..=more_turn { - let result = request(feedback.clone()) - .await - .and_then(|value| validate(&value).map(|_| value)); - match result { - Ok(value) => return Ok(value), - Err(error) if attempt < more_turn => feedback = Some(error), + let mut history = initial_history; + for attempt in 0..=max_retries { + let value = match requester(history.clone()).await { + Ok(value) => value, + Err(_error) if attempt < max_retries => continue, + Err(error) => return Err(error), + }; + match validater(&value) { + Ok(()) => return Ok(value), + Err(error) if attempt < max_retries => { + let serialized = serde_json::to_string(&value) + .map_err(|serialize_error| format!("序列化修复反馈失败：{serialize_error}"))?; + history.push(LlmMessage::system(format!( + "上一次模型输出：\n{serialized}\n\n业务校验失败：\n{error}\n\n请修正并完整返回。" + ))); + } Err(error) => return Err(error), } } - unreachable!("repair harness always returns within requested turns") + unreachable!("repair history runner always returns within requested retries") } pub(crate) fn parse_limited_llm_tool_arguments( @@ -173,15 +182,17 @@ mod tests { } #[tokio::test] - async fn feedback_harness_zero_more_turn_calls_once_without_feedback() { + async fn repair_history_zero_retries_calls_once_with_initial_history() { let calls = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); let seen = calls.clone(); - let result = request_with_feedback( + let initial_history = vec![LlmMessage::user("初始 prompt")]; + let result = run_with_repair_history( 0, - move |feedback| { + initial_history.clone(), + move |history| { let seen = seen.clone(); async move { - seen.lock().unwrap().push(feedback); + seen.lock().unwrap().push(history); Ok::<_, String>(serde_json::json!({"ok": true})) } }, @@ -190,7 +201,85 @@ mod tests { .await .expect("single turn should succeed"); assert_eq!(result, serde_json::json!({"ok": true})); - assert_eq!(calls.lock().unwrap().as_slice(), &[None]); + assert_eq!(calls.lock().unwrap().as_slice(), &[initial_history]); + } + + #[tokio::test] + async fn repair_history_request_error_retries_without_appending_history() { + let calls = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let attempts = std::sync::Arc::new(std::sync::Mutex::new(0usize)); + let seen_calls = calls.clone(); + let seen_attempts = attempts.clone(); + let initial_history = vec![LlmMessage::user("初始 prompt")]; + let result = run_with_repair_history( + 1, + initial_history.clone(), + move |history| { + seen_calls.lock().unwrap().push(history); + let attempt = { + let mut attempts = seen_attempts.lock().unwrap(); + let attempt = *attempts; + *attempts += 1; + attempt + }; + async move { + if attempt == 0 { + Err("网络错误".to_string()) + } else { + Ok::<_, String>(serde_json::json!({"ok": true})) + } + } + }, + |_| Ok(()), + ) + .await + .expect("retry-only error should recover"); + assert_eq!(result, serde_json::json!({"ok": true})); + assert_eq!( + calls.lock().unwrap().as_slice(), + &[initial_history.clone(), initial_history] + ); + } + + #[tokio::test] + async fn repair_history_business_failure_appends_serialized_value_and_error() { + let calls = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let attempts = std::sync::Arc::new(std::sync::Mutex::new(0usize)); + let seen_calls = calls.clone(); + let seen_attempts = attempts.clone(); + let initial_history = vec![LlmMessage::user("初始 prompt")]; + let result = run_with_repair_history( + 1, + initial_history.clone(), + move |history| { + seen_calls.lock().unwrap().push(history); + let mut attempts = seen_attempts.lock().unwrap(); + let attempt = *attempts; + *attempts += 1; + async move { Ok::<_, String>(serde_json::json!({"attempt": attempt})) } + }, + |value: &serde_json::Value| { + if value["attempt"] == 0 { + Err("业务校验失败".to_string()) + } else { + Ok(()) + } + }, + ) + .await + .expect("business feedback should recover"); + assert_eq!(result, serde_json::json!({"attempt": 1})); + let calls = calls.lock().unwrap(); + assert_eq!(calls.len(), 2); + assert_eq!(calls[0], initial_history); + assert_eq!(calls[1].len(), 2); + assert_eq!(calls[1][0], LlmMessage::user("初始 prompt")); + assert_eq!( + calls[1][1], + LlmMessage::system( + "上一次模型输出：\n{\"attempt\":0}\n\n业务校验失败：\n业务校验失败\n\n请修正并完整返回。" + ) + ); } #[test] diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 00092043a..2e6658f85 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -8034,6 +8034,12 @@ CI 上 `background_agent_runtime_recovers_stale_running_before_pending_task` 在 - AGC LLM 对话入口在解析 Router 凭据和访问上游前先读取用户 `wallet_balance`。余额为 `0` 时直接返回 `409 MUD_POINTS_INSUFFICIENT`，客户端显示“泥点余额不足”；不创建、续期或使用 Router 账号。余额读取失败同样失败关闭，返回“泥点余额暂时不可用”。 - 余额大于 `0` 的请求继续走 Router，成功后仍按 best-effort 后置结算；退款占用、冻结或扣费时余额不足的处理继续由钱包事务和既有结算规则负责。 +## 2026-09-09 UI Editor 结构化请求 repair history + +- UI Editor 的结构化 LLM repair 由 `run_with_repair_history` 统一维护 append-only `LlmMessage` history；调用方只构造初始 prompt 并提供 `requester(history) -> Result`。 +- `validater(&T) -> Result<(), String>` 只负责业务校验。网络、模型、tool 缺失、JSON 或反序列化错误只按原 history 重试；只有业务校验失败才把序列化后的响应和校验错误合并为一条 system message 追加到 history。 +- history 仅存在本次请求内存中，不重复图片、不截断、不扩展 `platform-llm` 消息协议；重试次数参数统一使用 `max_retries`。 + ## 2026-08-29 DirectProject 受控联网搜索默认与边界 - 正式产品本次只覆盖 `DirectProject` 单 Codex Agent。`Provider`、`ToolHost`、`DirectHome` 不是 Agent，也不是本次联网主链路；不新增全路由联网或工具桥。唯一受控联网工具为 `agc_tools.agc_web_search`，链路固定为 Codex MCP 工具目录 -> 客户端 loopback `DirectToolBridge` -> 有界 Bing RSS HTTPS -> 过滤 / 脱敏 -> MCP 结果回传。 From 53a919e39bad9afe206c8f2edeca1707f18f492a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 19:04:56 +0800 Subject: [PATCH 065/248] =?UTF-8?q?=E4=BC=98=E5=8C=96=20UI=20=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E5=B7=A5=E4=BD=9C=E6=B5=81=E7=A8=8B=E7=9A=84=E6=8F=90?= =?UTF-8?q?=E7=A4=BA=E7=94=9F=E6=88=90=E4=B8=8E=E5=8E=86=E5=8F=B2=E5=A4=84?= =?UTF-8?q?=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 调整 initial_history 构建逻辑，增强系统与用户消息的一致性。 - 简化 SeparationNode 提示生成逻辑，仅保留节点 ID 与注释。 - 明确 extracted_area 的权威性为 processed 图像，并更新生成提示的规则。 --- .../ui_editor/commands/separation/model.rs | 6 +--- .../ui_editor/commands/separation/prompt.rs | 5 ++- .../ui_editor/commands/separation/workflow.rs | 32 ++++++++++++------- 3 files changed, 25 insertions(+), 18 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs index 16b617403..985e8fda3 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs @@ -42,12 +42,8 @@ pub struct SeparationNode { impl SeparationNode { pub fn as_prompt(&self) -> String { format!( - "node_id={} area=({}, {}, {}, {}) {}", + "node_id={} note: {}", self.id.as_str(), - self.global_pos_x_px, - self.global_pos_y_px, - self.width_px, - self.height_px, self.note.as_prompt() ) } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs index d6df93e9f..ab32085b0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs @@ -33,7 +33,6 @@ pub(super) fn gen_extract_prompt(separation_notes: Vec) -> Strin pub(super) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { let binding_system_prompt = format!( r#" - You are working under a UI elements separation workflow. You will be given a src UI design image and a processed image, where some ui elements are separated. Here were the separation requirements: ``` @@ -42,6 +41,10 @@ pub(super) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { You need to recognize and review the separation using the given tool. field notes: * extracted_area MUST be the recognized area from the processed image, INSTEAD OF from the src image. + The processed image is the only authoritative image for extracted_area. + Return the pixel bounding box of the extracted element as it appears in the processed image. + Do not copy, infer, or reuse the source node rectangle. + The src image is only for identifying which semantic UI element belongs to to_node. these node need handle: "# diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 1081ff2fc..2c3a629bb 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -259,6 +259,7 @@ async fn visual_binding( e.to_string() })? .llm; + let client = build_game_creator_llm_client_from_llm_config(&llm_config, "llm").map_err(|e| { app_log!("ui_separation.error stage=visual_binding reason=build_client error={e}"); @@ -274,16 +275,23 @@ async fn visual_binding( schema, ) .with_strict(true); - let base_prompt = gen_binding_prompt(nodes.to_vec()); - let initial_history = vec![LlmMessage::user_multimodal(vec![ - LlmMessageContentPart::InputText { text: base_prompt }, - LlmMessageContentPart::InputImage { - image_url: source_url.clone(), - }, - LlmMessageContentPart::InputImage { - image_url: processed_url.clone(), - }, - ])]; + let initial_history = vec![ + LlmMessage::system(gen_binding_prompt(nodes.to_vec())), + LlmMessage::user_multimodal(vec![ + LlmMessageContentPart::InputText { + text: "processed image:".to_string(), + }, + LlmMessageContentPart::InputImage { + image_url: processed_url.clone(), + }, + LlmMessageContentPart::InputText { + text: "src image:".to_string(), + }, + LlmMessageContentPart::InputImage { + image_url: source_url.clone(), + }, + ]), + ]; let result = run_with_repair_history( 2, initial_history, @@ -518,13 +526,13 @@ pub(crate) async fn separate_ui_impl( for decision in &binding.decisions { if let BindingDecision::Ok { to_node, - extracted_area: separated_image_area, + extracted_area, } = decision { let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); match cut_processed_image( processed_path.clone(), - *separated_image_area, + *extracted_area, cut_path.clone(), ) .await From 3e1e172f9cdd846a10eb97d8809c39f06dc36ace Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 20:03:27 +0800 Subject: [PATCH 066/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E5=89=8D=E7=AB=AF=E6=8E=A5=E5=85=A5=E5=90=88?= =?UTF-8?q?=E5=90=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 明确 UI 编辑器单次调用 separation 与 sidecar 恢复弹窗 补充资源登记、State 回填、保存和 finalize 顺序 划定旧 binding 与 Runtime workflow 的本次不改范围 --- ...【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index 2ec3ae1b1..f3620ee74 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -64,6 +64,17 @@ - 每次重做产生新的 SpriteAssetId，不假设 NodeId 到 SpriteAssetId 的稳定映射。 - sidecar 中的图片保留，正式 SpriteAsset 的最终清理策略列 TODO。 +## 前端正式接入 + +- UI 编辑器点击“自动分离”时，前端只调用一次 `separate_ui`，消费完整 `SeparationDTO`；separation 内部 batch 不向前端暴露，也不在 UI 中显示 batch 进度。 +- 如果 sidecar 已存在未完成的 `state.json`，点击入口时先打开独立弹窗，由用户选择“继续上次分离”或“开始新的分离”。继续复用 sidecar 的 pending tree；重新开始只替换当前 `state.json`，不删除 sidecar 图片。 +- `BoundNode.cut_image_path` 必须是项目根相对路径。前端使用现有 `import_local_project_image_assets` 登记 cut 图片；由于该通用命令单次最多 100 个路径，前端可以在资源登记阶段按 100 条分组调用，但这不属于 separation batch，也不向用户展示。 +- 现有本地资源导入按清洗后的文件名 stem 与内容摘要生成目标路径；相同目标路径直接复用已有 manifest asset ID，内容不同则拒绝覆盖或生成不同摘要路径。前端不自行猜测 SpriteAsset 是否存在，也不从 NodeId 派生 SpriteAssetId。 +- 全部可登记图片完成导入后，前端在一个 `runWithStateLocked` 中复用 `addSpriteAssets` 的内部 State 变换逻辑，加入返回的 SpriteAsset 并回填仍匹配 Node 的第一个未绑定 Image component，最后一次性提交 State。公开 `addSpriteAssets` 的普通 mutation guard 不放宽。 +- UI tree 在 separation 期间发生变化时，已登记的 cut 图片和可匹配节点的回填保留；找不到 Node 或没有未绑定 Image 的结果产生明确问题提示，不回滚已登记资源，不静默跳过。 +- State 保存成功后才调用 `finalize_separation` 删除 sidecar `state.json`；登记、回填或保存失败时保留 sidecar，允许下次选择继续。sidecar 图片按当前 debug 策略保留。 +- 本次只替换 UI 编辑器独立页面的 `bindComponents` 前端入口。`binding.rs`、`bind_components` Tauri 命令及 Runtime `workflow.rs` 的旧 binding 链路暂列后续退役/迁移事项。 + ## 重启与 Raw GPT Image 2 - 已保存的 separation state 是跨重启继续工作的最小单位；重启后从上一个已保存 batch 的状态继续。 @@ -77,7 +88,7 @@ - `Vec` 重构为一种组件类型最多一个的容器。 - 当前第一个 Image component 回填规则的正式替代方案。 - 正在执行 batch 的持久化和恢复。 -- 前端复制、登记 SpriteAsset、回填 State 的精确 IPC/提交合同。 +- 前端登记 SpriteAsset、回填 State、sidecar 恢复弹窗和 `finalize_separation` 的实现与测试。 - 临时图片清理/归档策略。 - 手动抠图能力。 - problematic 对更高层 workflow 完成门禁的最终定义。 From 615d1a28cd4b497f945fc5ad75f051f90f096720 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 20:16:09 +0800 Subject: [PATCH 067/248] =?UTF-8?q?=E8=A1=A5=E9=BD=90=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=81=A2=E5=A4=8D=E7=8A=B6=E6=80=81=E8=83=B6?= =?UTF-8?q?=E6=B0=B4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增 separation 恢复探测、完成清理命令与 DTO。 持久化项目相对 cut 图片路径并保留 state sidecar 到最终提交。 --- .../src-tauri/src/main.rs | 27 +++++++++++ .../src-tauri/src/ui_editor/commands/mod.rs | 2 +- .../ui_editor/commands/separation/model.rs | 9 ++++ .../commands/separation/persistence.rs | 48 +++++++++++++++++++ .../ui_editor/commands/separation/workflow.rs | 21 ++------ 5 files changed, 90 insertions(+), 17 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/main.rs b/apps/ai-game-creator-shell/src-tauri/src/main.rs index 3f4293087..1f51a98ae 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/main.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/main.rs @@ -337,6 +337,30 @@ async fn separate_ui( ui_editor::commands::separate_ui_impl(project_path, asset_id, state).await } +#[tauri::command] +fn inspect_separation_recovery( + project_path: String, + asset_id: String, +) -> Result { + let root = Path::new(project_path.trim()); + enforce_project_permission_policy(root, "asset.list")?; + ui_editor::commands::separation::inspect_separation_recovery(root, &asset_id) +} + +#[tauri::command] +fn finalize_separation(project_path: String, asset_id: String) -> Result<(), String> { + let root = Path::new(project_path.trim()); + enforce_project_permission_policy(root, "asset.register")?; + ui_editor::commands::separation::finalize_separation(root, &asset_id) +} + +#[tauri::command] +fn discard_separation_recovery(project_path: String, asset_id: String) -> Result<(), String> { + let root = Path::new(project_path.trim()); + enforce_project_permission_policy(root, "asset.register")?; + ui_editor::commands::separation::finalize_separation(root, &asset_id) +} + #[tauri::command] async fn merge_ui(state: ui_editor::state::State) -> Result { ui_editor::commands::merge_ui_impl(state).await @@ -2569,6 +2593,9 @@ fn main() { suggest_ui_design_semantic, recognize_ui, separate_ui, + inspect_separation_recovery, + finalize_separation, + discard_separation_recovery, merge_ui, bind_components, load_ui_design_state, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs index 181931f2d..d85432e07 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/mod.rs @@ -12,6 +12,6 @@ pub(crate) use merge::{merge_ui_impl, merge_ui_impl_with_provider}; pub use recognition::RecognitionDTO; pub(crate) use recognition::{recognize_ui_impl, recognize_ui_impl_with_provider}; pub(crate) use separation::separate_ui_impl; -pub use separation::SeparationDTO; +pub use separation::{SeparationDTO, SeparationRecoveryDTO}; pub(crate) use ui_design_suggestion::suggest_ui_design_semantic_impl; pub use ui_design_suggestion::UIDesignSuggestionTreeNode; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs index 985e8fda3..12fca349c 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs @@ -84,6 +84,15 @@ pub struct SeparationDTO { pub problematic_nodes: Vec, } +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationRecoveryDTO { + pub exists: bool, + pub bound_node_count: usize, + pub problematic_node_count: usize, + pub has_pending_tree: bool, +} + #[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] #[schemars(deny_unknown_fields)] pub struct BindingArea { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index e9d7c9296..3ec6aa8fb 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -25,6 +25,21 @@ pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result Result { + Ok(separation_sidecar_dir(root, asset_id)?.join("state.json")) +} + +pub fn project_relative_path(root: &Path, path: &Path) -> Result { + let relative = path + .strip_prefix(root) + .map_err(|_| "separation 产物必须位于项目目录内".to_string())?; + let value = relative.to_string_lossy().replace('\\', "/"); + if value.is_empty() || value.starts_with('/') || value.split('/').any(|part| part == "..") { + return Err("separation 产物相对路径无效".to_string()); + } + Ok(value) +} + pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<(), String> { if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { app_log!("ui_separation.error stage=state_write reason=schema_mismatch"); @@ -116,3 +131,36 @@ pub fn separation_dto(state: &SeparationState) -> SeparationDTO { problematic_nodes: state.problematic_nodes.clone(), } } + +pub fn inspect_separation_recovery( + root: &Path, + asset_id: &str, +) -> Result { + let state_path = separation_state_path(root, asset_id)?; + if !state_path.exists() { + return Ok(SeparationRecoveryDTO { + exists: false, + bound_node_count: 0, + problematic_node_count: 0, + has_pending_tree: false, + }); + } + let state = read_separation_state(&state_path)?; + Ok(SeparationRecoveryDTO { + exists: true, + bound_node_count: state.bound.len(), + problematic_node_count: state.problematic_nodes.len(), + has_pending_tree: state.trees.iter().any(|tree| { + !tree.root.children.is_empty() || tree.root_extractable + }), + }) +} + +pub fn finalize_separation(root: &Path, asset_id: &str) -> Result<(), String> { + let state_path = separation_state_path(root, asset_id)?; + match fs::remove_file(&state_path) { + Ok(()) => Ok(()), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(format!("删除 separation state 失败：{error}")), + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 2c3a629bb..da3a1288e 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -367,7 +367,7 @@ pub(crate) async fn separate_ui_impl( ); format!("创建 separation sidecar 失败：{e}") })?; - let state_path = sidecar.join("state.json"); + let state_path = separation_state_path(root, &asset_id)?; let restored = state_path.exists(); let mut separation = if restored { app_log!("ui_separation.state_restore.start asset_id={}", asset_id); @@ -538,8 +538,10 @@ pub(crate) async fn separate_ui_impl( .await { Ok(()) => { - cut_paths - .insert(to_node.clone(), cut_path.to_string_lossy().to_string()); + cut_paths.insert( + to_node.clone(), + project_relative_path(root, &cut_path)?, + ); } Err(error) => { app_log!( @@ -577,19 +579,6 @@ pub(crate) async fn separate_ui_impl( batch_index += 1; } } - match fs::remove_file(&state_path) { - Ok(()) => app_log!("ui_separation.state_removed asset_id={}", asset_id), - Err(error) if error.kind() == std::io::ErrorKind::NotFound => { - app_log!( - "ui_separation.state_remove_skipped asset_id={} reason=not_found", - asset_id - ) - } - Err(error) => app_log!( - "ui_separation.error stage=state_remove asset_id={} error={error}", - asset_id - ), - } app_log!( "ui_separation.completed asset_id={} bound_nodes={} problematic_nodes={}", asset_id, From 5844afa14d493e881b77647372b000bbbe62840d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 20:17:11 +0800 Subject: [PATCH 068/248] =?UTF-8?q?=E6=8A=BD=E5=8F=96=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E7=B4=A0=E6=9D=90=E7=8A=B6=E6=80=81=E5=8F=98?= =?UTF-8?q?=E6=8D=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复用 SpriteAsset 校验并支持锁内构造下一份 State。 保留公开 addSpriteAssets 的锁保护与历史提交行为。 --- .../features/ui-editor/useUiEditorState.ts | 63 +++++++++++-------- 1 file changed, 37 insertions(+), 26 deletions(-) diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts b/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts index a65ed50dc..d04c52f93 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts @@ -524,6 +524,40 @@ function spriteResourceValidationError(sprite: SpriteAsset) { return border.ok ? null : border.message; } +/** + * Apply the SpriteAsset resource checks without acquiring the editor mutation + * guard. Workflow adapters use this while a State lock is already held so + * resource insertion and component backfill can be committed atomically. + */ +export function addSpriteAssetsToState( + current: State, + assets: readonly SpriteAsset[], +): UiEditorOperationResult { + const unique = new Map(); + for (const asset of assets) { + const candidate = + unique.get(asset.asset_id) ?? current.sprite_assets[asset.asset_id]; + if (candidate && !sameResource(candidate, asset)) { + return { ok: false, reason: 'duplicate' }; + } + unique.set(asset.asset_id, asset); + } + const invalidSprite = [...unique.values()] + .map((asset) => ({ asset, error: spriteResourceValidationError(asset) })) + .find((item) => item.error); + if (invalidSprite?.error) { + return { + ok: false, + reason: `invalid:${invalidSprite.asset.asset_id}：${invalidSprite.error}`, + }; + } + const next = cloneState(current); + for (const asset of unique.values()) { + next.sprite_assets[asset.asset_id] = structuredClone(asset); + } + return { ok: true, value: next }; +} + function fontResourceValidationError(font: FontAsset) { if (font.asset_id.trim().length === 0) return '缺少字体 ID'; if (font.path.trim().length === 0) return '缺少字体路径'; @@ -782,32 +816,9 @@ export function useUiEditorState(initialState: State = EMPTY_UI_EDITOR_STATE) { const blocked = guard(); if (blocked) return blocked; const current = stateRef.current; - const unique = new Map(); - for (const asset of assets) { - const candidate = - unique.get(asset.asset_id) ?? current.sprite_assets[asset.asset_id]; - if (candidate && !sameResource(candidate, asset)) { - return { ok: false, reason: 'duplicate' }; - } - unique.set(asset.asset_id, asset); - } - const invalidSprite = [...unique.values()] - .map((asset) => ({ - asset, - error: spriteResourceValidationError(asset), - })) - .find((item) => item.error); - if (invalidSprite?.error) { - return { - ok: false, - reason: `invalid:${invalidSprite.asset.asset_id}：${invalidSprite.error}`, - }; - } - const next = cloneState(current); - for (const asset of unique.values()) { - next.sprite_assets[asset.asset_id] = structuredClone(asset); - } - commit(next); + const result = addSpriteAssetsToState(current, assets); + if (!result.ok) return result; + commit(result.value); return { ok: true, value: undefined }; }, [commit, guard], From 7fb7d246c045fa4f1d5d1f28c89f8d7dd058453e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 20:51:18 +0800 Subject: [PATCH 069/248] =?UTF-8?q?=E5=B0=86=E8=A7=86=E8=A7=89=E6=AD=A5?= =?UTF-8?q?=E9=AA=A4=E5=88=87=E6=8D=A2=E4=B8=BA=E8=87=AA=E5=8A=A8=E5=88=86?= =?UTF-8?q?=E7=A6=BB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 前端仅调用一次 separate_ui，并完成 cut 图片登记、SpriteAsset 回填和 State 保存。 增加 sidecar 恢复选择面板，旧 bind_components 标记为未调用遗留命令。 --- .../scripts/check-config.mjs | 2 + .../ui-editor/types/SeparationRecoveryDTO.ts | 3 + .../components/WorkflowActionCard.tsx | 51 +++- .../src/view/ui-editor/useUiEditorPage.ts | 272 +++++++++++++++--- 4 files changed, 280 insertions(+), 48 deletions(-) create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationRecoveryDTO.ts diff --git a/apps/ai-game-creator-shell/scripts/check-config.mjs b/apps/ai-game-creator-shell/scripts/check-config.mjs index 0149d953a..f2ef7b4b8 100644 --- a/apps/ai-game-creator-shell/scripts/check-config.mjs +++ b/apps/ai-game-creator-shell/scripts/check-config.mjs @@ -108,6 +108,8 @@ const rustSharedContractSource = fs.readFileSync( ); const allowedUncalledTauriCommands = [ 'append_direct_project_conversation_message', + // TODO: Remove the retired binding command after the legacy runtime path is removed. + 'bind_components', 'chat_with_game_creator_agent', 'check_ui_editor_font_glyph_coverage', 'create_ui_design_resource', diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationRecoveryDTO.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationRecoveryDTO.ts new file mode 100644 index 000000000..38c5bb8b2 --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationRecoveryDTO.ts @@ -0,0 +1,3 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SeparationRecoveryDTO = { exists: boolean, bound_node_count: number, problematic_node_count: number, has_pending_tree: boolean, }; diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx index c5bf945e4..695c10163 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx @@ -1,3 +1,4 @@ +import { ThemedModal } from '../../../components/modal/ThemedModal'; import type { UiEditorWorkflowProjection } from '../useUiEditorPage'; export function WorkflowActionCard({ @@ -10,19 +11,19 @@ export function WorkflowActionCard({ (workflow.activeStep === 'reference-analysis' && workflow.isSuggesting) || (workflow.activeStep === 'structure-recognition' && workflow.isRecognizing) || - (workflow.activeStep === 'visual-binding' && workflow.isBinding); + (workflow.activeStep === 'visual-binding' && workflow.isSeparating); const status = workflow.activeStep === 'reference-analysis' ? workflow.suggestionStatus : workflow.activeStep === 'structure-recognition' ? workflow.recognitionStatus - : workflow.bindingStatus; + : workflow.separationStatus; const hasRun = workflow.activeStep === 'reference-analysis' ? workflow.hasSuggested : workflow.activeStep === 'structure-recognition' ? workflow.hasRecognized - : workflow.hasBound; + : workflow.hasSeparated; return (
@@ -56,6 +57,44 @@ export function WorkflowActionCard({ ) : null} + +

发现未完成的自动分离

+

+ 上次分离留下了可恢复状态（已登记{' '} + {workflow.separationRecovery?.bound_node_count ?? 0}{' '} + 个节点）。请选择继续上次分离，或开始新的分离。 +

+
+ + + +
+
); } @@ -76,8 +115,8 @@ function getStepAction(workflow: UiEditorWorkflowProjection) { }; } return { - label: '绑定视觉素材', - runningLabel: '绑定中…', - action: workflow.bindComponents, + label: '自动分离并绑定视觉素材', + runningLabel: '自动分离中…', + action: workflow.separateUi, }; } diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index a580a652d..4ff18bcf9 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -2,7 +2,6 @@ import { invoke } from '@tauri-apps/api/core'; import { useCallback, useEffect, useMemo, useState } from 'react'; import type { ImportedAsset } from '../../components/AssetImporter'; -import { applyBindingResult } from '../../features/ui-editor/binding'; import { prepareDesignImageBatch, prepareFontAssetBatch, @@ -15,7 +14,6 @@ import { type StageStatusField, } from '../../features/ui-editor/stageStatusOverview'; import { collectUiNodeIds } from '../../features/ui-editor/treeUtils'; -import type { BindingDTO } from '../../features/ui-editor/types/BindingDTO'; import type { ChildrenDisplayMode } from '../../features/ui-editor/types/ChildrenDisplayMode'; import type { Component } from '../../features/ui-editor/types/Component'; import type { FontAssetId } from '../../features/ui-editor/types/FontAssetId'; @@ -23,6 +21,8 @@ import type { MergeDTO } from '../../features/ui-editor/types/MergeDTO'; import type { Node as UiNode } from '../../features/ui-editor/types/Node'; import type { NodeId } from '../../features/ui-editor/types/NodeId'; import type { RecognitionDTO } from '../../features/ui-editor/types/RecognitionDTO'; +import type { SeparationDTO } from '../../features/ui-editor/types/SeparationDTO'; +import type { SeparationRecoveryDTO } from '../../features/ui-editor/types/SeparationRecoveryDTO'; import type { SpriteAssetId } from '../../features/ui-editor/types/SpriteAssetId'; import type { SpriteBorder } from '../../features/ui-editor/types/SpriteBorder'; import type { State } from '../../features/ui-editor/types/State'; @@ -37,6 +37,7 @@ import { } from '../../features/ui-editor/uiDesignStateStore'; import { applyUiDesignSuggestions } from '../../features/ui-editor/uiDesignSuggestions'; import { useUiEditorFontFaces } from '../../features/ui-editor/useUiEditorFontFaces'; +import { addSpriteAssetsToState } from '../../features/ui-editor/useUiEditorState'; import { EMPTY_UI_EDITOR_STATE, type NodeLayoutPatch, @@ -70,7 +71,15 @@ import { } from './model'; import { useUiEditorNodeFocus } from './useUiEditorNodeFocus'; -const ASSET_BATCH_SIZE = 5; +const SEPARATION_IMPORT_BATCH_SIZE = 100; + +type LocalImageImportResponse = { + assets: Array<{ id: string; localPath: string; assetKind?: string | null }>; +}; + +function normalizeProjectRelativePath(path: string): string { + return path.replaceAll('\\', '/').replace(/^\/+/, ''); +} type StatusFieldHighlight = { nodeId: NodeId; @@ -250,11 +259,13 @@ export function useUiEditorSession( ); const [isMerging, setIsMerging] = useState(false); const [mergeStatus, setMergeStatus] = useState(null); - const [isBinding, setIsBinding] = useState(false); - const [bindingStatus, setBindingStatus] = useState(null); + const [isSeparating, setIsSeparating] = useState(false); + const [separationStatus, setSeparationStatus] = useState(null); + const [separationRecovery, setSeparationRecovery] = + useState(null); const [hasSuggested, setHasSuggested] = useState(false); const [hasRecognized, setHasRecognized] = useState(false); - const [hasBound, setHasBound] = useState(false); + const [hasSeparated, setHasSeparated] = useState(false); const [completionNotice, setCompletionNotice] = useState(null); @@ -393,7 +404,8 @@ export function useUiEditorSession( image.metadata.role === 'Page' && !isSlaveToDescendant(images, id as UIDesignImageId, activeImageId), ); - const isAiRunning = isSuggesting || isRecognizing || isBinding || isMerging; + const isAiRunning = + isSuggesting || isRecognizing || isMerging || isSeparating; const isWorkflowBusy = isAiRunning || isSaving || isGenerating || isLoading || editor.isLocked; const stateSignature = JSON.stringify(editor.state); @@ -1061,53 +1073,227 @@ export function useUiEditorSession( } } - async function bindComponents() { - if (isBinding || isWorkflowBusy) return; + async function runSeparationWorkflow() { + if (!resourceId) return; + setIsSeparating(true); + setSeparationStatus(null); setCompletionNotice(null); - setBindingStatus(null); - setIsBinding(true); + let preparedSprites: Awaited> = + []; try { + let backfillErrors: string[] = []; + let separationResult: SeparationDTO | null = null; await editor.runWithStateLocked(async (snapshot) => { - const allSpriteIds = Object.keys(snapshot.sprite_assets); - const batches: string[][] = []; + const result = await invoke('separate_ui', { + projectPath, + assetId: resourceId, + state: snapshot, + }); + separationResult = result; + + const uniquePaths = [ + ...new Set( + result.bound_nodes.map((bound) => + normalizeProjectRelativePath(bound.cut_image_path), + ), + ), + ]; + const importedByPath = new Map< + string, + { id: string; localPath: string; assetKind: string | null } + >(); for ( let index = 0; - index < allSpriteIds.length; - index += ASSET_BATCH_SIZE + index < uniquePaths.length; + index += SEPARATION_IMPORT_BATCH_SIZE ) { - batches.push(allSpriteIds.slice(index, index + ASSET_BATCH_SIZE)); + const relativePaths = uniquePaths.slice( + index, + index + SEPARATION_IMPORT_BATCH_SIZE, + ); + const imported = await invoke( + 'import_local_project_image_assets', + { projectPath, relativePaths }, + ); + for (const [assetIndex, asset] of imported.assets.entries()) { + const normalizedAsset = { + id: asset.id, + localPath: normalizeProjectRelativePath(asset.localPath), + assetKind: asset.assetKind ?? null, + }; + // The importer may copy a sidecar file into assets/uploads and + // therefore return a different localPath. Keep both identities: + // the cut path is the separation contract, while the returned + // path is the SpriteAsset resource path. + importedByPath.set(normalizedAsset.localPath, normalizedAsset); + const requestedPath = relativePaths[assetIndex]; + if (requestedPath) { + importedByPath.set( + normalizeProjectRelativePath(requestedPath), + normalizedAsset, + ); + } + } } - if (batches.length === 0) batches.push([]); - let current = snapshot; - for (const [index, spriteIds] of batches.entries()) { - setBindingStatus(`绑定组件中（${index + 1}/${batches.length}）…`); - const result = await invoke('bind_components', { - projectPath, - state: current, - spriteIds, - }); - current = applyBindingResult(current, result); - editor.replaceState(current, { - history: index < batches.length - 1 ? 'skip' : 'record', - }); + + const missingImports = uniquePaths.filter( + (path) => !importedByPath.has(path), + ); + backfillErrors = missingImports.map( + (path) => `未能登记分离图片：${path}`, + ); + const importedAssets: ImportedAsset[] = [ + ...new Map( + [...importedByPath.values()].map((asset) => [asset.id, asset]), + ).values(), + ]; + preparedSprites = await prepareSpriteAssetBatch( + projectPath, + importedAssets, + ); + const spriteById = new Map( + preparedSprites.map((item) => [ + item.resource.asset_id, + item.resource, + ]), + ); + const spriteByPath = new Map( + [...importedByPath.entries()].flatMap(([path, asset]) => { + const sprite = spriteById.get(asset.id); + return sprite ? [[path, sprite] as const] : []; + }), + ); + const added = addSpriteAssetsToState( + snapshot, + preparedSprites.map((item) => item.resource), + ); + if (!added.ok) { + throw new Error(uiEditorOperationError(added.reason)); } + const next = added.value; + for (const bound of result.bound_nodes) { + const path = normalizeProjectRelativePath(bound.cut_image_path); + const sprite = spriteByPath.get(path); + if (!sprite) { + backfillErrors.push( + `节点 ${bound.node_id} 缺少已登记的分离图片：${path}`, + ); + continue; + } + const location = next.ui_trees + .map((tree) => findUiNodeLocation(tree.root, bound.node_id)) + .find((candidate) => candidate !== null); + if (!location) { + backfillErrors.push(`节点 ${bound.node_id} 已不存在，素材已保留`); + continue; + } + const imageComponent = location.node.components.find( + (component): component is Extract => + 'Image' in component && component.Image.target_graphic === null, + ); + if (!imageComponent || !('Image' in imageComponent)) { + backfillErrors.push( + `节点 ${bound.node_id} 没有可回填的未绑定 Image 组件，素材已保留`, + ); + continue; + } + imageComponent.Image.target_graphic = sprite.asset_id; + } + editor.replaceState(next); + }); + + setPreviewUrls((current) => ({ + ...current, + ...Object.fromEntries( + preparedSprites.map((item) => [ + item.resource.asset_id, + item.previewUrl, + ]), + ), + })); + if (separationResult === null) throw new Error('自动分离没有返回结果'); + const completedResult = separationResult as SeparationDTO; + if (!(await save())) { + throw new Error( + '分离结果已写入编辑器，但 State 保存失败；sidecar 已保留，可继续恢复。', + ); + } + if (backfillErrors.length > 0) { reportWorkflowCompletion( 'visual-binding', - 'success', - `视觉素材绑定完成：已处理 ${batches.length}/${batches.length} 个批次`, - setBindingStatus, + 'failure', + `自动分离已完成，但有 ${backfillErrors.length} 项未能回填；已登记素材并保留恢复状态。\n${backfillErrors.join('\n')}`, + setSeparationStatus, ); - setHasBound(true); + return; + } + await invoke('finalize_separation', { + projectPath, + assetId: resourceId, }); + setHasSeparated(true); + reportWorkflowCompletion( + 'visual-binding', + 'success', + `自动分离完成：${completedResult.bound_nodes.length} 个已绑定，${completedResult.problematic_nodes.length} 个待处理。`, + setSeparationStatus, + ); } catch (cause) { reportWorkflowCompletion( 'visual-binding', 'failure', cause instanceof Error ? cause.message : String(cause), - setBindingStatus, + setSeparationStatus, ); } finally { - setIsBinding(false); + setIsSeparating(false); + } + } + + async function separateUi() { + if ( + isSeparating || + isWorkflowBusy || + separationRecovery !== null || + !resourceId + ) { + return; + } + try { + const recovery = await invoke( + 'inspect_separation_recovery', + { projectPath, assetId: resourceId }, + ); + if (recovery.exists) { + setSeparationRecovery(recovery); + return; + } + await runSeparationWorkflow(); + } catch (cause) { + setSeparationStatus( + cause instanceof Error ? cause.message : String(cause), + ); + } + } + + async function continueSeparation() { + setSeparationRecovery(null); + await runSeparationWorkflow(); + } + + async function restartSeparation() { + if (!resourceId) return; + setSeparationRecovery(null); + try { + await invoke('discard_separation_recovery', { + projectPath, + assetId: resourceId, + }); + await runSeparationWorkflow(); + } catch (cause) { + setSeparationStatus( + cause instanceof Error ? cause.message : String(cause), + ); } } @@ -1244,17 +1430,15 @@ export function useUiEditorSession( selectedNodeId, focusRequest, operations: { - isBinding, isMerging, isRecognizing, isSuggesting, - bindingStatus, mergeStatus, recognitionStatus, suggestionStatus, }, checkPrerequisites, - bindComponents, + separateUi, mergeUi, recognizeUi, suggestUiDesignSemantics, @@ -1367,11 +1551,15 @@ export function useUiEditorSession( hasRecognized, recognitionStatus, recognizeUi, - isBinding, - hasBound, - bindingStatus, + hasSeparated, completionNotice, - bindComponents, + separateUi, + isSeparating, + separationStatus, + separationRecovery, + continueSeparation, + restartSeparation, + cancelSeparationRecovery: () => setSeparationRecovery(null), requestStepChange, continueToNextStep: () => { if (nextStep) requestStepChange(nextStep); From 8cd7c43b9eaecb66cc4b52da7fc1c6fa75fe2a28 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Wed, 9 Sep 2026 21:02:09 +0800 Subject: [PATCH 070/248] =?UTF-8?q?=E6=9B=B4=E6=96=B0=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=8E=A5=E5=85=A5=E6=96=87=E6=A1=A3=E7=8A=B6?= =?UTF-8?q?=E6=80=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将前端接入 TODO 更新为联调与失败注入测试待办。 --- docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index f3620ee74..ee9635f7b 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -88,7 +88,7 @@ - `Vec` 重构为一种组件类型最多一个的容器。 - 当前第一个 Image component 回填规则的正式替代方案。 - 正在执行 batch 的持久化和恢复。 -- 前端登记 SpriteAsset、回填 State、sidecar 恢复弹窗和 `finalize_separation` 的实现与测试。 +- 前端自动分离接入已实现；仍需补齐真实 Tauri/前端联调回归测试与失败注入测试。 - 临时图片清理/归档策略。 - 手动抠图能力。 - problematic 对更高层 workflow 完成门禁的最终定义。 From b3d6543d73e5e9c1249966cc393bbb5ad72134aa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 10:32:34 +0800 Subject: [PATCH 071/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=96=87=E5=AD=97=E9=81=AE=E7=BD=A9=E6=96=B9?= =?UTF-8?q?=E6=A1=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录普通文字归属、遮罩字段与标记图绘制规则 明确不进入分离树、绑定结果和视觉模型契约 --- .../【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index ee9635f7b..5fa60277f 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -27,6 +27,14 @@ - batch 成功后只把结果追加到 bound 容器，失败节点在达到返工上限后追加到 problematic 容器；树拓扑不变，流程继续消费剩余树。 - 不额外维护节点状态枚举；节点是否仍在 pending tree、`rework_count` 和 problematic 容器共同表达状态。 +### 普通文字遮罩 + +- 普通 `Text` 仍是正式 UI tree 中的独立 UI 元素，不进入 separation tree，也不参与 batch、绿色框、visual binding 或 bound 结果。 +- 构造 separation tree 时，把 Text 节点的布局矩形转换为页面像素坐标，挂到最近的未绑定图片节点（`ImageComponent.target_graphic == None`）的 `text_mask_areas`。纯容器只透传；嵌套图片下归最近图片；没有可切图片祖先的 Text 直接忽略。 +- `text_mask_areas` 只保存 `global_pos_x_px`、`global_pos_y_px`、`width_px`、`height_px`。不保存 NodeId、父节点、文字内容、字体样式，也不做 OCR、字形估算、偏差检查、合并或去重。 +- marker 阶段在 image-edit 前把当前 batch 节点自身的文字矩形填充为紫色；它与子图片区域一起绘制，绿色框随后绘制并位于最上层。文字遮罩不递归读取后代节点的 mask。 +- mask 仅是 image-edit 输入标记，未对 image-edit 残留文字增加 OCR 或视觉复核；正式文字语义仍由 `TextComponent` 保持。 + ## 图片编辑与视觉绑定 - image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。处理父节点时，紫色填充其 children 的矩形区域（包括已 problematic 的 children），再在父节点自身外围绘制绿色框和角到角的绿色交叉线；绿色标记覆盖在紫色之上。叶节点只绘制绿色框和角到角的绿色交叉线，不填充自身。 From 761d32cebfeeb043cb9b71c8cf57c85f00bf1e6c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 10:43:00 +0800 Subject: [PATCH 072/248] =?UTF-8?q?=E6=8B=86=E5=88=86=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=A8=A1=E5=9E=8B=E5=B9=B6=E5=A2=9E=E5=8A=A0?= =?UTF-8?q?=E6=96=87=E5=AD=97=E9=81=AE=E7=BD=A9=E5=8C=BA=E5=9F=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 separation model 拆为 note、node、binding、result 模块 为 SeparationNode 增加 text_mask_areas 字段并同步 TypeScript 类型 --- .../src/ui_editor/commands/separation/mod.rs | 6 + .../ui_editor/commands/separation/model.rs | 138 ------------------ .../commands/separation/model/binding.rs | 50 +++++++ .../commands/separation/model/mod.rs | 13 ++ .../commands/separation/model/node.rs | 44 ++++++ .../commands/separation/model/note.rs | 23 +++ .../commands/separation/model/result.rs | 43 ++++++ .../ui-editor/types/SeparationNode.ts | 3 +- .../features/ui-editor/types/TextMaskArea.ts | 3 + 9 files changed, 184 insertions(+), 139 deletions(-) delete mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/result.rs create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index bf51348a7..91dc38009 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -118,6 +118,7 @@ mod tests { description: "image".to_string(), rework_notes: Vec::new(), }, + text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -204,6 +205,7 @@ mod tests { width_px: 1, height_px: 1, note: SeparationNote::default(), + text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -263,6 +265,7 @@ mod tests { width_px: 10, height_px: 10, note: SeparationNote::default(), + text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -273,6 +276,7 @@ mod tests { width_px: 10, height_px: 10, note: SeparationNote::default(), + text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -283,6 +287,7 @@ mod tests { width_px: 5, height_px: 5, note: SeparationNote::default(), + text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -295,6 +300,7 @@ mod tests { width_px: 100, height_px: 100, note: SeparationNote::default(), + text_mask_areas: vec![], children: vec![a, b, c], rework_count: 0, }, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs deleted file mode 100644 index 12fca349c..000000000 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model.rs +++ /dev/null @@ -1,138 +0,0 @@ -use crate::ui_editor::utils::{NodeId, UIDesignImageId}; -use schemars::JsonSchema; -use serde::{Deserialize, Serialize}; -use ts_rs::TS; - -pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; -pub const MAX_REWORK_COUNT: u32 = 3; -pub const MAX_REWORK_NOTE_CHARS: usize = 512; - -#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationNote { - pub description: String, - pub rework_notes: Vec, -} -impl SeparationNote { - pub fn as_prompt(&self) -> String { - let mut prompt = format!("desc: {}", self.description); - if !self.rework_notes.is_empty() { - prompt.push_str("\nprevious rework notes:"); - for note in &self.rework_notes { - prompt.push_str("\n- "); - prompt.push_str(note); - } - } - prompt - } -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationNode { - pub id: NodeId, - pub global_pos_x_px: u32, - pub global_pos_y_px: u32, - pub width_px: u32, - pub height_px: u32, - pub note: SeparationNote, - pub children: Vec, - pub rework_count: u32, -} -impl SeparationNode { - pub fn as_prompt(&self) -> String { - format!( - "node_id={} note: {}", - self.id.as_str(), - self.note.as_prompt() - ) - } -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationTree { - pub src_ui_design: UIDesignImageId, - pub root: SeparationNode, - pub root_extractable: bool, -} -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct BoundNode { - pub node_id: NodeId, - pub cut_image_path: String, -} -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct ProblematicNode { - pub node_id: NodeId, - pub problem_description: String, - pub rework_count: u32, -} -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationState { - pub schema_version: String, - pub trees: Vec, - pub bound: Vec, - pub problematic_nodes: Vec, -} -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationDTO { - pub bound_nodes: Vec, - pub problematic_nodes: Vec, -} - -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] -#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct SeparationRecoveryDTO { - pub exists: bool, - pub bound_node_count: usize, - pub problematic_node_count: usize, - pub has_pending_tree: bool, -} - -#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] -#[schemars(deny_unknown_fields)] -pub struct BindingArea { - pub global_pos_x_px: u32, - pub global_pos_y_px: u32, - pub width_px: u32, - pub height_px: u32, -} -impl BindingArea { - pub fn validate_in(&self, w: u32, h: u32) -> Result<(), String> { - if self.width_px == 0 || self.height_px == 0 { - return Err("BindingArea 宽度和高度必须大于 0".into()); - } - if self - .global_pos_x_px - .checked_add(self.width_px) - .is_none_or(|v| v > w) - || self - .global_pos_y_px - .checked_add(self.height_px) - .is_none_or(|v| v > h) - { - return Err("BindingArea 超出处理图边界".into()); - } - Ok(()) - } -} -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] -#[schemars(deny_unknown_fields)] -pub enum BindingDecision { - Ok { - extracted_area: BindingArea, - to_node: NodeId, - }, - NeedRework { - problem_description: String, - to_node: NodeId, - }, -} -#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] -pub struct BindingResp { - pub decisions: Vec, -} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs new file mode 100644 index 000000000..d552e8b12 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs @@ -0,0 +1,50 @@ +use crate::ui_editor::utils::NodeId; +use schemars::JsonSchema; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub struct BindingArea { + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, +} + +impl BindingArea { + pub fn validate_in(&self, w: u32, h: u32) -> Result<(), String> { + if self.width_px == 0 || self.height_px == 0 { + return Err("BindingArea 宽度和高度必须大于 0".into()); + } + if self + .global_pos_x_px + .checked_add(self.width_px) + .is_none_or(|v| v > w) + || self + .global_pos_y_px + .checked_add(self.height_px) + .is_none_or(|v| v > h) + { + return Err("BindingArea 超出处理图边界".into()); + } + Ok(()) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[schemars(deny_unknown_fields)] +pub enum BindingDecision { + Ok { + extracted_area: BindingArea, + to_node: NodeId, + }, + NeedRework { + problem_description: String, + to_node: NodeId, + }, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +pub struct BindingResp { + pub decisions: Vec, +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs new file mode 100644 index 000000000..7e1032bac --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs @@ -0,0 +1,13 @@ +mod binding; +mod node; +mod note; +mod result; + +pub use binding::*; +pub use node::*; +pub use note::*; +pub use result::*; + +pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; +pub const MAX_REWORK_COUNT: u32 = 3; +pub const MAX_REWORK_NOTE_CHARS: usize = 512; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs new file mode 100644 index 000000000..e0ab3c8fb --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs @@ -0,0 +1,44 @@ +use crate::ui_editor::utils::{NodeId, UIDesignImageId}; +use serde::{Deserialize, Serialize}; +use ts_rs::TS; + +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct TextMaskArea { + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationNode { + pub id: NodeId, + pub global_pos_x_px: u32, + pub global_pos_y_px: u32, + pub width_px: u32, + pub height_px: u32, + pub note: super::SeparationNote, + pub text_mask_areas: Vec, + pub children: Vec, + pub rework_count: u32, +} + +impl SeparationNode { + pub fn as_prompt(&self) -> String { + format!( + "node_id={} note: {}", + self.id.as_str(), + self.note.as_prompt() + ) + } +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationTree { + pub src_ui_design: UIDesignImageId, + pub root: SeparationNode, + pub root_extractable: bool, +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs new file mode 100644 index 000000000..15c500b59 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs @@ -0,0 +1,23 @@ +use serde::{Deserialize, Serialize}; +use ts_rs::TS; + +#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationNote { + pub description: String, + pub rework_notes: Vec, +} + +impl SeparationNote { + pub fn as_prompt(&self) -> String { + let mut prompt = format!("desc: {}", self.description); + if !self.rework_notes.is_empty() { + prompt.push_str("\nprevious rework notes:"); + for note in &self.rework_notes { + prompt.push_str("\n- "); + prompt.push_str(note); + } + } + prompt + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/result.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/result.rs new file mode 100644 index 000000000..82b34bdcd --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/result.rs @@ -0,0 +1,43 @@ +use crate::ui_editor::utils::NodeId; +use serde::{Deserialize, Serialize}; +use ts_rs::TS; + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct BoundNode { + pub node_id: NodeId, + pub cut_image_path: String, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct ProblematicNode { + pub node_id: NodeId, + pub problem_description: String, + pub rework_count: u32, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationState { + pub schema_version: String, + pub trees: Vec, + pub bound: Vec, + pub problematic_nodes: Vec, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationDTO { + pub bound_nodes: Vec, + pub problematic_nodes: Vec, +} + +#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub struct SeparationRecoveryDTO { + pub exists: bool, + pub bound_node_count: usize, + pub problematic_node_count: usize, + pub has_pending_tree: bool, +} diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts index 7aaa94d6f..09a0d8a53 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts @@ -1,5 +1,6 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. import type { NodeId } from "./NodeId"; import type { SeparationNote } from "./SeparationNote"; +import type { TextMaskArea } from "./TextMaskArea"; -export type SeparationNode = { id: NodeId, global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, note: SeparationNote, children: Array, rework_count: number, }; +export type SeparationNode = { id: NodeId, global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, note: SeparationNote, text_mask_areas: Array, children: Array, rework_count: number, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts new file mode 100644 index 000000000..5e450d070 --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts @@ -0,0 +1,3 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type TextMaskArea = { global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, }; From c84df3d88f14df62550b7b994c616ab2fa3acc99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 10:49:07 +0800 Subject: [PATCH 073/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E6=96=87=E5=AD=97?= =?UTF-8?q?=E9=81=AE=E7=BD=A9=E7=8A=B6=E6=80=81=E5=AD=97=E6=AE=B5=E5=A5=91?= =?UTF-8?q?=E7=BA=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 明确 text_mask_areas 为当前 separation state 必选字段 保持开发阶段 v1 且不提供旧 sidecar 迁移回退 --- docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index 5fa60277f..8089b481e 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -54,6 +54,7 @@ - sidecar 目录按 UI manifest `asset_id` 生成，复用 `generated_file_stem(asset_id)` 的安全字符替换和 SHA-256 摘要规则，位于项目 `ui/` 下。 - 目录只保存一份当前 separation state，而不是每 batch 一个状态文件。 - state 文件只保留 `schema_version`、separation trees、bound 结果和 problematic 节点，不重复保存 `projectId / assetId / uiStateRevision`。 +- `SeparationNode.text_mask_areas` 是 separation tree 的必选字段，当前开发阶段继续使用 `ui-editor-separation-state.v1`，不提供旧 sidecar 迁移或回退。 - sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 - 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 - 临时图片可跨重启保留。raw image-edit 返回图、绿色/紫色标记图、处理图和 cut 图片当前都保留用于 debug；理论上只应在内存中，清理/归档策略列 TODO。 From 56f8c9a6a17fd3d116d5e1cad0d9dd100b334292 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 10:49:45 +0800 Subject: [PATCH 074/248] =?UTF-8?q?=E6=9E=84=E9=80=A0=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=96=87=E5=AD=97=E9=81=AE=E7=BD=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按最近未绑定图片祖先收集 Text 布局像素矩形 补充嵌套图片与根图片遮罩构造测试 --- .../src/ui_editor/commands/separation/mod.rs | 64 +++++++++++++++++++ .../src/ui_editor/commands/separation/tree.rs | 52 ++++++++++++++- 2 files changed, 114 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 91dc38009..38a2d5dda 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -16,6 +16,7 @@ mod tests { use super::*; use crate::ui_editor::component::image::{ImageComponent, ImageType}; use crate::ui_editor::component::Component; + use crate::ui_editor::component::text::TextComponent; use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; use crate::ui_editor::layout::control_layout::ControlLayout; use crate::ui_editor::layout::node::Node; @@ -106,6 +107,69 @@ mod tests { assert_eq!(tree.root.id.as_str(), "root-image"); assert!(tree.root_extractable); } + + #[test] + fn construction_attaches_text_mask_to_nearest_unbound_image() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let text = Component::Text(TextComponent::new("按钮")); + let root = node( + "root", + vec![], + vec![node( + "outer-image", + vec![image.clone()], + vec![node("text", vec![text.clone()], vec![])], + )], + ); + let result = construct_separation_state(&state(root)); + let outer = &result.trees[0].root.children[0]; + assert_eq!(outer.id.as_str(), "outer-image"); + assert_eq!(outer.text_mask_areas.len(), 1); + assert!(outer.children.is_empty()); + + let nested_root = node( + "root", + vec![], + vec![node( + "outer-image", + vec![image.clone()], + vec![node( + "inner-image", + vec![image], + vec![node("text", vec![text], vec![])], + )], + )], + ); + let nested = construct_separation_state(&state(nested_root)); + let inner = &nested.trees[0].root.children[0].children[0]; + assert_eq!(inner.text_mask_areas.len(), 1); + assert!(nested.trees[0].root.children[0].text_mask_areas.is_empty()); + } + + #[test] + fn root_image_receives_text_mask() { + let root = node( + "root-image", + vec![Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + })], + vec![node( + "text", + vec![Component::Text(TextComponent::new("标题"))], + vec![], + )], + ); + let result = construct_separation_state(&state(root)); + assert_eq!(result.trees[0].root.text_mask_areas.len(), 1); + } #[test] fn binding_validation_requires_exact_batch_coverage() { let node = SeparationNode { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 42e09f701..56328dfef 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -3,6 +3,7 @@ use crate::ui_editor::component::{image::ImageComponent, Component}; use crate::ui_editor::layout::node::Node; use crate::ui_editor::state::State; use crate::ui_editor::utils::NodeId; +use std::collections::HashMap; use std::collections::HashSet; fn is_unbound_image(node: &Node) -> bool { @@ -17,6 +18,18 @@ fn is_unbound_image(node: &Node) -> bool { }) } +fn has_image_component(node: &Node) -> bool { + node.components + .iter() + .any(|component| matches!(component, Component::Image(_))) +} + +fn has_text_component(node: &Node) -> bool { + node.components + .iter() + .any(|component| matches!(component, Component::Text(_))) +} + fn node_pixel_rect( node: &Node, parent: &crate::ui_editor::layout::dimension::UIRect, @@ -47,11 +60,12 @@ fn collect_todo_nodes( parent: &crate::ui_editor::layout::dimension::UIRect, ppu: f32, output: &mut Vec, + text_masks: &mut HashMap>, ) { let rect = node.layout.transform.resolve(parent); let mut children = Vec::new(); for child in &node.children { - collect_todo_nodes(child, &rect, ppu, &mut children); + collect_todo_nodes(child, &rect, ppu, &mut children, text_masks); } if is_unbound_image(node) { let (x, y, w, h) = node_pixel_rect(node, parent, ppu); @@ -65,6 +79,7 @@ fn collect_todo_nodes( description: node_description(node), rework_notes: Vec::new(), }, + text_mask_areas: text_masks.remove(&node.id).unwrap_or_default(), children, rework_count: 0, }); @@ -73,6 +88,36 @@ fn collect_todo_nodes( } } +fn collect_text_masks( + node: &Node, + parent: &crate::ui_editor::layout::dimension::UIRect, + ppu: f32, + nearest_image: Option, + output: &mut HashMap>, +) { + let node_is_image = is_unbound_image(node); + let nearest_image = if node_is_image { + Some(node.id.clone()) + } else { + nearest_image + }; + if has_text_component(node) && !has_image_component(node) { + if let Some(image_id) = nearest_image.clone() { + let (x, y, w, h) = node_pixel_rect(node, parent, ppu); + output.entry(image_id).or_default().push(TextMaskArea { + global_pos_x_px: x, + global_pos_y_px: y, + width_px: w, + height_px: h, + }); + } + } + let rect = node.layout.transform.resolve(parent); + for child in &node.children { + collect_text_masks(child, &rect, ppu, nearest_image.clone(), output); + } +} + pub fn construct_separation_state(state: &State) -> SeparationState { let trees = state .ui_trees @@ -83,9 +128,11 @@ pub fn construct_separation_state(state: &State) -> SeparationState { let size = image.pixel_size / ppu; let root_rect = crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); + let mut text_masks = HashMap::new(); + collect_text_masks(&tree.root, &root_rect, ppu, None, &mut text_masks); let mut children = Vec::new(); for child in &tree.root.children { - collect_todo_nodes(child, &root_rect, ppu, &mut children); + collect_todo_nodes(child, &root_rect, ppu, &mut children, &mut text_masks); } let root_extractable = is_unbound_image(&tree.root); if !root_extractable && children.is_empty() { @@ -104,6 +151,7 @@ pub fn construct_separation_state(state: &State) -> SeparationState { description: node_description(&tree.root), rework_notes: Vec::new(), }, + text_mask_areas: text_masks.remove(&tree.root.id).unwrap_or_default(), children, rework_count: 0, }, From 22d9958d18a0af19403399fe02656ab8e6f14214 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 10:51:20 +0800 Subject: [PATCH 075/248] =?UTF-8?q?=E5=9C=A8=E5=88=86=E7=A6=BB=E6=A0=87?= =?UTF-8?q?=E8=AE=B0=E5=9B=BE=E4=B8=AD=E9=81=AE=E7=BD=A9=E6=99=AE=E9=80=9A?= =?UTF-8?q?=E6=96=87=E5=AD=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 绘制文字紫色区域并保持绿色元素框在最上层 增加文字遮罩图层顺序测试 --- .../ui_editor/commands/separation/marker.rs | 61 ++++++++++++++++++- 1 file changed, 59 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index 9c8cfd120..b02abafe1 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -1,9 +1,10 @@ -use super::model::SeparationNode; +use super::model::{SeparationNode, SeparationNote, TextMaskArea}; use base64::Engine as _; use std::path::Path; use std::path::PathBuf; const MARKER_LINE_WIDTH: u32 = 2; +const PURPLE_FILL: image::Rgba = image::Rgba([180, 0, 180, 120]); pub async fn build_marked_image( source_url: String, @@ -41,7 +42,19 @@ fn build_marked_image_blocking( child.global_pos_y_px, child.width_px, child.height_px, - image::Rgba([180, 0, 180, 120]), + PURPLE_FILL, + width, + height, + ); + } + for mask in &node.text_mask_areas { + fill_rect( + &mut image, + mask.global_pos_x_px, + mask.global_pos_y_px, + mask.width_px, + mask.height_px, + PURPLE_FILL, width, height, ); @@ -262,4 +275,48 @@ mod tests { } assert_eq!(*image.get_pixel(1, 1), image::Rgba([1, 2, 3, 255])); } + + #[test] + fn text_mask_is_purple_before_green_frame() { + let mut image = image::RgbaImage::from_pixel(8, 8, image::Rgba([1, 2, 3, 255])); + let node = SeparationNode { + id: crate::ui_editor::utils::NodeId::new("image").unwrap(), + global_pos_x_px: 1, + global_pos_y_px: 1, + width_px: 6, + height_px: 6, + note: SeparationNote::default(), + text_mask_areas: vec![TextMaskArea { + global_pos_x_px: 2, + global_pos_y_px: 3, + width_px: 2, + height_px: 2, + }], + children: vec![], + rework_count: 0, + }; + for mask in &node.text_mask_areas { + fill_rect( + &mut image, + mask.global_pos_x_px, + mask.global_pos_y_px, + mask.width_px, + mask.height_px, + PURPLE_FILL, + 8, + 8, + ); + } + draw_frame( + &mut image, + node.global_pos_x_px, + node.global_pos_y_px, + node.width_px, + node.height_px, + 8, + 8, + ); + assert_eq!(*image.get_pixel(2, 4), PURPLE_FILL); + assert_eq!(*image.get_pixel(1, 1), image::Rgba([0, 255, 0, 255])); + } } From 989440173fd488f497a752fb9574ff776293f8de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 11:03:01 +0800 Subject: [PATCH 076/248] =?UTF-8?q?=E5=AE=8C=E5=96=84=E7=BB=91=E5=AE=9A?= =?UTF-8?q?=E5=8C=BA=E5=9F=9F=E5=80=BC=E7=B1=BB=E5=9E=8B=E7=BA=A6=E6=9D=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 BindingArea 增加 Eq 以支持分离区域比较 --- .../ui_editor/commands/separation/marker.rs | 18 ++++++++++++------ .../commands/separation/model/binding.rs | 2 +- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index b02abafe1..a14245995 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -1,4 +1,4 @@ -use super::model::{SeparationNode, SeparationNote, TextMaskArea}; +use super::model::SeparationNode; use base64::Engine as _; use std::path::Path; use std::path::PathBuf; @@ -246,6 +246,7 @@ fn draw_brush( #[cfg(test)] mod tests { use super::*; + use crate::ui_editor::commands::separation::{SeparationNote, TextMaskArea}; #[test] fn draw_frame_adds_green_cross_corner_lines() { @@ -287,10 +288,15 @@ mod tests { height_px: 6, note: SeparationNote::default(), text_mask_areas: vec![TextMaskArea { - global_pos_x_px: 2, - global_pos_y_px: 3, - width_px: 2, - height_px: 2, + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + }, TextMaskArea { + global_pos_x_px: 1, + global_pos_y_px: 1, + width_px: 1, + height_px: 1, }], children: vec![], rework_count: 0, @@ -316,7 +322,7 @@ mod tests { 8, 8, ); - assert_eq!(*image.get_pixel(2, 4), PURPLE_FILL); + assert_eq!(*image.get_pixel(0, 0), PURPLE_FILL); assert_eq!(*image.get_pixel(1, 1), image::Rgba([0, 255, 0, 255])); } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs index d552e8b12..c21b4388d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs @@ -2,7 +2,7 @@ use crate::ui_editor::utils::NodeId; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; -#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize, JsonSchema)] #[schemars(deny_unknown_fields)] pub struct BindingArea { pub global_pos_x_px: u32, From 5d0f3e6be64e7a41c5f745f9bab50fd762b8fb63 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 12:13:04 +0800 Subject: [PATCH 077/248] =?UTF-8?q?UI=E7=BC=96=E8=BE=91=E5=99=A8=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E5=8C=BA=E5=9F=9F=E5=83=8F=E7=B4=A0=E8=BE=B9=E7=95=8C?= =?UTF-8?q?=E5=BD=92=E4=B8=80=E5=8C=96?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增独立 BindingArea 像素边界归一化模块与回归测试 在 cut 裁切前按透明像素收缩或扩展四条边并记录约束结果 接入分离工作流并保留原始与归一化区域日志 --- .../src/ui_editor/commands/separation/area.rs | 401 ++++++++++++++++++ .../src/ui_editor/commands/separation/mod.rs | 2 + .../ui_editor/commands/separation/workflow.rs | 38 +- 3 files changed, 432 insertions(+), 9 deletions(-) create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs new file mode 100644 index 000000000..08a218b35 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -0,0 +1,401 @@ +use super::model::BindingArea; +use image::RgbaImage; + +/// Each edge may move by at most half of the area dimension returned by the +/// visual model. Keep this policy explicit so changing it is an intentional +/// workflow decision rather than a scattered numeric literal. +pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT: u32 = 50; + +#[derive(Clone, Copy, Debug, PartialEq)] +pub(crate) struct NormalizedBindingArea { + pub(crate) area: BindingArea, + pub(crate) changed: bool, + pub(crate) clamped: bool, + pub(crate) transparent: bool, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum EdgeDirection { + Inward, + Outward, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +struct Rect { + left: u32, + top: u32, + right: u32, + bottom: u32, +} + +impl Rect { + fn from_area(area: BindingArea) -> Self { + Self { + left: area.global_pos_x_px, + top: area.global_pos_y_px, + right: area.global_pos_x_px + area.width_px, + bottom: area.global_pos_y_px + area.height_px, + } + } + + fn into_area(self) -> BindingArea { + BindingArea { + global_pos_x_px: self.left, + global_pos_y_px: self.top, + width_px: self.right - self.left, + height_px: self.bottom - self.top, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum Edge { + Left, + Right, + Top, + Bottom, +} + +impl Edge { + const ALL: [Self; 4] = [Self::Left, Self::Right, Self::Top, Self::Bottom]; +} + +fn edge_has_visible_pixel(image: &RgbaImage, rect: Rect, edge: Edge) -> bool { + match edge { + Edge::Left | Edge::Right => { + let x = if edge == Edge::Left { + rect.left + } else { + rect.right - 1 + }; + (rect.top..rect.bottom).any(|y| image.get_pixel(x, y).0[3] > 0) + } + Edge::Top | Edge::Bottom => { + let y = if edge == Edge::Top { + rect.top + } else { + rect.bottom - 1 + }; + (rect.left..rect.right).any(|x| image.get_pixel(x, y).0[3] > 0) + } + } +} + +fn rect_has_visible_pixel(image: &RgbaImage, rect: Rect) -> bool { + (rect.top..rect.bottom).any(|y| (rect.left..rect.right).any(|x| image.get_pixel(x, y).0[3] > 0)) +} + +fn max_edge_adjustment(dimension: u32) -> u32 { + ((u64::from(dimension) * u64::from(MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT)) / 100) + .min(u64::from(u32::MAX)) as u32 +} + +fn edge_direction(image: &RgbaImage, rect: Rect, edge: Edge) -> EdgeDirection { + if edge_has_visible_pixel(image, rect, edge) { + EdgeDirection::Outward + } else { + EdgeDirection::Inward + } +} + +fn move_edge(rect: &mut Rect, edge: Edge, direction: EdgeDirection) { + match (edge, direction) { + (Edge::Left, EdgeDirection::Inward) => rect.left += 1, + (Edge::Left, EdgeDirection::Outward) => rect.left -= 1, + (Edge::Right, EdgeDirection::Inward) => rect.right -= 1, + (Edge::Right, EdgeDirection::Outward) => rect.right += 1, + (Edge::Top, EdgeDirection::Inward) => rect.top += 1, + (Edge::Top, EdgeDirection::Outward) => rect.top -= 1, + (Edge::Bottom, EdgeDirection::Inward) => rect.bottom -= 1, + (Edge::Bottom, EdgeDirection::Outward) => rect.bottom += 1, + } +} + +fn edge_coordinate(rect: Rect, edge: Edge) -> u32 { + match edge { + Edge::Left => rect.left, + Edge::Right => rect.right, + Edge::Top => rect.top, + Edge::Bottom => rect.bottom, + } +} + +fn edge_displacement(original: Rect, current: Rect, edge: Edge) -> u32 { + edge_coordinate(original, edge).abs_diff(edge_coordinate(current, edge)) +} + +fn edge_adjustment_limit(original: Rect, edge: Edge) -> u32 { + max_edge_adjustment(match edge { + Edge::Left | Edge::Right => original.right - original.left, + Edge::Top | Edge::Bottom => original.bottom - original.top, + }) +} + +fn reached_adjustment_limit(original: Rect, current: Rect, edge: Edge) -> bool { + edge_displacement(original, current, edge) >= edge_adjustment_limit(original, edge) +} + +fn can_move_geometrically( + image: &RgbaImage, + current: Rect, + edge: Edge, + direction: EdgeDirection, +) -> bool { + match (edge, direction) { + (Edge::Left, EdgeDirection::Inward) => current.left + 1 < current.right, + (Edge::Left, EdgeDirection::Outward) => current.left > 0, + (Edge::Right, EdgeDirection::Inward) => current.right > current.left + 1, + (Edge::Right, EdgeDirection::Outward) => current.right < image.width(), + (Edge::Top, EdgeDirection::Inward) => current.top + 1 < current.bottom, + (Edge::Top, EdgeDirection::Outward) => current.top > 0, + (Edge::Bottom, EdgeDirection::Inward) => current.bottom > current.top + 1, + (Edge::Bottom, EdgeDirection::Outward) => current.bottom < image.height(), + } +} + +fn next_edge_rect(rect: Rect, edge: Edge, direction: EdgeDirection) -> Option { + let mut next = rect; + match (edge, direction) { + (Edge::Left, EdgeDirection::Inward) if rect.left + 1 < rect.right => next.left += 1, + (Edge::Left, EdgeDirection::Outward) if rect.left > 0 => next.left -= 1, + (Edge::Right, EdgeDirection::Inward) if rect.right > rect.left + 1 => next.right -= 1, + (Edge::Right, EdgeDirection::Outward) => next.right = next.right.checked_add(1)?, + (Edge::Top, EdgeDirection::Inward) if rect.top + 1 < rect.bottom => next.top += 1, + (Edge::Top, EdgeDirection::Outward) if rect.top > 0 => next.top -= 1, + (Edge::Bottom, EdgeDirection::Inward) if rect.bottom > rect.top + 1 => next.bottom -= 1, + (Edge::Bottom, EdgeDirection::Outward) => next.bottom = next.bottom.checked_add(1)?, + _ => return None, + } + Some(next) +} + +fn edge_requires_move(image: &RgbaImage, rect: Rect, edge: Edge, direction: EdgeDirection) -> bool { + match direction { + EdgeDirection::Inward => !edge_has_visible_pixel(image, rect, edge), + EdgeDirection::Outward => { + if !edge_has_visible_pixel(image, rect, edge) { + return false; + } + if !can_move_geometrically(image, rect, edge, direction) { + return true; + } + next_edge_rect(rect, edge, direction) + .is_some_and(|next| edge_has_visible_pixel(image, next, edge)) + } + } +} + +fn apply_edge_step( + image: &RgbaImage, + original: Rect, + current: Rect, + edge: Edge, + direction: EdgeDirection, +) -> (Rect, bool, bool) { + if !edge_requires_move(image, current, edge, direction) { + return (current, false, false); + } + if reached_adjustment_limit(original, current, edge) + || !can_move_geometrically(image, current, edge, direction) + { + return (current, false, true); + } + let mut next = current; + move_edge(&mut next, edge, direction); + (next, true, false) +} + +/// Normalizes a model-provided area using visible pixels on the processed +/// transparent image. Each edge chooses inward/outward direction once from +/// its initial scan and then moves monotonically, so sparse pixels cannot make +/// the boundary oscillate. The four edge steps are calculated from the same +/// rectangle on each round. +pub(crate) fn normalize_binding_area( + image: &RgbaImage, + original_area: BindingArea, +) -> Result { + original_area.validate_in(image.width(), image.height())?; + let original = Rect::from_area(original_area); + let directions = Edge::ALL.map(|edge| edge_direction(image, original, edge)); + let mut current = original; + let mut clamped = false; + let mut active = [true; 4]; + + // TODO: Replace the deliberately simple pixel-by-pixel scan if real UI + // design sizes show this path to be a measurable bottleneck. + while active.iter().any(|value| *value) { + let before = current; + let mut next = current; + let mut moved = [false; 4]; + for (index, edge) in Edge::ALL.into_iter().enumerate() { + if !active[index] { + continue; + } + let (candidate, did_move, reached_limit) = + apply_edge_step(image, original, current, edge, directions[index]); + if reached_limit { + clamped = true; + active[index] = false; + } else if !did_move { + active[index] = false; + } + moved[index] = did_move; + match edge { + Edge::Left => next.left = candidate.left, + Edge::Right => next.right = candidate.right, + Edge::Top => next.top = candidate.top, + Edge::Bottom => next.bottom = candidate.bottom, + } + } + if next.left >= next.right { + clamped = true; + if moved[0] { + active[0] = false; + } + if moved[1] { + active[1] = false; + } + next.left = current.left; + next.right = current.right; + } + if next.top >= next.bottom { + clamped = true; + if moved[2] { + active[2] = false; + } + if moved[3] { + active[3] = false; + } + next.top = current.top; + next.bottom = current.bottom; + } + current = next; + if current == before { + break; + } + } + + let area = current.into_area(); + Ok(NormalizedBindingArea { + changed: area != original_area, + area, + clamped, + transparent: !rect_has_visible_pixel(image, current), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use image::{Rgba, RgbaImage}; + + fn image_with_rect( + width: u32, + height: u32, + left: u32, + top: u32, + right: u32, + bottom: u32, + ) -> RgbaImage { + let mut image = RgbaImage::from_pixel(width, height, Rgba([0, 0, 0, 0])); + for y in top..bottom { + for x in left..right { + image.put_pixel(x, y, Rgba([255, 255, 255, 255])); + } + } + image + } + + fn area(x: u32, y: u32, width: u32, height: u32) -> BindingArea { + BindingArea { + global_pos_x_px: x, + global_pos_y_px: y, + width_px: width, + height_px: height, + } + } + + #[test] + fn shrinks_empty_edges_to_visible_bounds() { + let image = image_with_rect(32, 32, 10, 11, 16, 18); + let result = normalize_binding_area(&image, area(6, 7, 14, 16)).unwrap(); + assert_eq!(result.area, area(10, 11, 6, 7)); + assert!(result.changed); + assert!(!result.clamped); + assert!(!result.transparent); + } + + #[test] + fn expands_visible_edges_to_cover_the_element() { + let image = image_with_rect(32, 32, 10, 11, 16, 18); + let result = normalize_binding_area(&image, area(11, 12, 4, 5)).unwrap(); + assert_eq!(result.area, area(10, 11, 6, 7)); + assert!(result.changed); + assert!(!result.clamped); + assert!(!result.transparent); + } + + #[test] + fn adjusts_each_edge_independently() { + let image = image_with_rect(32, 32, 10, 11, 16, 18); + let result = normalize_binding_area(&image, area(10, 12, 10, 3)).unwrap(); + assert_eq!(result.area, area(10, 11, 6, 5)); + } + + #[test] + fn keeps_nonzero_alpha_antialias_pixels() { + let mut image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0])); + image.put_pixel(5, 6, Rgba([255, 255, 255, 1])); + image.put_pixel(7, 8, Rgba([255, 255, 255, 255])); + let result = normalize_binding_area(&image, area(4, 5, 5, 5)).unwrap(); + assert_eq!(result.area, area(5, 6, 3, 3)); + } + + #[test] + fn fully_transparent_image_uses_the_same_path() { + let image = RgbaImage::from_pixel(32, 32, Rgba([0, 0, 0, 0])); + let result = normalize_binding_area(&image, area(10, 10, 10, 10)).unwrap(); + assert_eq!(result.area, area(14, 14, 2, 2)); + assert!(result.changed); + assert!(result.transparent); + } + + #[test] + fn caps_each_edge_at_half_of_the_original_dimension() { + let image = image_with_rect(64, 64, 0, 0, 64, 64); + let result = normalize_binding_area(&image, area(16, 16, 8, 8)).unwrap(); + assert_eq!(result.area, area(12, 12, 16, 16)); + assert!(result.clamped); + } + + #[test] + fn clamps_expansion_to_image_edges() { + let image = image_with_rect(16, 16, 0, 0, 4, 4); + let result = normalize_binding_area(&image, area(1, 1, 2, 2)).unwrap(); + assert_eq!(result.area, area(0, 0, 4, 4)); + assert!(result.clamped); + } + + #[test] + fn exact_split_at_adjustment_limit_is_not_clamped() { + let image = image_with_rect(16, 16, 4, 4, 8, 8); + let result = normalize_binding_area(&image, area(5, 5, 2, 2)).unwrap(); + assert_eq!(result.area, area(4, 4, 4, 4)); + assert!(!result.clamped); + } + + #[test] + fn one_pixel_area_stays_nonzero_when_adjustment_limit_is_zero() { + let image = image_with_rect(8, 8, 2, 2, 5, 5); + let result = normalize_binding_area(&image, area(3, 3, 1, 1)).unwrap(); + assert_eq!(result.area, area(3, 3, 1, 1)); + assert!(result.clamped); + } + + #[test] + fn rejects_zero_sized_or_out_of_bounds_model_areas() { + let image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0])); + assert!(normalize_binding_area(&image, area(0, 0, 0, 1)).is_err()); + assert!(normalize_binding_area(&image, area(15, 15, 2, 2)).is_err()); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 38a2d5dda..c2bcc3ce7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -1,3 +1,4 @@ +mod area; mod marker; mod model; mod persistence; @@ -5,6 +6,7 @@ mod prompt; mod tree; mod workflow; +pub(crate) use area::{normalize_binding_area, NormalizedBindingArea}; pub(crate) use marker::build_marked_image; pub use model::*; pub use persistence::*; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index da3a1288e..16f6117e4 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -618,14 +618,34 @@ fn cut_processed_image_blocking( area: &BindingArea, target: &Path, ) -> Result<(), String> { - let image = image::open(source).map_err(|e| format!("读取处理图失败：{e}"))?; - area.validate_in(image.width(), image.height())?; - let cropped = image.crop_imm( - area.global_pos_x_px, - area.global_pos_y_px, - area.width_px, - area.height_px, + let image = image::open(source) + .map_err(|e| format!("读取处理图失败：{e}"))? + .to_rgba8(); + let normalized = normalize_binding_area(&image, *area)?; + let original_area = *area; + let normalized_area = normalized.area; + app_log!( + "ui_separation.cut_image.normalized changed={} clamped={} transparent={} original_area=({}, {}, {}, {}) normalized_area=({}, {}, {}, {})", + normalized.changed, + normalized.clamped, + normalized.transparent, + original_area.global_pos_x_px, + original_area.global_pos_y_px, + original_area.width_px, + original_area.height_px, + normalized_area.global_pos_x_px, + normalized_area.global_pos_y_px, + normalized_area.width_px, + normalized_area.height_px ); + let cropped = image::imageops::crop_imm( + &image, + normalized_area.global_pos_x_px, + normalized_area.global_pos_y_px, + normalized_area.width_px, + normalized_area.height_px, + ) + .to_image(); cropped .save_with_format(target, ImageFormat::Png) .map_err(|e| format!("写入 cut 图片失败：{e}"))?; @@ -635,8 +655,8 @@ fn cut_processed_image_blocking( .file_name() .and_then(|name| name.to_str()) .unwrap_or(""), - area.width_px, - area.height_px + normalized_area.width_px, + normalized_area.height_px ); Ok(()) } From 3733e86272f604fc283ac7f4b7956293445a4de5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 12:13:21 +0800 Subject: [PATCH 078/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E5=8C=BA=E5=9F=9F=E5=83=8F=E7=B4=A0=E5=BD=92=E4=B8=80=E5=8C=96?= =?UTF-8?q?=E7=BA=A6=E5=AE=9A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录透明像素边界扫描、方向锁定与四边同步推进规则 明确 50% 与图像边界约束、透明图处理、日志字段和性能 TODO --- docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md index 8089b481e..34ca8322b 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md @@ -43,6 +43,7 @@ `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 +- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的 `50%` 位移上限。该上限由模块级常量定义。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到 `50%` 上限、图像边界或非零尺寸约束。性能优化列 TODO。 - `NeedRework` 携带短问题描述（最多 512 个 Unicode 字符）；通过校验后按产生顺序追加到目标 `SeparationNode.note.rework_notes`，下一次该节点进入 image-edit 时全部意见会注入提取 prompt。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 - 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 From 086aa91d7020157c6be8a286f3a3a5271e624372 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 12:24:18 +0800 Subject: [PATCH 079/248] =?UTF-8?q?=E6=94=B6=E7=AA=84=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E5=8C=BA=E5=9F=9F=E6=A8=A1=E5=9D=97=E5=AF=BC=E5=87=BA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 让裁切工作流直接依赖面积归一化模块 移除未使用的公共重导出，保持分离模块边界清晰 --- .../src/ui_editor/commands/separation/mod.rs | 43 +++++++++---------- .../ui_editor/commands/separation/workflow.rs | 1 + 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index c2bcc3ce7..3050f6a47 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -6,7 +6,6 @@ mod prompt; mod tree; mod workflow; -pub(crate) use area::{normalize_binding_area, NormalizedBindingArea}; pub(crate) use marker::build_marked_image; pub use model::*; pub use persistence::*; @@ -31,7 +30,7 @@ mod tests { use std::path::Path; use typed_floats::tf32::StrictlyPositiveFinite; - fn node(id: &str, components: Vec, children: Vec) -> Node { + fn node(id: &str, component: Option, children: Vec) -> Node { Node { id: NodeId::new(id).unwrap(), layout: ControlLayout::default(), @@ -39,12 +38,12 @@ mod tests { name: id.to_string(), description: String::new(), layout_status: StageStatus::NoProblem, - components_status: StageStatus::NoProblem, + component_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::Llm, }, - components, + component, children_display_mode: ChildrenDisplayMode::Stack, children, } @@ -84,11 +83,11 @@ mod tests { }); let root = node( "root", - vec![], + None, vec![node( "container", - vec![], - vec![node("image", vec![image], vec![])], + None, + vec![node("image", Some(image), vec![])], )], ); let result = construct_separation_state(&state(root)); @@ -103,7 +102,7 @@ mod tests { preserve_aspect: false, }, }); - let root = node("root-image", vec![image], vec![]); + let root = node("root-image", Some(image), vec![]); let result = construct_separation_state(&state(root)); let tree = &result.trees[0]; assert_eq!(tree.root.id.as_str(), "root-image"); @@ -121,11 +120,11 @@ mod tests { let text = Component::Text(TextComponent::new("按钮")); let root = node( "root", - vec![], + None, vec![node( "outer-image", - vec![image.clone()], - vec![node("text", vec![text.clone()], vec![])], + Some(image.clone()), + vec![node("text", Some(text.clone()), vec![])], )], ); let result = construct_separation_state(&state(root)); @@ -136,14 +135,14 @@ mod tests { let nested_root = node( "root", - vec![], + None, vec![node( "outer-image", - vec![image.clone()], + Some(image.clone()), vec![node( "inner-image", - vec![image], - vec![node("text", vec![text], vec![])], + Some(image), + vec![node("text", Some(text), vec![])], )], )], ); @@ -157,15 +156,15 @@ mod tests { fn root_image_receives_text_mask() { let root = node( "root-image", - vec![Component::Image(ImageComponent { + Some(Component::Image(ImageComponent { target_graphic: None, image_type: ImageType::Simple { preserve_aspect: false, }, - })], + })), vec![node( "text", - vec![Component::Text(TextComponent::new("标题"))], + Some(Component::Text(TextComponent::new("标题"))), vec![], )], ); @@ -219,8 +218,8 @@ mod tests { }); let mut separation = construct_separation_state(&state(node( "root", - vec![], - vec![node("image", vec![image], vec![])], + None, + vec![node("image", Some(image), vec![])], ))); let id = NodeId::new("image").unwrap(); let paths = HashMap::new(); @@ -303,8 +302,8 @@ mod tests { }); let mut state = construct_separation_state(&state(node( "root", - vec![], - vec![node("image", vec![image], vec![])], + None, + vec![node("image", Some(image), vec![])], ))); let id = NodeId::new("image").unwrap(); let decisions = vec![BindingDecision::Ok { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 16f6117e4..8d45c6a09 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -1,3 +1,4 @@ +use super::area::normalize_binding_area; use super::model::*; use super::prompt::{gen_binding_prompt, gen_extract_prompt}; use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; From d6c720ecdd67855067871de328dc3df31b0c4d43 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 12:26:39 +0800 Subject: [PATCH 080/248] =?UTF-8?q?=E7=BA=A0=E6=AD=A3=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A8=A1=E5=9D=97=E5=AF=BC=E5=87=BA=E6=8F=90=E4=BA=A4=E8=BE=B9?= =?UTF-8?q?=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 撤销误入的组件字段重命名测试改动 仅保留裁切工作流对面积归一化模块的直接依赖 --- .../src/ui_editor/commands/separation/mod.rs | 42 +++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 3050f6a47..a68f7ed13 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -30,7 +30,7 @@ mod tests { use std::path::Path; use typed_floats::tf32::StrictlyPositiveFinite; - fn node(id: &str, component: Option, children: Vec) -> Node { + fn node(id: &str, components: Vec, children: Vec) -> Node { Node { id: NodeId::new(id).unwrap(), layout: ControlLayout::default(), @@ -38,12 +38,12 @@ mod tests { name: id.to_string(), description: String::new(), layout_status: StageStatus::NoProblem, - component_status: StageStatus::NoProblem, + components_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::Llm, }, - component, + components, children_display_mode: ChildrenDisplayMode::Stack, children, } @@ -83,11 +83,11 @@ mod tests { }); let root = node( "root", - None, + vec![], vec![node( "container", - None, - vec![node("image", Some(image), vec![])], + vec![], + vec![node("image", vec![image], vec![])], )], ); let result = construct_separation_state(&state(root)); @@ -102,7 +102,7 @@ mod tests { preserve_aspect: false, }, }); - let root = node("root-image", Some(image), vec![]); + let root = node("root-image", vec![image], vec![]); let result = construct_separation_state(&state(root)); let tree = &result.trees[0]; assert_eq!(tree.root.id.as_str(), "root-image"); @@ -120,11 +120,11 @@ mod tests { let text = Component::Text(TextComponent::new("按钮")); let root = node( "root", - None, + vec![], vec![node( "outer-image", - Some(image.clone()), - vec![node("text", Some(text.clone()), vec![])], + vec![image.clone()], + vec![node("text", vec![text.clone()], vec![])], )], ); let result = construct_separation_state(&state(root)); @@ -135,14 +135,14 @@ mod tests { let nested_root = node( "root", - None, + vec![], vec![node( "outer-image", - Some(image.clone()), + vec![image.clone()], vec![node( "inner-image", - Some(image), - vec![node("text", Some(text), vec![])], + vec![image], + vec![node("text", vec![text], vec![])], )], )], ); @@ -156,15 +156,15 @@ mod tests { fn root_image_receives_text_mask() { let root = node( "root-image", - Some(Component::Image(ImageComponent { + vec![Component::Image(ImageComponent { target_graphic: None, image_type: ImageType::Simple { preserve_aspect: false, }, - })), + })], vec![node( "text", - Some(Component::Text(TextComponent::new("标题"))), + vec![Component::Text(TextComponent::new("标题"))], vec![], )], ); @@ -218,8 +218,8 @@ mod tests { }); let mut separation = construct_separation_state(&state(node( "root", - None, - vec![node("image", Some(image), vec![])], + vec![], + vec![node("image", vec![image], vec![])], ))); let id = NodeId::new("image").unwrap(); let paths = HashMap::new(); @@ -302,8 +302,8 @@ mod tests { }); let mut state = construct_separation_state(&state(node( "root", - None, - vec![node("image", Some(image), vec![])], + vec![], + vec![node("image", vec![image], vec![])], ))); let id = NodeId::new("image").unwrap(); let decisions = vec![BindingDecision::Ok { From 51f629b2f9d3248014e0dc932eaede592e68f14a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 13:08:29 +0800 Subject: [PATCH 081/248] =?UTF-8?q?UI=E7=BC=96=E8=BE=91=E5=99=A8=E8=8A=82?= =?UTF-8?q?=E7=82=B9=E6=94=B9=E4=B8=BA=E5=8D=95=E7=BB=84=E4=BB=B6=E6=A8=A1?= =?UTF-8?q?=E5=9E=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 修改 Node、Binding DTO、识别与分离链路为 component 单字段 为识别和绑定 LLM 工具引入 PureNode/WithComponent 显式载荷 移除前端组件索引、排序和组件栈操作 同步生成 ts-rs TypeScript 类型、状态校验、测试与专题文档 --- .../src/ui_editor/commands/binding.rs | 123 ++++--- .../src-tauri/src/ui_editor/commands/merge.rs | 10 +- .../src/ui_editor/commands/recognition.rs | 59 ++-- .../src/ui_editor/commands/separation/mod.rs | 44 +-- .../src/ui_editor/commands/separation/tree.rs | 24 +- .../src-tauri/src/ui_editor/component/mod.rs | 24 ++ .../src/ui_editor/html_renderer/mod.rs | 13 +- .../src-tauri/src/ui_editor/layout/node.rs | 4 +- .../src-tauri/src/ui_editor/persistence.rs | 71 +++- .../src-tauri/src/ui_editor/workflow.rs | 10 +- .../src/features/ui-editor/binding.ts | 6 +- .../src/features/ui-editor/bindingOverview.ts | 15 +- .../src/features/ui-editor/requisites.ts | 7 +- .../features/ui-editor/stageStatusOverview.ts | 2 +- .../features/ui-editor/types/BindingChange.ts | 4 +- .../src/features/ui-editor/types/Node.ts | 2 +- .../features/ui-editor/types/NodeComponent.ts | 12 + .../features/ui-editor/types/NodeMetadata.ts | 2 +- .../src/features/ui-editor/types/UIRect.ts | 9 + .../features/ui-editor/useUiEditorState.ts | 167 ++------- .../ui-editor/components/ImportOverview.tsx | 2 +- .../ui-editor/components/InputSidebar.tsx | 4 +- .../Inspector/Components/ComponentPanel.tsx | 327 +++++------------- .../Components/componentEditorTypes.ts | 15 +- .../components/Inspector/InspectorSidebar.tsx | 34 +- .../view/ui-editor/components/UiTreePanel.tsx | 6 +- .../components/preview/UiTreeRenderer.tsx | 10 +- .../src/view/ui-editor/index.tsx | 2 +- .../src/view/ui-editor/useUiEditorPage.ts | 61 +--- .../tests/bindingOverview.test.ts | 27 +- .../tests/nodeTransformGeometry.test.ts | 2 +- .../tests/previewWorkspaceZoom.test.tsx | 4 +- .../tests/stageStatusOverview.test.ts | 4 +- .../tests/uiEditorPage.test.ts | 8 +- .../tests/uiEditorPreview.test.tsx | 4 +- .../tests/uiEditorState.test.ts | 68 ++-- .../tests/uiTreeUtils.test.ts | 2 +- .../useNodeTransformInteraction.test.tsx | 4 +- ...术方案】UI编辑器自动分离工作流-2026-09-08.md | 24 +- 39 files changed, 488 insertions(+), 728 deletions(-) create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/NodeComponent.ts create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/UIRect.ts diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs index 4585eee85..17979cbca 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs @@ -5,11 +5,9 @@ use crate::ui_editor::commands::utils::{ strict_json_schema, }; use crate::ui_editor::component::text::FontSource; -use crate::ui_editor::component::Component; +use crate::ui_editor::component::{Component, NodeComponent}; use crate::ui_editor::layout::node::{Node, StageStatus}; -use crate::ui_editor::persistence::{ - UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE, UI_DESIGN_STATE_MAX_NODES, -}; +use crate::ui_editor::persistence::UI_DESIGN_STATE_MAX_NODES; use crate::ui_editor::state::State; use crate::ui_editor::utils::{FontAssetId, NodeId, SpriteAssetId}; use platform_llm::{ @@ -31,10 +29,10 @@ const SYSTEM_PROMPT: &str = r#" 你是游戏 UI 组件绑定器。你会看到全部 UI 参考图、可编辑节点说明，以及本批独立素材的真实像素。 * 只对视觉上确实需要改变组件的节点返回 changes； -* 每个 change 的 components 是该节点完整的新渲染栈，空数组表示明确清空。数组顺序从底到顶渲染。 -* 对每个 Component，直接完整返回其全部参数. +* 每个 change 的 component 是该节点完整的新组件；纯结构节点返回 "PureNode"，有组件返回 {"WithComponent": <完整 Component>}。 +* 对 Component，直接完整返回其全部参数. * 有任何困难或者不确定把状态设为 NeedReview，说明中文原因。 -* 纯结构节点可以返回空数组并标为 NoProblem。 +* 纯结构节点可以返回 "PureNode" 并标为 NoProblem。 * 容器背景等推荐使用Simple + preserve_aspect: false 实现与node大小一致 * 面向用户的 reason 使用中文。 @@ -53,8 +51,8 @@ enum DraftStatus { #[schemars(deny_unknown_fields)] struct BindingChangeDraft { node_id: NodeId, - components: Vec, - components_status: DraftStatus, + component: NodeComponent, + component_status: DraftStatus, } #[derive(Clone, Debug, Deserialize, JsonSchema)] @@ -68,8 +66,8 @@ struct BindingResponse { #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] pub struct BindingChange { pub node_id: NodeId, - pub components: Vec, - pub components_status: StageStatus, + pub component: NodeComponent, + pub component_status: StageStatus, } #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] @@ -83,7 +81,7 @@ struct EditableNodeContext<'a> { node_id: &'a NodeId, name: &'a str, description: &'a str, - components: &'a [Component], + component: Option<&'a Component>, } #[derive(Debug, Serialize)] @@ -106,7 +104,7 @@ fn collect_editable_nodes<'a>(node: &'a Node, output: &mut Vec UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE { - return Err(format!( - "单个组件绑定栈不能超过 {UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE} 个组件" - )); + if !change + .as_object() + .is_some_and(|object| object.contains_key("component")) + { + return Err("组件绑定 change 缺少 component 字段".to_string()); } } Ok(()) @@ -212,7 +207,7 @@ fn validate_and_materialize( if !changed_ids.insert(change.node_id.clone()) { return Err(format!("组件绑定重复返回节点：{}", change.node_id.as_str())); } - for component in &change.components { + if let NodeComponent::WithComponent(component) = &change.component { match component { Component::Image(image) => { if image @@ -232,18 +227,22 @@ fn validate_and_materialize( } } } - let components = change.components; - let components_status = match change.components_status { + let component_status = match change.component_status { DraftStatus::NoProblem => StageStatus::NoProblem, DraftStatus::NeedReview(reason) if reason.trim().is_empty() => { return Err("组件待审状态必须包含原因".to_string()) } - DraftStatus::NeedReview(reason) => StageStatus::NeedReview(reason), + DraftStatus::NeedReview(reason) => { + if matches!(&change.component, NodeComponent::PureNode) { + return Err("纯结构节点不能标记为组件待审".to_string()); + } + StageStatus::NeedReview(reason) + } }; materialized.push(BindingChange { node_id: change.node_id, - components, - components_status, + component: change.component, + component_status, }); } Ok(BindingDTO { @@ -440,8 +439,8 @@ mod tests { ]); let unapproved = BindingChangeDraft { node_id: id("other"), - components: Vec::new(), - components_status: DraftStatus::NoProblem, + component: NodeComponent::PureNode, + component_status: DraftStatus::NoProblem, }; assert!( validate_and_materialize(vec![unapproved], &editable, &known, &HashSet::new()).is_err() @@ -450,7 +449,7 @@ mod tests { // References to sprites from another batch are allowed once they exist in the project. let other_batch = BindingChangeDraft { node_id: id("editable"), - components: vec![Component::Image( + component: NodeComponent::WithComponent(Component::Image( crate::ui_editor::component::image::ImageComponent { target_graphic: Some( SpriteAssetId::new("other-batch-sprite").expect("valid sprite"), @@ -459,8 +458,8 @@ mod tests { preserve_aspect: false, }, }, - )], - components_status: DraftStatus::NoProblem, + )), + component_status: DraftStatus::NoProblem, }; assert!( validate_and_materialize(vec![other_batch], &editable, &known, &HashSet::new()).is_ok() @@ -469,15 +468,15 @@ mod tests { // References to sprites that do not exist in the project at all are still rejected. let unknown = BindingChangeDraft { node_id: id("editable"), - components: vec![Component::Image( + component: NodeComponent::WithComponent(Component::Image( crate::ui_editor::component::image::ImageComponent { target_graphic: Some(SpriteAssetId::new("unknown").expect("valid sprite")), image_type: crate::ui_editor::component::image::ImageType::Simple { preserve_aspect: false, }, }, - )], - components_status: DraftStatus::NoProblem, + )), + component_status: DraftStatus::NoProblem, }; assert!( validate_and_materialize(vec![unknown], &editable, &known, &HashSet::new()).is_err() @@ -491,8 +490,8 @@ mod tests { text.font = FontSource::Bound(FontAssetId::new("unknown-font").expect("valid font")); let change = BindingChangeDraft { node_id: id("editable"), - components: vec![Component::Text(text)], - components_status: DraftStatus::NoProblem, + component: NodeComponent::WithComponent(Component::Text(text)), + component_status: DraftStatus::NoProblem, }; let error = validate_and_materialize( @@ -506,13 +505,13 @@ mod tests { } #[test] - fn materialization_preserves_changed_only_empty_component_lists() { + fn materialization_preserves_pure_node_change() { let editable = HashSet::from([id("editable")]); let result = validate_and_materialize( vec![BindingChangeDraft { node_id: id("editable"), - components: Vec::new(), - components_status: DraftStatus::NoProblem, + component: NodeComponent::PureNode, + component_status: DraftStatus::NoProblem, }], &editable, &HashSet::new(), @@ -520,8 +519,28 @@ mod tests { ) .expect("valid changed-only clear"); assert_eq!(result.changes.len(), 1); - assert!(result.changes[0].components.is_empty()); - assert_eq!(result.changes[0].components_status, StageStatus::NoProblem); + assert!(matches!( + result.changes[0].component, + NodeComponent::PureNode + )); + assert_eq!(result.changes[0].component_status, StageStatus::NoProblem); + } + + #[test] + fn materialization_rejects_problematic_pure_node() { + let editable = HashSet::from([id("editable")]); + let error = validate_and_materialize( + vec![BindingChangeDraft { + node_id: id("editable"), + component: NodeComponent::PureNode, + component_status: DraftStatus::NeedReview("缺少可确认的组件".to_string()), + }], + &editable, + &HashSet::new(), + &HashSet::new(), + ) + .expect_err("pure node cannot carry a component review status"); + assert!(error.contains("纯结构节点")); } #[test] @@ -575,20 +594,26 @@ mod tests { } #[test] - fn binding_response_bounds_changes_and_each_component_stack() { + fn binding_response_bounds_changes_and_uses_single_component_shape() { let too_many_changes = serde_json::json!({ - "changes": [{"components": []}, {"components": []}] + "changes": [{"component": "PureNode"}, {"component": "PureNode"}] }); assert!(validate_binding_response_shape(&too_many_changes, 1).is_err()); - let too_many_components = serde_json::json!({ + let one_component = serde_json::json!({ "changes": [{ - "components": (0..=UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE) - .map(|_| serde_json::Value::Null) - .collect::>() + "node_id": "editable", + "component": "PureNode", + "component_status": "NoProblem" }] }); - assert!(validate_binding_response_shape(&too_many_components, 1).is_err()); + assert!(validate_binding_response_shape(&one_component, 1).is_ok()); + let parsed = parse_binding_response(&one_component.to_string(), 1) + .expect("explicit PureNode payload should parse"); + assert!(matches!( + parsed.changes[0].component, + NodeComponent::PureNode + )); } #[tokio::test] diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/merge.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/merge.rs index e8a789948..9fe0ad71f 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/merge.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/merge.rs @@ -295,12 +295,12 @@ mod materialize { name: container_name, description: container_description, layout_status: StageStatus::NoProblem, - components_status: StageStatus::NoProblem, + component_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::Llm, }, - components: Vec::new(), + component: None, children_display_mode: ChildrenDisplayMode::Exclusive, children: members.into_iter().map(|member| member.node).collect(), }, @@ -542,12 +542,12 @@ mod tests { name: id.to_string(), description: String::new(), layout_status: StageStatus::NoProblem, - components_status: StageStatus::NoProblem, + component_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::Human, }, - components: Vec::::new(), + component: None, children_display_mode: ChildrenDisplayMode::Stack, children, } @@ -586,7 +586,7 @@ mod tests { ChildrenDisplayMode::Exclusive ); assert_eq!( - result.root.metadata.components_status, + result.root.metadata.component_status, StageStatus::NoProblem ); assert_eq!(result.root.children.len(), 2); diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index ccfb53ac3..49750b04e 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -4,7 +4,7 @@ use crate::ui_editor::commands::utils::{ parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, strict_json_schema, }; -use crate::ui_editor::component::Component; +use crate::ui_editor::component::{Component, NodeComponent}; use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; use crate::ui_editor::layout::control_layout::ControlLayout; use crate::ui_editor::layout::dimension::UIRect; @@ -44,11 +44,11 @@ const SYSTEM_PROMPT: &str = r#" * 由于每个截图未必是完整的, 可能是局部的, 每棵树描述清楚每个截图上UI的层次结构即可 * 不同树的共用框架/层次/...请使用使用相同的名称描述. 不同状态/变体名称使用相同的前缀, 用后缀区别 * 粒度要求: 尽可能细致, 以可交互,方便程序化控制的最小单位为准. 包括不限于: icon, 进度条的底槽、填充和外框; slider的底槽, dragger等. -* 为每个节点直接返回完整 components. - 无背景的逻辑容器返回空数组. + * 为每个节点直接返回 component. + 无背景的逻辑容器返回 "PureNode"，不要返回 null. 有背景的容器推荐使用Simple+不锁定宽高比的Image component. 目前我们只做识别, 不要求图片字体参数. - 每个节点当前最多返回一个 Image component 和一个 Text component。 + 每个节点最多返回一个 component；需要多个视觉层时拆成多个节点。 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. "#; @@ -113,7 +113,7 @@ struct RecognitionNode { description: String, children: Vec, confidence: Confidence, - components: Vec, + component: NodeComponent, } #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] @@ -312,12 +312,12 @@ fn convert_node( name: source.name.clone(), description: source.description.clone(), layout_status: status, - components_status: StageStatus::NoProblem, + component_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::Llm, }, - components: source.components.clone(), + component: source.component.clone().into_option(), children_display_mode: ChildrenDisplayMode::Stack, children, }) @@ -325,40 +325,25 @@ fn convert_node( fn validate_confidence(nodes: &[RecognitionNode]) -> Result<(), String> { for node in nodes { - let image_count = node - .components - .iter() - .filter(|component| matches!(component, Component::Image(_))) - .count(); - let text_count = node - .components - .iter() - .filter(|component| matches!(component, Component::Text(_))) - .count(); - if image_count > 1 || text_count > 1 { - return Err("单个节点当前最多包含一个 Image 和一个 Text component".to_string()); - } - if node.components.iter().any(|component| { - matches!( + if let NodeComponent::WithComponent(component) = &node.component { + if matches!( component, Component::Image(crate::ui_editor::component::image::ImageComponent { target_graphic: Some(_), .. }) - ) - }) { - return Err("识别阶段不能返回已绑定的 SpriteAssetId".to_string()); - } - if node.components.iter().any(|component| { - matches!( + ) { + return Err("识别阶段不能返回已绑定的 SpriteAssetId".to_string()); + } + if matches!( component, Component::Text(crate::ui_editor::component::text::TextComponent { font: crate::ui_editor::component::text::FontSource::Bound(_), .. }) - ) - }) { - return Err("识别阶段不能返回已绑定的字体素材".to_string()); + ) { + return Err("识别阶段不能返回已绑定的字体素材".to_string()); + } } if let Confidence::UnSure(reason) = &node.confidence { if reason.trim().is_empty() { @@ -424,7 +409,7 @@ mod tests { description: String::new(), children: Vec::new(), confidence: Confidence::Confident, - components: Vec::new(), + component: NodeComponent::PureNode, } } @@ -528,7 +513,7 @@ mod tests { description: String::new(), children: Vec::new(), confidence: Confidence::UnSure(String::new()), - components: Vec::new(), + component: NodeComponent::PureNode, }; assert!(validate_confidence(&[node]).is_err()); } @@ -552,7 +537,7 @@ mod tests { description: String::new(), children: Vec::new(), confidence: Confidence::Confident, - components: vec![Component::Text(text)], + component: NodeComponent::WithComponent(Component::Text(text)), }; assert!(validate_confidence(&[node]).is_err()); } @@ -569,7 +554,7 @@ mod tests { converted.layout.transform.resolve(&root_rect), UIRect::new(Point2::new(50.0, 25.0), Vector2::new(100.0, 50.0)), ); - assert_eq!(converted.metadata.components_status, StageStatus::NoProblem); + assert_eq!(converted.metadata.component_status, StageStatus::NoProblem); } #[test] @@ -850,12 +835,12 @@ pub(crate) async fn recognize_ui_impl_with_provider( name: "页面根节点".to_string(), description: String::new(), layout_status: StageStatus::NoProblem, - components_status: StageStatus::NoProblem, + component_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::System, }, - components: Vec::new(), + component: None, children_display_mode: ChildrenDisplayMode::Stack, children, }; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index a68f7ed13..4dadea3a6 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -16,8 +16,8 @@ pub(crate) use workflow::separate_ui_impl; mod tests { use super::*; use crate::ui_editor::component::image::{ImageComponent, ImageType}; - use crate::ui_editor::component::Component; use crate::ui_editor::component::text::TextComponent; + use crate::ui_editor::component::Component; use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode; use crate::ui_editor::layout::control_layout::ControlLayout; use crate::ui_editor::layout::node::Node; @@ -30,7 +30,7 @@ mod tests { use std::path::Path; use typed_floats::tf32::StrictlyPositiveFinite; - fn node(id: &str, components: Vec, children: Vec) -> Node { + fn node(id: &str, component: Option, children: Vec) -> Node { Node { id: NodeId::new(id).unwrap(), layout: ControlLayout::default(), @@ -38,12 +38,12 @@ mod tests { name: id.to_string(), description: String::new(), layout_status: StageStatus::NoProblem, - components_status: StageStatus::NoProblem, + component_status: StageStatus::NoProblem, allow_llm_edit_layout: true, allow_llm_edit_component: true, source: NodeSource::Llm, }, - components, + component, children_display_mode: ChildrenDisplayMode::Stack, children, } @@ -83,11 +83,11 @@ mod tests { }); let root = node( "root", - vec![], + None, vec![node( "container", - vec![], - vec![node("image", vec![image], vec![])], + None, + vec![node("image", Some(image), vec![])], )], ); let result = construct_separation_state(&state(root)); @@ -102,7 +102,7 @@ mod tests { preserve_aspect: false, }, }); - let root = node("root-image", vec![image], vec![]); + let root = node("root-image", Some(image), vec![]); let result = construct_separation_state(&state(root)); let tree = &result.trees[0]; assert_eq!(tree.root.id.as_str(), "root-image"); @@ -120,11 +120,11 @@ mod tests { let text = Component::Text(TextComponent::new("按钮")); let root = node( "root", - vec![], + None, vec![node( "outer-image", - vec![image.clone()], - vec![node("text", vec![text.clone()], vec![])], + Some(image.clone()), + vec![node("text", Some(text.clone()), vec![])], )], ); let result = construct_separation_state(&state(root)); @@ -135,14 +135,14 @@ mod tests { let nested_root = node( "root", - vec![], + None, vec![node( "outer-image", - vec![image.clone()], + Some(image.clone()), vec![node( "inner-image", - vec![image], - vec![node("text", vec![text], vec![])], + Some(image), + vec![node("text", Some(text), vec![])], )], )], ); @@ -156,15 +156,15 @@ mod tests { fn root_image_receives_text_mask() { let root = node( "root-image", - vec![Component::Image(ImageComponent { + Some(Component::Image(ImageComponent { target_graphic: None, image_type: ImageType::Simple { preserve_aspect: false, }, - })], + })), vec![node( "text", - vec![Component::Text(TextComponent::new("标题"))], + Some(Component::Text(TextComponent::new("标题"))), vec![], )], ); @@ -218,8 +218,8 @@ mod tests { }); let mut separation = construct_separation_state(&state(node( "root", - vec![], - vec![node("image", vec![image], vec![])], + None, + vec![node("image", Some(image), vec![])], ))); let id = NodeId::new("image").unwrap(); let paths = HashMap::new(); @@ -302,8 +302,8 @@ mod tests { }); let mut state = construct_separation_state(&state(node( "root", - vec![], - vec![node("image", vec![image], vec![])], + None, + vec![node("image", Some(image), vec![])], ))); let id = NodeId::new("image").unwrap(); let decisions = vec![BindingDecision::Ok { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 56328dfef..6cd9e2be3 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -7,27 +7,21 @@ use std::collections::HashMap; use std::collections::HashSet; fn is_unbound_image(node: &Node) -> bool { - node.components.iter().any(|component| { - matches!( - component, - Component::Image(ImageComponent { - target_graphic: None, - .. - }) - ) - }) + matches!( + node.component.as_ref(), + Some(Component::Image(ImageComponent { + target_graphic: None, + .. + })) + ) } fn has_image_component(node: &Node) -> bool { - node.components - .iter() - .any(|component| matches!(component, Component::Image(_))) + matches!(node.component.as_ref(), Some(Component::Image(_))) } fn has_text_component(node: &Node) -> bool { - node.components - .iter() - .any(|component| matches!(component, Component::Text(_))) + matches!(node.component.as_ref(), Some(Component::Text(_))) } fn node_pixel_rect( diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/component/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/component/mod.rs index 14ab9c854..664f40e42 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/component/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/component/mod.rs @@ -9,3 +9,27 @@ pub enum Component { Image(image::ImageComponent), Text(text::TextComponent), } + +/// LLM 工具返回的节点组件载荷。 +/// +/// 这里不能直接使用 `Option`：部分模型在严格工具 schema 下不会稳定地产生 +/// `null`。用显式的 `PureNode` / `WithComponent` 外部枚举表达两种情况，既保留纯结构节点 +/// 的语义，也让工具调用始终返回一个可判别的对象；落入编辑器 `Node` 时再映射为 +/// `Option`。 +#[derive( + Clone, Debug, PartialEq, schemars::JsonSchema, serde::Deserialize, serde::Serialize, ts_rs::TS, +)] +#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] +pub enum NodeComponent { + PureNode, + WithComponent(Component), +} + +impl NodeComponent { + pub fn into_option(self) -> Option { + match self { + Self::PureNode => None, + Self::WithComponent(component) => Some(component), + } + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/html_renderer/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/html_renderer/mod.rs index e0c8199e7..e908a3117 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/html_renderer/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/html_renderer/mod.rs @@ -199,14 +199,13 @@ fn render_node_with_scale( json!({"childrenDisplayMode": "Exclusive", "childrenRendered": "all"}), ) }); - let components = node - .components - .iter() + let component = node + .component + .as_ref() .map(|component| render_component(state, component)) - .collect::, _>>()? - .into_iter() + .transpose()? .map(|fragment| fragment.into_string()) - .collect::>(); + .unwrap_or_default(); let children = node .children .iter() @@ -226,7 +225,7 @@ fn render_node_with_scale( (comment) @if let Some(group_comment) = exclusive_comment { (group_comment) } div ui-node-id=(node.id.as_str()) style=(style) { - (PreEscaped(components.concat())) + (PreEscaped(component)) (PreEscaped(children.concat())) } }) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/layout/node.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/layout/node.rs index 9375e913f..be70cdca6 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/layout/node.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/layout/node.rs @@ -11,7 +11,7 @@ pub struct Node { pub id: NodeId, pub layout: ControlLayout, pub metadata: NodeMetadata, - pub components: Vec, + pub component: Option, pub children_display_mode: ChildrenDisplayMode, pub children: Vec, } @@ -50,7 +50,7 @@ pub struct NodeMetadata { pub name: String, pub description: String, pub layout_status: StageStatus, - pub components_status: StageStatus, + pub component_status: StageStatus, pub allow_llm_edit_layout: bool, pub allow_llm_edit_component: bool, pub source: NodeSource, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs index 2384476b0..ab4c07c78 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/persistence.rs @@ -25,7 +25,6 @@ const UI_DESIGN_STATE_MAX_IMAGES: usize = 4; const UI_DESIGN_STATE_MAX_SPRITES: usize = 1_024; pub(crate) const UI_DESIGN_STATE_MAX_NODES: usize = 10_000; const UI_DESIGN_STATE_MAX_DEPTH: usize = 128; -pub(crate) const UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE: usize = 64; const UI_DESIGN_STATE_MAX_SAFE_REVISION: u64 = 9_007_199_254_740_991; #[derive(Clone, Debug, Deserialize, PartialEq, Serialize)] @@ -747,12 +746,8 @@ fn validate_node( } } } - if node.components.len() > UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE { - return Err(format!( - "单个 UI 节点最多支持 {UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE} 个组件" - )); - } - for component in &node.components { + validate_component_status(node.component.as_ref(), &node.metadata.component_status)?; + if let Some(component) = &node.component { match component { Component::Image(image) => { if image @@ -778,6 +773,22 @@ fn validate_node( Ok(()) } +fn validate_component_status( + component: Option<&Component>, + status: &crate::ui_editor::layout::node::StageStatus, +) -> Result<(), String> { + if component.is_none() + && matches!( + status, + crate::ui_editor::layout::node::StageStatus::NeedReview(_) + | crate::ui_editor::layout::node::StageStatus::Blocked(_) + ) + { + return Err("纯结构节点的 component_status 必须为 NoProblem".to_string()); + } + Ok(()) +} + fn validate_id(value: &str, label: &str) -> Result<(), String> { if value.is_empty() || value.trim() != value || value.chars().any(char::is_control) { return Err(format!("{label} 无效")); @@ -881,12 +892,12 @@ mod tests { "name": "页面根节点", "description": "", "layout_status": "NoProblem", - "components_status": "NoProblem", + "component_status": "NoProblem", "allow_llm_edit_layout": true, "allow_llm_edit_component": true, "source": "System" }, - "components": [], + "component": null, "children_display_mode": "Stack", "children": [{ "id": "dragged-node", @@ -907,17 +918,17 @@ mod tests { "name": "拖拽节点", "description": "", "layout_status": "NoProblem", - "components_status": "NoProblem", + "component_status": "NoProblem", "allow_llm_edit_layout": true, "allow_llm_edit_component": true, "source": "Human" }, - "components": [{ + "component": { "Image": { "target_graphic": "spirit", "image_type": { "Simple": { "preserve_aspect": false } } } - }], + }, "children_display_mode": "Stack", "children": [] }] @@ -1095,12 +1106,12 @@ mod tests { "name": "根节点", "description": "", "layout_status": "NoProblem", - "components_status": "NoProblem", + "component_status": "NoProblem", "allow_llm_edit_layout": true, "allow_llm_edit_component": true, "source": "System" }, - "components": [], + "component": null, "children_display_mode": "Stack", "children": [] } @@ -1296,12 +1307,12 @@ mod tests { "name": "根节点", "description": "", "layout_status": "NoProblem", - "components_status": "NoProblem", + "component_status": "NoProblem", "allow_llm_edit_layout": false, "allow_llm_edit_component": false, "source": "System" }, - "components": [{ + "component": { "Text": { "content": "标题", "font": {"Bound": "missing-font"}, @@ -1313,7 +1324,7 @@ mod tests { "vertical_overflow": "Truncate", "line_spacing": 1.0 } - }], + }, "children_display_mode": "Stack", "children": [] } @@ -1340,4 +1351,30 @@ mod tests { .expect_err("missing Text font reference must be rejected"); assert!(error.contains("Text 组件引用了不存在的字体素材")); } + + #[test] + fn component_status_matrix_keeps_pure_nodes_unproblematic() { + use crate::ui_editor::component::image::ImageComponent; + + assert!(validate_component_status( + None, + &crate::ui_editor::layout::node::StageStatus::NoProblem, + ) + .is_ok()); + assert!(validate_component_status( + None, + &crate::ui_editor::layout::node::StageStatus::NeedReview("原因".to_string()), + ) + .is_err()); + assert!(validate_component_status( + None, + &crate::ui_editor::layout::node::StageStatus::Blocked("原因".to_string()), + ) + .is_err()); + assert!(validate_component_status( + Some(&Component::Image(ImageComponent::new())), + &crate::ui_editor::layout::node::StageStatus::NeedReview("等待素材".to_string()), + ) + .is_ok()); + } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/workflow.rs index 829782148..ea905233e 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/workflow.rs @@ -1145,8 +1145,8 @@ fn apply_binding_changes( ) -> usize { let mut changed = 0; if let Some(change) = changes.get(&node.id) { - node.components = change.components.clone(); - node.metadata.components_status = change.components_status.clone(); + node.component = change.component.clone().into_option(); + node.metadata.component_status = change.component_status.clone(); changed += 1; } for child in &mut node.children { @@ -1163,7 +1163,7 @@ fn apply_binding_changes( fn state_has_renderable_component(state: &crate::ui_editor::state::State) -> bool { fn has_component(node: &Node) -> bool { - !node.components.is_empty() || node.children.iter().any(has_component) + node.component.is_some() || node.children.iter().any(has_component) } state.ui_trees.iter().any(|tree| has_component(&tree.root)) } @@ -1315,14 +1315,14 @@ fn derive_page_status( } fn collect_binding_blockers(node: &Node, component_count: &mut usize, blockers: &mut Vec) { - *component_count += node.components.len(); + *component_count += usize::from(node.component.is_some()); if let StageStatus::NeedReview(reason) | StageStatus::Blocked(reason) = &node.metadata.layout_status { blockers.push(format!("{} 布局未通过：{reason}", node.metadata.name)); } if let StageStatus::NeedReview(reason) | StageStatus::Blocked(reason) = - &node.metadata.components_status + &node.metadata.component_status { blockers.push(format!("{} 组件未通过：{reason}", node.metadata.name)); } diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts b/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts index 47a8a3a17..3f3422b18 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts @@ -7,8 +7,10 @@ function applyChanges(node: Node, result: BindingDTO): void { (candidate) => candidate.node_id === node.id, ); if (change) { - node.components = structuredClone(change.components); - node.metadata.components_status = structuredClone(change.components_status); + node.component = structuredClone( + change.component === 'PureNode' ? null : change.component.WithComponent, + ); + node.metadata.component_status = structuredClone(change.component_status); } for (const child of node.children) applyChanges(child, result); } diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/bindingOverview.ts b/apps/ai-game-creator-shell/src/features/ui-editor/bindingOverview.ts index 84e6cc6ce..fe770e022 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/bindingOverview.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/bindingOverview.ts @@ -90,13 +90,13 @@ export function getBindingOverview( }; for (const { node } of collectUiTreeNodeTargets(uiTrees)) { - const status = node.metadata.components_status; + const status = node.metadata.component_status; if (isBlocked(status)) overview.blocked += 1; if (isBlocked(status) || isNeedReview(status)) { overview.needsAttention += 1; } - for (const component of node.components) { - addBindingCounts(overview, getComponentBindingCounts(component)); + if (node.component) { + addBindingCounts(overview, getComponentBindingCounts(node.component)); } } @@ -104,16 +104,17 @@ export function getBindingOverview( } export function nodeHasPendingBinding(target: UiTreeNodeTarget): boolean { - return target.node.components.some( - (component) => getComponentBindingCounts(component).pendingSlots > 0, + return ( + target.node.component !== null && + getComponentBindingCounts(target.node.component).pendingSlots > 0 ); } export function nodeNeedsComponentReview(target: UiTreeNodeTarget): boolean { - const status = target.node.metadata.components_status; + const status = target.node.metadata.component_status; return isBlocked(status) || isNeedReview(status); } export function nodeHasBlockedComponents(target: UiTreeNodeTarget): boolean { - return isBlocked(target.node.metadata.components_status); + return isBlocked(target.node.metadata.component_status); } diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts b/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts index af3e0c929..1fcd649c8 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts @@ -106,7 +106,8 @@ export function validateLayoutGenerationPrerequisites( const issues = validateAssetRecognitionPrerequisites(state); const visit = (nodes: State['ui_trees'][number]['root']['children']) => { for (const node of nodes) { - for (const component of node.components) { + const component = node.component; + if (component) { if ( 'Image' in component && component.Image.target_graphic !== null && @@ -219,10 +220,10 @@ export function validateVisualBindingResult( const issues: UiEditorPrerequisiteIssue[] = []; for (const tree of state.ui_trees) { visitNodes([tree.root], (node) => { - if (node.components.length == 0) { + if (node.component === null) { return; } - const issue = componentStatusIssue(node.metadata.components_status); + const issue = componentStatusIssue(node.metadata.component_status); if (issue) issues.push(issue); }); } diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/stageStatusOverview.ts b/apps/ai-game-creator-shell/src/features/ui-editor/stageStatusOverview.ts index c2f9e9c00..80dc27fac 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/stageStatusOverview.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/stageStatusOverview.ts @@ -6,7 +6,7 @@ import type { UITree } from './types/UITree'; export type StageStatusField = Extract< keyof NodeMetadata, - 'layout_status' | 'components_status' + 'layout_status' | 'component_status' >; export type UiTreeNodeTarget = { diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts index ec71edbfd..3bb2c5c4d 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts @@ -1,6 +1,6 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. -import type { Component } from "./Component"; +import type { NodeComponent } from "./NodeComponent"; import type { NodeId } from "./NodeId"; import type { StageStatus } from "./StageStatus"; -export type BindingChange = { node_id: NodeId, components: Array, components_status: StageStatus, }; +export type BindingChange = { node_id: NodeId, component: NodeComponent, component_status: StageStatus, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/Node.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/Node.ts index c05d8743a..bac60a9ca 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/Node.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/Node.ts @@ -5,4 +5,4 @@ import type { ControlLayout } from "./ControlLayout"; import type { NodeId } from "./NodeId"; import type { NodeMetadata } from "./NodeMetadata"; -export type Node = { id: NodeId, layout: ControlLayout, metadata: NodeMetadata, components: Array, children_display_mode: ChildrenDisplayMode, children: Array, }; +export type Node = { id: NodeId, layout: ControlLayout, metadata: NodeMetadata, component: Component | null, children_display_mode: ChildrenDisplayMode, children: Array, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeComponent.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeComponent.ts new file mode 100644 index 000000000..b3116576c --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeComponent.ts @@ -0,0 +1,12 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. +import type { Component } from "./Component"; + +/** + * LLM 工具返回的节点组件载荷。 + * + * 这里不能直接使用 `Option`：部分模型在严格工具 schema 下不会稳定地产生 + * `null`。用显式的 `PureNode` / `WithComponent` 外部枚举表达两种情况，既保留纯结构节点 + * 的语义，也让工具调用始终返回一个可判别的对象；落入编辑器 `Node` 时再映射为 + * `Option`。 + */ +export type NodeComponent = "PureNode" | { "WithComponent": Component }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeMetadata.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeMetadata.ts index 43aee349e..f822b7d4b 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeMetadata.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/NodeMetadata.ts @@ -2,4 +2,4 @@ import type { NodeSource } from "./NodeSource"; import type { StageStatus } from "./StageStatus"; -export type NodeMetadata = { name: string, description: string, layout_status: StageStatus, components_status: StageStatus, allow_llm_edit_layout: boolean, allow_llm_edit_component: boolean, source: NodeSource, }; +export type NodeMetadata = { name: string, description: string, layout_status: StageStatus, component_status: StageStatus, allow_llm_edit_layout: boolean, allow_llm_edit_component: boolean, source: NodeSource, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/UIRect.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/UIRect.ts new file mode 100644 index 000000000..d1db67fd7 --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/UIRect.ts @@ -0,0 +1,9 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * UI 布局在父级坐标系中的轴对齐矩形。 + * + * `min` 是矩形的最小角，`size` 是沿两个坐标轴的尺寸。这里不规定 Y 轴方向， + * 因而既能用于 Y 轴向上的游戏坐标，也能用于 Y 轴向下的画布坐标。 + */ +export type UIRect = { min: [number, number], size: [number, number], }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts b/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts index d04c52f93..a83ab93b4 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/useUiEditorState.ts @@ -64,14 +64,12 @@ export type NodeMetadataPatch = Partial< | 'name' | 'description' | 'layout_status' - | 'components_status' + | 'component_status' | 'allow_llm_edit_layout' | 'allow_llm_edit_component' > >; -export type ComponentIndex = number; - export type NodeTransformOptions = { keepChildrenUnchanged?: boolean; }; @@ -106,6 +104,12 @@ function visitNodes(node: Node, visit: (node: Node) => void): void { for (const child of node.children) visitNodes(child, visit); } +function isProblematicComponentStatus( + status: NodeMetadata['component_status'], +): boolean { + return typeof status !== 'string'; +} + function existingNodeIds(state: State): Set { const ids = new Set(); for (const tree of state.ui_trees) { @@ -144,12 +148,12 @@ function createPageRoot(state: State): Node { name: '页面根节点', description: '', layout_status: 'NoProblem', - components_status: 'NoProblem', + component_status: 'NoProblem', allow_llm_edit_layout: true, allow_llm_edit_component: true, source: 'System', }, - components: [], + component: null, children_display_mode: 'Stack', children: [], }; @@ -175,12 +179,12 @@ function createHumanNode(state: State): Node { name: '新节点', description: '', layout_status: 'NoProblem', - components_status: 'NoProblem', + component_status: 'NoProblem', allow_llm_edit_layout: true, allow_llm_edit_component: true, source: 'Human', }, - components: [], + component: null, children_display_mode: 'Stack', children: [], }; @@ -283,9 +287,7 @@ function sameResource( function visitComponents(nodes: Node[], visit: (component: Component) => void) { for (const node of nodes) { - for (const component of node.components) { - visit(component); - } + if (node.component) visit(node.component); visitComponents(node.children, visit); } } @@ -1121,15 +1123,15 @@ export function useUiEditorState(initialState: State = EMPTY_UI_EDITOR_STATE) { [commit, guard], ); - const setNodeComponents = useCallback( + const setNodeComponent = useCallback( ( treeId: UIDesignImageId, nodeId: NodeId, - components: Component[], + component: Component | null, ): UiEditorOperationResult => { const blocked = guard(); if (blocked) return blocked; - if (!components.every(isValidComponent)) { + if (component && !isValidComponent(component)) { return { ok: false, reason: 'invalid' }; } const current = stateRef.current; @@ -1143,126 +1145,9 @@ export function useUiEditorState(initialState: State = EMPTY_UI_EDITOR_STATE) { const nextTree = next.ui_trees.find( (candidate) => candidate.src_ui_design === treeId, )!; - findNodeLocation(nextTree.root, nodeId)!.node.components = - structuredClone(components); - commit(next); - return { ok: true, value: undefined }; - }, - [commit, guard], - ); - - const insertComponent = useCallback( - ( - treeId: UIDesignImageId, - nodeId: NodeId, - index: ComponentIndex, - component: Component, - ): UiEditorOperationResult => { - const blocked = guard(); - if (blocked) return blocked; - if ( - !isValidComponent(component) || - !Number.isInteger(index) || - index < 0 - ) { - return { ok: false, reason: 'invalid' }; - } - const current = stateRef.current; - const tree = current.ui_trees.find( - (candidate) => candidate.src_ui_design === treeId, - ); - if (!tree) return { ok: false, reason: 'missing' }; - const location = findNodeLocation(tree.root, nodeId); - if (!location) return { ok: false, reason: 'missing' }; - if (index > location.node.components.length) { - return { ok: false, reason: 'invalid' }; - } - const next = cloneState(current); - const nextNode = findNodeLocation( - next.ui_trees.find((candidate) => candidate.src_ui_design === treeId)! - .root, - nodeId, - )!.node; - nextNode.components.splice(index, 0, structuredClone(component)); - commit(next); - return { ok: true, value: undefined }; - }, - [commit, guard], - ); - - const deleteComponent = useCallback( - ( - treeId: UIDesignImageId, - nodeId: NodeId, - index: ComponentIndex, - ): UiEditorOperationResult => { - const blocked = guard(); - if (blocked) return blocked; - if (!Number.isInteger(index) || index < 0) - return { ok: false, reason: 'invalid' }; - const current = stateRef.current; - const tree = current.ui_trees.find( - (candidate) => candidate.src_ui_design === treeId, - ); - if (!tree) return { ok: false, reason: 'missing' }; - const location = findNodeLocation(tree.root, nodeId); - if (!location) return { ok: false, reason: 'missing' }; - if (index >= location.node.components.length) { - return { ok: false, reason: 'invalid' }; - } - const next = cloneState(current); - const nextNode = findNodeLocation( - next.ui_trees.find((candidate) => candidate.src_ui_design === treeId)! - .root, - nodeId, - )!.node; - nextNode.components.splice(index, 1); - commit(next); - return { ok: true, value: undefined }; - }, - [commit, guard], - ); - - const moveComponent = useCallback( - ( - treeId: UIDesignImageId, - nodeId: NodeId, - fromIndex: ComponentIndex, - toIndex: ComponentIndex, - ): UiEditorOperationResult => { - const blocked = guard(); - if (blocked) return blocked; - if ( - !Number.isInteger(fromIndex) || - !Number.isInteger(toIndex) || - fromIndex < 0 || - toIndex < 0 - ) { - return { ok: false, reason: 'invalid' }; - } - const current = stateRef.current; - const tree = current.ui_trees.find( - (candidate) => candidate.src_ui_design === treeId, - ); - if (!tree) return { ok: false, reason: 'missing' }; - const location = findNodeLocation(tree.root, nodeId); - if (!location) return { ok: false, reason: 'missing' }; - if ( - fromIndex >= location.node.components.length || - toIndex >= location.node.components.length - ) { - return { ok: false, reason: 'invalid' }; - } - if (fromIndex === toIndex) return { ok: true, value: undefined }; - const next = cloneState(current); - const nextNode = findNodeLocation( - next.ui_trees.find((candidate) => candidate.src_ui_design === treeId)! - .root, - nodeId, - )!.node; - const [component] = nextNode.components.splice(fromIndex, 1); - if (!component) return { ok: false, reason: 'invalid' }; - nextNode.components.splice(toIndex, 0, component); + const nextNode = findNodeLocation(nextTree.root, nodeId)!.node; + nextNode.component = structuredClone(component); + nextNode.metadata.component_status = 'NoProblem'; commit(next); return { ok: true, value: undefined }; }, @@ -1290,13 +1175,20 @@ export function useUiEditorState(initialState: State = EMPTY_UI_EDITOR_STATE) { .root, nodeId, )!.node; + if ( + patch.component_status !== undefined && + node.component === null && + isProblematicComponentStatus(patch.component_status) + ) { + return { ok: false, reason: 'invalid' }; + } if (patch.name !== undefined) node.metadata.name = patch.name; if (patch.description !== undefined) node.metadata.description = patch.description; if (patch.layout_status !== undefined) node.metadata.layout_status = patch.layout_status; - if (patch.components_status !== undefined) - node.metadata.components_status = patch.components_status; + if (patch.component_status !== undefined) + node.metadata.component_status = patch.component_status; if (patch.allow_llm_edit_layout !== undefined) node.metadata.allow_llm_edit_layout = patch.allow_llm_edit_layout; if (patch.allow_llm_edit_component !== undefined) @@ -1575,10 +1467,7 @@ export function useUiEditorState(initialState: State = EMPTY_UI_EDITOR_STATE) { deleteNode, setNodeTransform, setNodeLayout, - setNodeComponents, - insertComponent, - deleteComponent, - moveComponent, + setNodeComponent, setNodeMetadata, setNodeChildrenDisplayMode, moveNode, diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/ImportOverview.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/ImportOverview.tsx index cd76e9fec..0823299bd 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/ImportOverview.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/ImportOverview.tsx @@ -6,7 +6,7 @@ import type { UITree } from '../../../features/ui-editor/types/UITree'; function countUiComponents(nodes: UiNode[]): number { return nodes.reduce( (total, node) => - total + node.components.length + countUiComponents(node.children), + total + (node.component ? 1 : 0) + countUiComponents(node.children), 0, ); } diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/InputSidebar.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/InputSidebar.tsx index 495db5bd9..b473accb2 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/InputSidebar.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/InputSidebar.tsx @@ -62,12 +62,12 @@ export function InputSidebar({ input }: { input: UiEditorInputProjection }) { name: 'UI Trees', description: '', layout_status: 'NoProblem', - components_status: 'NoProblem', + component_status: 'NoProblem', allow_llm_edit_layout: false, allow_llm_edit_component: false, source: 'System', }, - components: [], + component: null, children_display_mode: 'Stack', children: uiTrees.map((tree) => tree.root), }; diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index ab030488d..89d20155c 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -1,12 +1,5 @@ -import { - ArrowDown, - ArrowUp, - ChevronDown, - ChevronRight, - Plus, - Trash2, -} from 'lucide-react'; -import { useEffect, useState } from 'react'; +import { ChevronDown, ChevronRight, Plus, Trash2 } from 'lucide-react'; +import { useState } from 'react'; import type { Component } from '../../../../../features/ui-editor/types/Component'; import type { UiEditorOperationResult } from '../../../../../features/ui-editor/useUiEditorState'; @@ -18,111 +11,30 @@ import { TextPanel } from './TextPanel'; import { createDefaultTextComponent } from './TextPanelDefaults'; export function ComponentPanel(props: ComponentPanelProps) { - const { - components, - readOnly: propReadOnly, - onSetComponents, - onInsertComponent, - onDeleteComponent, - onMoveComponent, - } = props; + const { component, readOnly: propReadOnly, onSetComponent } = props; const inspectorReadOnly = useInspectorReadOnly(); const readOnly = propReadOnly || inspectorReadOnly; const [addKind, setAddKind] = useState<'Image' | 'Text'>('Image'); - const [expandedIndexes, setExpandedIndexes] = useState>( - () => new Set(), - ); + const [expanded, setExpanded] = useState(Boolean(component)); const [error, setError] = useState(null); - useEffect(() => { - setExpandedIndexes((current) => { - const next = new Set( - [...current].filter((index) => index >= 0 && index < components.length), - ); - if (next.size === current.size) return current; - return next; - }); - }, [components.length]); - - const updateComponent = (index: number, next: Component) => { + function setComponent(next: Component | null) { if (readOnly) return undefined; - const nextComponents = components.slice(); - nextComponents[index] = next; - const result = onSetComponents(nextComponents); - if (result && !result.ok) setError('组件字段无效，更新未应用。'); + const result = onSetComponent(next); + if (result && !result.ok) setError('组件更新失败。'); else setError(null); return result; - }; - - function addComponent() { - if (readOnly) return; - let component: Component; - switch (addKind) { - case 'Image': - component = { Image: createDefaultImageComponent() }; - break; - case 'Text': - component = { Text: createDefaultTextComponent() }; - break; - } - const result = onInsertComponent(components.length, component); - if (result?.ok) { - setExpandedIndexes((current) => new Set(current).add(components.length)); - setError(null); - } else if (result) { - setError('组件新增失败。'); - } } - function deleteComponentAt(index: number) { - if (readOnly) return; - const result = onDeleteComponent(index); - if (result?.ok) { - setExpandedIndexes((current) => { - const next = new Set(); - for (const expanded of current) { - if (expanded === index) continue; - if (expanded > index) next.add(expanded - 1); - else next.add(expanded); - } - return next; - }); - setError(null); - } else if (result) { - setError('组件删除失败。'); - } + function createComponent(): Component { + return addKind === 'Image' + ? { Image: createDefaultImageComponent() } + : { Text: createDefaultTextComponent() }; } - function moveComponent(index: number, direction: 'up' | 'down') { - if (readOnly) return; - // Components are rendered in array order. The last item therefore sits - // visually at the top of the stack. - let nextIndex: number; - switch (direction) { - case 'up': - nextIndex = index + 1; - break; - case 'down': - nextIndex = index - 1; - break; - } - if (nextIndex < 0 || nextIndex >= components.length) return; - const result = onMoveComponent(index, nextIndex); - if (result?.ok) { - setExpandedIndexes((current) => { - const next = new Set(current); - const wasCurrentExpanded = next.has(index); - const wasTargetExpanded = next.has(nextIndex); - next.delete(index); - next.delete(nextIndex); - if (wasCurrentExpanded) next.add(nextIndex); - if (wasTargetExpanded) next.add(index); - return next; - }); - setError(null); - } else if (result) { - setError('组件顺序更新失败。'); - } + function replaceComponent() { + const result = setComponent(createComponent()); + if (result?.ok) setExpanded(true); } return ( @@ -130,7 +42,7 @@ export function ComponentPanel(props: ComponentPanelProps) {

组件

- {components.length} 个 + {component ? componentKind(component) : '无'}
@@ -142,7 +54,7 @@ export function ComponentPanel(props: ComponentPanelProps) { onChange={(event) => setAddKind(event.target.value as 'Image' | 'Text') } - aria-label="新增组件类型" + aria-label="组件类型" > @@ -151,93 +63,49 @@ export function ComponentPanel(props: ComponentPanelProps) { type="button" className="flex h-8 items-center gap-1 rounded-lg bg-blue-600 px-3 text-xs font-semibold text-white disabled:cursor-not-allowed disabled:opacity-40" disabled={readOnly} - onClick={addComponent} - aria-label="新增组件" - title="新增组件" + onClick={replaceComponent} + aria-label={component ? '替换组件' : '新增组件'} + title={component ? '替换组件' : '新增组件'} > - {componentKindLabel(addKind)} + {component ? '替换' : '新增'} - {components.length > 0 && ( -
- {[...components].reverse().map((component, reverseIndex) => { - const index = components.length - reverseIndex - 1; - const expanded = expandedIndexes.has(index); - return ( -
-
- - - - -
- {expanded && ( -
- {renderComponentEditor( - component, - index, - props, - readOnly, - updateComponent, - )} -
- )} -
- ); - })} + {component ? ( +
+
+ + +
+ {expanded && ( +
+ {renderComponentEditor(component, props, readOnly, setComponent)} +
+ )}
- )} - {components.length === 0 && ( + ) : (

当前节点没有组件。

@@ -248,72 +116,45 @@ export function ComponentPanel(props: ComponentPanelProps) { } function componentKind(component: Component): string { - switch (true) { - case 'Image' in component: - return '图片'; - case 'Text' in component: - return '文本'; - default: - return '未知'; - } -} - -function componentKindLabel(kind: 'Image' | 'Text'): string { - switch (kind) { - case 'Image': - return '图片'; - case 'Text': - return '文本'; - } -} - -function componentLayerLabel(index: number, count: number): string { - if (index === count - 1) return '顶部'; - return `层级 ${index + 1}`; -} - -function ExpandIcon({ expanded }: { expanded: boolean }) { - if (expanded) return ; - return ; + if ('Image' in component) return '图片'; + if ('Text' in component) return '文本'; + return '未知'; } function renderComponentEditor( component: Component, - index: number, props: ComponentPanelProps, readOnly: boolean, updateComponent: ( - index: number, - next: Component, + next: Component | null, ) => UiEditorOperationResult | undefined, ) { - switch (true) { - case 'Image' in component: - return ( - updateComponent(index, { Image: next })} - /> - ); - case 'Text' in component: - return ( - updateComponent(index, { Text: next })} - /> - ); - default: - return ( -

- 当前组件类型暂不支持编辑。 -

- ); + if ('Image' in component) { + return ( + updateComponent({ Image: next })} + /> + ); } + if ('Text' in component) { + return ( + updateComponent({ Text: next })} + /> + ); + } + return ( +

+ 当前组件类型暂不支持编辑。 +

+ ); } diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/componentEditorTypes.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/componentEditorTypes.ts index 9d3adc06a..0319745b5 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/componentEditorTypes.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/componentEditorTypes.ts @@ -7,24 +7,15 @@ import type { UiEditorFontFaceState } from '../../../../../features/ui-editor/us import type { UiEditorOperationResult } from '../../../../../features/ui-editor/useUiEditorState'; export type ComponentPanelProps = { - components: Component[]; + component: Component | null; sprites: Record; previewUrls: Record; fonts: Record; fontFaces: Record; projectPath: string; readOnly: boolean; - onSetComponents: ( - components: Component[], - ) => UiEditorOperationResult | undefined; - onInsertComponent: ( - index: number, - component: Component, - ) => UiEditorOperationResult | undefined; - onDeleteComponent: (index: number) => UiEditorOperationResult | undefined; - onMoveComponent: ( - fromIndex: number, - toIndex: number, + onSetComponent: ( + component: Component | null, ) => UiEditorOperationResult | undefined; }; diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/InspectorSidebar.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/InspectorSidebar.tsx index 7557c3727..97c962104 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/InspectorSidebar.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/InspectorSidebar.tsx @@ -115,10 +115,7 @@ export function InspectorSidebar({ fonts={inspector.fonts} fontFaces={inspector.fontFaces} projectPath={inspector.projectPath} - onSetComponents={inspector.setNodeComponents} - onInsertComponent={inspector.insertNodeComponent} - onDeleteComponent={inspector.deleteNodeComponent} - onMoveComponent={inspector.moveNodeComponent} + onSetComponent={inspector.setNodeComponent} onDeleteNode={() => inspector.deleteNode(view.node.id)} deleteDisabled={ inspector.isLocked || view.node.id === inspector.tree?.root.id @@ -293,10 +290,7 @@ function NodeInspector({ fonts, fontFaces, projectPath, - onSetComponents, - onInsertComponent, - onDeleteComponent, - onMoveComponent, + onSetComponent, onDeleteNode, deleteDisabled, }: { @@ -323,10 +317,7 @@ function NodeInspector({ fonts: UiEditorInspectorProjection['fonts']; fontFaces: UiEditorInspectorProjection['fontFaces']; projectPath: string; - onSetComponents: UiEditorInspectorProjection['setNodeComponents']; - onInsertComponent: UiEditorInspectorProjection['insertNodeComponent']; - onDeleteComponent: UiEditorInspectorProjection['deleteNodeComponent']; - onMoveComponent: UiEditorInspectorProjection['moveNodeComponent']; + onSetComponent: UiEditorInspectorProjection['setNodeComponent']; onDeleteNode: () => void; deleteDisabled: boolean; }) { @@ -414,7 +405,7 @@ function NodeInspector({ 来源：{node.metadata.source}
- 组件：{node.components.length} + 组件：{node.component ? 1 : 0}
@@ -434,18 +425,18 @@ function NodeInspector({ }} /> { - if (!isReadOnly) onMetadataChange({ components_status }); + onChange={(component_status) => { + if (!isReadOnly) onMetadataChange({ component_status }); }} />
@@ -498,17 +489,14 @@ function NodeInspector({ onChange={onLayoutChange} /> @@ -115,8 +117,8 @@ function getStepAction(workflow: UiEditorWorkflowProjection) { }; } return { - label: '自动分离并绑定视觉素材', - runningLabel: '自动分离中…', + label: '自动切分素材', + runningLabel: '素材切分中…', action: workflow.separateUi, }; } diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts index 54b77ef1f..1f0b8f943 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts @@ -21,7 +21,7 @@ export function prerequisiteIssuesForStep( return []; case 'structure-recognition': return validateComponentRecognitionPrerequisites(state); - case 'visual-binding': + case 'asset-separation': return validateAssetRecognitionPrerequisites(state); } } @@ -35,7 +35,7 @@ export function postCheckIssuesForStep( return validateReferenceAnalysisResult(state); case 'structure-recognition': return validateStructureRecognitionResult(state); - case 'visual-binding': + case 'asset-separation': return validateVisualBindingResult(state); } } @@ -59,7 +59,7 @@ export function activeStepPrerequisiteIssues( return validateComponentRecognitionPrerequisites(state); case 'structure-recognition': return validateAssetRecognitionPrerequisites(state); - case 'visual-binding': + case 'asset-separation': return validateLayoutReviewPrerequisites(state); } } diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/workflowCompletionNotice.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/workflowCompletionNotice.ts index ff90b6dd5..02c5b317a 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/workflowCompletionNotice.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/workflowCompletionNotice.ts @@ -18,8 +18,8 @@ export function workflowStepLabel(step: UiEditorStepId): string { return '分析参考图'; case 'structure-recognition': return '识别界面结构'; - case 'visual-binding': - return '绑定视觉素材'; + case 'asset-separation': + return '自动切分素材'; } const exhaustiveCheck: never = step; return exhaustiveCheck; diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/index.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/index.tsx index 11c2c82df..2ea43b868 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/index.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/index.tsx @@ -7,13 +7,13 @@ import { type IUiDesignStateStore, uiDesignStateStore, } from '../../features/ui-editor/uiDesignStateStore'; -import { BindingOverview } from './components/BindingOverview'; import { EditorDialogs } from './components/EditorDialogs'; import { ImportOverview } from './components/ImportOverview'; import { InputSidebar } from './components/InputSidebar'; import { InspectorSidebar } from './components/Inspector/InspectorSidebar'; import { PreviewWorkspace } from './components/preview/PreviewWorkspace'; import { RecognitionOverview } from './components/RecognitionOverview'; +import { SeparationOverview } from './components/SeparationOverview'; import { ToolNavigation } from './components/ToolNavigation'; import { WorkflowActionCard } from './components/WorkflowActionCard'; import { WorkflowCompletionModal } from './components/WorkflowCompletionModal'; @@ -265,8 +265,8 @@ export default function UiEditorPage({ session.input.highlightStatusField(nodeId, 'layout_status'); }} /> - ) : session.input.activeStep === 'visual-binding' ? ( - { diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/model.ts b/apps/ai-game-creator-shell/src/view/ui-editor/model.ts index 13ec54cc7..f03f6ef47 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/model.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/model.ts @@ -6,7 +6,7 @@ import type { RemovalImpact } from '../../features/ui-editor/useUiEditorState'; export type UiEditorStepId = | 'reference-analysis' | 'structure-recognition' - | 'visual-binding'; + | 'asset-separation'; export type UiEditorImportKind = 'design-image' | 'font' | 'sprite'; export type UiEditorNodeFocusRequest = { @@ -27,7 +27,7 @@ export const UI_EDITOR_STEPS: Array<{ }> = [ { id: 'reference-analysis', label: '分析参考图' }, { id: 'structure-recognition', label: '识别界面结构' }, - { id: 'visual-binding', label: '绑定视觉素材' }, + { id: 'asset-separation', label: '自动切分素材' }, ]; export const UI_DESIGN_IMAGE_ROLES: Array<{ diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index e09978662..c449a56e7 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -417,7 +417,7 @@ export function useUiEditorSession( activeStep === 'reference-analysis' ? 'structure-recognition' : activeStep === 'structure-recognition' - ? 'visual-binding' + ? 'asset-separation' : null; const spriteReferenceCounts = useMemo(() => { @@ -1111,7 +1111,7 @@ export function useUiEditorSession( (path) => !importedByPath.has(path), ); backfillErrors = missingImports.map( - (path) => `未能登记分离图片：${path}`, + (path) => `未能登记自动切分素材图片：${path}`, ); const importedAssets: ImportedAsset[] = [ ...new Map( @@ -1147,7 +1147,7 @@ export function useUiEditorSession( const sprite = spriteByPath.get(path); if (!sprite) { backfillErrors.push( - `节点 ${bound.node_id} 缺少已登记的分离图片：${path}`, + `节点 ${bound.node_id} 缺少已登记的自动切分素材图片：${path}`, ); continue; } @@ -1183,18 +1183,19 @@ export function useUiEditorSession( ]), ), })); - if (separationResult === null) throw new Error('自动分离没有返回结果'); + if (separationResult === null) + throw new Error('自动切分素材没有返回结果'); const completedResult = separationResult as SeparationDTO; if (!(await save())) { throw new Error( - '分离结果已写入编辑器，但 State 保存失败；sidecar 已保留，可继续恢复。', + '自动切分素材结果已写入编辑器，但 State 保存失败；sidecar 已保留，可继续恢复。', ); } if (backfillErrors.length > 0) { reportWorkflowCompletion( - 'visual-binding', + 'asset-separation', 'failure', - `自动分离已完成，但有 ${backfillErrors.length} 项未能回填；已登记素材并保留恢复状态。\n${backfillErrors.join('\n')}`, + `自动切分素材已完成，但有 ${backfillErrors.length} 项未能回填；已登记素材并保留恢复状态。\n${backfillErrors.join('\n')}`, setSeparationStatus, ); return; @@ -1205,14 +1206,14 @@ export function useUiEditorSession( }); setHasSeparated(true); reportWorkflowCompletion( - 'visual-binding', + 'asset-separation', 'success', - `自动分离完成：${completedResult.bound_nodes.length} 个已绑定，${completedResult.problematic_nodes.length} 个待处理。`, + `自动切分素材完成：${completedResult.bound_nodes.length} 个已切分并回填，${completedResult.problematic_nodes.length} 个待处理。`, setSeparationStatus, ); } catch (cause) { reportWorkflowCompletion( - 'visual-binding', + 'asset-separation', 'failure', cause instanceof Error ? cause.message : String(cause), setSeparationStatus, diff --git a/apps/ai-game-creator-shell/tests/bindingOverview.test.ts b/apps/ai-game-creator-shell/tests/separationOverview.test.ts similarity index 89% rename from apps/ai-game-creator-shell/tests/bindingOverview.test.ts rename to apps/ai-game-creator-shell/tests/separationOverview.test.ts index c7ab782a5..91b39ca72 100644 --- a/apps/ai-game-creator-shell/tests/bindingOverview.test.ts +++ b/apps/ai-game-creator-shell/tests/separationOverview.test.ts @@ -1,11 +1,11 @@ import { describe, expect, it } from 'vitest'; import { - getBindingOverview, + getSeparationOverview, nodeHasBlockedComponents, - nodeHasPendingBinding, + nodeHasPendingSeparation, nodeNeedsComponentReview, -} from '../src/features/ui-editor/bindingOverview'; +} from '../src/features/ui-editor/separationOverview'; import { getNextMatchingUiTreeNodeTarget } from '../src/features/ui-editor/stageStatusOverview'; import type { Component } from '../src/features/ui-editor/types/Component'; import type { Node } from '../src/features/ui-editor/types/Node'; @@ -97,9 +97,9 @@ const trees: UITree[] = [ }, ]; -describe('getBindingOverview', () => { +describe('getSeparationOverview', () => { it('uses per-component helpers to include both image and text slots', () => { - expect(getBindingOverview(trees, sprites)).toEqual({ + expect(getSeparationOverview(trees, sprites)).toEqual({ componentsNeedingAssets: 2, assetSlots: 2, boundSlots: 1, @@ -110,15 +110,15 @@ describe('getBindingOverview', () => { }); }); - it('reuses the common preorder next-target search for every binding queue', () => { + it('reuses the common preorder next-target search for every separation queue', () => { expect( getNextMatchingUiTreeNodeTarget(trees, null, (target) => - nodeHasPendingBinding(target), + nodeHasPendingSeparation(target), )?.node.id, ).toBe('review'); expect( getNextMatchingUiTreeNodeTarget(trees, 'review', (target) => - nodeHasPendingBinding(target), + nodeHasPendingSeparation(target), )?.node.id, ).toBe('review'); expect( diff --git a/apps/ai-game-creator-shell/tests/uiEditorPage.test.ts b/apps/ai-game-creator-shell/tests/uiEditorPage.test.ts index 2f4bc4114..f9615a790 100644 --- a/apps/ai-game-creator-shell/tests/uiEditorPage.test.ts +++ b/apps/ai-game-creator-shell/tests/uiEditorPage.test.ts @@ -347,12 +347,14 @@ describe('UiEditorPage', () => { fireEvent.click(screen.getByRole('button', { name: '仍然继续' })); expect(screen.getByRole('heading', { name: '识别概览' })).toBeTruthy(); - fireEvent.click(screen.getByRole('button', { name: /绑定视觉素材/ })); + fireEvent.click(screen.getByRole('button', { name: /自动切分素材/ })); fireEvent.click(screen.getByRole('button', { name: '仍然继续' })); - expect(screen.getByRole('heading', { name: '绑定概览' })).toBeTruthy(); + expect( + screen.getByRole('heading', { name: '自动切分素材概览' }), + ).toBeTruthy(); }); - it('opens a completed workflow directly at the visual binding review stage', async () => { + it('opens a completed workflow directly at the asset separation review stage', async () => { const stateStore: IUiDesignStateStore = { load: vi.fn().mockResolvedValue({ revision: 3, @@ -367,25 +369,25 @@ describe('UiEditorPage', () => { projectPath: '/tmp/ui-editor-final-review', resourceId: 'ui-resource', stateStore, - initialStep: 'visual-binding', + initialStep: 'asset-separation', initialFurthestStepIndex: 2, }), ); expect( - await screen.findByRole('heading', { name: '绑定概览' }), + await screen.findByRole('heading', { name: '自动切分素材概览' }), ).toBeTruthy(); expect( screen .getByRole('navigation', { name: 'UI 编辑流程' }) .querySelector('button[aria-current="step"]')?.textContent, - ).toContain('绑定视觉素材'); + ).toContain('自动切分素材'); }); it('keeps the pending binding count informational instead of navigable', () => { render(createElement(UiEditorPage, { projectPath: '/tmp/ui-editor' })); - fireEvent.click(screen.getByRole('button', { name: /绑定视觉素材/ })); + fireEvent.click(screen.getByRole('button', { name: /自动切分素材/ })); fireEvent.click(screen.getByRole('button', { name: '仍然继续' })); expect( @@ -396,7 +398,7 @@ describe('UiEditorPage', () => { it('switches tools freely without inventing completed workflow state', () => { render(createElement(UiEditorPage, { projectPath: '/tmp/ui-editor' })); - fireEvent.click(screen.getByRole('button', { name: /绑定视觉素材/ })); + fireEvent.click(screen.getByRole('button', { name: /自动切分素材/ })); expect(screen.getByRole('heading', { name: '检查发现问题' })).toBeTruthy(); }); diff --git a/apps/ai-game-creator-shell/tests/workflowCompletionNotice.test.ts b/apps/ai-game-creator-shell/tests/workflowCompletionNotice.test.ts index 01016c4ae..af3f2d1cc 100644 --- a/apps/ai-game-creator-shell/tests/workflowCompletionNotice.test.ts +++ b/apps/ai-game-creator-shell/tests/workflowCompletionNotice.test.ts @@ -15,6 +15,6 @@ describe('workflow completion notice helpers', () => { it('maps every workflow step to a user-facing label', () => { expect(workflowStepLabel('reference-analysis')).toBe('分析参考图'); expect(workflowStepLabel('structure-recognition')).toBe('识别界面结构'); - expect(workflowStepLabel('visual-binding')).toBe('绑定视觉素材'); + expect(workflowStepLabel('asset-separation')).toBe('自动切分素材'); }); }); diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 2e6658f85..5e45daf45 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -7903,7 +7903,7 @@ CI 上 `background_agent_runtime_recovers_stale_running_before_pending_task` 在 ## 2026-08-24 AGC UI 原型桥接与自主 UI workflow - 决策：`ui-prototype` 图片与 `UI` JSON 编辑资源保持两种正式类型。Agent 通过受控 `ui.workflow.run` 按 `prepare -> recognize -> status -> finalize` 创建页面资源、关联源图、持久化 UI State 和 manifest 阶段；`recognize` 直接复用 UI Editor 的 provider-backed 结构识别、多树合并与组件绑定命令，按 `reference-ready -> structure-ready -> merge-ready -> binding-ready` 逐阶段写入并推进项目 revision。页面可显式关联已登记图片/图标和字体，图片/图标按 5 项一批绑定，字体安全元数据进入绑定上下文且未知引用失败关闭。Runtime 回执携带 `revisionAdvanceCount`；Provider 未配置、请求失败、工具调用缺失、结果不匹配、未产出可渲染组件或仍有待审节点时保留最近真实阶段，禁止用 deterministic seed 冒充语义处理完成。 -- 客户端：画布点击 `ui-prototype` 先幂等桥接到 `UI` JSON，并立即刷新 manifest；关联查找按 canonical resource identity 且优先已完成 workflow 资源。全部页面完成后，工作台自动打开首个页面的 UI 编辑器 `visual-binding` 最终阶段。 +- 客户端：画布点击 `ui-prototype` 先幂等桥接到 `UI` JSON，并立即刷新 manifest；关联查找按 canonical resource identity 且优先已完成 workflow 资源。全部页面完成后，工作台自动打开首个页面的 UI 编辑器 `asset-separation` 最终阶段。 - 完成门：`finalize` 必须为每个页面提供 `game/` 下真实 UTF-8 应用文件并安装当前 UI State revision 标记；缺少结构、组件、页面或标记时拒绝完成。详细输入、阶段与恢复契约见 [`docs/【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md`](../../【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md)。 - 验证：前端 bridge 6/6、资源实时集成 19/19、AppSurface 410/410、AGC typecheck、Rust workflow 定向测试覆盖 provider 前的 reference 阶段与真实调用失败关闭、Rust bridge 1/1、编码、格式和 diff 门禁通过；认证登录与真实 Provider 生成的桌面端 E2E 尚未具备可用会话，保持未验证。 diff --git a/docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md b/docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md index 2f63f00e6..8f5321635 100644 --- a/docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md +++ b/docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md @@ -106,7 +106,7 @@ UI Editor Inspector 的全局只读状态唯一来源是 `controller.editor.isLo ## 2026-08-18 UI Editor 结构识别、合并与增量导入边界 -UI Editor 当前把“识别界面结构”定义为结构草稿阶段，而不是完整视觉还原阶段。识别 DTO 只负责输出节点层级、几何、名称、描述和置信度；节点组件暂为空，由后续“绑定视觉素材”阶段补齐 Image / Text 组件。`applyRecognitionResult` 可以整体替换当前 `ui_trees`，但该替换只代表结构结果，不能宣称已经保留截图中的视觉内容；组件状态使用 `NoProblem`，前置检查仍会根据空组件和素材绑定情况阻止跳过绑定阶段。 +UI Editor 当前把“识别界面结构”定义为结构草稿阶段，而不是完整视觉还原阶段。识别 DTO 只负责输出节点层级、几何、名称、描述和置信度；节点组件暂为空，由后续“自动切分素材”阶段补齐 Image / Text 组件。`applyRecognitionResult` 可以整体替换当前 `ui_trees`，但该替换只代表结构结果，不能宣称已经保留截图中的视觉内容；组件状态使用 `NoProblem`，前置检查仍会根据空组件和素材切分情况阻止跳过自动切分阶段。 结构识别、界面语义建议、多图合并和组件绑定只接受不超过 `1 MiB` 的 LLM 工具调用 arguments，并在递归业务类型反序列化前先解析为通用 JSON、迭代检查结构预算。结构识别按每棵返回树独立限制为最多 `512` 个 LLM 节点和 `32` 层，不汇总多棵树的节点数，也不计 Rust 自动补建的页面根；界面语义建议最多 `4` 个节点和 `4` 层；合并计划最多 `512` 个计划节点和 `32` 层，`Simple.children` 与 `Merged.merged_from` 使用同一计数和深度口径；组件绑定 `changes` 不得超过当前可编辑节点数且绝对上限为 `10,000`，每个 change 的完整组件栈最多 `64` 个组件。任何超限结果均整次拒绝，不截断、不返回部分结果，也不把工具 arguments 正文写入日志。 @@ -1286,7 +1286,7 @@ game-project/ DirectProject 使用 `approvalPolicy=never`，避免每次原生调用再经过泛化 ToolHost 包装；原生命令网络保持关闭，联网资料继续走受控 `agc_web_search`。多 Agent、Apps、完整插件 Runtime、hooks、Goals、Workspace Dependencies、Tool Suggestion 和原生浏览器/电脑控制仍关闭，避免绕过 AGC durable delegation、浏览器证据和副作用审计；图片生成通过客户端审核的 `agc_tools.agc_generate_image` 暴露普通单图、角色图、视觉规范图和 UI 设计图，完整游戏美术包继续使用 `agc_tools.taonier_prepare_game_art`，两者都复用同一客户端登录态、幂等账本、下载校验和 manifest/revision 投影，不开放 Codex 原生 image tool。app-server 使用隔离 `CODEX_HOME`：内置 `agc_tools` 由客户端启动参数注入，用户在客户端扩展列表启用的独立第三方 MCP 以原生配置写入该次隔离 home；全局 Codex MCP、禁用项、Plugin hooks/apps 和其它插件能力不进入 DirectProject。第三方项固定非 required，配置或启动失败只记录该项，不替换 `agc_tools`；provider session token、工具桥地址和受控搜索标记不得通过第三方 MCP 的环境转发字段泄露。配置了 AGC LLM Key 或可解析的 `OPENAI_API_KEY` 登录态时，真实 provider 凭据只由 AGC 本地 provider proxy 持有，Codex 仅使用连接级随机代理令牌；无法安全代理的 OAuth `auth.json` 继续关闭 native shell/unified exec。`agc_tools` 的平台授权由 AGC 客户端当前登录会话和受控后端完成，普通客户端不得把 DirectProject 请求改成外部 API Key 请求；401/403 只投影为客户端登录或权限异常，不向用户索要凭据或暴露内部 URL。shell 子进程采用 `shell_environment_policy` core 继承及 secret/proxy/bridge 排除，provider key 和桥接凭据不得进入命令环境。系统提示词不再预注入项目源码快照或 Skill 正文，Codex 按需读取当前 cwd 文件。 ## 2026-08-24 AGC UI 原型桥接与自主 UI workflow -- 2026-08-24 起，`ui-prototype` 与 UI 编辑器的 `UI` JSON 资源明确分离。设计图生成后必须由白名单 `ui.workflow.run` 按页面执行 `prepare → recognize → status → finalize`：为每个功能页面创建并关联 `UI` JSON，载入页面设计图和已登记图片/图标/字体，调用 UI Editor 的 provider-backed 结构识别、多树合并与分批组件绑定，持久化 State/revision，写入 `game/` 应用标记，并把 `reference-ready → structure-ready → merge-ready → binding-ready → application-ready → completed` 各阶段的 `generationKind` 和 manifest revision 投影给客户端。Provider 未配置、请求失败、工具调用缺失、结果不匹配、未知字体引用、未产出可渲染组件或仍有待审节点时保留最近真实阶段并返回 blocker，不得使用 deterministic seed 冒充完成。工作台点击 `ui-prototype` 时通过 `ensure_ui_design_resource_for_prototype` 幂等补齐关联资源；工作流完成后自动打开首个页面的 UI 编辑器 `visual-binding` 最终阶段，交给用户检查和手动调整。UI 编辑器独立的语义建议请求也必须复用统一 LLM 传输选择，`llm.stream=true` 时发送 `stream=true` 并聚合完整工具调用后再校验结果。只生成图片、登记空 JSON 或进入普通图片画布均不构成 UI 工作流完成，详见 [`【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md`](../【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md)。 +- 2026-08-24 起，`ui-prototype` 与 UI 编辑器的 `UI` JSON 资源明确分离。设计图生成后必须由白名单 `ui.workflow.run` 按页面执行 `prepare → recognize → status → finalize`：为每个功能页面创建并关联 `UI` JSON，载入页面设计图和已登记图片/图标/字体，调用 UI Editor 的 provider-backed 结构识别、多树合并与分批自动切分素材，持久化 State/revision，写入 `game/` 应用标记，并把 `reference-ready → structure-ready → merge-ready → binding-ready → application-ready → completed` 各阶段的 `generationKind` 和 manifest revision 投影给客户端。Provider 未配置、请求失败、工具调用缺失、结果不匹配、未知字体引用、未产出可渲染组件或仍有待审节点时保留最近真实阶段并返回 blocker，不得使用 deterministic seed 冒充完成。工作台点击 `ui-prototype` 时通过 `ensure_ui_design_resource_for_prototype` 幂等补齐关联资源；工作流完成后自动打开首个页面的 UI 编辑器 `asset-separation` 最终阶段，交给用户检查和手动调整。UI 编辑器独立的语义建议请求也必须复用统一 LLM 传输选择，`llm.stream=true` 时发送 `stream=true` 并聚合完整工具调用后再校验结果。只生成图片、登记空 JSON 或进入普通图片画布均不构成 UI 工作流完成，详见 [`【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md`](../【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md)。 ## 2026-08-28 AGC 自主构建 relaxed 编排覆盖 diff --git a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md similarity index 93% rename from docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md rename to docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index a0691c6b6..d06c1ef77 100644 --- a/docs/technical/【技术方案】UI编辑器自动分离工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -1,10 +1,10 @@ -# UI 编辑器自动分离工作流 +# UI 编辑器自动切分素材工作流 更新时间：`2026-09-08` ## 目标 -将 UI 编辑器现有“用户先提供独立图片/图标，再执行组件绑定”的入口替换为自动分离：结构识别阶段直接返回可渲染组件草稿，分离阶段按整页叶节点批次调用图片编辑模型，再由视觉模型确认处理图中的区域与目标节点。 +将 UI 编辑器现有“用户先提供独立图片/图标，再执行组件绑定”的入口替换为自动切分素材：结构识别阶段直接返回可渲染组件草稿，切分阶段按整页叶节点批次调用图片编辑模型，再由视觉模型确认处理图中的区域与目标节点。 ## 识别结果 @@ -84,8 +84,8 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## 前端正式接入 -- UI 编辑器点击“自动分离”时，前端只调用一次 `separate_ui`，消费完整 `SeparationDTO`；separation 内部 batch 不向前端暴露，也不在 UI 中显示 batch 进度。 -- 如果 sidecar 已存在未完成的 `state.json`，点击入口时先打开独立弹窗，由用户选择“继续上次分离”或“开始新的分离”。继续复用 sidecar 的 pending tree；重新开始只替换当前 `state.json`，不删除 sidecar 图片。 +- UI 编辑器点击“自动切分素材”时，前端只调用一次 `separate_ui`，消费完整 `SeparationDTO`；separation 内部 batch 不向前端暴露，也不在 UI 中显示 batch 进度。 +- 如果 sidecar 已存在未完成的 `state.json`，点击入口时先打开独立弹窗，由用户选择“继续上次自动切分素材”或“开始新的自动切分素材”。继续复用 sidecar 的 pending tree；重新开始只替换当前 `state.json`，不删除 sidecar 图片。 - `BoundNode.cut_image_path` 必须是项目根相对路径。前端使用现有 `import_local_project_image_assets` 登记 cut 图片；由于该通用命令单次最多 100 个路径，前端可以在资源登记阶段按 100 条分组调用，但这不属于 separation batch，也不向用户展示。 - 现有本地资源导入按清洗后的文件名 stem 与内容摘要生成目标路径；相同目标路径直接复用已有 manifest asset ID，内容不同则拒绝覆盖或生成不同摘要路径。前端不自行猜测 SpriteAsset 是否存在，也不从 NodeId 派生 SpriteAssetId。 - 全部可登记图片完成导入后，前端在一个 `runWithStateLocked` 中复用 `addSpriteAssets` 的内部 State 变换逻辑，加入返回的 SpriteAsset 并回填仍匹配 Node 的唯一未绑定 Image component，最后一次性提交 State。公开 `addSpriteAssets` 的普通 mutation guard 不放宽。 @@ -104,7 +104,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## TODO - 正在执行 batch 的持久化和恢复。 -- 前端自动分离接入已实现；仍需补齐真实 Tauri/前端联调回归测试与失败注入测试。 +- 前端自动切分素材接入已实现；仍需补齐真实 Tauri/前端联调回归测试与失败注入测试。 - 临时图片清理/归档策略。 - 手动抠图能力。 - problematic 对更高层 workflow 完成门禁的最终定义。 diff --git a/docs/technical/【设计】UI编辑器工作流完成通知弹窗-2026-09-04.md b/docs/technical/【设计】UI编辑器工作流完成通知弹窗-2026-09-04.md index 5cdc52832..385dcea54 100644 --- a/docs/technical/【设计】UI编辑器工作流完成通知弹窗-2026-09-04.md +++ b/docs/technical/【设计】UI编辑器工作流完成通知弹窗-2026-09-04.md @@ -2,7 +2,7 @@ ## 目标 -UI 编辑器的“分析参考图”“识别界面结构”“绑定视觉素材”三个工作流动作在每次运行结束后，用独立的阻塞通知弹窗明确反馈结果，避免仅依赖卡片内一行状态文本而被忽略。 +UI 编辑器的“分析参考图”“识别界面结构”“自动切分素材”三个工作流动作在每次运行结束后，用独立的阻塞通知弹窗明确反馈结果，避免仅依赖卡片内一行状态文本而被忽略。 ## 交互约定 @@ -15,13 +15,13 @@ UI 编辑器的“分析参考图”“识别界面结构”“绑定视觉素 ## 文案 -弹窗标题由步骤名和结果态组成，例如“识别界面结构完成”或“绑定视觉素材失败”。正文复用卡片状态文本，并逐条扩展结果信息，统一以“请检查”收尾。 +弹窗标题由步骤名和结果态组成，例如“识别界面结构完成”或“自动切分素材失败”。正文复用卡片状态文本，并逐条扩展结果信息，统一以“请检查”收尾。 成功状态的基线文案： - 分析参考图：保留已应用的语义建议数量；若现有状态可可靠取得问题/待确认数量，则一并展示。 - 识别界面结构：保留替换的界面树数量，并展示识别结果中的待检查/必须修复数量（若可取得）。 -- 绑定视觉素材：保留现有 `B/B` 批次计数，改为用户可读的绑定结果。 +- 自动切分素材：保留现有 `B/B` 批次计数，改为用户可读的切分结果。 失败状态保留实际错误文本，仅在弹窗标题中补充步骤和失败上下文，正文同样以“请检查”收尾。 diff --git a/docs/【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md b/docs/【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md index fe5c971a3..61258c68e 100644 --- a/docs/【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md +++ b/docs/【技术方案】UI工作流资源桥接与Runtime执行-2026-08-24.md @@ -38,7 +38,7 @@ Agent 通过白名单工具 `ui.workflow.run` 发起工作流。项目路径由 1. `prepare` 为每个页面创建确定性的 `kind=UI` JSON 资源，引用源 `ui-prototype` 和页面设计图，载入设计图尺寸与相对路径到 `ui_design_images`，并保存 State。 2. `recognize` 依次执行 Provider 多模态结构识别、现有多树合并器、最多每批 5 项的图片/图标组件绑定，并把已登记字体的安全元数据提供给绑定器；阶段分别持久化为 `structure-ready`、`merge-ready`、`binding-ready`，重复执行从最近真实阶段恢复。 3. `status` 只回读 State、页面阶段和 blockers，不推进项目 revision。 -4. `finalize` 只接受 `game/` 下的真实 UTF-8 文件，写入与 UI State revision 绑定的应用标记；所有页面通过应用门禁后才返回 `visual-binding` 最终阶段路由。缺少页面、资源、组件或应用标记时拒绝伪造完成。 +4. `finalize` 只接受 `game/` 下的真实 UTF-8 文件，写入与 UI State revision 绑定的应用标记；所有页面通过应用门禁后才返回 `asset-separation` 最终阶段路由。缺少页面、资源、组件或应用标记时拒绝伪造完成。 每次 State 或 manifest 阶段变化都推进项目 revision。Runtime 回执带有 `revisionAdvanceCount`，用于并发项目 revision 门禁；manifest 资产的 `source.generationKind` 依次记录： @@ -67,7 +67,7 @@ ui-workflow.completed - 没有关联时原子创建 `ui/UI 设计 N.json`，登记 `kind=UI`、`application/json`，并把原型图作为首张页面设计图载入 State。 - 成功后通过 `onManifestChange` 更新客户端资源投影，再打开 UI 编辑器；普通桥接从 `reference-analysis` 开始。 -点击已有 `UI` 资源直接打开 UI 编辑器。若 manifest 阶段为 `ui-workflow.completed`，工作台自动打开该资源的 `visual-binding` 阶段（最远步骤为 2），交给用户做最终检查和手动调整。 +点击已有 `UI` 资源直接打开 UI 编辑器。若 manifest 阶段为 `ui-workflow.completed`，工作台自动打开该资源的 `asset-separation` 阶段（最远步骤为 2），交给用户做最终检查和手动调整。 ## 诚实完成门禁 From 74f726e65d8cdca2bc600fcccf33579439ebc0bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 18:43:31 +0800 Subject: [PATCH 086/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E7=B4=A0=E6=9D=90=E6=A3=80=E6=9F=A5=E6=9D=A1?= =?UTF-8?q?=E4=BB=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 移除自动切分对预先导入独立素材的错误要求。 统一自动切分前置与结果检查，并补齐图片和字体引用校验。 在新建和重新开始自动切分时执行前置检查，恢复流程继续尽力完成。 --- .../src/features/ui-editor/requisites.ts | 79 +++++++------------ .../ui-editor/components/WorkflowChecks.ts | 15 ++-- .../src/view/ui-editor/useUiEditorPage.ts | 26 ++++++ ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 1 + 4 files changed, 62 insertions(+), 59 deletions(-) diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts b/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts index 1fcd649c8..759792923 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts @@ -72,7 +72,7 @@ export function validateComponentRecognitionPrerequisites( return issues; } -export function validateAssetRecognitionPrerequisites( +export function validateAssetSeparationPrerequisites( state: State, ): UiEditorPrerequisiteIssue[] { const issues = validateComponentRecognitionPrerequisites(state); @@ -91,58 +91,9 @@ export function validateAssetRecognitionPrerequisites( }); } } - if (Object.keys(state.sprite_assets).length === 0) { - issues.push({ - code: 'missing-sprite-assets', - message: '请先导入独立素材', - }); - } return issues; } -export function validateLayoutGenerationPrerequisites( - state: State, -): UiEditorPrerequisiteIssue[] { - const issues = validateAssetRecognitionPrerequisites(state); - const visit = (nodes: State['ui_trees'][number]['root']['children']) => { - for (const node of nodes) { - const component = node.component; - if (component) { - if ( - 'Image' in component && - component.Image.target_graphic !== null && - !(component.Image.target_graphic in state.sprite_assets) - ) { - issues.push({ - code: 'missing-target-graphic', - message: '图片组件引用的独立素材不存在', - resourceId: component.Image.target_graphic, - }); - } - if ('Text' in component) { - const font = component.Text.font; - if (typeof font !== 'string' && !(font.Bound in state.font_assets)) { - issues.push({ - code: 'missing-font', - message: '文本组件引用的字体不存在', - resourceId: font.Bound, - }); - } - } - } - visit(node.children); - } - }; - for (const tree of state.ui_trees) visit([tree.root]); - return issues; -} - -export function validateLayoutReviewPrerequisites( - state: State, -): UiEditorPrerequisiteIssue[] { - return validateLayoutGenerationPrerequisites(state); -} - function layoutStatusIssue( status: StageStatus, ): UiEditorPrerequisiteIssue | null { @@ -214,7 +165,7 @@ export function validateStructureRecognitionResult( return issues; } -export function validateVisualBindingResult( +export function validateAssetSeparationResult( state: State, ): UiEditorPrerequisiteIssue[] { const issues: UiEditorPrerequisiteIssue[] = []; @@ -225,6 +176,32 @@ export function validateVisualBindingResult( } const issue = componentStatusIssue(node.metadata.component_status); if (issue) issues.push(issue); + const component = node.component; + if ('Image' in component) { + const targetGraphic = component.Image.target_graphic; + if (targetGraphic === null) { + issues.push({ + code: 'missing-separated-image', + message: '图片组件尚未完成素材切分', + }); + } else if (!(targetGraphic in state.sprite_assets)) { + issues.push({ + code: 'missing-target-graphic', + message: '图片组件引用的切分素材不存在', + resourceId: targetGraphic, + }); + } + } + if ('Text' in component) { + const font = component.Text.font; + if (typeof font !== 'string' && !(font.Bound in state.font_assets)) { + issues.push({ + code: 'missing-font', + message: '文本组件引用的字体不存在', + resourceId: font.Bound, + }); + } + } }); } return issues; diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts index 1f0b8f943..fbf4d1582 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts @@ -1,11 +1,10 @@ import { type UiEditorPrerequisiteIssue, - validateAssetRecognitionPrerequisites, + validateAssetSeparationPrerequisites, + validateAssetSeparationResult, validateComponentRecognitionPrerequisites, - validateLayoutReviewPrerequisites, validateReferenceAnalysisResult, validateStructureRecognitionResult, - validateVisualBindingResult, } from '../../../features/ui-editor/requisites'; import type { State } from '../../../features/ui-editor/types/State'; import type { UiEditorStepId } from '../model'; @@ -22,7 +21,7 @@ export function prerequisiteIssuesForStep( case 'structure-recognition': return validateComponentRecognitionPrerequisites(state); case 'asset-separation': - return validateAssetRecognitionPrerequisites(state); + return validateAssetSeparationPrerequisites(state); } } @@ -36,7 +35,7 @@ export function postCheckIssuesForStep( case 'structure-recognition': return validateStructureRecognitionResult(state); case 'asset-separation': - return validateVisualBindingResult(state); + return validateAssetSeparationResult(state); } } @@ -46,7 +45,7 @@ export function postCheckIssuesForSave( return [ ...validateReferenceAnalysisResult(state), ...validateStructureRecognitionResult(state), - ...validateVisualBindingResult(state), + ...validateAssetSeparationResult(state), ]; } @@ -58,8 +57,8 @@ export function activeStepPrerequisiteIssues( case 'reference-analysis': return validateComponentRecognitionPrerequisites(state); case 'structure-recognition': - return validateAssetRecognitionPrerequisites(state); + return validateComponentRecognitionPrerequisites(state); case 'asset-separation': - return validateLayoutReviewPrerequisites(state); + return validateAssetSeparationPrerequisites(state); } } diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index c449a56e7..735a02af7 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1241,6 +1241,19 @@ export function useUiEditorSession( setSeparationRecovery(recovery); return; } + const prerequisiteIssues = prerequisiteIssuesForStep( + editor.state, + 'asset-separation', + ); + if (prerequisiteIssues.length > 0) { + reportWorkflowCompletion( + 'asset-separation', + 'failure', + prerequisiteIssues.map((issue) => issue.message).join('；'), + setSeparationStatus, + ); + return; + } await runSeparationWorkflow(); } catch (cause) { setSeparationStatus( @@ -1257,6 +1270,19 @@ export function useUiEditorSession( async function restartSeparation() { if (!resourceId) return; setSeparationRecovery(null); + const prerequisiteIssues = prerequisiteIssuesForStep( + editor.state, + 'asset-separation', + ); + if (prerequisiteIssues.length > 0) { + reportWorkflowCompletion( + 'asset-separation', + 'failure', + prerequisiteIssues.map((issue) => issue.message).join('；'), + setSeparationStatus, + ); + return; + } try { await invoke('discard_separation_recovery', { projectPath, diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index d06c1ef77..053da070b 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -85,6 +85,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## 前端正式接入 - UI 编辑器点击“自动切分素材”时，前端只调用一次 `separate_ui`，消费完整 `SeparationDTO`；separation 内部 batch 不向前端暴露，也不在 UI 中显示 batch 进度。 +- 自动切分的前置检查只要求界面图及对应 UI tree 有效，不要求用户预先导入 SpriteAsset；结果检查负责报告未回填的 Image、丢失的切分素材引用、丢失的字体引用及组件审阅状态。新建和重新开始切分时执行前置检查，继续已有 sidecar 时直接按恢复状态尽力完成。 - 如果 sidecar 已存在未完成的 `state.json`，点击入口时先打开独立弹窗，由用户选择“继续上次自动切分素材”或“开始新的自动切分素材”。继续复用 sidecar 的 pending tree；重新开始只替换当前 `state.json`，不删除 sidecar 图片。 - `BoundNode.cut_image_path` 必须是项目根相对路径。前端使用现有 `import_local_project_image_assets` 登记 cut 图片；由于该通用命令单次最多 100 个路径，前端可以在资源登记阶段按 100 条分组调用，但这不属于 separation batch，也不向用户展示。 - 现有本地资源导入按清洗后的文件名 stem 与内容摘要生成目标路径；相同目标路径直接复用已有 manifest asset ID，内容不同则拒绝覆盖或生成不同摘要路径。前端不自行猜测 SpriteAsset 是否存在，也不从 NodeId 派生 SpriteAssetId。 From 69a0804c035e300ce4495039f6c6e9dd7cc71d10 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 18:46:39 +0800 Subject: [PATCH 087/248] =?UTF-8?q?=E8=B0=83=E6=95=B4=E5=88=87=E5=88=86?= =?UTF-8?q?=E5=8C=BA=E5=9F=9F=E8=BE=B9=E7=95=8C=E6=89=A9=E5=B1=95=E4=B8=8A?= =?UTF-8?q?=E9=99=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将像素边界每边调整上限常量设为百分之百 同步更新受影响的区域归一化测试 技术方案仅引用上限常量，不重复写死数值 --- .../src/ui_editor/commands/separation/area.rs | 21 ++++++++++--------- ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 2 +- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs index 08a218b35..68cc9ba47 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -1,10 +1,11 @@ use super::model::BindingArea; use image::RgbaImage; -/// Each edge may move by at most half of the area dimension returned by the -/// visual model. Keep this policy explicit so changing it is an intentional -/// workflow decision rather than a scattered numeric literal. -pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT: u32 = 50; +/// Each edge may move by at most this percentage of the corresponding area +/// dimension returned by the visual model. Keep this policy explicit so +/// changing it is an intentional workflow decision rather than a scattered +/// numeric literal. +pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT: u32 = 100; #[derive(Clone, Copy, Debug, PartialEq)] pub(crate) struct NormalizedBindingArea { @@ -339,7 +340,7 @@ mod tests { fn adjusts_each_edge_independently() { let image = image_with_rect(32, 32, 10, 11, 16, 18); let result = normalize_binding_area(&image, area(10, 12, 10, 3)).unwrap(); - assert_eq!(result.area, area(10, 11, 6, 5)); + assert_eq!(result.area, area(10, 11, 6, 7)); } #[test] @@ -361,10 +362,10 @@ mod tests { } #[test] - fn caps_each_edge_at_half_of_the_original_dimension() { + fn caps_each_edge_at_original_dimension() { let image = image_with_rect(64, 64, 0, 0, 64, 64); let result = normalize_binding_area(&image, area(16, 16, 8, 8)).unwrap(); - assert_eq!(result.area, area(12, 12, 16, 16)); + assert_eq!(result.area, area(8, 8, 24, 24)); assert!(result.clamped); } @@ -385,11 +386,11 @@ mod tests { } #[test] - fn one_pixel_area_stays_nonzero_when_adjustment_limit_is_zero() { + fn one_pixel_area_expands_with_configured_adjustment_limit() { let image = image_with_rect(8, 8, 2, 2, 5, 5); let result = normalize_binding_area(&image, area(3, 3, 1, 1)).unwrap(); - assert_eq!(result.area, area(3, 3, 1, 1)); - assert!(result.clamped); + assert_eq!(result.area, area(2, 2, 3, 3)); + assert!(!result.clamped); } #[test] diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 053da070b..5aa3921af 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -51,7 +51,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 -- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的 `50%` 位移上限。该上限由模块级常量定义。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到 `50%` 上限、图像边界或非零尺寸约束。性能优化列 TODO。 +- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的位移上限。该上限由模块级常量 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT` 定义。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到该常量上限、图像边界或非零尺寸约束。性能优化列 TODO。 - `NeedRework` 携带短问题描述（最多 512 个 Unicode 字符）；通过校验后按产生顺序追加到目标 `SeparationNode.note.rework_notes`，下一次该节点进入 image-edit 时全部意见会注入提取 prompt。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 - 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 From 8c521ca8916518e15a6e177c973aa6f33c2620d8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 18:48:56 +0800 Subject: [PATCH 088/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=20UI=20=E9=A2=84?= =?UTF-8?q?=E8=A7=88=E9=80=82=E9=85=8D=E4=B8=8E=E7=BC=A9=E6=94=BE=E8=BE=93?= =?UTF-8?q?=E5=85=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 删除预览工具栏适配按钮、适配回调与对应快捷键 将缩放百分比改为只读显示并移除输入组件及测试 --- .../components/preview/PreviewWorkspace.tsx | 150 +++++++----------- .../preview/ZoomPercentageInput.tsx | 79 --------- .../components/preview/previewZoomKeyboard.ts | 11 +- .../tests/ZoomPercentageInput.test.tsx | 67 -------- .../tests/previewWorkspaceZoom.test.tsx | 10 +- .../tests/previewZoomKeyboard.test.ts | 10 +- 6 files changed, 63 insertions(+), 264 deletions(-) delete mode 100644 apps/ai-game-creator-shell/src/view/ui-editor/components/preview/ZoomPercentageInput.tsx delete mode 100644 apps/ai-game-creator-shell/tests/ZoomPercentageInput.test.tsx diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx index c923ca875..8df4e4adf 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx @@ -1,10 +1,12 @@ import { CANVAS_ZOOM_IN_FACTOR, CANVAS_ZOOM_OUT_FACTOR, + canvasDisplayScaleToViewportScale, type CanvasViewport, createPanDragState, type DragState, fitViewportToBounds, + formatCanvasDisplayScalePercent, moveViewportFromPan, resolveViewportFromWheel, scaleViewportFromScreenPoint, @@ -12,7 +14,6 @@ import { import { CanvasViewport as SharedCanvasViewport, CanvasWorld, - ZoomControls, } from '@genarrative/image-canvas-react'; import { Image as ImageIcon, Minus, Plus } from 'lucide-react'; import { @@ -37,7 +38,6 @@ import { } from './previewZoomKeyboard'; import { type UiEditorRenderMode, UiTreeRenderer } from './UiTreeRenderer'; import { useNodeTransformInteraction } from './useNodeTransformInteraction'; -import { ZoomPercentageInput } from './ZoomPercentageInput'; export function PreviewWorkspace({ canvas, @@ -140,26 +140,6 @@ export function PreviewWorkspace({ setViewportState(next); }, []); - const fitToCanvas = useCallback(() => { - if (!logicalSize) return; - const element = viewportElementRef.current; - const size = { - width: element?.clientWidth || 900, - height: element?.clientHeight || 640, - }; - setViewport( - fitViewportToBounds({ - bounds: { - x: 0, - y: 0, - width: logicalSize.width, - height: logicalSize.height, - }, - canvasSize: size, - }), - ); - }, [logicalSize, setViewport]); - const scaleViewportFromCenter = useCallback( (nextScale: number) => { const element = viewportElementRef.current; @@ -188,6 +168,15 @@ export function PreviewWorkspace({ scaleViewportFromCenter(viewportRef.current.scale * CANVAS_ZOOM_OUT_FACTOR); }, [scaleViewportFromCenter]); + const displayPercent = formatCanvasDisplayScalePercent(viewport.scale); + + const zoomToDisplayScale = useCallback( + (displayScale: number) => { + scaleViewportFromCenter(canvasDisplayScaleToViewportScale(displayScale)); + }, + [scaleViewportFromCenter], + ); + useEffect(() => { const element = viewportElementRef.current; if (!element) return; @@ -203,12 +192,6 @@ export function PreviewWorkspace({ return () => observer.disconnect(); }, []); - useEffect(() => { - fitToCanvas(); - // This effect intentionally follows the active image, not every controller render. - // eslint-disable-next-line react-hooks/exhaustive-deps - }, [activeImageId, fitToCanvas]); - useEffect(() => { const request = canvas.focusRequest; if ( @@ -282,7 +265,6 @@ export function PreviewWorkspace({ usesMetaModifier, }, { - fit: fitToCanvas, resetToActualSize, zoomIn, zoomOut, @@ -301,7 +283,7 @@ export function PreviewWorkspace({ window.removeEventListener('keyup', onKeyUp); window.removeEventListener('blur', onWindowBlur); }; - }, [fitToCanvas, logicalSize, resetToActualSize, zoomIn, zoomOut]); + }, [logicalSize, resetToActualSize, zoomIn, zoomOut]); const handlePointerDown = (event: ReactPointerEvent) => { if (event.button === 0 && !isPreviewZoomInteractiveTarget(event.target)) { @@ -507,70 +489,52 @@ export function PreviewWorkspace({ onDelete={(nodeId) => canvas.deleteNode(nodeId)} /> ) : null} - { + if ( + event.target instanceof Element && + event.target.closest('button') + ) { + event.preventDefault(); + } + }} > - {(actions) => ( -
{ - if ( - event.target instanceof Element && - event.target.closest('button') - ) { - event.preventDefault(); - } - }} - > - - - actions.zoomToDisplayScale(Number(event.target.value) / 100) - } - /> - - actions.zoomToDisplayScale(percent / 100) - } - /> - - -
- )} - + + + zoomToDisplayScale(Number(event.target.value) / 100) + } + /> + + {displayPercent} + + + ) : (
diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/ZoomPercentageInput.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/ZoomPercentageInput.tsx deleted file mode 100644 index 5a9ff0d78..000000000 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/ZoomPercentageInput.tsx +++ /dev/null @@ -1,79 +0,0 @@ -import { useEffect, useRef, useState } from 'react'; - -const MIN_ZOOM_PERCENT = 25; -const MAX_ZOOM_PERCENT = 200; - -function displayPercentToValue(displayPercent: string) { - return displayPercent.replace('%', ''); -} - -export function ZoomPercentageInput({ - displayPercent, - onCommit, -}: { - displayPercent: string; - onCommit: (percent: number) => void; -}) { - const [draft, setDraft] = useState(() => - displayPercentToValue(displayPercent), - ); - const draftRef = useRef(draft); - const isEditingRef = useRef(false); - - useEffect(() => { - if (!isEditingRef.current) { - setDraft(displayPercentToValue(displayPercent)); - draftRef.current = displayPercentToValue(displayPercent); - } - }, [displayPercent]); - - const commit = () => { - const currentPercent = Number.parseFloat( - displayPercentToValue(displayPercent), - ); - const parsed = Number.parseFloat(draftRef.current); - const nextPercent = Number.isFinite(parsed) - ? Math.min(MAX_ZOOM_PERCENT, Math.max(MIN_ZOOM_PERCENT, parsed)) - : currentPercent; - setDraft(String(nextPercent)); - draftRef.current = String(nextPercent); - isEditingRef.current = false; - if (nextPercent !== currentPercent) { - onCommit(nextPercent); - } - }; - - return ( - - ); -} diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/previewZoomKeyboard.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/previewZoomKeyboard.ts index 85941ca1e..ce09b1495 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/previewZoomKeyboard.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/previewZoomKeyboard.ts @@ -1,8 +1,4 @@ -export type PreviewZoomShortcut = - | 'fit' - | 'actual-size' - | 'zoom-in' - | 'zoom-out'; +export type PreviewZoomShortcut = 'actual-size' | 'zoom-in' | 'zoom-out'; export type PreviewZoomKeyboardContext = { hasZoomableViewport: boolean; @@ -12,7 +8,6 @@ export type PreviewZoomKeyboardContext = { }; export type PreviewZoomKeyboardActions = { - fit: () => void; resetToActualSize: () => void; zoomIn: () => void; zoomOut: () => void; @@ -61,7 +56,6 @@ export function resolvePreviewZoomShortcut( event: KeyboardEvent, ): PreviewZoomShortcut | null { if (event.altKey) return null; - if (event.key === '0') return 'fit'; if (event.key === '1') return 'actual-size'; if (event.code === 'NumpadAdd' || event.key === '+' || event.key === '=') { return 'zoom-in'; @@ -92,8 +86,7 @@ export function handlePreviewZoomKeyDown( event.preventDefault(); event.stopPropagation(); - if (shortcut === 'fit') actions.fit(); - else if (shortcut === 'actual-size') actions.resetToActualSize(); + if (shortcut === 'actual-size') actions.resetToActualSize(); else if (shortcut === 'zoom-in') actions.zoomIn(); else actions.zoomOut(); return true; diff --git a/apps/ai-game-creator-shell/tests/ZoomPercentageInput.test.tsx b/apps/ai-game-creator-shell/tests/ZoomPercentageInput.test.tsx deleted file mode 100644 index f02a71e3d..000000000 --- a/apps/ai-game-creator-shell/tests/ZoomPercentageInput.test.tsx +++ /dev/null @@ -1,67 +0,0 @@ -// @vitest-environment jsdom - -import { fireEvent, render, screen } from '@testing-library/react'; -import { describe, expect, it, vi } from 'vitest'; - -import { ZoomPercentageInput } from '../src/view/ui-editor/components/preview/ZoomPercentageInput'; - -describe('ZoomPercentageInput', () => { - it('edits the displayed percentage and commits on blur without fitting', () => { - const onCommit = vi.fn(); - render(); - - const input = screen.getByRole('spinbutton', { name: /画布缩放/ }); - expect((input as HTMLInputElement).value).toBe('50'); - - fireEvent.focus(input); - fireEvent.change(input, { target: { value: '125' } }); - expect(onCommit).not.toHaveBeenCalled(); - fireEvent.blur(input); - - expect(onCommit).toHaveBeenCalledWith(125); - expect((input as HTMLInputElement).value).toBe('125'); - }); - - it.each([ - { value: '0', expected: 25 }, - { value: '999', expected: 200 }, - ])('clamps $value to $expected on blur', ({ value, expected }) => { - const onCommit = vi.fn(); - render(); - - const input = screen.getByRole('spinbutton', { name: /画布缩放/ }); - fireEvent.focus(input); - fireEvent.change(input, { target: { value } }); - fireEvent.blur(input); - - expect(onCommit).toHaveBeenCalledWith(expected); - expect((input as HTMLInputElement).value).toBe(String(expected)); - }); - - it('restores the current percentage when the draft is invalid', () => { - const onCommit = vi.fn(); - render(); - - const input = screen.getByRole('spinbutton', { name: /画布缩放/ }); - fireEvent.focus(input); - fireEvent.change(input, { target: { value: '' } }); - fireEvent.blur(input); - - expect(onCommit).not.toHaveBeenCalled(); - expect((input as HTMLInputElement).value).toBe('80'); - }); - - it('tracks viewport updates while not editing', () => { - const onCommit = vi.fn(); - const view = render( - , - ); - const input = screen.getByRole('spinbutton', { name: /画布缩放/ }); - - view.rerender( - , - ); - - expect((input as HTMLInputElement).value).toBe('140'); - }); -}); diff --git a/apps/ai-game-creator-shell/tests/previewWorkspaceZoom.test.tsx b/apps/ai-game-creator-shell/tests/previewWorkspaceZoom.test.tsx index 589bc5306..1159785c3 100644 --- a/apps/ai-game-creator-shell/tests/previewWorkspaceZoom.test.tsx +++ b/apps/ai-game-creator-shell/tests/previewWorkspaceZoom.test.tsx @@ -160,10 +160,9 @@ describe('PreviewWorkspace quick zoom', () => { expect(renderedScale(rendered.container)).toBeCloseTo(initial * 0.86); }); - it('keeps actual-size and fit shortcuts within the preview scope', () => { + it('keeps the actual-size shortcut within the preview scope', () => { const rendered = render(); const preview = screen.getByRole('region', { name: 'UI 预览画布' }); - const fitted = renderedScale(rendered.container); fireEvent.focus(preview); fireEvent.keyDown(window, { @@ -172,13 +171,6 @@ describe('PreviewWorkspace quick zoom', () => { cancelable: true, }); expect(renderedScale(rendered.container)).toBe(1); - - fireEvent.keyDown(window, { - key: '0', - ctrlKey: true, - cancelable: true, - }); - expect(renderedScale(rendered.container)).toBe(fitted); }); it('leaves browser zoom untouched when the preview has no content', () => { diff --git a/apps/ai-game-creator-shell/tests/previewZoomKeyboard.test.ts b/apps/ai-game-creator-shell/tests/previewZoomKeyboard.test.ts index 5ef72b7db..519c3247f 100644 --- a/apps/ai-game-creator-shell/tests/previewZoomKeyboard.test.ts +++ b/apps/ai-game-creator-shell/tests/previewZoomKeyboard.test.ts @@ -12,7 +12,6 @@ import { function createActions(): PreviewZoomKeyboardActions { return { - fit: vi.fn(), resetToActualSize: vi.fn(), zoomIn: vi.fn(), zoomOut: vi.fn(), @@ -96,15 +95,12 @@ describe('preview zoom keyboard shortcuts', () => { expect(keyboardEvent?.defaultPrevented).toBe(false); }); - it.each([ - { key: '0', action: 'fit' as const }, - { key: '1', action: 'resetToActualSize' as const }, - ])('keeps the existing $key shortcut', ({ key, action }) => { + it('keeps the existing actual-size shortcut', () => { const { actions } = dispatchShortcut({ - event: { key, ctrlKey: true, cancelable: true }, + event: { key: '1', ctrlKey: true, cancelable: true }, }); - expect(actions[action]).toHaveBeenCalledTimes(1); + expect(actions.resetToActualSize).toHaveBeenCalledTimes(1); }); it('uses Cmd on Apple platforms and Ctrl elsewhere', () => { From 4bbc77fc32e9a4315610bd3bed645d88dfc52350 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 18:51:53 +0800 Subject: [PATCH 089/248] =?UTF-8?q?=E5=A2=9E=E5=8A=A0UI=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E8=80=97=E6=97=B6=E6=97=A5=E5=BF=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 补充 marker、area、LLM、图片编辑及 batch 阶段耗时观测 记录成功与失败路径的性能字段 同步自动切分工作流技术方案中的日志约定 --- .../src/ui_editor/commands/separation/area.rs | 37 ++++++++- .../ui_editor/commands/separation/marker.rs | 29 ++++++- .../ui_editor/commands/separation/workflow.rs | 80 +++++++++++++++++-- ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 1 + 4 files changed, 136 insertions(+), 11 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs index 68cc9ba47..739492b6c 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -1,5 +1,6 @@ use super::model::BindingArea; use image::RgbaImage; +use std::time::Instant; /// Each edge may move by at most this percentage of the corresponding area /// dimension returned by the visual model. Keep this policy explicit so @@ -215,16 +216,31 @@ pub(crate) fn normalize_binding_area( image: &RgbaImage, original_area: BindingArea, ) -> Result { - original_area.validate_in(image.width(), image.height())?; + let started = Instant::now(); + if let Err(error) = original_area.validate_in(image.width(), image.height()) { + app_log!( + "ui_separation.area.timing outcome=error elapsed_us={} rounds=0 image_width={} image_height={} area=({}, {}, {}, {})", + started.elapsed().as_micros(), + image.width(), + image.height(), + original_area.global_pos_x_px, + original_area.global_pos_y_px, + original_area.width_px, + original_area.height_px + ); + return Err(error); + } let original = Rect::from_area(original_area); let directions = Edge::ALL.map(|edge| edge_direction(image, original, edge)); let mut current = original; let mut clamped = false; let mut active = [true; 4]; + let mut rounds = 0u32; // TODO: Replace the deliberately simple pixel-by-pixel scan if real UI // design sizes show this path to be a measurable bottleneck. while active.iter().any(|value| *value) { + rounds = rounds.saturating_add(1); let before = current; let mut next = current; let mut moved = [false; 4]; @@ -277,12 +293,27 @@ pub(crate) fn normalize_binding_area( } let area = current.into_area(); - Ok(NormalizedBindingArea { + let result = Ok(NormalizedBindingArea { changed: area != original_area, area, clamped, transparent: !rect_has_visible_pixel(image, current), - }) + }); + app_log!( + "ui_separation.area.timing outcome=ok elapsed_us={} rounds={} image_width={} image_height={} area=({}, {}, {}, {}) changed={} clamped={} transparent={}", + started.elapsed().as_micros(), + rounds, + image.width(), + image.height(), + original_area.global_pos_x_px, + original_area.global_pos_y_px, + original_area.width_px, + original_area.height_px, + result.as_ref().expect("normalization result exists").changed, + result.as_ref().expect("normalization result exists").clamped, + result.as_ref().expect("normalization result exists").transparent + ); + result } #[cfg(test)] diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index 7ded9027e..1964f7311 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -2,6 +2,7 @@ use super::model::SeparationNode; use base64::Engine as _; use std::path::Path; use std::path::PathBuf; +use std::time::Instant; const MARKER_LINE_WIDTH: u32 = 2; const PURPLE_FILL: image::Rgba = image::Rgba([180, 0, 180, 120]); @@ -11,9 +12,31 @@ pub async fn build_marked_image( nodes: Vec, target: PathBuf, ) -> Result { - tokio::task::spawn_blocking(move || build_marked_image_blocking(&source_url, &nodes, &target)) - .await - .map_err(|error| format!("构建标记图任务失败：{error}"))? + let node_count = nodes.len(); + let started = Instant::now(); + let result = match tokio::task::spawn_blocking(move || { + let blocking_started = Instant::now(); + let result = build_marked_image_blocking(&source_url, &nodes, &target); + app_log!( + "ui_separation.marker.blocking_timing outcome={} elapsed_ms={} nodes={}", + if result.is_ok() { "ok" } else { "error" }, + blocking_started.elapsed().as_millis(), + node_count + ); + result + }) + .await + { + Ok(result) => result, + Err(error) => Err(format!("构建标记图任务失败：{error}")), + }; + app_log!( + "ui_separation.marker.timing outcome={} elapsed_ms={} nodes={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis(), + node_count + ); + result } fn build_marked_image_blocking( diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index a13f4dd98..771cec10a 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -18,6 +18,7 @@ use platform_llm::{ use serde::Deserialize; use std::fs; use std::path::{Path, PathBuf}; +use std::time::Instant; pub fn apply_batch_patch( state: &mut SeparationState, tree_index: usize, @@ -65,7 +66,7 @@ pub fn apply_batch_patch( } BindingDecision::NeedRework { to_node, - problem_description, + advice: problem_description, } => { append_rework_note(&mut tree.root, to_node, problem_description); let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; @@ -125,6 +126,25 @@ async fn raw_image_edit( prompt: &str, width: u32, height: u32, +) -> Result { + let started = Instant::now(); + let result = raw_image_edit_inner(session, image_data_url, prompt, width, height).await; + app_log!( + "ui_separation.image_edit.timing outcome={} elapsed_ms={} width={} height={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis(), + width, + height + ); + result +} + +async fn raw_image_edit_inner( + session: &crate::platform_session::PlatformSessionSnapshot, + image_data_url: &str, + prompt: &str, + width: u32, + height: u32, ) -> Result { app_log!( "ui_separation.image_edit.start width={} height={} prompt_chars={}", @@ -208,6 +228,17 @@ async fn raw_image_edit( } async fn write_processed_image(processed_url: String, target: PathBuf) -> Result<(), String> { + let started = Instant::now(); + let result = write_processed_image_inner(processed_url, target).await; + app_log!( + "ui_separation.processed_image.write.timing outcome={} elapsed_ms={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis() + ); + result +} + +async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> Result<(), String> { app_log!( "ui_separation.processed_image.write.start target_file={} data_url_chars={}", target @@ -247,6 +278,22 @@ async fn visual_binding( source_url: String, processed_url: String, nodes: &[&SeparationNode], +) -> Result { + let started = Instant::now(); + let result = visual_binding_inner(source_url, processed_url, nodes).await; + app_log!( + "ui_separation.visual_binding.timing outcome={} elapsed_ms={} nodes={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis(), + nodes.len() + ); + result +} + +async fn visual_binding_inner( + source_url: String, + processed_url: String, + nodes: &[&SeparationNode], ) -> Result { app_log!( "ui_separation.visual_binding.start nodes={} source_url_chars={} processed_url_chars={}", @@ -304,8 +351,14 @@ async fn visual_binding( let request = LlmRunRequest::new(history) .with_function_tools(vec![tool.clone()]) .with_tool_choice(LlmToolChoice::Required); - request_ui_editor_llm(&client, &llm_config, request) - .await + let request_started = Instant::now(); + let response = request_ui_editor_llm(&client, &llm_config, request).await; + app_log!( + "ui_separation.llm.timing outcome={} elapsed_ms={}", + if response.is_ok() { "ok" } else { "error" }, + request_started.elapsed().as_millis() + ); + response .map_err(|e| e.to_string()) .and_then(|response| { response @@ -425,6 +478,7 @@ pub(crate) async fn separate_ui_impl( })?; let mut batch_index = 0usize; loop { + let batch_started = Instant::now(); let Some(current_tree) = separation.trees.get(tree_index) else { break; }; @@ -568,12 +622,13 @@ pub(crate) async fn separate_ui_impl( apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; write_separation_state(&state_path, &separation)?; app_log!( - "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={}", + "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={} elapsed_ms={}", tree_index, batch_index, cut_paths.len(), separation.bound.len(), - separation.problematic_nodes.len() + separation.problematic_nodes.len(), + batch_started.elapsed().as_millis() ); batch_index += 1; } @@ -591,6 +646,21 @@ async fn cut_processed_image( source: PathBuf, area: BindingArea, target: PathBuf, +) -> Result<(), String> { + let started = Instant::now(); + let result = cut_processed_image_inner(source, area, target).await; + app_log!( + "ui_separation.cut_image.timing outcome={} elapsed_ms={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis() + ); + result +} + +async fn cut_processed_image_inner( + source: PathBuf, + area: BindingArea, + target: PathBuf, ) -> Result<(), String> { app_log!( "ui_separation.cut_image.start source_file={} target_file={} area=({}, {}, {}, {})", diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 5aa3921af..4f19e0e78 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -56,6 +56,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 - 父节点背景重建由 image-edit 模型完成，不由 Rust 硬编码重建算法完成。 +- 性能观测沿用 `app_log!`：marker 记录端到端与 `spawn_blocking` 耗时，area 记录 `elapsed_us` 与扫描轮数，image-edit 与 visual binding 记录整个请求耗时，处理图写入、cut 和 batch 记录阶段耗时；日志不写入 prompt、图片内容、绝对路径或模型原文。 ## 临时 sidecar From c5c7339c4a55bfef7842f83a4b76327e38ecfe53 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 18:54:10 +0800 Subject: [PATCH 090/248] =?UTF-8?q?=E9=87=8D=E5=91=BD=E5=90=8D=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E6=A8=A1=E5=9D=97=20`problem=5Fdescription`=20?= =?UTF-8?q?=E4=B8=BA=20`advice`=20=E5=B9=B6=E5=90=8C=E6=AD=A5=E6=9B=B4?= =?UTF-8?q?=E6=96=B0=E7=9B=B8=E5=85=B3=E6=A0=A1=E9=AA=8C=E9=80=BB=E8=BE=91?= =?UTF-8?q?=E4=B8=8E=E6=B5=8B=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/mod.rs | 4 ++-- .../src/ui_editor/commands/separation/model/binding.rs | 2 +- .../src-tauri/src/ui_editor/commands/separation/tree.rs | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 4dadea3a6..fa3cc23d2 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -230,7 +230,7 @@ mod tests { 0, &[BindingDecision::NeedRework { to_node: id.clone(), - problem_description: note.to_string(), + advice: note.to_string(), }], &paths, ) @@ -276,7 +276,7 @@ mod tests { }; let decision = BindingDecision::NeedRework { to_node: node.id.clone(), - problem_description: "x".repeat(MAX_REWORK_NOTE_CHARS + 1), + advice: "x".repeat(MAX_REWORK_NOTE_CHARS + 1), }; assert!(validate_binding_response( &BindingResp { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs index c21b4388d..d24f249eb 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs @@ -39,7 +39,7 @@ pub enum BindingDecision { to_node: NodeId, }, NeedRework { - problem_description: String, + advice: String, to_node: NodeId, }, } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 6cd9e2be3..6e942e260 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -252,14 +252,14 @@ pub fn validate_binding_response( return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); } if let BindingDecision::NeedRework { - problem_description, + advice, .. } = decision { - if problem_description.trim().is_empty() { + if advice.trim().is_empty() { return Err("NeedRework 必须包含问题描述".to_string()); } - if problem_description.chars().count() > MAX_REWORK_NOTE_CHARS { + if advice.chars().count() > MAX_REWORK_NOTE_CHARS { return Err(format!( "NeedRework 问题描述不能超过 {MAX_REWORK_NOTE_CHARS} 个字符" )); From b69bd88e5e410fc55fb1e5ca95d979c2c828c401 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:01:18 +0800 Subject: [PATCH 091/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=20UI=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E5=99=A8=E6=A0=A1=E9=AA=8C=E5=99=A8=E5=91=BD=E5=90=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 run_with_repair_history 的 Validater/validater 更正为 Validator/validator。 不改变重试流程和运行时行为。 --- .../src-tauri/src/ui_editor/commands/utils.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index 4f849d227..c4a51ca6b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -28,17 +28,17 @@ pub(crate) async fn request_ui_editor_llm( } /// 按 append-only history 重试结构化 LLM 请求；仅业务校验失败会追加反馈消息。 -pub(crate) async fn run_with_repair_history( +pub(crate) async fn run_with_repair_history( max_retries: usize, initial_history: Vec, requester: Requester, - validater: Validater, + validator: Validator, ) -> Result where T: Serialize, Requester: Fn(Vec) -> Fut, Fut: Future>, - Validater: Fn(&T) -> Result<(), String>, + Validator: Fn(&T) -> Result<(), String>, { let mut history = initial_history; for attempt in 0..=max_retries { @@ -47,7 +47,7 @@ where Err(_error) if attempt < max_retries => continue, Err(error) => return Err(error), }; - match validater(&value) { + match validator(&value) { Ok(()) => return Ok(value), Err(error) if attempt < max_retries => { let serialized = serde_json::to_string(&value) From ae2f5fc80b446907beb932ad282457a55be1840f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:01:32 +0800 Subject: [PATCH 092/248] =?UTF-8?q?=E8=AE=B0=E5=BD=95=20UI=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E5=99=A8=E9=87=8D=E8=AF=95=E4=B8=AD=E7=9A=84=E8=AF=B7?= =?UTF-8?q?=E6=B1=82=E9=94=99=E8=AF=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在结构化 LLM 请求的中间重试失败时写入尝试次数和错误日志。 保留原有重试次数、历史消息和最终错误语义。 --- .../src-tauri/src/ui_editor/commands/utils.rs | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index c4a51ca6b..84f7fa178 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -44,7 +44,15 @@ where for attempt in 0..=max_retries { let value = match requester(history.clone()).await { Ok(value) => value, - Err(_error) if attempt < max_retries => continue, + Err(error) if attempt < max_retries => { + app_log!( + "ui_editor.llm.retry request_error attempt={} max_retries={} error={}", + attempt + 1, + max_retries, + error + ); + continue; + } Err(error) => return Err(error), }; match validator(&value) { From f200cb4d3c9af99dc383fbabfb688d051f9f2093 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:02:09 +0800 Subject: [PATCH 093/248] =?UTF-8?q?=E9=81=BF=E5=85=8D=E6=A0=87=E8=AE=B0?= =?UTF-8?q?=E5=9B=BE=E7=BC=96=E7=A0=81=E9=87=8D=E5=A4=8D=E5=A4=8D=E5=88=B6?= =?UTF-8?q?=E5=83=8F=E7=B4=A0=E7=BC=93=E5=86=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复用同一个 DynamicImage 完成文件保存和返回数据编码。 保持标记图内容和错误处理不变。 --- .../src-tauri/src/ui_editor/commands/separation/marker.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index 1964f7311..771d1f352 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -94,11 +94,12 @@ fn build_marked_image_blocking( height, ); } - image::DynamicImage::ImageRgba8(image.clone()) + let image = image::DynamicImage::ImageRgba8(image); + image .save_with_format(target, image::ImageFormat::Png) .map_err(|e| format!("写入标记图失败：{e}"))?; let mut png = Vec::new(); - image::DynamicImage::ImageRgba8(image) + image .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) .map_err(|e| format!("编码标记图失败：{e}"))?; Ok(format!( From 5b441d389c4ca9a37f097556acd378bc563bf876 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:02:40 +0800 Subject: [PATCH 094/248] =?UTF-8?q?=E5=BF=BD=E7=95=A5=E5=AE=8C=E5=85=A8?= =?UTF-8?q?=E4=BD=8D=E4=BA=8E=E5=9B=BE=E7=89=87=E5=A4=96=E7=9A=84=E5=88=87?= =?UTF-8?q?=E5=88=86=E6=A0=87=E8=AE=B0=E7=9F=A9=E5=BD=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 矩形起点超出图片边界时直接返回 None，避免在边缘绘制伪影。 新增越界矩形回归测试。 --- .../src/ui_editor/commands/separation/marker.rs | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs index 771d1f352..117c266b7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs @@ -119,8 +119,11 @@ fn clipped_rect( if width == 0 || height == 0 || w == 0 || h == 0 { return None; } - let x0 = x.min(width - 1); - let y0 = y.min(height - 1); + if x >= width || y >= height { + return None; + } + let x0 = x; + let y0 = y; let x1 = x.saturating_add(w).min(width).saturating_sub(1); let y1 = y.saturating_add(h).min(height).saturating_sub(1); (x0 <= x1 && y0 <= y1).then_some((x0, y0, x1, y1)) @@ -301,6 +304,12 @@ mod tests { assert_eq!(*image.get_pixel(1, 1), image::Rgba([1, 2, 3, 255])); } + #[test] + fn clipped_rect_ignores_rectangles_starting_outside_image() { + assert_eq!(clipped_rect(4, 0, 1, 1, 4, 4), None); + assert_eq!(clipped_rect(0, 4, 1, 1, 4, 4), None); + } + #[test] fn text_mask_is_purple_before_green_frame() { let mut image = image::RgbaImage::from_pixel(8, 8, image::Rgba([1, 2, 3, 255])); From 1eb4bd5ec0c89a3043a5b6325b7787f8f4a483e5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:03:29 +0800 Subject: [PATCH 095/248] =?UTF-8?q?=E6=98=8E=E7=A1=AE=E6=8B=92=E7=BB=9D?= =?UTF-8?q?=E6=97=A0=E6=95=88=E7=9A=84=E5=A4=84=E7=90=86=E5=9B=BE=E6=95=B0?= =?UTF-8?q?=E6=8D=AE=E5=9C=B0=E5=9D=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 处理图缺少 data URL 分隔符时直接返回格式错误。 避免先写入空文件再在后续裁切阶段暴露误导性错误。 --- .../src/ui_editor/commands/separation/workflow.rs | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index 771cec10a..da0c9e6b0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -248,13 +248,12 @@ async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> processed_url.chars().count() ); tokio::task::spawn_blocking(move || { + let encoded = processed_url + .split_once(',') + .map(|(_, data)| data) + .ok_or_else(|| "处理图 data URL 无效".to_string())?; let processed_bytes = base64::engine::general_purpose::STANDARD - .decode( - processed_url - .split_once(',') - .map(|(_, data)| data) - .unwrap_or_default(), - ) + .decode(encoded) .map_err(|error| format!("解析处理图失败：{error}"))?; let byte_len = processed_bytes.len(); fs::write(&target, processed_bytes) From a50cbfb6c895a451903228a753d9145b2b5e7865 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:04:40 +0800 Subject: [PATCH 096/248] =?UTF-8?q?=E8=BF=87=E6=BB=A4=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E7=BC=96=E8=BE=91=E5=93=8D=E5=BA=94=E4=B8=AD=E7=9A=84=E7=A9=BA?= =?UTF-8?q?=E5=9B=BE=E5=83=8F=E6=95=B0=E6=8D=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 忽略上游返回的空 b64_json 条目，避免把空字符串转发为图像结果。 新增空条目与有效条目混合响应的回归测试。 --- .../src/vector_engine/raw_edit.rs | 23 +++++++++++++++---- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index d215adbbc..984d37add 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -264,11 +264,7 @@ pub async fn create_vector_engine_raw_image_edit( }); } }; - let b64_images = payload - .data - .into_iter() - .map(|entry| entry.b64_json) - .collect::>(); + let b64_images = collect_b64_images(payload.data); if b64_images.is_empty() { let message = format!("{failure_context}：上游未返回 b64_json 图片"); let audit = build_failure_audit( @@ -299,6 +295,13 @@ pub async fn create_vector_engine_raw_image_edit( }) } +fn collect_b64_images(data: Vec) -> Vec { + data.into_iter() + .map(|entry| entry.b64_json) + .filter(|value| !value.is_empty()) + .collect() +} + fn invalid_request(context: &str, message: String) -> PlatformImageError { PlatformImageError::InvalidRequest { provider: VECTOR_ENGINE_PROVIDER, @@ -415,6 +418,16 @@ mod tests { assert!(serde_json::from_str::(r#"{"data":[{}]}"#).is_err()); } + #[test] + fn raw_success_response_discards_empty_b64_json_entries() { + let payload: RawImageEditResponsePayload = serde_json::from_str( + r#"{"data":[{"b64_json":""},{"b64_json":"valid"}]}"#, + ) + .expect("response envelope"); + + assert_eq!(collect_b64_images(payload.data), vec!["valid"]); + } + #[test] fn raw_image_edit_dimensions_enforce_strict_contract() { assert!(validate_raw_image_edit_dimensions(1024, 640).is_ok()); From 4e1064266f4d4fc2b4aaade7661c28dc56dc1574 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:05:48 +0800 Subject: [PATCH 097/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E7=BB=93=E6=9E=9C=E7=9A=84=E5=86=97=E4=BD=99?= =?UTF-8?q?=E7=B1=BB=E5=9E=8B=E6=96=AD=E8=A8=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 依赖空值检查后的 TypeScript 类型收窄直接使用 SeparationDTO。 不改变自动切分流程或运行时行为。 --- .../ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 735a02af7..39726526d 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1185,7 +1185,7 @@ export function useUiEditorSession( })); if (separationResult === null) throw new Error('自动切分素材没有返回结果'); - const completedResult = separationResult as SeparationDTO; + const completedResult = separationResult; if (!(await save())) { throw new Error( '自动切分素材结果已写入编辑器，但 State 保存失败；sidecar 已保留，可继续恢复。', From 462caa33c545e889c5a8d76eb25e57ac069b8524 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:06:20 +0800 Subject: [PATCH 098/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E5=B7=A5=E4=BD=9C?= =?UTF-8?q?=E6=B5=81=E5=AE=8C=E6=88=90=E5=BC=B9=E7=AA=97=E7=9A=84=E5=A4=9A?= =?UTF-8?q?=E8=A1=8C=E9=94=99=E8=AF=AF=E5=B1=95=E7=A4=BA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 让完成通知中的换行字符按实际换行渲染，避免多项回填错误挤成一行。 保持通知内容和弹窗交互不变。 --- .../src/view/ui-editor/components/WorkflowCompletionModal.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowCompletionModal.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowCompletionModal.tsx index fbda3f214..c96e8d670 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowCompletionModal.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowCompletionModal.tsx @@ -23,7 +23,7 @@ export function WorkflowCompletionModal({ {stepLabel} {outcomeLabel} -

+

{notice.message}

From b8d8072267809c6c3c32e6d69420f5ca5d626eb0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Thu, 10 Sep 2026 19:49:41 +0800 Subject: [PATCH 099/248] =?UTF-8?q?=E5=AE=8C=E5=96=84=E8=AF=86=E5=88=AB?= =?UTF-8?q?=E5=91=BD=E4=BB=A4=E8=AF=B4=E6=98=8E=EF=BC=8C=E6=96=B0=E5=A2=9E?= =?UTF-8?q?=E7=A6=81=E6=AD=A2=E5=86=97=E4=BD=99=E8=83=8C=E6=99=AF=E8=8A=82?= =?UTF-8?q?=E7=82=B9=E7=9A=84=E6=8C=87=E5=BC=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/recognition.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index c95a4c781..15e71f2aa 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -50,7 +50,8 @@ const SYSTEM_PROMPT: &str = r#" 目前我们只做识别, 不要求图片字体参数. 每个节点最多返回一个 component；需要多个视觉层时拆成多个节点。 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. -* 不鼓励兄弟节点相互重叠, 对于背景等元素, 请promote为父节点的组件, 不要再建单独的兄弟节点承载. +* 不鼓励兄弟节点相互重叠. +* 对于面板等容器的背景等, 必须作为父节点的组件, 禁止新增冗余的所谓"背景节点". "#; #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] From 95ed49f2643be0244e3faecaa7c2de4be064aab8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 10:28:32 +0800 Subject: [PATCH 100/248] =?UTF-8?q?=E8=B0=83=E6=95=B4=E7=B4=A0=E6=9D=90?= =?UTF-8?q?=E5=88=87=E5=88=86=E8=BE=B9=E7=95=8C=E5=83=8F=E7=B4=A0=E4=B8=8A?= =?UTF-8?q?=E9=99=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将视觉绑定区域每条边的调整上限改为固定 32px。 同步 UI 编辑器自动切分技术方案与项目决策记录。 更新边界测试以覆盖绝对像素限制。 --- .../src/ui_editor/commands/separation/area.rs | 31 +++++++------------ .../shared-memory/decision-log.md | 4 +++ ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 4 +-- 3 files changed, 17 insertions(+), 22 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs index 739492b6c..30dd4e4cf 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -2,11 +2,10 @@ use super::model::BindingArea; use image::RgbaImage; use std::time::Instant; -/// Each edge may move by at most this percentage of the corresponding area -/// dimension returned by the visual model. Keep this policy explicit so -/// changing it is an intentional workflow decision rather than a scattered -/// numeric literal. -pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT: u32 = 100; +/// Each edge may move by at most this many pixels from the area returned by +/// the visual model. Keep this policy explicit so changing it is an +/// intentional workflow decision rather than a scattered numeric literal. +pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX: u32 = 32; #[derive(Clone, Copy, Debug, PartialEq)] pub(crate) struct NormalizedBindingArea { @@ -87,11 +86,6 @@ fn rect_has_visible_pixel(image: &RgbaImage, rect: Rect) -> bool { (rect.top..rect.bottom).any(|y| (rect.left..rect.right).any(|x| image.get_pixel(x, y).0[3] > 0)) } -fn max_edge_adjustment(dimension: u32) -> u32 { - ((u64::from(dimension) * u64::from(MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT)) / 100) - .min(u64::from(u32::MAX)) as u32 -} - fn edge_direction(image: &RgbaImage, rect: Rect, edge: Edge) -> EdgeDirection { if edge_has_visible_pixel(image, rect, edge) { EdgeDirection::Outward @@ -126,15 +120,12 @@ fn edge_displacement(original: Rect, current: Rect, edge: Edge) -> u32 { edge_coordinate(original, edge).abs_diff(edge_coordinate(current, edge)) } -fn edge_adjustment_limit(original: Rect, edge: Edge) -> u32 { - max_edge_adjustment(match edge { - Edge::Left | Edge::Right => original.right - original.left, - Edge::Top | Edge::Bottom => original.bottom - original.top, - }) +fn edge_adjustment_limit() -> u32 { + MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX } fn reached_adjustment_limit(original: Rect, current: Rect, edge: Edge) -> bool { - edge_displacement(original, current, edge) >= edge_adjustment_limit(original, edge) + edge_displacement(original, current, edge) >= edge_adjustment_limit() } fn can_move_geometrically( @@ -393,10 +384,10 @@ mod tests { } #[test] - fn caps_each_edge_at_original_dimension() { - let image = image_with_rect(64, 64, 0, 0, 64, 64); - let result = normalize_binding_area(&image, area(16, 16, 8, 8)).unwrap(); - assert_eq!(result.area, area(8, 8, 24, 24)); + fn caps_each_edge_at_absolute_pixel_limit() { + let image = image_with_rect(128, 128, 0, 0, 128, 128); + let result = normalize_binding_area(&image, area(48, 48, 8, 8)).unwrap(); + assert_eq!(result.area, area(16, 16, 72, 72)); assert!(result.clamped); } diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 5e45daf45..2d6cae156 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -8040,6 +8040,10 @@ CI 上 `background_agent_runtime_recovers_stale_running_before_pending_task` 在 - `validater(&T) -> Result<(), String>` 只负责业务校验。网络、模型、tool 缺失、JSON 或反序列化错误只按原 history 重试；只有业务校验失败才把序列化后的响应和校验错误合并为一条 system message 追加到 history。 - history 仅存在本次请求内存中，不重复图片、不截断、不扩展 `platform-llm` 消息协议；重试次数参数统一使用 `max_retries`。 +## 2026-09-11 UI 编辑器素材切分边界使用固定像素上限 + +- UI 编辑器自动切分在 cut 前对视觉模型返回的 `BindingArea` 做像素边界归一化时，每条边相对原始区域最多移动 `32px`，不再按原始区域宽高的百分比计算；Rust 常量为 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX`，技术方案同步记录该固定上限。 + ## 2026-08-29 DirectProject 受控联网搜索默认与边界 - 正式产品本次只覆盖 `DirectProject` 单 Codex Agent。`Provider`、`ToolHost`、`DirectHome` 不是 Agent，也不是本次联网主链路；不新增全路由联网或工具桥。唯一受控联网工具为 `agc_tools.agc_web_search`，链路固定为 Codex MCP 工具目录 -> 客户端 loopback `DirectToolBridge` -> 有界 Bing RSS HTTPS -> 过滤 / 脱敏 -> MCP 结果回传。 diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 4f19e0e78..7bce4bfc5 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -1,6 +1,6 @@ # UI 编辑器自动切分素材工作流 -更新时间：`2026-09-08` +更新时间：`2026-09-11` ## 目标 @@ -51,7 +51,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 -- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的位移上限。该上限由模块级常量 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PERCENT` 定义。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到该常量上限、图像边界或非零尺寸约束。性能优化列 TODO。 +- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的位移上限。每条边最多相对原始 area 移动 `32px`，由模块级常量 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX` 定义，与原始 area 尺寸无关。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到该常量上限、图像边界或非零尺寸约束。性能优化列 TODO。 - `NeedRework` 携带短问题描述（最多 512 个 Unicode 字符）；通过校验后按产生顺序追加到目标 `SeparationNode.note.rework_notes`，下一次该节点进入 image-edit 时全部意见会注入提取 prompt。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 - 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 From 5c83142fecbd4b10d2722474aa0052a66375dce6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 11:01:55 +0800 Subject: [PATCH 101/248] =?UTF-8?q?=E6=9B=B4=E6=96=B0UI=E7=BC=96=E8=BE=91?= =?UTF-8?q?=E5=99=A8=E8=87=AA=E5=8A=A8=E5=88=87=E5=88=86=E7=B4=A0=E6=9D=90?= =?UTF-8?q?=E5=B7=A5=E4=BD=9C=E6=B5=81=E6=96=B9=E6=A1=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将多轮整图分层与DFS面积批次写入权威专题 明确TextRemovalOnly、视觉绑定和atlas输出合同 删除平行专题文档并保留后续TODO 补充UI编辑器自动分离多轮分层方案 新增整图语义分层与多轮 image-edit 方案 明确 DFS 批次、TextRemovalOnly 和视觉绑定合同 记录面积预算、恢复边界与后续 TODO --- ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 44 ++++++++++--------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 7bce4bfc5..76e9864d4 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -4,12 +4,12 @@ ## 目标 -将 UI 编辑器现有“用户先提供独立图片/图标，再执行组件绑定”的入口替换为自动切分素材：结构识别阶段直接返回可渲染组件草稿，切分阶段按整页叶节点批次调用图片编辑模型，再由视觉模型确认处理图中的区域与目标节点。 +将 UI 编辑器现有“用户先提供独立图片/图标，再执行组件绑定”的入口替换为自动切分素材：结构识别阶段返回完整 UI 树和组件语义，切分阶段让 image-edit 模型按整页分层说明生成单张透明 atlas，再由视觉模型确认 atlas 中图片区域与目标节点的对应关系。 ## 识别结果 - `recognize` 返回完整 `Node.component` 草稿，不再要求用户先导入独立素材。 -- `component = null` 表示纯节点。 +- `NodeComponent::PureNode` 表示纯节点；`NodeComponent::WithComponent` 携带完整组件。 - `ImageComponent.target_graphic = None` 表示图片组件等待分离结果回填；它不是“明确没有图片”。 - 一个 Node 最多承载一个 `Component`；需要多个视觉层时使用多个 Node 表达。 - 组件草稿直接保存在正式 UI Node 中；临时 separation tree 不复制组件。 @@ -26,37 +26,36 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## Separation tree - recognition 完成后由 UI tree 构造临时 separation tree；每棵树保留原 UI tree 的 `src_ui_design` 与真实 `root`，不生成 synthetic root。 -- 非 root 的纯容器、纯 Text 节点和不需要切图的节点在构造时过滤，被过滤节点的 children 向上透传。真实 root 始终保留；`root_extractable` 表示 root 是否含未绑定图片组件并可作为候选。 +- 非 root 的纯容器和已绑定图片节点在构造时过滤并透传 children；真实 root 始终保留，`root_extractable` 表示 root 是否含未绑定图片组件并可作为候选。 - 节点的 `children` 在整个 workflow 中始终保留，不能因处理成功或失败而从树上删除。节点终态由 `bound`、`problematic_nodes` 反查；两者均不存在时仍待处理，`rework_count` 仅记录视觉模型返工次数。 -- 逻辑叶必须是未终止、可处理且所有 children 都已终止的节点；root 在 `root_extractable=true` 时按普通节点参与，否则只递归其 children。 -- 候选按 DFS 和 children 原顺序遍历，使用简单贪心选择与已选矩形无正面积交集的节点组成 batch；边或角接触不算重叠，不做面积或偏差排序检查。正常树结构下有候选时至少选中一个。 -- 一个 batch 是当前树中整批互不重叠的逻辑叶节点。 -- 一个 batch 的最小处理单元是：一次 image-edit + 一次 visual binding。 -- batch 成功后只把结果追加到 bound 容器，失败节点在达到返工上限后追加到 problematic 容器；树拓扑不变，流程继续消费剩余树。 +- `SeparationNodeKind::ImageTarget` 表示可以产出 Sprite 的未绑定 Image；`SeparationNodeKind::TextRemovalOnly` 表示普通 Text 的层级和矩形上下文。Text 不进入目标 batch、visual binding、bound 或 problematic。 +- 候选按前序 DFS 和 children 原顺序遍历。root 在 `root_extractable=true` 时按普通 Image 目标参与，否则只递归其 children；已终止 Image 跳过自身但继续访问 descendants。 +- 一个 batch 是 DFS 顺序上的连续 Image 目标片段，不做 overlap 筛选、排序或面积抵消。按源矩形 `width_px × height_px` 使用 `u64` 累加，正常上限为 `2880 × 2880 × IMAGE_EDIT_AREA_UTILIZATION`；利用率是命名常量，当前为 `0.8`。加入下一个目标会超限时停止；若当前为空则强制加入第一个 Image 目标以保证进度。root/超大图片的专门 atlas 策略列 TODO。 +- parent 与 child 可以同批；若 child 因面积预算未入批，仍作为 prompt 上下文，要求 parent 背景一并移除该 child，下一批再输出 child。 +- 一个 batch 的最小处理单元是：一次 image-edit + 一次 visual binding。batch 成功后只把 Image 结果追加到 bound，NeedRework 节点在达到返工上限后追加到 problematic；树拓扑不变，流程继续消费剩余树。 - 不额外维护节点状态枚举；节点是否仍在 pending tree、`rework_count` 和 problematic 容器共同表达状态。 -### 普通文字遮罩 +### 普通文字 -- 普通 `Text` 仍是正式 UI tree 中的独立 UI 元素，不进入 separation tree，也不参与 batch、绿色框、visual binding 或 bound 结果。 -- 构造 separation tree 时，把 Text 节点的布局矩形转换为页面像素坐标，挂到最近的未绑定图片节点（`ImageComponent.target_graphic == None`）的 `text_mask_areas`。纯容器只透传；嵌套图片下归最近图片；没有可切图片祖先的 Text 直接忽略。 -- `text_mask_areas` 只保存 `global_pos_x_px`、`global_pos_y_px`、`width_px`、`height_px`。不保存 NodeId、父节点、文字内容、字体样式，也不做 OCR、字形估算、偏差检查、合并或去重。 -- marker 阶段在 image-edit 前把当前 batch 节点自身的文字矩形填充为紫色；它与子图片区域一起绘制，绿色框随后绘制并位于最上层。文字遮罩不递归读取后代节点的 mask。 -- mask 仅是 image-edit 输入标记，未对 image-edit 残留文字增加 OCR 或视觉复核；正式文字语义仍由 `TextComponent` 保持。 +- 普通 `Text` 仍是正式 UI tree 中的独立 UI 元素，并作为 `TextRemovalOnly` 节点保留在 separation tree；使用现有 name/description 提供语义，不新增 OCR 或文字内容字段。 +- Text 只进入完整页面上下文，不进入 Image 目标集合，也不传给 visual binding。image-edit 只需从所属父图片/背景层中移除并重建普通文字，不生成文字 Sprite。 +- Text 的 NodeId、矩形和父子关系必须保留，使 parent/child 分层 prompt 能说明文字所在层；Text 不计入 batch 面积或目标数量，也不阻塞树完成。 ## 图片编辑与视觉绑定 -- image-edit 使用源 UI design 图片及由 Rust 生成的绿色标记/紫色重建输入。处理父节点时，紫色填充其 children 的矩形区域（包括已 problematic 的 children），再在父节点自身外围绘制绿色框和角到角的绿色交叉线；绿色标记覆盖在紫色之上。叶节点只绘制绿色框和角到角的绿色交叉线，不填充自身。 +- image-edit 直接使用原始 UI design PNG，不再生成或发送绿色框、紫色填充等 marker 图。Rust 保留现有 extraction prompt，并在其末尾追加由当前 separation state 生成的完整页面分层清单和 batch 状态；本次实现不改写既有 prompt 文案，由维护者手工整合 marker 旧句子。 +- 追加清单区分本轮 Image 输出目标、已完成 Image、仅作父子/遮挡上下文的 Image，以及只需从父图片移除的 Text。清单使用人类可读的编号、name/description、位置和层级，不向 image-edit 暴露 opaque NodeId。 +- 由于 raw endpoint 每次只返回一张 PNG，prompt 要求 image-edit 输出透明 atlas：本轮图片层可以移动和缩放，放置在不会互相遮挡的位置；视觉模型返回每层在 processed 图中的实际区域。源节点矩形只用于语义定位，不用于裁切区域推断。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 -- 标记图构建、处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 - `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 -- 视觉 binding 输入源图与处理图，必须为当前 batch 每个节点恰好返回一次 `Ok` 或 `NeedRework`。 +- 处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 +- 视觉 binding 输入源图与处理图，只接收当前 batch 的 Image targets，必须为每个 Image target 恰好返回一次 `Ok` 或 `NeedRework`。每个决定继续携带 `to_node: NodeId`；Text 不出现在请求或 schema 中。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 - cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的位移上限。每条边最多相对原始 area 移动 `32px`，由模块级常量 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX` 定义，与原始 area 尺寸无关。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到该常量上限、图像边界或非零尺寸约束。性能优化列 TODO。 - `NeedRework` 携带短问题描述（最多 512 个 Unicode 字符）；通过校验后按产生顺序追加到目标 `SeparationNode.note.rework_notes`，下一次该节点进入 image-edit 时全部意见会注入提取 prompt。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 - 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 - 父节点背景重建由 image-edit 模型完成，不由 Rust 硬编码重建算法完成。 -- 性能观测沿用 `app_log!`：marker 记录端到端与 `spawn_blocking` 耗时，area 记录 `elapsed_us` 与扫描轮数，image-edit 与 visual binding 记录整个请求耗时，处理图写入、cut 和 batch 记录阶段耗时；日志不写入 prompt、图片内容、绝对路径或模型原文。 +- 性能观测沿用 `app_log!`：image-edit 与 visual binding 记录整个请求耗时，处理图写入、cut 和 batch 记录阶段耗时；日志不写入 prompt、图片内容、绝对路径或模型原文。 ## 临时 sidecar @@ -64,10 +63,10 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - sidecar 目录按 UI manifest `asset_id` 生成，复用 `generated_file_stem(asset_id)` 的安全字符替换和 SHA-256 摘要规则，位于项目 `ui/` 下。 - 目录只保存一份当前 separation state，而不是每 batch 一个状态文件。 - state 文件只保留 `schema_version`、separation trees、bound 结果和 problematic 节点，不重复保存 `projectId / assetId / uiStateRevision`。 -- `SeparationNode.text_mask_areas` 是 separation tree 的必选字段，当前开发阶段继续使用 `ui-editor-separation-state.v1`，不提供旧 sidecar 迁移或回退。 +- `SeparationNode.kind` 与 tree children 一起持久化；当前数据结构变更提升 separation state schema 版本，不提供旧 sidecar 迁移或回退。 - sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 - 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 -- 临时图片可跨重启保留。raw image-edit 返回图、绿色/紫色标记图、处理图和 cut 图片当前都保留用于 debug；理论上只应在内存中，清理/归档策略列 TODO。 +- 临时图片可跨重启保留。raw image-edit 返回图、处理图和 cut 图片当前都保留用于 debug；理论上 raw/processed 中间图只应在内存中，清理/归档策略列 TODO。 - 并发边界：当前由前端 `isSeparating` 与 `runWithStateLocked` 保证同一 UI 编辑会话 同时只有一次 separation。sidecar 是临时恢复状态，不是正式 UI 资产真相，不参与 manifest 或项目 revision，因此当前不额外持有项目写锁；若未来支持多窗口/多进程并发， @@ -112,3 +111,6 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - problematic 对更高层 workflow 完成门禁的最终定义。 - separation workflow 与 manifest/stage 的接入。 - Raw GPT Image 2 后端 raw operation 持久状态及恢复 worker。 +- 完整页面上下文对 image-edit 效果的人工评估。 +- raw prompt 上限从 4 KiB 扩展到 16 KiB 的代理合同与预算。 +- root/超大图片超过 atlas 面积预算时的专门策略。 From 13701d01441e3ff072fba52a55e590cb1ddb34d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 11:52:21 +0800 Subject: [PATCH 102/248] =?UTF-8?q?=E6=BE=84=E6=B8=85UI=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E7=9A=84=E5=8E=9F=E5=9B=BE=E8=BE=93=E5=85=A5?= =?UTF-8?q?=E4=B8=8E=E8=B0=83=E8=AF=95=E4=BA=A7=E7=89=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 明确图片编辑直接使用原始界面图，不再生成辅助输入图 明确处理图与裁切图使用随机文件名并保留用于调试 --- .../【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 76e9864d4..28a48888e 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -43,7 +43,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## 图片编辑与视觉绑定 -- image-edit 直接使用原始 UI design PNG，不再生成或发送绿色框、紫色填充等 marker 图。Rust 保留现有 extraction prompt，并在其末尾追加由当前 separation state 生成的完整页面分层清单和 batch 状态；本次实现不改写既有 prompt 文案，由维护者手工整合 marker 旧句子。 +- image-edit 直接使用原始 UI design PNG，不再生成或发送绿色框、紫色填充等额外辅助输入图。提取 prompt 直接描述完整页面分层清单和当前 batch 状态。 - 追加清单区分本轮 Image 输出目标、已完成 Image、仅作父子/遮挡上下文的 Image，以及只需从父图片移除的 Text。清单使用人类可读的编号、name/description、位置和层级，不向 image-edit 暴露 opaque NodeId。 - 由于 raw endpoint 每次只返回一张 PNG，prompt 要求 image-edit 输出透明 atlas：本轮图片层可以移动和缩放，放置在不会互相遮挡的位置；视觉模型返回每层在 processed 图中的实际区域。源节点矩形只用于语义定位，不用于裁切区域推断。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 @@ -66,7 +66,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - `SeparationNode.kind` 与 tree children 一起持久化；当前数据结构变更提升 separation state schema 版本，不提供旧 sidecar 迁移或回退。 - sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 - 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 -- 临时图片可跨重启保留。raw image-edit 返回图、处理图和 cut 图片当前都保留用于 debug；理论上 raw/processed 中间图只应在内存中，清理/归档策略列 TODO。 +- 临时图片可跨重启保留。image-edit 返回的 processed 图和 cut 图片当前都保留用于 debug；理论上 processed 中间图只应在内存中，清理/归档策略列 TODO。 - 并发边界：当前由前端 `isSeparating` 与 `runWithStateLocked` 保证同一 UI 编辑会话 同时只有一次 separation。sidecar 是临时恢复状态，不是正式 UI 资产真相，不参与 manifest 或项目 revision，因此当前不额外持有项目写锁；若未来支持多窗口/多进程并发， From 1db75971414f2b82a28aec181f3c5699f2e872b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 12:07:19 +0800 Subject: [PATCH 103/248] =?UTF-8?q?=E9=87=8D=E6=9E=84UI=E7=B4=A0=E6=9D=90?= =?UTF-8?q?=E5=88=87=E5=88=86=E6=A0=91=E4=B8=8E=E6=89=B9=E6=AC=A1=E9=80=89?= =?UTF-8?q?=E6=8B=A9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 移除文本遮罩和标记图生成流程，图片编辑直接读取原始界面图 引入节点类型并保留文字移除上下文，按前序DFS与面积预算选择图片批次 补充根节点、重叠节点、文字过滤和超面积推进测试并同步TypeScript类型 --- .../ui_editor/commands/separation/marker.rs | 364 ------------------ .../src/ui_editor/commands/separation/mod.rs | 161 +++++++- .../commands/separation/model/mod.rs | 7 +- .../commands/separation/model/node.rs | 13 +- .../src/ui_editor/commands/separation/tree.rs | 155 +++----- .../ui_editor/commands/separation/workflow.rs | 35 +- .../ui-editor/types/SeparationNode.ts | 4 +- .../ui-editor/types/SeparationNodeKind.ts | 3 + .../features/ui-editor/types/TextMaskArea.ts | 3 - 9 files changed, 231 insertions(+), 514 deletions(-) delete mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs create mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNodeKind.ts delete mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs deleted file mode 100644 index 117c266b7..000000000 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/marker.rs +++ /dev/null @@ -1,364 +0,0 @@ -use super::model::SeparationNode; -use base64::Engine as _; -use std::path::Path; -use std::path::PathBuf; -use std::time::Instant; - -const MARKER_LINE_WIDTH: u32 = 2; -const PURPLE_FILL: image::Rgba = image::Rgba([180, 0, 180, 120]); - -pub async fn build_marked_image( - source_url: String, - nodes: Vec, - target: PathBuf, -) -> Result { - let node_count = nodes.len(); - let started = Instant::now(); - let result = match tokio::task::spawn_blocking(move || { - let blocking_started = Instant::now(); - let result = build_marked_image_blocking(&source_url, &nodes, &target); - app_log!( - "ui_separation.marker.blocking_timing outcome={} elapsed_ms={} nodes={}", - if result.is_ok() { "ok" } else { "error" }, - blocking_started.elapsed().as_millis(), - node_count - ); - result - }) - .await - { - Ok(result) => result, - Err(error) => Err(format!("构建标记图任务失败：{error}")), - }; - app_log!( - "ui_separation.marker.timing outcome={} elapsed_ms={} nodes={}", - if result.is_ok() { "ok" } else { "error" }, - started.elapsed().as_millis(), - node_count - ); - result -} - -fn build_marked_image_blocking( - source_url: &str, - nodes: &[SeparationNode], - target: &Path, -) -> Result { - let encoded = source_url - .split_once(',') - .map(|(_, d)| d) - .ok_or_else(|| "源图 data URL 无效".to_string())?; - let bytes = base64::engine::general_purpose::STANDARD - .decode(encoded) - .map_err(|e| format!("解码源图失败：{e}"))?; - let mut image = image::load_from_memory(&bytes) - .map_err(|e| format!("读取源图失败：{e}"))? - .to_rgba8(); - let width = image.width(); - let height = image.height(); - // 先填充紫色重建区域，随后绘制绿色框，确保绿色框位于最上层。 - for node in nodes { - for child in &node.children { - fill_rect( - &mut image, - child.global_pos_x_px, - child.global_pos_y_px, - child.width_px, - child.height_px, - PURPLE_FILL, - width, - height, - ); - } - for mask in &node.text_mask_areas { - fill_rect( - &mut image, - mask.global_pos_x_px, - mask.global_pos_y_px, - mask.width_px, - mask.height_px, - PURPLE_FILL, - width, - height, - ); - } - } - for node in nodes { - draw_frame( - &mut image, - node.global_pos_x_px, - node.global_pos_y_px, - node.width_px, - node.height_px, - width, - height, - ); - } - let image = image::DynamicImage::ImageRgba8(image); - image - .save_with_format(target, image::ImageFormat::Png) - .map_err(|e| format!("写入标记图失败：{e}"))?; - let mut png = Vec::new(); - image - .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) - .map_err(|e| format!("编码标记图失败：{e}"))?; - Ok(format!( - "data:image/png;base64,{}", - base64::engine::general_purpose::STANDARD.encode(png) - )) -} - -fn clipped_rect( - x: u32, - y: u32, - w: u32, - h: u32, - width: u32, - height: u32, -) -> Option<(u32, u32, u32, u32)> { - if width == 0 || height == 0 || w == 0 || h == 0 { - return None; - } - if x >= width || y >= height { - return None; - } - let x0 = x; - let y0 = y; - let x1 = x.saturating_add(w).min(width).saturating_sub(1); - let y1 = y.saturating_add(h).min(height).saturating_sub(1); - (x0 <= x1 && y0 <= y1).then_some((x0, y0, x1, y1)) -} - -fn fill_rect( - image: &mut image::RgbaImage, - x: u32, - y: u32, - w: u32, - h: u32, - color: image::Rgba, - width: u32, - height: u32, -) { - let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { - return; - }; - for yy in y0..=y1 { - for xx in x0..=x1 { - image.put_pixel(xx, yy, color); - } - } -} - -fn draw_frame( - image: &mut image::RgbaImage, - x: u32, - y: u32, - w: u32, - h: u32, - width: u32, - height: u32, -) { - let Some((x0, y0, x1, y1)) = clipped_rect(x, y, w, h, width, height) else { - return; - }; - let green = image::Rgba([0, 255, 0, 255]); - draw_line( - image, - (x0, y0), - (x1, y0), - green, - MARKER_LINE_WIDTH, - (x0, y0, x1, y1), - ); - draw_line( - image, - (x0, y1), - (x1, y1), - green, - MARKER_LINE_WIDTH, - (x0, y0, x1, y1), - ); - draw_line( - image, - (x0, y0), - (x0, y1), - green, - MARKER_LINE_WIDTH, - (x0, y0, x1, y1), - ); - draw_line( - image, - (x1, y0), - (x1, y1), - green, - MARKER_LINE_WIDTH, - (x0, y0, x1, y1), - ); - draw_line( - image, - (x0, y0), - (x1, y1), - green, - MARKER_LINE_WIDTH, - (x0, y0, x1, y1), - ); - draw_line( - image, - (x1, y0), - (x0, y1), - green, - MARKER_LINE_WIDTH, - (x0, y0, x1, y1), - ); -} - -fn draw_line( - image: &mut image::RgbaImage, - start: (u32, u32), - end: (u32, u32), - color: image::Rgba, - line_width: u32, - bounds: (u32, u32, u32, u32), -) { - let mut x = start.0 as i64; - let mut y = start.1 as i64; - let target_x = end.0 as i64; - let target_y = end.1 as i64; - let dx = (target_x - x).abs(); - let sx = if x < target_x { 1 } else { -1 }; - let dy = -(target_y - y).abs(); - let sy = if y < target_y { 1 } else { -1 }; - let mut error = dx + dy; - - loop { - draw_brush(image, x, y, color, line_width, bounds); - if x == target_x && y == target_y { - break; - } - let twice_error = error * 2; - if twice_error >= dy { - error += dy; - x += sx; - } - if twice_error <= dx { - error += dx; - y += sy; - } - } -} - -fn draw_brush( - image: &mut image::RgbaImage, - x: i64, - y: i64, - color: image::Rgba, - line_width: u32, - bounds: (u32, u32, u32, u32), -) { - let (x0, y0, x1, y1) = bounds; - let line_width = line_width.max(1) as i64; - let before = (line_width - 1) / 2; - let after = line_width / 2; - let min_x = (x - before).max(x0 as i64); - let max_x = (x + after).min(x1 as i64); - let min_y = (y - before).max(y0 as i64); - let max_y = (y + after).min(y1 as i64); - for yy in min_y..=max_y { - for xx in min_x..=max_x { - image.put_pixel(xx as u32, yy as u32, color); - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::ui_editor::commands::separation::{SeparationNote, TextMaskArea}; - - #[test] - fn draw_frame_adds_green_cross_corner_lines() { - let mut image = image::RgbaImage::from_pixel(8, 6, image::Rgba([1, 2, 3, 255])); - draw_frame(&mut image, 1, 1, 5, 3, 8, 6); - let green = image::Rgba([0, 255, 0, 255]); - - for &(x, y) in &[(1, 1), (5, 1), (1, 3), (5, 3), (3, 2)] { - assert_eq!(*image.get_pixel(x, y), green, "pixel ({x}, {y})"); - } - assert_eq!(*image.get_pixel(3, 1), green); - assert_eq!(*image.get_pixel(3, 3), green); - assert_eq!(*image.get_pixel(2, 2), green); - assert_eq!(*image.get_pixel(4, 2), green); - assert_eq!(*image.get_pixel(0, 0), image::Rgba([1, 2, 3, 255])); - } - - #[test] - fn draw_frame_keeps_cross_inside_clipped_rect() { - let mut image = image::RgbaImage::from_pixel(4, 4, image::Rgba([1, 2, 3, 255])); - draw_frame(&mut image, 2, 2, 4, 4, 4, 4); - let green = image::Rgba([0, 255, 0, 255]); - for y in 2..4 { - for x in 2..4 { - assert_eq!(*image.get_pixel(x, y), green); - } - } - assert_eq!(*image.get_pixel(1, 1), image::Rgba([1, 2, 3, 255])); - } - - #[test] - fn clipped_rect_ignores_rectangles_starting_outside_image() { - assert_eq!(clipped_rect(4, 0, 1, 1, 4, 4), None); - assert_eq!(clipped_rect(0, 4, 1, 1, 4, 4), None); - } - - #[test] - fn text_mask_is_purple_before_green_frame() { - let mut image = image::RgbaImage::from_pixel(8, 8, image::Rgba([1, 2, 3, 255])); - let node = SeparationNode { - id: crate::ui_editor::utils::NodeId::new("image").unwrap(), - global_pos_x_px: 1, - global_pos_y_px: 1, - width_px: 6, - height_px: 6, - note: SeparationNote::default(), - text_mask_areas: vec![ - TextMaskArea { - global_pos_x_px: 0, - global_pos_y_px: 0, - width_px: 1, - height_px: 1, - }, - TextMaskArea { - global_pos_x_px: 1, - global_pos_y_px: 1, - width_px: 1, - height_px: 1, - }, - ], - children: vec![], - rework_count: 0, - }; - for mask in &node.text_mask_areas { - fill_rect( - &mut image, - mask.global_pos_x_px, - mask.global_pos_y_px, - mask.width_px, - mask.height_px, - PURPLE_FILL, - 8, - 8, - ); - } - draw_frame( - &mut image, - node.global_pos_x_px, - node.global_pos_y_px, - node.width_px, - node.height_px, - 8, - 8, - ); - assert_eq!(*image.get_pixel(0, 0), PURPLE_FILL); - assert_eq!(*image.get_pixel(1, 1), image::Rgba([0, 255, 0, 255])); - } -} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index fa3cc23d2..c7e48d309 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -1,12 +1,10 @@ mod area; -mod marker; mod model; mod persistence; mod prompt; mod tree; mod workflow; -pub(crate) use marker::build_marked_image; pub use model::*; pub use persistence::*; pub use tree::*; @@ -110,7 +108,7 @@ mod tests { } #[test] - fn construction_attaches_text_mask_to_nearest_unbound_image() { + fn construction_keeps_text_as_removal_only_context() { let image = Component::Image(ImageComponent { target_graphic: None, image_type: ImageType::Simple { @@ -130,8 +128,8 @@ mod tests { let result = construct_separation_state(&state(root)); let outer = &result.trees[0].root.children[0]; assert_eq!(outer.id.as_str(), "outer-image"); - assert_eq!(outer.text_mask_areas.len(), 1); - assert!(outer.children.is_empty()); + assert_eq!(outer.kind, SeparationNodeKind::ImageTarget); + assert_eq!(outer.children[0].kind, SeparationNodeKind::TextRemovalOnly); let nested_root = node( "root", @@ -148,12 +146,12 @@ mod tests { ); let nested = construct_separation_state(&state(nested_root)); let inner = &nested.trees[0].root.children[0].children[0]; - assert_eq!(inner.text_mask_areas.len(), 1); - assert!(nested.trees[0].root.children[0].text_mask_areas.is_empty()); + assert_eq!(inner.kind, SeparationNodeKind::ImageTarget); + assert_eq!(inner.children[0].kind, SeparationNodeKind::TextRemovalOnly); } #[test] - fn root_image_receives_text_mask() { + fn root_image_keeps_text_as_removal_only_context() { let root = node( "root-image", Some(Component::Image(ImageComponent { @@ -169,12 +167,16 @@ mod tests { )], ); let result = construct_separation_state(&state(root)); - assert_eq!(result.trees[0].root.text_mask_areas.len(), 1); + assert_eq!( + result.trees[0].root.children[0].kind, + SeparationNodeKind::TextRemovalOnly + ); } #[test] fn binding_validation_requires_exact_batch_coverage() { let node = SeparationNode { id: NodeId::new("image").unwrap(), + kind: SeparationNodeKind::ImageTarget, global_pos_x_px: 0, global_pos_y_px: 0, width_px: 1, @@ -183,7 +185,6 @@ mod tests { description: "image".to_string(), rework_notes: Vec::new(), }, - text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -248,7 +249,7 @@ mod tests { MAX_REWORK_COUNT ); assert_eq!(node.rework_count, MAX_REWORK_COUNT); - assert!(next_leaf_batch(&separation, &separation.trees[0]).is_empty()); + assert!(next_image_batch(&separation, &separation.trees[0]).is_empty()); } #[test] @@ -265,12 +266,12 @@ mod tests { fn binding_validation_rejects_overlong_rework_note() { let node = SeparationNode { id: NodeId::new("image").unwrap(), + kind: SeparationNodeKind::ImageTarget, global_pos_x_px: 0, global_pos_y_px: 0, width_px: 1, height_px: 1, note: SeparationNote::default(), - text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -292,6 +293,7 @@ mod tests { assert!(dir.to_string_lossy().contains("ui_1-")); assert!(dir.to_string_lossy().ends_with("-separation")); } + #[test] fn patch_collects_bound_and_keeps_tree_topology() { let image = Component::Image(ImageComponent { @@ -322,37 +324,37 @@ mod tests { } #[test] - fn batch_selection_greedily_skips_overlapping_leaves() { + fn batch_selection_keeps_dfs_order_even_when_rectangles_overlap() { let a = SeparationNode { id: NodeId::new("a").unwrap(), + kind: SeparationNodeKind::ImageTarget, global_pos_x_px: 0, global_pos_y_px: 0, width_px: 10, height_px: 10, note: SeparationNote::default(), - text_mask_areas: vec![], children: vec![], rework_count: 0, }; let b = SeparationNode { id: NodeId::new("b").unwrap(), + kind: SeparationNodeKind::ImageTarget, global_pos_x_px: 5, global_pos_y_px: 5, width_px: 10, height_px: 10, note: SeparationNote::default(), - text_mask_areas: vec![], children: vec![], rework_count: 0, }; let c = SeparationNode { id: NodeId::new("c").unwrap(), + kind: SeparationNodeKind::ImageTarget, global_pos_x_px: 20, global_pos_y_px: 0, width_px: 5, height_px: 5, note: SeparationNote::default(), - text_mask_areas: vec![], children: vec![], rework_count: 0, }; @@ -360,12 +362,12 @@ mod tests { src_ui_design: UIDesignImageId::new("page").unwrap(), root: SeparationNode { id: NodeId::new("root").unwrap(), + kind: SeparationNodeKind::PureContainer, global_pos_x_px: 0, global_pos_y_px: 0, width_px: 100, height_px: 100, note: SeparationNote::default(), - text_mask_areas: vec![], children: vec![a, b, c], rework_count: 0, }, @@ -377,13 +379,134 @@ mod tests { bound: vec![], problematic_nodes: vec![], }; - let batch = next_leaf_batch(&state, &tree); + let batch = next_image_batch(&state, &tree); assert_eq!( batch .iter() .map(|node| node.id.as_str()) .collect::>(), - vec!["a", "c"] + vec!["a", "b", "c"] ); } + + #[test] + fn batch_selection_skips_text_removal_only_nodes() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let text = Component::Text(TextComponent::new("标题")); + let separation = construct_separation_state(&state(node( + "root", + None, + vec![ + node("text", Some(text), vec![]), + node("image", Some(image), vec![]), + ], + ))); + + let batch = next_image_batch(&separation, &separation.trees[0]); + assert_eq!( + batch + .iter() + .map(|node| node.id.as_str()) + .collect::>(), + vec!["image"] + ); + } + + #[test] + fn batch_selection_includes_extractable_root_before_children() { + let image = Component::Image(ImageComponent { + target_graphic: None, + image_type: ImageType::Simple { + preserve_aspect: false, + }, + }); + let separation = construct_separation_state(&state(node( + "root-image", + Some(image.clone()), + vec![node("child-image", Some(image), vec![])], + ))); + + let batch = next_image_batch(&separation, &separation.trees[0]); + assert_eq!( + batch + .iter() + .map(|node| node.id.as_str()) + .collect::>(), + vec!["root-image", "child-image"] + ); + } + + fn image_separation_node(id: &str, width_px: u32, height_px: u32) -> SeparationNode { + SeparationNode { + id: NodeId::new(id).unwrap(), + kind: SeparationNodeKind::ImageTarget, + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px, + height_px, + note: SeparationNote::default(), + children: vec![], + rework_count: 0, + } + } + + fn separation_with_children( + children: Vec, + ) -> (SeparationState, SeparationTree) { + let tree = SeparationTree { + src_ui_design: UIDesignImageId::new("page").unwrap(), + root: SeparationNode { + id: NodeId::new("root").unwrap(), + kind: SeparationNodeKind::PureContainer, + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 100, + height_px: 100, + note: SeparationNote::default(), + children, + rework_count: 0, + }, + root_extractable: false, + }; + let separation = SeparationState { + schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), + trees: vec![tree.clone()], + bound: vec![], + problematic_nodes: vec![], + }; + (separation, tree) + } + + #[test] + fn batch_selection_stops_before_second_node_that_exceeds_area_budget() { + let first_area = IMAGE_EDIT_AREA_LIMIT_PX / 2 + 1; + let first = image_separation_node("first", first_area as u32, 1); + let second = image_separation_node("second", first_area as u32, 1); + let (separation, tree) = separation_with_children(vec![first, second]); + + let batch = next_image_batch(&separation, &tree); + assert_eq!( + batch + .iter() + .map(|node| node.id.as_str()) + .collect::>(), + vec!["first"] + ); + } + + #[test] + fn batch_selection_accepts_first_oversized_node_to_guarantee_progress() { + let oversized = + image_separation_node("oversized", (IMAGE_EDIT_AREA_LIMIT_PX + 1) as u32, 1); + let (separation, tree) = separation_with_children(vec![oversized]); + + let batch = next_image_batch(&separation, &tree); + assert_eq!(batch.len(), 1); + assert_eq!(batch[0].id.as_str(), "oversized"); + } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs index 7e1032bac..16f549feb 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs @@ -8,6 +8,11 @@ pub use node::*; pub use note::*; pub use result::*; -pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v1"; +pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v2"; pub const MAX_REWORK_COUNT: u32 = 3; pub const MAX_REWORK_NOTE_CHARS: usize = 512; +pub const IMAGE_EDIT_MAX_DIMENSION_PX: u64 = 2880; +pub const IMAGE_EDIT_AREA_UTILIZATION_PERCENT: u64 = 80; +pub const IMAGE_EDIT_AREA_LIMIT_PX: u64 = + IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_AREA_UTILIZATION_PERCENT + / 100; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs index e0ab3c8fb..d925b7e36 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs @@ -2,25 +2,24 @@ use crate::ui_editor::utils::{NodeId, UIDesignImageId}; use serde::{Deserialize, Serialize}; use ts_rs::TS; -#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Serialize, TS)] +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize, TS)] #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] -pub struct TextMaskArea { - pub global_pos_x_px: u32, - pub global_pos_y_px: u32, - pub width_px: u32, - pub height_px: u32, +pub enum SeparationNodeKind { + ImageTarget, + TextRemovalOnly, + PureContainer, } #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)] #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] pub struct SeparationNode { pub id: NodeId, + pub kind: SeparationNodeKind, pub global_pos_x_px: u32, pub global_pos_y_px: u32, pub width_px: u32, pub height_px: u32, pub note: super::SeparationNote, - pub text_mask_areas: Vec, pub children: Vec, pub rework_count: u32, } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 6e942e260..01c46b583 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -3,7 +3,6 @@ use crate::ui_editor::component::{image::ImageComponent, Component}; use crate::ui_editor::layout::node::Node; use crate::ui_editor::state::State; use crate::ui_editor::utils::NodeId; -use std::collections::HashMap; use std::collections::HashSet; fn is_unbound_image(node: &Node) -> bool { @@ -54,61 +53,41 @@ fn collect_todo_nodes( parent: &crate::ui_editor::layout::dimension::UIRect, ppu: f32, output: &mut Vec, - text_masks: &mut HashMap>, ) { let rect = node.layout.transform.resolve(parent); let mut children = Vec::new(); for child in &node.children { - collect_todo_nodes(child, &rect, ppu, &mut children, text_masks); + collect_todo_nodes(child, &rect, ppu, &mut children); } - if is_unbound_image(node) { - let (x, y, w, h) = node_pixel_rect(node, parent, ppu); - output.push(SeparationNode { - id: node.id.clone(), - global_pos_x_px: x, - global_pos_y_px: y, - width_px: w, - height_px: h, - note: SeparationNote { - description: node_description(node), - rework_notes: Vec::new(), - }, - text_mask_areas: text_masks.remove(&node.id).unwrap_or_default(), - children, - rework_count: 0, - }); + let kind = if is_unbound_image(node) { + Some(SeparationNodeKind::ImageTarget) + } else if has_text_component(node) && !has_image_component(node) { + Some(SeparationNodeKind::TextRemovalOnly) } else { - output.extend(children); - } -} - -fn collect_text_masks( - node: &Node, - parent: &crate::ui_editor::layout::dimension::UIRect, - ppu: f32, - nearest_image: Option, - output: &mut HashMap>, -) { - let node_is_image = is_unbound_image(node); - let nearest_image = if node_is_image { - Some(node.id.clone()) - } else { - nearest_image + None }; - if has_text_component(node) && !has_image_component(node) { - if let Some(image_id) = nearest_image.clone() { - let (x, y, w, h) = node_pixel_rect(node, parent, ppu); - output.entry(image_id).or_default().push(TextMaskArea { + if let Some(kind) = kind { + let (x, y, w, h) = node_pixel_rect(node, parent, ppu); + if w > 0 && h > 0 { + output.push(SeparationNode { + id: node.id.clone(), + kind, global_pos_x_px: x, global_pos_y_px: y, width_px: w, height_px: h, + note: SeparationNote { + description: node_description(node), + rework_notes: Vec::new(), + }, + children, + rework_count: 0, }); + } else { + output.extend(children); } - } - let rect = node.layout.transform.resolve(parent); - for child in &node.children { - collect_text_masks(child, &rect, ppu, nearest_image.clone(), output); + } else { + output.extend(children); } } @@ -122,11 +101,9 @@ pub fn construct_separation_state(state: &State) -> SeparationState { let size = image.pixel_size / ppu; let root_rect = crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); - let mut text_masks = HashMap::new(); - collect_text_masks(&tree.root, &root_rect, ppu, None, &mut text_masks); let mut children = Vec::new(); for child in &tree.root.children { - collect_todo_nodes(child, &root_rect, ppu, &mut children, &mut text_masks); + collect_todo_nodes(child, &root_rect, ppu, &mut children); } let root_extractable = is_unbound_image(&tree.root); if !root_extractable && children.is_empty() { @@ -137,6 +114,13 @@ pub fn construct_separation_state(state: &State) -> SeparationState { src_ui_design: tree.src_ui_design.clone(), root: SeparationNode { id: tree.root.id.clone(), + kind: if root_extractable { + SeparationNodeKind::ImageTarget + } else if has_text_component(&tree.root) && !has_image_component(&tree.root) { + SeparationNodeKind::TextRemovalOnly + } else { + SeparationNodeKind::PureContainer + }, global_pos_x_px: x, global_pos_y_px: y, width_px: w, @@ -145,7 +129,6 @@ pub fn construct_separation_state(state: &State) -> SeparationState { description: node_description(&tree.root), rework_notes: Vec::new(), }, - text_mask_areas: text_masks.remove(&tree.root.id).unwrap_or_default(), children, rework_count: 0, }, @@ -170,62 +153,55 @@ fn terminal_ids(state: &SeparationState) -> HashSet { .collect() } -fn logical_leaves<'a>( +fn collect_dfs_batch<'a>( node: &'a SeparationNode, - extractable: bool, + is_root: bool, + root_extractable: bool, terminal: &HashSet, - output: &mut Vec<&'a SeparationNode>, -) { - let is_terminal = terminal.contains(&node.id); - let children_terminal = node - .children - .iter() - .all(|child| terminal.contains(&child.id)); - if extractable && !is_terminal && children_terminal { - output.push(node); - return; + selected: &mut Vec<&'a SeparationNode>, + area: &mut u64, +) -> bool { + let is_target = matches!(node.kind, SeparationNodeKind::ImageTarget) + && (!is_root || root_extractable) + && !terminal.contains(&node.id); + if is_target { + let node_area = u64::from(node.width_px).saturating_mul(u64::from(node.height_px)); + let would_exceed = area.saturating_add(node_area) > IMAGE_EDIT_AREA_LIMIT_PX; + if selected.is_empty() || !would_exceed { + selected.push(node); + *area = area.saturating_add(node_area); + } else { + return true; + } } for child in &node.children { - logical_leaves(child, true, terminal, output); + if collect_dfs_batch(child, false, root_extractable, terminal, selected, area) { + return true; + } } + false } -fn overlaps(a: &SeparationNode, b: &SeparationNode) -> bool { - let ax1 = a.global_pos_x_px as u64 + a.width_px as u64; - let ay1 = a.global_pos_y_px as u64 + a.height_px as u64; - let bx1 = b.global_pos_x_px as u64 + b.width_px as u64; - let by1 = b.global_pos_y_px as u64 + b.height_px as u64; - let width = ax1 - .min(bx1) - .saturating_sub(a.global_pos_x_px.max(b.global_pos_x_px) as u64); - let height = ay1 - .min(by1) - .saturating_sub(a.global_pos_y_px.max(b.global_pos_y_px) as u64); - width > 0 && height > 0 -} - -pub fn next_leaf_batch<'a>( +pub fn next_image_batch<'a>( state: &SeparationState, tree: &'a SeparationTree, ) -> Vec<&'a SeparationNode> { let terminal = terminal_ids(state); - let mut candidates = Vec::new(); - logical_leaves( + let mut selected = Vec::new(); + let mut area = 0; + collect_dfs_batch( &tree.root, + true, tree.root_extractable, &terminal, - &mut candidates, + &mut selected, + &mut area, ); - let mut selected: Vec<&'a SeparationNode> = Vec::new(); - for candidate in candidates { - if selected.iter().all(|other| !overlaps(candidate, other)) { - selected.push(candidate); - } - } app_log!( - "ui_separation.batch_selected image_id={} leaf_nodes={}", + "ui_separation.batch_selected image_id={} image_nodes={} area_px={}", tree.src_ui_design.as_str(), - selected.len() + selected.len(), + area ); selected } @@ -236,6 +212,7 @@ pub fn validate_binding_response( ) -> Result<(), String> { let expected = batch .iter() + .filter(|node| matches!(node.kind, SeparationNodeKind::ImageTarget)) .map(|node| node.id.clone()) .collect::>(); let mut seen = HashSet::new(); @@ -251,11 +228,7 @@ pub fn validate_binding_response( if !seen.insert(node_id.clone()) { return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); } - if let BindingDecision::NeedRework { - advice, - .. - } = decision - { + if let BindingDecision::NeedRework { advice, .. } = decision { if advice.trim().is_empty() { return Err("NeedRework 必须包含问题描述".to_string()); } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs index da0c9e6b0..08f85b7d8 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs @@ -19,6 +19,7 @@ use serde::Deserialize; use std::fs; use std::path::{Path, PathBuf}; use std::time::Instant; + pub fn apply_batch_patch( state: &mut SeparationState, tree_index: usize, @@ -36,7 +37,7 @@ pub fn apply_batch_patch( .trees .get(tree_index) .ok_or_else(|| "separation tree 索引无效".to_string())?; - next_leaf_batch(state, tree) + next_image_batch(state, tree) .into_iter() .cloned() .collect::>() @@ -160,13 +161,13 @@ async fn raw_image_edit_inner( .and_then(|v| v.strip_suffix(";base64")) .unwrap_or("image/png"); if !mime.eq_ignore_ascii_case("image/png") { - return Err("图片分离请求只支持 PNG 标记图".to_string()); + return Err("图片分离请求只支持 PNG 源图".to_string()); } let image_bytes = base64::engine::general_purpose::STANDARD .decode(data.trim()) - .map_err(|error| format!("解码标记图失败：{error}"))?; + .map_err(|error| format!("解码源图失败：{error}"))?; if image_bytes.is_empty() { - return Err("标记图不能为空".to_string()); + return Err("源图不能为空".to_string()); } let client = crate::http_client::agc_main_site_client_builder() .build() @@ -481,7 +482,7 @@ pub(crate) async fn separate_ui_impl( let Some(current_tree) = separation.trees.get(tree_index) else { break; }; - let batch_nodes = next_leaf_batch(&separation, current_tree) + let batch_nodes = next_image_batch(&separation, current_tree) .into_iter() .cloned() .collect::>(); @@ -496,8 +497,7 @@ pub(crate) async fn separate_ui_impl( break; } let batch = batch_nodes.iter().collect::>(); - let prompt = - gen_extract_prompt(batch.iter().map(|n| n.note.clone()).collect::>()); + let prompt = gen_extract_prompt(&separation, current_tree, &batch); app_log!( "ui_separation.batch_start tree_index={} batch_index={} nodes={} prompt_chars={} rework_total={}", tree_index, @@ -506,28 +506,9 @@ pub(crate) async fn separate_ui_impl( prompt.chars().count(), batch.iter().map(|node| node.rework_count).sum::() ); - let marker_path = sidecar.join(format!("marked-{}.png", separation.bound.len())); - let marked_url = match build_marked_image( - source_url.clone(), - batch_nodes.clone(), - marker_path, - ) - .await - { - Ok(value) => value, - Err(error) => { - app_log!( - "ui_separation.error stage=mark_image tree_index={} batch_index={} error={error}", - tree_index, - batch_index - ); - write_separation_state(&state_path, &separation)?; - return Err(error); - } - }; let processed_url = match raw_image_edit( &session, - &marked_url, + &source_url, &prompt, image.pixel_size.x as u32, image.pixel_size.y as u32, diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts index 09a0d8a53..a3fffd4e0 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNode.ts @@ -1,6 +1,6 @@ // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. import type { NodeId } from "./NodeId"; +import type { SeparationNodeKind } from "./SeparationNodeKind"; import type { SeparationNote } from "./SeparationNote"; -import type { TextMaskArea } from "./TextMaskArea"; -export type SeparationNode = { id: NodeId, global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, note: SeparationNote, text_mask_areas: Array, children: Array, rework_count: number, }; +export type SeparationNode = { id: NodeId, kind: SeparationNodeKind, global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, note: SeparationNote, children: Array, rework_count: number, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNodeKind.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNodeKind.ts new file mode 100644 index 000000000..86f060e82 --- /dev/null +++ b/apps/ai-game-creator-shell/src/features/ui-editor/types/SeparationNodeKind.ts @@ -0,0 +1,3 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +export type SeparationNodeKind = "ImageTarget" | "TextRemovalOnly" | "PureContainer"; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts deleted file mode 100644 index 5e450d070..000000000 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/TextMaskArea.ts +++ /dev/null @@ -1,3 +0,0 @@ -// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. - -export type TextMaskArea = { global_pos_x_px: number, global_pos_y_px: number, width_px: number, height_px: number, }; From 1303592a86d79816401bff777d553e735ee04ea0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 13:54:33 +0800 Subject: [PATCH 104/248] =?UTF-8?q?=E9=87=8D=E6=9E=84UI=E7=BC=96=E8=BE=91?= =?UTF-8?q?=E5=99=A8=E7=B4=A0=E6=9D=90=E5=88=87=E5=88=86=E5=B7=A5=E4=BD=9C?= =?UTF-8?q?=E6=B5=81=E6=A8=A1=E5=9D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 拆分提取与绑定提示词模块 拆分图片编辑、视觉绑定、裁切和批次 patch 实现 同步技术方案实现组织说明并更新提示词测试 --- .../src/ui_editor/commands/separation/mod.rs | 29 +- .../ui_editor/commands/separation/prompt.rs | 70 -- .../commands/separation/prompt/binding.rs | 38 + .../commands/separation/prompt/extract.rs | 101 +++ .../commands/separation/prompt/mod.rs | 5 + .../ui_editor/commands/separation/workflow.rs | 711 ------------------ .../commands/separation/workflow/binding.rs | 130 ++++ .../commands/separation/workflow/cut.rs | 93 +++ .../separation/workflow/image_edit.rs | 172 +++++ .../commands/separation/workflow/mod.rs | 222 ++++++ .../commands/separation/workflow/patch.rs | 98 +++ ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 6 + 12 files changed, 890 insertions(+), 785 deletions(-) delete mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/mod.rs delete mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/image_edit.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index c7e48d309..6b8545876 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -254,11 +254,32 @@ mod tests { #[test] fn next_extract_prompt_contains_previous_rework_notes() { - let note = SeparationNote { - description: "图标".to_string(), - rework_notes: vec!["不要带父背景".to_string()], + let node = SeparationNode { + id: NodeId::new("image").unwrap(), + kind: SeparationNodeKind::ImageTarget, + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + note: SeparationNote { + description: "图标".to_string(), + rework_notes: vec!["不要带父背景".to_string()], + }, + children: vec![], + rework_count: 0, }; - let prompt = super::prompt::gen_extract_prompt(vec![note]); + let tree = SeparationTree { + src_ui_design: UIDesignImageId::new("page").unwrap(), + root: node.clone(), + root_extractable: true, + }; + let separation = SeparationState { + schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), + trees: vec![tree.clone()], + bound: vec![], + problematic_nodes: vec![], + }; + let prompt = super::prompt::gen_extract_prompt(&separation, &tree, &[&node]); assert!(prompt.contains("previous rework notes:\n- 不要带父背景")); } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs deleted file mode 100644 index a91a94889..000000000 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt.rs +++ /dev/null @@ -1,70 +0,0 @@ -use crate::ui_editor::commands::separation::{SeparationNode, SeparationNote}; - -const SHARED_SEPARATION_REQ: &str = r#" - - MUST hard edges; preserve no glow/blur beyond the exact visible shape. - NEVER keep its parent's background with it. - NEVER include any text unless requested. - - UI elements that needs to extract has been marked with GREEN line frames box with crossline inside. (only for mark purpose, NEVER wrap a frame in your extraction). - On some UI elements, there is some PURPLE filled area, they were removed UI elements, reconstruct the background under where they were. - - MUST extract exactly these marked UI elements area. -"#; -pub(super) fn gen_extract_prompt(separation_notes: Vec) -> String { - let extract_system_prompt = format!( - r#" - This is a UI design image, not a normal photo/illustration. Extract it strictly as UI elements/layers, not as a generic foreground/background extraction. - Treat distinct UI element as its own layer with hard, clean, pixel-accurate edges and full transparency outside the element. - - MUST keep each element at its original position on a transparent canvas. - {SHARED_SEPARATION_REQ} - here are UI elements to extract: - - "# - ); - let mut result = extract_system_prompt; - result.reserve(512); - for elem in separation_notes { - result.push_str(&elem.as_prompt()); - result.push('\n'); - } - result -} -pub(super) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { - let binding_system_prompt = format!( - r#" - You will be given a src UI design image and a processed image, where some ui elements are separated. - You need to recognize and review the separation using the given tool. - field notes: - * extracted_area MUST be the recognized area from the processed image, INSTEAD OF from the src image. - The processed image is the only authoritative image for extracted_area. - Return the pixel bounding box of the extracted element as it appears in the processed image. - Do not copy, infer, or reuse the source node rectangle. - The src image is only for identifying which semantic UI element belongs to to_node. - - Here were the separation requirements: - ``` - {SHARED_SEPARATION_REQ} - ``` - And you should also review if the extracted's successfully meet the src image: - * shape - * color - * style - * edge process - ... - - if not, use `NeedRework` data structure in the tool to indicate the (it should be from which) node id and advice. - your advice (less than 20 words) will be used to improve the separation in the next time. - - these node need handle: - "# - ); - let mut result = binding_system_prompt; - result.reserve(512); - for elem in nodes { - result.push_str(&elem.as_prompt()); - result.push('\n'); - } - result -} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs new file mode 100644 index 000000000..0f621c248 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs @@ -0,0 +1,38 @@ +use crate::ui_editor::commands::separation::SeparationNode; + +pub(crate) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { + let binding_system_prompt = r#" + You will be given a src UI design image and a processed image, where some ui elements are separated. + You need to recognize and review the separation using the given tool. + field notes: + * extracted_area MUST be the recognized area from the processed image, INSTEAD OF from the src image. + The processed image is the only authoritative image for extracted_area. + Return the pixel bounding box of the extracted element as it appears in the processed image. + Do not copy, infer, or reuse the source node rectangle. + The src image is only for identifying which semantic UI element belongs to to_node. + + Here were the separation requirements: + Preserve hard edges and the exact visible shape. + The processed image is a transparent atlas containing the requested image layers. + Do not use the source node rectangle as the extracted area. + And you should also review if the extracted's successfully meet the src image: + * shape + * color + * style + * edge process + ... + + if not, use `NeedRework` data structure in the tool to indicate the (it should be from which) node id and advice. + your advice (less than 20 words) will be used to improve the separation in the next time. + + these node need handle: + "# + .to_string(); + let mut result = binding_system_prompt; + result.reserve(512); + for elem in nodes { + result.push_str(&elem.as_prompt()); + result.push('\n'); + } + result +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs new file mode 100644 index 000000000..be839b24b --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs @@ -0,0 +1,101 @@ +use crate::ui_editor::commands::separation::{SeparationNode, SeparationNodeKind}; +use std::collections::HashSet; + +use crate::ui_editor::commands::separation::model::{SeparationState, SeparationTree}; +use crate::ui_editor::utils::NodeId; + +pub(crate) fn gen_extract_prompt( + state: &SeparationState, + tree: &SeparationTree, + batch: &[&SeparationNode], +) -> String { + let mut result = r#" + This is a UI design image, not a normal photo/illustration. Extract it strictly as UI elements/layers, not as a generic foreground/background extraction. + Treat distinct UI element as its own layer with hard, clean, pixel-accurate edges and full transparency outside the element. + Generate one transparent atlas at the requested canvas size. You may move or scale output layers so they do not cover one another. + Preserve hard edges and the exact visible shape. Never split a scene/background into multiple scene layers. + Ordinary text is editable UI text: remove it from its parent image/background and do not generate a text raster layer. + Extract only the image nodes marked OUTPUT_THIS_TURN. Reconstruct every child/text layer that is listed under a parent but is not an output target. + + UI layer tree: +"# + .to_string(); + result.reserve(2048); + let target_ids = batch + .iter() + .map(|node| node.id.clone()) + .collect::>(); + let terminal_ids = state + .bound + .iter() + .map(|node| node.node_id.clone()) + .chain( + state + .problematic_nodes + .iter() + .map(|node| node.node_id.clone()), + ) + .collect::>(); + let mut lines = Vec::new(); + let mut index = 1usize; + append_context_lines( + &tree.root, + 0, + true, + &target_ids, + &terminal_ids, + &mut index, + &mut lines, + ); + result.push_str(&lines.join("\n")); + result +} + +fn append_context_lines( + node: &SeparationNode, + depth: usize, + is_root: bool, + target_ids: &HashSet, + terminal_ids: &HashSet, + index: &mut usize, + lines: &mut Vec, +) { + let status = match node.kind { + SeparationNodeKind::TextRemovalOnly => "REMOVE_ONLY", + SeparationNodeKind::PureContainer => "CONTEXT_ONLY", + SeparationNodeKind::ImageTarget => { + if target_ids.contains(&node.id) { + "OUTPUT_THIS_TURN" + } else if terminal_ids.contains(&node.id) { + "DONE" + } else { + "CONTEXT_ONLY" + } + } + }; + let role = if is_root { "root" } else { "child" }; + lines.push(format!( + "{}. [{}] depth={} role={} rect=({}, {}, {}, {}) {}", + *index, + status, + depth, + role, + node.global_pos_x_px, + node.global_pos_y_px, + node.width_px, + node.height_px, + node.note.as_prompt() + )); + *index += 1; + for child in &node.children { + append_context_lines( + child, + depth + 1, + false, + target_ids, + terminal_ids, + index, + lines, + ); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/mod.rs new file mode 100644 index 000000000..219a1164c --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/mod.rs @@ -0,0 +1,5 @@ +mod binding; +mod extract; + +pub(super) use binding::gen_binding_prompt; +pub(super) use extract::gen_extract_prompt; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs deleted file mode 100644 index 08f85b7d8..000000000 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow.rs +++ /dev/null @@ -1,711 +0,0 @@ -use super::area::normalize_binding_area; -use super::model::*; -use super::prompt::{gen_binding_prompt, gen_extract_prompt}; -use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; -use crate::platform_session::current_platform_session; -use crate::ui_editor::commands::separation::*; -use crate::ui_editor::commands::utils::{ - parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm, - run_with_repair_history, strict_json_schema, -}; -use crate::ui_editor::state::State; -use crate::ui_editor::utils::NodeId; -use base64::Engine as _; -use image::ImageFormat; -use platform_llm::{ - LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice, -}; -use serde::Deserialize; -use std::fs; -use std::path::{Path, PathBuf}; -use std::time::Instant; - -pub fn apply_batch_patch( - state: &mut SeparationState, - tree_index: usize, - decisions: &[BindingDecision], - cut_paths: &std::collections::HashMap, -) -> Result<(), String> { - app_log!( - "ui_separation.batch_patch.start tree_index={} decisions={} cut_paths={}", - tree_index, - decisions.len(), - cut_paths.len() - ); - let batch_nodes = { - let tree = state - .trees - .get(tree_index) - .ok_or_else(|| "separation tree 索引无效".to_string())?; - next_image_batch(state, tree) - .into_iter() - .cloned() - .collect::>() - }; - let batch = batch_nodes.iter().collect::>(); - validate_binding_response( - &BindingResp { - decisions: decisions.to_vec(), - }, - &batch, - )?; - let rework_counts = batch_nodes - .iter() - .map(|node| (node.id.clone(), node.rework_count)) - .collect::>(); - let tree = state.trees.get_mut(tree_index).expect("tree index checked"); - for decision in decisions { - match decision { - BindingDecision::Ok { to_node, .. } => { - let path = cut_paths - .get(to_node) - .ok_or_else(|| format!("缺少节点 {} 的 cut 图片", to_node.as_str()))?; - state.bound.push(BoundNode { - node_id: to_node.clone(), - cut_image_path: path.clone(), - }); - } - BindingDecision::NeedRework { - to_node, - advice: problem_description, - } => { - append_rework_note(&mut tree.root, to_node, problem_description); - let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; - increment_rework_count(&mut tree.root, to_node, count); - if count >= MAX_REWORK_COUNT { - state.problematic_nodes.push(ProblematicNode { - node_id: to_node.clone(), - problem_description: problem_description.clone(), - rework_count: count, - }); - } - } - } - } - app_log!( - "ui_separation.batch_patch.completed tree_index={} bound={} problematic={} pending_root_children={}", - tree_index, - state.bound.len(), - state.problematic_nodes.len(), - tree.root.children.len() - ); - Ok(()) -} - -fn append_rework_note(node: &mut SeparationNode, id: &NodeId, note: &str) -> bool { - if node.id == *id { - node.note.rework_notes.push(note.to_string()); - return true; - } - node.children - .iter_mut() - .any(|child| append_rework_note(child, id, note)) -} - -fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { - if node.id == *id { - node.rework_count = count; - return; - } - for child in &mut node.children { - increment_rework_count(child, id, count); - } -} - -#[derive(Deserialize)] -struct RawEditResponse { - data: Vec, -} -#[derive(Deserialize)] -struct RawEditItem { - b64_json: String, -} - -async fn raw_image_edit( - session: &crate::platform_session::PlatformSessionSnapshot, - image_data_url: &str, - prompt: &str, - width: u32, - height: u32, -) -> Result { - let started = Instant::now(); - let result = raw_image_edit_inner(session, image_data_url, prompt, width, height).await; - app_log!( - "ui_separation.image_edit.timing outcome={} elapsed_ms={} width={} height={}", - if result.is_ok() { "ok" } else { "error" }, - started.elapsed().as_millis(), - width, - height - ); - result -} - -async fn raw_image_edit_inner( - session: &crate::platform_session::PlatformSessionSnapshot, - image_data_url: &str, - prompt: &str, - width: u32, - height: u32, -) -> Result { - app_log!( - "ui_separation.image_edit.start width={} height={} prompt_chars={}", - width, - height, - prompt.chars().count() - ); - let (mime, data) = image_data_url - .split_once(",") - .ok_or_else(|| "界面图 data URL 无效".to_string())?; - let mime = mime - .strip_prefix("data:") - .and_then(|v| v.strip_suffix(";base64")) - .unwrap_or("image/png"); - if !mime.eq_ignore_ascii_case("image/png") { - return Err("图片分离请求只支持 PNG 源图".to_string()); - } - let image_bytes = base64::engine::general_purpose::STANDARD - .decode(data.trim()) - .map_err(|error| format!("解码源图失败：{error}"))?; - if image_bytes.is_empty() { - return Err("源图不能为空".to_string()); - } - let client = crate::http_client::agc_main_site_client_builder() - .build() - .map_err(|e| format!("创建图片编辑客户端失败：{e}"))?; - let url = format!( - "{}/api/raw/v1/images/edit", - session.api_base_url.trim_end_matches('/') - ); - let image_part = reqwest::multipart::Part::bytes(image_bytes) - .file_name("image.png") - .mime_str("image/png") - .map_err(|error| format!("构造图片编辑文件部件失败：{error}"))?; - let body = reqwest::multipart::Form::new() - .part("image", image_part) - .text("prompt", prompt.to_string()) - .text("width", width.to_string()) - .text("height", height.to_string()) - .text("output_format", "png") - .text("background", "transparent"); - let response = crate::http_client::with_agc_main_site_marker( - client - .post(url) - .bearer_auth(&session.access_token) - .multipart(body), - ) - .send() - .await - .map_err(|e| { - app_log!("ui_separation.error stage=image_edit reason=send error={e}"); - format!("图片分离请求失败：{e}") - })?; - if !response.status().is_success() { - app_log!( - "ui_separation.error stage=image_edit reason=http_status status={}", - response.status() - ); - return Err(format!("图片分离请求失败（HTTP {}）", response.status())); - } - let payload = response.json::().await.map_err(|e| { - app_log!("ui_separation.error stage=image_edit reason=parse_response error={e}"); - format!("解析图片分离响应失败：{e}") - })?; - let result = payload - .data - .into_iter() - .next() - .map(|item| format!("data:image/png;base64,{}", item.b64_json)) - .ok_or_else(|| "图片分离响应没有图像".to_string()); - match &result { - Ok(value) => app_log!( - "ui_separation.image_edit.completed data_url_chars={}", - value.chars().count() - ), - Err(error) => { - app_log!("ui_separation.error stage=image_edit reason=empty_result error={error}") - } - } - result -} - -async fn write_processed_image(processed_url: String, target: PathBuf) -> Result<(), String> { - let started = Instant::now(); - let result = write_processed_image_inner(processed_url, target).await; - app_log!( - "ui_separation.processed_image.write.timing outcome={} elapsed_ms={}", - if result.is_ok() { "ok" } else { "error" }, - started.elapsed().as_millis() - ); - result -} - -async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> Result<(), String> { - app_log!( - "ui_separation.processed_image.write.start target_file={} data_url_chars={}", - target - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or(""), - processed_url.chars().count() - ); - tokio::task::spawn_blocking(move || { - let encoded = processed_url - .split_once(',') - .map(|(_, data)| data) - .ok_or_else(|| "处理图 data URL 无效".to_string())?; - let processed_bytes = base64::engine::general_purpose::STANDARD - .decode(encoded) - .map_err(|error| format!("解析处理图失败：{error}"))?; - let byte_len = processed_bytes.len(); - fs::write(&target, processed_bytes) - .map_err(|error| format!("写入处理图失败：{}: {error}", target.display())) - .map(|_| { - app_log!( - "ui_separation.processed_image.write.completed target_file={} bytes={}", - target - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or(""), - byte_len - ); - }) - }) - .await - .map_err(|error| format!("写入处理图任务失败：{error}"))? -} - -async fn visual_binding( - source_url: String, - processed_url: String, - nodes: &[&SeparationNode], -) -> Result { - let started = Instant::now(); - let result = visual_binding_inner(source_url, processed_url, nodes).await; - app_log!( - "ui_separation.visual_binding.timing outcome={} elapsed_ms={} nodes={}", - if result.is_ok() { "ok" } else { "error" }, - started.elapsed().as_millis(), - nodes.len() - ); - result -} - -async fn visual_binding_inner( - source_url: String, - processed_url: String, - nodes: &[&SeparationNode], -) -> Result { - app_log!( - "ui_separation.visual_binding.start nodes={} source_url_chars={} processed_url_chars={}", - nodes.len(), - source_url.chars().count(), - processed_url.chars().count() - ); - let llm_config = load_game_creator_app_config() - .map_err(|e| { - app_log!("ui_separation.error stage=visual_binding reason=load_config error={e}"); - e.to_string() - })? - .llm; - - let client = - build_game_creator_llm_client_from_llm_config(&llm_config, "llm").map_err(|e| { - app_log!("ui_separation.error stage=visual_binding reason=build_client error={e}"); - e.to_string() - })?; - let schema = strict_json_schema::().map_err(|error| { - app_log!("ui_separation.error stage=visual_binding reason=build_schema error={error}"); - error - })?; - let tool = LlmFunctionTool::new( - "bind_ui_elements", - "确认处理图中的区域对应哪些 UI 节点", - schema, - ) - .with_strict(true); - let initial_history = vec![ - LlmMessage::system(gen_binding_prompt(nodes.to_vec())), - LlmMessage::user_multimodal(vec![ - LlmMessageContentPart::InputText { - text: "processed image:".to_string(), - }, - LlmMessageContentPart::InputImage { - image_url: processed_url.clone(), - }, - LlmMessageContentPart::InputText { - text: "src image:".to_string(), - }, - LlmMessageContentPart::InputImage { - image_url: source_url.clone(), - }, - ]), - ]; - let result = run_with_repair_history( - 2, - initial_history, - |history| { - let tool = tool.clone(); - let client = client.clone(); - let llm_config = llm_config.clone(); - async move { - let request = LlmRunRequest::new(history) - .with_function_tools(vec![tool.clone()]) - .with_tool_choice(LlmToolChoice::Required); - let request_started = Instant::now(); - let response = request_ui_editor_llm(&client, &llm_config, request).await; - app_log!( - "ui_separation.llm.timing outcome={} elapsed_ms={}", - if response.is_ok() { "ok" } else { "error" }, - request_started.elapsed().as_millis() - ); - response - .map_err(|e| e.to_string()) - .and_then(|response| { - response - .tool_calls - .into_iter() - .find(|call| call.name == "bind_ui_elements") - .map(|call| call.arguments) - .ok_or_else(|| "视觉绑定模型未返回工具调用".to_string()) - }) - .and_then(|arguments| parse_limited_llm_tool_arguments(&arguments)) - .and_then(|args| { - serde_json::from_value::(args) - .map_err(|e| format!("视觉绑定结果无效：{e}")) - }) - .and_then(|parsed| Ok(parsed)) - } - }, - |value: &BindingResp| validate_binding_response(value, nodes), - ) - .await; - match &result { - Ok(value) => app_log!( - "ui_separation.visual_binding.completed nodes={} decisions={}", - nodes.len(), - value.decisions.len() - ), - Err(error) => app_log!( - "ui_separation.error stage=visual_binding reason=failed nodes={} error={error}", - nodes.len() - ), - } - result -} - -pub(crate) async fn separate_ui_impl( - project_path: String, - asset_id: String, - state: State, -) -> Result { - app_log!( - "ui_separation.start asset_id={} ui_trees={} ui_images={} sprites={}", - asset_id, - state.ui_trees.len(), - state.ui_design_images.len(), - state.sprite_assets.len() - ); - let session = current_platform_session().ok_or_else(|| "请先登录平台账号".to_string())?; - let root = Path::new(project_path.trim()); - let sidecar = separation_sidecar_dir(root, &asset_id).map_err(|error| { - app_log!( - "ui_separation.error stage=sidecar_dir asset_id={} error={error}", - asset_id - ); - error - })?; - fs::create_dir_all(&sidecar).map_err(|e| { - app_log!( - "ui_separation.error stage=sidecar_create asset_id={} error={e}", - asset_id - ); - format!("创建 separation sidecar 失败：{e}") - })?; - let state_path = separation_state_path(root, &asset_id)?; - let restored = state_path.exists(); - let mut separation = if restored { - app_log!("ui_separation.state_restore.start asset_id={}", asset_id); - read_separation_state(&state_path).map_err(|error| { - app_log!( - "ui_separation.error stage=state_restore asset_id={} error={error}", - asset_id - ); - error - })? - } else { - app_log!("ui_separation.state_construct.start asset_id={}", asset_id); - construct_separation_state(&state) - }; - app_log!( - "ui_separation.state_ready asset_id={} restored={} trees={} bound={} problematic={}", - asset_id, - restored, - separation.trees.len(), - separation.bound.len(), - separation.problematic_nodes.len() - ); - for tree_index in 0..separation.trees.len() { - let tree = &separation.trees[tree_index]; - let image_id = tree.src_ui_design.clone(); - let image = state - .ui_design_images - .get(&image_id) - .ok_or_else(|| "缺少源界面图".to_string())?; - let source_path = crate::project::resolve_local_project_path(root, &image.path)?; - let source_url = read_ui_reference_image_data_url(source_path) - .await - .map_err(|error| { - app_log!( - "ui_separation.error stage=read_source tree_index={} image_id={} error={error}", - tree_index, - image_id.as_str() - ); - error - })?; - app_log!( - "ui_separation.tree_start tree_index={} image_id={} width={} height={}", - tree_index, - image_id.as_str(), - image.pixel_size.x.round() as u32, - image.pixel_size.y.round() as u32 - ); - write_separation_state(&state_path, &separation).map_err(|error| { - app_log!( - "ui_separation.error stage=state_checkpoint tree_index={} error={error}", - tree_index - ); - error - })?; - let mut batch_index = 0usize; - loop { - let batch_started = Instant::now(); - let Some(current_tree) = separation.trees.get(tree_index) else { - break; - }; - let batch_nodes = next_image_batch(&separation, current_tree) - .into_iter() - .cloned() - .collect::>(); - if batch_nodes.is_empty() { - app_log!( - "ui_separation.tree_completed tree_index={} image_id={} bound={} problematic={}", - tree_index, - image_id.as_str(), - separation.bound.len(), - separation.problematic_nodes.len() - ); - break; - } - let batch = batch_nodes.iter().collect::>(); - let prompt = gen_extract_prompt(&separation, current_tree, &batch); - app_log!( - "ui_separation.batch_start tree_index={} batch_index={} nodes={} prompt_chars={} rework_total={}", - tree_index, - batch_index, - batch.len(), - prompt.chars().count(), - batch.iter().map(|node| node.rework_count).sum::() - ); - let processed_url = match raw_image_edit( - &session, - &source_url, - &prompt, - image.pixel_size.x as u32, - image.pixel_size.y as u32, - ) - .await - { - Ok(value) => value, - Err(error) => { - app_log!( - "ui_separation.error stage=image_edit tree_index={} batch_index={} error={error}", - tree_index, - batch_index - ); - write_separation_state(&state_path, &separation)?; - return Err(error); - } - }; - let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); - if let Err(error) = - write_processed_image(processed_url.clone(), processed_path.clone()).await - { - app_log!( - "ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", - tree_index, - batch_index - ); - write_separation_state(&state_path, &separation)?; - return Err(error); - } - let binding = match visual_binding(source_url.clone(), processed_url, &batch).await { - Ok(value) => value, - Err(error) => { - app_log!( - "ui_separation.error stage=visual_binding tree_index={} batch_index={} error={error}", - tree_index, - batch_index - ); - write_separation_state(&state_path, &separation)?; - return Err(error); - } - }; - app_log!( - "ui_separation.binding_decisions tree_index={} batch_index={} decisions={}", - tree_index, - batch_index, - binding.decisions.len() - ); - let mut cut_paths = std::collections::HashMap::new(); - let mut cut_error = None; - for decision in &binding.decisions { - if let BindingDecision::Ok { - to_node, - extracted_area, - } = decision - { - let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); - match cut_processed_image( - processed_path.clone(), - *extracted_area, - cut_path.clone(), - ) - .await - { - Ok(()) => { - cut_paths - .insert(to_node.clone(), project_relative_path(root, &cut_path)?); - } - Err(error) => { - app_log!( - "ui_separation.error stage=cut_image tree_index={} batch_index={} node_id={} error={error}", - tree_index, - batch_index, - to_node.as_str() - ); - cut_error = - Some(format!("节点 {} 的分离区域无效：{error}", to_node.as_str())); - break; - } - } - } - } - if let Some(error) = cut_error { - app_log!( - "ui_separation.error stage=cut_batch tree_index={} batch_index={} error={error}", - tree_index, - batch_index - ); - write_separation_state(&state_path, &separation)?; - return Err(error); - } - apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; - write_separation_state(&state_path, &separation)?; - app_log!( - "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={} elapsed_ms={}", - tree_index, - batch_index, - cut_paths.len(), - separation.bound.len(), - separation.problematic_nodes.len(), - batch_started.elapsed().as_millis() - ); - batch_index += 1; - } - } - app_log!( - "ui_separation.completed asset_id={} bound_nodes={} problematic_nodes={}", - asset_id, - separation.bound.len(), - separation.problematic_nodes.len() - ); - Ok(separation_dto(&separation)) -} - -async fn cut_processed_image( - source: PathBuf, - area: BindingArea, - target: PathBuf, -) -> Result<(), String> { - let started = Instant::now(); - let result = cut_processed_image_inner(source, area, target).await; - app_log!( - "ui_separation.cut_image.timing outcome={} elapsed_ms={}", - if result.is_ok() { "ok" } else { "error" }, - started.elapsed().as_millis() - ); - result -} - -async fn cut_processed_image_inner( - source: PathBuf, - area: BindingArea, - target: PathBuf, -) -> Result<(), String> { - app_log!( - "ui_separation.cut_image.start source_file={} target_file={} area=({}, {}, {}, {})", - source - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or(""), - target - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or(""), - area.global_pos_x_px, - area.global_pos_y_px, - area.width_px, - area.height_px - ); - tokio::task::spawn_blocking(move || cut_processed_image_blocking(&source, &area, &target)) - .await - .map_err(|error| format!("裁切处理图任务失败：{error}"))? -} - -fn cut_processed_image_blocking( - source: &Path, - area: &BindingArea, - target: &Path, -) -> Result<(), String> { - let image = image::open(source) - .map_err(|e| format!("读取处理图失败：{e}"))? - .to_rgba8(); - let normalized = normalize_binding_area(&image, *area)?; - let original_area = *area; - let normalized_area = normalized.area; - app_log!( - "ui_separation.cut_image.normalized changed={} clamped={} transparent={} original_area=({}, {}, {}, {}) normalized_area=({}, {}, {}, {})", - normalized.changed, - normalized.clamped, - normalized.transparent, - original_area.global_pos_x_px, - original_area.global_pos_y_px, - original_area.width_px, - original_area.height_px, - normalized_area.global_pos_x_px, - normalized_area.global_pos_y_px, - normalized_area.width_px, - normalized_area.height_px - ); - let cropped = image::imageops::crop_imm( - &image, - normalized_area.global_pos_x_px, - normalized_area.global_pos_y_px, - normalized_area.width_px, - normalized_area.height_px, - ) - .to_image(); - cropped - .save_with_format(target, ImageFormat::Png) - .map_err(|e| format!("写入 cut 图片失败：{e}"))?; - app_log!( - "ui_separation.cut_image.completed target_file={} width={} height={}", - target - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or(""), - normalized_area.width_px, - normalized_area.height_px - ); - Ok(()) -} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs new file mode 100644 index 000000000..a2c614930 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs @@ -0,0 +1,130 @@ +use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; +use crate::ui_editor::commands::separation::prompt::gen_binding_prompt; +use crate::ui_editor::commands::separation::{ + validate_binding_response, BindingResp, SeparationNode, +}; +use crate::ui_editor::commands::utils::{ + parse_limited_llm_tool_arguments, request_ui_editor_llm, run_with_repair_history, + strict_json_schema, +}; +use platform_llm::{ + LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice, +}; +use std::time::Instant; + +pub(super) async fn visual_binding( + source_url: String, + processed_url: String, + nodes: &[&SeparationNode], +) -> Result { + let started = Instant::now(); + let result = visual_binding_inner(source_url, processed_url, nodes).await; + app_log!( + "ui_separation.visual_binding.timing outcome={} elapsed_ms={} nodes={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis(), + nodes.len() + ); + result +} + +async fn visual_binding_inner( + source_url: String, + processed_url: String, + nodes: &[&SeparationNode], +) -> Result { + app_log!( + "ui_separation.visual_binding.start nodes={} source_url_chars={} processed_url_chars={}", + nodes.len(), + source_url.chars().count(), + processed_url.chars().count() + ); + let llm_config = load_game_creator_app_config() + .map_err(|e| { + app_log!("ui_separation.error stage=visual_binding reason=load_config error={e}"); + e.to_string() + })? + .llm; + let client = + build_game_creator_llm_client_from_llm_config(&llm_config, "llm").map_err(|e| { + app_log!("ui_separation.error stage=visual_binding reason=build_client error={e}"); + e.to_string() + })?; + let schema = strict_json_schema::().map_err(|error| { + app_log!("ui_separation.error stage=visual_binding reason=build_schema error={error}"); + error + })?; + let tool = LlmFunctionTool::new( + "bind_ui_elements", + "确认处理图中的区域对应哪些 UI 节点", + schema, + ) + .with_strict(true); + let initial_history = vec![ + LlmMessage::system(gen_binding_prompt(nodes.to_vec())), + LlmMessage::user_multimodal(vec![ + LlmMessageContentPart::InputText { + text: "processed image:".to_string(), + }, + LlmMessageContentPart::InputImage { + image_url: processed_url.clone(), + }, + LlmMessageContentPart::InputText { + text: "src image:".to_string(), + }, + LlmMessageContentPart::InputImage { + image_url: source_url.clone(), + }, + ]), + ]; + let result = run_with_repair_history( + 2, + initial_history, + |history| { + let tool = tool.clone(); + let client = client.clone(); + let llm_config = llm_config.clone(); + async move { + let request = LlmRunRequest::new(history) + .with_function_tools(vec![tool.clone()]) + .with_tool_choice(LlmToolChoice::Required); + let request_started = Instant::now(); + let response = request_ui_editor_llm(&client, &llm_config, request).await; + app_log!( + "ui_separation.llm.timing outcome={} elapsed_ms={}", + if response.is_ok() { "ok" } else { "error" }, + request_started.elapsed().as_millis() + ); + response + .map_err(|e| e.to_string()) + .and_then(|response| { + response + .tool_calls + .into_iter() + .find(|call| call.name == "bind_ui_elements") + .map(|call| call.arguments) + .ok_or_else(|| "视觉绑定模型未返回工具调用".to_string()) + }) + .and_then(|arguments| parse_limited_llm_tool_arguments(&arguments)) + .and_then(|args| { + serde_json::from_value::(args) + .map_err(|e| format!("视觉绑定结果无效：{e}")) + }) + } + }, + |value: &BindingResp| validate_binding_response(value, nodes), + ) + .await; + match &result { + Ok(value) => app_log!( + "ui_separation.visual_binding.completed nodes={} decisions={}", + nodes.len(), + value.decisions.len() + ), + Err(error) => app_log!( + "ui_separation.error stage=visual_binding reason=failed nodes={} error={error}", + nodes.len() + ), + } + result +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs new file mode 100644 index 000000000..8b083c8b5 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs @@ -0,0 +1,93 @@ +use crate::ui_editor::commands::separation::area::normalize_binding_area; +use crate::ui_editor::commands::separation::model::BindingArea; +use image::ImageFormat; +use std::path::{Path, PathBuf}; +use std::time::Instant; + +pub(super) async fn cut_processed_image( + source: PathBuf, + area: BindingArea, + target: PathBuf, +) -> Result<(), String> { + let started = Instant::now(); + let result = cut_processed_image_inner(source, area, target).await; + app_log!( + "ui_separation.cut_image.timing outcome={} elapsed_ms={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis() + ); + result +} + +async fn cut_processed_image_inner( + source: PathBuf, + area: BindingArea, + target: PathBuf, +) -> Result<(), String> { + app_log!( + "ui_separation.cut_image.start source_file={} target_file={} area=({}, {}, {}, {})", + source + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + area.global_pos_x_px, + area.global_pos_y_px, + area.width_px, + area.height_px + ); + tokio::task::spawn_blocking(move || cut_processed_image_blocking(&source, &area, &target)) + .await + .map_err(|error| format!("裁切处理图任务失败：{error}"))? +} + +fn cut_processed_image_blocking( + source: &Path, + area: &BindingArea, + target: &Path, +) -> Result<(), String> { + let image = image::open(source) + .map_err(|e| format!("读取处理图失败：{e}"))? + .to_rgba8(); + let normalized = normalize_binding_area(&image, *area)?; + let original_area = *area; + let normalized_area = normalized.area; + app_log!( + "ui_separation.cut_image.normalized changed={} clamped={} transparent={} original_area=({}, {}, {}, {}) normalized_area=({}, {}, {}, {})", + normalized.changed, + normalized.clamped, + normalized.transparent, + original_area.global_pos_x_px, + original_area.global_pos_y_px, + original_area.width_px, + original_area.height_px, + normalized_area.global_pos_x_px, + normalized_area.global_pos_y_px, + normalized_area.width_px, + normalized_area.height_px + ); + let cropped = image::imageops::crop_imm( + &image, + normalized_area.global_pos_x_px, + normalized_area.global_pos_y_px, + normalized_area.width_px, + normalized_area.height_px, + ) + .to_image(); + cropped + .save_with_format(target, ImageFormat::Png) + .map_err(|e| format!("写入 cut 图片失败：{e}"))?; + app_log!( + "ui_separation.cut_image.completed target_file={} width={} height={}", + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + normalized_area.width_px, + normalized_area.height_px + ); + Ok(()) +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/image_edit.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/image_edit.rs new file mode 100644 index 000000000..88a48d5f7 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/image_edit.rs @@ -0,0 +1,172 @@ +use crate::platform_session::PlatformSessionSnapshot; +use base64::Engine as _; +use serde::Deserialize; +use std::fs; +use std::path::PathBuf; +use std::time::Instant; + +#[derive(Deserialize)] +struct RawEditResponse { + data: Vec, +} + +#[derive(Deserialize)] +struct RawEditItem { + b64_json: String, +} + +pub(super) async fn raw_image_edit( + session: &PlatformSessionSnapshot, + image_data_url: &str, + prompt: &str, + width: u32, + height: u32, +) -> Result { + let started = Instant::now(); + let result = raw_image_edit_inner(session, image_data_url, prompt, width, height).await; + app_log!( + "ui_separation.image_edit.timing outcome={} elapsed_ms={} width={} height={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis(), + width, + height + ); + result +} + +async fn raw_image_edit_inner( + session: &PlatformSessionSnapshot, + image_data_url: &str, + prompt: &str, + width: u32, + height: u32, +) -> Result { + app_log!( + "ui_separation.image_edit.start width={} height={} prompt_chars={}", + width, + height, + prompt.chars().count() + ); + let (mime, data) = image_data_url + .split_once(',') + .ok_or_else(|| "界面图 data URL 无效".to_string())?; + let mime = mime + .strip_prefix("data:") + .and_then(|value| value.strip_suffix(";base64")) + .unwrap_or("image/png"); + if !mime.eq_ignore_ascii_case("image/png") { + return Err("图片分离请求只支持 PNG 源图".to_string()); + } + let image_bytes = base64::engine::general_purpose::STANDARD + .decode(data.trim()) + .map_err(|error| format!("解码源图失败：{error}"))?; + if image_bytes.is_empty() { + return Err("源图不能为空".to_string()); + } + let client = crate::http_client::agc_main_site_client_builder() + .build() + .map_err(|error| format!("创建图片编辑客户端失败：{error}"))?; + let url = format!( + "{}/api/raw/v1/images/edit", + session.api_base_url.trim_end_matches('/') + ); + let image_part = reqwest::multipart::Part::bytes(image_bytes) + .file_name("image.png") + .mime_str("image/png") + .map_err(|error| format!("构造图片编辑文件部件失败：{error}"))?; + let body = reqwest::multipart::Form::new() + .part("image", image_part) + .text("prompt", prompt.to_string()) + .text("width", width.to_string()) + .text("height", height.to_string()) + .text("output_format", "png") + .text("background", "transparent"); + let response = crate::http_client::with_agc_main_site_marker( + client + .post(url) + .bearer_auth(&session.access_token) + .multipart(body), + ) + .send() + .await + .map_err(|error| { + app_log!("ui_separation.error stage=image_edit reason=send error={error}"); + format!("图片分离请求失败：{error}") + })?; + if !response.status().is_success() { + app_log!( + "ui_separation.error stage=image_edit reason=http_status status={}", + response.status() + ); + return Err(format!("图片分离请求失败（HTTP {}）", response.status())); + } + let payload = response.json::().await.map_err(|error| { + app_log!("ui_separation.error stage=image_edit reason=parse_response error={error}"); + format!("解析图片分离响应失败：{error}") + })?; + let result = payload + .data + .into_iter() + .next() + .map(|item| format!("data:image/png;base64,{}", item.b64_json)) + .ok_or_else(|| "图片分离响应没有图像".to_string()); + match &result { + Ok(value) => app_log!( + "ui_separation.image_edit.completed data_url_chars={}", + value.chars().count() + ), + Err(error) => { + app_log!("ui_separation.error stage=image_edit reason=empty_result error={error}") + } + } + result +} + +pub(super) async fn write_processed_image( + processed_url: String, + target: PathBuf, +) -> Result<(), String> { + let started = Instant::now(); + let result = write_processed_image_inner(processed_url, target).await; + app_log!( + "ui_separation.processed_image.write.timing outcome={} elapsed_ms={}", + if result.is_ok() { "ok" } else { "error" }, + started.elapsed().as_millis() + ); + result +} + +async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> Result<(), String> { + app_log!( + "ui_separation.processed_image.write.start target_file={} data_url_chars={}", + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + processed_url.chars().count() + ); + tokio::task::spawn_blocking(move || { + let encoded = processed_url + .split_once(',') + .map(|(_, data)| data) + .ok_or_else(|| "处理图 data URL 无效".to_string())?; + let processed_bytes = base64::engine::general_purpose::STANDARD + .decode(encoded) + .map_err(|error| format!("解析处理图失败：{error}"))?; + let byte_len = processed_bytes.len(); + fs::write(&target, processed_bytes) + .map_err(|error| format!("写入处理图失败：{}: {error}", target.display())) + .map(|_| { + app_log!( + "ui_separation.processed_image.write.completed target_file={} bytes={}", + target + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + byte_len + ); + }) + }) + .await + .map_err(|error| format!("写入处理图任务失败：{error}"))? +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs new file mode 100644 index 000000000..0a9a5c652 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -0,0 +1,222 @@ +mod binding; +mod cut; +mod image_edit; +mod patch; + +pub use patch::apply_batch_patch; + +use super::model::*; +use super::persistence::{ + project_relative_path, read_separation_state, separation_dto, separation_sidecar_dir, + separation_state_path, write_separation_state, +}; +use super::prompt::gen_extract_prompt; +use super::tree::next_image_batch; +use crate::platform_session::current_platform_session; +use crate::ui_editor::commands::utils::read_ui_reference_image_data_url; +use crate::ui_editor::state::State; +use std::collections::HashMap; +use std::fs; +use std::path::Path; +use std::time::Instant; + +pub(crate) async fn separate_ui_impl( + project_path: String, + asset_id: String, + state: State, +) -> Result { + app_log!( + "ui_separation.start asset_id={} ui_trees={} ui_images={} sprites={}", + asset_id, + state.ui_trees.len(), + state.ui_design_images.len(), + state.sprite_assets.len() + ); + let session = current_platform_session().ok_or_else(|| "请先登录平台账号".to_string())?; + let root = Path::new(project_path.trim()); + let sidecar = separation_sidecar_dir(root, &asset_id).map_err(|error| { + app_log!( + "ui_separation.error stage=sidecar_dir asset_id={} error={error}", + asset_id + ); + error + })?; + fs::create_dir_all(&sidecar).map_err(|error| { + app_log!( + "ui_separation.error stage=sidecar_create asset_id={} error={error}", + asset_id + ); + format!("创建 separation sidecar 失败：{error}") + })?; + let state_path = separation_state_path(root, &asset_id)?; + let restored = state_path.exists(); + let mut separation = if restored { + app_log!("ui_separation.state_restore.start asset_id={asset_id}"); + read_separation_state(&state_path).map_err(|error| { + app_log!( + "ui_separation.error stage=state_restore asset_id={} error={error}", + asset_id + ); + error + })? + } else { + app_log!("ui_separation.state_construct.start asset_id={asset_id}"); + super::tree::construct_separation_state(&state) + }; + app_log!( + "ui_separation.state_ready asset_id={} restored={} trees={} bound={} problematic={}", + asset_id, + restored, + separation.trees.len(), + separation.bound.len(), + separation.problematic_nodes.len() + ); + + for tree_index in 0..separation.trees.len() { + let tree = &separation.trees[tree_index]; + let image_id = tree.src_ui_design.clone(); + let image = state + .ui_design_images + .get(&image_id) + .ok_or_else(|| "缺少源界面图".to_string())?; + let source_path = crate::project::resolve_local_project_path(root, &image.path)?; + let source_url = read_ui_reference_image_data_url(source_path) + .await + .map_err(|error| { + app_log!( + "ui_separation.error stage=read_source tree_index={} image_id={} error={error}", + tree_index, + image_id.as_str() + ); + error + })?; + app_log!( + "ui_separation.tree_start tree_index={} image_id={} width={} height={}", + tree_index, + image_id.as_str(), + image.pixel_size.x.round() as u32, + image.pixel_size.y.round() as u32 + ); + write_separation_state(&state_path, &separation).map_err(|error| { + app_log!( + "ui_separation.error stage=state_checkpoint tree_index={} error={error}", + tree_index + ); + error + })?; + let mut batch_index = 0usize; + loop { + let batch_started = Instant::now(); + let Some(current_tree) = separation.trees.get(tree_index) else { + break; + }; + let batch_nodes = next_image_batch(&separation, current_tree) + .into_iter() + .cloned() + .collect::>(); + if batch_nodes.is_empty() { + app_log!( + "ui_separation.tree_completed tree_index={} image_id={} bound={} problematic={}", + tree_index, image_id.as_str(), separation.bound.len(), separation.problematic_nodes.len() + ); + break; + } + let batch = batch_nodes.iter().collect::>(); + let prompt = gen_extract_prompt(&separation, current_tree, &batch); + app_log!( + "ui_separation.batch_start tree_index={} batch_index={} nodes={} prompt_chars={} rework_total={}", + tree_index, batch_index, batch.len(), prompt.chars().count(), + batch.iter().map(|node| node.rework_count).sum::() + ); + let processed_url = match image_edit::raw_image_edit( + &session, + &source_url, + &prompt, + image.pixel_size.x as u32, + image.pixel_size.y as u32, + ) + .await + { + Ok(value) => value, + Err(error) => { + app_log!("ui_separation.error stage=image_edit tree_index={} batch_index={} error={error}", tree_index, batch_index); + write_separation_state(&state_path, &separation)?; + return Err(error); + } + }; + let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); + if let Err(error) = + image_edit::write_processed_image(processed_url.clone(), processed_path.clone()) + .await + { + app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); + write_separation_state(&state_path, &separation)?; + return Err(error); + } + let binding = match binding::visual_binding(source_url.clone(), processed_url, &batch) + .await + { + Ok(value) => value, + Err(error) => { + app_log!("ui_separation.error stage=visual_binding tree_index={} batch_index={} error={error}", tree_index, batch_index); + write_separation_state(&state_path, &separation)?; + return Err(error); + } + }; + app_log!( + "ui_separation.binding_decisions tree_index={} batch_index={} decisions={}", + tree_index, + batch_index, + binding.decisions.len() + ); + let mut cut_paths = HashMap::new(); + let mut cut_error = None; + for decision in &binding.decisions { + if let BindingDecision::Ok { + to_node, + extracted_area, + } = decision + { + let cut_path = sidecar.join(format!("cut-{}.png", to_node.as_str())); + match cut::cut_processed_image( + processed_path.clone(), + *extracted_area, + cut_path.clone(), + ) + .await + { + Ok(()) => { + cut_paths + .insert(to_node.clone(), project_relative_path(root, &cut_path)?); + } + Err(error) => { + app_log!("ui_separation.error stage=cut_image tree_index={} batch_index={} node_id={} error={error}", tree_index, batch_index, to_node.as_str()); + cut_error = + Some(format!("节点 {} 的分离区域无效：{error}", to_node.as_str())); + break; + } + } + } + } + if let Some(error) = cut_error { + app_log!("ui_separation.error stage=cut_batch tree_index={} batch_index={} error={error}", tree_index, batch_index); + write_separation_state(&state_path, &separation)?; + return Err(error); + } + patch::apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; + write_separation_state(&state_path, &separation)?; + app_log!( + "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={} elapsed_ms={}", + tree_index, batch_index, cut_paths.len(), separation.bound.len(), separation.problematic_nodes.len(), batch_started.elapsed().as_millis() + ); + batch_index += 1; + } + } + app_log!( + "ui_separation.completed asset_id={} bound_nodes={} problematic_nodes={}", + asset_id, + separation.bound.len(), + separation.problematic_nodes.len() + ); + Ok(separation_dto(&separation)) +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs new file mode 100644 index 000000000..c2fdd0e84 --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs @@ -0,0 +1,98 @@ +use crate::ui_editor::commands::separation::{ + next_image_batch, validate_binding_response, BindingDecision, BindingResp, BoundNode, + ProblematicNode, SeparationNode, SeparationState, MAX_REWORK_COUNT, +}; +use crate::ui_editor::utils::NodeId; +use std::collections::HashMap; + +pub fn apply_batch_patch( + state: &mut SeparationState, + tree_index: usize, + decisions: &[BindingDecision], + cut_paths: &HashMap, +) -> Result<(), String> { + app_log!( + "ui_separation.batch_patch.start tree_index={} decisions={} cut_paths={}", + tree_index, + decisions.len(), + cut_paths.len() + ); + let batch_nodes = { + let tree = state + .trees + .get(tree_index) + .ok_or_else(|| "separation tree 索引无效".to_string())?; + next_image_batch(state, tree) + .into_iter() + .cloned() + .collect::>() + }; + let batch = batch_nodes.iter().collect::>(); + validate_binding_response( + &BindingResp { + decisions: decisions.to_vec(), + }, + &batch, + )?; + let rework_counts = batch_nodes + .iter() + .map(|node| (node.id.clone(), node.rework_count)) + .collect::>(); + let tree = state.trees.get_mut(tree_index).expect("tree index checked"); + for decision in decisions { + match decision { + BindingDecision::Ok { to_node, .. } => { + let path = cut_paths + .get(to_node) + .ok_or_else(|| format!("缺少节点 {} 的 cut 图片", to_node.as_str()))?; + state.bound.push(BoundNode { + node_id: to_node.clone(), + cut_image_path: path.clone(), + }); + } + BindingDecision::NeedRework { + to_node, + advice: problem_description, + } => { + append_rework_note(&mut tree.root, to_node, problem_description); + let count = rework_counts.get(to_node).copied().unwrap_or(0) + 1; + increment_rework_count(&mut tree.root, to_node, count); + if count >= MAX_REWORK_COUNT { + state.problematic_nodes.push(ProblematicNode { + node_id: to_node.clone(), + problem_description: problem_description.clone(), + rework_count: count, + }); + } + } + } + } + app_log!( + "ui_separation.batch_patch.completed tree_index={} bound={} problematic={} pending_root_children={}", + tree_index, + state.bound.len(), + state.problematic_nodes.len(), + tree.root.children.len() + ); + Ok(()) +} + +fn append_rework_note(node: &mut SeparationNode, id: &NodeId, note: &str) -> bool { + if node.id == *id { + node.note.rework_notes.push(note.to_string()); + return true; + } + node.children + .iter_mut() + .any(|child| append_rework_note(child, id, note)) +} + +fn increment_rework_count(node: &mut SeparationNode, id: &NodeId, count: u32) { + if node.id == *id { + node.rework_count = count; + return; + } + for child in &mut node.children { + increment_rework_count(child, id, count); + } +} diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 28a48888e..de3f0c774 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -102,6 +102,12 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - 图片编辑调用当前 Raw GPT Image 2 multipart 合同：`image`（PNG 文件）、`prompt`、`width`、`height`、`output_format=png`、`background=transparent`；不再发送旧 JSON/base64 请求体。 - 后端 raw operation 的持久状态与扣费后崩溃恢复窗口，遵循 Raw GPT Image 2 方案中的独立 TODO。 +## 实现组织 + +- separation prompt 按职责拆分为 `prompt/extract.rs` 与 `prompt/binding.rs`，由 `prompt/mod.rs` 统一导出。 +- separation workflow 按执行边界拆分为 `workflow/image_edit.rs`（Raw image-edit 与处理图写入）、`workflow/binding.rs`（视觉绑定）、`workflow/cut.rs`（像素归一化与裁切）、`workflow/patch.rs`（批次状态 patch）；`workflow/mod.rs` 仅负责批次编排与 sidecar 检查点。 +- 上述拆分只调整 Rust 模块边界，不改变批次选择、重试、sidecar 持久化、绑定校验或错误恢复语义。 + ## TODO - 正在执行 batch 的持久化和恢复。 From 7c7d55dcabb8b79c031cd7bd1c9804b040df33a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 14:54:00 +0800 Subject: [PATCH 105/248] =?UTF-8?q?=E9=87=8D=E6=9E=84=E6=8F=90=E5=8F=96?= =?UTF-8?q?=E6=8F=90=E7=A4=BA=E8=AF=8D=E7=94=9F=E6=88=90=E9=80=BB=E8=BE=91?= =?UTF-8?q?=EF=BC=8C=E7=A7=BB=E9=99=A4=E6=96=87=E6=9C=AC=E6=8B=BC=E6=8E=A5?= =?UTF-8?q?=E6=96=B9=E5=BC=8F=EF=BC=8C=E6=96=B0=E5=A2=9E=20YAML=20?= =?UTF-8?q?=E5=BA=8F=E5=88=97=E5=8C=96=E7=BB=93=E6=9E=84=E4=B8=8E=E6=B5=8B?= =?UTF-8?q?=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/separation/mod.rs | 31 --- .../commands/separation/prompt/extract.rs | 178 ++++++++++++------ ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 1 + 3 files changed, 124 insertions(+), 86 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 6b8545876..6e8a0f6fa 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -252,37 +252,6 @@ mod tests { assert!(next_image_batch(&separation, &separation.trees[0]).is_empty()); } - #[test] - fn next_extract_prompt_contains_previous_rework_notes() { - let node = SeparationNode { - id: NodeId::new("image").unwrap(), - kind: SeparationNodeKind::ImageTarget, - global_pos_x_px: 0, - global_pos_y_px: 0, - width_px: 1, - height_px: 1, - note: SeparationNote { - description: "图标".to_string(), - rework_notes: vec!["不要带父背景".to_string()], - }, - children: vec![], - rework_count: 0, - }; - let tree = SeparationTree { - src_ui_design: UIDesignImageId::new("page").unwrap(), - root: node.clone(), - root_extractable: true, - }; - let separation = SeparationState { - schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(), - trees: vec![tree.clone()], - bound: vec![], - problematic_nodes: vec![], - }; - let prompt = super::prompt::gen_extract_prompt(&separation, &tree, &[&node]); - assert!(prompt.contains("previous rework notes:\n- 不要带父背景")); - } - #[test] fn binding_validation_rejects_overlong_rework_note() { let node = SeparationNode { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs index be839b24b..e0059e2b9 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs @@ -1,8 +1,45 @@ +use crate::ui_editor::commands::separation::model::{SeparationState, SeparationTree}; use crate::ui_editor::commands::separation::{SeparationNode, SeparationNodeKind}; +use crate::ui_editor::utils::NodeId; +use serde::Serialize; use std::collections::HashSet; -use crate::ui_editor::commands::separation::model::{SeparationState, SeparationTree}; -use crate::ui_editor::utils::NodeId; +#[derive(Serialize)] +struct ExtractPromptDocument { + ui_layer_tree: ExtractPromptNode, +} + +#[derive(Serialize)] +struct ExtractPromptNode { + index: usize, + status: ExtractPromptStatus, + rect: ExtractPromptRect, + description: String, + #[serde(skip_serializing_if = "Vec::is_empty")] + rework_notes: Vec, + #[serde(skip_serializing_if = "Vec::is_empty")] + children: Vec, +} + +#[derive(Serialize)] +enum ExtractPromptStatus { + #[serde(rename = "OUTPUT_THIS_TURN")] + OutputThisTurn, + #[serde(rename = "DONE")] + Done, + #[serde(rename = "CONTEXT_ONLY")] + ContextOnly, + #[serde(rename = "REMOVE_ONLY")] + RemoveOnly, +} + +#[derive(Serialize)] +struct ExtractPromptRect { + x: u32, + y: u32, + width: u32, + height: u32, +} pub(crate) fn gen_extract_prompt( state: &SeparationState, @@ -36,66 +73,97 @@ pub(crate) fn gen_extract_prompt( .map(|node| node.node_id.clone()), ) .collect::>(); - let mut lines = Vec::new(); - let mut index = 1usize; - append_context_lines( - &tree.root, - 0, - true, - &target_ids, - &terminal_ids, - &mut index, - &mut lines, - ); - result.push_str(&lines.join("\n")); + let mut index = 1; + let document = ExtractPromptDocument { + ui_layer_tree: project_node(&tree.root, &target_ids, &terminal_ids, &mut index), + }; + let yaml = serde_yaml::to_string(&document) + .expect("UI separation extract prompt projection must be serializable"); + + result.push_str("```yaml\n"); + result.push_str(&yaml); + result.push_str("```\n"); result } -fn append_context_lines( +fn project_node( node: &SeparationNode, - depth: usize, - is_root: bool, target_ids: &HashSet, terminal_ids: &HashSet, index: &mut usize, - lines: &mut Vec, -) { - let status = match node.kind { - SeparationNodeKind::TextRemovalOnly => "REMOVE_ONLY", - SeparationNodeKind::PureContainer => "CONTEXT_ONLY", - SeparationNodeKind::ImageTarget => { - if target_ids.contains(&node.id) { - "OUTPUT_THIS_TURN" - } else if terminal_ids.contains(&node.id) { - "DONE" - } else { - "CONTEXT_ONLY" - } - } - }; - let role = if is_root { "root" } else { "child" }; - lines.push(format!( - "{}. [{}] depth={} role={} rect=({}, {}, {}, {}) {}", - *index, - status, - depth, - role, - node.global_pos_x_px, - node.global_pos_y_px, - node.width_px, - node.height_px, - node.note.as_prompt() - )); +) -> ExtractPromptNode { + let current_index = *index; *index += 1; - for child in &node.children { - append_context_lines( - child, - depth + 1, - false, - target_ids, - terminal_ids, - index, - lines, - ); + + let status = match node.kind { + SeparationNodeKind::TextRemovalOnly => ExtractPromptStatus::RemoveOnly, + SeparationNodeKind::PureContainer => ExtractPromptStatus::ContextOnly, + SeparationNodeKind::ImageTarget if target_ids.contains(&node.id) => { + ExtractPromptStatus::OutputThisTurn + } + SeparationNodeKind::ImageTarget if terminal_ids.contains(&node.id) => { + ExtractPromptStatus::Done + } + SeparationNodeKind::ImageTarget => ExtractPromptStatus::ContextOnly, + }; + + let children = node + .children + .iter() + .map(|child| project_node(child, target_ids, terminal_ids, index)) + .collect(); + + ExtractPromptNode { + index: current_index, + status, + rect: ExtractPromptRect { + x: node.global_pos_x_px, + y: node.global_pos_y_px, + width: node.width_px, + height: node.height_px, + }, + description: node.note.description.clone(), + rework_notes: node.note.rework_notes.clone(), + children, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ui_editor::commands::separation::model::SeparationNote; + + #[test] + fn projects_source_node_to_prompt_view() { + let node = SeparationNode { + id: NodeId::new("image").unwrap(), + kind: SeparationNodeKind::ImageTarget, + global_pos_x_px: 1, + global_pos_y_px: 2, + width_px: 3, + height_px: 4, + note: SeparationNote { + description: "按钮".to_string(), + rework_notes: vec!["保留圆角".to_string()], + }, + children: vec![], + rework_count: 0, + }; + let mut index = 1; + let target_ids = HashSet::from([node.id.clone()]); + let projected = project_node(&node, &target_ids, &HashSet::new(), &mut index); + + assert_eq!(projected.index, 1); + assert!(matches!( + projected.status, + ExtractPromptStatus::OutputThisTurn + )); + assert_eq!(projected.rect.x, 1); + assert_eq!(projected.rect.y, 2); + assert_eq!(projected.rect.width, 3); + assert_eq!(projected.rect.height, 4); + assert_eq!(projected.description, "按钮"); + assert_eq!(projected.rework_notes, vec!["保留圆角".to_string()]); + assert!(projected.children.is_empty()); } } diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index de3f0c774..5c756464b 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -45,6 +45,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - image-edit 直接使用原始 UI design PNG，不再生成或发送绿色框、紫色填充等额外辅助输入图。提取 prompt 直接描述完整页面分层清单和当前 batch 状态。 - 追加清单区分本轮 Image 输出目标、已完成 Image、仅作父子/遮挡上下文的 Image，以及只需从父图片移除的 Text。清单使用人类可读的编号、name/description、位置和层级，不向 image-edit 暴露 opaque NodeId。 +- 提取 prompt 先把 separation tree 投影为小型 YAML 视图，再注入固定规则文本：每个节点只包含展示编号、状态、`x/y/width/height` 矩形、描述、可选返工意见和递归 children；YAML 不携带 opaque NodeId，树的嵌套关系替代 `depth/role` 字段。 - 由于 raw endpoint 每次只返回一张 PNG，prompt 要求 image-edit 输出透明 atlas：本轮图片层可以移动和缩放，放置在不会互相遮挡的位置；视觉模型返回每层在 processed 图中的实际区域。源节点矩形只用于语义定位，不用于裁切区域推断。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 - 处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 From 8f680ef3767584527c3745de5d18389880994c73 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 15:09:44 +0800 Subject: [PATCH 106/248] =?UTF-8?q?=E8=A7=86=E8=A7=89=E7=BB=91=E5=AE=9A?= =?UTF-8?q?=E5=A2=9E=E5=8A=A0=E9=80=8F=E6=98=8E=E6=A0=87=E8=AE=B0=E9=A2=84?= =?UTF-8?q?=E5=A4=84=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增视觉绑定 RGBA 预处理与阈值复用 在请求前生成不透明洋红标记图并记录耗时 更新英文提示词说明标记色语义 --- .../src/ui_editor/commands/separation/area.rs | 72 +++++++++++-- .../commands/separation/image_preprocess.rs | 102 ++++++++++++++++++ .../commands/separation/prompt/binding.rs | 14 ++- .../commands/separation/workflow/binding.rs | 27 ++++- .../workflow/{image_edit.rs => extract.rs} | 0 ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 12 ++- 6 files changed, 210 insertions(+), 17 deletions(-) create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs rename apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/{image_edit.rs => extract.rs} (100%) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs index 30dd4e4cf..f39c38f10 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -7,6 +7,14 @@ use std::time::Instant; /// intentional workflow decision rather than a scattered numeric literal. pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX: u32 = 32; +/// Alpha values below this threshold are treated as transparent for boundary +/// detection. The cropped pixels themselves are preserved unchanged. +pub(crate) const MIN_VISIBLE_ALPHA: u8 = 16; + +/// An edge needs this many consecutive visible pixels to count as supported. +/// The requirement is reduced to the edge length for one-pixel-wide elements. +pub(crate) const MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS: usize = 2; + #[derive(Clone, Copy, Debug, PartialEq)] pub(crate) struct NormalizedBindingArea { pub(crate) area: BindingArea, @@ -61,7 +69,35 @@ impl Edge { const ALL: [Self; 4] = [Self::Left, Self::Right, Self::Top, Self::Bottom]; } +fn pixel_is_visible(alpha: u8) -> bool { + alpha >= MIN_VISIBLE_ALPHA +} + +fn has_consecutive_visible_pixels(alphas: I, required: usize) -> bool +where + I: IntoIterator, +{ + let required = required.max(1); + let mut consecutive = 0usize; + for alpha in alphas { + if pixel_is_visible(alpha) { + consecutive = consecutive.saturating_add(1); + if consecutive >= required { + return true; + } + } else { + consecutive = 0; + } + } + false +} + fn edge_has_visible_pixel(image: &RgbaImage, rect: Rect, edge: Edge) -> bool { + let edge_length = match edge { + Edge::Left | Edge::Right => rect.bottom - rect.top, + Edge::Top | Edge::Bottom => rect.right - rect.left, + } as usize; + let required = MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS.max(1).min(edge_length); match edge { Edge::Left | Edge::Right => { let x = if edge == Edge::Left { @@ -69,7 +105,10 @@ fn edge_has_visible_pixel(image: &RgbaImage, rect: Rect, edge: Edge) -> bool { } else { rect.right - 1 }; - (rect.top..rect.bottom).any(|y| image.get_pixel(x, y).0[3] > 0) + has_consecutive_visible_pixels( + (rect.top..rect.bottom).map(|y| image.get_pixel(x, y).0[3]), + required, + ) } Edge::Top | Edge::Bottom => { let y = if edge == Edge::Top { @@ -77,13 +116,17 @@ fn edge_has_visible_pixel(image: &RgbaImage, rect: Rect, edge: Edge) -> bool { } else { rect.bottom - 1 }; - (rect.left..rect.right).any(|x| image.get_pixel(x, y).0[3] > 0) + has_consecutive_visible_pixels( + (rect.left..rect.right).map(|x| image.get_pixel(x, y).0[3]), + required, + ) } } } fn rect_has_visible_pixel(image: &RgbaImage, rect: Rect) -> bool { - (rect.top..rect.bottom).any(|y| (rect.left..rect.right).any(|x| image.get_pixel(x, y).0[3] > 0)) + (rect.top..rect.bottom) + .any(|y| (rect.left..rect.right).any(|x| pixel_is_visible(image.get_pixel(x, y).0[3]))) } fn edge_direction(image: &RgbaImage, rect: Rect, edge: Edge) -> EdgeDirection { @@ -366,12 +409,25 @@ mod tests { } #[test] - fn keeps_nonzero_alpha_antialias_pixels() { + fn ignores_low_alpha_halo_while_preserving_visible_bounds() { let mut image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0])); - image.put_pixel(5, 6, Rgba([255, 255, 255, 1])); - image.put_pixel(7, 8, Rgba([255, 255, 255, 255])); - let result = normalize_binding_area(&image, area(4, 5, 5, 5)).unwrap(); - assert_eq!(result.area, area(5, 6, 3, 3)); + for y in 6..10 { + for x in 5..9 { + image.put_pixel(x, y, Rgba([255, 255, 255, 255])); + } + } + image.put_pixel(4, 7, Rgba([255, 255, 255, 1])); + image.put_pixel(9, 8, Rgba([255, 255, 255, 8])); + let result = normalize_binding_area(&image, area(4, 5, 6, 6)).unwrap(); + assert_eq!(result.area, area(5, 6, 4, 4)); + } + + #[test] + fn ignores_isolated_visible_edge_pixel() { + let mut image = image_with_rect(16, 16, 4, 4, 6, 8); + image.put_pixel(6, 4, Rgba([255, 255, 255, 255])); + let result = normalize_binding_area(&image, area(4, 4, 2, 4)).unwrap(); + assert_eq!(result.area, area(4, 4, 2, 4)); } #[test] diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs new file mode 100644 index 000000000..d3615ef4f --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs @@ -0,0 +1,102 @@ +use super::area::MIN_VISIBLE_ALPHA; +use base64::Engine as _; +use image::{ImageFormat, Rgba, RgbaImage}; +use std::fs; +use std::io::Cursor; +use std::path::{Path, PathBuf}; + +pub(crate) const VISUAL_BINDING_TRANSPARENT_MARKER_RGBA: [u8; 4] = [255, 0, 255, 255]; + +pub(crate) async fn preprocess_for_visual_binding( + processed_url: String, + sidecar: PathBuf, +) -> Result { + tokio::task::spawn_blocking(move || { + preprocess_for_visual_binding_blocking(&processed_url, &sidecar) + }) + .await + .map_err(|error| format!("视觉绑定预处理任务失败：{error}"))? +} + +fn preprocess_for_visual_binding_blocking( + processed_url: &str, + sidecar: &Path, +) -> Result { + let encoded = processed_url + .split_once(',') + .map(|(_, data)| data) + .ok_or_else(|| "处理图 data URL 无效".to_string())?; + let bytes = base64::engine::general_purpose::STANDARD + .decode(encoded.trim()) + .map_err(|error| format!("解析处理图失败：{error}"))?; + let mut image = image::load_from_memory(&bytes) + .map_err(|error| format!("解码处理图失败：{error}"))? + .to_rgba8(); + for pixel in image.pixels_mut() { + if pixel.0[3] < MIN_VISIBLE_ALPHA { + *pixel = Rgba(VISUAL_BINDING_TRANSPARENT_MARKER_RGBA); + } else { + pixel.0[3] = 255; + } + } + let mut png = Vec::new(); + image::DynamicImage::ImageRgba8(image) + .write_to(&mut Cursor::new(&mut png), ImageFormat::Png) + .map_err(|error| format!("编码视觉绑定预览失败：{error}"))?; + let debug_name = format!("binding-{}.png", uuid::Uuid::new_v4().simple()); + fs::write(sidecar.join(&debug_name), &png) + .map_err(|error| format!("写入视觉绑定预览失败：{error}"))?; + Ok(format!( + "data:image/png;base64,{}", + base64::engine::general_purpose::STANDARD.encode(png) + )) +} + +#[cfg(test)] +mod tests { + use super::*; + use image::Rgba; + use tempfile::tempdir; + + fn data_url(image: RgbaImage) -> String { + let mut bytes = Vec::new(); + image::DynamicImage::ImageRgba8(image) + .write_to(&mut Cursor::new(&mut bytes), ImageFormat::Png) + .expect("encode fixture"); + format!( + "data:image/png;base64,{}", + base64::engine::general_purpose::STANDARD.encode(bytes) + ) + } + + #[test] + fn preprocesses_alpha_using_existing_visibility_threshold() { + let mut image = RgbaImage::from_pixel(4, 1, Rgba([10, 20, 30, 255])); + image.put_pixel(0, 0, Rgba([1, 2, 3, 0])); + image.put_pixel(1, 0, Rgba([4, 5, 6, MIN_VISIBLE_ALPHA - 1])); + image.put_pixel(2, 0, Rgba([7, 8, 9, MIN_VISIBLE_ALPHA])); + image.put_pixel(3, 0, Rgba([11, 12, 13, 254])); + let directory = tempdir().expect("create sidecar fixture"); + + let url = preprocess_for_visual_binding_blocking(&data_url(image), directory.path()) + .expect("preprocess fixture"); + let encoded = url.split_once(',').expect("data URL").1; + let bytes = base64::engine::general_purpose::STANDARD + .decode(encoded) + .expect("decode output"); + let output = image::load_from_memory(&bytes) + .expect("decode output png") + .to_rgba8(); + + assert_eq!( + output.get_pixel(0, 0).0, + VISUAL_BINDING_TRANSPARENT_MARKER_RGBA + ); + assert_eq!( + output.get_pixel(1, 0).0, + VISUAL_BINDING_TRANSPARENT_MARKER_RGBA + ); + assert_eq!(output.get_pixel(2, 0).0, [7, 8, 9, 255]); + assert_eq!(output.get_pixel(3, 0).0, [11, 12, 13, 255]); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs index 0f621c248..2b77bf71c 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs @@ -1,7 +1,12 @@ +use crate::ui_editor::commands::separation::image_preprocess::VISUAL_BINDING_TRANSPARENT_MARKER_RGBA; use crate::ui_editor::commands::separation::SeparationNode; pub(crate) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { - let binding_system_prompt = r#" + let [marker_red, marker_green, marker_blue, marker_alpha] = + VISUAL_BINDING_TRANSPARENT_MARKER_RGBA; + let marker_color = format!("rgba({marker_red}, {marker_green}, {marker_blue}, {marker_alpha})"); + let binding_system_prompt = format!( + r#" You will be given a src UI design image and a processed image, where some ui elements are separated. You need to recognize and review the separation using the given tool. field notes: @@ -13,7 +18,10 @@ pub(crate) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { Here were the separation requirements: Preserve hard edges and the exact visible shape. - The processed image is a transparent atlas containing the requested image layers. + The processed image is an opaque visual-binding preview containing the requested image layers. + The solid color {marker_color} is an intentional transparency marker added by this workflow before this request. + It is not an image-edit defect and is not part of any UI element. + Do not include this marker color in the extracted area. Do not use the source node rectangle as the extracted area. And you should also review if the extracted's successfully meet the src image: * shape @@ -27,7 +35,7 @@ pub(crate) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { these node need handle: "# - .to_string(); + ); let mut result = binding_system_prompt; result.reserve(512); for elem in nodes { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs index a2c614930..d29dd226a 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs @@ -1,4 +1,5 @@ use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config}; +use crate::ui_editor::commands::separation::image_preprocess; use crate::ui_editor::commands::separation::prompt::gen_binding_prompt; use crate::ui_editor::commands::separation::{ validate_binding_response, BindingResp, SeparationNode, @@ -10,15 +11,17 @@ use crate::ui_editor::commands::utils::{ use platform_llm::{ LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice, }; +use std::path::PathBuf; use std::time::Instant; pub(super) async fn visual_binding( source_url: String, processed_url: String, + sidecar: PathBuf, nodes: &[&SeparationNode], ) -> Result { let started = Instant::now(); - let result = visual_binding_inner(source_url, processed_url, nodes).await; + let result = visual_binding_inner(source_url, processed_url, sidecar, nodes).await; app_log!( "ui_separation.visual_binding.timing outcome={} elapsed_ms={} nodes={}", if result.is_ok() { "ok" } else { "error" }, @@ -31,6 +34,7 @@ pub(super) async fn visual_binding( async fn visual_binding_inner( source_url: String, processed_url: String, + sidecar: PathBuf, nodes: &[&SeparationNode], ) -> Result { app_log!( @@ -39,6 +43,25 @@ async fn visual_binding_inner( source_url.chars().count(), processed_url.chars().count() ); + let preprocess_started = Instant::now(); + let binding_processed_url = + match image_preprocess::preprocess_for_visual_binding(processed_url, sidecar).await { + Ok(value) => { + app_log!( + "ui_separation.visual_binding.preprocess.timing outcome=ok elapsed_ms={}", + preprocess_started.elapsed().as_millis() + ); + value + } + Err(error) => { + app_log!( + "ui_separation.visual_binding.preprocess.timing outcome=error elapsed_ms={}", + preprocess_started.elapsed().as_millis() + ); + app_log!("ui_separation.error stage=visual_binding_preprocess error={error}"); + return Err(error); + } + }; let llm_config = load_game_creator_app_config() .map_err(|e| { app_log!("ui_separation.error stage=visual_binding reason=load_config error={e}"); @@ -67,7 +90,7 @@ async fn visual_binding_inner( text: "processed image:".to_string(), }, LlmMessageContentPart::InputImage { - image_url: processed_url.clone(), + image_url: binding_processed_url, }, LlmMessageContentPart::InputText { text: "src image:".to_string(), diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/image_edit.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs similarity index 100% rename from apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/image_edit.rs rename to apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index 5c756464b..e69506c13 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -44,14 +44,17 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## 图片编辑与视觉绑定 - image-edit 直接使用原始 UI design PNG，不再生成或发送绿色框、紫色填充等额外辅助输入图。提取 prompt 直接描述完整页面分层清单和当前 batch 状态。 +- 原因：部分视觉模型不会可靠读取 PNG alpha；image-edit 返回的处理图还可能出现只包含 `1..254`、缺少 `0` 和 `255` 的异常 alpha，导致 visual binding 无法稳定区分透明区域与素材内容。该问题只影响视觉模型的观察输入，不改变正式 cut 使用的 RGBA 真相。 - 追加清单区分本轮 Image 输出目标、已完成 Image、仅作父子/遮挡上下文的 Image，以及只需从父图片移除的 Text。清单使用人类可读的编号、name/description、位置和层级，不向 image-edit 暴露 opaque NodeId。 - 提取 prompt 先把 separation tree 投影为小型 YAML 视图，再注入固定规则文本：每个节点只包含展示编号、状态、`x/y/width/height` 矩形、描述、可选返工意见和递归 children；YAML 不携带 opaque NodeId，树的嵌套关系替代 `depth/role` 字段。 - 由于 raw endpoint 每次只返回一张 PNG，prompt 要求 image-edit 输出透明 atlas：本轮图片层可以移动和缩放，放置在不会互相遮挡的位置；视觉模型返回每层在 processed 图中的实际区域。源节点矩形只用于语义定位，不用于裁切区域推断。 - 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 - 处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 -- 视觉 binding 输入源图与处理图，只接收当前 batch 的 Image targets，必须为每个 Image target 恰好返回一次 `Ok` 或 `NeedRework`。每个决定继续携带 `to_node: NodeId`；Text 不出现在请求或 schema 中。 +- visual binding 请求前新增本地预处理：复用 `MIN_VISIBLE_ALPHA`，将 alpha 小于该阈值的像素替换为不透明洋红标记色 `[255, 0, 255, 255]`，其余像素保留 RGB 并将 alpha 设为 `255`。预处理图只用于 visual binding，原始 processed RGBA 继续用于 cut；不新增质量门禁、alpha 统计判定或重试。 +- 视觉 binding 输入源图与预处理后的不透明处理图，只接收当前 batch 的 Image targets，必须为每个 Image target 恰好返回一次 `Ok` 或 `NeedRework`。每个决定继续携带 `to_node: NodeId`；Text 不出现在请求或 schema 中。 +- binding prompt 使用英文明确说明：洋红色是本工作流在请求前注入的透明区域标记，不是 image-edit 缺陷，也不是 UI 素材；模型不得将该颜色计入 extracted area。颜色文本由 `VISUAL_BINDING_TRANSPARENT_MARKER_RGBA` 常量生成，避免提示词与实现漂移。 - `Ok` 返回 `NodeId + BindingArea`；Rust 仅校验 NodeId、区域边界和非零尺寸，不检查与原节点框的偏差，也不要求区域不重叠。 -- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 `alpha > 0`。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的位移上限。每条边最多相对原始 area 移动 `32px`，由模块级常量 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX` 定义，与原始 area 尺寸无关。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到该常量上限、图像边界或非零尺寸约束。性能优化列 TODO。 +- cut 前会对视觉模型返回的 `BindingArea` 做本地像素边界归一化。处理图是透明 PNG，有效像素定义为 alpha 不低于模块级常量 `MIN_VISIBLE_ALPHA`（当前为 `16`）；边缘扫描还要求至少连续 `MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS` 个有效像素（当前为 `2`），避免半透明光晕和孤立噪点驱动边界移动。四条边以模型 area 为起点，每条边根据首次扫描结果固定方向：边上无有效像素则只向内收缩，边上有有效像素则只向外扩展；四边每轮从同一矩形快照同时逐像素推进，直到达到“内侧有像素、外侧无像素”的分界、图像边界或每条边相对原始 area 的位移上限。每条边最多相对原始 area 移动 `32px`，由模块级常量 `MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX` 定义，与原始 area 尺寸无关。方向固定用于避免稀疏像素造成边界来回振荡；没有理想分界时使用受限范围内的最终 area，不重新请求视觉模型，也不转 problematic。全透明处理图不走特殊错误分支，仍沿同一规则得到最终 area 后裁切。归一化只影响本地 cut，不改写原始 `BindingDecision`、sidecar 或 DTO；日志记录原始 area、最终 area、是否变更，以及仍需移动时是否受到该常量上限、图像边界或非零尺寸约束。性能优化列 TODO。 - `NeedRework` 携带短问题描述（最多 512 个 Unicode 字符）；通过校验后按产生顺序追加到目标 `SeparationNode.note.rework_notes`，下一次该节点进入 image-edit 时全部意见会注入提取 prompt。结构化工具调用失败时使用可复用 repair harness，把错误反馈给模型并额外请求一次；image-edit 不使用该 harness。 - 达到返工上限时仍先保留最后一条视觉模型意见，再把节点追加到 problematic；网络、IO、裁切等基础设施错误不写入节点意见。 - 达到模块级重做常量后，节点移入 problematic；不中断整条工作流，最终统一通知用户。image-edit、图像写入或裁切失败保留当前 state 并返回错误，不自动把整批标记为 problematic。 @@ -67,7 +70,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - `SeparationNode.kind` 与 tree children 一起持久化；当前数据结构变更提升 separation state schema 版本，不提供旧 sidecar 迁移或回退。 - sidecar 只在 separation 未完成期间存在；完成后删除 state JSON。 - 当前只持久化已经完成的 batch；正在执行 batch 的恢复语义列 TODO。 -- 临时图片可跨重启保留。image-edit 返回的 processed 图和 cut 图片当前都保留用于 debug；理论上 processed 中间图只应在内存中，清理/归档策略列 TODO。 +- 临时图片可跨重启保留。image-edit 返回的 processed 图、visual binding 预处理图和 cut 图片当前都保留用于 debug；预处理图位于同一 sidecar，命名为 `binding-<随机 UUID>.png`，不写入 separation state。清理/归档策略列 TODO。 - 并发边界：当前由前端 `isSeparating` 与 `runWithStateLocked` 保证同一 UI 编辑会话 同时只有一次 separation。sidecar 是临时恢复状态，不是正式 UI 资产真相，不参与 manifest 或项目 revision，因此当前不额外持有项目写锁；若未来支持多窗口/多进程并发， @@ -106,7 +109,8 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 ## 实现组织 - separation prompt 按职责拆分为 `prompt/extract.rs` 与 `prompt/binding.rs`，由 `prompt/mod.rs` 统一导出。 -- separation workflow 按执行边界拆分为 `workflow/image_edit.rs`（Raw image-edit 与处理图写入）、`workflow/binding.rs`（视觉绑定）、`workflow/cut.rs`（像素归一化与裁切）、`workflow/patch.rs`（批次状态 patch）；`workflow/mod.rs` 仅负责批次编排与 sidecar 检查点。 +- separation workflow 按执行边界拆分为 `workflow/image_edit.rs`（Raw image-edit 与处理图写入）、`workflow/binding.rs`（视觉绑定）、`workflow/cut.rs`（像素归一化与裁切）、`workflow/patch.rs`（批次状态 patch）；新增 `image_preprocess.rs`（visual binding 请求前的 RGBA 标记图转换、PNG 写入和 data URL 生成）；`workflow/mod.rs` 仅负责批次编排与 sidecar 检查点。 +- `image_preprocess.rs` 的像素转换和 debug 文件写入运行在独立 `spawn_blocking` 任务；`visual_binding` 记录预处理阶段的 `outcome` 与 `elapsed_ms`，不记录图片内容或绝对路径。预处理单元测试只验证像素转换和 PNG 可解码，不把 debug 文件是否存在作为测试契约。 - 上述拆分只调整 Rust 模块边界，不改变批次选择、重试、sidecar 持久化、绑定校验或错误恢复语义。 ## TODO From b9b4ebd38e911e944b05f79b7e697fbd70f63652 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 15:18:05 +0800 Subject: [PATCH 107/248] =?UTF-8?q?=E9=87=8D=E5=91=BD=E5=90=8D=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E4=B8=BA=E6=8F=90=E5=8F=96=E6=A8=A1?= =?UTF-8?q?=E5=9D=97=E5=B9=B6=E6=9B=B4=E6=96=B0=E7=9B=B8=E5=85=B3=E8=B0=83?= =?UTF-8?q?=E7=94=A8=E7=BB=98=E5=88=B6=E6=B5=81=E7=A8=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/separation/mod.rs | 1 + .../commands/separation/workflow/extract.rs | 6 +++--- .../ui_editor/commands/separation/workflow/mod.rs | 15 ++++++++++----- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 6e8a0f6fa..5350db82d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -1,4 +1,5 @@ mod area; +pub(crate) mod image_preprocess; mod model; mod persistence; mod prompt; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs index 88a48d5f7..45096b14a 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs @@ -15,7 +15,7 @@ struct RawEditItem { b64_json: String, } -pub(super) async fn raw_image_edit( +pub(super) async fn raw_extract( session: &PlatformSessionSnapshot, image_data_url: &str, prompt: &str, @@ -23,7 +23,7 @@ pub(super) async fn raw_image_edit( height: u32, ) -> Result { let started = Instant::now(); - let result = raw_image_edit_inner(session, image_data_url, prompt, width, height).await; + let result = raw_extract_inner(session, image_data_url, prompt, width, height).await; app_log!( "ui_separation.image_edit.timing outcome={} elapsed_ms={} width={} height={}", if result.is_ok() { "ok" } else { "error" }, @@ -34,7 +34,7 @@ pub(super) async fn raw_image_edit( result } -async fn raw_image_edit_inner( +async fn raw_extract_inner( session: &PlatformSessionSnapshot, image_data_url: &str, prompt: &str, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 0a9a5c652..b6b245a8b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -1,6 +1,6 @@ mod binding; mod cut; -mod image_edit; +mod extract; mod patch; pub use patch::apply_batch_patch; @@ -128,7 +128,7 @@ pub(crate) async fn separate_ui_impl( tree_index, batch_index, batch.len(), prompt.chars().count(), batch.iter().map(|node| node.rework_count).sum::() ); - let processed_url = match image_edit::raw_image_edit( + let processed_url = match extract::raw_extract( &session, &source_url, &prompt, @@ -146,15 +146,20 @@ pub(crate) async fn separate_ui_impl( }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); if let Err(error) = - image_edit::write_processed_image(processed_url.clone(), processed_path.clone()) + extract::write_processed_image(processed_url.clone(), processed_path.clone()) .await { app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); write_separation_state(&state_path, &separation)?; return Err(error); } - let binding = match binding::visual_binding(source_url.clone(), processed_url, &batch) - .await + let binding = match binding::visual_binding( + source_url.clone(), + processed_url, + sidecar.clone(), + &batch, + ) + .await { Ok(value) => value, Err(error) => { From 9acc23b848788cc7f134483460b5fa540f0b7b0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 15:34:30 +0800 Subject: [PATCH 108/248] =?UTF-8?q?=E6=8A=BD=E7=A6=BB=20DFS=20=E6=89=B9?= =?UTF-8?q?=E6=AC=A1=E9=80=BB=E8=BE=91=E8=87=B3=E7=8B=AC=E7=AB=8B=E6=A8=A1?= =?UTF-8?q?=E5=9D=97=EF=BC=8C=E5=B9=B6=E4=BC=98=E5=8C=96=E8=8A=82=E7=82=B9?= =?UTF-8?q?=E9=9D=A2=E7=A7=AF=E8=AE=A1=E7=AE=97=E4=B8=8E=E7=BB=88=E7=AB=AF?= =?UTF-8?q?=E8=BF=87=E6=BB=A4=E6=B5=81=E7=A8=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src/ui_editor/commands/separation/mod.rs | 1 + .../src/ui_editor/commands/separation/tree.rs | 63 ------------------ .../commands/separation/workflow/batch.rs | 65 +++++++++++++++++++ .../commands/separation/workflow/mod.rs | 3 +- .../commands/separation/workflow/patch.rs | 5 +- 5 files changed, 71 insertions(+), 66 deletions(-) create mode 100644 apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 5350db82d..a0b15a162 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -10,6 +10,7 @@ pub use model::*; pub use persistence::*; pub use tree::*; pub use workflow::apply_batch_patch; +pub use workflow::batch::next_image_batch; pub(crate) use workflow::separate_ui_impl; #[cfg(test)] mod tests { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 01c46b583..0b483d8e4 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -2,7 +2,6 @@ use super::model::*; use crate::ui_editor::component::{image::ImageComponent, Component}; use crate::ui_editor::layout::node::Node; use crate::ui_editor::state::State; -use crate::ui_editor::utils::NodeId; use std::collections::HashSet; fn is_unbound_image(node: &Node) -> bool { @@ -144,68 +143,6 @@ pub fn construct_separation_state(state: &State) -> SeparationState { } } -fn terminal_ids(state: &SeparationState) -> HashSet { - state - .bound - .iter() - .map(|n| n.node_id.clone()) - .chain(state.problematic_nodes.iter().map(|n| n.node_id.clone())) - .collect() -} - -fn collect_dfs_batch<'a>( - node: &'a SeparationNode, - is_root: bool, - root_extractable: bool, - terminal: &HashSet, - selected: &mut Vec<&'a SeparationNode>, - area: &mut u64, -) -> bool { - let is_target = matches!(node.kind, SeparationNodeKind::ImageTarget) - && (!is_root || root_extractable) - && !terminal.contains(&node.id); - if is_target { - let node_area = u64::from(node.width_px).saturating_mul(u64::from(node.height_px)); - let would_exceed = area.saturating_add(node_area) > IMAGE_EDIT_AREA_LIMIT_PX; - if selected.is_empty() || !would_exceed { - selected.push(node); - *area = area.saturating_add(node_area); - } else { - return true; - } - } - for child in &node.children { - if collect_dfs_batch(child, false, root_extractable, terminal, selected, area) { - return true; - } - } - false -} - -pub fn next_image_batch<'a>( - state: &SeparationState, - tree: &'a SeparationTree, -) -> Vec<&'a SeparationNode> { - let terminal = terminal_ids(state); - let mut selected = Vec::new(); - let mut area = 0; - collect_dfs_batch( - &tree.root, - true, - tree.root_extractable, - &terminal, - &mut selected, - &mut area, - ); - app_log!( - "ui_separation.batch_selected image_id={} image_nodes={} area_px={}", - tree.src_ui_design.as_str(), - selected.len(), - area - ); - selected -} - pub fn validate_binding_response( response: &BindingResp, batch: &[&SeparationNode], diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs new file mode 100644 index 000000000..487fc312c --- /dev/null +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs @@ -0,0 +1,65 @@ +use crate::ui_editor::commands::separation::model::*; +use crate::ui_editor::utils::NodeId; +use std::collections::HashSet; + +fn terminal_ids(state: &SeparationState) -> HashSet { + state + .bound + .iter() + .map(|n| n.node_id.clone()) + .chain(state.problematic_nodes.iter().map(|n| n.node_id.clone())) + .collect() +} + +fn collect_dfs_batch<'a>( + node: &'a SeparationNode, + is_root: bool, + root_extractable: bool, + terminal: &HashSet, + selected: &mut Vec<&'a SeparationNode>, + area: &mut u64, +) -> bool { + let is_target = matches!(node.kind, SeparationNodeKind::ImageTarget) + && (!is_root || root_extractable) + && !terminal.contains(&node.id); + if is_target { + let node_area = u64::from(node.width_px).saturating_mul(u64::from(node.height_px)); + let would_exceed = area.saturating_add(node_area) > IMAGE_EDIT_AREA_LIMIT_PX; + if selected.is_empty() || !would_exceed { + selected.push(node); + *area = area.saturating_add(node_area); + } else { + return true; + } + } + for child in &node.children { + if collect_dfs_batch(child, false, root_extractable, terminal, selected, area) { + return true; + } + } + false +} + +pub fn next_image_batch<'a>( + state: &SeparationState, + tree: &'a SeparationTree, +) -> Vec<&'a SeparationNode> { + let terminal = terminal_ids(state); + let mut selected = Vec::new(); + let mut area = 0; + collect_dfs_batch( + &tree.root, + true, + tree.root_extractable, + &terminal, + &mut selected, + &mut area, + ); + app_log!( + "ui_separation.batch_selected image_id={} image_nodes={} area_px={}", + tree.src_ui_design.as_str(), + selected.len(), + area + ); + selected +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index b6b245a8b..4bba6a8da 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -1,3 +1,4 @@ +pub mod batch; mod binding; mod cut; mod extract; @@ -5,13 +6,13 @@ mod patch; pub use patch::apply_batch_patch; +use self::batch::next_image_batch; use super::model::*; use super::persistence::{ project_relative_path, read_separation_state, separation_dto, separation_sidecar_dir, separation_state_path, write_separation_state, }; use super::prompt::gen_extract_prompt; -use super::tree::next_image_batch; use crate::platform_session::current_platform_session; use crate::ui_editor::commands::utils::read_ui_reference_image_data_url; use crate::ui_editor::state::State; diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs index c2fdd0e84..341416fd6 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs @@ -1,6 +1,7 @@ +use super::batch::next_image_batch; use crate::ui_editor::commands::separation::{ - next_image_batch, validate_binding_response, BindingDecision, BindingResp, BoundNode, - ProblematicNode, SeparationNode, SeparationState, MAX_REWORK_COUNT, + validate_binding_response, BindingDecision, BindingResp, BoundNode, ProblematicNode, + SeparationNode, SeparationState, MAX_REWORK_COUNT, }; use crate::ui_editor::utils::NodeId; use std::collections::HashMap; From 4bdbab29c0c122fffc93ae82dde8c433a3414c51 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:25:25 +0800 Subject: [PATCH 109/248] =?UTF-8?q?=E4=BC=98=E5=8C=96=E5=8E=9F=E5=A7=8B?= =?UTF-8?q?=E5=9B=BE=E7=89=87=E8=AF=B7=E6=B1=82=E7=BC=93=E5=86=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 ReferenceImage 与 multipart 输入改用 bytes::Bytes。 通过流式 multipart part 避免 reqwest 路径的额外整图复制。 --- server-rs/crates/api-server/src/raw_image.rs | 13 +++++++------ server-rs/crates/platform-image/Cargo.toml | 1 + .../platform-image/src/vector_engine/client.rs | 2 +- .../src/vector_engine/curl_transport.rs | 4 ++-- .../src/vector_engine/image_source.rs | 5 +++-- .../platform-image/src/vector_engine/raw_edit.rs | 4 ++-- .../platform-image/src/vector_engine/request.rs | 6 +++--- .../platform-image/src/vector_engine/types.rs | 3 ++- .../crates/platform-image/tests/vector_engine.rs | 4 ++-- 9 files changed, 23 insertions(+), 19 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index fc0d33a6d..b32883a0f 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -3,6 +3,7 @@ use axum::{ extract::{Extension, Multipart, State}, http::StatusCode, }; +use bytes::Bytes; use image::{GenericImageView, ImageFormat, ImageReader}; use platform_image::{ RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RawImageEditOptions, ReferenceImage, @@ -30,7 +31,7 @@ use time::OffsetDateTime; #[derive(Debug)] struct RawImageData { - pub(crate) bytes: Vec, + pub(crate) bytes: Bytes, pub(crate) mime_type: String, pub(crate) file_name: String, } @@ -229,7 +230,7 @@ async fn read_multipart_image( return Err(bad_request(format!("{name} 文件不能为空"))); } Ok(RawImageData { - bytes: bytes.to_vec(), + bytes, mime_type: "image/png".to_string(), file_name: format!("{name}.png"), }) @@ -425,12 +426,12 @@ mod tests { fn request(image: Vec, mask: Option>) -> RawImageEditRequest { RawImageEditRequest { image: RawImageData { - bytes: image, + bytes: Bytes::from(image), mime_type: "image/png".to_string(), file_name: "image.png".to_string(), }, mask: mask.map(|bytes| RawImageData { - bytes, + bytes: Bytes::from(bytes), mime_type: "image/png".to_string(), file_name: "mask.png".to_string(), }), @@ -520,7 +521,7 @@ mod tests { let invalid_bytes = RawImageEditRequest { image: RawImageData { - bytes: b"hello".to_vec(), + bytes: Bytes::from_static(b"hello"), mime_type: "image/png".to_string(), file_name: "image.png".to_string(), }, @@ -530,7 +531,7 @@ mod tests { let invalid_mime = RawImageEditRequest { image: RawImageData { - bytes: png_bytes(1, 1), + bytes: Bytes::from(png_bytes(1, 1)), mime_type: "image/jpeg".to_string(), file_name: "image.jpg".to_string(), }, diff --git a/server-rs/crates/platform-image/Cargo.toml b/server-rs/crates/platform-image/Cargo.toml index 2030dab06..c4acbcd2a 100644 --- a/server-rs/crates/platform-image/Cargo.toml +++ b/server-rs/crates/platform-image/Cargo.toml @@ -6,6 +6,7 @@ license.workspace = true [dependencies] base64 = { workspace = true } +bytes = { workspace = true } curl = { workspace = true } image = { workspace = true, features = ["jpeg", "png", "webp"] } reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls"] } diff --git a/server-rs/crates/platform-image/src/vector_engine/client.rs b/server-rs/crates/platform-image/src/vector_engine/client.rs index 24b10196a..0dbf0be84 100644 --- a/server-rs/crates/platform-image/src/vector_engine/client.rs +++ b/server-rs/crates/platform-image/src/vector_engine/client.rs @@ -1107,7 +1107,7 @@ mod tests { fn reference_image(index: usize) -> ReferenceImage { ReferenceImage { - bytes: vec![index as u8], + bytes: bytes::Bytes::from(vec![index as u8]), mime_type: "image/png".to_string(), file_name: format!("reference-{index}.png"), } diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index fbe94e1b1..0c981f1a6 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -257,7 +257,7 @@ fn send_multipart_edit_request_with_curl_blocking( form.part("image") .buffer( reference_image.file_name.as_str(), - reference_image.bytes.clone(), + reference_image.bytes.to_vec(), ) .content_type(reference_image.mime_type.as_str()) .add()?; @@ -338,7 +338,7 @@ mod tests { "1024x1024", 1, &[ReferenceImage { - bytes: b"reference".to_vec(), + bytes: bytes::Bytes::from_static(b"reference"), mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }], diff --git a/server-rs/crates/platform-image/src/vector_engine/image_source.rs b/server-rs/crates/platform-image/src/vector_engine/image_source.rs index dbc0b38a4..29fe42a45 100644 --- a/server-rs/crates/platform-image/src/vector_engine/image_source.rs +++ b/server-rs/crates/platform-image/src/vector_engine/image_source.rs @@ -1,4 +1,5 @@ use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; +use bytes::Bytes; use reqwest::header; use std::time::Instant; @@ -161,7 +162,7 @@ pub(crate) async fn resolve_reference_images( .await .map_err(|error| contextualize_reference_download_error(error, failure_context))?; resolved.push(ReferenceImage { - bytes: downloaded.bytes, + bytes: Bytes::from(downloaded.bytes), mime_type: downloaded.mime_type.clone(), file_name: format!( "reference-{index}.{}", @@ -213,7 +214,7 @@ pub(crate) fn parse_reference_image_data_url( })?; let mime_type = normalize_downloaded_image_mime_type(mime_type); Ok(Some(ReferenceImage { - bytes, + bytes: Bytes::from(bytes), file_name: format!( "reference-{index}.{}", mime_to_extension(mime_type.as_str()) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 984d37add..02f08c998 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -138,7 +138,7 @@ pub async fn create_vector_engine_raw_image_edit( .text("size", format!("{}x{}", options.width, options.height)) .part( "image", - Part::bytes(image_bytes) + Part::stream_with_length(image_bytes.clone(), image_bytes.len() as u64) .file_name(image_file_name) .mime_str(image_mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, @@ -155,7 +155,7 @@ pub async fn create_vector_engine_raw_image_edit( if let Some(mask) = options.mask { form = form.part( "mask", - Part::bytes(mask.bytes) + Part::stream_with_length(mask.bytes.clone(), mask.bytes.len() as u64) .file_name(mask.file_name) .mime_str(mask.mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, diff --git a/server-rs/crates/platform-image/src/vector_engine/request.rs b/server-rs/crates/platform-image/src/vector_engine/request.rs index 4dc53443e..fe331625b 100644 --- a/server-rs/crates/platform-image/src/vector_engine/request.rs +++ b/server-rs/crates/platform-image/src/vector_engine/request.rs @@ -65,7 +65,7 @@ pub fn build_vector_engine_nanobanana_generate_content_request_body( "mime_type": reference_image.mime_type, "data": base64::Engine::encode( &base64::engine::general_purpose::STANDARD, - reference_image.bytes.as_slice() + reference_image.bytes.as_ref() ), }, })); @@ -346,12 +346,12 @@ mod tests { 9, &[ ReferenceImage { - bytes: vec![1, 2, 3, 4, 5], + bytes: bytes::Bytes::from_static(&[1, 2, 3, 4, 5]), mime_type: "image/png".to_string(), file_name: "reference-a.png".to_string(), }, ReferenceImage { - bytes: vec![8; 7], + bytes: bytes::Bytes::from(vec![8; 7]), mime_type: "image/jpeg".to_string(), file_name: "reference-b.jpg".to_string(), }, diff --git a/server-rs/crates/platform-image/src/vector_engine/types.rs b/server-rs/crates/platform-image/src/vector_engine/types.rs index d25ad5c12..0ceb6c4c3 100644 --- a/server-rs/crates/platform-image/src/vector_engine/types.rs +++ b/server-rs/crates/platform-image/src/vector_engine/types.rs @@ -1,4 +1,5 @@ use super::audit::PlatformImageFailureAudit; +use bytes::Bytes; #[derive(Clone, Debug)] pub struct VectorEngineImageSettings { @@ -31,7 +32,7 @@ pub struct DownloadedImage { #[derive(Clone, Debug)] pub struct ReferenceImage { - pub bytes: Vec, + pub bytes: Bytes, pub mime_type: String, pub file_name: String, } diff --git a/server-rs/crates/platform-image/tests/vector_engine.rs b/server-rs/crates/platform-image/tests/vector_engine.rs index f1bd4470b..c764d3e67 100644 --- a/server-rs/crates/platform-image/tests/vector_engine.rs +++ b/server-rs/crates/platform-image/tests/vector_engine.rs @@ -259,7 +259,7 @@ async fn vector_engine_image_edit_retries_send_timeout_once_and_succeeds() { let http_client = build_vector_engine_image_http_client(&settings).expect("client should build"); let reference_image = ReferenceImage { - bytes: b"reference".to_vec(), + bytes: bytes::Bytes::from_static(b"reference"), mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }; @@ -598,7 +598,7 @@ async fn vector_engine_image_edit_falls_back_when_preferred_model_is_unsupported let http_client = build_vector_engine_image_http_client(&settings).expect("client should build"); let reference = ReferenceImage { - bytes: b"reference".to_vec(), + bytes: bytes::Bytes::from_static(b"reference"), mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }; From 46df12c08667e4a373b1bf6db411acd253d64786 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:26:33 +0800 Subject: [PATCH 110/248] =?UTF-8?q?=E6=94=B6=E6=95=9B=E5=8E=9F=E5=A7=8B?= =?UTF-8?q?=E5=9B=BE=E7=89=87=20multipart=20=E9=94=99=E8=AF=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 服务端记录解析原始错误，客户端仅返回稳定分类文案。 --- server-rs/crates/api-server/src/raw_image.rs | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index b32883a0f..1b071e05e 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -168,7 +168,10 @@ async fn parse_multipart_request( while let Some(field) = multipart .next_field() .await - .map_err(|error| bad_request(format!("multipart 字段读取失败：{error}")))? + .map_err(|error| { + tracing::warn!(error = %error, "raw image multipart 字段解析失败"); + bad_request("multipart 请求无效") + })? { let name = field .name() @@ -225,7 +228,10 @@ async fn read_multipart_image( let bytes = field .bytes() .await - .map_err(|error| bad_request(format!("{name} 文件读取失败：{error}")))?; + .map_err(|error| { + tracing::warn!(field = name, error = %error, "raw image multipart 图片读取失败"); + bad_request(format!("{name} 字段读取失败")) + })?; if bytes.is_empty() { return Err(bad_request(format!("{name} 文件不能为空"))); } @@ -244,12 +250,10 @@ async fn set_text_field( if target.is_some() { return Err(bad_request(format!("{name} 字段不能重复"))); } - *target = Some( - field - .text() - .await - .map_err(|error| bad_request(format!("{name} 字段读取失败：{error}")))?, - ); + *target = Some(field.text().await.map_err(|error| { + tracing::warn!(field = name, error = %error, "raw image multipart 文本读取失败"); + bad_request(format!("{name} 字段读取失败")) + })?); Ok(()) } From 48c5638893ed03234f597e20a5f83e9cb2a9123d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:28:13 +0800 Subject: [PATCH 111/248] =?UTF-8?q?=E6=89=A9=E5=A4=A7=E5=8E=9F=E5=A7=8B?= =?UTF-8?q?=E5=9B=BE=E7=89=87=E6=96=87=E6=9C=AC=E5=AD=97=E6=AE=B5=E4=B8=8A?= =?UTF-8?q?=E9=99=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit multipart 文本字段改为 chunk 流式读取并限制 16KB。 统一 UTF-8 校验和超限错误，避免先完整分配超长 prompt。 --- server-rs/crates/api-server/src/raw_image.rs | 31 +++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 1b071e05e..7bde75405 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -58,7 +58,7 @@ pub(crate) struct RawImageEditResponse { pub(crate) data: Vec, } -const RAW_IMAGE_MAX_PROMPT_BYTES: usize = 4 * 1024; +const RAW_IMAGE_MAX_TEXT_FIELD_BYTES: usize = 16 * 1024; pub(crate) async fn edit_raw_image( State(state): State, @@ -244,15 +244,32 @@ async fn read_multipart_image( async fn set_text_field( target: &mut Option, - field: axum::extract::multipart::Field<'_>, + mut field: axum::extract::multipart::Field<'_>, name: &str, ) -> Result<(), AppError> { if target.is_some() { return Err(bad_request(format!("{name} 字段不能重复"))); } - *target = Some(field.text().await.map_err(|error| { + let mut bytes = Vec::new(); + while let Some(chunk) = field.chunk().await.map_err(|error| { tracing::warn!(field = name, error = %error, "raw image multipart 文本读取失败"); bad_request(format!("{name} 字段读取失败")) + })? { + if bytes.len().saturating_add(chunk.len()) > RAW_IMAGE_MAX_TEXT_FIELD_BYTES { + tracing::warn!( + field = name, + limit_bytes = RAW_IMAGE_MAX_TEXT_FIELD_BYTES, + "raw image multipart 文本字段超过大小限制" + ); + return Err(bad_request(format!( + "{name} 字段不能超过 {RAW_IMAGE_MAX_TEXT_FIELD_BYTES} 字节" + ))); + } + bytes.extend_from_slice(&chunk); + } + *target = Some(String::from_utf8(bytes).map_err(|error| { + tracing::warn!(field = name, error = %error, "raw image multipart 文本字段不是有效 UTF-8"); + bad_request(format!("{name} 字段必须为有效 UTF-8 文本")) })?); Ok(()) } @@ -269,9 +286,9 @@ fn prepare_request(payload: RawImageEditRequest) -> Result RAW_IMAGE_MAX_PROMPT_BYTES { + if payload.prompt.len() > RAW_IMAGE_MAX_TEXT_FIELD_BYTES { return Err(bad_request(format!( - "prompt 不能超过 {RAW_IMAGE_MAX_PROMPT_BYTES} 字节" + "prompt 不能超过 {RAW_IMAGE_MAX_TEXT_FIELD_BYTES} 字节" ))); } validate_raw_image_edit_dimensions(payload.width, payload.height) @@ -567,12 +584,12 @@ mod tests { #[test] fn prompt_uses_raw_utf8_byte_limit() { let mut parsed = request(png_bytes(1, 1), None); - parsed.prompt = "a".repeat(RAW_IMAGE_MAX_PROMPT_BYTES + 1); + parsed.prompt = "a".repeat(RAW_IMAGE_MAX_TEXT_FIELD_BYTES + 1); let error = match prepare_request(parsed) { Ok(_) => panic!("oversized prompt should fail before image decode and billing"), Err(error) => error, }; - assert!(format!("{error:?}").contains("prompt 不能超过 4096 字节")); + assert!(format!("{error:?}").contains("prompt 不能超过 16384 字节")); } } From b357d36262be23fa18c4222816c758d05433585b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:29:11 +0800 Subject: [PATCH 112/248] =?UTF-8?q?=E6=98=8E=E7=A1=AE=20GPT=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E7=BC=96=E8=BE=91=E5=93=8D=E5=BA=94=E6=A0=BC=E5=BC=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 原始图片编辑请求固定声明 response_format=b64_json。 补充 GPT-Image-2 仅返回 b64_json 的契约行内注释。 --- server-rs/crates/platform-image/src/vector_engine/raw_edit.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 02f08c998..3c26e1bd0 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -136,6 +136,8 @@ pub async fn create_vector_engine_raw_image_edit( .text("n", "1".to_string()) .text("prompt", prompt.to_string()) .text("size", format!("{}x{}", options.width, options.height)) + // GPT-Image-2 的图片编辑响应契约只返回 b64_json，不处理 URL 响应。 + .text("response_format", "b64_json".to_string()) .part( "image", Part::stream_with_length(image_bytes.clone(), image_bytes.len() as u64) From 8cd6e34af0e160bd73873ae90971126c27e7e463 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:33:26 +0800 Subject: [PATCH 113/248] =?UTF-8?q?=E8=B0=83=E6=95=B4=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E5=88=87=E5=88=86=E6=89=B9=E6=AC=A1=E8=AF=B7=E6=B1=82=E5=B0=BA?= =?UTF-8?q?=E5=AF=B8=E8=AE=A1=E7=AE=97=E9=80=BB=E8=BE=91?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 基于批次源矩形总面积动态计算 image-edit 请求尺寸，新增面积对齐和最小/最大值限制，扩展相关模块和测试以支持调整后的工作流。 --- .../src/ui_editor/commands/separation/mod.rs | 2 +- .../commands/separation/model/mod.rs | 2 + .../commands/separation/workflow/batch.rs | 91 ++++++++++++++++++- .../commands/separation/workflow/cut.rs | 36 ++++++++ .../commands/separation/workflow/mod.rs | 20 ++-- ...案】UI编辑器自动切分素材工作流-2026-09-08.md | 2 +- 6 files changed, 138 insertions(+), 15 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index a0b15a162..067c83349 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -10,7 +10,7 @@ pub use model::*; pub use persistence::*; pub use tree::*; pub use workflow::apply_batch_patch; -pub use workflow::batch::next_image_batch; +pub use workflow::batch::{image_edit_dimension_for_area, next_image_batch}; pub(crate) use workflow::separate_ui_impl; #[cfg(test)] mod tests { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs index 16f549feb..74a2c773b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/mod.rs @@ -12,6 +12,8 @@ pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v2 pub const MAX_REWORK_COUNT: u32 = 3; pub const MAX_REWORK_NOTE_CHARS: usize = 512; pub const IMAGE_EDIT_MAX_DIMENSION_PX: u64 = 2880; +pub const IMAGE_EDIT_MIN_DIMENSION_PX: u64 = 816; +pub const IMAGE_EDIT_DIMENSION_ALIGNMENT_PX: u64 = 16; pub const IMAGE_EDIT_AREA_UTILIZATION_PERCENT: u64 = 80; pub const IMAGE_EDIT_AREA_LIMIT_PX: u64 = IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_AREA_UTILIZATION_PERCENT diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs index 487fc312c..f4033880c 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs @@ -2,6 +2,44 @@ use crate::ui_editor::commands::separation::model::*; use crate::ui_editor::utils::NodeId; use std::collections::HashSet; +#[derive(Debug)] +pub struct ImageBatch<'a> { + pub nodes: Vec<&'a SeparationNode>, + pub area_px: u64, + pub image_edit_dimension_px: u32, +} + +/// Derive the square raw image-edit canvas from the selected source area. +/// The calculation lives beside batch selection so the area budget and +/// request size cannot drift apart. +pub fn image_edit_dimension_for_area(area_px: u64) -> u32 { + let max = u128::from(IMAGE_EDIT_MAX_DIMENSION_PX); + let limit = u128::from(IMAGE_EDIT_AREA_LIMIT_PX); + let area = u128::from(area_px); + let raw = if area >= limit { + IMAGE_EDIT_MAX_DIMENSION_PX + } else if area == 0 { + 0 + } else { + // Find floor(max * sqrt(area / limit)) without floating-point rounding. + let target = max * max * area; + let mut low = 0u64; + let mut high = IMAGE_EDIT_MAX_DIMENSION_PX; + while low < high { + let mid = low + (high - low + 1) / 2; + if u128::from(mid) * u128::from(mid) * limit <= target { + low = mid; + } else { + high = mid - 1; + } + } + low + }; + let alignment = IMAGE_EDIT_DIMENSION_ALIGNMENT_PX; + let aligned = raw / alignment * alignment; + aligned.clamp(IMAGE_EDIT_MIN_DIMENSION_PX, IMAGE_EDIT_MAX_DIMENSION_PX) as u32 +} + fn terminal_ids(state: &SeparationState) -> HashSet { state .bound @@ -40,10 +78,10 @@ fn collect_dfs_batch<'a>( false } -pub fn next_image_batch<'a>( +pub fn next_image_batch_with_size<'a>( state: &SeparationState, tree: &'a SeparationTree, -) -> Vec<&'a SeparationNode> { +) -> ImageBatch<'a> { let terminal = terminal_ids(state); let mut selected = Vec::new(); let mut area = 0; @@ -55,11 +93,54 @@ pub fn next_image_batch<'a>( &mut selected, &mut area, ); + let image_edit_dimension_px = image_edit_dimension_for_area(area); app_log!( - "ui_separation.batch_selected image_id={} image_nodes={} area_px={}", + "ui_separation.batch_selected image_id={} image_nodes={} area_px={} image_edit_dimension_px={}", tree.src_ui_design.as_str(), selected.len(), - area + area, + image_edit_dimension_px ); - selected + ImageBatch { + nodes: selected, + area_px: area, + image_edit_dimension_px, + } +} + +pub fn next_image_batch<'a>( + state: &SeparationState, + tree: &'a SeparationTree, +) -> Vec<&'a SeparationNode> { + next_image_batch_with_size(state, tree).nodes +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn image_edit_dimension_uses_minimum_and_alignment() { + assert_eq!(image_edit_dimension_for_area(0), 816); + assert_eq!(image_edit_dimension_for_area(1), 816); + assert_eq!( + image_edit_dimension_for_area(IMAGE_EDIT_AREA_LIMIT_PX / 16), + 816 + ); + assert_eq!( + image_edit_dimension_for_area(IMAGE_EDIT_AREA_LIMIT_PX), + 2880 + ); + assert_eq!(image_edit_dimension_for_area(u64::MAX), 2880); + } + + #[test] + fn image_edit_dimension_rounds_down_to_sixteen_pixels() { + let max_squared = IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_MAX_DIMENSION_PX; + let area_just_below_1536 = IMAGE_EDIT_AREA_LIMIT_PX * 1536 * 1536 / max_squared; + let area_at_1536 = (IMAGE_EDIT_AREA_LIMIT_PX * 1536 * 1536).div_ceil(max_squared); + + assert_eq!(image_edit_dimension_for_area(area_just_below_1536), 1520); + assert_eq!(image_edit_dimension_for_area(area_at_1536), 1536); + } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs index 8b083c8b5..bd4e9d2c7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/cut.rs @@ -91,3 +91,39 @@ fn cut_processed_image_blocking( ); Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + use image::{Rgba, RgbaImage}; + + #[test] + fn cuts_using_small_processed_image_dimensions() { + let directory = tempfile::tempdir().expect("创建临时目录失败"); + let source = directory.path().join("processed.png"); + let target = directory.path().join("cut.png"); + let mut image = RgbaImage::from_pixel(816, 816, Rgba([0, 0, 0, 0])); + for y in 120..152 { + for x in 700..800 { + image.put_pixel(x, y, Rgba([255, 255, 255, 255])); + } + } + image.save(&source).expect("写入处理图失败"); + + cut_processed_image_blocking( + &source, + &BindingArea { + global_pos_x_px: 700, + global_pos_y_px: 120, + width_px: 100, + height_px: 32, + }, + &target, + ) + .expect("裁切处理图失败"); + + let cropped = image::open(target).expect("读取 cut 图片失败"); + assert_eq!(cropped.width(), 100); + assert_eq!(cropped.height(), 32); + } +} diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 4bba6a8da..8c492b9b6 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -6,7 +6,7 @@ mod patch; pub use patch::apply_batch_patch; -use self::batch::next_image_batch; +use self::batch::next_image_batch_with_size; use super::model::*; use super::persistence::{ project_relative_path, read_separation_state, separation_dto, separation_sidecar_dir, @@ -111,7 +111,11 @@ pub(crate) async fn separate_ui_impl( let Some(current_tree) = separation.trees.get(tree_index) else { break; }; - let batch_nodes = next_image_batch(&separation, current_tree) + let batch_selection = next_image_batch_with_size(&separation, current_tree); + let batch_area_px = batch_selection.area_px; + let image_edit_dimension_px = batch_selection.image_edit_dimension_px; + let batch_nodes = batch_selection + .nodes .into_iter() .cloned() .collect::>(); @@ -125,16 +129,17 @@ pub(crate) async fn separate_ui_impl( let batch = batch_nodes.iter().collect::>(); let prompt = gen_extract_prompt(&separation, current_tree, &batch); app_log!( - "ui_separation.batch_start tree_index={} batch_index={} nodes={} prompt_chars={} rework_total={}", - tree_index, batch_index, batch.len(), prompt.chars().count(), + "ui_separation.batch_start tree_index={} batch_index={} nodes={} area_px={} image_edit_dimension_px={} prompt_chars={} rework_total={}", + tree_index, batch_index, batch.len(), batch_area_px, image_edit_dimension_px, + prompt.chars().count(), batch.iter().map(|node| node.rework_count).sum::() ); let processed_url = match extract::raw_extract( &session, &source_url, &prompt, - image.pixel_size.x as u32, - image.pixel_size.y as u32, + image_edit_dimension_px, + image_edit_dimension_px, ) .await { @@ -147,8 +152,7 @@ pub(crate) async fn separate_ui_impl( }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); if let Err(error) = - extract::write_processed_image(processed_url.clone(), processed_path.clone()) - .await + extract::write_processed_image(processed_url.clone(), processed_path.clone()).await { app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); write_separation_state(&state_path, &separation)?; diff --git a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md index e69506c13..ce0b51f80 100644 --- a/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md +++ b/docs/technical/【技术方案】UI编辑器自动切分素材工作流-2026-09-08.md @@ -48,7 +48,7 @@ Rust 在接收并校验工具结果后，才把这两个枚举变体映射为正 - 追加清单区分本轮 Image 输出目标、已完成 Image、仅作父子/遮挡上下文的 Image，以及只需从父图片移除的 Text。清单使用人类可读的编号、name/description、位置和层级，不向 image-edit 暴露 opaque NodeId。 - 提取 prompt 先把 separation tree 投影为小型 YAML 视图，再注入固定规则文本：每个节点只包含展示编号、状态、`x/y/width/height` 矩形、描述、可选返工意见和递归 children；YAML 不携带 opaque NodeId，树的嵌套关系替代 `depth/role` 字段。 - 由于 raw endpoint 每次只返回一张 PNG，prompt 要求 image-edit 输出透明 atlas：本轮图片层可以移动和缩放，放置在不会互相遮挡的位置；视觉模型返回每层在 processed 图中的实际区域。源节点矩形只用于语义定位，不用于裁切区域推断。 -- 请求尺寸始终使用源 UI design 尺寸；Raw GPT Image 2 API 保证返回相同尺寸，客户端不额外做尺寸拒绝检查。 +- image-edit 请求尺寸由当前 batch 的源矩形面积决定，不再始终使用源 UI design 尺寸。设 batch 面积为 `A`、面积上限为 `L = IMAGE_EDIT_AREA_LIMIT_PX`，先计算 `floor(IMAGE_EDIT_MAX_DIMENSION_PX × sqrt(A / L))`，再按 `IMAGE_EDIT_DIMENSION_ALIGNMENT_PX` 向下对齐，并限制在 `IMAGE_EDIT_MIN_DIMENSION_PX` 至 `IMAGE_EDIT_MAX_DIMENSION_PX`（当前为 `816` 至 `2880`）之间；请求使用正方形 `N × N`。Raw GPT Image 2 API 保证返回与请求相同尺寸，客户端不额外做尺寸拒绝检查。最终 cut 继续依据 processed PNG 的实际尺寸执行，不使用源图尺寸换算。 - 处理图解码/写入和 cut 裁切属于本地 CPU/文件操作，放入独立的 `spawn_blocking` 任务；image-edit 与 visual binding 网络请求仍运行在 async future 中。 - visual binding 请求前新增本地预处理：复用 `MIN_VISIBLE_ALPHA`，将 alpha 小于该阈值的像素替换为不透明洋红标记色 `[255, 0, 255, 255]`，其余像素保留 RGB 并将 alpha 设为 `255`。预处理图只用于 visual binding，原始 processed RGBA 继续用于 cut；不新增质量门禁、alpha 统计判定或重试。 - 视觉 binding 输入源图与预处理后的不透明处理图，只接收当前 batch 的 Image targets，必须为每个 Image target 恰好返回一次 `Ok` 或 `NeedRework`。每个决定继续携带 `to_node: NodeId`；Text 不出现在请求或 schema 中。 From 842d080a753bd815483c8411d060c7a9a625cef2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:37:32 +0800 Subject: [PATCH 114/248] =?UTF-8?q?=E9=94=81=E5=AE=9A=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E7=BC=93=E5=86=B2=E4=BE=9D=E8=B5=96?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 platform-image 使用的 bytes 依赖写入 Cargo.lock。 --- server-rs/Cargo.lock | 1 + 1 file changed, 1 insertion(+) diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index b77676b63..07ddbe576 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -4088,6 +4088,7 @@ name = "platform-image" version = "0.1.0" dependencies = [ "base64", + "bytes", "curl", "image", "platform-oss", From 0cebc39b6ac443d4269add724c7e79a454d7d9cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:37:58 +0800 Subject: [PATCH 115/248] =?UTF-8?q?=E5=90=8C=E6=AD=A5=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E4=BB=A3=E7=90=86=E5=A5=91=E7=BA=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录文本字段 16KiB 流式限制。 记录 GPT-Image-2 固定 b64_json 响应格式。 --- .../【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index ed130ed2a..b37b8502c 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -27,7 +27,7 @@ width: 1536 height: 1024 ``` -`image` 和 `mask` 必须是 `image/png` 文件字段；服务端不信任客户端文件名，转发时使用固定文件名。空文件、非 PNG 字节、MIME 不匹配或 mask 与 image 尺寸不一致均在扣费前返回 400。`prompt` 必填，UTF-8 原始字节长度不得超过 `4 KiB`；超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。字段不能重复，未知字段拒绝；缺失的必填字段拒绝。 +`image` 和 `mask` 必须是 `image/png` 文件字段；服务端不信任客户端文件名，转发时使用固定文件名。空文件、非 PNG 字节、MIME 不匹配或 mask 与 image 尺寸不一致均在扣费前返回 400。`prompt` 必填，UTF-8 原始字节长度不得超过 `16 KiB`；其它文本字段也使用同一 `16 KiB` 有界流式读取，超限在扣费前返回 400。`quality`、`background` 和 `output_format` 采用 GPT Image 模型支持的值。字段不能重复，未知字段拒绝；缺失的必填字段拒绝。 `width`、`height` 使用严格输出尺寸规则，均在扣费前校验： @@ -38,13 +38,14 @@ height: 1024 校验通过后按整数尺寸发送给 provider，不静默 clamp 或改写调用者尺寸。 -Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；文件字段由 multipart 解析器直接收集为字节，随后在阻塞线程中完成 PNG 解码。PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 +Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；图片字段由 multipart 解析器收集为可共享字节缓冲，文本字段按 chunk 流式读取并在达到 `16 KiB` 时立即拒绝，随后在阻塞线程中完成 PNG 解码。PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 服务端发送给 `platform-image` 时固定注入： ```text model = gpt-image-2 n = 1 +response_format = b64_json ``` 请求不暴露 `model`、`n`、`response_format`、`style`、`user` 或 `output_compression`。 @@ -85,7 +86,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、multipart 字段解析、PNG 预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 +provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 的 provider 请求显式固定 `response_format=b64_json`，因此不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 ## 代码拆分 From 99f4961c5faef70e1895d1f49c54f982b1937694 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:47:31 +0800 Subject: [PATCH 116/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=20GPT=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=B0=BA=E5=AF=B8=E8=AE=A1=E8=B4=B9=E9=98=88=E5=80=BC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 提取共享的 1536 长边 tier 常量并复用到 raw 与编辑器计费。 同步 Raw 图片代理技术合同中的阈值来源。 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 4 +-- .../crates/api-server/src/editor_project.rs | 14 ++++++-- server-rs/crates/api-server/src/raw_image.rs | 34 +++++++++++++++++-- .../src/vector_engine/constants.rs | 2 ++ .../platform-image/src/vector_engine/mod.rs | 4 +-- 5 files changed, 49 insertions(+), 9 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index b37b8502c..ff244f984 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -38,7 +38,7 @@ height: 1024 校验通过后按整数尺寸发送给 provider，不静默 clamp 或改写调用者尺寸。 -Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；图片字段由 multipart 解析器收集为可共享字节缓冲，文本字段按 chunk 流式读取并在达到 `16 KiB` 时立即拒绝，随后在阻塞线程中完成 PNG 解码。PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 +Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；图片字段由 multipart 解析器收集为可共享字节缓冲，文本字段按 chunk 流式读取并在达到 `16 KiB` 时立即拒绝，随后在阻塞线程中完成 PNG 解码。PNG 仅接受 8-bit/channel（`png::BitDepth::Eight`），其它位深在解码前以 400 返回“`{field} 必须为 8-bit PNG（每通道 8 位）`”；PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 服务端发送给 `platform-image` 时固定注入： @@ -72,7 +72,7 @@ response_format = b64_json 检查通过后，api-server 进入现有资产操作计费边界，通过 SpacetimeDB 钱包事务 procedure 原子完成： -1. 按现有图片编辑算法解析价格：GPT Image 2 长边不超过 1536 使用 1K 价格，否则使用 2K 价格；当前默认价格为 3 / 5 泥点； +1. 按共享常量 `platform-image::GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD`（当前值 `1536`）解析图片价格：长边不超过阈值使用 1K 价格，否则使用 2K 价格；当前默认价格为 3 / 5 泥点； 2. 以认证后的用户、`raw-image-edit` 命名空间和请求 ID 组成幂等扣费流水 ID； 3. 原子扣除用户泥点并写入 `asset_operation_consume` 流水。 diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index b0663a4b1..0e89d6be9 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -19,7 +19,7 @@ use module_assets::{ AssetObjectAccessPolicy, AssetObjectFieldError, AssetObjectUpsertInput, build_asset_object_upsert_input, generate_asset_object_id, }; -use platform_image::DownloadedImage; +use platform_image::{DownloadedImage, GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD}; use platform_oss::{ LegacyAssetPrefix, OssHeadObjectRequest, OssObjectAccess, OssSignedGetObjectUrlRequest, }; @@ -3841,7 +3841,11 @@ fn editor_image_price_size_from_pixels(size: &str) -> &'static str { let Ok(height) = height.parse::() else { return "1K"; }; - if width.max(height) > 1536 { "2K" } else { "1K" } + if width.max(height) > GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD { + "2K" + } else { + "1K" + } } fn editor_image_edit_uses_nanobanana_generate_content(model: &str) -> bool { @@ -3890,7 +3894,11 @@ fn infer_editor_image_edit_size_tier(model: &str, size: Option<&str>) -> Option< if model == EDITOR_IMAGE_MODEL_NANOBANANA2 && long_edge <= 768 { return Some("0.5K"); } - Some(if long_edge > 1536 { "2K" } else { "1K" }) + Some(if long_edge > GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD { + "2K" + } else { + "1K" + }) } fn parse_editor_image_edit_pixel_size(size: &str) -> Option<(u32, u32)> { diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 7bde75405..29541924d 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -6,7 +6,8 @@ use axum::{ use bytes::Bytes; use image::{GenericImageView, ImageFormat, ImageReader}; use platform_image::{ - RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RawImageEditOptions, ReferenceImage, + GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, + RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, }; use serde::Serialize; @@ -380,6 +381,15 @@ fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32 if reader.format() != Some(ImageFormat::Png) { return Err(bad_request(format!("{field} 文件必须是有效 PNG 文件"))); } + let png_decoder = png::Decoder::new(Cursor::new(bytes.as_ref())); + let png_reader = png_decoder + .read_info() + .map_err(|_| bad_request(format!("{field} 文件必须是有效 PNG 文件")))?; + if png_reader.info().bit_depth != png::BitDepth::Eight { + return Err(bad_request(format!( + "{field} 必须为 8-bit PNG（每通道 8 位）" + ))); + } let decoded = reader .decode() .map_err(|error| map_decode_image_error(field, error))?; @@ -406,7 +416,11 @@ fn map_decode_image_error(field: &str, error: image::ImageError) -> AppError { } async fn raw_image_edit_price(state: &AppState, width: u32, height: u32) -> Result { - let tier = if width.max(height) > 1536 { "2K" } else { "1K" }; + let tier = if width.max(height) > GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD { + "2K" + } else { + "1K" + }; state .editor_generation_pricing() .await @@ -561,6 +575,22 @@ mod tests { assert!(prepare_request(invalid_mime).is_err()); } + #[test] + fn input_rejects_non_eight_bit_png() { + let mut bytes = Vec::new(); + { + let mut encoder = png::Encoder::new(&mut bytes, 1, 1); + encoder.set_color(png::ColorType::Rgba); + encoder.set_depth(png::BitDepth::Sixteen); + let mut writer = encoder.write_header().expect("PNG header"); + writer + .write_image_data(&[0, 0, 0, 0, 0, 0, 0, 0]) + .expect("PNG body"); + } + let error = prepare_request(request(bytes, None)).expect_err("16-bit PNG should fail"); + assert!(format!("{error:?}").contains("image 必须为 8-bit PNG（每通道 8 位）")); + } + #[test] fn oversized_valid_png_reports_resource_limit() { let error = match prepare_request(request(png_bytes(RAW_IMAGE_MAX_EDGE + 1, 1), None)) { diff --git a/server-rs/crates/platform-image/src/vector_engine/constants.rs b/server-rs/crates/platform-image/src/vector_engine/constants.rs index dfb70f0f5..87572ca73 100644 --- a/server-rs/crates/platform-image/src/vector_engine/constants.rs +++ b/server-rs/crates/platform-image/src/vector_engine/constants.rs @@ -5,6 +5,8 @@ pub const VECTOR_ENGINE_GPT_IMAGE_2_MODEL: &str = GPT_IMAGE_2_MODEL; pub const VECTOR_ENGINE_PROVIDER: &str = "vector-engine"; pub const VECTOR_ENGINE_IMAGE_EDIT_MAX_REFERENCE_IMAGES: usize = 5; pub const VECTOR_ENGINE_NANOBANANA_MAX_REFERENCE_IMAGES: usize = 14; +/// GPT-Image-2 pricing uses 1K through this inclusive long-edge threshold. +pub const GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD: u32 = 1536; pub(crate) const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; pub(crate) const GPT_IMAGE_2_MAX_PIXELS: u64 = 8_294_400; diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index 351244650..55a8e572e 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -21,8 +21,8 @@ pub use client::{ create_vector_engine_nanobanana_generate_content, }; pub use constants::{ - GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, NANOBANANA_2_MODEL, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, - VECTOR_ENGINE_PROVIDER, + GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, + NANOBANANA_2_MODEL, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, }; pub use error::{PlatformImageError, PlatformImageStatusHint}; pub use image_source::download_remote_image; From 4c88e5e924fad8bc860707962f784ecc817a3239 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:48:01 +0800 Subject: [PATCH 117/248] =?UTF-8?q?=E9=99=90=E5=88=B6=20Raw=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E4=BB=85=E6=8E=A5=E6=94=B6=E5=85=AB=E4=BD=8D=20PNG?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 使用 png 解码器读取位深并在 image 解码前拒绝非八位输入。 统一返回稳定的 8-bit PNG 客户端错误并补充十六位 PNG 测试。 --- server-rs/Cargo.lock | 1 + server-rs/Cargo.toml | 1 + server-rs/crates/api-server/Cargo.toml | 1 + 3 files changed, 3 insertions(+) diff --git a/server-rs/Cargo.lock b/server-rs/Cargo.lock index 07ddbe576..d76f865be 100644 --- a/server-rs/Cargo.lock +++ b/server-rs/Cargo.lock @@ -240,6 +240,7 @@ dependencies = [ "platform-oss", "platform-speech", "platform-wechat", + "png", "regex", "reqwest", "ring", diff --git a/server-rs/Cargo.toml b/server-rs/Cargo.toml index 83e472426..157646847 100644 --- a/server-rs/Cargo.toml +++ b/server-rs/Cargo.toml @@ -87,6 +87,7 @@ http-body-util = "0.1" httpdate = "1" hex = "0.4" image = { version = "0.25", default-features = false } +png = "0.18" jsonwebtoken = "9" log = "0.4" mime_guess = "2.0.5" diff --git a/server-rs/crates/api-server/Cargo.toml b/server-rs/crates/api-server/Cargo.toml index 0a166cd5f..a5b2e3cec 100644 --- a/server-rs/crates/api-server/Cargo.toml +++ b/server-rs/crates/api-server/Cargo.toml @@ -14,6 +14,7 @@ bytes = { workspace = true } dotenvy = { workspace = true } hex = { workspace = true } image = { workspace = true, features = ["jpeg", "png", "webp"] } +png = { workspace = true } http-body-util = { workspace = true } reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls"] } regex = { workspace = true } From 31fd1da4959ef8b8443abe0334a604ba0b39be34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:48:55 +0800 Subject: [PATCH 118/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=20GPT=20=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E5=93=8D=E5=BA=94=E5=A5=91=E7=BA=A6=E8=AF=B4=E6=98=8E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 仅保留 GPT-Image-2 原生 b64_json 行为的行内注释。 不发送 response_format 参数，也不引入 URL 兼容下载。 --- server-rs/crates/platform-image/src/vector_engine/raw_edit.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 3c26e1bd0..233b6cd9f 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -136,8 +136,7 @@ pub async fn create_vector_engine_raw_image_edit( .text("n", "1".to_string()) .text("prompt", prompt.to_string()) .text("size", format!("{}x{}", options.width, options.height)) - // GPT-Image-2 的图片编辑响应契约只返回 b64_json，不处理 URL 响应。 - .text("response_format", "b64_json".to_string()) + // GPT-Image-2 原生只返回 b64_json；不要添加 URL 下载或 response_format 兼容分支。 .part( "image", Part::stream_with_length(image_bytes.clone(), image_bytes.len() as u64) From 4a32ee37c49ad8c4d7c6e976551245849b96828f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:52:00 +0800 Subject: [PATCH 119/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=20Bytes=20=E8=BF=81?= =?UTF-8?q?=E7=A7=BB=E5=90=8E=E7=9A=84=E5=9B=BE=E7=89=87=E7=B1=BB=E5=9E=8B?= =?UTF-8?q?=E8=BE=B9=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 统一使用 Bytes 的只读引用并在旧 Vec 接口处显式转换。 修复编辑器参考图对齐与 Raw PNG 解码编译错误。 --- server-rs/crates/api-server/src/editor_project.rs | 5 +++-- server-rs/crates/api-server/src/editor_project_icon.rs | 2 +- server-rs/crates/api-server/src/raw_image.rs | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index 0e89d6be9..c7069e175 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -4646,7 +4646,7 @@ fn prepare_editor_image_edit_references( ); } for (index, reference) in reference_images.iter_mut().enumerate() { - let decoded = image::load_from_memory(reference.bytes.as_slice()).map_err(|error| { + let decoded = image::load_from_memory(reference.bytes.as_ref()).map_err(|error| { AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ "provider": "editor-image-edit", "message": format!("图片改造参考图不是有效图片：{error}"), @@ -4677,7 +4677,8 @@ fn prepare_editor_image_edit_references( image::DynamicImage::ImageRgba8(aligned), StatusCode::BAD_REQUEST, "图片改造参考图 16 对齐失败", - )?; + )? + .into(); reference.mime_type = "image/png".to_string(); reference.file_name = format!("editor-image-edit-reference-{}.png", index + 1); } diff --git a/server-rs/crates/api-server/src/editor_project_icon.rs b/server-rs/crates/api-server/src/editor_project_icon.rs index f7f318b86..a065f626b 100644 --- a/server-rs/crates/api-server/src/editor_project_icon.rs +++ b/server-rs/crates/api-server/src/editor_project_icon.rs @@ -2324,7 +2324,7 @@ pub async fn split_editor_icon_spritesheet( Err(_) => return Err(editor_icon_spritesheet_processing_timeout_error()), }; let source = DownloadedImage { - bytes: reference.bytes, + bytes: reference.bytes.to_vec(), mime_type: reference.mime_type, extension: "png".to_string(), }; diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 29541924d..811d316d4 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -370,7 +370,7 @@ fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32 if bytes.is_empty() { return Err(bad_request(format!("{field} 文件不能为空"))); } - let mut reader = ImageReader::new(Cursor::new(bytes.as_slice())) + let mut reader = ImageReader::new(Cursor::new(bytes.as_ref())) .with_guessed_format() .map_err(|_| bad_request(format!("{field} 文件必须是有效 PNG 文件")))?; let mut limits = image::Limits::default(); From da19bde99a4f92d5c5a1fe6517618ba87668358d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:52:26 +0800 Subject: [PATCH 120/248] =?UTF-8?q?=E5=90=8C=E6=AD=A5=E5=85=AB=E4=BD=8D=20?= =?UTF-8?q?PNG=20=E4=B8=8E=20b64=20=E5=A5=91=E7=BA=A6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录八位 PNG 校验规则与稳定错误文案。 说明 GPT-Image-2 原生 b64_json，不发送 response_format。 --- .../【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index ff244f984..998d0aed4 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -45,7 +45,6 @@ Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段； ```text model = gpt-image-2 n = 1 -response_format = b64_json ``` 请求不暴露 `model`、`n`、`response_format`、`style`、`user` 或 `output_compression`。 @@ -86,7 +85,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、multipart 字段解析、PNG 预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 的 provider 请求显式固定 `response_format=b64_json`，因此不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 +provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 原生只返回 `b64_json`，因此不发送 `response_format` 参数，也不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 ## 代码拆分 From 3e85c710600ea95173d429250b331837035c0f0e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 17:55:50 +0800 Subject: [PATCH 121/248] =?UTF-8?q?=E6=95=B4=E7=90=86=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E4=BB=A3=E7=90=86=E4=BB=A3=E7=A0=81=E6=A0=BC=E5=BC=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按 Rust 格式化规范整理 multipart 与响应解析代码。 --- server-rs/crates/api-server/src/raw_image.rs | 27 +++++++------------ .../platform-image/src/vector_engine/mod.rs | 4 +-- .../src/vector_engine/raw_edit.rs | 7 +++-- 3 files changed, 15 insertions(+), 23 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 811d316d4..c8fab56e3 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -7,8 +7,8 @@ use bytes::Bytes; use image::{GenericImageView, ImageFormat, ImageReader}; use platform_image::{ GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, - RawImageEditOptions, ReferenceImage, - create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, + RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, + validate_raw_image_edit_dimensions, }; use serde::Serialize; use serde_json::json; @@ -166,14 +166,10 @@ async fn parse_multipart_request( let mut width = None; let mut height = None; - while let Some(field) = multipart - .next_field() - .await - .map_err(|error| { - tracing::warn!(error = %error, "raw image multipart 字段解析失败"); - bad_request("multipart 请求无效") - })? - { + while let Some(field) = multipart.next_field().await.map_err(|error| { + tracing::warn!(error = %error, "raw image multipart 字段解析失败"); + bad_request("multipart 请求无效") + })? { let name = field .name() .ok_or_else(|| bad_request("multipart 字段缺少名称"))? @@ -226,13 +222,10 @@ async fn read_multipart_image( if !mime_type.eq_ignore_ascii_case("image/png") { return Err(bad_request(format!("{name} 必须为 image/png"))); } - let bytes = field - .bytes() - .await - .map_err(|error| { - tracing::warn!(field = name, error = %error, "raw image multipart 图片读取失败"); - bad_request(format!("{name} 字段读取失败")) - })?; + let bytes = field.bytes().await.map_err(|error| { + tracing::warn!(field = name, error = %error, "raw image multipart 图片读取失败"); + bad_request(format!("{name} 字段读取失败")) + })?; if bytes.is_empty() { return Err(bad_request(format!("{name} 文件不能为空"))); } diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index 55a8e572e..44941cefb 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -21,8 +21,8 @@ pub use client::{ create_vector_engine_nanobanana_generate_content, }; pub use constants::{ - GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, - NANOBANANA_2_MODEL, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, + GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, NANOBANANA_2_MODEL, + VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, }; pub use error::{PlatformImageError, PlatformImageStatusHint}; pub use image_source::download_remote_image; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 233b6cd9f..a1f13e20f 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -421,10 +421,9 @@ mod tests { #[test] fn raw_success_response_discards_empty_b64_json_entries() { - let payload: RawImageEditResponsePayload = serde_json::from_str( - r#"{"data":[{"b64_json":""},{"b64_json":"valid"}]}"#, - ) - .expect("response envelope"); + let payload: RawImageEditResponsePayload = + serde_json::from_str(r#"{"data":[{"b64_json":""},{"b64_json":"valid"}]}"#) + .expect("response envelope"); assert_eq!(collect_b64_images(payload.data), vec!["valid"]); } From 4749339bad0b1debef014c0167094c1fd23bb0b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:02:04 +0800 Subject: [PATCH 122/248] =?UTF-8?q?=E8=A1=A5=E9=BD=90=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E5=AD=97=E8=8A=82=E7=B1=BB=E5=9E=8B=E5=AF=BC=E5=87=BA=E4=B8=8E?= =?UTF-8?q?=E8=BD=AC=E6=8D=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 导出共享 GPT 图片尺寸阈值常量。 修复编辑器参考图 Bytes 与 Vec 边界转换。 --- server-rs/crates/api-server/src/editor_project.rs | 4 ++-- server-rs/crates/platform-image/src/lib.rs | 13 +++++++------ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index c7069e175..eb63aa79e 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -13522,7 +13522,7 @@ pub(crate) async fn read_editor_reference_image_object_with_client( .to_string(); let extension = editor_reference_image_extension(mime_type.as_str()); Ok(OpenAiReferenceImage { - bytes, + bytes: bytes.into(), mime_type, file_name: format!("editor-reference.{extension}"), }) @@ -13536,7 +13536,7 @@ async fn download_editor_persisted_image_object( Ok(DownloadedOpenAiImage { extension: editor_reference_image_extension(image.mime_type.as_str()).to_string(), mime_type: image.mime_type, - bytes: image.bytes, + bytes: image.bytes.to_vec(), }) } diff --git a/server-rs/crates/platform-image/src/lib.rs b/server-rs/crates/platform-image/src/lib.rs index c527d4c7d..6c7d480fb 100644 --- a/server-rs/crates/platform-image/src/lib.rs +++ b/server-rs/crates/platform-image/src/lib.rs @@ -9,12 +9,13 @@ pub use pixel_art_snapper::{ snap_pixel_art_with_deadline, }; pub use vector_engine::{ - DownloadedImage, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, GeneratedImages, NANOBANANA_2_MODEL, - PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, - RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, - RawImageEditDimensionError, RawImageEditOptions, RawImageEditResult, ReferenceImage, - VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, VectorEngineImageSettings, - build_vector_engine_image_http_client, build_vector_engine_image_request_body, + DownloadedImage, GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, + GeneratedImages, NANOBANANA_2_MODEL, PlatformImageError, PlatformImageFailureAudit, + PlatformImageStatusHint, RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, + RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, RawImageEditDimensionError, RawImageEditOptions, + RawImageEditResult, ReferenceImage, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, + VectorEngineImageSettings, build_vector_engine_image_http_client, + build_vector_engine_image_request_body, build_vector_engine_nanobanana_generate_content_request_body, create_vector_engine_image_edit, create_vector_engine_image_edit_with_references, create_vector_engine_image_edit_with_references_and_model, From 2af3d9a965889167e5c0a085ec074b3980acc367 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:50:59 +0800 Subject: [PATCH 123/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E7=BB=93=E6=9E=84?= =?UTF-8?q?=E8=AF=86=E5=88=AB=E6=AD=A5=E9=AA=A4=E7=9A=84=E5=89=8D=E7=BD=AE?= =?UTF-8?q?=E6=A0=A1=E9=AA=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将结构识别活动步骤改为校验素材切分前置条件。 --- .../src/view/ui-editor/components/WorkflowChecks.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts index fbf4d1582..cfff7ab03 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts @@ -57,7 +57,7 @@ export function activeStepPrerequisiteIssues( case 'reference-analysis': return validateComponentRecognitionPrerequisites(state); case 'structure-recognition': - return validateComponentRecognitionPrerequisites(state); + return validateAssetSeparationPrerequisites(state); case 'asset-separation': return validateAssetSeparationPrerequisites(state); } From 8d7ccc0d759a954b79859f8f7d7c73b47306c64f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:51:30 +0800 Subject: [PATCH 124/248] =?UTF-8?q?=E7=AE=80=E5=8C=96=E5=B7=A5=E4=BD=9C?= =?UTF-8?q?=E6=B5=81=E6=93=8D=E4=BD=9C=E5=8D=A1=E7=8A=B6=E6=80=81=E6=98=A0?= =?UTF-8?q?=E5=B0=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 使用步骤映射替代嵌套三元表达式。 --- .../components/WorkflowActionCard.tsx | 22 +++++++++---------- 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx index f102006f2..7c392369b 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowActionCard.tsx @@ -12,18 +12,16 @@ export function WorkflowActionCard({ (workflow.activeStep === 'structure-recognition' && workflow.isRecognizing) || (workflow.activeStep === 'asset-separation' && workflow.isSeparating); - const status = - workflow.activeStep === 'reference-analysis' - ? workflow.suggestionStatus - : workflow.activeStep === 'structure-recognition' - ? workflow.recognitionStatus - : workflow.separationStatus; - const hasRun = - workflow.activeStep === 'reference-analysis' - ? workflow.hasSuggested - : workflow.activeStep === 'structure-recognition' - ? workflow.hasRecognized - : workflow.hasSeparated; + const status = { + 'reference-analysis': workflow.suggestionStatus, + 'structure-recognition': workflow.recognitionStatus, + 'asset-separation': workflow.separationStatus, + }[workflow.activeStep]; + const hasRun = { + 'reference-analysis': workflow.hasSuggested, + 'structure-recognition': workflow.hasRecognized, + 'asset-separation': workflow.hasSeparated, + }[workflow.activeStep]; return (
From cf68b8212d15d5c00e0ca128bf76228a207f069a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:52:10 +0800 Subject: [PATCH 125/248] =?UTF-8?q?=E8=AE=A9=E6=8F=90=E5=8F=96=E6=8F=90?= =?UTF-8?q?=E7=A4=BA=E8=AF=8D=E5=BA=8F=E5=88=97=E5=8C=96=E9=94=99=E8=AF=AF?= =?UTF-8?q?=E5=8F=AF=E6=81=A2=E5=A4=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将提取提示词生成改为返回带上下文的错误。 --- .../src/ui_editor/commands/separation/prompt/extract.rs | 6 +++--- .../src/ui_editor/commands/separation/workflow/mod.rs | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs index e0059e2b9..9b6e7a2ed 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs @@ -45,7 +45,7 @@ pub(crate) fn gen_extract_prompt( state: &SeparationState, tree: &SeparationTree, batch: &[&SeparationNode], -) -> String { +) -> Result { let mut result = r#" This is a UI design image, not a normal photo/illustration. Extract it strictly as UI elements/layers, not as a generic foreground/background extraction. Treat distinct UI element as its own layer with hard, clean, pixel-accurate edges and full transparency outside the element. @@ -78,12 +78,12 @@ pub(crate) fn gen_extract_prompt( ui_layer_tree: project_node(&tree.root, &target_ids, &terminal_ids, &mut index), }; let yaml = serde_yaml::to_string(&document) - .expect("UI separation extract prompt projection must be serializable"); + .map_err(|error| format!("UI separation extract prompt projection failed: {error}"))?; result.push_str("```yaml\n"); result.push_str(&yaml); result.push_str("```\n"); - result + Ok(result) } fn project_node( diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 8c492b9b6..b145d8e9b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -127,7 +127,7 @@ pub(crate) async fn separate_ui_impl( break; } let batch = batch_nodes.iter().collect::>(); - let prompt = gen_extract_prompt(&separation, current_tree, &batch); + let prompt = gen_extract_prompt(&separation, current_tree, &batch)?; app_log!( "ui_separation.batch_start tree_index={} batch_index={} nodes={} area_px={} image_edit_dimension_px={} prompt_chars={} rework_total={}", tree_index, batch_index, batch.len(), batch_area_px, image_edit_dimension_px, From 8d15125f12164846ba65567bfa4d4240c83428f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:52:59 +0800 Subject: [PATCH 126/248] =?UTF-8?q?=E9=99=90=E5=88=B6=E5=88=87=E5=88=86?= =?UTF-8?q?=E6=8F=90=E7=A4=BA=E8=AF=8D=E4=B8=AD=E7=9A=84=E8=8A=82=E7=82=B9?= =?UTF-8?q?=E5=A4=87=E6=B3=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 对用户备注清理控制字符和围栏符号并限制长度。 --- .../commands/separation/model/note.rs | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs index 15c500b59..cb1df4fd7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs @@ -1,6 +1,24 @@ use serde::{Deserialize, Serialize}; use ts_rs::TS; +const MAX_PROMPT_NOTE_CHARS: usize = 512; + +fn sanitize_prompt_text(value: &str) -> String { + value + .chars() + .filter_map(|character| { + if character == '`' { + Some('\'') + } else if character.is_control() { + Some(' ') + } else { + Some(character) + } + }) + .take(MAX_PROMPT_NOTE_CHARS) + .collect() +} + #[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] pub struct SeparationNote { @@ -10,12 +28,12 @@ pub struct SeparationNote { impl SeparationNote { pub fn as_prompt(&self) -> String { - let mut prompt = format!("desc: {}", self.description); + let mut prompt = format!("desc: {}", sanitize_prompt_text(&self.description)); if !self.rework_notes.is_empty() { prompt.push_str("\nprevious rework notes:"); for note in &self.rework_notes { prompt.push_str("\n- "); - prompt.push_str(note); + prompt.push_str(&sanitize_prompt_text(note)); } } prompt From 3774d7c1e188bba0d018c52f3b315163e53dafd0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:53:34 +0800 Subject: [PATCH 127/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=E4=BF=AE=E5=A4=8D?= =?UTF-8?q?=E9=87=8D=E8=AF=95=E6=B5=81=E7=A8=8B=E4=B8=AD=E7=9A=84=E9=9A=90?= =?UTF-8?q?=E5=BC=8F=E6=81=90=E6=85=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在循环意外结束时返回可恢复错误而不是触发 unreachable。 --- .../src-tauri/src/ui_editor/commands/utils.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index 84f7fa178..e40a2805d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -67,7 +67,7 @@ where Err(error) => return Err(error), } } - unreachable!("repair history runner always returns within requested retries") + Err("LLM 修复重试流程未产生结果".to_string()) } pub(crate) fn parse_limited_llm_tool_arguments( From 87a7294a04cafbfaa24f27de1fe4e52a60db82cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:54:21 +0800 Subject: [PATCH 128/248] =?UTF-8?q?=E7=BB=86=E5=8C=96=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E7=BB=91=E5=AE=9A=E8=BD=BD=E8=8D=B7=E5=BD=A2=E7=8A=B6=E6=A0=A1?= =?UTF-8?q?=E9=AA=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在反序列化前拒绝空值和无效的节点组件外部枚举。 --- .../src/ui_editor/commands/binding.rs | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs index 17979cbca..d6eb3ba8d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/binding.rs @@ -170,11 +170,22 @@ fn validate_binding_response_shape( return Err(format!("组件绑定 changes 不能超过 {max_changes} 条")); } for change in changes { - if !change - .as_object() - .is_some_and(|object| object.contains_key("component")) - { + let Some(object) = change.as_object() else { return Err("组件绑定 change 缺少 component 字段".to_string()); + }; + let Some(component) = object.get("component") else { + return Err("组件绑定 change 缺少 component 字段".to_string()); + }; + let valid_component = component == "PureNode" + || component + .as_object() + .and_then(|value| value.get("WithComponent")) + .is_some_and(serde_json::Value::is_object); + if !valid_component { + return Err( + "组件绑定 change 的 component 必须是 PureNode 或 WithComponent 对象" + .to_string(), + ); } } Ok(()) From 6db6de5b998238a3b1cf21bc89f42c76e49f84ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:54:39 +0800 Subject: [PATCH 129/248] =?UTF-8?q?=E5=90=8C=E6=AD=A5=E6=A3=80=E6=9F=A5?= =?UTF-8?q?=E5=99=A8=E7=BB=84=E4=BB=B6=E9=9D=A2=E6=9D=BF=E5=B1=95=E5=BC=80?= =?UTF-8?q?=E7=8A=B6=E6=80=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 切换节点或组件时重置展开状态，避免复用旧节点的折叠状态。 --- .../components/Inspector/Components/ComponentPanel.tsx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index 89d20155c..30089990f 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -1,5 +1,5 @@ import { ChevronDown, ChevronRight, Plus, Trash2 } from 'lucide-react'; -import { useState } from 'react'; +import { useEffect, useState } from 'react'; import type { Component } from '../../../../../features/ui-editor/types/Component'; import type { UiEditorOperationResult } from '../../../../../features/ui-editor/useUiEditorState'; @@ -18,6 +18,10 @@ export function ComponentPanel(props: ComponentPanelProps) { const [expanded, setExpanded] = useState(Boolean(component)); const [error, setError] = useState(null); + useEffect(() => { + setExpanded(Boolean(component)); + }, [component]); + function setComponent(next: Component | null) { if (readOnly) return undefined; const result = onSetComponent(next); From bfe010a19259b11480a72c0a551b55a8ea2106d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:55:06 +0800 Subject: [PATCH 130/248] =?UTF-8?q?=E8=AE=A9=E8=A7=86=E8=A7=89=E7=BB=91?= =?UTF-8?q?=E5=AE=9A=E8=B0=83=E8=AF=95=E9=A2=84=E8=A7=88=E5=86=99=E5=85=A5?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E5=8F=AF=E9=99=8D=E7=BA=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 调试 PNG 写入失败时记录警告并继续绑定流程。 --- .../src/ui_editor/commands/separation/image_preprocess.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs index d3615ef4f..d04782faf 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/image_preprocess.rs @@ -44,8 +44,12 @@ fn preprocess_for_visual_binding_blocking( .write_to(&mut Cursor::new(&mut png), ImageFormat::Png) .map_err(|error| format!("编码视觉绑定预览失败：{error}"))?; let debug_name = format!("binding-{}.png", uuid::Uuid::new_v4().simple()); - fs::write(sidecar.join(&debug_name), &png) - .map_err(|error| format!("写入视觉绑定预览失败：{error}"))?; + if let Err(error) = fs::write(sidecar.join(&debug_name), &png) { + app_log!( + "ui_separation.warning stage=visual_binding_preview_write file={} error={error}", + debug_name + ); + } Ok(format!( "data:image/png;base64,{}", base64::engine::general_purpose::STANDARD.encode(png) From b8691add3addb23e477836a5d5934bd5c0031238 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:55:28 +0800 Subject: [PATCH 131/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=E5=8C=BA=E5=9F=9F?= =?UTF-8?q?=E5=BD=92=E4=B8=80=E5=8C=96=E6=97=A5=E5=BF=97=E8=B7=AF=E5=BE=84?= =?UTF-8?q?=E4=B8=AD=E7=9A=84=E6=81=90=E6=85=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 直接记录归一化结果，避免诊断日志调用 expect。 --- .../src/ui_editor/commands/separation/area.rs | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs index f39c38f10..9685a0884 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -327,12 +327,12 @@ pub(crate) fn normalize_binding_area( } let area = current.into_area(); - let result = Ok(NormalizedBindingArea { + let normalized = NormalizedBindingArea { changed: area != original_area, area, clamped, transparent: !rect_has_visible_pixel(image, current), - }); + }; app_log!( "ui_separation.area.timing outcome=ok elapsed_us={} rounds={} image_width={} image_height={} area=({}, {}, {}, {}) changed={} clamped={} transparent={}", started.elapsed().as_micros(), @@ -343,11 +343,11 @@ pub(crate) fn normalize_binding_area( original_area.global_pos_y_px, original_area.width_px, original_area.height_px, - result.as_ref().expect("normalization result exists").changed, - result.as_ref().expect("normalization result exists").clamped, - result.as_ref().expect("normalization result exists").transparent + normalized.changed, + normalized.clamped, + normalized.transparent ); - result + Ok(normalized) } #[cfg(test)] From 26677b193c1360bcb9ecf112c396caf10a1feefe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 18:56:13 +0800 Subject: [PATCH 132/248] =?UTF-8?q?=E5=A4=8D=E7=94=A8=E7=95=8C=E9=9D=A2?= =?UTF-8?q?=E6=A0=91=E7=BB=84=E4=BB=B6=E8=AE=A1=E6=95=B0=E5=80=BC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在同一行中复用组件数量，避免重复计算。 --- .../src/view/ui-editor/components/UiTreePanel.tsx | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/UiTreePanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/UiTreePanel.tsx index 0936dbf28..6bba111ce 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/UiTreePanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/UiTreePanel.tsx @@ -103,6 +103,7 @@ function TreeRow({ const data = node.data; const nodeLabel = data.metadata.name || '未命名节点'; const isVisible = isNodeVisible(data.id); + const componentCount = data.component ? 1 : 0; return (
+ +
+
); } From 53008959c55d6b5b0c51df34b3fe67b4319a0432 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 19:09:22 +0800 Subject: [PATCH 146/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E7=BB=93=E6=9E=9C=E7=9A=84=E7=B1=BB=E5=9E=8B?= =?UTF-8?q?=E6=94=B6=E7=AA=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在异步闭包赋值后明确已完成结果的非空类型。 --- .../ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 561c237c9..6f262cabe 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1190,7 +1190,7 @@ export function useUiEditorSession( })); if (separationResult === null) throw new Error('自动切分素材没有返回结果'); - const completedResult = separationResult; + const completedResult = separationResult as SeparationDTO; if (!(await save())) { throw new Error( '自动切分素材结果已写入编辑器，但 State 保存失败；sidecar 已保留，可继续恢复。', From a5e936dd27f975554c724fa6ff34f8911014f47e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 19:39:59 +0800 Subject: [PATCH 147/248] =?UTF-8?q?=E6=B8=85=E7=90=86=E5=B7=B2=E9=80=80?= =?UTF-8?q?=E5=BD=B9=E7=9A=84=E5=89=8D=E7=AB=AF=E7=BB=91=E5=AE=9A=E6=A8=A1?= =?UTF-8?q?=E5=9D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 删除无现役导入的 binding.ts 及其 BindingDTO、BindingChange 类型。 --- .../src/features/ui-editor/binding.ts | 23 ------------------- .../features/ui-editor/types/BindingChange.ts | 6 ----- .../features/ui-editor/types/BindingDTO.ts | 4 ---- 3 files changed, 33 deletions(-) delete mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/binding.ts delete mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts delete mode 100644 apps/ai-game-creator-shell/src/features/ui-editor/types/BindingDTO.ts diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts b/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts deleted file mode 100644 index 3f3422b18..000000000 --- a/apps/ai-game-creator-shell/src/features/ui-editor/binding.ts +++ /dev/null @@ -1,23 +0,0 @@ -import type { BindingDTO } from './types/BindingDTO'; -import type { Node } from './types/Node'; -import type { State } from './types/State'; - -function applyChanges(node: Node, result: BindingDTO): void { - const change = result.changes.find( - (candidate) => candidate.node_id === node.id, - ); - if (change) { - node.component = structuredClone( - change.component === 'PureNode' ? null : change.component.WithComponent, - ); - node.metadata.component_status = structuredClone(change.component_status); - } - for (const child of node.children) applyChanges(child, result); -} - -/** Applies only explicit component changes; omitted nodes remain untouched. */ -export function applyBindingResult(state: State, result: BindingDTO): State { - const next = structuredClone(state); - for (const tree of next.ui_trees) applyChanges(tree.root, result); - return next; -} diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts deleted file mode 100644 index 3bb2c5c4d..000000000 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingChange.ts +++ /dev/null @@ -1,6 +0,0 @@ -// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. -import type { NodeComponent } from "./NodeComponent"; -import type { NodeId } from "./NodeId"; -import type { StageStatus } from "./StageStatus"; - -export type BindingChange = { node_id: NodeId, component: NodeComponent, component_status: StageStatus, }; diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingDTO.ts b/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingDTO.ts deleted file mode 100644 index beb340894..000000000 --- a/apps/ai-game-creator-shell/src/features/ui-editor/types/BindingDTO.ts +++ /dev/null @@ -1,4 +0,0 @@ -// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. -import type { BindingChange } from "./BindingChange"; - -export type BindingDTO = { changes: Array, }; From f933db169e63f97e40ddcfca2f758f361751e0e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 19:41:21 +0800 Subject: [PATCH 148/248] =?UTF-8?q?=E8=AF=B4=E6=98=8E=E7=BB=93=E6=9E=84?= =?UTF-8?q?=E5=8C=96=E9=87=8D=E8=AF=95=E7=9A=84=E6=9C=89=E9=99=90=E4=B8=8A?= =?UTF-8?q?=E4=B8=8B=E6=96=87=E8=BE=B9=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录现役调用方的小重试次数和保留完整反馈的设计依据。 --- .../src-tauri/src/ui_editor/commands/utils.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index e40a2805d..5111aadd8 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -28,6 +28,10 @@ pub(crate) async fn request_ui_editor_llm( } /// 按 append-only history 重试结构化 LLM 请求；仅业务校验失败会追加反馈消息。 +/// +/// 现役调用方把重试次数固定在很小的范围（生产路径为 2 次），初始提示词和工具 +/// schema 也受 provider 的请求预算约束；因此最多追加两份反馈，不会形成需要额外 +/// 截断策略的上下文无限增长。这里保留完整模型输出，便于模型修正业务校验失败。 pub(crate) async fn run_with_repair_history( max_retries: usize, initial_history: Vec, From 4cf3642b531072cd003a4990c8936509ad70914d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 19:41:48 +0800 Subject: [PATCH 149/248] =?UTF-8?q?=E8=AF=B4=E6=98=8E=E8=A7=86=E8=A7=89?= =?UTF-8?q?=E7=BB=91=E5=AE=9A=E8=A7=A3=E6=9E=90=E9=94=99=E8=AF=AF=E7=9A=84?= =?UTF-8?q?=E6=9C=89=E9=99=90=E9=87=8D=E8=AF=95=E7=AD=96=E7=95=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录严格工具 schema 和响应损坏重试边界。 --- .../src/ui_editor/commands/separation/workflow/binding.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs index d29dd226a..32d3f5416 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs @@ -100,6 +100,8 @@ async fn visual_binding_inner( }, ]), ]; + // 严格工具 schema 由 provider 负责约束正常模型输出；这里的解析失败只代表极小概率 + // 的传输/响应损坏，因此沿用有限重试，不再为理论上的坏载荷扩展业务修复协议。 let result = run_with_repair_history( 2, initial_history, From 8c8b6cc75acffe58e7074b664fb6966705332e4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 19:44:01 +0800 Subject: [PATCH 150/248] =?UTF-8?q?=E5=B0=86=E8=87=AA=E5=8A=A8=E5=88=87?= =?UTF-8?q?=E5=88=86=E7=8A=B6=E6=80=81=E5=86=99=E7=9B=98=E7=A7=BB=E5=87=BA?= =?UTF-8?q?=20Tokio=20worker?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 用 spawn_blocking 承载 JSON 序列化和文件写入，并在工作流中等待结果。 --- .../commands/separation/persistence.rs | 8 +++++- .../commands/separation/workflow/mod.rs | 26 ++++++++++--------- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 32b49adc6..05a4ed683 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -42,7 +42,13 @@ pub fn project_relative_path(root: &Path, path: &Path) -> Result Ok(value) } -pub fn write_separation_state(path: &Path, state: &SeparationState) -> Result<(), String> { +pub async fn write_separation_state(path: PathBuf, state: SeparationState) -> Result<(), String> { + tokio::task::spawn_blocking(move || write_separation_state_blocking(&path, &state)) + .await + .map_err(|error| format!("写入 separation state 任务失败：{error}"))? +} + +fn write_separation_state_blocking(path: &Path, state: &SeparationState) -> Result<(), String> { if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { app_log!("ui_separation.error stage=state_write reason=schema_mismatch"); return Err("不支持的 separation state schema".to_string()); diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 62dae2474..c4195e9ad 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -98,13 +98,15 @@ pub(crate) async fn separate_ui_impl( image.pixel_size.x.round() as u32, image.pixel_size.y.round() as u32 ); - write_separation_state(&state_path, &separation).map_err(|error| { - app_log!( - "ui_separation.error stage=state_checkpoint tree_index={} error={error}", - tree_index - ); - error - })?; + write_separation_state(state_path.clone(), separation.clone()) + .await + .map_err(|error| { + app_log!( + "ui_separation.error stage=state_checkpoint tree_index={} error={error}", + tree_index + ); + error + })?; let mut batch_index = 0usize; loop { let batch_started = Instant::now(); @@ -146,7 +148,7 @@ pub(crate) async fn separate_ui_impl( Ok(value) => value, Err(error) => { app_log!("ui_separation.error stage=image_edit tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(&state_path, &separation)?; + write_separation_state(state_path.clone(), separation.clone()).await?; return Err(error); } }; @@ -155,7 +157,7 @@ pub(crate) async fn separate_ui_impl( extract::write_processed_image(processed_url.clone(), processed_path.clone()).await { app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(&state_path, &separation)?; + write_separation_state(state_path.clone(), separation.clone()).await?; return Err(error); } let binding = match binding::visual_binding( @@ -169,7 +171,7 @@ pub(crate) async fn separate_ui_impl( Ok(value) => value, Err(error) => { app_log!("ui_separation.error stage=visual_binding tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(&state_path, &separation)?; + write_separation_state(state_path.clone(), separation.clone()).await?; return Err(error); } }; @@ -220,11 +222,11 @@ pub(crate) async fn separate_ui_impl( } if let Some(error) = cut_error { app_log!("ui_separation.error stage=cut_batch tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(&state_path, &separation)?; + write_separation_state(state_path.clone(), separation.clone()).await?; return Err(error); } patch::apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; - write_separation_state(&state_path, &separation)?; + write_separation_state(state_path.clone(), separation.clone()).await?; app_log!( "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={} elapsed_ms={}", tree_index, batch_index, cut_paths.len(), separation.bound.len(), separation.problematic_nodes.len(), batch_started.elapsed().as_millis() From c743ca146ef1737c19ee3f67f0c47a0a22e0982a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 19:53:14 +0800 Subject: [PATCH 151/248] =?UTF-8?q?=E5=A4=8D=E7=94=A8=E5=9B=BE=E5=83=8F?= =?UTF-8?q?=E4=B8=8B=E8=BD=BD=E7=9A=84=E5=85=B1=E4=BA=AB=E5=AD=97=E8=8A=82?= =?UTF-8?q?=E7=BC=93=E5=86=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 DownloadedImage.bytes 改为 bytes::Bytes。 让远程下载和图标切片下游避免不必要的完整复制。 同步更新图像处理实现与测试 fixture。 --- .../crates/api-server/src/editor_project.rs | 2 +- .../api-server/src/editor_project_icon.rs | 6 ++-- .../src/generated_asset_sheets/sheet.rs | 28 +++++++++---------- .../platform-image/src/pixel_art_snapper.rs | 6 ++-- .../src/vector_engine/image_source.rs | 6 ++-- .../platform-image/src/vector_engine/types.rs | 3 +- .../tests/generated_asset_sheets.rs | 2 +- 7 files changed, 27 insertions(+), 26 deletions(-) diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index eb63aa79e..c6529863a 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -19064,7 +19064,7 @@ mod tests { #[tokio::test] async fn icon_spritesheet_expired_deadline_skips_blocking_image_work() { let source = DownloadedImage { - bytes: b"not-an-image".to_vec(), + bytes: bytes::Bytes::from_static(b"not-an-image"), mime_type: "image/png".to_string(), extension: "png".to_string(), }; diff --git a/server-rs/crates/api-server/src/editor_project_icon.rs b/server-rs/crates/api-server/src/editor_project_icon.rs index a065f626b..39deb0bd7 100644 --- a/server-rs/crates/api-server/src/editor_project_icon.rs +++ b/server-rs/crates/api-server/src/editor_project_icon.rs @@ -2324,7 +2324,7 @@ pub async fn split_editor_icon_spritesheet( Err(_) => return Err(editor_icon_spritesheet_processing_timeout_error()), }; let source = DownloadedImage { - bytes: reference.bytes.to_vec(), + bytes: reference.bytes, mime_type: reference.mime_type, extension: "png".to_string(), }; @@ -2653,7 +2653,7 @@ pub(crate) fn editor_icon_spritesheet_warning_after_persist_error( } fn validate_editor_icon_spritesheet_source(source: &DownloadedImage) -> Result<(), AppError> { - let reader = image::ImageReader::new(Cursor::new(source.bytes.as_slice())) + let reader = image::ImageReader::new(Cursor::new(source.bytes.as_ref())) .with_guessed_format() .map_err(|error| { AppError::from_status(StatusCode::UNPROCESSABLE_ENTITY).with_details(json!({ @@ -2931,7 +2931,7 @@ mod tests { .write_to(&mut std::io::Cursor::new(&mut bytes), ImageFormat::Png) .expect("fixture png should encode"); let source = DownloadedImage { - bytes, + bytes: bytes.into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; diff --git a/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs b/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs index b0d7ec81b..496bb9824 100644 --- a/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs +++ b/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs @@ -34,7 +34,7 @@ pub fn slice_generated_asset_sheet( let grid_size_u32 = u32::try_from(grid_size).map_err(|_| { GeneratedAssetSheetError::invalid_request("系列素材图集的 n 超出可支持范围。") })?; - let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("系列素材图集解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source); @@ -94,7 +94,7 @@ pub fn slice_generated_asset_sheet_two_items_per_row( let grid_size_u32 = u32::try_from(grid_size).map_err(|_| { GeneratedAssetSheetError::invalid_request("系列素材图集的 n 超出可支持范围。") })?; - let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("系列素材图集解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source); @@ -243,7 +243,7 @@ pub fn prepare_generated_icon_spritesheet_all_by_connected_components( "图标 spritesheet 累计裁剪像素上限必须大于 0。", )); } - let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("图标 spritesheet 解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source); @@ -263,7 +263,7 @@ pub fn prepare_generated_icon_spritesheet_all_by_connected_components( pub fn prepare_generated_icon_spritesheet_grid_2x2( image: &crate::DownloadedImage, ) -> Result { - let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("图标 spritesheet 解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source).into_rgba8(); @@ -942,7 +942,7 @@ mod tests { } } crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), } @@ -1031,7 +1031,7 @@ mod tests { sheet.put_pixel(x0 + 40, y0 + 8, Rgba(color)); } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1063,7 +1063,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1095,7 +1095,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1137,7 +1137,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1165,7 +1165,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1206,7 +1206,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1240,7 +1240,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1272,7 +1272,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1303,7 +1303,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet), + bytes: encode_png(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; diff --git a/server-rs/crates/platform-image/src/pixel_art_snapper.rs b/server-rs/crates/platform-image/src/pixel_art_snapper.rs index 589a26a6f..9f7ac5c33 100644 --- a/server-rs/crates/platform-image/src/pixel_art_snapper.rs +++ b/server-rs/crates/platform-image/src/pixel_art_snapper.rs @@ -266,7 +266,7 @@ fn decode_rgba_source( return Err(PixelArtSnapError::InvalidInput(format!("{input} 为空"))); } - let dimension_reader = image::ImageReader::new(Cursor::new(source.bytes.as_slice())) + let dimension_reader = image::ImageReader::new(Cursor::new(source.bytes.as_ref())) .with_guessed_format() .map_err(|error| PixelArtSnapError::Decode { input, @@ -281,7 +281,7 @@ fn decode_rgba_source( })?; validate_dimensions(width, height, input)?; - let mut reader = image::ImageReader::new(Cursor::new(source.bytes.as_slice())) + let mut reader = image::ImageReader::new(Cursor::new(source.bytes.as_ref())) .with_guessed_format() .map_err(|error| PixelArtSnapError::Decode { input, @@ -465,7 +465,7 @@ fn encode_png(image: RgbaImage) -> Result { .write_to(&mut cursor, ImageFormat::Png) .map_err(|error| PixelArtSnapError::Encode(error.to_string()))?; Ok(DownloadedImage { - bytes: cursor.into_inner(), + bytes: cursor.into_inner().into(), mime_type: "image/png".to_string(), extension: "png".to_string(), }) diff --git a/server-rs/crates/platform-image/src/vector_engine/image_source.rs b/server-rs/crates/platform-image/src/vector_engine/image_source.rs index 29fe42a45..7bbe073d4 100644 --- a/server-rs/crates/platform-image/src/vector_engine/image_source.rs +++ b/server-rs/crates/platform-image/src/vector_engine/image_source.rs @@ -52,7 +52,7 @@ pub async fn download_remote_image( Ok(DownloadedImage { extension: mime_to_extension(normalized_mime_type.as_str()).to_string(), mime_type: normalized_mime_type, - bytes: body.to_vec(), + bytes: body, }) } @@ -162,7 +162,7 @@ pub(crate) async fn resolve_reference_images( .await .map_err(|error| contextualize_reference_download_error(error, failure_context))?; resolved.push(ReferenceImage { - bytes: Bytes::from(downloaded.bytes), + bytes: downloaded.bytes, mime_type: downloaded.mime_type.clone(), file_name: format!( "reference-{index}.{}", @@ -248,7 +248,7 @@ pub(crate) fn decode_generated_image_base64(raw: &str) -> Option, + /// Shared immutable image bytes so callers can pass downloaded buffers downstream without copying. + pub bytes: Bytes, pub mime_type: String, pub extension: String, } diff --git a/server-rs/crates/platform-image/tests/generated_asset_sheets.rs b/server-rs/crates/platform-image/tests/generated_asset_sheets.rs index d3bc61705..b3c860528 100644 --- a/server-rs/crates/platform-image/tests/generated_asset_sheets.rs +++ b/server-rs/crates/platform-image/tests/generated_asset_sheets.rs @@ -42,7 +42,7 @@ fn build_test_sheet(width: u32, height: u32) -> DownloadedImage { } DownloadedImage { - bytes: encode_image(sheet), + bytes: encode_image(sheet).into(), mime_type: "image/png".to_string(), extension: "png".to_string(), } From e7cd008d7ce07204e329fb9b7fbc3b6fd5c389c7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 20:06:09 +0800 Subject: [PATCH 152/248] =?UTF-8?q?=E8=B0=83=E6=95=B4=E6=96=87=E6=9C=AC?= =?UTF-8?q?=E7=BB=84=E4=BB=B6=E7=9A=84=E8=AF=86=E5=88=AB=E8=A7=84=E5=88=99?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 简化多行文本节点逻辑，明确文字与图片组件的绑定要求。 --- .../src-tauri/src/ui_editor/commands/recognition.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index 15e71f2aa..c71475440 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -47,9 +47,9 @@ const SYSTEM_PROMPT: &str = r#" * 为每个节点直接返回 component. 无背景的逻辑容器返回 "PureNode"，不要返回 null. 有背景的容器推荐使用Simple+不锁定宽高比的Image component. - 目前我们只做识别, 不要求图片字体参数. - 每个节点最多返回一个 component；需要多个视觉层时拆成多个节点。 + 目前我们只做识别, 不要求图片字体的具体绑定参数. 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. +* 多行文本只使用一个节点. * 不鼓励兄弟节点相互重叠. * 对于面板等容器的背景等, 必须作为父节点的组件, 禁止新增冗余的所谓"背景节点". "#; From e7af7cf4e4b2ebe4c5661e349b0c4e7d97ebdab2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 20:11:11 +0800 Subject: [PATCH 153/248] =?UTF-8?q?=E6=A0=A1=E9=AA=8C=E5=A4=84=E7=90=86?= =?UTF-8?q?=E5=9B=BE=E5=8C=BA=E5=9F=9F=E5=B9=B6=E6=98=8E=E7=A1=AE=20multip?= =?UTF-8?q?art=20=E5=AD=97=E8=8A=82=E4=B8=8A=E4=BC=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将处理图真实尺寸传入视觉绑定校验，提前拒绝零尺寸和越界区域。 保持非法区域直接失败，不转换为 NeedRework。 将 image 和 mask multipart 部分改为 Part::bytes。 --- .../src/ui_editor/commands/separation/mod.rs | 39 +++++++++++++++++-- .../src/ui_editor/commands/separation/tree.rs | 6 +++ .../commands/separation/workflow/binding.rs | 13 ++++++- .../commands/separation/workflow/extract.rs | 15 +++++-- .../commands/separation/workflow/mod.rs | 27 +++++++++---- .../commands/separation/workflow/patch.rs | 2 + .../src/vector_engine/raw_edit.rs | 4 +- 7 files changed, 89 insertions(+), 17 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index 067c83349..d6400e05d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -190,7 +190,38 @@ mod tests { children: vec![], rework_count: 0, }; - assert!(validate_binding_response(&BindingResp { decisions: vec![] }, &[&node]).is_err()); + assert!( + validate_binding_response(&BindingResp { decisions: vec![] }, &[&node], (1, 1)) + .is_err() + ); + } + + #[test] + fn binding_validation_rejects_area_outside_processed_image() { + let node = SeparationNode { + id: NodeId::new("image").unwrap(), + kind: SeparationNodeKind::ImageTarget, + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + note: SeparationNote::default(), + children: vec![], + rework_count: 0, + }; + let response = BindingResp { + decisions: vec![BindingDecision::Ok { + to_node: node.id.clone(), + extracted_area: BindingArea { + global_pos_x_px: 1, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + }, + }], + }; + let error = validate_binding_response(&response, &[&node], (1, 1)).unwrap_err(); + assert!(error.contains("超出处理图边界")); } #[test] @@ -236,6 +267,7 @@ mod tests { advice: note.to_string(), }], &paths, + (1, 1), ) .unwrap(); } @@ -275,7 +307,8 @@ mod tests { &BindingResp { decisions: vec![decision] }, - &[&node] + &[&node], + (1, 1) ) .is_err()); } @@ -310,7 +343,7 @@ mod tests { }, }]; let paths = HashMap::from([(id.clone(), "ui/.sidecar/cut.png".to_string())]); - apply_batch_patch(&mut state, 0, &decisions, &paths).unwrap(); + apply_batch_patch(&mut state, 0, &decisions, &paths, (1, 1)).unwrap(); assert_eq!(state.bound[0].node_id, id); assert_eq!(state.trees[0].root.children.len(), 1); } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 0b483d8e4..f1d024827 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -146,6 +146,7 @@ pub fn construct_separation_state(state: &State) -> SeparationState { pub fn validate_binding_response( response: &BindingResp, batch: &[&SeparationNode], + processed_dimensions: (u32, u32), ) -> Result<(), String> { let expected = batch .iter() @@ -165,6 +166,11 @@ pub fn validate_binding_response( if !seen.insert(node_id.clone()) { return Err(format!("视觉绑定重复返回节点：{}", node_id.as_str())); } + if let BindingDecision::Ok { extracted_area, .. } = decision { + extracted_area + .validate_in(processed_dimensions.0, processed_dimensions.1) + .map_err(|error| format!("节点 {} 的分离区域无效：{error}", node_id.as_str()))?; + } if let BindingDecision::NeedRework { advice, .. } = decision { if advice.trim().is_empty() { return Err("NeedRework 必须包含问题描述".to_string()); diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs index 32d3f5416..7b13beaaf 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs @@ -19,9 +19,17 @@ pub(super) async fn visual_binding( processed_url: String, sidecar: PathBuf, nodes: &[&SeparationNode], + processed_dimensions: (u32, u32), ) -> Result { let started = Instant::now(); - let result = visual_binding_inner(source_url, processed_url, sidecar, nodes).await; + let result = visual_binding_inner( + source_url, + processed_url, + sidecar, + nodes, + processed_dimensions, + ) + .await; app_log!( "ui_separation.visual_binding.timing outcome={} elapsed_ms={} nodes={}", if result.is_ok() { "ok" } else { "error" }, @@ -36,6 +44,7 @@ async fn visual_binding_inner( processed_url: String, sidecar: PathBuf, nodes: &[&SeparationNode], + processed_dimensions: (u32, u32), ) -> Result { app_log!( "ui_separation.visual_binding.start nodes={} source_url_chars={} processed_url_chars={}", @@ -137,7 +146,7 @@ async fn visual_binding_inner( }) } }, - |value: &BindingResp| validate_binding_response(value, nodes), + |value: &BindingResp| validate_binding_response(value, nodes, processed_dimensions), ) .await; match &result { diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs index 45096b14a..a5f08e433 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs @@ -1,9 +1,9 @@ use crate::platform_session::PlatformSessionSnapshot; use base64::Engine as _; use serde::Deserialize; -use std::fs; use std::path::PathBuf; use std::time::Instant; +use std::{fs, io::Cursor}; #[derive(Deserialize)] struct RawEditResponse { @@ -125,7 +125,7 @@ async fn raw_extract_inner( pub(super) async fn write_processed_image( processed_url: String, target: PathBuf, -) -> Result<(), String> { +) -> Result<(u32, u32), String> { let started = Instant::now(); let result = write_processed_image_inner(processed_url, target).await; app_log!( @@ -136,7 +136,10 @@ pub(super) async fn write_processed_image( result } -async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> Result<(), String> { +async fn write_processed_image_inner( + processed_url: String, + target: PathBuf, +) -> Result<(u32, u32), String> { app_log!( "ui_separation.processed_image.write.start target_file={} data_url_chars={}", target @@ -153,6 +156,11 @@ async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> let processed_bytes = base64::engine::general_purpose::STANDARD .decode(encoded) .map_err(|error| format!("解析处理图失败：{error}"))?; + let dimensions = image::ImageReader::new(Cursor::new(processed_bytes.as_slice())) + .with_guessed_format() + .map_err(|error| format!("识别处理图格式失败：{error}"))? + .into_dimensions() + .map_err(|error| format!("读取处理图尺寸失败：{error}"))?; let byte_len = processed_bytes.len(); fs::write(&target, processed_bytes) .map_err(|error| format!("写入处理图失败：{}: {error}", target.display())) @@ -165,6 +173,7 @@ async fn write_processed_image_inner(processed_url: String, target: PathBuf) -> .unwrap_or(""), byte_len ); + dimensions }) }) .await diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index c4195e9ad..fb1c1e4dd 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -153,18 +153,25 @@ pub(crate) async fn separate_ui_impl( } }; let processed_path = sidecar.join(format!("processed-{}.png", separation.bound.len())); - if let Err(error) = - extract::write_processed_image(processed_url.clone(), processed_path.clone()).await + let processed_dimensions = match extract::write_processed_image( + processed_url.clone(), + processed_path.clone(), + ) + .await { - app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(state_path.clone(), separation.clone()).await?; - return Err(error); - } + Ok(dimensions) => dimensions, + Err(error) => { + app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); + write_separation_state(state_path.clone(), separation.clone()).await?; + return Err(error); + } + }; let binding = match binding::visual_binding( source_url.clone(), processed_url, sidecar.clone(), &batch, + processed_dimensions, ) .await { @@ -225,7 +232,13 @@ pub(crate) async fn separate_ui_impl( write_separation_state(state_path.clone(), separation.clone()).await?; return Err(error); } - patch::apply_batch_patch(&mut separation, tree_index, &binding.decisions, &cut_paths)?; + patch::apply_batch_patch( + &mut separation, + tree_index, + &binding.decisions, + &cut_paths, + processed_dimensions, + )?; write_separation_state(state_path.clone(), separation.clone()).await?; app_log!( "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={} elapsed_ms={}", diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs index 341416fd6..42985ac4f 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs @@ -11,6 +11,7 @@ pub fn apply_batch_patch( tree_index: usize, decisions: &[BindingDecision], cut_paths: &HashMap, + processed_dimensions: (u32, u32), ) -> Result<(), String> { app_log!( "ui_separation.batch_patch.start tree_index={} decisions={} cut_paths={}", @@ -34,6 +35,7 @@ pub fn apply_batch_patch( decisions: decisions.to_vec(), }, &batch, + processed_dimensions, )?; let rework_counts = batch_nodes .iter() diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index a1f13e20f..d15340b07 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -139,7 +139,7 @@ pub async fn create_vector_engine_raw_image_edit( // GPT-Image-2 原生只返回 b64_json；不要添加 URL 下载或 response_format 兼容分支。 .part( "image", - Part::stream_with_length(image_bytes.clone(), image_bytes.len() as u64) + Part::bytes(image_bytes.to_vec()) .file_name(image_file_name) .mime_str(image_mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, @@ -156,7 +156,7 @@ pub async fn create_vector_engine_raw_image_edit( if let Some(mask) = options.mask { form = form.part( "mask", - Part::stream_with_length(mask.bytes.clone(), mask.bytes.len() as u64) + Part::bytes(mask.bytes.to_vec()) .file_name(mask.file_name) .mime_str(mask.mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, From c86221e3dad286c6fadb43beaced0eaace87d7e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 20:12:16 +0800 Subject: [PATCH 154/248] =?UTF-8?q?=E6=B7=BB=E5=8A=A0=20TODO=20=E6=B3=A8?= =?UTF-8?q?=E9=87=8A=E4=BB=A5=E6=8F=90=E7=A4=BA=E6=81=A2=E5=A4=8D=E5=AD=97?= =?UTF-8?q?=E7=AC=A6=E9=99=90=E5=88=B6=E9=94=99=E8=AF=AF=E4=BF=A1=E6=81=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/separation/tree.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index f1d024827..8f9a454d0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -176,6 +176,7 @@ pub fn validate_binding_response( return Err("NeedRework 必须包含问题描述".to_string()); } if advice.chars().count() > MAX_REWORK_NOTE_CHARS { + // TODO add this back in prompt return Err(format!( "NeedRework 问题描述不能超过 {MAX_REWORK_NOTE_CHARS} 个字符" )); From 86c268eca32c8943f2bed83bef154d3cf185fb27 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 21:54:17 +0800 Subject: [PATCH 155/248] =?UTF-8?q?=E5=88=86=E7=A6=BB=E6=81=A2=E5=A4=8D?= =?UTF-8?q?=E4=BD=BF=E7=94=A8=E7=8B=AC=E7=AB=8B=E4=B8=A2=E5=BC=83=E6=93=8D?= =?UTF-8?q?=E4=BD=9C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增独立的恢复状态丢弃入口 完成与 finalize 的共享状态删除逻辑 --- apps/ai-game-creator-shell/src-tauri/src/main.rs | 2 +- .../src/ui_editor/commands/separation/persistence.rs | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/main.rs b/apps/ai-game-creator-shell/src-tauri/src/main.rs index b8096e5d1..1fb7e138b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/main.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/main.rs @@ -353,7 +353,7 @@ fn inspect_separation_recovery( fn finalize_separation(project_path: String, asset_id: String) -> Result<(), String> { let root = Path::new(project_path.trim()); enforce_project_permission_policy(root, "asset.register")?; - ui_editor::commands::separation::finalize_separation(root, &asset_id) + ui_editor::commands::separation::discard_separation_recovery(root, &asset_id) } #[tauri::command] diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 05a4ed683..5b663c609 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -187,6 +187,14 @@ pub fn inspect_separation_recovery( } pub fn finalize_separation(root: &Path, asset_id: &str) -> Result<(), String> { + remove_separation_state(root, asset_id) +} + +pub fn discard_separation_recovery(root: &Path, asset_id: &str) -> Result<(), String> { + remove_separation_state(root, asset_id) +} + +fn remove_separation_state(root: &Path, asset_id: &str) -> Result<(), String> { let state_path = separation_state_path(root, asset_id)?; match fs::remove_file(&state_path) { Ok(()) => Ok(()), From 49e88bbd9f420595e2d2cf6a96e11d027adad867 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Fri, 11 Sep 2026 21:55:15 +0800 Subject: [PATCH 156/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=8F=90=E7=A4=BA=E5=A4=87=E6=B3=A8=E9=95=BF=E5=BA=A6=E9=99=90?= =?UTF-8?q?=E5=88=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复用模型层的重做备注字符上限 导出提示文本清理函数供投影复用 --- .../src/ui_editor/commands/separation/model/note.rs | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs index cb1df4fd7..1c3065627 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs @@ -1,9 +1,7 @@ use serde::{Deserialize, Serialize}; use ts_rs::TS; -const MAX_PROMPT_NOTE_CHARS: usize = 512; - -fn sanitize_prompt_text(value: &str) -> String { +pub(crate) fn sanitize_prompt_text(value: &str) -> String { value .chars() .filter_map(|character| { @@ -15,7 +13,7 @@ fn sanitize_prompt_text(value: &str) -> String { Some(character) } }) - .take(MAX_PROMPT_NOTE_CHARS) + .take(super::MAX_REWORK_NOTE_CHARS) .collect() } From 4d9e69841e0dfee01ade369336c6b60652d5caed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:04:32 +0800 Subject: [PATCH 157/248] =?UTF-8?q?=E5=8A=A0=E5=9B=BA=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=8F=90=E7=A4=BA=E8=AF=8D=E7=94=9F=E6=88=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 绑定提示改用切片参数避免多余复制 清理提取提示中的描述与返工备注文本 --- .../ui_editor/commands/separation/prompt/binding.rs | 2 +- .../ui_editor/commands/separation/prompt/extract.rs | 13 ++++++++++--- .../commands/separation/workflow/binding.rs | 2 +- 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs index 2b77bf71c..575ebb913 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs @@ -1,7 +1,7 @@ use crate::ui_editor::commands::separation::image_preprocess::VISUAL_BINDING_TRANSPARENT_MARKER_RGBA; use crate::ui_editor::commands::separation::SeparationNode; -pub(crate) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String { +pub(crate) fn gen_binding_prompt(nodes: &[&SeparationNode]) -> String { let [marker_red, marker_green, marker_blue, marker_alpha] = VISUAL_BINDING_TRANSPARENT_MARKER_RGBA; let marker_color = format!("rgba({marker_red}, {marker_green}, {marker_blue}, {marker_alpha})"); diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs index 9b6e7a2ed..d6edfe0e5 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs @@ -1,4 +1,6 @@ -use crate::ui_editor::commands::separation::model::{SeparationState, SeparationTree}; +use crate::ui_editor::commands::separation::model::{ + sanitize_prompt_text, SeparationState, SeparationTree, +}; use crate::ui_editor::commands::separation::{SeparationNode, SeparationNodeKind}; use crate::ui_editor::utils::NodeId; use serde::Serialize; @@ -122,8 +124,13 @@ fn project_node( width: node.width_px, height: node.height_px, }, - description: node.note.description.clone(), - rework_notes: node.note.rework_notes.clone(), + description: sanitize_prompt_text(&node.note.description), + rework_notes: node + .note + .rework_notes + .iter() + .map(|note| sanitize_prompt_text(note)) + .collect(), children, } } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs index 7b13beaaf..255a204c7 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/binding.rs @@ -93,7 +93,7 @@ async fn visual_binding_inner( ) .with_strict(true); let initial_history = vec![ - LlmMessage::system(gen_binding_prompt(nodes.to_vec())), + LlmMessage::system(gen_binding_prompt(nodes)), LlmMessage::user_multimodal(vec![ LlmMessageContentPart::InputText { text: "processed image:".to_string(), From d7776f2a170977d79b07f51e116192670b9e59a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:05:18 +0800 Subject: [PATCH 158/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E6=96=87=E7=94=9F?= =?UTF-8?q?=E7=BB=91=E5=AE=9A=E6=8F=90=E7=A4=BA=E8=AF=AD=E6=B3=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 改进分离要求和返工建议英文表述 --- .../src/ui_editor/commands/separation/prompt/binding.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs index 575ebb913..e9253e810 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/binding.rs @@ -16,7 +16,7 @@ pub(crate) fn gen_binding_prompt(nodes: &[&SeparationNode]) -> String { Do not copy, infer, or reuse the source node rectangle. The src image is only for identifying which semantic UI element belongs to to_node. - Here were the separation requirements: + Here are the separation requirements: Preserve hard edges and the exact visible shape. The processed image is an opaque visual-binding preview containing the requested image layers. The solid color {marker_color} is an intentional transparency marker added by this workflow before this request. @@ -30,10 +30,10 @@ pub(crate) fn gen_binding_prompt(nodes: &[&SeparationNode]) -> String { * edge process ... - if not, use `NeedRework` data structure in the tool to indicate the (it should be from which) node id and advice. - your advice (less than 20 words) will be used to improve the separation in the next time. + if not, use the `NeedRework` data structure in the tool to indicate the node id and advice. + your advice (less than 20 words) will be used to improve the separation next time. - these node need handle: + these nodes need handling: "# ); let mut result = binding_system_prompt; From e04075fd957e12efc2689d6554fda4d6bbf43374 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:07:02 +0800 Subject: [PATCH 159/248] =?UTF-8?q?=E4=B8=BA=E5=88=86=E7=A6=BB=E6=A8=A1?= =?UTF-8?q?=E5=9E=8B=E8=AF=B7=E6=B1=82=E5=A2=9E=E5=8A=A0=E9=87=8D=E8=AF=95?= =?UTF-8?q?=E9=80=80=E9=81=BF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 瞬时请求失败时按递增毫秒延迟后重试 将重试循环末尾改为不可达断言 --- .../src-tauri/src/ui_editor/commands/utils.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index 5111aadd8..fe38199ab 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -55,6 +55,8 @@ where max_retries, error ); + tokio::time::sleep(std::time::Duration::from_millis(200 * (attempt as u64 + 1))) + .await; continue; } Err(error) => return Err(error), @@ -71,7 +73,7 @@ where Err(error) => return Err(error), } } - Err("LLM 修复重试流程未产生结果".to_string()) + unreachable!("重试循环在最后一次尝试时始终返回结果") } pub(crate) fn parse_limited_llm_tool_arguments( From 560422286ba2dd0e3c9725ed464215b1987d540c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:08:56 +0800 Subject: [PATCH 160/248] =?UTF-8?q?=E7=A8=B3=E5=AE=9A=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E9=9D=A2=E6=9D=BF=E5=B1=95=E5=BC=80=E4=B8=8E=E7=A7=BB=E9=99=A4?= =?UTF-8?q?=E4=BA=A4=E4=BA=92?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 仅在组件存在性变化时自动展开面板 为移除组件增加确认并保留失败弹窗 --- .../Inspector/Components/ComponentPanel.tsx | 50 +++++++++++++++++-- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index c4fa25f93..2e3955fd0 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -1,5 +1,5 @@ import { ChevronDown, ChevronRight, Plus, Trash2 } from 'lucide-react'; -import { useEffect, useState } from 'react'; +import { useEffect, useRef, useState } from 'react'; import { ThemedModal } from '../../../../../components/modal/ThemedModal'; import type { Component } from '../../../../../features/ui-editor/types/Component'; @@ -19,9 +19,15 @@ export function ComponentPanel(props: ComponentPanelProps) { const [expanded, setExpanded] = useState(Boolean(component)); const [error, setError] = useState(null); const [replaceConfirmationOpen, setReplaceConfirmationOpen] = useState(false); + const [removeConfirmationOpen, setRemoveConfirmationOpen] = useState(false); + const previousComponentRef = useRef(component); useEffect(() => { - setExpanded(Boolean(component)); + const previous = previousComponentRef.current; + previousComponentRef.current = component; + if (Boolean(component) !== Boolean(previous)) { + setExpanded(Boolean(component)); + } }, [component]); function setComponent(next: Component | null) { @@ -48,8 +54,15 @@ export function ComponentPanel(props: ComponentPanelProps) { function applyReplacement() { const result = setComponent(createComponent()); - if (result?.ok) setExpanded(true); - setReplaceConfirmationOpen(false); + if (result?.ok) { + setExpanded(true); + setReplaceConfirmationOpen(false); + } + } + + function removeComponent() { + const result = setComponent(null); + if (result?.ok) setRemoveConfirmationOpen(false); } return ( @@ -107,7 +120,7 @@ export function ComponentPanel(props: ComponentPanelProps) { type="button" className="grid size-7 place-items-center rounded-lg border border-red-200 bg-red-50 text-red-700 disabled:cursor-not-allowed disabled:opacity-35" disabled={readOnly} - onClick={() => setComponent(null)} + onClick={() => setRemoveConfirmationOpen(true)} aria-label="移除组件" title="移除组件" > @@ -153,6 +166,33 @@ export function ComponentPanel(props: ComponentPanelProps) {
+ setRemoveConfirmationOpen(false)} + panelClassName="w-full max-w-sm rounded-2xl p-5" + > +

确认移除组件？

+

+ 当前组件配置将被移除。 +

+
+ + +
+ ); } From ed2cfbad362b8faf21d3f49bae1390cf6f2a8d90 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:11:22 +0800 Subject: [PATCH 161/248] =?UTF-8?q?=E9=81=BF=E5=85=8D=E9=A2=84=E8=A7=88?= =?UTF-8?q?=E5=AE=B9=E5=99=A8=E5=8F=98=E5=8C=96=E9=87=8D=E7=BD=AE=E7=BC=A9?= =?UTF-8?q?=E6=94=BE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 自动适配仅在图片或逻辑尺寸变化时执行 通过尺寸引用保留初始视口兜底值 --- .../components/preview/PreviewWorkspace.tsx | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx index 62ce5f4d5..f9bfaa56d 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx @@ -55,6 +55,8 @@ export function PreviewWorkspace({ viewportRef.current, ); const [canvasSize, setCanvasSize] = useState({ width: 900, height: 640 }); + const canvasSizeRef = useRef(canvasSize); + canvasSizeRef.current = canvasSize; const [spaceHeld, setSpaceHeld] = useState(false); const [renderMode, setRenderMode] = useState('editor-overlay'); @@ -152,18 +154,12 @@ export function PreviewWorkspace({ height: logicalSize.height, }, canvasSize: { - width: element?.clientWidth || canvasSize.width, - height: element?.clientHeight || canvasSize.height, + width: element?.clientWidth || canvasSizeRef.current.width, + height: element?.clientHeight || canvasSizeRef.current.height, }, }), ); - }, [ - activeImageId, - canvasSize.height, - canvasSize.width, - logicalSize, - setViewport, - ]); + }, [activeImageId, logicalSize, setViewport]); const scaleViewportFromCenter = useCallback( (nextScale: number) => { From 0e67fbf80ab016540f57faf5f6bbd373d41f5bf7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:12:15 +0800 Subject: [PATCH 162/248] =?UTF-8?q?=E5=81=9C=E6=AD=A2=E5=AF=B9=E6=9C=80?= =?UTF-8?q?=E7=BB=88=E5=B7=A5=E4=BD=9C=E6=B5=81=E6=AD=A5=E9=AA=A4=E9=87=8D?= =?UTF-8?q?=E5=A4=8D=E9=A2=84=E6=A3=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 资产分离作为最后步骤不再校验不存在的下一步 --- .../src/view/ui-editor/components/WorkflowChecks.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts index cfff7ab03..5df5d0d9b 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/WorkflowChecks.ts @@ -59,6 +59,6 @@ export function activeStepPrerequisiteIssues( case 'structure-recognition': return validateAssetSeparationPrerequisites(state); case 'asset-separation': - return validateAssetSeparationPrerequisites(state); + return []; } } From 478e08fb92038ef2292a79adb294d8352b71a228 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:13:52 +0800 Subject: [PATCH 163/248] =?UTF-8?q?=E5=B0=86=20PNG=20=E4=BE=9D=E8=B5=96?= =?UTF-8?q?=E9=99=90=E5=AE=9A=E4=B8=BA=E6=B5=8B=E8=AF=95=E7=8E=AF=E5=A2=83?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 从生产依赖移除仅测试使用的 png crate 在 dev-dependencies 中声明 PNG 测试依赖 --- server-rs/crates/api-server/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server-rs/crates/api-server/Cargo.toml b/server-rs/crates/api-server/Cargo.toml index a5b2e3cec..415afd5b9 100644 --- a/server-rs/crates/api-server/Cargo.toml +++ b/server-rs/crates/api-server/Cargo.toml @@ -14,7 +14,6 @@ bytes = { workspace = true } dotenvy = { workspace = true } hex = { workspace = true } image = { workspace = true, features = ["jpeg", "png", "webp"] } -png = { workspace = true } http-body-util = { workspace = true } reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls"] } regex = { workspace = true } @@ -63,6 +62,7 @@ zip = { workspace = true, features = ["deflate"] } windows-sys = { workspace = true, features = ["Win32_Foundation", "Win32_System_Diagnostics_ToolHelp", "Win32_System_ProcessStatus", "Win32_System_Threading"] } [dev-dependencies] +png = { workspace = true } base64 = { workspace = true } http-body-util = { workspace = true } reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls"] } From 12fbb61fada325747dd132c3880a6cd7e51f1b0b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:17:55 +0800 Subject: [PATCH 164/248] =?UTF-8?q?=E9=99=90=E5=88=B6=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E7=8A=B6=E6=80=81=E5=86=99=E5=85=A5=E5=A4=A7?= =?UTF-8?q?=E5=B0=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 写盘前校验与读取一致的八兆字节上限 超限时记录原因并拒绝不可恢复检查点 --- .../src/ui_editor/commands/separation/persistence.rs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 5b663c609..a2f0728c4 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -66,6 +66,17 @@ fn write_separation_state_blocking(path: &Path, state: &SeparationState) -> Resu app_log!("ui_separation.error stage=state_write reason=serialize error={error}"); format!("序列化 separation state 失败：{error}") })?; + if bytes.len() > SEPARATION_STATE_MAX_BYTES { + app_log!( + "ui_separation.error stage=state_write reason=too_large bytes={} max_bytes={}", + bytes.len(), + SEPARATION_STATE_MAX_BYTES + ); + return Err(format!( + "separation state 超过 {} 字节上限", + SEPARATION_STATE_MAX_BYTES + )); + } let parent = path.parent().ok_or_else(|| { app_log!("ui_separation.error stage=state_write reason=missing_parent"); "separation state 路径缺少父目录".to_string() From e8cf773aacd046bb8f45507ffcf505b547cebf9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:19:48 +0800 Subject: [PATCH 165/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E5=A4=B1=E8=B4=A5=E5=AE=8C=E6=88=90=E6=8F=90?= =?UTF-8?q?=E7=A4=BA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 分离入口异常时写入工作流完成失败通知 --- .../src/view/ui-editor/useUiEditorPage.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 6f262cabe..1b8ea4f52 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1261,8 +1261,11 @@ export function useUiEditorSession( } await runSeparationWorkflow(); } catch (cause) { - setSeparationStatus( + reportWorkflowCompletion( + 'asset-separation', + 'failure', cause instanceof Error ? cause.message : String(cause), + setSeparationStatus, ); } } From eada20c19969cdd3cb3727b8ee160167fc4e8092 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:20:52 +0800 Subject: [PATCH 166/248] =?UTF-8?q?=E6=81=A2=E5=A4=8D=E8=87=AA=E5=8A=A8?= =?UTF-8?q?=E5=88=87=E5=88=86=E5=89=8D=E9=87=8D=E6=96=B0=E6=A0=A1=E9=AA=8C?= =?UTF-8?q?=E7=8A=B6=E6=80=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 继续分离前确认 sidecar 存在并检查前置条件 恢复入口异常统一写入工作流失败通知 --- .../src/view/ui-editor/useUiEditorPage.ts | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 1b8ea4f52..312a29aac 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1271,8 +1271,38 @@ export function useUiEditorSession( } async function continueSeparation() { + if (!resourceId) return; setSeparationRecovery(null); - await runSeparationWorkflow(); + try { + const recovery = await invoke( + 'inspect_separation_recovery', + { projectPath, assetId: resourceId }, + ); + if (!recovery.exists) { + throw new Error('自动切分恢复状态不存在，请重新开始。'); + } + const prerequisiteIssues = prerequisiteIssuesForStep( + editor.state, + 'asset-separation', + ); + if (prerequisiteIssues.length > 0) { + reportWorkflowCompletion( + 'asset-separation', + 'failure', + prerequisiteIssues.map((issue) => issue.message).join('；'), + setSeparationStatus, + ); + return; + } + await runSeparationWorkflow(); + } catch (cause) { + reportWorkflowCompletion( + 'asset-separation', + 'failure', + cause instanceof Error ? cause.message : String(cause), + setSeparationStatus, + ); + } } async function restartSeparation() { From bec468bdc97e9d68ac72bc0baebfde8131a9c021 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:23:02 +0800 Subject: [PATCH 167/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E7=BB=88=E6=AD=A2=E8=8A=82=E7=82=B9=E5=88=A4=E5=AE=9A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 批次选择与提取提示复用同一终止节点集合 --- .../ui_editor/commands/separation/prompt/extract.rs | 13 ++----------- .../ui_editor/commands/separation/workflow/batch.rs | 4 ++-- 2 files changed, 4 insertions(+), 13 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs index d6edfe0e5..aceb5c4df 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs @@ -1,6 +1,7 @@ use crate::ui_editor::commands::separation::model::{ sanitize_prompt_text, SeparationState, SeparationTree, }; +use crate::ui_editor::commands::separation::workflow::batch::terminal_node_ids; use crate::ui_editor::commands::separation::{SeparationNode, SeparationNodeKind}; use crate::ui_editor::utils::NodeId; use serde::Serialize; @@ -64,17 +65,7 @@ pub(crate) fn gen_extract_prompt( .iter() .map(|node| node.id.clone()) .collect::>(); - let terminal_ids = state - .bound - .iter() - .map(|node| node.node_id.clone()) - .chain( - state - .problematic_nodes - .iter() - .map(|node| node.node_id.clone()), - ) - .collect::>(); + let terminal_ids = terminal_node_ids(state); let mut index = 1; let document = ExtractPromptDocument { ui_layer_tree: project_node(&tree.root, &target_ids, &terminal_ids, &mut index), diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs index f4033880c..13024a78b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/batch.rs @@ -40,7 +40,7 @@ pub fn image_edit_dimension_for_area(area_px: u64) -> u32 { aligned.clamp(IMAGE_EDIT_MIN_DIMENSION_PX, IMAGE_EDIT_MAX_DIMENSION_PX) as u32 } -fn terminal_ids(state: &SeparationState) -> HashSet { +pub(crate) fn terminal_node_ids(state: &SeparationState) -> HashSet { state .bound .iter() @@ -82,7 +82,7 @@ pub fn next_image_batch_with_size<'a>( state: &SeparationState, tree: &'a SeparationTree, ) -> ImageBatch<'a> { - let terminal = terminal_ids(state); + let terminal = terminal_node_ids(state); let mut selected = Vec::new(); let mut area = 0; collect_dfs_batch( From 319aa69d6e3bb05417eb5674e4c84f5ff17cf64c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 02:24:01 +0800 Subject: [PATCH 168/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=A0=91=E6=A0=B9=E8=8A=82=E7=82=B9=E5=9D=90=E6=A0=87=E7=A9=BA?= =?UTF-8?q?=E9=97=B4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 子节点收集复用根节点实际解析矩形 --- .../src-tauri/src/ui_editor/commands/separation/tree.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs index 8f9a454d0..d632796ef 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/tree.rs @@ -100,9 +100,10 @@ pub fn construct_separation_state(state: &State) -> SeparationState { let size = image.pixel_size / ppu; let root_rect = crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size); + let root_resolved = tree.root.layout.transform.resolve(&root_rect); let mut children = Vec::new(); for child in &tree.root.children { - collect_todo_nodes(child, &root_rect, ppu, &mut children); + collect_todo_nodes(child, &root_resolved, ppu, &mut children); } let root_extractable = is_unbound_image(&tree.root); if !root_extractable && children.is_empty() { From 69ce232a746ffe9176d9ebfb9abb17c55f3703c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 10:20:17 +0800 Subject: [PATCH 169/248] =?UTF-8?q?=E4=B8=BA=E7=BB=91=E5=AE=9A=E5=8C=BA?= =?UTF-8?q?=E5=9F=9F=E6=A0=A1=E9=AA=8C=E5=BC=95=E5=85=A5=E7=B1=BB=E5=9E=8B?= =?UTF-8?q?=E9=94=99=E8=AF=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 用 ZeroDimension 和 OutOfBounds 区分边界校验失败 补充 typed error 单元测试并保留展示层中文文案 --- .../commands/separation/model/binding.rs | 53 +++++++++++++++++-- 1 file changed, 50 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs index d24f249eb..e527c7613 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/binding.rs @@ -2,6 +2,23 @@ use crate::ui_editor::utils::NodeId; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum BindingAreaValidationError { + ZeroDimension, + OutOfBounds, +} + +impl std::fmt::Display for BindingAreaValidationError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str(match self { + Self::ZeroDimension => "BindingArea 宽度和高度必须大于 0", + Self::OutOfBounds => "BindingArea 超出处理图边界", + }) + } +} + +impl std::error::Error for BindingAreaValidationError {} + #[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize, JsonSchema)] #[schemars(deny_unknown_fields)] pub struct BindingArea { @@ -12,9 +29,9 @@ pub struct BindingArea { } impl BindingArea { - pub fn validate_in(&self, w: u32, h: u32) -> Result<(), String> { + pub fn validate_in(&self, w: u32, h: u32) -> Result<(), BindingAreaValidationError> { if self.width_px == 0 || self.height_px == 0 { - return Err("BindingArea 宽度和高度必须大于 0".into()); + return Err(BindingAreaValidationError::ZeroDimension); } if self .global_pos_x_px @@ -25,7 +42,7 @@ impl BindingArea { .checked_add(self.height_px) .is_none_or(|v| v > h) { - return Err("BindingArea 超出处理图边界".into()); + return Err(BindingAreaValidationError::OutOfBounds); } Ok(()) } @@ -48,3 +65,33 @@ pub enum BindingDecision { pub struct BindingResp { pub decisions: Vec, } + +#[cfg(test)] +mod tests { + use super::{BindingArea, BindingAreaValidationError}; + + #[test] + fn validates_binding_area_with_typed_errors() { + let zero = BindingArea { + global_pos_x_px: 0, + global_pos_y_px: 0, + width_px: 0, + height_px: 1, + }; + assert_eq!( + zero.validate_in(10, 10), + Err(BindingAreaValidationError::ZeroDimension) + ); + + let outside = BindingArea { + global_pos_x_px: 10, + global_pos_y_px: 0, + width_px: 1, + height_px: 1, + }; + assert_eq!( + outside.validate_in(10, 10), + Err(BindingAreaValidationError::OutOfBounds) + ); + } +} From 915006c98f4f05822d3d527d5585571d6feea6e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 10:23:21 +0800 Subject: [PATCH 170/248] =?UTF-8?q?=E9=80=82=E9=85=8D=E7=BB=91=E5=AE=9A?= =?UTF-8?q?=E5=8C=BA=E5=9F=9F=E7=B1=BB=E5=9E=8B=E9=94=99=E8=AF=AF=E8=B0=83?= =?UTF-8?q?=E7=94=A8=E6=96=B9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在区域归一化边界转换 typed error 为展示文案 --- .../src-tauri/src/ui_editor/commands/separation/area.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs index 9685a0884..d861884d8 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/area.rs @@ -262,7 +262,7 @@ pub(crate) fn normalize_binding_area( original_area.width_px, original_area.height_px ); - return Err(error); + return Err(error.to_string()); } let original = Rect::from_area(original_area); let directions = Edge::ALL.map(|edge| edge_direction(image, original, edge)); From 2a50614c11a9da9f78c5c9577dd85cce0bd1a20e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 10:23:32 +0800 Subject: [PATCH 171/248] =?UTF-8?q?=E5=B0=86=E5=88=86=E7=A6=BB=E6=81=A2?= =?UTF-8?q?=E5=A4=8D=E9=98=BB=E5=A1=9E=20I/O=20=E7=A7=BB=E5=87=BA=E5=BC=82?= =?UTF-8?q?=E6=AD=A5=E7=BA=BF=E7=A8=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 新增 spawn_blocking 状态读取封装 sidecar 目录创建改用阻塞任务执行 --- .../commands/separation/persistence.rs | 6 +++++ .../commands/separation/workflow/mod.rs | 26 ++++++++++++------- 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index a2f0728c4..00221e9a8 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -159,6 +159,12 @@ pub fn read_separation_state(path: &Path) -> Result { Ok(state) } +pub async fn read_separation_state_async(path: PathBuf) -> Result { + tokio::task::spawn_blocking(move || read_separation_state(&path)) + .await + .map_err(|error| format!("读取 separation state 任务失败：{error}"))? +} + pub fn separation_dto(state: &SeparationState) -> SeparationDTO { app_log!( "ui_separation.dto bound_nodes={} problematic_nodes={} remaining_trees={}", diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index fb1c1e4dd..7a24146fc 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -9,7 +9,7 @@ pub use patch::apply_batch_patch; use self::batch::next_image_batch_with_size; use super::model::*; use super::persistence::{ - project_relative_path, read_separation_state, separation_dto, separation_sidecar_dir, + project_relative_path, read_separation_state_async, separation_dto, separation_sidecar_dir, separation_state_path, write_separation_state, }; use super::prompt::gen_extract_prompt; @@ -42,7 +42,13 @@ pub(crate) async fn separate_ui_impl( ); error })?; - fs::create_dir_all(&sidecar).map_err(|error| { + tokio::task::spawn_blocking({ + let sidecar = sidecar.clone(); + move || fs::create_dir_all(sidecar) + }) + .await + .map_err(|error| format!("创建 separation sidecar 任务失败：{error}"))? + .map_err(|error| { app_log!( "ui_separation.error stage=sidecar_create asset_id={} error={error}", asset_id @@ -53,13 +59,15 @@ pub(crate) async fn separate_ui_impl( let restored = state_path.exists(); let mut separation = if restored { app_log!("ui_separation.state_restore.start asset_id={asset_id}"); - read_separation_state(&state_path).map_err(|error| { - app_log!( - "ui_separation.error stage=state_restore asset_id={} error={error}", - asset_id - ); - error - })? + read_separation_state_async(state_path.clone()) + .await + .map_err(|error| { + app_log!( + "ui_separation.error stage=state_restore asset_id={} error={error}", + asset_id + ); + error + })? } else { app_log!("ui_separation.state_construct.start asset_id={asset_id}"); super::tree::construct_separation_state(&state) From 545f64cf9e91e19bb56ea91626505f37c8977c84 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 10:37:04 +0800 Subject: [PATCH 172/248] =?UTF-8?q?=E8=A1=A5=E9=BD=90=E5=8E=9F=E5=9B=BE?= =?UTF-8?q?=E7=BC=96=E8=BE=91=E5=A4=B1=E8=B4=A5=E8=BF=BD=E8=B8=AA=E4=BA=8B?= =?UTF-8?q?=E4=BB=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit provider 失败时记录 external_generation_run 失败状态 保留现有外部 API 失败审计与错误映射 --- server-rs/crates/api-server/src/raw_image.rs | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index b8d58a2d2..6d549c4c0 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -107,6 +107,21 @@ pub(crate) async fn edit_raw_image( Ok(generated) => generated, Err(error) => { record_openai_image_failure_if_configured(&audit_settings, &error).await; + if let Some(state) = tracking_state.as_ref() { + record_external_generation_run_after_success( + state, + platform_image::VECTOR_ENGINE_PROVIDER, + "raw_image_edit", + "raw_image_edit", + tracking_payload, + started_at_micros, + false, + Some(error.message().to_string()), + None, + None, + ) + .await; + } return Err(map_platform_image_error(error)); } }; From fc2f4b71ce6f75501c8d79f94e8edb75ff6b6b35 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 10:38:00 +0800 Subject: [PATCH 173/248] =?UTF-8?q?=E6=94=AF=E6=8C=81=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=B5=81=E7=A8=8B=E8=BD=AC=E6=8D=A2=E9=9D=9E=20PNG=20=E6=BA=90?= =?UTF-8?q?=E5=9B=BE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在上传图片编辑请求前将 JPEG 和 WebP 转码为 PNG 新增 JPEG 转 PNG 单元测试并将转码放入阻塞任务 --- .../commands/separation/workflow/extract.rs | 40 +++++++++++++++++-- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs index a5f08e433..08d8cffc4 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs @@ -54,15 +54,20 @@ async fn raw_extract_inner( .strip_prefix("data:") .and_then(|value| value.strip_suffix(";base64")) .unwrap_or("image/png"); - if !mime.eq_ignore_ascii_case("image/png") { - return Err("图片分离请求只支持 PNG 源图".to_string()); - } + let is_png = mime.eq_ignore_ascii_case("image/png"); let image_bytes = base64::engine::general_purpose::STANDARD .decode(data.trim()) .map_err(|error| format!("解码源图失败：{error}"))?; if image_bytes.is_empty() { return Err("源图不能为空".to_string()); } + let image_bytes = if is_png { + image_bytes + } else { + tokio::task::spawn_blocking(move || normalize_source_image_to_png(image_bytes)) + .await + .map_err(|error| format!("转换源图任务失败：{error}"))?? + }; let client = crate::http_client::agc_main_site_client_builder() .build() .map_err(|error| format!("创建图片编辑客户端失败：{error}"))?; @@ -122,6 +127,35 @@ async fn raw_extract_inner( result } +fn normalize_source_image_to_png(image_bytes: Vec) -> Result, String> { + let image = image::load_from_memory(&image_bytes) + .map_err(|error| format!("解码非 PNG 源图失败：{error}"))?; + let mut png_bytes = Vec::new(); + image + .write_to(&mut Cursor::new(&mut png_bytes), image::ImageFormat::Png) + .map_err(|error| format!("将源图转换为 PNG 失败：{error}"))?; + Ok(png_bytes) +} + +#[cfg(test)] +mod tests { + use super::normalize_source_image_to_png; + use image::{DynamicImage, ImageFormat, Rgb, RgbImage}; + use std::io::Cursor; + + #[test] + fn converts_jpeg_source_to_png() { + let image = DynamicImage::ImageRgb8(RgbImage::from_pixel(2, 2, Rgb([255, 0, 0]))); + let mut jpeg = Vec::new(); + image + .write_to(&mut Cursor::new(&mut jpeg), ImageFormat::Jpeg) + .expect("encode jpeg"); + + let png = normalize_source_image_to_png(jpeg).expect("convert jpeg"); + assert_eq!(&png[..8], b"\x89PNG\r\n\x1a\n"); + } +} + pub(super) async fn write_processed_image( processed_url: String, target: PathBuf, From afc9ecbd517a40088547be96b854aaf523cfc85b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 10:42:15 +0800 Subject: [PATCH 174/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20WebP=20=E6=BA=90?= =?UTF-8?q?=E5=9B=BE=E8=BD=AC=E7=A0=81=E6=B5=8B=E8=AF=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 验证 WebP 输入可转换为 PNG 上传格式 --- .../commands/separation/workflow/extract.rs | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs index 08d8cffc4..204d9141d 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs @@ -154,6 +154,18 @@ mod tests { let png = normalize_source_image_to_png(jpeg).expect("convert jpeg"); assert_eq!(&png[..8], b"\x89PNG\r\n\x1a\n"); } + + #[test] + fn converts_webp_source_to_png() { + let image = DynamicImage::ImageRgb8(RgbImage::from_pixel(2, 2, Rgb([0, 128, 255]))); + let mut webp = Vec::new(); + image + .write_to(&mut Cursor::new(&mut webp), ImageFormat::WebP) + .expect("encode webp"); + + let png = normalize_source_image_to_png(webp).expect("convert webp"); + assert_eq!(&png[..8], b"\x89PNG\r\n\x1a\n"); + } } pub(super) async fn write_processed_image( From 7edf4a8f8bc12f4516bf968d4a89d097a09649bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 11:06:05 +0800 Subject: [PATCH 175/248] =?UTF-8?q?=E6=81=A2=E5=A4=8D=E6=97=A7=E5=9B=BE?= =?UTF-8?q?=E7=89=87=E9=93=BE=E8=B7=AF=E5=AD=97=E8=8A=82=E7=B1=BB=E5=9E=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 撤回非 raw_edit 的全局 Bytes 迁移 保留既有 VectorEngine 与编辑器路径的 Vec 字节契约 --- .../crates/api-server/src/editor_project.rs | 25 ++++++------------- .../src/vector_engine/client.rs | 2 +- .../src/vector_engine/constants.rs | 7 ------ .../src/vector_engine/curl_transport.rs | 4 +-- .../src/vector_engine/image_source.rs | 7 +++--- .../src/vector_engine/request.rs | 19 ++++++-------- .../platform-image/src/vector_engine/types.rs | 12 ++------- 7 files changed, 24 insertions(+), 52 deletions(-) diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index c6529863a..b0663a4b1 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -19,7 +19,7 @@ use module_assets::{ AssetObjectAccessPolicy, AssetObjectFieldError, AssetObjectUpsertInput, build_asset_object_upsert_input, generate_asset_object_id, }; -use platform_image::{DownloadedImage, GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD}; +use platform_image::DownloadedImage; use platform_oss::{ LegacyAssetPrefix, OssHeadObjectRequest, OssObjectAccess, OssSignedGetObjectUrlRequest, }; @@ -3841,11 +3841,7 @@ fn editor_image_price_size_from_pixels(size: &str) -> &'static str { let Ok(height) = height.parse::() else { return "1K"; }; - if width.max(height) > GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD { - "2K" - } else { - "1K" - } + if width.max(height) > 1536 { "2K" } else { "1K" } } fn editor_image_edit_uses_nanobanana_generate_content(model: &str) -> bool { @@ -3894,11 +3890,7 @@ fn infer_editor_image_edit_size_tier(model: &str, size: Option<&str>) -> Option< if model == EDITOR_IMAGE_MODEL_NANOBANANA2 && long_edge <= 768 { return Some("0.5K"); } - Some(if long_edge > GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD { - "2K" - } else { - "1K" - }) + Some(if long_edge > 1536 { "2K" } else { "1K" }) } fn parse_editor_image_edit_pixel_size(size: &str) -> Option<(u32, u32)> { @@ -4646,7 +4638,7 @@ fn prepare_editor_image_edit_references( ); } for (index, reference) in reference_images.iter_mut().enumerate() { - let decoded = image::load_from_memory(reference.bytes.as_ref()).map_err(|error| { + let decoded = image::load_from_memory(reference.bytes.as_slice()).map_err(|error| { AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ "provider": "editor-image-edit", "message": format!("图片改造参考图不是有效图片：{error}"), @@ -4677,8 +4669,7 @@ fn prepare_editor_image_edit_references( image::DynamicImage::ImageRgba8(aligned), StatusCode::BAD_REQUEST, "图片改造参考图 16 对齐失败", - )? - .into(); + )?; reference.mime_type = "image/png".to_string(); reference.file_name = format!("editor-image-edit-reference-{}.png", index + 1); } @@ -13522,7 +13513,7 @@ pub(crate) async fn read_editor_reference_image_object_with_client( .to_string(); let extension = editor_reference_image_extension(mime_type.as_str()); Ok(OpenAiReferenceImage { - bytes: bytes.into(), + bytes, mime_type, file_name: format!("editor-reference.{extension}"), }) @@ -13536,7 +13527,7 @@ async fn download_editor_persisted_image_object( Ok(DownloadedOpenAiImage { extension: editor_reference_image_extension(image.mime_type.as_str()).to_string(), mime_type: image.mime_type, - bytes: image.bytes.to_vec(), + bytes: image.bytes, }) } @@ -19064,7 +19055,7 @@ mod tests { #[tokio::test] async fn icon_spritesheet_expired_deadline_skips_blocking_image_work() { let source = DownloadedImage { - bytes: bytes::Bytes::from_static(b"not-an-image"), + bytes: b"not-an-image".to_vec(), mime_type: "image/png".to_string(), extension: "png".to_string(), }; diff --git a/server-rs/crates/platform-image/src/vector_engine/client.rs b/server-rs/crates/platform-image/src/vector_engine/client.rs index 0dbf0be84..24b10196a 100644 --- a/server-rs/crates/platform-image/src/vector_engine/client.rs +++ b/server-rs/crates/platform-image/src/vector_engine/client.rs @@ -1107,7 +1107,7 @@ mod tests { fn reference_image(index: usize) -> ReferenceImage { ReferenceImage { - bytes: bytes::Bytes::from(vec![index as u8]), + bytes: vec![index as u8], mime_type: "image/png".to_string(), file_name: format!("reference-{index}.png"), } diff --git a/server-rs/crates/platform-image/src/vector_engine/constants.rs b/server-rs/crates/platform-image/src/vector_engine/constants.rs index 87572ca73..6480fba73 100644 --- a/server-rs/crates/platform-image/src/vector_engine/constants.rs +++ b/server-rs/crates/platform-image/src/vector_engine/constants.rs @@ -5,10 +5,3 @@ pub const VECTOR_ENGINE_GPT_IMAGE_2_MODEL: &str = GPT_IMAGE_2_MODEL; pub const VECTOR_ENGINE_PROVIDER: &str = "vector-engine"; pub const VECTOR_ENGINE_IMAGE_EDIT_MAX_REFERENCE_IMAGES: usize = 5; pub const VECTOR_ENGINE_NANOBANANA_MAX_REFERENCE_IMAGES: usize = 14; -/// GPT-Image-2 pricing uses 1K through this inclusive long-edge threshold. -pub const GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD: u32 = 1536; - -pub(crate) const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; -pub(crate) const GPT_IMAGE_2_MAX_PIXELS: u64 = 8_294_400; -pub(crate) const GPT_IMAGE_2_MAX_EDGE: u32 = 3_840; -pub(crate) const GPT_IMAGE_2_DIMENSION_ALIGNMENT: u32 = 16; diff --git a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs index 0c981f1a6..fbe94e1b1 100644 --- a/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs +++ b/server-rs/crates/platform-image/src/vector_engine/curl_transport.rs @@ -257,7 +257,7 @@ fn send_multipart_edit_request_with_curl_blocking( form.part("image") .buffer( reference_image.file_name.as_str(), - reference_image.bytes.to_vec(), + reference_image.bytes.clone(), ) .content_type(reference_image.mime_type.as_str()) .add()?; @@ -338,7 +338,7 @@ mod tests { "1024x1024", 1, &[ReferenceImage { - bytes: bytes::Bytes::from_static(b"reference"), + bytes: b"reference".to_vec(), mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }], diff --git a/server-rs/crates/platform-image/src/vector_engine/image_source.rs b/server-rs/crates/platform-image/src/vector_engine/image_source.rs index 7bbe073d4..dbc0b38a4 100644 --- a/server-rs/crates/platform-image/src/vector_engine/image_source.rs +++ b/server-rs/crates/platform-image/src/vector_engine/image_source.rs @@ -1,5 +1,4 @@ use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; -use bytes::Bytes; use reqwest::header; use std::time::Instant; @@ -52,7 +51,7 @@ pub async fn download_remote_image( Ok(DownloadedImage { extension: mime_to_extension(normalized_mime_type.as_str()).to_string(), mime_type: normalized_mime_type, - bytes: body, + bytes: body.to_vec(), }) } @@ -214,7 +213,7 @@ pub(crate) fn parse_reference_image_data_url( })?; let mime_type = normalize_downloaded_image_mime_type(mime_type); Ok(Some(ReferenceImage { - bytes: Bytes::from(bytes), + bytes, file_name: format!( "reference-{index}.{}", mime_to_extension(mime_type.as_str()) @@ -248,7 +247,7 @@ pub(crate) fn decode_generated_image_base64(raw: &str) -> Option String { } fn clamp_gpt_image_2_pixel_size(size: &str) -> String { - const MIN_PIXELS: u64 = GPT_IMAGE_2_MIN_PIXELS; - const MAX_PIXELS: u64 = GPT_IMAGE_2_MAX_PIXELS; - const MAX_EDGE: u32 = GPT_IMAGE_2_MAX_EDGE; - const DIMENSION_ALIGNMENT: u32 = GPT_IMAGE_2_DIMENSION_ALIGNMENT; + const MIN_PIXELS: u64 = 655_360; + const MAX_PIXELS: u64 = 8_294_400; + const MAX_EDGE: u32 = 3_840; + const DIMENSION_ALIGNMENT: u32 = 16; const MAX_ASPECT_RATIO: f64 = 3.0; // 中文注释：这里是 VectorEngine 的共享发送边界，只处理 gpt-image-2 的显式像素尺寸。 @@ -346,12 +343,12 @@ mod tests { 9, &[ ReferenceImage { - bytes: bytes::Bytes::from_static(&[1, 2, 3, 4, 5]), + bytes: vec![1, 2, 3, 4, 5], mime_type: "image/png".to_string(), file_name: "reference-a.png".to_string(), }, ReferenceImage { - bytes: bytes::Bytes::from(vec![8; 7]), + bytes: vec![8; 7], mime_type: "image/jpeg".to_string(), file_name: "reference-b.jpg".to_string(), }, diff --git a/server-rs/crates/platform-image/src/vector_engine/types.rs b/server-rs/crates/platform-image/src/vector_engine/types.rs index 17de6090b..77fbd19f9 100644 --- a/server-rs/crates/platform-image/src/vector_engine/types.rs +++ b/server-rs/crates/platform-image/src/vector_engine/types.rs @@ -1,5 +1,4 @@ use super::audit::PlatformImageFailureAudit; -use bytes::Bytes; #[derive(Clone, Debug)] pub struct VectorEngineImageSettings { @@ -17,23 +16,16 @@ pub struct GeneratedImages { pub recovered_failure_audits: Vec, } -#[derive(Clone, Debug)] -pub struct RawImageEditResult { - pub b64_images: Vec, - pub recovered_failure_audits: Vec, -} - #[derive(Clone, Debug)] pub struct DownloadedImage { - /// Shared immutable image bytes so callers can pass downloaded buffers downstream without copying. - pub bytes: Bytes, + pub bytes: Vec, pub mime_type: String, pub extension: String, } #[derive(Clone, Debug)] pub struct ReferenceImage { - pub bytes: Bytes, + pub bytes: Vec, pub mime_type: String, pub file_name: String, } From d05440dcca209ed18f8297ebfd1b878398e20fc3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 11:13:27 +0800 Subject: [PATCH 176/248] =?UTF-8?q?=E4=B8=BA=20Raw=20=E7=BC=96=E8=BE=91?= =?UTF-8?q?=E5=BB=BA=E7=AB=8B=E7=8B=AC=E7=AB=8B=E5=AD=97=E8=8A=82=E7=B1=BB?= =?UTF-8?q?=E5=9E=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 恢复旧图片链路的 Vec 字节契约 让新增 raw_edit 输入和结果类型独立于既有 ReferenceImage --- .../api-server/src/editor_project_icon.rs | 4 +- server-rs/crates/api-server/src/raw_image.rs | 11 ++++-- .../src/generated_asset_sheets/sheet.rs | 28 +++++++------- server-rs/crates/platform-image/src/lib.rs | 6 +-- .../platform-image/src/pixel_art_snapper.rs | 6 +-- .../platform-image/src/vector_engine/mod.rs | 13 +++---- .../src/vector_engine/raw_edit.rs | 37 ++++++++++++++----- .../tests/generated_asset_sheets.rs | 2 +- .../platform-image/tests/vector_engine.rs | 4 +- 9 files changed, 65 insertions(+), 46 deletions(-) diff --git a/server-rs/crates/api-server/src/editor_project_icon.rs b/server-rs/crates/api-server/src/editor_project_icon.rs index 39deb0bd7..f7f318b86 100644 --- a/server-rs/crates/api-server/src/editor_project_icon.rs +++ b/server-rs/crates/api-server/src/editor_project_icon.rs @@ -2653,7 +2653,7 @@ pub(crate) fn editor_icon_spritesheet_warning_after_persist_error( } fn validate_editor_icon_spritesheet_source(source: &DownloadedImage) -> Result<(), AppError> { - let reader = image::ImageReader::new(Cursor::new(source.bytes.as_ref())) + let reader = image::ImageReader::new(Cursor::new(source.bytes.as_slice())) .with_guessed_format() .map_err(|error| { AppError::from_status(StatusCode::UNPROCESSABLE_ENTITY).with_details(json!({ @@ -2931,7 +2931,7 @@ mod tests { .write_to(&mut std::io::Cursor::new(&mut bytes), ImageFormat::Png) .expect("fixture png should encode"); let source = DownloadedImage { - bytes: bytes.into(), + bytes, mime_type: "image/png".to_string(), extension: "png".to_string(), }; diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 6d549c4c0..1af3d4c91 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -7,7 +7,7 @@ use bytes::Bytes; use image::{GenericImageView, ImageFormat, ImageReader}; use platform_image::{ GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, - RawImageEditOptions, ReferenceImage, create_vector_engine_raw_image_edit, + RawImageEditImage, RawImageEditOptions, create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, }; use serde::Serialize; @@ -162,7 +162,7 @@ pub(crate) async fn edit_raw_image( } struct PreparedRawImageEdit { - image: ReferenceImage, + image: RawImageEditImage, prompt: String, options: RawImageEditOptions, width: u32, @@ -367,7 +367,10 @@ fn validate_optional_value( Err(bad_request(format!("{field} 值无效"))) } -fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32, u32), AppError> { +fn decode_image( + value: RawImageData, + field: &str, +) -> Result<(RawImageEditImage, u32, u32), AppError> { let mime_type = value.mime_type.trim().to_string(); if !mime_type.eq_ignore_ascii_case("image/png") { return Err(bad_request(format!( @@ -405,7 +408,7 @@ fn decode_image(value: RawImageData, field: &str) -> Result<(ReferenceImage, u32 } let (width, height) = decoded.dimensions(); Ok(( - ReferenceImage { + RawImageEditImage { bytes, file_name: value.file_name, mime_type, diff --git a/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs b/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs index 496bb9824..b0d7ec81b 100644 --- a/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs +++ b/server-rs/crates/platform-image/src/generated_asset_sheets/sheet.rs @@ -34,7 +34,7 @@ pub fn slice_generated_asset_sheet( let grid_size_u32 = u32::try_from(grid_size).map_err(|_| { GeneratedAssetSheetError::invalid_request("系列素材图集的 n 超出可支持范围。") })?; - let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("系列素材图集解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source); @@ -94,7 +94,7 @@ pub fn slice_generated_asset_sheet_two_items_per_row( let grid_size_u32 = u32::try_from(grid_size).map_err(|_| { GeneratedAssetSheetError::invalid_request("系列素材图集的 n 超出可支持范围。") })?; - let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("系列素材图集解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source); @@ -243,7 +243,7 @@ pub fn prepare_generated_icon_spritesheet_all_by_connected_components( "图标 spritesheet 累计裁剪像素上限必须大于 0。", )); } - let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("图标 spritesheet 解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source); @@ -263,7 +263,7 @@ pub fn prepare_generated_icon_spritesheet_all_by_connected_components( pub fn prepare_generated_icon_spritesheet_grid_2x2( image: &crate::DownloadedImage, ) -> Result { - let source = image::load_from_memory(image.bytes.as_ref()).map_err(|error| { + let source = image::load_from_memory(image.bytes.as_slice()).map_err(|error| { GeneratedAssetSheetError::decode_image(format!("图标 spritesheet 解码失败：{error}")) })?; let source = apply_generated_asset_sheet_green_screen_alpha(source).into_rgba8(); @@ -942,7 +942,7 @@ mod tests { } } crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), } @@ -1031,7 +1031,7 @@ mod tests { sheet.put_pixel(x0 + 40, y0 + 8, Rgba(color)); } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1063,7 +1063,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1095,7 +1095,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1137,7 +1137,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1165,7 +1165,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1206,7 +1206,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1240,7 +1240,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1272,7 +1272,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; @@ -1303,7 +1303,7 @@ mod tests { } let source = crate::DownloadedImage { - bytes: encode_png(sheet).into(), + bytes: encode_png(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), }; diff --git a/server-rs/crates/platform-image/src/lib.rs b/server-rs/crates/platform-image/src/lib.rs index 6c7d480fb..9ef40103c 100644 --- a/server-rs/crates/platform-image/src/lib.rs +++ b/server-rs/crates/platform-image/src/lib.rs @@ -12,9 +12,9 @@ pub use vector_engine::{ DownloadedImage, GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, GeneratedImages, NANOBANANA_2_MODEL, PlatformImageError, PlatformImageFailureAudit, PlatformImageStatusHint, RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, - RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, RawImageEditDimensionError, RawImageEditOptions, - RawImageEditResult, ReferenceImage, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, - VectorEngineImageSettings, build_vector_engine_image_http_client, + RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, RawImageEditDimensionError, RawImageEditImage, + RawImageEditOptions, RawImageEditResult, ReferenceImage, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, + VECTOR_ENGINE_PROVIDER, VectorEngineImageSettings, build_vector_engine_image_http_client, build_vector_engine_image_request_body, build_vector_engine_nanobanana_generate_content_request_body, create_vector_engine_image_edit, create_vector_engine_image_edit_with_references, diff --git a/server-rs/crates/platform-image/src/pixel_art_snapper.rs b/server-rs/crates/platform-image/src/pixel_art_snapper.rs index 9f7ac5c33..589a26a6f 100644 --- a/server-rs/crates/platform-image/src/pixel_art_snapper.rs +++ b/server-rs/crates/platform-image/src/pixel_art_snapper.rs @@ -266,7 +266,7 @@ fn decode_rgba_source( return Err(PixelArtSnapError::InvalidInput(format!("{input} 为空"))); } - let dimension_reader = image::ImageReader::new(Cursor::new(source.bytes.as_ref())) + let dimension_reader = image::ImageReader::new(Cursor::new(source.bytes.as_slice())) .with_guessed_format() .map_err(|error| PixelArtSnapError::Decode { input, @@ -281,7 +281,7 @@ fn decode_rgba_source( })?; validate_dimensions(width, height, input)?; - let mut reader = image::ImageReader::new(Cursor::new(source.bytes.as_ref())) + let mut reader = image::ImageReader::new(Cursor::new(source.bytes.as_slice())) .with_guessed_format() .map_err(|error| PixelArtSnapError::Decode { input, @@ -465,7 +465,7 @@ fn encode_png(image: RgbaImage) -> Result { .write_to(&mut cursor, ImageFormat::Png) .map_err(|error| PixelArtSnapError::Encode(error.to_string()))?; Ok(DownloadedImage { - bytes: cursor.into_inner().into(), + bytes: cursor.into_inner(), mime_type: "image/png".to_string(), extension: "png".to_string(), }) diff --git a/server-rs/crates/platform-image/src/vector_engine/mod.rs b/server-rs/crates/platform-image/src/vector_engine/mod.rs index 44941cefb..25e984180 100644 --- a/server-rs/crates/platform-image/src/vector_engine/mod.rs +++ b/server-rs/crates/platform-image/src/vector_engine/mod.rs @@ -21,14 +21,15 @@ pub use client::{ create_vector_engine_nanobanana_generate_content, }; pub use constants::{ - GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, NANOBANANA_2_MODEL, - VECTOR_ENGINE_GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, + GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL, NANOBANANA_2_MODEL, VECTOR_ENGINE_GPT_IMAGE_2_MODEL, + VECTOR_ENGINE_PROVIDER, }; pub use error::{PlatformImageError, PlatformImageStatusHint}; pub use image_source::download_remote_image; pub use raw_edit::{ - RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, - RawImageEditDimensionError, RawImageEditOptions, create_vector_engine_raw_image_edit, + GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, RAW_IMAGE_DIMENSION_ALIGNMENT, RAW_IMAGE_MAX_EDGE, + RAW_IMAGE_MAX_PIXELS, RAW_IMAGE_MIN_PIXELS, RawImageEditDimensionError, RawImageEditImage, + RawImageEditOptions, RawImageEditResult, create_vector_engine_raw_image_edit, validate_raw_image_edit_dimensions, }; pub use request::{ @@ -38,6 +39,4 @@ pub use request::{ vector_engine_nanobanana_generate_content_url, }; pub use transport::build_vector_engine_image_http_client; -pub use types::{ - DownloadedImage, GeneratedImages, RawImageEditResult, ReferenceImage, VectorEngineImageSettings, -}; +pub use types::{DownloadedImage, GeneratedImages, ReferenceImage, VectorEngineImageSettings}; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index d15340b07..c4399f3de 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -1,18 +1,16 @@ use std::time::{Duration, Instant}; +use bytes::Bytes; use reqwest::multipart::{Form, Part}; use serde::Deserialize; use super::{ audit::build_failure_audit, budget::{effective_request_timeout_ms, request_budget_exhausted_error}, - constants::{ - GPT_IMAGE_2_DIMENSION_ALIGNMENT, GPT_IMAGE_2_MAX_EDGE, GPT_IMAGE_2_MAX_PIXELS, - GPT_IMAGE_2_MIN_PIXELS, GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, - }, + constants::{GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER}, error::PlatformImageError, request::vector_engine_images_edit_url, - types::{RawImageEditResult, ReferenceImage, VectorEngineImageSettings}, + types::VectorEngineImageSettings, util::truncate_raw, }; @@ -23,9 +21,27 @@ pub struct RawImageEditOptions { pub output_format: Option, pub width: u32, pub height: u32, - pub mask: Option, + pub mask: Option, } +#[derive(Clone, Debug)] +pub struct RawImageEditImage { + pub bytes: Bytes, + pub mime_type: String, + pub file_name: String, +} + +#[derive(Clone, Debug)] +pub struct RawImageEditResult { + pub b64_images: Vec, + pub recovered_failure_audits: Vec, +} + +const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; +const GPT_IMAGE_2_MAX_PIXELS: u64 = 8_294_400; +const GPT_IMAGE_2_MAX_EDGE: u32 = 3_840; +const GPT_IMAGE_2_DIMENSION_ALIGNMENT: u32 = 16; + #[derive(Debug, Deserialize)] struct RawImageEditResponsePayload { data: Vec, @@ -40,6 +56,7 @@ pub const RAW_IMAGE_MAX_EDGE: u32 = GPT_IMAGE_2_MAX_EDGE; pub const RAW_IMAGE_DIMENSION_ALIGNMENT: u32 = GPT_IMAGE_2_DIMENSION_ALIGNMENT; pub const RAW_IMAGE_MIN_PIXELS: u64 = GPT_IMAGE_2_MIN_PIXELS; pub const RAW_IMAGE_MAX_PIXELS: u64 = GPT_IMAGE_2_MAX_PIXELS; +pub const GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD: u32 = 1536; #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum RawImageEditDimensionError { @@ -104,7 +121,7 @@ pub fn validate_raw_image_edit_dimensions( pub async fn create_vector_engine_raw_image_edit( settings: &VectorEngineImageSettings, prompt: &str, - image: ReferenceImage, + image: RawImageEditImage, options: RawImageEditOptions, failure_context: &str, ) -> Result { @@ -126,7 +143,7 @@ pub async fn create_vector_engine_raw_image_edit( reference_image_count, )); }; - let ReferenceImage { + let RawImageEditImage { bytes: image_bytes, file_name: image_file_name, mime_type: image_mime_type, @@ -139,7 +156,7 @@ pub async fn create_vector_engine_raw_image_edit( // GPT-Image-2 原生只返回 b64_json；不要添加 URL 下载或 response_format 兼容分支。 .part( "image", - Part::bytes(image_bytes.to_vec()) + Part::stream(reqwest::Body::from(image_bytes)) .file_name(image_file_name) .mime_str(image_mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, @@ -156,7 +173,7 @@ pub async fn create_vector_engine_raw_image_edit( if let Some(mask) = options.mask { form = form.part( "mask", - Part::bytes(mask.bytes.to_vec()) + Part::stream(reqwest::Body::from(mask.bytes)) .file_name(mask.file_name) .mime_str(mask.mime_type.as_str()) .map_err(|error| invalid_request(failure_context, error.to_string()))?, diff --git a/server-rs/crates/platform-image/tests/generated_asset_sheets.rs b/server-rs/crates/platform-image/tests/generated_asset_sheets.rs index b3c860528..d3bc61705 100644 --- a/server-rs/crates/platform-image/tests/generated_asset_sheets.rs +++ b/server-rs/crates/platform-image/tests/generated_asset_sheets.rs @@ -42,7 +42,7 @@ fn build_test_sheet(width: u32, height: u32) -> DownloadedImage { } DownloadedImage { - bytes: encode_image(sheet).into(), + bytes: encode_image(sheet), mime_type: "image/png".to_string(), extension: "png".to_string(), } diff --git a/server-rs/crates/platform-image/tests/vector_engine.rs b/server-rs/crates/platform-image/tests/vector_engine.rs index c764d3e67..f1bd4470b 100644 --- a/server-rs/crates/platform-image/tests/vector_engine.rs +++ b/server-rs/crates/platform-image/tests/vector_engine.rs @@ -259,7 +259,7 @@ async fn vector_engine_image_edit_retries_send_timeout_once_and_succeeds() { let http_client = build_vector_engine_image_http_client(&settings).expect("client should build"); let reference_image = ReferenceImage { - bytes: bytes::Bytes::from_static(b"reference"), + bytes: b"reference".to_vec(), mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }; @@ -598,7 +598,7 @@ async fn vector_engine_image_edit_falls_back_when_preferred_model_is_unsupported let http_client = build_vector_engine_image_http_client(&settings).expect("client should build"); let reference = ReferenceImage { - bytes: bytes::Bytes::from_static(b"reference"), + bytes: b"reference".to_vec(), mime_type: "image/png".to_string(), file_name: "reference.png".to_string(), }; From 551fb6609a2eede96dd3b67ea35496bf46874086 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 11:15:15 +0800 Subject: [PATCH 177/248] =?UTF-8?q?=E4=B8=BA=20Raw=20=E7=BC=96=E8=BE=91?= =?UTF-8?q?=E5=BB=BA=E7=AB=8B=E7=8B=AC=E7=AB=8B=E5=AD=97=E8=8A=82=E7=B1=BB?= =?UTF-8?q?=E5=9E=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 恢复旧图片链路的 Vec 字节契约 让新增 raw_edit 输入和结果类型独立于既有 ReferenceImage --- server-rs/crates/platform-image/src/vector_engine/raw_edit.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index c4399f3de..a749f06db 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -154,6 +154,8 @@ pub async fn create_vector_engine_raw_image_edit( .text("prompt", prompt.to_string()) .text("size", format!("{}x{}", options.width, options.height)) // GPT-Image-2 原生只返回 b64_json；不要添加 URL 下载或 response_format 兼容分支。 + // `Part::bytes` 只接受 `Cow<[u8]>`，会迫使 `Bytes` 复制成 `Vec`；stream Body + // 直接接管共享缓冲，避免 raw_edit 请求的整图/掩码额外分配和峰值内存。 .part( "image", Part::stream(reqwest::Body::from(image_bytes)) From 9c04561419eb7e6f8a4282070aa133abd1a3ae99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 11:18:14 +0800 Subject: [PATCH 178/248] =?UTF-8?q?=E6=98=8E=E7=A1=AE=20Raw=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E9=9B=B6=E6=8B=B7=E8=B4=9D=E4=B8=8A=E4=BC=A0=E7=BA=A6?= =?UTF-8?q?=E6=9D=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 补充 Bytes 所有权与 streaming multipart 注释 增加共享字节缓冲不转换的回归测试 --- .../platform-image/src/vector_engine/raw_edit.rs | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index a749f06db..1bb1d2e25 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -25,6 +25,8 @@ pub struct RawImageEditOptions { } #[derive(Clone, Debug)] +/// Multipart input owned by the raw-edit endpoint. `Bytes` keeps the Axum +/// upload buffer shared until reqwest consumes it, avoiding a full-image copy. pub struct RawImageEditImage { pub bytes: Bytes, pub mime_type: String, @@ -489,4 +491,18 @@ mod tests { "raw_image_edit：构造请求客户端失败：builder failed" ); } + + #[test] + fn raw_edit_image_owns_shared_bytes_without_conversion() { + let bytes = bytes::Bytes::from_static(b"raw-image"); + let pointer = bytes.as_ptr(); + let image = RawImageEditImage { + bytes, + mime_type: "image/png".to_string(), + file_name: "image.png".to_string(), + }; + + assert_eq!(image.bytes.as_ptr(), pointer); + assert_eq!(image.bytes.as_ref(), b"raw-image"); + } } From b8a4523a4fa3f73ee0a510cd426b5ebcc1956a53 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 11:20:53 +0800 Subject: [PATCH 179/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=20Raw=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E5=AD=97=E8=8A=82=E8=BE=B9=E7=95=8C=E8=AF=B4=E6=98=8E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 记录 raw_edit 使用 Bytes streaming multipart 的零拷贝约束 明确既有图片链路继续保持 Vec 字节契约 --- .../【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 998d0aed4..4fb70506b 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -91,6 +91,8 @@ provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端 - `server-rs/crates/api-server/src/raw_image.rs`：独立路由 handler、multipart 字段解析、请求/响应 DTO、PNG 输入校验、预检查和 raw billing 编排。 - `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、严格尺寸校验、独立 provider 请求映射和 `b64_json` 响应透传；不复用现有 editor 图片编辑 client 或其 multipart transport。 + +raw-edit 的图片输入使用独立的 `RawImageEditImage`（`bytes::Bytes`），由 reqwest `Part::stream(Body::from(Bytes))` 直接接管 multipart 请求体，避免整图和掩码在 `Part::bytes` 的 `Cow<[u8]>` 转换中再次复制。既有 `ReferenceImage`、`DownloadedImage` 及 curl/编辑器链路继续保持 `Vec` 契约，不因 raw-edit 引入全局字节类型迁移。 - `server-rs/crates/api-server/src/modules/raw.rs`：只注册 `/api/raw/v1/images/edit` 并挂载 Bearer middleware。 不修改 External v1 OpenAPI；不在 `external_editor_api.rs`、编辑器项目模块或外部生成 worker 中增加 raw 分支。 From 23fdc005fecfc21046860566305f0738b948edeb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 11:22:50 +0800 Subject: [PATCH 180/248] =?UTF-8?q?=E6=95=B4=E7=90=86=20Raw=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E6=96=B9=E6=A1=88=E6=96=87=E6=A1=A3=E6=A0=BC=E5=BC=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 分隔字节边界说明与代码拆分列表 --- .../【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 4fb70506b..393eb6fb4 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -93,6 +93,7 @@ provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端 - `server-rs/crates/platform-image/src/vector_engine/raw_edit.rs`：raw 编辑选项、严格尺寸校验、独立 provider 请求映射和 `b64_json` 响应透传；不复用现有 editor 图片编辑 client 或其 multipart transport。 raw-edit 的图片输入使用独立的 `RawImageEditImage`（`bytes::Bytes`），由 reqwest `Part::stream(Body::from(Bytes))` 直接接管 multipart 请求体，避免整图和掩码在 `Part::bytes` 的 `Cow<[u8]>` 转换中再次复制。既有 `ReferenceImage`、`DownloadedImage` 及 curl/编辑器链路继续保持 `Vec` 契约，不因 raw-edit 引入全局字节类型迁移。 + - `server-rs/crates/api-server/src/modules/raw.rs`：只注册 `/api/raw/v1/images/edit` 并挂载 Bearer middleware。 不修改 External v1 OpenAPI；不在 `external_editor_api.rs`、编辑器项目模块或外部生成 worker 中增加 raw 分支。 From a8a30ecbc8d558c86f7c884e20b77cd5784450ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:13:06 +0800 Subject: [PATCH 181/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E5=88=87=E5=88=86?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E5=91=BD=E4=BB=A4=E6=98=A0=E5=B0=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit finalize_separation 调用 finalize helper discard_separation_recovery 调用 discard helper --- apps/ai-game-creator-shell/src-tauri/src/main.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/main.rs b/apps/ai-game-creator-shell/src-tauri/src/main.rs index 811214345..2d1e31d15 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/main.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/main.rs @@ -353,14 +353,14 @@ fn inspect_separation_recovery( fn finalize_separation(project_path: String, asset_id: String) -> Result<(), String> { let root = Path::new(project_path.trim()); enforce_project_permission_policy(root, "asset.register")?; - ui_editor::commands::separation::discard_separation_recovery(root, &asset_id) + ui_editor::commands::separation::finalize_separation(root, &asset_id) } #[tauri::command] fn discard_separation_recovery(project_path: String, asset_id: String) -> Result<(), String> { let root = Path::new(project_path.trim()); enforce_project_permission_policy(root, "asset.register")?; - ui_editor::commands::separation::finalize_separation(root, &asset_id) + ui_editor::commands::separation::discard_separation_recovery(root, &asset_id) } #[tauri::command] From 46984a4945d25c950621af1b1dc4cb5ea9235004 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:13:25 +0800 Subject: [PATCH 182/248] =?UTF-8?q?=E5=87=80=E5=8C=96=E5=88=87=E5=88=86?= =?UTF-8?q?=E6=8F=90=E7=A4=BA=E4=B8=AD=E7=9A=84=E8=8A=82=E7=82=B9=E6=A0=87?= =?UTF-8?q?=E8=AF=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 复用提示文本清洗逻辑，阻断控制字符与反引号注入 --- .../src-tauri/src/ui_editor/commands/separation/model/node.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs index d925b7e36..f06c56fd1 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs @@ -28,7 +28,7 @@ impl SeparationNode { pub fn as_prompt(&self) -> String { format!( "node_id={} note: {}", - self.id.as_str(), + super::sanitize_prompt_text(self.id.as_str()), self.note.as_prompt() ) } From 8beade5b4df81fddc792b480129d444cb91f8baf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:13:56 +0800 Subject: [PATCH 183/248] =?UTF-8?q?=E4=B8=BA=E4=B8=9A=E5=8A=A1=E6=A0=A1?= =?UTF-8?q?=E9=AA=8C=E9=87=8D=E8=AF=95=E5=A2=9E=E5=8A=A0=E9=80=80=E9=81=BF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 校验失败重试复用请求错误的递增等待并记录日志 --- .../src-tauri/src/ui_editor/commands/utils.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index fe38199ab..d432c09e1 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -64,6 +64,16 @@ where match validator(&value) { Ok(()) => return Ok(value), Err(error) if attempt < max_retries => { + app_log!( + "ui_editor.llm.retry validation_error attempt={} max_retries={} error={}", + attempt + 1, + max_retries, + error + ); + tokio::time::sleep(std::time::Duration::from_millis( + 200 * (attempt as u64 + 1), + )) + .await; let serialized = serde_json::to_string(&value) .map_err(|serialize_error| format!("序列化修复反馈失败：{serialize_error}"))?; history.push(LlmMessage::system(format!( From 0726106818a69f9409f5c77c72f72197c693d076 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:14:22 +0800 Subject: [PATCH 184/248] =?UTF-8?q?=E4=B8=BA=E5=9B=BE=E7=89=87=E5=88=86?= =?UTF-8?q?=E7=A6=BB=E8=AF=B7=E6=B1=82=E8=AE=BE=E7=BD=AE=E8=B6=85=E6=97=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将原图编辑 HTTP 请求限制为 120 秒，避免工作流永久阻塞 --- .../src/ui_editor/commands/separation/workflow/extract.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs index 204d9141d..b359fb32e 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs @@ -69,6 +69,7 @@ async fn raw_extract_inner( .map_err(|error| format!("转换源图任务失败：{error}"))?? }; let client = crate::http_client::agc_main_site_client_builder() + .timeout(std::time::Duration::from_secs(120)) .build() .map_err(|error| format!("创建图片编辑客户端失败：{error}"))?; let url = format!( From d8a4e3523257d4a7660f9f5454ff52fb1eb8ef5b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:16:25 +0800 Subject: [PATCH 185/248] =?UTF-8?q?=E5=87=8F=E5=B0=91=E5=88=87=E5=88=86?= =?UTF-8?q?=E7=8A=B6=E6=80=81=E5=86=99=E5=85=A5=E7=9A=84=E9=87=8D=E5=A4=8D?= =?UTF-8?q?=E5=85=8B=E9=9A=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 让状态写入接口借用状态并在阻塞任务中写入序列化字节 保留 schema 与大小校验，避免每批次复制完整状态树 --- .../commands/separation/persistence.rs | 40 ++++++++----------- .../commands/separation/workflow/mod.rs | 12 +++--- .../src-tauri/src/ui_editor/commands/utils.rs | 6 +-- 3 files changed, 25 insertions(+), 33 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 00221e9a8..c35737d86 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -42,30 +42,19 @@ pub fn project_relative_path(root: &Path, path: &Path) -> Result Ok(value) } -pub async fn write_separation_state(path: PathBuf, state: SeparationState) -> Result<(), String> { +pub async fn write_separation_state(path: PathBuf, state: &SeparationState) -> Result<(), String> { + if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { + app_log!("ui_separation.error stage=state_write reason=schema_mismatch"); + return Err("不支持的 separation state schema".to_string()); + } + let state = serde_json::to_vec_pretty(state) + .map_err(|error| format!("序列化 separation state 失败：{error}"))?; tokio::task::spawn_blocking(move || write_separation_state_blocking(&path, &state)) .await .map_err(|error| format!("写入 separation state 任务失败：{error}"))? } -fn write_separation_state_blocking(path: &Path, state: &SeparationState) -> Result<(), String> { - if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION { - app_log!("ui_separation.error stage=state_write reason=schema_mismatch"); - return Err("不支持的 separation state schema".to_string()); - } - app_log!( - "ui_separation.state_write.start file={} trees={} bound={} problematic={}", - path.file_name() - .and_then(|name| name.to_str()) - .unwrap_or(""), - state.trees.len(), - state.bound.len(), - state.problematic_nodes.len() - ); - let bytes = serde_json::to_vec_pretty(state).map_err(|error| { - app_log!("ui_separation.error stage=state_write reason=serialize error={error}"); - format!("序列化 separation state 失败：{error}") - })?; +fn write_separation_state_blocking(path: &Path, bytes: &[u8]) -> Result<(), String> { if bytes.len() > SEPARATION_STATE_MAX_BYTES { app_log!( "ui_separation.error stage=state_write reason=too_large bytes={} max_bytes={}", @@ -77,6 +66,13 @@ fn write_separation_state_blocking(path: &Path, state: &SeparationState) -> Resu SEPARATION_STATE_MAX_BYTES )); } + app_log!( + "ui_separation.state_write.start file={} bytes={}", + path.file_name() + .and_then(|name| name.to_str()) + .unwrap_or(""), + bytes.len() + ); let parent = path.parent().ok_or_else(|| { app_log!("ui_separation.error stage=state_write reason=missing_parent"); "separation state 路径缺少父目录".to_string() @@ -95,16 +91,14 @@ fn write_separation_state_blocking(path: &Path, state: &SeparationState) -> Resu format!("安装 separation state 失败：{error}") })?; app_log!( - "ui_separation.state_write.completed file={} bytes={} trees={} bound={} problematic={}", + "ui_separation.state_write.completed file={} bytes={}", path.file_name() .and_then(|name| name.to_str()) .unwrap_or(""), fs::metadata(path) .map(|metadata| metadata.len()) .unwrap_or(0), - state.trees.len(), - state.bound.len(), - state.problematic_nodes.len() + bytes.len() ); Ok(()) } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 7a24146fc..960cc7ee9 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -106,7 +106,7 @@ pub(crate) async fn separate_ui_impl( image.pixel_size.x.round() as u32, image.pixel_size.y.round() as u32 ); - write_separation_state(state_path.clone(), separation.clone()) + write_separation_state(state_path.clone(), &separation) .await .map_err(|error| { app_log!( @@ -156,7 +156,7 @@ pub(crate) async fn separate_ui_impl( Ok(value) => value, Err(error) => { app_log!("ui_separation.error stage=image_edit tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(state_path.clone(), separation.clone()).await?; + write_separation_state(state_path.clone(), &separation).await?; return Err(error); } }; @@ -170,7 +170,7 @@ pub(crate) async fn separate_ui_impl( Ok(dimensions) => dimensions, Err(error) => { app_log!("ui_separation.error stage=processed_image_write tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(state_path.clone(), separation.clone()).await?; + write_separation_state(state_path.clone(), &separation).await?; return Err(error); } }; @@ -186,7 +186,7 @@ pub(crate) async fn separate_ui_impl( Ok(value) => value, Err(error) => { app_log!("ui_separation.error stage=visual_binding tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(state_path.clone(), separation.clone()).await?; + write_separation_state(state_path.clone(), &separation).await?; return Err(error); } }; @@ -237,7 +237,7 @@ pub(crate) async fn separate_ui_impl( } if let Some(error) = cut_error { app_log!("ui_separation.error stage=cut_batch tree_index={} batch_index={} error={error}", tree_index, batch_index); - write_separation_state(state_path.clone(), separation.clone()).await?; + write_separation_state(state_path.clone(), &separation).await?; return Err(error); } patch::apply_batch_patch( @@ -247,7 +247,7 @@ pub(crate) async fn separate_ui_impl( &cut_paths, processed_dimensions, )?; - write_separation_state(state_path.clone(), separation.clone()).await?; + write_separation_state(state_path.clone(), &separation).await?; app_log!( "ui_separation.batch_completed tree_index={} batch_index={} cuts={} bound={} problematic={} elapsed_ms={}", tree_index, batch_index, cut_paths.len(), separation.bound.len(), separation.problematic_nodes.len(), batch_started.elapsed().as_millis() diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs index d432c09e1..c9b5e2bc9 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/utils.rs @@ -70,10 +70,8 @@ where max_retries, error ); - tokio::time::sleep(std::time::Duration::from_millis( - 200 * (attempt as u64 + 1), - )) - .await; + tokio::time::sleep(std::time::Duration::from_millis(200 * (attempt as u64 + 1))) + .await; let serialized = serde_json::to_string(&value) .map_err(|serialize_error| format!("序列化修复反馈失败：{serialize_error}"))?; history.push(LlmMessage::system(format!( From cdb6fcfe979583546050d998dfce547d048f3bee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:17:01 +0800 Subject: [PATCH 186/248] =?UTF-8?q?=E5=88=87=E6=8D=A2=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E6=97=B6=E9=87=8D=E7=BD=AE=E6=A3=80=E6=9F=A5=E9=9D=A2=E6=9D=BF?= =?UTF-8?q?=E5=B1=95=E5=BC=80=E7=8A=B6=E6=80=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按组件身份变化同步展开状态，避免沿用上一个节点的折叠状态 --- .../components/Inspector/Components/ComponentPanel.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index 2e3955fd0..c8ba9afad 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -25,7 +25,7 @@ export function ComponentPanel(props: ComponentPanelProps) { useEffect(() => { const previous = previousComponentRef.current; previousComponentRef.current = component; - if (Boolean(component) !== Boolean(previous)) { + if (component !== previous) { setExpanded(Boolean(component)); } }, [component]); From 983b15c3a72556d09b48210ef3be031224f3b0f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:17:38 +0800 Subject: [PATCH 187/248] =?UTF-8?q?=E5=85=81=E8=AE=B8=E6=81=A2=E5=A4=8D?= =?UTF-8?q?=E6=B5=81=E7=A8=8B=E8=AF=86=E5=88=AB=E5=B7=B2=E5=AE=8C=E6=88=90?= =?UTF-8?q?=E7=9A=84=E7=B4=A0=E6=9D=90=E5=9B=9E=E5=A1=AB?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 同一 Sprite 已绑定的节点直接跳过 仅对缺失 Image 或绑定其他素材报告错误 --- .../src/view/ui-editor/useUiEditorPage.ts | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 312a29aac..9e279db75 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1164,13 +1164,18 @@ export function useUiEditorSession( continue; } const imageComponent = location.node.component; - if ( - !imageComponent || - !('Image' in imageComponent) || - imageComponent.Image.target_graphic !== null - ) { + if (!imageComponent || !('Image' in imageComponent)) { backfillErrors.push( - `节点 ${bound.node_id} 没有可回填的未绑定 Image 组件，素材已保留`, + `节点 ${bound.node_id} 没有可回填的 Image 组件，素材已保留`, + ); + continue; + } + if (imageComponent.Image.target_graphic === sprite.asset_id) { + continue; + } + if (imageComponent.Image.target_graphic !== null) { + backfillErrors.push( + `节点 ${bound.node_id} 已绑定其他素材，自动切分素材已保留`, ); continue; } From a4b1d8112378943555233290a6df07196d890af1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:17:52 +0800 Subject: [PATCH 188/248] =?UTF-8?q?=E9=81=BF=E5=85=8D=E5=BF=99=E7=A2=8C?= =?UTF-8?q?=E6=97=B6=E9=87=8D=E5=90=AF=E5=88=87=E5=88=86=E4=B8=A2=E5=A4=B1?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E7=8A=B6=E6=80=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit restartSeparation 在清理 sidecar 前复用并发工作流守卫 --- .../ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 9e279db75..584ec4c9f 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1311,7 +1311,7 @@ export function useUiEditorSession( } async function restartSeparation() { - if (!resourceId) return; + if (!resourceId || isSeparating || isWorkflowBusy) return; setSeparationRecovery(null); const prerequisiteIssues = prerequisiteIssuesForStep( editor.state, From 7e7d6f1563846c213728e810f37791b60da44988 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:18:13 +0800 Subject: [PATCH 189/248] =?UTF-8?q?=E9=81=BF=E5=85=8D=E5=BF=99=E7=A2=8C?= =?UTF-8?q?=E6=97=B6=E7=BB=A7=E7=BB=AD=E5=88=87=E5=88=86=E9=9A=90=E8=97=8F?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E6=8F=90=E7=A4=BA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在清理恢复对话框前检查分离与工作流忙碌状态 --- .../ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 584ec4c9f..5074d58b2 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1276,7 +1276,7 @@ export function useUiEditorSession( } async function continueSeparation() { - if (!resourceId) return; + if (!resourceId || isSeparating || isWorkflowBusy) return; setSeparationRecovery(null); try { const recovery = await invoke( From 4b8e26881679e46bd9343c2f233f4e3df67a91dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:18:30 +0800 Subject: [PATCH 190/248] =?UTF-8?q?=E7=BB=9F=E4=B8=80=E9=87=8D=E5=90=AF?= =?UTF-8?q?=E5=88=87=E5=88=86=E5=A4=B1=E8=B4=A5=E9=80=9A=E7=9F=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit restartSeparation 失败时复用工作流完成上报并清理完成提示 --- .../src/view/ui-editor/useUiEditorPage.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index 5074d58b2..eabaae116 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1333,8 +1333,11 @@ export function useUiEditorSession( }); await runSeparationWorkflow(); } catch (cause) { - setSeparationStatus( + reportWorkflowCompletion( + 'asset-separation', + 'failure', cause instanceof Error ? cause.message : String(cause), + setSeparationStatus, ); } } From 206b4830a194c3e8360419ecb0cb7cf9a6181fff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:19:10 +0800 Subject: [PATCH 191/248] =?UTF-8?q?=E5=A4=8D=E7=94=A8=E5=8E=9F=E5=9B=BE?= =?UTF-8?q?=E7=BC=96=E8=BE=91=E9=94=99=E8=AF=AF=E6=91=98=E8=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 解析失败分支只计算一次截断响应并同时用于审计与错误返回 --- .../crates/platform-image/src/vector_engine/raw_edit.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 1bb1d2e25..5d722ffa8 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -262,6 +262,7 @@ pub async fn create_vector_engine_raw_image_edit( Ok(payload) => payload, Err(error) => { let message = format!("{failure_context}：上游响应不是 JSON：{error}"); + let raw_excerpt = truncate_raw(body.as_str()); let audit = build_failure_audit( url.as_str(), failure_context, @@ -272,7 +273,7 @@ pub async fn create_vector_engine_raw_image_edit( false, message.as_str(), Some(error.to_string()), - Some(truncate_raw(body.as_str())), + Some(raw_excerpt.clone()), Some(started_at.elapsed().as_millis() as u64), prompt_chars, reference_image_count, @@ -281,7 +282,7 @@ pub async fn create_vector_engine_raw_image_edit( return Err(PlatformImageError::ResponseParse { provider: VECTOR_ENGINE_PROVIDER, message, - raw_excerpt: truncate_raw(body.as_str()), + raw_excerpt, audit: Some(audit), }); } From ef04a19d5aeec898ba36cc579d286571fa9ade36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:19:59 +0800 Subject: [PATCH 192/248] =?UTF-8?q?=E6=8A=BD=E5=8F=96=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E7=A1=AE=E8=AE=A4=E5=BC=B9=E7=AA=97=E5=85=AC=E5=85=B1=E7=BB=93?= =?UTF-8?q?=E6=9E=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 统一替换与移除操作的确认弹窗布局 保留各操作独立文案与按钮样式 --- .../Inspector/Components/ComponentPanel.tsx | 112 ++++++++++-------- 1 file changed, 63 insertions(+), 49 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index c8ba9afad..abf81c924 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -1,5 +1,5 @@ import { ChevronDown, ChevronRight, Plus, Trash2 } from 'lucide-react'; -import { useEffect, useRef, useState } from 'react'; +import { type ReactNode, useEffect, useRef, useState } from 'react'; import { ThemedModal } from '../../../../../components/modal/ThemedModal'; import type { Component } from '../../../../../features/ui-editor/types/Component'; @@ -139,64 +139,78 @@ export function ComponentPanel(props: ComponentPanelProps) {

)} {error &&

{error}

} - setReplaceConfirmationOpen(false)} - panelClassName="w-full max-w-sm rounded-2xl p-5" - > -

确认替换组件？

-

- 当前组件的配置将被新的默认组件覆盖。 -

-
- - -
- - + setRemoveConfirmationOpen(false)} - panelClassName="w-full max-w-sm rounded-2xl p-5" - > -

确认移除组件？

-

- 当前组件配置将被移除。 -

-
- - -
- + onConfirm={removeComponent} + /> ); } +function ConfirmationModal({ + open, + ariaLabel, + title, + description, + confirmLabel, + confirmClassName, + onClose, + onConfirm, +}: { + open: boolean; + ariaLabel: string; + title: string; + description: ReactNode; + confirmLabel: string; + confirmClassName: string; + onClose: () => void; + onConfirm: () => void; +}) { + return ( + +

{title}

+

{description}

+
+ + +
+ + ); +} + function componentKind(component: Component): string { if ('Image' in component) return '图片'; if ('Text' in component) return '文本'; From 379611d5525715f064e52b22382743c932fedc45 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:20:30 +0800 Subject: [PATCH 193/248] =?UTF-8?q?=E9=81=BF=E5=85=8D=E6=B8=B2=E6=9F=93?= =?UTF-8?q?=E9=98=B6=E6=AE=B5=E5=86=99=E5=85=A5=E7=94=BB=E5=B8=83=E5=B0=BA?= =?UTF-8?q?=E5=AF=B8=E5=BC=95=E7=94=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 在 ResizeObserver 尺寸更新回调中同步 canvasSizeRef --- .../view/ui-editor/components/preview/PreviewWorkspace.tsx | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx index f9bfaa56d..ba749ed99 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx @@ -56,7 +56,6 @@ export function PreviewWorkspace({ ); const [canvasSize, setCanvasSize] = useState({ width: 900, height: 640 }); const canvasSizeRef = useRef(canvasSize); - canvasSizeRef.current = canvasSize; const [spaceHeld, setSpaceHeld] = useState(false); const [renderMode, setRenderMode] = useState('editor-overlay'); @@ -202,10 +201,12 @@ export function PreviewWorkspace({ const element = viewportElementRef.current; if (!element) return; const updateSize = () => { - setCanvasSize({ + const nextSize = { width: element.clientWidth || 900, height: element.clientHeight || 640, - }); + }; + canvasSizeRef.current = nextSize; + setCanvasSize(nextSize); }; updateSize(); const observer = new ResizeObserver(updateSize); From b1751784ba251c9bdfd60e6aeb07050ac71f0950 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:26:36 +0800 Subject: [PATCH 194/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E5=88=87=E5=88=86?= =?UTF-8?q?=E7=8A=B6=E6=80=81=E5=86=99=E5=85=A5=E6=97=A5=E5=BF=97=E6=A0=BC?= =?UTF-8?q?=E5=BC=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 移除状态写入完成日志中多余的格式参数 --- .../src-tauri/src/ui_editor/commands/separation/persistence.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index c35737d86..637722fa1 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -97,8 +97,7 @@ fn write_separation_state_blocking(path: &Path, bytes: &[u8]) -> Result<(), Stri .unwrap_or(""), fs::metadata(path) .map(|metadata| metadata.len()) - .unwrap_or(0), - bytes.len() + .unwrap_or(0) ); Ok(()) } From bd127846f70d678a78ef7fc42fe0cdf8497034ac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:36:30 +0800 Subject: [PATCH 195/248] =?UTF-8?q?=E5=85=81=E8=AE=B8=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=B5=81=E7=A8=8B=E5=9C=A8=E5=BF=99=E7=A2=8C=E6=80=81=E4=BF=9D?= =?UTF-8?q?=E5=AD=98=E7=BB=93=E6=9E=9C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 保留 isSeparating 属于 isAiRunning 的全局语义 为内部分离保存增加受控 allowDuringSeparation 选项 --- .../src/view/ui-editor/useUiEditorPage.ts | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index eabaae116..b7637d6c1 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1196,7 +1196,7 @@ export function useUiEditorSession( if (separationResult === null) throw new Error('自动切分素材没有返回结果'); const completedResult = separationResult as SeparationDTO; - if (!(await save())) { + if (!(await save({ allowDuringSeparation: true }))) { throw new Error( '自动切分素材结果已写入编辑器，但 State 保存失败；sidecar 已保留，可继续恢复。', ); @@ -1342,13 +1342,19 @@ export function useUiEditorSession( } } - async function save() { + async function save(options?: { allowDuringSeparation?: boolean }) { + const allowDuringSeparation = + options?.allowDuringSeparation === true && + isSeparating && + !isSuggesting && + !isRecognizing && + !isMerging; if ( !resourceId || isSaving || isGenerating || isLoading || - isAiRunning || + (isAiRunning && !allowDuringSeparation) || loadError || persistedRevision === null || editor.isLocked From aaa9d3fb6f02cd3124e83bd9bb8a18716113c731 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:44:17 +0800 Subject: [PATCH 196/248] =?UTF-8?q?=E8=A1=A5=E9=BD=90=20Raw=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E6=81=A2=E5=A4=8D=E5=AE=A1=E8=AE=A1=E8=AF=AD=E4=B9=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 保留 recovered_failure_audits 公共字段并注明当前同步无回退不变量 让 raw handler 记录未来恢复审计并写入成功摘要计数 同步 Raw GPT Image 2 技术方案文档 --- ...术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- .../api-server/src/openai_image_generation.rs | 2 +- server-rs/crates/api-server/src/raw_image.rs | 13 ++++++++++--- .../platform-image/src/vector_engine/raw_edit.rs | 7 +++++++ 4 files changed, 19 insertions(+), 5 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 393eb6fb4..c6a4cd3e1 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -85,7 +85,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、multipart 字段解析、PNG 预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 原生只返回 `b64_json`，因此不发送 `response_format` 参数，也不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 +provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 原生只返回 `b64_json`，因此不发送 `response_format` 参数，也不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。`RawImageEditResult.recovered_failure_audits` 保留共享结果契约，但 raw-edit 当前只有一次同步 provider 调用、没有 fallback，因此成功结果该列表恒为空；若后续增加 fallback，api-server 必须先落库其中的每条失败审计，再写成功运行摘要。 ## 代码拆分 diff --git a/server-rs/crates/api-server/src/openai_image_generation.rs b/server-rs/crates/api-server/src/openai_image_generation.rs index f0fa7591c..e1debf606 100644 --- a/server-rs/crates/api-server/src/openai_image_generation.rs +++ b/server-rs/crates/api-server/src/openai_image_generation.rs @@ -492,7 +492,7 @@ pub(crate) async fn record_openai_image_failure_if_configured( record_openai_image_failure_audit_if_configured(settings, audit).await; } -async fn record_openai_image_failure_audit_if_configured( +pub(crate) async fn record_openai_image_failure_audit_if_configured( settings: &OpenAiImageSettings, audit: &platform_image::PlatformImageFailureAudit, ) { diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 1af3d4c91..873b0398c 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -21,8 +21,8 @@ use crate::{ auth::AuthenticatedAccessToken, http_error::AppError, openai_image_generation::{ - map_platform_image_error, record_openai_image_failure_if_configured, - require_openai_image_settings, + map_platform_image_error, record_openai_image_failure_audit_if_configured, + record_openai_image_failure_if_configured, require_openai_image_settings, }, request_context::RequestContext, state::AppState, @@ -125,6 +125,10 @@ pub(crate) async fn edit_raw_image( return Err(map_platform_image_error(error)); } }; + for audit in &generated.recovered_failure_audits { + record_openai_image_failure_audit_if_configured(&audit_settings, audit).await; + } + let recovered_failure_count = generated.recovered_failure_audits.len(); let data: Vec = generated .b64_images .into_iter() @@ -141,7 +145,10 @@ pub(crate) async fn edit_raw_image( true, None, Some("raw-image-edit".to_string()), - Some(json!({ "imageCount": data.len() })), + Some(json!({ + "imageCount": data.len(), + "recoveredFailureCount": recovered_failure_count, + })), ) .await; } diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 5d722ffa8..ea944a190 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -34,8 +34,15 @@ pub struct RawImageEditImage { } #[derive(Clone, Debug)] +/// Successful response from the synchronous raw GPT Image 2 edit call. pub struct RawImageEditResult { pub b64_images: Vec, + /// Failure audits recovered by an internal fallback attempt. + /// + /// Raw-edit currently performs one provider call and has no fallback, so + /// this is empty on every successful result. The field remains part of the + /// shared result contract so adding a fallback later cannot silently drop + /// its audit trail from callers that already handle recovered failures. pub recovered_failure_audits: Vec, } From e4d70fbe1b7260377788e9770449813e8b7d597e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 13:50:25 +0800 Subject: [PATCH 197/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=20Raw=20=E7=BC=96?= =?UTF-8?q?=E8=BE=91=E6=97=A0=E6=95=88=E6=81=A2=E5=A4=8D=E5=AE=A1=E8=AE=A1?= =?UTF-8?q?=E5=AD=97=E6=AE=B5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RawImageEditResult 仅保留实际透传的 b64_images 删除 raw-edit 不会产生的恢复审计转发与文档描述 补充 PreparedRawImageEdit 的 Debug 派生以通过现有测试 --- ...术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- .../api-server/src/openai_image_generation.rs | 2 +- server-rs/crates/api-server/src/raw_image.rs | 14 ++++---------- .../platform-image/src/vector_engine/raw_edit.rs | 13 +------------ 4 files changed, 7 insertions(+), 24 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index c6a4cd3e1..393eb6fb4 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -85,7 +85,7 @@ raw 操作使用独立的 operation / ledger 命名空间，例如 `raw-image-ed `platform-image` 保留 VectorEngine 协议细节。raw handler 只负责：认证、multipart 字段解析、PNG 预检查、计费编排和响应映射。provider 请求仍由 `platform-image` 统一构造，并携带 `model`、`n`、`quality`、`background`、`output_format`、尺寸及图片参考字节。 -provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 原生只返回 `b64_json`，因此不发送 `response_format` 参数，也不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。`RawImageEditResult.recovered_failure_audits` 保留共享结果契约，但 raw-edit 当前只有一次同步 provider 调用、没有 fallback，因此成功结果该列表恒为空；若后续增加 fallback，api-server 必须先落库其中的每条失败审计，再写成功运行摘要。 +provider 响应只提取并透传 `data[].b64_json` 字符串，不在服务端解码图片 base64，也不读取或回传 provider 的 `output_format`（该字段只是请求参数回显）。GPT-Image-2 原生只返回 `b64_json`，因此不发送 `response_format` 参数，也不实现 URL 响应下载或兼容分支。发送、响应读取、上游状态、响应解析和缺图失败必须生成 `PlatformImageFailureAudit`，由 api-server 写入现有外部 API 失败审计链；成功结果同时写入统一的 `external_generation_run` 追踪事件。raw handler 只将上游 `b64_json` 原样写入 `data[].b64_json`。 ## 代码拆分 diff --git a/server-rs/crates/api-server/src/openai_image_generation.rs b/server-rs/crates/api-server/src/openai_image_generation.rs index e1debf606..f0fa7591c 100644 --- a/server-rs/crates/api-server/src/openai_image_generation.rs +++ b/server-rs/crates/api-server/src/openai_image_generation.rs @@ -492,7 +492,7 @@ pub(crate) async fn record_openai_image_failure_if_configured( record_openai_image_failure_audit_if_configured(settings, audit).await; } -pub(crate) async fn record_openai_image_failure_audit_if_configured( +async fn record_openai_image_failure_audit_if_configured( settings: &OpenAiImageSettings, audit: &platform_image::PlatformImageFailureAudit, ) { diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 873b0398c..a39aca114 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -21,8 +21,8 @@ use crate::{ auth::AuthenticatedAccessToken, http_error::AppError, openai_image_generation::{ - map_platform_image_error, record_openai_image_failure_audit_if_configured, - record_openai_image_failure_if_configured, require_openai_image_settings, + map_platform_image_error, record_openai_image_failure_if_configured, + require_openai_image_settings, }, request_context::RequestContext, state::AppState, @@ -125,10 +125,6 @@ pub(crate) async fn edit_raw_image( return Err(map_platform_image_error(error)); } }; - for audit in &generated.recovered_failure_audits { - record_openai_image_failure_audit_if_configured(&audit_settings, audit).await; - } - let recovered_failure_count = generated.recovered_failure_audits.len(); let data: Vec = generated .b64_images .into_iter() @@ -145,10 +141,7 @@ pub(crate) async fn edit_raw_image( true, None, Some("raw-image-edit".to_string()), - Some(json!({ - "imageCount": data.len(), - "recoveredFailureCount": recovered_failure_count, - })), + Some(json!({ "imageCount": data.len() })), ) .await; } @@ -168,6 +161,7 @@ pub(crate) async fn edit_raw_image( Ok(Json(result)) } +#[derive(Debug)] struct PreparedRawImageEdit { image: RawImageEditImage, prompt: String, diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index ea944a190..6061f885b 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -34,16 +34,8 @@ pub struct RawImageEditImage { } #[derive(Clone, Debug)] -/// Successful response from the synchronous raw GPT Image 2 edit call. pub struct RawImageEditResult { pub b64_images: Vec, - /// Failure audits recovered by an internal fallback attempt. - /// - /// Raw-edit currently performs one provider call and has no fallback, so - /// this is empty on every successful result. The field remains part of the - /// shared result contract so adding a fallback later cannot silently drop - /// its audit trail from callers that already handle recovered failures. - pub recovered_failure_audits: Vec, } const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; @@ -319,10 +311,7 @@ pub async fn create_vector_engine_raw_image_edit( audit: Some(audit), }); } - Ok(RawImageEditResult { - b64_images, - recovered_failure_audits: Vec::new(), - }) + Ok(RawImageEditResult { b64_images }) } fn collect_b64_images(data: Vec) -> Vec { From 329de5fce9c33bf7cb273a67be2f8af68491d7d8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:46:08 +0800 Subject: [PATCH 198/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E9=9D=A2=E6=9D=BF=E5=B1=95=E5=BC=80=E7=8A=B6=E6=80=81=E9=87=8D?= =?UTF-8?q?=E7=BD=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 仅在组件存在性发生变化时同步展开状态，避免深拷贝更新意外重新展开。 --- .../components/Inspector/Components/ComponentPanel.tsx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index abf81c924..8ce36ca7b 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -25,7 +25,7 @@ export function ComponentPanel(props: ComponentPanelProps) { useEffect(() => { const previous = previousComponentRef.current; previousComponentRef.current = component; - if (component !== previous) { + if (Boolean(component) !== Boolean(previous)) { setExpanded(Boolean(component)); } }, [component]); From 9788087271fa476e532a0f97d20d02d6118cf0f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:47:01 +0800 Subject: [PATCH 199/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E7=BB=84=E4=BB=B6?= =?UTF-8?q?=E6=93=8D=E4=BD=9C=E5=A4=B1=E8=B4=A5=E6=97=B6=E7=A1=AE=E8=AE=A4?= =?UTF-8?q?=E5=BC=B9=E7=AA=97=E5=8D=A1=E4=BD=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 当组件更新返回 undefined 时关闭替换或移除确认弹窗，避免操作无反馈地停留。 --- .../components/Inspector/Components/ComponentPanel.tsx | 3 +++ 1 file changed, 3 insertions(+) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx index 8ce36ca7b..eee3d1d51 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/Inspector/Components/ComponentPanel.tsx @@ -57,12 +57,15 @@ export function ComponentPanel(props: ComponentPanelProps) { if (result?.ok) { setExpanded(true); setReplaceConfirmationOpen(false); + } else if (result === undefined) { + setReplaceConfirmationOpen(false); } } function removeComponent() { const result = setComponent(null); if (result?.ok) setRemoveConfirmationOpen(false); + else if (result === undefined) setRemoveConfirmationOpen(false); } return ( From 398edd6e8609b14054f358be111374cb5dc01919 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:48:00 +0800 Subject: [PATCH 200/248] =?UTF-8?q?=E5=B0=86=E5=88=86=E7=A6=BB=E7=8A=B6?= =?UTF-8?q?=E6=80=81=E5=BA=8F=E5=88=97=E5=8C=96=E7=A7=BB=E5=85=A5=E9=98=BB?= =?UTF-8?q?=E5=A1=9E=E4=BB=BB=E5=8A=A1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 避免在异步执行器线程同步格式化大型 separation state。 --- .../src/ui_editor/commands/separation/persistence.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 637722fa1..9275b5523 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -47,9 +47,12 @@ pub async fn write_separation_state(path: PathBuf, state: &SeparationState) -> R app_log!("ui_separation.error stage=state_write reason=schema_mismatch"); return Err("不支持的 separation state schema".to_string()); } - let state = serde_json::to_vec_pretty(state) - .map_err(|error| format!("序列化 separation state 失败：{error}"))?; - tokio::task::spawn_blocking(move || write_separation_state_blocking(&path, &state)) + let owned = state.clone(); + tokio::task::spawn_blocking(move || { + let bytes = serde_json::to_vec_pretty(&owned) + .map_err(|error| format!("序列化 separation state 失败：{error}"))?; + write_separation_state_blocking(&path, &bytes) + }) .await .map_err(|error| format!("写入 separation state 任务失败：{error}"))? } From b52a33cc283170ce9bc27ff81657b7efe63341f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:48:42 +0800 Subject: [PATCH 201/248] =?UTF-8?q?=E6=98=8E=E7=A1=AE=E6=8F=90=E7=A4=BA?= =?UTF-8?q?=E8=AF=8D=E6=B8=85=E6=B4=97=E7=9A=84=E5=AD=97=E7=AC=A6=E6=98=A0?= =?UTF-8?q?=E5=B0=84=E6=84=8F=E5=9B=BE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 使用 map 替代始终返回 Some 的 filter_map，保持清洗行为不变。 --- .../src/ui_editor/commands/separation/model/note.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs index 1c3065627..6ffb109a1 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs @@ -4,13 +4,13 @@ use ts_rs::TS; pub(crate) fn sanitize_prompt_text(value: &str) -> String { value .chars() - .filter_map(|character| { + .map(|character| { if character == '`' { - Some('\'') + '\'' } else if character.is_control() { - Some(' ') + ' ' } else { - Some(character) + character } }) .take(super::MAX_REWORK_NOTE_CHARS) From b0e9741e816437a906ca37f660967c694878a545 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:49:30 +0800 Subject: [PATCH 202/248] =?UTF-8?q?=E5=9C=A8=E9=A2=84=E8=A7=88=E5=8C=BA?= =?UTF-8?q?=E5=B0=BA=E5=AF=B8=E5=8F=98=E5=8C=96=E6=97=B6=E9=87=8D=E6=96=B0?= =?UTF-8?q?=E9=80=82=E9=85=8D=E7=94=BB=E5=B8=83?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将画布尺寸纳入适配视口 effect 依赖，恢复面板调整后的自动 fit。 --- .../ui-editor/components/preview/PreviewWorkspace.tsx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx index ba749ed99..8fbc37dbb 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx @@ -158,7 +158,13 @@ export function PreviewWorkspace({ }, }), ); - }, [activeImageId, logicalSize, setViewport]); + }, [ + activeImageId, + canvasSize.height, + canvasSize.width, + logicalSize, + setViewport, + ]); const scaleViewportFromCenter = useCallback( (nextScale: number) => { From 2bfc8708544346813f413dec569edb02876b48c7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:54:05 +0800 Subject: [PATCH 203/248] =?UTF-8?q?=E5=A4=8D=E7=94=A8=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=B5=81=E7=A8=8B=E5=B7=B2=E9=80=89=E6=89=B9=E6=AC=A1=E8=8A=82?= =?UTF-8?q?=E7=82=B9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 让批次校验和应用使用同一份已选节点，避免重复选择造成目标漂移。 --- .../src/ui_editor/commands/separation/mod.rs | 11 ++++++++++- .../ui_editor/commands/separation/workflow/mod.rs | 1 + .../commands/separation/workflow/patch.rs | 15 ++++----------- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs index d6400e05d..3bb71b767 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/mod.rs @@ -259,9 +259,14 @@ mod tests { let paths = HashMap::new(); for note in ["第一次意见", "第二次意见", "最后一次意见"] { + let batch_nodes = next_image_batch(&separation, &separation.trees[0]) + .into_iter() + .cloned() + .collect::>(); apply_batch_patch( &mut separation, 0, + &batch_nodes, &[BindingDecision::NeedRework { to_node: id.clone(), advice: note.to_string(), @@ -343,7 +348,11 @@ mod tests { }, }]; let paths = HashMap::from([(id.clone(), "ui/.sidecar/cut.png".to_string())]); - apply_batch_patch(&mut state, 0, &decisions, &paths, (1, 1)).unwrap(); + let batch_nodes = next_image_batch(&state, &state.trees[0]) + .into_iter() + .cloned() + .collect::>(); + apply_batch_patch(&mut state, 0, &batch_nodes, &decisions, &paths, (1, 1)).unwrap(); assert_eq!(state.bound[0].node_id, id); assert_eq!(state.trees[0].root.children.len(), 1); } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 960cc7ee9..181964feb 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -243,6 +243,7 @@ pub(crate) async fn separate_ui_impl( patch::apply_batch_patch( &mut separation, tree_index, + &batch_nodes, &binding.decisions, &cut_paths, processed_dimensions, diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs index 42985ac4f..39618548b 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/patch.rs @@ -1,4 +1,3 @@ -use super::batch::next_image_batch; use crate::ui_editor::commands::separation::{ validate_binding_response, BindingDecision, BindingResp, BoundNode, ProblematicNode, SeparationNode, SeparationState, MAX_REWORK_COUNT, @@ -9,6 +8,7 @@ use std::collections::HashMap; pub fn apply_batch_patch( state: &mut SeparationState, tree_index: usize, + batch_nodes: &[SeparationNode], decisions: &[BindingDecision], cut_paths: &HashMap, processed_dimensions: (u32, u32), @@ -19,16 +19,9 @@ pub fn apply_batch_patch( decisions.len(), cut_paths.len() ); - let batch_nodes = { - let tree = state - .trees - .get(tree_index) - .ok_or_else(|| "separation tree 索引无效".to_string())?; - next_image_batch(state, tree) - .into_iter() - .cloned() - .collect::>() - }; + if state.trees.get(tree_index).is_none() { + return Err("separation tree 索引无效".to_string()); + } let batch = batch_nodes.iter().collect::>(); validate_binding_response( &BindingResp { From 3bcabe36fa734ccdd98405e2e75c3091301e62eb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:54:49 +0800 Subject: [PATCH 204/248] =?UTF-8?q?=E6=B8=85=E7=90=86=E5=AD=97=E4=BD=93?= =?UTF-8?q?=E6=A0=A1=E9=AA=8C=E4=B8=AD=E7=9A=84=E5=86=97=E4=BD=99=E7=A9=BA?= =?UTF-8?q?=E5=80=BC=E5=88=A4=E6=96=AD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按 FontSource 的实际联合类型保留运行时字体资源校验。 --- .../src/features/ui-editor/requisites.ts | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts b/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts index 14a77a269..759792923 100644 --- a/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts +++ b/apps/ai-game-creator-shell/src/features/ui-editor/requisites.ts @@ -194,12 +194,7 @@ export function validateAssetSeparationResult( } if ('Text' in component) { const font = component.Text.font; - if ( - font !== null && - font !== undefined && - typeof font !== 'string' && - !(font.Bound in state.font_assets) - ) { + if (typeof font !== 'string' && !(font.Bound in state.font_assets)) { issues.push({ code: 'missing-font', message: '文本组件引用的字体不存在', From f0559617145a0fba45d347f4de8e5e834f67c4d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:55:35 +0800 Subject: [PATCH 205/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=BA=90=E5=9B=BE=20MIME=20=E8=AF=86=E5=88=AB=E5=9B=9E?= =?UTF-8?q?=E9=80=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 缺失或未知 MIME 时强制走 PNG 规范化，避免非 PNG 数据被误标为 image/png。 --- .../src/ui_editor/commands/separation/workflow/extract.rs | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs index b359fb32e..f55db59e3 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/extract.rs @@ -52,9 +52,8 @@ async fn raw_extract_inner( .ok_or_else(|| "界面图 data URL 无效".to_string())?; let mime = mime .strip_prefix("data:") - .and_then(|value| value.strip_suffix(";base64")) - .unwrap_or("image/png"); - let is_png = mime.eq_ignore_ascii_case("image/png"); + .and_then(|value| value.strip_suffix(";base64")); + let is_png = mime.is_some_and(|value| value.eq_ignore_ascii_case("image/png")); let image_bytes = base64::engine::general_purpose::STANDARD .decode(data.trim()) .map_err(|error| format!("解码源图失败：{error}"))?; From 39a64a864adde6fe54cdfa2523c2dd26cbbad31c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:56:54 +0800 Subject: [PATCH 206/248] =?UTF-8?q?=E5=A3=B0=E6=98=8E=E5=9B=BE=E7=89=87?= =?UTF-8?q?=E5=B9=B3=E5=8F=B0=E4=BD=BF=E7=94=A8=20reqwest=20=E6=B5=81?= =?UTF-8?q?=E5=BC=8F=E7=89=B9=E6=80=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 Part::stream 补齐 platform-image 的直接依赖特性，支持独立构建。 --- server-rs/crates/platform-image/Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server-rs/crates/platform-image/Cargo.toml b/server-rs/crates/platform-image/Cargo.toml index c4acbcd2a..166d6e011 100644 --- a/server-rs/crates/platform-image/Cargo.toml +++ b/server-rs/crates/platform-image/Cargo.toml @@ -9,7 +9,7 @@ base64 = { workspace = true } bytes = { workspace = true } curl = { workspace = true } image = { workspace = true, features = ["jpeg", "png", "webp"] } -reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls"] } +reqwest = { workspace = true, features = ["json", "multipart", "rustls-tls", "stream"] } serde = { workspace = true } serde_json = { workspace = true } tokio = { workspace = true, features = ["io-util", "macros", "net", "time"] } From 664bdd6877f2770a547cdb7170204525d05453a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 14:57:19 +0800 Subject: [PATCH 207/248] =?UTF-8?q?=E9=81=BF=E5=85=8D=E4=BC=AA=E9=80=A0?= =?UTF-8?q?=E5=8E=9F=E5=A7=8B=E5=9B=BE=E7=89=87=E4=BE=9B=E5=BA=94=E5=95=86?= =?UTF-8?q?=E8=AF=B7=E6=B1=82=E7=BC=96=E5=8F=B7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 原始图片响应未提供供应商 request id 时不再写入操作名作为关联编号。 --- server-rs/crates/api-server/src/raw_image.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index a39aca114..bead3a52e 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -140,7 +140,7 @@ pub(crate) async fn edit_raw_image( started_at_micros, true, None, - Some("raw-image-edit".to_string()), + None, Some(json!({ "imageCount": data.len() })), ) .await; From e9d01800542bc3aac1c90a0de1f0179a9eb7fe81 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 15:00:55 +0800 Subject: [PATCH 208/248] =?UTF-8?q?=E9=9B=86=E4=B8=AD=E7=AE=A1=E7=90=86=20?= =?UTF-8?q?GPT=20Image=202=20=E5=B0=BA=E5=AF=B8=E9=99=90=E5=88=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将请求尺寸压缩与原始编辑校验共用同一组像素、边长和对齐常量。 --- .../src/vector_engine/constants.rs | 4 +++ .../src/vector_engine/raw_edit.rs | 10 +++---- .../src/vector_engine/request.rs | 27 +++++++++---------- 3 files changed, 21 insertions(+), 20 deletions(-) diff --git a/server-rs/crates/platform-image/src/vector_engine/constants.rs b/server-rs/crates/platform-image/src/vector_engine/constants.rs index 6480fba73..2ae789387 100644 --- a/server-rs/crates/platform-image/src/vector_engine/constants.rs +++ b/server-rs/crates/platform-image/src/vector_engine/constants.rs @@ -5,3 +5,7 @@ pub const VECTOR_ENGINE_GPT_IMAGE_2_MODEL: &str = GPT_IMAGE_2_MODEL; pub const VECTOR_ENGINE_PROVIDER: &str = "vector-engine"; pub const VECTOR_ENGINE_IMAGE_EDIT_MAX_REFERENCE_IMAGES: usize = 5; pub const VECTOR_ENGINE_NANOBANANA_MAX_REFERENCE_IMAGES: usize = 14; +pub const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; +pub const GPT_IMAGE_2_MAX_PIXELS: u64 = 8_294_400; +pub const GPT_IMAGE_2_MAX_EDGE: u32 = 3_840; +pub const GPT_IMAGE_2_DIMENSION_ALIGNMENT: u32 = 16; diff --git a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs index 6061f885b..6731432d9 100644 --- a/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs +++ b/server-rs/crates/platform-image/src/vector_engine/raw_edit.rs @@ -7,7 +7,10 @@ use serde::Deserialize; use super::{ audit::build_failure_audit, budget::{effective_request_timeout_ms, request_budget_exhausted_error}, - constants::{GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER}, + constants::{ + GPT_IMAGE_2_DIMENSION_ALIGNMENT, GPT_IMAGE_2_MAX_EDGE, GPT_IMAGE_2_MAX_PIXELS, + GPT_IMAGE_2_MIN_PIXELS, GPT_IMAGE_2_MODEL, VECTOR_ENGINE_PROVIDER, + }, error::PlatformImageError, request::vector_engine_images_edit_url, types::VectorEngineImageSettings, @@ -38,11 +41,6 @@ pub struct RawImageEditResult { pub b64_images: Vec, } -const GPT_IMAGE_2_MIN_PIXELS: u64 = 655_360; -const GPT_IMAGE_2_MAX_PIXELS: u64 = 8_294_400; -const GPT_IMAGE_2_MAX_EDGE: u32 = 3_840; -const GPT_IMAGE_2_DIMENSION_ALIGNMENT: u32 = 16; - #[derive(Debug, Deserialize)] struct RawImageEditResponsePayload { data: Vec, diff --git a/server-rs/crates/platform-image/src/vector_engine/request.rs b/server-rs/crates/platform-image/src/vector_engine/request.rs index af232dbc8..0b008d2a2 100644 --- a/server-rs/crates/platform-image/src/vector_engine/request.rs +++ b/server-rs/crates/platform-image/src/vector_engine/request.rs @@ -1,7 +1,10 @@ use serde_json::{Map, Value, json}; use super::{ - constants::{GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_MODEL}, + constants::{ + GPT_IMAGE_2_C_MODEL, GPT_IMAGE_2_DIMENSION_ALIGNMENT, GPT_IMAGE_2_MAX_EDGE, + GPT_IMAGE_2_MAX_PIXELS, GPT_IMAGE_2_MIN_PIXELS, GPT_IMAGE_2_MODEL, + }, types::{ReferenceImage, VectorEngineImageSettings}, }; @@ -129,10 +132,6 @@ fn normalize_explicit_pixel_size(value: &str) -> String { } fn clamp_gpt_image_2_pixel_size(size: &str) -> String { - const MIN_PIXELS: u64 = 655_360; - const MAX_PIXELS: u64 = 8_294_400; - const MAX_EDGE: u32 = 3_840; - const DIMENSION_ALIGNMENT: u32 = 16; const MAX_ASPECT_RATIO: f64 = 3.0; // 中文注释：这里是 VectorEngine 的共享发送边界，只处理 gpt-image-2 的显式像素尺寸。 @@ -154,31 +153,31 @@ fn clamp_gpt_image_2_pixel_size(size: &str) -> String { let pixels = width * height; // 中文注释：等比缩小，优先保留已收紧后的画面比例，同时满足最大边和最大总像素。 - let scale = (MAX_EDGE as f64 / width.max(height)) - .min((MAX_PIXELS as f64 / pixels).sqrt()) + let scale = (GPT_IMAGE_2_MAX_EDGE as f64 / width.max(height)) + .min((GPT_IMAGE_2_MAX_PIXELS as f64 / pixels).sqrt()) .min(1.0); width *= scale; height *= scale; let pixels = width * height; // 中文注释：小于最小总像素时等比放大；此前已处理比例，放大不会重新突破 3:1。 - if pixels < MIN_PIXELS as f64 { - let scale = (MIN_PIXELS as f64 / pixels).sqrt(); + if pixels < GPT_IMAGE_2_MIN_PIXELS as f64 { + let scale = (GPT_IMAGE_2_MIN_PIXELS as f64 / pixels).sqrt(); width *= scale; height *= scale; } // 中文注释：provider 要求两边均为 16px 倍数，向上取整避免对齐后落到最小像素以下。 - let width = align_dimension_up(width, DIMENSION_ALIGNMENT); - let height = align_dimension_up(height, DIMENSION_ALIGNMENT); + let width = align_dimension_up(width, GPT_IMAGE_2_DIMENSION_ALIGNMENT); + let height = align_dimension_up(height, GPT_IMAGE_2_DIMENSION_ALIGNMENT); // TODO unlikely but can improve // 中文注释：对齐可能触碰最大边或最大像素，因此发送前重新核验全部约束。 if is_valid_gpt_image_2_size( width, height, - MIN_PIXELS, - MAX_PIXELS, - MAX_EDGE, + GPT_IMAGE_2_MIN_PIXELS, + GPT_IMAGE_2_MAX_PIXELS, + GPT_IMAGE_2_MAX_EDGE, MAX_ASPECT_RATIO, ) { return format!("{width}x{height}"); From 9bbfc56cd7b1a61b5d707c8117b3f957453aa14f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 15:02:56 +0800 Subject: [PATCH 209/248] =?UTF-8?q?=E4=BF=AE=E6=AD=A3=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=9C=9F=E9=97=B4=E4=BF=9D=E5=AD=98=E9=80=89=E9=A1=B9=E7=9A=84?= =?UTF-8?q?=E9=97=AD=E5=8C=85=E5=88=A4=E6=96=AD?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 由调用方明确传入的 allowDuringSeparation 直接生效，避免依赖点击时捕获的过期状态。 --- .../src/view/ui-editor/useUiEditorPage.ts | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts index b7637d6c1..56ad362ab 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts +++ b/apps/ai-game-creator-shell/src/view/ui-editor/useUiEditorPage.ts @@ -1343,12 +1343,7 @@ export function useUiEditorSession( } async function save(options?: { allowDuringSeparation?: boolean }) { - const allowDuringSeparation = - options?.allowDuringSeparation === true && - isSeparating && - !isSuggesting && - !isRecognizing && - !isMerging; + const allowDuringSeparation = options?.allowDuringSeparation === true; if ( !resourceId || isSaving || From 8a9d37d5ad9e03abb571c9f632621f2d74af79eb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 15:19:24 +0800 Subject: [PATCH 210/248] =?UTF-8?q?=E7=A7=BB=E9=99=A4=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E6=8F=90=E7=A4=BA=E8=AF=8D=E7=9A=84=E7=A0=B4=E5=9D=8F=E6=80=A7?= =?UTF-8?q?=E6=B8=85=E6=B4=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 保留描述、返工意见和 NodeId 原文，删除重复截断与字符改写。 返工意见仍由 validate_binding_response 统一执行 MAX_REWORK_NOTE_CHARS 校验。 --- .../commands/separation/model/node.rs | 2 +- .../commands/separation/model/note.rs | 20 ++----------------- .../commands/separation/prompt/extract.rs | 13 +++--------- 3 files changed, 6 insertions(+), 29 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs index f06c56fd1..d925b7e36 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/node.rs @@ -28,7 +28,7 @@ impl SeparationNode { pub fn as_prompt(&self) -> String { format!( "node_id={} note: {}", - super::sanitize_prompt_text(self.id.as_str()), + self.id.as_str(), self.note.as_prompt() ) } diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs index 6ffb109a1..15c500b59 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/model/note.rs @@ -1,22 +1,6 @@ use serde::{Deserialize, Serialize}; use ts_rs::TS; -pub(crate) fn sanitize_prompt_text(value: &str) -> String { - value - .chars() - .map(|character| { - if character == '`' { - '\'' - } else if character.is_control() { - ' ' - } else { - character - } - }) - .take(super::MAX_REWORK_NOTE_CHARS) - .collect() -} - #[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)] #[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))] pub struct SeparationNote { @@ -26,12 +10,12 @@ pub struct SeparationNote { impl SeparationNote { pub fn as_prompt(&self) -> String { - let mut prompt = format!("desc: {}", sanitize_prompt_text(&self.description)); + let mut prompt = format!("desc: {}", self.description); if !self.rework_notes.is_empty() { prompt.push_str("\nprevious rework notes:"); for note in &self.rework_notes { prompt.push_str("\n- "); - prompt.push_str(&sanitize_prompt_text(note)); + prompt.push_str(note); } } prompt diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs index aceb5c4df..c6cf4abfa 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/prompt/extract.rs @@ -1,6 +1,4 @@ -use crate::ui_editor::commands::separation::model::{ - sanitize_prompt_text, SeparationState, SeparationTree, -}; +use crate::ui_editor::commands::separation::model::{SeparationState, SeparationTree}; use crate::ui_editor::commands::separation::workflow::batch::terminal_node_ids; use crate::ui_editor::commands::separation::{SeparationNode, SeparationNodeKind}; use crate::ui_editor::utils::NodeId; @@ -115,13 +113,8 @@ fn project_node( width: node.width_px, height: node.height_px, }, - description: sanitize_prompt_text(&node.note.description), - rework_notes: node - .note - .rework_notes - .iter() - .map(|note| sanitize_prompt_text(note)) - .collect(), + description: node.note.description.clone(), + rework_notes: node.note.rework_notes.iter().cloned().collect(), children, } } From 6235fa3298e95dfe7877bc853d3e4413b5359f1c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 15:40:12 +0800 Subject: [PATCH 211/248] =?UTF-8?q?=E4=BF=AE=E5=A4=8D=E5=88=86=E7=A6=BB?= =?UTF-8?q?=E7=8A=B6=E6=80=81=E8=B7=A8=E5=B9=B3=E5=8F=B0=E5=8E=9F=E5=AD=90?= =?UTF-8?q?=E6=9B=BF=E6=8D=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 使用唯一临时文件并同步写入，避免 checkpoint 临时文件互相覆盖。 Unix 使用 rename，Windows 使用 MoveFileExW 原子覆盖已有 state.json。 拒绝符号链接和非普通文件目标，并增加覆盖已有目标回归测试。 --- .../commands/separation/persistence.rs | 99 +++++++++++++++++-- 1 file changed, 91 insertions(+), 8 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs index 9275b5523..c27336066 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/persistence.rs @@ -1,7 +1,9 @@ use super::model::*; use crate::ui_editor::commands::separation::*; -use std::fs; +use std::fs::{self, OpenOptions}; +use std::io::Write; use std::path::{Path, PathBuf}; +use uuid::Uuid; const SEPARATION_STATE_MAX_BYTES: usize = 8 * 1024 * 1024; pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result { @@ -53,8 +55,8 @@ pub async fn write_separation_state(path: PathBuf, state: &SeparationState) -> R .map_err(|error| format!("序列化 separation state 失败：{error}"))?; write_separation_state_blocking(&path, &bytes) }) - .await - .map_err(|error| format!("写入 separation state 任务失败：{error}"))? + .await + .map_err(|error| format!("写入 separation state 任务失败：{error}"))? } fn write_separation_state_blocking(path: &Path, bytes: &[u8]) -> Result<(), String> { @@ -84,15 +86,40 @@ fn write_separation_state_blocking(path: &Path, bytes: &[u8]) -> Result<(), Stri app_log!("ui_separation.error stage=state_write reason=create_parent error={error}"); format!("创建 separation sidecar 失败：{error}") })?; - let temporary = path.with_extension("json.tmp"); - fs::write(&temporary, bytes).map_err(|error| { + match fs::symlink_metadata(path) { + Ok(metadata) if metadata.file_type().is_symlink() || !metadata.is_file() => { + app_log!("ui_separation.error stage=state_write reason=unsafe_target"); + return Err("separation state 目标必须是普通文件".to_string()); + } + Ok(_) => {} + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => { + app_log!("ui_separation.error stage=state_write reason=target_metadata error={error}"); + return Err(format!("检查 separation state 目标失败：{error}")); + } + } + let file_name = path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or("state.json"); + let temporary = parent.join(format!(".{file_name}.tmp.{}", Uuid::new_v4())); + let mut temporary_file = OpenOptions::new(); + temporary_file.write(true).create_new(true); + let mut file = temporary_file.open(&temporary).map_err(|error| { app_log!("ui_separation.error stage=state_write reason=write_temp error={error}"); format!("写入 separation state 失败：{error}") })?; - fs::rename(&temporary, path).map_err(|error| { + if let Err(error) = file.write_all(bytes).and_then(|_| file.sync_data()) { + let _ = fs::remove_file(&temporary); + app_log!("ui_separation.error stage=state_write reason=write_temp error={error}"); + return Err(format!("写入 separation state 失败：{error}")); + } + drop(file); + if let Err(error) = replace_separation_state_atomically(&temporary, path) { + let _ = fs::remove_file(&temporary); app_log!("ui_separation.error stage=state_write reason=install error={error}"); - format!("安装 separation state 失败：{error}") - })?; + return Err(format!("安装 separation state 失败：{error}")); + } app_log!( "ui_separation.state_write.completed file={} bytes={}", path.file_name() @@ -105,6 +132,42 @@ fn write_separation_state_blocking(path: &Path, bytes: &[u8]) -> Result<(), Stri Ok(()) } +#[cfg(not(windows))] +fn replace_separation_state_atomically(temporary: &Path, target: &Path) -> std::io::Result<()> { + fs::rename(temporary, target) +} + +#[cfg(windows)] +fn replace_separation_state_atomically(temporary: &Path, target: &Path) -> std::io::Result<()> { + use std::os::windows::ffi::OsStrExt; + use windows_sys::Win32::Storage::FileSystem::{ + MOVEFILE_REPLACE_EXISTING, MOVEFILE_WRITE_THROUGH, MoveFileExW, + }; + + let source = temporary + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + let destination = target + .as_os_str() + .encode_wide() + .chain(std::iter::once(0)) + .collect::>(); + let moved = unsafe { + MoveFileExW( + source.as_ptr(), + destination.as_ptr(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH, + ) + }; + if moved == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } +} + pub fn read_separation_state(path: &Path) -> Result { app_log!( "ui_separation.state_read.start file={}", @@ -215,3 +278,23 @@ fn remove_separation_state(root: &Path, asset_id: &str) -> Result<(), String> { Err(error) => Err(format!("删除 separation state 失败：{error}")), } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn atomic_state_replacement_overwrites_existing_target() { + let directory = tempfile::tempdir().expect("create temporary state directory"); + let target = directory.path().join("state.json"); + let temporary = directory.path().join("state.json.tmp"); + fs::write(&target, b"old").expect("write old state"); + fs::write(&temporary, b"new").expect("write new state"); + + replace_separation_state_atomically(&temporary, &target) + .expect("replacement should overwrite existing state"); + + assert_eq!(fs::read(&target).expect("read replaced state"), b"new"); + assert!(!temporary.exists()); + } +} From 413707da33041d01383712bb33e3f585465ae833 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 15:40:26 +0800 Subject: [PATCH 212/248] =?UTF-8?q?=E6=8F=90=E5=89=8D=E6=8B=92=E7=BB=9D?= =?UTF-8?q?=E9=AB=98=E4=BD=8D=E6=B7=B1=20PNG=20=E8=A7=A3=E7=A0=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 通过 image crate 的 ImageReader decoder 元数据在像素缓冲分配前拒绝 16-bit PNG。 继续使用同一外部 decoder 执行受限像素读取校验，不手写 PNG 解析器。 --- server-rs/crates/api-server/src/raw_image.rs | 26 ++++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index bead3a52e..6f3b55c52 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -4,7 +4,7 @@ use axum::{ http::StatusCode, }; use bytes::Bytes; -use image::{GenericImageView, ImageFormat, ImageReader}; +use image::{ImageDecoder, ImageFormat, ImageReader}; use platform_image::{ GPT_IMAGE_2_2K_LONG_EDGE_THRESHOLD, RAW_IMAGE_MAX_EDGE, RAW_IMAGE_MAX_PIXELS, RawImageEditImage, RawImageEditOptions, create_vector_engine_raw_image_edit, @@ -393,11 +393,11 @@ fn decode_image( if reader.format() != Some(ImageFormat::Png) { return Err(bad_request(format!("{field} 文件必须是有效 PNG 文件"))); } - let decoded = reader - .decode() + let decoder = reader + .into_decoder() .map_err(|error| map_decode_image_error(field, error))?; if matches!( - decoded.color(), + decoder.color_type(), image::ColorType::Rgba16 | image::ColorType::Rgb16 | image::ColorType::L16 @@ -407,7 +407,23 @@ fn decode_image( "{field} 必须为 8-bit PNG（每通道 8 位）" ))); } - let (width, height) = decoded.dimensions(); + let (width, height) = decoder.dimensions(); + let decoded_bytes = usize::try_from(decoder.total_bytes()).map_err(|_| { + bad_request(format!( + "{field} 文件超出 PNG 尺寸或解码资源上限（单边不超过 {RAW_IMAGE_MAX_EDGE}px）" + )) + })?; + let max_decoded_bytes = + usize::try_from(RAW_IMAGE_MAX_PIXELS.saturating_mul(4)).unwrap_or(usize::MAX); + if decoded_bytes > max_decoded_bytes { + return Err(bad_request(format!( + "{field} 文件超出 PNG 尺寸或解码资源上限（单边不超过 {RAW_IMAGE_MAX_EDGE}px）" + ))); + } + let mut decoded = vec![0_u8; decoded_bytes]; + decoder + .read_image(&mut decoded) + .map_err(|error| map_decode_image_error(field, error))?; Ok(( RawImageEditImage { bytes, From bb2f080d3bf6f1cadc20211a6ecc8428bbdc9a79 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 16:18:13 +0800 Subject: [PATCH 213/248] =?UTF-8?q?=E5=88=86=E7=A6=BB=E4=BA=A7=E7=89=A9?= =?UTF-8?q?=E4=BD=BF=E7=94=A8=20UUID=20=E6=96=87=E4=BB=B6=E5=90=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 切分图片文件名改用 UUID，避免把 NodeId 当作物理路径。 继续通过 cut_paths 写入 BoundNode.cut_image_path，保留逻辑节点 ID 与产物路径的独立性。 --- .../src/ui_editor/commands/separation/workflow/mod.rs | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs index 181964feb..ffa1092da 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/separation/workflow/mod.rs @@ -20,6 +20,7 @@ use std::collections::HashMap; use std::fs; use std::path::Path; use std::time::Instant; +use uuid::Uuid; pub(crate) async fn separate_ui_impl( project_path: String, @@ -205,13 +206,7 @@ pub(crate) async fn separate_ui_impl( } = decision { let node_id = to_node.as_str(); - if !node_id.chars().all(|character| { - character.is_ascii_alphanumeric() || matches!(character, '-' | '_') - }) { - cut_error = Some(format!("节点 {} 的名称包含非法路径字符", node_id)); - break; - } - let cut_path = sidecar.join(format!("cut-{node_id}.png")); + let cut_path = sidecar.join(format!("cut-{}.png", Uuid::new_v4())); if !cut_path.starts_with(&sidecar) { cut_error = Some(format!("节点 {} 的路径越界", node_id)); break; From 0944667c370747f49c9fe1b2aae61f271449d10f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 16:22:06 +0800 Subject: [PATCH 214/248] =?UTF-8?q?=E9=99=90=E5=88=B6=E5=8E=9F=E5=A7=8B?= =?UTF-8?q?=E5=9B=BE=E7=89=87=E8=A7=A3=E7=A0=81=E5=B9=B6=E5=8F=91?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 为 raw image PNG 预处理增加 AppState 专用 semaphore。 按请求截止时间限制槽位等待和 blocking 解码等待，并让 permit 持有到任务结束。 --- server-rs/crates/api-server/src/raw_image.rs | 59 ++++++++++++++++++-- server-rs/crates/api-server/src/state.rs | 11 ++++ 2 files changed, 64 insertions(+), 6 deletions(-) diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index 6f3b55c52..c6bdf1d65 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -13,6 +13,7 @@ use platform_image::{ use serde::Serialize; use serde_json::json; use std::io::Cursor; +use std::time::{Duration, Instant}; use crate::{ asset_billing::{ @@ -25,7 +26,7 @@ use crate::{ require_openai_image_settings, }, request_context::RequestContext, - state::AppState, + state::{AppState, RAW_IMAGE_DECODE_MAX_CONCURRENCY}, tracking::record_external_generation_run_after_success, }; use time::OffsetDateTime; @@ -60,6 +61,7 @@ pub(crate) struct RawImageEditResponse { } const RAW_IMAGE_MAX_TEXT_FIELD_BYTES: usize = 16 * 1024; +const RAW_IMAGE_PREPARE_TIMEOUT: Duration = Duration::from_secs(30); pub(crate) async fn edit_raw_image( State(state): State, @@ -68,11 +70,47 @@ pub(crate) async fn edit_raw_image( multipart: Multipart, ) -> Result, AppError> { let payload = parse_multipart_request(multipart).await?; - let prepared = tokio::task::spawn_blocking(move || prepare_request(payload)) - .await - .map_err(|error| { - AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_message(error.to_string()) - })??; + let local_deadline = Instant::now() + .checked_add(RAW_IMAGE_PREPARE_TIMEOUT) + .unwrap_or_else(Instant::now); + let processing_deadline = request_context + .external_call_deadline() + .map(|deadline| deadline.min(local_deadline)) + .unwrap_or(local_deadline); + let permit = match tokio::time::timeout_at( + tokio::time::Instant::from_std(processing_deadline), + state.raw_image_decode_limiter().acquire_owned(), + ) + .await + { + Ok(Ok(permit)) => permit, + Ok(Err(error)) => { + return Err( + AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({ + "provider": "raw-image-edit", + "code": "RAW_IMAGE_DECODE_LIMITER_UNAVAILABLE", + "message": format!("raw 图片解码并发控制器不可用：{error}"), + })), + ); + } + Err(_) => return Err(raw_image_prepare_timeout_error("等待 raw 图片解码槽位超时")), + }; + let worker = tokio::task::spawn_blocking(move || { + // 超时只能停止 async 等待，permit 必须由 blocking closure 持有到解码真正结束。 + let _permit = permit; + prepare_request(payload) + }); + let prepared = + match tokio::time::timeout_at(tokio::time::Instant::from_std(processing_deadline), worker) + .await + { + Ok(Ok(result)) => result?, + Ok(Err(error)) => { + return Err(AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR) + .with_message(error.to_string())); + } + Err(_) => return Err(raw_image_prepare_timeout_error("raw 图片解码处理超时")), + }; let settings = require_openai_image_settings(&state)?.with_external_api_audit_context( &request_context, Some(authenticated.claims().user_id().to_string()), @@ -472,6 +510,15 @@ fn bad_request(message: impl Into) -> AppError { })) } +fn raw_image_prepare_timeout_error(message: &str) -> AppError { + AppError::from_status(StatusCode::GATEWAY_TIMEOUT).with_details(json!({ + "provider": "raw-image-edit", + "code": "RAW_IMAGE_PREPARE_TIMEOUT", + "message": message, + "maxConcurrency": RAW_IMAGE_DECODE_MAX_CONCURRENCY, + })) +} + #[cfg(test)] mod tests { use super::*; diff --git a/server-rs/crates/api-server/src/state.rs b/server-rs/crates/api-server/src/state.rs index dc53af8cc..dc7b08503 100644 --- a/server-rs/crates/api-server/src/state.rs +++ b/server-rs/crates/api-server/src/state.rs @@ -56,6 +56,8 @@ const ADMIN_ROLE: &str = "admin"; const EDITOR_AGENT_LLM_MAX_RETRIES: u32 = 1; const EDITOR_AGENT_LLM_MAX_RETRY_BACKOFF_MS: u64 = 60_000; pub(crate) const CHARACTER_ANIMATION_OSS_MAX_CONCURRENCY: usize = 8; +// Raw image PNG 解码会进入 Tokio blocking pool；单独限流，避免图片请求挤占其它 blocking 工作。 +pub(crate) const RAW_IMAGE_DECODE_MAX_CONCURRENCY: usize = 4; // P=8：父侧成功图片读取/解码槽。配 N=16 是内存与出口吞吐的折中， // 极端完整 body 内存按 (N + P) × 32 MiB 评估（见调度方案 §9.2）。 pub(crate) const BGFILTER_IMAGE_VALIDATION_MAX_CONCURRENCY: usize = 8; @@ -304,6 +306,7 @@ pub struct AppStateInner { matting_client: Option, bgfilter_provider_http_client: reqwest::Client, bgfilter_worker_http_client: reqwest::Client, + raw_image_decode_limiter: Arc, bgfilter_image_validation_limiter: Arc, character_animation_oss_http_client: reqwest::Client, character_animation_oss_io_limiter: Arc, @@ -612,6 +615,9 @@ impl AppState { let bgfilter_image_validation_limiter = Arc::new(Semaphore::new( bgfilter_image_validation_concurrency.min(Semaphore::MAX_PERMITS), )); + let raw_image_decode_limiter = Arc::new(Semaphore::new( + RAW_IMAGE_DECODE_MAX_CONCURRENCY.min(Semaphore::MAX_PERMITS), + )); let character_animation_oss_http_client = build_character_animation_oss_http_client()?; let character_animation_oss_io_limiter = Arc::new(Semaphore::new(CHARACTER_ANIMATION_OSS_MAX_CONCURRENCY)); @@ -674,6 +680,7 @@ impl AppState { matting_client, bgfilter_provider_http_client, bgfilter_worker_http_client, + raw_image_decode_limiter, bgfilter_image_validation_limiter, character_animation_oss_http_client, character_animation_oss_io_limiter, @@ -1584,6 +1591,10 @@ impl AppState { &self.bgfilter_worker_http_client } + pub fn raw_image_decode_limiter(&self) -> Arc { + self.raw_image_decode_limiter.clone() + } + pub fn bgfilter_worker_reached(&self) -> bool { self.bgfilter_worker_reached.load(Ordering::Relaxed) } From 6764950aecb08ede58d6d6be1d53826fd0f89b2b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 16:27:23 +0800 Subject: [PATCH 215/248] =?UTF-8?q?=E9=99=90=E5=88=B6=E5=8E=9F=E5=A7=8B?= =?UTF-8?q?=E5=9B=BE=E7=89=87=E4=B8=8A=E4=BC=A0=E5=AD=97=E8=8A=82?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将 image 和 mask multipart 字段改为分块读取并限制单字段大小。 新增图片总字节上限、413 错误和字段/总量边界测试。 同步 Raw GPT Image 2 代理技术方案的资源防护合同。 --- ...案】Raw GPT Image 2图片编辑代理-2026-09-07.md | 2 +- server-rs/crates/api-server/src/raw_image.rs | 128 +++++++++++++++++- 2 files changed, 123 insertions(+), 7 deletions(-) diff --git a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md index 393eb6fb4..81f44f5e7 100644 --- a/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md +++ b/docs/technical/【技术方案】Raw GPT Image 2图片编辑代理-2026-09-07.md @@ -38,7 +38,7 @@ height: 1024 校验通过后按整数尺寸发送给 provider，不静默 clamp 或改写调用者尺寸。 -Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；图片字段由 multipart 解析器收集为可共享字节缓冲，文本字段按 chunk 流式读取并在达到 `16 KiB` 时立即拒绝，随后在阻塞线程中完成 PNG 解码。PNG 仅接受 8-bit/channel（`png::BitDepth::Eight`），其它位深在解码前以 400 返回“`{field} 必须为 8-bit PNG（每通道 8 位）`”；PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 +Raw 路由的 multipart body limit 为 `64 MiB`，覆盖图片和文本字段；每个 `image` / `mask` 字段最多 `32 MiB`，两者图片字节总计最多 `48 MiB`，图片字段按 chunk 流式收集，越过任一上限立即返回 `413`，文本字段按 chunk 流式读取并在达到 `16 KiB` 时立即拒绝。随后图片校验在独立的 raw-image 解码 semaphore（进程内最多 4 个 blocking 解码任务）中执行，并受 30 秒本地处理截止时间约束；超时只停止等待，不会提前释放仍在运行任务持有的槽位。PNG 仅接受 8-bit/channel（`png::BitDepth::Eight`），其它位深在解码前以 400 返回“`{field} 必须为 8-bit PNG（每通道 8 位）`”；PNG 解码使用与输出合同一致的资源上限：宽高各不超过 `3840`，解码分配不超过 `8294400 × 4` 字节；不再执行 base64 入站解码。 服务端发送给 `platform-image` 时固定注入： diff --git a/server-rs/crates/api-server/src/raw_image.rs b/server-rs/crates/api-server/src/raw_image.rs index c6bdf1d65..30932c3a1 100644 --- a/server-rs/crates/api-server/src/raw_image.rs +++ b/server-rs/crates/api-server/src/raw_image.rs @@ -61,6 +61,8 @@ pub(crate) struct RawImageEditResponse { } const RAW_IMAGE_MAX_TEXT_FIELD_BYTES: usize = 16 * 1024; +const RAW_IMAGE_MAX_FILE_BYTES: usize = 32 * 1024 * 1024; +const RAW_IMAGE_MAX_INPUT_BYTES: usize = 48 * 1024 * 1024; const RAW_IMAGE_PREPARE_TIMEOUT: Duration = Duration::from_secs(30); pub(crate) async fn edit_raw_image( @@ -219,6 +221,7 @@ async fn parse_multipart_request( let mut output_format = None; let mut width = None; let mut height = None; + let mut image_bytes_total = 0usize; while let Some(field) = multipart.next_field().await.map_err(|error| { tracing::warn!(error = %error, "raw image multipart 字段解析失败"); @@ -233,13 +236,13 @@ async fn parse_multipart_request( if image.is_some() { return Err(bad_request("image 字段不能重复")); } - image = Some(read_multipart_image(field, "image").await?); + image = Some(read_multipart_image(field, "image", &mut image_bytes_total).await?); } "mask" => { if mask.is_some() { return Err(bad_request("mask 字段不能重复")); } - mask = Some(read_multipart_image(field, "mask").await?); + mask = Some(read_multipart_image(field, "mask", &mut image_bytes_total).await?); } "prompt" => set_text_field(&mut prompt, field, "prompt").await?, "quality" => set_text_field(&mut quality, field, "quality").await?, @@ -269,27 +272,79 @@ async fn parse_multipart_request( } async fn read_multipart_image( - field: axum::extract::multipart::Field<'_>, + mut field: axum::extract::multipart::Field<'_>, name: &str, + total_bytes: &mut usize, ) -> Result { let mime_type = field.content_type().unwrap_or_default().to_string(); if !mime_type.eq_ignore_ascii_case("image/png") { return Err(bad_request(format!("{name} 必须为 image/png"))); } - let bytes = field.bytes().await.map_err(|error| { + let mut bytes = Vec::new(); + let mut field_bytes = 0usize; + while let Some(chunk) = field.chunk().await.map_err(|error| { tracing::warn!(field = name, error = %error, "raw image multipart 图片读取失败"); bad_request(format!("{name} 字段读取失败")) - })?; + })? { + append_bounded_image_chunk( + &mut bytes, + &mut field_bytes, + total_bytes, + &chunk, + name, + RAW_IMAGE_MAX_FILE_BYTES, + RAW_IMAGE_MAX_INPUT_BYTES, + )?; + } if bytes.is_empty() { return Err(bad_request(format!("{name} 文件不能为空"))); } Ok(RawImageData { - bytes, + bytes: Bytes::from(bytes), mime_type: "image/png".to_string(), file_name: format!("{name}.png"), }) } +fn append_bounded_image_chunk( + bytes: &mut Vec, + field_bytes: &mut usize, + total_bytes: &mut usize, + chunk: &[u8], + name: &str, + max_field_bytes: usize, + max_total_bytes: usize, +) -> Result<(), AppError> { + let next_field_bytes = field_bytes.saturating_add(chunk.len()); + if next_field_bytes > max_field_bytes { + tracing::warn!( + field = name, + bytes = next_field_bytes, + max_bytes = max_field_bytes, + "raw image multipart 图片字段超过大小限制" + ); + return Err(payload_too_large(format!( + "{name} 图片字段不能超过 {max_field_bytes} 字节" + ))); + } + let next_total_bytes = total_bytes.saturating_add(chunk.len()); + if next_total_bytes > max_total_bytes { + tracing::warn!( + field = name, + bytes = next_total_bytes, + max_bytes = max_total_bytes, + "raw image multipart 图片总输入超过大小限制" + ); + return Err(payload_too_large(format!( + "image 和 mask 图片总大小不能超过 {max_total_bytes} 字节" + ))); + } + bytes.extend_from_slice(chunk); + *field_bytes = next_field_bytes; + *total_bytes = next_total_bytes; + Ok(()) +} + async fn set_text_field( target: &mut Option, mut field: axum::extract::multipart::Field<'_>, @@ -510,6 +565,15 @@ fn bad_request(message: impl Into) -> AppError { })) } +fn payload_too_large(message: impl Into) -> AppError { + AppError::from_status(StatusCode::PAYLOAD_TOO_LARGE).with_details(json!({ + "provider": "raw-image-edit", + "message": message.into(), + "maxFileBytes": RAW_IMAGE_MAX_FILE_BYTES, + "maxInputBytes": RAW_IMAGE_MAX_INPUT_BYTES, + })) +} + fn raw_image_prepare_timeout_error(message: &str) -> AppError { AppError::from_status(StatusCode::GATEWAY_TIMEOUT).with_details(json!({ "provider": "raw-image-edit", @@ -699,4 +763,56 @@ mod tests { assert!(format!("{error:?}").contains("prompt 不能超过 16384 字节")); } + + #[test] + fn image_chunks_enforce_field_and_total_limits() { + let mut bytes = Vec::new(); + let mut field_bytes = 0; + let mut total_bytes = 0; + append_bounded_image_chunk( + &mut bytes, + &mut field_bytes, + &mut total_bytes, + b"abc", + "image", + 3, + 8, + ) + .expect("chunk at field limit should pass"); + assert_eq!(bytes, b"abc"); + assert_eq!(field_bytes, 3); + assert_eq!(total_bytes, 3); + + let error = append_bounded_image_chunk( + &mut bytes, + &mut field_bytes, + &mut total_bytes, + b"d", + "image", + 3, + 8, + ) + .expect_err("chunk over field limit should fail"); + assert_eq!(error.status_code(), StatusCode::PAYLOAD_TOO_LARGE); + assert_eq!(bytes, b"abc"); + assert_eq!(field_bytes, 3); + assert_eq!(total_bytes, 3); + + let mut other_field = Vec::new(); + let mut other_field_bytes = 0; + let error = append_bounded_image_chunk( + &mut other_field, + &mut other_field_bytes, + &mut total_bytes, + b"123456", + "mask", + 8, + 8, + ) + .expect_err("chunk over total limit should fail"); + assert_eq!(error.status_code(), StatusCode::PAYLOAD_TOO_LARGE); + assert!(other_field.is_empty()); + assert_eq!(other_field_bytes, 0); + assert_eq!(total_bytes, 3); + } } From d44c6ab62c06ee7d1c42a6d889bed5c69c719c2e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 16:28:31 +0800 Subject: [PATCH 216/248] =?UTF-8?q?=E8=A1=A5=E5=85=85=E8=83=8C=E6=99=AF?= =?UTF-8?q?=E8=8A=82=E7=82=B9=E8=A7=84=E5=88=99=E7=A4=BA=E4=BE=8B=E8=AF=B4?= =?UTF-8?q?=E6=98=8E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../src-tauri/src/ui_editor/commands/recognition.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs index c71475440..f6d087ea0 100644 --- a/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs +++ b/apps/ai-game-creator-shell/src-tauri/src/ui_editor/commands/recognition.rs @@ -51,7 +51,8 @@ const SYSTEM_PROMPT: &str = r#" 文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别. * 多行文本只使用一个节点. * 不鼓励兄弟节点相互重叠. -* 对于面板等容器的背景等, 必须作为父节点的组件, 禁止新增冗余的所谓"背景节点". +* 对于面板等容器的背景等, 必须作为父节点的组件, 禁止新增冗余的所谓"背景节点". 例如:对于全局的背景可以直接作为根节点的图片组件, 不用另起节点 + "#; #[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)] From c7fe531b16a243f8e59d4b9106b5eba56be9b556 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=8B=E5=BE=B7=E5=AE=87?= Date: Sat, 12 Sep 2026 16:30:53 +0800 Subject: [PATCH 217/248] =?UTF-8?q?=E6=81=A2=E5=A4=8D=E9=A2=84=E8=A7=88?= =?UTF-8?q?=E7=99=BE=E5=88=86=E6=AF=94=E6=8C=89=E9=92=AE=E9=80=82=E9=85=8D?= =?UTF-8?q?=E7=94=BB=E5=B8=83?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 将缩放百分比显示改为可点击的适配画布按钮 复用工具栏按钮的鼠标焦点处理并按当前容器重新计算视口 补充预览缩放组件回归测试并同步交互设计文档 --- .../components/preview/PreviewWorkspace.tsx | 29 ++++++++++++++++--- .../tests/previewWorkspaceZoom.test.tsx | 15 ++++++++++ ...【交互设计】预览画布缩放滑杆-2026-09-05.md | 4 +-- 3 files changed, 42 insertions(+), 6 deletions(-) diff --git a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx index 8fbc37dbb..8ec753d09 100644 --- a/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx +++ b/apps/ai-game-creator-shell/src/view/ui-editor/components/preview/PreviewWorkspace.tsx @@ -141,6 +141,25 @@ export function PreviewWorkspace({ setViewportState(next); }, []); + const fitToCanvas = useCallback(() => { + if (!logicalSize) return; + const element = viewportElementRef.current; + setViewport( + fitViewportToBounds({ + bounds: { + x: 0, + y: 0, + width: logicalSize.width, + height: logicalSize.height, + }, + canvasSize: { + width: element?.clientWidth || canvasSizeRef.current.width, + height: element?.clientHeight || canvasSizeRef.current.height, + }, + }), + ); + }, [logicalSize, setViewport]); + useEffect(() => { if (!activeImageId || !logicalSize) return; const element = viewportElementRef.current; @@ -548,12 +567,14 @@ export function PreviewWorkspace({ zoomToDisplayScale(Number(event.target.value) / 100) } /> - {displayPercent} - +