Merge branch 'master' into fix/cant-pan-after-ui-editor
Project CI / AI game creator shell Rust shard 2/4 (pull_request) Waiting to run
Project CI / AI game creator shell Rust shard 3/4 (pull_request) Waiting to run
Project CI / AI game creator shell Rust shard 4/4 (pull_request) Waiting to run
Project CI / AI game creator shell Rust smoke (pull_request) Waiting to run
Project CI / AI game creator shell Rust crates (pull_request) Waiting to run
Project CI / Backend tests (pull_request) Waiting to run
Project CI / Native shell tests (pull_request) Waiting to run
Project CI / Frontend tests (pull_request) Waiting to run
Project CI / Repository checks (pull_request) Waiting to run
Project CI / AI game creator shell web tests (pull_request) Waiting to run
Project CI / AI game creator shell Rust shard 1/4 (pull_request) Has started running

This commit is contained in:
2026-09-14 19:00:47 +08:00
134 changed files with 6861 additions and 1389 deletions
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "@genarrative/ai-game-creator-shell",
"private": true,
"version": "0.1.27",
"version": "0.1.29",
"type": "module",
"scripts": {
"dev": "node scripts/start-tauri-dev.mjs",
@@ -108,6 +108,8 @@ const rustSharedContractSource = fs.readFileSync(
);
const allowedUncalledTauriCommands = [
'append_direct_project_conversation_message',
// TODO: Remove the retired binding command after the legacy runtime path is removed.
'bind_components',
'chat_with_game_creator_agent',
'check_ui_editor_font_glyph_coverage',
'create_ui_design_resource',
@@ -8,6 +8,7 @@ export const SKILL_PACK_SCHEMA_VERSION = 'agc-skill-pack.v1';
export const EXPECTED_SKILL_NAMES = Object.freeze([
'agc-browser-playtest',
'agc-client-projection',
'agc-game-production-workflow',
'agc-project-structure',
'agc-web-game-development',
'taonier-art-assets',
+1 -1
View File
@@ -1725,7 +1725,7 @@ dependencies = [
[[package]]
name = "genarrative-ai-game-creator-shell"
version = "0.1.27"
version = "0.1.29"
dependencies = [
"agent-runtime-core",
"axum",
@@ -1,6 +1,6 @@
[package]
name = "genarrative-ai-game-creator-shell"
version = "0.1.27"
version = "0.1.29"
edition = "2021"
publish = false
@@ -0,0 +1,26 @@
---
name: agc-game-production-workflow
description: Orchestrate a complete AGC game delivery from an approved brief to a playable, art-integrated, browser-validated product. Use when creating a new game, implementing a substantial game brief, or turning a planning document into a finished game.
---
# AGC Game Production Workflow
Use this Skill as the top-level SOP for a new game or a substantial game brief. The tools are stages in one delivery chain, not independent suggestions. Do not stop after producing a plan, after writing code, or after generating an image.
## Stage flow
1. **Brief and scope** — Read the current planning output and project instructions. Extract the game loop, player actions, entities, visual requirements, target viewports, and the completion evidence. If the brief is incomplete, ask focused questions before side effects.
2. **Project and asset inventory** — Inspect the existing project structure and call `agc_list_registered_assets` (and `agc_list_project_files` when needed). Record which requested visuals already have usable registered identities and which are missing. Do not invent asset identities from filenames.
3. **Visual production** — For missing or unsuitable visuals, call the reviewed `agc_tools` workflow: use `taonier_prepare_game_art` for a complete package, or `agc_generate_image` / `agc_edit_image` for focused assets. Read returned paths, identities, and warnings. A warning or partial package requires a narrower retry or independent assets before continuing.
4. **Game implementation** — Implement the complete playable loop and wire the returned project-relative asset paths into the actual runtime. Every required character, object, background, effect, and UI visual must have a real source or an explicit brief-level decision to remain code-native. Generated assets that are unused, documentation-only, or replaced by emoji/CSS placeholders do not satisfy this stage.
5. **Build and local verification** — Run the projects bootstrap/install and verify/build commands. Confirm the actual playable entry under `dist` (or the editor runtime for a supported editor project) and fix build or asset-loading failures before preview.
6. **Browser playtest** — Call `agc_browser_playtest` for desktop and mobile evidence after meaningful changes. Check the game loop, input, layout, asset loading, and visible use of the generated art. Fix findings and repeat stages 46 until the evidence is clean.
7. **Delivery** — Report the implemented behavior, real asset paths and identities used, build result, playtest evidence, warnings, and any explicit remaining gap. Do not claim complete while a required stage is failed, skipped without the briefs justification, or missing evidence.
## Stage transitions
Advance only when the current stage has its output: brief → inventory; inventory → art decision; art decision → usable registered assets or an explicit no-art decision; implementation → source references to those assets; build → playable entry; playtest → evidence; delivery → truthful report. If a tool fails, preserve its error and stop or repair at that stage instead of silently substituting a later-stage placeholder.
For a small edit to an existing game where the brief and suitable assets are unchanged, use the focused edit path and do not regenerate art. This exception does not apply to a new game or a substantial planning brief.
Read the referenced specialist Skills for their detailed contracts: `agc-project-structure`, `taonier-art-assets`, `agc-web-game-development`, `agc-client-projection`, and `agc-browser-playtest`.
@@ -0,0 +1,4 @@
interface:
display_name: "完整游戏生产流程"
short_description: "从策划案到真实美术接入和试玩验收的连续交付"
default_prompt: "Use $agc-game-production-workflow to take the current game brief through inventory, art, implementation, build, playtest, and delivery."
@@ -0,0 +1,5 @@
# Workflow contract
The production Skill owns sequencing and transition evidence. Specialist Skills own the detailed safety and data rules for each tool family. A specialist tool result is never a delivery result by itself: image generation must be followed by registered identity inspection and runtime integration; code writing must be followed by build verification; a successful preview launch must be followed by desktop and mobile playtest evidence when the brief targets both.
The no-art exception is valid only when the brief explicitly requests a code-native visual treatment or the inventory proves that all required visuals are already registered and suitable. Emoji, CSS primitives, random local files, and generated files that are not referenced by the runtime are not evidence of an integrated art package.
@@ -14,7 +14,7 @@ Implement the user's actual game request in the current project as an npm-manage
3. Build with the project's npm script before previewing. The playable entry is the package directory's `dist/index.html`; never report an unbuilt bare-module page as playable. Import assets or configure public assets so all runtime media is included in dist; preview and exports cannot read outside it.
4. Build a complete playable loop: visible objective, responsive input, meaningful state changes, success or failure feedback, and a reliable restart path where the game needs one.
5. Fit the active game scene to desktop and mobile viewports without accidental page scrollbars. Reserve deliberate safe space for HUD elements instead of covering interactive content.
6. Reuse registered Taonier art when available through `agc_tools`. Load media defensively and keep gameplay usable when an optional derivative is absent; never relabel a local placeholder as platform art.
6. Invoke `taonier-art-assets` for every new game brief that needs visual assets. First reuse suitable registered Taonier art; when the brief's required visual elements are missing or unsuitable, call the reviewed `agc_tools` generation/edit workflow in the same task. After the tool returns, wire its relative paths into the game and verify the rendered result. A game with unused generated assets or placeholder emoji/CSS where requested art should appear is not complete. Load media defensively only for genuinely optional effects, and never relabel a local placeholder as platform art.
7. Let Phaser own the render loop and input dispatch. Avoid duplicate scenes, stale event listeners, and state that survives restart unintentionally.
8. After a meaningful game change, use the browser playtest Skill and fix issues shown by real evidence before reporting completion.
@@ -1,7 +1,29 @@
{
"schemaVersion": "agc-skill-pack.v1",
"version": "2026-08-26.12",
"version": "2026-08-26.13",
"skills": [
{
"name": "agc-game-production-workflow",
"purpose": "把完整游戏从策划案按阶段推进到真实素材接入、构建、试玩和交付",
"triggers": [
"从策划案创建完整游戏",
"实现完整游戏交付",
"需要衔接策划、素材、代码、构建和试玩"
],
"requiredTools": [
"agc_tools.agc_list_registered_assets",
"agc_tools.agc_generate_image",
"agc_tools.agc_edit_image",
"agc_tools.taonier_prepare_game_art",
"agc_tools.agc_browser_playtest"
],
"files": [
"SKILL.md",
"agents/openai.yaml",
"references/workflow-contract.md"
],
"sha256": "91082fdff4123f1e1fcf930af433cbea51a8c9d26991678b19028b344ea49f39"
},
{
"name": "agc-project-structure",
"purpose": "约束当前项目根、游戏代码、美术素材与客户端状态的职责边界",
@@ -31,6 +53,7 @@
"已有陶泥儿素材需要接入玩法"
],
"requiredTools": [
"agc_tools.agc_list_registered_assets",
"agc_tools.agc_generate_image",
"agc_tools.agc_edit_image",
"agc_tools.taonier_prepare_game_art"
@@ -40,7 +63,7 @@
"agents/openai.yaml",
"references/platform-art-contract.md"
],
"sha256": "82e4b2ee8ca8147b51ca206b0565b3cc244dc5d3cddb8343875001c0beb4711f"
"sha256": "bd1e415aac0cd0f97090296f34c67898dd731d1e177ec91a56027f9b68a88b37"
},
{
"name": "agc-web-game-development",
@@ -57,7 +80,7 @@
"agents/openai.yaml",
"references/game-quality-checklist.md"
],
"sha256": "0649c72dd53e05ad7c87b28def1397c2badf61b0c308091196c40f7c48a8b36a"
"sha256": "05b5cfbf7a40fd303717491f5cea84ff339a73359c9678b283fd54d2b5c45efd"
},
{
"name": "agc-browser-playtest",
@@ -5,7 +5,15 @@ description: Prepare, recover, inspect, and integrate real Taonier platform game
# Taonier Art Assets
Use real platform assets only through the reviewed `agc_tools` MCP server. Use
Use real platform assets only through the reviewed `agc_tools` MCP server. When
building a new game from a brief that names characters, objects, backgrounds,
effects, or other visual elements, this Skill is an execution step: inspect
existing assets, generate or reuse suitable art, process it when needed, and
integrate the returned paths into the playable game before reporting the game
complete. Do not treat the art step as optional merely because the user did
not repeat “生图” in the latest message.
Use
`agc_generate_image` for a single ordinary image, character image, visual-spec
image, UI design image, or publication material; use `agc_edit_image` for an
edit of an existing registered image; use `taonier_prepare_game_art` only for
@@ -17,11 +25,11 @@ the complete game-art package and its canonical slices.
## Workflow
1. Inspect existing `assets/` and registered project evidence before requesting new art. Reuse suitable assets when the user did not ask to regenerate them.
1. Inspect existing `assets/` and registered project evidence before requesting new art. Reuse suitable assets when they satisfy the current brief. If the brief requires visual elements that are absent or unsuitable, call the appropriate generation tool during the same game implementation task; do not continue with placeholder art and silently defer generation.
2. For one new image, call `agc_generate_image` with `kind="image"` (or `character`, `icon-spec`, `ui-prototype`, or `publication-material` when that is the explicit intent). For changes to an existing registered image, call `agc_edit_image` with its `sourceLocalAssetId`; do not fake an edit with a new-image request. For a complete game-art package, call `taonier_prepare_game_art` only when the current intent requires new or recoverable platform art. Use `mode="regenerate"` only after the latest User message is a standalone reviewed immediate-confirmation command such as `请重新生成美术`; punctuation may end it, but no brief, condition, negation, alternative, cost qualifier, deferral, or other text may accompany it. Describe the desired style and gameplay constraints in an earlier non-billable turn, then obtain the standalone confirmation turn; otherwise use `mode="reuse-or-create"`. Quoted UI copy or examples, explanations, questions, historical wording, model/MCP arguments do not authorize regeneration. Pass a concise game-specific visual brief that names the required gameplay entities, background exclusions, tiling needs, and viewport constraints. Do not call either generation tool for greetings, date questions, or text-only code fixes.
3. Treat the tool result as authoritative. Read `mode`, `assetPaths`, `slicePaths`, `resources`, and every entry in both `warnings` and `sliceWarnings`. `resources` is the client's safe projection of registered Canvas identities; use only its returned relative paths and identities. Never invent a resource, slice, platform identity, warning-free result, or successful regeneration.
4. A newly created or explicitly regenerated standard package is complete only when `slicePaths` contains the four canonical independent slices. An empty or partial `slicePaths` result never satisfies an independent-asset requirement; stop and report the warning instead of guessing atlas coordinates or fabricating derivatives. A trusted legacy complete sheet may still be used without slices only when the current request does not require independent assets.
5. Inspect the returned background, complete sheet, and available slice previews before integrating them. Then use suitable returned runtime assets in the game's actual visible experience and confirm their visible use in desktop and mobile playtest evidence. `art-spec.png` is a reference specification, not a runtime background, character, prop, or effect. Background exclusions, seamless tiling, entity semantics, and final draw dimensions are visual/runtime acceptance checks; a prompt alone does not prove them. A hidden or side-panel preview does not count as gameplay use.
4. A newly created or explicitly regenerated standard package is complete only when `slicePaths` contains the four canonical independent slices. An empty or partial `slicePaths` result never satisfies an independent-asset requirement: if a `sliceWarning` reports too many or unusable elements, narrow the edit/generation brief or generate the needed independent images and continue the integration; do not guess atlas coordinates, fabricate derivatives, or silently fall back to placeholders. A trusted legacy complete sheet may still be used without slices only when the current request does not require independent assets.
5. Inspect the returned background, complete sheet, and available slice previews before integrating them. Then use suitable returned runtime assets in the game's actual visible experience and confirm their visible use in desktop and mobile playtest evidence. The implementation is incomplete while generated assets remain unused, are referenced only by documentation, or are replaced by emoji, CSS shapes, or other placeholders where the brief requires the generated art. `art-spec.png` is a reference specification, not a runtime background, character, prop, or effect. Background exclusions, seamless tiling, entity semantics, and final draw dimensions are visual/runtime acceptance checks; a prompt alone does not prove them. A hidden or side-panel preview does not count as gameplay use.
6. Preserve warning details in the final report. If the tool reports missing credentials, uncertain operation state, invalid provenance, download failure, or decode failure, stop and report the actionable reason; do not substitute generated CSS shapes and call the platform step complete.
Before interpreting async recovery, source-preserved warnings, or slice warnings, read `references/platform-art-contract.md`.
@@ -4999,7 +4999,7 @@ case "$extra_roots" in *'"method":"skills/extraRoots/set"'*) ;; *) exit 87 ;; es
printf '%s\n' '{"id":2,"result":{}}'
IFS= read -r skills_list
case "$skills_list" in *'"method":"skills/list"'*) ;; *) exit 88 ;; esac
printf '%s\n' '{"id":3,"result":{"data":[{"skills":[{"name":"agc-browser-playtest"},{"name":"agc-client-projection"},{"name":"agc-project-structure"},{"name":"agc-web-game-development"},{"name":"taonier-art-assets"}],"errors":[]}]}}'
printf '%s\n' '{"id":3,"result":{"data":[{"skills":[{"name":"agc-browser-playtest"},{"name":"agc-client-projection"},{"name":"agc-game-production-workflow"},{"name":"agc-project-structure"},{"name":"agc-web-game-development"},{"name":"taonier-art-assets"}],"errors":[]}]}}'
while IFS= read -r line; do :; done
"#,
)
File diff suppressed because one or more lines are too long
@@ -1156,6 +1156,10 @@ mod tests {
assert!(with_canvas.contains("根据当前玩法需求编写规格和界面建议"));
assert!(with_canvas.contains("用途、数量、输出路径、尺寸、参考资源和是否需要 spritesheet"));
assert!(with_canvas.contains("再调用 canvas.asset_generate"));
assert!(with_canvas.contains("调用 canvas.asset_generate"));
assert!(with_canvas.contains("不要使用固定图片合同"));
assert!(with_canvas.contains("不修改 game/index.html"));
assert!(!with_canvas.contains("不调用 canvas.asset_generate"));
}
#[test]
@@ -801,7 +801,7 @@ fn agent_runtime_action_receipt_safe_detail_with_owner(
let initial_step = route.get("initialStep")?.as_str()?;
let render_mode = route.get("renderMode")?.as_str()?;
if resource_id.is_empty()
|| initial_step != "visual-binding"
|| initial_step != "asset-separation"
|| render_mode != "final-preview"
{
return None;
@@ -6,15 +6,16 @@ use std::path::{Component, Path};
const AGC_SKILL_PACK_MANIFEST: &[u8] = include_bytes!("../../resources/agc-skills/manifest.json");
const AGC_SKILL_PACK_SCHEMA_VERSION: &str = "agc-skill-pack.v1";
pub(crate) const AGC_SKILL_PACK_EXPECTED_NAMES: [&str; 5] = [
pub(crate) const AGC_SKILL_PACK_EXPECTED_NAMES: [&str; 6] = [
"agc-browser-playtest",
"agc-client-projection",
"agc-game-production-workflow",
"agc-project-structure",
"agc-web-game-development",
"taonier-art-assets",
];
const AGC_SKILL_PACK_FILES: [(&str, &[u8]); 15] = [
const AGC_SKILL_PACK_FILES: [(&str, &[u8]); 18] = [
(
"agc-browser-playtest/SKILL.md",
include_bytes!("../../resources/agc-skills/agc-browser-playtest/SKILL.md"),
@@ -43,6 +44,20 @@ const AGC_SKILL_PACK_FILES: [(&str, &[u8]); 15] = [
"../../resources/agc-skills/agc-client-projection/references/projection-contract.md"
),
),
(
"agc-game-production-workflow/SKILL.md",
include_bytes!("../../resources/agc-skills/agc-game-production-workflow/SKILL.md"),
),
(
"agc-game-production-workflow/agents/openai.yaml",
include_bytes!("../../resources/agc-skills/agc-game-production-workflow/agents/openai.yaml"),
),
(
"agc-game-production-workflow/references/workflow-contract.md",
include_bytes!(
"../../resources/agc-skills/agc-game-production-workflow/references/workflow-contract.md"
),
),
(
"agc-project-structure/SKILL.md",
include_bytes!("../../resources/agc-skills/agc-project-structure/SKILL.md"),
@@ -291,10 +306,10 @@ mod tests {
use super::*;
#[test]
fn bundled_skill_pack_is_exactly_the_five_reviewed_skills() {
fn bundled_skill_pack_is_exactly_the_six_reviewed_skills() {
let manifest = validated_skill_pack_manifest().expect("validated manifest");
assert_eq!(manifest.schema_version, "agc-skill-pack.v1");
assert_eq!(manifest.skills.len(), 5);
assert_eq!(manifest.skills.len(), 6);
assert!(manifest.skills.iter().all(|entry| entry.sha256.len() == 64));
let serialized = serde_json::to_string(
&manifest
@@ -339,6 +339,41 @@ async fn recognize_ui(
ui_editor::commands::recognize_ui_impl(project_path, state).await
}
#[tauri::command]
async fn separate_ui(
project_path: String,
asset_id: String,
state: ui_editor::state::State,
) -> Result<ui_editor::commands::SeparationDTO, String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.register")?;
ui_editor::commands::separate_ui_impl(project_path, asset_id, state).await
}
#[tauri::command]
fn inspect_separation_recovery(
project_path: String,
asset_id: String,
) -> Result<ui_editor::commands::SeparationRecoveryDTO, String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.list")?;
ui_editor::commands::separation::inspect_separation_recovery(root, &asset_id)
}
#[tauri::command]
fn finalize_separation(project_path: String, asset_id: String) -> Result<(), String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.register")?;
ui_editor::commands::separation::finalize_separation(root, &asset_id)
}
#[tauri::command]
fn discard_separation_recovery(project_path: String, asset_id: String) -> Result<(), String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.register")?;
ui_editor::commands::separation::discard_separation_recovery(root, &asset_id)
}
#[tauri::command]
async fn merge_ui(state: ui_editor::state::State) -> Result<ui_editor::commands::MergeDTO, String> {
ui_editor::commands::merge_ui_impl(state).await
@@ -2689,6 +2724,10 @@ fn main() {
check_ui_editor_font_glyph_coverage,
suggest_ui_design_semantic,
recognize_ui,
separate_ui,
inspect_separation_recovery,
finalize_separation,
discard_separation_recovery,
merge_ui,
bind_components,
load_ui_design_state,
@@ -5,11 +5,9 @@ use crate::ui_editor::commands::utils::{
strict_json_schema,
};
use crate::ui_editor::component::text::FontSource;
use crate::ui_editor::component::Component;
use crate::ui_editor::component::{Component, NodeComponent};
use crate::ui_editor::layout::node::{Node, StageStatus};
use crate::ui_editor::persistence::{
UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE, UI_DESIGN_STATE_MAX_NODES,
};
use crate::ui_editor::persistence::UI_DESIGN_STATE_MAX_NODES;
use crate::ui_editor::state::State;
use crate::ui_editor::utils::{FontAssetId, NodeId, SpriteAssetId};
use platform_llm::{
@@ -31,10 +29,10 @@ const SYSTEM_PROMPT: &str = r#"
你是游戏 UI 组件绑定器。你会看到全部 UI 参考图、可编辑节点说明,以及本批独立素材的真实像素。
* 只对视觉上确实需要改变组件的节点返回 changes;
* 每个 change 的 components 是该节点完整的新渲染栈,空数组表示明确清空。数组顺序从底到顶渲染
* 对每个 Component,直接完整返回其全部参数.
* 每个 change 的 component 是该节点完整的新组件;纯结构节点返回 "PureNode",有组件返回 {"WithComponent": <完整 Component>}
* 对 Component,直接完整返回其全部参数.
* 有任何困难或者不确定把状态设为 NeedReview,说明中文原因。
* 纯结构节点可以返回空数组并标为 NoProblem。
* 纯结构节点可以返回 "PureNode" 并标为 NoProblem。
* 容器背景等推荐使用Simple + preserve_aspect: false 实现与node大小一致
* 面向用户的 reason 使用中文。
@@ -53,8 +51,8 @@ enum DraftStatus {
#[schemars(deny_unknown_fields)]
struct BindingChangeDraft {
node_id: NodeId,
components: Vec<Component>,
components_status: DraftStatus,
component: NodeComponent,
component_status: DraftStatus,
}
#[derive(Clone, Debug, Deserialize, JsonSchema)]
@@ -68,8 +66,8 @@ struct BindingResponse {
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct BindingChange {
pub node_id: NodeId,
pub components: Vec<Component>,
pub components_status: StageStatus,
pub component: NodeComponent,
pub component_status: StageStatus,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
@@ -83,7 +81,7 @@ struct EditableNodeContext<'a> {
node_id: &'a NodeId,
name: &'a str,
description: &'a str,
components: &'a [Component],
component: Option<&'a Component>,
}
#[derive(Debug, Serialize)]
@@ -106,7 +104,7 @@ fn collect_editable_nodes<'a>(node: &'a Node, output: &mut Vec<EditableNodeConte
node_id: &node.id,
name: &node.metadata.name,
description: &node.metadata.description,
components: &node.components,
component: node.component.as_ref(),
});
}
for child in &node.children {
@@ -172,14 +170,21 @@ fn validate_binding_response_shape(
return Err(format!("组件绑定 changes 不能超过 {max_changes}"));
}
for change in changes {
let components = change
.get("components")
.and_then(serde_json::Value::as_array)
.ok_or_else(|| "组件绑定 change 缺少 components 数组".to_string())?;
if components.len() > UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE {
return Err(format!(
"单个组件绑定栈不能超过 {UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE} 个组件"
));
let Some(object) = change.as_object() else {
return Err("组件绑定 change 缺少 component 字段".to_string());
};
let Some(component) = object.get("component") else {
return Err("组件绑定 change 缺少 component 字段".to_string());
};
let valid_component = component == "PureNode"
|| component
.as_object()
.and_then(|value| value.get("WithComponent"))
.is_some_and(serde_json::Value::is_object);
if !valid_component {
return Err(
"组件绑定 change 的 component 必须是 PureNode 或 WithComponent 对象".to_string(),
);
}
}
Ok(())
@@ -212,7 +217,7 @@ fn validate_and_materialize(
if !changed_ids.insert(change.node_id.clone()) {
return Err(format!("组件绑定重复返回节点:{}", change.node_id.as_str()));
}
for component in &change.components {
if let NodeComponent::WithComponent(component) = &change.component {
match component {
Component::Image(image) => {
if image
@@ -232,18 +237,22 @@ fn validate_and_materialize(
}
}
}
let components = change.components;
let components_status = match change.components_status {
let component_status = match change.component_status {
DraftStatus::NoProblem => StageStatus::NoProblem,
DraftStatus::NeedReview(reason) if reason.trim().is_empty() => {
return Err("组件待审状态必须包含原因".to_string())
}
DraftStatus::NeedReview(reason) => StageStatus::NeedReview(reason),
DraftStatus::NeedReview(reason) => {
if matches!(&change.component, NodeComponent::PureNode) {
return Err("纯结构节点不能标记为组件待审".to_string());
}
StageStatus::NeedReview(reason)
}
};
materialized.push(BindingChange {
node_id: change.node_id,
components,
components_status,
component: change.component,
component_status,
});
}
Ok(BindingDTO {
@@ -440,8 +449,8 @@ mod tests {
]);
let unapproved = BindingChangeDraft {
node_id: id("other"),
components: Vec::new(),
components_status: DraftStatus::NoProblem,
component: NodeComponent::PureNode,
component_status: DraftStatus::NoProblem,
};
assert!(
validate_and_materialize(vec![unapproved], &editable, &known, &HashSet::new()).is_err()
@@ -450,7 +459,7 @@ mod tests {
// References to sprites from another batch are allowed once they exist in the project.
let other_batch = BindingChangeDraft {
node_id: id("editable"),
components: vec![Component::Image(
component: NodeComponent::WithComponent(Component::Image(
crate::ui_editor::component::image::ImageComponent {
target_graphic: Some(
SpriteAssetId::new("other-batch-sprite").expect("valid sprite"),
@@ -459,8 +468,8 @@ mod tests {
preserve_aspect: false,
},
},
)],
components_status: DraftStatus::NoProblem,
)),
component_status: DraftStatus::NoProblem,
};
assert!(
validate_and_materialize(vec![other_batch], &editable, &known, &HashSet::new()).is_ok()
@@ -469,15 +478,15 @@ mod tests {
// References to sprites that do not exist in the project at all are still rejected.
let unknown = BindingChangeDraft {
node_id: id("editable"),
components: vec![Component::Image(
component: NodeComponent::WithComponent(Component::Image(
crate::ui_editor::component::image::ImageComponent {
target_graphic: Some(SpriteAssetId::new("unknown").expect("valid sprite")),
image_type: crate::ui_editor::component::image::ImageType::Simple {
preserve_aspect: false,
},
},
)],
components_status: DraftStatus::NoProblem,
)),
component_status: DraftStatus::NoProblem,
};
assert!(
validate_and_materialize(vec![unknown], &editable, &known, &HashSet::new()).is_err()
@@ -491,8 +500,8 @@ mod tests {
text.font = FontSource::Bound(FontAssetId::new("unknown-font").expect("valid font"));
let change = BindingChangeDraft {
node_id: id("editable"),
components: vec![Component::Text(text)],
components_status: DraftStatus::NoProblem,
component: NodeComponent::WithComponent(Component::Text(text)),
component_status: DraftStatus::NoProblem,
};
let error = validate_and_materialize(
@@ -506,13 +515,13 @@ mod tests {
}
#[test]
fn materialization_preserves_changed_only_empty_component_lists() {
fn materialization_preserves_pure_node_change() {
let editable = HashSet::from([id("editable")]);
let result = validate_and_materialize(
vec![BindingChangeDraft {
node_id: id("editable"),
components: Vec::new(),
components_status: DraftStatus::NoProblem,
component: NodeComponent::PureNode,
component_status: DraftStatus::NoProblem,
}],
&editable,
&HashSet::new(),
@@ -520,8 +529,28 @@ mod tests {
)
.expect("valid changed-only clear");
assert_eq!(result.changes.len(), 1);
assert!(result.changes[0].components.is_empty());
assert_eq!(result.changes[0].components_status, StageStatus::NoProblem);
assert!(matches!(
result.changes[0].component,
NodeComponent::PureNode
));
assert_eq!(result.changes[0].component_status, StageStatus::NoProblem);
}
#[test]
fn materialization_rejects_problematic_pure_node() {
let editable = HashSet::from([id("editable")]);
let error = validate_and_materialize(
vec![BindingChangeDraft {
node_id: id("editable"),
component: NodeComponent::PureNode,
component_status: DraftStatus::NeedReview("缺少可确认的组件".to_string()),
}],
&editable,
&HashSet::new(),
&HashSet::new(),
)
.expect_err("pure node cannot carry a component review status");
assert!(error.contains("纯结构节点"));
}
#[test]
@@ -575,20 +604,26 @@ mod tests {
}
#[test]
fn binding_response_bounds_changes_and_each_component_stack() {
fn binding_response_bounds_changes_and_uses_single_component_shape() {
let too_many_changes = serde_json::json!({
"changes": [{"components": []}, {"components": []}]
"changes": [{"component": "PureNode"}, {"component": "PureNode"}]
});
assert!(validate_binding_response_shape(&too_many_changes, 1).is_err());
let too_many_components = serde_json::json!({
let one_component = serde_json::json!({
"changes": [{
"components": (0..=UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE)
.map(|_| serde_json::Value::Null)
.collect::<Vec<_>>()
"node_id": "editable",
"component": "PureNode",
"component_status": "NoProblem"
}]
});
assert!(validate_binding_response_shape(&too_many_components, 1).is_err());
assert!(validate_binding_response_shape(&one_component, 1).is_ok());
let parsed = parse_binding_response(&one_component.to_string(), 1)
.expect("explicit PureNode payload should parse");
assert!(matches!(
parsed.changes[0].component,
NodeComponent::PureNode
));
}
#[tokio::test]
@@ -174,7 +174,7 @@ mod materialize {
};
use crate::ui_editor::layout::transform::Transform;
use crate::ui_editor::state::{State, UITree};
use crate::ui_editor::utils::{NodeId, UIDesignImageId};
use crate::ui_editor::utils::{random_node_id, NodeId, UIDesignImageId};
use std::collections::{HashMap, HashSet};
#[derive(Clone)]
@@ -230,12 +230,11 @@ mod materialize {
Ok(best_index)
}
fn random_node_id(occupied: &mut HashSet<NodeId>) -> Result<NodeId, String> {
fn unique_random_node_id(occupied: &mut HashSet<NodeId>) -> NodeId {
loop {
let id = NodeId::new(uuid::Uuid::new_v4().simple().to_string())
.map_err(|error| format!("生成合并容器节点 ID 失败:{error}"))?;
let id = random_node_id();
if occupied.insert(id.clone()) {
return Ok(id);
return id;
}
}
}
@@ -286,7 +285,7 @@ mod materialize {
}
Ok(BuiltNode {
node: LayoutNode {
id: random_node_id(occupied_ids)?,
id: unique_random_node_id(occupied_ids),
layout:
crate::ui_editor::layout::control_layout::ControlLayout::with_transform(
original_transform,
@@ -295,12 +294,12 @@ mod materialize {
name: container_name,
description: container_description,
layout_status: StageStatus::NoProblem,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::Llm,
},
components: Vec::new(),
component: None,
children_display_mode: ChildrenDisplayMode::Exclusive,
children: members.into_iter().map(|member| member.node).collect(),
},
@@ -525,7 +524,6 @@ mod tests {
materialize, validate_merge_input_state, validate_merge_plan_shape, MAX_MERGE_INPUT_DEPTH,
MAX_MERGE_INPUT_NODES, MAX_MERGE_PLAN_DEPTH, MAX_MERGE_PLAN_NODES,
};
use crate::ui_editor::component::Component;
use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode;
use crate::ui_editor::layout::control_layout::ControlLayout;
use crate::ui_editor::layout::node::{Node, NodeMetadata, NodeSource, StageStatus};
@@ -542,12 +540,12 @@ mod tests {
name: id.to_string(),
description: String::new(),
layout_status: StageStatus::NoProblem,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::Human,
},
components: Vec::<Component>::new(),
component: None,
children_display_mode: ChildrenDisplayMode::Stack,
children,
}
@@ -586,7 +584,7 @@ mod tests {
ChildrenDisplayMode::Exclusive
);
assert_eq!(
result.root.metadata.components_status,
result.root.metadata.component_status,
StageStatus::NoProblem
);
assert_eq!(result.root.children.len(), 2);
@@ -1,6 +1,7 @@
pub mod binding;
pub mod merge;
pub mod recognition;
pub mod separation;
pub mod ui_design_suggestion;
pub mod utils;
@@ -10,5 +11,7 @@ pub use merge::MergeDTO;
pub(crate) use merge::{merge_ui_impl, merge_ui_impl_with_provider};
pub use recognition::RecognitionDTO;
pub(crate) use recognition::{recognize_ui_impl, recognize_ui_impl_with_provider};
pub(crate) use separation::separate_ui_impl;
pub use separation::{SeparationDTO, SeparationRecoveryDTO};
pub(crate) use ui_design_suggestion::suggest_ui_design_semantic_impl;
pub use ui_design_suggestion::UIDesignSuggestionTreeNode;
@@ -4,6 +4,7 @@ use crate::ui_editor::commands::utils::{
parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm,
strict_json_schema,
};
use crate::ui_editor::component::{Component, NodeComponent};
use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode;
use crate::ui_editor::layout::control_layout::ControlLayout;
use crate::ui_editor::layout::dimension::UIRect;
@@ -11,7 +12,7 @@ use crate::ui_editor::layout::node::{Node as LayoutNode, NodeMetadata, NodeSourc
use crate::ui_editor::layout::transform::Transform;
use crate::ui_editor::resource::ui_design_image::UIDesignImage;
use crate::ui_editor::state::{State, UITree};
use crate::ui_editor::utils::{NodeId, UIDesignImageId};
use crate::ui_editor::utils::{random_node_id, NodeId, UIDesignImageId};
use nalgebra::{Point2, Vector2};
use platform_llm::{
LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice,
@@ -27,25 +28,31 @@ const MAX_RECOGNITION_TREE_NODES: usize = 512;
const MAX_RECOGNITION_TREE_DEPTH: usize = 32;
const SYSTEM_PROMPT: &str = r#"
角色:
你是游戏 UI 多图结构识别器。
任务:
同时分析同一 UI 系统的全部参考图,建立UI树
用户会给你一些UI截图(它们从属于同一个UI系统)和对应的元数据, 请用给定的工具描述UI结构
识别规则:
* 只识别 UI,不识别场景人物、地形、建筑、光影和背景装饰。
* 只识别 UI元素. 要区分动态内容, 不要白费力气识别应该由程序生成/绘制的内容.(此类内容应该用一个整体节点+自然语言描述) 除此之外必须完整包含所有元素,结构.
* 无法确定类型、层级、关系时,在 UnSure 中写明原因。
* 返回的 trees 必须与输入图片一一对应,每张输入图片只能有一棵树,不能合并多张图片的树。 每棵树的 src_ui_design_image_id 必须等于对应输入图片标注的 id。
* 每棵树root的 global_pos_x_px、global_pos_y_px、width_px、height_px、local_anchor 仅为占位并会被忽略,给合法值即可.
* 每棵树必须使用自己的输入图片原始像素坐标系(0,0 as left top)输出
global_pos_x_px、global_pos_y_px、width_px、height_px
* 为了响应式布局, 我们提供了类似godot的Anchor参数, 可以使用语义化的预设或者可custom的直接操作min max, 请准确地根据父子布局的关系使用
* 面向用户的字段如名称描述等请用中文
* 由于每个截图未必是完整的, 可能是局部的, 每棵树描述清楚每个截图上UI的层次结构即可
* 不同树的共用框架/层次/...请使用使用相同的名称描述. 不同状态/变体名称使用相同的前缀, 用后缀区别
* 粒度要求: 尽可能细致, 最小单元举例: 进度条的底槽、填充和外框; slider的底槽, dragger等
* 粒度要求: 尽可能细致, 以可交互,方便程序化控制的最小单位为准. 包括不限于: icon, 进度条的底槽、填充和外框; slider的底槽, dragger等.
* 为每个节点直接返回 component.
无背景的逻辑容器返回 "PureNode",不要返回 null.
有背景的容器推荐使用Simple+不锁定宽高比的Image component.
目前我们只做识别, 不要求图片字体的具体绑定参数.
文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别.
* 多行文本只使用一个节点.
* 不鼓励兄弟节点相互重叠.
* 对于面板等容器的背景等, 必须作为父节点的组件, 禁止新增冗余的所谓"背景节点". 例如:对于全局的背景直接作为root节点的图片组件, 禁止另外添加节点
"#;
@@ -109,13 +116,14 @@ struct RecognitionNode {
description: String,
children: Vec<RecognitionNode>,
confidence: Confidence,
component: NodeComponent,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
#[schemars(deny_unknown_fields)]
struct RecognitionTree {
src_ui_design_image_id: UIDesignImageId,
children: Vec<RecognitionNode>,
root: RecognitionNode,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
@@ -143,10 +151,14 @@ fn validate_recognition_response_shape(value: &serde_json::Value) -> Result<(),
return Err(format!("识别结果最多包含 {MAX_REFERENCES} 棵界面树"));
}
for tree in trees {
let children = tree
let root = tree
.get("root")
.and_then(serde_json::Value::as_object)
.ok_or_else(|| "识别树缺少 root 节点".to_string())?;
let children = root
.get("children")
.and_then(serde_json::Value::as_array)
.ok_or_else(|| "识别树缺少 children 数组".to_string())?;
.ok_or_else(|| "识别树根节点缺少 children 数组".to_string())?;
let mut stack = children
.iter()
.map(|node| (node, 1usize))
@@ -215,11 +227,6 @@ fn anchor_ranges(anchor: &Anchor) -> Result<(Vector2<f32>, Vector2<f32>), String
Ok((min, max))
}
fn random_node_id() -> Result<NodeId, String> {
NodeId::new(uuid::Uuid::new_v4().simple().to_string())
.map_err(|error| format!("生成节点 ID 失败:{error}"))
}
fn image_layout_size(image: &UIDesignImage) -> Result<Vector2<f32>, String> {
let pixels_per_unit = image.pixels_per_unit.get();
if !pixels_per_unit.is_finite() || pixels_per_unit <= 0.0 {
@@ -301,21 +308,18 @@ fn convert_node(
.map(|child| convert_node(child, image_id, image, target_rect))
.collect::<Result<Vec<_>, _>>()?;
Ok(LayoutNode {
id: random_node_id()?,
id: random_node_id(),
layout: ControlLayout::with_transform(transform),
metadata: NodeMetadata {
name: source.name.clone(),
description: source.description.clone(),
layout_status: status,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::Llm,
},
// V1 有意把识别结果限定为“结构草稿”:组件绑定属于后续独立阶段。
// 因此空组件不是丢失数据,而是等待 visual-binding 阶段补齐 Image/Text。
// 约定见 docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md。
components: Vec::new(),
component: source.component.clone().into_option(),
children_display_mode: ChildrenDisplayMode::Stack,
children,
})
@@ -323,6 +327,26 @@ fn convert_node(
fn validate_confidence(nodes: &[RecognitionNode]) -> Result<(), String> {
for node in nodes {
if let NodeComponent::WithComponent(component) = &node.component {
if matches!(
component,
Component::Image(crate::ui_editor::component::image::ImageComponent {
target_graphic: Some(_),
..
})
) {
return Err("识别阶段不能返回已绑定的 SpriteAssetId".to_string());
}
if matches!(
component,
Component::Text(crate::ui_editor::component::text::TextComponent {
font: crate::ui_editor::component::text::FontSource::Bound(_),
..
})
) {
return Err("识别阶段不能返回已绑定的字体素材".to_string());
}
}
if let Confidence::UnSure(reason) = &node.confidence {
if reason.trim().is_empty() {
return Err("UnSure 必须包含审阅原因".to_string());
@@ -346,7 +370,7 @@ fn validate_tree_image_ids(
if !seen.insert(tree.src_ui_design_image_id.clone()) {
return Err("LLM 为同一界面图返回了重复 UI 树".to_string());
}
validate_confidence(&tree.children)?;
validate_confidence(std::slice::from_ref(&tree.root))?;
}
if seen.len() != allowed.len() {
return Err("LLM 未为当前识别上下文的每张界面图返回 UI 树".to_string());
@@ -387,6 +411,7 @@ mod tests {
description: String::new(),
children: Vec::new(),
confidence: Confidence::Confident,
component: NodeComponent::PureNode,
}
}
@@ -425,8 +450,8 @@ mod tests {
.collect::<Vec<_>>();
let response = serde_json::json!({
"trees": [
{"children": leaves.clone()},
{"children": leaves}
{"root": {"children": leaves.clone()}},
{"root": {"children": leaves}}
]
});
validate_recognition_response_shape(&response)
@@ -439,7 +464,7 @@ mod tests {
.map(|_| serde_json::json!({"children": []}))
.collect::<Vec<_>>();
assert!(validate_recognition_response_shape(&serde_json::json!({
"trees": [{"children": oversized}]
"trees": [{"root": {"children": oversized}}]
}))
.is_err());
@@ -448,7 +473,7 @@ mod tests {
nested = serde_json::json!({"children": [nested]});
}
assert!(validate_recognition_response_shape(&serde_json::json!({
"trees": [{"children": [nested]}]
"trees": [{"root": {"children": [nested]}}]
}))
.is_err());
}
@@ -490,6 +515,31 @@ mod tests {
description: String::new(),
children: Vec::new(),
confidence: Confidence::UnSure(String::new()),
component: NodeComponent::PureNode,
};
assert!(validate_confidence(&[node]).is_err());
}
#[test]
fn recognition_rejects_bound_font_references() {
let mut text = crate::ui_editor::component::text::TextComponent::default();
text.font = crate::ui_editor::component::text::FontSource::Bound(
crate::ui_editor::utils::FontAssetId::new("font").expect("valid font id"),
);
let node = RecognitionNode {
global_pos_x_px: 0,
global_pos_y_px: 0,
width_px: 1,
height_px: 1,
local_anchor: Anchor::Preset(PresetAnchor {
horizontal: HorizontalAnchor::Left,
vertical: VerticalAnchor::Top,
}),
name: "文本".to_string(),
description: String::new(),
children: Vec::new(),
confidence: Confidence::Confident,
component: NodeComponent::WithComponent(Component::Text(text)),
};
assert!(validate_confidence(&[node]).is_err());
}
@@ -506,7 +556,7 @@ mod tests {
converted.layout.transform.resolve(&root_rect),
UIRect::new(Point2::new(50.0, 25.0), Vector2::new(100.0, 50.0)),
);
assert_eq!(converted.metadata.components_status, StageStatus::NoProblem);
assert_eq!(converted.metadata.component_status, StageStatus::NoProblem);
}
#[test]
@@ -542,7 +592,7 @@ mod tests {
let slave = UIDesignImageId::new("slave").expect("valid image id");
let tree = |id: UIDesignImageId| RecognitionTree {
src_ui_design_image_id: id,
children: Vec::new(),
root: test_node(),
};
assert!(validate_tree_image_ids(
@@ -775,24 +825,37 @@ pub(crate) async fn recognize_ui_impl_with_provider(
.ok_or_else(|| format!("缺少界面图 {} 的识别树", image_id.as_str()))?;
let size = image_layout_size(image)?;
let root_rect = UIRect::new(Point2::origin(), size);
let children = tree
// The recognition root is a real UI node. Its pixel geometry and
// anchor are intentionally ignored; the page root always fills
// the design image while the other recognition fields take effect.
let recognition_root = tree.root;
let children = recognition_root
.children
.iter()
.map(|node| convert_node(node, &image_id, image, root_rect))
.collect::<Result<Vec<_>, _>>()?;
let root_layout_status = match recognition_root.confidence {
Confidence::Confident => StageStatus::NoProblem,
Confidence::UnSure(reason) => StageStatus::NeedReview(reason),
};
let root_name = if recognition_root.name.trim().is_empty() {
"页面根节点".to_string()
} else {
recognition_root.name
};
let root = LayoutNode {
id: random_node_id()?,
id: random_node_id(),
layout: ControlLayout::with_transform(Transform::stretch()),
metadata: NodeMetadata {
name: "页面根节点".to_string(),
description: String::new(),
layout_status: StageStatus::NoProblem,
components_status: StageStatus::NoProblem,
name: root_name,
description: recognition_root.description,
layout_status: root_layout_status,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::System,
source: NodeSource::Llm,
},
components: Vec::new(),
component: recognition_root.component.into_option(),
children_display_mode: ChildrenDisplayMode::Stack,
children,
};
@@ -0,0 +1,480 @@
use super::model::BindingArea;
use image::RgbaImage;
use std::time::Instant;
/// Each edge may move by at most this many pixels from the area returned by
/// the visual model. Keep this policy explicit so changing it is an
/// intentional workflow decision rather than a scattered numeric literal.
pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX: u32 = 32;
/// Alpha values below this threshold are treated as transparent for boundary
/// detection. The cropped pixels themselves are preserved unchanged.
pub(crate) const MIN_VISIBLE_ALPHA: u8 = 16;
/// An edge needs this many consecutive visible pixels to count as supported.
/// The requirement is reduced to the edge length for one-pixel-wide elements.
pub(crate) const MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS: usize = 2;
#[derive(Clone, Copy, Debug, PartialEq)]
pub(crate) struct NormalizedBindingArea {
pub(crate) area: BindingArea,
pub(crate) changed: bool,
pub(crate) clamped: bool,
pub(crate) transparent: bool,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum EdgeDirection {
Inward,
Outward,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct Rect {
left: u32,
top: u32,
right: u32,
bottom: u32,
}
impl Rect {
fn from_area(area: BindingArea) -> Self {
Self {
left: area.global_pos_x_px,
top: area.global_pos_y_px,
right: area.global_pos_x_px + area.width_px,
bottom: area.global_pos_y_px + area.height_px,
}
}
fn into_area(self) -> BindingArea {
BindingArea {
global_pos_x_px: self.left,
global_pos_y_px: self.top,
width_px: self.right - self.left,
height_px: self.bottom - self.top,
}
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum Edge {
Left,
Right,
Top,
Bottom,
}
impl Edge {
const ALL: [Self; 4] = [Self::Left, Self::Right, Self::Top, Self::Bottom];
}
fn pixel_is_visible(alpha: u8) -> bool {
alpha >= MIN_VISIBLE_ALPHA
}
fn has_consecutive_visible_pixels<I>(alphas: I, required: usize) -> bool
where
I: IntoIterator<Item = u8>,
{
let required = required.max(1);
let mut consecutive = 0usize;
for alpha in alphas {
if pixel_is_visible(alpha) {
consecutive = consecutive.saturating_add(1);
if consecutive >= required {
return true;
}
} else {
consecutive = 0;
}
}
false
}
fn edge_has_visible_pixel(image: &RgbaImage, rect: Rect, edge: Edge) -> bool {
let edge_length = match edge {
Edge::Left | Edge::Right => rect.bottom - rect.top,
Edge::Top | Edge::Bottom => rect.right - rect.left,
} as usize;
let required = MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS.max(1).min(edge_length);
match edge {
Edge::Left | Edge::Right => {
let x = if edge == Edge::Left {
rect.left
} else {
rect.right - 1
};
has_consecutive_visible_pixels(
(rect.top..rect.bottom).map(|y| image.get_pixel(x, y).0[3]),
required,
)
}
Edge::Top | Edge::Bottom => {
let y = if edge == Edge::Top {
rect.top
} else {
rect.bottom - 1
};
has_consecutive_visible_pixels(
(rect.left..rect.right).map(|x| image.get_pixel(x, y).0[3]),
required,
)
}
}
}
fn rect_has_visible_pixel(image: &RgbaImage, rect: Rect) -> bool {
(rect.top..rect.bottom)
.any(|y| (rect.left..rect.right).any(|x| pixel_is_visible(image.get_pixel(x, y).0[3])))
}
fn edge_direction(image: &RgbaImage, rect: Rect, edge: Edge) -> EdgeDirection {
if edge_has_visible_pixel(image, rect, edge) {
EdgeDirection::Outward
} else {
EdgeDirection::Inward
}
}
fn move_edge(rect: &mut Rect, edge: Edge, direction: EdgeDirection) {
match (edge, direction) {
(Edge::Left, EdgeDirection::Inward) => rect.left += 1,
(Edge::Left, EdgeDirection::Outward) => rect.left -= 1,
(Edge::Right, EdgeDirection::Inward) => rect.right -= 1,
(Edge::Right, EdgeDirection::Outward) => rect.right += 1,
(Edge::Top, EdgeDirection::Inward) => rect.top += 1,
(Edge::Top, EdgeDirection::Outward) => rect.top -= 1,
(Edge::Bottom, EdgeDirection::Inward) => rect.bottom -= 1,
(Edge::Bottom, EdgeDirection::Outward) => rect.bottom += 1,
}
}
fn edge_coordinate(rect: Rect, edge: Edge) -> u32 {
match edge {
Edge::Left => rect.left,
Edge::Right => rect.right,
Edge::Top => rect.top,
Edge::Bottom => rect.bottom,
}
}
fn edge_displacement(original: Rect, current: Rect, edge: Edge) -> u32 {
edge_coordinate(original, edge).abs_diff(edge_coordinate(current, edge))
}
fn edge_adjustment_limit() -> u32 {
MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX
}
fn reached_adjustment_limit(original: Rect, current: Rect, edge: Edge) -> bool {
edge_displacement(original, current, edge) >= edge_adjustment_limit()
}
fn can_move_geometrically(
image: &RgbaImage,
current: Rect,
edge: Edge,
direction: EdgeDirection,
) -> bool {
match (edge, direction) {
(Edge::Left, EdgeDirection::Inward) => current.left + 1 < current.right,
(Edge::Left, EdgeDirection::Outward) => current.left > 0,
(Edge::Right, EdgeDirection::Inward) => current.right > current.left + 1,
(Edge::Right, EdgeDirection::Outward) => current.right < image.width(),
(Edge::Top, EdgeDirection::Inward) => current.top + 1 < current.bottom,
(Edge::Top, EdgeDirection::Outward) => current.top > 0,
(Edge::Bottom, EdgeDirection::Inward) => current.bottom > current.top + 1,
(Edge::Bottom, EdgeDirection::Outward) => current.bottom < image.height(),
}
}
fn next_edge_rect(rect: Rect, edge: Edge, direction: EdgeDirection) -> Option<Rect> {
let mut next = rect;
match (edge, direction) {
(Edge::Left, EdgeDirection::Inward) if rect.left + 1 < rect.right => next.left += 1,
(Edge::Left, EdgeDirection::Outward) if rect.left > 0 => next.left -= 1,
(Edge::Right, EdgeDirection::Inward) if rect.right > rect.left + 1 => next.right -= 1,
(Edge::Right, EdgeDirection::Outward) => next.right = next.right.checked_add(1)?,
(Edge::Top, EdgeDirection::Inward) if rect.top + 1 < rect.bottom => next.top += 1,
(Edge::Top, EdgeDirection::Outward) if rect.top > 0 => next.top -= 1,
(Edge::Bottom, EdgeDirection::Inward) if rect.bottom > rect.top + 1 => next.bottom -= 1,
(Edge::Bottom, EdgeDirection::Outward) => next.bottom = next.bottom.checked_add(1)?,
_ => return None,
}
Some(next)
}
fn edge_requires_move(image: &RgbaImage, rect: Rect, edge: Edge, direction: EdgeDirection) -> bool {
match direction {
EdgeDirection::Inward => !edge_has_visible_pixel(image, rect, edge),
EdgeDirection::Outward => {
if !edge_has_visible_pixel(image, rect, edge) {
return false;
}
if !can_move_geometrically(image, rect, edge, direction) {
return true;
}
next_edge_rect(rect, edge, direction)
.is_some_and(|next| edge_has_visible_pixel(image, next, edge))
}
}
}
fn apply_edge_step(
image: &RgbaImage,
original: Rect,
current: Rect,
edge: Edge,
direction: EdgeDirection,
) -> (Rect, bool, bool) {
if !edge_requires_move(image, current, edge, direction) {
return (current, false, false);
}
if reached_adjustment_limit(original, current, edge)
|| !can_move_geometrically(image, current, edge, direction)
{
return (current, false, true);
}
let mut next = current;
move_edge(&mut next, edge, direction);
(next, true, false)
}
/// Normalizes a model-provided area using visible pixels on the processed
/// transparent image. Each edge chooses inward/outward direction once from
/// its initial scan and then moves monotonically, so sparse pixels cannot make
/// the boundary oscillate. The four edge steps are calculated from the same
/// rectangle on each round.
pub(crate) fn normalize_binding_area(
image: &RgbaImage,
original_area: BindingArea,
) -> Result<NormalizedBindingArea, String> {
let started = Instant::now();
if let Err(error) = original_area.validate_in(image.width(), image.height()) {
app_log!(
"ui_separation.area.timing outcome=error elapsed_us={} rounds=0 image_width={} image_height={} area=({}, {}, {}, {})",
started.elapsed().as_micros(),
image.width(),
image.height(),
original_area.global_pos_x_px,
original_area.global_pos_y_px,
original_area.width_px,
original_area.height_px
);
return Err(error.to_string());
}
let original = Rect::from_area(original_area);
let directions = Edge::ALL.map(|edge| edge_direction(image, original, edge));
let mut current = original;
let mut clamped = false;
let mut active = [true; 4];
let mut rounds = 0u32;
// TODO: Replace the deliberately simple pixel-by-pixel scan if real UI
// design sizes show this path to be a measurable bottleneck.
while active.iter().any(|value| *value) {
rounds = rounds.saturating_add(1);
let before = current;
let mut next = current;
let mut moved = [false; 4];
for (index, edge) in Edge::ALL.into_iter().enumerate() {
if !active[index] {
continue;
}
let (candidate, did_move, reached_limit) =
apply_edge_step(image, original, current, edge, directions[index]);
if reached_limit {
clamped = true;
active[index] = false;
} else if !did_move {
active[index] = false;
}
moved[index] = did_move;
match edge {
Edge::Left => next.left = candidate.left,
Edge::Right => next.right = candidate.right,
Edge::Top => next.top = candidate.top,
Edge::Bottom => next.bottom = candidate.bottom,
}
}
if next.left >= next.right {
clamped = true;
if moved[0] {
active[0] = false;
}
if moved[1] {
active[1] = false;
}
next.left = current.left;
next.right = current.right;
}
if next.top >= next.bottom {
clamped = true;
if moved[2] {
active[2] = false;
}
if moved[3] {
active[3] = false;
}
next.top = current.top;
next.bottom = current.bottom;
}
current = next;
if current == before {
break;
}
}
let area = current.into_area();
let normalized = NormalizedBindingArea {
changed: area != original_area,
area,
clamped,
transparent: !rect_has_visible_pixel(image, current),
};
app_log!(
"ui_separation.area.timing outcome=ok elapsed_us={} rounds={} image_width={} image_height={} area=({}, {}, {}, {}) changed={} clamped={} transparent={}",
started.elapsed().as_micros(),
rounds,
image.width(),
image.height(),
original_area.global_pos_x_px,
original_area.global_pos_y_px,
original_area.width_px,
original_area.height_px,
normalized.changed,
normalized.clamped,
normalized.transparent
);
Ok(normalized)
}
#[cfg(test)]
mod tests {
use super::*;
use image::{Rgba, RgbaImage};
fn image_with_rect(
width: u32,
height: u32,
left: u32,
top: u32,
right: u32,
bottom: u32,
) -> RgbaImage {
let mut image = RgbaImage::from_pixel(width, height, Rgba([0, 0, 0, 0]));
for y in top..bottom {
for x in left..right {
image.put_pixel(x, y, Rgba([255, 255, 255, 255]));
}
}
image
}
fn area(x: u32, y: u32, width: u32, height: u32) -> BindingArea {
BindingArea {
global_pos_x_px: x,
global_pos_y_px: y,
width_px: width,
height_px: height,
}
}
#[test]
fn shrinks_empty_edges_to_visible_bounds() {
let image = image_with_rect(32, 32, 10, 11, 16, 18);
let result = normalize_binding_area(&image, area(6, 7, 14, 16)).unwrap();
assert_eq!(result.area, area(10, 11, 6, 7));
assert!(result.changed);
assert!(!result.clamped);
assert!(!result.transparent);
}
#[test]
fn expands_visible_edges_to_cover_the_element() {
let image = image_with_rect(32, 32, 10, 11, 16, 18);
let result = normalize_binding_area(&image, area(11, 12, 4, 5)).unwrap();
assert_eq!(result.area, area(10, 11, 6, 7));
assert!(result.changed);
assert!(!result.clamped);
assert!(!result.transparent);
}
#[test]
fn adjusts_each_edge_independently() {
let image = image_with_rect(32, 32, 10, 11, 16, 18);
let result = normalize_binding_area(&image, area(10, 12, 10, 3)).unwrap();
assert_eq!(result.area, area(10, 11, 6, 7));
}
#[test]
fn ignores_low_alpha_halo_while_preserving_visible_bounds() {
let mut image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0]));
for y in 6..10 {
for x in 5..9 {
image.put_pixel(x, y, Rgba([255, 255, 255, 255]));
}
}
image.put_pixel(4, 7, Rgba([255, 255, 255, 1]));
image.put_pixel(9, 8, Rgba([255, 255, 255, 8]));
let result = normalize_binding_area(&image, area(4, 5, 6, 6)).unwrap();
assert_eq!(result.area, area(5, 6, 4, 4));
}
#[test]
fn ignores_isolated_visible_edge_pixel() {
let mut image = image_with_rect(16, 16, 4, 4, 6, 8);
image.put_pixel(6, 4, Rgba([255, 255, 255, 255]));
let result = normalize_binding_area(&image, area(4, 4, 2, 4)).unwrap();
assert_eq!(result.area, area(4, 4, 2, 4));
}
#[test]
fn fully_transparent_image_uses_the_same_path() {
let image = RgbaImage::from_pixel(32, 32, Rgba([0, 0, 0, 0]));
let result = normalize_binding_area(&image, area(10, 10, 10, 10)).unwrap();
assert_eq!(result.area, area(14, 14, 2, 2));
assert!(result.changed);
assert!(result.transparent);
}
#[test]
fn caps_each_edge_at_absolute_pixel_limit() {
let image = image_with_rect(128, 128, 0, 0, 128, 128);
let result = normalize_binding_area(&image, area(48, 48, 8, 8)).unwrap();
assert_eq!(result.area, area(16, 16, 72, 72));
assert!(result.clamped);
}
#[test]
fn clamps_expansion_to_image_edges() {
let image = image_with_rect(16, 16, 0, 0, 4, 4);
let result = normalize_binding_area(&image, area(1, 1, 2, 2)).unwrap();
assert_eq!(result.area, area(0, 0, 4, 4));
assert!(result.clamped);
}
#[test]
fn exact_split_at_adjustment_limit_is_not_clamped() {
let image = image_with_rect(16, 16, 4, 4, 8, 8);
let result = normalize_binding_area(&image, area(5, 5, 2, 2)).unwrap();
assert_eq!(result.area, area(4, 4, 4, 4));
assert!(!result.clamped);
}
#[test]
fn one_pixel_area_expands_with_configured_adjustment_limit() {
let image = image_with_rect(8, 8, 2, 2, 5, 5);
let result = normalize_binding_area(&image, area(3, 3, 1, 1)).unwrap();
assert_eq!(result.area, area(2, 2, 3, 3));
assert!(!result.clamped);
}
#[test]
fn rejects_zero_sized_or_out_of_bounds_model_areas() {
let image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0]));
assert!(normalize_binding_area(&image, area(0, 0, 0, 1)).is_err());
assert!(normalize_binding_area(&image, area(15, 15, 2, 2)).is_err());
}
}
@@ -0,0 +1,123 @@
use super::area::MIN_VISIBLE_ALPHA;
use base64::Engine as _;
use image::{ImageFormat, ImageReader, Rgba, RgbaImage};
use std::fs;
use std::io::Cursor;
use std::path::{Path, PathBuf};
pub(crate) const VISUAL_BINDING_TRANSPARENT_MARKER_RGBA: [u8; 4] = [255, 0, 255, 255];
const MAX_PROCESSED_IMAGE_BYTES: usize = 64 * 1024 * 1024;
const MAX_PROCESSED_IMAGE_DIMENSION: u32 = 2880;
pub(crate) async fn preprocess_for_visual_binding(
processed_url: String,
sidecar: PathBuf,
) -> Result<String, String> {
tokio::task::spawn_blocking(move || {
preprocess_for_visual_binding_blocking(&processed_url, &sidecar)
})
.await
.map_err(|error| format!("视觉绑定预处理任务失败:{error}"))?
}
fn preprocess_for_visual_binding_blocking(
processed_url: &str,
sidecar: &Path,
) -> Result<String, String> {
let encoded = processed_url
.split_once(',')
.map(|(_, data)| data)
.ok_or_else(|| "处理图 data URL 无效".to_string())?;
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded.trim())
.map_err(|error| format!("解析处理图失败:{error}"))?;
if bytes.len() > MAX_PROCESSED_IMAGE_BYTES {
return Err(format!(
"处理图超过 {} MiB 字节上限",
MAX_PROCESSED_IMAGE_BYTES / 1024 / 1024
));
}
let dimensions = ImageReader::new(Cursor::new(&bytes))
.with_guessed_format()
.map_err(|error| format!("解析处理图格式失败:{error}"))?
.into_dimensions()
.map_err(|error| format!("读取处理图尺寸失败:{error}"))?;
if dimensions.0 > MAX_PROCESSED_IMAGE_DIMENSION || dimensions.1 > MAX_PROCESSED_IMAGE_DIMENSION
{
return Err("处理图尺寸超出上限".to_string());
}
let mut image = image::load_from_memory(&bytes)
.map_err(|error| format!("解码处理图失败:{error}"))?
.to_rgba8();
for pixel in image.pixels_mut() {
if pixel.0[3] < MIN_VISIBLE_ALPHA {
*pixel = Rgba(VISUAL_BINDING_TRANSPARENT_MARKER_RGBA);
} else {
pixel.0[3] = 255;
}
}
let mut png = Vec::new();
image::DynamicImage::ImageRgba8(image)
.write_to(&mut Cursor::new(&mut png), ImageFormat::Png)
.map_err(|error| format!("编码视觉绑定预览失败:{error}"))?;
let debug_name = format!("binding-{}.png", uuid::Uuid::new_v4().simple());
if let Err(error) = fs::write(sidecar.join(&debug_name), &png) {
app_log!(
"ui_separation.warning stage=visual_binding_preview_write file={} error={error}",
debug_name
);
}
Ok(format!(
"data:image/png;base64,{}",
base64::engine::general_purpose::STANDARD.encode(png)
))
}
#[cfg(test)]
mod tests {
use super::*;
use image::Rgba;
use tempfile::tempdir;
fn data_url(image: RgbaImage) -> String {
let mut bytes = Vec::new();
image::DynamicImage::ImageRgba8(image)
.write_to(&mut Cursor::new(&mut bytes), ImageFormat::Png)
.expect("encode fixture");
format!(
"data:image/png;base64,{}",
base64::engine::general_purpose::STANDARD.encode(bytes)
)
}
#[test]
fn preprocesses_alpha_using_existing_visibility_threshold() {
let mut image = RgbaImage::from_pixel(4, 1, Rgba([10, 20, 30, 255]));
image.put_pixel(0, 0, Rgba([1, 2, 3, 0]));
image.put_pixel(1, 0, Rgba([4, 5, 6, MIN_VISIBLE_ALPHA - 1]));
image.put_pixel(2, 0, Rgba([7, 8, 9, MIN_VISIBLE_ALPHA]));
image.put_pixel(3, 0, Rgba([11, 12, 13, 254]));
let directory = tempdir().expect("create sidecar fixture");
let url = preprocess_for_visual_binding_blocking(&data_url(image), directory.path())
.expect("preprocess fixture");
let encoded = url.split_once(',').expect("data URL").1;
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded)
.expect("decode output");
let output = image::load_from_memory(&bytes)
.expect("decode output png")
.to_rgba8();
assert_eq!(
output.get_pixel(0, 0).0,
VISUAL_BINDING_TRANSPARENT_MARKER_RGBA
);
assert_eq!(
output.get_pixel(1, 0).0,
VISUAL_BINDING_TRANSPARENT_MARKER_RGBA
);
assert_eq!(output.get_pixel(2, 0).0, [7, 8, 9, 255]);
assert_eq!(output.get_pixel(3, 0).0, [11, 12, 13, 255]);
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,97 @@
use crate::ui_editor::utils::NodeId;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum BindingAreaValidationError {
ZeroDimension,
OutOfBounds,
}
impl std::fmt::Display for BindingAreaValidationError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str(match self {
Self::ZeroDimension => "BindingArea 宽度和高度必须大于 0",
Self::OutOfBounds => "BindingArea 超出处理图边界",
})
}
}
impl std::error::Error for BindingAreaValidationError {}
#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize, JsonSchema)]
#[schemars(deny_unknown_fields)]
pub struct BindingArea {
pub global_pos_x_px: u32,
pub global_pos_y_px: u32,
pub width_px: u32,
pub height_px: u32,
}
impl BindingArea {
pub fn validate_in(&self, w: u32, h: u32) -> Result<(), BindingAreaValidationError> {
if self.width_px == 0 || self.height_px == 0 {
return Err(BindingAreaValidationError::ZeroDimension);
}
if self
.global_pos_x_px
.checked_add(self.width_px)
.is_none_or(|v| v > w)
|| self
.global_pos_y_px
.checked_add(self.height_px)
.is_none_or(|v| v > h)
{
return Err(BindingAreaValidationError::OutOfBounds);
}
Ok(())
}
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
#[schemars(deny_unknown_fields)]
pub enum BindingDecision {
Ok {
extracted_area: BindingArea,
to_node: NodeId,
},
NeedRework {
advice: String,
to_node: NodeId,
},
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
pub struct BindingResp {
pub decisions: Vec<BindingDecision>,
}
#[cfg(test)]
mod tests {
use super::{BindingArea, BindingAreaValidationError};
#[test]
fn validates_binding_area_with_typed_errors() {
let zero = BindingArea {
global_pos_x_px: 0,
global_pos_y_px: 0,
width_px: 0,
height_px: 1,
};
assert_eq!(
zero.validate_in(10, 10),
Err(BindingAreaValidationError::ZeroDimension)
);
let outside = BindingArea {
global_pos_x_px: 10,
global_pos_y_px: 0,
width_px: 1,
height_px: 1,
};
assert_eq!(
outside.validate_in(10, 10),
Err(BindingAreaValidationError::OutOfBounds)
);
}
}

Some files were not shown because too many files have changed in this diff Show More