Compare commits

...

126 Commits

Author SHA1 Message Date
k88936 c86221e3da 添加 TODO 注释以提示恢复字符限制错误信息
Project CI / Repository checks (pull_request) Failing after 15s
Project CI / Backend tests (pull_request) Failing after 15s
Project CI / Frontend tests (pull_request) Successful in 3m3s
Project CI / Native shell tests (pull_request) Failing after 14m57s
2026-09-11 20:12:16 +08:00
k88936 e7af7cf4e4 校验处理图区域并明确 multipart 字节上传
将处理图真实尺寸传入视觉绑定校验,提前拒绝零尺寸和越界区域。

保持非法区域直接失败,不转换为 NeedRework。

将 image 和 mask multipart 部分改为 Part::bytes。
2026-09-11 20:11:11 +08:00
k88936 e7cd008d7c 调整文本组件的识别规则
简化多行文本节点逻辑,明确文字与图片组件的绑定要求。
2026-09-11 20:06:09 +08:00
k88936 c743ca146e 复用图像下载的共享字节缓冲
将 DownloadedImage.bytes 改为 bytes::Bytes。

让远程下载和图标切片下游避免不必要的完整复制。

同步更新图像处理实现与测试 fixture。
2026-09-11 19:53:14 +08:00
k88936 8c8b6cc75a 将自动切分状态写盘移出 Tokio worker
用 spawn_blocking 承载 JSON 序列化和文件写入,并在工作流中等待结果。
2026-09-11 19:44:01 +08:00
k88936 4cf3642b53 说明视觉绑定解析错误的有限重试策略
记录严格工具 schema 和响应损坏重试边界。
2026-09-11 19:41:48 +08:00
k88936 f933db169e 说明结构化重试的有限上下文边界
记录现役调用方的小重试次数和保留完整反馈的设计依据。
2026-09-11 19:41:21 +08:00
k88936 a5e936dd27 清理已退役的前端绑定模块
删除无现役导入的 binding.ts 及其 BindingDTO、BindingChange 类型。
2026-09-11 19:39:59 +08:00
k88936 53008959c5 修正自动切分结果的类型收窄
在异步闭包赋值后明确已完成结果的非空类型。
2026-09-11 19:09:22 +08:00
k88936 57dbb3de52 替换已有组件前增加确认
使用独立确认弹窗避免一次点击覆盖已配置组件。
2026-09-11 19:07:12 +08:00
k88936 e189aff485 更新设计基础提示词契约测试
将测试断言同步到当前按需图片生成与动态产物合同。
2026-09-11 19:05:57 +08:00
k88936 248c094da9 清理合并命令测试中的无用导入
移除组件类型未再使用的测试模块导入。
2026-09-11 19:05:03 +08:00
k88936 e66eb2fb55 切换预览图片时自动适配画布
在活动图片变化后重新按完整画布边界计算视口。
2026-09-11 19:04:33 +08:00
k88936 d455dc18a9 按实际保存快照更新脏状态签名
保存成功后直接记录持久化快照签名,避免闭包中的旧编辑器状态阻止清脏。
2026-09-11 19:03:51 +08:00
k88936 04a0d21351 避免原始 PNG 的重复解析
使用解码后的颜色类型校验位深,将读取限制统一留在 image 解码器。
2026-09-11 19:03:03 +08:00
k88936 a9358bf3ce 防止字体前置校验访问空值
对外部状态中的空字体引用返回正常校验结果而不是抛出异常。
2026-09-11 19:00:49 +08:00
k88936 c3de2ce958 为自动切分命令增加项目写权限校验
在启动分离工作流前复用 asset.register 权限门禁。
2026-09-11 19:00:19 +08:00
k88936 287d2f94da 限制自动切分恢复状态读取大小
在解析 sidecar state 前检查文件大小,避免异常分配。
2026-09-11 18:59:51 +08:00
k88936 1343f487d0 为自动切分工作流增加忙碌保护
统一直接调用入口的并发防护,避免状态锁冲突。
2026-09-11 18:59:10 +08:00
k88936 73fb89a8f4 阻止本地素材登记结果错位回填
资源导入返回数量与请求路径不一致时立即中止位置映射。
2026-09-11 18:58:35 +08:00
k88936 bf2ffd2bf9 拒绝保存无可见像素的切分区域
透明裁切区域直接返回可恢复错误,避免登记空素材。
2026-09-11 18:58:13 +08:00
k88936 3f55f043a1 拒绝切分产物路径中的非法节点标识
在写入 cut 图片前限制节点 ID 字符并确认路径仍位于 sidecar 内。
2026-09-11 18:57:19 +08:00
k88936 26677b193c 复用界面树组件计数值
在同一行中复用组件数量,避免重复计算。
2026-09-11 18:56:13 +08:00
k88936 b8691add3a 移除区域归一化日志路径中的恐慌
直接记录归一化结果,避免诊断日志调用 expect。
2026-09-11 18:55:28 +08:00
k88936 bfe010a192 让视觉绑定调试预览写入失败可降级
调试 PNG 写入失败时记录警告并继续绑定流程。
2026-09-11 18:55:06 +08:00
k88936 6db6de5b99 同步检查器组件面板展开状态
切换节点或组件时重置展开状态,避免复用旧节点的折叠状态。
2026-09-11 18:54:39 +08:00
k88936 87a7294a04 细化组件绑定载荷形状校验
在反序列化前拒绝空值和无效的节点组件外部枚举。
2026-09-11 18:54:21 +08:00
k88936 3774d7c1e1 移除修复重试流程中的隐式恐慌
在循环意外结束时返回可恢复错误而不是触发 unreachable。
2026-09-11 18:53:34 +08:00
k88936 8d15125f12 限制切分提示词中的节点备注
对用户备注清理控制字符和围栏符号并限制长度。
2026-09-11 18:52:59 +08:00
k88936 cf68b8212d 让提取提示词序列化错误可恢复
将提取提示词生成改为返回带上下文的错误。
2026-09-11 18:52:10 +08:00
k88936 8d7ccc0d75 简化工作流操作卡状态映射
使用步骤映射替代嵌套三元表达式。
2026-09-11 18:51:30 +08:00
k88936 2af3d9a965 修正结构识别步骤的前置校验
将结构识别活动步骤改为校验素材切分前置条件。
2026-09-11 18:50:59 +08:00
k88936 4749339bad 补齐图片字节类型导出与转换
Project CI / Repository checks (pull_request) Failing after 31s
Project CI / Backend tests (pull_request) Failing after 13s
Project CI / Frontend tests (pull_request) Successful in 3m56s
Project CI / Native shell tests (pull_request) Failing after 3m33s
导出共享 GPT 图片尺寸阈值常量。

修复编辑器参考图 Bytes 与 Vec 边界转换。
2026-09-11 18:02:04 +08:00
k88936 3e85c71060 整理图片代理代码格式
按 Rust 格式化规范整理 multipart 与响应解析代码。
2026-09-11 17:55:50 +08:00
k88936 da19bde99a 同步八位 PNG 与 b64 契约
记录八位 PNG 校验规则与稳定错误文案。

说明 GPT-Image-2 原生 b64_json,不发送 response_format。
2026-09-11 17:52:26 +08:00
k88936 4a32ee37c4 修复 Bytes 迁移后的图片类型边界
统一使用 Bytes 的只读引用并在旧 Vec 接口处显式转换。

修复编辑器参考图对齐与 Raw PNG 解码编译错误。
2026-09-11 17:52:00 +08:00
k88936 31fd1da495 修正 GPT 图片响应契约说明
仅保留 GPT-Image-2 原生 b64_json 行为的行内注释。

不发送 response_format 参数,也不引入 URL 兼容下载。
2026-09-11 17:48:55 +08:00
k88936 4c88e5e924 限制 Raw 图片仅接收八位 PNG
使用 png 解码器读取位深并在 image 解码前拒绝非八位输入。

统一返回稳定的 8-bit PNG 客户端错误并补充十六位 PNG 测试。
2026-09-11 17:48:01 +08:00
k88936 99f4961c5f 统一 GPT 图片尺寸计费阈值
提取共享的 1536 长边 tier 常量并复用到 raw 与编辑器计费。

同步 Raw 图片代理技术合同中的阈值来源。
2026-09-11 17:47:31 +08:00
k88936 0cebc39b6a 同步 Raw 图片代理契约
记录文本字段 16KiB 流式限制。

记录 GPT-Image-2 固定 b64_json 响应格式。
2026-09-11 17:37:58 +08:00
k88936 842d080a75 锁定图片缓冲依赖
将 platform-image 使用的 bytes 依赖写入 Cargo.lock。
2026-09-11 17:37:32 +08:00
k88936 8cd6e34af0 调整图片切分批次请求尺寸计算逻辑
基于批次源矩形总面积动态计算 image-edit 请求尺寸,新增面积对齐和最小/最大值限制,扩展相关模块和测试以支持调整后的工作流。
2026-09-11 17:33:26 +08:00
k88936 b357d36262 明确 GPT 图片编辑响应格式
原始图片编辑请求固定声明 response_format=b64_json。

补充 GPT-Image-2 仅返回 b64_json 的契约行内注释。
2026-09-11 17:29:11 +08:00
k88936 48c5638893 扩大原始图片文本字段上限
multipart 文本字段改为 chunk 流式读取并限制 16KB。

统一 UTF-8 校验和超限错误,避免先完整分配超长 prompt。
2026-09-11 17:28:13 +08:00
k88936 46df12c086 收敛原始图片 multipart 错误
服务端记录解析原始错误,客户端仅返回稳定分类文案。
2026-09-11 17:26:33 +08:00
k88936 4bdbab29c0 优化原始图片请求缓冲
将 ReferenceImage 与 multipart 输入改用 bytes::Bytes。

通过流式 multipart part 避免 reqwest 路径的额外整图复制。
2026-09-11 17:25:25 +08:00
k88936 9acc23b848 抽离 DFS 批次逻辑至独立模块,并优化节点面积计算与终端过滤流程
Project CI / Repository checks (pull_request) Failing after 15s
Project CI / Backend tests (pull_request) Failing after 15s
Project CI / Frontend tests (pull_request) Successful in 8m16s
Project CI / Native shell tests (pull_request) Failing after 11m7s
2026-09-11 15:34:30 +08:00
k88936 b9b4ebd38e 重命名图片编辑为提取模块并更新相关调用绘制流程
Project CI / Repository checks (pull_request) Failing after 16s
Project CI / Backend tests (pull_request) Failing after 15s
Project CI / Native shell tests (pull_request) Failing after 4m22s
Project CI / Frontend tests (pull_request) Successful in 11m36s
2026-09-11 15:18:05 +08:00
k88936 8f680ef376 视觉绑定增加透明标记预处理
新增视觉绑定 RGBA 预处理与阈值复用

在请求前生成不透明洋红标记图并记录耗时

更新英文提示词说明标记色语义
2026-09-11 15:09:44 +08:00
k88936 7c7d55dcab 重构提取提示词生成逻辑,移除文本拼接方式,新增 YAML 序列化结构与测试 2026-09-11 14:54:00 +08:00
k88936 1303592a86 重构UI编辑器素材切分工作流模块
Project CI / Repository checks (pull_request) Failing after 14s
Project CI / Backend tests (pull_request) Failing after 15s
Project CI / Frontend tests (pull_request) Successful in 5m17s
Project CI / Native shell tests (pull_request) Failing after 6m41s
拆分提取与绑定提示词模块

拆分图片编辑、视觉绑定、裁切和批次 patch 实现

同步技术方案实现组织说明并更新提示词测试
2026-09-11 13:54:33 +08:00
k88936 1db7597141 重构UI素材切分树与批次选择
移除文本遮罩和标记图生成流程,图片编辑直接读取原始界面图

引入节点类型并保留文字移除上下文,按前序DFS与面积预算选择图片批次

补充根节点、重叠节点、文字过滤和超面积推进测试并同步TypeScript类型
2026-09-11 12:55:53 +08:00
k88936 13701d0144 澄清UI自动切分的原图输入与调试产物
明确图片编辑直接使用原始界面图,不再生成辅助输入图

明确处理图与裁切图使用随机文件名并保留用于调试
2026-09-11 11:56:33 +08:00
k88936 5c83142fec 更新UI编辑器自动切分素材工作流方案
将多轮整图分层与DFS面积批次写入权威专题

明确TextRemovalOnly、视觉绑定和atlas输出合同

删除平行专题文档并保留后续TODO

补充UI编辑器自动分离多轮分层方案

新增整图语义分层与多轮 image-edit 方案

明确 DFS 批次、TextRemovalOnly 和视觉绑定合同

记录面积预算、恢复边界与后续 TODO
2026-09-11 11:10:51 +08:00
k88936 95ed49f264 调整素材切分边界像素上限
将视觉绑定区域每条边的调整上限改为固定 32px。

同步 UI 编辑器自动切分技术方案与项目决策记录。

更新边界测试以覆盖绝对像素限制。
2026-09-11 10:28:32 +08:00
k88936 b8d8072267 完善识别命令说明,新增禁止冗余背景节点的指引
Project CI / Repository checks (pull_request) Failing after 9s
Project CI / Backend tests (pull_request) Failing after 10s
Project CI / Frontend tests (pull_request) Successful in 2m44s
Project CI / Native shell tests (pull_request) Failing after 3m5s
2026-09-10 19:54:46 +08:00
k88936 462caa33c5 修复工作流完成弹窗的多行错误展示
让完成通知中的换行字符按实际换行渲染,避免多项回填错误挤成一行。

保持通知内容和弹窗交互不变。
2026-09-10 19:06:20 +08:00
k88936 4e1064266f 移除自动切分结果的冗余类型断言
依赖空值检查后的 TypeScript 类型收窄直接使用 SeparationDTO。

不改变自动切分流程或运行时行为。
2026-09-10 19:05:48 +08:00
k88936 a50cbfb6c8 过滤图片编辑响应中的空图像数据
忽略上游返回的空 b64_json 条目,避免把空字符串转发为图像结果。

新增空条目与有效条目混合响应的回归测试。
2026-09-10 19:04:40 +08:00
k88936 1eb4bd5ec0 明确拒绝无效的处理图数据地址
处理图缺少 data URL 分隔符时直接返回格式错误。

避免先写入空文件再在后续裁切阶段暴露误导性错误。
2026-09-10 19:03:29 +08:00
k88936 5b441d389c 忽略完全位于图片外的切分标记矩形
矩形起点超出图片边界时直接返回 None,避免在边缘绘制伪影。

新增越界矩形回归测试。
2026-09-10 19:02:40 +08:00
k88936 f200cb4d3c 避免标记图编码重复复制像素缓冲
复用同一个 DynamicImage 完成文件保存和返回数据编码。

保持标记图内容和错误处理不变。
2026-09-10 19:02:09 +08:00
k88936 ae2f5fc80b 记录 UI 编辑器重试中的请求错误
在结构化 LLM 请求的中间重试失败时写入尝试次数和错误日志。

保留原有重试次数、历史消息和最终错误语义。
2026-09-10 19:01:32 +08:00
k88936 b69bd88e5e 修正 UI 编辑器校验器命名
将 run_with_repair_history 的 Validater/validater 更正为 Validator/validator。

不改变重试流程和运行时行为。
2026-09-10 19:01:18 +08:00
k88936 c5c7339c4a 重命名分离模块 problem_descriptionadvice 并同步更新相关校验逻辑与测试
Project CI / Repository checks (pull_request) Failing after 13s
Project CI / Backend tests (pull_request) Failing after 9s
Project CI / Frontend tests (pull_request) Successful in 2m28s
Project CI / Native shell tests (pull_request) Failing after 14m32s
2026-09-10 18:54:10 +08:00
k88936 4bbc77fc32 增加UI自动切分耗时日志
补充 marker、area、LLM、图片编辑及 batch 阶段耗时观测
记录成功与失败路径的性能字段
同步自动切分工作流技术方案中的日志约定
2026-09-10 18:51:53 +08:00
k88936 8c521ca891 移除 UI 预览适配与缩放输入
删除预览工具栏适配按钮、适配回调与对应快捷键
将缩放百分比改为只读显示并移除输入组件及测试
2026-09-10 18:48:56 +08:00
k88936 69a0804c03 调整切分区域边界扩展上限
将像素边界每边调整上限常量设为百分之百

同步更新受影响的区域归一化测试

技术方案仅引用上限常量,不重复写死数值
2026-09-10 18:46:39 +08:00
k88936 74f726e65d 修正自动切分素材检查条件
移除自动切分对预先导入独立素材的错误要求。
统一自动切分前置与结果检查,并补齐图片和字体引用校验。
在新建和重新开始自动切分时执行前置检查,恢复流程继续尽力完成。
2026-09-10 18:43:31 +08:00
k88936 1509235c1b 统一自动切分素材工作流命名
将 UI 编辑器第三步和内部流程 ID 统一为自动切分素材。
重命名前端切分概览模块并更新完成、恢复和导航文案。
同步 Runtime 最终编辑器路由、测试与相关技术文档。
2026-09-10 15:11:29 +08:00
k88936 1c992c69fe 完善分离与识别命令说明,调整嵌套格式与结构化指引 2026-09-10 14:05:37 +08:00
k88936 d9b153ed78 完善分离命令说明,新增禁止未请求文本的指引 2026-09-10 13:15:42 +08:00
k88936 30c469e9a1 规范分离模块格式化写法 2026-09-10 13:15:34 +08:00
k88936 51f629b2f9 UI编辑器节点改为单组件模型
Project CI / Repository checks (pull_request) Failing after 14s
Project CI / Backend tests (pull_request) Failing after 14s
Project CI / Frontend tests (pull_request) Successful in 2m43s
Project CI / Native shell tests (pull_request) Failing after 14m29s
修改 Node、Binding DTO、识别与分离链路为 component 单字段

为识别和绑定 LLM 工具引入 PureNode/WithComponent 显式载荷

移除前端组件索引、排序和组件栈操作

同步生成 ts-rs TypeScript 类型、状态校验、测试与专题文档
2026-09-10 13:08:29 +08:00
k88936 d6c720ecdd 纠正分离模块导出提交边界
撤销误入的组件字段重命名测试改动

仅保留裁切工作流对面积归一化模块的直接依赖
2026-09-10 12:26:39 +08:00
k88936 086aa91d70 收窄分离区域模块导出
让裁切工作流直接依赖面积归一化模块

移除未使用的公共重导出,保持分离模块边界清晰
2026-09-10 12:24:18 +08:00
k88936 3733e86272 补充分离区域像素归一化约定
记录透明像素边界扫描、方向锁定与四边同步推进规则

明确 50% 与图像边界约束、透明图处理、日志字段和性能 TODO
2026-09-10 12:13:21 +08:00
k88936 5d0f3e6be6 UI编辑器分离区域像素边界归一化
新增独立 BindingArea 像素边界归一化模块与回归测试

在 cut 裁切前按透明像素收缩或扩展四条边并记录约束结果

接入分离工作流并保留原始与归一化区域日志
2026-09-10 12:13:04 +08:00
k88936 989440173f 完善绑定区域值类型约束
为 BindingArea 增加 Eq 以支持分离区域比较
2026-09-10 11:52:46 +08:00
k88936 22d9958d18 在分离标记图中遮罩普通文字
绘制文字紫色区域并保持绿色元素框在最上层

增加文字遮罩图层顺序测试
2026-09-10 10:57:50 +08:00
k88936 56f8c9a6a1 构造自动分离文字遮罩
按最近未绑定图片祖先收集 Text 布局像素矩形

补充嵌套图片与根图片遮罩构造测试
2026-09-10 10:49:45 +08:00
k88936 c84df3d88f 补充文字遮罩状态字段契约
明确 text_mask_areas 为当前 separation state 必选字段

保持开发阶段 v1 且不提供旧 sidecar 迁移回退
2026-09-10 10:49:07 +08:00
k88936 761d32cebf 拆分自动分离模型并增加文字遮罩区域
将 separation model 拆为 note、node、binding、result 模块

为 SeparationNode 增加 text_mask_areas 字段并同步 TypeScript 类型
2026-09-10 10:43:00 +08:00
k88936 b3d6543d73 补充自动分离文字遮罩方案
记录普通文字归属、遮罩字段与标记图绘制规则

明确不进入分离树、绑定结果和视觉模型契约
2026-09-10 10:32:34 +08:00
k88936 8cd7c43b9e 更新自动分离接入文档状态
将前端接入 TODO 更新为联调与失败注入测试待办。
2026-09-09 21:02:09 +08:00
k88936 7fb7d246c0 将视觉步骤切换为自动分离
前端仅调用一次 separate_ui,并完成 cut 图片登记、SpriteAsset 回填和 State 保存。

增加 sidecar 恢复选择面板,旧 bind_components 标记为未调用遗留命令。
2026-09-09 21:00:53 +08:00
k88936 5844afa14d 抽取自动分离素材状态变换
复用 SpriteAsset 校验并支持锁内构造下一份 State。

保留公开 addSpriteAssets 的锁保护与历史提交行为。
2026-09-09 20:17:11 +08:00
k88936 615d1a28cd 补齐自动分离恢复状态胶水
新增 separation 恢复探测、完成清理命令与 DTO。

持久化项目相对 cut 图片路径并保留 state sidecar 到最终提交。
2026-09-09 20:16:09 +08:00
k88936 3e1e172f9c 补充自动分离前端接入合同
明确 UI 编辑器单次调用 separation 与 sidecar 恢复弹窗

补充资源登记、State 回填、保存和 finalize 顺序

划定旧 binding 与 Runtime workflow 的本次不改范围
2026-09-09 20:03:27 +08:00
k88936 53a919e39b 优化 UI 分离工作流程的提示生成与历史处理
Project CI / Repository checks (pull_request) Failing after 14s
Project CI / Backend tests (pull_request) Failing after 14s
Project CI / Frontend tests (pull_request) Successful in 2m50s
Project CI / Native shell tests (pull_request) Failing after 3m4s
- 调整 initial_history 构建逻辑,增强系统与用户消息的一致性。
- 简化 SeparationNode 提示生成逻辑,仅保留节点 ID 与注释。
- 明确 extracted_area 的权威性为 processed 图像,并更新生成提示的规则。
2026-09-09 19:04:56 +08:00
k88936 d4b5f9fc14 重构 UI Editor LLM repair history
将 request_with_feedback 改为 run_with_repair_history 并统一 max_retries 与 validater 命名。

让 requester 接收 append-only LlmMessage history,业务校验失败追加序列化响应和错误反馈。

迁移 visual_binding 调用方并补充 retry-only 与业务反馈 history 测试。

同步 UI Editor 结构化请求 repair history 决策文档。
2026-09-09 16:46:29 +08:00
k88936 6e4f9a6f1f 完善 UI 自动分离返工意见链路
记录视觉模型 NeedRework 意见并注入后续提取提示

同步返工次数、problematic 终态和 512 字符校验

补充 separation DTO 类型、回归测试与技术方案
2026-09-09 16:29:54 +08:00
k88936 8a0dcf8be7 增强分离标记框的可视辨识度
Project CI / Backend tests (pull_request) Failing after 15s
Project CI / Repository checks (pull_request) Failing after 15s
Project CI / Frontend tests (pull_request) Successful in 3m4s
Project CI / Native shell tests (pull_request) Failing after 4m38s
为分离标记框增加两条绿色角到角交叉线。

新增 MARKER_LINE_WIDTH 常量统一配置标记线宽,并确保绘制结果裁剪在矩形内。

补充标记框像素级回归测试并同步更新自动分离专题文档。
2026-09-09 16:24:11 +08:00
k88936 aa2a78f7e0 重命名 separated_image_area 为 extracted_area,优化命名一致性并明确区域含义 2026-09-09 15:48:32 +08:00
k88936 7aa96e8438 统一分离标记图中文注释
说明紫色重建与绿色框绘制层级
2026-09-09 15:16:39 +08:00
k88936 feb6bc0ad4 修正分离失败恢复边界
图片编辑、写入和裁切失败保留状态并返回错误
仅视觉模型返工结果进入 problematic
2026-09-09 15:07:49 +08:00
k88936 f349350d60 同步分离状态 TypeScript 类型
加入 root_extractable 并将树字段统一为 trees
2026-09-09 15:03:48 +08:00
k88936 df3a172bc9 补充 Raw GPT Image 2 multipart 合同
明确分离图片编辑请求字段与旧协议边界
2026-09-09 14:57:55 +08:00
k88936 9b4924c73e 补充分离批次重叠筛选测试
验证 DFS 顺序下贪心跳过重叠叶节点
2026-09-09 14:54:37 +08:00
k88936 d30bc462c5 格式化自动分离 Rust 模块
统一树选择、标记图和批次流程代码风格
2026-09-09 14:50:19 +08:00
k88936 26b9ad7fad 重构分离树与非重叠批次选择
保留真实根节点并记录 root_extractable
按终态反查和 DFS 贪心筛选逻辑叶
拆出父子区域标记图生成模块
2026-09-09 14:48:18 +08:00
k88936 bc5398894e 补充分离树与批次算法约定
明确真实根节点、逻辑叶和非重叠贪心选择
补充父节点紫色重建与错误恢复边界
2026-09-09 14:29:38 +08:00
k88936 11d4d3d930 适配 UI 分离图片编辑 multipart 接口
Project CI / Frontend tests (pull_request) Successful in 4m57s
Project CI / Repository checks (pull_request) Failing after 10s
Project CI / Backend tests (pull_request) Failing after 11s
Project CI / Native shell tests (pull_request) Failing after 5m40s
将标记图 data URL 解码为 PNG 文件部件。

按 Raw GPT Image 2 新合同发送 prompt、尺寸和输出参数。

保留响应 data[].b64_json 解析与 separation 流程不变。
2026-09-09 13:17:36 +08:00
k88936 418d5732de 调整UI识别指令,细化粒度要求并优化组件返回规则 2026-09-09 13:12:51 +08:00
k88936 592b6b4ee7 简化识别树验证测试中的断言格式 2026-09-09 13:12:51 +08:00
k88936 99c19e3ed9 补充分离流程排障日志
记录 separation sidecar 状态读写与恢复阶段

记录树构造、批次选择、视觉绑定、裁切和 patch 结果

避免输出 base64、完整提示词和敏感凭据
2026-09-09 13:12:51 +08:00
k88936 15e30acb0f 调整UI识别指令细节,明确文字组件识别规则与粒度标准 2026-09-09 13:12:51 +08:00
k88936 56fff8325b 将自动分离图像处理移出异步执行器
在 spawn_blocking 中构建标记图并编码写入处理图

在 spawn_blocking 中完成处理图裁切与文件输出

保留 image-edit 和视觉绑定网络请求的 async 调度边界
2026-09-09 13:12:51 +08:00
k88936 884c5ec5ee 补充自动分离 sidecar 并发边界约束
记录前端单次运行与 runWithStateLocked 的并发前提

明确 sidecar 不参与正式资产写入和项目 revision

补充未来多窗口多进程场景的进程级锁 TODO

记录图像本地处理使用 spawn_blocking 的执行边界
2026-09-09 13:12:51 +08:00
k88936 207b0820dd 统一分离树文件格式
按仓库 rustfmt 规范整理 synthetic root 实现
2026-09-09 13:12:51 +08:00
k88936 19f3d62d78 隔离非法分离区域导致的批次失败
将裁切错误记录为 problematic 并继续流程
避免单个模型区域终止整页分离
2026-09-09 13:12:51 +08:00
k88936 3c1e7bd599 保留分离问题节点的真实重做次数
避免基础设施失败伪造达到上限
保持 problematic 节点诊断信息准确
2026-09-09 13:12:51 +08:00
k88936 dda6b3946a 阻止识别结果引用未授权字体素材
补齐识别阶段字体引用校验
增加伪造字体绑定回归测试
2026-09-09 13:12:51 +08:00
k88936 aff40f794d 修复分离树合成根节点 ID 冲突
避免单图根节点重复进入重做计数
补充根图片构造回归测试
2026-09-09 13:12:50 +08:00
k88936 187faa607d 保留原始分离提示并拆分模块
按模型、树、持久化、提示词和工作流拆分 separation

提取支持 more_turn 的反馈重试 harness

删除旧 separation.rs 文件
2026-09-09 13:12:50 +08:00
k88936 109cc81740 拆分UI自动分离命令模块
保留原始图片分离与绑定提示词

提取模型、树、持久化和工作流子模块

新增可配置 more_turn 反馈重试 harness
2026-09-09 13:12:50 +08:00
k88936 f796aef7e2 完善UI自动分离批次处理
复用资源ID安全路径并修正像素坐标

加入标记图、Raw编辑、视觉绑定修复与问题节点继续处理
2026-09-09 13:12:50 +08:00
k88936 4cc7f4dd64 实现UI自动分离命令
新增分离树、批次处理与sidecar状态

接入Raw GPT Image 2与视觉绑定重试

注册Tauri命令并补充识别组件草稿
2026-09-09 13:12:50 +08:00
k88936 b2d4690f94 补充UI编辑器自动分离工作流合同
新增自动分离、临时 sidecar 与视觉绑定设计合同

明确前端资产登记职责与恢复 TODO
2026-09-09 13:12:50 +08:00
k88936 0400103a19 将 Raw 图片编辑入站改为 multipart
移除 JSON/base64 入站兼容,仅接受 multipart 图片与参数字段

复用扣费前 PNG、尺寸、mask 和 prompt 校验并补充 multipart 测试

优化上游图片字节转发,避免再次克隆 image bytes

同步 Raw 图片编辑技术方案与 Axum multipart 依赖
2026-09-09 13:06:22 +08:00
k88936 c049d68b8e 收紧 Raw 成功响应字段类型
要求 data 与 b64_json 必填并移除无用 provider id

prompt 上限调整为 4 KiB 原始字节并同步文档
2026-09-09 12:03:03 +08:00
k88936 acfcf0c158 同步 platform-image 的 serde 锁定依赖
为 Raw typed 响应记录 serde workspace 依赖
2026-09-09 11:51:25 +08:00
k88936 5bbedd9a5c 限制 Raw prompt 原始字节大小
将 prompt UTF-8 原始字节限制为 16 KiB

补充超限回归测试并同步请求合同
2026-09-09 11:50:30 +08:00
k88936 40316d82c4 在扣费前校验 Raw mask 尺寸
解码时保留 image 与 mask 宽高并拒绝不一致输入

同步 Raw 图片编辑请求合同与回归测试
2026-09-09 11:45:44 +08:00
k88936 6c77de4a36 为 Raw 成功响应收紧类型透传
使用 typed JSON DTO 提取并原样转发 b64_json

不解码图片内容且忽略 output_format 回显
2026-09-09 11:44:31 +08:00
k88936 b3693765ef 透传 Raw 图片结果避免解码回显字段
仅转发上游 data.b64_json 并忽略 output_format

同步 Raw 技术方案并补充不解码回归测试
2026-09-09 11:34:41 +08:00
111 changed files with 4862 additions and 1542 deletions
@@ -108,6 +108,8 @@ const rustSharedContractSource = fs.readFileSync(
);
const allowedUncalledTauriCommands = [
'append_direct_project_conversation_message',
// TODO: Remove the retired binding command after the legacy runtime path is removed.
'bind_components',
'chat_with_game_creator_agent',
'check_ui_editor_font_glyph_coverage',
'create_ui_design_resource',
@@ -1822,27 +1822,11 @@ mod tests {
let with_canvas =
game_creator_design_foundation_tool_plan_prompt("shared runtime contract", true);
assert!(with_canvas.contains("再调用 canvas.asset_generate"));
assert!(with_canvas.contains("assets/art-spec.png"));
assert!(with_canvas.contains("referenceImageSrcs 第一项"));
assert!(with_canvas.contains("assets/ui-prototype.png"));
assert!(with_canvas.contains("调用 image.inspect"));
assert!(with_canvas.contains("ui-prototype.v2"));
assert!(with_canvas.contains("成功只表示候选图片已生成并登记,不等于视觉验收完成"));
assert!(with_canvas.contains("由 Runtime 在收束门内同时核对固定 owner 文档"));
assert!(!with_canvas.contains("成功动作本身就是当前 revision 的验证"));
assert!(with_canvas.contains("调用 ui.workflow.run"));
assert!(with_canvas.contains("visual-binding 最终编辑器路由"));
assert!(with_canvas.contains("每个功能页面各写一行 @genarrative-ui-page"));
assert!(with_canvas.contains("ui.workflow.run 的 discover"));
assert!(with_canvas.contains("assets/ui-pages/{pageId}.png"));
assert!(with_canvas.contains("informationHud"));
assert!(with_canvas.contains("failureRestartFlow"));
assert!(with_canvas.contains("不得假设为塔防"));
assert!(with_canvas
.contains("POST /api/external/v1/editor/images/generationskind=ui-design"));
assert!(with_canvas
.contains("不得误用 POST /api/external/v1/editor/ui-designs/assets/extractions"));
assert!(with_canvas.contains("根据当前玩法需求编写规格和界面建议"));
assert!(with_canvas.contains("调用 canvas.asset_generate"));
assert!(with_canvas.contains("不要使用固定图片合同"));
assert!(with_canvas.contains("不修改 game/index.html"));
assert!(!with_canvas.contains("调用 canvas.asset_generate"));
}
#[test]
@@ -801,7 +801,7 @@ fn agent_runtime_action_receipt_safe_detail_with_owner(
let initial_step = route.get("initialStep")?.as_str()?;
let render_mode = route.get("renderMode")?.as_str()?;
if resource_id.is_empty()
|| initial_step != "visual-binding"
|| initial_step != "asset-separation"
|| render_mode != "final-preview"
{
return None;
@@ -328,6 +328,41 @@ async fn recognize_ui(
ui_editor::commands::recognize_ui_impl(project_path, state).await
}
#[tauri::command]
async fn separate_ui(
project_path: String,
asset_id: String,
state: ui_editor::state::State,
) -> Result<ui_editor::commands::SeparationDTO, String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.register")?;
ui_editor::commands::separate_ui_impl(project_path, asset_id, state).await
}
#[tauri::command]
fn inspect_separation_recovery(
project_path: String,
asset_id: String,
) -> Result<ui_editor::commands::SeparationRecoveryDTO, String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.list")?;
ui_editor::commands::separation::inspect_separation_recovery(root, &asset_id)
}
#[tauri::command]
fn finalize_separation(project_path: String, asset_id: String) -> Result<(), String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.register")?;
ui_editor::commands::separation::finalize_separation(root, &asset_id)
}
#[tauri::command]
fn discard_separation_recovery(project_path: String, asset_id: String) -> Result<(), String> {
let root = Path::new(project_path.trim());
enforce_project_permission_policy(root, "asset.register")?;
ui_editor::commands::separation::finalize_separation(root, &asset_id)
}
#[tauri::command]
async fn merge_ui(state: ui_editor::state::State) -> Result<ui_editor::commands::MergeDTO, String> {
ui_editor::commands::merge_ui_impl(state).await
@@ -2559,6 +2594,10 @@ fn main() {
check_ui_editor_font_glyph_coverage,
suggest_ui_design_semantic,
recognize_ui,
separate_ui,
inspect_separation_recovery,
finalize_separation,
discard_separation_recovery,
merge_ui,
bind_components,
load_ui_design_state,
@@ -5,11 +5,9 @@ use crate::ui_editor::commands::utils::{
strict_json_schema,
};
use crate::ui_editor::component::text::FontSource;
use crate::ui_editor::component::Component;
use crate::ui_editor::component::{Component, NodeComponent};
use crate::ui_editor::layout::node::{Node, StageStatus};
use crate::ui_editor::persistence::{
UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE, UI_DESIGN_STATE_MAX_NODES,
};
use crate::ui_editor::persistence::UI_DESIGN_STATE_MAX_NODES;
use crate::ui_editor::state::State;
use crate::ui_editor::utils::{FontAssetId, NodeId, SpriteAssetId};
use platform_llm::{
@@ -31,10 +29,10 @@ const SYSTEM_PROMPT: &str = r#"
你是游戏 UI 组件绑定器。你会看到全部 UI 参考图、可编辑节点说明,以及本批独立素材的真实像素。
* 只对视觉上确实需要改变组件的节点返回 changes;
* 每个 change 的 components 是该节点完整的新渲染栈,空数组表示明确清空。数组顺序从底到顶渲染
* 对每个 Component,直接完整返回其全部参数.
* 每个 change 的 component 是该节点完整的新组件;纯结构节点返回 "PureNode",有组件返回 {"WithComponent": <完整 Component>}
* 对 Component,直接完整返回其全部参数.
* 有任何困难或者不确定把状态设为 NeedReview,说明中文原因。
* 纯结构节点可以返回空数组并标为 NoProblem。
* 纯结构节点可以返回 "PureNode" 并标为 NoProblem。
* 容器背景等推荐使用Simple + preserve_aspect: false 实现与node大小一致
* 面向用户的 reason 使用中文。
@@ -53,8 +51,8 @@ enum DraftStatus {
#[schemars(deny_unknown_fields)]
struct BindingChangeDraft {
node_id: NodeId,
components: Vec<Component>,
components_status: DraftStatus,
component: NodeComponent,
component_status: DraftStatus,
}
#[derive(Clone, Debug, Deserialize, JsonSchema)]
@@ -68,8 +66,8 @@ struct BindingResponse {
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct BindingChange {
pub node_id: NodeId,
pub components: Vec<Component>,
pub components_status: StageStatus,
pub component: NodeComponent,
pub component_status: StageStatus,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
@@ -83,7 +81,7 @@ struct EditableNodeContext<'a> {
node_id: &'a NodeId,
name: &'a str,
description: &'a str,
components: &'a [Component],
component: Option<&'a Component>,
}
#[derive(Debug, Serialize)]
@@ -106,7 +104,7 @@ fn collect_editable_nodes<'a>(node: &'a Node, output: &mut Vec<EditableNodeConte
node_id: &node.id,
name: &node.metadata.name,
description: &node.metadata.description,
components: &node.components,
component: node.component.as_ref(),
});
}
for child in &node.children {
@@ -172,14 +170,21 @@ fn validate_binding_response_shape(
return Err(format!("组件绑定 changes 不能超过 {max_changes}"));
}
for change in changes {
let components = change
.get("components")
.and_then(serde_json::Value::as_array)
.ok_or_else(|| "组件绑定 change 缺少 components 数组".to_string())?;
if components.len() > UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE {
return Err(format!(
"单个组件绑定栈不能超过 {UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE} 个组件"
));
let Some(object) = change.as_object() else {
return Err("组件绑定 change 缺少 component 字段".to_string());
};
let Some(component) = object.get("component") else {
return Err("组件绑定 change 缺少 component 字段".to_string());
};
let valid_component = component == "PureNode"
|| component
.as_object()
.and_then(|value| value.get("WithComponent"))
.is_some_and(serde_json::Value::is_object);
if !valid_component {
return Err(
"组件绑定 change 的 component 必须是 PureNode 或 WithComponent 对象".to_string(),
);
}
}
Ok(())
@@ -212,7 +217,7 @@ fn validate_and_materialize(
if !changed_ids.insert(change.node_id.clone()) {
return Err(format!("组件绑定重复返回节点:{}", change.node_id.as_str()));
}
for component in &change.components {
if let NodeComponent::WithComponent(component) = &change.component {
match component {
Component::Image(image) => {
if image
@@ -232,18 +237,22 @@ fn validate_and_materialize(
}
}
}
let components = change.components;
let components_status = match change.components_status {
let component_status = match change.component_status {
DraftStatus::NoProblem => StageStatus::NoProblem,
DraftStatus::NeedReview(reason) if reason.trim().is_empty() => {
return Err("组件待审状态必须包含原因".to_string())
}
DraftStatus::NeedReview(reason) => StageStatus::NeedReview(reason),
DraftStatus::NeedReview(reason) => {
if matches!(&change.component, NodeComponent::PureNode) {
return Err("纯结构节点不能标记为组件待审".to_string());
}
StageStatus::NeedReview(reason)
}
};
materialized.push(BindingChange {
node_id: change.node_id,
components,
components_status,
component: change.component,
component_status,
});
}
Ok(BindingDTO {
@@ -440,8 +449,8 @@ mod tests {
]);
let unapproved = BindingChangeDraft {
node_id: id("other"),
components: Vec::new(),
components_status: DraftStatus::NoProblem,
component: NodeComponent::PureNode,
component_status: DraftStatus::NoProblem,
};
assert!(
validate_and_materialize(vec![unapproved], &editable, &known, &HashSet::new()).is_err()
@@ -450,7 +459,7 @@ mod tests {
// References to sprites from another batch are allowed once they exist in the project.
let other_batch = BindingChangeDraft {
node_id: id("editable"),
components: vec![Component::Image(
component: NodeComponent::WithComponent(Component::Image(
crate::ui_editor::component::image::ImageComponent {
target_graphic: Some(
SpriteAssetId::new("other-batch-sprite").expect("valid sprite"),
@@ -459,8 +468,8 @@ mod tests {
preserve_aspect: false,
},
},
)],
components_status: DraftStatus::NoProblem,
)),
component_status: DraftStatus::NoProblem,
};
assert!(
validate_and_materialize(vec![other_batch], &editable, &known, &HashSet::new()).is_ok()
@@ -469,15 +478,15 @@ mod tests {
// References to sprites that do not exist in the project at all are still rejected.
let unknown = BindingChangeDraft {
node_id: id("editable"),
components: vec![Component::Image(
component: NodeComponent::WithComponent(Component::Image(
crate::ui_editor::component::image::ImageComponent {
target_graphic: Some(SpriteAssetId::new("unknown").expect("valid sprite")),
image_type: crate::ui_editor::component::image::ImageType::Simple {
preserve_aspect: false,
},
},
)],
components_status: DraftStatus::NoProblem,
)),
component_status: DraftStatus::NoProblem,
};
assert!(
validate_and_materialize(vec![unknown], &editable, &known, &HashSet::new()).is_err()
@@ -491,8 +500,8 @@ mod tests {
text.font = FontSource::Bound(FontAssetId::new("unknown-font").expect("valid font"));
let change = BindingChangeDraft {
node_id: id("editable"),
components: vec![Component::Text(text)],
components_status: DraftStatus::NoProblem,
component: NodeComponent::WithComponent(Component::Text(text)),
component_status: DraftStatus::NoProblem,
};
let error = validate_and_materialize(
@@ -506,13 +515,13 @@ mod tests {
}
#[test]
fn materialization_preserves_changed_only_empty_component_lists() {
fn materialization_preserves_pure_node_change() {
let editable = HashSet::from([id("editable")]);
let result = validate_and_materialize(
vec![BindingChangeDraft {
node_id: id("editable"),
components: Vec::new(),
components_status: DraftStatus::NoProblem,
component: NodeComponent::PureNode,
component_status: DraftStatus::NoProblem,
}],
&editable,
&HashSet::new(),
@@ -520,8 +529,28 @@ mod tests {
)
.expect("valid changed-only clear");
assert_eq!(result.changes.len(), 1);
assert!(result.changes[0].components.is_empty());
assert_eq!(result.changes[0].components_status, StageStatus::NoProblem);
assert!(matches!(
result.changes[0].component,
NodeComponent::PureNode
));
assert_eq!(result.changes[0].component_status, StageStatus::NoProblem);
}
#[test]
fn materialization_rejects_problematic_pure_node() {
let editable = HashSet::from([id("editable")]);
let error = validate_and_materialize(
vec![BindingChangeDraft {
node_id: id("editable"),
component: NodeComponent::PureNode,
component_status: DraftStatus::NeedReview("缺少可确认的组件".to_string()),
}],
&editable,
&HashSet::new(),
&HashSet::new(),
)
.expect_err("pure node cannot carry a component review status");
assert!(error.contains("纯结构节点"));
}
#[test]
@@ -575,20 +604,26 @@ mod tests {
}
#[test]
fn binding_response_bounds_changes_and_each_component_stack() {
fn binding_response_bounds_changes_and_uses_single_component_shape() {
let too_many_changes = serde_json::json!({
"changes": [{"components": []}, {"components": []}]
"changes": [{"component": "PureNode"}, {"component": "PureNode"}]
});
assert!(validate_binding_response_shape(&too_many_changes, 1).is_err());
let too_many_components = serde_json::json!({
let one_component = serde_json::json!({
"changes": [{
"components": (0..=UI_DESIGN_STATE_MAX_COMPONENTS_PER_NODE)
.map(|_| serde_json::Value::Null)
.collect::<Vec<_>>()
"node_id": "editable",
"component": "PureNode",
"component_status": "NoProblem"
}]
});
assert!(validate_binding_response_shape(&too_many_components, 1).is_err());
assert!(validate_binding_response_shape(&one_component, 1).is_ok());
let parsed = parse_binding_response(&one_component.to_string(), 1)
.expect("explicit PureNode payload should parse");
assert!(matches!(
parsed.changes[0].component,
NodeComponent::PureNode
));
}
#[tokio::test]
@@ -295,12 +295,12 @@ mod materialize {
name: container_name,
description: container_description,
layout_status: StageStatus::NoProblem,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::Llm,
},
components: Vec::new(),
component: None,
children_display_mode: ChildrenDisplayMode::Exclusive,
children: members.into_iter().map(|member| member.node).collect(),
},
@@ -525,7 +525,6 @@ mod tests {
materialize, validate_merge_input_state, validate_merge_plan_shape, MAX_MERGE_INPUT_DEPTH,
MAX_MERGE_INPUT_NODES, MAX_MERGE_PLAN_DEPTH, MAX_MERGE_PLAN_NODES,
};
use crate::ui_editor::component::Component;
use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode;
use crate::ui_editor::layout::control_layout::ControlLayout;
use crate::ui_editor::layout::node::{Node, NodeMetadata, NodeSource, StageStatus};
@@ -542,12 +541,12 @@ mod tests {
name: id.to_string(),
description: String::new(),
layout_status: StageStatus::NoProblem,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::Human,
},
components: Vec::<Component>::new(),
component: None,
children_display_mode: ChildrenDisplayMode::Stack,
children,
}
@@ -586,7 +585,7 @@ mod tests {
ChildrenDisplayMode::Exclusive
);
assert_eq!(
result.root.metadata.components_status,
result.root.metadata.component_status,
StageStatus::NoProblem
);
assert_eq!(result.root.children.len(), 2);
@@ -1,6 +1,7 @@
pub mod binding;
pub mod merge;
pub mod recognition;
pub mod separation;
pub mod ui_design_suggestion;
pub mod utils;
@@ -10,5 +11,7 @@ pub use merge::MergeDTO;
pub(crate) use merge::{merge_ui_impl, merge_ui_impl_with_provider};
pub use recognition::RecognitionDTO;
pub(crate) use recognition::{recognize_ui_impl, recognize_ui_impl_with_provider};
pub(crate) use separation::separate_ui_impl;
pub use separation::{SeparationDTO, SeparationRecoveryDTO};
pub(crate) use ui_design_suggestion::suggest_ui_design_semantic_impl;
pub use ui_design_suggestion::UIDesignSuggestionTreeNode;
@@ -4,6 +4,7 @@ use crate::ui_editor::commands::utils::{
parse_limited_llm_tool_arguments, read_ui_reference_image_data_url, request_ui_editor_llm,
strict_json_schema,
};
use crate::ui_editor::component::{Component, NodeComponent};
use crate::ui_editor::layout::children_display_mode::ChildrenDisplayMode;
use crate::ui_editor::layout::control_layout::ControlLayout;
use crate::ui_editor::layout::dimension::UIRect;
@@ -27,16 +28,13 @@ const MAX_RECOGNITION_TREE_NODES: usize = 512;
const MAX_RECOGNITION_TREE_DEPTH: usize = 32;
const SYSTEM_PROMPT: &str = r#"
角色:
你是游戏 UI 多图结构识别器。
任务:
同时分析同一 UI 系统的全部参考图,建立UI树
用户会给你一些UI截图(它们从属于同一个UI系统)和对应的元数据, 请用给定的工具描述UI结构
识别规则:
* 只识别 UI,不识别场景人物、地形、建筑、光影和背景装饰。
* 只识别 UI元素. 要区分动态内容, 不要白费力气识别应该由程序生成/绘制的内容.(此类内容应该用一个整体节点+自然语言描述) 除此之外必须完整包含所有元素,结构.
* 无法确定类型、层级、关系时,在 UnSure 中写明原因。
* 返回的 trees 必须与输入图片一一对应,每张输入图片只能有一棵树,不能合并多张图片的树。 每棵树的 src_ui_design_image_id 必须等于对应输入图片标注的 id。
* 每棵树必须使用自己的输入图片原始像素坐标系(0,0 as left top)输出
@@ -45,8 +43,15 @@ const SYSTEM_PROMPT: &str = r#"
* 面向用户的字段如名称描述等请用中文
* 由于每个截图未必是完整的, 可能是局部的, 每棵树描述清楚每个截图上UI的层次结构即可
* 不同树的共用框架/层次/...请使用使用相同的名称描述. 不同状态/变体名称使用相同的前缀, 用后缀区别
* 粒度要求: 尽可能细致, 最小单元举例: 进度条的底槽、填充和外框; slider的底槽, dragger等
* 粒度要求: 尽可能细致, 以可交互,方便程序化控制的最小单位为准. 包括不限于: icon, 进度条的底槽、填充和外框; slider的底槽, dragger等.
* 为每个节点直接返回 component.
无背景的逻辑容器返回 "PureNode",不要返回 null.
有背景的容器推荐使用Simple+不锁定宽高比的Image component.
目前我们只做识别, 不要求图片字体的具体绑定参数.
文字组件要求: 艺术字等作为图片组件, 其余正常文字要作为单独的节点识别.
* 多行文本只使用一个节点.
* 不鼓励兄弟节点相互重叠.
* 对于面板等容器的背景等, 必须作为父节点的组件, 禁止新增冗余的所谓"背景节点".
"#;
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
@@ -109,6 +114,7 @@ struct RecognitionNode {
description: String,
children: Vec<RecognitionNode>,
confidence: Confidence,
component: NodeComponent,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
@@ -307,15 +313,12 @@ fn convert_node(
name: source.name.clone(),
description: source.description.clone(),
layout_status: status,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::Llm,
},
// V1 有意把识别结果限定为“结构草稿”:组件绑定属于后续独立阶段。
// 因此空组件不是丢失数据,而是等待 visual-binding 阶段补齐 Image/Text。
// 约定见 docs/technical/【技术方案】AI游戏创作智能体App实施计划-2026-06-24.md。
components: Vec::new(),
component: source.component.clone().into_option(),
children_display_mode: ChildrenDisplayMode::Stack,
children,
})
@@ -323,6 +326,26 @@ fn convert_node(
fn validate_confidence(nodes: &[RecognitionNode]) -> Result<(), String> {
for node in nodes {
if let NodeComponent::WithComponent(component) = &node.component {
if matches!(
component,
Component::Image(crate::ui_editor::component::image::ImageComponent {
target_graphic: Some(_),
..
})
) {
return Err("识别阶段不能返回已绑定的 SpriteAssetId".to_string());
}
if matches!(
component,
Component::Text(crate::ui_editor::component::text::TextComponent {
font: crate::ui_editor::component::text::FontSource::Bound(_),
..
})
) {
return Err("识别阶段不能返回已绑定的字体素材".to_string());
}
}
if let Confidence::UnSure(reason) = &node.confidence {
if reason.trim().is_empty() {
return Err("UnSure 必须包含审阅原因".to_string());
@@ -387,6 +410,7 @@ mod tests {
description: String::new(),
children: Vec::new(),
confidence: Confidence::Confident,
component: NodeComponent::PureNode,
}
}
@@ -490,6 +514,31 @@ mod tests {
description: String::new(),
children: Vec::new(),
confidence: Confidence::UnSure(String::new()),
component: NodeComponent::PureNode,
};
assert!(validate_confidence(&[node]).is_err());
}
#[test]
fn recognition_rejects_bound_font_references() {
let mut text = crate::ui_editor::component::text::TextComponent::default();
text.font = crate::ui_editor::component::text::FontSource::Bound(
crate::ui_editor::utils::FontAssetId::new("font").expect("valid font id"),
);
let node = RecognitionNode {
global_pos_x_px: 0,
global_pos_y_px: 0,
width_px: 1,
height_px: 1,
local_anchor: Anchor::Preset(PresetAnchor {
horizontal: HorizontalAnchor::Left,
vertical: VerticalAnchor::Top,
}),
name: "文本".to_string(),
description: String::new(),
children: Vec::new(),
confidence: Confidence::Confident,
component: NodeComponent::WithComponent(Component::Text(text)),
};
assert!(validate_confidence(&[node]).is_err());
}
@@ -506,7 +555,7 @@ mod tests {
converted.layout.transform.resolve(&root_rect),
UIRect::new(Point2::new(50.0, 25.0), Vector2::new(100.0, 50.0)),
);
assert_eq!(converted.metadata.components_status, StageStatus::NoProblem);
assert_eq!(converted.metadata.component_status, StageStatus::NoProblem);
}
#[test]
@@ -787,12 +836,12 @@ pub(crate) async fn recognize_ui_impl_with_provider(
name: "页面根节点".to_string(),
description: String::new(),
layout_status: StageStatus::NoProblem,
components_status: StageStatus::NoProblem,
component_status: StageStatus::NoProblem,
allow_llm_edit_layout: true,
allow_llm_edit_component: true,
source: NodeSource::System,
},
components: Vec::new(),
component: None,
children_display_mode: ChildrenDisplayMode::Stack,
children,
};
@@ -0,0 +1,480 @@
use super::model::BindingArea;
use image::RgbaImage;
use std::time::Instant;
/// Each edge may move by at most this many pixels from the area returned by
/// the visual model. Keep this policy explicit so changing it is an
/// intentional workflow decision rather than a scattered numeric literal.
pub(crate) const MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX: u32 = 32;
/// Alpha values below this threshold are treated as transparent for boundary
/// detection. The cropped pixels themselves are preserved unchanged.
pub(crate) const MIN_VISIBLE_ALPHA: u8 = 16;
/// An edge needs this many consecutive visible pixels to count as supported.
/// The requirement is reduced to the edge length for one-pixel-wide elements.
pub(crate) const MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS: usize = 2;
#[derive(Clone, Copy, Debug, PartialEq)]
pub(crate) struct NormalizedBindingArea {
pub(crate) area: BindingArea,
pub(crate) changed: bool,
pub(crate) clamped: bool,
pub(crate) transparent: bool,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum EdgeDirection {
Inward,
Outward,
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
struct Rect {
left: u32,
top: u32,
right: u32,
bottom: u32,
}
impl Rect {
fn from_area(area: BindingArea) -> Self {
Self {
left: area.global_pos_x_px,
top: area.global_pos_y_px,
right: area.global_pos_x_px + area.width_px,
bottom: area.global_pos_y_px + area.height_px,
}
}
fn into_area(self) -> BindingArea {
BindingArea {
global_pos_x_px: self.left,
global_pos_y_px: self.top,
width_px: self.right - self.left,
height_px: self.bottom - self.top,
}
}
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum Edge {
Left,
Right,
Top,
Bottom,
}
impl Edge {
const ALL: [Self; 4] = [Self::Left, Self::Right, Self::Top, Self::Bottom];
}
fn pixel_is_visible(alpha: u8) -> bool {
alpha >= MIN_VISIBLE_ALPHA
}
fn has_consecutive_visible_pixels<I>(alphas: I, required: usize) -> bool
where
I: IntoIterator<Item = u8>,
{
let required = required.max(1);
let mut consecutive = 0usize;
for alpha in alphas {
if pixel_is_visible(alpha) {
consecutive = consecutive.saturating_add(1);
if consecutive >= required {
return true;
}
} else {
consecutive = 0;
}
}
false
}
fn edge_has_visible_pixel(image: &RgbaImage, rect: Rect, edge: Edge) -> bool {
let edge_length = match edge {
Edge::Left | Edge::Right => rect.bottom - rect.top,
Edge::Top | Edge::Bottom => rect.right - rect.left,
} as usize;
let required = MIN_CONSECUTIVE_VISIBLE_EDGE_PIXELS.max(1).min(edge_length);
match edge {
Edge::Left | Edge::Right => {
let x = if edge == Edge::Left {
rect.left
} else {
rect.right - 1
};
has_consecutive_visible_pixels(
(rect.top..rect.bottom).map(|y| image.get_pixel(x, y).0[3]),
required,
)
}
Edge::Top | Edge::Bottom => {
let y = if edge == Edge::Top {
rect.top
} else {
rect.bottom - 1
};
has_consecutive_visible_pixels(
(rect.left..rect.right).map(|x| image.get_pixel(x, y).0[3]),
required,
)
}
}
}
fn rect_has_visible_pixel(image: &RgbaImage, rect: Rect) -> bool {
(rect.top..rect.bottom)
.any(|y| (rect.left..rect.right).any(|x| pixel_is_visible(image.get_pixel(x, y).0[3])))
}
fn edge_direction(image: &RgbaImage, rect: Rect, edge: Edge) -> EdgeDirection {
if edge_has_visible_pixel(image, rect, edge) {
EdgeDirection::Outward
} else {
EdgeDirection::Inward
}
}
fn move_edge(rect: &mut Rect, edge: Edge, direction: EdgeDirection) {
match (edge, direction) {
(Edge::Left, EdgeDirection::Inward) => rect.left += 1,
(Edge::Left, EdgeDirection::Outward) => rect.left -= 1,
(Edge::Right, EdgeDirection::Inward) => rect.right -= 1,
(Edge::Right, EdgeDirection::Outward) => rect.right += 1,
(Edge::Top, EdgeDirection::Inward) => rect.top += 1,
(Edge::Top, EdgeDirection::Outward) => rect.top -= 1,
(Edge::Bottom, EdgeDirection::Inward) => rect.bottom -= 1,
(Edge::Bottom, EdgeDirection::Outward) => rect.bottom += 1,
}
}
fn edge_coordinate(rect: Rect, edge: Edge) -> u32 {
match edge {
Edge::Left => rect.left,
Edge::Right => rect.right,
Edge::Top => rect.top,
Edge::Bottom => rect.bottom,
}
}
fn edge_displacement(original: Rect, current: Rect, edge: Edge) -> u32 {
edge_coordinate(original, edge).abs_diff(edge_coordinate(current, edge))
}
fn edge_adjustment_limit() -> u32 {
MAX_BINDING_AREA_EDGE_ADJUSTMENT_PX
}
fn reached_adjustment_limit(original: Rect, current: Rect, edge: Edge) -> bool {
edge_displacement(original, current, edge) >= edge_adjustment_limit()
}
fn can_move_geometrically(
image: &RgbaImage,
current: Rect,
edge: Edge,
direction: EdgeDirection,
) -> bool {
match (edge, direction) {
(Edge::Left, EdgeDirection::Inward) => current.left + 1 < current.right,
(Edge::Left, EdgeDirection::Outward) => current.left > 0,
(Edge::Right, EdgeDirection::Inward) => current.right > current.left + 1,
(Edge::Right, EdgeDirection::Outward) => current.right < image.width(),
(Edge::Top, EdgeDirection::Inward) => current.top + 1 < current.bottom,
(Edge::Top, EdgeDirection::Outward) => current.top > 0,
(Edge::Bottom, EdgeDirection::Inward) => current.bottom > current.top + 1,
(Edge::Bottom, EdgeDirection::Outward) => current.bottom < image.height(),
}
}
fn next_edge_rect(rect: Rect, edge: Edge, direction: EdgeDirection) -> Option<Rect> {
let mut next = rect;
match (edge, direction) {
(Edge::Left, EdgeDirection::Inward) if rect.left + 1 < rect.right => next.left += 1,
(Edge::Left, EdgeDirection::Outward) if rect.left > 0 => next.left -= 1,
(Edge::Right, EdgeDirection::Inward) if rect.right > rect.left + 1 => next.right -= 1,
(Edge::Right, EdgeDirection::Outward) => next.right = next.right.checked_add(1)?,
(Edge::Top, EdgeDirection::Inward) if rect.top + 1 < rect.bottom => next.top += 1,
(Edge::Top, EdgeDirection::Outward) if rect.top > 0 => next.top -= 1,
(Edge::Bottom, EdgeDirection::Inward) if rect.bottom > rect.top + 1 => next.bottom -= 1,
(Edge::Bottom, EdgeDirection::Outward) => next.bottom = next.bottom.checked_add(1)?,
_ => return None,
}
Some(next)
}
fn edge_requires_move(image: &RgbaImage, rect: Rect, edge: Edge, direction: EdgeDirection) -> bool {
match direction {
EdgeDirection::Inward => !edge_has_visible_pixel(image, rect, edge),
EdgeDirection::Outward => {
if !edge_has_visible_pixel(image, rect, edge) {
return false;
}
if !can_move_geometrically(image, rect, edge, direction) {
return true;
}
next_edge_rect(rect, edge, direction)
.is_some_and(|next| edge_has_visible_pixel(image, next, edge))
}
}
}
fn apply_edge_step(
image: &RgbaImage,
original: Rect,
current: Rect,
edge: Edge,
direction: EdgeDirection,
) -> (Rect, bool, bool) {
if !edge_requires_move(image, current, edge, direction) {
return (current, false, false);
}
if reached_adjustment_limit(original, current, edge)
|| !can_move_geometrically(image, current, edge, direction)
{
return (current, false, true);
}
let mut next = current;
move_edge(&mut next, edge, direction);
(next, true, false)
}
/// Normalizes a model-provided area using visible pixels on the processed
/// transparent image. Each edge chooses inward/outward direction once from
/// its initial scan and then moves monotonically, so sparse pixels cannot make
/// the boundary oscillate. The four edge steps are calculated from the same
/// rectangle on each round.
pub(crate) fn normalize_binding_area(
image: &RgbaImage,
original_area: BindingArea,
) -> Result<NormalizedBindingArea, String> {
let started = Instant::now();
if let Err(error) = original_area.validate_in(image.width(), image.height()) {
app_log!(
"ui_separation.area.timing outcome=error elapsed_us={} rounds=0 image_width={} image_height={} area=({}, {}, {}, {})",
started.elapsed().as_micros(),
image.width(),
image.height(),
original_area.global_pos_x_px,
original_area.global_pos_y_px,
original_area.width_px,
original_area.height_px
);
return Err(error);
}
let original = Rect::from_area(original_area);
let directions = Edge::ALL.map(|edge| edge_direction(image, original, edge));
let mut current = original;
let mut clamped = false;
let mut active = [true; 4];
let mut rounds = 0u32;
// TODO: Replace the deliberately simple pixel-by-pixel scan if real UI
// design sizes show this path to be a measurable bottleneck.
while active.iter().any(|value| *value) {
rounds = rounds.saturating_add(1);
let before = current;
let mut next = current;
let mut moved = [false; 4];
for (index, edge) in Edge::ALL.into_iter().enumerate() {
if !active[index] {
continue;
}
let (candidate, did_move, reached_limit) =
apply_edge_step(image, original, current, edge, directions[index]);
if reached_limit {
clamped = true;
active[index] = false;
} else if !did_move {
active[index] = false;
}
moved[index] = did_move;
match edge {
Edge::Left => next.left = candidate.left,
Edge::Right => next.right = candidate.right,
Edge::Top => next.top = candidate.top,
Edge::Bottom => next.bottom = candidate.bottom,
}
}
if next.left >= next.right {
clamped = true;
if moved[0] {
active[0] = false;
}
if moved[1] {
active[1] = false;
}
next.left = current.left;
next.right = current.right;
}
if next.top >= next.bottom {
clamped = true;
if moved[2] {
active[2] = false;
}
if moved[3] {
active[3] = false;
}
next.top = current.top;
next.bottom = current.bottom;
}
current = next;
if current == before {
break;
}
}
let area = current.into_area();
let normalized = NormalizedBindingArea {
changed: area != original_area,
area,
clamped,
transparent: !rect_has_visible_pixel(image, current),
};
app_log!(
"ui_separation.area.timing outcome=ok elapsed_us={} rounds={} image_width={} image_height={} area=({}, {}, {}, {}) changed={} clamped={} transparent={}",
started.elapsed().as_micros(),
rounds,
image.width(),
image.height(),
original_area.global_pos_x_px,
original_area.global_pos_y_px,
original_area.width_px,
original_area.height_px,
normalized.changed,
normalized.clamped,
normalized.transparent
);
Ok(normalized)
}
#[cfg(test)]
mod tests {
use super::*;
use image::{Rgba, RgbaImage};
fn image_with_rect(
width: u32,
height: u32,
left: u32,
top: u32,
right: u32,
bottom: u32,
) -> RgbaImage {
let mut image = RgbaImage::from_pixel(width, height, Rgba([0, 0, 0, 0]));
for y in top..bottom {
for x in left..right {
image.put_pixel(x, y, Rgba([255, 255, 255, 255]));
}
}
image
}
fn area(x: u32, y: u32, width: u32, height: u32) -> BindingArea {
BindingArea {
global_pos_x_px: x,
global_pos_y_px: y,
width_px: width,
height_px: height,
}
}
#[test]
fn shrinks_empty_edges_to_visible_bounds() {
let image = image_with_rect(32, 32, 10, 11, 16, 18);
let result = normalize_binding_area(&image, area(6, 7, 14, 16)).unwrap();
assert_eq!(result.area, area(10, 11, 6, 7));
assert!(result.changed);
assert!(!result.clamped);
assert!(!result.transparent);
}
#[test]
fn expands_visible_edges_to_cover_the_element() {
let image = image_with_rect(32, 32, 10, 11, 16, 18);
let result = normalize_binding_area(&image, area(11, 12, 4, 5)).unwrap();
assert_eq!(result.area, area(10, 11, 6, 7));
assert!(result.changed);
assert!(!result.clamped);
assert!(!result.transparent);
}
#[test]
fn adjusts_each_edge_independently() {
let image = image_with_rect(32, 32, 10, 11, 16, 18);
let result = normalize_binding_area(&image, area(10, 12, 10, 3)).unwrap();
assert_eq!(result.area, area(10, 11, 6, 7));
}
#[test]
fn ignores_low_alpha_halo_while_preserving_visible_bounds() {
let mut image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0]));
for y in 6..10 {
for x in 5..9 {
image.put_pixel(x, y, Rgba([255, 255, 255, 255]));
}
}
image.put_pixel(4, 7, Rgba([255, 255, 255, 1]));
image.put_pixel(9, 8, Rgba([255, 255, 255, 8]));
let result = normalize_binding_area(&image, area(4, 5, 6, 6)).unwrap();
assert_eq!(result.area, area(5, 6, 4, 4));
}
#[test]
fn ignores_isolated_visible_edge_pixel() {
let mut image = image_with_rect(16, 16, 4, 4, 6, 8);
image.put_pixel(6, 4, Rgba([255, 255, 255, 255]));
let result = normalize_binding_area(&image, area(4, 4, 2, 4)).unwrap();
assert_eq!(result.area, area(4, 4, 2, 4));
}
#[test]
fn fully_transparent_image_uses_the_same_path() {
let image = RgbaImage::from_pixel(32, 32, Rgba([0, 0, 0, 0]));
let result = normalize_binding_area(&image, area(10, 10, 10, 10)).unwrap();
assert_eq!(result.area, area(14, 14, 2, 2));
assert!(result.changed);
assert!(result.transparent);
}
#[test]
fn caps_each_edge_at_absolute_pixel_limit() {
let image = image_with_rect(128, 128, 0, 0, 128, 128);
let result = normalize_binding_area(&image, area(48, 48, 8, 8)).unwrap();
assert_eq!(result.area, area(16, 16, 72, 72));
assert!(result.clamped);
}
#[test]
fn clamps_expansion_to_image_edges() {
let image = image_with_rect(16, 16, 0, 0, 4, 4);
let result = normalize_binding_area(&image, area(1, 1, 2, 2)).unwrap();
assert_eq!(result.area, area(0, 0, 4, 4));
assert!(result.clamped);
}
#[test]
fn exact_split_at_adjustment_limit_is_not_clamped() {
let image = image_with_rect(16, 16, 4, 4, 8, 8);
let result = normalize_binding_area(&image, area(5, 5, 2, 2)).unwrap();
assert_eq!(result.area, area(4, 4, 4, 4));
assert!(!result.clamped);
}
#[test]
fn one_pixel_area_expands_with_configured_adjustment_limit() {
let image = image_with_rect(8, 8, 2, 2, 5, 5);
let result = normalize_binding_area(&image, area(3, 3, 1, 1)).unwrap();
assert_eq!(result.area, area(2, 2, 3, 3));
assert!(!result.clamped);
}
#[test]
fn rejects_zero_sized_or_out_of_bounds_model_areas() {
let image = RgbaImage::from_pixel(16, 16, Rgba([0, 0, 0, 0]));
assert!(normalize_binding_area(&image, area(0, 0, 0, 1)).is_err());
assert!(normalize_binding_area(&image, area(15, 15, 2, 2)).is_err());
}
}
@@ -0,0 +1,106 @@
use super::area::MIN_VISIBLE_ALPHA;
use base64::Engine as _;
use image::{ImageFormat, Rgba, RgbaImage};
use std::fs;
use std::io::Cursor;
use std::path::{Path, PathBuf};
pub(crate) const VISUAL_BINDING_TRANSPARENT_MARKER_RGBA: [u8; 4] = [255, 0, 255, 255];
pub(crate) async fn preprocess_for_visual_binding(
processed_url: String,
sidecar: PathBuf,
) -> Result<String, String> {
tokio::task::spawn_blocking(move || {
preprocess_for_visual_binding_blocking(&processed_url, &sidecar)
})
.await
.map_err(|error| format!("视觉绑定预处理任务失败:{error}"))?
}
fn preprocess_for_visual_binding_blocking(
processed_url: &str,
sidecar: &Path,
) -> Result<String, String> {
let encoded = processed_url
.split_once(',')
.map(|(_, data)| data)
.ok_or_else(|| "处理图 data URL 无效".to_string())?;
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded.trim())
.map_err(|error| format!("解析处理图失败:{error}"))?;
let mut image = image::load_from_memory(&bytes)
.map_err(|error| format!("解码处理图失败:{error}"))?
.to_rgba8();
for pixel in image.pixels_mut() {
if pixel.0[3] < MIN_VISIBLE_ALPHA {
*pixel = Rgba(VISUAL_BINDING_TRANSPARENT_MARKER_RGBA);
} else {
pixel.0[3] = 255;
}
}
let mut png = Vec::new();
image::DynamicImage::ImageRgba8(image)
.write_to(&mut Cursor::new(&mut png), ImageFormat::Png)
.map_err(|error| format!("编码视觉绑定预览失败:{error}"))?;
let debug_name = format!("binding-{}.png", uuid::Uuid::new_v4().simple());
if let Err(error) = fs::write(sidecar.join(&debug_name), &png) {
app_log!(
"ui_separation.warning stage=visual_binding_preview_write file={} error={error}",
debug_name
);
}
Ok(format!(
"data:image/png;base64,{}",
base64::engine::general_purpose::STANDARD.encode(png)
))
}
#[cfg(test)]
mod tests {
use super::*;
use image::Rgba;
use tempfile::tempdir;
fn data_url(image: RgbaImage) -> String {
let mut bytes = Vec::new();
image::DynamicImage::ImageRgba8(image)
.write_to(&mut Cursor::new(&mut bytes), ImageFormat::Png)
.expect("encode fixture");
format!(
"data:image/png;base64,{}",
base64::engine::general_purpose::STANDARD.encode(bytes)
)
}
#[test]
fn preprocesses_alpha_using_existing_visibility_threshold() {
let mut image = RgbaImage::from_pixel(4, 1, Rgba([10, 20, 30, 255]));
image.put_pixel(0, 0, Rgba([1, 2, 3, 0]));
image.put_pixel(1, 0, Rgba([4, 5, 6, MIN_VISIBLE_ALPHA - 1]));
image.put_pixel(2, 0, Rgba([7, 8, 9, MIN_VISIBLE_ALPHA]));
image.put_pixel(3, 0, Rgba([11, 12, 13, 254]));
let directory = tempdir().expect("create sidecar fixture");
let url = preprocess_for_visual_binding_blocking(&data_url(image), directory.path())
.expect("preprocess fixture");
let encoded = url.split_once(',').expect("data URL").1;
let bytes = base64::engine::general_purpose::STANDARD
.decode(encoded)
.expect("decode output");
let output = image::load_from_memory(&bytes)
.expect("decode output png")
.to_rgba8();
assert_eq!(
output.get_pixel(0, 0).0,
VISUAL_BINDING_TRANSPARENT_MARKER_RGBA
);
assert_eq!(
output.get_pixel(1, 0).0,
VISUAL_BINDING_TRANSPARENT_MARKER_RGBA
);
assert_eq!(output.get_pixel(2, 0).0, [7, 8, 9, 255]);
assert_eq!(output.get_pixel(3, 0).0, [11, 12, 13, 255]);
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,50 @@
use crate::ui_editor::utils::NodeId;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize, JsonSchema)]
#[schemars(deny_unknown_fields)]
pub struct BindingArea {
pub global_pos_x_px: u32,
pub global_pos_y_px: u32,
pub width_px: u32,
pub height_px: u32,
}
impl BindingArea {
pub fn validate_in(&self, w: u32, h: u32) -> Result<(), String> {
if self.width_px == 0 || self.height_px == 0 {
return Err("BindingArea 宽度和高度必须大于 0".into());
}
if self
.global_pos_x_px
.checked_add(self.width_px)
.is_none_or(|v| v > w)
|| self
.global_pos_y_px
.checked_add(self.height_px)
.is_none_or(|v| v > h)
{
return Err("BindingArea 超出处理图边界".into());
}
Ok(())
}
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
#[schemars(deny_unknown_fields)]
pub enum BindingDecision {
Ok {
extracted_area: BindingArea,
to_node: NodeId,
},
NeedRework {
advice: String,
to_node: NodeId,
},
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, JsonSchema)]
pub struct BindingResp {
pub decisions: Vec<BindingDecision>,
}
@@ -0,0 +1,20 @@
mod binding;
mod node;
mod note;
mod result;
pub use binding::*;
pub use node::*;
pub use note::*;
pub use result::*;
pub const SEPARATION_STATE_SCHEMA_VERSION: &str = "ui-editor-separation-state.v2";
pub const MAX_REWORK_COUNT: u32 = 3;
pub const MAX_REWORK_NOTE_CHARS: usize = 512;
pub const IMAGE_EDIT_MAX_DIMENSION_PX: u64 = 2880;
pub const IMAGE_EDIT_MIN_DIMENSION_PX: u64 = 816;
pub const IMAGE_EDIT_DIMENSION_ALIGNMENT_PX: u64 = 16;
pub const IMAGE_EDIT_AREA_UTILIZATION_PERCENT: u64 = 80;
pub const IMAGE_EDIT_AREA_LIMIT_PX: u64 =
IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_AREA_UTILIZATION_PERCENT
/ 100;
@@ -0,0 +1,43 @@
use crate::ui_editor::utils::{NodeId, UIDesignImageId};
use serde::{Deserialize, Serialize};
use ts_rs::TS;
#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub enum SeparationNodeKind {
ImageTarget,
TextRemovalOnly,
PureContainer,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct SeparationNode {
pub id: NodeId,
pub kind: SeparationNodeKind,
pub global_pos_x_px: u32,
pub global_pos_y_px: u32,
pub width_px: u32,
pub height_px: u32,
pub note: super::SeparationNote,
pub children: Vec<SeparationNode>,
pub rework_count: u32,
}
impl SeparationNode {
pub fn as_prompt(&self) -> String {
format!(
"node_id={} note: {}",
self.id.as_str(),
self.note.as_prompt()
)
}
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct SeparationTree {
pub src_ui_design: UIDesignImageId,
pub root: SeparationNode,
pub root_extractable: bool,
}
@@ -0,0 +1,41 @@
use serde::{Deserialize, Serialize};
use ts_rs::TS;
const MAX_PROMPT_NOTE_CHARS: usize = 512;
fn sanitize_prompt_text(value: &str) -> String {
value
.chars()
.filter_map(|character| {
if character == '`' {
Some('\'')
} else if character.is_control() {
Some(' ')
} else {
Some(character)
}
})
.take(MAX_PROMPT_NOTE_CHARS)
.collect()
}
#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct SeparationNote {
pub description: String,
pub rework_notes: Vec<String>,
}
impl SeparationNote {
pub fn as_prompt(&self) -> String {
let mut prompt = format!("desc: {}", sanitize_prompt_text(&self.description));
if !self.rework_notes.is_empty() {
prompt.push_str("\nprevious rework notes:");
for note in &self.rework_notes {
prompt.push_str("\n- ");
prompt.push_str(&sanitize_prompt_text(note));
}
}
prompt
}
}
@@ -0,0 +1,43 @@
use crate::ui_editor::utils::NodeId;
use serde::{Deserialize, Serialize};
use ts_rs::TS;
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct BoundNode {
pub node_id: NodeId,
pub cut_image_path: String,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct ProblematicNode {
pub node_id: NodeId,
pub problem_description: String,
pub rework_count: u32,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct SeparationState {
pub schema_version: String,
pub trees: Vec<super::SeparationTree>,
pub bound: Vec<BoundNode>,
pub problematic_nodes: Vec<ProblematicNode>,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct SeparationDTO {
pub bound_nodes: Vec<BoundNode>,
pub problematic_nodes: Vec<ProblematicNode>,
}
#[derive(Clone, Debug, Deserialize, PartialEq, Serialize, TS)]
#[ts(export, export_to = concat!(env!("CARGO_MANIFEST_DIR"), "/../src/features/ui-editor/types/"))]
pub struct SeparationRecoveryDTO {
pub exists: bool,
pub bound_node_count: usize,
pub problematic_node_count: usize,
pub has_pending_tree: bool,
}
@@ -0,0 +1,196 @@
use super::model::*;
use crate::ui_editor::commands::separation::*;
use std::fs;
use std::path::{Path, PathBuf};
const SEPARATION_STATE_MAX_BYTES: usize = 8 * 1024 * 1024;
pub fn separation_sidecar_dir(root: &Path, asset_id: &str) -> Result<PathBuf, String> {
if asset_id.trim().is_empty() || asset_id.trim() != asset_id {
app_log!("ui_separation.error stage=sidecar_dir reason=invalid_asset_id");
return Err("UI 资源 ID 无效".to_string());
}
let dir = root.join("ui").join(format!(
".{}-separation",
crate::ui_editor::persistence::generated_file_stem(asset_id)
));
if !dir.starts_with(root) {
app_log!("ui_separation.error stage=sidecar_dir reason=path_escape");
return Err("separation sidecar 路径越界".to_string());
}
app_log!(
"ui_separation.sidecar_resolved asset_id={} directory={}",
asset_id,
dir.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>")
);
Ok(dir)
}
pub fn separation_state_path(root: &Path, asset_id: &str) -> Result<PathBuf, String> {
Ok(separation_sidecar_dir(root, asset_id)?.join("state.json"))
}
pub fn project_relative_path(root: &Path, path: &Path) -> Result<String, String> {
let relative = path
.strip_prefix(root)
.map_err(|_| "separation 产物必须位于项目目录内".to_string())?;
let value = relative.to_string_lossy().replace('\\', "/");
if value.is_empty() || value.starts_with('/') || value.split('/').any(|part| part == "..") {
return Err("separation 产物相对路径无效".to_string());
}
Ok(value)
}
pub async fn write_separation_state(path: PathBuf, state: SeparationState) -> Result<(), String> {
tokio::task::spawn_blocking(move || write_separation_state_blocking(&path, &state))
.await
.map_err(|error| format!("写入 separation state 任务失败:{error}"))?
}
fn write_separation_state_blocking(path: &Path, state: &SeparationState) -> Result<(), String> {
if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION {
app_log!("ui_separation.error stage=state_write reason=schema_mismatch");
return Err("不支持的 separation state schema".to_string());
}
app_log!(
"ui_separation.state_write.start file={} trees={} bound={} problematic={}",
path.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
state.trees.len(),
state.bound.len(),
state.problematic_nodes.len()
);
let bytes = serde_json::to_vec_pretty(state).map_err(|error| {
app_log!("ui_separation.error stage=state_write reason=serialize error={error}");
format!("序列化 separation state 失败:{error}")
})?;
let parent = path.parent().ok_or_else(|| {
app_log!("ui_separation.error stage=state_write reason=missing_parent");
"separation state 路径缺少父目录".to_string()
})?;
fs::create_dir_all(parent).map_err(|error| {
app_log!("ui_separation.error stage=state_write reason=create_parent error={error}");
format!("创建 separation sidecar 失败:{error}")
})?;
let temporary = path.with_extension("json.tmp");
fs::write(&temporary, bytes).map_err(|error| {
app_log!("ui_separation.error stage=state_write reason=write_temp error={error}");
format!("写入 separation state 失败:{error}")
})?;
fs::rename(&temporary, path).map_err(|error| {
app_log!("ui_separation.error stage=state_write reason=install error={error}");
format!("安装 separation state 失败:{error}")
})?;
app_log!(
"ui_separation.state_write.completed file={} bytes={} trees={} bound={} problematic={}",
path.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
fs::metadata(path)
.map(|metadata| metadata.len())
.unwrap_or(0),
state.trees.len(),
state.bound.len(),
state.problematic_nodes.len()
);
Ok(())
}
pub fn read_separation_state(path: &Path) -> Result<SeparationState, String> {
app_log!(
"ui_separation.state_read.start file={}",
path.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>")
);
let metadata = fs::metadata(path).map_err(|error| {
app_log!("ui_separation.error stage=state_read reason=metadata error={error}");
format!("读取 separation state 信息失败:{error}")
})?;
if metadata.len() > SEPARATION_STATE_MAX_BYTES as u64 {
app_log!(
"ui_separation.error stage=state_read reason=too_large bytes={} max_bytes={}",
metadata.len(),
SEPARATION_STATE_MAX_BYTES
);
return Err(format!(
"separation state 超过 {} 字节上限",
SEPARATION_STATE_MAX_BYTES
));
}
let bytes = fs::read(path).map_err(|error| {
app_log!("ui_separation.error stage=state_read reason=read error={error}");
format!("读取 separation state 失败:{error}")
})?;
if bytes.len() > SEPARATION_STATE_MAX_BYTES {
return Err(format!(
"separation state 超过 {} 字节上限",
SEPARATION_STATE_MAX_BYTES
));
}
let state: SeparationState = serde_json::from_slice(&bytes).map_err(|error| {
app_log!("ui_separation.error stage=state_read reason=parse error={error}");
format!("解析 separation state 失败:{error}")
})?;
if state.schema_version != SEPARATION_STATE_SCHEMA_VERSION {
app_log!("ui_separation.error stage=state_read reason=schema_mismatch");
return Err("不支持的 separation state schema".to_string());
}
app_log!(
"ui_separation.state_read.completed bytes={} trees={} bound={} problematic={}",
bytes.len(),
state.trees.len(),
state.bound.len(),
state.problematic_nodes.len()
);
Ok(state)
}
pub fn separation_dto(state: &SeparationState) -> SeparationDTO {
app_log!(
"ui_separation.dto bound_nodes={} problematic_nodes={} remaining_trees={}",
state.bound.len(),
state.problematic_nodes.len(),
state.trees.len()
);
SeparationDTO {
bound_nodes: state.bound.clone(),
problematic_nodes: state.problematic_nodes.clone(),
}
}
pub fn inspect_separation_recovery(
root: &Path,
asset_id: &str,
) -> Result<SeparationRecoveryDTO, String> {
let state_path = separation_state_path(root, asset_id)?;
if !state_path.exists() {
return Ok(SeparationRecoveryDTO {
exists: false,
bound_node_count: 0,
problematic_node_count: 0,
has_pending_tree: false,
});
}
let state = read_separation_state(&state_path)?;
Ok(SeparationRecoveryDTO {
exists: true,
bound_node_count: state.bound.len(),
problematic_node_count: state.problematic_nodes.len(),
has_pending_tree: state
.trees
.iter()
.any(|tree| !tree.root.children.is_empty() || tree.root_extractable),
})
}
pub fn finalize_separation(root: &Path, asset_id: &str) -> Result<(), String> {
let state_path = separation_state_path(root, asset_id)?;
match fs::remove_file(&state_path) {
Ok(()) => Ok(()),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(error) => Err(format!("删除 separation state 失败:{error}")),
}
}
@@ -0,0 +1,46 @@
use crate::ui_editor::commands::separation::image_preprocess::VISUAL_BINDING_TRANSPARENT_MARKER_RGBA;
use crate::ui_editor::commands::separation::SeparationNode;
pub(crate) fn gen_binding_prompt(nodes: Vec<&SeparationNode>) -> String {
let [marker_red, marker_green, marker_blue, marker_alpha] =
VISUAL_BINDING_TRANSPARENT_MARKER_RGBA;
let marker_color = format!("rgba({marker_red}, {marker_green}, {marker_blue}, {marker_alpha})");
let binding_system_prompt = format!(
r#"
You will be given a src UI design image and a processed image, where some ui elements are separated.
You need to recognize and review the separation using the given tool.
field notes:
* extracted_area MUST be the recognized area from the processed image, INSTEAD OF from the src image.
The processed image is the only authoritative image for extracted_area.
Return the pixel bounding box of the extracted element as it appears in the processed image.
Do not copy, infer, or reuse the source node rectangle.
The src image is only for identifying which semantic UI element belongs to to_node.
Here were the separation requirements:
Preserve hard edges and the exact visible shape.
The processed image is an opaque visual-binding preview containing the requested image layers.
The solid color {marker_color} is an intentional transparency marker added by this workflow before this request.
It is not an image-edit defect and is not part of any UI element.
Do not include this marker color in the extracted area.
Do not use the source node rectangle as the extracted area.
And you should also review if the extracted's successfully meet the src image:
* shape
* color
* style
* edge process
...
if not, use `NeedRework` data structure in the tool to indicate the (it should be from which) node id and advice.
your advice (less than 20 words) will be used to improve the separation in the next time.
these node need handle:
"#
);
let mut result = binding_system_prompt;
result.reserve(512);
for elem in nodes {
result.push_str(&elem.as_prompt());
result.push('\n');
}
result
}
@@ -0,0 +1,169 @@
use crate::ui_editor::commands::separation::model::{SeparationState, SeparationTree};
use crate::ui_editor::commands::separation::{SeparationNode, SeparationNodeKind};
use crate::ui_editor::utils::NodeId;
use serde::Serialize;
use std::collections::HashSet;
#[derive(Serialize)]
struct ExtractPromptDocument {
ui_layer_tree: ExtractPromptNode,
}
#[derive(Serialize)]
struct ExtractPromptNode {
index: usize,
status: ExtractPromptStatus,
rect: ExtractPromptRect,
description: String,
#[serde(skip_serializing_if = "Vec::is_empty")]
rework_notes: Vec<String>,
#[serde(skip_serializing_if = "Vec::is_empty")]
children: Vec<ExtractPromptNode>,
}
#[derive(Serialize)]
enum ExtractPromptStatus {
#[serde(rename = "OUTPUT_THIS_TURN")]
OutputThisTurn,
#[serde(rename = "DONE")]
Done,
#[serde(rename = "CONTEXT_ONLY")]
ContextOnly,
#[serde(rename = "REMOVE_ONLY")]
RemoveOnly,
}
#[derive(Serialize)]
struct ExtractPromptRect {
x: u32,
y: u32,
width: u32,
height: u32,
}
pub(crate) fn gen_extract_prompt(
state: &SeparationState,
tree: &SeparationTree,
batch: &[&SeparationNode],
) -> Result<String, String> {
let mut result = r#"
This is a UI design image, not a normal photo/illustration. Extract it strictly as UI elements/layers, not as a generic foreground/background extraction.
Treat distinct UI element as its own layer with hard, clean, pixel-accurate edges and full transparency outside the element.
Generate one transparent atlas at the requested canvas size. You may move or scale output layers so they do not cover one another.
Preserve hard edges and the exact visible shape. Never split a scene/background into multiple scene layers.
Ordinary text is editable UI text: remove it from its parent image/background and do not generate a text raster layer.
Extract only the image nodes marked OUTPUT_THIS_TURN. Reconstruct every child/text layer that is listed under a parent but is not an output target.
UI layer tree:
"#
.to_string();
result.reserve(2048);
let target_ids = batch
.iter()
.map(|node| node.id.clone())
.collect::<HashSet<_>>();
let terminal_ids = state
.bound
.iter()
.map(|node| node.node_id.clone())
.chain(
state
.problematic_nodes
.iter()
.map(|node| node.node_id.clone()),
)
.collect::<HashSet<_>>();
let mut index = 1;
let document = ExtractPromptDocument {
ui_layer_tree: project_node(&tree.root, &target_ids, &terminal_ids, &mut index),
};
let yaml = serde_yaml::to_string(&document)
.map_err(|error| format!("UI separation extract prompt projection failed: {error}"))?;
result.push_str("```yaml\n");
result.push_str(&yaml);
result.push_str("```\n");
Ok(result)
}
fn project_node(
node: &SeparationNode,
target_ids: &HashSet<NodeId>,
terminal_ids: &HashSet<NodeId>,
index: &mut usize,
) -> ExtractPromptNode {
let current_index = *index;
*index += 1;
let status = match node.kind {
SeparationNodeKind::TextRemovalOnly => ExtractPromptStatus::RemoveOnly,
SeparationNodeKind::PureContainer => ExtractPromptStatus::ContextOnly,
SeparationNodeKind::ImageTarget if target_ids.contains(&node.id) => {
ExtractPromptStatus::OutputThisTurn
}
SeparationNodeKind::ImageTarget if terminal_ids.contains(&node.id) => {
ExtractPromptStatus::Done
}
SeparationNodeKind::ImageTarget => ExtractPromptStatus::ContextOnly,
};
let children = node
.children
.iter()
.map(|child| project_node(child, target_ids, terminal_ids, index))
.collect();
ExtractPromptNode {
index: current_index,
status,
rect: ExtractPromptRect {
x: node.global_pos_x_px,
y: node.global_pos_y_px,
width: node.width_px,
height: node.height_px,
},
description: node.note.description.clone(),
rework_notes: node.note.rework_notes.clone(),
children,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ui_editor::commands::separation::model::SeparationNote;
#[test]
fn projects_source_node_to_prompt_view() {
let node = SeparationNode {
id: NodeId::new("image").unwrap(),
kind: SeparationNodeKind::ImageTarget,
global_pos_x_px: 1,
global_pos_y_px: 2,
width_px: 3,
height_px: 4,
note: SeparationNote {
description: "按钮".to_string(),
rework_notes: vec!["保留圆角".to_string()],
},
children: vec![],
rework_count: 0,
};
let mut index = 1;
let target_ids = HashSet::from([node.id.clone()]);
let projected = project_node(&node, &target_ids, &HashSet::new(), &mut index);
assert_eq!(projected.index, 1);
assert!(matches!(
projected.status,
ExtractPromptStatus::OutputThisTurn
));
assert_eq!(projected.rect.x, 1);
assert_eq!(projected.rect.y, 2);
assert_eq!(projected.rect.width, 3);
assert_eq!(projected.rect.height, 4);
assert_eq!(projected.description, "按钮");
assert_eq!(projected.rework_notes, vec!["保留圆角".to_string()]);
assert!(projected.children.is_empty());
}
}
@@ -0,0 +1,5 @@
mod binding;
mod extract;
pub(super) use binding::gen_binding_prompt;
pub(super) use extract::gen_extract_prompt;
@@ -0,0 +1,190 @@
use super::model::*;
use crate::ui_editor::component::{image::ImageComponent, Component};
use crate::ui_editor::layout::node::Node;
use crate::ui_editor::state::State;
use std::collections::HashSet;
fn is_unbound_image(node: &Node) -> bool {
matches!(
node.component.as_ref(),
Some(Component::Image(ImageComponent {
target_graphic: None,
..
}))
)
}
fn has_image_component(node: &Node) -> bool {
matches!(node.component.as_ref(), Some(Component::Image(_)))
}
fn has_text_component(node: &Node) -> bool {
matches!(node.component.as_ref(), Some(Component::Text(_)))
}
fn node_pixel_rect(
node: &Node,
parent: &crate::ui_editor::layout::dimension::UIRect,
ppu: f32,
) -> (u32, u32, u32, u32) {
let rect = node.layout.transform.resolve(parent);
(
(rect.min.x * ppu).max(0.0).round() as u32,
(rect.min.y * ppu).max(0.0).round() as u32,
(rect.size.x * ppu).max(0.0).round() as u32,
(rect.size.y * ppu).max(0.0).round() as u32,
)
}
fn node_description(node: &Node) -> String {
let name = node.metadata.name.trim();
let description = node.metadata.description.trim();
match (name.is_empty(), description.is_empty()) {
(true, true) => "未命名 UI 图片元素".to_string(),
(false, true) => name.to_string(),
(true, false) => description.to_string(),
(false, false) => format!("{name}{description}"),
}
}
fn collect_todo_nodes(
node: &Node,
parent: &crate::ui_editor::layout::dimension::UIRect,
ppu: f32,
output: &mut Vec<SeparationNode>,
) {
let rect = node.layout.transform.resolve(parent);
let mut children = Vec::new();
for child in &node.children {
collect_todo_nodes(child, &rect, ppu, &mut children);
}
let kind = if is_unbound_image(node) {
Some(SeparationNodeKind::ImageTarget)
} else if has_text_component(node) && !has_image_component(node) {
Some(SeparationNodeKind::TextRemovalOnly)
} else {
None
};
if let Some(kind) = kind {
let (x, y, w, h) = node_pixel_rect(node, parent, ppu);
if w > 0 && h > 0 {
output.push(SeparationNode {
id: node.id.clone(),
kind,
global_pos_x_px: x,
global_pos_y_px: y,
width_px: w,
height_px: h,
note: SeparationNote {
description: node_description(node),
rework_notes: Vec::new(),
},
children,
rework_count: 0,
});
} else {
output.extend(children);
}
} else {
output.extend(children);
}
}
pub fn construct_separation_state(state: &State) -> SeparationState {
let trees = state
.ui_trees
.iter()
.filter_map(|tree| {
let image = state.ui_design_images.get(&tree.src_ui_design)?;
let ppu = image.pixels_per_unit.get();
let size = image.pixel_size / ppu;
let root_rect =
crate::ui_editor::layout::dimension::UIRect::new(nalgebra::Point2::origin(), size);
let mut children = Vec::new();
for child in &tree.root.children {
collect_todo_nodes(child, &root_rect, ppu, &mut children);
}
let root_extractable = is_unbound_image(&tree.root);
if !root_extractable && children.is_empty() {
return None;
}
let (x, y, w, h) = node_pixel_rect(&tree.root, &root_rect, ppu);
Some(SeparationTree {
src_ui_design: tree.src_ui_design.clone(),
root: SeparationNode {
id: tree.root.id.clone(),
kind: if root_extractable {
SeparationNodeKind::ImageTarget
} else if has_text_component(&tree.root) && !has_image_component(&tree.root) {
SeparationNodeKind::TextRemovalOnly
} else {
SeparationNodeKind::PureContainer
},
global_pos_x_px: x,
global_pos_y_px: y,
width_px: w,
height_px: h,
note: SeparationNote {
description: node_description(&tree.root),
rework_notes: Vec::new(),
},
children,
rework_count: 0,
},
root_extractable,
})
})
.collect();
SeparationState {
schema_version: SEPARATION_STATE_SCHEMA_VERSION.to_string(),
trees,
bound: Vec::new(),
problematic_nodes: Vec::new(),
}
}
pub fn validate_binding_response(
response: &BindingResp,
batch: &[&SeparationNode],
processed_dimensions: (u32, u32),
) -> Result<(), String> {
let expected = batch
.iter()
.filter(|node| matches!(node.kind, SeparationNodeKind::ImageTarget))
.map(|node| node.id.clone())
.collect::<HashSet<_>>();
let mut seen = HashSet::new();
for decision in &response.decisions {
let node_id = match decision {
BindingDecision::Ok { to_node, .. } | BindingDecision::NeedRework { to_node, .. } => {
to_node
}
};
if !expected.contains(node_id) {
return Err(format!("视觉绑定返回了未知节点:{}", node_id.as_str()));
}
if !seen.insert(node_id.clone()) {
return Err(format!("视觉绑定重复返回节点:{}", node_id.as_str()));
}
if let BindingDecision::Ok { extracted_area, .. } = decision {
extracted_area
.validate_in(processed_dimensions.0, processed_dimensions.1)
.map_err(|error| format!("节点 {} 的分离区域无效:{error}", node_id.as_str()))?;
}
if let BindingDecision::NeedRework { advice, .. } = decision {
if advice.trim().is_empty() {
return Err("NeedRework 必须包含问题描述".to_string());
}
if advice.chars().count() > MAX_REWORK_NOTE_CHARS {
// TODO add this back in prompt
return Err(format!(
"NeedRework 问题描述不能超过 {MAX_REWORK_NOTE_CHARS} 个字符"
));
}
}
}
if seen.len() != expected.len() {
return Err("视觉绑定未覆盖当前 batch 的全部节点".to_string());
}
Ok(())
}
@@ -0,0 +1,146 @@
use crate::ui_editor::commands::separation::model::*;
use crate::ui_editor::utils::NodeId;
use std::collections::HashSet;
#[derive(Debug)]
pub struct ImageBatch<'a> {
pub nodes: Vec<&'a SeparationNode>,
pub area_px: u64,
pub image_edit_dimension_px: u32,
}
/// Derive the square raw image-edit canvas from the selected source area.
/// The calculation lives beside batch selection so the area budget and
/// request size cannot drift apart.
pub fn image_edit_dimension_for_area(area_px: u64) -> u32 {
let max = u128::from(IMAGE_EDIT_MAX_DIMENSION_PX);
let limit = u128::from(IMAGE_EDIT_AREA_LIMIT_PX);
let area = u128::from(area_px);
let raw = if area >= limit {
IMAGE_EDIT_MAX_DIMENSION_PX
} else if area == 0 {
0
} else {
// Find floor(max * sqrt(area / limit)) without floating-point rounding.
let target = max * max * area;
let mut low = 0u64;
let mut high = IMAGE_EDIT_MAX_DIMENSION_PX;
while low < high {
let mid = low + (high - low + 1) / 2;
if u128::from(mid) * u128::from(mid) * limit <= target {
low = mid;
} else {
high = mid - 1;
}
}
low
};
let alignment = IMAGE_EDIT_DIMENSION_ALIGNMENT_PX;
let aligned = raw / alignment * alignment;
aligned.clamp(IMAGE_EDIT_MIN_DIMENSION_PX, IMAGE_EDIT_MAX_DIMENSION_PX) as u32
}
fn terminal_ids(state: &SeparationState) -> HashSet<NodeId> {
state
.bound
.iter()
.map(|n| n.node_id.clone())
.chain(state.problematic_nodes.iter().map(|n| n.node_id.clone()))
.collect()
}
fn collect_dfs_batch<'a>(
node: &'a SeparationNode,
is_root: bool,
root_extractable: bool,
terminal: &HashSet<NodeId>,
selected: &mut Vec<&'a SeparationNode>,
area: &mut u64,
) -> bool {
let is_target = matches!(node.kind, SeparationNodeKind::ImageTarget)
&& (!is_root || root_extractable)
&& !terminal.contains(&node.id);
if is_target {
let node_area = u64::from(node.width_px).saturating_mul(u64::from(node.height_px));
let would_exceed = area.saturating_add(node_area) > IMAGE_EDIT_AREA_LIMIT_PX;
if selected.is_empty() || !would_exceed {
selected.push(node);
*area = area.saturating_add(node_area);
} else {
return true;
}
}
for child in &node.children {
if collect_dfs_batch(child, false, root_extractable, terminal, selected, area) {
return true;
}
}
false
}
pub fn next_image_batch_with_size<'a>(
state: &SeparationState,
tree: &'a SeparationTree,
) -> ImageBatch<'a> {
let terminal = terminal_ids(state);
let mut selected = Vec::new();
let mut area = 0;
collect_dfs_batch(
&tree.root,
true,
tree.root_extractable,
&terminal,
&mut selected,
&mut area,
);
let image_edit_dimension_px = image_edit_dimension_for_area(area);
app_log!(
"ui_separation.batch_selected image_id={} image_nodes={} area_px={} image_edit_dimension_px={}",
tree.src_ui_design.as_str(),
selected.len(),
area,
image_edit_dimension_px
);
ImageBatch {
nodes: selected,
area_px: area,
image_edit_dimension_px,
}
}
pub fn next_image_batch<'a>(
state: &SeparationState,
tree: &'a SeparationTree,
) -> Vec<&'a SeparationNode> {
next_image_batch_with_size(state, tree).nodes
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn image_edit_dimension_uses_minimum_and_alignment() {
assert_eq!(image_edit_dimension_for_area(0), 816);
assert_eq!(image_edit_dimension_for_area(1), 816);
assert_eq!(
image_edit_dimension_for_area(IMAGE_EDIT_AREA_LIMIT_PX / 16),
816
);
assert_eq!(
image_edit_dimension_for_area(IMAGE_EDIT_AREA_LIMIT_PX),
2880
);
assert_eq!(image_edit_dimension_for_area(u64::MAX), 2880);
}
#[test]
fn image_edit_dimension_rounds_down_to_sixteen_pixels() {
let max_squared = IMAGE_EDIT_MAX_DIMENSION_PX * IMAGE_EDIT_MAX_DIMENSION_PX;
let area_just_below_1536 = IMAGE_EDIT_AREA_LIMIT_PX * 1536 * 1536 / max_squared;
let area_at_1536 = (IMAGE_EDIT_AREA_LIMIT_PX * 1536 * 1536).div_ceil(max_squared);
assert_eq!(image_edit_dimension_for_area(area_just_below_1536), 1520);
assert_eq!(image_edit_dimension_for_area(area_at_1536), 1536);
}
}
@@ -0,0 +1,164 @@
use crate::config::{build_game_creator_llm_client_from_llm_config, load_game_creator_app_config};
use crate::ui_editor::commands::separation::image_preprocess;
use crate::ui_editor::commands::separation::prompt::gen_binding_prompt;
use crate::ui_editor::commands::separation::{
validate_binding_response, BindingResp, SeparationNode,
};
use crate::ui_editor::commands::utils::{
parse_limited_llm_tool_arguments, request_ui_editor_llm, run_with_repair_history,
strict_json_schema,
};
use platform_llm::{
LlmFunctionTool, LlmMessage, LlmMessageContentPart, LlmRunRequest, LlmToolChoice,
};
use std::path::PathBuf;
use std::time::Instant;
pub(super) async fn visual_binding(
source_url: String,
processed_url: String,
sidecar: PathBuf,
nodes: &[&SeparationNode],
processed_dimensions: (u32, u32),
) -> Result<BindingResp, String> {
let started = Instant::now();
let result = visual_binding_inner(
source_url,
processed_url,
sidecar,
nodes,
processed_dimensions,
)
.await;
app_log!(
"ui_separation.visual_binding.timing outcome={} elapsed_ms={} nodes={}",
if result.is_ok() { "ok" } else { "error" },
started.elapsed().as_millis(),
nodes.len()
);
result
}
async fn visual_binding_inner(
source_url: String,
processed_url: String,
sidecar: PathBuf,
nodes: &[&SeparationNode],
processed_dimensions: (u32, u32),
) -> Result<BindingResp, String> {
app_log!(
"ui_separation.visual_binding.start nodes={} source_url_chars={} processed_url_chars={}",
nodes.len(),
source_url.chars().count(),
processed_url.chars().count()
);
let preprocess_started = Instant::now();
let binding_processed_url =
match image_preprocess::preprocess_for_visual_binding(processed_url, sidecar).await {
Ok(value) => {
app_log!(
"ui_separation.visual_binding.preprocess.timing outcome=ok elapsed_ms={}",
preprocess_started.elapsed().as_millis()
);
value
}
Err(error) => {
app_log!(
"ui_separation.visual_binding.preprocess.timing outcome=error elapsed_ms={}",
preprocess_started.elapsed().as_millis()
);
app_log!("ui_separation.error stage=visual_binding_preprocess error={error}");
return Err(error);
}
};
let llm_config = load_game_creator_app_config()
.map_err(|e| {
app_log!("ui_separation.error stage=visual_binding reason=load_config error={e}");
e.to_string()
})?
.llm;
let client =
build_game_creator_llm_client_from_llm_config(&llm_config, "llm").map_err(|e| {
app_log!("ui_separation.error stage=visual_binding reason=build_client error={e}");
e.to_string()
})?;
let schema = strict_json_schema::<BindingResp>().map_err(|error| {
app_log!("ui_separation.error stage=visual_binding reason=build_schema error={error}");
error
})?;
let tool = LlmFunctionTool::new(
"bind_ui_elements",
"确认处理图中的区域对应哪些 UI 节点",
schema,
)
.with_strict(true);
let initial_history = vec![
LlmMessage::system(gen_binding_prompt(nodes.to_vec())),
LlmMessage::user_multimodal(vec![
LlmMessageContentPart::InputText {
text: "processed image:".to_string(),
},
LlmMessageContentPart::InputImage {
image_url: binding_processed_url,
},
LlmMessageContentPart::InputText {
text: "src image:".to_string(),
},
LlmMessageContentPart::InputImage {
image_url: source_url.clone(),
},
]),
];
// 严格工具 schema 由 provider 负责约束正常模型输出;这里的解析失败只代表极小概率
// 的传输/响应损坏,因此沿用有限重试,不再为理论上的坏载荷扩展业务修复协议。
let result = run_with_repair_history(
2,
initial_history,
|history| {
let tool = tool.clone();
let client = client.clone();
let llm_config = llm_config.clone();
async move {
let request = LlmRunRequest::new(history)
.with_function_tools(vec![tool.clone()])
.with_tool_choice(LlmToolChoice::Required);
let request_started = Instant::now();
let response = request_ui_editor_llm(&client, &llm_config, request).await;
app_log!(
"ui_separation.llm.timing outcome={} elapsed_ms={}",
if response.is_ok() { "ok" } else { "error" },
request_started.elapsed().as_millis()
);
response
.map_err(|e| e.to_string())
.and_then(|response| {
response
.tool_calls
.into_iter()
.find(|call| call.name == "bind_ui_elements")
.map(|call| call.arguments)
.ok_or_else(|| "视觉绑定模型未返回工具调用".to_string())
})
.and_then(|arguments| parse_limited_llm_tool_arguments(&arguments))
.and_then(|args| {
serde_json::from_value::<BindingResp>(args)
.map_err(|e| format!("视觉绑定结果无效:{e}"))
})
}
},
|value: &BindingResp| validate_binding_response(value, nodes, processed_dimensions),
)
.await;
match &result {
Ok(value) => app_log!(
"ui_separation.visual_binding.completed nodes={} decisions={}",
nodes.len(),
value.decisions.len()
),
Err(error) => app_log!(
"ui_separation.error stage=visual_binding reason=failed nodes={} error={error}",
nodes.len()
),
}
result
}
@@ -0,0 +1,132 @@
use crate::ui_editor::commands::separation::area::normalize_binding_area;
use crate::ui_editor::commands::separation::model::BindingArea;
use image::ImageFormat;
use std::path::{Path, PathBuf};
use std::time::Instant;
pub(super) async fn cut_processed_image(
source: PathBuf,
area: BindingArea,
target: PathBuf,
) -> Result<(), String> {
let started = Instant::now();
let result = cut_processed_image_inner(source, area, target).await;
app_log!(
"ui_separation.cut_image.timing outcome={} elapsed_ms={}",
if result.is_ok() { "ok" } else { "error" },
started.elapsed().as_millis()
);
result
}
async fn cut_processed_image_inner(
source: PathBuf,
area: BindingArea,
target: PathBuf,
) -> Result<(), String> {
app_log!(
"ui_separation.cut_image.start source_file={} target_file={} area=({}, {}, {}, {})",
source
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
target
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
area.global_pos_x_px,
area.global_pos_y_px,
area.width_px,
area.height_px
);
tokio::task::spawn_blocking(move || cut_processed_image_blocking(&source, &area, &target))
.await
.map_err(|error| format!("裁切处理图任务失败:{error}"))?
}
fn cut_processed_image_blocking(
source: &Path,
area: &BindingArea,
target: &Path,
) -> Result<(), String> {
let image = image::open(source)
.map_err(|e| format!("读取处理图失败:{e}"))?
.to_rgba8();
let normalized = normalize_binding_area(&image, *area)?;
if normalized.transparent {
return Err("分离区域没有可见像素".to_string());
}
let original_area = *area;
let normalized_area = normalized.area;
app_log!(
"ui_separation.cut_image.normalized changed={} clamped={} transparent={} original_area=({}, {}, {}, {}) normalized_area=({}, {}, {}, {})",
normalized.changed,
normalized.clamped,
normalized.transparent,
original_area.global_pos_x_px,
original_area.global_pos_y_px,
original_area.width_px,
original_area.height_px,
normalized_area.global_pos_x_px,
normalized_area.global_pos_y_px,
normalized_area.width_px,
normalized_area.height_px
);
let cropped = image::imageops::crop_imm(
&image,
normalized_area.global_pos_x_px,
normalized_area.global_pos_y_px,
normalized_area.width_px,
normalized_area.height_px,
)
.to_image();
cropped
.save_with_format(target, ImageFormat::Png)
.map_err(|e| format!("写入 cut 图片失败:{e}"))?;
app_log!(
"ui_separation.cut_image.completed target_file={} width={} height={}",
target
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
normalized_area.width_px,
normalized_area.height_px
);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use image::{Rgba, RgbaImage};
#[test]
fn cuts_using_small_processed_image_dimensions() {
let directory = tempfile::tempdir().expect("创建临时目录失败");
let source = directory.path().join("processed.png");
let target = directory.path().join("cut.png");
let mut image = RgbaImage::from_pixel(816, 816, Rgba([0, 0, 0, 0]));
for y in 120..152 {
for x in 700..800 {
image.put_pixel(x, y, Rgba([255, 255, 255, 255]));
}
}
image.save(&source).expect("写入处理图失败");
cut_processed_image_blocking(
&source,
&BindingArea {
global_pos_x_px: 700,
global_pos_y_px: 120,
width_px: 100,
height_px: 32,
},
&target,
)
.expect("裁切处理图失败");
let cropped = image::open(target).expect("读取 cut 图片失败");
assert_eq!(cropped.width(), 100);
assert_eq!(cropped.height(), 32);
}
}
@@ -0,0 +1,181 @@
use crate::platform_session::PlatformSessionSnapshot;
use base64::Engine as _;
use serde::Deserialize;
use std::path::PathBuf;
use std::time::Instant;
use std::{fs, io::Cursor};
#[derive(Deserialize)]
struct RawEditResponse {
data: Vec<RawEditItem>,
}
#[derive(Deserialize)]
struct RawEditItem {
b64_json: String,
}
pub(super) async fn raw_extract(
session: &PlatformSessionSnapshot,
image_data_url: &str,
prompt: &str,
width: u32,
height: u32,
) -> Result<String, String> {
let started = Instant::now();
let result = raw_extract_inner(session, image_data_url, prompt, width, height).await;
app_log!(
"ui_separation.image_edit.timing outcome={} elapsed_ms={} width={} height={}",
if result.is_ok() { "ok" } else { "error" },
started.elapsed().as_millis(),
width,
height
);
result
}
async fn raw_extract_inner(
session: &PlatformSessionSnapshot,
image_data_url: &str,
prompt: &str,
width: u32,
height: u32,
) -> Result<String, String> {
app_log!(
"ui_separation.image_edit.start width={} height={} prompt_chars={}",
width,
height,
prompt.chars().count()
);
let (mime, data) = image_data_url
.split_once(',')
.ok_or_else(|| "界面图 data URL 无效".to_string())?;
let mime = mime
.strip_prefix("data:")
.and_then(|value| value.strip_suffix(";base64"))
.unwrap_or("image/png");
if !mime.eq_ignore_ascii_case("image/png") {
return Err("图片分离请求只支持 PNG 源图".to_string());
}
let image_bytes = base64::engine::general_purpose::STANDARD
.decode(data.trim())
.map_err(|error| format!("解码源图失败:{error}"))?;
if image_bytes.is_empty() {
return Err("源图不能为空".to_string());
}
let client = crate::http_client::agc_main_site_client_builder()
.build()
.map_err(|error| format!("创建图片编辑客户端失败:{error}"))?;
let url = format!(
"{}/api/raw/v1/images/edit",
session.api_base_url.trim_end_matches('/')
);
let image_part = reqwest::multipart::Part::bytes(image_bytes)
.file_name("image.png")
.mime_str("image/png")
.map_err(|error| format!("构造图片编辑文件部件失败:{error}"))?;
let body = reqwest::multipart::Form::new()
.part("image", image_part)
.text("prompt", prompt.to_string())
.text("width", width.to_string())
.text("height", height.to_string())
.text("output_format", "png")
.text("background", "transparent");
let response = crate::http_client::with_agc_main_site_marker(
client
.post(url)
.bearer_auth(&session.access_token)
.multipart(body),
)
.send()
.await
.map_err(|error| {
app_log!("ui_separation.error stage=image_edit reason=send error={error}");
format!("图片分离请求失败:{error}")
})?;
if !response.status().is_success() {
app_log!(
"ui_separation.error stage=image_edit reason=http_status status={}",
response.status()
);
return Err(format!("图片分离请求失败(HTTP {}", response.status()));
}
let payload = response.json::<RawEditResponse>().await.map_err(|error| {
app_log!("ui_separation.error stage=image_edit reason=parse_response error={error}");
format!("解析图片分离响应失败:{error}")
})?;
let result = payload
.data
.into_iter()
.next()
.map(|item| format!("data:image/png;base64,{}", item.b64_json))
.ok_or_else(|| "图片分离响应没有图像".to_string());
match &result {
Ok(value) => app_log!(
"ui_separation.image_edit.completed data_url_chars={}",
value.chars().count()
),
Err(error) => {
app_log!("ui_separation.error stage=image_edit reason=empty_result error={error}")
}
}
result
}
pub(super) async fn write_processed_image(
processed_url: String,
target: PathBuf,
) -> Result<(u32, u32), String> {
let started = Instant::now();
let result = write_processed_image_inner(processed_url, target).await;
app_log!(
"ui_separation.processed_image.write.timing outcome={} elapsed_ms={}",
if result.is_ok() { "ok" } else { "error" },
started.elapsed().as_millis()
);
result
}
async fn write_processed_image_inner(
processed_url: String,
target: PathBuf,
) -> Result<(u32, u32), String> {
app_log!(
"ui_separation.processed_image.write.start target_file={} data_url_chars={}",
target
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
processed_url.chars().count()
);
tokio::task::spawn_blocking(move || {
let encoded = processed_url
.split_once(',')
.map(|(_, data)| data)
.ok_or_else(|| "处理图 data URL 无效".to_string())?;
let processed_bytes = base64::engine::general_purpose::STANDARD
.decode(encoded)
.map_err(|error| format!("解析处理图失败:{error}"))?;
let dimensions = image::ImageReader::new(Cursor::new(processed_bytes.as_slice()))
.with_guessed_format()
.map_err(|error| format!("识别处理图格式失败:{error}"))?
.into_dimensions()
.map_err(|error| format!("读取处理图尺寸失败:{error}"))?;
let byte_len = processed_bytes.len();
fs::write(&target, processed_bytes)
.map_err(|error| format!("写入处理图失败:{}: {error}", target.display()))
.map(|_| {
app_log!(
"ui_separation.processed_image.write.completed target_file={} bytes={}",
target
.file_name()
.and_then(|name| name.to_str())
.unwrap_or("<unknown>"),
byte_len
);
dimensions
})
})
.await
.map_err(|error| format!("写入处理图任务失败:{error}"))?
}

Some files were not shown because too many files have changed in this diff Show More