3 Commits

Author SHA1 Message Date
Ukawka b3b51a48cf 增加测试数据 2026-09-23 20:54:36 +08:00
Ukawka 00f476022d 第三阶段改进 2026-09-22 22:18:34 +08:00
Ukawka 8080d52a99 接通第四步 2026-09-22 01:02:21 +08:00
17 changed files with 1828 additions and 88 deletions
+3
View File
@@ -16,3 +16,6 @@ references/external/kenney-platformer/
# OS/editor noise # OS/editor noise
.DS_Store .DS_Store
#data
data/
+42 -20
View File
@@ -20,11 +20,11 @@
## 跨会话上下文与阅读顺序 ## 跨会话上下文与阅读顺序
本节于 2026-09-21 根据当前代码与已检查产物整理。它是后续对话的上下文入口;后续修改应同步更新对应状态,实际源码、配置和具体实验 manifest 优先于历史总结。不要把旧实验统计套到新输出上。 本节于 2026-09-22 接入通用第四阶段及 adaptive 补全后更新。它是后续对话的上下文入口;后续修改应同步更新对应状态,实际源码、配置和具体实验 manifest 优先于历史总结。不要把旧实验统计套到新输出上。
1. 先阅读本文件和 `git status --short`,保留用户已有修改。 1. 先阅读本文件和 `git status --short`,保留用户已有修改。
2. 阅读 [README](README.md) 了解项目、环境、配置与命令。 2. 阅读 [README](README.md) 了解项目、环境、配置与命令。
3. 按任务阅读 [第一阶段](docs/1-understanding-stage.md)、[第二阶段](docs/2-segmentation-stage.md)、[第三阶段](docs/3-completion-stage.md),再核对相关实现与实际产物。 3. 按任务阅读 [第一阶段](docs/1-understanding-stage.md)、[第二阶段](docs/2-segmentation-stage.md)、[第三阶段](docs/3-completion-stage.md)、[第四阶段](docs/4-spine-stage.md),再核对相关实现与实际产物。
4. 用户说“继续”时,先核对工作区和最近任务进展,不能只凭历史总结重复请求付费模型或覆盖实验。 4. 用户说“继续”时,先核对工作区和最近任务进展,不能只凭历史总结重复请求付费模型或覆盖实验。
## 用户目标与固定偏好 ## 用户目标与固定偏好
@@ -37,8 +37,8 @@
## 文档维护约定 ## 文档维护约定
- 各文档文件名保持不变。`docs/` 当前保留三份阶段文档;`docs/model-stages.md` 在本次文档整理前已处于用户删除状态,不自行恢复,相关说明归入 README 和阶段文档。 - 原有文档文件名保持不变。`docs/` 保留前三份阶段文档,并新增 `4-spine-stage.md`;`docs/model-stages.md` 在此前文档整理前已处于用户删除状态,不自行恢复,相关说明归入 README 和阶段文档。
- 三份阶段文档有且仅有以下八个二级章节,细节必须归入对应章节: - 四份阶段文档有且仅有以下八个二级章节,细节必须归入对应章节:
1. 阶段简介 1. 阶段简介
2. 输入、输出与执行链路 2. 输入、输出与执行链路
3. 阶段执行命令示例及参数说明 3. 阶段执行命令示例及参数说明
@@ -63,9 +63,9 @@
- `configs/tiantoken.json`:独立 TIANTOKEN_BASE_URL / TIANTOKEN_API_KEY,地址 https://api.tiantoken.com;理解 gpt-6-astra,补全 gpt-image-2;模型变量为 TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL。 - `configs/tiantoken.json`:独立 TIANTOKEN_BASE_URL / TIANTOKEN_API_KEY,地址 https://api.tiantoken.com;理解 gpt-6-astra,补全 gpt-image-2;模型变量为 TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL。
- CLI 模型 > stage.model > 对应模型环境变量 > default_model;shell 同名变量优先于 .env。模型 ID 原样发送,不加隐式别名。 - CLI 模型 > stage.model > 对应模型环境变量 > default_model;shell 同名变量优先于 .env。模型 ID 原样发送,不加隐式别名。
- 用户提供的密钥仅留在本地忽略的 `.env`;禁止在文档、命令日志、聊天或提交中暴露,不要读取/打印完整 .env 来诊断。 - 用户提供的密钥仅留在本地忽略的 `.env`;禁止在文档、命令日志、聊天或提交中暴露,不要读取/打印完整 .env 来诊断。
- 三份配置的图片编辑能力均为 multiple_images=false、mask=false。接口已预留分支不代表中转已支持;开启前需验证。Astra 理解接口与图片编辑接口不同,当前没有 Astra Responses 图片生成工具链。 - 三份配置的图片编辑能力仍为 multiple_images=false、mask=false。adaptive auto 在此配置下发送单图参考板,包含局部和完整角色参考,mask 仅本地约束。独立多图/mask 和单部件探针已接入,但本轮未调用付费模型,不能宣称中转已支持。Astra 理解接口与图片编辑接口不同,当前没有 Astra Responses 图片生成工具链。
## 三阶段实现与不能丢失的边界 ## 四阶段实现与不能丢失的边界
### 第一阶段:理解与几何候选 ### 第一阶段:理解与几何候选
@@ -87,19 +87,34 @@
### 第三阶段:受限部件补全 ### 第三阶段:受限部件补全
- 入口 `--stage completion --input-dir SAM目录`;实现为 stages/completion-stage.mjs、completion_geometry.py、providers/image-edits.mjs。 - 入口 `--stage completion --input-dir SAM目录`;实现为 stages/completion-stage.mjs、completion_geometry.py、completion_planning.py、providers/image-edits.mjs。默认 completion-profile=adaptive,legacy 保留旧行为和候选兼容。
- 输入须含 source.png、v2 计划、segmentation.json 及所有记录的全画布 masks;单独 parts/ 不够。 - 输入须含 source.png、v2 计划、segmentation.json 及所有记录的全画布 masks;单独 parts/ 不够。
- 仅处理 needs_completion=true、非隐藏、有可见 mask 且允许区域非空的部件。--part 不绕过筛选;目前不重新评估 SAM 缺损。 - adaptive 从实际 mask 重建遮挡/关节、闭合漏分孔洞、附件底图清理需求,不单凭 needs_completion 筛选。自动孔洞只取原图前景且无其他 mask 归属的区域;开放缺口和歧义归属仍需复核。无明确遮挡的近重复 mask(IoU≥0.8)要求人工确认。
- 默认 1024 方形局部画布,原像素不缩放。允许区来自轮廓外 32 像素范围与遮挡 mask / pivot 圆盘交集,排除可见 mask;不是完整隐藏形状生成。 - adaptive preserve=SAM−editable−removed;仅保留区锁定原 RGBA,清理和替换区可重绘,人工 remove 可删除。completion-review JSON(schema_version=1、parts 映射、confirmed=true)支持 repair/replace/remove 全画布二值 masks 与部件半径,原文件及哈希留档,不改 SAM。
- 当前真实请求只发白底 input.png;context.png、edit-mask.png 默认不发送。开启 multiple_images 也只是发送同坐标局部 context,并非独立全角色参考。 - 默认请求边长 1024;adaptive 半径按 mask 短边×0.15 向上取整并限于 4~96 原像素,CLI 或部件 review 可覆盖。native 裁剪边长约目标包围框最长边×1.15;小部件放大、大部件缩小,保存正反仿射和工作区。逆缩放后恢复保留原像素,坐标可供第四阶段使用。
- reference auto:multiple_images=true 时 input→context→reference,默认 false 时发送 board(左目标,右局部和全角色参考),参考面板不进入最终资产。支持 none/local/full/both/board;mask auto/required/off 独立选择。request-plan.json 记录计划,request.json 仅实际发送时生成,含有序图片角色、哈希、尺寸、字段、mask 哈希与透明语义。
- legacy 才沿用 needs_completion、原像素不缩放、默认 32 半径和原可见像素全保护。旧候选复检必须显式 --completion-profile legacy。新旧画布/参考布局/复核文件不同不能混用候选。
- 图片解码已修复“空 b64_json 但有有效 URL”情况,回退下载 URL;该问题与画布尺寸不符不同。 - 图片解码已修复“空 b64_json 但有有效 URL”情况,回退下载 URL;该问题与画布尺寸不符不同。
- 在线请求现显式传 size,配置冲突失败。strict 默认拒绝尺寸不同;uniform 仅恢复同宽高比整画布尺寸,保留原候选,继续原有验收,不单独移动/缩放物体。 - 在线请求现显式传 size,配置冲突失败。strict 默认拒绝尺寸不同;uniform 仅恢复同宽高比整画布尺寸,保留原候选,继续原有验收,不单独移动/缩放物体。
- 验收阈值:可见不透明像素 RGB MAE≤30、保留率≥95%;有效新增须在允许区且连接可见部件;原可见 RGBA 随后逐像素恢复。 - 对齐阈值保持 RGB MAE≤30、保留率≥95%;adaptive 在请求工作区的保留不透明像素上检查,再逆采样,避免把插值 Alpha 误作位移。编辑像素必须位于允许区且连接保留区;可见缺口使用原图纹理,移除区清零。
- unfulfilled_regions 当前只记录、不硬拒绝;语义错误、接缝、关节补够与否尚无可靠验收。错误 SAM 可见像素会被强制保留。 - adaptive 清理/人工替换要求≥95% 有效填充,其余 unfulfilled_regions 仍仅诊断。语义、接缝和关节质量不能由填充率证明。未明确编辑的误分仍被保护,需复核文件修正。
- --probe-edit-capabilities 要求 adaptive 在线且恰好一个 --part,发送主图/局部/全角色+mask,n=1、maxRetries=0。记录 capability-evidence 和合并前 raw_outside_edit 指标;不会把 reference_use_verified/mask_semantics_verified 自动置 true 或修改配置。prepare-only/mock 不是中转实测。
- prepare-only 不联网;candidates-dir 可读平铺 ID.png 或匹配 manifest 的旧任务首张候选。--resume 不是失败断点续发,准备/生成/复检使用独立目录。 - prepare-only 不联网;candidates-dir 可读平铺 ID.png 或匹配 manifest 的旧任务首张候选。--resume 不是失败断点续发,准备/生成/复检使用独立目录。
- parts/ 只含通过检查的补全部件,不包含全部原拆件。完整角色组装、旋转验收、新管线 Spine 导出尚未接入。 - parts/ 只含通过检查的补全部件,不包含全部原拆件。完整角色组装、Spine 导出与旋转检查动画已独立接入第四阶段;运动质量自动验收仍未实现。
- 旧 `--stage completion --image ... --prompt-file ...` 是接口探针,不能和 input-dir 混用,不执行完整几何验收。 - 旧 `--stage completion --image ... --prompt-file ...` 是接口探针,不能和 input-dir 混用,不执行完整几何验收。
### 第四阶段:通用组装与 Spine 导出
- 用户明确要求可复用管线,不能写死具体角色、部件 ID、数量或人形比例。入口 `--stage spine --input-dir SAM目录 --completion-dir 补全目录 --out-dir 新目录`;completion-dir 可省略。
- `stages/spine-stage.mjs` 与 `spine_assembly.py` 完全离线,校验 v2、输入快照及 masks 哈希、PNG 和裁剪坐标;不读取模型配置/密钥,不调用 SAM。
- 原资产从 source + mask 重建,采用补全结果前检查 selected_file、状态、qa.accepted、image、crop_xyxy。legacy 全可见 RGBA 保持;adaptive 验证 preservation 分区、保留 RGBA 精确匹配、编辑/移除互斥及输出不越界,契约 mask 哈希留档。不能混用其他实验。
- 默认补全失败回退 SAM 并记录原因;fallback-policy=error 拒绝需补全但无通过结果的可见部件。缺可见 mask 直接报错;隐藏骨骼保留但不生成图片。
- pivot 转换为底边中点原点的 Spine 坐标;骨骼 setup 旋转为零,局部位置取父子 pivot 差;附件中心相对绑定骨骼偏移。bone_part 使用 parent,其他角色使用 bind_to。
- occluded_by 稳定拓扑排序;无约束重叠写入诊断,可用 draw-order-file 全量覆盖,冲突留档。骨骼/遮挡环路拒绝。
- 导出完整 images、assembly.json、静态组装/对比/覆盖诊断、skeleton.json、atlas、report.html 和真实 preview.html。依赖外部只读图集和播放器脚本,其哈希纳入复用指纹。
- 目标 Spine 4.2;setup 静态姿态及 check__ID 单骨骼检查动画。没有成品 idle/walk、mesh/weights/IK,semantic_verified=false、quality_status=needs_review;几何通过和可加载不代表资产合格。
- 生成离线;播放器从 CDN 加载资源,打开骨架预览需要网络。静态报告可离线查看。
## 已检查实验与质量结论 ## 已检查实验与质量结论
以下为已有实验记录,不代表后续每次运行的结果;不必为了文档修改重跑 GPU 或付费模型。 以下为已有实验记录,不代表后续每次运行的结果;不必为了文档修改重跑 GPU 或付费模型。
@@ -108,30 +123,37 @@
| --- | --- | | --- | --- |
| output/sombrero-grounding-v2 | 旧计划离线几何检查,28 个程序候选通过前景检查;未证明语义正确 | | output/sombrero-grounding-v2 | 旧计划离线几何检查,28 个程序候选通过前景检查;未证明语义正确 |
| output/sombrero-sam31-v3 | 较早 SAM 实验,28 个 mask,约 67.18% 覆盖率;肢体/鞋子重复、漏分,不合格 | | output/sombrero-sam31-v3 | 较早 SAM 实验,28 个 mask,约 67.18% 覆盖率;肢体/鞋子重复、漏分,不合格 |
| output/20260918-183532/sam | 当前讨论基线:32 个计划项、10 个 hidden、22 个 mask;约 83.3% 覆盖,32719 像素未分配;前臂/手混入衣料、袖子与上衣缺损 | | output/20260918-183532/sam | 较早基线:32 个计划项、10 个 hidden、22 个 mask;约 83.3% 覆盖,32719 像素未分配;前臂/手混入衣料、袖子与上衣缺损 |
| output/20260918-183532/complete2_gpt | 真实 Tiantoken gpt-image-2 请求 12 部件;旧请求未传 size,返回 1248/1254,预期 1024,全部 CANVAS_SIZE_MISMATCH | | output/20260918-183532/complete2_gpt | 真实 Tiantoken gpt-image-2 请求 12 部件;旧请求未传 size,返回 1248/1254,预期 1024,全部 CANVAS_SIZE_MISMATCH |
| output/20260918-183532/complete2_gpt-size-recheck-v2 | 离线 uniform 复检,未重生成:8 个几何通过;head、pants_thigh_l、pants_thigh_r、shoe_l 4 个拒绝;通过项仍有误分和未完成区域 | | output/20260918-183532/complete2_gpt-size-recheck-v2 | 离线 uniform 复检,未重生成:8 个几何通过;head、pants_thigh_l、pants_thigh_r、shoe_l 4 个拒绝;通过项仍有误分和未完成区域 |
| output/20260918-183532/complete-recovered-head-check | 更早 Qwen 头部候选恢复检查,不能与 GPT 批次混为一谈 | | output/20260918-183532/complete-recovered-head-check | 更早 Qwen 头部候选恢复检查,不能与 GPT 批次混为一谈 |
| output/20260920-225137 | 较新实验:27 个语义项、2 个 hidden、25 个 SAM 部件;补全 16 项中 13 个几何通过、3 个对齐拒绝 |
| output/spine-validation-latest-20260922 | 第四阶段离线验证:25 个 slot、16 个骨骼(含 root);采用 13 个补全部件,5 个需补全部件回退 SAM(含两袖无允许区域);上游缺口仍存在 |
| output/spine-validation-baseline-20260922 | 另一批输入验证:22 个 slot、16 个骨骼(含 root);采用 8 个补全部件、4 个回退;输入源为旧基线和 size-recheck-v2 |
| output/completion-adaptive-prepare-20260922 | adaptive 离线准备:16 个任务、11 个跳过;上衣规划 3708 个封闭漏分像素,头部规划 10205 个附件清理像素;这些是待复核区域,不是补全成功结果,未请求模型 |
- 当前 22→12 的原因:sombrero、sleeve_l、sleeve_r、eye_l、eye_r、eyebrow_l、eyebrow_r、nose、mouth、mustache 共 10 个被第一阶段标记 NO_COMPLETION_NEEDED;不能解释为它们的实际 SAM mask 已完整。 - 当前 22→12 的原因:sombrero、sleeve_l、sleeve_r、eye_l、eye_r、eyebrow_l、eyebrow_r、nose、mouth、mustache 共 10 个被第一阶段标记 NO_COMPLETION_NEEDED;不能解释为它们的实际 SAM mask 已完整。
- 2026-09-20 尺寸修复后曾通过 22 项 Node 测试(含 Python 补全回归)。这是历史验证;新改动按范围重新验证,不自动沿用“全部通过”。显式 size 构造已测试,修复后中转是否遵守尚无新实测。 - 2026-09-20 尺寸修复后曾通过 22 项 Node 测试(含 Python 补全回归)。这是历史验证;新改动按范围重新验证,不自动沿用“全部通过”。显式 size 构造已测试,修复后中转是否遵守尚无新实测。
- 总体判断:上游部件归属、模型缺少原图参考与遮罩、区域规则粗糙、验收不足共同导致低质量。不能仅换模型、强制处理全部 mask、放宽阈值或解决尺寸错误就宣称修好。 - 总体判断:上游部件归属、模型缺少原图参考与遮罩、区域规则粗糙、验收不足共同导致低质量。不能仅换模型、强制处理全部 mask、放宽阈值或解决尺寸错误就宣称修好。
- 2026-09-22 第四阶段接入后,Conda ai-spine 下 npm test 共 26 项通过(含新增 6 项 Python 组装用例)。另使用官方 Spine 4.2 core IIFE 对上述两份导出分别验证 setup 顶点和 30 个单关节旋转采样,全部通过。最新导出的 preview.html 已在 headless Chrome 实际加载渲染,截图为 output/spine-runtime-validation-20260922/preview.png;这不是动作或资产质量验收。复检工具为 `tools/verify-spine-runtime.mjs`,需调用者提供本地官方 runtime 文件,不自动下载/安装依赖。
- 随后接入 adaptive 补全,Conda ai-spine 下 npm test 共 29 项全部通过(含 8 项新增 Python adaptive 用例):覆盖参考顺序/mask mock、放大/缩小、位移拒绝、实际缺口、清理与区域契约、候选身份及第四阶段 strict 回退。未进行付费中转试验,真实参考使用与 mask 语义仍待验证。
## 当前改进方向与后续顺序 ## 当前改进方向与后续顺序
本轮用户要求重整文档,以下功能是待实现需求,不因写入文档就成为已完成功能: 当前用户已要求落实第三阶段的参考图/区域重建/自适应画布,必须保持可复用且与第四阶段贯通。当前状态:
1. 第二阶段输出仅色彩遮罩组成的完整角色图,保留当前原图叠色预览,并明确未分配/重叠区域。 1. 第二阶段输出仅色彩遮罩组成的完整角色图,保留当前原图叠色预览,并明确未分配/重叠区域。
2. 第二阶段每个 SAM 部件可选经过一次大模型去毛边,保留原始拆件、原图参考、坐标、变更区域与前后对照。 2. 第二阶段每个 SAM 部件可选经过一次大模型去毛边,保留原始拆件、原图参考、坐标、变更区域与前后对照。
3. 第三阶段将角色原图或部件相关原图局部作为参考交给补全模型,验证中转多图/遮罩能力及实际发送内容。 3. 第三阶段参考图、实际 mask 规划、自适应区域/可逆缩放、发送审计和单部件能力探针已实现并离线验证;真实多图/mask 语义和新布局生成质量仍待付费小样实测,默认 capabilities 不变。
4. 第三阶段输出使用补全部件组装的完整角色,复用无需补全的原部件,明确失败/回退状态。 4. 完整角色组装及失败回退记录已在第四阶段实现;接缝与关节旋转质量自动验收仍待实现。
5. 技术优先级:先修正 SAM 部件归属与输入门槛,再按实际 mask 重建补全计划;用头、前臂、上衣三类小样验证参考图和遮罩,然后增加回贴与关节旋转验收,最后批量与 Spine 导出。 5. 后续质量工作:复核自动规划和 SAM 归属、实测参考图与遮罩、加强接缝/动作验收。第三/四阶段遵循统一输入契约,不为某个实验硬编码修正。
详细改进说明保留在各阶段文档第 8 节;若后续用户改变优先级,应更新此处。 详细改进说明保留在各阶段文档第 8 节;若后续用户改变优先级,应更新此处。
## 旧管线与范围识别 ## 旧管线与范围识别
- `pipeline.mjs` 未指定 --mode annotated-parts 时走旧“生成拆件图→CV 定位→Spine”流程,不是新三阶段自动串联。 - `pipeline.mjs` 未指定 --mode annotated-parts 时走旧“生成拆件图→CV 定位→Spine”流程,不是新四阶段自动串联。
- `segment.py` 为旧拆件图分割;`compose_preview.py` 为旧布局组装;`run_case_matrix.py` 为旧定位对照工具。不要因为 IDE 当前打开这些文件就误改新阶段入口。 - `segment.py` 为旧拆件图分割;`compose_preview.py` 为旧布局组装;`run_case_matrix.py` 为旧定位对照工具。不要因为 IDE 当前打开这些文件就误改新阶段入口。
- 相邻 `spine-animation-ai` 为外部只读依赖,保留其 PolyForm Noncommercial 许可证约束;新阶段尚未复用为完成的导出流程。 - 相邻 `spine-animation-ai` 为外部只读依赖,保留其 PolyForm Noncommercial 许可证约束;第四阶段复用图集和播放器脚本,不复用旧特征定位或 root 平铺骨架逻辑。
- 不要把文档改写任务扩大成付费图片生成、SAM 推理、依赖安装或功能实现;需要实现时遵从用户后续任务范围。 - 不要把文档改写任务扩大成付费图片生成、SAM 推理、依赖安装或功能实现;需要实现时遵从用户后续任务范围。
+32 -8
View File
@@ -1,16 +1,17 @@
# AI Spine 角色拆件与动画管线 # AI Spine 角色拆件与动画管线
本项目探索将单张 2D 角色图转换为可用于 Spine 动画的部件与骨骼资源。当前主线分为三个阶段:视觉 LLM 理解结构、本地 SAM 分割可见部件、图片编辑模型补全隐藏区域和关节连接。 本项目探索将单张 2D 角色图转换为可用于 Spine 动画的部件与骨骼资源。当前主线分为四个阶段:视觉 LLM 理解结构、本地 SAM 分割可见部件、图片编辑模型补全隐藏区域和关节连接、离线组装与 Spine 导出。
| 阶段 | 输入 → 输出 | 当前状态 | | 阶段 | 输入 → 输出 | 当前状态 |
| --- | --- | --- | | --- | --- | --- |
| 1. 结构理解 | PNG → v2 语义计划、粗框、pivot、程序候选点 | 可在线理解或离线复检;语义待复核 | | 1. 结构理解 | PNG → v2 语义计划、粗框、pivot、程序候选点 | 可在线理解或离线复检;语义待复核 |
| 2. SAM 分割 | 第一阶段目录 → mask、原像素透明部件、标注图与拆解图 | 本机 GPU 实测可运行;存在误分与漏分 | | 2. SAM 分割 | 第一阶段目录 → mask、原像素透明部件、标注图与拆解图 | 本机 GPU 实测可运行;存在误分与漏分 |
| 3. 部件补全 | SAM 目录 → 局部编辑候选、几何 QA、通过检查的补全部件 | 已真实调用图片模型;结果质量尚不合格 | | 3. 部件补全 | SAM 目录 → 局部编辑候选、几何 QA、通过检查的补全部件 | 已真实调用图片模型;结果质量尚不合格 |
| 4. Spine 导出 | SAM + 可选补全目录 → 完整资产、原位组装、骨骼、图集、预览 | 通用离线入口;含单骨骼旋转检查动画,资产与动作质量需复核 |
第三阶段尚未输出使用补全部件组装的完整角色,新管线 Spine 导出也未完成。执行成功、mask 覆盖率高、几何检查通过,均不等于动画资产可用。 第三阶段输出通过几何检查的补全部件;第四阶段整合所有可绘制部件、组装并导出 Spine 4.2。执行成功、mask 覆盖率高、几何检查通过,均不等于动画资产可用。
仓库保留旧版“生成白底拆件图 → OpenCV 拆分 → Genielabs 定位 → Spine 导出”作为对照;它与当前三阶段主线并存,产物不能直接混用。 仓库保留旧版“生成白底拆件图 → OpenCV 拆分 → Genielabs 定位 → Spine 导出”作为对照;它与当前四阶段主线并存,产物不能直接混用。
## 文档入口 ## 文档入口
@@ -18,6 +19,7 @@
- [第一阶段](docs/1-understanding-stage.md):理解、数据契约、Alpha 候选与粗标注。 - [第一阶段](docs/1-understanding-stage.md):理解、数据契约、Alpha 候选与粗标注。
- [第二阶段](docs/2-segmentation-stage.md):本地 SAM、mask 选择、拆件与质量边界。 - [第二阶段](docs/2-segmentation-stage.md):本地 SAM、mask 选择、拆件与质量边界。
- [第三阶段](docs/3-completion-stage.md):筛选、局部画布、图片请求、几何验收及补全改进。 - [第三阶段](docs/3-completion-stage.md):筛选、局部画布、图片请求、几何验收及补全改进。
- [第四阶段](docs/4-spine-stage.md):通用资产整合、坐标与骨骼、绘制顺序、Spine 导出与回退策略。
阶段文档统一采用八节:简介;输入输出与链路;命令参数;输入格式;重要代码原理;职责索引;产物;可改进点。配置与实验说明已归入上述文档。 阶段文档统一采用八节:简介;输入输出与链路;命令参数;输入格式;重要代码原理;职责索引;产物;可改进点。配置与实验说明已归入上述文档。
@@ -101,7 +103,7 @@ SAM_CODE_DIR=F:/Study/SAM3/sam3
CLI > 配置 stage.model > 对应模型环境变量 > default_model;shell 环境变量优先于 `.env`。不指定配置时入口默认 astra-qwen.json,建议实验始终显式指定。模型 ID 原样发送,中转能力以实际请求为准。Tiantoken 图片编辑已真实执行;第三阶段尚未适配 Astra Responses 图片生成工具。 CLI > 配置 stage.model > 对应模型环境变量 > default_model;shell 环境变量优先于 `.env`。不指定配置时入口默认 astra-qwen.json,建议实验始终显式指定。模型 ID 原样发送,中转能力以实际请求为准。Tiantoken 图片编辑已真实执行;第三阶段尚未适配 Astra Responses 图片生成工具。
三个配置当前均关闭多图参考与编辑 mask 能力。第三阶段默认只发送白底单部件图;本地 context.png/edit-mask.png 已生成但不默认发送。改进计划见第三阶段文档。 三个配置当前仍关闭原生多图与编辑 mask 能力。第三阶段默认 adaptive:实际 mask 重建需求、自适应半径和可逆缩放;当前配置下 auto 会发送含目标、局部参考、完整角色参考的单张参考板。只有支持原生多图时才独立发送参考图;mask 是否发送由配置和 completion-mask 决定,未发送时明确标记 local_only。不能把本地裁剪与恢复当成模型遵循 mask 的证据。
查询模型列表和检查配置: 查询模型列表和检查配置:
@@ -112,7 +114,7 @@ node .\tools\model-check.mjs --config .\configs\tiantoken.json --dry-run
前者联网,后者不请求模型;列表存在某个 ID 不代表其所有能力已验证。 前者联网,后者不请求模型;列表存在某个 ID 不代表其所有能力已验证。
## 三阶段执行命令 ## 四阶段执行命令
以下命令按顺序在同一个 PowerShell 会话执行。反引号后不能带空格。每次实验使用新的 runId;分阶段重开终端时,将路径变量设置成已有的对应目录。 以下命令按顺序在同一个 PowerShell 会话执行。反引号后不能带空格。每次实验使用新的 runId;分阶段重开终端时,将路径变量设置成已有的对应目录。
@@ -160,6 +162,8 @@ node .\tools\model-check.mjs --stage segmentation `
### 第三阶段:准备与补全 ### 第三阶段:准备与补全
默认 `--completion-profile adaptive` 会重新检查实际 SAM 缺口、骨骼连接和附件重叠,即使旧计划 needs_completion=false 也可能生成修复任务。先检查 preparation 报告的区域与参考图,再请求模型;缺口归属和附件底图仍需语义复核。
先离线准备任务,不请求模型: 先离线准备任务,不请求模型:
```powershell ```powershell
@@ -189,7 +193,9 @@ node .\tools\model-check.mjs --stage completion `
--out-dir "$runRoot\complete" --out-dir "$runRoot\complete"
``` ```
省略 part 不意味着强制补全全部 mask。检查 `completion-plan.json` 的跳过原因和 `completion-result.json` 的 QA;parts/ 仅包含通过检查的补全部件。所有结果仍需语义复核。 省略 part 不意味着强制补全全部 mask。检查 `completion-plan.json` 的 assessments/跳过原因、每个任务的 input.png、request-plan.json 和编辑区域;parts/ 仅包含通过检查的补全部件。所有结果仍需语义复核。`--completion-review FILE` 可按 ID 提供修复/替换/移除区域及部件半径,格式见[第三阶段文档](docs/3-completion-stage.md)。
`--completion-reference auto` 默认在原生多图可用时发送主图→局部参考→全角色参考,否则使用 board;可显式选择 board/local/full/both/none。`--completion-mask auto|required|off` 控制 mask 请求,auto 遵循配置。`--probe-edit-capabilities --part ID` 是一次真实多图+mask 小样,固定 n=1、无 HTTP 重试,保存原候选及能力证据,不自动启用 capabilities。该实测会产生费用;当前实现只完成离线验证。
旧候选尺寸不符时,可离线导入复检,避免重新生成: 旧候选尺寸不符时,可离线导入复检,避免重新生成:
@@ -199,10 +205,28 @@ node .\tools\model-check.mjs --stage completion `
--input-dir .\output\20260918-183532\sam ` --input-dir .\output\20260918-183532\sam `
--candidates-dir .\output\20260918-183532\complete2_gpt ` --candidates-dir .\output\20260918-183532\complete2_gpt `
--out-dir "$runRoot\complete-size-recheck" ` --out-dir "$runRoot\complete-size-recheck" `
--completion-profile legacy `
--candidate-size-policy uniform --candidate-size-policy uniform
``` ```
uniform 仅恢复同宽高比的整画布尺寸,不纠正物体位移、变形或语义。在线请求现已显式传 size;参数及中转实测边界见[第三阶段文档](docs/3-completion-stage.md)。 旧画布候选必须指定 legacy。adaptive 候选需匹配源图、masks、profile、参考布局、半径和复核文件哈希;不能把旧候选直接映射到新画布。uniform 仅恢复同宽高比整画布尺寸,不纠正物体位移、变形或语义。在线请求显式传 size;参数及中转实测边界见[第三阶段文档](docs/3-completion-stage.md)。
### 第四阶段:离线组装与 Spine 导出
接续本轮 SAM 与补全目录,无需模型配置、密钥或 GPU:
```powershell
node .\tools\model-check.mjs --stage spine `
--input-dir "$runRoot\sam" `
--completion-dir "$runRoot\complete" `
--out-dir "$runRoot\spine"
```
省略 `--completion-dir` 可只用 SAM 资产;加 `--dry-run` 完整检查输入而不写输出。默认补全失败回退 SAM 并记录原因;`--fallback-policy error` 要求需补全的可见部件全部有通过结果。可见部件缺 mask、混用不同输入或非法补全坐标会直接报错。
检查 `report.html`、`comparison.png` 和 `assembly.json`;`preview.html` 显示真实骨架,setup 为静态姿态,`check__骨骼ID` 为单关节旋转检查动画。生成过程离线;播放器页面从 CDN 加载 Spine Player,需要网络。`skeleton.json` 配合 `images/` 供编辑器导入,`atlas/` 用于运行时。
部件数量、名称、父子关系均由 v2 计划驱动;隐藏骨骼保留,附件按 bind_to 绑定。第四阶段只复用外部只读 `spine-animation-ai` 的图集和播放器脚本,保留其 PolyForm Noncommercial 约束。完整参数、输入契约和绘制顺序覆盖见[第四阶段文档](docs/4-spine-stage.md)。
## 旧管线与验证 ## 旧管线与验证
@@ -218,7 +242,7 @@ node .\pipeline.mjs `
--animations idle,walk,wave --animations idle,walk,wave
``` ```
`segment.py` 为旧版拆件分割,`compose_preview.py` 组装预览,`run_case_matrix.py` 对照定位方案;当前 SAM 入口为 sam_segment.py。旧产物需检查 assembled.png 和 preview-rigged.html,不能据此声称新三阶段已经完成 Spine 导出。 `segment.py` 为旧版拆件分割,`compose_preview.py` 组装预览,`run_case_matrix.py` 对照定位方案;当前 SAM 入口为 sam_segment.py。旧产物检查 assembled.png 和 preview-rigged.html;新管线导出使用独立的第四阶段入口。
测试与命令帮助: 测试与命令帮助:
+21 -5
View File
@@ -145,6 +145,9 @@ def merge_candidate(folder, candidate_file, output_file, size_policy='strict'):
raise ValueError('Unknown candidate size policy') raise ValueError('Unknown candidate size policy')
folder = Path(folder) folder = Path(folder)
task = read_json(folder/'task.json') task = read_json(folder/'task.json')
if task.get('profile') == 'adaptive':
from completion_planning import merge_adaptive
return merge_adaptive(folder, candidate_file, output_file, size_policy)
original = np.array(Image.open(folder/'visible.png').convert('RGBA')) original = np.array(Image.open(folder/'visible.png').convert('RGBA'))
allowed = np.array(Image.open(folder/'allowed-mask.png').convert('L')) > 0 allowed = np.array(Image.open(folder/'allowed-mask.png').convert('L')) > 0
candidate = Image.open(candidate_file).convert('RGBA') candidate = Image.open(candidate_file).convert('RGBA')
@@ -228,6 +231,9 @@ def finalize(directory):
meta = read_json(out/'tasks'/name/'task.json') meta = read_json(out/'tasks'/name/'task.json')
ox, oy = meta['origin_xy'] ox, oy = meta['origin_xy']
task.update(image=f'parts/{name}.png', crop_xyxy=[bbox[0]+ox, bbox[1]+oy, bbox[2]+ox, bbox[3]+oy]) task.update(image=f'parts/{name}.png', crop_xyxy=[bbox[0]+ox, bbox[1]+oy, bbox[2]+ox, bbox[3]+oy])
if meta.get('preservation'):
task['preservation'] = meta['preservation']
task['transform'] = meta['transform']
view = f'<img src="{task["image"]}">' if task.get('image') else '' view = f'<img src="{task["image"]}">' if task.get('image') else ''
for index, item in enumerate(task.get('candidates', []), 1): for index, item in enumerate(task.get('candidates', []), 1):
view += f'<p><a href="{html.escape(item["file"], quote=True)}">原候选 {index}</a> · <a href="tasks/{name}/qa-{index}.json">QA</a>' view += f'<p><a href="{html.escape(item["file"], quote=True)}">原候选 {index}</a> · <a href="tasks/{name}/qa-{index}.json">QA</a>'
@@ -235,11 +241,14 @@ def finalize(directory):
view += f' · <a href="tasks/{name}/merged-{index}-normalized.png">缩放后候选</a>' view += f' · <a href="tasks/{name}/merged-{index}-normalized.png">缩放后候选</a>'
view += '</p>' view += '</p>'
warnings = html.escape(', '.join(task.get('upstream_warnings', []))) warnings = html.escape(', '.join(task.get('upstream_warnings', [])))
rows.append(f'<tr><td>{html.escape(name)}<p>{warnings}</p><a href="tasks/{name}/task.json">任务参数</a> · <a href="tasks/{name}/prompt.txt">提示词</a></td><td>{html.escape(task["status"])}</td><td><a href="tasks/{name}/visible.png"><img src="tasks/{name}/visible.png"></a></td><td><a href="tasks/{name}/allowed-mask.png"><img src="tasks/{name}/allowed-mask.png"></a><p><a href="tasks/{name}/context.png">原图局部参考</a></p></td><td>{view}</td><td><pre>{html.escape(json.dumps(qa, ensure_ascii=False, indent=2))}</pre></td></tr>') reference = f'<p><a href="tasks/{name}/input.png">实际请求主图</a> · <a href="tasks/{name}/request-plan.json">发送计划</a></p>'
if (out/'tasks'/name/'reference.png').exists():
reference += f'<a href="tasks/{name}/reference.png">完整角色参考</a>'
rows.append(f'<tr><td>{html.escape(name)}<p>{warnings}</p><a href="tasks/{name}/task.json">任务参数</a> · <a href="tasks/{name}/prompt.txt">提示词</a>{reference}</td><td>{html.escape(task["status"])}</td><td><a href="tasks/{name}/visible.png"><img src="tasks/{name}/visible.png"></a></td><td><a href="tasks/{name}/allowed-mask.png"><img src="tasks/{name}/allowed-mask.png"></a><p><a href="tasks/{name}/context.png">原图局部参考</a></p></td><td>{view}</td><td><pre>{html.escape(json.dumps(qa, ensure_ascii=False, indent=2))}</pre></td></tr>')
write_json(out/'completion-result.json', result) write_json(out/'completion-result.json', result)
(out/'report.html').write_text('''<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>部件补全</title> (out/'report.html').write_text('''<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>部件补全</title>
<style>body{font:16px system-ui;margin:24px}td,th{border:1px solid #ccc;padding:8px}img{max-width:260px;max-height:300px;background:#eee}pre{white-space:pre-wrap}table{border-collapse:collapse}</style> <style>body{font:16px system-ui;margin:24px}td,th{border:1px solid #ccc;padding:8px}img{max-width:260px;max-height:300px;background:#eee}pre{white-space:pre-wrap}table{border-collapse:collapse}</style>
<h1>第三阶段:部件补全</h1><p>新增区域受允许掩码限制,原图可见像素逐像素恢复。几何通过仅表示候选可供复核,不代表遮挡和关节纹理已正确补全。上游 SAM 错分仍需修正。</p> <h1>第三阶段:部件补全</h1><p>输出受允许掩码限制,保留区域逐像素恢复;adaptive 允许明确规划的附件清理、替换和移除,原始 SAM 始终保留。几何通过仅表示候选可供复核,不代表遮挡和关节纹理已正确补全。</p>
<table><tr><th>部件 / 上游告警</th><th>状态</th><th>原可见区域</th><th>允许新增区域(白色)</th><th>补全候选</th><th>几何检查</th></tr>'''+''.join(rows)+'</table><h2>跳过</h2><pre>'+html.escape(json.dumps(result['skipped'], ensure_ascii=False, indent=2))+'</pre></html>', encoding='utf-8') <table><tr><th>部件 / 上游告警</th><th>状态</th><th>原可见区域</th><th>允许新增区域(白色)</th><th>补全候选</th><th>几何检查</th></tr>'''+''.join(rows)+'</table><h2>跳过</h2><pre>'+html.escape(json.dumps(result['skipped'], ensure_ascii=False, indent=2))+'</pre></html>', encoding='utf-8')
@@ -249,16 +258,23 @@ def main():
parser.add_argument('--out', required=True) parser.add_argument('--out', required=True)
parser.add_argument('--part', action='append', default=[]) parser.add_argument('--part', action='append', default=[])
parser.add_argument('--size', type=int, default=1024) parser.add_argument('--size', type=int, default=1024)
parser.add_argument('--radius', type=int, default=32) parser.add_argument('--radius', type=int)
parser.add_argument('--profile', choices=['legacy', 'adaptive'], default='legacy')
parser.add_argument('--reference-mode', choices=['none', 'local', 'full', 'both', 'board'], default='board')
parser.add_argument('--review-file')
parser.add_argument('--candidate') parser.add_argument('--candidate')
parser.add_argument('--merged') parser.add_argument('--merged')
parser.add_argument('--qa') parser.add_argument('--qa')
parser.add_argument('--candidate-size-policy', choices=['strict', 'uniform'], default='strict') parser.add_argument('--candidate-size-policy', choices=['strict', 'uniform'], default='strict')
args = parser.parse_args() args = parser.parse_args()
if args.action == 'prepare': if args.action == 'prepare':
if not 128 <= args.size <= 4096 or not 1 <= args.radius <= 256: if not 128 <= args.size <= 4096 or (args.radius is not None and not 1 <= args.radius <= 256):
parser.error('size must be 128..4096 and radius 1..256') parser.error('size must be 128..4096 and radius 1..256')
prepare(args.out, args.part, args.size, args.radius) if args.profile == 'adaptive':
from completion_planning import prepare_adaptive
prepare_adaptive(args.out, args.part, args.size, args.radius, args.reference_mode, args.review_file)
else:
prepare(args.out, args.part, args.size, args.radius or 32)
elif args.action == 'merge': elif args.action == 'merge':
write_json(args.qa, merge_candidate(args.out, args.candidate, args.merged, args.candidate_size_policy)) write_json(args.qa, merge_candidate(args.out, args.candidate, args.merged, args.candidate_size_policy))
else: else:
+343
View File
@@ -0,0 +1,343 @@
"""Adaptive completion planning, reference canvases and reversible native-pixel merging."""
import hashlib
import json
import math
from pathlib import Path
import cv2
import numpy as np
from PIL import Image, ImageDraw
from normalize_reference import prepare_rgba
def read(file):
return json.loads(Path(file).read_text(encoding='utf-8-sig'))
def write(file, value):
Path(file).write_text(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False), encoding='utf-8')
def binary(file, shape):
a = np.array(Image.open(file))
if a.shape != shape or not np.isin(a, [0, 255]).all():
raise ValueError(f'Expected full-canvas binary PNG: {file}')
return a > 0
def review_inputs(file, ids, shape):
"""Review masks are explicitly authored data; never guess ownership from scores."""
if not file:
return {}, {}
file = Path(file).resolve()
data = read(file)
if data.get('schema_version') != 1 or not isinstance(data.get('parts'), dict):
raise ValueError('Completion review requires schema_version=1 and parts object')
hashes = {'review.json': hashlib.sha256(file.read_bytes()).hexdigest()}
result = {}
for name, entry in data['parts'].items():
if name not in ids or not isinstance(entry, dict) or entry.get('confirmed') is not True:
raise ValueError(f'Review entry requires known ID and confirmed=true: {name}')
if set(entry) - {'confirmed', 'extension_radius', 'repair_mask', 'replace_mask', 'remove_mask'}:
raise ValueError(f'Unknown review fields: {name}')
item = {'confirmed': True}
if 'extension_radius' in entry:
radius = entry['extension_radius']
if type(radius) is not int or not 1 <= radius <= 256:
raise ValueError(f'Invalid reviewed radius: {name}')
item['extension_radius'] = radius
for key in ['repair_mask', 'replace_mask', 'remove_mask']:
if key not in entry:
continue
ref = entry[key]
if not isinstance(ref, str) or Path(ref).is_absolute():
raise ValueError('Review mask must be relative to review file')
mask_file = (file.parent / ref).resolve()
if not mask_file.is_relative_to(file.parent):
raise ValueError('Review mask path escapes review directory')
item[key] = binary(mask_file, shape)
hashes[f'{name}/{key}'] = hashlib.sha256(mask_file.read_bytes()).hexdigest()
result[name] = item
return result, hashes
def bounding(mask):
ys, xs = np.nonzero(mask)
return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1] if len(xs) else None
def enclosed(mask):
padded = np.pad(mask.astype(np.uint8), 1)
fill = padded.copy()
cv2.floodFill(fill, None, (0, 0), 1)
return ~fill[1:-1, 1:-1].astype(bool)
def white_image(array):
im = Image.fromarray(array).convert('RGBA')
return Image.alpha_composite(Image.new('RGBA', im.size, 'white'), im).convert('RGB')
def prepare_adaptive(directory, selected_ids, size=1024, radius=None, reference_mode='board', review_file=None):
# Import shared native coordinate helpers lazily to avoid a module cycle.
from completion_geometry import localize, allowed_regions
out = Path(directory)
plan, seg = read(out / 'semantic-parts.json'), read(out / 'segmentation.json')
rgba = np.array(Image.open(out / 'source.png').convert('RGBA'))
h, w = rgba.shape[:2]
ids = {p['id']: p for p in plan['parts']}
if set(selected_ids) - set(ids):
raise ValueError('Unknown requested part ID')
records = {p['id']: p for p in seg['parts']}
if len(records) != len(seg['parts']) or set(records) - set(ids):
raise ValueError('Invalid segmentation IDs')
masks = {name: binary(out / 'masks' / f'{name}.png', (h, w)) for name in records}
for name, mask in masks.items():
if not mask.any() or np.any(mask & (rgba[:, :, 3] == 0)):
raise ValueError(f'Empty mask or mask outside original alpha: {name}')
reviews, review_hashes = review_inputs(review_file, ids, (h, w))
coverage = np.zeros((h, w), np.uint16)
for mask in masks.values():
coverage += mask
tasks, skipped, assessments = [], [], []
(out / 'preservation').mkdir(exist_ok=True)
for part in plan['parts']:
name = part['id']
if selected_ids and name not in selected_ids:
continue
if name not in masks or part['visibility'] == 'hidden':
skipped.append({'id': name, 'reason': 'NO_VISIBLE_SAM_MASK'})
continue
visible = masks[name]
box = bounding(visible)
review = reviews.get(name, {})
resolved_radius = review.get('extension_radius', radius if radius is not None else
int(np.clip(math.ceil(min(box[2]-box[0], box[3]-box[1]) * .15), 4, 96)))
inferred = dict(part)
inferred['completion'] = {
'occlusion_fill': bool(part['occluded_by']),
'joint_extension_parent': part['rig_role'] == 'bone_part' and part['parent'] is not None,
'joint_extension_child': part['rig_role'] == 'bone_part' and any(
p['parent'] == name and p['rig_role'] == 'bone_part' for p in plan['parts'])}
_, regions, warnings = allowed_regions(inferred, plan, masks, visible, resolved_radius)
# Only enclosed source foreground without another mask owner is an automatic
# repair candidate. Open gaps and ambiguous ownership stay diagnostic.
gap = enclosed(visible) & (rgba[:, :, 3] > 0) & (coverage == 0)
cleanup = np.zeros_like(visible)
for front in part['occluded_by']:
p = ids[front]
if front in masks and p['rig_role'] != 'bone_part' and p['bind_to'] == name:
cleanup |= visible & masks[front]
replace = review.get('replace_mask', np.zeros_like(visible))
removed = review.get('remove_mask', np.zeros_like(visible))
repair = review.get('repair_mask', np.zeros_like(visible))
if np.any((replace | removed) & ~visible) or np.any(repair & visible) or np.any(replace & removed):
raise ValueError(f'{name}: replacement/removal must be disjoint subsets of SAM; repair must be outside SAM')
regions['segmentation_gap'] = gap | repair
regions['attachment_cleanup'] = cleanup & ~removed
regions['reviewed_replacement'] = replace
allowed = np.zeros_like(visible)
for region in regions.values():
allowed |= region
allowed &= ~removed
protected = visible & ~allowed & ~removed
unexpected = []
duplicates = []
for other, mask in masks.items():
if other == name:
continue
overlap = int((mask & visible).sum())
related = other in part['occluded_by'] or name in ids[other]['occluded_by']
if overlap and not related:
unexpected.append({'id': other, 'pixels': overlap})
if overlap / max(1, int((mask | visible).sum())) >= .8:
duplicates.append(other)
assessment = {'id': name, 'original_needs_completion': part['needs_completion'],
'inferred_completion': inferred['completion'], 'radius': resolved_radius,
'region_pixels': {k: int(v.sum()) for k, v in regions.items()},
'protected_pixels': int(protected.sum()), 'removed_pixels': int(removed.sum()),
'unexpected_overlaps': unexpected, 'review_confirmed': bool(review)}
assessments.append(assessment)
if duplicates and not review:
skipped.append({'id': name, 'reason': 'AMBIGUOUS_MASK_REQUIRES_REVIEW', 'overlaps': duplicates})
continue
if not protected.any() or not (protected & (rgba[:, :, 3] >= 240)).any():
skipped.append({'id': name, 'reason': 'NO_PRESERVED_ALIGNMENT_ANCHOR'})
continue
if not allowed.any() and not removed.any():
skipped.append({'id': name, 'reason': 'NO_OBSERVED_EDIT_REGION'})
continue
region_box = bounding(visible | allowed)
bw, bh = region_box[2] - region_box[0], region_box[3] - region_box[1]
native_size = max(8, math.ceil(max(bw, bh) * 1.15))
origin = [region_box[0] - (native_size-bw)//2, region_box[1] - (native_size-bh)//2]
work_size = int(size * .68) if reference_mode == 'board' else size
offset = [int(size * .02), (size-work_size)//2] if reference_mode == 'board' else [0, 0]
scale = work_size / native_size
folder = out / 'tasks' / name
folder.mkdir(parents=True, exist_ok=True)
native_source = localize(rgba, origin, native_size)
native_visible = localize(visible, origin, native_size)
native_keep = localize(protected, origin, native_size)
native_edit = localize(allowed, origin, native_size)
native_remove = localize(removed, origin, native_size)
cut = native_source.copy()
cut[~native_keep] = 0
Image.fromarray(cut).save(folder / 'native-visible.png')
Image.fromarray(native_source).save(folder / 'native-source.png')
for label, mask in [('keep', native_keep), ('allowed', native_edit), ('remove', native_remove), ('sam', native_visible)]:
Image.fromarray(mask.astype(np.uint8)*255).save(folder / f'native-{label}.png')
def project(array, is_mask=False):
small = np.array(Image.fromarray(array.astype(np.uint8)*255 if is_mask else array).resize(
(work_size, work_size), Image.Resampling.NEAREST if is_mask else Image.Resampling.LANCZOS))
canvas = np.zeros((size, size) + small.shape[2:], np.uint8)
canvas[offset[1]:offset[1]+work_size, offset[0]:offset[0]+work_size] = small
return canvas
model_visible = project(cut)
Image.fromarray(model_visible).save(folder / 'visible.png')
model_allowed = project(native_edit, True) > 0
model_remove = project(native_remove, True) > 0
Image.fromarray(model_allowed.astype(np.uint8)*255).save(folder / 'allowed-mask.png')
mask_rgba = np.full((size, size, 4), 255, np.uint8)
mask_rgba[model_allowed | model_remove, 3] = 0
Image.fromarray(mask_rgba).save(folder / 'edit-mask.png')
context = white_image(project(native_source))
context.save(folder / 'context.png')
full_reference = white_image(rgba)
full_reference.thumbnail((size, size), Image.Resampling.LANCZOS)
full_reference.save(folder / 'reference.png')
model_input = white_image(model_visible)
panel_boxes = []
if reference_mode == 'board':
draw = ImageDraw.Draw(model_input)
x = int(size * .74)
draw.line([(x-4, 0), (x-4, size)], fill='#777777', width=1)
for label, image, top in [('LOCAL REFERENCE', white_image(native_source), 0),
('FULL REFERENCE', full_reference.copy(), size//2)]:
draw.text((x, top+2), label, fill='black')
image.thumbnail((size-x-4, size//2-24), Image.Resampling.LANCZOS)
model_input.paste(image, (x, top+22))
panel_boxes.append([x, top+22, x+image.width, top+22+image.height])
model_input.save(folder / 'input.png')
bounds, native_counts = {}, {}
for kind, region in regions.items():
native = localize(region & allowed, origin, native_size)
model = project(native, True)
Image.fromarray(native.astype(np.uint8)*255).save(folder / f'native-{kind}.png')
Image.fromarray(model).save(folder / f'{kind}.png')
bounds[kind] = bounding(model > 0)
native_counts[kind] = int(native.sum())
preservation = {}
for key, mask in [('preserve_mask', protected), ('editable_mask', allowed), ('removed_mask', removed)]:
relative = f'preservation/{name}-{key}.png'
Image.fromarray(mask.astype(np.uint8)*255).save(out / relative)
preservation[key] = relative
if unexpected:
warnings.append('UNEXPECTED_OVERLAP_REVIEW_REQUIRED')
if cleanup.any():
warnings.append('ATTACHMENT_BASE_CLEANUP_NEEDS_REVIEW')
task = {'id': name, 'semantic': part['semantic'], 'profile': 'adaptive',
'completion': inferred['completion'], 'original_completion': part['completion'],
'origin_xy': origin, 'native_size': native_size, 'canvas_size': [size, size], 'source_size': [w, h],
'extension_radius': resolved_radius, 'allowed_pixels': int(native_edit.sum()),
'removed_pixels': int(native_remove.sum()), 'regions': native_counts,
'region_bounds_xyxy': bounds, 'occluded_by': part['occluded_by'],
'warnings': records[name].get('warnings', []) + warnings, 'preservation': preservation,
'reference_mode': reference_mode, 'reference_panel_boxes': panel_boxes,
'transform': {'scale': scale, 'working_xyxy': [*offset, offset[0]+work_size, offset[1]+work_size],
'source_to_canvas': [scale, 0, offset[0]-scale*origin[0]+(scale-1)/2,
0, scale, offset[1]-scale*origin[1]+(scale-1)/2],
'canvas_to_source': [1/scale, 0, origin[0]-offset[0]/scale+(1/scale-1)/2,
0, 1/scale, origin[1]-offset[1]/scale+(1/scale-1)/2]},
'semantic_verified': False, 'status': 'prepared'}
write(folder / 'task.json', task)
tasks.append(task)
result = {'schema_version': 2, 'profile': 'adaptive', 'tasks': tasks, 'skipped': skipped,
'assessments': assessments, 'review_hashes': review_hashes, 'reference_mode': reference_mode,
'canvas_size': size, 'quality_status': 'needs_review'}
write(out / 'completion-plan.json', result)
return result
def merge_adaptive(folder, candidate_file, output_file, size_policy):
folder = Path(folder)
task = read(folder / 'task.json')
candidate = Image.open(candidate_file).convert('RGBA')
qa = {'accepted': False, 'semantic_verified': False, 'reasons': [], 'profile': 'adaptive',
'expected_size': task['canvas_size'], 'returned_size': list(candidate.size),
'size_policy': size_policy, 'canvas_resampled': False, 'transform': task['transform']}
tw, th = task['canvas_size']
if candidate.size != (tw, th):
if size_policy == 'strict' or candidate.width * th != candidate.height * tw:
qa['reasons'].append('CANVAS_SIZE_MISMATCH' if size_policy == 'strict' else 'CANVAS_ASPECT_RATIO_MISMATCH')
return qa
qa.update(canvas_resampled=True, canvas_scale=tw/candidate.width, resampling='LANCZOS')
candidate = candidate.resize((tw, th), Image.Resampling.LANCZOS)
candidate.save(Path(output_file).with_name(Path(output_file).stem + '-normalized.png'))
# Record raw behavior before masking/restoration; an accepted HTTP response is
# not evidence that the relay/model respected the supplied edit mask.
expected_input = np.array(Image.open(folder / 'input.png').convert('RGB')).astype(float)
raw = np.array(white_image(np.array(candidate))).astype(float)
editable = np.array(Image.open(folder / 'edit-mask.png'))[:, :, 3] == 0
outside_delta = np.abs(raw - expected_input).mean(axis=2)
qa['raw_outside_edit_mae'] = float(outside_delta[~editable].mean()) if (~editable).any() else 0
qa['raw_outside_edit_changed_pixels'] = int(((outside_delta > 30) & ~editable).sum())
rect = task['transform']['working_xyxy']
try:
cropped, preprocessing = prepare_rgba(candidate.crop(rect))
except ValueError as error:
qa.update(reasons=['CANNOT_EXTRACT_FOREGROUND'], detail=str(error))
return qa
side = task['native_size']
generated = np.array(cropped.resize((side, side), Image.Resampling.LANCZOS))
original = np.array(Image.open(folder / 'native-visible.png').convert('RGBA'))
source = np.array(Image.open(folder / 'native-source.png').convert('RGBA'))
keep = binary(folder / 'native-keep.png', (side, side))
allowed = binary(folder / 'native-allowed.png', (side, side))
removed = binary(folder / 'native-remove.png', (side, side))
# Judge alignment before the inverse resampling. Alpha at native boundary
# pixels changes during interpolation even for an exactly aligned candidate.
model_original = np.array(Image.open(folder / 'visible.png').convert('RGBA').crop(rect))
model_generated = np.array(cropped)
opaque = model_original[:, :, 3] >= 240
if not opaque.any():
qa['reasons'].append('NO_OPAQUE_ALIGNMENT_ANCHOR')
return qa
rgb = np.array(white_image(model_generated)).astype(float)
mae = float(np.abs(rgb[opaque] - model_original[:, :, :3][opaque]).mean())
retained = float((model_generated[:, :, 3][opaque] >= 240).mean())
proposed = (generated[:, :, 3] > 0) & ~keep
additions = proposed & allowed
_, labels = cv2.connectedComponents((keep | additions).astype(np.uint8), connectivity=8)
touching = np.unique(labels[keep])
connected = additions & np.isin(labels, touching[touching != 0])
counts = {kind: int((connected & binary(folder / f'native-{kind}.png', (side, side))).sum())
for kind in task['regions']}
qa.update(alignment_mae=mae, visible_retention=retained, alignment_space='request_work_panel', new_pixels=int(connected.sum()),
removed_pixels=int(removed.sum()), region_additions=counts,
unfulfilled_regions=[k for k, n in task['regions'].items() if n and not counts[k]],
discarded_outside_allowed=int((proposed & ~allowed).sum()), preprocessing=preprocessing)
if mae > 30 or retained < .95:
qa['reasons'].append('VISIBLE_ALIGNMENT_MISMATCH')
if not connected.any() and not removed.any():
qa['reasons'].append('NO_CONNECTED_NEW_PIXELS')
for kind in ['attachment_cleanup', 'reviewed_replacement']:
if task['regions'][kind] and counts[kind] / task['regions'][kind] < .95:
qa['reasons'].append(f'INCOMPLETE_{kind.upper()}')
if qa['reasons']:
return qa
merged = np.zeros_like(original)
merged[connected] = generated[connected]
# Visible segmentation gaps already have ground-truth texture in source.
# Restore it where the proposed fill connects, rather than trusting a repaint.
repair = binary(folder / 'native-segmentation_gap.png', (side, side)) & connected & (source[:, :, 3] > 0)
merged[repair] = source[repair]
merged[keep] = original[keep]
merged[removed] = 0
qa.update(accepted=True, quality_status='needs_review', visible_pixels_exact=True,
preservation_scope='preserve_mask', repaired_source_pixels=int(repair.sum()))
Image.fromarray(merged).save(output_file)
return qa
+79 -27
View File
@@ -2,9 +2,11 @@
## 1. 阶段简介 ## 1. 阶段简介
读取 SAM 可见 mask,为需要遮挡补全或关节延伸的部件准备局部画布,调用中转站图片编辑模型,再将允许区域内的新增像素与原可见像素合并。Python 几何处理固定使用 Conda `ai-spine`;图片生成在远端完成,本阶段不加载本地 SAM。 读取 SAM 可见 mask,规划分割缺口修复、遮挡补全、关节延伸和附件底图清理,向图片模型提供部件与原图参考,再把结果映射回原图坐标。Python 几何处理固定使用 Conda `ai-spine`;图片生成在远端完成,本阶段不加载本地 SAM。
当前已实现准备、真实图片请求、离线候选导入、几何检查与透明补全部件输出。尚未输出使用补全部件组装的完整角色,尚未接入新管线 Spine 导出。几何通过仍为 `needs_review`,不能等同于部件可用。 默认 `--completion-profile adaptive` 使用实际 mask 重建需求、自适应半径和可逆缩放;`legacy` 保留旧画布与筛选规则,供旧候选复检。原 SAM 输入始终留档,只有保留区域锁定原像素;明确规划的编辑区域可以替换,人工确认的移除区域可删除。
当前已实现准备、真实图片请求、离线候选导入、几何检查与透明补全部件输出。[第四阶段](4-spine-stage.md) 独立整合 SAM 与补全部件,输出完整角色组装、Spine 骨架和预览。几何通过仍为 `needs_review`,不能等同于部件可用。
## 2. 输入、输出与执行链路 ## 2. 输入、输出与执行链路
@@ -14,18 +16,19 @@
tools/model-check.mjs --stage completion --input-dir SAM目录 tools/model-check.mjs --stage completion --input-dir SAM目录
→ stages/completion-stage.mjs 校验计划、mask 文件、参数及 Conda → stages/completion-stage.mjs 校验计划、mask 文件、参数及 Conda
→ completion_geometry.py prepare → completion_geometry.py prepare
筛选需求 → 计算允许区域 → 建立局部画布与编辑 mask adaptive:completion_planning.py 重建需求 → 区分保留/编辑/移除 → 缩放工作区并加入参考
legacy:沿用 needs_completion → 固定半径 → 原像素平移
→ prepare-only:保存任务和报告后结束 → prepare-only:保存任务和报告后结束
→ 否则:串行请求 images/edits,或读取离线候选 → 否则:串行请求 images/edits,或读取离线候选
→ completion_geometry.py merge → completion_geometry.py merge
检查画布/前景/对齐 → 限制新增区域与连通性 → 恢复原可见像素 检查画布/前景/对齐 → 逆变换回原像素尺寸 → 限制编辑和连通性 → 恢复保留像素
→ 每部件选第一个通过几何检查的候选 → 每部件选第一个通过几何检查的候选
→ finalize 输出透明部件、原图坐标和报告 → finalize 输出透明部件、原图坐标和报告
``` ```
任务必须同时满足:计划 `needs_completion=true`、非 hidden、有可见 SAM mask、允许新增区域非空。指定 `--part` 只缩小范围,不强制绕过筛选。分别记录 `NO_COMPLETION_NEEDED`、`NO_VISIBLE_SAM_MASK`、`NO_ALLOWED_EXTENSION`。 adaptive 检查所有选中部件的实际 mask,不再以第一阶段 needs_completion 为唯一门槛。必须有可见 mask、保留对齐锚点,以及非空编辑或移除区域。隐藏/缺 mask、无观察到的编辑区域、无保留锚点分别跳过;无明确遮挡关系的近重复 mask(IoU≥0.8)要求人工复核。一般有歧义的重叠只记录诊断,不自动猜测归属。指定 `--part` 只缩小处理范围。
以 `output/20260918-183532/sam` 为例:32 个计划项中 10 个 hidden 没有 mask;22 个有 mask,其中 12 个进入补全。帽子、两袖、两眼、两眉、鼻、嘴、胡子这 10 个因第一阶段标记不需补全而跳过,没有重新由模型判断实际 mask 完整性。 legacy 才要求 `needs_completion=true` 且允许新增非空。旧实验 `output/20260918-183532/sam` 的 22 个可见部件中 12 个进入补全,是旧筛选结果,不能套用于 adaptive。
## 3. 阶段执行命令示例及参数说明 ## 3. 阶段执行命令示例及参数说明
@@ -68,6 +71,7 @@ node .\tools\model-check.mjs --stage completion `
--input-dir $samInput ` --input-dir $samInput `
--candidates-dir .\output\20260918-183532\complete2_gpt ` --candidates-dir .\output\20260918-183532\complete2_gpt `
--out-dir ".\output\$runId\complete-recheck" ` --out-dir ".\output\$runId\complete-recheck" `
--completion-profile legacy `
--candidate-size-policy uniform --candidate-size-policy uniform
``` ```
@@ -83,6 +87,18 @@ node .\tools\model-check.mjs --stage completion `
探针只保存候选,不执行部件准备与几何验收,不能与 `--input-dir` 混用。 探针只保存候选,不执行部件准备与几何验收,不能与 `--input-dir` 混用。
新的单部件多图/mask 能力探针会产生一次真实请求,并执行新几何链路;不会修改默认 capabilities。仅在决定进行付费小样时运行,替换 part 为实际存在的可执行任务:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir $samInput `
--out-dir ".\output\$runId\capability-probe" `
--part head --probe-edit-capabilities
```
该探针发送主图、局部参考、全角色参考和透明可编辑 mask,强制 n=1、禁用 HTTP 重试。查看 request.json、原候选、QA 和 capability-evidence.json;HTTP 接受不等于参考图被使用或 mask 语义正确,因此不会自动启用能力。prepare-only 可配 `--completion-reference both --completion-mask required` 检查计划发送内容而不联网。
| 参数 | 说明 | | 参数 | 说明 |
| --- | --- | | --- | --- |
| `--input-dir DIR` | 完整 SAM 输出目录,部件补全必需 | | `--input-dir DIR` | 完整 SAM 输出目录,部件补全必需 |
@@ -90,8 +106,13 @@ node .\tools\model-check.mjs --stage completion `
| `--part ID` | 可重复,ID 必须存在且不重复;省略时处理所有符合筛选条件的部件 | | `--part ID` | 可重复,ID 必须存在且不重复;省略时处理所有符合筛选条件的部件 |
| `--prepare-only` | 只生成画布、区域、提示词和报告,不请求模型 | | `--prepare-only` | 只生成画布、区域、提示词和报告,不请求模型 |
| `--candidates-dir DIR` | 离线导入候选,与 prepare-only 互斥 | | `--candidates-dir DIR` | 离线导入候选,与 prepare-only 互斥 |
| `--completion-size N` | 方形画布边长,默认 1024,整数 128~4096;原像素不缩放 | | `--completion-profile adaptive/legacy` | 默认 adaptive;旧候选需显式 legacy |
| `--extension-radius N` | 轮廓延展距离与关节圆盘半径,默认 32,整数 1~256 | | `--completion-reference auto/board/local/full/both/none` | auto:支持多图时 local+full,否则单图参考板;legacy auto 为旧单图/局部参考行为 |
| `--completion-mask auto/required/off` | auto 遵循配置;required 要求已配置支持,或用单部件能力探针试验;off 不发送但仍本地限制编辑 |
| `--completion-review FILE` | 人工确认的补全区域文件,格式见第 4 节;只用于 adaptive |
| `--probe-edit-capabilities` | adaptive 在线单部件试验多图+mask,不持久修改配置,不自动认定能力支持 |
| `--completion-size N` | 请求方形画布边长,默认 1024,整数 128~4096;adaptive 记录显式缩放和逆变换 |
| `--extension-radius N` | 原图像素半径覆盖,整数 1~256;adaptive 默认 min(mask宽,高)×0.15 向上取整并限于 4~96;legacy 默认 32 |
| `--candidate-size-policy NAME` | strict 默认拒绝尺寸不符;uniform 允许同宽高比整画布缩放后复检 | | `--candidate-size-policy NAME` | strict 默认拒绝尺寸不符;uniform 允许同宽高比整画布缩放后复检 |
| `--completion-model ID` | 图片模型 ID,原样发送 | | `--completion-model ID` | 图片模型 ID,原样发送 |
| `--config FILE` / `--env-file FILE` | 默认 astra-qwen.json / 项目 .env;shell 同名变量优先 | | `--config FILE` / `--env-file FILE` | 默认 astra-qwen.json / 项目 .env;shell 同名变量优先 |
@@ -122,19 +143,42 @@ node .\tools\model-check.mjs --stage completion `
不同原图、mask 或参数的候选不能混用。尺寸之外,部件位置、朝向、比例也必须保持一致。 不同原图、mask 或参数的候选不能混用。尺寸之外,部件位置、朝向、比例也必须保持一致。
adaptive 的旧运行候选还要匹配 profile、参考布局、复核文件/掩码哈希和规划版本;board 与独立多图的画布不同,不可混用。平铺候选仍需人工保证它对应当前请求画布。
人工复核文件示例(JSON 中的 ID 取自当前语义计划,字段均可省略但 confirmed 必须为 true):
```json
{
"schema_version": 1,
"parts": {
"part_id": {
"confirmed": true,
"extension_radius": 24,
"repair_mask": "review-masks/repair.png",
"replace_mask": "review-masks/replace.png",
"remove_mask": "review-masks/remove.png"
}
}
}
```
mask 是与 source 同尺寸的单通道 0/255 PNG,白色表示区域;路径相对于复核 JSON 且不得越出其目录。repair 必须在当前 SAM mask 外;replace/remove 必须位于 SAM 内且互不重叠。replace 用于误分重绘或额外底图清理,remove 用于明确不属于该部件的像素。半径优先级为部件复核值 > CLI 全局值 > 自适应值。文件及 masks 会复制进输出,哈希纳入复用身份。
## 5. 执行链路中重要部分代码文件及原理说明(包括重点说明) ## 5. 执行链路中重要部分代码文件及原理说明(包括重点说明)
**允许区域。** `completion_geometry.py::allowed_regions()` 先计算可见 mask 外、距轮廓不超过 extension-radius 的区域,再分别与以下区域取交集:遮挡补全用 occluded_by 的 mask 并集;父关节延展用本部件 pivot 周围圆盘;子关节延展用直接子骨骼部件 pivot 圆盘。始终排除原可见 mask,新增范围限制在原图画布内。它是保守几何范围,不是完整隐藏形状推断。 **按实际 mask 重建需求。** 遮挡延伸使用轮廓外半径带与 occluded_by 的实际 mask 交集;父/子关节需求来自骨骼层级与 pivot。自动分割修复仅选择 mask 内部闭合孔洞、原图有前景且无其他 mask 归属的像素;开放缺口和重叠归属不自动猜测,可用复核文件指定。自动底图清理仅针对 occluded_by 中直接 bind_to 当前骨骼的附件/五官与当前 mask 的交集。这些仍是几何候选,语义不自动获证。
**局部坐标。** `prepare()` 将可见区和允许区居中放入方形画布,至少留 16 像素边距,只平移和补边。`origin_xy` 是局部左上角在原图的位置,可为负数;原图位置=局部位置+origin。放不下时失败,不能偷偷缩放原像素。 **保留与编辑。** editable 为遮挡/关节/修复/清理/人工替换区域的并集,扣除 removed;preserve = SAM − editable − removed。保留原 mask 和源图,并另存三种全画布区域供第四阶段核对。只有原始 SAM 或单一图片目录不足以描述这个契约。
**重点:当前实际发送什么。** `stages/completion-stage.mjs` 默认只发送白底 `input.png`。三个现有配置均为 `multiple_images=false`、`mask=false`,因此本地生成 context.png 和 edit-mask.png 不意味着发给模型。只有开启并验证对应能力后,分支才发送第二张同坐标局部原图参考和编辑遮罩;当前没有独立全角色参考图输入。文字边界说明及返回后的区域裁剪不能替代模型侧遮罩约束。 **可逆画布。** native 方形裁剪边长为可见+编辑区域最长边的 1.15 倍向上取整,记录 origin。工作区缩放到请求画布(board 工作区边长为请求边长约 68%,其余为参考板;独立参考时工作区占整幅)。RGBA 用 LANCZOS,区域 mask 用最近邻。task.json 保存 native_size、scale、working_xyxy 和含像素中心约定的正反仿射变换。返回后只取工作区,逆缩放到 native,再裁剪回原图坐标;保留像素从原图恢复,所以缩放不改变最终保留像素。legacy 仍仅平移补边。
**实际发送内容。** 当前三个提供方默认仍关闭 multiple_images/mask。adaptive auto 因此发送一张 input.png:左侧是待编辑工作区,右侧为局部及完整角色参考,参考区域不参与最终资产。支持多图时 auto 顺序为 input.png → context.png → reference.png;也可选 local/full/both。request-plan.json 保存计划;只有实际请求才写 request.json,包含有序文件角色、SHA-256、尺寸、image 字段、mask 哈希、透明可编辑语义与是否发送。mask 不支持时明确标 local_only,不能宣称模型侧已受 mask 约束。
**图片解码与尺寸。** `providers/image-edits.mjs` 使用 multipart images/edits,支持非空 b64_json 或返回 URL 下载;空 b64_json 会回退有效 URL。当前部件请求显式发送 size=画布宽x高,并写入 request.json;配置 params.size 冲突会报错。strict 模式拒绝 CANVAS_SIZE_MISMATCH;uniform 仅将同宽高比整画布 LANCZOS 缩放到目标尺寸,保留原候选,不平移、不裁剪、不单独缩放物体,也不改变原图/mask。 **图片解码与尺寸。** `providers/image-edits.mjs` 使用 multipart images/edits,支持非空 b64_json 或返回 URL 下载;空 b64_json 会回退有效 URL。当前部件请求显式发送 size=画布宽x高,并写入 request.json;配置 params.size 冲突会报错。strict 模式拒绝 CANVAS_SIZE_MISMATCH;uniform 仅将同宽高比整画布 LANCZOS 缩放到目标尺寸,保留原候选,不平移、不裁剪、不单独缩放物体,也不改变原图/mask。
**几何验收与合并。** `merge_candidate()` 去除边界连通白底,在原部件 Alpha≥240 的像素处检查 RGB 平均绝对误差≤30、可见保留率≥95%;无不透明锚点拒绝。只取允许区域内且与原部件连通的新增像素,没有有效新增即拒绝。通过后逐像素恢复原可见 RGBA,模型只提供新增部分。 **几何验收与合并。** adaptive 在请求工作区的保留不透明像素处检查 RGB MAE≤30、保留率≥95%,先验对齐再逆采样,避免把逆采样边界 Alpha 变化误判成位移。返回前景只保留 editable 内且连接 preserve 的像素;清理/人工替换区域至少 95% 有效填充,否则拒绝。没有有效修改且没有人工移除则拒绝。最后恢复 preserve 的原 RGBA,已连接的可见缺口使用原图纹理,removed 清零。legacy 沿用原可见像素全保护规则。
**重点:验收仍有缺口。** `unfulfilled_regions` 记录未补到的子区域,但当前不是硬性拒绝条件;某一区域新增少量像素也不代表关节补够。语义、接缝、材质、运动露底没有自动验收。错误 SAM 像素会被原样恢复;若头部 mask 已含眼睛/胡子,当前规则也不能清除这些内部附件生成干净脸底。 **能力证据与质量边界。** adaptive QA 记录合并前的 raw_outside_edit_mae / changed_pixels,避免本地恢复掩盖模型越界修改。capability-evidence.json 记录探针的输入顺序、尺寸、候选和 QA,但 reference_use_verified、mask_semantics_verified 不自动置 true,仍需视觉验证。区域填满不能证明生成的是正确底图;未明确划入 editable/remove 的误分仍受保护。接缝、材质、关节补足和运动露底尚无可靠自动验收。
## 6. 各个代码文件职责索引 ## 6. 各个代码文件职责索引
@@ -143,6 +187,7 @@ node .\tools\model-check.mjs --stage completion `
| [tools/model-check.mjs](../tools/model-check.mjs) | 部件补全与独立探针分发、参数互斥检查 | | [tools/model-check.mjs](../tools/model-check.mjs) | 部件补全与独立探针分发、参数互斥检查 |
| [stages/completion-stage.mjs](../stages/completion-stage.mjs) | 输入指纹、准备、串行请求/离线读取、候选选择、状态留档 | | [stages/completion-stage.mjs](../stages/completion-stage.mjs) | 输入指纹、准备、串行请求/离线读取、候选选择、状态留档 |
| [completion_geometry.py](../completion_geometry.py) | allowed_regions、prepare、merge_candidate、finalize 与报告 | | [completion_geometry.py](../completion_geometry.py) | allowed_regions、prepare、merge_candidate、finalize 与报告 |
| [completion_planning.py](../completion_planning.py) | adaptive 需求重建、复核输入、参考板、正反变换、编辑分区与合并 |
| [normalize_reference.py](../normalize_reference.py) | 候选 Alpha 保留、边界连通白底提取 | | [normalize_reference.py](../normalize_reference.py) | 候选 Alpha 保留、边界连通白底提取 |
| [providers/image-edits.mjs](../providers/image-edits.mjs) | multipart 编辑请求、能力检查、Base64/URL 图片解码 | | [providers/image-edits.mjs](../providers/image-edits.mjs) | multipart 编辑请求、能力检查、Base64/URL 图片解码 |
| [providers/http.mjs](../providers/http.mjs) | HTTP、超时、重试、诊断和脱敏 | | [providers/http.mjs](../providers/http.mjs) | HTTP、超时、重试、诊断和脱敏 |
@@ -153,6 +198,7 @@ node .\tools\model-check.mjs --stage completion `
| [prompts/complete-v1.txt](../prompts/complete-v1.txt) | 保持姿势、尺寸、可见纹理与隐藏区域补全约束 | | [prompts/complete-v1.txt](../prompts/complete-v1.txt) | 保持姿势、尺寸、可见纹理与隐藏区域补全约束 |
| [Tiantoken](../configs/tiantoken.json)、[DeepSeek/Qwen](../configs/deepseek-qwen.json)、[Astra/Qwen](../configs/astra-qwen.json) 配置 | 图片模型、认证变量、请求参数与能力开关 | | [Tiantoken](../configs/tiantoken.json)、[DeepSeek/Qwen](../configs/deepseek-qwen.json)、[Astra/Qwen](../configs/astra-qwen.json) 配置 | 图片模型、认证变量、请求参数与能力开关 |
| [tests/completion-stage.test.mjs](../tests/completion-stage.test.mjs) / [tests/test_completion.py](../tests/test_completion.py) | mock 编辑、离线复用、尺寸/对齐拒绝、像素保真与区域回归 | | [tests/completion-stage.test.mjs](../tests/completion-stage.test.mjs) / [tests/test_completion.py](../tests/test_completion.py) | mock 编辑、离线复用、尺寸/对齐拒绝、像素保真与区域回归 |
| [tests/test_completion_adaptive.py](../tests/test_completion_adaptive.py) | 放大/缩小回贴、漏分重建、附件清理、区域复核、拒绝位移和第四阶段契约 |
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 提供方与探针相关回归 | | [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 提供方与探针相关回归 |
## 7. 产物详细说明 ## 7. 产物详细说明
@@ -160,34 +206,40 @@ node .\tools\model-check.mjs --stage completion `
| 路径 | 内容与用途 | | 路径 | 内容与用途 |
| --- | --- | | --- | --- |
| `source.png`、`semantic-parts.json`、`segmentation.json`、`masks/` | 输入快照,用于复现与离线候选身份核对 | | `source.png`、`semantic-parts.json`、`segmentation.json`、`masks/` | 输入快照,用于复现与离线候选身份核对 |
| `completion-plan.json` | 准备任务、允许区域、参数和跳过原因 | | `completion-plan.json` | 准备任务、区域、跳过原因;adaptive 含 assessments、重建需求、半径及歧义重叠 |
| `tasks/<id>/input.png` | 白底局部可见部件,默认唯一模型输入图 | | `tasks/<id>/input.png` | 实际请求主图;board 含目标、局部参考与全角色参考 |
| `tasks/<id>/visible.png` | 原图可见 RGBA,合并时恢复像素 | | `tasks/<id>/visible.png` | 请求坐标的保留 RGBA;legacy 为原可见 RGBA |
| `tasks/<id>/context.png` | 同坐标局部原图参考,当前默认不发送 | | `tasks/<id>/context.png` / `reference.png` | 局部 / 全角色原图参考,由 reference mode 决定是否独立发送 |
| `tasks/<id>/edit-mask.png` | RGBA 编辑遮罩,透明处可编辑,当前默认不发送 | | `tasks/<id>/edit-mask.png` | RGBA 编辑遮罩,透明处为 editable 或 removed;是否发送以 request.json 为准 |
| `tasks/<id>/allowed-mask.png` | 白色表示允许新增区域 | | `tasks/<id>/allowed-mask.png` | 白色表示允许新增区域 |
| `tasks/<id>/<补全字段>.png` | 启用的补全子区域 mask | | `tasks/<id>/<补全字段>.png` | 启用的补全子区域 mask |
| `tasks/<id>/task.json` / `prompt.txt` | origin、区域、需求和告警 / 实际提示词 | | `tasks/<id>/task.json` / `prompt.txt` | origin、区域、变换、参考布局、重建需求和告警 / 实际提示词 |
| `tasks/<id>/request.json` | 在线请求模型、图片哈希、是否发送 mask;修复后含 size | | `tasks/<id>/native-*.png` | 原像素尺寸的源图、保留像素与各区域,用于逆映射和精确恢复 |
| `preservation/<id>-*.png` | 原图全画布 preserve/editable/removed 契约,第四阶段核对用 |
| `review-source.json`、`review.json`、`review/` | 可选原始复核文件、路径规范化快照及对应 masks |
| `tasks/<id>/request-plan.json` / `request.json` | 计划 / 实际请求;记录有序图片角色、哈希、尺寸、mask 语义和发送状态 |
| `tasks/<id>/capability-evidence.json` | 仅能力探针:请求接受情况及原始越界指标,能力需人工确认 |
| `tasks/<id>/response.raw.json` / `request-metadata.json` | 在线响应留档 / 请求 ID、耗时等元数据 | | `tasks/<id>/response.raw.json` / `request-metadata.json` | 在线响应留档 / 请求 ID、耗时等元数据 |
| `tasks/<id>/response.error.json` | 可获取时保存的脱敏请求失败诊断 | | `tasks/<id>/response.error.json` | 可获取时保存的脱敏请求失败诊断 |
| `tasks/<id>/candidate-N.*` | 模型或离线导入的原始候选,拒绝也保留 | | `tasks/<id>/candidate-N.*` | 模型或离线导入的原始候选,拒绝也保留 |
| `tasks/<id>/merged-N-normalized.png` | uniform 实际重采样时保存的候选;不代表通过 | | `tasks/<id>/merged-N-normalized.png` | uniform 实际重采样时保存的候选;不代表通过 |
| `tasks/<id>/qa-N.json` | 尺寸、对齐、新增/丢弃像素、各区域完成情况、拒绝原因 | | `tasks/<id>/qa-N.json` | 尺寸、对齐、新增/丢弃像素、各区域完成情况、拒绝原因 |
| `tasks/<id>/merged-N.png` | 通过几何检查且恢复原可见像素后的局部 RGBA | | `tasks/<id>/merged-N.png` | 原像素尺寸的局部 RGBA;adaptive 已逆采样并恢复 preserve |
| `parts/<id>.png` | 每部件选中结果的紧裁剪透明 PNG | | `parts/<id>.png` | 每部件选中结果的紧裁剪透明 PNG |
| `completion-result.json` | 每部件状态、候选 QA、原图 crop_xyxy 和路径 | | `completion-result.json` | 每部件状态、候选 QA、原图 crop_xyxy、preservation 路径和变换 |
| `report.html` / `manifest.json` | 人工检查页 / 模式、网络标记、状态、输入/代码/产物哈希 | | `report.html` / `manifest.json` | 人工检查页 / 模式、网络标记、状态、输入/代码/产物哈希 |
parts/ 只含本次选中且通过检查的补全部件,不是全部 SAM 部件集合。无需补全、失败和未选中的部件不会自动复制进来,也没有完整角色组装图。 parts/ 只含本次选中且通过检查的补全部件,不是全部 SAM 部件集合。无需补全、失败和未选中的部件不会自动复制进来;完整角色组装由第四阶段读取 SAM 与本目录后生成。
当前实验:`output/20260918-183532/complete2_gpt` 已真实请求 Tiantoken gpt-image-2 共 12 个部件。旧请求未传 size,返回 1248/1254 方形图,预期为 1024,全部严格尺寸拒绝。`complete2_gpt-size-recheck-v2` 离线复用原候选后,8 个几何通过;head、pants_thigh_l、pants_thigh_r、shoe_l 共 4 个因对齐或无有效新增拒绝。通过项中仍有衣料误分、错误手部和未补全子区域。显式 size 的请求构造已验证,但尚无修复后中转遵守 size 的新实测结论。 当前实验:`output/20260918-183532/complete2_gpt` 已真实请求 Tiantoken gpt-image-2 共 12 个部件。旧请求未传 size,返回 1248/1254 方形图,预期为 1024,全部严格尺寸拒绝。`complete2_gpt-size-recheck-v2` 离线复用原候选后,8 个几何通过;head、pants_thigh_l、pants_thigh_r、shoe_l 共 4 个因对齐或无有效新增拒绝。通过项中仍有衣料误分、错误手部和未补全子区域。显式 size 的请求构造已验证,但尚无修复后中转遵守 size 的新实测结论。
2026-09-22 adaptive 验证:Conda ai-spine 下 npm test 共 29 项通过(包括 8 项新 Python adaptive 用例),覆盖 board/独立参考顺序、mask mock、放大与缩小回贴、位移拒绝、实际孔洞、附件清理、人工编辑契约及第四阶段接续。`output/completion-adaptive-prepare-20260922` 使用既有 SAM 数据离线生成 16 个任务、11 个跳过,上衣规划 3708 个孔洞像素、头部规划 10205 个底图清理像素。它们是准备阶段的候选区域,不代表补全质量;本轮未请求付费模型,也未验证中转真实能力。
## 8. 当前阶段可改进点说明 ## 8. 当前阶段可改进点说明
- **补全后输出使用补全部件的完整角色。** 按原图坐标和遮挡顺序组装,复用无需补全的原拆件,明确标记失败项及回退来源;输出补全前后对比、接缝/重叠诊断。不能只展示少数 parts 文件作为完整结果。 - **完整角色组装已接入第四阶段。** 按原坐标与遮挡约束组装,复用原拆件并记录失败回退,输出补全前后对比、覆盖与重叠诊断及单骨骼旋转检查动画。可靠的接缝与运动质量自动验收仍待实现。
- **把原图或部件相关原图区域交给补全模型参考。** 接入并实测全角色/局部上下文参考和编辑 mask,记录实际发送内容。不能仅把 capabilities 改为 true 就宣称中转支持;需验证图片顺序、mask 语义、返回尺寸和画面保持能力。 - **参考输入已实现,真实能力待验证。** 支持 board 与独立 local/full/both、mask 发送策略、有序请求记录及单部件能力探针。离线 mock 已覆盖图片顺序、mask、返回尺寸和越界记录;本轮未请求付费模型,未宣称中转支持或生成质量提升。
- **根据 SAM 实际结果重建需求。** 区分分割缺口修复、遮挡补全、关节延伸、附件底图清理;对应建立保留区域与可编辑区域,避免误分像素被永久保护。 - **实际 mask 需求重建已实现。** 分别规划缺口、遮挡、关节和附件清理,保存 preserve/editable/removed,并支持人工复核区域。保守孔洞检测不会自动解决所有开放缺口、误分与语义歧义;这些仍需复核。
- **改进画布与补全区域。** 固定 32 像素不适配所有部件;小部件在 1024 大白底中占比低。可验证按部件设计区域、显式局部放大并记录逆变换,保证回贴坐标可检查。 - **自适应画布与区域已实现。** 半径随部件尺寸变化并可覆盖;小部件放大、大部件缩小,请求与原图间变换留档,最终恢复原像素尺寸和保护区。离线测试覆盖放大/缩小、Alpha 保真、位移拒绝及第四阶段接续;真实模型效果待小样验证。
- **加强验收。** 对必需区域定义有效覆盖要求,增加部件语义、接缝、材质、回贴和关节旋转露底检查;不能放宽阈值来掩盖重绘、位移或错误输入。 - **加强验收。** 对必需区域定义有效覆盖要求,增加部件语义、接缝、材质、回贴和关节旋转露底检查;不能放宽阈值来掩盖重绘、位移或错误输入。
- **按小样验证后批量。** 先修正 SAM 输入,再用头、前臂、上衣验证参考图与遮罩,比较原输入、原候选、合并图和回贴图,确认改善后再跑全量。 - **按小样验证后批量。** 先修正 SAM 输入,再用头、前臂、上衣验证参考图与遮罩,比较原输入、原候选、合并图和回贴图,确认改善后再跑全量。
+128
View File
@@ -0,0 +1,128 @@
# 第四阶段:角色组装与 Spine 导出
## 1. 阶段简介
将第二阶段的可见部件与第三阶段通过几何检查的补全部件整合,按原图坐标组装,并依据 v2 语义计划生成 Spine 4.2 骨架、region 附件、图集及预览。所有部件名称、数量、骨骼父子关系和附件绑定来自输入,不使用角色名称或人形比例模板。
生成过程完全离线,不调用图片模型、不运行 SAM。骨架预览页面嵌入资产,但会从 CDN 加载 Spine Player 4.2,打开播放器需要网络。静态报告与 PNG 可离线查看。
导出完成仍标记 `quality_status=needs_review`。当前只有 setup 与单骨骼旋转检查动画;没有成品 idle/walk,也没有 mesh、权重或 IK。关节、语义、接缝及运动质量需人工复核。
## 2. 输入、输出与执行链路
```text
SAM 目录 + 可选补全目录 + 可选绘制顺序覆盖
→ 校验 v2 计划与骨骼关系
→ 对比两阶段 source / 计划 / segmentation / masks 的 SHA-256
→ 合并部件:已选补全 / 原 SAM 像素 / 隐藏无纹理节点
→ 根据遮挡约束稳定拓扑排序,记录没有约束的重叠
→ 按 crop_xyxy 组装,输出来源与覆盖诊断
→ 根据 parent / bind_to / pivot_norm 创建骨架与附件
→ 生成 skeleton.json、图集、播放器、报告与 manifest
```
SAM 原部件直接由 source + mask 重建,保持 RGB/Alpha 和原坐标;无需依赖 SAM 的紧裁剪 parts/。补全部件使用 `completion-result.json` 中选中结果的 `image` 和 `crop_xyxy`,同时检查尺寸及可见 RGBA 是否保持不变。
## 3. 阶段执行命令示例及参数说明
在项目根目录运行;环境统一为 Conda `ai-spine`。以下路径仅是现有输入示例,换角色时替换输入目录,不修改代码。输出目录每次独立。
```powershell
Set-Location F:\Study\taonier\AI_spine
conda activate ai-spine
$samDir = '.\output\20260920-225137\sam'
$completionDir = '.\output\20260920-225137\complete'
$spineOut = '.\output\spine-' + (Get-Date -Format 'yyyyMMdd-HHmmss')
node .\tools\model-check.mjs --stage spine `
--input-dir $samDir `
--completion-dir $completionDir `
--out-dir $spineOut
Start-Process "$spineOut\report.html"
```
| 参数 | 含义 |
| --- | --- |
| `--input-dir DIR` | 必需,SAM 输出目录 |
| `--completion-dir DIR` | 可选,同批输入的补全输出;省略则使用 SAM 资产 |
| `--out-dir DIR` | 独立输出目录;dry-run 时可省略 |
| `--fallback-policy sam/error` | 默认 sam;需要补全但没有通过结果时回退可见像素并记录原因。error 则终止 |
| `--draw-order-file FILE` | JSON 对象,`draw_order` 包含所有可绘制 ID,按从后到前排列,各出现一次 |
| `--check-angle FLOAT` | 单骨骼检查动画正负角度,默认 10,范围 (0,45] |
| `--genielabs-dir DIR` | 默认相邻 `spine-animation-ai`;复用其图集和播放器脚本,保留 PolyForm Noncommercial 约束 |
| `--python FILE` | 仅允许 ai-spine 解释器 |
| `--dry-run` | 完整检查输入、PNG、坐标及依赖脚本,不写输出,不联网 |
| `--resume` | 仅复用输入、参数、代码与产物哈希全部匹配的完整运行 |
该阶段不读取模型配置与密钥,也不接受 `--config`、模型参数或补全请求参数。自动化可用 `conda run --no-capture-output -n ai-spine node ...`。
可选 runtime 复检:将已有的官方 Spine 4.2 core IIFE 文件路径赋给 `$runtimeFile`,执行以下命令。工具不下载或安装依赖;检查图集尺寸、静态附件顶点和正负旋转下的后代跟随,不验证浏览器 WebGL 渲染。
```powershell
$runtimeFile = '.\output\spine-runtime-validation-20260922\spine-core.js'
conda run -n ai-spine node .\tools\verify-spine-runtime.mjs `
--runtime-file $runtimeFile --input-dir $spineOut
```
## 4. 输入格式要求
SAM 目录需含 `source.png`、有效 v2 `semantic-parts.json`、`segmentation.json`、其中记录的全画布二值 masks。可见部件缺少 mask 直接报错;hidden 不得有可见 mask。
补全目录需含相同输入快照、对应 masks 和 `completion-result.json`。只有 `selected_file` 非空、`qa.accepted=true` 且 `status=geometry_passed_needs_review` 的结果会被采用;其 `image` 文件必须存在,裁剪坐标为原画布内的整数半开区间,尺寸与 PNG 一致。legacy 可见像素必须逐像素匹配 SAM;adaptive 使用 preservation 的 preserve/editable/removed 全画布二值区域,要求 preserve=SAM−editable−removed、编辑与移除不重叠、保留像素精确匹配、输出没有超出保留/编辑区域。输入哈希不一致时拒绝混用。图片和契约路径必须位于补全目录内。
补全失败、未处理、未提供或没有允许区域的需补全部件遵守 fallback-policy;adaptive 已重建并尝试修复的部件也视为有需求,不因原计划 needs_completion=false 就忽略失败。hidden 保留语义骨骼但不生成纹理,也不因缺纹理触发严格回退错误。
可选绘制顺序文件格式为 `{"draw_order":["部件ID一","部件ID二"]}`。必须列出本次全部可绘制部件,不包含 hidden。显式顺序允许覆盖模型遮挡关系,冲突会在报告中列出。
## 5. 执行链路中重要部分代码文件及原理说明(包括重点说明)
**资产整合。** 按 ID 合并,保留来源、跳过或拒绝原因、补全 QA 与上游告警。通过几何检查的补全仍是待复核资产,失败结果不会当作补全成功使用。缺少图片或非法坐标不会被静默回退掩盖。
**位置与骨骼。** 图像坐标原点为左上角,Spine 原点设为原画布底边中点:`X=x-W/2`,`Y=H-y`。pivot 使用连续坐标 `x=pivot_norm[0]*W`、`y=pivot_norm[1]*H`。所有骨骼 setup 旋转为零、缩放为一,因此子骨骼局部位置是世界 pivot 减父 pivot;附件位置是裁剪中心减绑定骨骼 pivot。骨骼按父先子后输出,hidden bone_part 也创建骨骼。attachment/facial_feature 绑定 bind_to,null 绑定 root,不创建额外独立骨骼。
**绘制顺序。** `A.occluded_by` 包含 B 时,A 排在 B 后方。排序保留隐藏节点的传递约束,最终只输出有纹理的 slot。没有约束时使用计划中的稳定顺序;若这些部件确有 Alpha 重叠,记录 unconstrained_overlaps,不能声称已确定遮挡。循环关系拒绝导出。
**检查动画。** 每个语义骨骼生成 `check__ID`:0→正角度→0→负角度→0,时长 2 秒,子骨骼与附件按层级跟随。setup 是静态姿态。它们供人工检查绑定和露底,不自动判定动作质量。
Spine 字段依据 [官方 JSON 格式说明](https://esotericsoftware.com/spine-json-format),导出目标为 4.2;旋转时间线使用 4.2 的 `value` 字段。
## 6. 各个代码文件职责索引
| 文件 | 职责 |
| --- | --- |
| `tools/model-check.mjs` | 独立离线入口、参数与其他阶段隔离 |
| `stages/spine-stage.mjs` | 验证计划、Conda 调用、指纹、导出调度与复用 |
| `spine_assembly.py` | 资产核对、顺序、组装、诊断、骨架及报告 |
| `lib/validate-parts.mjs` | v2 Schema、引用、骨骼和遮挡环路校验 |
| `lib/run-artifacts.mjs` | 独立输出目录和产物哈希 |
| 外部 `scripts/make_atlas.py` | 透明 PNG 图集打包,不修改外部依赖 |
| 外部 `scripts/generate_spine_player.py` | 嵌入骨架和图集的 Spine Player HTML |
| `tests/test_spine_assembly.py` / `tests/spine-stage.test.mjs` | 非人形合成数据、坐标、父子绑定、输入身份、回退、CLI 与完整导出回归 |
| `tools/verify-spine-runtime.mjs` | 可选官方 runtime 加载、图集匹配与世界顶点/旋转复检,需提供本地 core IIFE |
## 7. 产物详细说明
| 文件 | 用途 |
| --- | --- |
| `source.png` / `semantic-parts.json` | 原图与语义计划快照 |
| `images/<id>.png` | 完整可绘制部件集合,供 Spine 编辑器导入 |
| `assembly.json` | 坐标、来源、回退、顺序、QA、输入哈希与覆盖诊断 |
| `assembled-sam.png` / `assembled.png` | 同一绘制顺序的原 SAM / 最终资产透明组装图 |
| `comparison.png` | 左原图、中 SAM 组装、右最终组装 |
| `coverage-diagnostic.png` | 红色为原图前景未覆盖,黄色为多部件重叠;重叠不一定是错误 |
| `skeleton.json` | Spine 4.2 骨骼、slot、region、setup 和关节检查动画;images 路径为 `./images/` |
| `atlas/skeleton.atlas` / `atlas/skeleton.png` | 运行时图集 |
| `preview.html` | 真正的骨架预览,可切换检查动画;需要联网加载播放器 |
| `report.html` | 静态对比、诊断、来源列表与预览入口 |
| `manifest.json` | 执行状态、质量状态、输入/代码/产物哈希、计数、依赖及限制 |
2026-09-22 验证记录:`conda run --no-capture-output -n ai-spine npm test` 的 26 项测试全部通过,包含 6 项新增 Python 组装用例。`output/spine-validation-latest-20260922` 和 `output/spine-validation-baseline-20260922` 分别导出 25/22 个 slot、各 16 个骨骼(含 root);官方 Spine 4.2 runtime 对两者的 setup 顶点与各 30 个旋转采样验证通过。最新导出的播放器另经 headless Chrome 实际加载,显示 Loaded successfully 并渲染角色,截图位于 `output/spine-runtime-validation-20260922/preview.png`。这不代表资产或动作质量合格,现有输入的漏分仍显示在组装对比图中。
## 8. 当前阶段可改进点说明
- 待实现:针对未确定遮挡、pivot 和绑定的交互修正界面;目前可通过语义计划与绘制顺序文件调整。
- 待实现:可靠的接缝、露底与旋转质量自动验收;目前输出检查动画与诊断,质量状态始终 needs_review。
- 待实现:mesh、权重、IK,以及由动作语义驱动的成品动画;不通过固定部件名称硬套人形模板。
- 待验证:不同角色类别与更复杂遮挡关系的资产质量。合成回归只验证数据契约及几何逻辑。
- 待实现:本地播放器资源打包,使骨架预览也可完全离线打开。
+8 -4
View File
@@ -1,5 +1,8 @@
Complete only the hidden portion of the specified character part on the first Repair and complete the specified character part on the first input image.
input image (the local canvas). Any subsequent images are context references. The task declares editable regions for segmentation gaps, occlusion, joint
extensions, and removal of overlaid attachments from their underlying base.
Only regions explicitly marked editable may be changed. Preserve all other
visible pixels. Any subsequent images or labeled reference panels are context.
Preserve the exact original pose, scale, screen direction, perspective, colors, Preserve the exact original pose, scale, screen direction, perspective, colors,
line thickness and shading. Do not mirror, rotate, resize, recenter or redesign. line thickness and shading. Do not mirror, rotate, resize, recenter or redesign.
Keep the canvas dimensions and the position of every already-visible feature. Keep the canvas dimensions and the position of every already-visible feature.
@@ -8,5 +11,6 @@ body parts, props, labels, annotations or extra versions of the same part.
If an edit mask is supplied, transparent mask pixels are editable. Otherwise, If an edit mask is supplied, transparent mask pixels are editable. Otherwise,
use the task's bounded extension instructions and keep the remaining background use the task's bounded extension instructions and keep the remaining background
pure white or transparent. Return the same canvas size without recentering. pure white or transparent. Return the same canvas size without recentering.
Downstream checks will reject misalignment, restrict additions to allowed regions, Downstream checks reject misalignment and constrain modifications to edit regions,
and restore the source visible pixels. Do not rely on restoration to hide changes. restore protected source pixels, and reverse the documented canvas transform.
Do not rely on restoration to hide movement or redesign.
+345
View File
@@ -0,0 +1,345 @@
"""Offline asset assembly and Spine 4.2 region export, driven by semantic v2 data."""
import argparse
import hashlib
import html
import json
from pathlib import Path
import numpy as np
from PIL import Image
def write_json(file, data):
Path(file).write_text(json.dumps(data, ensure_ascii=False, indent=2, allow_nan=False) + '\n', encoding='utf-8')
def safe_file(root, name):
if not isinstance(name, str) or not name or Path(name).is_absolute():
raise ValueError(f'Expected relative asset path: {name}')
root = Path(root).resolve()
file = (root / name).resolve()
if not file.is_relative_to(root) or not file.is_file():
raise ValueError(f'Missing asset or path outside input directory: {name}')
return file
def ordered(nodes, edges):
"""Stable topological order; do not silently break contradictory constraints."""
pending = list(nodes)
result = []
while pending:
ready = next((n for n in pending if not any(b == n and a in pending for a, b in edges)), None)
if ready is None:
raise ValueError('Cyclic bone or draw-order constraints')
pending.remove(ready)
result.append(ready)
return result
def load_assets(input_dir, completion_dir=None, fallback_policy='sam', draw_order_file=None):
root = Path(input_dir).resolve()
hashes = {}
def tracked(folder, name, label):
file = safe_file(folder, name)
hashes[f'{label}/{name}'] = hashlib.sha256(file.read_bytes()).hexdigest()
return file
def read(folder, name, label):
return json.loads(tracked(folder, name, label).read_text(encoding='utf-8-sig'))
source = Image.open(tracked(root, 'source.png', 'sam')).convert('RGBA')
rgba = np.array(source)
w, h = source.size
plan = read(root, 'semantic-parts.json', 'sam')
seg = read(root, 'segmentation.json', 'sam')
parts = plan['parts']
by_id = {p['id']: p for p in parts}
if len(by_id) != len(parts):
raise ValueError('Duplicate semantic ID')
def records(items, label):
result = {}
for item in items:
name = item['id']
if name not in by_id or name in result:
raise ValueError(f'{label}: unknown or duplicate ID: {name}')
result[name] = item
return result
segmented = records(seg['parts'], 'segmentation')
completed, skipped = {}, {}
completion = Path(completion_dir).resolve() if completion_dir else None
# A completion run snapshots its SAM inputs, so compare content, not directory names.
if completion:
for name in ['source.png', 'semantic-parts.json', 'segmentation.json']:
tracked(completion, name, 'completion')
if hashes[f'sam/{name}'] != hashes[f'completion/{name}']:
raise ValueError(f'Completion input mismatch: {name}')
result = read(completion, 'completion-result.json', 'completion')
completed = records(result['parts'], 'completion')
skipped = records(result.get('skipped', []), 'completion skipped')
if set(completed) & set(skipped):
raise ValueError('Completion ID occurs in both parts and skipped')
images, original_images, asset_records = {}, {}, []
masks = {}
for name, record in segmented.items():
if by_id[name]['visibility'] == 'hidden':
raise ValueError(f'Hidden part unexpectedly has a SAM mask: {name}')
mask_name = record.get('mask', f'masks/{name}.png')
mask = np.array(Image.open(tracked(root, mask_name, 'sam')))
if mask.shape != (h, w) or not np.isin(mask, [0, 255]).all():
raise ValueError(f'Invalid full-canvas binary mask: {name}')
mask = mask > 0
if not mask.any() or np.any(mask & (rgba[:, :, 3] == 0)):
raise ValueError(f'Empty mask or mask outside source alpha: {name}')
if completion:
tracked(completion, mask_name, 'completion')
if hashes[f'sam/{mask_name}'] != hashes[f'completion/{mask_name}']:
raise ValueError(f'Completion mask mismatch: {name}')
masks[name] = mask
for part in parts:
name = part['id']
if part['visibility'] == 'hidden':
asset_records.append({'id': name, 'source': 'hidden', 'image': None,
'reason': 'HIDDEN_NO_TEXTURE', 'semantic_verified': False})
continue
if name not in masks:
raise ValueError(f'Visible part has no SAM mask: {name}')
mask = masks[name]
ys, xs = np.nonzero(mask)
original_box = [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1]
cut = rgba.copy()
cut[~mask] = 0
original = Image.fromarray(cut).crop(original_box)
original_images[name] = (original, original_box)
candidate = completed.get(name, {})
accepted = bool(candidate.get('selected_file')) and candidate.get('qa', {}).get('accepted') is True
warnings = list(segmented[name].get('warnings', []))
if accepted:
if candidate.get('status') != 'geometry_passed_needs_review':
raise ValueError(f'Inconsistent selected completion status: {name}')
image = Image.open(tracked(completion, candidate.get('image'), 'completion')).convert('RGBA')
box = candidate.get('crop_xyxy')
if (not isinstance(box, list) or len(box) != 4 or
any(type(x) is not int for x in box) or
not (0 <= box[0] < box[2] <= w and 0 <= box[1] < box[3] <= h) or
image.size != (box[2] - box[0], box[3] - box[1])):
raise ValueError(f'Invalid completion crop/size: {name}')
full = Image.new('RGBA', source.size)
full.paste(image, (box[0], box[1]))
preserve = mask
preservation = candidate.get('preservation')
if preservation:
areas = {}
for key in ['preserve_mask', 'editable_mask', 'removed_mask']:
area = np.array(Image.open(tracked(completion, preservation.get(key), 'completion')))
if area.shape != (h, w) or not np.isin(area, [0, 255]).all():
raise ValueError(f'Invalid completion preservation mask: {name}/{key}')
areas[key] = area > 0
preserve, editable, removed = (areas[k] for k in ['preserve_mask', 'editable_mask', 'removed_mask'])
if (np.any(editable & removed) or np.any(removed & ~mask) or
not np.array_equal(preserve, mask & ~editable & ~removed) or not preserve.any()):
raise ValueError(f'Inconsistent completion preservation partition: {name}')
alpha = np.array(full)[:, :, 3] > 0
if np.any(alpha & ~(preserve | editable)):
raise ValueError(f'Completion pixels outside preservation/edit contract: {name}')
warnings.append('VISIBLE_EDITS_DECLARED_NEEDS_REVIEW')
if not np.array_equal(np.array(full)[preserve], rgba[preserve]):
raise ValueError(f'Completion does not preserve original visible RGBA: {name}')
origin, reason = 'completion', 'GEOMETRY_PASSED_NEEDS_REVIEW'
if candidate.get('qa', {}).get('unfulfilled_regions'):
warnings.append('UNFULFILLED_COMPLETION_REGIONS')
else:
image, box = original, original_box
origin = 'sam'
inferred_need = bool(candidate) or skipped.get(name, {}).get('reason') == 'AMBIGUOUS_MASK_REQUIRES_REVIEW'
if not part['needs_completion'] and not inferred_need:
reason = 'NO_COMPLETION_NEEDED'
else:
reason = skipped.get(name, {}).get('reason') or (
'COMPLETION_' + str(candidate.get('status', 'NOT_PROVIDED')).upper())
warnings.append('USING_UNCOMPLETED_SAM')
if fallback_policy == 'error':
raise ValueError(f'Completion required for {name}: {reason}')
images[name] = (image, box)
asset_records.append({'id': name, 'source': origin, 'reason': reason, 'image': f'images/{name}.png',
'crop_xyxy': box, 'original_crop_xyxy': original_box,
'semantic_verified': False, 'warnings': warnings,
'completion_status': candidate.get('status'),
'preservation': candidate.get('preservation'),
'completion_qa': candidate.get('qa')})
if not images:
raise ValueError('No drawable assets')
edges = [(p['id'], front) for p in parts for front in p['occluded_by']]
# Keep hidden nodes while sorting so transitive constraints survive their removal.
draw_order = [n for n in ordered(by_id, edges) if n in images]
overridden = False
if draw_order_file:
file = Path(draw_order_file).resolve()
hashes['draw_order_override'] = hashlib.sha256(file.read_bytes()).hexdigest()
override = json.loads(file.read_text(encoding='utf-8-sig'))
draw_order = override['draw_order']
if not isinstance(draw_order, list) or len(draw_order) != len(images) or set(draw_order) != set(images):
raise ValueError('draw_order must contain each drawable ID exactly once')
overridden = True
reach = {n: set() for n in by_id}
for n in reversed(ordered(by_id, edges)):
for a, b in edges:
if a == n:
reach[n].add(b)
reach[n].update(reach[b])
unresolved = []
for i, a in enumerate(draw_order):
for b in draw_order[i + 1:]:
if b in reach[a] or a in reach[b]:
continue
ai, ab = images[a]
bi, bb = images[b]
x0, y0, x1, y1 = max(ab[0], bb[0]), max(ab[1], bb[1]), min(ab[2], bb[2]), min(ab[3], bb[3])
if x0 < x1 and y0 < y1:
am = np.array(ai.getchannel('A'))[y0-ab[1]:y1-ab[1], x0-ab[0]:x1-ab[0]] > 0
bm = np.array(bi.getchannel('A'))[y0-bb[1]:y1-bb[1], x0-bb[0]:x1-bb[0]] > 0
overlap = int((am & bm).sum())
if overlap:
unresolved.append({'parts': [a, b], 'overlap_pixels': overlap})
violated = [[a, b] for a in draw_order for b in reach[a]
if b in images and draw_order.index(a) > draw_order.index(b)]
assembly = {'schema_version': 1, 'quality_status': 'needs_review', 'semantic_verified': False,
'width': w, 'height': h, 'coordinate_space': 'pixel_xy_top_left',
'parts': asset_records, 'draw_order': draw_order,
'draw_order_source': 'override' if overridden else 'occlusion_stable_topological',
'unconstrained_overlaps': unresolved, 'overridden_occlusion_constraints': violated,
'input_hashes': hashes}
return source, plan, assembly, images, original_images
def build_skeleton(plan, assembly, check_angle=10):
w, h = assembly['width'], assembly['height']
bone_parts = {p['id']: p for p in plan['parts'] if p['rig_role'] == 'bone_part'}
edges = [(p['parent'], name) for name, p in bone_parts.items() if p['parent']]
bone_order = ordered(bone_parts, edges)
world = {'root': (0, 0)}
bones = [{'name': 'root'}]
for name in bone_order:
part = bone_parts[name]
world[name] = (part['pivot_norm'][0] * w - w / 2, h - part['pivot_norm'][1] * h)
parent = part['parent'] or 'root'
x, y = world[name]
px, py = world[parent]
# Setup axes remain aligned with the canvas. Parent rotation still propagates
# to children; attachment offsets below preserve the exact original pose.
bones.append({'name': name, 'parent': parent, 'x': x - px, 'y': y - py})
semantic = {p['id']: p for p in plan['parts']}
assets = {p['id']: p for p in assembly['parts']}
slots, attachments = [], {}
for name in assembly['draw_order']:
part, asset = semantic[name], assets[name]
bone = name if part['rig_role'] == 'bone_part' else part['bind_to'] or 'root'
bx, by = world[bone]
x0, y0, x1, y1 = asset['crop_xyxy']
slots.append({'name': name, 'bone': bone, 'attachment': name})
attachments[name] = {name: {'type': 'region', 'path': name,
'x': (x0 + x1) / 2 - w / 2 - bx,
'y': h - (y0 + y1) / 2 - by,
'width': x1 - x0, 'height': y1 - y0}}
animations = {'setup': {}}
for name in bone_order:
animations[f'check__{name}'] = {'bones': {name: {'rotate': [
{'time': 0, 'value': 0}, {'time': .5, 'value': check_angle},
{'time': 1, 'value': 0}, {'time': 1.5, 'value': -check_angle}, {'time': 2, 'value': 0}]}}}
return {'skeleton': {'spine': '4.2.0', 'x': -w / 2, 'y': 0, 'width': w, 'height': h,
'images': './images/'},
'bones': bones, 'slots': slots, 'skins': [{'name': 'default', 'attachments': attachments}],
'animations': animations}
def compose(size, images, order):
canvas = Image.new('RGBA', size)
coverage = np.zeros((size[1], size[0]), np.uint16)
for name in order:
image, (x0, y0, x1, y1) = images[name]
canvas.alpha_composite(image, (x0, y0))
coverage[y0:y1, x0:x1] += np.array(image.getchannel('A')) > 0
return canvas, coverage
def export_assets(data, output, check_angle=10):
source, plan, assembly, images, original_images = data
out = Path(output)
(out / 'images').mkdir(parents=True, exist_ok=True)
source.save(out / 'source.png')
write_json(out / 'semantic-parts.json', plan)
for name, (im, _) in images.items():
im.save(out / 'images' / f'{name}.png')
assembled, coverage = compose(source.size, images, assembly['draw_order'])
before, _ = compose(source.size, original_images, assembly['draw_order'])
assembled.save(out / 'assembled.png')
before.save(out / 'assembled-sam.png')
foreground = np.array(source.getchannel('A')) > 0
missing = foreground & (coverage == 0)
diagnostic = np.zeros((source.height, source.width, 4), np.uint8)
diagnostic[coverage > 1] = [255, 180, 0, 220]
diagnostic[missing] = [255, 0, 80, 255]
Image.fromarray(diagnostic).save(out / 'coverage-diagnostic.png')
comparison = Image.new('RGBA', (source.width * 3, source.height), 'white')
for i, im in enumerate([source, before, assembled]):
comparison.alpha_composite(im, (i * source.width, 0))
comparison.convert('RGB').save(out / 'comparison.png')
assembly['diagnostics'] = {'uncovered_source_pixels': int(missing.sum()),
'overlap_pixels': int((coverage > 1).sum()),
'rotation_quality_verified': False}
write_json(out / 'assembly.json', assembly)
skeleton = build_skeleton(plan, assembly, check_angle)
write_json(out / 'skeleton.json', skeleton)
rows = []
for part in assembly['parts']:
picture = f'<img src="{part["image"]}">' if part['image'] else ''
rows.append('<tr><td>' + html.escape(part['id']) + '</td><td>' + html.escape(part['source']) +
'</td><td>' + html.escape(part['reason']) + '</td><td>' + picture + '</td></tr>')
notes = {key: assembly[key] for key in ['diagnostics', 'draw_order_source', 'unconstrained_overlaps',
'overridden_occlusion_constraints']}
(out / 'report.html').write_text('''<!doctype html><html lang="zh-CN"><meta charset="utf-8">
<title>第四阶段:Spine 组装与导出</title>
<style>body{font:16px system-ui;margin:24px}img{max-width:100%;background:#ddd}td img{max-height:150px}td,th{border:1px solid #ccc;padding:8px}pre{white-space:pre-wrap}</style>
<h1>第四阶段:Spine 组装与导出</h1>
<p>状态:needs_review。几何通过与成功导出不代表语义、接缝或动作质量合格。</p>
<p><a href="preview.html">Spine 预览</a>(播放器从 CDN 加载,需要网络);选择 setup 查看静态姿态,选择 check__骨骼ID 检查单关节旋转及子骨骼跟随。检查动画不代表成品动作。</p>
<p><a href="skeleton.json">骨架 JSON</a> · <a href="assembly.json">来源与诊断</a></p>
<h2>原图 / SAM 组装 / 最终组装</h2><img src="comparison.png">
<h2>覆盖诊断:红色为原图漏分,黄色为部件重叠</h2><img src="coverage-diagnostic.png">
<p>重叠可能是正常遮挡;覆盖率不能判断语义。没有遮挡约束的重叠使用语义计划中的稳定顺序,需复核。</p>
<pre>''' + html.escape(json.dumps(notes, ensure_ascii=False, indent=2)) + '''</pre>
<h2>资产来源</h2><table><tr><th>ID</th><th>来源</th><th>原因</th><th>部件</th></tr>''' +
''.join(rows) + '</table></html>', encoding='utf-8')
return {'parts': len(images), 'bones': len(skeleton['bones']),
'completion_parts': sum(p['source'] == 'completion' for p in assembly['parts']),
'fallback_parts': sum('USING_UNCOMPLETED_SAM' in p.get('warnings', []) for p in assembly['parts']),
'quality_status': 'needs_review', 'input_hashes': assembly['input_hashes']}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--input-dir', required=True)
parser.add_argument('--completion-dir')
parser.add_argument('--output')
parser.add_argument('--draw-order-file')
parser.add_argument('--fallback-policy', choices=['sam', 'error'], default='sam')
parser.add_argument('--check-angle', type=float, default=10)
args = parser.parse_args()
if not 0 < args.check_angle <= 45:
parser.error('check-angle must be in (0, 45]')
data = load_assets(args.input_dir, args.completion_dir, args.fallback_policy, args.draw_order_file)
if args.output:
result = export_assets(data, args.output, args.check_angle)
else:
skeleton = build_skeleton(data[1], data[2], args.check_angle)
result = {'input_hashes': data[2]['input_hashes'], 'parts': len(data[3]), 'bones': len(skeleton['bones'])}
print(json.dumps(result, ensure_ascii=False))
if __name__ == '__main__':
main()
+96 -15
View File
@@ -23,14 +23,34 @@ function filesIn(dir, prefix = '') {
} }
export async function runCompletion({stage, inputDir, outDir, partIds = [], prepareOnly = false, export async function runCompletion({stage, inputDir, outDir, partIds = [], prepareOnly = false,
candidatesDir, completionSize = 1024, extensionRadius = 32, python: requestedPython, candidatesDir, completionSize = 1024, extensionRadius, python: requestedPython,
resume = false, dryRun = false, candidateSizePolicy = 'strict'}, transport) { resume = false, dryRun = false, candidateSizePolicy = 'strict', completionProfile = 'adaptive',
completionReference = 'auto', completionMask = 'auto', completionReview, probeEditCapabilities = false}, transport) {
if (!inputDir) throw new Error('部件补全需要 --input-dir 指向 SAM 输出目录'); if (!inputDir) throw new Error('部件补全需要 --input-dir 指向 SAM 输出目录');
if (!outDir && !dryRun) throw new Error('部件补全需要独立的 --out-dir'); if (!outDir && !dryRun) throw new Error('部件补全需要独立的 --out-dir');
if (prepareOnly && candidatesDir) throw new Error('--prepare-only 与 --candidates-dir 不能同时使用'); if (prepareOnly && candidatesDir) throw new Error('--prepare-only 与 --candidates-dir 不能同时使用');
const size = Number(completionSize), radius = Number(extensionRadius); const size = Number(completionSize), radius = extensionRadius === undefined
? completionProfile === 'legacy' ? 32 : null : Number(extensionRadius);
if (!['legacy', 'adaptive'].includes(completionProfile)) throw new Error('--completion-profile 仅支持 adaptive / legacy');
if (!['auto', 'none', 'local', 'full', 'both', 'board'].includes(completionReference)) throw new Error('不支持的 --completion-reference');
if (!['auto', 'required', 'off'].includes(completionMask)) throw new Error('--completion-mask 仅支持 auto / required / off');
if (probeEditCapabilities && (partIds.length !== 1 || prepareOnly || candidatesDir || completionProfile !== 'adaptive')) {
throw new Error('--probe-edit-capabilities 需要 adaptive、恰好一个 --part 和在线请求');
}
const referenceMode = probeEditCapabilities ? 'both' : completionReference === 'auto'
? completionProfile === 'legacy' ? stage.capabilities.multiple_images ? 'local' : 'none'
: stage.capabilities.multiple_images ? 'both' : 'board' : completionReference;
if (completionProfile === 'legacy' && (completionReview || !['local', 'none'].includes(referenceMode))) {
throw new Error('legacy 不支持 review 或 full/both/board 参考,请使用 adaptive');
}
const nativeReferences = ['local', 'full', 'both'].includes(referenceMode);
const maskSent = probeEditCapabilities || completionMask === 'required' || (completionMask === 'auto' && stage.capabilities.mask);
const capabilitiesAvailable = (!nativeReferences || stage.capabilities.multiple_images) && (!maskSent || stage.capabilities.mask);
if (!capabilitiesAvailable && !probeEditCapabilities && !prepareOnly && !candidatesDir && !dryRun) {
throw new Error('请求需要未确认的多图/mask 能力;可先 prepare-only,或用单部件 --probe-edit-capabilities 实测,不要直接更改默认 capabilities');
}
if (!['strict', 'uniform'].includes(candidateSizePolicy)) throw new Error('--candidate-size-policy 仅支持 strict / uniform'); if (!['strict', 'uniform'].includes(candidateSizePolicy)) throw new Error('--candidate-size-policy 仅支持 strict / uniform');
if (!Number.isInteger(size) || size < 128 || size > 4096 || !Number.isInteger(radius) || radius < 1 || radius > 256) { if (!Number.isInteger(size) || size < 128 || size > 4096 || (radius !== null && (!Number.isInteger(radius) || radius < 1 || radius > 256))) {
throw new Error('--completion-size 必须为 128~4096 的整数;--extension-radius 必须为 1~256 的整数'); throw new Error('--completion-size 必须为 128~4096 的整数;--extension-radius 必须为 1~256 的整数');
} }
const input = path.resolve(inputDir); const input = path.resolve(inputDir);
@@ -46,15 +66,43 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
const source = readImage(path.join(input, 'source.png'), {pngOnly: true}); const source = readImage(path.join(input, 'source.png'), {pngOnly: true});
const inputFiles = ['source.png', 'semantic-parts.json', 'segmentation.json', ...segmentation.parts.map((p) => `masks/${p.id}.png`)]; const inputFiles = ['source.png', 'semantic-parts.json', 'segmentation.json', ...segmentation.parts.map((p) => `masks/${p.id}.png`)];
const inputs = Object.fromEntries(inputFiles.map((file) => [file, sha256(readFileSync(path.join(input, file)))])); const inputs = Object.fromEntries(inputFiles.map((file) => [file, sha256(readFileSync(path.join(input, file)))]));
const reviewFiles = [], reviewHashes = {};
let reviewData;
if (completionReview) {
const file = path.resolve(completionReview);
reviewData = readJson(file);
if (reviewData.schema_version !== 1 || !reviewData.parts || Array.isArray(reviewData.parts)) throw new Error('review 需要 schema_version=1 和 parts 对象');
reviewHashes['review.json'] = sha256(readFileSync(file));
reviewFiles.push({file, target: 'review-source.json'});
for (const [id, item] of Object.entries(reviewData.parts)) {
if (!names.has(id) || !item || item.confirmed !== true) throw new Error('review 部件必须有效且 confirmed=true');
for (const field of ['repair_mask', 'replace_mask', 'remove_mask']) {
if (!Object.hasOwn(item, field)) continue;
const ref = item[field];
if (typeof ref !== 'string' || path.isAbsolute(ref)) throw new Error('review mask 必须是相对路径');
const maskFile = path.resolve(path.dirname(file), ref), relative = path.relative(path.dirname(file), maskFile);
if (relative === '..' || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) throw new Error('review mask 不能越出 review 目录');
const target = `review/${id}-${field}.png`;
reviewHashes[`${id}/${field}`] = sha256(readFileSync(maskFile));
reviewFiles.push({file: maskFile, target});
item[field] = target;
}
}
}
const planning = {profile: completionProfile, reference_mode: referenceMode, review_hashes: reviewHashes,
extension_radius: radius, planner_version: completionProfile === 'adaptive' ? 2 : 1};
const imported = {}; const imported = {};
if (candidatesDir) { if (candidatesDir) {
const priorManifest = path.resolve(candidatesDir, 'manifest.json'); const priorManifest = path.resolve(candidatesDir, 'manifest.json');
const previousRun = existsSync(priorManifest) ? readJson(priorManifest) : null; const previousRun = existsSync(priorManifest) ? readJson(priorManifest) : null;
const previousPlanning = previousRun?.planning || {profile: 'legacy', reference_mode: previousRun?.resolved_stage?.capabilities?.multiple_images ? 'local' : 'none',
review_hashes: {}, extension_radius: previousRun?.extension_radius, planner_version: 1};
if (previousRun && (previousRun.output_kind !== 'bounded_part_completion' || previousRun.completion_size !== size || previousRun.extension_radius !== radius || if (previousRun && (previousRun.output_kind !== 'bounded_part_completion' || previousRun.completion_size !== size || previousRun.extension_radius !== radius ||
JSON.stringify(previousPlanning) !== JSON.stringify(planning) ||
Object.entries(inputs).some(([file, hash]) => previousRun.inputs?.[file] !== hash))) { Object.entries(inputs).some(([file, hash]) => previousRun.inputs?.[file] !== hash))) {
throw new Error('旧补全目录的输入或画布/延展参数不一致,不能复用其候选'); throw new Error('旧补全目录的输入或画布/延展参数不一致,不能复用其候选');
} }
for (const p of plan.parts.filter((p) => p.needs_completion && (!partIds.length || partIds.includes(p.id)))) { for (const p of plan.parts.filter((p) => (completionProfile === 'adaptive' || p.needs_completion) && (!partIds.length || partIds.includes(p.id)))) {
const file = previousRun const file = previousRun
? ['png', 'jpg', 'webp'].map((ext) => path.resolve(candidatesDir, 'tasks', p.id, `candidate-1.${ext}`)).find(existsSync) ? ['png', 'jpg', 'webp'].map((ext) => path.resolve(candidatesDir, 'tasks', p.id, `candidate-1.${ext}`)).find(existsSync)
: path.resolve(candidatesDir, `${p.id}.png`); : path.resolve(candidatesDir, `${p.id}.png`);
@@ -66,16 +114,18 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
if (dryRun) { if (dryRun) {
console.log(JSON.stringify({stage: 'completion', input, width: source.width, height: source.height, console.log(JSON.stringify({stage: 'completion', input, width: source.width, height: source.height,
selected_parts: partIds, completion_size: size, extension_radius: radius, candidate_size_policy: candidateSizePolicy, selected_parts: partIds, completion_size: size, extension_radius: radius, candidate_size_policy: candidateSizePolicy,
planning, mask_requested: maskSent, capabilities_available: capabilitiesAvailable,
mode: prepareOnly ? 'prepare' : candidatesDir ? 'offline_candidates' : 'image_edit', network_requested: false}, null, 2)); mode: prepareOnly ? 'prepare' : candidatesDir ? 'offline_candidates' : 'image_edit', network_requested: false}, null, 2));
return {dryRun: true}; return {dryRun: true};
} }
const codeFiles = ['stages/completion-stage.mjs', 'completion_geometry.py', 'normalize_reference.py', const codeFiles = ['stages/completion-stage.mjs', 'completion_geometry.py', 'completion_planning.py', 'normalize_reference.py',
'lib/config.mjs', 'lib/python-runtime.mjs', 'lib/run-artifacts.mjs', 'lib/images.mjs', 'lib/config.mjs', 'lib/python-runtime.mjs', 'lib/run-artifacts.mjs', 'lib/images.mjs',
'lib/validate-parts.mjs', 'schemas/semantic-parts.schema.json', 'providers/image-edits.mjs', 'providers/http.mjs']; 'lib/validate-parts.mjs', 'schemas/semantic-parts.schema.json', 'providers/image-edits.mjs', 'providers/http.mjs'];
const identity = {schema_version: 1, stage: 'completion', output_kind: 'bounded_part_completion', const identity = {schema_version: 1, stage: 'completion', output_kind: 'bounded_part_completion',
mode: prepareOnly ? 'prepare' : candidatesDir ? 'offline_candidates' : 'image_edit', mode: prepareOnly ? 'prepare' : candidatesDir ? 'offline_candidates' : 'image_edit',
resolved_stage: publicStage(stage), endpoint_sha256: sha256(stage.baseUrl), prompt_sha256: sha256(stage.prompt), resolved_stage: publicStage(stage), endpoint_sha256: sha256(stage.baseUrl), prompt_sha256: sha256(stage.prompt),
inputs, part_ids: partIds, completion_size: size, extension_radius: radius, candidate_size_policy: candidateSizePolicy, inputs, part_ids: partIds, completion_size: size, extension_radius: radius, candidate_size_policy: candidateSizePolicy,
planning, mask_requested: maskSent, capability_probe: probeEditCapabilities,
imported_candidates: Object.fromEntries(Object.entries(imported).map(([id, value]) => [id, value.sha256])), imported_candidates: Object.fromEntries(Object.entries(imported).map(([id, value]) => [id, value.sha256])),
code_sha256: sha256(codeFiles.map((file) => `${file}\n${readFileSync(path.join(projectRoot, file), 'utf8')}`).join('\n'))}; code_sha256: sha256(codeFiles.map((file) => `${file}\n${readFileSync(path.join(projectRoot, file), 'utf8')}`).join('\n'))};
const fingerprint = sha256(JSON.stringify(identity)); const fingerprint = sha256(JSON.stringify(identity));
@@ -83,24 +133,51 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
if (run.reused) return {out: run.out, reused: true}; if (run.reused) return {out: run.out, reused: true};
const manifest = {...identity, fingerprint, status: 'running', quality_status: 'pending', started_at: new Date().toISOString(), network_requested: false}; const manifest = {...identity, fingerprint, status: 'running', quality_status: 'pending', started_at: new Date().toISOString(), network_requested: false};
jsonFile(run.manifestPath, manifest, stage.apiKey); jsonFile(run.manifestPath, manifest, stage.apiKey);
const result = {schema_version: 1, quality_status: 'needs_review', parts: [], skipped: []}; const result = {schema_version: completionProfile === 'adaptive' ? 2 : 1, profile: completionProfile,
quality_status: 'needs_review', parts: [], skipped: []};
try { try {
const requestSize = `${size}x${size}`; const requestSize = `${size}x${size}`;
if (!prepareOnly && !candidatesDir && stage.params.size !== undefined && stage.params.size !== requestSize) { if (!prepareOnly && !candidatesDir && stage.params.size !== undefined && stage.params.size !== requestSize) {
throw new Error(`completion.params.size 必须与局部画布一致:${requestSize}`); throw new Error(`completion.params.size 必须与局部画布一致:${requestSize}`);
} }
const editStage = {...stage, params: {...stage.params, size: requestSize}}; const editStage = {...stage, params: {...stage.params, size: requestSize},
...(probeEditCapabilities ? {capabilities: {multiple_images: true, mask: true}, maxRetries: 0} : {})};
if (probeEditCapabilities) editStage.params.n = 1;
for (const file of inputFiles) { for (const file of inputFiles) {
mkdirSync(path.dirname(path.join(run.out, file)), {recursive: true}); mkdirSync(path.dirname(path.join(run.out, file)), {recursive: true});
copyFileSync(path.join(input, file), path.join(run.out, file)); copyFileSync(path.join(input, file), path.join(run.out, file));
} }
pythonCall(python, ['prepare', '--out', run.out, '--size', String(size), '--radius', String(radius), ...partIds.flatMap((id) => ['--part', id])]); for (const {file, target} of reviewFiles) {
mkdirSync(path.dirname(path.join(run.out, target)), {recursive: true});
copyFileSync(file, path.join(run.out, target));
}
if (reviewData) jsonFile(path.join(run.out, 'review.json'), reviewData);
pythonCall(python, ['prepare', '--out', run.out, '--size', String(size), '--profile', completionProfile,
'--reference-mode', referenceMode, ...(radius === null ? [] : ['--radius', String(radius)]),
...(reviewData ? ['--review-file', path.join(run.out, 'review.json')] : []), ...partIds.flatMap((id) => ['--part', id])]);
const prepared = readJson(path.join(run.out, 'completion-plan.json')); const prepared = readJson(path.join(run.out, 'completion-plan.json'));
result.skipped = prepared.skipped; result.skipped = prepared.skipped;
result.parts = prepared.tasks.map((task) => ({id: task.id, status: 'prepared', upstream_warnings: task.warnings, semantic_verified: false})); result.parts = prepared.tasks.map((task) => ({id: task.id, status: 'prepared', upstream_warnings: task.warnings, semantic_verified: false}));
if (probeEditCapabilities && prepared.tasks.length !== 1) throw new Error('所选部件没有可执行任务;请检查 completion-plan.json,不会发送能力探针');
for (const [index, task] of prepared.tasks.entries()) { for (const [index, task] of prepared.tasks.entries()) {
const folder = path.join(run.out, 'tasks', task.id); const folder = path.join(run.out, 'tasks', task.id);
const prompt = `${stage.prompt}\n\nPart: ${task.id}: ${task.semantic}\nCompletion requirements: ${JSON.stringify(task.completion)}\nCanvas: ${size} x ${size} pixels. Keep the part at its current position and native scale.\nExtend only the hidden contour and requested joint connections, by at most ${radius} pixels from the visible silhouette. Keep all other background pure white or transparent. Do not add the occluder or neighboring body parts.\n${stage.capabilities.multiple_images ? 'The second image is local context only; do not copy its other parts into the output.' : 'Only the isolated visible part is provided.'}\nReturn exactly the same canvas dimensions.\n`; const imageRoles = [{file: 'input.png', role: referenceMode === 'board' ? 'target_with_reference_panels' : 'target'}];
if (['local', 'both'].includes(referenceMode)) imageRoles.push({file: 'context.png', role: 'local_reference'});
if (['full', 'both'].includes(referenceMode)) imageRoles.push({file: 'reference.png', role: 'full_character_reference'});
const images = imageRoles.map(({file}) => readImage(path.join(folder, file), {pngOnly: true}));
const mask = maskSent ? readImage(path.join(folder, 'edit-mask.png'), {pngOnly: true}) : null;
const requestPlan = {model: stage.model, size: requestSize, reference_mode: referenceMode,
images: images.map((im) => im.sha256), image_roles: images.map((im, i) => ({...imageRoles[i], index: i, sha256: im.sha256, width: im.width, height: im.height})),
image_field: stage.imageField, mask_editable: mask ? stage.maskEditable : null,
edit_mask_sent: Boolean(mask), mask_sha256: mask?.sha256 ?? null,
mask_enforcement: mask ? 'provider_requested_and_local' : 'local_only',
capability_status: probeEditCapabilities ? 'unverified_probe' : 'configured_not_proven_by_this_run',
allowed_mask_sha256: sha256(readFileSync(path.join(folder, 'allowed-mask.png')))};
jsonFile(path.join(folder, 'request-plan.json'), {...requestPlan, request_sent: false}, stage.apiKey);
const referenceInstruction = referenceMode === 'board'
? `The LEFT work panel ${JSON.stringify(task.transform.working_xyxy)} is the only output target. RIGHT panels are LOCAL and FULL character references. Keep the entire ${size}x${size} board layout unchanged; never place reference characters in the target panel. Preserve reference panels.`
: `Image order: ${imageRoles.map((im, i) => `${i + 1}=${im.role}`).join(', ')}. Only edit the first image. Subsequent images are references, never extra output parts.`;
const prompt = `${stage.prompt}\n\nPart: ${task.id}: ${task.semantic}\nCompletion requirements: ${JSON.stringify(task.completion)}\nCanvas: ${size} x ${size} pixels. Preserve the target's exact current position and displayed scale.\nTask regions: ${JSON.stringify(task.regions)}. segmentation_gap: restore source texture missing from segmentation; attachment_cleanup: paint the underlying material WITHOUT the overlaid attachment; reviewed_replacement: repair incorrect pixels; joint/occlusion regions: extend this part only.\n${referenceInstruction}\n${mask ? 'Transparent mask pixels are editable; opaque mask pixels must remain unchanged.' : 'No provider mask is sent. Follow the bounded edit instructions; output is constrained locally afterwards.'}\nReturn exactly the same canvas dimensions.\n`;
const fullPrompt = `${prompt}\nOccluded by: ${task.occluded_by.join(', ') || 'not specified'}.\nLocal pixel bounds for proposed extensions (xyxy, exclusive upper bounds): ${JSON.stringify(task.region_bounds_xyxy)}. Null means no supported region for that requirement. These boxes bound the allowed regions; do not fill whole rectangles.\n`; const fullPrompt = `${prompt}\nOccluded by: ${task.occluded_by.join(', ') || 'not specified'}.\nLocal pixel bounds for proposed extensions (xyxy, exclusive upper bounds): ${JSON.stringify(task.region_bounds_xyxy)}. Null means no supported region for that requirement. These boxes bound the allowed regions; do not fill whole rectangles.\n`;
writeFileSync(path.join(folder, 'prompt.txt'), fullPrompt); writeFileSync(path.join(folder, 'prompt.txt'), fullPrompt);
if (prepareOnly) continue; if (prepareOnly) continue;
@@ -111,11 +188,7 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
if (!imported[task.id]) { record.status = 'missing_candidate'; continue; } if (!imported[task.id]) { record.status = 'missing_candidate'; continue; }
outputs = [{bytes: readFileSync(imported[task.id].file), extension: path.extname(imported[task.id].file).slice(1)}]; outputs = [{bytes: readFileSync(imported[task.id].file), extension: path.extname(imported[task.id].file).slice(1)}];
} else { } else {
const images = [readImage(path.join(folder, 'input.png'), {pngOnly: true})]; jsonFile(path.join(folder, 'request.json'), {...requestPlan, request_sent: true}, stage.apiKey);
if (stage.capabilities.multiple_images) images.push(readImage(path.join(folder, 'context.png'), {pngOnly: true}));
const mask = stage.capabilities.mask ? readImage(path.join(folder, 'edit-mask.png'), {pngOnly: true}) : null;
jsonFile(path.join(folder, 'request.json'), {model: stage.model, size: requestSize, images: images.map((im) => im.sha256),
edit_mask_sent: Boolean(mask), allowed_mask_sha256: sha256(readFileSync(path.join(folder, 'allowed-mask.png')))}, stage.apiKey);
manifest.network_requested = true; manifest.network_requested = true;
jsonFile(run.manifestPath, manifest, stage.apiKey); jsonFile(run.manifestPath, manifest, stage.apiKey);
try { try {
@@ -126,6 +199,10 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
} catch (error) { } catch (error) {
record.status = 'request_failed'; record.status = 'request_failed';
if (error.diagnostics) jsonFile(path.join(folder, 'response.error.json'), error.diagnostics, stage.apiKey); if (error.diagnostics) jsonFile(path.join(folder, 'response.error.json'), error.diagnostics, stage.apiKey);
if (probeEditCapabilities) jsonFile(path.join(folder, 'capability-evidence.json'), {
request_accepted: false, request: requestPlan, error: error.message,
reference_use_verified: false, mask_semantics_verified: false,
verdict: 'request_failed_no_capability_promotion'}, stage.apiKey);
throw error; throw error;
} }
} }
@@ -145,6 +222,10 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
record.status = record.selected_file ? 'geometry_passed_needs_review' : 'rejected'; record.status = record.selected_file ? 'geometry_passed_needs_review' : 'rejected';
if (!record.qa) record.qa = {...record.candidates[0]?.qa, accepted: false, if (!record.qa) record.qa = {...record.candidates[0]?.qa, accepted: false,
reasons: [...new Set(record.candidates.flatMap((c) => c.qa.reasons))]}; reasons: [...new Set(record.candidates.flatMap((c) => c.qa.reasons))]};
if (probeEditCapabilities) jsonFile(path.join(folder, 'capability-evidence.json'), {
request_accepted: true, request: requestPlan, candidates: record.candidates.map((c) => c.qa),
reference_use_verified: false, mask_semantics_verified: false,
verdict: 'needs_visual_review_no_automatic_capability_promotion'}, stage.apiKey);
jsonFile(path.join(run.out, 'completion-result.json'), result, stage.apiKey); jsonFile(path.join(run.out, 'completion-result.json'), result, stage.apiKey);
} }
manifest.quality_status = prepareOnly ? 'prepared_needs_review' : 'needs_review'; manifest.quality_status = prepareOnly ? 'prepared_needs_review' : 'needs_review';
+79
View File
@@ -0,0 +1,79 @@
import {execFileSync} from 'node:child_process';
import {readFileSync, readdirSync} from 'node:fs';
import path from 'node:path';
import {projectRoot} from '../lib/config.mjs';
import {selectPython} from '../lib/python-runtime.mjs';
import {validateParts} from '../lib/validate-parts.mjs';
import {sha256} from '../lib/images.mjs';
import {prepareRun, jsonFile, finishRun} from '../lib/run-artifacts.mjs';
const readJson = (file) => JSON.parse(readFileSync(file, 'utf8').replace(/^\uFEFF/u, ''));
const filesIn = (dir, prefix = '') => readdirSync(dir, {withFileTypes: true}).flatMap((entry) => entry.isDirectory()
? filesIn(path.join(dir, entry.name), `${prefix}${entry.name}/`) : [`${prefix}${entry.name}`]);
export async function runSpine(options) {
if (!options.inputDir || (!options.outDir && !options.dryRun)) throw new Error('Spine 阶段需要 --input-dir SAM目录 和独立的 --out-dir');
const input = path.resolve(options.inputDir);
const completion = options.completionDir ? path.resolve(options.completionDir) : undefined;
const plan = readJson(path.join(input, 'semantic-parts.json'));
const validation = validateParts(plan);
if (!validation.valid || plan.schema_version !== 2) throw new Error(`Spine 阶段需要有效 v2 计划:${validation.errors.join(', ')}`);
const fallback = options.fallbackPolicy || 'sam';
const angle = Number(options.checkAngle ?? 10);
if (!['sam', 'error'].includes(fallback)) throw new Error('--fallback-policy 仅支持 sam / error');
if (!(angle > 0 && angle <= 45)) throw new Error('--check-angle 必须在 (0, 45] 度之间');
const python = selectPython(options.python);
const external = path.resolve(options.genielabsDir || path.join(projectRoot, '..', 'spine-animation-ai'));
const scripts = ['make_atlas.py', 'generate_spine_player.py'].map((name) => path.join(external, 'scripts', name));
const codeFiles = ['spine_assembly.py', 'stages/spine-stage.mjs', 'tools/model-check.mjs',
'lib/python-runtime.mjs', 'lib/run-artifacts.mjs', 'lib/images.mjs', 'lib/validate-parts.mjs',
'schemas/semantic-parts.schema.json'].map((name) => path.join(projectRoot, name));
const codeHash = sha256([...codeFiles, ...scripts].map((file) => readFileSync(file)).map((bytes) => sha256(bytes)).join('\n'));
function call(script, args) {
try {
return execFileSync(python, [script, ...args], {cwd: projectRoot, windowsHide: true,
encoding: 'utf8', timeout: 120000, maxBuffer: 8 * 1024 * 1024,
env: {...process.env, PYTHONUTF8: '1'}, stdio: ['ignore', 'pipe', 'pipe']});
} catch (error) { throw new Error(`Spine 离线处理失败:${error.stderr || error.message}`); }
}
const worker = path.join(projectRoot, 'spine_assembly.py');
const args = ['--input-dir', input, '--fallback-policy', fallback, '--check-angle', String(angle)];
if (completion) args.push('--completion-dir', completion);
if (options.drawOrderFile) args.push('--draw-order-file', path.resolve(options.drawOrderFile));
// Inspect all masks, selected PNGs, crop coordinates and input identities before creating output.
const inspected = JSON.parse(call(worker, args));
if (options.dryRun) {
const result = {dryRun: true, stage: 'spine', ...inspected, network_requested: false};
console.log(JSON.stringify(result, null, 2));
return result;
}
const identity = {schema_version: 1, stage: 'spine', spine_version: '4.2.0',
inputs: inspected.input_hashes, code_sha256: codeHash, fallback_policy: fallback, check_angle: angle};
const fingerprint = sha256(JSON.stringify(identity));
const run = prepareRun(options.outDir, fingerprint, options.resume);
if (run.reused) return {out: run.out, reused: true};
const manifest = {...identity, fingerprint, status: 'running', quality_status: 'needs_review',
network_requested: false, started_at: new Date().toISOString(),
external_dependency: {name: 'spine-animation-ai', license: 'PolyForm-Noncommercial-1.0.0'},
limitations: ['region attachments only; no mesh, weights or IK',
'setup and per-bone check animations only; semantic and motion quality need review',
'preview embeds assets but loads Spine Player 4.2 from CDN']};
jsonFile(run.manifestPath, manifest);
try {
const result = JSON.parse(call(worker, [...args, '--output', run.out]));
if (JSON.stringify(result.input_hashes) !== JSON.stringify(inspected.input_hashes)) throw new Error('处理期间输入发生变化,请使用新输出目录重跑');
call(scripts[0], ['--parts', path.join(run.out, 'images'), '--output', path.join(run.out, 'atlas'), '--name', 'skeleton']);
call(scripts[1], ['--skeleton', path.join(run.out, 'skeleton.json'),
'--atlas', path.join(run.out, 'atlas', 'skeleton.atlas'), '--atlas-image', path.join(run.out, 'atlas', 'skeleton.png'),
'--output', path.join(run.out, 'preview.html'), '--animation', 'setup', '--title', 'Spine Assembly Review']);
Object.assign(manifest, {part_count: result.parts, bone_count: result.bones,
completion_part_count: result.completion_parts, fallback_part_count: result.fallback_parts,
completed_at: new Date().toISOString()});
finishRun(run, manifest, filesIn(run.out).filter((name) => name !== 'manifest.json'));
return {out: run.out, reused: false, ...result};
} catch (error) {
manifest.status = 'failed'; manifest.error = error.message;
jsonFile(run.manifestPath, manifest);
throw error;
}
}
+63 -4
View File
@@ -31,7 +31,7 @@ test('completion geometry rejects moved/empty outputs and preserves original alp
test('part completion prepares without HTTP, sends declared capabilities, merges, resumes, and detects changed masks', async (t) => { test('part completion prepares without HTTP, sends declared capabilities, merges, resumes, and detects changed masks', async (t) => {
const {root, input, stage} = fixture(t); const {root, input, stage} = fixture(t);
const opts = {stage, inputDir: input, outDir: path.join(root, 'run'), completionSize: 128, extensionRadius: 16, partIds: ['torso']}; const opts = {stage, inputDir: input, outDir: path.join(root, 'run'), completionProfile: 'legacy', completionSize: 128, extensionRadius: 16, partIds: ['torso']};
let requests = 0; let requests = 0;
const transport = {fetchImpl: async (_url, request) => { const transport = {fetchImpl: async (_url, request) => {
requests++; requests++;
@@ -66,7 +66,7 @@ test('offline candidate import needs no HTTP; mismatched candidate is retained a
const candidates = path.join(root, 'candidates'); mkdirSync(candidates); const candidates = path.join(root, 'candidates'); mkdirSync(candidates);
writeFileSync(path.join(candidates, 'torso.png'), readFileSync(path.join(input, 'source.png'))); writeFileSync(path.join(candidates, 'torso.png'), readFileSync(path.join(input, 'source.png')));
const outDir = path.join(root, 'run'); const outDir = path.join(root, 'run');
const result = await runCompletion({stage, inputDir: input, outDir, candidatesDir: candidates, completionSize: 128}, {fetchImpl: () => { throw new Error('Must remain offline'); }}); const result = await runCompletion({stage, inputDir: input, outDir, candidatesDir: candidates, completionProfile: 'legacy', completionSize: 128}, {fetchImpl: () => { throw new Error('Must remain offline'); }});
assert.equal(result.rejected, 1); assert.equal(result.rejected, 1);
assert.equal(result.accepted, 0); assert.equal(result.accepted, 0);
const data = JSON.parse(readFileSync(path.join(outDir, 'completion-result.json'))); const data = JSON.parse(readFileSync(path.join(outDir, 'completion-result.json')));
@@ -81,7 +81,7 @@ test('provider mask/context capabilities are explicit and failure preserves diag
stage.capabilities = {mask: true, multiple_images: true}; stage.capabilities = {mask: true, multiple_images: true};
stage.maxRetries = 0; stage.maxRetries = 0;
const outDir = path.join(root, 'run'); const outDir = path.join(root, 'run');
await assert.rejects(runCompletion({stage, inputDir: input, outDir, completionSize: 128}, {fetchImpl: async (_url, request) => { await assert.rejects(runCompletion({stage, inputDir: input, outDir, completionProfile: 'legacy', completionSize: 128}, {fetchImpl: async (_url, request) => {
assert.equal(request.body.getAll('image').length, 2); assert.equal(request.body.getAll('image').length, 2);
assert.ok(request.body.get('mask')); assert.ok(request.body.get('mask'));
return new Response(JSON.stringify({error: {message: 'test-secret failed'}}), {status: 400}); return new Response(JSON.stringify({error: {message: 'test-secret failed'}}), {status: 400});
@@ -98,7 +98,7 @@ test('completion CLI prepares without credentials and rejects mixed probe argume
const envFile = path.join(root, 'empty.env'); writeFileSync(envFile, ''); const envFile = path.join(root, 'empty.env'); writeFileSync(envFile, '');
const outDir = path.join(root, 'cli'); const outDir = path.join(root, 'cli');
const args = ['tools/model-check.mjs', '--stage', 'completion', '--input-dir', input, const args = ['tools/model-check.mjs', '--stage', 'completion', '--input-dir', input,
'--out-dir', outDir, '--prepare-only', '--part', 'torso', '--completion-size', '128', '--env-file', envFile]; '--out-dir', outDir, '--prepare-only', '--part', 'torso', '--completion-profile', 'legacy', '--completion-size', '128', '--env-file', envFile];
const options = {cwd: projectRoot, windowsHide: true, encoding: 'utf8', const options = {cwd: projectRoot, windowsHide: true, encoding: 'utf8',
env: {...process.env, VECTOR_ENGINE_BASE_URL: '', VECTOR_ENGINE_API_KEY: ''}, stdio: ['ignore', 'pipe', 'pipe']}; env: {...process.env, VECTOR_ENGINE_BASE_URL: '', VECTOR_ENGINE_API_KEY: ''}, stdio: ['ignore', 'pipe', 'pipe']};
assert.match(execFileSync(process.execPath, args, options), /未请求模型/u); assert.match(execFileSync(process.execPath, args, options), /未请求模型/u);
@@ -107,3 +107,62 @@ test('completion CLI prepares without credentials and rejects mixed probe argume
assert.equal(manifest.prepared_count, 1); assert.equal(manifest.prepared_count, 1);
assert.throws(() => execFileSync(process.execPath, [...args, '--image', path.join(input, 'source.png')], options), (error) => /不接受探针输入/u.test(error.stderr)); assert.throws(() => execFileSync(process.execPath, [...args, '--image', path.join(input, 'source.png')], options), (error) => /不接受探针输入/u.test(error.stderr));
}); });
test('adaptive geometry: reversible scaling, real-mask replanning, cleanup, review masks and Spine contract', () => {
assert.equal(execFileSync(python, ['-m', 'unittest', 'discover', '-s', 'tests', '-p', 'test_completion_adaptive.py'],
{cwd: projectRoot, windowsHide: true, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe']}), '');
});
function adaptiveResponse(folder) {
execFileSync(python, ['-c', 'import sys; sys.path.insert(0, "tests"); from test_completion_adaptive import mock_candidate; mock_candidate(sys.argv[1])', folder],
{cwd: projectRoot, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe']});
return new Response(JSON.stringify({data: [{b64_json: readFileSync(path.join(folder, 'test-candidate.png')).toString('base64')}]}),
{headers: {'Content-Type': 'application/json'}});
}
test('adaptive default sends a single reference board and keeps native coordinates; planning changes invalidate candidates', async (t) => {
const {root, input, stage} = fixture(t);
const outDir = path.join(root, 'adaptive');
const opts = {stage, inputDir: input, outDir, partIds: ['torso'], completionSize: 256, extensionRadius: 16};
const result = await runCompletion(opts, {fetchImpl: async (_url, request) => {
assert.equal(request.body.getAll('image').length, 1);
assert.equal(request.body.has('mask'), false);
assert.match(request.body.get('prompt'), /RIGHT panels/u);
return adaptiveResponse(path.join(outDir, 'tasks/torso'));
}});
assert.equal(result.accepted, 1);
const record = JSON.parse(readFileSync(path.join(outDir, 'completion-result.json'))).parts[0];
assert.deepEqual(record.crop_xyxy, [20, 20, 50, 40]);
assert.ok(record.preservation.preserve_mask);
const request = JSON.parse(readFileSync(path.join(outDir, 'tasks/torso/request.json')));
assert.equal(request.reference_mode, 'board');
assert.equal(request.image_roles[0].role, 'target_with_reference_panels');
assert.equal(request.mask_enforcement, 'local_only');
assert.equal((await runCompletion({...opts, resume: true})).reused, true);
const recovered = await runCompletion({...opts, outDir: path.join(root, 'adaptive-recheck'), candidatesDir: outDir});
assert.equal(recovered.accepted, 1);
await assert.rejects(runCompletion({...opts, outDir: path.join(root, 'bad-recheck'), candidatesDir: outDir, completionReference: 'none'}), /画布/u);
});
test('native full/local references and mask preserve order and hashes; probe never promotes capabilities', async (t) => {
const {root, input, stage} = fixture(t);
const opts = {stage, inputDir: input, outDir: path.join(root, 'probe'), partIds: ['torso'], completionSize: 256};
await assert.rejects(runCompletion({...opts, completionReference: 'both', completionMask: 'required'}), /未确认/u);
await assert.rejects(runCompletion({...opts, partIds: [], probeEditCapabilities: true}), /恰好一个/u);
await runCompletion({...opts, probeEditCapabilities: true}, {fetchImpl: async (_url, request) => {
const images = request.body.getAll('image');
assert.deepEqual(images.map((im) => im.name), ['input.png', 'context.png', 'reference.png']);
assert.equal(request.body.get('mask').name, 'edit-mask.png');
assert.equal(request.body.get('n'), '1');
return adaptiveResponse(path.join(opts.outDir, 'tasks/torso'));
}});
assert.deepEqual(stage.capabilities, {multiple_images: false, mask: false});
const evidence = JSON.parse(readFileSync(path.join(opts.outDir, 'tasks/torso/capability-evidence.json')));
assert.equal(evidence.request_accepted, true);
assert.equal(evidence.mask_semantics_verified, false);
assert.equal(evidence.reference_use_verified, false);
assert.equal(evidence.request.image_roles.length, 3);
assert.ok(evidence.request.mask_sha256);
const qa = JSON.parse(readFileSync(path.join(opts.outDir, 'tasks/torso/qa-1.json')));
assert.equal(typeof qa.raw_outside_edit_changed_pixels, 'number');
});
+79
View File
@@ -0,0 +1,79 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import {execFileSync} from 'node:child_process';
import {mkdtempSync, readFileSync, writeFileSync, realpathSync, rmSync, existsSync} from 'node:fs';
import path from 'node:path';
import {tmpdir} from 'node:os';
import {projectRoot} from '../lib/config.mjs';
import {selectPython} from '../lib/python-runtime.mjs';
import {runSpine} from '../stages/spine-stage.mjs';
const python = selectPython();
const execution = {cwd: projectRoot, windowsHide: true, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe']};
function fixture(t) {
const base = realpathSync(tmpdir());
const root = mkdtempSync(path.join(base, 'spine-export-'));
t.after(() => {
const resolved = realpathSync(root);
assert.equal(path.dirname(resolved), base);
assert.ok(path.basename(resolved).startsWith('spine-export-'));
rmSync(resolved, {recursive: true});
});
execFileSync(python, ['-c', 'import sys; sys.path.insert(0, "tests"); from test_spine_assembly import make_fixture; make_fixture(sys.argv[1])', root], execution);
return {root, inputDir: path.join(root, 'sam'), completionDir: path.join(root, 'complete'), outDir: path.join(root, 'out')};
}
test('Spine assembly preserves coordinates, hidden hierarchy, bindings and validates asset identity', () => {
assert.equal(execFileSync(python, ['-m', 'unittest', 'discover', '-s', 'tests', '-p', 'test_spine_assembly.py'], execution), '');
});
test('Spine exports complete assets, atlas, player, diagnostics and verifies resume hashes', async (t) => {
const opts = fixture(t);
const result = await runSpine(opts);
assert.equal(result.parts, 4);
assert.equal(result.bones, 4);
for (const file of ['assembly.json', 'skeleton.json', 'atlas/skeleton.atlas', 'atlas/skeleton.png',
'images/shell.png', 'assembled.png', 'comparison.png', 'coverage-diagnostic.png', 'report.html', 'preview.html']) {
assert.ok(existsSync(path.join(opts.outDir, file)), file);
}
const manifest = JSON.parse(readFileSync(path.join(opts.outDir, 'manifest.json')));
assert.equal(manifest.status, 'completed');
assert.equal(manifest.quality_status, 'needs_review');
assert.equal(manifest.network_requested, false);
assert.equal(manifest.fallback_part_count, 1);
const skeleton = JSON.parse(readFileSync(path.join(opts.outDir, 'skeleton.json')));
assert.equal(skeleton.skeleton.images, './images/');
const atlas = readFileSync(path.join(opts.outDir, 'atlas/skeleton.atlas'), 'utf8').replaceAll('\r\n', '\n');
for (const slot of skeleton.slots) assert.ok(atlas.includes(`\n${slot.attachment}\n`));
assert.match(readFileSync(path.join(opts.outDir, 'preview.html'), 'utf8'), /animation: "setup"/u);
assert.equal((await runSpine({...opts, resume: true})).reused, true);
await assert.rejects(runSpine(opts), /非空/u);
await assert.rejects(runSpine({...opts, resume: true, checkAngle: 5}), /变化/u);
writeFileSync(path.join(opts.outDir, 'images/shell.png'), 'tampered');
await assert.rejects(runSpine({...opts, resume: true}), /产物缺失或被修改/u);
});
test('Spine CLI is independent of model config and dry-run writes nothing', (t) => {
const opts = fixture(t);
const args = ['tools/model-check.mjs', '--stage', 'spine', '--input-dir', opts.inputDir,
'--completion-dir', opts.completionDir, '--out-dir', opts.outDir];
const env = {...process.env, VECTOR_ENGINE_API_KEY: '', TIANTOKEN_API_KEY: ''};
assert.match(execFileSync(process.execPath, [...args, '--dry-run'], {...execution, env}), /"network_requested": false/u);
assert.equal(existsSync(opts.outDir), false);
assert.throws(() => execFileSync(process.execPath, [...args, '--config', 'missing-config.json'], execution),
(error) => /独立离线阶段/u.test(error.stderr));
assert.throws(() => execFileSync(process.execPath, [...args, '--fallback-policy', 'error'], execution),
(error) => /Completion required/u.test(error.stderr));
assert.equal(existsSync(opts.outDir), false);
assert.match(execFileSync(process.execPath, args, {...execution, env}), /导出完成/u);
});
test('Spine rejects malformed semantic hierarchy before export', async (t) => {
const opts = fixture(t);
const file = path.join(opts.inputDir, 'semantic-parts.json');
const plan = JSON.parse(readFileSync(file));
plan.parts.find((p) => p.id === 'hub').parent = 'tip_q';
writeFileSync(file, JSON.stringify(plan));
await assert.rejects(runSpine(opts), /存在环/u);
assert.equal(existsSync(opts.outDir), false);
});
+205
View File
@@ -0,0 +1,205 @@
import copy
import json
from pathlib import Path
import tempfile
import unittest
import numpy as np
from PIL import Image
from completion_geometry import localize, merge_candidate, finalize
from completion_planning import prepare_adaptive, read, write, review_inputs
from spine_assembly import load_assets
from test_completion import make_fixture
def candidate_for(folder, native):
"""Independent fixture: insert known native ground truth into documented work rectangle."""
folder = Path(folder)
task = read(folder / 'task.json')
rect = task['transform']['working_xyxy']
canvas = Image.open(folder / 'input.png').convert('RGBA')
content = Image.fromarray(native).resize((rect[2]-rect[0], rect[3]-rect[1]), Image.Resampling.NEAREST)
canvas.paste(content, (rect[0], rect[1]))
target = folder / 'test-candidate.png'
canvas.save(target)
return target
def mock_candidate(folder):
folder = Path(folder)
native = np.array(Image.open(folder / 'native-visible.png'))
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
native[edit] = [180, 50, 20, 255]
return candidate_for(folder, native)
class AdaptiveTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.source = make_fixture(self.root)
def task(self, mode='board', size=256, review=None):
plan = prepare_adaptive(self.root, ['torso'], size, 16, mode, review)
return plan['tasks'][0], self.root / 'tasks' / 'torso'
def test_small_part_scales_up_roundtrip_and_reference_panels_are_excluded(self):
task, folder = self.task()
self.assertGreater(task['transform']['scale'], 1)
self.assertEqual(len(task['reference_panel_boxes']), 2)
forward = task['transform']['source_to_canvas']
inverse = task['transform']['canvas_to_source']
for x, y in [(20, 20), (49, 39), (0, 79)]:
mx, my = forward[0]*x+forward[2], forward[4]*y+forward[5]
self.assertAlmostEqual(inverse[0]*mx+inverse[2], x)
self.assertAlmostEqual(inverse[4]*my+inverse[5], y)
native = np.array(Image.open(folder / 'native-visible.png'))
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
native[edit] = [180, 50, 20, 255]
output = folder / 'merged.png'
qa = merge_candidate(folder, candidate_for(folder, native), output)
self.assertTrue(qa['accepted'], qa)
keep = np.array(Image.open(folder / 'native-keep.png')) > 0
np.testing.assert_array_equal(np.array(Image.open(output))[keep], native[keep])
write(self.root / 'completion-result.json', {'parts': [{'id': 'torso', 'status': 'geometry_passed_needs_review',
'selected_file': 'tasks/torso/merged.png', 'qa': qa}], 'skipped': []})
finalize(self.root)
result = read(self.root / 'completion-result.json')['parts'][0]
self.assertEqual(result['crop_xyxy'], [20, 20, 50, 40])
self.assertIn('preservation', result)
self.assertTrue((folder / 'reference.png').exists())
mask = np.array(Image.open(folder / 'edit-mask.png'))
for x0, y0, x1, y1 in task['reference_panel_boxes']:
self.assertTrue(np.all(mask[y0:y1, x0:x1, 3] == 255))
def test_actual_hole_creates_task_despite_false_semantic_flag(self):
plan = read(self.root / 'semantic-parts.json')
plan['parts'][0].update(needs_completion=False, visibility='visible', occluded_by=[],
completion={k: False for k in plan['parts'][0]['completion']})
write(self.root / 'semantic-parts.json', plan)
mask = np.array(Image.open(self.root / 'masks/torso.png'))
mask[26:30, 26:30] = 0
Image.fromarray(mask).save(self.root / 'masks/torso.png')
task, folder = self.task()
self.assertEqual(task['regions']['segmentation_gap'], 16)
native = np.array(Image.open(folder / 'native-visible.png'))
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
native[edit] = [200, 40, 10, 255]
output = folder / 'merged.png'
qa = merge_candidate(folder, candidate_for(folder, native), output)
self.assertTrue(qa['accepted'], qa)
# Repair restores source texture, even though the model proposed a different colour.
gap = np.array(Image.open(folder / 'native-segmentation_gap.png')) > 0
source = np.array(Image.open(folder / 'native-source.png'))
np.testing.assert_array_equal(np.array(Image.open(output))[gap], source[gap])
def test_attachment_cleanup_releases_only_declared_overlay_and_spine_accepts_contract(self):
# Move collar inside torso to represent an overlaid detail needing a clean base.
mask = np.zeros((80, 80), np.uint8)
mask[26:30, 26:30] = 255
Image.fromarray(mask).save(self.root / 'masks/collar.png')
task, folder = self.task('both')
self.assertEqual(task['regions']['attachment_cleanup'], 16)
native = np.array(Image.open(folder / 'native-visible.png'))
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
native[edit] = [20, 100, 200, 255]
output = folder / 'merged.png'
qa = merge_candidate(folder, candidate_for(folder, native), output)
self.assertTrue(qa['accepted'], qa)
write(self.root / 'completion-result.json', {'parts': [{'id': 'torso', 'status': 'geometry_passed_needs_review',
'selected_file': 'tasks/torso/merged.png', 'qa': qa}], 'skipped': []})
finalize(self.root)
data = load_assets(self.root, self.root)
self.assertEqual(data[2]['parts'][0]['source'], 'completion')
# Contract tampering cannot turn a protected pixel into an untracked edit.
result = read(self.root / 'completion-result.json')['parts'][0]
image = np.array(Image.open(self.root / result['image']))
image[0, 0] = [255, 0, 0, 255]
Image.fromarray(image).save(self.root / result['image'])
with self.assertRaisesRegex(ValueError, 'preserve original visible'):
load_assets(self.root, self.root)
def test_review_replacement_removal_and_invalid_partition(self):
replace = np.zeros((80, 80), np.uint8)
replace[26:29, 26:29] = 255
remove = np.zeros_like(replace)
remove[35:40, 35:40] = 255
Image.fromarray(replace).save(self.root / 'replace.png')
Image.fromarray(remove).save(self.root / 'remove.png')
review = self.root / 'review-input.json'
write(review, {'schema_version': 1, 'parts': {'torso': {'confirmed': True,
'replace_mask': 'replace.png', 'remove_mask': 'remove.png', 'extension_radius': 8}}})
task, folder = self.task(review=review)
self.assertEqual(task['extension_radius'], 8)
native = np.array(Image.open(folder / 'native-visible.png'))
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
native[edit] = [180, 50, 20, 255]
qa = merge_candidate(folder, candidate_for(folder, native), folder / 'merged.png')
self.assertTrue(qa['accepted'], qa)
deleted = np.array(Image.open(folder / 'native-remove.png')) > 0
self.assertTrue(np.all(np.array(Image.open(folder / 'merged.png'))[deleted] == 0))
Image.fromarray(remove).save(self.root / 'replace.png')
with self.assertRaisesRegex(ValueError, 'disjoint'):
self.task(review=review)
def test_large_part_downscales_and_invalid_review_cannot_escape(self):
# Upsize the entire source and masks to exceed request dimensions.
for file in [self.root / 'source.png', *list((self.root / 'masks').glob('*.png'))]:
im = Image.open(file)
im.resize((800, 800), Image.Resampling.NEAREST).save(file)
task, folder = self.task(size=128)
self.assertLess(task['transform']['scale'], 1)
native = np.array(Image.open(folder / 'native-visible.png'))
native[np.array(Image.open(folder / 'native-allowed.png')) > 0] = [180, 50, 20, 255]
output = folder / 'merged.png'
qa = merge_candidate(folder, candidate_for(folder, native), output)
self.assertTrue(qa['accepted'], qa)
keep = np.array(Image.open(folder / 'native-keep.png')) > 0
np.testing.assert_array_equal(np.array(Image.open(output))[keep], native[keep])
review = self.root / 'bad-review.json'
write(review, {'schema_version': 1, 'parts': {'torso': {'confirmed': True, 'replace_mask': '../escape.png'}}})
with self.assertRaisesRegex(ValueError, 'escapes'):
review_inputs(review, ['torso'], (800, 800))
def test_adaptive_rejects_movement_and_missing_cleanup(self):
task, folder = self.task()
candidate = mock_candidate(folder)
image = np.array(Image.open(candidate))
Image.fromarray(np.roll(image, 30, axis=1)).save(candidate)
qa = merge_candidate(folder, candidate, folder / 'bad.png')
self.assertFalse(qa['accepted'])
self.assertIn('VISIBLE_ALIGNMENT_MISMATCH', qa['reasons'])
mask = np.zeros((80, 80), np.uint8)
mask[26:30, 26:30] = 255
Image.fromarray(mask).save(self.root / 'masks/collar.png')
_, folder = self.task('both')
no_fill = np.array(Image.open(folder / 'native-visible.png'))
qa = merge_candidate(folder, candidate_for(folder, no_fill), folder / 'bad-cleanup.png')
self.assertFalse(qa['accepted'])
self.assertIn('INCOMPLETE_ATTACHMENT_CLEANUP', qa['reasons'])
def test_duplicate_masks_blocked_until_reviewed(self):
plan = read(self.root / 'semantic-parts.json')
plan['parts'][0]['occluded_by'] = []
write(self.root / 'semantic-parts.json', plan)
Image.open(self.root / 'masks/torso.png').save(self.root / 'masks/collar.png')
result = prepare_adaptive(self.root, ['torso'])
self.assertEqual(result['skipped'][0]['reason'], 'AMBIGUOUS_MASK_REQUIRES_REVIEW')
def test_spine_strict_policy_respects_rebuilt_need_even_if_original_flag_was_false(self):
plan = read(self.root / 'semantic-parts.json')
plan['parts'][0].update(needs_completion=False, visibility='visible', occluded_by=[],
completion={k: False for k in plan['parts'][0]['completion']})
write(self.root / 'semantic-parts.json', plan)
write(self.root / 'completion-result.json', {'schema_version': 2, 'profile': 'adaptive', 'parts': [
{'id': 'torso', 'status': 'rejected', 'qa': {'accepted': False}}], 'skipped': []})
with self.assertRaisesRegex(ValueError, 'Completion required for torso'):
load_assets(self.root, self.root, fallback_policy='error')
data = load_assets(self.root, self.root)
self.assertEqual(data[2]['parts'][0]['reason'], 'COMPLETION_REJECTED')
if __name__ == '__main__':
unittest.main()
+180
View File
@@ -0,0 +1,180 @@
import copy
import json
from pathlib import Path
import shutil
import tempfile
import unittest
import numpy as np
from PIL import Image
from spine_assembly import load_assets, build_skeleton, export_assets, write_json
def make_fixture(directory):
"""An asymmetric mechanical fixture; no humanoid names or assumed part count."""
root = Path(directory)
sam, completion = root / 'sam', root / 'complete'
(sam / 'masks').mkdir(parents=True)
source = np.zeros((48, 64, 4), np.uint8)
specs = [('shell', [16, 12, 30, 24], [40, 70, 130, 255]),
('link_z', [30, 22, 40, 32], [160, 70, 10, 255]),
('tip_q', [40, 28, 45, 36], [80, 170, 30, 255]),
('decal', [19, 15, 22, 18], [200, 50, 130, 255])]
for _, (x0, y0, x1, y1), color in specs:
source[y0:y1, x0:x1] = color
source[12, 16, 3] = 90
source[25, 33] = [230, 140, 40, 255]
Image.fromarray(source).save(sam / 'source.png')
def part(name, parent, role, pivot, box, hidden=False, needs=False, occluded=None):
return {'id': name, 'semantic': name, 'parent': parent, 'rig_role': role,
'bind_to': parent if role != 'bone_part' else None,
'pivot_norm': [pivot[0] / 64, pivot[1] / 48],
'visibility': 'hidden' if hidden else 'visible',
'bbox_norm_xyxy': None if hidden else [box[0]/64, box[1]/48, box[2]/64, box[3]/48],
'positive_points_norm': [], 'negative_points_norm': [],
'occluded_by': occluded or [], 'confidence': .9,
'needs_completion': hidden or needs,
'completion': {'occlusion_fill': hidden, 'joint_extension_parent': needs,
'joint_extension_child': False}}
parts = [part('tip_q', 'link_z', 'bone_part', [40, 29], specs[2][1], needs=True),
part('decal', 'hub', 'attachment', [19, 15], specs[3][1]),
part('shell', 'hub', 'attachment', [20, 16], specs[0][1], occluded=['decal']),
part('link_z', 'hub', 'bone_part', [31, 23], specs[1][1], needs=True),
part('hub', None, 'bone_part', [21, 17], None, hidden=True, occluded=['shell'])]
write_json(sam / 'semantic-parts.json', {'schema_version': 2, 'coordinate_space': 'normalized_xy_top_left',
'parts': parts, 'notes': []})
records = []
for name, (x0, y0, x1, y1), _ in specs:
mask = np.zeros(source.shape[:2], np.uint8)
mask[y0:y1, x0:x1] = 255
Image.fromarray(mask).save(sam / 'masks' / f'{name}.png')
records.append({'id': name, 'mask': f'masks/{name}.png', 'crop_xyxy': [x0, y0, x1, y1]})
write_json(sam / 'segmentation.json', {'schema_version': 1, 'parts': records})
shutil.copytree(sam, completion)
(completion / 'parts').mkdir()
# Add pixels to the left of link_z; the original visible pixels are exact.
candidate = np.zeros((10, 14, 4), np.uint8)
candidate[:, :4] = [160, 70, 10, 255]
candidate[:, 4:] = source[22:32, 30:40]
Image.fromarray(candidate).save(completion / 'parts' / 'link_z.png')
write_json(completion / 'completion-result.json', {'parts': [
{'id': 'link_z', 'status': 'geometry_passed_needs_review', 'selected_file': 'tasks/link_z/merged-1.png',
'qa': {'accepted': True}, 'image': 'parts/link_z.png', 'crop_xyxy': [26, 22, 40, 32]},
{'id': 'tip_q', 'status': 'rejected', 'qa': {'accepted': False, 'reasons': ['VISIBLE_ALIGNMENT_MISMATCH']}}
], 'skipped': [{'id': 'hub', 'reason': 'NO_VISIBLE_SAM_MASK'},
{'id': 'shell', 'reason': 'NO_COMPLETION_NEEDED'},
{'id': 'decal', 'reason': 'NO_COMPLETION_NEEDED'}]})
class AssemblyTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
make_fixture(self.root)
self.sam, self.complete = self.root / 'sam', self.root / 'complete'
def load(self, **kwargs):
return load_assets(self.sam, self.complete, **kwargs)
def result(self):
return json.loads((self.complete / 'completion-result.json').read_text())
def test_pose_coordinates_hidden_hierarchy_and_attachment_binding(self):
data = self.load()
result = export_assets(data, self.root / 'out')
self.assertEqual((result['parts'], result['bones'], result['completion_parts'], result['fallback_parts']), (4, 4, 1, 1))
skeleton = build_skeleton(data[1], data[2])
self.assertEqual([b['name'] for b in skeleton['bones']], ['root', 'hub', 'link_z', 'tip_q'])
bones = {b['name']: b for b in skeleton['bones']}
self.assertAlmostEqual(bones['hub']['x'], -11)
self.assertAlmostEqual(bones['hub']['y'], 31)
self.assertAlmostEqual(bones['link_z']['x'], 10)
self.assertAlmostEqual(bones['link_z']['y'], -6)
self.assertAlmostEqual(bones['tip_q']['x'], 9)
self.assertAlmostEqual(bones['tip_q']['y'], -6)
slots = {s['name']: s for s in skeleton['slots']}
self.assertEqual(slots['shell']['bone'], 'hub')
self.assertNotIn('hub', slots)
self.assertLess(data[2]['draw_order'].index('shell'), data[2]['draw_order'].index('decal'))
# Independent forward kinematics reconstructs each crop centre exactly.
world = {'root': (0, 0)}
for bone in skeleton['bones'][1:]:
px, py = world[bone['parent']]
world[bone['name']] = px + bone['x'], py + bone['y']
for name, (_, box) in data[3].items():
att = skeleton['skins'][0]['attachments'][name][name]
bx, by = world[slots[name]['bone']]
self.assertAlmostEqual(att['x'] + bx + 32, (box[0] + box[2]) / 2)
self.assertAlmostEqual(48 - att['y'] - by, (box[1] + box[3]) / 2)
image = np.array(Image.open(self.root / 'out' / 'assembled.png'))
np.testing.assert_array_equal(image[12, 16], [40, 70, 130, 90])
self.assertEqual(skeleton['animations']['check__link_z']['bones']['link_z']['rotate'][1]['value'], 10)
def test_rejection_fallback_and_strict_policy(self):
data = self.load()
assets = {p['id']: p for p in data[2]['parts']}
self.assertEqual(assets['tip_q']['source'], 'sam')
self.assertEqual(assets['tip_q']['reason'], 'COMPLETION_REJECTED')
with self.assertRaisesRegex(ValueError, 'Completion required for tip_q'):
self.load(fallback_policy='error')
only_sam = load_assets(self.sam)
self.assertTrue(all(p['source'] in ['sam', 'hidden'] for p in only_sam[2]['parts']))
def test_mixed_runs_and_changed_masks_rejected(self):
mask = self.complete / 'masks' / 'tip_q.png'
Image.new('L', (64, 48), 255).save(mask)
with self.assertRaisesRegex(ValueError, 'Completion mask mismatch'):
self.load()
shutil.copy2(self.sam / 'masks' / 'tip_q.png', mask)
(self.complete / 'semantic-parts.json').write_text('{}')
with self.assertRaisesRegex(ValueError, 'Completion input mismatch'):
self.load()
def test_bad_crop_alignment_path_and_missing_visible_parts_rejected(self):
original = self.result()
for box in [[26, 22, 39, 32], [-1, 22, 13, 32], [27, 22, 41, 32]]:
result = copy.deepcopy(original)
result['parts'][0]['crop_xyxy'] = box
write_json(self.complete / 'completion-result.json', result)
with self.assertRaises(ValueError):
self.load()
result = copy.deepcopy(original)
result['parts'][0]['image'] = '../sam/source.png'
write_json(self.complete / 'completion-result.json', result)
with self.assertRaisesRegex(ValueError, 'outside input'):
self.load()
seg = json.loads((self.sam / 'segmentation.json').read_text())
seg['parts'] = seg['parts'][1:]
write_json(self.sam / 'segmentation.json', seg)
with self.assertRaisesRegex(ValueError, 'Visible part has no SAM mask'):
load_assets(self.sam)
def test_draw_order_override_and_unconstrained_overlap(self):
data = self.load()
self.assertTrue(data[2]['unconstrained_overlaps'])
override = self.root / 'order.json'
order = ['decal', 'shell', 'link_z', 'tip_q']
write_json(override, {'draw_order': order})
modified = self.load(draw_order_file=override)[2]
self.assertEqual(modified['draw_order'], order)
self.assertIn(['shell', 'decal'], modified['overridden_occlusion_constraints'])
write_json(override, {'draw_order': ['shell'] * 4})
with self.assertRaisesRegex(ValueError, 'exactly once'):
self.load(draw_order_file=override)
def test_attachment_only_and_root_binding(self):
plan = {'parts': [{'id': 'panel', 'rig_role': 'attachment', 'bind_to': None}]}
assembly = {'width': 64, 'height': 48, 'draw_order': ['panel'],
'parts': [{'id': 'panel', 'crop_xyxy': [1, 2, 10, 12]}]}
skeleton = build_skeleton(plan, assembly)
self.assertEqual(skeleton['bones'], [{'name': 'root'}])
self.assertEqual(skeleton['slots'][0]['bone'], 'root')
self.assertEqual(skeleton['animations'], {'setup': {}})
if __name__ == '__main__':
unittest.main()
+35 -5
View File
@@ -7,12 +7,14 @@ import {runUnderstanding, runCompletionProbe} from '../stages/model-stage.mjs';
import {readImage} from '../lib/images.mjs'; import {readImage} from '../lib/images.mjs';
import {runSegmentation} from '../stages/segmentation-stage.mjs'; import {runSegmentation} from '../stages/segmentation-stage.mjs';
import {runCompletion} from '../stages/completion-stage.mjs'; import {runCompletion} from '../stages/completion-stage.mjs';
import {runSpine} from '../stages/spine-stage.mjs';
const help = `模型接入与结构理解(新增模式,不调用旧拆件流程) const help = `模型接入与结构理解(新增模式,不调用旧拆件流程)
node pipeline.mjs --mode annotated-parts --reference image.png --out-dir output/run --stop-after understand node pipeline.mjs --mode annotated-parts --reference image.png --out-dir output/run --stop-after understand
node tools/model-check.mjs --stage completion --image local.png --prompt-file task.txt --out-dir output/edit node tools/model-check.mjs --stage completion --image local.png --prompt-file task.txt --out-dir output/edit
node tools/model-check.mjs --stage segmentation --input-dir output/understand --out-dir output/segment --sam-dir F:/Study/SAM3 node tools/model-check.mjs --stage segmentation --input-dir output/understand --out-dir output/segment --sam-dir F:/Study/SAM3
node tools/model-check.mjs --stage completion --input-dir output/segment --out-dir output/complete --part head --prepare-only node tools/model-check.mjs --stage completion --input-dir output/segment --out-dir output/complete --part head --prepare-only
node tools/model-check.mjs --stage spine --input-dir output/segment --completion-dir output/complete --out-dir output/spine
node tools/model-check.mjs --list-models node tools/model-check.mjs --list-models
node tools/model-check.mjs --dry-run node tools/model-check.mjs --dry-run
@@ -28,12 +30,22 @@ const help = `模型接入与结构理解(新增模式,不调用旧拆件流
--parts-file FILE 离线读取已有 v1/v2 计划,做 Alpha 校验与种子生成,不请求模型 --parts-file FILE 离线读取已有 v1/v2 计划,做 Alpha 校验与种子生成,不请求模型
--python FILE ai-spine Conda 环境解释器(默认使用已激活环境) --python FILE ai-spine Conda 环境解释器(默认使用已激活环境)
--input-dir DIR SAM 阶段读理解目录;补全阶段读 SAM 目录 --input-dir DIR SAM 阶段读理解目录;补全阶段读 SAM 目录
--completion-dir DIR Spine 阶段可选补全目录;input-dir 指向同批 SAM 目录
--fallback-policy sam|error Spine 未完成补全时回退 SAM(默认)或报错
--draw-order-file FILE Spine 绘制顺序覆盖:{"draw_order":[全部可绘制ID,从后到前]}
--check-angle FLOAT Spine 单骨骼检查动画角度,默认 10,范围 (0,45]
--genielabs-dir DIR 外部 spine-animation-ai 路径,仅用于 Spine 图集和播放器
--part ID 补全部件 ID,可重复;默认处理所有需补全部件 --part ID 补全部件 ID,可重复;默认处理所有需补全部件
--prepare-only 仅准备补全画布、掩码和提示词,不请求模型 --prepare-only 仅准备补全画布、掩码和提示词,不请求模型
--candidates-dir DIR 离线导入 ID.png,或旧补全目录的首张候选,不请求模型 --candidates-dir DIR 离线导入 ID.png,或旧补全目录的首张候选,不请求模型
--candidate-size-policy NAME strict(默认)/ uniform(同宽高比缩放后重新检查对齐) --candidate-size-policy NAME strict(默认)/ uniform(同宽高比缩放后重新检查对齐)
--completion-size INT 补全方形画布边长,默认 1024,不缩放原像素 --completion-profile NAME adaptive(默认)重建需求与可逆缩放;legacy 复用旧候选
--extension-radius INT 补全区域距可见轮廓的最大像素距离,默认 32 --completion-reference NAME auto(默认)/ board / local / full / both / none
--completion-mask NAME auto(默认)/ required / off;auto 遵循提供方能力
--completion-review FILE 复核 JSON,按 ID 指定修复/替换/移除 mask 与半径
--probe-edit-capabilities 单部件真实多图+mask 探针,不自动修改提供方能力配置
--completion-size INT 请求方形画布边长,默认 1024;adaptive 显式记录缩放逆变换
--extension-radius INT 全局原图像素半径覆盖;adaptive 默认按部件尺寸,legacy 默认 32
--sam-dir DIR 本地 SAM 根目录,包含 sam3 与 sam3.1checkpoint --sam-dir DIR 本地 SAM 根目录,包含 sam3 与 sam3.1checkpoint
--sam-code-dir DIR SAM 源码仓库目录覆盖 --sam-code-dir DIR SAM 源码仓库目录覆盖
--sam-checkpoint FILE 本地权重路径覆盖 --sam-checkpoint FILE 本地权重路径覆盖
@@ -44,7 +56,7 @@ const help = `模型接入与结构理解(新增模式,不调用旧拆件流
--dry-run 检查配置与本地输入,不请求模型、不生成产物 --dry-run 检查配置与本地输入,不请求模型、不生成产物
--stop-after 支持 understand / segment。SAM 在临时 GPU 子进程运行,完成后退出。 --stop-after 支持 understand / segment。SAM 在临时 GPU 子进程运行,完成后退出。
部件补全使用 --stage completion --input-dir;可见像素恢复与几何检查已接入,Spine 导出尚未接入。`; 部件补全使用 --stage completion --input-dir;--stage spine 离线组装并导出 Spine 4.2,资产和动作质量仍需复核。`;
export function parseOptions(argv) { export function parseOptions(argv) {
const options = {stage: 'understanding', imagePaths: [], partIds: []}; const options = {stage: 'understanding', imagePaths: [], partIds: []};
@@ -58,6 +70,10 @@ export function parseOptions(argv) {
['--sam-checkpoint', 'samCheckpoint'], ['--sam-confidence', 'samConfidence'], ['--sam-precision', 'samPrecision'], ['--sam-checkpoint', 'samCheckpoint'], ['--sam-confidence', 'samConfidence'], ['--sam-precision', 'samPrecision'],
['--candidates-dir', 'candidatesDir'], ['--completion-size', 'completionSize'], ['--extension-radius', 'extensionRadius'], ['--candidates-dir', 'candidatesDir'], ['--completion-size', 'completionSize'], ['--extension-radius', 'extensionRadius'],
['--candidate-size-policy', 'candidateSizePolicy'], ['--candidate-size-policy', 'candidateSizePolicy'],
['--completion-dir', 'completionDir'], ['--fallback-policy', 'fallbackPolicy'],
['--draw-order-file', 'drawOrderFile'], ['--check-angle', 'checkAngle'], ['--genielabs-dir', 'genielabsDir'],
['--completion-profile', 'completionProfile'], ['--completion-reference', 'completionReference'],
['--completion-mask', 'completionMask'], ['--completion-review', 'completionReview'],
]); ]);
for (let i = 0; i < argv.length; i += 1) { for (let i = 0; i < argv.length; i += 1) {
const flag = argv[i]; const flag = argv[i];
@@ -65,6 +81,7 @@ export function parseOptions(argv) {
else if (flag === '--resume') options.resume = true; else if (flag === '--resume') options.resume = true;
else if (flag === '--dry-run') options.dryRun = true; else if (flag === '--dry-run') options.dryRun = true;
else if (flag === '--prepare-only') options.prepareOnly = true; else if (flag === '--prepare-only') options.prepareOnly = true;
else if (flag === '--probe-edit-capabilities') options.probeEditCapabilities = true;
else if (flag === '--list-models') options.listModels = true; else if (flag === '--list-models') options.listModels = true;
else if (valueFlags.has(flag) || flag === '--image' || flag === '--part') { else if (valueFlags.has(flag) || flag === '--image' || flag === '--part') {
const value = argv[++i]; const value = argv[++i];
@@ -80,7 +97,19 @@ export function parseOptions(argv) {
export async function main(argv = process.argv.slice(2)) { export async function main(argv = process.argv.slice(2)) {
const options = parseOptions(argv); const options = parseOptions(argv);
if (options.help) { console.log(help); return; } if (options.help) { console.log(help); return; }
if (!['understanding', 'completion', 'segmentation'].includes(options.stage)) throw new Error('--stage 仅支持 understanding / completion / segmentation'); if (!['understanding', 'completion', 'segmentation', 'spine'].includes(options.stage)) throw new Error('--stage 仅支持 understanding / completion / segmentation / spine');
if (options.stage === 'spine') {
const allowed = new Set(['stage', 'inputDir', 'outDir', 'completionDir', 'fallbackPolicy', 'drawOrderFile',
'checkAngle', 'genielabsDir', 'python', 'resume', 'dryRun', 'mode', 'imagePaths', 'partIds']);
if (Object.keys(options).some((key) => !allowed.has(key)) || options.imagePaths.length || options.partIds.length ||
(options.mode && options.mode !== 'annotated-parts')) throw new Error('Spine 为独立离线阶段,不接受模型配置、请求或补全参数');
const result = await runSpine(options);
if (!result.dryRun) console.log(`Spine ${result.reused ? '复用完成' : '导出完成,需复核'}:${path.join(result.out, 'report.html')}`);
return result;
}
if (['completionDir', 'fallbackPolicy', 'drawOrderFile', 'checkAngle', 'genielabsDir'].some((key) => options[key] !== undefined)) {
throw new Error('Spine 导出参数仅用于 --stage spine');
}
if (options.partsFile && (options.stage !== 'understanding' || options.listModels)) throw new Error('--parts-file 仅用于离线结构理解'); if (options.partsFile && (options.stage !== 'understanding' || options.listModels)) throw new Error('--parts-file 仅用于离线结构理解');
if (options.streamRaw !== undefined && !['true', 'false'].includes(options.streamRaw)) throw new Error('--stream 仅支持 true 或 false'); if (options.streamRaw !== undefined && !['true', 'false'].includes(options.streamRaw)) throw new Error('--stream 仅支持 true 或 false');
options.stream = options.streamRaw === undefined ? undefined : options.streamRaw === 'true'; options.stream = options.streamRaw === undefined ? undefined : options.streamRaw === 'true';
@@ -90,7 +119,8 @@ export async function main(argv = process.argv.slice(2)) {
} }
const loaded = loadConfig(options); const loaded = loadConfig(options);
const partCompletion = options.stage === 'completion' && options.inputDir; const partCompletion = options.stage === 'completion' && options.inputDir;
if (!partCompletion && (options.prepareOnly || options.candidatesDir || options.partIds.length || options.completionSize || options.extensionRadius || options.candidateSizePolicy)) { if (!partCompletion && (options.prepareOnly || options.candidatesDir || options.partIds.length || options.completionSize || options.extensionRadius || options.candidateSizePolicy ||
options.completionProfile || options.completionReference || options.completionMask || options.completionReview || options.probeEditCapabilities)) {
throw new Error('部件补全参数需要 --stage completion --input-dir'); throw new Error('部件补全参数需要 --stage completion --input-dir');
} }
if (partCompletion) { if (partCompletion) {
+90
View File
@@ -0,0 +1,90 @@
// Optional, offline compatibility check using a caller-provided official Spine 4.2 core IIFE.
// node tools/verify-spine-runtime.mjs --runtime-file /path/spine-core.js --input-dir output/spine
import assert from 'node:assert/strict';
import {readFileSync} from 'node:fs';
import path from 'node:path';
import vm from 'node:vm';
import {readImage} from '../lib/images.mjs';
const options = {};
for (let i = 2; i < process.argv.length; i += 2) {
const flag = process.argv[i];
if (!['--runtime-file', '--input-dir'].includes(flag) || !process.argv[i + 1]) {
throw new Error('Usage: node tools/verify-spine-runtime.mjs --runtime-file spine-core.js --input-dir output/spine');
}
options[flag] = process.argv[i + 1];
}
if (!options['--runtime-file'] || !options['--input-dir']) throw new Error('Both --runtime-file and --input-dir are required');
const context = vm.createContext({console});
vm.runInContext(readFileSync(options['--runtime-file'], 'utf8'), context, {timeout: 10000});
const spine = context.spine;
const input = path.resolve(options['--input-dir']);
const assembly = JSON.parse(readFileSync(path.join(input, 'assembly.json')));
const json = JSON.parse(readFileSync(path.join(input, 'skeleton.json')));
assert.match(json.skeleton.spine, /^4\.2\./u);
const atlas = new spine.TextureAtlas(readFileSync(path.join(input, 'atlas/skeleton.atlas'), 'utf8'));
for (const page of atlas.pages) {
const png = readImage(path.join(input, 'atlas', page.name), {pngOnly: true});
assert.equal(page.width, png.width);
assert.equal(page.height, png.height);
// CPU geometry needs region metadata, not a WebGL texture upload.
page.setTexture({setFilters() {}, setWraps() {}, getImage() { return {width: png.width, height: png.height}; }});
}
const data = new spine.SkeletonJson(new spine.AtlasAttachmentLoader(atlas)).readSkeletonData(json);
const skeleton = new spine.Skeleton(data);
function vertices(slot) {
const values = new Float32Array(8);
slot.getAttachment().computeWorldVertices(slot, values, 0, 2);
assert.ok([...values].every(Number.isFinite));
return values;
}
function close(actual, expected, label) {
assert.ok(Math.abs(actual - expected) < .002, `${label}: ${actual} != ${expected}`);
}
skeleton.setToSetupPose();
skeleton.updateWorldTransform(spine.Physics.none);
const originals = new Map();
const assets = new Map(assembly.parts.map((p) => [p.id, p]));
for (const slot of skeleton.slots) {
const values = vertices(slot);
originals.set(slot.data.name, values);
const [x0, y0, x1, y1] = assets.get(slot.data.name).crop_xyxy;
const xs = [values[0], values[2], values[4], values[6]];
const ys = [values[1], values[3], values[5], values[7]];
close(Math.min(...xs), x0 - assembly.width / 2, 'setup x0');
close(Math.max(...xs), x1 - assembly.width / 2, 'setup x1');
close(Math.min(...ys), assembly.height - y1, 'setup y0');
close(Math.max(...ys), assembly.height - y0, 'setup y1');
const region = atlas.findRegion(slot.data.name);
assert.equal(region.originalWidth, x1 - x0);
assert.equal(region.originalHeight, y1 - y0);
}
let samples = 0;
for (const animation of data.animations) {
if (animation.name === 'setup') continue;
const boneName = animation.name.replace(/^check__/u, '');
const frames = json.animations[animation.name].bones[boneName].rotate;
for (const time of [.5, 1.5]) {
skeleton.setToSetupPose();
skeleton.updateWorldTransform(spine.Physics.none);
const bone = skeleton.findBone(boneName);
const px = bone.worldX, py = bone.worldY;
const angle = frames.find((f) => f.time === time).value * Math.PI / 180;
animation.apply(skeleton, 0, time, false, [], 1, spine.MixBlend.replace, spine.MixDirection.mixIn);
skeleton.updateWorldTransform(spine.Physics.none);
for (const slot of skeleton.slots) {
let parent = slot.bone;
while (parent && parent !== bone) parent = parent.parent;
const before = originals.get(slot.data.name), after = vertices(slot);
for (let i = 0; i < 8; i += 2) {
const x = before[i] - px, y = before[i + 1] - py;
close(after[i], parent ? px + x * Math.cos(angle) - y * Math.sin(angle) : before[i], 'rotated x');
close(after[i + 1], parent ? py + x * Math.sin(angle) + y * Math.cos(angle) : before[i + 1], 'rotated y');
}
}
samples++;
}
}
console.log(JSON.stringify({input, spine_version: data.version, bones: data.bones.length,
slots: data.slots.length, animations: data.animations.length, rotation_samples: samples,
setup_geometry_verified: true, descendant_rotation_verified: true, browser_render_verified: false}, null, 2));