第三阶段改进
This commit is contained in:
Ukawka
@@ -20,7 +20,7 @@
|
||||
|
||||
## 跨会话上下文与阅读顺序
|
||||
|
||||
本节于 2026-09-22 接入通用第四阶段后更新。它是后续对话的上下文入口；后续修改应同步更新对应状态，实际源码、配置和具体实验 manifest 优先于历史总结。不要把旧实验统计套到新输出上。
|
||||
本节于 2026-09-22 接入通用第四阶段及 adaptive 补全后更新。它是后续对话的上下文入口；后续修改应同步更新对应状态，实际源码、配置和具体实验 manifest 优先于历史总结。不要把旧实验统计套到新输出上。
|
||||
|
||||
1. 先阅读本文件和 `git status --short`，保留用户已有修改。
|
||||
2. 阅读 [README](README.md) 了解项目、环境、配置与命令。
|
||||
@@ -63,7 +63,7 @@
|
||||
- `configs/tiantoken.json`：独立 TIANTOKEN_BASE_URL / TIANTOKEN_API_KEY，地址 https://api.tiantoken.com；理解 gpt-6-astra，补全 gpt-image-2；模型变量为 TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL。
|
||||
- CLI 模型 > stage.model > 对应模型环境变量 > default_model；shell 同名变量优先于 .env。模型 ID 原样发送，不加隐式别名。
|
||||
- 用户提供的密钥仅留在本地忽略的 `.env`；禁止在文档、命令日志、聊天或提交中暴露，不要读取/打印完整 .env 来诊断。
|
||||
- 三份配置的图片编辑能力均为 multiple_images=false、mask=false。接口已预留分支不代表中转已支持；开启前需验证。Astra 理解接口与图片编辑接口不同，当前没有 Astra Responses 图片生成工具链。
|
||||
- 三份配置的图片编辑能力仍为 multiple_images=false、mask=false。adaptive auto 在此配置下发送单图参考板，包含局部和完整角色参考，mask 仅本地约束。独立多图/mask 和单部件探针已接入，但本轮未调用付费模型，不能宣称中转已支持。Astra 理解接口与图片编辑接口不同，当前没有 Astra Responses 图片生成工具链。
|
||||
|
||||
## 四阶段实现与不能丢失的边界
|
||||
|
||||
@@ -87,15 +87,18 @@
|
||||
|
||||
### 第三阶段：受限部件补全
|
||||
|
||||
- 入口 `--stage completion --input-dir SAM目录`；实现为 stages/completion-stage.mjs、completion_geometry.py、providers/image-edits.mjs。
|
||||
- 入口 `--stage completion --input-dir SAM目录`；实现为 stages/completion-stage.mjs、completion_geometry.py、completion_planning.py、providers/image-edits.mjs。默认 completion-profile=adaptive，legacy 保留旧行为和候选兼容。
|
||||
- 输入须含 source.png、v2 计划、segmentation.json 及所有记录的全画布 masks；单独 parts/ 不够。
|
||||
- 仅处理 needs_completion=true、非隐藏、有可见 mask 且允许区域非空的部件。--part 不绕过筛选；目前不重新评估 SAM 缺损。
|
||||
- 默认 1024 方形局部画布，原像素不缩放。允许区来自轮廓外 32 像素范围与遮挡 mask / pivot 圆盘交集，排除可见 mask；不是完整隐藏形状生成。
|
||||
- 当前真实请求只发白底 input.png；context.png、edit-mask.png 默认不发送。开启 multiple_images 也只是发送同坐标局部 context，并非独立全角色参考。
|
||||
- adaptive 从实际 mask 重建遮挡/关节、闭合漏分孔洞、附件底图清理需求，不单凭 needs_completion 筛选。自动孔洞只取原图前景且无其他 mask 归属的区域；开放缺口和歧义归属仍需复核。无明确遮挡的近重复 mask（IoU≥0.8）要求人工确认。
|
||||
- adaptive preserve=SAM−editable−removed；仅保留区锁定原 RGBA，清理和替换区可重绘，人工 remove 可删除。completion-review JSON（schema_version=1、parts 映射、confirmed=true）支持 repair/replace/remove 全画布二值 masks 与部件半径，原文件及哈希留档，不改 SAM。
|
||||
- 默认请求边长 1024；adaptive 半径按 mask 短边×0.15 向上取整并限于 4～96 原像素，CLI 或部件 review 可覆盖。native 裁剪边长约目标包围框最长边×1.15；小部件放大、大部件缩小，保存正反仿射和工作区。逆缩放后恢复保留原像素，坐标可供第四阶段使用。
|
||||
- reference auto：multiple_images=true 时 input→context→reference，默认 false 时发送 board（左目标，右局部和全角色参考），参考面板不进入最终资产。支持 none/local/full/both/board；mask auto/required/off 独立选择。request-plan.json 记录计划，request.json 仅实际发送时生成，含有序图片角色、哈希、尺寸、字段、mask 哈希与透明语义。
|
||||
- legacy 才沿用 needs_completion、原像素不缩放、默认 32 半径和原可见像素全保护。旧候选复检必须显式 --completion-profile legacy。新旧画布/参考布局/复核文件不同不能混用候选。
|
||||
- 图片解码已修复“空 b64_json 但有有效 URL”情况，回退下载 URL；该问题与画布尺寸不符不同。
|
||||
- 在线请求现显式传 size，配置冲突失败。strict 默认拒绝尺寸不同；uniform 仅恢复同宽高比整画布尺寸，保留原候选，继续原有验收，不单独移动/缩放物体。
|
||||
- 验收阈值：可见不透明像素 RGB MAE≤30、保留率≥95%；有效新增须在允许区且连接可见部件；原可见 RGBA 随后逐像素恢复。
|
||||
- unfulfilled_regions 当前只记录、不硬拒绝；语义错误、接缝、关节补够与否尚无可靠验收。错误 SAM 可见像素会被强制保留。
|
||||
- 对齐阈值保持 RGB MAE≤30、保留率≥95%；adaptive 在请求工作区的保留不透明像素上检查，再逆采样，避免把插值 Alpha 误作位移。编辑像素必须位于允许区且连接保留区；可见缺口使用原图纹理，移除区清零。
|
||||
- adaptive 清理/人工替换要求≥95% 有效填充，其余 unfulfilled_regions 仍仅诊断。语义、接缝和关节质量不能由填充率证明。未明确编辑的误分仍被保护，需复核文件修正。
|
||||
- --probe-edit-capabilities 要求 adaptive 在线且恰好一个 --part，发送主图/局部/全角色+mask，n=1、maxRetries=0。记录 capability-evidence 和合并前 raw_outside_edit 指标；不会把 reference_use_verified/mask_semantics_verified 自动置 true 或修改配置。prepare-only/mock 不是中转实测。
|
||||
- prepare-only 不联网；candidates-dir 可读平铺 ID.png 或匹配 manifest 的旧任务首张候选。--resume 不是失败断点续发，准备/生成/复检使用独立目录。
|
||||
- parts/ 只含通过检查的补全部件，不包含全部原拆件。完整角色组装、Spine 导出与旋转检查动画已独立接入第四阶段；运动质量自动验收仍未实现。
|
||||
- 旧 `--stage completion --image ... --prompt-file ...` 是接口探针，不能和 input-dir 混用，不执行完整几何验收。
|
||||
@@ -104,7 +107,7 @@
|
||||
|
||||
- 用户明确要求可复用管线，不能写死具体角色、部件 ID、数量或人形比例。入口 `--stage spine --input-dir SAM目录 --completion-dir 补全目录 --out-dir 新目录`；completion-dir 可省略。
|
||||
- `stages/spine-stage.mjs` 与 `spine_assembly.py` 完全离线，校验 v2、输入快照及 masks 哈希、PNG 和裁剪坐标；不读取模型配置/密钥，不调用 SAM。
|
||||
- 原资产从 source + mask 重建，采用补全结果前检查 selected_file、状态、qa.accepted、image、crop_xyxy 以及原可见 RGBA 保持。不能混用其他实验。
|
||||
- 原资产从 source + mask 重建，采用补全结果前检查 selected_file、状态、qa.accepted、image、crop_xyxy。legacy 全可见 RGBA 保持；adaptive 验证 preservation 分区、保留 RGBA 精确匹配、编辑/移除互斥及输出不越界，契约 mask 哈希留档。不能混用其他实验。
|
||||
- 默认补全失败回退 SAM 并记录原因；fallback-policy=error 拒绝需补全但无通过结果的可见部件。缺可见 mask 直接报错；隐藏骨骼保留但不生成图片。
|
||||
- pivot 转换为底边中点原点的 Spine 坐标；骨骼 setup 旋转为零，局部位置取父子 pivot 差；附件中心相对绑定骨骼偏移。bone_part 使用 parent，其他角色使用 bind_to。
|
||||
- occluded_by 稳定拓扑排序；无约束重叠写入诊断，可用 draw-order-file 全量覆盖，冲突留档。骨骼/遮挡环路拒绝。
|
||||
@@ -127,21 +130,24 @@
|
||||
| output/20260920-225137 | 较新实验：27 个语义项、2 个 hidden、25 个 SAM 部件；补全 16 项中 13 个几何通过、3 个对齐拒绝 |
|
||||
| output/spine-validation-latest-20260922 | 第四阶段离线验证：25 个 slot、16 个骨骼（含 root）；采用 13 个补全部件，5 个需补全部件回退 SAM（含两袖无允许区域）；上游缺口仍存在 |
|
||||
| output/spine-validation-baseline-20260922 | 另一批输入验证：22 个 slot、16 个骨骼（含 root）；采用 8 个补全部件、4 个回退；输入源为旧基线和 size-recheck-v2 |
|
||||
| output/completion-adaptive-prepare-20260922 | adaptive 离线准备：16 个任务、11 个跳过；上衣规划 3708 个封闭漏分像素，头部规划 10205 个附件清理像素；这些是待复核区域，不是补全成功结果，未请求模型 |
|
||||
|
||||
- 当前 22→12 的原因：sombrero、sleeve_l、sleeve_r、eye_l、eye_r、eyebrow_l、eyebrow_r、nose、mouth、mustache 共 10 个被第一阶段标记 NO_COMPLETION_NEEDED；不能解释为它们的实际 SAM mask 已完整。
|
||||
- 2026-09-20 尺寸修复后曾通过 22 项 Node 测试（含 Python 补全回归）。这是历史验证；新改动按范围重新验证，不自动沿用“全部通过”。显式 size 构造已测试，修复后中转是否遵守尚无新实测。
|
||||
- 总体判断：上游部件归属、模型缺少原图参考与遮罩、区域规则粗糙、验收不足共同导致低质量。不能仅换模型、强制处理全部 mask、放宽阈值或解决尺寸错误就宣称修好。
|
||||
- 2026-09-22 第四阶段接入后，Conda ai-spine 下 npm test 共 26 项通过（含新增 6 项 Python 组装用例）。另使用官方 Spine 4.2 core IIFE 对上述两份导出分别验证 setup 顶点和 30 个单关节旋转采样，全部通过。最新导出的 preview.html 已在 headless Chrome 实际加载渲染，截图为 output/spine-runtime-validation-20260922/preview.png；这不是动作或资产质量验收。复检工具为 `tools/verify-spine-runtime.mjs`，需调用者提供本地官方 runtime 文件，不自动下载/安装依赖。
|
||||
|
||||
- 随后接入 adaptive 补全，Conda ai-spine 下 npm test 共 29 项全部通过（含 8 项新增 Python adaptive 用例）：覆盖参考顺序/mask mock、放大/缩小、位移拒绝、实际缺口、清理与区域契约、候选身份及第四阶段 strict 回退。未进行付费中转试验，真实参考使用与 mask 语义仍待验证。
|
||||
|
||||
## 当前改进方向与后续顺序
|
||||
|
||||
当前用户优先推进可复用第四阶段；以下上游质量改进仍待实现，不因导出链路完成而自动解决：
|
||||
当前用户已要求落实第三阶段的参考图/区域重建/自适应画布，必须保持可复用且与第四阶段贯通。当前状态：
|
||||
|
||||
1. 第二阶段输出仅色彩遮罩组成的完整角色图，保留当前原图叠色预览，并明确未分配/重叠区域。
|
||||
2. 第二阶段每个 SAM 部件可选经过一次大模型去毛边，保留原始拆件、原图参考、坐标、变更区域与前后对照。
|
||||
3. 第三阶段将角色原图或部件相关原图局部作为参考交给补全模型，验证中转多图/遮罩能力及实际发送内容。
|
||||
3. 第三阶段参考图、实际 mask 规划、自适应区域/可逆缩放、发送审计和单部件能力探针已实现并离线验证；真实多图/mask 语义和新布局生成质量仍待付费小样实测，默认 capabilities 不变。
|
||||
4. 完整角色组装及失败回退记录已在第四阶段实现；接缝与关节旋转质量自动验收仍待实现。
|
||||
5. 后续质量工作：修正 SAM 归属和补全计划、验证参考图与遮罩，然后加强接缝/动作验收。第四阶段遵循统一输入契约，不为某个实验硬编码修正。
|
||||
5. 后续质量工作：复核自动规划和 SAM 归属、实测参考图与遮罩、加强接缝/动作验收。第三/四阶段遵循统一输入契约，不为某个实验硬编码修正。
|
||||
|
||||
详细改进说明保留在各阶段文档第 8 节；若后续用户改变优先级，应更新此处。
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ SAM_CODE_DIR=F:/Study/SAM3/sam3
|
||||
|
||||
CLI > 配置 stage.model > 对应模型环境变量 > default_model；shell 环境变量优先于 `.env`。不指定配置时入口默认 astra-qwen.json，建议实验始终显式指定。模型 ID 原样发送，中转能力以实际请求为准。Tiantoken 图片编辑已真实执行；第三阶段尚未适配 Astra Responses 图片生成工具。
|
||||
|
||||
三个配置当前均关闭多图参考与编辑 mask 能力。第三阶段默认只发送白底单部件图；本地 context.png/edit-mask.png 已生成但不默认发送。改进计划见第三阶段文档。
|
||||
三个配置当前仍关闭原生多图与编辑 mask 能力。第三阶段默认 adaptive：实际 mask 重建需求、自适应半径和可逆缩放；当前配置下 auto 会发送含目标、局部参考、完整角色参考的单张参考板。只有支持原生多图时才独立发送参考图；mask 是否发送由配置和 completion-mask 决定，未发送时明确标记 local_only。不能把本地裁剪与恢复当成模型遵循 mask 的证据。
|
||||
|
||||
查询模型列表和检查配置：
|
||||
|
||||
@@ -162,6 +162,8 @@ node .\tools\model-check.mjs --stage segmentation `
|
||||
|
||||
### 第三阶段：准备与补全
|
||||
|
||||
默认 `--completion-profile adaptive` 会重新检查实际 SAM 缺口、骨骼连接和附件重叠，即使旧计划 needs_completion=false 也可能生成修复任务。先检查 preparation 报告的区域与参考图，再请求模型；缺口归属和附件底图仍需语义复核。
|
||||
|
||||
先离线准备任务，不请求模型：
|
||||
|
||||
```powershell
|
||||
@@ -191,7 +193,9 @@ node .\tools\model-check.mjs --stage completion `
|
||||
--out-dir "$runRoot\complete"
|
||||
```
|
||||
|
||||
省略 part 不意味着强制补全全部 mask。检查 `completion-plan.json` 的跳过原因和 `completion-result.json` 的 QA；parts/ 仅包含通过检查的补全部件。所有结果仍需语义复核。
|
||||
省略 part 不意味着强制补全全部 mask。检查 `completion-plan.json` 的 assessments/跳过原因、每个任务的 input.png、request-plan.json 和编辑区域；parts/ 仅包含通过检查的补全部件。所有结果仍需语义复核。`--completion-review FILE` 可按 ID 提供修复/替换/移除区域及部件半径，格式见[第三阶段文档](docs/3-completion-stage.md)。
|
||||
|
||||
`--completion-reference auto` 默认在原生多图可用时发送主图→局部参考→全角色参考，否则使用 board；可显式选择 board/local/full/both/none。`--completion-mask auto|required|off` 控制 mask 请求，auto 遵循配置。`--probe-edit-capabilities --part ID` 是一次真实多图+mask 小样，固定 n=1、无 HTTP 重试，保存原候选及能力证据，不自动启用 capabilities。该实测会产生费用；当前实现只完成离线验证。
|
||||
|
||||
旧候选尺寸不符时，可离线导入复检，避免重新生成：
|
||||
|
||||
@@ -201,10 +205,11 @@ node .\tools\model-check.mjs --stage completion `
|
||||
--input-dir .\output\20260918-183532\sam `
|
||||
--candidates-dir .\output\20260918-183532\complete2_gpt `
|
||||
--out-dir "$runRoot\complete-size-recheck" `
|
||||
--completion-profile legacy `
|
||||
--candidate-size-policy uniform
|
||||
```
|
||||
|
||||
uniform 仅恢复同宽高比的整画布尺寸，不纠正物体位移、变形或语义。在线请求现已显式传 size；参数及中转实测边界见[第三阶段文档](docs/3-completion-stage.md)。
|
||||
旧画布候选必须指定 legacy。adaptive 候选需匹配源图、masks、profile、参考布局、半径和复核文件哈希；不能把旧候选直接映射到新画布。uniform 仅恢复同宽高比整画布尺寸，不纠正物体位移、变形或语义。在线请求显式传 size；参数及中转实测边界见[第三阶段文档](docs/3-completion-stage.md)。
|
||||
|
||||
### 第四阶段：离线组装与 Spine 导出
|
||||
|
||||
|
||||
+21
-5
@@ -145,6 +145,9 @@ def merge_candidate(folder, candidate_file, output_file, size_policy='strict'):
|
||||
raise ValueError('Unknown candidate size policy')
|
||||
folder = Path(folder)
|
||||
task = read_json(folder/'task.json')
|
||||
if task.get('profile') == 'adaptive':
|
||||
from completion_planning import merge_adaptive
|
||||
return merge_adaptive(folder, candidate_file, output_file, size_policy)
|
||||
original = np.array(Image.open(folder/'visible.png').convert('RGBA'))
|
||||
allowed = np.array(Image.open(folder/'allowed-mask.png').convert('L')) > 0
|
||||
candidate = Image.open(candidate_file).convert('RGBA')
|
||||
@@ -228,6 +231,9 @@ def finalize(directory):
|
||||
meta = read_json(out/'tasks'/name/'task.json')
|
||||
ox, oy = meta['origin_xy']
|
||||
task.update(image=f'parts/{name}.png', crop_xyxy=[bbox[0]+ox, bbox[1]+oy, bbox[2]+ox, bbox[3]+oy])
|
||||
if meta.get('preservation'):
|
||||
task['preservation'] = meta['preservation']
|
||||
task['transform'] = meta['transform']
|
||||
view = f'<img src="{task["image"]}">' if task.get('image') else ''
|
||||
for index, item in enumerate(task.get('candidates', []), 1):
|
||||
view += f'<p><a href="{html.escape(item["file"], quote=True)}">原候选 {index}</a> · <a href="tasks/{name}/qa-{index}.json">QA</a>'
|
||||
@@ -235,11 +241,14 @@ def finalize(directory):
|
||||
view += f' · <a href="tasks/{name}/merged-{index}-normalized.png">缩放后候选</a>'
|
||||
view += '</p>'
|
||||
warnings = html.escape(', '.join(task.get('upstream_warnings', [])))
|
||||
rows.append(f'<tr><td>{html.escape(name)}<p>{warnings}</p><a href="tasks/{name}/task.json">任务参数</a> · <a href="tasks/{name}/prompt.txt">提示词</a></td><td>{html.escape(task["status"])}</td><td><a href="tasks/{name}/visible.png"><img src="tasks/{name}/visible.png"></a></td><td><a href="tasks/{name}/allowed-mask.png"><img src="tasks/{name}/allowed-mask.png"></a><p><a href="tasks/{name}/context.png">原图局部参考</a></p></td><td>{view}</td><td><pre>{html.escape(json.dumps(qa, ensure_ascii=False, indent=2))}</pre></td></tr>')
|
||||
reference = f'<p><a href="tasks/{name}/input.png">实际请求主图</a> · <a href="tasks/{name}/request-plan.json">发送计划</a></p>'
|
||||
if (out/'tasks'/name/'reference.png').exists():
|
||||
reference += f'<a href="tasks/{name}/reference.png">完整角色参考</a>'
|
||||
rows.append(f'<tr><td>{html.escape(name)}<p>{warnings}</p><a href="tasks/{name}/task.json">任务参数</a> · <a href="tasks/{name}/prompt.txt">提示词</a>{reference}</td><td>{html.escape(task["status"])}</td><td><a href="tasks/{name}/visible.png"><img src="tasks/{name}/visible.png"></a></td><td><a href="tasks/{name}/allowed-mask.png"><img src="tasks/{name}/allowed-mask.png"></a><p><a href="tasks/{name}/context.png">原图局部参考</a></p></td><td>{view}</td><td><pre>{html.escape(json.dumps(qa, ensure_ascii=False, indent=2))}</pre></td></tr>')
|
||||
write_json(out/'completion-result.json', result)
|
||||
(out/'report.html').write_text('''<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>部件补全</title>
|
||||
<style>body{font:16px system-ui;margin:24px}td,th{border:1px solid #ccc;padding:8px}img{max-width:260px;max-height:300px;background:#eee}pre{white-space:pre-wrap}table{border-collapse:collapse}</style>
|
||||
<h1>第三阶段：部件补全</h1><p>新增区域受允许掩码限制，原图可见像素逐像素恢复。几何通过仅表示候选可供复核，不代表遮挡和关节纹理已正确补全。上游 SAM 错分仍需修正。</p>
|
||||
<h1>第三阶段：部件补全</h1><p>输出受允许掩码限制，保留区域逐像素恢复；adaptive 允许明确规划的附件清理、替换和移除，原始 SAM 始终保留。几何通过仅表示候选可供复核，不代表遮挡和关节纹理已正确补全。</p>
|
||||
<table><tr><th>部件 / 上游告警</th><th>状态</th><th>原可见区域</th><th>允许新增区域（白色）</th><th>补全候选</th><th>几何检查</th></tr>'''+''.join(rows)+'</table><h2>跳过</h2><pre>'+html.escape(json.dumps(result['skipped'], ensure_ascii=False, indent=2))+'</pre></html>', encoding='utf-8')
|
||||
|
||||
|
||||
@@ -249,16 +258,23 @@ def main():
|
||||
parser.add_argument('--out', required=True)
|
||||
parser.add_argument('--part', action='append', default=[])
|
||||
parser.add_argument('--size', type=int, default=1024)
|
||||
parser.add_argument('--radius', type=int, default=32)
|
||||
parser.add_argument('--radius', type=int)
|
||||
parser.add_argument('--profile', choices=['legacy', 'adaptive'], default='legacy')
|
||||
parser.add_argument('--reference-mode', choices=['none', 'local', 'full', 'both', 'board'], default='board')
|
||||
parser.add_argument('--review-file')
|
||||
parser.add_argument('--candidate')
|
||||
parser.add_argument('--merged')
|
||||
parser.add_argument('--qa')
|
||||
parser.add_argument('--candidate-size-policy', choices=['strict', 'uniform'], default='strict')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'prepare':
|
||||
if not 128 <= args.size <= 4096 or not 1 <= args.radius <= 256:
|
||||
if not 128 <= args.size <= 4096 or (args.radius is not None and not 1 <= args.radius <= 256):
|
||||
parser.error('size must be 128..4096 and radius 1..256')
|
||||
prepare(args.out, args.part, args.size, args.radius)
|
||||
if args.profile == 'adaptive':
|
||||
from completion_planning import prepare_adaptive
|
||||
prepare_adaptive(args.out, args.part, args.size, args.radius, args.reference_mode, args.review_file)
|
||||
else:
|
||||
prepare(args.out, args.part, args.size, args.radius or 32)
|
||||
elif args.action == 'merge':
|
||||
write_json(args.qa, merge_candidate(args.out, args.candidate, args.merged, args.candidate_size_policy))
|
||||
else:
|
||||
|
||||
@@ -0,0 +1,343 @@
|
||||
"""Adaptive completion planning, reference canvases and reversible native-pixel merging."""
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
from normalize_reference import prepare_rgba
|
||||
|
||||
|
||||
def read(file):
|
||||
return json.loads(Path(file).read_text(encoding='utf-8-sig'))
|
||||
|
||||
|
||||
def write(file, value):
|
||||
Path(file).write_text(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False), encoding='utf-8')
|
||||
|
||||
|
||||
def binary(file, shape):
|
||||
a = np.array(Image.open(file))
|
||||
if a.shape != shape or not np.isin(a, [0, 255]).all():
|
||||
raise ValueError(f'Expected full-canvas binary PNG: {file}')
|
||||
return a > 0
|
||||
|
||||
|
||||
def review_inputs(file, ids, shape):
|
||||
"""Review masks are explicitly authored data; never guess ownership from scores."""
|
||||
if not file:
|
||||
return {}, {}
|
||||
file = Path(file).resolve()
|
||||
data = read(file)
|
||||
if data.get('schema_version') != 1 or not isinstance(data.get('parts'), dict):
|
||||
raise ValueError('Completion review requires schema_version=1 and parts object')
|
||||
hashes = {'review.json': hashlib.sha256(file.read_bytes()).hexdigest()}
|
||||
result = {}
|
||||
for name, entry in data['parts'].items():
|
||||
if name not in ids or not isinstance(entry, dict) or entry.get('confirmed') is not True:
|
||||
raise ValueError(f'Review entry requires known ID and confirmed=true: {name}')
|
||||
if set(entry) - {'confirmed', 'extension_radius', 'repair_mask', 'replace_mask', 'remove_mask'}:
|
||||
raise ValueError(f'Unknown review fields: {name}')
|
||||
item = {'confirmed': True}
|
||||
if 'extension_radius' in entry:
|
||||
radius = entry['extension_radius']
|
||||
if type(radius) is not int or not 1 <= radius <= 256:
|
||||
raise ValueError(f'Invalid reviewed radius: {name}')
|
||||
item['extension_radius'] = radius
|
||||
for key in ['repair_mask', 'replace_mask', 'remove_mask']:
|
||||
if key not in entry:
|
||||
continue
|
||||
ref = entry[key]
|
||||
if not isinstance(ref, str) or Path(ref).is_absolute():
|
||||
raise ValueError('Review mask must be relative to review file')
|
||||
mask_file = (file.parent / ref).resolve()
|
||||
if not mask_file.is_relative_to(file.parent):
|
||||
raise ValueError('Review mask path escapes review directory')
|
||||
item[key] = binary(mask_file, shape)
|
||||
hashes[f'{name}/{key}'] = hashlib.sha256(mask_file.read_bytes()).hexdigest()
|
||||
result[name] = item
|
||||
return result, hashes
|
||||
|
||||
|
||||
def bounding(mask):
|
||||
ys, xs = np.nonzero(mask)
|
||||
return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1] if len(xs) else None
|
||||
|
||||
|
||||
def enclosed(mask):
|
||||
padded = np.pad(mask.astype(np.uint8), 1)
|
||||
fill = padded.copy()
|
||||
cv2.floodFill(fill, None, (0, 0), 1)
|
||||
return ~fill[1:-1, 1:-1].astype(bool)
|
||||
|
||||
|
||||
def white_image(array):
|
||||
im = Image.fromarray(array).convert('RGBA')
|
||||
return Image.alpha_composite(Image.new('RGBA', im.size, 'white'), im).convert('RGB')
|
||||
|
||||
|
||||
def prepare_adaptive(directory, selected_ids, size=1024, radius=None, reference_mode='board', review_file=None):
|
||||
# Import shared native coordinate helpers lazily to avoid a module cycle.
|
||||
from completion_geometry import localize, allowed_regions
|
||||
out = Path(directory)
|
||||
plan, seg = read(out / 'semantic-parts.json'), read(out / 'segmentation.json')
|
||||
rgba = np.array(Image.open(out / 'source.png').convert('RGBA'))
|
||||
h, w = rgba.shape[:2]
|
||||
ids = {p['id']: p for p in plan['parts']}
|
||||
if set(selected_ids) - set(ids):
|
||||
raise ValueError('Unknown requested part ID')
|
||||
records = {p['id']: p for p in seg['parts']}
|
||||
if len(records) != len(seg['parts']) or set(records) - set(ids):
|
||||
raise ValueError('Invalid segmentation IDs')
|
||||
masks = {name: binary(out / 'masks' / f'{name}.png', (h, w)) for name in records}
|
||||
for name, mask in masks.items():
|
||||
if not mask.any() or np.any(mask & (rgba[:, :, 3] == 0)):
|
||||
raise ValueError(f'Empty mask or mask outside original alpha: {name}')
|
||||
reviews, review_hashes = review_inputs(review_file, ids, (h, w))
|
||||
coverage = np.zeros((h, w), np.uint16)
|
||||
for mask in masks.values():
|
||||
coverage += mask
|
||||
tasks, skipped, assessments = [], [], []
|
||||
(out / 'preservation').mkdir(exist_ok=True)
|
||||
for part in plan['parts']:
|
||||
name = part['id']
|
||||
if selected_ids and name not in selected_ids:
|
||||
continue
|
||||
if name not in masks or part['visibility'] == 'hidden':
|
||||
skipped.append({'id': name, 'reason': 'NO_VISIBLE_SAM_MASK'})
|
||||
continue
|
||||
visible = masks[name]
|
||||
box = bounding(visible)
|
||||
review = reviews.get(name, {})
|
||||
resolved_radius = review.get('extension_radius', radius if radius is not None else
|
||||
int(np.clip(math.ceil(min(box[2]-box[0], box[3]-box[1]) * .15), 4, 96)))
|
||||
inferred = dict(part)
|
||||
inferred['completion'] = {
|
||||
'occlusion_fill': bool(part['occluded_by']),
|
||||
'joint_extension_parent': part['rig_role'] == 'bone_part' and part['parent'] is not None,
|
||||
'joint_extension_child': part['rig_role'] == 'bone_part' and any(
|
||||
p['parent'] == name and p['rig_role'] == 'bone_part' for p in plan['parts'])}
|
||||
_, regions, warnings = allowed_regions(inferred, plan, masks, visible, resolved_radius)
|
||||
# Only enclosed source foreground without another mask owner is an automatic
|
||||
# repair candidate. Open gaps and ambiguous ownership stay diagnostic.
|
||||
gap = enclosed(visible) & (rgba[:, :, 3] > 0) & (coverage == 0)
|
||||
cleanup = np.zeros_like(visible)
|
||||
for front in part['occluded_by']:
|
||||
p = ids[front]
|
||||
if front in masks and p['rig_role'] != 'bone_part' and p['bind_to'] == name:
|
||||
cleanup |= visible & masks[front]
|
||||
replace = review.get('replace_mask', np.zeros_like(visible))
|
||||
removed = review.get('remove_mask', np.zeros_like(visible))
|
||||
repair = review.get('repair_mask', np.zeros_like(visible))
|
||||
if np.any((replace | removed) & ~visible) or np.any(repair & visible) or np.any(replace & removed):
|
||||
raise ValueError(f'{name}: replacement/removal must be disjoint subsets of SAM; repair must be outside SAM')
|
||||
regions['segmentation_gap'] = gap | repair
|
||||
regions['attachment_cleanup'] = cleanup & ~removed
|
||||
regions['reviewed_replacement'] = replace
|
||||
allowed = np.zeros_like(visible)
|
||||
for region in regions.values():
|
||||
allowed |= region
|
||||
allowed &= ~removed
|
||||
protected = visible & ~allowed & ~removed
|
||||
unexpected = []
|
||||
duplicates = []
|
||||
for other, mask in masks.items():
|
||||
if other == name:
|
||||
continue
|
||||
overlap = int((mask & visible).sum())
|
||||
related = other in part['occluded_by'] or name in ids[other]['occluded_by']
|
||||
if overlap and not related:
|
||||
unexpected.append({'id': other, 'pixels': overlap})
|
||||
if overlap / max(1, int((mask | visible).sum())) >= .8:
|
||||
duplicates.append(other)
|
||||
assessment = {'id': name, 'original_needs_completion': part['needs_completion'],
|
||||
'inferred_completion': inferred['completion'], 'radius': resolved_radius,
|
||||
'region_pixels': {k: int(v.sum()) for k, v in regions.items()},
|
||||
'protected_pixels': int(protected.sum()), 'removed_pixels': int(removed.sum()),
|
||||
'unexpected_overlaps': unexpected, 'review_confirmed': bool(review)}
|
||||
assessments.append(assessment)
|
||||
if duplicates and not review:
|
||||
skipped.append({'id': name, 'reason': 'AMBIGUOUS_MASK_REQUIRES_REVIEW', 'overlaps': duplicates})
|
||||
continue
|
||||
if not protected.any() or not (protected & (rgba[:, :, 3] >= 240)).any():
|
||||
skipped.append({'id': name, 'reason': 'NO_PRESERVED_ALIGNMENT_ANCHOR'})
|
||||
continue
|
||||
if not allowed.any() and not removed.any():
|
||||
skipped.append({'id': name, 'reason': 'NO_OBSERVED_EDIT_REGION'})
|
||||
continue
|
||||
region_box = bounding(visible | allowed)
|
||||
bw, bh = region_box[2] - region_box[0], region_box[3] - region_box[1]
|
||||
native_size = max(8, math.ceil(max(bw, bh) * 1.15))
|
||||
origin = [region_box[0] - (native_size-bw)//2, region_box[1] - (native_size-bh)//2]
|
||||
work_size = int(size * .68) if reference_mode == 'board' else size
|
||||
offset = [int(size * .02), (size-work_size)//2] if reference_mode == 'board' else [0, 0]
|
||||
scale = work_size / native_size
|
||||
folder = out / 'tasks' / name
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
native_source = localize(rgba, origin, native_size)
|
||||
native_visible = localize(visible, origin, native_size)
|
||||
native_keep = localize(protected, origin, native_size)
|
||||
native_edit = localize(allowed, origin, native_size)
|
||||
native_remove = localize(removed, origin, native_size)
|
||||
cut = native_source.copy()
|
||||
cut[~native_keep] = 0
|
||||
Image.fromarray(cut).save(folder / 'native-visible.png')
|
||||
Image.fromarray(native_source).save(folder / 'native-source.png')
|
||||
for label, mask in [('keep', native_keep), ('allowed', native_edit), ('remove', native_remove), ('sam', native_visible)]:
|
||||
Image.fromarray(mask.astype(np.uint8)*255).save(folder / f'native-{label}.png')
|
||||
|
||||
def project(array, is_mask=False):
|
||||
small = np.array(Image.fromarray(array.astype(np.uint8)*255 if is_mask else array).resize(
|
||||
(work_size, work_size), Image.Resampling.NEAREST if is_mask else Image.Resampling.LANCZOS))
|
||||
canvas = np.zeros((size, size) + small.shape[2:], np.uint8)
|
||||
canvas[offset[1]:offset[1]+work_size, offset[0]:offset[0]+work_size] = small
|
||||
return canvas
|
||||
|
||||
model_visible = project(cut)
|
||||
Image.fromarray(model_visible).save(folder / 'visible.png')
|
||||
model_allowed = project(native_edit, True) > 0
|
||||
model_remove = project(native_remove, True) > 0
|
||||
Image.fromarray(model_allowed.astype(np.uint8)*255).save(folder / 'allowed-mask.png')
|
||||
mask_rgba = np.full((size, size, 4), 255, np.uint8)
|
||||
mask_rgba[model_allowed | model_remove, 3] = 0
|
||||
Image.fromarray(mask_rgba).save(folder / 'edit-mask.png')
|
||||
context = white_image(project(native_source))
|
||||
context.save(folder / 'context.png')
|
||||
full_reference = white_image(rgba)
|
||||
full_reference.thumbnail((size, size), Image.Resampling.LANCZOS)
|
||||
full_reference.save(folder / 'reference.png')
|
||||
model_input = white_image(model_visible)
|
||||
panel_boxes = []
|
||||
if reference_mode == 'board':
|
||||
draw = ImageDraw.Draw(model_input)
|
||||
x = int(size * .74)
|
||||
draw.line([(x-4, 0), (x-4, size)], fill='#777777', width=1)
|
||||
for label, image, top in [('LOCAL REFERENCE', white_image(native_source), 0),
|
||||
('FULL REFERENCE', full_reference.copy(), size//2)]:
|
||||
draw.text((x, top+2), label, fill='black')
|
||||
image.thumbnail((size-x-4, size//2-24), Image.Resampling.LANCZOS)
|
||||
model_input.paste(image, (x, top+22))
|
||||
panel_boxes.append([x, top+22, x+image.width, top+22+image.height])
|
||||
model_input.save(folder / 'input.png')
|
||||
bounds, native_counts = {}, {}
|
||||
for kind, region in regions.items():
|
||||
native = localize(region & allowed, origin, native_size)
|
||||
model = project(native, True)
|
||||
Image.fromarray(native.astype(np.uint8)*255).save(folder / f'native-{kind}.png')
|
||||
Image.fromarray(model).save(folder / f'{kind}.png')
|
||||
bounds[kind] = bounding(model > 0)
|
||||
native_counts[kind] = int(native.sum())
|
||||
preservation = {}
|
||||
for key, mask in [('preserve_mask', protected), ('editable_mask', allowed), ('removed_mask', removed)]:
|
||||
relative = f'preservation/{name}-{key}.png'
|
||||
Image.fromarray(mask.astype(np.uint8)*255).save(out / relative)
|
||||
preservation[key] = relative
|
||||
if unexpected:
|
||||
warnings.append('UNEXPECTED_OVERLAP_REVIEW_REQUIRED')
|
||||
if cleanup.any():
|
||||
warnings.append('ATTACHMENT_BASE_CLEANUP_NEEDS_REVIEW')
|
||||
task = {'id': name, 'semantic': part['semantic'], 'profile': 'adaptive',
|
||||
'completion': inferred['completion'], 'original_completion': part['completion'],
|
||||
'origin_xy': origin, 'native_size': native_size, 'canvas_size': [size, size], 'source_size': [w, h],
|
||||
'extension_radius': resolved_radius, 'allowed_pixels': int(native_edit.sum()),
|
||||
'removed_pixels': int(native_remove.sum()), 'regions': native_counts,
|
||||
'region_bounds_xyxy': bounds, 'occluded_by': part['occluded_by'],
|
||||
'warnings': records[name].get('warnings', []) + warnings, 'preservation': preservation,
|
||||
'reference_mode': reference_mode, 'reference_panel_boxes': panel_boxes,
|
||||
'transform': {'scale': scale, 'working_xyxy': [*offset, offset[0]+work_size, offset[1]+work_size],
|
||||
'source_to_canvas': [scale, 0, offset[0]-scale*origin[0]+(scale-1)/2,
|
||||
0, scale, offset[1]-scale*origin[1]+(scale-1)/2],
|
||||
'canvas_to_source': [1/scale, 0, origin[0]-offset[0]/scale+(1/scale-1)/2,
|
||||
0, 1/scale, origin[1]-offset[1]/scale+(1/scale-1)/2]},
|
||||
'semantic_verified': False, 'status': 'prepared'}
|
||||
write(folder / 'task.json', task)
|
||||
tasks.append(task)
|
||||
result = {'schema_version': 2, 'profile': 'adaptive', 'tasks': tasks, 'skipped': skipped,
|
||||
'assessments': assessments, 'review_hashes': review_hashes, 'reference_mode': reference_mode,
|
||||
'canvas_size': size, 'quality_status': 'needs_review'}
|
||||
write(out / 'completion-plan.json', result)
|
||||
return result
|
||||
|
||||
|
||||
def merge_adaptive(folder, candidate_file, output_file, size_policy):
|
||||
folder = Path(folder)
|
||||
task = read(folder / 'task.json')
|
||||
candidate = Image.open(candidate_file).convert('RGBA')
|
||||
qa = {'accepted': False, 'semantic_verified': False, 'reasons': [], 'profile': 'adaptive',
|
||||
'expected_size': task['canvas_size'], 'returned_size': list(candidate.size),
|
||||
'size_policy': size_policy, 'canvas_resampled': False, 'transform': task['transform']}
|
||||
tw, th = task['canvas_size']
|
||||
if candidate.size != (tw, th):
|
||||
if size_policy == 'strict' or candidate.width * th != candidate.height * tw:
|
||||
qa['reasons'].append('CANVAS_SIZE_MISMATCH' if size_policy == 'strict' else 'CANVAS_ASPECT_RATIO_MISMATCH')
|
||||
return qa
|
||||
qa.update(canvas_resampled=True, canvas_scale=tw/candidate.width, resampling='LANCZOS')
|
||||
candidate = candidate.resize((tw, th), Image.Resampling.LANCZOS)
|
||||
candidate.save(Path(output_file).with_name(Path(output_file).stem + '-normalized.png'))
|
||||
# Record raw behavior before masking/restoration; an accepted HTTP response is
|
||||
# not evidence that the relay/model respected the supplied edit mask.
|
||||
expected_input = np.array(Image.open(folder / 'input.png').convert('RGB')).astype(float)
|
||||
raw = np.array(white_image(np.array(candidate))).astype(float)
|
||||
editable = np.array(Image.open(folder / 'edit-mask.png'))[:, :, 3] == 0
|
||||
outside_delta = np.abs(raw - expected_input).mean(axis=2)
|
||||
qa['raw_outside_edit_mae'] = float(outside_delta[~editable].mean()) if (~editable).any() else 0
|
||||
qa['raw_outside_edit_changed_pixels'] = int(((outside_delta > 30) & ~editable).sum())
|
||||
rect = task['transform']['working_xyxy']
|
||||
try:
|
||||
cropped, preprocessing = prepare_rgba(candidate.crop(rect))
|
||||
except ValueError as error:
|
||||
qa.update(reasons=['CANNOT_EXTRACT_FOREGROUND'], detail=str(error))
|
||||
return qa
|
||||
side = task['native_size']
|
||||
generated = np.array(cropped.resize((side, side), Image.Resampling.LANCZOS))
|
||||
original = np.array(Image.open(folder / 'native-visible.png').convert('RGBA'))
|
||||
source = np.array(Image.open(folder / 'native-source.png').convert('RGBA'))
|
||||
keep = binary(folder / 'native-keep.png', (side, side))
|
||||
allowed = binary(folder / 'native-allowed.png', (side, side))
|
||||
removed = binary(folder / 'native-remove.png', (side, side))
|
||||
# Judge alignment before the inverse resampling. Alpha at native boundary
|
||||
# pixels changes during interpolation even for an exactly aligned candidate.
|
||||
model_original = np.array(Image.open(folder / 'visible.png').convert('RGBA').crop(rect))
|
||||
model_generated = np.array(cropped)
|
||||
opaque = model_original[:, :, 3] >= 240
|
||||
if not opaque.any():
|
||||
qa['reasons'].append('NO_OPAQUE_ALIGNMENT_ANCHOR')
|
||||
return qa
|
||||
rgb = np.array(white_image(model_generated)).astype(float)
|
||||
mae = float(np.abs(rgb[opaque] - model_original[:, :, :3][opaque]).mean())
|
||||
retained = float((model_generated[:, :, 3][opaque] >= 240).mean())
|
||||
proposed = (generated[:, :, 3] > 0) & ~keep
|
||||
additions = proposed & allowed
|
||||
_, labels = cv2.connectedComponents((keep | additions).astype(np.uint8), connectivity=8)
|
||||
touching = np.unique(labels[keep])
|
||||
connected = additions & np.isin(labels, touching[touching != 0])
|
||||
counts = {kind: int((connected & binary(folder / f'native-{kind}.png', (side, side))).sum())
|
||||
for kind in task['regions']}
|
||||
qa.update(alignment_mae=mae, visible_retention=retained, alignment_space='request_work_panel', new_pixels=int(connected.sum()),
|
||||
removed_pixels=int(removed.sum()), region_additions=counts,
|
||||
unfulfilled_regions=[k for k, n in task['regions'].items() if n and not counts[k]],
|
||||
discarded_outside_allowed=int((proposed & ~allowed).sum()), preprocessing=preprocessing)
|
||||
if mae > 30 or retained < .95:
|
||||
qa['reasons'].append('VISIBLE_ALIGNMENT_MISMATCH')
|
||||
if not connected.any() and not removed.any():
|
||||
qa['reasons'].append('NO_CONNECTED_NEW_PIXELS')
|
||||
for kind in ['attachment_cleanup', 'reviewed_replacement']:
|
||||
if task['regions'][kind] and counts[kind] / task['regions'][kind] < .95:
|
||||
qa['reasons'].append(f'INCOMPLETE_{kind.upper()}')
|
||||
if qa['reasons']:
|
||||
return qa
|
||||
merged = np.zeros_like(original)
|
||||
merged[connected] = generated[connected]
|
||||
# Visible segmentation gaps already have ground-truth texture in source.
|
||||
# Restore it where the proposed fill connects, rather than trusting a repaint.
|
||||
repair = binary(folder / 'native-segmentation_gap.png', (side, side)) & connected & (source[:, :, 3] > 0)
|
||||
merged[repair] = source[repair]
|
||||
merged[keep] = original[keep]
|
||||
merged[removed] = 0
|
||||
qa.update(accepted=True, quality_status='needs_review', visible_pixels_exact=True,
|
||||
preservation_scope='preserve_mask', repaired_source_pixels=int(repair.sum()))
|
||||
Image.fromarray(merged).save(output_file)
|
||||
return qa
|
||||
+76
-24
@@ -2,7 +2,9 @@
|
||||
|
||||
## 1. 阶段简介
|
||||
|
||||
读取 SAM 可见 mask，为需要遮挡补全或关节延伸的部件准备局部画布，调用中转站图片编辑模型，再将允许区域内的新增像素与原可见像素合并。Python 几何处理固定使用 Conda `ai-spine`；图片生成在远端完成，本阶段不加载本地 SAM。
|
||||
读取 SAM 可见 mask，规划分割缺口修复、遮挡补全、关节延伸和附件底图清理，向图片模型提供部件与原图参考，再把结果映射回原图坐标。Python 几何处理固定使用 Conda `ai-spine`；图片生成在远端完成，本阶段不加载本地 SAM。
|
||||
|
||||
默认 `--completion-profile adaptive` 使用实际 mask 重建需求、自适应半径和可逆缩放；`legacy` 保留旧画布与筛选规则，供旧候选复检。原 SAM 输入始终留档，只有保留区域锁定原像素；明确规划的编辑区域可以替换，人工确认的移除区域可删除。
|
||||
|
||||
当前已实现准备、真实图片请求、离线候选导入、几何检查与透明补全部件输出。[第四阶段](4-spine-stage.md) 独立整合 SAM 与补全部件，输出完整角色组装、Spine 骨架和预览。几何通过仍为 `needs_review`，不能等同于部件可用。
|
||||
|
||||
@@ -14,18 +16,19 @@
|
||||
tools/model-check.mjs --stage completion --input-dir SAM目录
|
||||
→ stages/completion-stage.mjs 校验计划、mask 文件、参数及 Conda
|
||||
→ completion_geometry.py prepare
|
||||
筛选需求 → 计算允许区域 → 建立局部画布与编辑 mask
|
||||
adaptive：completion_planning.py 重建需求 → 区分保留/编辑/移除 → 缩放工作区并加入参考
|
||||
legacy：沿用 needs_completion → 固定半径 → 原像素平移
|
||||
→ prepare-only：保存任务和报告后结束
|
||||
→ 否则：串行请求 images/edits，或读取离线候选
|
||||
→ completion_geometry.py merge
|
||||
检查画布/前景/对齐 → 限制新增区域与连通性 → 恢复原可见像素
|
||||
检查画布/前景/对齐 → 逆变换回原像素尺寸 → 限制编辑和连通性 → 恢复保留像素
|
||||
→ 每部件选第一个通过几何检查的候选
|
||||
→ finalize 输出透明部件、原图坐标和报告
|
||||
```
|
||||
|
||||
任务必须同时满足：计划 `needs_completion=true`、非 hidden、有可见 SAM mask、允许新增区域非空。指定 `--part` 只缩小范围，不强制绕过筛选。分别记录 `NO_COMPLETION_NEEDED`、`NO_VISIBLE_SAM_MASK`、`NO_ALLOWED_EXTENSION`。
|
||||
adaptive 检查所有选中部件的实际 mask，不再以第一阶段 needs_completion 为唯一门槛。必须有可见 mask、保留对齐锚点，以及非空编辑或移除区域。隐藏/缺 mask、无观察到的编辑区域、无保留锚点分别跳过；无明确遮挡关系的近重复 mask（IoU≥0.8）要求人工复核。一般有歧义的重叠只记录诊断，不自动猜测归属。指定 `--part` 只缩小处理范围。
|
||||
|
||||
以 `output/20260918-183532/sam` 为例：32 个计划项中 10 个 hidden 没有 mask；22 个有 mask，其中 12 个进入补全。帽子、两袖、两眼、两眉、鼻、嘴、胡子这 10 个因第一阶段标记不需补全而跳过，没有重新由模型判断实际 mask 完整性。
|
||||
legacy 才要求 `needs_completion=true` 且允许新增非空。旧实验 `output/20260918-183532/sam` 的 22 个可见部件中 12 个进入补全，是旧筛选结果，不能套用于 adaptive。
|
||||
|
||||
## 3. 阶段执行命令示例及参数说明
|
||||
|
||||
@@ -68,6 +71,7 @@ node .\tools\model-check.mjs --stage completion `
|
||||
--input-dir $samInput `
|
||||
--candidates-dir .\output\20260918-183532\complete2_gpt `
|
||||
--out-dir ".\output\$runId\complete-recheck" `
|
||||
--completion-profile legacy `
|
||||
--candidate-size-policy uniform
|
||||
```
|
||||
|
||||
@@ -83,6 +87,18 @@ node .\tools\model-check.mjs --stage completion `
|
||||
|
||||
探针只保存候选，不执行部件准备与几何验收，不能与 `--input-dir` 混用。
|
||||
|
||||
新的单部件多图/mask 能力探针会产生一次真实请求，并执行新几何链路；不会修改默认 capabilities。仅在决定进行付费小样时运行，替换 part 为实际存在的可执行任务：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir $samInput `
|
||||
--out-dir ".\output\$runId\capability-probe" `
|
||||
--part head --probe-edit-capabilities
|
||||
```
|
||||
|
||||
该探针发送主图、局部参考、全角色参考和透明可编辑 mask，强制 n=1、禁用 HTTP 重试。查看 request.json、原候选、QA 和 capability-evidence.json；HTTP 接受不等于参考图被使用或 mask 语义正确，因此不会自动启用能力。prepare-only 可配 `--completion-reference both --completion-mask required` 检查计划发送内容而不联网。
|
||||
|
||||
| 参数 | 说明 |
|
||||
| --- | --- |
|
||||
| `--input-dir DIR` | 完整 SAM 输出目录，部件补全必需 |
|
||||
@@ -90,8 +106,13 @@ node .\tools\model-check.mjs --stage completion `
|
||||
| `--part ID` | 可重复，ID 必须存在且不重复；省略时处理所有符合筛选条件的部件 |
|
||||
| `--prepare-only` | 只生成画布、区域、提示词和报告，不请求模型 |
|
||||
| `--candidates-dir DIR` | 离线导入候选，与 prepare-only 互斥 |
|
||||
| `--completion-size N` | 方形画布边长，默认 1024，整数 128～4096；原像素不缩放 |
|
||||
| `--extension-radius N` | 轮廓延展距离与关节圆盘半径，默认 32，整数 1～256 |
|
||||
| `--completion-profile adaptive/legacy` | 默认 adaptive；旧候选需显式 legacy |
|
||||
| `--completion-reference auto/board/local/full/both/none` | auto：支持多图时 local+full，否则单图参考板；legacy auto 为旧单图/局部参考行为 |
|
||||
| `--completion-mask auto/required/off` | auto 遵循配置；required 要求已配置支持，或用单部件能力探针试验；off 不发送但仍本地限制编辑 |
|
||||
| `--completion-review FILE` | 人工确认的补全区域文件，格式见第 4 节；只用于 adaptive |
|
||||
| `--probe-edit-capabilities` | adaptive 在线单部件试验多图+mask，不持久修改配置，不自动认定能力支持 |
|
||||
| `--completion-size N` | 请求方形画布边长，默认 1024，整数 128～4096；adaptive 记录显式缩放和逆变换 |
|
||||
| `--extension-radius N` | 原图像素半径覆盖，整数 1～256；adaptive 默认 min(mask宽,高)×0.15 向上取整并限于 4～96；legacy 默认 32 |
|
||||
| `--candidate-size-policy NAME` | strict 默认拒绝尺寸不符；uniform 允许同宽高比整画布缩放后复检 |
|
||||
| `--completion-model ID` | 图片模型 ID，原样发送 |
|
||||
| `--config FILE` / `--env-file FILE` | 默认 astra-qwen.json / 项目 .env；shell 同名变量优先 |
|
||||
@@ -122,19 +143,42 @@ node .\tools\model-check.mjs --stage completion `
|
||||
|
||||
不同原图、mask 或参数的候选不能混用。尺寸之外，部件位置、朝向、比例也必须保持一致。
|
||||
|
||||
adaptive 的旧运行候选还要匹配 profile、参考布局、复核文件/掩码哈希和规划版本；board 与独立多图的画布不同，不可混用。平铺候选仍需人工保证它对应当前请求画布。
|
||||
|
||||
人工复核文件示例（JSON 中的 ID 取自当前语义计划，字段均可省略但 confirmed 必须为 true）：
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"parts": {
|
||||
"part_id": {
|
||||
"confirmed": true,
|
||||
"extension_radius": 24,
|
||||
"repair_mask": "review-masks/repair.png",
|
||||
"replace_mask": "review-masks/replace.png",
|
||||
"remove_mask": "review-masks/remove.png"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
mask 是与 source 同尺寸的单通道 0/255 PNG，白色表示区域；路径相对于复核 JSON 且不得越出其目录。repair 必须在当前 SAM mask 外；replace/remove 必须位于 SAM 内且互不重叠。replace 用于误分重绘或额外底图清理，remove 用于明确不属于该部件的像素。半径优先级为部件复核值 > CLI 全局值 > 自适应值。文件及 masks 会复制进输出，哈希纳入复用身份。
|
||||
|
||||
## 5. 执行链路中重要部分代码文件及原理说明（包括重点说明）
|
||||
|
||||
**允许区域。** `completion_geometry.py::allowed_regions()` 先计算可见 mask 外、距轮廓不超过 extension-radius 的区域，再分别与以下区域取交集：遮挡补全用 occluded_by 的 mask 并集；父关节延展用本部件 pivot 周围圆盘；子关节延展用直接子骨骼部件 pivot 圆盘。始终排除原可见 mask，新增范围限制在原图画布内。它是保守几何范围，不是完整隐藏形状推断。
|
||||
**按实际 mask 重建需求。** 遮挡延伸使用轮廓外半径带与 occluded_by 的实际 mask 交集；父/子关节需求来自骨骼层级与 pivot。自动分割修复仅选择 mask 内部闭合孔洞、原图有前景且无其他 mask 归属的像素；开放缺口和重叠归属不自动猜测，可用复核文件指定。自动底图清理仅针对 occluded_by 中直接 bind_to 当前骨骼的附件/五官与当前 mask 的交集。这些仍是几何候选，语义不自动获证。
|
||||
|
||||
**局部坐标。** `prepare()` 将可见区和允许区居中放入方形画布，至少留 16 像素边距，只平移和补边。`origin_xy` 是局部左上角在原图的位置，可为负数；原图位置=局部位置+origin。放不下时失败，不能偷偷缩放原像素。
|
||||
**保留与编辑。** editable 为遮挡/关节/修复/清理/人工替换区域的并集，扣除 removed；preserve = SAM − editable − removed。保留原 mask 和源图，并另存三种全画布区域供第四阶段核对。只有原始 SAM 或单一图片目录不足以描述这个契约。
|
||||
|
||||
**重点：当前实际发送什么。** `stages/completion-stage.mjs` 默认只发送白底 `input.png`。三个现有配置均为 `multiple_images=false`、`mask=false`，因此本地生成 context.png 和 edit-mask.png 不意味着发给模型。只有开启并验证对应能力后，分支才发送第二张同坐标局部原图参考和编辑遮罩；当前没有独立全角色参考图输入。文字边界说明及返回后的区域裁剪不能替代模型侧遮罩约束。
|
||||
**可逆画布。** native 方形裁剪边长为可见+编辑区域最长边的 1.15 倍向上取整，记录 origin。工作区缩放到请求画布（board 工作区边长为请求边长约 68%，其余为参考板；独立参考时工作区占整幅）。RGBA 用 LANCZOS，区域 mask 用最近邻。task.json 保存 native_size、scale、working_xyxy 和含像素中心约定的正反仿射变换。返回后只取工作区，逆缩放到 native，再裁剪回原图坐标；保留像素从原图恢复，所以缩放不改变最终保留像素。legacy 仍仅平移补边。
|
||||
|
||||
**实际发送内容。** 当前三个提供方默认仍关闭 multiple_images/mask。adaptive auto 因此发送一张 input.png：左侧是待编辑工作区，右侧为局部及完整角色参考，参考区域不参与最终资产。支持多图时 auto 顺序为 input.png → context.png → reference.png；也可选 local/full/both。request-plan.json 保存计划；只有实际请求才写 request.json，包含有序文件角色、SHA-256、尺寸、image 字段、mask 哈希、透明可编辑语义与是否发送。mask 不支持时明确标 local_only，不能宣称模型侧已受 mask 约束。
|
||||
|
||||
**图片解码与尺寸。** `providers/image-edits.mjs` 使用 multipart images/edits，支持非空 b64_json 或返回 URL 下载；空 b64_json 会回退有效 URL。当前部件请求显式发送 size=画布宽x高，并写入 request.json；配置 params.size 冲突会报错。strict 模式拒绝 CANVAS_SIZE_MISMATCH；uniform 仅将同宽高比整画布 LANCZOS 缩放到目标尺寸，保留原候选，不平移、不裁剪、不单独缩放物体，也不改变原图/mask。
|
||||
|
||||
**几何验收与合并。** `merge_candidate()` 去除边界连通白底，在原部件 Alpha≥240 的像素处检查 RGB 平均绝对误差≤30、可见保留率≥95%；无不透明锚点拒绝。只取允许区域内且与原部件连通的新增像素，没有有效新增即拒绝。通过后逐像素恢复原可见 RGBA，模型只提供新增部分。
|
||||
**几何验收与合并。** adaptive 在请求工作区的保留不透明像素处检查 RGB MAE≤30、保留率≥95%，先验对齐再逆采样，避免把逆采样边界 Alpha 变化误判成位移。返回前景只保留 editable 内且连接 preserve 的像素；清理/人工替换区域至少 95% 有效填充，否则拒绝。没有有效修改且没有人工移除则拒绝。最后恢复 preserve 的原 RGBA，已连接的可见缺口使用原图纹理，removed 清零。legacy 沿用原可见像素全保护规则。
|
||||
|
||||
**重点：验收仍有缺口。** `unfulfilled_regions` 记录未补到的子区域，但当前不是硬性拒绝条件；某一区域新增少量像素也不代表关节补够。语义、接缝、材质、运动露底没有自动验收。错误 SAM 像素会被原样恢复；若头部 mask 已含眼睛/胡子，当前规则也不能清除这些内部附件生成干净脸底。
|
||||
**能力证据与质量边界。** adaptive QA 记录合并前的 raw_outside_edit_mae / changed_pixels，避免本地恢复掩盖模型越界修改。capability-evidence.json 记录探针的输入顺序、尺寸、候选和 QA，但 reference_use_verified、mask_semantics_verified 不自动置 true，仍需视觉验证。区域填满不能证明生成的是正确底图；未明确划入 editable/remove 的误分仍受保护。接缝、材质、关节补足和运动露底尚无可靠自动验收。
|
||||
|
||||
## 6. 各个代码文件职责索引
|
||||
|
||||
@@ -143,6 +187,7 @@ node .\tools\model-check.mjs --stage completion `
|
||||
| [tools/model-check.mjs](../tools/model-check.mjs) | 部件补全与独立探针分发、参数互斥检查 |
|
||||
| [stages/completion-stage.mjs](../stages/completion-stage.mjs) | 输入指纹、准备、串行请求/离线读取、候选选择、状态留档 |
|
||||
| [completion_geometry.py](../completion_geometry.py) | allowed_regions、prepare、merge_candidate、finalize 与报告 |
|
||||
| [completion_planning.py](../completion_planning.py) | adaptive 需求重建、复核输入、参考板、正反变换、编辑分区与合并 |
|
||||
| [normalize_reference.py](../normalize_reference.py) | 候选 Alpha 保留、边界连通白底提取 |
|
||||
| [providers/image-edits.mjs](../providers/image-edits.mjs) | multipart 编辑请求、能力检查、Base64/URL 图片解码 |
|
||||
| [providers/http.mjs](../providers/http.mjs) | HTTP、超时、重试、诊断和脱敏 |
|
||||
@@ -153,6 +198,7 @@ node .\tools\model-check.mjs --stage completion `
|
||||
| [prompts/complete-v1.txt](../prompts/complete-v1.txt) | 保持姿势、尺寸、可见纹理与隐藏区域补全约束 |
|
||||
| [Tiantoken](../configs/tiantoken.json)、[DeepSeek/Qwen](../configs/deepseek-qwen.json)、[Astra/Qwen](../configs/astra-qwen.json) 配置 | 图片模型、认证变量、请求参数与能力开关 |
|
||||
| [tests/completion-stage.test.mjs](../tests/completion-stage.test.mjs) / [tests/test_completion.py](../tests/test_completion.py) | mock 编辑、离线复用、尺寸/对齐拒绝、像素保真与区域回归 |
|
||||
| [tests/test_completion_adaptive.py](../tests/test_completion_adaptive.py) | 放大/缩小回贴、漏分重建、附件清理、区域复核、拒绝位移和第四阶段契约 |
|
||||
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 提供方与探针相关回归 |
|
||||
|
||||
## 7. 产物详细说明
|
||||
@@ -160,34 +206,40 @@ node .\tools\model-check.mjs --stage completion `
|
||||
| 路径 | 内容与用途 |
|
||||
| --- | --- |
|
||||
| `source.png`、`semantic-parts.json`、`segmentation.json`、`masks/` | 输入快照，用于复现与离线候选身份核对 |
|
||||
| `completion-plan.json` | 准备任务、允许区域、参数和跳过原因 |
|
||||
| `tasks/<id>/input.png` | 白底局部可见部件，默认唯一模型输入图 |
|
||||
| `tasks/<id>/visible.png` | 原图可见 RGBA，合并时恢复像素 |
|
||||
| `tasks/<id>/context.png` | 同坐标局部原图参考，当前默认不发送 |
|
||||
| `tasks/<id>/edit-mask.png` | RGBA 编辑遮罩，透明处可编辑，当前默认不发送 |
|
||||
| `completion-plan.json` | 准备任务、区域、跳过原因；adaptive 含 assessments、重建需求、半径及歧义重叠 |
|
||||
| `tasks/<id>/input.png` | 实际请求主图；board 含目标、局部参考与全角色参考 |
|
||||
| `tasks/<id>/visible.png` | 请求坐标的保留 RGBA；legacy 为原可见 RGBA |
|
||||
| `tasks/<id>/context.png` / `reference.png` | 局部 / 全角色原图参考，由 reference mode 决定是否独立发送 |
|
||||
| `tasks/<id>/edit-mask.png` | RGBA 编辑遮罩，透明处为 editable 或 removed；是否发送以 request.json 为准 |
|
||||
| `tasks/<id>/allowed-mask.png` | 白色表示允许新增区域 |
|
||||
| `tasks/<id>/<补全字段>.png` | 启用的补全子区域 mask |
|
||||
| `tasks/<id>/task.json` / `prompt.txt` | origin、区域、需求和告警 / 实际提示词 |
|
||||
| `tasks/<id>/request.json` | 在线请求模型、图片哈希、是否发送 mask；修复后含 size |
|
||||
| `tasks/<id>/task.json` / `prompt.txt` | origin、区域、变换、参考布局、重建需求和告警 / 实际提示词 |
|
||||
| `tasks/<id>/native-*.png` | 原像素尺寸的源图、保留像素与各区域，用于逆映射和精确恢复 |
|
||||
| `preservation/<id>-*.png` | 原图全画布 preserve/editable/removed 契约，第四阶段核对用 |
|
||||
| `review-source.json`、`review.json`、`review/` | 可选原始复核文件、路径规范化快照及对应 masks |
|
||||
| `tasks/<id>/request-plan.json` / `request.json` | 计划 / 实际请求；记录有序图片角色、哈希、尺寸、mask 语义和发送状态 |
|
||||
| `tasks/<id>/capability-evidence.json` | 仅能力探针：请求接受情况及原始越界指标，能力需人工确认 |
|
||||
| `tasks/<id>/response.raw.json` / `request-metadata.json` | 在线响应留档 / 请求 ID、耗时等元数据 |
|
||||
| `tasks/<id>/response.error.json` | 可获取时保存的脱敏请求失败诊断 |
|
||||
| `tasks/<id>/candidate-N.*` | 模型或离线导入的原始候选，拒绝也保留 |
|
||||
| `tasks/<id>/merged-N-normalized.png` | uniform 实际重采样时保存的候选；不代表通过 |
|
||||
| `tasks/<id>/qa-N.json` | 尺寸、对齐、新增/丢弃像素、各区域完成情况、拒绝原因 |
|
||||
| `tasks/<id>/merged-N.png` | 通过几何检查且恢复原可见像素后的局部 RGBA |
|
||||
| `tasks/<id>/merged-N.png` | 原像素尺寸的局部 RGBA；adaptive 已逆采样并恢复 preserve |
|
||||
| `parts/<id>.png` | 每部件选中结果的紧裁剪透明 PNG |
|
||||
| `completion-result.json` | 每部件状态、候选 QA、原图 crop_xyxy 和路径 |
|
||||
| `completion-result.json` | 每部件状态、候选 QA、原图 crop_xyxy、preservation 路径和变换 |
|
||||
| `report.html` / `manifest.json` | 人工检查页 / 模式、网络标记、状态、输入/代码/产物哈希 |
|
||||
|
||||
parts/ 只含本次选中且通过检查的补全部件，不是全部 SAM 部件集合。无需补全、失败和未选中的部件不会自动复制进来；完整角色组装由第四阶段读取 SAM 与本目录后生成。
|
||||
|
||||
当前实验：`output/20260918-183532/complete2_gpt` 已真实请求 Tiantoken gpt-image-2 共 12 个部件。旧请求未传 size，返回 1248/1254 方形图，预期为 1024，全部严格尺寸拒绝。`complete2_gpt-size-recheck-v2` 离线复用原候选后，8 个几何通过；head、pants_thigh_l、pants_thigh_r、shoe_l 共 4 个因对齐或无有效新增拒绝。通过项中仍有衣料误分、错误手部和未补全子区域。显式 size 的请求构造已验证，但尚无修复后中转遵守 size 的新实测结论。
|
||||
|
||||
2026-09-22 adaptive 验证：Conda ai-spine 下 npm test 共 29 项通过（包括 8 项新 Python adaptive 用例），覆盖 board/独立参考顺序、mask mock、放大与缩小回贴、位移拒绝、实际孔洞、附件清理、人工编辑契约及第四阶段接续。`output/completion-adaptive-prepare-20260922` 使用既有 SAM 数据离线生成 16 个任务、11 个跳过，上衣规划 3708 个孔洞像素、头部规划 10205 个底图清理像素。它们是准备阶段的候选区域，不代表补全质量；本轮未请求付费模型，也未验证中转真实能力。
|
||||
|
||||
## 8. 当前阶段可改进点说明
|
||||
|
||||
- **完整角色组装已接入第四阶段。** 按原坐标与遮挡约束组装，复用原拆件并记录失败回退，输出补全前后对比、覆盖与重叠诊断及单骨骼旋转检查动画。可靠的接缝与运动质量自动验收仍待实现。
|
||||
- **把原图或部件相关原图区域交给补全模型参考。** 接入并实测全角色/局部上下文参考和编辑 mask，记录实际发送内容。不能仅把 capabilities 改为 true 就宣称中转支持；需验证图片顺序、mask 语义、返回尺寸和画面保持能力。
|
||||
- **根据 SAM 实际结果重建需求。** 区分分割缺口修复、遮挡补全、关节延伸、附件底图清理；对应建立保留区域与可编辑区域，避免误分像素被永久保护。
|
||||
- **改进画布与补全区域。** 固定 32 像素不适配所有部件；小部件在 1024 大白底中占比低。可验证按部件设计区域、显式局部放大并记录逆变换，保证回贴坐标可检查。
|
||||
- **参考输入已实现，真实能力待验证。** 支持 board 与独立 local/full/both、mask 发送策略、有序请求记录及单部件能力探针。离线 mock 已覆盖图片顺序、mask、返回尺寸和越界记录；本轮未请求付费模型，未宣称中转支持或生成质量提升。
|
||||
- **实际 mask 需求重建已实现。** 分别规划缺口、遮挡、关节和附件清理，保存 preserve/editable/removed，并支持人工复核区域。保守孔洞检测不会自动解决所有开放缺口、误分与语义歧义；这些仍需复核。
|
||||
- **自适应画布与区域已实现。** 半径随部件尺寸变化并可覆盖；小部件放大、大部件缩小，请求与原图间变换留档，最终恢复原像素尺寸和保护区。离线测试覆盖放大/缩小、Alpha 保真、位移拒绝及第四阶段接续；真实模型效果待小样验证。
|
||||
- **加强验收。** 对必需区域定义有效覆盖要求，增加部件语义、接缝、材质、回贴和关节旋转露底检查；不能放宽阈值来掩盖重绘、位移或错误输入。
|
||||
- **按小样验证后批量。** 先修正 SAM 输入，再用头、前臂、上衣验证参考图与遮罩，比较原输入、原候选、合并图和回贴图，确认改善后再跑全量。
|
||||
|
||||
@@ -69,9 +69,9 @@ conda run -n ai-spine node .\tools\verify-spine-runtime.mjs `
|
||||
|
||||
SAM 目录需含 `source.png`、有效 v2 `semantic-parts.json`、`segmentation.json`、其中记录的全画布二值 masks。可见部件缺少 mask 直接报错；hidden 不得有可见 mask。
|
||||
|
||||
补全目录需含相同输入快照、对应 masks 和 `completion-result.json`。只有 `selected_file` 非空、`qa.accepted=true` 且 `status=geometry_passed_needs_review` 的结果会被采用；其 `image` 文件必须存在，裁剪坐标为原画布内的整数半开区间，尺寸与 PNG 一致，可见像素必须逐像素匹配 SAM 源图。输入哈希不一致时拒绝混用。图片路径必须位于相应输入目录内。
|
||||
补全目录需含相同输入快照、对应 masks 和 `completion-result.json`。只有 `selected_file` 非空、`qa.accepted=true` 且 `status=geometry_passed_needs_review` 的结果会被采用；其 `image` 文件必须存在，裁剪坐标为原画布内的整数半开区间，尺寸与 PNG 一致。legacy 可见像素必须逐像素匹配 SAM；adaptive 使用 preservation 的 preserve/editable/removed 全画布二值区域，要求 preserve=SAM−editable−removed、编辑与移除不重叠、保留像素精确匹配、输出没有超出保留/编辑区域。输入哈希不一致时拒绝混用。图片和契约路径必须位于补全目录内。
|
||||
|
||||
补全失败、未处理、未提供或 `NO_ALLOWED_EXTENSION` 的需补全部件遵守 fallback-policy。hidden 保留语义骨骼但不生成纹理，也不因缺纹理触发严格回退错误。
|
||||
补全失败、未处理、未提供或没有允许区域的需补全部件遵守 fallback-policy；adaptive 已重建并尝试修复的部件也视为有需求，不因原计划 needs_completion=false 就忽略失败。hidden 保留语义骨骼但不生成纹理，也不因缺纹理触发严格回退错误。
|
||||
|
||||
可选绘制顺序文件格式为 `{"draw_order":["部件ID一","部件ID二"]}`。必须列出本次全部可绘制部件，不包含 hidden。显式顺序允许覆盖模型遮挡关系，冲突会在报告中列出。
|
||||
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
Complete only the hidden portion of the specified character part on the first
|
||||
input image (the local canvas). Any subsequent images are context references.
|
||||
Repair and complete the specified character part on the first input image.
|
||||
The task declares editable regions for segmentation gaps, occlusion, joint
|
||||
extensions, and removal of overlaid attachments from their underlying base.
|
||||
Only regions explicitly marked editable may be changed. Preserve all other
|
||||
visible pixels. Any subsequent images or labeled reference panels are context.
|
||||
Preserve the exact original pose, scale, screen direction, perspective, colors,
|
||||
line thickness and shading. Do not mirror, rotate, resize, recenter or redesign.
|
||||
Keep the canvas dimensions and the position of every already-visible feature.
|
||||
@@ -8,5 +11,6 @@ body parts, props, labels, annotations or extra versions of the same part.
|
||||
If an edit mask is supplied, transparent mask pixels are editable. Otherwise,
|
||||
use the task's bounded extension instructions and keep the remaining background
|
||||
pure white or transparent. Return the same canvas size without recentering.
|
||||
Downstream checks will reject misalignment, restrict additions to allowed regions,
|
||||
and restore the source visible pixels. Do not rely on restoration to hide changes.
|
||||
Downstream checks reject misalignment and constrain modifications to edit regions,
|
||||
restore protected source pixels, and reverse the documented canvas transform.
|
||||
Do not rely on restoration to hide movement or redesign.
|
||||
|
||||
+21
-2
@@ -130,7 +130,24 @@ def load_assets(input_dir, completion_dir=None, fallback_policy='sam', draw_orde
|
||||
raise ValueError(f'Invalid completion crop/size: {name}')
|
||||
full = Image.new('RGBA', source.size)
|
||||
full.paste(image, (box[0], box[1]))
|
||||
if not np.array_equal(np.array(full)[mask], rgba[mask]):
|
||||
preserve = mask
|
||||
preservation = candidate.get('preservation')
|
||||
if preservation:
|
||||
areas = {}
|
||||
for key in ['preserve_mask', 'editable_mask', 'removed_mask']:
|
||||
area = np.array(Image.open(tracked(completion, preservation.get(key), 'completion')))
|
||||
if area.shape != (h, w) or not np.isin(area, [0, 255]).all():
|
||||
raise ValueError(f'Invalid completion preservation mask: {name}/{key}')
|
||||
areas[key] = area > 0
|
||||
preserve, editable, removed = (areas[k] for k in ['preserve_mask', 'editable_mask', 'removed_mask'])
|
||||
if (np.any(editable & removed) or np.any(removed & ~mask) or
|
||||
not np.array_equal(preserve, mask & ~editable & ~removed) or not preserve.any()):
|
||||
raise ValueError(f'Inconsistent completion preservation partition: {name}')
|
||||
alpha = np.array(full)[:, :, 3] > 0
|
||||
if np.any(alpha & ~(preserve | editable)):
|
||||
raise ValueError(f'Completion pixels outside preservation/edit contract: {name}')
|
||||
warnings.append('VISIBLE_EDITS_DECLARED_NEEDS_REVIEW')
|
||||
if not np.array_equal(np.array(full)[preserve], rgba[preserve]):
|
||||
raise ValueError(f'Completion does not preserve original visible RGBA: {name}')
|
||||
origin, reason = 'completion', 'GEOMETRY_PASSED_NEEDS_REVIEW'
|
||||
if candidate.get('qa', {}).get('unfulfilled_regions'):
|
||||
@@ -138,7 +155,8 @@ def load_assets(input_dir, completion_dir=None, fallback_policy='sam', draw_orde
|
||||
else:
|
||||
image, box = original, original_box
|
||||
origin = 'sam'
|
||||
if not part['needs_completion']:
|
||||
inferred_need = bool(candidate) or skipped.get(name, {}).get('reason') == 'AMBIGUOUS_MASK_REQUIRES_REVIEW'
|
||||
if not part['needs_completion'] and not inferred_need:
|
||||
reason = 'NO_COMPLETION_NEEDED'
|
||||
else:
|
||||
reason = skipped.get(name, {}).get('reason') or (
|
||||
@@ -151,6 +169,7 @@ def load_assets(input_dir, completion_dir=None, fallback_policy='sam', draw_orde
|
||||
'crop_xyxy': box, 'original_crop_xyxy': original_box,
|
||||
'semantic_verified': False, 'warnings': warnings,
|
||||
'completion_status': candidate.get('status'),
|
||||
'preservation': candidate.get('preservation'),
|
||||
'completion_qa': candidate.get('qa')})
|
||||
|
||||
if not images:
|
||||
|
||||
+96
-15
@@ -23,14 +23,34 @@ function filesIn(dir, prefix = '') {
|
||||
}
|
||||
|
||||
export async function runCompletion({stage, inputDir, outDir, partIds = [], prepareOnly = false,
|
||||
candidatesDir, completionSize = 1024, extensionRadius = 32, python: requestedPython,
|
||||
resume = false, dryRun = false, candidateSizePolicy = 'strict'}, transport) {
|
||||
candidatesDir, completionSize = 1024, extensionRadius, python: requestedPython,
|
||||
resume = false, dryRun = false, candidateSizePolicy = 'strict', completionProfile = 'adaptive',
|
||||
completionReference = 'auto', completionMask = 'auto', completionReview, probeEditCapabilities = false}, transport) {
|
||||
if (!inputDir) throw new Error('部件补全需要 --input-dir 指向 SAM 输出目录');
|
||||
if (!outDir && !dryRun) throw new Error('部件补全需要独立的 --out-dir');
|
||||
if (prepareOnly && candidatesDir) throw new Error('--prepare-only 与 --candidates-dir 不能同时使用');
|
||||
const size = Number(completionSize), radius = Number(extensionRadius);
|
||||
const size = Number(completionSize), radius = extensionRadius === undefined
|
||||
? completionProfile === 'legacy' ? 32 : null : Number(extensionRadius);
|
||||
if (!['legacy', 'adaptive'].includes(completionProfile)) throw new Error('--completion-profile 仅支持 adaptive / legacy');
|
||||
if (!['auto', 'none', 'local', 'full', 'both', 'board'].includes(completionReference)) throw new Error('不支持的 --completion-reference');
|
||||
if (!['auto', 'required', 'off'].includes(completionMask)) throw new Error('--completion-mask 仅支持 auto / required / off');
|
||||
if (probeEditCapabilities && (partIds.length !== 1 || prepareOnly || candidatesDir || completionProfile !== 'adaptive')) {
|
||||
throw new Error('--probe-edit-capabilities 需要 adaptive、恰好一个 --part 和在线请求');
|
||||
}
|
||||
const referenceMode = probeEditCapabilities ? 'both' : completionReference === 'auto'
|
||||
? completionProfile === 'legacy' ? stage.capabilities.multiple_images ? 'local' : 'none'
|
||||
: stage.capabilities.multiple_images ? 'both' : 'board' : completionReference;
|
||||
if (completionProfile === 'legacy' && (completionReview || !['local', 'none'].includes(referenceMode))) {
|
||||
throw new Error('legacy 不支持 review 或 full/both/board 参考，请使用 adaptive');
|
||||
}
|
||||
const nativeReferences = ['local', 'full', 'both'].includes(referenceMode);
|
||||
const maskSent = probeEditCapabilities || completionMask === 'required' || (completionMask === 'auto' && stage.capabilities.mask);
|
||||
const capabilitiesAvailable = (!nativeReferences || stage.capabilities.multiple_images) && (!maskSent || stage.capabilities.mask);
|
||||
if (!capabilitiesAvailable && !probeEditCapabilities && !prepareOnly && !candidatesDir && !dryRun) {
|
||||
throw new Error('请求需要未确认的多图/mask 能力；可先 prepare-only，或用单部件 --probe-edit-capabilities 实测，不要直接更改默认 capabilities');
|
||||
}
|
||||
if (!['strict', 'uniform'].includes(candidateSizePolicy)) throw new Error('--candidate-size-policy 仅支持 strict / uniform');
|
||||
if (!Number.isInteger(size) || size < 128 || size > 4096 || !Number.isInteger(radius) || radius < 1 || radius > 256) {
|
||||
if (!Number.isInteger(size) || size < 128 || size > 4096 || (radius !== null && (!Number.isInteger(radius) || radius < 1 || radius > 256))) {
|
||||
throw new Error('--completion-size 必须为 128～4096 的整数；--extension-radius 必须为 1～256 的整数');
|
||||
}
|
||||
const input = path.resolve(inputDir);
|
||||
@@ -46,15 +66,43 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
|
||||
const source = readImage(path.join(input, 'source.png'), {pngOnly: true});
|
||||
const inputFiles = ['source.png', 'semantic-parts.json', 'segmentation.json', ...segmentation.parts.map((p) => `masks/${p.id}.png`)];
|
||||
const inputs = Object.fromEntries(inputFiles.map((file) => [file, sha256(readFileSync(path.join(input, file)))]));
|
||||
const reviewFiles = [], reviewHashes = {};
|
||||
let reviewData;
|
||||
if (completionReview) {
|
||||
const file = path.resolve(completionReview);
|
||||
reviewData = readJson(file);
|
||||
if (reviewData.schema_version !== 1 || !reviewData.parts || Array.isArray(reviewData.parts)) throw new Error('review 需要 schema_version=1 和 parts 对象');
|
||||
reviewHashes['review.json'] = sha256(readFileSync(file));
|
||||
reviewFiles.push({file, target: 'review-source.json'});
|
||||
for (const [id, item] of Object.entries(reviewData.parts)) {
|
||||
if (!names.has(id) || !item || item.confirmed !== true) throw new Error('review 部件必须有效且 confirmed=true');
|
||||
for (const field of ['repair_mask', 'replace_mask', 'remove_mask']) {
|
||||
if (!Object.hasOwn(item, field)) continue;
|
||||
const ref = item[field];
|
||||
if (typeof ref !== 'string' || path.isAbsolute(ref)) throw new Error('review mask 必须是相对路径');
|
||||
const maskFile = path.resolve(path.dirname(file), ref), relative = path.relative(path.dirname(file), maskFile);
|
||||
if (relative === '..' || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative)) throw new Error('review mask 不能越出 review 目录');
|
||||
const target = `review/${id}-${field}.png`;
|
||||
reviewHashes[`${id}/${field}`] = sha256(readFileSync(maskFile));
|
||||
reviewFiles.push({file: maskFile, target});
|
||||
item[field] = target;
|
||||
}
|
||||
}
|
||||
}
|
||||
const planning = {profile: completionProfile, reference_mode: referenceMode, review_hashes: reviewHashes,
|
||||
extension_radius: radius, planner_version: completionProfile === 'adaptive' ? 2 : 1};
|
||||
const imported = {};
|
||||
if (candidatesDir) {
|
||||
const priorManifest = path.resolve(candidatesDir, 'manifest.json');
|
||||
const previousRun = existsSync(priorManifest) ? readJson(priorManifest) : null;
|
||||
const previousPlanning = previousRun?.planning || {profile: 'legacy', reference_mode: previousRun?.resolved_stage?.capabilities?.multiple_images ? 'local' : 'none',
|
||||
review_hashes: {}, extension_radius: previousRun?.extension_radius, planner_version: 1};
|
||||
if (previousRun && (previousRun.output_kind !== 'bounded_part_completion' || previousRun.completion_size !== size || previousRun.extension_radius !== radius ||
|
||||
JSON.stringify(previousPlanning) !== JSON.stringify(planning) ||
|
||||
Object.entries(inputs).some(([file, hash]) => previousRun.inputs?.[file] !== hash))) {
|
||||
throw new Error('旧补全目录的输入或画布/延展参数不一致，不能复用其候选');
|
||||
}
|
||||
for (const p of plan.parts.filter((p) => p.needs_completion && (!partIds.length || partIds.includes(p.id)))) {
|
||||
for (const p of plan.parts.filter((p) => (completionProfile === 'adaptive' || p.needs_completion) && (!partIds.length || partIds.includes(p.id)))) {
|
||||
const file = previousRun
|
||||
? ['png', 'jpg', 'webp'].map((ext) => path.resolve(candidatesDir, 'tasks', p.id, `candidate-1.${ext}`)).find(existsSync)
|
||||
: path.resolve(candidatesDir, `${p.id}.png`);
|
||||
@@ -66,16 +114,18 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
|
||||
if (dryRun) {
|
||||
console.log(JSON.stringify({stage: 'completion', input, width: source.width, height: source.height,
|
||||
selected_parts: partIds, completion_size: size, extension_radius: radius, candidate_size_policy: candidateSizePolicy,
|
||||
planning, mask_requested: maskSent, capabilities_available: capabilitiesAvailable,
|
||||
mode: prepareOnly ? 'prepare' : candidatesDir ? 'offline_candidates' : 'image_edit', network_requested: false}, null, 2));
|
||||
return {dryRun: true};
|
||||
}
|
||||
const codeFiles = ['stages/completion-stage.mjs', 'completion_geometry.py', 'normalize_reference.py',
|
||||
const codeFiles = ['stages/completion-stage.mjs', 'completion_geometry.py', 'completion_planning.py', 'normalize_reference.py',
|
||||
'lib/config.mjs', 'lib/python-runtime.mjs', 'lib/run-artifacts.mjs', 'lib/images.mjs',
|
||||
'lib/validate-parts.mjs', 'schemas/semantic-parts.schema.json', 'providers/image-edits.mjs', 'providers/http.mjs'];
|
||||
const identity = {schema_version: 1, stage: 'completion', output_kind: 'bounded_part_completion',
|
||||
mode: prepareOnly ? 'prepare' : candidatesDir ? 'offline_candidates' : 'image_edit',
|
||||
resolved_stage: publicStage(stage), endpoint_sha256: sha256(stage.baseUrl), prompt_sha256: sha256(stage.prompt),
|
||||
inputs, part_ids: partIds, completion_size: size, extension_radius: radius, candidate_size_policy: candidateSizePolicy,
|
||||
planning, mask_requested: maskSent, capability_probe: probeEditCapabilities,
|
||||
imported_candidates: Object.fromEntries(Object.entries(imported).map(([id, value]) => [id, value.sha256])),
|
||||
code_sha256: sha256(codeFiles.map((file) => `${file}\n${readFileSync(path.join(projectRoot, file), 'utf8')}`).join('\n'))};
|
||||
const fingerprint = sha256(JSON.stringify(identity));
|
||||
@@ -83,24 +133,51 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
|
||||
if (run.reused) return {out: run.out, reused: true};
|
||||
const manifest = {...identity, fingerprint, status: 'running', quality_status: 'pending', started_at: new Date().toISOString(), network_requested: false};
|
||||
jsonFile(run.manifestPath, manifest, stage.apiKey);
|
||||
const result = {schema_version: 1, quality_status: 'needs_review', parts: [], skipped: []};
|
||||
const result = {schema_version: completionProfile === 'adaptive' ? 2 : 1, profile: completionProfile,
|
||||
quality_status: 'needs_review', parts: [], skipped: []};
|
||||
try {
|
||||
const requestSize = `${size}x${size}`;
|
||||
if (!prepareOnly && !candidatesDir && stage.params.size !== undefined && stage.params.size !== requestSize) {
|
||||
throw new Error(`completion.params.size 必须与局部画布一致：${requestSize}`);
|
||||
}
|
||||
const editStage = {...stage, params: {...stage.params, size: requestSize}};
|
||||
const editStage = {...stage, params: {...stage.params, size: requestSize},
|
||||
...(probeEditCapabilities ? {capabilities: {multiple_images: true, mask: true}, maxRetries: 0} : {})};
|
||||
if (probeEditCapabilities) editStage.params.n = 1;
|
||||
for (const file of inputFiles) {
|
||||
mkdirSync(path.dirname(path.join(run.out, file)), {recursive: true});
|
||||
copyFileSync(path.join(input, file), path.join(run.out, file));
|
||||
}
|
||||
pythonCall(python, ['prepare', '--out', run.out, '--size', String(size), '--radius', String(radius), ...partIds.flatMap((id) => ['--part', id])]);
|
||||
for (const {file, target} of reviewFiles) {
|
||||
mkdirSync(path.dirname(path.join(run.out, target)), {recursive: true});
|
||||
copyFileSync(file, path.join(run.out, target));
|
||||
}
|
||||
if (reviewData) jsonFile(path.join(run.out, 'review.json'), reviewData);
|
||||
pythonCall(python, ['prepare', '--out', run.out, '--size', String(size), '--profile', completionProfile,
|
||||
'--reference-mode', referenceMode, ...(radius === null ? [] : ['--radius', String(radius)]),
|
||||
...(reviewData ? ['--review-file', path.join(run.out, 'review.json')] : []), ...partIds.flatMap((id) => ['--part', id])]);
|
||||
const prepared = readJson(path.join(run.out, 'completion-plan.json'));
|
||||
result.skipped = prepared.skipped;
|
||||
result.parts = prepared.tasks.map((task) => ({id: task.id, status: 'prepared', upstream_warnings: task.warnings, semantic_verified: false}));
|
||||
if (probeEditCapabilities && prepared.tasks.length !== 1) throw new Error('所选部件没有可执行任务；请检查 completion-plan.json，不会发送能力探针');
|
||||
for (const [index, task] of prepared.tasks.entries()) {
|
||||
const folder = path.join(run.out, 'tasks', task.id);
|
||||
const prompt = `${stage.prompt}\n\nPart: ${task.id}: ${task.semantic}\nCompletion requirements: ${JSON.stringify(task.completion)}\nCanvas: ${size} x ${size} pixels. Keep the part at its current position and native scale.\nExtend only the hidden contour and requested joint connections, by at most ${radius} pixels from the visible silhouette. Keep all other background pure white or transparent. Do not add the occluder or neighboring body parts.\n${stage.capabilities.multiple_images ? 'The second image is local context only; do not copy its other parts into the output.' : 'Only the isolated visible part is provided.'}\nReturn exactly the same canvas dimensions.\n`;
|
||||
const imageRoles = [{file: 'input.png', role: referenceMode === 'board' ? 'target_with_reference_panels' : 'target'}];
|
||||
if (['local', 'both'].includes(referenceMode)) imageRoles.push({file: 'context.png', role: 'local_reference'});
|
||||
if (['full', 'both'].includes(referenceMode)) imageRoles.push({file: 'reference.png', role: 'full_character_reference'});
|
||||
const images = imageRoles.map(({file}) => readImage(path.join(folder, file), {pngOnly: true}));
|
||||
const mask = maskSent ? readImage(path.join(folder, 'edit-mask.png'), {pngOnly: true}) : null;
|
||||
const requestPlan = {model: stage.model, size: requestSize, reference_mode: referenceMode,
|
||||
images: images.map((im) => im.sha256), image_roles: images.map((im, i) => ({...imageRoles[i], index: i, sha256: im.sha256, width: im.width, height: im.height})),
|
||||
image_field: stage.imageField, mask_editable: mask ? stage.maskEditable : null,
|
||||
edit_mask_sent: Boolean(mask), mask_sha256: mask?.sha256 ?? null,
|
||||
mask_enforcement: mask ? 'provider_requested_and_local' : 'local_only',
|
||||
capability_status: probeEditCapabilities ? 'unverified_probe' : 'configured_not_proven_by_this_run',
|
||||
allowed_mask_sha256: sha256(readFileSync(path.join(folder, 'allowed-mask.png')))};
|
||||
jsonFile(path.join(folder, 'request-plan.json'), {...requestPlan, request_sent: false}, stage.apiKey);
|
||||
const referenceInstruction = referenceMode === 'board'
|
||||
? `The LEFT work panel ${JSON.stringify(task.transform.working_xyxy)} is the only output target. RIGHT panels are LOCAL and FULL character references. Keep the entire ${size}x${size} board layout unchanged; never place reference characters in the target panel. Preserve reference panels.`
|
||||
: `Image order: ${imageRoles.map((im, i) => `${i + 1}=${im.role}`).join(', ')}. Only edit the first image. Subsequent images are references, never extra output parts.`;
|
||||
const prompt = `${stage.prompt}\n\nPart: ${task.id}: ${task.semantic}\nCompletion requirements: ${JSON.stringify(task.completion)}\nCanvas: ${size} x ${size} pixels. Preserve the target's exact current position and displayed scale.\nTask regions: ${JSON.stringify(task.regions)}. segmentation_gap: restore source texture missing from segmentation; attachment_cleanup: paint the underlying material WITHOUT the overlaid attachment; reviewed_replacement: repair incorrect pixels; joint/occlusion regions: extend this part only.\n${referenceInstruction}\n${mask ? 'Transparent mask pixels are editable; opaque mask pixels must remain unchanged.' : 'No provider mask is sent. Follow the bounded edit instructions; output is constrained locally afterwards.'}\nReturn exactly the same canvas dimensions.\n`;
|
||||
const fullPrompt = `${prompt}\nOccluded by: ${task.occluded_by.join(', ') || 'not specified'}.\nLocal pixel bounds for proposed extensions (xyxy, exclusive upper bounds): ${JSON.stringify(task.region_bounds_xyxy)}. Null means no supported region for that requirement. These boxes bound the allowed regions; do not fill whole rectangles.\n`;
|
||||
writeFileSync(path.join(folder, 'prompt.txt'), fullPrompt);
|
||||
if (prepareOnly) continue;
|
||||
@@ -111,11 +188,7 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
|
||||
if (!imported[task.id]) { record.status = 'missing_candidate'; continue; }
|
||||
outputs = [{bytes: readFileSync(imported[task.id].file), extension: path.extname(imported[task.id].file).slice(1)}];
|
||||
} else {
|
||||
const images = [readImage(path.join(folder, 'input.png'), {pngOnly: true})];
|
||||
if (stage.capabilities.multiple_images) images.push(readImage(path.join(folder, 'context.png'), {pngOnly: true}));
|
||||
const mask = stage.capabilities.mask ? readImage(path.join(folder, 'edit-mask.png'), {pngOnly: true}) : null;
|
||||
jsonFile(path.join(folder, 'request.json'), {model: stage.model, size: requestSize, images: images.map((im) => im.sha256),
|
||||
edit_mask_sent: Boolean(mask), allowed_mask_sha256: sha256(readFileSync(path.join(folder, 'allowed-mask.png')))}, stage.apiKey);
|
||||
jsonFile(path.join(folder, 'request.json'), {...requestPlan, request_sent: true}, stage.apiKey);
|
||||
manifest.network_requested = true;
|
||||
jsonFile(run.manifestPath, manifest, stage.apiKey);
|
||||
try {
|
||||
@@ -126,6 +199,10 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
|
||||
} catch (error) {
|
||||
record.status = 'request_failed';
|
||||
if (error.diagnostics) jsonFile(path.join(folder, 'response.error.json'), error.diagnostics, stage.apiKey);
|
||||
if (probeEditCapabilities) jsonFile(path.join(folder, 'capability-evidence.json'), {
|
||||
request_accepted: false, request: requestPlan, error: error.message,
|
||||
reference_use_verified: false, mask_semantics_verified: false,
|
||||
verdict: 'request_failed_no_capability_promotion'}, stage.apiKey);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -145,6 +222,10 @@ export async function runCompletion({stage, inputDir, outDir, partIds = [], prep
|
||||
record.status = record.selected_file ? 'geometry_passed_needs_review' : 'rejected';
|
||||
if (!record.qa) record.qa = {...record.candidates[0]?.qa, accepted: false,
|
||||
reasons: [...new Set(record.candidates.flatMap((c) => c.qa.reasons))]};
|
||||
if (probeEditCapabilities) jsonFile(path.join(folder, 'capability-evidence.json'), {
|
||||
request_accepted: true, request: requestPlan, candidates: record.candidates.map((c) => c.qa),
|
||||
reference_use_verified: false, mask_semantics_verified: false,
|
||||
verdict: 'needs_visual_review_no_automatic_capability_promotion'}, stage.apiKey);
|
||||
jsonFile(path.join(run.out, 'completion-result.json'), result, stage.apiKey);
|
||||
}
|
||||
manifest.quality_status = prepareOnly ? 'prepared_needs_review' : 'needs_review';
|
||||
|
||||
@@ -31,7 +31,7 @@ test('completion geometry rejects moved/empty outputs and preserves original alp
|
||||
|
||||
test('part completion prepares without HTTP, sends declared capabilities, merges, resumes, and detects changed masks', async (t) => {
|
||||
const {root, input, stage} = fixture(t);
|
||||
const opts = {stage, inputDir: input, outDir: path.join(root, 'run'), completionSize: 128, extensionRadius: 16, partIds: ['torso']};
|
||||
const opts = {stage, inputDir: input, outDir: path.join(root, 'run'), completionProfile: 'legacy', completionSize: 128, extensionRadius: 16, partIds: ['torso']};
|
||||
let requests = 0;
|
||||
const transport = {fetchImpl: async (_url, request) => {
|
||||
requests++;
|
||||
@@ -66,7 +66,7 @@ test('offline candidate import needs no HTTP; mismatched candidate is retained a
|
||||
const candidates = path.join(root, 'candidates'); mkdirSync(candidates);
|
||||
writeFileSync(path.join(candidates, 'torso.png'), readFileSync(path.join(input, 'source.png')));
|
||||
const outDir = path.join(root, 'run');
|
||||
const result = await runCompletion({stage, inputDir: input, outDir, candidatesDir: candidates, completionSize: 128}, {fetchImpl: () => { throw new Error('Must remain offline'); }});
|
||||
const result = await runCompletion({stage, inputDir: input, outDir, candidatesDir: candidates, completionProfile: 'legacy', completionSize: 128}, {fetchImpl: () => { throw new Error('Must remain offline'); }});
|
||||
assert.equal(result.rejected, 1);
|
||||
assert.equal(result.accepted, 0);
|
||||
const data = JSON.parse(readFileSync(path.join(outDir, 'completion-result.json')));
|
||||
@@ -81,7 +81,7 @@ test('provider mask/context capabilities are explicit and failure preserves diag
|
||||
stage.capabilities = {mask: true, multiple_images: true};
|
||||
stage.maxRetries = 0;
|
||||
const outDir = path.join(root, 'run');
|
||||
await assert.rejects(runCompletion({stage, inputDir: input, outDir, completionSize: 128}, {fetchImpl: async (_url, request) => {
|
||||
await assert.rejects(runCompletion({stage, inputDir: input, outDir, completionProfile: 'legacy', completionSize: 128}, {fetchImpl: async (_url, request) => {
|
||||
assert.equal(request.body.getAll('image').length, 2);
|
||||
assert.ok(request.body.get('mask'));
|
||||
return new Response(JSON.stringify({error: {message: 'test-secret failed'}}), {status: 400});
|
||||
@@ -98,7 +98,7 @@ test('completion CLI prepares without credentials and rejects mixed probe argume
|
||||
const envFile = path.join(root, 'empty.env'); writeFileSync(envFile, '');
|
||||
const outDir = path.join(root, 'cli');
|
||||
const args = ['tools/model-check.mjs', '--stage', 'completion', '--input-dir', input,
|
||||
'--out-dir', outDir, '--prepare-only', '--part', 'torso', '--completion-size', '128', '--env-file', envFile];
|
||||
'--out-dir', outDir, '--prepare-only', '--part', 'torso', '--completion-profile', 'legacy', '--completion-size', '128', '--env-file', envFile];
|
||||
const options = {cwd: projectRoot, windowsHide: true, encoding: 'utf8',
|
||||
env: {...process.env, VECTOR_ENGINE_BASE_URL: '', VECTOR_ENGINE_API_KEY: ''}, stdio: ['ignore', 'pipe', 'pipe']};
|
||||
assert.match(execFileSync(process.execPath, args, options), /未请求模型/u);
|
||||
@@ -107,3 +107,62 @@ test('completion CLI prepares without credentials and rejects mixed probe argume
|
||||
assert.equal(manifest.prepared_count, 1);
|
||||
assert.throws(() => execFileSync(process.execPath, [...args, '--image', path.join(input, 'source.png')], options), (error) => /不接受探针输入/u.test(error.stderr));
|
||||
});
|
||||
|
||||
test('adaptive geometry: reversible scaling, real-mask replanning, cleanup, review masks and Spine contract', () => {
|
||||
assert.equal(execFileSync(python, ['-m', 'unittest', 'discover', '-s', 'tests', '-p', 'test_completion_adaptive.py'],
|
||||
{cwd: projectRoot, windowsHide: true, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe']}), '');
|
||||
});
|
||||
|
||||
function adaptiveResponse(folder) {
|
||||
execFileSync(python, ['-c', 'import sys; sys.path.insert(0, "tests"); from test_completion_adaptive import mock_candidate; mock_candidate(sys.argv[1])', folder],
|
||||
{cwd: projectRoot, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe']});
|
||||
return new Response(JSON.stringify({data: [{b64_json: readFileSync(path.join(folder, 'test-candidate.png')).toString('base64')}]}),
|
||||
{headers: {'Content-Type': 'application/json'}});
|
||||
}
|
||||
|
||||
test('adaptive default sends a single reference board and keeps native coordinates; planning changes invalidate candidates', async (t) => {
|
||||
const {root, input, stage} = fixture(t);
|
||||
const outDir = path.join(root, 'adaptive');
|
||||
const opts = {stage, inputDir: input, outDir, partIds: ['torso'], completionSize: 256, extensionRadius: 16};
|
||||
const result = await runCompletion(opts, {fetchImpl: async (_url, request) => {
|
||||
assert.equal(request.body.getAll('image').length, 1);
|
||||
assert.equal(request.body.has('mask'), false);
|
||||
assert.match(request.body.get('prompt'), /RIGHT panels/u);
|
||||
return adaptiveResponse(path.join(outDir, 'tasks/torso'));
|
||||
}});
|
||||
assert.equal(result.accepted, 1);
|
||||
const record = JSON.parse(readFileSync(path.join(outDir, 'completion-result.json'))).parts[0];
|
||||
assert.deepEqual(record.crop_xyxy, [20, 20, 50, 40]);
|
||||
assert.ok(record.preservation.preserve_mask);
|
||||
const request = JSON.parse(readFileSync(path.join(outDir, 'tasks/torso/request.json')));
|
||||
assert.equal(request.reference_mode, 'board');
|
||||
assert.equal(request.image_roles[0].role, 'target_with_reference_panels');
|
||||
assert.equal(request.mask_enforcement, 'local_only');
|
||||
assert.equal((await runCompletion({...opts, resume: true})).reused, true);
|
||||
const recovered = await runCompletion({...opts, outDir: path.join(root, 'adaptive-recheck'), candidatesDir: outDir});
|
||||
assert.equal(recovered.accepted, 1);
|
||||
await assert.rejects(runCompletion({...opts, outDir: path.join(root, 'bad-recheck'), candidatesDir: outDir, completionReference: 'none'}), /画布/u);
|
||||
});
|
||||
|
||||
test('native full/local references and mask preserve order and hashes; probe never promotes capabilities', async (t) => {
|
||||
const {root, input, stage} = fixture(t);
|
||||
const opts = {stage, inputDir: input, outDir: path.join(root, 'probe'), partIds: ['torso'], completionSize: 256};
|
||||
await assert.rejects(runCompletion({...opts, completionReference: 'both', completionMask: 'required'}), /未确认/u);
|
||||
await assert.rejects(runCompletion({...opts, partIds: [], probeEditCapabilities: true}), /恰好一个/u);
|
||||
await runCompletion({...opts, probeEditCapabilities: true}, {fetchImpl: async (_url, request) => {
|
||||
const images = request.body.getAll('image');
|
||||
assert.deepEqual(images.map((im) => im.name), ['input.png', 'context.png', 'reference.png']);
|
||||
assert.equal(request.body.get('mask').name, 'edit-mask.png');
|
||||
assert.equal(request.body.get('n'), '1');
|
||||
return adaptiveResponse(path.join(opts.outDir, 'tasks/torso'));
|
||||
}});
|
||||
assert.deepEqual(stage.capabilities, {multiple_images: false, mask: false});
|
||||
const evidence = JSON.parse(readFileSync(path.join(opts.outDir, 'tasks/torso/capability-evidence.json')));
|
||||
assert.equal(evidence.request_accepted, true);
|
||||
assert.equal(evidence.mask_semantics_verified, false);
|
||||
assert.equal(evidence.reference_use_verified, false);
|
||||
assert.equal(evidence.request.image_roles.length, 3);
|
||||
assert.ok(evidence.request.mask_sha256);
|
||||
const qa = JSON.parse(readFileSync(path.join(opts.outDir, 'tasks/torso/qa-1.json')));
|
||||
assert.equal(typeof qa.raw_outside_edit_changed_pixels, 'number');
|
||||
});
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
import copy
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from completion_geometry import localize, merge_candidate, finalize
|
||||
from completion_planning import prepare_adaptive, read, write, review_inputs
|
||||
from spine_assembly import load_assets
|
||||
from test_completion import make_fixture
|
||||
|
||||
|
||||
def candidate_for(folder, native):
|
||||
"""Independent fixture: insert known native ground truth into documented work rectangle."""
|
||||
folder = Path(folder)
|
||||
task = read(folder / 'task.json')
|
||||
rect = task['transform']['working_xyxy']
|
||||
canvas = Image.open(folder / 'input.png').convert('RGBA')
|
||||
content = Image.fromarray(native).resize((rect[2]-rect[0], rect[3]-rect[1]), Image.Resampling.NEAREST)
|
||||
canvas.paste(content, (rect[0], rect[1]))
|
||||
target = folder / 'test-candidate.png'
|
||||
canvas.save(target)
|
||||
return target
|
||||
|
||||
|
||||
def mock_candidate(folder):
|
||||
folder = Path(folder)
|
||||
native = np.array(Image.open(folder / 'native-visible.png'))
|
||||
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
|
||||
native[edit] = [180, 50, 20, 255]
|
||||
return candidate_for(folder, native)
|
||||
|
||||
|
||||
class AdaptiveTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.source = make_fixture(self.root)
|
||||
|
||||
def task(self, mode='board', size=256, review=None):
|
||||
plan = prepare_adaptive(self.root, ['torso'], size, 16, mode, review)
|
||||
return plan['tasks'][0], self.root / 'tasks' / 'torso'
|
||||
|
||||
def test_small_part_scales_up_roundtrip_and_reference_panels_are_excluded(self):
|
||||
task, folder = self.task()
|
||||
self.assertGreater(task['transform']['scale'], 1)
|
||||
self.assertEqual(len(task['reference_panel_boxes']), 2)
|
||||
forward = task['transform']['source_to_canvas']
|
||||
inverse = task['transform']['canvas_to_source']
|
||||
for x, y in [(20, 20), (49, 39), (0, 79)]:
|
||||
mx, my = forward[0]*x+forward[2], forward[4]*y+forward[5]
|
||||
self.assertAlmostEqual(inverse[0]*mx+inverse[2], x)
|
||||
self.assertAlmostEqual(inverse[4]*my+inverse[5], y)
|
||||
native = np.array(Image.open(folder / 'native-visible.png'))
|
||||
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
|
||||
native[edit] = [180, 50, 20, 255]
|
||||
output = folder / 'merged.png'
|
||||
qa = merge_candidate(folder, candidate_for(folder, native), output)
|
||||
self.assertTrue(qa['accepted'], qa)
|
||||
keep = np.array(Image.open(folder / 'native-keep.png')) > 0
|
||||
np.testing.assert_array_equal(np.array(Image.open(output))[keep], native[keep])
|
||||
write(self.root / 'completion-result.json', {'parts': [{'id': 'torso', 'status': 'geometry_passed_needs_review',
|
||||
'selected_file': 'tasks/torso/merged.png', 'qa': qa}], 'skipped': []})
|
||||
finalize(self.root)
|
||||
result = read(self.root / 'completion-result.json')['parts'][0]
|
||||
self.assertEqual(result['crop_xyxy'], [20, 20, 50, 40])
|
||||
self.assertIn('preservation', result)
|
||||
self.assertTrue((folder / 'reference.png').exists())
|
||||
mask = np.array(Image.open(folder / 'edit-mask.png'))
|
||||
for x0, y0, x1, y1 in task['reference_panel_boxes']:
|
||||
self.assertTrue(np.all(mask[y0:y1, x0:x1, 3] == 255))
|
||||
|
||||
def test_actual_hole_creates_task_despite_false_semantic_flag(self):
|
||||
plan = read(self.root / 'semantic-parts.json')
|
||||
plan['parts'][0].update(needs_completion=False, visibility='visible', occluded_by=[],
|
||||
completion={k: False for k in plan['parts'][0]['completion']})
|
||||
write(self.root / 'semantic-parts.json', plan)
|
||||
mask = np.array(Image.open(self.root / 'masks/torso.png'))
|
||||
mask[26:30, 26:30] = 0
|
||||
Image.fromarray(mask).save(self.root / 'masks/torso.png')
|
||||
task, folder = self.task()
|
||||
self.assertEqual(task['regions']['segmentation_gap'], 16)
|
||||
native = np.array(Image.open(folder / 'native-visible.png'))
|
||||
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
|
||||
native[edit] = [200, 40, 10, 255]
|
||||
output = folder / 'merged.png'
|
||||
qa = merge_candidate(folder, candidate_for(folder, native), output)
|
||||
self.assertTrue(qa['accepted'], qa)
|
||||
# Repair restores source texture, even though the model proposed a different colour.
|
||||
gap = np.array(Image.open(folder / 'native-segmentation_gap.png')) > 0
|
||||
source = np.array(Image.open(folder / 'native-source.png'))
|
||||
np.testing.assert_array_equal(np.array(Image.open(output))[gap], source[gap])
|
||||
|
||||
def test_attachment_cleanup_releases_only_declared_overlay_and_spine_accepts_contract(self):
|
||||
# Move collar inside torso to represent an overlaid detail needing a clean base.
|
||||
mask = np.zeros((80, 80), np.uint8)
|
||||
mask[26:30, 26:30] = 255
|
||||
Image.fromarray(mask).save(self.root / 'masks/collar.png')
|
||||
task, folder = self.task('both')
|
||||
self.assertEqual(task['regions']['attachment_cleanup'], 16)
|
||||
native = np.array(Image.open(folder / 'native-visible.png'))
|
||||
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
|
||||
native[edit] = [20, 100, 200, 255]
|
||||
output = folder / 'merged.png'
|
||||
qa = merge_candidate(folder, candidate_for(folder, native), output)
|
||||
self.assertTrue(qa['accepted'], qa)
|
||||
write(self.root / 'completion-result.json', {'parts': [{'id': 'torso', 'status': 'geometry_passed_needs_review',
|
||||
'selected_file': 'tasks/torso/merged.png', 'qa': qa}], 'skipped': []})
|
||||
finalize(self.root)
|
||||
data = load_assets(self.root, self.root)
|
||||
self.assertEqual(data[2]['parts'][0]['source'], 'completion')
|
||||
# Contract tampering cannot turn a protected pixel into an untracked edit.
|
||||
result = read(self.root / 'completion-result.json')['parts'][0]
|
||||
image = np.array(Image.open(self.root / result['image']))
|
||||
image[0, 0] = [255, 0, 0, 255]
|
||||
Image.fromarray(image).save(self.root / result['image'])
|
||||
with self.assertRaisesRegex(ValueError, 'preserve original visible'):
|
||||
load_assets(self.root, self.root)
|
||||
|
||||
def test_review_replacement_removal_and_invalid_partition(self):
|
||||
replace = np.zeros((80, 80), np.uint8)
|
||||
replace[26:29, 26:29] = 255
|
||||
remove = np.zeros_like(replace)
|
||||
remove[35:40, 35:40] = 255
|
||||
Image.fromarray(replace).save(self.root / 'replace.png')
|
||||
Image.fromarray(remove).save(self.root / 'remove.png')
|
||||
review = self.root / 'review-input.json'
|
||||
write(review, {'schema_version': 1, 'parts': {'torso': {'confirmed': True,
|
||||
'replace_mask': 'replace.png', 'remove_mask': 'remove.png', 'extension_radius': 8}}})
|
||||
task, folder = self.task(review=review)
|
||||
self.assertEqual(task['extension_radius'], 8)
|
||||
native = np.array(Image.open(folder / 'native-visible.png'))
|
||||
edit = np.array(Image.open(folder / 'native-allowed.png')) > 0
|
||||
native[edit] = [180, 50, 20, 255]
|
||||
qa = merge_candidate(folder, candidate_for(folder, native), folder / 'merged.png')
|
||||
self.assertTrue(qa['accepted'], qa)
|
||||
deleted = np.array(Image.open(folder / 'native-remove.png')) > 0
|
||||
self.assertTrue(np.all(np.array(Image.open(folder / 'merged.png'))[deleted] == 0))
|
||||
Image.fromarray(remove).save(self.root / 'replace.png')
|
||||
with self.assertRaisesRegex(ValueError, 'disjoint'):
|
||||
self.task(review=review)
|
||||
|
||||
def test_large_part_downscales_and_invalid_review_cannot_escape(self):
|
||||
# Upsize the entire source and masks to exceed request dimensions.
|
||||
for file in [self.root / 'source.png', *list((self.root / 'masks').glob('*.png'))]:
|
||||
im = Image.open(file)
|
||||
im.resize((800, 800), Image.Resampling.NEAREST).save(file)
|
||||
task, folder = self.task(size=128)
|
||||
self.assertLess(task['transform']['scale'], 1)
|
||||
native = np.array(Image.open(folder / 'native-visible.png'))
|
||||
native[np.array(Image.open(folder / 'native-allowed.png')) > 0] = [180, 50, 20, 255]
|
||||
output = folder / 'merged.png'
|
||||
qa = merge_candidate(folder, candidate_for(folder, native), output)
|
||||
self.assertTrue(qa['accepted'], qa)
|
||||
keep = np.array(Image.open(folder / 'native-keep.png')) > 0
|
||||
np.testing.assert_array_equal(np.array(Image.open(output))[keep], native[keep])
|
||||
review = self.root / 'bad-review.json'
|
||||
write(review, {'schema_version': 1, 'parts': {'torso': {'confirmed': True, 'replace_mask': '../escape.png'}}})
|
||||
with self.assertRaisesRegex(ValueError, 'escapes'):
|
||||
review_inputs(review, ['torso'], (800, 800))
|
||||
|
||||
def test_adaptive_rejects_movement_and_missing_cleanup(self):
|
||||
task, folder = self.task()
|
||||
candidate = mock_candidate(folder)
|
||||
image = np.array(Image.open(candidate))
|
||||
Image.fromarray(np.roll(image, 30, axis=1)).save(candidate)
|
||||
qa = merge_candidate(folder, candidate, folder / 'bad.png')
|
||||
self.assertFalse(qa['accepted'])
|
||||
self.assertIn('VISIBLE_ALIGNMENT_MISMATCH', qa['reasons'])
|
||||
mask = np.zeros((80, 80), np.uint8)
|
||||
mask[26:30, 26:30] = 255
|
||||
Image.fromarray(mask).save(self.root / 'masks/collar.png')
|
||||
_, folder = self.task('both')
|
||||
no_fill = np.array(Image.open(folder / 'native-visible.png'))
|
||||
qa = merge_candidate(folder, candidate_for(folder, no_fill), folder / 'bad-cleanup.png')
|
||||
self.assertFalse(qa['accepted'])
|
||||
self.assertIn('INCOMPLETE_ATTACHMENT_CLEANUP', qa['reasons'])
|
||||
|
||||
def test_duplicate_masks_blocked_until_reviewed(self):
|
||||
plan = read(self.root / 'semantic-parts.json')
|
||||
plan['parts'][0]['occluded_by'] = []
|
||||
write(self.root / 'semantic-parts.json', plan)
|
||||
Image.open(self.root / 'masks/torso.png').save(self.root / 'masks/collar.png')
|
||||
result = prepare_adaptive(self.root, ['torso'])
|
||||
self.assertEqual(result['skipped'][0]['reason'], 'AMBIGUOUS_MASK_REQUIRES_REVIEW')
|
||||
|
||||
def test_spine_strict_policy_respects_rebuilt_need_even_if_original_flag_was_false(self):
|
||||
plan = read(self.root / 'semantic-parts.json')
|
||||
plan['parts'][0].update(needs_completion=False, visibility='visible', occluded_by=[],
|
||||
completion={k: False for k in plan['parts'][0]['completion']})
|
||||
write(self.root / 'semantic-parts.json', plan)
|
||||
write(self.root / 'completion-result.json', {'schema_version': 2, 'profile': 'adaptive', 'parts': [
|
||||
{'id': 'torso', 'status': 'rejected', 'qa': {'accepted': False}}], 'skipped': []})
|
||||
with self.assertRaisesRegex(ValueError, 'Completion required for torso'):
|
||||
load_assets(self.root, self.root, fallback_policy='error')
|
||||
data = load_assets(self.root, self.root)
|
||||
self.assertEqual(data[2]['parts'][0]['reason'], 'COMPLETION_REJECTED')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+12
-3
@@ -39,8 +39,13 @@ const help = `模型接入与结构理解（新增模式，不调用旧拆件流
|
||||
--prepare-only 仅准备补全画布、掩码和提示词，不请求模型
|
||||
--candidates-dir DIR 离线导入 ID.png，或旧补全目录的首张候选，不请求模型
|
||||
--candidate-size-policy NAME strict（默认）/ uniform（同宽高比缩放后重新检查对齐）
|
||||
--completion-size INT 补全方形画布边长，默认 1024，不缩放原像素
|
||||
--extension-radius INT 补全区域距可见轮廓的最大像素距离，默认 32
|
||||
--completion-profile NAME adaptive（默认）重建需求与可逆缩放；legacy 复用旧候选
|
||||
--completion-reference NAME auto（默认）/ board / local / full / both / none
|
||||
--completion-mask NAME auto（默认）/ required / off；auto 遵循提供方能力
|
||||
--completion-review FILE 复核 JSON，按 ID 指定修复/替换/移除 mask 与半径
|
||||
--probe-edit-capabilities 单部件真实多图+mask 探针，不自动修改提供方能力配置
|
||||
--completion-size INT 请求方形画布边长，默认 1024；adaptive 显式记录缩放逆变换
|
||||
--extension-radius INT 全局原图像素半径覆盖；adaptive 默认按部件尺寸，legacy 默认 32
|
||||
--sam-dir DIR 本地 SAM 根目录，包含 sam3 与 sam3.1checkpoint
|
||||
--sam-code-dir DIR SAM 源码仓库目录覆盖
|
||||
--sam-checkpoint FILE 本地权重路径覆盖
|
||||
@@ -67,6 +72,8 @@ export function parseOptions(argv) {
|
||||
['--candidate-size-policy', 'candidateSizePolicy'],
|
||||
['--completion-dir', 'completionDir'], ['--fallback-policy', 'fallbackPolicy'],
|
||||
['--draw-order-file', 'drawOrderFile'], ['--check-angle', 'checkAngle'], ['--genielabs-dir', 'genielabsDir'],
|
||||
['--completion-profile', 'completionProfile'], ['--completion-reference', 'completionReference'],
|
||||
['--completion-mask', 'completionMask'], ['--completion-review', 'completionReview'],
|
||||
]);
|
||||
for (let i = 0; i < argv.length; i += 1) {
|
||||
const flag = argv[i];
|
||||
@@ -74,6 +81,7 @@ export function parseOptions(argv) {
|
||||
else if (flag === '--resume') options.resume = true;
|
||||
else if (flag === '--dry-run') options.dryRun = true;
|
||||
else if (flag === '--prepare-only') options.prepareOnly = true;
|
||||
else if (flag === '--probe-edit-capabilities') options.probeEditCapabilities = true;
|
||||
else if (flag === '--list-models') options.listModels = true;
|
||||
else if (valueFlags.has(flag) || flag === '--image' || flag === '--part') {
|
||||
const value = argv[++i];
|
||||
@@ -111,7 +119,8 @@ export async function main(argv = process.argv.slice(2)) {
|
||||
}
|
||||
const loaded = loadConfig(options);
|
||||
const partCompletion = options.stage === 'completion' && options.inputDir;
|
||||
if (!partCompletion && (options.prepareOnly || options.candidatesDir || options.partIds.length || options.completionSize || options.extensionRadius || options.candidateSizePolicy)) {
|
||||
if (!partCompletion && (options.prepareOnly || options.candidatesDir || options.partIds.length || options.completionSize || options.extensionRadius || options.candidateSizePolicy ||
|
||||
options.completionProfile || options.completionReference || options.completionMask || options.completionReview || options.probeEditCapabilities)) {
|
||||
throw new Error('部件补全参数需要 --stage completion --input-dir');
|
||||
}
|
||||
if (partCompletion) {
|
||||
|
||||
Reference in New Issue
Block a user