Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 ziyu
|
3753f110f4 |
@@ -3,19 +3,3 @@ VECTOR_ENGINE_BASE_URL=https://your-vector-engine.example
|
||||
VECTOR_ENGINE_API_KEY=replace-me
|
||||
# Optional request timeout in milliseconds
|
||||
VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS=1200000
|
||||
|
||||
# Optional independent relay: select with --config configs/tiantoken.json
|
||||
TIANTOKEN_BASE_URL=https://api.tiantoken.com
|
||||
TIANTOKEN_API_KEY=replace-me
|
||||
# Defaults in tiantoken.json; independent from SPINE_* model overrides.
|
||||
# TIANTOKEN_UNDERSTANDING_MODEL=gpt-6-astra
|
||||
# TIANTOKEN_COMPLETION_MODEL=gpt-image-2
|
||||
|
||||
# Independent model IDs for the annotated-parts pipeline. Use EXACT relay IDs.
|
||||
# Confirm with: node tools/model-check.mjs --list-models
|
||||
SPINE_UNDERSTANDING_MODEL=gpt-6-astra
|
||||
SPINE_COMPLETION_MODEL=qwen-image-3.0
|
||||
|
||||
# Local source + checkpoint root. SAM runs in a temporary ai-spine GPU process.
|
||||
SAM_MODEL_DIR=F:/Study/SAM3
|
||||
SAM_CODE_DIR=F:/Study/SAM3/sam3
|
||||
|
||||
@@ -16,6 +16,3 @@ references/external/kenney-platformer/
|
||||
|
||||
# OS/editor noise
|
||||
.DS_Store
|
||||
|
||||
#data
|
||||
data/
|
||||
@@ -1,159 +0,0 @@
|
||||
# 项目修改约定
|
||||
|
||||
本文件适用于 `AI_spine/` 及其所有子目录。每次修改代码、文档、依赖或运行测试前，都必须阅读并遵守。
|
||||
|
||||
## Python 环境
|
||||
|
||||
- 本项目固定使用名为 `ai-spine` 的 Conda 环境。
|
||||
- 不创建、不使用 `.venv`、`venv` 或其他 Python 虚拟环境；不把依赖安装到 Conda base 或系统 Python。
|
||||
- 交互式终端先执行 `conda activate ai-spine`；自动化命令使用 `conda run -n ai-spine ...`。
|
||||
- Python 依赖统一记录在 `requirements.txt`。安装命令使用 `conda run -n ai-spine python -m pip install -r requirements.txt`，避免裸 `pip` 指向错误环境。
|
||||
- Node 管线调用 Python 时也必须使用此 Conda 环境。`--python` 只能指向 `ai-spine` 环境中的解释器。
|
||||
- 不在代码中硬编码本机 Conda 安装路径，不自动创建环境或自动安装依赖。缺少环境或依赖时给出明确的安装/激活命令。
|
||||
|
||||
## 修改与验证
|
||||
|
||||
- 修改环境选择、依赖或启动方式时，同步更新 README、相关文档和 CLI 帮助，保持一致。
|
||||
- 测试在 `ai-spine` 环境下执行：`conda run -n ai-spine npm test`。不要为了绕过依赖问题临时新建环境。
|
||||
- 验证结果必须如实记录；缺少依赖导致未运行的检查，不得声称通过。
|
||||
- 保留用户已有修改和实验产物，不覆盖旧实验输出。
|
||||
|
||||
## 跨会话上下文与阅读顺序
|
||||
|
||||
本节于 2026-09-22 接入通用第四阶段及 adaptive 补全后更新。它是后续对话的上下文入口；后续修改应同步更新对应状态，实际源码、配置和具体实验 manifest 优先于历史总结。不要把旧实验统计套到新输出上。
|
||||
|
||||
1. 先阅读本文件和 `git status --short`，保留用户已有修改。
|
||||
2. 阅读 [README](README.md) 了解项目、环境、配置与命令。
|
||||
3. 按任务阅读 [第一阶段](docs/1-understanding-stage.md)、[第二阶段](docs/2-segmentation-stage.md)、[第三阶段](docs/3-completion-stage.md)、[第四阶段](docs/4-spine-stage.md)，再核对相关实现与实际产物。
|
||||
4. 用户说“继续”时，先核对工作区和最近任务进展，不能只凭历史总结重复请求付费模型或覆盖实验。
|
||||
|
||||
## 用户目标与固定偏好
|
||||
|
||||
- 目标是将单张 2D 角色图拆成可以用于 Spine 的部件，保留画风、姿态、左右、不对称性及原有像素关系，补足遮挡和关节连接。
|
||||
- 第二阶段使用本地 SAM 代码和权重，通过临时 Conda 子进程在本机 GPU 推理，完成后退出；不部署常驻独立服务。
|
||||
- 需要可直接执行的 PowerShell 命令；说明当前已实现行为、实测结果与未来计划，不能把几何通过写成资产合格。
|
||||
- 不确定的部件语义、遮挡和候选点需要复核，不得用前景检查、覆盖率或模型分数代替语义准确性。
|
||||
- 输出使用独立目录，保留原候选和诊断；已有候选优先离线复检，避免无意义重复生成。
|
||||
|
||||
## 文档维护约定
|
||||
|
||||
- 原有文档文件名保持不变。`docs/` 保留前三份阶段文档，并新增 `4-spine-stage.md`；`docs/model-stages.md` 在此前文档整理前已处于用户删除状态，不自行恢复，相关说明归入 README 和阶段文档。
|
||||
- 四份阶段文档有且仅有以下八个二级章节，细节必须归入对应章节：
|
||||
1. 阶段简介
|
||||
2. 输入、输出与执行链路
|
||||
3. 阶段执行命令示例及参数说明
|
||||
4. 输入格式要求
|
||||
5. 执行链路中重要部分代码文件及原理说明（包括重点说明）
|
||||
6. 各个代码文件职责索引
|
||||
7. 产物详细说明
|
||||
8. 当前阶段可改进点说明
|
||||
- 外层 README 负责整体介绍、Conda 准备、配置及每阶段命令；本文件负责约定和后续会话必要上下文。已存在的环境与修改约定必须保留。
|
||||
- 改进项明确写为待实现/待验证；实现后更新状态、命令、产物说明及本文件。PowerShell 反引号后不能有空格；示例变量先定义，示例输出目录相互独立。
|
||||
|
||||
## 工作区、环境与配置
|
||||
|
||||
- 当前实际项目目录：`F:\Study\taonier\AI_spine`。用户历史命令也出现过 `F:\Study\taonier\AI\_spine`，不要盲目复制旧路径，先确认 cwd。
|
||||
- 当前 SAM 根目录为 `F:\Study\SAM3`；源码仓库 `F:\Study\SAM3\sam3`；权重 `F:\Study\SAM3\sam3.1checkpoint\sam3.1_multiplex.pt`。早期 `F:\Study\sam3.1` 已不是当前约定路径。
|
||||
- 已验证组合：Python 3.11、torch 2.7.1+cu126、torchvision 0.22.1、NumPy 1.26.4、OpenCV 4.11.0.86；RTX 3060 Laptop 6 GB。机器环境可能变化，执行前按需探测。
|
||||
- Node.js >=20，无额外 npm 运行时依赖。自动化使用 Conda ai-spine 执行 Node/Python；运行 SAM 可用 `conda run --no-capture-output -n ai-spine node ...`。
|
||||
- 默认配置是 `configs/astra-qwen.json`；示例推荐明确传 config，避免环境变量导致选错提供方。
|
||||
- `configs/deepseek-qwen.json`：deepseek-flash / Chat Completions / 非流式；补全 qwen-image-3.0。
|
||||
- `configs/astra-qwen.json`：gpt-6-astra / Responses / 流式；补全 qwen-image-3.0。
|
||||
- 上述两份读取 VECTOR_ENGINE_BASE_URL / VECTOR_ENGINE_API_KEY，模型变量为 SPINE_UNDERSTANDING_MODEL / SPINE_COMPLETION_MODEL。
|
||||
- `configs/tiantoken.json`：独立 TIANTOKEN_BASE_URL / TIANTOKEN_API_KEY，地址 https://api.tiantoken.com；理解 gpt-6-astra，补全 gpt-image-2；模型变量为 TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL。
|
||||
- CLI 模型 > stage.model > 对应模型环境变量 > default_model；shell 同名变量优先于 .env。模型 ID 原样发送，不加隐式别名。
|
||||
- 用户提供的密钥仅留在本地忽略的 `.env`；禁止在文档、命令日志、聊天或提交中暴露，不要读取/打印完整 .env 来诊断。
|
||||
- 三份配置的图片编辑能力仍为 multiple_images=false、mask=false。adaptive auto 在此配置下发送单图参考板，包含局部和完整角色参考，mask 仅本地约束。独立多图/mask 和单部件探针已接入，但本轮未调用付费模型，不能宣称中转已支持。Astra 理解接口与图片编辑接口不同，当前没有 Astra Responses 图片生成工具链。
|
||||
|
||||
## 四阶段实现与不能丢失的边界
|
||||
|
||||
### 第一阶段：理解与几何候选
|
||||
|
||||
- 入口 `tools/model-check.mjs --stage understanding`，或 `pipeline.mjs --mode annotated-parts`；后者仅转发新入口。
|
||||
- `stages/model-stage.mjs` 调用视觉模型，`lib/validate-parts.mjs` 校验 v2：rig_role、bind_to、三项 completion 及 needs_completion 一致性。
|
||||
- `normalize_reference.py` 保留原画布，使用 Alpha 或边界连通近白背景处理；`geometry_grounding.py` 用 Alpha/ROI 距离变换生成候选，不是再次请求 AI。
|
||||
- 模型原始点仅诊断；程序候选也未获语义确认；pivot 独立保留，不吸附前景。
|
||||
- 第二阶段输入必须为同一实验的 source.png、semantic-parts.json、segmentation-prompts.json。
|
||||
- `--parts-file` 可离线迁移 v1/检查 v2 并生成新候选；旧 annotated.svg 不会自动更新。
|
||||
|
||||
### 第二阶段：SAM 可见拆件
|
||||
|
||||
- 入口 `--stage segmentation --input-dir 理解目录`；实现为 stages/segmentation-stage.mjs、sam_runtime.py、sam_segment.py、segmentation_outputs.py。
|
||||
- 环境探测与推理分别为短期子进程，禁止在线下载权重。multiplex detector 适配实际严格加载 1130 个张量，忽略 36 个未使用视频 neck 张量；不要因此改成 strict=False 或修改外部 SAM 源码。
|
||||
- BF16 仅用于推理 autocast；不要整体转换模型而破坏复数缓冲区/FP32 路径。
|
||||
- 使用文本和文本+粗框两组候选；当前图像概念接口未接收点提示，Alpha 种子仅参与 mask 排序。
|
||||
- 与原图 Alpha 取交集后提取原像素；mask 为原画布二值图，parts 为紧裁剪 RGBA。重叠保留、覆盖率仅为像素覆盖，semantic_verified=false。
|
||||
- annotated.png 为原图纹理上半透明颜色叠加，parts-sheet.png 为白底拆件；纯色遮罩角色图与可选大模型去毛边尚未实现。
|
||||
|
||||
### 第三阶段：受限部件补全
|
||||
|
||||
- 入口 `--stage completion --input-dir SAM目录`；实现为 stages/completion-stage.mjs、completion_geometry.py、completion_planning.py、providers/image-edits.mjs。默认 completion-profile=adaptive，legacy 保留旧行为和候选兼容。
|
||||
- 输入须含 source.png、v2 计划、segmentation.json 及所有记录的全画布 masks；单独 parts/ 不够。
|
||||
- adaptive 从实际 mask 重建遮挡/关节、闭合漏分孔洞、附件底图清理需求，不单凭 needs_completion 筛选。自动孔洞只取原图前景且无其他 mask 归属的区域；开放缺口和歧义归属仍需复核。无明确遮挡的近重复 mask（IoU≥0.8）要求人工确认。
|
||||
- adaptive preserve=SAM−editable−removed；仅保留区锁定原 RGBA，清理和替换区可重绘，人工 remove 可删除。completion-review JSON（schema_version=1、parts 映射、confirmed=true）支持 repair/replace/remove 全画布二值 masks 与部件半径，原文件及哈希留档，不改 SAM。
|
||||
- 默认请求边长 1024；adaptive 半径按 mask 短边×0.15 向上取整并限于 4～96 原像素，CLI 或部件 review 可覆盖。native 裁剪边长约目标包围框最长边×1.15；小部件放大、大部件缩小，保存正反仿射和工作区。逆缩放后恢复保留原像素，坐标可供第四阶段使用。
|
||||
- reference auto：multiple_images=true 时 input→context→reference，默认 false 时发送 board（左目标，右局部和全角色参考），参考面板不进入最终资产。支持 none/local/full/both/board；mask auto/required/off 独立选择。request-plan.json 记录计划，request.json 仅实际发送时生成，含有序图片角色、哈希、尺寸、字段、mask 哈希与透明语义。
|
||||
- legacy 才沿用 needs_completion、原像素不缩放、默认 32 半径和原可见像素全保护。旧候选复检必须显式 --completion-profile legacy。新旧画布/参考布局/复核文件不同不能混用候选。
|
||||
- 图片解码已修复“空 b64_json 但有有效 URL”情况，回退下载 URL；该问题与画布尺寸不符不同。
|
||||
- 在线请求现显式传 size，配置冲突失败。strict 默认拒绝尺寸不同；uniform 仅恢复同宽高比整画布尺寸，保留原候选，继续原有验收，不单独移动/缩放物体。
|
||||
- 对齐阈值保持 RGB MAE≤30、保留率≥95%；adaptive 在请求工作区的保留不透明像素上检查，再逆采样，避免把插值 Alpha 误作位移。编辑像素必须位于允许区且连接保留区；可见缺口使用原图纹理，移除区清零。
|
||||
- adaptive 清理/人工替换要求≥95% 有效填充，其余 unfulfilled_regions 仍仅诊断。语义、接缝和关节质量不能由填充率证明。未明确编辑的误分仍被保护，需复核文件修正。
|
||||
- --probe-edit-capabilities 要求 adaptive 在线且恰好一个 --part，发送主图/局部/全角色+mask，n=1、maxRetries=0。记录 capability-evidence 和合并前 raw_outside_edit 指标；不会把 reference_use_verified/mask_semantics_verified 自动置 true 或修改配置。prepare-only/mock 不是中转实测。
|
||||
- prepare-only 不联网；candidates-dir 可读平铺 ID.png 或匹配 manifest 的旧任务首张候选。--resume 不是失败断点续发，准备/生成/复检使用独立目录。
|
||||
- parts/ 只含通过检查的补全部件，不包含全部原拆件。完整角色组装、Spine 导出与旋转检查动画已独立接入第四阶段；运动质量自动验收仍未实现。
|
||||
- 旧 `--stage completion --image ... --prompt-file ...` 是接口探针，不能和 input-dir 混用，不执行完整几何验收。
|
||||
|
||||
### 第四阶段：通用组装与 Spine 导出
|
||||
|
||||
- 用户明确要求可复用管线，不能写死具体角色、部件 ID、数量或人形比例。入口 `--stage spine --input-dir SAM目录 --completion-dir 补全目录 --out-dir 新目录`；completion-dir 可省略。
|
||||
- `stages/spine-stage.mjs` 与 `spine_assembly.py` 完全离线，校验 v2、输入快照及 masks 哈希、PNG 和裁剪坐标；不读取模型配置/密钥，不调用 SAM。
|
||||
- 原资产从 source + mask 重建，采用补全结果前检查 selected_file、状态、qa.accepted、image、crop_xyxy。legacy 全可见 RGBA 保持；adaptive 验证 preservation 分区、保留 RGBA 精确匹配、编辑/移除互斥及输出不越界，契约 mask 哈希留档。不能混用其他实验。
|
||||
- 默认补全失败回退 SAM 并记录原因；fallback-policy=error 拒绝需补全但无通过结果的可见部件。缺可见 mask 直接报错；隐藏骨骼保留但不生成图片。
|
||||
- pivot 转换为底边中点原点的 Spine 坐标；骨骼 setup 旋转为零，局部位置取父子 pivot 差；附件中心相对绑定骨骼偏移。bone_part 使用 parent，其他角色使用 bind_to。
|
||||
- occluded_by 稳定拓扑排序；无约束重叠写入诊断，可用 draw-order-file 全量覆盖，冲突留档。骨骼/遮挡环路拒绝。
|
||||
- 导出完整 images、assembly.json、静态组装/对比/覆盖诊断、skeleton.json、atlas、report.html 和真实 preview.html。依赖外部只读图集和播放器脚本，其哈希纳入复用指纹。
|
||||
- 目标 Spine 4.2；setup 静态姿态及 check__ID 单骨骼检查动画。没有成品 idle/walk、mesh/weights/IK，semantic_verified=false、quality_status=needs_review；几何通过和可加载不代表资产合格。
|
||||
- 生成离线；播放器从 CDN 加载资源，打开骨架预览需要网络。静态报告可离线查看。
|
||||
|
||||
## 已检查实验与质量结论
|
||||
|
||||
以下为已有实验记录，不代表后续每次运行的结果；不必为了文档修改重跑 GPU 或付费模型。
|
||||
|
||||
| 目录（相对于项目） | 事实与意义 |
|
||||
| --- | --- |
|
||||
| output/sombrero-grounding-v2 | 旧计划离线几何检查，28 个程序候选通过前景检查；未证明语义正确 |
|
||||
| output/sombrero-sam31-v3 | 较早 SAM 实验，28 个 mask，约 67.18% 覆盖率；肢体/鞋子重复、漏分，不合格 |
|
||||
| output/20260918-183532/sam | 较早基线：32 个计划项、10 个 hidden、22 个 mask；约 83.3% 覆盖，32719 像素未分配；前臂/手混入衣料、袖子与上衣缺损 |
|
||||
| output/20260918-183532/complete2_gpt | 真实 Tiantoken gpt-image-2 请求 12 部件；旧请求未传 size，返回 1248/1254，预期 1024，全部 CANVAS_SIZE_MISMATCH |
|
||||
| output/20260918-183532/complete2_gpt-size-recheck-v2 | 离线 uniform 复检，未重生成：8 个几何通过；head、pants_thigh_l、pants_thigh_r、shoe_l 4 个拒绝；通过项仍有误分和未完成区域 |
|
||||
| output/20260918-183532/complete-recovered-head-check | 更早 Qwen 头部候选恢复检查，不能与 GPT 批次混为一谈 |
|
||||
| output/20260920-225137 | 较新实验：27 个语义项、2 个 hidden、25 个 SAM 部件；补全 16 项中 13 个几何通过、3 个对齐拒绝 |
|
||||
| output/spine-validation-latest-20260922 | 第四阶段离线验证：25 个 slot、16 个骨骼（含 root）；采用 13 个补全部件，5 个需补全部件回退 SAM（含两袖无允许区域）；上游缺口仍存在 |
|
||||
| output/spine-validation-baseline-20260922 | 另一批输入验证：22 个 slot、16 个骨骼（含 root）；采用 8 个补全部件、4 个回退；输入源为旧基线和 size-recheck-v2 |
|
||||
| output/completion-adaptive-prepare-20260922 | adaptive 离线准备：16 个任务、11 个跳过；上衣规划 3708 个封闭漏分像素，头部规划 10205 个附件清理像素；这些是待复核区域，不是补全成功结果，未请求模型 |
|
||||
|
||||
- 当前 22→12 的原因：sombrero、sleeve_l、sleeve_r、eye_l、eye_r、eyebrow_l、eyebrow_r、nose、mouth、mustache 共 10 个被第一阶段标记 NO_COMPLETION_NEEDED；不能解释为它们的实际 SAM mask 已完整。
|
||||
- 2026-09-20 尺寸修复后曾通过 22 项 Node 测试（含 Python 补全回归）。这是历史验证；新改动按范围重新验证，不自动沿用“全部通过”。显式 size 构造已测试，修复后中转是否遵守尚无新实测。
|
||||
- 总体判断：上游部件归属、模型缺少原图参考与遮罩、区域规则粗糙、验收不足共同导致低质量。不能仅换模型、强制处理全部 mask、放宽阈值或解决尺寸错误就宣称修好。
|
||||
- 2026-09-22 第四阶段接入后，Conda ai-spine 下 npm test 共 26 项通过（含新增 6 项 Python 组装用例）。另使用官方 Spine 4.2 core IIFE 对上述两份导出分别验证 setup 顶点和 30 个单关节旋转采样，全部通过。最新导出的 preview.html 已在 headless Chrome 实际加载渲染，截图为 output/spine-runtime-validation-20260922/preview.png；这不是动作或资产质量验收。复检工具为 `tools/verify-spine-runtime.mjs`，需调用者提供本地官方 runtime 文件，不自动下载/安装依赖。
|
||||
|
||||
- 随后接入 adaptive 补全，Conda ai-spine 下 npm test 共 29 项全部通过（含 8 项新增 Python adaptive 用例）：覆盖参考顺序/mask mock、放大/缩小、位移拒绝、实际缺口、清理与区域契约、候选身份及第四阶段 strict 回退。未进行付费中转试验，真实参考使用与 mask 语义仍待验证。
|
||||
|
||||
## 当前改进方向与后续顺序
|
||||
|
||||
当前用户已要求落实第三阶段的参考图/区域重建/自适应画布，必须保持可复用且与第四阶段贯通。当前状态：
|
||||
|
||||
1. 第二阶段输出仅色彩遮罩组成的完整角色图，保留当前原图叠色预览，并明确未分配/重叠区域。
|
||||
2. 第二阶段每个 SAM 部件可选经过一次大模型去毛边，保留原始拆件、原图参考、坐标、变更区域与前后对照。
|
||||
3. 第三阶段参考图、实际 mask 规划、自适应区域/可逆缩放、发送审计和单部件能力探针已实现并离线验证；真实多图/mask 语义和新布局生成质量仍待付费小样实测，默认 capabilities 不变。
|
||||
4. 完整角色组装及失败回退记录已在第四阶段实现；接缝与关节旋转质量自动验收仍待实现。
|
||||
5. 后续质量工作：复核自动规划和 SAM 归属、实测参考图与遮罩、加强接缝/动作验收。第三/四阶段遵循统一输入契约，不为某个实验硬编码修正。
|
||||
|
||||
详细改进说明保留在各阶段文档第 8 节；若后续用户改变优先级，应更新此处。
|
||||
|
||||
## 旧管线与范围识别
|
||||
|
||||
- `pipeline.mjs` 未指定 --mode annotated-parts 时走旧“生成拆件图→CV 定位→Spine”流程，不是新四阶段自动串联。
|
||||
- `segment.py` 为旧拆件图分割；`compose_preview.py` 为旧布局组装；`run_case_matrix.py` 为旧定位对照工具。不要因为 IDE 当前打开这些文件就误改新阶段入口。
|
||||
- 相邻 `spine-animation-ai` 为外部只读依赖，保留其 PolyForm Noncommercial 许可证约束；第四阶段复用图集和播放器脚本，不复用旧特征定位或 root 平铺骨架逻辑。
|
||||
- 不要把文档改写任务扩大成付费图片生成、SAM 推理、依赖安装或功能实现；需要实现时遵从用户后续任务范围。
|
||||
@@ -1,254 +1,93 @@
|
||||
# AI Spine 角色拆件与动画管线
|
||||
# Genarrative / Genielabs Spine Animation Pipeline
|
||||
|
||||
本项目探索将单张 2D 角色图转换为可用于 Spine 动画的部件与骨骼资源。当前主线分为四个阶段：视觉 LLM 理解结构、本地 SAM 分割可见部件、图片编辑模型补全隐藏区域和关节连接、离线组装与 Spine 导出。
|
||||
这是独立目录，不修改 Genarrative。它只参考 Genarrative 的 VectorEngine 图片生成协议：后端环境变量提供 `VECTOR_ENGINE_BASE_URL` 与 `VECTOR_ENGINE_API_KEY`，使用 OpenAI-compatible `POST /v1/images/edits` multipart 请求。
|
||||
|
||||
| 阶段 | 输入 → 输出 | 当前状态 |
|
||||
| --- | --- | --- |
|
||||
| 1. 结构理解 | PNG → v2 语义计划、粗框、pivot、程序候选点 | 可在线理解或离线复检；语义待复核 |
|
||||
| 2. SAM 分割 | 第一阶段目录 → mask、原像素透明部件、标注图与拆解图 | 本机 GPU 实测可运行；存在误分与漏分 |
|
||||
| 3. 部件补全 | SAM 目录 → 局部编辑候选、几何 QA、通过检查的补全部件 | 已真实调用图片模型；结果质量尚不合格 |
|
||||
| 4. Spine 导出 | SAM + 可选补全目录 → 完整资产、原位组装、骨骼、图集、预览 | 通用离线入口；含单骨骼旋转检查动画，资产与动作质量需复核 |
|
||||
当前上游没有 `gpt-image-2.5`，脚本会把它映射为 `gpt-image-2`，并在 manifest 中保留 `requestedModel`。
|
||||
|
||||
第三阶段输出通过几何检查的补全部件；第四阶段整合所有可绘制部件、组装并导出 Spine 4.2。执行成功、mask 覆盖率高、几何检查通过，均不等于动画资产可用。
|
||||
拆件提示词是角色无关的通用模板：以参考图为唯一 ground truth，保留原始注册位置、左右方向、遮挡关系和非对称结构，只在自然关节/遮挡边界切分，禁止镜像、拉直、补全隐藏部件和生成重复备件。图片接口对 429/5xx 会自动重试 3 次；使用新输出目录即可重新生成，旧目录可用 `--resume` 复用已生成的拆件图。
|
||||
|
||||
仓库保留旧版“生成白底拆件图 → OpenCV 拆分 → Genielabs 定位 → Spine 导出”作为对照；它与当前四阶段主线并存，产物不能直接混用。
|
||||
正式运行默认尝试 6 个通用拆件提示词变体，分别强调注册位置、关节边界、遮挡审计、保守拆分和像素保真；脚本按微碎片、巨型粘连、前景覆盖率和极端部件数量选择候选，不假定角色必须有固定部件数。可用 `--attempts 1` 只运行一个候选。
|
||||
|
||||
## 文档入口
|
||||
## 运行
|
||||
|
||||
- [AGENTS.md](AGENTS.md)：修改约定、跨会话上下文、当前基线及后续优先级。
|
||||
- [第一阶段](docs/1-understanding-stage.md)：理解、数据契约、Alpha 候选与粗标注。
|
||||
- [第二阶段](docs/2-segmentation-stage.md)：本地 SAM、mask 选择、拆件与质量边界。
|
||||
- [第三阶段](docs/3-completion-stage.md)：筛选、局部画布、图片请求、几何验收及补全改进。
|
||||
- [第四阶段](docs/4-spine-stage.md)：通用资产整合、坐标与骨骼、绘制顺序、Spine 导出与回退策略。
|
||||
|
||||
阶段文档统一采用八节：简介；输入输出与链路；命令参数；输入格式；重要代码原理；职责索引；产物；可改进点。配置与实验说明已归入上述文档。
|
||||
|
||||
## Conda 环境准备
|
||||
|
||||
所有 Python 操作固定使用 **ai-spine**，不使用 `.venv`、其他虚拟环境、Conda base 或系统 Python。Node.js 要求 20 或以上；项目没有额外 npm 运行时依赖。
|
||||
|
||||
在 PowerShell 中进入实际项目根目录。本工作区路径如下；若本机使用其他目录，仅替换这一行：
|
||||
|
||||
```powershell
|
||||
Set-Location F:\Study\taonier\AI_spine
|
||||
node --version
|
||||
conda env list
|
||||
```
|
||||
|
||||
仅在没有 ai-spine 环境的新机器上，手动创建一次；当前机器已有环境则跳过：
|
||||
|
||||
```powershell
|
||||
conda create -n ai-spine python=3.11 -y
|
||||
```
|
||||
|
||||
补齐项目依赖。以下是本机已验证的 CUDA 12.6 组合；已有匹配 PyTorch 时可跳过第一条安装命令。代码不会自动安装依赖或创建环境。
|
||||
|
||||
```powershell
|
||||
conda activate ai-spine
|
||||
python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
|
||||
python -m pip install -r .\requirements.txt
|
||||
python -m pip install --no-deps --no-build-isolation -e F:\Study\SAM3\sam3
|
||||
```
|
||||
|
||||
不激活终端时，明确指定目标环境：
|
||||
|
||||
```powershell
|
||||
conda run -n ai-spine python -m pip install -r .\requirements.txt
|
||||
```
|
||||
|
||||
本地 SAM 代码和权重由用户提供，当前位置为：
|
||||
|
||||
```text
|
||||
F:/Study/SAM3/sam3/ 源码仓库
|
||||
F:/Study/SAM3/sam3.1checkpoint/sam3.1_multiplex.pt 权重
|
||||
```
|
||||
|
||||
检查解释器、基础依赖与 GPU：
|
||||
|
||||
```powershell
|
||||
conda run -n ai-spine python -c "import sys, cv2, PIL, numpy, torch; print(sys.executable); print('torch', torch.__version__); print('CUDA', torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'No CUDA GPU')"
|
||||
```
|
||||
|
||||
第二阶段要求可用 CUDA GPU。本机已验证 RTX 3060 Laptop 6 GB；默认 BF16 推理保留模型参数和缓冲区所需精度。第一、三阶段的本地几何处理不运行 SAM。Node 入口会验证 Python 属于 ai-spine；`--python` 也不能绕过此约定。
|
||||
|
||||
## 中转站配置
|
||||
|
||||
仅在本地 `.env` 不存在时复制模板，随后自行填写密钥：
|
||||
|
||||
```powershell
|
||||
if (-not (Test-Path .\.env)) { Copy-Item .\.env.example .\.env }
|
||||
```
|
||||
|
||||
```dotenv
|
||||
# 原中转站
|
||||
VECTOR_ENGINE_BASE_URL=https://router.genarrative.world/v1
|
||||
VECTOR_ENGINE_API_KEY=填写你的令牌
|
||||
|
||||
# Tiantoken 独立配置
|
||||
TIANTOKEN_BASE_URL=https://api.tiantoken.com
|
||||
TIANTOKEN_API_KEY=填写你的令牌
|
||||
|
||||
# 本地 SAM
|
||||
SAM_MODEL_DIR=F:/Study/SAM3
|
||||
SAM_CODE_DIR=F:/Study/SAM3/sam3
|
||||
```
|
||||
|
||||
不要把密钥写入文档、JSON 配置或提交到版本库；不要输出 `.env` 内容。
|
||||
|
||||
| 配置文件 | 第一阶段 | 第三阶段 | 模型环境变量 |
|
||||
| --- | --- | --- | --- |
|
||||
| [deepseek-qwen.json](configs/deepseek-qwen.json) | deepseek-flash，Chat Completions，非流式 | qwen-image-3.0，images/edits | SPINE_UNDERSTANDING_MODEL / SPINE_COMPLETION_MODEL |
|
||||
| [astra-qwen.json](configs/astra-qwen.json) | gpt-6-astra，Responses，流式 | qwen-image-3.0，images/edits | 同上 |
|
||||
| [tiantoken.json](configs/tiantoken.json) | gpt-6-astra，Responses，流式 | gpt-image-2，images/edits | TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL |
|
||||
|
||||
CLI > 配置 stage.model > 对应模型环境变量 > default_model；shell 环境变量优先于 `.env`。不指定配置时入口默认 astra-qwen.json，建议实验始终显式指定。模型 ID 原样发送，中转能力以实际请求为准。Tiantoken 图片编辑已真实执行；第三阶段尚未适配 Astra Responses 图片生成工具。
|
||||
|
||||
三个配置当前仍关闭原生多图与编辑 mask 能力。第三阶段默认 adaptive：实际 mask 重建需求、自适应半径和可逆缩放；当前配置下 auto 会发送含目标、局部参考、完整角色参考的单张参考板。只有支持原生多图时才独立发送参考图；mask 是否发送由配置和 completion-mask 决定，未发送时明确标记 local_only。不能把本地裁剪与恢复当成模型遵循 mask 的证据。
|
||||
|
||||
查询模型列表和检查配置：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --config .\configs\tiantoken.json --list-models
|
||||
node .\tools\model-check.mjs --config .\configs\tiantoken.json --dry-run
|
||||
```
|
||||
|
||||
前者联网，后者不请求模型；列表存在某个 ID 不代表其所有能力已验证。
|
||||
|
||||
## 四阶段执行命令
|
||||
|
||||
以下命令按顺序在同一个 PowerShell 会话执行。反引号后不能带空格。每次实验使用新的 runId；分阶段重开终端时，将路径变量设置成已有的对应目录。
|
||||
|
||||
```powershell
|
||||
conda activate ai-spine
|
||||
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
$runRoot = ".\output\$runId"
|
||||
$reference = ".\output\20260918-183532\sam\source.png"
|
||||
```
|
||||
|
||||
示例复用现有角色原图，可替换 `$reference` 为自己的 PNG。不要混用不同实验的 source、计划和 mask。
|
||||
|
||||
### 第一阶段：理解角色结构
|
||||
|
||||
此命令调用视觉模型：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage understanding `
|
||||
--config .\configs\deepseek-qwen.json `
|
||||
--reference $reference `
|
||||
--out-dir "$runRoot\understand"
|
||||
```
|
||||
|
||||
检查 `understand/report.html`、`semantic-parts.json`、`geometry-validation.json`。程序候选点来自 Alpha 距离变换，落在前景不代表属于正确部件。
|
||||
|
||||
已有计划可离线重新检查；输出独立目录，不重跑模型：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage understanding `
|
||||
--reference .\output\20260918-183532\sam\source.png `
|
||||
--parts-file .\output\20260918-183532\sam\semantic-parts.json `
|
||||
--out-dir "$runRoot\understand-offline"
|
||||
```
|
||||
|
||||
### 第二阶段：本地 SAM 分割
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage segmentation `
|
||||
--input-dir "$runRoot\understand" `
|
||||
--out-dir "$runRoot\sam" `
|
||||
--sam-dir F:/Study/SAM3
|
||||
```
|
||||
|
||||
若使用前面的离线计划，input-dir 改为 understand-offline。该命令只运行本地临时 GPU 子进程；添加 `--dry-run` 可探测环境而不加载模型。检查 `sam/report.html`、`annotated.png`、`parts-sheet.png`、漏分图和 mask 归属，错误部件应先修正。
|
||||
|
||||
### 第三阶段：准备与补全
|
||||
|
||||
默认 `--completion-profile adaptive` 会重新检查实际 SAM 缺口、骨骼连接和附件重叠，即使旧计划 needs_completion=false 也可能生成修复任务。先检查 preparation 报告的区域与参考图，再请求模型；缺口归属和附件底图仍需语义复核。
|
||||
|
||||
先离线准备任务，不请求模型：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir "$runRoot\sam" `
|
||||
--out-dir "$runRoot\complete-prepare" `
|
||||
--prepare-only
|
||||
```
|
||||
|
||||
复核允许区域后，按实际存在的 ID 试一个部件；下面会调用图片模型：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir "$runRoot\sam" `
|
||||
--out-dir "$runRoot\complete-head" `
|
||||
--part head
|
||||
```
|
||||
|
||||
处理所有具备可见 mask、补全需求和允许区域的部件：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir "$runRoot\sam" `
|
||||
--out-dir "$runRoot\complete"
|
||||
```
|
||||
|
||||
省略 part 不意味着强制补全全部 mask。检查 `completion-plan.json` 的 assessments/跳过原因、每个任务的 input.png、request-plan.json 和编辑区域；parts/ 仅包含通过检查的补全部件。所有结果仍需语义复核。`--completion-review FILE` 可按 ID 提供修复/替换/移除区域及部件半径，格式见[第三阶段文档](docs/3-completion-stage.md)。
|
||||
|
||||
`--completion-reference auto` 默认在原生多图可用时发送主图→局部参考→全角色参考，否则使用 board；可显式选择 board/local/full/both/none。`--completion-mask auto|required|off` 控制 mask 请求，auto 遵循配置。`--probe-edit-capabilities --part ID` 是一次真实多图+mask 小样，固定 n=1、无 HTTP 重试，保存原候选及能力证据，不自动启用 capabilities。该实测会产生费用；当前实现只完成离线验证。
|
||||
|
||||
旧候选尺寸不符时，可离线导入复检，避免重新生成：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir .\output\20260918-183532\sam `
|
||||
--candidates-dir .\output\20260918-183532\complete2_gpt `
|
||||
--out-dir "$runRoot\complete-size-recheck" `
|
||||
--completion-profile legacy `
|
||||
--candidate-size-policy uniform
|
||||
```
|
||||
|
||||
旧画布候选必须指定 legacy。adaptive 候选需匹配源图、masks、profile、参考布局、半径和复核文件哈希；不能把旧候选直接映射到新画布。uniform 仅恢复同宽高比整画布尺寸，不纠正物体位移、变形或语义。在线请求显式传 size；参数及中转实测边界见[第三阶段文档](docs/3-completion-stage.md)。
|
||||
|
||||
### 第四阶段：离线组装与 Spine 导出
|
||||
|
||||
接续本轮 SAM 与补全目录，无需模型配置、密钥或 GPU：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage spine `
|
||||
--input-dir "$runRoot\sam" `
|
||||
--completion-dir "$runRoot\complete" `
|
||||
--out-dir "$runRoot\spine"
|
||||
```
|
||||
|
||||
省略 `--completion-dir` 可只用 SAM 资产；加 `--dry-run` 完整检查输入而不写输出。默认补全失败回退 SAM 并记录原因；`--fallback-policy error` 要求需补全的可见部件全部有通过结果。可见部件缺 mask、混用不同输入或非法补全坐标会直接报错。
|
||||
|
||||
检查 `report.html`、`comparison.png` 和 `assembly.json`；`preview.html` 显示真实骨架，setup 为静态姿态，`check__骨骼ID` 为单关节旋转检查动画。生成过程离线；播放器页面从 CDN 加载 Spine Player，需要网络。`skeleton.json` 配合 `images/` 供编辑器导入，`atlas/` 用于运行时。
|
||||
|
||||
部件数量、名称、父子关系均由 v2 计划驱动；隐藏骨骼保留，附件按 bind_to 绑定。第四阶段只复用外部只读 `spine-animation-ai` 的图集和播放器脚本，保留其 PolyForm Noncommercial 约束。完整参数、输入契约和绘制顺序覆盖见[第四阶段文档](docs/4-spine-stage.md)。
|
||||
|
||||
## 旧管线与验证
|
||||
|
||||
旧版依赖相邻 `spine-animation-ai` 外部项目，使用时遵守其 PolyForm Noncommercial 许可证；不要修改外部依赖。旧版命令会调用图片生成并尝试定位、导出：
|
||||
|
||||
```powershell
|
||||
node .\pipeline.mjs `
|
||||
--reference $reference `
|
||||
--out-dir "$runRoot\legacy" `
|
||||
--genielabs-dir ..\spine-animation-ai `
|
||||
--image-model qwen-image-3.0 `
|
||||
--attempts 1 `
|
||||
```bash
|
||||
node pipeline.mjs \
|
||||
--reference /absolute/path/character.png \
|
||||
--out-dir ./output/character-spine \
|
||||
--genielabs-dir /absolute/path/spine-animation-ai \
|
||||
--env-file /absolute/path/.env \
|
||||
--image-model gpt-image-2.5 \
|
||||
--animations idle,walk,wave
|
||||
```
|
||||
|
||||
`segment.py` 为旧版拆件分割，`compose_preview.py` 组装预览，`run_case_matrix.py` 对照定位方案；当前 SAM 入口为 sam_segment.py。旧产物检查 assembled.png 和 preview-rigged.html；新管线导出使用独立的第四阶段入口。
|
||||
脚本会依次执行：
|
||||
|
||||
测试与命令帮助：
|
||||
1. VectorEngine 图片编辑，生成白底拆件工作图；
|
||||
2. OpenCV 连通域拆件，输出透明部件 PNG；
|
||||
3. 默认调用 Genielabs `position_parts.py`，使用 SIFT/RANSAC、模板匹配回退和 z-order 计算把部件放回参考图；只有显式传入 `--layout-mode anchored` 才启用按画布锚点的保守回退；
|
||||
4. 生成 Spine 配置并运行 `build_spine_json.py`；
|
||||
5. 运行 `make_atlas.py` 和 `generate_spine_player.py`。
|
||||
|
||||
```powershell
|
||||
conda run --no-capture-output -n ai-spine npm test
|
||||
node .\tools\model-check.mjs --help
|
||||
为避免“部件爆炸”，定位后默认只保留 SIFT 高置信部件和模板匹配分数不低于 `0.3` 的部件，最多保留 28 个；可用 `--min-score`、`--max-parts` 调整。多候选选择还会比较 Genielabs 定位置信度和部件框重叠，降低重复角色层被选中的概率。若模型仍输出多套角色，先删除输出目录重新生成，不要直接把低质量拆件导入 Spine。
|
||||
|
||||
`matched` 依赖拆件图与参考图保持相同画风和可辨识纹理；低纹理部件会进入模板匹配回退，并进行白底归一化以避免透明像素产生 `NaN/inf`。`anchored` 只按拆件图的行布局做通用兜底，不应作为正式定位结果。
|
||||
|
||||
Python 环境探测顺序为显式 `--python`、Anaconda、Homebrew Python、系统 Python；要求导入 `cv2`、`PIL`、`numpy`。如果全部不可用，脚本会在本目录创建 `.venv` 并安装 `requirements.txt`。
|
||||
|
||||
脚本默认只读取本目录 `.env` 和当前 shell 环境，不会自动读取 Genarrative 的 secrets 文件。可以通过 `--env-file` 显式指定一个只在受控后端使用的环境文件。
|
||||
|
||||
输出目录包含 `generated-parts-sheet.png`、`parts/`、`layout.json`、`assembled.png`、`skeleton.json`、`atlas/`、`preview.html` 和 `manifest.json`。`assembled.png` 是按同一 layout 坐标生成的静态 QA 合成图；自动结果是 region attachments，mesh、weights、IK、FFD 和骨骼语义仍需在 Spine 4.2 中校正。
|
||||
|
||||
`preview.html` 现在显示参考图的真实姿态，适合确认形体是否正常；自动拆件结果放在 `preview-rigged.html`。这是因为参考图可能是左右不对称动作，不能用对称锚点强行还原，否则会出现手臂漂浮或错接。
|
||||
|
||||
## 离线多 case 回归
|
||||
|
||||
`run_case_matrix.py` 可以复用已经生成的多个拆件候选，不调用图片 API，只测试 Genielabs 定位和合成：
|
||||
|
||||
```bash
|
||||
python run_case_matrix.py \
|
||||
--reference output/demo-character-final/source.png \
|
||||
--genielabs-dir /absolute/path/spine-animation-ai \
|
||||
--out-dir output/cases-matrix \
|
||||
--candidate final output/demo-character-final/parts \
|
||||
--candidate attempt1 output/demo-character-multi/candidates/attempt-1/parts \
|
||||
--candidate attempt2 output/demo-character-multi/candidates/attempt-2/parts
|
||||
```
|
||||
|
||||
测试使用本地 mock 或注入响应，不请求真实付费模型。自动化运行任一阶段可用 `conda run --no-capture-output -n ai-spine node ...`。保留旧实验；resume 只复用完整匹配产物，失败任务另建输出目录。
|
||||
每个 case 会输出 `layout.json`、`assembled.png` 和 `case-summary.json`，总报告为 `case-report.json`。当前回归中，`final`/`matched` 保留 9 个有效部件，前景 MAE 约 17.68；`attempt1` 只有 7 个部件且出现手臂误拆，MAE 30.28；`attempt2` 有 15 个部件但包含重复/漂浮部件，MAE 27.59。默认候选选择和重叠过滤会优先保留 `final`/`matched` 这类结果。
|
||||
|
||||
Genielabs 以外部只读目录引用，需遵守其 PolyForm Noncommercial 许可证。生产集成时应把输出上传对象存储，只向前端返回 object key 或 asset object id。
|
||||
|
||||
## 外部参考角色
|
||||
|
||||
已下载 5 个 Kenney CC0 参考素材到 `references/characters/`，可直接作为下一轮图片拆件的 `--reference` 输入；来源、许可证和 SHA-256 见 [references/index.json](references/index.json)，原始包及许可证文件见 `references/external/`。缩略图对比见 [references/characters/contact-sheet.jpg](references/characters/contact-sheet.jpg)。
|
||||
|
||||
另外新增了高分辨率单角色回归素材，位于 `references/highres-characters/`。其中 `tiddybub-toon/crops-768/` 包含 6 个 768×1024、透明背景、CC0 的 Kenney Toon Characters 单角色图，适合测试头发、手臂、手、腿、鞋和机器人关节等不同结构。
|
||||
|
||||
## GPT Image 2 prompt 优化回归
|
||||
|
||||
已对原始 sombrero/cowboy 参考图执行 6 个通用 prompt 变体（每个变体一次请求），并对 6 个高分辨率角色执行最多 3 个候选：female-adventurer、male-adventurer、female-person、male-person、robot、zombie。所有 prompt 都不包含角色名称、左右手硬编码或单个案例的外观描述；候选只由参考图的像素、姿态、遮挡和非对称关系决定。
|
||||
|
||||
当前离线复评结果保存在被忽略的 `output/prompt-optimization/`：
|
||||
|
||||
| 参考角色 | 选中候选 | 拆件数 | 定位方式 | 备注 |
|
||||
| --- | --- | ---: | --- | --- |
|
||||
| cowboy | attempt-4 | 8 | 8 个 SIFT | 当前最佳；平均定位 0.793，重叠惩罚 0 |
|
||||
| female-adventurer | attempt-3 | 16 | 2 个 SIFT + 模板回退 | 形体基本完整，仍需检查衣服和手套层 |
|
||||
| male-adventurer | attempt-2 | 14 | 1 个 SIFT + 模板回退 | 可用作回归输入，细部依赖模板定位 |
|
||||
| male-person | attempt-2 | 17 | 4 个 SIFT + 模板回退 | 需要检查头发、脸和躯干的层级 |
|
||||
| robot | attempt-2 | 17 | 5 个 SIFT + 模板回退 | 机械关节拆分较清楚，部分小件会漂浮 |
|
||||
| zombie | attempt-2 | 11 | 模板回退 | 仍有明显错层，不能视为通过 |
|
||||
| female-person | attempt-1 | 21 | 3 个 SIFT + 模板回退 | 检测到完整人物副本风险，不能直接采用 |
|
||||
|
||||
综合对照图为 `output/prompt-optimization/reselection-contact-sheet.jpg`，机器报告为 `output/prompt-optimization/reselection-report.json`。这些评分用于缩小人工复核范围，不代表自动通过，也不会把完整人物副本误选为拆件。
|
||||
|
||||
如需在不调用图片 API 的情况下重新评估已有候选，可运行：
|
||||
|
||||
```bash
|
||||
python reselect_candidates.py \
|
||||
--out-root output/prompt-optimization \
|
||||
--genielabs-dir /absolute/path/spine-animation-ai
|
||||
```
|
||||
|
||||
本轮新的 GPT Image 请求在 female-person、robot 和 zombie 的后续尝试中收到 VectorEngine `403 user quota is not enough`，因此没有继续重试或伪造新结果。补充额度后，可直接用同一组 `prompts/v2-tested.json` 和 `--attempts 6` 继续 A/B 测试。
|
||||
|
||||
@@ -1,285 +0,0 @@
|
||||
"""Prepare bounded completion canvases and merge generated pixels conservatively."""
|
||||
import argparse
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from normalize_reference import prepare_rgba
|
||||
|
||||
|
||||
def write_json(file, value):
|
||||
Path(file).write_text(json.dumps(value, ensure_ascii=False, indent=2), encoding='utf-8')
|
||||
|
||||
|
||||
def read_json(file):
|
||||
return json.loads(Path(file).read_text(encoding='utf-8-sig'))
|
||||
|
||||
|
||||
def localize(array, origin, size):
|
||||
"""Copy source coordinates to a padded local canvas, without scaling."""
|
||||
out = np.zeros((size, size) + array.shape[2:], dtype=array.dtype)
|
||||
ox, oy = origin
|
||||
h, w = array.shape[:2]
|
||||
x0, y0, x1, y1 = max(0, ox), max(0, oy), min(w, ox + size), min(h, oy + size)
|
||||
if x1 > x0 and y1 > y0:
|
||||
out[y0-oy:y1-oy, x0-ox:x1-ox] = array[y0:y1, x0:x1]
|
||||
return out
|
||||
|
||||
|
||||
def allowed_regions(part, plan, masks, visible, radius):
|
||||
h, w = visible.shape
|
||||
distance = cv2.distanceTransform((~visible).astype(np.uint8), cv2.DIST_L2, 5)
|
||||
band = (distance <= radius) & ~visible
|
||||
regions, warnings = {}, []
|
||||
flags = part['completion']
|
||||
if flags['occlusion_fill']:
|
||||
occluders = np.zeros_like(visible)
|
||||
for name in part['occluded_by']:
|
||||
if name in masks:
|
||||
occluders |= masks[name]
|
||||
regions['occlusion_fill'] = band & occluders
|
||||
if not regions['occlusion_fill'].any():
|
||||
warnings.append('NO_OCCLUSION_REGION')
|
||||
anchors = {}
|
||||
if flags['joint_extension_parent']:
|
||||
anchors['joint_extension_parent'] = [part['pivot_norm']]
|
||||
if flags['joint_extension_child']:
|
||||
anchors['joint_extension_child'] = [p['pivot_norm'] for p in plan['parts']
|
||||
if p['parent'] == part['id'] and p['rig_role'] == 'bone_part']
|
||||
for kind, points in anchors.items():
|
||||
disks = np.zeros((h, w), np.uint8)
|
||||
for x, y in points:
|
||||
cv2.circle(disks, (min(w-1, round(x*w)), min(h-1, round(y*h))), radius, 1, -1)
|
||||
regions[kind] = band & (disks > 0)
|
||||
if not regions[kind].any():
|
||||
warnings.append(f'NO_{kind.upper()}_REGION')
|
||||
allowed = np.zeros_like(visible)
|
||||
for region in regions.values():
|
||||
allowed |= region
|
||||
return allowed, regions, warnings
|
||||
|
||||
|
||||
def prepare(directory, selected_ids, size=1024, radius=32):
|
||||
out = Path(directory)
|
||||
plan, segmentation = read_json(out/'semantic-parts.json'), read_json(out/'segmentation.json')
|
||||
source = np.array(Image.open(out/'source.png').convert('RGBA'))
|
||||
h, w = source.shape[:2]
|
||||
known = {p['id']: p for p in plan['parts']}
|
||||
if set(selected_ids) - set(known):
|
||||
raise ValueError('Unknown requested part ID')
|
||||
records = {p['id']: p for p in segmentation['parts']}
|
||||
if len(records) != len(segmentation['parts']) or set(records) - set(known):
|
||||
raise ValueError('Segmentation IDs do not match semantic plan')
|
||||
masks = {}
|
||||
for name in records:
|
||||
mask = np.array(Image.open(out/'masks'/f'{name}.png').convert('L'))
|
||||
if mask.shape != (h, w) or not np.isin(mask, [0, 255]).all():
|
||||
raise ValueError(f'{name}: mask must be binary and match the source canvas')
|
||||
mask = mask > 0
|
||||
if not mask.any() or (mask & (source[:, :, 3] == 0)).any():
|
||||
raise ValueError(f'{name}: empty mask or mask outside original alpha')
|
||||
masks[name] = mask
|
||||
tasks, skipped = [], []
|
||||
for part in plan['parts']:
|
||||
name = part['id']
|
||||
if selected_ids and name not in selected_ids:
|
||||
continue
|
||||
if not part['needs_completion']:
|
||||
skipped.append({'id': name, 'reason': 'NO_COMPLETION_NEEDED'})
|
||||
continue
|
||||
if name not in masks or part['visibility'] == 'hidden':
|
||||
skipped.append({'id': name, 'reason': 'NO_VISIBLE_SAM_MASK'})
|
||||
continue
|
||||
visible = masks[name]
|
||||
allowed, regions, warnings = allowed_regions(part, plan, masks, visible, radius)
|
||||
if not allowed.any():
|
||||
skipped.append({'id': name, 'reason': 'NO_ALLOWED_EXTENSION', 'warnings': warnings})
|
||||
continue
|
||||
ys, xs = np.nonzero(visible | allowed)
|
||||
bw, bh = int(xs.max()-xs.min()+1), int(ys.max()-ys.min()+1)
|
||||
if max(bw, bh) + 32 > size:
|
||||
raise ValueError(f'{name}: completion canvas too small; increase --completion-size (no resizing is performed)')
|
||||
origin = [int(xs.min()) - (size-bw)//2, int(ys.min()) - (size-bh)//2]
|
||||
folder = out/'tasks'/name
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
cut = source.copy()
|
||||
cut[~visible] = 0
|
||||
original = localize(cut, origin, size)
|
||||
editable = localize(allowed, origin, size)
|
||||
Image.fromarray(original).save(folder/'visible.png')
|
||||
white = Image.new('RGBA', (size, size), 'white')
|
||||
Image.alpha_composite(white, Image.fromarray(original)).convert('RGB').save(folder/'input.png')
|
||||
context = Image.fromarray(localize(source, origin, size))
|
||||
Image.alpha_composite(white, context).convert('RGB').save(folder/'context.png')
|
||||
edit_mask = np.full((size, size, 4), 255, np.uint8)
|
||||
edit_mask[editable, 3] = 0
|
||||
Image.fromarray(edit_mask).save(folder/'edit-mask.png')
|
||||
Image.fromarray(editable.astype(np.uint8)*255).save(folder/'allowed-mask.png')
|
||||
local_regions, region_bounds = {}, {}
|
||||
for kind, region in regions.items():
|
||||
local = localize(region, origin, size)
|
||||
Image.fromarray(local.astype(np.uint8)*255).save(folder/f'{kind}.png')
|
||||
local_regions[kind] = int(local.sum())
|
||||
ry, rx = np.nonzero(local)
|
||||
region_bounds[kind] = [int(rx.min()), int(ry.min()), int(rx.max())+1, int(ry.max())+1] if len(rx) else None
|
||||
task = {'id': name, 'semantic': part['semantic'], 'completion': part['completion'],
|
||||
'origin_xy': origin, 'canvas_size': [size, size], 'source_size': [w, h],
|
||||
'allowed_pixels': int(editable.sum()), 'regions': local_regions,
|
||||
'region_bounds_xyxy': region_bounds, 'occluded_by': part['occluded_by'],
|
||||
'warnings': records[name].get('warnings', []) + warnings,
|
||||
'semantic_verified': False, 'status': 'prepared'}
|
||||
write_json(folder/'task.json', task)
|
||||
tasks.append(task)
|
||||
result = {'schema_version': 1, 'tasks': tasks, 'skipped': skipped,
|
||||
'quality_status': 'needs_review', 'canvas_size': size, 'extension_radius': radius}
|
||||
write_json(out/'completion-plan.json', result)
|
||||
return result
|
||||
|
||||
|
||||
def merge_candidate(folder, candidate_file, output_file, size_policy='strict'):
|
||||
if size_policy not in ['strict', 'uniform']:
|
||||
raise ValueError('Unknown candidate size policy')
|
||||
folder = Path(folder)
|
||||
task = read_json(folder/'task.json')
|
||||
if task.get('profile') == 'adaptive':
|
||||
from completion_planning import merge_adaptive
|
||||
return merge_adaptive(folder, candidate_file, output_file, size_policy)
|
||||
original = np.array(Image.open(folder/'visible.png').convert('RGBA'))
|
||||
allowed = np.array(Image.open(folder/'allowed-mask.png').convert('L')) > 0
|
||||
candidate = Image.open(candidate_file).convert('RGBA')
|
||||
qa = {'accepted': False, 'semantic_verified': False, 'reasons': [],
|
||||
'expected_size': task['canvas_size'], 'returned_size': list(candidate.size),
|
||||
'size_policy': size_policy, 'canvas_resampled': False}
|
||||
if list(candidate.size) != task['canvas_size']:
|
||||
if size_policy == 'strict':
|
||||
qa['reasons'].append('CANVAS_SIZE_MISMATCH')
|
||||
return qa
|
||||
tw, th = task['canvas_size']
|
||||
cw, ch = candidate.size
|
||||
if cw * th != ch * tw:
|
||||
qa['reasons'].append('CANVAS_ASPECT_RATIO_MISMATCH')
|
||||
return qa
|
||||
# Only reverse uniform whole-canvas scaling. Do not translate, stretch,
|
||||
# crop, or register the object to make a changed pose appear correct.
|
||||
candidate = candidate.resize((tw, th), Image.Resampling.LANCZOS)
|
||||
qa.update(canvas_resampled=True, canvas_scale=tw/cw,
|
||||
resampling='LANCZOS', warnings=['CANDIDATE_CANVAS_RESAMPLED'])
|
||||
candidate.save(Path(output_file).with_name(Path(output_file).stem + '-normalized.png'))
|
||||
try:
|
||||
candidate, preprocessing = prepare_rgba(candidate)
|
||||
except ValueError as error:
|
||||
qa['reasons'].append('CANNOT_EXTRACT_FOREGROUND')
|
||||
qa['detail'] = str(error)
|
||||
return qa
|
||||
generated = np.array(candidate)
|
||||
visible = original[:, :, 3] > 0
|
||||
opaque = original[:, :, 3] >= 240
|
||||
if not opaque.any():
|
||||
qa['reasons'].append('NO_OPAQUE_ALIGNMENT_ANCHOR')
|
||||
return qa
|
||||
# Detect gross movement/redesign before restoring original pixels, otherwise
|
||||
# restoration could hide a shifted generation attached at the wrong place.
|
||||
white = Image.new('RGBA', candidate.size, 'white')
|
||||
generated_rgb = np.array(Image.alpha_composite(white, candidate))[:, :, :3].astype(float)
|
||||
mae = float(np.abs(generated_rgb[opaque] - original[:, :, :3][opaque]).mean())
|
||||
retained = float((generated[:, :, 3][opaque] >= 240).mean())
|
||||
proposed = (generated[:, :, 3] > 0) & ~visible
|
||||
added = proposed & allowed
|
||||
_, labels = cv2.connectedComponents((visible | added).astype(np.uint8), connectivity=8)
|
||||
touching = np.unique(labels[visible])
|
||||
connected = added & np.isin(labels, touching[touching != 0])
|
||||
region_additions = {kind: int((connected & (np.array(Image.open(folder/f'{kind}.png')) > 0)).sum())
|
||||
for kind in task['regions']}
|
||||
missing_regions = [kind for kind, pixels in task['regions'].items() if not pixels or not region_additions[kind]]
|
||||
qa.update(alignment_mae=mae, visible_retention=retained,
|
||||
new_pixels=int(connected.sum()), discarded_outside_allowed=int((proposed & ~allowed).sum()),
|
||||
disconnected_pixels=int((added & ~connected).sum()), region_additions=region_additions,
|
||||
unfulfilled_regions=missing_regions, preprocessing=preprocessing)
|
||||
if mae > 30 or retained < .95:
|
||||
qa['reasons'].append('VISIBLE_ALIGNMENT_MISMATCH')
|
||||
if not connected.any():
|
||||
qa['reasons'].append('NO_CONNECTED_NEW_PIXELS')
|
||||
if qa['reasons']:
|
||||
return qa
|
||||
merged = np.zeros_like(original)
|
||||
merged[connected] = generated[connected]
|
||||
merged[visible] = original[visible]
|
||||
qa['visible_pixels_exact'] = bool(np.array_equal(merged[visible], original[visible]))
|
||||
qa['accepted'] = True
|
||||
qa['quality_status'] = 'needs_review'
|
||||
Image.fromarray(merged).save(output_file)
|
||||
return qa
|
||||
|
||||
|
||||
def finalize(directory):
|
||||
out = Path(directory)
|
||||
result = read_json(out/'completion-result.json')
|
||||
rows = []
|
||||
for task in result['parts']:
|
||||
name = task['id']
|
||||
qa = task.get('qa', {})
|
||||
if task.get('selected_file'):
|
||||
im = Image.open(out/task['selected_file']).convert('RGBA')
|
||||
bbox = im.getbbox()
|
||||
crop = im.crop(bbox)
|
||||
(out/'parts').mkdir(exist_ok=True)
|
||||
crop.save(out/'parts'/f'{name}.png')
|
||||
meta = read_json(out/'tasks'/name/'task.json')
|
||||
ox, oy = meta['origin_xy']
|
||||
task.update(image=f'parts/{name}.png', crop_xyxy=[bbox[0]+ox, bbox[1]+oy, bbox[2]+ox, bbox[3]+oy])
|
||||
if meta.get('preservation'):
|
||||
task['preservation'] = meta['preservation']
|
||||
task['transform'] = meta['transform']
|
||||
view = f'<img src="{task["image"]}">' if task.get('image') else ''
|
||||
for index, item in enumerate(task.get('candidates', []), 1):
|
||||
view += f'<p><a href="{html.escape(item["file"], quote=True)}">原候选 {index}</a> · <a href="tasks/{name}/qa-{index}.json">QA</a>'
|
||||
if item['qa'].get('canvas_resampled'):
|
||||
view += f' · <a href="tasks/{name}/merged-{index}-normalized.png">缩放后候选</a>'
|
||||
view += '</p>'
|
||||
warnings = html.escape(', '.join(task.get('upstream_warnings', [])))
|
||||
reference = f'<p><a href="tasks/{name}/input.png">实际请求主图</a> · <a href="tasks/{name}/request-plan.json">发送计划</a></p>'
|
||||
if (out/'tasks'/name/'reference.png').exists():
|
||||
reference += f'<a href="tasks/{name}/reference.png">完整角色参考</a>'
|
||||
rows.append(f'<tr><td>{html.escape(name)}<p>{warnings}</p><a href="tasks/{name}/task.json">任务参数</a> · <a href="tasks/{name}/prompt.txt">提示词</a>{reference}</td><td>{html.escape(task["status"])}</td><td><a href="tasks/{name}/visible.png"><img src="tasks/{name}/visible.png"></a></td><td><a href="tasks/{name}/allowed-mask.png"><img src="tasks/{name}/allowed-mask.png"></a><p><a href="tasks/{name}/context.png">原图局部参考</a></p></td><td>{view}</td><td><pre>{html.escape(json.dumps(qa, ensure_ascii=False, indent=2))}</pre></td></tr>')
|
||||
write_json(out/'completion-result.json', result)
|
||||
(out/'report.html').write_text('''<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>部件补全</title>
|
||||
<style>body{font:16px system-ui;margin:24px}td,th{border:1px solid #ccc;padding:8px}img{max-width:260px;max-height:300px;background:#eee}pre{white-space:pre-wrap}table{border-collapse:collapse}</style>
|
||||
<h1>第三阶段：部件补全</h1><p>输出受允许掩码限制，保留区域逐像素恢复；adaptive 允许明确规划的附件清理、替换和移除，原始 SAM 始终保留。几何通过仅表示候选可供复核，不代表遮挡和关节纹理已正确补全。</p>
|
||||
<table><tr><th>部件 / 上游告警</th><th>状态</th><th>原可见区域</th><th>允许新增区域（白色）</th><th>补全候选</th><th>几何检查</th></tr>'''+''.join(rows)+'</table><h2>跳过</h2><pre>'+html.escape(json.dumps(result['skipped'], ensure_ascii=False, indent=2))+'</pre></html>', encoding='utf-8')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('action', choices=['prepare', 'merge', 'finalize'])
|
||||
parser.add_argument('--out', required=True)
|
||||
parser.add_argument('--part', action='append', default=[])
|
||||
parser.add_argument('--size', type=int, default=1024)
|
||||
parser.add_argument('--radius', type=int)
|
||||
parser.add_argument('--profile', choices=['legacy', 'adaptive'], default='legacy')
|
||||
parser.add_argument('--reference-mode', choices=['none', 'local', 'full', 'both', 'board'], default='board')
|
||||
parser.add_argument('--review-file')
|
||||
parser.add_argument('--candidate')
|
||||
parser.add_argument('--merged')
|
||||
parser.add_argument('--qa')
|
||||
parser.add_argument('--candidate-size-policy', choices=['strict', 'uniform'], default='strict')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'prepare':
|
||||
if not 128 <= args.size <= 4096 or (args.radius is not None and not 1 <= args.radius <= 256):
|
||||
parser.error('size must be 128..4096 and radius 1..256')
|
||||
if args.profile == 'adaptive':
|
||||
from completion_planning import prepare_adaptive
|
||||
prepare_adaptive(args.out, args.part, args.size, args.radius, args.reference_mode, args.review_file)
|
||||
else:
|
||||
prepare(args.out, args.part, args.size, args.radius or 32)
|
||||
elif args.action == 'merge':
|
||||
write_json(args.qa, merge_candidate(args.out, args.candidate, args.merged, args.candidate_size_policy))
|
||||
else:
|
||||
finalize(args.out)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,343 +0,0 @@
|
||||
"""Adaptive completion planning, reference canvases and reversible native-pixel merging."""
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
from normalize_reference import prepare_rgba
|
||||
|
||||
|
||||
def read(file):
|
||||
return json.loads(Path(file).read_text(encoding='utf-8-sig'))
|
||||
|
||||
|
||||
def write(file, value):
|
||||
Path(file).write_text(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False), encoding='utf-8')
|
||||
|
||||
|
||||
def binary(file, shape):
|
||||
a = np.array(Image.open(file))
|
||||
if a.shape != shape or not np.isin(a, [0, 255]).all():
|
||||
raise ValueError(f'Expected full-canvas binary PNG: {file}')
|
||||
return a > 0
|
||||
|
||||
|
||||
def review_inputs(file, ids, shape):
|
||||
"""Review masks are explicitly authored data; never guess ownership from scores."""
|
||||
if not file:
|
||||
return {}, {}
|
||||
file = Path(file).resolve()
|
||||
data = read(file)
|
||||
if data.get('schema_version') != 1 or not isinstance(data.get('parts'), dict):
|
||||
raise ValueError('Completion review requires schema_version=1 and parts object')
|
||||
hashes = {'review.json': hashlib.sha256(file.read_bytes()).hexdigest()}
|
||||
result = {}
|
||||
for name, entry in data['parts'].items():
|
||||
if name not in ids or not isinstance(entry, dict) or entry.get('confirmed') is not True:
|
||||
raise ValueError(f'Review entry requires known ID and confirmed=true: {name}')
|
||||
if set(entry) - {'confirmed', 'extension_radius', 'repair_mask', 'replace_mask', 'remove_mask'}:
|
||||
raise ValueError(f'Unknown review fields: {name}')
|
||||
item = {'confirmed': True}
|
||||
if 'extension_radius' in entry:
|
||||
radius = entry['extension_radius']
|
||||
if type(radius) is not int or not 1 <= radius <= 256:
|
||||
raise ValueError(f'Invalid reviewed radius: {name}')
|
||||
item['extension_radius'] = radius
|
||||
for key in ['repair_mask', 'replace_mask', 'remove_mask']:
|
||||
if key not in entry:
|
||||
continue
|
||||
ref = entry[key]
|
||||
if not isinstance(ref, str) or Path(ref).is_absolute():
|
||||
raise ValueError('Review mask must be relative to review file')
|
||||
mask_file = (file.parent / ref).resolve()
|
||||
if not mask_file.is_relative_to(file.parent):
|
||||
raise ValueError('Review mask path escapes review directory')
|
||||
item[key] = binary(mask_file, shape)
|
||||
hashes[f'{name}/{key}'] = hashlib.sha256(mask_file.read_bytes()).hexdigest()
|
||||
result[name] = item
|
||||
return result, hashes
|
||||
|
||||
|
||||
def bounding(mask):
|
||||
ys, xs = np.nonzero(mask)
|
||||
return [int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1] if len(xs) else None
|
||||
|
||||
|
||||
def enclosed(mask):
|
||||
padded = np.pad(mask.astype(np.uint8), 1)
|
||||
fill = padded.copy()
|
||||
cv2.floodFill(fill, None, (0, 0), 1)
|
||||
return ~fill[1:-1, 1:-1].astype(bool)
|
||||
|
||||
|
||||
def white_image(array):
|
||||
im = Image.fromarray(array).convert('RGBA')
|
||||
return Image.alpha_composite(Image.new('RGBA', im.size, 'white'), im).convert('RGB')
|
||||
|
||||
|
||||
def prepare_adaptive(directory, selected_ids, size=1024, radius=None, reference_mode='board', review_file=None):
|
||||
# Import shared native coordinate helpers lazily to avoid a module cycle.
|
||||
from completion_geometry import localize, allowed_regions
|
||||
out = Path(directory)
|
||||
plan, seg = read(out / 'semantic-parts.json'), read(out / 'segmentation.json')
|
||||
rgba = np.array(Image.open(out / 'source.png').convert('RGBA'))
|
||||
h, w = rgba.shape[:2]
|
||||
ids = {p['id']: p for p in plan['parts']}
|
||||
if set(selected_ids) - set(ids):
|
||||
raise ValueError('Unknown requested part ID')
|
||||
records = {p['id']: p for p in seg['parts']}
|
||||
if len(records) != len(seg['parts']) or set(records) - set(ids):
|
||||
raise ValueError('Invalid segmentation IDs')
|
||||
masks = {name: binary(out / 'masks' / f'{name}.png', (h, w)) for name in records}
|
||||
for name, mask in masks.items():
|
||||
if not mask.any() or np.any(mask & (rgba[:, :, 3] == 0)):
|
||||
raise ValueError(f'Empty mask or mask outside original alpha: {name}')
|
||||
reviews, review_hashes = review_inputs(review_file, ids, (h, w))
|
||||
coverage = np.zeros((h, w), np.uint16)
|
||||
for mask in masks.values():
|
||||
coverage += mask
|
||||
tasks, skipped, assessments = [], [], []
|
||||
(out / 'preservation').mkdir(exist_ok=True)
|
||||
for part in plan['parts']:
|
||||
name = part['id']
|
||||
if selected_ids and name not in selected_ids:
|
||||
continue
|
||||
if name not in masks or part['visibility'] == 'hidden':
|
||||
skipped.append({'id': name, 'reason': 'NO_VISIBLE_SAM_MASK'})
|
||||
continue
|
||||
visible = masks[name]
|
||||
box = bounding(visible)
|
||||
review = reviews.get(name, {})
|
||||
resolved_radius = review.get('extension_radius', radius if radius is not None else
|
||||
int(np.clip(math.ceil(min(box[2]-box[0], box[3]-box[1]) * .15), 4, 96)))
|
||||
inferred = dict(part)
|
||||
inferred['completion'] = {
|
||||
'occlusion_fill': bool(part['occluded_by']),
|
||||
'joint_extension_parent': part['rig_role'] == 'bone_part' and part['parent'] is not None,
|
||||
'joint_extension_child': part['rig_role'] == 'bone_part' and any(
|
||||
p['parent'] == name and p['rig_role'] == 'bone_part' for p in plan['parts'])}
|
||||
_, regions, warnings = allowed_regions(inferred, plan, masks, visible, resolved_radius)
|
||||
# Only enclosed source foreground without another mask owner is an automatic
|
||||
# repair candidate. Open gaps and ambiguous ownership stay diagnostic.
|
||||
gap = enclosed(visible) & (rgba[:, :, 3] > 0) & (coverage == 0)
|
||||
cleanup = np.zeros_like(visible)
|
||||
for front in part['occluded_by']:
|
||||
p = ids[front]
|
||||
if front in masks and p['rig_role'] != 'bone_part' and p['bind_to'] == name:
|
||||
cleanup |= visible & masks[front]
|
||||
replace = review.get('replace_mask', np.zeros_like(visible))
|
||||
removed = review.get('remove_mask', np.zeros_like(visible))
|
||||
repair = review.get('repair_mask', np.zeros_like(visible))
|
||||
if np.any((replace | removed) & ~visible) or np.any(repair & visible) or np.any(replace & removed):
|
||||
raise ValueError(f'{name}: replacement/removal must be disjoint subsets of SAM; repair must be outside SAM')
|
||||
regions['segmentation_gap'] = gap | repair
|
||||
regions['attachment_cleanup'] = cleanup & ~removed
|
||||
regions['reviewed_replacement'] = replace
|
||||
allowed = np.zeros_like(visible)
|
||||
for region in regions.values():
|
||||
allowed |= region
|
||||
allowed &= ~removed
|
||||
protected = visible & ~allowed & ~removed
|
||||
unexpected = []
|
||||
duplicates = []
|
||||
for other, mask in masks.items():
|
||||
if other == name:
|
||||
continue
|
||||
overlap = int((mask & visible).sum())
|
||||
related = other in part['occluded_by'] or name in ids[other]['occluded_by']
|
||||
if overlap and not related:
|
||||
unexpected.append({'id': other, 'pixels': overlap})
|
||||
if overlap / max(1, int((mask | visible).sum())) >= .8:
|
||||
duplicates.append(other)
|
||||
assessment = {'id': name, 'original_needs_completion': part['needs_completion'],
|
||||
'inferred_completion': inferred['completion'], 'radius': resolved_radius,
|
||||
'region_pixels': {k: int(v.sum()) for k, v in regions.items()},
|
||||
'protected_pixels': int(protected.sum()), 'removed_pixels': int(removed.sum()),
|
||||
'unexpected_overlaps': unexpected, 'review_confirmed': bool(review)}
|
||||
assessments.append(assessment)
|
||||
if duplicates and not review:
|
||||
skipped.append({'id': name, 'reason': 'AMBIGUOUS_MASK_REQUIRES_REVIEW', 'overlaps': duplicates})
|
||||
continue
|
||||
if not protected.any() or not (protected & (rgba[:, :, 3] >= 240)).any():
|
||||
skipped.append({'id': name, 'reason': 'NO_PRESERVED_ALIGNMENT_ANCHOR'})
|
||||
continue
|
||||
if not allowed.any() and not removed.any():
|
||||
skipped.append({'id': name, 'reason': 'NO_OBSERVED_EDIT_REGION'})
|
||||
continue
|
||||
region_box = bounding(visible | allowed)
|
||||
bw, bh = region_box[2] - region_box[0], region_box[3] - region_box[1]
|
||||
native_size = max(8, math.ceil(max(bw, bh) * 1.15))
|
||||
origin = [region_box[0] - (native_size-bw)//2, region_box[1] - (native_size-bh)//2]
|
||||
work_size = int(size * .68) if reference_mode == 'board' else size
|
||||
offset = [int(size * .02), (size-work_size)//2] if reference_mode == 'board' else [0, 0]
|
||||
scale = work_size / native_size
|
||||
folder = out / 'tasks' / name
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
native_source = localize(rgba, origin, native_size)
|
||||
native_visible = localize(visible, origin, native_size)
|
||||
native_keep = localize(protected, origin, native_size)
|
||||
native_edit = localize(allowed, origin, native_size)
|
||||
native_remove = localize(removed, origin, native_size)
|
||||
cut = native_source.copy()
|
||||
cut[~native_keep] = 0
|
||||
Image.fromarray(cut).save(folder / 'native-visible.png')
|
||||
Image.fromarray(native_source).save(folder / 'native-source.png')
|
||||
for label, mask in [('keep', native_keep), ('allowed', native_edit), ('remove', native_remove), ('sam', native_visible)]:
|
||||
Image.fromarray(mask.astype(np.uint8)*255).save(folder / f'native-{label}.png')
|
||||
|
||||
def project(array, is_mask=False):
|
||||
small = np.array(Image.fromarray(array.astype(np.uint8)*255 if is_mask else array).resize(
|
||||
(work_size, work_size), Image.Resampling.NEAREST if is_mask else Image.Resampling.LANCZOS))
|
||||
canvas = np.zeros((size, size) + small.shape[2:], np.uint8)
|
||||
canvas[offset[1]:offset[1]+work_size, offset[0]:offset[0]+work_size] = small
|
||||
return canvas
|
||||
|
||||
model_visible = project(cut)
|
||||
Image.fromarray(model_visible).save(folder / 'visible.png')
|
||||
model_allowed = project(native_edit, True) > 0
|
||||
model_remove = project(native_remove, True) > 0
|
||||
Image.fromarray(model_allowed.astype(np.uint8)*255).save(folder / 'allowed-mask.png')
|
||||
mask_rgba = np.full((size, size, 4), 255, np.uint8)
|
||||
mask_rgba[model_allowed | model_remove, 3] = 0
|
||||
Image.fromarray(mask_rgba).save(folder / 'edit-mask.png')
|
||||
context = white_image(project(native_source))
|
||||
context.save(folder / 'context.png')
|
||||
full_reference = white_image(rgba)
|
||||
full_reference.thumbnail((size, size), Image.Resampling.LANCZOS)
|
||||
full_reference.save(folder / 'reference.png')
|
||||
model_input = white_image(model_visible)
|
||||
panel_boxes = []
|
||||
if reference_mode == 'board':
|
||||
draw = ImageDraw.Draw(model_input)
|
||||
x = int(size * .74)
|
||||
draw.line([(x-4, 0), (x-4, size)], fill='#777777', width=1)
|
||||
for label, image, top in [('LOCAL REFERENCE', white_image(native_source), 0),
|
||||
('FULL REFERENCE', full_reference.copy(), size//2)]:
|
||||
draw.text((x, top+2), label, fill='black')
|
||||
image.thumbnail((size-x-4, size//2-24), Image.Resampling.LANCZOS)
|
||||
model_input.paste(image, (x, top+22))
|
||||
panel_boxes.append([x, top+22, x+image.width, top+22+image.height])
|
||||
model_input.save(folder / 'input.png')
|
||||
bounds, native_counts = {}, {}
|
||||
for kind, region in regions.items():
|
||||
native = localize(region & allowed, origin, native_size)
|
||||
model = project(native, True)
|
||||
Image.fromarray(native.astype(np.uint8)*255).save(folder / f'native-{kind}.png')
|
||||
Image.fromarray(model).save(folder / f'{kind}.png')
|
||||
bounds[kind] = bounding(model > 0)
|
||||
native_counts[kind] = int(native.sum())
|
||||
preservation = {}
|
||||
for key, mask in [('preserve_mask', protected), ('editable_mask', allowed), ('removed_mask', removed)]:
|
||||
relative = f'preservation/{name}-{key}.png'
|
||||
Image.fromarray(mask.astype(np.uint8)*255).save(out / relative)
|
||||
preservation[key] = relative
|
||||
if unexpected:
|
||||
warnings.append('UNEXPECTED_OVERLAP_REVIEW_REQUIRED')
|
||||
if cleanup.any():
|
||||
warnings.append('ATTACHMENT_BASE_CLEANUP_NEEDS_REVIEW')
|
||||
task = {'id': name, 'semantic': part['semantic'], 'profile': 'adaptive',
|
||||
'completion': inferred['completion'], 'original_completion': part['completion'],
|
||||
'origin_xy': origin, 'native_size': native_size, 'canvas_size': [size, size], 'source_size': [w, h],
|
||||
'extension_radius': resolved_radius, 'allowed_pixels': int(native_edit.sum()),
|
||||
'removed_pixels': int(native_remove.sum()), 'regions': native_counts,
|
||||
'region_bounds_xyxy': bounds, 'occluded_by': part['occluded_by'],
|
||||
'warnings': records[name].get('warnings', []) + warnings, 'preservation': preservation,
|
||||
'reference_mode': reference_mode, 'reference_panel_boxes': panel_boxes,
|
||||
'transform': {'scale': scale, 'working_xyxy': [*offset, offset[0]+work_size, offset[1]+work_size],
|
||||
'source_to_canvas': [scale, 0, offset[0]-scale*origin[0]+(scale-1)/2,
|
||||
0, scale, offset[1]-scale*origin[1]+(scale-1)/2],
|
||||
'canvas_to_source': [1/scale, 0, origin[0]-offset[0]/scale+(1/scale-1)/2,
|
||||
0, 1/scale, origin[1]-offset[1]/scale+(1/scale-1)/2]},
|
||||
'semantic_verified': False, 'status': 'prepared'}
|
||||
write(folder / 'task.json', task)
|
||||
tasks.append(task)
|
||||
result = {'schema_version': 2, 'profile': 'adaptive', 'tasks': tasks, 'skipped': skipped,
|
||||
'assessments': assessments, 'review_hashes': review_hashes, 'reference_mode': reference_mode,
|
||||
'canvas_size': size, 'quality_status': 'needs_review'}
|
||||
write(out / 'completion-plan.json', result)
|
||||
return result
|
||||
|
||||
|
||||
def merge_adaptive(folder, candidate_file, output_file, size_policy):
|
||||
folder = Path(folder)
|
||||
task = read(folder / 'task.json')
|
||||
candidate = Image.open(candidate_file).convert('RGBA')
|
||||
qa = {'accepted': False, 'semantic_verified': False, 'reasons': [], 'profile': 'adaptive',
|
||||
'expected_size': task['canvas_size'], 'returned_size': list(candidate.size),
|
||||
'size_policy': size_policy, 'canvas_resampled': False, 'transform': task['transform']}
|
||||
tw, th = task['canvas_size']
|
||||
if candidate.size != (tw, th):
|
||||
if size_policy == 'strict' or candidate.width * th != candidate.height * tw:
|
||||
qa['reasons'].append('CANVAS_SIZE_MISMATCH' if size_policy == 'strict' else 'CANVAS_ASPECT_RATIO_MISMATCH')
|
||||
return qa
|
||||
qa.update(canvas_resampled=True, canvas_scale=tw/candidate.width, resampling='LANCZOS')
|
||||
candidate = candidate.resize((tw, th), Image.Resampling.LANCZOS)
|
||||
candidate.save(Path(output_file).with_name(Path(output_file).stem + '-normalized.png'))
|
||||
# Record raw behavior before masking/restoration; an accepted HTTP response is
|
||||
# not evidence that the relay/model respected the supplied edit mask.
|
||||
expected_input = np.array(Image.open(folder / 'input.png').convert('RGB')).astype(float)
|
||||
raw = np.array(white_image(np.array(candidate))).astype(float)
|
||||
editable = np.array(Image.open(folder / 'edit-mask.png'))[:, :, 3] == 0
|
||||
outside_delta = np.abs(raw - expected_input).mean(axis=2)
|
||||
qa['raw_outside_edit_mae'] = float(outside_delta[~editable].mean()) if (~editable).any() else 0
|
||||
qa['raw_outside_edit_changed_pixels'] = int(((outside_delta > 30) & ~editable).sum())
|
||||
rect = task['transform']['working_xyxy']
|
||||
try:
|
||||
cropped, preprocessing = prepare_rgba(candidate.crop(rect))
|
||||
except ValueError as error:
|
||||
qa.update(reasons=['CANNOT_EXTRACT_FOREGROUND'], detail=str(error))
|
||||
return qa
|
||||
side = task['native_size']
|
||||
generated = np.array(cropped.resize((side, side), Image.Resampling.LANCZOS))
|
||||
original = np.array(Image.open(folder / 'native-visible.png').convert('RGBA'))
|
||||
source = np.array(Image.open(folder / 'native-source.png').convert('RGBA'))
|
||||
keep = binary(folder / 'native-keep.png', (side, side))
|
||||
allowed = binary(folder / 'native-allowed.png', (side, side))
|
||||
removed = binary(folder / 'native-remove.png', (side, side))
|
||||
# Judge alignment before the inverse resampling. Alpha at native boundary
|
||||
# pixels changes during interpolation even for an exactly aligned candidate.
|
||||
model_original = np.array(Image.open(folder / 'visible.png').convert('RGBA').crop(rect))
|
||||
model_generated = np.array(cropped)
|
||||
opaque = model_original[:, :, 3] >= 240
|
||||
if not opaque.any():
|
||||
qa['reasons'].append('NO_OPAQUE_ALIGNMENT_ANCHOR')
|
||||
return qa
|
||||
rgb = np.array(white_image(model_generated)).astype(float)
|
||||
mae = float(np.abs(rgb[opaque] - model_original[:, :, :3][opaque]).mean())
|
||||
retained = float((model_generated[:, :, 3][opaque] >= 240).mean())
|
||||
proposed = (generated[:, :, 3] > 0) & ~keep
|
||||
additions = proposed & allowed
|
||||
_, labels = cv2.connectedComponents((keep | additions).astype(np.uint8), connectivity=8)
|
||||
touching = np.unique(labels[keep])
|
||||
connected = additions & np.isin(labels, touching[touching != 0])
|
||||
counts = {kind: int((connected & binary(folder / f'native-{kind}.png', (side, side))).sum())
|
||||
for kind in task['regions']}
|
||||
qa.update(alignment_mae=mae, visible_retention=retained, alignment_space='request_work_panel', new_pixels=int(connected.sum()),
|
||||
removed_pixels=int(removed.sum()), region_additions=counts,
|
||||
unfulfilled_regions=[k for k, n in task['regions'].items() if n and not counts[k]],
|
||||
discarded_outside_allowed=int((proposed & ~allowed).sum()), preprocessing=preprocessing)
|
||||
if mae > 30 or retained < .95:
|
||||
qa['reasons'].append('VISIBLE_ALIGNMENT_MISMATCH')
|
||||
if not connected.any() and not removed.any():
|
||||
qa['reasons'].append('NO_CONNECTED_NEW_PIXELS')
|
||||
for kind in ['attachment_cleanup', 'reviewed_replacement']:
|
||||
if task['regions'][kind] and counts[kind] / task['regions'][kind] < .95:
|
||||
qa['reasons'].append(f'INCOMPLETE_{kind.upper()}')
|
||||
if qa['reasons']:
|
||||
return qa
|
||||
merged = np.zeros_like(original)
|
||||
merged[connected] = generated[connected]
|
||||
# Visible segmentation gaps already have ground-truth texture in source.
|
||||
# Restore it where the proposed fill connects, rather than trusting a repaint.
|
||||
repair = binary(folder / 'native-segmentation_gap.png', (side, side)) & connected & (source[:, :, 3] > 0)
|
||||
merged[repair] = source[repair]
|
||||
merged[keep] = original[keep]
|
||||
merged[removed] = 0
|
||||
qa.update(accepted=True, quality_status='needs_review', visible_pixels_exact=True,
|
||||
preservation_scope='preserve_mask', repaired_source_pixels=int(repair.sum()))
|
||||
Image.fromarray(merged).save(output_file)
|
||||
return qa
|
||||
@@ -1,45 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"providers": {
|
||||
"relay-vision": {
|
||||
"protocol": "responses",
|
||||
"base_url_env": "VECTOR_ENGINE_BASE_URL",
|
||||
"api_key_env": "VECTOR_ENGINE_API_KEY",
|
||||
"timeout_ms": 180000,
|
||||
"max_retries": 2
|
||||
},
|
||||
"relay-image-edit": {
|
||||
"protocol": "images-edits",
|
||||
"base_url_env": "VECTOR_ENGINE_BASE_URL",
|
||||
"api_key_env": "VECTOR_ENGINE_API_KEY",
|
||||
"timeout_ms": 1200000,
|
||||
"max_retries": 2,
|
||||
"capabilities": {"multiple_images": false, "mask": false},
|
||||
"image_field": "image",
|
||||
"mask_editable": "transparent"
|
||||
}
|
||||
},
|
||||
"stages": {
|
||||
"understanding": {
|
||||
"provider": "relay-vision",
|
||||
"model_env": "SPINE_UNDERSTANDING_MODEL",
|
||||
"default_model": "gpt-6-astra",
|
||||
"prompt": "../prompts/understand-v1.txt",
|
||||
"output_mode": "text-json",
|
||||
"stream": true,
|
||||
"params": {"store": false}
|
||||
},
|
||||
"completion": {
|
||||
"provider": "relay-image-edit",
|
||||
"model_env": "SPINE_COMPLETION_MODEL",
|
||||
"default_model": "qwen-image-3.0",
|
||||
"prompt": "../prompts/complete-v1.txt",
|
||||
"params": {"n": 1}
|
||||
},
|
||||
"segmentation": {
|
||||
"provider": "sam-local",
|
||||
"model_dir_env": "SAM_MODEL_DIR",
|
||||
"default_model_dir": "F:/Study/SAM3"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,45 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"providers": {
|
||||
"relay-vision": {
|
||||
"protocol": "chat-completions",
|
||||
"base_url_env": "VECTOR_ENGINE_BASE_URL",
|
||||
"api_key_env": "VECTOR_ENGINE_API_KEY",
|
||||
"timeout_ms": 180000,
|
||||
"max_retries": 2
|
||||
},
|
||||
"relay-image-edit": {
|
||||
"protocol": "images-edits",
|
||||
"base_url_env": "VECTOR_ENGINE_BASE_URL",
|
||||
"api_key_env": "VECTOR_ENGINE_API_KEY",
|
||||
"timeout_ms": 1200000,
|
||||
"max_retries": 2,
|
||||
"capabilities": {"multiple_images": false, "mask": false},
|
||||
"image_field": "image",
|
||||
"mask_editable": "transparent"
|
||||
}
|
||||
},
|
||||
"stages": {
|
||||
"understanding": {
|
||||
"provider": "relay-vision",
|
||||
"model_env": "SPINE_UNDERSTANDING_MODEL",
|
||||
"default_model": "deepseek-flash",
|
||||
"prompt": "../prompts/understand-v1.txt",
|
||||
"output_mode": "text-json",
|
||||
"stream": false,
|
||||
"params": {}
|
||||
},
|
||||
"completion": {
|
||||
"provider": "relay-image-edit",
|
||||
"model_env": "SPINE_COMPLETION_MODEL",
|
||||
"default_model": "qwen-image-3.0",
|
||||
"prompt": "../prompts/complete-v1.txt",
|
||||
"params": {"n": 1}
|
||||
},
|
||||
"segmentation": {
|
||||
"provider": "sam-local",
|
||||
"model_dir_env": "SAM_MODEL_DIR",
|
||||
"default_model_dir": "F:/Study/SAM3"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,45 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"providers": {
|
||||
"tiantoken-vision": {
|
||||
"protocol": "responses",
|
||||
"base_url_env": "TIANTOKEN_BASE_URL",
|
||||
"api_key_env": "TIANTOKEN_API_KEY",
|
||||
"timeout_ms": 180000,
|
||||
"max_retries": 2
|
||||
},
|
||||
"tiantoken-image-edit": {
|
||||
"protocol": "images-edits",
|
||||
"base_url_env": "TIANTOKEN_BASE_URL",
|
||||
"api_key_env": "TIANTOKEN_API_KEY",
|
||||
"timeout_ms": 1200000,
|
||||
"max_retries": 2,
|
||||
"capabilities": {"multiple_images": false, "mask": false},
|
||||
"image_field": "image",
|
||||
"mask_editable": "transparent"
|
||||
}
|
||||
},
|
||||
"stages": {
|
||||
"understanding": {
|
||||
"provider": "tiantoken-vision",
|
||||
"model_env": "TIANTOKEN_UNDERSTANDING_MODEL",
|
||||
"default_model": "gpt-6-astra",
|
||||
"prompt": "../prompts/understand-v1.txt",
|
||||
"output_mode": "text-json",
|
||||
"stream": true,
|
||||
"params": {"store": false}
|
||||
},
|
||||
"completion": {
|
||||
"provider": "tiantoken-image-edit",
|
||||
"model_env": "TIANTOKEN_COMPLETION_MODEL",
|
||||
"default_model": "gpt-image-2",
|
||||
"prompt": "../prompts/complete-v1.txt",
|
||||
"params": {"n": 1}
|
||||
},
|
||||
"segmentation": {
|
||||
"provider": "sam-local",
|
||||
"model_dir_env": "SAM_MODEL_DIR",
|
||||
"default_model_dir": "F:/Study/SAM3"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,147 +0,0 @@
|
||||
# 第一阶段：视觉 LLM 角色结构理解
|
||||
|
||||
## 1. 阶段简介
|
||||
|
||||
将单张角色图转换为部件语义计划：确定部件名称、骨骼与附件分类、运动层级、遮挡、可见区域粗框、关节点和补全需求。视觉模型负责结构理解；本地程序负责格式校验和前景候选点生成。
|
||||
|
||||
本阶段不生成精确 mask、透明拆件、隐藏纹理或 Spine 文件。结构合法不等于识别正确，结果仍为 `quality_status=needs_review`。
|
||||
|
||||
## 2. 输入、输出与执行链路
|
||||
|
||||
输入为 PNG 角色图；在线模式需要模型配置与密钥，离线模式需要已有语义计划。核心输出为语义计划和 SAM 提示。
|
||||
|
||||
```text
|
||||
tools/model-check.mjs（或 pipeline.mjs --mode annotated-parts）
|
||||
→ 读取配置、图片、提示词与 Schema，计算实验指纹
|
||||
→ Python 保留 Alpha / 去除边界连通白底，保持原画布
|
||||
→ 在线：视觉模型理解；离线：读取计划并按需迁移 v1
|
||||
→ 提取 JSON，校验字段、引用、层级和补全一致性
|
||||
→ 在 Alpha 前景与粗框交集中生成距离变换候选点
|
||||
→ 保存 semantic-parts.json、segmentation-prompts.json、诊断及报告
|
||||
→ 第二阶段读取 source.png + 计划 + SAM 提示
|
||||
```
|
||||
|
||||
`--parts-file` 不请求模型，也不运行 SAM。`--stop-after segment` 可接续第二阶段，输出到理解目录的 `segmentation/`；第三阶段独立调用。
|
||||
|
||||
## 3. 阶段执行命令示例及参数说明
|
||||
|
||||
以下 PowerShell 命令在项目根目录执行，环境准备见 [README](../README.md)。反引号必须在行尾，后面不能有空格。
|
||||
|
||||
```powershell
|
||||
conda activate ai-spine
|
||||
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
node .\tools\model-check.mjs --stage understanding `
|
||||
--config .\configs\deepseek-qwen.json `
|
||||
--reference .\output\20260918-183532\sam\source.png `
|
||||
--out-dir ".\output\$runId\understand"
|
||||
```
|
||||
|
||||
参考图可替换成自己的 PNG。切换 Tiantoken：将配置改为 `--config .\configs\tiantoken.json`，默认理解模型为 `gpt-6-astra`，也可显式传入 `--understanding-model gpt-6-astra`。
|
||||
|
||||
已有计划离线生成新候选，不重跑 AI：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage understanding `
|
||||
--reference .\output\20260918-183532\sam\source.png `
|
||||
--parts-file .\output\20260918-183532\sam\semantic-parts.json `
|
||||
--out-dir ".\output\$runId\understand-offline"
|
||||
```
|
||||
|
||||
| 参数 | 说明 |
|
||||
| --- | --- |
|
||||
| `--stage understanding` | 第一阶段，也是入口默认阶段 |
|
||||
| `--reference FILE` | 必需，角色 PNG |
|
||||
| `--out-dir DIR` | 实际执行必需，使用独立新目录 |
|
||||
| `--config FILE` | JSON 配置；默认 `configs/astra-qwen.json` |
|
||||
| `--env-file FILE` | 默认项目 `.env`；shell 同名变量优先 |
|
||||
| `--understanding-model ID` | 原样发送模型 ID，不转换别名 |
|
||||
| `--understanding-protocol NAME` | `chat-completions` 或 `responses` |
|
||||
| `--stream true/false` | 覆盖配置流式设置 |
|
||||
| `--output-mode NAME` | `text-json`、`json-object`、`json-schema`；需确认中转能力 |
|
||||
| `--parts-file FILE` | 离线读取有效 v2 或保守迁移 v1 |
|
||||
| `--python FILE` | 仅允许 Conda ai-spine 的解释器 |
|
||||
| `--stop-after understand/segment` | 理解结束或接续 SAM |
|
||||
| `--dry-run` | 检查配置与图片格式，不生成产物；不代表完整 Python 几何链路已验证 |
|
||||
| `--resume` | 仅复用已完成且指纹、产物哈希一致的结果，不是失败请求续发 |
|
||||
| `--list-models` | 联网查询所选提供方模型列表，不能与 dry-run 同用 |
|
||||
|
||||
模型优先级：CLI > `stage.model` > 配置指定的模型环境变量 > `stage.default_model`。提示词路径相对于配置文件。DeepSeek 预设为 Chat Completions 非流式；两个 Astra 预设为 Responses 流式。配置不代表中转所有能力均已验证。
|
||||
|
||||
## 4. 输入格式要求
|
||||
|
||||
参考图必须是有效 PNG，建议单角色、透明背景、完整画布。透明图保留 Alpha；不透明白底只去除与边界连通、RGB 各通道均 ≥240 的区域。复杂背景需先抠图，空前景会失败；白色主体接触边界可能被误删。
|
||||
|
||||
离线计划采用 [v2 Schema](../schemas/semantic-parts.schema.json)：顶层 `schema_version=2`、`coordinate_space=normalized_xy_top_left`、`parts` 数组和 `notes`。实际部件数量为 1～64。
|
||||
|
||||
| 部件字段 | 格式与含义 |
|
||||
| --- | --- |
|
||||
| `id` / `semantic` | 唯一小写 ASCII ID / 语义说明 |
|
||||
| `parent` | 父骨骼 ID 或 null；运动关系 |
|
||||
| `rig_role` | `bone_part`、`attachment`、`facial_feature` |
|
||||
| `bind_to` | 附件绑定的骨骼 ID 或 null（root）；骨骼自身为 null |
|
||||
| `visibility` | `visible`、`partial`、`hidden` |
|
||||
| `bbox_norm_xyxy` | `[xmin,ymin,xmax,ymax]` 可见粗框；隐藏为 null |
|
||||
| `pivot_norm` | `[x,y]` 关节或连接点，可处于遮挡区域 |
|
||||
| `positive_points_norm` / `negative_points_norm` | 旧模型点兼容字段，新提示词要求留空 |
|
||||
| `occluded_by` | 遮挡者 ID 数组，与运动关系分开 |
|
||||
| `completion` | `occlusion_fill`、`joint_extension_parent`、`joint_extension_child` 三个布尔值 |
|
||||
| `needs_completion` | 三项 completion 的逻辑或 |
|
||||
| `confidence` | 0～1 自报置信度；说明与不确定性写入顶层 notes，部件内没有 notes 字段 |
|
||||
|
||||
坐标以完整画布左上角为原点，归一化到 0～1。可见框须为正面积；partial/hidden 必须声明遮挡补全，hidden 无可见框和正点。附件及面部特征不能声明独立骨骼关节延展。引用必须存在，不能自引用或形成父子/遮挡环路，绑定目标必须是骨骼。完整类型与必填项以 Schema 为准。
|
||||
|
||||
v1 迁移依据 ID 推断角色分类和补全需求，推断写入 notes，不重新识别图片、不覆盖旧文件。计划必须对应同一张图、同一画布。
|
||||
|
||||
## 5. 执行链路中重要部分代码文件及原理说明（包括重点说明）
|
||||
|
||||
**请求和结构校验。** `stages/model-stage.mjs` 调度；`providers/vision.mjs` 将图片 Data URL、画布尺寸、提示词与 Schema 转成对应协议，再提取 JSON，拒绝 refusal、异常结束和未完成响应。`lib/validate-parts.mjs` 校验字段、引用、层级和补全一致性，成功后写正式计划。低置信度等产生告警，不能据结构校验认定语义正确。
|
||||
|
||||
**候选点并非重新由 AI 生成。** `geometry_grounding.py` 对粗框与 `alpha>0` 的交集计算距离变换，选取远离边界的像素中心；无有效前景则 blocked。归一化点转像素采用 floor，1 映射到最后一个像素；粗框按 floor/ceil 转半开范围。候选仍可能属于框内的其他部件。
|
||||
|
||||
**pivot 与种子独立。** 关节点不吸附到前景，模型原始正负点仅作诊断。第二阶段当前只用程序种子排序返回的 mask，未把它们作为 SAM 点提示。
|
||||
|
||||
**通信、可视化与留档。** `providers/http.mjs` 负责 HTTP、SSE、超时和脱敏；明确的 429/5xx 有限重试，超时/网络中断不自动重复请求。`lib/report.mjs` 用红/黄圆区分原模型点、青色方块表示程序候选、空心圆表示 pivot。旧输出不会随代码修改自动更新。`lib/run-artifacts.mjs` 按输入、模型、参数、提示词、Schema、代码和产物哈希复用结果；completed 仅表示执行完成。
|
||||
|
||||
## 6. 各个代码文件职责索引
|
||||
|
||||
| 文件 | 职责 |
|
||||
| --- | --- |
|
||||
| [pipeline.mjs](../pipeline.mjs) / [tools/model-check.mjs](../tools/model-check.mjs) | 新管线转发 / CLI、模型列表和阶段分发 |
|
||||
| [stages/model-stage.mjs](../stages/model-stage.mjs) | 理解总调度与留档；另含图片编辑探针 |
|
||||
| [lib/config.mjs](../lib/config.mjs) / [lib/images.mjs](../lib/images.mjs) | 配置解析 / 图片格式、尺寸、哈希和 Data URL |
|
||||
| [lib/python-runtime.mjs](../lib/python-runtime.mjs) / [lib/geometry.mjs](../lib/geometry.mjs) | Conda 检查 / Python 几何调用 |
|
||||
| [normalize_reference.py](../normalize_reference.py) / [geometry_grounding.py](../geometry_grounding.py) | Alpha 与去白底 / ROI 诊断和候选生成 |
|
||||
| [providers/vision.mjs](../providers/vision.mjs) / [providers/http.mjs](../providers/http.mjs) | 视觉协议 / 通信与脱敏 |
|
||||
| [lib/validate-parts.mjs](../lib/validate-parts.mjs) | 解析、Schema/关系校验、v1 迁移 |
|
||||
| [lib/report.mjs](../lib/report.mjs) / [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | SVG/HTML / 状态、指纹与缓存 |
|
||||
| [prompts/understand-v1.txt](../prompts/understand-v1.txt) | 理解提示词；文件名保留 v1，当前使用 v2 契约 |
|
||||
| [v2 Schema](../schemas/semantic-parts.schema.json) / [v1 Schema](../schemas/semantic-parts-v1.schema.json) | 正式契约 / 兼容输入 |
|
||||
| [DeepSeek](../configs/deepseek-qwen.json)、[Astra](../configs/astra-qwen.json)、[Tiantoken](../configs/tiantoken.json) 配置 | 模型、协议和环境变量预设 |
|
||||
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) / [tests/test_geometry.py](../tests/test_geometry.py) | 协议、计划、留档及几何回归 |
|
||||
|
||||
## 7. 产物详细说明
|
||||
|
||||
| 路径 | 内容与用途 |
|
||||
| --- | --- |
|
||||
| `source-original.png` | 原始输入副本 |
|
||||
| `source.png` | 预处理 RGBA，在线模型所见图片，后续坐标基准 |
|
||||
| `preprocessing.json` | 背景处理方式与像素统计 |
|
||||
| `semantic-parts.json` / `validation.json` | 正式 v2 计划 / 字段、关系校验及告警 |
|
||||
| `geometry-validation.json` | 原点诊断、ROI 前景比例、连通域、候选与阻塞原因 |
|
||||
| `segmentation-prompts.json` | 画布、部件 ID、candidate/hidden/blocked 状态、粗框、程序种子和 QA 要求 |
|
||||
| `annotated.svg` / `report.html` | 粗标注 / 综合复核页；不是 SAM 分割图 |
|
||||
| `prompt.txt` / `request.json` | 阶段提示词副本 / 请求身份摘要，不是完整 HTTP body |
|
||||
| `response.raw.json` / `response.txt` | 响应留档 / 提取文本；离线模式使用本地计划记录 |
|
||||
| `request-metadata.json` | 请求与返回模型、request ID、usage、耗时、次数和协议元数据 |
|
||||
| `response.error.json` | 可获取时保存的脱敏请求失败诊断 |
|
||||
| `manifest.json` | 模式、状态、输入/代码/产物哈希；SAM 路径仅记录，不加载模型 |
|
||||
|
||||
失败可能只有部分产物。第二阶段前应复核左右、粗框、层级、遮挡、分类和补全需求，不能只看点是否落在前景。
|
||||
|
||||
## 8. 当前阶段可改进点说明
|
||||
|
||||
- 明确角色左右与画面左右命名约定，提升粗框、遮挡及隐藏状态判断，减少 SAM 衣料/肢体混淆。
|
||||
- 根据预期动画审查拆分粒度与附件绑定，区分隐藏骨骼占位项和真正可见拆件。
|
||||
- 增加可审阅的计划修正流程，修正后重新生成几何提示；Alpha 候选不能代替语义复核。
|
||||
- 第二阶段后根据实际 mask 重新评估补全需求，不能因第一阶段 needs_completion=false 永久跳过有缺损的拆件。
|
||||
- 增加结构和像素证据一致性检查，区分“不确定”“结构合法”“人工确认正确”。
|
||||
@@ -1,156 +0,0 @@
|
||||
# 第二阶段：本地 SAM 分割
|
||||
|
||||
## 1. 阶段简介
|
||||
|
||||
读取第一阶段语义计划，用本地 SAM 3.1 图像检测分支生成部件 mask，再从原图提取可见像素。当前输出半透明颜色叠加的角色图、白底拆解图和透明部件。
|
||||
|
||||
SAM 在 Conda `ai-spine` 的临时 GPU 子进程中运行，结束后退出；不部署常驻服务、不下载权重、不调用中转站、不自动回退 CPU。本阶段尚无大模型去毛边步骤，也不补画隐藏纹理。
|
||||
|
||||
## 2. 输入、输出与执行链路
|
||||
|
||||
输入为第一阶段目录，加本地 SAM 代码和权重；输出为可见部件 mask、原像素拆件、诊断和预览。
|
||||
|
||||
```text
|
||||
tools/model-check.mjs --stage segmentation
|
||||
→ stages/segmentation-stage.mjs 检查输入、路径、参数与 Conda
|
||||
→ 临时探测子进程检查依赖和 CUDA，不加载权重
|
||||
→ 计算输入、代码、权重指纹，复制输入到新输出目录
|
||||
→ 临时推理子进程：sam_segment.py → sam_runtime.py
|
||||
→ 严格加载检测分支，一次编码原图
|
||||
→ 每部件分别取得文本、文本+粗框候选，排序选 mask
|
||||
→ 与原图 Alpha 取交集，提取原像素并渲染报告
|
||||
→ 子进程退出，manifest 记录产物与退出状态
|
||||
→ 第三阶段读取 source.png、计划、segmentation.json 和 masks/
|
||||
```
|
||||
|
||||
`hidden/blocked` 不进入推理；没有合格候选会记录跳过原因，不用矩形或种子伪造 mask。所有输出仍为 `semantic_verified=false`、`quality_status=needs_review`。
|
||||
|
||||
## 3. 阶段执行命令示例及参数说明
|
||||
|
||||
在项目根目录执行，环境安装见 [README](../README.md)。以下独立命令复用已有计划，无需模型密钥：
|
||||
|
||||
```powershell
|
||||
conda activate ai-spine
|
||||
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
node .\tools\model-check.mjs --stage segmentation `
|
||||
--input-dir .\output\sombrero-grounding-v2 `
|
||||
--out-dir ".\output\$runId\sam" `
|
||||
--sam-dir F:/Study/SAM3
|
||||
```
|
||||
|
||||
将 `--input-dir` 替换为本次第一阶段的 `understand` 目录，即可接续新实验。自动化可以使用 `conda run --no-capture-output -n ai-spine node ...`。
|
||||
|
||||
仅探测环境与输入，不进行模型推理：
|
||||
|
||||
```powershell
|
||||
conda run --no-capture-output -n ai-spine node .\tools\model-check.mjs `
|
||||
--stage segmentation `
|
||||
--input-dir .\output\sombrero-grounding-v2 `
|
||||
--sam-dir F:/Study/SAM3 `
|
||||
--dry-run
|
||||
```
|
||||
|
||||
| 参数 | 说明 |
|
||||
| --- | --- |
|
||||
| `--stage segmentation` | 独立运行第二阶段 |
|
||||
| `--input-dir DIR` | 必需，第一阶段输出目录 |
|
||||
| `--out-dir DIR` | 推理必需，使用独立新目录；dry-run 可省略 |
|
||||
| `--sam-dir DIR` | SAM 根目录，优先于 SAM_MODEL_DIR 和配置默认值 |
|
||||
| `--sam-code-dir DIR` | 源码仓库，优先于 SAM_CODE_DIR 和根目录下 sam3 |
|
||||
| `--sam-checkpoint FILE` | 显式本地权重，覆盖根目录自动查找 |
|
||||
| `--sam-confidence FLOAT` | 候选模型分数阈值，默认 0.2，严格介于 0 和 1；不是语义准确率 |
|
||||
| `--sam-precision NAME` | 默认 bf16，可选 fp32；后者显存占用更高 |
|
||||
| `--config FILE` / `--env-file FILE` | 配置与环境变量文件，默认 astra-qwen.json / 项目 .env |
|
||||
| `--python FILE` | 仅允许 ai-spine 的解释器 |
|
||||
| `--dry-run` | 检查路径、Conda、依赖导入和 CUDA；不校验完整权重加载或推理显存 |
|
||||
| `--resume` | 输入、参数、代码、权重和完整产物一致时复用；失败任务另建目录 |
|
||||
|
||||
也可在第一阶段命令上使用 `--stop-after segment --sam-dir F:/Study/SAM3`，输出位于理解目录的 `segmentation/`。该方式会先执行理解阶段，可能调用付费模型。
|
||||
|
||||
## 4. 输入格式要求
|
||||
|
||||
| 输入 | 要求 |
|
||||
| --- | --- |
|
||||
| `source.png` | 第一阶段预处理 RGBA，保留完整原画布 |
|
||||
| `semantic-parts.json` | 通过结构校验的 v2 计划，旧版须先离线迁移 |
|
||||
| `segmentation-prompts.json` | 宽高匹配 source；部件 ID 与计划完整一致 |
|
||||
| candidate 提示 | 非隐藏部件，seed_source 为 alpha_roi_distance_transform；整数粗框在画布内；至少一个整数种子在粗框与 Alpha 前景内 |
|
||||
| SAM 代码 | 源码仓库内部必须有 sam3/model_builder.py |
|
||||
| SAM 权重 | 本地 sam3.1_multiplex.pt，与本项目适配的检测分支结构匹配 |
|
||||
|
||||
当前本机布局：
|
||||
|
||||
```text
|
||||
F:/Study/SAM3/
|
||||
sam3/
|
||||
sam3/model_builder.py
|
||||
sam3.1checkpoint/
|
||||
sam3.1_multiplex.pt
|
||||
```
|
||||
|
||||
权重默认先查找上述子目录，再查找 SAM 根目录下同名文件。当前已验证环境为 Python 3.11、PyTorch 2.7.1+cu126、torchvision 0.22.1、NumPy 1.26.4、OpenCV 4.11.0.86，RTX 3060 Laptop 6 GB。这是本机验证组合，不是所有机器的兼容保证。
|
||||
|
||||
## 5. 执行链路中重要部分代码文件及原理说明（包括重点说明）
|
||||
|
||||
**本地生命周期与权重。** `stages/segmentation-stage.mjs` 先探测，再启动一次推理进程；离线环境变量禁止 Hugging Face 下载。推理超时 20 分钟，结束、报错或取消后退出。`sam_runtime.py` 从 multiplex 权重提取 detector 分支，适配三个特征尺度，移除未使用的第四尺度并取消 scalp；忽略不参与当前推理的视频 neck，实际加载参数使用 strict=True。外部 SAM 源码不作修改。
|
||||
|
||||
**精度不能简单全模型转换。** 模型参数和复数位置编码缓冲区保持原精度，在推理上下文开启 BF16 autocast；直接对整个模型调用 BF16 转换会影响复数缓冲区及要求 FP32 的路径。
|
||||
|
||||
**提示和选择。** `sam_segment.py` 将原图合成白底 RGB，GPU 只编码一次；每个部件分别收集 text_only 与 text_and_box 候选。排序结合模型分数、粗框内比例、Alpha 越界比例和种子命中；前景不足 4 像素或框内比例低于 10% 的候选被拒绝。粗框是提示，并非强制裁剪边界。
|
||||
|
||||
**重点：当前不是 SAM 点提示分割。** 所用图像概念接口接受文本和框。第一阶段程序种子仅参与返回 mask 排序，模型原始点不传入 SAM。提高种子前景命中率不能保证选中正确部件。
|
||||
|
||||
**像素与质量。** `segmentation_outputs.py` 用原图 Alpha 裁掉背景，从原图复制 RGB/Alpha，不重绘、不缩放；保留多部件重叠。记录越界、碎片、未命中种子、漏分、重叠和 IoU≥0.8 的疑似重复。覆盖率不是语义准确率，原像素保真也不能修复错误归属。
|
||||
|
||||
**当前预览含原图纹理。** `annotated.png` 是原图上 Alpha=100 的彩色 mask 叠加，不是仅用纯色区域组成的角色图；`parts-sheet.png` 为带 ID 的白底拆件。纯色遮罩角色图属于第 8 节的待实现项。
|
||||
|
||||
## 6. 各个代码文件职责索引
|
||||
|
||||
| 文件 | 职责 |
|
||||
| --- | --- |
|
||||
| [tools/model-check.mjs](../tools/model-check.mjs) | 独立分割入口、参数解析 |
|
||||
| [stages/segmentation-stage.mjs](../stages/segmentation-stage.mjs) | 输入校验、环境探测、子进程、哈希、日志与失败处理 |
|
||||
| [lib/config.mjs](../lib/config.mjs) | SAM 路径与配置解析 |
|
||||
| [lib/python-runtime.mjs](../lib/python-runtime.mjs) | ai-spine 解释器与基础依赖检查 |
|
||||
| [lib/images.mjs](../lib/images.mjs) / [lib/validate-parts.mjs](../lib/validate-parts.mjs) | PNG 元数据 / v2 计划结构校验 |
|
||||
| [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | 输出目录、复用、产物哈希 |
|
||||
| [sam_runtime.py](../sam_runtime.py) | 依赖/CUDA 探测、源码路径、检测权重适配与严格加载 |
|
||||
| [sam_segment.py](../sam_segment.py) | 提示校验、GPU 推理、候选排序、诊断与阶段组织 |
|
||||
| [segmentation_outputs.py](../segmentation_outputs.py) | mask QA、原像素提取、标注图、拆件图、HTML |
|
||||
| [tests/test_segmentation.py](../tests/test_segmentation.py) | 候选、提示、Alpha、裁剪保真和重叠回归 |
|
||||
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 阶段入口和相关本地回归调度 |
|
||||
|
||||
`segment.py` 是旧版白底拆件图分割器，不是本阶段 SAM 入口；`compose_preview.py`、`run_case_matrix.py` 是旧版定位/组装对照工具，不应与新阶段混用。
|
||||
|
||||
## 7. 产物详细说明
|
||||
|
||||
| 路径 | 内容与用途 |
|
||||
| --- | --- |
|
||||
| `source.png` / `semantic-parts.json` / `segmentation-prompts.json` | 输入副本，保留坐标与语义来源 |
|
||||
| `annotated.png` | 原尺寸原图 + 半透明彩色 mask，背景合成白色 |
|
||||
| `parts-sheet.png` | 原尺寸拆件分开排布的白底图，含 ID 与颜色标记 |
|
||||
| `parts/<id>.png` | 紧裁剪 RGBA 部件，保留原图像素 |
|
||||
| `masks/<id>.png` | 原画布尺寸的二值 mask，0/255，已裁掉 Alpha 背景 |
|
||||
| `candidates/<id>-raw.png` | 选中的原始 SAM mask，尚未裁 Alpha，用于越界诊断 |
|
||||
| `unassigned-mask.png` | 白色为没有任何部件覆盖的角色前景 |
|
||||
| `overlap-mask.png` | 白色为多个部件共同覆盖的区域 |
|
||||
| `segmentation.json` | 部件路径、crop_xyxy、sheet_xyxy、颜色、QA、跳过列表、覆盖率、运行环境、加载记录 |
|
||||
| `candidate-diagnostics.json` | 两种提示候选的分数、排序、拒绝原因和最终选择 |
|
||||
| `report.html` | 预览、部件链接、告警及漏分/重复复核 |
|
||||
| `worker.log` / `worker-error.json` | 日志 / 子进程错误诊断（失败时） |
|
||||
| `manifest.json` | 状态、输入/代码/权重/产物哈希；完成后 gpu_worker_exited=true |
|
||||
|
||||
`crop_xyxy` 为原图坐标，`sheet_xyxy` 为拆解图坐标，右下角均为开区间。第三阶段依赖全画布 masks，不以紧裁剪 parts 反推位置。
|
||||
|
||||
当前质量基线是 `output/20260918-183532/sam`：32 个计划项中 10 个 hidden 跳过，22 个 mask，前景覆盖率约 83.3%，32719 像素未分配。图中 forearm_r/hand_r 存在衣料误分，前臂混入手部，上衣及袖子存在缺失。更早 `sombrero-sam31-v3` 为另一批 28 个 mask、约 67.18% 覆盖率，不能混用统计。真实 GPU 推理跑通不代表拆件可直接用于动画。
|
||||
|
||||
## 8. 当前阶段可改进点说明
|
||||
|
||||
以下均为待实现或待验证的改进方向：
|
||||
|
||||
- **输出仅色彩遮罩的完整角色图。** 在原画布上以各部件纯色展示分区，不叠加原图纹理；保留现有 annotated.png。需明确重叠显示顺序，将未分配区域单独标识，不能为了“完整”伪造已识别区域。
|
||||
- **可选的大模型去毛边。** 对每个通过部件归属复核的 SAM 拆件增加一次可选边缘清理，提供对应原图局部参考，约束修改范围，保存清理前后、mask 差异和坐标。默认保留原始拆件；清理不能冒充隐藏补全，也不能把重绘结果称为原像素。
|
||||
- **先修正语义误分。** 明确左右、隐藏状态和粗框，复核衣料/皮肤/手部归属；对明显错误、缺失或疑似重复输出阻断性状态，再进入补全。
|
||||
- **改善覆盖和边界。** 对未分配前景、过度重叠及大孔洞做定位复核；区分合理遮挡与分割缺口，避免一刀切强制所有 mask 互斥。
|
||||
- **重评补全需求。** 基于实际 mask 检查袖子、衣服等缺损，反馈第三阶段；不能仅沿用第一阶段 needs_completion。
|
||||
- **增加语义验收。** 模型分数、前景命中和覆盖率只作辅助；将原图、候选、最终 mask 并排复核，记录人工修正及来源。
|
||||
@@ -1,245 +0,0 @@
|
||||
# 第三阶段：部件补全
|
||||
|
||||
## 1. 阶段简介
|
||||
|
||||
读取 SAM 可见 mask，规划分割缺口修复、遮挡补全、关节延伸和附件底图清理，向图片模型提供部件与原图参考，再把结果映射回原图坐标。Python 几何处理固定使用 Conda `ai-spine`；图片生成在远端完成，本阶段不加载本地 SAM。
|
||||
|
||||
默认 `--completion-profile adaptive` 使用实际 mask 重建需求、自适应半径和可逆缩放；`legacy` 保留旧画布与筛选规则，供旧候选复检。原 SAM 输入始终留档，只有保留区域锁定原像素；明确规划的编辑区域可以替换，人工确认的移除区域可删除。
|
||||
|
||||
当前已实现准备、真实图片请求、离线候选导入、几何检查与透明补全部件输出。[第四阶段](4-spine-stage.md) 独立整合 SAM 与补全部件，输出完整角色组装、Spine 骨架和预览。几何通过仍为 `needs_review`，不能等同于部件可用。
|
||||
|
||||
## 2. 输入、输出与执行链路
|
||||
|
||||
输入是第二阶段目录；输出是任务画布、候选、检查报告及通过几何检查的补全部件。
|
||||
|
||||
```text
|
||||
tools/model-check.mjs --stage completion --input-dir SAM目录
|
||||
→ stages/completion-stage.mjs 校验计划、mask 文件、参数及 Conda
|
||||
→ completion_geometry.py prepare
|
||||
adaptive：completion_planning.py 重建需求 → 区分保留/编辑/移除 → 缩放工作区并加入参考
|
||||
legacy：沿用 needs_completion → 固定半径 → 原像素平移
|
||||
→ prepare-only：保存任务和报告后结束
|
||||
→ 否则：串行请求 images/edits，或读取离线候选
|
||||
→ completion_geometry.py merge
|
||||
检查画布/前景/对齐 → 逆变换回原像素尺寸 → 限制编辑和连通性 → 恢复保留像素
|
||||
→ 每部件选第一个通过几何检查的候选
|
||||
→ finalize 输出透明部件、原图坐标和报告
|
||||
```
|
||||
|
||||
adaptive 检查所有选中部件的实际 mask，不再以第一阶段 needs_completion 为唯一门槛。必须有可见 mask、保留对齐锚点，以及非空编辑或移除区域。隐藏/缺 mask、无观察到的编辑区域、无保留锚点分别跳过；无明确遮挡关系的近重复 mask（IoU≥0.8）要求人工复核。一般有歧义的重叠只记录诊断，不自动猜测归属。指定 `--part` 只缩小处理范围。
|
||||
|
||||
legacy 才要求 `needs_completion=true` 且允许新增非空。旧实验 `output/20260918-183532/sam` 的 22 个可见部件中 12 个进入补全，是旧筛选结果，不能套用于 adaptive。
|
||||
|
||||
## 3. 阶段执行命令示例及参数说明
|
||||
|
||||
以下命令在项目根目录执行；输入示例是已有实验，不代表其 mask 已合格。
|
||||
|
||||
```powershell
|
||||
conda activate ai-spine
|
||||
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
|
||||
$samInput = ".\output\20260918-183532\sam"
|
||||
|
||||
# 离线准备所有符合条件的任务，无需密钥
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir $samInput `
|
||||
--out-dir ".\output\$runId\complete-prepare" `
|
||||
--prepare-only
|
||||
|
||||
# 单部件真实图片编辑请求
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir $samInput `
|
||||
--out-dir ".\output\$runId\complete-head" `
|
||||
--completion-model gpt-image-2 `
|
||||
--part head
|
||||
|
||||
# 所有符合条件的部件，逐部件请求模型
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir $samInput `
|
||||
--out-dir ".\output\$runId\complete"
|
||||
```
|
||||
|
||||
使用 Qwen 时改为 `--config .\configs\deepseek-qwen.json --completion-model qwen-image-3.0`，读取原中转配置。Tiantoken 配置第三阶段为 `gpt-image-2`，第一阶段的 `gpt-6-astra` 不是当前图片编辑接入的替代品；未实现 Astra Responses 图片生成工具链。
|
||||
|
||||
旧候选离线复检，不重新请求模型：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir $samInput `
|
||||
--candidates-dir .\output\20260918-183532\complete2_gpt `
|
||||
--out-dir ".\output\$runId\complete-recheck" `
|
||||
--completion-profile legacy `
|
||||
--candidate-size-policy uniform
|
||||
```
|
||||
|
||||
仅测试图片编辑接口的独立探针：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--image .\output\20260918-183532\complete2_gpt\tasks\head\input.png `
|
||||
--prompt-file .\output\20260918-183532\complete2_gpt\tasks\head\prompt.txt `
|
||||
--out-dir ".\output\$runId\edit-probe"
|
||||
```
|
||||
|
||||
探针只保存候选，不执行部件准备与几何验收，不能与 `--input-dir` 混用。
|
||||
|
||||
新的单部件多图/mask 能力探针会产生一次真实请求，并执行新几何链路；不会修改默认 capabilities。仅在决定进行付费小样时运行，替换 part 为实际存在的可执行任务：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs --stage completion `
|
||||
--config .\configs\tiantoken.json `
|
||||
--input-dir $samInput `
|
||||
--out-dir ".\output\$runId\capability-probe" `
|
||||
--part head --probe-edit-capabilities
|
||||
```
|
||||
|
||||
该探针发送主图、局部参考、全角色参考和透明可编辑 mask，强制 n=1、禁用 HTTP 重试。查看 request.json、原候选、QA 和 capability-evidence.json；HTTP 接受不等于参考图被使用或 mask 语义正确，因此不会自动启用能力。prepare-only 可配 `--completion-reference both --completion-mask required` 检查计划发送内容而不联网。
|
||||
|
||||
| 参数 | 说明 |
|
||||
| --- | --- |
|
||||
| `--input-dir DIR` | 完整 SAM 输出目录，部件补全必需 |
|
||||
| `--out-dir DIR` | 独立新目录，准备与生成使用不同目录 |
|
||||
| `--part ID` | 可重复，ID 必须存在且不重复；省略时处理所有符合筛选条件的部件 |
|
||||
| `--prepare-only` | 只生成画布、区域、提示词和报告，不请求模型 |
|
||||
| `--candidates-dir DIR` | 离线导入候选，与 prepare-only 互斥 |
|
||||
| `--completion-profile adaptive/legacy` | 默认 adaptive；旧候选需显式 legacy |
|
||||
| `--completion-reference auto/board/local/full/both/none` | auto：支持多图时 local+full，否则单图参考板；legacy auto 为旧单图/局部参考行为 |
|
||||
| `--completion-mask auto/required/off` | auto 遵循配置；required 要求已配置支持，或用单部件能力探针试验；off 不发送但仍本地限制编辑 |
|
||||
| `--completion-review FILE` | 人工确认的补全区域文件，格式见第 4 节；只用于 adaptive |
|
||||
| `--probe-edit-capabilities` | adaptive 在线单部件试验多图+mask，不持久修改配置，不自动认定能力支持 |
|
||||
| `--completion-size N` | 请求方形画布边长，默认 1024，整数 128～4096；adaptive 记录显式缩放和逆变换 |
|
||||
| `--extension-radius N` | 原图像素半径覆盖，整数 1～256；adaptive 默认 min(mask宽,高)×0.15 向上取整并限于 4～96；legacy 默认 32 |
|
||||
| `--candidate-size-policy NAME` | strict 默认拒绝尺寸不符；uniform 允许同宽高比整画布缩放后复检 |
|
||||
| `--completion-model ID` | 图片模型 ID，原样发送 |
|
||||
| `--config FILE` / `--env-file FILE` | 默认 astra-qwen.json / 项目 .env；shell 同名变量优先 |
|
||||
| `--python FILE` | 仅允许 ai-spine 解释器 |
|
||||
| `--dry-run` | 检查输入文件、计划、参数和 Python 基础环境，不准备画布或请求模型；完整像素级 mask 校验在 prepare 中执行 |
|
||||
| `--resume` | 仅复用已完成、输入/配置/源码/产物一致的任务，不自动续发失败请求 |
|
||||
| `--image FILE` / `--prompt-file FILE` | 仅用于探针：PNG 图片 / 文本提示词；多次 image 需配置多图能力 |
|
||||
| `--mask FILE` | 仅用于探针：透明处可编辑的 RGBA PNG，需配置 mask 能力 |
|
||||
|
||||
配置中默认 `n=1`，逐部件串行调用。明确的 429/5xx 有限重试，超时/断网不自动重试；请求失败保留诊断并停止后续部件，几何拒绝会记录结果并继续。`--stop-after` 不支持 completion。
|
||||
|
||||
## 4. 输入格式要求
|
||||
|
||||
| 输入 | 要求 |
|
||||
| --- | --- |
|
||||
| `source.png` | 原画布 PNG，使用第二阶段的 RGBA 原图 |
|
||||
| `semantic-parts.json` | 有效 v2 计划；部件分类、遮挡、pivot 和补全需求合法 |
|
||||
| `segmentation.json` | parts 数组，ID 唯一且存在于计划中 |
|
||||
| `masks/<id>.png` | segmentation.parts 中每个 ID 都必须有文件；原画布尺寸，灰度值仅 0/255，非空，不覆盖原图 Alpha=0 区域 |
|
||||
| 中转配置 | 在线模式需地址、密钥、images-edits 提供方及图片模型；prepare-only/离线导入无需密钥 |
|
||||
|
||||
只提供 SAM 的 `parts/` 或白底拆解图不满足要求。输入目录中无需存在 annotated.png 才能运行，但建议先通过报告复核语义。
|
||||
|
||||
离线候选支持两种形式：
|
||||
|
||||
- 平铺目录：`<id>.png`，必须对应本次局部画布坐标、尺寸与部件；平铺文件没有旧任务 manifest 可自动核对，需自行保证一致。
|
||||
- 旧补全目录：含 manifest 和 `tasks/<id>/candidate-1.png`，也支持 jpg/webp；逐项核对源图、计划、segmentation.json、全部 mask 哈希以及画布/延展参数。每部件只取首张候选。缺失记为 missing_candidate，不联网补发。
|
||||
|
||||
不同原图、mask 或参数的候选不能混用。尺寸之外，部件位置、朝向、比例也必须保持一致。
|
||||
|
||||
adaptive 的旧运行候选还要匹配 profile、参考布局、复核文件/掩码哈希和规划版本；board 与独立多图的画布不同，不可混用。平铺候选仍需人工保证它对应当前请求画布。
|
||||
|
||||
人工复核文件示例（JSON 中的 ID 取自当前语义计划，字段均可省略但 confirmed 必须为 true）：
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"parts": {
|
||||
"part_id": {
|
||||
"confirmed": true,
|
||||
"extension_radius": 24,
|
||||
"repair_mask": "review-masks/repair.png",
|
||||
"replace_mask": "review-masks/replace.png",
|
||||
"remove_mask": "review-masks/remove.png"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
mask 是与 source 同尺寸的单通道 0/255 PNG，白色表示区域；路径相对于复核 JSON 且不得越出其目录。repair 必须在当前 SAM mask 外；replace/remove 必须位于 SAM 内且互不重叠。replace 用于误分重绘或额外底图清理，remove 用于明确不属于该部件的像素。半径优先级为部件复核值 > CLI 全局值 > 自适应值。文件及 masks 会复制进输出，哈希纳入复用身份。
|
||||
|
||||
## 5. 执行链路中重要部分代码文件及原理说明（包括重点说明）
|
||||
|
||||
**按实际 mask 重建需求。** 遮挡延伸使用轮廓外半径带与 occluded_by 的实际 mask 交集；父/子关节需求来自骨骼层级与 pivot。自动分割修复仅选择 mask 内部闭合孔洞、原图有前景且无其他 mask 归属的像素；开放缺口和重叠归属不自动猜测，可用复核文件指定。自动底图清理仅针对 occluded_by 中直接 bind_to 当前骨骼的附件/五官与当前 mask 的交集。这些仍是几何候选，语义不自动获证。
|
||||
|
||||
**保留与编辑。** editable 为遮挡/关节/修复/清理/人工替换区域的并集，扣除 removed；preserve = SAM − editable − removed。保留原 mask 和源图，并另存三种全画布区域供第四阶段核对。只有原始 SAM 或单一图片目录不足以描述这个契约。
|
||||
|
||||
**可逆画布。** native 方形裁剪边长为可见+编辑区域最长边的 1.15 倍向上取整，记录 origin。工作区缩放到请求画布（board 工作区边长为请求边长约 68%，其余为参考板；独立参考时工作区占整幅）。RGBA 用 LANCZOS，区域 mask 用最近邻。task.json 保存 native_size、scale、working_xyxy 和含像素中心约定的正反仿射变换。返回后只取工作区，逆缩放到 native，再裁剪回原图坐标；保留像素从原图恢复，所以缩放不改变最终保留像素。legacy 仍仅平移补边。
|
||||
|
||||
**实际发送内容。** 当前三个提供方默认仍关闭 multiple_images/mask。adaptive auto 因此发送一张 input.png：左侧是待编辑工作区，右侧为局部及完整角色参考，参考区域不参与最终资产。支持多图时 auto 顺序为 input.png → context.png → reference.png；也可选 local/full/both。request-plan.json 保存计划；只有实际请求才写 request.json，包含有序文件角色、SHA-256、尺寸、image 字段、mask 哈希、透明可编辑语义与是否发送。mask 不支持时明确标 local_only，不能宣称模型侧已受 mask 约束。
|
||||
|
||||
**图片解码与尺寸。** `providers/image-edits.mjs` 使用 multipart images/edits，支持非空 b64_json 或返回 URL 下载；空 b64_json 会回退有效 URL。当前部件请求显式发送 size=画布宽x高，并写入 request.json；配置 params.size 冲突会报错。strict 模式拒绝 CANVAS_SIZE_MISMATCH；uniform 仅将同宽高比整画布 LANCZOS 缩放到目标尺寸，保留原候选，不平移、不裁剪、不单独缩放物体，也不改变原图/mask。
|
||||
|
||||
**几何验收与合并。** adaptive 在请求工作区的保留不透明像素处检查 RGB MAE≤30、保留率≥95%，先验对齐再逆采样，避免把逆采样边界 Alpha 变化误判成位移。返回前景只保留 editable 内且连接 preserve 的像素；清理/人工替换区域至少 95% 有效填充，否则拒绝。没有有效修改且没有人工移除则拒绝。最后恢复 preserve 的原 RGBA，已连接的可见缺口使用原图纹理，removed 清零。legacy 沿用原可见像素全保护规则。
|
||||
|
||||
**能力证据与质量边界。** adaptive QA 记录合并前的 raw_outside_edit_mae / changed_pixels，避免本地恢复掩盖模型越界修改。capability-evidence.json 记录探针的输入顺序、尺寸、候选和 QA，但 reference_use_verified、mask_semantics_verified 不自动置 true，仍需视觉验证。区域填满不能证明生成的是正确底图；未明确划入 editable/remove 的误分仍受保护。接缝、材质、关节补足和运动露底尚无可靠自动验收。
|
||||
|
||||
## 6. 各个代码文件职责索引
|
||||
|
||||
| 文件 | 职责 |
|
||||
| --- | --- |
|
||||
| [tools/model-check.mjs](../tools/model-check.mjs) | 部件补全与独立探针分发、参数互斥检查 |
|
||||
| [stages/completion-stage.mjs](../stages/completion-stage.mjs) | 输入指纹、准备、串行请求/离线读取、候选选择、状态留档 |
|
||||
| [completion_geometry.py](../completion_geometry.py) | allowed_regions、prepare、merge_candidate、finalize 与报告 |
|
||||
| [completion_planning.py](../completion_planning.py) | adaptive 需求重建、复核输入、参考板、正反变换、编辑分区与合并 |
|
||||
| [normalize_reference.py](../normalize_reference.py) | 候选 Alpha 保留、边界连通白底提取 |
|
||||
| [providers/image-edits.mjs](../providers/image-edits.mjs) | multipart 编辑请求、能力检查、Base64/URL 图片解码 |
|
||||
| [providers/http.mjs](../providers/http.mjs) | HTTP、超时、重试、诊断和脱敏 |
|
||||
| [stages/model-stage.mjs](../stages/model-stage.mjs) | runCompletionProbe，仅验证编辑接口 |
|
||||
| [lib/config.mjs](../lib/config.mjs) / [lib/images.mjs](../lib/images.mjs) | 提供方、模型与能力配置 / 图片格式、尺寸和哈希 |
|
||||
| [lib/python-runtime.mjs](../lib/python-runtime.mjs) / [lib/validate-parts.mjs](../lib/validate-parts.mjs) | Conda 检查 / v2 结构校验 |
|
||||
| [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | 输出目录、复用、状态和产物哈希 |
|
||||
| [prompts/complete-v1.txt](../prompts/complete-v1.txt) | 保持姿势、尺寸、可见纹理与隐藏区域补全约束 |
|
||||
| [Tiantoken](../configs/tiantoken.json)、[DeepSeek/Qwen](../configs/deepseek-qwen.json)、[Astra/Qwen](../configs/astra-qwen.json) 配置 | 图片模型、认证变量、请求参数与能力开关 |
|
||||
| [tests/completion-stage.test.mjs](../tests/completion-stage.test.mjs) / [tests/test_completion.py](../tests/test_completion.py) | mock 编辑、离线复用、尺寸/对齐拒绝、像素保真与区域回归 |
|
||||
| [tests/test_completion_adaptive.py](../tests/test_completion_adaptive.py) | 放大/缩小回贴、漏分重建、附件清理、区域复核、拒绝位移和第四阶段契约 |
|
||||
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 提供方与探针相关回归 |
|
||||
|
||||
## 7. 产物详细说明
|
||||
|
||||
| 路径 | 内容与用途 |
|
||||
| --- | --- |
|
||||
| `source.png`、`semantic-parts.json`、`segmentation.json`、`masks/` | 输入快照，用于复现与离线候选身份核对 |
|
||||
| `completion-plan.json` | 准备任务、区域、跳过原因；adaptive 含 assessments、重建需求、半径及歧义重叠 |
|
||||
| `tasks/<id>/input.png` | 实际请求主图；board 含目标、局部参考与全角色参考 |
|
||||
| `tasks/<id>/visible.png` | 请求坐标的保留 RGBA；legacy 为原可见 RGBA |
|
||||
| `tasks/<id>/context.png` / `reference.png` | 局部 / 全角色原图参考，由 reference mode 决定是否独立发送 |
|
||||
| `tasks/<id>/edit-mask.png` | RGBA 编辑遮罩，透明处为 editable 或 removed；是否发送以 request.json 为准 |
|
||||
| `tasks/<id>/allowed-mask.png` | 白色表示允许新增区域 |
|
||||
| `tasks/<id>/<补全字段>.png` | 启用的补全子区域 mask |
|
||||
| `tasks/<id>/task.json` / `prompt.txt` | origin、区域、变换、参考布局、重建需求和告警 / 实际提示词 |
|
||||
| `tasks/<id>/native-*.png` | 原像素尺寸的源图、保留像素与各区域，用于逆映射和精确恢复 |
|
||||
| `preservation/<id>-*.png` | 原图全画布 preserve/editable/removed 契约，第四阶段核对用 |
|
||||
| `review-source.json`、`review.json`、`review/` | 可选原始复核文件、路径规范化快照及对应 masks |
|
||||
| `tasks/<id>/request-plan.json` / `request.json` | 计划 / 实际请求；记录有序图片角色、哈希、尺寸、mask 语义和发送状态 |
|
||||
| `tasks/<id>/capability-evidence.json` | 仅能力探针：请求接受情况及原始越界指标，能力需人工确认 |
|
||||
| `tasks/<id>/response.raw.json` / `request-metadata.json` | 在线响应留档 / 请求 ID、耗时等元数据 |
|
||||
| `tasks/<id>/response.error.json` | 可获取时保存的脱敏请求失败诊断 |
|
||||
| `tasks/<id>/candidate-N.*` | 模型或离线导入的原始候选，拒绝也保留 |
|
||||
| `tasks/<id>/merged-N-normalized.png` | uniform 实际重采样时保存的候选；不代表通过 |
|
||||
| `tasks/<id>/qa-N.json` | 尺寸、对齐、新增/丢弃像素、各区域完成情况、拒绝原因 |
|
||||
| `tasks/<id>/merged-N.png` | 原像素尺寸的局部 RGBA；adaptive 已逆采样并恢复 preserve |
|
||||
| `parts/<id>.png` | 每部件选中结果的紧裁剪透明 PNG |
|
||||
| `completion-result.json` | 每部件状态、候选 QA、原图 crop_xyxy、preservation 路径和变换 |
|
||||
| `report.html` / `manifest.json` | 人工检查页 / 模式、网络标记、状态、输入/代码/产物哈希 |
|
||||
|
||||
parts/ 只含本次选中且通过检查的补全部件，不是全部 SAM 部件集合。无需补全、失败和未选中的部件不会自动复制进来；完整角色组装由第四阶段读取 SAM 与本目录后生成。
|
||||
|
||||
当前实验：`output/20260918-183532/complete2_gpt` 已真实请求 Tiantoken gpt-image-2 共 12 个部件。旧请求未传 size，返回 1248/1254 方形图，预期为 1024，全部严格尺寸拒绝。`complete2_gpt-size-recheck-v2` 离线复用原候选后，8 个几何通过；head、pants_thigh_l、pants_thigh_r、shoe_l 共 4 个因对齐或无有效新增拒绝。通过项中仍有衣料误分、错误手部和未补全子区域。显式 size 的请求构造已验证，但尚无修复后中转遵守 size 的新实测结论。
|
||||
|
||||
2026-09-22 adaptive 验证：Conda ai-spine 下 npm test 共 29 项通过（包括 8 项新 Python adaptive 用例），覆盖 board/独立参考顺序、mask mock、放大与缩小回贴、位移拒绝、实际孔洞、附件清理、人工编辑契约及第四阶段接续。`output/completion-adaptive-prepare-20260922` 使用既有 SAM 数据离线生成 16 个任务、11 个跳过，上衣规划 3708 个孔洞像素、头部规划 10205 个底图清理像素。它们是准备阶段的候选区域，不代表补全质量；本轮未请求付费模型，也未验证中转真实能力。
|
||||
|
||||
## 8. 当前阶段可改进点说明
|
||||
|
||||
- **完整角色组装已接入第四阶段。** 按原坐标与遮挡约束组装，复用原拆件并记录失败回退，输出补全前后对比、覆盖与重叠诊断及单骨骼旋转检查动画。可靠的接缝与运动质量自动验收仍待实现。
|
||||
- **参考输入已实现，真实能力待验证。** 支持 board 与独立 local/full/both、mask 发送策略、有序请求记录及单部件能力探针。离线 mock 已覆盖图片顺序、mask、返回尺寸和越界记录；本轮未请求付费模型，未宣称中转支持或生成质量提升。
|
||||
- **实际 mask 需求重建已实现。** 分别规划缺口、遮挡、关节和附件清理，保存 preserve/editable/removed，并支持人工复核区域。保守孔洞检测不会自动解决所有开放缺口、误分与语义歧义；这些仍需复核。
|
||||
- **自适应画布与区域已实现。** 半径随部件尺寸变化并可覆盖；小部件放大、大部件缩小，请求与原图间变换留档，最终恢复原像素尺寸和保护区。离线测试覆盖放大/缩小、Alpha 保真、位移拒绝及第四阶段接续；真实模型效果待小样验证。
|
||||
- **加强验收。** 对必需区域定义有效覆盖要求，增加部件语义、接缝、材质、回贴和关节旋转露底检查；不能放宽阈值来掩盖重绘、位移或错误输入。
|
||||
- **按小样验证后批量。** 先修正 SAM 输入，再用头、前臂、上衣验证参考图与遮罩，比较原输入、原候选、合并图和回贴图，确认改善后再跑全量。
|
||||
@@ -1,128 +0,0 @@
|
||||
# 第四阶段：角色组装与 Spine 导出
|
||||
|
||||
## 1. 阶段简介
|
||||
|
||||
将第二阶段的可见部件与第三阶段通过几何检查的补全部件整合，按原图坐标组装，并依据 v2 语义计划生成 Spine 4.2 骨架、region 附件、图集及预览。所有部件名称、数量、骨骼父子关系和附件绑定来自输入，不使用角色名称或人形比例模板。
|
||||
|
||||
生成过程完全离线，不调用图片模型、不运行 SAM。骨架预览页面嵌入资产，但会从 CDN 加载 Spine Player 4.2，打开播放器需要网络。静态报告与 PNG 可离线查看。
|
||||
|
||||
导出完成仍标记 `quality_status=needs_review`。当前只有 setup 与单骨骼旋转检查动画；没有成品 idle/walk，也没有 mesh、权重或 IK。关节、语义、接缝及运动质量需人工复核。
|
||||
|
||||
## 2. 输入、输出与执行链路
|
||||
|
||||
```text
|
||||
SAM 目录 + 可选补全目录 + 可选绘制顺序覆盖
|
||||
→ 校验 v2 计划与骨骼关系
|
||||
→ 对比两阶段 source / 计划 / segmentation / masks 的 SHA-256
|
||||
→ 合并部件：已选补全 / 原 SAM 像素 / 隐藏无纹理节点
|
||||
→ 根据遮挡约束稳定拓扑排序，记录没有约束的重叠
|
||||
→ 按 crop_xyxy 组装，输出来源与覆盖诊断
|
||||
→ 根据 parent / bind_to / pivot_norm 创建骨架与附件
|
||||
→ 生成 skeleton.json、图集、播放器、报告与 manifest
|
||||
```
|
||||
|
||||
SAM 原部件直接由 source + mask 重建，保持 RGB/Alpha 和原坐标；无需依赖 SAM 的紧裁剪 parts/。补全部件使用 `completion-result.json` 中选中结果的 `image` 和 `crop_xyxy`，同时检查尺寸及可见 RGBA 是否保持不变。
|
||||
|
||||
## 3. 阶段执行命令示例及参数说明
|
||||
|
||||
在项目根目录运行；环境统一为 Conda `ai-spine`。以下路径仅是现有输入示例，换角色时替换输入目录，不修改代码。输出目录每次独立。
|
||||
|
||||
```powershell
|
||||
Set-Location F:\Study\taonier\AI_spine
|
||||
conda activate ai-spine
|
||||
$samDir = '.\output\20260920-225137\sam'
|
||||
$completionDir = '.\output\20260920-225137\complete'
|
||||
$spineOut = '.\output\spine-' + (Get-Date -Format 'yyyyMMdd-HHmmss')
|
||||
|
||||
node .\tools\model-check.mjs --stage spine `
|
||||
--input-dir $samDir `
|
||||
--completion-dir $completionDir `
|
||||
--out-dir $spineOut
|
||||
|
||||
Start-Process "$spineOut\report.html"
|
||||
```
|
||||
|
||||
| 参数 | 含义 |
|
||||
| --- | --- |
|
||||
| `--input-dir DIR` | 必需，SAM 输出目录 |
|
||||
| `--completion-dir DIR` | 可选，同批输入的补全输出；省略则使用 SAM 资产 |
|
||||
| `--out-dir DIR` | 独立输出目录；dry-run 时可省略 |
|
||||
| `--fallback-policy sam/error` | 默认 sam；需要补全但没有通过结果时回退可见像素并记录原因。error 则终止 |
|
||||
| `--draw-order-file FILE` | JSON 对象，`draw_order` 包含所有可绘制 ID，按从后到前排列，各出现一次 |
|
||||
| `--check-angle FLOAT` | 单骨骼检查动画正负角度，默认 10，范围 (0,45] |
|
||||
| `--genielabs-dir DIR` | 默认相邻 `spine-animation-ai`；复用其图集和播放器脚本，保留 PolyForm Noncommercial 约束 |
|
||||
| `--python FILE` | 仅允许 ai-spine 解释器 |
|
||||
| `--dry-run` | 完整检查输入、PNG、坐标及依赖脚本，不写输出，不联网 |
|
||||
| `--resume` | 仅复用输入、参数、代码与产物哈希全部匹配的完整运行 |
|
||||
|
||||
该阶段不读取模型配置与密钥，也不接受 `--config`、模型参数或补全请求参数。自动化可用 `conda run --no-capture-output -n ai-spine node ...`。
|
||||
|
||||
可选 runtime 复检：将已有的官方 Spine 4.2 core IIFE 文件路径赋给 `$runtimeFile`，执行以下命令。工具不下载或安装依赖；检查图集尺寸、静态附件顶点和正负旋转下的后代跟随，不验证浏览器 WebGL 渲染。
|
||||
|
||||
```powershell
|
||||
$runtimeFile = '.\output\spine-runtime-validation-20260922\spine-core.js'
|
||||
conda run -n ai-spine node .\tools\verify-spine-runtime.mjs `
|
||||
--runtime-file $runtimeFile --input-dir $spineOut
|
||||
```
|
||||
|
||||
## 4. 输入格式要求
|
||||
|
||||
SAM 目录需含 `source.png`、有效 v2 `semantic-parts.json`、`segmentation.json`、其中记录的全画布二值 masks。可见部件缺少 mask 直接报错；hidden 不得有可见 mask。
|
||||
|
||||
补全目录需含相同输入快照、对应 masks 和 `completion-result.json`。只有 `selected_file` 非空、`qa.accepted=true` 且 `status=geometry_passed_needs_review` 的结果会被采用；其 `image` 文件必须存在，裁剪坐标为原画布内的整数半开区间，尺寸与 PNG 一致。legacy 可见像素必须逐像素匹配 SAM；adaptive 使用 preservation 的 preserve/editable/removed 全画布二值区域，要求 preserve=SAM−editable−removed、编辑与移除不重叠、保留像素精确匹配、输出没有超出保留/编辑区域。输入哈希不一致时拒绝混用。图片和契约路径必须位于补全目录内。
|
||||
|
||||
补全失败、未处理、未提供或没有允许区域的需补全部件遵守 fallback-policy；adaptive 已重建并尝试修复的部件也视为有需求，不因原计划 needs_completion=false 就忽略失败。hidden 保留语义骨骼但不生成纹理，也不因缺纹理触发严格回退错误。
|
||||
|
||||
可选绘制顺序文件格式为 `{"draw_order":["部件ID一","部件ID二"]}`。必须列出本次全部可绘制部件，不包含 hidden。显式顺序允许覆盖模型遮挡关系，冲突会在报告中列出。
|
||||
|
||||
## 5. 执行链路中重要部分代码文件及原理说明（包括重点说明）
|
||||
|
||||
**资产整合。** 按 ID 合并，保留来源、跳过或拒绝原因、补全 QA 与上游告警。通过几何检查的补全仍是待复核资产，失败结果不会当作补全成功使用。缺少图片或非法坐标不会被静默回退掩盖。
|
||||
|
||||
**位置与骨骼。** 图像坐标原点为左上角，Spine 原点设为原画布底边中点：`X=x-W/2`，`Y=H-y`。pivot 使用连续坐标 `x=pivot_norm[0]*W`、`y=pivot_norm[1]*H`。所有骨骼 setup 旋转为零、缩放为一，因此子骨骼局部位置是世界 pivot 减父 pivot；附件位置是裁剪中心减绑定骨骼 pivot。骨骼按父先子后输出，hidden bone_part 也创建骨骼。attachment/facial_feature 绑定 bind_to，null 绑定 root，不创建额外独立骨骼。
|
||||
|
||||
**绘制顺序。** `A.occluded_by` 包含 B 时，A 排在 B 后方。排序保留隐藏节点的传递约束，最终只输出有纹理的 slot。没有约束时使用计划中的稳定顺序；若这些部件确有 Alpha 重叠，记录 unconstrained_overlaps，不能声称已确定遮挡。循环关系拒绝导出。
|
||||
|
||||
**检查动画。** 每个语义骨骼生成 `check__ID`：0→正角度→0→负角度→0，时长 2 秒，子骨骼与附件按层级跟随。setup 是静态姿态。它们供人工检查绑定和露底，不自动判定动作质量。
|
||||
|
||||
Spine 字段依据 [官方 JSON 格式说明](https://esotericsoftware.com/spine-json-format)，导出目标为 4.2；旋转时间线使用 4.2 的 `value` 字段。
|
||||
|
||||
## 6. 各个代码文件职责索引
|
||||
|
||||
| 文件 | 职责 |
|
||||
| --- | --- |
|
||||
| `tools/model-check.mjs` | 独立离线入口、参数与其他阶段隔离 |
|
||||
| `stages/spine-stage.mjs` | 验证计划、Conda 调用、指纹、导出调度与复用 |
|
||||
| `spine_assembly.py` | 资产核对、顺序、组装、诊断、骨架及报告 |
|
||||
| `lib/validate-parts.mjs` | v2 Schema、引用、骨骼和遮挡环路校验 |
|
||||
| `lib/run-artifacts.mjs` | 独立输出目录和产物哈希 |
|
||||
| 外部 `scripts/make_atlas.py` | 透明 PNG 图集打包，不修改外部依赖 |
|
||||
| 外部 `scripts/generate_spine_player.py` | 嵌入骨架和图集的 Spine Player HTML |
|
||||
| `tests/test_spine_assembly.py` / `tests/spine-stage.test.mjs` | 非人形合成数据、坐标、父子绑定、输入身份、回退、CLI 与完整导出回归 |
|
||||
| `tools/verify-spine-runtime.mjs` | 可选官方 runtime 加载、图集匹配与世界顶点/旋转复检，需提供本地 core IIFE |
|
||||
|
||||
## 7. 产物详细说明
|
||||
|
||||
| 文件 | 用途 |
|
||||
| --- | --- |
|
||||
| `source.png` / `semantic-parts.json` | 原图与语义计划快照 |
|
||||
| `images/<id>.png` | 完整可绘制部件集合，供 Spine 编辑器导入 |
|
||||
| `assembly.json` | 坐标、来源、回退、顺序、QA、输入哈希与覆盖诊断 |
|
||||
| `assembled-sam.png` / `assembled.png` | 同一绘制顺序的原 SAM / 最终资产透明组装图 |
|
||||
| `comparison.png` | 左原图、中 SAM 组装、右最终组装 |
|
||||
| `coverage-diagnostic.png` | 红色为原图前景未覆盖，黄色为多部件重叠；重叠不一定是错误 |
|
||||
| `skeleton.json` | Spine 4.2 骨骼、slot、region、setup 和关节检查动画；images 路径为 `./images/` |
|
||||
| `atlas/skeleton.atlas` / `atlas/skeleton.png` | 运行时图集 |
|
||||
| `preview.html` | 真正的骨架预览，可切换检查动画；需要联网加载播放器 |
|
||||
| `report.html` | 静态对比、诊断、来源列表与预览入口 |
|
||||
| `manifest.json` | 执行状态、质量状态、输入/代码/产物哈希、计数、依赖及限制 |
|
||||
|
||||
2026-09-22 验证记录：`conda run --no-capture-output -n ai-spine npm test` 的 26 项测试全部通过，包含 6 项新增 Python 组装用例。`output/spine-validation-latest-20260922` 和 `output/spine-validation-baseline-20260922` 分别导出 25/22 个 slot、各 16 个骨骼（含 root）；官方 Spine 4.2 runtime 对两者的 setup 顶点与各 30 个旋转采样验证通过。最新导出的播放器另经 headless Chrome 实际加载，显示 Loaded successfully 并渲染角色，截图位于 `output/spine-runtime-validation-20260922/preview.png`。这不代表资产或动作质量合格，现有输入的漏分仍显示在组装对比图中。
|
||||
|
||||
## 8. 当前阶段可改进点说明
|
||||
|
||||
- 待实现：针对未确定遮挡、pivot 和绑定的交互修正界面；目前可通过语义计划与绘制顺序文件调整。
|
||||
- 待实现：可靠的接缝、露底与旋转质量自动验收；目前输出检查动画与诊断，质量状态始终 needs_review。
|
||||
- 待实现：mesh、权重、IK，以及由动作语义驱动的成品动画；不通过固定部件名称硬套人形模板。
|
||||
- 待验证：不同角色类别与更复杂遮挡关系的资产质量。合成回归只验证数据契约及几何逻辑。
|
||||
- 待实现：本地播放器资源打包，使骨架预览也可完全离线打开。
|
||||
@@ -1,129 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Alpha constraints and deterministic seed candidates, never semantic masks."""
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from normalize_reference import prepare_rgba
|
||||
|
||||
|
||||
def pixel(point, width, height):
|
||||
# Normalized points map to their containing pixel; 1 maps to the last pixel.
|
||||
return min(width - 1, int(point[0] * width)), min(height - 1, int(point[1] * height))
|
||||
|
||||
|
||||
def ground_parts(image, plan):
|
||||
alpha = np.array(image.convert('RGBA'))[:, :, 3]
|
||||
foreground = alpha > 0
|
||||
height, width = alpha.shape
|
||||
usable_alpha = bool(foreground.any() and not foreground.all())
|
||||
silhouette_distance = cv2.distanceTransform(np.pad(foreground.astype(np.uint8), 1), cv2.DIST_L2, cv2.DIST_MASK_PRECISE)[1:-1, 1:-1]
|
||||
diagnostics, prompts = [], []
|
||||
for part in plan['parts']:
|
||||
box = part['bbox_norm_xyxy']
|
||||
entry = {
|
||||
'id': part['id'], 'status': 'hidden' if part['visibility'] == 'hidden' else 'blocked',
|
||||
'bbox_px_xyxy': None, 'bbox_foreground_ratio': None,
|
||||
'foreground_bbox_px_xyxy': None, 'foreground_components': 0,
|
||||
'model_positive_points': [], 'candidate_points': [], 'warnings': [],
|
||||
'semantic_verified': False,
|
||||
}
|
||||
for point in part.get('positive_points_norm', []):
|
||||
x, y = pixel(point, width, height)
|
||||
in_box = bool(box and box[0] <= point[0] <= box[2] and box[1] <= point[1] <= box[3])
|
||||
on_foreground = bool(foreground[y, x])
|
||||
entry['model_positive_points'].append({
|
||||
'point_norm': point, 'point_px': [x, y], 'alpha': int(alpha[y, x]),
|
||||
'inside_bbox': in_box, 'on_foreground': on_foreground,
|
||||
'status': 'UNTRUSTED_SEED' if usable_alpha and on_foreground and in_box else 'INVALID_SEED',
|
||||
})
|
||||
if part['visibility'] != 'hidden' and box:
|
||||
x0, y0 = math.floor(box[0] * width), math.floor(box[1] * height)
|
||||
x1, y1 = math.ceil(box[2] * width), math.ceil(box[3] * height)
|
||||
entry['bbox_px_xyxy'] = [x0, y0, x1, y1]
|
||||
roi = foreground[y0:y1, x0:x1]
|
||||
entry['bbox_foreground_ratio'] = float(roi.mean()) if roi.size else 0
|
||||
if not usable_alpha:
|
||||
entry['warnings'].append('NO_USABLE_ALPHA')
|
||||
elif not roi.any():
|
||||
entry['warnings'].append('EMPTY_FOREGROUND_ROI')
|
||||
else:
|
||||
ys, xs = np.nonzero(roi)
|
||||
entry['foreground_bbox_px_xyxy'] = [int(xs.min()) + x0, int(ys.min()) + y0, int(xs.max()) + x0 + 1, int(ys.max()) + y0 + 1]
|
||||
count, _ = cv2.connectedComponents(roi.astype(np.uint8), connectivity=8)
|
||||
entry['foreground_components'] = count - 1
|
||||
# Pad the ROI with background so an opaque ROI has a finite
|
||||
# distance, and candidates stay away from both silhouette/box edges.
|
||||
distance = cv2.distanceTransform(np.pad(roi.astype(np.uint8), 1), cv2.DIST_L2, cv2.DIST_MASK_PRECISE)[1:-1, 1:-1]
|
||||
# Only centers strictly inside the original continuous ROI qualify.
|
||||
eligible_x = ((np.arange(x0, x1) + .5) / width >= box[0]) & ((np.arange(x0, x1) + .5) / width <= box[2])
|
||||
eligible_y = ((np.arange(y0, y1) + .5) / height >= box[1]) & ((np.arange(y0, y1) + .5) / height <= box[3])
|
||||
eligible = roi & eligible_y[:, None] & eligible_x[None, :]
|
||||
if eligible.any():
|
||||
scores = np.where(eligible, distance, -1)
|
||||
cy, cx = np.unravel_index(np.argmax(scores), scores.shape)
|
||||
x, y = int(cx) + x0, int(cy) + y0
|
||||
entry['candidate_points'] = [{
|
||||
'point_px': [x, y], 'point_norm': [(x + .5) / width, (y + .5) / height],
|
||||
'distance_to_roi_boundary_px': float(distance[cy, cx]),
|
||||
'distance_to_silhouette_px': float(silhouette_distance[y, x]),
|
||||
'source': 'alpha_roi_distance_transform',
|
||||
}]
|
||||
entry['status'] = 'candidate'
|
||||
else:
|
||||
entry['warnings'].append('NO_PIXEL_CENTER_IN_ROI')
|
||||
if entry['bbox_foreground_ratio'] < .5:
|
||||
entry['warnings'].append('LOW_FOREGROUND_RATIO')
|
||||
if count > 2:
|
||||
entry['warnings'].append('MULTIPLE_FOREGROUND_COMPONENTS')
|
||||
entry['warnings'].append('SEMANTIC_REVIEW_REQUIRED')
|
||||
diagnostics.append(entry)
|
||||
prompts.append({
|
||||
'id': part['id'], 'text_prompt': part['semantic'], 'status': entry['status'],
|
||||
'bbox_px_xyxy': entry['bbox_px_xyxy'],
|
||||
'positive_points_px': [p['point_px'] for p in entry['candidate_points']],
|
||||
'positive_points_norm': [p['point_norm'] for p in entry['candidate_points']],
|
||||
'negative_points_px': [], 'seed_source': 'alpha_roi_distance_transform',
|
||||
'semantic_verified': False, 'requires_mask_qa': True,
|
||||
})
|
||||
points = [p for entry in diagnostics for p in entry['model_positive_points']]
|
||||
return {
|
||||
'schema_version': 1, 'width': width, 'height': height,
|
||||
'alpha_threshold': 0, 'coordinate_mapping': 'floor(norm * size), clamped at size - 1; boxes are half-open',
|
||||
'summary': {'model_points': len(points), 'invalid_model_points': sum(p['status'] == 'INVALID_SEED' for p in points),
|
||||
'background_model_points': sum(not p['on_foreground'] for p in points),
|
||||
'candidate_parts': sum(p['status'] == 'candidate' for p in diagnostics),
|
||||
'blocked_parts': sum(p['status'] == 'blocked' for p in diagnostics)},
|
||||
'parts': diagnostics,
|
||||
}, {'schema_version': 1, 'width': width, 'height': height, 'quality_status': 'needs_review', 'parts': prompts}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--input', required=True)
|
||||
parser.add_argument('--output', required=True)
|
||||
parser.add_argument('--prepare', action='store_true')
|
||||
parser.add_argument('--parts')
|
||||
parser.add_argument('--prompts')
|
||||
args = parser.parse_args()
|
||||
with Image.open(args.input) as image:
|
||||
if args.prepare:
|
||||
prepared, metadata = prepare_rgba(image)
|
||||
prepared.save(args.output, format='PNG')
|
||||
print(json.dumps(metadata))
|
||||
else:
|
||||
if not args.parts or not args.prompts:
|
||||
parser.error('--parts and --prompts are required for grounding')
|
||||
plan = json.loads(Path(args.parts).read_text(encoding='utf-8'))
|
||||
diagnostics, prompts = ground_parts(image, plan)
|
||||
Path(args.output).write_text(json.dumps(diagnostics, indent=2), encoding='utf-8')
|
||||
Path(args.prompts).write_text(json.dumps(prompts, indent=2), encoding='utf-8')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,96 +0,0 @@
|
||||
import {existsSync, readFileSync} from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {fileURLToPath} from 'node:url';
|
||||
|
||||
export const projectRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
|
||||
|
||||
export function readEnvFile(file) {
|
||||
if (!existsSync(file)) return {};
|
||||
const values = {};
|
||||
for (const line of readFileSync(file, 'utf8').replace(/^\uFEFF/u, '').split(/\r?\n/u)) {
|
||||
const match = line.match(/^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*?)\s*$/u);
|
||||
if (!match) continue;
|
||||
let value = match[2];
|
||||
if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
|
||||
value = value.slice(1, -1);
|
||||
} else value = value.replace(/\s+#.*$/u, '');
|
||||
values[match[1]] = value;
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
export function loadConfig({configFile, envFile, env = process.env} = {}) {
|
||||
const file = path.resolve(configFile || path.join(projectRoot, 'configs', 'astra-qwen.json'));
|
||||
const config = JSON.parse(readFileSync(file, 'utf8').replace(/^\uFEFF/u, ''));
|
||||
if (config.schema_version !== 1 || !config.providers || !config.stages) {
|
||||
throw new Error('配置必须声明 schema_version: 1、providers 和 stages');
|
||||
}
|
||||
return {config, file, env: {...readEnvFile(envFile || path.join(projectRoot, '.env')), ...env}};
|
||||
}
|
||||
|
||||
export function resolveStage(loaded, name, overrides = {}, {requireKey = true} = {}) {
|
||||
const stage = loaded.config.stages[name];
|
||||
if (!stage) throw new Error(`没有配置阶段：${name}`);
|
||||
const provider = loaded.config.providers[stage.provider];
|
||||
if (!provider) throw new Error(`没有配置提供方：${stage.provider}`);
|
||||
const model = String(overrides.model || stage.model || loaded.env[stage.model_env] || stage.default_model || '').trim();
|
||||
if (!model) throw new Error(`缺少 ${name} 的模型 ID`);
|
||||
const baseUrl = String(loaded.env[provider.base_url_env] || '').trim().replace(/\/+$/u, '');
|
||||
const apiKey = String(loaded.env[provider.api_key_env] || '').trim();
|
||||
if (requireKey && (!baseUrl || !apiKey)) throw new Error(`需要配置 ${provider.base_url_env} 和 ${provider.api_key_env}`);
|
||||
if (baseUrl) {
|
||||
const url = new URL(baseUrl);
|
||||
if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password || url.search || url.hash) {
|
||||
throw new Error('提供方 base URL 必须是无用户名、密码、查询参数的 HTTP(S) 地址');
|
||||
}
|
||||
}
|
||||
const protocol = overrides.protocol || provider.protocol;
|
||||
const allowed = name === 'understanding' ? ['chat-completions', 'responses'] : ['images-edits'];
|
||||
if (!allowed.includes(protocol)) throw new Error(`${name} 不支持协议：${protocol}`);
|
||||
const params = {...stage.params, ...overrides.params};
|
||||
// Semantic fields are owned by adapters, never overridden via opaque params.
|
||||
for (const key of ['model', 'messages', 'input', 'instructions', 'prompt', 'image', 'image[]', 'mask', 'response_format', 'text', 'stream', 'Authorization', 'api_key', 'apiKey']) {
|
||||
if (Object.hasOwn(params, key)) throw new Error(`params 不能覆盖保留字段：${key}`);
|
||||
}
|
||||
const timeoutMs = provider.timeout_ms ?? 180000;
|
||||
const maxRetries = provider.max_retries ?? 2;
|
||||
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || !Number.isInteger(maxRetries) || maxRetries < 0 || maxRetries > 5) {
|
||||
throw new Error('timeout_ms 必须为正数，max_retries 必须为 0～5 的整数');
|
||||
}
|
||||
const outputMode = overrides.outputMode || stage.output_mode || 'text-json';
|
||||
if (!['text-json', 'json-object', 'json-schema'].includes(outputMode)) throw new Error(`未知 output_mode：${outputMode}`);
|
||||
const stream = overrides.stream ?? stage.stream ?? false;
|
||||
if (typeof stream !== 'boolean') throw new Error('stream 必须为 boolean');
|
||||
const promptPath = path.resolve(path.dirname(loaded.file), overrides.prompt || stage.prompt);
|
||||
const capabilities = {multiple_images: false, mask: false, ...provider.capabilities};
|
||||
if (typeof capabilities.mask !== 'boolean' || typeof capabilities.multiple_images !== 'boolean') {
|
||||
throw new Error('capabilities.mask 和 multiple_images 必须为 boolean');
|
||||
}
|
||||
const imageField = provider.image_field || 'image';
|
||||
if (!['image', 'image[]'].includes(imageField)) throw new Error('image_field 必须是 image 或 image[]');
|
||||
return {
|
||||
name, provider: stage.provider, protocol, model, baseUrl, apiKey, timeoutMs, maxRetries,
|
||||
params, outputMode, stream, promptPath, prompt: readFileSync(promptPath, 'utf8'), capabilities,
|
||||
imageField, maskEditable: provider.mask_editable || 'transparent',
|
||||
};
|
||||
}
|
||||
|
||||
export function publicStage(stage) {
|
||||
return {
|
||||
provider: stage.provider, protocol: stage.protocol, model: stage.model,
|
||||
params: stage.params, output_mode: stage.outputMode, stream: stage.stream, timeout_ms: stage.timeoutMs,
|
||||
max_retries: stage.maxRetries, capabilities: stage.capabilities,
|
||||
image_field: stage.imageField, mask_editable: stage.maskEditable,
|
||||
};
|
||||
}
|
||||
|
||||
export function samInfo(loaded, overrides = {}) {
|
||||
const stage = loaded.config.stages.segmentation || {};
|
||||
const modelDir = overrides.samDir || loaded.env[stage.model_dir_env] || stage.default_model_dir || '';
|
||||
const checkpoint = overrides.samCheckpoint || (modelDir ? [path.join(modelDir, 'sam3.1checkpoint', 'sam3.1_multiplex.pt'), path.join(modelDir, 'sam3.1_multiplex.pt')].find(existsSync) || path.join(modelDir, 'sam3.1checkpoint', 'sam3.1_multiplex.pt') : '');
|
||||
const codeDir = overrides.samCodeDir || loaded.env.SAM_CODE_DIR || (modelDir ? path.join(modelDir, 'sam3') : '');
|
||||
return {model_dir: modelDir, code_dir: codeDir, checkpoint,
|
||||
checkpoint_exists: Boolean(checkpoint && existsSync(checkpoint)),
|
||||
code_exists: Boolean(codeDir && existsSync(path.join(codeDir, 'sam3', 'model_builder.py'))),
|
||||
runtime_status: 'local_subprocess_not_probed'};
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
import {execFileSync} from 'node:child_process';
|
||||
import {readFileSync} from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {projectRoot} from './config.mjs';
|
||||
|
||||
export {selectPython as geometryPython} from './python-runtime.mjs';
|
||||
|
||||
export function prepareReference(python, input, output) {
|
||||
return JSON.parse(execFileSync(python, [path.join(projectRoot, 'geometry_grounding.py'), '--prepare', '--input', input, '--output', output], {encoding: 'utf8', windowsHide: true}));
|
||||
}
|
||||
|
||||
export function groundParts(python, out) {
|
||||
const diagnostics = path.join(out, 'geometry-validation.json');
|
||||
execFileSync(python, [path.join(projectRoot, 'geometry_grounding.py'), '--input', path.join(out, 'source.png'), '--parts', path.join(out, 'semantic-parts.json'), '--output', diagnostics, '--prompts', path.join(out, 'segmentation-prompts.json')], {encoding: 'utf8', windowsHide: true});
|
||||
return JSON.parse(readFileSync(diagnostics, 'utf8'));
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
import {readFileSync} from 'node:fs';
|
||||
import {createHash} from 'node:crypto';
|
||||
import path from 'node:path';
|
||||
|
||||
export function sha256(value) { return createHash('sha256').update(value).digest('hex'); }
|
||||
|
||||
export function imageFormat(bytes) {
|
||||
if (bytes.length >= 24 && bytes.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]))) {
|
||||
return {mime: 'image/png', extension: 'png', width: bytes.readUInt32BE(16), height: bytes.readUInt32BE(20)};
|
||||
}
|
||||
if (bytes.length >= 3 && bytes[0] === 255 && bytes[1] === 216 && bytes[2] === 255) return {mime: 'image/jpeg', extension: 'jpg'};
|
||||
if (bytes.length >= 12 && bytes.toString('ascii', 0, 4) === 'RIFF' && bytes.toString('ascii', 8, 12) === 'WEBP') return {mime: 'image/webp', extension: 'webp'};
|
||||
throw new Error('图片数据不是支持的 PNG/JPEG/WebP 格式');
|
||||
}
|
||||
|
||||
export function readImage(file, {pngOnly = false} = {}) {
|
||||
const bytes = readFileSync(file);
|
||||
const format = imageFormat(bytes);
|
||||
if (pngOnly && (format.extension !== 'png' || !format.width || !format.height)) throw new Error('结构理解与局部画布当前要求非空 PNG 图片');
|
||||
return {file: path.resolve(file), bytes, ...format, sha256: sha256(bytes)};
|
||||
}
|
||||
|
||||
export function dataUrl(image) { return `data:${image.mime};base64,${image.bytes.toString('base64')}`; }
|
||||
@@ -1,23 +0,0 @@
|
||||
import {spawnSync} from 'node:child_process';
|
||||
import path from 'node:path';
|
||||
|
||||
export function selectPython(requested) {
|
||||
const command = requested || (process.env.CONDA_PREFIX
|
||||
? path.join(process.env.CONDA_PREFIX, process.platform === 'win32' ? 'python.exe' : 'bin/python')
|
||||
: 'python');
|
||||
const probe = spawnSync(command, ['-c', [
|
||||
'import json, sys',
|
||||
'from pathlib import Path',
|
||||
"print(json.dumps({'executable': sys.executable, 'name': Path(sys.prefix).name, 'conda': (Path(sys.prefix) / 'conda-meta').is_dir()}))",
|
||||
].join('\n')], {encoding: 'utf8', windowsHide: true});
|
||||
let runtime;
|
||||
try { if (probe.status === 0) runtime = JSON.parse(probe.stdout.trim()); } catch { /* Report an actionable environment error below. */ }
|
||||
if (!runtime?.conda || runtime.name !== 'ai-spine') {
|
||||
throw new Error('本项目固定使用 Conda 环境 ai-spine。请先 conda activate ai-spine，或用 conda run -n ai-spine node ...；--python 也必须指向该环境的解释器');
|
||||
}
|
||||
const dependencies = spawnSync(runtime.executable, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore', windowsHide: true});
|
||||
if (dependencies.status !== 0) {
|
||||
throw new Error('ai-spine 环境缺少或无法加载 Python 依赖。请在 AI_spine 目录执行：conda run -n ai-spine python -m pip install -r requirements.txt');
|
||||
}
|
||||
return runtime.executable;
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
import {dataUrl} from './images.mjs';
|
||||
|
||||
const escape = (value) => String(value).replace(/[&<>"']/gu, (c) => ({'&': '&', '<': '<', '>': '>', '"': '"', "'": '''}[c]));
|
||||
|
||||
export function annotatedSvg(image, parts, geometry = null) {
|
||||
const {width: w, height: h} = image;
|
||||
const stroke = Math.max(w, h) / 400;
|
||||
const font = Math.max(w, h) / 65;
|
||||
const byId = new Map(parts.map((part) => [part.id, part]));
|
||||
const diagnostics = new Map((geometry?.parts || []).map((part) => [part.id, part]));
|
||||
const overlays = parts.map((part, index) => {
|
||||
const color = `hsl(${(index * 137.5) % 360} 85% 55%)`;
|
||||
const [px, py] = [part.pivot_norm[0] * w, part.pivot_norm[1] * h];
|
||||
const parent = byId.get(part.parent);
|
||||
const line = parent ? `<line x1="${px}" y1="${py}" x2="${parent.pivot_norm[0] * w}" y2="${parent.pivot_norm[1] * h}" stroke-dasharray="${stroke * 3}"/>` : '';
|
||||
let box = '';
|
||||
if (part.bbox_norm_xyxy) {
|
||||
const [x0, y0, x1, y1] = part.bbox_norm_xyxy;
|
||||
box = `<rect x="${x0 * w}" y="${y0 * h}" width="${(x1 - x0) * w}" height="${(y1 - y0) * h}"/>`;
|
||||
}
|
||||
const diagnostic = diagnostics.get(part.id);
|
||||
const points = part.positive_points_norm.map(([x, y], i) => {
|
||||
const status = diagnostic?.model_positive_points[i]?.status;
|
||||
const fill = status === 'INVALID_SEED' ? '#ff4444' : status === 'UNTRUSTED_SEED' ? '#ffc857' : '#999';
|
||||
return `<circle cx="${x * w}" cy="${y * h}" r="${stroke * 2}" stroke="#111" fill="${fill}"><title>${escape(part.id)}: ${status || 'UNTRUSTED_MODEL_POINT'}</title></circle>`;
|
||||
}).join('');
|
||||
const candidates = (diagnostic?.candidate_points || []).map(({point_norm: [x, y]}) => `<rect x="${x * w - stroke * 2}" y="${y * h - stroke * 2}" width="${stroke * 4}" height="${stroke * 4}" stroke="#111" fill="#44e5ff"><title>${escape(part.id)}: Alpha candidate, semantic unverified</title></rect>`).join('');
|
||||
return `<g fill="none" stroke="${color}" stroke-width="${stroke}">${line}${box}${points}${candidates}<circle cx="${px}" cy="${py}" r="${stroke * 3}"/><text x="${Math.min(px + 4, w * 0.6)}" y="${Math.max(font, py - 5)}" fill="${color}" stroke="#111" stroke-width="${stroke / 2}" paint-order="stroke" font-size="${font}" font-family="sans-serif">${escape(part.id)}</text></g>`;
|
||||
}).join('\n');
|
||||
return `<svg xmlns="http://www.w3.org/2000/svg" width="${w}" height="${h}" viewBox="0 0 ${w} ${h}"><rect width="100%" height="100%" fill="#262632"/><image href="${dataUrl(image)}" width="${w}" height="${h}"/>${overlays}</svg>`;
|
||||
}
|
||||
|
||||
export function understandingReport(image, data, validation, metadata, geometry = null) {
|
||||
const svg = data && validation.valid ? annotatedSvg(image, data.parts, geometry) : `<img src="${dataUrl(image)}" alt="source">`;
|
||||
const diagnostics = new Map((geometry?.parts || []).map((part) => [part.id, part]));
|
||||
const rows = data && validation.valid ? data.parts.map((part) => {
|
||||
const g = diagnostics.get(part.id);
|
||||
return `<tr><td>${escape(part.id)}</td><td>${escape(part.rig_role || '')}<br>${escape(part.bind_to ?? part.parent ?? 'root')}</td><td>${escape(part.visibility)}</td><td>${escape(part.occluded_by.join(', '))}</td><td>${escape(Object.entries(part.completion || {}).filter(([, value]) => value).map(([key]) => key).join(', ') || '无')}</td><td>${g?.bbox_foreground_ratio == null ? '—' : `${(g.bbox_foreground_ratio * 100).toFixed(1)}%`}</td><td>${escape(g?.status || 'pending')}<br>${escape((g?.warnings || []).join(', '))}</td><td>${part.confidence}</td></tr>`;
|
||||
}).join('') : '';
|
||||
return `<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>角色结构与几何检查</title><style>body{font:16px system-ui;max-width:1400px;margin:24px auto;padding:16px;background:#f5f5f5}svg,img{max-width:100%;height:auto;max-height:85vh}td,th{padding:8px;border:1px solid #ccc}table{border-collapse:collapse}pre{white-space:pre-wrap;overflow-wrap:anywhere}</style><h1>角色结构与几何检查</h1><p>尚未执行 SAM 分割或补全。红色实心圆：无效模型点；黄色实心圆：通过前景检查但不可信的模型点；青色方块：程序生成的 Alpha 候选。空心圆是原始关节点，不做前景吸附。粗框与候选均需语义复核，Alpha 不能证明点属于目标部件。</p><p>状态：${validation.valid ? 'needs_review（待检查）' : 'failed'}</p>${svg}<h2>几何摘要</h2><pre>${escape(JSON.stringify(geometry?.summary || {}, null, 2))}</pre><h2>部件</h2><table><tr><th>ID</th><th>角色/绑定</th><th>可见性</th><th>遮挡物</th><th>补全项</th><th>框内前景</th><th>候选状态</th><th>自报置信度</th></tr>${rows}</table><h2>校验</h2><pre>${escape(JSON.stringify(validation, null, 2))}</pre><h2>模型说明</h2><pre>${escape(JSON.stringify(data?.notes || [], null, 2))}</pre><h2>请求记录</h2><pre>${escape(JSON.stringify(metadata, null, 2))}</pre></html>`;
|
||||
}
|
||||
@@ -1,34 +0,0 @@
|
||||
import {existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync} from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {sha256} from './images.mjs';
|
||||
import {redact} from '../providers/http.mjs';
|
||||
|
||||
export function jsonFile(file, value, secret = '') {
|
||||
writeFileSync(file, `${redact(JSON.stringify(value, null, 2), secret)}\n`, 'utf8');
|
||||
}
|
||||
|
||||
export function prepareRun(outDir, fingerprint, resume) {
|
||||
const out = path.resolve(outDir);
|
||||
const manifestPath = path.join(out, 'manifest.json');
|
||||
if (existsSync(out) && readdirSync(out).length) {
|
||||
if (!resume) throw new Error(`输出目录非空，请使用新目录或 --resume：${out}`);
|
||||
if (!existsSync(manifestPath)) throw new Error('无法复用：目录没有 manifest.json');
|
||||
const previous = JSON.parse(readFileSync(manifestPath, 'utf8'));
|
||||
if (previous.fingerprint !== fingerprint) throw new Error('无法复用：模型、配置、输入或代码版本变化，请使用新的输出目录');
|
||||
if (previous.status !== 'completed') throw new Error('上次任务没有完整结束，请保留诊断文件并使用新目录重跑');
|
||||
for (const [name, hash] of Object.entries(previous.artifact_hashes || {})) {
|
||||
const file = path.join(out, name);
|
||||
if (!existsSync(file) || sha256(readFileSync(file)) !== hash) throw new Error(`缓存产物缺失或被修改：${name}`);
|
||||
}
|
||||
if (!Object.keys(previous.artifact_hashes || {}).length) throw new Error('缓存缺少产物校验记录');
|
||||
return {out, manifestPath, reused: true};
|
||||
}
|
||||
mkdirSync(out, {recursive: true});
|
||||
return {out, manifestPath, reused: false};
|
||||
}
|
||||
|
||||
export function finishRun(run, manifest, artifacts, secret) {
|
||||
manifest.artifact_hashes = Object.fromEntries(artifacts.map((name) => [name, sha256(readFileSync(path.join(run.out, name)))]));
|
||||
manifest.status = 'completed';
|
||||
jsonFile(run.manifestPath, manifest, secret);
|
||||
}
|
||||
@@ -1,141 +0,0 @@
|
||||
import {readFileSync} from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {projectRoot} from './config.mjs';
|
||||
|
||||
export const partsSchema = JSON.parse(readFileSync(path.join(projectRoot, 'schemas', 'semantic-parts.schema.json'), 'utf8'));
|
||||
const legacySchema = JSON.parse(readFileSync(path.join(projectRoot, 'schemas', 'semantic-parts-v1.schema.json'), 'utf8'));
|
||||
|
||||
// Validate the subset of JSON Schema used by the checked-in contract. Keeping
|
||||
// the validation driven by that contract avoids two independent field lists.
|
||||
function validate(value, schema, at, errors) {
|
||||
const types = Array.isArray(schema.type) ? schema.type : [schema.type];
|
||||
const matches = (type) => {
|
||||
if (type === 'null') return value === null;
|
||||
if (type === 'array') return Array.isArray(value);
|
||||
if (type === 'object') return value !== null && typeof value === 'object' && !Array.isArray(value);
|
||||
if (type === 'integer') return Number.isInteger(value);
|
||||
if (type === 'number') return typeof value === 'number' && Number.isFinite(value);
|
||||
return typeof value === type;
|
||||
};
|
||||
if (!types.some(matches)) { errors.push(`${at}: 类型必须为 ${types.join('/')}`); return; }
|
||||
if (schema.enum && !schema.enum.includes(value)) errors.push(`${at}: 值不在允许列表中`);
|
||||
if (value === null) return;
|
||||
if (typeof value === 'number') {
|
||||
if (value < (schema.minimum ?? -Infinity) || value > (schema.maximum ?? Infinity)) errors.push(`${at}: 数值越界`);
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
if (schema.pattern && !new RegExp(schema.pattern, 'u').test(value)) errors.push(`${at}: 格式不合法`);
|
||||
if (value.length < (schema.minLength ?? 0) || value.length > (schema.maxLength ?? Infinity)) errors.push(`${at}: 字符串长度不合法`);
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
if (value.length < (schema.minItems ?? 0) || value.length > (schema.maxItems ?? Infinity)) errors.push(`${at}: 数组长度不合法`);
|
||||
value.forEach((item, index) => validate(item, schema.items, `${at}[${index}]`, errors));
|
||||
} else if (typeof value === 'object') {
|
||||
for (const key of schema.required || []) if (!Object.hasOwn(value, key)) errors.push(`${at}.${key}: 必填`);
|
||||
for (const [key, item] of Object.entries(value)) {
|
||||
if (schema.properties[key]) validate(item, schema.properties[key], `${at}.${key}`, errors);
|
||||
else if (schema.additionalProperties === false) errors.push(`${at}.${key}: 未知字段`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function parseParts(text) {
|
||||
const cleaned = text.trim().replace(/^```(?:json)?\s*\n?([\s\S]*?)\n?```$/u, '$1');
|
||||
return JSON.parse(cleaned);
|
||||
}
|
||||
|
||||
export function validateParts(data) {
|
||||
const errors = [];
|
||||
const warnings = [];
|
||||
const legacy = data?.schema_version === 1;
|
||||
validate(data, legacy ? legacySchema : partsSchema, '$', errors);
|
||||
if (errors.length) return {valid: false, errors, warnings};
|
||||
const ids = new Set(data.parts.map((part) => part.id));
|
||||
const byId = new Map(data.parts.map((part) => [part.id, part]));
|
||||
if (legacy) warnings.push('旧版计划：rig_role、bind_to 和 completion 将按保守规则迁移，需复核');
|
||||
if (ids.size !== data.parts.length) errors.push('部件 ID 重复');
|
||||
for (const part of data.parts) {
|
||||
if (part.id === 'root') errors.push('root 为后续导出保留 ID');
|
||||
for (const ref of [part.parent, ...part.occluded_by, ...(!legacy ? [part.bind_to] : [])].filter((item) => item !== null)) {
|
||||
if (!ids.has(ref) || ref === part.id) errors.push(`${part.id}: 引用了不存在的部件或自身 ${ref}`);
|
||||
}
|
||||
if (new Set(part.occluded_by).size !== part.occluded_by.length) errors.push(`${part.id}: occluded_by 重复`);
|
||||
const box = part.bbox_norm_xyxy;
|
||||
if (part.visibility === 'hidden') {
|
||||
if (box !== null || part.positive_points_norm.length || !part.needs_completion) errors.push(`${part.id}: 完全隐藏部件应无可见框/正点，且需要补全`);
|
||||
warnings.push(`${part.id}: 完全隐藏部件需要人工确认，不能直接生成可见 mask`);
|
||||
} else {
|
||||
if (!box || box[0] >= box[2] || box[1] >= box[3]) errors.push(`${part.id}: 可见 bbox 必须具有正面积`);
|
||||
if (box) for (const [x, y] of part.positive_points_norm) {
|
||||
if (x < box[0] || x > box[2] || y < box[1] || y > box[3]) warnings.push(`${part.id}: 模型正点不在可见框内，将丢弃并重新生成候选`);
|
||||
}
|
||||
}
|
||||
if (part.positive_points_norm.length || part.negative_points_norm.length) warnings.push(`${part.id}: 模型提示点仅用于诊断，不直接传给 SAM`);
|
||||
if (!legacy) {
|
||||
if (part.rig_role === 'bone_part' && part.bind_to !== null) errors.push(`${part.id}: bone_part 使用 parent，bind_to 必须为 null`);
|
||||
if (part.bind_to !== null && byId.get(part.bind_to)?.rig_role !== 'bone_part') errors.push(`${part.id}: bind_to 必须引用 bone_part`);
|
||||
if (part.rig_role === 'bone_part' && part.parent !== null && byId.get(part.parent)?.rig_role !== 'bone_part') errors.push(`${part.id}: bone_part 的 parent 必须引用 bone_part`);
|
||||
if (part.needs_completion !== Object.values(part.completion).some(Boolean)) errors.push(`${part.id}: needs_completion 必须等于 completion 各项的逻辑或`);
|
||||
if (part.visibility !== 'visible' && !part.completion.occlusion_fill) errors.push(`${part.id}: partial/hidden 部件需要 occlusion_fill`);
|
||||
if (part.rig_role !== 'bone_part' && (part.completion.joint_extension_parent || part.completion.joint_extension_child)) errors.push(`${part.id}: 附件/面部特征不应声明独立骨骼关节延展`);
|
||||
}
|
||||
if (part.confidence < 0.6) warnings.push(`${part.id}: 模型自报低置信度，需要复核`);
|
||||
}
|
||||
if (!errors.length) {
|
||||
const parts = new Map(data.parts.map((part) => [part.id, part]));
|
||||
for (const relation of ['parent', 'occluded_by']) {
|
||||
const visiting = new Set();
|
||||
const done = new Set();
|
||||
const visit = (id) => {
|
||||
if (visiting.has(id)) { errors.push(`${relation} 存在环：${id}`); return; }
|
||||
if (done.has(id)) return;
|
||||
visiting.add(id);
|
||||
const part = parts.get(id);
|
||||
const refs = relation === 'parent' ? [part.parent].filter(Boolean) : part.occluded_by;
|
||||
refs.forEach(visit);
|
||||
visiting.delete(id);
|
||||
done.add(id);
|
||||
};
|
||||
ids.forEach(visit);
|
||||
}
|
||||
}
|
||||
return {valid: errors.length === 0, errors, warnings};
|
||||
}
|
||||
|
||||
// Legacy results remain readable. These inferred rig/completion decisions are
|
||||
// explicit review candidates, not a claim that the model supplied them.
|
||||
export function upgradeParts(data) {
|
||||
if (data.schema_version !== 1) return structuredClone(data);
|
||||
const result = structuredClone(data);
|
||||
result.schema_version = 2;
|
||||
const byId = new Map(result.parts.map((part) => [part.id, part]));
|
||||
for (const part of result.parts) {
|
||||
part.rig_role = /^(eye|eyes|eyebrow|nose|mouth|mustache)(_|$)/u.test(part.id) ? 'facial_feature'
|
||||
: /^(pelvis|torso|neck|head|upper_arm|forearm|hand|thigh|shin|foot)(_|$)/u.test(part.id) ? 'bone_part' : 'attachment';
|
||||
}
|
||||
const nearestBone = (id) => {
|
||||
const seen = new Set();
|
||||
while (id && !seen.has(id)) {
|
||||
seen.add(id);
|
||||
const part = byId.get(id);
|
||||
if (!part) return null;
|
||||
if (part.rig_role === 'bone_part') return id;
|
||||
id = part.parent;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
for (const part of result.parts) {
|
||||
const sleeveBone = part.id.replace(/^sleeve_/u, 'upper_arm_');
|
||||
part.bind_to = part.rig_role === 'bone_part' ? null
|
||||
: part.id.startsWith('sleeve_') && byId.get(sleeveBone)?.rig_role === 'bone_part' ? sleeveBone : nearestBone(part.parent);
|
||||
if (part.rig_role === 'bone_part') part.parent = nearestBone(part.parent);
|
||||
part.completion = {
|
||||
occlusion_fill: part.visibility !== 'visible' || part.needs_completion,
|
||||
joint_extension_parent: part.rig_role === 'bone_part' && part.parent !== null,
|
||||
joint_extension_child: part.rig_role === 'bone_part' && result.parts.some((child) => child.rig_role === 'bone_part' && nearestBone(child.parent) === part.id),
|
||||
};
|
||||
part.needs_completion = Object.values(part.completion).some(Boolean);
|
||||
}
|
||||
result.notes.push('Migrated from schema v1: rig roles/bindings and completion are heuristic review candidates. Original model points are diagnostic only; use segmentation-prompts.json for geometry candidates.');
|
||||
return result;
|
||||
}
|
||||
@@ -3,50 +3,14 @@
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from PIL import Image
|
||||
import numpy as np
|
||||
import cv2
|
||||
|
||||
|
||||
def prepare_rgba(image, white_threshold=240):
|
||||
"""Keep existing alpha; remove only border-connected near-white background.
|
||||
|
||||
No crop or rescale here: semantic coordinates must retain the full canvas.
|
||||
White regions enclosed by character outlines are deliberately preserved.
|
||||
"""
|
||||
rgba = np.array(image.convert('RGBA'))
|
||||
alpha = rgba[:, :, 3]
|
||||
method = 'existing_alpha'
|
||||
removed = 0
|
||||
if not np.any(alpha == 0):
|
||||
white = np.all(rgba[:, :, :3] >= white_threshold, axis=2).astype(np.uint8)
|
||||
_, labels = cv2.connectedComponents(white, connectivity=4)
|
||||
border = np.unique(np.concatenate((labels[0], labels[-1], labels[:, 0], labels[:, -1])))
|
||||
background = np.isin(labels, border[border != 0])
|
||||
removed = int(background.sum())
|
||||
rgba[background, 3] = 0
|
||||
method = 'border_connected_white'
|
||||
foreground = rgba[:, :, 3] > 0
|
||||
if not foreground.any():
|
||||
raise ValueError('Reference has no foreground after background removal')
|
||||
if foreground.all():
|
||||
raise ValueError('No transparent background or border-connected white background; supply a cutout RGBA PNG')
|
||||
return Image.fromarray(rgba), {
|
||||
'method': method, 'white_threshold': white_threshold,
|
||||
'removed_pixels': removed, 'foreground_pixels': int(foreground.sum()),
|
||||
'width': image.width, 'height': image.height,
|
||||
'needs_review': method != 'existing_alpha',
|
||||
}
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--input', required=True)
|
||||
p.add_argument('--output', required=True)
|
||||
p.add_argument('--target', type=int, default=1024)
|
||||
p.add_argument('--remove-background', action='store_true')
|
||||
a = p.parse_args()
|
||||
im = Image.open(a.input).convert('RGBA')
|
||||
if a.remove_background:
|
||||
im, _ = prepare_rgba(im)
|
||||
longest = max(im.width, im.height)
|
||||
if longest < a.target:
|
||||
scale = a.target / longest
|
||||
|
||||
+1
-6
@@ -2,10 +2,5 @@
|
||||
"name": "genarrative-genielabs-spine-pipeline",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"run": "node pipeline.mjs",
|
||||
"models": "node tools/model-check.mjs --list-models",
|
||||
"test": "node --test"
|
||||
},
|
||||
"engines": {"node": ">=20"}
|
||||
"scripts": {"run": "node pipeline.mjs"}
|
||||
}
|
||||
|
||||
+162
-34
@@ -8,30 +8,39 @@
|
||||
*/
|
||||
import {Buffer} from 'node:buffer';
|
||||
import {spawnSync} from 'node:child_process';
|
||||
import {cpSync, existsSync, mkdirSync, readFileSync, rmSync, unlinkSync, writeFileSync} from 'node:fs';
|
||||
import {fileURLToPath} from 'node:url';
|
||||
import {cpSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync} from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {selectPython} from './lib/python-runtime.mjs';
|
||||
|
||||
const repoRoot = path.dirname(fileURLToPath(import.meta.url));
|
||||
const defaultGenielabsDir = path.resolve(repoRoot, '..', 'spine-animation-ai');
|
||||
const repoRoot = path.dirname(new URL(import.meta.url).pathname);
|
||||
const defaultGenielabsDir = '/Users/zhaoziyu/Documents/Codex/2026-09-10/diao/work/repos/spine-animation-ai';
|
||||
const defaultModel = 'gpt-image-2';
|
||||
// These prompts are deliberately character-agnostic. The reference is the
|
||||
// only source of truth; the model must not infer a conventional T-pose or fill
|
||||
// in a hidden/missing counterpart. Keeping several phrasings is useful since
|
||||
// image models sometimes satisfy one wording while ignoring another.
|
||||
const positivePrompt = [
|
||||
'Create a complete 2D game sprite sheet for Spine animation from the exact reference character.',
|
||||
'Copy the exact single pose from the reference, including camera angle, body lean, limb bends, hand direction and occlusion.',
|
||||
'Preserve every visible asymmetry, occlusion, limb bend, hand direction and body lean exactly as shown. Do not mirror, straighten, normalize or symmetrize the pose.',
|
||||
'Output one set of isolated visible parts only. Do not add alternate poses, duplicate parts, extra props, effects or text. Keep the original style and lay parts on a white background.',
|
||||
'Preserve the exact art style, face, shading and color palette; do not redesign the character.',
|
||||
'Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image.',
|
||||
'Use the reference as the only ground truth. Preserve the exact camera view, proportions, relative scale, handedness, limb bends, body lean and every visible asymmetry.',
|
||||
'Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable).',
|
||||
'Move cutout pieces apart by translation only. Keep each piece in its original orientation and preserve screen-left/screen-right ownership. Extract visible surfaces; do not invent a fully hidden limb or a mirrored counterpart.',
|
||||
'Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part.',
|
||||
'Output isolated cutout pieces ONLY. Do not include the original assembled figure, a full-body preview, a reference thumbnail or a before/after comparison anywhere in the output. Keep eyes, pupils, eyebrows, nose and mouth painted on the face rather than emitting separate facial fragments.',
|
||||
'Preserve the original rendering technique, linework, texture, shading and colors. Leave clear white gaps between every pair of pieces. Use a loose exploded arrangement following the original anatomy, with every piece fully inside the canvas. Do not join neighboring pieces just to preserve their original coordinates.',
|
||||
].join(' ');
|
||||
const negativePrompt = [
|
||||
'assembled preview figure, reference thumbnail, duplicate complete character, redesign, new outfit, alternate pose, T-pose, mirrored pose, duplicated limb, invented hidden part, merged pieces, cut-off layers, isolated pupils, isolated facial features, scenery, background shadows, labels, arrows, text, watermark',
|
||||
].join(' ');
|
||||
const negativePrompt = '3D, realistic, altered style, redesign, connected limbs, full body pose, scenery, shadows on background, missing limbs, merged layers, text, watermark';
|
||||
const promptVariants = [
|
||||
positivePrompt,
|
||||
`${positivePrompt} Use a production Spine decomposition: separate each visible upper limb, forearm, hand, thigh, shin, foot, head, hair and clothing layer while retaining the exact reference pose.`,
|
||||
`Technical 2D cutout sheet. Treat the reference as ground truth. Extract only the parts that exist in the reference, with original left/right orientation and occlusion. Never invent a symmetric counterpart or a second version of any part. ${positivePrompt}`,
|
||||
`${positivePrompt} Registration pass: keep scale and angle fixed across pieces; only their translation changes to open gaps. Retain the exact screen-left/screen-right ownership.`,
|
||||
`${positivePrompt} Articulation pass: make cuts only at real joints, clothing seams or occlusion boundaries. Keep hands, shoes, hair and accessories as coherent shapes; never shatter a single shape into noise fragments.`,
|
||||
`${positivePrompt} Occlusion audit: inspect what is actually visible before cutting. Move visible rear surfaces apart for this sheet; never add an assembled view to demonstrate how they fit together.`,
|
||||
`${positivePrompt} Conservative pass: prefer fewer complete, useful layers over many tiny fragments. Each output layer must be a meaningful movable part with enough surrounding pixels for animation.`,
|
||||
`${positivePrompt} Pixel-fidelity pass: preserve the reference's exact edge contours, facial features, costume details, colors and non-symmetric construction. Do not redraw or reinterpret the artwork.`,
|
||||
];
|
||||
|
||||
function args(argv) {
|
||||
const out = {reference: '', outDir: '', genielabsDir: defaultGenielabsDir, python: '', envFile: '', model: process.env.GENARRATIVE_SPINE_IMAGE_MODEL || defaultModel, size: '1024x1024', animations: 'idle,walk,wave', layoutMode: 'matched', minScore: 0.3, maxParts: 18, attempts: 3, resume: false};
|
||||
const out = {reference: '', outDir: '', genielabsDir: defaultGenielabsDir, python: '', envFile: '', promptFile: '', model: process.env.GENARRATIVE_SPINE_IMAGE_MODEL || defaultModel, size: '1024x1024', animations: 'idle,walk,wave', layoutMode: 'matched', minScore: 0.3, maxParts: 28, attempts: 6, resume: false};
|
||||
for (let i = 2; i < argv.length; i += 1) {
|
||||
const key = argv[i];
|
||||
if (key === '--reference') out.reference = argv[++i] || '';
|
||||
@@ -39,13 +48,14 @@ function args(argv) {
|
||||
else if (key === '--genielabs-dir') out.genielabsDir = argv[++i] || '';
|
||||
else if (key === '--python') out.python = argv[++i] || 'python3';
|
||||
else if (key === '--env-file') out.envFile = argv[++i] || '';
|
||||
else if (key === '--prompt-file') out.promptFile = argv[++i] || '';
|
||||
else if (key === '--image-model') out.model = argv[++i] || defaultModel;
|
||||
else if (key === '--size') out.size = argv[++i] || '1024x1024';
|
||||
else if (key === '--animations') out.animations = argv[++i] || 'idle';
|
||||
else if (key === '--layout-mode') out.layoutMode = argv[++i] || 'matched';
|
||||
else if (key === '--min-score') out.minScore = Number(argv[++i] || 0.55);
|
||||
else if (key === '--max-parts') out.maxParts = Number(argv[++i] || 18);
|
||||
else if (key === '--attempts') out.attempts = Math.max(1, Math.min(3, Number(argv[++i] || 3)));
|
||||
else if (key === '--attempts') out.attempts = Math.max(1, Math.min(promptVariants.length, Number(argv[++i] || promptVariants.length)));
|
||||
else if (key === '--resume') out.resume = true;
|
||||
else if (key === '--help' || key === '-h') { console.log('用法：node pipeline.mjs --reference ref.png --out-dir ./output --genielabs-dir /path/to/spine-animation-ai'); process.exit(0); }
|
||||
}
|
||||
@@ -58,8 +68,12 @@ function run(command, commandArgs, cwd = repoRoot) {
|
||||
}
|
||||
|
||||
function resolveModel(model) {
|
||||
// Preserve exact provider IDs so experiments never silently use another model.
|
||||
return String(model || '').trim() || defaultModel;
|
||||
const normalized = String(model || '').trim();
|
||||
if (normalized === 'gpt-image-2.5' || normalized === 'gpt image 2.5') {
|
||||
console.warn('提示：VectorEngine 当前没有 gpt-image-2.5，已兼容映射为 gpt-image-2。');
|
||||
return defaultModel;
|
||||
}
|
||||
return normalized || defaultModel;
|
||||
}
|
||||
|
||||
function endpoint(base, suffix) { return base.endsWith('/v1') ? `${base}/${suffix}` : `${base}/v1/${suffix}`; }
|
||||
@@ -83,6 +97,24 @@ function loadEnvFile(file) {
|
||||
return result;
|
||||
}
|
||||
|
||||
function selectPython(requested) {
|
||||
const candidates = [requested, '/Users/zhaoziyu/anaconda3/bin/python', '/opt/homebrew/bin/python3', 'python3'].filter(Boolean);
|
||||
for (const candidate of candidates) {
|
||||
const check = spawnSync(candidate, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore'});
|
||||
if (check.status === 0) return candidate;
|
||||
}
|
||||
const venvPython = path.join(repoRoot, '.venv', 'bin', 'python');
|
||||
if (!existsSync(venvPython)) {
|
||||
const bootstrap = spawnSync('python3', ['-m', 'venv', path.join(repoRoot, '.venv')], {stdio: 'inherit'});
|
||||
if (bootstrap.status !== 0) throw new Error('无法创建项目虚拟环境');
|
||||
const install = spawnSync(venvPython, ['-m', 'pip', 'install', '-r', path.join(repoRoot, 'requirements.txt')], {stdio: 'inherit'});
|
||||
if (install.status !== 0) throw new Error('Python 依赖安装失败，请检查网络或手动安装 requirements.txt');
|
||||
}
|
||||
const check = spawnSync(venvPython, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore'});
|
||||
if (check.status === 0) return venvPython;
|
||||
throw new Error('虚拟环境安装后仍缺少 cv2、Pillow、numpy');
|
||||
}
|
||||
|
||||
async function requestAtlas(reference, model, size, config, prompt) {
|
||||
for (let attempt = 0; attempt < 3; attempt += 1) {
|
||||
const form = new FormData();
|
||||
@@ -95,7 +127,7 @@ async function requestAtlas(reference, model, size, config, prompt) {
|
||||
if (!response.ok) {
|
||||
const message = payload.error?.message || 'unknown error';
|
||||
if ((response.status === 429 || response.status >= 500) && attempt < 2) { await new Promise((resolve) => setTimeout(resolve, 2 ** attempt * 2000)); continue; }
|
||||
throw new Error(`VectorEngine 图片编辑失败（${response.status}）：${message}`);
|
||||
throw Object.assign(new Error(`VectorEngine 图片编辑失败（${response.status}）：${message}`), {status: response.status});
|
||||
}
|
||||
const item = payload.data?.[0] || payload.images?.[0] || payload.output?.[0];
|
||||
if (!item) throw new Error('VectorEngine 未返回图片');
|
||||
@@ -169,8 +201,6 @@ function filterLayout(layoutPath, partsDir, minScore, maxParts) {
|
||||
}
|
||||
for (const name of Object.keys(layout.parts || {})) {
|
||||
if (!keep.has(name)) {
|
||||
const file = path.join(partsDir, `${name}.png`);
|
||||
if (existsSync(file)) unlinkSync(file);
|
||||
delete layout.parts[name];
|
||||
}
|
||||
}
|
||||
@@ -186,6 +216,98 @@ function pngSize(file) {
|
||||
throw new Error('参考图目前需要 PNG 格式');
|
||||
}
|
||||
|
||||
// Score a generated sheet without assuming a particular character or an
|
||||
// expected number of limbs. Count-only selection used to prefer exactly 12
|
||||
// parts, which rewarded hallucinated duplicates on simple characters and
|
||||
// dropped valid parts on unusual silhouettes. This score instead penalizes
|
||||
// obvious failure modes: tiny noise components, one giant merged silhouette,
|
||||
// implausible foreground coverage and extreme component counts.
|
||||
function candidateQuality(stats) {
|
||||
const width = Number(stats.atlas_width || 0);
|
||||
const height = Number(stats.atlas_height || 0);
|
||||
const area = Math.max(1, width * height);
|
||||
const parts = Array.isArray(stats.parts) ? stats.parts : [];
|
||||
const count = parts.length;
|
||||
if (!count) return {score: Number.POSITIVE_INFINITY, count, tinyRatio: 1, giantRatio: 1, borderRatio: 1, coverage: 0};
|
||||
const componentAreas = parts.map((part) => Math.max(0, Number(part.area || 0)));
|
||||
const minUsefulArea = Math.max(500, area * 0.00045);
|
||||
const tiny = componentAreas.filter((value) => value < minUsefulArea).length;
|
||||
const giant = componentAreas.filter((value) => value > area * 0.42).length;
|
||||
const border = parts.filter((part) => Number(part.x || 0) <= 1 || Number(part.y || 0) <= 1 || Number(part.x || 0) + Number(part.width || 0) >= width - 1 || Number(part.y || 0) + Number(part.height || 0) >= height - 1).length;
|
||||
const coverage = componentAreas.reduce((sum, value) => sum + value, 0) / area;
|
||||
const countPenalty = count < 4 ? (4 - count) * 5 : count > 28 ? (count - 28) * 1.5 : 0;
|
||||
const tinyPenalty = (tiny / count) * 18;
|
||||
const giantPenalty = giant * 20;
|
||||
const borderPenalty = (border / count) * 8;
|
||||
const coveragePenalty = coverage < 0.015 ? 8 : coverage > 0.88 ? 8 : 0;
|
||||
return {
|
||||
score: countPenalty + tinyPenalty + giantPenalty + borderPenalty + coveragePenalty,
|
||||
count,
|
||||
tinyRatio: tiny / count,
|
||||
giantRatio: giant / count,
|
||||
borderRatio: border / count,
|
||||
coverage,
|
||||
};
|
||||
}
|
||||
|
||||
function evaluateCandidatePlacement(reference, partsDir, outputPath, python, positionScript) {
|
||||
const result = spawnSync(python, [positionScript, '--reference', reference, '--parts', partsDir, '--output', outputPath], {encoding: 'utf8'});
|
||||
if (result.status !== 0 || !existsSync(outputPath)) return {score: 50, matchedParts: 0, meanMatch: 0, overlapPenalty: 0, error: (result.stderr || '').trim().slice(-400)};
|
||||
try {
|
||||
const layout = JSON.parse(readFileSync(outputPath, 'utf8').replace(/\b(?:NaN|Infinity|-Infinity)\b/gu, 'null'));
|
||||
const parts = Object.values(layout.parts || {});
|
||||
const matchScores = parts.map((part) => Number(part.score || 0)).filter(Number.isFinite);
|
||||
let overlapPenalty = 0;
|
||||
for (let i = 0; i < parts.length; i += 1) {
|
||||
for (let j = i + 1; j < parts.length; j += 1) {
|
||||
const a = parts[i]; const b = parts[j];
|
||||
const ix = Math.max(0, Math.min(a.x + a.width, b.x + b.width) - Math.max(a.x, b.x));
|
||||
const iy = Math.max(0, Math.min(a.y + a.height, b.y + b.height) - Math.max(a.y, b.y));
|
||||
const intersection = ix * iy;
|
||||
const union = Math.max(1, a.width * a.height + b.width * b.height - intersection);
|
||||
const iou = intersection / union;
|
||||
if (iou > 0.55) overlapPenalty += (iou - 0.55) * 12;
|
||||
}
|
||||
}
|
||||
const meanMatch = matchScores.length ? matchScores.reduce((sum, value) => sum + value, 0) / matchScores.length : 0;
|
||||
const score = (1 - meanMatch) * 10 + overlapPenalty + Math.max(0, 5 - parts.length) * 2;
|
||||
return {score, matchedParts: parts.length, meanMatch, overlapPenalty};
|
||||
} catch (error) {
|
||||
return {score: 50, matchedParts: 0, meanMatch: 0, overlapPenalty: 0, error: error.message};
|
||||
}
|
||||
}
|
||||
|
||||
function repairLayerOrder(layoutPath) {
|
||||
const layout = JSON.parse(readFileSync(layoutPath, 'utf8').replace(/\b(?:NaN|Infinity|-Infinity)\b/gu, 'null'));
|
||||
const order = [...(layout.z_order || Object.keys(layout.parts || {}))];
|
||||
const index = new Map(order.map((name, i) => [name, i]));
|
||||
const parts = layout.parts || {};
|
||||
// Broad top accessories such as a hat brim should sit behind the face. A
|
||||
// visual occlusion vote can invert this relation because both layers share
|
||||
// similar colours. Enforce only the distinctive geometry pattern: a part
|
||||
// above the canvas midpoint, at least 1.45x wider than a lower overlapping
|
||||
// part. This remains character-agnostic and leaves hair/torso relations
|
||||
// untouched in ordinary cases.
|
||||
for (const [wideName, wide] of Object.entries(parts)) {
|
||||
const canvasHeight = Number(layout.canvas_height || 0);
|
||||
if (Number(wide.height) > Number(wide.width) * 0.65 || Number(wide.y) + Number(wide.height) / 2 > canvasHeight * 0.35) continue;
|
||||
for (const [narrowName, narrow] of Object.entries(parts)) {
|
||||
if (wideName === narrowName || Number(wide.y) >= Number(narrow.y) || Number(wide.width) < Number(narrow.width) * 1.45) continue;
|
||||
const ix = Math.max(0, Math.min(wide.x + wide.width, narrow.x + narrow.width) - Math.max(wide.x, narrow.x));
|
||||
const iy = Math.max(0, Math.min(wide.y + wide.height, narrow.y + narrow.height) - Math.max(wide.y, narrow.y));
|
||||
const overlap = (ix * iy) / Math.max(1, Number(narrow.width) * Number(narrow.height));
|
||||
if (overlap < 0.08) continue;
|
||||
const wi = index.get(wideName); const ni = index.get(narrowName);
|
||||
if (wi === undefined || ni === undefined || wi < ni) continue;
|
||||
order.splice(wi, 1);
|
||||
order.splice(order.indexOf(narrowName), 0, wideName);
|
||||
order.forEach((name, i) => index.set(name, i));
|
||||
}
|
||||
}
|
||||
layout.z_order = order;
|
||||
writeFileSync(layoutPath, JSON.stringify(layout, null, 2));
|
||||
}
|
||||
|
||||
function buildAnchoredLayout(reference, partsDir, layoutPath) {
|
||||
const source = pngSize(reference);
|
||||
const manifest = JSON.parse(readFileSync(path.join(partsDir, 'parts-manifest.json'), 'utf8'));
|
||||
@@ -227,12 +349,6 @@ function writeReferencePreview(reference, outputPath) {
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const modeIndex = process.argv.indexOf('--mode');
|
||||
if (modeIndex >= 0) {
|
||||
if (process.argv[modeIndex + 1] !== 'annotated-parts') throw new Error('新增 --mode 仅支持 annotated-parts；旧流程无需 --mode');
|
||||
const {main: modelMain} = await import('./tools/model-check.mjs');
|
||||
return modelMain(process.argv.slice(2));
|
||||
}
|
||||
const options = args(process.argv);
|
||||
if (!options.reference || !existsSync(options.reference) || !options.outDir || !options.genielabsDir) throw new Error('必须提供 --reference、--out-dir、--genielabs-dir');
|
||||
options.python = selectPython(options.python);
|
||||
@@ -244,7 +360,7 @@ async function main() {
|
||||
for (const file of [positionScript, buildScript, atlasScript, playerScript]) if (!existsSync(file)) throw new Error(`Genielabs 脚本不存在：${file}`);
|
||||
const out = path.resolve(options.outDir); const parts = path.join(out, 'parts'); const atlasDir = path.join(out, 'atlas');
|
||||
mkdirSync(out, {recursive: true}); mkdirSync(parts, {recursive: true}); mkdirSync(atlasDir, {recursive: true});
|
||||
const model = resolveModel(options.model); const config = envConfig(options.envFile);
|
||||
const model = resolveModel(options.model);
|
||||
const atlasPath = path.join(out, 'generated-parts-sheet.png');
|
||||
writeFileSync(path.join(out, 'source-original.png'), readFileSync(options.reference));
|
||||
const positionReference = path.join(out, 'reference-position.png');
|
||||
@@ -253,10 +369,12 @@ async function main() {
|
||||
writeFileSync(path.join(out, 'manifest.json'), JSON.stringify({status: 'generating', model, requestedModel: options.model, pipelineVersion: 'genarrative-genielabs-spine-v1', source: path.basename(options.reference)}, null, 2));
|
||||
const candidateRoot = path.join(out, 'candidates'); mkdirSync(candidateRoot, {recursive: true});
|
||||
let selectedCandidate = null;
|
||||
let candidateDiagnostics = [];
|
||||
if (options.resume && existsSync(atlasPath) && existsSync(path.join(parts, 'parts-manifest.json'))) {
|
||||
selectedCandidate = {atlas: atlasPath, parts};
|
||||
console.log(`复用已有拆件工作图：${atlasPath}`);
|
||||
} else {
|
||||
const config = envConfig(options.envFile);
|
||||
const candidates = [];
|
||||
for (let index = 0; index < options.attempts; index += 1) {
|
||||
const candidateDir = path.join(candidateRoot, `attempt-${index + 1}`); mkdirSync(candidateDir, {recursive: true});
|
||||
@@ -264,13 +382,22 @@ async function main() {
|
||||
const candidateParts = path.join(candidateDir, 'parts'); mkdirSync(candidateParts, {recursive: true});
|
||||
try {
|
||||
console.log(`尝试第 ${index + 1}/${options.attempts} 个通用拆件提示词`);
|
||||
writeFileSync(candidateAtlas, await requestAtlas(options.reference, model, options.size, config, promptVariants[index % promptVariants.length]));
|
||||
const prompt = options.promptFile ? readFileSync(options.promptFile, 'utf8') : promptVariants[index % promptVariants.length];
|
||||
writeFileSync(path.join(candidateDir, 'prompt.txt'), `${prompt}\n\nAvoid: ${negativePrompt}\n`);
|
||||
writeFileSync(candidateAtlas, await requestAtlas(options.reference, model, options.size, config, prompt));
|
||||
run(options.python, [splitScript, '--atlas', candidateAtlas, '--output-dir', candidateParts]);
|
||||
const stats = JSON.parse(readFileSync(path.join(candidateParts, 'parts-manifest.json'), 'utf8'));
|
||||
const count = stats.parts.length; const score = Math.abs(count - 12) + (count > 24 ? count - 24 : 0);
|
||||
candidates.push({atlas: candidateAtlas, parts: candidateParts, count, score});
|
||||
console.log(`候选 ${index + 1}: ${count} 个部件，质量分 ${score}`);
|
||||
} catch (error) { console.warn(`候选 ${index + 1} 失败：${error.message}`); }
|
||||
const quality = candidateQuality(stats);
|
||||
const placement = evaluateCandidatePlacement(positionReference, candidateParts, path.join(candidateDir, 'layout-eval.json'), options.python, positionScript);
|
||||
const totalScore = quality.score + placement.score;
|
||||
candidates.push({atlas: candidateAtlas, parts: candidateParts, count: quality.count, score: totalScore, quality, placement});
|
||||
candidateDiagnostics.push({attempt: index + 1, count: quality.count, score: totalScore, quality, placement});
|
||||
console.log(`候选 ${index + 1}: ${quality.count} 个部件，综合分 ${totalScore.toFixed(2)}（拆件 ${quality.score.toFixed(2)}，定位 ${placement.score.toFixed(2)}，匹配 ${(placement.meanMatch * 100).toFixed(0)}%，重叠罚分 ${placement.overlapPenalty.toFixed(2)}）`);
|
||||
} catch (error) {
|
||||
candidateDiagnostics.push({attempt: index + 1, status: 'failed', error: error.message});
|
||||
console.warn(`候选 ${index + 1} 失败：${error.message}`);
|
||||
if (error.status === 401 || error.status === 403) break;
|
||||
}
|
||||
}
|
||||
if (!candidates.length) throw new Error('所有拆件候选均失败');
|
||||
selectedCandidate = candidates.sort((a, b) => a.score - b.score)[0];
|
||||
@@ -281,6 +408,7 @@ async function main() {
|
||||
if (options.layoutMode === 'matched') {
|
||||
run(options.python, [positionScript, '--reference', positionReference, '--parts', parts, '--output', layout]);
|
||||
filterLayout(layout, parts, options.minScore, options.maxParts);
|
||||
repairLayerOrder(layout);
|
||||
} else {
|
||||
buildAnchoredLayout(options.reference, parts, layout);
|
||||
}
|
||||
@@ -295,8 +423,8 @@ async function main() {
|
||||
writeFileSync(skeleton, JSON.stringify(skeletonJson, null, 2));
|
||||
run(options.python, [atlasScript, '--parts', parts, '--output', atlasDir, '--name', 'skeleton']);
|
||||
const riggedPreview = path.join(out, 'preview-rigged.html'); run(options.python, [playerScript, '--skeleton', skeleton, '--atlas', path.join(atlasDir, 'skeleton.atlas'), '--atlas-image', path.join(atlasDir, 'skeleton.png'), '--output', riggedPreview, '--animation', options.animations.split(',')[0] || 'idle']);
|
||||
const preview = path.join(out, 'preview.html'); writeReferencePreview(options.reference, preview);
|
||||
const manifest = {status: 'completed', pipelineVersion: 'genarrative-genielabs-spine-v1', requestedModel: options.model, model, source: path.basename(options.reference), attempts: options.attempts, selectedCandidate: selectedCandidate.count ?? null, promptVariants: promptVariants.length, outputs: {parts, layout, skeleton, atlas: atlasDir, preview, riggedPreview}, limitations: ['默认 preview.html 显示参考图的真实非对称姿态，保证形体结构正常；preview-rigged.html 显示自动拆件 rig，需人工校正。', '当前自动生成的是 region attachments；mesh/weights/IK 仍需在 Spine 中人工校正。', 'Genielabs 脚本为外部依赖，请按其 PolyForm Noncommercial 许可证处理商业授权。']};
|
||||
const preview = path.join(out, 'preview.html'); writeReferencePreview(path.join(out, 'assembled.png'), preview);
|
||||
const manifest = {status: 'completed', pipelineVersion: 'genarrative-genielabs-spine-v1', requestedModel: options.model, model, source: path.basename(options.reference), attempts: options.attempts, selectedCandidate: selectedCandidate.count ?? null, selectedCandidateScore: selectedCandidate.score ?? null, candidateDiagnostics, promptVariants: promptVariants.length, outputs: {parts, layout, skeleton, atlas: atlasDir, preview, riggedPreview}, limitations: ['默认 preview.html 显示参考图的真实非对称姿态，保证形体结构正常；preview-rigged.html 显示自动拆件 rig，需人工校正。', '当前自动生成的是 region attachments；mesh/weights/IK 仍需在 Spine 中人工校正。', 'Genielabs 脚本为外部依赖，请按其 PolyForm Noncommercial 许可证处理商业授权。']};
|
||||
writeFileSync(path.join(out, 'manifest.json'), JSON.stringify(manifest, null, 2)); console.log(`Spine pipeline 完成：${out}`);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
Repair and complete the specified character part on the first input image.
|
||||
The task declares editable regions for segmentation gaps, occlusion, joint
|
||||
extensions, and removal of overlaid attachments from their underlying base.
|
||||
Only regions explicitly marked editable may be changed. Preserve all other
|
||||
visible pixels. Any subsequent images or labeled reference panels are context.
|
||||
Preserve the exact original pose, scale, screen direction, perspective, colors,
|
||||
line thickness and shading. Do not mirror, rotate, resize, recenter or redesign.
|
||||
Keep the canvas dimensions and the position of every already-visible feature.
|
||||
Extend the hidden joint area naturally behind its occluder, without adding other
|
||||
body parts, props, labels, annotations or extra versions of the same part.
|
||||
If an edit mask is supplied, transparent mask pixels are editable. Otherwise,
|
||||
use the task's bounded extension instructions and keep the remaining background
|
||||
pure white or transparent. Return the same canvas size without recentering.
|
||||
Downstream checks reject misalignment and constrain modifications to edit regions,
|
||||
restore protected source pixels, and reverse the documented canvas transform.
|
||||
Do not rely on restoration to hide movement or redesign.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user