1 Commits

Author SHA1 Message Date
ziyu 3753f110f4 Improve character cutout prompts and offline candidate evaluation 2026-09-16 01:52:43 +08:00
92 changed files with 34324 additions and 3940 deletions
-16
View File
@@ -3,19 +3,3 @@ VECTOR_ENGINE_BASE_URL=https://your-vector-engine.example
VECTOR_ENGINE_API_KEY=replace-me
# Optional request timeout in milliseconds
VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS=1200000
# Optional independent relay: select with --config configs/tiantoken.json
TIANTOKEN_BASE_URL=https://api.tiantoken.com
TIANTOKEN_API_KEY=replace-me
# Defaults in tiantoken.json; independent from SPINE_* model overrides.
# TIANTOKEN_UNDERSTANDING_MODEL=gpt-6-astra
# TIANTOKEN_COMPLETION_MODEL=gpt-image-2
# Independent model IDs for the annotated-parts pipeline. Use EXACT relay IDs.
# Confirm with: node tools/model-check.mjs --list-models
SPINE_UNDERSTANDING_MODEL=gpt-6-astra
SPINE_COMPLETION_MODEL=qwen-image-3.0
# Local source + checkpoint root. SAM runs in a temporary ai-spine GPU process.
SAM_MODEL_DIR=F:/Study/SAM3
SAM_CODE_DIR=F:/Study/SAM3/sam3
-137
View File
@@ -1,137 +0,0 @@
# 项目修改约定
本文件适用于 `AI_spine/` 及其所有子目录。每次修改代码、文档、依赖或运行测试前,都必须阅读并遵守。
## Python 环境
- 本项目固定使用名为 `ai-spine` 的 Conda 环境。
- 不创建、不使用 `.venv`、`venv` 或其他 Python 虚拟环境;不把依赖安装到 Conda base 或系统 Python。
- 交互式终端先执行 `conda activate ai-spine`;自动化命令使用 `conda run -n ai-spine ...`。
- Python 依赖统一记录在 `requirements.txt`。安装命令使用 `conda run -n ai-spine python -m pip install -r requirements.txt`,避免裸 `pip` 指向错误环境。
- Node 管线调用 Python 时也必须使用此 Conda 环境。`--python` 只能指向 `ai-spine` 环境中的解释器。
- 不在代码中硬编码本机 Conda 安装路径,不自动创建环境或自动安装依赖。缺少环境或依赖时给出明确的安装/激活命令。
## 修改与验证
- 修改环境选择、依赖或启动方式时,同步更新 README、相关文档和 CLI 帮助,保持一致。
- 测试在 `ai-spine` 环境下执行:`conda run -n ai-spine npm test`。不要为了绕过依赖问题临时新建环境。
- 验证结果必须如实记录;缺少依赖导致未运行的检查,不得声称通过。
- 保留用户已有修改和实验产物,不覆盖旧实验输出。
## 跨会话上下文与阅读顺序
本节于 2026-09-21 根据当前代码与已检查产物整理。它是后续对话的上下文入口;后续修改应同步更新对应状态,实际源码、配置和具体实验 manifest 优先于历史总结。不要把旧实验统计套到新输出上。
1. 先阅读本文件和 `git status --short`,保留用户已有修改。
2. 阅读 [README](README.md) 了解项目、环境、配置与命令。
3. 按任务阅读 [第一阶段](docs/1-understanding-stage.md)、[第二阶段](docs/2-segmentation-stage.md)、[第三阶段](docs/3-completion-stage.md),再核对相关实现与实际产物。
4. 用户说“继续”时,先核对工作区和最近任务进展,不能只凭历史总结重复请求付费模型或覆盖实验。
## 用户目标与固定偏好
- 目标是将单张 2D 角色图拆成可以用于 Spine 的部件,保留画风、姿态、左右、不对称性及原有像素关系,补足遮挡和关节连接。
- 第二阶段使用本地 SAM 代码和权重,通过临时 Conda 子进程在本机 GPU 推理,完成后退出;不部署常驻独立服务。
- 需要可直接执行的 PowerShell 命令;说明当前已实现行为、实测结果与未来计划,不能把几何通过写成资产合格。
- 不确定的部件语义、遮挡和候选点需要复核,不得用前景检查、覆盖率或模型分数代替语义准确性。
- 输出使用独立目录,保留原候选和诊断;已有候选优先离线复检,避免无意义重复生成。
## 文档维护约定
- 各文档文件名保持不变。`docs/` 当前保留三份阶段文档;`docs/model-stages.md` 在本次文档整理前已处于用户删除状态,不自行恢复,相关说明归入 README 和阶段文档。
- 三份阶段文档有且仅有以下八个二级章节,细节必须归入对应章节:
1. 阶段简介
2. 输入、输出与执行链路
3. 阶段执行命令示例及参数说明
4. 输入格式要求
5. 执行链路中重要部分代码文件及原理说明(包括重点说明)
6. 各个代码文件职责索引
7. 产物详细说明
8. 当前阶段可改进点说明
- 外层 README 负责整体介绍、Conda 准备、配置及每阶段命令;本文件负责约定和后续会话必要上下文。已存在的环境与修改约定必须保留。
- 改进项明确写为待实现/待验证;实现后更新状态、命令、产物说明及本文件。PowerShell 反引号后不能有空格;示例变量先定义,示例输出目录相互独立。
## 工作区、环境与配置
- 当前实际项目目录:`F:\Study\taonier\AI_spine`。用户历史命令也出现过 `F:\Study\taonier\AI\_spine`,不要盲目复制旧路径,先确认 cwd。
- 当前 SAM 根目录为 `F:\Study\SAM3`;源码仓库 `F:\Study\SAM3\sam3`;权重 `F:\Study\SAM3\sam3.1checkpoint\sam3.1_multiplex.pt`。早期 `F:\Study\sam3.1` 已不是当前约定路径。
- 已验证组合:Python 3.11、torch 2.7.1+cu126、torchvision 0.22.1、NumPy 1.26.4、OpenCV 4.11.0.86;RTX 3060 Laptop 6 GB。机器环境可能变化,执行前按需探测。
- Node.js >=20,无额外 npm 运行时依赖。自动化使用 Conda ai-spine 执行 Node/Python;运行 SAM 可用 `conda run --no-capture-output -n ai-spine node ...`。
- 默认配置是 `configs/astra-qwen.json`;示例推荐明确传 config,避免环境变量导致选错提供方。
- `configs/deepseek-qwen.json`:deepseek-flash / Chat Completions / 非流式;补全 qwen-image-3.0。
- `configs/astra-qwen.json`:gpt-6-astra / Responses / 流式;补全 qwen-image-3.0。
- 上述两份读取 VECTOR_ENGINE_BASE_URL / VECTOR_ENGINE_API_KEY,模型变量为 SPINE_UNDERSTANDING_MODEL / SPINE_COMPLETION_MODEL。
- `configs/tiantoken.json`:独立 TIANTOKEN_BASE_URL / TIANTOKEN_API_KEY,地址 https://api.tiantoken.com;理解 gpt-6-astra,补全 gpt-image-2;模型变量为 TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL。
- CLI 模型 > stage.model > 对应模型环境变量 > default_model;shell 同名变量优先于 .env。模型 ID 原样发送,不加隐式别名。
- 用户提供的密钥仅留在本地忽略的 `.env`;禁止在文档、命令日志、聊天或提交中暴露,不要读取/打印完整 .env 来诊断。
- 三份配置的图片编辑能力均为 multiple_images=false、mask=false。接口已预留分支不代表中转已支持;开启前需验证。Astra 理解接口与图片编辑接口不同,当前没有 Astra Responses 图片生成工具链。
## 三阶段实现与不能丢失的边界
### 第一阶段:理解与几何候选
- 入口 `tools/model-check.mjs --stage understanding`,或 `pipeline.mjs --mode annotated-parts`;后者仅转发新入口。
- `stages/model-stage.mjs` 调用视觉模型,`lib/validate-parts.mjs` 校验 v2:rig_role、bind_to、三项 completion 及 needs_completion 一致性。
- `normalize_reference.py` 保留原画布,使用 Alpha 或边界连通近白背景处理;`geometry_grounding.py` 用 Alpha/ROI 距离变换生成候选,不是再次请求 AI。
- 模型原始点仅诊断;程序候选也未获语义确认;pivot 独立保留,不吸附前景。
- 第二阶段输入必须为同一实验的 source.png、semantic-parts.json、segmentation-prompts.json。
- `--parts-file` 可离线迁移 v1/检查 v2 并生成新候选;旧 annotated.svg 不会自动更新。
### 第二阶段:SAM 可见拆件
- 入口 `--stage segmentation --input-dir 理解目录`;实现为 stages/segmentation-stage.mjs、sam_runtime.py、sam_segment.py、segmentation_outputs.py。
- 环境探测与推理分别为短期子进程,禁止在线下载权重。multiplex detector 适配实际严格加载 1130 个张量,忽略 36 个未使用视频 neck 张量;不要因此改成 strict=False 或修改外部 SAM 源码。
- BF16 仅用于推理 autocast;不要整体转换模型而破坏复数缓冲区/FP32 路径。
- 使用文本和文本+粗框两组候选;当前图像概念接口未接收点提示,Alpha 种子仅参与 mask 排序。
- 与原图 Alpha 取交集后提取原像素;mask 为原画布二值图,parts 为紧裁剪 RGBA。重叠保留、覆盖率仅为像素覆盖,semantic_verified=false。
- annotated.png 为原图纹理上半透明颜色叠加,parts-sheet.png 为白底拆件;纯色遮罩角色图与可选大模型去毛边尚未实现。
### 第三阶段:受限部件补全
- 入口 `--stage completion --input-dir SAM目录`;实现为 stages/completion-stage.mjs、completion_geometry.py、providers/image-edits.mjs。
- 输入须含 source.png、v2 计划、segmentation.json 及所有记录的全画布 masks;单独 parts/ 不够。
- 仅处理 needs_completion=true、非隐藏、有可见 mask 且允许区域非空的部件。--part 不绕过筛选;目前不重新评估 SAM 缺损。
- 默认 1024 方形局部画布,原像素不缩放。允许区来自轮廓外 32 像素范围与遮挡 mask / pivot 圆盘交集,排除可见 mask;不是完整隐藏形状生成。
- 当前真实请求只发白底 input.png;context.png、edit-mask.png 默认不发送。开启 multiple_images 也只是发送同坐标局部 context,并非独立全角色参考。
- 图片解码已修复“空 b64_json 但有有效 URL”情况,回退下载 URL;该问题与画布尺寸不符不同。
- 在线请求现显式传 size,配置冲突失败。strict 默认拒绝尺寸不同;uniform 仅恢复同宽高比整画布尺寸,保留原候选,继续原有验收,不单独移动/缩放物体。
- 验收阈值:可见不透明像素 RGB MAE≤30、保留率≥95%;有效新增须在允许区且连接可见部件;原可见 RGBA 随后逐像素恢复。
- unfulfilled_regions 当前只记录、不硬拒绝;语义错误、接缝、关节补够与否尚无可靠验收。错误 SAM 可见像素会被强制保留。
- prepare-only 不联网;candidates-dir 可读平铺 ID.png 或匹配 manifest 的旧任务首张候选。--resume 不是失败断点续发,准备/生成/复检使用独立目录。
- parts/ 只含通过检查的补全部件,不包含全部原拆件。完整角色组装、旋转验收、新管线 Spine 导出尚未接入。
- 旧 `--stage completion --image ... --prompt-file ...` 是接口探针,不能和 input-dir 混用,不执行完整几何验收。
## 已检查实验与质量结论
以下为已有实验记录,不代表后续每次运行的结果;不必为了文档修改重跑 GPU 或付费模型。
| 目录(相对于项目) | 事实与意义 |
| --- | --- |
| output/sombrero-grounding-v2 | 旧计划离线几何检查,28 个程序候选通过前景检查;未证明语义正确 |
| output/sombrero-sam31-v3 | 较早 SAM 实验,28 个 mask,约 67.18% 覆盖率;肢体/鞋子重复、漏分,不合格 |
| output/20260918-183532/sam | 当前讨论基线:32 个计划项、10 个 hidden、22 个 mask;约 83.3% 覆盖,32719 像素未分配;前臂/手混入衣料、袖子与上衣缺损 |
| output/20260918-183532/complete2_gpt | 真实 Tiantoken gpt-image-2 请求 12 部件;旧请求未传 size,返回 1248/1254,预期 1024,全部 CANVAS_SIZE_MISMATCH |
| output/20260918-183532/complete2_gpt-size-recheck-v2 | 离线 uniform 复检,未重生成:8 个几何通过;head、pants_thigh_l、pants_thigh_r、shoe_l 4 个拒绝;通过项仍有误分和未完成区域 |
| output/20260918-183532/complete-recovered-head-check | 更早 Qwen 头部候选恢复检查,不能与 GPT 批次混为一谈 |
- 当前 22→12 的原因:sombrero、sleeve_l、sleeve_r、eye_l、eye_r、eyebrow_l、eyebrow_r、nose、mouth、mustache 共 10 个被第一阶段标记 NO_COMPLETION_NEEDED;不能解释为它们的实际 SAM mask 已完整。
- 2026-09-20 尺寸修复后曾通过 22 项 Node 测试(含 Python 补全回归)。这是历史验证;新改动按范围重新验证,不自动沿用“全部通过”。显式 size 构造已测试,修复后中转是否遵守尚无新实测。
- 总体判断:上游部件归属、模型缺少原图参考与遮罩、区域规则粗糙、验收不足共同导致低质量。不能仅换模型、强制处理全部 mask、放宽阈值或解决尺寸错误就宣称修好。
## 当前改进方向与后续顺序
本轮用户要求重整文档,以下功能是待实现需求,不因写入文档就成为已完成功能:
1. 第二阶段输出仅色彩遮罩组成的完整角色图,保留当前原图叠色预览,并明确未分配/重叠区域。
2. 第二阶段每个 SAM 部件可选经过一次大模型去毛边,保留原始拆件、原图参考、坐标、变更区域与前后对照。
3. 第三阶段将角色原图或部件相关原图局部作为参考交给补全模型,验证中转多图/遮罩能力及实际发送内容。
4. 第三阶段输出使用补全部件组装的完整角色,复用无需补全的原部件,明确失败/回退状态。
5. 技术优先级:先修正 SAM 部件归属与输入门槛,再按实际 mask 重建补全计划;用头、前臂、上衣三类小样验证参考图和遮罩,然后增加回贴与关节旋转验收,最后批量与 Spine 导出。
详细改进说明保留在各阶段文档第 8 节;若后续用户改变优先级,应更新此处。
## 旧管线与范围识别
- `pipeline.mjs` 未指定 --mode annotated-parts 时走旧“生成拆件图→CV 定位→Spine”流程,不是新三阶段自动串联。
- `segment.py` 为旧拆件图分割;`compose_preview.py` 为旧布局组装;`run_case_matrix.py` 为旧定位对照工具。不要因为 IDE 当前打开这些文件就误改新阶段入口。
- 相邻 `spine-animation-ai` 为外部只读依赖,保留其 PolyForm Noncommercial 许可证约束;新阶段尚未复用为完成的导出流程。
- 不要把文档改写任务扩大成付费图片生成、SAM 推理、依赖安装或功能实现;需要实现时遵从用户后续任务范围。
+80 -217
View File
@@ -1,230 +1,93 @@
# AI Spine 角色拆件与动画管线
# Genarrative / Genielabs Spine Animation Pipeline
本项目探索将单张 2D 角色图转换为可用于 Spine 动画的部件与骨骼资源。当前主线分为三个阶段:视觉 LLM 理解结构、本地 SAM 分割可见部件、图片编辑模型补全隐藏区域和关节连接。
这是独立目录,不修改 Genarrative。它只参考 Genarrative 的 VectorEngine 图片生成协议:后端环境变量提供 `VECTOR_ENGINE_BASE_URL` 与 `VECTOR_ENGINE_API_KEY`,使用 OpenAI-compatible `POST /v1/images/edits` multipart 请求。
| 阶段 | 输入 → 输出 | 当前状态 |
| --- | --- | --- |
| 1. 结构理解 | PNG → v2 语义计划、粗框、pivot、程序候选点 | 可在线理解或离线复检;语义待复核 |
| 2. SAM 分割 | 第一阶段目录 → mask、原像素透明部件、标注图与拆解图 | 本机 GPU 实测可运行;存在误分与漏分 |
| 3. 部件补全 | SAM 目录 → 局部编辑候选、几何 QA、通过检查的补全部件 | 已真实调用图片模型;结果质量尚不合格 |
当前上游没有 `gpt-image-2.5`,脚本会把它映射为 `gpt-image-2`,并在 manifest 中保留 `requestedModel`。
第三阶段尚未输出使用补全部件组装的完整角色,新管线 Spine 导出也未完成。执行成功、mask 覆盖率高、几何检查通过,均不等于动画资产可用。
拆件提示词是角色无关的通用模板:以参考图为唯一 ground truth,保留原始注册位置、左右方向、遮挡关系和非对称结构,只在自然关节/遮挡边界切分,禁止镜像、拉直、补全隐藏部件和生成重复备件。图片接口对 429/5xx 会自动重试 3 次;使用新输出目录即可重新生成,旧目录可用 `--resume` 复用已生成的拆件图。
仓库保留旧版“生成白底拆件图 → OpenCV 拆分 → Genielabs 定位 → Spine 导出”作为对照;它与当前三阶段主线并存,产物不能直接混用。
正式运行默认尝试 6 个通用拆件提示词变体,分别强调注册位置、关节边界、遮挡审计、保守拆分和像素保真;脚本按微碎片、巨型粘连、前景覆盖率和极端部件数量选择候选,不假定角色必须有固定部件数。可用 `--attempts 1` 只运行一个候选。
## 文档入口
## 运行
- [AGENTS.md](AGENTS.md):修改约定、跨会话上下文、当前基线及后续优先级。
- [第一阶段](docs/1-understanding-stage.md):理解、数据契约、Alpha 候选与粗标注。
- [第二阶段](docs/2-segmentation-stage.md):本地 SAM、mask 选择、拆件与质量边界。
- [第三阶段](docs/3-completion-stage.md):筛选、局部画布、图片请求、几何验收及补全改进。
阶段文档统一采用八节:简介;输入输出与链路;命令参数;输入格式;重要代码原理;职责索引;产物;可改进点。配置与实验说明已归入上述文档。
## Conda 环境准备
所有 Python 操作固定使用 **ai-spine**,不使用 `.venv`、其他虚拟环境、Conda base 或系统 Python。Node.js 要求 20 或以上;项目没有额外 npm 运行时依赖。
在 PowerShell 中进入实际项目根目录。本工作区路径如下;若本机使用其他目录,仅替换这一行:
```powershell
Set-Location F:\Study\taonier\AI_spine
node --version
conda env list
```
仅在没有 ai-spine 环境的新机器上,手动创建一次;当前机器已有环境则跳过:
```powershell
conda create -n ai-spine python=3.11 -y
```
补齐项目依赖。以下是本机已验证的 CUDA 12.6 组合;已有匹配 PyTorch 时可跳过第一条安装命令。代码不会自动安装依赖或创建环境。
```powershell
conda activate ai-spine
python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r .\requirements.txt
python -m pip install --no-deps --no-build-isolation -e F:\Study\SAM3\sam3
```
不激活终端时,明确指定目标环境:
```powershell
conda run -n ai-spine python -m pip install -r .\requirements.txt
```
本地 SAM 代码和权重由用户提供,当前位置为:
```text
F:/Study/SAM3/sam3/ 源码仓库
F:/Study/SAM3/sam3.1checkpoint/sam3.1_multiplex.pt 权重
```
检查解释器、基础依赖与 GPU:
```powershell
conda run -n ai-spine python -c "import sys, cv2, PIL, numpy, torch; print(sys.executable); print('torch', torch.__version__); print('CUDA', torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'No CUDA GPU')"
```
第二阶段要求可用 CUDA GPU。本机已验证 RTX 3060 Laptop 6 GB;默认 BF16 推理保留模型参数和缓冲区所需精度。第一、三阶段的本地几何处理不运行 SAM。Node 入口会验证 Python 属于 ai-spine;`--python` 也不能绕过此约定。
## 中转站配置
仅在本地 `.env` 不存在时复制模板,随后自行填写密钥:
```powershell
if (-not (Test-Path .\.env)) { Copy-Item .\.env.example .\.env }
```
```dotenv
# 原中转站
VECTOR_ENGINE_BASE_URL=https://router.genarrative.world/v1
VECTOR_ENGINE_API_KEY=填写你的令牌
# Tiantoken 独立配置
TIANTOKEN_BASE_URL=https://api.tiantoken.com
TIANTOKEN_API_KEY=填写你的令牌
# 本地 SAM
SAM_MODEL_DIR=F:/Study/SAM3
SAM_CODE_DIR=F:/Study/SAM3/sam3
```
不要把密钥写入文档、JSON 配置或提交到版本库;不要输出 `.env` 内容。
| 配置文件 | 第一阶段 | 第三阶段 | 模型环境变量 |
| --- | --- | --- | --- |
| [deepseek-qwen.json](configs/deepseek-qwen.json) | deepseek-flash,Chat Completions,非流式 | qwen-image-3.0,images/edits | SPINE_UNDERSTANDING_MODEL / SPINE_COMPLETION_MODEL |
| [astra-qwen.json](configs/astra-qwen.json) | gpt-6-astra,Responses,流式 | qwen-image-3.0,images/edits | 同上 |
| [tiantoken.json](configs/tiantoken.json) | gpt-6-astra,Responses,流式 | gpt-image-2,images/edits | TIANTOKEN_UNDERSTANDING_MODEL / TIANTOKEN_COMPLETION_MODEL |
CLI > 配置 stage.model > 对应模型环境变量 > default_model;shell 环境变量优先于 `.env`。不指定配置时入口默认 astra-qwen.json,建议实验始终显式指定。模型 ID 原样发送,中转能力以实际请求为准。Tiantoken 图片编辑已真实执行;第三阶段尚未适配 Astra Responses 图片生成工具。
三个配置当前均关闭多图参考与编辑 mask 能力。第三阶段默认只发送白底单部件图;本地 context.png/edit-mask.png 已生成但不默认发送。改进计划见第三阶段文档。
查询模型列表和检查配置:
```powershell
node .\tools\model-check.mjs --config .\configs\tiantoken.json --list-models
node .\tools\model-check.mjs --config .\configs\tiantoken.json --dry-run
```
前者联网,后者不请求模型;列表存在某个 ID 不代表其所有能力已验证。
## 三阶段执行命令
以下命令按顺序在同一个 PowerShell 会话执行。反引号后不能带空格。每次实验使用新的 runId;分阶段重开终端时,将路径变量设置成已有的对应目录。
```powershell
conda activate ai-spine
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
$runRoot = ".\output\$runId"
$reference = ".\output\20260918-183532\sam\source.png"
```
示例复用现有角色原图,可替换 `$reference` 为自己的 PNG。不要混用不同实验的 source、计划和 mask。
### 第一阶段:理解角色结构
此命令调用视觉模型:
```powershell
node .\tools\model-check.mjs --stage understanding `
--config .\configs\deepseek-qwen.json `
--reference $reference `
--out-dir "$runRoot\understand"
```
检查 `understand/report.html`、`semantic-parts.json`、`geometry-validation.json`。程序候选点来自 Alpha 距离变换,落在前景不代表属于正确部件。
已有计划可离线重新检查;输出独立目录,不重跑模型:
```powershell
node .\tools\model-check.mjs --stage understanding `
--reference .\output\20260918-183532\sam\source.png `
--parts-file .\output\20260918-183532\sam\semantic-parts.json `
--out-dir "$runRoot\understand-offline"
```
### 第二阶段:本地 SAM 分割
```powershell
node .\tools\model-check.mjs --stage segmentation `
--input-dir "$runRoot\understand" `
--out-dir "$runRoot\sam" `
--sam-dir F:/Study/SAM3
```
若使用前面的离线计划,input-dir 改为 understand-offline。该命令只运行本地临时 GPU 子进程;添加 `--dry-run` 可探测环境而不加载模型。检查 `sam/report.html`、`annotated.png`、`parts-sheet.png`、漏分图和 mask 归属,错误部件应先修正。
### 第三阶段:准备与补全
先离线准备任务,不请求模型:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir "$runRoot\sam" `
--out-dir "$runRoot\complete-prepare" `
--prepare-only
```
复核允许区域后,按实际存在的 ID 试一个部件;下面会调用图片模型:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir "$runRoot\sam" `
--out-dir "$runRoot\complete-head" `
--part head
```
处理所有具备可见 mask、补全需求和允许区域的部件:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir "$runRoot\sam" `
--out-dir "$runRoot\complete"
```
省略 part 不意味着强制补全全部 mask。检查 `completion-plan.json` 的跳过原因和 `completion-result.json` 的 QA;parts/ 仅包含通过检查的补全部件。所有结果仍需语义复核。
旧候选尺寸不符时,可离线导入复检,避免重新生成:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir .\output\20260918-183532\sam `
--candidates-dir .\output\20260918-183532\complete2_gpt `
--out-dir "$runRoot\complete-size-recheck" `
--candidate-size-policy uniform
```
uniform 仅恢复同宽高比的整画布尺寸,不纠正物体位移、变形或语义。在线请求现已显式传 size;参数及中转实测边界见[第三阶段文档](docs/3-completion-stage.md)。
## 旧管线与验证
旧版依赖相邻 `spine-animation-ai` 外部项目,使用时遵守其 PolyForm Noncommercial 许可证;不要修改外部依赖。旧版命令会调用图片生成并尝试定位、导出:
```powershell
node .\pipeline.mjs `
--reference $reference `
--out-dir "$runRoot\legacy" `
--genielabs-dir ..\spine-animation-ai `
--image-model qwen-image-3.0 `
--attempts 1 `
```bash
node pipeline.mjs \
--reference /absolute/path/character.png \
--out-dir ./output/character-spine \
--genielabs-dir /absolute/path/spine-animation-ai \
--env-file /absolute/path/.env \
--image-model gpt-image-2.5 \
--animations idle,walk,wave
```
`segment.py` 为旧版拆件分割,`compose_preview.py` 组装预览,`run_case_matrix.py` 对照定位方案;当前 SAM 入口为 sam_segment.py。旧产物需检查 assembled.png 和 preview-rigged.html,不能据此声称新三阶段已经完成 Spine 导出。
脚本会依次执行:
测试与命令帮助:
1. VectorEngine 图片编辑,生成白底拆件工作图;
2. OpenCV 连通域拆件,输出透明部件 PNG;
3. 默认调用 Genielabs `position_parts.py`,使用 SIFT/RANSAC、模板匹配回退和 z-order 计算把部件放回参考图;只有显式传入 `--layout-mode anchored` 才启用按画布锚点的保守回退;
4. 生成 Spine 配置并运行 `build_spine_json.py`;
5. 运行 `make_atlas.py` 和 `generate_spine_player.py`。
```powershell
conda run --no-capture-output -n ai-spine npm test
node .\tools\model-check.mjs --help
为避免“部件爆炸”,定位后默认只保留 SIFT 高置信部件和模板匹配分数不低于 `0.3` 的部件,最多保留 28 个;可用 `--min-score`、`--max-parts` 调整。多候选选择还会比较 Genielabs 定位置信度和部件框重叠,降低重复角色层被选中的概率。若模型仍输出多套角色,先删除输出目录重新生成,不要直接把低质量拆件导入 Spine。
`matched` 依赖拆件图与参考图保持相同画风和可辨识纹理;低纹理部件会进入模板匹配回退,并进行白底归一化以避免透明像素产生 `NaN/inf`。`anchored` 只按拆件图的行布局做通用兜底,不应作为正式定位结果。
Python 环境探测顺序为显式 `--python`、Anaconda、Homebrew Python、系统 Python;要求导入 `cv2`、`PIL`、`numpy`。如果全部不可用,脚本会在本目录创建 `.venv` 并安装 `requirements.txt`。
脚本默认只读取本目录 `.env` 和当前 shell 环境,不会自动读取 Genarrative 的 secrets 文件。可以通过 `--env-file` 显式指定一个只在受控后端使用的环境文件。
输出目录包含 `generated-parts-sheet.png`、`parts/`、`layout.json`、`assembled.png`、`skeleton.json`、`atlas/`、`preview.html` 和 `manifest.json`。`assembled.png` 是按同一 layout 坐标生成的静态 QA 合成图;自动结果是 region attachments,mesh、weights、IK、FFD 和骨骼语义仍需在 Spine 4.2 中校正。
`preview.html` 现在显示参考图的真实姿态,适合确认形体是否正常;自动拆件结果放在 `preview-rigged.html`。这是因为参考图可能是左右不对称动作,不能用对称锚点强行还原,否则会出现手臂漂浮或错接。
## 离线多 case 回归
`run_case_matrix.py` 可以复用已经生成的多个拆件候选,不调用图片 API,只测试 Genielabs 定位和合成:
```bash
python run_case_matrix.py \
--reference output/demo-character-final/source.png \
--genielabs-dir /absolute/path/spine-animation-ai \
--out-dir output/cases-matrix \
--candidate final output/demo-character-final/parts \
--candidate attempt1 output/demo-character-multi/candidates/attempt-1/parts \
--candidate attempt2 output/demo-character-multi/candidates/attempt-2/parts
```
测试使用本地 mock 或注入响应,不请求真实付费模型。自动化运行任一阶段可用 `conda run --no-capture-output -n ai-spine node ...`。保留旧实验;resume 只复用完整匹配产物,失败任务另建输出目录。
每个 case 会输出 `layout.json`、`assembled.png` 和 `case-summary.json`,总报告为 `case-report.json`。当前回归中,`final`/`matched` 保留 9 个有效部件,前景 MAE 约 17.68;`attempt1` 只有 7 个部件且出现手臂误拆,MAE 30.28;`attempt2` 有 15 个部件但包含重复/漂浮部件,MAE 27.59。默认候选选择和重叠过滤会优先保留 `final`/`matched` 这类结果。
Genielabs 以外部只读目录引用,需遵守其 PolyForm Noncommercial 许可证。生产集成时应把输出上传对象存储,只向前端返回 object key 或 asset object id。
## 外部参考角色
已下载 5 个 Kenney CC0 参考素材到 `references/characters/`,可直接作为下一轮图片拆件的 `--reference` 输入;来源、许可证和 SHA-256 见 [references/index.json](references/index.json),原始包及许可证文件见 `references/external/`。缩略图对比见 [references/characters/contact-sheet.jpg](references/characters/contact-sheet.jpg)。
另外新增了高分辨率单角色回归素材,位于 `references/highres-characters/`。其中 `tiddybub-toon/crops-768/` 包含 6 个 768×1024、透明背景、CC0 的 Kenney Toon Characters 单角色图,适合测试头发、手臂、手、腿、鞋和机器人关节等不同结构。
## GPT Image 2 prompt 优化回归
已对原始 sombrero/cowboy 参考图执行 6 个通用 prompt 变体(每个变体一次请求),并对 6 个高分辨率角色执行最多 3 个候选:female-adventurer、male-adventurer、female-person、male-person、robot、zombie。所有 prompt 都不包含角色名称、左右手硬编码或单个案例的外观描述;候选只由参考图的像素、姿态、遮挡和非对称关系决定。
当前离线复评结果保存在被忽略的 `output/prompt-optimization/`:
| 参考角色 | 选中候选 | 拆件数 | 定位方式 | 备注 |
| --- | --- | ---: | --- | --- |
| cowboy | attempt-4 | 8 | 8 个 SIFT | 当前最佳;平均定位 0.793,重叠惩罚 0 |
| female-adventurer | attempt-3 | 16 | 2 个 SIFT + 模板回退 | 形体基本完整,仍需检查衣服和手套层 |
| male-adventurer | attempt-2 | 14 | 1 个 SIFT + 模板回退 | 可用作回归输入,细部依赖模板定位 |
| male-person | attempt-2 | 17 | 4 个 SIFT + 模板回退 | 需要检查头发、脸和躯干的层级 |
| robot | attempt-2 | 17 | 5 个 SIFT + 模板回退 | 机械关节拆分较清楚,部分小件会漂浮 |
| zombie | attempt-2 | 11 | 模板回退 | 仍有明显错层,不能视为通过 |
| female-person | attempt-1 | 21 | 3 个 SIFT + 模板回退 | 检测到完整人物副本风险,不能直接采用 |
综合对照图为 `output/prompt-optimization/reselection-contact-sheet.jpg`,机器报告为 `output/prompt-optimization/reselection-report.json`。这些评分用于缩小人工复核范围,不代表自动通过,也不会把完整人物副本误选为拆件。
如需在不调用图片 API 的情况下重新评估已有候选,可运行:
```bash
python reselect_candidates.py \
--out-root output/prompt-optimization \
--genielabs-dir /absolute/path/spine-animation-ai
```
本轮新的 GPT Image 请求在 female-person、robot 和 zombie 的后续尝试中收到 VectorEngine `403 user quota is not enough`,因此没有继续重试或伪造新结果。补充额度后,可直接用同一组 `prompts/v2-tested.json` 和 `--attempts 6` 继续 A/B 测试。
-269
View File
@@ -1,269 +0,0 @@
"""Prepare bounded completion canvases and merge generated pixels conservatively."""
import argparse
import html
import json
from pathlib import Path
import cv2
import numpy as np
from PIL import Image
from normalize_reference import prepare_rgba
def write_json(file, value):
Path(file).write_text(json.dumps(value, ensure_ascii=False, indent=2), encoding='utf-8')
def read_json(file):
return json.loads(Path(file).read_text(encoding='utf-8-sig'))
def localize(array, origin, size):
"""Copy source coordinates to a padded local canvas, without scaling."""
out = np.zeros((size, size) + array.shape[2:], dtype=array.dtype)
ox, oy = origin
h, w = array.shape[:2]
x0, y0, x1, y1 = max(0, ox), max(0, oy), min(w, ox + size), min(h, oy + size)
if x1 > x0 and y1 > y0:
out[y0-oy:y1-oy, x0-ox:x1-ox] = array[y0:y1, x0:x1]
return out
def allowed_regions(part, plan, masks, visible, radius):
h, w = visible.shape
distance = cv2.distanceTransform((~visible).astype(np.uint8), cv2.DIST_L2, 5)
band = (distance <= radius) & ~visible
regions, warnings = {}, []
flags = part['completion']
if flags['occlusion_fill']:
occluders = np.zeros_like(visible)
for name in part['occluded_by']:
if name in masks:
occluders |= masks[name]
regions['occlusion_fill'] = band & occluders
if not regions['occlusion_fill'].any():
warnings.append('NO_OCCLUSION_REGION')
anchors = {}
if flags['joint_extension_parent']:
anchors['joint_extension_parent'] = [part['pivot_norm']]
if flags['joint_extension_child']:
anchors['joint_extension_child'] = [p['pivot_norm'] for p in plan['parts']
if p['parent'] == part['id'] and p['rig_role'] == 'bone_part']
for kind, points in anchors.items():
disks = np.zeros((h, w), np.uint8)
for x, y in points:
cv2.circle(disks, (min(w-1, round(x*w)), min(h-1, round(y*h))), radius, 1, -1)
regions[kind] = band & (disks > 0)
if not regions[kind].any():
warnings.append(f'NO_{kind.upper()}_REGION')
allowed = np.zeros_like(visible)
for region in regions.values():
allowed |= region
return allowed, regions, warnings
def prepare(directory, selected_ids, size=1024, radius=32):
out = Path(directory)
plan, segmentation = read_json(out/'semantic-parts.json'), read_json(out/'segmentation.json')
source = np.array(Image.open(out/'source.png').convert('RGBA'))
h, w = source.shape[:2]
known = {p['id']: p for p in plan['parts']}
if set(selected_ids) - set(known):
raise ValueError('Unknown requested part ID')
records = {p['id']: p for p in segmentation['parts']}
if len(records) != len(segmentation['parts']) or set(records) - set(known):
raise ValueError('Segmentation IDs do not match semantic plan')
masks = {}
for name in records:
mask = np.array(Image.open(out/'masks'/f'{name}.png').convert('L'))
if mask.shape != (h, w) or not np.isin(mask, [0, 255]).all():
raise ValueError(f'{name}: mask must be binary and match the source canvas')
mask = mask > 0
if not mask.any() or (mask & (source[:, :, 3] == 0)).any():
raise ValueError(f'{name}: empty mask or mask outside original alpha')
masks[name] = mask
tasks, skipped = [], []
for part in plan['parts']:
name = part['id']
if selected_ids and name not in selected_ids:
continue
if not part['needs_completion']:
skipped.append({'id': name, 'reason': 'NO_COMPLETION_NEEDED'})
continue
if name not in masks or part['visibility'] == 'hidden':
skipped.append({'id': name, 'reason': 'NO_VISIBLE_SAM_MASK'})
continue
visible = masks[name]
allowed, regions, warnings = allowed_regions(part, plan, masks, visible, radius)
if not allowed.any():
skipped.append({'id': name, 'reason': 'NO_ALLOWED_EXTENSION', 'warnings': warnings})
continue
ys, xs = np.nonzero(visible | allowed)
bw, bh = int(xs.max()-xs.min()+1), int(ys.max()-ys.min()+1)
if max(bw, bh) + 32 > size:
raise ValueError(f'{name}: completion canvas too small; increase --completion-size (no resizing is performed)')
origin = [int(xs.min()) - (size-bw)//2, int(ys.min()) - (size-bh)//2]
folder = out/'tasks'/name
folder.mkdir(parents=True, exist_ok=True)
cut = source.copy()
cut[~visible] = 0
original = localize(cut, origin, size)
editable = localize(allowed, origin, size)
Image.fromarray(original).save(folder/'visible.png')
white = Image.new('RGBA', (size, size), 'white')
Image.alpha_composite(white, Image.fromarray(original)).convert('RGB').save(folder/'input.png')
context = Image.fromarray(localize(source, origin, size))
Image.alpha_composite(white, context).convert('RGB').save(folder/'context.png')
edit_mask = np.full((size, size, 4), 255, np.uint8)
edit_mask[editable, 3] = 0
Image.fromarray(edit_mask).save(folder/'edit-mask.png')
Image.fromarray(editable.astype(np.uint8)*255).save(folder/'allowed-mask.png')
local_regions, region_bounds = {}, {}
for kind, region in regions.items():
local = localize(region, origin, size)
Image.fromarray(local.astype(np.uint8)*255).save(folder/f'{kind}.png')
local_regions[kind] = int(local.sum())
ry, rx = np.nonzero(local)
region_bounds[kind] = [int(rx.min()), int(ry.min()), int(rx.max())+1, int(ry.max())+1] if len(rx) else None
task = {'id': name, 'semantic': part['semantic'], 'completion': part['completion'],
'origin_xy': origin, 'canvas_size': [size, size], 'source_size': [w, h],
'allowed_pixels': int(editable.sum()), 'regions': local_regions,
'region_bounds_xyxy': region_bounds, 'occluded_by': part['occluded_by'],
'warnings': records[name].get('warnings', []) + warnings,
'semantic_verified': False, 'status': 'prepared'}
write_json(folder/'task.json', task)
tasks.append(task)
result = {'schema_version': 1, 'tasks': tasks, 'skipped': skipped,
'quality_status': 'needs_review', 'canvas_size': size, 'extension_radius': radius}
write_json(out/'completion-plan.json', result)
return result
def merge_candidate(folder, candidate_file, output_file, size_policy='strict'):
if size_policy not in ['strict', 'uniform']:
raise ValueError('Unknown candidate size policy')
folder = Path(folder)
task = read_json(folder/'task.json')
original = np.array(Image.open(folder/'visible.png').convert('RGBA'))
allowed = np.array(Image.open(folder/'allowed-mask.png').convert('L')) > 0
candidate = Image.open(candidate_file).convert('RGBA')
qa = {'accepted': False, 'semantic_verified': False, 'reasons': [],
'expected_size': task['canvas_size'], 'returned_size': list(candidate.size),
'size_policy': size_policy, 'canvas_resampled': False}
if list(candidate.size) != task['canvas_size']:
if size_policy == 'strict':
qa['reasons'].append('CANVAS_SIZE_MISMATCH')
return qa
tw, th = task['canvas_size']
cw, ch = candidate.size
if cw * th != ch * tw:
qa['reasons'].append('CANVAS_ASPECT_RATIO_MISMATCH')
return qa
# Only reverse uniform whole-canvas scaling. Do not translate, stretch,
# crop, or register the object to make a changed pose appear correct.
candidate = candidate.resize((tw, th), Image.Resampling.LANCZOS)
qa.update(canvas_resampled=True, canvas_scale=tw/cw,
resampling='LANCZOS', warnings=['CANDIDATE_CANVAS_RESAMPLED'])
candidate.save(Path(output_file).with_name(Path(output_file).stem + '-normalized.png'))
try:
candidate, preprocessing = prepare_rgba(candidate)
except ValueError as error:
qa['reasons'].append('CANNOT_EXTRACT_FOREGROUND')
qa['detail'] = str(error)
return qa
generated = np.array(candidate)
visible = original[:, :, 3] > 0
opaque = original[:, :, 3] >= 240
if not opaque.any():
qa['reasons'].append('NO_OPAQUE_ALIGNMENT_ANCHOR')
return qa
# Detect gross movement/redesign before restoring original pixels, otherwise
# restoration could hide a shifted generation attached at the wrong place.
white = Image.new('RGBA', candidate.size, 'white')
generated_rgb = np.array(Image.alpha_composite(white, candidate))[:, :, :3].astype(float)
mae = float(np.abs(generated_rgb[opaque] - original[:, :, :3][opaque]).mean())
retained = float((generated[:, :, 3][opaque] >= 240).mean())
proposed = (generated[:, :, 3] > 0) & ~visible
added = proposed & allowed
_, labels = cv2.connectedComponents((visible | added).astype(np.uint8), connectivity=8)
touching = np.unique(labels[visible])
connected = added & np.isin(labels, touching[touching != 0])
region_additions = {kind: int((connected & (np.array(Image.open(folder/f'{kind}.png')) > 0)).sum())
for kind in task['regions']}
missing_regions = [kind for kind, pixels in task['regions'].items() if not pixels or not region_additions[kind]]
qa.update(alignment_mae=mae, visible_retention=retained,
new_pixels=int(connected.sum()), discarded_outside_allowed=int((proposed & ~allowed).sum()),
disconnected_pixels=int((added & ~connected).sum()), region_additions=region_additions,
unfulfilled_regions=missing_regions, preprocessing=preprocessing)
if mae > 30 or retained < .95:
qa['reasons'].append('VISIBLE_ALIGNMENT_MISMATCH')
if not connected.any():
qa['reasons'].append('NO_CONNECTED_NEW_PIXELS')
if qa['reasons']:
return qa
merged = np.zeros_like(original)
merged[connected] = generated[connected]
merged[visible] = original[visible]
qa['visible_pixels_exact'] = bool(np.array_equal(merged[visible], original[visible]))
qa['accepted'] = True
qa['quality_status'] = 'needs_review'
Image.fromarray(merged).save(output_file)
return qa
def finalize(directory):
out = Path(directory)
result = read_json(out/'completion-result.json')
rows = []
for task in result['parts']:
name = task['id']
qa = task.get('qa', {})
if task.get('selected_file'):
im = Image.open(out/task['selected_file']).convert('RGBA')
bbox = im.getbbox()
crop = im.crop(bbox)
(out/'parts').mkdir(exist_ok=True)
crop.save(out/'parts'/f'{name}.png')
meta = read_json(out/'tasks'/name/'task.json')
ox, oy = meta['origin_xy']
task.update(image=f'parts/{name}.png', crop_xyxy=[bbox[0]+ox, bbox[1]+oy, bbox[2]+ox, bbox[3]+oy])
view = f'<img src="{task["image"]}">' if task.get('image') else ''
for index, item in enumerate(task.get('candidates', []), 1):
view += f'<p><a href="{html.escape(item["file"], quote=True)}">原候选 {index}</a> · <a href="tasks/{name}/qa-{index}.json">QA</a>'
if item['qa'].get('canvas_resampled'):
view += f' · <a href="tasks/{name}/merged-{index}-normalized.png">缩放后候选</a>'
view += '</p>'
warnings = html.escape(', '.join(task.get('upstream_warnings', [])))
rows.append(f'<tr><td>{html.escape(name)}<p>{warnings}</p><a href="tasks/{name}/task.json">任务参数</a> · <a href="tasks/{name}/prompt.txt">提示词</a></td><td>{html.escape(task["status"])}</td><td><a href="tasks/{name}/visible.png"><img src="tasks/{name}/visible.png"></a></td><td><a href="tasks/{name}/allowed-mask.png"><img src="tasks/{name}/allowed-mask.png"></a><p><a href="tasks/{name}/context.png">原图局部参考</a></p></td><td>{view}</td><td><pre>{html.escape(json.dumps(qa, ensure_ascii=False, indent=2))}</pre></td></tr>')
write_json(out/'completion-result.json', result)
(out/'report.html').write_text('''<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>部件补全</title>
<style>body{font:16px system-ui;margin:24px}td,th{border:1px solid #ccc;padding:8px}img{max-width:260px;max-height:300px;background:#eee}pre{white-space:pre-wrap}table{border-collapse:collapse}</style>
<h1>第三阶段:部件补全</h1><p>新增区域受允许掩码限制,原图可见像素逐像素恢复。几何通过仅表示候选可供复核,不代表遮挡和关节纹理已正确补全。上游 SAM 错分仍需修正。</p>
<table><tr><th>部件 / 上游告警</th><th>状态</th><th>原可见区域</th><th>允许新增区域(白色)</th><th>补全候选</th><th>几何检查</th></tr>'''+''.join(rows)+'</table><h2>跳过</h2><pre>'+html.escape(json.dumps(result['skipped'], ensure_ascii=False, indent=2))+'</pre></html>', encoding='utf-8')
def main():
parser = argparse.ArgumentParser()
parser.add_argument('action', choices=['prepare', 'merge', 'finalize'])
parser.add_argument('--out', required=True)
parser.add_argument('--part', action='append', default=[])
parser.add_argument('--size', type=int, default=1024)
parser.add_argument('--radius', type=int, default=32)
parser.add_argument('--candidate')
parser.add_argument('--merged')
parser.add_argument('--qa')
parser.add_argument('--candidate-size-policy', choices=['strict', 'uniform'], default='strict')
args = parser.parse_args()
if args.action == 'prepare':
if not 128 <= args.size <= 4096 or not 1 <= args.radius <= 256:
parser.error('size must be 128..4096 and radius 1..256')
prepare(args.out, args.part, args.size, args.radius)
elif args.action == 'merge':
write_json(args.qa, merge_candidate(args.out, args.candidate, args.merged, args.candidate_size_policy))
else:
finalize(args.out)
if __name__ == '__main__':
main()
-45
View File
@@ -1,45 +0,0 @@
{
"schema_version": 1,
"providers": {
"relay-vision": {
"protocol": "responses",
"base_url_env": "VECTOR_ENGINE_BASE_URL",
"api_key_env": "VECTOR_ENGINE_API_KEY",
"timeout_ms": 180000,
"max_retries": 2
},
"relay-image-edit": {
"protocol": "images-edits",
"base_url_env": "VECTOR_ENGINE_BASE_URL",
"api_key_env": "VECTOR_ENGINE_API_KEY",
"timeout_ms": 1200000,
"max_retries": 2,
"capabilities": {"multiple_images": false, "mask": false},
"image_field": "image",
"mask_editable": "transparent"
}
},
"stages": {
"understanding": {
"provider": "relay-vision",
"model_env": "SPINE_UNDERSTANDING_MODEL",
"default_model": "gpt-6-astra",
"prompt": "../prompts/understand-v1.txt",
"output_mode": "text-json",
"stream": true,
"params": {"store": false}
},
"completion": {
"provider": "relay-image-edit",
"model_env": "SPINE_COMPLETION_MODEL",
"default_model": "qwen-image-3.0",
"prompt": "../prompts/complete-v1.txt",
"params": {"n": 1}
},
"segmentation": {
"provider": "sam-local",
"model_dir_env": "SAM_MODEL_DIR",
"default_model_dir": "F:/Study/SAM3"
}
}
}
-45
View File
@@ -1,45 +0,0 @@
{
"schema_version": 1,
"providers": {
"relay-vision": {
"protocol": "chat-completions",
"base_url_env": "VECTOR_ENGINE_BASE_URL",
"api_key_env": "VECTOR_ENGINE_API_KEY",
"timeout_ms": 180000,
"max_retries": 2
},
"relay-image-edit": {
"protocol": "images-edits",
"base_url_env": "VECTOR_ENGINE_BASE_URL",
"api_key_env": "VECTOR_ENGINE_API_KEY",
"timeout_ms": 1200000,
"max_retries": 2,
"capabilities": {"multiple_images": false, "mask": false},
"image_field": "image",
"mask_editable": "transparent"
}
},
"stages": {
"understanding": {
"provider": "relay-vision",
"model_env": "SPINE_UNDERSTANDING_MODEL",
"default_model": "deepseek-flash",
"prompt": "../prompts/understand-v1.txt",
"output_mode": "text-json",
"stream": false,
"params": {}
},
"completion": {
"provider": "relay-image-edit",
"model_env": "SPINE_COMPLETION_MODEL",
"default_model": "qwen-image-3.0",
"prompt": "../prompts/complete-v1.txt",
"params": {"n": 1}
},
"segmentation": {
"provider": "sam-local",
"model_dir_env": "SAM_MODEL_DIR",
"default_model_dir": "F:/Study/SAM3"
}
}
}
-45
View File
@@ -1,45 +0,0 @@
{
"schema_version": 1,
"providers": {
"tiantoken-vision": {
"protocol": "responses",
"base_url_env": "TIANTOKEN_BASE_URL",
"api_key_env": "TIANTOKEN_API_KEY",
"timeout_ms": 180000,
"max_retries": 2
},
"tiantoken-image-edit": {
"protocol": "images-edits",
"base_url_env": "TIANTOKEN_BASE_URL",
"api_key_env": "TIANTOKEN_API_KEY",
"timeout_ms": 1200000,
"max_retries": 2,
"capabilities": {"multiple_images": false, "mask": false},
"image_field": "image",
"mask_editable": "transparent"
}
},
"stages": {
"understanding": {
"provider": "tiantoken-vision",
"model_env": "TIANTOKEN_UNDERSTANDING_MODEL",
"default_model": "gpt-6-astra",
"prompt": "../prompts/understand-v1.txt",
"output_mode": "text-json",
"stream": true,
"params": {"store": false}
},
"completion": {
"provider": "tiantoken-image-edit",
"model_env": "TIANTOKEN_COMPLETION_MODEL",
"default_model": "gpt-image-2",
"prompt": "../prompts/complete-v1.txt",
"params": {"n": 1}
},
"segmentation": {
"provider": "sam-local",
"model_dir_env": "SAM_MODEL_DIR",
"default_model_dir": "F:/Study/SAM3"
}
}
}
-147
View File
@@ -1,147 +0,0 @@
# 第一阶段:视觉 LLM 角色结构理解
## 1. 阶段简介
将单张角色图转换为部件语义计划:确定部件名称、骨骼与附件分类、运动层级、遮挡、可见区域粗框、关节点和补全需求。视觉模型负责结构理解;本地程序负责格式校验和前景候选点生成。
本阶段不生成精确 mask、透明拆件、隐藏纹理或 Spine 文件。结构合法不等于识别正确,结果仍为 `quality_status=needs_review`。
## 2. 输入、输出与执行链路
输入为 PNG 角色图;在线模式需要模型配置与密钥,离线模式需要已有语义计划。核心输出为语义计划和 SAM 提示。
```text
tools/model-check.mjs(或 pipeline.mjs --mode annotated-parts)
→ 读取配置、图片、提示词与 Schema,计算实验指纹
→ Python 保留 Alpha / 去除边界连通白底,保持原画布
→ 在线:视觉模型理解;离线:读取计划并按需迁移 v1
→ 提取 JSON,校验字段、引用、层级和补全一致性
→ 在 Alpha 前景与粗框交集中生成距离变换候选点
→ 保存 semantic-parts.json、segmentation-prompts.json、诊断及报告
→ 第二阶段读取 source.png + 计划 + SAM 提示
```
`--parts-file` 不请求模型,也不运行 SAM。`--stop-after segment` 可接续第二阶段,输出到理解目录的 `segmentation/`;第三阶段独立调用。
## 3. 阶段执行命令示例及参数说明
以下 PowerShell 命令在项目根目录执行,环境准备见 [README](../README.md)。反引号必须在行尾,后面不能有空格。
```powershell
conda activate ai-spine
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
node .\tools\model-check.mjs --stage understanding `
--config .\configs\deepseek-qwen.json `
--reference .\output\20260918-183532\sam\source.png `
--out-dir ".\output\$runId\understand"
```
参考图可替换成自己的 PNG。切换 Tiantoken:将配置改为 `--config .\configs\tiantoken.json`,默认理解模型为 `gpt-6-astra`,也可显式传入 `--understanding-model gpt-6-astra`。
已有计划离线生成新候选,不重跑 AI:
```powershell
node .\tools\model-check.mjs --stage understanding `
--reference .\output\20260918-183532\sam\source.png `
--parts-file .\output\20260918-183532\sam\semantic-parts.json `
--out-dir ".\output\$runId\understand-offline"
```
| 参数 | 说明 |
| --- | --- |
| `--stage understanding` | 第一阶段,也是入口默认阶段 |
| `--reference FILE` | 必需,角色 PNG |
| `--out-dir DIR` | 实际执行必需,使用独立新目录 |
| `--config FILE` | JSON 配置;默认 `configs/astra-qwen.json` |
| `--env-file FILE` | 默认项目 `.env`;shell 同名变量优先 |
| `--understanding-model ID` | 原样发送模型 ID,不转换别名 |
| `--understanding-protocol NAME` | `chat-completions` 或 `responses` |
| `--stream true/false` | 覆盖配置流式设置 |
| `--output-mode NAME` | `text-json`、`json-object`、`json-schema`;需确认中转能力 |
| `--parts-file FILE` | 离线读取有效 v2 或保守迁移 v1 |
| `--python FILE` | 仅允许 Conda ai-spine 的解释器 |
| `--stop-after understand/segment` | 理解结束或接续 SAM |
| `--dry-run` | 检查配置与图片格式,不生成产物;不代表完整 Python 几何链路已验证 |
| `--resume` | 仅复用已完成且指纹、产物哈希一致的结果,不是失败请求续发 |
| `--list-models` | 联网查询所选提供方模型列表,不能与 dry-run 同用 |
模型优先级:CLI > `stage.model` > 配置指定的模型环境变量 > `stage.default_model`。提示词路径相对于配置文件。DeepSeek 预设为 Chat Completions 非流式;两个 Astra 预设为 Responses 流式。配置不代表中转所有能力均已验证。
## 4. 输入格式要求
参考图必须是有效 PNG,建议单角色、透明背景、完整画布。透明图保留 Alpha;不透明白底只去除与边界连通、RGB 各通道均 ≥240 的区域。复杂背景需先抠图,空前景会失败;白色主体接触边界可能被误删。
离线计划采用 [v2 Schema](../schemas/semantic-parts.schema.json):顶层 `schema_version=2`、`coordinate_space=normalized_xy_top_left`、`parts` 数组和 `notes`。实际部件数量为 1~64。
| 部件字段 | 格式与含义 |
| --- | --- |
| `id` / `semantic` | 唯一小写 ASCII ID / 语义说明 |
| `parent` | 父骨骼 ID 或 null;运动关系 |
| `rig_role` | `bone_part`、`attachment`、`facial_feature` |
| `bind_to` | 附件绑定的骨骼 ID 或 null(root);骨骼自身为 null |
| `visibility` | `visible`、`partial`、`hidden` |
| `bbox_norm_xyxy` | `[xmin,ymin,xmax,ymax]` 可见粗框;隐藏为 null |
| `pivot_norm` | `[x,y]` 关节或连接点,可处于遮挡区域 |
| `positive_points_norm` / `negative_points_norm` | 旧模型点兼容字段,新提示词要求留空 |
| `occluded_by` | 遮挡者 ID 数组,与运动关系分开 |
| `completion` | `occlusion_fill`、`joint_extension_parent`、`joint_extension_child` 三个布尔值 |
| `needs_completion` | 三项 completion 的逻辑或 |
| `confidence` | 0~1 自报置信度;说明与不确定性写入顶层 notes,部件内没有 notes 字段 |
坐标以完整画布左上角为原点,归一化到 0~1。可见框须为正面积;partial/hidden 必须声明遮挡补全,hidden 无可见框和正点。附件及面部特征不能声明独立骨骼关节延展。引用必须存在,不能自引用或形成父子/遮挡环路,绑定目标必须是骨骼。完整类型与必填项以 Schema 为准。
v1 迁移依据 ID 推断角色分类和补全需求,推断写入 notes,不重新识别图片、不覆盖旧文件。计划必须对应同一张图、同一画布。
## 5. 执行链路中重要部分代码文件及原理说明(包括重点说明)
**请求和结构校验。** `stages/model-stage.mjs` 调度;`providers/vision.mjs` 将图片 Data URL、画布尺寸、提示词与 Schema 转成对应协议,再提取 JSON,拒绝 refusal、异常结束和未完成响应。`lib/validate-parts.mjs` 校验字段、引用、层级和补全一致性,成功后写正式计划。低置信度等产生告警,不能据结构校验认定语义正确。
**候选点并非重新由 AI 生成。** `geometry_grounding.py` 对粗框与 `alpha>0` 的交集计算距离变换,选取远离边界的像素中心;无有效前景则 blocked。归一化点转像素采用 floor,1 映射到最后一个像素;粗框按 floor/ceil 转半开范围。候选仍可能属于框内的其他部件。
**pivot 与种子独立。** 关节点不吸附到前景,模型原始正负点仅作诊断。第二阶段当前只用程序种子排序返回的 mask,未把它们作为 SAM 点提示。
**通信、可视化与留档。** `providers/http.mjs` 负责 HTTP、SSE、超时和脱敏;明确的 429/5xx 有限重试,超时/网络中断不自动重复请求。`lib/report.mjs` 用红/黄圆区分原模型点、青色方块表示程序候选、空心圆表示 pivot。旧输出不会随代码修改自动更新。`lib/run-artifacts.mjs` 按输入、模型、参数、提示词、Schema、代码和产物哈希复用结果;completed 仅表示执行完成。
## 6. 各个代码文件职责索引
| 文件 | 职责 |
| --- | --- |
| [pipeline.mjs](../pipeline.mjs) / [tools/model-check.mjs](../tools/model-check.mjs) | 新管线转发 / CLI、模型列表和阶段分发 |
| [stages/model-stage.mjs](../stages/model-stage.mjs) | 理解总调度与留档;另含图片编辑探针 |
| [lib/config.mjs](../lib/config.mjs) / [lib/images.mjs](../lib/images.mjs) | 配置解析 / 图片格式、尺寸、哈希和 Data URL |
| [lib/python-runtime.mjs](../lib/python-runtime.mjs) / [lib/geometry.mjs](../lib/geometry.mjs) | Conda 检查 / Python 几何调用 |
| [normalize_reference.py](../normalize_reference.py) / [geometry_grounding.py](../geometry_grounding.py) | Alpha 与去白底 / ROI 诊断和候选生成 |
| [providers/vision.mjs](../providers/vision.mjs) / [providers/http.mjs](../providers/http.mjs) | 视觉协议 / 通信与脱敏 |
| [lib/validate-parts.mjs](../lib/validate-parts.mjs) | 解析、Schema/关系校验、v1 迁移 |
| [lib/report.mjs](../lib/report.mjs) / [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | SVG/HTML / 状态、指纹与缓存 |
| [prompts/understand-v1.txt](../prompts/understand-v1.txt) | 理解提示词;文件名保留 v1,当前使用 v2 契约 |
| [v2 Schema](../schemas/semantic-parts.schema.json) / [v1 Schema](../schemas/semantic-parts-v1.schema.json) | 正式契约 / 兼容输入 |
| [DeepSeek](../configs/deepseek-qwen.json)、[Astra](../configs/astra-qwen.json)、[Tiantoken](../configs/tiantoken.json) 配置 | 模型、协议和环境变量预设 |
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) / [tests/test_geometry.py](../tests/test_geometry.py) | 协议、计划、留档及几何回归 |
## 7. 产物详细说明
| 路径 | 内容与用途 |
| --- | --- |
| `source-original.png` | 原始输入副本 |
| `source.png` | 预处理 RGBA,在线模型所见图片,后续坐标基准 |
| `preprocessing.json` | 背景处理方式与像素统计 |
| `semantic-parts.json` / `validation.json` | 正式 v2 计划 / 字段、关系校验及告警 |
| `geometry-validation.json` | 原点诊断、ROI 前景比例、连通域、候选与阻塞原因 |
| `segmentation-prompts.json` | 画布、部件 ID、candidate/hidden/blocked 状态、粗框、程序种子和 QA 要求 |
| `annotated.svg` / `report.html` | 粗标注 / 综合复核页;不是 SAM 分割图 |
| `prompt.txt` / `request.json` | 阶段提示词副本 / 请求身份摘要,不是完整 HTTP body |
| `response.raw.json` / `response.txt` | 响应留档 / 提取文本;离线模式使用本地计划记录 |
| `request-metadata.json` | 请求与返回模型、request ID、usage、耗时、次数和协议元数据 |
| `response.error.json` | 可获取时保存的脱敏请求失败诊断 |
| `manifest.json` | 模式、状态、输入/代码/产物哈希;SAM 路径仅记录,不加载模型 |
失败可能只有部分产物。第二阶段前应复核左右、粗框、层级、遮挡、分类和补全需求,不能只看点是否落在前景。
## 8. 当前阶段可改进点说明
- 明确角色左右与画面左右命名约定,提升粗框、遮挡及隐藏状态判断,减少 SAM 衣料/肢体混淆。
- 根据预期动画审查拆分粒度与附件绑定,区分隐藏骨骼占位项和真正可见拆件。
- 增加可审阅的计划修正流程,修正后重新生成几何提示;Alpha 候选不能代替语义复核。
- 第二阶段后根据实际 mask 重新评估补全需求,不能因第一阶段 needs_completion=false 永久跳过有缺损的拆件。
- 增加结构和像素证据一致性检查,区分“不确定”“结构合法”“人工确认正确”。
-156
View File
@@ -1,156 +0,0 @@
# 第二阶段:本地 SAM 分割
## 1. 阶段简介
读取第一阶段语义计划,用本地 SAM 3.1 图像检测分支生成部件 mask,再从原图提取可见像素。当前输出半透明颜色叠加的角色图、白底拆解图和透明部件。
SAM 在 Conda `ai-spine` 的临时 GPU 子进程中运行,结束后退出;不部署常驻服务、不下载权重、不调用中转站、不自动回退 CPU。本阶段尚无大模型去毛边步骤,也不补画隐藏纹理。
## 2. 输入、输出与执行链路
输入为第一阶段目录,加本地 SAM 代码和权重;输出为可见部件 mask、原像素拆件、诊断和预览。
```text
tools/model-check.mjs --stage segmentation
→ stages/segmentation-stage.mjs 检查输入、路径、参数与 Conda
→ 临时探测子进程检查依赖和 CUDA,不加载权重
→ 计算输入、代码、权重指纹,复制输入到新输出目录
→ 临时推理子进程:sam_segment.py → sam_runtime.py
→ 严格加载检测分支,一次编码原图
→ 每部件分别取得文本、文本+粗框候选,排序选 mask
→ 与原图 Alpha 取交集,提取原像素并渲染报告
→ 子进程退出,manifest 记录产物与退出状态
→ 第三阶段读取 source.png、计划、segmentation.json 和 masks/
```
`hidden/blocked` 不进入推理;没有合格候选会记录跳过原因,不用矩形或种子伪造 mask。所有输出仍为 `semantic_verified=false`、`quality_status=needs_review`。
## 3. 阶段执行命令示例及参数说明
在项目根目录执行,环境安装见 [README](../README.md)。以下独立命令复用已有计划,无需模型密钥:
```powershell
conda activate ai-spine
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
node .\tools\model-check.mjs --stage segmentation `
--input-dir .\output\sombrero-grounding-v2 `
--out-dir ".\output\$runId\sam" `
--sam-dir F:/Study/SAM3
```
将 `--input-dir` 替换为本次第一阶段的 `understand` 目录,即可接续新实验。自动化可以使用 `conda run --no-capture-output -n ai-spine node ...`。
仅探测环境与输入,不进行模型推理:
```powershell
conda run --no-capture-output -n ai-spine node .\tools\model-check.mjs `
--stage segmentation `
--input-dir .\output\sombrero-grounding-v2 `
--sam-dir F:/Study/SAM3 `
--dry-run
```
| 参数 | 说明 |
| --- | --- |
| `--stage segmentation` | 独立运行第二阶段 |
| `--input-dir DIR` | 必需,第一阶段输出目录 |
| `--out-dir DIR` | 推理必需,使用独立新目录;dry-run 可省略 |
| `--sam-dir DIR` | SAM 根目录,优先于 SAM_MODEL_DIR 和配置默认值 |
| `--sam-code-dir DIR` | 源码仓库,优先于 SAM_CODE_DIR 和根目录下 sam3 |
| `--sam-checkpoint FILE` | 显式本地权重,覆盖根目录自动查找 |
| `--sam-confidence FLOAT` | 候选模型分数阈值,默认 0.2,严格介于 0 和 1;不是语义准确率 |
| `--sam-precision NAME` | 默认 bf16,可选 fp32;后者显存占用更高 |
| `--config FILE` / `--env-file FILE` | 配置与环境变量文件,默认 astra-qwen.json / 项目 .env |
| `--python FILE` | 仅允许 ai-spine 的解释器 |
| `--dry-run` | 检查路径、Conda、依赖导入和 CUDA;不校验完整权重加载或推理显存 |
| `--resume` | 输入、参数、代码、权重和完整产物一致时复用;失败任务另建目录 |
也可在第一阶段命令上使用 `--stop-after segment --sam-dir F:/Study/SAM3`,输出位于理解目录的 `segmentation/`。该方式会先执行理解阶段,可能调用付费模型。
## 4. 输入格式要求
| 输入 | 要求 |
| --- | --- |
| `source.png` | 第一阶段预处理 RGBA,保留完整原画布 |
| `semantic-parts.json` | 通过结构校验的 v2 计划,旧版须先离线迁移 |
| `segmentation-prompts.json` | 宽高匹配 source;部件 ID 与计划完整一致 |
| candidate 提示 | 非隐藏部件,seed_source 为 alpha_roi_distance_transform;整数粗框在画布内;至少一个整数种子在粗框与 Alpha 前景内 |
| SAM 代码 | 源码仓库内部必须有 sam3/model_builder.py |
| SAM 权重 | 本地 sam3.1_multiplex.pt,与本项目适配的检测分支结构匹配 |
当前本机布局:
```text
F:/Study/SAM3/
sam3/
sam3/model_builder.py
sam3.1checkpoint/
sam3.1_multiplex.pt
```
权重默认先查找上述子目录,再查找 SAM 根目录下同名文件。当前已验证环境为 Python 3.11、PyTorch 2.7.1+cu126、torchvision 0.22.1、NumPy 1.26.4、OpenCV 4.11.0.86,RTX 3060 Laptop 6 GB。这是本机验证组合,不是所有机器的兼容保证。
## 5. 执行链路中重要部分代码文件及原理说明(包括重点说明)
**本地生命周期与权重。** `stages/segmentation-stage.mjs` 先探测,再启动一次推理进程;离线环境变量禁止 Hugging Face 下载。推理超时 20 分钟,结束、报错或取消后退出。`sam_runtime.py` 从 multiplex 权重提取 detector 分支,适配三个特征尺度,移除未使用的第四尺度并取消 scalp;忽略不参与当前推理的视频 neck,实际加载参数使用 strict=True。外部 SAM 源码不作修改。
**精度不能简单全模型转换。** 模型参数和复数位置编码缓冲区保持原精度,在推理上下文开启 BF16 autocast;直接对整个模型调用 BF16 转换会影响复数缓冲区及要求 FP32 的路径。
**提示和选择。** `sam_segment.py` 将原图合成白底 RGB,GPU 只编码一次;每个部件分别收集 text_only 与 text_and_box 候选。排序结合模型分数、粗框内比例、Alpha 越界比例和种子命中;前景不足 4 像素或框内比例低于 10% 的候选被拒绝。粗框是提示,并非强制裁剪边界。
**重点:当前不是 SAM 点提示分割。** 所用图像概念接口接受文本和框。第一阶段程序种子仅参与返回 mask 排序,模型原始点不传入 SAM。提高种子前景命中率不能保证选中正确部件。
**像素与质量。** `segmentation_outputs.py` 用原图 Alpha 裁掉背景,从原图复制 RGB/Alpha,不重绘、不缩放;保留多部件重叠。记录越界、碎片、未命中种子、漏分、重叠和 IoU≥0.8 的疑似重复。覆盖率不是语义准确率,原像素保真也不能修复错误归属。
**当前预览含原图纹理。** `annotated.png` 是原图上 Alpha=100 的彩色 mask 叠加,不是仅用纯色区域组成的角色图;`parts-sheet.png` 为带 ID 的白底拆件。纯色遮罩角色图属于第 8 节的待实现项。
## 6. 各个代码文件职责索引
| 文件 | 职责 |
| --- | --- |
| [tools/model-check.mjs](../tools/model-check.mjs) | 独立分割入口、参数解析 |
| [stages/segmentation-stage.mjs](../stages/segmentation-stage.mjs) | 输入校验、环境探测、子进程、哈希、日志与失败处理 |
| [lib/config.mjs](../lib/config.mjs) | SAM 路径与配置解析 |
| [lib/python-runtime.mjs](../lib/python-runtime.mjs) | ai-spine 解释器与基础依赖检查 |
| [lib/images.mjs](../lib/images.mjs) / [lib/validate-parts.mjs](../lib/validate-parts.mjs) | PNG 元数据 / v2 计划结构校验 |
| [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | 输出目录、复用、产物哈希 |
| [sam_runtime.py](../sam_runtime.py) | 依赖/CUDA 探测、源码路径、检测权重适配与严格加载 |
| [sam_segment.py](../sam_segment.py) | 提示校验、GPU 推理、候选排序、诊断与阶段组织 |
| [segmentation_outputs.py](../segmentation_outputs.py) | mask QA、原像素提取、标注图、拆件图、HTML |
| [tests/test_segmentation.py](../tests/test_segmentation.py) | 候选、提示、Alpha、裁剪保真和重叠回归 |
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 阶段入口和相关本地回归调度 |
`segment.py` 是旧版白底拆件图分割器,不是本阶段 SAM 入口;`compose_preview.py`、`run_case_matrix.py` 是旧版定位/组装对照工具,不应与新阶段混用。
## 7. 产物详细说明
| 路径 | 内容与用途 |
| --- | --- |
| `source.png` / `semantic-parts.json` / `segmentation-prompts.json` | 输入副本,保留坐标与语义来源 |
| `annotated.png` | 原尺寸原图 + 半透明彩色 mask,背景合成白色 |
| `parts-sheet.png` | 原尺寸拆件分开排布的白底图,含 ID 与颜色标记 |
| `parts/<id>.png` | 紧裁剪 RGBA 部件,保留原图像素 |
| `masks/<id>.png` | 原画布尺寸的二值 mask,0/255,已裁掉 Alpha 背景 |
| `candidates/<id>-raw.png` | 选中的原始 SAM mask,尚未裁 Alpha,用于越界诊断 |
| `unassigned-mask.png` | 白色为没有任何部件覆盖的角色前景 |
| `overlap-mask.png` | 白色为多个部件共同覆盖的区域 |
| `segmentation.json` | 部件路径、crop_xyxy、sheet_xyxy、颜色、QA、跳过列表、覆盖率、运行环境、加载记录 |
| `candidate-diagnostics.json` | 两种提示候选的分数、排序、拒绝原因和最终选择 |
| `report.html` | 预览、部件链接、告警及漏分/重复复核 |
| `worker.log` / `worker-error.json` | 日志 / 子进程错误诊断(失败时) |
| `manifest.json` | 状态、输入/代码/权重/产物哈希;完成后 gpu_worker_exited=true |
`crop_xyxy` 为原图坐标,`sheet_xyxy` 为拆解图坐标,右下角均为开区间。第三阶段依赖全画布 masks,不以紧裁剪 parts 反推位置。
当前质量基线是 `output/20260918-183532/sam`:32 个计划项中 10 个 hidden 跳过,22 个 mask,前景覆盖率约 83.3%,32719 像素未分配。图中 forearm_r/hand_r 存在衣料误分,前臂混入手部,上衣及袖子存在缺失。更早 `sombrero-sam31-v3` 为另一批 28 个 mask、约 67.18% 覆盖率,不能混用统计。真实 GPU 推理跑通不代表拆件可直接用于动画。
## 8. 当前阶段可改进点说明
以下均为待实现或待验证的改进方向:
- **输出仅色彩遮罩的完整角色图。** 在原画布上以各部件纯色展示分区,不叠加原图纹理;保留现有 annotated.png。需明确重叠显示顺序,将未分配区域单独标识,不能为了“完整”伪造已识别区域。
- **可选的大模型去毛边。** 对每个通过部件归属复核的 SAM 拆件增加一次可选边缘清理,提供对应原图局部参考,约束修改范围,保存清理前后、mask 差异和坐标。默认保留原始拆件;清理不能冒充隐藏补全,也不能把重绘结果称为原像素。
- **先修正语义误分。** 明确左右、隐藏状态和粗框,复核衣料/皮肤/手部归属;对明显错误、缺失或疑似重复输出阻断性状态,再进入补全。
- **改善覆盖和边界。** 对未分配前景、过度重叠及大孔洞做定位复核;区分合理遮挡与分割缺口,避免一刀切强制所有 mask 互斥。
- **重评补全需求。** 基于实际 mask 检查袖子、衣服等缺损,反馈第三阶段;不能仅沿用第一阶段 needs_completion。
- **增加语义验收。** 模型分数、前景命中和覆盖率只作辅助;将原图、候选、最终 mask 并排复核,记录人工修正及来源。
-193
View File
@@ -1,193 +0,0 @@
# 第三阶段:部件补全
## 1. 阶段简介
读取 SAM 可见 mask,为需要遮挡补全或关节延伸的部件准备局部画布,调用中转站图片编辑模型,再将允许区域内的新增像素与原可见像素合并。Python 几何处理固定使用 Conda `ai-spine`;图片生成在远端完成,本阶段不加载本地 SAM。
当前已实现准备、真实图片请求、离线候选导入、几何检查与透明补全部件输出。尚未输出使用补全部件组装的完整角色,尚未接入新管线 Spine 导出。几何通过仍为 `needs_review`,不能等同于部件可用。
## 2. 输入、输出与执行链路
输入是第二阶段目录;输出是任务画布、候选、检查报告及通过几何检查的补全部件。
```text
tools/model-check.mjs --stage completion --input-dir SAM目录
→ stages/completion-stage.mjs 校验计划、mask 文件、参数及 Conda
→ completion_geometry.py prepare
筛选需求 → 计算允许区域 → 建立局部画布与编辑 mask
→ prepare-only:保存任务和报告后结束
→ 否则:串行请求 images/edits,或读取离线候选
→ completion_geometry.py merge
检查画布/前景/对齐 → 限制新增区域与连通性 → 恢复原可见像素
→ 每部件选第一个通过几何检查的候选
→ finalize 输出透明部件、原图坐标和报告
```
任务必须同时满足:计划 `needs_completion=true`、非 hidden、有可见 SAM mask、允许新增区域非空。指定 `--part` 只缩小范围,不强制绕过筛选。分别记录 `NO_COMPLETION_NEEDED`、`NO_VISIBLE_SAM_MASK`、`NO_ALLOWED_EXTENSION`。
以 `output/20260918-183532/sam` 为例:32 个计划项中 10 个 hidden 没有 mask;22 个有 mask,其中 12 个进入补全。帽子、两袖、两眼、两眉、鼻、嘴、胡子这 10 个因第一阶段标记不需补全而跳过,没有重新由模型判断实际 mask 完整性。
## 3. 阶段执行命令示例及参数说明
以下命令在项目根目录执行;输入示例是已有实验,不代表其 mask 已合格。
```powershell
conda activate ai-spine
$runId = Get-Date -Format "yyyyMMdd-HHmmss"
$samInput = ".\output\20260918-183532\sam"
# 离线准备所有符合条件的任务,无需密钥
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir $samInput `
--out-dir ".\output\$runId\complete-prepare" `
--prepare-only
# 单部件真实图片编辑请求
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir $samInput `
--out-dir ".\output\$runId\complete-head" `
--completion-model gpt-image-2 `
--part head
# 所有符合条件的部件,逐部件请求模型
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir $samInput `
--out-dir ".\output\$runId\complete"
```
使用 Qwen 时改为 `--config .\configs\deepseek-qwen.json --completion-model qwen-image-3.0`,读取原中转配置。Tiantoken 配置第三阶段为 `gpt-image-2`,第一阶段的 `gpt-6-astra` 不是当前图片编辑接入的替代品;未实现 Astra Responses 图片生成工具链。
旧候选离线复检,不重新请求模型:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--input-dir $samInput `
--candidates-dir .\output\20260918-183532\complete2_gpt `
--out-dir ".\output\$runId\complete-recheck" `
--candidate-size-policy uniform
```
仅测试图片编辑接口的独立探针:
```powershell
node .\tools\model-check.mjs --stage completion `
--config .\configs\tiantoken.json `
--image .\output\20260918-183532\complete2_gpt\tasks\head\input.png `
--prompt-file .\output\20260918-183532\complete2_gpt\tasks\head\prompt.txt `
--out-dir ".\output\$runId\edit-probe"
```
探针只保存候选,不执行部件准备与几何验收,不能与 `--input-dir` 混用。
| 参数 | 说明 |
| --- | --- |
| `--input-dir DIR` | 完整 SAM 输出目录,部件补全必需 |
| `--out-dir DIR` | 独立新目录,准备与生成使用不同目录 |
| `--part ID` | 可重复,ID 必须存在且不重复;省略时处理所有符合筛选条件的部件 |
| `--prepare-only` | 只生成画布、区域、提示词和报告,不请求模型 |
| `--candidates-dir DIR` | 离线导入候选,与 prepare-only 互斥 |
| `--completion-size N` | 方形画布边长,默认 1024,整数 128~4096;原像素不缩放 |
| `--extension-radius N` | 轮廓延展距离与关节圆盘半径,默认 32,整数 1~256 |
| `--candidate-size-policy NAME` | strict 默认拒绝尺寸不符;uniform 允许同宽高比整画布缩放后复检 |
| `--completion-model ID` | 图片模型 ID,原样发送 |
| `--config FILE` / `--env-file FILE` | 默认 astra-qwen.json / 项目 .env;shell 同名变量优先 |
| `--python FILE` | 仅允许 ai-spine 解释器 |
| `--dry-run` | 检查输入文件、计划、参数和 Python 基础环境,不准备画布或请求模型;完整像素级 mask 校验在 prepare 中执行 |
| `--resume` | 仅复用已完成、输入/配置/源码/产物一致的任务,不自动续发失败请求 |
| `--image FILE` / `--prompt-file FILE` | 仅用于探针:PNG 图片 / 文本提示词;多次 image 需配置多图能力 |
| `--mask FILE` | 仅用于探针:透明处可编辑的 RGBA PNG,需配置 mask 能力 |
配置中默认 `n=1`,逐部件串行调用。明确的 429/5xx 有限重试,超时/断网不自动重试;请求失败保留诊断并停止后续部件,几何拒绝会记录结果并继续。`--stop-after` 不支持 completion。
## 4. 输入格式要求
| 输入 | 要求 |
| --- | --- |
| `source.png` | 原画布 PNG,使用第二阶段的 RGBA 原图 |
| `semantic-parts.json` | 有效 v2 计划;部件分类、遮挡、pivot 和补全需求合法 |
| `segmentation.json` | parts 数组,ID 唯一且存在于计划中 |
| `masks/<id>.png` | segmentation.parts 中每个 ID 都必须有文件;原画布尺寸,灰度值仅 0/255,非空,不覆盖原图 Alpha=0 区域 |
| 中转配置 | 在线模式需地址、密钥、images-edits 提供方及图片模型;prepare-only/离线导入无需密钥 |
只提供 SAM 的 `parts/` 或白底拆解图不满足要求。输入目录中无需存在 annotated.png 才能运行,但建议先通过报告复核语义。
离线候选支持两种形式:
- 平铺目录:`<id>.png`,必须对应本次局部画布坐标、尺寸与部件;平铺文件没有旧任务 manifest 可自动核对,需自行保证一致。
- 旧补全目录:含 manifest 和 `tasks/<id>/candidate-1.png`,也支持 jpg/webp;逐项核对源图、计划、segmentation.json、全部 mask 哈希以及画布/延展参数。每部件只取首张候选。缺失记为 missing_candidate,不联网补发。
不同原图、mask 或参数的候选不能混用。尺寸之外,部件位置、朝向、比例也必须保持一致。
## 5. 执行链路中重要部分代码文件及原理说明(包括重点说明)
**允许区域。** `completion_geometry.py::allowed_regions()` 先计算可见 mask 外、距轮廓不超过 extension-radius 的区域,再分别与以下区域取交集:遮挡补全用 occluded_by 的 mask 并集;父关节延展用本部件 pivot 周围圆盘;子关节延展用直接子骨骼部件 pivot 圆盘。始终排除原可见 mask,新增范围限制在原图画布内。它是保守几何范围,不是完整隐藏形状推断。
**局部坐标。** `prepare()` 将可见区和允许区居中放入方形画布,至少留 16 像素边距,只平移和补边。`origin_xy` 是局部左上角在原图的位置,可为负数;原图位置=局部位置+origin。放不下时失败,不能偷偷缩放原像素。
**重点:当前实际发送什么。** `stages/completion-stage.mjs` 默认只发送白底 `input.png`。三个现有配置均为 `multiple_images=false`、`mask=false`,因此本地生成 context.png 和 edit-mask.png 不意味着发给模型。只有开启并验证对应能力后,分支才发送第二张同坐标局部原图参考和编辑遮罩;当前没有独立全角色参考图输入。文字边界说明及返回后的区域裁剪不能替代模型侧遮罩约束。
**图片解码与尺寸。** `providers/image-edits.mjs` 使用 multipart images/edits,支持非空 b64_json 或返回 URL 下载;空 b64_json 会回退有效 URL。当前部件请求显式发送 size=画布宽x高,并写入 request.json;配置 params.size 冲突会报错。strict 模式拒绝 CANVAS_SIZE_MISMATCH;uniform 仅将同宽高比整画布 LANCZOS 缩放到目标尺寸,保留原候选,不平移、不裁剪、不单独缩放物体,也不改变原图/mask。
**几何验收与合并。** `merge_candidate()` 去除边界连通白底,在原部件 Alpha≥240 的像素处检查 RGB 平均绝对误差≤30、可见保留率≥95%;无不透明锚点拒绝。只取允许区域内且与原部件连通的新增像素,没有有效新增即拒绝。通过后逐像素恢复原可见 RGBA,模型只提供新增部分。
**重点:验收仍有缺口。** `unfulfilled_regions` 记录未补到的子区域,但当前不是硬性拒绝条件;某一区域新增少量像素也不代表关节补够。语义、接缝、材质、运动露底没有自动验收。错误 SAM 像素会被原样恢复;若头部 mask 已含眼睛/胡子,当前规则也不能清除这些内部附件生成干净脸底。
## 6. 各个代码文件职责索引
| 文件 | 职责 |
| --- | --- |
| [tools/model-check.mjs](../tools/model-check.mjs) | 部件补全与独立探针分发、参数互斥检查 |
| [stages/completion-stage.mjs](../stages/completion-stage.mjs) | 输入指纹、准备、串行请求/离线读取、候选选择、状态留档 |
| [completion_geometry.py](../completion_geometry.py) | allowed_regions、prepare、merge_candidate、finalize 与报告 |
| [normalize_reference.py](../normalize_reference.py) | 候选 Alpha 保留、边界连通白底提取 |
| [providers/image-edits.mjs](../providers/image-edits.mjs) | multipart 编辑请求、能力检查、Base64/URL 图片解码 |
| [providers/http.mjs](../providers/http.mjs) | HTTP、超时、重试、诊断和脱敏 |
| [stages/model-stage.mjs](../stages/model-stage.mjs) | runCompletionProbe,仅验证编辑接口 |
| [lib/config.mjs](../lib/config.mjs) / [lib/images.mjs](../lib/images.mjs) | 提供方、模型与能力配置 / 图片格式、尺寸和哈希 |
| [lib/python-runtime.mjs](../lib/python-runtime.mjs) / [lib/validate-parts.mjs](../lib/validate-parts.mjs) | Conda 检查 / v2 结构校验 |
| [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | 输出目录、复用、状态和产物哈希 |
| [prompts/complete-v1.txt](../prompts/complete-v1.txt) | 保持姿势、尺寸、可见纹理与隐藏区域补全约束 |
| [Tiantoken](../configs/tiantoken.json)、[DeepSeek/Qwen](../configs/deepseek-qwen.json)、[Astra/Qwen](../configs/astra-qwen.json) 配置 | 图片模型、认证变量、请求参数与能力开关 |
| [tests/completion-stage.test.mjs](../tests/completion-stage.test.mjs) / [tests/test_completion.py](../tests/test_completion.py) | mock 编辑、离线复用、尺寸/对齐拒绝、像素保真与区域回归 |
| [tests/model-stages.test.mjs](../tests/model-stages.test.mjs) | 提供方与探针相关回归 |
## 7. 产物详细说明
| 路径 | 内容与用途 |
| --- | --- |
| `source.png`、`semantic-parts.json`、`segmentation.json`、`masks/` | 输入快照,用于复现与离线候选身份核对 |
| `completion-plan.json` | 准备任务、允许区域、参数和跳过原因 |
| `tasks/<id>/input.png` | 白底局部可见部件,默认唯一模型输入图 |
| `tasks/<id>/visible.png` | 原图可见 RGBA,合并时恢复像素 |
| `tasks/<id>/context.png` | 同坐标局部原图参考,当前默认不发送 |
| `tasks/<id>/edit-mask.png` | RGBA 编辑遮罩,透明处可编辑,当前默认不发送 |
| `tasks/<id>/allowed-mask.png` | 白色表示允许新增区域 |
| `tasks/<id>/<补全字段>.png` | 启用的补全子区域 mask |
| `tasks/<id>/task.json` / `prompt.txt` | origin、区域、需求和告警 / 实际提示词 |
| `tasks/<id>/request.json` | 在线请求模型、图片哈希、是否发送 mask;修复后含 size |
| `tasks/<id>/response.raw.json` / `request-metadata.json` | 在线响应留档 / 请求 ID、耗时等元数据 |
| `tasks/<id>/response.error.json` | 可获取时保存的脱敏请求失败诊断 |
| `tasks/<id>/candidate-N.*` | 模型或离线导入的原始候选,拒绝也保留 |
| `tasks/<id>/merged-N-normalized.png` | uniform 实际重采样时保存的候选;不代表通过 |
| `tasks/<id>/qa-N.json` | 尺寸、对齐、新增/丢弃像素、各区域完成情况、拒绝原因 |
| `tasks/<id>/merged-N.png` | 通过几何检查且恢复原可见像素后的局部 RGBA |
| `parts/<id>.png` | 每部件选中结果的紧裁剪透明 PNG |
| `completion-result.json` | 每部件状态、候选 QA、原图 crop_xyxy 和路径 |
| `report.html` / `manifest.json` | 人工检查页 / 模式、网络标记、状态、输入/代码/产物哈希 |
parts/ 只含本次选中且通过检查的补全部件,不是全部 SAM 部件集合。无需补全、失败和未选中的部件不会自动复制进来,也没有完整角色组装图。
当前实验:`output/20260918-183532/complete2_gpt` 已真实请求 Tiantoken gpt-image-2 共 12 个部件。旧请求未传 size,返回 1248/1254 方形图,预期为 1024,全部严格尺寸拒绝。`complete2_gpt-size-recheck-v2` 离线复用原候选后,8 个几何通过;head、pants_thigh_l、pants_thigh_r、shoe_l 共 4 个因对齐或无有效新增拒绝。通过项中仍有衣料误分、错误手部和未补全子区域。显式 size 的请求构造已验证,但尚无修复后中转遵守 size 的新实测结论。
## 8. 当前阶段可改进点说明
- **补全后输出使用补全部件的完整角色。** 按原图坐标和遮挡顺序组装,复用无需补全的原拆件,明确标记失败项及回退来源;输出补全前后对比、接缝/重叠诊断。不能只展示少数 parts 文件作为完整结果。
- **把原图或部件相关原图区域交给补全模型参考。** 接入并实测全角色/局部上下文参考和编辑 mask,记录实际发送内容。不能仅把 capabilities 改为 true 就宣称中转支持;需验证图片顺序、mask 语义、返回尺寸和画面保持能力。
- **根据 SAM 实际结果重建需求。** 区分分割缺口修复、遮挡补全、关节延伸、附件底图清理;对应建立保留区域与可编辑区域,避免误分像素被永久保护。
- **改进画布与补全区域。** 固定 32 像素不适配所有部件;小部件在 1024 大白底中占比低。可验证按部件设计区域、显式局部放大并记录逆变换,保证回贴坐标可检查。
- **加强验收。** 对必需区域定义有效覆盖要求,增加部件语义、接缝、材质、回贴和关节旋转露底检查;不能放宽阈值来掩盖重绘、位移或错误输入。
- **按小样验证后批量。** 先修正 SAM 输入,再用头、前臂、上衣验证参考图与遮罩,比较原输入、原候选、合并图和回贴图,确认改善后再跑全量。
-129
View File
@@ -1,129 +0,0 @@
#!/usr/bin/env python3
"""Alpha constraints and deterministic seed candidates, never semantic masks."""
import argparse
import json
import math
from pathlib import Path
import cv2
import numpy as np
from PIL import Image
from normalize_reference import prepare_rgba
def pixel(point, width, height):
# Normalized points map to their containing pixel; 1 maps to the last pixel.
return min(width - 1, int(point[0] * width)), min(height - 1, int(point[1] * height))
def ground_parts(image, plan):
alpha = np.array(image.convert('RGBA'))[:, :, 3]
foreground = alpha > 0
height, width = alpha.shape
usable_alpha = bool(foreground.any() and not foreground.all())
silhouette_distance = cv2.distanceTransform(np.pad(foreground.astype(np.uint8), 1), cv2.DIST_L2, cv2.DIST_MASK_PRECISE)[1:-1, 1:-1]
diagnostics, prompts = [], []
for part in plan['parts']:
box = part['bbox_norm_xyxy']
entry = {
'id': part['id'], 'status': 'hidden' if part['visibility'] == 'hidden' else 'blocked',
'bbox_px_xyxy': None, 'bbox_foreground_ratio': None,
'foreground_bbox_px_xyxy': None, 'foreground_components': 0,
'model_positive_points': [], 'candidate_points': [], 'warnings': [],
'semantic_verified': False,
}
for point in part.get('positive_points_norm', []):
x, y = pixel(point, width, height)
in_box = bool(box and box[0] <= point[0] <= box[2] and box[1] <= point[1] <= box[3])
on_foreground = bool(foreground[y, x])
entry['model_positive_points'].append({
'point_norm': point, 'point_px': [x, y], 'alpha': int(alpha[y, x]),
'inside_bbox': in_box, 'on_foreground': on_foreground,
'status': 'UNTRUSTED_SEED' if usable_alpha and on_foreground and in_box else 'INVALID_SEED',
})
if part['visibility'] != 'hidden' and box:
x0, y0 = math.floor(box[0] * width), math.floor(box[1] * height)
x1, y1 = math.ceil(box[2] * width), math.ceil(box[3] * height)
entry['bbox_px_xyxy'] = [x0, y0, x1, y1]
roi = foreground[y0:y1, x0:x1]
entry['bbox_foreground_ratio'] = float(roi.mean()) if roi.size else 0
if not usable_alpha:
entry['warnings'].append('NO_USABLE_ALPHA')
elif not roi.any():
entry['warnings'].append('EMPTY_FOREGROUND_ROI')
else:
ys, xs = np.nonzero(roi)
entry['foreground_bbox_px_xyxy'] = [int(xs.min()) + x0, int(ys.min()) + y0, int(xs.max()) + x0 + 1, int(ys.max()) + y0 + 1]
count, _ = cv2.connectedComponents(roi.astype(np.uint8), connectivity=8)
entry['foreground_components'] = count - 1
# Pad the ROI with background so an opaque ROI has a finite
# distance, and candidates stay away from both silhouette/box edges.
distance = cv2.distanceTransform(np.pad(roi.astype(np.uint8), 1), cv2.DIST_L2, cv2.DIST_MASK_PRECISE)[1:-1, 1:-1]
# Only centers strictly inside the original continuous ROI qualify.
eligible_x = ((np.arange(x0, x1) + .5) / width >= box[0]) & ((np.arange(x0, x1) + .5) / width <= box[2])
eligible_y = ((np.arange(y0, y1) + .5) / height >= box[1]) & ((np.arange(y0, y1) + .5) / height <= box[3])
eligible = roi & eligible_y[:, None] & eligible_x[None, :]
if eligible.any():
scores = np.where(eligible, distance, -1)
cy, cx = np.unravel_index(np.argmax(scores), scores.shape)
x, y = int(cx) + x0, int(cy) + y0
entry['candidate_points'] = [{
'point_px': [x, y], 'point_norm': [(x + .5) / width, (y + .5) / height],
'distance_to_roi_boundary_px': float(distance[cy, cx]),
'distance_to_silhouette_px': float(silhouette_distance[y, x]),
'source': 'alpha_roi_distance_transform',
}]
entry['status'] = 'candidate'
else:
entry['warnings'].append('NO_PIXEL_CENTER_IN_ROI')
if entry['bbox_foreground_ratio'] < .5:
entry['warnings'].append('LOW_FOREGROUND_RATIO')
if count > 2:
entry['warnings'].append('MULTIPLE_FOREGROUND_COMPONENTS')
entry['warnings'].append('SEMANTIC_REVIEW_REQUIRED')
diagnostics.append(entry)
prompts.append({
'id': part['id'], 'text_prompt': part['semantic'], 'status': entry['status'],
'bbox_px_xyxy': entry['bbox_px_xyxy'],
'positive_points_px': [p['point_px'] for p in entry['candidate_points']],
'positive_points_norm': [p['point_norm'] for p in entry['candidate_points']],
'negative_points_px': [], 'seed_source': 'alpha_roi_distance_transform',
'semantic_verified': False, 'requires_mask_qa': True,
})
points = [p for entry in diagnostics for p in entry['model_positive_points']]
return {
'schema_version': 1, 'width': width, 'height': height,
'alpha_threshold': 0, 'coordinate_mapping': 'floor(norm * size), clamped at size - 1; boxes are half-open',
'summary': {'model_points': len(points), 'invalid_model_points': sum(p['status'] == 'INVALID_SEED' for p in points),
'background_model_points': sum(not p['on_foreground'] for p in points),
'candidate_parts': sum(p['status'] == 'candidate' for p in diagnostics),
'blocked_parts': sum(p['status'] == 'blocked' for p in diagnostics)},
'parts': diagnostics,
}, {'schema_version': 1, 'width': width, 'height': height, 'quality_status': 'needs_review', 'parts': prompts}
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--input', required=True)
parser.add_argument('--output', required=True)
parser.add_argument('--prepare', action='store_true')
parser.add_argument('--parts')
parser.add_argument('--prompts')
args = parser.parse_args()
with Image.open(args.input) as image:
if args.prepare:
prepared, metadata = prepare_rgba(image)
prepared.save(args.output, format='PNG')
print(json.dumps(metadata))
else:
if not args.parts or not args.prompts:
parser.error('--parts and --prompts are required for grounding')
plan = json.loads(Path(args.parts).read_text(encoding='utf-8'))
diagnostics, prompts = ground_parts(image, plan)
Path(args.output).write_text(json.dumps(diagnostics, indent=2), encoding='utf-8')
Path(args.prompts).write_text(json.dumps(prompts, indent=2), encoding='utf-8')
if __name__ == '__main__':
main()
-96
View File
@@ -1,96 +0,0 @@
import {existsSync, readFileSync} from 'node:fs';
import path from 'node:path';
import {fileURLToPath} from 'node:url';
export const projectRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
export function readEnvFile(file) {
if (!existsSync(file)) return {};
const values = {};
for (const line of readFileSync(file, 'utf8').replace(/^\uFEFF/u, '').split(/\r?\n/u)) {
const match = line.match(/^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*?)\s*$/u);
if (!match) continue;
let value = match[2];
if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) {
value = value.slice(1, -1);
} else value = value.replace(/\s+#.*$/u, '');
values[match[1]] = value;
}
return values;
}
export function loadConfig({configFile, envFile, env = process.env} = {}) {
const file = path.resolve(configFile || path.join(projectRoot, 'configs', 'astra-qwen.json'));
const config = JSON.parse(readFileSync(file, 'utf8').replace(/^\uFEFF/u, ''));
if (config.schema_version !== 1 || !config.providers || !config.stages) {
throw new Error('配置必须声明 schema_version: 1、providers 和 stages');
}
return {config, file, env: {...readEnvFile(envFile || path.join(projectRoot, '.env')), ...env}};
}
export function resolveStage(loaded, name, overrides = {}, {requireKey = true} = {}) {
const stage = loaded.config.stages[name];
if (!stage) throw new Error(`没有配置阶段:${name}`);
const provider = loaded.config.providers[stage.provider];
if (!provider) throw new Error(`没有配置提供方:${stage.provider}`);
const model = String(overrides.model || stage.model || loaded.env[stage.model_env] || stage.default_model || '').trim();
if (!model) throw new Error(`缺少 ${name} 的模型 ID`);
const baseUrl = String(loaded.env[provider.base_url_env] || '').trim().replace(/\/+$/u, '');
const apiKey = String(loaded.env[provider.api_key_env] || '').trim();
if (requireKey && (!baseUrl || !apiKey)) throw new Error(`需要配置 ${provider.base_url_env} 和 ${provider.api_key_env}`);
if (baseUrl) {
const url = new URL(baseUrl);
if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password || url.search || url.hash) {
throw new Error('提供方 base URL 必须是无用户名、密码、查询参数的 HTTP(S) 地址');
}
}
const protocol = overrides.protocol || provider.protocol;
const allowed = name === 'understanding' ? ['chat-completions', 'responses'] : ['images-edits'];
if (!allowed.includes(protocol)) throw new Error(`${name} 不支持协议:${protocol}`);
const params = {...stage.params, ...overrides.params};
// Semantic fields are owned by adapters, never overridden via opaque params.
for (const key of ['model', 'messages', 'input', 'instructions', 'prompt', 'image', 'image[]', 'mask', 'response_format', 'text', 'stream', 'Authorization', 'api_key', 'apiKey']) {
if (Object.hasOwn(params, key)) throw new Error(`params 不能覆盖保留字段:${key}`);
}
const timeoutMs = provider.timeout_ms ?? 180000;
const maxRetries = provider.max_retries ?? 2;
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || !Number.isInteger(maxRetries) || maxRetries < 0 || maxRetries > 5) {
throw new Error('timeout_ms 必须为正数,max_retries 必须为 0~5 的整数');
}
const outputMode = overrides.outputMode || stage.output_mode || 'text-json';
if (!['text-json', 'json-object', 'json-schema'].includes(outputMode)) throw new Error(`未知 output_mode:${outputMode}`);
const stream = overrides.stream ?? stage.stream ?? false;
if (typeof stream !== 'boolean') throw new Error('stream 必须为 boolean');
const promptPath = path.resolve(path.dirname(loaded.file), overrides.prompt || stage.prompt);
const capabilities = {multiple_images: false, mask: false, ...provider.capabilities};
if (typeof capabilities.mask !== 'boolean' || typeof capabilities.multiple_images !== 'boolean') {
throw new Error('capabilities.mask 和 multiple_images 必须为 boolean');
}
const imageField = provider.image_field || 'image';
if (!['image', 'image[]'].includes(imageField)) throw new Error('image_field 必须是 image 或 image[]');
return {
name, provider: stage.provider, protocol, model, baseUrl, apiKey, timeoutMs, maxRetries,
params, outputMode, stream, promptPath, prompt: readFileSync(promptPath, 'utf8'), capabilities,
imageField, maskEditable: provider.mask_editable || 'transparent',
};
}
export function publicStage(stage) {
return {
provider: stage.provider, protocol: stage.protocol, model: stage.model,
params: stage.params, output_mode: stage.outputMode, stream: stage.stream, timeout_ms: stage.timeoutMs,
max_retries: stage.maxRetries, capabilities: stage.capabilities,
image_field: stage.imageField, mask_editable: stage.maskEditable,
};
}
export function samInfo(loaded, overrides = {}) {
const stage = loaded.config.stages.segmentation || {};
const modelDir = overrides.samDir || loaded.env[stage.model_dir_env] || stage.default_model_dir || '';
const checkpoint = overrides.samCheckpoint || (modelDir ? [path.join(modelDir, 'sam3.1checkpoint', 'sam3.1_multiplex.pt'), path.join(modelDir, 'sam3.1_multiplex.pt')].find(existsSync) || path.join(modelDir, 'sam3.1checkpoint', 'sam3.1_multiplex.pt') : '');
const codeDir = overrides.samCodeDir || loaded.env.SAM_CODE_DIR || (modelDir ? path.join(modelDir, 'sam3') : '');
return {model_dir: modelDir, code_dir: codeDir, checkpoint,
checkpoint_exists: Boolean(checkpoint && existsSync(checkpoint)),
code_exists: Boolean(codeDir && existsSync(path.join(codeDir, 'sam3', 'model_builder.py'))),
runtime_status: 'local_subprocess_not_probed'};
}
-16
View File
@@ -1,16 +0,0 @@
import {execFileSync} from 'node:child_process';
import {readFileSync} from 'node:fs';
import path from 'node:path';
import {projectRoot} from './config.mjs';
export {selectPython as geometryPython} from './python-runtime.mjs';
export function prepareReference(python, input, output) {
return JSON.parse(execFileSync(python, [path.join(projectRoot, 'geometry_grounding.py'), '--prepare', '--input', input, '--output', output], {encoding: 'utf8', windowsHide: true}));
}
export function groundParts(python, out) {
const diagnostics = path.join(out, 'geometry-validation.json');
execFileSync(python, [path.join(projectRoot, 'geometry_grounding.py'), '--input', path.join(out, 'source.png'), '--parts', path.join(out, 'semantic-parts.json'), '--output', diagnostics, '--prompts', path.join(out, 'segmentation-prompts.json')], {encoding: 'utf8', windowsHide: true});
return JSON.parse(readFileSync(diagnostics, 'utf8'));
}
-23
View File
@@ -1,23 +0,0 @@
import {readFileSync} from 'node:fs';
import {createHash} from 'node:crypto';
import path from 'node:path';
export function sha256(value) { return createHash('sha256').update(value).digest('hex'); }
export function imageFormat(bytes) {
if (bytes.length >= 24 && bytes.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]))) {
return {mime: 'image/png', extension: 'png', width: bytes.readUInt32BE(16), height: bytes.readUInt32BE(20)};
}
if (bytes.length >= 3 && bytes[0] === 255 && bytes[1] === 216 && bytes[2] === 255) return {mime: 'image/jpeg', extension: 'jpg'};
if (bytes.length >= 12 && bytes.toString('ascii', 0, 4) === 'RIFF' && bytes.toString('ascii', 8, 12) === 'WEBP') return {mime: 'image/webp', extension: 'webp'};
throw new Error('图片数据不是支持的 PNG/JPEG/WebP 格式');
}
export function readImage(file, {pngOnly = false} = {}) {
const bytes = readFileSync(file);
const format = imageFormat(bytes);
if (pngOnly && (format.extension !== 'png' || !format.width || !format.height)) throw new Error('结构理解与局部画布当前要求非空 PNG 图片');
return {file: path.resolve(file), bytes, ...format, sha256: sha256(bytes)};
}
export function dataUrl(image) { return `data:${image.mime};base64,${image.bytes.toString('base64')}`; }
-23
View File
@@ -1,23 +0,0 @@
import {spawnSync} from 'node:child_process';
import path from 'node:path';
export function selectPython(requested) {
const command = requested || (process.env.CONDA_PREFIX
? path.join(process.env.CONDA_PREFIX, process.platform === 'win32' ? 'python.exe' : 'bin/python')
: 'python');
const probe = spawnSync(command, ['-c', [
'import json, sys',
'from pathlib import Path',
"print(json.dumps({'executable': sys.executable, 'name': Path(sys.prefix).name, 'conda': (Path(sys.prefix) / 'conda-meta').is_dir()}))",
].join('\n')], {encoding: 'utf8', windowsHide: true});
let runtime;
try { if (probe.status === 0) runtime = JSON.parse(probe.stdout.trim()); } catch { /* Report an actionable environment error below. */ }
if (!runtime?.conda || runtime.name !== 'ai-spine') {
throw new Error('本项目固定使用 Conda 环境 ai-spine。请先 conda activate ai-spine,或用 conda run -n ai-spine node ...;--python 也必须指向该环境的解释器');
}
const dependencies = spawnSync(runtime.executable, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore', windowsHide: true});
if (dependencies.status !== 0) {
throw new Error('ai-spine 环境缺少或无法加载 Python 依赖。请在 AI_spine 目录执行:conda run -n ai-spine python -m pip install -r requirements.txt');
}
return runtime.executable;
}
-41
View File
@@ -1,41 +0,0 @@
import {dataUrl} from './images.mjs';
const escape = (value) => String(value).replace(/[&<>"']/gu, (c) => ({'&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;'}[c]));
export function annotatedSvg(image, parts, geometry = null) {
const {width: w, height: h} = image;
const stroke = Math.max(w, h) / 400;
const font = Math.max(w, h) / 65;
const byId = new Map(parts.map((part) => [part.id, part]));
const diagnostics = new Map((geometry?.parts || []).map((part) => [part.id, part]));
const overlays = parts.map((part, index) => {
const color = `hsl(${(index * 137.5) % 360} 85% 55%)`;
const [px, py] = [part.pivot_norm[0] * w, part.pivot_norm[1] * h];
const parent = byId.get(part.parent);
const line = parent ? `<line x1="${px}" y1="${py}" x2="${parent.pivot_norm[0] * w}" y2="${parent.pivot_norm[1] * h}" stroke-dasharray="${stroke * 3}"/>` : '';
let box = '';
if (part.bbox_norm_xyxy) {
const [x0, y0, x1, y1] = part.bbox_norm_xyxy;
box = `<rect x="${x0 * w}" y="${y0 * h}" width="${(x1 - x0) * w}" height="${(y1 - y0) * h}"/>`;
}
const diagnostic = diagnostics.get(part.id);
const points = part.positive_points_norm.map(([x, y], i) => {
const status = diagnostic?.model_positive_points[i]?.status;
const fill = status === 'INVALID_SEED' ? '#ff4444' : status === 'UNTRUSTED_SEED' ? '#ffc857' : '#999';
return `<circle cx="${x * w}" cy="${y * h}" r="${stroke * 2}" stroke="#111" fill="${fill}"><title>${escape(part.id)}: ${status || 'UNTRUSTED_MODEL_POINT'}</title></circle>`;
}).join('');
const candidates = (diagnostic?.candidate_points || []).map(({point_norm: [x, y]}) => `<rect x="${x * w - stroke * 2}" y="${y * h - stroke * 2}" width="${stroke * 4}" height="${stroke * 4}" stroke="#111" fill="#44e5ff"><title>${escape(part.id)}: Alpha candidate, semantic unverified</title></rect>`).join('');
return `<g fill="none" stroke="${color}" stroke-width="${stroke}">${line}${box}${points}${candidates}<circle cx="${px}" cy="${py}" r="${stroke * 3}"/><text x="${Math.min(px + 4, w * 0.6)}" y="${Math.max(font, py - 5)}" fill="${color}" stroke="#111" stroke-width="${stroke / 2}" paint-order="stroke" font-size="${font}" font-family="sans-serif">${escape(part.id)}</text></g>`;
}).join('\n');
return `<svg xmlns="http://www.w3.org/2000/svg" width="${w}" height="${h}" viewBox="0 0 ${w} ${h}"><rect width="100%" height="100%" fill="#262632"/><image href="${dataUrl(image)}" width="${w}" height="${h}"/>${overlays}</svg>`;
}
export function understandingReport(image, data, validation, metadata, geometry = null) {
const svg = data && validation.valid ? annotatedSvg(image, data.parts, geometry) : `<img src="${dataUrl(image)}" alt="source">`;
const diagnostics = new Map((geometry?.parts || []).map((part) => [part.id, part]));
const rows = data && validation.valid ? data.parts.map((part) => {
const g = diagnostics.get(part.id);
return `<tr><td>${escape(part.id)}</td><td>${escape(part.rig_role || '')}<br>${escape(part.bind_to ?? part.parent ?? 'root')}</td><td>${escape(part.visibility)}</td><td>${escape(part.occluded_by.join(', '))}</td><td>${escape(Object.entries(part.completion || {}).filter(([, value]) => value).map(([key]) => key).join(', ') || '无')}</td><td>${g?.bbox_foreground_ratio == null ? '—' : `${(g.bbox_foreground_ratio * 100).toFixed(1)}%`}</td><td>${escape(g?.status || 'pending')}<br>${escape((g?.warnings || []).join(', '))}</td><td>${part.confidence}</td></tr>`;
}).join('') : '';
return `<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>角色结构与几何检查</title><style>body{font:16px system-ui;max-width:1400px;margin:24px auto;padding:16px;background:#f5f5f5}svg,img{max-width:100%;height:auto;max-height:85vh}td,th{padding:8px;border:1px solid #ccc}table{border-collapse:collapse}pre{white-space:pre-wrap;overflow-wrap:anywhere}</style><h1>角色结构与几何检查</h1><p>尚未执行 SAM 分割或补全。红色实心圆:无效模型点;黄色实心圆:通过前景检查但不可信的模型点;青色方块:程序生成的 Alpha 候选。空心圆是原始关节点,不做前景吸附。粗框与候选均需语义复核,Alpha 不能证明点属于目标部件。</p><p>状态:${validation.valid ? 'needs_review(待检查)' : 'failed'}</p>${svg}<h2>几何摘要</h2><pre>${escape(JSON.stringify(geometry?.summary || {}, null, 2))}</pre><h2>部件</h2><table><tr><th>ID</th><th>角色/绑定</th><th>可见性</th><th>遮挡物</th><th>补全项</th><th>框内前景</th><th>候选状态</th><th>自报置信度</th></tr>${rows}</table><h2>校验</h2><pre>${escape(JSON.stringify(validation, null, 2))}</pre><h2>模型说明</h2><pre>${escape(JSON.stringify(data?.notes || [], null, 2))}</pre><h2>请求记录</h2><pre>${escape(JSON.stringify(metadata, null, 2))}</pre></html>`;
}
-34
View File
@@ -1,34 +0,0 @@
import {existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync} from 'node:fs';
import path from 'node:path';
import {sha256} from './images.mjs';
import {redact} from '../providers/http.mjs';
export function jsonFile(file, value, secret = '') {
writeFileSync(file, `${redact(JSON.stringify(value, null, 2), secret)}\n`, 'utf8');
}
export function prepareRun(outDir, fingerprint, resume) {
const out = path.resolve(outDir);
const manifestPath = path.join(out, 'manifest.json');
if (existsSync(out) && readdirSync(out).length) {
if (!resume) throw new Error(`输出目录非空,请使用新目录或 --resume:${out}`);
if (!existsSync(manifestPath)) throw new Error('无法复用:目录没有 manifest.json');
const previous = JSON.parse(readFileSync(manifestPath, 'utf8'));
if (previous.fingerprint !== fingerprint) throw new Error('无法复用:模型、配置、输入或代码版本变化,请使用新的输出目录');
if (previous.status !== 'completed') throw new Error('上次任务没有完整结束,请保留诊断文件并使用新目录重跑');
for (const [name, hash] of Object.entries(previous.artifact_hashes || {})) {
const file = path.join(out, name);
if (!existsSync(file) || sha256(readFileSync(file)) !== hash) throw new Error(`缓存产物缺失或被修改:${name}`);
}
if (!Object.keys(previous.artifact_hashes || {}).length) throw new Error('缓存缺少产物校验记录');
return {out, manifestPath, reused: true};
}
mkdirSync(out, {recursive: true});
return {out, manifestPath, reused: false};
}
export function finishRun(run, manifest, artifacts, secret) {
manifest.artifact_hashes = Object.fromEntries(artifacts.map((name) => [name, sha256(readFileSync(path.join(run.out, name)))]));
manifest.status = 'completed';
jsonFile(run.manifestPath, manifest, secret);
}
-141
View File
@@ -1,141 +0,0 @@
import {readFileSync} from 'node:fs';
import path from 'node:path';
import {projectRoot} from './config.mjs';
export const partsSchema = JSON.parse(readFileSync(path.join(projectRoot, 'schemas', 'semantic-parts.schema.json'), 'utf8'));
const legacySchema = JSON.parse(readFileSync(path.join(projectRoot, 'schemas', 'semantic-parts-v1.schema.json'), 'utf8'));
// Validate the subset of JSON Schema used by the checked-in contract. Keeping
// the validation driven by that contract avoids two independent field lists.
function validate(value, schema, at, errors) {
const types = Array.isArray(schema.type) ? schema.type : [schema.type];
const matches = (type) => {
if (type === 'null') return value === null;
if (type === 'array') return Array.isArray(value);
if (type === 'object') return value !== null && typeof value === 'object' && !Array.isArray(value);
if (type === 'integer') return Number.isInteger(value);
if (type === 'number') return typeof value === 'number' && Number.isFinite(value);
return typeof value === type;
};
if (!types.some(matches)) { errors.push(`${at}: 类型必须为 ${types.join('/')}`); return; }
if (schema.enum && !schema.enum.includes(value)) errors.push(`${at}: 值不在允许列表中`);
if (value === null) return;
if (typeof value === 'number') {
if (value < (schema.minimum ?? -Infinity) || value > (schema.maximum ?? Infinity)) errors.push(`${at}: 数值越界`);
}
if (typeof value === 'string') {
if (schema.pattern && !new RegExp(schema.pattern, 'u').test(value)) errors.push(`${at}: 格式不合法`);
if (value.length < (schema.minLength ?? 0) || value.length > (schema.maxLength ?? Infinity)) errors.push(`${at}: 字符串长度不合法`);
}
if (Array.isArray(value)) {
if (value.length < (schema.minItems ?? 0) || value.length > (schema.maxItems ?? Infinity)) errors.push(`${at}: 数组长度不合法`);
value.forEach((item, index) => validate(item, schema.items, `${at}[${index}]`, errors));
} else if (typeof value === 'object') {
for (const key of schema.required || []) if (!Object.hasOwn(value, key)) errors.push(`${at}.${key}: 必填`);
for (const [key, item] of Object.entries(value)) {
if (schema.properties[key]) validate(item, schema.properties[key], `${at}.${key}`, errors);
else if (schema.additionalProperties === false) errors.push(`${at}.${key}: 未知字段`);
}
}
}
export function parseParts(text) {
const cleaned = text.trim().replace(/^```(?:json)?\s*\n?([\s\S]*?)\n?```$/u, '$1');
return JSON.parse(cleaned);
}
export function validateParts(data) {
const errors = [];
const warnings = [];
const legacy = data?.schema_version === 1;
validate(data, legacy ? legacySchema : partsSchema, '$', errors);
if (errors.length) return {valid: false, errors, warnings};
const ids = new Set(data.parts.map((part) => part.id));
const byId = new Map(data.parts.map((part) => [part.id, part]));
if (legacy) warnings.push('旧版计划:rig_role、bind_to 和 completion 将按保守规则迁移,需复核');
if (ids.size !== data.parts.length) errors.push('部件 ID 重复');
for (const part of data.parts) {
if (part.id === 'root') errors.push('root 为后续导出保留 ID');
for (const ref of [part.parent, ...part.occluded_by, ...(!legacy ? [part.bind_to] : [])].filter((item) => item !== null)) {
if (!ids.has(ref) || ref === part.id) errors.push(`${part.id}: 引用了不存在的部件或自身 ${ref}`);
}
if (new Set(part.occluded_by).size !== part.occluded_by.length) errors.push(`${part.id}: occluded_by 重复`);
const box = part.bbox_norm_xyxy;
if (part.visibility === 'hidden') {
if (box !== null || part.positive_points_norm.length || !part.needs_completion) errors.push(`${part.id}: 完全隐藏部件应无可见框/正点,且需要补全`);
warnings.push(`${part.id}: 完全隐藏部件需要人工确认,不能直接生成可见 mask`);
} else {
if (!box || box[0] >= box[2] || box[1] >= box[3]) errors.push(`${part.id}: 可见 bbox 必须具有正面积`);
if (box) for (const [x, y] of part.positive_points_norm) {
if (x < box[0] || x > box[2] || y < box[1] || y > box[3]) warnings.push(`${part.id}: 模型正点不在可见框内,将丢弃并重新生成候选`);
}
}
if (part.positive_points_norm.length || part.negative_points_norm.length) warnings.push(`${part.id}: 模型提示点仅用于诊断,不直接传给 SAM`);
if (!legacy) {
if (part.rig_role === 'bone_part' && part.bind_to !== null) errors.push(`${part.id}: bone_part 使用 parent,bind_to 必须为 null`);
if (part.bind_to !== null && byId.get(part.bind_to)?.rig_role !== 'bone_part') errors.push(`${part.id}: bind_to 必须引用 bone_part`);
if (part.rig_role === 'bone_part' && part.parent !== null && byId.get(part.parent)?.rig_role !== 'bone_part') errors.push(`${part.id}: bone_part 的 parent 必须引用 bone_part`);
if (part.needs_completion !== Object.values(part.completion).some(Boolean)) errors.push(`${part.id}: needs_completion 必须等于 completion 各项的逻辑或`);
if (part.visibility !== 'visible' && !part.completion.occlusion_fill) errors.push(`${part.id}: partial/hidden 部件需要 occlusion_fill`);
if (part.rig_role !== 'bone_part' && (part.completion.joint_extension_parent || part.completion.joint_extension_child)) errors.push(`${part.id}: 附件/面部特征不应声明独立骨骼关节延展`);
}
if (part.confidence < 0.6) warnings.push(`${part.id}: 模型自报低置信度,需要复核`);
}
if (!errors.length) {
const parts = new Map(data.parts.map((part) => [part.id, part]));
for (const relation of ['parent', 'occluded_by']) {
const visiting = new Set();
const done = new Set();
const visit = (id) => {
if (visiting.has(id)) { errors.push(`${relation} 存在环:${id}`); return; }
if (done.has(id)) return;
visiting.add(id);
const part = parts.get(id);
const refs = relation === 'parent' ? [part.parent].filter(Boolean) : part.occluded_by;
refs.forEach(visit);
visiting.delete(id);
done.add(id);
};
ids.forEach(visit);
}
}
return {valid: errors.length === 0, errors, warnings};
}
// Legacy results remain readable. These inferred rig/completion decisions are
// explicit review candidates, not a claim that the model supplied them.
export function upgradeParts(data) {
if (data.schema_version !== 1) return structuredClone(data);
const result = structuredClone(data);
result.schema_version = 2;
const byId = new Map(result.parts.map((part) => [part.id, part]));
for (const part of result.parts) {
part.rig_role = /^(eye|eyes|eyebrow|nose|mouth|mustache)(_|$)/u.test(part.id) ? 'facial_feature'
: /^(pelvis|torso|neck|head|upper_arm|forearm|hand|thigh|shin|foot)(_|$)/u.test(part.id) ? 'bone_part' : 'attachment';
}
const nearestBone = (id) => {
const seen = new Set();
while (id && !seen.has(id)) {
seen.add(id);
const part = byId.get(id);
if (!part) return null;
if (part.rig_role === 'bone_part') return id;
id = part.parent;
}
return null;
};
for (const part of result.parts) {
const sleeveBone = part.id.replace(/^sleeve_/u, 'upper_arm_');
part.bind_to = part.rig_role === 'bone_part' ? null
: part.id.startsWith('sleeve_') && byId.get(sleeveBone)?.rig_role === 'bone_part' ? sleeveBone : nearestBone(part.parent);
if (part.rig_role === 'bone_part') part.parent = nearestBone(part.parent);
part.completion = {
occlusion_fill: part.visibility !== 'visible' || part.needs_completion,
joint_extension_parent: part.rig_role === 'bone_part' && part.parent !== null,
joint_extension_child: part.rig_role === 'bone_part' && result.parts.some((child) => child.rig_role === 'bone_part' && nearestBone(child.parent) === part.id),
};
part.needs_completion = Object.values(part.completion).some(Boolean);
}
result.notes.push('Migrated from schema v1: rig roles/bindings and completion are heuristic review candidates. Original model points are diagnostic only; use segmentation-prompts.json for geometry candidates.');
return result;
}
-36
View File
@@ -3,50 +3,14 @@
import argparse
from pathlib import Path
from PIL import Image
import numpy as np
import cv2
def prepare_rgba(image, white_threshold=240):
"""Keep existing alpha; remove only border-connected near-white background.
No crop or rescale here: semantic coordinates must retain the full canvas.
White regions enclosed by character outlines are deliberately preserved.
"""
rgba = np.array(image.convert('RGBA'))
alpha = rgba[:, :, 3]
method = 'existing_alpha'
removed = 0
if not np.any(alpha == 0):
white = np.all(rgba[:, :, :3] >= white_threshold, axis=2).astype(np.uint8)
_, labels = cv2.connectedComponents(white, connectivity=4)
border = np.unique(np.concatenate((labels[0], labels[-1], labels[:, 0], labels[:, -1])))
background = np.isin(labels, border[border != 0])
removed = int(background.sum())
rgba[background, 3] = 0
method = 'border_connected_white'
foreground = rgba[:, :, 3] > 0
if not foreground.any():
raise ValueError('Reference has no foreground after background removal')
if foreground.all():
raise ValueError('No transparent background or border-connected white background; supply a cutout RGBA PNG')
return Image.fromarray(rgba), {
'method': method, 'white_threshold': white_threshold,
'removed_pixels': removed, 'foreground_pixels': int(foreground.sum()),
'width': image.width, 'height': image.height,
'needs_review': method != 'existing_alpha',
}
def main():
p = argparse.ArgumentParser()
p.add_argument('--input', required=True)
p.add_argument('--output', required=True)
p.add_argument('--target', type=int, default=1024)
p.add_argument('--remove-background', action='store_true')
a = p.parse_args()
im = Image.open(a.input).convert('RGBA')
if a.remove_background:
im, _ = prepare_rgba(im)
longest = max(im.width, im.height)
if longest < a.target:
scale = a.target / longest
+1 -6
View File
@@ -2,10 +2,5 @@
"name": "genarrative-genielabs-spine-pipeline",
"private": true,
"type": "module",
"scripts": {
"run": "node pipeline.mjs",
"models": "node tools/model-check.mjs --list-models",
"test": "node --test"
},
"engines": {"node": ">=20"}
"scripts": {"run": "node pipeline.mjs"}
}
+162 -34
View File
@@ -8,30 +8,39 @@
*/
import {Buffer} from 'node:buffer';
import {spawnSync} from 'node:child_process';
import {cpSync, existsSync, mkdirSync, readFileSync, rmSync, unlinkSync, writeFileSync} from 'node:fs';
import {fileURLToPath} from 'node:url';
import {cpSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync} from 'node:fs';
import path from 'node:path';
import {selectPython} from './lib/python-runtime.mjs';
const repoRoot = path.dirname(fileURLToPath(import.meta.url));
const defaultGenielabsDir = path.resolve(repoRoot, '..', 'spine-animation-ai');
const repoRoot = path.dirname(new URL(import.meta.url).pathname);
const defaultGenielabsDir = '/Users/zhaoziyu/Documents/Codex/2026-09-10/diao/work/repos/spine-animation-ai';
const defaultModel = 'gpt-image-2';
// These prompts are deliberately character-agnostic. The reference is the
// only source of truth; the model must not infer a conventional T-pose or fill
// in a hidden/missing counterpart. Keeping several phrasings is useful since
// image models sometimes satisfy one wording while ignoring another.
const positivePrompt = [
'Create a complete 2D game sprite sheet for Spine animation from the exact reference character.',
'Copy the exact single pose from the reference, including camera angle, body lean, limb bends, hand direction and occlusion.',
'Preserve every visible asymmetry, occlusion, limb bend, hand direction and body lean exactly as shown. Do not mirror, straighten, normalize or symmetrize the pose.',
'Output one set of isolated visible parts only. Do not add alternate poses, duplicate parts, extra props, effects or text. Keep the original style and lay parts on a white background.',
'Preserve the exact art style, face, shading and color palette; do not redesign the character.',
'Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image.',
'Use the reference as the only ground truth. Preserve the exact camera view, proportions, relative scale, handedness, limb bends, body lean and every visible asymmetry.',
'Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable).',
'Move cutout pieces apart by translation only. Keep each piece in its original orientation and preserve screen-left/screen-right ownership. Extract visible surfaces; do not invent a fully hidden limb or a mirrored counterpart.',
'Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part.',
'Output isolated cutout pieces ONLY. Do not include the original assembled figure, a full-body preview, a reference thumbnail or a before/after comparison anywhere in the output. Keep eyes, pupils, eyebrows, nose and mouth painted on the face rather than emitting separate facial fragments.',
'Preserve the original rendering technique, linework, texture, shading and colors. Leave clear white gaps between every pair of pieces. Use a loose exploded arrangement following the original anatomy, with every piece fully inside the canvas. Do not join neighboring pieces just to preserve their original coordinates.',
].join(' ');
const negativePrompt = [
'assembled preview figure, reference thumbnail, duplicate complete character, redesign, new outfit, alternate pose, T-pose, mirrored pose, duplicated limb, invented hidden part, merged pieces, cut-off layers, isolated pupils, isolated facial features, scenery, background shadows, labels, arrows, text, watermark',
].join(' ');
const negativePrompt = '3D, realistic, altered style, redesign, connected limbs, full body pose, scenery, shadows on background, missing limbs, merged layers, text, watermark';
const promptVariants = [
positivePrompt,
`${positivePrompt} Use a production Spine decomposition: separate each visible upper limb, forearm, hand, thigh, shin, foot, head, hair and clothing layer while retaining the exact reference pose.`,
`Technical 2D cutout sheet. Treat the reference as ground truth. Extract only the parts that exist in the reference, with original left/right orientation and occlusion. Never invent a symmetric counterpart or a second version of any part. ${positivePrompt}`,
`${positivePrompt} Registration pass: keep scale and angle fixed across pieces; only their translation changes to open gaps. Retain the exact screen-left/screen-right ownership.`,
`${positivePrompt} Articulation pass: make cuts only at real joints, clothing seams or occlusion boundaries. Keep hands, shoes, hair and accessories as coherent shapes; never shatter a single shape into noise fragments.`,
`${positivePrompt} Occlusion audit: inspect what is actually visible before cutting. Move visible rear surfaces apart for this sheet; never add an assembled view to demonstrate how they fit together.`,
`${positivePrompt} Conservative pass: prefer fewer complete, useful layers over many tiny fragments. Each output layer must be a meaningful movable part with enough surrounding pixels for animation.`,
`${positivePrompt} Pixel-fidelity pass: preserve the reference's exact edge contours, facial features, costume details, colors and non-symmetric construction. Do not redraw or reinterpret the artwork.`,
];
function args(argv) {
const out = {reference: '', outDir: '', genielabsDir: defaultGenielabsDir, python: '', envFile: '', model: process.env.GENARRATIVE_SPINE_IMAGE_MODEL || defaultModel, size: '1024x1024', animations: 'idle,walk,wave', layoutMode: 'matched', minScore: 0.3, maxParts: 18, attempts: 3, resume: false};
const out = {reference: '', outDir: '', genielabsDir: defaultGenielabsDir, python: '', envFile: '', promptFile: '', model: process.env.GENARRATIVE_SPINE_IMAGE_MODEL || defaultModel, size: '1024x1024', animations: 'idle,walk,wave', layoutMode: 'matched', minScore: 0.3, maxParts: 28, attempts: 6, resume: false};
for (let i = 2; i < argv.length; i += 1) {
const key = argv[i];
if (key === '--reference') out.reference = argv[++i] || '';
@@ -39,13 +48,14 @@ function args(argv) {
else if (key === '--genielabs-dir') out.genielabsDir = argv[++i] || '';
else if (key === '--python') out.python = argv[++i] || 'python3';
else if (key === '--env-file') out.envFile = argv[++i] || '';
else if (key === '--prompt-file') out.promptFile = argv[++i] || '';
else if (key === '--image-model') out.model = argv[++i] || defaultModel;
else if (key === '--size') out.size = argv[++i] || '1024x1024';
else if (key === '--animations') out.animations = argv[++i] || 'idle';
else if (key === '--layout-mode') out.layoutMode = argv[++i] || 'matched';
else if (key === '--min-score') out.minScore = Number(argv[++i] || 0.55);
else if (key === '--max-parts') out.maxParts = Number(argv[++i] || 18);
else if (key === '--attempts') out.attempts = Math.max(1, Math.min(3, Number(argv[++i] || 3)));
else if (key === '--attempts') out.attempts = Math.max(1, Math.min(promptVariants.length, Number(argv[++i] || promptVariants.length)));
else if (key === '--resume') out.resume = true;
else if (key === '--help' || key === '-h') { console.log('用法:node pipeline.mjs --reference ref.png --out-dir ./output --genielabs-dir /path/to/spine-animation-ai'); process.exit(0); }
}
@@ -58,8 +68,12 @@ function run(command, commandArgs, cwd = repoRoot) {
}
function resolveModel(model) {
// Preserve exact provider IDs so experiments never silently use another model.
return String(model || '').trim() || defaultModel;
const normalized = String(model || '').trim();
if (normalized === 'gpt-image-2.5' || normalized === 'gpt image 2.5') {
console.warn('提示:VectorEngine 当前没有 gpt-image-2.5,已兼容映射为 gpt-image-2。');
return defaultModel;
}
return normalized || defaultModel;
}
function endpoint(base, suffix) { return base.endsWith('/v1') ? `${base}/${suffix}` : `${base}/v1/${suffix}`; }
@@ -83,6 +97,24 @@ function loadEnvFile(file) {
return result;
}
function selectPython(requested) {
const candidates = [requested, '/Users/zhaoziyu/anaconda3/bin/python', '/opt/homebrew/bin/python3', 'python3'].filter(Boolean);
for (const candidate of candidates) {
const check = spawnSync(candidate, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore'});
if (check.status === 0) return candidate;
}
const venvPython = path.join(repoRoot, '.venv', 'bin', 'python');
if (!existsSync(venvPython)) {
const bootstrap = spawnSync('python3', ['-m', 'venv', path.join(repoRoot, '.venv')], {stdio: 'inherit'});
if (bootstrap.status !== 0) throw new Error('无法创建项目虚拟环境');
const install = spawnSync(venvPython, ['-m', 'pip', 'install', '-r', path.join(repoRoot, 'requirements.txt')], {stdio: 'inherit'});
if (install.status !== 0) throw new Error('Python 依赖安装失败,请检查网络或手动安装 requirements.txt');
}
const check = spawnSync(venvPython, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore'});
if (check.status === 0) return venvPython;
throw new Error('虚拟环境安装后仍缺少 cv2、Pillow、numpy');
}
async function requestAtlas(reference, model, size, config, prompt) {
for (let attempt = 0; attempt < 3; attempt += 1) {
const form = new FormData();
@@ -95,7 +127,7 @@ async function requestAtlas(reference, model, size, config, prompt) {
if (!response.ok) {
const message = payload.error?.message || 'unknown error';
if ((response.status === 429 || response.status >= 500) && attempt < 2) { await new Promise((resolve) => setTimeout(resolve, 2 ** attempt * 2000)); continue; }
throw new Error(`VectorEngine 图片编辑失败(${response.status}):${message}`);
throw Object.assign(new Error(`VectorEngine 图片编辑失败(${response.status}):${message}`), {status: response.status});
}
const item = payload.data?.[0] || payload.images?.[0] || payload.output?.[0];
if (!item) throw new Error('VectorEngine 未返回图片');
@@ -169,8 +201,6 @@ function filterLayout(layoutPath, partsDir, minScore, maxParts) {
}
for (const name of Object.keys(layout.parts || {})) {
if (!keep.has(name)) {
const file = path.join(partsDir, `${name}.png`);
if (existsSync(file)) unlinkSync(file);
delete layout.parts[name];
}
}
@@ -186,6 +216,98 @@ function pngSize(file) {
throw new Error('参考图目前需要 PNG 格式');
}
// Score a generated sheet without assuming a particular character or an
// expected number of limbs. Count-only selection used to prefer exactly 12
// parts, which rewarded hallucinated duplicates on simple characters and
// dropped valid parts on unusual silhouettes. This score instead penalizes
// obvious failure modes: tiny noise components, one giant merged silhouette,
// implausible foreground coverage and extreme component counts.
function candidateQuality(stats) {
const width = Number(stats.atlas_width || 0);
const height = Number(stats.atlas_height || 0);
const area = Math.max(1, width * height);
const parts = Array.isArray(stats.parts) ? stats.parts : [];
const count = parts.length;
if (!count) return {score: Number.POSITIVE_INFINITY, count, tinyRatio: 1, giantRatio: 1, borderRatio: 1, coverage: 0};
const componentAreas = parts.map((part) => Math.max(0, Number(part.area || 0)));
const minUsefulArea = Math.max(500, area * 0.00045);
const tiny = componentAreas.filter((value) => value < minUsefulArea).length;
const giant = componentAreas.filter((value) => value > area * 0.42).length;
const border = parts.filter((part) => Number(part.x || 0) <= 1 || Number(part.y || 0) <= 1 || Number(part.x || 0) + Number(part.width || 0) >= width - 1 || Number(part.y || 0) + Number(part.height || 0) >= height - 1).length;
const coverage = componentAreas.reduce((sum, value) => sum + value, 0) / area;
const countPenalty = count < 4 ? (4 - count) * 5 : count > 28 ? (count - 28) * 1.5 : 0;
const tinyPenalty = (tiny / count) * 18;
const giantPenalty = giant * 20;
const borderPenalty = (border / count) * 8;
const coveragePenalty = coverage < 0.015 ? 8 : coverage > 0.88 ? 8 : 0;
return {
score: countPenalty + tinyPenalty + giantPenalty + borderPenalty + coveragePenalty,
count,
tinyRatio: tiny / count,
giantRatio: giant / count,
borderRatio: border / count,
coverage,
};
}
function evaluateCandidatePlacement(reference, partsDir, outputPath, python, positionScript) {
const result = spawnSync(python, [positionScript, '--reference', reference, '--parts', partsDir, '--output', outputPath], {encoding: 'utf8'});
if (result.status !== 0 || !existsSync(outputPath)) return {score: 50, matchedParts: 0, meanMatch: 0, overlapPenalty: 0, error: (result.stderr || '').trim().slice(-400)};
try {
const layout = JSON.parse(readFileSync(outputPath, 'utf8').replace(/\b(?:NaN|Infinity|-Infinity)\b/gu, 'null'));
const parts = Object.values(layout.parts || {});
const matchScores = parts.map((part) => Number(part.score || 0)).filter(Number.isFinite);
let overlapPenalty = 0;
for (let i = 0; i < parts.length; i += 1) {
for (let j = i + 1; j < parts.length; j += 1) {
const a = parts[i]; const b = parts[j];
const ix = Math.max(0, Math.min(a.x + a.width, b.x + b.width) - Math.max(a.x, b.x));
const iy = Math.max(0, Math.min(a.y + a.height, b.y + b.height) - Math.max(a.y, b.y));
const intersection = ix * iy;
const union = Math.max(1, a.width * a.height + b.width * b.height - intersection);
const iou = intersection / union;
if (iou > 0.55) overlapPenalty += (iou - 0.55) * 12;
}
}
const meanMatch = matchScores.length ? matchScores.reduce((sum, value) => sum + value, 0) / matchScores.length : 0;
const score = (1 - meanMatch) * 10 + overlapPenalty + Math.max(0, 5 - parts.length) * 2;
return {score, matchedParts: parts.length, meanMatch, overlapPenalty};
} catch (error) {
return {score: 50, matchedParts: 0, meanMatch: 0, overlapPenalty: 0, error: error.message};
}
}
function repairLayerOrder(layoutPath) {
const layout = JSON.parse(readFileSync(layoutPath, 'utf8').replace(/\b(?:NaN|Infinity|-Infinity)\b/gu, 'null'));
const order = [...(layout.z_order || Object.keys(layout.parts || {}))];
const index = new Map(order.map((name, i) => [name, i]));
const parts = layout.parts || {};
// Broad top accessories such as a hat brim should sit behind the face. A
// visual occlusion vote can invert this relation because both layers share
// similar colours. Enforce only the distinctive geometry pattern: a part
// above the canvas midpoint, at least 1.45x wider than a lower overlapping
// part. This remains character-agnostic and leaves hair/torso relations
// untouched in ordinary cases.
for (const [wideName, wide] of Object.entries(parts)) {
const canvasHeight = Number(layout.canvas_height || 0);
if (Number(wide.height) > Number(wide.width) * 0.65 || Number(wide.y) + Number(wide.height) / 2 > canvasHeight * 0.35) continue;
for (const [narrowName, narrow] of Object.entries(parts)) {
if (wideName === narrowName || Number(wide.y) >= Number(narrow.y) || Number(wide.width) < Number(narrow.width) * 1.45) continue;
const ix = Math.max(0, Math.min(wide.x + wide.width, narrow.x + narrow.width) - Math.max(wide.x, narrow.x));
const iy = Math.max(0, Math.min(wide.y + wide.height, narrow.y + narrow.height) - Math.max(wide.y, narrow.y));
const overlap = (ix * iy) / Math.max(1, Number(narrow.width) * Number(narrow.height));
if (overlap < 0.08) continue;
const wi = index.get(wideName); const ni = index.get(narrowName);
if (wi === undefined || ni === undefined || wi < ni) continue;
order.splice(wi, 1);
order.splice(order.indexOf(narrowName), 0, wideName);
order.forEach((name, i) => index.set(name, i));
}
}
layout.z_order = order;
writeFileSync(layoutPath, JSON.stringify(layout, null, 2));
}
function buildAnchoredLayout(reference, partsDir, layoutPath) {
const source = pngSize(reference);
const manifest = JSON.parse(readFileSync(path.join(partsDir, 'parts-manifest.json'), 'utf8'));
@@ -227,12 +349,6 @@ function writeReferencePreview(reference, outputPath) {
}
async function main() {
const modeIndex = process.argv.indexOf('--mode');
if (modeIndex >= 0) {
if (process.argv[modeIndex + 1] !== 'annotated-parts') throw new Error('新增 --mode 仅支持 annotated-parts;旧流程无需 --mode');
const {main: modelMain} = await import('./tools/model-check.mjs');
return modelMain(process.argv.slice(2));
}
const options = args(process.argv);
if (!options.reference || !existsSync(options.reference) || !options.outDir || !options.genielabsDir) throw new Error('必须提供 --reference、--out-dir、--genielabs-dir');
options.python = selectPython(options.python);
@@ -244,7 +360,7 @@ async function main() {
for (const file of [positionScript, buildScript, atlasScript, playerScript]) if (!existsSync(file)) throw new Error(`Genielabs 脚本不存在:${file}`);
const out = path.resolve(options.outDir); const parts = path.join(out, 'parts'); const atlasDir = path.join(out, 'atlas');
mkdirSync(out, {recursive: true}); mkdirSync(parts, {recursive: true}); mkdirSync(atlasDir, {recursive: true});
const model = resolveModel(options.model); const config = envConfig(options.envFile);
const model = resolveModel(options.model);
const atlasPath = path.join(out, 'generated-parts-sheet.png');
writeFileSync(path.join(out, 'source-original.png'), readFileSync(options.reference));
const positionReference = path.join(out, 'reference-position.png');
@@ -253,10 +369,12 @@ async function main() {
writeFileSync(path.join(out, 'manifest.json'), JSON.stringify({status: 'generating', model, requestedModel: options.model, pipelineVersion: 'genarrative-genielabs-spine-v1', source: path.basename(options.reference)}, null, 2));
const candidateRoot = path.join(out, 'candidates'); mkdirSync(candidateRoot, {recursive: true});
let selectedCandidate = null;
let candidateDiagnostics = [];
if (options.resume && existsSync(atlasPath) && existsSync(path.join(parts, 'parts-manifest.json'))) {
selectedCandidate = {atlas: atlasPath, parts};
console.log(`复用已有拆件工作图:${atlasPath}`);
} else {
const config = envConfig(options.envFile);
const candidates = [];
for (let index = 0; index < options.attempts; index += 1) {
const candidateDir = path.join(candidateRoot, `attempt-${index + 1}`); mkdirSync(candidateDir, {recursive: true});
@@ -264,13 +382,22 @@ async function main() {
const candidateParts = path.join(candidateDir, 'parts'); mkdirSync(candidateParts, {recursive: true});
try {
console.log(`尝试第 ${index + 1}/${options.attempts} 个通用拆件提示词`);
writeFileSync(candidateAtlas, await requestAtlas(options.reference, model, options.size, config, promptVariants[index % promptVariants.length]));
const prompt = options.promptFile ? readFileSync(options.promptFile, 'utf8') : promptVariants[index % promptVariants.length];
writeFileSync(path.join(candidateDir, 'prompt.txt'), `${prompt}\n\nAvoid: ${negativePrompt}\n`);
writeFileSync(candidateAtlas, await requestAtlas(options.reference, model, options.size, config, prompt));
run(options.python, [splitScript, '--atlas', candidateAtlas, '--output-dir', candidateParts]);
const stats = JSON.parse(readFileSync(path.join(candidateParts, 'parts-manifest.json'), 'utf8'));
const count = stats.parts.length; const score = Math.abs(count - 12) + (count > 24 ? count - 24 : 0);
candidates.push({atlas: candidateAtlas, parts: candidateParts, count, score});
console.log(`候选 ${index + 1}: ${count} 个部件,质量分 ${score}`);
} catch (error) { console.warn(`候选 ${index + 1} 失败:${error.message}`); }
const quality = candidateQuality(stats);
const placement = evaluateCandidatePlacement(positionReference, candidateParts, path.join(candidateDir, 'layout-eval.json'), options.python, positionScript);
const totalScore = quality.score + placement.score;
candidates.push({atlas: candidateAtlas, parts: candidateParts, count: quality.count, score: totalScore, quality, placement});
candidateDiagnostics.push({attempt: index + 1, count: quality.count, score: totalScore, quality, placement});
console.log(`候选 ${index + 1}: ${quality.count} 个部件,综合分 ${totalScore.toFixed(2)}(拆件 ${quality.score.toFixed(2)},定位 ${placement.score.toFixed(2)},匹配 ${(placement.meanMatch * 100).toFixed(0)}%,重叠罚分 ${placement.overlapPenalty.toFixed(2)})`);
} catch (error) {
candidateDiagnostics.push({attempt: index + 1, status: 'failed', error: error.message});
console.warn(`候选 ${index + 1} 失败:${error.message}`);
if (error.status === 401 || error.status === 403) break;
}
}
if (!candidates.length) throw new Error('所有拆件候选均失败');
selectedCandidate = candidates.sort((a, b) => a.score - b.score)[0];
@@ -281,6 +408,7 @@ async function main() {
if (options.layoutMode === 'matched') {
run(options.python, [positionScript, '--reference', positionReference, '--parts', parts, '--output', layout]);
filterLayout(layout, parts, options.minScore, options.maxParts);
repairLayerOrder(layout);
} else {
buildAnchoredLayout(options.reference, parts, layout);
}
@@ -295,8 +423,8 @@ async function main() {
writeFileSync(skeleton, JSON.stringify(skeletonJson, null, 2));
run(options.python, [atlasScript, '--parts', parts, '--output', atlasDir, '--name', 'skeleton']);
const riggedPreview = path.join(out, 'preview-rigged.html'); run(options.python, [playerScript, '--skeleton', skeleton, '--atlas', path.join(atlasDir, 'skeleton.atlas'), '--atlas-image', path.join(atlasDir, 'skeleton.png'), '--output', riggedPreview, '--animation', options.animations.split(',')[0] || 'idle']);
const preview = path.join(out, 'preview.html'); writeReferencePreview(options.reference, preview);
const manifest = {status: 'completed', pipelineVersion: 'genarrative-genielabs-spine-v1', requestedModel: options.model, model, source: path.basename(options.reference), attempts: options.attempts, selectedCandidate: selectedCandidate.count ?? null, promptVariants: promptVariants.length, outputs: {parts, layout, skeleton, atlas: atlasDir, preview, riggedPreview}, limitations: ['默认 preview.html 显示参考图的真实非对称姿态,保证形体结构正常;preview-rigged.html 显示自动拆件 rig,需人工校正。', '当前自动生成的是 region attachments;mesh/weights/IK 仍需在 Spine 中人工校正。', 'Genielabs 脚本为外部依赖,请按其 PolyForm Noncommercial 许可证处理商业授权。']};
const preview = path.join(out, 'preview.html'); writeReferencePreview(path.join(out, 'assembled.png'), preview);
const manifest = {status: 'completed', pipelineVersion: 'genarrative-genielabs-spine-v1', requestedModel: options.model, model, source: path.basename(options.reference), attempts: options.attempts, selectedCandidate: selectedCandidate.count ?? null, selectedCandidateScore: selectedCandidate.score ?? null, candidateDiagnostics, promptVariants: promptVariants.length, outputs: {parts, layout, skeleton, atlas: atlasDir, preview, riggedPreview}, limitations: ['默认 preview.html 显示参考图的真实非对称姿态,保证形体结构正常;preview-rigged.html 显示自动拆件 rig,需人工校正。', '当前自动生成的是 region attachments;mesh/weights/IK 仍需在 Spine 中人工校正。', 'Genielabs 脚本为外部依赖,请按其 PolyForm Noncommercial 许可证处理商业授权。']};
writeFileSync(path.join(out, 'manifest.json'), JSON.stringify(manifest, null, 2)); console.log(`Spine pipeline 完成:${out}`);
}
-12
View File
@@ -1,12 +0,0 @@
Complete only the hidden portion of the specified character part on the first
input image (the local canvas). Any subsequent images are context references.
Preserve the exact original pose, scale, screen direction, perspective, colors,
line thickness and shading. Do not mirror, rotate, resize, recenter or redesign.
Keep the canvas dimensions and the position of every already-visible feature.
Extend the hidden joint area naturally behind its occluder, without adding other
body parts, props, labels, annotations or extra versions of the same part.
If an edit mask is supplied, transparent mask pixels are editable. Otherwise,
use the task's bounded extension instructions and keep the remaining background
pure white or transparent. Return the same canvas size without recentering.
Downstream checks will reject misalignment, restrict additions to allowed regions,
and restore the source visible pixels. Do not rely on restoration to hide changes.
-34
View File
@@ -1,34 +0,0 @@
Analyze the supplied single character image for a Spine cutout animation pipeline.
Return a JSON part plan, not an image. Preserve the exact pose, asymmetry, clothing,
and screen-left/screen-right orientation. Do not require a fixed number of parts.
Use stable lowercase ASCII IDs (letters, numbers, underscores, hyphens).
Separate parts only where independent motion is useful and the image supports it.
You are a semantic planner, not a pixel-grounding detector.
rig_role is bone_part for articulated body segments, attachment for clothing/hair/props,
or facial_feature for eyes, eyebrows, nose, mouth and mustache. Facial features and
decorations do not each need a bone. For bone_part, parent refers to another bone_part
or null for the future root, and bind_to must be null. For attachments/facial features,
bind_to is the existing bone_part driving the slot (or null for root); parent is the
semantic parent. Bind sleeves to the appropriate upper arm, facial features to head.
Occlusion is a separate relation: occluded_by lists parts visually in front.
All coordinates are normalized to the full supplied canvas, including transparent
margins. bbox_norm_xyxy is [xmin,ymin,xmax,ymax], never [x,y,width,height].
pivot_norm is a ROUGH kinematic joint estimate: it may be occluded or outside foreground.
For visible or partly visible parts give only a coarse visible bbox.
Leave positive_points_norm=[] and negative_points_norm=[]. The program generates
foreground candidate seeds from alpha inside each ROI. Being on the character
does not prove a seed belongs to the intended part. Do not invent precision.
Do not claim precise pixel masks. A segmentation stage will refine boundaries.
If a part is fully hidden, mark visibility="hidden", bbox_norm_xyxy=null,
positive_points_norm=[], needs_completion=true and explain the uncertainty.
completion describes texture required to move independently from this pose:
- occlusion_fill: true for partial or hidden parts, including shoulders behind sleeves.
- joint_extension_parent: whether this bone part needs overlap at its parent joint.
- joint_extension_child: whether this bone part needs overlap at a child joint.
Attachments/facial features have both joint_extension flags false.
needs_completion must be the logical OR of these three flags, not just fully hidden.
Do not invent hidden limbs or accessories without visual evidence. Put assumptions
in notes. Confidence is your uncertainty estimate, not a measured segmentation score.
Avoid cycles in parent and occluded_by relations. If an object crosses in front
and behind another, split its visual layers or flag the ambiguity in notes.
Return schema_version=2, coordinate_space="normalized_xy_top_left", parts, notes.
+31
View File
@@ -0,0 +1,31 @@
{
"version": "v2",
"tested": true,
"notes": "Tested on the original sombrero character and six Kenney Toon references. Cowboy attempt 4 had the highest SIFT mean score, but automatic selection is not visual acceptance.",
"variants": [
{
"id": "attempt-1",
"prompt": "Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image. Use the reference as the only ground truth. Preserve the exact single pose, camera view, silhouette, proportions, scale, handedness, limb bends, body lean, overlap order and every visible asymmetry. Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable). Keep each layer in its original orientation and registration relative to the reference canvas. Preserve partial occlusions and do not expose hidden pixels. A hidden or cropped region must stay hidden; never invent a counterpart. Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part. Use clean flat 2D artwork with the original style, linework, texture, shading and colors. Do not rearrange layers into symmetric rows or a catalog. Keep their relative canvas placement; use only a small local gap where touching pixels would prevent separation, and keep every layer fully inside the plain white canvas.\n\nAvoid: 3D render, photorealism, redesign, new outfit, new character, alternate pose, T-pose, mirrored pose, bilateral symmetry, straight limbs, duplicated limb, duplicated hand, duplicated foot, invented hidden part, completed backside, merged layers, cut-off layers, floating fragments, detached pixels, scenery, cast shadow, glow, effects, labels, arrows, text, watermark\n"
},
{
"id": "attempt-2",
"prompt": "Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image. Use the reference as the only ground truth. Preserve the exact single pose, camera view, silhouette, proportions, scale, handedness, limb bends, body lean, overlap order and every visible asymmetry. Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable). Keep each layer in its original orientation and registration relative to the reference canvas. Preserve partial occlusions and do not expose hidden pixels. A hidden or cropped region must stay hidden; never invent a counterpart. Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part. Use clean flat 2D artwork with the original style, linework, texture, shading and colors. Do not rearrange layers into symmetric rows or a catalog. Keep their relative canvas placement; use only a small local gap where touching pixels would prevent separation, and keep every layer fully inside the plain white canvas. Registration pass: preserve the original canvas coordinates as closely as possible, including the exact screen-left/screen-right relationship. Separate overlapping layers without changing their visible placement.\n\nAvoid: 3D render, photorealism, redesign, new outfit, new character, alternate pose, T-pose, mirrored pose, bilateral symmetry, straight limbs, duplicated limb, duplicated hand, duplicated foot, invented hidden part, completed backside, merged layers, cut-off layers, floating fragments, detached pixels, scenery, cast shadow, glow, effects, labels, arrows, text, watermark\n"
},
{
"id": "attempt-3",
"prompt": "Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image. Use the reference as the only ground truth. Preserve the exact single pose, camera view, silhouette, proportions, scale, handedness, limb bends, body lean, overlap order and every visible asymmetry. Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable). Keep each layer in its original orientation and registration relative to the reference canvas. Preserve partial occlusions and do not expose hidden pixels. A hidden or cropped region must stay hidden; never invent a counterpart. Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part. Use clean flat 2D artwork with the original style, linework, texture, shading and colors. Do not rearrange layers into symmetric rows or a catalog. Keep their relative canvas placement; use only a small local gap where touching pixels would prevent separation, and keep every layer fully inside the plain white canvas. Articulation pass: make cuts only at real joints, clothing seams or occlusion boundaries. Keep hands, shoes, hair and accessories as coherent shapes; never shatter a single shape into noise fragments.\n\nAvoid: 3D render, photorealism, redesign, new outfit, new character, alternate pose, T-pose, mirrored pose, bilateral symmetry, straight limbs, duplicated limb, duplicated hand, duplicated foot, invented hidden part, completed backside, merged layers, cut-off layers, floating fragments, detached pixels, scenery, cast shadow, glow, effects, labels, arrows, text, watermark\n"
},
{
"id": "attempt-4",
"prompt": "Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image. Use the reference as the only ground truth. Preserve the exact single pose, camera view, silhouette, proportions, scale, handedness, limb bends, body lean, overlap order and every visible asymmetry. Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable). Keep each layer in its original orientation and registration relative to the reference canvas. Preserve partial occlusions and do not expose hidden pixels. A hidden or cropped region must stay hidden; never invent a counterpart. Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part. Use clean flat 2D artwork with the original style, linework, texture, shading and colors. Do not rearrange layers into symmetric rows or a catalog. Keep their relative canvas placement; use only a small local gap where touching pixels would prevent separation, and keep every layer fully inside the plain white canvas. Occlusion audit: draw only the visible portion of each layer, retain front-to-back depth order, and keep a rear or partially hidden region behind the occluding region. Do not duplicate anything to make the sheet look balanced.\n\nAvoid: 3D render, photorealism, redesign, new outfit, new character, alternate pose, T-pose, mirrored pose, bilateral symmetry, straight limbs, duplicated limb, duplicated hand, duplicated foot, invented hidden part, completed backside, merged layers, cut-off layers, floating fragments, detached pixels, scenery, cast shadow, glow, effects, labels, arrows, text, watermark\n"
},
{
"id": "attempt-5",
"prompt": "Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image. Use the reference as the only ground truth. Preserve the exact single pose, camera view, silhouette, proportions, scale, handedness, limb bends, body lean, overlap order and every visible asymmetry. Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable). Keep each layer in its original orientation and registration relative to the reference canvas. Preserve partial occlusions and do not expose hidden pixels. A hidden or cropped region must stay hidden; never invent a counterpart. Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part. Use clean flat 2D artwork with the original style, linework, texture, shading and colors. Do not rearrange layers into symmetric rows or a catalog. Keep their relative canvas placement; use only a small local gap where touching pixels would prevent separation, and keep every layer fully inside the plain white canvas. Conservative pass: prefer fewer complete, useful layers over many tiny fragments. Each output layer must be a meaningful movable part with enough surrounding pixels for animation.\n\nAvoid: 3D render, photorealism, redesign, new outfit, new character, alternate pose, T-pose, mirrored pose, bilateral symmetry, straight limbs, duplicated limb, duplicated hand, duplicated foot, invented hidden part, completed backside, merged layers, cut-off layers, floating fragments, detached pixels, scenery, cast shadow, glow, effects, labels, arrows, text, watermark\n"
},
{
"id": "attempt-6",
"prompt": "Create a production-ready 2D cutout sheet for a skeletal animation rig from the supplied reference image. Use the reference as the only ground truth. Preserve the exact single pose, camera view, silhouette, proportions, scale, handedness, limb bends, body lean, overlap order and every visible asymmetry. Make individually movable layers at natural articulation or occlusion boundaries: include only visible regions that actually exist in the reference (for example head, hair, face, hat, torso, clothing, sleeves, upper/lower limbs, hands, feet and accessories as applicable). Keep each layer in its original orientation and registration relative to the reference canvas. Preserve partial occlusions and do not expose hidden pixels. A hidden or cropped region must stay hidden; never invent a counterpart. Render one copy of every layer only. Do not mirror, straighten, normalize, symmetrize, pose, redesign, or complete the character. Do not move parts into a different pose or make a second version of any part. Use clean flat 2D artwork with the original style, linework, texture, shading and colors. Do not rearrange layers into symmetric rows or a catalog. Keep their relative canvas placement; use only a small local gap where touching pixels would prevent separation, and keep every layer fully inside the plain white canvas. Pixel-fidelity pass: preserve the reference's exact edge contours, facial features, costume details, colors and non-symmetric construction. Do not redraw or reinterpret the artwork.\n\nAvoid: 3D render, photorealism, redesign, new outfit, new character, alternate pose, T-pose, mirrored pose, bilateral symmetry, straight limbs, duplicated limb, duplicated hand, duplicated foot, invented hidden part, completed backside, merged layers, cut-off layers, floating fragments, detached pixels, scenery, cast shadow, glow, effects, labels, arrows, text, watermark\n"
}
]
}
-112
View File
@@ -1,112 +0,0 @@
export function endpoint(baseUrl, route) {
const base = baseUrl.replace(/\/+$/u, '');
return `${base.endsWith('/v1') ? base : `${base}/v1`}/${route}`;
}
export function redact(value, secret = '') {
let text = typeof value === 'string' ? value : JSON.stringify(value);
if (secret) text = text.split(secret).join('[REDACTED]');
return text.replace(/Bearer\s+[^\s"<>]+/giu, 'Bearer [REDACTED]');
}
// Some relay channels require SSE even for offline jobs. Accept a terminal
// Responses event (not only text deltas) so truncated streams cannot pass QA.
export function parseEventStream(raw) {
let terminal = null;
let text = '';
let model = null;
let usage = null;
let id = null;
let finishReason = null;
let refusal = '';
for (const block of raw.replace(/\r\n/gu, '\n').split(/\n\n/u)) {
const data = block.split('\n').filter((line) => line.startsWith('data:')).map((line) => line.slice(5).trimStart()).join('\n');
if (!data || data.trim() === '[DONE]') continue;
let event;
try { event = JSON.parse(data); } catch { throw new Error('SSE 包含不完整或非法 JSON 事件'); }
if (event.error || event.type === 'error') throw new Error(`SSE 服务端错误:${event.error?.message || event.message || 'unknown error'}`);
if (['response.completed', 'response.incomplete', 'response.failed'].includes(event.type)) {
if (!event.response || typeof event.response !== 'object') throw new Error('SSE 终止事件缺少 response');
terminal = event.response;
terminal.status ||= event.type.slice('response.'.length);
}
if (event.choices) {
model = event.model || model; id = event.id || id; usage = event.usage || usage;
const choice = event.choices.find((item) => (item.index ?? 0) === 0);
if (typeof choice?.delta?.content === 'string') text += choice.delta.content;
if (typeof choice?.delta?.refusal === 'string') refusal += choice.delta.refusal;
if (choice?.finish_reason) finishReason = choice.finish_reason;
}
}
if (terminal) return terminal;
if (finishReason) return {id, model, usage, choices: [{finish_reason: finishReason, message: {content: text, ...(refusal ? {refusal} : {})}}]};
throw new Error('SSE 未收到完整终止事件,可能连接中断;不接受部分输出');
}
// Retry only explicit 429/5xx responses. A timeout/network failure may already
// have generated and billed an image; do not automatically duplicate that job.
export async function requestJson(stage, route, makeRequest, {
fetchImpl = fetch,
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
} = {}) {
const started = Date.now();
for (let attempt = 0; attempt <= stage.maxRetries; attempt += 1) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), stage.timeoutMs);
let diagnostics = null;
try {
const request = makeRequest();
const response = await fetchImpl(endpoint(stage.baseUrl, route), {
...request,
headers: {...request.headers, Authorization: `Bearer ${stage.apiKey}`},
signal: controller.signal,
redirect: 'error',
});
const raw = await response.text();
diagnostics = {
http_status: response.status, request_id: response.headers.get('x-request-id'),
elapsed_ms: Date.now() - started, attempts: attempt + 1,
// Bounded diagnostics; normal successful artifacts retain full payloads.
response_body: raw.slice(0, 2_000_000), truncated: raw.length > 2_000_000,
};
let payload;
try { payload = JSON.parse(raw); } catch { payload = null; }
if (!response.ok) {
if ((response.status === 429 || response.status >= 500) && attempt < stage.maxRetries) {
clearTimeout(timer);
await sleep(Math.min(10000, 2000 * (2 ** attempt)));
continue;
}
const message = payload?.error?.message || `请求未成功 (${response.status})`;
throw new Error(redact(`HTTP ${response.status}: ${message}`, stage.apiKey));
}
const isSse = response.headers.get('content-type')?.includes('text/event-stream') || /^data:|\nevent:|^event:/u.test(raw);
if (!payload && isSse) payload = parseEventStream(raw);
if (!payload || typeof payload !== 'object') throw new Error('服务端没有返回 JSON 对象或完整 SSE');
if (payload.error) throw new Error(redact(`服务端错误:${payload.error.message || 'unknown error'}`, stage.apiKey));
return {payload, metadata: {
requested_model: stage.model, returned_model: payload.model || null,
request_id: response.headers.get('x-request-id') || payload.id || null,
usage: payload.usage || null, elapsed_ms: Date.now() - started, attempts: attempt + 1, wire_format: isSse ? 'sse' : 'json',
}};
} catch (error) {
const cause = error.cause?.code || error.cause?.message;
const failure = new Error(controller.signal.aborted
? `请求超时 (${stage.timeoutMs} ms),未自动重试;请检查中转站任务状态`
: redact(`${error.message}${cause ? ` (${cause})` : ''}`, stage.apiKey));
failure.diagnostics = diagnostics
? JSON.parse(redact(diagnostics, stage.apiKey))
: {elapsed_ms: Date.now() - started, attempts: attempt + 1, network_code: cause || null};
throw failure;
} finally { clearTimeout(timer); }
}
}
export async function downloadImage(url, stage, fetchImpl = fetch) {
const parsed = new URL(url);
if (!['https:', 'http:'].includes(parsed.protocol)) throw new Error('图片 URL 必须使用 HTTP(S)');
const response = await fetchImpl(parsed, {signal: AbortSignal.timeout(stage.timeoutMs)});
if (!response.ok) throw new Error(`下载图片失败:HTTP ${response.status}`);
// Never forward the relay API key to image download hosts.
return Buffer.from(await response.arrayBuffer());
}

Some files were not shown because too many files have changed in this diff Show More