增加几何校验
This commit is contained in:
Ukawka
@@ -0,0 +1,19 @@
|
||||
# 项目修改约定
|
||||
|
||||
本文件适用于 `AI_spine/` 及其所有子目录。每次修改代码、文档、依赖或运行测试前，都必须阅读并遵守。
|
||||
|
||||
## Python 环境
|
||||
|
||||
- 本项目固定使用名为 `ai-spine` 的 Conda 环境。
|
||||
- 不创建、不使用 `.venv`、`venv` 或其他 Python 虚拟环境；不把依赖安装到 Conda base 或系统 Python。
|
||||
- 交互式终端先执行 `conda activate ai-spine`；自动化命令使用 `conda run -n ai-spine ...`。
|
||||
- Python 依赖统一记录在 `requirements.txt`。安装命令使用 `conda run -n ai-spine python -m pip install -r requirements.txt`，避免裸 `pip` 指向错误环境。
|
||||
- Node 管线调用 Python 时也必须使用此 Conda 环境。`--python` 只能指向 `ai-spine` 环境中的解释器。
|
||||
- 不在代码中硬编码本机 Conda 安装路径，不自动创建环境或自动安装依赖。缺少环境或依赖时给出明确的安装/激活命令。
|
||||
|
||||
## 修改与验证
|
||||
|
||||
- 修改环境选择、依赖或启动方式时，同步更新 README、相关文档和 CLI 帮助，保持一致。
|
||||
- 测试在 `ai-spine` 环境下执行：`conda run -n ai-spine npm test`。不要为了绕过依赖问题临时新建环境。
|
||||
- 验证结果必须如实记录；缺少依赖导致未运行的检查，不得声称通过。
|
||||
- 保留用户已有修改和实验产物，不覆盖旧实验输出。
|
||||
@@ -4,8 +4,8 @@
|
||||
|
||||
当前正在实现的新管线分为三个核心阶段：
|
||||
|
||||
1. 视觉 LLM 理解角色结构，输出部件语义、层级、遮挡关系、粗框和分割提示点。
|
||||
2. 使用 SAM 根据提示点生成精确 mask，并从原图提取可见像素。
|
||||
1. 视觉 LLM 输出语义、层级、骨骼/附件分类、遮挡、粗框和粗关节；程序用 Alpha 检查 ROI 并生成候选种子。
|
||||
2. 使用 SAM 根据语义、粗框和程序候选生成 mask，经语义/几何检查后从原图提取可见像素。
|
||||
3. 使用图片编辑模型补全被遮挡的部件，再进行几何验收和 Spine 导出。
|
||||
|
||||
目前第一阶段已经可以完整运行。SAM 3.1 checkpoint 路径和图片编辑适配器已经接入配置，但 SAM 推理、补全结果合并及新管线的 Spine 导出尚未完成。
|
||||
@@ -14,6 +14,7 @@
|
||||
|
||||
## 文档
|
||||
|
||||
- [项目修改约定](AGENTS.md)：每次修改都必须遵守，Python 固定使用 `ai-spine` Conda 环境。
|
||||
- [第一阶段：视觉 LLM 角色结构理解](docs/1-understanding-stage.md)：执行流程、代码对应关系、数据格式和输出文件。
|
||||
- [模型阶段配置与实验说明](docs/model-stages.md)：模型切换、协议适配、图片编辑探针、缓存和错误处理。
|
||||
|
||||
@@ -21,11 +22,11 @@
|
||||
|
||||
### Node.js
|
||||
|
||||
第一阶段要求 Node.js 20 或更高版本，不需要 Python，也没有额外 npm 运行时依赖。
|
||||
第一阶段要求 Node.js 20 或更高版本，以及下方 Python 几何依赖；没有额外 npm 运行时依赖。
|
||||
|
||||
```powershell
|
||||
node --version
|
||||
npm test
|
||||
conda run -n ai-spine npm test
|
||||
```
|
||||
|
||||
### 中转站配置
|
||||
@@ -54,27 +55,40 @@ SAM_MODEL_DIR=F:/Study/sam3.1
|
||||
|
||||
### Python 与外部项目
|
||||
|
||||
运行第一阶段不需要 Python。
|
||||
本项目固定使用 **Conda 环境 `ai-spine`**，不使用或创建 `.venv`。第一阶段的 Alpha 校验需要 Python 3、`Pillow`、`numpy`、`opencv-python`。
|
||||
|
||||
只有旧版拆件和 Spine 导出流程需要：
|
||||
执行项目命令前先 `conda activate ai-spine`。Node 管线会检查 Python 属于该 Conda 环境；`--python` 也只能指定该环境的解释器。未激活环境或缺少依赖时会报错，不会自动创建环境或安装依赖。
|
||||
|
||||
旧版拆件和 Spine 导出还需要：
|
||||
|
||||
- Python 3；
|
||||
- `Pillow`、`numpy`、`opencv-python`；
|
||||
- 相邻目录中的 `spine-animation-ai` 项目。
|
||||
|
||||
安装旧流程依赖：
|
||||
在 `AI_spine` 目录安装缺失的 Python 依赖（已满足版本要求的包会保留）：
|
||||
|
||||
```powershell
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
conda activate ai-spine
|
||||
python -m pip install -r .\requirements.txt
|
||||
```
|
||||
|
||||
不激活终端也可明确指定安装目标：
|
||||
|
||||
```powershell
|
||||
conda run -n ai-spine python -m pip install -r .\requirements.txt
|
||||
```
|
||||
|
||||
检查解释器和依赖：
|
||||
|
||||
```powershell
|
||||
conda run -n ai-spine python -c "import sys, cv2, PIL, numpy; print(sys.executable); print('Dependencies OK')"
|
||||
```
|
||||
|
||||
`spine-animation-ai` 是外部只读依赖，使用时需要遵守其 PolyForm Noncommercial 许可证。
|
||||
|
||||
## 常用命令
|
||||
|
||||
以下命令默认在 `AI_spine` 目录运行。
|
||||
以下命令默认在 `AI_spine` 目录、已激活的 `ai-spine` Conda 环境中运行。自动化可使用 `conda run -n ai-spine node ...`。
|
||||
|
||||
### 查询中转站模型
|
||||
|
||||
@@ -108,10 +122,25 @@ node .\pipeline.mjs --mode annotated-parts `
|
||||
重点检查输出目录中的：
|
||||
|
||||
- `report.html`：人工复核页面；
|
||||
- `semantic-parts.json`：后续 SAM 阶段的正式输入；
|
||||
- `semantic-parts.json`：v2 语义计划，包含 `rig_role`、`bind_to`、三项 `completion`；
|
||||
- `geometry-validation.json`：原始点诊断、框内前景比例、候选及阻塞原因；
|
||||
- `segmentation-prompts.json`：程序生成的 SAM 候选提示；模型原始正负点不会传入；
|
||||
- `validation.json`：本地结构校验结果；
|
||||
- `manifest.json`：模型、输入、代码版本和产物记录。
|
||||
|
||||
输入保留画布尺寸。已有透明背景直接保留 Alpha；不透明白底只移除与边界连通的近白区域。其他背景需先提供抠图 PNG。候选来自粗框内前景的距离变换，不能保证属于目标部件，仍需语义复核。关节点不会吸附到前景。
|
||||
|
||||
已有结果可离线检查，无需再次请求模型：
|
||||
|
||||
```powershell
|
||||
node .\tools\model-check.mjs `
|
||||
--reference .\output\sombrero-understand-deepseek-flash\source.png `
|
||||
--parts-file .\output\sombrero-understand-deepseek-flash\semantic-parts.json `
|
||||
--out-dir .\output\sombrero-grounding-review
|
||||
```
|
||||
|
||||
旧版 v1 计划会保守迁移到 v2，迁移推断写入 notes，仍需复核；原文件不覆盖。
|
||||
|
||||
### 使用 GPT-6 Astra 做对照实验
|
||||
|
||||
```powershell
|
||||
@@ -168,7 +197,7 @@ node .\pipeline.mjs `
|
||||
### 运行测试
|
||||
|
||||
```powershell
|
||||
npm test
|
||||
conda run -n ai-spine npm test
|
||||
```
|
||||
|
||||
测试使用本地 mock HTTP 服务或注入响应，不调用真实付费模型。
|
||||
|
||||
@@ -13,7 +13,8 @@
|
||||
- 部件之间的运动父子关系；
|
||||
- 部件之间的视觉遮挡关系；
|
||||
- 可见、部分可见或完全隐藏状态；
|
||||
- 可见区域粗框、预计关节位置和 SAM 提示点；
|
||||
- 可见区域粗框、预计关节位置，骨骼/附件角色与绑定目标；
|
||||
- 程序用 Alpha 校验后另行输出的 SAM 候选点；
|
||||
- 是否需要在第三阶段补全；
|
||||
- 模型对判断结果的自报置信度。
|
||||
|
||||
@@ -22,6 +23,8 @@
|
||||
~~~text
|
||||
角色原图
|
||||
↓
|
||||
保留 Alpha 或去除边界连通白底（不缩放裁剪）
|
||||
↓
|
||||
读取模型配置、提示词和输出 Schema
|
||||
↓
|
||||
构造视觉 LLM 请求
|
||||
@@ -34,6 +37,8 @@ Schema 校验和部件关系校验
|
||||
↓
|
||||
semantic-parts.json
|
||||
↓
|
||||
Alpha / ROI 校验、距离变换候选点、segmentation-prompts.json
|
||||
↓
|
||||
生成 annotated.svg 和 report.html
|
||||
↓
|
||||
人工复核后交给 SAM 分割阶段
|
||||
@@ -97,7 +102,9 @@ Shell 环境变量优先于 <code>.env</code>。模型 ID 会原样传给中转
|
||||
- SHA-256；
|
||||
- 模型请求使用的 Base64 Data URL。
|
||||
|
||||
本阶段要求参考图为 PNG。实际提交给模型的输入会原样复制为输出目录中的 <code>source.png</code>，不缩放、不裁剪。
|
||||
本阶段要求参考图为 PNG。原始输入保存为 <code>source-original.png</code>，模型实际看到的预处理 RGBA 保存为 <code>source.png</code>，不缩放、不裁剪。已有透明背景时保留原像素；不透明白底使用连通域仅去除与画布边界相连、RGB 各通道均 ≥240 的区域。其他背景或空前景会在请求模型前失败，需先提供抠图 PNG。去白底存在白色主体接触边界的局限，预处理记录保存在 <code>preprocessing.json</code>。
|
||||
|
||||
这一层调用 <code>normalize_reference.py</code> 和 <code>geometry_grounding.py</code>，固定使用 Conda 环境 <code>ai-spine</code> 及 requirements.txt 中的依赖。先执行 <code>conda activate ai-spine</code>；自动化使用 <code>conda run -n ai-spine node ...</code>。--python 也只能指定该环境的解释器，不创建其他环境。每次修改须遵守 [AGENTS.md](../AGENTS.md)。
|
||||
|
||||
## 5. 理解提示词
|
||||
|
||||
@@ -107,7 +114,7 @@ Shell 环境变量优先于 <code>.env</code>。模型 ID 会原样传给中转
|
||||
- 仅在独立运动有意义且图像支持时拆分部件；
|
||||
- 区分运动父子关系和视觉遮挡关系；
|
||||
- 使用完整画布的归一化坐标；
|
||||
- 输出粗框、预计关节和 SAM 提示点；
|
||||
- 输出语义、粗框、预计关节和 rig_role/bind_to；正负提示点留空；
|
||||
- 不把粗框声称为像素级 mask；
|
||||
- 不凭空创造没有视觉证据的隐藏肢体或附件；
|
||||
- 将不确定性和假设写入 <code>notes</code>。
|
||||
@@ -120,7 +127,7 @@ Shell 环境变量优先于 <code>.env</code>。模型 ID 会原样传给中转
|
||||
|
||||
~~~json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"schema_version": 2,
|
||||
"coordinate_space": "normalized_xy_top_left",
|
||||
"parts": [],
|
||||
"notes": []
|
||||
@@ -134,13 +141,16 @@ Shell 环境变量优先于 <code>.env</code>。模型 ID 会原样传给中转
|
||||
| <code>id</code> | 稳定、唯一的小写 ASCII 部件 ID |
|
||||
| <code>semantic</code> | 部件语义说明 |
|
||||
| <code>parent</code> | 运动层级中的父部件；<code>null</code> 表示未来挂到 root |
|
||||
| <code>rig_role</code> | bone_part / attachment / facial_feature，附件不自动创建骨骼 |
|
||||
| <code>bind_to</code> | 附件绑定的 bone_part；骨骼部件使用 parent，bind_to 为 null |
|
||||
| <code>visibility</code> | <code>visible</code>、<code>partial</code> 或 <code>hidden</code> |
|
||||
| <code>bbox_norm_xyxy</code> | 可见区域粗框 <code>[xmin, ymin, xmax, ymax]</code> |
|
||||
| <code>pivot_norm</code> | 预计的关节或连接点 |
|
||||
| <code>positive_points_norm</code> | 后续 SAM 的正提示点 |
|
||||
| <code>negative_points_norm</code> | 后续 SAM 的负提示点 |
|
||||
| <code>positive_points_norm</code> | 模型粗估点，仅为旧版诊断兼容保留，新请求留空 |
|
||||
| <code>negative_points_norm</code> | 模型负点，仅保留作记录，不传给 SAM |
|
||||
| <code>occluded_by</code> | 视觉上遮挡当前部件的其他部件 |
|
||||
| <code>needs_completion</code> | 是否需要补全不可见区域 |
|
||||
| <code>completion</code> | occlusion_fill / joint_extension_parent / joint_extension_child 三项 boolean |
|
||||
| <code>needs_completion</code> | 三项 completion 的逻辑或；包括独立旋转需要的隐藏纹理和关节重叠 |
|
||||
| <code>confidence</code> | LLM 自报置信度，范围 0～1 |
|
||||
|
||||
所有坐标都以完整画布左上角为原点并归一化到 0～1。例如：
|
||||
@@ -234,19 +244,26 @@ POST /v1/responses
|
||||
- 部件不能引用自己；
|
||||
- 父子关系和遮挡关系不能形成环；
|
||||
- 可见部件必须具有正面积 bbox；
|
||||
- 可见部件必须至少有一个正提示点；
|
||||
- 正提示点必须位于 bbox 内；
|
||||
- 模型正点可为空；落在 bbox 外只产生警告并丢弃其提示作用；
|
||||
- bind_to 必须引用骨骼部件，骨骼的 parent 也必须是骨骼部件；
|
||||
- partial/hidden 必须声明 occlusion_fill，needs_completion 与三项补全字段一致；
|
||||
- 附件和面部特征不能声明独立骨骼关节延展；
|
||||
- 完全隐藏部件必须没有可见框和正点，并设置 <code>needs_completion=true</code>；
|
||||
- 自报置信度低于 0.6 时产生人工复核警告。
|
||||
|
||||
校验结果写入 <code>validation.json</code>。只有校验通过时才会写出正式的 <code>semantic-parts.json</code>。
|
||||
|
||||
随后进行像素几何检查：以 <code>alpha > 0</code> 为前景，记录原始模型点是否落在前景和粗框内。在粗框与前景交集中，用距离变换选择远离轮廓和 ROI 边界的像素作为候选。像素坐标使用 floor(norm × size)，1 映射到最后一个像素；候选归一化坐标使用像素中心。粗框按 floor/ceil 转为半开像素范围。
|
||||
|
||||
框内无前景、没有像素中心或 Alpha 不可用时标记 blocked，不生成点。隐藏部件标记 hidden。记录框内前景比例、连通域数量和候选到轮廓距离，不把候选声称为已确认的语义种子；原始 pivot 完全保留。
|
||||
|
||||
## 10. 可视化与人工复核
|
||||
|
||||
[lib/report.mjs](../lib/report.mjs) 将结构理解结果叠加到原图：
|
||||
|
||||
- 彩色矩形：模型预测的可见区域粗框；
|
||||
- 实心圆：SAM 正提示点；
|
||||
- 红色实心圆：无效模型点；黄色实心圆：前景上的不可信模型点；
|
||||
- 青色方块：程序生成的 Alpha 候选，仍需确认部件语义；
|
||||
- 空心圆：部件 pivot；
|
||||
- 虚线：当前部件到父部件的关系；
|
||||
- 文本：部件 ID。
|
||||
@@ -293,7 +310,11 @@ POST /v1/responses
|
||||
|
||||
| 文件 | 作用 |
|
||||
| --- | --- |
|
||||
| <code>source.png</code> | 模型实际看到的原始输入副本 |
|
||||
| <code>source-original.png</code> | 原始输入副本 |
|
||||
| <code>source.png</code> | 模型实际看到的预处理 RGBA |
|
||||
| <code>preprocessing.json</code> | 背景处理方式和像素统计 |
|
||||
| <code>geometry-validation.json</code> | 原始点诊断、框内前景比例、连通域、候选和阻塞原因 |
|
||||
| <code>segmentation-prompts.json</code> | 语义、粗框、程序生成的正点；负点为空，requires_mask_qa=true |
|
||||
| <code>prompt.txt</code> | 本次解析后的阶段提示词 |
|
||||
| <code>request.json</code> | 可复现实验的请求摘要，不包含 API Key 或完整 Base64 请求体 |
|
||||
| <code>response.raw.json</code> | 中转站原始成功响应 |
|
||||
@@ -323,6 +344,9 @@ POST /v1/responses
|
||||
| [providers/vision.mjs](../providers/vision.mjs) | Chat Completions/Responses 请求构造与文本提取 |
|
||||
| [providers/http.mjs](../providers/http.mjs) | HTTP、认证、超时、重试、SSE、元数据和脱敏 |
|
||||
| [lib/validate-parts.mjs](../lib/validate-parts.mjs) | JSON 解析、Schema 校验和关系校验 |
|
||||
| [lib/geometry.mjs](../lib/geometry.mjs) | Python 环境选择与几何阶段调用 |
|
||||
| [geometry_grounding.py](../geometry_grounding.py) | Alpha/ROI 诊断与距离变换候选生成 |
|
||||
| [normalize_reference.py](../normalize_reference.py) | RGBA 保留与边界连通白底去除 |
|
||||
| [lib/report.mjs](../lib/report.mjs) | SVG 标注和 HTML 检查报告 |
|
||||
| [lib/run-artifacts.mjs](../lib/run-artifacts.mjs) | 输出目录、恢复校验、状态和产物哈希 |
|
||||
| [stages/model-stage.mjs](../stages/model-stage.mjs) | 第一阶段总调度与失败处理 |
|
||||
@@ -333,12 +357,14 @@ POST /v1/responses
|
||||
|
||||
~~~text
|
||||
视觉 LLM 输出
|
||||
= 部件语义 + 层级 + 遮挡 + 粗框 + 提示点
|
||||
= 部件语义 + 层级 + rig_role/bind_to + 遮挡 + 粗框 + 粗关节 + 补全需求
|
||||
|
||||
视觉 LLM 不输出
|
||||
= 可信的像素 mask + 透明拆件 + 隐藏区域补全 + 最终骨骼
|
||||
~~~
|
||||
|
||||
下一阶段应读取 <code>semantic-parts.json</code>，把每个可见或部分可见部件的粗框和正负提示点转换为像素坐标，再交给 SAM 3.1 生成候选 mask。
|
||||
下一阶段应读取 <code>semantic-parts.json</code> 和 <code>segmentation-prompts.json</code>，仅使用 status=candidate 的语义、粗框和程序候选点。模型原始正负点不传给 SAM。候选始终 semantic_verified=false：框内可能混有多个部件，未来仍需要 mask 语义选择、越出角色 Alpha 比例等 QA。
|
||||
|
||||
已有 v1 结果可以通过 <code>--parts-file 旧计划.json --reference 原图.png --out-dir 新目录</code> 离线迁移与校验，不请求模型。迁移按 ID 推断骨骼/附件类型，袖子优先绑定同侧上臂，面部绑定头部；部分可见部件开启遮挡补全，骨骼连接处保守建议延展。这些推断在 notes 标注为待复核，不替代模型或人工判断。输入计划哈希和所有新产物参与 resume 校验。
|
||||
|
||||
<code>manifest.json</code> 当前只记录 SAM checkpoint 是否存在，<code>runtime_status</code> 仍为 <code>not_integrated</code>，不会加载权重或占用 GPU。
|
||||
|
||||
+11
-4
@@ -1,12 +1,12 @@
|
||||
# 第一阶段：可替换模型接口与角色结构理解
|
||||
|
||||
本轮实现 GPT6Astra 结构理解、Qwen Image 3.0 图片编辑适配器、结构 JSON 校验、粗标注预览、实验记录和缓存。新增模式不需要 Python，不运行旧的“拆件图 → SIFT 定位”流程。
|
||||
本轮实现结构理解、图片编辑适配器、结构 JSON 校验、Alpha 几何校验、候选种子、诊断预览和实验缓存。几何层需要 Python 与 requirements.txt，不运行旧的“拆件图 → SIFT 定位”流程。
|
||||
|
||||
结构理解当前停在 `understand`；SAM 分割、掩码修正、隐藏部件合并、Spine 骨骼导出是后续阶段。图片编辑测试保存的是原始候选，不是验收通过的最终资产。
|
||||
|
||||
## 环境与模型
|
||||
|
||||
要求 Node.js 20 或更高版本，无新增 npm 依赖。在 `AI_spine` 目录运行。
|
||||
要求 Node.js 20 或更高版本，无新增 npm 依赖。在 `AI_spine` 目录、已激活的 `ai-spine` Conda 环境中运行。每次修改须遵守 [AGENTS.md](../AGENTS.md)。
|
||||
|
||||
复用现有 `.env` 的地址和密钥，可增加：
|
||||
|
||||
@@ -74,8 +74,11 @@ node .\pipeline.mjs --mode annotated-parts `
|
||||
|
||||
| 文件 | 含义 |
|
||||
| --- | --- |
|
||||
| `source.png` | 实际提交的原图，不缩放、不裁剪 |
|
||||
| `source-original.png` / `source.png` | 原始输入 / 预处理 RGBA，不缩放、不裁剪 |
|
||||
| `preprocessing.json` | Alpha 保留或边界连通白底去除记录 |
|
||||
| `semantic-parts.json` | 已通过结构校验的部件清单 |
|
||||
| `geometry-validation.json` | 原始点 Alpha 检查、ROI 前景比例、候选来源和阻塞原因 |
|
||||
| `segmentation-prompts.json` | 语义 + 粗框 + 程序候选点；不转发模型原始点，始终需要 mask QA |
|
||||
| `annotated.svg` | 原图上的粗框、关节点、正点和父子连线，可直接浏览器打开 |
|
||||
| `report.html` | 自包含检查页面，带部件表和模型记录 |
|
||||
| `validation.json` | 字段、坐标、引用、父子环和遮挡环校验结果 |
|
||||
@@ -86,6 +89,10 @@ node .\pipeline.mjs --mode annotated-parts `
|
||||
|
||||
`status=completed` 仅表示这一步执行完成。`quality_status=needs_review` 表示仍需人工检查语义和框的位置。粗框不是像素 mask，输出也不是自动批准的骨骼。
|
||||
|
||||
v2 计划新增 `rig_role`、`bind_to` 和 `completion`（遮挡补全、父关节延展、子关节延展），`needs_completion` 必须等于三项逻辑或。新提示词让模型正负点留空；旧点仅用于诊断，程序从 ROI ∩ Alpha 前景重新生成候选，pivot 不参与前景吸附。空 ROI 标记 blocked，不生成替代点。
|
||||
|
||||
使用 `--parts-file semantic-parts.json --reference source.png --out-dir 新目录` 可离线迁移 v1 或检查 v2，无需密钥或网络。Python 固定使用 Conda 环境 `ai-spine`，`--python` 也必须指向该环境。依赖安装命令为 `conda run -n ai-spine python -m pip install -r requirements.txt`，完整说明见 README。背景预处理无法获得可用 Alpha 时在模型请求前失败。边界连通白底去除是一种保守启发式，贴到画布边缘的白色角色区域仍需人工检查。
|
||||
|
||||
校验失败保存原始响应、validation 和报告，退出码为非零，不继续分割。网络/API 失败保存失败 manifest。拒绝、截断和不完整 Responses 都不会被当作成功。
|
||||
|
||||
HTTP/SSE/网络失败还保存 `response.error.json`：状态码、请求 ID、耗时、尝试次数、响应体（最多 2 MB）或网络错误代码，密钥会脱敏。若服务端返回 overloaded，使用新目录稍后重跑；这不表示 schema 或分割模型出错。
|
||||
@@ -157,7 +164,7 @@ node .\tools\model-check.mjs --stage completion --image .\local-part.png --promp
|
||||
## 测试
|
||||
|
||||
```powershell
|
||||
npm test
|
||||
conda run -n ai-spine npm test
|
||||
```
|
||||
|
||||
测试使用本地 mock HTTP 服务或注入响应，不调用真实付费模型。覆盖模型优先级、协议请求/响应、multipart、能力限制、数值/引用/环校验、错误响应、重试、缓存和 CLI 全链路。
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Alpha constraints and deterministic seed candidates, never semantic masks."""
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from normalize_reference import prepare_rgba
|
||||
|
||||
|
||||
def pixel(point, width, height):
|
||||
# Normalized points map to their containing pixel; 1 maps to the last pixel.
|
||||
return min(width - 1, int(point[0] * width)), min(height - 1, int(point[1] * height))
|
||||
|
||||
|
||||
def ground_parts(image, plan):
|
||||
alpha = np.array(image.convert('RGBA'))[:, :, 3]
|
||||
foreground = alpha > 0
|
||||
height, width = alpha.shape
|
||||
usable_alpha = bool(foreground.any() and not foreground.all())
|
||||
silhouette_distance = cv2.distanceTransform(np.pad(foreground.astype(np.uint8), 1), cv2.DIST_L2, cv2.DIST_MASK_PRECISE)[1:-1, 1:-1]
|
||||
diagnostics, prompts = [], []
|
||||
for part in plan['parts']:
|
||||
box = part['bbox_norm_xyxy']
|
||||
entry = {
|
||||
'id': part['id'], 'status': 'hidden' if part['visibility'] == 'hidden' else 'blocked',
|
||||
'bbox_px_xyxy': None, 'bbox_foreground_ratio': None,
|
||||
'foreground_bbox_px_xyxy': None, 'foreground_components': 0,
|
||||
'model_positive_points': [], 'candidate_points': [], 'warnings': [],
|
||||
'semantic_verified': False,
|
||||
}
|
||||
for point in part.get('positive_points_norm', []):
|
||||
x, y = pixel(point, width, height)
|
||||
in_box = bool(box and box[0] <= point[0] <= box[2] and box[1] <= point[1] <= box[3])
|
||||
on_foreground = bool(foreground[y, x])
|
||||
entry['model_positive_points'].append({
|
||||
'point_norm': point, 'point_px': [x, y], 'alpha': int(alpha[y, x]),
|
||||
'inside_bbox': in_box, 'on_foreground': on_foreground,
|
||||
'status': 'UNTRUSTED_SEED' if usable_alpha and on_foreground and in_box else 'INVALID_SEED',
|
||||
})
|
||||
if part['visibility'] != 'hidden' and box:
|
||||
x0, y0 = math.floor(box[0] * width), math.floor(box[1] * height)
|
||||
x1, y1 = math.ceil(box[2] * width), math.ceil(box[3] * height)
|
||||
entry['bbox_px_xyxy'] = [x0, y0, x1, y1]
|
||||
roi = foreground[y0:y1, x0:x1]
|
||||
entry['bbox_foreground_ratio'] = float(roi.mean()) if roi.size else 0
|
||||
if not usable_alpha:
|
||||
entry['warnings'].append('NO_USABLE_ALPHA')
|
||||
elif not roi.any():
|
||||
entry['warnings'].append('EMPTY_FOREGROUND_ROI')
|
||||
else:
|
||||
ys, xs = np.nonzero(roi)
|
||||
entry['foreground_bbox_px_xyxy'] = [int(xs.min()) + x0, int(ys.min()) + y0, int(xs.max()) + x0 + 1, int(ys.max()) + y0 + 1]
|
||||
count, _ = cv2.connectedComponents(roi.astype(np.uint8), connectivity=8)
|
||||
entry['foreground_components'] = count - 1
|
||||
# Pad the ROI with background so an opaque ROI has a finite
|
||||
# distance, and candidates stay away from both silhouette/box edges.
|
||||
distance = cv2.distanceTransform(np.pad(roi.astype(np.uint8), 1), cv2.DIST_L2, cv2.DIST_MASK_PRECISE)[1:-1, 1:-1]
|
||||
# Only centers strictly inside the original continuous ROI qualify.
|
||||
eligible_x = ((np.arange(x0, x1) + .5) / width >= box[0]) & ((np.arange(x0, x1) + .5) / width <= box[2])
|
||||
eligible_y = ((np.arange(y0, y1) + .5) / height >= box[1]) & ((np.arange(y0, y1) + .5) / height <= box[3])
|
||||
eligible = roi & eligible_y[:, None] & eligible_x[None, :]
|
||||
if eligible.any():
|
||||
scores = np.where(eligible, distance, -1)
|
||||
cy, cx = np.unravel_index(np.argmax(scores), scores.shape)
|
||||
x, y = int(cx) + x0, int(cy) + y0
|
||||
entry['candidate_points'] = [{
|
||||
'point_px': [x, y], 'point_norm': [(x + .5) / width, (y + .5) / height],
|
||||
'distance_to_roi_boundary_px': float(distance[cy, cx]),
|
||||
'distance_to_silhouette_px': float(silhouette_distance[y, x]),
|
||||
'source': 'alpha_roi_distance_transform',
|
||||
}]
|
||||
entry['status'] = 'candidate'
|
||||
else:
|
||||
entry['warnings'].append('NO_PIXEL_CENTER_IN_ROI')
|
||||
if entry['bbox_foreground_ratio'] < .5:
|
||||
entry['warnings'].append('LOW_FOREGROUND_RATIO')
|
||||
if count > 2:
|
||||
entry['warnings'].append('MULTIPLE_FOREGROUND_COMPONENTS')
|
||||
entry['warnings'].append('SEMANTIC_REVIEW_REQUIRED')
|
||||
diagnostics.append(entry)
|
||||
prompts.append({
|
||||
'id': part['id'], 'text_prompt': part['semantic'], 'status': entry['status'],
|
||||
'bbox_px_xyxy': entry['bbox_px_xyxy'],
|
||||
'positive_points_px': [p['point_px'] for p in entry['candidate_points']],
|
||||
'positive_points_norm': [p['point_norm'] for p in entry['candidate_points']],
|
||||
'negative_points_px': [], 'seed_source': 'alpha_roi_distance_transform',
|
||||
'semantic_verified': False, 'requires_mask_qa': True,
|
||||
})
|
||||
points = [p for entry in diagnostics for p in entry['model_positive_points']]
|
||||
return {
|
||||
'schema_version': 1, 'width': width, 'height': height,
|
||||
'alpha_threshold': 0, 'coordinate_mapping': 'floor(norm * size), clamped at size - 1; boxes are half-open',
|
||||
'summary': {'model_points': len(points), 'invalid_model_points': sum(p['status'] == 'INVALID_SEED' for p in points),
|
||||
'background_model_points': sum(not p['on_foreground'] for p in points),
|
||||
'candidate_parts': sum(p['status'] == 'candidate' for p in diagnostics),
|
||||
'blocked_parts': sum(p['status'] == 'blocked' for p in diagnostics)},
|
||||
'parts': diagnostics,
|
||||
}, {'schema_version': 1, 'width': width, 'height': height, 'quality_status': 'needs_review', 'parts': prompts}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--input', required=True)
|
||||
parser.add_argument('--output', required=True)
|
||||
parser.add_argument('--prepare', action='store_true')
|
||||
parser.add_argument('--parts')
|
||||
parser.add_argument('--prompts')
|
||||
args = parser.parse_args()
|
||||
with Image.open(args.input) as image:
|
||||
if args.prepare:
|
||||
prepared, metadata = prepare_rgba(image)
|
||||
prepared.save(args.output, format='PNG')
|
||||
print(json.dumps(metadata))
|
||||
else:
|
||||
if not args.parts or not args.prompts:
|
||||
parser.error('--parts and --prompts are required for grounding')
|
||||
plan = json.loads(Path(args.parts).read_text(encoding='utf-8'))
|
||||
diagnostics, prompts = ground_parts(image, plan)
|
||||
Path(args.output).write_text(json.dumps(diagnostics, indent=2), encoding='utf-8')
|
||||
Path(args.prompts).write_text(json.dumps(prompts, indent=2), encoding='utf-8')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,16 @@
|
||||
import {execFileSync} from 'node:child_process';
|
||||
import {readFileSync} from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import {projectRoot} from './config.mjs';
|
||||
|
||||
export {selectPython as geometryPython} from './python-runtime.mjs';
|
||||
|
||||
export function prepareReference(python, input, output) {
|
||||
return JSON.parse(execFileSync(python, [path.join(projectRoot, 'geometry_grounding.py'), '--prepare', '--input', input, '--output', output], {encoding: 'utf8', windowsHide: true}));
|
||||
}
|
||||
|
||||
export function groundParts(python, out) {
|
||||
const diagnostics = path.join(out, 'geometry-validation.json');
|
||||
execFileSync(python, [path.join(projectRoot, 'geometry_grounding.py'), '--input', path.join(out, 'source.png'), '--parts', path.join(out, 'semantic-parts.json'), '--output', diagnostics, '--prompts', path.join(out, 'segmentation-prompts.json')], {encoding: 'utf8', windowsHide: true});
|
||||
return JSON.parse(readFileSync(diagnostics, 'utf8'));
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
import {spawnSync} from 'node:child_process';
|
||||
import path from 'node:path';
|
||||
|
||||
export function selectPython(requested) {
|
||||
const command = requested || (process.env.CONDA_PREFIX
|
||||
? path.join(process.env.CONDA_PREFIX, process.platform === 'win32' ? 'python.exe' : 'bin/python')
|
||||
: 'python');
|
||||
const probe = spawnSync(command, ['-c', [
|
||||
'import json, sys',
|
||||
'from pathlib import Path',
|
||||
"print(json.dumps({'executable': sys.executable, 'name': Path(sys.prefix).name, 'conda': (Path(sys.prefix) / 'conda-meta').is_dir()}))",
|
||||
].join('\n')], {encoding: 'utf8', windowsHide: true});
|
||||
let runtime;
|
||||
try { if (probe.status === 0) runtime = JSON.parse(probe.stdout.trim()); } catch { /* Report an actionable environment error below. */ }
|
||||
if (!runtime?.conda || runtime.name !== 'ai-spine') {
|
||||
throw new Error('本项目固定使用 Conda 环境 ai-spine。请先 conda activate ai-spine，或用 conda run -n ai-spine node ...；--python 也必须指向该环境的解释器');
|
||||
}
|
||||
const dependencies = spawnSync(runtime.executable, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore', windowsHide: true});
|
||||
if (dependencies.status !== 0) {
|
||||
throw new Error('ai-spine 环境缺少或无法加载 Python 依赖。请在 AI_spine 目录执行：conda run -n ai-spine python -m pip install -r requirements.txt');
|
||||
}
|
||||
return runtime.executable;
|
||||
}
|
||||
+18
-7
@@ -2,11 +2,12 @@ import {dataUrl} from './images.mjs';
|
||||
|
||||
const escape = (value) => String(value).replace(/[&<>"']/gu, (c) => ({'&': '&', '<': '<', '>': '>', '"': '"', "'": '''}[c]));
|
||||
|
||||
export function annotatedSvg(image, parts) {
|
||||
export function annotatedSvg(image, parts, geometry = null) {
|
||||
const {width: w, height: h} = image;
|
||||
const stroke = Math.max(w, h) / 400;
|
||||
const font = Math.max(w, h) / 65;
|
||||
const byId = new Map(parts.map((part) => [part.id, part]));
|
||||
const diagnostics = new Map((geometry?.parts || []).map((part) => [part.id, part]));
|
||||
const overlays = parts.map((part, index) => {
|
||||
const color = `hsl(${(index * 137.5) % 360} 85% 55%)`;
|
||||
const [px, py] = [part.pivot_norm[0] * w, part.pivot_norm[1] * h];
|
||||
@@ -17,14 +18,24 @@ export function annotatedSvg(image, parts) {
|
||||
const [x0, y0, x1, y1] = part.bbox_norm_xyxy;
|
||||
box = `<rect x="${x0 * w}" y="${y0 * h}" width="${(x1 - x0) * w}" height="${(y1 - y0) * h}"/>`;
|
||||
}
|
||||
const points = part.positive_points_norm.map(([x, y]) => `<circle cx="${x * w}" cy="${y * h}" r="${stroke * 1.5}" fill="${color}"/>`).join('');
|
||||
return `<g fill="none" stroke="${color}" stroke-width="${stroke}">${line}${box}${points}<circle cx="${px}" cy="${py}" r="${stroke * 3}"/><text x="${Math.min(px + 4, w * 0.6)}" y="${Math.max(font, py - 5)}" fill="${color}" stroke="#111" stroke-width="${stroke / 2}" paint-order="stroke" font-size="${font}" font-family="sans-serif">${escape(part.id)}</text></g>`;
|
||||
const diagnostic = diagnostics.get(part.id);
|
||||
const points = part.positive_points_norm.map(([x, y], i) => {
|
||||
const status = diagnostic?.model_positive_points[i]?.status;
|
||||
const fill = status === 'INVALID_SEED' ? '#ff4444' : status === 'UNTRUSTED_SEED' ? '#ffc857' : '#999';
|
||||
return `<circle cx="${x * w}" cy="${y * h}" r="${stroke * 2}" stroke="#111" fill="${fill}"><title>${escape(part.id)}: ${status || 'UNTRUSTED_MODEL_POINT'}</title></circle>`;
|
||||
}).join('');
|
||||
const candidates = (diagnostic?.candidate_points || []).map(({point_norm: [x, y]}) => `<rect x="${x * w - stroke * 2}" y="${y * h - stroke * 2}" width="${stroke * 4}" height="${stroke * 4}" stroke="#111" fill="#44e5ff"><title>${escape(part.id)}: Alpha candidate, semantic unverified</title></rect>`).join('');
|
||||
return `<g fill="none" stroke="${color}" stroke-width="${stroke}">${line}${box}${points}${candidates}<circle cx="${px}" cy="${py}" r="${stroke * 3}"/><text x="${Math.min(px + 4, w * 0.6)}" y="${Math.max(font, py - 5)}" fill="${color}" stroke="#111" stroke-width="${stroke / 2}" paint-order="stroke" font-size="${font}" font-family="sans-serif">${escape(part.id)}</text></g>`;
|
||||
}).join('\n');
|
||||
return `<svg xmlns="http://www.w3.org/2000/svg" width="${w}" height="${h}" viewBox="0 0 ${w} ${h}"><rect width="100%" height="100%" fill="#262632"/><image href="${dataUrl(image)}" width="${w}" height="${h}"/>${overlays}</svg>`;
|
||||
}
|
||||
|
||||
export function understandingReport(image, data, validation, metadata) {
|
||||
const svg = data && validation.valid ? annotatedSvg(image, data.parts) : `<img src="${dataUrl(image)}" alt="source">`;
|
||||
const rows = data && validation.valid ? data.parts.map((part) => `<tr><td>${escape(part.id)}</td><td>${escape(part.parent ?? 'root')}</td><td>${escape(part.visibility)}</td><td>${escape(part.occluded_by.join(', '))}</td><td>${part.confidence}</td></tr>`).join('') : '';
|
||||
return `<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>角色结构理解检查</title><style>body{font:16px system-ui;max-width:1100px;margin:24px auto;padding:16px;background:#f5f5f5}svg,img{max-width:100%;height:auto;max-height:85vh}td,th{padding:8px;border:1px solid #ccc}table{border-collapse:collapse}pre{white-space:pre-wrap;overflow-wrap:anywhere}</style><h1>角色结构理解</h1><p>此图仅包含模型预测的粗框、关节点（空心圆）和正点（实心圆），尚未执行 SAM 分割或补全。结构校验通过不代表语义正确。</p><p>状态：${validation.valid ? 'needs_review（待检查）' : 'failed'}</p>${svg}<h2>校验</h2><pre>${escape(JSON.stringify(validation, null, 2))}</pre><h2>部件</h2><table><tr><th>ID</th><th>父部件</th><th>可见性</th><th>遮挡物</th><th>自报置信度</th></tr>${rows}</table><h2>模型说明</h2><pre>${escape(JSON.stringify(data?.notes || [], null, 2))}</pre><h2>请求记录</h2><pre>${escape(JSON.stringify(metadata, null, 2))}</pre></html>`;
|
||||
export function understandingReport(image, data, validation, metadata, geometry = null) {
|
||||
const svg = data && validation.valid ? annotatedSvg(image, data.parts, geometry) : `<img src="${dataUrl(image)}" alt="source">`;
|
||||
const diagnostics = new Map((geometry?.parts || []).map((part) => [part.id, part]));
|
||||
const rows = data && validation.valid ? data.parts.map((part) => {
|
||||
const g = diagnostics.get(part.id);
|
||||
return `<tr><td>${escape(part.id)}</td><td>${escape(part.rig_role || '')}<br>${escape(part.bind_to ?? part.parent ?? 'root')}</td><td>${escape(part.visibility)}</td><td>${escape(part.occluded_by.join(', '))}</td><td>${escape(Object.entries(part.completion || {}).filter(([, value]) => value).map(([key]) => key).join(', ') || '无')}</td><td>${g?.bbox_foreground_ratio == null ? '—' : `${(g.bbox_foreground_ratio * 100).toFixed(1)}%`}</td><td>${escape(g?.status || 'pending')}<br>${escape((g?.warnings || []).join(', '))}</td><td>${part.confidence}</td></tr>`;
|
||||
}).join('') : '';
|
||||
return `<!doctype html><html lang="zh-CN"><meta charset="utf-8"><title>角色结构与几何检查</title><style>body{font:16px system-ui;max-width:1400px;margin:24px auto;padding:16px;background:#f5f5f5}svg,img{max-width:100%;height:auto;max-height:85vh}td,th{padding:8px;border:1px solid #ccc}table{border-collapse:collapse}pre{white-space:pre-wrap;overflow-wrap:anywhere}</style><h1>角色结构与几何检查</h1><p>尚未执行 SAM 分割或补全。红色实心圆：无效模型点；黄色实心圆：通过前景检查但不可信的模型点；青色方块：程序生成的 Alpha 候选。空心圆是原始关节点，不做前景吸附。粗框与候选均需语义复核，Alpha 不能证明点属于目标部件。</p><p>状态：${validation.valid ? 'needs_review（待检查）' : 'failed'}</p>${svg}<h2>几何摘要</h2><pre>${escape(JSON.stringify(geometry?.summary || {}, null, 2))}</pre><h2>部件</h2><table><tr><th>ID</th><th>角色/绑定</th><th>可见性</th><th>遮挡物</th><th>补全项</th><th>框内前景</th><th>候选状态</th><th>自报置信度</th></tr>${rows}</table><h2>校验</h2><pre>${escape(JSON.stringify(validation, null, 2))}</pre><h2>模型说明</h2><pre>${escape(JSON.stringify(data?.notes || [], null, 2))}</pre><h2>请求记录</h2><pre>${escape(JSON.stringify(metadata, null, 2))}</pre></html>`;
|
||||
}
|
||||
|
||||
+54
-4
@@ -3,6 +3,7 @@ import path from 'node:path';
|
||||
import {projectRoot} from './config.mjs';
|
||||
|
||||
export const partsSchema = JSON.parse(readFileSync(path.join(projectRoot, 'schemas', 'semantic-parts.schema.json'), 'utf8'));
|
||||
const legacySchema = JSON.parse(readFileSync(path.join(projectRoot, 'schemas', 'semantic-parts-v1.schema.json'), 'utf8'));
|
||||
|
||||
// Validate the subset of JSON Schema used by the checked-in contract. Keeping
|
||||
// the validation driven by that contract avoids two independent field lists.
|
||||
@@ -46,13 +47,16 @@ export function parseParts(text) {
|
||||
export function validateParts(data) {
|
||||
const errors = [];
|
||||
const warnings = [];
|
||||
validate(data, partsSchema, '$', errors);
|
||||
const legacy = data?.schema_version === 1;
|
||||
validate(data, legacy ? legacySchema : partsSchema, '$', errors);
|
||||
if (errors.length) return {valid: false, errors, warnings};
|
||||
const ids = new Set(data.parts.map((part) => part.id));
|
||||
const byId = new Map(data.parts.map((part) => [part.id, part]));
|
||||
if (legacy) warnings.push('旧版计划：rig_role、bind_to 和 completion 将按保守规则迁移，需复核');
|
||||
if (ids.size !== data.parts.length) errors.push('部件 ID 重复');
|
||||
for (const part of data.parts) {
|
||||
if (part.id === 'root') errors.push('root 为后续导出保留 ID');
|
||||
for (const ref of [part.parent, ...part.occluded_by].filter((item) => item !== null)) {
|
||||
for (const ref of [part.parent, ...part.occluded_by, ...(!legacy ? [part.bind_to] : [])].filter((item) => item !== null)) {
|
||||
if (!ids.has(ref) || ref === part.id) errors.push(`${part.id}: 引用了不存在的部件或自身 ${ref}`);
|
||||
}
|
||||
if (new Set(part.occluded_by).size !== part.occluded_by.length) errors.push(`${part.id}: occluded_by 重复`);
|
||||
@@ -62,11 +66,19 @@ export function validateParts(data) {
|
||||
warnings.push(`${part.id}: 完全隐藏部件需要人工确认，不能直接生成可见 mask`);
|
||||
} else {
|
||||
if (!box || box[0] >= box[2] || box[1] >= box[3]) errors.push(`${part.id}: 可见 bbox 必须具有正面积`);
|
||||
if (!part.positive_points_norm.length) errors.push(`${part.id}: 可见部件需要至少一个正点`);
|
||||
if (box) for (const [x, y] of part.positive_points_norm) {
|
||||
if (x < box[0] || x > box[2] || y < box[1] || y > box[3]) errors.push(`${part.id}: 正点不在可见框内`);
|
||||
if (x < box[0] || x > box[2] || y < box[1] || y > box[3]) warnings.push(`${part.id}: 模型正点不在可见框内，将丢弃并重新生成候选`);
|
||||
}
|
||||
}
|
||||
if (part.positive_points_norm.length || part.negative_points_norm.length) warnings.push(`${part.id}: 模型提示点仅用于诊断，不直接传给 SAM`);
|
||||
if (!legacy) {
|
||||
if (part.rig_role === 'bone_part' && part.bind_to !== null) errors.push(`${part.id}: bone_part 使用 parent，bind_to 必须为 null`);
|
||||
if (part.bind_to !== null && byId.get(part.bind_to)?.rig_role !== 'bone_part') errors.push(`${part.id}: bind_to 必须引用 bone_part`);
|
||||
if (part.rig_role === 'bone_part' && part.parent !== null && byId.get(part.parent)?.rig_role !== 'bone_part') errors.push(`${part.id}: bone_part 的 parent 必须引用 bone_part`);
|
||||
if (part.needs_completion !== Object.values(part.completion).some(Boolean)) errors.push(`${part.id}: needs_completion 必须等于 completion 各项的逻辑或`);
|
||||
if (part.visibility !== 'visible' && !part.completion.occlusion_fill) errors.push(`${part.id}: partial/hidden 部件需要 occlusion_fill`);
|
||||
if (part.rig_role !== 'bone_part' && (part.completion.joint_extension_parent || part.completion.joint_extension_child)) errors.push(`${part.id}: 附件/面部特征不应声明独立骨骼关节延展`);
|
||||
}
|
||||
if (part.confidence < 0.6) warnings.push(`${part.id}: 模型自报低置信度，需要复核`);
|
||||
}
|
||||
if (!errors.length) {
|
||||
@@ -89,3 +101,41 @@ export function validateParts(data) {
|
||||
}
|
||||
return {valid: errors.length === 0, errors, warnings};
|
||||
}
|
||||
|
||||
// Legacy results remain readable. These inferred rig/completion decisions are
|
||||
// explicit review candidates, not a claim that the model supplied them.
|
||||
export function upgradeParts(data) {
|
||||
if (data.schema_version !== 1) return structuredClone(data);
|
||||
const result = structuredClone(data);
|
||||
result.schema_version = 2;
|
||||
const byId = new Map(result.parts.map((part) => [part.id, part]));
|
||||
for (const part of result.parts) {
|
||||
part.rig_role = /^(eye|eyes|eyebrow|nose|mouth|mustache)(_|$)/u.test(part.id) ? 'facial_feature'
|
||||
: /^(pelvis|torso|neck|head|upper_arm|forearm|hand|thigh|shin|foot)(_|$)/u.test(part.id) ? 'bone_part' : 'attachment';
|
||||
}
|
||||
const nearestBone = (id) => {
|
||||
const seen = new Set();
|
||||
while (id && !seen.has(id)) {
|
||||
seen.add(id);
|
||||
const part = byId.get(id);
|
||||
if (!part) return null;
|
||||
if (part.rig_role === 'bone_part') return id;
|
||||
id = part.parent;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
for (const part of result.parts) {
|
||||
const sleeveBone = part.id.replace(/^sleeve_/u, 'upper_arm_');
|
||||
part.bind_to = part.rig_role === 'bone_part' ? null
|
||||
: part.id.startsWith('sleeve_') && byId.get(sleeveBone)?.rig_role === 'bone_part' ? sleeveBone : nearestBone(part.parent);
|
||||
if (part.rig_role === 'bone_part') part.parent = nearestBone(part.parent);
|
||||
part.completion = {
|
||||
occlusion_fill: part.visibility !== 'visible' || part.needs_completion,
|
||||
joint_extension_parent: part.rig_role === 'bone_part' && part.parent !== null,
|
||||
joint_extension_child: part.rig_role === 'bone_part' && result.parts.some((child) => child.rig_role === 'bone_part' && nearestBone(child.parent) === part.id),
|
||||
};
|
||||
part.needs_completion = Object.values(part.completion).some(Boolean);
|
||||
}
|
||||
result.notes.push('Migrated from schema v1: rig roles/bindings and completion are heuristic review candidates. Original model points are diagnostic only; use segmentation-prompts.json for geometry candidates.');
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -3,14 +3,50 @@
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from PIL import Image
|
||||
import numpy as np
|
||||
import cv2
|
||||
|
||||
|
||||
def prepare_rgba(image, white_threshold=240):
|
||||
"""Keep existing alpha; remove only border-connected near-white background.
|
||||
|
||||
No crop or rescale here: semantic coordinates must retain the full canvas.
|
||||
White regions enclosed by character outlines are deliberately preserved.
|
||||
"""
|
||||
rgba = np.array(image.convert('RGBA'))
|
||||
alpha = rgba[:, :, 3]
|
||||
method = 'existing_alpha'
|
||||
removed = 0
|
||||
if not np.any(alpha == 0):
|
||||
white = np.all(rgba[:, :, :3] >= white_threshold, axis=2).astype(np.uint8)
|
||||
_, labels = cv2.connectedComponents(white, connectivity=4)
|
||||
border = np.unique(np.concatenate((labels[0], labels[-1], labels[:, 0], labels[:, -1])))
|
||||
background = np.isin(labels, border[border != 0])
|
||||
removed = int(background.sum())
|
||||
rgba[background, 3] = 0
|
||||
method = 'border_connected_white'
|
||||
foreground = rgba[:, :, 3] > 0
|
||||
if not foreground.any():
|
||||
raise ValueError('Reference has no foreground after background removal')
|
||||
if foreground.all():
|
||||
raise ValueError('No transparent background or border-connected white background; supply a cutout RGBA PNG')
|
||||
return Image.fromarray(rgba), {
|
||||
'method': method, 'white_threshold': white_threshold,
|
||||
'removed_pixels': removed, 'foreground_pixels': int(foreground.sum()),
|
||||
'width': image.width, 'height': image.height,
|
||||
'needs_review': method != 'existing_alpha',
|
||||
}
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument('--input', required=True)
|
||||
p.add_argument('--output', required=True)
|
||||
p.add_argument('--target', type=int, default=1024)
|
||||
p.add_argument('--remove-background', action='store_true')
|
||||
a = p.parse_args()
|
||||
im = Image.open(a.input).convert('RGBA')
|
||||
if a.remove_background:
|
||||
im, _ = prepare_rgba(im)
|
||||
longest = max(im.width, im.height)
|
||||
if longest < a.target:
|
||||
scale = a.target / longest
|
||||
|
||||
+1
-21
@@ -11,6 +11,7 @@ import {spawnSync} from 'node:child_process';
|
||||
import {cpSync, existsSync, mkdirSync, readFileSync, rmSync, unlinkSync, writeFileSync} from 'node:fs';
|
||||
import {fileURLToPath} from 'node:url';
|
||||
import path from 'node:path';
|
||||
import {selectPython} from './lib/python-runtime.mjs';
|
||||
|
||||
const repoRoot = path.dirname(fileURLToPath(import.meta.url));
|
||||
const defaultGenielabsDir = path.resolve(repoRoot, '..', 'spine-animation-ai');
|
||||
@@ -82,27 +83,6 @@ function loadEnvFile(file) {
|
||||
return result;
|
||||
}
|
||||
|
||||
function selectPython(requested) {
|
||||
const venvPython = path.join(repoRoot, '.venv', process.platform === 'win32' ? 'Scripts' : 'bin', process.platform === 'win32' ? 'python.exe' : 'python');
|
||||
const candidates = requested ? [requested] : ['python', 'python3', venvPython];
|
||||
for (const candidate of candidates) {
|
||||
const check = spawnSync(candidate, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore'});
|
||||
if (check.status === 0) return candidate;
|
||||
}
|
||||
if (requested) throw new Error(`指定的 Python 无法导入 cv2、PIL、numpy：${requested}；请在该环境安装 requirements.txt`);
|
||||
if (!existsSync(venvPython)) {
|
||||
const bootstrapPython = ['python', 'python3'].find((candidate) => spawnSync(candidate, ['-c', 'import sys'], {stdio: 'ignore'}).status === 0);
|
||||
if (!bootstrapPython) throw new Error('未找到 Python，请激活环境或通过 --python 指定');
|
||||
const bootstrap = spawnSync(bootstrapPython, ['-m', 'venv', path.join(repoRoot, '.venv')], {stdio: 'inherit'});
|
||||
if (bootstrap.status !== 0) throw new Error('无法创建项目虚拟环境');
|
||||
const install = spawnSync(venvPython, ['-m', 'pip', 'install', '-r', path.join(repoRoot, 'requirements.txt')], {stdio: 'inherit'});
|
||||
if (install.status !== 0) throw new Error('Python 依赖安装失败，请检查网络或手动安装 requirements.txt');
|
||||
}
|
||||
const check = spawnSync(venvPython, ['-c', 'import cv2, PIL, numpy'], {stdio: 'ignore'});
|
||||
if (check.status === 0) return venvPython;
|
||||
throw new Error('虚拟环境安装后仍缺少 cv2、Pillow、numpy');
|
||||
}
|
||||
|
||||
async function requestAtlas(reference, model, size, config, prompt) {
|
||||
for (let attempt = 0; attempt < 3; attempt += 1) {
|
||||
const form = new FormData();
|
||||
|
||||
@@ -3,18 +3,32 @@ Return a JSON part plan, not an image. Preserve the exact pose, asymmetry, cloth
|
||||
and screen-left/screen-right orientation. Do not require a fixed number of parts.
|
||||
Use stable lowercase ASCII IDs (letters, numbers, underscores, hyphens).
|
||||
Separate parts only where independent motion is useful and the image supports it.
|
||||
Parent refers to the motion hierarchy; null means attached to the future root bone.
|
||||
You are a semantic planner, not a pixel-grounding detector.
|
||||
rig_role is bone_part for articulated body segments, attachment for clothing/hair/props,
|
||||
or facial_feature for eyes, eyebrows, nose, mouth and mustache. Facial features and
|
||||
decorations do not each need a bone. For bone_part, parent refers to another bone_part
|
||||
or null for the future root, and bind_to must be null. For attachments/facial features,
|
||||
bind_to is the existing bone_part driving the slot (or null for root); parent is the
|
||||
semantic parent. Bind sleeves to the appropriate upper arm, facial features to head.
|
||||
Occlusion is a separate relation: occluded_by lists parts visually in front.
|
||||
All coordinates are normalized to the full supplied canvas, including transparent
|
||||
margins. bbox_norm_xyxy is [xmin,ymin,xmax,ymax], never [x,y,width,height].
|
||||
pivot_norm is the estimated attachment joint, not necessarily the bbox center.
|
||||
For visible or partly visible parts give a coarse visible bbox and positive points
|
||||
inside visible pixels; optional negative points identify nearby other parts.
|
||||
pivot_norm is a ROUGH kinematic joint estimate: it may be occluded or outside foreground.
|
||||
For visible or partly visible parts give only a coarse visible bbox.
|
||||
Leave positive_points_norm=[] and negative_points_norm=[]. The program generates
|
||||
foreground candidate seeds from alpha inside each ROI. Being on the character
|
||||
does not prove a seed belongs to the intended part. Do not invent precision.
|
||||
Do not claim precise pixel masks. A segmentation stage will refine boundaries.
|
||||
If a part is fully hidden, mark visibility="hidden", bbox_norm_xyxy=null,
|
||||
positive_points_norm=[], needs_completion=true and explain the uncertainty.
|
||||
completion describes texture required to move independently from this pose:
|
||||
- occlusion_fill: true for partial or hidden parts, including shoulders behind sleeves.
|
||||
- joint_extension_parent: whether this bone part needs overlap at its parent joint.
|
||||
- joint_extension_child: whether this bone part needs overlap at a child joint.
|
||||
Attachments/facial features have both joint_extension flags false.
|
||||
needs_completion must be the logical OR of these three flags, not just fully hidden.
|
||||
Do not invent hidden limbs or accessories without visual evidence. Put assumptions
|
||||
in notes. Confidence is your uncertainty estimate, not a measured segmentation score.
|
||||
Avoid cycles in parent and occluded_by relations. If an object crosses in front
|
||||
and behind another, split its visual layers or flag the ambiguity in notes.
|
||||
Return schema_version=1, coordinate_space="normalized_xy_top_left", parts, notes.
|
||||
Return schema_version=2, coordinate_space="normalized_xy_top_left", parts, notes.
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_version", "coordinate_space", "parts", "notes"],
|
||||
"properties": {
|
||||
"schema_version": {"type": "integer", "enum": [1]},
|
||||
"coordinate_space": {"type": "string", "enum": ["normalized_xy_top_left"]},
|
||||
"notes": {"type": "array", "items": {"type": "string"}},
|
||||
"parts": {
|
||||
"type": "array", "minItems": 1, "maxItems": 64,
|
||||
"items": {
|
||||
"type": "object", "additionalProperties": false,
|
||||
"required": ["id", "semantic", "parent", "visibility", "bbox_norm_xyxy", "pivot_norm", "positive_points_norm", "negative_points_norm", "occluded_by", "needs_completion", "confidence"],
|
||||
"properties": {
|
||||
"id": {"type": "string", "pattern": "^[a-z][a-z0-9_-]*$", "maxLength": 64},
|
||||
"semantic": {"type": "string", "minLength": 1},
|
||||
"parent": {"type": ["string", "null"]},
|
||||
"visibility": {"type": "string", "enum": ["visible", "partial", "hidden"]},
|
||||
"bbox_norm_xyxy": {"type": ["array", "null"], "minItems": 4, "maxItems": 4, "items": {"type": "number", "minimum": 0, "maximum": 1}},
|
||||
"pivot_norm": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "number", "minimum": 0, "maximum": 1}},
|
||||
"positive_points_norm": {"type": "array", "maxItems": 32, "items": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "number", "minimum": 0, "maximum": 1}}},
|
||||
"negative_points_norm": {"type": "array", "maxItems": 32, "items": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "number", "minimum": 0, "maximum": 1}}},
|
||||
"occluded_by": {"type": "array", "items": {"type": "string"}},
|
||||
"needs_completion": {"type": "boolean"},
|
||||
"confidence": {"type": "number", "minimum": 0, "maximum": 1}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,18 +3,29 @@
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_version", "coordinate_space", "parts", "notes"],
|
||||
"properties": {
|
||||
"schema_version": {"type": "integer", "enum": [1]},
|
||||
"schema_version": {"type": "integer", "enum": [2]},
|
||||
"coordinate_space": {"type": "string", "enum": ["normalized_xy_top_left"]},
|
||||
"notes": {"type": "array", "items": {"type": "string"}},
|
||||
"parts": {
|
||||
"type": "array", "minItems": 1, "maxItems": 64,
|
||||
"items": {
|
||||
"type": "object", "additionalProperties": false,
|
||||
"required": ["id", "semantic", "parent", "visibility", "bbox_norm_xyxy", "pivot_norm", "positive_points_norm", "negative_points_norm", "occluded_by", "needs_completion", "confidence"],
|
||||
"required": ["id", "semantic", "parent", "rig_role", "bind_to", "completion", "visibility", "bbox_norm_xyxy", "pivot_norm", "positive_points_norm", "negative_points_norm", "occluded_by", "needs_completion", "confidence"],
|
||||
"properties": {
|
||||
"id": {"type": "string", "pattern": "^[a-z][a-z0-9_-]*$", "maxLength": 64},
|
||||
"semantic": {"type": "string", "minLength": 1},
|
||||
"parent": {"type": ["string", "null"]},
|
||||
"rig_role": {"type": "string", "enum": ["bone_part", "attachment", "facial_feature"]},
|
||||
"bind_to": {"type": ["string", "null"], "description": "bone_part: null; attachment/facial_feature: existing bone_part ID or null for root"},
|
||||
"completion": {
|
||||
"type": "object", "additionalProperties": false,
|
||||
"required": ["occlusion_fill", "joint_extension_parent", "joint_extension_child"],
|
||||
"properties": {
|
||||
"occlusion_fill": {"type": "boolean"},
|
||||
"joint_extension_parent": {"type": "boolean"},
|
||||
"joint_extension_child": {"type": "boolean"}
|
||||
}
|
||||
},
|
||||
"visibility": {"type": "string", "enum": ["visible", "partial", "hidden"]},
|
||||
"bbox_norm_xyxy": {"type": ["array", "null"], "minItems": 4, "maxItems": 4, "items": {"type": "number", "minimum": 0, "maximum": 1}},
|
||||
"pivot_norm": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "number", "minimum": 0, "maximum": 1}},
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@ import sys
|
||||
try:
|
||||
import cv2
|
||||
except ImportError as error:
|
||||
raise SystemExit("缺少 OpenCV，请先执行：python3 -m pip install opencv-python") from error
|
||||
raise SystemExit("缺少 OpenCV，请在 AI_spine 目录执行：conda run -n ai-spine python -m pip install -r requirements.txt") from error
|
||||
|
||||
|
||||
def segment(atlas_path: str, output_dir: str, min_area: int, padding: int, bg_threshold: int):
|
||||
|
||||
+36
-10
@@ -3,7 +3,8 @@ import {execFileSync} from 'node:child_process';
|
||||
import path from 'node:path';
|
||||
import {projectRoot, publicStage, samInfo} from '../lib/config.mjs';
|
||||
import {readImage, sha256} from '../lib/images.mjs';
|
||||
import {partsSchema, parseParts, validateParts} from '../lib/validate-parts.mjs';
|
||||
import {partsSchema, parseParts, validateParts, upgradeParts} from '../lib/validate-parts.mjs';
|
||||
import {geometryPython, prepareReference, groundParts} from '../lib/geometry.mjs';
|
||||
import {annotatedSvg, understandingReport} from '../lib/report.mjs';
|
||||
import {prepareRun, jsonFile, finishRun} from '../lib/run-artifacts.mjs';
|
||||
import {analyze, extractVisionText} from '../providers/vision.mjs';
|
||||
@@ -16,7 +17,8 @@ function codeVersion() {
|
||||
'stages/model-stage.mjs', 'lib/config.mjs', 'lib/images.mjs', 'lib/report.mjs',
|
||||
'lib/run-artifacts.mjs', 'lib/validate-parts.mjs', 'providers/http.mjs',
|
||||
'providers/vision.mjs', 'providers/image-edits.mjs', 'tools/model-check.mjs',
|
||||
'schemas/semantic-parts.schema.json',
|
||||
'schemas/semantic-parts.schema.json', 'schemas/semantic-parts-v1.schema.json',
|
||||
'lib/geometry.mjs', 'lib/python-runtime.mjs', 'geometry_grounding.py', 'normalize_reference.py',
|
||||
];
|
||||
const sourceHash = sha256(files.map((file) => `${file}\n${readFileSync(path.join(projectRoot, file), 'utf8')}`).join('\n'));
|
||||
let commit = null;
|
||||
@@ -33,37 +35,61 @@ function runIdentity(stage, images, extra) {
|
||||
};
|
||||
}
|
||||
|
||||
export async function runUnderstanding({loaded, stage, reference, outDir, resume = false}, transport) {
|
||||
const image = readImage(reference, {pngOnly: true});
|
||||
const identity = runIdentity(stage, [image], {schema_sha256: sha256(JSON.stringify(partsSchema))});
|
||||
export async function runUnderstanding({loaded, stage, reference, outDir, resume = false, python, partsFile}, transport) {
|
||||
const original = readImage(reference, {pngOnly: true});
|
||||
const importedText = partsFile ? readFileSync(partsFile, 'utf8') : null;
|
||||
const identity = runIdentity(stage, [original], {schema_sha256: sha256(JSON.stringify(partsSchema)),
|
||||
input_mode: partsFile ? 'offline_plan' : 'vision_request', imported_plan_sha256: importedText === null ? null : sha256(importedText)});
|
||||
const run = prepareRun(outDir, sha256(JSON.stringify(identity)), resume);
|
||||
if (run.reused) return {out: run.out, reused: true};
|
||||
const manifest = {...identity, fingerprint: sha256(JSON.stringify(identity)), status: 'running', quality_status: 'pending', started_at: new Date().toISOString(), sam: samInfo(loaded)};
|
||||
jsonFile(run.manifestPath, manifest, stage.apiKey);
|
||||
writeFileSync(path.join(run.out, 'source.png'), image.bytes);
|
||||
writeFileSync(path.join(run.out, 'source-original.png'), original.bytes);
|
||||
writeFileSync(path.join(run.out, 'prompt.txt'), stage.prompt);
|
||||
jsonFile(path.join(run.out, 'request.json'), {...identity, reference: 'source.png'}, stage.apiKey);
|
||||
try {
|
||||
const {payload, metadata} = await analyze(stage, image, partsSchema, transport);
|
||||
const runtime = geometryPython(python);
|
||||
const preprocessing = prepareReference(runtime, path.join(run.out, 'source-original.png'), path.join(run.out, 'source.png'));
|
||||
const image = readImage(path.join(run.out, 'source.png'), {pngOnly: true});
|
||||
jsonFile(path.join(run.out, 'preprocessing.json'), preprocessing, stage.apiKey);
|
||||
manifest.prepared_input = {sha256: image.sha256, width: image.width, height: image.height};
|
||||
jsonFile(path.join(run.out, 'request.json'), {...identity, reference: 'source.png', prepared_input: manifest.prepared_input}, stage.apiKey);
|
||||
const {payload, metadata} = importedText !== null
|
||||
? {payload: {imported_plan_text: importedText}, metadata: {source: 'offline_plan', network_requested: false}}
|
||||
: await analyze(stage, image, partsSchema, transport);
|
||||
jsonFile(path.join(run.out, 'response.raw.json'), payload, stage.apiKey);
|
||||
jsonFile(path.join(run.out, 'request-metadata.json'), metadata, stage.apiKey);
|
||||
let data = null;
|
||||
let validation;
|
||||
try {
|
||||
const text = extractVisionText(payload, stage.protocol);
|
||||
const text = importedText ?? extractVisionText(payload, stage.protocol);
|
||||
writeFileSync(path.join(run.out, 'response.txt'), redact(text, stage.apiKey));
|
||||
data = parseParts(text);
|
||||
validation = validateParts(data);
|
||||
if (validation.valid) {
|
||||
const warnings = validation.warnings;
|
||||
data = upgradeParts(data);
|
||||
validation = validateParts(data);
|
||||
validation.warnings = [...new Set([...warnings, ...validation.warnings])];
|
||||
}
|
||||
} catch (error) { validation = {valid: false, errors: [redact(error.message, stage.apiKey)], warnings: []}; }
|
||||
jsonFile(path.join(run.out, 'validation.json'), validation, stage.apiKey);
|
||||
writeFileSync(path.join(run.out, 'report.html'), redact(understandingReport(image, data, validation, metadata), stage.apiKey));
|
||||
if (!validation.valid) throw new Error('结构理解结果未通过本地校验，请查看 validation.json、response.raw.json 和 report.html');
|
||||
jsonFile(path.join(run.out, 'semantic-parts.json'), data, stage.apiKey);
|
||||
writeFileSync(path.join(run.out, 'annotated.svg'), redact(annotatedSvg(image, data.parts), stage.apiKey));
|
||||
const geometry = groundParts(runtime, run.out);
|
||||
// Geometry failures block individual SAM prompts, while retaining the plan
|
||||
// and diagnostics for correction. They never fabricate fallback points.
|
||||
validation.geometry = geometry.summary;
|
||||
for (const part of geometry.parts) for (const warning of part.warnings) validation.warnings.push(`${part.id}: ${warning}`);
|
||||
jsonFile(path.join(run.out, 'validation.json'), validation, stage.apiKey);
|
||||
writeFileSync(path.join(run.out, 'report.html'), redact(understandingReport(image, data, validation, metadata, geometry), stage.apiKey));
|
||||
writeFileSync(path.join(run.out, 'annotated.svg'), redact(annotatedSvg(image, data.parts, geometry), stage.apiKey));
|
||||
manifest.request = metadata;
|
||||
manifest.quality_status = 'needs_review';
|
||||
manifest.geometry = geometry.summary;
|
||||
manifest.completed_at = new Date().toISOString();
|
||||
finishRun(run, manifest, ['source.png', 'prompt.txt', 'request.json', 'response.raw.json', 'response.txt', 'request-metadata.json', 'validation.json', 'report.html', 'semantic-parts.json', 'annotated.svg'], stage.apiKey);
|
||||
finishRun(run, manifest, ['source-original.png', 'source.png', 'preprocessing.json', 'geometry-validation.json', 'segmentation-prompts.json', 'prompt.txt', 'request.json', 'response.raw.json', 'response.txt', 'request-metadata.json', 'validation.json', 'report.html', 'semantic-parts.json', 'annotated.svg'], stage.apiKey);
|
||||
return {out: run.out, reused: false, parts: data.parts.length};
|
||||
} catch (error) {
|
||||
if (error.diagnostics) jsonFile(path.join(run.out, 'response.error.json'), error.diagnostics, stage.apiKey);
|
||||
|
||||
@@ -8,7 +8,8 @@ import {execFile} from 'node:child_process';
|
||||
import {promisify} from 'node:util';
|
||||
import {loadConfig, resolveStage, projectRoot, publicStage} from '../lib/config.mjs';
|
||||
import {readImage} from '../lib/images.mjs';
|
||||
import {validateParts, partsSchema} from '../lib/validate-parts.mjs';
|
||||
import {validateParts, partsSchema, upgradeParts} from '../lib/validate-parts.mjs';
|
||||
import {geometryPython} from '../lib/geometry.mjs';
|
||||
import {requestJson, parseEventStream} from '../providers/http.mjs';
|
||||
import {buildVisionRequest, extractVisionText} from '../providers/vision.mjs';
|
||||
import {editImage, decodeEditedImages, validateEditInputs} from '../providers/image-edits.mjs';
|
||||
@@ -17,8 +18,10 @@ import {runUnderstanding, runCompletionProbe} from '../stages/model-stage.mjs';
|
||||
const exec = promisify(execFile);
|
||||
const reference = path.join(projectRoot, 'references', 'characters', 'kenney-beige-idle.png');
|
||||
const image = readImage(reference, {pngOnly: true});
|
||||
const part = (id, parent = null) => ({id, semantic: id, parent, visibility: 'visible', bbox_norm_xyxy: [0.1, 0.1, 0.9, 0.9], pivot_norm: [0.5, 0.5], positive_points_norm: [[0.5, 0.5]], negative_points_norm: [], occluded_by: [], needs_completion: false, confidence: 0.8});
|
||||
const goodParts = () => ({schema_version: 1, coordinate_space: 'normalized_xy_top_left', parts: [part('torso'), part('head', 'torso')], notes: []});
|
||||
const part = (id, parent = null) => ({id, semantic: id, parent, rig_role: 'bone_part', bind_to: null,
|
||||
completion: {occlusion_fill: false, joint_extension_parent: false, joint_extension_child: false},
|
||||
visibility: 'visible', bbox_norm_xyxy: [0.1, 0.1, 0.9, 0.9], pivot_norm: [0.5, 0.5], positive_points_norm: [], negative_points_norm: [], occluded_by: [], needs_completion: false, confidence: 0.8});
|
||||
const goodParts = () => ({schema_version: 2, coordinate_space: 'normalized_xy_top_left', parts: [part('torso'), part('head', 'torso')], notes: []});
|
||||
const jsonResponse = (payload, status = 200) => new Response(JSON.stringify(payload), {status, headers: {'Content-Type': 'application/json', 'x-request-id': 'test-request'}});
|
||||
const stageFor = (name = 'understanding') => resolveStage(loadConfig({envFile: '__no_test_env__', env: {VECTOR_ENGINE_BASE_URL: 'https://relay.invalid/v1', VECTOR_ENGINE_API_KEY: 'test-secret'}}), name, name === 'understanding' ? {protocol: 'chat-completions'} : {});
|
||||
const chatPayload = (parts = goodParts()) => ({model: 'gpt-6-astra', choices: [{finish_reason: 'stop', message: {content: JSON.stringify(parts)}}], usage: {total_tokens: 12}});
|
||||
@@ -77,7 +80,10 @@ test('schema + semantic validation reject invalid coordinates, references and bo
|
||||
(d) => { d.parts[0].occluded_by = ['head']; d.parts[1].occluded_by = ['torso']; },
|
||||
(d) => { d.parts[1].id = 'torso'; },
|
||||
(d) => { d.parts[0].visibility = 'hidden'; },
|
||||
(d) => { d.parts[0].positive_points_norm = [[0, 0]]; },
|
||||
(d) => { d.parts[0].positive_points_norm = [[-0.1, 0]]; },
|
||||
(d) => { d.parts[0].bind_to = 'head'; },
|
||||
(d) => { d.parts[0].completion.occlusion_fill = true; },
|
||||
(d) => { d.parts[0].visibility = 'partial'; },
|
||||
(d) => { d.parts[0].unknown = 1; },
|
||||
]) {
|
||||
const data = goodParts(); mutate(data);
|
||||
@@ -85,6 +91,39 @@ test('schema + semantic validation reject invalid coordinates, references and bo
|
||||
}
|
||||
});
|
||||
|
||||
test('v1 migration separates facial attachments and completion, while preserving pivots and diagnostics', () => {
|
||||
const data = goodParts();
|
||||
data.parts.push({...part('eye_left', 'head'), visibility: 'partial'});
|
||||
data.schema_version = 1;
|
||||
for (const p of data.parts) { delete p.rig_role; delete p.bind_to; delete p.completion; }
|
||||
data.parts[0].positive_points_norm = [[0, 0]];
|
||||
assert.equal(validateParts(data).valid, true);
|
||||
const migrated = upgradeParts(data);
|
||||
assert.equal(validateParts(migrated).valid, true);
|
||||
assert.equal(migrated.parts[2].rig_role, 'facial_feature');
|
||||
assert.equal(migrated.parts[2].bind_to, 'head');
|
||||
assert.equal(migrated.parts[2].completion.occlusion_fill, true);
|
||||
assert.equal(migrated.parts[2].needs_completion, true);
|
||||
assert.equal(migrated.parts[0].completion.joint_extension_child, true);
|
||||
assert.deepEqual(migrated.parts.map((p) => p.pivot_norm), data.parts.map((p) => p.pivot_norm));
|
||||
assert.equal(data.parts[2].needs_completion, false);
|
||||
});
|
||||
|
||||
test('attachments cannot bind to other attachments or declare bone overlap', () => {
|
||||
const data = goodParts();
|
||||
data.parts.push({...part('eye_left', 'head'), rig_role: 'facial_feature', bind_to: 'head'});
|
||||
assert.equal(validateParts(data).valid, true);
|
||||
data.parts[2].bind_to = 'missing';
|
||||
assert.equal(validateParts(data).valid, false);
|
||||
data.parts[2].bind_to = 'head';
|
||||
data.parts[1].rig_role = 'attachment';
|
||||
assert.equal(validateParts(data).valid, false);
|
||||
data.parts[1].rig_role = 'bone_part';
|
||||
data.parts[2].completion.joint_extension_parent = true;
|
||||
data.parts[2].needs_completion = true;
|
||||
assert.equal(validateParts(data).valid, false);
|
||||
});
|
||||
|
||||
test('SSE requires terminal completion, supports Responses and Chat, rejects truncated/error streams', async () => {
|
||||
const payload = {status: 'completed', model: 'gpt-6-astra', output: [{content: [{type: 'output_text', text: '{}'}]}]};
|
||||
const stream = `event: response.completed\r\ndata: ${JSON.stringify({type: 'response.completed', response: payload})}\r\n\r\ndata: [DONE]\r\n\r\n`;
|
||||
@@ -156,6 +195,11 @@ test('understanding run saves report and provenance; resume checks inputs/model/
|
||||
assert.equal(manifest.quality_status, 'needs_review');
|
||||
assert.equal(manifest.request.returned_model, 'gpt-6-astra');
|
||||
assert.equal(manifest.inputs[0].width, 128);
|
||||
assert.equal(manifest.geometry.candidate_parts, 2);
|
||||
const prompts = JSON.parse(readFileSync(path.join(outDir, 'segmentation-prompts.json')));
|
||||
assert.equal(prompts.parts[0].semantic_verified, false);
|
||||
assert.equal(prompts.parts[0].positive_points_px.length, 1);
|
||||
assert.equal(JSON.parse(readFileSync(path.join(outDir, 'semantic-parts.json'))).parts[0].positive_points_norm.length, 0);
|
||||
assert.match(readFileSync(path.join(outDir, 'report.html'), 'utf8'), /尚未执行 SAM/u);
|
||||
await assert.rejects(runUnderstanding(options, transport), /非空/u);
|
||||
assert.equal((await runUnderstanding({...options, resume: true}, transport)).reused, true);
|
||||
@@ -202,7 +246,7 @@ test('completion probe produces raw candidates and never claims final assets', a
|
||||
assert.deepEqual(readFileSync(path.join(outDir, 'candidate-1.png')), image.bytes);
|
||||
});
|
||||
|
||||
test('pipeline new mode reaches a mock relay without Python; model listing and unsupported stages', async (t) => {
|
||||
test('pipeline new mode reaches a mock relay with geometry; model listing and unsupported stages', async (t) => {
|
||||
const dir = temporary(t);
|
||||
let requests = 0;
|
||||
const server = createServer(async (req, res) => {
|
||||
@@ -218,7 +262,7 @@ test('pipeline new mode reaches a mock relay without Python; model listing and u
|
||||
t.after(() => new Promise((resolve) => server.close(resolve)));
|
||||
const env = {...process.env, VECTOR_ENGINE_BASE_URL: `http://127.0.0.1:${server.address().port}`, VECTOR_ENGINE_API_KEY: 'mock-key'};
|
||||
const outDir = path.join(dir, 'cli');
|
||||
const result = await exec(process.execPath, ['pipeline.mjs', '--mode', 'annotated-parts', '--reference', reference, '--out-dir', outDir, '--stop-after', 'understand', '--understanding-model', 'relay-vision-test', '--python', 'deliberately-nonexistent-python'], {cwd: projectRoot, env});
|
||||
const result = await exec(process.execPath, ['pipeline.mjs', '--mode', 'annotated-parts', '--reference', reference, '--out-dir', outDir, '--stop-after', 'understand', '--understanding-model', 'relay-vision-test'], {cwd: projectRoot, env});
|
||||
assert.match(result.stdout, /结构理解完成/u);
|
||||
const list = await exec(process.execPath, ['tools/model-check.mjs', '--list-models'], {cwd: projectRoot, env});
|
||||
assert.match(list.stdout, /relay-vision-test/u);
|
||||
@@ -226,3 +270,23 @@ test('pipeline new mode reaches a mock relay without Python; model listing and u
|
||||
await assert.rejects(exec(process.execPath, ['pipeline.mjs', '--mode', 'annotated-parts', '--stop-after', 'segment'], {cwd: projectRoot, env}), (error) => /仅支持 understand/u.test(error.stderr));
|
||||
assert.equal(requests, 2);
|
||||
});
|
||||
|
||||
test('offline CLI needs no credentials or network and fingerprints imported plans', async (t) => {
|
||||
const dir = temporary(t);
|
||||
const plan = path.join(dir, 'plan.json'); writeFileSync(plan, JSON.stringify(goodParts()));
|
||||
const envFile = path.join(dir, 'empty.env'); writeFileSync(envFile, '');
|
||||
const env = {...process.env, VECTOR_ENGINE_BASE_URL: '', VECTOR_ENGINE_API_KEY: ''};
|
||||
const args = ['tools/model-check.mjs', '--env-file', envFile, '--reference', reference,
|
||||
'--parts-file', plan, '--out-dir', path.join(dir, 'offline')];
|
||||
await exec(process.execPath, args, {cwd: projectRoot, env});
|
||||
const metadata = JSON.parse(readFileSync(path.join(dir, 'offline', 'request-metadata.json')));
|
||||
assert.equal(metadata.network_requested, false);
|
||||
await exec(process.execPath, [...args, '--resume'], {cwd: projectRoot, env});
|
||||
writeFileSync(plan, JSON.stringify({...goodParts(), notes: ['changed']}));
|
||||
await assert.rejects(exec(process.execPath, [...args, '--resume'], {cwd: projectRoot, env}), (e) => /变化/u.test(e.stderr));
|
||||
});
|
||||
|
||||
test('Python geometry regression suite', async () => {
|
||||
const result = await exec(geometryPython(), ['-m', 'unittest', 'discover', '-s', 'tests', '-p', 'test_geometry.py'], {cwd: projectRoot});
|
||||
assert.match(result.stderr, /OK/u);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
import copy
|
||||
import unittest
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from geometry_grounding import ground_parts, pixel
|
||||
from normalize_reference import prepare_rgba
|
||||
|
||||
|
||||
def plan(box=None, points=None):
|
||||
return {'parts': [{'id': 'hand_left', 'semantic': 'screen-left hand',
|
||||
'visibility': 'visible', 'bbox_norm_xyxy': box or [0, 0, 1, 1],
|
||||
'positive_points_norm': points or [], 'pivot_norm': [0, 0]}]}
|
||||
|
||||
|
||||
class GeometryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.rgba = np.zeros((20, 30, 4), dtype=np.uint8)
|
||||
self.rgba[5:15, 10:20] = [20, 40, 60, 255]
|
||||
self.image = Image.fromarray(self.rgba)
|
||||
|
||||
def test_existing_alpha_and_visible_pixels_unchanged(self):
|
||||
self.rgba[6, 12] = [255, 255, 255, 128]
|
||||
image, meta = prepare_rgba(Image.fromarray(self.rgba))
|
||||
np.testing.assert_array_equal(image, self.rgba)
|
||||
self.assertEqual(meta['method'], 'existing_alpha')
|
||||
|
||||
def test_only_border_connected_white_is_removed(self):
|
||||
rgb = np.full((20, 20, 3), 255, dtype=np.uint8)
|
||||
rgb[4:16, 4:16] = 20
|
||||
rgb[7:13, 7:13] = 255
|
||||
prepared, meta = prepare_rgba(Image.fromarray(rgb))
|
||||
data = np.array(prepared)
|
||||
self.assertEqual(data[0, 0, 3], 0)
|
||||
self.assertEqual(data[8, 8, 3], 255)
|
||||
np.testing.assert_array_equal(data[:, :, :3], rgb)
|
||||
self.assertEqual(meta['method'], 'border_connected_white')
|
||||
self.assertEqual(prepared.size, (20, 20))
|
||||
|
||||
def test_colored_opaque_and_blank_inputs_rejected(self):
|
||||
for color in [(0, 30, 80, 255), (255, 255, 255, 255), (0, 0, 0, 0)]:
|
||||
with self.assertRaises(ValueError):
|
||||
prepare_rgba(Image.new('RGBA', (10, 10), color))
|
||||
|
||||
def test_invalid_model_points_replaced_without_changing_pivot(self):
|
||||
data = plan(points=[[0, 0], [1, 1], [.4, .4]])
|
||||
original = copy.deepcopy(data)
|
||||
report, prompts = ground_parts(self.image, data)
|
||||
self.assertEqual(report['summary']['invalid_model_points'], 2)
|
||||
self.assertEqual(report['summary']['background_model_points'], 2)
|
||||
candidate = report['parts'][0]['candidate_points'][0]
|
||||
x, y = candidate['point_px']
|
||||
self.assertGreater(self.rgba[y, x, 3], 0)
|
||||
self.assertGreaterEqual(candidate['distance_to_roi_boundary_px'], 5)
|
||||
self.assertEqual(pixel(candidate['point_norm'], 30, 20), (x, y))
|
||||
self.assertFalse(prompts['parts'][0]['semantic_verified'])
|
||||
self.assertEqual(prompts['parts'][0]['negative_points_px'], [])
|
||||
self.assertEqual(data, original)
|
||||
self.assertEqual(report['parts'][0]['model_positive_points'][2]['status'], 'UNTRUSTED_SEED')
|
||||
|
||||
def test_bbox_with_no_foreground_has_no_seed(self):
|
||||
report, prompts = ground_parts(self.image, plan([0, 0, .2, .2]))
|
||||
self.assertEqual(report['summary']['blocked_parts'], 1)
|
||||
self.assertEqual(prompts['parts'][0]['positive_points_px'], [])
|
||||
self.assertIn('EMPTY_FOREGROUND_ROI', report['parts'][0]['warnings'])
|
||||
|
||||
def test_solid_roi_has_finite_interior_distance(self):
|
||||
report, _ = ground_parts(self.image, plan([10/30, 5/20, 20/30, 15/20]))
|
||||
part = report['parts'][0]
|
||||
self.assertEqual(part['bbox_foreground_ratio'], 1)
|
||||
self.assertEqual(part['candidate_points'][0]['distance_to_roi_boundary_px'], 5)
|
||||
|
||||
def test_disconnected_roi_remains_semantically_unverified(self):
|
||||
self.rgba[2:4, 2:4] = [255, 0, 0, 255]
|
||||
report, _ = ground_parts(Image.fromarray(self.rgba), plan())
|
||||
self.assertEqual(report['parts'][0]['foreground_components'], 2)
|
||||
self.assertIn('MULTIPLE_FOREGROUND_COMPONENTS', report['parts'][0]['warnings'])
|
||||
self.assertFalse(report['parts'][0]['semantic_verified'])
|
||||
|
||||
def test_opaque_image_does_not_validate_every_point(self):
|
||||
report, prompts = ground_parts(Image.new('RGBA', (30, 20), 'black'), plan(points=[[.5, .5]]))
|
||||
self.assertEqual(report['parts'][0]['status'], 'blocked')
|
||||
self.assertEqual(prompts['parts'][0]['positive_points_px'], [])
|
||||
|
||||
def test_subpixel_roi_without_pixel_center_is_blocked(self):
|
||||
report, _ = ground_parts(self.image, plan([.4, .4, .401, .401]))
|
||||
self.assertEqual(report['parts'][0]['status'], 'blocked')
|
||||
self.assertIn('NO_PIXEL_CENTER_IN_ROI', report['parts'][0]['warnings'])
|
||||
|
||||
def test_hidden_part_never_gets_foreground_seed(self):
|
||||
data = plan()
|
||||
data['parts'][0].update(visibility='hidden', bbox_norm_xyxy=None)
|
||||
report, prompts = ground_parts(self.image, data)
|
||||
self.assertEqual(report['parts'][0]['status'], 'hidden')
|
||||
self.assertEqual(prompts['parts'][0]['positive_points_px'], [])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -21,6 +21,8 @@ const help = `模型接入与结构理解（新增模式，不调用旧拆件流
|
||||
--stream true|false 覆盖理解阶段的流式设置
|
||||
--output-mode NAME text-json / json-object / json-schema
|
||||
--resume 输入/配置/代码和产物哈希相同时复用
|
||||
--parts-file FILE 离线读取已有 v1/v2 计划，做 Alpha 校验与种子生成，不请求模型
|
||||
--python FILE ai-spine Conda 环境解释器（默认使用已激活环境）
|
||||
--image FILE 图片编辑输入，可重复；多图需配置 capabilities
|
||||
--mask FILE 透明处可编辑的 RGBA PNG；需配置 capabilities
|
||||
--dry-run 检查配置与本地输入，不请求模型、不生成产物
|
||||
@@ -34,7 +36,7 @@ export function parseOptions(argv) {
|
||||
['--understanding-model', 'understandingModel'], ['--completion-model', 'completionModel'],
|
||||
['--understanding-protocol', 'understandingProtocol'], ['--output-mode', 'outputMode'],
|
||||
['--stage', 'stage'], ['--mask', 'maskPath'], ['--prompt-file', 'promptFile'], ['--stop-after', 'stopAfter'],
|
||||
['--mode', 'mode'], ['--python', 'python'], ['--stream', 'streamRaw'],
|
||||
['--mode', 'mode'], ['--python', 'python'], ['--stream', 'streamRaw'], ['--parts-file', 'partsFile'],
|
||||
]);
|
||||
for (let i = 0; i < argv.length; i += 1) {
|
||||
const flag = argv[i];
|
||||
@@ -56,6 +58,7 @@ export async function main(argv = process.argv.slice(2)) {
|
||||
const options = parseOptions(argv);
|
||||
if (options.help) { console.log(help); return; }
|
||||
if (!['understanding', 'completion'].includes(options.stage)) throw new Error('--stage 仅支持 understanding 或 completion');
|
||||
if (options.partsFile && (options.stage !== 'understanding' || options.listModels)) throw new Error('--parts-file 仅用于离线结构理解');
|
||||
if (options.streamRaw !== undefined && !['true', 'false'].includes(options.streamRaw)) throw new Error('--stream 仅支持 true 或 false');
|
||||
options.stream = options.streamRaw === undefined ? undefined : options.streamRaw === 'true';
|
||||
if (options.mode && options.mode !== 'annotated-parts') throw new Error('新入口仅支持 --mode annotated-parts');
|
||||
@@ -66,7 +69,7 @@ export async function main(argv = process.argv.slice(2)) {
|
||||
const overrides = options.stage === 'understanding'
|
||||
? {model: options.understandingModel, protocol: options.understandingProtocol, outputMode: options.outputMode, stream: options.stream}
|
||||
: {model: options.completionModel};
|
||||
const stage = resolveStage(loaded, options.stage, overrides, {requireKey: !options.dryRun});
|
||||
const stage = resolveStage(loaded, options.stage, overrides, {requireKey: !options.dryRun && !options.partsFile});
|
||||
if (options.listModels) {
|
||||
if (options.dryRun) throw new Error('--list-models 会请求模型列表，不能与 --dry-run 同时使用');
|
||||
const {payload} = await requestJson(stage, 'models', () => ({method: 'GET'}));
|
||||
@@ -87,7 +90,7 @@ export async function main(argv = process.argv.slice(2)) {
|
||||
if (options.stage === 'understanding') {
|
||||
if (!options.reference) throw new Error('结构理解需要 --reference');
|
||||
if (options.imagePaths.length || options.maskPath || options.promptFile) throw new Error('结构理解不接受图片编辑参数');
|
||||
console.log(`角色结构理解：${stage.model} (${stage.protocol})`);
|
||||
console.log(options.partsFile ? '角色结构理解：离线计划几何校验' : `角色结构理解：${stage.model} (${stage.protocol})`);
|
||||
result = await runUnderstanding({...options, loaded, stage});
|
||||
console.log(`${result.reused ? '复用已校验产物' : '结构理解完成，语义待复核'}：${path.join(result.out, 'report.html')}`);
|
||||
} else {
|
||||
|
||||
Reference in New Issue
Block a user