补齐外部生成服务OpenSSL路径

为worker与controller systemd单元补齐LD_LIBRARY_PATH

避免服务器动态链接OpenSSL失败

.dockerignore

@@ -22,6 +22,7 @@ tmp
 .env.secrets.*
 spacetime.local.json
 deploy/container/api-server.env
+deploy/container/worker-smoke
 
 server-rs/target
 server-rs/target-*

.gitignore

@@ -42,6 +42,7 @@ temp*build*/
 .env.secrets.local
 spacetime.local.json
 deploy/container/api-server.env
+deploy/container/worker-smoke/
 
 # Local load-test data extracted from private migration files
 scripts/loadtest/data/*.local.json

.hermes/shared-memory/decision-log.md

@@ -16,6 +16,14 @@
 
 ---
 
+## 2026-06-11 本地服务器管理入口采用 SSH alias + egui 桌面面板
+
+- 背景：release / dev 等服务器的日常巡检已有 systemd、健康巡检 timer 和 HTTP 探测口径，但开发者本地仍需要在多个 SSH alias 间手工切换命令并重复执行启停操作。
+- 决策：新增 `server-rs/crates/server-manager-panel` 作为本地 egui 桌面工具；服务器来源只读取本机 `~/.ssh/config` 的具体 `Host` alias，不保存服务器密钥或凭据；巡检通过 `ssh <alias> sh -s` 执行只读脚本，服务操作只允许 `start`、`stop`、`restart` 并限制 systemd unit 名字符集。
+- 影响范围：本地运维工具入口、`package.json` 的 `server-manager:panel`、开发运维文档和团队共享工作流。
+- 验证方式：`cargo check -p server-manager-panel --manifest-path server-rs/Cargo.toml`、`cargo test -p server-manager-panel --manifest-path server-rs/Cargo.toml`、`npm run check:encoding`。
+- 关联文档：`docs/technical/【开发运维】本地SSH服务器管理面板技术方案-2026-06-11.md`。
+
 ## 2026-06-10 公开作品互动能力进入后台全局配置
 
 - 背景：作品详情页的点赞和改造能力原本由前端和各玩法 handler 的硬编码能力矩阵决定，后台无法临时关闭某类公开作品的互动入口，直接关闭创作入口又会误伤已有作品读取和游玩。
@@ -32,6 +40,13 @@
 - 验证方式：从 release 执行 `git ls-remote http://10.2.0.10/GenarrativeAI/Genarrative.git HEAD` 应返回 HEAD；公网来源伪造 `Host: 10.2.0.10` 访问 dev 公网 80 应返回 `403`；`https://git.genarrative.world/` 原入口应保持 `200`。
 - 关联文档：`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
 
+## 2026-06-08 通用分享统一为作品分享卡片
+
+- 背景：已发布作品的分享入口需要同时支持网页复制链接、下载可传播的分享卡，以及微信小程序内的九宫切图；推荐页在小程序内直接使用系统“分享到聊天”时，宿主快照只截页面中部，容易裁掉游戏主体，且原生分享默认只能拿到小程序页面启动参数。
+- 决策：统一分享入口继续收口到 `PublishShareModal`，分享卡展示作品封面、作品类型、作品名称和公开作品号，底部提供“复制链接”和“下载卡片”。普通 H5 复制公开作品 H5 URL；微信小程序 WebView 内复制小程序 `pages/web-view/index` 路径，缺少直达参数时补 `targetPath=/works/detail` 与 `work=<公开作品号>`，由小程序原生 WebView 页转成 H5 作品详情 URL。当 H5 运行在微信小程序 WebView 内且存在封面图时，额外显示“九宫切图”，跳转小程序原生 `pages/share-grid/index`，由原生页按 3x3 从左到右、从上到下裁切并保存。推荐页当前作品会通过 `wx.miniProgram.postMessage` 同步给小程序原生 `web-view` 页，右上角系统分享优先使用该目标生成带作品参数的小程序路径。小程序运行态通过根节点标记启用推荐页 runtime 快照安全区，把游戏画面等比缩放到分享快照中部。
+- 影响范围：`src/components/common/PublishShareModal.tsx`、`src/components/common/publishShareModalModel.ts`、`src/components/common/publishShareCardImage.ts`、`src/services/wechatMiniProgramShareGrid.ts`、`src/services/wechatMiniProgramShareTarget.ts`、`miniprogram/pages/web-view/`、`miniprogram/pages/share-grid/`、推荐页 runtime CSS 和平台玩法链路文档。
+- 验证方式：`npm run test -- src/components/common/PublishShareModal.test.tsx miniprogram/pages/web-view/index.test.js src/services/wechatMiniProgramShareTarget.test.ts`、`npm run test -- miniprogram/pages/share-grid/index.test.js`、`npm run test -- src/index.test.ts -t "mini program recommend runtime"`、`npm run typecheck`、`npm run check:encoding`。
+- 关联文档：`docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`。
 ## 2026-06-08 微信能力按领域收口
 
 - 背景：微信登录、订阅消息、普通微信支付和小程序虚拟支付能力曾分散在 `api-server` 根模块、`platform-auth` 与 `platform-wechat`，支付协议细节和业务 handler 边界不够清晰。
@@ -116,7 +131,7 @@
 ## 2026-06-05 Server-Provision 全程在目标部署 agent 执行且不安装构建链
 
 - 背景：`Genarrative-Server-Provision` 的 `DEPLOY_TARGET=development` 语义是部署到 dev 服务器，不是构建机 dry-run。旧流水线把 development 映射到 `linux && genarrative-build`，还先在 build 节点准备 `provision-tools/` 再 stash 给后续阶段，导致真实 dev 初始化可能跑到 Jenkins controller / build 节点；脚本还安装 clang / lld / pkg-config / OpenSSL headers / sccache 等构建链依赖，超出了服务器初始化职责。
-- 决策：Server-Provision 只做服务器初始化，全程运行在目标部署 agent：development 使用 `linux && genarrative-dev-deploy`，release 使用 `linux && genarrative-release-deploy`。`Prepare Provision Tools` 与 `Provision Server` 在同一个目标 agent workspace 顺序执行，不再切到 `linux && genarrative-build`，不再 `stash/unstash` 工具包。`scripts/jenkins-server-provision.sh` 不再安装 clang / lld / pkg-config / libssl-dev / sccache；非 dry-run 仍要求目标 dev / release agent 具备 root 权限，因为 provision 会写 systemd、Nginx、`/etc` 和系统用户。Job 的 `Pipeline script from SCM` 与 Jenkinsfile 参数 `SOURCE_GIT_REMOTE_URL` 都必须使用本机路径或目标 agent 可访问的内网 Git 源，不允许公网 Git fallback。
+- 决策：Server-Provision 只做服务器初始化，全程运行在目标部署 agent：development 使用 `linux && genarrative-dev-deploy`，release 使用 `linux && genarrative-release-deploy`。`Prepare Provision Tools` 与 `Provision Server` 在同一个目标 agent workspace 顺序执行，不再切到 `linux && genarrative-build`，不再 `stash/unstash` 工具包。`scripts/jenkins-server-provision.sh` 不再安装 clang / lld / pkg-config / libssl-dev / sccache；当前 OpenSSL 3.2 独立运行时自举会安装 `build-essential` 等最小工具，这是满足 api-server/libcurl 运行时符号的受控例外，不代表 provision 承担 api-server 构建职责。非 dry-run 仍要求目标 dev / release agent 具备 root 权限，因为 provision 会写 systemd、Nginx、`/etc` 和系统用户。Job 的 `Pipeline script from SCM` 与 Jenkinsfile 参数 `SOURCE_GIT_REMOTE_URL` 都必须使用本机路径或目标 agent 可访问的内网 Git 源，不允许公网 Git fallback。
 - 追加决策（2026-06-10）：`Prepare Provision Tools` 必须先读取目标机现状，再准备需要的文件。目标机 `/usr/local/bin/otelcol-contrib` 版本匹配 `OTELCOL_VERSION` 时直接复用；`${SPACETIME_ROOT}/bin/current/spacetimedb-cli` 和 `spacetimedb-standalone` 存在且 CLI 版本匹配 `SPACETIME_EXPECTED_VERSION` 或 `SPACETIME_DOWNLOAD_ROOT` 中的版本时，直接复用当前安装生成 `provision-tools/`。只有目标机缺失、不可执行或版本不匹配时，才消费 `PROVISION_DOWNLOADS_DIR` 中的本地包或进入下载分支。
 - 影响范围：`jenkins/Jenkinsfile.production-server-provision`、`scripts/jenkins-server-provision.sh`、生产运维文档、Server-Provision 排障口径。
 - 验证方式：Jenkins 日志中 Server-Provision 的 `Prepare`、`Checkout Provision Files`、`Prepare Provision Tools` 和 `Provision Server` 都在目标 dev / release agent 上执行；日志不出现 `Running on Jenkins`、`linux && genarrative-build`、`stash 'server-provision-tools'`、`Git 主地址拉取失败...改用备用地址`、`https://git.genarrative.world/GenarrativeAI/Genarrative.git` 或构建依赖 / sccache 安装步骤；`bash -n scripts/jenkins-server-provision.sh` 和编码检查通过。
@@ -154,6 +169,24 @@
 - 验证方式：关闭任一创作入口后，新建创作请求返回 `creation_entry_disabled`；公开作品列表 / 详情 / 启动 / 运行态动作不返回该错误；进入平台首页不弹“平台首页：creation_entry_disabled”；关闭态入口卡显示锁定状态且不显示 `10-20泥点数`。
 - 关联文档：`docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`。
 
+## 2026-06-03 外部内容生成改为持久队列加 worker 角色
+
+- 背景：拼图首图、图集、音频等外部生成链路长期占用 `api-server` HTTP handler，导致扩容只能放大 API 进程，且 HTTP 超时和外部 provider 波动会直接影响创作入口。
+- 决策：外部生成任务统一进入 SpacetimeDB `external_generation_job` 持久队列，由 `api-server` 的 `external-generation-worker` 进程角色 claim lease 后执行；HTTP 角色只做鉴权、表单/状态初始化、入队和返回 `queued/running/completed/failed` 操作状态。生产通过 systemd worker 模板增加实例数或提高 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩容，`GENARRATIVE_PROCESS_ROLE=all` 仅用于本地 smoke。拼图 `compile_puzzle_draft`、结果页 `generate_puzzle_images` 与 `generate_puzzle_ui_background` 已接入 worker；业务写回必须在 SpacetimeDB transaction 内校验 `external_generation_job` 的 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，其中首图 worker 的前置 `compile_puzzle_agent_draft` 也必须带 guard。worker 核心业务写回失败不能返回内存快照并把 job 标成 completed；失败态业务写回成功后才能把 job 标成 failed，失败态未写回则保留租约等待后续重领。拼图业务失败不自动重试，只保留 lease 过期后的崩溃重领，避免钱包扣退费幂等漂移。生产发布会启用默认 `genarrative-external-generation-worker@1.service` 并等待 worker active，worker 停机时停止 claim 新任务并 drain 当前任务。
+- 2026-06-07 追加：`GENARRATIVE_EXTERNAL_GENERATION_MODE` 使用 `queue|inline` 显式策略；生产和容器扩缩容验证保持 `queue`。本地开发若需要同步等待结果，应通过 `.env.local` 或本机环境显式配置为 `inline`，由 HTTP handler 复用同一 worker executor 直接返回 `completed`，不创建 `external_generation_job`，不支持 worker 动态扩缩容；脚本不得硬编码该策略。拼图写回 guard 字段改为可选，queue 路径仍必须完整校验 `job_id + worker_id + lease_token`；inline 路径只允许三项同时为空，半空 guard 仍拒绝。
+- 2026-06-11 追加：生产新增固定 `external-generation-controller` 进程角色和 `genarrative-external-generation-controller.service`。controller 只读取 `get_external_generation_queue_stats_and_return` 队列统计并管理 `genarrative-external-generation-worker@N.service`，不监听 HTTP、不执行外部生成任务；默认保留 `@1`，按 `claimable_pending + running_active + expired_running` 计算目标实例数，上限由 `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MAX_WORKERS` 控制，缩容需要连续空闲轮数且每轮只停最高编号一个实例。
+- 影响范围：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/spacetime-client/src/external_generation.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/external_generation_worker_controller.rs`、`deploy/systemd/genarrative-external-generation-worker@.service`、`deploy/systemd/genarrative-external-generation-controller.service`、`deploy/env/external-generation-controller.env.example`、`scripts/deploy/production-api-deploy.sh`、`scripts/jenkins-server-provision.sh`、拼图 `compile_puzzle_draft`、拼图 `generate_puzzle_images`、拼图 `generate_puzzle_ui_background`、生产 env 模板和运维文档。
+- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`npm run check:server-rs-ddd`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`，并在 queue 模式下用 `GENARRATIVE_PROCESS_ROLE=all npm run dev` smoke 至少一次 queued -> worker 完成链路；本地 inline 排查只确认不创建 `external_generation_job`。
+- 关联文档：`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`、`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`。
+
+## 2026-06-03 外部生成 worker lease 使用 SpacetimeDB 时间和 token 栅栏
+
+- 背景：外部生成 worker 支持多进程动态缩扩容后，长任务超过单次 lease、worker 本机时钟漂移或复用 worker id 都可能导致同一任务被重复领取并被过期执行者回写。
+- 决策：`external_generation_job` 新增末尾字段 `lease_token`；`claim` 使用 SpacetimeDB `ctx.timestamp` 计算 lease，生成本次 claim token；worker 执行期间调用 `renew_external_generation_job_lease_and_return` 续租；`complete/fail` 必须带 `worker_id + lease_token` 才能回写。拼图 `compile_puzzle_draft` 的 dedupe key 包含本次 `extgen-` job id，避免同一 session 的失败或完成 job 吞掉后续重新生成。拼图首图前置 `compile_puzzle_agent_draft`、图片保存、UI 背景与失败态业务写回同样必须携带 lease guard，并在 `compile_puzzle_agent_draft`、`save_puzzle_generated_images`、`save_puzzle_ui_background`、`mark_puzzle_draft_generation_failed`、`mark_puzzle_level_generation_failed` 的 SpacetimeDB 事务内校验。
+- 影响范围：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/spacetime-module/src/puzzle.rs`、`server-rs/crates/module-puzzle/src/commands.rs`、`server-rs/crates/spacetime-client/src/external_generation.rs`、`server-rs/crates/spacetime-client/src/puzzle.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/puzzle/handlers.rs`、`server-rs/crates/api-server/src/puzzle/draft.rs`、`server-rs/crates/api-server/src/puzzle/generation.rs`。
+- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`cargo test -p spacetime-module external_generation --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml`、`GENARRATIVE_PROCESS_ROLE=all npm run dev` 后检查 `/healthz`。
+- 关联文档：`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
+
 ## 2026-06-04 Draft Generation Shelf 剩余草稿打开 intent 收口
 
 - 背景：拼图 / 抓大鹅草稿打开 intent 已归入 `platformDraftGenerationShelfModel.ts`，但方洞挑战、大鱼吃小鱼和视觉小说仍在平台壳层内联判断已发布详情、缺 session、active generating、当前结果页和普通草稿恢复。
@@ -279,7 +312,7 @@
 - 背景：Match3D、SquareHole、Puzzle、Jump Hop 等 runtime client 重复手写 path segment 编码、JSON header / body、runtime guest token、auth options 和 retry options，新增玩法容易遗漏同一请求骨架。
 - 决策：新增 `src/services/runtimeRequest.ts`，以 `buildRuntimeApiPath` 统一 runtime path 编码，以 `requestRuntimeJson` 统一 JSON 请求、runtime guest auth 和 retry 合并。Match3D 与 SquareHole runtime client 已先迁移，保留原导出函数名、错误文案、返回契约和重试常量。
 - 追加决策：Big Fish 与 Bark Battle runtime client 也迁入 `runtimeRequest.ts`；玩法专属 payload 归一化（如 Bark Battle start / finish 自动补 `workId`、`runId`）仍留在各玩法 client，通用 Module 只承接请求骨架。
-- 追加决策：Puzzle 的 start / get / swap / drag / next-level / leaderboard 与 Jump Hop 的 start / jump / restart 也迁入 `runtimeRequest.ts`；Puzzle `pause` 与 `props` 仍保留原账号态 auth options，不直接接入 runtime guest auth。
+- 追加决策：Puzzle 的 start / get / swap / drag / next-level / leaderboard / pause / props 与 Jump Hop 的 start / jump / restart 也迁入 `runtimeRequest.ts`；只要调用方传入 Runtime Guest Token，所有正式 runtime 请求都统一带局部 Authorization、`skipAuth` 与 `skipRefresh`。
 - 追加决策：Wooden Fish 的 start / checkpoint / finish 与 Visual Novel 的 gallery / run / history / regenerate JSON 请求也迁入 `runtimeRequest.ts`；Wooden Fish 的 `clientEventId` 生成仍留在木鱼 client，Visual Novel start 因 `timeoutMs`、SSE 因流式 `fetchWithApiAuth` 仍暂留原实现。
 - 影响范围：`src/services/runtimeRequest.ts`、Match3D / SquareHole / Big Fish / Bark Battle / Puzzle / Jump Hop / Wooden Fish / Visual Novel runtime client。
 - 验证方式：`npm run test -- src/services/runtimeRequest.test.ts src/services/recommendedRuntimeGuestLaunch.test.ts src/services/match3d-runtime/match3dRuntimeAdapter.test.ts`、`npm run typecheck`、`npm run check:encoding`、相关文件 ESLint 通过。
@@ -428,7 +461,7 @@
 ## 2026-05-25 抓大鹅运行态 HUD 收敛为拼图同款低遮挡样式
 
 - 背景：抓大鹅游玩阶段 UI 需要继续对齐拼图运行态的观感，同时移除右上角设置入口、灰白半透底板和显眼锅壳，让棋盘区域更专注。
-- 决策：抓大鹅运行态只保留左上透明返回按钮，右上不再显示设置入口；顶部关卡名和倒计时直接复用拼图同款的铭牌 + 下挂计时牌结构、同色板、同造型和 `media/logo.png` 产品 logo；底部备选栏和道具图标保持交互边界但不再显示灰白半透底；中央容器图层可以视觉隐藏，但棋盘命中边界和既有交互逻辑保留。
+- 决策：抓大鹅运行态只保留左上透明返回按钮，右上不再显示设置入口；顶部关卡名和倒计时直接复用拼图同款的铭牌 + 下挂计时牌结构、同色板、同造型和 `media/logo-runtime-hud.webp` 产品 logo 小图；底部备选栏和道具图标保持交互边界但不再显示灰白半透底；中央容器图层可以视觉隐藏，但棋盘命中边界和既有交互逻辑保留。
 - 影响范围：`src/components/match3d-runtime/Match3DRuntimeShell.tsx`、`src/components/match3d-runtime/Match3DRuntimeShell.test.tsx`、`src/index.css`、抓大鹅玩法链路文档。
 - 验证方式：运行态页面不再渲染“打开抓大鹅设置”，顶部仍显示关卡名和倒计时，底部槽位和道具按钮 class 中不含旧白底视觉；相关测试通过后保持该口径。
 - 关联文档：`docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`。
@@ -710,7 +743,7 @@
 
 - 背景：Windows 本机直连极高 VU 压测会放大本地连接与发送缓冲行为，和线上 Linux + Nginx + systemd 拓扑不一致；需要一个更接近生产网络层的模拟方案，但不能扰动当前生产发布链路。
 - 决策：新增 `deploy/container/` 容器化方案，使用 Docker Compose 组合 Linux release `api-server`、容器 SpacetimeDB、容器 Nginx、`otelcol-contrib` debug exporter 和可选 k6。该方案只用于本机或预发压测模拟，不替换当前生产 `systemd + Nginx + Jenkins` 路径。
-- 服务器模拟参数：2026-05-18 通过 `ssh genarrative-release` 采样，目标机器为 2 vCPU / 约 2 GiB RAM / Ubuntu 24.04 / Nginx `worker_connections=768`；容器方案按待发布运行口径使用 `nofile=4096`，并在 compose 中限制 `spacetimedb cpus=1.0 mem_limit=768m`、`api-server cpus=2.0 mem_limit=1g`、`nginx cpus=0.25 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=0.5 mem_limit=512m`；Collector 镜像默认使用 `otel/opentelemetry-collector-contrib:0.151.0`。
+- 服务器模拟参数：2026-05-18 通过 `ssh genarrative-release` 采样，目标机器为 2 vCPU / 约 2 GiB RAM / Ubuntu 24.04 / Nginx `worker_connections=768`；容器方案按待发布运行口径使用 `nofile=4096`，并在 compose 中限制 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`external-generation-worker cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`；Collector 镜像默认使用 `otel/opentelemetry-collector-contrib:0.151.0`。
 - 隔离边界：容器方案使用独立 `deploy/container/api-server.env`、独立 Nginx 配置、独立 compose 命令和默认 `18080` 端口；真实 token 不进入镜像、不提交 Git；生产 systemd 单元、Jenkins 发布脚本和 `deploy/nginx/` 模板仍是正式线上来源。
 - 生产 Collector：server-provision 可安装 `otelcol-contrib.service` 和本机 debug exporter 配置，但二进制由 Jenkins 构建机先准备 `provision-tools/otelcol-contrib` 再上传到 release 部署 agent，目标机不从 GitHub 下载；api-server 是否发送 OTLP 仍由 `GENARRATIVE_OTEL_ENABLED` 控制。
 - 影响范围：`deploy/container/`、`scripts/container-compose.mjs`、`package.json` 容器命令、开发运维文档和容器 build context 排除规则。
@@ -1728,3 +1761,11 @@
 - 影响范围：`shared-contracts` 默认 spec、`module-runtime` 入口配置响应、`spacetime-module` 后台保存校验、后台入口开关页摘要和前端 fallback spec。
 - 验证方式：`GET /api/creation-entry/config` 中各玩法 `unifiedCreationSpec.title` 等于已保存契约内容；后台只修改入口名称时不应隐式改写已保存的统一创作页表头。
 - 关联文档：`docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`。
+
+## 2026-06-11 资产计费边界改为 fail-closed 并补偿退款
+
+- 背景：图片 / 资产生成入口曾在钱包或 SpacetimeDB 预扣费连通性异常时允许继续生成，且失败后同步退款如果遇到 SpacetimeDB 短暂不可用缺少本地补偿；拼图首图后台任务还使用 api-server 进程内 HashSet 互斥，多实例下不能防重复。
+- 决策：暂不实现 token 限流。所有资产生成预扣费改为 fail-closed，预扣费失败直接返回错误；支持 retry 的计费 ledger id 统一包含 HTTP `request_id`，前端静默刷新重试复用同一个 `x-request-id`。生成失败后的退款先同步调用 SpacetimeDB，失败则写入 `wallet-refund-outbox` 本地文件并由后台 worker 重放。拼图首图后台生成互斥改为 SpacetimeDB `puzzle_background_compile_task` 表，使用 `task_id + request_id` 作为 claim id，释放时校验 claim id，避免旧任务误删新租约。
+- 影响范围：`api-server` 资产计费包裹、钱包退款补偿、拼图首图后台生成、`spacetime-module` 拼图 task 表、`spacetime-client` bindings/facade、前端 API request id 复用和后端架构文档。
+- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`npm run check:spacetime-runtime-access`、`node scripts/check-server-rs-ddd-boundaries.mjs`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server --manifest-path server-rs/Cargo.toml wallet_refund_outbox`、`cargo test -p api-server --manifest-path server-rs/Cargo.toml asset_operation`、`npm run test -- src/services/apiClient.test.ts`、`npm run check:encoding`。
+- 关联文档：`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`。

.hermes/shared-memory/development-workflow.md

@@ -52,6 +52,8 @@ npm run dev
 
 Linux 多用户共享同一台机器开发时，本地 dev 脚本会为当前 Linux 用户分配一个固定端口段并写入系统级注册表 `/var/tmp/genarrative-dev-port-ranges/registry.json`，自动分配从 `10000-10099` 开始，每段 100 个端口，四个 dev 服务依次使用 `start` 到 `start + 3`。可用 `GENARRATIVE_DEV_PORT_RANGE` 或 `npm run dev -- --port-range` 手动指定端口段用于特殊场景；注册表会阻止不同用户使用相同或重叠段，并让同一用户后续启动继续复用自己已占用的固定段。该机制只在 Linux 生效，Windows 仍沿用原有端口探测与漂移逻辑。
 
+本地 `npm run dev`、`npm run dev:spacetime` 和 `npm run dev:api-server` 会在 Rust 子进程环境中绕过项目默认 `sccache` wrapper，避免损坏的本机 cache daemon 阻断 `spacetime publish` 或 `api-server` 启动；显式设置的非 sccache 自定义 wrapper 会被保留。生产 / Jenkins 构建仍按流水线自身的 sccache 策略执行。
+
 该命令会启动：
 
 - SpacetimeDB standalone
@@ -95,6 +97,15 @@ npm run dev:web
 npm run dev:admin-web
 ```
 
+本地 SSH 服务器管理面板：
+
+```bash
+npm run server-manager:panel
+```
+
+该命令启动 `server-rs/crates/server-manager-panel` 的 egui 桌面工具，从本机 `~/.ssh/config` 读取可用 `Host` alias，支持多服务器健康巡检、可折叠侧边栏和受控 systemd 服务启停。服务操作通过远端 `sudo -n systemctl start|stop|restart <unit>` 执行，目标服务器需要提前配置对应 unit 的免交互 sudo 权限。
+面板启动时会自动注入本机中文字体；如开发机中文仍显示为方块，可设置 `GENARRATIVE_SERVER_PANEL_CJK_FONT=/path/to/font.ttc|index` 指向本机 CJK 字体。
+
 `npm run dev:api-server` 会保留终端实时输出，并把同一份输出持久化到 `logs/api-server/api-server-<timestamp>.log`。完整联调入口 `npm run dev` 启动的 Rust `api-server` 使用同一套日志规则。如需改写路径，可设置 `GENARRATIVE_API_SERVER_LOG_FILE`；如只改目录，可设置 `GENARRATIVE_API_SERVER_LOG_DIR`。
 
 开发态 `npm run dev` / `npm run dev:api-server` 默认打开 `GENARRATIVE_DEV_PASSWORD_ENTRY_AUTO_REGISTER_ENABLED=true`，密码入口可以直接注册未知手机号账号；生产默认仍关闭该开关。
@@ -109,6 +120,16 @@ npm run dev:admin-web
 npm run dev:spacetime:logs
 ```
 
+本机隔离验证外部生成 worker 队列、API-only 更新和 worker 动态扩缩容时，优先使用：
+
+```bash
+npm run container:worker-smoke -- smoke
+```
+
+该命令生成 `deploy/container/worker-smoke/` 下的 gitignored env 与端口 state，启动独立 compose project 和独立 SpacetimeDB，用 unsupported job 验证 worker claim / fail 回写；排查时用 `api-update` 确认 API 重建不触碰 worker，用 `scale <n>` 调整 worker 数量。
+`external_generation_job` 是 private table，worker-smoke 通过 worker 日志里的 job_id 和 unsupported 记录确认消费，不通过 CLI SQL 查询队列表。
+worker-smoke 默认把本机 `spacetime` CLI 打成轻量 SpacetimeDB 镜像，避免首次 smoke 依赖官方大镜像下载。若容器内 Cargo 下载依赖不稳定，追加 `--local-binary`，让容器内 Cargo 复用本机 Cargo 缓存构建当前 `api-server` 二进制，并把产物放进 Debian bookworm smoke runtime；可用 `GENARRATIVE_WORKER_SMOKE_LOCAL_BASE_IMAGE` 覆盖运行时基础镜像；隔离端口或库数据需要重建时追加 `--force`。
+
 后台管理前端：
 
 ```bash

.hermes/shared-memory/pitfalls.md

@@ -15,6 +15,38 @@
 - 关联：相关文件、文档、提交或 Issue
 ```
 
+## 外部生成 worker 业务失败重试会撞上钱包扣退费幂等
+
+- 现象：同一个外部生成 job 如果第一次业务失败后退款，再用同一个业务资源 ID 自动重试并成功，钱包 `consume` ledger 可能因为同 ID 已存在而跳过，最终出现“失败已退、成功不再扣”的余额漂移。
+- 原因：资产操作扣费和退款都用稳定 ledger id 做幂等；这能保护 lease 过期后的崩溃重领不重复扣费，但不适合“已明确失败且已退款”的自动业务重试。
+- 处理：拼图 `puzzle_compile_draft` 首期设置 `max_attempts=1`，业务失败直接 failed，只保留 running lease 过期后的崩溃重领。后续若要恢复自动 retry，必须先引入 attempt-aware billing 或可配对撤销的账本接口。
+- 验证：检查 `external_generation_job.max_attempts`、worker 失败回写和钱包 ledger；失败后草稿进入 failed，重试应由用户重新触发新任务，而不是旧 job 自动 pending。
+- 关联：`server-rs/crates/api-server/src/puzzle/handlers.rs`、`server-rs/crates/api-server/src/asset_billing.rs`、`server-rs/crates/spacetime-module/src/runtime/profile.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。
+
+## 外部生成队列不再由 HTTP 进程兜底执行
+
+- 现象：拼图首关生成接口返回 `queued`，但生成页长时间不完成，重启 `genarrative-api.service` 也没有推进任务。
+- 原因：HTTP 角色只入队，不再直接调用外部 provider；如果没有运行 `GENARRATIVE_PROCESS_ROLE=external-generation-worker` 或 `all` 的进程，`external_generation_job` 会停留在 `pending/running`，直到有 worker claim。
+- 处理：生产用 `systemctl enable --now genarrative-external-generation-worker@1.service genarrative-external-generation-controller.service` 启动保底 worker 和 controller；首次 API deploy 会在默认 worker pattern 下自动启用并启动 `@1`、等待 worker active，并重启验活 controller。扩容默认交给 controller 按队列统计启动 `@2.service` 等实例，手动扩缩容只作为兜底；worker 收到停机信号后会停止 claim 新任务并等待当前任务完成。本地 smoke 可临时用 `GENARRATIVE_PROCESS_ROLE=all npm run dev`；本地若只想同步排查可通过 `.env.local` 或本机环境设置 `GENARRATIVE_EXTERNAL_GENERATION_MODE=inline`，但这不会创建 job，也不能验证 worker 扩缩容。
+- 验证：`systemctl status genarrative-external-generation-controller.service 'genarrative-external-generation-worker@*.service'` 能看到 controller 和 worker 实例；queue 模式下任务被 claim 后 `worker_id` 与 `lease_expires_at` 会更新，完成后 session 进入 ready 或 failed；inline 模式下不应产生新的 `external_generation_job`。
+- 关联：`deploy/systemd/genarrative-external-generation-worker@.service`、`deploy/systemd/genarrative-external-generation-controller.service`、`deploy/env/external-generation-controller.env.example`、`server-rs/crates/spacetime-module/src/external_generation.rs`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
+
+## 外部生成 worker 业务写回必须同事务校验 lease guard
+
+- 现象：worker `complete/fail` 已校验 `worker_id + lease_token`，但如果玩法 session / work profile 写回在此之前单独调用，过期 worker 仍可能先写入业务状态，随后才在 job complete/fail 阶段失败；带计费包装的旧 worker 还可能因为 stale guard 错误触发补偿退款。
+- 原因：队列状态栅栏只保护 `external_generation_job` 自身，不会自动保护玩法 procedure。业务写回必须自己带 claim 后的 `job_id / worker_id / lease_token`，并在同一个 SpacetimeDB transaction 内校验 job 仍为 `running`、lease 未过期、job kind、owner 和 source entity 匹配。
+- 处理：拼图首图 worker 的前置 `compile_puzzle_agent_draft`、`save_puzzle_generated_images`、`save_puzzle_ui_background`、`mark_puzzle_draft_generation_failed` 和 `mark_puzzle_level_generation_failed` 已接入 `external_generation_job` lease guard；api-server 的资产扣费包装遇到这类 stale worker lease guard 错误时不执行补偿退款，错误文本包含 `external_generation_job 当前不是 running 状态` 或 `external_generation_job 不存在` 时也按 stale guard 处理。inline 模式只允许 `job_id / worker_id / lease_token` 三项同时为空，半空 guard 仍拒绝。后续迁移其它玩法 worker 时必须复用该模式，不能只在 worker 进程内保存一份 token。
+- 验证：`cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server asset_operation_billing_does_not_refund_stale_worker_lease_errors --manifest-path server-rs/Cargo.toml`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`。
+- 关联：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/spacetime-module/src/puzzle.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/asset_billing.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。
+
+## 外部生成 worker 核心业务写回失败不能完成 job
+
+- 现象：worker 已经生成图片并拿到本地合成 session 快照，但 SpacetimeDB 业务写回因连接、旧 wasm 或 lease guard 失败没有真实落库；如果此时仍把 `external_generation_job` 标成 `completed`，前端只会看到队列完成而 session 长时间不变化，后续也没有 worker 会重领修复。
+- 原因：同步 HTTP handler 的“外部 provider 已成功但 SpacetimeDB 短暂不可用时返回内存快照”降级语义，不能直接搬进异步 worker。worker 的完成状态必须代表核心业务事实已经持久化。
+- 处理：worker 路径的 `save_puzzle_generated_images` / `save_puzzle_ui_background` 等核心业务写回失败时直接返回错误；只有核心写回已经成功后的非关键投影回写才允许降级记录 warning。业务失败态也必须先写回 session / work profile，写回成功后才允许把队列 job 标为 failed；失败态未写回时保留租约，等待 lease 过期后重领。生产首装和首次 API deploy 都必须至少启用一个 worker 实例，例如 `systemctl enable --now genarrative-external-generation-worker@1.service`。
+- 验证：`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server asset_operation_billing_does_not_refund_stale_worker_lease_errors --manifest-path server-rs/Cargo.toml`，并在 smoke 时确认 queued 任务被 worker 消费后 session 真实更新。
+- 关联：`server-rs/crates/api-server/src/puzzle/draft.rs`、`server-rs/crates/api-server/src/puzzle/generation.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。
+
 ## 生产冷备份后 API 不能只依赖 SpacetimeDB 自恢复
 
 - 现象：release 机器 `03:20` 冷备份后，`spacetimedb.service` 已恢复，但作品列表、创作入口配置或公开 gallery 继续超时 / 502 / 504，`genarrative-api.service` 保持 stopped。
@@ -1149,6 +1181,7 @@
 - 追加处理：generated 私有图片换签 `/api/assets/read-url` 也属于展示层后台请求；推荐页拼图运行态挂载后会立即解析封面图，若换签 401 触发全局鉴权事件，也会表现成“进入拼图作品后瞬间未登录”。资源换签失败只应让当前图片为空，不应清 token、广播 auth 事件或主动 refresh。
 - 追加处理：从推荐页点进公开拼图作品并启动完整运行态后，`startPuzzleRun`、通关自动 `submitPuzzleLeaderboard`、下一关 `advancePuzzleNextLevel` 和重开同样属于当前玩法局部同步；这些请求失败时只应留在拼图错误态，不应清 token 或广播 auth 事件。
 - 追加处理：通关后 `refreshSaveArchives()`、首屏 bootstrap 的个人看板/作品架/浏览历史读写也只是平台投影刷新，失败应显示局部错误，不能充当全局登录态判定。
+- 追加处理：未登录推荐页启动任一公开正式玩法时，`/api/runtime/*` 局内路由必须使用 `RuntimePrincipal`，前端通过 `PlatformEntryFlowShellImpl` 的统一 request options helper 给 start / checkpoint / finish / input / drop / click / restart / time-up / leaderboard / next-level 等动作透传 runtime guest token；公开 runtime detail 读取如跳一跳、敲木鱼必须显式 `skipAuth/skipRefresh`，匿名推荐流不能补读受保护创作详情，否则会在真正开局前打出 `/api/auth/refresh 401`。
 - 验证：`npm run test -- src/services/apiClient.test.ts src/services/assetReadUrlService.test.ts`、`npm run test -- src/components/rpg-entry/RpgEntryFlowShell.agent.interaction.test.tsx -t "home recommendation starts embedded puzzle"`、`npm run test -- src/components/rpg-entry/RpgEntryFlowShell.agent.interaction.test.tsx -t "formal puzzle runtime uses frontend move merge logic and backend leaderboard"`、`npm run test -- src/components/rpg-entry/RpgEntryFlowShell.agent.interaction.test.tsx -t "formal puzzle similar work keeps current run level progression"`。
 - 关联：`src/services/apiClient.ts`、`src/services/assetReadUrlService.ts`、`src/services/puzzle-runtime/puzzleRuntimeClient.ts`、`src/components/platform-entry/PlatformEntryFlowShellImpl.tsx`、`docs/technical/RECOMMEND_RUNTIME_AUTH_FAILURE_ISOLATION_FIX_2026-05-09.md`。
 
@@ -1279,8 +1312,8 @@
 
 - 现象：Cargo 报 `could not execute process sccache ... rustc.exe -vV (never executed)`、`sccache: error: Timed out waiting for server startup`，或 `sccache: caused by: Failed to send data to or receive data from server / Failed to read response header / failed to fill whole buffer`；真实 `rustc -Vv` 可以执行，但构建在调用包装器时失败。
 - 原因：环境、Jenkinsfile 或 `server-rs/.cargo/config.toml` 启用了 `sccache` wrapper，但当前 agent 没有可执行的 `sccache`、PATH 中 shim 损坏，或本地 sccache server/client 通道状态损坏。Windows 本机若配置了 `SCCACHE_OSS_*`，sccache daemon 冷启动会先经 OSS/本机代理完成缓存读写检查，再监听 `127.0.0.1:4226`；代理或 OSS 链路慢时，Cargo 的 `sccache rustc -vV` 可能先超时。
-- 处理：保留 `server-rs/.cargo/config.toml` 的 `rustc-wrapper = "sccache"`；Windows 本机优先在 `%APPDATA%\Mozilla\sccache\config\config` 写入 `server_startup_timeout_ms = 60000`，拉长 client 等待 daemon 完成 OSS 初始化的时间，然后删除 `server-rs/target/.rustc_info.json` 里缓存的失败探测结果并重跑原始 Cargo 命令。冷启动验证优先用 `sccache --stop-server`，不要在另一个 `cargo` / `rustc` 仍在编译时 `taskkill /F /IM sccache.exe /T`，否则 proc-macro crate 可能被打断并表现为 `serde_derive` / `spacetimedb-bindings-macro` 的 `sccache ... exit code: 1`。若只做临时排障，可在 Git Bash 中执行 `RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo build ...`，或在 PowerShell 用 `cargo check -p api-server --config "build.rustc-wrapper=''"` 一次性绕过 wrapper；生产流水线必须先实际执行 `sccache --version`，失败时移除 `RUSTC_WRAPPER` 并回退到直接 `rustc`。
-- 验证：`rustc -Vv` 能输出版本；冷启动后原始 `cargo check -p api-server` 和 `cargo check -p spacetime-module` 能通过；`sccache --show-stats` 显示 `Cache location oss, name: genarrative-sccache`，证明仍在使用 sccache/OSS 缓存；Jenkins 日志出现“未找到可用 sccache，改用 rustc 直接构建”后仍继续真实构建。
+- 处理：保留 `server-rs/.cargo/config.toml` 的 `rustc-wrapper = "sccache"`；本地 `npm run dev` / `npm run dev:spacetime` / `npm run dev:api-server` 由 `scripts/dev.mjs` 给 Rust 子进程注入直通 wrapper，自动绕过项目默认 sccache，避免损坏的 daemon 阻断 `spacetime publish` 或 `api-server` 启动；显式设置的非 sccache 自定义 wrapper 会被保留。Windows 本机优先在 `%APPDATA%\Mozilla\sccache\config\config` 写入 `server_startup_timeout_ms = 60000`，拉长 client 等待 daemon 完成 OSS 初始化的时间，然后删除 `server-rs/target/.rustc_info.json` 里缓存的失败探测结果并重跑原始 Cargo 命令。冷启动验证优先用 `sccache --stop-server`，不要在另一个 `cargo` / `rustc` 仍在编译时 `taskkill /F /IM sccache.exe /T`，否则 proc-macro crate 可能被打断并表现为 `serde_derive` / `spacetimedb-bindings-macro` 的 `sccache ... exit code: 1`。若只做临时排障，可在 Git Bash 中执行 `RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo build ...`，或在 PowerShell 用 `cargo check -p api-server --config "build.rustc-wrapper=''"` 一次性绕过 wrapper；生产流水线必须先实际执行 `sccache --version`，失败时移除 `RUSTC_WRAPPER` 并回退到直接 `rustc`。
+- 验证：`rustc -Vv` 能输出版本；本地 `npm run dev` 能完成 `spacetime publish`、`api-server` `/healthz`、主站 Vite 和后台 Vite 启动；冷启动后原始 `cargo check -p api-server` 和 `cargo check -p spacetime-module` 能通过；`sccache --show-stats` 显示 `Cache location oss, name: genarrative-sccache`，证明原始 Cargo/Jenkins 路径仍可使用 sccache/OSS 缓存；Jenkins 日志出现“未找到可用 sccache，改用 rustc 直接构建”后仍继续真实构建。
 - 关联：`scripts/dev.mjs`、`jenkins/Jenkinsfile.production-stdb-module-build`、`docs/technical/SPACETIMEDB_PUBLISH_SCCACHE_FALLBACK_2026-05-09.md`、`docs/technical/PRODUCTION_DEPLOYMENT_PLAN_2026-05-02.md`。
 
 ## 生产发布入口不要沿用旧 Jenkinsfile / 一体化脚本
@@ -2189,3 +2222,11 @@
 - 待处理：将跳一跳生成改为后端任务化 / 可轮询真实阶段进度，按背景、返回按钮、图集、切片、持久化、写草稿分阶段落库；统一后端全局生成 deadline、VectorEngine 重试预算、前端等待窗口和失败态回写。超时后再次进入同一 session 应优先恢复正在运行或已完成的任务，不应重复生图。
 - 验证：模拟首张 image2 超长耗时或超时重试时，生成页应显示真实阶段和可恢复状态；前端请求超时不应把最终成功草稿标记为失败；刷新 `/creation/jump-hop/generating?sessionId=<id>` 后应能恢复到后端真实状态；同一 session 重试不得重复生成已完成阶段。
 - 关联：`src/services/jump-hop/jumpHopClient.ts`、`src/services/miniGameDraftGenerationProgress.ts`、`server-rs/crates/api-server/src/jump_hop.rs`、`server-rs/crates/platform-image/src/vector_engine/client.rs`、`docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`。
+
+## SpacetimeDB 连接池租约必须有 Drop 兜底，acquire 不允许无界自旋
+
+- 现象：release 上 api-server 周期性出现全量 `spacetime_stage="pool_acquire" elapsed_ms=45000` 业务超时，`/readyz` 503（`reason=spacetime_unhealthy, stage=pool_acquire`），`/healthz` 仍 200，只有重启能恢复，过若干小时复发。
+- 原因：旧 `PooledConnectionLease` 只能显式 `release_connection` 归还；HTTP 请求方在等待 StDB 回包期间断开时 handler future 被取消，permit 自动归还但槽位 `in_use` 永不复位。后续 acquire 在拿到 permit 后进入无界 `loop + yield_now` 扫描空闲槽位，泄漏积累到 pool_size 后整池挂死。
+- 处理：租约持有 `Arc<SpacetimeConnectionPool>` 并实现 `Drop` 统一复位槽位/归还连接；槽位改 `AtomicBool` CAS 抢占，删除自旋循环（持有 permit 必然命中空闲槽位）。任何新的"显式归还"资源在 async 取消语义下都要先想 Drop 兜底。
+- 验证：`cargo test -p spacetime-client --manifest-path server-rs/Cargo.toml --lib`（`dropped_lease_releases_slot_and_permit`、`acquire_times_out_at_pool_acquire_when_pool_is_busy`）。
+- 关联：`server-rs/crates/spacetime-client/src/lib.rs`、`docs/【后端架构】SpacetimeDB连接池租约Drop兜底与取消安全-2026-06-11.md`。

deploy/container/README.md

@@ -9,12 +9,14 @@ Docker Compose
 ├─ spacetimedb  :3101，独立数据卷，供 api-server 连接
 ├─ nginx        :80 -> api-server:8082，负责静态站点、/admin/、/api/ 反代、upstream timing log、连接限制
 ├─ api-server   :8082，Linux release 构建，连接 compose 内 SpacetimeDB
+├─ external-generation-worker，独立 worker 进程，消费 external_generation_job 队列
 ├─ otelcol      :4317/4318，debug exporter，接收 traces / metrics / logs
 └─ k6           profile=loadtest 时临时启动，在 compose 网络内压 nginx
 ```
 
-当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`。SpacetimeDB 同时设置 `--page_pool_max_size=402653184`，给 reducer、订阅与运行时保留更多非 page pool 内存。
+当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`external-generation-worker cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`。SpacetimeDB 同时设置 `--page_pool_max_size=402653184`，给 reducer、订阅与运行时保留更多非 page pool 内存。
 容器 `api-server` 默认 `GENARRATIVE_API_WORKER_THREADS=4`，用于让 Tokio 在 2 vCPU 配额内有更多 I/O 调度 worker；该值不会突破 compose 里的 `cpus=2.0` CPU 上限。
+容器默认 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`，用于验证 `api-server -> external_generation_job -> external-generation-worker` 链路；如只想本地同步排查 provider/OSS/SpacetimeDB 写回，可在本机 env 临时改为 `inline`，但该模式不会覆盖 worker 动态扩缩容验证。
 Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。
 生产服务器若启用 Collector，则由 `deploy/systemd/otelcol-contrib.service` 和 `deploy/otelcol/genarrative-debug.yaml` 托管，不走容器镜像。
 
@@ -55,7 +57,7 @@ Linux Docker Engine 若要从宿主机 CLI 连到容器内服务，直接用 `ht
 
 ## 构建工具链
 
-`api-server` 容器镜像只构建 Linux release API 二进制，不构建 `spacetime-module`。当前 `api-server -> spacetime-client -> spacetimedb-sdk 2.4.1` 依赖链要求 Rust 1.93，因此 `deploy/container/api-server.Dockerfile` 的 Rust builder 固定为 `rust:1.93-bookworm`。如果本机 Docker Hub 拉取失败，可以先在本机准备同名本地 builder 镜像，但不要把临时 bootstrap 容器或私有 registry 凭据写入仓库。
+`api-server` 容器镜像只构建 Linux release API 二进制，不构建 `spacetime-module`。当前 `api-server -> spacetime-client -> spacetimedb-sdk 2.4.1` 依赖链要求 Rust 1.93，因此 `deploy/container/api-server.Dockerfile` 的 Rust builder 固定为 `rust:1.93-bookworm`。镜像构建阶段会同时复制 `public/`，用于满足 API 二进制里 `include_bytes!` 引用的内置素材；不要把 `public/generated-*` 放入镜像上下文。如果本机 Docker Hub 拉取失败，可以先在本机准备同名本地 builder 镜像，但不要把临时 bootstrap 容器或私有 registry 凭据写入仓库。
 
 ## 启动与验证
 
@@ -74,6 +76,7 @@ curl -sS http://127.0.0.1:18080/api/runtime/puzzle/gallery
 ```bash
 npm run container:logs -- nginx
 npm run container:logs -- api-server
+npm run container:logs -- external-generation-worker
 npm run container:logs -- otelcol
 ```
 
@@ -85,6 +88,73 @@ npm run container:config -- --print
 
 如果 `deploy/container/api-server.env` 已写入真实 token，不要把完整展开结果贴到公开渠道。
 
+动态扩缩容外部生成 worker 时，只调整 `external-generation-worker` service：
+
+```bash
+npm run container:up -- --scale external-generation-worker=3 external-generation-worker
+npm run container:up -- --scale external-generation-worker=1 external-generation-worker
+```
+
+动态扩缩容验证必须保持 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`；`inline` 模式下生成请求由 `api-server` 同步执行，不会被这些 worker 实例消费。
+
+### 外部生成 Worker 隔离 Smoke
+
+如果只想在本机隔离验证 worker 模式，不复用 `deploy/container/api-server.env`，使用专用脚本：
+
+```bash
+npm run container:worker-smoke -- smoke
+```
+
+该脚本会生成 gitignored 的 `deploy/container/worker-smoke/api-server.env` 与端口 state，使用独立 compose project、独立 SpacetimeDB 数据卷和独立 host 端口，完成 `build -> up-spacetime -> publish -> up -> enqueue -> api-update -> enqueue`。测试 job 使用 `worker_smoke_unsupported` 类型，不访问真实 VectorEngine、LLM 或 OSS；预期结果是 worker 领取队列任务后按“不支持的任务类型”执行失败分支，从而验证队列 claim、lease、失败回写路径和 API / worker 进程隔离。`external_generation_job` 是 private table，脚本通过 worker 日志里的 job_id 和 unsupported 记录确认消费，不通过 CLI SQL 绕过权限。`smoke` 默认只启动 `api-server` 与 `external-generation-worker`，避免无关前端 / Nginx 镜像构建；需要同时验证 Nginx 时可分步执行 `up --with-nginx`。
+
+分步排查时可执行：
+
+```bash
+npm run container:worker-smoke -- init --force
+npm run container:worker-smoke -- build
+npm run container:worker-smoke -- up-spacetime
+npm run container:worker-smoke -- publish
+npm run container:worker-smoke -- up
+npm run container:worker-smoke -- enqueue before-update
+npm run container:worker-smoke -- api-update
+npm run container:worker-smoke -- enqueue after-update
+npm run container:worker-smoke -- status
+```
+
+如果隔离端口或库数据需要重置：
+
+```bash
+npm run container:worker-smoke -- smoke --force
+```
+
+`container:worker-smoke` 默认会把本机 `spacetime` 2.4.1 CLI 打成轻量 SpacetimeDB 镜像，避免首次 smoke 必须拉取官方大镜像；普通 `npm run container:*` 压测仍默认使用 `clockworklabs/spacetime:v2.4.1`。如果 Docker build 阶段在容器内拉取 crates.io 依赖不稳定，可让容器内 Cargo 复用本机 Cargo 缓存构建当前二进制，再打入临时 smoke 镜像。该模式默认使用 `rust:1.93-bookworm` 作为 builder、Debian bookworm smoke runtime 承载构建产物；需要换 builder 镜像时设置 `GENARRATIVE_WORKER_SMOKE_CARGO_IMAGE`，需要换运行时基础镜像时设置 `GENARRATIVE_WORKER_SMOKE_LOCAL_BASE_IMAGE`：
+
+```bash
+npm run container:worker-smoke -- smoke --local-binary
+```
+
+`api-update` 只会 `--force-recreate api-server`，并校验 `external-generation-worker` 容器 ID 不变；如要同时重建 API 镜像，使用：
+
+```bash
+npm run container:worker-smoke -- api-update --build
+```
+
+验证 worker 动态扩缩容：
+
+```bash
+npm run container:worker-smoke -- scale 3
+npm run container:worker-smoke -- ps
+npm run container:worker-smoke -- enqueue scaled-workers
+npm run container:worker-smoke -- scale 1
+```
+
+查看或清理隔离环境：
+
+```bash
+npm run container:worker-smoke -- logs external-generation-worker
+npm run container:worker-smoke -- down -v
+```
+
 停止：
 
 ```bash

deploy/container/api-server.Dockerfile

@@ -2,6 +2,7 @@ FROM rust:1.93-bookworm AS rust-builder
 WORKDIR /workspace
 
 COPY server-rs ./server-rs
+COPY public ./public
 RUN cargo build --release -p api-server --manifest-path server-rs/Cargo.toml && \
     cp server-rs/target/release/api-server /tmp/api-server
 

deploy/container/api-server.env.example

@@ -8,6 +8,14 @@ GENARRATIVE_API_PORT=8082
 GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
 GENARRATIVE_API_WORKER_THREADS=4
+# 容器 smoke 可临时设 all；压测或预发按 api / external-generation-worker 拆进程。
+GENARRATIVE_PROCESS_ROLE=api
+# 默认 queue 进入 external_generation_job；本地/小流量同步排查可显式设 inline。
+GENARRATIVE_EXTERNAL_GENERATION_MODE=queue
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=2
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=3600
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
 GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320
 GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64

deploy/container/docker-compose.loadtest.yml

@@ -2,7 +2,7 @@ name: genarrative-container-loadtest
 
 services:
   spacetimedb:
-    image: clockworklabs/spacetime:v2.4.1
+    image: ${GENARRATIVE_CONTAINER_SPACETIME_IMAGE:-clockworklabs/spacetime:v2.4.1}
     user: root
     command:
       [
@@ -44,7 +44,7 @@ services:
     cpus: "2.0"
     mem_limit: 1g
     env_file:
-      - ./api-server.env
+      - ${GENARRATIVE_CONTAINER_API_ENV_FILE:-./api-server.env}
     environment:
       GENARRATIVE_API_HOST: 0.0.0.0
       GENARRATIVE_API_PORT: 8082
@@ -69,6 +69,32 @@ services:
       retries: 12
       start_period: 20s
 
+  external-generation-worker:
+    build:
+      context: ../..
+      dockerfile: deploy/container/api-server.Dockerfile
+      target: api-runtime
+    cpus: "2.0"
+    mem_limit: 1g
+    env_file:
+      - ${GENARRATIVE_CONTAINER_API_ENV_FILE:-./api-server.env}
+    environment:
+      GENARRATIVE_PROCESS_ROLE: external-generation-worker
+      GENARRATIVE_TRACKING_OUTBOX_DIR: /var/lib/genarrative/tracking-outbox-worker
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4318
+      OTEL_SERVICE_NAME: genarrative-external-generation-worker
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
+    depends_on:
+      spacetimedb:
+        condition: service_healthy
+      otelcol:
+        condition: service_started
+
   nginx:
     build:
       context: ../..

deploy/env/api-server.env.example

@@ -7,6 +7,14 @@ GENARRATIVE_API_PORT=8082
 GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
 GENARRATIVE_API_WORKER_THREADS=4
+# api 只监听 HTTP；外部生成 worker 用独立进程设置为 external-generation-worker 后横向扩缩。
+GENARRATIVE_PROCESS_ROLE=api
+# 默认 queue 进入 external_generation_job；本地/小流量同步排查可显式设 inline。
+GENARRATIVE_EXTERNAL_GENERATION_MODE=queue
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=2
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=3600
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
 GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320
 GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64

deploy/env/external-generation-controller.env.example

@@ -0,0 +1,13 @@
+# 复制到 /etc/genarrative/external-generation-controller.env 后按机器容量调整。
+# controller 只管理 systemd worker 实例；SpacetimeDB、外部 provider 密钥继续复用 api-server.env。
+# systemd unit 会强制设置 GENARRATIVE_PROCESS_ROLE=external-generation-controller。
+
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MIN_WORKERS=1
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MAX_WORKERS=8
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_TARGET_JOBS_PER_WORKER=2
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_POLL_INTERVAL_MS=10000
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_SCALE_DOWN_IDLE_ROUNDS=6
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_SERVICE_TEMPLATE=genarrative-external-generation-worker@{}.service
+GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_DRY_RUN=false
+GENARRATIVE_API_LOG=info,tower_http=info
+OTEL_SERVICE_NAME=genarrative-external-generation-controller

deploy/env/external-generation-worker.env.example

@@ -0,0 +1,11 @@
+# 复制到 /etc/genarrative/external-generation-worker.env 后按机器容量调整。
+# 该文件只覆盖 worker 专属参数；SpacetimeDB、外部 provider 密钥继续复用 api-server.env。
+# systemd 模板会强制设置 GENARRATIVE_PROCESS_ROLE=external-generation-worker
+# 和 GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=%H-%i，避免多实例 ID 冲突。
+
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=2
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
+# 单次 lease 会由 worker 自动续租；该值覆盖心跳抖动窗口即可。
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=3600
+GENARRATIVE_API_LOG=info,tower_http=info
+OTEL_SERVICE_NAME=genarrative-external-generation-worker

deploy/systemd/genarrative-external-generation-controller.service

@@ -0,0 +1,28 @@
+[Unit]
+Description=Genarrative External Generation Worker Controller
+After=network-online.target spacetimedb.service
+Wants=network-online.target
+Requires=spacetimedb.service
+
+[Service]
+Type=simple
+WorkingDirectory=/opt/genarrative/current
+EnvironmentFile=/etc/genarrative/api-server.env
+EnvironmentFile=-/etc/genarrative/external-generation-controller.env
+Environment="LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib"
+ExecStart=/usr/bin/env GENARRATIVE_PROCESS_ROLE=external-generation-controller GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox/controller OTEL_SERVICE_NAME=genarrative-external-generation-controller /opt/genarrative/current/api-server
+Restart=always
+RestartSec=5
+KillSignal=SIGINT
+TimeoutStopSec=120
+LimitNOFILE=65535
+TasksMax=512
+
+# controller 需要调用 systemctl 管理 worker@N 实例，因此不降为 genarrative 用户。
+# 它只复用 api-server 发布包和 SpacetimeDB 配置，不直接执行外部生成任务。
+PrivateTmp=true
+ProtectSystem=full
+ReadWritePaths=/opt/genarrative /var/lib/genarrative
+
+[Install]
+WantedBy=multi-user.target

deploy/systemd/genarrative-external-generation-worker@.service

@@ -0,0 +1,30 @@
+[Unit]
+Description=Genarrative External Generation Worker %i
+After=network-online.target spacetimedb.service
+Wants=network-online.target
+Requires=spacetimedb.service
+
+[Service]
+Type=simple
+User=genarrative
+Group=genarrative
+WorkingDirectory=/opt/genarrative/current
+EnvironmentFile=/etc/genarrative/api-server.env
+EnvironmentFile=-/etc/genarrative/external-generation-worker.env
+Environment="LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib"
+ExecStart=/usr/bin/env GENARRATIVE_PROCESS_ROLE=external-generation-worker GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=%H-%i GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox/%H-%i OTEL_SERVICE_NAME=genarrative-external-generation-worker /opt/genarrative/current/api-server
+Restart=always
+RestartSec=5
+KillSignal=SIGINT
+TimeoutStopSec=7200
+LimitNOFILE=65535
+TasksMax=2048
+
+# worker 复用 api-server 发布目录；外部生成审计与临时运行态只写服务端私有目录。
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ReadWritePaths=/opt/genarrative /var/lib/genarrative
+
+[Install]
+WantedBy=multi-user.target

docs/README.md

@@ -21,6 +21,8 @@
 
 微信小程序虚拟支付接入、`wechat_mp_virtual` 渠道、`wx.requestVirtualPayment` 承接页和后端签名配置见 [【技术方案】微信虚拟支付接入-2026-05-26.md](./%E3%80%90%E6%8A%80%E6%9C%AF%E6%96%B9%E6%A1%88%E3%80%91%E5%BE%AE%E4%BF%A1%E8%99%9A%E6%8B%9F%E6%94%AF%E4%BB%98%E6%8E%A5%E5%85%A5-2026-05-26.md)。
 
+本地通过 SSH alias 管理多台服务器、查看硬件 / systemd / HTTP 健康状态并执行受控服务启停的 egui 桌面工具见 [【开发运维】本地SSH服务器管理面板技术方案-2026-06-11.md](./technical/【开发运维】本地SSH服务器管理面板技术方案-2026-06-11.md)。
+
 生产部署切换到 systemd + Nginx + SpacetimeDB 自托管的总方案见 [PRODUCTION_DEPLOYMENT_PLAN_2026-05-02.md](./technical/PRODUCTION_DEPLOYMENT_PLAN_2026-05-02.md)，该文档也是当前生产 Jenkinsfile 的唯一入口。SpacetimeDB 表结构变更、自动迁移边界和保留旧数据的分阶段迁移流程见 [SPACETIMEDB_SCHEMA_CHANGE_CONSTRAINTS.md](./technical/SPACETIMEDB_SCHEMA_CHANGE_CONSTRAINTS.md)；private 表迁移 JSON 导入导出、HTTP 413 分片导入和旧数据库迁移流水线经验见 [SPACETIMEDB_JSON_STRING_MIGRATION_PROCEDURE_2026-04-27.md](./technical/SPACETIMEDB_JSON_STRING_MIGRATION_PROCEDURE_2026-04-27.md) 与 [JENKINS_SPACETIMEDB_DATABASE_MIGRATION_PIPELINES_2026-04-29.md](./technical/JENKINS_SPACETIMEDB_DATABASE_MIGRATION_PIPELINES_2026-04-29.md)；后台管理独立前端工程技术方案见 [ADMIN_WEB_CONSOLE_TECHNICAL_SOLUTION_2026-04-30.md](./technical/ADMIN_WEB_CONSOLE_TECHNICAL_SOLUTION_2026-04-30.md)。
 
 SpacetimeDB 表结构变更、自动迁移边界和保留旧数据的分阶段迁移流程见 [SPACETIMEDB_SCHEMA_CHANGE_CONSTRAINTS.md](./technical/SPACETIMEDB_SCHEMA_CHANGE_CONSTRAINTS.md)。

docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md

@@ -0,0 +1,172 @@
+# 外部生成 Worker 化方案
+
+更新时间：`2026-06-07`
+
+## 背景
+
+当前 VectorEngine `gpt-image-2`、音频、LLM 等外部生成链路多数由 `api-server` 的 HTTP handler 直接等待上游、OSS 持久化和 SpacetimeDB 回写完成。前端虽然有生成页和会话轮询，但 HTTP 进程仍承担长耗时副作用，导致接入更多玩法或大图生成时只能放大 API 进程，而不能单独扩展外部生成吞吐。
+
+## 目标
+
+- 默认 `queue` 模式下，`api-server` 的 HTTP 角色只负责鉴权、入参校验、扣费前置/状态初始化、任务入队和返回 `queued` 操作结果。
+- 外部生成副作用由独立 `external-generation-worker` 角色执行。
+- 多个 worker 进程通过 SpacetimeDB 任务表抢占任务，依赖 lease 超时恢复，支持按进程数和单进程并发动态缩扩容。
+- 本地或小流量同步排查可显式启用 `inline` 模式，由 HTTP handler 复用同一 worker executor 同步执行并返回 `completed`；该模式不创建队列任务，也不具备 worker 横向扩容能力。
+- SpacetimeDB reducer / procedure 只做任务状态流转，不做网络、文件系统或外部 provider I/O。
+- 已接入拼图 `compile_puzzle_draft`、结果页 `generate_puzzle_images` 与结果页 `generate_puzzle_ui_background`；后续玩法继续复用同一队列 Module，不再为每个玩法发明独立队列。
+
+## Module 与 Interface
+
+新增深一点的 **外部生成任务 Module**，Interface 收敛为：
+
+- `enqueue_external_generation_job_and_return`：按 `dedupe_key` 幂等创建或返回现有任务。
+- `claim_external_generation_jobs_and_return`：worker 按 `worker_id`、`limit` 和 lease 时长抢占 `pending` 或 lease 过期的 `running` 任务，返回本次 claim 的 `lease_token`。
+- `renew_external_generation_job_lease_and_return`：worker 长任务执行期间按 `worker_id + lease_token` 续租，防止外部生成超过单次 lease 后被重复领取。
+- `complete_external_generation_job_and_return`：worker 成功后按 `worker_id + lease_token` 写入 `result_payload_json`，任务进入 `completed`。
+- `fail_external_generation_job_and_return`：worker 失败后按 `worker_id + lease_token` 回写错误，并按 `max_attempts` 决定回到 `pending` 重试或进入 `failed`。
+- `get_external_generation_queue_stats_and_return`：controller 读取队列积压、运行中任务和过期 lease 数量，用于计算 worker 目标实例数；该 procedure 只读 `external_generation_job`，不直接操作 systemd。
+
+这个 Module 的 **Seam** 在 SpacetimeDB procedure + `spacetime-client` facade；`api-server` HTTP role 和 worker role 都只依赖这个 Interface。外部 provider、OSS、计费补偿、玩法草稿回写仍留在 `api-server` worker implementation 内，不进入 SpacetimeDB reducer。
+
+## 任务表
+
+新增私有表 `external_generation_job`：
+
+| 字段 | 说明 |
+| --- | --- |
+| `job_id` | 主键，`extgen-` 前缀 UUID |
+| `dedupe_key` | 唯一键，建议为 `play/action/session/scope` |
+| `job_kind` | 执行类型，当前拼图为 `puzzle_compile_draft`、`puzzle_generate_images`、`puzzle_generate_ui_background` |
+| `owner_user_id` | 触发用户 |
+| `source_module` | 玩法或能力名，例如 `puzzle` |
+| `source_entity_id` | session/profile/work 等作用域 |
+| `request_label` | 排障标签 |
+| `request_payload_json` | worker 执行入参 JSON |
+| `status` | `pending/running/completed/failed/cancelled` |
+| `attempt` / `max_attempts` | 当前尝试次数与最大尝试次数 |
+| `last_error_message` | 最近失败原因 |
+| `worker_id` | 当前 lease owner |
+| `lease_expires_at` | lease 到期时间 |
+| `lease_token` | 本次 claim 的 fencing token，用于阻止过期 worker 回写 |
+| `available_at` | 下次可领取时间 |
+| `result_payload_json` | 完成摘要 |
+| `created_at/started_at/completed_at/updated_at` | 审计时间 |
+
+索引：
+
+- `by_external_generation_job_status_available(status, available_at)`
+- `by_external_generation_job_worker_id(worker_id)`
+- `by_external_generation_job_source(source_module, source_entity_id)`
+- `by_external_generation_job_owner_user_id(owner_user_id)`
+
+## 状态机
+
+```text
+pending -> running -> completed
+pending -> running -> pending   (可重试失败)
+pending -> running -> failed    (达到最大重试次数)
+pending/running -> cancelled    (预留)
+```
+
+`claim` 只领取 `pending` 且 `available_at <= now` 的任务，或 `running` 且 `lease_expires_at <= now` 的任务。领取时递增 `attempt`、写入 `worker_id`、`started_at`、新的 `lease_expires_at` 和 `lease_token`。SpacetimeDB procedure 使用 `ctx.timestamp` 作为状态流转时间，只从 worker 入参读取“时长差值”，不信任 worker 本机绝对时间。worker 每次执行只处理自己 claim 到的任务；续租、完成或失败时必须带同一个 `worker_id + lease_token`，且当前 lease 尚未过期，防止过期 worker 覆盖新 lease。
+
+玩法业务写回也必须在 SpacetimeDB 同一事务里校验 lease fencing。拼图的 `compile_puzzle_agent_draft` worker 调用、`save_puzzle_generated_images`、`save_puzzle_ui_background`、`mark_puzzle_draft_generation_failed` 和 `mark_puzzle_level_generation_failed` 在 `queue` 模式下会带 `external_generation_job_id / worker_id / lease_token`，并校验 job 仍为 `running`、token 未过期、`job_kind`、`owner_user_id`、`source_module` 和 `source_entity_id` 均匹配后才写 session / work profile。`inline` 模式不创建 `external_generation_job`，因此这三个 guard 字段必须同时为空；transaction 只把三项全空识别为 api-server 受控同步写回，三项半空仍按非法请求拒绝。worker 路径的核心业务写回失败不能返回内存快照并把 job 标为 `completed`；失败态业务回写成功后才允许把队列 job 标为 `failed`，失败态仍未写回时保留当前租约并等待后续 lease 过期重领，避免队列状态和真实 session 脱节。api-server 的资产扣费包装遇到这类 stale worker lease guard 错误时不执行补偿退款，避免旧 worker 冲掉后续合法 worker 的同一账本扣费。
+
+## 执行模式与进程角色
+
+外部生成执行模式由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制：
+
+- `queue`：默认值，HTTP handler 入队 `external_generation_job`，由 `external-generation-worker` 角色 claim lease 后执行；生产、预发和压测默认使用该模式。
+- `inline`：HTTP handler 直接调用同一个 worker executor，同步等待 provider、OSS 和 SpacetimeDB 写回完成后返回 `operation.status = completed`；只用于本地或低并发排查，不提供队列持久化、lease 重领和 worker 横向扩容。
+
+同一个 Rust binary 通过 `GENARRATIVE_PROCESS_ROLE` 切换：
+
+- `api`：只启动 HTTP server。
+- `external-generation-worker`：只启动外部生成 worker，不监听 HTTP。
+- `external-generation-controller`：只启动 worker controller，不监听 HTTP，也不直接执行外部生成任务。
+- `all`：本地开发可同时启动 HTTP 与 worker。
+
+worker 配置：
+
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`：实例 ID；未配置时用 hostname/pid 派生。
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY`：单进程并发领取/执行数量。
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS`：空队列轮询间隔。
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS`：任务 lease 时长；worker 会按约三分之一 lease、最长 30 秒的间隔续租。该值应覆盖一次心跳网络抖动窗口，不需要大于完整外部生成链路耗时。
+
+controller 配置：
+
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MIN_WORKERS`：保底 worker 实例数，生产默认 `1`，controller 不会主动停止 `@1`。
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MAX_WORKERS`：自动扩容上限，生产模板默认 `8`。
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_TARGET_JOBS_PER_WORKER`：每个 worker 实例承担的目标未完成任务数，默认 `2`；目标实例数按 `claimable_pending + running_active + expired_running` 计算后夹在 min/max 之间，避免把已包含过期 running 的 `claimable_count` 重复计入。
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_POLL_INTERVAL_MS`：controller 轮询队列统计的间隔，默认 `10000`。
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_SCALE_DOWN_IDLE_ROUNDS`：连续多少轮无可领取、无运行中、无过期 running 后才允许缩容，默认 `6`；缩容每轮只停止最高编号的一个实例。
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_SERVICE_TEMPLATE`：systemd worker 模板，默认 `genarrative-external-generation-worker@{}.service`。
+- `GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_DRY_RUN`：只记录决策不执行 systemctl，默认 `false`。
+
+动态缩扩容方式：生产默认由 `deploy/systemd/genarrative-external-generation-controller.service` 启动 `GENARRATIVE_PROCESS_ROLE=external-generation-controller`，controller 读取 `get_external_generation_queue_stats_and_return` 后对 `genarrative-external-generation-worker@N.service` 执行精确 `systemctl start/stop`；无需改变 HTTP 进程数。controller 只操作 `@1..@MAX` 中的缺口或最高编号多余实例，保留 `@1` 作为保底 worker。缩容或发布重启 worker 时，进程收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成；若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`，未完成任务会在 lease 过期后被其它 worker 重新领取。容器链路已有独立 `external-generation-worker` compose service；扩 worker 必须扩这个 worker service，不能只扩 `api-server` HTTP service。
+
+## 已接入的拼图纵切
+
+`compile_puzzle_draft`：
+
+1. HTTP handler 保存拼图表单草稿；`queue` 模式下 `queued/running` 的持久事实源是 `external_generation_job`，不把 HTTP 进程变成外部生成执行者。
+2. `queue` 模式下 HTTP handler 入队 `puzzle_compile_draft`，返回 `operation.status = queued` 和当前 session。拼图 dedupe key 包含本次 `extgen-` job id，只保证同一任务行唯一，不把同一 session 后续重新生成吞掉。`inline` 模式下 HTTP handler 复用同一 executor 同步执行，成功后直接返回 `completed` 和最新 session。
+3. 前端保持 `puzzle-generating`，继续轮询 `getPuzzleAgentSession`；首期不把 `queued/running` 写回 `puzzle_agent_session`，因此刷新或跨设备恢复生成中状态仍是后续 read model 工作。
+4. worker claim 后执行原有 `compile_puzzle_draft_with_initial_cover` 或 `compile_puzzle_draft_with_uploaded_cover`；前置 `compile_puzzle_agent_draft` 也必须携带本次 `job_id / worker_id / lease_token`，防止过期 worker 先把草稿卡和 session 写到 ready。
+5. 成功后沿原有 SpacetimeDB 拼图会话/作品写回，前端轮询看到 `progressPercent >= 94/96/100` 和 ready 草稿。
+6. 失败后调用 `mark_puzzle_draft_generation_failed`，拼图首期业务失败直接进入 failed；只有失败态写回成功才把队列 job 标为 failed，失败态写回失败则保留租约等待重领。队列仍保留 lease 过期后的崩溃重领，避免 worker 退款后再次成功导致钱包账本漂移。前端通过现有失败草稿/弹窗机制展示来源错误。
+
+`generate_puzzle_images`：
+
+1. HTTP handler 校验本次 `levelsJson` 快照；`queue` 模式下入队 `puzzle_generate_images` 并返回 `operation.status = queued/running/completed/failed`，`inline` 模式下同步执行原 worker executor 并在成功后返回 `completed`。
+2. worker 执行原结果页关卡图链路：自动命名、VectorEngine / 上传图直用、关卡场景图、UI spritesheet、关卡背景资产包、OSS 持久化和 SpacetimeDB 回写。
+3. 成功后 `save_puzzle_generated_images` 写回目标关卡和草稿卡；失败后 `mark_puzzle_level_generation_failed` 只标记目标关卡 `failed`，不污染已 ready 的其它关卡。队列 job 只有在目标关卡失败态写回成功后才进入 failed。
+4. 前端结果页对 `queued/running` 操作继续轮询 `getPuzzleAgentSession`，目标关卡变为 ready 或 failed 后收敛。
+
+`generate_puzzle_ui_background`：
+
+1. HTTP handler 校验本次 `levelsJson` 快照；`queue` 模式下入队 `puzzle_generate_ui_background` 并返回 `operation.status = queued/running/completed/failed`，`inline` 模式下同步执行原 worker executor 并在成功后返回 `completed`。
+2. worker 执行原结果页 UI 背景链路：归一化提示词、VectorEngine 生成、OSS 持久化和 `save_puzzle_ui_background` 写回。
+3. 成功后目标关卡写入 `uiBackgroundPrompt/uiBackgroundImageSrc/uiBackgroundImageObjectKey`；失败后复用 `mark_puzzle_level_generation_failed` 标记目标关卡 `failed`，并在失败态写回成功后才终结队列 job，让前端轮询能收敛。
+
+Match3D、Wooden Fish、Visual Novel 音频等后续外部生成 action 按同一模式迁移。
+
+## 验收
+
+基础检查：
+
+```bash
+npm run spacetime:generate
+npm run check:spacetime-schema
+npm run check:server-rs-ddd
+cargo check -p api-server --manifest-path server-rs/Cargo.toml
+```
+
+定向测试：
+
+```bash
+cargo test -p spacetime-module external_generation --manifest-path server-rs/Cargo.toml
+cargo test -p spacetime-module level_generation_failure --manifest-path server-rs/Cargo.toml
+cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml
+npm run test -- src/components/puzzle-result/PuzzleResultView.test.tsx -t "keeps generation progress visible"
+npm run test -- src/components/rpg-entry/RpgEntryFlowShell.agent.interaction.test.tsx -t "compile_puzzle_draft"
+```
+
+本地 smoke：
+
+```bash
+GENARRATIVE_PROCESS_ROLE=all npm run dev
+curl -f http://127.0.0.1:<api-port>/healthz
+```
+
+本地同步排查可显式使用 `GENARRATIVE_EXTERNAL_GENERATION_MODE=inline npm run dev:api-server`，用于确认 provider、OSS 和 SpacetimeDB 写回链路本身是否可行；该模式不覆盖 worker 队列 smoke。生产 smoke 需要保持 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`，并至少启动一个 `api` 角色、一个 `external-generation-worker` 角色和一个 `external-generation-controller` 角色；发布脚本会在默认 worker pattern 下自动启用并启动 `genarrative-external-generation-worker@1.service`，重启并验活 `genarrative-external-generation-controller.service`。若 worker 数量归零，生成任务会保持 `queued/running`，不会由 HTTP 进程偷偷执行。
+
+systemd 生产 controller 与手动兜底示例：
+
+```bash
+systemctl enable --now genarrative-external-generation-worker@1.service
+systemctl enable --now genarrative-external-generation-controller.service
+systemctl start genarrative-external-generation-worker@2.service
+systemctl stop genarrative-external-generation-worker@2.service
+systemctl status genarrative-external-generation-controller.service 'genarrative-external-generation-worker@*.service'
+```

docs/technical/【开发运维】本地SSH服务器管理面板技术方案-2026-06-11.md

@@ -0,0 +1,82 @@
+# 本地 SSH 服务器管理面板技术方案
+
+日期：`2026-06-11`
+
+## 背景
+
+release / dev 等服务器的日常巡检已经有 `genarrative-health-patrol.timer`、`/readyz`、`/healthz`、SpacetimeDB `/v1/ping` 和 systemd 状态文件，但开发者本地仍需要在多个 SSH alias 之间切换命令。服务器管理面板用于把这些只读巡检和少量 systemd 服务操作收敛到一个本地桌面入口。
+
+## 范围
+
+- 使用 Rust `egui` / `eframe` 实现本地桌面面板，不接入线上 Web 后台，不暴露公网端口。
+- 从本机 `~/.ssh/config` 的 `Host` alias 发现服务器；只展示不含通配符的 alias。
+- 支持多个服务器，左侧服务器侧边栏可收起。
+- 主面板展示硬件状态、服务状态、HTTP 健康探测和生产健康巡检状态。
+- 支持对允许的 systemd unit 执行启动、关闭、重启。
+
+## 命令入口
+
+```bash
+npm run server-manager:panel
+```
+
+等价于：
+
+```bash
+cargo run -p server-manager-panel --manifest-path server-rs/Cargo.toml
+```
+
+面板启动时会自动查找本机中文字体并注入 egui 字体集，优先使用 `Noto Sans CJK SC`，其次使用文泉驿 / Droid fallback。若某台开发机字体路径特殊，可用 `GENARRATIVE_SERVER_PANEL_CJK_FONT=/path/to/font.ttc|index` 指定；普通 `.ttf` 可省略 `|index`。
+
+## SSH 约定
+
+本地 `~/.ssh/config` 中需要存在类似：
+
+```sshconfig
+Host dev
+  HostName 10.2.0.10
+  User genarrative
+
+Host release
+  HostName genarrative.world
+  User genarrative
+```
+
+面板通过 `ssh <alias> sh -s` 执行远端只读巡检脚本。服务操作使用：
+
+```bash
+sudo -n systemctl <start|stop|restart> <unit>
+```
+
+若 SSH 用户是 root，则直接执行 `systemctl`。非 root 用户需要提前配置只允许目标 unit 的无密码 sudo；否则面板会显示 sudo 权限错误，不会弹出交互密码输入。
+
+## 健康检查内容
+
+只读巡检覆盖：
+
+- 主机名、内核、运行时长、CPU 核数 / 型号、load average。
+- 内存 / swap 使用情况。
+- `/`、`/var`、`/opt`、`/stdb`、`/data` 中存在路径的磁盘使用率。
+- `genarrative-api.service`、`spacetimedb.service`、`nginx.service`、`genarrative-health-patrol.timer`、`genarrative-database-backup.timer` 的 systemd 状态。
+- `http://127.0.0.1:8082/healthz`、`/readyz`、`http://127.0.0.1:3101/v1/ping` 和代表性公开接口。
+- `/var/lib/genarrative/health-patrol/status.json` 的最近巡检状态。
+- 若服务器安装了 `sensors`，附带温度 / 风扇等硬件传感器摘要。
+
+## 服务操作安全边界
+
+面板只允许 `start`、`stop`、`restart` 三种动作，并且 unit 名必须匹配安全字符集：
+
+```text
+A-Z a-z 0-9 . _ - @ :
+```
+
+服务操作会先出现确认弹窗，避免误点。第一版默认列出 Genarrative 生产相关 unit，并提供“其他 unit”输入框；该输入框仍只会执行 `systemctl` 的三种受控动作，不提供任意命令执行入口。
+
+## 状态判定
+
+- service / HTTP 探测失败：`CRITICAL`。
+- 磁盘使用率 `>= 95%`：`CRITICAL`，`>= 85%`：`WARNING`。
+- 内存使用率 `>= 95%`：`CRITICAL`，`>= 85%`：`WARNING`。
+- 生产健康巡检状态沿用 `OK / WARNING / CRITICAL`。
+
+面板状态只是本地巡检视图，最终运维事实仍以服务器上的 systemd、journal、Nginx 日志、`production-health-patrol.mjs` 输出和现有部署文档为准。

docs/【后端架构】SpacetimeDB连接池租约Drop兜底与取消安全-2026-06-11.md

@@ -0,0 +1,40 @@
+# SpacetimeDB 连接池租约 Drop 兜底与取消安全
+
+- 日期：2026-06-11
+- 关联故障：release 环境 api-server 周期性全量 `spacetime_stage="pool_acquire" elapsed_ms=45000` 超时，`/readyz` 503（`reason=spacetime_unhealthy, stage=pool_acquire`），重启后临时恢复。
+- 涉及代码：`server-rs/crates/spacetime-client/src/lib.rs`
+
+## 故障根因
+
+修复前的连接池存在两个叠加缺陷：
+
+1. **租约没有 Drop 兜底**。`PooledConnectionLease` 只能通过显式 `release_connection` 归还。当 HTTP 请求方在等待 StDB 回包期间断开（前端超时、用户刷新、Nginx 截断），axum/hyper 会直接丢弃 handler future，租约被 Drop：permit 因 `OwnedSemaphorePermit` 自动归还，但槽位的 `in_use` 标记永远不会复位。
+2. **acquire 在槽位泄漏后永久空转**。后续请求拿到 permit 后进入 `loop { 扫描槽位; yield_now }`，找不到空闲槽位就无限自旋，且这段自旋不受 `procedure_timeout` 约束，自旋期间 permit 不归还。
+
+叠加效果：StDB 一旦变慢（请求占用连接接近 45 秒），客户端取消请求的概率大增，每次取消泄漏一个槽位并连带吞掉一个 permit；泄漏数量达到 `pool_size`（release 为 8）后，所有业务请求与健康检查全部在 `pool_acquire` 阶段 45 秒超时，服务表现为"连不上 StDB"，只有重启能恢复。
+
+## 本地复现
+
+不需要真实 SpacetimeDB，单元测试即可复现机制（位于 `spacetime-client` tests 模块）：
+
+- 修复前：将一个槽位置为 `in_use=true` 后调用 `acquire_connection_with_timeout(200ms)`，acquire 在 5 秒守护窗口内不返回（永久自旋），测试红。
+- `dropped_lease_releases_slot_and_permit`：模拟"请求被取消、租约未经 release 直接 Drop"，断言槽位与 permit 都被复位归还。
+- `acquire_times_out_at_pool_acquire_when_pool_is_busy`：池内 permit 全部被占用时，acquire 必须在超时窗口内返回 `PoolAcquire + Timeout`，不允许无限等待。
+
+## 修复方案
+
+1. `PooledConnectionSlot` 改为 `in_use: AtomicBool + connection: Mutex<Option<PooledConnection>>`，槽位占用标记不再依赖异步锁。
+2. `PooledConnectionLease` 持有 `Arc<SpacetimeConnectionPool>` 并实现 `Drop`：无论显式归还还是 future 被取消，统一在 Drop 中复位槽位、按 broken 状态决定连接是否回池，permit 随后自动归还。Drop 体先复位 `in_use` 再释放 permit（字段在 Drop 体之后析构），保证新请求拿到 permit 时必有空闲槽位。
+3. acquire 改为 CAS 抢占槽位：持有 permit 即保证并发持有者不超过 `pool_size`，扫描一轮必然命中空闲槽位，彻底删除自旋循环；建连失败直接返回错误，槽位由租约 Drop 复位。
+4. `release_connection` 退化为 `drop(lease)`，显式与隐式归还共用同一条兜底路径。
+
+## 验收
+
+- `cargo test -p spacetime-client --manifest-path server-rs/Cargo.toml --lib`（35 通过，含上述新测试）
+- `cargo test -p api-server --manifest-path server-rs/Cargo.toml readyz`（2 通过）
+- `cargo check -p api-server --manifest-path server-rs/Cargo.toml`
+
+## 运维提示
+
+- 此修复解决的是"取消导致的永久泄漏"。StDB 真慢时仍会出现成批 45 秒超时（连接被在途请求合法占用），那是容量/上游问题，应结合 `GENARRATIVE_SPACETIME_POOL_SIZE` 与 StDB 负载排查，不要再怀疑池泄漏。
+- 健康检查 `/readyz` 在池被在途请求占满时仍可能短暂 503（stage=pool_acquire），恢复后自动转好，无需重启。

docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md

@@ -113,7 +113,7 @@ npm run check:server-rs-ddd
 - `server-rs/crates/api-server/src/state.rs` 中的 `PuzzleApiState` 是拼图 HTTP/BFF 的 Feature State，集中暴露 `SpacetimeClient`、`PuzzleGalleryCache`、OSS client、作者查询所需认证服务、拼图 LLM client 和少量 VectorEngine / Agent 配置快照。拼图 handler 只提取 `State<PuzzleApiState>`，不得重新改回 `State<AppState>`。
 - `server-rs/crates/api-server/src/puzzle.rs` 只作为聚合入口，保留共享 import / 常量、内部模块声明和 handler re-export，不继续承载大段实现。
 - `server-rs/crates/api-server/src/puzzle/handlers.rs` 承接 Axum handler，负责 extract、鉴权上下文、调用 SpacetimeDB facade / 编排 helper，并返回 HTTP/SSE 响应。
-- `server-rs/crates/api-server/src/puzzle/draft.rs` 承接表单草稿保存、草稿编译、首关命名、UI 背景 prompt、降级 snapshot 和初始资产就绪校验。
+- `server-rs/crates/api-server/src/puzzle/draft.rs` 承接表单草稿保存、草稿编译、首关命名、UI 背景 prompt 和初始资产就绪校验。
 - `server-rs/crates/api-server/src/puzzle/generation.rs` 承接拼图图片与 UI 背景的生成编排、计费包裹和 reference image 路径选择。
 - `server-rs/crates/api-server/src/puzzle/vector_engine.rs` 承接 VectorEngine 请求体、HTTP 调用、下载 / base64 解码、OSS 写入、asset object / binding 持久化和上游错误归一。
 - `server-rs/crates/api-server/src/puzzle/mappers.rs` 承接 SpacetimeDB record 到 shared-contracts DTO 的映射。
@@ -125,11 +125,14 @@ npm run check:server-rs-ddd
 
 `/api/runtime/puzzle/runs*` 当前接受 `RuntimePrincipal`，可同时识别登录用户 Bearer 和 runtime guest token。推荐页嵌入运行态的正式开局、交换、拖拽、下一关、暂停、道具与排行榜请求，应由前端在登录态下继续携带账号 access token；匿名游客仅在确认为未登录时走 runtime guest token。不要再把拼图 runtime 当成只认普通 Bearer 的纯账号接口。
 
+公开正式 runtime 的启动与局内同步动作统一接受 `RuntimePrincipal`，包括拼图、拼消消、跳一跳、敲木鱼、抓大鹅 Match3D、方洞挑战、视觉小说、大鱼吃小鱼和汪汪声浪。登录用户仍使用账号 Bearer；未登录推荐页或公开运行态使用 Runtime Guest Token，后端以 `principal.subject()` 作为本局 owner / player subject，并用 `WorkPlayTrackingDraft::runtime_principal(...)` 记录游玩。创作、个人作品、删除、发布、Remix、点赞等账号或所有权动作不得改成 runtime guest 鉴权。
+
 抓大鹅 Match3D `api-server` 内部拆分：
 
 - `server-rs/crates/api-server/src/modules/match3d.rs` 继续负责路由装配和 body limit；对外 handler 名称保持不变。
 - `server-rs/crates/api-server/src/match3d.rs` 只作为聚合入口，保留共享 import / 常量 / 内部类型、模块声明和 handler re-export。
 - `server-rs/crates/api-server/src/match3d/handlers.rs` 承接 Axum handler，负责 extract、鉴权上下文、调用 SpacetimeDB facade / 编排 helper，并返回 HTTP 响应。
+- `/api/runtime/match3d/works/{profile_id}/runs`、`/api/runtime/match3d/runs/{run_id}`、`/click`、`/stop`、`/restart` 与 `/time-up` 属于正式运行态局部请求，必须接受 `RuntimePrincipal`；登录用户使用账号 Bearer，推荐页匿名游客使用 runtime guest token，后端以 principal subject 作为本局 owner，不得退回只认普通 Bearer 的路由。
 - `server-rs/crates/api-server/src/match3d/draft.rs` 承接 Agent session、草稿编译、题材 / 难度 / 物品计划和草稿持久化编排。
 - `server-rs/crates/api-server/src/match3d/works.rs` 承接作品 CRUD、封面 / 背景 / 容器资产生成入口、发布 / Remix / 点赞 / 游玩记录和作品级 helper。
 - `server-rs/crates/api-server/src/match3d/item_assets.rs` 承接物品生成批次编排、append / replace / delete / sort / merge、计费外层和草稿素材映射；sheet prompt、绿幕 / 近白底透明化、切图和切片持久化复用 `generated_asset_sheets` 通用模块。
@@ -187,6 +190,10 @@ npm run check:server-rs-ddd
 1. `creation_entry_type_config.unified_creation_spec_json` 内的 `mudPointCost` 是玩法新建草稿初始生成的泥点成本真相源，同时供入口卡展示和前端余额前置校验使用；旧契约缺失时允许按代码默认成本兜底。
 2. `api-server` 执行拼图首图生成、抓大鹅完整草稿生成和汪汪声浪初始三图生成时，必须通过 `GET /api/creation-entry/config` 同源配置解析对应玩法成本后再调用钱包扣费 procedure，不得继续使用前端或后端硬编码常量作为实际扣费真相。
 3. 结果页单图重生成、发布、道具使用和其它独立资产操作仍按各自业务操作成本执行；不要把初始草稿成本误套到这些单次操作上。
+4. 资产操作的预扣费必须 fail-closed：钱包或 SpacetimeDB 预扣费不可达、超时或返回业务错误时，`api-server` 直接返回错误，不允许继续调用图片、音频、GLB 等外部生成 provider。
+5. 需要支持 HTTP retry 的计费 ledger id 必须包含当前请求的 `request_id`；前端 `fetchWithApiAuth` 同一次业务请求的静默刷新重试复用同一个 `x-request-id`，后端不得再使用 prompt 指纹或随机 asset id 作为扣费幂等键。
+6. 外部生成已预扣费但后续失败时必须先同步调用钱包退款；若 SpacetimeDB 暂不可用，退款请求写入 `wallet-refund-outbox` 本地文件并由后台 worker 重放。默认启用，配置项为 `GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED`、`GENARRATIVE_WALLET_REFUND_OUTBOX_DIR`、`GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE`、`GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS` 和 `GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES`。outbox 文件按 refund ledger id 幂等落盘；成功重放后删除，坏文件隔离为 `corrupt-*`。
+7. 拼图首图后台生成的跨实例互斥锁必须落在 SpacetimeDB `puzzle_background_compile_task` 表，claim id 由 `task_id + request_id` 构成，释放时必须校验 claim id，避免旧后台任务释放新请求抢到的租约。
 
 ## 外部服务与资产
 
@@ -226,6 +233,12 @@ npm run check:server-rs-ddd
 - Rust 结构体：`AiTaskStage`
 - 源码：`server-rs/crates/spacetime-module/src/ai/stages.rs`
 
+### `external_generation_job`
+
+- Rust 结构体：`ExternalGenerationJob`
+- 源码：`server-rs/crates/spacetime-module/src/external_generation.rs`
+- 用途：外部生成 worker 的持久任务队列；`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时，`api-server` HTTP 角色只入队，`external-generation-worker` 角色通过 claim lease 领取、续租、执行，并用 `lease_token` 栅栏回写完成 / 失败。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft`、`generate_puzzle_images` 与 `generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，避免过期 worker 写 session / work profile；`GENARRATIVE_EXTERNAL_GENERATION_MODE=inline` 时不创建该队列行，三个 external generation guard 字段必须同时为空才允许 api-server 受控同步写回，半空 guard 仍会拒绝。worker 成功写回业务事实后才能 complete job；业务失败态写回成功后才能 fail job，失败态未写回时保留租约等待后续重领。
+
 ### `ai_text_chunk`
 
 - Rust 结构体：`AiTextChunk`
@@ -637,6 +650,12 @@ npm run check:server-rs-ddd
 - Rust 结构体：`PuzzleAgentSessionRow`
 - 源码：`server-rs/crates/spacetime-module/src/puzzle.rs`
 
+### `puzzle_background_compile_task`
+
+- Rust 结构体：`PuzzleBackgroundCompileTaskRow`
+- 源码：`server-rs/crates/spacetime-module/src/puzzle.rs`
+- 说明：拼图首图后台生成的跨 api-server 实例互斥 claim 表，只保存活动任务租约，不表达最终生成结果；`task_id` 为主键，`claim_id` 用于释放时防止误删新租约，租约超时时间为 30 分钟。
+
 ### `puzzle_event`
 
 - Rust 结构体：`PuzzleEvent`

docs/【开发运维】本地开发验证与生产运维-2026-05-15.md

@@ -51,6 +51,10 @@ Linux 本机多用户并发开发时，`npm run dev` 和 `npm run dev:*` 单模
 
 开发态 `npm run dev` 与 `npm run dev:api-server` 会默认注入 `GENARRATIVE_DEV_PASSWORD_ENTRY_AUTO_REGISTER_ENABLED=true`，因此密码登录在本地开发环境可直接注册未知手机号账号；生产环境仍按 `api-server` 配置默认关闭该开关。
 
+本地排查外部内容生成 worker 时，可临时用 `GENARRATIVE_PROCESS_ROLE=all npm run dev:api-server` 让同一 Rust 进程同时监听 HTTP 并消费 `external_generation_job` 队列。该模式只用于 smoke；生产默认 `GENARRATIVE_PROCESS_ROLE=api`，外部生成任务由独立 `GENARRATIVE_PROCESS_ROLE=external-generation-worker` 进程消费。外部生成执行策略由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制，生产与容器扩缩容验证保持 `queue`，拼图首图 `compile_puzzle_draft`、结果页关卡图片 `generate_puzzle_images` 和结果页 UI 背景 `generate_puzzle_ui_background` 会进入持久队列；worker 数量为 0 时，HTTP 只返回 queued/running，不会兜底执行外部 provider。本地如果要让 `npm run dev` 或 `npm run dev:api-server` 同步等待生成结果，应在 `.env.local` 或本机环境显式配置 `GENARRATIVE_EXTERNAL_GENERATION_MODE=inline`，由 handler 直接复用 worker executor 并在完成后返回 `completed`；该配置不得硬编码进 `scripts/dev.mjs`，且 inline 不创建 `external_generation_job`、不提供动态扩缩容能力。
+
+需要验证“更新 API 不停 worker”和“worker 是否持续消费队列”时，优先使用隔离容器 smoke：`npm run container:worker-smoke -- smoke`。该脚本生成 gitignored 的 `deploy/container/worker-smoke/api-server.env`，启动独立 compose project 与独立 SpacetimeDB，发布当前 `spacetime-module` 后写入 `worker_smoke_unsupported` 测试 job；预期 worker claim 后执行 unsupported 失败分支，再执行 API-only recreate 并确认 worker 容器 ID 不变，最后再次入队验证 API 更新后队列仍可消费。`external_generation_job` 是 private table，脚本通过 worker 日志确认 job_id 被消费，不用 CLI SQL 查询私表。该 smoke 不读取 `.env.local`，也不依赖真实 VectorEngine / OSS 密钥；真实生图链路联调再在本地私有 env 中补齐 provider 配置。worker-smoke 默认把本机 `spacetime` CLI 打成轻量 SpacetimeDB 镜像，避免本机首次 smoke 依赖官方大镜像下载。若容器内 Cargo 拉取 crates.io 依赖不稳定，可用 `npm run container:worker-smoke -- smoke --local-binary` 让容器内 Cargo 复用本机 Cargo 缓存构建当前二进制，再打入 Debian bookworm smoke runtime 临时镜像；可用 `GENARRATIVE_WORKER_SMOKE_LOCAL_BASE_IMAGE` 覆盖运行时基础镜像；若隔离端口或库数据需要重建，追加 `--force`。
+
 本地只做账号/UI smoke 且需要短信登录时，`SMS_AUTH_PROVIDER` 应显式设为 `mock`，并把 `SMS_AUTH_MOCK_VERIFY_CODE` 设为固定值（当前常用 `123456`），再重启 `npm run dev` 或 `npm run dev:api-server`。如果 `.env.local` 还保留 `SMS_AUTH_PROVIDER=aliyun`，`POST /api/auth/phone/login` 用 mock 验证码会稳定报“验证码错误”，不是前端表单问题。真实短信联调再切回 `aliyun` 并重启。
 
 微信小程序虚拟支付使用 `WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_OFFER_ID`、`WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_APP_KEY`、`WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_SANDBOX_APP_KEY` 和 `WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_ENV` 配置。小程序充值统一走 `wechat_mp_virtual` / `wx.requestVirtualPayment`：泥点属于代币（`coin`），`buyQuantity` 按当前充值商品快照里的 `points_amount` 传；会员和后台新增道具类商品走 `short_series_goods`，`productId` 对应微信后台道具 ID。旧登录快照若缺 `session_key`，需要用户在小程序内重新登录后再支付；客户端成功回调不是最终到账，仍以后端通知或查询确认订单为准。详细口径见 `docs/【技术方案】微信虚拟支付接入-2026-05-26.md`。
@@ -202,7 +206,7 @@ UI 相关修改要重点验证：
 
 ## SpacetimeDB 操作规则
 
-1. 不在人工命令、本地联调或文档示例中使用 `spacetime --root-dir`。
+1. 不在人工命令、本地联调或文档示例中使用 `spacetime --root-dir`；CI/CD 脚本内部为隔离运行用户登录态的受控用法例外，但不得写成手工排障命令。
 2. 本地开发使用项目脚本维护数据目录；需要清空本地数据时先确认可丢弃，再停止服务并处理本地数据目录。
 3. 发布目标必须显式 `--server` / `--server-url`。
 4. 身份问题先查 `spacetime login show`、`spacetime server list` 和目标库权限，不通过切回旧 Node / PostgreSQL 绕过。
@@ -212,7 +216,7 @@ UI 相关修改要重点验证：
 
 ### SpacetimeDB 数据目录 OSS 备份
 
-数据库备份不放进 `spacetime-module` reducer / procedure：备份属于文件系统与 OSS 外部副作用，必须由运维脚本在 SpacetimeDB 宿主外执行。当前统一脚本为；生产 provision 还会安装 `genarrative-database-backup.timer`，每天 `03:20` 左右自动执行一次 OSS 冷备份：
+数据库备份不放进 `spacetime-module` reducer / procedure：备份属于文件系统与 OSS 外部副作用，必须由运维脚本在 SpacetimeDB 宿主外执行。当前统一脚本为 `scripts/database-backup-to-oss.mjs`（npm 命令 `npm run database:backup:oss`）；生产 provision 还会安装 `genarrative-database-backup.timer`，每天 `03:20` 左右自动执行一次 OSS 冷备份：
 
 ```bash
 npm run database:backup:oss -- --data-dir /stdb --stop-service spacetimedb.service --restart-service-after genarrative-api.service
@@ -260,11 +264,12 @@ Jenkins 按 web / api / Spacetime module / build / deploy / publish 拆分
 
 `Genarrative-Server-Provision` 会安装并启用 `genarrative-health-patrol.timer`，默认每 5 分钟运行一次 `genarrative-health-patrol.service`。巡检脚本随 API release 归档到 `/opt/genarrative/current/scripts/ops/production-health-patrol.mjs`，只读检查：
 
-- `genarrative-api.service`、`spacetimedb.service`、`nginx.service` 是否 active。
+- `genarrative-api.service`、`genarrative-external-generation-controller.service`、`spacetimedb.service`、`nginx.service` 是否 active。
+- 至少一个 `genarrative-external-generation-worker@*.service` 实例是否 active；如果 controller 存活但 worker 全部退出，巡检直接返回 `CRITICAL`，避免外部生成队列长期无人消费。
 - API 直连 `/healthz`、`/readyz`。
 - SpacetimeDB 直连 `/v1/ping`。
 - 默认直连 API 端口检查 `/api/creation-entry/config`、`/api/runtime/puzzle/gallery`、`/api/runtime/custom-world-gallery`；如需走 Nginx / 公网域名，在 `/etc/genarrative/health-patrol.env` 配置 `GENARRATIVE_HEALTH_PATROL_PUBLIC_BASE_URL=https://<域名>`。
-- 最近 15 分钟 `genarrative-api.service`、`spacetimedb.service`、`nginx.service` 的 `err..alert` 日志。
+- 最近 15 分钟 `genarrative-api.service`、`genarrative-external-generation-controller.service`、`genarrative-external-generation-worker@*.service`、`spacetimedb.service`、`nginx.service` 的 `err..alert` 日志。
 
 巡检输出总状态 `OK / WARNING / CRITICAL`；只有 `CRITICAL` 默认让 systemd service 失败，`WARNING` 只写日志和状态文件，避免历史日志噪声把 timer 长期打成失败。最近一次结果写入 `/var/lib/genarrative/health-patrol/status.json`。手动执行：
 
@@ -302,7 +307,9 @@ dev 服务器上的 Gitea 内网入口固定为 `http://10.2.0.10/GenarrativeAI/
 
 生产环境变量模板：`deploy/env/api-server.env.example`。真实密钥只放服务器，不提交 Git，不写入文档示例。
 
-`Genarrative-Server-Provision` 会安装 systemd 模板和 Nginx 站点模板，不再安装 clang / lld / pkg-config / OpenSSL headers / sccache 等通用构建链依赖。因 VectorEngine 图片上游 POST 已改用 `libcurl`，当前 Linux release 构建出的 `api-server` 运行时需要 `OPENSSL_3.2.0` 符号；Ubuntu 24.04 apt 默认只提供 OpenSSL 3.0.x，不能直接满足该符号版本。Provision 会把 OpenSSL `3.2.0` 独立安装到 `/opt/genarrative/openssl-3.2.0`，校验官方 tarball SHA256，并只通过 `genarrative-api.service` 的 `LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib` 让 api-server 使用，避免替换系统 OpenSSL 或影响 ssh / nginx / apt。Ubuntu / apt 目标机会额外安装 `libnginx-mod-http-brotli-filter` 与 `libnginx-mod-http-brotli-static`，随后由 `scripts/jenkins-server-provision.sh` 通过临时 `nginx -t` 配置探测 Brotli 指令是否可用；该临时配置必须先 `include /etc/nginx/modules-enabled/*.conf`，因为 apt 安装的 Brotli 是动态模块，不会出现在普通 `nginx -V` 编译参数里。探测成功才在渲染后的 `deploy/nginx/genarrative.conf` / `genarrative-dev-http.conf` 中启用 Brotli，避免未安装模块的机器直接写入无效配置。Provision 写入 Genarrative Nginx 站点时会把 `/etc/nginx/sites-enabled/default*` 移到 `/etc/nginx/sites-disabled/`，避免 Debian / Certbot 默认站点继续占用 `genarrative.world` / `www.genarrative.world` 并在 `nginx -T` 中出现 `conflicting server name ... ignored`。如果 `nginx -t` 失败，脚本会恢复写入前的 Genarrative 配置和被移动的默认站点。
+`api-server` 进程角色由 `GENARRATIVE_PROCESS_ROLE` 控制：`api` 只监听 HTTP，`external-generation-worker` 只消费外部生成队列，`external-generation-controller` 只管理 worker systemd 实例，`all` 仅用于本地或临时 smoke，不隐式启动 controller。外部生成策略由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制，生产和容器压测默认保持 `queue`；`inline` 只用于本地或低并发同步排查，HTTP handler 会直接复用 worker executor，完成后返回 `completed`，但不会落 `external_generation_job`，也不能通过增加 worker 进程扩吞吐。外部生成 worker 使用同一发布包和同一套 SpacetimeDB 配置，按实例数和 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩；生产默认由 `genarrative-external-generation-controller.service` 读取 `get_external_generation_queue_stats_and_return`，按 `claimable_pending + running_active + expired_running` 计算目标 worker 数，并对 `genarrative-external-generation-worker@N.service` 精确执行 `systemctl start/stop`。controller 参数模板是 `deploy/env/external-generation-controller.env.example`：默认保底 `MIN_WORKERS=1`、上限 `MAX_WORKERS=8`、每 worker 目标 `TARGET_JOBS_PER_WORKER=2`、`POLL_INTERVAL_MS=10000`、连续 `SCALE_DOWN_IDLE_ROUNDS=6` 轮完全空闲才缩容；缩容每轮只停止最高编号的一个实例，且不主动停止 `@1`。worker 收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成；若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`，未完成任务才会在 lease 过期后由其它 worker 重领。每个 worker 实例应设置唯一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`，默认会用主机名和 pid 兜底；systemd 生产模板 `deploy/systemd/genarrative-external-generation-worker@.service` 会用 `%H-%i` 生成实例 ID，并把 tracking outbox 隔离到 `/var/lib/genarrative/tracking-outbox/%H-%i`。`Genarrative-Server-Provision` 会安装 worker 模板、controller unit 和两份专属 env 模板，默认 enable 首个 `genarrative-external-generation-worker@1.service` 与 `genarrative-external-generation-controller.service`；首次 API deploy 会在默认 worker pattern 下自动 `enable --now genarrative-external-generation-worker@1.service` 并等待 worker active，同时重启并验活 controller。手动兜底扩容仍可用 `systemctl start genarrative-external-generation-worker@2.service` / `@3.service`，缩容用 `systemctl stop genarrative-external-generation-worker@N.service`；controller 下轮会按队列压力修正到目标实例数。worker 专属参数模板是 `deploy/env/external-generation-worker.env.example`，密钥与 SpacetimeDB 连接仍复用 `/etc/genarrative/api-server.env`。API 发布脚本默认会重启并验活 `genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service`；若本次只发 HTTP 且不希望滚动 worker，可传 `--no-worker-services`，若不希望重启 controller 可传 `--no-worker-controller`。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 控制空队列轮询间隔，`GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS` 控制单次 lease，worker 会约每三分之一 lease、最长 30 秒续租；该值应覆盖一次心跳网络抖动窗口，不需要大于完整外部生成链路耗时。SpacetimeDB 使用自身事务时间计算 claim/renew/complete/fail，完成和失败回写还会校验 `lease_token` 与未过期 lease，避免同一 job 被过期 worker 覆盖。当前拼图首关生成只做 lease 崩溃重领，不做业务失败自动重试，避免 worker 退款和重试成功之间产生钱包账本漂移。
+
+`Genarrative-Server-Provision` 会安装 systemd 模板和 Nginx 站点模板，不再安装 clang / lld / pkg-config / OpenSSL headers / sccache 等通用构建链依赖。因 VectorEngine 图片上游 POST 已改用 `libcurl`，当前 Linux release 构建出的 `api-server` 运行时需要 `OPENSSL_3.2.0` 符号；Ubuntu 24.04 apt 默认只提供 OpenSSL 3.0.x，不能直接满足该符号版本。Provision 会把 OpenSSL `3.2.0` 独立安装到 `/opt/genarrative/openssl-3.2.0`，校验官方 tarball SHA256，并只通过 `genarrative-api.service` 的 `LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib` 让 api-server 使用，避免替换系统 OpenSSL 或影响 ssh / nginx / apt。Ubuntu / apt 目标机为完成这一步会安装 `build-essential`、`ca-certificates`、`curl`、`perl`、`tar` 等 OpenSSL 运行时自举工具；这只服务于独立 OpenSSL 运行时安装，不代表 provision 重新承担 api-server 构建职责。Ubuntu / apt 目标机会额外安装 `libnginx-mod-http-brotli-filter` 与 `libnginx-mod-http-brotli-static`，随后由 `scripts/jenkins-server-provision.sh` 通过临时 `nginx -t` 配置探测 Brotli 指令是否可用；该临时配置必须先 `include /etc/nginx/modules-enabled/*.conf`，因为 apt 安装的 Brotli 是动态模块，不会出现在普通 `nginx -V` 编译参数里。探测成功才在渲染后的 `deploy/nginx/genarrative.conf` / `genarrative-dev-http.conf` 中启用 Brotli，避免未安装模块的机器直接写入无效配置。Provision 写入 Genarrative Nginx 站点时会把 `/etc/nginx/sites-enabled/default*` 移到 `/etc/nginx/sites-disabled/`，避免 Debian / Certbot 默认站点继续占用 `genarrative.world` / `www.genarrative.world` 并在 `nginx -T` 中出现 `conflicting server name ... ignored`。如果 `nginx -t` 失败，脚本会恢复写入前的 Genarrative 配置和被移动的默认站点。
 
 50 HTTP req/s 首版压测优化口径：
 
@@ -319,7 +326,7 @@ dev 服务器上的 Gitea 内网入口固定为 `http://10.2.0.10/GenarrativeAI/
 - 作品列表短期继续由 `api-server` / BFF 订阅 SpacetimeDB 公开 read model 后读本地 cache，不让浏览器前端直接订阅完整列表；未来如新增 `public_work_gallery_entry` 等专用公开作品列表 read model，前端只可订阅稳定、低基数、公开的专用投影，禁止订阅 `puzzle_work_profile`、`custom_world_profile` 等玩法源表后自行 join、聚合或判断权限。前端直订阅落地前必须先补齐权限、字段契约、排序 / 分页、埋点和 BFF 回退策略。
 - 50 HTTP req/s 验收目标为 `http_req_failed < 1%`、`p95 < 2s`、`dropped_iterations = 0`，同时压测窗口内 Nginx 无新增 502。2026-05-19 容器 2C / 2G 连续 10 轮不重启 SpacetimeDB 压测：`PEAK_RPS=2500` 等价约 5000 HTTP req/s，平均实际吞吐约 `4219 HTTP req/s`，10 轮总计 `1,897,357` 个 200、`212,542` 个 429、`0` 个 5xx，200 请求平均 `p95=123ms`、`p99=234ms`；该档会把 SpacetimeDB 容器内存从约 `366MiB` 推到约 `885MiB / 896MiB`，因此当前不要继续抬公开 gallery 入口并发，应优先处理 SpacetimeDB 侧连接 / 订阅 / tracking 写入后的内存高水位。
 
-容器化压测与隔离部署方案单独放在 `deploy/container/`，用于本机或预发模拟 Linux release + Nginx + OTLP Collector 拓扑，不替换当前生产 `systemd + Nginx + Jenkins` 发布路径。当前容器模拟参数按 `genarrative-release` 采样值收口为 2 vCPU / 2 GiB RAM / `nofile=4096` / `worker_connections=768`，并在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=768m`、`api-server cpus=2.0 mem_limit=1g`、`nginx cpus=0.25 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=0.5 mem_limit=512m`。容器 `api-server` 默认 `GENARRATIVE_API_WORKER_THREADS=4`，只增加 Tokio worker 调度并发，不突破 `api-server cpus=2.0` 的 CPU 配额：
+容器化压测与隔离部署方案单独放在 `deploy/container/`，用于本机或预发模拟 Linux release + Nginx + OTLP Collector 拓扑，不替换当前生产 `systemd + Nginx + Jenkins` 发布路径。当前容器模拟参数按 `genarrative-release` 采样值收口为 2 vCPU / 2 GiB RAM / `nofile=4096` / `worker_connections=768`，并在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`external-generation-worker cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`。容器 `api-server` 默认 `GENARRATIVE_API_WORKER_THREADS=4`，只增加 Tokio worker 调度并发，不突破 `api-server cpus=2.0` 的 CPU 配额；容器默认 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`，可用 `npm run container:up -- --scale external-generation-worker=N external-generation-worker` 验证外部生成 worker 动态扩缩容，`inline` 模式不参与该验证：
 
 ```bash
 npm run container:init
@@ -332,6 +339,7 @@ npm run container:down
 
 容器方案默认暴露 `http://127.0.0.1:18080`，`api-server` 在容器内监听 `0.0.0.0:8082`，Nginx 通过 `api-server:8082` upstream 反代 `/api/` 和 `/admin/api/`。SpacetimeDB 也纳入 compose，容器内由 `spacetimedb:3101` 提供服务，宿主机通过 `http://127.0.0.1:13101` 进行模块发布；Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。生产 provision 侧现在由目标 dev / release agent 自己准备 `provision-tools/otelcol-contrib`，并安装本机 `otelcol-contrib.service`，真实库名、token 和外部服务密钥只写本地 `deploy/container/api-server.env`，不提交 Git。完整拓扑、端口、k6 参数和 OTLP debug exporter 使用方法见 `deploy/container/README.md`。
 `npm run container:config` 默认只做 quiet 校验，避免把本地 env 中的 token 展开到终端；确需排查完整 compose 时再传 `-- --print`。
+隔离验证 worker 队列和 API-only 更新时使用 `npm run container:worker-smoke -- smoke`。该命令不复用 `deploy/container/api-server.env`，会在 `deploy/container/worker-smoke/` 生成本机专用 env 与端口 state，并使用 unsupported job 验证 worker claim / fail 回写，不需要真实外部生成密钥；本机 crates.io 网络不稳时使用 `--local-binary`，由容器内 Cargo 复用本机 Cargo 缓存构建，并把产物放进 Debian bookworm smoke runtime。
 
 OpenTelemetry 现阶段默认开启 OTLP traces / metrics / logs，但本地日志与 Nginx 文件日志仍保留：
 

docs/【玩法创作】平台入口与玩法链路-2026-05-15.md

@@ -146,12 +146,14 @@ RPG / 拼图等运行态存档仍以 `/api/profile/save-archives` 的后端列
 - 拼图试玩和正式运行态刷新恢复不复用创作私有 query。进入 `/runtime/puzzle` 时必须写入 `runtimeProfileId`、草稿 `runtimeSessionId`、可选 `runtimeLevelId`、公开作品 `work` 和 `mode=draft|published`；进入运行态的导航顺序必须先切到 `/runtime/puzzle`，再写这些 runtime query，避免被阶段导航清掉后刷新停在“正在进入拼图关卡”。
 - 结果页生成关卡图时若关卡名为空，前端必须传 `shouldAutoNameLevel=true`，后端复用首关命名契约先按画面描述生成关卡名，再在图片生成后用视觉命名结果精修，并把生成名和 UI 背景提示词随本次关卡快照写回。
 - 拼图运行态背景优先读取当前关卡 `levelBackgroundImageSrc/levelBackgroundImageObjectKey`，旧数据才兼容 `uiBackgroundImageSrc/uiBackgroundImageObjectKey`；本地试玩、直达指定关卡和正式 `next-level` 推进时，目标关卡缺关卡背景时必须继承同作品首个可用关卡背景，仍缺失时才沿用当前运行态快照背景或默认 UI。运行态按钮视觉优先读取当前关卡 `uiSpritesheetImageSrc/uiSpritesheetImageObjectKey`，先按透明 alpha 自动边界检测识别 spritesheet 中的独立按钮展示矩形，再按原图位置从左到右、从上到下映射到返回、设置、下一关、提示、原图、冻结；同一组件还要按较高 alpha 阈值派生紧致点击热区，透明留白和柔边低 alpha 区域尽量不响应点击。检测失败时回退旧固定六格裁切，缺失时才用现有图标按钮兜底。有 spritesheet 时，返回、设置和下一关的点击容器只提供透明点击区，不再叠加默认白色圆形底、胶囊主按钮底或额外文字；下一关按钮在通关弹窗和底部入口中都直接使用 spritesheet 裁切出的 next 素材作为按钮本体。底部提示、原图、冻结三枚素材按检测矩形的原始宽高比显示，不能强行拉伸成正圆或铺满整列。底部道具区不再使用连片胶囊背景，提示、原图、冻结三个按钮均匀分布；运行态只展示按钮素材本身，不额外叠加“提示 / 原图 / 冻结”文字。
-- 推荐页本身不是登录门禁入口，未登录用户点击底部或侧边栏的推荐 Tab 应直接进入嵌入运行态，不主动打开登录弹窗。推荐页嵌入运行态必须按真实身份分流：已登录用户或本地已有 access token 时，启动拼图和后续排行榜 / 下一关等正式请求继续走账号 Bearer；只有确认为匿名访客时才申请并透传 runtime guest token。`/api/runtime/puzzle/runs*` 后端统一接受 `RuntimePrincipal`，可识别账号用户和匿名 runtime guest；推荐卡片的后台读写请求仍使用 local auth impact，避免单卡 401 清空整站登录态。创作、个人作品、删除、发布、Remix 等账号或所有权动作仍保持普通用户鉴权。
+- 推荐页本身不是登录门禁入口，未登录用户点击底部或侧边栏的推荐 Tab 应直接进入嵌入运行态，不主动打开登录弹窗。推荐页嵌入运行态必须按真实身份分流：已登录用户或本地已有 access token 时，正式 runtime 启动与后续局内动作继续走账号 Bearer；只有确认为匿名访客时才申请并透传 runtime guest token。平台壳统一通过 `buildRecommendRuntimeRequestOptions(...)` 为各玩法的 start / checkpoint / finish / input / drop / click / restart / time-up / leaderboard / next-level 等动作生成局部 request options，不允许每个玩法各写一套匿名分支。后端 `/api/runtime/*` 正式运行态写请求统一接受 `RuntimePrincipal`，可识别账号用户和匿名 runtime guest；推荐卡片的后台读写请求仍使用 local auth impact，避免单卡 401 清空整站登录态。创作、个人作品、删除、发布、Remix 等账号或所有权动作仍保持普通用户鉴权。
+- 推荐页作品队列只能通过 `buildPlatformRecommendFeedEntries(...)` 生成，首页卡片窗口、桌面推荐格、嵌入 runtime 自动启动和上一条 / 下一条切换都必须消费同一队列。不得在首页和 `PlatformEntryFlowShellImpl` 内分别按“最新列表顺序”和“评分推荐顺序”各算一套相邻作品，否则连续切换会出现视觉上跳过作品或回跳。
+- 推荐页作品信息区的分享按钮统一唤起发布分享弹窗 `PublishShareModal`，不在推荐卡内部单独拼接分享文案或只做剪贴板复制反馈；拼图推荐作品的 H5 分享链接继续沿用 `/gallery/puzzle/detail?work=...`，其它统一公开作品默认走 `/works/detail?work=...`。微信小程序 WebView 内复制动作必须改为小程序 `pages/web-view/index` 路径并补齐 `targetPath=/works/detail` 与 `work` 参数。推荐页当前 active 作品必须通过 `wx.miniProgram.postMessage` 同步给原生 `web-view` 页，让右上角系统“转发给朋友”和“分享到朋友圈”也使用当前作品参数生成小程序短链背后的 path。微信小程序 WebView 内的推荐页运行态需要启用分享快照安全区，把游戏画面等比缩放并保持在页面中部，避免用户直接点击小程序自带“分享到聊天”时只截到游戏画面局部。
 - 拼图运行态棋盘不叠加分块蒙版、描边、阴影、选中底色或合并块 SVG 轮廓；拼图片本体需要裁切为圆角形状，单块使用独立圆角裁切，合并块使用 SVG 原生 `clipPath` 裁切整体外轮廓，外凸角和内凹角分别计算半径，内凹角半径要比外凸角更明显以避免手机 WebView 中看起来仍是直角。原图道具只在用户主动确认后打开独立原图查看层，不在当前拼图棋盘上叠加原图。
 - 拼图运行态拖拽必须完全跟随手指或鼠标位置，`pointermove` 期间即时写入可见拼块的 transform，不依赖等待后端回包、React 重渲染或下一帧动画队列；进入拖动后不展示拼块选中态或“已选择”提示，松手后再提交目标格同步规则真相。
 - 拼图运行态的提示、设置等点击弹层跟随当前运行态主色主题，使用普通圆角主题面板，不复用像素九宫格素材框。
 - 拼图运行态壳层自身要补齐 `platform-ui-shell` / `platform-theme` / `platform-theme--light|dark`，不能依赖外层平台壳来提供主题变量；`/puzzle` 直达页和平台内嵌页都必须渲染同一套主题语义类。
-- 拼图运行态顶部关卡信息采用游戏化铭牌样式：橘棕横向关卡名牌承载 `第 N 关` 和关卡名，左侧固定使用 `media/logo.png` 卡通形象；倒计时作为下挂米白小牌独立显示，紧贴铭牌但不遮挡棋盘。该样式只改变运行态 HUD 视觉，不改变计时、暂停、失败同步或关卡推进规则。
+- 拼图运行态顶部关卡信息采用游戏化铭牌样式：橘棕横向关卡名牌承载 `第 N 关` 和关卡名，左侧固定使用 `media/logo-runtime-hud.webp` 卡通形象小图；倒计时作为下挂米白小牌独立显示，紧贴铭牌但不遮挡棋盘。该样式只改变运行态 HUD 视觉，不改变计时、暂停、失败同步或关卡推进规则。
 - 拼图运行态进行中关卡的 `elapsedMs` 仍是结算字段，设置面板的“当前用时”必须按 `startedAtMs`、暂停累计和冻结累计实时派生；不要直接把进行中的 `currentLevel.elapsedMs` 当作展示值。
 - 推荐页嵌入拼图运行态时，通关结算弹层必须挂到页面级 fixed 浮层，不能留在推荐卡片视觉区内的 absolute 覆盖层；推荐页滑动卡片和运行态视口都使用 `overflow: hidden`，半屏内容区会裁剪排行榜、下一关按钮和相似作品卡。
 - 推荐页嵌入拼图运行态时，“下一关”应优先切到相似作品；如果当前推荐候选为空，才回退到同作品下一关，避免匿名推荐流在多关卡作品上持续停留在同一作品内。下一关请求 pending 期间必须保留当前 `PuzzleRuntimeShell` 和棋盘，不得把推荐卡整体切回 `加载中...` 占位态；局部同步状态由拼图运行态自己的 busy 表现承接。后端返回的新关卡属于其它作品时，前端必须同步 `selectedPuzzleDetail`、推荐页 `puzzleGalleryEntries` 缓存和 `activeRecommendEntryKey`，让底部作品信息、分享 / 点赞 / 改造和下一次“下一个”基准都指向新作品。
@@ -304,7 +306,7 @@ RPG / 拼图等运行态存档仍以 `/api/profile/save-archives` 的后端列
 - 难度只决定本局加载的物品种类数量：轻松 3、标准 9、进阶 15、硬核 20。硬核仍保留 21 次消除和 63 件总物品，运行态按 20 种素材循环复用，不要求生成第 21 种素材。
 - 运行态启动前要预加载 `generatedItemAssets[].imageViews[]`、顶层 `generatedBackgroundAsset`、物品挂载 `backgroundAsset` 中的背景、UI spritesheet 和物品 spritesheet；首次生成自动试玩、结果页手动试玩、推荐流和公开详情启动都必须传入提升后的 profile。卡片摘要缺图集字段时，进入运行态前必须补读 work detail。补读后的 profile 也要再次提升 `generatedItemAssets[].backgroundAsset`，确保背景和图集字段传给 `Match3DRuntimeShell`。
 - 背景图作为运行态全屏背景，图内已经保留容器；旧 `containerImage*` 只作为历史透明容器兼容字段。若 `containerImage*` 与 `uiSpritesheetImage*` 同源，运行态不得把 UI spritesheet 当中心容器图叠到棋盘上。
-- 抓大鹅运行态 HUD 需贴近拼图顶部信息条的视觉口径：左上只保留透明返回按钮；右上不再暴露设置入口；顶部关卡名和倒计时直接复用拼图同款的铭牌 + 下挂计时牌结构、同色板和同造型，并在牌面左侧挂上 `media/logo.png` 产品 logo；下方备选栏和道具图标只保留内容与交互边界，不再显示灰白半透底板；中央容器图层视觉可隐藏，但棋盘命中边界仍保留。
+- 抓大鹅运行态 HUD 需贴近拼图顶部信息条的视觉口径：左上只保留透明返回按钮；右上不再暴露设置入口；顶部关卡名和倒计时直接复用拼图同款的铭牌 + 下挂计时牌结构、同色板和同造型，并在牌面左侧挂上 `media/logo-runtime-hud.webp` 产品 logo 小图；下方备选栏和道具图标只保留内容与交互边界，不再显示灰白半透底板；中央容器图层视觉可隐藏，但棋盘命中边界仍保留。
 - generated 私有图换签未完成时，局内物品先隐藏等待，不得短暂显示默认积木；同一批资源在重启 run 时保留已解析签名 URL，只有资源源列表变化或换签失败后才允许进入兜底视觉。
 - `itemSize` 只缩放生成 2D 图片本体：`大`、`中`、`小` 均按相对尺寸缩放，其中 `大` 也比原始图片略小，`中` 和 `小` 进一步缩小；不改变后端下发的布局半径、点击半径或三消规则。
 - 物品进入底部物品栏时按同类型插入：如果物品栏已有同类物品，新物品插到该类型最后一个物品后面，后续物品整体后移；没有同类时追加到当前末尾。达到三件同类时，在飞入物品栏动画结束后，左侧和右侧同类物品向中间合成，三件一起消失，播放合成音效，不展示星星图标，后面的物品再向前补位。该动效只是前端表现层，后端和本地试玩仍负责权威插入、指定点击类型清除与补位后的槽位快照。

miniprogram/app.json

@@ -1,6 +1,7 @@
 {
   "pages": [
     "pages/web-view/index",
+    "pages/share-grid/index",
     "pages/wechat-pay/index",
     "pages/subscribe-message/index"
   ],

miniprogram/pages/share-grid/index.js

@@ -0,0 +1,206 @@
+/* global Page, wx */
+/* eslint-disable no-console */
+
+const {
+  buildShareGridTileFileName,
+  buildShareGridTilePlan,
+  normalizeShareGridQuery,
+} = require('./index.shared');
+
+function downloadImage(imageUrl) {
+  return new Promise((resolve, reject) => {
+    wx.downloadFile({
+      url: imageUrl,
+      success(response) {
+        if (response.statusCode >= 200 && response.statusCode < 300) {
+          resolve(response.tempFilePath);
+          return;
+        }
+        reject(new Error(`封面下载失败：${response.statusCode}`));
+      },
+      fail(error) {
+        reject(new Error(error.errMsg || '封面下载失败'));
+      },
+    });
+  });
+}
+
+function getImageInfo(src) {
+  return new Promise((resolve, reject) => {
+    wx.getImageInfo({
+      src,
+      success: resolve,
+      fail(error) {
+        reject(new Error(error.errMsg || '读取封面失败'));
+      },
+    });
+  });
+}
+
+function getCanvasNode(page) {
+  return new Promise((resolve, reject) => {
+    wx.createSelectorQuery()
+      .in(page)
+      .select('#share-grid-canvas')
+      .fields({ node: true, size: true })
+      .exec((results) => {
+        const canvas = results && results[0] && results[0].node;
+        if (canvas) {
+          resolve(canvas);
+          return;
+        }
+        reject(new Error('切图画布初始化失败'));
+      });
+  });
+}
+
+function canvasToTempFilePath(canvas, width, height) {
+  return new Promise((resolve, reject) => {
+    wx.canvasToTempFilePath({
+      canvas,
+      width,
+      height,
+      destWidth: width,
+      destHeight: height,
+      fileType: 'png',
+      success(response) {
+        resolve(response.tempFilePath);
+      },
+      fail(error) {
+        reject(new Error(error.errMsg || '导出切图失败'));
+      },
+    });
+  });
+}
+
+function saveImageToAlbum(filePath) {
+  return new Promise((resolve, reject) => {
+    wx.saveImageToPhotosAlbum({
+      filePath,
+      success() {
+        resolve();
+      },
+      fail(error) {
+        reject(new Error(error.errMsg || '保存到相册失败'));
+      },
+    });
+  });
+}
+
+function copyTempFileWithName(tempFilePath, fileName) {
+  const fileSystem = wx.getFileSystemManager && wx.getFileSystemManager();
+  const userDataPath = wx.env && wx.env.USER_DATA_PATH;
+  if (!fileSystem || !userDataPath || typeof fileSystem.copyFile !== 'function') {
+    return Promise.resolve(tempFilePath);
+  }
+
+  const targetPath = `${userDataPath}/${fileName}`;
+  return new Promise((resolve) => {
+    fileSystem.copyFile({
+      srcPath: tempFilePath,
+      destPath: targetPath,
+      success() {
+        resolve(targetPath);
+      },
+      fail() {
+        resolve(tempFilePath);
+      },
+    });
+  });
+}
+
+async function saveGridTiles(page, params, localImagePath, imageInfo) {
+  const canvas = await getCanvasNode(page);
+  const context = canvas.getContext('2d');
+  const image = canvas.createImage();
+  await new Promise((resolve, reject) => {
+    image.onload = resolve;
+    image.onerror = () => reject(new Error('封面绘制失败'));
+    image.src = localImagePath;
+  });
+
+  const plan = buildShareGridTilePlan(imageInfo.width, imageInfo.height);
+  for (const tile of plan) {
+    canvas.width = tile.sourceWidth;
+    canvas.height = tile.sourceHeight;
+    context.clearRect(0, 0, tile.sourceWidth, tile.sourceHeight);
+    context.drawImage(
+      image,
+      tile.sourceX,
+      tile.sourceY,
+      tile.sourceWidth,
+      tile.sourceHeight,
+      0,
+      0,
+      tile.sourceWidth,
+      tile.sourceHeight,
+    );
+
+    const tempFilePath = await canvasToTempFilePath(
+      canvas,
+      tile.sourceWidth,
+      tile.sourceHeight,
+    );
+    const namedFilePath = await copyTempFileWithName(
+      tempFilePath,
+      buildShareGridTileFileName(params, tile.index),
+    );
+    await saveImageToAlbum(namedFilePath);
+    page.setData({
+      savedCount: tile.index + 1,
+    });
+  }
+}
+
+Page({
+  data: {
+    errorMessage: '',
+    loading: true,
+    savedCount: 0,
+    title: '九宫切图',
+  },
+
+  async onLoad(query = {}) {
+    const params = normalizeShareGridQuery(query);
+    this._shareGridParams = params;
+    this.setData({
+      errorMessage: '',
+      loading: true,
+      savedCount: 0,
+      title: params.title,
+    });
+
+    if (!params.imageUrl) {
+      this.setData({
+        errorMessage: '缺少封面图。',
+        loading: false,
+      });
+      return;
+    }
+
+    try {
+      const localImagePath = await downloadImage(params.imageUrl);
+      const imageInfo = await getImageInfo(localImagePath);
+      await saveGridTiles(this, params, localImagePath, imageInfo);
+      this.setData({
+        loading: false,
+        savedCount: 9,
+      });
+      wx.showToast({
+        title: '已保存',
+        icon: 'success',
+      });
+    } catch (error) {
+      console.error('[share-grid] save failed', error);
+      this.setData({
+        errorMessage:
+          error && error.message ? error.message : '九宫切图保存失败。',
+        loading: false,
+      });
+    }
+  },
+
+  handleBack() {
+    wx.navigateBack();
+  },
+});

miniprogram/pages/share-grid/index.json

@@ -0,0 +1,3 @@
+{
+  "navigationBarTitleText": "九宫切图"
+}

miniprogram/pages/share-grid/index.shared.js

@@ -0,0 +1,62 @@
+const GRID_SIZE = 3;
+const TILE_COUNT = GRID_SIZE * GRID_SIZE;
+
+function normalizeQueryValue(value) {
+  return String(value || '').trim();
+}
+
+function sanitizeFileNamePart(value) {
+  const normalized = normalizeQueryValue(value)
+    .replace(/[\\/:*?"<>|]/g, '')
+    .replace(/\s+/g, '-')
+    .slice(0, 32);
+  return normalized || 'taonier';
+}
+
+function buildShareGridTileFileName(params, tileIndex) {
+  const safeTitle = sanitizeFileNamePart(params.title || params.publicWorkCode);
+  const safeCode = sanitizeFileNamePart(params.publicWorkCode || 'share');
+  const order = String(tileIndex + 1).padStart(2, '0');
+  return `${safeTitle}-${safeCode}-${order}.png`;
+}
+
+function normalizeShareGridQuery(query) {
+  return {
+    imageUrl: normalizeQueryValue(query && query.imageUrl),
+    title: normalizeQueryValue(query && query.title) || '我的作品',
+    publicWorkCode: normalizeQueryValue(query && query.publicWorkCode),
+  };
+}
+
+function buildShareGridTilePlan(imageWidth, imageHeight) {
+  const tileWidth = Math.floor(imageWidth / GRID_SIZE);
+  const tileHeight = Math.floor(imageHeight / GRID_SIZE);
+  const plan = [];
+
+  for (let row = 0; row < GRID_SIZE; row += 1) {
+    for (let col = 0; col < GRID_SIZE; col += 1) {
+      const index = row * GRID_SIZE + col;
+      const sourceX = col * tileWidth;
+      const sourceY = row * tileHeight;
+      plan.push({
+        index,
+        row,
+        col,
+        sourceX,
+        sourceY,
+        sourceWidth: col === GRID_SIZE - 1 ? imageWidth - sourceX : tileWidth,
+        sourceHeight: row === GRID_SIZE - 1 ? imageHeight - sourceY : tileHeight,
+      });
+    }
+  }
+
+  return plan;
+}
+
+module.exports = {
+  GRID_SIZE,
+  TILE_COUNT,
+  buildShareGridTileFileName,
+  buildShareGridTilePlan,
+  normalizeShareGridQuery,
+};

miniprogram/pages/share-grid/index.test.js

@@ -0,0 +1,67 @@
+import { describe, expect, test } from 'vitest';
+
+import shareGridBridge from './index.shared.js';
+
+const {
+  buildShareGridTileFileName,
+  buildShareGridTilePlan,
+  normalizeShareGridQuery,
+} = shareGridBridge;
+
+describe('share-grid mini program bridge', () => {
+  test('normalizes query values and keeps a fallback title', () => {
+    expect(
+      normalizeShareGridQuery({
+        imageUrl: ' https://web.test/cover.png ',
+        publicWorkCode: ' PZ-0001 ',
+      }),
+    ).toEqual({
+      imageUrl: 'https://web.test/cover.png',
+      title: '我的作品',
+      publicWorkCode: 'PZ-0001',
+    });
+  });
+
+  test('names tiles by title, public code and left-to-right order', () => {
+    const params = {
+      title: '星港:拼图',
+      publicWorkCode: 'PZ-0001',
+    };
+
+    expect(buildShareGridTileFileName(params, 0)).toBe(
+      '星港拼图-PZ-0001-01.png',
+    );
+    expect(buildShareGridTileFileName(params, 8)).toBe(
+      '星港拼图-PZ-0001-09.png',
+    );
+  });
+
+  test('builds a 3x3 crop plan in reading order', () => {
+    const plan = buildShareGridTilePlan(900, 600);
+
+    expect(plan).toHaveLength(9);
+    expect(plan[0]).toMatchObject({
+      index: 0,
+      row: 0,
+      col: 0,
+      sourceX: 0,
+      sourceY: 0,
+      sourceWidth: 300,
+      sourceHeight: 200,
+    });
+    expect(plan[4]).toMatchObject({
+      index: 4,
+      row: 1,
+      col: 1,
+      sourceX: 300,
+      sourceY: 200,
+    });
+    expect(plan[8]).toMatchObject({
+      index: 8,
+      row: 2,
+      col: 2,
+      sourceX: 600,
+      sourceY: 400,
+    });
+  });
+});

miniprogram/pages/share-grid/index.wxml

@@ -0,0 +1,20 @@
+<view class="share-grid-page">
+  <view class="share-grid-card">
+    <view class="share-grid-title">{{title}}</view>
+    <view wx:if="{{loading}}" class="share-grid-text">
+      正在保存 {{savedCount}}/9
+    </view>
+    <view wx:elif="{{errorMessage}}" class="share-grid-text share-grid-text--danger">
+      {{errorMessage}}
+    </view>
+    <view wx:else class="share-grid-text">已保存 9/9</view>
+    <button class="share-grid-button" bindtap="handleBack">
+      返回
+    </button>
+  </view>
+  <canvas
+    id="share-grid-canvas"
+    type="2d"
+    class="share-grid-canvas"
+  ></canvas>
+</view>

miniprogram/pages/share-grid/index.wxss

@@ -0,0 +1,60 @@
+page {
+  background: #fffdf9;
+}
+
+.share-grid-page {
+  min-height: 100vh;
+  box-sizing: border-box;
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  padding: 48rpx;
+  background: #fffdf9;
+}
+
+.share-grid-card {
+  width: 100%;
+  max-width: 560rpx;
+  box-sizing: border-box;
+  border: 1rpx solid rgba(127, 85, 57, 0.18);
+  border-radius: 16rpx;
+  background: rgba(255, 255, 255, 0.92);
+  padding: 36rpx;
+  box-shadow: 0 24rpx 68rpx rgba(127, 85, 57, 0.12);
+}
+
+.share-grid-title {
+  color: #332820;
+  font-size: 34rpx;
+  font-weight: 700;
+  line-height: 1.35;
+}
+
+.share-grid-text {
+  margin-top: 18rpx;
+  color: rgba(51, 40, 32, 0.68);
+  font-size: 26rpx;
+  line-height: 1.55;
+}
+
+.share-grid-text--danger {
+  color: #b84a3d;
+}
+
+.share-grid-button {
+  margin-top: 28rpx;
+  width: 100%;
+  border-radius: 8rpx;
+  background: #7f5539;
+  color: #fffdf9;
+  font-size: 28rpx;
+  line-height: 2.6;
+}
+
+.share-grid-canvas {
+  position: fixed;
+  left: -9999px;
+  top: -9999px;
+  width: 1px;
+  height: 1px;
+}

miniprogram/pages/web-view/index.js

@@ -10,6 +10,13 @@ const {
   WEB_VIEW_ENTRY_URL,
   WEB_VIEW_SOURCE_QUERY,
 } = require('../../config');
+const {
+  appendHashParams,
+  buildWebViewSharePath,
+  buildWebViewShareTimelineQuery,
+  resolveShareTargetFromWebViewMessage,
+  resolveWebViewUrlFromRuntimeConfig,
+} = require('./index.shared');
 
 const MINI_PROGRAM_CLIENT_TYPE = 'mini_program';
 const MINI_PROGRAM_CLIENT_RUNTIME = 'wechat_mini_program';
@@ -19,7 +26,6 @@ const AUTH_RESULT_STORAGE_KEY = 'genarrative:mini-program-auth-result';
 const AUTH_ACTION_LOGIN = 'login';
 const PAY_RESULT_RECHECK_DELAY_MS = 120;
 const WEB_VIEW_SHARE_TITLE = '陶泥儿';
-const WEB_VIEW_SHARE_PATH = '/pages/web-view/index';
 
 function showWebViewShareMenu() {
   if (typeof wx.showShareMenu !== 'function') {
@@ -32,17 +38,25 @@ function showWebViewShareMenu() {
   });
 }
 
-function buildWebViewShareAppMessage() {
+function resolveNativeShareQuery(page) {
+  return (
+    (page && page._currentShareTarget) ||
+    (page && page._lastLaunchQuery) ||
+    {}
+  );
+}
+
+function buildWebViewShareAppMessage(query = {}) {
   return {
     title: WEB_VIEW_SHARE_TITLE,
-    path: WEB_VIEW_SHARE_PATH,
+    path: buildWebViewSharePath(query),
   };
 }
 
-function buildWebViewShareTimeline() {
+function buildWebViewShareTimeline(query = {}) {
   return {
     title: WEB_VIEW_SHARE_TITLE,
-    query: '',
+    query: buildWebViewShareTimelineQuery(query),
   };
 }
 
@@ -59,50 +73,6 @@ function isConfiguredApiBaseUrl(value) {
   return /^https:\/\/[^/]+/i.test(String(value || '').trim());
 }
 
-function appendQuery(url, query) {
-  const pairs = Object.keys(query)
-    .filter((key) => query[key])
-    .map(
-      (key) =>
-        `${encodeURIComponent(key)}=${encodeURIComponent(String(query[key]))}`,
-    );
-
-  if (pairs.length === 0) {
-    return url;
-  }
-
-  return `${url}${url.includes('?') ? '&' : '?'}${pairs.join('&')}`;
-}
-
-function appendHashParams(url, params) {
-  const nextKeys = new Set(Object.keys(params).filter((key) => params[key]));
-  const pairs = Object.keys(params)
-    .filter((key) => params[key])
-    .map(
-      (key) =>
-        `${encodeURIComponent(key)}=${encodeURIComponent(String(params[key]))}`,
-    );
-  if (pairs.length === 0) {
-    return url;
-  }
-
-  const hashIndex = url.indexOf('#');
-  const baseUrl = hashIndex >= 0 ? url.slice(0, hashIndex) : url;
-  const rawHash = hashIndex >= 0 ? url.slice(hashIndex + 1) : '';
-  const keptHashParts = rawHash.split('&').filter((part) => {
-    if (!part) {
-      return false;
-    }
-    const [rawKey = ''] = part.split('=');
-    try {
-      return !nextKeys.has(decodeURIComponent(rawKey));
-    } catch (_error) {
-      return !nextKeys.has(rawKey);
-    }
-  });
-  return `${baseUrl}#${keptHashParts.concat(pairs).join('&')}`;
-}
-
 function parseBooleanQueryFlag(value) {
   return value === true || value === '1' || value === 'true' || value === 'yes';
 }
@@ -233,22 +203,16 @@ function shouldReturnToPreviousPage(query) {
   return String((query && query.returnTo) || '').trim() === 'previous';
 }
 
-function resolveWebViewUrl(authResult) {
+function resolveWebViewUrl(authResult, launchQuery = {}) {
   const runtimeConfig = resolveMiniProgramRuntimeConfig();
   const entryUrl = String(runtimeConfig.webViewEntryUrl || '').trim();
   if (!isConfiguredEntryUrl(entryUrl)) {
     return '';
   }
 
-  const sourcedUrl = appendQuery(entryUrl, runtimeConfig.sourceQuery);
-  if (!authResult || !authResult.token) {
-    return sourcedUrl;
-  }
-
-  return appendHashParams(sourcedUrl, {
-    auth_provider: 'wechat',
-    auth_token: authResult.token,
-    auth_binding_status: authResult.bindingStatus,
+  return resolveWebViewUrlFromRuntimeConfig(authResult, launchQuery, {
+    ...runtimeConfig,
+    webViewEntryUrl: String(runtimeConfig.webViewEntryUrl || '').trim(),
   });
 }
 
@@ -467,7 +431,7 @@ Page({
         loading: false,
         phoneBindingRequired: false,
         returnToPreviousPage: false,
-        webViewUrl: resolveWebViewUrl(null),
+        webViewUrl: resolveWebViewUrl(null, query),
       });
       return;
     }
@@ -572,7 +536,7 @@ Page({
         nicknameRequired: false,
         phoneBindingRequired: false,
         returnToPreviousPage,
-        webViewUrl: resolveWebViewUrl(authResult),
+        webViewUrl: resolveWebViewUrl(authResult, this._lastLaunchQuery || {}),
       });
     } catch (error) {
       this.setData({
@@ -600,7 +564,7 @@ Page({
         loading: false,
         nicknameRequired: false,
         phoneBindingRequired: false,
-        webViewUrl: resolveWebViewUrl(authResult),
+        webViewUrl: resolveWebViewUrl(authResult, this._lastLaunchQuery || {}),
       });
     }
 
@@ -674,7 +638,10 @@ Page({
         loading: false,
         nicknameRequired: false,
         phoneBindingRequired: false,
-        webViewUrl: resolveWebViewUrl(nextAuthResult),
+        webViewUrl: resolveWebViewUrl(
+          nextAuthResult,
+          this._lastLaunchQuery || {},
+        ),
       });
     } catch (error) {
       this.setData({
@@ -712,15 +679,19 @@ Page({
   },
 
   handleWebViewMessage(event) {
+    const shareTarget = resolveShareTargetFromWebViewMessage(event.detail);
+    if (shareTarget) {
+      this._currentShareTarget = shareTarget;
+    }
     // 中文注释：支付和订阅消息都由独立 native 页面承接，web-view 消息只保留调试输出。
     console.info('[web-view] message', event.detail);
   },
 
   onShareAppMessage() {
-    return buildWebViewShareAppMessage();
+    return buildWebViewShareAppMessage(resolveNativeShareQuery(this));
   },
 
   onShareTimeline() {
-    return buildWebViewShareTimeline();
+    return buildWebViewShareTimeline(resolveNativeShareQuery(this));
   },
 });

miniprogram/pages/web-view/index.shared.js

@@ -0,0 +1,188 @@
+const ALLOWED_TARGET_PATHS = new Set(['/works/detail']);
+const SHARE_TARGET_MESSAGE_TYPE = 'genarrative:share-target';
+const WEB_VIEW_SHARE_PATH = '/pages/web-view/index';
+
+function trimTrailingSlash(value) {
+  return String(value || '').trim().replace(/\/+$/u, '');
+}
+
+function appendQuery(url, query) {
+  const rawUrl = String(url || '');
+  const pairs = Object.keys(query)
+    .filter((key) => query[key])
+    .map(
+      (key) =>
+        `${encodeURIComponent(key)}=${encodeURIComponent(String(query[key]))}`,
+    );
+
+  if (pairs.length === 0) {
+    return rawUrl;
+  }
+
+  const hashIndex = rawUrl.indexOf('#');
+  const baseUrl = hashIndex >= 0 ? rawUrl.slice(0, hashIndex) : rawUrl;
+  const hash = hashIndex >= 0 ? rawUrl.slice(hashIndex) : '';
+  return `${baseUrl}${baseUrl.includes('?') ? '&' : '?'}${pairs.join('&')}${hash}`;
+}
+
+function appendHashParams(url, params) {
+  const nextKeys = new Set(Object.keys(params).filter((key) => params[key]));
+  const pairs = Object.keys(params)
+    .filter((key) => params[key])
+    .map(
+      (key) =>
+        `${encodeURIComponent(key)}=${encodeURIComponent(String(params[key]))}`,
+    );
+  if (pairs.length === 0) {
+    return url;
+  }
+
+  const hashIndex = url.indexOf('#');
+  const baseUrl = hashIndex >= 0 ? url.slice(0, hashIndex) : url;
+  const rawHash = hashIndex >= 0 ? url.slice(hashIndex + 1) : '';
+  const keptHashParts = rawHash.split('&').filter((part) => {
+    if (!part) {
+      return false;
+    }
+    const [rawKey = ''] = part.split('=');
+    try {
+      return !nextKeys.has(decodeURIComponent(rawKey));
+    } catch (_error) {
+      return !nextKeys.has(rawKey);
+    }
+  });
+  return `${baseUrl}#${keptHashParts.concat(pairs).join('&')}`;
+}
+
+function normalizeTargetPath(value) {
+  const trimmed = String(value || '').trim();
+  if (!trimmed.startsWith('/')) {
+    return '';
+  }
+
+  const normalized = trimmed.replace(/\/+$/u, '') || '/';
+  return ALLOWED_TARGET_PATHS.has(normalized) ? normalized : '';
+}
+
+function resolveLaunchTargetQuery(query) {
+  const targetPath = normalizeTargetPath(query && query.targetPath);
+  const work = String((query && query.work) || '').trim();
+  if (!targetPath || !work) {
+    return {};
+  }
+
+  return {
+    targetPath,
+    work,
+  };
+}
+
+function buildWebViewSharePath(query = {}, basePath = WEB_VIEW_SHARE_PATH) {
+  const launchTarget = resolveLaunchTargetQuery(query);
+  if (!launchTarget.targetPath) {
+    return basePath;
+  }
+
+  return appendQuery(basePath, {
+    targetPath: launchTarget.targetPath,
+    work: launchTarget.work,
+  });
+}
+
+function buildWebViewShareTimelineQuery(query = {}) {
+  const launchTarget = resolveLaunchTargetQuery(query);
+  if (!launchTarget.targetPath) {
+    return '';
+  }
+
+  return new URLSearchParams({
+    targetPath: launchTarget.targetPath,
+    work: launchTarget.work,
+  }).toString();
+}
+
+function normalizeShareTargetMessageData(value) {
+  const message = value && value.data ? value.data : value;
+  if (!message || message.type !== SHARE_TARGET_MESSAGE_TYPE) {
+    return null;
+  }
+
+  const payload = message.payload || {};
+  const launchTarget = resolveLaunchTargetQuery(payload);
+  if (!launchTarget.targetPath) {
+    return null;
+  }
+
+  return {
+    ...launchTarget,
+    title: String(payload.title || '').trim(),
+  };
+}
+
+function resolveShareTargetFromWebViewMessage(detail) {
+  const dataList = detail && Array.isArray(detail.data) ? detail.data : [];
+  for (let index = dataList.length - 1; index >= 0; index -= 1) {
+    const target = normalizeShareTargetMessageData(dataList[index]);
+    if (target) {
+      return target;
+    }
+  }
+
+  return normalizeShareTargetMessageData(detail);
+}
+
+function appendLaunchTargetToEntryUrl(entryUrl, query) {
+  const launchTarget = resolveLaunchTargetQuery(query);
+  if (!launchTarget.targetPath) {
+    return entryUrl;
+  }
+
+  const rawEntryUrl = String(entryUrl || '').trim();
+  const hashIndex = rawEntryUrl.indexOf('#');
+  const entryWithoutHash =
+    hashIndex >= 0 ? rawEntryUrl.slice(0, hashIndex) : rawEntryUrl;
+  const hash = hashIndex >= 0 ? rawEntryUrl.slice(hashIndex) : '';
+  const queryIndex = entryWithoutHash.indexOf('?');
+  const entryBase =
+    queryIndex >= 0 ? entryWithoutHash.slice(0, queryIndex) : entryWithoutHash;
+  const entrySearch =
+    queryIndex >= 0 ? entryWithoutHash.slice(queryIndex) : '';
+  const targetUrl = `${trimTrailingSlash(entryBase)}${launchTarget.targetPath}${entrySearch}${hash}`;
+
+  return appendQuery(targetUrl, {
+    work: launchTarget.work,
+  });
+}
+
+function resolveWebViewUrlFromRuntimeConfig(
+  authResult,
+  launchQuery = {},
+  runtimeConfig = {},
+) {
+  const entryUrl = appendLaunchTargetToEntryUrl(
+    String(runtimeConfig.webViewEntryUrl || '').trim(),
+    launchQuery,
+  );
+  const sourcedUrl = appendQuery(entryUrl, runtimeConfig.sourceQuery || {});
+  if (!authResult || !authResult.token) {
+    return sourcedUrl;
+  }
+
+  return appendHashParams(sourcedUrl, {
+    auth_provider: 'wechat',
+    auth_token: authResult.token,
+    auth_binding_status: authResult.bindingStatus,
+  });
+}
+
+module.exports = {
+  appendHashParams,
+  appendLaunchTargetToEntryUrl,
+  appendQuery,
+  buildWebViewSharePath,
+  buildWebViewShareTimelineQuery,
+  normalizeTargetPath,
+  resolveShareTargetFromWebViewMessage,
+  resolveLaunchTargetQuery,
+  resolveWebViewUrlFromRuntimeConfig,
+};

miniprogram/pages/web-view/index.test.js

@@ -0,0 +1,110 @@
+import { describe, expect, test } from 'vitest';
+
+import webViewBridge from './index.shared.js';
+
+const {
+  appendLaunchTargetToEntryUrl,
+  buildWebViewSharePath,
+  buildWebViewShareTimelineQuery,
+  resolveShareTargetFromWebViewMessage,
+  resolveWebViewUrlFromRuntimeConfig,
+} = webViewBridge;
+
+const runtimeConfig = {
+  sourceQuery: {
+    clientType: 'mini_program',
+    clientRuntime: 'wechat_mini_program',
+  },
+  webViewEntryUrl: 'https://www.genarrative.world',
+};
+
+describe('mini program web-view launch target', () => {
+  test('opens the H5 public work detail when launch query carries work params', () => {
+    expect(
+      appendLaunchTargetToEntryUrl('https://www.genarrative.world?foo=bar', {
+        targetPath: '/works/detail',
+        work: 'BB-12345678',
+      }),
+    ).toBe(
+      'https://www.genarrative.world/works/detail?foo=bar&work=BB-12345678',
+    );
+
+    const webViewUrl = resolveWebViewUrlFromRuntimeConfig(
+      null,
+      {
+        targetPath: '/works/detail',
+        work: 'BB-12345678',
+      },
+      runtimeConfig,
+    );
+    const url = new URL(webViewUrl);
+    expect(url.pathname).toBe('/works/detail');
+    expect(url.searchParams.get('work')).toBe('BB-12345678');
+    expect(url.searchParams.get('clientRuntime')).toBe('wechat_mini_program');
+  });
+
+  test('ignores unsupported launch target paths', () => {
+    const webViewUrl = resolveWebViewUrlFromRuntimeConfig(
+      null,
+      {
+        targetPath: '/admin',
+        work: 'BB-12345678',
+      },
+      runtimeConfig,
+    );
+    const url = new URL(webViewUrl);
+    expect(url.pathname).toBe('/');
+    expect(url.searchParams.get('work')).toBeNull();
+  });
+
+  test('keeps public work params in native mini program share paths', () => {
+    const sharePath = buildWebViewSharePath({
+      targetPath: '/works/detail',
+      work: 'BB-12345678',
+    });
+    const url = new URL(sharePath, 'https://mini.test');
+
+    expect(url.pathname).toBe('/pages/web-view/index');
+    expect(url.searchParams.get('targetPath')).toBe('/works/detail');
+    expect(url.searchParams.get('work')).toBe('BB-12345678');
+    expect(
+      buildWebViewShareTimelineQuery({
+        targetPath: '/works/detail',
+        work: 'BB-12345678',
+      }),
+    ).toBe('targetPath=%2Fworks%2Fdetail&work=BB-12345678');
+  });
+
+  test('reads the latest H5 recommended work share target from web-view messages', () => {
+    expect(
+      resolveShareTargetFromWebViewMessage({
+        data: [
+          {
+            data: {
+              type: 'genarrative:share-target',
+              payload: {
+                targetPath: '/works/detail',
+                work: 'PZ-0001',
+                title: '旧作品',
+              },
+            },
+          },
+          {
+            data: {
+              type: 'genarrative:share-target',
+              payload: {
+                targetPath: '/works/detail',
+                work: 'BB-12345678',
+                title: '汪汪声浪',
+              },
+            },
+          },
+        ],
+      }),
+    ).toEqual({
+      targetPath: '/works/detail',
+      work: 'BB-12345678',
+      title: '汪汪声浪',
+    });
+  });
+});

package.json

@@ -9,6 +9,7 @@
     "dev:api-server": "node scripts/dev.mjs api-server",
     "dev:web": "node scripts/dev.mjs web",
     "dev:admin-web": "node scripts/dev.mjs admin-web",
+    "server-manager:panel": "cargo run -p server-manager-panel --manifest-path server-rs/Cargo.toml",
     "dev:spacetime:logs": "node scripts/run-bash-script.mjs scripts/spacetime-logs-local.sh",
     "otel:debug": "node scripts/run-otelcol.mjs debug",
     "otel:rider": "node scripts/run-otelcol.mjs rider",
@@ -54,6 +55,7 @@
     "container:ps": "node scripts/container-compose.mjs ps",
     "container:config": "node scripts/container-compose.mjs config",
     "container:k6": "node scripts/container-compose.mjs k6",
+    "container:worker-smoke": "node scripts/container-worker-smoke.mjs",
     "check": "npm run lint && npm run test && npm run build && npm run check:content",
     "check:data": "node scripts/run-tsx.cjs scripts/validate-content.ts",
     "check:overrides": "node scripts/run-tsx.cjs scripts/validate-overrides.ts",

scripts/check-production-ops-guardrails.mjs

@@ -23,6 +23,46 @@ const checks = [
     includes: 'genarrative-health-patrol.timer',
     reason: 'Server-Provision 必须安装并启用健康巡检 timer。',
   },
+  {
+    file: 'scripts/jenkins-server-provision.sh',
+    includes: 'genarrative-external-generation-controller.service',
+    reason: 'Server-Provision 必须安装并启用外部生成 worker controller。',
+  },
+  {
+    file: 'scripts/jenkins-server-provision.sh',
+    includes: 'genarrative-external-generation-worker@1.service',
+    reason: 'Server-Provision 必须启用外部生成保底 worker 实例。',
+  },
+  {
+    file: 'scripts/deploy/production-api-deploy.sh',
+    includes: 'ensure_default_worker_service',
+    reason: 'API Deploy 必须在缺少 worker 实例时补启动默认外部生成 worker。',
+  },
+  {
+    file: 'scripts/deploy/production-api-deploy.sh',
+    includes: 'wait_for_worker_services',
+    reason: 'API Deploy 必须等待外部生成 worker 实例 active。',
+  },
+  {
+    file: 'scripts/deploy/production-api-deploy.sh',
+    includes: 'wait_for_worker_controller_service',
+    reason: 'API Deploy 必须重启并验活外部生成 worker controller。',
+  },
+  {
+    file: 'deploy/systemd/genarrative-external-generation-worker@.service',
+    includes: 'GENARRATIVE_PROCESS_ROLE=external-generation-worker',
+    reason: '外部生成 worker 模板必须作为独立 worker 进程角色运行。',
+  },
+  {
+    file: 'deploy/systemd/genarrative-external-generation-controller.service',
+    includes: 'GENARRATIVE_PROCESS_ROLE=external-generation-controller',
+    reason: '外部生成 worker controller 必须作为独立进程角色运行。',
+  },
+  {
+    file: 'scripts/ops/production-health-patrol.mjs',
+    includes: 'checkActiveWorkerInstances',
+    reason: '生产健康巡检必须检查至少一个外部生成 worker 实例 active。',
+  },
   {
     file: 'scripts/build-production-release.sh',
     includes: 'production-health-patrol.mjs',

scripts/check-spacetime-schema-guard.mjs

@@ -475,14 +475,14 @@ function loadBaseSources(baseRef) {
 
 function getChangedFiles(baseRef) {
   const diffOutput = tryGit(['diff', '--name-only', '-z', baseRef, '--']) ?? '';
-  const untrackedOutput =
+  const untrackedModuleOutput =
     tryGit(['ls-files', '--others', '--exclude-standard', '-z', moduleSrcRoot]) ?? '';
   const untrackedBindingsOutput =
     tryGit(['ls-files', '--others', '--exclude-standard', '-z', bindingsRoot]) ?? '';
   return new Set(
     [
       ...diffOutput.split(/\u0000/u),
-      ...untrackedOutput.split(/\u0000/u),
+      ...untrackedModuleOutput.split(/\u0000/u),
       ...untrackedBindingsOutput.split(/\u0000/u),
     ]
       .map(normalizePath)

scripts/container-worker-smoke.mjs

@@ -0,0 +1,839 @@
+import {spawn} from 'node:child_process';
+import {
+  chmodSync,
+  copyFileSync,
+  existsSync,
+  mkdirSync,
+  readFileSync,
+  writeFileSync,
+} from 'node:fs';
+import net from 'node:net';
+import path from 'node:path';
+
+const [, , rawCommand = 'help', ...rawArgs] = process.argv;
+
+const projectRoot = process.cwd();
+const composeFile = path.join('deploy', 'container', 'docker-compose.loadtest.yml');
+const smokeDir = path.join('deploy', 'container', 'worker-smoke');
+const envPath = path.join(smokeDir, 'api-server.env');
+const statePath = path.join(smokeDir, 'state.json');
+const localImageDir = path.join(smokeDir, 'image');
+const localImageDockerfilePath = path.join(localImageDir, 'Dockerfile.local');
+const localImageBinaryPath = path.join(localImageDir, 'api-server');
+const localCargoTargetDir = path.join('server-rs', 'target-worker-smoke');
+const localSpacetimeImageDir = path.join(smokeDir, 'spacetimedb-image');
+const localSpacetimeDockerfilePath = path.join(localSpacetimeImageDir, 'Dockerfile.local');
+const localSpacetimeBinaryPath = path.join(localSpacetimeImageDir, 'spacetime');
+const localSpacetimeStandalonePath = path.join(
+  localSpacetimeImageDir,
+  'spacetimedb-standalone',
+);
+const projectName = process.env.GENARRATIVE_WORKER_SMOKE_PROJECT || 'genarrative-worker-smoke';
+const defaultDatabase =
+  process.env.GENARRATIVE_WORKER_SMOKE_DATABASE || 'genarrative-worker-smoke';
+
+const command = rawCommand.trim();
+const supportedCommands = new Set([
+  'help',
+  'init',
+  'build',
+  'up-spacetime',
+  'publish',
+  'up',
+  'enqueue',
+  'status',
+  'api-update',
+  'scale',
+  'logs',
+  'ps',
+  'down',
+  'smoke',
+]);
+
+if (!supportedCommands.has(command)) {
+  printHelp(true);
+  process.exit(1);
+}
+
+try {
+  await main();
+} catch (error) {
+  console.error(`[worker-smoke] ${error.message}`);
+  process.exit(1);
+}
+
+async function main() {
+  switch (command) {
+    case 'help':
+      printHelp(false);
+      return;
+    case 'init':
+      await ensureStateAndEnv({force: rawArgs.includes('--force')});
+      return;
+    case 'build':
+      await ensureStateAndEnv();
+      await buildRuntimeImages();
+      return;
+    case 'up-spacetime':
+      await ensureStateAndEnv();
+      await ensureSpacetimeImage();
+      await dockerCompose(['up', '-d', 'spacetimedb', 'otelcol']);
+      await waitForSpacetime();
+      return;
+    case 'publish':
+      await ensureStateAndEnv();
+      await publishModule();
+      return;
+    case 'up':
+      await ensureStateAndEnv();
+      await upRuntime();
+      await waitForApi();
+      return;
+    case 'enqueue':
+      await ensureStateAndEnv();
+      await enqueueSmokeJob();
+      return;
+    case 'status':
+      await ensureStateAndEnv();
+      await printQueueStatus();
+      return;
+    case 'api-update':
+      await ensureStateAndEnv();
+      await apiOnlyUpdate({build: rawArgs.includes('--build')});
+      return;
+    case 'scale':
+      await ensureStateAndEnv();
+      await scaleWorkers(rawArgs[0] ?? '1');
+      return;
+    case 'logs':
+      await ensureStateAndEnv();
+      await dockerCompose(['logs', ...rawArgs]);
+      return;
+    case 'ps':
+      await ensureStateAndEnv();
+      await dockerCompose(['ps', ...rawArgs]);
+      return;
+    case 'down':
+      await ensureStateAndEnv({create: false});
+      await dockerCompose(['down', ...rawArgs]);
+      return;
+    case 'smoke':
+      await runSmoke();
+      return;
+    default:
+      throw new Error(`未知命令: ${command}`);
+  }
+}
+
+async function runSmoke() {
+  if (rawArgs.includes('--force')) {
+    await ensureStateAndEnv();
+    await dockerComposeCapture(['down', '-v'], {allowFailure: true});
+  }
+  const state = await ensureStateAndEnv({force: rawArgs.includes('--force')});
+  await assertSavedPortsAvailableForNewProject(state);
+  console.log(
+    `[worker-smoke] 使用隔离环境 project=${projectName} database=${state.database}`,
+  );
+  await buildRuntimeImages();
+  await ensureSpacetimeImage();
+  await dockerCompose(['up', '-d', 'spacetimedb', 'otelcol']);
+  await waitForSpacetime();
+  await publishModule();
+  await upRuntime();
+  await waitForApi();
+  await assertWorkersRunning();
+
+  const beforeWorkerIds = await getContainerIds('external-generation-worker');
+  console.log(`[worker-smoke] worker 容器: ${beforeWorkerIds.join(', ')}`);
+
+  const firstJobId = await enqueueSmokeJob({label: 'before-api-update'});
+  await waitForJobConsumed(firstJobId);
+
+  await apiOnlyUpdate({build: false});
+  const afterWorkerIds = await getContainerIds('external-generation-worker');
+  if (beforeWorkerIds.join('\n') !== afterWorkerIds.join('\n')) {
+    throw new Error(
+      `api-update 后 worker 容器发生变化: before=${beforeWorkerIds.join(',')} after=${afterWorkerIds.join(',')}`,
+    );
+  }
+  console.log('[worker-smoke] api-only 更新未重建 worker 容器。');
+
+  const secondJobId = await enqueueSmokeJob({label: 'after-api-update'});
+  await waitForJobConsumed(secondJobId);
+  await printQueueStatus();
+  console.log('[worker-smoke] smoke 通过：worker 独立消费队列，API-only 更新未停止 worker。');
+}
+
+async function buildRuntimeImages() {
+  const imageMode = resolveImageMode();
+  if (imageMode === 'local-binary') {
+    await buildLocalBinaryRuntimeImages();
+    return;
+  }
+  await dockerCompose(['build', 'api-server', 'external-generation-worker']);
+}
+
+function resolveImageMode() {
+  if (rawArgs.includes('--local-binary')) {
+    return 'local-binary';
+  }
+  const envMode = process.env.GENARRATIVE_WORKER_SMOKE_IMAGE_MODE;
+  if (!envMode || envMode === 'dockerfile') {
+    return 'dockerfile';
+  }
+  if (envMode === 'local-binary') {
+    return 'local-binary';
+  }
+  throw new Error(
+    `GENARRATIVE_WORKER_SMOKE_IMAGE_MODE 仅支持 dockerfile 或 local-binary: ${envMode}`,
+  );
+}
+
+async function buildLocalBinaryRuntimeImages() {
+  const profile =
+    rawArgs.includes('--release') ||
+    process.env.GENARRATIVE_WORKER_SMOKE_CARGO_PROFILE === 'release'
+      ? 'release'
+      : 'debug';
+  const buildArgs = ['build', '-p', 'api-server', '--manifest-path', 'server-rs/Cargo.toml'];
+  if (profile === 'release') {
+    buildArgs.push('--release');
+  }
+  const cargoImage = resolveLocalBinaryCargoImage();
+  const cargoHome = resolveLocalBinaryCargoHome();
+  mkdirSync(cargoHome, {recursive: true});
+
+  console.log(
+    `[worker-smoke] 使用 ${cargoImage} 复用本机 Cargo 缓存构建 ${profile} api-server 二进制。`,
+  );
+  await run('docker', [
+    'run',
+    '--rm',
+    '-u',
+    currentUserSpec(),
+    '-v',
+    `${projectRoot}:/workspace`,
+    '-v',
+    `${cargoHome}:/cargo-home`,
+    '-w',
+    '/workspace',
+    '-e',
+    'HOME=/cargo-home',
+    '-e',
+    'CARGO_HOME=/cargo-home',
+    '-e',
+    `CARGO_TARGET_DIR=/workspace/${toContainerPath(localCargoTargetDir)}`,
+    cargoImage,
+    'cargo',
+    '--config',
+    'build.rustc-wrapper=""',
+    '--config',
+    'target.x86_64-unknown-linux-gnu.linker="cc"',
+    '--config',
+    'target.x86_64-unknown-linux-gnu.rustflags=[]',
+    ...buildArgs,
+  ]);
+
+  const sourceBinaryPath = path.join(localCargoTargetDir, profile, 'api-server');
+  if (!existsSync(sourceBinaryPath)) {
+    throw new Error(`未找到 worker smoke api-server 二进制: ${sourceBinaryPath}`);
+  }
+
+  mkdirSync(localImageDir, {recursive: true});
+  copyFileSync(sourceBinaryPath, localImageBinaryPath);
+  chmodSync(localImageBinaryPath, 0o755);
+
+  const baseImage = await resolveLocalBinaryBaseImage();
+  writeFileSync(localImageDockerfilePath, buildLocalBinaryDockerfile(baseImage), 'utf8');
+
+  await run('docker', [
+    'build',
+    '-f',
+    localImageDockerfilePath,
+    '-t',
+    `${projectName}-api-server`,
+    '-t',
+    `${projectName}-external-generation-worker`,
+    localImageDir,
+  ]);
+}
+
+function resolveLocalBinaryCargoImage() {
+  return process.env.GENARRATIVE_WORKER_SMOKE_CARGO_IMAGE || 'rust:1.93-bookworm';
+}
+
+function resolveLocalBinaryCargoHome() {
+  if (process.env.GENARRATIVE_WORKER_SMOKE_CARGO_HOME) {
+    return path.resolve(process.env.GENARRATIVE_WORKER_SMOKE_CARGO_HOME);
+  }
+  if (!process.env.HOME) {
+    throw new Error('未找到 HOME，无法挂载本机 Cargo 缓存。');
+  }
+  return path.join(process.env.HOME, '.cargo');
+}
+
+function currentUserSpec() {
+  if (typeof process.getuid === 'function' && typeof process.getgid === 'function') {
+    return `${process.getuid()}:${process.getgid()}`;
+  }
+  return '0:0';
+}
+
+async function ensureSpacetimeImage() {
+  if (process.env.GENARRATIVE_WORKER_SMOKE_SPACETIME_IMAGE_MODE === 'official') {
+    return;
+  }
+  const imageName = localSpacetimeImageName();
+  const existingImage = await runCapture('docker', ['image', 'inspect', imageName], {
+    allowFailure: true,
+    quiet: true,
+  });
+  if (existingImage.code === 0 && !rawArgs.includes('--force')) {
+    return;
+  }
+
+  const spacetimePath = await resolveSpacetimeBinaryPath();
+  if (!spacetimePath) {
+    throw new Error('未找到本机 spacetime CLI，无法构建隔离 SpacetimeDB 镜像。');
+  }
+
+  mkdirSync(localSpacetimeImageDir, {recursive: true});
+  copyFileSync(spacetimePath, localSpacetimeBinaryPath);
+  chmodSync(localSpacetimeBinaryPath, 0o755);
+  const standalonePath = path.join(path.dirname(spacetimePath), 'spacetimedb-standalone');
+  if (!existsSync(standalonePath)) {
+    throw new Error(`未找到本机 spacetimedb-standalone: ${standalonePath}`);
+  }
+  copyFileSync(standalonePath, localSpacetimeStandalonePath);
+  chmodSync(localSpacetimeStandalonePath, 0o755);
+  writeFileSync(localSpacetimeDockerfilePath, buildLocalSpacetimeDockerfile(), 'utf8');
+
+  console.log(`[worker-smoke] 使用本机 spacetime CLI 构建隔离镜像: ${imageName}`);
+  await run('docker', [
+    'build',
+    '-f',
+    localSpacetimeDockerfilePath,
+    '-t',
+    imageName,
+    localSpacetimeImageDir,
+  ]);
+}
+
+function buildLocalSpacetimeDockerfile() {
+  return `FROM debian:bookworm-slim
+WORKDIR /var/lib/spacetimedb
+RUN apt-get update && \\
+    apt-get install -y --no-install-recommends ca-certificates libstdc++6 zlib1g && \\
+    rm -rf /var/lib/apt/lists/*
+COPY spacetime /usr/local/bin/spacetime
+COPY spacetimedb-standalone /usr/local/bin/spacetimedb-standalone
+RUN chmod 0755 /usr/local/bin/spacetime /usr/local/bin/spacetimedb-standalone
+ENTRYPOINT ["spacetime"]
+`;
+}
+
+async function resolveSpacetimeBinaryPath() {
+  if (process.env.GENARRATIVE_WORKER_SMOKE_SPACETIME_BIN) {
+    return process.env.GENARRATIVE_WORKER_SMOKE_SPACETIME_BIN;
+  }
+  const versionResult = await runCapture('spacetime', ['--version'], {quiet: true});
+  const pathMatch = versionResult.stdout.match(/^spacetime Path:\s*(.+)$/mu);
+  if (pathMatch?.[1]) {
+    return pathMatch[1].trim();
+  }
+  const whichResult = await runCapture('which', ['spacetime'], {quiet: true});
+  return whichResult.stdout.trim();
+}
+
+async function resolveLocalBinaryBaseImage() {
+  if (process.env.GENARRATIVE_WORKER_SMOKE_LOCAL_BASE_IMAGE) {
+    return process.env.GENARRATIVE_WORKER_SMOKE_LOCAL_BASE_IMAGE;
+  }
+  return 'debian:bookworm-slim';
+}
+
+function buildLocalBinaryDockerfile(baseImage) {
+  return `FROM ${baseImage}
+WORKDIR /srv/genarrative
+RUN apt-get update && \\
+    apt-get install -y --no-install-recommends ca-certificates curl libssl3 zlib1g libzstd1 && \\
+    rm -rf /var/lib/apt/lists/* && \\
+    (id -u genarrative >/dev/null 2>&1 || useradd --system --create-home --home-dir /srv/genarrative --shell /usr/sbin/nologin genarrative)
+COPY api-server /usr/local/bin/api-server
+RUN chmod 0755 /usr/local/bin/api-server && \\
+    mkdir -p /var/lib/genarrative/auth /var/lib/genarrative/tracking-outbox && \\
+    chown -R genarrative:genarrative /srv/genarrative /var/lib/genarrative
+USER genarrative
+EXPOSE 8082
+ENV GENARRATIVE_ENV=container \\
+    GENARRATIVE_API_HOST=0.0.0.0 \\
+    GENARRATIVE_API_PORT=8082 \\
+    GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox
+CMD ["api-server"]
+`;
+}
+
+function toContainerPath(localPath) {
+  return localPath.split(path.sep).join('/');
+}
+
+async function upRuntime() {
+  const services = ['api-server', 'external-generation-worker'];
+  if (rawArgs.includes('--with-nginx')) {
+    services.push('nginx');
+  }
+  await dockerCompose(['up', '-d', ...services]);
+}
+
+async function ensureStateAndEnv(options = {}) {
+  const {force = false, create = true} = options;
+  if (!create && !existsSync(statePath)) {
+    return defaultState();
+  }
+  mkdirSync(smokeDir, {recursive: true});
+
+  if (!existsSync(statePath) || force) {
+    const state = {
+      database: defaultDatabase,
+      spacetimePort: await findAvailablePort(
+        Number(process.env.GENARRATIVE_WORKER_SMOKE_SPACETIME_PORT || 19101),
+      ),
+      httpPort: await findAvailablePort(
+        Number(process.env.GENARRATIVE_WORKER_SMOKE_HTTP_PORT || 19080),
+      ),
+      otlpGrpcPort: await findAvailablePort(
+        Number(process.env.GENARRATIVE_WORKER_SMOKE_OTLP_GRPC_PORT || 15317),
+      ),
+      otlpHttpPort: await findAvailablePort(
+        Number(process.env.GENARRATIVE_WORKER_SMOKE_OTLP_HTTP_PORT || 15318),
+      ),
+      createdAt: new Date().toISOString(),
+    };
+    writeFileSync(statePath, `${JSON.stringify(state, null, 2)}\n`, 'utf8');
+  }
+
+  const state = readState();
+  if (!existsSync(envPath) || force) {
+    writeFileSync(envPath, buildSmokeEnv(state), 'utf8');
+  }
+  console.log(`[worker-smoke] env=${envPath}`);
+  console.log(`[worker-smoke] state=${statePath}`);
+  console.log(`[worker-smoke] SpacetimeDB=http://127.0.0.1:${state.spacetimePort}`);
+  console.log(`[worker-smoke] Nginx=http://127.0.0.1:${state.httpPort}`);
+  return state;
+}
+
+function buildSmokeEnv(state) {
+  return `# 本文件由 scripts/container-worker-smoke.mjs 生成，仅用于本机隔离 worker smoke。
+# 不要在这里写真实生产密钥；目录 deploy/container/worker-smoke/ 已被 gitignore。
+GENARRATIVE_ENV=container-worker-smoke
+GENARRATIVE_API_HOST=0.0.0.0
+GENARRATIVE_API_PORT=8082
+GENARRATIVE_API_LOG=info,tower_http=info
+GENARRATIVE_API_LISTEN_BACKLOG=256
+GENARRATIVE_API_WORKER_THREADS=2
+GENARRATIVE_PROCESS_ROLE=api
+GENARRATIVE_EXTERNAL_GENERATION_MODE=queue
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=1
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=500
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=60
+GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=64
+GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=32
+GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=16
+GENARRATIVE_API_ADMIN_MAX_CONCURRENT_REQUESTS=8
+GENARRATIVE_TRACKING_OUTBOX_ENABLED=false
+GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox
+
+GENARRATIVE_OTEL_ENABLED=false
+OTEL_SERVICE_NAME=genarrative-worker-smoke-api
+OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4318
+OTEL_RESOURCE_ATTRIBUTES=deployment.environment=worker-smoke,service.namespace=genarrative
+
+GENARRATIVE_INTERNAL_API_SECRET=worker-smoke-internal-secret
+GENARRATIVE_JWT_ISSUER=genarrative-worker-smoke
+GENARRATIVE_JWT_SECRET=worker-smoke-jwt-secret
+AUTH_REFRESH_COOKIE_SECURE=false
+GENARRATIVE_DEV_PASSWORD_ENTRY_AUTO_REGISTER_ENABLED=true
+
+GENARRATIVE_SPACETIME_SERVER_URL=http://spacetimedb:3101
+GENARRATIVE_SPACETIME_DATABASE=${state.database}
+GENARRATIVE_SPACETIME_TOKEN=
+GENARRATIVE_SPACETIME_POOL_SIZE=2
+GENARRATIVE_SPACETIME_PROCEDURE_TIMEOUT_SECONDS=15
+
+GENARRATIVE_LLM_PROVIDER=openai-compatible
+GENARRATIVE_LLM_BASE_URL=
+GENARRATIVE_LLM_API_KEY=
+GENARRATIVE_LLM_MODEL=
+VECTOR_ENGINE_BASE_URL=
+VECTOR_ENGINE_API_KEY=
+ALIYUN_OSS_BUCKET=
+ALIYUN_OSS_ENDPOINT=oss-cn-shanghai.aliyuncs.com
+ALIYUN_OSS_ACCESS_KEY_ID=
+ALIYUN_OSS_ACCESS_KEY_SECRET=
+WECHAT_MINIPROGRAM_MESSAGE_TOKEN=
+WECHAT_MINIPROGRAM_MESSAGE_ENCODING_AES_KEY=
+`;
+}
+
+function defaultState() {
+  return {
+    database: defaultDatabase,
+    spacetimePort: 19101,
+    httpPort: 19080,
+    otlpGrpcPort: 15317,
+    otlpHttpPort: 15318,
+  };
+}
+
+function readState() {
+  if (!existsSync(statePath)) {
+    return defaultState();
+  }
+  return JSON.parse(readFileSync(statePath, 'utf8'));
+}
+
+async function findAvailablePort(startPort) {
+  for (let port = startPort; port < startPort + 100; port += 1) {
+    if (await isPortAvailable(port)) {
+      return port;
+    }
+  }
+  throw new Error(`未找到可用端口: ${startPort}-${startPort + 99}`);
+}
+
+function isPortAvailable(port) {
+  return new Promise((resolve) => {
+    const server = net.createServer();
+    server.once('error', () => resolve(false));
+    server.once('listening', () => {
+      server.close(() => resolve(true));
+    });
+    server.listen(port, '127.0.0.1');
+  });
+}
+
+async function publishModule() {
+  const state = readState();
+  const serverUrl = spacetimeServerUrl(state);
+  const publishArgs = [
+    'publish',
+    state.database,
+    '--server',
+    serverUrl,
+    '--module-path',
+    'server-rs/crates/spacetime-module',
+    '--delete-data=on-conflict',
+    '--anonymous',
+    '--yes=all',
+    '--no-config',
+  ];
+  const buildOptions = process.env.GENARRATIVE_WORKER_SMOKE_STDB_BUILD_OPTIONS;
+  if (buildOptions) {
+    publishArgs.push('--build-options', buildOptions);
+  }
+  await run('spacetime', publishArgs);
+}
+
+async function enqueueSmokeJob(options = {}) {
+  if (!rawArgs.includes('--no-worker-check')) {
+    await assertWorkersRunning();
+  }
+  const state = readState();
+  const nowMicros = Date.now() * 1000;
+  const suffix = `${Date.now()}-${Math.random().toString(16).slice(2, 8)}`;
+  const jobId = `extgen-smoke-${suffix}`;
+  const label = options.label || rawArgs[0] || 'manual';
+  const input = {
+    job_id: jobId,
+    dedupe_key: `worker-smoke:${label}:${suffix}`,
+    job_kind: 'worker_smoke_unsupported',
+    owner_user_id: 'worker-smoke-user',
+    source_module: 'worker-smoke',
+    source_entity_id: `worker-smoke-entity-${suffix}`,
+    request_label: `worker-smoke ${label}`,
+    request_payload_json: JSON.stringify({label, suffix}),
+    max_attempts: 1,
+    available_at_micros: nowMicros,
+    created_at_micros: nowMicros,
+  };
+
+  await run('spacetime', [
+    'call',
+    '--server',
+    spacetimeServerUrl(state),
+    '--anonymous',
+    '--yes',
+    '--no-config',
+    state.database,
+    'enqueue_external_generation_job_and_return',
+    JSON.stringify(input),
+  ]);
+  console.log(`[worker-smoke] 已入队测试 job: ${jobId}`);
+  return jobId;
+}
+
+async function printQueueStatus() {
+  console.log('[worker-smoke] external_generation_job 是 private table，status 显示最近 worker 日志：');
+  await printServiceLogs('external-generation-worker', 120);
+}
+
+async function waitForJobConsumed(jobId) {
+  const deadline = Date.now() + 60_000;
+  let lastOutput = '';
+  while (Date.now() < deadline) {
+    const result = await dockerComposeCapture(
+      ['logs', '--no-color', 'external-generation-worker'],
+      {allowFailure: true, quiet: true},
+    );
+    lastOutput = `${result.stdout}\n${result.stderr}`;
+    if (lastOutput.includes(jobId) && lastOutput.includes('暂不支持的任务类型')) {
+      console.log(`[worker-smoke] job ${jobId} 已被 worker 领取并执行到 unsupported 分支。`);
+      return;
+    }
+    await sleep(1000);
+  }
+  await printServiceLogs('external-generation-worker', 120);
+  throw new Error(`等待 worker 消费 job ${jobId} 超时，最后输出:\n${lastOutput}`);
+}
+
+async function assertSavedPortsAvailableForNewProject(state) {
+  const existingContainers = await getProjectContainerIds();
+  if (existingContainers.length > 0) {
+    return;
+  }
+  const ports = [
+    ['SpacetimeDB', state.spacetimePort],
+    ['Nginx', state.httpPort],
+    ['OTLP gRPC', state.otlpGrpcPort],
+    ['OTLP HTTP', state.otlpHttpPort],
+  ];
+  for (const [label, port] of ports) {
+    if (!(await isPortAvailable(port))) {
+      throw new Error(
+        `${label} 端口 ${port} 已被占用；可执行 npm run container:worker-smoke -- smoke --force 重新分配隔离端口。`,
+      );
+    }
+  }
+}
+
+async function getProjectContainerIds() {
+  const result = await dockerComposeCapture(['ps', '-q'], {
+    allowFailure: true,
+    quiet: true,
+  });
+  if (result.code !== 0) {
+    return [];
+  }
+  return result.stdout
+    .split(/\r?\n/u)
+    .map((line) => line.trim())
+    .filter(Boolean);
+}
+
+async function assertWorkersRunning() {
+  const result = await dockerComposeCapture(
+    ['ps', '--status', 'running', '-q', 'external-generation-worker'],
+    {allowFailure: true, quiet: true},
+  );
+  const workerIds = result.stdout
+    .split(/\r?\n/u)
+    .map((line) => line.trim())
+    .filter(Boolean);
+  if (result.code === 0 && workerIds.length > 0) {
+    return;
+  }
+  await printServiceLogs('external-generation-worker', 80);
+  throw new Error('external-generation-worker 未处于 running 状态，已输出最近日志。');
+}
+
+async function printServiceLogs(service, tail = 80) {
+  await dockerComposeCapture(['logs', '--tail', String(tail), service], {
+    allowFailure: true,
+  });
+}
+
+async function waitForSpacetime() {
+  const state = readState();
+  const url = `${spacetimeServerUrl(state)}/v1/ping`;
+  await waitForHttp(url, 'SpacetimeDB');
+}
+
+async function waitForApi() {
+  const deadline = Date.now() + 120_000;
+  while (Date.now() < deadline) {
+    const result = await dockerComposeCapture(
+      ['exec', '-T', 'api-server', 'curl', '-fsS', 'http://127.0.0.1:8082/healthz'],
+      {allowFailure: true, quiet: true},
+    );
+    if (result.code === 0) {
+      console.log('[worker-smoke] api-server 已就绪: api-server:8082/healthz');
+      return;
+    }
+    await sleep(2000);
+  }
+  throw new Error('api-server 等待超时: api-server:8082/healthz');
+}
+
+async function waitForHttp(url, label) {
+  const deadline = Date.now() + 120_000;
+  while (Date.now() < deadline) {
+    const result = await runCapture('curl', ['-fsS', '--max-time', '3', url], {
+      allowFailure: true,
+    });
+    if (result.code === 0) {
+      console.log(`[worker-smoke] ${label} 已就绪: ${url}`);
+      return;
+    }
+    await sleep(2000);
+  }
+  throw new Error(`${label} 等待超时: ${url}`);
+}
+
+async function apiOnlyUpdate({build}) {
+  const beforeWorkerIds = await getContainerIds('external-generation-worker');
+  const args = ['up', '-d', '--no-deps', '--force-recreate'];
+  if (build) {
+    args.push('--build');
+  }
+  args.push('api-server');
+  await dockerCompose(args);
+  await waitForApi();
+  const afterWorkerIds = await getContainerIds('external-generation-worker');
+  if (beforeWorkerIds.join('\n') !== afterWorkerIds.join('\n')) {
+    throw new Error('API-only 更新不应重建 external-generation-worker 容器');
+  }
+  console.log('[worker-smoke] API-only 更新完成，worker 容器保持不变。');
+}
+
+async function scaleWorkers(rawCount) {
+  const count = Number.parseInt(rawCount, 10);
+  if (!Number.isInteger(count) || count < 0 || count > 16) {
+    throw new Error(`worker 数量必须是 0-16 的整数: ${rawCount}`);
+  }
+  await dockerCompose([
+    'up',
+    '-d',
+    '--scale',
+    `external-generation-worker=${count}`,
+    'external-generation-worker',
+  ]);
+}
+
+async function getContainerIds(service) {
+  const result = await dockerComposeCapture(['ps', '-q', service]);
+  return result.stdout
+    .split(/\r?\n/u)
+    .map((line) => line.trim())
+    .filter(Boolean)
+    .sort();
+}
+
+async function dockerCompose(args) {
+  await run('docker', composeArgs(args), {env: composeEnv()});
+}
+
+async function dockerComposeCapture(args, options = {}) {
+  return runCapture('docker', composeArgs(args), {
+    env: composeEnv(),
+    ...options,
+  });
+}
+
+function composeArgs(args) {
+  return ['compose', '-p', projectName, '-f', composeFile, ...args];
+}
+
+function composeEnv() {
+  const state = readState();
+  return {
+    ...process.env,
+    GENARRATIVE_CONTAINER_API_ENV_FILE: './worker-smoke/api-server.env',
+    GENARRATIVE_CONTAINER_SPACETIME_IMAGE:
+      process.env.GENARRATIVE_CONTAINER_SPACETIME_IMAGE || localSpacetimeImageName(),
+    GENARRATIVE_CONTAINER_SPACETIME_PORT: String(state.spacetimePort),
+    GENARRATIVE_CONTAINER_HTTP_PORT: String(state.httpPort),
+    GENARRATIVE_CONTAINER_OTLP_GRPC_PORT: String(state.otlpGrpcPort),
+    GENARRATIVE_CONTAINER_OTLP_HTTP_PORT: String(state.otlpHttpPort),
+  };
+}
+
+function localSpacetimeImageName() {
+  return `${projectName}-spacetimedb:2.4.1`;
+}
+
+function spacetimeServerUrl(state) {
+  return `http://127.0.0.1:${state.spacetimePort}`;
+}
+
+function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+async function run(commandName, args, options = {}) {
+  const result = await runCapture(commandName, args, options);
+  if (result.code !== 0 && !options.allowFailure) {
+    throw new Error(`${commandName} ${args.join(' ')} 失败，exit=${result.code}`);
+  }
+  return result;
+}
+
+function runCapture(commandName, args, options = {}) {
+  return new Promise((resolve, reject) => {
+    const child = spawn(commandName, args, {
+      cwd: projectRoot,
+      env: options.env ?? process.env,
+      shell: false,
+    });
+    let stdout = '';
+    let stderr = '';
+    child.stdout?.on('data', (chunk) => {
+      const text = chunk.toString();
+      stdout += text;
+      if (!options.quiet) {
+        process.stdout.write(text);
+      }
+    });
+    child.stderr?.on('data', (chunk) => {
+      const text = chunk.toString();
+      stderr += text;
+      if (!options.quiet) {
+        process.stderr.write(text);
+      }
+    });
+    child.on('error', reject);
+    child.on('exit', (code, signal) => {
+      if (signal) {
+        reject(new Error(`${commandName} 被信号终止: ${signal}`));
+        return;
+      }
+      resolve({code: code ?? 0, stdout, stderr});
+    });
+  });
+}
+
+function printHelp(isError) {
+  const output = isError ? console.error : console.log;
+  output(`Usage: npm run container:worker-smoke -- <command>
+
+Commands:
+  init [--force]      生成隔离 env 与端口 state
+  build [--local-binary] [--release]
+                      构建 api-server / worker 镜像；--local-binary 让容器内 Cargo 复用本机缓存
+  up-spacetime        启动隔离 SpacetimeDB 与 otelcol
+  publish             向隔离 SpacetimeDB 发布 spacetime-module
+  up [--with-nginx]   启动 api-server / worker；需要 Nginx 时显式加 --with-nginx
+  enqueue [label] [--no-worker-check]
+                      写入一个 unsupported 测试 job，验证 worker claim/fail
+  status              查看最近 worker 日志；external_generation_job 是 private table
+  api-update [--build] 仅重建/重启 api-server，不触碰 worker
+  scale <n>           调整 external-generation-worker 实例数
+  ps                  查看隔离 compose 状态
+  logs [service]      查看隔离 compose 日志
+  down [-v]           停止隔离 compose，-v 会清理数据卷
+  smoke [--force] [--local-binary] [--release]
+                      一键执行 build -> publish -> up -> enqueue -> api-update -> enqueue
+`);
+}

scripts/deploy/production-api-deploy.sh

@@ -5,10 +5,11 @@ set -euo pipefail
 usage() {
   cat <<'EOF'
 用法:
-  ./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101]
+  ./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--worker-controller-service genarrative-external-generation-controller.service] [--no-worker-controller] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101]
 
 说明:
   进入维护模式，校验并发布 api-server 单文件，更新 current 链接，重启 systemd 服务并执行 readiness 检查。
+  默认同时重启外部生成 worker controller 和已加载的 worker 实例；未启用 worker 单元时会自动跳过。
   若传入 --database，会在重启前把 GENARRATIVE_SPACETIME_DATABASE 写入 api-server 环境文件，避免服务继续读取旧库。
   失败时保留维护模式。
 EOF
@@ -223,12 +224,144 @@ ensure_runtime_env_and_dirs() {
   fi
 }
 
+list_worker_services() {
+  local pattern="$1"
+
+  if [[ -z "${pattern}" ]]; then
+    return 0
+  fi
+
+  systemctl list-units --all --plain --no-legend "${pattern}" 2>/dev/null | awk '{print $1}' | sort -u
+}
+
+ensure_default_worker_service() {
+  local pattern="$1"
+  local default_service="genarrative-external-generation-worker@1.service"
+  local template_service="genarrative-external-generation-worker@.service"
+  local services=()
+
+  if [[ -z "${pattern}" ]]; then
+    return 0
+  fi
+
+  if [[ "${pattern}" != "genarrative-external-generation-worker@*.service" ]]; then
+    return 0
+  fi
+
+  if ! systemctl cat "${template_service}" >/dev/null 2>&1; then
+    echo "[production-api-deploy] 缺少外部生成 worker systemd 模板: ${template_service}" >&2
+    return 1
+  fi
+
+  mapfile -t services < <(list_worker_services "${pattern}")
+  if [[ "${#services[@]}" -gt 0 ]]; then
+    return 0
+  fi
+
+  echo "[production-api-deploy] 未发现外部生成 worker 实例，启用并启动默认实例: ${default_service}"
+  systemctl enable --now "${default_service}"
+}
+
+restart_worker_services() {
+  local pattern="$1"
+  local services=()
+
+  if [[ -z "${pattern}" ]]; then
+    echo "[production-api-deploy] 跳过外部生成 worker 重启。"
+    return 0
+  fi
+
+  ensure_default_worker_service "${pattern}"
+  mapfile -t services < <(list_worker_services "${pattern}")
+  if [[ "${#services[@]}" -eq 0 ]]; then
+    echo "[production-api-deploy] 未发现已加载的外部生成 worker 单元: ${pattern}" >&2
+    return 1
+  fi
+
+  echo "[production-api-deploy] 重启外部生成 worker: ${services[*]}"
+  systemctl restart "${services[@]}"
+}
+
+wait_for_worker_services() {
+  local pattern="$1"
+  local services=()
+  local all_active
+
+  if [[ -z "${pattern}" ]]; then
+    return 0
+  fi
+
+  mapfile -t services < <(list_worker_services "${pattern}")
+  if [[ "${#services[@]}" -eq 0 ]]; then
+    echo "[production-api-deploy] 外部生成 worker 单元不存在，发布失败: ${pattern}" >&2
+    return 1
+  fi
+
+  echo "[production-api-deploy] 等待外部生成 worker active: ${services[*]}"
+  for _ in {1..30}; do
+    all_active=1
+    for service in "${services[@]}"; do
+      if ! systemctl is-active --quiet "${service}"; then
+        all_active=0
+        break
+      fi
+    done
+    if [[ "${all_active}" -eq 1 ]]; then
+      return 0
+    fi
+    sleep 2
+  done
+
+  systemctl --no-pager --full status "${services[@]}" || true
+  echo "[production-api-deploy] 外部生成 worker 未在超时时间内进入 active，发布失败。" >&2
+  return 1
+}
+
+ensure_worker_controller_service() {
+  local service="$1"
+
+  if [[ -z "${service}" ]]; then
+    return 0
+  fi
+
+  if ! systemctl cat "${service}" >/dev/null 2>&1; then
+    echo "[production-api-deploy] 缺少外部生成 worker controller systemd 单元: ${service}" >&2
+    return 1
+  fi
+
+  echo "[production-api-deploy] 启用并重启外部生成 worker controller: ${service}"
+  systemctl enable "${service}"
+  systemctl restart "${service}"
+}
+
+wait_for_worker_controller_service() {
+  local service="$1"
+
+  if [[ -z "${service}" ]]; then
+    return 0
+  fi
+
+  echo "[production-api-deploy] 等待外部生成 worker controller active: ${service}"
+  for _ in {1..30}; do
+    if systemctl is-active --quiet "${service}"; then
+      return 0
+    fi
+    sleep 2
+  done
+
+  systemctl --no-pager --full status "${service}" || true
+  echo "[production-api-deploy] 外部生成 worker controller 未在超时时间内进入 active，发布失败。" >&2
+  return 1
+}
+
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
 SOURCE_DIR=""
 VERSION=""
 RELEASE_ROOT="/opt/genarrative/releases"
 CURRENT_LINK="/opt/genarrative/current"
 SERVICE_NAME="genarrative-api.service"
+WORKER_SERVICE_PATTERN="genarrative-external-generation-worker@*.service"
+WORKER_CONTROLLER_SERVICE="genarrative-external-generation-controller.service"
 HEALTH_URL="http://127.0.0.1:8082/readyz"
 API_ENV_FILE="/etc/genarrative/api-server.env"
 DATABASE=""
@@ -261,6 +394,22 @@ while [[ $# -gt 0 ]]; do
       SERVICE_NAME="${2:?缺少 --service 的值}"
       shift 2
       ;;
+    --worker-service-pattern)
+      WORKER_SERVICE_PATTERN="${2:?缺少 --worker-service-pattern 的值}"
+      shift 2
+      ;;
+    --no-worker-services)
+      WORKER_SERVICE_PATTERN=""
+      shift
+      ;;
+    --worker-controller-service)
+      WORKER_CONTROLLER_SERVICE="${2:?缺少 --worker-controller-service 的值}"
+      shift 2
+      ;;
+    --no-worker-controller)
+      WORKER_CONTROLLER_SERVICE=""
+      shift
+      ;;
     --health-url)
       HEALTH_URL="${2:?缺少 --health-url 的值}"
       shift 2
@@ -383,6 +532,10 @@ ln -sfn "${RELEASE_DIR}" "${CURRENT_LINK}"
 
 echo "[production-api-deploy] 重启服务: ${SERVICE_NAME}"
 systemctl restart "${SERVICE_NAME}"
+restart_worker_services "${WORKER_SERVICE_PATTERN}"
+wait_for_worker_services "${WORKER_SERVICE_PATTERN}"
+ensure_worker_controller_service "${WORKER_CONTROLLER_SERVICE}"
+wait_for_worker_controller_service "${WORKER_CONTROLLER_SERVICE}"
 
 echo "[production-api-deploy] 等待 readiness: ${HEALTH_URL}"
 for _ in {1..30}; do

scripts/dev.mjs

@@ -37,6 +37,7 @@ const manifestPath = resolve(serverRsDir, 'Cargo.toml');
 const modulePath = resolve(serverRsDir, 'crates/spacetime-module');
 const viteCliPath = resolve(repoRoot, 'scripts/vite-cli.mjs');
 const adminWebDir = resolve(repoRoot, 'apps/admin-web');
+const LOCAL_DEV_RUSTC_WRAPPER_BYPASS = process.platform === 'win32' ? 'rustc' : '/usr/bin/env';
 
 const SERVICE_NAMES = ['spacetime', 'api-server', 'web', 'admin-web'];
 const SERVICE_ALIASES = new Map([
@@ -399,6 +400,39 @@ function requireCommand(command) {
   }
 }
 
+function isSccacheRustcWrapper(value) {
+  const wrapper = String(value ?? '').trim();
+  if (!wrapper) {
+    return false;
+  }
+
+  const command = wrapper.split(/[\\/]/).pop()?.toLowerCase();
+  return command === 'sccache' || command === 'sccache.exe';
+}
+
+function buildLocalRustProcessEnv(env, options = {}) {
+  const mergedEnv = {...env};
+  const wrappers = [
+    String(mergedEnv.RUSTC_WRAPPER ?? '').trim(),
+    String(mergedEnv.CARGO_BUILD_RUSTC_WRAPPER ?? '').trim(),
+  ].filter(Boolean);
+  const customWrapper = wrappers.find((wrapper) => !isSccacheRustcWrapper(wrapper));
+  if (customWrapper) {
+    mergedEnv.RUSTC_WRAPPER = customWrapper;
+    mergedEnv.CARGO_BUILD_RUSTC_WRAPPER = customWrapper;
+    return mergedEnv;
+  }
+
+  mergedEnv.RUSTC_WRAPPER = LOCAL_DEV_RUSTC_WRAPPER_BYPASS;
+  mergedEnv.CARGO_BUILD_RUSTC_WRAPPER = LOCAL_DEV_RUSTC_WRAPPER_BYPASS;
+  if (options.log !== false) {
+    console.warn(
+      '[dev:rust] 本地 dev 构建绕过项目 sccache wrapper，避免缓存进程异常阻断启动。',
+    );
+  }
+  return mergedEnv;
+}
+
 function readWorkspaceSpacetimeVersion() {
   const manifestText = readFileSync(manifestPath, 'utf8');
   const match = /^spacetimedb\s*=\s*(?:"([^"]+)"|\{[^}]*version\s*=\s*"([^"]+)")/mu.exec(
@@ -776,7 +810,7 @@ class DevRunner {
     this.writeDevStackState();
   }
 
-  async prepareLinuxPortRange(command) {
+  async prepareLinuxPortRange() {
     if (process.platform !== 'linux') {
       return;
     }
@@ -1228,7 +1262,7 @@ class DevRunner {
   }
 
   async publishSpacetimeModule() {
-    const env = {...this.baseEnv};
+    const env = buildLocalRustProcessEnv(this.baseEnv);
     this.prepareMigrationBootstrapSecret(env);
 
     const args = buildSpacetimePublishArgs({
@@ -1291,7 +1325,7 @@ class DevRunner {
     await this.ensureApiServerSpacetimeToken();
 
     const mergedEnv = buildApiServerProcessEnv({
-      baseEnv: this.baseEnv,
+      baseEnv: buildLocalRustProcessEnv(this.baseEnv),
       options: this.options,
       state: this.state,
     });
@@ -2124,19 +2158,20 @@ function buildApiServerProcessEnv({baseEnv, options, state}) {
 }
 
 export {
-  DevRunner,
   assertReusableSpacetimeProcessVersionMatchesWorkspace,
   assertSpacetimeToolVersionMatchesWorkspace,
   buildApiServerProcessEnv,
   buildDevStackSnapshot,
+  buildLocalRustProcessEnv,
   buildSpacetimePublishArgs,
   createDevServerSpawnOptions,
   createWatchConfigs,
-  isSpacetimePublishPermissionError,
+  DevRunner,
   isDirectModuleExecution,
+  isSpacetimePublishPermissionError,
   normalizeCargoVersionRequirement,
-  parseSpacetimeToolVersion,
   parseArgs,
+  parseSpacetimeToolVersion,
   resolveDevStackStatePath,
   shouldAcceptWatchEvent,
 };

scripts/dev.test.ts

@@ -5,19 +5,20 @@ import {join} from 'node:path';
 import {afterEach, describe, expect, test, vi} from 'vitest';
 
 import {
-  DevRunner,
   assertReusableSpacetimeProcessVersionMatchesWorkspace,
   assertSpacetimeToolVersionMatchesWorkspace,
   buildApiServerProcessEnv,
   buildDevStackSnapshot,
+  buildLocalRustProcessEnv,
   buildSpacetimePublishArgs,
   createDevServerSpawnOptions,
   createWatchConfigs,
+  DevRunner,
   isDirectModuleExecution,
   isSpacetimePublishPermissionError,
   normalizeCargoVersionRequirement,
-  parseSpacetimeToolVersion,
   parseArgs,
+  parseSpacetimeToolVersion,
   resolveDevStackStatePath,
   shouldAcceptWatchEvent,
 } from './dev.mjs';
@@ -185,6 +186,35 @@ describe('dev scheduler api-server env', () => {
   });
 });
 
+describe('dev scheduler Rust build env', () => {
+  test('local dev Rust env bypasses project sccache wrapper', () => {
+    const env = buildLocalRustProcessEnv(
+      {
+        RUSTC_WRAPPER: '/usr/bin/sccache',
+        CARGO_BUILD_RUSTC_WRAPPER: 'sccache',
+      },
+      {log: false},
+    );
+
+    expect(env.RUSTC_WRAPPER).not.toBe('/usr/bin/sccache');
+    expect(env.RUSTC_WRAPPER).not.toBe('sccache');
+    expect(env.CARGO_BUILD_RUSTC_WRAPPER).toBe(env.RUSTC_WRAPPER);
+  });
+
+  test('local dev Rust env keeps healthy custom wrapper untouched', () => {
+    const env = buildLocalRustProcessEnv(
+      {
+        RUSTC_WRAPPER: 'custom-wrapper',
+        CARGO_BUILD_RUSTC_WRAPPER: 'sccache',
+      },
+      {log: false},
+    );
+
+    expect(env.RUSTC_WRAPPER).toBe('custom-wrapper');
+    expect(env.CARGO_BUILD_RUSTC_WRAPPER).toBe('custom-wrapper');
+  });
+});
+
 describe('dev scheduler stack state file', () => {
   test('状态文件路径固定在根目录 .app/dev-stack.json', () => {
     expect(resolveDevStackStatePath('C:\\repo\\Genarrative')).toBe(

scripts/jenkins-server-provision.sh

@@ -4,6 +4,8 @@ set -euo pipefail
 PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}"
 SPACETIME_BIN_SOURCE="${SPACETIME_BIN_SOURCE:-${PROVISION_TOOLS_DIR}/spacetime/spacetime}"
 OTELCOL_BIN_SOURCE="${OTELCOL_BIN_SOURCE:-${PROVISION_TOOLS_DIR}/otelcol-contrib}"
+WORKER_ENV_FILE="${WORKER_ENV_FILE:-/etc/genarrative/external-generation-worker.env}"
+CONTROLLER_ENV_FILE="${CONTROLLER_ENV_FILE:-/etc/genarrative/external-generation-controller.env}"
 GENARRATIVE_OPENSSL_VERSION="${GENARRATIVE_OPENSSL_VERSION:-3.2.0}"
 GENARRATIVE_OPENSSL_PREFIX="${GENARRATIVE_OPENSSL_PREFIX:-/opt/genarrative/openssl-3.2.0}"
 GENARRATIVE_OPENSSL_SOURCE_URL="${GENARRATIVE_OPENSSL_SOURCE_URL:-https://github.com/openssl/openssl/releases/download/openssl-${GENARRATIVE_OPENSSL_VERSION}/openssl-${GENARRATIVE_OPENSSL_VERSION}.tar.gz}"
@@ -458,6 +460,7 @@ ensure_spacetime_owner_client_token() {
     echo "[server-provision] 已生成 SpacetimeDB client identity 并写入 GENARRATIVE_SPACETIME_TOKEN: ${identity_preview}..."
   fi
 
+  # 中文注释：这里是 provision 内部为 spacetimedb 运行用户隔离 CLI 登录态的受控用法，不作为人工 spacetime 命令示例。
   if ! login_output="$(runuser -u spacetimedb -- "${cli_path}" --root-dir "${SPACETIME_ROOT}" login --token "${token}" 2>&1)"; then
     echo "[server-provision] 使用 GENARRATIVE_SPACETIME_TOKEN 登录 SpacetimeDB CLI 失败。" >&2
     printf "%s\\n" "${login_output}" | sed -E "s/[A-Za-z0-9_.=-]{24,}/[REDACTED]/g" >&2
@@ -536,6 +539,14 @@ render_api_env_example() {
     deploy/env/api-server.env.example
 }
 
+render_external_generation_worker_env_example() {
+  cat deploy/env/external-generation-worker.env.example
+}
+
+render_external_generation_controller_env_example() {
+  cat deploy/env/external-generation-controller.env.example
+}
+
 render_otelcol_service() {
   cat deploy/systemd/otelcol-contrib.service
 }
@@ -722,6 +733,30 @@ render_api_service() {
     deploy/systemd/genarrative-api.service
 }
 
+render_external_generation_worker_service() {
+  local current_escaped api_env_escaped worker_env_escaped
+  current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
+  api_env_escaped="$(escape_sed_replacement "${API_ENV_FILE}")"
+  worker_env_escaped="$(escape_sed_replacement "${WORKER_ENV_FILE}")"
+  sed \
+    -e "s|/opt/genarrative/current|${current_escaped}|g" \
+    -e "s|/etc/genarrative/api-server.env|${api_env_escaped}|g" \
+    -e "s|/etc/genarrative/external-generation-worker.env|${worker_env_escaped}|g" \
+    deploy/systemd/genarrative-external-generation-worker@.service
+}
+
+render_external_generation_controller_service() {
+  local current_escaped api_env_escaped controller_env_escaped
+  current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
+  api_env_escaped="$(escape_sed_replacement "${API_ENV_FILE}")"
+  controller_env_escaped="$(escape_sed_replacement "${CONTROLLER_ENV_FILE}")"
+  sed \
+    -e "s|/opt/genarrative/current|${current_escaped}|g" \
+    -e "s|/etc/genarrative/api-server.env|${api_env_escaped}|g" \
+    -e "s|/etc/genarrative/external-generation-controller.env|${controller_env_escaped}|g" \
+    deploy/systemd/genarrative-external-generation-controller.service
+}
+
 render_database_backup_service() {
   local current_escaped env_escaped
   current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
@@ -742,6 +777,8 @@ render_health_patrol_service() {
 
 require_path deploy/systemd/spacetimedb.service
 require_path deploy/systemd/genarrative-api.service
+require_path deploy/systemd/genarrative-external-generation-worker@.service
+require_path deploy/systemd/genarrative-external-generation-controller.service
 require_path deploy/systemd/genarrative-database-backup.service
 require_path deploy/systemd/genarrative-database-backup.timer
 require_path deploy/systemd/genarrative-health-patrol.service
@@ -752,6 +789,8 @@ require_path deploy/nginx/genarrative.conf
 require_path deploy/nginx/genarrative-dev-http.conf
 require_path deploy/nginx/snippets/genarrative-maintenance.conf
 require_path deploy/env/api-server.env.example
+require_path deploy/env/external-generation-worker.env.example
+require_path deploy/env/external-generation-controller.env.example
 require_path scripts/deploy/maintenance-on.sh
 require_path scripts/deploy/maintenance-off.sh
 require_path scripts/deploy/maintenance-status.sh
@@ -795,19 +834,25 @@ sync_spacetime_install "${SPACETIME_ROOT}"
 
 spacetimedb_service="$(mktemp)"
 api_service="$(mktemp)"
+external_generation_worker_service="$(mktemp)"
+external_generation_controller_service="$(mktemp)"
 database_backup_service="$(mktemp)"
 health_patrol_service="$(mktemp)"
 render_spacetimedb_service >"${spacetimedb_service}"
 render_api_service >"${api_service}"
+render_external_generation_worker_service >"${external_generation_worker_service}"
+render_external_generation_controller_service >"${external_generation_controller_service}"
 render_database_backup_service >"${database_backup_service}"
 render_health_patrol_service >"${health_patrol_service}"
 install_file "${spacetimedb_service}" /etc/systemd/system/spacetimedb.service 0644
 install_file "${api_service}" /etc/systemd/system/genarrative-api.service 0644
+install_file "${external_generation_worker_service}" /etc/systemd/system/genarrative-external-generation-worker@.service 0644
+install_file "${external_generation_controller_service}" /etc/systemd/system/genarrative-external-generation-controller.service 0644
 install_file "${database_backup_service}" /etc/systemd/system/genarrative-database-backup.service 0644
 install_file deploy/systemd/genarrative-database-backup.timer /etc/systemd/system/genarrative-database-backup.timer 0644
 install_file "${health_patrol_service}" /etc/systemd/system/genarrative-health-patrol.service 0644
 install_file deploy/systemd/genarrative-health-patrol.timer /etc/systemd/system/genarrative-health-patrol.timer 0644
-rm -f "${spacetimedb_service}" "${api_service}" "${database_backup_service}" "${health_patrol_service}"
+rm -f "${spacetimedb_service}" "${api_service}" "${external_generation_worker_service}" "${external_generation_controller_service}" "${database_backup_service}" "${health_patrol_service}"
 
 if [[ ! -f "${API_ENV_FILE}" ]]; then
   echo "+ create ${API_ENV_FILE} from example"
@@ -821,6 +866,28 @@ else
 fi
 ensure_api_runtime_env_defaults
 
+if [[ ! -f "${WORKER_ENV_FILE}" ]]; then
+  echo "+ create ${WORKER_ENV_FILE} from example"
+  if [[ "${DRY_RUN}" != "true" ]]; then
+    render_external_generation_worker_env_example >"${WORKER_ENV_FILE}"
+    chmod 0600 "${WORKER_ENV_FILE}"
+    chown root:root "${WORKER_ENV_FILE}"
+  fi
+else
+  echo "[server-provision] 已存在 worker 环境文件，保留不覆盖: ${WORKER_ENV_FILE}"
+fi
+
+if [[ ! -f "${CONTROLLER_ENV_FILE}" ]]; then
+  echo "+ create ${CONTROLLER_ENV_FILE} from example"
+  if [[ "${DRY_RUN}" != "true" ]]; then
+    render_external_generation_controller_env_example >"${CONTROLLER_ENV_FILE}"
+    chmod 0600 "${CONTROLLER_ENV_FILE}"
+    chown root:root "${CONTROLLER_ENV_FILE}"
+  fi
+else
+  echo "[server-provision] 已存在 controller 环境文件，保留不覆盖: ${CONTROLLER_ENV_FILE}"
+fi
+
 if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
   sync_otelcol_install
   otelcol_service="$(mktemp)"
@@ -842,7 +909,7 @@ if [[ "${ENABLE_SERVICES}" == "true" ]]; then
   if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
     run_cmd systemctl enable otelcol-contrib.service
   fi
-  run_cmd systemctl enable spacetimedb.service genarrative-api.service genarrative-database-backup.timer genarrative-health-patrol.timer
+  run_cmd systemctl enable spacetimedb.service genarrative-api.service genarrative-database-backup.timer genarrative-external-generation-worker@1.service genarrative-external-generation-controller.service genarrative-health-patrol.timer
   if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
     run_cmd systemctl restart otelcol-contrib.service
   fi
@@ -851,8 +918,12 @@ if [[ "${ENABLE_SERVICES}" == "true" ]]; then
   ensure_spacetime_owner_client_token
   if [[ -x "${CURRENT_LINK}/api-server" ]]; then
     run_cmd systemctl restart genarrative-api.service
+    run_cmd systemctl enable --now genarrative-external-generation-worker@1.service
+    run_cmd systemctl restart genarrative-external-generation-worker@1.service
+    run_cmd systemctl enable --now genarrative-external-generation-controller.service
+    run_cmd systemctl restart genarrative-external-generation-controller.service
   else
-    echo "[server-provision] 尚未发现 ${CURRENT_LINK}/api-server，跳过 api-server 首次启动。后续 API deploy 会重启服务。"
+    echo "[server-provision] 尚未发现 ${CURRENT_LINK}/api-server，跳过 api-server、外部生成 worker 和 controller 首次启动。后续 API deploy 会启用并启动默认 worker 与 controller。"
   fi
 fi
 

scripts/ops/production-health-patrol.mjs

@@ -20,9 +20,11 @@ const DEFAULT_PUBLIC_PATHS = [
 
 const DEFAULT_SERVICES = [
   'genarrative-api.service',
+  'genarrative-external-generation-controller.service',
   'spacetimedb.service',
   'nginx.service',
 ];
+const WORKER_SERVICE_PATTERN = 'genarrative-external-generation-worker@*.service';
 
 function usage() {
   console.log(`Usage:
@@ -216,6 +218,61 @@ async function checkService(serviceName, timeoutMs) {
   );
 }
 
+async function checkActiveWorkerInstances(config) {
+  const result = await runCommand(
+    'systemctl',
+    [
+      'list-units',
+      WORKER_SERVICE_PATTERN,
+      '--type=service',
+      '--state=active',
+      '--no-legend',
+      '--plain',
+      '--no-pager',
+    ],
+    config.timeoutMs,
+  );
+  if (result.code !== 0) {
+    return checkResult(
+      'service:external-generation-workers',
+      'CRITICAL',
+      '无法枚举外部生成 worker 实例',
+      {
+        command: result.command,
+        stderr: result.stderr.trim() || result.error,
+      },
+    );
+  }
+
+  const services = result.stdout
+    .split('\n')
+    .map((line) => line.trim().split(/\s+/u)[0])
+    .filter((service) =>
+      /^genarrative-external-generation-worker@.+\.service$/u.test(service),
+    );
+
+  if (services.length === 0) {
+    return checkResult(
+      'service:external-generation-workers',
+      'CRITICAL',
+      '没有 active 的外部生成 worker 实例',
+      {
+        command: result.command,
+      },
+    );
+  }
+
+  return checkResult(
+    'service:external-generation-workers',
+    'OK',
+    `${services.length} 个 worker active`,
+    {
+      command: result.command,
+      services,
+    },
+  );
+}
+
 function requestUrl(url, timeoutMs) {
   return new Promise((resolve) => {
     const startedAt = Date.now();
@@ -310,6 +367,10 @@ async function checkRecentJournal(config) {
     '-u',
     'genarrative-api.service',
     '-u',
+    'genarrative-external-generation-controller.service',
+    '-u',
+    WORKER_SERVICE_PATTERN,
+    '-u',
     'spacetimedb.service',
     '-u',
     'nginx.service',
@@ -426,6 +487,7 @@ async function main() {
   for (const serviceName of DEFAULT_SERVICES) {
     checks.push(await checkService(serviceName, config.timeoutMs));
   }
+  checks.push(await checkActiveWorkerInstances(config));
 
   checks.push(await checkHttp('api:/healthz', joinUrl(config.apiBaseUrl, '/healthz'), config));
   checks.push(await checkHttp('api:/readyz', joinUrl(config.apiBaseUrl, '/readyz'), config));

server-rs/Cargo.toml

@@ -40,6 +40,7 @@ members = [
     "crates/platform-wechat",
     "crates/platform-speech",
     "crates/platform-agent",
+    "crates/server-manager-panel",
     "crates/shared-contracts",
     "crates/shared-kernel",
     "crates/shared-logging",

server-rs/crates/api-server/Cargo.toml

@@ -56,7 +56,7 @@ shared-kernel = { workspace = true }
 shared-logging = { workspace = true }
 socket2 = { workspace = true }
 spacetime-client = { workspace = true }
-tokio = { workspace = true, features = ["macros", "rt-multi-thread", "net", "time", "sync", "fs", "io-util", "signal"] }
+tokio = { workspace = true, features = ["macros", "rt-multi-thread", "net", "time", "sync", "fs", "io-util", "signal", "process"] }
 tokio-stream = { workspace = true }
 futures-util = { workspace = true }
 time = { workspace = true, features = ["formatting"] }

server-rs/crates/api-server/src/asset_billing.rs

@@ -4,7 +4,11 @@ use axum::http::StatusCode;
 use serde_json::json;
 use spacetime_client::SpacetimeClientError;
 
-use crate::{http_error::AppError, state::AppState};
+use crate::{
+    http_error::AppError,
+    state::AppState,
+    wallet_refund_outbox::{WalletRefundOutboxEnqueueOutcome, WalletRefundOutboxRecord},
+};
 
 pub(crate) const ASSET_OPERATION_POINTS_COST: u64 = 1;
 
@@ -48,7 +52,7 @@ where
     match operation.await {
         Ok(value) => Ok(value),
         Err(error) => {
-            if points_consumed {
+            if points_consumed && should_refund_asset_operation_error(&error) {
                 refund_asset_operation_points(
                     state,
                     owner_user_id,
@@ -63,6 +67,20 @@ where
     }
 }
 
+pub(crate) fn should_refund_asset_operation_error(error: &AppError) -> bool {
+    let message = error.body_text();
+    // 中文注释：worker lease guard 拒绝表示当前进程已失去队列写权限；
+    // 这类 stale worker 失败不能补偿退款，否则可能冲掉后续合法 worker 的同一账本扣费。
+    !(message.contains("external_generation_job")
+        && (message.contains("lease")
+            || message.contains("worker")
+            || message.contains("job_kind")
+            || message.contains("source_")
+            || message.contains("owner_user_id")
+            || message.contains("不存在")
+            || message.contains("不是 running 状态")))
+}
+
 /// 资产操作统一预扣泥点；扣费流水 ID 由业务资源 ID 参与构造，保证重试幂等。
 async fn consume_asset_operation_points(
     state: &AppState,
@@ -90,22 +108,11 @@ async fn consume_asset_operation_points(
         .await
     {
         Ok(_) => Ok(true),
-        Err(error) if should_skip_asset_operation_billing_for_connectivity(&error) => {
-            // 中文注释：外部生图不应被 Maincloud 钱包短暂 503 阻断；此时跳过扣费，让业务链路继续，避免用户重复点击。
-            tracing::warn!(
-                owner_user_id,
-                asset_kind,
-                asset_id,
-                error = %error,
-                "资产操作泥点预扣因 SpacetimeDB 连接不可用而降级跳过"
-            );
-            Ok(false)
-        }
         Err(error) => Err(map_asset_operation_wallet_error(error)),
     }
 }
 
-/// 外部生成或发布 mutation 失败后补偿退款；退款失败只记日志，避免覆盖原始业务错误。
+/// 外部生成或发布 mutation 失败后补偿退款；立即退款失败会进入 outbox，避免覆盖原始业务错误。
 async fn refund_asset_operation_points(
     state: &AppState,
     owner_user_id: &str,
@@ -117,22 +124,74 @@ async fn refund_asset_operation_points(
         "asset_operation_refund:{}:{}:{}",
         owner_user_id, asset_kind, asset_id
     );
+    let created_at_micros = current_utc_micros();
     if let Err(error) = state
         .spacetime_client()
         .refund_profile_wallet_points(
             owner_user_id.to_string(),
             points_cost,
-            ledger_id,
-            current_utc_micros(),
+            ledger_id.clone(),
+            created_at_micros,
         )
         .await
     {
+        let refund_error = error.to_string();
+        if let Some(outbox) = state.wallet_refund_outbox() {
+            match outbox
+                .enqueue(WalletRefundOutboxRecord {
+                    owner_user_id: owner_user_id.to_string(),
+                    amount: points_cost,
+                    ledger_id: ledger_id.clone(),
+                    created_at_micros,
+                    asset_kind: asset_kind.to_string(),
+                    asset_id: asset_id.to_string(),
+                })
+                .await
+            {
+                Ok(WalletRefundOutboxEnqueueOutcome::Enqueued) => {
+                    tracing::warn!(
+                        owner_user_id,
+                        asset_kind,
+                        asset_id,
+                        ledger_id,
+                        error = %refund_error,
+                        "资产操作失败后的泥点退款立即执行失败，已写入 wallet refund outbox"
+                    );
+                    return;
+                }
+                Ok(WalletRefundOutboxEnqueueOutcome::Dropped { reason }) => {
+                    tracing::error!(
+                        owner_user_id,
+                        asset_kind,
+                        asset_id,
+                        ledger_id,
+                        reason,
+                        error = %refund_error,
+                        "资产操作失败后的泥点退款立即执行失败，且 wallet refund outbox 因容量限制丢弃"
+                    );
+                    return;
+                }
+                Err(outbox_error) => {
+                    tracing::error!(
+                        owner_user_id,
+                        asset_kind,
+                        asset_id,
+                        ledger_id,
+                        refund_error = %refund_error,
+                        outbox_error = %outbox_error,
+                        "资产操作失败后的泥点退款立即执行失败，且写入 wallet refund outbox 失败"
+                    );
+                    return;
+                }
+            }
+        }
         tracing::error!(
             owner_user_id,
             asset_kind,
             asset_id,
-            error = %error,
-            "资产操作失败后的泥点退款失败"
+            ledger_id,
+            error = %refund_error,
+            "资产操作失败后的泥点退款失败，且 wallet refund outbox 未启用"
         );
     }
 }
@@ -185,7 +244,7 @@ mod tests {
     use super::*;
 
     #[test]
-    fn asset_operation_billing_skips_spacetime_connectivity_errors() {
+    fn asset_operation_connectivity_errors_are_classified_for_non_billing_fallbacks() {
         assert_eq!(ASSET_OPERATION_POINTS_COST, 1);
         assert!(should_skip_asset_operation_billing_for_connectivity(
             &SpacetimeClientError::ConnectDropped
@@ -204,4 +263,31 @@ mod tests {
             &SpacetimeClientError::Procedure("泥点余额不足".to_string()),
         ));
     }
+
+    #[test]
+    fn asset_operation_billing_does_not_refund_stale_worker_lease_errors() {
+        let stale_error = AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+            "provider": "spacetimedb",
+            "message": "external_generation_job lease 已过期",
+        }));
+        let completed_job_error =
+            AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+                "provider": "spacetimedb",
+                "message": "external_generation_job 当前不是 running 状态",
+            }));
+        let missing_job_error =
+            AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+                "provider": "spacetimedb",
+                "message": "external_generation_job 不存在",
+            }));
+        let ordinary_error = AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+            "provider": "vector-engine",
+            "message": "图片生成失败",
+        }));
+
+        assert!(!should_refund_asset_operation_error(&stale_error));
+        assert!(!should_refund_asset_operation_error(&completed_job_error));
+        assert!(!should_refund_asset_operation_error(&missing_job_error));
+        assert!(should_refund_asset_operation_error(&ordinary_error));
+    }
 }

server-rs/crates/api-server/src/bark_battle.rs

@@ -37,7 +37,7 @@ use time::{Duration as TimeDuration, OffsetDateTime};
 use crate::{
     api_response::json_success_body,
     asset_billing::execute_billable_asset_operation_with_cost,
-    auth::AuthenticatedAccessToken,
+    auth::{AuthenticatedAccessToken, RuntimePrincipal},
     generated_image_assets::{
         GeneratedImageAssetAdapter, GeneratedImageAssetDataUrl,
         adapter::{GeneratedImageAssetAdapterMetadata, GeneratedImageAssetPersistInput},
@@ -306,11 +306,12 @@ pub async fn generate_bark_battle_image_asset(
         .filter(|value| !value.is_empty())
         .map(ToString::to_string);
     let points_cost = resolve_bark_battle_image_asset_points_cost(&state, &payload).await;
+    let billing_asset_id = request_context.request_id().to_string();
     let result = execute_billable_asset_operation_with_cost(
         &state,
         &owner_user_id,
         bark_battle_slot_asset_kind(&slot),
-        asset_id.as_str(),
+        billing_asset_id.as_str(),
         points_cost,
         async {
             generate_and_persist_bark_battle_image_asset(
@@ -506,13 +507,13 @@ pub async fn get_bark_battle_runtime_config(
     State(state): State<AppState>,
     Path(work_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, &work_id, "workId")?;
 
     let config = state
         .spacetime_client()
-        .get_bark_battle_runtime_config(work_id, Some(authenticated.claims().user_id().to_string()))
+        .get_bark_battle_runtime_config(work_id, Some(principal.subject().to_string()))
         .await
         .map_err(|error| {
             bark_battle_error_response(&request_context, map_bark_battle_client_error(error))
@@ -526,7 +527,7 @@ pub async fn start_bark_battle_run(
     State(state): State<AppState>,
     Path(work_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<BarkBattleRunStartRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let maybe_payload = payload.ok().map(|Json(payload)| payload);
@@ -543,7 +544,7 @@ pub async fn start_bark_battle_run(
     };
     ensure_non_empty(&request_context, &work_id, "workId")?;
 
-    let owner_user_id = authenticated.claims().user_id().to_string();
+    let owner_user_id = principal.subject().to_string();
     let runtime_config = state
         .spacetime_client()
         .get_bark_battle_runtime_config(work_id.clone(), Some(owner_user_id.clone()))
@@ -593,12 +594,13 @@ pub async fn start_bark_battle_run(
     record_work_play_start_after_success(
         &state,
         &request_context,
-        WorkPlayTrackingDraft::new(
+        WorkPlayTrackingDraft::runtime_principal(
             BARK_BATTLE_PLAY_TYPE_ID,
             work_id.clone(),
-            &authenticated,
+            &principal,
             "/api/runtime/bark-battle/...",
         )
+        .owner_user_id(owner_user_id.clone())
         .extra(json!({
             "runId": run_snapshot.run_id,
             "workId": work_id,
@@ -607,6 +609,7 @@ pub async fn start_bark_battle_run(
             "difficultyPreset": runtime_config.difficulty_preset,
             "sourceRoute": request.source_route,
             "clientRuntimeVersion": request.client_runtime_version,
+            "principalKind": principal.kind().as_str(),
         })),
     )
     .await;
@@ -638,12 +641,12 @@ pub async fn get_bark_battle_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, &run_id, "runId")?;
     let run = state
         .spacetime_client()
-        .get_bark_battle_run(run_id, authenticated.claims().user_id().to_string())
+        .get_bark_battle_run(run_id, principal.subject().to_string())
         .await
         .map_err(|error| {
             bark_battle_error_response(&request_context, map_bark_battle_client_error(error))
@@ -657,7 +660,7 @@ pub async fn finish_bark_battle_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<BarkBattleRunFinishRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = bark_battle_json(payload, &request_context)?;
@@ -698,7 +701,7 @@ pub async fn finish_bark_battle_run(
         .finish_bark_battle_run(BarkBattleRunFinishRecordInput {
             run_id,
             run_token: payload.run_token,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             work_id: payload.work_id.clone(),
             config_version: u64::from(payload.config_version),
             ruleset_version: payload.ruleset_version.clone(),

server-rs/crates/api-server/src/big_fish.rs

@@ -63,7 +63,7 @@ use crate::{
     },
     api_response::json_success_body,
     asset_billing::execute_billable_asset_operation,
-    auth::AuthenticatedAccessToken,
+    auth::{AuthenticatedAccessToken, RuntimePrincipal},
     character_visual_assets::try_apply_background_alpha_to_png,
     http_error::AppError,
     platform_errors::map_oss_error,
@@ -224,7 +224,7 @@ pub async fn record_big_fish_play(
     State(state): State<AppState>,
     Path(session_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<RecordBigFishPlayRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = payload.map_err(|error| {
@@ -242,7 +242,7 @@ pub async fn record_big_fish_play(
         .spacetime_client()
         .record_big_fish_play(BigFishPlayReportRecordInput {
             session_id: session_id.clone(),
-            user_id: authenticated.claims().user_id().to_string(),
+            user_id: principal.subject().to_string(),
             elapsed_ms: payload.elapsed_ms.unwrap_or(0),
             reported_at_micros: current_utc_micros(),
         })
@@ -254,13 +254,14 @@ pub async fn record_big_fish_play(
     record_work_play_start_after_success(
         &state,
         &request_context,
-        WorkPlayTrackingDraft::new(
+        WorkPlayTrackingDraft::runtime_principal(
             "big-fish",
             session_id.clone(),
-            &authenticated,
+            &principal,
             "/api/runtime/big-fish/sessions/{session_id}/play",
         )
-        .run_id(session_id.clone()),
+        .run_id(session_id.clone())
+        .owner_user_id(principal.subject().to_string()),
     )
     .await;
 
@@ -279,7 +280,7 @@ pub async fn start_big_fish_run(
     State(state): State<AppState>,
     Path(session_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, &session_id, "sessionId")?;
 
@@ -288,7 +289,7 @@ pub async fn start_big_fish_run(
         .start_big_fish_run(BigFishRunStartRecordInput {
             run_id: build_prefixed_uuid_id("big-fish-run-"),
             session_id,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             started_at_micros: current_utc_micros(),
         })
         .await
@@ -339,13 +340,13 @@ pub async fn get_big_fish_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, &run_id, "runId")?;
 
     let run = state
         .spacetime_client()
-        .get_big_fish_run(run_id, authenticated.claims().user_id().to_string())
+        .get_big_fish_run(run_id, principal.subject().to_string())
         .await
         .map_err(|error| {
             big_fish_error_response(&request_context, map_big_fish_client_error(error))
@@ -363,7 +364,7 @@ pub async fn submit_big_fish_input(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<SubmitBigFishInputRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = payload.map_err(|error| {
@@ -384,7 +385,7 @@ pub async fn submit_big_fish_input(
         .spacetime_client()
         .submit_big_fish_input(BigFishInputSubmitRecordInput {
             run_id,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             x: payload.x,
             y: payload.y,
             submitted_at_micros: current_utc_micros(),
@@ -721,7 +722,7 @@ pub async fn execute_big_fish_action(
         "big_fish_publish_game" => Some("big_fish_publish_game"),
         _ => None,
     };
-    let billing_asset_id = format!("{session_id}:{now}");
+    let billing_asset_id = format!("{}:{}:{}", session_id, action, request_context.request_id());
     let session_operation = async {
         match action.as_str() {
             "big_fish_compile_draft" => {

server-rs/crates/api-server/src/config.rs

@@ -22,6 +22,19 @@ pub struct AppConfig {
     pub bind_port: u16,
     pub listen_backlog: i32,
     pub worker_threads: Option<usize>,
+    pub process_role: ProcessRole,
+    pub external_generation_mode: ExternalGenerationMode,
+    pub external_generation_worker_id: String,
+    pub external_generation_worker_concurrency: usize,
+    pub external_generation_worker_poll_interval: Duration,
+    pub external_generation_worker_lease: Duration,
+    pub external_generation_controller_min_workers: usize,
+    pub external_generation_controller_max_workers: usize,
+    pub external_generation_controller_target_jobs_per_worker: usize,
+    pub external_generation_controller_poll_interval: Duration,
+    pub external_generation_controller_scale_down_idle_rounds: u32,
+    pub external_generation_controller_service_template: String,
+    pub external_generation_controller_dry_run: bool,
     pub max_concurrent_requests: Option<usize>,
     pub gallery_max_concurrent_requests: Option<usize>,
     pub detail_max_concurrent_requests: Option<usize>,
@@ -32,6 +45,11 @@ pub struct AppConfig {
     pub tracking_outbox_batch_size: usize,
     pub tracking_outbox_flush_interval: Duration,
     pub tracking_outbox_max_bytes: u64,
+    pub wallet_refund_outbox_enabled: bool,
+    pub wallet_refund_outbox_dir: PathBuf,
+    pub wallet_refund_outbox_batch_size: usize,
+    pub wallet_refund_outbox_flush_interval: Duration,
+    pub wallet_refund_outbox_max_bytes: u64,
     pub log_filter: String,
     pub otel_enabled: bool,
     pub admin_username: Option<String>,
@@ -166,6 +184,56 @@ pub struct AppConfig {
     pub slow_request_threshold_ms: u64,
 }
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum ProcessRole {
+    Api,
+    ExternalGenerationWorker,
+    ExternalGenerationController,
+    All,
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum ExternalGenerationMode {
+    Inline,
+    Queue,
+}
+
+impl ExternalGenerationMode {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Inline => "inline",
+            Self::Queue => "queue",
+        }
+    }
+
+    pub fn is_inline(self) -> bool {
+        matches!(self, Self::Inline)
+    }
+}
+
+impl ProcessRole {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Api => "api",
+            Self::ExternalGenerationWorker => "external-generation-worker",
+            Self::ExternalGenerationController => "external-generation-controller",
+            Self::All => "all",
+        }
+    }
+
+    pub fn runs_http(self) -> bool {
+        matches!(self, Self::Api | Self::All)
+    }
+
+    pub fn runs_external_generation_worker(self) -> bool {
+        matches!(self, Self::ExternalGenerationWorker | Self::All)
+    }
+
+    pub fn runs_external_generation_controller(self) -> bool {
+        matches!(self, Self::ExternalGenerationController)
+    }
+}
+
 impl Default for AppConfig {
     fn default() -> Self {
         Self {
@@ -173,6 +241,20 @@ impl Default for AppConfig {
             bind_port: 3000,
             listen_backlog: 1024,
             worker_threads: None,
+            process_role: ProcessRole::Api,
+            external_generation_mode: ExternalGenerationMode::Queue,
+            external_generation_worker_id: default_external_generation_worker_id(),
+            external_generation_worker_concurrency: 2,
+            external_generation_worker_poll_interval: Duration::from_millis(2_000),
+            external_generation_worker_lease: Duration::from_secs(3_600),
+            external_generation_controller_min_workers: 1,
+            external_generation_controller_max_workers: 8,
+            external_generation_controller_target_jobs_per_worker: 2,
+            external_generation_controller_poll_interval: Duration::from_millis(10_000),
+            external_generation_controller_scale_down_idle_rounds: 6,
+            external_generation_controller_service_template:
+                "genarrative-external-generation-worker@{}.service".to_string(),
+            external_generation_controller_dry_run: false,
             max_concurrent_requests: None,
             gallery_max_concurrent_requests: None,
             detail_max_concurrent_requests: None,
@@ -183,6 +265,11 @@ impl Default for AppConfig {
             tracking_outbox_batch_size: 500,
             tracking_outbox_flush_interval: Duration::from_millis(1_000),
             tracking_outbox_max_bytes: 256 * 1024 * 1024,
+            wallet_refund_outbox_enabled: true,
+            wallet_refund_outbox_dir: PathBuf::from("server-rs/.data/wallet-refund-outbox"),
+            wallet_refund_outbox_batch_size: 100,
+            wallet_refund_outbox_flush_interval: Duration::from_millis(1_000),
+            wallet_refund_outbox_max_bytes: 64 * 1024 * 1024,
             log_filter: "info,tower_http=info".to_string(),
             otel_enabled: false,
             admin_username: None,
@@ -364,6 +451,78 @@ impl AppConfig {
         if let Some(worker_threads) = read_first_usize_env(&["GENARRATIVE_API_WORKER_THREADS"]) {
             config.worker_threads = Some(worker_threads);
         }
+        if let Some(process_role) = read_first_process_role_env(&["GENARRATIVE_PROCESS_ROLE"]) {
+            config.process_role = process_role;
+        }
+        if let Some(external_generation_mode) =
+            read_first_external_generation_mode_env(&["GENARRATIVE_EXTERNAL_GENERATION_MODE"])
+        {
+            config.external_generation_mode = external_generation_mode;
+        }
+        if let Some(worker_id) =
+            read_first_non_empty_env(&["GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID"])
+        {
+            config.external_generation_worker_id = worker_id;
+        }
+        if let Some(concurrency) =
+            read_first_usize_env(&["GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY"])
+        {
+            config.external_generation_worker_concurrency = concurrency.max(1);
+        }
+        if let Some(poll_interval_ms) = read_first_positive_u64_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS",
+        ]) {
+            config.external_generation_worker_poll_interval =
+                Duration::from_millis(poll_interval_ms);
+        }
+        if let Some(lease_seconds) = read_first_duration_seconds_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS",
+        ]) {
+            config.external_generation_worker_lease = Duration::from_secs(lease_seconds.max(1));
+        }
+        if let Some(min_workers) =
+            read_first_usize_env(&["GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MIN_WORKERS"])
+        {
+            config.external_generation_controller_min_workers = min_workers;
+        }
+        if let Some(max_workers) =
+            read_first_usize_env(&["GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_MAX_WORKERS"])
+        {
+            config.external_generation_controller_max_workers = max_workers;
+        }
+        if config.external_generation_controller_max_workers
+            < config.external_generation_controller_min_workers
+        {
+            config.external_generation_controller_max_workers =
+                config.external_generation_controller_min_workers;
+        }
+        if let Some(target_jobs_per_worker) = read_first_usize_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_TARGET_JOBS_PER_WORKER",
+        ]) {
+            config.external_generation_controller_target_jobs_per_worker =
+                target_jobs_per_worker.max(1);
+        }
+        if let Some(poll_interval_ms) = read_first_positive_u64_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_POLL_INTERVAL_MS",
+        ]) {
+            config.external_generation_controller_poll_interval =
+                Duration::from_millis(poll_interval_ms);
+        }
+        if let Some(idle_rounds) = read_first_u32_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_SCALE_DOWN_IDLE_ROUNDS",
+        ]) {
+            config.external_generation_controller_scale_down_idle_rounds = idle_rounds;
+        }
+        if let Some(service_template) = read_first_non_empty_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_SERVICE_TEMPLATE",
+        ]) {
+            config.external_generation_controller_service_template = service_template;
+        }
+        if let Some(dry_run) =
+            read_first_bool_env(&["GENARRATIVE_EXTERNAL_GENERATION_CONTROLLER_DRY_RUN"])
+        {
+            config.external_generation_controller_dry_run = dry_run;
+        }
         if let Some(max_concurrent_requests) =
             read_first_usize_env(&["GENARRATIVE_API_MAX_CONCURRENT_REQUESTS"])
         {
@@ -409,6 +568,27 @@ impl AppConfig {
         {
             config.tracking_outbox_max_bytes = max_bytes;
         }
+        if let Some(enabled) = read_first_bool_env(&["GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED"]) {
+            config.wallet_refund_outbox_enabled = enabled;
+        }
+        if let Some(dir) = read_first_non_empty_env(&["GENARRATIVE_WALLET_REFUND_OUTBOX_DIR"]) {
+            config.wallet_refund_outbox_dir = PathBuf::from(dir);
+        }
+        if let Some(batch_size) =
+            read_first_usize_env(&["GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE"])
+        {
+            config.wallet_refund_outbox_batch_size = batch_size;
+        }
+        if let Some(flush_interval_ms) =
+            read_first_positive_u64_env(&["GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS"])
+        {
+            config.wallet_refund_outbox_flush_interval = Duration::from_millis(flush_interval_ms);
+        }
+        if let Some(max_bytes) =
+            read_first_positive_u64_env(&["GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES"])
+        {
+            config.wallet_refund_outbox_max_bytes = max_bytes;
+        }
         if let Some(otel_enabled) = read_first_bool_env(&["GENARRATIVE_OTEL_ENABLED"]) {
             config.otel_enabled = otel_enabled;
         }
@@ -1022,6 +1202,22 @@ fn read_first_llm_provider_env(keys: &[&str]) -> Option<LlmProvider> {
     })
 }
 
+fn read_first_process_role_env(keys: &[&str]) -> Option<ProcessRole> {
+    keys.iter().find_map(|key| {
+        env::var(key)
+            .ok()
+            .and_then(|value| parse_process_role(&value))
+    })
+}
+
+fn read_first_external_generation_mode_env(keys: &[&str]) -> Option<ExternalGenerationMode> {
+    keys.iter().find_map(|key| {
+        env::var(key)
+            .ok()
+            .and_then(|value| parse_external_generation_mode(&value))
+    })
+}
+
 fn read_first_positive_u32_env(keys: &[&str]) -> Option<u32> {
     keys.iter().find_map(|key| {
         env::var(key)
@@ -1069,6 +1265,49 @@ fn read_first_u8_env(keys: &[&str]) -> Option<u8> {
         .find_map(|key| env::var(key).ok().and_then(|value| parse_u8(&value)))
 }
 
+fn default_external_generation_worker_id() -> String {
+    let host = env::var("HOSTNAME")
+        .or_else(|_| env::var("COMPUTERNAME"))
+        .unwrap_or_else(|_| "local".to_string());
+    format!("{}-{}", host.trim(), std::process::id())
+}
+
+fn parse_process_role(value: &str) -> Option<ProcessRole> {
+    match trim_quoted_env_value(value).to_ascii_lowercase().as_str() {
+        "api" => Some(ProcessRole::Api),
+        "external-generation-worker" | "external_generation_worker" | "worker" => {
+            Some(ProcessRole::ExternalGenerationWorker)
+        }
+        "external-generation-controller" | "external_generation_controller" | "controller" => {
+            Some(ProcessRole::ExternalGenerationController)
+        }
+        "all" => Some(ProcessRole::All),
+        _ => None,
+    }
+}
+
+fn parse_external_generation_mode(value: &str) -> Option<ExternalGenerationMode> {
+    match trim_quoted_env_value(value).to_ascii_lowercase().as_str() {
+        "inline" | "sync" | "synchronous" => Some(ExternalGenerationMode::Inline),
+        "queue" | "queued" | "worker" | "async" | "asynchronous" => {
+            Some(ExternalGenerationMode::Queue)
+        }
+        _ => None,
+    }
+}
+
+fn trim_quoted_env_value(raw: &str) -> &str {
+    let raw = raw.trim();
+    raw.strip_prefix('"')
+        .and_then(|value| value.strip_suffix('"'))
+        .or_else(|| {
+            raw.strip_prefix('\'')
+                .and_then(|value| value.strip_suffix('\''))
+        })
+        .unwrap_or(raw)
+        .trim()
+}
+
 fn read_first_positive_u16_env(keys: &[&str]) -> Option<u16> {
     keys.iter().find_map(|key| {
         env::var(key)
@@ -1189,7 +1428,8 @@ fn parse_positive_u16(raw: &str) -> Option<u16> {
 #[cfg(test)]
 mod tests {
     use super::{
-        AppConfig, DEFAULT_VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS, LlmProvider, parse_bool,
+        AppConfig, DEFAULT_VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS, ExternalGenerationMode,
+        LlmProvider, ProcessRole, parse_bool, parse_external_generation_mode, parse_process_role,
     };
     use std::sync::{Mutex, OnceLock};
 
@@ -1231,6 +1471,91 @@ mod tests {
         assert_eq!(parse_bool("'off'"), Some(false));
     }
 
+    #[test]
+    fn process_role_controls_http_and_external_generation_worker_roles() {
+        assert_eq!(parse_process_role("api"), Some(ProcessRole::Api));
+        assert_eq!(
+            parse_process_role("\"external-generation-worker\""),
+            Some(ProcessRole::ExternalGenerationWorker)
+        );
+        assert_eq!(
+            parse_process_role("'external_generation_worker'"),
+            Some(ProcessRole::ExternalGenerationWorker)
+        );
+        assert_eq!(
+            parse_process_role("worker"),
+            Some(ProcessRole::ExternalGenerationWorker)
+        );
+        assert_eq!(
+            parse_process_role("controller"),
+            Some(ProcessRole::ExternalGenerationController)
+        );
+        assert_eq!(
+            parse_process_role("'external_generation_controller'"),
+            Some(ProcessRole::ExternalGenerationController)
+        );
+        assert_eq!(parse_process_role("all"), Some(ProcessRole::All));
+        assert_eq!(parse_process_role("unknown"), None);
+
+        assert!(ProcessRole::Api.runs_http());
+        assert!(!ProcessRole::Api.runs_external_generation_worker());
+        assert!(!ProcessRole::Api.runs_external_generation_controller());
+        assert!(!ProcessRole::ExternalGenerationWorker.runs_http());
+        assert!(ProcessRole::ExternalGenerationWorker.runs_external_generation_worker());
+        assert!(!ProcessRole::ExternalGenerationWorker.runs_external_generation_controller());
+        assert!(!ProcessRole::ExternalGenerationController.runs_http());
+        assert!(!ProcessRole::ExternalGenerationController.runs_external_generation_worker());
+        assert!(ProcessRole::ExternalGenerationController.runs_external_generation_controller());
+        assert!(ProcessRole::All.runs_http());
+        assert!(ProcessRole::All.runs_external_generation_worker());
+        assert!(!ProcessRole::All.runs_external_generation_controller());
+    }
+
+    #[test]
+    fn external_generation_mode_parses_inline_and_queue_aliases() {
+        assert_eq!(
+            parse_external_generation_mode("inline"),
+            Some(ExternalGenerationMode::Inline)
+        );
+        assert_eq!(
+            parse_external_generation_mode("'sync'"),
+            Some(ExternalGenerationMode::Inline)
+        );
+        assert_eq!(
+            parse_external_generation_mode("\"queue\""),
+            Some(ExternalGenerationMode::Queue)
+        );
+        assert_eq!(
+            parse_external_generation_mode("worker"),
+            Some(ExternalGenerationMode::Queue)
+        );
+        assert_eq!(parse_external_generation_mode("unknown"), None);
+
+        assert!(ExternalGenerationMode::Inline.is_inline());
+        assert!(!ExternalGenerationMode::Queue.is_inline());
+    }
+
+    #[test]
+    fn from_env_reads_external_generation_mode() {
+        let _guard = ENV_LOCK
+            .get_or_init(|| Mutex::new(()))
+            .lock()
+            .expect("env lock");
+        unsafe {
+            std::env::set_var("GENARRATIVE_EXTERNAL_GENERATION_MODE", "inline");
+        }
+
+        let config = AppConfig::from_env();
+
+        assert_eq!(
+            config.external_generation_mode,
+            ExternalGenerationMode::Inline
+        );
+        unsafe {
+            std::env::remove_var("GENARRATIVE_EXTERNAL_GENERATION_MODE");
+        }
+    }
+
     #[test]
     fn from_env_reads_sms_enabled_when_shell_value_keeps_quotes() {
         let _guard = ENV_LOCK
@@ -1380,6 +1705,11 @@ mod tests {
             std::env::remove_var("GENARRATIVE_TRACKING_OUTBOX_BATCH_SIZE");
             std::env::remove_var("GENARRATIVE_TRACKING_OUTBOX_FLUSH_INTERVAL_MS");
             std::env::remove_var("GENARRATIVE_TRACKING_OUTBOX_MAX_BYTES");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_DIR");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES");
             std::env::remove_var("GENARRATIVE_OTEL_ENABLED");
             std::env::set_var("GENARRATIVE_API_LISTEN_BACKLOG", "2048");
             std::env::set_var("GENARRATIVE_API_WORKER_THREADS", "6");
@@ -1396,6 +1726,14 @@ mod tests {
             std::env::set_var("GENARRATIVE_TRACKING_OUTBOX_BATCH_SIZE", "250");
             std::env::set_var("GENARRATIVE_TRACKING_OUTBOX_FLUSH_INTERVAL_MS", "2000");
             std::env::set_var("GENARRATIVE_TRACKING_OUTBOX_MAX_BYTES", "1048576");
+            std::env::set_var("GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED", "false");
+            std::env::set_var(
+                "GENARRATIVE_WALLET_REFUND_OUTBOX_DIR",
+                "/tmp/genarrative-wallet-refund-outbox",
+            );
+            std::env::set_var("GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE", "50");
+            std::env::set_var("GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS", "3000");
+            std::env::set_var("GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES", "524288");
             std::env::set_var("GENARRATIVE_OTEL_ENABLED", "true");
         }
 
@@ -1421,6 +1759,17 @@ mod tests {
             std::time::Duration::from_millis(2_000)
         );
         assert_eq!(config.tracking_outbox_max_bytes, 1_048_576);
+        assert!(!config.wallet_refund_outbox_enabled);
+        assert_eq!(
+            config.wallet_refund_outbox_dir,
+            std::path::PathBuf::from("/tmp/genarrative-wallet-refund-outbox")
+        );
+        assert_eq!(config.wallet_refund_outbox_batch_size, 50);
+        assert_eq!(
+            config.wallet_refund_outbox_flush_interval,
+            std::time::Duration::from_millis(3_000)
+        );
+        assert_eq!(config.wallet_refund_outbox_max_bytes, 524_288);
         assert!(config.otel_enabled);
 
         unsafe {
@@ -1436,6 +1785,11 @@ mod tests {
             std::env::remove_var("GENARRATIVE_TRACKING_OUTBOX_BATCH_SIZE");
             std::env::remove_var("GENARRATIVE_TRACKING_OUTBOX_FLUSH_INTERVAL_MS");
             std::env::remove_var("GENARRATIVE_TRACKING_OUTBOX_MAX_BYTES");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_DIR");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS");
+            std::env::remove_var("GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES");
             std::env::remove_var("GENARRATIVE_OTEL_ENABLED");
         }
     }

server-rs/crates/api-server/src/custom_world_ai.rs

@@ -547,11 +547,12 @@ pub async fn generate_custom_world_scene_image(
     require_openai_image_settings(&state)
         .map_err(|error| custom_world_ai_error_response(&request_context, error))?;
     let asset_id = format!("custom-scene-{}", current_utc_millis());
+    let billing_asset_id = request_context.request_id().to_string();
     let asset = execute_billable_asset_operation(
         &state,
         &owner_user_id,
         "scene_image",
-        asset_id.as_str(),
+        billing_asset_id.as_str(),
         async {
             let settings = require_openai_image_settings(&state)?.with_external_api_audit_context(
                 &request_context,
@@ -806,11 +807,12 @@ pub async fn generate_custom_world_cover_image(
     require_dashscope_settings(&state)
         .map_err(|error| custom_world_ai_error_response(&request_context, error))?;
     let asset_id = format!("custom-cover-{}", current_utc_millis());
+    let billing_asset_id = request_context.request_id().to_string();
     let asset = execute_billable_asset_operation(
         &state,
         &owner_user_id,
         "custom_world_cover",
-        asset_id.as_str(),
+        billing_asset_id.as_str(),
         async {
             let settings = require_dashscope_settings(&state)?;
             let http_client = build_dashscope_http_client(&settings)?;
@@ -1011,11 +1013,12 @@ pub async fn generate_custom_world_opening_cg(
         .map_err(|error| custom_world_ai_error_response(&request_context, error))?;
 
     let opening_cg_id = normalized.opening_cg_id.clone();
+    let billing_asset_id = request_context.request_id().to_string();
     let generated = execute_billable_asset_operation_with_cost(
         &state,
         &owner_user_id,
         "custom_world_opening_cg",
-        opening_cg_id.as_str(),
+        billing_asset_id.as_str(),
         OPENING_CG_POINTS_COST,
         async {
             let image_settings = require_openai_image_settings(&state)?

server-rs/crates/api-server/src/external_generation_worker.rs

@@ -0,0 +1,583 @@
+use std::{future::Future, io, pin::Pin, time::Duration};
+
+use axum::extract::FromRef;
+use serde_json::json;
+use shared_kernel::offset_datetime_to_unix_micros;
+use spacetime_client::{
+    ExternalGenerationJobClaimRecordInput, ExternalGenerationJobCompleteRecordInput,
+    ExternalGenerationJobFailRecordInput, ExternalGenerationJobRecord,
+    ExternalGenerationJobRenewLeaseRecordInput,
+};
+use tokio::{
+    task::JoinSet,
+    time::{Instant, sleep},
+};
+use tracing::{error, info, warn};
+
+use crate::{
+    puzzle::{
+        ExternalGenerationWriteLeaseGuard, PuzzleCompileDraftWorkerPayload,
+        PuzzleGenerateImagesWorkerPayload, PuzzleGenerateUiBackgroundWorkerPayload,
+        execute_puzzle_compile_draft_worker_job, execute_puzzle_generate_images_worker_job,
+        execute_puzzle_generate_ui_background_worker_job, release_puzzle_compile_background_claim,
+    },
+    request_context::RequestContext,
+    state::{AppState, PuzzleApiState},
+};
+
+pub(crate) const PUZZLE_COMPILE_DRAFT_JOB_KIND: &str = "puzzle_compile_draft";
+pub(crate) const PUZZLE_GENERATE_IMAGES_JOB_KIND: &str = "puzzle_generate_images";
+pub(crate) const PUZZLE_GENERATE_UI_BACKGROUND_JOB_KIND: &str = "puzzle_generate_ui_background";
+
+pub(crate) async fn run_external_generation_worker(state: AppState) -> Result<(), io::Error> {
+    let worker_id = state.config.external_generation_worker_id.clone();
+    let concurrency = state.config.external_generation_worker_concurrency.max(1);
+    let poll_interval = state.config.external_generation_worker_poll_interval;
+    let lease = state.config.external_generation_worker_lease;
+    let mut tasks = JoinSet::new();
+    let mut shutdown = external_generation_worker_shutdown_signal();
+
+    info!(
+        worker_id,
+        concurrency,
+        poll_interval_ms = poll_interval.as_millis(),
+        lease_seconds = lease.as_secs(),
+        "external generation worker 已启动"
+    );
+
+    loop {
+        while tasks.len() >= concurrency {
+            if await_worker_task_or_shutdown(&mut tasks, &mut shutdown).await {
+                drain_external_generation_worker_tasks(&mut tasks).await;
+                return Ok(());
+            }
+        }
+
+        let available = concurrency.saturating_sub(tasks.len()).max(1);
+        let now_micros = current_utc_micros();
+        let lease_expires_at_micros = now_micros.saturating_add(duration_micros_i64(lease));
+
+        let claim_jobs = state.spacetime_client().claim_external_generation_jobs(
+            ExternalGenerationJobClaimRecordInput {
+                worker_id: worker_id.clone(),
+                limit: available.min(u32::MAX as usize) as u32,
+                lease_expires_at_micros,
+                claimed_at_micros: now_micros,
+            },
+        );
+        tokio::pin!(claim_jobs);
+        let jobs = match tokio::select! {
+            _ = shutdown.as_mut() => {
+                drain_external_generation_worker_tasks(&mut tasks).await;
+                return Ok(());
+            }
+            result = &mut claim_jobs => result
+        } {
+            Ok(jobs) => jobs,
+            Err(error) => {
+                error!(error = %error, "领取外部生成任务失败，等待下一轮重试");
+                if await_one_task_or_sleep_or_shutdown(
+                    &mut tasks,
+                    sleep(poll_interval),
+                    &mut shutdown,
+                )
+                .await
+                {
+                    drain_external_generation_worker_tasks(&mut tasks).await;
+                    return Ok(());
+                }
+                continue;
+            }
+        };
+
+        if jobs.is_empty() {
+            if await_one_task_or_sleep_or_shutdown(&mut tasks, sleep(poll_interval), &mut shutdown)
+                .await
+            {
+                drain_external_generation_worker_tasks(&mut tasks).await;
+                return Ok(());
+            }
+            continue;
+        }
+
+        for job in jobs {
+            let state = state.clone();
+            let worker_id = worker_id.clone();
+            tasks.spawn(async move {
+                if let Err(error) =
+                    process_external_generation_job(state, worker_id, lease, job).await
+                {
+                    error!(error = %error, "external generation worker 执行任务失败");
+                }
+            });
+        }
+    }
+}
+
+type ExternalGenerationShutdownSignal = Pin<Box<dyn Future<Output = ()> + Send>>;
+
+fn external_generation_worker_shutdown_signal() -> ExternalGenerationShutdownSignal {
+    Box::pin(async {
+        wait_for_external_generation_worker_shutdown_signal().await;
+    })
+}
+
+#[cfg(unix)]
+async fn wait_for_external_generation_worker_shutdown_signal() {
+    use tokio::signal::unix::{SignalKind, signal};
+
+    let mut sigterm = signal(SignalKind::terminate()).ok();
+    tokio::select! {
+        result = tokio::signal::ctrl_c() => {
+            if let Err(error) = result {
+                warn!(error = %error, "external generation worker 监听 SIGINT 失败");
+            }
+        }
+        _ = async {
+            if let Some(sigterm) = sigterm.as_mut() {
+                sigterm.recv().await;
+            } else {
+                std::future::pending::<()>().await;
+            }
+        } => {}
+    }
+}
+
+#[cfg(not(unix))]
+async fn wait_for_external_generation_worker_shutdown_signal() {
+    if let Err(error) = tokio::signal::ctrl_c().await {
+        warn!(error = %error, "external generation worker 监听 Ctrl-C 失败");
+    }
+}
+
+async fn await_worker_task(tasks: &mut JoinSet<()>) {
+    if let Some(result) = tasks.join_next().await
+        && let Err(error) = result
+    {
+        error!(error = %error, "external generation worker 子任务 panic");
+    }
+}
+
+async fn await_worker_task_or_shutdown(
+    tasks: &mut JoinSet<()>,
+    shutdown: &mut ExternalGenerationShutdownSignal,
+) -> bool {
+    tokio::select! {
+        _ = shutdown.as_mut() => true,
+        _ = await_worker_task(tasks) => false,
+    }
+}
+
+async fn await_one_task_or_sleep_or_shutdown(
+    tasks: &mut JoinSet<()>,
+    sleeper: impl Future<Output = ()>,
+    shutdown: &mut ExternalGenerationShutdownSignal,
+) -> bool {
+    tokio::pin!(sleeper);
+    if tasks.is_empty() {
+        tokio::select! {
+            _ = shutdown.as_mut() => true,
+            _ = &mut sleeper => false,
+        }
+    } else {
+        tokio::select! {
+            _ = shutdown.as_mut() => true,
+            _ = &mut sleeper => false,
+            result = tasks.join_next() => {
+                if let Some(Err(error)) = result {
+                    error!(error = %error, "external generation worker 子任务 panic");
+                }
+                false
+            }
+        }
+    }
+}
+
+async fn drain_external_generation_worker_tasks(tasks: &mut JoinSet<()>) {
+    info!(
+        in_flight_jobs = tasks.len(),
+        "external generation worker 收到停机信号，停止领取新任务并等待当前任务完成"
+    );
+    while !tasks.is_empty() {
+        await_worker_task(tasks).await;
+    }
+    info!("external generation worker 已完成优雅停机");
+}
+
+async fn process_external_generation_job(
+    state: AppState,
+    worker_id: String,
+    lease: Duration,
+    job: ExternalGenerationJobRecord,
+) -> Result<(), String> {
+    let heartbeat_interval = external_generation_worker_heartbeat_interval(lease);
+    let work = process_external_generation_job_once(state.clone(), worker_id.clone(), job.clone());
+    tokio::pin!(work);
+    let heartbeat = sleep(heartbeat_interval);
+    tokio::pin!(heartbeat);
+
+    loop {
+        tokio::select! {
+            biased;
+            result = &mut work => return result,
+            _ = &mut heartbeat => {
+                renew_job_lease(&state, &worker_id, &job, lease).await?;
+                heartbeat.as_mut().reset(Instant::now() + heartbeat_interval);
+            }
+        }
+    }
+}
+
+async fn process_external_generation_job_once(
+    state: AppState,
+    worker_id: String,
+    job: ExternalGenerationJobRecord,
+) -> Result<(), String> {
+    match job.job_kind.as_str() {
+        PUZZLE_COMPILE_DRAFT_JOB_KIND => {
+            let payload = match serde_json::from_str::<PuzzleCompileDraftWorkerPayload>(
+                job.request_payload_json.as_str(),
+            ) {
+                Ok(payload) => payload,
+                Err(error) => {
+                    let message = format!("拼图生成任务参数解析失败：{error}");
+                    fail_job(&state, &worker_id, &job, message.clone()).await?;
+                    return Err(message);
+                }
+            };
+            let request_context = RequestContext::new(
+                format!("external-generation-worker-{}", job.job_id),
+                format!("external-generation-worker {}", job.job_kind),
+                std::time::Duration::ZERO,
+                false,
+            );
+            let puzzle_state = PuzzleApiState::from_ref(&state);
+            let write_guard = build_external_generation_write_lease_guard(&worker_id, &job)?;
+            match execute_puzzle_compile_draft_worker_job(
+                &puzzle_state,
+                &request_context,
+                payload.clone(),
+                write_guard,
+            )
+            .await
+            {
+                Ok(session) => {
+                    let result = complete_job(
+                        &state,
+                        &worker_id,
+                        &job,
+                        Some(
+                            json!({
+                                "sessionId": session.session_id,
+                                "progressPercent": session.progress_percent,
+                            })
+                            .to_string(),
+                        ),
+                    )
+                    .await;
+                    if result.is_ok() {
+                        release_puzzle_compile_background_claim(&puzzle_state, &payload);
+                    }
+                    result
+                }
+                Err(error) => {
+                    let message = error.body_text();
+                    let should_release_claim = error.should_fail_queue_job();
+                    let result = fail_queue_job_after_worker_error(
+                        &state, &worker_id, &job, &error, &message,
+                    )
+                    .await;
+                    if result.is_ok() && should_release_claim {
+                        release_puzzle_compile_background_claim(&puzzle_state, &payload);
+                    }
+                    result?;
+                    Err(message)
+                }
+            }
+        }
+        PUZZLE_GENERATE_IMAGES_JOB_KIND => {
+            let payload = match serde_json::from_str::<PuzzleGenerateImagesWorkerPayload>(
+                job.request_payload_json.as_str(),
+            ) {
+                Ok(payload) => payload,
+                Err(error) => {
+                    let message = format!("拼图关卡图片生成任务参数解析失败：{error}");
+                    fail_job(&state, &worker_id, &job, message.clone()).await?;
+                    return Err(message);
+                }
+            };
+            let request_context = RequestContext::new(
+                format!("external-generation-worker-{}", job.job_id),
+                format!("external-generation-worker {}", job.job_kind),
+                std::time::Duration::ZERO,
+                false,
+            );
+            let puzzle_state = PuzzleApiState::from_ref(&state);
+            let write_guard = build_external_generation_write_lease_guard(&worker_id, &job)?;
+            match execute_puzzle_generate_images_worker_job(
+                &puzzle_state,
+                &request_context,
+                payload,
+                write_guard,
+            )
+            .await
+            {
+                Ok(session) => {
+                    complete_job(
+                        &state,
+                        &worker_id,
+                        &job,
+                        Some(
+                            json!({
+                                "sessionId": session.session_id,
+                                "progressPercent": session.progress_percent,
+                            })
+                            .to_string(),
+                        ),
+                    )
+                    .await
+                }
+                Err(error) => {
+                    let message = error.body_text();
+                    fail_queue_job_after_worker_error(&state, &worker_id, &job, &error, &message)
+                        .await?;
+                    Err(message)
+                }
+            }
+        }
+        PUZZLE_GENERATE_UI_BACKGROUND_JOB_KIND => {
+            let payload = match serde_json::from_str::<PuzzleGenerateUiBackgroundWorkerPayload>(
+                job.request_payload_json.as_str(),
+            ) {
+                Ok(payload) => payload,
+                Err(error) => {
+                    let message = format!("拼图 UI 背景图生成任务参数解析失败：{error}");
+                    fail_job(&state, &worker_id, &job, message.clone()).await?;
+                    return Err(message);
+                }
+            };
+            let request_context = RequestContext::new(
+                format!("external-generation-worker-{}", job.job_id),
+                format!("external-generation-worker {}", job.job_kind),
+                std::time::Duration::ZERO,
+                false,
+            );
+            let puzzle_state = PuzzleApiState::from_ref(&state);
+            let write_guard = build_external_generation_write_lease_guard(&worker_id, &job)?;
+            match execute_puzzle_generate_ui_background_worker_job(
+                &puzzle_state,
+                &request_context,
+                payload,
+                write_guard,
+            )
+            .await
+            {
+                Ok(session) => {
+                    complete_job(
+                        &state,
+                        &worker_id,
+                        &job,
+                        Some(
+                            json!({
+                                "sessionId": session.session_id,
+                                "progressPercent": session.progress_percent,
+                            })
+                            .to_string(),
+                        ),
+                    )
+                    .await
+                }
+                Err(error) => {
+                    let message = error.body_text();
+                    fail_queue_job_after_worker_error(&state, &worker_id, &job, &error, &message)
+                        .await?;
+                    Err(message)
+                }
+            }
+        }
+        unknown => {
+            warn!(
+                job_id = job.job_id,
+                job_kind = unknown,
+                "external generation worker 收到暂不支持的任务类型"
+            );
+            fail_job(
+                &state,
+                &worker_id,
+                &job,
+                format!("暂不支持的外部生成任务类型：{unknown}"),
+            )
+            .await
+        }
+    }
+}
+
+async fn fail_queue_job_after_worker_error(
+    state: &AppState,
+    worker_id: &str,
+    job: &ExternalGenerationJobRecord,
+    error: &crate::puzzle::PuzzleExternalGenerationWorkerError,
+    message: &str,
+) -> Result<(), String> {
+    if error.should_fail_queue_job() {
+        return fail_job(state, worker_id, job, message.to_string()).await;
+    }
+
+    warn!(
+        job_id = job.job_id,
+        job_kind = job.job_kind,
+        "external generation worker 业务失败态尚未写回，保留任务租约等待后续重试"
+    );
+    Ok(())
+}
+
+async fn complete_job(
+    state: &AppState,
+    worker_id: &str,
+    job: &ExternalGenerationJobRecord,
+    result_payload_json: Option<String>,
+) -> Result<(), String> {
+    state
+        .spacetime_client()
+        .complete_external_generation_job(ExternalGenerationJobCompleteRecordInput {
+            job_id: job.job_id.clone(),
+            worker_id: worker_id.to_string(),
+            lease_token: require_job_lease_token(job)?,
+            result_payload_json,
+            completed_at_micros: current_utc_micros(),
+        })
+        .await
+        .map(|_| ())
+        .map_err(|error| error.to_string())
+}
+
+async fn fail_job(
+    state: &AppState,
+    worker_id: &str,
+    job: &ExternalGenerationJobRecord,
+    error_message: String,
+) -> Result<(), String> {
+    let now_micros = current_utc_micros();
+    state
+        .spacetime_client()
+        .fail_external_generation_job(ExternalGenerationJobFailRecordInput {
+            job_id: job.job_id.clone(),
+            worker_id: worker_id.to_string(),
+            lease_token: require_job_lease_token(job)?,
+            error_message,
+            retry_after_micros: now_micros.saturating_add(60_000_000),
+            failed_at_micros: now_micros,
+        })
+        .await
+        .map(|_| ())
+        .map_err(|error| error.to_string())
+}
+
+async fn renew_job_lease(
+    state: &AppState,
+    worker_id: &str,
+    job: &ExternalGenerationJobRecord,
+    lease: Duration,
+) -> Result<(), String> {
+    let now_micros = current_utc_micros();
+    state
+        .spacetime_client()
+        .renew_external_generation_job_lease(ExternalGenerationJobRenewLeaseRecordInput {
+            job_id: job.job_id.clone(),
+            worker_id: worker_id.to_string(),
+            lease_token: require_job_lease_token(job)?,
+            lease_expires_at_micros: now_micros.saturating_add(duration_micros_i64(lease)),
+            renewed_at_micros: now_micros,
+        })
+        .await
+        .map(|_| ())
+        .map_err(|error| error.to_string())
+}
+
+fn require_job_lease_token(job: &ExternalGenerationJobRecord) -> Result<String, String> {
+    job.lease_token
+        .as_deref()
+        .map(str::trim)
+        .filter(|value| !value.is_empty())
+        .map(ToOwned::to_owned)
+        .ok_or_else(|| format!("external_generation_job {} 缺少 lease token", job.job_id))
+}
+
+fn build_external_generation_write_lease_guard(
+    worker_id: &str,
+    job: &ExternalGenerationJobRecord,
+) -> Result<ExternalGenerationWriteLeaseGuard, String> {
+    Ok(ExternalGenerationWriteLeaseGuard::from_claimed_job(
+        job.job_id.clone(),
+        worker_id.to_string(),
+        require_job_lease_token(job)?,
+    ))
+}
+
+fn duration_micros_i64(duration: Duration) -> i64 {
+    duration.as_micros().min(i64::MAX as u128) as i64
+}
+
+fn external_generation_worker_heartbeat_interval(lease: Duration) -> Duration {
+    let heartbeat_millis = (lease.as_millis() / 3).clamp(250, 30_000) as u64;
+    Duration::from_millis(heartbeat_millis)
+}
+
+fn current_utc_micros() -> i64 {
+    offset_datetime_to_unix_micros(time::OffsetDateTime::now_utc())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn worker_write_guard_uses_claimed_job_lease_token() {
+        let job = external_generation_job_record_fixture(Some("lease-1"));
+
+        let guard = build_external_generation_write_lease_guard("worker-a", &job)
+            .expect("guard should build");
+
+        assert_eq!(guard.job_id.as_deref(), Some("extgen-1"));
+        assert_eq!(guard.worker_id.as_deref(), Some("worker-a"));
+        assert_eq!(guard.lease_token.as_deref(), Some("lease-1"));
+    }
+
+    #[test]
+    fn worker_write_guard_requires_claimed_job_lease_token() {
+        let job = external_generation_job_record_fixture(None);
+
+        let error = build_external_generation_write_lease_guard("worker-a", &job)
+            .expect_err("missing token should fail");
+
+        assert!(error.contains("缺少 lease token"));
+    }
+
+    fn external_generation_job_record_fixture(
+        lease_token: Option<&str>,
+    ) -> ExternalGenerationJobRecord {
+        ExternalGenerationJobRecord {
+            job_id: "extgen-1".to_string(),
+            dedupe_key: "puzzle:generate_puzzle_images:session-1:extgen-1".to_string(),
+            job_kind: PUZZLE_GENERATE_IMAGES_JOB_KIND.to_string(),
+            owner_user_id: "user-1".to_string(),
+            source_module: "puzzle".to_string(),
+            source_entity_id: "session-1:puzzle-level-1".to_string(),
+            request_label: "拼图关卡图片生成".to_string(),
+            request_payload_json: "{}".to_string(),
+            status: "running".to_string(),
+            attempt: 1,
+            max_attempts: 1,
+            last_error_message: None,
+            worker_id: Some("worker-a".to_string()),
+            lease_expires_at: Some("2026-06-03T00:00:00Z".to_string()),
+            available_at: "2026-06-03T00:00:00Z".to_string(),
+            result_payload_json: None,
+            created_at: "2026-06-03T00:00:00Z".to_string(),
+            started_at: Some("2026-06-03T00:00:00Z".to_string()),
+            completed_at: None,
+            updated_at: "2026-06-03T00:00:00Z".to_string(),
+            lease_token: lease_token.map(ToOwned::to_owned),
+        }
+    }
+}

server-rs/crates/api-server/src/external_generation_worker_controller.rs

@@ -0,0 +1,465 @@
+use std::{collections::BTreeSet, future::Future, io, pin::Pin, process::Stdio, time::Duration};
+
+use spacetime_client::ExternalGenerationQueueStatsRecord;
+use tokio::{
+    process::Command,
+    time::{Instant, sleep},
+};
+use tracing::{error, info, warn};
+
+use crate::state::AppState;
+
+#[derive(Clone, Debug)]
+struct ExternalGenerationWorkerControllerConfig {
+    min_workers: usize,
+    max_workers: usize,
+    target_jobs_per_worker: usize,
+    poll_interval: Duration,
+    scale_down_idle_rounds: u32,
+    service_template: String,
+    dry_run: bool,
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+struct ExternalGenerationWorkerControllerDecision {
+    desired_workers: usize,
+    should_scale_down: bool,
+    idle_rounds: u32,
+}
+
+#[derive(Debug, Default)]
+struct ExternalGenerationWorkerControllerState {
+    idle_rounds: u32,
+}
+
+pub(crate) async fn run_external_generation_worker_controller(
+    state: AppState,
+) -> Result<(), io::Error> {
+    let config = ExternalGenerationWorkerControllerConfig::from_state(&state);
+    let mut controller_state = ExternalGenerationWorkerControllerState::default();
+    let mut shutdown = external_generation_controller_shutdown_signal();
+
+    info!(
+        min_workers = config.min_workers,
+        max_workers = config.max_workers,
+        target_jobs_per_worker = config.target_jobs_per_worker,
+        poll_interval_ms = config.poll_interval.as_millis(),
+        scale_down_idle_rounds = config.scale_down_idle_rounds,
+        service_template = config.service_template,
+        dry_run = config.dry_run,
+        "external generation worker controller 已启动"
+    );
+
+    loop {
+        let tick = run_external_generation_controller_tick(&state, &config, &mut controller_state);
+        tokio::select! {
+            _ = shutdown.as_mut() => {
+                info!("external generation worker controller 收到停机信号");
+                return Ok(());
+            }
+            result = tick => {
+                if let Err(error) = result {
+                    error!(error = %error, "external generation worker controller 本轮扩缩容失败");
+                }
+            }
+        }
+
+        let next_tick = sleep(config.poll_interval);
+        tokio::pin!(next_tick);
+        tokio::select! {
+            _ = shutdown.as_mut() => {
+                info!("external generation worker controller 收到停机信号");
+                return Ok(());
+            }
+            _ = &mut next_tick => {}
+        }
+    }
+}
+
+async fn run_external_generation_controller_tick(
+    state: &AppState,
+    config: &ExternalGenerationWorkerControllerConfig,
+    controller_state: &mut ExternalGenerationWorkerControllerState,
+) -> Result<(), String> {
+    let stats = state
+        .spacetime_client()
+        .get_external_generation_queue_stats()
+        .await
+        .map_err(|error| format!("读取 external_generation_job 队列统计失败：{error}"))?;
+    let active_instances = list_active_external_generation_worker_instances(config).await?;
+    let current_workers = active_instances.len();
+    let decision = decide_external_generation_worker_target(
+        &stats,
+        current_workers,
+        controller_state.idle_rounds,
+        config,
+    );
+    controller_state.idle_rounds = decision.idle_rounds;
+
+    info!(
+        pending = stats.pending_count,
+        delayed_pending = stats.delayed_pending_count,
+        claimable = stats.claimable_count,
+        running_active = stats.running_active_count,
+        expired_running = stats.expired_running_count,
+        oldest_claimable_age_ms = stats.oldest_claimable_age_micros.unwrap_or(0) / 1_000,
+        current_workers,
+        desired_workers = decision.desired_workers,
+        idle_rounds = decision.idle_rounds,
+        "external generation worker controller 完成队列评估"
+    );
+
+    reconcile_external_generation_worker_instances(config, &active_instances, &decision).await
+}
+
+fn decide_external_generation_worker_target(
+    stats: &ExternalGenerationQueueStatsRecord,
+    current_workers: usize,
+    previous_idle_rounds: u32,
+    config: &ExternalGenerationWorkerControllerConfig,
+) -> ExternalGenerationWorkerControllerDecision {
+    let pressure = stats
+        .claimable_pending_count
+        .saturating_add(stats.running_active_count)
+        .saturating_add(stats.expired_running_count);
+    let desired_from_pressure =
+        ceil_div_usize(pressure as usize, config.target_jobs_per_worker.max(1));
+    let desired_workers = desired_from_pressure.clamp(config.min_workers, config.max_workers);
+    let is_idle = stats.claimable_count == 0
+        && stats.expired_running_count == 0
+        && stats.running_active_count == 0
+        && desired_workers <= config.min_workers;
+    let idle_rounds = if is_idle {
+        previous_idle_rounds.saturating_add(1)
+    } else {
+        0
+    };
+    let should_scale_down = current_workers > desired_workers
+        && idle_rounds >= config.scale_down_idle_rounds
+        && config.scale_down_idle_rounds > 0;
+
+    ExternalGenerationWorkerControllerDecision {
+        desired_workers,
+        should_scale_down,
+        idle_rounds,
+    }
+}
+
+async fn reconcile_external_generation_worker_instances(
+    config: &ExternalGenerationWorkerControllerConfig,
+    active_instances: &BTreeSet<usize>,
+    decision: &ExternalGenerationWorkerControllerDecision,
+) -> Result<(), String> {
+    let current_workers = active_instances.len();
+    let mut started = 0usize;
+    for instance in 1..=config.max_workers {
+        if current_workers.saturating_add(started) >= decision.desired_workers {
+            break;
+        }
+        if !active_instances.contains(&instance) {
+            systemctl_worker_instance(config, "start", instance).await?;
+            started = started.saturating_add(1);
+        }
+    }
+
+    if decision.desired_workers > current_workers && started == 0 {
+        warn!(
+            current_workers,
+            desired_workers = decision.desired_workers,
+            "external generation worker controller 未找到可启动的缺口实例"
+        );
+    }
+    if started > 0 {
+        return Ok(());
+    }
+
+    if decision.should_scale_down && decision.desired_workers < current_workers {
+        if let Some(instance) = active_instances
+            .iter()
+            .rev()
+            .copied()
+            .find(|instance| *instance > config.min_workers.max(1))
+        {
+            systemctl_worker_instance(config, "stop", instance).await?;
+        }
+    }
+
+    Ok(())
+}
+
+async fn list_active_external_generation_worker_instances(
+    config: &ExternalGenerationWorkerControllerConfig,
+) -> Result<BTreeSet<usize>, String> {
+    let mut active_instances = BTreeSet::new();
+    for instance in 1..=config.max_workers {
+        if is_external_generation_worker_instance_active(config, instance).await? {
+            active_instances.insert(instance);
+        }
+    }
+    Ok(active_instances)
+}
+
+async fn is_external_generation_worker_instance_active(
+    config: &ExternalGenerationWorkerControllerConfig,
+    instance: usize,
+) -> Result<bool, String> {
+    let service = format_worker_service_name(&config.service_template, instance)?;
+    if config.dry_run {
+        return Ok(instance <= config.min_workers);
+    }
+
+    let output = Command::new("systemctl")
+        .arg("is-active")
+        .arg("--quiet")
+        .arg(&service)
+        .stdin(Stdio::null())
+        .stdout(Stdio::null())
+        .stderr(Stdio::null())
+        .output()
+        .await
+        .map_err(|error| format!("执行 systemctl is-active {service} 失败：{error}"))?;
+    Ok(output.status.success())
+}
+
+async fn systemctl_worker_instance(
+    config: &ExternalGenerationWorkerControllerConfig,
+    action: &str,
+    instance: usize,
+) -> Result<(), String> {
+    let service = format_worker_service_name(&config.service_template, instance)?;
+    if config.dry_run {
+        info!(
+            action,
+            service, "external generation worker controller dry-run 跳过 systemctl"
+        );
+        return Ok(());
+    }
+
+    let started_at = Instant::now();
+    let output = Command::new("systemctl")
+        .arg(action)
+        .arg(&service)
+        .stdin(Stdio::null())
+        .stdout(Stdio::piped())
+        .stderr(Stdio::piped())
+        .output()
+        .await
+        .map_err(|error| format!("执行 systemctl {action} {service} 失败：{error}"))?;
+    if !output.status.success() {
+        let stderr = String::from_utf8_lossy(&output.stderr);
+        return Err(format!(
+            "systemctl {action} {service} 返回失败 status={} stderr={}",
+            output.status, stderr
+        ));
+    }
+
+    info!(
+        action,
+        service,
+        elapsed_ms = started_at.elapsed().as_millis(),
+        "external generation worker controller 已执行 systemctl"
+    );
+    Ok(())
+}
+
+fn format_worker_service_name(template: &str, instance: usize) -> Result<String, String> {
+    let instance = instance.to_string();
+    if template.contains("{}") {
+        return Ok(template.replacen("{}", &instance, 1));
+    }
+    if template.contains("%i") {
+        return Ok(template.replacen("%i", &instance, 1));
+    }
+    Err("external generation controller service template 必须包含 {} 或 %i".to_string())
+}
+
+fn ceil_div_usize(value: usize, divisor: usize) -> usize {
+    if value == 0 {
+        0
+    } else {
+        value.saturating_add(divisor.saturating_sub(1)) / divisor.max(1)
+    }
+}
+
+impl ExternalGenerationWorkerControllerConfig {
+    fn from_state(state: &AppState) -> Self {
+        let min_workers = state.config.external_generation_controller_min_workers;
+        let max_workers = state
+            .config
+            .external_generation_controller_max_workers
+            .max(min_workers);
+        Self {
+            min_workers,
+            max_workers,
+            target_jobs_per_worker: state
+                .config
+                .external_generation_controller_target_jobs_per_worker
+                .max(1),
+            poll_interval: state.config.external_generation_controller_poll_interval,
+            scale_down_idle_rounds: state
+                .config
+                .external_generation_controller_scale_down_idle_rounds,
+            service_template: state
+                .config
+                .external_generation_controller_service_template
+                .clone(),
+            dry_run: state.config.external_generation_controller_dry_run,
+        }
+    }
+}
+
+type ExternalGenerationControllerShutdownSignal = Pin<Box<dyn Future<Output = ()> + Send>>;
+
+fn external_generation_controller_shutdown_signal() -> ExternalGenerationControllerShutdownSignal {
+    Box::pin(async {
+        wait_for_external_generation_controller_shutdown_signal().await;
+    })
+}
+
+#[cfg(unix)]
+async fn wait_for_external_generation_controller_shutdown_signal() {
+    use tokio::signal::unix::{SignalKind, signal};
+
+    let mut sigterm = signal(SignalKind::terminate()).ok();
+    tokio::select! {
+        result = tokio::signal::ctrl_c() => {
+            if let Err(error) = result {
+                warn!(error = %error, "external generation worker controller 监听 SIGINT 失败");
+            }
+        }
+        _ = async {
+            if let Some(sigterm) = sigterm.as_mut() {
+                sigterm.recv().await;
+            } else {
+                std::future::pending::<()>().await;
+            }
+        } => {}
+    }
+}
+
+#[cfg(not(unix))]
+async fn wait_for_external_generation_controller_shutdown_signal() {
+    if let Err(error) = tokio::signal::ctrl_c().await {
+        warn!(error = %error, "external generation worker controller 监听 Ctrl-C 失败");
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn scales_up_to_max_when_queue_pressure_is_high() {
+        let config = controller_config_fixture();
+        let stats = stats_fixture(120, 0, 8);
+
+        let decision = decide_external_generation_worker_target(&stats, 1, 0, &config);
+
+        assert_eq!(decision.desired_workers, 8);
+        assert!(!decision.should_scale_down);
+        assert_eq!(decision.idle_rounds, 0);
+    }
+
+    #[test]
+    fn scale_down_requires_consecutive_idle_rounds() {
+        let config = controller_config_fixture();
+        let stats = stats_fixture(0, 0, 0);
+
+        let first = decide_external_generation_worker_target(&stats, 5, 0, &config);
+        let ready = decide_external_generation_worker_target(
+            &stats,
+            5,
+            config.scale_down_idle_rounds.saturating_sub(1),
+            &config,
+        );
+
+        assert_eq!(first.desired_workers, config.min_workers);
+        assert!(!first.should_scale_down);
+        assert!(ready.should_scale_down);
+    }
+
+    #[test]
+    fn running_jobs_hold_capacity_before_scale_down() {
+        let config = controller_config_fixture();
+        let stats = stats_fixture(0, 6, 0);
+
+        let decision = decide_external_generation_worker_target(&stats, 5, 5, &config);
+
+        assert_eq!(decision.desired_workers, 3);
+        assert!(!decision.should_scale_down);
+        assert_eq!(decision.idle_rounds, 0);
+    }
+
+    #[test]
+    fn expired_running_jobs_are_not_counted_twice_as_claimable_pressure() {
+        let config = controller_config_fixture();
+        let stats = stats_fixture(0, 0, 3);
+
+        let decision = decide_external_generation_worker_target(&stats, 1, 0, &config);
+
+        assert_eq!(decision.desired_workers, 2);
+        assert!(!decision.should_scale_down);
+    }
+
+    #[test]
+    fn formats_worker_service_name_with_supported_templates() {
+        assert_eq!(
+            format_worker_service_name("genarrative-external-generation-worker@{}.service", 3)
+                .expect("format"),
+            "genarrative-external-generation-worker@3.service"
+        );
+        assert_eq!(
+            format_worker_service_name("worker@%i.service", 7).expect("format"),
+            "worker@7.service"
+        );
+        assert!(format_worker_service_name("worker.service", 1).is_err());
+    }
+
+    #[tokio::test]
+    async fn dry_run_reconcile_does_not_start_low_number_gaps_when_capacity_is_enough() {
+        let config = controller_config_fixture();
+        let active_instances = BTreeSet::from([3usize, 4usize]);
+        let decision = ExternalGenerationWorkerControllerDecision {
+            desired_workers: 2,
+            should_scale_down: false,
+            idle_rounds: 0,
+        };
+
+        let result =
+            reconcile_external_generation_worker_instances(&config, &active_instances, &decision)
+                .await;
+
+        assert!(result.is_ok());
+    }
+
+    fn controller_config_fixture() -> ExternalGenerationWorkerControllerConfig {
+        ExternalGenerationWorkerControllerConfig {
+            min_workers: 1,
+            max_workers: 8,
+            target_jobs_per_worker: 2,
+            poll_interval: Duration::from_secs(10),
+            scale_down_idle_rounds: 3,
+            service_template: "genarrative-external-generation-worker@{}.service".to_string(),
+            dry_run: true,
+        }
+    }
+
+    fn stats_fixture(
+        claimable_pending_count: u32,
+        running_active_count: u32,
+        expired_running_count: u32,
+    ) -> ExternalGenerationQueueStatsRecord {
+        let claimable_count = claimable_pending_count.saturating_add(expired_running_count);
+        ExternalGenerationQueueStatsRecord {
+            pending_count: claimable_pending_count,
+            delayed_pending_count: 0,
+            claimable_pending_count,
+            running_active_count,
+            expired_running_count,
+            terminal_count: 0,
+            claimable_count,
+            oldest_claimable_age_micros: None,
+            now_micros: 0,
+        }
+    }
+}

server-rs/crates/api-server/src/main.rs

@@ -40,6 +40,8 @@ mod edutainment_baby_drawing;
 mod edutainment_baby_object;
 mod error_middleware;
 mod external_api_audit;
+mod external_generation_worker;
+mod external_generation_worker_controller;
 pub(crate) mod generated_asset_sheets;
 mod generated_image_assets;
 mod health;
@@ -89,6 +91,7 @@ mod tracking_outbox;
 mod vector_engine_audio_generation;
 mod visual_novel;
 mod volcengine_speech;
+mod wallet_refund_outbox;
 mod wechat;
 mod wooden_fish;
 mod work_author;
@@ -113,8 +116,11 @@ use tracing::{error, info, warn};
 use crate::{
     app::{build_router, build_spacetime_unavailable_router},
     config::AppConfig,
+    external_generation_worker::run_external_generation_worker,
+    external_generation_worker_controller::run_external_generation_worker_controller,
     state::{AppState, AppStateInitError},
     tracking_outbox::TrackingOutbox,
+    wallet_refund_outbox::WalletRefundOutbox,
 };
 
 const API_SERVER_STARTUP_STACK_SIZE_BYTES: usize = 32 * 1024 * 1024;
@@ -125,6 +131,7 @@ const AUTH_STORE_STARTUP_RETRY_INTERVAL: Duration = Duration::from_secs(5);
 struct ShutdownContext {
     app_state: Option<AppState>,
     tracking_outbox: Option<Arc<TrackingOutbox>>,
+    wallet_refund_outbox: Option<Arc<WalletRefundOutbox>>,
     outbox_flush_timeout: Duration,
 }
 
@@ -164,27 +171,66 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
     process_metrics::register_process_metrics();
     telemetry::register_http_runtime_metrics();
 
+    if !config.process_role.runs_http() {
+        return run_worker_only(config).await;
+    }
+
+    run_http_role(config).await
+}
+
+async fn run_worker_only(config: AppConfig) -> Result<(), io::Error> {
+    let process_role = config.process_role;
+    let state = restore_app_state_for_startup(config)
+        .await
+        .map_err(|error| {
+            io::Error::other(format!(
+                "初始化 external generation worker 状态失败：{error}"
+            ))
+        })?;
+    spawn_app_state_background_workers(&state);
+    info!(
+        process_role = process_role.as_str(),
+        "api-server 以非 HTTP 角色启动"
+    );
+    if process_role.runs_external_generation_worker() {
+        run_external_generation_worker(state).await
+    } else if process_role.runs_external_generation_controller() {
+        run_external_generation_worker_controller(state).await
+    } else {
+        Err(io::Error::other(format!(
+            "不支持的非 HTTP 进程角色：{}",
+            process_role.as_str()
+        )))
+    }
+}
+
+async fn run_http_role(config: AppConfig) -> Result<(), io::Error> {
     let bind_address = config.bind_socket_addr();
     let listen_backlog = config.listen_backlog;
     let worker_threads = config.worker_threads;
     let otel_enabled = config.otel_enabled;
+    let process_role = config.process_role;
     let outbox_flush_timeout = config.shutdown_outbox_flush_timeout;
     let listener = build_tcp_listener(bind_address, listen_backlog)?;
 
-    let (router, shutdown_context) = match restore_app_state_for_startup(config).await {
+    let (router, shutdown_context, worker_state) = match restore_app_state_for_startup(config).await
+    {
         Ok(state) => {
-            state.puzzle_gallery_cache().spawn_cleanup_task();
+            spawn_app_state_background_workers(&state);
             let tracking_outbox = state.tracking_outbox();
-            if let Some(outbox) = tracking_outbox.clone() {
-                outbox.spawn_worker();
-            }
+            let wallet_refund_outbox = state.wallet_refund_outbox();
+            let worker_state = process_role
+                .runs_external_generation_worker()
+                .then(|| state.clone());
             (
                 build_router(state.clone()),
                 ShutdownContext {
                     app_state: Some(state),
                     tracking_outbox,
+                    wallet_refund_outbox,
                     outbox_flush_timeout,
                 },
+                worker_state,
             )
         }
         Err(AppStateInitError::DependencyUnavailable(message)) => (
@@ -192,8 +238,10 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
             ShutdownContext {
                 app_state: None,
                 tracking_outbox: None,
+                wallet_refund_outbox: None,
                 outbox_flush_timeout,
             },
+            None,
         ),
         Err(error) => {
             return Err(std::io::Error::other(format!(
@@ -207,12 +255,20 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
         listen_backlog,
         worker_threads = worker_threads.unwrap_or(0),
         otel_enabled,
+        process_role = process_role.as_str(),
         "api-server 已完成 tracing 初始化并开始监听"
     );
 
-    let result = axum::serve(listener, router)
-        .with_graceful_shutdown(shutdown_signal(shutdown_context.clone()))
-        .await;
+    let http_server = axum::serve(listener, router)
+        .with_graceful_shutdown(shutdown_signal(shutdown_context.clone()));
+    let result = if let Some(worker_state) = worker_state {
+        tokio::select! {
+            result = http_server => result,
+            result = run_external_generation_worker(worker_state) => result,
+        }
+    } else {
+        http_server.await
+    };
     finalize_shutdown(shutdown_context).await;
     result
 }
@@ -271,12 +327,8 @@ async fn finalize_shutdown(context: ShutdownContext) {
         state.mark_not_ready();
     }
 
-    let Some(outbox) = context.tracking_outbox else {
-        return;
-    };
-
     if context.outbox_flush_timeout.is_zero() {
-        warn!("api-server 退出时 tracking outbox flush timeout 为 0，跳过主动 flush");
+        warn!("api-server 退出时 outbox flush timeout 为 0，跳过主动 flush");
         return;
     }
 
@@ -284,26 +336,59 @@ async fn finalize_shutdown(context: ShutdownContext) {
         .outbox_flush_timeout
         .as_millis()
         .min(u128::from(u64::MAX)) as u64;
-    info!(timeout_ms, "api-server 退出前封存并 flush tracking outbox");
-    match timeout(context.outbox_flush_timeout, outbox.flush_for_shutdown()).await {
-        Ok(Ok(())) => {
-            info!("api-server 退出前 tracking outbox flush 完成");
+    if let Some(outbox) = context.tracking_outbox {
+        info!(timeout_ms, "api-server 退出前封存并 flush tracking outbox");
+        match timeout(context.outbox_flush_timeout, outbox.flush_for_shutdown()).await {
+            Ok(Ok(())) => {
+                info!("api-server 退出前 tracking outbox flush 完成");
+            }
+            Ok(Err(error)) => {
+                warn!(
+                    error = %error,
+                    "api-server 退出前 tracking outbox flush 未完成，已保留本地文件等待下次启动重试"
+                );
+            }
+            Err(_) => {
+                warn!(
+                    timeout_ms,
+                    "api-server 退出前 tracking outbox flush 超时，已保留本地文件等待下次启动重试"
+                );
+            }
         }
-        Ok(Err(error)) => {
-            warn!(
-                error = %error,
-                "api-server 退出前 tracking outbox flush 未完成，已保留本地文件等待下次启动重试"
-            );
-        }
-        Err(_) => {
-            warn!(
-                timeout_ms,
-                "api-server 退出前 tracking outbox flush 超时，已保留本地文件等待下次启动重试"
-            );
+    }
+
+    if let Some(outbox) = context.wallet_refund_outbox {
+        info!(timeout_ms, "api-server 退出前 flush wallet refund outbox");
+        match timeout(context.outbox_flush_timeout, outbox.flush_for_shutdown()).await {
+            Ok(Ok(())) => {
+                info!("api-server 退出前 wallet refund outbox flush 完成");
+            }
+            Ok(Err(error)) => {
+                warn!(
+                    error = %error,
+                    "api-server 退出前 wallet refund outbox flush 未完成，已保留本地文件等待下次启动重试"
+                );
+            }
+            Err(_) => {
+                warn!(
+                    timeout_ms,
+                    "api-server 退出前 wallet refund outbox flush 超时，已保留本地文件等待下次启动重试"
+                );
+            }
         }
     }
 }
 
+fn spawn_app_state_background_workers(state: &AppState) {
+    state.puzzle_gallery_cache().spawn_cleanup_task();
+    if let Some(outbox) = state.tracking_outbox() {
+        outbox.spawn_worker();
+    }
+    if let Some(outbox) = state.wallet_refund_outbox() {
+        outbox.spawn_worker();
+    }
+}
+
 fn build_tcp_listener(
     bind_address: SocketAddr,
     listen_backlog: i32,

server-rs/crates/api-server/src/match3d.rs

@@ -1,4 +1,4 @@
-use std::{
+use std::{
     collections::BTreeMap,
     convert::Infallible,
     future::Future,
@@ -65,11 +65,8 @@ use spacetime_client::{
 
 use crate::{
     api_response::json_success_body,
-    asset_billing::{
-        execute_billable_asset_operation_with_cost, map_asset_operation_wallet_error,
-        should_skip_asset_operation_billing_for_connectivity,
-    },
-    auth::AuthenticatedAccessToken,
+    asset_billing::{execute_billable_asset_operation_with_cost, map_asset_operation_wallet_error},
+    auth::{AuthenticatedAccessToken, RuntimePrincipal},
     config::AppConfig,
     generated_asset_sheets::apply_generated_asset_sheet_green_screen_alpha,
     http_error::AppError,
@@ -354,13 +351,6 @@ impl Match3DItemAssetsGenerationPlan {
             Self::Replace(plan) => plan.requested_item_names.len(),
         }
     }
-
-    fn billing_fingerprint_source(&self) -> String {
-        match self {
-            Self::Append(plan) => format!("append:{}", plan.requested_item_names.join("|")),
-            Self::Replace(plan) => format!("replace:{}", plan.requested_item_names.join("|")),
-        }
-    }
 }
 
 fn serialize_match3d_generated_item_assets(assets: &[Match3DGeneratedItemAsset]) -> Option<String> {

server-rs/crates/api-server/src/match3d/draft.rs

@@ -162,7 +162,12 @@ pub(super) async fn compile_match3d_draft_for_session(
     let initial_tags = requested_tags
         .clone()
         .unwrap_or_else(|| fallback_work_metadata.tags.clone());
-    let billing_asset_id = format!("{}:{}:{}", session_id, profile_id, current_utc_micros());
+    let billing_asset_id = format!(
+        "{}:{}:{}",
+        session_id,
+        profile_id,
+        request_context.request_id()
+    );
     let points_cost = crate::creation_entry_config::resolve_creation_entry_mud_point_cost(
         state,
         "match3d",
@@ -514,15 +519,6 @@ async fn consume_match3d_draft_generation_points(
         .await
     {
         Ok(_) => Ok(true),
-        Err(error) if should_skip_asset_operation_billing_for_connectivity(&error) => {
-            tracing::warn!(
-                owner_user_id,
-                billing_asset_id,
-                error = %error,
-                "抓大鹅草稿泥点预扣因 SpacetimeDB 连接不可用而降级跳过"
-            );
-            Ok(false)
-        }
         Err(error) => Err(match3d_error_response(
             request_context,
             MATCH3D_AGENT_PROVIDER,

server-rs/crates/api-server/src/match3d/handlers.rs

@@ -751,7 +751,6 @@ pub async fn generate_match3d_background_image_for_work(
     )?;
     let prompt = normalize_match3d_background_prompt(payload.prompt.as_str());
     ensure_non_empty(&request_context, MATCH3D_WORKS_PROVIDER, &prompt, "prompt")?;
-    let prompt_fingerprint = build_match3d_prompt_fingerprint(prompt.as_str());
 
     let context =
         load_match3d_work_asset_context(&state, &request_context, &authenticated, &profile_id)
@@ -763,7 +762,12 @@ pub async fn generate_match3d_background_image_for_work(
         config,
         assets,
     } = context;
-    let billing_asset_id = format!("{}:{}:{}", session_id, profile_id, prompt_fingerprint);
+    let billing_asset_id = format!(
+        "{}:{}:{}",
+        session_id,
+        profile_id,
+        request_context.request_id()
+    );
     let (generated_background, generated_assets) = execute_billable_asset_operation_with_cost(
         &state,
         owner_user_id.as_str(),
@@ -860,7 +864,6 @@ pub async fn generate_match3d_container_image_for_work(
     )?;
     let prompt = normalize_match3d_background_prompt(payload.prompt.as_str());
     ensure_non_empty(&request_context, MATCH3D_WORKS_PROVIDER, &prompt, "prompt")?;
-    let prompt_fingerprint = build_match3d_prompt_fingerprint(prompt.as_str());
 
     let context =
         load_match3d_work_asset_context(&state, &request_context, &authenticated, &profile_id)
@@ -874,7 +877,9 @@ pub async fn generate_match3d_container_image_for_work(
     } = context;
     let billing_asset_id = format!(
         "{}:{}:{}:container",
-        session_id, profile_id, prompt_fingerprint
+        session_id,
+        profile_id,
+        request_context.request_id()
     );
     let (generated_background, generated_assets) = execute_billable_asset_operation_with_cost(
         &state,
@@ -1017,7 +1022,7 @@ pub async fn generate_match3d_item_assets_for_work(
         session_id,
         profile_id,
         billed_item_count,
-        build_match3d_prompt_fingerprint(generation_plan.billing_fingerprint_source().as_str())
+        request_context.request_id()
     );
     let generated_assets = execute_billable_asset_operation_with_cost(
         &state,
@@ -1171,7 +1176,7 @@ pub async fn start_match3d_run(
     State(state): State<AppState>,
     Path(profile_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<StartMatch3DRunRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let maybe_payload = payload.ok().map(|Json(payload)| payload);
@@ -1191,7 +1196,7 @@ pub async fn start_match3d_run(
         .spacetime_client()
         .start_match3d_run(Match3DRunStartRecordInput {
             run_id: build_prefixed_uuid_id(MATCH3D_RUN_ID_PREFIX),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             profile_id: profile_id.clone(),
             started_at_ms: current_utc_ms(),
             item_type_count_override: maybe_payload
@@ -1211,15 +1216,17 @@ pub async fn start_match3d_run(
     record_work_play_start_after_success(
         &state,
         &request_context,
-        WorkPlayTrackingDraft::new(
+        WorkPlayTrackingDraft::runtime_principal(
             "match3d",
             profile_id.clone(),
-            &authenticated,
+            &principal,
             "/api/runtime/match3d/...",
         )
         .profile_id(profile_id.clone())
+        .owner_user_id(principal.subject().to_string())
         .extra(json!({
             "runId": run.run_id,
+            "principalKind": principal.kind().as_str(),
         })),
     )
     .await;
@@ -1236,13 +1243,13 @@ pub async fn get_match3d_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, MATCH3D_RUNTIME_PROVIDER, &run_id, "runId")?;
 
     let run = state
         .spacetime_client()
-        .get_match3d_run(run_id, authenticated.claims().user_id().to_string())
+        .get_match3d_run(run_id, principal.subject().to_string())
         .await
         .map_err(|error| {
             match3d_error_response(
@@ -1264,7 +1271,7 @@ pub async fn click_match3d_item(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<ClickMatch3DItemRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = match3d_json(payload, &request_context, MATCH3D_RUNTIME_PROVIDER)?;
@@ -1286,7 +1293,7 @@ pub async fn click_match3d_item(
         .spacetime_client()
         .click_match3d_item(Match3DRunClickRecordInput {
             run_id: payload.run_id.unwrap_or(run_id),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             item_instance_id: payload.item_instance_id,
             client_snapshot_version: payload.client_snapshot_version.min(u32::MAX as u64) as u32,
             client_event_id: payload.client_event_id,
@@ -1313,7 +1320,7 @@ pub async fn stop_match3d_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<StopMatch3DRunRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let _ = payload.ok();
@@ -1323,7 +1330,7 @@ pub async fn stop_match3d_run(
         .spacetime_client()
         .stop_match3d_run(Match3DRunStopRecordInput {
             run_id,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             stopped_at_ms: current_utc_ms(),
         })
         .await
@@ -1347,7 +1354,7 @@ pub async fn restart_match3d_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, MATCH3D_RUNTIME_PROVIDER, &run_id, "runId")?;
 
@@ -1356,7 +1363,7 @@ pub async fn restart_match3d_run(
         .restart_match3d_run(Match3DRunRestartRecordInput {
             source_run_id: run_id,
             next_run_id: build_prefixed_uuid_id(MATCH3D_RUN_ID_PREFIX),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             restarted_at_ms: current_utc_ms(),
         })
         .await
@@ -1380,7 +1387,7 @@ pub async fn finish_match3d_time_up(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&request_context, MATCH3D_RUNTIME_PROVIDER, &run_id, "runId")?;
 
@@ -1388,7 +1395,7 @@ pub async fn finish_match3d_time_up(
         .spacetime_client()
         .finish_match3d_time_up(Match3DRunTimeUpRecordInput {
             run_id,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             finished_at_ms: current_utc_ms(),
         })
         .await

server-rs/crates/api-server/src/match3d/item_assets.rs

@@ -1208,14 +1208,6 @@ pub(super) fn normalize_match3d_background_prompt(raw: &str) -> String {
         .to_string()
 }
 
-pub(super) fn build_match3d_prompt_fingerprint(value: &str) -> String {
-    let mut hash = 0u32;
-    for character in value.chars() {
-        hash = hash.wrapping_mul(31).wrapping_add(character as u32);
-    }
-    format!("{hash:08x}")
-}
-
 pub(super) fn build_fallback_match3d_background_prompt(config: &Match3DConfigJson) -> String {
     let theme = config.theme_text.trim();
     let normalized_theme = if theme.is_empty() { "抓大鹅" } else { theme };

server-rs/crates/api-server/src/modules/bark_battle.rs

@@ -4,7 +4,7 @@ use axum::{
 };
 
 use crate::{
-    auth::require_bearer_auth,
+    auth::{require_bearer_auth, require_runtime_principal_auth},
     bark_battle::{
         create_bark_battle_draft, delete_bark_battle_work, finish_bark_battle_run,
         generate_bark_battle_image_asset, get_bark_battle_run, get_bark_battle_runtime_config,
@@ -66,26 +66,28 @@ pub fn router(state: AppState) -> Router<AppState> {
             "/api/runtime/bark-battle/works/{work_id}/config",
             get(get_bark_battle_runtime_config).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/bark-battle/works/{work_id}/runs",
             post(start_bark_battle_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/bark-battle/runs/{run_id}",
             get(get_bark_battle_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/bark-battle/runs/{run_id}/finish",
-            post(finish_bark_battle_run)
-                .route_layer(middleware::from_fn_with_state(state, require_bearer_auth)),
+            post(finish_bark_battle_run).route_layer(middleware::from_fn_with_state(
+                state,
+                require_runtime_principal_auth,
+            )),
         )
 }

server-rs/crates/api-server/src/modules/big_fish.rs

@@ -4,7 +4,7 @@ use axum::{
 };
 
 use crate::{
-    auth::require_bearer_auth,
+    auth::{require_bearer_auth, require_runtime_principal_auth},
     big_fish::{
         create_big_fish_session, delete_big_fish_work, execute_big_fish_action, get_big_fish_run,
         get_big_fish_session, get_big_fish_works, list_big_fish_gallery,
@@ -85,35 +85,35 @@ pub fn router(state: AppState) -> Router<AppState> {
             "/api/runtime/big-fish/sessions/{session_id}/play",
             post(record_big_fish_play).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/big-fish/works/{session_id}/play",
             post(record_big_fish_play).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/big-fish/sessions/{session_id}/runs",
             post(start_big_fish_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/big-fish/runs/{run_id}",
             get(get_big_fish_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/big-fish/runs/{run_id}/input",
             post(submit_big_fish_input).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
 }

server-rs/crates/api-server/src/modules/match3d.rs

@@ -4,7 +4,7 @@ use axum::{
 };
 
 use crate::{
-    auth::require_bearer_auth,
+    auth::{require_bearer_auth, require_runtime_principal_auth},
     match3d::{
         click_match3d_item, compile_match3d_agent_draft, create_match3d_agent_session,
         delete_match3d_work, execute_match3d_agent_action, finish_match3d_time_up,
@@ -139,42 +139,42 @@ pub fn router(state: AppState) -> Router<AppState> {
             "/api/runtime/match3d/works/{profile_id}/runs",
             post(start_match3d_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/match3d/runs/{run_id}",
             get(get_match3d_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/match3d/runs/{run_id}/click",
             post(click_match3d_item).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/match3d/runs/{run_id}/stop",
             post(stop_match3d_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/match3d/runs/{run_id}/restart",
             post(restart_match3d_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/match3d/runs/{run_id}/time-up",
             post(finish_match3d_time_up).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
 }

server-rs/crates/api-server/src/modules/square_hole.rs

@@ -4,7 +4,7 @@ use axum::{
 };
 
 use crate::{
-    auth::require_bearer_auth,
+    auth::{require_bearer_auth, require_runtime_principal_auth},
     square_hole::{
         compile_square_hole_agent_draft, create_square_hole_agent_session, delete_square_hole_work,
         drop_square_hole_shape, execute_square_hole_agent_action, finish_square_hole_time_up,
@@ -101,42 +101,42 @@ pub fn router(state: AppState) -> Router<AppState> {
             "/api/runtime/square-hole/works/{profile_id}/runs",
             post(start_square_hole_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/square-hole/runs/{run_id}",
             get(get_square_hole_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/square-hole/runs/{run_id}/drop",
             post(drop_square_hole_shape).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/square-hole/runs/{run_id}/stop",
             post(stop_square_hole_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/square-hole/runs/{run_id}/restart",
             post(restart_square_hole_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/square-hole/runs/{run_id}/time-up",
             post(finish_square_hole_time_up).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
 }

server-rs/crates/api-server/src/modules/visual_novel.rs

@@ -4,7 +4,7 @@ use axum::{
 };
 
 use crate::{
-    auth::require_bearer_auth,
+    auth::{require_bearer_auth, require_runtime_principal_auth},
     state::AppState,
     vector_engine_audio_generation::{
         create_background_music_task, create_sound_effect_task,
@@ -151,33 +151,35 @@ pub fn router(state: AppState) -> Router<AppState> {
             "/api/runtime/visual-novel/works/{profile_id}/runs",
             post(start_visual_novel_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/visual-novel/runs/{run_id}",
             get(get_visual_novel_run).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/visual-novel/runs/{run_id}/actions/stream",
             post(stream_visual_novel_action).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/visual-novel/runs/{run_id}/history",
             get(list_visual_novel_history).route_layer(middleware::from_fn_with_state(
                 state.clone(),
-                require_bearer_auth,
+                require_runtime_principal_auth,
             )),
         )
         .route(
             "/api/runtime/visual-novel/runs/{run_id}/regenerate",
-            post(regenerate_visual_novel_run)
-                .route_layer(middleware::from_fn_with_state(state, require_bearer_auth)),
+            post(regenerate_visual_novel_run).route_layer(middleware::from_fn_with_state(
+                state,
+                require_runtime_principal_auth,
+            )),
         )
 }

server-rs/crates/api-server/src/puzzle.rs

@@ -1,6 +1,5 @@
 use std::{
-    collections::{BTreeMap, HashSet},
-    sync::{Mutex, OnceLock},
+    collections::BTreeMap,
     time::{Instant, SystemTime, UNIX_EPOCH},
 };
 
@@ -53,16 +52,18 @@ use shared_contracts::{
 };
 use shared_kernel::{build_prefixed_uuid_id, format_timestamp_micros};
 use spacetime_client::{
-    PuzzleAgentMessageRecord, PuzzleAgentMessageSubmitRecordInput,
-    PuzzleAgentSessionCreateRecordInput, PuzzleAgentSessionRecord,
-    PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord, PuzzleAnchorPackRecord,
-    PuzzleAudioAssetRecord, PuzzleCreatorIntentRecord, PuzzleDraftCompileFailureRecordInput,
-    PuzzleDraftLevelRecord, PuzzleFormDraftRecord, PuzzleFormDraftSaveRecordInput,
-    PuzzleGeneratedImageCandidateRecord, PuzzleGeneratedImagesSaveRecordInput,
-    PuzzleLeaderboardEntryRecord, PuzzleLeaderboardSubmitRecordInput, PuzzlePublishRecordInput,
-    PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord, PuzzleResultPreviewBlockerRecord,
-    PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord, PuzzleRunDragRecordInput,
-    PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput, PuzzleRunRecord,
+    ExternalGenerationJobEnqueueRecordInput, PuzzleAgentMessageRecord,
+    PuzzleAgentMessageSubmitRecordInput, PuzzleAgentSessionCreateRecordInput,
+    PuzzleAgentSessionRecord, PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord,
+    PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBackgroundCompileTaskClaimRecordInput,
+    PuzzleBackgroundCompileTaskReleaseRecordInput, PuzzleCreatorIntentRecord,
+    PuzzleDraftCompileFailureRecordInput, PuzzleDraftLevelRecord, PuzzleFormDraftRecord,
+    PuzzleFormDraftSaveRecordInput, PuzzleGeneratedImageCandidateRecord,
+    PuzzleGeneratedImagesSaveRecordInput, PuzzleLeaderboardEntryRecord,
+    PuzzleLeaderboardSubmitRecordInput, PuzzleLevelGenerationFailureRecordInput,
+    PuzzlePublishRecordInput, PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord,
+    PuzzleResultPreviewBlockerRecord, PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord,
+    PuzzleRunDragRecordInput, PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput, PuzzleRunRecord,
     PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput, PuzzleSelectCoverImageRecordInput,
     PuzzleUiBackgroundSaveRecordInput, PuzzleWorkLikeReportRecordInput,
     PuzzleWorkPointIncentiveClaimRecordInput, PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput,
@@ -78,6 +79,10 @@ use crate::{
         should_skip_asset_operation_billing_for_connectivity,
     },
     auth::{AuthenticatedAccessToken, RuntimePrincipal},
+    external_generation_worker::{
+        PUZZLE_COMPILE_DRAFT_JOB_KIND, PUZZLE_GENERATE_IMAGES_JOB_KIND,
+        PUZZLE_GENERATE_UI_BACKGROUND_JOB_KIND,
+    },
     generated_asset_sheets::apply_generated_asset_sheet_green_screen_alpha,
     http_error::AppError,
     llm_model_routing::{CREATION_TEMPLATE_LLM_MODEL, PUZZLE_LEVEL_NAME_VISION_LLM_MODEL},
@@ -134,43 +139,75 @@ const PUZZLE_UI_BACKGROUND_PROMPT_FALLBACK_MARKER: &str =
 const PUZZLE_VECTOR_ENGINE_SQUARE_IMAGE_SIZE: &str = "1024x1024";
 const PUZZLE_VECTOR_ENGINE_PORTRAIT_IMAGE_SIZE: &str = "1024x1536";
 
-static PUZZLE_BACKGROUND_COMPILE_TASKS: OnceLock<Mutex<HashSet<String>>> = OnceLock::new();
-
-fn puzzle_background_compile_tasks() -> &'static Mutex<HashSet<String>> {
-    PUZZLE_BACKGROUND_COMPILE_TASKS.get_or_init(|| Mutex::new(HashSet::new()))
+fn build_puzzle_background_compile_task_id(session_id: &str) -> String {
+    format!("puzzle_initial_background:{session_id}")
 }
 
-fn try_register_puzzle_background_compile_task(session_id: &str) -> bool {
-    match puzzle_background_compile_tasks().lock() {
-        Ok(mut tasks) => tasks.insert(session_id.to_string()),
-        Err(error) => {
+fn build_puzzle_background_compile_claim_id(task_id: &str, request_id: &str) -> String {
+    format!("{task_id}:{request_id}")
+}
+
+async fn release_claimed_puzzle_background_compile_task(
+    state: &PuzzleApiState,
+    task_id: &str,
+    claim_id: &str,
+    session_id: &str,
+    owner_user_id: &str,
+) {
+    let result = state
+        .spacetime_client()
+        .release_puzzle_background_compile_task(PuzzleBackgroundCompileTaskReleaseRecordInput {
+            task_id: task_id.to_string(),
+            claim_id: claim_id.to_string(),
+            session_id: session_id.to_string(),
+            owner_user_id: owner_user_id.to_string(),
+        })
+        .await;
+    match result {
+        Ok(true) => {}
+        Ok(false) => {
             tracing::warn!(
                 provider = PUZZLE_AGENT_API_BASE_PROVIDER,
+                task_id,
+                claim_id,
                 session_id,
-                error = %error,
-                "拼图后台生成任务注册表锁已损坏，允许本次任务继续"
+                owner_user_id,
+                "拼图首图后台生成任务释放未命中当前 claim"
             );
-            true
-        }
-    }
-}
-
-fn unregister_puzzle_background_compile_task(session_id: &str) {
-    match puzzle_background_compile_tasks().lock() {
-        Ok(mut tasks) => {
-            tasks.remove(session_id);
         }
         Err(error) => {
             tracing::warn!(
                 provider = PUZZLE_AGENT_API_BASE_PROVIDER,
+                task_id,
+                claim_id,
                 session_id,
+                owner_user_id,
                 error = %error,
-                "拼图后台生成任务注册表解锁失败，忽略清理"
+                "拼图首图后台生成任务释放失败"
             );
         }
     }
 }
 
+pub(crate) fn spawn_release_claimed_puzzle_background_compile_task(
+    state: PuzzleApiState,
+    task_id: String,
+    claim_id: String,
+    session_id: String,
+    owner_user_id: String,
+) {
+    tokio::spawn(async move {
+        release_claimed_puzzle_background_compile_task(
+            &state,
+            &task_id,
+            &claim_id,
+            &session_id,
+            &owner_user_id,
+        )
+        .await;
+    });
+}
+
 fn has_puzzle_cover_image_src(value: &Option<String>) -> bool {
     value
         .as_deref()
@@ -201,6 +238,65 @@ fn mark_puzzle_initial_generation_started_snapshot(
     session
 }
 
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub(crate) struct ExternalGenerationWriteLeaseGuard {
+    pub(crate) job_id: Option<String>,
+    pub(crate) worker_id: Option<String>,
+    pub(crate) lease_token: Option<String>,
+}
+
+impl ExternalGenerationWriteLeaseGuard {
+    pub(crate) fn inline() -> Self {
+        Self {
+            job_id: None,
+            worker_id: None,
+            lease_token: None,
+        }
+    }
+
+    pub(crate) fn from_claimed_job(job_id: String, worker_id: String, lease_token: String) -> Self {
+        Self {
+            job_id: Some(job_id),
+            worker_id: Some(worker_id),
+            lease_token: Some(lease_token),
+        }
+    }
+}
+
+#[derive(Debug)]
+pub(crate) struct PuzzleExternalGenerationWorkerError {
+    error: AppError,
+    should_fail_queue_job: bool,
+}
+
+impl PuzzleExternalGenerationWorkerError {
+    pub(crate) fn with_failure_state_written(error: AppError) -> Self {
+        Self {
+            error,
+            should_fail_queue_job: true,
+        }
+    }
+
+    pub(crate) fn with_failure_state_pending(error: AppError) -> Self {
+        Self {
+            error,
+            should_fail_queue_job: false,
+        }
+    }
+
+    pub(crate) fn body_text(&self) -> String {
+        self.error.body_text()
+    }
+
+    pub(crate) fn into_app_error(self) -> AppError {
+        self.error
+    }
+
+    pub(crate) fn should_fail_queue_job(&self) -> bool {
+        self.should_fail_queue_job
+    }
+}
+
 pub(crate) fn format_puzzle_reference_image_upload_bytes(bytes: usize) -> String {
     format!("{:.1}MB", bytes as f64 / 1024.0 / 1024.0)
 }
@@ -223,7 +319,7 @@ mod mappers;
 use self::mappers::*;
 
 mod draft;
-use self::draft::*;
+pub(crate) use self::draft::*;
 
 mod tags;
 
@@ -232,7 +328,7 @@ use self::tags::*;
 mod generation;
 mod vector_engine;
 
-use self::generation::*;
+pub(crate) use self::generation::*;
 use self::vector_engine::*;
 
 #[cfg(test)]

server-rs/crates/api-server/src/puzzle/draft.rs

@@ -137,6 +137,213 @@ pub(crate) async fn create_seeded_puzzle_session_when_form_save_missing(
     Ok(replacement.session_id)
 }
 
+fn default_puzzle_image_generation_points_cost() -> u64 {
+    PUZZLE_IMAGE_GENERATION_POINTS_COST
+}
+
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct PuzzleCompileDraftWorkerPayload {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub billing_asset_id: String,
+    pub ai_redraw: bool,
+    #[serde(default = "default_puzzle_image_generation_points_cost")]
+    pub billing_points_cost: u64,
+    #[serde(default)]
+    pub prompt_text: Option<String>,
+    #[serde(default)]
+    pub reference_image_src: Option<String>,
+    #[serde(default)]
+    pub image_model: Option<String>,
+    pub requested_at_micros: i64,
+    #[serde(default)]
+    pub background_task_id: Option<String>,
+    #[serde(default)]
+    pub background_claim_id: Option<String>,
+}
+
+pub(crate) async fn execute_puzzle_compile_draft_worker_job(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    payload: PuzzleCompileDraftWorkerPayload,
+    external_generation_guard: ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, PuzzleExternalGenerationWorkerError> {
+    let now = current_utc_micros();
+    let session = if payload.ai_redraw {
+        execute_billable_asset_operation_with_cost(
+            state.root_state(),
+            &payload.owner_user_id,
+            "puzzle_initial_image",
+            &payload.billing_asset_id,
+            payload.billing_points_cost,
+            async {
+                compile_puzzle_draft_with_initial_cover(
+                    state,
+                    request_context,
+                    payload.session_id.clone(),
+                    payload.owner_user_id.clone(),
+                    payload.prompt_text.as_deref(),
+                    payload.reference_image_src.as_deref(),
+                    payload.image_model.as_deref(),
+                    now,
+                    &external_generation_guard,
+                )
+                .await
+            },
+        )
+        .await
+    } else {
+        compile_puzzle_draft_with_uploaded_cover(
+            state,
+            request_context,
+            payload.session_id.clone(),
+            payload.owner_user_id.clone(),
+            payload.prompt_text.as_deref(),
+            payload.reference_image_src.as_deref(),
+            now,
+            &external_generation_guard,
+        )
+        .await
+    };
+
+    match session {
+        Ok(session) => {
+            if session
+                .draft
+                .as_ref()
+                .is_some_and(|draft| draft.generation_status == "ready")
+            {
+                send_generation_result_subscribe_message_after_completion(
+                    state.root_state(),
+                    GenerationResultSubscribeMessage {
+                        owner_user_id: payload.owner_user_id.clone(),
+                        task_name: Some("拼图".to_string()),
+                        work_name: session.draft.as_ref().map(|draft| draft.work_title.clone()),
+                        status: GenerationResultSubscribeMessageStatus::Succeeded,
+                        consumed_points: if payload.ai_redraw {
+                            payload.billing_points_cost
+                        } else {
+                            0
+                        },
+                        completed_at_micros: current_utc_micros(),
+                        page: Some("/pages/web-view/index".to_string()),
+                    },
+                )
+                .await;
+            }
+            release_inline_puzzle_compile_background_claim(
+                state,
+                &payload,
+                &external_generation_guard,
+            );
+            Ok(session)
+        }
+        Err(error) => {
+            match mark_puzzle_compile_failure_for_worker(
+                state,
+                &payload.session_id,
+                &payload.owner_user_id,
+                error.body_text(),
+                now,
+                &external_generation_guard,
+            )
+            .await
+            {
+                Ok(()) => {
+                    send_generation_result_subscribe_message_after_completion(
+                        state.root_state(),
+                        GenerationResultSubscribeMessage {
+                            owner_user_id: payload.owner_user_id.clone(),
+                            task_name: Some("拼图".to_string()),
+                            work_name: None,
+                            status: GenerationResultSubscribeMessageStatus::Failed,
+                            consumed_points: 0,
+                            completed_at_micros: now,
+                            page: Some("/pages/web-view/index".to_string()),
+                        },
+                    )
+                    .await;
+                    release_inline_puzzle_compile_background_claim(
+                        state,
+                        &payload,
+                        &external_generation_guard,
+                    );
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_written(error))
+                }
+                Err(mark_error) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_pending(mark_error))
+                }
+            }
+        }
+    }
+}
+
+fn release_inline_puzzle_compile_background_claim(
+    state: &PuzzleApiState,
+    payload: &PuzzleCompileDraftWorkerPayload,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) {
+    if external_generation_guard.job_id.is_some() {
+        return;
+    }
+    release_puzzle_compile_background_claim(state, payload);
+}
+
+pub(crate) fn release_puzzle_compile_background_claim(
+    state: &PuzzleApiState,
+    payload: &PuzzleCompileDraftWorkerPayload,
+) {
+    let (Some(task_id), Some(claim_id)) = (
+        payload.background_task_id.as_ref(),
+        payload.background_claim_id.as_ref(),
+    ) else {
+        return;
+    };
+
+    spawn_release_claimed_puzzle_background_compile_task(
+        state.clone(),
+        task_id.clone(),
+        claim_id.clone(),
+        payload.session_id.clone(),
+        payload.owner_user_id.clone(),
+    );
+}
+
+pub(crate) async fn mark_puzzle_compile_failure_for_worker(
+    state: &PuzzleApiState,
+    session_id: &str,
+    owner_user_id: &str,
+    error_message: String,
+    failed_at_micros: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<(), AppError> {
+    let result = state
+        .spacetime_client()
+        .mark_puzzle_draft_generation_failed(PuzzleDraftCompileFailureRecordInput {
+            session_id: session_id.to_string(),
+            owner_user_id: owner_user_id.to_string(),
+            error_message,
+            failed_at_micros,
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
+        })
+        .await;
+    if let Err(error) = result {
+        tracing::warn!(
+            provider = PUZZLE_AGENT_API_BASE_PROVIDER,
+            session_id,
+            owner_user_id,
+            message = %error,
+            "拼图 worker 草稿失败态回写失败"
+        );
+        return Err(map_puzzle_client_error(error));
+    }
+
+    Ok(())
+}
+
 pub(crate) fn select_puzzle_level_for_api(
     draft: &PuzzleResultDraftRecord,
     level_id: Option<&str>,
@@ -1163,6 +1370,44 @@ pub(crate) fn find_puzzle_level_for_initial_asset_check<'a>(
         .or_else(|| levels.first())
 }
 
+pub(crate) async fn compile_puzzle_draft_with_initial_cover(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    session_id: String,
+    owner_user_id: String,
+    prompt_text: Option<&str>,
+    reference_image_src: Option<&str>,
+    image_model: Option<&str>,
+    now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, AppError> {
+    let compiled_session = state
+        .spacetime_client()
+        .compile_puzzle_agent_draft_with_external_generation_guard(
+            session_id,
+            owner_user_id.clone(),
+            now,
+            external_generation_guard.job_id.clone(),
+            external_generation_guard.worker_id.clone(),
+            external_generation_guard.lease_token.clone(),
+        )
+        .await
+        .map_err(map_puzzle_compile_error)?;
+
+    generate_puzzle_initial_cover_from_compiled_session(
+        state,
+        request_context,
+        compiled_session,
+        owner_user_id,
+        prompt_text,
+        reference_image_src,
+        image_model,
+        now,
+        external_generation_guard,
+    )
+    .await
+}
+
 pub(crate) async fn generate_puzzle_initial_cover_from_compiled_session(
     state: &PuzzleApiState,
     request_context: &RequestContext,
@@ -1172,6 +1417,7 @@ pub(crate) async fn generate_puzzle_initial_cover_from_compiled_session(
     reference_image_src: Option<&str>,
     image_model: Option<&str>,
     now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
 ) -> Result<PuzzleAgentSessionRecord, AppError> {
     let draft = compiled_session.draft.clone().ok_or_else(|| {
         AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
@@ -1322,6 +1568,9 @@ pub(crate) async fn generate_puzzle_initial_cover_from_compiled_session(
             levels_json: levels_json_with_generated_name.clone(),
             candidates_json,
             saved_at_micros: current_utc_micros(),
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
         })
         .await
         .map_err(map_puzzle_client_error)
@@ -1435,6 +1684,7 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
     prompt_text: Option<&str>,
     reference_image_src: Option<&str>,
     now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
 ) -> Result<PuzzleAgentSessionRecord, AppError> {
     let uploaded_image_src = reference_image_src
         .map(str::trim)
@@ -1469,7 +1719,14 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
             })?;
     let compiled_session = state
         .spacetime_client()
-        .compile_puzzle_agent_draft(session_id.clone(), owner_user_id.clone(), now)
+        .compile_puzzle_agent_draft_with_external_generation_guard(
+            session_id.clone(),
+            owner_user_id.clone(),
+            now,
+            external_generation_guard.job_id.clone(),
+            external_generation_guard.worker_id.clone(),
+            external_generation_guard.lease_token.clone(),
+        )
         .await
         .map_err(map_puzzle_compile_error)?;
     let draft = compiled_session.draft.clone().ok_or_else(|| {
@@ -1618,6 +1875,9 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
             levels_json: levels_json_with_generated_name.clone(),
             candidates_json,
             saved_at_micros: current_utc_micros(),
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
         })
         .await
         .map_err(map_puzzle_client_error)

server-rs/crates/api-server/src/puzzle/generation.rs

@@ -22,6 +22,510 @@ pub(crate) fn should_use_uploaded_puzzle_image_directly(
             .is_some_and(|value| !value.is_empty())
 }
 
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct PuzzleGenerateImagesWorkerPayload {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub billing_asset_id: String,
+    #[serde(default)]
+    pub level_id: Option<String>,
+    #[serde(default)]
+    pub prompt_text: Option<String>,
+    #[serde(default)]
+    pub reference_image_src: Option<String>,
+    #[serde(default)]
+    pub reference_image_srcs: Vec<String>,
+    #[serde(default)]
+    pub reference_image_asset_object_id: Option<String>,
+    #[serde(default)]
+    pub reference_image_asset_object_ids: Vec<String>,
+    #[serde(default)]
+    pub image_model: Option<String>,
+    #[serde(default)]
+    pub ai_redraw: Option<bool>,
+    #[serde(default)]
+    pub should_auto_name_level: Option<bool>,
+    #[serde(default)]
+    pub work_title: Option<String>,
+    #[serde(default)]
+    pub work_description: Option<String>,
+    #[serde(default)]
+    pub picture_description: Option<String>,
+    #[serde(default)]
+    pub summary: Option<String>,
+    #[serde(default)]
+    pub theme_tags: Option<Vec<String>>,
+    #[serde(default)]
+    pub levels_json: Option<String>,
+    pub requested_at_micros: i64,
+}
+
+impl PuzzleGenerateImagesWorkerPayload {
+    fn to_action_request(&self) -> ExecutePuzzleAgentActionRequest {
+        ExecutePuzzleAgentActionRequest {
+            action: "generate_puzzle_images".to_string(),
+            prompt_text: self.prompt_text.clone(),
+            reference_image_src: self.reference_image_src.clone(),
+            reference_image_srcs: self.reference_image_srcs.clone(),
+            reference_image_asset_object_id: self.reference_image_asset_object_id.clone(),
+            reference_image_asset_object_ids: self.reference_image_asset_object_ids.clone(),
+            image_model: self.image_model.clone(),
+            ai_redraw: self.ai_redraw,
+            candidate_count: Some(1),
+            should_auto_name_level: self.should_auto_name_level,
+            candidate_id: None,
+            level_id: self.level_id.clone(),
+            work_title: self.work_title.clone(),
+            work_description: self.work_description.clone(),
+            picture_description: self.picture_description.clone(),
+            level_name: None,
+            summary: self.summary.clone(),
+            theme_tags: self.theme_tags.clone(),
+            levels_json: self.levels_json.clone(),
+        }
+    }
+}
+
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct PuzzleGenerateUiBackgroundWorkerPayload {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub billing_asset_id: String,
+    #[serde(default)]
+    pub level_id: Option<String>,
+    #[serde(default)]
+    pub prompt_text: Option<String>,
+    #[serde(default)]
+    pub levels_json: Option<String>,
+    pub requested_at_micros: i64,
+}
+
+impl PuzzleGenerateUiBackgroundWorkerPayload {
+    fn to_action_request(&self) -> ExecutePuzzleAgentActionRequest {
+        ExecutePuzzleAgentActionRequest {
+            action: "generate_puzzle_ui_background".to_string(),
+            prompt_text: self.prompt_text.clone(),
+            reference_image_src: None,
+            reference_image_srcs: Vec::new(),
+            reference_image_asset_object_id: None,
+            reference_image_asset_object_ids: Vec::new(),
+            image_model: None,
+            ai_redraw: None,
+            candidate_count: None,
+            should_auto_name_level: None,
+            candidate_id: None,
+            level_id: self.level_id.clone(),
+            work_title: None,
+            work_description: None,
+            picture_description: None,
+            level_name: None,
+            summary: None,
+            theme_tags: None,
+            levels_json: self.levels_json.clone(),
+        }
+    }
+}
+
+pub(crate) async fn execute_puzzle_generate_images_worker_job(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    payload: PuzzleGenerateImagesWorkerPayload,
+    external_generation_guard: ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, PuzzleExternalGenerationWorkerError> {
+    let now = current_utc_micros();
+    let session = execute_billable_asset_operation_with_cost(
+        state.root_state(),
+        &payload.owner_user_id,
+        "puzzle_generated_image",
+        &payload.billing_asset_id,
+        PUZZLE_IMAGE_GENERATION_POINTS_COST,
+        async {
+            execute_puzzle_generate_images_worker_job_inner(
+                state,
+                request_context,
+                &payload,
+                now,
+                &external_generation_guard,
+            )
+            .await
+        },
+    )
+    .await;
+
+    match session {
+        Ok(session) => Ok(session),
+        Err(error) => {
+            match mark_puzzle_level_generation_failure_for_worker(
+                state,
+                &payload,
+                error.body_text(),
+                now,
+                &external_generation_guard,
+            )
+            .await
+            {
+                Ok(()) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_written(error))
+                }
+                Err(mark_error) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_pending(mark_error))
+                }
+            }
+        }
+    }
+}
+
+pub(crate) async fn execute_puzzle_generate_ui_background_worker_job(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    payload: PuzzleGenerateUiBackgroundWorkerPayload,
+    external_generation_guard: ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, PuzzleExternalGenerationWorkerError> {
+    let now = current_utc_micros();
+    let session = execute_billable_asset_operation_with_cost(
+        state.root_state(),
+        &payload.owner_user_id,
+        "puzzle_ui_background_image",
+        &payload.billing_asset_id,
+        PUZZLE_IMAGE_GENERATION_POINTS_COST,
+        async {
+            execute_puzzle_generate_ui_background_worker_job_inner(
+                state,
+                request_context,
+                &payload,
+                now,
+                &external_generation_guard,
+            )
+            .await
+        },
+    )
+    .await;
+
+    match session {
+        Ok(session) => Ok(session),
+        Err(error) => {
+            match mark_puzzle_level_generation_failure_for_external_generation(
+                state,
+                &payload.session_id,
+                &payload.owner_user_id,
+                payload.level_id.clone(),
+                payload.levels_json.clone(),
+                error.body_text(),
+                now,
+                &external_generation_guard,
+            )
+            .await
+            {
+                Ok(()) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_written(error))
+                }
+                Err(mark_error) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_pending(mark_error))
+                }
+            }
+        }
+    }
+}
+
+async fn execute_puzzle_generate_images_worker_job_inner(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    payload: &PuzzleGenerateImagesWorkerPayload,
+    now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, AppError> {
+    let action_payload = payload.to_action_request();
+    let target_level_id = payload.level_id.clone();
+    let levels_json = payload.levels_json.clone();
+    let session = get_puzzle_session_for_image_generation(
+        state,
+        payload.session_id.clone(),
+        payload.owner_user_id.clone(),
+        &action_payload,
+        levels_json.as_deref(),
+        now,
+    )
+    .await?;
+    let mut draft = session.draft.clone().ok_or_else(|| {
+        AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
+            "provider": PUZZLE_AGENT_API_BASE_PROVIDER,
+            "message": "拼图结果页草稿尚未生成",
+        }))
+    })?;
+    if let Some(levels_json) = levels_json.as_ref() {
+        draft.levels = parse_puzzle_level_records_from_module_json(levels_json)?;
+    }
+    let mut target_level = select_puzzle_level_for_api(&draft, target_level_id.as_deref())?;
+    let prompt = resolve_puzzle_level_image_prompt(
+        payload.prompt_text.as_deref(),
+        &target_level.picture_description,
+        &draft.summary,
+    );
+    let should_auto_name_level = payload
+        .should_auto_name_level
+        .unwrap_or_else(|| target_level.level_name.trim().is_empty());
+    if should_auto_name_level {
+        let naming =
+            generate_puzzle_first_level_name(state, target_level.picture_description.as_str())
+                .await;
+        target_level.level_name = naming.level_name.clone();
+        target_level.ui_background_prompt = naming.ui_background_prompt.clone();
+    }
+    let reference_image_sources = collect_puzzle_reference_image_sources(
+        payload.reference_image_src.as_deref(),
+        payload.reference_image_srcs.as_slice(),
+        payload.reference_image_asset_object_id.as_deref(),
+        payload.reference_image_asset_object_ids.as_slice(),
+    );
+    let primary_reference_image_src = reference_image_sources.first().map(String::as_str);
+    // 中文注释：拼图结果页从多候选抽卡收口为单图替换，前端传入的旧 candidateCount 只做兼容忽略。
+    let candidate_start_index = target_level.candidates.len();
+    let ai_redraw = payload.ai_redraw.unwrap_or(true);
+    let mut candidates =
+        if should_use_uploaded_puzzle_image_directly(primary_reference_image_src, ai_redraw) {
+            vec![
+                create_uploaded_puzzle_image_candidate(
+                    state,
+                    payload.owner_user_id.as_str(),
+                    &session.session_id,
+                    &target_level.level_name,
+                    &prompt,
+                    primary_reference_image_src.expect("checked reference image"),
+                    candidate_start_index,
+                )
+                .await?,
+            ]
+        } else {
+            let (_, profile_id) = build_stable_puzzle_work_ids(&session.session_id);
+            generate_puzzle_image_candidates(
+                state,
+                payload.owner_user_id.as_str(),
+                Some(profile_id.as_str()),
+                &session.session_id,
+                &target_level.level_name,
+                &prompt,
+                primary_reference_image_src,
+                ai_redraw,
+                payload.image_model.as_deref(),
+                1,
+                candidate_start_index,
+            )
+            .await
+            .map_err(map_puzzle_generation_endpoint_error)?
+        };
+    if candidates.is_empty() {
+        return Err(
+            AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+                "provider": PUZZLE_AGENT_API_BASE_PROVIDER,
+                "message": "拼图候选图生成结果为空",
+            })),
+        );
+    }
+    if let Some(refined_naming) = generate_puzzle_first_level_name_from_image(
+        state,
+        target_level.picture_description.as_str(),
+        &candidates[0].downloaded_image,
+    )
+    .await
+    .filter(|_| should_auto_name_level)
+    {
+        target_level.level_name = refined_naming.level_name.clone();
+        if refined_naming.ui_background_prompt.is_some() {
+            target_level.ui_background_prompt = refined_naming.ui_background_prompt.clone();
+        }
+    }
+    let mut updated_levels =
+        build_puzzle_levels_with_primary_update(&draft, &target_level, primary_reference_image_src);
+    for candidate in &mut candidates {
+        candidate.record.prompt = prompt.clone();
+    }
+    let selected_candidate = candidates
+        .iter()
+        .find(|candidate| candidate.record.selected)
+        .or_else(|| candidates.first())
+        .ok_or_else(|| {
+            AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+                "provider": PUZZLE_AGENT_API_BASE_PROVIDER,
+                "message": "拼图候选图生成结果为空",
+            }))
+        })?;
+    let asset_bundle = generate_puzzle_level_asset_bundle_required(
+        state,
+        request_context,
+        payload.owner_user_id.as_str(),
+        &session.session_id,
+        &target_level,
+        &selected_candidate.downloaded_image,
+    )
+    .await?;
+    attach_puzzle_level_asset_bundle(
+        &mut updated_levels,
+        target_level.level_id.as_str(),
+        asset_bundle,
+    );
+    attach_selected_puzzle_candidate_to_levels(
+        &mut updated_levels,
+        target_level.level_id.as_str(),
+        &selected_candidate.record,
+    );
+    let levels_json_with_generated_name =
+        Some(serialize_puzzle_level_records_for_module(&updated_levels)?);
+    let candidates_json = serde_json::to_string(
+        &candidates
+            .iter()
+            .map(|candidate| to_puzzle_generated_image_candidate(&candidate.record))
+            .collect::<Vec<_>>(),
+    )
+    .map_err(|error| {
+        AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
+            "provider": PUZZLE_AGENT_API_BASE_PROVIDER,
+            "message": format!("拼图候选图序列化失败：{error}"),
+        }))
+    })?;
+    state
+        .spacetime_client()
+        .save_puzzle_generated_images(PuzzleGeneratedImagesSaveRecordInput {
+            session_id: session.session_id.clone(),
+            owner_user_id: payload.owner_user_id.clone(),
+            level_id: Some(target_level.level_id.clone()),
+            levels_json: levels_json_with_generated_name,
+            candidates_json,
+            saved_at_micros: now,
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
+        })
+        .await
+        .map_err(map_puzzle_client_error)
+}
+
+async fn execute_puzzle_generate_ui_background_worker_job_inner(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    payload: &PuzzleGenerateUiBackgroundWorkerPayload,
+    now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, AppError> {
+    let action_payload = payload.to_action_request();
+    let target_level_id = payload.level_id.clone();
+    let levels_json = payload.levels_json.clone();
+    let session = get_puzzle_session_for_image_generation(
+        state,
+        payload.session_id.clone(),
+        payload.owner_user_id.clone(),
+        &action_payload,
+        levels_json.as_deref(),
+        now,
+    )
+    .await?;
+    let mut draft = session.draft.clone().ok_or_else(|| {
+        AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
+            "provider": PUZZLE_AGENT_API_BASE_PROVIDER,
+            "message": "拼图结果页草稿尚未生成",
+        }))
+    })?;
+    if let Some(levels_json) = levels_json.as_ref() {
+        draft.levels = parse_puzzle_level_records_from_module_json(levels_json)?;
+    }
+    let target_level = select_puzzle_level_for_api(&draft, target_level_id.as_deref())?;
+    let raw_prompt = payload
+        .prompt_text
+        .as_deref()
+        .map(str::trim)
+        .filter(|value| !value.is_empty())
+        .unwrap_or_default()
+        .to_string();
+    let resolved_prompt =
+        normalize_puzzle_ui_background_prompt(raw_prompt.as_str(), &draft, &target_level);
+    let generated = generate_puzzle_ui_background_image(
+        state,
+        request_context,
+        payload.owner_user_id.as_str(),
+        &session.session_id,
+        &target_level.level_name,
+        resolved_prompt.as_str(),
+    )
+    .await
+    .map_err(map_puzzle_generation_endpoint_error)?;
+    state
+        .spacetime_client()
+        .save_puzzle_ui_background(PuzzleUiBackgroundSaveRecordInput {
+            session_id: session.session_id.clone(),
+            owner_user_id: payload.owner_user_id.clone(),
+            level_id: Some(target_level.level_id.clone()),
+            levels_json,
+            prompt: resolved_prompt.clone(),
+            image_src: generated.image_src.clone(),
+            image_object_key: Some(generated.object_key.clone()),
+            saved_at_micros: now,
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
+        })
+        .await
+        .map_err(map_puzzle_client_error)
+}
+
+pub(crate) async fn mark_puzzle_level_generation_failure_for_worker(
+    state: &PuzzleApiState,
+    payload: &PuzzleGenerateImagesWorkerPayload,
+    error_message: String,
+    failed_at_micros: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<(), AppError> {
+    mark_puzzle_level_generation_failure_for_external_generation(
+        state,
+        &payload.session_id,
+        &payload.owner_user_id,
+        payload.level_id.clone(),
+        payload.levels_json.clone(),
+        error_message,
+        failed_at_micros,
+        external_generation_guard,
+    )
+    .await
+}
+
+async fn mark_puzzle_level_generation_failure_for_external_generation(
+    state: &PuzzleApiState,
+    session_id: &str,
+    owner_user_id: &str,
+    level_id: Option<String>,
+    levels_json: Option<String>,
+    error_message: String,
+    failed_at_micros: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<(), AppError> {
+    let result = state
+        .spacetime_client()
+        .mark_puzzle_level_generation_failed(PuzzleLevelGenerationFailureRecordInput {
+            session_id: session_id.to_string(),
+            owner_user_id: owner_user_id.to_string(),
+            level_id,
+            levels_json,
+            error_message,
+            failed_at_micros,
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
+        })
+        .await;
+    if let Err(error) = result {
+        tracing::warn!(
+            provider = PUZZLE_AGENT_API_BASE_PROVIDER,
+            session_id = %session_id,
+            owner_user_id = %owner_user_id,
+            message = %error,
+            "拼图 worker 关卡生图失败态回写失败"
+        );
+        return Err(map_puzzle_client_error(error));
+    }
+
+    Ok(())
+}
+
 pub(crate) async fn create_uploaded_puzzle_image_candidate(
     state: &PuzzleApiState,
     owner_user_id: &str,

server-rs/crates/api-server/src/puzzle/tests.rs

@@ -535,6 +535,108 @@ fn puzzle_image_generation_fallback_session_ready_when_asset_pack_complete() {
     assert_eq!(session.stage, "ready_to_publish");
 }
 
+#[test]
+fn puzzle_generate_images_worker_payload_keeps_action_snapshot() {
+    let raw_levels_json = serde_json::to_string(&vec![json!({
+        "levelId": "puzzle-level-2",
+        "levelName": "",
+        "pictureDescription": "新关卡里有一座发光钟楼。",
+        "candidates": [],
+        "selectedCandidateId": null,
+        "coverImageSrc": null,
+        "coverAssetId": null,
+        "generationStatus": "generating",
+    })])
+    .expect("levels json");
+    let levels_json = normalize_puzzle_levels_json_for_module(Some(raw_levels_json.as_str()))
+        .expect("levels should normalize")
+        .expect("levels json should exist");
+    let payload = PuzzleGenerateImagesWorkerPayload {
+        session_id: "puzzle-session-1".to_string(),
+        owner_user_id: "user-1".to_string(),
+        billing_asset_id: "puzzle-session-1:123".to_string(),
+        level_id: Some("puzzle-level-2".to_string()),
+        prompt_text: Some("发光钟楼".to_string()),
+        reference_image_src: None,
+        reference_image_srcs: vec!["data:image/png;base64,abc".to_string()],
+        reference_image_asset_object_id: Some("asset-object-1".to_string()),
+        reference_image_asset_object_ids: vec!["asset-object-2".to_string()],
+        image_model: Some(PUZZLE_IMAGE_MODEL_GPT_IMAGE_2.to_string()),
+        ai_redraw: Some(true),
+        should_auto_name_level: Some(true),
+        work_title: Some("暖灯猫街作品".to_string()),
+        work_description: Some("一套雨夜猫街主题拼图。".to_string()),
+        picture_description: None,
+        summary: Some("一套雨夜猫街主题拼图。".to_string()),
+        theme_tags: Some(vec!["猫咪".to_string(), "雨夜".to_string()]),
+        levels_json: Some(levels_json.clone()),
+        requested_at_micros: 123,
+    };
+
+    let encoded = serde_json::to_string(&payload).expect("payload should serialize");
+    let decoded: PuzzleGenerateImagesWorkerPayload =
+        serde_json::from_str(encoded.as_str()).expect("payload should deserialize");
+
+    assert_eq!(decoded.level_id.as_deref(), Some("puzzle-level-2"));
+    assert_eq!(decoded.reference_image_srcs.len(), 1);
+    assert_eq!(
+        decoded.reference_image_asset_object_ids,
+        vec!["asset-object-2".to_string()]
+    );
+    assert_eq!(decoded.should_auto_name_level, Some(true));
+    let records = parse_puzzle_level_records_from_module_json(
+        decoded.levels_json.as_deref().expect("levels json"),
+    )
+    .expect("levels should parse as module json");
+    assert_eq!(records[0].level_id, "puzzle-level-2");
+    assert_eq!(records[0].generation_status, "generating");
+}
+
+#[test]
+fn puzzle_generate_ui_background_worker_payload_keeps_action_snapshot() {
+    let raw_levels_json = serde_json::to_string(&vec![json!({
+        "levelId": "puzzle-level-3",
+        "levelName": "钟楼回廊",
+        "pictureDescription": "新关卡里有一座发光钟楼。",
+        "uiBackgroundPrompt": "发光钟楼延展成竖屏回廊，远处有暖色窗光。",
+        "candidates": [],
+        "selectedCandidateId": null,
+        "coverImageSrc": null,
+        "coverAssetId": null,
+        "generationStatus": "generating",
+    })])
+    .expect("levels json");
+    let levels_json = normalize_puzzle_levels_json_for_module(Some(raw_levels_json.as_str()))
+        .expect("levels should normalize")
+        .expect("levels json should exist");
+    let payload = PuzzleGenerateUiBackgroundWorkerPayload {
+        session_id: "puzzle-session-1".to_string(),
+        owner_user_id: "user-1".to_string(),
+        billing_asset_id: "puzzle-session-1:456".to_string(),
+        level_id: Some("puzzle-level-3".to_string()),
+        prompt_text: Some("发光钟楼延展成竖屏回廊".to_string()),
+        levels_json: Some(levels_json.clone()),
+        requested_at_micros: 456,
+    };
+
+    let encoded = serde_json::to_string(&payload).expect("payload should serialize");
+    let decoded: PuzzleGenerateUiBackgroundWorkerPayload =
+        serde_json::from_str(encoded.as_str()).expect("payload should deserialize");
+
+    assert_eq!(decoded.level_id.as_deref(), Some("puzzle-level-3"));
+    assert_eq!(
+        decoded.prompt_text.as_deref(),
+        Some("发光钟楼延展成竖屏回廊")
+    );
+    assert_eq!(decoded.requested_at_micros, 456);
+    let records = parse_puzzle_level_records_from_module_json(
+        decoded.levels_json.as_deref().expect("levels json"),
+    )
+    .expect("levels should parse as module json");
+    assert_eq!(records[0].level_id, "puzzle-level-3");
+    assert_eq!(records[0].generation_status, "generating");
+}
+
 #[test]
 fn puzzle_first_level_name_parser_accepts_json_and_normalizes_text() {
     assert_eq!(

server-rs/crates/api-server/src/square_hole.rs

@@ -69,7 +69,7 @@ use crate::generated_image_assets::{
 use crate::{
     ai_generation_drafts::{AiGenerationDraftContext, AiGenerationDraftWriter},
     api_response::json_success_body,
-    auth::AuthenticatedAccessToken,
+    auth::{AuthenticatedAccessToken, RuntimePrincipal},
     http_error::AppError,
     openai_image_generation::{
         DownloadedOpenAiImage, build_openai_image_http_client, create_openai_image_generation,
@@ -739,7 +739,7 @@ pub async fn start_square_hole_run(
     State(state): State<AppState>,
     Path(profile_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<StartSquareHoleRunRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let maybe_payload = payload.ok().map(|Json(payload)| payload);
@@ -758,7 +758,7 @@ pub async fn start_square_hole_run(
         .spacetime_client()
         .start_square_hole_run(SquareHoleRunStartRecordInput {
             run_id: build_prefixed_uuid_id(SQUARE_HOLE_RUN_ID_PREFIX),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             profile_id: profile_id.clone(),
             started_at_ms: current_utc_ms(),
         })
@@ -774,15 +774,17 @@ pub async fn start_square_hole_run(
     record_work_play_start_after_success(
         &state,
         &request_context,
-        WorkPlayTrackingDraft::new(
+        WorkPlayTrackingDraft::runtime_principal(
             "square-hole",
             profile_id.clone(),
-            &authenticated,
+            &principal,
             "/api/runtime/square-hole/...",
         )
         .profile_id(profile_id.clone())
+        .owner_user_id(principal.subject().to_string())
         .extra(json!({
             "runId": run.run_id,
+            "principalKind": principal.kind().as_str(),
         })),
     )
     .await;
@@ -799,7 +801,7 @@ pub async fn get_square_hole_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(
         &request_context,
@@ -810,7 +812,7 @@ pub async fn get_square_hole_run(
 
     let run = state
         .spacetime_client()
-        .get_square_hole_run(run_id, authenticated.claims().user_id().to_string())
+        .get_square_hole_run(run_id, principal.subject().to_string())
         .await
         .map_err(|error| {
             square_hole_error_response(
@@ -832,7 +834,7 @@ pub async fn drop_square_hole_shape(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<DropSquareHoleShapeRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = square_hole_json(payload, &request_context, SQUARE_HOLE_RUNTIME_PROVIDER)?;
@@ -859,7 +861,7 @@ pub async fn drop_square_hole_shape(
         .spacetime_client()
         .drop_square_hole_shape(SquareHoleRunDropRecordInput {
             run_id: payload.run_id.unwrap_or(run_id),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             hole_id: payload.hole_id,
             client_snapshot_version: payload.client_snapshot_version,
             client_event_id: payload.client_event_id,
@@ -887,7 +889,7 @@ pub async fn stop_square_hole_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<StopSquareHoleRunRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let _ = payload.ok();
@@ -902,7 +904,7 @@ pub async fn stop_square_hole_run(
         .spacetime_client()
         .stop_square_hole_run(SquareHoleRunStopRecordInput {
             run_id,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             stopped_at_ms: current_utc_ms(),
         })
         .await
@@ -926,7 +928,7 @@ pub async fn restart_square_hole_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(
         &request_context,
@@ -940,7 +942,7 @@ pub async fn restart_square_hole_run(
         .restart_square_hole_run(SquareHoleRunRestartRecordInput {
             source_run_id: run_id,
             next_run_id: build_prefixed_uuid_id(SQUARE_HOLE_RUN_ID_PREFIX),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             restarted_at_ms: current_utc_ms(),
         })
         .await
@@ -964,7 +966,7 @@ pub async fn finish_square_hole_time_up(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(
         &request_context,
@@ -977,7 +979,7 @@ pub async fn finish_square_hole_time_up(
         .spacetime_client()
         .finish_square_hole_time_up(SquareHoleRunTimeUpRecordInput {
             run_id,
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             finished_at_ms: current_utc_ms(),
         })
         .await

server-rs/crates/api-server/src/state.rs

@@ -41,6 +41,7 @@ use tracing::{info, warn};
 use crate::config::AppConfig;
 use crate::puzzle_gallery_cache::PuzzleGalleryCache;
 use crate::tracking_outbox::TrackingOutbox;
+use crate::wallet_refund_outbox::WalletRefundOutbox;
 use crate::wechat::pay::{build_wechat_pay_config, map_wechat_pay_init_error};
 use crate::wechat::provider::build_wechat_provider;
 use crate::work_author::{
@@ -263,6 +264,7 @@ pub struct AppStateInner {
     spacetime_client: SpacetimeClient,
     puzzle_gallery_cache: PuzzleGalleryCache,
     tracking_outbox: Option<Arc<TrackingOutbox>>,
+    wallet_refund_outbox: Option<Arc<WalletRefundOutbox>>,
     llm_client: Option<LlmClient>,
     creative_agent_gpt5_client: Option<LlmClient>,
     creative_agent_executor: Arc<MockLangChainRustAgentExecutor>,
@@ -406,6 +408,8 @@ impl AppState {
             procedure_timeout: config.spacetime_procedure_timeout,
         });
         let tracking_outbox = TrackingOutbox::from_config(&config, spacetime_client.clone());
+        let wallet_refund_outbox =
+            WalletRefundOutbox::from_config(&config, spacetime_client.clone());
         let llm_client = build_llm_client(&config)?;
         let creative_agent_gpt5_client = build_creative_agent_gpt5_client(&config)?;
         let http_request_permit_pools = HttpRequestPermitPools::from_config(&config);
@@ -441,6 +445,7 @@ impl AppState {
             spacetime_client,
             puzzle_gallery_cache: PuzzleGalleryCache::new(),
             tracking_outbox,
+            wallet_refund_outbox,
             llm_client,
             creative_agent_gpt5_client,
             creative_agent_executor: Arc::new(MockLangChainRustAgentExecutor),
@@ -922,6 +927,10 @@ impl AppState {
         self.tracking_outbox.clone()
     }
 
+    pub fn wallet_refund_outbox(&self) -> Option<Arc<WalletRefundOutbox>> {
+        self.wallet_refund_outbox.clone()
+    }
+
     pub fn llm_client(&self) -> Option<&LlmClient> {
         self.llm_client.as_ref()
     }

server-rs/crates/api-server/src/visual_novel.rs

@@ -30,7 +30,7 @@ use time::OffsetDateTime;
 
 use crate::{
     api_response::json_success_body,
-    auth::AuthenticatedAccessToken,
+    auth::{AuthenticatedAccessToken, RuntimePrincipal},
     http_error::AppError,
     prompt::visual_novel as vn_prompt,
     request_context::RequestContext,
@@ -434,7 +434,7 @@ pub async fn start_visual_novel_run(
     State(state): State<AppState>,
     Path(profile_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<contract::VisualNovelStartRunRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = parse_json_payload(&request_context, payload)?;
@@ -453,7 +453,7 @@ pub async fn start_visual_novel_run(
         .spacetime_client()
         .start_visual_novel_run(VisualNovelRunStartRecordInput {
             run_id: build_prefixed_uuid_id(domain::VISUAL_NOVEL_RUN_ID_PREFIX),
-            owner_user_id: authenticated.claims().user_id().to_string(),
+            owner_user_id: principal.subject().to_string(),
             profile_id: profile_id.clone(),
             mode: run_mode_to_wire(&payload.mode).to_string(),
             snapshot_json: None,
@@ -467,16 +467,18 @@ pub async fn start_visual_novel_run(
     record_work_play_start_after_success(
         &state,
         &request_context,
-        WorkPlayTrackingDraft::new(
+        WorkPlayTrackingDraft::runtime_principal(
             "visual-novel",
             profile_id.clone(),
-            &authenticated,
+            &principal,
             "/api/runtime/visual-novel/...",
         )
         .profile_id(profile_id.clone())
+        .owner_user_id(principal.subject().to_string())
         .extra(json!({
             "mode": run_mode_to_wire(&payload.mode),
             "runId": run.run_id,
+            "principalKind": principal.kind().as_str(),
         })),
     )
     .await;
@@ -493,12 +495,12 @@ pub async fn get_visual_novel_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&run_id, "runId")?;
     let run = state
         .spacetime_client()
-        .get_visual_novel_run(run_id, authenticated.claims().user_id().to_string())
+        .get_visual_novel_run(run_id, principal.subject().to_string())
         .await
         .map_err(|error| {
             visual_novel_error_response(&request_context, map_spacetime_error(error))
@@ -516,13 +518,13 @@ pub async fn stream_visual_novel_action(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<contract::VisualNovelRuntimeActionRequest>, JsonRejection>,
 ) -> Result<Response, Response> {
     let Json(payload) = parse_json_payload(&request_context, payload)?;
     ensure_non_empty(&run_id, "runId")?;
     ensure_non_empty(&payload.client_event_id, "clientEventId")?;
-    let owner_user_id = authenticated.claims().user_id().to_string();
+    let owner_user_id = principal.subject().to_string();
     let run = state
         .spacetime_client()
         .get_visual_novel_run(run_id.clone(), owner_user_id.clone())
@@ -569,12 +571,12 @@ pub async fn list_visual_novel_history(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
 ) -> Result<Json<Value>, Response> {
     ensure_non_empty(&run_id, "runId")?;
     let history = state
         .spacetime_client()
-        .list_visual_novel_runtime_history(run_id, authenticated.claims().user_id().to_string())
+        .list_visual_novel_runtime_history(run_id, principal.subject().to_string())
         .await
         .map_err(|error| {
             visual_novel_error_response(&request_context, map_spacetime_error(error))
@@ -595,13 +597,13 @@ pub async fn regenerate_visual_novel_run(
     State(state): State<AppState>,
     Path(run_id): Path<String>,
     Extension(request_context): Extension<RequestContext>,
-    Extension(authenticated): Extension<AuthenticatedAccessToken>,
+    Extension(principal): Extension<RuntimePrincipal>,
     payload: Result<Json<contract::VisualNovelRegenerateRequest>, JsonRejection>,
 ) -> Result<Json<Value>, Response> {
     let Json(payload) = parse_json_payload(&request_context, payload)?;
     ensure_non_empty(&run_id, "runId")?;
     ensure_non_empty(&payload.history_entry_id, "historyEntryId")?;
-    let owner_user_id = authenticated.claims().user_id().to_string();
+    let owner_user_id = principal.subject().to_string();
     let run = state
         .spacetime_client()
         .get_visual_novel_run(run_id.clone(), owner_user_id.clone())

server-rs/crates/api-server/src/wallet_refund_outbox.rs

@@ -0,0 +1,463 @@
+use std::{
+    fmt,
+    path::{Path, PathBuf},
+    sync::Arc,
+    time::{Duration, SystemTime, UNIX_EPOCH},
+};
+
+use serde::{Deserialize, Serialize};
+use sha2::{Digest, Sha256};
+use spacetime_client::{SpacetimeClient, SpacetimeClientError};
+use tokio::{
+    fs::{self, File, OpenOptions},
+    io::{AsyncReadExt, AsyncWriteExt},
+    sync::{Mutex, Notify},
+    time::sleep,
+};
+use tracing::{debug, warn};
+
+use crate::config::AppConfig;
+
+const PENDING_FILE_PREFIX: &str = "refund-";
+const CORRUPT_FILE_PREFIX: &str = "corrupt-";
+const TEMP_FILE_PREFIX: &str = "tmp-";
+const OUTBOX_FILE_EXTENSION: &str = ".json";
+
+#[derive(Clone)]
+pub struct WalletRefundOutbox {
+    dir: PathBuf,
+    batch_size: usize,
+    flush_interval: Duration,
+    max_bytes: u64,
+    spacetime_client: SpacetimeClient,
+    enqueue_lock: Arc<Mutex<()>>,
+    flush_notify: Arc<Notify>,
+}
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub(crate) struct WalletRefundOutboxRecord {
+    pub owner_user_id: String,
+    pub amount: u64,
+    pub ledger_id: String,
+    pub created_at_micros: i64,
+    pub asset_kind: String,
+    pub asset_id: String,
+}
+
+#[derive(Debug)]
+pub enum WalletRefundOutboxEnqueueOutcome {
+    Enqueued,
+    Dropped { reason: &'static str },
+}
+
+#[derive(Debug)]
+pub enum WalletRefundOutboxError {
+    Io(std::io::Error),
+    Json(serde_json::Error),
+    Spacetime(SpacetimeClientError),
+}
+
+impl WalletRefundOutbox {
+    pub fn from_config(config: &AppConfig, spacetime_client: SpacetimeClient) -> Option<Arc<Self>> {
+        if !config.wallet_refund_outbox_enabled {
+            return None;
+        }
+
+        Some(Arc::new(Self {
+            dir: config.wallet_refund_outbox_dir.clone(),
+            batch_size: config.wallet_refund_outbox_batch_size.max(1),
+            flush_interval: config.wallet_refund_outbox_flush_interval,
+            max_bytes: config.wallet_refund_outbox_max_bytes,
+            spacetime_client,
+            enqueue_lock: Arc::new(Mutex::new(())),
+            flush_notify: Arc::new(Notify::new()),
+        }))
+    }
+
+    pub async fn enqueue(
+        &self,
+        record: WalletRefundOutboxRecord,
+    ) -> Result<WalletRefundOutboxEnqueueOutcome, WalletRefundOutboxError> {
+        let _guard = self.enqueue_lock.lock().await;
+        fs::create_dir_all(&self.dir).await?;
+
+        let pending_path = self.pending_path_for_ledger(&record.ledger_id);
+        if fs::metadata(&pending_path).await.is_ok() {
+            self.flush_notify.notify_one();
+            return Ok(WalletRefundOutboxEnqueueOutcome::Enqueued);
+        }
+
+        let bytes = serde_json::to_vec(&record)?;
+        let line_bytes = bytes.len().min(u64::MAX as usize) as u64;
+        let current_bytes = directory_size_if_exists(&self.dir).unwrap_or(0);
+        if current_bytes.saturating_add(line_bytes) > self.max_bytes {
+            return Ok(WalletRefundOutboxEnqueueOutcome::Dropped {
+                reason: "max_bytes",
+            });
+        }
+
+        let temp_path = self.temp_path();
+        let mut file = OpenOptions::new()
+            .create_new(true)
+            .write(true)
+            .open(&temp_path)
+            .await?;
+        file.write_all(&bytes).await?;
+        file.flush().await?;
+        file.sync_data().await?;
+        drop(file);
+        if fs::metadata(&pending_path).await.is_ok() {
+            let _ = fs::remove_file(&temp_path).await;
+            self.flush_notify.notify_one();
+            return Ok(WalletRefundOutboxEnqueueOutcome::Enqueued);
+        }
+        fs::rename(&temp_path, &pending_path).await?;
+        sync_directory_metadata(&self.dir).await?;
+        self.flush_notify.notify_one();
+        Ok(WalletRefundOutboxEnqueueOutcome::Enqueued)
+    }
+
+    pub fn spawn_worker(self: Arc<Self>) {
+        tokio::spawn(async move {
+            loop {
+                tokio::select! {
+                    _ = sleep(self.flush_interval) => {
+                        if let Err(error) = self.flush_pending_files_once().await {
+                            warn!(error = %error, "wallet refund outbox 重放退款失败，将保留文件等待重试");
+                        }
+                    }
+                    _ = self.flush_notify.notified() => {
+                        if let Err(error) = self.flush_pending_files_once().await {
+                            warn!(error = %error, "wallet refund outbox 主动重放退款失败，将保留文件等待重试");
+                        }
+                    }
+                }
+            }
+        });
+    }
+
+    pub async fn flush_for_shutdown(&self) -> Result<(), WalletRefundOutboxError> {
+        self.flush_pending_files_once().await
+    }
+
+    async fn flush_pending_files_once(&self) -> Result<(), WalletRefundOutboxError> {
+        fs::create_dir_all(&self.dir).await?;
+        let pending_files = self.list_pending_files().await?;
+        for path in pending_files.into_iter().take(self.batch_size) {
+            let record = match read_refund_record(&path).await {
+                Ok(record) => record,
+                Err(error) if error.is_data_corruption() => {
+                    let corrupt_path = self.corrupt_path_for(&path);
+                    fs::rename(&path, &corrupt_path).await?;
+                    sync_directory_metadata(&self.dir).await?;
+                    warn!(
+                        error = %error,
+                        source = %path.display(),
+                        target = %corrupt_path.display(),
+                        "wallet refund outbox 文件无法解析，已隔离"
+                    );
+                    continue;
+                }
+                Err(error) => return Err(error),
+            };
+
+            match self
+                .spacetime_client
+                .refund_profile_wallet_points(
+                    record.owner_user_id.clone(),
+                    record.amount,
+                    record.ledger_id.clone(),
+                    record.created_at_micros,
+                )
+                .await
+            {
+                Ok(_) => {
+                    match fs::remove_file(&path).await {
+                        Ok(()) => {}
+                        Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
+                        Err(error) => return Err(error.into()),
+                    }
+                    sync_directory_metadata(&self.dir).await?;
+                    debug!(
+                        ledger_id = %record.ledger_id,
+                        owner_user_id = %record.owner_user_id,
+                        asset_kind = %record.asset_kind,
+                        asset_id = %record.asset_id,
+                        path = %path.display(),
+                        "wallet refund outbox 退款已重放并删除文件"
+                    );
+                }
+                Err(error) => return Err(WalletRefundOutboxError::Spacetime(error)),
+            }
+        }
+        Ok(())
+    }
+
+    async fn list_pending_files(&self) -> Result<Vec<PathBuf>, WalletRefundOutboxError> {
+        let mut entries = fs::read_dir(&self.dir).await?;
+        let mut files = Vec::new();
+        while let Some(entry) = entries.next_entry().await? {
+            let path = entry.path();
+            let Some(name) = path.file_name().and_then(|value| value.to_str()) else {
+                continue;
+            };
+            if name.starts_with(PENDING_FILE_PREFIX) && name.ends_with(OUTBOX_FILE_EXTENSION) {
+                files.push(path);
+            }
+        }
+        files.sort();
+        Ok(files)
+    }
+
+    fn pending_path_for_ledger(&self, ledger_id: &str) -> PathBuf {
+        self.dir.join(format!(
+            "{PENDING_FILE_PREFIX}{}{OUTBOX_FILE_EXTENSION}",
+            ledger_id_hash(ledger_id)
+        ))
+    }
+
+    fn temp_path(&self) -> PathBuf {
+        self.dir.join(format!(
+            "{TEMP_FILE_PREFIX}{}-{uuid}{OUTBOX_FILE_EXTENSION}",
+            current_unix_micros(),
+            uuid = uuid::Uuid::new_v4()
+        ))
+    }
+
+    fn corrupt_path_for(&self, path: &Path) -> PathBuf {
+        let name = path
+            .file_name()
+            .and_then(|value| value.to_str())
+            .unwrap_or("unknown.json");
+        self.dir.join(format!(
+            "{CORRUPT_FILE_PREFIX}{}-{uuid}-{name}",
+            current_unix_micros(),
+            uuid = uuid::Uuid::new_v4()
+        ))
+    }
+}
+
+impl fmt::Debug for WalletRefundOutbox {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("WalletRefundOutbox")
+            .field("dir", &self.dir)
+            .field("batch_size", &self.batch_size)
+            .field("flush_interval", &self.flush_interval)
+            .field("max_bytes", &self.max_bytes)
+            .finish()
+    }
+}
+
+impl fmt::Display for WalletRefundOutboxError {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match self {
+            Self::Io(error) => write!(f, "{error}"),
+            Self::Json(error) => write!(f, "{error}"),
+            Self::Spacetime(error) => write!(f, "{error}"),
+        }
+    }
+}
+
+impl From<std::io::Error> for WalletRefundOutboxError {
+    fn from(value: std::io::Error) -> Self {
+        Self::Io(value)
+    }
+}
+
+impl From<serde_json::Error> for WalletRefundOutboxError {
+    fn from(value: serde_json::Error) -> Self {
+        Self::Json(value)
+    }
+}
+
+impl WalletRefundOutboxError {
+    fn is_data_corruption(&self) -> bool {
+        matches!(self, Self::Json(_))
+    }
+}
+
+async fn read_refund_record(
+    path: &Path,
+) -> Result<WalletRefundOutboxRecord, WalletRefundOutboxError> {
+    let mut file = File::open(path).await?;
+    let mut bytes = Vec::new();
+    file.read_to_end(&mut bytes).await?;
+    Ok(serde_json::from_slice::<WalletRefundOutboxRecord>(&bytes)?)
+}
+
+fn directory_size_if_exists(path: &Path) -> Result<u64, std::io::Error> {
+    if !path.is_dir() {
+        return Ok(0);
+    }
+
+    let mut total = 0u64;
+    for entry in std::fs::read_dir(path)? {
+        let entry = entry?;
+        if !is_pending_outbox_file_name(&entry.file_name()) {
+            continue;
+        }
+        let metadata = entry.metadata()?;
+        if metadata.is_file() {
+            total = total.saturating_add(metadata.len());
+        }
+    }
+    Ok(total)
+}
+
+fn current_unix_micros() -> u128 {
+    SystemTime::now()
+        .duration_since(UNIX_EPOCH)
+        .unwrap_or_default()
+        .as_micros()
+}
+
+fn ledger_id_hash(ledger_id: &str) -> String {
+    hex::encode(Sha256::digest(ledger_id.as_bytes()))
+}
+
+fn is_pending_outbox_file_name(name: &std::ffi::OsStr) -> bool {
+    name.to_str().is_some_and(|value| {
+        value.starts_with(PENDING_FILE_PREFIX) && value.ends_with(OUTBOX_FILE_EXTENSION)
+    })
+}
+
+async fn sync_directory_metadata(path: &Path) -> Result<(), WalletRefundOutboxError> {
+    let path = path.to_path_buf();
+    tokio::task::spawn_blocking(move || {
+        let dir = std::fs::File::open(path)?;
+        dir.sync_all()
+    })
+    .await
+    .map_err(|error| std::io::Error::new(std::io::ErrorKind::Other, error.to_string()))??;
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn sample_record(ledger_id: &str) -> WalletRefundOutboxRecord {
+        WalletRefundOutboxRecord {
+            owner_user_id: "user-1".to_string(),
+            amount: 2,
+            ledger_id: ledger_id.to_string(),
+            created_at_micros: 1_713_680_000_000_000,
+            asset_kind: "puzzle_initial_image".to_string(),
+            asset_id: "asset-1".to_string(),
+        }
+    }
+
+    fn test_dir(name: &str) -> PathBuf {
+        let dir = std::env::temp_dir().join(format!(
+            "genarrative-wallet-refund-outbox-{name}-{}",
+            current_unix_micros()
+        ));
+        let _ = std::fs::remove_dir_all(&dir);
+        dir
+    }
+
+    fn test_outbox(dir: PathBuf, max_bytes: u64) -> Arc<WalletRefundOutbox> {
+        let config = AppConfig {
+            wallet_refund_outbox_dir: dir,
+            wallet_refund_outbox_batch_size: 500,
+            wallet_refund_outbox_flush_interval: Duration::from_secs(60),
+            wallet_refund_outbox_max_bytes: max_bytes,
+            ..AppConfig::default()
+        };
+        WalletRefundOutbox::from_config(
+            &config,
+            SpacetimeClient::new(spacetime_client::SpacetimeClientConfig {
+                server_url: "http://127.0.0.1:1".to_string(),
+                database: "missing".to_string(),
+                token: None,
+                pool_size: 1,
+                procedure_timeout: Duration::from_millis(10),
+            }),
+        )
+        .expect("outbox should be enabled")
+    }
+
+    #[tokio::test]
+    async fn enqueue_is_idempotent_per_ledger_id() {
+        let dir = test_dir("idempotent");
+        let outbox = test_outbox(dir.clone(), 1024 * 1024);
+
+        outbox.enqueue(sample_record("ledger-1")).await.unwrap();
+        outbox.enqueue(sample_record("ledger-1")).await.unwrap();
+
+        let pending_count = std::fs::read_dir(&dir)
+            .unwrap()
+            .filter_map(Result::ok)
+            .filter(|entry| is_pending_outbox_file_name(&entry.file_name()))
+            .count();
+        assert_eq!(pending_count, 1);
+
+        let _ = std::fs::remove_dir_all(dir);
+    }
+
+    #[tokio::test]
+    async fn enqueue_drops_when_outbox_exceeds_max_bytes() {
+        let dir = test_dir("max-bytes");
+        let outbox = test_outbox(dir.clone(), 1);
+
+        let outcome = outbox.enqueue(sample_record("ledger-1")).await.unwrap();
+
+        assert!(matches!(
+            outcome,
+            WalletRefundOutboxEnqueueOutcome::Dropped {
+                reason: "max_bytes"
+            }
+        ));
+        assert!(!dir.exists() || std::fs::read_dir(&dir).unwrap().next().is_none());
+
+        let _ = std::fs::remove_dir_all(dir);
+    }
+
+    #[tokio::test]
+    async fn flush_quarantines_corrupt_file() {
+        let dir = test_dir("corrupt");
+        std::fs::create_dir_all(&dir).unwrap();
+        let pending_path = dir.join(format!("{PENDING_FILE_PREFIX}bad{OUTBOX_FILE_EXTENSION}"));
+        std::fs::write(&pending_path, b"{not-json}").unwrap();
+        let outbox = test_outbox(dir.clone(), 1024 * 1024);
+
+        outbox.flush_pending_files_once().await.unwrap();
+
+        assert!(!pending_path.exists());
+        let corrupt_count = std::fs::read_dir(&dir)
+            .unwrap()
+            .filter_map(Result::ok)
+            .filter(|entry| {
+                entry
+                    .file_name()
+                    .to_str()
+                    .is_some_and(|name| name.starts_with(CORRUPT_FILE_PREFIX))
+            })
+            .count();
+        assert_eq!(corrupt_count, 1);
+
+        let _ = std::fs::remove_dir_all(dir);
+    }
+
+    #[tokio::test]
+    async fn shutdown_flush_keeps_file_when_spacetime_is_unavailable() {
+        let dir = test_dir("shutdown");
+        let outbox = test_outbox(dir.clone(), 1024 * 1024);
+
+        outbox.enqueue(sample_record("ledger-1")).await.unwrap();
+        let result = outbox.flush_for_shutdown().await;
+
+        assert!(
+            matches!(result, Err(WalletRefundOutboxError::Spacetime(_))),
+            "missing test SpacetimeDB should keep refund file for retry"
+        );
+        let pending_count = std::fs::read_dir(&dir)
+            .unwrap()
+            .filter_map(Result::ok)
+            .filter(|entry| is_pending_outbox_file_name(&entry.file_name()))
+            .count();
+        assert_eq!(pending_count, 1);
+
+        let _ = std::fs::remove_dir_all(dir);
+    }
+}

server-rs/crates/module-puzzle/src/application.rs

@@ -20,6 +20,14 @@ pub struct PuzzleAgentSessionProcedureResult {
     pub error_message: Option<String>,
 }
 
+#[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PuzzleBackgroundCompileTaskProcedureResult {
+    pub ok: bool,
+    pub claimed: bool,
+    pub error_message: Option<String>,
+}
+
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
 #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
 pub struct PuzzleWorksProcedureResult {

server-rs/crates/module-puzzle/src/commands.rs

@@ -66,6 +66,28 @@ pub struct PuzzleDraftCompileInput {
     pub session_id: String,
     pub owner_user_id: String,
     pub compiled_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
+}
+
+#[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PuzzleBackgroundCompileTaskClaimInput {
+    pub task_id: String,
+    pub claim_id: String,
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub claimed_at_micros: i64,
+}
+
+#[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PuzzleBackgroundCompileTaskReleaseInput {
+    pub task_id: String,
+    pub claim_id: String,
+    pub session_id: String,
+    pub owner_user_id: String,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
@@ -75,6 +97,23 @@ pub struct PuzzleDraftCompileFailureInput {
     pub owner_user_id: String,
     pub error_message: String,
     pub failed_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
+}
+
+#[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PuzzleLevelGenerationFailureInput {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub level_id: Option<String>,
+    pub levels_json: Option<String>,
+    pub error_message: String,
+    pub failed_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
@@ -86,6 +125,9 @@ pub struct PuzzleGeneratedImagesSaveInput {
     pub levels_json: Option<String>,
     pub candidates_json: String,
     pub saved_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
@@ -99,6 +141,9 @@ pub struct PuzzleUiBackgroundSaveInput {
     pub image_src: String,
     pub image_object_key: Option<String>,
     pub saved_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]

server-rs/crates/server-manager-panel/Cargo.toml

@@ -0,0 +1,13 @@
+[package]
+name = "server-manager-panel"
+edition.workspace = true
+version.workspace = true
+license.workspace = true
+
+[dependencies]
+eframe = { version = "0.33", default-features = false, features = [
+    "default_fonts",
+    "glow",
+    "wayland",
+    "x11",
+] }

server-rs/crates/server-manager-panel/src/app.rs

@@ -0,0 +1,577 @@
+use eframe::egui;
+
+use crate::health::{
+    DiskSnapshot, HealthLevel, MemorySnapshot, ProbeSnapshot, ServerHealthReport, ServiceSnapshot,
+};
+use crate::remote::{
+    RemoteEvent, RemoteReceiver, RemoteSender, ServiceAction, channel, spawn_health_check,
+    spawn_service_action,
+};
+use crate::ssh_config::{SshAlias, discover_ssh_aliases};
+
+const DEFAULT_MANAGED_SERVICES: &[&str] = &[
+    "genarrative-api.service",
+    "spacetimedb.service",
+    "nginx.service",
+    "genarrative-health-patrol.timer",
+    "genarrative-database-backup.timer",
+];
+
+#[derive(Debug)]
+pub struct ServerManagerApp {
+    servers: Vec<ServerState>,
+    selected_alias: Option<String>,
+    sidebar_collapsed: bool,
+    tx: RemoteSender,
+    rx: RemoteReceiver,
+    pending_confirmation: Option<ServiceConfirmation>,
+    custom_service_name: String,
+}
+
+impl Default for ServerManagerApp {
+    fn default() -> Self {
+        let (tx, rx) = channel();
+        let aliases = discover_ssh_aliases();
+        let selected_alias = aliases.first().map(|alias| alias.name.clone());
+        Self {
+            servers: aliases.into_iter().map(ServerState::new).collect(),
+            selected_alias,
+            sidebar_collapsed: false,
+            tx,
+            rx,
+            pending_confirmation: None,
+            custom_service_name: String::new(),
+        }
+    }
+}
+
+impl eframe::App for ServerManagerApp {
+    fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
+        self.drain_remote_events(ctx);
+        self.render_confirm_dialog(ctx);
+
+        egui::TopBottomPanel::top("top_bar").show(ctx, |ui| {
+            ui.horizontal(|ui| {
+                if ui
+                    .button(if self.sidebar_collapsed {
+                        "展开侧栏"
+                    } else {
+                        "收起侧栏"
+                    })
+                    .clicked()
+                {
+                    self.sidebar_collapsed = !self.sidebar_collapsed;
+                }
+                if ui.button("重新读取 SSH alias").clicked() {
+                    self.reload_aliases();
+                }
+                if let Some(alias) = self.selected_alias.clone() {
+                    if ui.button("刷新当前服务器").clicked() {
+                        self.refresh_server(&alias);
+                    }
+                }
+                ui.separator();
+                ui.label("本地 SSH alias 管理");
+            });
+        });
+
+        if !self.sidebar_collapsed {
+            egui::SidePanel::left("server_sidebar")
+                .resizable(true)
+                .default_width(260.0)
+                .width_range(180.0..=360.0)
+                .show(ctx, |ui| self.render_sidebar(ui));
+        }
+
+        egui::CentralPanel::default().show(ctx, |ui| {
+            if self.servers.is_empty() {
+                self.render_empty_state(ui);
+                return;
+            }
+
+            let Some(alias) = self.selected_alias.clone() else {
+                self.render_empty_state(ui);
+                return;
+            };
+
+            if let Some(index) = self.server_index(&alias) {
+                self.render_server_detail(ui, index);
+            } else {
+                ui.label("请选择服务器");
+            }
+        });
+    }
+}
+
+impl ServerManagerApp {
+    fn drain_remote_events(&mut self, ctx: &egui::Context) {
+        while let Ok(event) = self.rx.try_recv() {
+            match event {
+                RemoteEvent::Health { alias, result } => {
+                    if let Some(server) = self.server_mut(&alias) {
+                        server.loading = false;
+                        match result {
+                            Ok(report) => {
+                                server.error = None;
+                                server.report = Some(report);
+                            }
+                            Err(error) => {
+                                server.error = Some(error);
+                            }
+                        }
+                    }
+                }
+                RemoteEvent::ServiceAction {
+                    alias,
+                    service,
+                    action,
+                    result,
+                } => {
+                    if let Some(server) = self.server_mut(&alias) {
+                        server.action_in_progress = None;
+                        server.action_log = Some(format!(
+                            "{} {}: {}\n{}{}",
+                            action.label(),
+                            service,
+                            result.summary,
+                            result.stdout,
+                            result.stderr
+                        ));
+                        server.loading = true;
+                        spawn_health_check(alias, self.tx.clone());
+                    }
+                }
+            }
+            ctx.request_repaint();
+        }
+    }
+
+    fn render_sidebar(&mut self, ui: &mut egui::Ui) {
+        ui.heading("服务器");
+        ui.add_space(8.0);
+        let mut refresh_alias: Option<String> = None;
+
+        for server in &mut self.servers {
+            let selected = self.selected_alias.as_deref() == Some(server.alias.name.as_str());
+            let response = ui.selectable_label(selected, server_label(server));
+            if response.clicked() {
+                self.selected_alias = Some(server.alias.name.clone());
+            }
+            response.on_hover_text(server.alias.source.display().to_string());
+            ui.horizontal(|ui| {
+                let status = server.status();
+                ui.colored_label(level_color(status), status.label());
+                if server.loading {
+                    ui.spinner();
+                }
+                if ui.small_button("刷新").clicked() {
+                    refresh_alias = Some(server.alias.name.clone());
+                }
+            });
+            ui.add_space(6.0);
+        }
+
+        if let Some(alias) = refresh_alias {
+            self.refresh_server(&alias);
+        }
+    }
+
+    fn render_empty_state(&mut self, ui: &mut egui::Ui) {
+        ui.vertical_centered(|ui| {
+            ui.heading("未发现 SSH alias");
+            ui.label("请在 ~/.ssh/config 中配置 Host alias 后重新读取。");
+            if ui.button("重新读取").clicked() {
+                self.reload_aliases();
+            }
+        });
+    }
+
+    fn render_server_detail(&mut self, ui: &mut egui::Ui, index: usize) {
+        let alias = self.servers[index].alias.name.clone();
+        let status = self.servers[index].status();
+        let loading = self.servers[index].loading;
+        let report = self.servers[index].report.clone();
+        let error = self.servers[index].error.clone();
+        let action_log = self.servers[index].action_log.clone();
+
+        ui.horizontal(|ui| {
+            ui.heading(&alias);
+            ui.colored_label(level_color(status), status.label());
+            if loading {
+                ui.spinner();
+            }
+        });
+        ui.add_space(8.0);
+
+        if let Some(error) = error {
+            ui.colored_label(warning_color(), format!("SSH 巡检失败：{error}"));
+            ui.add_space(8.0);
+        }
+
+        if let Some(report) = report {
+            self.render_report(ui, &alias, &report);
+        } else {
+            ui.label("尚未执行巡检。");
+        }
+
+        ui.add_space(12.0);
+        self.render_service_controls(ui, &alias, index);
+
+        if let Some(log) = action_log {
+            ui.add_space(12.0);
+            egui::CollapsingHeader::new("最近一次服务操作输出")
+                .default_open(true)
+                .show(ui, |ui| {
+                    ui.add(
+                        egui::TextEdit::multiline(&mut log.clone())
+                            .font(egui::TextStyle::Monospace)
+                            .desired_rows(8)
+                            .interactive(false),
+                    );
+                });
+        }
+    }
+
+    fn render_report(&self, ui: &mut egui::Ui, alias: &str, report: &ServerHealthReport) {
+        egui::ScrollArea::vertical().show(ui, |ui| {
+            ui.horizontal_wrapped(|ui| {
+                info_chip(ui, "主机", value_or_dash(&report.host.hostname));
+                info_chip(ui, "内核", value_or_dash(&report.host.kernel));
+                info_chip(ui, "运行时间", value_or_dash(&report.host.uptime));
+                info_chip(ui, "检查时间", value_or_dash(&report.checked_at));
+            });
+
+            ui.add_space(10.0);
+            egui::CollapsingHeader::new("硬件状态")
+                .default_open(true)
+                .show(ui, |ui| {
+                    ui.horizontal_wrapped(|ui| {
+                        info_chip(ui, "CPU", value_or_dash(&report.hardware.cpu_model));
+                        info_chip(ui, "核心", value_or_dash(&report.hardware.cpu_cores));
+                        info_chip(ui, "负载", value_or_dash(&report.hardware.load_average));
+                    });
+                    ui.add_space(6.0);
+                    memory_row(ui, "内存", &report.hardware.memory);
+                    memory_row(ui, "Swap", &report.hardware.swap);
+                    ui.add_space(6.0);
+                    for disk in &report.hardware.disks {
+                        disk_row(ui, disk);
+                    }
+                    ui.add_space(6.0);
+                    for sensor in &report.hardware.sensors {
+                        ui.label(sensor);
+                    }
+                });
+
+            egui::CollapsingHeader::new("服务状态")
+                .default_open(true)
+                .show(ui, |ui| {
+                    egui::Grid::new(format!("{alias}_services"))
+                        .striped(true)
+                        .show(ui, |ui| {
+                            ui.strong("服务");
+                            ui.strong("状态");
+                            ui.strong("子状态");
+                            ui.strong("Unit");
+                            ui.end_row();
+                            for service in &report.services {
+                                service_row(ui, service);
+                            }
+                        });
+                });
+
+            egui::CollapsingHeader::new("HTTP 探测")
+                .default_open(true)
+                .show(ui, |ui| {
+                    egui::Grid::new(format!("{alias}_probes"))
+                        .striped(true)
+                        .show(ui, |ui| {
+                            ui.strong("探测");
+                            ui.strong("状态码");
+                            ui.strong("耗时");
+                            ui.strong("目标");
+                            ui.end_row();
+                            for probe in &report.probes {
+                                probe_row(ui, probe);
+                            }
+                        });
+                });
+
+            if let Some(patrol) = &report.health_patrol {
+                egui::CollapsingHeader::new("生产健康巡检")
+                    .default_open(true)
+                    .show(ui, |ui| {
+                        ui.horizontal(|ui| {
+                            ui.colored_label(level_color(patrol.level), &patrol.status);
+                            ui.label(value_or_dash(&patrol.checked_at));
+                            ui.label(value_or_dash(&patrol.summary));
+                        });
+                    });
+            }
+
+            egui::CollapsingHeader::new("原始巡检输出").show(ui, |ui| {
+                ui.add(
+                    egui::TextEdit::multiline(&mut report.raw_output.clone())
+                        .font(egui::TextStyle::Monospace)
+                        .desired_rows(12)
+                        .interactive(false),
+                );
+            });
+        });
+    }
+
+    fn render_service_controls(&mut self, ui: &mut egui::Ui, alias: &str, index: usize) {
+        ui.heading("服务控制");
+        ui.add_space(4.0);
+
+        let action_in_progress = self.servers[index].action_in_progress.clone();
+        for service in DEFAULT_MANAGED_SERVICES {
+            ui.horizontal(|ui| {
+                ui.label(*service);
+                for action in [
+                    ServiceAction::Start,
+                    ServiceAction::Stop,
+                    ServiceAction::Restart,
+                ] {
+                    let disabled = action_in_progress.is_some();
+                    if ui
+                        .add_enabled(!disabled, egui::Button::new(action.label()))
+                        .clicked()
+                    {
+                        self.pending_confirmation = Some(ServiceConfirmation {
+                            alias: alias.to_owned(),
+                            service: (*service).to_owned(),
+                            action,
+                        });
+                    }
+                }
+            });
+        }
+
+        ui.add_space(8.0);
+        ui.horizontal(|ui| {
+            ui.label("其他 unit");
+            ui.text_edit_singleline(&mut self.custom_service_name);
+            if ui.button("启动").clicked() {
+                self.confirm_custom_service(alias, ServiceAction::Start);
+            }
+            if ui.button("关闭").clicked() {
+                self.confirm_custom_service(alias, ServiceAction::Stop);
+            }
+            if ui.button("重启").clicked() {
+                self.confirm_custom_service(alias, ServiceAction::Restart);
+            }
+        });
+
+        if let Some(action) = action_in_progress {
+            ui.label(format!("正在执行：{action}"));
+        }
+    }
+
+    fn render_confirm_dialog(&mut self, ctx: &egui::Context) {
+        let Some(confirmation) = self.pending_confirmation.clone() else {
+            return;
+        };
+
+        egui::Window::new("确认服务操作")
+            .collapsible(false)
+            .resizable(false)
+            .show(ctx, |ui| {
+                ui.label(format!(
+                    "确认在 {} 上{} {}？",
+                    confirmation.alias,
+                    confirmation.action.label(),
+                    confirmation.service
+                ));
+                ui.add_space(8.0);
+                ui.horizontal(|ui| {
+                    if ui.button("确认").clicked() {
+                        self.execute_service_action(&confirmation);
+                        self.pending_confirmation = None;
+                    }
+                    if ui.button("取消").clicked() {
+                        self.pending_confirmation = None;
+                    }
+                });
+            });
+    }
+
+    fn reload_aliases(&mut self) {
+        let aliases = discover_ssh_aliases();
+        self.servers = aliases.into_iter().map(ServerState::new).collect();
+        self.selected_alias = self.servers.first().map(|server| server.alias.name.clone());
+    }
+
+    fn refresh_server(&mut self, alias: &str) {
+        if let Some(server) = self.server_mut(alias) {
+            server.loading = true;
+            server.error = None;
+        }
+        spawn_health_check(alias.to_owned(), self.tx.clone());
+    }
+
+    fn confirm_custom_service(&mut self, alias: &str, action: ServiceAction) {
+        let service = self.custom_service_name.trim();
+        if service.is_empty() {
+            return;
+        }
+        self.pending_confirmation = Some(ServiceConfirmation {
+            alias: alias.to_owned(),
+            service: service.to_owned(),
+            action,
+        });
+    }
+
+    fn execute_service_action(&mut self, confirmation: &ServiceConfirmation) {
+        if let Some(server) = self.server_mut(&confirmation.alias) {
+            server.action_in_progress = Some(format!(
+                "{} {}",
+                confirmation.action.label(),
+                confirmation.service
+            ));
+            server.action_log = None;
+        }
+        spawn_service_action(
+            confirmation.alias.clone(),
+            confirmation.service.clone(),
+            confirmation.action,
+            self.tx.clone(),
+        );
+    }
+
+    fn server_index(&self, alias: &str) -> Option<usize> {
+        self.servers
+            .iter()
+            .position(|server| server.alias.name == alias)
+    }
+
+    fn server_mut(&mut self, alias: &str) -> Option<&mut ServerState> {
+        self.servers
+            .iter_mut()
+            .find(|server| server.alias.name == alias)
+    }
+}
+
+#[derive(Debug, Clone)]
+struct ServiceConfirmation {
+    alias: String,
+    service: String,
+    action: ServiceAction,
+}
+
+#[derive(Debug)]
+struct ServerState {
+    alias: SshAlias,
+    report: Option<ServerHealthReport>,
+    loading: bool,
+    error: Option<String>,
+    action_in_progress: Option<String>,
+    action_log: Option<String>,
+}
+
+impl ServerState {
+    fn new(alias: SshAlias) -> Self {
+        Self {
+            alias,
+            report: None,
+            loading: false,
+            error: None,
+            action_in_progress: None,
+            action_log: None,
+        }
+    }
+
+    fn status(&self) -> HealthLevel {
+        if self.error.is_some() {
+            HealthLevel::Critical
+        } else {
+            self.report
+                .as_ref()
+                .map(|report| report.status)
+                .unwrap_or(HealthLevel::Unknown)
+        }
+    }
+}
+
+fn server_label(server: &ServerState) -> String {
+    let prefix = match server.status() {
+        HealthLevel::Ok => "[OK]",
+        HealthLevel::Warning => "[!]",
+        HealthLevel::Critical => "[X]",
+        HealthLevel::Unknown => "[?]",
+    };
+    format!("{prefix} {}", server.alias.name)
+}
+
+fn service_row(ui: &mut egui::Ui, service: &ServiceSnapshot) {
+    ui.label(&service.name);
+    ui.colored_label(level_color(service.level), &service.active);
+    ui.label(&service.sub);
+    ui.label(&service.unit_file);
+    ui.end_row();
+}
+
+fn probe_row(ui: &mut egui::Ui, probe: &ProbeSnapshot) {
+    ui.label(&probe.name);
+    ui.colored_label(level_color(probe.level), &probe.http_code);
+    ui.label(
+        probe
+            .elapsed_ms
+            .map(|elapsed| format!("{elapsed}ms"))
+            .unwrap_or_else(|| "-".to_owned()),
+    );
+    ui.label(&probe.target);
+    ui.end_row();
+}
+
+fn memory_row(ui: &mut egui::Ui, label: &str, memory: &MemorySnapshot) {
+    let percent = memory.used_percent.unwrap_or_default();
+    ui.horizontal(|ui| {
+        ui.label(label);
+        ui.add(egui::ProgressBar::new(f32::from(percent) / 100.0).text(format!("{percent}%")));
+        ui.label(format!(
+            "已用 {} / 总计 {}，可用 {}",
+            value_or_dash(&memory.used),
+            value_or_dash(&memory.total),
+            value_or_dash(&memory.available)
+        ));
+    });
+}
+
+fn disk_row(ui: &mut egui::Ui, disk: &DiskSnapshot) {
+    let percent = disk.used_percent.unwrap_or_default();
+    ui.horizontal(|ui| {
+        ui.label(&disk.mount);
+        ui.add(egui::ProgressBar::new(f32::from(percent) / 100.0).text(format!("{percent}%")));
+        ui.label(format!(
+            "{} 已用 {} / {}，可用 {}",
+            disk.filesystem, disk.used, disk.size, disk.available
+        ));
+    });
+}
+
+fn info_chip(ui: &mut egui::Ui, label: &str, value: &str) {
+    ui.group(|ui| {
+        ui.vertical(|ui| {
+            ui.small(label);
+            ui.label(value);
+        });
+    });
+}
+
+fn value_or_dash(value: &str) -> &str {
+    if value.trim().is_empty() { "-" } else { value }
+}
+
+fn level_color(level: HealthLevel) -> egui::Color32 {
+    match level {
+        HealthLevel::Ok => egui::Color32::from_rgb(38, 166, 91),
+        HealthLevel::Warning => egui::Color32::from_rgb(214, 137, 16),
+        HealthLevel::Critical => egui::Color32::from_rgb(205, 66, 70),
+        HealthLevel::Unknown => egui::Color32::from_rgb(120, 126, 136),
+    }
+}
+
+fn warning_color() -> egui::Color32 {
+    egui::Color32::from_rgb(205, 66, 70)
+}

server-rs/crates/server-manager-panel/src/fonts.rs

@@ -0,0 +1,128 @@
+use std::path::{Path, PathBuf};
+use std::process::Command;
+use std::sync::Arc;
+
+use eframe::egui::{FontData, FontDefinitions, FontFamily};
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CjkFontCandidate {
+    pub path: PathBuf,
+    pub index: u32,
+}
+
+pub fn install_cjk_font(ctx: &eframe::egui::Context) -> Option<CjkFontCandidate> {
+    let candidate = find_cjk_font_candidate()?;
+    let bytes = std::fs::read(&candidate.path).ok()?;
+    let mut font_data = FontData::from_owned(bytes);
+    font_data.index = candidate.index;
+
+    let mut definitions = FontDefinitions::default();
+    definitions
+        .font_data
+        .insert("genarrative-cjk".to_owned(), Arc::new(font_data));
+
+    // 中文注释：作为 fallback 注入，保留 egui 默认拉丁/图标字体，同时补齐中文 glyph。
+    for family in [FontFamily::Proportional, FontFamily::Monospace] {
+        definitions
+            .families
+            .entry(family)
+            .or_default()
+            .push("genarrative-cjk".to_owned());
+    }
+
+    ctx.set_fonts(definitions);
+    Some(candidate)
+}
+
+pub fn find_cjk_font_candidate() -> Option<CjkFontCandidate> {
+    if let Ok(path) = std::env::var("GENARRATIVE_SERVER_PANEL_CJK_FONT") {
+        if let Some(candidate) = parse_font_spec(&path) {
+            return Some(candidate);
+        }
+    }
+
+    const KNOWN_PATHS: &[(&str, u32)] = &[
+        ("/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc", 2),
+        ("/usr/share/fonts/opentype/noto/NotoSansCJK-Medium.ttc", 2),
+        ("/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc", 0),
+        ("/usr/share/fonts/truetype/wqy/wqy-microhei.ttc", 0),
+        (
+            "/usr/share/fonts/truetype/droid/DroidSansFallbackFull.ttf",
+            0,
+        ),
+        (
+            "/home/dsk/.local/share/fonts/genarrative-cjk/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc",
+            0,
+        ),
+    ];
+
+    for (path, index) in KNOWN_PATHS {
+        if Path::new(path).is_file() {
+            return Some(CjkFontCandidate {
+                path: PathBuf::from(path),
+                index: *index,
+            });
+        }
+    }
+
+    for family in [
+        "Noto Sans CJK SC",
+        "WenQuanYi Zen Hei",
+        "Droid Sans Fallback",
+    ] {
+        if let Some(candidate) = find_with_fc_match(family) {
+            return Some(candidate);
+        }
+    }
+
+    None
+}
+
+fn parse_font_spec(raw: &str) -> Option<CjkFontCandidate> {
+    let trimmed = raw.trim();
+    if trimmed.is_empty() {
+        return None;
+    }
+    let (path, index) = trimmed
+        .rsplit_once('|')
+        .and_then(|(path, index)| Some((path, index.parse().ok()?)))
+        .unwrap_or((trimmed, 0));
+    let path = PathBuf::from(path);
+    path.is_file().then_some(CjkFontCandidate { path, index })
+}
+
+fn find_with_fc_match(family: &str) -> Option<CjkFontCandidate> {
+    let output = Command::new("fc-match")
+        .arg("-f")
+        .arg("%{file}|%{index}\n")
+        .arg(family)
+        .output()
+        .ok()?;
+    if !output.status.success() {
+        return None;
+    }
+    let stdout = String::from_utf8_lossy(&output.stdout);
+    stdout.lines().find_map(parse_font_spec)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn parses_font_path_with_index() {
+        let candidate = parse_font_spec("/tmp/missing-font.ttc|2");
+        assert_eq!(candidate, None);
+    }
+
+    #[test]
+    fn finds_existing_system_cjk_font() {
+        let candidate = find_cjk_font_candidate();
+        assert!(
+            candidate
+                .as_ref()
+                .is_some_and(|candidate| candidate.path.is_file()),
+            "expected at least one CJK font on this development host"
+        );
+    }
+}

server-rs/crates/server-manager-panel/src/health.rs

@@ -0,0 +1,474 @@
+use std::collections::BTreeMap;
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
+pub enum HealthLevel {
+    Unknown,
+    Ok,
+    Warning,
+    Critical,
+}
+
+impl HealthLevel {
+    pub fn label(self) -> &'static str {
+        match self {
+            HealthLevel::Unknown => "未知",
+            HealthLevel::Ok => "正常",
+            HealthLevel::Warning => "警告",
+            HealthLevel::Critical => "异常",
+        }
+    }
+
+    pub fn rank(self) -> u8 {
+        match self {
+            HealthLevel::Unknown => 1,
+            HealthLevel::Ok => 0,
+            HealthLevel::Warning => 2,
+            HealthLevel::Critical => 3,
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct ServerHealthReport {
+    pub status: HealthLevel,
+    pub checked_at: String,
+    pub host: HostSnapshot,
+    pub hardware: HardwareSnapshot,
+    pub services: Vec<ServiceSnapshot>,
+    pub probes: Vec<ProbeSnapshot>,
+    pub health_patrol: Option<HealthPatrolSnapshot>,
+    pub raw_output: String,
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct HostSnapshot {
+    pub hostname: String,
+    pub kernel: String,
+    pub uptime: String,
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct HardwareSnapshot {
+    pub cpu_model: String,
+    pub cpu_cores: String,
+    pub load_average: String,
+    pub memory: MemorySnapshot,
+    pub swap: MemorySnapshot,
+    pub disks: Vec<DiskSnapshot>,
+    pub sensors: Vec<String>,
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct MemorySnapshot {
+    pub total: String,
+    pub used: String,
+    pub free: String,
+    pub available: String,
+    pub used_percent: Option<u8>,
+}
+
+#[derive(Debug, Clone, Default)]
+pub struct DiskSnapshot {
+    pub mount: String,
+    pub filesystem: String,
+    pub size: String,
+    pub used: String,
+    pub available: String,
+    pub used_percent: Option<u8>,
+}
+
+#[derive(Debug, Clone)]
+pub struct ServiceSnapshot {
+    pub name: String,
+    pub active: String,
+    pub sub: String,
+    pub unit_file: String,
+    pub level: HealthLevel,
+}
+
+#[derive(Debug, Clone)]
+pub struct ProbeSnapshot {
+    pub name: String,
+    pub target: String,
+    pub http_code: String,
+    pub elapsed_ms: Option<u64>,
+    pub level: HealthLevel,
+}
+
+#[derive(Debug, Clone)]
+pub struct HealthPatrolSnapshot {
+    pub status: String,
+    pub checked_at: String,
+    pub summary: String,
+    pub level: HealthLevel,
+}
+
+pub fn parse_health_report(raw_output: &str) -> ServerHealthReport {
+    let mut sections: BTreeMap<String, Vec<String>> = BTreeMap::new();
+    let mut current = String::new();
+
+    for line in raw_output.lines() {
+        if let Some(name) = parse_section_marker(line) {
+            current = name.to_owned();
+            sections.entry(current.clone()).or_default();
+        } else if !current.is_empty() {
+            sections
+                .entry(current.clone())
+                .or_default()
+                .push(line.to_owned());
+        }
+    }
+
+    let mut report = ServerHealthReport {
+        status: HealthLevel::Unknown,
+        checked_at: section_value(&sections, "checked_at").unwrap_or_default(),
+        host: parse_host(&sections),
+        hardware: parse_hardware(&sections),
+        services: parse_services(&sections),
+        probes: parse_probes(&sections),
+        health_patrol: parse_health_patrol(&sections),
+        raw_output: raw_output.to_owned(),
+    };
+    report.status = summarize_report(&report);
+    report
+}
+
+pub fn summarize_report(report: &ServerHealthReport) -> HealthLevel {
+    let mut status = HealthLevel::Ok;
+    for level in report
+        .services
+        .iter()
+        .map(|service| service.level)
+        .chain(report.probes.iter().map(|probe| probe.level))
+        .chain(report.health_patrol.iter().map(|patrol| patrol.level))
+    {
+        if level.rank() > status.rank() {
+            status = level;
+        }
+    }
+
+    if let Some(used_percent) = report.hardware.memory.used_percent {
+        let memory_level = if used_percent >= 95 {
+            HealthLevel::Critical
+        } else if used_percent >= 85 {
+            HealthLevel::Warning
+        } else {
+            HealthLevel::Ok
+        };
+        if memory_level.rank() > status.rank() {
+            status = memory_level;
+        }
+    }
+
+    for disk in &report.hardware.disks {
+        let disk_level = match disk.used_percent {
+            Some(percent) if percent >= 95 => HealthLevel::Critical,
+            Some(percent) if percent >= 85 => HealthLevel::Warning,
+            _ => HealthLevel::Ok,
+        };
+        if disk_level.rank() > status.rank() {
+            status = disk_level;
+        }
+    }
+
+    status
+}
+
+fn parse_section_marker(line: &str) -> Option<&str> {
+    line.strip_prefix("==GENARRATIVE_PANEL:")
+        .and_then(|rest| rest.strip_suffix("=="))
+}
+
+fn section_value(sections: &BTreeMap<String, Vec<String>>, name: &str) -> Option<String> {
+    sections.get(name).and_then(|lines| {
+        lines
+            .iter()
+            .map(|line| line.trim())
+            .find(|line| !line.is_empty())
+            .map(str::to_owned)
+    })
+}
+
+fn parse_host(sections: &BTreeMap<String, Vec<String>>) -> HostSnapshot {
+    HostSnapshot {
+        hostname: section_value(sections, "hostname").unwrap_or_default(),
+        kernel: section_value(sections, "kernel").unwrap_or_default(),
+        uptime: section_value(sections, "uptime").unwrap_or_default(),
+    }
+}
+
+fn parse_hardware(sections: &BTreeMap<String, Vec<String>>) -> HardwareSnapshot {
+    HardwareSnapshot {
+        cpu_model: section_value(sections, "cpu_model").unwrap_or_default(),
+        cpu_cores: section_value(sections, "cpu_cores").unwrap_or_default(),
+        load_average: section_value(sections, "load_average").unwrap_or_default(),
+        memory: parse_memory(section_value(sections, "memory").as_deref()),
+        swap: parse_memory(section_value(sections, "swap").as_deref()),
+        disks: parse_disks(sections),
+        sensors: sections.get("sensors").cloned().unwrap_or_default(),
+    }
+}
+
+fn parse_memory(value: Option<&str>) -> MemorySnapshot {
+    let Some(value) = value else {
+        return MemorySnapshot::default();
+    };
+    let parts: Vec<&str> = value.split('|').collect();
+    MemorySnapshot {
+        total: parts.first().copied().unwrap_or_default().to_owned(),
+        used: parts.get(1).copied().unwrap_or_default().to_owned(),
+        free: parts.get(2).copied().unwrap_or_default().to_owned(),
+        available: parts.get(3).copied().unwrap_or_default().to_owned(),
+        used_percent: parts.get(4).and_then(|value| parse_percent(value)),
+    }
+}
+
+fn parse_disks(sections: &BTreeMap<String, Vec<String>>) -> Vec<DiskSnapshot> {
+    sections
+        .get("disks")
+        .into_iter()
+        .flatten()
+        .filter_map(|line| {
+            let parts: Vec<&str> = line.split('|').collect();
+            (parts.len() >= 6).then(|| DiskSnapshot {
+                filesystem: parts[0].to_owned(),
+                size: parts[1].to_owned(),
+                used: parts[2].to_owned(),
+                available: parts[3].to_owned(),
+                used_percent: parse_percent(parts[4]),
+                mount: parts[5].to_owned(),
+            })
+        })
+        .collect()
+}
+
+fn parse_services(sections: &BTreeMap<String, Vec<String>>) -> Vec<ServiceSnapshot> {
+    sections
+        .get("services")
+        .into_iter()
+        .flatten()
+        .filter_map(|line| {
+            let parts: Vec<&str> = line.split('|').collect();
+            (parts.len() >= 4).then(|| {
+                let active = parts[1].to_owned();
+                let sub = parts[2].to_owned();
+                let level = if active == "active" {
+                    HealthLevel::Ok
+                } else if active == "unknown" || active == "inactive" {
+                    HealthLevel::Warning
+                } else {
+                    HealthLevel::Critical
+                };
+                ServiceSnapshot {
+                    name: parts[0].to_owned(),
+                    active,
+                    sub,
+                    unit_file: parts[3].to_owned(),
+                    level,
+                }
+            })
+        })
+        .collect()
+}
+
+fn parse_probes(sections: &BTreeMap<String, Vec<String>>) -> Vec<ProbeSnapshot> {
+    sections
+        .get("probes")
+        .into_iter()
+        .flatten()
+        .filter_map(|line| {
+            let parts: Vec<&str> = line.split('|').collect();
+            (parts.len() >= 4).then(|| {
+                let http_code = parts[2].to_owned();
+                let elapsed_ms = parts[3].parse().ok();
+                let level = if http_code.starts_with('2') {
+                    HealthLevel::Ok
+                } else if http_code == "000" {
+                    HealthLevel::Critical
+                } else {
+                    HealthLevel::Critical
+                };
+                ProbeSnapshot {
+                    name: parts[0].to_owned(),
+                    target: parts[1].to_owned(),
+                    http_code,
+                    elapsed_ms,
+                    level,
+                }
+            })
+        })
+        .collect()
+}
+
+fn parse_health_patrol(sections: &BTreeMap<String, Vec<String>>) -> Option<HealthPatrolSnapshot> {
+    let line = section_value(sections, "health_patrol")?;
+    let parts: Vec<&str> = line.split('|').collect();
+    let status = parts.first().copied().unwrap_or_default().to_owned();
+    let level = match status.as_str() {
+        "OK" => HealthLevel::Ok,
+        "WARNING" => HealthLevel::Warning,
+        "CRITICAL" => HealthLevel::Critical,
+        _ => HealthLevel::Unknown,
+    };
+    Some(HealthPatrolSnapshot {
+        status,
+        checked_at: parts.get(1).copied().unwrap_or_default().to_owned(),
+        summary: parts.get(2).copied().unwrap_or_default().to_owned(),
+        level,
+    })
+}
+
+fn parse_percent(value: &str) -> Option<u8> {
+    value.trim_end_matches('%').parse().ok()
+}
+
+pub const HEALTH_SCRIPT: &str = r#"set -eu
+
+print_section() {
+  printf '==GENARRATIVE_PANEL:%s==\n' "$1"
+}
+
+print_section checked_at
+date -Is 2>/dev/null || date
+
+print_section hostname
+hostname 2>/dev/null || true
+
+print_section kernel
+uname -srmo 2>/dev/null || uname -a 2>/dev/null || true
+
+print_section uptime
+uptime -p 2>/dev/null || uptime 2>/dev/null || true
+
+print_section cpu_model
+awk -F: '/model name/ {gsub(/^[ \t]+/, "", $2); print $2; exit}' /proc/cpuinfo 2>/dev/null || true
+
+print_section cpu_cores
+nproc 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null || true
+
+print_section load_average
+cat /proc/loadavg 2>/dev/null | awk '{print $1" "$2" "$3}' || true
+
+print_section memory
+awk '
+  /^MemTotal:/ {total=$2}
+  /^MemFree:/ {free=$2}
+  /^MemAvailable:/ {available=$2}
+  END {
+    if (total > 0) {
+      used = total - free
+      percent = int((used * 100 + total / 2) / total)
+      printf "%.1f GiB|%.1f GiB|%.1f GiB|%.1f GiB|%d%%\n", total/1048576, used/1048576, free/1048576, available/1048576, percent
+    }
+  }
+' /proc/meminfo 2>/dev/null || true
+
+print_section swap
+awk '
+  /^SwapTotal:/ {total=$2}
+  /^SwapFree:/ {free=$2}
+  END {
+    if (total > 0) {
+      used = total - free
+      percent = int((used * 100 + total / 2) / total)
+      printf "%.1f GiB|%.1f GiB|%.1f GiB|%.1f GiB|%d%%\n", total/1048576, used/1048576, free/1048576, free/1048576, percent
+    } else {
+      print "0 GiB|0 GiB|0 GiB|0 GiB|0%"
+    }
+  }
+' /proc/meminfo 2>/dev/null || true
+
+print_section disks
+for mount in / /var /opt /stdb /data; do
+  if [ -e "$mount" ]; then
+    df -hP "$mount" 2>/dev/null | awk 'NR == 2 {print $1"|"$2"|"$3"|"$4"|"$5"|"$6}'
+  fi
+done | awk '!seen[$6]++'
+
+print_section sensors
+if command -v sensors >/dev/null 2>&1; then
+  sensors 2>/dev/null | sed -n '1,20p'
+else
+  echo "sensors 未安装"
+fi
+
+print_section services
+for service in genarrative-api.service spacetimedb.service nginx.service genarrative-health-patrol.timer genarrative-database-backup.timer; do
+  active=$(systemctl is-active "$service" 2>/dev/null || true)
+  sub=$(systemctl show "$service" -p SubState --value 2>/dev/null || true)
+  unit_file=$(systemctl show "$service" -p UnitFileState --value 2>/dev/null || true)
+  [ -n "$active" ] || active="unknown"
+  [ -n "$sub" ] || sub="unknown"
+  [ -n "$unit_file" ] || unit_file="unknown"
+  printf '%s|%s|%s|%s\n' "$service" "$active" "$sub" "$unit_file"
+done
+
+print_section probes
+probe() {
+  name="$1"
+  url="$2"
+  tmp=$(mktemp)
+  code=$(curl -fsS -m 5 -o /dev/null -w '%{http_code}|%{time_total}' "$url" 2>"$tmp" || true)
+  if [ -z "$code" ]; then
+    code="000|0"
+  fi
+  http_code=${code%%|*}
+  time_total=${code#*|}
+  elapsed_ms=$(awk "BEGIN {printf \"%d\", $time_total * 1000}")
+  printf '%s|%s|%s|%s\n' "$name" "$url" "$http_code" "$elapsed_ms"
+  rm -f "$tmp"
+}
+probe "api:/healthz" "http://127.0.0.1:8082/healthz"
+probe "api:/readyz" "http://127.0.0.1:8082/readyz"
+probe "spacetimedb:/v1/ping" "http://127.0.0.1:3101/v1/ping"
+probe "public:/api/creation-entry/config" "http://127.0.0.1:8082/api/creation-entry/config"
+probe "public:/api/runtime/puzzle/gallery" "http://127.0.0.1:8082/api/runtime/puzzle/gallery"
+
+print_section health_patrol
+if [ -r /var/lib/genarrative/health-patrol/status.json ]; then
+  node -e '
+    const fs = require("fs");
+    const payload = JSON.parse(fs.readFileSync("/var/lib/genarrative/health-patrol/status.json", "utf8"));
+    const status = payload.status || "UNKNOWN";
+    const checkedAt = payload.checkedAt || "";
+    const checks = Array.isArray(payload.checks) ? payload.checks : [];
+    const summary = checks.filter((check) => check.status && check.status !== "OK").slice(0, 3).map((check) => `${check.name}:${check.status}`).join(",");
+    console.log(`${status}|${checkedAt}|${summary}`);
+  ' 2>/dev/null || echo "UNKNOWN||状态文件解析失败"
+else
+  echo "UNKNOWN||未找到 /var/lib/genarrative/health-patrol/status.json"
+fi
+"#;
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn parses_report_sections() {
+        let report = parse_health_report(
+            r#"==GENARRATIVE_PANEL:checked_at==
+2026-06-11T12:00:00+08:00
+==GENARRATIVE_PANEL:hostname==
+release
+==GENARRATIVE_PANEL:memory==
+2.0 GiB|1.0 GiB|1.0 GiB|1.0 GiB|50%
+==GENARRATIVE_PANEL:disks==
+/dev/sda1|40G|20G|20G|50%|/
+==GENARRATIVE_PANEL:services==
+genarrative-api.service|active|running|enabled
+spacetimedb.service|failed|failed|enabled
+==GENARRATIVE_PANEL:probes==
+api:/readyz|http://127.0.0.1:8082/readyz|200|18
+==GENARRATIVE_PANEL:health_patrol==
+WARNING|2026-06-11T11:59:00Z|journal:WARNING
+"#,
+        );
+
+        assert_eq!(report.host.hostname, "release");
+        assert_eq!(report.hardware.memory.used_percent, Some(50));
+        assert_eq!(report.services.len(), 2);
+        assert_eq!(report.probes[0].http_code, "200");
+        assert_eq!(report.status, HealthLevel::Critical);
+    }
+}

server-rs/crates/server-manager-panel/src/lib.rs

@@ -0,0 +1,5 @@
+pub mod app;
+pub mod fonts;
+pub mod health;
+pub mod remote;
+pub mod ssh_config;

server-rs/crates/server-manager-panel/src/main.rs

@@ -0,0 +1,21 @@
+use eframe::egui;
+use server_manager_panel::app::ServerManagerApp;
+use server_manager_panel::fonts::install_cjk_font;
+
+fn main() -> eframe::Result<()> {
+    let native_options = eframe::NativeOptions {
+        viewport: egui::ViewportBuilder::default()
+            .with_inner_size([1180.0, 760.0])
+            .with_min_inner_size([920.0, 620.0]),
+        ..Default::default()
+    };
+
+    eframe::run_native(
+        "Genarrative 服务器管理面板",
+        native_options,
+        Box::new(|cc| {
+            install_cjk_font(&cc.egui_ctx);
+            Ok(Box::new(ServerManagerApp::default()))
+        }),
+    )
+}

server-rs/crates/server-manager-panel/src/remote.rs

@@ -0,0 +1,231 @@
+use std::io::Write;
+use std::process::{Command, Stdio};
+use std::sync::mpsc;
+use std::thread;
+use std::time::{Duration, Instant};
+
+use crate::health::{HEALTH_SCRIPT, ServerHealthReport, parse_health_report};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ServiceAction {
+    Start,
+    Stop,
+    Restart,
+}
+
+impl ServiceAction {
+    pub fn as_systemctl_arg(self) -> &'static str {
+        match self {
+            ServiceAction::Start => "start",
+            ServiceAction::Stop => "stop",
+            ServiceAction::Restart => "restart",
+        }
+    }
+
+    pub fn label(self) -> &'static str {
+        match self {
+            ServiceAction::Start => "启动",
+            ServiceAction::Stop => "关闭",
+            ServiceAction::Restart => "重启",
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct RemoteCommandResult {
+    pub success: bool,
+    pub summary: String,
+    pub stdout: String,
+    pub stderr: String,
+}
+
+#[derive(Debug)]
+pub enum RemoteEvent {
+    Health {
+        alias: String,
+        result: Result<ServerHealthReport, String>,
+    },
+    ServiceAction {
+        alias: String,
+        service: String,
+        action: ServiceAction,
+        result: RemoteCommandResult,
+    },
+}
+
+pub type RemoteSender = mpsc::Sender<RemoteEvent>;
+pub type RemoteReceiver = mpsc::Receiver<RemoteEvent>;
+
+pub fn channel() -> (RemoteSender, RemoteReceiver) {
+    mpsc::channel()
+}
+
+pub fn spawn_health_check(alias: String, tx: RemoteSender) {
+    thread::spawn(move || {
+        let result =
+            run_ssh_script(&alias, HEALTH_SCRIPT, Duration::from_secs(20)).and_then(|output| {
+                if output.success {
+                    Ok(parse_health_report(&output.stdout))
+                } else {
+                    Err(format_remote_error(&output))
+                }
+            });
+        let _ = tx.send(RemoteEvent::Health { alias, result });
+    });
+}
+
+pub fn spawn_service_action(
+    alias: String,
+    service: String,
+    action: ServiceAction,
+    tx: RemoteSender,
+) {
+    thread::spawn(move || {
+        let result = if is_safe_service_name(&service) {
+            run_ssh_script(
+                &alias,
+                &build_service_action_script(&service, action),
+                Duration::from_secs(20),
+            )
+            .unwrap_or_else(|error| RemoteCommandResult {
+                success: false,
+                summary: error,
+                stdout: String::new(),
+                stderr: String::new(),
+            })
+        } else {
+            RemoteCommandResult {
+                success: false,
+                summary: "服务名包含不允许的字符".to_owned(),
+                stdout: String::new(),
+                stderr: String::new(),
+            }
+        };
+        let _ = tx.send(RemoteEvent::ServiceAction {
+            alias,
+            service,
+            action,
+            result,
+        });
+    });
+}
+
+pub fn is_safe_service_name(service: &str) -> bool {
+    !service.is_empty()
+        && service.len() <= 128
+        && service.bytes().all(|byte| {
+            matches!(
+                byte,
+                b'a'..=b'z'
+                    | b'A'..=b'Z'
+                    | b'0'..=b'9'
+                    | b'.'
+                    | b'_'
+                    | b'-'
+                    | b'@'
+                    | b':'
+            )
+        })
+}
+
+fn build_service_action_script(service: &str, action: ServiceAction) -> String {
+    format!(
+        r#"set -eu
+service='{service}'
+action='{action}'
+if [ "$(id -u)" = "0" ]; then
+  systemctl "$action" "$service"
+else
+  sudo -n systemctl "$action" "$service"
+fi
+systemctl is-active "$service" || true
+systemctl status "$service" --no-pager -l -n 12 || true
+"#,
+        service = service,
+        action = action.as_systemctl_arg()
+    )
+}
+
+fn run_ssh_script(
+    alias: &str,
+    script: &str,
+    timeout: Duration,
+) -> Result<RemoteCommandResult, String> {
+    let started = Instant::now();
+    let mut child = Command::new("ssh")
+        .arg("-o")
+        .arg("BatchMode=yes")
+        .arg("-o")
+        .arg("ConnectTimeout=8")
+        .arg(alias)
+        .arg("sh")
+        .arg("-s")
+        .stdin(Stdio::piped())
+        .stdout(Stdio::piped())
+        .stderr(Stdio::piped())
+        .spawn()
+        .map_err(|error| format!("无法启动 ssh: {error}"))?;
+
+    {
+        // 中文注释：写完脚本后必须关闭 stdin，让远端 `sh -s` 收到 EOF 并开始退出。
+        let Some(mut stdin) = child.stdin.take() else {
+            return Err("无法写入 ssh stdin".to_owned());
+        };
+        stdin
+            .write_all(script.as_bytes())
+            .map_err(|error| format!("写入远端脚本失败: {error}"))?;
+    }
+
+    loop {
+        match child.try_wait() {
+            Ok(Some(_status)) => {
+                let output = child
+                    .wait_with_output()
+                    .map_err(|error| format!("读取 ssh 输出失败: {error}"))?;
+                let success = output.status.success();
+                return Ok(RemoteCommandResult {
+                    success,
+                    summary: if success {
+                        "执行成功".to_owned()
+                    } else {
+                        format!("ssh 退出码 {:?}", output.status.code())
+                    },
+                    stdout: String::from_utf8_lossy(&output.stdout).into_owned(),
+                    stderr: String::from_utf8_lossy(&output.stderr).into_owned(),
+                });
+            }
+            Ok(None) if started.elapsed() >= timeout => {
+                let _ = child.kill();
+                let _ = child.wait();
+                return Err(format!("ssh 执行超过 {} 秒", timeout.as_secs()));
+            }
+            Ok(None) => thread::sleep(Duration::from_millis(80)),
+            Err(error) => return Err(format!("等待 ssh 进程失败: {error}")),
+        }
+    }
+}
+
+fn format_remote_error(result: &RemoteCommandResult) -> String {
+    let stderr = result.stderr.trim();
+    let stdout = result.stdout.trim();
+    if !stderr.is_empty() {
+        format!("{}: {}", result.summary, stderr)
+    } else if !stdout.is_empty() {
+        format!("{}: {}", result.summary, stdout)
+    } else {
+        result.summary.clone()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn allows_systemd_unit_names_only() {
+        assert!(is_safe_service_name("genarrative-api.service"));
+        assert!(is_safe_service_name("worker@1.service"));
+        assert!(!is_safe_service_name("api.service;rm -rf /"));
+        assert!(!is_safe_service_name(""));
+    }
+}

server-rs/crates/server-manager-panel/src/ssh_config.rs

@@ -0,0 +1,143 @@
+use std::collections::HashSet;
+use std::fs;
+use std::path::{Path, PathBuf};
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct SshAlias {
+    pub name: String,
+    pub source: PathBuf,
+}
+
+pub fn discover_ssh_aliases() -> Vec<SshAlias> {
+    let Some(home) = std::env::var_os("HOME") else {
+        return Vec::new();
+    };
+    let config_path = PathBuf::from(home).join(".ssh/config");
+    discover_from_file(&config_path)
+}
+
+pub fn discover_from_file(path: &Path) -> Vec<SshAlias> {
+    let mut visited = HashSet::new();
+    let mut aliases = Vec::new();
+    discover_inner(path, &mut visited, &mut aliases);
+    dedupe_aliases(aliases)
+}
+
+fn discover_inner(path: &Path, visited: &mut HashSet<PathBuf>, aliases: &mut Vec<SshAlias>) {
+    let Ok(canonical) = path.canonicalize() else {
+        return;
+    };
+    if !visited.insert(canonical.clone()) {
+        return;
+    }
+    let Ok(content) = fs::read_to_string(&canonical) else {
+        return;
+    };
+
+    for line in content.lines() {
+        let trimmed = trim_comment(line);
+        let mut parts = trimmed.split_whitespace();
+        let Some(keyword) = parts.next() else {
+            continue;
+        };
+        if keyword.eq_ignore_ascii_case("host") {
+            aliases.extend(parts.filter_map(|name| {
+                is_concrete_alias(name).then(|| SshAlias {
+                    name: name.to_owned(),
+                    source: canonical.clone(),
+                })
+            }));
+        } else if keyword.eq_ignore_ascii_case("include") {
+            for include in parts {
+                for include_path in expand_include_path(include, canonical.parent()) {
+                    discover_inner(&include_path, visited, aliases);
+                }
+            }
+        }
+    }
+}
+
+fn dedupe_aliases(aliases: Vec<SshAlias>) -> Vec<SshAlias> {
+    let mut seen = HashSet::new();
+    let mut deduped = Vec::new();
+    for alias in aliases {
+        if seen.insert(alias.name.clone()) {
+            deduped.push(alias);
+        }
+    }
+    deduped
+}
+
+fn trim_comment(line: &str) -> &str {
+    line.split('#').next().unwrap_or("").trim()
+}
+
+fn is_concrete_alias(value: &str) -> bool {
+    !value.is_empty()
+        && !value.starts_with('-')
+        && !value.starts_with('!')
+        && !value.contains('*')
+        && !value.contains('?')
+        && !value.contains('%')
+        && !value.contains('/')
+}
+
+fn expand_include_path(raw: &str, parent: Option<&Path>) -> Vec<PathBuf> {
+    if raw.contains('*') || raw.contains('?') {
+        // 中文注释：SSH Include 支持复杂 glob；面板只解析普通文件，避免误扫过大目录。
+        return Vec::new();
+    }
+    let expanded = if let Some(rest) = raw.strip_prefix("~/") {
+        std::env::var_os("HOME")
+            .map(PathBuf::from)
+            .map(|home| home.join(rest))
+    } else {
+        let path = PathBuf::from(raw);
+        if path.is_absolute() {
+            Some(path)
+        } else {
+            parent.map(|base| base.join(path))
+        }
+    };
+    expanded.into_iter().collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn host_parser_ignores_wildcards_and_negations() {
+        let mut aliases = Vec::new();
+        let source = PathBuf::from("/tmp/config");
+        for line in [
+            "Host dev release *.internal !blocked",
+            "Host github.com",
+            "Host ?pattern",
+            "Host -bad",
+        ] {
+            let trimmed = trim_comment(line);
+            let mut parts = trimmed.split_whitespace();
+            let keyword = parts.next().unwrap();
+            if keyword.eq_ignore_ascii_case("host") {
+                aliases.extend(parts.filter_map(|name| {
+                    is_concrete_alias(name).then(|| SshAlias {
+                        name: name.to_owned(),
+                        source: source.clone(),
+                    })
+                }));
+            }
+        }
+
+        let names: Vec<_> = dedupe_aliases(aliases)
+            .into_iter()
+            .map(|alias| alias.name)
+            .collect();
+        assert_eq!(names, ["dev", "release", "github.com"]);
+    }
+
+    #[test]
+    fn comment_trimming_keeps_plain_aliases() {
+        assert_eq!(trim_comment("  Host dev # release host "), "Host dev");
+    }
+}

server-rs/crates/spacetime-client/src/bark_battle.rs

@@ -104,9 +104,7 @@ impl SpacetimeClient {
                             if result.ok {
                                 Ok(())
                             } else {
-                                Err(SpacetimeClientError::procedure_failed(
-                                    result.error_message,
-                                ))
+                                Err(SpacetimeClientError::procedure_failed(result.error_message))
                             }
                         });
                     send_once(&sender, mapped);

server-rs/crates/spacetime-client/src/external_generation.rs

@@ -0,0 +1,148 @@
+use super::*;
+use crate::mapper::*;
+
+impl SpacetimeClient {
+    pub async fn enqueue_external_generation_job(
+        &self,
+        input: ExternalGenerationJobEnqueueRecordInput,
+    ) -> Result<ExternalGenerationJobRecord, SpacetimeClientError> {
+        let procedure_input = input.into();
+
+        self.call_after_connect(
+            "enqueue_external_generation_job_and_return",
+            move |connection, sender| {
+                connection
+                    .procedures()
+                    .enqueue_external_generation_job_and_return_then(
+                        procedure_input,
+                        move |_, result| {
+                            let mapped = result
+                                .map_err(SpacetimeClientError::from_sdk_error)
+                                .and_then(map_external_generation_job_procedure_result);
+                            send_once(&sender, mapped);
+                        },
+                    );
+            },
+        )
+        .await
+    }
+
+    pub async fn claim_external_generation_jobs(
+        &self,
+        input: ExternalGenerationJobClaimRecordInput,
+    ) -> Result<Vec<ExternalGenerationJobRecord>, SpacetimeClientError> {
+        let procedure_input = input.into();
+
+        self.call_after_connect(
+            "claim_external_generation_jobs_and_return",
+            move |connection, sender| {
+                connection
+                    .procedures()
+                    .claim_external_generation_jobs_and_return_then(
+                        procedure_input,
+                        move |_, result| {
+                            let mapped = result
+                                .map_err(SpacetimeClientError::from_sdk_error)
+                                .and_then(map_external_generation_job_claim_result);
+                            send_once(&sender, mapped);
+                        },
+                    );
+            },
+        )
+        .await
+    }
+
+    pub async fn complete_external_generation_job(
+        &self,
+        input: ExternalGenerationJobCompleteRecordInput,
+    ) -> Result<ExternalGenerationJobRecord, SpacetimeClientError> {
+        let procedure_input = input.into();
+
+        self.call_after_connect(
+            "complete_external_generation_job_and_return",
+            move |connection, sender| {
+                connection
+                    .procedures()
+                    .complete_external_generation_job_and_return_then(
+                        procedure_input,
+                        move |_, result| {
+                            let mapped = result
+                                .map_err(SpacetimeClientError::from_sdk_error)
+                                .and_then(map_external_generation_job_procedure_result);
+                            send_once(&sender, mapped);
+                        },
+                    );
+            },
+        )
+        .await
+    }
+
+    pub async fn renew_external_generation_job_lease(
+        &self,
+        input: ExternalGenerationJobRenewLeaseRecordInput,
+    ) -> Result<ExternalGenerationJobRecord, SpacetimeClientError> {
+        let procedure_input = input.into();
+
+        self.call_after_connect(
+            "renew_external_generation_job_lease_and_return",
+            move |connection, sender| {
+                connection
+                    .procedures()
+                    .renew_external_generation_job_lease_and_return_then(
+                        procedure_input,
+                        move |_, result| {
+                            let mapped = result
+                                .map_err(SpacetimeClientError::from_sdk_error)
+                                .and_then(map_external_generation_job_procedure_result);
+                            send_once(&sender, mapped);
+                        },
+                    );
+            },
+        )
+        .await
+    }
+
+    pub async fn fail_external_generation_job(
+        &self,
+        input: ExternalGenerationJobFailRecordInput,
+    ) -> Result<ExternalGenerationJobRecord, SpacetimeClientError> {
+        let procedure_input = input.into();
+
+        self.call_after_connect(
+            "fail_external_generation_job_and_return",
+            move |connection, sender| {
+                connection
+                    .procedures()
+                    .fail_external_generation_job_and_return_then(
+                        procedure_input,
+                        move |_, result| {
+                            let mapped = result
+                                .map_err(SpacetimeClientError::from_sdk_error)
+                                .and_then(map_external_generation_job_procedure_result);
+                            send_once(&sender, mapped);
+                        },
+                    );
+            },
+        )
+        .await
+    }
+
+    pub async fn get_external_generation_queue_stats(
+        &self,
+    ) -> Result<ExternalGenerationQueueStatsRecord, SpacetimeClientError> {
+        self.call_after_connect(
+            "get_external_generation_queue_stats_and_return",
+            move |connection, sender| {
+                connection
+                    .procedures()
+                    .get_external_generation_queue_stats_and_return_then(move |_, result| {
+                        let mapped = result
+                            .map_err(SpacetimeClientError::from_sdk_error)
+                            .and_then(map_external_generation_queue_stats_result);
+                        send_once(&sender, mapped);
+                    });
+            },
+        )
+        .await
+    }
+}

server-rs/crates/spacetime-client/src/jump_hop.rs

@@ -233,15 +233,14 @@ impl SpacetimeClient {
         };
 
         self.call_after_connect("delete_jump_hop_work", move |connection, sender| {
-            connection.procedures().delete_jump_hop_work_then(
-                procedure_input,
-                move |_, result| {
+            connection
+                .procedures()
+                .delete_jump_hop_work_then(procedure_input, move |_, result| {
                     let mapped = result
                         .map_err(SpacetimeClientError::from_sdk_error)
                         .and_then(map_jump_hop_works_procedure_result);
                     send_once(&sender, mapped);
-                },
-            );
+                });
         })
         .await
     }

server-rs/crates/spacetime-client/src/lib.rs

@@ -30,8 +30,12 @@ pub use mapper::{
     CustomWorldPublishGateRecord, CustomWorldPublishWorldRecord,
     CustomWorldPublishWorldRecordInput, CustomWorldPublishedProfileCompileRecord,
     CustomWorldResultPreviewBlockerRecord, CustomWorldSupportedActionRecord,
-    CustomWorldWorkSummaryRecord, JumpHopActionRequest, JumpHopActionResponse, JumpHopActionType,
-    JumpHopCharacterAsset, JumpHopDifficulty, JumpHopDraftResponse, JumpHopGalleryCardResponse,
+    CustomWorldWorkSummaryRecord, ExternalGenerationJobClaimRecordInput,
+    ExternalGenerationJobCompleteRecordInput, ExternalGenerationJobEnqueueRecordInput,
+    ExternalGenerationJobFailRecordInput, ExternalGenerationJobRecord,
+    ExternalGenerationJobRenewLeaseRecordInput, ExternalGenerationQueueStatsRecord,
+    JumpHopActionRequest, JumpHopActionResponse, JumpHopActionType, JumpHopCharacterAsset,
+    JumpHopDifficulty, JumpHopDraftResponse, JumpHopGalleryCardResponse,
     JumpHopGalleryDetailResponse, JumpHopGalleryResponse, JumpHopGenerationStatus,
     JumpHopJumpRequest, JumpHopJumpResponse, JumpHopJumpResult, JumpHopLastJump, JumpHopPath,
     JumpHopPlatform, JumpHopRestartRunRequest, JumpHopRunResponse, JumpHopRunStatus,
@@ -51,7 +55,8 @@ pub use mapper::{
     PublicWorkGalleryEntryRecord, PuzzleAgentMessageFinalizeRecordInput, PuzzleAgentMessageRecord,
     PuzzleAgentMessageSubmitRecordInput, PuzzleAgentSessionCreateRecordInput,
     PuzzleAgentSessionRecord, PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord,
-    PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBoardRecord, PuzzleCellPositionRecord,
+    PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBackgroundCompileTaskClaimRecordInput,
+    PuzzleBackgroundCompileTaskReleaseRecordInput, PuzzleBoardRecord, PuzzleCellPositionRecord,
     PuzzleClearActionRequest, PuzzleClearActionResponse, PuzzleClearActionType,
     PuzzleClearBoardCell, PuzzleClearBoardSnapshot, PuzzleClearCardAsset, PuzzleClearDraftResponse,
     PuzzleClearGenerationStatus, PuzzleClearImageAsset, PuzzleClearNextLevelRequest,
@@ -64,12 +69,13 @@ pub use mapper::{
     PuzzleDraftCompileFailureRecordInput, PuzzleDraftLevelRecord, PuzzleFormDraftRecord,
     PuzzleFormDraftSaveRecordInput, PuzzleGalleryCardRecord, PuzzleGeneratedImageCandidateRecord,
     PuzzleGeneratedImagesSaveRecordInput, PuzzleLeaderboardEntryRecord,
-    PuzzleLeaderboardSubmitRecordInput, PuzzleMergedGroupRecord, PuzzlePieceStateRecord,
-    PuzzlePublishRecordInput, PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord,
-    PuzzleResultPreviewBlockerRecord, PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord,
-    PuzzleRunDragRecordInput, PuzzleRunNextLevelRecordInput, PuzzleRunPauseRecordInput,
-    PuzzleRunPropRecordInput, PuzzleRunRecord, PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput,
-    PuzzleRuntimeLevelRecord, PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput,
+    PuzzleLeaderboardSubmitRecordInput, PuzzleLevelGenerationFailureRecordInput,
+    PuzzleMergedGroupRecord, PuzzlePieceStateRecord, PuzzlePublishRecordInput,
+    PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord, PuzzleResultPreviewBlockerRecord,
+    PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord, PuzzleRunDragRecordInput,
+    PuzzleRunNextLevelRecordInput, PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput,
+    PuzzleRunRecord, PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput, PuzzleRuntimeLevelRecord,
+    PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput,
     PuzzleWorkLikeReportRecordInput, PuzzleWorkPointIncentiveClaimRecordInput,
     PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput, PuzzleWorkUpsertRecordInput,
     ResolveCombatActionRecord, ResolveNpcBattleInteractionInput,
@@ -111,6 +117,7 @@ pub use bark_battle::{
 pub mod big_fish;
 pub mod combat;
 pub mod custom_world;
+pub mod external_generation;
 pub mod inventory;
 pub mod jump_hop;
 pub mod match3d;
@@ -348,7 +355,7 @@ type ProcedureResultSender<T> =
 type ReducerResultSender = Arc<Mutex<Option<oneshot::Sender<Result<(), SpacetimeClientError>>>>>;
 
 struct SpacetimeConnectionPool {
-    slots: Vec<tokio::sync::Mutex<PooledConnectionSlot>>,
+    slots: Vec<PooledConnectionSlot>,
     permits: Arc<Semaphore>,
 }
 
@@ -371,8 +378,10 @@ impl SpacetimeStageError {
 }
 
 struct PooledConnectionSlot {
-    connection: Option<PooledConnection>,
-    in_use: bool,
+    // 槽位占用标记独立成原子量：抢占/复位不依赖锁，租约 Drop 兜底可以同步完成。
+    in_use: AtomicBool,
+    // in_use=true 的持有者独占本槽连接，正常情况下锁上不会有竞争。
+    connection: tokio::sync::Mutex<Option<PooledConnection>>,
 }
 
 struct PooledConnection {
@@ -385,9 +394,28 @@ struct PooledConnection {
 struct PooledConnectionLease {
     slot_index: usize,
     connection: Option<PooledConnection>,
+    pool: Arc<SpacetimeConnectionPool>,
     _permit: OwnedSemaphorePermit,
 }
 
+impl Drop for PooledConnectionLease {
+    // 租约 Drop 兜底：请求 future 被取消（如客户端断开导致 handler 被丢弃）时，
+    // 也必须归还连接并复位槽位，否则槽位会永久停留在 in_use 状态、连接池逐渐耗尽。
+    fn drop(&mut self) {
+        let slot = &self.pool.slots[self.slot_index];
+        if let Some(connection) = self.connection.take() {
+            if !connection.is_broken() {
+                if let Ok(mut slot_connection) = slot.connection.try_lock() {
+                    *slot_connection = Some(connection);
+                }
+                // try_lock 理论上不会失败（in_use 持有者独占）；万一失败只丢弃连接，不丢槽位。
+            }
+        }
+        slot.in_use.store(false, Ordering::Release);
+        // _permit 随 Drop 自动归还信号量。
+    }
+}
+
 impl SpacetimeClient {
     pub fn new(config: SpacetimeClientConfig) -> Self {
         let pool_size = config.pool_size.max(1) as usize;
@@ -400,11 +428,9 @@ impl SpacetimeClient {
             ..config
         };
         let slots = (0..pool_size)
-            .map(|_| {
-                tokio::sync::Mutex::new(PooledConnectionSlot {
-                    connection: None,
-                    in_use: false,
-                })
+            .map(|_| PooledConnectionSlot {
+                in_use: AtomicBool::new(false),
+                connection: tokio::sync::Mutex::new(None),
             })
             .collect::<Vec<_>>();
         let pool = Arc::new(SpacetimeConnectionPool {
@@ -678,42 +704,49 @@ impl SpacetimeClient {
                 )
             })?;
 
-        loop {
-            for (slot_index, slot) in self.pool.slots.iter().enumerate() {
-                if let Ok(mut slot_guard) = slot.try_lock() {
-                    if slot_guard.in_use {
-                        continue;
-                    }
-                    let reusable_connection = slot_guard
-                        .connection
-                        .take()
-                        .filter(|connection| !connection.is_broken());
-                    slot_guard.in_use = true;
-                    drop(slot_guard);
+        // 持有 permit 即保证最多 pool_size 个并发持有者，必然能抢到一个空闲槽位；
+        // CAS 抢占后立即构造租约，后续任何失败/取消都由租约 Drop 兜底复位槽位。
+        let slot_index = self
+            .pool
+            .slots
+            .iter()
+            .position(|slot| {
+                slot.in_use
+                    .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire)
+                    .is_ok()
+            })
+            .ok_or_else(|| {
+                SpacetimeStageError::new(
+                    SpacetimeClientStage::PoolAcquire,
+                    SpacetimeClientError::Runtime(
+                        "SpacetimeDB 连接池 permit 与槽位状态不一致".to_string(),
+                    ),
+                )
+            })?;
 
-                    let connection = if let Some(connection) = reusable_connection {
-                        connection
-                    } else {
-                        match self.build_pooled_connection(operation_timeout).await {
-                            Ok(connection) => connection,
-                            Err(error) => {
-                                let mut slot_guard = self.pool.slots[slot_index].lock().await;
-                                slot_guard.in_use = false;
-                                return Err(error);
-                            }
-                        }
-                    };
+        let mut lease = PooledConnectionLease {
+            slot_index,
+            connection: None,
+            pool: self.pool.clone(),
+            _permit: permit,
+        };
 
-                    return Ok(PooledConnectionLease {
-                        slot_index,
-                        connection: Some(connection),
-                        _permit: permit,
-                    });
-                }
-            }
+        let reusable_connection = self.pool.slots[slot_index]
+            .connection
+            .lock()
+            .await
+            .take()
+            .filter(|connection| !connection.is_broken());
 
-            tokio::task::yield_now().await;
-        }
+        let connection = if let Some(connection) = reusable_connection {
+            connection
+        } else {
+            // 建连失败时直接返回错误，槽位与 permit 由 lease Drop 自动归还。
+            self.build_pooled_connection(operation_timeout).await?
+        };
+
+        lease.connection = Some(connection);
+        Ok(lease)
     }
 
     async fn build_pooled_connection(
@@ -911,18 +944,10 @@ impl SpacetimeClient {
         Ok(subscription)
     }
 
-    async fn release_connection(&self, mut lease: PooledConnectionLease) {
-        let mut slot_guard = self.pool.slots[lease.slot_index].lock().await;
-        slot_guard.in_use = false;
-        let Some(connection) = lease.connection.take() else {
-            slot_guard.connection = None;
-            return;
-        };
-        if connection.is_broken() {
-            slot_guard.connection = None;
-        } else {
-            slot_guard.connection = Some(connection);
-        }
+    async fn release_connection(&self, lease: PooledConnectionLease) {
+        // 显式归还与“请求被取消”的隐式归还共用同一套租约 Drop 兜底逻辑，
+        // 保证任何路径下槽位与 permit 都会复位，连接池不会被慢慢泄漏占满。
+        drop(lease);
     }
 
     // 超时后必须统一归还租约；若连接已先一步断开则回传断线，否则标记坏连接并回传超时。
@@ -1127,4 +1152,78 @@ mod tests {
             SpacetimeClientError::Runtime(_)
         ));
     }
+
+    fn test_client(pool_size: u32, procedure_timeout: Duration) -> SpacetimeClient {
+        SpacetimeClient::new(SpacetimeClientConfig {
+            // 指向本机不可达端口：测试只验证连接池行为，不需要真实 SpacetimeDB。
+            server_url: "http://127.0.0.1:9".to_string(),
+            database: "pool-test".to_string(),
+            token: None,
+            pool_size,
+            procedure_timeout,
+        })
+    }
+
+    /// 复现线上故障机制：修复前请求 future 被取消时租约不会归还，槽位永久停留在 in_use，
+    /// 后续 acquire 拿着 permit 空转挂死。修复后租约 Drop 必须同时复位槽位与 permit。
+    #[tokio::test]
+    async fn dropped_lease_releases_slot_and_permit() {
+        let client = test_client(1, Duration::from_millis(200));
+        let permit = client
+            .pool
+            .permits
+            .clone()
+            .acquire_owned()
+            .await
+            .expect("permit should acquire");
+        client.pool.slots[0].in_use.store(true, Ordering::SeqCst);
+        assert_eq!(client.pool.permits.available_permits(), 0);
+
+        // 模拟请求被取消：租约未经过 release_connection 直接被 Drop。
+        let lease = PooledConnectionLease {
+            slot_index: 0,
+            connection: None,
+            pool: client.pool.clone(),
+            _permit: permit,
+        };
+        drop(lease);
+
+        assert!(
+            !client.pool.slots[0].in_use.load(Ordering::SeqCst),
+            "租约 Drop 后槽位必须复位，否则连接池会被泄漏占满"
+        );
+        assert_eq!(
+            client.pool.permits.available_permits(),
+            1,
+            "租约 Drop 后 permit 必须归还"
+        );
+    }
+
+    /// 池内 permit 全部被占用（持续在途请求）时，acquire 必须在超时窗口内返回
+    /// pool_acquire 超时，而不是无限等待。
+    #[tokio::test]
+    async fn acquire_times_out_at_pool_acquire_when_pool_is_busy() {
+        let client = test_client(1, Duration::from_millis(200));
+        let _held_permit = client
+            .pool
+            .permits
+            .clone()
+            .acquire_owned()
+            .await
+            .expect("permit should acquire");
+
+        let result = tokio::time::timeout(
+            Duration::from_secs(5),
+            client.acquire_connection_with_timeout(Duration::from_millis(200)),
+        )
+        .await
+        .expect("acquire 必须在超时窗口内返回，而不是空转挂死");
+
+        let error = match result {
+            Ok(_) => panic!("池占满时应返回 pool_acquire 超时"),
+            Err(error) => error,
+        };
+        assert_eq!(error.stage, SpacetimeClientStage::PoolAcquire);
+        assert!(matches!(error.error, SpacetimeClientError::Timeout));
+    }
 }

server-rs/crates/spacetime-client/src/mapper.rs

@@ -8,6 +8,7 @@ mod big_fish;
 mod combat;
 mod common;
 mod custom_world;
+mod external_generation;
 mod inventory;
 mod jump_hop;
 mod match3d;
@@ -68,6 +69,12 @@ pub use self::common::{
     VisualNovelRunSnapshotRecordInput, VisualNovelRunStartRecordInput,
     VisualNovelWorkCompileRecordInput,
 };
+pub use self::external_generation::{
+    ExternalGenerationJobClaimRecordInput, ExternalGenerationJobCompleteRecordInput,
+    ExternalGenerationJobEnqueueRecordInput, ExternalGenerationJobFailRecordInput,
+    ExternalGenerationJobRecord, ExternalGenerationJobRenewLeaseRecordInput,
+    ExternalGenerationQueueStatsRecord,
+};
 pub use self::jump_hop::{
     JumpHopActionRequest, JumpHopActionResponse, JumpHopActionType, JumpHopCharacterAsset,
     JumpHopDifficulty, JumpHopDraftResponse, JumpHopGalleryCardResponse,
@@ -101,17 +108,18 @@ pub use self::puzzle::{
     PuzzleAgentMessageFinalizeRecordInput, PuzzleAgentMessageRecord,
     PuzzleAgentMessageSubmitRecordInput, PuzzleAgentSessionCreateRecordInput,
     PuzzleAgentSessionRecord, PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord,
-    PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBoardRecord, PuzzleCellPositionRecord,
+    PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleBackgroundCompileTaskClaimRecordInput,
+    PuzzleBackgroundCompileTaskReleaseRecordInput, PuzzleBoardRecord, PuzzleCellPositionRecord,
     PuzzleCreatorIntentRecord, PuzzleDraftCompileFailureRecordInput, PuzzleDraftLevelRecord,
     PuzzleFormDraftRecord, PuzzleFormDraftSaveRecordInput, PuzzleGalleryCardRecord,
     PuzzleGeneratedImageCandidateRecord, PuzzleGeneratedImagesSaveRecordInput,
-    PuzzleLeaderboardEntryRecord, PuzzleLeaderboardSubmitRecordInput, PuzzleMergedGroupRecord,
-    PuzzlePieceStateRecord, PuzzlePublishRecordInput, PuzzleRecommendedNextWorkRecord,
-    PuzzleResultDraftRecord, PuzzleResultPreviewBlockerRecord, PuzzleResultPreviewFindingRecord,
-    PuzzleResultPreviewRecord, PuzzleRunDragRecordInput, PuzzleRunNextLevelRecordInput,
-    PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput, PuzzleRunRecord,
-    PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput, PuzzleRuntimeLevelRecord,
-    PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput,
+    PuzzleLeaderboardEntryRecord, PuzzleLeaderboardSubmitRecordInput,
+    PuzzleLevelGenerationFailureRecordInput, PuzzleMergedGroupRecord, PuzzlePieceStateRecord,
+    PuzzlePublishRecordInput, PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord,
+    PuzzleResultPreviewBlockerRecord, PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord,
+    PuzzleRunDragRecordInput, PuzzleRunNextLevelRecordInput, PuzzleRunPauseRecordInput,
+    PuzzleRunPropRecordInput, PuzzleRunRecord, PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput,
+    PuzzleRuntimeLevelRecord, PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput,
     PuzzleWorkLikeReportRecordInput, PuzzleWorkPointIncentiveClaimRecordInput,
     PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput, PuzzleWorkUpsertRecordInput,
 };
@@ -177,6 +185,10 @@ pub(crate) use self::custom_world::{
     parse_rpg_agent_operation_status_record, parse_rpg_agent_operation_type_record,
     parse_rpg_agent_stage_record,
 };
+pub(crate) use self::external_generation::{
+    map_external_generation_job_claim_result, map_external_generation_job_procedure_result,
+    map_external_generation_queue_stats_result,
+};
 pub(crate) use self::inventory::{
     map_runtime_inventory_state_procedure_result, map_runtime_item_reward_item_snapshot,
     map_runtime_item_reward_item_snapshot_back,
@@ -199,10 +211,10 @@ pub(crate) use self::public_work::{
     map_public_work_gallery_entry, map_public_work_gallery_entry_to_detail_entry,
 };
 pub(crate) use self::puzzle::{
-    map_puzzle_agent_session_procedure_result, map_puzzle_gallery_card_view_row,
-    map_puzzle_run_procedure_result, map_puzzle_work_procedure_result,
-    map_puzzle_works_procedure_result, map_runtime_profile_wallet_ledger_source_type_back,
-    parse_puzzle_agent_stage_record,
+    map_puzzle_agent_session_procedure_result, map_puzzle_background_compile_task_procedure_result,
+    map_puzzle_gallery_card_view_row, map_puzzle_run_procedure_result,
+    map_puzzle_work_procedure_result, map_puzzle_works_procedure_result,
+    map_runtime_profile_wallet_ledger_source_type_back, parse_puzzle_agent_stage_record,
 };
 pub(crate) use self::puzzle_clear::{
     map_puzzle_clear_agent_session_procedure_result, map_puzzle_clear_gallery_card_view_row,

server-rs/crates/spacetime-client/src/mapper/external_generation.rs

@@ -0,0 +1,238 @@
+use super::*;
+
+impl From<ExternalGenerationJobEnqueueRecordInput> for ExternalGenerationJobEnqueueInput {
+    fn from(input: ExternalGenerationJobEnqueueRecordInput) -> Self {
+        Self {
+            job_id: input.job_id,
+            dedupe_key: input.dedupe_key,
+            job_kind: input.job_kind,
+            owner_user_id: input.owner_user_id,
+            source_module: input.source_module,
+            source_entity_id: input.source_entity_id,
+            request_label: input.request_label,
+            request_payload_json: input.request_payload_json,
+            max_attempts: input.max_attempts,
+            available_at_micros: input.available_at_micros,
+            created_at_micros: input.created_at_micros,
+        }
+    }
+}
+
+impl From<ExternalGenerationJobClaimRecordInput> for ExternalGenerationJobClaimInput {
+    fn from(input: ExternalGenerationJobClaimRecordInput) -> Self {
+        Self {
+            worker_id: input.worker_id,
+            limit: input.limit,
+            lease_expires_at_micros: input.lease_expires_at_micros,
+            claimed_at_micros: input.claimed_at_micros,
+        }
+    }
+}
+
+impl From<ExternalGenerationJobCompleteRecordInput> for ExternalGenerationJobCompleteInput {
+    fn from(input: ExternalGenerationJobCompleteRecordInput) -> Self {
+        Self {
+            job_id: input.job_id,
+            worker_id: input.worker_id,
+            lease_token: input.lease_token,
+            result_payload_json: input.result_payload_json,
+            completed_at_micros: input.completed_at_micros,
+        }
+    }
+}
+
+impl From<ExternalGenerationJobRenewLeaseRecordInput> for ExternalGenerationJobRenewLeaseInput {
+    fn from(input: ExternalGenerationJobRenewLeaseRecordInput) -> Self {
+        Self {
+            job_id: input.job_id,
+            worker_id: input.worker_id,
+            lease_token: input.lease_token,
+            lease_expires_at_micros: input.lease_expires_at_micros,
+            renewed_at_micros: input.renewed_at_micros,
+        }
+    }
+}
+
+impl From<ExternalGenerationJobFailRecordInput> for ExternalGenerationJobFailInput {
+    fn from(input: ExternalGenerationJobFailRecordInput) -> Self {
+        Self {
+            job_id: input.job_id,
+            worker_id: input.worker_id,
+            lease_token: input.lease_token,
+            error_message: input.error_message,
+            retry_after_micros: input.retry_after_micros,
+            failed_at_micros: input.failed_at_micros,
+        }
+    }
+}
+
+pub(crate) fn map_external_generation_job_procedure_result(
+    result: ExternalGenerationJobProcedureResult,
+) -> Result<ExternalGenerationJobRecord, SpacetimeClientError> {
+    if !result.ok {
+        return Err(SpacetimeClientError::procedure_failed(result.error_message));
+    }
+
+    let job = result
+        .job
+        .ok_or_else(|| SpacetimeClientError::missing_snapshot("external_generation_job 快照"))?;
+
+    Ok(map_external_generation_job_snapshot(job))
+}
+
+pub(crate) fn map_external_generation_job_claim_result(
+    result: ExternalGenerationJobProcedureResult,
+) -> Result<Vec<ExternalGenerationJobRecord>, SpacetimeClientError> {
+    if !result.ok {
+        return Err(SpacetimeClientError::procedure_failed(result.error_message));
+    }
+
+    Ok(result
+        .jobs
+        .into_iter()
+        .map(map_external_generation_job_snapshot)
+        .collect())
+}
+
+pub(crate) fn map_external_generation_queue_stats_result(
+    result: ExternalGenerationQueueStatsProcedureResult,
+) -> Result<ExternalGenerationQueueStatsRecord, SpacetimeClientError> {
+    if !result.ok {
+        return Err(SpacetimeClientError::procedure_failed(result.error_message));
+    }
+
+    let stats = result.stats.ok_or_else(|| {
+        SpacetimeClientError::missing_snapshot("external_generation queue stats 快照")
+    })?;
+
+    Ok(ExternalGenerationQueueStatsRecord {
+        pending_count: stats.pending_count,
+        delayed_pending_count: stats.delayed_pending_count,
+        claimable_pending_count: stats.claimable_pending_count,
+        running_active_count: stats.running_active_count,
+        expired_running_count: stats.expired_running_count,
+        terminal_count: stats.terminal_count,
+        claimable_count: stats.claimable_count,
+        oldest_claimable_age_micros: stats.oldest_claimable_age_micros,
+        now_micros: stats.now_micros,
+    })
+}
+
+fn map_external_generation_job_snapshot(
+    snapshot: ExternalGenerationJobSnapshot,
+) -> ExternalGenerationJobRecord {
+    ExternalGenerationJobRecord {
+        job_id: snapshot.job_id,
+        dedupe_key: snapshot.dedupe_key,
+        job_kind: snapshot.job_kind,
+        owner_user_id: snapshot.owner_user_id,
+        source_module: snapshot.source_module,
+        source_entity_id: snapshot.source_entity_id,
+        request_label: snapshot.request_label,
+        request_payload_json: snapshot.request_payload_json,
+        status: snapshot.status,
+        attempt: snapshot.attempt,
+        max_attempts: snapshot.max_attempts,
+        last_error_message: snapshot.last_error_message,
+        worker_id: snapshot.worker_id,
+        lease_expires_at: snapshot
+            .lease_expires_at_micros
+            .map(format_timestamp_micros),
+        available_at: format_timestamp_micros(snapshot.available_at_micros),
+        result_payload_json: snapshot.result_payload_json,
+        created_at: format_timestamp_micros(snapshot.created_at_micros),
+        started_at: snapshot.started_at_micros.map(format_timestamp_micros),
+        completed_at: snapshot.completed_at_micros.map(format_timestamp_micros),
+        updated_at: format_timestamp_micros(snapshot.updated_at_micros),
+        lease_token: snapshot.lease_token,
+    }
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationJobEnqueueRecordInput {
+    pub job_id: String,
+    pub dedupe_key: String,
+    pub job_kind: String,
+    pub owner_user_id: String,
+    pub source_module: String,
+    pub source_entity_id: String,
+    pub request_label: String,
+    pub request_payload_json: String,
+    pub max_attempts: u32,
+    pub available_at_micros: i64,
+    pub created_at_micros: i64,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationJobClaimRecordInput {
+    pub worker_id: String,
+    pub limit: u32,
+    pub lease_expires_at_micros: i64,
+    pub claimed_at_micros: i64,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationJobCompleteRecordInput {
+    pub job_id: String,
+    pub worker_id: String,
+    pub lease_token: String,
+    pub result_payload_json: Option<String>,
+    pub completed_at_micros: i64,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationJobRenewLeaseRecordInput {
+    pub job_id: String,
+    pub worker_id: String,
+    pub lease_token: String,
+    pub lease_expires_at_micros: i64,
+    pub renewed_at_micros: i64,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationJobFailRecordInput {
+    pub job_id: String,
+    pub worker_id: String,
+    pub lease_token: String,
+    pub error_message: String,
+    pub retry_after_micros: i64,
+    pub failed_at_micros: i64,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationJobRecord {
+    pub job_id: String,
+    pub dedupe_key: String,
+    pub job_kind: String,
+    pub owner_user_id: String,
+    pub source_module: String,
+    pub source_entity_id: String,
+    pub request_label: String,
+    pub request_payload_json: String,
+    pub status: String,
+    pub attempt: u32,
+    pub max_attempts: u32,
+    pub last_error_message: Option<String>,
+    pub worker_id: Option<String>,
+    pub lease_expires_at: Option<String>,
+    pub available_at: String,
+    pub result_payload_json: Option<String>,
+    pub created_at: String,
+    pub started_at: Option<String>,
+    pub completed_at: Option<String>,
+    pub updated_at: String,
+    pub lease_token: Option<String>,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct ExternalGenerationQueueStatsRecord {
+    pub pending_count: u32,
+    pub delayed_pending_count: u32,
+    pub claimable_pending_count: u32,
+    pub running_active_count: u32,
+    pub expired_running_count: u32,
+    pub terminal_count: u32,
+    pub claimable_count: u32,
+    pub oldest_claimable_age_micros: Option<i64>,
+    pub now_micros: i64,
+}

server-rs/crates/spacetime-client/src/mapper/puzzle.rs

@@ -13,6 +13,16 @@ pub(crate) fn map_puzzle_agent_session_procedure_result(
     Ok(map_puzzle_agent_session_snapshot(session))
 }
 
+pub(crate) fn map_puzzle_background_compile_task_procedure_result(
+    result: PuzzleBackgroundCompileTaskProcedureResult,
+) -> Result<bool, SpacetimeClientError> {
+    if !result.ok {
+        return Err(SpacetimeClientError::procedure_failed(result.error_message));
+    }
+
+    Ok(result.claimed)
+}
+
 pub(crate) fn map_puzzle_work_procedure_result(
     result: PuzzleWorkProcedureResult,
 ) -> Result<PuzzleWorkProfileRecord, SpacetimeClientError> {
@@ -614,6 +624,23 @@ pub struct PuzzleFormDraftSaveRecordInput {
     pub saved_at_micros: i64,
 }
 
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct PuzzleBackgroundCompileTaskClaimRecordInput {
+    pub task_id: String,
+    pub claim_id: String,
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub claimed_at_micros: i64,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct PuzzleBackgroundCompileTaskReleaseRecordInput {
+    pub task_id: String,
+    pub claim_id: String,
+    pub session_id: String,
+    pub owner_user_id: String,
+}
+
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct PuzzleAgentMessageSubmitRecordInput {
     pub session_id: String,
@@ -642,6 +669,22 @@ pub struct PuzzleDraftCompileFailureRecordInput {
     pub owner_user_id: String,
     pub error_message: String,
     pub failed_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct PuzzleLevelGenerationFailureRecordInput {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub level_id: Option<String>,
+    pub levels_json: Option<String>,
+    pub error_message: String,
+    pub failed_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[derive(Clone, Debug, PartialEq, Eq)]
@@ -652,6 +695,9 @@ pub struct PuzzleGeneratedImagesSaveRecordInput {
     pub levels_json: Option<String>,
     pub candidates_json: String,
     pub saved_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[derive(Clone, Debug, PartialEq, Eq)]
@@ -664,6 +710,9 @@ pub struct PuzzleUiBackgroundSaveRecordInput {
     pub image_src: String,
     pub image_object_key: Option<String>,
     pub saved_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[derive(Clone, Debug, PartialEq, Eq)]

server-rs/crates/spacetime-client/src/module_bindings.rs

@@ -203,7 +203,9 @@ pub mod chapter_progression_snapshot_type;
 pub mod chapter_progression_table;
 pub mod chapter_progression_type;
 pub mod checkpoint_wooden_fish_run_procedure;
+pub mod claim_external_generation_jobs_and_return_procedure;
 pub mod claim_profile_task_reward_and_return_procedure;
+pub mod claim_puzzle_background_compile_task_procedure;
 pub mod claim_puzzle_work_point_incentive_procedure;
 pub mod clear_database_migration_import_chunks_procedure;
 pub mod clear_platform_browse_history_and_return_procedure;
@@ -220,6 +222,7 @@ pub mod compile_visual_novel_work_profile_procedure;
 pub mod compile_wooden_fish_draft_procedure;
 pub mod complete_ai_stage_and_return_procedure;
 pub mod complete_ai_task_and_return_procedure;
+pub mod complete_external_generation_job_and_return_procedure;
 pub mod confirm_asset_object_and_return_procedure;
 pub mod confirm_asset_object_reducer;
 pub mod consume_inventory_item_input_type;
@@ -342,12 +345,25 @@ pub mod delete_visual_novel_work_procedure;
 pub mod delete_wooden_fish_work_procedure;
 pub mod drag_puzzle_piece_or_group_procedure;
 pub mod drop_square_hole_shape_procedure;
+pub mod enqueue_external_generation_job_and_return_procedure;
 pub mod ensure_analytics_date_dimension_for_date_reducer;
 pub mod equip_inventory_item_input_type;
 pub mod execute_custom_world_agent_action_procedure;
 pub mod export_auth_store_snapshot_from_tables_procedure;
 pub mod export_database_migration_to_file_procedure;
+pub mod external_generation_job_claim_input_type;
+pub mod external_generation_job_complete_input_type;
+pub mod external_generation_job_enqueue_input_type;
+pub mod external_generation_job_fail_input_type;
+pub mod external_generation_job_procedure_result_type;
+pub mod external_generation_job_renew_lease_input_type;
+pub mod external_generation_job_snapshot_type;
+pub mod external_generation_job_table;
+pub mod external_generation_job_type;
+pub mod external_generation_queue_stats_procedure_result_type;
+pub mod external_generation_queue_stats_snapshot_type;
 pub mod fail_ai_task_and_return_procedure;
+pub mod fail_external_generation_job_and_return_procedure;
 pub mod finalize_big_fish_agent_message_turn_procedure;
 pub mod finalize_custom_world_agent_message_turn_procedure;
 pub mod finalize_match_3_d_agent_message_turn_procedure;
@@ -372,6 +388,7 @@ pub mod get_custom_world_agent_session_procedure;
 pub mod get_custom_world_gallery_detail_by_code_procedure;
 pub mod get_custom_world_gallery_detail_procedure;
 pub mod get_custom_world_library_detail_procedure;
+pub mod get_external_generation_queue_stats_and_return_procedure;
 pub mod get_jump_hop_agent_session_procedure;
 pub mod get_jump_hop_leaderboard_procedure;
 pub mod get_jump_hop_run_procedure;
@@ -496,6 +513,7 @@ pub mod list_wooden_fish_works_procedure;
 pub mod mark_profile_recharge_order_paid_and_return_procedure;
 pub mod mark_puzzle_clear_level_time_up_procedure;
 pub mod mark_puzzle_draft_generation_failed_procedure;
+pub mod mark_puzzle_level_generation_failed_procedure;
 pub mod match_3_d_agent_message_finalize_input_type;
 pub mod match_3_d_agent_message_row_type;
 pub mod match_3_d_agent_message_snapshot_type;
@@ -628,6 +646,11 @@ pub mod puzzle_anchor_item_type;
 pub mod puzzle_anchor_pack_type;
 pub mod puzzle_anchor_status_type;
 pub mod puzzle_audio_asset_type;
+pub mod puzzle_background_compile_task_claim_input_type;
+pub mod puzzle_background_compile_task_procedure_result_type;
+pub mod puzzle_background_compile_task_release_input_type;
+pub mod puzzle_background_compile_task_row_type;
+pub mod puzzle_background_compile_task_table;
 pub mod puzzle_board_snapshot_type;
 pub mod puzzle_cell_position_type;
 pub mod puzzle_clear_agent_session_create_input_type;
@@ -686,6 +709,7 @@ pub mod puzzle_leaderboard_entry_row_type;
 pub mod puzzle_leaderboard_entry_table;
 pub mod puzzle_leaderboard_entry_type;
 pub mod puzzle_leaderboard_submit_input_type;
+pub mod puzzle_level_generation_failure_input_type;
 pub mod puzzle_merged_group_state_type;
 pub mod puzzle_piece_state_type;
 pub mod puzzle_publication_status_type;
@@ -766,9 +790,11 @@ pub mod redeem_profile_reward_code_procedure;
 pub mod refresh_session_table;
 pub mod refresh_session_type;
 pub mod refund_profile_wallet_points_and_return_procedure;
+pub mod release_puzzle_background_compile_task_procedure;
 pub mod remix_big_fish_work_procedure;
 pub mod remix_custom_world_profile_procedure;
 pub mod remix_puzzle_work_procedure;
+pub mod renew_external_generation_job_lease_and_return_procedure;
 pub mod resolve_combat_action_and_return_procedure;
 pub mod resolve_combat_action_input_type;
 pub mod resolve_combat_action_procedure_result_type;
@@ -1311,7 +1337,9 @@ pub use chapter_progression_snapshot_type::ChapterProgressionSnapshot;
 pub use chapter_progression_table::*;
 pub use chapter_progression_type::ChapterProgression;
 pub use checkpoint_wooden_fish_run_procedure::checkpoint_wooden_fish_run;
+pub use claim_external_generation_jobs_and_return_procedure::claim_external_generation_jobs_and_return;
 pub use claim_profile_task_reward_and_return_procedure::claim_profile_task_reward_and_return;
+pub use claim_puzzle_background_compile_task_procedure::claim_puzzle_background_compile_task;
 pub use claim_puzzle_work_point_incentive_procedure::claim_puzzle_work_point_incentive;
 pub use clear_database_migration_import_chunks_procedure::clear_database_migration_import_chunks;
 pub use clear_platform_browse_history_and_return_procedure::clear_platform_browse_history_and_return;
@@ -1328,6 +1356,7 @@ pub use compile_visual_novel_work_profile_procedure::compile_visual_novel_work_p
 pub use compile_wooden_fish_draft_procedure::compile_wooden_fish_draft;
 pub use complete_ai_stage_and_return_procedure::complete_ai_stage_and_return;
 pub use complete_ai_task_and_return_procedure::complete_ai_task_and_return;
+pub use complete_external_generation_job_and_return_procedure::complete_external_generation_job_and_return;
 pub use confirm_asset_object_and_return_procedure::confirm_asset_object_and_return;
 pub use confirm_asset_object_reducer::confirm_asset_object;
 pub use consume_inventory_item_input_type::ConsumeInventoryItemInput;
@@ -1450,12 +1479,25 @@ pub use delete_visual_novel_work_procedure::delete_visual_novel_work;
 pub use delete_wooden_fish_work_procedure::delete_wooden_fish_work;
 pub use drag_puzzle_piece_or_group_procedure::drag_puzzle_piece_or_group;
 pub use drop_square_hole_shape_procedure::drop_square_hole_shape;
+pub use enqueue_external_generation_job_and_return_procedure::enqueue_external_generation_job_and_return;
 pub use ensure_analytics_date_dimension_for_date_reducer::ensure_analytics_date_dimension_for_date;
 pub use equip_inventory_item_input_type::EquipInventoryItemInput;
 pub use execute_custom_world_agent_action_procedure::execute_custom_world_agent_action;
 pub use export_auth_store_snapshot_from_tables_procedure::export_auth_store_snapshot_from_tables;
 pub use export_database_migration_to_file_procedure::export_database_migration_to_file;
+pub use external_generation_job_claim_input_type::ExternalGenerationJobClaimInput;
+pub use external_generation_job_complete_input_type::ExternalGenerationJobCompleteInput;
+pub use external_generation_job_enqueue_input_type::ExternalGenerationJobEnqueueInput;
+pub use external_generation_job_fail_input_type::ExternalGenerationJobFailInput;
+pub use external_generation_job_procedure_result_type::ExternalGenerationJobProcedureResult;
+pub use external_generation_job_renew_lease_input_type::ExternalGenerationJobRenewLeaseInput;
+pub use external_generation_job_snapshot_type::ExternalGenerationJobSnapshot;
+pub use external_generation_job_table::*;
+pub use external_generation_job_type::ExternalGenerationJob;
+pub use external_generation_queue_stats_procedure_result_type::ExternalGenerationQueueStatsProcedureResult;
+pub use external_generation_queue_stats_snapshot_type::ExternalGenerationQueueStatsSnapshot;
 pub use fail_ai_task_and_return_procedure::fail_ai_task_and_return;
+pub use fail_external_generation_job_and_return_procedure::fail_external_generation_job_and_return;
 pub use finalize_big_fish_agent_message_turn_procedure::finalize_big_fish_agent_message_turn;
 pub use finalize_custom_world_agent_message_turn_procedure::finalize_custom_world_agent_message_turn;
 pub use finalize_match_3_d_agent_message_turn_procedure::finalize_match_3_d_agent_message_turn;
@@ -1480,6 +1522,7 @@ pub use get_custom_world_agent_session_procedure::get_custom_world_agent_session
 pub use get_custom_world_gallery_detail_by_code_procedure::get_custom_world_gallery_detail_by_code;
 pub use get_custom_world_gallery_detail_procedure::get_custom_world_gallery_detail;
 pub use get_custom_world_library_detail_procedure::get_custom_world_library_detail;
+pub use get_external_generation_queue_stats_and_return_procedure::get_external_generation_queue_stats_and_return;
 pub use get_jump_hop_agent_session_procedure::get_jump_hop_agent_session;
 pub use get_jump_hop_leaderboard_procedure::get_jump_hop_leaderboard;
 pub use get_jump_hop_run_procedure::get_jump_hop_run;
@@ -1604,6 +1647,7 @@ pub use list_wooden_fish_works_procedure::list_wooden_fish_works;
 pub use mark_profile_recharge_order_paid_and_return_procedure::mark_profile_recharge_order_paid_and_return;
 pub use mark_puzzle_clear_level_time_up_procedure::mark_puzzle_clear_level_time_up;
 pub use mark_puzzle_draft_generation_failed_procedure::mark_puzzle_draft_generation_failed;
+pub use mark_puzzle_level_generation_failed_procedure::mark_puzzle_level_generation_failed;
 pub use match_3_d_agent_message_finalize_input_type::Match3DAgentMessageFinalizeInput;
 pub use match_3_d_agent_message_row_type::Match3DAgentMessageRow;
 pub use match_3_d_agent_message_snapshot_type::Match3DAgentMessageSnapshot;
@@ -1736,6 +1780,11 @@ pub use puzzle_anchor_item_type::PuzzleAnchorItem;
 pub use puzzle_anchor_pack_type::PuzzleAnchorPack;
 pub use puzzle_anchor_status_type::PuzzleAnchorStatus;
 pub use puzzle_audio_asset_type::PuzzleAudioAsset;
+pub use puzzle_background_compile_task_claim_input_type::PuzzleBackgroundCompileTaskClaimInput;
+pub use puzzle_background_compile_task_procedure_result_type::PuzzleBackgroundCompileTaskProcedureResult;
+pub use puzzle_background_compile_task_release_input_type::PuzzleBackgroundCompileTaskReleaseInput;
+pub use puzzle_background_compile_task_row_type::PuzzleBackgroundCompileTaskRow;
+pub use puzzle_background_compile_task_table::*;
 pub use puzzle_board_snapshot_type::PuzzleBoardSnapshot;
 pub use puzzle_cell_position_type::PuzzleCellPosition;
 pub use puzzle_clear_agent_session_create_input_type::PuzzleClearAgentSessionCreateInput;
@@ -1794,6 +1843,7 @@ pub use puzzle_leaderboard_entry_row_type::PuzzleLeaderboardEntryRow;
 pub use puzzle_leaderboard_entry_table::*;
 pub use puzzle_leaderboard_entry_type::PuzzleLeaderboardEntry;
 pub use puzzle_leaderboard_submit_input_type::PuzzleLeaderboardSubmitInput;
+pub use puzzle_level_generation_failure_input_type::PuzzleLevelGenerationFailureInput;
 pub use puzzle_merged_group_state_type::PuzzleMergedGroupState;
 pub use puzzle_piece_state_type::PuzzlePieceState;
 pub use puzzle_publication_status_type::PuzzlePublicationStatus;
@@ -1874,9 +1924,11 @@ pub use redeem_profile_reward_code_procedure::redeem_profile_reward_code;
 pub use refresh_session_table::*;
 pub use refresh_session_type::RefreshSession;
 pub use refund_profile_wallet_points_and_return_procedure::refund_profile_wallet_points_and_return;
+pub use release_puzzle_background_compile_task_procedure::release_puzzle_background_compile_task;
 pub use remix_big_fish_work_procedure::remix_big_fish_work;
 pub use remix_custom_world_profile_procedure::remix_custom_world_profile;
 pub use remix_puzzle_work_procedure::remix_puzzle_work;
+pub use renew_external_generation_job_lease_and_return_procedure::renew_external_generation_job_lease_and_return;
 pub use resolve_combat_action_and_return_procedure::resolve_combat_action_and_return;
 pub use resolve_combat_action_input_type::ResolveCombatActionInput;
 pub use resolve_combat_action_procedure_result_type::ResolveCombatActionProcedureResult;
@@ -2533,6 +2585,7 @@ pub struct DbUpdate {
     custom_world_session: __sdk::TableUpdate<CustomWorldSession>,
     database_migration_import_chunk: __sdk::TableUpdate<DatabaseMigrationImportChunk>,
     database_migration_operator: __sdk::TableUpdate<DatabaseMigrationOperator>,
+    external_generation_job: __sdk::TableUpdate<ExternalGenerationJob>,
     inventory_slot: __sdk::TableUpdate<InventorySlot>,
     jump_hop_agent_session: __sdk::TableUpdate<JumpHopAgentSessionRow>,
     jump_hop_event: __sdk::TableUpdate<JumpHopEventRow>,
@@ -2569,6 +2622,7 @@ pub struct DbUpdate {
     public_work_play_daily_stat: __sdk::TableUpdate<PublicWorkPlayDailyStat>,
     puzzle_agent_message: __sdk::TableUpdate<PuzzleAgentMessageRow>,
     puzzle_agent_session: __sdk::TableUpdate<PuzzleAgentSessionRow>,
+    puzzle_background_compile_task: __sdk::TableUpdate<PuzzleBackgroundCompileTaskRow>,
     puzzle_clear_agent_session: __sdk::TableUpdate<PuzzleClearAgentSessionRow>,
     puzzle_clear_event: __sdk::TableUpdate<PuzzleClearEventRow>,
     puzzle_clear_gallery_card_view: __sdk::TableUpdate<PuzzleClearGalleryCardViewRow>,
@@ -2744,6 +2798,9 @@ impl TryFrom<__ws::v2::TransactionUpdate> for DbUpdate {
                 "database_migration_operator" => db_update.database_migration_operator.append(
                     database_migration_operator_table::parse_table_update(table_update)?,
                 ),
+                "external_generation_job" => db_update.external_generation_job.append(
+                    external_generation_job_table::parse_table_update(table_update)?,
+                ),
                 "inventory_slot" => db_update
                     .inventory_slot
                     .append(inventory_slot_table::parse_table_update(table_update)?),
@@ -2854,6 +2911,11 @@ impl TryFrom<__ws::v2::TransactionUpdate> for DbUpdate {
                 "puzzle_agent_session" => db_update.puzzle_agent_session.append(
                     puzzle_agent_session_table::parse_table_update(table_update)?,
                 ),
+                "puzzle_background_compile_task" => {
+                    db_update.puzzle_background_compile_task.append(
+                        puzzle_background_compile_task_table::parse_table_update(table_update)?,
+                    )
+                }
                 "puzzle_clear_agent_session" => db_update.puzzle_clear_agent_session.append(
                     puzzle_clear_agent_session_table::parse_table_update(table_update)?,
                 ),
@@ -3199,6 +3261,12 @@ impl __sdk::DbUpdate for DbUpdate {
                 &self.database_migration_operator,
             )
             .with_updates_by_pk(|row| &row.operator_identity);
+        diff.external_generation_job = cache
+            .apply_diff_to_table::<ExternalGenerationJob>(
+                "external_generation_job",
+                &self.external_generation_job,
+            )
+            .with_updates_by_pk(|row| &row.job_id);
         diff.inventory_slot = cache
             .apply_diff_to_table::<InventorySlot>("inventory_slot", &self.inventory_slot)
             .with_updates_by_pk(|row| &row.slot_id);
@@ -3373,6 +3441,12 @@ impl __sdk::DbUpdate for DbUpdate {
                 &self.puzzle_agent_session,
             )
             .with_updates_by_pk(|row| &row.session_id);
+        diff.puzzle_background_compile_task = cache
+            .apply_diff_to_table::<PuzzleBackgroundCompileTaskRow>(
+                "puzzle_background_compile_task",
+                &self.puzzle_background_compile_task,
+            )
+            .with_updates_by_pk(|row| &row.task_id);
         diff.puzzle_clear_agent_session = cache
             .apply_diff_to_table::<PuzzleClearAgentSessionRow>(
                 "puzzle_clear_agent_session",
@@ -3720,6 +3794,9 @@ impl __sdk::DbUpdate for DbUpdate {
                 "database_migration_operator" => db_update
                     .database_migration_operator
                     .append(__sdk::parse_row_list_as_inserts(table_rows.rows)?),
+                "external_generation_job" => db_update
+                    .external_generation_job
+                    .append(__sdk::parse_row_list_as_inserts(table_rows.rows)?),
                 "inventory_slot" => db_update
                     .inventory_slot
                     .append(__sdk::parse_row_list_as_inserts(table_rows.rows)?),
@@ -3828,6 +3905,9 @@ impl __sdk::DbUpdate for DbUpdate {
                 "puzzle_agent_session" => db_update
                     .puzzle_agent_session
                     .append(__sdk::parse_row_list_as_inserts(table_rows.rows)?),
+                "puzzle_background_compile_task" => db_update
+                    .puzzle_background_compile_task
+                    .append(__sdk::parse_row_list_as_inserts(table_rows.rows)?),
                 "puzzle_clear_agent_session" => db_update
                     .puzzle_clear_agent_session
                     .append(__sdk::parse_row_list_as_inserts(table_rows.rows)?),
@@ -4084,6 +4164,9 @@ impl __sdk::DbUpdate for DbUpdate {
                 "database_migration_operator" => db_update
                     .database_migration_operator
                     .append(__sdk::parse_row_list_as_deletes(table_rows.rows)?),
+                "external_generation_job" => db_update
+                    .external_generation_job
+                    .append(__sdk::parse_row_list_as_deletes(table_rows.rows)?),
                 "inventory_slot" => db_update
                     .inventory_slot
                     .append(__sdk::parse_row_list_as_deletes(table_rows.rows)?),
@@ -4192,6 +4275,9 @@ impl __sdk::DbUpdate for DbUpdate {
                 "puzzle_agent_session" => db_update
                     .puzzle_agent_session
                     .append(__sdk::parse_row_list_as_deletes(table_rows.rows)?),
+                "puzzle_background_compile_task" => db_update
+                    .puzzle_background_compile_task
+                    .append(__sdk::parse_row_list_as_deletes(table_rows.rows)?),
                 "puzzle_clear_agent_session" => db_update
                     .puzzle_clear_agent_session
                     .append(__sdk::parse_row_list_as_deletes(table_rows.rows)?),
@@ -4374,6 +4460,7 @@ pub struct AppliedDiff<'r> {
     custom_world_session: __sdk::TableAppliedDiff<'r, CustomWorldSession>,
     database_migration_import_chunk: __sdk::TableAppliedDiff<'r, DatabaseMigrationImportChunk>,
     database_migration_operator: __sdk::TableAppliedDiff<'r, DatabaseMigrationOperator>,
+    external_generation_job: __sdk::TableAppliedDiff<'r, ExternalGenerationJob>,
     inventory_slot: __sdk::TableAppliedDiff<'r, InventorySlot>,
     jump_hop_agent_session: __sdk::TableAppliedDiff<'r, JumpHopAgentSessionRow>,
     jump_hop_event: __sdk::TableAppliedDiff<'r, JumpHopEventRow>,
@@ -4410,6 +4497,7 @@ pub struct AppliedDiff<'r> {
     public_work_play_daily_stat: __sdk::TableAppliedDiff<'r, PublicWorkPlayDailyStat>,
     puzzle_agent_message: __sdk::TableAppliedDiff<'r, PuzzleAgentMessageRow>,
     puzzle_agent_session: __sdk::TableAppliedDiff<'r, PuzzleAgentSessionRow>,
+    puzzle_background_compile_task: __sdk::TableAppliedDiff<'r, PuzzleBackgroundCompileTaskRow>,
     puzzle_clear_agent_session: __sdk::TableAppliedDiff<'r, PuzzleClearAgentSessionRow>,
     puzzle_clear_event: __sdk::TableAppliedDiff<'r, PuzzleClearEventRow>,
     puzzle_clear_gallery_card_view: __sdk::TableAppliedDiff<'r, PuzzleClearGalleryCardViewRow>,
@@ -4653,6 +4741,11 @@ impl<'r> __sdk::AppliedDiff<'r> for AppliedDiff<'r> {
             &self.database_migration_operator,
             event,
         );
+        callbacks.invoke_table_row_callbacks::<ExternalGenerationJob>(
+            "external_generation_job",
+            &self.external_generation_job,
+            event,
+        );
         callbacks.invoke_table_row_callbacks::<InventorySlot>(
             "inventory_slot",
             &self.inventory_slot,
@@ -4829,6 +4922,11 @@ impl<'r> __sdk::AppliedDiff<'r> for AppliedDiff<'r> {
             &self.puzzle_agent_session,
             event,
         );
+        callbacks.invoke_table_row_callbacks::<PuzzleBackgroundCompileTaskRow>(
+            "puzzle_background_compile_task",
+            &self.puzzle_background_compile_task,
+            event,
+        );
         callbacks.invoke_table_row_callbacks::<PuzzleClearAgentSessionRow>(
             "puzzle_clear_agent_session",
             &self.puzzle_clear_agent_session,
@@ -5730,6 +5828,7 @@ impl __sdk::SpacetimeModule for RemoteModule {
         custom_world_session_table::register_table(client_cache);
         database_migration_import_chunk_table::register_table(client_cache);
         database_migration_operator_table::register_table(client_cache);
+        external_generation_job_table::register_table(client_cache);
         inventory_slot_table::register_table(client_cache);
         jump_hop_agent_session_table::register_table(client_cache);
         jump_hop_event_table::register_table(client_cache);
@@ -5766,6 +5865,7 @@ impl __sdk::SpacetimeModule for RemoteModule {
         public_work_play_daily_stat_table::register_table(client_cache);
         puzzle_agent_message_table::register_table(client_cache);
         puzzle_agent_session_table::register_table(client_cache);
+        puzzle_background_compile_task_table::register_table(client_cache);
         puzzle_clear_agent_session_table::register_table(client_cache);
         puzzle_clear_event_table::register_table(client_cache);
         puzzle_clear_gallery_card_view_table::register_table(client_cache);
@@ -5849,6 +5949,7 @@ impl __sdk::SpacetimeModule for RemoteModule {
         "custom_world_session",
         "database_migration_import_chunk",
         "database_migration_operator",
+        "external_generation_job",
         "inventory_slot",
         "jump_hop_agent_session",
         "jump_hop_event",
@@ -5885,6 +5986,7 @@ impl __sdk::SpacetimeModule for RemoteModule {
         "public_work_play_daily_stat",
         "puzzle_agent_message",
         "puzzle_agent_session",
+        "puzzle_background_compile_task",
         "puzzle_clear_agent_session",
         "puzzle_clear_event",
         "puzzle_clear_gallery_card_view",

server-rs/crates/spacetime-client/src/module_bindings/claim_external_generation_jobs_and_return_procedure.rs

@@ -0,0 +1,59 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+use super::external_generation_job_claim_input_type::ExternalGenerationJobClaimInput;
+use super::external_generation_job_procedure_result_type::ExternalGenerationJobProcedureResult;
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+struct ClaimExternalGenerationJobsAndReturnArgs {
+    pub input: ExternalGenerationJobClaimInput,
+}
+
+impl __sdk::InModule for ClaimExternalGenerationJobsAndReturnArgs {
+    type Module = super::RemoteModule;
+}
+
+#[allow(non_camel_case_types)]
+/// Extension trait for access to the procedure `claim_external_generation_jobs_and_return`.
+///
+/// Implemented for [`super::RemoteProcedures`].
+pub trait claim_external_generation_jobs_and_return {
+    fn claim_external_generation_jobs_and_return(&self, input: ExternalGenerationJobClaimInput) {
+        self.claim_external_generation_jobs_and_return_then(input, |_, _| {});
+    }
+
+    fn claim_external_generation_jobs_and_return_then(
+        &self,
+        input: ExternalGenerationJobClaimInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<ExternalGenerationJobProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    );
+}
+
+impl claim_external_generation_jobs_and_return for super::RemoteProcedures {
+    fn claim_external_generation_jobs_and_return_then(
+        &self,
+        input: ExternalGenerationJobClaimInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<ExternalGenerationJobProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    ) {
+        self.imp
+            .invoke_procedure_with_callback::<_, ExternalGenerationJobProcedureResult>(
+                "claim_external_generation_jobs_and_return",
+                ClaimExternalGenerationJobsAndReturnArgs { input },
+                __callback,
+            );
+    }
+}

server-rs/crates/spacetime-client/src/module_bindings/claim_puzzle_background_compile_task_procedure.rs

@@ -0,0 +1,59 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+use super::puzzle_background_compile_task_claim_input_type::PuzzleBackgroundCompileTaskClaimInput;
+use super::puzzle_background_compile_task_procedure_result_type::PuzzleBackgroundCompileTaskProcedureResult;
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+struct ClaimPuzzleBackgroundCompileTaskArgs {
+    pub input: PuzzleBackgroundCompileTaskClaimInput,
+}
+
+impl __sdk::InModule for ClaimPuzzleBackgroundCompileTaskArgs {
+    type Module = super::RemoteModule;
+}
+
+#[allow(non_camel_case_types)]
+/// Extension trait for access to the procedure `claim_puzzle_background_compile_task`.
+///
+/// Implemented for [`super::RemoteProcedures`].
+pub trait claim_puzzle_background_compile_task {
+    fn claim_puzzle_background_compile_task(&self, input: PuzzleBackgroundCompileTaskClaimInput) {
+        self.claim_puzzle_background_compile_task_then(input, |_, _| {});
+    }
+
+    fn claim_puzzle_background_compile_task_then(
+        &self,
+        input: PuzzleBackgroundCompileTaskClaimInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<PuzzleBackgroundCompileTaskProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    );
+}
+
+impl claim_puzzle_background_compile_task for super::RemoteProcedures {
+    fn claim_puzzle_background_compile_task_then(
+        &self,
+        input: PuzzleBackgroundCompileTaskClaimInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<PuzzleBackgroundCompileTaskProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    ) {
+        self.imp
+            .invoke_procedure_with_callback::<_, PuzzleBackgroundCompileTaskProcedureResult>(
+                "claim_puzzle_background_compile_task",
+                ClaimPuzzleBackgroundCompileTaskArgs { input },
+                __callback,
+            );
+    }
+}

server-rs/crates/spacetime-client/src/module_bindings/complete_external_generation_job_and_return_procedure.rs

@@ -0,0 +1,62 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+use super::external_generation_job_complete_input_type::ExternalGenerationJobCompleteInput;
+use super::external_generation_job_procedure_result_type::ExternalGenerationJobProcedureResult;
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+struct CompleteExternalGenerationJobAndReturnArgs {
+    pub input: ExternalGenerationJobCompleteInput,
+}
+
+impl __sdk::InModule for CompleteExternalGenerationJobAndReturnArgs {
+    type Module = super::RemoteModule;
+}
+
+#[allow(non_camel_case_types)]
+/// Extension trait for access to the procedure `complete_external_generation_job_and_return`.
+///
+/// Implemented for [`super::RemoteProcedures`].
+pub trait complete_external_generation_job_and_return {
+    fn complete_external_generation_job_and_return(
+        &self,
+        input: ExternalGenerationJobCompleteInput,
+    ) {
+        self.complete_external_generation_job_and_return_then(input, |_, _| {});
+    }
+
+    fn complete_external_generation_job_and_return_then(
+        &self,
+        input: ExternalGenerationJobCompleteInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<ExternalGenerationJobProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    );
+}
+
+impl complete_external_generation_job_and_return for super::RemoteProcedures {
+    fn complete_external_generation_job_and_return_then(
+        &self,
+        input: ExternalGenerationJobCompleteInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<ExternalGenerationJobProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    ) {
+        self.imp
+            .invoke_procedure_with_callback::<_, ExternalGenerationJobProcedureResult>(
+                "complete_external_generation_job_and_return",
+                CompleteExternalGenerationJobAndReturnArgs { input },
+                __callback,
+            );
+    }
+}

server-rs/crates/spacetime-client/src/module_bindings/enqueue_external_generation_job_and_return_procedure.rs

@@ -0,0 +1,59 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+use super::external_generation_job_enqueue_input_type::ExternalGenerationJobEnqueueInput;
+use super::external_generation_job_procedure_result_type::ExternalGenerationJobProcedureResult;
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+struct EnqueueExternalGenerationJobAndReturnArgs {
+    pub input: ExternalGenerationJobEnqueueInput,
+}
+
+impl __sdk::InModule for EnqueueExternalGenerationJobAndReturnArgs {
+    type Module = super::RemoteModule;
+}
+
+#[allow(non_camel_case_types)]
+/// Extension trait for access to the procedure `enqueue_external_generation_job_and_return`.
+///
+/// Implemented for [`super::RemoteProcedures`].
+pub trait enqueue_external_generation_job_and_return {
+    fn enqueue_external_generation_job_and_return(&self, input: ExternalGenerationJobEnqueueInput) {
+        self.enqueue_external_generation_job_and_return_then(input, |_, _| {});
+    }
+
+    fn enqueue_external_generation_job_and_return_then(
+        &self,
+        input: ExternalGenerationJobEnqueueInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<ExternalGenerationJobProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    );
+}
+
+impl enqueue_external_generation_job_and_return for super::RemoteProcedures {
+    fn enqueue_external_generation_job_and_return_then(
+        &self,
+        input: ExternalGenerationJobEnqueueInput,
+
+        __callback: impl FnOnce(
+                &super::ProcedureEventContext,
+                Result<ExternalGenerationJobProcedureResult, __sdk::InternalError>,
+            ) + Send
+            + 'static,
+    ) {
+        self.imp
+            .invoke_procedure_with_callback::<_, ExternalGenerationJobProcedureResult>(
+                "enqueue_external_generation_job_and_return",
+                EnqueueExternalGenerationJobAndReturnArgs { input },
+                __callback,
+            );
+    }
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_claim_input_type.rs

@@ -0,0 +1,18 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobClaimInput {
+    pub worker_id: String,
+    pub limit: u32,
+    pub lease_expires_at_micros: i64,
+    pub claimed_at_micros: i64,
+}
+
+impl __sdk::InModule for ExternalGenerationJobClaimInput {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_complete_input_type.rs

@@ -0,0 +1,19 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobCompleteInput {
+    pub job_id: String,
+    pub worker_id: String,
+    pub lease_token: String,
+    pub result_payload_json: Option<String>,
+    pub completed_at_micros: i64,
+}
+
+impl __sdk::InModule for ExternalGenerationJobCompleteInput {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_enqueue_input_type.rs

@@ -0,0 +1,25 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobEnqueueInput {
+    pub job_id: String,
+    pub dedupe_key: String,
+    pub job_kind: String,
+    pub owner_user_id: String,
+    pub source_module: String,
+    pub source_entity_id: String,
+    pub request_label: String,
+    pub request_payload_json: String,
+    pub max_attempts: u32,
+    pub available_at_micros: i64,
+    pub created_at_micros: i64,
+}
+
+impl __sdk::InModule for ExternalGenerationJobEnqueueInput {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_fail_input_type.rs

@@ -0,0 +1,20 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobFailInput {
+    pub job_id: String,
+    pub worker_id: String,
+    pub lease_token: String,
+    pub error_message: String,
+    pub retry_after_micros: i64,
+    pub failed_at_micros: i64,
+}
+
+impl __sdk::InModule for ExternalGenerationJobFailInput {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_procedure_result_type.rs

@@ -0,0 +1,20 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+use super::external_generation_job_snapshot_type::ExternalGenerationJobSnapshot;
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobProcedureResult {
+    pub ok: bool,
+    pub job: Option<ExternalGenerationJobSnapshot>,
+    pub jobs: Vec<ExternalGenerationJobSnapshot>,
+    pub error_message: Option<String>,
+}
+
+impl __sdk::InModule for ExternalGenerationJobProcedureResult {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_renew_lease_input_type.rs

@@ -0,0 +1,19 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobRenewLeaseInput {
+    pub job_id: String,
+    pub worker_id: String,
+    pub lease_token: String,
+    pub lease_expires_at_micros: i64,
+    pub renewed_at_micros: i64,
+}
+
+impl __sdk::InModule for ExternalGenerationJobRenewLeaseInput {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_snapshot_type.rs

@@ -0,0 +1,35 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+#[derive(__lib::ser::Serialize, __lib::de::Deserialize, Clone, PartialEq, Debug)]
+#[sats(crate = __lib)]
+pub struct ExternalGenerationJobSnapshot {
+    pub job_id: String,
+    pub dedupe_key: String,
+    pub job_kind: String,
+    pub owner_user_id: String,
+    pub source_module: String,
+    pub source_entity_id: String,
+    pub request_label: String,
+    pub request_payload_json: String,
+    pub status: String,
+    pub attempt: u32,
+    pub max_attempts: u32,
+    pub last_error_message: Option<String>,
+    pub worker_id: Option<String>,
+    pub lease_expires_at_micros: Option<i64>,
+    pub available_at_micros: i64,
+    pub result_payload_json: Option<String>,
+    pub created_at_micros: i64,
+    pub started_at_micros: Option<i64>,
+    pub completed_at_micros: Option<i64>,
+    pub updated_at_micros: i64,
+    pub lease_token: Option<String>,
+}
+
+impl __sdk::InModule for ExternalGenerationJobSnapshot {
+    type Module = super::RemoteModule;
+}

server-rs/crates/spacetime-client/src/module_bindings/external_generation_job_table.rs

@@ -0,0 +1,192 @@
+// THIS FILE IS AUTOMATICALLY GENERATED BY SPACETIMEDB. EDITS TO THIS FILE
+// WILL NOT BE SAVED. MODIFY TABLES IN YOUR MODULE SOURCE CODE INSTEAD.
+
+#![allow(unused, clippy::all)]
+use super::external_generation_job_type::ExternalGenerationJob;
+use spacetimedb_sdk::__codegen::{self as __sdk, __lib, __sats, __ws};
+
+/// Table handle for the table `external_generation_job`.
+///
+/// Obtain a handle from the [`ExternalGenerationJobTableAccess::external_generation_job`] method on [`super::RemoteTables`],
+/// like `ctx.db.external_generation_job()`.
+///
+/// Users are encouraged not to explicitly reference this type,
+/// but to directly chain method calls,
+/// like `ctx.db.external_generation_job().on_insert(...)`.
+pub struct ExternalGenerationJobTableHandle<'ctx> {
+    imp: __sdk::TableHandle<ExternalGenerationJob>,
+    ctx: std::marker::PhantomData<&'ctx super::RemoteTables>,
+}
+
+#[allow(non_camel_case_types)]
+/// Extension trait for access to the table `external_generation_job`.
+///
+/// Implemented for [`super::RemoteTables`].
+pub trait ExternalGenerationJobTableAccess {
+    #[allow(non_snake_case)]
+    /// Obtain a [`ExternalGenerationJobTableHandle`], which mediates access to the table `external_generation_job`.
+    fn external_generation_job(&self) -> ExternalGenerationJobTableHandle<'_>;
+}
+
+impl ExternalGenerationJobTableAccess for super::RemoteTables {
+    fn external_generation_job(&self) -> ExternalGenerationJobTableHandle<'_> {
+        ExternalGenerationJobTableHandle {
+            imp: self
+                .imp
+                .get_table::<ExternalGenerationJob>("external_generation_job"),
+            ctx: std::marker::PhantomData,
+        }
+    }
+}
+
+pub struct ExternalGenerationJobInsertCallbackId(__sdk::CallbackId);
+pub struct ExternalGenerationJobDeleteCallbackId(__sdk::CallbackId);
+
+impl<'ctx> __sdk::Table for ExternalGenerationJobTableHandle<'ctx> {
+    type Row = ExternalGenerationJob;
+    type EventContext = super::EventContext;
+
+    fn count(&self) -> u64 {
+        self.imp.count()
+    }
+    fn iter(&self) -> impl Iterator<Item = ExternalGenerationJob> + '_ {
+        self.imp.iter()
+    }
+
+    type InsertCallbackId = ExternalGenerationJobInsertCallbackId;
+
+    fn on_insert(
+        &self,
+        callback: impl FnMut(&Self::EventContext, &Self::Row) + Send + 'static,
+    ) -> ExternalGenerationJobInsertCallbackId {
+        ExternalGenerationJobInsertCallbackId(self.imp.on_insert(Box::new(callback)))
+    }
+
+    fn remove_on_insert(&self, callback: ExternalGenerationJobInsertCallbackId) {
+        self.imp.remove_on_insert(callback.0)
+    }
+
+    type DeleteCallbackId = ExternalGenerationJobDeleteCallbackId;
+
+    fn on_delete(
+        &self,
+        callback: impl FnMut(&Self::EventContext, &Self::Row) + Send + 'static,
+    ) -> ExternalGenerationJobDeleteCallbackId {
+        ExternalGenerationJobDeleteCallbackId(self.imp.on_delete(Box::new(callback)))
+    }
+
+    fn remove_on_delete(&self, callback: ExternalGenerationJobDeleteCallbackId) {
+        self.imp.remove_on_delete(callback.0)
+    }
+}
+
+pub struct ExternalGenerationJobUpdateCallbackId(__sdk::CallbackId);
+
+impl<'ctx> __sdk::TableWithPrimaryKey for ExternalGenerationJobTableHandle<'ctx> {
+    type UpdateCallbackId = ExternalGenerationJobUpdateCallbackId;
+
+    fn on_update(
+        &self,
+        callback: impl FnMut(&Self::EventContext, &Self::Row, &Self::Row) + Send + 'static,
+    ) -> ExternalGenerationJobUpdateCallbackId {
+        ExternalGenerationJobUpdateCallbackId(self.imp.on_update(Box::new(callback)))
+    }
+
+    fn remove_on_update(&self, callback: ExternalGenerationJobUpdateCallbackId) {
+        self.imp.remove_on_update(callback.0)
+    }
+}
+
+/// Access to the `job_id` unique index on the table `external_generation_job`,
+/// which allows point queries on the field of the same name
+/// via the [`ExternalGenerationJobJobIdUnique::find`] method.
+///
+/// Users are encouraged not to explicitly reference this type,
+/// but to directly chain method calls,
+/// like `ctx.db.external_generation_job().job_id().find(...)`.
+pub struct ExternalGenerationJobJobIdUnique<'ctx> {
+    imp: __sdk::UniqueConstraintHandle<ExternalGenerationJob, String>,
+    phantom: std::marker::PhantomData<&'ctx super::RemoteTables>,
+}
+
+impl<'ctx> ExternalGenerationJobTableHandle<'ctx> {
+    /// Get a handle on the `job_id` unique index on the table `external_generation_job`.
+    pub fn job_id(&self) -> ExternalGenerationJobJobIdUnique<'ctx> {
+        ExternalGenerationJobJobIdUnique {
+            imp: self.imp.get_unique_constraint::<String>("job_id"),
+            phantom: std::marker::PhantomData,
+        }
+    }
+}
+
+impl<'ctx> ExternalGenerationJobJobIdUnique<'ctx> {
+    /// Find the subscribed row whose `job_id` column value is equal to `col_val`,
+    /// if such a row is present in the client cache.
+    pub fn find(&self, col_val: &String) -> Option<ExternalGenerationJob> {
+        self.imp.find(col_val)
+    }
+}
+
+/// Access to the `dedupe_key` unique index on the table `external_generation_job`,
+/// which allows point queries on the field of the same name
+/// via the [`ExternalGenerationJobDedupeKeyUnique::find`] method.
+///
+/// Users are encouraged not to explicitly reference this type,
+/// but to directly chain method calls,
+/// like `ctx.db.external_generation_job().dedupe_key().find(...)`.
+pub struct ExternalGenerationJobDedupeKeyUnique<'ctx> {
+    imp: __sdk::UniqueConstraintHandle<ExternalGenerationJob, String>,
+    phantom: std::marker::PhantomData<&'ctx super::RemoteTables>,
+}
+
+impl<'ctx> ExternalGenerationJobTableHandle<'ctx> {
+    /// Get a handle on the `dedupe_key` unique index on the table `external_generation_job`.
+    pub fn dedupe_key(&self) -> ExternalGenerationJobDedupeKeyUnique<'ctx> {
+        ExternalGenerationJobDedupeKeyUnique {
+            imp: self.imp.get_unique_constraint::<String>("dedupe_key"),
+            phantom: std::marker::PhantomData,
+        }
+    }
+}
+
+impl<'ctx> ExternalGenerationJobDedupeKeyUnique<'ctx> {
+    /// Find the subscribed row whose `dedupe_key` column value is equal to `col_val`,
+    /// if such a row is present in the client cache.
+    pub fn find(&self, col_val: &String) -> Option<ExternalGenerationJob> {
+        self.imp.find(col_val)
+    }
+}
+
+#[doc(hidden)]
+pub(super) fn register_table(client_cache: &mut __sdk::ClientCache<super::RemoteModule>) {
+    let _table = client_cache.get_or_make_table::<ExternalGenerationJob>("external_generation_job");
+    _table.add_unique_constraint::<String>("job_id", |row| &row.job_id);
+    _table.add_unique_constraint::<String>("dedupe_key", |row| &row.dedupe_key);
+}
+
+#[doc(hidden)]
+pub(super) fn parse_table_update(
+    raw_updates: __ws::v2::TableUpdate,
+) -> __sdk::Result<__sdk::TableUpdate<ExternalGenerationJob>> {
+    __sdk::TableUpdate::parse_table_update(raw_updates).map_err(|e| {
+        __sdk::InternalError::failed_parse("TableUpdate<ExternalGenerationJob>", "TableUpdate")
+            .with_cause(e)
+            .into()
+    })
+}
+
+#[allow(non_camel_case_types)]
+/// Extension trait for query builder access to the table `ExternalGenerationJob`.
+///
+/// Implemented for [`__sdk::QueryTableAccessor`].
+pub trait external_generation_jobQueryTableAccess {
+    #[allow(non_snake_case)]
+    /// Get a query builder for the table `ExternalGenerationJob`.
+    fn external_generation_job(&self) -> __sdk::__query_builder::Table<ExternalGenerationJob>;
+}
+
+impl external_generation_jobQueryTableAccess for __sdk::QueryTableAccessor {
+    fn external_generation_job(&self) -> __sdk::__query_builder::Table<ExternalGenerationJob> {
+        __sdk::__query_builder::Table::new("external_generation_job")
+    }
+}