feat: workerize external generation

.hermes/shared-memory/decision-log.md

@@ -48,6 +48,14 @@
 - 验证方式：关闭任一创作入口后，新建创作请求返回 `creation_entry_disabled`；公开作品列表 / 详情 / 启动 / 运行态动作不返回该错误；进入平台首页不弹“平台首页：creation_entry_disabled”；关闭态入口卡显示锁定状态且不显示 `10-20泥点数`。
 - 关联文档：`docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`。
 
+## 2026-06-03 外部内容生成改为持久队列加 worker 角色
+
+- 背景：拼图首图、图集、音频等外部生成链路长期占用 `api-server` HTTP handler，导致扩容只能放大 API 进程，且 HTTP 超时和外部 provider 波动会直接影响创作入口。
+- 决策：外部生成任务统一进入 SpacetimeDB `external_generation_job` 持久队列，由 `api-server` 的 `external-generation-worker` 进程角色 claim lease 后执行；HTTP 角色只做鉴权、表单/状态初始化、入队和返回 `queued/running/completed/failed` 操作状态。生产通过 systemd worker 模板增加实例数或提高 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩容，`GENARRATIVE_PROCESS_ROLE=all` 仅用于本地 smoke。拼图 `compile_puzzle_draft`、结果页 `generate_puzzle_images` 与 `generate_puzzle_ui_background` 已接入 worker；业务写回必须在 SpacetimeDB transaction 内校验 `external_generation_job` 的 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，其中首图 worker 的前置 `compile_puzzle_agent_draft` 也必须带 guard。worker 核心业务写回失败不能返回内存快照并把 job 标成 completed；失败态业务写回成功后才能把 job 标成 failed，失败态未写回则保留租约等待后续重领。拼图业务失败不自动重试，只保留 lease 过期后的崩溃重领，避免钱包扣退费幂等漂移。生产发布会启用默认 `genarrative-external-generation-worker@1.service` 并等待 worker active，worker 停机时停止 claim 新任务并 drain 当前任务。
+- 影响范围：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/spacetime-client/src/external_generation.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`deploy/systemd/genarrative-external-generation-worker@.service`、`scripts/deploy/production-api-deploy.sh`、`scripts/jenkins-server-provision.sh`、拼图 `compile_puzzle_draft`、拼图 `generate_puzzle_images`、拼图 `generate_puzzle_ui_background`、生产 env 模板和运维文档。
+- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`npm run check:server-rs-ddd`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`，并用 `GENARRATIVE_PROCESS_ROLE=all npm run dev` smoke 至少一次 queued -> worker 完成链路。
+- 关联文档：`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`、`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`。
+
 ## 2026-06-03 最近创作只复用创作模板入口
 
 - 背景：底部加号创作入口的“最近创作”最初由真实作品架摘要驱动，但页面曾按作品标题、摘要和生成状态渲染独立最近创作卡，和其它模板页签的卡片样式及点击语义不一致。
@@ -1247,3 +1255,11 @@
 - 影响范围：`server-rs/crates/api-server/src/state.rs`、`server-rs/crates/module-auth/src/lib.rs`、`server-rs/crates/spacetime-module/src/auth/procedures.rs`、`server-rs/crates/spacetime-client/src/auth.rs`、对应生成 bindings。
 - 验证方式：`cargo check -p module-auth --manifest-path server-rs/Cargo.toml`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`cargo test -p module-auth password --manifest-path server-rs/Cargo.toml -- --nocapture`、`npm run check:spacetime-schema`、`npm run check:encoding`、`cargo test -p api-server spacetime_unavailable_router_returns_service_unavailable_for_requests --manifest-path server-rs/Cargo.toml -- --nocapture`。
 - 关联文档：`docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
+
+## 2026-06-03 外部生成 worker lease 使用 SpacetimeDB 时间和 token 栅栏
+
+- 背景：外部生成 worker 支持多进程动态缩扩容后，长任务超过单次 lease、worker 本机时钟漂移或复用 worker id 都可能导致同一任务被重复领取并被过期执行者回写。
+- 决策：`external_generation_job` 新增末尾字段 `lease_token`；`claim` 使用 SpacetimeDB `ctx.timestamp` 计算 lease，生成本次 claim token；worker 执行期间调用 `renew_external_generation_job_lease_and_return` 续租；`complete/fail` 必须带 `worker_id + lease_token` 才能回写。拼图 `compile_puzzle_draft` 的 dedupe key 包含本次 `extgen-` job id，避免同一 session 的失败或完成 job 吞掉后续重新生成。拼图首图前置 `compile_puzzle_agent_draft`、图片保存、UI 背景与失败态业务写回同样必须携带 lease guard，并在 `compile_puzzle_agent_draft`、`save_puzzle_generated_images`、`save_puzzle_ui_background`、`mark_puzzle_draft_generation_failed`、`mark_puzzle_level_generation_failed` 的 SpacetimeDB 事务内校验。
+- 影响范围：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/spacetime-module/src/puzzle.rs`、`server-rs/crates/module-puzzle/src/commands.rs`、`server-rs/crates/spacetime-client/src/external_generation.rs`、`server-rs/crates/spacetime-client/src/puzzle.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/puzzle/handlers.rs`、`server-rs/crates/api-server/src/puzzle/draft.rs`、`server-rs/crates/api-server/src/puzzle/generation.rs`。
+- 验证方式：`npm run spacetime:generate`、`npm run check:spacetime-schema`、`cargo test -p spacetime-module external_generation --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml`、`GENARRATIVE_PROCESS_ROLE=all npm run dev` 后检查 `/healthz`。
+- 关联文档：`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。

.hermes/shared-memory/pitfalls.md

@@ -15,6 +15,38 @@
 - 关联：相关文件、文档、提交或 Issue
 ```
 
+## 外部生成 worker 业务失败重试会撞上钱包扣退费幂等
+
+- 现象：同一个外部生成 job 如果第一次业务失败后退款，再用同一个业务资源 ID 自动重试并成功，钱包 `consume` ledger 可能因为同 ID 已存在而跳过，最终出现“失败已退、成功不再扣”的余额漂移。
+- 原因：资产操作扣费和退款都用稳定 ledger id 做幂等；这能保护 lease 过期后的崩溃重领不重复扣费，但不适合“已明确失败且已退款”的自动业务重试。
+- 处理：拼图 `puzzle_compile_draft` 首期设置 `max_attempts=1`，业务失败直接 failed，只保留 running lease 过期后的崩溃重领。后续若要恢复自动 retry，必须先引入 attempt-aware billing 或可配对撤销的账本接口。
+- 验证：检查 `external_generation_job.max_attempts`、worker 失败回写和钱包 ledger；失败后草稿进入 failed，重试应由用户重新触发新任务，而不是旧 job 自动 pending。
+- 关联：`server-rs/crates/api-server/src/puzzle/handlers.rs`、`server-rs/crates/api-server/src/asset_billing.rs`、`server-rs/crates/spacetime-module/src/runtime/profile.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。
+
+## 外部生成队列不再由 HTTP 进程兜底执行
+
+- 现象：拼图首关生成接口返回 `queued`，但生成页长时间不完成，重启 `genarrative-api.service` 也没有推进任务。
+- 原因：HTTP 角色只入队，不再直接调用外部 provider；如果没有运行 `GENARRATIVE_PROCESS_ROLE=external-generation-worker` 或 `all` 的进程，`external_generation_job` 会停留在 `pending/running`，直到有 worker claim。
+- 处理：生产用 `systemctl enable --now genarrative-external-generation-worker@1.service` 启动至少一个 worker；首次 API deploy 会在默认 worker pattern 下自动启用并启动 `@1`，并等待 worker active。扩容继续启动 `@2.service` 等实例，缩容停止多余实例；worker 收到停机信号后会停止 claim 新任务并等待当前任务完成。本地 smoke 可临时用 `GENARRATIVE_PROCESS_ROLE=all npm run dev`。
+- 验证：`systemctl status 'genarrative-external-generation-worker@*.service'` 能看到 worker 实例；队列任务被 claim 后 `worker_id` 与 `lease_expires_at` 会更新，完成后 session 进入 ready 或 failed。
+- 关联：`deploy/systemd/genarrative-external-generation-worker@.service`、`deploy/env/external-generation-worker.env.example`、`server-rs/crates/spacetime-module/src/external_generation.rs`、`docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`。
+
+## 外部生成 worker 业务写回必须同事务校验 lease guard
+
+- 现象：worker `complete/fail` 已校验 `worker_id + lease_token`，但如果玩法 session / work profile 写回在此之前单独调用，过期 worker 仍可能先写入业务状态，随后才在 job complete/fail 阶段失败；带计费包装的旧 worker 还可能因为 stale guard 错误触发补偿退款。
+- 原因：队列状态栅栏只保护 `external_generation_job` 自身，不会自动保护玩法 procedure。业务写回必须自己带 claim 后的 `job_id / worker_id / lease_token`，并在同一个 SpacetimeDB transaction 内校验 job 仍为 `running`、lease 未过期、job kind、owner 和 source entity 匹配。
+- 处理：拼图首图 worker 的前置 `compile_puzzle_agent_draft`、`save_puzzle_generated_images`、`save_puzzle_ui_background`、`mark_puzzle_draft_generation_failed` 和 `mark_puzzle_level_generation_failed` 已接入 `external_generation_job` lease guard；api-server 的资产扣费包装遇到这类 stale worker lease guard 错误时不执行补偿退款，错误文本包含 `external_generation_job 当前不是 running 状态` 或 `external_generation_job 不存在` 时也按 stale guard 处理。后续迁移其它玩法 worker 时必须复用该模式，不能只在 worker 进程内保存一份 token。
+- 验证：`cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server asset_operation_billing_does_not_refund_stale_worker_lease_errors --manifest-path server-rs/Cargo.toml`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`。
+- 关联：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/spacetime-module/src/puzzle.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/asset_billing.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。
+
+## 外部生成 worker 核心业务写回失败不能完成 job
+
+- 现象：worker 已经生成图片并拿到本地合成 session 快照，但 SpacetimeDB 业务写回因连接、旧 wasm 或 lease guard 失败没有真实落库；如果此时仍把 `external_generation_job` 标成 `completed`，前端只会看到队列完成而 session 长时间不变化，后续也没有 worker 会重领修复。
+- 原因：同步 HTTP handler 的“外部 provider 已成功但 SpacetimeDB 短暂不可用时返回内存快照”降级语义，不能直接搬进异步 worker。worker 的完成状态必须代表核心业务事实已经持久化。
+- 处理：worker 路径的 `save_puzzle_generated_images` / `save_puzzle_ui_background` 等核心业务写回失败时直接返回错误；只有核心写回已经成功后的非关键投影回写才允许降级记录 warning。业务失败态也必须先写回 session / work profile，写回成功后才允许把队列 job 标为 failed；失败态未写回时保留租约，等待 lease 过期后重领。生产首装和首次 API deploy 都必须至少启用一个 worker 实例，例如 `systemctl enable --now genarrative-external-generation-worker@1.service`。
+- 验证：`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server asset_operation_billing_does_not_refund_stale_worker_lease_errors --manifest-path server-rs/Cargo.toml`，并在 smoke 时确认 queued 任务被 worker 消费后 session 真实更新。
+- 关联：`server-rs/crates/api-server/src/puzzle/draft.rs`、`server-rs/crates/api-server/src/puzzle/generation.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。
+
 ## 平台异步错误必须带来源弹窗，不要只显示裸错误
 
 - 现象：用户先后触发多个拼图或草稿生成时，旧请求失败后会在当前页面显示“图片生成失败”等裸错误，容易误判为当前正在看的拼图失败；错误文本也不便复制给开发排查。
@@ -1761,3 +1793,11 @@
 - 处理：推荐页拖拽只校验当前是否有作品、多作品可切换以及是否正在提交动画，不再要求登录；登录态相关操作仍由点赞、改造等按钮自身权限控制。
 - 验证：`npx vitest run src/components/rpg-entry/RpgEntryHomeView.recharge.test.tsx` 覆盖访客态纵向滑动不弹登录且触发下一条推荐。
 - 关联：`src/components/rpg-entry/RpgEntryHomeView.tsx`、`src/components/rpg-entry/RpgEntryHomeView.recharge.test.tsx`。
+
+## 外部生成 worker 不能只靠 worker_id 判定 lease owner
+
+- 现象：外部生成任务超过单次 lease、worker 机器时钟漂移，或 systemd 实例误复用同一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID` 时，同一 job 可能被新 worker 重领，但旧 worker 仍在执行并尝试 complete/fail。
+- 原因：如果队列只校验 `worker_id`，过期执行者仍可能覆盖当前 lease；如果 claim 使用 worker 本机绝对时间，动态扩缩容时的时钟漂移会造成提前抢占或长期锁死。
+- 处理：`external_generation_job` 使用 SpacetimeDB `ctx.timestamp` 计算 claim/renew/complete/fail 时间，并在 claim 时生成 `lease_token`；worker 长任务期间调用 renew，complete/fail 必须携带同一个 `worker_id + lease_token`，且 lease 尚未过期。排查时先看 job 快照里的 `attempt`、`worker_id`、`lease_expires_at` 和 `lease_token` 是否按 claim 递增切换。
+- 验证：`cargo test -p spacetime-module external_generation --manifest-path server-rs/Cargo.toml` 应覆盖 token 和时长 helper；`npm run check:spacetime-schema` 应确认新增字段在表末尾且有默认值。
+- 关联：`server-rs/crates/spacetime-module/src/external_generation.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`。

deploy/container/README.md

@@ -9,11 +9,12 @@ Docker Compose
 ├─ spacetimedb  :3101，独立数据卷，供 api-server 连接
 ├─ nginx        :80 -> api-server:8082，负责静态站点、/admin/、/api/ 反代、upstream timing log、连接限制
 ├─ api-server   :8082，Linux release 构建，连接 compose 内 SpacetimeDB
+├─ external-generation-worker，独立 worker 进程，消费 external_generation_job 队列
 ├─ otelcol      :4317/4318，debug exporter，接收 traces / metrics / logs
 └─ k6           profile=loadtest 时临时启动，在 compose 网络内压 nginx
 ```
 
-当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`。SpacetimeDB 同时设置 `--page_pool_max_size=402653184`，给 reducer、订阅与运行时保留更多非 page pool 内存。
+当前容器模拟参数按 `genarrative-release` 服务器采样值收口为 2 vCPU / 2 GiB RAM / 4096 soft nofile / 768 worker_connections，并已在 compose 里落实到 `spacetimedb cpus=1.0 mem_limit=896m`、`api-server cpus=2.0 mem_limit=1g`、`external-generation-worker cpus=2.0 mem_limit=1g`、`nginx cpus=0.5 mem_limit=128m`、`otelcol cpus=0.25 mem_limit=128m`、`k6 cpus=1.0 mem_limit=512m`。SpacetimeDB 同时设置 `--page_pool_max_size=402653184`，给 reducer、订阅与运行时保留更多非 page pool 内存。
 容器 `api-server` 默认 `GENARRATIVE_API_WORKER_THREADS=4`，用于让 Tokio 在 2 vCPU 配额内有更多 I/O 调度 worker；该值不会突破 compose 里的 `cpus=2.0` CPU 上限。
 Collector 镜像使用 `otel/opentelemetry-collector-contrib:0.151.0`。
 生产服务器若启用 Collector，则由 `deploy/systemd/otelcol-contrib.service` 和 `deploy/otelcol/genarrative-debug.yaml` 托管，不走容器镜像。
@@ -74,6 +75,7 @@ curl -sS http://127.0.0.1:18080/api/runtime/puzzle/gallery
 ```bash
 npm run container:logs -- nginx
 npm run container:logs -- api-server
+npm run container:logs -- external-generation-worker
 npm run container:logs -- otelcol
 ```
 

deploy/container/api-server.env.example

@@ -8,6 +8,12 @@ GENARRATIVE_API_PORT=8082
 GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
 GENARRATIVE_API_WORKER_THREADS=4
+# 容器 smoke 可临时设 all；压测或预发按 api / external-generation-worker 拆进程。
+GENARRATIVE_PROCESS_ROLE=api
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=2
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=3600
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
 GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320
 GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64

deploy/container/docker-compose.loadtest.yml

@@ -69,6 +69,32 @@ services:
       retries: 12
       start_period: 20s
 
+  external-generation-worker:
+    build:
+      context: ../..
+      dockerfile: deploy/container/api-server.Dockerfile
+      target: api-runtime
+    cpus: "2.0"
+    mem_limit: 1g
+    env_file:
+      - ./api-server.env
+    environment:
+      GENARRATIVE_PROCESS_ROLE: external-generation-worker
+      GENARRATIVE_TRACKING_OUTBOX_DIR: /var/lib/genarrative/tracking-outbox-worker
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://otelcol:4318
+      OTEL_SERVICE_NAME: genarrative-external-generation-worker
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
+    depends_on:
+      spacetimedb:
+        condition: service_healthy
+      otelcol:
+        condition: service_started
+
   nginx:
     build:
       context: ../..

deploy/env/api-server.env.example

@@ -7,6 +7,12 @@ GENARRATIVE_API_PORT=8082
 GENARRATIVE_API_LOG=info,tower_http=info
 GENARRATIVE_API_LISTEN_BACKLOG=1024
 GENARRATIVE_API_WORKER_THREADS=4
+# api 只监听 HTTP；外部生成 worker 用独立进程设置为 external-generation-worker 后横向扩缩。
+GENARRATIVE_PROCESS_ROLE=api
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=2
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=3600
 GENARRATIVE_API_MAX_CONCURRENT_REQUESTS=512
 GENARRATIVE_API_GALLERY_MAX_CONCURRENT_REQUESTS=320
 GENARRATIVE_API_DETAIL_MAX_CONCURRENT_REQUESTS=64

deploy/env/external-generation-worker.env.example

@@ -0,0 +1,11 @@
+# 复制到 /etc/genarrative/external-generation-worker.env 后按机器容量调整。
+# 该文件只覆盖 worker 专属参数；SpacetimeDB、外部 provider 密钥继续复用 api-server.env。
+# systemd 模板会强制设置 GENARRATIVE_PROCESS_ROLE=external-generation-worker
+# 和 GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=%H-%i，避免多实例 ID 冲突。
+
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY=2
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS=2000
+# 单次 lease 会由 worker 自动续租；该值覆盖心跳抖动窗口即可。
+GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS=3600
+GENARRATIVE_API_LOG=info,tower_http=info
+OTEL_SERVICE_NAME=genarrative-external-generation-worker

deploy/systemd/genarrative-external-generation-worker@.service

@@ -0,0 +1,29 @@
+[Unit]
+Description=Genarrative External Generation Worker %i
+After=network-online.target spacetimedb.service
+Wants=network-online.target
+Requires=spacetimedb.service
+
+[Service]
+Type=simple
+User=genarrative
+Group=genarrative
+WorkingDirectory=/opt/genarrative/current
+EnvironmentFile=/etc/genarrative/api-server.env
+EnvironmentFile=-/etc/genarrative/external-generation-worker.env
+ExecStart=/usr/bin/env GENARRATIVE_PROCESS_ROLE=external-generation-worker GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID=%H-%i GENARRATIVE_TRACKING_OUTBOX_DIR=/var/lib/genarrative/tracking-outbox/%H-%i OTEL_SERVICE_NAME=genarrative-external-generation-worker /opt/genarrative/current/api-server
+Restart=always
+RestartSec=5
+KillSignal=SIGINT
+TimeoutStopSec=7200
+LimitNOFILE=65535
+TasksMax=2048
+
+# worker 复用 api-server 发布目录；外部生成审计与临时运行态只写服务端私有目录。
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=full
+ReadWritePaths=/opt/genarrative /var/lib/genarrative
+
+[Install]
+WantedBy=multi-user.target

docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md

@@ -0,0 +1,153 @@
+# 外部生成 Worker 化方案
+
+更新时间：`2026-06-03`
+
+## 背景
+
+当前 VectorEngine `gpt-image-2`、音频、LLM 等外部生成链路多数由 `api-server` 的 HTTP handler 直接等待上游、OSS 持久化和 SpacetimeDB 回写完成。前端虽然有生成页和会话轮询，但 HTTP 进程仍承担长耗时副作用，导致接入更多玩法或大图生成时只能放大 API 进程，而不能单独扩展外部生成吞吐。
+
+## 目标
+
+- `api-server` 的 HTTP 角色只负责鉴权、入参校验、扣费前置/状态初始化、任务入队和返回 `queued` 操作结果。
+- 外部生成副作用由独立 `external-generation-worker` 角色执行。
+- 多个 worker 进程通过 SpacetimeDB 任务表抢占任务，依赖 lease 超时恢复，支持按进程数和单进程并发动态缩扩容。
+- SpacetimeDB reducer / procedure 只做任务状态流转，不做网络、文件系统或外部 provider I/O。
+- 已接入拼图 `compile_puzzle_draft` 与结果页 `generate_puzzle_images`；后续玩法继续复用同一队列 Module，不再为每个玩法发明独立队列。
+
+## Module 与 Interface
+
+新增深一点的 **外部生成任务 Module**，Interface 收敛为：
+
+- `enqueue_external_generation_job_and_return`：按 `dedupe_key` 幂等创建或返回现有任务。
+- `claim_external_generation_jobs_and_return`：worker 按 `worker_id`、`limit` 和 lease 时长抢占 `pending` 或 lease 过期的 `running` 任务，返回本次 claim 的 `lease_token`。
+- `renew_external_generation_job_lease_and_return`：worker 长任务执行期间按 `worker_id + lease_token` 续租，防止外部生成超过单次 lease 后被重复领取。
+- `complete_external_generation_job_and_return`：worker 成功后按 `worker_id + lease_token` 写入 `result_payload_json`，任务进入 `completed`。
+- `fail_external_generation_job_and_return`：worker 失败后按 `worker_id + lease_token` 回写错误，并按 `max_attempts` 决定回到 `pending` 重试或进入 `failed`。
+
+这个 Module 的 **Seam** 在 SpacetimeDB procedure + `spacetime-client` facade；`api-server` HTTP role 和 worker role 都只依赖这个 Interface。外部 provider、OSS、计费补偿、玩法草稿回写仍留在 `api-server` worker implementation 内，不进入 SpacetimeDB reducer。
+
+## 任务表
+
+新增私有表 `external_generation_job`：
+
+| 字段 | 说明 |
+| --- | --- |
+| `job_id` | 主键，`extgen-` 前缀 UUID |
+| `dedupe_key` | 唯一键，建议为 `play/action/session/scope` |
+| `job_kind` | 执行类型，当前拼图为 `puzzle_compile_draft`、`puzzle_generate_images`、`puzzle_generate_ui_background` |
+| `owner_user_id` | 触发用户 |
+| `source_module` | 玩法或能力名，例如 `puzzle` |
+| `source_entity_id` | session/profile/work 等作用域 |
+| `request_label` | 排障标签 |
+| `request_payload_json` | worker 执行入参 JSON |
+| `status` | `pending/running/completed/failed/cancelled` |
+| `attempt` / `max_attempts` | 当前尝试次数与最大尝试次数 |
+| `last_error_message` | 最近失败原因 |
+| `worker_id` | 当前 lease owner |
+| `lease_expires_at` | lease 到期时间 |
+| `lease_token` | 本次 claim 的 fencing token，用于阻止过期 worker 回写 |
+| `available_at` | 下次可领取时间 |
+| `result_payload_json` | 完成摘要 |
+| `created_at/started_at/completed_at/updated_at` | 审计时间 |
+
+索引：
+
+- `by_external_generation_job_status_available(status, available_at)`
+- `by_external_generation_job_worker_id(worker_id)`
+- `by_external_generation_job_source(source_module, source_entity_id)`
+- `by_external_generation_job_owner_user_id(owner_user_id)`
+
+## 状态机
+
+```text
+pending -> running -> completed
+pending -> running -> pending   (可重试失败)
+pending -> running -> failed    (达到最大重试次数)
+pending/running -> cancelled    (预留)
+```
+
+`claim` 只领取 `pending` 且 `available_at <= now` 的任务，或 `running` 且 `lease_expires_at <= now` 的任务。领取时递增 `attempt`、写入 `worker_id`、`started_at`、新的 `lease_expires_at` 和 `lease_token`。SpacetimeDB procedure 使用 `ctx.timestamp` 作为状态流转时间，只从 worker 入参读取“时长差值”，不信任 worker 本机绝对时间。worker 每次执行只处理自己 claim 到的任务；续租、完成或失败时必须带同一个 `worker_id + lease_token`，且当前 lease 尚未过期，防止过期 worker 覆盖新 lease。
+
+玩法业务写回也必须在 SpacetimeDB 同一事务里校验 lease fencing。拼图的 `compile_puzzle_agent_draft` worker 调用、`save_puzzle_generated_images`、`save_puzzle_ui_background`、`mark_puzzle_draft_generation_failed` 和 `mark_puzzle_level_generation_failed` 会带 `external_generation_job_id / worker_id / lease_token`，并校验 job 仍为 `running`、token 未过期、`job_kind`、`owner_user_id`、`source_module` 和 `source_entity_id` 均匹配后才写 session / work profile。worker 路径的核心业务写回失败不能返回内存快照并把 job 标为 `completed`；失败态业务回写成功后才允许把队列 job 标为 `failed`，失败态仍未写回时保留当前租约并等待后续 lease 过期重领，避免队列状态和真实 session 脱节。api-server 的资产扣费包装遇到这类 stale worker lease guard 错误时不执行补偿退款，避免旧 worker 冲掉后续合法 worker 的同一账本扣费。
+
+## 进程角色
+
+同一个 Rust binary 通过 `GENARRATIVE_PROCESS_ROLE` 切换：
+
+- `api`：只启动 HTTP server。
+- `external-generation-worker`：只启动外部生成 worker，不监听 HTTP。
+- `all`：本地开发可同时启动 HTTP 与 worker。
+
+worker 配置：
+
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`：实例 ID；未配置时用 hostname/pid 派生。
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY`：单进程并发领取/执行数量。
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS`：空队列轮询间隔。
+- `GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS`：任务 lease 时长；worker 会按约三分之一 lease、最长 30 秒的间隔续租。该值应覆盖一次心跳网络抖动窗口，不需要大于完整外部生成链路耗时。
+
+动态缩扩容方式：生产通过 `deploy/systemd/genarrative-external-generation-worker@.service` 或进程管理器启动更多 `external-generation-worker` 实例；无需改变 HTTP 进程数。缩容或发布重启 worker 时，进程收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成；若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`，未完成任务会在 lease 过期后被其它 worker 重新领取。容器链路已有独立 `external-generation-worker` compose service；扩 worker 必须扩这个 worker service，不能只扩 `api-server` HTTP service。
+
+## 已接入的拼图纵切
+
+`compile_puzzle_draft`：
+
+1. HTTP handler 保存拼图表单草稿；`queued/running` 的持久事实源是 `external_generation_job`，不把 HTTP 进程变成外部生成执行者。
+2. HTTP handler 入队 `puzzle_compile_draft`，返回 `operation.status = queued` 和当前 session。拼图 dedupe key 包含本次 `extgen-` job id，只保证同一任务行唯一，不把同一 session 后续重新生成吞掉。
+3. 前端保持 `puzzle-generating`，继续轮询 `getPuzzleAgentSession`；首期不把 `queued/running` 写回 `puzzle_agent_session`，因此刷新或跨设备恢复生成中状态仍是后续 read model 工作。
+4. worker claim 后执行原有 `compile_puzzle_draft_with_initial_cover` 或 `compile_puzzle_draft_with_uploaded_cover`；前置 `compile_puzzle_agent_draft` 也必须携带本次 `job_id / worker_id / lease_token`，防止过期 worker 先把草稿卡和 session 写到 ready。
+5. 成功后沿原有 SpacetimeDB 拼图会话/作品写回，前端轮询看到 `progressPercent >= 94/96/100` 和 ready 草稿。
+6. 失败后调用 `mark_puzzle_draft_generation_failed`，拼图首期业务失败直接进入 failed；只有失败态写回成功才把队列 job 标为 failed，失败态写回失败则保留租约等待重领。队列仍保留 lease 过期后的崩溃重领，避免 worker 退款后再次成功导致钱包账本漂移。前端通过现有失败草稿/弹窗机制展示来源错误。
+
+`generate_puzzle_images`：
+
+1. HTTP handler 校验本次 `levelsJson` 快照后入队 `puzzle_generate_images`，返回 `operation.status = queued/running/completed/failed`。
+2. worker 执行原结果页关卡图链路：自动命名、VectorEngine / 上传图直用、关卡场景图、UI spritesheet、关卡背景资产包、OSS 持久化和 SpacetimeDB 回写。
+3. 成功后 `save_puzzle_generated_images` 写回目标关卡和草稿卡；失败后 `mark_puzzle_level_generation_failed` 只标记目标关卡 `failed`，不污染已 ready 的其它关卡。队列 job 只有在目标关卡失败态写回成功后才进入 failed。
+4. 前端结果页对 `queued/running` 操作继续轮询 `getPuzzleAgentSession`，目标关卡变为 ready 或 failed 后收敛。
+
+`generate_puzzle_ui_background`：
+
+1. HTTP handler 校验本次 `levelsJson` 快照后入队 `puzzle_generate_ui_background`，返回 `operation.status = queued/running/completed/failed`。
+2. worker 执行原结果页 UI 背景链路：归一化提示词、VectorEngine 生成、OSS 持久化和 `save_puzzle_ui_background` 写回。
+3. 成功后目标关卡写入 `uiBackgroundPrompt/uiBackgroundImageSrc/uiBackgroundImageObjectKey`；失败后复用 `mark_puzzle_level_generation_failed` 标记目标关卡 `failed`，并在失败态写回成功后才终结队列 job，让前端轮询能收敛。
+
+Match3D、Wooden Fish、Visual Novel 音频等后续外部生成 action 按同一模式迁移。
+
+## 验收
+
+基础检查：
+
+```bash
+npm run spacetime:generate
+npm run check:spacetime-schema
+npm run check:server-rs-ddd
+cargo check -p api-server --manifest-path server-rs/Cargo.toml
+```
+
+定向测试：
+
+```bash
+cargo test -p spacetime-module external_generation --manifest-path server-rs/Cargo.toml
+cargo test -p spacetime-module level_generation_failure --manifest-path server-rs/Cargo.toml
+cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml
+npm run test -- src/components/puzzle-result/PuzzleResultView.test.tsx -t "keeps generation progress visible"
+npm run test -- src/components/rpg-entry/RpgEntryFlowShell.agent.interaction.test.tsx -t "compile_puzzle_draft"
+```
+
+本地 smoke：
+
+```bash
+GENARRATIVE_PROCESS_ROLE=all npm run dev
+curl -f http://127.0.0.1:<api-port>/healthz
+```
+
+生产 smoke 需要至少启动一个 `api` 角色和一个 `external-generation-worker` 角色；发布脚本会在默认 worker pattern 下自动启用并启动 `genarrative-external-generation-worker@1.service`，并等待 worker active。若 worker 数量归零，生成任务会保持 `queued/running`，不会由 HTTP 进程偷偷执行。
+
+systemd 生产扩缩容示例：
+
+```bash
+systemctl enable --now genarrative-external-generation-worker@1.service
+systemctl start genarrative-external-generation-worker@2.service
+systemctl stop genarrative-external-generation-worker@2.service
+systemctl status 'genarrative-external-generation-worker@*.service'
+```

docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md

@@ -98,7 +98,7 @@ npm run check:server-rs-ddd
 - `server-rs/crates/api-server/src/state.rs` 中的 `PuzzleApiState` 是拼图 HTTP/BFF 的 Feature State，集中暴露 `SpacetimeClient`、`PuzzleGalleryCache`、OSS client、作者查询所需认证服务、拼图 LLM client 和少量 VectorEngine / Agent 配置快照。拼图 handler 只提取 `State<PuzzleApiState>`，不得重新改回 `State<AppState>`。
 - `server-rs/crates/api-server/src/puzzle.rs` 只作为聚合入口，保留共享 import / 常量、内部模块声明和 handler re-export，不继续承载大段实现。
 - `server-rs/crates/api-server/src/puzzle/handlers.rs` 承接 Axum handler，负责 extract、鉴权上下文、调用 SpacetimeDB facade / 编排 helper，并返回 HTTP/SSE 响应。
-- `server-rs/crates/api-server/src/puzzle/draft.rs` 承接表单草稿保存、草稿编译、首关命名、UI 背景 prompt、降级 snapshot 和初始资产就绪校验。
+- `server-rs/crates/api-server/src/puzzle/draft.rs` 承接表单草稿保存、草稿编译、首关命名、UI 背景 prompt 和初始资产就绪校验。
 - `server-rs/crates/api-server/src/puzzle/generation.rs` 承接拼图图片与 UI 背景的生成编排、计费包裹和 reference image 路径选择。
 - `server-rs/crates/api-server/src/puzzle/vector_engine.rs` 承接 VectorEngine 请求体、HTTP 调用、下载 / base64 解码、OSS 写入、asset object / binding 持久化和上游错误归一。
 - `server-rs/crates/api-server/src/puzzle/mappers.rs` 承接 SpacetimeDB record 到 shared-contracts DTO 的映射。
@@ -203,6 +203,12 @@ npm run check:server-rs-ddd
 - Rust 结构体：`AiTaskStage`
 - 源码：`server-rs/crates/spacetime-module/src/ai/stages.rs`
 
+### `external_generation_job`
+
+- Rust 结构体：`ExternalGenerationJob`
+- 源码：`server-rs/crates/spacetime-module/src/external_generation.rs`
+- 用途：外部生成 worker 的持久任务队列；`api-server` HTTP 角色只入队，`external-generation-worker` 角色通过 claim lease 领取、续租、执行，并用 `lease_token` 栅栏回写完成 / 失败。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft`、`generate_puzzle_images` 与 `generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，避免过期 worker 写 session / work profile。worker 成功写回业务事实后才能 complete job；业务失败态写回成功后才能 fail job，失败态未写回时保留租约等待后续重领。
+
 ### `ai_text_chunk`
 
 - Rust 结构体：`AiTextChunk`

docs/【开发运维】本地开发验证与生产运维-2026-05-15.md

@@ -51,6 +51,8 @@ Linux 本机多用户并发开发时，`npm run dev` 和 `npm run dev:*` 单模
 
 开发态 `npm run dev` 与 `npm run dev:api-server` 会默认注入 `GENARRATIVE_DEV_PASSWORD_ENTRY_AUTO_REGISTER_ENABLED=true`，因此密码登录在本地开发环境可直接注册未知手机号账号；生产环境仍按 `api-server` 配置默认关闭该开关。
 
+本地排查外部内容生成 worker 时，可临时用 `GENARRATIVE_PROCESS_ROLE=all npm run dev:api-server` 让同一 Rust 进程同时监听 HTTP 并消费 `external_generation_job` 队列。该模式只用于 smoke；生产默认 `GENARRATIVE_PROCESS_ROLE=api`，外部生成任务由独立 `GENARRATIVE_PROCESS_ROLE=external-generation-worker` 进程消费。拼图首图 `compile_puzzle_draft`、结果页关卡图片 `generate_puzzle_images` 和结果页 UI 背景 `generate_puzzle_ui_background` 都走该队列；worker 数量为 0 时，HTTP 只返回 queued/running，不会兜底执行外部 provider。
+
 微信小程序虚拟支付使用 `WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_OFFER_ID`、`WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_APP_KEY`、`WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_SANDBOX_APP_KEY` 和 `WECHAT_MINI_PROGRAM_VIRTUAL_PAYMENT_ENV` 配置。小程序充值统一走 `wechat_mp_virtual` / `wx.requestVirtualPayment`：泥点属于代币（`coin`），`buyQuantity` 按当前充值商品快照里的 `points_amount` 传；会员和后台新增道具类商品走 `short_series_goods`，`productId` 对应微信后台道具 ID。旧登录快照若缺 `session_key`，需要用户在小程序内重新登录后再支付；客户端成功回调不是最终到账，仍以后端通知或查询确认订单为准。详细口径见 `docs/【技术方案】微信虚拟支付接入-2026-05-26.md`。
 
 如果本地 `GET /api/creation-entry/config` 返回 `No such procedure`，或 `api-server` 日志出现 `no such table: puzzle_gallery_card_view` / `no such table: wooden_fish_gallery_card_view` 这类公开 view 缺失，通常是 `.env.local` 指向的 SpacetimeDB 库还没有发布当前 `spacetime-module`，或当前 CLI 身份无权发布该库。debug 构建的 `api-server` 会临时使用后端默认入口配置兜底，避免创作作品架整块消失；正式修复仍应切换到拥有目标库权限的 SpacetimeDB 身份后重新运行 `npm run dev` 完成发布，或用 gitignored 的 `spacetime.local.json` 指向可发布的本地库。
@@ -255,6 +257,8 @@ Jenkins 按 web / api / Spacetime module / build / deploy / publish 拆分
 
 生产环境变量模板：`deploy/env/api-server.env.example`。真实密钥只放服务器，不提交 Git，不写入文档示例。
 
+`api-server` 进程角色由 `GENARRATIVE_PROCESS_ROLE` 控制：`api` 只监听 HTTP，`external-generation-worker` 只消费外部生成队列，`all` 仅用于本地或临时 smoke。外部生成 worker 使用同一发布包和同一套 SpacetimeDB 配置，按实例数和 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩；扩容时增加 worker 进程或提高单进程并发，缩容时停止多余 worker。worker 收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成；若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`，未完成任务才会在 lease 过期后由其它 worker 重领。每个 worker 实例应设置唯一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`，默认会用主机名和 pid 兜底；systemd 生产模板 `deploy/systemd/genarrative-external-generation-worker@.service` 会用 `%H-%i` 生成实例 ID，并把 tracking outbox 隔离到 `/var/lib/genarrative/tracking-outbox/%H-%i`。`Genarrative-Server-Provision` 会默认 enable 首个 `genarrative-external-generation-worker@1.service`，并在已存在 `/opt/genarrative/current/api-server` 时随 API 一起重启；首次 API deploy 会在默认 worker pattern 下自动 `enable --now genarrative-external-generation-worker@1.service` 并等待 worker active。手动持久化首个实例可用 `systemctl enable --now genarrative-external-generation-worker@1.service`，横向扩容用 `systemctl start genarrative-external-generation-worker@2.service` / `@3.service`，缩容用 `systemctl stop genarrative-external-generation-worker@N.service`。worker 专属参数模板是 `deploy/env/external-generation-worker.env.example`，密钥与 SpacetimeDB 连接仍复用 `/etc/genarrative/api-server.env`。API 发布脚本默认会重启并验活 `genarrative-external-generation-worker@*.service`；若本次只发 HTTP 且不希望滚动 worker，可传 `--no-worker-services`。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 控制空队列轮询间隔，`GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS` 控制单次 lease，worker 会约每三分之一 lease、最长 30 秒续租；该值应覆盖一次心跳网络抖动窗口，不需要大于完整外部生成链路耗时。SpacetimeDB 使用自身事务时间计算 claim/renew/complete/fail，完成和失败回写还会校验 `lease_token` 与未过期 lease，避免同一 job 被过期 worker 覆盖。当前拼图首关生成只做 lease 崩溃重领，不做业务失败自动重试，避免 worker 退款和重试成功之间产生钱包账本漂移。
+
 `Genarrative-Server-Provision` 会安装 systemd 模板和 Nginx 站点模板，不再安装 clang / lld / pkg-config / OpenSSL headers / sccache 等构建链依赖。Ubuntu / apt 目标机会额外安装 `libnginx-mod-http-brotli-filter` 与 `libnginx-mod-http-brotli-static`，随后由 `scripts/jenkins-server-provision.sh` 通过临时 `nginx -t` 配置探测 Brotli 指令是否可用；该临时配置必须先 `include /etc/nginx/modules-enabled/*.conf`，因为 apt 安装的 Brotli 是动态模块，不会出现在普通 `nginx -V` 编译参数里。探测成功才在渲染后的 `deploy/nginx/genarrative.conf` / `genarrative-dev-http.conf` 中启用 Brotli，避免未安装模块的机器直接写入无效配置。Provision 写入 Genarrative Nginx 站点时会把 `/etc/nginx/sites-enabled/default*` 移到 `/etc/nginx/sites-disabled/`，避免 Debian / Certbot 默认站点继续占用 `genarrative.world` / `www.genarrative.world` 并在 `nginx -T` 中出现 `conflicting server name ... ignored`。如果 `nginx -t` 失败，脚本会恢复写入前的 Genarrative 配置和被移动的默认站点。
 
 50 HTTP req/s 首版压测优化口径：

scripts/check-spacetime-schema-guard.mjs

@@ -475,10 +475,16 @@ function loadBaseSources(baseRef) {
 
 function getChangedFiles(baseRef) {
   const diffOutput = tryGit(['diff', '--name-only', '-z', baseRef, '--']) ?? '';
-  const untrackedOutput =
+  const untrackedModuleOutput =
     tryGit(['ls-files', '--others', '--exclude-standard', '-z', moduleSrcRoot]) ?? '';
+  const untrackedBindingsOutput =
+    tryGit(['ls-files', '--others', '--exclude-standard', '-z', bindingsRoot]) ?? '';
   return new Set(
-    [...diffOutput.split(/\u0000/u), ...untrackedOutput.split(/\u0000/u)]
+    [
+      ...diffOutput.split(/\u0000/u),
+      ...untrackedModuleOutput.split(/\u0000/u),
+      ...untrackedBindingsOutput.split(/\u0000/u),
+    ]
       .map(normalizePath)
       .filter(Boolean),
   );

scripts/deploy/production-api-deploy.sh

@@ -5,10 +5,11 @@ set -euo pipefail
 usage() {
   cat <<'EOF'
 用法:
-  ./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101]
+  ./scripts/deploy/production-api-deploy.sh --source-dir build/<version> [--version <version>] [--release-root /opt/genarrative/releases] [--current-link /opt/genarrative/current] [--service genarrative-api.service] [--worker-service-pattern 'genarrative-external-generation-worker@*.service'] [--no-worker-services] [--health-url http://127.0.0.1:8082/readyz] [--api-env-file /etc/genarrative/api-server.env] [--database genarrative-prod] [--spacetime-server-url http://127.0.0.1:3101]
 
 说明:
   进入维护模式，校验并发布 api-server 单文件，更新 current 链接，重启 systemd 服务并执行 readiness 检查。
+  默认同时重启已加载的外部生成 worker 实例；未启用 worker 单元时会自动跳过。
   若传入 --database，会在重启前把 GENARRATIVE_SPACETIME_DATABASE 写入 api-server 环境文件，避免服务继续读取旧库。
   失败时保留维护模式。
 EOF
@@ -223,12 +224,106 @@ ensure_runtime_env_and_dirs() {
   fi
 }
 
+list_worker_services() {
+  local pattern="$1"
+
+  if [[ -z "${pattern}" ]]; then
+    return 0
+  fi
+
+  systemctl list-units --all --plain --no-legend "${pattern}" 2>/dev/null | awk '{print $1}' | sort -u
+}
+
+ensure_default_worker_service() {
+  local pattern="$1"
+  local default_service="genarrative-external-generation-worker@1.service"
+  local template_service="genarrative-external-generation-worker@.service"
+  local services=()
+
+  if [[ -z "${pattern}" ]]; then
+    return 0
+  fi
+
+  if [[ "${pattern}" != "genarrative-external-generation-worker@*.service" ]]; then
+    return 0
+  fi
+
+  if ! systemctl cat "${template_service}" >/dev/null 2>&1; then
+    echo "[production-api-deploy] 缺少外部生成 worker systemd 模板: ${template_service}" >&2
+    return 1
+  fi
+
+  mapfile -t services < <(list_worker_services "${pattern}")
+  if [[ "${#services[@]}" -gt 0 ]]; then
+    return 0
+  fi
+
+  echo "[production-api-deploy] 未发现外部生成 worker 实例，启用并启动默认实例: ${default_service}"
+  systemctl enable --now "${default_service}"
+}
+
+restart_worker_services() {
+  local pattern="$1"
+  local services=()
+
+  if [[ -z "${pattern}" ]]; then
+    echo "[production-api-deploy] 跳过外部生成 worker 重启。"
+    return 0
+  fi
+
+  ensure_default_worker_service "${pattern}"
+  mapfile -t services < <(list_worker_services "${pattern}")
+  if [[ "${#services[@]}" -eq 0 ]]; then
+    echo "[production-api-deploy] 未发现已加载的外部生成 worker 单元: ${pattern}" >&2
+    return 1
+  fi
+
+  echo "[production-api-deploy] 重启外部生成 worker: ${services[*]}"
+  systemctl restart "${services[@]}"
+}
+
+wait_for_worker_services() {
+  local pattern="$1"
+  local services=()
+  local all_active
+
+  if [[ -z "${pattern}" ]]; then
+    return 0
+  fi
+
+  mapfile -t services < <(list_worker_services "${pattern}")
+  if [[ "${#services[@]}" -eq 0 ]]; then
+    echo "[production-api-deploy] 外部生成 worker 单元不存在，发布失败: ${pattern}" >&2
+    return 1
+  fi
+
+  echo "[production-api-deploy] 等待外部生成 worker active: ${services[*]}"
+  for _ in {1..30}; do
+    all_active=1
+    for service in "${services[@]}"; do
+      if ! systemctl is-active --quiet "${service}"; then
+        all_active=0
+        break
+      fi
+    done
+    if [[ "${all_active}" -eq 1 ]]; then
+      return 0
+    fi
+    sleep 2
+  done
+
+  systemctl --no-pager --full status "${services[@]}" || true
+  echo "[production-api-deploy] 外部生成 worker 未在超时时间内进入 active，发布失败。" >&2
+  return 1
+}
+
 SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
 SOURCE_DIR=""
 VERSION=""
 RELEASE_ROOT="/opt/genarrative/releases"
 CURRENT_LINK="/opt/genarrative/current"
 SERVICE_NAME="genarrative-api.service"
+WORKER_SERVICE_PATTERN="genarrative-external-generation-worker@*.service"
 HEALTH_URL="http://127.0.0.1:8082/readyz"
 API_ENV_FILE="/etc/genarrative/api-server.env"
 DATABASE=""
@@ -261,6 +356,14 @@ while [[ $# -gt 0 ]]; do
       SERVICE_NAME="${2:?缺少 --service 的值}"
       shift 2
       ;;
+    --worker-service-pattern)
+      WORKER_SERVICE_PATTERN="${2:?缺少 --worker-service-pattern 的值}"
+      shift 2
+      ;;
+    --no-worker-services)
+      WORKER_SERVICE_PATTERN=""
+      shift
+      ;;
     --health-url)
       HEALTH_URL="${2:?缺少 --health-url 的值}"
       shift 2
@@ -362,6 +465,8 @@ ln -sfn "${RELEASE_DIR}" "${CURRENT_LINK}"
 
 echo "[production-api-deploy] 重启服务: ${SERVICE_NAME}"
 systemctl restart "${SERVICE_NAME}"
+restart_worker_services "${WORKER_SERVICE_PATTERN}"
+wait_for_worker_services "${WORKER_SERVICE_PATTERN}"
 
 echo "[production-api-deploy] 等待 readiness: ${HEALTH_URL}"
 for _ in {1..30}; do

scripts/jenkins-server-provision.sh

@@ -4,6 +4,7 @@ set -euo pipefail
 PROVISION_TOOLS_DIR="${PROVISION_TOOLS_DIR:-provision-tools}"
 SPACETIME_BIN_SOURCE="${SPACETIME_BIN_SOURCE:-${PROVISION_TOOLS_DIR}/spacetime/spacetime}"
 OTELCOL_BIN_SOURCE="${OTELCOL_BIN_SOURCE:-${PROVISION_TOOLS_DIR}/otelcol-contrib}"
+WORKER_ENV_FILE="${WORKER_ENV_FILE:-/etc/genarrative/external-generation-worker.env}"
 
 require_non_root_relative_path() {
   local label="$1"
@@ -417,6 +418,10 @@ render_api_env_example() {
     deploy/env/api-server.env.example
 }
 
+render_external_generation_worker_env_example() {
+  cat deploy/env/external-generation-worker.env.example
+}
+
 render_otelcol_service() {
   cat deploy/systemd/otelcol-contrib.service
 }
@@ -603,6 +608,18 @@ render_api_service() {
     deploy/systemd/genarrative-api.service
 }
 
+render_external_generation_worker_service() {
+  local current_escaped api_env_escaped worker_env_escaped
+  current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
+  api_env_escaped="$(escape_sed_replacement "${API_ENV_FILE}")"
+  worker_env_escaped="$(escape_sed_replacement "${WORKER_ENV_FILE}")"
+  sed \
+    -e "s|/opt/genarrative/current|${current_escaped}|g" \
+    -e "s|/etc/genarrative/api-server.env|${api_env_escaped}|g" \
+    -e "s|/etc/genarrative/external-generation-worker.env|${worker_env_escaped}|g" \
+    deploy/systemd/genarrative-external-generation-worker@.service
+}
+
 render_database_backup_service() {
   local current_escaped env_escaped
   current_escaped="$(escape_sed_replacement "${CURRENT_LINK}")"
@@ -615,6 +632,7 @@ render_database_backup_service() {
 
 require_path deploy/systemd/spacetimedb.service
 require_path deploy/systemd/genarrative-api.service
+require_path deploy/systemd/genarrative-external-generation-worker@.service
 require_path deploy/systemd/genarrative-database-backup.service
 require_path deploy/systemd/genarrative-database-backup.timer
 require_path deploy/systemd/otelcol-contrib.service
@@ -623,6 +641,7 @@ require_path deploy/nginx/genarrative.conf
 require_path deploy/nginx/genarrative-dev-http.conf
 require_path deploy/nginx/snippets/genarrative-maintenance.conf
 require_path deploy/env/api-server.env.example
+require_path deploy/env/external-generation-worker.env.example
 require_path scripts/deploy/maintenance-on.sh
 require_path scripts/deploy/maintenance-off.sh
 require_path scripts/deploy/maintenance-status.sh
@@ -665,15 +684,18 @@ sync_spacetime_install "${SPACETIME_ROOT}"
 
 spacetimedb_service="$(mktemp)"
 api_service="$(mktemp)"
+external_generation_worker_service="$(mktemp)"
 database_backup_service="$(mktemp)"
 render_spacetimedb_service >"${spacetimedb_service}"
 render_api_service >"${api_service}"
+render_external_generation_worker_service >"${external_generation_worker_service}"
 render_database_backup_service >"${database_backup_service}"
 install_file "${spacetimedb_service}" /etc/systemd/system/spacetimedb.service 0644
 install_file "${api_service}" /etc/systemd/system/genarrative-api.service 0644
+install_file "${external_generation_worker_service}" /etc/systemd/system/genarrative-external-generation-worker@.service 0644
 install_file "${database_backup_service}" /etc/systemd/system/genarrative-database-backup.service 0644
 install_file deploy/systemd/genarrative-database-backup.timer /etc/systemd/system/genarrative-database-backup.timer 0644
-rm -f "${spacetimedb_service}" "${api_service}" "${database_backup_service}"
+rm -f "${spacetimedb_service}" "${api_service}" "${external_generation_worker_service}" "${database_backup_service}"
 
 if [[ ! -f "${API_ENV_FILE}" ]]; then
   echo "+ create ${API_ENV_FILE} from example"
@@ -687,6 +709,17 @@ else
 fi
 ensure_api_runtime_env_defaults
 
+if [[ ! -f "${WORKER_ENV_FILE}" ]]; then
+  echo "+ create ${WORKER_ENV_FILE} from example"
+  if [[ "${DRY_RUN}" != "true" ]]; then
+    render_external_generation_worker_env_example >"${WORKER_ENV_FILE}"
+    chmod 0600 "${WORKER_ENV_FILE}"
+    chown root:root "${WORKER_ENV_FILE}"
+  fi
+else
+  echo "[server-provision] 已存在 worker 环境文件，保留不覆盖: ${WORKER_ENV_FILE}"
+fi
+
 if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
   sync_otelcol_install
   otelcol_service="$(mktemp)"
@@ -708,7 +741,7 @@ if [[ "${ENABLE_SERVICES}" == "true" ]]; then
   if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
     run_cmd systemctl enable otelcol-contrib.service
   fi
-  run_cmd systemctl enable spacetimedb.service genarrative-api.service genarrative-database-backup.timer
+  run_cmd systemctl enable spacetimedb.service genarrative-api.service genarrative-database-backup.timer genarrative-external-generation-worker@1.service
   if [[ "${ENABLE_OTELCOL:-true}" == "true" ]]; then
     run_cmd systemctl restart otelcol-contrib.service
   fi
@@ -717,8 +750,10 @@ if [[ "${ENABLE_SERVICES}" == "true" ]]; then
   ensure_spacetime_owner_client_token
   if [[ -x "${CURRENT_LINK}/api-server" ]]; then
     run_cmd systemctl restart genarrative-api.service
+    run_cmd systemctl enable --now genarrative-external-generation-worker@1.service
+    run_cmd systemctl restart genarrative-external-generation-worker@1.service
   else
-    echo "[server-provision] 尚未发现 ${CURRENT_LINK}/api-server，跳过 api-server 首次启动。后续 API deploy 会重启服务。"
+    echo "[server-provision] 尚未发现 ${CURRENT_LINK}/api-server，跳过 api-server 和外部生成 worker 首次启动。后续 API deploy 会启用并启动默认 worker 实例。"
   fi
 fi
 

server-rs/crates/api-server/src/app.rs

@@ -4119,8 +4119,7 @@ mod tests {
             .await
             .expect("banners body should collect")
             .to_bytes();
-        let payload: Value =
-            serde_json::from_slice(&body).expect("banners payload should be json");
+        let payload: Value = serde_json::from_slice(&body).expect("banners payload should be json");
 
         assert_eq!(payload["eventBanners"][0]["title"], "后台表单公告");
         assert_eq!(payload["eventBanners"][0]["renderMode"], "html");

server-rs/crates/api-server/src/asset_billing.rs

@@ -48,7 +48,7 @@ where
     match operation.await {
         Ok(value) => Ok(value),
         Err(error) => {
-            if points_consumed {
+            if points_consumed && should_refund_asset_operation_error(&error) {
                 refund_asset_operation_points(
                     state,
                     owner_user_id,
@@ -63,6 +63,20 @@ where
     }
 }
 
+pub(crate) fn should_refund_asset_operation_error(error: &AppError) -> bool {
+    let message = error.body_text();
+    // 中文注释：worker lease guard 拒绝表示当前进程已失去队列写权限；
+    // 这类 stale worker 失败不能补偿退款，否则可能冲掉后续合法 worker 的同一账本扣费。
+    !(message.contains("external_generation_job")
+        && (message.contains("lease")
+            || message.contains("worker")
+            || message.contains("job_kind")
+            || message.contains("source_")
+            || message.contains("owner_user_id")
+            || message.contains("不存在")
+            || message.contains("不是 running 状态")))
+}
+
 /// 资产操作统一预扣泥点；扣费流水 ID 由业务资源 ID 参与构造，保证重试幂等。
 async fn consume_asset_operation_points(
     state: &AppState,
@@ -200,4 +214,31 @@ mod tests {
             &SpacetimeClientError::Procedure("泥点余额不足".to_string()),
         ));
     }
+
+    #[test]
+    fn asset_operation_billing_does_not_refund_stale_worker_lease_errors() {
+        let stale_error = AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+            "provider": "spacetimedb",
+            "message": "external_generation_job lease 已过期",
+        }));
+        let completed_job_error =
+            AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+                "provider": "spacetimedb",
+                "message": "external_generation_job 当前不是 running 状态",
+            }));
+        let missing_job_error =
+            AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+                "provider": "spacetimedb",
+                "message": "external_generation_job 不存在",
+            }));
+        let ordinary_error = AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
+            "provider": "vector-engine",
+            "message": "图片生成失败",
+        }));
+
+        assert!(!should_refund_asset_operation_error(&stale_error));
+        assert!(!should_refund_asset_operation_error(&completed_job_error));
+        assert!(!should_refund_asset_operation_error(&missing_job_error));
+        assert!(should_refund_asset_operation_error(&ordinary_error));
+    }
 }

server-rs/crates/api-server/src/config.rs

@@ -21,6 +21,11 @@ pub struct AppConfig {
     pub bind_port: u16,
     pub listen_backlog: i32,
     pub worker_threads: Option<usize>,
+    pub process_role: ProcessRole,
+    pub external_generation_worker_id: String,
+    pub external_generation_worker_concurrency: usize,
+    pub external_generation_worker_poll_interval: Duration,
+    pub external_generation_worker_lease: Duration,
     pub max_concurrent_requests: Option<usize>,
     pub gallery_max_concurrent_requests: Option<usize>,
     pub detail_max_concurrent_requests: Option<usize>,
@@ -159,6 +164,31 @@ pub struct AppConfig {
     pub slow_request_threshold_ms: u64,
 }
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum ProcessRole {
+    Api,
+    ExternalGenerationWorker,
+    All,
+}
+
+impl ProcessRole {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Api => "api",
+            Self::ExternalGenerationWorker => "external-generation-worker",
+            Self::All => "all",
+        }
+    }
+
+    pub fn runs_http(self) -> bool {
+        matches!(self, Self::Api | Self::All)
+    }
+
+    pub fn runs_external_generation_worker(self) -> bool {
+        matches!(self, Self::ExternalGenerationWorker | Self::All)
+    }
+}
+
 impl Default for AppConfig {
     fn default() -> Self {
         Self {
@@ -166,6 +196,11 @@ impl Default for AppConfig {
             bind_port: 3000,
             listen_backlog: 1024,
             worker_threads: None,
+            process_role: ProcessRole::Api,
+            external_generation_worker_id: default_external_generation_worker_id(),
+            external_generation_worker_concurrency: 2,
+            external_generation_worker_poll_interval: Duration::from_millis(2_000),
+            external_generation_worker_lease: Duration::from_secs(3_600),
             max_concurrent_requests: None,
             gallery_max_concurrent_requests: None,
             detail_max_concurrent_requests: None,
@@ -347,6 +382,30 @@ impl AppConfig {
         if let Some(worker_threads) = read_first_usize_env(&["GENARRATIVE_API_WORKER_THREADS"]) {
             config.worker_threads = Some(worker_threads);
         }
+        if let Some(process_role) = read_first_process_role_env(&["GENARRATIVE_PROCESS_ROLE"]) {
+            config.process_role = process_role;
+        }
+        if let Some(worker_id) =
+            read_first_non_empty_env(&["GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID"])
+        {
+            config.external_generation_worker_id = worker_id;
+        }
+        if let Some(concurrency) =
+            read_first_usize_env(&["GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY"])
+        {
+            config.external_generation_worker_concurrency = concurrency.max(1);
+        }
+        if let Some(poll_interval_ms) = read_first_positive_u64_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS",
+        ]) {
+            config.external_generation_worker_poll_interval =
+                Duration::from_millis(poll_interval_ms);
+        }
+        if let Some(lease_seconds) = read_first_duration_seconds_env(&[
+            "GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS",
+        ]) {
+            config.external_generation_worker_lease = Duration::from_secs(lease_seconds.max(1));
+        }
         if let Some(max_concurrent_requests) =
             read_first_usize_env(&["GENARRATIVE_API_MAX_CONCURRENT_REQUESTS"])
         {
@@ -979,6 +1038,14 @@ fn read_first_llm_provider_env(keys: &[&str]) -> Option<LlmProvider> {
     })
 }
 
+fn read_first_process_role_env(keys: &[&str]) -> Option<ProcessRole> {
+    keys.iter().find_map(|key| {
+        env::var(key)
+            .ok()
+            .and_then(|value| parse_process_role(&value))
+    })
+}
+
 fn read_first_positive_u32_env(keys: &[&str]) -> Option<u32> {
     keys.iter().find_map(|key| {
         env::var(key)
@@ -1026,6 +1093,36 @@ fn read_first_u8_env(keys: &[&str]) -> Option<u8> {
         .find_map(|key| env::var(key).ok().and_then(|value| parse_u8(&value)))
 }
 
+fn default_external_generation_worker_id() -> String {
+    let host = env::var("HOSTNAME")
+        .or_else(|_| env::var("COMPUTERNAME"))
+        .unwrap_or_else(|_| "local".to_string());
+    format!("{}-{}", host.trim(), std::process::id())
+}
+
+fn parse_process_role(value: &str) -> Option<ProcessRole> {
+    match trim_quoted_env_value(value).to_ascii_lowercase().as_str() {
+        "api" => Some(ProcessRole::Api),
+        "external-generation-worker" | "external_generation_worker" | "worker" => {
+            Some(ProcessRole::ExternalGenerationWorker)
+        }
+        "all" => Some(ProcessRole::All),
+        _ => None,
+    }
+}
+
+fn trim_quoted_env_value(raw: &str) -> &str {
+    let raw = raw.trim();
+    raw.strip_prefix('"')
+        .and_then(|value| value.strip_suffix('"'))
+        .or_else(|| {
+            raw.strip_prefix('\'')
+                .and_then(|value| value.strip_suffix('\''))
+        })
+        .unwrap_or(raw)
+        .trim()
+}
+
 fn read_first_positive_u16_env(keys: &[&str]) -> Option<u16> {
     keys.iter().find_map(|key| {
         env::var(key)
@@ -1146,7 +1243,8 @@ fn parse_positive_u16(raw: &str) -> Option<u16> {
 #[cfg(test)]
 mod tests {
     use super::{
-        AppConfig, DEFAULT_VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS, LlmProvider, parse_bool,
+        AppConfig, DEFAULT_VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS, LlmProvider, ProcessRole,
+        parse_bool, parse_process_role,
     };
     use std::sync::{Mutex, OnceLock};
 
@@ -1188,6 +1286,32 @@ mod tests {
         assert_eq!(parse_bool("'off'"), Some(false));
     }
 
+    #[test]
+    fn process_role_controls_http_and_external_generation_worker_roles() {
+        assert_eq!(parse_process_role("api"), Some(ProcessRole::Api));
+        assert_eq!(
+            parse_process_role("\"external-generation-worker\""),
+            Some(ProcessRole::ExternalGenerationWorker)
+        );
+        assert_eq!(
+            parse_process_role("'external_generation_worker'"),
+            Some(ProcessRole::ExternalGenerationWorker)
+        );
+        assert_eq!(
+            parse_process_role("worker"),
+            Some(ProcessRole::ExternalGenerationWorker)
+        );
+        assert_eq!(parse_process_role("all"), Some(ProcessRole::All));
+        assert_eq!(parse_process_role("unknown"), None);
+
+        assert!(ProcessRole::Api.runs_http());
+        assert!(!ProcessRole::Api.runs_external_generation_worker());
+        assert!(!ProcessRole::ExternalGenerationWorker.runs_http());
+        assert!(ProcessRole::ExternalGenerationWorker.runs_external_generation_worker());
+        assert!(ProcessRole::All.runs_http());
+        assert!(ProcessRole::All.runs_external_generation_worker());
+    }
+
     #[test]
     fn from_env_reads_sms_enabled_when_shell_value_keeps_quotes() {
         let _guard = ENV_LOCK

server-rs/crates/api-server/src/main.rs

@@ -40,6 +40,7 @@ mod edutainment_baby_drawing;
 mod edutainment_baby_object;
 mod error_middleware;
 mod external_api_audit;
+mod external_generation_worker;
 pub(crate) mod generated_asset_sheets;
 mod generated_image_assets;
 mod health;
@@ -114,6 +115,7 @@ use tracing::{error, info, warn};
 use crate::{
     app::{build_router, build_spacetime_unavailable_router},
     config::AppConfig,
+    external_generation_worker::run_external_generation_worker,
     state::{AppState, AppStateInitError},
     tracking_outbox::TrackingOutbox,
 };
@@ -164,20 +166,47 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
     process_metrics::register_process_metrics();
     telemetry::register_http_runtime_metrics();
 
+    if !config.process_role.runs_http() {
+        return run_worker_only(config).await;
+    }
+
+    run_http_role(config).await
+}
+
+async fn run_worker_only(config: AppConfig) -> Result<(), io::Error> {
+    let process_role = config.process_role;
+    let state = restore_app_state_for_startup(config)
+        .await
+        .map_err(|error| {
+            io::Error::other(format!(
+                "初始化 external generation worker 状态失败：{error}"
+            ))
+        })?;
+    spawn_app_state_background_workers(&state);
+    info!(
+        process_role = process_role.as_str(),
+        "api-server 以 worker 角色启动"
+    );
+    run_external_generation_worker(state).await
+}
+
+async fn run_http_role(config: AppConfig) -> Result<(), io::Error> {
     let bind_address = config.bind_socket_addr();
     let listen_backlog = config.listen_backlog;
     let worker_threads = config.worker_threads;
     let otel_enabled = config.otel_enabled;
+    let process_role = config.process_role;
     let outbox_flush_timeout = config.shutdown_outbox_flush_timeout;
     let listener = build_tcp_listener(bind_address, listen_backlog)?;
 
-    let (router, shutdown_context) = match restore_app_state_for_startup(config).await {
+    let (router, shutdown_context, worker_state) = match restore_app_state_for_startup(config).await
+    {
         Ok(state) => {
-            state.puzzle_gallery_cache().spawn_cleanup_task();
+            spawn_app_state_background_workers(&state);
             let tracking_outbox = state.tracking_outbox();
-            if let Some(outbox) = tracking_outbox.clone() {
-                outbox.spawn_worker();
-            }
+            let worker_state = process_role
+                .runs_external_generation_worker()
+                .then(|| state.clone());
             (
                 build_router(state.clone()),
                 ShutdownContext {
@@ -185,6 +214,7 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
                     tracking_outbox,
                     outbox_flush_timeout,
                 },
+                worker_state,
             )
         }
         Err(AppStateInitError::DependencyUnavailable(message)) => (
@@ -194,6 +224,7 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
                 tracking_outbox: None,
                 outbox_flush_timeout,
             },
+            None,
         ),
         Err(error) => {
             return Err(std::io::Error::other(format!(
@@ -207,12 +238,20 @@ async fn run_server(config: AppConfig) -> Result<(), io::Error> {
         listen_backlog,
         worker_threads = worker_threads.unwrap_or(0),
         otel_enabled,
+        process_role = process_role.as_str(),
         "api-server 已完成 tracing 初始化并开始监听"
     );
 
-    let result = axum::serve(listener, router)
-        .with_graceful_shutdown(shutdown_signal(shutdown_context.clone()))
-        .await;
+    let http_server = axum::serve(listener, router)
+        .with_graceful_shutdown(shutdown_signal(shutdown_context.clone()));
+    let result = if let Some(worker_state) = worker_state {
+        tokio::select! {
+            result = http_server => result,
+            result = run_external_generation_worker(worker_state) => result,
+        }
+    } else {
+        http_server.await
+    };
     finalize_shutdown(shutdown_context).await;
     result
 }
@@ -304,6 +343,13 @@ async fn finalize_shutdown(context: ShutdownContext) {
     }
 }
 
+fn spawn_app_state_background_workers(state: &AppState) {
+    state.puzzle_gallery_cache().spawn_cleanup_task();
+    if let Some(outbox) = state.tracking_outbox() {
+        outbox.spawn_worker();
+    }
+}
+
 fn build_tcp_listener(
     bind_address: SocketAddr,
     listen_backlog: i32,

server-rs/crates/api-server/src/puzzle.rs

@@ -52,21 +52,21 @@ use shared_contracts::{
 };
 use shared_kernel::{build_prefixed_uuid_id, format_timestamp_micros};
 use spacetime_client::{
-    PuzzleAgentMessageRecord, PuzzleAgentMessageSubmitRecordInput,
-    PuzzleAgentSessionCreateRecordInput, PuzzleAgentSessionRecord,
-    PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord, PuzzleAnchorPackRecord,
-    PuzzleAudioAssetRecord, PuzzleCreatorIntentRecord, PuzzleDraftCompileFailureRecordInput,
-    PuzzleDraftLevelRecord, PuzzleFormDraftRecord, PuzzleFormDraftSaveRecordInput,
-    PuzzleGalleryCardRecord, PuzzleGeneratedImageCandidateRecord,
+    ExternalGenerationJobEnqueueRecordInput, PuzzleAgentMessageRecord,
+    PuzzleAgentMessageSubmitRecordInput, PuzzleAgentSessionCreateRecordInput,
+    PuzzleAgentSessionRecord, PuzzleAgentSuggestedActionRecord, PuzzleAnchorItemRecord,
+    PuzzleAnchorPackRecord, PuzzleAudioAssetRecord, PuzzleCreatorIntentRecord,
+    PuzzleDraftCompileFailureRecordInput, PuzzleDraftLevelRecord, PuzzleFormDraftRecord,
+    PuzzleFormDraftSaveRecordInput, PuzzleGalleryCardRecord, PuzzleGeneratedImageCandidateRecord,
     PuzzleGeneratedImagesSaveRecordInput, PuzzleLeaderboardEntryRecord,
-    PuzzleLeaderboardSubmitRecordInput, PuzzlePublishRecordInput, PuzzleRecommendedNextWorkRecord,
-    PuzzleResultDraftRecord, PuzzleResultPreviewBlockerRecord, PuzzleResultPreviewFindingRecord,
-    PuzzleResultPreviewRecord, PuzzleRunDragRecordInput, PuzzleRunPauseRecordInput,
-    PuzzleRunPropRecordInput, PuzzleRunRecord, PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput,
-    PuzzleSelectCoverImageRecordInput, PuzzleUiBackgroundSaveRecordInput,
-    PuzzleWorkLikeReportRecordInput, PuzzleWorkPointIncentiveClaimRecordInput,
-    PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput, PuzzleWorkUpsertRecordInput,
-    SpacetimeClientError,
+    PuzzleLeaderboardSubmitRecordInput, PuzzleLevelGenerationFailureRecordInput,
+    PuzzlePublishRecordInput, PuzzleRecommendedNextWorkRecord, PuzzleResultDraftRecord,
+    PuzzleResultPreviewBlockerRecord, PuzzleResultPreviewFindingRecord, PuzzleResultPreviewRecord,
+    PuzzleRunDragRecordInput, PuzzleRunPauseRecordInput, PuzzleRunPropRecordInput, PuzzleRunRecord,
+    PuzzleRunStartRecordInput, PuzzleRunSwapRecordInput, PuzzleSelectCoverImageRecordInput,
+    PuzzleUiBackgroundSaveRecordInput, PuzzleWorkLikeReportRecordInput,
+    PuzzleWorkPointIncentiveClaimRecordInput, PuzzleWorkProfileRecord, PuzzleWorkRemixRecordInput,
+    PuzzleWorkUpsertRecordInput, SpacetimeClientError,
 };
 use std::convert::Infallible;
 
@@ -78,6 +78,10 @@ use crate::{
         should_skip_asset_operation_billing_for_connectivity,
     },
     auth::{AuthenticatedAccessToken, RuntimePrincipal},
+    external_generation_worker::{
+        PUZZLE_COMPILE_DRAFT_JOB_KIND, PUZZLE_GENERATE_IMAGES_JOB_KIND,
+        PUZZLE_GENERATE_UI_BACKGROUND_JOB_KIND,
+    },
     generated_asset_sheets::apply_generated_asset_sheet_green_screen_alpha,
     http_error::AppError,
     llm_model_routing::{CREATION_TEMPLATE_LLM_MODEL, PUZZLE_LEVEL_NAME_VISION_LLM_MODEL},
@@ -130,6 +134,43 @@ const PUZZLE_UI_BACKGROUND_PROMPT_FALLBACK_MARKER: &str =
 const PUZZLE_VECTOR_ENGINE_SQUARE_IMAGE_SIZE: &str = "1024x1024";
 const PUZZLE_VECTOR_ENGINE_PORTRAIT_IMAGE_SIZE: &str = "1024x1536";
 
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub(crate) struct ExternalGenerationWriteLeaseGuard {
+    pub(crate) job_id: String,
+    pub(crate) worker_id: String,
+    pub(crate) lease_token: String,
+}
+
+#[derive(Debug)]
+pub(crate) struct PuzzleExternalGenerationWorkerError {
+    error: AppError,
+    should_fail_queue_job: bool,
+}
+
+impl PuzzleExternalGenerationWorkerError {
+    pub(crate) fn with_failure_state_written(error: AppError) -> Self {
+        Self {
+            error,
+            should_fail_queue_job: true,
+        }
+    }
+
+    pub(crate) fn with_failure_state_pending(error: AppError) -> Self {
+        Self {
+            error,
+            should_fail_queue_job: false,
+        }
+    }
+
+    pub(crate) fn body_text(&self) -> String {
+        self.error.body_text()
+    }
+
+    pub(crate) fn should_fail_queue_job(&self) -> bool {
+        self.should_fail_queue_job
+    }
+}
+
 pub(crate) fn format_puzzle_reference_image_upload_bytes(bytes: usize) -> String {
     format!("{:.1}MB", bytes as f64 / 1024.0 / 1024.0)
 }
@@ -152,7 +193,7 @@ mod mappers;
 use self::mappers::*;
 
 mod draft;
-use self::draft::*;
+pub(crate) use self::draft::*;
 
 mod tags;
 
@@ -161,7 +202,7 @@ use self::tags::*;
 mod generation;
 mod vector_engine;
 
-use self::generation::*;
+pub(crate) use self::generation::*;
 use self::vector_engine::*;
 
 #[cfg(test)]

server-rs/crates/api-server/src/puzzle/draft.rs

@@ -137,6 +137,124 @@ pub(crate) async fn create_seeded_puzzle_session_when_form_save_missing(
     Ok(replacement.session_id)
 }
 
+#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub(crate) struct PuzzleCompileDraftWorkerPayload {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub billing_asset_id: String,
+    pub ai_redraw: bool,
+    #[serde(default)]
+    pub prompt_text: Option<String>,
+    #[serde(default)]
+    pub reference_image_src: Option<String>,
+    #[serde(default)]
+    pub image_model: Option<String>,
+    pub requested_at_micros: i64,
+}
+
+pub(crate) async fn execute_puzzle_compile_draft_worker_job(
+    state: &PuzzleApiState,
+    request_context: &RequestContext,
+    payload: PuzzleCompileDraftWorkerPayload,
+    external_generation_guard: ExternalGenerationWriteLeaseGuard,
+) -> Result<PuzzleAgentSessionRecord, PuzzleExternalGenerationWorkerError> {
+    let now = current_utc_micros();
+    let session = if payload.ai_redraw {
+        execute_billable_asset_operation_with_cost(
+            state.root_state(),
+            &payload.owner_user_id,
+            "puzzle_initial_image",
+            &payload.billing_asset_id,
+            PUZZLE_IMAGE_GENERATION_POINTS_COST,
+            async {
+                compile_puzzle_draft_with_initial_cover(
+                    state,
+                    request_context,
+                    payload.session_id.clone(),
+                    payload.owner_user_id.clone(),
+                    payload.prompt_text.as_deref(),
+                    payload.reference_image_src.as_deref(),
+                    payload.image_model.as_deref(),
+                    now,
+                    &external_generation_guard,
+                )
+                .await
+            },
+        )
+        .await
+    } else {
+        compile_puzzle_draft_with_uploaded_cover(
+            state,
+            request_context,
+            payload.session_id.clone(),
+            payload.owner_user_id.clone(),
+            payload.prompt_text.as_deref(),
+            payload.reference_image_src.as_deref(),
+            now,
+            &external_generation_guard,
+        )
+        .await
+    };
+
+    match session {
+        Ok(session) => Ok(session),
+        Err(error) => {
+            match mark_puzzle_compile_failure_for_worker(
+                state,
+                &payload.session_id,
+                &payload.owner_user_id,
+                error.body_text(),
+                now,
+                &external_generation_guard,
+            )
+            .await
+            {
+                Ok(()) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_written(error))
+                }
+                Err(mark_error) => {
+                    Err(PuzzleExternalGenerationWorkerError::with_failure_state_pending(mark_error))
+                }
+            }
+        }
+    }
+}
+
+pub(crate) async fn mark_puzzle_compile_failure_for_worker(
+    state: &PuzzleApiState,
+    session_id: &str,
+    owner_user_id: &str,
+    error_message: String,
+    failed_at_micros: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
+) -> Result<(), AppError> {
+    let result = state
+        .spacetime_client()
+        .mark_puzzle_draft_generation_failed(PuzzleDraftCompileFailureRecordInput {
+            session_id: session_id.to_string(),
+            owner_user_id: owner_user_id.to_string(),
+            error_message,
+            failed_at_micros,
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
+        })
+        .await;
+    if let Err(error) = result {
+        tracing::warn!(
+            provider = PUZZLE_AGENT_API_BASE_PROVIDER,
+            session_id,
+            owner_user_id,
+            message = %error,
+            "拼图 worker 草稿失败态回写失败"
+        );
+        return Err(map_puzzle_client_error(error));
+    }
+
+    Ok(())
+}
+
 pub(crate) fn select_puzzle_level_for_api(
     draft: &PuzzleResultDraftRecord,
     level_id: Option<&str>,
@@ -1186,10 +1304,18 @@ pub(crate) async fn compile_puzzle_draft_with_initial_cover(
     reference_image_src: Option<&str>,
     image_model: Option<&str>,
     now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
 ) -> Result<PuzzleAgentSessionRecord, AppError> {
     let compiled_session = state
         .spacetime_client()
-        .compile_puzzle_agent_draft(session_id.clone(), owner_user_id.clone(), now)
+        .compile_puzzle_agent_draft_with_external_generation_guard(
+            session_id.clone(),
+            owner_user_id.clone(),
+            now,
+            external_generation_guard.job_id.clone(),
+            external_generation_guard.worker_id.clone(),
+            external_generation_guard.lease_token.clone(),
+        )
         .await
         .map_err(map_puzzle_compile_error)?;
     let draft = compiled_session.draft.clone().ok_or_else(|| {
@@ -1332,7 +1458,7 @@ pub(crate) async fn compile_puzzle_draft_with_initial_cover(
             "message": format!("拼图候选图序列化失败：{error}"),
         }))
     })?;
-    let (saved_session, save_used_fallback) = state
+    let saved_session = state
         .spacetime_client()
         .save_puzzle_generated_images(PuzzleGeneratedImagesSaveRecordInput {
             session_id: compiled_session.session_id.clone(),
@@ -1341,42 +1467,12 @@ pub(crate) async fn compile_puzzle_draft_with_initial_cover(
             levels_json: levels_json_with_generated_name.clone(),
             candidates_json,
             saved_at_micros: current_utc_micros(),
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
         })
         .await
-        .map_err(map_puzzle_client_error)
-        .map(|session| (session, false))
-        .or_else(|error| {
-            if is_spacetimedb_connectivity_app_error(&error) {
-                // 中文注释：首图已落 OSS 时，SpacetimeDB 短暂不可用先返回本地快照，避免整次 VectorEngine 生图被判失败。
-                tracing::warn!(
-                    provider = PUZZLE_AGENT_API_BASE_PROVIDER,
-                    session_id = %compiled_session.session_id,
-                    owner_user_id = %owner_user_id,
-                    message = %error.body_text(),
-                    "拼图首图已生成但 SpacetimeDB 草稿回写不可用，降级返回本次生成快照"
-                );
-                let session = apply_generated_puzzle_candidates_to_session_snapshot(
-                    apply_generated_puzzle_levels_to_session_snapshot(
-                        apply_generated_puzzle_first_level_name_to_session_snapshot(
-                            compiled_session.clone(),
-                            target_level.level_id.as_str(),
-                            generated_level_name.as_str(),
-                            fallback_level_name.as_str(),
-                            now,
-                        ),
-                        updated_levels.clone(),
-                        now,
-                    ),
-                    target_level.level_id.as_str(),
-                    candidates.into_records(),
-                    reference_image_src,
-                    now,
-                );
-                Ok((session, true))
-            } else {
-                Err(error)
-            }
-        })?;
+        .map_err(map_puzzle_client_error)?;
     match state
         .spacetime_client()
         .update_puzzle_work(PuzzleWorkUpsertRecordInput {
@@ -1413,9 +1509,6 @@ pub(crate) async fn compile_puzzle_draft_with_initial_cover(
         fallback_level_name.as_str(),
         now,
     );
-    if save_used_fallback {
-        return Ok(saved_session);
-    }
     match state
         .spacetime_client()
         .select_puzzle_cover_image(PuzzleSelectCoverImageRecordInput {
@@ -1454,6 +1547,7 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
     prompt_text: Option<&str>,
     reference_image_src: Option<&str>,
     now: i64,
+    external_generation_guard: &ExternalGenerationWriteLeaseGuard,
 ) -> Result<PuzzleAgentSessionRecord, AppError> {
     let uploaded_image_src = reference_image_src
         .map(str::trim)
@@ -1488,7 +1582,14 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
             })?;
     let compiled_session = state
         .spacetime_client()
-        .compile_puzzle_agent_draft(session_id.clone(), owner_user_id.clone(), now)
+        .compile_puzzle_agent_draft_with_external_generation_guard(
+            session_id.clone(),
+            owner_user_id.clone(),
+            now,
+            external_generation_guard.job_id.clone(),
+            external_generation_guard.worker_id.clone(),
+            external_generation_guard.lease_token.clone(),
+        )
         .await
         .map_err(map_puzzle_compile_error)?;
     let draft = compiled_session.draft.clone().ok_or_else(|| {
@@ -1628,7 +1729,7 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
             "message": format!("拼图上传图候选序列化失败：{error}"),
         }))
     })?;
-    let (saved_session, save_used_fallback) = state
+    let saved_session = state
         .spacetime_client()
         .save_puzzle_generated_images(PuzzleGeneratedImagesSaveRecordInput {
             session_id: compiled_session.session_id.clone(),
@@ -1637,41 +1738,12 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
             levels_json: levels_json_with_generated_name.clone(),
             candidates_json,
             saved_at_micros: current_utc_micros(),
+            external_generation_job_id: external_generation_guard.job_id.clone(),
+            external_generation_worker_id: external_generation_guard.worker_id.clone(),
+            external_generation_lease_token: external_generation_guard.lease_token.clone(),
         })
         .await
-        .map_err(map_puzzle_client_error)
-        .map(|session| (session, false))
-        .or_else(|error| {
-            if is_spacetimedb_connectivity_app_error(&error) {
-                tracing::warn!(
-                    provider = PUZZLE_AGENT_API_BASE_PROVIDER,
-                    session_id = %compiled_session.session_id,
-                    owner_user_id = %owner_user_id,
-                    message = %error.body_text(),
-                    "拼图上传图草稿回写不可用，降级返回本地快照"
-                );
-                let session = apply_generated_puzzle_candidates_to_session_snapshot(
-                    apply_generated_puzzle_levels_to_session_snapshot(
-                        apply_generated_puzzle_first_level_name_to_session_snapshot(
-                            compiled_session.clone(),
-                            target_level.level_id.as_str(),
-                            generated_level_name.as_str(),
-                            fallback_level_name.as_str(),
-                            now,
-                        ),
-                        updated_levels.clone(),
-                        now,
-                    ),
-                    target_level.level_id.as_str(),
-                    vec![candidate.clone()],
-                    reference_image_src,
-                    now,
-                );
-                Ok((session, true))
-            } else {
-                Err(error)
-            }
-        })?;
+        .map_err(map_puzzle_client_error)?;
     let (_, profile_id) = build_stable_puzzle_work_ids(&compiled_session.session_id);
     match state
         .spacetime_client()
@@ -1709,9 +1781,6 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
         fallback_level_name.as_str(),
         now,
     );
-    if save_used_fallback {
-        return Ok(saved_session);
-    }
     match state
         .spacetime_client()
         .select_puzzle_cover_image(PuzzleSelectCoverImageRecordInput {
@@ -1742,6 +1811,7 @@ pub(crate) async fn compile_puzzle_draft_with_uploaded_cover(
     }
 }
 
+#[cfg(test)]
 pub(crate) fn apply_generated_puzzle_candidates_to_session_snapshot(
     mut session: PuzzleAgentSessionRecord,
     target_level_id: &str,
@@ -1794,23 +1864,7 @@ pub(crate) fn apply_generated_puzzle_candidates_to_session_snapshot(
     session
 }
 
-pub(crate) fn apply_generated_puzzle_levels_to_session_snapshot(
-    mut session: PuzzleAgentSessionRecord,
-    levels: Vec<PuzzleDraftLevelRecord>,
-    updated_at_micros: i64,
-) -> PuzzleAgentSessionRecord {
-    let Some(draft) = session.draft.as_mut() else {
-        return session;
-    };
-    if levels.is_empty() {
-        return session;
-    }
-    draft.levels = levels;
-    sync_puzzle_primary_draft_fields_from_level(draft);
-    session.updated_at = format_timestamp_micros(updated_at_micros);
-    session
-}
-
+#[cfg(test)]
 pub(crate) fn apply_generated_puzzle_first_level_name_to_session_snapshot(
     mut session: PuzzleAgentSessionRecord,
     target_level_id: &str,
@@ -1863,45 +1917,6 @@ pub(crate) fn apply_generated_puzzle_initial_metadata_to_session_snapshot(
     session
 }
 
-pub(crate) fn apply_generated_puzzle_metadata_to_session_snapshot(
-    mut session: PuzzleAgentSessionRecord,
-    target_level_id: &str,
-    metadata: &PuzzleLevelNaming,
-    previous_level_name: &str,
-    updated_at_micros: i64,
-) -> PuzzleAgentSessionRecord {
-    let Some(draft) = session.draft.as_mut() else {
-        return session;
-    };
-    let Some(target_index) = draft
-        .levels
-        .iter()
-        .position(|level| level.level_id == target_level_id)
-        .or_else(|| (!draft.levels.is_empty()).then_some(0))
-    else {
-        return session;
-    };
-
-    draft.levels[target_index].level_name = metadata.level_name.clone();
-    if metadata.ui_background_prompt.is_some() {
-        draft.levels[target_index].ui_background_prompt = metadata.ui_background_prompt.clone();
-    }
-
-    if target_index == 0 {
-        apply_generated_puzzle_initial_metadata_to_draft(
-            draft,
-            metadata,
-            previous_level_name,
-            updated_at_micros,
-        );
-    } else {
-        sync_puzzle_primary_draft_fields_from_level(draft);
-    }
-
-    session.updated_at = format_timestamp_micros(updated_at_micros);
-    session
-}
-
 pub(crate) fn apply_generated_puzzle_initial_metadata_to_draft(
     draft: &mut PuzzleResultDraftRecord,
     metadata: &PuzzleLevelNaming,
@@ -1951,45 +1966,3 @@ pub(crate) fn sync_puzzle_primary_draft_fields_from_level(draft: &mut PuzzleResu
         });
     }
 }
-
-pub(crate) fn replace_puzzle_session_draft_snapshot(
-    mut session: PuzzleAgentSessionRecord,
-    draft: PuzzleResultDraftRecord,
-    updated_at_micros: i64,
-) -> PuzzleAgentSessionRecord {
-    session.draft = Some(draft);
-    session.updated_at = format_timestamp_micros(updated_at_micros);
-    session
-}
-
-pub(crate) fn apply_generated_puzzle_ui_background_to_session_snapshot(
-    mut session: PuzzleAgentSessionRecord,
-    target_level_id: &str,
-    prompt: String,
-    image_src: String,
-    image_object_key: Option<String>,
-    updated_at_micros: i64,
-) -> PuzzleAgentSessionRecord {
-    let Some(draft) = session.draft.as_mut() else {
-        return session;
-    };
-    let Some(target_index) = draft
-        .levels
-        .iter()
-        .position(|level| level.level_id == target_level_id)
-        .or_else(|| (!draft.levels.is_empty()).then_some(0))
-    else {
-        return session;
-    };
-    let level = &mut draft.levels[target_index];
-    level.ui_background_prompt = Some(prompt);
-    level.ui_background_image_src = Some(image_src);
-    level.ui_background_image_object_key = image_object_key;
-    if target_index == 0 {
-        sync_puzzle_primary_draft_fields_from_level(draft);
-    }
-    session.progress_percent = session.progress_percent.max(96);
-    session.last_assistant_reply = Some("拼图 UI 背景图已生成。".to_string());
-    session.updated_at = format_timestamp_micros(updated_at_micros);
-    session
-}

server-rs/crates/api-server/src/puzzle/tests.rs

@@ -484,6 +484,108 @@ fn puzzle_image_generation_builds_fallback_session_from_levels_snapshot() {
     );
 }
 
+#[test]
+fn puzzle_generate_images_worker_payload_keeps_action_snapshot() {
+    let raw_levels_json = serde_json::to_string(&vec![json!({
+        "levelId": "puzzle-level-2",
+        "levelName": "",
+        "pictureDescription": "新关卡里有一座发光钟楼。",
+        "candidates": [],
+        "selectedCandidateId": null,
+        "coverImageSrc": null,
+        "coverAssetId": null,
+        "generationStatus": "generating",
+    })])
+    .expect("levels json");
+    let levels_json = normalize_puzzle_levels_json_for_module(Some(raw_levels_json.as_str()))
+        .expect("levels should normalize")
+        .expect("levels json should exist");
+    let payload = PuzzleGenerateImagesWorkerPayload {
+        session_id: "puzzle-session-1".to_string(),
+        owner_user_id: "user-1".to_string(),
+        billing_asset_id: "puzzle-session-1:123".to_string(),
+        level_id: Some("puzzle-level-2".to_string()),
+        prompt_text: Some("发光钟楼".to_string()),
+        reference_image_src: None,
+        reference_image_srcs: vec!["data:image/png;base64,abc".to_string()],
+        reference_image_asset_object_id: Some("asset-object-1".to_string()),
+        reference_image_asset_object_ids: vec!["asset-object-2".to_string()],
+        image_model: Some(PUZZLE_IMAGE_MODEL_GPT_IMAGE_2.to_string()),
+        ai_redraw: Some(true),
+        should_auto_name_level: Some(true),
+        work_title: Some("暖灯猫街作品".to_string()),
+        work_description: Some("一套雨夜猫街主题拼图。".to_string()),
+        picture_description: None,
+        summary: Some("一套雨夜猫街主题拼图。".to_string()),
+        theme_tags: Some(vec!["猫咪".to_string(), "雨夜".to_string()]),
+        levels_json: Some(levels_json.clone()),
+        requested_at_micros: 123,
+    };
+
+    let encoded = serde_json::to_string(&payload).expect("payload should serialize");
+    let decoded: PuzzleGenerateImagesWorkerPayload =
+        serde_json::from_str(encoded.as_str()).expect("payload should deserialize");
+
+    assert_eq!(decoded.level_id.as_deref(), Some("puzzle-level-2"));
+    assert_eq!(decoded.reference_image_srcs.len(), 1);
+    assert_eq!(
+        decoded.reference_image_asset_object_ids,
+        vec!["asset-object-2".to_string()]
+    );
+    assert_eq!(decoded.should_auto_name_level, Some(true));
+    let records = parse_puzzle_level_records_from_module_json(
+        decoded.levels_json.as_deref().expect("levels json"),
+    )
+    .expect("levels should parse as module json");
+    assert_eq!(records[0].level_id, "puzzle-level-2");
+    assert_eq!(records[0].generation_status, "generating");
+}
+
+#[test]
+fn puzzle_generate_ui_background_worker_payload_keeps_action_snapshot() {
+    let raw_levels_json = serde_json::to_string(&vec![json!({
+        "levelId": "puzzle-level-3",
+        "levelName": "钟楼回廊",
+        "pictureDescription": "新关卡里有一座发光钟楼。",
+        "uiBackgroundPrompt": "发光钟楼延展成竖屏回廊，远处有暖色窗光。",
+        "candidates": [],
+        "selectedCandidateId": null,
+        "coverImageSrc": null,
+        "coverAssetId": null,
+        "generationStatus": "generating",
+    })])
+    .expect("levels json");
+    let levels_json = normalize_puzzle_levels_json_for_module(Some(raw_levels_json.as_str()))
+        .expect("levels should normalize")
+        .expect("levels json should exist");
+    let payload = PuzzleGenerateUiBackgroundWorkerPayload {
+        session_id: "puzzle-session-1".to_string(),
+        owner_user_id: "user-1".to_string(),
+        billing_asset_id: "puzzle-session-1:456".to_string(),
+        level_id: Some("puzzle-level-3".to_string()),
+        prompt_text: Some("发光钟楼延展成竖屏回廊".to_string()),
+        levels_json: Some(levels_json.clone()),
+        requested_at_micros: 456,
+    };
+
+    let encoded = serde_json::to_string(&payload).expect("payload should serialize");
+    let decoded: PuzzleGenerateUiBackgroundWorkerPayload =
+        serde_json::from_str(encoded.as_str()).expect("payload should deserialize");
+
+    assert_eq!(decoded.level_id.as_deref(), Some("puzzle-level-3"));
+    assert_eq!(
+        decoded.prompt_text.as_deref(),
+        Some("发光钟楼延展成竖屏回廊")
+    );
+    assert_eq!(decoded.requested_at_micros, 456);
+    let records = parse_puzzle_level_records_from_module_json(
+        decoded.levels_json.as_deref().expect("levels json"),
+    )
+    .expect("levels should parse as module json");
+    assert_eq!(records[0].level_id, "puzzle-level-3");
+    assert_eq!(records[0].generation_status, "generating");
+}
+
 #[test]
 fn puzzle_first_level_name_parser_accepts_json_and_normalizes_text() {
     assert_eq!(

server-rs/crates/module-puzzle/src/commands.rs

@@ -66,6 +66,9 @@ pub struct PuzzleDraftCompileInput {
     pub session_id: String,
     pub owner_user_id: String,
     pub compiled_at_micros: i64,
+    pub external_generation_job_id: Option<String>,
+    pub external_generation_worker_id: Option<String>,
+    pub external_generation_lease_token: Option<String>,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
@@ -75,6 +78,23 @@ pub struct PuzzleDraftCompileFailureInput {
     pub owner_user_id: String,
     pub error_message: String,
     pub failed_at_micros: i64,
+    pub external_generation_job_id: String,
+    pub external_generation_worker_id: String,
+    pub external_generation_lease_token: String,
+}
+
+#[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PuzzleLevelGenerationFailureInput {
+    pub session_id: String,
+    pub owner_user_id: String,
+    pub level_id: Option<String>,
+    pub levels_json: Option<String>,
+    pub error_message: String,
+    pub failed_at_micros: i64,
+    pub external_generation_job_id: String,
+    pub external_generation_worker_id: String,
+    pub external_generation_lease_token: String,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
@@ -86,6 +106,9 @@ pub struct PuzzleGeneratedImagesSaveInput {
     pub levels_json: Option<String>,
     pub candidates_json: String,
     pub saved_at_micros: i64,
+    pub external_generation_job_id: String,
+    pub external_generation_worker_id: String,
+    pub external_generation_lease_token: String,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]
@@ -99,6 +122,9 @@ pub struct PuzzleUiBackgroundSaveInput {
     pub image_src: String,
     pub image_object_key: Option<String>,
     pub saved_at_micros: i64,
+    pub external_generation_job_id: String,
+    pub external_generation_worker_id: String,
+    pub external_generation_lease_token: String,
 }
 
 #[cfg_attr(feature = "spacetime-types", derive(SpacetimeType))]