升级外部生成为正式任务列表

将 external_generation_job 扩展为含价格、确认时间、退款流水和事件审计的正式生成任务事实源

新增外部生成任务列表与确认 BFF,并让平台入口按后端任务补弹完成或失败提示

保护画布生成队列超时、快照回写和自动保存覆盖场景

将 worker 计费退款流水写入 externalGenerationJobId,并补充冷备份后 worker/controller 恢复守护
This commit is contained in:
2026-06-24 15:17:12 +08:00
parent ed6b17442a
commit 73a17193d4
66 changed files with 2416 additions and 122 deletions
@@ -9,9 +9,10 @@ User=root
Group=root
WorkingDirectory=/opt/genarrative/current
EnvironmentFile=/etc/genarrative/api-server.env
ExecStart=/usr/bin/node -- /opt/genarrative/current/scripts/database-backup-to-oss.mjs --env-file /etc/genarrative/api-server.env --stop-service spacetimedb.service --restart-service-after genarrative-api.service
ExecStart=/usr/bin/node -- /opt/genarrative/current/scripts/database-backup-to-oss.mjs --env-file /etc/genarrative/api-server.env --stop-service spacetimedb.service --restart-service-after genarrative-api.service --restart-service-after genarrative-external-generation-worker@1.service --restart-service-after genarrative-external-generation-controller.service
# 备份需要停止 / 启动 spacetimedb.service,并读取 /stdb、写入 /var/lib/genarrative/database-backups。
# 停止 SpacetimeDB 会连带停止 Requires 它的 API / worker / controller,冷备份后必须显式拉起。
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/stdb /var/lib/genarrative
@@ -16,6 +16,14 @@
---
## 2026-06-24 外部生成队列升级为正式生成任务列表
- 背景:外部生成队列已经承载画板和玩法的付费生成,但前端只展示排队概览,缺少可追溯任务列表、后端确认状态、完成提示补弹和退款记录到任务的追踪关系。
- 决策:`external_generation_job` 同时作为正式生成任务列表事实源,保存 `price_mud_points``refund_ledger_id``notification_acknowledged_at`;新增 `external_generation_job_event` 追加状态转换审计。BFF 新增当前账号任务列表和 acknowledge 接口;前端只展示后端任务状态,完成 / 失败提示关闭时由后端写确认时间,未确认终态任务在下次登录后按列表集中弹出。任务触发的钱包扣费 / 退款流水 metadata 必须写 `externalGenerationJobId`,本机退款 outbox 重放也保留该任务 ID。
- 影响范围:`spacetime-module` 外部生成 schema / procedure、`spacetime-client` bindings/facade、`api-server` 外部生成 BFF、worker 失败回写和资产计费退款链路、平台入口“我的”页任务卡和完成提示弹窗。
- 验证方式:运行 `npm run spacetime:generate``npm run check:spacetime-schema``cargo test -p spacetime-module external_generation --manifest-path server-rs/Cargo.toml``cargo test -p api-server wallet_refund_outbox --manifest-path server-rs/Cargo.toml``cargo check -p api-server --manifest-path server-rs/Cargo.toml``npm run typecheck``npm run check:encoding``git diff --check`
- 关联文档:`docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md``docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md`
## 2026-06-23 编辑器宣发素材固定 gpt-image-2
- 背景:画板宣发素材的游戏首图、详情五图和运营海报只应使用稳定的宣发图生成链路,不能被图片模型上次选择或旧请求切到 `nanobanana2`
+21 -5
View File
@@ -23,6 +23,14 @@
- 验证:`ImageCanvasEditorModel.test.ts` 覆盖素材库 source resource 保留,`useImageCanvasAssetCanvasBridge.test.tsx` 覆盖资源 ID 级联清理,`ImageCanvasEditorAssetsIntegration.test.tsx` 覆盖删除后保存的新 layout 不再包含被删图层。
- 关联:`src/components/image-editor/ImageCanvasEditorModel.ts``src/components/image-editor/useImageCanvasAssetCanvasBridge.ts``src/components/image-editor/ImageCanvasEditorAssetsIntegration.test.tsx`
## 画板外部生成排队超时不是失败
- 现象:画板发起付费图片生成后,前端弹出 `生成任务仍在队列中,请稍后刷新画布查看结果`,但后端任务仍在队列或执行中,后续可能正常完成。
- 原因:画板生成已经接入后端外部生成任务队列,`queued` / `running` 是正式任务状态;旧前端轮询等待窗口到期时直接抛错,导致正常排队被提交流程 catch 成失败 UI。
- 处理:`waitForEditorGenerationQueue` 等待超时只返回“仍在后端继续执行”,调用方停止本次前端等待并保留生成中状态;只有后端任务终态为 `failed` 才展示失败。
- 验证:画板生成 workflow 测试覆盖 queueState 持续 `running` 到前端等待窗口结束时,不进入 failed、不显示该排队文案、不添加本地临时结果层。
- 关联:`src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts``src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx`
## 编辑器生成按钮显示泥点后仍要查真实钱包预扣
- 现象:画板生成按钮显示 `N泥点`,后端也能按模型配置计算出价格,但用户点击后钱包余额不变。
@@ -319,12 +327,12 @@
- 验证:`cargo check -p api-server --manifest-path server-rs/Cargo.toml``cargo test -p api-server asset_operation_billing_does_not_refund_stale_worker_lease_errors --manifest-path server-rs/Cargo.toml`,并在 smoke 时确认 queued 任务被 worker 消费后 session 真实更新。
- 关联:`server-rs/crates/api-server/src/puzzle/draft.rs``server-rs/crates/api-server/src/puzzle/generation.rs``server-rs/crates/api-server/src/external_generation_worker.rs``docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md`
## 生产冷备份后 API 不能只依赖 SpacetimeDB 自恢复
## 生产冷备份后 API 和外部生成 worker 不能只依赖 SpacetimeDB 自恢复
- 现象:release 机器 `03:20` 冷备份后,`spacetimedb.service` 已恢复,但作品列表、创作入口配置或公开 gallery 继续超时 / 502 / 504`genarrative-api.service` 保持 stopped。
- 原因:`genarrative-api.service` 配置了 `Requires=spacetimedb.service`,冷备份停止 `spacetimedb.service` API 会被 systemd 依赖关系一并停止;如果 `genarrative-database-backup.service``--stop-service spacetimedb.service` 而漏掉 `--restart-service-after genarrative-api.service`,备份脚本只会恢复数据库,不会再拉起 API
- 处理:生产冷备份 unit 和发布脚本必须带 `--restart-service-after genarrative-api.service`;仓库用 `npm run check:production-ops` 检查 systemd 模板、API build/deploy 归档和健康巡检链路。现场修复后执行 `systemctl daemon-reload`,但不要为了验证而手动触发冷备份。
- 验证:`systemctl cat genarrative-database-backup.service` 应包含参数;`systemctl is-active spacetimedb.service genarrative-api.service nginx.service` 全为 `active``curl -fsS http://127.0.0.1:3101/v1/ping``/healthz``/readyz` 和代表性 `/api/runtime/puzzle/gallery` 均成功。
- 现象:release 机器 `03:20` 冷备份后,`spacetimedb.service` 已恢复,但作品列表、创作入口配置或公开 gallery 继续超时 / 502 / 504`genarrative-api.service` 保持 stopped;或图片画布生成请求返回队列态后长期显示排队,`external_generation_job` 有 claimable pending,但 `genarrative-external-generation-worker@1.service` / controller 是 inactive
- 原因:`genarrative-api.service``genarrative-external-generation-worker@*.service``genarrative-external-generation-controller.service`配置了 `Requires=spacetimedb.service`,冷备份停止 `spacetimedb.service`这些服务会被 systemd 依赖关系一并停止;如果 `genarrative-database-backup.service`恢复数据库或只重启 API,外部生成队列就不会被消费
- 处理:生产冷备份 unit 和发布脚本必须带 `--restart-service-after genarrative-api.service``--restart-service-after genarrative-external-generation-worker@1.service``--restart-service-after genarrative-external-generation-controller.service`;仓库用 `npm run check:production-ops` 检查 systemd 模板、API build/deploy 归档和健康巡检链路。现场修复后执行 `systemctl daemon-reload`,但不要为了验证而手动触发冷备份。
- 验证:`systemctl cat genarrative-database-backup.service` 应包含这些参数;`systemctl is-active spacetimedb.service genarrative-api.service genarrative-external-generation-worker@1.service genarrative-external-generation-controller.service nginx.service` 全为 `active``curl -fsS http://127.0.0.1:3101/v1/ping``/healthz``/readyz` 和代表性 `/api/runtime/puzzle/gallery` 均成功`get_external_generation_queue_stats_and_return` 不应长期出现 claimable pending
- 关联:`deploy/systemd/genarrative-database-backup.service``scripts/database-backup-to-oss.mjs``scripts/ops/production-health-patrol.mjs``docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`
## Pingora Brotli 不能只看 Content-Encoding
@@ -2619,6 +2627,14 @@
- 验证:模拟首张 image2 超长耗时或超时重试时,生成页应显示真实阶段和可恢复状态;前端请求超时不应把最终成功草稿标记为失败;刷新 `/creation/jump-hop/generating?sessionId=<id>` 后应能恢复到后端真实状态;同一 session 重试不得重复生成已完成阶段。
- 关联:`src/services/jump-hop/jumpHopClient.ts``src/services/miniGameDraftGenerationProgress.ts``server-rs/crates/api-server/src/jump_hop.rs``server-rs/crates/platform-image/src/vector_engine/client.rs``docs/【玩法创作】平台入口与玩法链路-2026-05-15.md`
## 画布生成完成态不能被旧 autosave 覆盖
- 现象:release 外部生成 worker 补跑完成后,生成图已进入素材库或项目资源,但画布生成器仍显示 `generating`;刷新后可能仍看到历史生成框卡住。
- 原因:画布前端在提交生成后会把 `generating` layout 放入 450ms 自动保存队列;worker 完成后后端会写入 `idle + generatedLayerId + 生成层`,但旧的 pending / in-flight layout save 可能晚到并覆盖完成态。另有历史 inline 请求在 api-server 重启时只留下前端已保存的 `generating` 框,没有终态任务或生成资源。
- 处理:前端 `applyProjectSnapshot` 必须取消 pending layout save,并跳过一次由后端快照恢复触发的 autosave;后端 `save_editor_project_layout` 要保护已完成的 generation dialog,如果传入旧 `generating` 且无 `generatedLayerId`,而当前 layout 已有同一 dialog 的完成态,则保留完成态和生成层。线上脏数据只在确认无任务 / 无资源时标成 `failed` 并保留原 prompt 供用户重试。
- 验证:`npm run test -- src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx src/components/image-editor/useImageCanvasProjectPersistence.test.tsx``cargo test -p spacetime-module --manifest-path server-rs/Cargo.toml editor_project_storage --lib`release 排障用 `list_editor_projects_and_return` / `get_editor_project_and_return``generation-dialog` 状态,不要只看素材库。
- 关联:`src/components/image-editor/useImageCanvasProjectPersistence.ts``server-rs/crates/spacetime-module/src/editor_project_storage.rs``server-rs/crates/api-server/src/external_generation_worker.rs`
## Pingora 静态缓存不能只写 Cache-Control
- 现象:直连 Pingora 后,HTML 入口虽然是 `Cache-Control: no-cache`,但浏览器每次都重新下载完整入口页或普通静态文件;或者 Vite 指纹资源长期缓存正常,但旧标签页刷新时协商缓存行为和 Nginx 直连不同。
@@ -1,6 +1,6 @@
# 外部生成 Worker 化方案
更新时间:`2026-06-22`
更新时间:`2026-06-24`
## 背景
@@ -26,6 +26,8 @@
- `renew_external_generation_job_lease_and_return`worker 长任务执行期间按 `worker_id + lease_token` 续租,防止外部生成超过单次 lease 后被重复领取。
- `complete_external_generation_job_and_return`worker 成功后按 `worker_id + lease_token` 写入 `result_payload_json`,任务进入 `completed`
- `fail_external_generation_job_and_return`worker 失败后按 `worker_id + lease_token` 回写错误,并按 `max_attempts` 决定回到 `pending` 重试或进入 `failed`
- `list_external_generation_jobs_and_return`:按当前账号读取正式生成任务列表,返回 pending / running / 未确认终态数量、任务价格和完成提示确认状态。
- `acknowledge_external_generation_jobs_and_return`:按当前账号确认已终态任务的完成 / 失败提示,写入 `notification_acknowledged_at` 并追加审计事件。
- `get_external_generation_queue_stats_and_return`controller 读取队列积压、运行中任务和过期 lease 数量,用于计算 worker 目标实例数;该 procedure 只读 `external_generation_job`,不直接操作 systemd。
- `get_external_generation_job_and_return`:按 `job_id` 读取单个任务状态,给 BFF 和生成页展示使用;必须只返回调用者有权读取的任务,不能暴露其它用户的 payload、错误详情或 worker 内部字段。
@@ -35,10 +37,12 @@
队列状态对前端只通过 `api-server` BFF 暴露,不允许前端直接查询 SpacetimeDB private table
- `GET /api/runtime/external-generation/queue-overview`:队列概览,用于 `我的` 页签、调试面板或后台观测当前用户可见的等待状态。返回 pending / running / completed / failed / cancelled 数量、最早等待时间、当前可见 job 摘要,以及是否存在过期 lease 需要等待 worker 重领
- `GET /api/runtime/external-generation/queue-overview`当前账号队列概览,用于兼容旧展示和轻量状态读取。返回 pendingrunning、未确认终态数量和更新时间
- `GET /api/runtime/external-generation/jobs?limit=20&includeAcknowledgedTerminal=false`:当前账号正式生成任务列表,用于 `我的` 页签任务列表和完成 / 失败提示。返回每个任务的 job id、kind、source、可展示 label、状态、进度、错误、`priceMudPoints``refundLedgerId``notificationAcknowledgedAt` 和时间戳。默认不返回已确认的终态任务。
- `POST /api/runtime/external-generation/jobs/acknowledge`:用户关闭生成完成 / 失败提示时调用,BFF 只传当前账号 job ids,后端只确认属于当前账号且已终态的任务。
- `GET /api/runtime/external-generation/jobs/{jobId}`:单 job 状态,用于生成页轮询某次动作。返回 `jobId``jobKind``sourceModule``sourceEntityId``status``attempt``maxAttempts``createdAt``startedAt``completedAt``updatedAt`、可展示的 `requestLabel`、可展示的 `lastErrorMessage`、以及业务侧下一次轮询所需的 source 标识。
BFF 只做鉴权、授权裁剪、字段脱敏和契约映射;队列事实仍以 `external_generation_job` 为准,业务结果仍以玩法 session / work profile 为准。生成页 / 进度页只展示当前玩法业务进度;用户可见队列概览放在 `我的` 页签,必要时再用单 job 状态补充排障信息,并继续按原玩法 session/detail 接口收敛到 ready 或 failed。队列接口不替代玩法恢复接口,也不把 private `request_payload_json` 原样传给前端。
BFF 只做鉴权、授权裁剪、字段脱敏和契约映射;队列事实仍以 `external_generation_job` 为准,业务结果仍以玩法 session / work profile 为准。生成页 / 进度页只展示当前玩法业务进度;用户可见任务列表放在 `我的` 页签,必要时再用单 job 状态补充排障信息,并继续按原玩法 session/detail 接口收敛到 ready 或 failed。队列接口不替代玩法恢复接口,也不把 private `request_payload_json` 原样传给前端。终态提示的弹出与否以后端 `notification_acknowledged_at` 为准;前端关闭提示只能调用 acknowledge 接口,不能本地永久吞掉任务。
## 任务表
@@ -63,6 +67,11 @@ BFF 只做鉴权、授权裁剪、字段脱敏和契约映射;队列事实仍
| `available_at` | 下次可领取时间 |
| `result_payload_json` | 完成摘要 |
| `created_at/started_at/completed_at/updated_at` | 审计时间 |
| `price_mud_points` | 后端计算的本任务价格,用于任务列表展示和排障 |
| `refund_ledger_id` | 失败退款产生的钱包退款流水 ID,便于从任务追到退款记录 |
| `notification_acknowledged_at` | 用户已确认完成 / 失败提示的时间,未确认终态任务下次登录继续集中弹出 |
新增私有审计表 `external_generation_job_event`,记录 `enqueued/claimed/lease_renewed/completed/failed/acknowledged` 等事件。事件表只追加状态转换事实,不作为当前状态源;排障时先看 `external_generation_job` 当前状态,再按 `job_id``external_generation_job_event` 时间线。
索引:
@@ -205,7 +214,7 @@ curl -f http://127.0.0.1:<api-port>/healthz
本地 `npm run dev` 默认保持 `inline` 开发体验:未显式配置 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时,普通本地联调可以同步确认 provider、OSS 和 SpacetimeDB 写回链路本身是否可行。需要验证 worker 队列、BFF 队列状态、lease 重领或扩缩容时,必须显式使用 `queue`,并启动 worker 角色;可以用 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue GENARRATIVE_PROCESS_ROLE=all npm run dev:api-server` 做临时单进程 smoke,也可以使用隔离容器 smoke。
生产 smoke 需要保持 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`,并至少启动一个 `api` 角色、一个 `external-generation-worker` 角色和一个 `external-generation-controller` 角色;发布脚本会在默认 worker pattern 下自动启用并启动 `genarrative-external-generation-worker@1.service`,重启并验活 `genarrative-external-generation-controller.service`。若 worker 数量归零,生成任务会保持 `queued/running`,不会由 HTTP 进程偷偷执行。部署验证除 `/healthz` / `/readyz` 外,还要确认队列概览 BFF 可读、单 job 状态能从 `queued/running` 收敛到业务 session/detail 的 ready 或 failed。
生产 smoke 需要保持 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue`,并至少启动一个 `api` 角色、一个 `external-generation-worker` 角色和一个 `external-generation-controller` 角色;发布脚本会在默认 worker pattern 下自动启用并启动 `genarrative-external-generation-worker@1.service`,重启并验活 `genarrative-external-generation-controller.service`。若 worker 数量归零,生成任务会保持 `queued/running`,不会由 HTTP 进程偷偷执行。部署验证除 `/healthz` / `/readyz` 外,还要确认任务列表 BFF 可读、未确认终态任务会弹出提示、acknowledge 后不再弹出,单 job 状态能从 `queued/running` 收敛到业务 session/detail 的 ready 或 failed。
systemd 生产 controller 与手动兜底示例:
@@ -1,6 +1,6 @@
# server-rs 与 SpacetimeDB 数据契约
更新时间:`2026-06-12`
更新时间:`2026-06-24`
## 后端主线
@@ -193,7 +193,7 @@ npm run check:server-rs-ddd
3. 结果页单图重生成、发布、道具使用和其它独立资产操作仍按各自业务操作成本执行;不要把初始草稿成本误套到这些单次操作上。
4. 资产操作的预扣费必须 fail-closed:钱包或 SpacetimeDB 预扣费不可达、超时或返回业务错误时,`api-server` 直接返回错误,不允许继续调用图片、音频、GLB 等外部生成 provider。
5. 需要支持 HTTP retry 的计费 ledger id 必须包含当前请求的 `request_id`;前端 `fetchWithApiAuth` 同一次业务请求的静默刷新重试复用同一个 `x-request-id`,后端不得再使用 prompt 指纹或随机 asset id 作为扣费幂等键。
6. 外部生成已预扣费但后续失败时必须先同步调用钱包退款;若 SpacetimeDB 暂不可用,退款请求写入 `wallet-refund-outbox` 本地文件并由后台 worker 重放。默认启用,配置项为 `GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED``GENARRATIVE_WALLET_REFUND_OUTBOX_DIR``GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE``GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS``GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES`。outbox 文件按 refund ledger id 幂等落盘;成功重放后删除,坏文件隔离为 `corrupt-*`
6. 外部生成已预扣费但后续失败时必须先同步调用钱包退款;若 SpacetimeDB 暂不可用,退款请求写入 `wallet-refund-outbox` 本地文件并由后台 worker 重放。默认启用,配置项为 `GENARRATIVE_WALLET_REFUND_OUTBOX_ENABLED``GENARRATIVE_WALLET_REFUND_OUTBOX_DIR``GENARRATIVE_WALLET_REFUND_OUTBOX_BATCH_SIZE``GENARRATIVE_WALLET_REFUND_OUTBOX_FLUSH_INTERVAL_MS``GENARRATIVE_WALLET_REFUND_OUTBOX_MAX_BYTES`。outbox 文件按 refund ledger id 幂等落盘;成功重放后删除,坏文件隔离为 `corrupt-*`外部生成任务触发的扣费和退款必须在 `profile_wallet_ledger.metadata_json` 中写入 `externalGenerationJobId`,outbox 重放也必须保留同一任务 ID,便于从退款记录追溯到正式生成任务。
7. 拼图首图后台生成的跨实例互斥锁必须落在 SpacetimeDB `puzzle_background_compile_task` 表,claim id 由 `task_id + request_id` 构成,释放时必须校验 claim id,避免旧后台任务释放新请求抢到的租约。
## 用户钱包与编辑器生成扣费契约
@@ -245,7 +245,13 @@ npm run check:server-rs-ddd
- Rust 结构体:`ExternalGenerationJob`
- 源码:`server-rs/crates/spacetime-module/src/external_generation.rs`
- 用途:外部生成 worker 的持久任务队列;`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时,`api-server` HTTP 角色只入队,`external-generation-worker` 角色通过 claim lease 领取、续租、执行,并用 `lease_token` 栅栏回写完成 / 失败。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft``generate_puzzle_images``generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity,避免过期 worker 写 session / work profile;图片画布编辑器的 `editor_image_generation``editor_image_edit``editor_icon_spritesheet_generation``editor_ui_design_asset_extraction``editor_character_animation_generation``editor_video_generation``editor_sound_effect_generation``editor_background_music_generation` 复用同一队列表,worker 成功后经 `api-server` facade 写入 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`,前端只通过 BFF job 状态轮询和项目快照读取恢复完成态。`GENARRATIVE_EXTERNAL_GENERATION_MODE=inline` 时不创建该队列行,三个 external generation guard 字段必须同时为空才允许 api-server 受控同步写回,半空 guard 仍会拒绝。worker 成功写回业务事实后才能 complete job;业务失败态写回成功后才能 fail job,失败态未写回时保留租约等待后续重领。
- 用途:外部生成 worker 的持久任务队列和用户可见生成任务列表`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时,`api-server` HTTP 角色只入队,`external-generation-worker` 角色通过 claim lease 领取、续租、执行,并用 `lease_token` 栅栏回写完成 / 失败。队列行同时保存 `price_mud_points``refund_ledger_id``notification_acknowledged_at`BFF 通过 `GET /api/runtime/external-generation/jobs` 返回当前账号的正式生成任务列表、价格、状态和未确认终态数量;前端只能展示该后端事实,完成 / 失败提示关闭时必须调用 `POST /api/runtime/external-generation/jobs/acknowledge` 由后端写确认时间,未确认终态任务会在下次登录后再次集中弹出。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft``generate_puzzle_images``generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity,避免过期 worker 写 session / work profile;图片画布编辑器的 `editor_image_generation``editor_image_edit``editor_icon_spritesheet_generation``editor_ui_design_asset_extraction``editor_character_animation_generation``editor_video_generation``editor_sound_effect_generation``editor_background_music_generation` 复用同一队列表,worker 成功后经 `api-server` facade 写入 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`,前端只通过 BFF job 状态轮询和项目快照读取恢复完成态。`GENARRATIVE_EXTERNAL_GENERATION_MODE=inline` 时不创建该队列行,三个 external generation guard 字段必须同时为空才允许 api-server 受控同步写回,半空 guard 仍会拒绝。worker 成功写回业务事实后才能 complete job;业务失败态写回成功后才能 fail job,失败态未写回时保留租约等待后续重领。
### `external_generation_job_event`
- Rust 结构体:`ExternalGenerationJobEvent`
- 源码:`server-rs/crates/spacetime-module/src/external_generation.rs`
- 用途:外部生成任务审计事件表,按 `job_id``owner_user_id` 记录 `enqueued``claimed``lease_renewed``completed``failed``acknowledged` 等状态转换事实。状态转换只能由 SpacetimeDB procedure 写入,不由前端或 worker 直接改表;该表用于追溯任务生命周期和排障,不替代 `external_generation_job` 当前状态。
### `ai_text_chunk`
@@ -679,6 +685,7 @@ npm run check:server-rs-ddd
- Rust 结构体:`ProfileWalletLedger`
- 源码:`server-rs/crates/spacetime-module/src/runtime/profile.rs`
- 说明:账号钱包流水表。`metadata_json` 为可选 JSON 对象字符串,旧行缺失时读取层按 `{}` 归一;外部生成扣费 / 退款写入 `externalGenerationJobId`,使退款记录可以追溯到对应 `external_generation_job`
### `profile_wallet_config`
@@ -229,10 +229,10 @@ UI 相关修改要重点验证:
数据库备份不放进 `spacetime-module` reducer / procedure:备份属于文件系统与 OSS 外部副作用,必须由运维脚本在 SpacetimeDB 宿主外执行。当前统一脚本为 `scripts/database-backup-to-oss.mjs`npm 命令 `npm run database:backup:oss`);生产 provision 还会安装 `genarrative-database-backup.timer`,每天 `03:20` 左右自动执行一次 OSS 冷备份:
```bash
npm run database:backup:oss -- --data-dir /stdb --stop-service spacetimedb.service --restart-service-after genarrative-api.service
npm run database:backup:oss -- --data-dir /stdb --stop-service spacetimedb.service --restart-service-after genarrative-api.service --restart-service-after genarrative-external-generation-worker@1.service --restart-service-after genarrative-external-generation-controller.service
```
脚本会将数据目录打包成 `tar.gz`,上传到 `oss://<bucket>/<prefix>/<database>/<database>-<UTC时间>.tar.gz`。生产建议做冷备份:传入 `--stop-service spacetimedb.service`,脚本会在打包前停止服务、打包后恢复服务,再上传 OSS;因 `genarrative-api.service` 依赖 `spacetimedb.service`,生产定时冷备份还必须传入 `--restart-service-after genarrative-api.service`,确保备份后 API 随数据库一起恢复。`2026-06-10` release 故障就是现场 unit 漏掉参数,`03:20` 冷备份停止 SpacetimeDB 后 API 被依赖关系一并停止,备份脚本只恢复了 SpacetimeDB,API 直到人工重启前都不可用;后续现场变更、provision 模板和 Jenkins 归档都必须通过 `npm run check:production-ops` 防止回退。由于 OSS 上传可能受服务器带宽限制,`Genarrative-Stdb-Module-Publish` 默认使用 `DATABASE_BACKUP_MODE=async`:先在 publish 前用 `--defer-upload` 生成本地冷备份和 `.manifest.json`,随后继续执行 publish;发布脚本退出前会用后台 `node -- ... --upload-archive <tar.gz>` 上传同一份发布前备份,不等待上传完成。发布脚本在校验 wasm 后、执行 `spacetime publish` 前会等待显式 `SPACETIME_SERVER_URL``/v1/ping` 就绪,默认最多等待 `60` 秒;如生产机器冷备份恢复 `spacetimedb.service` 较慢,可临时设置 `GENARRATIVE_STDB_PUBLISH_READY_TIMEOUT_SECONDS` 调整等待时间。需要强一致发布闸门时改用 `DATABASE_BACKUP_MODE=sync`(等价脚本参数 `--backup-mode sync`),备份会在 publish 前同步打包并上传,失败会阻断 publish;确认已有其他备份窗口时才使用 `DATABASE_BACKUP_MODE=skip`(兼容脚本参数 `--skip-backup`)。若业务不能接受停机窗口,应先规划 SpacetimeDB 原生快照或主备策略,不要直接在写入中的数据目录上做热拷贝并当作强一致备份。
脚本会将数据目录打包成 `tar.gz`,上传到 `oss://<bucket>/<prefix>/<database>/<database>-<UTC时间>.tar.gz`。生产建议做冷备份:传入 `--stop-service spacetimedb.service`,脚本会在打包前停止服务、打包后恢复服务,再上传 OSS;因 `genarrative-api.service``genarrative-external-generation-worker@*.service``genarrative-external-generation-controller.service`依赖 `spacetimedb.service`,生产定时冷备份还必须传入对应的 `--restart-service-after`,确保备份后 API、保底 worker 和 controller 随数据库一起恢复。`2026-06-10` release 故障就是现场 unit 漏掉 API 重启参数,`03:20` 冷备份停止 SpacetimeDB 后 API 被依赖关系一并停止,备份脚本只恢复了 SpacetimeDB,API 直到人工重启前都不可用;`2026-06-24` release 又出现同类依赖停机后只恢复 API、未恢复外部生成 worker/controller,导致图片画布生成任务长期停留在队列中。后续现场变更、provision 模板和 Jenkins 归档都必须通过 `npm run check:production-ops` 防止回退。由于 OSS 上传可能受服务器带宽限制,`Genarrative-Stdb-Module-Publish` 默认使用 `DATABASE_BACKUP_MODE=async`:先在 publish 前用 `--defer-upload` 生成本地冷备份和 `.manifest.json`,随后继续执行 publish;发布脚本退出前会用后台 `node -- ... --upload-archive <tar.gz>` 上传同一份发布前备份,不等待上传完成。发布脚本在校验 wasm 后、执行 `spacetime publish` 前会等待显式 `SPACETIME_SERVER_URL``/v1/ping` 就绪,默认最多等待 `60` 秒;如生产机器冷备份恢复 `spacetimedb.service` 较慢,可临时设置 `GENARRATIVE_STDB_PUBLISH_READY_TIMEOUT_SECONDS` 调整等待时间。需要强一致发布闸门时改用 `DATABASE_BACKUP_MODE=sync`(等价脚本参数 `--backup-mode sync`),备份会在 publish 前同步打包并上传,失败会阻断 publish;确认已有其他备份窗口时才使用 `DATABASE_BACKUP_MODE=skip`(兼容脚本参数 `--skip-backup`)。若业务不能接受停机窗口,应先规划 SpacetimeDB 原生快照或主备策略,不要直接在写入中的数据目录上做热拷贝并当作强一致备份。
生产环境变量模板在 `deploy/env/api-server.env.example`
@@ -7,6 +7,7 @@ export type ExternalGenerationJobStatus =
export interface ExternalGenerationQueueOverview {
pendingCount: number;
runningCount: number;
unacknowledgedTerminalCount: number;
updatedAtMicros: number;
}
@@ -27,3 +28,37 @@ export interface ExternalGenerationJobStatusRecord {
export interface ExternalGenerationJobStatusResponse {
job: ExternalGenerationJobStatusRecord;
}
export interface ExternalGenerationTaskRecord {
jobId: string;
jobKind: string;
sourceModule: string;
sourceEntityId: string;
requestLabel: string;
status: ExternalGenerationJobStatus;
phaseLabel: string;
phaseDetail: string;
progress: number;
error?: string | null;
priceMudPoints: number;
refundLedgerId?: string | null;
notificationAcknowledgedAt?: string | null;
createdAt: string;
startedAt?: string | null;
completedAt?: string | null;
updatedAt: string;
updatedAtMicros: number;
}
export interface ExternalGenerationTaskListResponse {
tasks: ExternalGenerationTaskRecord[];
overview: ExternalGenerationQueueOverview;
}
export interface ExternalGenerationTaskAcknowledgeRequest {
jobIds: string[];
}
export interface ExternalGenerationTaskAcknowledgeResponse {
acknowledgedTasks: ExternalGenerationTaskRecord[];
}
@@ -8,6 +8,16 @@ const checks = [
includes: '--restart-service-after genarrative-api.service',
reason: '生产冷备份恢复 SpacetimeDB 后必须显式拉起依赖它的 API 服务。',
},
{
file: 'deploy/systemd/genarrative-database-backup.service',
includes: '--restart-service-after genarrative-external-generation-worker@1.service',
reason: '生产冷备份恢复 SpacetimeDB 后必须显式拉起保底外部生成 worker。',
},
{
file: 'deploy/systemd/genarrative-database-backup.service',
includes: '--restart-service-after genarrative-external-generation-controller.service',
reason: '生产冷备份恢复 SpacetimeDB 后必须显式拉起外部生成 worker controller。',
},
{
file: 'deploy/systemd/genarrative-database-backup.service',
includes: 'ExecStart=/usr/bin/node -- /opt/genarrative/current/scripts/database-backup-to-oss.mjs --env-file',
@@ -12,6 +12,22 @@ use crate::{
pub(crate) const ASSET_OPERATION_POINTS_COST: u64 = 1;
tokio::task_local! {
static CURRENT_EXTERNAL_GENERATION_JOB_ID: String;
}
pub(crate) async fn with_external_generation_billing_task_id<T, Fut>(
job_id: String,
future: Fut,
) -> T
where
Fut: Future<Output = T>,
{
CURRENT_EXTERNAL_GENERATION_JOB_ID
.scope(job_id, future)
.await
}
/// 资产操作统一执行入口:业务层只声明操作类型与资源 ID,钱包扣退费由服务层收口。
pub(crate) async fn execute_billable_asset_operation<T, Fut>(
state: &AppState,
@@ -99,11 +115,12 @@ async fn consume_asset_operation_points(
);
match state
.spacetime_client()
.consume_profile_wallet_points(
.consume_profile_wallet_points_with_metadata(
owner_user_id.to_string(),
points_cost,
ledger_id,
current_utc_micros(),
current_wallet_metadata_json(),
)
.await
{
@@ -125,13 +142,16 @@ async fn refund_asset_operation_points(
owner_user_id, asset_kind, asset_id
);
let created_at_micros = current_utc_micros();
let external_generation_job_id = current_external_generation_job_id();
let metadata_json = wallet_metadata_json(external_generation_job_id.as_deref());
if let Err(error) = state
.spacetime_client()
.refund_profile_wallet_points(
.refund_profile_wallet_points_with_metadata(
owner_user_id.to_string(),
points_cost,
ledger_id.clone(),
created_at_micros,
metadata_json,
)
.await
{
@@ -145,6 +165,7 @@ async fn refund_asset_operation_points(
created_at_micros,
asset_kind: asset_kind.to_string(),
asset_id: asset_id.to_string(),
external_generation_job_id: external_generation_job_id.clone(),
})
.await
{
@@ -153,6 +174,7 @@ async fn refund_asset_operation_points(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
error = %refund_error,
"资产操作失败后的泥点退款立即执行失败,已写入 wallet refund outbox"
@@ -164,6 +186,7 @@ async fn refund_asset_operation_points(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
reason,
error = %refund_error,
@@ -176,6 +199,7 @@ async fn refund_asset_operation_points(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
refund_error = %refund_error,
outbox_error = %outbox_error,
@@ -189,6 +213,7 @@ async fn refund_asset_operation_points(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
error = %refund_error,
"资产操作失败后的泥点退款失败,且 wallet refund outbox 未启用"
@@ -196,6 +221,32 @@ async fn refund_asset_operation_points(
}
}
fn current_external_generation_job_id() -> Option<String> {
CURRENT_EXTERNAL_GENERATION_JOB_ID
.try_with(|value| value.trim().to_string())
.ok()
.filter(|value| !value.is_empty())
}
fn current_wallet_metadata_json() -> String {
let external_generation_job_id = current_external_generation_job_id();
wallet_metadata_json(external_generation_job_id.as_deref())
}
fn wallet_metadata_json(external_generation_job_id: Option<&str>) -> String {
let Some(external_generation_job_id) = external_generation_job_id
.map(str::trim)
.filter(|value| !value.is_empty())
else {
return module_runtime::PROFILE_INVITE_CODE_METADATA_DEFAULT_JSON.to_string();
};
json!({
"externalGenerationJobId": external_generation_job_id,
})
.to_string()
}
pub(crate) fn map_asset_operation_wallet_error(error: SpacetimeClientError) -> AppError {
let message = error.to_string();
tracing::warn!(
@@ -545,6 +545,18 @@ pub async fn generate_editor_character_animation(
})?;
let owner_user_id = authenticated.claims().user_id().to_string();
if !state.config.external_generation_mode.is_inline() {
let pricing = state.editor_generation_pricing().map_err(|error| {
character_animation_error_response(
&request_context,
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({
"provider": "editor-generation-pricing",
"message": error.to_string(),
})),
)
})?;
let normalized =
normalize_editor_character_animation_request_with_pricing(payload.clone(), &pricing)
.map_err(|error| character_animation_error_response(&request_context, error))?;
let source_entity_id = editor_generation_source_entity_id(
payload.project_id.as_deref(),
payload.source_layer_id.as_str(),
@@ -556,6 +568,7 @@ pub async fn generate_editor_character_animation(
EDITOR_CHARACTER_ANIMATION_GENERATION_JOB_KIND,
source_entity_id,
"图片画布生成角色动作",
u64::from(normalized.price_mud_points),
&payload,
)
.await
@@ -760,6 +773,17 @@ pub async fn generate_editor_video(
})?;
let owner_user_id = authenticated.claims().user_id().to_string();
if !state.config.external_generation_mode.is_inline() {
let pricing = state.editor_generation_pricing().map_err(|error| {
editor_video_error_response(
&request_context,
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({
"provider": "editor-generation-pricing",
"message": error.to_string(),
})),
)
})?;
let normalized = normalize_editor_video_request_with_pricing(payload.clone(), &pricing)
.map_err(|error| editor_video_error_response(&request_context, error))?;
let source_entity_id =
editor_generation_source_entity_id(payload.project_id.as_deref(), "editor-video");
let queue_job = enqueue_editor_generation_job(
@@ -769,6 +793,7 @@ pub async fn generate_editor_video(
EDITOR_VIDEO_GENERATION_JOB_KIND,
source_entity_id,
"图片画布生成视频",
u64::from(normalized.price_mud_points),
&payload,
)
.await
+9 -13
View File
@@ -1,4 +1,4 @@
use std::{env, fs, net::SocketAddr, path::PathBuf, time::Duration};
use std::{env, fs, net::SocketAddr, path::PathBuf, time::Duration};
use platform_llm::{
DEFAULT_ARK_BASE_URL, DEFAULT_MAX_RETRIES, DEFAULT_REQUEST_TIMEOUT_MS,
@@ -1113,21 +1113,17 @@ impl AppConfig {
config.ark_character_video_model = ark_character_video_model;
}
if let Some(character_animation_ffmpeg_path) =
read_first_non_empty_env(&[
"GENARRATIVE_CHARACTER_ANIMATION_FFMPEG_PATH",
"CHARACTER_ANIMATION_FFMPEG_PATH",
])
{
if let Some(character_animation_ffmpeg_path) = read_first_non_empty_env(&[
"GENARRATIVE_CHARACTER_ANIMATION_FFMPEG_PATH",
"CHARACTER_ANIMATION_FFMPEG_PATH",
]) {
config.character_animation_ffmpeg_path = character_animation_ffmpeg_path;
}
if let Some(character_animation_ffprobe_path) =
read_first_non_empty_env(&[
"GENARRATIVE_CHARACTER_ANIMATION_FFPROBE_PATH",
"CHARACTER_ANIMATION_FFPROBE_PATH",
])
{
if let Some(character_animation_ffprobe_path) = read_first_non_empty_env(&[
"GENARRATIVE_CHARACTER_ANIMATION_FFPROBE_PATH",
"CHARACTER_ANIMATION_FFPROBE_PATH",
]) {
config.character_animation_ffprobe_path = character_animation_ffprobe_path;
}
@@ -95,10 +95,8 @@ impl EditorGenerationPricingConfig {
image_size: Option<&str>,
) -> u32 {
let normalized_model = normalize_editor_image_model(model);
let normalized_size = normalize_editor_generation_image_price_size(
normalized_model,
image_size,
);
let normalized_size =
normalize_editor_generation_image_price_size(normalized_model, image_size);
read_tier_price(
&self.models,
normalized_model,
@@ -358,7 +356,10 @@ fn normalize_editor_image_model(model: Option<&str>) -> &'static str {
}
}
fn normalize_editor_generation_image_price_size(model: &str, image_size: Option<&str>) -> &'static str {
fn normalize_editor_generation_image_price_size(
model: &str,
image_size: Option<&str>,
) -> &'static str {
match image_size.map(str::trim).filter(|value| !value.is_empty()) {
Some("0.5K" | "0.5k") if model == EDITOR_IMAGE_MODEL_NANOBANANA2 => IMAGE_PRICE_SIZE_0_5K,
Some("2K" | "2k") => IMAGE_PRICE_SIZE_2K,
@@ -575,7 +576,11 @@ mod tests {
EditorGenerationPricingUnit::PerSecond
);
assert!(!config.models.contains_key("spec:gpt-image-2"));
assert!(!config.models.contains_key("character-animation:seedance2.0-fast"));
assert!(
!config
.models
.contains_key("character-animation:seedance2.0-fast")
);
}
#[test]
@@ -597,19 +602,11 @@ mod tests {
12
);
assert_eq!(
editor_image_generation_mud_points(
Some("ui-design"),
Some("gpt-image-2"),
Some("1K")
),
editor_image_generation_mud_points(Some("ui-design"), Some("gpt-image-2"), Some("1K")),
3
);
assert_eq!(
editor_image_generation_mud_points(
Some("ui-design"),
Some("gpt-image-2"),
Some("2K")
),
editor_image_generation_mud_points(Some("ui-design"), Some("gpt-image-2"), Some("2K")),
5
);
assert_eq!(
@@ -697,7 +694,10 @@ mod tests {
let config = load_editor_generation_pricing_from_paths(Some(&override_path))
.expect("pricing should load");
assert_eq!(config.image_model_mud_points(Some("gpt-image-2"), Some("2K")), 62);
assert_eq!(
config.image_model_mud_points(Some("gpt-image-2"), Some("2K")),
62
);
assert_eq!(config.spec_model_mud_points(Some("gpt-image-2")), 62);
assert_eq!(
config.video_model_mud_points(Some("seedance2.0"), "720p", 5),
@@ -38,6 +38,7 @@ pub(crate) async fn enqueue_editor_generation_job<T>(
job_kind: &str,
source_entity_id: impl Into<String>,
request_label: impl Into<String>,
price_mud_points: u64,
payload: &T,
) -> Result<ExternalGenerationJobRecord, AppError>
where
@@ -65,6 +66,7 @@ where
max_attempts: 1,
available_at_micros: now_micros,
created_at_micros: now_micros,
price_mud_points,
})
.await
.map_err(|error| {
@@ -963,6 +963,34 @@ pub async fn generate_editor_image(
) -> Result<Json<Value>, AppError> {
let caller = EditorGenerationCaller::from_authenticated(&authenticated);
if !state.config.external_generation_mode.is_inline() {
let normalized_kind = payload.kind.as_deref().map(str::trim);
let is_ui_design_generation = matches!(normalized_kind, Some("ui-design"));
let is_publication_material_generation =
matches!(normalized_kind, Some("publication-material"));
let generation_options = normalize_editor_generation_options(
if is_ui_design_generation || is_publication_material_generation {
Some(GPT_IMAGE_2_MODEL)
} else {
payload.model.as_deref()
},
payload.aspect_ratio.as_deref(),
payload.image_size.as_deref(),
);
let price_mud_points = u64::from(
state
.editor_generation_pricing()
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({
"provider": "editor-generation-pricing",
"message": error.to_string(),
}))
})?
.image_generation_mud_points(
normalized_kind,
Some(generation_options.model),
Some(generation_options.image_size),
),
);
let source_entity_id = editor_generation_source_entity_id(
payload.project_id.as_deref(),
"editor-image-generation",
@@ -974,6 +1002,7 @@ pub async fn generate_editor_image(
EDITOR_IMAGE_GENERATION_JOB_KIND,
source_entity_id,
"图片画布生成图片",
price_mud_points,
&payload,
)
.await?;
@@ -1439,6 +1468,7 @@ pub async fn edit_editor_image(
) -> Result<Json<Value>, AppError> {
let caller = EditorGenerationCaller::from_authenticated(&authenticated);
if !state.config.external_generation_mode.is_inline() {
let price_mud_points = u64::from(resolve_editor_image_edit_price(&state)?);
let source_entity_id =
editor_generation_source_entity_id(payload.project_id.as_deref(), "editor-image-edit");
let queue_job = enqueue_editor_generation_job(
@@ -1448,6 +1478,7 @@ pub async fn edit_editor_image(
EDITOR_IMAGE_EDIT_JOB_KIND,
source_entity_id,
"图片画布修改图片",
price_mud_points,
&payload,
)
.await?;
@@ -1592,6 +1623,16 @@ pub async fn generate_editor_icon_spritesheet(
) -> Result<Json<Value>, AppError> {
let caller = EditorGenerationCaller::from_authenticated(&authenticated);
if !state.config.external_generation_mode.is_inline() {
let generation_options = normalize_editor_generation_options(
payload.model.as_deref(),
payload.aspect_ratio.as_deref(),
payload.image_size.as_deref(),
);
let price_mud_points = u64::from(resolve_editor_icon_spritesheet_price(
&state,
Some(generation_options.model),
Some(generation_options.image_size),
)?);
let source_entity_id = editor_generation_source_entity_id(
payload.project_id.as_deref(),
"editor-icon-spritesheet-generation",
@@ -1603,6 +1644,7 @@ pub async fn generate_editor_icon_spritesheet(
EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND,
source_entity_id,
"图片画布生成图标素材",
price_mud_points,
&payload,
)
.await?;
@@ -1860,6 +1902,14 @@ pub async fn extract_editor_ui_design_assets(
) -> Result<Json<Value>, AppError> {
let caller = EditorGenerationCaller::from_authenticated(&authenticated);
if !state.config.external_generation_mode.is_inline() {
let generation_options = normalize_editor_ui_design_asset_extraction_options(
payload.aspect_ratio.as_str(),
payload.image_size.as_str(),
)?;
let price_mud_points = u64::from(resolve_editor_ui_design_asset_extraction_price(
&state,
Some(generation_options.image_size),
)?);
let source_entity_id = editor_generation_source_entity_id(
payload.project_id.as_deref(),
"editor-ui-design-asset-extraction",
@@ -1871,6 +1921,7 @@ pub async fn extract_editor_ui_design_assets(
EDITOR_UI_DESIGN_ASSET_EXTRACTION_JOB_KIND,
source_entity_id,
"图片画布提取UI设计图素材",
price_mud_points,
&payload,
)
.await?;
@@ -1,18 +1,22 @@
use axum::{
Json,
extract::{Extension, Path, State},
extract::{Extension, Path, Query, State},
http::StatusCode,
response::Response,
};
use serde::Deserialize;
use serde_json::json;
use shared_contracts::external_generation::{
ExternalGenerationJobStatus, ExternalGenerationJobStatusRecord,
ExternalGenerationJobStatusResponse, ExternalGenerationQueueOverview,
ExternalGenerationQueueOverviewResponse,
ExternalGenerationQueueOverviewResponse, ExternalGenerationTaskAcknowledgeRequest,
ExternalGenerationTaskAcknowledgeResponse, ExternalGenerationTaskListResponse,
ExternalGenerationTaskRecord,
};
use spacetime_client::{
ExternalGenerationJobGetRecordInput, ExternalGenerationJobRecord,
ExternalGenerationQueueStatsRecord, SpacetimeClientError,
ExternalGenerationJobAcknowledgeRecordInput, ExternalGenerationJobGetRecordInput,
ExternalGenerationJobListRecord, ExternalGenerationJobListRecordInput,
ExternalGenerationJobRecord, SpacetimeClientError,
};
use crate::{
@@ -22,20 +26,92 @@ use crate::{
const EXTERNAL_GENERATION_PROVIDER: &str = "external_generation";
#[derive(Debug, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ExternalGenerationTaskListQuery {
limit: Option<u32>,
include_acknowledged_terminal: Option<bool>,
}
pub async fn get_external_generation_queue_overview(
State(state): State<AppState>,
Extension(request_context): Extension<RequestContext>,
Extension(authenticated): Extension<AuthenticatedAccessToken>,
) -> Result<Json<serde_json::Value>, Response> {
let stats = state
let owner_user_id = authenticated.claims().user_id().to_string();
let list = state
.spacetime_client()
.get_external_generation_queue_stats()
.list_external_generation_jobs(ExternalGenerationJobListRecordInput {
owner_user_id,
limit: 1,
include_acknowledged_terminal: false,
})
.await
.map_err(|error| external_generation_error_response(&request_context, error))?;
Ok(json_success_body(
Some(&request_context),
ExternalGenerationQueueOverviewResponse {
overview: map_external_generation_queue_overview(stats),
overview: map_external_generation_queue_overview(&list),
},
))
}
pub async fn list_external_generation_tasks(
State(state): State<AppState>,
Extension(request_context): Extension<RequestContext>,
Extension(authenticated): Extension<AuthenticatedAccessToken>,
Query(query): Query<ExternalGenerationTaskListQuery>,
) -> Result<Json<serde_json::Value>, Response> {
let owner_user_id = authenticated.claims().user_id().to_string();
let list = state
.spacetime_client()
.list_external_generation_jobs(ExternalGenerationJobListRecordInput {
owner_user_id,
limit: query.limit.unwrap_or(50).clamp(1, 100),
include_acknowledged_terminal: query.include_acknowledged_terminal.unwrap_or(false),
})
.await
.map_err(|error| external_generation_error_response(&request_context, error))?;
Ok(json_success_body(
Some(&request_context),
ExternalGenerationTaskListResponse {
overview: map_external_generation_queue_overview(&list),
tasks: list
.jobs
.into_iter()
.map(map_external_generation_task_record)
.collect(),
},
))
}
pub async fn acknowledge_external_generation_tasks(
State(state): State<AppState>,
Extension(request_context): Extension<RequestContext>,
Extension(authenticated): Extension<AuthenticatedAccessToken>,
Json(payload): Json<ExternalGenerationTaskAcknowledgeRequest>,
) -> Result<Json<serde_json::Value>, Response> {
let owner_user_id = authenticated.claims().user_id().to_string();
let acknowledged = state
.spacetime_client()
.acknowledge_external_generation_jobs(ExternalGenerationJobAcknowledgeRecordInput {
owner_user_id,
job_ids: payload.job_ids,
acknowledged_at_micros: current_utc_micros(),
})
.await
.map_err(|error| external_generation_error_response(&request_context, error))?;
Ok(json_success_body(
Some(&request_context),
ExternalGenerationTaskAcknowledgeResponse {
acknowledged_tasks: acknowledged
.jobs
.into_iter()
.map(map_external_generation_task_record)
.collect(),
},
))
}
@@ -65,12 +141,13 @@ pub async fn get_external_generation_job_status(
}
fn map_external_generation_queue_overview(
stats: ExternalGenerationQueueStatsRecord,
list: &ExternalGenerationJobListRecord,
) -> ExternalGenerationQueueOverview {
ExternalGenerationQueueOverview {
pending_count: stats.pending_count,
running_count: stats.running_active_count,
updated_at_micros: stats.now_micros,
pending_count: list.pending_count,
running_count: list.running_count,
unacknowledged_terminal_count: list.unacknowledged_terminal_count,
updated_at_micros: list.now_micros,
}
}
@@ -85,16 +162,46 @@ fn map_external_generation_job_status(
};
ExternalGenerationJobStatusRecord {
operation_id: job.job_id,
operation_id: job.job_id.clone(),
status,
phase_label: job.request_label,
phase_label: job.request_label.clone(),
phase_detail: phase_detail.to_string(),
progress,
error: job.last_error_message,
error: job.last_error_message.clone(),
updated_at_micros: job.updated_at_micros,
}
}
fn map_external_generation_task_record(
job: ExternalGenerationJobRecord,
) -> ExternalGenerationTaskRecord {
let status_record = map_external_generation_job_status(job.clone());
ExternalGenerationTaskRecord {
job_id: job.job_id,
job_kind: job.job_kind,
source_module: job.source_module,
source_entity_id: job.source_entity_id,
request_label: job.request_label,
status: status_record.status,
phase_label: status_record.phase_label,
phase_detail: status_record.phase_detail,
progress: status_record.progress,
error: status_record.error,
price_mud_points: job.price_mud_points,
refund_ledger_id: job.refund_ledger_id,
notification_acknowledged_at: job.notification_acknowledged_at,
created_at: job.created_at,
started_at: job.started_at,
completed_at: job.completed_at,
updated_at: job.updated_at,
updated_at_micros: job.updated_at_micros,
}
}
fn current_utc_micros() -> i64 {
shared_kernel::offset_datetime_to_unix_micros(time::OffsetDateTime::now_utc())
}
fn external_generation_error_response(
request_context: &RequestContext,
error: SpacetimeClientError,
@@ -15,6 +15,7 @@ use tokio::{
use tracing::{error, info, warn};
use crate::{
asset_billing::with_external_generation_billing_task_id,
character_animation_assets::{
generate_editor_character_animation_for_owner, generate_editor_video_for_owner,
},
@@ -242,7 +243,10 @@ async fn process_external_generation_job(
job: ExternalGenerationJobRecord,
) -> Result<(), String> {
let heartbeat_interval = external_generation_worker_heartbeat_interval(lease);
let work = process_external_generation_job_once(state.clone(), worker_id.clone(), job.clone());
let work = with_external_generation_billing_task_id(
job.job_id.clone(),
process_external_generation_job_once(state.clone(), worker_id.clone(), job.clone()),
);
tokio::pin!(work);
let heartbeat = sleep(heartbeat_interval);
tokio::pin!(heartbeat);
@@ -918,6 +922,7 @@ async fn fail_job(
error_message,
retry_after_micros: now_micros.saturating_add(60_000_000),
failed_at_micros: now_micros,
refund_ledger_id: None,
})
.await
.map(|_| ())
@@ -1030,6 +1035,10 @@ mod tests {
updated_at: "2026-06-03T00:00:00Z".to_string(),
updated_at_micros: 1_780_444_800_000_000,
lease_token: lease_token.map(ToOwned::to_owned),
price_mud_points: 2,
refund_ledger_id: None,
notification_acknowledged_at: None,
notification_acknowledged_at_micros: None,
}
}
}
@@ -301,6 +301,7 @@ async fn enqueue_jump_hop_compile_draft_job(
) -> Result<ExternalGenerationJobRecord, Response> {
let job_id = build_prefixed_uuid_id("extgen-");
let now_micros = current_utc_micros();
let price_mud_points = resolve_jump_hop_generation_points_cost(state).await;
let request_payload_json = serde_json::to_string(&JumpHopCompileDraftWorkerPayload {
session_id: session_id.to_string(),
owner_user_id: owner_user_id.to_string(),
@@ -329,6 +330,7 @@ async fn enqueue_jump_hop_compile_draft_job(
max_attempts: 1,
available_at_micros: now_micros,
created_at_micros: now_micros,
price_mud_points,
})
.await
.map_err(|error| {
@@ -1,9 +1,13 @@
use axum::{Router, middleware, routing::get};
use axum::{
Router, middleware,
routing::{get, post},
};
use crate::{
auth::require_bearer_auth,
external_generation::{
get_external_generation_job_status, get_external_generation_queue_overview,
acknowledge_external_generation_tasks, get_external_generation_job_status,
get_external_generation_queue_overview, list_external_generation_tasks,
},
state::AppState,
};
@@ -16,6 +20,19 @@ pub fn router(state: AppState) -> Router<AppState> {
middleware::from_fn_with_state(state.clone(), require_bearer_auth),
),
)
.route(
"/api/runtime/external-generation/jobs",
get(list_external_generation_tasks).route_layer(middleware::from_fn_with_state(
state.clone(),
require_bearer_auth,
)),
)
.route(
"/api/runtime/external-generation/jobs/acknowledge",
post(acknowledge_external_generation_tasks).route_layer(
middleware::from_fn_with_state(state.clone(), require_bearer_auth),
),
)
.route(
"/api/runtime/external-generation/jobs/{job_id}",
get(get_external_generation_job_status).route_layer(middleware::from_fn_with_state(
@@ -123,7 +123,8 @@ fn build_npc_animation_prompt(
"角色固定为同一人,侧身朝右,镜头稳定,轮廓清晰,武器不可丢失。".to_string(),
"动作连贯,避免服装、发型、面部、武器随机漂移。".to_string(),
if use_chroma_key {
"背景为单一纯绿色 #00FF00 / RGB(0,255,0) 绿幕,无其他人物和场景元素,方便后期抠像。".to_string()
"背景为单一纯绿色 #00FF00 / RGB(0,255,0) 绿幕,无其他人物和场景元素,方便后期抠像。"
.to_string()
} else {
"背景简洁纯净,无复杂场景。".to_string()
},
@@ -820,6 +820,7 @@ pub async fn execute_puzzle_agent_action(
max_attempts: 1,
available_at_micros: now,
created_at_micros: now,
price_mud_points: puzzle_draft_generation_points_cost,
})
.await
.map_err(|error| {
@@ -1045,6 +1046,7 @@ pub async fn execute_puzzle_agent_action(
max_attempts: 1,
available_at_micros: now,
created_at_micros: now,
price_mud_points: PUZZLE_IMAGE_GENERATION_POINTS_COST,
})
.await
.map_err(|error| {
@@ -1187,6 +1189,7 @@ pub async fn execute_puzzle_agent_action(
max_attempts: 1,
available_at_micros: now,
created_at_micros: now,
price_mud_points: PUZZLE_IMAGE_GENERATION_POINTS_COST,
})
.await
.map_err(|error| {
@@ -284,6 +284,7 @@ async fn enqueue_puzzle_clear_compile_draft_job(
})),
)
})?;
let price_mud_points = resolve_puzzle_clear_generation_points_cost(state).await;
state
.spacetime_client()
.enqueue_external_generation_job(ExternalGenerationJobEnqueueRecordInput {
@@ -298,6 +299,7 @@ async fn enqueue_puzzle_clear_compile_draft_job(
max_attempts: 1,
available_at_micros: now_micros,
created_at_micros: now_micros,
price_mud_points,
})
.await
.map_err(|error| {
@@ -309,6 +311,15 @@ async fn enqueue_puzzle_clear_compile_draft_job(
})
}
async fn resolve_puzzle_clear_generation_points_cost(state: &AppState) -> u64 {
crate::creation_entry_config::resolve_creation_entry_mud_point_cost(
state,
PUZZLE_CLEAR_TEMPLATE_ID,
u64::from(shared_contracts::creation_entry_config::DEFAULT_UNIFIED_CREATION_MUD_POINT_COST),
)
.await
}
fn map_puzzle_clear_queue_job_status(
job: ExternalGenerationJobRecord,
) -> ExternalGenerationJobStatusRecord {
@@ -143,6 +143,17 @@ pub async fn generate_editor_sound_effect(
let Json(payload) = parse_json_payload(&request_context, payload)?;
let owner_user_id = authenticated.claims().user_id().to_string();
if !state.config.external_generation_mode.is_inline() {
let pricing = state.editor_generation_pricing().map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_details(json!({
"provider": "editor-generation-pricing",
"message": error.to_string(),
}))
.into_response_with_context(Some(&request_context))
})?;
let normalized =
normalize_editor_sound_effect_request_with_pricing(payload.clone(), &pricing)
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
let source_entity_id = editor_generation_source_entity_id(
payload.project_id.as_deref(),
"editor-sound-effect",
@@ -154,6 +165,7 @@ pub async fn generate_editor_sound_effect(
EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND,
source_entity_id,
"图片画布生成音效",
u64::from(normalized.price_mud_points),
&payload,
)
.await
@@ -306,6 +318,17 @@ pub async fn generate_editor_background_music(
let Json(payload) = parse_json_payload(&request_context, payload)?;
let owner_user_id = authenticated.claims().user_id().to_string();
if !state.config.external_generation_mode.is_inline() {
let pricing = state.editor_generation_pricing().map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_details(json!({
"provider": "editor-generation-pricing",
"message": error.to_string(),
}))
.into_response_with_context(Some(&request_context))
})?;
let normalized =
normalize_editor_background_music_request_with_pricing(payload.clone(), &pricing)
.map_err(|error| error.into_response_with_context(Some(&request_context)))?;
let source_entity_id = editor_generation_source_entity_id(
payload.project_id.as_deref(),
"editor-background-music",
@@ -317,6 +340,7 @@ pub async fn generate_editor_background_music(
EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND,
source_entity_id,
"图片画布生成背景音乐",
u64::from(normalized.price_mud_points),
&payload,
)
.await
@@ -42,6 +42,8 @@ pub(crate) struct WalletRefundOutboxRecord {
pub created_at_micros: i64,
pub asset_kind: String,
pub asset_id: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub external_generation_job_id: Option<String>,
}
#[derive(Debug)]
@@ -163,11 +165,12 @@ impl WalletRefundOutbox {
match self
.spacetime_client
.refund_profile_wallet_points(
.refund_profile_wallet_points_with_metadata(
record.owner_user_id.clone(),
record.amount,
record.ledger_id.clone(),
record.created_at_micros,
refund_metadata_json(record.external_generation_job_id.as_deref()),
)
.await
{
@@ -183,6 +186,7 @@ impl WalletRefundOutbox {
owner_user_id = %record.owner_user_id,
asset_kind = %record.asset_kind,
asset_id = %record.asset_id,
external_generation_job_id = ?record.external_generation_job_id,
path = %path.display(),
"wallet refund outbox 退款已重放并删除文件"
);
@@ -237,6 +241,20 @@ impl WalletRefundOutbox {
}
}
fn refund_metadata_json(external_generation_job_id: Option<&str>) -> String {
let Some(external_generation_job_id) = external_generation_job_id
.map(str::trim)
.filter(|value| !value.is_empty())
else {
return module_runtime::PROFILE_INVITE_CODE_METADATA_DEFAULT_JSON.to_string();
};
serde_json::json!({
"externalGenerationJobId": external_generation_job_id,
})
.to_string()
}
impl fmt::Debug for WalletRefundOutbox {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("WalletRefundOutbox")
@@ -344,6 +362,7 @@ mod tests {
created_at_micros: 1_713_680_000_000_000,
asset_kind: "puzzle_initial_image".to_string(),
asset_id: "asset-1".to_string(),
external_generation_job_id: Some("extgen-test".to_string()),
}
}
@@ -287,6 +287,7 @@ async fn enqueue_wooden_fish_generate_image_assets_job(
) -> Result<ExternalGenerationJobRecord, Response> {
let job_id = build_prefixed_uuid_id("extgen-");
let now_micros = current_utc_micros();
let price_mud_points = resolve_wooden_fish_generation_points_cost(state).await;
let request_payload_json = serde_json::to_string(&WoodenFishGenerateImageAssetsWorkerPayload {
session_id: session_id.to_string(),
owner_user_id: owner_user_id.to_string(),
@@ -316,6 +317,7 @@ async fn enqueue_wooden_fish_generate_image_assets_job(
max_attempts: 1,
available_at_micros: now_micros,
created_at_micros: now_micros,
price_mud_points,
})
.await
.map_err(|error| {
@@ -839,6 +839,7 @@ pub fn build_runtime_profile_wallet_ledger_entry_record(
source_type: snapshot.source_type,
created_at: format_utc_micros(snapshot.created_at_micros),
created_at_micros: snapshot.created_at_micros,
metadata_json: snapshot.metadata_json,
}
}

Some files were not shown because too many files have changed in this diff Show More