diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 1dba0f6e6..9da4bea1f 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -2632,3 +2632,11 @@ - 2026-06-22 补充：所有明确设置绿幕用于后续抠图的 prompt 都必须固定写明 `#00FF00 / RGB(0,255,0)`，不能只写“纯绿色绿幕”或“接近 #00FF00”；编辑器角色图通用抠图额外开启暗绿 / 灰绿绿幕背景识别，只作为生成模型偏离标准亮绿时的兜底。该宽松识别只参与从画布边缘连通扩散出的背景清理，不参与全图断开绿色区域删除，避免误伤角色衣物或纹理。 - 影响范围：`server-rs/crates/api-server/src/editor_project.rs`、`docs/【编辑器】画板角色形象生成入口设计-2026-06-15.md`。 - 验证方式：`cargo test -p api-server --manifest-path server-rs/Cargo.toml editor_character_image_general_cutout`、`cargo test -p platform-image --manifest-path server-rs/Cargo.toml generated_asset_sheet_muted_green_alpha_requires_explicit_option`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`npm run check:encoding`、`git diff --check`。 + +## 2026-06-22 图片画布外部生成统一改走 worker 队列 + +- 背景：图片画布的图片、改图、图标素材、UI 素材提取、角色动作、视频和音频生成都可能长时间等待外部 provider；如果继续由 HTTP handler 同步执行，生产只能扩 API 进程，不能独立扩生成吞吐。 +- 决策：`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 下，画板所有外部 provider 生成入口统一入 `external_generation_job`，job kind 使用 `editor_image_generation`、`editor_image_edit`、`editor_icon_spritesheet_generation`、`editor_ui_design_asset_extraction`、`editor_character_animation_generation`、`editor_video_generation`、`editor_sound_effect_generation` 和 `editor_background_music_generation`。worker 成功后由后端写 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`；前端只轮询 BFF job 状态并重新读取项目快照，不从队列 payload 或本地临时状态重建完成图层。 +- 补充：带 `dialogId` 的 `canvasCompletion` 必须读取后端当前 layout 中的最新 generation dialog placeholder；等待期间用户移动占位时，结果层要跟随最新占位。无 dialog 的重绘 / UI 素材提取等入口使用明确的右侧完成占位；生成器已删除时不把结果重新塞回画布。 +- 影响范围：`server-rs/crates/api-server/src/editor_generation_queue.rs`、`server-rs/crates/api-server/src/external_generation_worker.rs`、`server-rs/crates/api-server/src/editor_project.rs`、`server-rs/crates/api-server/src/character_animation_assets.rs`、`server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs`、`src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts`、`src/services/image-editor/editorProjectClient.ts`。 +- 验证方式：`cargo test -p api-server external_generation_worker --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server editor_canvas_generation --manifest-path server-rs/Cargo.toml`、`cargo test -p shared-contracts --manifest-path server-rs/Cargo.toml`、`npm run test -- src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx src/services/image-editor/editorProjectClient.test.ts`、`npm run typecheck`、`npm run check:encoding`、`git diff --check`。 diff --git a/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md b/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md index d5d6b2b23..709d3c21e 100644 --- a/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md +++ b/docs/technical/【后端架构】外部生成Worker化方案-2026-06-03.md @@ -1,6 +1,6 @@ # 外部生成 Worker 化方案 -更新时间：`2026-06-12` +更新时间：`2026-06-22` ## 背景 @@ -13,7 +13,7 @@ - 多个 worker 进程通过 SpacetimeDB 任务表抢占任务，依赖 lease 超时恢复，支持按进程数和单进程并发动态缩扩容。 - 本地或小流量同步排查可显式启用 `inline` 模式，由 HTTP handler 复用同一 worker executor 同步执行并返回 `completed`；该模式不创建队列任务，也不具备 worker 横向扩容能力。 - SpacetimeDB reducer / procedure 只做任务状态流转，不做网络、文件系统或外部 provider I/O。 -- 已接入拼图 `compile_puzzle_draft`、结果页 `generate_puzzle_images` 与结果页 `generate_puzzle_ui_background`；本轮扩展到跳一跳、拼消消和敲木鱼的外部图片生成动作。后续玩法继续复用同一队列 Module，不再为每个玩法发明独立队列。 +- 已接入拼图 `compile_puzzle_draft`、结果页 `generate_puzzle_images` 与结果页 `generate_puzzle_ui_background`，跳一跳、拼消消和敲木鱼的外部图片生成动作，以及图片画布编辑器的图片、改图、图标 spritesheet、UI 素材提取、角色动作、视频、音效和背景音乐生成。后续玩法和编辑器生成入口继续复用同一队列 Module，不再为每个入口发明独立队列。 - 第一版外部生成队列粒度固定为“单个用户动作对应单个 job”。例如草稿编译、结果页单槽重生、图集重生都各自入一个 job；job 内部可以串行或并行调用 provider、OSS、SpacetimeDB 写回，但不再拆成“提示词 / 生图 / 切图 / 去背景 / 持久化 / 回写”等阶段 job。阶段进度只作为 `request_payload_json` / 业务 session 的展示状态，不作为队列调度单位。 - 不调用外部图片 / 音频 / LLM provider 的动作继续 inline 执行，不为了统一排队而进入 `external_generation_job`。 @@ -48,7 +48,7 @@ BFF 只做鉴权、授权裁剪、字段脱敏和契约映射；队列事实仍 | --- | --- | | `job_id` | 主键，`extgen-` 前缀 UUID | | `dedupe_key` | 唯一键，建议为 `play/action/session/scope` | -| `job_kind` | 执行类型，当前拼图为 `puzzle_compile_draft`、`puzzle_generate_images`、`puzzle_generate_ui_background` | +| `job_kind` | 执行类型，当前覆盖 `puzzle_compile_draft`、`puzzle_generate_images`、`puzzle_generate_ui_background`、跳一跳 / 拼消消 / 敲木鱼生成动作，以及 `editor_image_generation`、`editor_image_edit`、`editor_icon_spritesheet_generation`、`editor_ui_design_asset_extraction`、`editor_character_animation_generation`、`editor_video_generation`、`editor_sound_effect_generation`、`editor_background_music_generation` | | `owner_user_id` | 触发用户 | | `source_module` | 玩法或能力名，例如 `puzzle` | | `source_entity_id` | session/profile/work 等作用域 | @@ -161,6 +161,20 @@ controller 配置： 每个玩法迁移时必须同时接入业务写回 lease guard：worker 路径带 `external_generation_job_id / worker_id / lease_token`，inline 路径三项同时为空。过期 worker 不得写 session / work profile；业务失败态写回成功后才允许 job 进入 `failed`。 +### 图片画布编辑器 + +图片画布 `/editor/canvas` 下所有会调用外部生成 provider 的入口在 `queue` 模式下入 `external_generation_job`，HTTP handler 只返回 `queueState`： + +- `editor_image_generation`：普通图片、生成规范、角色形象、UI 设计图、宣发素材和图片快速编辑。 +- `editor_image_edit`：图片编辑 / 修改结果。 +- `editor_icon_spritesheet_generation`：图标素材 spritesheet 生成和拆分。 +- `editor_ui_design_asset_extraction`：UI 设计图红框素材提取。 +- `editor_character_animation_generation`：角色动作视频和帧素材生成。 +- `editor_video_generation`：画布视频生成和视频素材快速编辑。 +- `editor_sound_effect_generation` / `editor_background_music_generation`：画布音效与背景音乐。 + +画板结果的业务真相仍是 `editor_project_resource`、账号级 `editor_asset` 和 `editor_canvas.layers_json`。请求携带 `projectId + canvasCompletion` 时，worker 成功后读取当前项目 layout，用最新 generation dialog placeholder 或无 dialog 完成占位写入结果图层，并保存项目快照；前端轮询单 job 到 completed 后重新读取项目快照，不从队列 payload 或本地临时响应重建正式图层。生成器已被删除时，worker 只保留生成出的资源 / 素材记录，不把结果重新塞回画布。 + ## 验收 基础检查： diff --git a/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md b/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md index 529aa8dff..b89419150 100644 --- a/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md +++ b/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md @@ -245,7 +245,7 @@ npm run check:server-rs-ddd - Rust 结构体：`ExternalGenerationJob` - 源码：`server-rs/crates/spacetime-module/src/external_generation.rs` -- 用途：外部生成 worker 的持久任务队列；`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时，`api-server` HTTP 角色只入队，`external-generation-worker` 角色通过 claim lease 领取、续租、执行，并用 `lease_token` 栅栏回写完成 / 失败。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft`、`generate_puzzle_images` 与 `generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，避免过期 worker 写 session / work profile；`GENARRATIVE_EXTERNAL_GENERATION_MODE=inline` 时不创建该队列行，三个 external generation guard 字段必须同时为空才允许 api-server 受控同步写回，半空 guard 仍会拒绝。worker 成功写回业务事实后才能 complete job；业务失败态写回成功后才能 fail job，失败态未写回时保留租约等待后续重领。 +- 用途：外部生成 worker 的持久任务队列；`GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时，`api-server` HTTP 角色只入队，`external-generation-worker` 角色通过 claim lease 领取、续租、执行，并用 `lease_token` 栅栏回写完成 / 失败。拼图 `compile_puzzle_draft` 的前置 `compile_puzzle_agent_draft`、`generate_puzzle_images` 与 `generate_puzzle_ui_background` 的业务写回也在对应 SpacetimeDB transaction 内校验 `job_id + worker_id + lease_token`、job kind、owner 和 source entity，避免过期 worker 写 session / work profile；图片画布编辑器的 `editor_image_generation`、`editor_image_edit`、`editor_icon_spritesheet_generation`、`editor_ui_design_asset_extraction`、`editor_character_animation_generation`、`editor_video_generation`、`editor_sound_effect_generation` 和 `editor_background_music_generation` 复用同一队列表，worker 成功后经 `api-server` facade 写入 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`，前端只通过 BFF job 状态轮询和项目快照读取恢复完成态。`GENARRATIVE_EXTERNAL_GENERATION_MODE=inline` 时不创建该队列行，三个 external generation guard 字段必须同时为空才允许 api-server 受控同步写回，半空 guard 仍会拒绝。worker 成功写回业务事实后才能 complete job；业务失败态写回成功后才能 fail job，失败态未写回时保留租约等待后续重领。 ### `ai_text_chunk` diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index ac17e3b2d..6cffa142c 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -57,7 +57,7 @@ Windows 本地如果已在 `%LOCALAPPDATA%\Genarrative\ffmpeg\bin` 安装 FFmpeg 本地 `npm run dev` 和 `npm run dev:api-server` 默认保留 inline 开发体验：未显式设置 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue` 时，外部生成 handler 会同步复用 worker executor，完成后返回 `completed`，便于快速确认 provider、OSS 和 SpacetimeDB 写回链路。inline 不创建 `external_generation_job`，也不能验证 worker lease、队列等待展示或动态扩缩容。 -本地排查外部内容生成 worker 队列时，必须显式使用 queue，例如 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue GENARRATIVE_PROCESS_ROLE=all npm run dev:api-server`，让同一 Rust 进程同时监听 HTTP 并消费 `external_generation_job` 队列；更接近生产的验证应分别启动 `api`、`external-generation-worker` 和 `external-generation-controller`。生产默认 `GENARRATIVE_PROCESS_ROLE=api`，外部生成任务由独立 `GENARRATIVE_PROCESS_ROLE=external-generation-worker` 进程消费；生产与容器扩缩容验证保持 `queue`。当前进入持久队列的外部图片生成动作包括：拼图 `compile_puzzle_draft` / `generate_puzzle_images` / `generate_puzzle_ui_background`，跳一跳 `compile-draft` / `regenerate-tiles`，拼消消 `compile-draft` / `regenerate-atlas`，敲木鱼 `compile-draft` / `regenerate-hit-object`。非外部图片生成动作继续 inline，不进入队列。worker 数量为 0 时，HTTP 只返回 queued/running，不会兜底执行外部 provider。 +本地排查外部内容生成 worker 队列时，必须显式使用 queue，例如 `GENARRATIVE_EXTERNAL_GENERATION_MODE=queue GENARRATIVE_PROCESS_ROLE=all npm run dev:api-server`，让同一 Rust 进程同时监听 HTTP 并消费 `external_generation_job` 队列；更接近生产的验证应分别启动 `api`、`external-generation-worker` 和 `external-generation-controller`。生产默认 `GENARRATIVE_PROCESS_ROLE=api`，外部生成任务由独立 `GENARRATIVE_PROCESS_ROLE=external-generation-worker` 进程消费；生产与容器扩缩容验证保持 `queue`。当前进入持久队列的外部生成动作包括：拼图 `compile_puzzle_draft` / `generate_puzzle_images` / `generate_puzzle_ui_background`，跳一跳 `compile-draft` / `regenerate-tiles`，拼消消 `compile-draft` / `regenerate-atlas`，敲木鱼 `compile-draft` / `regenerate-hit-object`，以及图片画布 `editor_image_generation` / `editor_image_edit` / `editor_icon_spritesheet_generation` / `editor_ui_design_asset_extraction` / `editor_character_animation_generation` / `editor_video_generation` / `editor_sound_effect_generation` / `editor_background_music_generation`。非外部 provider 生成动作继续 inline，不进入队列。worker 数量为 0 时，HTTP 只返回 queued/running，不会兜底执行外部 provider。 `我的` 页签或排障面板展示队列等待时，只读取 BFF 队列接口：`GET /api/runtime/external-generation/queue-overview` 查看当前用户可见队列概览，`GET /api/runtime/external-generation/jobs/{jobId}` 查看单 job 状态。生成页 / 进度页不承接队列概览，只展示当前玩法业务进度；队列接口只提供等待 / 运行 / 失败 / 完成状态补充，最终草稿、作品和结果页仍要轮询对应玩法 session/detail 接口收敛到 ready 或 failed；不要直接查询 `external_generation_job` private table，也不要把 worker 内部 payload 暴露到前端。 @@ -387,7 +387,7 @@ Pingora current release 自审脚本 `scripts/ops/pingora-current-release-audit. 生产环境变量模板：`deploy/env/api-server.env.example`。真实密钥只放服务器，不提交 Git，不写入文档示例。 -`api-server` 进程角色由 `GENARRATIVE_PROCESS_ROLE` 控制：`api` 只监听 HTTP，`external-generation-worker` 只消费外部生成队列，`external-generation-controller` 只管理 worker systemd 实例，`all` 仅用于本地或临时 smoke，不隐式启动 controller。外部生成策略由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制；生产和容器压测默认保持 `queue`，本地 `npm run dev` 默认保留 `inline` 开发体验，只有显式配置 `queue` 才会落 `external_generation_job`。`inline` 只用于本地或低并发同步排查，HTTP handler 会直接复用 worker executor，完成后返回 `completed`，但不会落 `external_generation_job`，也不能通过增加 worker 进程扩吞吐。外部生成 worker 使用同一发布包和同一套 SpacetimeDB 配置，按实例数和 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩；生产默认由 `genarrative-external-generation-controller.service` 读取 `get_external_generation_queue_stats_and_return`，按 `claimable_pending + running_active + expired_running` 计算目标 worker 数，并对 `genarrative-external-generation-worker@N.service` 精确执行 `systemctl start/stop`。controller 参数模板是 `deploy/env/external-generation-controller.env.example`：默认保底 `MIN_WORKERS=1`、上限 `MAX_WORKERS=8`、每 worker 目标 `TARGET_JOBS_PER_WORKER=2`、`POLL_INTERVAL_MS=10000`、连续 `SCALE_DOWN_IDLE_ROUNDS=6` 轮完全空闲才缩容；缩容每轮只停止最高编号的一个实例，且不主动停止 `@1`。worker 收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成；若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`，未完成任务才会在 lease 过期后由其它 worker 重领。每个 worker 实例应设置唯一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`，默认会用主机名和 pid 兜底；systemd 生产模板 `deploy/systemd/genarrative-external-generation-worker@.service` 会用 `%H-%i` 生成实例 ID，并把 tracking outbox 隔离到 `/var/lib/genarrative/tracking-outbox/%H-%i`。`Genarrative-Server-Provision` 会安装 worker 模板、controller unit 和两份专属 env 模板，默认 enable 首个 `genarrative-external-generation-worker@1.service` 与 `genarrative-external-generation-controller.service`；首次 API deploy 会在默认 worker pattern 下自动 `enable --now genarrative-external-generation-worker@1.service` 并等待 worker active，同时重启并验活 controller。手动兜底扩容仍可用 `systemctl start genarrative-external-generation-worker@2.service` / `@3.service`，缩容用 `systemctl stop genarrative-external-generation-worker@N.service`；controller 下轮会按队列压力修正到目标实例数。worker 专属参数模板是 `deploy/env/external-generation-worker.env.example`，密钥与 SpacetimeDB 连接仍复用 `/etc/genarrative/api-server.env`。API 发布脚本默认会重启并验活 `genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service`；若本次只发 HTTP 且不希望滚动 worker，可传 `--no-worker-services`，若不希望重启 controller 可传 `--no-worker-controller`。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 控制空队列轮询间隔，`GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS` 控制单次 lease，worker 会约每三分之一 lease、最长 30 秒续租；该值应覆盖一次心跳网络抖动窗口，不需要大于完整外部生成链路耗时。SpacetimeDB 使用自身事务时间计算 claim/renew/complete/fail，完成和失败回写还会校验 `lease_token` 与未过期 lease，避免同一 job 被过期 worker 覆盖。首版 worker 粒度是单动作单 job，不拆阶段 job；当前外部图片生成动作覆盖拼图、跳一跳、拼消消和敲木鱼，纯元信息保存、发布、试玩启动、运行态动作和公开读取继续 inline。当前生成业务失败只做用户重新触发，不做自动业务重试，避免 worker 退款和重试成功之间产生钱包账本漂移。 +`api-server` 进程角色由 `GENARRATIVE_PROCESS_ROLE` 控制：`api` 只监听 HTTP，`external-generation-worker` 只消费外部生成队列，`external-generation-controller` 只管理 worker systemd 实例，`all` 仅用于本地或临时 smoke，不隐式启动 controller。外部生成策略由 `GENARRATIVE_EXTERNAL_GENERATION_MODE` 控制；生产和容器压测默认保持 `queue`，本地 `npm run dev` 默认保留 `inline` 开发体验，只有显式配置 `queue` 才会落 `external_generation_job`。`inline` 只用于本地或低并发同步排查，HTTP handler 会直接复用 worker executor，完成后返回 `completed`，但不会落 `external_generation_job`，也不能通过增加 worker 进程扩吞吐。外部生成 worker 使用同一发布包和同一套 SpacetimeDB 配置，按实例数和 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_CONCURRENCY` 动态扩缩；生产默认由 `genarrative-external-generation-controller.service` 读取 `get_external_generation_queue_stats_and_return`，按 `claimable_pending + running_active + expired_running` 计算目标 worker 数，并对 `genarrative-external-generation-worker@N.service` 精确执行 `systemctl start/stop`。controller 参数模板是 `deploy/env/external-generation-controller.env.example`：默认保底 `MIN_WORKERS=1`、上限 `MAX_WORKERS=8`、每 worker 目标 `TARGET_JOBS_PER_WORKER=2`、`POLL_INTERVAL_MS=10000`、连续 `SCALE_DOWN_IDLE_ROUNDS=6` 轮完全空闲才缩容；缩容每轮只停止最高编号的一个实例，且不主动停止 `@1`。worker 收到 SIGINT/SIGTERM 后会停止 claim 新任务并等待当前任务完成；若进程被硬杀、机器断电或超过 systemd `TimeoutStopSec`，未完成任务才会在 lease 过期后由其它 worker 重领。每个 worker 实例应设置唯一 `GENARRATIVE_EXTERNAL_GENERATION_WORKER_ID`，默认会用主机名和 pid 兜底；systemd 生产模板 `deploy/systemd/genarrative-external-generation-worker@.service` 会用 `%H-%i` 生成实例 ID，并把 tracking outbox 隔离到 `/var/lib/genarrative/tracking-outbox/%H-%i`。`Genarrative-Server-Provision` 会安装 worker 模板、controller unit 和两份专属 env 模板，默认 enable 首个 `genarrative-external-generation-worker@1.service` 与 `genarrative-external-generation-controller.service`；首次 API deploy 会在默认 worker pattern 下自动 `enable --now genarrative-external-generation-worker@1.service` 并等待 worker active，同时重启并验活 controller。手动兜底扩容仍可用 `systemctl start genarrative-external-generation-worker@2.service` / `@3.service`，缩容用 `systemctl stop genarrative-external-generation-worker@N.service`；controller 下轮会按队列压力修正到目标实例数。worker 专属参数模板是 `deploy/env/external-generation-worker.env.example`，密钥与 SpacetimeDB 连接仍复用 `/etc/genarrative/api-server.env`。API 发布脚本默认会重启并验活 `genarrative-external-generation-worker@*.service` 和 `genarrative-external-generation-controller.service`；若本次只发 HTTP 且不希望滚动 worker，可传 `--no-worker-services`，若不希望重启 controller 可传 `--no-worker-controller`。`GENARRATIVE_EXTERNAL_GENERATION_WORKER_POLL_INTERVAL_MS` 控制空队列轮询间隔，`GENARRATIVE_EXTERNAL_GENERATION_WORKER_LEASE_SECONDS` 控制单次 lease，worker 会约每三分之一 lease、最长 30 秒续租；该值应覆盖一次心跳网络抖动窗口，不需要大于完整外部生成链路耗时。SpacetimeDB 使用自身事务时间计算 claim/renew/complete/fail，完成和失败回写还会校验 `lease_token` 与未过期 lease，避免同一 job 被过期 worker 覆盖。首版 worker 粒度是单动作单 job，不拆阶段 job；当前外部生成动作覆盖拼图、跳一跳、拼消消、敲木鱼和图片画布编辑器生成入口，纯元信息保存、发布、试玩启动、运行态动作和公开读取继续 inline。图片画布 worker 成功后由后端保存 `editor_project_resource` / `editor_asset` / `editor_canvas.layers_json`，前端只轮询 job 并重新读取项目快照。当前生成业务失败只做用户重新触发，不做自动业务重试，避免 worker 退款和重试成功之间产生钱包账本漂移。 `Genarrative-Server-Provision` 会安装 systemd 模板和 Nginx 站点模板，不再安装 clang / lld / pkg-config / OpenSSL headers / sccache 等通用构建链依赖。因 VectorEngine 图片上游 POST 已改用 `libcurl`，当前 Linux release 构建出的 `api-server` 运行时需要 `OPENSSL_3.2.0` 符号；Ubuntu 24.04 apt 默认只提供 OpenSSL 3.0.x，不能直接满足该符号版本。Provision 会把 OpenSSL `3.2.0` 独立安装到 `/opt/genarrative/openssl-3.2.0`，校验官方 tarball SHA256，并只通过 `genarrative-api.service` 的 `LD_LIBRARY_PATH=/opt/genarrative/openssl-3.2.0/lib64:/opt/genarrative/openssl-3.2.0/lib` 让 api-server 使用，避免替换系统 OpenSSL 或影响 ssh / nginx / apt。Ubuntu / apt 目标机为完成这一步会安装 `build-essential`、`ca-certificates`、`curl`、`perl`、`tar` 等 OpenSSL 运行时自举工具；这只服务于独立 OpenSSL 运行时安装，不代表 provision 重新承担 api-server 构建职责。Ubuntu / apt 目标机会额外安装 `libnginx-mod-http-brotli-filter` 与 `libnginx-mod-http-brotli-static`，随后由 `scripts/jenkins-server-provision.sh` 通过临时 `nginx -t` 配置探测 Brotli 指令是否可用；该临时配置必须先 `include /etc/nginx/modules-enabled/*.conf`，因为 apt 安装的 Brotli 是动态模块，不会出现在普通 `nginx -V` 编译参数里。探测成功才在渲染后的 `deploy/nginx/genarrative.conf` / `genarrative-dev-http.conf` 中启用 Brotli，避免未安装模块的机器直接写入无效配置。Provision 写入 Genarrative Nginx 站点时会把 `/etc/nginx/sites-enabled/default*` 移到 `/etc/nginx/sites-disabled/`，避免 Debian / Certbot 默认站点继续占用 `genarrative.world` / `www.genarrative.world` 并在 `nginx -T` 中出现 `conflicting server name ... ignored`。如果 `nginx -t` 失败，脚本会恢复写入前的 Genarrative 配置和被移动的默认站点。 diff --git a/server-rs/crates/api-server/src/app.rs b/server-rs/crates/api-server/src/app.rs index 55e3d04d7..e86aabf9a 100644 --- a/server-rs/crates/api-server/src/app.rs +++ b/server-rs/crates/api-server/src/app.rs @@ -283,7 +283,10 @@ mod tests { use tokio::net::TcpListener; use tower::ServiceExt; - use crate::{config::AppConfig, state::AppState}; + use crate::{ + config::{AppConfig, ExternalGenerationMode}, + state::AppState, + }; use super::{build_router, build_spacetime_unavailable_router}; @@ -1472,7 +1475,11 @@ mod tests { #[tokio::test] async fn editor_character_animation_accepts_character_image_body_above_default_limit() { - let state = AppState::new(AppConfig::default()).expect("state should build"); + let state = AppState::new(AppConfig { + external_generation_mode: ExternalGenerationMode::Inline, + ..AppConfig::default() + }) + .expect("state should build"); let seed_user = seed_phone_user_with_password(&state, "13800138190", TEST_PASSWORD).await; let token = sign_test_user_token(&state, &seed_user, "sess_editor_character_body"); let app = build_router(state); @@ -1554,7 +1561,11 @@ mod tests { #[tokio::test] async fn editor_ui_design_asset_extraction_accepts_image_body_above_default_limit() { - let state = AppState::new(AppConfig::default()).expect("state should build"); + let state = AppState::new(AppConfig { + external_generation_mode: ExternalGenerationMode::Inline, + ..AppConfig::default() + }) + .expect("state should build"); let seed_user = seed_phone_user_with_password(&state, "13800138188", TEST_PASSWORD).await; let token = sign_test_user_token(&state, &seed_user, "sess_editor_ui_extract_body"); let app = build_router(state); @@ -1599,7 +1610,11 @@ mod tests { #[tokio::test] async fn editor_image_generation_accepts_reference_body_above_default_limit() { - let state = AppState::new(AppConfig::default()).expect("state should build"); + let state = AppState::new(AppConfig { + external_generation_mode: ExternalGenerationMode::Inline, + ..AppConfig::default() + }) + .expect("state should build"); let seed_user = seed_phone_user_with_password(&state, "13800138028", TEST_PASSWORD).await; let token = sign_test_user_token(&state, &seed_user, "sess_editor_image_reference_body"); let app = build_router(state); diff --git a/server-rs/crates/api-server/src/character_animation_assets.rs b/server-rs/crates/api-server/src/character_animation_assets.rs index 3b552054d..fc3a325d3 100644 --- a/server-rs/crates/api-server/src/character_animation_assets.rs +++ b/server-rs/crates/api-server/src/character_animation_assets.rs @@ -1,4 +1,4 @@ -use std::{ +use std::{ collections::BTreeMap, fs, path::{Path, PathBuf}, @@ -55,6 +55,15 @@ use crate::{ build_character_animation_prompt, build_fallback_moderation_safe_animation_prompt, }, editor_generation_config::EditorGenerationPricingConfig, + editor_generation_queue::{ + EDITOR_CHARACTER_ANIMATION_GENERATION_JOB_KIND, EDITOR_VIDEO_GENERATION_JOB_KIND, + EditorGenerationQueuedResponse, editor_generation_queue_state, + editor_generation_source_entity_id, enqueue_editor_generation_job, + }, + editor_project::{ + EditorCanvasGeneratedLayerInput, build_editor_canvas_generated_layer_item, + complete_editor_canvas_generation_with_items, + }, http_error::AppError, platform_errors::map_oss_error, prompt::role_asset_studio::{ @@ -525,11 +534,44 @@ pub async fn generate_editor_character_animation( Extension(authenticated): Extension, payload: Result, JsonRejection>, ) -> Result, Response> { + let Json(payload) = payload.map_err(|error| { + character_animation_error_response( + &request_context, + AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ + "provider": "editor-character-animation", + "message": error.body_text(), + })), + ) + })?; + let owner_user_id = authenticated.claims().user_id().to_string(); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = editor_generation_source_entity_id( + payload.project_id.as_deref(), + payload.source_layer_id.as_str(), + ); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + owner_user_id.as_str(), + EDITOR_CHARACTER_ANIMATION_GENERATION_JOB_KIND, + source_entity_id, + "图片画布生成角色动作", + &payload, + ) + .await + .map_err(|error| error.into_response_with_context(Some(&request_context)))?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } generate_editor_character_animation_for_owner( state, request_context, - authenticated.claims().user_id().to_string(), - payload, + owner_user_id, + Ok(Json(payload)), ) .await } @@ -549,6 +591,10 @@ pub(crate) async fn generate_editor_character_animation_for_owner( })), ) })?; + let project_id = payload.project_id.clone(); + let canvas_completion = payload.canvas_completion.clone(); + let generation_inputs = payload.generation_inputs.clone(); + let source_resource_id = payload.source_resource_id.clone(); let pricing = state.editor_generation_pricing().map_err(|error| { character_animation_error_response( @@ -613,6 +659,68 @@ pub(crate) async fn generate_editor_character_animation_for_owner( let (generated, frames) = result.map_err(|error| character_animation_error_response(&request_context, error))?; + let completed_project = if let Some(completion) = canvas_completion.as_ref() { + let frame_payload_json = serde_json::to_value(&frames).map_err(|error| { + character_animation_error_response( + &request_context, + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ + "provider": "editor-character-animation", + "message": format!("角色动作序列帧无法写入画布：{error}"), + })), + ) + })?; + let preview_frame = frames.first(); + let layer_id = format!("layer-editor-character-animation-{task_id}"); + let item = build_editor_canvas_generated_layer_item( + completion, + EditorCanvasGeneratedLayerInput { + layer_id: layer_id.clone(), + resource_id: format!("local-resource-editor-character-animation-{task_id}"), + title: completion.title.trim().to_string(), + src: preview_frame + .map(|frame| frame.image_src.clone()) + .unwrap_or_else(|| generated.preview_video_path.clone()), + media_type: Some("image-sequence".to_string()), + thumbnail_src: preview_frame.map(|frame| frame.image_src.clone()), + original_width: preview_frame + .map(|frame| frame.width) + .unwrap_or(normalized.frame_width), + original_height: preview_frame + .map(|frame| frame.height) + .unwrap_or(normalized.frame_height), + display_width: completion.placeholder.width, + display_height: completion.placeholder.height, + source_type: "generated".to_string(), + prompt: Some(generated.submitted_prompt.clone()), + actual_prompt: Some(generated.submitted_prompt.clone()), + model: Some(EDITOR_CHARACTER_ANIMATION_MODEL.to_string()), + provider: Some("ark".to_string()), + task_id: Some(task_id.clone()), + object_key: None, + asset_object_id: None, + duration_seconds: Some(normalized.duration_seconds), + source_resource_id, + asset_kind: Some("character-animation".to_string()), + generation_inputs, + image_sequence_frames: Some(frame_payload_json), + preview_video_path: Some(generated.preview_video_path.clone()), + }, + ); + complete_editor_canvas_generation_with_items( + &state, + owner_user_id.as_str(), + project_id.as_deref(), + canvas_completion.as_ref(), + vec![item], + Some(layer_id), + ) + .await + .map_err(|error| character_animation_error_response(&request_context, error))? + } else { + None + }; + let completed_project = + completed_project.and_then(|project| serde_json::to_value(project).ok()); Ok(json_success_body( Some(&request_context), @@ -629,6 +737,8 @@ pub(crate) async fn generate_editor_character_animation_for_owner( fps: normalized.fps, price_mud_points: normalized.price_mud_points, frames, + project: completed_project, + queue_state: None, }, )) } @@ -639,13 +749,38 @@ pub async fn generate_editor_video( Extension(authenticated): Extension, payload: Result, JsonRejection>, ) -> Result, Response> { - generate_editor_video_for_owner( - state, - request_context, - authenticated.claims().user_id().to_string(), - payload, - ) - .await + let Json(payload) = payload.map_err(|error| { + editor_video_error_response( + &request_context, + AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ + "provider": "editor-video", + "message": error.body_text(), + })), + ) + })?; + let owner_user_id = authenticated.claims().user_id().to_string(); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = + editor_generation_source_entity_id(payload.project_id.as_deref(), "editor-video"); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + owner_user_id.as_str(), + EDITOR_VIDEO_GENERATION_JOB_KIND, + source_entity_id, + "图片画布生成视频", + &payload, + ) + .await + .map_err(|error| error.into_response_with_context(Some(&request_context)))?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } + generate_editor_video_for_owner(state, request_context, owner_user_id, Ok(Json(payload))).await } pub(crate) async fn generate_editor_video_for_owner( @@ -663,6 +798,11 @@ pub(crate) async fn generate_editor_video_for_owner( })), ) })?; + let project_id = payload.project_id.clone(); + let canvas_completion = payload.canvas_completion.clone(); + let generation_inputs = payload.generation_inputs.clone(); + let source_resource_id = payload.source_resource_id.clone(); + let asset_kind = payload.asset_kind.clone(); let pricing = state.editor_generation_pricing().map_err(|error| { editor_video_error_response( @@ -698,6 +838,52 @@ pub(crate) async fn generate_editor_video_for_owner( ) .await .map_err(|error| editor_video_error_response(&request_context, error))?; + let completed_project = if let Some(completion) = canvas_completion.as_ref() { + let layer_id = format!("layer-editor-video-{task_id}"); + let item = build_editor_canvas_generated_layer_item( + completion, + EditorCanvasGeneratedLayerInput { + layer_id: layer_id.clone(), + resource_id: format!("local-resource-editor-video-{task_id}"), + title: completion.title.trim().to_string(), + src: generated.preview_video_path.clone(), + media_type: Some("video".to_string()), + thumbnail_src: None, + original_width: normalized.width, + original_height: normalized.height, + display_width: completion.placeholder.width, + display_height: completion.placeholder.height, + source_type: "generated".to_string(), + prompt: Some(normalized.prompt.clone()), + actual_prompt: Some(generated.submitted_prompt.clone()), + model: Some(normalized.model.clone()), + provider: Some("VectorEngine".to_string()), + task_id: Some(task_id.clone()), + object_key: generated.object_key.clone(), + asset_object_id: generated.asset_object_id.clone(), + duration_seconds: Some(normalized.duration_seconds), + source_resource_id, + asset_kind: Some(asset_kind.unwrap_or_else(|| "video".to_string())), + generation_inputs, + image_sequence_frames: None, + preview_video_path: None, + }, + ); + complete_editor_canvas_generation_with_items( + &state, + owner_user_id.as_str(), + project_id.as_deref(), + canvas_completion.as_ref(), + vec![item], + Some(layer_id), + ) + .await + .map_err(|error| editor_video_error_response(&request_context, error))? + } else { + None + }; + let completed_project = + completed_project.and_then(|project| serde_json::to_value(project).ok()); Ok(json_success_body( Some(&request_context), @@ -717,6 +903,8 @@ pub(crate) async fn generate_editor_video_for_owner( duration_seconds: normalized.duration_seconds, resolution: normalized.resolution, price_mud_points: normalized.price_mud_points, + project: completed_project, + queue_state: None, }, )) } @@ -4909,6 +5097,10 @@ mod tests { frame_count: 48, duration_seconds: 6, model: EDITOR_CHARACTER_ANIMATION_MODEL.to_string(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, }) .expect("editor request should normalize"); @@ -4943,6 +5135,10 @@ mod tests { frame_count: 32, duration_seconds: 4, model: EDITOR_CHARACTER_ANIMATION_MODEL.to_string(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, }) .expect("editor request should normalize"); @@ -4966,6 +5162,10 @@ mod tests { frame_count: 48, duration_seconds: 4, model: EDITOR_CHARACTER_ANIMATION_MODEL.to_string(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, }) .expect_err("invalid frame/duration pair should fail"); @@ -5121,6 +5321,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("editor video request should normalize"); @@ -5146,6 +5351,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("seedance fast request should normalize"); @@ -5168,6 +5378,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("seedance standard request should normalize"); @@ -5190,6 +5405,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("seedance standard 1080p request should normalize"); @@ -5213,6 +5433,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect_err("seedance fast should reject 1080p"); @@ -5235,6 +5460,11 @@ mod tests { "generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(), ], reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()], + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("seedance multimodal request should normalize"); @@ -5266,6 +5496,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: vec!["data:video/mp4;base64,video".to_string()], reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect_err("Seedance reference videos must not accept Base64 data URLs"); @@ -5287,6 +5522,11 @@ mod tests { reference_image_srcs: vec!["data:image/png;base64,image".to_string()], reference_video_srcs: vec!["https://assets.example.com/reference.mp4".to_string()], reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()], + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("seedance multimodal request should normalize"); @@ -5328,6 +5568,11 @@ mod tests { "generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(), ], reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("seedance objectKey request should normalize"); @@ -5363,6 +5608,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: vec!["data:audio/wav;base64,audio".to_string()], + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect_err("audio-only references should fail"); @@ -5404,7 +5654,6 @@ mod tests { ); } - fn assert_function_contains(source: &str, start: &str, end: &str, snippets: &[&str]) { let start_index = source .find(start) diff --git a/server-rs/crates/api-server/src/editor_generation_queue.rs b/server-rs/crates/api-server/src/editor_generation_queue.rs new file mode 100644 index 000000000..08355bdc6 --- /dev/null +++ b/server-rs/crates/api-server/src/editor_generation_queue.rs @@ -0,0 +1,105 @@ +use axum::http::StatusCode; +use serde::Serialize; +use serde_json::json; +use shared_contracts::external_generation::{ + ExternalGenerationJobStatus, ExternalGenerationJobStatusRecord, +}; +use shared_kernel::{build_prefixed_uuid_id, offset_datetime_to_unix_micros}; +use spacetime_client::{ExternalGenerationJobEnqueueRecordInput, ExternalGenerationJobRecord}; + +use crate::{http_error::AppError, request_context::RequestContext, state::AppState}; + +pub(crate) const EDITOR_IMAGE_GENERATION_JOB_KIND: &str = "editor_image_generation"; +pub(crate) const EDITOR_IMAGE_EDIT_JOB_KIND: &str = "editor_image_edit"; +pub(crate) const EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND: &str = + "editor_icon_spritesheet_generation"; +pub(crate) const EDITOR_UI_DESIGN_ASSET_EXTRACTION_JOB_KIND: &str = + "editor_ui_design_asset_extraction"; +pub(crate) const EDITOR_CHARACTER_ANIMATION_GENERATION_JOB_KIND: &str = + "editor_character_animation_generation"; +pub(crate) const EDITOR_VIDEO_GENERATION_JOB_KIND: &str = "editor_video_generation"; +pub(crate) const EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND: &str = "editor_sound_effect_generation"; +pub(crate) const EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND: &str = + "editor_background_music_generation"; + +pub(crate) const EDITOR_GENERATION_QUEUE_SOURCE_MODULE: &str = "editor-canvas"; +const EDITOR_GENERATION_QUEUE_PROVIDER: &str = "editor-generation-worker"; + +#[derive(Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub(crate) struct EditorGenerationQueuedResponse { + pub(crate) queue_state: ExternalGenerationJobStatusRecord, +} + +pub(crate) async fn enqueue_editor_generation_job( + state: &AppState, + _request_context: &RequestContext, + owner_user_id: &str, + job_kind: &str, + source_entity_id: impl Into, + request_label: impl Into, + payload: &T, +) -> Result +where + T: Serialize, +{ + let job_id = build_prefixed_uuid_id("extgen-"); + let request_payload_json = serde_json::to_string(payload).map_err(|error| { + AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({ + "provider": EDITOR_GENERATION_QUEUE_PROVIDER, + "message": format!("编辑器 worker 任务参数序列化失败：{error}"), + })) + })?; + let now_micros = current_utc_micros(); + state + .spacetime_client() + .enqueue_external_generation_job(ExternalGenerationJobEnqueueRecordInput { + dedupe_key: format!("editor-canvas:{job_kind}:{job_id}"), + job_id, + job_kind: job_kind.to_string(), + owner_user_id: owner_user_id.to_string(), + source_module: EDITOR_GENERATION_QUEUE_SOURCE_MODULE.to_string(), + source_entity_id: source_entity_id.into(), + request_label: request_label.into(), + request_payload_json, + max_attempts: 1, + available_at_micros: now_micros, + created_at_micros: now_micros, + }) + .await + .map_err(|error| { + AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({ + "provider": EDITOR_GENERATION_QUEUE_PROVIDER, + "message": error.to_string(), + })) + }) +} + +pub(crate) fn editor_generation_queue_state( + job: ExternalGenerationJobRecord, +) -> ExternalGenerationJobStatusRecord { + ExternalGenerationJobStatusRecord { + operation_id: job.job_id, + status: ExternalGenerationJobStatus::Queued, + phase_label: job.request_label, + phase_detail: "排队中。".to_string(), + progress: 8, + error: job.last_error_message, + updated_at_micros: job.updated_at_micros, + } +} + +pub(crate) fn editor_generation_source_entity_id( + project_id: Option<&str>, + fallback: &str, +) -> String { + project_id + .map(str::trim) + .filter(|value| !value.is_empty()) + .unwrap_or(fallback) + .to_string() +} + +fn current_utc_micros() -> i64 { + offset_datetime_to_unix_micros(time::OffsetDateTime::now_utc()) +} diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index 6c7e306ed..c4f1a85ed 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -22,6 +22,10 @@ use platform_image::{ use platform_oss::{LegacyAssetPrefix, OssHeadObjectRequest, OssObjectAccess}; use serde::{Deserialize, Serialize}; use serde_json::{Value, json}; +use shared_contracts::assets::{ + EditorCanvasGenerationCompletionPayload as EditorCanvasGenerationCompletionRequest, + EditorCanvasGenerationPlaceholderPayload, +}; use shared_kernel::build_prefixed_uuid_id; use spacetime_client::{ EditorAssetCreateRecordInput, EditorAssetDeleteRecordInput, EditorAssetFolderCreateRecordInput, @@ -37,6 +41,12 @@ use crate::{ api_response::json_success_body, asset_billing::execute_billable_asset_operation_with_cost, auth::AuthenticatedAccessToken, + editor_generation_queue::{ + EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, EDITOR_IMAGE_EDIT_JOB_KIND, + EDITOR_IMAGE_GENERATION_JOB_KIND, EDITOR_UI_DESIGN_ASSET_EXTRACTION_JOB_KIND, + EditorGenerationQueuedResponse, editor_generation_queue_state, + editor_generation_source_entity_id, enqueue_editor_generation_job, + }, generated_image_assets::{ GeneratedImageAssetAdapter, GeneratedImageAssetDataUrl, adapter::{GeneratedImageAssetAdapterMetadata, GeneratedImageAssetPersistInput}, @@ -162,7 +172,7 @@ pub struct EditorAssetUpdateRequest { folder_id: Option, } -#[derive(Debug, Deserialize)] +#[derive(Clone, Debug, Deserialize, Serialize)] #[serde(rename_all = "camelCase")] pub struct EditorImageGenerationRequest { pub(crate) prompt: String, @@ -181,26 +191,7 @@ pub struct EditorImageGenerationRequest { pub(crate) canvas_completion: Option, } -#[derive(Clone, Debug, Deserialize)] -#[serde(rename_all = "camelCase")] -pub struct EditorCanvasGenerationCompletionRequest { - pub(crate) dialog_id: String, - pub(crate) title: String, - pub(crate) placeholder: EditorCanvasGenerationPlaceholderPayload, -} - -#[derive(Clone, Debug, Deserialize)] -#[serde(rename_all = "camelCase")] -pub struct EditorCanvasGenerationPlaceholderPayload { - pub(crate) x: f64, - pub(crate) y: f64, - pub(crate) width: f64, - pub(crate) height: f64, - pub(crate) original_width: f64, - pub(crate) original_height: f64, -} - -#[derive(Debug, Deserialize)] +#[derive(Clone, Debug, Deserialize, Serialize)] #[serde(rename_all = "camelCase")] pub struct EditorImageEditRequest { pub(crate) prompt: String, @@ -211,9 +202,10 @@ pub struct EditorImageEditRequest { pub(crate) asset_folder_id: Option, pub(crate) asset_label: Option, pub(crate) source_resource_id: Option, + pub(crate) canvas_completion: Option, } -#[derive(Debug, Deserialize)] +#[derive(Clone, Debug, Deserialize, Serialize)] #[serde(rename_all = "camelCase")] pub struct EditorIconSpritesheetGenerationRequest { pub(crate) reference_image_src: String, @@ -224,9 +216,10 @@ pub struct EditorIconSpritesheetGenerationRequest { pub(crate) project_id: Option, pub(crate) generation_inputs: Option, pub(crate) asset_folder_id: Option, + pub(crate) canvas_completion: Option, } -#[derive(Debug, Deserialize)] +#[derive(Clone, Debug, Deserialize, Serialize)] #[serde(rename_all = "camelCase")] pub struct EditorUiDesignAssetExtractionRequest { pub(crate) source_image_src: String, @@ -236,6 +229,7 @@ pub struct EditorUiDesignAssetExtractionRequest { pub(crate) generation_inputs: Option, pub(crate) asset_folder_id: Option, pub(crate) spritesheet_label: Option, + pub(crate) canvas_completion: Option, } pub(crate) struct EditorGenerationCaller { @@ -341,6 +335,33 @@ struct PersistEditorGeneratedAssetInput { generation_inputs: Option, } +pub(crate) struct EditorCanvasGeneratedLayerInput { + pub(crate) layer_id: String, + pub(crate) resource_id: String, + pub(crate) title: String, + pub(crate) src: String, + pub(crate) media_type: Option, + pub(crate) thumbnail_src: Option, + pub(crate) original_width: u32, + pub(crate) original_height: u32, + pub(crate) display_width: f64, + pub(crate) display_height: f64, + pub(crate) source_type: String, + pub(crate) prompt: Option, + pub(crate) actual_prompt: Option, + pub(crate) model: Option, + pub(crate) provider: Option, + pub(crate) task_id: Option, + pub(crate) object_key: Option, + pub(crate) asset_object_id: Option, + pub(crate) duration_seconds: Option, + pub(crate) source_resource_id: Option, + pub(crate) asset_kind: Option, + pub(crate) generation_inputs: Option, + pub(crate) image_sequence_frames: Option, + pub(crate) preview_video_path: Option, +} + #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct EditorImageGenerationResponse { @@ -386,6 +407,7 @@ pub struct EditorIconSpritesheetGenerationResponse { price_mud_points: u32, spritesheet_resource: Option, spritesheet_asset: Option, + project: Option, } #[derive(Debug, Serialize)] @@ -895,13 +917,30 @@ pub async fn generate_editor_image( Extension(authenticated): Extension, Json(payload): Json, ) -> Result, AppError> { - generate_editor_image_for_owner( - &state, - &request_context, - EditorGenerationCaller::from_authenticated(&authenticated), - payload, - ) - .await + let caller = EditorGenerationCaller::from_authenticated(&authenticated); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = editor_generation_source_entity_id( + payload.project_id.as_deref(), + "editor-image-generation", + ); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + caller.owner_user_id.as_str(), + EDITOR_IMAGE_GENERATION_JOB_KIND, + source_entity_id, + "图片画布生成图片", + &payload, + ) + .await?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } + generate_editor_image_for_owner(&state, &request_context, caller, payload).await } pub(crate) async fn generate_editor_image_for_owner( @@ -1332,13 +1371,28 @@ pub async fn edit_editor_image( Extension(authenticated): Extension, Json(payload): Json, ) -> Result, AppError> { - edit_editor_image_for_owner( - &state, - &request_context, - EditorGenerationCaller::from_authenticated(&authenticated), - payload, - ) - .await + let caller = EditorGenerationCaller::from_authenticated(&authenticated); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = + editor_generation_source_entity_id(payload.project_id.as_deref(), "editor-image-edit"); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + caller.owner_user_id.as_str(), + EDITOR_IMAGE_EDIT_JOB_KIND, + source_entity_id, + "图片画布修改图片", + &payload, + ) + .await?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } + edit_editor_image_for_owner(&state, &request_context, caller, payload).await } pub(crate) async fn edit_editor_image_for_owner( @@ -1400,11 +1454,12 @@ pub(crate) async fn edit_editor_image_for_owner( image.mime_type, BASE64_STANDARD.encode(image.bytes.as_slice()) ); + let owner_user_id = caller.owner_user_id.clone(); let generated_asset = persist_editor_generated_asset( state, PersistEditorGeneratedAssetInput { - project_id: payload.project_id, - owner_user_id: caller.owner_user_id, + project_id: payload.project_id.clone(), + owner_user_id: owner_user_id.clone(), folder_id: payload.asset_folder_id, label: normalize_optional_string(payload.asset_label) .unwrap_or_else(|| "图片修改结果".to_string()), @@ -1424,6 +1479,14 @@ pub(crate) async fn edit_editor_image_for_owner( }, ) .await?; + let completed_project = complete_editor_canvas_generation( + state, + owner_user_id.as_str(), + payload.project_id.as_deref(), + payload.canvas_completion.as_ref(), + generated_asset.resource.as_ref(), + ) + .await?; Ok(json_success_body( Some(&request_context), @@ -1441,7 +1504,7 @@ pub(crate) async fn edit_editor_image_for_owner( task_id: generated.task_id, resource: generated_asset.resource, asset: generated_asset.asset, - project: None, + project: completed_project, }, )) } @@ -1452,13 +1515,30 @@ pub async fn generate_editor_icon_spritesheet( Extension(authenticated): Extension, Json(payload): Json, ) -> Result, AppError> { - generate_editor_icon_spritesheet_for_owner( - &state, - &request_context, - EditorGenerationCaller::from_authenticated(&authenticated), - payload, - ) - .await + let caller = EditorGenerationCaller::from_authenticated(&authenticated); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = editor_generation_source_entity_id( + payload.project_id.as_deref(), + "editor-icon-spritesheet-generation", + ); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + caller.owner_user_id.as_str(), + EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, + source_entity_id, + "图片画布生成图标素材", + &payload, + ) + .await?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } + generate_editor_icon_spritesheet_for_owner(&state, &request_context, caller, payload).await } pub(crate) async fn generate_editor_icon_spritesheet_for_owner( @@ -1618,6 +1698,25 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner( ..icon_response }); } + let (canvas_items, primary_layer_id) = + if let Some(completion) = payload.canvas_completion.as_ref() { + build_icon_spritesheet_canvas_layer_items( + completion, + spritesheet_record.resource.as_ref(), + icon_image_srcs.as_slice(), + ) + } else { + (Vec::new(), None) + }; + let completed_project = complete_editor_canvas_generation_with_items( + state, + owner_user_id.as_str(), + payload.project_id.as_deref(), + payload.canvas_completion.as_ref(), + canvas_items, + primary_layer_id, + ) + .await?; Ok(json_success_body( Some(&request_context), @@ -1634,6 +1733,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner( price_mud_points: expected_price_mud_points, spritesheet_resource: spritesheet_record.resource, spritesheet_asset: spritesheet_record.asset, + project: completed_project, }, )) } @@ -1644,13 +1744,30 @@ pub async fn extract_editor_ui_design_assets( Extension(authenticated): Extension, Json(payload): Json, ) -> Result, AppError> { - extract_editor_ui_design_assets_for_owner( - &state, - &request_context, - EditorGenerationCaller::from_authenticated(&authenticated), - payload, - ) - .await + let caller = EditorGenerationCaller::from_authenticated(&authenticated); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = editor_generation_source_entity_id( + payload.project_id.as_deref(), + "editor-ui-design-asset-extraction", + ); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + caller.owner_user_id.as_str(), + EDITOR_UI_DESIGN_ASSET_EXTRACTION_JOB_KIND, + source_entity_id, + "图片画布提取UI设计图素材", + &payload, + ) + .await?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } + extract_editor_ui_design_assets_for_owner(&state, &request_context, caller, payload).await } pub(crate) async fn extract_editor_ui_design_assets_for_owner( @@ -1785,6 +1902,25 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner( ..icon_response }); } + let (canvas_items, primary_layer_id) = + if let Some(completion) = payload.canvas_completion.as_ref() { + build_icon_spritesheet_canvas_layer_items( + completion, + spritesheet_record.resource.as_ref(), + icon_image_srcs.as_slice(), + ) + } else { + (Vec::new(), None) + }; + let completed_project = complete_editor_canvas_generation_with_items( + state, + owner_user_id.as_str(), + payload.project_id.as_deref(), + payload.canvas_completion.as_ref(), + canvas_items, + primary_layer_id, + ) + .await?; Ok(json_success_body( Some(&request_context), @@ -1801,6 +1937,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner( price_mud_points: expected_price_mud_points, spritesheet_resource: spritesheet_record.resource, spritesheet_asset: spritesheet_record.asset, + project: completed_project, }, )) } @@ -1994,7 +2131,7 @@ async fn persist_editor_generated_asset( Ok(EditorGeneratedAssetRecord { resource, asset }) } -async fn complete_editor_canvas_generation( +pub(crate) async fn complete_editor_canvas_generation( state: &AppState, owner_user_id: &str, project_id: Option<&str>, @@ -2006,7 +2143,7 @@ async fn complete_editor_canvas_generation( return Ok(None); }; let project_id = project_id.trim(); - if project_id.is_empty() || completion.dialog_id.trim().is_empty() { + if project_id.is_empty() { return Ok(None); } @@ -2018,8 +2155,21 @@ async fn complete_editor_canvas_generation( }) .await .map_err(map_editor_project_error)?; - let completion_result = - apply_editor_canvas_generation_completion(project.layers, completion, resource)?; + let Some(placeholder) = + resolve_canvas_completion_placeholder(project.layers.clone(), completion)? + else { + return Ok(None); + }; + let layer_id = generated_canvas_layer_id(resource.resource_id.as_str()); + let item = + build_generated_canvas_layer_item(completion, &placeholder, resource, layer_id.as_str(), 0); + let completion_result = apply_editor_canvas_generation_items( + project.layers, + completion, + placeholder, + vec![item], + Some(layer_id), + )?; if !completion_result.changed { return Ok(None); } @@ -2038,69 +2188,214 @@ async fn complete_editor_canvas_generation( Ok(Some(editor_project_payload_from_record(saved))) } -fn apply_editor_canvas_generation_completion( +pub(crate) async fn complete_editor_canvas_generation_with_items( + state: &AppState, + owner_user_id: &str, + project_id: Option<&str>, + completion: Option<&EditorCanvasGenerationCompletionRequest>, + items: Vec, + primary_layer_id: Option, +) -> Result, AppError> { + let (Some(project_id), Some(completion)) = (project_id, completion) else { + return Ok(None); + }; + let project_id = project_id.trim(); + if project_id.is_empty() || items.is_empty() { + return Ok(None); + } + + let project = state + .spacetime_client() + .get_editor_project(EditorProjectGetRecordInput { + project_id: project_id.to_string(), + owner_user_id: owner_user_id.to_string(), + }) + .await + .map_err(map_editor_project_error)?; + let Some(placeholder) = + resolve_canvas_completion_placeholder(project.layers.clone(), completion)? + else { + return Ok(None); + }; + let items = + translate_editor_canvas_generation_items(items, &completion.placeholder, &placeholder); + let completion_result = apply_editor_canvas_generation_items( + project.layers, + completion, + placeholder, + items, + primary_layer_id, + )?; + if !completion_result.changed { + return Ok(None); + } + let saved = state + .spacetime_client() + .save_editor_project_layout(EditorProjectLayoutSaveRecordInput { + project_id: project_id.to_string(), + owner_user_id: owner_user_id.to_string(), + viewport: project.viewport.clone(), + layers_json: serialize_editor_layers(completion_result.layers)?, + updated_at_micros: current_utc_micros(), + }) + .await + .map_err(map_editor_project_error)?; + + Ok(Some(editor_project_payload_from_record(saved))) +} + +fn translate_editor_canvas_generation_items( + mut items: Vec, + submitted_placeholder: &EditorCanvasGenerationPlaceholderPayload, + resolved_placeholder: &EditorCanvasGenerationPlaceholderPayload, +) -> Vec { + let dx = resolved_placeholder.x - submitted_placeholder.x; + let dy = resolved_placeholder.y - submitted_placeholder.y; + if !dx.is_finite() || !dy.is_finite() || (dx == 0.0 && dy == 0.0) { + return items; + } + + for item in &mut items { + let Some(object) = item.as_object_mut() else { + continue; + }; + if let Some(x) = object.get("x").and_then(Value::as_f64) + && (x + dx).is_finite() + { + object.insert("x".to_string(), json!(x + dx)); + } + if let Some(y) = object.get("y").and_then(Value::as_f64) + && (y + dy).is_finite() + { + object.insert("y".to_string(), json!(y + dy)); + } + } + + items +} + +fn resolve_canvas_completion_placeholder( layers: Value, completion: &EditorCanvasGenerationCompletionRequest, - resource: &EditorProjectResourcePayload, +) -> Result, AppError> { + let dialog_id = normalized_canvas_completion_dialog_id(completion); + let items = match layers { + Value::Array(items) => items, + _ => Vec::new(), + }; + if let Some(dialog_id) = dialog_id.as_deref() { + return items + .iter() + .find(|item| generation_dialog_item_matches(item, dialog_id)) + .map(|item| { + generation_dialog_item_placeholder(item, dialog_id) + .unwrap_or_else(|| completion.placeholder.clone()) + }) + .map(Some) + .map_or(Ok(None), Ok); + } + Ok(Some(completion.placeholder.clone())) +} + +fn apply_editor_canvas_generation_items( + layers: Value, + completion: &EditorCanvasGenerationCompletionRequest, + _placeholder: EditorCanvasGenerationPlaceholderPayload, + generated_items: Vec, + primary_layer_id: Option, ) -> Result { let mut items = match layers { Value::Array(items) => items, _ => Vec::new(), }; - let Some(placeholder) = items - .iter() - .find(|item| generation_dialog_item_matches(item, completion.dialog_id.as_str())) - .map(|item| { - generation_dialog_item_placeholder(item, completion.dialog_id.as_str()) - .unwrap_or_else(|| completion.placeholder.clone()) - }) - else { + let dialog_id = normalized_canvas_completion_dialog_id(completion); + if let Some(dialog_id) = dialog_id.as_deref() + && !items + .iter() + .any(|item| generation_dialog_item_matches(item, dialog_id)) + { return Ok(EditorCanvasGenerationCompletionResult { layers: Value::Array(items), changed: false, }); - }; - let generated_layer_id = generated_canvas_layer_id(resource.resource_id.as_str()); - let completed_layer_id = if let Some(existing_layer_id) = items.iter().find_map(|item| { - let resource_id = item.get("resourceId").and_then(Value::as_str)?; - if resource_id == resource.resource_id { - item.get("layerId") - .and_then(Value::as_str) - .map(str::to_string) - } else { - None - } - }) { - existing_layer_id - } else { - let z_index = next_canvas_layer_z_index(&items); - items.push(build_generated_canvas_layer_item( - completion, - &placeholder, - resource, - generated_layer_id.as_str(), - z_index, - )); - generated_layer_id - }; + } - for item in &mut items { - if generation_dialog_item_matches(item, completion.dialog_id.as_str()) { - if let Some(dialog) = item.get_mut("dialog").and_then(Value::as_object_mut) { + let mut changed = false; + let mut next_z_index = next_canvas_layer_z_index(&items); + let mut completed_layer_id = primary_layer_id; + for mut item in generated_items { + let existing_layer_id = + item.get("resourceId") + .and_then(Value::as_str) + .and_then(|resource_id| { + items.iter().find_map(|existing| { + let existing_resource_id = + existing.get("resourceId").and_then(Value::as_str)?; + if existing_resource_id == resource_id { + existing + .get("layerId") + .and_then(Value::as_str) + .map(str::to_string) + } else { + None + } + }) + }); + if let Some(existing_layer_id) = existing_layer_id { + completed_layer_id = Some(existing_layer_id); + continue; + } + let needs_z_index = match item.get("zIndex").and_then(Value::as_i64) { + Some(value) => value <= 0, + None => true, + }; + if needs_z_index && let Some(object) = item.as_object_mut() { + object.insert("zIndex".to_string(), json!(next_z_index)); + next_z_index += 1; + } + if completed_layer_id.is_none() { + completed_layer_id = item + .get("layerId") + .and_then(Value::as_str) + .map(str::to_string); + } + items.push(item); + changed = true; + } + + if let Some(dialog_id) = dialog_id.as_deref() { + for item in &mut items { + if generation_dialog_item_matches(item, dialog_id) + && let Some(dialog) = item.get_mut("dialog").and_then(Value::as_object_mut) + { dialog.insert("status".to_string(), json!("idle")); dialog.insert("composerOpen".to_string(), json!(true)); - dialog.insert("generatedLayerId".to_string(), json!(completed_layer_id)); + if let Some(completed_layer_id) = completed_layer_id.as_deref() { + dialog.insert("generatedLayerId".to_string(), json!(completed_layer_id)); + } dialog.remove("errorMessage"); + changed = true; } } } Ok(EditorCanvasGenerationCompletionResult { layers: Value::Array(items), - changed: true, + changed, }) } +fn normalized_canvas_completion_dialog_id( + completion: &EditorCanvasGenerationCompletionRequest, +) -> Option { + completion + .dialog_id + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(ToOwned::to_owned) +} + fn build_generated_canvas_layer_item( completion: &EditorCanvasGenerationCompletionRequest, placeholder: &EditorCanvasGenerationPlaceholderPayload, @@ -2122,6 +2417,7 @@ fn build_generated_canvas_layer_item( "layerId": generated_layer_id, "resourceId": resource.resource_id, "title": completion.title.trim(), + "src": resource.image_src, "x": x, "y": y, "width": width, @@ -2139,6 +2435,52 @@ fn build_generated_canvas_layer_item( "objectKey": resource.object_key, "assetObjectId": resource.asset_object_id, "sourceResourceId": resource.source_resource_id, + "assetKind": resource.asset_kind, + "generationInputs": resource.generation_inputs, + }) +} + +pub(crate) fn build_editor_canvas_generated_layer_item( + completion: &EditorCanvasGenerationCompletionRequest, + input: EditorCanvasGeneratedLayerInput, +) -> Value { + let placeholder = &completion.placeholder; + let original_width = positive_f64(input.original_width as f64, placeholder.original_width); + let original_height = positive_f64(input.original_height as f64, placeholder.original_height); + let width = positive_f64(input.display_width, placeholder.width); + let height = positive_f64(input.display_height, placeholder.height); + let placeholder_width = positive_f64(placeholder.width, width); + let placeholder_height = positive_f64(placeholder.height, height); + let x = placeholder.x + placeholder_width / 2.0 - width / 2.0; + let y = placeholder.y + placeholder_height / 2.0 - height / 2.0; + + json!({ + "layerId": input.layer_id, + "resourceId": input.resource_id, + "title": input.title, + "src": input.src, + "mediaType": input.media_type, + "thumbnailSrc": input.thumbnail_src, + "x": x, + "y": y, + "width": width, + "height": height, + "originalWidth": original_width, + "originalHeight": original_height, + "sourceType": input.source_type, + "prompt": input.prompt, + "actualPrompt": input.actual_prompt, + "model": input.model, + "provider": input.provider, + "taskId": input.task_id, + "objectKey": input.object_key, + "assetObjectId": input.asset_object_id, + "durationSeconds": input.duration_seconds, + "sourceResourceId": input.source_resource_id, + "assetKind": input.asset_kind, + "generationInputs": input.generation_inputs, + "imageSequenceFrames": input.image_sequence_frames, + "previewVideoPath": input.preview_video_path, }) } @@ -2164,6 +2506,75 @@ fn generated_canvas_media_type(asset_kind: Option<&str>) -> &'static str { } } +fn build_icon_spritesheet_canvas_layer_items( + completion: &EditorCanvasGenerationCompletionRequest, + spritesheet_resource: Option<&EditorProjectResourcePayload>, + icons: &[EditorIconSpritesheetIconResponse], +) -> (Vec, Option) { + let mut items = Vec::new(); + let mut primary_layer_id = None; + let placeholder = completion.placeholder.clone(); + let Some(spritesheet_resource) = spritesheet_resource else { + return (items, primary_layer_id); + }; + + let spritesheet_layer_id = generated_canvas_layer_id(spritesheet_resource.resource_id.as_str()); + primary_layer_id = Some(spritesheet_layer_id.clone()); + let spritesheet_item = build_generated_canvas_layer_item( + completion, + &placeholder, + spritesheet_resource, + spritesheet_layer_id.as_str(), + 0, + ); + let sheet_width = spritesheet_item + .get("width") + .and_then(Value::as_f64) + .unwrap_or(placeholder.width.max(1.0)); + items.push(spritesheet_item); + + let mut cursor_x = placeholder.x + sheet_width + 32.0; + let mut cursor_y = placeholder.y; + let mut row_height = 0.0_f64; + let max_row_width = 560.0_f64; + let start_x = cursor_x; + let spacing = 24.0_f64; + + for icon in icons { + let Some(resource) = icon.resource.as_ref() else { + continue; + }; + let width = positive_f64(resource.width as f64, 128.0); + let height = positive_f64(resource.height as f64, 128.0); + if cursor_x > start_x && cursor_x + width - start_x > max_row_width { + cursor_x = start_x; + cursor_y += row_height + spacing; + row_height = 0.0; + } + let layer_id = generated_canvas_layer_id(resource.resource_id.as_str()); + let mut item = build_generated_canvas_layer_item( + completion, + &placeholder, + resource, + layer_id.as_str(), + 0, + ); + if let Some(object) = item.as_object_mut() { + object.insert("x".to_string(), json!(cursor_x)); + object.insert("y".to_string(), json!(cursor_y)); + object.insert("width".to_string(), json!(width)); + object.insert("height".to_string(), json!(height)); + object.insert("originalWidth".to_string(), json!(width)); + object.insert("originalHeight".to_string(), json!(height)); + } + cursor_x += width + spacing; + row_height = row_height.max(height); + items.push(item); + } + + (items, primary_layer_id) +} + fn next_canvas_layer_z_index(items: &[Value]) -> i64 { items .iter() @@ -2682,6 +3093,49 @@ mod tests { use super::*; use crate::config::AppConfig; + fn apply_editor_canvas_generation_completion( + layers: Value, + completion: &EditorCanvasGenerationCompletionRequest, + resource: &EditorProjectResourcePayload, + ) -> Result { + let placeholder = resolve_canvas_completion_placeholder(layers.clone(), completion)? + .unwrap_or_else(|| completion.placeholder.clone()); + let layer_id = generated_canvas_layer_id(resource.resource_id.as_str()); + let item = build_generated_canvas_layer_item( + completion, + &placeholder, + resource, + layer_id.as_str(), + 0, + ); + apply_editor_canvas_generation_items( + layers, + completion, + placeholder, + vec![item], + Some(layer_id), + ) + } + + fn apply_editor_canvas_generation_multi_items( + layers: Value, + completion: &EditorCanvasGenerationCompletionRequest, + items: Vec, + primary_layer_id: Option, + ) -> Result { + let placeholder = resolve_canvas_completion_placeholder(layers.clone(), completion)? + .unwrap_or_else(|| completion.placeholder.clone()); + let items = + translate_editor_canvas_generation_items(items, &completion.placeholder, &placeholder); + apply_editor_canvas_generation_items( + layers, + completion, + placeholder, + items, + primary_layer_id, + ) + } + #[test] fn editor_image_generation_size_keeps_quick_edit_canvas_ratio_presets() { assert_eq!(normalize_editor_image_generation_size(None), "1024x1024"); @@ -2991,7 +3445,7 @@ mod tests { } ]); let completion = EditorCanvasGenerationCompletionRequest { - dialog_id: "dialog-character".to_string(), + dialog_id: Some("dialog-character".to_string()), title: "角色形象 1".to_string(), placeholder: EditorCanvasGenerationPlaceholderPayload { x: 200.0, @@ -3050,9 +3504,8 @@ mod tests { assert_eq!(result_layer["zIndex"], json!(5)); assert_eq!(result_layer["mediaType"], json!("image")); assert_eq!( - result_layer.get("src"), - None, - "layout must not embed heavy image payloads" + result_layer["src"], + json!("data:image/png;base64,character") ); let dialog = items @@ -3091,7 +3544,7 @@ mod tests { } ]); let completion = EditorCanvasGenerationCompletionRequest { - dialog_id: "dialog-character".to_string(), + dialog_id: Some("dialog-character".to_string()), title: "角色形象 1".to_string(), placeholder: EditorCanvasGenerationPlaceholderPayload { x: 0.0, @@ -3171,7 +3624,7 @@ mod tests { } ]); let completion = EditorCanvasGenerationCompletionRequest { - dialog_id: "dialog-character".to_string(), + dialog_id: Some("dialog-character".to_string()), title: "角色形象 1".to_string(), placeholder: EditorCanvasGenerationPlaceholderPayload { x: 0.0, @@ -3217,6 +3670,71 @@ mod tests { assert_eq!(result_layer["y"], json!(-88.0)); } + #[test] + fn editor_canvas_generation_multi_item_completion_uses_latest_layout_placeholder() { + let layers = json!([ + { + "itemType": "generation-dialog", + "layerId": "generation-dialog:dialog-icons", + "resourceId": "generation-dialog:dialog-icons", + "dialog": { + "id": "dialog-icons", + "mode": "icon", + "prompt": "返回按钮", + "status": "generating", + "composerOpen": false, + "placeholder": { + "x": 300, + "y": 240, + "width": 200, + "height": 200, + "originalWidth": 200, + "originalHeight": 200 + } + } + } + ]); + let completion = EditorCanvasGenerationCompletionRequest { + dialog_id: Some("dialog-icons".to_string()), + title: "图标素材图集".to_string(), + placeholder: EditorCanvasGenerationPlaceholderPayload { + x: 100.0, + y: 80.0, + width: 200.0, + height: 200.0, + original_width: 200.0, + original_height: 200.0, + }, + }; + let item = json!({ + "layerId": "layer-icon-sheet", + "resourceId": "resource-icon-sheet", + "title": "图标素材图集", + "x": 150.0, + "y": 130.0, + "width": 100.0, + "height": 100.0 + }); + + let completed = apply_editor_canvas_generation_multi_items( + layers, + &completion, + vec![item], + Some("layer-icon-sheet".to_string()), + ) + .expect("multi-item completion should apply"); + let result_layer = completed + .layers + .as_array() + .expect("layers should stay array") + .iter() + .find(|item| item["resourceId"] == "resource-icon-sheet") + .expect("result layer should be inserted"); + + assert_eq!(result_layer["x"], json!(350.0)); + assert_eq!(result_layer["y"], json!(290.0)); + } + #[test] fn editor_canvas_generation_completion_skips_when_dialog_was_removed() { let layers = json!([ @@ -3227,7 +3745,7 @@ mod tests { } ]); let completion = EditorCanvasGenerationCompletionRequest { - dialog_id: "dialog-character".to_string(), + dialog_id: Some("dialog-character".to_string()), title: "角色形象 1".to_string(), placeholder: EditorCanvasGenerationPlaceholderPayload { x: 200.0, @@ -3326,8 +3844,7 @@ mod tests { 12 ); assert_eq!( - resolve_editor_icon_spritesheet_price(&state, Some("gpt-image-2"), Some("1K")) - .unwrap(), + resolve_editor_icon_spritesheet_price(&state, Some("gpt-image-2"), Some("1K")).unwrap(), 3 ); let response = EditorIconSpritesheetGenerationResponse { @@ -3354,6 +3871,7 @@ mod tests { ), spritesheet_resource: None, spritesheet_asset: None, + project: None, }; let payload = serde_json::to_value(response).expect("response should serialize"); @@ -3378,7 +3896,10 @@ mod tests { assert_eq!(generation_options.model, GPT_IMAGE_2_MODEL); assert_eq!(generation_options.size, "2048x2048"); assert_eq!( - resolve_editor_ui_design_asset_extraction_price(&state, Some(generation_options.image_size)) + resolve_editor_ui_design_asset_extraction_price( + &state, + Some(generation_options.image_size) + ) .unwrap(), 5 ); diff --git a/server-rs/crates/api-server/src/external_generation_worker.rs b/server-rs/crates/api-server/src/external_generation_worker.rs index fe110c2ed..ccc02ac3d 100644 --- a/server-rs/crates/api-server/src/external_generation_worker.rs +++ b/server-rs/crates/api-server/src/external_generation_worker.rs @@ -1,6 +1,6 @@ use std::{future::Future, io, pin::Pin, time::Duration}; -use axum::extract::FromRef; +use axum::{Json, extract::FromRef}; use serde_json::json; use shared_kernel::offset_datetime_to_unix_micros; use spacetime_client::{ @@ -15,6 +15,22 @@ use tokio::{ use tracing::{error, info, warn}; use crate::{ + character_animation_assets::{ + generate_editor_character_animation_for_owner, generate_editor_video_for_owner, + }, + editor_generation_queue::{ + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, + EDITOR_CHARACTER_ANIMATION_GENERATION_JOB_KIND, + EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND, EDITOR_IMAGE_EDIT_JOB_KIND, + EDITOR_IMAGE_GENERATION_JOB_KIND, EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, + EDITOR_UI_DESIGN_ASSET_EXTRACTION_JOB_KIND, EDITOR_VIDEO_GENERATION_JOB_KIND, + }, + editor_project::{ + EditorGenerationCaller, EditorIconSpritesheetGenerationRequest, EditorImageEditRequest, + EditorImageGenerationRequest, EditorUiDesignAssetExtractionRequest, + edit_editor_image_for_owner, extract_editor_ui_design_assets_for_owner, + generate_editor_icon_spritesheet_for_owner, generate_editor_image_for_owner, + }, jump_hop::{ JUMP_HOP_COMPILE_DRAFT_JOB_KIND, JumpHopCompileDraftWorkerPayload, execute_jump_hop_compile_draft_worker_job, @@ -31,6 +47,9 @@ use crate::{ }, request_context::RequestContext, state::{AppState, PuzzleApiState}, + vector_engine_audio_generation::{ + generate_editor_background_music_for_owner, generate_editor_sound_effect_for_owner, + }, wooden_fish::{ WOODEN_FISH_GENERATE_IMAGE_ASSETS_JOB_KIND, WoodenFishGenerateImageAssetsWorkerPayload, execute_wooden_fish_generate_image_assets_worker_job, @@ -536,6 +555,234 @@ async fn process_external_generation_job_once( } } } + EDITOR_IMAGE_GENERATION_JOB_KIND => { + let payload = match serde_json::from_str::( + job.request_payload_json.as_str(), + ) { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布生图任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match generate_editor_image_for_owner( + &state, + &request_context, + editor_generation_worker_caller(&job), + payload, + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(error) => { + let message = error.body_text(); + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_IMAGE_EDIT_JOB_KIND => { + let payload = match serde_json::from_str::( + job.request_payload_json.as_str(), + ) { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布改图任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match edit_editor_image_for_owner( + &state, + &request_context, + editor_generation_worker_caller(&job), + payload, + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(error) => { + let message = error.body_text(); + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_ICON_SPRITESHEET_GENERATION_JOB_KIND => { + let payload = match serde_json::from_str::( + job.request_payload_json.as_str(), + ) { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布图标素材任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match generate_editor_icon_spritesheet_for_owner( + &state, + &request_context, + editor_generation_worker_caller(&job), + payload, + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(error) => { + let message = error.body_text(); + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_UI_DESIGN_ASSET_EXTRACTION_JOB_KIND => { + let payload = match serde_json::from_str::( + job.request_payload_json.as_str(), + ) { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布 UI 素材提取任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match extract_editor_ui_design_assets_for_owner( + &state, + &request_context, + editor_generation_worker_caller(&job), + payload, + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(error) => { + let message = error.body_text(); + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_CHARACTER_ANIMATION_GENERATION_JOB_KIND => { + let payload = match serde_json::from_str::< + shared_contracts::assets::EditorCharacterAnimationGenerateRequest, + >(job.request_payload_json.as_str()) + { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布角色动作任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match generate_editor_character_animation_for_owner( + state.clone(), + request_context, + job.owner_user_id.clone(), + Ok(Json(payload)), + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(response) => { + let message = response_error_message(response).await; + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_VIDEO_GENERATION_JOB_KIND => { + let payload = match serde_json::from_str::< + shared_contracts::assets::EditorVideoGenerateRequest, + >(job.request_payload_json.as_str()) + { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布视频生成任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match generate_editor_video_for_owner( + state.clone(), + request_context, + job.owner_user_id.clone(), + Ok(Json(payload)), + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(response) => { + let message = response_error_message(response).await; + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND => { + let payload = match serde_json::from_str::< + shared_contracts::assets::EditorSoundEffectGenerateRequest, + >(job.request_payload_json.as_str()) + { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布音效生成任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match generate_editor_sound_effect_for_owner( + state.clone(), + request_context, + job.owner_user_id.clone(), + Ok(Json(payload)), + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(response) => { + let message = response_error_message(response).await; + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND => { + let payload = match serde_json::from_str::< + shared_contracts::assets::EditorBackgroundMusicGenerateRequest, + >(job.request_payload_json.as_str()) + { + Ok(payload) => payload, + Err(error) => { + let message = format!("图片画布背景音乐生成任务参数解析失败：{error}"); + fail_job(&state, &worker_id, &job, message.clone()).await?; + return Err(message); + } + }; + let request_context = worker_request_context(&job); + match generate_editor_background_music_for_owner( + state.clone(), + request_context, + job.owner_user_id.clone(), + Ok(Json(payload)), + ) + .await + { + Ok(_) => complete_editor_generation_job(&state, &worker_id, &job).await, + Err(response) => { + let message = response_error_message(response).await; + fail_job(&state, &worker_id, &job, message.clone()).await?; + Err(message) + } + } + } unknown => { warn!( job_id = job.job_id, @@ -598,6 +845,43 @@ async fn fail_queue_job_after_worker_error( Ok(()) } +fn worker_request_context(job: &ExternalGenerationJobRecord) -> RequestContext { + RequestContext::new( + format!("external-generation-worker-{}", job.job_id), + format!("external-generation-worker {}", job.job_kind), + std::time::Duration::ZERO, + false, + ) +} + +fn editor_generation_worker_caller(job: &ExternalGenerationJobRecord) -> EditorGenerationCaller { + EditorGenerationCaller { + owner_user_id: job.owner_user_id.clone(), + audit_subject_user_id: Some(job.owner_user_id.clone()), + audit_project_id: Some(job.source_entity_id.clone()), + } +} + +async fn complete_editor_generation_job( + state: &AppState, + worker_id: &str, + job: &ExternalGenerationJobRecord, +) -> Result<(), String> { + complete_job( + state, + worker_id, + job, + Some( + json!({ + "sourceModule": job.source_module.clone(), + "sourceEntityId": job.source_entity_id.clone(), + }) + .to_string(), + ), + ) + .await +} + async fn complete_job( state: &AppState, worker_id: &str, diff --git a/server-rs/crates/api-server/src/main.rs b/server-rs/crates/api-server/src/main.rs index e9da1150a..529060bb9 100644 --- a/server-rs/crates/api-server/src/main.rs +++ b/server-rs/crates/api-server/src/main.rs @@ -37,6 +37,7 @@ mod custom_world_foundation_draft; mod custom_world_result_prompts; mod custom_world_rpg_draft_prompts; mod editor_generation_config; +mod editor_generation_queue; mod editor_project; mod edutainment_baby_drawing; mod edutainment_baby_object; diff --git a/server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs b/server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs index 07eff0393..0296deb8a 100644 --- a/server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs +++ b/server-rs/crates/api-server/src/vector_engine_audio_generation/generation.rs @@ -11,9 +11,21 @@ use serde_json::json; use shared_contracts::{assets, creation_audio}; use crate::{ - api_response::json_success_body, auth::AuthenticatedAccessToken, - editor_generation_config::EditorGenerationPricingConfig, http_error::AppError, - request_context::RequestContext, state::AppState, + api_response::json_success_body, + auth::AuthenticatedAccessToken, + editor_generation_config::EditorGenerationPricingConfig, + editor_generation_queue::{ + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, + EditorGenerationQueuedResponse, editor_generation_queue_state, + editor_generation_source_entity_id, enqueue_editor_generation_job, + }, + editor_project::{ + EditorCanvasGeneratedLayerInput, build_editor_canvas_generated_layer_item, + complete_editor_canvas_generation_with_items, + }, + http_error::AppError, + request_context::RequestContext, + state::AppState, tracking::record_external_generation_run_after_success, }; @@ -128,13 +140,33 @@ pub async fn generate_editor_sound_effect( Extension(authenticated): Extension, payload: Result, JsonRejection>, ) -> Result, Response> { - generate_editor_sound_effect_for_owner( - state, - request_context, - authenticated.claims().user_id().to_string(), - payload, - ) - .await + let Json(payload) = parse_json_payload(&request_context, payload)?; + let owner_user_id = authenticated.claims().user_id().to_string(); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = editor_generation_source_entity_id( + payload.project_id.as_deref(), + "editor-sound-effect", + ); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + owner_user_id.as_str(), + EDITOR_SOUND_EFFECT_GENERATION_JOB_KIND, + source_entity_id, + "图片画布生成音效", + &payload, + ) + .await + .map_err(|error| error.into_response_with_context(Some(&request_context)))?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } + generate_editor_sound_effect_for_owner(state, request_context, owner_user_id, Ok(Json(payload))) + .await } pub(crate) async fn generate_editor_sound_effect_for_owner( @@ -144,6 +176,9 @@ pub(crate) async fn generate_editor_sound_effect_for_owner( payload: Result, JsonRejection>, ) -> Result, Response> { let Json(payload) = parse_json_payload(&request_context, payload)?; + let project_id = payload.project_id.clone(); + let canvas_completion = payload.canvas_completion.clone(); + let generation_inputs = payload.generation_inputs.clone(); let pricing = state .editor_generation_pricing() .map_err(|error| { @@ -192,6 +227,52 @@ pub(crate) async fn generate_editor_sound_effect_for_owner( .audio_src .ok_or_else(|| vector_engine_bad_gateway("音效生成完成但缺少播放地址")) .map_err(|error| error.into_response_with_context(Some(&request_context)))?; + let completed_project = if let Some(completion) = canvas_completion.as_ref() { + let layer_id = format!("layer-editor-sound-effect-{}", generated.task_id); + let item = build_editor_canvas_generated_layer_item( + completion, + EditorCanvasGeneratedLayerInput { + layer_id: layer_id.clone(), + resource_id: format!("local-resource-editor-sound-effect-{}", generated.task_id), + title: completion.title.trim().to_string(), + src: audio_src.clone(), + media_type: Some("audio".to_string()), + thumbnail_src: None, + original_width: EDITOR_AUDIO_WIDTH, + original_height: EDITOR_AUDIO_HEIGHT, + display_width: completion.placeholder.width, + display_height: completion.placeholder.height, + source_type: "generated".to_string(), + prompt: Some(normalized.prompt.clone()), + actual_prompt: Some(normalized.prompt.clone()), + model: Some(normalized.model.clone()), + provider: Some(generated.provider.clone()), + task_id: Some(generated.task_id.clone()), + object_key: generated.object_key.clone(), + asset_object_id: generated.asset_object_id.clone(), + duration_seconds: Some(u32::from(normalized.duration)), + source_resource_id: None, + asset_kind: Some("sound-effect".to_string()), + generation_inputs, + image_sequence_frames: None, + preview_video_path: None, + }, + ); + complete_editor_canvas_generation_with_items( + &state, + owner_user_id.as_str(), + project_id.as_deref(), + canvas_completion.as_ref(), + vec![item], + Some(layer_id), + ) + .await + .map_err(|error| error.into_response_with_context(Some(&request_context)))? + } else { + None + }; + let completed_project = + completed_project.and_then(|project| serde_json::to_value(project).ok()); Ok(json_success_body( Some(&request_context), @@ -210,6 +291,8 @@ pub(crate) async fn generate_editor_sound_effect_for_owner( task_id: generated.task_id, price_mud_points: normalized.price_mud_points, audio_kind: "sound-effect".to_string(), + project: completed_project, + queue_state: None, }, )) } @@ -220,11 +303,36 @@ pub async fn generate_editor_background_music( Extension(authenticated): Extension, payload: Result, JsonRejection>, ) -> Result, Response> { + let Json(payload) = parse_json_payload(&request_context, payload)?; + let owner_user_id = authenticated.claims().user_id().to_string(); + if !state.config.external_generation_mode.is_inline() { + let source_entity_id = editor_generation_source_entity_id( + payload.project_id.as_deref(), + "editor-background-music", + ); + let queue_job = enqueue_editor_generation_job( + &state, + &request_context, + owner_user_id.as_str(), + EDITOR_BACKGROUND_MUSIC_GENERATION_JOB_KIND, + source_entity_id, + "图片画布生成背景音乐", + &payload, + ) + .await + .map_err(|error| error.into_response_with_context(Some(&request_context)))?; + return Ok(json_success_body( + Some(&request_context), + EditorGenerationQueuedResponse { + queue_state: editor_generation_queue_state(queue_job), + }, + )); + } generate_editor_background_music_for_owner( state, request_context, - authenticated.claims().user_id().to_string(), - payload, + owner_user_id, + Ok(Json(payload)), ) .await } @@ -236,6 +344,9 @@ pub(crate) async fn generate_editor_background_music_for_owner( payload: Result, JsonRejection>, ) -> Result, Response> { let Json(payload) = parse_json_payload(&request_context, payload)?; + let project_id = payload.project_id.clone(); + let canvas_completion = payload.canvas_completion.clone(); + let generation_inputs = payload.generation_inputs.clone(); let pricing = state .editor_generation_pricing() .map_err(|error| { @@ -282,6 +393,55 @@ pub(crate) async fn generate_editor_background_music_for_owner( .audio_src .ok_or_else(|| vector_engine_bad_gateway("背景音乐生成完成但缺少播放地址")) .map_err(|error| error.into_response_with_context(Some(&request_context)))?; + let completed_project = if let Some(completion) = canvas_completion.as_ref() { + let layer_id = format!("layer-editor-background-music-{}", generated.task_id); + let item = build_editor_canvas_generated_layer_item( + completion, + EditorCanvasGeneratedLayerInput { + layer_id: layer_id.clone(), + resource_id: format!( + "local-resource-editor-background-music-{}", + generated.task_id + ), + title: completion.title.trim().to_string(), + src: audio_src.clone(), + media_type: Some("audio".to_string()), + thumbnail_src: None, + original_width: EDITOR_AUDIO_WIDTH, + original_height: EDITOR_AUDIO_HEIGHT, + display_width: completion.placeholder.width, + display_height: completion.placeholder.height, + source_type: "generated".to_string(), + prompt: Some(normalized.gpt_description_prompt.clone()), + actual_prompt: Some(normalized.gpt_description_prompt.clone()), + model: Some(platform_audio::SUNO_DEFAULT_MODEL.to_string()), + provider: Some(generated.provider.clone()), + task_id: Some(generated.task_id.clone()), + object_key: generated.object_key.clone(), + asset_object_id: generated.asset_object_id.clone(), + duration_seconds: None, + source_resource_id: None, + asset_kind: Some("background-music".to_string()), + generation_inputs, + image_sequence_frames: None, + preview_video_path: None, + }, + ); + complete_editor_canvas_generation_with_items( + &state, + owner_user_id.as_str(), + project_id.as_deref(), + canvas_completion.as_ref(), + vec![item], + Some(layer_id), + ) + .await + .map_err(|error| error.into_response_with_context(Some(&request_context)))? + } else { + None + }; + let completed_project = + completed_project.and_then(|project| serde_json::to_value(project).ok()); Ok(json_success_body( Some(&request_context), @@ -300,6 +460,8 @@ pub(crate) async fn generate_editor_background_music_for_owner( task_id: generated.task_id, price_mud_points: normalized.price_mud_points, audio_kind: "background-music".to_string(), + project: completed_project, + queue_state: None, }, )) } diff --git a/server-rs/crates/api-server/src/vector_engine_audio_generation/tests.rs b/server-rs/crates/api-server/src/vector_engine_audio_generation/tests.rs index 50c3d7ac6..53913ee9d 100644 --- a/server-rs/crates/api-server/src/vector_engine_audio_generation/tests.rs +++ b/server-rs/crates/api-server/src/vector_engine_audio_generation/tests.rs @@ -109,6 +109,9 @@ fn editor_sound_effect_request_normalizes_prompt_duration_and_resolves_price() { prompt: " 金币掉落叮当声 ".to_string(), model: None, duration: 7, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect("editor sound effect request should normalize"); @@ -125,6 +128,9 @@ fn editor_sound_effect_request_accepts_only_vidu_audio_model() { prompt: "金币掉落叮当声".to_string(), model: Some(" audio1.0 ".to_string()), duration: 5, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect("Vidu audio model should be accepted"); @@ -134,6 +140,9 @@ fn editor_sound_effect_request_accepts_only_vidu_audio_model() { prompt: "金币掉落叮当声".to_string(), model: Some(platform_audio::SUNO_DEFAULT_MODEL.to_string()), duration: 5, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect_err("Suno text-to-sound should be disabled for editor sound effects"); @@ -148,6 +157,9 @@ fn editor_sound_effect_request_rejects_duration_outside_vidu_range() { prompt: "按钮点击".to_string(), model: None, duration, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect_err("duration outside 2-10 seconds should fail"); @@ -183,6 +195,9 @@ fn editor_background_music_request_forces_instrumental_and_resolves_price() { normalize_editor_background_music_request(assets::EditorBackgroundMusicGenerateRequest { gpt_description_prompt: " 森林冒险背景音乐 ".to_string(), make_instrumental: false, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect("editor background music request should normalize"); @@ -197,6 +212,9 @@ fn editor_background_music_request_rejects_prompt_over_documented_limit() { normalize_editor_background_music_request(assets::EditorBackgroundMusicGenerateRequest { gpt_description_prompt: "乐".repeat(201), make_instrumental: true, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect_err("Suno gpt_description_prompt should follow Apifox 200 char limit"); diff --git a/server-rs/crates/shared-contracts/src/assets.rs b/server-rs/crates/shared-contracts/src/assets.rs index 20b21ada0..42fe67d42 100644 --- a/server-rs/crates/shared-contracts/src/assets.rs +++ b/server-rs/crates/shared-contracts/src/assets.rs @@ -3,6 +3,8 @@ use std::collections::BTreeMap; use serde::{Deserialize, Serialize}; use serde_json::Value; +use crate::external_generation::ExternalGenerationJobStatusRecord; + #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct CreateDirectUploadTicketRequest { @@ -304,7 +306,7 @@ pub struct CharacterAnimationGenerateResponse { pub preview_video_path: Option, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorCharacterAnimationGenerateRequest { pub source_layer_id: String, @@ -317,6 +319,14 @@ pub struct EditorCharacterAnimationGenerateRequest { pub frame_count: u32, pub duration_seconds: u32, pub model: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canvas_completion: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub generation_inputs: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source_resource_id: Option, } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] @@ -328,7 +338,7 @@ pub struct EditorCharacterAnimationFramePayload { pub height: u32, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorCharacterAnimationGenerateResponse { pub ok: bool, @@ -343,9 +353,13 @@ pub struct EditorCharacterAnimationGenerateResponse { pub frame_height: u32, pub fps: u32, pub price_mud_points: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub queue_state: Option, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorVideoGenerateRequest { pub prompt: String, @@ -363,13 +377,23 @@ pub struct EditorVideoGenerateRequest { pub reference_video_srcs: Vec, #[serde(default)] pub reference_audio_srcs: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canvas_completion: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub generation_inputs: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source_resource_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub asset_kind: Option, } fn default_editor_video_web_search_enabled() -> bool { true } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorVideoGenerateResponse { pub ok: bool, @@ -390,6 +414,30 @@ pub struct EditorVideoGenerateResponse { pub duration_seconds: u32, pub resolution: String, pub price_mud_points: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub queue_state: Option, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorCanvasGenerationPlaceholderPayload { + pub x: f64, + pub y: f64, + pub width: f64, + pub height: f64, + pub original_width: f64, + pub original_height: f64, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct EditorCanvasGenerationCompletionPayload { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub dialog_id: Option, + pub title: String, + pub placeholder: EditorCanvasGenerationPlaceholderPayload, } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] @@ -430,23 +478,35 @@ pub struct EditorIconSpritesheetGenerateResponse { pub price_mud_points: u32, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorSoundEffectGenerateRequest { pub prompt: String, #[serde(default, skip_serializing_if = "Option::is_none")] pub model: Option, pub duration: u8, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canvas_completion: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub generation_inputs: Option, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorBackgroundMusicGenerateRequest { pub gpt_description_prompt: String, pub make_instrumental: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canvas_completion: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub generation_inputs: Option, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] #[serde(rename_all = "camelCase")] pub struct EditorAudioGenerateResponse { pub ok: bool, @@ -466,6 +526,10 @@ pub struct EditorAudioGenerateResponse { pub task_id: String, pub price_mud_points: u32, pub audio_kind: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub project: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub queue_state: Option, } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq)] @@ -992,6 +1056,21 @@ mod tests { frame_count: 48, duration_seconds: 6, model: "seedance2.0-fast".to_string(), + project_id: Some("editor-project-1".to_string()), + canvas_completion: Some(EditorCanvasGenerationCompletionPayload { + dialog_id: Some("dialog-1".to_string()), + title: "角色动作".to_string(), + placeholder: EditorCanvasGenerationPlaceholderPayload { + x: 10.0, + y: 20.0, + width: 420.0, + height: 420.0, + original_width: 1024.0, + original_height: 1024.0, + }, + }), + generation_inputs: None, + source_resource_id: Some("resource-source".to_string()), }) .expect("request should serialize"); @@ -1002,6 +1081,9 @@ mod tests { ); assert!(payload.get("priceMudPoints").is_none()); assert_eq!(payload["model"], json!("seedance2.0-fast")); + assert_eq!(payload["projectId"], json!("editor-project-1")); + assert_eq!(payload["canvasCompletion"]["dialogId"], json!("dialog-1")); + assert_eq!(payload["sourceResourceId"], json!("resource-source")); } #[test] @@ -1024,6 +1106,8 @@ mod tests { frame_height: 1024, fps: 8, price_mud_points: 40, + project: None, + queue_state: None, }) .expect("response should serialize"); @@ -1052,6 +1136,11 @@ mod tests { reference_image_srcs: Vec::new(), reference_video_srcs: Vec::new(), reference_audio_srcs: Vec::new(), + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("request should serialize"); @@ -1078,6 +1167,11 @@ mod tests { "generated-character-drafts/editor/seedance-references/video/demo.mp4".to_string(), ], reference_audio_srcs: vec!["data:audio/mpeg;base64,audio".to_string()], + project_id: None, + canvas_completion: None, + generation_inputs: None, + source_resource_id: None, + asset_kind: None, }) .expect("request should serialize"); @@ -1100,7 +1194,9 @@ mod tests { let payload = serde_json::to_value(EditorVideoGenerateResponse { ok: true, video_src: "/generated-editor-videos/task-1/preview.mp4".to_string(), - object_key: Some("generated-character-drafts/editor-videos/task-1/preview.mp4".to_string()), + object_key: Some( + "generated-character-drafts/editor-videos/task-1/preview.mp4".to_string(), + ), asset_object_id: Some("assetobj_video_1".to_string()), width: 1280, height: 720, @@ -1113,6 +1209,8 @@ mod tests { duration_seconds: 5, resolution: "480p".to_string(), price_mud_points: 100, + project: None, + queue_state: None, }) .expect("response should serialize"); @@ -1178,6 +1276,9 @@ mod tests { prompt: "金币掉落叮当声".to_string(), model: Some("audio1.0".to_string()), duration: 7, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect("sound request should serialize"); assert_eq!(sound_payload["prompt"], json!("金币掉落叮当声")); @@ -1198,6 +1299,9 @@ mod tests { prompt: "按钮确认短促音".to_string(), model: None, duration: 5, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect("sound request with unset model should serialize"); assert!(unset_model_payload.get("model").is_none()); @@ -1206,6 +1310,9 @@ mod tests { let music_payload = serde_json::to_value(EditorBackgroundMusicGenerateRequest { gpt_description_prompt: "森林冒险背景音乐".to_string(), make_instrumental: true, + project_id: None, + canvas_completion: None, + generation_inputs: None, }) .expect("background music request should serialize"); assert_eq!( @@ -1229,6 +1336,8 @@ mod tests { task_id: "sound-task-1".to_string(), price_mud_points: 10, audio_kind: "sound-effect".to_string(), + project: None, + queue_state: None, }) .expect("audio response should serialize"); assert_eq!( diff --git a/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts b/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts index cdb30bff1..0079b4120 100644 --- a/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts +++ b/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts @@ -11,6 +11,7 @@ import { resolveEditorImageReferenceDataUrlForGeneration, } from '../../services/image-editor/editorImageReference'; import type { + EditorCanvasGenerationCompletionInput, EditorAssetSnapshot, EditorProjectSnapshot, } from '../../services/image-editor/editorProjectClient'; @@ -23,7 +24,10 @@ import { generateEditorImage, generateEditorSoundEffect, generateEditorVideo, + loadEditorProject, } from '../../services/image-editor/editorProjectClient'; +import { getExternalGenerationJobStatus } from '../../services/external-generation'; +import type { ExternalGenerationJobStatusRecord } from '../../../packages/shared/src/contracts/externalGeneration'; import type { CanvasGenerationDialogState, CanvasGenerationInputs, @@ -87,6 +91,9 @@ type UiDesignAssetExtractionOptions = { suppressAlert?: boolean; }; +const EDITOR_GENERATION_QUEUE_POLL_INTERVAL_MS = 1600; +const EDITOR_GENERATION_QUEUE_TIMEOUT_MS = 20 * 60 * 1000; + type GenerationSubmissionWorkflowOptions = { layers: CanvasLayer[]; canvasSize: CanvasSize; @@ -147,6 +154,86 @@ async function normalizePublicationReferenceImages( }; } +function delay(ms: number) { + return new Promise((resolve) => window.setTimeout(resolve, ms)); +} + +function queuedStateFromResponse( + response: { queueState?: ExternalGenerationJobStatusRecord | null } | null, +) { + return response?.queueState ?? null; +} + +async function waitForEditorGenerationQueue( + queueState: ExternalGenerationJobStatusRecord, +) { + const startedAt = Date.now(); + let current = queueState; + while (current.status === 'queued' || current.status === 'running') { + if (Date.now() - startedAt > EDITOR_GENERATION_QUEUE_TIMEOUT_MS) { + throw new Error('生成任务仍在队列中，请稍后刷新画布查看结果'); + } + await delay(EDITOR_GENERATION_QUEUE_POLL_INTERVAL_MS); + current = ( + await getExternalGenerationJobStatus(current.operationId) + ).job; + } + if (current.status === 'failed') { + throw new Error(current.error?.trim() || '生成任务失败'); + } + return current; +} + +async function applyQueuedEditorGenerationProject( + response: { queueState?: ExternalGenerationJobStatusRecord | null }, + projectId: string | null | undefined, + applyProjectSnapshot: ((project: EditorProjectSnapshot) => void) | undefined, +) { + const queueState = queuedStateFromResponse(response); + if (!queueState) { + return false; + } + await waitForEditorGenerationQueue(queueState); + if (projectId && applyProjectSnapshot) { + applyProjectSnapshot(await loadEditorProject(projectId)); + } + return true; +} + +function positiveCanvasSize(value: number | null | undefined, fallback = 1) { + const resolved = Number.isFinite(value) && (value ?? 0) > 0 ? value : fallback; + return Math.max(1, Math.round(resolved ?? 1)); +} + +function buildRightSideCanvasCompletionPlaceholder( + sourceLayer: CanvasLayer, + size?: { + width?: number; + height?: number; + originalWidth?: number; + originalHeight?: number; + }, +): EditorCanvasGenerationCompletionInput['placeholder'] { + const originalWidth = positiveCanvasSize( + size?.originalWidth ?? size?.width, + sourceLayer.originalWidth || sourceLayer.width || 1, + ); + const originalHeight = positiveCanvasSize( + size?.originalHeight ?? size?.height, + sourceLayer.originalHeight || sourceLayer.height || 1, + ); + const width = positiveCanvasSize(size?.width, originalWidth); + const height = positiveCanvasSize(size?.height, originalHeight); + return { + x: sourceLayer.x + sourceLayer.width + 32, + y: sourceLayer.y, + width, + height, + originalWidth, + originalHeight, + }; +} + function isVideoQuickEditSource(layer: CanvasLayer) { return layer.mediaType === 'video' || layer.assetKind === 'video'; } @@ -589,15 +676,6 @@ export function useImageCanvasGenerationSubmissionWorkflow({ const extractionPlan = resolveUiAssetExtractionGenerationPlan( options.marks?.length ?? 0, ); - const generated = await extractEditorUiDesignAssets({ - sourceImageSrc, - projectId, - generationInputs, - assetFolderId, - spritesheetLabel: `${sourceLayer.title} 素材图集`, - aspectRatio: extractionPlan.aspectRatio, - imageSize: extractionPlan.imageSize, - }); const extractionFrame = { x: sourceLayer.x + sourceLayer.width + 32, y: sourceLayer.y, @@ -606,6 +684,36 @@ export function useImageCanvasGenerationSubmissionWorkflow({ originalWidth: ICON_FRAME_ORIGINAL_SIZE.width, originalHeight: ICON_FRAME_ORIGINAL_SIZE.height, }; + const generated = await extractEditorUiDesignAssets({ + sourceImageSrc, + projectId, + generationInputs, + assetFolderId, + spritesheetLabel: `${sourceLayer.title} 素材图集`, + aspectRatio: extractionPlan.aspectRatio, + imageSize: extractionPlan.imageSize, + ...(projectId + ? { + canvasCompletion: { + title: `${sourceLayer.title} 素材图集`, + placeholder: extractionFrame, + }, + } + : {}), + }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } const nextLayers = addIconSpritesheetResultLayers( generated, generated.iconImageSrcs, @@ -630,7 +738,13 @@ export function useImageCanvasGenerationSubmissionWorkflow({ ); } }, - [addIconSpritesheetResultLayers, fitLayers, projectId, assetFolderId], + [ + addIconSpritesheetResultLayers, + applyProjectSnapshot, + fitLayers, + projectId, + assetFolderId, + ], ); const submitIconSpritesheetGeneration = useCallback( @@ -671,18 +785,42 @@ export function useImageCanvasGenerationSubmissionWorkflow({ }); try { + const canvasCompletionPlaceholder = + getGeneratingDialogPlaceholder(dialog); const generated = await generateEditorIconSpritesheet({ ...submissionPlan.input, projectId, generationInputs: submissionPlan.generationInputs, assetFolderId, + ...(projectId && canvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: canvasDialog.id, + title: '图标素材图集', + placeholder: canvasCompletionPlaceholder, + }, + } + : {}), }); rememberImageModel(submissionPlan.rememberImageModel); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addIconSpritesheetResultLayers( generated, generated.iconImageSrcs, submissionPlan.generationInputs, - getGeneratingDialogPlaceholder(dialog), + canvasCompletionPlaceholder, canvasDialog.id, ); } catch (error) { @@ -697,6 +835,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({ }, [ addIconSpritesheetResultLayers, + applyProjectSnapshot, getGeneratingDialogPlaceholder, rememberImageModel, projectId, @@ -801,6 +940,14 @@ export function useImageCanvasGenerationSubmissionWorkflow({ status: 'generating', }) : undefined; + const quickEditCanvasCompletionPlaceholder = + quickEditFrame ?? + buildRightSideCanvasCompletionPlaceholder(quickEditSourceLayer, { + width: quickEditOutputSize.width, + height: quickEditOutputSize.height, + originalWidth: quickEditOutputSize.width, + originalHeight: quickEditOutputSize.height, + }); if ( panelMode === 'quick-edit' && isCharacterAnimationQuickEditSource(quickEditSourceLayer) @@ -824,7 +971,32 @@ export function useImageCanvasGenerationSubmissionWorkflow({ frameCount: animationFrameCount, durationSeconds: animationDurationSeconds, model: CHARACTER_ANIMATION_MODEL, + projectId, + generationInputs, + sourceResourceId: quickEditSourceLayer.resourceId, + ...(projectId && quickEditCanvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: quickEditDialogId, + title: `${quickEditSourceLayer.title} 快速编辑`, + placeholder: quickEditCanvasCompletionPlaceholder, + }, + } + : {}), }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addCharacterAnimationResultLayer( generated, `${quickEditSourceLayer.title} 快速编辑`, @@ -862,7 +1034,33 @@ export function useImageCanvasGenerationSubmissionWorkflow({ ? { referenceImageSrcs: extraReferenceImageSrcs } : {}), referenceVideoSrcs: [sourceVideoSrc], + projectId, + generationInputs, + sourceResourceId: quickEditSourceLayer.resourceId, + assetKind: quickEditSourceLayer.assetKind, + ...(projectId && quickEditCanvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: quickEditDialogId, + title: `${quickEditSourceLayer.title} 快速编辑`, + placeholder: quickEditCanvasCompletionPlaceholder, + }, + } + : {}), }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addVideoResultLayer( generated, `${quickEditSourceLayer.title} 快速编辑`, @@ -927,7 +1125,31 @@ export function useImageCanvasGenerationSubmissionWorkflow({ panelMode === 'redraw' ? '重绘' : '快速编辑' }`, sourceResourceId: quickEditSourceLayer.resourceId, + ...(projectId && quickEditCanvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: quickEditDialogId, + title: `${quickEditSourceLayer.title} ${ + panelMode === 'redraw' ? '重绘' : '快速编辑' + }`, + placeholder: quickEditCanvasCompletionPlaceholder, + }, + } + : {}), }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addQuickEditResultLayer( generated, quickEditSourceLayer, @@ -967,6 +1189,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({ addCharacterAnimationResultLayer, addQuickEditResultLayer, addVideoResultLayer, + applyProjectSnapshot, canvasGenerationDialogs, canvasSize, getGeneratingDialogPlaceholder, @@ -1022,6 +1245,19 @@ export function useImageCanvasGenerationSubmissionWorkflow({ submissionPlan.sourceLayer.objectKey?.trim() || submissionPlan.sourceLayer.src, ); + const canvasCompletionPlaceholder = + getGeneratingDialogPlaceholder(dialog); + const editCanvasCompletionPlaceholder = + canvasCompletionPlaceholder ?? + buildRightSideCanvasCompletionPlaceholder( + submissionPlan.sourceLayer, + { + width: 1024, + height: 1024, + originalWidth: 1024, + originalHeight: 1024, + }, + ); const generated = await editEditorImage({ prompt: submissionPlan.normalizedPrompt, sourceImageSrc: referenceImageSrc, @@ -1031,30 +1267,121 @@ export function useImageCanvasGenerationSubmissionWorkflow({ assetFolderId, assetLabel: `${submissionPlan.sourceLayer.title} 修改结果`, sourceResourceId: submissionPlan.sourceLayer.resourceId, + ...(projectId && editCanvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: canvasDialog?.id, + title: `${submissionPlan.sourceLayer.title} 修改结果`, + placeholder: editCanvasCompletionPlaceholder, + }, + } + : {}), }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addGeneratedResultLayer(generated, { sourceLayer: submissionPlan.sourceLayer, generationInputs: submissionPlan.generationInputs, }); } else if (submissionPlan.kind === 'video') { - const generated = await generateEditorVideo(submissionPlan.input); + const canvasCompletionPlaceholder = + getGeneratingDialogPlaceholder(dialog); + const generated = await generateEditorVideo({ + ...submissionPlan.input, + projectId, + generationInputs: submissionPlan.result.generationInputs, + ...(projectId && canvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: canvasDialog?.id, + title: submissionPlan.result.title, + placeholder: canvasCompletionPlaceholder, + }, + } + : {}), + }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addVideoResultLayer( generated, submissionPlan.result.title, submissionPlan.result.generationInputs, - getGeneratingDialogPlaceholder(dialog), + canvasCompletionPlaceholder, canvasDialog?.id, ); } else if (submissionPlan.kind === 'audio') { + const canvasCompletionPlaceholder = + getGeneratingDialogPlaceholder(dialog); const generated = submissionPlan.audioKind === 'sound-effect' - ? await generateEditorSoundEffect(submissionPlan.input) - : await generateEditorBackgroundMusic(submissionPlan.input); + ? await generateEditorSoundEffect({ + ...submissionPlan.input, + projectId, + generationInputs: submissionPlan.result.generationInputs, + ...(projectId && canvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: canvasDialog?.id, + title: submissionPlan.result.title, + placeholder: canvasCompletionPlaceholder, + }, + } + : {}), + }) + : await generateEditorBackgroundMusic({ + ...submissionPlan.input, + projectId, + generationInputs: submissionPlan.result.generationInputs, + ...(projectId && canvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: canvasDialog?.id, + title: submissionPlan.result.title, + placeholder: canvasCompletionPlaceholder, + }, + } + : {}), + }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (generated.project && applyProjectSnapshot) { + applyProjectSnapshot(generated.project); + return; + } addAudioResultLayer( generated, submissionPlan.result.title, submissionPlan.result.generationInputs, - getGeneratingDialogPlaceholder(dialog), + canvasCompletionPlaceholder, canvasDialog?.id, ); } else { @@ -1074,16 +1401,25 @@ export function useImageCanvasGenerationSubmissionWorkflow({ generationInputs: submissionPlan.result.generationInputs, assetFolderId, assetLabel: resultTitle, - ...(canvasDialog && projectId && canvasCompletionPlaceholder + ...(projectId && canvasCompletionPlaceholder ? { canvasCompletion: { - dialogId: canvasDialog.id, + dialogId: canvasDialog?.id, title: resultTitle, placeholder: canvasCompletionPlaceholder, }, } : {}), }); + if ( + await applyQueuedEditorGenerationProject( + generated, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } if (submissionPlan.rememberImageModel) { rememberImageModel(submissionPlan.rememberImageModel); } @@ -1190,18 +1526,47 @@ export function useImageCanvasGenerationSubmissionWorkflow({ } try { - const result = await generateEditorCharacterAnimation( - submissionPlan.input, + const generationInputs = buildCharacterAnimationGenerationInputs( + submissionPlan.promptText, + characterAnimationSourceLayer, ); + const canvasCompletionPlaceholder = canvasDialog + ? getGeneratingDialogPlaceholder(canvasDialog) + : undefined; + const result = await generateEditorCharacterAnimation({ + ...submissionPlan.input, + projectId, + generationInputs, + sourceResourceId: characterAnimationSourceLayer.resourceId, + ...(canvasDialog && projectId && canvasCompletionPlaceholder + ? { + canvasCompletion: { + dialogId: canvasDialog.id, + title: '角色动作', + placeholder: canvasCompletionPlaceholder, + }, + } + : {}), + }); + if ( + await applyQueuedEditorGenerationProject( + result, + projectId, + applyProjectSnapshot, + ) + ) { + return; + } + if (result.project && applyProjectSnapshot) { + applyProjectSnapshot(result.project); + return; + } if (canvasDialog) { addCharacterAnimationResultLayer( result, '角色动作', - buildCharacterAnimationGenerationInputs( - submissionPlan.promptText, - characterAnimationSourceLayer, - ), - getGeneratingDialogPlaceholder(canvasDialog), + generationInputs, + canvasCompletionPlaceholder, canvasDialog.id, { sourceLayer: characterAnimationSourceLayer, @@ -1256,6 +1621,8 @@ export function useImageCanvasGenerationSubmissionWorkflow({ characterAnimationPanel, characterAnimationSourceLayer, getGeneratingDialogPlaceholder, + applyProjectSnapshot, + projectId, setCharacterAnimationPanel, updateCanvasGenerationDialogById, ]); diff --git a/src/services/image-editor/editorProjectClient.ts b/src/services/image-editor/editorProjectClient.ts index c7113d315..054fdf5f1 100644 --- a/src/services/image-editor/editorProjectClient.ts +++ b/src/services/image-editor/editorProjectClient.ts @@ -1,4 +1,5 @@ import { requestJson } from '../apiClient'; +import type { ExternalGenerationJobStatusRecord } from '../../../packages/shared/src/contracts/externalGeneration'; const EDITOR_PROJECT_API_BASE = '/api/editor/projects'; const EDITOR_ASSET_API_BASE = '/api/editor/assets'; @@ -135,7 +136,7 @@ export type EditorImageGenerationInput = { }; export type EditorCanvasGenerationCompletionInput = { - dialogId: string; + dialogId?: string; title: string; placeholder: { x: number; @@ -156,6 +157,7 @@ export type EditorIconSpritesheetGenerationInput = { projectId?: string | null; generationInputs?: EditorAssetGenerationInputs | null; assetFolderId?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; }; export type EditorUiDesignAssetExtractionInput = { @@ -166,6 +168,7 @@ export type EditorUiDesignAssetExtractionInput = { generationInputs?: EditorAssetGenerationInputs | null; assetFolderId?: string | null; spritesheetLabel?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; }; export type EditorImageEditInput = { @@ -179,6 +182,7 @@ export type EditorImageEditInput = { assetFolderId?: string | null; assetLabel?: string | null; sourceResourceId?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; }; export type EditorImageGenerationResult = { @@ -196,6 +200,7 @@ export type EditorImageGenerationResult = { resource?: EditorProjectResourceSnapshot | null; asset?: EditorAssetSnapshot | null; project?: EditorProjectSnapshot | null; + queueState?: ExternalGenerationJobStatusRecord | null; }; export type EditorIconSpritesheetIconResult = { @@ -220,6 +225,8 @@ export type EditorIconSpritesheetGenerationResult = { priceMudPoints: number; spritesheetResource?: EditorProjectResourceSnapshot | null; spritesheetAsset?: EditorAssetSnapshot | null; + project?: EditorProjectSnapshot | null; + queueState?: ExternalGenerationJobStatusRecord | null; }; export type EditorCharacterAnimationResolution = '480p' | '720p'; @@ -247,6 +254,10 @@ export type EditorCharacterAnimationGenerationInput = { frameCount: EditorCharacterAnimationFrameCount; durationSeconds: EditorCharacterAnimationDurationSeconds; model: 'seedance2.0-fast'; + projectId?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; + generationInputs?: EditorAssetGenerationInputs | null; + sourceResourceId?: string | null; }; export type EditorCharacterAnimationFrameResult = { @@ -266,6 +277,8 @@ export type EditorCharacterAnimationGenerationResult = { durationSeconds: number; fps: number; priceMudPoints: number; + project?: EditorProjectSnapshot | null; + queueState?: ExternalGenerationJobStatusRecord | null; }; export type EditorVideoModel = @@ -298,6 +311,11 @@ export type EditorVideoGenerationInput = { referenceImageSrcs?: string[]; referenceVideoSrcs?: string[]; referenceAudioSrcs?: string[]; + projectId?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; + generationInputs?: EditorAssetGenerationInputs | null; + sourceResourceId?: string | null; + assetKind?: string | null; }; export type EditorVideoGenerationResult = { @@ -313,17 +331,25 @@ export type EditorVideoGenerationResult = { provider: string; taskId: string; priceMudPoints?: number; + project?: EditorProjectSnapshot | null; + queueState?: ExternalGenerationJobStatusRecord | null; }; export type EditorSoundEffectGenerationInput = { prompt: string; model: 'audio1.0'; duration: number; + projectId?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; + generationInputs?: EditorAssetGenerationInputs | null; }; export type EditorBackgroundMusicGenerationInput = { gptDescriptionPrompt: string; makeInstrumental: true; + projectId?: string | null; + canvasCompletion?: EditorCanvasGenerationCompletionInput | null; + generationInputs?: EditorAssetGenerationInputs | null; }; export type EditorAudioGenerationResult = { @@ -341,6 +367,8 @@ export type EditorAudioGenerationResult = { priceMudPoints: number; audioKind: 'sound-effect' | 'background-music'; durationSeconds?: number | null; + project?: EditorProjectSnapshot | null; + queueState?: ExternalGenerationJobStatusRecord | null; }; export type EditorGenerationPricingUnit = 'perGeneration' | 'perSecond'; @@ -699,6 +727,9 @@ export async function generateEditorIconSpritesheet( ? { generationInputs: input.generationInputs } : {}), ...(input.assetFolderId ? { assetFolderId: input.assetFolderId } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), }), '生成图标素材失败', { @@ -727,6 +758,9 @@ export async function extractEditorUiDesignAssets( : {}), ...(input.aspectRatio ? { aspectRatio: input.aspectRatio } : {}), ...(input.imageSize ? { imageSize: input.imageSize } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), }), '提取UI设计图素材失败', { @@ -756,6 +790,9 @@ export async function editEditorImage(input: EditorImageEditInput) { ...(input.sourceResourceId ? { sourceResourceId: input.sourceResourceId } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), }), '修改图片失败', { @@ -772,7 +809,19 @@ export async function generateEditorCharacterAnimation( ) { return requestJson( EDITOR_CHARACTER_ANIMATION_GENERATION_API, - jsonRequest('POST', input), + jsonRequest('POST', { + ...input, + ...(input.projectId ? { projectId: input.projectId } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), + ...(input.generationInputs + ? { generationInputs: input.generationInputs } + : {}), + ...(input.sourceResourceId + ? { sourceResourceId: input.sourceResourceId } + : {}), + }), '生成角色动画失败', { timeoutMs: 1_200_000, @@ -804,6 +853,17 @@ export async function generateEditorVideo(input: EditorVideoGenerationInput) { ...(input.referenceAudioSrcs?.length ? { referenceAudioSrcs: input.referenceAudioSrcs } : {}), + ...(input.projectId ? { projectId: input.projectId } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), + ...(input.generationInputs + ? { generationInputs: input.generationInputs } + : {}), + ...(input.sourceResourceId + ? { sourceResourceId: input.sourceResourceId } + : {}), + ...(input.assetKind ? { assetKind: input.assetKind } : {}), }), '生成视频失败', { @@ -824,6 +884,13 @@ export async function generateEditorSoundEffect( prompt: input.prompt, model: input.model, duration: input.duration, + ...(input.projectId ? { projectId: input.projectId } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), + ...(input.generationInputs + ? { generationInputs: input.generationInputs } + : {}), }), '生成游戏音效失败', { @@ -843,6 +910,13 @@ export async function generateEditorBackgroundMusic( jsonRequest('POST', { gptDescriptionPrompt: input.gptDescriptionPrompt, makeInstrumental: input.makeInstrumental, + ...(input.projectId ? { projectId: input.projectId } : {}), + ...(input.canvasCompletion + ? { canvasCompletion: input.canvasCompletion } + : {}), + ...(input.generationInputs + ? { generationInputs: input.generationInputs } + : {}), }), '生成游戏背景音乐失败', {