diff --git a/docs/project-memory/plans/【实施计划】SFX生成优化V2.0任务拆解-2026-08-06.md b/docs/project-memory/plans/【实施计划】SFX生成优化V2.0任务拆解-2026-08-06.md index 00099725d..fe7a6592b 100644 --- a/docs/project-memory/plans/【实施计划】SFX生成优化V2.0任务拆解-2026-08-06.md +++ b/docs/project-memory/plans/【实施计划】SFX生成优化V2.0任务拆解-2026-08-06.md @@ -2,7 +2,7 @@ 日期:`2026-08-06` -状态:`T1、T2、T3 已完成,可以继续进入 T4;T2 翻译 service 与 T3 ElevenLabs adapter 均待 T5 正式接线,当前仍不是可发布切点` +状态:`T1、T2、T3、T4 已完成,可以继续进入 T5;T2 翻译 service、T3 ElevenLabs adapter 与 T4 前端冻结参数均待 T5 正式接线,当前仍不是可发布切点` 开发分支:`feat/sound_opt` @@ -163,6 +163,12 @@ External v1 `model` 先删除首尾 Unicode `White_Space`,再按大小写敏 - 在第一个 await 前取得 AI / 提交 operation,只锁当前 SFX dialog;迟到响应和 scope 切换不写新面板。 - T4 不切换 provider,不是可发布切点;与 T5 同一发布列车。 +实施记录(`2026-08-07`):T4 已完成。前端增加独立于 BGM 的 dialog-scoped SFX Prompt 状态模型与 controller,优化和提交都在第一个 `await` 前同步取得 operation;账号、项目、dialog、mode 和 `AbortController` 共同隔离迟到响应。优化成功形成一层 canonical Prompt 交换快照,失败清除本次临时快照且不恢复更早快照,预设写入清快照。现有 BGM 跑马灯已抽出无业务语义的音频内核,BGM / SFX 各保留 wrapper;SFX wrapper 展示 T1 冻结的 `40 + 12` 预设。 + +共享音频 composer 的 SFX 分支现已展示 `0 / 2048` 计数、一键优化、单层撤销、自动 / 手动时长、`0.5-30s` 且 `0.1s` 步进的 slider、Loop、固定 `ElevenLabs` 胶囊和新模型前端 `5` 泥点兜底。dialog layout 保存并恢复 `soundDurationMode / soundDurationSeconds / soundLoop`;历史 Vidu dialog 和改造入口统一打开 SFX V2 模型面板。同步提交 claim 冻结 canonical Prompt、时长模式、最近手动值和 Loop,只锁当前 dialog,并在 scope 失效后拒绝旧 UI 写回。 + +T4 没有修改 Worker、provider 调用、正式请求的 nullable duration / Loop 映射、服务端动态定价、计费、OSS、持久化详情、External v1、OpenAPI 或 SpacetimeDB schema;这些继续严格归属 T5。T4 单独合入仍不是可发布切点,也未执行真实 LLM、ElevenLabs 或付费生成。 + ### T5:正式提交、Worker、计费、持久化、详情和 External v1 - 前端提交冻结 canonical Prompt、duration 和 Loop,正式 POST 不 unsafe retry。 diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 6ffb6ea08..750acd5eb 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -6751,3 +6751,11 @@ - 翻译 service:在现有 `vector_engine_audio_generation` 内新增仅 crate 内部生成流水线可见、没有同步 HTTP 路由的翻译 service。每个业务语义轮固定 Luna、OpenAI Chat、Low、`max_tokens=8192`,不发送 temperature 或 tools;候选严格执行五字段结构、`true / true / true / false`、`1-2048` code points、至少一个 Latin alphabetic code point,且所有 alphabetic code point 都属于 Latin Script。Han、假名、韩文、西里尔、希腊和阿拉伯字母均失败,Common / Inherited 数字、标点、空白和符号允许。 - 轮次与错误:首轮成功返回但结构、判断、Script、长度或 `finish_reason=length` 不合格时,只以同一 canonical `userPrompt` 开启唯一第二业务语义轮;不得读取或传递首轮候选。首轮 `content_filter` 和 transport / timeout / 上游最终失败直接结束;`LlmClient` 内部 transport retry 仍属于当前业务轮。第二轮任何失败均最终失败。service 错误展示只提供 `translation_invalid / translation_upstream_failed` 分类和安全中文消息,不保存或输出未通过候选。 - 阶段边界:T2 没有调用 ElevenLabs、创建额外任务、扣费、修改队列 payload、持久化 `actual_prompt` 或变更 External v1 / OpenAPI / SpacetimeDB schema。T5 接入正式 Worker 时必须移除 T2 的 staged dead-code 豁免,并把翻译结果作为唯一 `actualPrompt` 进入 provider;T2 单独合入仍不是可发布切点。 + +## 2026-08-07 SFX 生成优化 V2.0 T4 前端 controller 与参数 UI + +- 状态模型:SFX 使用独立于 BGM 的纯状态模型和 dialog-scoped controller。优化与提交均在第一个 `await` 前同步 claim operation;账号、项目、dialog、mode 与 `AbortController` 共同判定响应归属。关闭、删除、mode / scope 切换后的旧响应不能写回新面板;同一按钮双击只有第一个 operation 生效。 +- Prompt 与撤销:计数、优化、预设、撤销和提交统一复用 T1 的 Unicode `White_Space` canonicalizer 与 `1-2048` code point 规则。优化开始时以当前 canonical Prompt 替换旧快照,成功转为单层交换快照,失败清除本次临时快照且不恢复更早快照;预设写入清快照,手动编辑优化结果后仍可在两个 canonical 版本间反复交换。 +- 预设视图:BGM 预设跑马灯抽出无业务语义的音频内核,保留单一可访问控件队列、无缝滚动、hover、触摸、页面可见性和 reduced-motion 行为;BGM / SFX 各自保留 wrapper、预设模型和业务 class。SFX wrapper 展示固定 `40 + 12` 预设,不保存展开、滚动或 hover 状态。 +- 参数与布局:SFX 首次打开为手动 `5s`、Loop false;手动 slider 为 `0.5-30s`、步进 `0.1s`,自动模式禁用 slider 但保留最近手动值。layout 恢复 `soundDurationMode / soundDurationSeconds / soundLoop`,历史 Vidu dialog 与改造入口统一打开固定 `eleven_text_to_sound_v2` / `ElevenLabs` 面板。前端显示新模型 `5` 泥点兜底,正式价格仍以后端 T5 入队冻结值为真相。 +- 锁与阶段边界:优化、提交或既有生成态只锁当前 SFX dialog 的输入、预设、滚动、参数、撤销和生成。提交 claim 同步冻结 canonical Prompt、时长模式、最近手动值与 Loop;T4 不改变正式请求的 nullable duration / Loop 映射,不接 Worker 翻译或 ElevenLabs,不修改服务端动态定价、计费、OSS、持久化详情、External v1、OpenAPI 或 SpacetimeDB schema。T4 必须与 T5 同一发布列车,不能单独发布。 diff --git a/docs/【编辑器】画板音乐生成入口设计-2026-06-18.md b/docs/【编辑器】画板音乐生成入口设计-2026-06-18.md index fa28aba84..f848d8bce 100644 --- a/docs/【编辑器】画板音乐生成入口设计-2026-06-18.md +++ b/docs/【编辑器】画板音乐生成入口设计-2026-06-18.md @@ -2,7 +2,7 @@ 日期:`2026-06-18` -更新时间:`2026-08-06` +更新时间:`2026-08-07` ## 范围 @@ -12,7 +12,7 @@ 音效与背景音乐继续使用同一个音频 composer,并由组件内的 `isSoundEffect = dialog.mode === 'audio-sound-effect'` 隔离行为。共享视图不等于共享业务规则:BGM 继续使用 Suno、200 字 canonical Prompt、30 个预设、AI 补全 / 简化、单层撤销和方案 A 提交锁;SFX V2 固定使用 ElevenLabs `eleven_text_to_sound_v2`、52 个预设、一键优化、自动中译英、自动 / 手动时长和 Loop。两条路径的 Prompt 模型、controller、预设 wrapper、锁和提交契约必须分别维护,不得交叉复用业务状态。 -SFX V2 已完成产品与技术口径冻结,T0 已通过,T1、T2 已完成,可以继续进入 T3、T4;这不表示功能已上线。T2 已落地登录态一键优化 BFF 和仅生成流水线内部可见的 Worker 翻译 service,但翻译到正式 Worker、ElevenLabs、计费、持久化和 External v1 的接线仍属于 T5。在正式切换完成前,当前运行代码仍是 Vidu SFX V1 行为。历史 Vidu 素材继续只读展示;重绘时使用历史用户 Prompt 打开 SFX V2 面板,新任务统一走 ElevenLabs,不回退 Vidu。 +SFX V2 已完成产品与技术口径冻结,T0 已通过,T1、T2、T3、T4 已完成,可以继续进入 T5;这不表示功能已上线。T2 已落地登录态一键优化 BFF 和仅生成流水线内部可见的 Worker 翻译 service,T3 已落地 ElevenLabs 直接二进制 adapter,T4 已落地 dialog-scoped 前端 controller、52 预设、优化 / 撤销、自动 / 手动时长与 Loop UI。翻译到正式 Worker、ElevenLabs、计费、持久化和 External v1 的接线仍属于 T5。在正式切换完成前,当前正式生成代码仍是 Vidu SFX V1 行为,T4 前端也不得单独发布。历史 Vidu 素材继续只读展示;重绘时使用历史用户 Prompt 打开 SFX V2 面板,新任务统一走 ElevenLabs,不回退 Vidu。 ## 入口与交互 diff --git a/src/components/image-editor/ImageCanvasAudioPresetMarquee.tsx b/src/components/image-editor/ImageCanvasAudioPresetMarquee.tsx new file mode 100644 index 000000000..35c8128ba --- /dev/null +++ b/src/components/image-editor/ImageCanvasAudioPresetMarquee.tsx @@ -0,0 +1,489 @@ +import { + ChevronDown, + ChevronLeft, + ChevronRight, + ChevronUp, +} from 'lucide-react'; +import { + type PointerEvent as ReactPointerEvent, + type ReactNode, + useCallback, + useEffect, + useId, + useRef, + useState, +} from 'react'; + +import { PlatformActionButton } from '../common/PlatformActionButton'; + +export type AudioPromptPreset = { + id: string; + group: string; + label: string; +}; + +/** 多份等宽队列拼接实现无缝循环:首尾之间不跳回也不留白。 */ +const PRESET_QUEUE_COPIES = 3; +/** 初始 scrollLeft 正对中间队列,因此它同时承担唯一的键盘与屏幕阅读器入口。 */ +const PRESET_ACCESSIBLE_QUEUE_INDEX = 1; +/** 默认缓慢循环与 hover 加速的速度,单位 px/s。 */ +const PRESET_BASE_VELOCITY = 26; +/** + * 加速档按真机实测下调到原值 340 的 70%。左右 15% 控制区与左右箭头共用同一个速度: + * 需求要求“箭头和滚动栏两边都有加速滚动”,两者相邻,拆成两个速度会让指针在箭头与 + * 控制区之间移动时出现速度突变。 + */ +const PRESET_FAST_VELOCITY = 238; +/** 左右控制区各占 viewport 宽度的 15%,中间 70% 为暂停区。 */ +const PRESET_SIDE_ZONE_RATIO = 0.15; +/** 箭头点击的一次离散滚动量,按 viewport 宽度比例计算。 */ +const PRESET_ARROW_STEP_RATIO = 0.6; +/** 单帧最大步进:标签页被挂起后回到前台时不产生一次巨大跳变。 */ +const PRESET_MAX_FRAME_MS = 100; +/** + * 触摸抬手后判定“惯性已经停下”的空闲窗口。原生惯性可能持续一两秒,固定等待再恢复 + * rAF 一定会打断它,所以这里等的是滚动事件停止,而不是一个固定时长。 + */ +const PRESET_TOUCH_IDLE_MS = 160; +const PRESET_REDUCED_MOTION_QUERY = '(prefers-reduced-motion: reduce)'; +/** 桌面 hover 控制区只在真正支持悬停的精确指针设备上启用。 */ +const PRESET_HOVER_CONTROL_QUERY = '(hover: hover) and (pointer: fine)'; + +type AudioPresetZone = 'left' | 'center' | 'right' | null; + +function readsMediaQueryMatches(query: string) { + if ( + typeof window === 'undefined' || + typeof window.matchMedia !== 'function' + ) { + return false; + } + return window.matchMedia(query).matches; +} + +function useMediaQueryMatches(query: string) { + const [matches, setMatches] = useState(() => readsMediaQueryMatches(query)); + + useEffect(() => { + if ( + typeof window === 'undefined' || + typeof window.matchMedia !== 'function' + ) { + return undefined; + } + const mediaQuery = window.matchMedia(query); + // 运行中切换(系统偏好变化、外接鼠标插拔)也要立即生效,不能只读初值。 + setMatches(mediaQuery.matches); + const handleChange = (event: MediaQueryListEvent) => { + setMatches(event.matches); + }; + mediaQuery.addEventListener('change', handleChange); + + return () => { + mediaQuery.removeEventListener('change', handleChange); + }; + }, [query]); + + return matches; +} + +export function ImageCanvasAudioPresetMarquee< + TPreset extends AudioPromptPreset, +>({ + presets, + isLocked, + onSelectPreset, + headerTrailing, + getPresetClassName, +}: { + presets: readonly TPreset[]; + isLocked: boolean; + onSelectPreset: (preset: TPreset) => void; + /** 标题行右端的展示位。字数计数要和「预设」同排,但它属于 composer 的状态。 */ + headerTrailing?: ReactNode; + getPresetClassName: (preset: TPreset) => string; +}) { + const trackId = useId(); + const [isExpanded, setIsExpanded] = useState(false); + const [zone, setZone] = useState(null); + const [isKeyboardFocused, setIsKeyboardFocused] = useState(false); + const [isPageVisible, setIsPageVisible] = useState( + () => + typeof document === 'undefined' || document.visibilityState !== 'hidden', + ); + const prefersReducedMotion = useMediaQueryMatches( + PRESET_REDUCED_MOTION_QUERY, + ); + const supportsHoverControls = useMediaQueryMatches( + PRESET_HOVER_CONTROL_QUERY, + ); + const [isTouchScrolling, setIsTouchScrolling] = useState(false); + + const viewportRef = useRef(null); + const queueRef = useRef(null); + const queueWidthRef = useRef(0); + const frameRef = useRef(null); + const frameTimestampRef = useRef(null); + const touchSettleTimerRef = useRef | null>( + null, + ); + + // 归位只按实测的单份队列宽度取模,不硬编码 chip 或轨道宽度。 + const normalizeScrollPosition = useCallback(() => { + const viewport = viewportRef.current; + const queueWidth = queueWidthRef.current; + if (!viewport || queueWidth <= 0) { + return; + } + if (viewport.scrollLeft < queueWidth * 0.5) { + viewport.scrollLeft += queueWidth; + return; + } + if (viewport.scrollLeft > queueWidth * 1.5) { + viewport.scrollLeft -= queueWidth; + } + }, []); + + const velocity = (() => { + if (prefersReducedMotion || isLocked || !isExpanded || !isPageVisible) { + return 0; + } + if (isTouchScrolling || isKeyboardFocused || zone === 'center') { + return 0; + } + if (zone === 'left') { + return -PRESET_FAST_VELOCITY; + } + if (zone === 'right') { + return PRESET_FAST_VELOCITY; + } + return PRESET_BASE_VELOCITY; + })(); + + useEffect(() => { + if (velocity === 0) { + return undefined; + } + const step = (timestamp: number) => { + const previousTimestamp = frameTimestampRef.current ?? timestamp; + const deltaMs = Math.min( + Math.max(timestamp - previousTimestamp, 0), + PRESET_MAX_FRAME_MS, + ); + frameTimestampRef.current = timestamp; + const viewport = viewportRef.current; + if (viewport) { + viewport.scrollLeft += (velocity * deltaMs) / 1000; + normalizeScrollPosition(); + } + frameRef.current = requestAnimationFrame(step); + }; + frameRef.current = requestAnimationFrame(step); + + return () => { + frameTimestampRef.current = null; + if (frameRef.current !== null) { + cancelAnimationFrame(frameRef.current); + frameRef.current = null; + } + }; + }, [normalizeScrollPosition, velocity]); + + useEffect(() => { + if (typeof document === 'undefined') { + return undefined; + } + const handleVisibilityChange = () => { + setIsPageVisible(document.visibilityState !== 'hidden'); + }; + document.addEventListener('visibilitychange', handleVisibilityChange); + + return () => { + document.removeEventListener('visibilitychange', handleVisibilityChange); + }; + }, []); + + useEffect(() => { + const queue = queueRef.current; + if (!queue || typeof ResizeObserver === 'undefined') { + return undefined; + } + const observer = new ResizeObserver(() => { + const nextQueueWidth = queue.getBoundingClientRect().width; + if (nextQueueWidth <= 0) { + return; + } + const previousQueueWidth = queueWidthRef.current; + queueWidthRef.current = nextQueueWidth; + const viewport = viewportRef.current; + if (!viewport) { + return; + } + // 尺寸变化后按新宽度重新建立基准,避免停在旧队列的相位上。 + if (previousQueueWidth <= 0) { + viewport.scrollLeft = nextQueueWidth; + return; + } + normalizeScrollPosition(); + }); + observer.observe(queue); + + return () => { + observer.disconnect(); + }; + }, [isExpanded, normalizeScrollPosition]); + + useEffect( + () => () => { + if (touchSettleTimerRef.current !== null) { + clearTimeout(touchSettleTimerRef.current); + } + }, + [], + ); + + useEffect(() => { + if (isExpanded && !isLocked && supportsHoverControls) { + return; + } + // 收起、锁定或失去 hover 能力时回到默认状态。最后一种尤其重要:没有 hover 的 + // 设备不会再有 pointerleave,残留的 `center` 会让循环永久停住。 + setZone(null); + setIsKeyboardFocused(false); + }, [isExpanded, isLocked, supportsHoverControls]); + + const handleViewportPointerMove = ( + event: ReactPointerEvent, + ) => { + // 只在真正支持悬停的精确指针设备上启用;触摸指针再挡一道。控制区按 viewport + // 相对 X 计算,不在 chip 上方覆盖三个透明 div。 + if (!supportsHoverControls || event.pointerType === 'touch' || isLocked) { + return; + } + const bounds = event.currentTarget.getBoundingClientRect(); + if (bounds.width <= 0) { + return; + } + const ratio = (event.clientX - bounds.left) / bounds.width; + // 位置算不出来时保持当前区域:默认落到“中间暂停”会让指针一进入轨道就停住循环。 + if (!Number.isFinite(ratio)) { + return; + } + if (ratio < PRESET_SIDE_ZONE_RATIO) { + setZone('left'); + return; + } + if (ratio > 1 - PRESET_SIDE_ZONE_RATIO) { + setZone('right'); + return; + } + setZone('center'); + }; + + const handleViewportPointerLeave = ( + event: ReactPointerEvent, + ) => { + if (event.pointerType === 'touch') { + return; + } + setZone(null); + }; + + const armTouchSettleWatch = () => { + if (touchSettleTimerRef.current !== null) { + clearTimeout(touchSettleTimerRef.current); + } + touchSettleTimerRef.current = setTimeout(() => { + touchSettleTimerRef.current = null; + setIsTouchScrolling(false); + }, PRESET_TOUCH_IDLE_MS); + }; + + const beginTouchScroll = (event: ReactPointerEvent) => { + if (event.pointerType !== 'touch') { + return; + } + if (touchSettleTimerRef.current !== null) { + clearTimeout(touchSettleTimerRef.current); + touchSettleTimerRef.current = null; + } + setIsTouchScrolling(true); + }; + + const endTouchScroll = (event: ReactPointerEvent) => { + if (event.pointerType !== 'touch') { + return; + } + // 抬手只是开始等待:原生惯性可能还要跑一两秒,真正的恢复条件是滚动停止。 + armTouchSettleWatch(); + }; + + const handleViewportScroll = () => { + // 只有在等待惯性稳定的窗口内才顺延;否则收到的是 rAF 自己写 scrollLeft 触发的 + // scroll,会把恢复条件永远推后。 + if (touchSettleTimerRef.current === null) { + return; + } + armTouchSettleWatch(); + }; + + const scrollByArrow = (direction: -1 | 1) => { + const viewport = viewportRef.current; + if (!viewport) { + return; + } + const step = + viewport.getBoundingClientRect().width * PRESET_ARROW_STEP_RATIO; + // 离散滚动不使用平滑动画,供触摸和键盘使用。 + viewport.scrollLeft += direction * (step || PRESET_FAST_VELOCITY); + normalizeScrollPosition(); + }; + + return ( +
+
+ + 预设 + + setIsExpanded((expanded) => !expanded)} + > + {isExpanded ? ( + + {headerTrailing} +
+ {isExpanded ? ( +
+ +
setIsKeyboardFocused(true)} + onBlur={() => setIsKeyboardFocused(false)} + > + {Array.from({ length: PRESET_QUEUE_COPIES }, (_, copyIndex) => { + const isAccessibleQueue = + copyIndex === PRESET_ACCESSIBLE_QUEUE_INDEX; + return ( +
+ {presets.map((preset) => { + const className = getPresetClassName(preset); + if (isAccessibleQueue) { + return ( + + ); + } + return ( + { + if (!isLocked) { + onSelectPreset(preset); + } + }} + > + {preset.label} + + ); + })} +
+ ); + })} +
+ + + ) : null} +
+ ); +} diff --git a/src/components/image-editor/ImageCanvasBackgroundMusicPresetMarquee.tsx b/src/components/image-editor/ImageCanvasBackgroundMusicPresetMarquee.tsx index 7f42497ba..d338cc549 100644 --- a/src/components/image-editor/ImageCanvasBackgroundMusicPresetMarquee.tsx +++ b/src/components/image-editor/ImageCanvasBackgroundMusicPresetMarquee.tsx @@ -1,87 +1,8 @@ -import { - ChevronDown, - ChevronLeft, - ChevronRight, - ChevronUp, -} from 'lucide-react'; -import { - type PointerEvent as ReactPointerEvent, - type ReactNode, - useCallback, - useEffect, - useId, - useRef, - useState, -} from 'react'; +import type { ReactNode } from 'react'; -import { PlatformActionButton } from '../common/PlatformActionButton'; +import { ImageCanvasAudioPresetMarquee } from './ImageCanvasAudioPresetMarquee'; import type { BackgroundMusicPromptPreset } from './ImageCanvasBackgroundMusicPresetModel'; -/** 多份等宽队列拼接实现无缝循环:首尾之间不跳回也不留白。 */ -const PRESET_QUEUE_COPIES = 3; -/** 初始 scrollLeft 正对中间队列,因此它同时承担唯一的键盘与屏幕阅读器入口。 */ -const PRESET_ACCESSIBLE_QUEUE_INDEX = 1; -/** 默认缓慢循环与 hover 加速的速度,单位 px/s。 */ -const PRESET_BASE_VELOCITY = 26; -/** - * 加速档按真机实测下调到原值 340 的 70%。左右 15% 控制区与左右箭头共用同一个速度: - * 需求要求“箭头和滚动栏两边都有加速滚动”,两者相邻,拆成两个速度会让指针在箭头与 - * 控制区之间移动时出现速度突变。 - */ -const PRESET_FAST_VELOCITY = 238; -/** 左右控制区各占 viewport 宽度的 15%,中间 70% 为暂停区。 */ -const PRESET_SIDE_ZONE_RATIO = 0.15; -/** 箭头点击的一次离散滚动量,按 viewport 宽度比例计算。 */ -const PRESET_ARROW_STEP_RATIO = 0.6; -/** 单帧最大步进:标签页被挂起后回到前台时不产生一次巨大跳变。 */ -const PRESET_MAX_FRAME_MS = 100; -/** - * 触摸抬手后判定“惯性已经停下”的空闲窗口。原生惯性可能持续一两秒,固定等待再恢复 - * rAF 一定会打断它,所以这里等的是滚动事件停止,而不是一个固定时长。 - */ -const PRESET_TOUCH_IDLE_MS = 160; -const PRESET_REDUCED_MOTION_QUERY = '(prefers-reduced-motion: reduce)'; -/** 桌面 hover 控制区只在真正支持悬停的精确指针设备上启用。 */ -const PRESET_HOVER_CONTROL_QUERY = '(hover: hover) and (pointer: fine)'; - -type BackgroundMusicPresetZone = 'left' | 'center' | 'right' | null; - -function readsMediaQueryMatches(query: string) { - if ( - typeof window === 'undefined' || - typeof window.matchMedia !== 'function' - ) { - return false; - } - return window.matchMedia(query).matches; -} - -function useMediaQueryMatches(query: string) { - const [matches, setMatches] = useState(() => readsMediaQueryMatches(query)); - - useEffect(() => { - if ( - typeof window === 'undefined' || - typeof window.matchMedia !== 'function' - ) { - return undefined; - } - const mediaQuery = window.matchMedia(query); - // 运行中切换(系统偏好变化、外接鼠标插拔)也要立即生效,不能只读初值。 - setMatches(mediaQuery.matches); - const handleChange = (event: MediaQueryListEvent) => { - setMatches(event.matches); - }; - mediaQuery.addEventListener('change', handleChange); - - return () => { - mediaQuery.removeEventListener('change', handleChange); - }; - }, [query]); - - return matches; -} - export function ImageCanvasBackgroundMusicPresetMarquee({ presets, isLocked, @@ -91,390 +12,17 @@ export function ImageCanvasBackgroundMusicPresetMarquee({ presets: readonly BackgroundMusicPromptPreset[]; isLocked: boolean; onSelectPreset: (preset: BackgroundMusicPromptPreset) => void; - /** 标题行右端的展示位。字数计数要和「预设」同排,但它属于 composer 的状态。 */ headerTrailing?: ReactNode; }) { - const trackId = useId(); - const [isExpanded, setIsExpanded] = useState(false); - const [zone, setZone] = useState(null); - const [isKeyboardFocused, setIsKeyboardFocused] = useState(false); - const [isPageVisible, setIsPageVisible] = useState( - () => - typeof document === 'undefined' || document.visibilityState !== 'hidden', - ); - const prefersReducedMotion = useMediaQueryMatches( - PRESET_REDUCED_MOTION_QUERY, - ); - const supportsHoverControls = useMediaQueryMatches( - PRESET_HOVER_CONTROL_QUERY, - ); - const [isTouchScrolling, setIsTouchScrolling] = useState(false); - - const viewportRef = useRef(null); - const queueRef = useRef(null); - const queueWidthRef = useRef(0); - const frameRef = useRef(null); - const frameTimestampRef = useRef(null); - const touchSettleTimerRef = useRef | null>( - null, - ); - - // 归位只按实测的单份队列宽度取模,不硬编码 chip 或轨道宽度。 - const normalizeScrollPosition = useCallback(() => { - const viewport = viewportRef.current; - const queueWidth = queueWidthRef.current; - if (!viewport || queueWidth <= 0) { - return; - } - if (viewport.scrollLeft < queueWidth * 0.5) { - viewport.scrollLeft += queueWidth; - return; - } - if (viewport.scrollLeft > queueWidth * 1.5) { - viewport.scrollLeft -= queueWidth; - } - }, []); - - const velocity = (() => { - if (prefersReducedMotion || isLocked || !isExpanded || !isPageVisible) { - return 0; - } - if (isTouchScrolling || isKeyboardFocused || zone === 'center') { - return 0; - } - if (zone === 'left') { - return -PRESET_FAST_VELOCITY; - } - if (zone === 'right') { - return PRESET_FAST_VELOCITY; - } - return PRESET_BASE_VELOCITY; - })(); - - useEffect(() => { - if (velocity === 0) { - return undefined; - } - const step = (timestamp: number) => { - const previousTimestamp = frameTimestampRef.current ?? timestamp; - const deltaMs = Math.min( - Math.max(timestamp - previousTimestamp, 0), - PRESET_MAX_FRAME_MS, - ); - frameTimestampRef.current = timestamp; - const viewport = viewportRef.current; - if (viewport) { - viewport.scrollLeft += (velocity * deltaMs) / 1000; - normalizeScrollPosition(); - } - frameRef.current = requestAnimationFrame(step); - }; - frameRef.current = requestAnimationFrame(step); - - return () => { - frameTimestampRef.current = null; - if (frameRef.current !== null) { - cancelAnimationFrame(frameRef.current); - frameRef.current = null; - } - }; - }, [normalizeScrollPosition, velocity]); - - useEffect(() => { - if (typeof document === 'undefined') { - return undefined; - } - const handleVisibilityChange = () => { - setIsPageVisible(document.visibilityState !== 'hidden'); - }; - document.addEventListener('visibilitychange', handleVisibilityChange); - - return () => { - document.removeEventListener('visibilitychange', handleVisibilityChange); - }; - }, []); - - useEffect(() => { - const queue = queueRef.current; - if (!queue || typeof ResizeObserver === 'undefined') { - return undefined; - } - const observer = new ResizeObserver(() => { - const nextQueueWidth = queue.getBoundingClientRect().width; - if (nextQueueWidth <= 0) { - return; - } - const previousQueueWidth = queueWidthRef.current; - queueWidthRef.current = nextQueueWidth; - const viewport = viewportRef.current; - if (!viewport) { - return; - } - // 尺寸变化后按新宽度重新建立基准,避免停在旧队列的相位上。 - if (previousQueueWidth <= 0) { - viewport.scrollLeft = nextQueueWidth; - return; - } - normalizeScrollPosition(); - }); - observer.observe(queue); - - return () => { - observer.disconnect(); - }; - }, [isExpanded, normalizeScrollPosition]); - - useEffect( - () => () => { - if (touchSettleTimerRef.current !== null) { - clearTimeout(touchSettleTimerRef.current); - } - }, - [], - ); - - useEffect(() => { - if (isExpanded && !isLocked && supportsHoverControls) { - return; - } - // 收起、锁定或失去 hover 能力时回到默认状态。最后一种尤其重要:没有 hover 的 - // 设备不会再有 pointerleave,残留的 `center` 会让循环永久停住。 - setZone(null); - setIsKeyboardFocused(false); - }, [isExpanded, isLocked, supportsHoverControls]); - - const handleViewportPointerMove = ( - event: ReactPointerEvent, - ) => { - // 只在真正支持悬停的精确指针设备上启用;触摸指针再挡一道。控制区按 viewport - // 相对 X 计算,不在 chip 上方覆盖三个透明 div。 - if (!supportsHoverControls || event.pointerType === 'touch' || isLocked) { - return; - } - const bounds = event.currentTarget.getBoundingClientRect(); - if (bounds.width <= 0) { - return; - } - const ratio = (event.clientX - bounds.left) / bounds.width; - // 位置算不出来时保持当前区域:默认落到“中间暂停”会让指针一进入轨道就停住循环。 - if (!Number.isFinite(ratio)) { - return; - } - if (ratio < PRESET_SIDE_ZONE_RATIO) { - setZone('left'); - return; - } - if (ratio > 1 - PRESET_SIDE_ZONE_RATIO) { - setZone('right'); - return; - } - setZone('center'); - }; - - const handleViewportPointerLeave = ( - event: ReactPointerEvent, - ) => { - if (event.pointerType === 'touch') { - return; - } - setZone(null); - }; - - const armTouchSettleWatch = () => { - if (touchSettleTimerRef.current !== null) { - clearTimeout(touchSettleTimerRef.current); - } - touchSettleTimerRef.current = setTimeout(() => { - touchSettleTimerRef.current = null; - setIsTouchScrolling(false); - }, PRESET_TOUCH_IDLE_MS); - }; - - const beginTouchScroll = (event: ReactPointerEvent) => { - if (event.pointerType !== 'touch') { - return; - } - if (touchSettleTimerRef.current !== null) { - clearTimeout(touchSettleTimerRef.current); - touchSettleTimerRef.current = null; - } - setIsTouchScrolling(true); - }; - - const endTouchScroll = (event: ReactPointerEvent) => { - if (event.pointerType !== 'touch') { - return; - } - // 抬手只是开始等待:原生惯性可能还要跑一两秒,真正的恢复条件是滚动停止。 - armTouchSettleWatch(); - }; - - const handleViewportScroll = () => { - // 只有在等待惯性稳定的窗口内才顺延;否则收到的是 rAF 自己写 scrollLeft 触发的 - // scroll,会把恢复条件永远推后。 - if (touchSettleTimerRef.current === null) { - return; - } - armTouchSettleWatch(); - }; - - const scrollByArrow = (direction: -1 | 1) => { - const viewport = viewportRef.current; - if (!viewport) { - return; - } - const step = - viewport.getBoundingClientRect().width * PRESET_ARROW_STEP_RATIO; - // 离散滚动不使用平滑动画,供触摸和键盘使用。 - viewport.scrollLeft += direction * (step || PRESET_FAST_VELOCITY); - normalizeScrollPosition(); - }; - return ( -
-
- - 预设 - - setIsExpanded((expanded) => !expanded)} - > - {isExpanded ? ( - - {headerTrailing} -
- {isExpanded ? ( -
- -
setIsKeyboardFocused(true)} - onBlur={() => setIsKeyboardFocused(false)} - > - {Array.from({ length: PRESET_QUEUE_COPIES }, (_, copyIndex) => { - const isAccessibleQueue = - copyIndex === PRESET_ACCESSIBLE_QUEUE_INDEX; - return ( -
- {presets.map((preset) => { - const className = `image-canvas-editor__background-music-preset image-canvas-editor__background-music-preset--${preset.group}`; - if (isAccessibleQueue) { - return ( - - ); - } - return ( - { - if (!isLocked) { - onSelectPreset(preset); - } - }} - > - {preset.label} - - ); - })} -
- ); - })} -
- - - ) : null} -
+ + `image-canvas-editor__background-music-preset image-canvas-editor__background-music-preset--${preset.group}` + } + /> ); } diff --git a/src/components/image-editor/ImageCanvasEditorModel.test.ts b/src/components/image-editor/ImageCanvasEditorModel.test.ts index 333c9bd27..4b629df58 100644 --- a/src/components/image-editor/ImageCanvasEditorModel.test.ts +++ b/src/components/image-editor/ImageCanvasEditorModel.test.ts @@ -1130,7 +1130,10 @@ describe('ImageCanvasEditorModel', () => { prompt: '金币掉落叮当声', status: 'idle', composerOpen: true, - soundDurationSeconds: 8, + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'auto', + soundDurationSeconds: 7.3, + soundLoop: true, generatedLayerId: 'layer-audio', placeholder: { x: 100, @@ -1154,7 +1157,10 @@ describe('ImageCanvasEditorModel', () => { id: 'generation-dialog-audio', mode: 'audio-sound-effect', prompt: '金币掉落叮当声', - soundDurationSeconds: 8, + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'auto', + soundDurationSeconds: 7.3, + soundLoop: true, generatedLayerId: 'layer-audio', }); }); diff --git a/src/components/image-editor/ImageCanvasEditorModel.ts b/src/components/image-editor/ImageCanvasEditorModel.ts index 6dba4dcbf..bffe87927 100644 --- a/src/components/image-editor/ImageCanvasEditorModel.ts +++ b/src/components/image-editor/ImageCanvasEditorModel.ts @@ -1360,11 +1360,27 @@ export function hydrateCanvasGenerationDialog( characterAnimationResult: hydrateCharacterAnimationResult( snapshot.characterAnimationResult, ), - soundModel: snapshot.soundModel === 'audio1.0' ? 'audio1.0' : undefined, - soundDurationSeconds: soundDurationOrDefault( - snapshot.soundDurationSeconds, - snapshot.audioDurationSeconds, - ), + soundModel: + snapshot.mode === 'audio-sound-effect' + ? EDITOR_SOUND_EFFECT_MODEL + : undefined, + soundDurationMode: + snapshot.mode === 'audio-sound-effect' + ? snapshot.soundDurationMode === 'auto' + ? 'auto' + : 'manual' + : undefined, + soundDurationSeconds: + snapshot.mode === 'audio-sound-effect' + ? soundDurationOrDefault( + snapshot.soundDurationSeconds, + snapshot.audioDurationSeconds, + ) + : undefined, + soundLoop: + snapshot.mode === 'audio-sound-effect' + ? snapshot.soundLoop === true + : undefined, makeInstrumental: typeof snapshot.makeInstrumental === 'boolean' ? snapshot.makeInstrumental @@ -2020,7 +2036,10 @@ export function soundDurationOrDefault( if (!resolvedValue) { return undefined; } - return Math.min(10, Math.max(2, Math.round(resolvedValue))); + return Math.min( + SOUND_EFFECT_DURATION_MAX_SECONDS, + Math.max(SOUND_EFFECT_DURATION_MIN_SECONDS, resolvedValue), + ); } export function stringOrNull(value: unknown) { diff --git a/src/components/image-editor/ImageCanvasEditorTypes.ts b/src/components/image-editor/ImageCanvasEditorTypes.ts index 974e3bd62..da3ca3807 100644 --- a/src/components/image-editor/ImageCanvasEditorTypes.ts +++ b/src/components/image-editor/ImageCanvasEditorTypes.ts @@ -1,4 +1,8 @@ -import type { EditorSoundEffectGenerationMetadataV2 } from '../../../packages/shared/src/contracts/editorAudio'; +import type { + EditorSoundEffectDurationMode, + EditorSoundEffectGenerationMetadataV2, + EditorSoundEffectModel, +} from '../../../packages/shared/src/contracts/editorAudio'; import type { EditorAssetSnapshot, EditorCharacterAnimationFrameCount, @@ -249,9 +253,10 @@ export type GenerateDialogState = { characterAnimationFrameCount?: EditorCharacterAnimationFrameCount; characterAnimationDurationSeconds?: 4 | 5 | 6; characterAnimationResult?: EditorCharacterAnimationGenerationResult; - soundModel?: 'audio1.0'; - // 中文注释:Vidu 文生音频 duration 只允许 2-10 秒,默认 5 秒。 + soundModel?: EditorSoundEffectModel | 'audio1.0'; + soundDurationMode?: EditorSoundEffectDurationMode; soundDurationSeconds?: number; + soundLoop?: boolean; makeInstrumental?: boolean; audioDurationSeconds?: number | null; aspectRatio?: string; diff --git a/src/components/image-editor/ImageCanvasGenerationComposerView.test.tsx b/src/components/image-editor/ImageCanvasGenerationComposerView.test.tsx index 5619ee875..8ef17d601 100644 --- a/src/components/image-editor/ImageCanvasGenerationComposerView.test.tsx +++ b/src/components/image-editor/ImageCanvasGenerationComposerView.test.tsx @@ -21,6 +21,10 @@ import type { BackgroundMusicPromptAssistComposerController, BackgroundMusicPromptAssistDialogState, } from './useImageCanvasBackgroundMusicPromptAssist'; +import type { + SoundEffectPromptAssistComposerController, + SoundEffectPromptAssistDialogState, +} from './useImageCanvasSoundEffectPromptAssist'; function mockStateSetter() { return vi.fn() as unknown as Dispatch>; @@ -54,6 +58,7 @@ function createComposerProps( isPickingUiDesignSpecFromCanvas: false, isPickingPublicationReferenceFromCanvas: false, generateDialog, + soundEffectPromptAssist: createSoundEffectPromptAssistStub(), updateCanvasGenerationDialogById: vi.fn(), generationComposerStyle: { left: 320, top: 240 }, iconComposerStyle: { left: 320, top: 240, width: '32rem' }, @@ -172,6 +177,32 @@ function createBackgroundMusicPromptAssistStub( }; } +function createSoundEffectPromptAssistStub( + assistState: Partial = {}, + overrides: Partial = {}, +): SoundEffectPromptAssistComposerController { + return { + getDialogState: (dialogId: string) => ({ + dialogId, + status: 'idle' as const, + operationId: null, + undoPromptSnapshot: null, + temporaryPromptSnapshot: null, + errorMessage: null, + ...assistState, + }), + optimizePrompt: vi.fn(async () => ({ + started: true, + applied: true, + reason: 'applied' as const, + errorMessage: null, + })), + preparePreset: vi.fn(() => ''), + undoPrompt: vi.fn(() => true), + ...overrides, + }; +} + function renderBackgroundMusicComposer({ dialog = createBackgroundMusicDialog(), promptAssist = createBackgroundMusicPromptAssistStub(), @@ -877,21 +908,40 @@ describe('ImageCanvasGenerationComposerView', () => { ); }); - it('生成游戏音效面板使用 Vidu duration 参数且不再显示类型和 BPM', () => { + it('生成游戏音效面板展示 SFX V2 预设、参数和 ElevenLabs', () => { const onSubmitImageGeneration = vi.fn(); function AudioHarness() { const [dialog, setDialog] = useState({ + id: 'dialog-sfx', mode: 'audio-sound-effect', - prompt: '', + prompt: '开门声', status: 'idle', composerOpen: true, - soundModel: 'audio1.0', + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'manual', soundDurationSeconds: 5, + soundLoop: false, }); + const updateCanvasGenerationDialogById = ( + dialogId: string, + updater: ( + current: CanvasGenerationDialogState, + ) => CanvasGenerationDialogState | null, + ) => { + setDialog((current) => { + if (current.id !== dialogId) { + return current; + } + return updater(current as CanvasGenerationDialogState) ?? current; + }); + }; return ( <> > } @@ -914,36 +964,42 @@ describe('ImageCanvasGenerationComposerView', () => { expect(footer?.children[0]?.className).toContain( 'image-canvas-editor__option-popover-anchor--dimensions', ); - expect(footer?.children[1]?.className).toContain( + expect(footer?.children[2]?.className).toContain( 'image-canvas-editor__option-popover-anchor--model', ); - expect(footer?.children[2]?.className).toContain( + expect(footer?.children[3]?.className).toContain( 'image-canvas-editor__generation-submit', ); const prompt = within(panel).getByRole('textbox', { name: 'prompt' }); expect(prompt.className).toContain('auto-grow-text-area'); expect(prompt.className).not.toContain('platform-text-field'); - expect(screen.getByLabelText('当前音效模型').textContent).toBe('audio1.0'); + expect(screen.getByLabelText('当前音效模型').textContent).toBe( + 'eleven_text_to_sound_v2', + ); const soundModelButton = within(panel).getByRole('button', { - name: '音效模型 Vidu', + name: '音效模型 ElevenLabs', }); expect((soundModelButton as HTMLButtonElement).disabled).toBe(true); - expect(within(panel).getByText('Vidu')).toBeTruthy(); + expect(within(panel).getByText('ElevenLabs')).toBeTruthy(); expect(within(panel).queryByText(/Suno/u)).toBeNull(); expect( within(panel).getByRole('button', { name: '音效时长 5秒' }), ).toBeTruthy(); expect(within(panel).queryByText('单次')).toBeNull(); - expect(within(panel).queryByText('循环')).toBeNull(); + expect( + within(panel) + .getByRole('button', { name: 'Loop' }) + .getAttribute('aria-pressed'), + ).toBe('false'); expect(within(panel).queryByText(/BPM/u)).toBeNull(); expect(within(panel).queryByRole('button', { name: 'AI 补全' })).toBeNull(); expect( within(panel).queryByRole('button', { name: '一键简化' }), ).toBeNull(); expect( - within(panel).queryByRole('button', { name: '一键优化' }), - ).toBeNull(); - expect(within(panel).queryByRole('button', { name: '展开' })).toBeNull(); + within(panel).getByRole('button', { name: '一键优化' }), + ).toBeTruthy(); + expect(within(panel).getByRole('button', { name: '展开' })).toBeTruthy(); expect( within(panel).getByRole('button', { name: '生成游戏音效' }).textContent, ).toBe('生成5泥点'); @@ -953,30 +1009,92 @@ describe('ImageCanvasGenerationComposerView', () => { ); const optionPanel = screen.getByRole('menu', { name: '音效时长选项' }); const durationSlider = within(optionPanel).getByRole('slider', { - name: '音效时长', + name: '手动音效时长', }) as HTMLInputElement; - expect(durationSlider.min).toBe('2'); - expect(durationSlider.max).toBe('10'); - expect(durationSlider.step).toBe('1'); + expect(durationSlider.min).toBe('0.5'); + expect(durationSlider.max).toBe('30'); + expect(durationSlider.step).toBe('0.1'); expect(durationSlider.value).toBe('5'); expect( within(optionPanel).queryByRole('button', { name: '时长 2秒' }), ).toBeNull(); expect(within(optionPanel).queryByText(/BPM/u)).toBeNull(); + const automaticDuration = within(optionPanel).getByRole('checkbox', { + name: '自动时长', + }) as HTMLInputElement; + fireEvent.click(automaticDuration); + expect( + screen.getByRole('button', { name: '音效时长 自动时长' }), + ).toBeTruthy(); + expect(durationSlider.disabled).toBe(true); + fireEvent.click(automaticDuration); + expect(durationSlider.disabled).toBe(false); + expect(durationSlider.value).toBe('5'); fireEvent.change(durationSlider, { target: { value: '8' } }); expect(screen.getByLabelText('当前音效时长').textContent).toBe('8'); expect(screen.getByRole('button', { name: '音效时长 8秒' })).toBeTruthy(); + fireEvent.click(within(panel).getByRole('button', { name: 'Loop' })); fireEvent.submit(panel); expect(onSubmitImageGeneration).toHaveBeenCalledWith( expect.objectContaining({ mode: 'audio-sound-effect', soundDurationSeconds: 8, + soundLoop: true, }), ); }); + it('SFX 优化中保留撤销按钮并只锁当前 dialog 的全部控件', () => { + renderComposer( + { + id: 'dialog-sfx-locked', + mode: 'audio-sound-effect', + prompt: '金币落地声', + status: 'idle', + composerOpen: true, + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'manual', + soundDurationSeconds: 5, + soundLoop: false, + }, + { + soundEffectPromptAssist: createSoundEffectPromptAssistStub({ + status: 'optimizing', + operationId: 'operation-sfx-1', + temporaryPromptSnapshot: '金币落地声', + }), + }, + ); + + const panel = screen.getByRole('dialog', { name: '生成游戏音效' }); + expect(panel.getAttribute('aria-busy')).toBe('true'); + expect( + ( + within(panel).getByRole('textbox', { + name: 'prompt', + }) as HTMLTextAreaElement + ).readOnly, + ).toBe(true); + for (const name of [ + '展开', + '一键优化', + '撤销', + '音效时长 5秒', + 'Loop', + '生成游戏音效', + ]) { + expect( + (within(panel).getByRole('button', { name }) as HTMLButtonElement) + .disabled, + ).toBe(true); + } + expect(within(panel).getByText('优化中')).toBeTruthy(); + expect(within(panel).queryByText('Suno')).toBeNull(); + expect(within(panel).queryByRole('button', { name: 'AI 补全' })).toBeNull(); + }); + it('生成游戏背景音乐面板只展示 gpt_description_prompt 并固定隐藏 instrumental 开关', () => { renderComposer( { @@ -1397,13 +1515,14 @@ describe('ImageCanvasGenerationComposerView', () => { ).toBeNull(); }); - it('缺少稳定 ID 和 BGM controller 不影响 SFX 正常渲染及默认时长', () => { + it('BGM controller 不影响带稳定 ID 的 SFX 正常渲染及默认时长', () => { renderComposer({ + id: 'dialog-sfx', mode: 'audio-sound-effect', prompt: '', status: 'idle', composerOpen: true, - soundModel: 'audio1.0', + soundModel: 'eleven_text_to_sound_v2', }); const panel = screen.getByRole('dialog', { name: '生成游戏音效' }); @@ -1460,11 +1579,12 @@ describe('ImageCanvasGenerationComposerView', () => { operationId: 'operation-1', }); const soundEffectDialog: GenerateDialogState = { + id: 'dialog-sfx', mode: 'audio-sound-effect', prompt: '开门声', status: 'idle', composerOpen: true, - soundModel: 'audio1.0', + soundModel: 'eleven_text_to_sound_v2', soundDurationSeconds: 5, }; const view = renderComposer(soundEffectDialog, { diff --git a/src/components/image-editor/ImageCanvasGenerationComposerView.tsx b/src/components/image-editor/ImageCanvasGenerationComposerView.tsx index 2e900e3ac..9c085bb8b 100644 --- a/src/components/image-editor/ImageCanvasGenerationComposerView.tsx +++ b/src/components/image-editor/ImageCanvasGenerationComposerView.tsx @@ -21,6 +21,10 @@ import { useState, } from 'react'; +import { + SOUND_EFFECT_DURATION_MAX_SECONDS, + SOUND_EFFECT_DURATION_MIN_SECONDS, +} from '../../../packages/shared/src/contracts/editorAudio'; import { AutoGrowTextArea } from '../common/AutoGrowTextArea'; import { PlatformActionButton } from '../common/PlatformActionButton'; import { @@ -83,9 +87,23 @@ import { ImageCanvasPublicationMaterialsDemoPanelView } from './ImageCanvasPubli import { getPublicationMaterialsWorkflow } from './ImageCanvasPublicationMaterialsModel'; import { ImageCanvasQuickEditPanelView } from './ImageCanvasQuickEditPanelView'; import { ImageCanvasReferenceSlot } from './ImageCanvasReferenceSlot'; +import { ImageCanvasSoundEffectPresetMarquee } from './ImageCanvasSoundEffectPresetMarquee'; +import { + appendSoundEffectPromptPreset, + SOUND_EFFECT_PROMPT_PRESETS, + type SoundEffectPromptPreset, +} from './ImageCanvasSoundEffectPresetModel'; +import { + canGenerateSoundEffectFromPrompt, + canOptimizeSoundEffectPrompt, + countSoundEffectPromptCodePoints, + SOUND_EFFECT_PROMPT_MAX_CODE_POINTS, + toSoundEffectGenerationDialog, +} from './ImageCanvasSoundEffectPromptModel'; import { ImageCanvasSpecGenerationPanelView } from './ImageCanvasSpecGenerationPanelView'; import type { BackgroundMusicPromptAssistComposerController } from './useImageCanvasBackgroundMusicPromptAssist'; import { useImageCanvasFloatingOptionDismiss } from './useImageCanvasFloatingOptionDismiss'; +import type { SoundEffectPromptAssistComposerController } from './useImageCanvasSoundEffectPromptAssist'; type ImageCanvasGenerationComposerViewProps = { specToolWrapRef: RefObject; @@ -111,6 +129,7 @@ type ImageCanvasGenerationComposerViewProps = { hasPendingImageReferenceUploads?: boolean; generateDialog: GenerateDialogState | null; backgroundMusicPromptAssist?: BackgroundMusicPromptAssistComposerController; + soundEffectPromptAssist?: SoundEffectPromptAssistComposerController; updateCanvasGenerationDialogById: ( dialogId: string, updater: ( @@ -775,15 +794,10 @@ function normalizeSoundDuration(value: number | null | undefined) { if (typeof value !== 'number' || !Number.isFinite(value)) { return DEFAULT_SOUND_EFFECT_DURATION_SECONDS; } - return Math.min(10, Math.max(2, Math.round(value))); -} - -function resetFailedAudioDialogStatus(dialog: GenerateDialogState) { - return { - ...dialog, - status: dialog.status === 'failed' ? 'idle' : dialog.status, - errorMessage: dialog.status === 'failed' ? undefined : dialog.errorMessage, - }; + return Math.min( + SOUND_EFFECT_DURATION_MAX_SECONDS, + Math.max(SOUND_EFFECT_DURATION_MIN_SECONDS, value), + ); } const BACKGROUND_MUSIC_COMPOSER_LABEL = '生成游戏背景音乐'; @@ -796,22 +810,22 @@ const BACKGROUND_MUSIC_MODEL_LABEL = 'Suno'; function ImageCanvasAudioGenerationComposerView({ dialog, style, - setGenerateDialog, renderEditorPortal, buildPortalMenuStyle, promptAssist, + soundEffectPromptAssist, updateCanvasGenerationDialogById, onSubmit, }: { dialog: GenerateDialogState; style: CSSProperties; - setGenerateDialog: Dispatch>; renderEditorPortal: (node: ReactNode) => ReactNode; buildPortalMenuStyle: ( anchor: HTMLElement | null, placement: 'above' | 'below', ) => CSSProperties; promptAssist?: BackgroundMusicPromptAssistComposerController; + soundEffectPromptAssist?: SoundEffectPromptAssistComposerController; updateCanvasGenerationDialogById: ( dialogId: string, updater: ( @@ -1083,23 +1097,67 @@ function ImageCanvasAudioGenerationComposerView({ ); } - const soundModel = dialog.soundModel ?? DEFAULT_SOUND_EFFECT_MODEL; + const soundEffectDialog = toSoundEffectGenerationDialog(dialog); + const controller = soundEffectPromptAssist; + if (!soundEffectDialog || !controller) { + return null; + } + + const assistState = controller.getDialogState(soundEffectDialog.id); + const isAiProcessing = assistState.status === 'optimizing'; + const isSubmitting = assistState.status === 'submitting'; + const isLocked = isAiProcessing || isSubmitting || isGenerating; + const promptCodePointCount = countSoundEffectPromptCodePoints( + soundEffectDialog.prompt, + ); + const isPromptOverLimit = + promptCodePointCount > SOUND_EFFECT_PROMPT_MAX_CODE_POINTS; + const canOptimize = + !isLocked && canOptimizeSoundEffectPrompt(soundEffectDialog.prompt); + const canGenerate = + !isLocked && canGenerateSoundEffectFromPrompt(soundEffectDialog.prompt); + const isUndoVisible = + assistState.undoPromptSnapshot !== null || + assistState.temporaryPromptSnapshot !== null; + const canUndo = isUndoVisible && !isLocked; + const soundModel = DEFAULT_SOUND_EFFECT_MODEL; const currentSoundModel = EDITOR_SOUND_EFFECT_MODEL_OPTIONS.find( (option) => option.value === soundModel, ) ?? EDITOR_SOUND_EFFECT_MODEL_OPTIONS[0]; - const soundDuration = normalizeSoundDuration(dialog.soundDurationSeconds); - const soundOptionLabel = `${soundDuration}秒`; + const soundDuration = normalizeSoundDuration( + soundEffectDialog.soundDurationSeconds, + ); + const durationMode = + soundEffectDialog.soundDurationMode === 'auto' ? 'auto' : 'manual'; + const isAutomaticDuration = durationMode === 'auto'; + const soundLoop = soundEffectDialog.soundLoop === true; + const soundOptionLabel = isAutomaticDuration + ? '自动时长' + : `${soundDuration}秒`; const dialogLabel = '生成游戏音效'; const fixedModelLabel = currentSoundModel.label; const fixedModelAriaLabel = `音效模型 ${fixedModelLabel}`; const cost = calculateEditorSoundEffectPrice(soundModel); - const updateAudioDialog = (patch: Partial) => { - setGenerateDialog((currentDialog) => - currentDialog?.mode === dialog.mode + const updateAudioDialog = ( + patch: Partial< + Pick< + GenerateDialogState, + 'prompt' | 'soundDurationMode' | 'soundDurationSeconds' | 'soundLoop' + > + >, + ) => { + updateCanvasGenerationDialogById(soundEffectDialog.id, (currentDialog) => + currentDialog.mode === 'audio-sound-effect' ? { - ...resetFailedAudioDialogStatus(currentDialog), + ...currentDialog, + status: + currentDialog.status === 'failed' ? 'idle' : currentDialog.status, + errorMessage: + currentDialog.status === 'failed' + ? undefined + : currentDialog.errorMessage, ...patch, } : currentDialog, @@ -1118,23 +1176,53 @@ function ImageCanvasAudioGenerationComposerView({ style={style} role="dialog" aria-label={dialogLabel} + aria-busy={isLocked} onPointerDown={(event) => event.stopPropagation()} onSubmit={(event) => { event.preventDefault(); - if (!isGenerating) { - onSubmit(dialog); + if (canGenerate) { + onSubmit(soundEffectDialog); } }} > updateAudioDialog({ prompt: event.target.value })} /> - {dialog.status === 'failed' ? ( + { + const canonicalPrompt = controller.preparePreset( + soundEffectDialog.id, + ); + if (canonicalPrompt === null) { + return; + } + updateAudioDialog({ + prompt: appendSoundEffectPromptPreset(canonicalPrompt, preset), + }); + }} + headerTrailing={ + + {`${promptCodePointCount} / ${SOUND_EFFECT_PROMPT_MAX_CODE_POINTS}`} + + } + /> + {assistState.errorMessage ? ( - {dialog.errorMessage} + {assistState.errorMessage} ) : null} + {soundEffectDialog.status === 'failed' ? ( + + {soundEffectDialog.errorMessage} + + ) : null} +
+ { + void controller.optimizePrompt(soundEffectDialog.id); + }} + > + + {isAiProcessing ? '优化中' : '一键优化'} + + {isUndoVisible ? ( + controller.undoPrompt(soundEffectDialog.id)} + > + + 撤销 + + ) : null} +
} onClick={() => setIsSoundOptionsOpen((isOpen) => !isOpen)} > @@ -1174,15 +1314,31 @@ function ImageCanvasAudioGenerationComposerView({ 时长 +
updateSoundDuration(Number(event.target.value)) } @@ -1195,6 +1351,15 @@ function ImageCanvasAudioGenerationComposerView({ ) : null}
+ updateAudioDialog({ soundLoop: !soundLoop })} + > + Loop +
{isGenerating ? ( @@ -1261,6 +1426,7 @@ export function ImageCanvasGenerationComposerView({ isPickingUiDesignSpecFromCanvas, generateDialog, backgroundMusicPromptAssist, + soundEffectPromptAssist, updateCanvasGenerationDialogById, generationComposerStyle, iconComposerStyle, @@ -1478,7 +1644,7 @@ export function ImageCanvasGenerationComposerView({ dialog={generateDialog} style={generationComposerStyle} promptAssist={backgroundMusicPromptAssist} - setGenerateDialog={setGenerateDialog} + soundEffectPromptAssist={soundEffectPromptAssist} renderEditorPortal={renderEditorPortal} buildPortalMenuStyle={buildPortalMenuStyle} updateCanvasGenerationDialogById={updateCanvasGenerationDialogById} diff --git a/src/components/image-editor/ImageCanvasGenerationDialogModel.test.ts b/src/components/image-editor/ImageCanvasGenerationDialogModel.test.ts index 65ef52064..070b0ad95 100644 --- a/src/components/image-editor/ImageCanvasGenerationDialogModel.test.ts +++ b/src/components/image-editor/ImageCanvasGenerationDialogModel.test.ts @@ -253,8 +253,10 @@ describe('ImageCanvasGenerationDialogModel', () => { prompt: '', status: 'idle', composerOpen: true, - soundModel: 'audio1.0', + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'manual', soundDurationSeconds: 5, + soundLoop: false, placeholder: { x: 270, y: 300, @@ -313,8 +315,10 @@ describe('ImageCanvasGenerationDialogModel', () => { mode: 'audio-sound-effect', sourceLayerId: 'layer-sfx', prompt: '金币收集音', - soundModel: 'audio1.0', + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'manual', soundDurationSeconds: 7, + soundLoop: false, composerOpen: true, placeholder: { x: 472, diff --git a/src/components/image-editor/ImageCanvasGenerationDialogModel.ts b/src/components/image-editor/ImageCanvasGenerationDialogModel.ts index 2759f28f4..0fbc876b6 100644 --- a/src/components/image-editor/ImageCanvasGenerationDialogModel.ts +++ b/src/components/image-editor/ImageCanvasGenerationDialogModel.ts @@ -1,3 +1,7 @@ +import { + SOUND_EFFECT_DURATION_MAX_SECONDS, + SOUND_EFFECT_DURATION_MIN_SECONDS, +} from '../../../packages/shared/src/contracts/editorAudio'; import { formatImageSizeValue } from './ImageCanvasEditorModel'; import type { CanvasGenerationDialogState, @@ -400,7 +404,9 @@ export function createSoundEffectGenerationDialogDraft({ status: 'idle', composerOpen: true, soundModel: DEFAULT_SOUND_EFFECT_MODEL, + soundDurationMode: 'manual', soundDurationSeconds: DEFAULT_SOUND_EFFECT_DURATION_SECONDS, + soundLoop: false, placeholder: { x: worldCenter.x - AUDIO_FRAME_DISPLAY_SIZE.width / 2, y: worldCenter.y - AUDIO_FRAME_DISPLAY_SIZE.height / 2, @@ -521,13 +527,10 @@ function resolveAudioRedrawMode(sourceLayer: CanvasLayer) { } function resolveAudioRedrawSoundModel( - sourceLayer: CanvasLayer, + _sourceLayer: CanvasLayer, ): NonNullable { - const fieldValue = findGenerationInputFieldValue(sourceLayer, [ - 'model', - ])?.trim(); - // 中文注释:编辑器音效改造入口当前只暴露 Vidu audio1.0,历史快照里的其他模型统一回落到默认模型。 - return fieldValue === DEFAULT_SOUND_EFFECT_MODEL ? 'audio1.0' : 'audio1.0'; + // 历史 Vidu 素材只提供用户 Prompt 与可兼容的手动时长;重绘始终打开 SFX V2。 + return DEFAULT_SOUND_EFFECT_MODEL; } function resolveAudioRedrawSoundDuration(sourceLayer: CanvasLayer) { @@ -535,12 +538,15 @@ function resolveAudioRedrawSoundDuration(sourceLayer: CanvasLayer) { '时长', 'duration', ]); - const matchedValue = fieldValue?.match(/\d+/u)?.[0]; - const duration = matchedValue ? Number.parseInt(matchedValue, 10) : null; + const matchedValue = fieldValue?.match(/\d+(?:\.\d+)?/u)?.[0]; + const duration = matchedValue ? Number.parseFloat(matchedValue) : null; if (!duration || !Number.isFinite(duration)) { return DEFAULT_SOUND_EFFECT_DURATION_SECONDS; } - return Math.min(10, Math.max(2, duration)); + return Math.min( + SOUND_EFFECT_DURATION_MAX_SECONDS, + Math.max(SOUND_EFFECT_DURATION_MIN_SECONDS, duration), + ); } export function createAudioRedrawGenerationDialogDraft( @@ -576,7 +582,9 @@ export function createAudioRedrawGenerationDialogDraft( return { ...baseDraft, soundModel: resolveAudioRedrawSoundModel(sourceLayer), + soundDurationMode: 'manual', soundDurationSeconds: resolveAudioRedrawSoundDuration(sourceLayer), + soundLoop: false, }; } @@ -1033,8 +1041,11 @@ export function createSameSourceGenerationDialogDraft({ ...draft, prompt, soundModel: sourceDialog?.soundModel ?? draft.soundModel, + soundDurationMode: + sourceDialog?.soundDurationMode ?? draft.soundDurationMode, soundDurationSeconds: sourceDialog?.soundDurationSeconds ?? draft.soundDurationSeconds, + soundLoop: sourceDialog?.soundLoop ?? draft.soundLoop, makeInstrumental: sourceDialog?.makeInstrumental ?? draft.makeInstrumental, }; diff --git a/src/components/image-editor/ImageCanvasGenerationModel.ts b/src/components/image-editor/ImageCanvasGenerationModel.ts index e25047f94..e6814225b 100644 --- a/src/components/image-editor/ImageCanvasGenerationModel.ts +++ b/src/components/image-editor/ImageCanvasGenerationModel.ts @@ -1,3 +1,4 @@ +import { EDITOR_SOUND_EFFECT_MODEL } from '../../../packages/shared/src/contracts/editorAudio'; import { ApiClientError } from '../../services/apiClient'; import type { EditorGenerationPricingConfig } from '../../services/image-editor/editorProjectClient'; import type { @@ -316,7 +317,7 @@ export const VIDEO_MODEL_VEO_3_1_FAST = 'veo3.1-fast'; export const DEFAULT_VIDEO_MODEL = VIDEO_MODEL_SEEDANCE_2_FAST; export const CHARACTER_ANIMATION_MODEL = VIDEO_MODEL_SEEDANCE_2_FAST; export const SOUND_EFFECT_MODEL_VIDU = 'audio1.0'; -export const DEFAULT_SOUND_EFFECT_MODEL = SOUND_EFFECT_MODEL_VIDU; +export const DEFAULT_SOUND_EFFECT_MODEL = EDITOR_SOUND_EFFECT_MODEL; export const DEFAULT_SOUND_EFFECT_DURATION_SECONDS = 5; export const SOUND_EFFECT_DURATION_OPTIONS = Array.from( { length: 9 }, @@ -354,6 +355,7 @@ export const EDITOR_VIDEO_MODEL_MUD_POINT_CONFIG = { [VIDEO_MODEL_VEO_3_1_FAST]: EDITOR_VIDEO_RESOLUTION_RATE_CONFIG, } as const; export const EDITOR_SOUND_EFFECT_MODEL_MUD_POINT_CONFIG = { + [EDITOR_SOUND_EFFECT_MODEL]: 5, [SOUND_EFFECT_MODEL_VIDU]: 5, } as const; export const EDITOR_BACKGROUND_MUSIC_MODEL_MUD_POINT_CONFIG = { @@ -396,6 +398,11 @@ export const EDITOR_MODEL_MUD_POINT_CONFIG = { unit: 'perGeneration', price: EDITOR_SOUND_EFFECT_MODEL_MUD_POINT_CONFIG[SOUND_EFFECT_MODEL_VIDU], }, + [EDITOR_SOUND_EFFECT_MODEL]: { + unit: 'perGeneration', + price: + EDITOR_SOUND_EFFECT_MODEL_MUD_POINT_CONFIG[EDITOR_SOUND_EFFECT_MODEL], + }, [BACKGROUND_MUSIC_MODEL_SUNO]: { unit: 'perGeneration', price: @@ -408,7 +415,7 @@ let runtimeEditorGenerationPricingConfig: EditorGenerationPricingConfig = { models: cloneModelPricing(EDITOR_MODEL_MUD_POINT_CONFIG), }; export const EDITOR_SOUND_EFFECT_MODEL_OPTIONS = [ - { label: 'Vidu', value: SOUND_EFFECT_MODEL_VIDU }, + { label: 'ElevenLabs', value: EDITOR_SOUND_EFFECT_MODEL }, ] as const; export const SEEDANCE_VIDEO_REFERENCE_LIMITS = { image: 9, @@ -958,7 +965,9 @@ export function calculateEditorSoundEffectPrice( model: string | null | undefined, ) { const normalizedModel = (model ?? DEFAULT_SOUND_EFFECT_MODEL).trim(); - return readModelFlatPrice(normalizedModel, DEFAULT_SOUND_EFFECT_MODEL); + // T4 只切前端固定模型展示;新模型的动态定价键由 T5 同批接入。中间态继续以 + // 历史 Vidu 的 5 泥点前端兜底展示,服务端价格始终是正式真相。 + return readModelFlatPrice(normalizedModel, SOUND_EFFECT_MODEL_VIDU); } export function calculateEditorBackgroundMusicPrice( diff --git a/src/components/image-editor/ImageCanvasSoundEffectPresetMarquee.test.tsx b/src/components/image-editor/ImageCanvasSoundEffectPresetMarquee.test.tsx new file mode 100644 index 000000000..772937ae8 --- /dev/null +++ b/src/components/image-editor/ImageCanvasSoundEffectPresetMarquee.test.tsx @@ -0,0 +1,76 @@ +/* @vitest-environment jsdom */ + +import { fireEvent, render, screen } from '@testing-library/react'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { ImageCanvasSoundEffectPresetMarquee } from './ImageCanvasSoundEffectPresetMarquee'; +import { SOUND_EFFECT_PROMPT_PRESETS } from './ImageCanvasSoundEffectPresetModel'; + +describe('ImageCanvasSoundEffectPresetMarquee', () => { + beforeEach(() => { + vi.stubGlobal( + 'requestAnimationFrame', + vi.fn(() => 1), + ); + vi.stubGlobal('cancelAnimationFrame', vi.fn()); + }); + + it('renders one accessible control for each of the 40 + 12 frozen presets', () => { + const onSelectPreset = vi.fn(); + render( + 0 / 2048} + />, + ); + + expect(screen.getByText('0 / 2048')).toBeTruthy(); + fireEvent.click(screen.getByRole('button', { name: '展开' })); + for (const preset of SOUND_EFFECT_PROMPT_PRESETS) { + const button = screen.getByRole('button', { name: preset.label }); + expect(button.className).toContain( + `image-canvas-editor__sound-effect-preset--${preset.group}`, + ); + } + expect(SOUND_EFFECT_PROMPT_PRESETS).toHaveLength(52); + + fireEvent.click( + screen.getByRole('button', { + name: SOUND_EFFECT_PROMPT_PRESETS[0].label, + }), + ); + expect(onSelectPreset).toHaveBeenCalledWith(SOUND_EFFECT_PROMPT_PRESETS[0]); + }); + + it('keeps expansion and preset controls truly disabled while locked', () => { + const view = render( + , + ); + fireEvent.click(screen.getByRole('button', { name: '展开' })); + view.rerender( + , + ); + + expect( + (screen.getByRole('button', { name: '收起' }) as HTMLButtonElement) + .disabled, + ).toBe(true); + expect( + ( + screen.getByRole('button', { + name: SOUND_EFFECT_PROMPT_PRESETS[0].label, + }) as HTMLButtonElement + ).disabled, + ).toBe(true); + }); +}); diff --git a/src/components/image-editor/ImageCanvasSoundEffectPresetMarquee.tsx b/src/components/image-editor/ImageCanvasSoundEffectPresetMarquee.tsx new file mode 100644 index 000000000..cf116ecfe --- /dev/null +++ b/src/components/image-editor/ImageCanvasSoundEffectPresetMarquee.tsx @@ -0,0 +1,28 @@ +import type { ReactNode } from 'react'; + +import { ImageCanvasAudioPresetMarquee } from './ImageCanvasAudioPresetMarquee'; +import type { SoundEffectPromptPreset } from './ImageCanvasSoundEffectPresetModel'; + +export function ImageCanvasSoundEffectPresetMarquee({ + presets, + isLocked, + onSelectPreset, + headerTrailing, +}: { + presets: readonly SoundEffectPromptPreset[]; + isLocked: boolean; + onSelectPreset: (preset: SoundEffectPromptPreset) => void; + headerTrailing?: ReactNode; +}) { + return ( + + `image-canvas-editor__background-music-preset image-canvas-editor__sound-effect-preset--${preset.group}` + } + /> + ); +} diff --git a/src/components/image-editor/ImageCanvasSoundEffectPromptModel.test.ts b/src/components/image-editor/ImageCanvasSoundEffectPromptModel.test.ts index 3fa25dff6..bf8c399c2 100644 --- a/src/components/image-editor/ImageCanvasSoundEffectPromptModel.test.ts +++ b/src/components/image-editor/ImageCanvasSoundEffectPromptModel.test.ts @@ -4,6 +4,7 @@ import soundEffectPromptCanonicalizationCases from '../../../packages/shared/tes import { canonicalizeSoundEffectPrompt, countSoundEffectPromptCodePoints, + createSoundEffectPromptStateModel, SOUND_EFFECT_PROMPT_MAX_CODE_POINTS, validateSoundEffectPrompt, } from './ImageCanvasSoundEffectPromptModel'; @@ -67,4 +68,94 @@ describe('ImageCanvasSoundEffectPromptModel', () => { }); } }); + + it('creates one dialog-scoped optimization operation and swaps one undo snapshot', () => { + const model = createSoundEffectPromptStateModel(); + const started = model.beginOperation({ + dialogId: 'dialog-a', + status: 'optimizing', + prompt: '\u0085金币落地声\u3000', + }); + expect(started.started).toBe(true); + expect(started.canonicalPrompt).toBe('金币落地声'); + expect(started.state).toMatchObject({ + status: 'optimizing', + undoPromptSnapshot: null, + temporaryPromptSnapshot: '金币落地声', + }); + expect( + model.beginOperation({ + dialogId: 'dialog-a', + status: 'optimizing', + prompt: '另一个输入', + }).reason, + ).toBe('duplicate'); + + const resolved = model.resolveAiOperation( + started.operation!, + '清脆明亮的金币落地声', + ); + expect(resolved).toMatchObject({ + applied: true, + prompt: '清脆明亮的金币落地声', + state: { + status: 'idle', + undoPromptSnapshot: '金币落地声', + temporaryPromptSnapshot: null, + }, + }); + expect( + model.swapUndoSnapshot('dialog-a', '清脆明亮的金币落地声'), + ).toMatchObject({ + applied: true, + prompt: '金币落地声', + state: { undoPromptSnapshot: '清脆明亮的金币落地声' }, + }); + }); + + it('clears the replaced snapshot when a later optimization fails', () => { + const model = createSoundEffectPromptStateModel(); + const first = model.beginOperation({ + dialogId: 'dialog-a', + status: 'optimizing', + prompt: '原始描述', + }); + model.resolveAiOperation(first.operation!, '第一次优化'); + const second = model.beginOperation({ + dialogId: 'dialog-a', + status: 'optimizing', + prompt: '手动修改后的描述', + }); + expect(second.state.undoPromptSnapshot).toBeNull(); + expect(model.rejectOperation(second.operation!)).toMatchObject({ + applied: true, + prompt: '手动修改后的描述', + state: { + status: 'idle', + undoPromptSnapshot: null, + temporaryPromptSnapshot: null, + }, + }); + }); + + it('keeps submission and preset state isolated by dialog ID', () => { + const model = createSoundEffectPromptStateModel(); + const submission = model.beginOperation({ + dialogId: 'dialog-a', + status: 'submitting', + prompt: ' 提交描述 ', + }); + expect(submission.started).toBe(true); + expect(model.preparePreset('dialog-a', '不应写入').applied).toBe(false); + expect(model.preparePreset('dialog-b', ' 另一面板 ').prompt).toBe( + '另一面板', + ); + expect( + model.completeSubmittingOperation(submission.operation!), + ).toMatchObject({ + applied: true, + prompt: '提交描述', + state: { status: 'idle' }, + }); + }); }); diff --git a/src/components/image-editor/ImageCanvasSoundEffectPromptModel.ts b/src/components/image-editor/ImageCanvasSoundEffectPromptModel.ts index 3fb159dc2..2b9503643 100644 --- a/src/components/image-editor/ImageCanvasSoundEffectPromptModel.ts +++ b/src/components/image-editor/ImageCanvasSoundEffectPromptModel.ts @@ -1,4 +1,10 @@ +import type { + CanvasGenerationDialogState, + GenerateDialogState, +} from './ImageCanvasEditorTypes'; + export const SOUND_EFFECT_PROMPT_MAX_CODE_POINTS = 2048; +const SOUND_EFFECT_PROMPT_OPERATION_ID_PREFIX = 'sound-effect-prompt-operation'; const LEADING_UNICODE_WHITE_SPACE = /^\p{White_Space}+/u; const TRAILING_UNICODE_WHITE_SPACE = /\p{White_Space}+$/u; @@ -16,6 +22,40 @@ export type SoundEffectPromptValidationResult = reason: 'empty' | 'too-long'; }; +export type SoundEffectPromptOperationStatus = 'optimizing' | 'submitting'; + +export type SoundEffectPromptDialogState = { + dialogId: string; + status: 'idle' | SoundEffectPromptOperationStatus; + operationId: string | null; + undoPromptSnapshot: string | null; + temporaryPromptSnapshot: string | null; +}; + +export type SoundEffectPromptOperation = { + dialogId: string; + operationId: string; + status: SoundEffectPromptOperationStatus; + canonicalPrompt: string; +}; + +type InternalSoundEffectPromptDialogState = SoundEffectPromptDialogState & { + activeCanonicalPrompt: string | null; +}; + +export type SoundEffectGenerationDialogState = CanvasGenerationDialogState & { + mode: 'audio-sound-effect'; +}; + +export function toSoundEffectGenerationDialog( + dialog: GenerateDialogState | null, +): SoundEffectGenerationDialogState | null { + if (!dialog || dialog.mode !== 'audio-sound-effect' || !dialog.id) { + return null; + } + return { ...dialog, id: dialog.id, mode: 'audio-sound-effect' }; +} + /** * 只删除首尾 Unicode `White_Space`。内部空白、组合字符、ZWJ、U+200B、U+FEFF * 及其它 code point 均原样保留;不能替换为语义不同的原生 `trim()`。 @@ -53,3 +93,297 @@ export function validateSoundEffectPrompt( } return { ok: true, prompt: canonicalPrompt, charCount }; } + +export function canOptimizeSoundEffectPrompt(prompt: string) { + return validateSoundEffectPrompt(prompt).ok; +} + +export function canGenerateSoundEffectFromPrompt(prompt: string) { + return validateSoundEffectPrompt(prompt).ok; +} + +function createIdleSoundEffectPromptDialogState( + dialogId: string, +): InternalSoundEffectPromptDialogState { + return { + dialogId, + status: 'idle', + operationId: null, + undoPromptSnapshot: null, + temporaryPromptSnapshot: null, + activeCanonicalPrompt: null, + }; +} + +function toPublicSoundEffectPromptDialogState( + state: InternalSoundEffectPromptDialogState, +): SoundEffectPromptDialogState { + return { + dialogId: state.dialogId, + status: state.status, + operationId: state.operationId, + undoPromptSnapshot: state.undoPromptSnapshot, + temporaryPromptSnapshot: state.temporaryPromptSnapshot, + }; +} + +/** + * SFX Prompt 的纯状态机。operation 在调用方第一次 await 前同步取得;模型只保存 + * dialog 级临时状态,不把快照、错误或预设 UI 状态写入画布 layout。 + */ +export function createSoundEffectPromptStateModel() { + const dialogStates = new Map(); + let operationSequence = 0; + + const readInternalState = (dialogId: string) => + dialogStates.get(dialogId) ?? + createIdleSoundEffectPromptDialogState(dialogId); + + const saveState = (state: InternalSoundEffectPromptDialogState) => { + dialogStates.set(state.dialogId, state); + return toPublicSoundEffectPromptDialogState(state); + }; + + const getDialogState = (dialogId: string) => + toPublicSoundEffectPromptDialogState(readInternalState(dialogId)); + + const beginOperation = ({ + dialogId, + status, + prompt, + }: { + dialogId: string; + status: SoundEffectPromptOperationStatus; + prompt: string; + }) => { + const currentState = readInternalState(dialogId); + const validation = validateSoundEffectPrompt(prompt); + const canonicalPrompt = validation.prompt; + + if (currentState.status === status && currentState.operationId !== null) { + return { + started: false as const, + reason: 'duplicate' as const, + canonicalPrompt: currentState.activeCanonicalPrompt ?? canonicalPrompt, + operation: null, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + if (currentState.status !== 'idle') { + return { + started: false as const, + reason: 'locked' as const, + canonicalPrompt: currentState.activeCanonicalPrompt ?? canonicalPrompt, + operation: null, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + if (!validation.ok) { + return { + started: false as const, + reason: 'ineligible' as const, + canonicalPrompt, + operation: null, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + + operationSequence += 1; + const operation: SoundEffectPromptOperation = { + dialogId, + operationId: `${SOUND_EFFECT_PROMPT_OPERATION_ID_PREFIX}-${operationSequence}`, + status, + canonicalPrompt, + }; + const isAiOperation = status === 'optimizing'; + const nextState: InternalSoundEffectPromptDialogState = { + ...currentState, + status, + operationId: operation.operationId, + undoPromptSnapshot: isAiOperation + ? null + : currentState.undoPromptSnapshot, + temporaryPromptSnapshot: isAiOperation ? canonicalPrompt : null, + activeCanonicalPrompt: canonicalPrompt, + }; + return { + started: true as const, + reason: 'started' as const, + canonicalPrompt, + operation, + state: saveState(nextState), + }; + }; + + const isCurrentOperation = ( + operation: SoundEffectPromptOperation, + state: InternalSoundEffectPromptDialogState, + ) => + state.dialogId === operation.dialogId && + state.operationId === operation.operationId && + state.status === operation.status; + + const resolveAiOperation = ( + operation: SoundEffectPromptOperation, + prompt: string, + ) => { + const currentState = readInternalState(operation.dialogId); + if ( + operation.status !== 'optimizing' || + !isCurrentOperation(operation, currentState) + ) { + return { + applied: false, + prompt: null, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + const validation = validateSoundEffectPrompt(prompt); + if (!validation.ok) { + const failedState: InternalSoundEffectPromptDialogState = { + ...currentState, + status: 'idle', + operationId: null, + undoPromptSnapshot: null, + temporaryPromptSnapshot: null, + activeCanonicalPrompt: null, + }; + return { + applied: false, + prompt: null, + state: saveState(failedState), + }; + } + const completedState: InternalSoundEffectPromptDialogState = { + ...currentState, + status: 'idle', + operationId: null, + undoPromptSnapshot: currentState.temporaryPromptSnapshot, + temporaryPromptSnapshot: null, + activeCanonicalPrompt: null, + }; + return { + applied: true, + prompt: validation.prompt, + state: saveState(completedState), + }; + }; + + const rejectOperation = (operation: SoundEffectPromptOperation) => { + const currentState = readInternalState(operation.dialogId); + if (!isCurrentOperation(operation, currentState)) { + return { + applied: false, + prompt: null, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + const isAiOperation = operation.status === 'optimizing'; + const failedState: InternalSoundEffectPromptDialogState = { + ...currentState, + status: 'idle', + operationId: null, + undoPromptSnapshot: isAiOperation + ? null + : currentState.undoPromptSnapshot, + temporaryPromptSnapshot: null, + activeCanonicalPrompt: null, + }; + return { + applied: true, + prompt: currentState.activeCanonicalPrompt, + state: saveState(failedState), + }; + }; + + const completeSubmittingOperation = ( + operation: SoundEffectPromptOperation, + ) => { + const currentState = readInternalState(operation.dialogId); + if ( + operation.status !== 'submitting' || + !isCurrentOperation(operation, currentState) + ) { + return { + applied: false, + prompt: null, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + const completedState: InternalSoundEffectPromptDialogState = { + ...currentState, + status: 'idle', + operationId: null, + temporaryPromptSnapshot: null, + activeCanonicalPrompt: null, + }; + return { + applied: true, + prompt: currentState.activeCanonicalPrompt ?? operation.canonicalPrompt, + state: saveState(completedState), + }; + }; + + const preparePreset = (dialogId: string, prompt: string) => { + const currentState = readInternalState(dialogId); + const canonicalPrompt = canonicalizeSoundEffectPrompt(prompt); + if (currentState.status !== 'idle') { + return { + applied: false, + prompt: currentState.activeCanonicalPrompt ?? canonicalPrompt, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + const nextState: InternalSoundEffectPromptDialogState = { + ...currentState, + undoPromptSnapshot: null, + temporaryPromptSnapshot: null, + activeCanonicalPrompt: null, + }; + return { + applied: true, + prompt: canonicalPrompt, + state: saveState(nextState), + }; + }; + + const swapUndoSnapshot = (dialogId: string, prompt: string) => { + const currentState = readInternalState(dialogId); + if ( + currentState.status !== 'idle' || + currentState.undoPromptSnapshot === null + ) { + return { + applied: false, + prompt, + state: toPublicSoundEffectPromptDialogState(currentState), + }; + } + const canonicalPrompt = canonicalizeSoundEffectPrompt(prompt); + const nextPrompt = currentState.undoPromptSnapshot; + const nextState: InternalSoundEffectPromptDialogState = { + ...currentState, + undoPromptSnapshot: canonicalPrompt, + }; + return { + applied: true, + prompt: nextPrompt, + state: saveState(nextState), + }; + }; + + const closeDialog = (dialogId: string) => dialogStates.delete(dialogId); + const reset = () => dialogStates.clear(); + + return { + getDialogState, + beginOperation, + resolveAiOperation, + rejectOperation, + completeSubmittingOperation, + preparePreset, + swapUndoSnapshot, + closeDialog, + reset, + }; +} diff --git a/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx b/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx index 4a7ae1980..e6daf7b52 100644 --- a/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx +++ b/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx @@ -31,6 +31,7 @@ import { resolveEditorGenerationMediaReference, useImageCanvasGenerationSubmissionWorkflow, } from './useImageCanvasGenerationSubmissionWorkflow'; +import { useImageCanvasSoundEffectPromptAssist } from './useImageCanvasSoundEffectPromptAssist'; const resolveEditorImageReferenceDataUrlMock = vi.hoisted(() => vi.fn()); const resolveEditorImageReferenceDataUrlForGenerationMock = vi.hoisted(() => @@ -266,15 +267,23 @@ function SubmissionWorkflowHarness({ ); const layerCounterRef = useRef(0); const dialogs = useCanvasGenerationDialogs(); - const backgroundMusicPromptAssist = - useImageCanvasBackgroundMusicPromptAssist({ + const backgroundMusicPromptAssist = useImageCanvasBackgroundMusicPromptAssist( + { canvasGenerationDialogs: dialogs.canvasGenerationDialogs, getCanvasGenerationDialogById: dialogs.getCanvasGenerationDialogById, updateCanvasGenerationDialogById: dialogs.updateCanvasGenerationDialogById, currentUserId, projectId, - }); + }, + ); + const soundEffectPromptAssist = useImageCanvasSoundEffectPromptAssist({ + canvasGenerationDialogs: dialogs.canvasGenerationDialogs, + getCanvasGenerationDialogById: dialogs.getCanvasGenerationDialogById, + updateCanvasGenerationDialogById: dialogs.updateCanvasGenerationDialogById, + currentUserId, + projectId, + }); const sourceLayer = layers[0] ?? null; const quickEditSourceLayer = quickEditPanel @@ -313,6 +322,7 @@ function SubmissionWorkflowHarness({ getCanvasGenerationDialogById: dialogs.getCanvasGenerationDialogById, activeCanvasGenerationDialogId: activeCanvasDialog?.id, backgroundMusicPromptAssist, + soundEffectPromptAssist, getGeneratingDialogPlaceholder: dialogs.getGeneratingDialogPlaceholder, appendCanvasLayersWithResources: (nextLayers) => { onAppendCanvasLayers?.(nextLayers); @@ -383,6 +393,11 @@ function SubmissionWorkflowHarness({ ? backgroundMusicPromptAssist.getDialogState(trackedDialogId).status : '-'} + + {trackedDialogId + ? soundEffectPromptAssist.getDialogState(trackedDialogId).status + : '-'} + {quickEditPanel ? `${quickEditPanel.sourceLayerId}:${quickEditPanel.status}:${quickEditPanel.prompt || '-'}:${quickEditPanel.errorMessage ?? '-'}` @@ -1980,6 +1995,67 @@ describe('useImageCanvasGenerationSubmissionWorkflow', () => { ); }); + it('claims and canonicalizes one SFX submission before the first await', async () => { + const generation = + createDeferred>(); + generateEditorSoundEffectMock.mockReturnValueOnce(generation.promise); + render( + , + ); + + fireEvent.click(screen.getByRole('button', { name: '设置初始对话' })); + fireEvent.click(screen.getByRole('button', { name: '提交当前生成' })); + fireEvent.click(screen.getByRole('button', { name: '提交当前生成' })); + + expect(generateEditorSoundEffectMock).toHaveBeenCalledTimes(1); + expect(screen.getByTestId('tracked-dialog').textContent).toBe( + 'audio-sound-effect:循环环境声:idle:open:-', + ); + expect(screen.getByTestId('sound-effect-assist').textContent).toBe( + 'submitting', + ); + + await act(async () => { + generation.resolve( + createAudioGenerated({ + prompt: '循环环境声', + audioKind: 'sound-effect', + }), + ); + await generation.promise; + }); + await waitFor(() => { + expect(screen.getByTestId('sound-effect-assist').textContent).toBe( + 'idle', + ); + expect(screen.getByTestId('layers').textContent).toContain( + 'sound-effect', + ); + }); + }); + it('claims and canonicalizes one background music submission before the first await', async () => { const representativeComplexPromptCase = backgroundMusicPromptCanonicalizationCases.find( diff --git a/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts b/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts index c74e99063..cae713ff0 100644 --- a/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts +++ b/src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts @@ -91,6 +91,7 @@ import type { import { resolveUiAssetExtractionGenerationPlan } from './ImageCanvasUiAssetExtractionModel'; import { renderUiDesignAssetExtractionMarkedImage } from './ImageCanvasUiAssetExtractionRasterModel'; import type { BackgroundMusicPromptAssistController } from './useImageCanvasBackgroundMusicPromptAssist'; +import type { SoundEffectPromptAssistController } from './useImageCanvasSoundEffectPromptAssist'; type CanvasSize = { width: number; height: number }; @@ -330,6 +331,7 @@ type GenerationSubmissionWorkflowOptions = { ) => CanvasGenerationDialogState | undefined; activeCanvasGenerationDialogId?: string | null; backgroundMusicPromptAssist: BackgroundMusicPromptAssistController; + soundEffectPromptAssist: SoundEffectPromptAssistController; getGeneratingDialogPlaceholder: ( dialog: GenerateDialogState, ) => GenerateDialogState['placeholder']; @@ -642,6 +644,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({ getCanvasGenerationDialogById, activeCanvasGenerationDialogId, backgroundMusicPromptAssist, + soundEffectPromptAssist, getGeneratingDialogPlaceholder, appendCanvasLayersWithResources, captureCanvasHistory, @@ -704,6 +707,26 @@ export function useImageCanvasGenerationSubmissionWorkflow({ [getCanvasGenerationDialogById], ); + const isSoundEffectSubmissionUiTargetCurrent = useCallback( + (scope: { + currentUserId: string | null; + projectId: string | null; + dialogId: string; + scopeVersion: number; + }) => { + const currentScope = currentBackgroundMusicSubmissionScopeRef.current; + const currentDialog = getCanvasGenerationDialogById(scope.dialogId); + return ( + currentScope.mounted && + currentScope.version === scope.scopeVersion && + Object.is(scope.currentUserId, currentScope.currentUserId) && + Object.is(scope.projectId, currentScope.projectId) && + currentDialog?.mode === 'audio-sound-effect' + ); + }, + [getCanvasGenerationDialogById], + ); + const isBackgroundMusicSubmissionAccountCurrent = useCallback( (scope: { currentUserId: string | null }) => { const currentScope = currentBackgroundMusicSubmissionScopeRef.current; @@ -1862,15 +1885,45 @@ export function useImageCanvasGenerationSubmissionWorkflow({ const canvasDialog = isCanvasGenerationDialog(dialog) ? dialog : null; const backgroundMusicDialog = canvasDialog?.mode === 'audio-background-music' ? canvasDialog : null; + const soundEffectDialog = + canvasDialog?.mode === 'audio-sound-effect' ? canvasDialog : null; if (dialog.mode === 'audio-background-music' && !backgroundMusicDialog) { return; } + if (dialog.mode === 'audio-sound-effect' && !soundEffectDialog) { + return; + } const backgroundMusicClaim = backgroundMusicDialog ? backgroundMusicPromptAssist.beginSubmission(backgroundMusicDialog.id) : null; if (backgroundMusicDialog && !backgroundMusicClaim) { return; } + const soundEffectClaim = soundEffectDialog + ? soundEffectPromptAssist.beginSubmission(soundEffectDialog.id) + : null; + if (soundEffectDialog && !soundEffectClaim) { + const validation = validateSoundEffectPrompt(soundEffectDialog.prompt); + if (!validation.ok) { + updateCanvasGenerationDialogById( + soundEffectDialog.id, + (currentDialog) => + currentDialog.mode === 'audio-sound-effect' + ? { + ...currentDialog, + prompt: validation.prompt, + status: 'failed', + composerOpen: true, + errorMessage: + validation.reason === 'empty' + ? '音效描述不能为空' + : '音效描述不能超过 2048 个字符', + } + : currentDialog, + ); + } + return; + } const backgroundMusicSubmission = backgroundMusicDialog && backgroundMusicClaim ? { @@ -1883,23 +1936,34 @@ export function useImageCanvasGenerationSubmissionWorkflow({ currentBackgroundMusicSubmissionScopeRef.current.version, } : null; - const soundEffectPrompt = - dialog.mode === 'audio-sound-effect' - ? validateSoundEffectPrompt(dialog.prompt) + const soundEffectSubmission = + soundEffectDialog && soundEffectClaim + ? { + currentUserId: currentUserId ?? null, + projectId: projectId ?? null, + dialogId: soundEffectDialog.id, + ...soundEffectClaim, + scopeVersion: + currentBackgroundMusicSubmissionScopeRef.current.version, + } : null; const normalizedPrompt = backgroundMusicSubmission?.prompt ?? - soundEffectPrompt?.prompt ?? + soundEffectSubmission?.prompt ?? (dialog.prompt.trim() || (dialog.mode === 'edit' ? '修改当前图片' : 'AI 生成图片')); - if (!backgroundMusicSubmission && canvasDialog) { + if ( + !backgroundMusicSubmission && + !soundEffectSubmission && + canvasDialog + ) { updateCanvasGenerationDialogById(canvasDialog.id, (currentDialog) => ({ ...currentDialog, prompt: normalizedPrompt, status: 'generating', composerOpen: false, })); - } else if (!backgroundMusicSubmission) { + } else if (!backgroundMusicSubmission && !soundEffectSubmission) { setGenerateDialog({ ...dialog, prompt: normalizedPrompt, @@ -1911,9 +1975,21 @@ export function useImageCanvasGenerationSubmissionWorkflow({ let backgroundMusicSubmissionAccepted = false; let backgroundMusicSubmissionUiOwned = false; let backgroundMusicQueueAcceptedWhileCurrentAndOpen = false; + let soundEffectSubmissionAccepted = false; + let soundEffectSubmissionUiOwned = false; + let soundEffectQueueAcceptedWhileCurrentAndOpen = false; try { + const dialogForSubmission = soundEffectSubmission + ? { + ...dialog, + prompt: soundEffectSubmission.prompt, + soundDurationMode: soundEffectSubmission.durationMode, + soundDurationSeconds: soundEffectSubmission.manualDurationSeconds, + soundLoop: soundEffectSubmission.loop, + } + : dialog; const submissionPlan = buildImageGenerationSubmissionPlan({ - dialog, + dialog: dialogForSubmission, layers, nextGeneratedIndex: layerCounterRef.current + 1, canonicalBackgroundMusicPrompt: backgroundMusicSubmission?.prompt, @@ -2145,9 +2221,13 @@ export function useImageCanvasGenerationSubmissionWorkflow({ }); const isBackgroundMusic = submissionPlan.audioKind === 'background-music'; + const isSoundEffect = submissionPlan.audioKind === 'sound-effect'; const backgroundMusicQueueState = isBackgroundMusic ? queuedStateFromResponse(generated) : null; + const soundEffectQueueState = isSoundEffect + ? queuedStateFromResponse(generated) + : null; if (isBackgroundMusic && backgroundMusicSubmission) { backgroundMusicSubmissionAccepted = true; const currentScope = @@ -2192,7 +2272,50 @@ export function useImageCanvasGenerationSubmissionWorkflow({ backgroundMusicSubmission, ) ) { - notifyWalletBalanceMayHaveChanged(onWalletBalanceMayHaveChanged); + notifyWalletBalanceMayHaveChanged( + onWalletBalanceMayHaveChanged, + ); + } + } + if (isSoundEffect && soundEffectSubmission) { + soundEffectSubmissionAccepted = true; + const currentScope = + currentBackgroundMusicSubmissionScopeRef.current; + const currentDialog = getCanvasGenerationDialogById( + soundEffectSubmission.dialogId, + ); + soundEffectQueueAcceptedWhileCurrentAndOpen = + Boolean(soundEffectQueueState) && + currentScope.activeCanvasGenerationDialogId === + soundEffectSubmission.dialogId && + currentDialog?.composerOpen !== false; + soundEffectSubmissionUiOwned = + soundEffectPromptAssist.finishSubmission({ + operation: soundEffectSubmission.operation, + accepted: true, + }); + if ( + soundEffectQueueState && + soundEffectSubmissionUiOwned && + isSoundEffectSubmissionUiTargetCurrent(soundEffectSubmission) + ) { + updateCanvasGenerationDialogById( + soundEffectSubmission.dialogId, + (currentDialog) => + currentDialog.mode === 'audio-sound-effect' + ? { + ...currentDialog, + prompt: soundEffectSubmission.prompt, + soundDurationMode: soundEffectSubmission.durationMode, + soundDurationSeconds: + soundEffectSubmission.manualDurationSeconds, + soundLoop: soundEffectSubmission.loop, + status: 'generating', + composerOpen: false, + errorMessage: undefined, + } + : currentDialog, + ); } } if ( @@ -2210,7 +2333,18 @@ export function useImageCanvasGenerationSubmissionWorkflow({ applyProjectSnapshot(projectSnapshot); } } - : applyProjectSnapshot, + : soundEffectSubmission && applyProjectSnapshot + ? (projectSnapshot) => { + if ( + soundEffectSubmissionUiOwned && + isSoundEffectSubmissionUiTargetCurrent( + soundEffectSubmission, + ) + ) { + applyProjectSnapshot(projectSnapshot); + } + } + : applyProjectSnapshot, backgroundMusicSubmission && onQueuedGenerationTask ? () => { if ( @@ -2221,7 +2355,17 @@ export function useImageCanvasGenerationSubmissionWorkflow({ onQueuedGenerationTask(); } } - : onQueuedGenerationTask, + : soundEffectSubmission && onQueuedGenerationTask + ? () => { + if ( + isBackgroundMusicSubmissionProjectCurrent( + soundEffectSubmission, + ) + ) { + onQueuedGenerationTask(); + } + } + : onQueuedGenerationTask, backgroundMusicSubmission && onWalletBalanceMayHaveChanged ? () => { if ( @@ -2232,7 +2376,17 @@ export function useImageCanvasGenerationSubmissionWorkflow({ onWalletBalanceMayHaveChanged(); } } - : onWalletBalanceMayHaveChanged, + : soundEffectSubmission && onWalletBalanceMayHaveChanged + ? () => { + if ( + isBackgroundMusicSubmissionAccountCurrent( + soundEffectSubmission, + ) + ) { + onWalletBalanceMayHaveChanged(); + } + } + : onWalletBalanceMayHaveChanged, ) ) { return; @@ -2246,6 +2400,13 @@ export function useImageCanvasGenerationSubmissionWorkflow({ ) { return; } + if ( + soundEffectSubmission && + (!soundEffectSubmissionUiOwned || + !isSoundEffectSubmissionUiTargetCurrent(soundEffectSubmission)) + ) { + return; + } if (generated.project && applyProjectSnapshot) { applyProjectSnapshot(generated.project); if (generated.asset) { @@ -2354,9 +2515,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({ backgroundMusicSubmission, ) ) { - notifyWalletBalanceMayHaveChanged( - onWalletBalanceMayHaveChanged, - ); + notifyWalletBalanceMayHaveChanged(onWalletBalanceMayHaveChanged); } } if ( @@ -2391,6 +2550,48 @@ export function useImageCanvasGenerationSubmissionWorkflow({ : latestDialog, ); } + } else if (soundEffectSubmission) { + if (!soundEffectSubmissionAccepted) { + soundEffectSubmissionUiOwned = + soundEffectPromptAssist.finishSubmission({ + operation: soundEffectSubmission.operation, + accepted: false, + }); + } + if ( + soundEffectSubmissionUiOwned && + isSoundEffectSubmissionUiTargetCurrent(soundEffectSubmission) + ) { + const currentScope = + currentBackgroundMusicSubmissionScopeRef.current; + const currentDialog = getCanvasGenerationDialogById( + soundEffectSubmission.dialogId, + ); + const shouldReopenComposer = + currentScope.activeCanvasGenerationDialogId === + soundEffectSubmission.dialogId && + (currentDialog?.composerOpen !== false || + soundEffectQueueAcceptedWhileCurrentAndOpen); + updateCanvasGenerationDialogById( + soundEffectSubmission.dialogId, + (latestDialog) => + latestDialog.mode === 'audio-sound-effect' + ? { + ...latestDialog, + prompt: soundEffectSubmission.prompt, + soundDurationMode: soundEffectSubmission.durationMode, + soundDurationSeconds: + soundEffectSubmission.manualDurationSeconds, + soundLoop: soundEffectSubmission.loop, + status: 'failed', + composerOpen: shouldReopenComposer + ? true + : latestDialog.composerOpen, + errorMessage: resolveImageGenerationErrorMessage(error), + } + : latestDialog, + ); + } } else if (canvasDialog) { updateCanvasGenerationDialogById(canvasDialog.id, () => ({ ...canvasDialog, @@ -2422,6 +2623,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({ isBackgroundMusicSubmissionAccountCurrent, isBackgroundMusicSubmissionProjectCurrent, isBackgroundMusicSubmissionUiTargetCurrent, + isSoundEffectSubmissionUiTargetCurrent, layerCounterRef, layers, quickEditSelectionState, @@ -2430,6 +2632,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({ assetFolderId, applyProjectSnapshot, setGenerateDialog, + soundEffectPromptAssist, updateCanvasGenerationDialogById, upsertGeneratedAsset, onGenerationWarning, diff --git a/src/components/image-editor/useImageCanvasGenerationSurface.tsx b/src/components/image-editor/useImageCanvasGenerationSurface.tsx index bb9d2bd7b..6d4e37f4f 100644 --- a/src/components/image-editor/useImageCanvasGenerationSurface.tsx +++ b/src/components/image-editor/useImageCanvasGenerationSurface.tsx @@ -438,6 +438,7 @@ export function useImageCanvasGenerationSurface({ backgroundMusicPromptAssist={ generationWorkflow.backgroundMusicPromptAssist } + soundEffectPromptAssist={generationWorkflow.soundEffectPromptAssist} updateCanvasGenerationDialogById={updateCanvasGenerationDialogById} hasPendingImageReferenceUploads={hasPendingImageReferenceUploads} generationComposerStyle={generationComposerStyle} diff --git a/src/components/image-editor/useImageCanvasGenerationWorkflow.test.tsx b/src/components/image-editor/useImageCanvasGenerationWorkflow.test.tsx index 60fb3baae..bdb3556da 100644 --- a/src/components/image-editor/useImageCanvasGenerationWorkflow.test.tsx +++ b/src/components/image-editor/useImageCanvasGenerationWorkflow.test.tsx @@ -1671,12 +1671,12 @@ describe('useImageCanvasGenerationWorkflow', () => { expect(generateEditorSoundEffectMock).toHaveBeenCalledWith( expect.objectContaining({ prompt: '金币跳出音', - model: 'audio1.0', + model: 'eleven_text_to_sound_v2', duration: 8, generationInputs: { fields: [ { title: 'prompt', value: '金币跳出音' }, - { title: 'model', value: 'audio1.0' }, + { title: 'model', value: 'eleven_text_to_sound_v2' }, { title: '时长', value: '8秒' }, ], references: [], diff --git a/src/components/image-editor/useImageCanvasGenerationWorkflow.ts b/src/components/image-editor/useImageCanvasGenerationWorkflow.ts index 959118db7..3ad5b7999 100644 --- a/src/components/image-editor/useImageCanvasGenerationWorkflow.ts +++ b/src/components/image-editor/useImageCanvasGenerationWorkflow.ts @@ -133,6 +133,7 @@ import { resolveEditorGenerationMediaReference, useImageCanvasGenerationSubmissionWorkflow, } from './useImageCanvasGenerationSubmissionWorkflow'; +import { useImageCanvasSoundEffectPromptAssist } from './useImageCanvasSoundEffectPromptAssist'; import { useInlineGenerationPlaceholderOwnership } from './useInlineGenerationPlaceholderExpiry'; type CanvasSize = { width: number; height: number }; @@ -1024,6 +1025,13 @@ export function useImageCanvasGenerationWorkflow({ currentUserId, projectId, }); + const soundEffectPromptAssist = useImageCanvasSoundEffectPromptAssist({ + canvasGenerationDialogs, + getCanvasGenerationDialogById, + updateCanvasGenerationDialogById, + currentUserId, + projectId, + }); const previousTaskCountRef = useRef(canvasGenerationDialogs.length); const splittingIconSpritesheetLayerIdsRef = useRef(new Set()); const [ @@ -3635,6 +3643,7 @@ export function useImageCanvasGenerationWorkflow({ ? generateDialog.id : null, backgroundMusicPromptAssist, + soundEffectPromptAssist, getGeneratingDialogPlaceholder, appendCanvasLayersWithResources, captureCanvasHistory, @@ -4245,6 +4254,7 @@ export function useImageCanvasGenerationWorkflow({ quickEditSelectionState, quickEditSelectionSourceLayer, backgroundMusicPromptAssist, + soundEffectPromptAssist, changeUiAssetExtractionTool, changeUiAssetExtractionModel, appendUiAssetExtractionReferences, @@ -4351,6 +4361,7 @@ export function useImageCanvasGenerationWorkflow({ [ effectiveCharacterAnimationPanel, backgroundMusicPromptAssist, + soundEffectPromptAssist, characterAnimationPrice, characterAnimationSourceLayer, clearDeletedLayerGenerationState, diff --git a/src/components/image-editor/useImageCanvasSoundEffectPromptAssist.test.tsx b/src/components/image-editor/useImageCanvasSoundEffectPromptAssist.test.tsx new file mode 100644 index 000000000..7ed310d39 --- /dev/null +++ b/src/components/image-editor/useImageCanvasSoundEffectPromptAssist.test.tsx @@ -0,0 +1,392 @@ +/* @vitest-environment jsdom */ + +import { act, renderHook, waitFor } from '@testing-library/react'; +import { type SetStateAction, useCallback, useRef, useState } from 'react'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import type { CanvasGenerationDialogState } from './ImageCanvasEditorTypes'; +import { useImageCanvasSoundEffectPromptAssist } from './useImageCanvasSoundEffectPromptAssist'; + +const clientMocks = vi.hoisted(() => ({ + optimize: vi.fn(), +})); + +vi.mock('../../services/image-editor/editorProjectClient', () => ({ + optimizeEditorSoundEffectPrompt: clientMocks.optimize, +})); + +type OptimizeResponse = { prompt: string; charCount: number }; + +function createDeferred() { + let resolve!: (value: T) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + return { promise, resolve, reject }; +} + +function createDialog( + id: string, + prompt: string, + overrides: Partial = {}, +): CanvasGenerationDialogState { + return { + id, + mode: 'audio-sound-effect', + prompt, + status: 'idle', + composerOpen: true, + soundModel: 'eleven_text_to_sound_v2', + soundDurationMode: 'manual', + soundDurationSeconds: 5, + soundLoop: false, + ...overrides, + }; +} + +function requireDialog( + dialogs: CanvasGenerationDialogState[], + dialogId: string, +) { + const dialog = dialogs.find((candidate) => candidate.id === dialogId); + if (!dialog) { + throw new Error(`missing test dialog: ${dialogId}`); + } + return dialog; +} + +function useHarness({ + initialDialogs, + currentUserId, + projectId, +}: { + initialDialogs: CanvasGenerationDialogState[]; + currentUserId?: string | null; + projectId?: string | null; +}) { + const [dialogs, setDialogs] = useState(initialDialogs); + const dialogsRef = useRef(dialogs); + dialogsRef.current = dialogs; + + const getCanvasGenerationDialogById = useCallback( + (dialogId: string) => + dialogsRef.current.find((dialog) => dialog.id === dialogId), + [], + ); + const updateCanvasGenerationDialogById = useCallback( + ( + dialogId: string, + updater: ( + dialog: CanvasGenerationDialogState, + ) => CanvasGenerationDialogState | null, + ) => { + const currentDialog = dialogsRef.current.find( + (dialog) => dialog.id === dialogId, + ); + if (!currentDialog) { + return; + } + const nextDialog = updater(currentDialog); + const nextDialogs = nextDialog + ? dialogsRef.current.map((dialog) => + dialog.id === dialogId ? nextDialog : dialog, + ) + : dialogsRef.current.filter((dialog) => dialog.id !== dialogId); + dialogsRef.current = nextDialogs; + setDialogs(nextDialogs); + }, + [], + ); + const updateDialogs = useCallback( + (updater: SetStateAction) => { + const nextDialogs = + typeof updater === 'function' ? updater(dialogsRef.current) : updater; + dialogsRef.current = nextDialogs; + setDialogs(nextDialogs); + }, + [], + ); + const setPrompt = useCallback( + (dialogId: string, prompt: string) => { + updateCanvasGenerationDialogById(dialogId, (dialog) => ({ + ...dialog, + prompt, + })); + }, + [updateCanvasGenerationDialogById], + ); + const removeDialog = useCallback( + (dialogId: string) => { + updateDialogs((currentDialogs) => + currentDialogs.filter((dialog) => dialog.id !== dialogId), + ); + }, + [updateDialogs], + ); + + const promptAssist = useImageCanvasSoundEffectPromptAssist({ + canvasGenerationDialogs: dialogs, + getCanvasGenerationDialogById, + updateCanvasGenerationDialogById, + currentUserId, + projectId, + }); + + return { dialogs, promptAssist, setPrompt, removeDialog }; +} + +describe('useImageCanvasSoundEffectPromptAssist', () => { + beforeEach(() => { + clientMocks.optimize.mockReset(); + }); + + it('canonicalizes before await, collapses double clicks, and creates one undo snapshot', async () => { + const deferred = createDeferred(); + clientMocks.optimize.mockReturnValueOnce(deferred.promise); + const { result } = renderHook(() => + useHarness({ + initialDialogs: [ + createDialog('dialog-a', '\u0085金币落地声\u3000', { + status: 'failed', + errorMessage: '旧错误', + }), + ], + currentUserId: 'user-a', + projectId: 'project-a', + }), + ); + + let firstAction!: ReturnType< + typeof result.current.promptAssist.optimizePrompt + >; + let duplicateResult!: Awaited; + act(() => { + firstAction = result.current.promptAssist.optimizePrompt('dialog-a'); + }); + await act(async () => { + duplicateResult = + await result.current.promptAssist.optimizePrompt('dialog-a'); + }); + + expect(clientMocks.optimize).toHaveBeenCalledTimes(1); + expect(clientMocks.optimize).toHaveBeenCalledWith( + { currentPrompt: '金币落地声' }, + { signal: expect.any(AbortSignal) }, + ); + expect(requireDialog(result.current.dialogs, 'dialog-a')).toMatchObject({ + prompt: '金币落地声', + status: 'idle', + errorMessage: undefined, + }); + expect(duplicateResult.reason).toBe('duplicate'); + expect( + result.current.promptAssist.getDialogState('dialog-a'), + ).toMatchObject({ + status: 'optimizing', + undoPromptSnapshot: null, + temporaryPromptSnapshot: '金币落地声', + }); + + let actionResult!: Awaited; + await act(async () => { + deferred.resolve({ prompt: '清脆明亮的金币落地声', charCount: 10 }); + actionResult = await firstAction; + }); + expect(actionResult.reason).toBe('applied'); + expect(requireDialog(result.current.dialogs, 'dialog-a').prompt).toBe( + '清脆明亮的金币落地声', + ); + expect( + result.current.promptAssist.getDialogState('dialog-a'), + ).toMatchObject({ + status: 'idle', + undoPromptSnapshot: '金币落地声', + temporaryPromptSnapshot: null, + }); + + act(() => { + result.current.setPrompt('dialog-a', '手动改过的结果'); + result.current.promptAssist.undoPrompt('dialog-a'); + }); + expect(requireDialog(result.current.dialogs, 'dialog-a').prompt).toBe( + '金币落地声', + ); + expect( + result.current.promptAssist.getDialogState('dialog-a').undoPromptSnapshot, + ).toBe('手动改过的结果'); + }); + + it('rejects mismatched response counts without exposing the candidate or restoring an older snapshot', async () => { + clientMocks.optimize + .mockResolvedValueOnce({ prompt: '第一次优化结果', charCount: 7 }) + .mockResolvedValueOnce({ prompt: '不应写入的候选', charCount: 999 }); + const { result } = renderHook(() => + useHarness({ + initialDialogs: [createDialog('dialog-a', '原始描述')], + }), + ); + + await act(async () => { + await result.current.promptAssist.optimizePrompt('dialog-a'); + }); + act(() => result.current.setPrompt('dialog-a', '第二次输入')); + let actionResult!: Awaited< + ReturnType + >; + await act(async () => { + actionResult = + await result.current.promptAssist.optimizePrompt('dialog-a'); + }); + + expect(actionResult).toMatchObject({ applied: false, reason: 'failed' }); + expect(requireDialog(result.current.dialogs, 'dialog-a').prompt).toBe( + '第二次输入', + ); + expect(JSON.stringify(result.current)).not.toContain('不应写入的候选'); + expect( + result.current.promptAssist.getDialogState('dialog-a'), + ).toMatchObject({ + status: 'idle', + undoPromptSnapshot: null, + temporaryPromptSnapshot: null, + errorMessage: 'AI 返回的游戏音效描述无效,请稍后重试', + }); + }); + + it('drops a late response after account or project scope changes', async () => { + const deferred = createDeferred(); + clientMocks.optimize.mockReturnValueOnce(deferred.promise); + const { result, rerender } = renderHook( + ({ currentUserId, projectId }) => + useHarness({ + initialDialogs: [createDialog('dialog-a', '旧作用域输入')], + currentUserId, + projectId, + }), + { + initialProps: { + currentUserId: 'user-a', + projectId: 'project-a', + }, + }, + ); + + let action!: ReturnType; + act(() => { + action = result.current.promptAssist.optimizePrompt('dialog-a'); + }); + act(() => { + rerender({ currentUserId: 'user-b', projectId: 'project-b' }); + result.current.setPrompt('dialog-a', '新作用域输入'); + }); + let actionResult!: Awaited; + await act(async () => { + deferred.resolve({ prompt: '旧响应候选', charCount: 6 }); + actionResult = await action; + }); + + expect(actionResult.reason).toBe('stale'); + expect(requireDialog(result.current.dialogs, 'dialog-a').prompt).toBe( + '新作用域输入', + ); + expect(JSON.stringify(result.current)).not.toContain('旧响应候选'); + }); + + it('removes dialog state and ignores a response after dialog deletion', async () => { + const deferred = createDeferred(); + clientMocks.optimize.mockReturnValueOnce(deferred.promise); + const { result } = renderHook(() => + useHarness({ initialDialogs: [createDialog('dialog-a', '输入')] }), + ); + let action!: ReturnType; + act(() => { + action = result.current.promptAssist.optimizePrompt('dialog-a'); + result.current.removeDialog('dialog-a'); + }); + await act(async () => { + deferred.resolve({ prompt: '迟到候选', charCount: 4 }); + await action; + }); + expect(result.current.dialogs).toEqual([]); + expect( + result.current.promptAssist.dialogStates['dialog-a'], + ).toBeUndefined(); + }); + + it('freezes duration mode, manual value, and Loop in one synchronous submission claim', () => { + const { result } = renderHook(() => + useHarness({ + initialDialogs: [ + createDialog('dialog-a', ' \u0085循环环境声\u3000', { + soundDurationMode: 'auto', + soundDurationSeconds: 7.3, + soundLoop: true, + }), + ], + }), + ); + + let claim!: ReturnType; + act(() => { + claim = result.current.promptAssist.beginSubmission('dialog-a'); + }); + expect(claim).toMatchObject({ + prompt: '循环环境声', + durationMode: 'auto', + manualDurationSeconds: 7.3, + loop: true, + }); + expect(result.current.promptAssist.getDialogState('dialog-a').status).toBe( + 'submitting', + ); + let duplicateClaim: ReturnType< + typeof result.current.promptAssist.beginSubmission + > = claim; + act(() => { + duplicateClaim = result.current.promptAssist.beginSubmission('dialog-a'); + }); + expect(duplicateClaim).toBeNull(); + + let finished = false; + act(() => { + finished = result.current.promptAssist.finishSubmission({ + operation: claim!.operation, + accepted: false, + }); + }); + expect(finished).toBe(true); + expect(result.current.promptAssist.getDialogState('dialog-a').status).toBe( + 'idle', + ); + expect(requireDialog(result.current.dialogs, 'dialog-a').prompt).toBe( + '循环环境声', + ); + }); + + it('clears the undo snapshot when a preset boundary is prepared', async () => { + clientMocks.optimize.mockResolvedValueOnce({ + prompt: '优化结果', + charCount: 4, + }); + const { result } = renderHook(() => + useHarness({ initialDialogs: [createDialog('dialog-a', '原始输入')] }), + ); + await act(async () => { + await result.current.promptAssist.optimizePrompt('dialog-a'); + }); + expect( + result.current.promptAssist.getDialogState('dialog-a').undoPromptSnapshot, + ).toBe('原始输入'); + act(() => { + result.current.promptAssist.preparePreset('dialog-a'); + }); + await waitFor(() => { + expect( + result.current.promptAssist.getDialogState('dialog-a') + .undoPromptSnapshot, + ).toBeNull(); + }); + }); +}); diff --git a/src/components/image-editor/useImageCanvasSoundEffectPromptAssist.ts b/src/components/image-editor/useImageCanvasSoundEffectPromptAssist.ts new file mode 100644 index 000000000..31b5a04a3 --- /dev/null +++ b/src/components/image-editor/useImageCanvasSoundEffectPromptAssist.ts @@ -0,0 +1,600 @@ +import { + useCallback, + useEffect, + useLayoutEffect, + useMemo, + useRef, + useState, +} from 'react'; + +import { + type EditorSoundEffectDurationMode, + SOUND_EFFECT_DURATION_MAX_SECONDS, + SOUND_EFFECT_DURATION_MIN_SECONDS, +} from '../../../packages/shared/src/contracts/editorAudio'; +import { optimizeEditorSoundEffectPrompt } from '../../services/image-editor/editorProjectClient'; +import type { CanvasGenerationDialogState } from './ImageCanvasEditorTypes'; +import { + canonicalizeSoundEffectPrompt, + countSoundEffectPromptCodePoints, + createSoundEffectPromptStateModel, + type SoundEffectPromptDialogState, + type SoundEffectPromptOperation, + validateSoundEffectPrompt, +} from './ImageCanvasSoundEffectPromptModel'; + +const DEFAULT_MANUAL_DURATION_SECONDS = 5; + +type CanvasGenerationDialogUpdater = ( + dialog: CanvasGenerationDialogState, +) => CanvasGenerationDialogState | null; + +type ActiveSoundEffectOperation = { + operation: SoundEffectPromptOperation; + abortController: AbortController | null; + scopeKey: string; +}; + +export type SoundEffectPromptAssistDialogState = + SoundEffectPromptDialogState & { + errorMessage: string | null; + }; + +export type SoundEffectPromptAssistActionResult = { + started: boolean; + applied: boolean; + reason: + | 'applied' + | 'duplicate' + | 'ineligible' + | 'locked' + | 'stale' + | 'failed'; + errorMessage: string | null; +}; + +export type SoundEffectPromptSubmissionClaim = { + operation: SoundEffectPromptOperation; + prompt: string; + durationMode: EditorSoundEffectDurationMode; + manualDurationSeconds: number; + loop: boolean; +}; + +export type SoundEffectPromptAssistController = ReturnType< + typeof useImageCanvasSoundEffectPromptAssist +>; + +export type SoundEffectPromptAssistComposerController = Pick< + SoundEffectPromptAssistController, + 'getDialogState' | 'optimizePrompt' | 'preparePreset' | 'undoPrompt' +>; + +function isSoundEffectDialog( + dialog: CanvasGenerationDialogState | undefined, +): dialog is CanvasGenerationDialogState & { mode: 'audio-sound-effect' } { + return dialog?.mode === 'audio-sound-effect'; +} + +function isAbortError(error: unknown) { + return ( + (typeof DOMException !== 'undefined' && + error instanceof DOMException && + error.name === 'AbortError') || + (error instanceof Error && error.name === 'AbortError') + ); +} + +function resolvePromptAssistErrorMessage(error: unknown) { + if (isAbortError(error)) { + return null; + } + if (error instanceof Error && error.message.trim()) { + return error.message.trim(); + } + return 'AI 暂时无法优化游戏音效描述,请稍后重试'; +} + +function invalidPromptAssistResponseError() { + return new Error('AI 返回的游戏音效描述无效,请稍后重试'); +} + +function resolveScopeKey( + currentUserId: string | null | undefined, + projectId: string | null | undefined, +) { + return JSON.stringify([currentUserId ?? null, projectId ?? null]); +} + +function normalizeManualDuration(value: number | null | undefined) { + if (typeof value !== 'number' || !Number.isFinite(value)) { + return DEFAULT_MANUAL_DURATION_SECONDS; + } + return Math.min( + SOUND_EFFECT_DURATION_MAX_SECONDS, + Math.max(SOUND_EFFECT_DURATION_MIN_SECONDS, value), + ); +} + +export function useImageCanvasSoundEffectPromptAssist({ + canvasGenerationDialogs, + getCanvasGenerationDialogById, + updateCanvasGenerationDialogById, + currentUserId, + projectId, +}: { + canvasGenerationDialogs: CanvasGenerationDialogState[]; + getCanvasGenerationDialogById: ( + dialogId: string, + ) => CanvasGenerationDialogState | undefined; + updateCanvasGenerationDialogById: ( + dialogId: string, + updater: CanvasGenerationDialogUpdater, + ) => void; + currentUserId?: string | null; + projectId?: string | null; +}) { + const modelRef = useRef(createSoundEffectPromptStateModel()); + const activeOperationsRef = useRef( + new Map(), + ); + const errorMessagesRef = useRef(new Map()); + const knownDialogIdsRef = useRef(new Set()); + const scopeKey = useMemo( + () => resolveScopeKey(currentUserId, projectId), + [currentUserId, projectId], + ); + const committedScopeKeyRef = useRef(scopeKey); + const [dialogStates, setDialogStates] = useState< + Record + >({}); + + const publishDialogState = useCallback((dialogId: string) => { + const nextState: SoundEffectPromptAssistDialogState = { + ...modelRef.current.getDialogState(dialogId), + errorMessage: errorMessagesRef.current.get(dialogId) ?? null, + }; + setDialogStates((currentStates) => ({ + ...currentStates, + [dialogId]: nextState, + })); + return nextState; + }, []); + + const updateDialogPrompt = useCallback( + (dialogId: string, prompt: string) => { + updateCanvasGenerationDialogById(dialogId, (dialog) => + dialog.mode === 'audio-sound-effect' + ? { + ...dialog, + prompt, + status: dialog.status === 'failed' ? 'idle' : dialog.status, + errorMessage: + dialog.status === 'failed' ? undefined : dialog.errorMessage, + } + : dialog, + ); + }, + [updateCanvasGenerationDialogById], + ); + + const clearActiveOperation = useCallback( + ( + dialogId: string, + options: { abort: boolean; removeDialogState: boolean }, + ) => { + const activeOperation = activeOperationsRef.current.get(dialogId); + if (activeOperation) { + if (options.abort) { + activeOperation.abortController?.abort(); + } + modelRef.current.rejectOperation(activeOperation.operation); + activeOperationsRef.current.delete(dialogId); + } + errorMessagesRef.current.delete(dialogId); + if (options.removeDialogState) { + modelRef.current.closeDialog(dialogId); + setDialogStates((currentStates) => { + if (!(dialogId in currentStates)) { + return currentStates; + } + const nextStates = { ...currentStates }; + delete nextStates[dialogId]; + return nextStates; + }); + return; + } + if (activeOperation) { + publishDialogState(dialogId); + } + }, + [publishDialogState], + ); + + const discardStaleOperation = useCallback( + (operation: SoundEffectPromptOperation) => { + if ( + modelRef.current.getDialogState(operation.dialogId).operationId !== + operation.operationId + ) { + return; + } + modelRef.current.rejectOperation(operation); + activeOperationsRef.current.delete(operation.dialogId); + publishDialogState(operation.dialogId); + }, + [publishDialogState], + ); + + useEffect(() => { + const currentDialogIds = new Set( + canvasGenerationDialogs + .filter(isSoundEffectDialog) + .map((dialog) => dialog.id), + ); + for (const dialogId of knownDialogIdsRef.current) { + if (!currentDialogIds.has(dialogId)) { + clearActiveOperation(dialogId, { + abort: true, + removeDialogState: true, + }); + } + } + for (const [dialogId, activeOperation] of activeOperationsRef.current) { + if (activeOperation.operation.status === 'submitting') { + continue; + } + const dialog = getCanvasGenerationDialogById(dialogId); + if (!isSoundEffectDialog(dialog) || dialog.composerOpen === false) { + clearActiveOperation(dialogId, { + abort: true, + removeDialogState: !isSoundEffectDialog(dialog), + }); + } + } + knownDialogIdsRef.current = currentDialogIds; + }, [ + canvasGenerationDialogs, + clearActiveOperation, + getCanvasGenerationDialogById, + ]); + + useLayoutEffect(() => { + if (Object.is(committedScopeKeyRef.current, scopeKey)) { + return; + } + committedScopeKeyRef.current = scopeKey; + for (const activeOperation of activeOperationsRef.current.values()) { + activeOperation.abortController?.abort(); + } + activeOperationsRef.current.clear(); + errorMessagesRef.current.clear(); + modelRef.current.reset(); + knownDialogIdsRef.current = new Set( + canvasGenerationDialogs + .filter(isSoundEffectDialog) + .map((dialog) => dialog.id), + ); + setDialogStates({}); + }, [canvasGenerationDialogs, scopeKey]); + + useEffect( + () => () => { + for (const activeOperation of activeOperationsRef.current.values()) { + activeOperation.abortController?.abort(); + } + activeOperationsRef.current.clear(); + }, + [], + ); + + const optimizePrompt = useCallback( + async (dialogId: string): Promise => { + const dialog = getCanvasGenerationDialogById(dialogId); + if (!isSoundEffectDialog(dialog) || dialog.composerOpen === false) { + return { + started: false, + applied: false, + reason: 'stale', + errorMessage: null, + }; + } + const canonicalPrompt = canonicalizeSoundEffectPrompt(dialog.prompt); + updateDialogPrompt(dialogId, canonicalPrompt); + const start = modelRef.current.beginOperation({ + dialogId, + status: 'optimizing', + prompt: canonicalPrompt, + }); + if (!start.started || !start.operation) { + publishDialogState(dialogId); + return { + started: false, + applied: false, + reason: start.reason, + errorMessage: null, + }; + } + + const abortController = new AbortController(); + const operation = start.operation; + const operationScopeKey = committedScopeKeyRef.current; + activeOperationsRef.current.set(dialogId, { + operation, + abortController, + scopeKey: operationScopeKey, + }); + errorMessagesRef.current.delete(dialogId); + publishDialogState(dialogId); + + try { + const response = await optimizeEditorSoundEffectPrompt( + { currentPrompt: operation.canonicalPrompt }, + { signal: abortController.signal }, + ); + const activeOperation = activeOperationsRef.current.get(dialogId); + const latestDialog = getCanvasGenerationDialogById(dialogId); + if ( + activeOperation?.operation.operationId !== operation.operationId || + !Object.is(activeOperation.scopeKey, operationScopeKey) || + !Object.is(operationScopeKey, committedScopeKeyRef.current) + ) { + discardStaleOperation(operation); + return { + started: true, + applied: false, + reason: 'stale', + errorMessage: null, + }; + } + if ( + !isSoundEffectDialog(latestDialog) || + latestDialog.composerOpen === false + ) { + clearActiveOperation(dialogId, { + abort: false, + removeDialogState: !isSoundEffectDialog(latestDialog), + }); + return { + started: true, + applied: false, + reason: 'stale', + errorMessage: null, + }; + } + + const validation = validateSoundEffectPrompt(response.prompt); + if ( + !validation.ok || + !Number.isInteger(response.charCount) || + response.charCount !== + countSoundEffectPromptCodePoints(validation.prompt) + ) { + throw invalidPromptAssistResponseError(); + } + const resolution = modelRef.current.resolveAiOperation( + operation, + validation.prompt, + ); + activeOperationsRef.current.delete(dialogId); + if (!resolution.applied || !resolution.prompt) { + throw invalidPromptAssistResponseError(); + } + errorMessagesRef.current.delete(dialogId); + updateDialogPrompt(dialogId, resolution.prompt); + publishDialogState(dialogId); + return { + started: true, + applied: true, + reason: 'applied', + errorMessage: null, + }; + } catch (error) { + const activeOperation = activeOperationsRef.current.get(dialogId); + if ( + activeOperation?.operation.operationId !== operation.operationId || + !Object.is(activeOperation.scopeKey, operationScopeKey) || + !Object.is(operationScopeKey, committedScopeKeyRef.current) + ) { + discardStaleOperation(operation); + return { + started: true, + applied: false, + reason: 'stale', + errorMessage: null, + }; + } + const latestDialog = getCanvasGenerationDialogById(dialogId); + if ( + !isSoundEffectDialog(latestDialog) || + latestDialog.composerOpen === false + ) { + clearActiveOperation(dialogId, { + abort: false, + removeDialogState: !isSoundEffectDialog(latestDialog), + }); + return { + started: true, + applied: false, + reason: 'stale', + errorMessage: null, + }; + } + const resolution = modelRef.current.rejectOperation(operation); + activeOperationsRef.current.delete(dialogId); + if (resolution.applied && resolution.prompt) { + updateDialogPrompt(dialogId, resolution.prompt); + } + const errorMessage = resolvePromptAssistErrorMessage(error); + if (errorMessage) { + errorMessagesRef.current.set(dialogId, errorMessage); + } else { + errorMessagesRef.current.delete(dialogId); + } + publishDialogState(dialogId); + return { + started: true, + applied: false, + reason: errorMessage ? 'failed' : 'stale', + errorMessage, + }; + } finally { + const activeOperation = activeOperationsRef.current.get(dialogId); + if ( + activeOperation?.operation.operationId === operation.operationId && + Object.is(activeOperation.scopeKey, operationScopeKey) + ) { + activeOperationsRef.current.delete(dialogId); + } + } + }, + [ + clearActiveOperation, + discardStaleOperation, + getCanvasGenerationDialogById, + publishDialogState, + updateDialogPrompt, + ], + ); + + const preparePreset = useCallback( + (dialogId: string) => { + const dialog = getCanvasGenerationDialogById(dialogId); + if (!isSoundEffectDialog(dialog) || dialog.composerOpen === false) { + return null; + } + const prepared = modelRef.current.preparePreset(dialogId, dialog.prompt); + if (!prepared.applied) { + publishDialogState(dialogId); + return null; + } + updateDialogPrompt(dialogId, prepared.prompt); + errorMessagesRef.current.delete(dialogId); + publishDialogState(dialogId); + return prepared.prompt; + }, + [getCanvasGenerationDialogById, publishDialogState, updateDialogPrompt], + ); + + const undoPrompt = useCallback( + (dialogId: string) => { + const dialog = getCanvasGenerationDialogById(dialogId); + if (!isSoundEffectDialog(dialog) || dialog.composerOpen === false) { + return false; + } + const canonicalPrompt = canonicalizeSoundEffectPrompt(dialog.prompt); + updateDialogPrompt(dialogId, canonicalPrompt); + const swapped = modelRef.current.swapUndoSnapshot( + dialogId, + canonicalPrompt, + ); + if (!swapped.applied) { + return false; + } + errorMessagesRef.current.delete(dialogId); + updateDialogPrompt(dialogId, swapped.prompt); + publishDialogState(dialogId); + return true; + }, + [getCanvasGenerationDialogById, publishDialogState, updateDialogPrompt], + ); + + const beginSubmission = useCallback( + (dialogId: string): SoundEffectPromptSubmissionClaim | null => { + const dialog = getCanvasGenerationDialogById(dialogId); + if (!isSoundEffectDialog(dialog) || dialog.composerOpen === false) { + return null; + } + const start = modelRef.current.beginOperation({ + dialogId, + status: 'submitting', + prompt: dialog.prompt, + }); + if (start.reason !== 'duplicate') { + updateDialogPrompt(dialogId, start.canonicalPrompt); + } + if (!start.started || !start.operation) { + publishDialogState(dialogId); + return null; + } + const operationScopeKey = committedScopeKeyRef.current; + activeOperationsRef.current.set(dialogId, { + operation: start.operation, + abortController: null, + scopeKey: operationScopeKey, + }); + errorMessagesRef.current.delete(dialogId); + publishDialogState(dialogId); + return { + operation: start.operation, + prompt: start.canonicalPrompt, + durationMode: dialog.soundDurationMode === 'auto' ? 'auto' : 'manual', + manualDurationSeconds: normalizeManualDuration( + dialog.soundDurationSeconds, + ), + loop: dialog.soundLoop === true, + }; + }, + [getCanvasGenerationDialogById, publishDialogState, updateDialogPrompt], + ); + + const finishSubmission = useCallback( + ({ + operation, + accepted, + }: { + operation: SoundEffectPromptOperation; + accepted: boolean; + }) => { + const activeOperation = activeOperationsRef.current.get( + operation.dialogId, + ); + if ( + activeOperation?.operation.operationId !== operation.operationId || + !Object.is(activeOperation.scopeKey, committedScopeKeyRef.current) + ) { + return false; + } + const resolution = accepted + ? modelRef.current.completeSubmittingOperation(operation) + : modelRef.current.rejectOperation(operation); + activeOperationsRef.current.delete(operation.dialogId); + if (!resolution.applied) { + return false; + } + publishDialogState(operation.dialogId); + return true; + }, + [publishDialogState], + ); + + const getDialogState = useCallback( + (dialogId: string) => + dialogStates[dialogId] ?? { + ...modelRef.current.getDialogState(dialogId), + errorMessage: errorMessagesRef.current.get(dialogId) ?? null, + }, + [dialogStates], + ); + + return useMemo( + () => ({ + dialogStates, + getDialogState, + optimizePrompt, + preparePreset, + undoPrompt, + beginSubmission, + finishSubmission, + }), + [ + beginSubmission, + dialogStates, + finishSubmission, + getDialogState, + optimizePrompt, + preparePreset, + undoPrompt, + ], + ); +} diff --git a/src/index.css b/src/index.css index 7a3afb392..9a81e9423 100644 --- a/src/index.css +++ b/src/index.css @@ -16892,6 +16892,40 @@ button { min-width: 0; } +.image-canvas-editor__sound-effect-prompt-actions { + grid-column: 1 / -1; +} + +.image-canvas-editor__sound-effect-prompt-action--optimize:not(:disabled) { + border-color: var(--image-canvas-brand-border-strong); + color: var(--image-canvas-brand-accent); +} + +.image-canvas-editor__sound-effect-prompt-action--optimize:not( + :disabled + ):hover { + background: var(--image-canvas-brand-soft); +} + +.image-canvas-editor__sound-effect-auto-duration { + display: flex; + align-items: center; + gap: 0.45rem; + color: #475569; + font-size: 0.76rem; + font-weight: 700; +} + +.image-canvas-editor__sound-effect-auto-duration input { + margin: 0; +} + +.image-canvas-editor__sound-effect-loop[aria-pressed='true'] { + border-color: var(--image-canvas-brand-border-strong); + background: var(--image-canvas-brand-soft); + color: var(--image-canvas-brand-accent); +} + /* * 助手动作按钮:`.platform-button` 给的是 2.9rem 高与 0.7rem/1rem 内边距,比同一面板里 * 2.25rem 的生成按钮高出一截。这里按两级选择器压回去,避免整排按钮比主操作还重。 @@ -17122,6 +17156,34 @@ button { color: #0f766e; } +.image-canvas-editor__sound-effect-preset--ui-operation, +.image-canvas-editor__sound-effect-preset--pickup-reward { + border-color: rgba(37, 99, 235, 0.24); + background: rgba(219, 234, 254, 0.7); + color: #1d4ed8; +} + +.image-canvas-editor__sound-effect-preset--growth-result, +.image-canvas-editor__sound-effect-preset--character-combat { + border-color: rgba(217, 119, 6, 0.24); + background: rgba(254, 243, 199, 0.7); + color: #b45309; +} + +.image-canvas-editor__sound-effect-preset--skill-status, +.image-canvas-editor__sound-effect-preset--mechanism-scene-interaction { + border-color: rgba(13, 148, 136, 0.24); + background: rgba(204, 251, 241, 0.7); + color: #0f766e; +} + +.image-canvas-editor__sound-effect-preset--style-direction, +.image-canvas-editor__sound-effect-preset--feedback-requirement { + border-color: rgba(124, 58, 237, 0.24); + background: rgba(237, 233, 254, 0.72); + color: #6d28d9; +} + .image-canvas-editor__background-music-presets-hairline { position: absolute; right: 1.74rem; diff --git a/src/services/image-editor/editorProjectClient.test.ts b/src/services/image-editor/editorProjectClient.test.ts index d5200a8d3..bc41024f3 100644 --- a/src/services/image-editor/editorProjectClient.test.ts +++ b/src/services/image-editor/editorProjectClient.test.ts @@ -12,6 +12,7 @@ import { deleteEditorProject, editEditorImage, EDITOR_BACKGROUND_MUSIC_PROMPT_ASSIST_TIMEOUT_MS, + EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZE_TIMEOUT_MS, extractEditorUiDesignAssets, generateEditorBackgroundMusic, generateEditorCharacterAnimation, @@ -25,6 +26,7 @@ import { loadEditorGenerationPricing, loadEditorProject, loadOrCreateRecentEditorProject, + optimizeEditorSoundEffectPrompt, removeEditorImageBackground, renameEditorProject, saveEditorProjectLayout, @@ -2035,6 +2037,44 @@ describe('editorProjectClient', () => { expect(requestJsonMock.mock.calls[0][3]).toEqual(promptAssistRequestOptions); }); + it('optimizes an SFX prompt through the authenticated internal BFF without retries or client-controlled limits', async () => { + requestJsonMock.mockResolvedValueOnce({ + prompt: '清脆明亮的金币落地声', + charCount: 10, + }); + const controller = new AbortController(); + + const result = await optimizeEditorSoundEffectPrompt( + { + currentPrompt: '金币落地声', + targetChars: 2048, + model: 'client-must-not-control', + } as Parameters[0] & { + targetChars: number; + model: string; + }, + { signal: controller.signal }, + ); + + expect(result).toEqual({ + prompt: '清脆明亮的金币落地声', + charCount: 10, + }); + expect(requestJsonMock).toHaveBeenCalledWith( + '/api/editor/audios/sound-effects/prompts/optimizations', + { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ currentPrompt: '金币落地声' }), + signal: controller.signal, + }, + '优化游戏音效提示词失败', + { timeoutMs: EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZE_TIMEOUT_MS }, + ); + expect(EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZE_TIMEOUT_MS).toBe(180_000); + expect(requestJsonMock.mock.calls[0]).toHaveLength(4); + }); + it.each([ [ 'completion', diff --git a/src/services/image-editor/editorProjectClient.ts b/src/services/image-editor/editorProjectClient.ts index ad75b602b..33b6ff077 100644 --- a/src/services/image-editor/editorProjectClient.ts +++ b/src/services/image-editor/editorProjectClient.ts @@ -5,6 +5,8 @@ import { type EditorSoundEffectGenerationRequest, SOUND_EFFECT_DURATION_MAX_SECONDS, SOUND_EFFECT_DURATION_MIN_SECONDS, + type SoundEffectPromptOptimizeRequest, + type SoundEffectPromptOptimizeResponse, } from '../../../packages/shared/src/contracts/editorAudio'; import type { ExternalGenerationJobStatusRecord } from '../../../packages/shared/src/contracts/externalGeneration'; import { requestJson } from '../apiClient'; @@ -36,11 +38,14 @@ const EDITOR_BACKGROUND_MUSIC_PROMPT_COMPLETION_API = '/api/editor/audios/background-music/prompts/completions'; const EDITOR_BACKGROUND_MUSIC_PROMPT_SIMPLIFICATION_API = '/api/editor/audios/background-music/prompts/simplifications'; +const EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZATION_API = + '/api/editor/audios/sound-effects/prompts/optimizations'; /** * 覆盖简化两个业务语义轮加单轮 transport retry 的服务端预算,同时为浏览器到 BFF 的 * 悬挂连接提供有界退出;超时后走既有失败路径恢复 idle,不写回候选。 */ export const EDITOR_BACKGROUND_MUSIC_PROMPT_ASSIST_TIMEOUT_MS = 180_000; +export const EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZE_TIMEOUT_MS = 180_000; const EDITOR_GENERATION_PRICING_API = '/api/editor/generation-pricing'; const EDITOR_IMAGE_MODEL_NANOBANANA2 = 'gemini-3.1-flash-image-preview'; const EDITOR_IMAGE_REFERENCE_LIMIT = 5; @@ -575,6 +580,10 @@ export type EditorBackgroundMusicPromptAssistOptions = { signal?: AbortSignal; }; +export type EditorSoundEffectPromptOptimizeOptions = { + signal?: AbortSignal; +}; + export type EditorAudioGenerationResult = { audioSrc: string; objectKey?: string | null; @@ -1437,6 +1446,25 @@ export function simplifyEditorBackgroundMusicPrompt( ); } +export function optimizeEditorSoundEffectPrompt( + input: SoundEffectPromptOptimizeRequest, + options: EditorSoundEffectPromptOptimizeOptions = {}, +) { + return requestJson( + EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZATION_API, + { + ...jsonRequest('POST', { + currentPrompt: input.currentPrompt, + }), + signal: options.signal, + }, + '优化游戏音效提示词失败', + { + timeoutMs: EDITOR_SOUND_EFFECT_PROMPT_OPTIMIZE_TIMEOUT_MS, + }, + ); +} + export async function generateEditorBackgroundMusic( input: EditorBackgroundMusicGenerationInput, ) {