diff --git a/docs/project-memory/shared-memory/decision-log.md b/docs/project-memory/shared-memory/decision-log.md index 3595be133..14ccc89bb 100644 --- a/docs/project-memory/shared-memory/decision-log.md +++ b/docs/project-memory/shared-memory/decision-log.md @@ -16,10 +16,17 @@ --- +## 2026-07-17 生成后抠图原图以 OSS 作为内存生命周期边界 + +- 背景：角色形象、图标图集和 UI 素材图集的生成原图虽然已先落私有 OSS，但 api-server 仍把带背景原图字节保留到 BgFilter / 阿里云 / 本地 fallback 结束，造成并发任务下的内存峰值叠加。 +- 决策：目标链路的带背景原图上传 OSS 时消费 `DownloadedImage` 所有权，不为上传克隆整张字节缓冲；上传完成后不再跨 BgFilter 调用常驻。BgFilter 只读取 600 秒签名 URL；进入阿里云 fallback 时由 `platform-matting` 新 URL 接口下载原图、上传 `AuthorizeFileUpload` 临时对象，并在开始阿里云推理前结束下载缓冲作用域；阿里云继续失败时，api-server 再从私有 OSS 独立下载原图供本地键色，产出后释放本次原图下载缓冲。 +- 边界：不改变接口 DTO、资源记录、画布原图展示、图集切分行为和降级顺序；“释放”指 Rust 所有权和 `Vec` 析构，RSS 不保证同步下降。 +- 验证方式：`platform-matting` 测试覆盖 URL 下载缓冲在临时上传后结束、降尺寸 Alpha 回贴；`api-server` 结构测试覆盖带背景原图 owned 上传、URL 阿里云 fallback、本地重新下载与原图释放；随后运行两个 crate 的测试与编译检查。 + ## 2026-07-15 BgFilter 输入改用私有 OSS 短期签名 URL - 背景：角色形象、图标图集和 UI 素材图集在调用 BgFilter 前已经把带背景原图持久化到私有 OSS；继续由 api-server 把同一图片作为 multipart `file` 再上传一次，会重复传输图片字节并占用 API 进程网络带宽。 -- 决策：上述生成后抠图链路统一复用已持久化原图的 object key，签发 600 秒 OSS GET URL，并通过 BgFilter multipart 的 `image_url` 字段提交；请求中不再携带 `file`。签名 URL 只交给 BgFilter，不写日志或持久化。BgFilter 失败或熔断打开后才使用已保留的图片字节进入原有“阿里云通用抠图 → 本地键色”兜底链。 +- 决策：上述生成后抠图链路统一复用已持久化原图的 object key，签发 600 秒 OSS GET URL，并通过 BgFilter multipart 的 `image_url` 字段提交；请求中不再携带 `file`。签名 URL 只交给 BgFilter，不写日志或持久化。2026-07-17 起，原图上传后不再保留图片字节；进入“阿里云通用抠图 → 本地键色”兜底链时按阶段从私有 OSS 重新下载。 - 影响范围：仅 `api-server/editor_project` 的 BgFilter 请求输入与相关文档；不改变 BgFilter endpoint、鉴权、`screen_color`、`seg_model`、输出校验、熔断规则、阿里云上传协议或降级顺序。 - 验证方式：定向测试必须断言 BgFilter 请求函数包含 `image_url` 与 600 秒 OSS 换签，不包含 multipart `file` 或源图字节读取；随后运行 `cargo check -p api-server --manifest-path server-rs/Cargo.toml`。 diff --git a/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md b/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md index a15052ec9..43fd27693 100644 --- a/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md +++ b/docs/【后端架构】server-rs与SpacetimeDB数据契约-2026-05-15.md @@ -241,7 +241,7 @@ npm run check:server-rs-ddd - LLM：通用 LLM 门面继续使用 `GENARRATIVE_LLM_*`；创意 Agent `gpt-5.4-mini` Chat Completions 文本链路已于 2026-06 从 APIMart 迁移到 VectorEngine，使用 `VECTOR_ENGINE_BASE_URL` / `VECTOR_ENGINE_API_KEY` 构造 OpenAI-compatible client，`api-server` 会把未带 `/v1` 的 VectorEngine base URL 规范化到 `/v1` 后请求 `/chat/completions`。通用 `/api/llm/chat/completions` 代理使用 `GENARRATIVE_LLM_PROVIDER=openai-compatible`、`GENARRATIVE_LLM_BASE_URL=https://api.vectorengine.cn/v1`、`GENARRATIVE_LLM_MODEL=gpt-5.4-mini`；未单独配置 `GENARRATIVE_LLM_API_KEY` 时可复用 `VECTOR_ENGINE_API_KEY`。`APIMART_BASE_URL` / `APIMART_API_KEY` 只作为历史残留，不再作为创意 Agent gpt-5.4-mini 客户端来源；后续排障时优先确认 VectorEngine `/v1/models`、`/v1/chat/completions` 和 `/v1/responses` 可用性。 - 图片生成：VectorEngine `gpt-image-2` 图片 provider 归属 `platform-image`，密钥只在后端环境变量中；`api-server` 内的 `openai_image_generation.rs` 只是兼容调用面和外部失败审计桥接，不再承载 provider 协议实现。实际外部生成运行记录统一落 `tracking_event`，`event_key = external_generation_run`，metadata 记录开始 / 结束时间、耗时、状态、成功标记、失败原因、provider task id 和结果摘要，不再写回过时的 `ai_task`。DashScope 只按仍在使用的历史能力单独处理，不作为 GPT-image-2 兜底。VectorEngine `/v1/images/generations` 和 `/v1/images/edits` 上游 POST 使用 `libcurl` 发送；`reqwest` 只保留给参考图 URL 下载和响应中图片 URL 下载。`/v1/images/edits` 的 multipart 参考图必须作为 libcurl 文件上传 part 发送，字段名为 `image`，实现上使用 `Form::buffer(file_name, bytes)` 并设置 `Content-Type`；不能只用 `contents(...).filename(...)`，否则上游会把请求转码为缺少图片并返回 `image is required`。`request_send` 阶段的 curl timeout / connect error 按可重试传输错误处理，最多尝试 5 次，并使用指数退避加短抖动；排障时优先看 `attempt`、`max_attempts`、`retry_delay_ms`、`reference_image_bytes_total` 和 `request_params`，不要把 `SendRequest` 当成上游业务错误。 - 编辑器抠图服务：手动 `POST /api/editor/images/background-removals` 继续代理独立 BiRefNet 服务，配置为 `GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_BASE_URL`、`GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_TOKEN` 和 `GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_REQUEST_TIMEOUT_MS`。角色形象生成、图标 spritesheet 生成和 UI 设计图素材提取的生成后纯色背景透明化改走独立 BgFilter 服务，配置为 `GENARRATIVE_EDITOR_BGFILTER_BASE_URL`、`GENARRATIVE_EDITOR_BGFILTER_TOKEN` 和 `GENARRATIVE_EDITOR_BGFILTER_REQUEST_TIMEOUT_MS`，默认 base URL 为 `http://58.87.105.82/bgfilter`，默认请求超时为 `180000ms`（BgFilter 当前为 CPU 推理，单次抠图较慢，必须留足超时），token 未配置时复用 BiRefNet token。BgFilter 请求必须显式传 `screen_color=` 和 `seg_model=`；前端用户路径不展示抠图模型选择并固定提交默认 `birefnet`，后端仍识别内部保留的 `anime-seg`，其中 `birefnet` 只表示 BgFilter 管线内部后端，不等同于手动去背景的独立 BiRefNet 服务。BgFilter 调用失败，或连续失败达到 `GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_FAILURE_THRESHOLD`（默认 `3`）并在 `GENARRATIVE_EDITOR_BGFILTER_CIRCUIT_COOLDOWN_SECONDS`（默认 `300`）内打开熔断时，均跳过或结束 BgFilter 调用后复用同一兜底链：先调用阿里云通用抠图，阿里云失败才使用本地 `editor_green_screen` 键色扣除；熔断期不得直接退化到本地兜底。角色动作视频生成的背景色已与生图链路统一：`screenColor=auto` 时由视觉 LLM（`gpt-5-mini`，Responses 协议、low 推理档）读源角色图自动决策，并经硬过滤器剔除与前景 / 皮肤撞色的候选，手动 hex 则尊重用户选择；透明源角色图在提交 Ark 图生视频前先合成到选定背景色实色，使视频背景等于抠图键色。抽帧后逐帧优先走阿里云通用抠图，失败时降级本地 `editor_green_screen` 键色兜底（按生成时选定的背景色，而非固定 `#00FF00`）。阿里云通用抠图配置为 `GENARRATIVE_ALIYUN_MATTING_ENABLED`、`GENARRATIVE_ALIYUN_MATTING_ENDPOINT`、`GENARRATIVE_ALIYUN_MATTING_ACCESS_KEY_ID`、`GENARRATIVE_ALIYUN_MATTING_ACCESS_KEY_SECRET` 和 `GENARRATIVE_ALIYUN_MATTING_REQUEST_TIMEOUT_MS`；未配置专用 AK/SK 时可复用 `ALIBABA_CLOUD_ACCESS_KEY_ID` / `ALIBABA_CLOUD_ACCESS_KEY_SECRET`，默认 endpoint 为 `imageseg.cn-shanghai.aliyuncs.com`。BgFilter 与阿里云抠图失败都写入 `external_api_call_failure` 审计。 -- BgFilter 处理已经持久化到私有 OSS 的带背景原图时，`api-server` 必须签发 600 秒 GET URL 并通过 multipart `image_url` 提交，不再下载对象或用 `file` 重传图片字节；签名 URL 不得写入日志、审计或持久化。只有 BgFilter 失败或熔断打开进入 fallback 后，才允许使用图片字节继续调用阿里云通用抠图或本地键色。 +- 生成后抠图以内存中的 `DownloadedImage` → 私有 OSS owned 上传作为带背景原图生命周期边界：上传消费字节所有权，完成后不保留原图缓冲。BgFilter 必须签发 600 秒 GET URL 并通过 multipart `image_url` 提交，不下载对象或用 `file` 重传；进入阿里云 fallback 时由 `platform-matting` URL 接口单独下载并上传 `AuthorizeFileUpload` 临时对象，在推理前释放下载缓冲；继续 fallback 到本地键色时再单独下载一次原图，本地产出后释放本次原图下载缓冲。签名 URL 不得写入日志、审计或持久化。 - 阿里云通用抠图的非上海地域输入不得使用 `viapiutils/GetOssStsToken`、固定 `viapi-customer-temp` 或 OSS V1 PUT。`platform-matting` 必须按官方新版 SDK Advance 协议调用 `AuthorizeFileUpload`，使用动态返回的单对象 Policy 执行 multipart POST，再把临时上海 OSS URL 交给 `SegmentCommonImage`；输入归一化、结果下载与原尺寸 Alpha 回贴继续留在同一适配器内。该协议仍上传图片字节，不等同于阿里云服务端直接抓取任意公网 URL，也不改变上层 BgFilter → 阿里云 → 本地降级顺序。 - Match3D 物品 sheet：关卡整图完成后走 VectorEngine `/v1/images/edits` multipart `image`，模型为 `gpt-image-2`，`2K 1:1` 输出 `10*10` spritesheet；物品 sheet prompt 固定要求单一纯绿色 `#00FF00 / RGB(0,255,0)` 绿幕背景，后端上传 OSS 前必须把绿幕扣成透明 PNG，并把透明整图写入 `itemSpritesheetImageSrc/itemSpritesheetImageObjectKey`。后端优先按透明 alpha 连通域从该 sheet 识别真实素材矩形并持久化 20 个物品、每个 5 个形态；识别数量不足时才回退 `10*10` 固定网格。通用系列素材图集的行列索引按每行 2 个物品计算，必须落在 `1..=10`，难度只决定运行态加载 3 / 9 / 15 / 20 种。 - Match3D UI spritesheet 和背景派生图：关卡整图作为参考图并发生成 `1K 1:1` UI spritesheet 与 `1K 9:16` 背景图，模型均为 `gpt-image-2`。UI spritesheet prompt 固定要求单一纯绿色 `#00FF00 / RGB(0,255,0)` 绿幕背景，后端上传 OSS 前必须把绿幕扣成透明 PNG；背景图必须合成为全画幅不透明 PNG。 diff --git a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md index 315174a3e..c4e646474 100644 --- a/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md +++ b/docs/【开发运维】本地开发验证与生产运维-2026-05-15.md @@ -67,7 +67,7 @@ lease 过期后不代表任务一定再次执行：claim transaction 只有在 ` 阿里云通用抠图的非上海地域输入使用 `AuthorizeFileUpload → Policy POST → SegmentCommonImage` 正式链路，上传 Bucket / Endpoint / ObjectKey 由阿里云动态返回；不得恢复 `GetOssStsToken`、固定 `viapi-customer-temp`、临时 AK/SK 或 OSS V1 PUT。该切换不新增环境变量；真实链路冒烟可运行 `cargo run -p platform-matting --example segment_smoke --manifest-path server-rs/Cargo.toml -- <图片路径>`，预期日志中的输入 host 为授权响应返回的上海 OSS host，并完成结果下载。图片字节仍经过执行任务的 api-server / worker，排障时不要把 Advance 路径误判为阿里云直接抓取任意公网 URL。 -BgFilter 对已经落入私有 OSS 的生成原图直接使用 600 秒签名 URL：`api-server` 的 multipart 只提交 `image_url`、`screen_color` 和 `seg_model`，不再提交 `file`，也不会在 BgFilter 调用前重新下载 OSS 对象。排障日志只应出现 object key 与签名有效期，不得记录带 `x-oss-*` 查询参数的完整 URL；BgFilter 失败或熔断打开后仍按既有顺序进入阿里云通用抠图和本地键色 fallback。 +BgFilter 对已经落入私有 OSS 的生成原图直接使用 600 秒签名 URL：`api-server` 的 multipart 只提交 `image_url`、`screen_color` 和 `seg_model`，不再提交 `file`，也不会在 BgFilter 调用前重新下载 OSS 对象。带背景原图上传完成后应已消费并释放字节所有权；BgFilter 失败或熔断打开后，阿里云 fallback 才单独下载源对象并上传动态临时桶，临时上传完成即释放本次下载缓冲；阿里云继续失败时本地 fallback 再独立下载，并在本地处理产出后释放本次原图缓冲。排障日志只应出现 object key 与签名有效期，不得记录带 `x-oss-*` 查询参数的完整 URL。这里的释放是 Rust 缓冲析构，不以操作系统 RSS 立即下降作为判据。 `我的` 页签或排障面板展示队列等待时，只读取 BFF 队列接口：`GET /api/runtime/external-generation/queue-overview` 查看当前用户可见队列概览，`GET /api/runtime/external-generation/jobs/{jobId}` 查看单 job 状态。生成页 / 进度页不承接队列概览，只展示当前玩法业务进度；队列接口只提供等待 / 运行 / 失败 / 完成状态补充，最终草稿、作品和结果页仍要轮询对应玩法 session/detail 接口收敛到 ready 或 failed；不要直接查询 `external_generation_job` private table，也不要把 worker 内部 payload 暴露到前端。 diff --git a/server-rs/crates/api-server/src/aliyun_matting.rs b/server-rs/crates/api-server/src/aliyun_matting.rs index 6094e6171..d60004f75 100644 --- a/server-rs/crates/api-server/src/aliyun_matting.rs +++ b/server-rs/crates/api-server/src/aliyun_matting.rs @@ -43,6 +43,39 @@ pub(crate) async fn segment_image_with_aliyun_matting( }) } +/// 私有 OSS 签名 URL → 延迟下载 → 阿里云临时桶 → 原尺寸透明 PNG。 +/// 下载和临时上传由 platform-matting 收口，源图缓冲不会跨越整次阿里云推理常驻。 +pub(crate) async fn segment_image_url_with_aliyun_matting( + state: &AppState, + image_url: &str, + log_label: &str, +) -> Result { + let matting_client = state + .matting_client() + .ok_or_else(aliyun_matting_unconfigured_error)?; + + let file_name = format!("{log_label}.png"); + let started_at = std::time::Instant::now(); + let output_bytes = matting_client + .segment_image_url_to_transparent_png(image_url, &file_name) + .await + .map_err(|error| { + aliyun_matting_failure_to_app_error(&error, started_at.elapsed().as_millis() as u64) + })?; + tracing::info!( + provider = "aliyun-matting", + log_label, + elapsed_ms = started_at.elapsed().as_millis() as u64, + "阿里云通用抠图 URL 输入完成" + ); + + Ok(DownloadedOpenAiImage { + bytes: output_bytes, + mime_type: "image/png".to_string(), + extension: "png".to_string(), + }) +} + /// 把 platform-matting 的错误映射成审计友好的 AppError。 /// /// 分类（是否外部调用、超时、传输层故障、上游 HTTP 状态）由 platform-matting 在错误发生处 @@ -99,6 +132,22 @@ mod tests { assert!(!crate::external_api_audit::matting_failure_external_call_attempted(&error)); } + #[test] + fn url_input_path_delegates_download_and_temp_upload_to_platform_matting() { + let source = include_str!("aliyun_matting.rs"); + let start = source + .find("pub(crate) async fn segment_image_url_with_aliyun_matting") + .expect("URL input adapter should exist"); + let tail = &source[start..]; + let end = tail + .find("/// 把 platform-matting 的错误映射") + .expect("URL input adapter should end before error mapper"); + let body = &tail[..end]; + + assert!(body.contains("segment_image_url_to_transparent_png")); + assert!(!body.contains("segment_image_to_transparent_png(image.bytes")); + } + #[test] fn local_preflight_failures_do_not_count_as_external_call() { for error in [ diff --git a/server-rs/crates/api-server/src/editor_project.rs b/server-rs/crates/api-server/src/editor_project.rs index 344193301..8cbedbf9e 100644 --- a/server-rs/crates/api-server/src/editor_project.rs +++ b/server-rs/crates/api-server/src/editor_project.rs @@ -126,7 +126,7 @@ const EDITOR_LEGACY_GREEN_SCREEN_SOURCE_ASSET_KIND: &str = "editor_green_screen_ const EDITOR_PROVIDER_SOURCE_SLOT: &str = "provider_source"; const EDITOR_BGFILTER_DEFAULT_SEG_MODEL: &str = "birefnet"; const EDITOR_BGFILTER_SEG_MODEL_ANIME_SEG: &str = "anime-seg"; -const EDITOR_BGFILTER_IMAGE_URL_EXPIRE_SECONDS: u64 = 600; +const EDITOR_MATTING_SOURCE_URL_EXPIRE_SECONDS: u64 = 600; const EDITOR_PUBLICATION_MATERIAL_ASSET_KIND: &str = "editor_publication_material"; const EDITOR_LEGACY_INLINE_IMAGE_ASSET_KIND: &str = "editor_legacy_inline_image"; @@ -1623,7 +1623,7 @@ pub(crate) async fn generate_editor_image_for_owner( state, caller.owner_user_id.as_str(), generated.task_id.as_str(), - &image, + image, submitted_prompt.as_str(), generated.actual_prompt.as_deref(), storage_profile.asset_kind, @@ -1670,7 +1670,6 @@ pub(crate) async fn generate_editor_image_for_owner( let removal = remove_editor_generated_screen_background_with_bgfilter( state, source_object_key.as_str(), - &image, screen_color.expect("character generation should have screen color"), seg_model.expect("character generation should have BgFilter seg model"), &matting_audit, @@ -2759,7 +2758,6 @@ struct EditorScreenBackgroundRemovalOutput { async fn remove_editor_generated_screen_background_with_bgfilter( state: &AppState, source_object_key: &str, - fallback_image: &DownloadedOpenAiImage, screen_color: EditorScreenBackgroundColor, seg_model: &str, audit: &crate::external_api_audit::ExternalApiAuditContext, @@ -2776,7 +2774,7 @@ async fn remove_editor_generated_screen_background_with_bgfilter( ); return fallback_editor_screen_background_removal( state, - fallback_image, + source_object_key, screen_color, audit, ) @@ -2825,7 +2823,7 @@ async fn remove_editor_generated_screen_background_with_bgfilter( .await; fallback_editor_screen_background_removal( state, - fallback_image, + source_object_key, screen_color, audit, ) @@ -2838,17 +2836,26 @@ async fn remove_editor_generated_screen_background_with_bgfilter( /// 熔断打开路径与 BgFilter 调用失败路径共用此链，确保 cooldown 内仍是「阿里云 → 本地」而非直接本地。 async fn fallback_editor_screen_background_removal( state: &AppState, - image: &DownloadedOpenAiImage, + source_object_key: &str, screen_color: EditorScreenBackgroundColor, audit: &crate::external_api_audit::ExternalApiAuditContext, ) -> Result { - match crate::aliyun_matting::segment_image_with_aliyun_matting( + let aliyun_result = match sign_editor_private_object_read_url( state, - image, - "editor-screen-background", - ) - .await - { + source_object_key, + EDITOR_MATTING_SOURCE_URL_EXPIRE_SECONDS, + ) { + Ok(source_url) => { + crate::aliyun_matting::segment_image_url_with_aliyun_matting( + state, + source_url.as_str(), + "editor-screen-background", + ) + .await + } + Err(error) => Err(error), + }; + match aliyun_result { Ok(image) => Ok(EditorScreenBackgroundRemovalOutput { image, provider: "Aliyun Matting", @@ -2878,12 +2885,13 @@ async fn fallback_editor_screen_background_removal( ) .await; } - remove_editor_generated_green_screen_background(image, screen_color).map(|image| { - EditorScreenBackgroundRemovalOutput { - image, - provider: "Genarrative Local", - model: "screen-color-keying".to_string(), - } + let source_image = download_editor_persisted_image_object(state, source_object_key).await?; + let output = remove_editor_generated_green_screen_background(&source_image, screen_color); + drop(source_image); + output.map(|image| EditorScreenBackgroundRemovalOutput { + image, + provider: "Genarrative Local", + model: "screen-color-keying".to_string(), }) } } @@ -2947,19 +2955,11 @@ async fn request_editor_generated_screen_background_with_bgfilter( seg_model: &str, ) -> Result { let url = editor_bgfilter_endpoint(state)?; - let oss_client = state.oss_client().ok_or_else(|| { - AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({ - "provider": "aliyun-oss", - "reason": "OSS 未完成环境变量配置", - })) - })?; - let signed = oss_client - .sign_get_object_url(OssSignedGetObjectUrlRequest { - object_key: source_object_key.to_string(), - expire_seconds: Some(EDITOR_BGFILTER_IMAGE_URL_EXPIRE_SECONDS), - }) - .map_err(|error| map_oss_error(error, "aliyun-oss"))?; - let signed_image_url = signed.signed_url; + let signed_image_url = sign_editor_private_object_read_url( + state, + source_object_key, + EDITOR_MATTING_SOURCE_URL_EXPIRE_SECONDS, + )?; let call_id = format!("bgfilter-call-{}", current_utc_micros()); let request_started_at = Instant::now(); let timeout_ms = state.config.editor_bgfilter_request_timeout_ms.max(1); @@ -2967,7 +2967,7 @@ async fn request_editor_generated_screen_background_with_bgfilter( %call_id, upstream_url = %url, source_object_key, - image_url_expire_seconds = EDITOR_BGFILTER_IMAGE_URL_EXPIRE_SECONDS, + image_url_expire_seconds = EDITOR_MATTING_SOURCE_URL_EXPIRE_SECONDS, screen_color = screen_color.hex, seg_model, timeout_ms, @@ -3257,6 +3257,26 @@ fn editor_bgfilter_endpoint(state: &AppState) -> Result { )) } +fn sign_editor_private_object_read_url( + state: &AppState, + object_key: &str, + expire_seconds: u64, +) -> Result { + let oss_client = state.oss_client().ok_or_else(|| { + AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({ + "provider": "aliyun-oss", + "reason": "OSS 未完成环境变量配置", + })) + })?; + oss_client + .sign_get_object_url(OssSignedGetObjectUrlRequest { + object_key: object_key.to_string(), + expire_seconds: Some(expire_seconds), + }) + .map(|signed| signed.signed_url) + .map_err(|error| map_oss_error(error, "aliyun-oss")) +} + fn sanitize_editor_bgfilter_upstream_message(message: &str, signed_image_url: &str) -> String { let sanitized = message.replace(signed_image_url, "[signed OSS URL redacted]"); if sanitized.to_ascii_lowercase().contains("x-oss-") { @@ -3714,7 +3734,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner( state, caller.owner_user_id.as_str(), generated.task_id.as_str(), - &image, + image, prompt.as_str(), generated.actual_prompt.as_deref(), EDITOR_ICON_SPRITESHEET_ASSET_KIND, @@ -3755,7 +3775,6 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner( let removal = remove_editor_generated_screen_background_with_bgfilter( state, source_object_key.as_str(), - &image, screen_color, seg_model, &matting_audit, @@ -3891,7 +3910,7 @@ pub(crate) async fn generate_editor_icon_spritesheet_for_owner( }), ) } - }; + }; let (canvas_items, primary_layer_id) = if let Some(completion) = payload.canvas_completion.as_ref() { build_icon_spritesheet_canvas_layer_items( @@ -4325,7 +4344,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner( state, caller.owner_user_id.as_str(), generated.task_id.as_str(), - &image, + image, prompt.as_str(), generated.actual_prompt.as_deref(), EDITOR_UI_DESIGN_SPRITESHEET_ASSET_KIND, @@ -4366,7 +4385,6 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner( let removal = remove_editor_generated_screen_background_with_bgfilter( state, source_object_key.as_str(), - &image, screen_color, seg_model, &matting_audit, @@ -4496,7 +4514,7 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner( }), ) } - }; + }; let (canvas_items, primary_layer_id) = if let Some(completion) = payload.canvas_completion.as_ref() { build_icon_spritesheet_canvas_layer_items( @@ -6359,6 +6377,71 @@ pub(crate) async fn persist_editor_generated_image( file_stem: &str, slot: &str, provider: &str, +) -> Result { + persist_editor_generated_image_data( + state, + owner_user_id, + task_id, + GeneratedImageAssetDataUrl { + format: normalize_generated_image_asset_mime(image.mime_type.as_str()), + bytes: image.bytes.clone(), + }, + prompt, + actual_prompt, + asset_kind, + path_kind, + file_stem, + slot, + provider, + ) + .await +} + +async fn persist_editor_generated_image_owned( + state: &AppState, + owner_user_id: &str, + task_id: &str, + image: DownloadedOpenAiImage, + prompt: &str, + actual_prompt: Option<&str>, + asset_kind: &str, + path_kind: &str, + file_stem: &str, + slot: &str, + provider: &str, +) -> Result { + let image_data = GeneratedImageAssetDataUrl { + format: normalize_generated_image_asset_mime(image.mime_type.as_str()), + bytes: image.bytes, + }; + persist_editor_generated_image_data( + state, + owner_user_id, + task_id, + image_data, + prompt, + actual_prompt, + asset_kind, + path_kind, + file_stem, + slot, + provider, + ) + .await +} + +async fn persist_editor_generated_image_data( + state: &AppState, + owner_user_id: &str, + task_id: &str, + image: GeneratedImageAssetDataUrl, + prompt: &str, + actual_prompt: Option<&str>, + asset_kind: &str, + path_kind: &str, + file_stem: &str, + slot: &str, + provider: &str, ) -> Result { let oss_client = state.oss_client().ok_or_else(|| { AppError::from_status(StatusCode::SERVICE_UNAVAILABLE).with_details(json!({ @@ -6375,10 +6458,7 @@ pub(crate) async fn persist_editor_generated_image( sanitize_editor_storage_segment(task_id, "task"), ], file_stem: sanitize_editor_storage_segment(file_stem, "image"), - image: GeneratedImageAssetDataUrl { - format: normalize_generated_image_asset_mime(image.mime_type.as_str()), - bytes: image.bytes.clone(), - }, + image, access: OssObjectAccess::Private, metadata: GeneratedImageAssetAdapterMetadata { asset_kind: Some(asset_kind.to_string()), @@ -6454,14 +6534,14 @@ async fn persist_editor_provider_source_image( state: &AppState, owner_user_id: &str, task_id: &str, - image: &DownloadedOpenAiImage, + image: DownloadedOpenAiImage, prompt: &str, actual_prompt: Option<&str>, asset_kind: &str, path_kind: &str, file_stem: &str, ) -> Result { - persist_editor_generated_image( + persist_editor_generated_image_owned( state, owner_user_id, task_id, @@ -6808,7 +6888,7 @@ async fn read_editor_reference_image_object( expire_seconds: Some(EDITOR_REFERENCE_IMAGE_READ_EXPIRE_SECONDS), }) .map_err(|error| map_oss_error(error, "aliyun-oss"))?; - let response = reqwest::Client::new() + let mut response = reqwest::Client::new() .get(signed.signed_url.as_str()) .send() .await @@ -6840,12 +6920,20 @@ async fn read_editor_reference_image_object( .and_then(|value| value.to_str().ok()) .and_then(normalize_editor_reference_image_mime_type) .map(ToOwned::to_owned); - let bytes = response.bytes().await.map_err(|error| { + let mut bytes = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(|error| { AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({ "provider": "aliyun-oss", "message": format!("读取图片参考图内容失败：{error}"), })) - })?; + })? { + if bytes.len().saturating_add(chunk.len()) as u64 + > EDITOR_REFERENCE_IMAGE_MAX_SIZE_BYTES + { + return Err(editor_reference_image_too_large()); + } + bytes.extend_from_slice(chunk.as_ref()); + } if bytes.is_empty() { return Err( AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ @@ -6855,11 +6943,8 @@ async fn read_editor_reference_image_object( })), ); } - if bytes.len() as u64 > EDITOR_REFERENCE_IMAGE_MAX_SIZE_BYTES { - return Err(editor_reference_image_too_large()); - } let mime_type = content_type_mime - .or_else(|| infer_editor_reference_image_mime_type(bytes.as_ref()).map(ToOwned::to_owned)) + .or_else(|| infer_editor_reference_image_mime_type(bytes.as_slice()).map(ToOwned::to_owned)) .ok_or_else(|| { AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({ "provider": "editor-reference-image", @@ -6871,12 +6956,24 @@ async fn read_editor_reference_image_object( .to_string(); let extension = editor_reference_image_extension(mime_type.as_str()); Ok(OpenAiReferenceImage { - bytes: bytes.to_vec(), + bytes, mime_type, file_name: format!("editor-reference.{extension}"), }) } +async fn download_editor_persisted_image_object( + state: &AppState, + object_key: &str, +) -> Result { + let image = read_editor_reference_image_object(state, object_key).await?; + Ok(DownloadedOpenAiImage { + extension: editor_reference_image_extension(image.mime_type.as_str()).to_string(), + mime_type: image.mime_type, + bytes: image.bytes, + }) +} + fn normalize_editor_reference_image_mime_type(content_type: &str) -> Option<&str> { let mime_type = content_type.split(';').next()?.trim(); mime_type.starts_with("image/").then_some(mime_type) @@ -9445,8 +9542,10 @@ mod tests { "async fn fallback_editor_screen_background_removal", "async fn request_editor_generated_screen_background_with_bgfilter", &[ - "segment_image_with_aliyun_matting", + "segment_image_url_with_aliyun_matting", + "download_editor_persisted_image_object", "remove_editor_generated_green_screen_background", + "drop(source_image)", ], ); assert_function_contains( @@ -9455,8 +9554,8 @@ mod tests { "async fn request_editor_background_removal_image", &[ "editor_bgfilter_endpoint", - "sign_get_object_url", - "EDITOR_BGFILTER_IMAGE_URL_EXPIRE_SECONDS", + "sign_editor_private_object_read_url", + "EDITOR_MATTING_SOURCE_URL_EXPIRE_SECONDS", "\"image_url\"", "editor_bgfilter_request_timeout_ms.max(1)", "\"screen_color\"", @@ -9541,6 +9640,95 @@ mod tests { ); } + #[test] + fn editor_matting_releases_source_buffers_at_oss_boundaries() { + let source = include_str!("editor_project.rs"); + for (start, end) in [ + ( + "pub(crate) async fn generate_editor_image_for_owner", + "fn normalize_editor_image_generation_size", + ), + ( + "pub(crate) async fn generate_editor_icon_spritesheet_for_owner", + "pub async fn extract_editor_ui_design_assets", + ), + ( + "pub(crate) async fn extract_editor_ui_design_assets_for_owner", + "pub(crate) fn editor_project_payload_from_record", + ), + ] { + assert_function_contains_in_order( + source, + start, + end, + &[ + "persist_editor_provider_source_image", + "remove_editor_generated_screen_background_with_bgfilter", + "persist_editor_generated_image", + ], + ); + } + assert_function_contains( + source, + "async fn persist_editor_provider_source_image", + "async fn persist_editor_provider_source_resource", + &["persist_editor_generated_image_owned"], + ); + for (start, end) in [ + ( + "pub(crate) async fn generate_editor_icon_spritesheet_for_owner", + "pub async fn extract_editor_ui_design_assets", + ), + ( + "pub(crate) async fn extract_editor_ui_design_assets_for_owner", + "pub(crate) fn editor_project_payload_from_record", + ), + ] { + assert_function_contains( + source, + start, + end, + &["bytes: image.bytes.clone()", "slice_source"], + ); + assert_function_not_contains( + source, + start, + end, + &["download_editor_persisted_image_object", "drop(slice_source)"], + ); + } + assert_function_not_contains( + source, + "async fn remove_editor_generated_screen_background_with_bgfilter", + "async fn fallback_editor_screen_background_removal", + &["fallback_image", "DownloadedOpenAiImage"], + ); + assert_function_contains( + source, + "async fn persist_editor_generated_image_owned", + "async fn persist_editor_generated_image_data", + &["bytes: image.bytes"], + ); + assert_function_not_contains( + source, + "async fn persist_editor_generated_image_owned", + "async fn persist_editor_generated_image_data", + &["image.bytes.clone()"], + ); + assert_function_contains( + source, + "async fn read_editor_reference_image_object", + "async fn download_editor_persisted_image_object", + &["response.chunk().await", "bytes.extend_from_slice"], + ); + assert_function_not_contains( + source, + "async fn read_editor_reference_image_object", + "async fn download_editor_persisted_image_object", + &["response.bytes().await", "bytes.to_vec()"], + ); + } + #[test] fn editor_paid_image_postprocess_keeps_provider_outputs_recoverable() { let source = include_str!("editor_project.rs"); diff --git a/server-rs/crates/platform-matting/src/lib.rs b/server-rs/crates/platform-matting/src/lib.rs index 1cf2c7312..70f8f9728 100644 --- a/server-rs/crates/platform-matting/src/lib.rs +++ b/server-rs/crates/platform-matting/src/lib.rs @@ -407,16 +407,9 @@ impl MattingClient { let image_url = self .upload_temp_image(upload_bytes, file_name, "image/png") .await?; - - let result = self - .segment_common_image(SegmentCommonImageRequest { - image_url, - // 默认 ReturnForm：原尺寸 + 透明背景，无需本地合成。 - return_form: None, - }) + let result_image = self + .segment_uploaded_input_to_rgba(image_url, (source_width, source_height), upload_dims) .await?; - let result_bytes = self.download_result_image(&result.image_url).await?; - let result_image = decode_result_image(&result_bytes)?.to_rgba8(); if !downscaled { if result_image.dimensions() != (source_width, source_height) { @@ -434,18 +427,140 @@ impl MattingClient { return encode_rgba_png(&result_image); } - // 缩图送抠的场景：只取结果 alpha，上采样回原尺寸后贴回原图 RGB。 - let alpha_mask = image::DynamicImage::ImageRgba8(result_image).resize_exact( - source_width, - source_height, - image::imageops::FilterType::Triangle, - ); - let alpha_mask = alpha_mask.to_rgba8(); - let mut output = source_rgba; - for (target, mask) in output.pixels_mut().zip(alpha_mask.pixels()) { - target.0[3] = target.0[3].min(mask.0[3]); + compose_source_with_result_alpha(source_rgba, result_image) + } + + /// 私有 OSS 签名 URL → 下载 → AuthorizeFileUpload 临时对象 → 通用抠图 → 透明 PNG。 + /// + /// 源图下载缓冲和解码图只活到临时对象上传完成；若输入发生过降尺寸，结果返回后再单独下载 + /// 一次源图完成 Alpha 回贴，避免在整个阿里云推理期间常驻原图内存。 + pub async fn segment_image_url_to_transparent_png( + &self, + source_url: &str, + file_name: &str, + ) -> Result, MattingError> { + let source_url = source_url.trim(); + if source_url.is_empty() { + return Err(MattingError::InvalidRequest( + "待抠图源图 URL 不能为空".to_string(), + )); } - encode_rgba_png(&output) + + let (source_dims, upload_dims, image_url) = { + let source_bytes = self.download_source_image(source_url).await?; + let source = decode_source_image(&source_bytes)?; + let source_dims = (source.width(), source.height()); + validate_source_dimensions(source_dims)?; + let (upload_bytes, upload_dims) = normalize_matting_input_png(&source)?; + let image_url = self + .upload_temp_image(upload_bytes, file_name, "image/png") + .await?; + (source_dims, upload_dims, image_url) + }; + + let result_image = self + .segment_uploaded_input_to_rgba(image_url, source_dims, upload_dims) + .await?; + if upload_dims == source_dims { + return encode_rgba_png(&result_image); + } + + // 只有降尺寸送抠时才重新下载源图恢复原始 RGB；第一次下载缓冲已在上传临时对象后释放。 + let source_rgba = { + let source_bytes = self.download_source_image(source_url).await?; + let source = decode_source_image(&source_bytes)?; + if (source.width(), source.height()) != source_dims { + return Err(MattingError::upstream_response_error( + "待抠图源图在处理期间尺寸发生变化".to_string(), + None, + )); + } + source.to_rgba8() + }; + compose_source_with_result_alpha(source_rgba, result_image) + } + + async fn segment_uploaded_input_to_rgba( + &self, + image_url: String, + source_dims: (u32, u32), + upload_dims: (u32, u32), + ) -> Result { + let result = self + .segment_common_image(SegmentCommonImageRequest { + image_url, + // 默认 ReturnForm：原尺寸 + 透明背景，无需本地合成。 + return_form: None, + }) + .await?; + let result_image = { + let result_bytes = self.download_result_image(&result.image_url).await?; + decode_result_image(&result_bytes)?.to_rgba8() + }; + if upload_dims == source_dims && result_image.dimensions() != source_dims { + return Err(MattingError::upstream_response_error( + format!( + "抠图结果尺寸 {}x{} 与输入 {}x{} 不一致", + result_image.width(), + result_image.height(), + source_dims.0, + source_dims.1 + ), + None, + )); + } + Ok(result_image) + } + + async fn download_source_image(&self, url: &str) -> Result, MattingError> { + let mut response = self.client.get(url).send().await.map_err(|error| { + MattingError::upstream_transport_error( + format!( + "下载待抠图源图失败（transport={}, timeout={}, connect={}）", + classify_reqwest_error(&error), + error.is_timeout(), + error.is_connect() + ), + error.is_timeout(), + ) + })?; + let status = response.status(); + if !status.is_success() { + return Err(MattingError::upstream_http_error( + format!("下载待抠图源图失败（HTTP {}）", status.as_u16()), + status.as_u16(), + )); + } + if response + .content_length() + .is_some_and(|length| length > MAX_RESULT_RESPONSE_BYTES as u64) + { + return Err(source_response_too_large_error()); + } + let mut bytes = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(|error| { + MattingError::upstream_transport_error( + format!( + "读取待抠图源图失败（transport={}, timeout={}, connect={}）", + classify_reqwest_error(&error), + error.is_timeout(), + error.is_connect() + ), + error.is_timeout(), + ) + })? { + if bytes.len().saturating_add(chunk.len()) > MAX_RESULT_RESPONSE_BYTES { + return Err(source_response_too_large_error()); + } + bytes.extend_from_slice(chunk.as_ref()); + } + if bytes.is_empty() { + return Err(MattingError::upstream_response_error( + "待抠图源图为空".to_string(), + Some(status.as_u16()), + )); + } + Ok(bytes) } async fn download_result_image(&self, url: &str) -> Result, MattingError> { @@ -663,6 +778,33 @@ fn normalize_matting_input_png( normalize_matting_input_png_within(source, MAX_INPUT_BYTES) } +fn validate_source_dimensions(source_dims: (u32, u32)) -> Result<(), MattingError> { + if source_dims.0 <= MIN_INPUT_EDGE || source_dims.1 <= MIN_INPUT_EDGE { + return Err(MattingError::InvalidRequest(format!( + "待抠图图片尺寸 {}x{} 过小，阿里云通用抠图要求每条边大于 {MIN_INPUT_EDGE} 像素", + source_dims.0, source_dims.1 + ))); + } + Ok(()) +} + +fn compose_source_with_result_alpha( + mut source_rgba: image::RgbaImage, + result_image: image::RgbaImage, +) -> Result, MattingError> { + let (source_width, source_height) = source_rgba.dimensions(); + let alpha_mask = image::DynamicImage::ImageRgba8(result_image).resize_exact( + source_width, + source_height, + image::imageops::FilterType::Triangle, + ); + let alpha_mask = alpha_mask.to_rgba8(); + for (target, mask) in source_rgba.pixels_mut().zip(alpha_mask.pixels()) { + target.0[3] = target.0[3].min(mask.0[3]); + } + encode_rgba_png(&source_rgba) +} + /// `normalize_matting_input_png` 的可注入体积上限版本，便于单测用小阈值触发降尺寸循环。 fn normalize_matting_input_png_within( source: &image::DynamicImage, @@ -720,6 +862,13 @@ fn result_response_too_large_error() -> MattingError { ) } +fn source_response_too_large_error() -> MattingError { + MattingError::upstream_response_error( + format!("待抠图源图响应过大，超过 {MAX_RESULT_RESPONSE_BYTES} 字节上限"), + None, + ) +} + /// 解码待抠图源图时套上尺寸 / 分配上限。源图直接来自请求体 / 生成产物，缺省 /// `image::Limits` 无尺寸上限、仅 512MiB alloc 兜底，12MB 请求体即可构造巨幅压缩图 /// （解压炸弹）在缩图前撑爆内存；逐帧动画并发调用时风险叠加。 @@ -1311,4 +1460,47 @@ mod tests { "降尺寸后仍是 PNG" ); } + + #[test] + fn url_input_upload_scope_ends_before_aliyun_inference() { + let source = include_str!("lib.rs"); + let start = source + .find("pub async fn segment_image_url_to_transparent_png") + .expect("URL input method should exist"); + let tail = &source[start..]; + let end = tail + .find("async fn segment_uploaded_input_to_rgba") + .expect("uploaded input helper should follow URL method"); + let body = &tail[..end]; + + let download = body + .find("let source_bytes = self.download_source_image(source_url).await?") + .expect("source should download lazily"); + let upload = body + .find(".upload_temp_image(upload_bytes, file_name, \"image/png\")") + .expect("source should upload to temporary OSS"); + let scope_end = body + .find("let result_image = self") + .expect("Aliyun inference should start after upload scope"); + assert!(download < upload && upload < scope_end); + assert!(body.contains("let (source_dims, upload_dims, image_url) = {")); + assert_eq!(body.matches("download_source_image(source_url)").count(), 2); + } + + #[test] + fn compose_source_with_result_alpha_preserves_rgb() { + let source = image::RgbaImage::from_pixel(2, 2, image::Rgba([10, 20, 30, 180])); + let mask = image::RgbaImage::from_pixel(1, 1, image::Rgba([200, 210, 220, 90])); + + let encoded = compose_source_with_result_alpha(source, mask) + .expect("alpha composition should encode"); + let output = image::load_from_memory(&encoded) + .expect("output should decode") + .to_rgba8(); + + assert_eq!(output.dimensions(), (2, 2)); + assert!(output + .pixels() + .all(|pixel| pixel.0 == [10, 20, 30, 90])); + } }