修复图片改造尺寸保持

保持图片改造提交与画布回填的原图尺寸
在 VectorEngine 边界对齐非 16 倍数参考图并恢复目标尺寸
补充前后端回归测试与编辑器尺寸契约文档
This commit is contained in:
2026-07-11 18:10:00 +08:00
parent d756e8e04b
commit 300e69c97d
9 changed files with 243 additions and 54 deletions
@@ -368,6 +368,14 @@
- 验证:`npm run test -- src/services/image-editor/editorProjectClient.test.ts src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx -- --runInBand`;后端验证至少覆盖 `editor_image_edit_request_omits_price_mud_points``editor_image_edit_can_complete_by_replacing_target_layer`
- 关联:`src/services/image-editor/editorProjectClient.ts``src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts``server-rs/crates/api-server/src/editor_project.rs`
## 图片画布快速编辑尺寸要区分业务目标和 provider 对齐尺寸
- 现象:原图经过快速编辑后 Resolution 变成近似比例的 1K / 2K 预设;原图或框选标记图宽高不是 16 的倍数时,VectorEngine edits 直接拒绝请求。
- 原因:前端已有源图精确 `originalWidth/originalHeight`,提交时却按最近常用比例和 K 档重新计算 `size`;后端又把非 16 倍数的目标尺寸和原始参考图字节直接放进 multipart,并以 provider 回图宽高落库和覆盖画布图层。
- 处理:画布快速编辑提交源图精确尺寸作为业务目标;api-server 只在 provider 边界向右、向下复制边缘像素,把每张参考图和目标尺寸临时补齐到 16 的倍数,收到回图后裁回业务目标尺寸再持久化。若上游异常返回其他尺寸,先按目标比例裁切缩放;临时对齐尺寸不能进入 OSS 元数据、`editor_project_resource``editor_asset` 或画布 Resolution。前端 inline 回填也保留源图显示尺寸和 Resolution,避免旧回包再次放大图层。
- 验证:`npm run test -- src/components/image-editor/ImageCanvasGenerationSubmissionModel.test.ts src/components/image-editor/ImageCanvasGenerationLayerModel.test.ts src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.test.tsx` 覆盖 `1537x1025` 精确提交和源图回填;`cargo test -p api-server editor_image_edit_aligns_provider_images_and_restores_source_dimensions --manifest-path server-rs/Cargo.toml` 覆盖 provider `1552x1040`、额外参考图独立对齐和回图恢复。
- 关联:`src/components/image-editor/ImageCanvasGenerationSubmissionModel.ts``src/components/image-editor/useImageCanvasGenerationSubmissionWorkflow.ts``src/components/image-editor/ImageCanvasGenerationLayerModel.ts``server-rs/crates/api-server/src/editor_project.rs`
## 图片画布快速编辑元数据必须记录原图引用
- 现象:快速编辑生成的新图可以替换画布,但打开图片信息时“生成输入”里看不到被修改的原图。
@@ -87,7 +87,7 @@
- `POST /api/editor/images/background-removals`:接收当前图片源,校验登录态后由 api-server 解析为图片文件并转发到 BiRefNet 去背景服务;请求可携带 `projectId``targetLayerId``assetFolderId``assetLabel``sourceResourceId``canvasCompletion`,有 `canvasCompletion` 时完成后按生成占位写入结果图层,否则沿用旧的目标图层替换路径;响应返回 `imageSrc``objectKey``assetObjectId``width``height``taskId``elapsedMs``provider` 和可选 `project` 快照。服务地址由 `GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_BASE_URL` 配置,令牌只在服务端通过 `GENARRATIVE_EDITOR_BACKGROUND_REMOVAL_TOKEN` 注入。
- `POST /api/editor/icon-spritesheets/generations`:按图标规范图和素材描述数组生成 spritesheet,生成成功后 api-server 先保存带纯色背景 spritesheet 源图,再调用 BgFilter 生成透明 spritesheet。请求支持 `model``screenColor``segModel``aspectRatio``imageSize``priceMudPoints``projectId``assetFolderId``generationInputs``priceMudPoints` 必须来自编辑器生成计费配置中对应生图模型的尺寸档位(如 `nanobanana2``0.5K / 1K / 2K``gpt-image-2``1K / 2K`),后端用 `editor_generation_config` 校验后才调用上游;`nanobanana2` 走原生 `generateContent` 并写入 `generationConfig.imageConfig.aspectRatio/imageSize``0.5K``"512"``gpt-image-2``/v1/images/edits`。后端保存透明 spritesheet project resource / 账号素材,并随响应返回对应快照。
- `POST /api/editor/ui-designs/assets/extractions`:前端把红色框选轮廓绘入本地临时图后,先将该图上传 OSS 并确认 asset object,再以返回的 `objectKey` 作为参考图入队;Data URL / Blob URL 只允许停留在上传前的浏览器临时态。接口固定 `gpt-image-2` 和自动决策纯色背景素材提取提示词生成素材 spritesheet,生成成功后 api-server 先保存带纯色背景 spritesheet 源图,再调用 BgFilter 生成透明 spritesheet,并按连通域自动拆分为 `素材 1..N`,返回结构复用图标 spritesheet 响应。请求必须携带 `screenColor``segModel``aspectRatio: "1:1"``imageSize: "1K" | "2K"``priceMudPoints`;框选数量不超过 6 个时前端按 `1:1·1K` 与 gpt-image-2 1K 价格提交,超过 6 个时按 `1:1·2K` 与 2K 价格提交。后端必须在调用上游前校验比例、尺寸和泥点价格,只允许 `1:1 / 1K / 2K`。请求可携带 `projectId``assetFolderId``generationInputs``spritesheetLabel`,后端保存 spritesheet / 拆分素材并返回对应 resource / asset 快照;前端必须把 spritesheet 原图与拆分素材都加入画布。
- `POST /api/editor/images/edits`:按提示词和当前图片的已登记 `objectKey` / `resourceId` 调用 VectorEngine edits,返回新的生成图片元数据;图片快速编辑当前只提交 `sourceImageSrc`,不提交隐藏的 `referenceImageSrcs`。本地红框标记图必须先上传再提交 objectKey;请求携带 project / asset 上下文时由后端创建新 resource / asset,前端只消费响应快照。
- `POST /api/editor/images/edits`:按提示词和当前图片的已登记 `objectKey` / `resourceId` 调用 VectorEngine edits,返回新的生成图片元数据;图片快速编辑当前只提交 `sourceImageSrc`,不提交隐藏的 `referenceImageSrcs`画布快速编辑必须把源图精确 `originalWidth x originalHeight` 作为业务目标 `size` 提交,不能重新映射为近似比例或 1K / 2K 预设;api-server 在 VectorEngine provider 边界把目标尺寸和所有 multipart 参考图临时补齐到 16 的倍数,回图后恢复到业务目标精确尺寸,再落 OSS、project resource、账号素材和画布快照。16 对齐尺寸不得泄漏到响应、持久化资源或图层 Resolution。本地红框标记图必须先上传再提交 objectKey;请求携带 project / asset 上下文时由后端创建新 resource / asset,前端只消费响应快照。
- `POST /api/editor/videos/generations`:按视频描述、模型、比例、时长、分辨率、模式、声音、默认联网搜索标记和泥点价格生成视频。前端可选模型为 `seedance2.0-fast``seedance2.0``kling3.0``kling3.0-omni`,默认 `seedance2.0-fast`;后端必须将 `seedance2.0-fast` 映射到 `doubao-seedance-2-0-fast-260128`,将 `seedance2.0` 映射到 `doubao-seedance-2-0-260128`,两者不得混用。后端允许 6 类比例、4 到 15 秒整数、`480p / 720p / 1080p`,并拒绝 `seedance2.0-fast + 1080p``sound=on/off` 映射 Ark `generate_audio=true/false`。后端复用 Ark / VectorEngine content generation task 轮询链路,下载最终视频并持久化到 OSS;请求携带 `projectId` / `assetFolderId` 时同步创建 project resource / 账号素材并返回 `project` / `asset` 快照,基础响应返回 `videoSrc`、尺寸、prompt、model、provider、taskId、durationSeconds、resolution 和 `priceMudPoints`
- `POST /api/editor/audios/sound-effects/generations``POST /api/editor/audios/background-music/generations`:按音效 / 背景音乐参数生成音频并持久化到 OSS;请求携带 `projectId` / `assetFolderId` 时同步创建 project resource / 账号素材并返回 `project` / `resource` / `asset` 快照,基础响应返回 `audioSrc`、prompt、model、provider、taskId、duration、歌词和 `priceMudPoints`
@@ -1910,6 +1910,148 @@ fn editor_generation_provider_image_size_for_model(
image_size
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct EditorImageEditDimensions {
target_width: u32,
target_height: u32,
provider_width: u32,
provider_height: u32,
}
fn align_editor_image_edit_dimension(value: u32) -> u32 {
value.saturating_add(15) / 16 * 16
}
fn encode_editor_image_edit_png(
image: image::DynamicImage,
failure_status: StatusCode,
failure_message: &str,
) -> Result<Vec<u8>, AppError> {
let mut bytes = Cursor::new(Vec::new());
image
.write_to(&mut bytes, image::ImageFormat::Png)
.map_err(|error| {
AppError::from_status(failure_status).with_details(json!({
"provider": "editor-image-edit",
"message": format!("{failure_message}{error}"),
}))
})?;
Ok(bytes.into_inner())
}
fn prepare_editor_image_edit_references(
reference_images: &mut [OpenAiReferenceImage],
target_size: &str,
) -> Result<EditorImageEditDimensions, AppError> {
let (target_width, target_height) = target_size.split_once('x').ok_or_else(|| {
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
"provider": "editor-image-edit",
"message": "图片改造目标尺寸无效",
}))
})?;
let target_width = target_width.parse::<u32>().map_err(|_| {
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
"provider": "editor-image-edit",
"message": "图片改造目标宽度无效",
}))
})?;
let target_height = target_height.parse::<u32>().map_err(|_| {
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
"provider": "editor-image-edit",
"message": "图片改造目标高度无效",
}))
})?;
let provider_width = align_editor_image_edit_dimension(target_width);
let provider_height = align_editor_image_edit_dimension(target_height);
if reference_images.is_empty() {
return Err(
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
"provider": "editor-image-edit",
"message": "图片改造缺少原图",
})),
);
}
for (index, reference) in reference_images.iter_mut().enumerate() {
let decoded = image::load_from_memory(reference.bytes.as_slice()).map_err(|error| {
AppError::from_status(StatusCode::BAD_REQUEST).with_details(json!({
"provider": "editor-image-edit",
"message": format!("图片改造参考图不是有效图片:{error}"),
"referenceIndex": index,
}))
})?;
let width = decoded.width();
let height = decoded.height();
let reference_provider_width = align_editor_image_edit_dimension(width);
let reference_provider_height = align_editor_image_edit_dimension(height);
if reference_provider_width == width && reference_provider_height == height {
continue;
}
// 中文注释:只在 provider 边界向右、向下复制边缘像素补齐到 16 倍数,
// 不缩放、不裁切用户原图;生成结果持久化前会恢复到业务目标精确尺寸。
let source = decoded.to_rgba8();
let mut aligned =
image::RgbaImage::new(reference_provider_width, reference_provider_height);
for y in 0..reference_provider_height {
let source_y = y.min(height.saturating_sub(1));
for x in 0..reference_provider_width {
let source_x = x.min(width.saturating_sub(1));
aligned.put_pixel(x, y, *source.get_pixel(source_x, source_y));
}
}
reference.bytes = encode_editor_image_edit_png(
image::DynamicImage::ImageRgba8(aligned),
StatusCode::BAD_REQUEST,
"图片改造参考图 16 对齐失败",
)?;
reference.mime_type = "image/png".to_string();
reference.file_name = format!("editor-image-edit-reference-{}.png", index + 1);
}
Ok(EditorImageEditDimensions {
target_width,
target_height,
provider_width,
provider_height,
})
}
fn restore_editor_image_edit_output_dimensions(
output: DownloadedOpenAiImage,
dimensions: &EditorImageEditDimensions,
) -> Result<DownloadedOpenAiImage, AppError> {
let decoded = image::load_from_memory(output.bytes.as_slice()).map_err(|error| {
AppError::from_status(StatusCode::BAD_GATEWAY).with_details(json!({
"provider": "vector-engine",
"message": format!("图片改造结果不是有效图片:{error}"),
}))
})?;
if decoded.width() == dimensions.target_width && decoded.height() == dimensions.target_height {
return Ok(output);
}
let restored = if decoded.width() == dimensions.provider_width
&& decoded.height() == dimensions.provider_height
{
decoded.crop_imm(0, 0, dimensions.target_width, dimensions.target_height)
} else {
decoded.resize_to_fill(
dimensions.target_width,
dimensions.target_height,
image::imageops::FilterType::Lanczos3,
)
};
Ok(DownloadedOpenAiImage {
bytes: encode_editor_image_edit_png(
restored,
StatusCode::BAD_GATEWAY,
"恢复图片改造结果目标尺寸失败",
)?,
mime_type: "image/png".to_string(),
extension: "png".to_string(),
})
}
pub async fn edit_editor_image(
State(state): State<AppState>,
Extension(request_context): Extension<RequestContext>,
@@ -1965,7 +2107,7 @@ pub(crate) async fn edit_editor_image_for_owner(
}
let generation_options =
normalize_editor_generation_options(payload.model.as_deref(), None, None);
let image_size = normalize_editor_image_generation_size(payload.size.as_deref());
let requested_image_size = normalize_editor_image_generation_size(payload.size.as_deref());
let mut reference_images =
Vec::with_capacity(1 + payload.reference_image_srcs.as_ref().map_or(0, Vec::len));
reference_images.push(
@@ -1983,10 +2125,6 @@ pub(crate) async fn edit_editor_image_for_owner(
parse_editor_reference_image(state, caller.owner_user_id.as_str(), source).await?,
);
}
let expected_price_mud_points =
resolve_editor_image_edit_price(state, generation_options.model, image_size.as_ref())
.await?;
let settings = require_openai_image_settings(state)?.with_external_api_audit_context(
request_context,
caller.audit_subject_user_id.clone(),
@@ -1996,6 +2134,15 @@ pub(crate) async fn edit_editor_image_for_owner(
.or_else(|| payload.project_id.clone()),
);
let http_client = build_openai_image_http_client(&settings)?;
let edit_dimensions =
prepare_editor_image_edit_references(&mut reference_images, requested_image_size.as_ref())?;
let image_size = format!(
"{}x{}",
edit_dimensions.provider_width, edit_dimensions.provider_height
);
let expected_price_mud_points =
resolve_editor_image_edit_price(state, generation_options.model, image_size.as_ref())
.await?;
let generated = execute_billable_asset_operation_with_cost(
state,
caller.owner_user_id.as_str(),
@@ -2021,6 +2168,7 @@ pub(crate) async fn edit_editor_image_for_owner(
"message": "VectorEngine 未返回图片",
}))
})?;
let image = restore_editor_image_edit_output_dimensions(image, &edit_dimensions)?;
let (width, height) = image::load_from_memory(image.bytes.as_slice())
.map(|image| (image.width(), image.height()))
.unwrap_or((1024, 1024));
@@ -6435,6 +6583,59 @@ mod tests {
);
}
#[test]
fn editor_image_edit_aligns_provider_images_and_restores_source_dimensions() {
fn encode_png(width: u32, height: u32) -> Vec<u8> {
let image = image::DynamicImage::new_rgba8(width, height);
let mut bytes = Cursor::new(Vec::new());
image
.write_to(&mut bytes, image::ImageFormat::Png)
.expect("test image should encode");
bytes.into_inner()
}
let mut references = vec![
OpenAiReferenceImage {
bytes: encode_png(1537, 1025),
mime_type: "image/png".to_string(),
file_name: "source.png".to_string(),
},
OpenAiReferenceImage {
bytes: encode_png(501, 333),
mime_type: "image/png".to_string(),
file_name: "style.png".to_string(),
},
];
let dimensions = prepare_editor_image_edit_references(&mut references, "1537x1025")
.expect("references should align for provider");
assert_eq!(dimensions.target_width, 1537);
assert_eq!(dimensions.target_height, 1025);
assert_eq!(dimensions.provider_width, 1552);
assert_eq!(dimensions.provider_height, 1040);
let source = image::load_from_memory(references[0].bytes.as_slice()).unwrap();
assert_eq!((source.width(), source.height()), (1552, 1040));
let extra = image::load_from_memory(references[1].bytes.as_slice()).unwrap();
assert_eq!((extra.width(), extra.height()), (512, 336));
let restored = restore_editor_image_edit_output_dimensions(
DownloadedOpenAiImage {
bytes: encode_png(1552, 1040),
mime_type: "image/png".to_string(),
extension: "png".to_string(),
},
&dimensions,
)
.expect("provider output should restore source dimensions");
let restored_image = image::load_from_memory(restored.bytes.as_slice()).unwrap();
assert_eq!(
(restored_image.width(), restored_image.height()),
(1537, 1025)
);
assert_eq!(restored.mime_type, "image/png");
assert_eq!(restored.extension, "png");
}
#[test]
fn editor_generation_dimensions_follow_model_options() {
let default_generation = normalize_editor_generation_options(None, Some("1:1"), Some("1K"));
@@ -287,8 +287,8 @@ describe('ImageCanvasGenerationLayerModel', () => {
const sourceLayer = createSourceLayer({
width: 320,
height: 240,
originalWidth: 1024,
originalHeight: 768,
originalWidth: 1537,
originalHeight: 1025,
});
const layer = applyImageEditResultToSourceLayer({
generated: createGenerated({
@@ -313,10 +313,10 @@ describe('ImageCanvasGenerationLayerModel', () => {
id: 'layer-source',
title: '源图',
src: 'data:image/png;base64,edited',
width: 1536,
height: 1024,
originalWidth: 1536,
originalHeight: 1024,
width: 320,
height: 240,
originalWidth: 1537,
originalHeight: 1025,
resourceId: 'resource-edited',
sourceResourceId: 'resource-source',
objectKey: 'generated/edited.png',
@@ -247,18 +247,10 @@ export function applyImageEditResultToSourceLayer({
sourceLayer: CanvasLayer;
generationInputs: CanvasGenerationInputs;
}): CanvasLayer {
const originalWidth =
generated.width || sourceLayer.originalWidth || sourceLayer.width;
const originalHeight =
generated.height || sourceLayer.originalHeight || sourceLayer.height;
const { width, height } = resolveLayerResolutionSize(
originalWidth,
originalHeight,
{
width: sourceLayer.width,
height: sourceLayer.height,
},
);
const originalWidth = sourceLayer.originalWidth || sourceLayer.width;
const originalHeight = sourceLayer.originalHeight || sourceLayer.height;
const width = sourceLayer.width;
const height = sourceLayer.height;
return applyGeneratedMetadata(
{
...sourceLayer,
@@ -122,6 +122,8 @@ describe('ImageCanvasGenerationSubmissionModel', () => {
const sourceLayer = createLayer({
objectKey: 'generated-character-drafts/editor/source.png',
assetKind: 'character',
originalWidth: 1537,
originalHeight: 1025,
});
const plan = buildImageGenerationSubmissionPlan({
@@ -150,7 +152,7 @@ describe('ImageCanvasGenerationSubmissionModel', () => {
normalizedPrompt: '把当前图改成雨天',
sourceLayer,
editInput: {
size: '1024x768',
size: '1537x1025',
model: IMAGE_MODEL_GPT_IMAGE_2,
},
result: {
@@ -41,11 +41,8 @@ import {
DEFAULT_VIDEO_WEB_SEARCH_ENABLED,
ICON_DESCRIPTION_LIMIT,
IMAGE_MODEL_GPT_IMAGE_2,
inferEditorImageAspectRatio,
inferEditorImageSizeLabel,
normalizeEditorImageModel,
resolveCharacterAnimationSourceImageSrc,
resolveEditorImageGenerationPixelSize,
SEEDANCE_VIDEO_REFERENCE_LIMITS,
SPEC_GENERATION_ASPECT_RATIO,
SPEC_GENERATION_IMAGE_SIZE,
@@ -260,29 +257,12 @@ export function buildImageGenerationSubmissionPlan({
const basePrompt = dialog.prompt.trim() || '快速编辑图片';
const normalizedQuickEditPrompt = basePrompt;
const imageModel = IMAGE_MODEL_GPT_IMAGE_2;
const aspectRatio =
dialog.aspectRatio ??
inferEditorImageAspectRatio(
sourceLayer.originalWidth,
sourceLayer.originalHeight,
);
const imageSize =
dialog.imageSize ??
inferEditorImageSizeLabel(
sourceLayer.originalWidth,
sourceLayer.originalHeight,
);
const outputSize = resolveEditorImageGenerationPixelSize({
model: imageModel,
aspectRatio,
imageSize,
});
return {
kind: 'quick-edit',
normalizedPrompt: normalizedQuickEditPrompt,
sourceLayer,
editInput: {
size: `${outputSize.width}x${outputSize.height}`,
size: `${sourceLayer.originalWidth}x${sourceLayer.originalHeight}`,
model: imageModel,
},
result: {
@@ -535,7 +535,13 @@ describe('useImageCanvasGenerationSubmissionWorkflow', () => {
editEditorImageMock.mockResolvedValueOnce(
createGenerated({ prompt: '快速修图' }),
);
render(<SubmissionWorkflowHarness />);
render(
<SubmissionWorkflowHarness
initialLayers={[
createLayer({ originalWidth: 1537, originalHeight: 1025 }),
]}
/>,
);
fireEvent.click(screen.getByRole('button', { name: '打开快速编辑' }));
fireEvent.click(screen.getByRole('button', { name: '填写快速编辑' }));
@@ -550,7 +556,7 @@ describe('useImageCanvasGenerationSubmissionWorkflow', () => {
expect.objectContaining({
prompt: '快速修图',
sourceImageSrc: 'resource-source',
size: '1024x768',
size: '1537x1025',
model: 'gpt-image-2',
}),
);
@@ -1268,12 +1268,11 @@ export function useImageCanvasGenerationSubmissionWorkflow({
quickEditSourceLayer.originalWidth,
quickEditSourceLayer.originalHeight,
);
const quickEditOutputSize = resolveEditorImageGenerationPixelSize({
model: quickEditPanel.model,
aspectRatio: quickEditAspectRatio,
imageSize: quickEditImageSize,
});
const quickEditSize = `${quickEditOutputSize.width}x${quickEditOutputSize.height}`;
const quickEditOutputSize = {
width: quickEditSourceLayer.originalWidth,
height: quickEditSourceLayer.originalHeight,
};
const quickEditSize = `${quickEditSourceLayer.originalWidth}x${quickEditSourceLayer.originalHeight}`;
const basePrompt =
quickEditPanel.prompt.trim() ||
(panelMode === 'redraw' ? '重绘图片' : '快速编辑图片');
@@ -1678,6 +1677,7 @@ export function useImageCanvasGenerationSubmissionWorkflow({
editEditorImage({
prompt: submissionPlan.normalizedPrompt,
sourceImageSrc: referenceImageSrc,
size: `${submissionPlan.sourceLayer.originalWidth}x${submissionPlan.sourceLayer.originalHeight}`,
projectId,
assetKind: submissionPlan.sourceLayer.assetKind,
generationInputs: submissionPlan.generationInputs,