删除失效的图片模型兜底审计链路
Project CI / AI game creator shell Rust crates (pull_request) Successful in 3m0s
Project CI / Backend tests (pull_request) Failing after 12s
Project CI / AI game creator shell Rust smoke (pull_request) Successful in 3m50s
Project CI / Frontend tests (pull_request) Successful in 4m2s
Project CI / Repository checks (pull_request) Failing after 11s
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Successful in 8m46s
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Successful in 10m1s
Project CI / Native shell tests (pull_request) Successful in 7m3s
Project CI / AI game creator shell web tests (pull_request) Successful in 2m37s

- platform-image 移除 PlatformImageError::FallbackFailed、recovered_failure_audits()、with_recovered_failure_audits 与 into_final_error
- GeneratedImages 去掉恒为空的 recovered_failure_audits 字段,image_source 两处构造同步收敛
- executor 去掉空 Vec 累积、成功路径空 append 与 finish_image_model_fallback_error 包装,失败分支直接返回原错误
- api-server 去掉成功摘要的 recoveredFailureCount 与两处 recovered audit 落库循环,map_platform_image_error 去掉不可达分支
- 测试移除恒空断言,image_generation_does_not_retry_as_another_model 改为断言终态 Upstream 401
- 同步开发运维文档与 decision-log(含说明 2026-07-21 兜底条目的 recoveredFailureCount 已退役)
This commit is contained in:
2026-09-22 14:17:44 +08:00
parent c0d4e76a01
commit ff89951a9b
9 changed files with 58 additions and 149 deletions
@@ -1,5 +1,13 @@
# 决策记录
## 2026-09-22 删除失效的图片模型兜底审计链路
- 背景:跨模型兜底 `gpt-image-2-c` 删除后,`platform-image` 已没有任何地方写入 `recovered_failure_audits`executor 只创建空 Vec、成功路径空 append、失败路径原样返回,`PlatformImageError::FallbackFailed` 不可能被构造,`GeneratedImages.recovered_failure_audits` 和错误上的 `recovered_failure_audits()` 恒为空,api-server 成功运行摘要的 `recoveredFailureCount` 恒为 `0`。这条链路既是死代码,也会让排障误以为仍存在兜底审计。
- 决策:整体删除——`PlatformImageError::FallbackFailed``recovered_failure_audits()``with_recovered_failure_audits``into_final_error``GeneratedImages.recovered_failure_audits`、executor 的 `finish_image_model_fallback_error` 包装与空 Vec 累积,以及 api-server 成功运行摘要里的 `recoveredFailureCount` 和两处 recovered audit 落库循环。终态错误 audit(`error.audit()``external_api_call_failure`)语义保持不变。
- 边界:不改重试策略(同模型 408/429/5xx 重试仍只写 provider 日志)、不改 `external_api_call_failure` 的终态记录语义、不改扣费与 HTTP 错误映射。本文件 2026-07-21「VectorEngine 图片首选 gpt-image-2 并以 gpt-image-2-c 兜底」条目里的 `recoveredFailureCount` 观测字段自本次起不再存在;数据库历史审计字符串按原样保留。
- 影响范围:`server-rs/crates/platform-image/src/image_provider/runtime/{error.rs,util.rs,types.rs,executor.rs,image_source.rs}``server-rs/crates/platform-image/tests/image_provider.rs``server-rs/crates/api-server/src/openai_image_generation.rs``docs/【开发运维】本地开发验证与生产运维-2026-05-15.md`
- 验证:`cargo test -p platform-image``cargo check -p api-server``npm run check:encoding``npm run check:doc-index``git diff --check`
## 2026-09-22 图片 provider 单次 attempt 超时解耦
- 背景:`config.rs``TIANTOKEN_IMAGE_REQUEST_TIMEOUT_MS`(回退 `VECTOR_ENGINE_IMAGE_REQUEST_TIMEOUT_MS`)统一写进 `vector_engine_image_request_timeout_ms``state.rs` 又用这一个字段同时构造 Tiantoken 与 VectorEngine 两个图片 client`OpenAiImageSettings` 也取同一个值;结果是两个 provider 无法各自设置单次 attempt 超时,配置 Tiantoken 变量会顺带改写 VectorEngine 的 deadline(反之亦然)。
@@ -244,7 +244,7 @@ spacetime sql <database> "SELECT * FROM runtime_setting LIMIT 1" --server http:/
SFX V2 发布必须使用维护窗:先关闭 SFX 入队,再对显式目标执行只读 `spacetime sql <database> --server <server-url> --format json "SELECT job_id, status, request_payload_json FROM external_generation_job WHERE job_kind = 'editor_sound_effect_generation' AND (status = 'pending' OR status = 'running')"`;结果非零时保持旧 Worker drain,不得删除任务或让新 Worker 解析旧 Vidu payload。禁止依赖默认 server,禁止使用 `--root-dir`。清零后先部署共享 env 已对齐的 api-server / external-generation worker,检查 `/healthz` 和 Worker 启动,再部署 Web 并小流量开放 SFX。灰度对账 job 完成数、退款数、ElevenLabs POST 数、完成资源数和孤儿资源;翻译失败仍调用 provider、单 job provider POST 大于一次、成功退款或失败未退款均应立即停止放量。回滚先停止入队并收口 V2 pending / running job,不自动切回 Vidu,不执行 SpacetimeDB schema 或数据回滚。完整清单见 `docs/【实施记录】SFX生成优化V2.0T6测试与发布门禁-2026-08-07.md`
VectorEngine 图片生成 / 编辑在 `request_send` 阶段出现 `timeout``connect`、libcurl 35 SSL connect reset、libcurl 56 receive error / `unexpected eof while reading`、recv failure 等临时传输错误,或在 `upstream_status` 阶段收到 408 / 429 / 5xx(例如 Nginx HTML `502 Bad Gateway`)时,`platform-image` 会在一次业务请求总上限 5 次内处理;multipart 图片编辑每次重试都会重新构造 form,避免复用已消费的 body。每次 provider attempt 都按任务确定的具体模型发送(`gpt-image-2.5-flare-c` 生成 / `gpt-image-2.5-sunburst-c` 编辑 / `gemini-3.1-flash-image-preview` nanobanana);明确模型不可用、408 / 非拒绝类 429 / 5xx、响应解析失败或非拒绝类缺图时只在同一具体模型内重试,不切模型,历史兜底模型 `gpt-image-2-c` 已从代码整体删除。发送 / 连接错误无法确认上游是否已受理,同样只重试同一模型;认证、普通参数、安全拒绝、图片下载和 budget 错误不重试。worker 从 job 开始的同一时钟起点计算绝对 deadline,常规保留最后 `60` 秒给审计、OSS 和终态写回;job 预算小于 `120` 秒时保留一半。VectorEngine 单次 attempt timeout 取配置值和剩余 provider 预算的较小值;退避后已没有下一次 attempt 的预算时立即停止。该 deadline 覆盖参考图、provider 请求 / 响应和响应图片下载的整次 provider future,但只在 worker 进程内通过 `RequestContext` 传递;普通 HTTP / `inline` 没有该 deadline,继续保持原有 timeout 和重试行为。重试过程中的失败 attempt 仍写入 `external_api_call_failure`;即使后续 attempt 成功,成功运行摘要`recoveredFailureCount` 同时递增。排查生产失败时应同时统计 retry 日志和最终 audit,避免把一次用户请求内的多次发送误判成多个用户请求。这项收口不修改 lease 续租 / fencing、迟到写回仲裁、attempt 耗尽与原子退款语义。
VectorEngine 图片生成 / 编辑在 `request_send` 阶段出现 `timeout``connect`、libcurl 35 SSL connect reset、libcurl 56 receive error / `unexpected eof while reading`、recv failure 等临时传输错误,或在 `upstream_status` 阶段收到 408 / 429 / 5xx(例如 Nginx HTML `502 Bad Gateway`)时,`platform-image` 会在一次业务请求总上限 5 次内处理;multipart 图片编辑每次重试都会重新构造 form,避免复用已消费的 body。每次 provider attempt 都按任务确定的具体模型发送(`gpt-image-2.5-flare-c` 生成 / `gpt-image-2.5-sunburst-c` 编辑 / `gemini-3.1-flash-image-preview` nanobanana);明确模型不可用、408 / 非拒绝类 429 / 5xx、响应解析失败或非拒绝类缺图时只在同一具体模型内重试,不切模型,历史兜底模型 `gpt-image-2-c` 已从代码整体删除。发送 / 连接错误无法确认上游是否已受理,同样只重试同一模型;认证、普通参数、安全拒绝、图片下载和 budget 错误不重试。worker 从 job 开始的同一时钟起点计算绝对 deadline,常规保留最后 `60` 秒给审计、OSS 和终态写回;job 预算小于 `120` 秒时保留一半。VectorEngine 单次 attempt timeout 取配置值和剩余 provider 预算的较小值;退避后已没有下一次 attempt 的预算时立即停止。该 deadline 覆盖参考图、provider 请求 / 响应和响应图片下载的整次 provider future,但只在 worker 进程内通过 `RequestContext` 传递;普通 HTTP / `inline` 没有该 deadline,继续保持原有 timeout 和重试行为。重试过程中的失败 attempt 只写 provider 日志;`external_api_call_failure` 只记录终态的 provider 错误 audit,成功运行摘要不再包含已删除`recoveredFailureCount`。排查生产失败时应同时统计 retry 日志和最终 audit,避免把一次用户请求内的多次发送误判成多个用户请求。这项收口不修改 lease 续租 / fencing、迟到写回仲裁、attempt 耗尽与原子退款语义。
图片编辑器生成属于持久队列长任务:提交接口返回 job 后,前端通过 `/api/runtime/external-generation/jobs/{jobId}` 与编辑器项目资源状态收敛。生产排查小程序或 WebView `Failed to fetch` 时,若 Nginx access log 为 `499``upstream_status=-`,先按提交请求的 `request_id`、job id、worker 日志和 `external_api_call_failure` 对齐真实任务,不把客户端断开直接判定为 provider 失败。
@@ -364,9 +364,6 @@ async fn map_platform_image_result(
) -> Result<OpenAiGeneratedImages, AppError> {
match result {
Ok(value) => {
for audit in &value.recovered_failure_audits {
record_openai_image_failure_audit_if_configured(settings, audit).await;
}
if let Some(state) = settings.external_api_audit_state.as_ref() {
record_external_generation_run_after_success(
state,
@@ -381,7 +378,6 @@ async fn map_platform_image_result(
Some(json!({
"imageCount": value.images.len(),
"actualPromptChars": value.actual_prompt.as_ref().map(|prompt| prompt.chars().count()),
"recoveredFailureCount": value.recovered_failure_audits.len(),
})),
)
.await;
@@ -389,9 +385,6 @@ async fn map_platform_image_result(
Ok(value)
}
Err(error) => {
for audit in error.recovered_failure_audits() {
record_openai_image_failure_audit_if_configured(settings, audit).await;
}
if let Some(state) = settings.external_api_audit_state.as_ref() {
record_external_generation_run_after_success(
state,
@@ -445,7 +438,6 @@ pub(crate) fn build_openai_image_failure_audit_draft(
}
pub(crate) fn map_platform_image_error(error: PlatformImageError) -> AppError {
let error = error.into_final_error();
let status = match error.status_hint() {
PlatformImageStatusHint::BadRequest => StatusCode::BAD_REQUEST,
PlatformImageStatusHint::ServiceUnavailable => StatusCode::SERVICE_UNAVAILABLE,
@@ -490,9 +482,6 @@ pub(crate) fn map_platform_image_error(error: PlatformImageError) -> AppError {
details["rawExcerpt"] = json!(raw_excerpt);
}
PlatformImageError::MissingImage { .. } => {}
PlatformImageError::FallbackFailed { .. } => {
unreachable!("fallback wrapper should be removed before HTTP error mapping")
}
}
if let Some(audit) = error.audit() {
@@ -55,10 +55,6 @@ pub enum PlatformImageError {
message: String,
audit: Option<PlatformImageFailureAudit>,
},
FallbackFailed {
final_error: Box<PlatformImageError>,
recovered_failure_audits: Vec<PlatformImageFailureAudit>,
},
}
impl PlatformImageError {
@@ -70,7 +66,6 @@ impl PlatformImageError {
| Self::Upstream { provider, .. }
| Self::ResponseParse { provider, .. }
| Self::MissingImage { provider, .. } => provider,
Self::FallbackFailed { final_error, .. } => final_error.provider(),
}
}
@@ -82,7 +77,6 @@ impl PlatformImageError {
| Self::Upstream { message, .. }
| Self::ResponseParse { message, .. }
| Self::MissingImage { message, .. } => message,
Self::FallbackFailed { final_error, .. } => final_error.message(),
}
}
@@ -92,42 +86,10 @@ impl PlatformImageError {
| Self::Upstream { audit, .. }
| Self::ResponseParse { audit, .. }
| Self::MissingImage { audit, .. } => audit.as_ref(),
Self::FallbackFailed { final_error, .. } => final_error.audit(),
Self::InvalidConfig { .. } | Self::InvalidRequest { .. } => None,
}
}
pub fn recovered_failure_audits(&self) -> &[PlatformImageFailureAudit] {
match self {
Self::FallbackFailed {
recovered_failure_audits,
..
} => recovered_failure_audits.as_slice(),
_ => &[],
}
}
pub(crate) fn with_recovered_failure_audits(
self,
recovered_failure_audits: Vec<PlatformImageFailureAudit>,
) -> Self {
if recovered_failure_audits.is_empty() {
self
} else {
Self::FallbackFailed {
final_error: Box::new(self),
recovered_failure_audits,
}
}
}
pub fn into_final_error(self) -> Self {
match self {
Self::FallbackFailed { final_error, .. } => final_error.into_final_error(),
error => error,
}
}
pub fn status_hint(&self) -> PlatformImageStatusHint {
match self {
Self::InvalidConfig { .. } => PlatformImageStatusHint::ServiceUnavailable,
@@ -144,7 +106,6 @@ impl PlatformImageError {
| Self::Upstream { .. }
| Self::ResponseParse { .. }
| Self::MissingImage { .. } => PlatformImageStatusHint::BadGateway,
Self::FallbackFailed { final_error, .. } => final_error.status_hint(),
}
}
}
@@ -102,7 +102,6 @@ pub async fn create_image_generation_with_model(
let normalized_size = normalize_image_size_for_model(requested_model, size);
let started_at = std::time::Instant::now();
let upstream_model = preferred_image_upstream_model(requested_model);
let mut recovered_failure_audits = Vec::new();
let mut attempt = 1;
loop {
let request_body = build_image_request_body_with_model(
@@ -116,17 +115,14 @@ pub async fn create_image_generation_with_model(
let Some(attempt_timeout_ms) =
effective_request_timeout_ms(settings.request_timeout_ms, settings.request_deadline)
else {
return Err(finish_image_model_fallback_error(
request_budget_exhausted_error(
settings.provider,
request_url.as_str(),
failure_context,
auditable_image_model(upstream_model),
Some(started_at.elapsed().as_millis() as u64),
Some(prompt.chars().count()),
Some(reference_images.len()),
),
&mut recovered_failure_audits,
return Err(request_budget_exhausted_error(
settings.provider,
request_url.as_str(),
failure_context,
auditable_image_model(upstream_model),
Some(started_at.elapsed().as_millis() as u64),
Some(prompt.chars().count()),
Some(reference_images.len()),
));
};
let response = match send_image_json_request_with_curl(
@@ -189,20 +185,17 @@ pub async fn create_image_generation_with_model(
continue;
}
}
return Err(finish_image_model_fallback_error(
map_curl_error(
settings.provider,
format!("{failure_context}:创建图片生成任务失败").as_str(),
request_url.as_str(),
"request_send",
auditable_image_model(upstream_model),
error,
started_at.elapsed().as_millis() as u64,
Some(prompt.chars().count()),
Some(reference_images.len()),
Some(&request_body),
),
&mut recovered_failure_audits,
return Err(map_curl_error(
settings.provider,
format!("{failure_context}:创建图片生成任务失败").as_str(),
request_url.as_str(),
"request_send",
auditable_image_model(upstream_model),
error,
started_at.elapsed().as_millis() as u64,
Some(prompt.chars().count()),
Some(reference_images.len()),
Some(&request_body),
));
}
};
@@ -240,18 +233,8 @@ pub async fn create_image_generation_with_model(
)
.await
{
Ok(mut generated) => {
generated
.recovered_failure_audits
.append(&mut recovered_failure_audits);
return Ok(generated);
}
Err(error) => {
return Err(finish_image_model_fallback_error(
error,
&mut recovered_failure_audits,
));
}
Ok(generated) => return Ok(generated),
Err(error) => return Err(error),
}
}
}
@@ -516,7 +499,6 @@ pub async fn create_image_edit_with_references_and_model(
reference_images.iter().map(|image| image.bytes.len()).sum();
let started_at = std::time::Instant::now();
let upstream_model = preferred_image_upstream_model(requested_model);
let mut recovered_failure_audits = Vec::new();
let mut attempt = 1;
loop {
let request_params = build_image_edit_request_log_params(
@@ -552,17 +534,14 @@ pub async fn create_image_edit_with_references_and_model(
let Some(attempt_timeout_ms) =
effective_request_timeout_ms(settings.request_timeout_ms, settings.request_deadline)
else {
return Err(finish_image_model_fallback_error(
request_budget_exhausted_error(
settings.provider,
request_url.as_str(),
failure_context,
auditable_image_model(upstream_model),
Some(started_at.elapsed().as_millis() as u64),
Some(prompt.chars().count()),
Some(reference_image_count),
),
&mut recovered_failure_audits,
return Err(request_budget_exhausted_error(
settings.provider,
request_url.as_str(),
failure_context,
auditable_image_model(upstream_model),
Some(started_at.elapsed().as_millis() as u64),
Some(prompt.chars().count()),
Some(reference_image_count),
));
};
let response = match send_image_multipart_edit_request_with_curl(
@@ -630,20 +609,17 @@ pub async fn create_image_edit_with_references_and_model(
continue;
}
}
return Err(finish_image_model_fallback_error(
map_curl_error(
settings.provider,
format!("{failure_context}:创建图片编辑任务失败").as_str(),
request_url.as_str(),
"request_send",
auditable_image_model(upstream_model),
error,
started_at.elapsed().as_millis() as u64,
Some(prompt.chars().count()),
Some(reference_image_count),
Some(&request_params),
),
&mut recovered_failure_audits,
return Err(map_curl_error(
settings.provider,
format!("{failure_context}:创建图片编辑任务失败").as_str(),
request_url.as_str(),
"request_send",
auditable_image_model(upstream_model),
error,
started_at.elapsed().as_millis() as u64,
Some(prompt.chars().count()),
Some(reference_image_count),
Some(&request_params),
));
}
};
@@ -683,18 +659,8 @@ pub async fn create_image_edit_with_references_and_model(
)
.await
{
Ok(mut generated) => {
generated
.recovered_failure_audits
.append(&mut recovered_failure_audits);
return Ok(generated);
}
Err(error) => {
return Err(finish_image_model_fallback_error(
error,
&mut recovered_failure_audits,
));
}
Ok(generated) => return Ok(generated),
Err(error) => return Err(error),
}
}
}
@@ -744,13 +710,6 @@ fn preferred_image_upstream_model(requested_model: &str) -> &str {
requested_model
}
fn finish_image_model_fallback_error(
error: PlatformImageError,
recovered_failure_audits: &mut Vec<super::audit::PlatformImageFailureAudit>,
) -> PlatformImageError {
error.with_recovered_failure_audits(std::mem::take(recovered_failure_audits))
}
fn auditable_image_model(model: &str) -> Option<&'static str> {
match model {
GPT_IMAGE_2_MODEL => Some(GPT_IMAGE_2_MODEL),
@@ -127,7 +127,6 @@ pub(crate) async fn download_images_from_urls(
task_id,
actual_prompt: None,
images,
recovered_failure_audits: Vec::new(),
})
}
@@ -248,7 +247,6 @@ pub(crate) fn images_from_base64(
task_id,
actual_prompt: None,
images,
recovered_failure_audits: Vec::new(),
}
}
@@ -1,5 +1,3 @@
use crate::image_provider::audit::PlatformImageFailureAudit;
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum ImageProvider {
VectorEngine,
@@ -81,7 +79,6 @@ pub struct GeneratedImages {
pub task_id: String,
pub actual_prompt: Option<String>,
pub images: Vec<DownloadedImage>,
pub recovered_failure_audits: Vec<PlatformImageFailureAudit>,
}
#[derive(Clone, Debug)]
@@ -79,13 +79,6 @@ impl PlatformImageError {
message,
audit: Some(audit),
},
Self::FallbackFailed {
final_error,
recovered_failure_audits,
} => Self::FallbackFailed {
final_error: Box::new(final_error.with_audit(audit)),
recovered_failure_audits,
},
Self::InvalidConfig { .. } | Self::InvalidRequest { .. } => self,
}
}
@@ -325,7 +325,6 @@ async fn image_edit_retries_send_timeout_once_and_succeeds() {
assert_eq!(generated.images.len(), 1);
assert_eq!(generated.images[0].mime_type, "image/png");
assert!(generated.recovered_failure_audits.is_empty());
assert_eq!(request_count.load(Ordering::SeqCst), 2);
let requests = requests.lock().await;
assert!(
@@ -600,7 +599,6 @@ async fn image_generation_uses_default_generation_model_on_success() {
.expect("preferred model should generate image");
assert_eq!(generated.images.len(), 1);
assert!(generated.recovered_failure_audits.is_empty());
let requests = requests.lock().await;
assert_eq!(requests.len(), 1);
assert!(requests[0].contains(&format!("\"model\":\"{GPT_IMAGE_2_5_GENERATION_MODEL}\"")));
@@ -786,7 +784,13 @@ async fn image_generation_does_not_retry_as_another_model() {
.await
.expect_err("upstream failure should remain terminal");
assert!(error.recovered_failure_audits().is_empty());
assert!(matches!(
error,
PlatformImageError::Upstream {
upstream_status: 401,
..
}
));
let requests = requests.lock().await;
assert_eq!(requests.len(), 2);
assert!(requests[0].contains(&format!("\"model\":\"{GPT_IMAGE_2_5_GENERATION_MODEL}\"")));