修复动作帧 OSS 重试错误分类
排除确定性 reqwest 客户端错误的无效重试 识别 OSS 400 RequestTimeout 并保留请求标识 补充有界错误体解析、回归测试和运维文档
This commit is contained in:
@@ -19,7 +19,7 @@
|
||||
## 2026-07-20 角色动画帧 OSS 请求使用专用连接池、并发保护与结构化重试
|
||||
|
||||
- 背景:角色动作逐帧流水线会同时发起源帧 PUT、透明帧 PUT 和最终帧 HEAD;原路径每次请求新建 `reqwest::Client`,且 OSS 请求错误丢失 HTTP 状态和 timeout/connect/transport 分类,多个动画任务叠加时无法在进程级限制 OSS 在途请求,也无法安全区分 PUT 与 HEAD 的失败。
|
||||
- 决策:`AppState` 仅为角色动画帧初始化一次 OSS HTTP Client 和 8 路 `Semaphore`。全帧 Future 仍保持 `buffer_unordered(frame_count.max(1))`,BgFilter、阿里云抠图和本地处理不占 OSS permit;每次 PUT/HEAD 网络 attempt 单独获取 permit,退避期间释放。`platform-oss` 保留 `OssErrorKind::Request`,但在 `OssError::Request` 中保留 operation、status、timeout、connect、transport 和原脱敏 message,并为动画帧提供 3 次 attempt、250ms/500ms 退避的 PUT/HEAD 独立重试。仅无响应传输错误、timeout、408、429 和 5xx 可重试;确定性 4xx、配置、签名、URL、空请求体、抠图和素材登记错误不重试。重试体在 platform-oss 内一次转为可复用 `Bytes`,每次重新签名和构造 Request,不复制整帧字节。
|
||||
- 决策:`AppState` 仅为角色动画帧初始化一次 OSS HTTP Client 和 8 路 `Semaphore`。全帧 Future 仍保持 `buffer_unordered(frame_count.max(1))`,BgFilter、阿里云抠图和本地处理不占 OSS permit;每次 PUT/HEAD 网络 attempt 单独获取 permit,退避期间释放。`platform-oss` 保留 `OssErrorKind::Request`,但在 `OssError::Request` 中保留 operation、status、timeout、connect、transport、OSS code、OSS request-id 和原脱敏 message,并为动画帧提供 3 次 attempt、250ms/500ms 退避的 PUT/HEAD 独立重试。仅无响应传输错误、timeout、OSS PutObject 的 `400 + RequestTimeout`、408、429 和 5xx 可重试;动作帧 PUT 对 400 错误体最多读取 16 KiB,只提取 `Code` 和响应头优先的 `x-oss-request-id`,不记录完整 XML。除 `RequestTimeout` 外的确定性 4xx、配置、签名、URL、空请求体、抠图和素材登记错误不重试。重试体在 platform-oss 内一次转为可复用 `Bytes`,每次重新签名和构造 Request,不复制整帧字节。
|
||||
- 失败语义:最终帧 PUT 成功后才执行 HEAD;HEAD 失败只重试 HEAD,不重复 PUT。任一帧最终失败仍排空已启动的 Future、整段动作退款并禁止发布缺帧动画,帧结果继续按原始序号排序。
|
||||
- 影响范围:`state.rs`、`platform-oss/lib.rs`、`character_animation_assets.rs`、对应 Cargo 依赖和架构 / 运维文档;不改变其他 OSS 调用方、BgFilter/阿里云降级、worker、计费退款、SpacetimeDB schema/DTO 或前端接口。
|
||||
- 验证方式:`cargo test -p platform-oss --manifest-path server-rs/Cargo.toml`、`cargo test -p api-server character_animation --manifest-path server-rs/Cargo.toml`、`cargo check -p api-server --manifest-path server-rs/Cargo.toml`、`npm run check:encoding`、`git diff --check`。
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -409,7 +409,7 @@ curl -fsS --max-time 5 http://127.0.0.1/api/runtime/puzzle/gallery >/dev/null
|
||||
|
||||
### 角色动画帧 OSS 排障
|
||||
|
||||
角色动画源帧 PUT、透明帧 PUT 和最终帧 HEAD 使用 `AppState` 内同一个 OSS HTTP Client/连接池,并受进程级 8 路 OSS permit 保护;BgFilter、阿里云抠图和本地处理不占用该 permit。每个 OSS attempt 最多 3 次(首次 + 2 次重试),退避为 250ms、500ms;只重试 timeout、无 HTTP 响应传输错误、408、429 和 500–599。日志字段包括 `frame_index`、`object_key`、`operation=source_put|final_put|final_head`、`attempt`、`max_attempts`、`retryable`、`will_retry`、`retry_delay_ms`、`permit_wait_ms`、`timeout`、`connect`、`transport`、`status` 和 `elapsed_ms`。`请求 OSS 失败` 时,`timeout/connect/transport=true` 表示传输类失败,`status=429` 或 `500–599` 表示暂时性服务端失败;`400/401/403/404`、配置、URL 和签名错误是确定性失败,不会重试。最终帧 HEAD 失败只会重试 HEAD,不会重复 PUT;如果任一帧最终失败,确认整段动作已排空已启动 Future,并检查任务按现有契约退款且没有发布缺帧动画。
|
||||
角色动画源帧 PUT、透明帧 PUT 和最终帧 HEAD 使用 `AppState` 内同一个 OSS HTTP Client/连接池,并受进程级 8 路 OSS permit 保护;BgFilter、阿里云抠图和本地处理不占用该 permit。每个 OSS attempt 最多 3 次(首次 + 2 次重试),退避为 250ms、500ms;只重试 timeout、无 HTTP 响应传输错误、OSS PutObject 的 `400 + RequestTimeout`、408、429 和 500–599。动作帧 PUT 收到 400 时只读取最多 16 KiB OSS 错误 XML,提取 `Code` 和 `RequestId`;`oss_request_id` 优先使用响应头 `x-oss-request-id`,XML 字段只作回退。日志字段包括 `frame_index`、`object_key`、`operation=source_put|final_put|final_head`、`attempt`、`max_attempts`、`retryable`、`will_retry`、`retry_delay_ms`、`permit_wait_ms`、`timeout`、`connect`、`transport`、`oss_code`、`oss_request_id`、`status` 和 `elapsed_ms`。`请求 OSS 失败` 时,`timeout/connect/transport=true` 表示传输类失败;`status=400, oss_code=RequestTimeout, timeout=true`、`status=429` 或 `500–599` 表示暂时性失败。除该 `RequestTimeout` 例外外,其他 400、401/403/404、配置、URL 和签名错误是确定性失败,不会重试。最终帧 HEAD 失败只会重试 HEAD,不会重复 PUT;如果任一帧最终失败,确认整段动作已排空已启动 Future,并检查任务按现有契约退款且没有发布缺帧动画。
|
||||
|
||||
## 生产运维
|
||||
|
||||
@@ -605,7 +605,7 @@ OpenTelemetry 现阶段默认开启 OTLP traces / metrics / logs,但本地日
|
||||
- api-server 会随 metrics 发送进程级指标:`process.memory.usage`、`process.memory.virtual`、`process.cpu.time`、`genarrative.process.cpu.usage_percent`、`process.thread.count`、`genarrative.process.memory.private`;Windows 额外发送 `process.windows.handle.count`,Linux 额外发送 `process.unix.file_descriptor.count`。这些指标只描述当前进程,不携带请求、用户或作品 label。
|
||||
- HTTP 运行态补充发送 `genarrative.http.server.response_bodies.in_flight` 与 `genarrative.http.server.request_permits.available`,后者带低基数 `pool=default|gallery|detail|admin` label,用于区分业务 handler / 背压 permit 是否仍被占用;拼图广场热点缓存补充发送 `genarrative.puzzle_gallery.cache.*` 指标,记录 fresh hit、stale hit、未命中、后台刷新开始 / 失败、重建耗时和预序列化 data JSON 字节数。
|
||||
- 外部 API 失败统一发送 OTLP 并落库。当前 VectorEngine `gpt-image-2` 图片生成 / 编辑失败由 `platform-image` provider 输出结构化日志字段,字段包括 provider、endpoint、failure_stage、status、source、source_chain、source_chain_depth、timeout、retryable、latency_ms、prompt_chars、reference_image_count、image_model、request_params 和 raw_excerpt;图片编辑请求参数日志还会带 reference_image_bytes_total,并在 request_params.referenceImages 中记录每个 multipart `image` part 的 fileName、mimeType 和 bytes,不记录 API key 或原始图片 bytes;`api-server` 再记录指标 `genarrative.external_api.failures{provider,failure_stage,status_class,retryable}`,并写入 `tracking_event`,`event_key = external_api_call_failure`、`module_key = external-api`、`scope_kind = module`、`scope_id = provider`。调用方能拿到身份上下文时,失败事件还会在行级 `user_id` / `owner_user_id` / `profile_id` 和 `metadata_json.userId` / `metadata_json.profileId` / `metadata_json.requestId` / `metadata_json.errorSource` 中记录触发者、草稿 / 作品作用域、请求标识和传输错误链。排障时先按 provider / failureStage 聚合,再下钻 userId / profileId,最后结合 request 日志、errorSource 和上游响应 excerpt 判断是限流、超时、解析失败还是未返回图片。
|
||||
- OSS 平台适配器也输出结构化日志,覆盖 `sign_post_object`、`sign_get_object_url`、`head_object` 和 `put_object`。排查资产签名、上传或确认失败时,先按 `provider=aliyun-oss` 与 `operation` 过滤,再看 `object_key` / `key_prefix`、`status`、`status_class`、`error_kind`、`content_length`、`content_type` 和 `elapsed_ms`;角色动画逐帧额外按 `frame_index`、`operation=source_put|final_put|final_head`、`attempt/max_attempts` 和 `will_retry` 对齐同一对象的请求尝试。`请求 OSS 失败` 时,`timeout/connect/transport=true` 表示传输类失败,`status=429` 或 `500–599` 表示服务端暂时失败;`400/401/403/404`、配置、URL 和签名错误是确定性失败,不会重试。最终帧 HEAD 失败只会重试 HEAD,不会重复 PUT。日志不得包含 AccessKey、policy、signature、Authorization header 或完整 signed URL。排查 generated 图片重复下载时,先确认前端输入是否为 `/generated-*` legacy path 或可归一化的 `https://*.oss-*.aliyuncs.com/generated-*`;正确链路应先调 `/api/assets/read-url`,再由浏览器请求 signed URL,且同一路径、同一 `refreshKey` 版本和未临近过期的 signed URL 应复用。新上传 generated 私有对象应带 `Cache-Control: public, max-age=31536000, immutable`;旧对象若只有 `ETag` / `Last-Modified`,浏览器会走 304 协商缓存而不是长期强缓存,可通过刷新 OSS 元数据或 CDN 配置补齐。
|
||||
- OSS 平台适配器也输出结构化日志,覆盖 `sign_post_object`、`sign_get_object_url`、`head_object` 和 `put_object`。排查资产签名、上传或确认失败时,先按 `provider=aliyun-oss` 与 `operation` 过滤,再看 `object_key` / `key_prefix`、`status`、`status_class`、`error_kind`、`content_length`、`content_type` 和 `elapsed_ms`;角色动画逐帧额外按 `frame_index`、`operation=source_put|final_put|final_head`、`attempt/max_attempts`、`will_retry`、`oss_code` 和 `oss_request_id` 对齐同一对象的请求尝试。`请求 OSS 失败` 时,`timeout/connect/transport=true` 表示传输类失败,OSS PutObject 的 `status=400, oss_code=RequestTimeout, timeout=true`、`status=429` 或 `500–599` 表示暂时性失败;除该 `RequestTimeout` 例外外,其他 400、401/403/404、配置、URL 和签名错误是确定性失败,不会重试。最终帧 HEAD 失败只会重试 HEAD,不会重复 PUT。日志不得包含 AccessKey、policy、signature、Authorization header、完整 signed URL 或 OSS 错误响应体;`oss_request_id` 只用于关联 OSS 服务端排障。排查 generated 图片重复下载时,先确认前端输入是否为 `/generated-*` legacy path 或可归一化的 `https://*.oss-*.aliyuncs.com/generated-*`;正确链路应先调 `/api/assets/read-url`,再由浏览器请求 signed URL,且同一路径、同一 `refreshKey` 版本和未临近过期的 signed URL 应复用。新上传 generated 私有对象应带 `Cache-Control: public, max-age=31536000, immutable`;旧对象若只有 `ETag` / `Last-Modified`,浏览器会走 304 协商缓存而不是长期强缓存,可通过刷新 OSS 元数据或 CDN 配置补齐。
|
||||
- SpacetimeDB 观测分为两类:procedure / reducer 调用继续用 `genarrative.spacetime.procedure.*`,订阅本地 cache 读使用 `genarrative.spacetime.read.*`。`read=list_puzzle_gallery` 表示拼图广场当前从 `puzzle_gallery_card_view` 本地 cache 读取,不再每个 HTTP 请求调用 `list_puzzle_gallery` procedure。
|
||||
- 本地 Windows 直连压测的内存高水位要结合 K6 VU / 连接数解释。250 RPS 下过高 `PREALLOCATED_VUS` 可能让 300 个本地 Established 连接把 `api-server` private memory 瞬时推到 GB 级,且 `/healthz` 小响应也能复现;若压测结束后回落、`response_bodies.in_flight` 和背压 permit 未显示业务积压,应优先按连接 / 发送链路高水位处理,而不是判断为 SpacetimeDB 或 JSON 缓存泄漏。
|
||||
- Rider 的 Logs 面板只展示 log event 自身字段,不会自动展开父 span 的全部 attributes;请求完成日志会直接带 `request_id`、`http.request.method`、`http.route`、`url.scheme`、`url.path`、`http.response.status_code`、`status_class`、`latency_ms` 和 `slow_request`,完整链路继续到 Traces 面板按 trace/span 查看。
|
||||
|
||||
@@ -226,6 +226,8 @@ pub struct OssRequestError {
|
||||
pub timeout: bool,
|
||||
pub connect: bool,
|
||||
pub transport: bool,
|
||||
pub oss_code: Option<String>,
|
||||
pub oss_request_id: Option<String>,
|
||||
pub operation: OssRequestOperation,
|
||||
pub message: String,
|
||||
}
|
||||
@@ -259,6 +261,9 @@ pub struct OssRequestAttemptContext {
|
||||
|
||||
const CHARACTER_ANIMATION_OSS_MAX_ATTEMPTS: u32 = 3;
|
||||
const CHARACTER_ANIMATION_OSS_RETRY_DELAYS_MS: [u64; 2] = [250, 500];
|
||||
const CHARACTER_ANIMATION_OSS_ERROR_BODY_MAX_BYTES: usize = 16 * 1024;
|
||||
const OSS_ERROR_CODE_MAX_BYTES: usize = 128;
|
||||
const OSS_REQUEST_ID_MAX_BYTES: usize = 256;
|
||||
|
||||
struct PreparedHeadObject {
|
||||
object_key: String,
|
||||
@@ -1148,11 +1153,7 @@ impl OssClient {
|
||||
.map_err(|error| request_error_from_reqwest(OssRequestOperation::Put, error))?;
|
||||
|
||||
if !response.status().is_success() {
|
||||
return Err(request_status_error(
|
||||
OssRequestOperation::Put,
|
||||
response.status().as_u16(),
|
||||
format!("OSS PutObject 失败,状态码:{}", response.status()),
|
||||
));
|
||||
return Err(request_status_error_from_oss_put_response(response).await);
|
||||
}
|
||||
|
||||
let headers = response.headers();
|
||||
@@ -1190,11 +1191,103 @@ fn request_error(operation: OssRequestOperation, message: &str) -> OssError {
|
||||
timeout: false,
|
||||
connect: false,
|
||||
transport: false,
|
||||
oss_code: None,
|
||||
oss_request_id: None,
|
||||
operation,
|
||||
message: message.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
async fn request_status_error_from_oss_put_response(mut response: reqwest::Response) -> OssError {
|
||||
let status = response.status();
|
||||
let header_request_id = response
|
||||
.headers()
|
||||
.get("x-oss-request-id")
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.and_then(|value| normalize_oss_error_field(value, OSS_REQUEST_ID_MAX_BYTES));
|
||||
let body = if status == reqwest::StatusCode::BAD_REQUEST {
|
||||
read_bounded_oss_error_body(&mut response).await
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
|
||||
request_status_error_from_oss_parts(
|
||||
OssRequestOperation::Put,
|
||||
status.as_u16(),
|
||||
header_request_id,
|
||||
&body,
|
||||
)
|
||||
}
|
||||
|
||||
async fn read_bounded_oss_error_body(response: &mut reqwest::Response) -> Vec<u8> {
|
||||
let mut body = Vec::new();
|
||||
while body.len() < CHARACTER_ANIMATION_OSS_ERROR_BODY_MAX_BYTES {
|
||||
let chunk = match response.chunk().await {
|
||||
Ok(Some(chunk)) => chunk,
|
||||
Ok(None) | Err(_) => break,
|
||||
};
|
||||
let remaining = CHARACTER_ANIMATION_OSS_ERROR_BODY_MAX_BYTES - body.len();
|
||||
body.extend_from_slice(&chunk[..chunk.len().min(remaining)]);
|
||||
if chunk.len() > remaining {
|
||||
break;
|
||||
}
|
||||
}
|
||||
body
|
||||
}
|
||||
|
||||
fn request_status_error_from_oss_parts(
|
||||
operation: OssRequestOperation,
|
||||
status: u16,
|
||||
header_request_id: Option<String>,
|
||||
body: &[u8],
|
||||
) -> OssError {
|
||||
let header_request_id = header_request_id
|
||||
.as_deref()
|
||||
.and_then(|value| normalize_oss_error_field(value, OSS_REQUEST_ID_MAX_BYTES));
|
||||
let bounded_body = &body[..body.len().min(CHARACTER_ANIMATION_OSS_ERROR_BODY_MAX_BYTES)];
|
||||
let oss_code = extract_oss_error_xml_field(bounded_body, "Code", OSS_ERROR_CODE_MAX_BYTES);
|
||||
let xml_request_id =
|
||||
extract_oss_error_xml_field(bounded_body, "RequestId", OSS_REQUEST_ID_MAX_BYTES);
|
||||
let oss_request_id = header_request_id.or(xml_request_id);
|
||||
let timeout = status == reqwest::StatusCode::BAD_REQUEST.as_u16()
|
||||
&& oss_code.as_deref() == Some("RequestTimeout");
|
||||
let mut message = format!("OSS PutObject 失败,状态码:{status}");
|
||||
if let Some(oss_code) = oss_code.as_deref() {
|
||||
message.push_str(&format!(",OSS 错误码:{oss_code}"));
|
||||
}
|
||||
if let Some(oss_request_id) = oss_request_id.as_deref() {
|
||||
message.push_str(&format!(",OSS Request ID:{oss_request_id}"));
|
||||
}
|
||||
|
||||
OssError::Request(OssRequestError {
|
||||
status: Some(status),
|
||||
timeout,
|
||||
connect: false,
|
||||
transport: false,
|
||||
oss_code,
|
||||
oss_request_id,
|
||||
operation,
|
||||
message,
|
||||
})
|
||||
}
|
||||
|
||||
fn extract_oss_error_xml_field(body: &[u8], field: &str, max_bytes: usize) -> Option<String> {
|
||||
let body = std::str::from_utf8(body).ok()?;
|
||||
let start_tag = format!("<{field}>");
|
||||
let end_tag = format!("</{field}>");
|
||||
let value_start = body.find(&start_tag)? + start_tag.len();
|
||||
let value_end = value_start + body[value_start..].find(&end_tag)?;
|
||||
normalize_oss_error_field(&body[value_start..value_end], max_bytes)
|
||||
}
|
||||
|
||||
fn normalize_oss_error_field(value: &str, max_bytes: usize) -> Option<String> {
|
||||
let value = value.trim();
|
||||
if value.is_empty() || value.len() > max_bytes || value.chars().any(char::is_control) {
|
||||
return None;
|
||||
}
|
||||
Some(value.to_string())
|
||||
}
|
||||
|
||||
async fn run_animation_request_with_retry<T, F, Fut>(
|
||||
io_limiter: Arc<Semaphore>,
|
||||
attempt_context: OssRequestAttemptContext,
|
||||
@@ -1250,11 +1343,18 @@ where
|
||||
}
|
||||
|
||||
fn request_error_from_reqwest(operation: OssRequestOperation, error: reqwest::Error) -> OssError {
|
||||
let status = error.status().map(|status| status.as_u16());
|
||||
let timeout = error.is_timeout();
|
||||
let connect = error.is_connect();
|
||||
let transport = !timeout && !connect && (error.is_request() || error.is_body());
|
||||
|
||||
OssError::Request(OssRequestError {
|
||||
status: error.status().map(|status| status.as_u16()),
|
||||
timeout: error.is_timeout(),
|
||||
connect: error.is_connect(),
|
||||
transport: error.status().is_none() && !error.is_timeout() && !error.is_connect(),
|
||||
status,
|
||||
timeout,
|
||||
connect,
|
||||
transport,
|
||||
oss_code: None,
|
||||
oss_request_id: None,
|
||||
operation,
|
||||
message: format!("请求 OSS 失败:{error}"),
|
||||
})
|
||||
@@ -1266,6 +1366,8 @@ fn request_status_error(operation: OssRequestOperation, status: u16, message: St
|
||||
timeout: false,
|
||||
connect: false,
|
||||
transport: false,
|
||||
oss_code: None,
|
||||
oss_request_id: None,
|
||||
operation,
|
||||
message,
|
||||
})
|
||||
@@ -1276,23 +1378,28 @@ fn oss_error_is_retryable(error: &OssError) -> bool {
|
||||
return false;
|
||||
};
|
||||
|
||||
match request_error.status {
|
||||
Some(408 | 429 | 500..=599) => true,
|
||||
Some(_) => false,
|
||||
None => request_error.timeout || request_error.connect || request_error.transport,
|
||||
match (request_error.status, request_error.oss_code.as_deref()) {
|
||||
(Some(400), Some("RequestTimeout")) => true,
|
||||
(Some(408 | 429 | 500..=599), _) => true,
|
||||
(Some(_), _) => false,
|
||||
(None, _) => request_error.timeout || request_error.connect || request_error.transport,
|
||||
}
|
||||
}
|
||||
|
||||
fn request_error_details(error: Option<&OssError>) -> (Option<u16>, bool, bool, bool) {
|
||||
fn request_error_details(
|
||||
error: Option<&OssError>,
|
||||
) -> (Option<u16>, bool, bool, bool, Option<&str>, Option<&str>) {
|
||||
match error {
|
||||
Some(OssError::Request(request_error)) => (
|
||||
request_error.status,
|
||||
request_error.timeout,
|
||||
request_error.connect,
|
||||
request_error.transport,
|
||||
request_error.oss_code.as_deref(),
|
||||
request_error.oss_request_id.as_deref(),
|
||||
),
|
||||
Some(OssError::ObjectNotFound(_)) => (Some(404), false, false, false),
|
||||
_ => (None, false, false, false),
|
||||
Some(OssError::ObjectNotFound(_)) => (Some(404), false, false, false, None, None),
|
||||
_ => (None, false, false, false, None, None),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1308,7 +1415,8 @@ fn log_animation_request_attempt(
|
||||
success_status: Option<u16>,
|
||||
error: Option<&OssError>,
|
||||
) {
|
||||
let (error_status, timeout, connect, transport) = request_error_details(error);
|
||||
let (error_status, timeout, connect, transport, oss_code, oss_request_id) =
|
||||
request_error_details(error);
|
||||
let status = success_status.or(error_status);
|
||||
if error.is_none() {
|
||||
info!(
|
||||
@@ -1325,6 +1433,8 @@ fn log_animation_request_attempt(
|
||||
timeout,
|
||||
connect,
|
||||
transport,
|
||||
oss_code = oss_code.unwrap_or_default(),
|
||||
oss_request_id = oss_request_id.unwrap_or_default(),
|
||||
status = status.unwrap_or_default(),
|
||||
elapsed_ms,
|
||||
"角色动画 OSS 请求 attempt 完成"
|
||||
@@ -1345,6 +1455,8 @@ fn log_animation_request_attempt(
|
||||
timeout,
|
||||
connect,
|
||||
transport,
|
||||
oss_code = oss_code.unwrap_or_default(),
|
||||
oss_request_id = oss_request_id.unwrap_or_default(),
|
||||
status = status.unwrap_or_default(),
|
||||
elapsed_ms,
|
||||
error_kind = error.map(oss_error_kind_label),
|
||||
@@ -2058,6 +2170,8 @@ mod tests {
|
||||
timeout,
|
||||
connect,
|
||||
transport,
|
||||
oss_code: None,
|
||||
oss_request_id: None,
|
||||
operation: OssRequestOperation::Put,
|
||||
message: "mock request failure".to_string(),
|
||||
})
|
||||
@@ -2086,6 +2200,8 @@ mod tests {
|
||||
timeout: false,
|
||||
connect: false,
|
||||
transport: true,
|
||||
oss_code: None,
|
||||
oss_request_id: None,
|
||||
operation: OssRequestOperation::Put,
|
||||
message: "network".to_string(),
|
||||
})
|
||||
@@ -2099,12 +2215,113 @@ mod tests {
|
||||
let error = OssError::ObjectNotFound("missing".to_string());
|
||||
assert_eq!(
|
||||
request_error_details(Some(&error)),
|
||||
(Some(404), false, false, false)
|
||||
(Some(404), false, false, false, None, None)
|
||||
);
|
||||
assert_eq!(oss_error_kind_label(&error), "object_not_found");
|
||||
assert!(!oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn oss_request_timeout_400_is_retryable_and_prefers_header_request_id() {
|
||||
let body = br#"<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Error><Code>RequestTimeout</Code><RequestId>xml-request-id</RequestId></Error>"#;
|
||||
let error = request_status_error_from_oss_parts(
|
||||
OssRequestOperation::Put,
|
||||
400,
|
||||
Some("header-request-id".to_string()),
|
||||
body,
|
||||
);
|
||||
let OssError::Request(request_error) = &error else {
|
||||
panic!("OSS status failure should remain a request error");
|
||||
};
|
||||
|
||||
assert_eq!(request_error.status, Some(400));
|
||||
assert!(request_error.timeout);
|
||||
assert_eq!(request_error.oss_code.as_deref(), Some("RequestTimeout"));
|
||||
assert_eq!(
|
||||
request_error.oss_request_id.as_deref(),
|
||||
Some("header-request-id")
|
||||
);
|
||||
assert!(oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn oss_request_timeout_400_uses_xml_request_id_when_header_is_missing() {
|
||||
let body = br#"<Error>
|
||||
<Code>RequestTimeout</Code><RequestId>xml-request-id</RequestId>
|
||||
</Error>"#;
|
||||
let error = request_status_error_from_oss_parts(OssRequestOperation::Put, 400, None, body);
|
||||
let OssError::Request(request_error) = &error else {
|
||||
panic!("OSS status failure should remain a request error");
|
||||
};
|
||||
|
||||
assert_eq!(
|
||||
request_error.oss_request_id.as_deref(),
|
||||
Some("xml-request-id")
|
||||
);
|
||||
assert!(oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn other_oss_400_errors_and_malformed_xml_are_not_retryable() {
|
||||
for body in [
|
||||
b"<Error><Code>InvalidArgument</Code></Error>".as_slice(),
|
||||
b"<Error><Code>RequestTimeout".as_slice(),
|
||||
b"not xml".as_slice(),
|
||||
] {
|
||||
let error =
|
||||
request_status_error_from_oss_parts(OssRequestOperation::Put, 400, None, body);
|
||||
assert!(!oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
let error = request_status_error_from_oss_parts(
|
||||
OssRequestOperation::Put,
|
||||
403,
|
||||
None,
|
||||
b"<Error><Code>RequestTimeout</Code></Error>",
|
||||
);
|
||||
assert!(!oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn oss_error_xml_fields_beyond_body_limit_are_ignored() {
|
||||
let mut body = vec![b' '; CHARACTER_ANIMATION_OSS_ERROR_BODY_MAX_BYTES];
|
||||
body.extend_from_slice(
|
||||
b"<Error><Code>RequestTimeout</Code><RequestId>late</RequestId></Error>",
|
||||
);
|
||||
let error = request_status_error_from_oss_parts(OssRequestOperation::Put, 400, None, &body);
|
||||
let OssError::Request(request_error) = &error else {
|
||||
panic!("OSS status failure should remain a request error");
|
||||
};
|
||||
|
||||
assert_eq!(request_error.oss_code, None);
|
||||
assert_eq!(request_error.oss_request_id, None);
|
||||
assert!(!request_error.timeout);
|
||||
assert!(!oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reqwest_builder_error_is_not_retryable_transport() {
|
||||
let error = reqwest::Client::new()
|
||||
.put("https://example.com")
|
||||
.header("x-oss-meta-invalid", "first line\nsecond line")
|
||||
.send()
|
||||
.await
|
||||
.expect_err("invalid header must fail while building the request");
|
||||
assert!(error.is_builder());
|
||||
|
||||
let error = request_error_from_reqwest(OssRequestOperation::Put, error);
|
||||
let OssError::Request(request_error) = &error else {
|
||||
panic!("builder failure should remain an OSS request error");
|
||||
};
|
||||
|
||||
assert_eq!(request_error.status, None);
|
||||
assert!(!request_error.timeout);
|
||||
assert!(!request_error.connect);
|
||||
assert!(!request_error.transport);
|
||||
assert!(!oss_error_is_retryable(&error));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn animation_retry_retries_transport_then_succeeds() {
|
||||
let attempts = Arc::new(AtomicUsize::new(0));
|
||||
|
||||
Reference in New Issue
Block a user