Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 920bf90e5a | |||
| 896fcecbc1 |
+1
@@ -3931,6 +3931,7 @@ dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -32,6 +32,8 @@
|
||||
- 修改 `/api/external/v1` 时,同批更新 `docs/openapi/genarrative-external-v1.openapi.json` 与契约测试。
|
||||
- 修改 SpacetimeDB schema 时遵守字段追加/default 约束,同步 migration、表目录、生成绑定,并运行 schema 检查;删除、改名、重排或改类型前先确认迁移计划。
|
||||
- 日志不递归输出完整配置、应用状态或 provider client;新增字段默认不进入安全摘要。
|
||||
- HTTP 横切能力集中在 Axum/Tower 中间件:正常与降级路由复用追踪层;指标与 trace 使用 `MatchedPath` 模板及固定兜底,不把请求 ID、实际资源 ID 或 query 放入指标标签。在途请求通过 RAII guard 覆盖 Future 取消与 panic unwind;请求执行和响应体存活分别计量,不能把 handler 耗时当作 SSE 全生命周期。
|
||||
- 业务依赖在组合根显式装配,Axum `FromRef` 只抽取可浅拷贝的窄能力。项目元数据与 External API 鉴权不持有完整 `AppState`,测试经相同接口注入替代依赖。集中鉴权仍保留方法级 fallback、公开入口、MCP 和 body limit 顺序;Provider span 跳过完整参数,不隐藏计费、重试、幂等或事务规则。
|
||||
- 中文文案、注释和文档保持 UTF-8,优先局部补丁,不擅自翻译成英文。
|
||||
|
||||
## 文档生命周期
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
# 画布验收后续修复
|
||||
|
||||
状态:待排查与方案确认。承接 Issue #411;首批修复已由 PR #412 合入,基准为 master `9a21690fe`。本提交仅登记后续范围,不代表以下问题已经修复。
|
||||
|
||||
## 优先排查
|
||||
|
||||
- 动画生成持续失败:读取具体操作错误及原请求状态后再定位,不盲目重复付费生成。
|
||||
- AI 润色无变化;恢复原文退出编辑面板、再次打开丢稿。
|
||||
- 文档全文预览入口不可用或不可发现,需在当前版本复验。
|
||||
- 标签面板无法查看更多标签;资源面板多选后卡死。
|
||||
- 未完成编辑缺少继续入口;快速编辑的素材名称校验报错。
|
||||
|
||||
## 后续需求
|
||||
|
||||
- 资源卡顶部独立名称区与全类型统一样式,确认 assetName/用户素材名的正式来源。
|
||||
- 生成任务列表移到画布右上角,进行中/已完成分组、失败可见、限高滚动及自动开合。
|
||||
- 素材替换关系与画布点选替换;聊天拖拽批量引用、复制对话保留有效引用。
|
||||
|
||||
## 边界与验收
|
||||
|
||||
先核对测试包/提交,避免旧包反馈与已合入修复混淆。每个问题先复现、明确原因与最小修复,再补回归和独立 review;复杂需求按现有规范驱动流程推进。首批本地通过不等于远程 CI 或真实 Provider 验收通过,遗留 Rust shard 2 失败仍需跟踪实际 SHA。不得为验证擅自提交付费生成、部署或修改他人项目。
|
||||
|
||||
完成条件:逐项记录修复提交、实际执行的检查和客户端验收;长期规则回归主规范,全部关闭后删除本待办。
|
||||
@@ -47,6 +47,15 @@ SpacetimeDB 版本口径:当前 Rust crate `spacetimedb`、`spacetimedb-sdk`
|
||||
npm run check:server-rs-ddd
|
||||
```
|
||||
|
||||
## 依赖装配与横切能力
|
||||
|
||||
- 启动入口负责构造共享依赖,业务入口通过 Axum `State` / `FromRef` 获取所需能力。项目元数据链路与 External API 鉴权使用窄状态;窄状态不得持有完整 `AppState`、通过 `Deref` 暴露完整配置,或提供 `root_state()` 逃逸。仅在需要替换或隔离测试的外部能力处抽取接口,生产实现仍复用现有 `spacetime-client` facade,不新建数据库访问通道。
|
||||
- 后台、External API 和编辑器路由按同一种身份策略聚合鉴权,公开登录、公开文档、公开精选读取及 MCP 保持独立边界。路由、HTTP 方法、404/405、请求大小限制、授权和错误响应合同保持不变;后台 member 的实时权限校验和资源 owner 校验继续由现有权威逻辑执行。
|
||||
- HTTP 观测使用统一 Tower 追踪层与 `MatchedPath` 模板;请求计数通过 RAII 覆盖取消与 unwind,响应体存活单独统计。具体指标合同见开发运维文档。
|
||||
- Provider 函数级追踪采用现有 tracing 能力,显式列出 provider、operation、model 等非秘密字段,跳过完整参数、配置、凭据和消息正文;异步 span 覆盖实际执行与等待,不能只记录 Future 创建。装饰与追踪保持原始成功值、错误、重试次数、流式回调和取消语义。
|
||||
- 扣费、退款、幂等、资源登记、重试判定与事务边界保持显式业务流程;不引入自动扫描 IOC 容器、通用 AOP 框架或跨外部副作用的隐式事务。
|
||||
- 验收覆盖:所有受保护路由的匿名拒绝及公开入口可访问;原有 404/405 和 body limit;窄依赖可独立构造和替换;Provider 成功/失败、异步追踪归属与脱敏;已有 API、计费和幂等回归。真实本地服务 smoke 必须记录所用数据库及依赖可用性,不能用单元测试代替运行时证据。
|
||||
|
||||
## `spacetime-client` mapper 组织
|
||||
|
||||
`spacetime-client` 的 Cargo `lib.path` 指向 `src/active.rs`,现役 mapper 聚合入口是 `src/active/mapper.rs`;原旧 facade 和 mapper 已删除。
|
||||
|
||||
@@ -808,13 +808,15 @@ OpenTelemetry 现阶段默认开启 OTLP traces / metrics / logs,但本地日
|
||||
- 应用日志按进程查看:父 API 使用 `journalctl -u genarrative-api.service`,独立 BgFilter worker 使用 `journalctl -u genarrative-bgfilter-worker.service`;Nginx 日志仍写文件。日志等级继续用 `GENARRATIVE_API_LOG` / `RUST_LOG` 控制,例如 `info,tower_http=info,spacetime_client=info`。
|
||||
- debug exporter / Rider 转发都会同时接收 traces、metrics 和 logs。
|
||||
- api-server 会随 metrics 发送进程级指标:`process.memory.usage`、`process.memory.virtual`、`process.cpu.time`、`genarrative.process.cpu.usage_percent`、`process.thread.count`、`genarrative.process.memory.private`;Windows 额外发送 `process.windows.handle.count`,Linux 额外发送 `process.unix.file_descriptor.count`。这些指标只描述当前进程,不携带请求、用户或作品 label。
|
||||
- HTTP 运行态补充发送 `genarrative.http.server.response_bodies.in_flight` 与 `genarrative.http.server.request_permits.available`,后者带低基数 `pool=default|gallery|detail|admin` label,用于区分业务 handler / 背压 permit 是否仍被占用;拼图广场热点缓存补充发送 `genarrative.puzzle_gallery.cache.*` 指标,记录 fresh hit、stale hit、未命中、后台刷新开始 / 失败、重建耗时和预序列化 data JSON 字节数。
|
||||
- HTTP 请求在途指标 `http.server.active_requests` 从进入观测中间件计数,到生成 `Response` 时结束;使用 RAII guard 保证正常返回、Future 被取消释放和 panic unwind 时都按原始 method / route 标签且仅递减一次。`http.server.request.duration` 采用同一执行区间,仅记录已生成响应的请求,不表示响应已发送到客户端,也不表示 SSE 已结束。响应体存活另由 `genarrative.http.server.response_bodies.in_flight` 统计;背压 permit 由 `genarrative.http.server.request_permits.available{pool=default|admin}` 统计。
|
||||
- 正常服务与 SpacetimeDB 不可用时的降级路由复用同一 HTTP `TraceLayer` 构造函数。请求上下文在追踪层外侧,错误归一化和响应头回写在追踪层内侧,保证提前拒绝和降级响应仍记录同一 request ID 与最终状态。
|
||||
- `platform-llm` 的普通与流式调用统一生成 `llm.request` 子 span,字段白名单为 `provider`、`operation`、`api_kind`、`model`,继承调用方当前 span;范围覆盖响应读取、流式回调和异步等待,取消释放 Future 时结束。完整 client/config、API Key、请求/响应正文不进入该 span;既有 Provider 错误和重试策略保持原样,不能用遥测包装吞掉失败或自动重放。
|
||||
- 外部 API 失败统一发送 OTLP 并落库。当前 VectorEngine 图片生成 / 编辑失败由 `platform-image` provider 输出结构化日志字段,字段包括 provider、endpoint、failure_stage、status、source、source_chain、source_chain_depth、timeout、retryable、latency_ms、prompt_chars、reference_image_count、实际 provider `image_model`、request_params 和 raw_excerpt;发生模型回退时另带 `fallback_from_model` / `fallback_to_model`。图片编辑请求参数日志还会带 reference_image_bytes_total,并在 request_params.referenceImages 中记录每个 multipart `image` part 的 fileName、mimeType 和 bytes,不记录 API key 或原始图片 bytes;`api-server` 再记录指标 `genarrative.external_api.failures{provider,failure_stage,status_class,retryable}`,并写入 `tracking_event`,`event_key = external_api_call_failure`、`module_key = external-api`、`scope_kind = module`、`scope_id = provider`。调用方能拿到身份上下文时,失败事件还会在行级 `user_id` / `owner_user_id` / `profile_id` 和 `metadata_json.userId` / `metadata_json.profileId` / `metadata_json.requestId` / `metadata_json.errorSource` 中记录触发者、草稿 / 作品作用域、请求标识和传输错误链。排障时先按 provider / failureStage / imageModel 聚合,再下钻 userId / profileId,最后结合 request 日志、errorSource 和上游响应 excerpt 判断是模型不可用、限流、超时、解析失败还是未返回图片。
|
||||
- OSS 平台适配器也输出结构化日志,覆盖 `sign_post_object`、`sign_get_object_url`、`head_object` 和 `put_object`。排查资产签名、上传或确认失败时,先按 `provider=aliyun-oss` 与 `operation` 过滤,再看 `object_key` / `key_prefix`、`status`、`status_class`、`error_kind`、`content_length`、`content_type` 和 `elapsed_ms`;角色动画逐帧额外按 `frame_index`、`operation=source_put|final_put|final_head`、`attempt/max_attempts`、`will_retry`、`oss_code` 和 `oss_request_id` 对齐同一对象的请求尝试。`请求 OSS 失败` 时,`timeout/connect/transport=true` 表示传输类失败,OSS PutObject 的 `status=400, oss_code=RequestTimeout, timeout=true`、`status=429` 或 `500–599` 表示暂时性失败,PUT 的 `status=400`、`oss_code` 为空且 `timeout=true` 或 `transport=true`(message 含「错误响应体读取失败」,即 400 错误体读取超时/断流)也会重试;除这两类例外外,其他 400、401/403/404、配置、URL 和签名错误是确定性失败,不会重试。最终帧 HEAD 失败只会重试 HEAD,不会重复 PUT。日志不得包含 AccessKey、policy、signature、Authorization header、完整 signed URL 或 OSS 错误响应体;`oss_request_id` 只用于关联 OSS 服务端排障。排查 generated 图片重复下载时,先确认前端输入是否为 `/generated-*` legacy path 或可归一化的 `https://*.oss-*.aliyuncs.com/generated-*`;正确链路应先调 `/api/assets/read-url`,再由浏览器请求 signed URL,且同一路径、同一 `refreshKey` 版本和未临近过期的 signed URL 应复用。新上传 generated 私有对象应带 `Cache-Control: public, max-age=31536000, immutable`;旧对象若只有 `ETag` / `Last-Modified`,浏览器会走 304 协商缓存而不是长期强缓存,可通过刷新 OSS 元数据或 CDN 配置补齐。
|
||||
- SpacetimeDB 观测分为两类:procedure / reducer 调用继续用 `genarrative.spacetime.procedure.*`,订阅本地 cache 读使用 `genarrative.spacetime.read.*`。`read=list_puzzle_gallery` 表示拼图广场当前从 `puzzle_gallery_card_view` 本地 cache 读取,不再每个 HTTP 请求调用 `list_puzzle_gallery` procedure。
|
||||
- 本地 Windows 直连压测的内存高水位要结合 K6 VU / 连接数解释。250 RPS 下过高 `PREALLOCATED_VUS` 可能让 300 个本地 Established 连接把 `api-server` private memory 瞬时推到 GB 级,且 `/healthz` 小响应也能复现;若压测结束后回落、`response_bodies.in_flight` 和背压 permit 未显示业务积压,应优先按连接 / 发送链路高水位处理,而不是判断为 SpacetimeDB 或 JSON 缓存泄漏。
|
||||
- Rider 的 Logs 面板只展示 log event 自身字段,不会自动展开父 span 的全部 attributes;请求完成日志会直接带 `request_id`、`http.request.method`、`http.route`、`url.scheme`、`url.path`、`http.response.status_code`、`status_class`、`latency_ms` 和 `slow_request`,完整链路继续到 Traces 面板按 trace/span 查看。
|
||||
- 指标 label 只允许低基数字段:HTTP 使用 `method`、`route`、`status_class`,SpacetimeDB 调用使用 `procedure`、`status_class`;`request_id` 只进入 trace/log attribute,不进入 metric label。
|
||||
- 指标 label 只允许低基数字段:HTTP 使用 `http.request.method`、`http.route`、`status_class`,SpacetimeDB 调用使用 `procedure`、`status_class`;`request_id` 只进入 trace/log attribute,不进入 metric label。HTTP trace、完成日志与指标共用 Axum `MatchedPath` 路由模板,例如 `/api/editor/projects/{project_id}`;未匹配路由或无路由模板的降级请求只使用 `/api/*`、`/admin/api/*`、`other` 三种固定兜底,不把实际 ID 或 query 写入 route 标签。原来按 `/api/*` 聚合的监控查询应改为按路由模板汇总。
|
||||
|
||||
常见外部服务变量:
|
||||
|
||||
|
||||
Generated
+2
@@ -4110,6 +4110,8 @@ dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
use std::time::Duration;
|
||||
|
||||
use axum::{
|
||||
Router,
|
||||
body::Body,
|
||||
@@ -9,8 +11,8 @@ use axum::{
|
||||
};
|
||||
use serde_json::json;
|
||||
use tower_http::{
|
||||
classify::ServerErrorsFailureClass,
|
||||
trace::{DefaultOnRequest, TraceLayer},
|
||||
classify::{ServerErrorsAsFailures, ServerErrorsFailureClass, SharedClassifier},
|
||||
trace::{DefaultOnBodyChunk, DefaultOnEos, DefaultOnRequest, TraceLayer},
|
||||
};
|
||||
use tracing::{Level, Span, error, info_span};
|
||||
|
||||
@@ -85,66 +87,7 @@ pub fn build_router(state: AppState) -> Router {
|
||||
state.clone(),
|
||||
record_http_observability,
|
||||
))
|
||||
// 当前阶段先统一挂接 HTTP tracing,后续 request_id、响应头与错误中间件继续在这里扩展。
|
||||
.layer(
|
||||
TraceLayer::new_for_http()
|
||||
.make_span_with(|request: &Request<Body>| {
|
||||
let request_id =
|
||||
resolve_request_id(request).unwrap_or_else(|| "unknown".to_string());
|
||||
let route = crate::telemetry::observability_route(request.uri().path());
|
||||
let scheme = crate::telemetry::resolve_request_scheme(request.headers());
|
||||
let span_name = format!("{} {}", request.method(), route);
|
||||
|
||||
info_span!(
|
||||
"http.request",
|
||||
otel.kind = "server",
|
||||
otel.name = %span_name,
|
||||
otel.status_code = tracing::field::Empty,
|
||||
http.response.status_code = tracing::field::Empty,
|
||||
method = %request.method(),
|
||||
http.request.method = %request.method(),
|
||||
http.route = %route,
|
||||
url.scheme = %scheme,
|
||||
url.path = %request.uri().path(),
|
||||
request_id = %request_id,
|
||||
status = tracing::field::Empty,
|
||||
latency_ms = tracing::field::Empty,
|
||||
)
|
||||
})
|
||||
.on_request(DefaultOnRequest::new().level(Level::INFO))
|
||||
.on_response(
|
||||
|response: &axum::response::Response,
|
||||
latency: std::time::Duration,
|
||||
span: &Span| {
|
||||
let latency_ms = latency.as_millis().min(u64::MAX as u128) as u64;
|
||||
let status = response.status().as_u16();
|
||||
span.record("status", status);
|
||||
span.record("http.response.status_code", status);
|
||||
span.record(
|
||||
"otel.status_code",
|
||||
if response.status().is_server_error() {
|
||||
"ERROR"
|
||||
} else {
|
||||
"OK"
|
||||
},
|
||||
);
|
||||
span.record("latency_ms", latency_ms);
|
||||
},
|
||||
)
|
||||
.on_failure(
|
||||
|failure: ServerErrorsFailureClass,
|
||||
latency: std::time::Duration,
|
||||
span: &Span| {
|
||||
let latency_ms = latency.as_millis().min(u64::MAX as u128) as u64;
|
||||
error!(
|
||||
parent: span,
|
||||
latency_ms,
|
||||
failure = %failure,
|
||||
"http request failed"
|
||||
);
|
||||
},
|
||||
),
|
||||
)
|
||||
.layer(http_trace_layer())
|
||||
// request_id 中间件先进入请求链,确保后续 tracing、错误处理和响应头层都能复用同一份请求标识。
|
||||
.layer(middleware::from_fn(attach_request_context))
|
||||
.with_state(state)
|
||||
@@ -159,68 +102,72 @@ pub fn build_spacetime_unavailable_router(message: String) -> Router {
|
||||
// 依赖不可用模式不挂业务 state,统一返回 503,并继续保留 request_id / API 版本 / 耗时响应头。
|
||||
.layer(middleware::from_fn(normalize_error_response))
|
||||
.layer(middleware::from_fn(propagate_request_id_header))
|
||||
.layer(
|
||||
TraceLayer::new_for_http()
|
||||
.make_span_with(|request: &Request<Body>| {
|
||||
let request_id =
|
||||
resolve_request_id(request).unwrap_or_else(|| "unknown".to_string());
|
||||
let route = crate::telemetry::observability_route(request.uri().path());
|
||||
let scheme = crate::telemetry::resolve_request_scheme(request.headers());
|
||||
let span_name = format!("{} {}", request.method(), route);
|
||||
|
||||
info_span!(
|
||||
"http.request",
|
||||
otel.kind = "server",
|
||||
otel.name = %span_name,
|
||||
otel.status_code = tracing::field::Empty,
|
||||
http.response.status_code = tracing::field::Empty,
|
||||
method = %request.method(),
|
||||
http.request.method = %request.method(),
|
||||
http.route = %route,
|
||||
url.scheme = %scheme,
|
||||
url.path = %request.uri().path(),
|
||||
request_id = %request_id,
|
||||
status = tracing::field::Empty,
|
||||
latency_ms = tracing::field::Empty,
|
||||
)
|
||||
})
|
||||
.on_request(DefaultOnRequest::new().level(Level::INFO))
|
||||
.on_response(
|
||||
|response: &axum::response::Response,
|
||||
latency: std::time::Duration,
|
||||
span: &Span| {
|
||||
let latency_ms = latency.as_millis().min(u64::MAX as u128) as u64;
|
||||
let status = response.status().as_u16();
|
||||
span.record("status", status);
|
||||
span.record("http.response.status_code", status);
|
||||
span.record(
|
||||
"otel.status_code",
|
||||
if response.status().is_server_error() {
|
||||
"ERROR"
|
||||
} else {
|
||||
"OK"
|
||||
},
|
||||
);
|
||||
span.record("latency_ms", latency_ms);
|
||||
},
|
||||
)
|
||||
.on_failure(
|
||||
|failure: ServerErrorsFailureClass,
|
||||
latency: std::time::Duration,
|
||||
span: &Span| {
|
||||
let latency_ms = latency.as_millis().min(u64::MAX as u128) as u64;
|
||||
error!(
|
||||
parent: span,
|
||||
latency_ms,
|
||||
failure = %failure,
|
||||
"http request failed"
|
||||
);
|
||||
},
|
||||
),
|
||||
)
|
||||
.layer(http_trace_layer())
|
||||
.layer(middleware::from_fn(attach_request_context))
|
||||
}
|
||||
|
||||
type HttpTraceLayer = TraceLayer<
|
||||
SharedClassifier<ServerErrorsAsFailures>,
|
||||
fn(&Request<Body>) -> Span,
|
||||
DefaultOnRequest,
|
||||
fn(&Response, Duration, &Span),
|
||||
DefaultOnBodyChunk,
|
||||
DefaultOnEos,
|
||||
fn(ServerErrorsFailureClass, Duration, &Span),
|
||||
>;
|
||||
|
||||
fn http_trace_layer() -> HttpTraceLayer {
|
||||
TraceLayer::new_for_http()
|
||||
.make_span_with(make_http_request_span as fn(&Request<Body>) -> Span)
|
||||
.on_request(DefaultOnRequest::new().level(Level::INFO))
|
||||
.on_response(record_http_response_span as fn(&Response, Duration, &Span))
|
||||
.on_failure(record_http_failure as fn(ServerErrorsFailureClass, Duration, &Span))
|
||||
}
|
||||
|
||||
fn make_http_request_span(request: &Request<Body>) -> Span {
|
||||
let request_id = resolve_request_id(request).unwrap_or_else(|| "unknown".to_string());
|
||||
let route = crate::telemetry::observability_route(request);
|
||||
let scheme = crate::telemetry::resolve_request_scheme(request.headers());
|
||||
let span_name = format!("{} {}", request.method(), route);
|
||||
|
||||
info_span!(
|
||||
"http.request",
|
||||
otel.kind = "server",
|
||||
otel.name = %span_name,
|
||||
otel.status_code = tracing::field::Empty,
|
||||
http.response.status_code = tracing::field::Empty,
|
||||
method = %request.method(),
|
||||
http.request.method = %request.method(),
|
||||
http.route = %route,
|
||||
url.scheme = %scheme,
|
||||
url.path = %request.uri().path(),
|
||||
request_id = %request_id,
|
||||
status = tracing::field::Empty,
|
||||
latency_ms = tracing::field::Empty,
|
||||
)
|
||||
}
|
||||
|
||||
fn record_http_response_span(response: &Response, latency: Duration, span: &Span) {
|
||||
let latency_ms = latency.as_millis().min(u64::MAX as u128) as u64;
|
||||
let status = response.status().as_u16();
|
||||
span.record("status", status);
|
||||
span.record("http.response.status_code", status);
|
||||
span.record(
|
||||
"otel.status_code",
|
||||
if response.status().is_server_error() {
|
||||
"ERROR"
|
||||
} else {
|
||||
"OK"
|
||||
},
|
||||
);
|
||||
span.record("latency_ms", latency_ms);
|
||||
}
|
||||
|
||||
fn record_http_failure(failure: ServerErrorsFailureClass, latency: Duration, span: &Span) {
|
||||
let latency_ms = latency.as_millis().min(u64::MAX as u128) as u64;
|
||||
error!(parent: span, latency_ms, failure = %failure, "http request failed");
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct SpacetimeUnavailableState {
|
||||
message: std::sync::Arc<str>,
|
||||
@@ -308,6 +255,201 @@ mod tests {
|
||||
const TEST_PASSWORD: &str = "secret123";
|
||||
const INTERNAL_TEST_SECRET: &str = "test-internal-secret";
|
||||
|
||||
mod http_tracing {
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
|
||||
use axum::extract::FromRef;
|
||||
use tracing::{
|
||||
Metadata, Subscriber,
|
||||
field::{Field, Visit},
|
||||
instrument::WithSubscriber,
|
||||
span::{Attributes, Id, Record},
|
||||
};
|
||||
|
||||
use super::*;
|
||||
use crate::state::{BackpressureState, HttpRequestPermitPoolKind};
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct HttpSpanCapture(Arc<Mutex<Vec<BTreeMap<String, String>>>>);
|
||||
|
||||
struct SpanFields<'a>(&'a mut BTreeMap<String, String>);
|
||||
|
||||
impl Visit for SpanFields<'_> {
|
||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||
self.0
|
||||
.insert(field.name().to_string(), format!("{value:?}"));
|
||||
}
|
||||
|
||||
fn record_str(&mut self, field: &Field, value: &str) {
|
||||
self.0.insert(field.name().to_string(), value.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
impl Subscriber for HttpSpanCapture {
|
||||
fn enabled(&self, metadata: &Metadata<'_>) -> bool {
|
||||
metadata.is_span() && metadata.name() == "http.request"
|
||||
}
|
||||
|
||||
fn new_span(&self, attributes: &Attributes<'_>) -> Id {
|
||||
let mut spans = self.0.lock().expect("span capture should lock");
|
||||
let mut fields = BTreeMap::new();
|
||||
attributes.record(&mut SpanFields(&mut fields));
|
||||
spans.push(fields);
|
||||
Id::from_u64(spans.len() as u64)
|
||||
}
|
||||
|
||||
fn record(&self, span: &Id, values: &Record<'_>) {
|
||||
let mut spans = self.0.lock().expect("span capture should lock");
|
||||
values.record(&mut SpanFields(&mut spans[span.into_u64() as usize - 1]));
|
||||
}
|
||||
|
||||
fn record_follows_from(&self, _: &Id, _: &Id) {}
|
||||
fn event(&self, _: &tracing::Event<'_>) {}
|
||||
fn enter(&self, _: &Id) {}
|
||||
fn exit(&self, _: &Id) {}
|
||||
}
|
||||
|
||||
async fn assert_rejection_observed(
|
||||
app: Router,
|
||||
request: Request<Body>,
|
||||
expected_status: StatusCode,
|
||||
expected_route: &str,
|
||||
expected_code: &str,
|
||||
) -> (Value, String) {
|
||||
let expected_request_id = request.headers().get("x-request-id").cloned();
|
||||
let path = request.uri().path().to_string();
|
||||
let capture = HttpSpanCapture::default();
|
||||
let response = app
|
||||
.oneshot(request)
|
||||
.with_subscriber(capture.clone())
|
||||
.await
|
||||
.expect("rejected request should complete");
|
||||
|
||||
assert_eq!(response.status(), expected_status);
|
||||
let request_id = response.headers()["x-request-id"]
|
||||
.to_str()
|
||||
.expect("request id should be a header string")
|
||||
.to_string();
|
||||
assert!(!request_id.is_empty());
|
||||
assert_ne!(request_id, "unknown");
|
||||
if let Some(expected_request_id) = expected_request_id {
|
||||
assert_eq!(response.headers()["x-request-id"], expected_request_id);
|
||||
}
|
||||
for name in ["x-api-version", "x-route-version"] {
|
||||
assert_eq!(response.headers()[name], shared_contracts::api::API_VERSION);
|
||||
}
|
||||
assert!(
|
||||
response.headers()["x-response-time-ms"]
|
||||
.to_str()
|
||||
.unwrap()
|
||||
.parse::<u64>()
|
||||
.is_ok()
|
||||
);
|
||||
if expected_status == StatusCode::TOO_MANY_REQUESTS {
|
||||
assert_eq!(response.headers()["retry-after"], "1");
|
||||
}
|
||||
let payload = read_json_response(response).await;
|
||||
assert_eq!(payload["error"]["code"], expected_code);
|
||||
|
||||
let spans = capture.0.lock().expect("span capture should lock");
|
||||
assert_eq!(
|
||||
spans.len(),
|
||||
1,
|
||||
"each rejected request should have one HTTP span"
|
||||
);
|
||||
let fields = &spans[0];
|
||||
assert_eq!(fields["request_id"], request_id);
|
||||
assert_eq!(fields["http.route"], expected_route);
|
||||
assert_eq!(fields["otel.name"], format!("GET {expected_route}"));
|
||||
assert_eq!(fields["url.path"], path);
|
||||
assert_eq!(fields["http.request.method"], "GET");
|
||||
assert_eq!(fields["status"], expected_status.as_u16().to_string());
|
||||
assert_eq!(
|
||||
fields["http.response.status_code"],
|
||||
expected_status.as_u16().to_string()
|
||||
);
|
||||
assert_eq!(
|
||||
fields["otel.status_code"],
|
||||
if expected_status.is_server_error() {
|
||||
"ERROR"
|
||||
} else {
|
||||
"OK"
|
||||
}
|
||||
);
|
||||
assert!(fields["latency_ms"].parse::<u64>().is_ok());
|
||||
(payload, request_id)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn auth_rejections_share_the_matched_route_template() {
|
||||
let app =
|
||||
build_router(AppState::new(AppConfig::default()).expect("state should build"));
|
||||
for project_id in ["project-one", "project-two"] {
|
||||
assert_rejection_observed(
|
||||
app.clone(),
|
||||
Request::builder()
|
||||
.uri(format!(
|
||||
"/api/editor/projects/{project_id}/agent-conversations"
|
||||
))
|
||||
.header("x-request-id", format!("req-trace-{project_id}"))
|
||||
.body(Body::empty())
|
||||
.expect("request should build"),
|
||||
StatusCode::UNAUTHORIZED,
|
||||
"/api/editor/projects/{project_id}/agent-conversations",
|
||||
"UNAUTHORIZED",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn backpressure_rejection_keeps_generated_context_and_headers() {
|
||||
let config = AppConfig {
|
||||
max_concurrent_requests: Some(1),
|
||||
..AppConfig::default()
|
||||
};
|
||||
let state = AppState::new(config).expect("state should build");
|
||||
let (_, pool) = BackpressureState::from_ref(&state)
|
||||
.request_permit_pool(HttpRequestPermitPoolKind::Default)
|
||||
.expect("default request pool should exist");
|
||||
let _held_permit = pool
|
||||
.try_acquire_owned()
|
||||
.expect("pool should have one permit");
|
||||
|
||||
let (payload, request_id) = assert_rejection_observed(
|
||||
build_router(state),
|
||||
Request::builder()
|
||||
.uri("/api/editor/projects/project-one/agent-conversations")
|
||||
.body(Body::empty())
|
||||
.expect("request should build"),
|
||||
StatusCode::TOO_MANY_REQUESTS,
|
||||
"/api/editor/projects/{project_id}/agent-conversations",
|
||||
"TOO_MANY_REQUESTS",
|
||||
)
|
||||
.await;
|
||||
assert_eq!(payload["meta"]["requestId"], request_id);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unavailable_router_rejection_keeps_generated_context_and_headers() {
|
||||
let (payload, request_id) = assert_rejection_observed(
|
||||
build_spacetime_unavailable_router("test unavailable".to_string()),
|
||||
Request::builder()
|
||||
.uri("/api/auth/login-options")
|
||||
.body(Body::empty())
|
||||
.expect("request should build"),
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"/api/*",
|
||||
"SERVICE_UNAVAILABLE",
|
||||
)
|
||||
.await;
|
||||
assert_eq!(payload["meta"]["requestId"], request_id);
|
||||
}
|
||||
}
|
||||
|
||||
async fn seed_phone_user_with_password(
|
||||
state: &AppState,
|
||||
phone_number: &str,
|
||||
|
||||
@@ -12,7 +12,7 @@ use std::{
|
||||
|
||||
use axum::{
|
||||
Json,
|
||||
extract::{Extension, Path, Query, State, rejection::JsonRejection},
|
||||
extract::{Extension, FromRef, Path, Query, State, rejection::JsonRejection},
|
||||
http::{HeaderMap, HeaderValue, StatusCode},
|
||||
};
|
||||
use module_assets::{
|
||||
@@ -121,7 +121,7 @@ use crate::{
|
||||
platform_errors::map_oss_error,
|
||||
prompt::editor_scene::build_editor_scene_prompt,
|
||||
request_context::RequestContext,
|
||||
state::AppState,
|
||||
state::{AppState, EditorMediaStorageState, EditorProjectState},
|
||||
work_author::{ORPHAN_WORK_AUTHOR_PUBLIC_USER_CODE, ORPHAN_WORK_OWNER_USER_ID},
|
||||
};
|
||||
|
||||
@@ -1892,19 +1892,19 @@ pub struct EditorAssetPayload {
|
||||
}
|
||||
|
||||
pub async fn load_recent_editor_project(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
) -> Result<Json<Value>, AppError> {
|
||||
let owner_user_id = authenticated.claims().user_id().to_string();
|
||||
let project = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.get_recent_editor_project(owner_user_id)
|
||||
.await
|
||||
.map_err(map_editor_project_error)?;
|
||||
let project = match project {
|
||||
Some(project) => Some(editor_project_payload_from_record(
|
||||
repair_editor_project_record_inline_media(&state, project).await,
|
||||
state.repair_project_media(project).await,
|
||||
)),
|
||||
None => None,
|
||||
};
|
||||
@@ -1930,20 +1930,20 @@ pub async fn get_editor_generation_pricing(
|
||||
}
|
||||
|
||||
pub async fn list_editor_projects(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
) -> Result<Json<Value>, AppError> {
|
||||
let owner_user_id = authenticated.claims().user_id().to_string();
|
||||
let project_records = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.list_editor_projects(owner_user_id)
|
||||
.await
|
||||
.map_err(map_editor_project_error)?;
|
||||
let mut projects = Vec::with_capacity(project_records.len());
|
||||
for project in project_records {
|
||||
projects.push(editor_project_payload_from_record(
|
||||
repair_editor_project_record_inline_media(&state, project).await,
|
||||
state.repair_project_media(project).await,
|
||||
));
|
||||
}
|
||||
|
||||
@@ -2048,7 +2048,7 @@ fn parse_editor_timestamp_micros(value: &str) -> Option<i64> {
|
||||
}
|
||||
|
||||
pub async fn create_editor_project(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
headers: HeaderMap,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
@@ -2072,7 +2072,7 @@ pub async fn create_editor_project(
|
||||
})
|
||||
.unwrap_or_else(|| build_prefixed_uuid_id(EDITOR_PROJECT_ID_PREFIX));
|
||||
let project = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.create_editor_project(EditorProjectCreateRecordInput {
|
||||
project_id,
|
||||
owner_user_id,
|
||||
@@ -2091,20 +2091,20 @@ pub async fn create_editor_project(
|
||||
}
|
||||
|
||||
pub async fn get_editor_project(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
Path(project_id): Path<String>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
) -> Result<Json<Value>, AppError> {
|
||||
let project = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.get_editor_project(EditorProjectGetRecordInput {
|
||||
project_id,
|
||||
owner_user_id: authenticated.claims().user_id().to_string(),
|
||||
})
|
||||
.await
|
||||
.map_err(map_editor_project_error)?;
|
||||
let project = repair_editor_project_record_inline_media(&state, project).await;
|
||||
let project = state.repair_project_media(project).await;
|
||||
|
||||
Ok(json_success_body(
|
||||
Some(&request_context),
|
||||
@@ -2115,7 +2115,7 @@ pub async fn get_editor_project(
|
||||
}
|
||||
|
||||
pub async fn save_editor_project_layout(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
Path(project_id): Path<String>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
@@ -2127,7 +2127,7 @@ pub async fn save_editor_project_layout(
|
||||
let owner_user_id = authenticated.claims().user_id().to_string();
|
||||
let updated_at_micros = current_utc_micros();
|
||||
let ack = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.save_editor_project_layout_v2_ack(EditorProjectLayoutSaveV2RecordInput {
|
||||
project_id,
|
||||
owner_user_id,
|
||||
@@ -2151,14 +2151,14 @@ pub async fn save_editor_project_layout(
|
||||
}
|
||||
|
||||
pub async fn rename_editor_project(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
Path(project_id): Path<String>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
Json(payload): Json<EditorProjectRenameRequest>,
|
||||
) -> Result<Json<Value>, AppError> {
|
||||
let project = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.rename_editor_project(EditorProjectRenameRecordInput {
|
||||
project_id,
|
||||
owner_user_id: authenticated.claims().user_id().to_string(),
|
||||
@@ -2177,13 +2177,13 @@ pub async fn rename_editor_project(
|
||||
}
|
||||
|
||||
pub async fn delete_editor_project(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<EditorProjectState>,
|
||||
Path(project_id): Path<String>,
|
||||
Extension(request_context): Extension<RequestContext>,
|
||||
Extension(authenticated): Extension<AuthenticatedAccessToken>,
|
||||
) -> Result<Json<Value>, AppError> {
|
||||
let deleted_project_id = state
|
||||
.spacetime_client()
|
||||
.projects()
|
||||
.delete_editor_project(EditorProjectDeleteRecordInput {
|
||||
project_id,
|
||||
owner_user_id: authenticated.claims().user_id().to_string(),
|
||||
@@ -7788,7 +7788,7 @@ pub async fn snap_editor_image_to_pixel_art(
|
||||
}
|
||||
result = async move {
|
||||
let mut uploaded = upload_editor_generated_image_object_prepared(
|
||||
&state,
|
||||
&EditorMediaStorageState::from_ref(&state),
|
||||
owner_user_id.as_str(),
|
||||
persistence_identity.task_id.as_str(),
|
||||
prepared_upload,
|
||||
@@ -8980,8 +8980,8 @@ pub(crate) async fn extract_editor_ui_design_assets_for_owner(
|
||||
))
|
||||
}
|
||||
|
||||
async fn repair_editor_project_record_inline_media(
|
||||
state: &AppState,
|
||||
pub(crate) async fn repair_editor_project_record_inline_media(
|
||||
state: &EditorMediaStorageState,
|
||||
mut record: EditorProjectRecord,
|
||||
) -> EditorProjectRecord {
|
||||
let owner_user_id = record.owner_user_id.clone();
|
||||
@@ -9010,7 +9010,7 @@ async fn repair_editor_asset_library_inline_media(
|
||||
}
|
||||
|
||||
async fn repair_editor_project_resource_inline_media(
|
||||
state: &AppState,
|
||||
state: &EditorMediaStorageState,
|
||||
owner_user_id: &str,
|
||||
resource: EditorProjectResourceRecord,
|
||||
) -> EditorProjectResourceRecord {
|
||||
@@ -9092,7 +9092,7 @@ async fn repair_editor_asset_inline_media(
|
||||
}
|
||||
|
||||
let repair = persist_editor_legacy_inline_image(
|
||||
state,
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
owner_user_id,
|
||||
asset.image_src.as_str(),
|
||||
asset.task_id.as_deref().unwrap_or(asset.asset_id.as_str()),
|
||||
@@ -9146,7 +9146,7 @@ async fn repair_editor_asset_inline_media(
|
||||
}
|
||||
|
||||
async fn persist_editor_legacy_inline_image(
|
||||
state: &AppState,
|
||||
state: &EditorMediaStorageState,
|
||||
owner_user_id: &str,
|
||||
image_src: &str,
|
||||
task_id: &str,
|
||||
@@ -9167,11 +9167,14 @@ async fn persist_editor_legacy_inline_image(
|
||||
mime_type: decoded.format.mime_type,
|
||||
extension: decoded.format.extension,
|
||||
};
|
||||
persist_editor_generated_image(
|
||||
persist_editor_generated_image_data(
|
||||
state,
|
||||
owner_user_id,
|
||||
task_id,
|
||||
&image,
|
||||
GeneratedImageAssetDataUrl {
|
||||
format: normalize_generated_image_asset_mime(image.mime_type.as_str()),
|
||||
bytes: image.bytes,
|
||||
},
|
||||
prompt,
|
||||
actual_prompt,
|
||||
asset_kind.unwrap_or(EDITOR_LEGACY_INLINE_IMAGE_ASSET_KIND),
|
||||
@@ -10441,7 +10444,11 @@ pub(crate) async fn complete_editor_canvas_generation(
|
||||
})
|
||||
.await
|
||||
.map_err(map_editor_project_error)?;
|
||||
let project = repair_editor_project_record_inline_media(state, project).await;
|
||||
let project = repair_editor_project_record_inline_media(
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
project,
|
||||
)
|
||||
.await;
|
||||
let expected_revision = project.canvas.revision;
|
||||
let viewport = project.viewport.clone();
|
||||
let project_payload = editor_project_payload_from_record(project);
|
||||
@@ -10717,7 +10724,11 @@ pub(crate) async fn prepare_editor_canvas_generation_layout(
|
||||
})
|
||||
.await
|
||||
.map_err(map_editor_project_error)?;
|
||||
let project = repair_editor_project_record_inline_media(state, project).await;
|
||||
let project = repair_editor_project_record_inline_media(
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
project,
|
||||
)
|
||||
.await;
|
||||
let expected_revision = project.canvas.revision;
|
||||
let viewport = project.viewport.clone();
|
||||
let project_payload = editor_project_payload_from_record(project);
|
||||
@@ -10810,7 +10821,11 @@ async fn prepare_editor_canvas_background_removal_layout(
|
||||
})
|
||||
.await
|
||||
.map_err(map_editor_project_error)?;
|
||||
let project = repair_editor_project_record_inline_media(state, project).await;
|
||||
let project = repair_editor_project_record_inline_media(
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
project,
|
||||
)
|
||||
.await;
|
||||
let expected_revision = project.canvas.revision;
|
||||
let viewport = project.viewport.clone();
|
||||
let project_payload = editor_project_payload_from_record(project);
|
||||
@@ -12157,7 +12172,7 @@ pub(crate) async fn persist_editor_generated_image(
|
||||
provider: &str,
|
||||
) -> Result<PersistedEditorGeneratedImage, AppError> {
|
||||
persist_editor_generated_image_data(
|
||||
state,
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
owner_user_id,
|
||||
task_id,
|
||||
GeneratedImageAssetDataUrl {
|
||||
@@ -12196,7 +12211,7 @@ pub(crate) async fn prepare_editor_generated_image(
|
||||
}))
|
||||
})?;
|
||||
let uploaded = upload_editor_generated_image_object_data(
|
||||
state,
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
caller.owner_user_id.as_str(),
|
||||
task_id,
|
||||
GeneratedImageAssetDataUrl {
|
||||
@@ -12241,7 +12256,7 @@ async fn persist_editor_generated_image_owned(
|
||||
bytes: image.bytes,
|
||||
};
|
||||
persist_editor_generated_image_data(
|
||||
state,
|
||||
&EditorMediaStorageState::from_ref(state),
|
||||
owner_user_id,
|
||||
task_id,
|
||||
image_data,
|
||||
@@ -12257,7 +12272,7 @@ async fn persist_editor_generated_image_owned(
|
||||
}
|
||||
|
||||
async fn persist_editor_generated_image_data(
|
||||
state: &AppState,
|
||||
state: &EditorMediaStorageState,
|
||||
owner_user_id: &str,
|
||||
task_id: &str,
|
||||
image: GeneratedImageAssetDataUrl,
|
||||
@@ -12304,7 +12319,7 @@ async fn persist_editor_generated_image_data(
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn upload_editor_generated_image_object_data(
|
||||
state: &AppState,
|
||||
state: &EditorMediaStorageState,
|
||||
owner_user_id: &str,
|
||||
task_id: &str,
|
||||
image: GeneratedImageAssetDataUrl,
|
||||
@@ -12340,7 +12355,7 @@ async fn upload_editor_generated_image_object_data(
|
||||
}
|
||||
|
||||
async fn upload_editor_generated_image_object_prepared(
|
||||
state: &AppState,
|
||||
state: &EditorMediaStorageState,
|
||||
owner_user_id: &str,
|
||||
task_id: &str,
|
||||
prepared: GeneratedImageAssetPreparedPut,
|
||||
@@ -13713,6 +13728,9 @@ pub(crate) fn current_utc_micros() -> i64 {
|
||||
i64::try_from(duration.as_micros()).expect("current unix micros should fit in i64")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod metadata_tests;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -0,0 +1,364 @@
|
||||
use super::*;
|
||||
use crate::{
|
||||
request_context::attach_request_context,
|
||||
state::project_metadata::{EditorProjectMediaRepair, EditorProjectRepository},
|
||||
};
|
||||
use axum::{
|
||||
Router,
|
||||
body::{Body, to_bytes},
|
||||
http::Request,
|
||||
middleware,
|
||||
routing::{get, patch},
|
||||
};
|
||||
use futures_util::future::BoxFuture;
|
||||
use platform_auth::{AccessTokenClaims, AuthProvider, BindingStatus};
|
||||
use spacetime_client::EditorProjectLayoutSaveV2AckRecord;
|
||||
use std::sync::Mutex;
|
||||
use tower::ServiceExt;
|
||||
|
||||
#[derive(Default)]
|
||||
struct RecordingProjects {
|
||||
calls: Mutex<Vec<(&'static str, Value)>>,
|
||||
error: Option<&'static str>,
|
||||
}
|
||||
|
||||
impl RecordingProjects {
|
||||
fn finish<T: Send + 'static>(
|
||||
&self,
|
||||
operation: &'static str,
|
||||
input: Value,
|
||||
result: T,
|
||||
) -> BoxFuture<'_, Result<T, SpacetimeClientError>> {
|
||||
self.calls.lock().unwrap().push((operation, input));
|
||||
Box::pin(async move {
|
||||
match self.error {
|
||||
Some(message) => Err(SpacetimeClientError::Procedure(message.to_string())),
|
||||
None => Ok(result),
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn project_record(project_id: &str, owner: &str, title: &str) -> EditorProjectRecord {
|
||||
let viewport = EditorCanvasViewportRecord {
|
||||
x: 1.0,
|
||||
y: 2.0,
|
||||
scale: 1.0,
|
||||
};
|
||||
EditorProjectRecord {
|
||||
project_id: project_id.to_string(),
|
||||
owner_user_id: owner.to_string(),
|
||||
title: title.to_string(),
|
||||
canvas: EditorCanvasRecord {
|
||||
canvas_id: "canvas-fixture".to_string(),
|
||||
project_id: project_id.to_string(),
|
||||
title: title.to_string(),
|
||||
viewport: viewport.clone(),
|
||||
layers: json!([]),
|
||||
revision: 7,
|
||||
layout_storage_version: 2,
|
||||
background_color: None,
|
||||
created_at: "0.000000Z".to_string(),
|
||||
updated_at: "0.000000Z".to_string(),
|
||||
},
|
||||
viewport,
|
||||
layers: json!([]),
|
||||
resources: vec![],
|
||||
created_at: "0.000000Z".to_string(),
|
||||
updated_at: "0.000000Z".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
impl EditorProjectRepository for RecordingProjects {
|
||||
fn get_recent_editor_project(
|
||||
&self,
|
||||
owner_user_id: String,
|
||||
) -> BoxFuture<'_, Result<Option<EditorProjectRecord>, SpacetimeClientError>> {
|
||||
let record = project_record("project-fixture", &owner_user_id, "原名称");
|
||||
self.finish("recent", json!({"owner": owner_user_id}), Some(record))
|
||||
}
|
||||
|
||||
fn list_editor_projects(
|
||||
&self,
|
||||
owner_user_id: String,
|
||||
) -> BoxFuture<'_, Result<Vec<EditorProjectRecord>, SpacetimeClientError>> {
|
||||
let record = project_record("project-fixture", &owner_user_id, "原名称");
|
||||
self.finish("list", json!({"owner": owner_user_id}), vec![record])
|
||||
}
|
||||
|
||||
fn get_editor_project(
|
||||
&self,
|
||||
input: EditorProjectGetRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>> {
|
||||
let record = project_record(&input.project_id, &input.owner_user_id, "原名称");
|
||||
self.finish(
|
||||
"get",
|
||||
json!({"owner": input.owner_user_id, "project": input.project_id}),
|
||||
record,
|
||||
)
|
||||
}
|
||||
|
||||
fn create_editor_project(
|
||||
&self,
|
||||
input: EditorProjectCreateRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>> {
|
||||
let record = project_record(&input.project_id, &input.owner_user_id, &input.title);
|
||||
self.finish("create", json!({"owner": input.owner_user_id, "project": input.project_id, "title": input.title, "at": input.now_micros}), record)
|
||||
}
|
||||
|
||||
fn rename_editor_project(
|
||||
&self,
|
||||
input: EditorProjectRenameRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>> {
|
||||
let record = project_record(&input.project_id, &input.owner_user_id, &input.title);
|
||||
self.finish("rename", json!({"owner": input.owner_user_id, "project": input.project_id, "title": input.title, "at": input.updated_at_micros}), record)
|
||||
}
|
||||
|
||||
fn delete_editor_project(
|
||||
&self,
|
||||
input: EditorProjectDeleteRecordInput,
|
||||
) -> BoxFuture<'_, Result<String, SpacetimeClientError>> {
|
||||
self.finish(
|
||||
"delete",
|
||||
json!({"owner": input.owner_user_id, "project": input.project_id}),
|
||||
input.project_id,
|
||||
)
|
||||
}
|
||||
|
||||
fn save_editor_project_layout_v2_ack(
|
||||
&self,
|
||||
input: EditorProjectLayoutSaveV2RecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectLayoutSaveV2AckRecord, SpacetimeClientError>> {
|
||||
let ack = EditorProjectLayoutSaveV2AckRecord {
|
||||
project_id: input.project_id.clone(),
|
||||
canvas_id: "canvas-fixture".to_string(),
|
||||
revision: input.expected_revision + 1,
|
||||
updated_at: "1.000000Z".to_string(),
|
||||
updated_at_micros: input.updated_at_micros,
|
||||
};
|
||||
self.finish("layout", json!({"owner": input.owner_user_id, "project": input.project_id, "revision": input.expected_revision, "layers": input.layers_json, "viewport": input.viewport}), ack)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct RecordingMediaRepair {
|
||||
calls: Mutex<Vec<(String, String)>>,
|
||||
}
|
||||
|
||||
impl EditorProjectMediaRepair for RecordingMediaRepair {
|
||||
fn repair_project(
|
||||
&self,
|
||||
mut project: EditorProjectRecord,
|
||||
) -> BoxFuture<'_, EditorProjectRecord> {
|
||||
self.calls
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push((project.project_id.clone(), project.owner_user_id.clone()));
|
||||
Box::pin(async move {
|
||||
project.updated_at = "media-repaired".to_string();
|
||||
project
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn authenticated_owner() -> AuthenticatedAccessToken {
|
||||
AuthenticatedAccessToken::new(AccessTokenClaims {
|
||||
iss: "fixture".to_string(),
|
||||
sub: "owner-from-token".to_string(),
|
||||
sid: "fixture-session".to_string(),
|
||||
provider: AuthProvider::Password,
|
||||
roles: vec!["user".to_string()],
|
||||
ver: 1,
|
||||
phone_verified: true,
|
||||
binding_status: BindingStatus::Active,
|
||||
display_name: None,
|
||||
device: None,
|
||||
iat: 1,
|
||||
exp: u64::MAX,
|
||||
})
|
||||
}
|
||||
|
||||
fn metadata_router(projects: Arc<RecordingProjects>, media: Arc<RecordingMediaRepair>) -> Router {
|
||||
// 直接以窄状态组装实际 handler,不构造 AppConfig、连接数据库或初始化其它平台。
|
||||
Router::new()
|
||||
.route("/projects/recent", get(load_recent_editor_project))
|
||||
.route(
|
||||
"/projects",
|
||||
get(list_editor_projects).post(create_editor_project),
|
||||
)
|
||||
.route(
|
||||
"/projects/{project_id}",
|
||||
get(get_editor_project)
|
||||
.patch(save_editor_project_layout)
|
||||
.delete(delete_editor_project),
|
||||
)
|
||||
.route(
|
||||
"/projects/{project_id}/metadata",
|
||||
patch(rename_editor_project),
|
||||
)
|
||||
.layer(Extension(authenticated_owner()))
|
||||
.layer(middleware::from_fn(attach_request_context))
|
||||
.with_state(EditorProjectState::new(projects, media))
|
||||
}
|
||||
|
||||
async fn send(router: &Router, method: &str, path: &str, body: Value) -> (StatusCode, Value) {
|
||||
let response = router
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.method(method)
|
||||
.uri(path)
|
||||
.header("content-type", "application/json")
|
||||
.header("idempotency-key", "metadata-fixture")
|
||||
.body(Body::from(body.to_string()))
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let status = response.status();
|
||||
let body = to_bytes(response.into_body(), 65_536).await.unwrap();
|
||||
(status, serde_json::from_slice(&body).unwrap())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn narrow_project_reads_preserve_owner_and_use_media_repair_results() {
|
||||
let projects = Arc::new(RecordingProjects::default());
|
||||
let media = Arc::new(RecordingMediaRepair::default());
|
||||
let router = metadata_router(projects.clone(), media.clone());
|
||||
for (path, result_path) in [
|
||||
("/projects/recent", "/project"),
|
||||
("/projects", "/projects/0"),
|
||||
("/projects/project-fixture", "/project"),
|
||||
] {
|
||||
let (status, body) = send(&router, "GET", path, Value::Null).await;
|
||||
assert_eq!(status, StatusCode::OK);
|
||||
let result = body.pointer(result_path).unwrap();
|
||||
assert_eq!(result["projectId"], "project-fixture");
|
||||
assert_eq!(result["updatedAt"], "media-repaired");
|
||||
}
|
||||
let calls = projects.calls.lock().unwrap();
|
||||
assert_eq!(
|
||||
calls.iter().map(|call| call.0).collect::<Vec<_>>(),
|
||||
["recent", "list", "get"]
|
||||
);
|
||||
assert!(
|
||||
calls
|
||||
.iter()
|
||||
.all(|call| call.1["owner"] == "owner-from-token")
|
||||
);
|
||||
assert_eq!(
|
||||
*media.calls.lock().unwrap(),
|
||||
vec![
|
||||
(
|
||||
"project-fixture".to_string(),
|
||||
"owner-from-token".to_string()
|
||||
);
|
||||
3
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn narrow_project_writes_preserve_owner_idempotency_and_revision() {
|
||||
let projects = Arc::new(RecordingProjects::default());
|
||||
let media = Arc::new(RecordingMediaRepair::default());
|
||||
let router = metadata_router(projects.clone(), media.clone());
|
||||
let (status, created) = send(
|
||||
&router,
|
||||
"POST",
|
||||
"/projects",
|
||||
json!({"title": " 测试项目 ", "ownerUserId": "untrusted-owner"}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(status, StatusCode::OK);
|
||||
let project_id = editor_idempotent_create_id(
|
||||
EDITOR_PROJECT_ID_PREFIX,
|
||||
"owner-from-token",
|
||||
"internal-editor-project",
|
||||
"metadata-fixture",
|
||||
);
|
||||
assert_eq!(created["project"]["projectId"], project_id);
|
||||
assert_eq!(created["project"]["title"], "测试项目");
|
||||
|
||||
let (status, renamed) = send(
|
||||
&router,
|
||||
"PATCH",
|
||||
"/projects/project-fixture/metadata",
|
||||
json!({"title": "新名称"}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(status, StatusCode::OK);
|
||||
assert_eq!(renamed["project"]["title"], "新名称");
|
||||
|
||||
let (status, saved) = send(
|
||||
&router,
|
||||
"PATCH",
|
||||
"/projects/project-fixture",
|
||||
json!({"viewport": {"x": 10, "y": 20, "scale": 2}, "layers": [], "expectedRevision": 7}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(status, StatusCode::OK);
|
||||
assert_eq!(saved["revision"], 8);
|
||||
assert_eq!(saved["projectId"], "project-fixture");
|
||||
|
||||
let (status, deleted) = send(&router, "DELETE", "/projects/project-fixture", Value::Null).await;
|
||||
assert_eq!(status, StatusCode::OK);
|
||||
assert_eq!(deleted["deletedProjectId"], "project-fixture");
|
||||
let calls = projects.calls.lock().unwrap();
|
||||
assert_eq!(
|
||||
calls.iter().map(|call| call.0).collect::<Vec<_>>(),
|
||||
["create", "rename", "layout", "delete"]
|
||||
);
|
||||
assert!(
|
||||
calls
|
||||
.iter()
|
||||
.all(|call| call.1["owner"] == "owner-from-token")
|
||||
);
|
||||
assert_eq!(calls[0].1["project"], project_id);
|
||||
assert_eq!(calls[2].1["revision"], 7);
|
||||
assert_eq!(calls[2].1["layers"], "[]");
|
||||
assert_eq!(
|
||||
calls[2].1["viewport"],
|
||||
json!({"x": 10.0, "y": 20.0, "scale": 2.0})
|
||||
);
|
||||
assert!(media.calls.lock().unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn narrow_project_handlers_propagate_facade_errors_without_media_side_effects() {
|
||||
for (message, expected_status) in [
|
||||
("项目不存在", StatusCode::NOT_FOUND),
|
||||
("无权访问该项目", StatusCode::FORBIDDEN),
|
||||
("项目版本冲突", StatusCode::CONFLICT),
|
||||
] {
|
||||
let projects = Arc::new(RecordingProjects {
|
||||
error: Some(message),
|
||||
..Default::default()
|
||||
});
|
||||
let media = Arc::new(RecordingMediaRepair::default());
|
||||
let router = metadata_router(projects.clone(), media.clone());
|
||||
for (method, path, body) in [
|
||||
("GET", "/projects/recent", Value::Null),
|
||||
("GET", "/projects", Value::Null),
|
||||
("GET", "/projects/project-fixture", Value::Null),
|
||||
("POST", "/projects", json!({"title": "测试"})),
|
||||
(
|
||||
"PATCH",
|
||||
"/projects/project-fixture/metadata",
|
||||
json!({"title": "测试"}),
|
||||
),
|
||||
(
|
||||
"PATCH",
|
||||
"/projects/project-fixture",
|
||||
json!({"viewport": {"x": 0, "y": 0, "scale": 1}, "layers": [], "expectedRevision": 7}),
|
||||
),
|
||||
("DELETE", "/projects/project-fixture", Value::Null),
|
||||
] {
|
||||
let (status, body) = send(&router, method, path, body).await;
|
||||
assert_eq!(status, expected_status, "{method} {path}");
|
||||
assert!(body.to_string().contains(message));
|
||||
}
|
||||
assert_eq!(projects.calls.lock().unwrap().len(), 7);
|
||||
assert!(media.calls.lock().unwrap().is_empty());
|
||||
}
|
||||
}
|
||||
@@ -16,7 +16,7 @@ use crate::{
|
||||
external_api_keys::{hash_external_api_key, map_external_api_key_error},
|
||||
http_error::AppError,
|
||||
request_context::RequestContext,
|
||||
state::AppState,
|
||||
state::ExternalApiAuthState,
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
@@ -50,7 +50,7 @@ impl ExternalApiPrincipal {
|
||||
}
|
||||
|
||||
pub async fn require_external_api_key(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<ExternalApiAuthState>,
|
||||
mut request: Request,
|
||||
next: Next,
|
||||
) -> Result<Response, AppError> {
|
||||
@@ -61,7 +61,7 @@ pub async fn require_external_api_key(
|
||||
.unwrap_or_else(|| "unknown".to_string());
|
||||
let raw_key = extract_external_api_bearer(request.headers())?;
|
||||
let key = state
|
||||
.spacetime_client()
|
||||
.authenticator()
|
||||
.authenticate_external_api_key(ExternalApiKeyAuthenticateRecordInput {
|
||||
key_hash: hash_external_api_key(raw_key.as_str()),
|
||||
used_at_micros: current_utc_micros(),
|
||||
@@ -88,7 +88,7 @@ pub async fn require_external_api_key(
|
||||
}
|
||||
|
||||
pub async fn require_external_mcp_api_key(
|
||||
State(state): State<AppState>,
|
||||
State(state): State<ExternalApiAuthState>,
|
||||
request: Request,
|
||||
next: Next,
|
||||
) -> Result<Response, AppError> {
|
||||
@@ -171,6 +171,184 @@ fn extract_external_api_bearer(headers: &HeaderMap) -> Result<String, AppError>
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::state::external_api_auth::ExternalApiKeyAuthenticator;
|
||||
use axum::{
|
||||
Router,
|
||||
body::{Body, to_bytes},
|
||||
extract::Extension,
|
||||
middleware,
|
||||
routing::get,
|
||||
};
|
||||
use futures_util::future::BoxFuture;
|
||||
use spacetime_client::{ExternalApiKeyRecord, SpacetimeClientError};
|
||||
use std::sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicUsize, Ordering},
|
||||
};
|
||||
use tower::ServiceExt;
|
||||
|
||||
struct RecordingAuthenticator {
|
||||
requests: Mutex<Vec<ExternalApiKeyAuthenticateRecordInput>>,
|
||||
error: Option<&'static str>,
|
||||
}
|
||||
|
||||
impl ExternalApiKeyAuthenticator for RecordingAuthenticator {
|
||||
fn authenticate_external_api_key(
|
||||
&self,
|
||||
input: ExternalApiKeyAuthenticateRecordInput,
|
||||
) -> BoxFuture<'_, Result<ExternalApiKeyRecord, SpacetimeClientError>> {
|
||||
self.requests.lock().unwrap().push(input);
|
||||
Box::pin(async move {
|
||||
if let Some(error) = self.error {
|
||||
return Err(SpacetimeClientError::Procedure(error.to_string()));
|
||||
}
|
||||
Ok(ExternalApiKeyRecord {
|
||||
key_id: "key-from-store".to_string(),
|
||||
owner_user_id: "owner-from-store".to_string(),
|
||||
name: "测试密钥".to_string(),
|
||||
key_prefix: "tnr_sk_fixture".to_string(),
|
||||
scopes: vec!["editor:project".to_string()],
|
||||
created_at: "0.000000Z".to_string(),
|
||||
last_used_at: None,
|
||||
revoked_at: None,
|
||||
updated_at: "0.000000Z".to_string(),
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn auth_test_router(
|
||||
dependency: Arc<RecordingAuthenticator>,
|
||||
entered: Arc<AtomicUsize>,
|
||||
mcp: bool,
|
||||
) -> Router {
|
||||
let state = ExternalApiAuthState::new(dependency);
|
||||
let router = Router::new().route("/protected", get(move |Extension(principal): Extension<ExternalApiPrincipal>| {
|
||||
let entered = entered.clone();
|
||||
async move {
|
||||
entered.fetch_add(1, Ordering::Relaxed);
|
||||
axum::Json(json!({"owner": principal.owner_user_id(), "projectScope": principal.has_scope("editor:project")}))
|
||||
}
|
||||
}));
|
||||
if mcp {
|
||||
router.layer(middleware::from_fn_with_state(
|
||||
state,
|
||||
require_external_mcp_api_key,
|
||||
))
|
||||
} else {
|
||||
router.layer(middleware::from_fn_with_state(
|
||||
state,
|
||||
require_external_api_key,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn narrow_external_auth_forwards_store_identity_and_only_hashes_credentials() {
|
||||
let dependency = Arc::new(RecordingAuthenticator {
|
||||
requests: Mutex::default(),
|
||||
error: None,
|
||||
});
|
||||
let entered = Arc::new(AtomicUsize::new(0));
|
||||
let response = auth_test_router(dependency.clone(), entered.clone(), false)
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.uri("/protected")
|
||||
.header(AUTHORIZATION, "Bearer tnr_sk_fixture_secret")
|
||||
.body(Body::empty())
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), StatusCode::OK);
|
||||
let principal = response.extensions().get::<ExternalApiPrincipal>().unwrap();
|
||||
assert_eq!(principal.owner_user_id(), "owner-from-store");
|
||||
assert_eq!(principal.key_id(), "key-from-store");
|
||||
let body: serde_json::Value =
|
||||
serde_json::from_slice(&to_bytes(response.into_body(), 1024).await.unwrap()).unwrap();
|
||||
assert_eq!(
|
||||
body,
|
||||
json!({"owner": "owner-from-store", "projectScope": true})
|
||||
);
|
||||
assert_eq!(entered.load(Ordering::Relaxed), 1);
|
||||
let requests = dependency.requests.lock().unwrap();
|
||||
assert_eq!(requests.len(), 1);
|
||||
assert_eq!(
|
||||
requests[0].key_hash,
|
||||
hash_external_api_key("tnr_sk_fixture_secret")
|
||||
);
|
||||
assert!(requests[0].used_at_micros > 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn narrow_external_auth_preserves_failure_mapping_and_mcp_guide() {
|
||||
for (message, expected_status) in [
|
||||
("API Key 不存在", StatusCode::UNAUTHORIZED),
|
||||
("无权使用此密钥", StatusCode::FORBIDDEN),
|
||||
("校验请求失败", StatusCode::BAD_REQUEST),
|
||||
] {
|
||||
for mcp in [false, true] {
|
||||
let dependency = Arc::new(RecordingAuthenticator {
|
||||
requests: Mutex::default(),
|
||||
error: Some(message),
|
||||
});
|
||||
let entered = Arc::new(AtomicUsize::new(0));
|
||||
let response = auth_test_router(dependency.clone(), entered.clone(), mcp)
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.uri("/protected")
|
||||
.header(AUTHORIZATION, "Bearer invalid-fixture-key")
|
||||
.body(Body::empty())
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), expected_status);
|
||||
assert!(
|
||||
response
|
||||
.extensions()
|
||||
.get::<ExternalApiPrincipal>()
|
||||
.is_none()
|
||||
);
|
||||
assert_eq!(entered.load(Ordering::Relaxed), 0);
|
||||
assert_eq!(dependency.requests.lock().unwrap().len(), 1);
|
||||
let body = String::from_utf8(
|
||||
to_bytes(response.into_body(), 16_384)
|
||||
.await
|
||||
.unwrap()
|
||||
.to_vec(),
|
||||
)
|
||||
.unwrap();
|
||||
if mcp && expected_status == StatusCode::UNAUTHORIZED {
|
||||
assert!(body.contains("MCP_AUTHENTICATION_REQUIRED"));
|
||||
assert!(!body.contains(message));
|
||||
} else {
|
||||
assert!(body.contains(message));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn narrow_external_auth_rejects_missing_credentials_before_store_access() {
|
||||
let dependency = Arc::new(RecordingAuthenticator {
|
||||
requests: Mutex::default(),
|
||||
error: None,
|
||||
});
|
||||
let entered = Arc::new(AtomicUsize::new(0));
|
||||
let response = auth_test_router(dependency.clone(), entered.clone(), false)
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.uri("/protected")
|
||||
.body(Body::empty())
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), StatusCode::UNAUTHORIZED);
|
||||
assert!(dependency.requests.lock().unwrap().is_empty());
|
||||
assert_eq!(entered.load(Ordering::Relaxed), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mcp_authentication_guide_replaces_sensitive_key_diagnostics() {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -52,6 +52,12 @@ use crate::work_author::{
|
||||
ORPHAN_WORK_AUTHOR_DISPLAY_NAME, ORPHAN_WORK_AUTHOR_PUBLIC_USER_CODE, ORPHAN_WORK_OWNER_USER_ID,
|
||||
};
|
||||
|
||||
pub(crate) mod external_api_auth;
|
||||
pub(crate) mod project_metadata;
|
||||
pub use external_api_auth::ExternalApiAuthState;
|
||||
pub(crate) use project_metadata::EditorMediaStorageState;
|
||||
pub use project_metadata::EditorProjectState;
|
||||
|
||||
const ADMIN_ROLE: &str = "admin";
|
||||
const EDITOR_AGENT_LLM_MAX_RETRIES: u32 = 1;
|
||||
const EDITOR_AGENT_LLM_MAX_RETRY_BACKOFF_MS: u64 = 60_000;
|
||||
@@ -297,6 +303,8 @@ pub struct AppStateInner {
|
||||
#[cfg_attr(not(test), allow(dead_code))]
|
||||
ai_task_service: AiTaskService,
|
||||
spacetime_client: SpacetimeClient,
|
||||
external_api_auth_state: ExternalApiAuthState,
|
||||
editor_project_state: EditorProjectState,
|
||||
#[cfg(any())]
|
||||
puzzle_gallery_cache: PuzzleGalleryCache,
|
||||
tracking_outbox: Option<Arc<TrackingOutbox>>,
|
||||
@@ -638,6 +646,14 @@ impl AppState {
|
||||
let character_animation_oss_io_limiter =
|
||||
Arc::new(Semaphore::new(CHARACTER_ANIMATION_OSS_MAX_CONCURRENCY));
|
||||
let editor_oss_http_client = build_editor_oss_http_client()?;
|
||||
let editor_project_state = EditorProjectState::new(
|
||||
Arc::new(spacetime_client.clone()),
|
||||
Arc::new(EditorMediaStorageState::new(
|
||||
spacetime_client.clone(),
|
||||
oss_client.clone(),
|
||||
editor_oss_http_client.clone(),
|
||||
)),
|
||||
);
|
||||
let http_request_permit_pools = HttpRequestPermitPools::from_config(&config);
|
||||
let (profile_recharge_order_updates, _) = broadcast::channel(128);
|
||||
// `ensure_orphan_work_owner_user` 只为公开作品作者回退提供进程内占位账号,
|
||||
@@ -685,6 +701,8 @@ impl AppState {
|
||||
wechat_client,
|
||||
wechat_pay_client,
|
||||
ai_task_service,
|
||||
external_api_auth_state: ExternalApiAuthState::new(Arc::new(spacetime_client.clone())),
|
||||
editor_project_state,
|
||||
spacetime_client,
|
||||
#[cfg(any())]
|
||||
puzzle_gallery_cache: PuzzleGalleryCache::new(),
|
||||
@@ -1577,6 +1595,14 @@ impl AppState {
|
||||
&self.spacetime_client
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn with_external_api_auth_state(mut self, state: ExternalApiAuthState) -> Self {
|
||||
Arc::get_mut(&mut self.0)
|
||||
.expect("configure external authentication before sharing AppState")
|
||||
.external_api_auth_state = state;
|
||||
self
|
||||
}
|
||||
|
||||
#[cfg(any())]
|
||||
pub fn puzzle_gallery_cache(&self) -> &PuzzleGalleryCache {
|
||||
&self.puzzle_gallery_cache
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::extract::FromRef;
|
||||
use futures_util::future::BoxFuture;
|
||||
use spacetime_client::{
|
||||
ExternalApiKeyAuthenticateRecordInput, ExternalApiKeyRecord, SpacetimeClient,
|
||||
SpacetimeClientError,
|
||||
};
|
||||
|
||||
use super::AppState;
|
||||
|
||||
pub(crate) trait ExternalApiKeyAuthenticator: Send + Sync {
|
||||
fn authenticate_external_api_key(
|
||||
&self,
|
||||
input: ExternalApiKeyAuthenticateRecordInput,
|
||||
) -> BoxFuture<'_, Result<ExternalApiKeyRecord, SpacetimeClientError>>;
|
||||
}
|
||||
|
||||
impl ExternalApiKeyAuthenticator for SpacetimeClient {
|
||||
fn authenticate_external_api_key(
|
||||
&self,
|
||||
input: ExternalApiKeyAuthenticateRecordInput,
|
||||
) -> BoxFuture<'_, Result<ExternalApiKeyRecord, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::authenticate_external_api_key(self, input))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct ExternalApiAuthState {
|
||||
authenticator: Arc<dyn ExternalApiKeyAuthenticator>,
|
||||
}
|
||||
|
||||
impl ExternalApiAuthState {
|
||||
pub(crate) fn new(authenticator: Arc<dyn ExternalApiKeyAuthenticator>) -> Self {
|
||||
Self { authenticator }
|
||||
}
|
||||
|
||||
pub(crate) fn authenticator(&self) -> &dyn ExternalApiKeyAuthenticator {
|
||||
self.authenticator.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
impl FromRef<AppState> for ExternalApiAuthState {
|
||||
fn from_ref(state: &AppState) -> Self {
|
||||
state.external_api_auth_state.clone()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::extract::FromRef;
|
||||
use futures_util::future::BoxFuture;
|
||||
use platform_oss::OssClient;
|
||||
use spacetime_client::{
|
||||
EditorProjectCreateRecordInput, EditorProjectDeleteRecordInput, EditorProjectGetRecordInput,
|
||||
EditorProjectLayoutSaveV2AckRecord, EditorProjectLayoutSaveV2RecordInput, EditorProjectRecord,
|
||||
EditorProjectRenameRecordInput, SpacetimeClient, SpacetimeClientError,
|
||||
};
|
||||
|
||||
use super::AppState;
|
||||
|
||||
// 元数据入口只依赖现役 typed facade 的项目能力,不暴露其它领域的数据库操作。
|
||||
pub(crate) trait EditorProjectRepository: Send + Sync {
|
||||
fn get_recent_editor_project(
|
||||
&self,
|
||||
owner_user_id: String,
|
||||
) -> BoxFuture<'_, Result<Option<EditorProjectRecord>, SpacetimeClientError>>;
|
||||
|
||||
fn list_editor_projects(
|
||||
&self,
|
||||
owner_user_id: String,
|
||||
) -> BoxFuture<'_, Result<Vec<EditorProjectRecord>, SpacetimeClientError>>;
|
||||
|
||||
fn get_editor_project(
|
||||
&self,
|
||||
input: EditorProjectGetRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>>;
|
||||
|
||||
fn create_editor_project(
|
||||
&self,
|
||||
input: EditorProjectCreateRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>>;
|
||||
|
||||
fn rename_editor_project(
|
||||
&self,
|
||||
input: EditorProjectRenameRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>>;
|
||||
|
||||
fn delete_editor_project(
|
||||
&self,
|
||||
input: EditorProjectDeleteRecordInput,
|
||||
) -> BoxFuture<'_, Result<String, SpacetimeClientError>>;
|
||||
|
||||
fn save_editor_project_layout_v2_ack(
|
||||
&self,
|
||||
input: EditorProjectLayoutSaveV2RecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectLayoutSaveV2AckRecord, SpacetimeClientError>>;
|
||||
}
|
||||
|
||||
impl EditorProjectRepository for SpacetimeClient {
|
||||
fn get_recent_editor_project(
|
||||
&self,
|
||||
owner_user_id: String,
|
||||
) -> BoxFuture<'_, Result<Option<EditorProjectRecord>, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::get_recent_editor_project(
|
||||
self,
|
||||
owner_user_id,
|
||||
))
|
||||
}
|
||||
|
||||
fn list_editor_projects(
|
||||
&self,
|
||||
owner_user_id: String,
|
||||
) -> BoxFuture<'_, Result<Vec<EditorProjectRecord>, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::list_editor_projects(self, owner_user_id))
|
||||
}
|
||||
|
||||
fn get_editor_project(
|
||||
&self,
|
||||
input: EditorProjectGetRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::get_editor_project(self, input))
|
||||
}
|
||||
|
||||
fn create_editor_project(
|
||||
&self,
|
||||
input: EditorProjectCreateRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::create_editor_project(self, input))
|
||||
}
|
||||
|
||||
fn rename_editor_project(
|
||||
&self,
|
||||
input: EditorProjectRenameRecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectRecord, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::rename_editor_project(self, input))
|
||||
}
|
||||
|
||||
fn delete_editor_project(
|
||||
&self,
|
||||
input: EditorProjectDeleteRecordInput,
|
||||
) -> BoxFuture<'_, Result<String, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::delete_editor_project(self, input))
|
||||
}
|
||||
|
||||
fn save_editor_project_layout_v2_ack(
|
||||
&self,
|
||||
input: EditorProjectLayoutSaveV2RecordInput,
|
||||
) -> BoxFuture<'_, Result<EditorProjectLayoutSaveV2AckRecord, SpacetimeClientError>> {
|
||||
Box::pin(SpacetimeClient::save_editor_project_layout_v2_ack(
|
||||
self, input,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) trait EditorProjectMediaRepair: Send + Sync {
|
||||
fn repair_project(&self, project: EditorProjectRecord) -> BoxFuture<'_, EditorProjectRecord>;
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct EditorProjectState {
|
||||
projects: Arc<dyn EditorProjectRepository>,
|
||||
media_repair: Arc<dyn EditorProjectMediaRepair>,
|
||||
}
|
||||
|
||||
impl EditorProjectState {
|
||||
pub(crate) fn new(
|
||||
projects: Arc<dyn EditorProjectRepository>,
|
||||
media_repair: Arc<dyn EditorProjectMediaRepair>,
|
||||
) -> Self {
|
||||
Self {
|
||||
projects,
|
||||
media_repair,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn projects(&self) -> &dyn EditorProjectRepository {
|
||||
self.projects.as_ref()
|
||||
}
|
||||
|
||||
pub(crate) async fn repair_project_media(
|
||||
&self,
|
||||
project: EditorProjectRecord,
|
||||
) -> EditorProjectRecord {
|
||||
self.media_repair.repair_project(project).await
|
||||
}
|
||||
}
|
||||
|
||||
impl FromRef<AppState> for EditorProjectState {
|
||||
fn from_ref(state: &AppState) -> Self {
|
||||
state.editor_project_state.clone()
|
||||
}
|
||||
}
|
||||
|
||||
// 读项目时的内联媒体修复与生成写入共用相同存储依赖,保持原有上传和登记顺序。
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct EditorMediaStorageState {
|
||||
spacetime_client: SpacetimeClient,
|
||||
oss_client: Option<OssClient>,
|
||||
http_client: reqwest::Client,
|
||||
}
|
||||
|
||||
impl EditorMediaStorageState {
|
||||
pub(super) fn new(
|
||||
spacetime_client: SpacetimeClient,
|
||||
oss_client: Option<OssClient>,
|
||||
http_client: reqwest::Client,
|
||||
) -> Self {
|
||||
Self {
|
||||
spacetime_client,
|
||||
oss_client,
|
||||
http_client,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn spacetime_client(&self) -> &SpacetimeClient {
|
||||
&self.spacetime_client
|
||||
}
|
||||
|
||||
pub(crate) fn oss_client(&self) -> Option<&OssClient> {
|
||||
self.oss_client.as_ref()
|
||||
}
|
||||
|
||||
pub(crate) fn editor_oss_http_client(&self) -> &reqwest::Client {
|
||||
&self.http_client
|
||||
}
|
||||
}
|
||||
|
||||
impl FromRef<AppState> for EditorMediaStorageState {
|
||||
fn from_ref(state: &AppState) -> Self {
|
||||
Self::new(
|
||||
state.spacetime_client().clone(),
|
||||
state.oss_client().cloned(),
|
||||
state.editor_oss_http_client().clone(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl EditorProjectMediaRepair for EditorMediaStorageState {
|
||||
fn repair_project(&self, project: EditorProjectRecord) -> BoxFuture<'_, EditorProjectRecord> {
|
||||
Box::pin(crate::editor_project::repair_editor_project_record_inline_media(self, project))
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
use axum::{
|
||||
body::Body,
|
||||
extract::State,
|
||||
extract::{MatchedPath, State},
|
||||
http::{HeaderMap, Request, Response},
|
||||
middleware::Next,
|
||||
};
|
||||
@@ -28,28 +28,42 @@ pub async fn record_http_observability(
|
||||
State(state): State<AppState>,
|
||||
request: Request<Body>,
|
||||
next: Next,
|
||||
) -> Response<Body> {
|
||||
observe_http_request(
|
||||
http_metrics(),
|
||||
state.config.slow_request_threshold_ms,
|
||||
request,
|
||||
next,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn observe_http_request(
|
||||
metrics: &HttpMetrics,
|
||||
slow_request_threshold_ms: u64,
|
||||
request: Request<Body>,
|
||||
next: Next,
|
||||
) -> Response<Body> {
|
||||
let method = request.method().as_str().to_string();
|
||||
let route = observability_route(request.uri().path());
|
||||
let route = observability_route(&request);
|
||||
let scheme = resolve_request_scheme(request.headers());
|
||||
let path = request.uri().path().to_string();
|
||||
let request_id = resolve_request_id(&request).unwrap_or_else(|| "unknown".to_string());
|
||||
let base_labels = http_base_labels(method.clone(), route.clone());
|
||||
let metrics = http_metrics();
|
||||
metrics.in_flight.add(1, &base_labels);
|
||||
let in_flight = RequestInFlightGuard::new(&metrics.in_flight, base_labels.clone());
|
||||
let started_at = std::time::Instant::now();
|
||||
|
||||
let response = next.run(request).await;
|
||||
let status = response.status().as_u16();
|
||||
let status_class = status_class(status);
|
||||
let latency_ms = started_at.elapsed().as_millis().min(u64::MAX as u128) as u64;
|
||||
let slow_request = latency_ms >= state.config.slow_request_threshold_ms;
|
||||
let slow_request = latency_ms >= slow_request_threshold_ms;
|
||||
let labels = http_response_labels(base_labels, status);
|
||||
metrics.requests.add(1, &labels);
|
||||
metrics
|
||||
.duration
|
||||
.record(started_at.elapsed().as_secs_f64(), &labels);
|
||||
metrics.in_flight.add(-1, &labels[..2]);
|
||||
drop(in_flight);
|
||||
|
||||
if slow_request {
|
||||
warn!(
|
||||
@@ -201,12 +215,32 @@ fn track_response_body_in_flight(response: Response<Body>) -> Response<Body> {
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct HttpMetrics {
|
||||
requests: Counter<u64>,
|
||||
in_flight: opentelemetry::metrics::UpDownCounter<i64>,
|
||||
duration: opentelemetry::metrics::Histogram<f64>,
|
||||
}
|
||||
|
||||
// 请求 Future 被取消或 panic unwind 时也必须释放计数;响应体存活由另一 guard 统计。
|
||||
struct RequestInFlightGuard<'a> {
|
||||
counter: &'a opentelemetry::metrics::UpDownCounter<i64>,
|
||||
labels: Vec<KeyValue>,
|
||||
}
|
||||
|
||||
impl<'a> RequestInFlightGuard<'a> {
|
||||
fn new(counter: &'a opentelemetry::metrics::UpDownCounter<i64>, labels: Vec<KeyValue>) -> Self {
|
||||
counter.add(1, &labels);
|
||||
Self { counter, labels }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for RequestInFlightGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
self.counter.add(-1, &self.labels);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(any())]
|
||||
struct PuzzleGalleryCacheMetrics {
|
||||
hits: Counter<u64>,
|
||||
@@ -467,7 +501,11 @@ fn status_class(status: u16) -> &'static str {
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn observability_route(path: &str) -> String {
|
||||
pub(crate) fn observability_route<B>(request: &Request<B>) -> String {
|
||||
if let Some(path) = request.extensions().get::<MatchedPath>() {
|
||||
return path.as_str().to_string();
|
||||
}
|
||||
let path = request.uri().path();
|
||||
if path.starts_with("/admin/api/") {
|
||||
"/admin/api/*".to_string()
|
||||
} else if path.starts_with("/api/") {
|
||||
@@ -490,22 +528,232 @@ pub(crate) fn resolve_request_scheme(headers: &HeaderMap) -> String {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use axum::http::{HeaderMap, HeaderValue};
|
||||
use axum::{
|
||||
Router,
|
||||
body::{Body, Bytes},
|
||||
http::{HeaderMap, HeaderValue, Request, StatusCode},
|
||||
middleware,
|
||||
routing::get,
|
||||
};
|
||||
use http_body_util::BodyExt;
|
||||
use opentelemetry::{
|
||||
KeyValue,
|
||||
metrics::{Counter, Histogram, SyncInstrument, UpDownCounter},
|
||||
};
|
||||
use std::{
|
||||
convert::Infallible,
|
||||
sync::{Arc, Mutex},
|
||||
time::Duration,
|
||||
};
|
||||
use tokio::sync::Notify;
|
||||
use tower::ServiceExt;
|
||||
|
||||
use super::{observability_route, resolve_request_scheme};
|
||||
use super::{HttpMetrics, observability_route, observe_http_request, resolve_request_scheme};
|
||||
|
||||
#[derive(Default)]
|
||||
struct Measurements<T> {
|
||||
values: Mutex<Vec<(T, Vec<KeyValue>)>>,
|
||||
}
|
||||
|
||||
impl<T: Send> SyncInstrument<T> for Measurements<T> {
|
||||
fn measure(&self, value: T, attributes: &[KeyValue]) {
|
||||
self.values
|
||||
.lock()
|
||||
.expect("measurement lock")
|
||||
.push((value, attributes.to_vec()));
|
||||
}
|
||||
}
|
||||
|
||||
fn observed_router(router: Router) -> (Router, Arc<Measurements<i64>>) {
|
||||
let in_flight = Arc::new(Measurements::default());
|
||||
let metrics = HttpMetrics {
|
||||
requests: Counter::new(Arc::new(Measurements::<u64>::default())),
|
||||
in_flight: UpDownCounter::new(in_flight.clone()),
|
||||
duration: Histogram::new(Arc::new(Measurements::<f64>::default())),
|
||||
};
|
||||
let router = router.layer(middleware::from_fn(move |request, next| {
|
||||
let metrics = metrics.clone();
|
||||
async move { observe_http_request(&metrics, u64::MAX, request, next).await }
|
||||
}));
|
||||
(router, in_flight)
|
||||
}
|
||||
|
||||
fn request(uri: &str) -> Request<Body> {
|
||||
Request::builder()
|
||||
.uri(uri)
|
||||
.body(Body::empty())
|
||||
.expect("request")
|
||||
}
|
||||
|
||||
fn assert_request_finished(measurements: &Measurements<i64>) {
|
||||
let values = measurements.values.lock().expect("measurement lock");
|
||||
assert_eq!(
|
||||
values.iter().map(|(value, _)| *value).collect::<Vec<_>>(),
|
||||
vec![1, -1]
|
||||
);
|
||||
assert_eq!(
|
||||
values[0].1, values[1].1,
|
||||
"decrement must use the original labels"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cancelled_request_releases_in_flight_measurement() {
|
||||
let entered = Arc::new(Notify::new());
|
||||
let handler_entered = entered.clone();
|
||||
let (router, measurements) = observed_router(Router::new().route(
|
||||
"/api/pending/{id}",
|
||||
get(move || {
|
||||
let entered = handler_entered.clone();
|
||||
async move {
|
||||
entered.notify_one();
|
||||
std::future::pending::<StatusCode>().await
|
||||
}
|
||||
}),
|
||||
));
|
||||
let task = tokio::spawn(router.oneshot(request("/api/pending/123")));
|
||||
tokio::time::timeout(Duration::from_secs(5), entered.notified())
|
||||
.await
|
||||
.expect("handler entered");
|
||||
assert_eq!(
|
||||
measurements.values.lock().expect("measurement lock")[0].0,
|
||||
1
|
||||
);
|
||||
|
||||
task.abort();
|
||||
assert!(
|
||||
task.await
|
||||
.expect_err("request should be cancelled")
|
||||
.is_cancelled()
|
||||
);
|
||||
assert_request_finished(&measurements);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn panicking_request_releases_in_flight_measurement() {
|
||||
async fn panic_handler() -> StatusCode {
|
||||
panic!("handler panic for cancellation cleanup test");
|
||||
}
|
||||
let (router, measurements) =
|
||||
observed_router(Router::new().route("/panic", get(panic_handler)));
|
||||
let task = tokio::spawn(router.oneshot(request("/panic")));
|
||||
assert!(task.await.expect_err("handler should panic").is_panic());
|
||||
assert_request_finished(&measurements);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn responses_release_in_flight_once_for_success_and_errors() {
|
||||
for status in [
|
||||
StatusCode::OK,
|
||||
StatusCode::UNAUTHORIZED,
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
] {
|
||||
let (router, measurements) = observed_router(
|
||||
Router::new().route("/response", get(move || async move { status })),
|
||||
);
|
||||
let response = router
|
||||
.oneshot(request("/response"))
|
||||
.await
|
||||
.expect("response");
|
||||
assert_eq!(response.status(), status);
|
||||
assert_request_finished(&measurements);
|
||||
drop(response);
|
||||
assert_request_finished(&measurements);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn streaming_response_releases_request_before_body_completion() {
|
||||
use futures_util::StreamExt;
|
||||
|
||||
let (router, measurements) = observed_router(Router::new().route(
|
||||
"/events",
|
||||
get(|| async {
|
||||
let stream = futures_util::stream::iter([Ok::<_, Infallible>(Bytes::from_static(
|
||||
b"data: ready\n\n",
|
||||
))])
|
||||
.chain(futures_util::stream::pending());
|
||||
(
|
||||
[("content-type", "text/event-stream")],
|
||||
Body::from_stream(stream),
|
||||
)
|
||||
}),
|
||||
));
|
||||
let mut response = router
|
||||
.oneshot(request("/events"))
|
||||
.await
|
||||
.expect("streaming response");
|
||||
assert_request_finished(&measurements);
|
||||
let frame = response
|
||||
.body_mut()
|
||||
.frame()
|
||||
.await
|
||||
.expect("first frame")
|
||||
.expect("body frame");
|
||||
assert_eq!(frame.into_data().expect("data frame"), "data: ready\n\n");
|
||||
drop(response);
|
||||
assert_request_finished(&measurements);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn matched_routes_share_templates_and_preserve_distinct_endpoints() {
|
||||
let routes = Router::new().nest(
|
||||
"/api",
|
||||
Router::new()
|
||||
.route("/projects/{project_id}", get(|| async {}))
|
||||
.route("/assets/{asset_id}", get(|| async {})),
|
||||
);
|
||||
let (router, measurements) = observed_router(routes);
|
||||
for (uri, template) in [
|
||||
(
|
||||
"/api/projects/project-1?cursor=private",
|
||||
"/api/projects/{project_id}",
|
||||
),
|
||||
("/api/projects/project-2", "/api/projects/{project_id}"),
|
||||
("/api/assets/asset-1", "/api/assets/{asset_id}"),
|
||||
("/api/missing/private-id?token=private", "/api/*"),
|
||||
] {
|
||||
let response = router
|
||||
.clone()
|
||||
.oneshot(request(uri))
|
||||
.await
|
||||
.expect("response");
|
||||
drop(response);
|
||||
assert_request_finished(&measurements);
|
||||
let mut values = measurements.values.lock().expect("measurement lock");
|
||||
assert_eq!(
|
||||
values[0].1,
|
||||
vec![
|
||||
KeyValue::new("http.request.method", "GET"),
|
||||
KeyValue::new("http.route", template)
|
||||
]
|
||||
);
|
||||
values.clear();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn observability_route_keeps_metrics_labels_low_cardinality() {
|
||||
assert_eq!(
|
||||
observability_route("/api/editor/showcase/resources?cursor=abc"),
|
||||
observability_route(&request("/api/editor/showcase/resources?cursor=abc")),
|
||||
"/api/*"
|
||||
);
|
||||
assert_eq!(
|
||||
observability_route("/api/editor/projects/project-1"),
|
||||
observability_route(&request("/api/editor/projects/project-1")),
|
||||
"/api/*"
|
||||
);
|
||||
assert_eq!(observability_route("/api/runtime/settings"), "/api/*");
|
||||
assert_eq!(observability_route("/admin/api/debug/http"), "/admin/api/*");
|
||||
assert_eq!(
|
||||
observability_route(&request("/api/runtime/settings")),
|
||||
"/api/*"
|
||||
);
|
||||
assert_eq!(
|
||||
observability_route(&request("/admin/api/debug/http")),
|
||||
"/admin/api/*"
|
||||
);
|
||||
assert_eq!(
|
||||
observability_route(&request("/missing/private-id")),
|
||||
"other"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -11,6 +11,8 @@ reqwest = { workspace = true, features = ["json", "rustls-tls-native-roots", "st
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
tokio = { workspace = true, features = ["time"] }
|
||||
tracing = { workspace = true }
|
||||
|
||||
[dev-dependencies]
|
||||
tokio = { workspace = true, features = ["macros", "rt"] }
|
||||
tokio = { workspace = true, features = ["macros", "rt", "sync"] }
|
||||
tracing-subscriber = { workspace = true }
|
||||
|
||||
@@ -18,6 +18,9 @@ use tokio::time::sleep;
|
||||
mod provider_adapter;
|
||||
pub mod router_billing;
|
||||
|
||||
#[cfg(test)]
|
||||
mod observability_tests;
|
||||
|
||||
pub use provider_adapter::{
|
||||
ANTHROPIC_PROVIDER_INSTANCE_ID, ANTHROPIC_PROVIDER_PROTOCOL_ID, AnthropicProviderAdapter,
|
||||
OPENAI_CHAT_PROVIDER_INSTANCE_ID, OPENAI_CHAT_PROVIDER_PROTOCOL_ID,
|
||||
@@ -1640,6 +1643,16 @@ impl LlmClient {
|
||||
self
|
||||
}
|
||||
|
||||
#[tracing::instrument(
|
||||
name = "llm.request",
|
||||
skip_all,
|
||||
fields(
|
||||
provider = self.config.provider().as_str(),
|
||||
operation = "run",
|
||||
api_kind = request.api_kind.as_str(),
|
||||
model = request.resolved_model(self.config.model()),
|
||||
)
|
||||
)]
|
||||
pub async fn run(&self, request: LlmRunRequest) -> Result<LlmRunResponse, LlmError> {
|
||||
request.validate()?;
|
||||
let resolved_model = request.resolved_model(self.config.model()).to_string();
|
||||
@@ -1687,6 +1700,16 @@ impl LlmClient {
|
||||
.await
|
||||
}
|
||||
|
||||
#[tracing::instrument(
|
||||
name = "llm.request",
|
||||
skip_all,
|
||||
fields(
|
||||
provider = self.config.provider().as_str(),
|
||||
operation = "stream_run",
|
||||
api_kind = request.api_kind.as_str(),
|
||||
model = request.resolved_model(self.config.model()),
|
||||
)
|
||||
)]
|
||||
pub async fn stream_run<F>(
|
||||
&self,
|
||||
request: LlmRunRequest,
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
io::{Read, Write},
|
||||
net::TcpListener,
|
||||
sync::{Arc, Mutex, mpsc},
|
||||
thread,
|
||||
time::Duration,
|
||||
};
|
||||
|
||||
use tokio::sync::oneshot;
|
||||
use tracing::{
|
||||
Instrument, Subscriber,
|
||||
field::{Field, Visit},
|
||||
instrument::WithSubscriber,
|
||||
span::{Attributes, Id, Record},
|
||||
};
|
||||
use tracing_subscriber::{Layer, layer::Context, prelude::*, registry::LookupSpan};
|
||||
|
||||
use super::{LlmClient, LlmConfig, LlmError, LlmProvider, LlmRunRequest};
|
||||
|
||||
const PRIVATE_INPUT: &str = "PRIVATE_MESSAGE_MUST_NOT_ENTER_SPAN";
|
||||
const PRIVATE_KEY: &str = "PRIVATE_API_KEY_MUST_NOT_ENTER_SPAN";
|
||||
|
||||
#[derive(Clone, Debug, Default)]
|
||||
struct CapturedSpan {
|
||||
name: String,
|
||||
parent: Option<u64>,
|
||||
fields: BTreeMap<String, String>,
|
||||
closed: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct Capture {
|
||||
spans: Arc<Mutex<BTreeMap<u64, CapturedSpan>>>,
|
||||
delta_parents: Arc<Mutex<Vec<u64>>>,
|
||||
}
|
||||
|
||||
struct Fields<'a>(&'a mut BTreeMap<String, String>);
|
||||
|
||||
impl Visit for Fields<'_> {
|
||||
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
|
||||
self.0.insert(field.name().into(), format!("{value:?}"));
|
||||
}
|
||||
|
||||
fn record_str(&mut self, field: &Field, value: &str) {
|
||||
self.0.insert(field.name().into(), value.into());
|
||||
}
|
||||
}
|
||||
|
||||
impl<S: Subscriber + for<'a> LookupSpan<'a>> Layer<S> for Capture {
|
||||
fn on_new_span(&self, attributes: &Attributes<'_>, id: &Id, context: Context<'_, S>) {
|
||||
let span = context.span(id).expect("registered span");
|
||||
let mut captured = CapturedSpan {
|
||||
name: attributes.metadata().name().into(),
|
||||
parent: span.parent().map(|parent| parent.id().into_u64()),
|
||||
..CapturedSpan::default()
|
||||
};
|
||||
attributes.record(&mut Fields(&mut captured.fields));
|
||||
self.spans.lock().unwrap().insert(id.into_u64(), captured);
|
||||
}
|
||||
|
||||
fn on_record(&self, id: &Id, values: &Record<'_>, _: Context<'_, S>) {
|
||||
let mut spans = self.spans.lock().unwrap();
|
||||
values.record(&mut Fields(
|
||||
&mut spans.get_mut(&id.into_u64()).unwrap().fields,
|
||||
));
|
||||
}
|
||||
|
||||
fn on_close(&self, id: Id, _: Context<'_, S>) {
|
||||
self.spans
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get_mut(&id.into_u64())
|
||||
.unwrap()
|
||||
.closed = true;
|
||||
}
|
||||
|
||||
fn on_event(&self, event: &tracing::Event<'_>, context: Context<'_, S>) {
|
||||
if event.metadata().target() == "llm_observability_test_delta" {
|
||||
let parent = context
|
||||
.event_span(event)
|
||||
.expect("delta should have a parent");
|
||||
self.delta_parents
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(parent.id().into_u64());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Capture {
|
||||
fn provider_span(&self) -> (u64, CapturedSpan) {
|
||||
let spans = self.spans.lock().unwrap();
|
||||
let provider_spans = spans
|
||||
.iter()
|
||||
.filter(|(_, span)| span.name == "llm.request")
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(provider_spans.len(), 1);
|
||||
let (id, span) = provider_spans[0];
|
||||
(*id, span.clone())
|
||||
}
|
||||
|
||||
fn assert_completed(&self, operation: &str) {
|
||||
let (_, span) = self.provider_span();
|
||||
assert!(
|
||||
span.closed,
|
||||
"provider span must close after completion or cancellation"
|
||||
);
|
||||
assert_eq!(
|
||||
span.fields.get("operation").map(String::as_str),
|
||||
Some(operation)
|
||||
);
|
||||
assert_eq!(
|
||||
span.fields.get("provider").map(String::as_str),
|
||||
Some("openai_compatible")
|
||||
);
|
||||
assert_eq!(
|
||||
span.fields.get("api_kind").map(String::as_str),
|
||||
Some("openai_chat")
|
||||
);
|
||||
assert_eq!(
|
||||
span.fields.get("model").map(String::as_str),
|
||||
Some("requested-model")
|
||||
);
|
||||
let spans = self.spans.lock().unwrap();
|
||||
assert_eq!(
|
||||
spans[&span.parent.expect("HTTP/application parent")].name,
|
||||
"test.request"
|
||||
);
|
||||
let captured = format!("{spans:?}");
|
||||
assert!(!captured.contains(PRIVATE_INPUT));
|
||||
assert!(!captured.contains(PRIVATE_KEY));
|
||||
assert!(!span.fields.contains_key("self"));
|
||||
assert!(!span.fields.contains_key("request"));
|
||||
}
|
||||
}
|
||||
|
||||
struct ProviderFixture {
|
||||
client: LlmClient,
|
||||
entered: oneshot::Receiver<()>,
|
||||
release: mpsc::Sender<()>,
|
||||
server: thread::JoinHandle<()>,
|
||||
}
|
||||
|
||||
// 使用本地替代上游和通道控制响应时点,不依赖实际 Provider 或计时猜测。
|
||||
fn provider_fixture(status: &str, content_type: &str, body: &str) -> ProviderFixture {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
let address = listener.local_addr().unwrap();
|
||||
let response = format!(
|
||||
"HTTP/1.1 {status}\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
);
|
||||
let (entered, entered_rx) = oneshot::channel();
|
||||
let (release, release_rx) = mpsc::channel();
|
||||
let server = thread::spawn(move || {
|
||||
listener.set_nonblocking(true).unwrap();
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(10);
|
||||
let mut stream = loop {
|
||||
match listener.accept() {
|
||||
Ok((stream, _)) => break stream,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {
|
||||
assert!(
|
||||
std::time::Instant::now() < deadline,
|
||||
"request did not arrive"
|
||||
);
|
||||
thread::sleep(Duration::from_millis(5));
|
||||
}
|
||||
Err(error) => panic!("fixture accept: {error}"),
|
||||
}
|
||||
};
|
||||
stream.set_nonblocking(false).unwrap();
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(5)))
|
||||
.unwrap();
|
||||
let mut bytes = Vec::new();
|
||||
let mut chunk = [0; 4096];
|
||||
loop {
|
||||
let count = stream.read(&mut chunk).unwrap();
|
||||
assert_ne!(count, 0);
|
||||
bytes.extend_from_slice(&chunk[..count]);
|
||||
if let Some(headers_end) = bytes.windows(4).position(|part| part == b"\r\n\r\n") {
|
||||
let headers = String::from_utf8_lossy(&bytes[..headers_end]);
|
||||
let length = headers
|
||||
.lines()
|
||||
.find_map(|line| {
|
||||
let (name, value) = line.split_once(':')?;
|
||||
name.eq_ignore_ascii_case("content-length")
|
||||
.then(|| value.trim().parse::<usize>().unwrap())
|
||||
})
|
||||
.unwrap_or(0);
|
||||
if bytes.len() >= headers_end + 4 + length {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
entered.send(()).unwrap();
|
||||
release_rx.recv_timeout(Duration::from_secs(10)).unwrap();
|
||||
// 取消用例已关闭客户端连接,允许写回失败。
|
||||
let _ = stream.write_all(response.as_bytes());
|
||||
});
|
||||
let config = LlmConfig::new(
|
||||
LlmProvider::OpenAiCompatible,
|
||||
format!("http://{address}"),
|
||||
PRIVATE_KEY.into(),
|
||||
"default-model".into(),
|
||||
5_000,
|
||||
0,
|
||||
1,
|
||||
)
|
||||
.unwrap();
|
||||
ProviderFixture {
|
||||
client: LlmClient::new(config).unwrap(),
|
||||
entered: entered_rx,
|
||||
release,
|
||||
server,
|
||||
}
|
||||
}
|
||||
|
||||
fn request() -> LlmRunRequest {
|
||||
LlmRunRequest::single_turn("system", PRIVATE_INPUT)
|
||||
.with_openai_chat()
|
||||
.with_model("requested-model")
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn provider_span_covers_awaited_execution_and_keeps_parent_without_arguments() {
|
||||
let fixture = provider_fixture(
|
||||
"200 OK",
|
||||
"application/json",
|
||||
r#"{"choices":[{"message":{"content":"completed"},"finish_reason":"stop"}]}"#,
|
||||
);
|
||||
let capture = Capture::default();
|
||||
let response =
|
||||
async {
|
||||
async {
|
||||
let mut future = Box::pin(fixture.client.run(request()));
|
||||
assert!(capture.spans.lock().unwrap().values().all(|span| span.name != "llm.request"));
|
||||
tokio::select! {
|
||||
response = &mut future => panic!("request returned before released: {response:?}"),
|
||||
entered = fixture.entered => entered.unwrap(),
|
||||
}
|
||||
assert!(!capture.provider_span().1.closed);
|
||||
// 挂起时不得把 Provider span 留在当前异步执行上下文。
|
||||
assert_eq!(tracing::Span::current().metadata().unwrap().name(), "test.request");
|
||||
fixture.release.send(()).unwrap();
|
||||
future.await.unwrap()
|
||||
}.instrument(tracing::info_span!("test.request")).await
|
||||
}
|
||||
.with_subscriber(tracing_subscriber::registry().with(capture.clone()))
|
||||
.await;
|
||||
fixture.server.join().unwrap();
|
||||
assert_eq!(response.text, "completed");
|
||||
capture.assert_completed("run");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stream_callbacks_inherit_provider_span_and_keep_result() {
|
||||
let fixture = provider_fixture(
|
||||
"200 OK",
|
||||
"text/event-stream",
|
||||
"data: {\"choices\":[{\"delta\":{\"content\":\"hello\"},\"finish_reason\":null}]}\n\ndata: {\"choices\":[{\"delta\":{},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n",
|
||||
);
|
||||
let capture = Capture::default();
|
||||
let mut deltas = Vec::new();
|
||||
let response =
|
||||
async {
|
||||
async {
|
||||
let mut future = Box::pin(fixture.client.stream_run(request(), |delta| {
|
||||
tracing::info!(target: "llm_observability_test_delta", "delta received");
|
||||
deltas.push(delta.delta_text.clone());
|
||||
}));
|
||||
tokio::select! {
|
||||
response = &mut future => panic!("stream returned before released: {response:?}"),
|
||||
entered = fixture.entered => entered.unwrap(),
|
||||
}
|
||||
fixture.release.send(()).unwrap();
|
||||
future.await.unwrap()
|
||||
}.instrument(tracing::info_span!("test.request")).await
|
||||
}
|
||||
.with_subscriber(tracing_subscriber::registry().with(capture.clone()))
|
||||
.await;
|
||||
fixture.server.join().unwrap();
|
||||
assert_eq!(response.text, "hello");
|
||||
assert_eq!(deltas.concat(), "hello");
|
||||
capture.assert_completed("stream_run");
|
||||
let id = capture.provider_span().0;
|
||||
let parents = capture.delta_parents.lock().unwrap();
|
||||
assert!(!parents.is_empty());
|
||||
assert!(parents.iter().all(|parent| *parent == id));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn provider_span_preserves_upstream_errors_and_closes_on_cancellation() {
|
||||
let fixture = provider_fixture(
|
||||
"401 Unauthorized",
|
||||
"application/json",
|
||||
r#"{"error":{"message":"upstream-rejected"}}"#,
|
||||
);
|
||||
let capture = Capture::default();
|
||||
let error = async {
|
||||
async {
|
||||
fixture.release.send(()).unwrap();
|
||||
fixture.client.run(request()).await.unwrap_err()
|
||||
}
|
||||
.instrument(tracing::info_span!("test.request"))
|
||||
.await
|
||||
}
|
||||
.with_subscriber(tracing_subscriber::registry().with(capture.clone()))
|
||||
.await;
|
||||
fixture.server.join().unwrap();
|
||||
assert!(
|
||||
matches!(error, LlmError::Upstream { status_code: 401, message } if message.contains("upstream-rejected"))
|
||||
);
|
||||
capture.assert_completed("run");
|
||||
|
||||
let fixture = provider_fixture("200 OK", "application/json", "{}");
|
||||
let capture = Capture::default();
|
||||
async {
|
||||
async {
|
||||
let mut future = Box::pin(fixture.client.run(request()));
|
||||
tokio::select! {
|
||||
response = &mut future => panic!("request returned before cancellation: {response:?}"),
|
||||
entered = fixture.entered => entered.unwrap(),
|
||||
}
|
||||
assert!(!capture.provider_span().1.closed);
|
||||
drop(future);
|
||||
fixture.release.send(()).unwrap();
|
||||
}.instrument(tracing::info_span!("test.request")).await
|
||||
}.with_subscriber(tracing_subscriber::registry().with(capture.clone())).await;
|
||||
fixture.server.join().unwrap();
|
||||
capture.assert_completed("run");
|
||||
}
|
||||
Reference in New Issue
Block a user