将退款 outbox 主路径迁入 SpacetimeDB (#204)
Project CI / Repository checks (push) Successful in 4m11s
Project CI / Frontend tests (push) Successful in 8m38s
Project CI / Backend tests (push) Successful in 10m42s
Project CI / Native shell tests (push) Successful in 32m11s

## 变更内容

- 新增 `profile_wallet_refund_outbox` 表与 enqueue/process procedure,退款主路径进入 SpacetimeDB。
- 外部生成失败事务、inline 资产失败和跨节点 worker 统一使用库内 outbox,按 ledger 幂等并在事务内完成退款与删除。
- SpacetimeDB 完全不可达时才写本机 emergency spool,恢复时重新入库;兼容旧 spool 文件并保留 attempt 追踪。
- 更新 SpacetimeDB migration、生成 bindings、架构文档、运维恢复说明和项目决策记录。

## 验证

- `cargo check -p spacetime-module -p spacetime-client -p api-server --manifest-path server-rs/Cargo.toml`
- api-server / spacetime-client / spacetime-module / module-runtime 定向测试
- `npm run check:spacetime-schema`
- `npm run check:spacetime-runtime-access`
- `npm run check:server-rs-ddd`
- `npm run check:encoding`
- `git diff --check`

Reviewed-on: http://192.168.35.82/git/GenarrativeAI/Genarrative/pulls/204
Co-authored-by: kdletters <kdletters@qq.com>
Co-committed-by: kdletters <kdletters@qq.com>
This commit was merged in pull request #204.
This commit is contained in:
2026-08-27 22:04:50 +08:00
committed by 段舒康
parent a7337c67a1
commit 458371a73d
55 changed files with 3574 additions and 256 deletions
@@ -53,6 +53,7 @@ services:
- "host.docker.internal:host-gateway"
volumes:
- api-tracking-outbox:/var/lib/genarrative/tracking-outbox
- api-wallet-refund-outbox:/var/lib/genarrative/wallet-refund-outbox
ulimits:
nofile:
soft: 4096
@@ -85,6 +86,9 @@ services:
OTEL_SERVICE_NAME: genarrative-external-generation-worker
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- external-generation-tracking-outbox:/var/lib/genarrative/tracking-outbox-worker
- external-generation-wallet-refund-outbox:/var/lib/genarrative/wallet-refund-outbox
ulimits:
nofile:
soft: 4096
@@ -142,4 +146,7 @@ services:
volumes:
spacetime-data:
api-tracking-outbox:
api-wallet-refund-outbox:
external-generation-tracking-outbox:
external-generation-wallet-refund-outbox:
nginx-logs:
@@ -16,6 +16,22 @@
---
## 2026-08-27 退款 emergency spool 容量溢出保持可恢复
- 背景:本机 emergency spool 仅作为 SpacetimeDB 完全不可达时的最后恢复路径,原有 `MAX_BYTES` 分支会直接返回 `Dropped`,导致扣费已经完成但没有可重放记录。
- 决策:达到普通 outbox `MAX_BYTES` 时,将退款记录写入同一持久目录的 `refund-overflow-*` 文件;该文件与普通 pending 文件一样由启动恢复和后台 worker 重放到 SpacetimeDB,且按 refund ledger id 保持幂等。溢出文件不计入普通阈值,但必须触发容量告警;底层磁盘写入失败仍进入关键退款人工补偿流程。
- 影响范围:api-server wallet refund emergency spool、资产失败退款日志、loadtest / 预览 Compose 持久卷、后端架构与开发运维文档。
- 验证方式:运行 api-server `wallet_refund_outbox` 定向测试,确认超限写入并保留 overflow 文件;运行 SpacetimeDB profile 测试、Compose 配置校验、编码和 diff 门禁。
## 2026-08-27 短期认证状态进入共享 typed projection
- 背景:短信验证码和微信 OAuth state 仍只存在 API 进程内 HashMap,多节点请求或 API 重启会直接丢失,无法满足无粘性会话的鉴权恢复要求。
- 决策:`AuthStoreProjectionView` 增加 `phone_codes``wechat_states` typed 字段,由 `auth_store_projection_meta` 以 JSON 投影持久化;启动恢复、CAS 同步和失败后的权威刷新都覆盖这两类短期状态。验证码哈希使用部署级稳定盐(当前复用 `GENARRATIVE_JWT_SECRET`),各 API 节点必须一致;发码前先刷新权威投影并用占位验证码记录做一次 projection CAS,只有占用成功才调用短信 provider,避免跨节点冷却竞态;认证 handler 在发码、消费验证码、创建/消费微信 state 后都要完成 projection sync,失败即返回服务错误;所有会读取或变更本机认证工作集的认证主链路(登录、刷新、`/me`、会话管理、密码、绑定和微信 state)在领域操作前先从正式投影做一次受 CAS 保护的只读刷新,受保护 Bearer 中间件也会在进入业务 handler 前执行同样的刷新,刷新失败时 fail closed,不能依赖粘性会话;同步遇到 CAS 冲突时,若本次尝试期间没有新的本地变更则恢复正式快照,若仍有待同步 revision 则由后续认证请求重试,避免节点永久卡在 pending。微信 OAuth state 设置有界活动数量,避免单个 JSON 投影无界膨胀。短期状态仍由 `module-auth` 内存工作集执行领域校验,但不再把本机 HashMap 当作持久化或跨节点真相。
- 影响范围:`module-auth` projection、`spacetime-module` auth schema/procedure、`spacetime-client` bindings/facade、api-server 手机号 / 微信 handler、认证架构与运维文档。
- 验证方式:运行 module-auth projection roundtrip(验证码可跨恢复校验、微信 state 可跨恢复消费)、SpacetimeDB schema/runtime/DDD 门禁、api-server 定向测试、编码和 diff 检查。
---
## 2026-08-27 外部生成历史采用受控保留清理
- 背景:`external_generation_job``external_generation_job_summary``external_generation_job_event` 都是持久化表;摘要和 payload 边界收紧后,已确认的终态历史仍会继续占用 SpacetimeDB 常驻内存,且事件审计链会随任务数量增长。
@@ -7772,6 +7788,12 @@ CI 上 `background_agent_runtime_recovers_stale_running_before_pending_task` 在
- 路由:涉及 SpacetimeDB 的任务统一先读取项目适配 skill,再按需读取 `spacetimedb:concepts``spacetimedb:rust-server``spacetimedb:cli``spacetimedb:typescript-client``spacetimedb:mcp`。插件通用示例不得覆盖项目禁止 `maincloud`、禁止人工 `spacetime --root-dir`、显式 server 和后端分层等约束。
- 安装:团队环境缺少插件时使用 `codex plugin marketplace add clockworklabs/SpacetimeDB --sparse .agents --sparse codex-plugin``codex plugin add spacetimedb\@spacetimedb-plugins`;个人配置、缓存和凭据不进入仓库。
## 2026-08-27 退款 outbox 主路径迁入 SpacetimeDB
- 决策:`profile_wallet_refund_outbox` 是跨 API 节点退款的正式持久化队列。扣费失败、外部生成 attempt 失败或最终 lease 过期时,在同一个 SpacetimeDB 事务内按 `refund_ledger_id` 幂等写入 pending 行;worker 从库内 pending 行批量处理,退款账本写入与 outbox 成功删除保持在同一事务内,失败由 `available_at` / `attempts` 驱动重试。`asset_operation_wallet_settlement` 继续负责退款先于 consume 可见时的取消 intent,阻止迟到扣费。只有 SpacetimeDB 完全不可达时,api-server 才写本机 `wallet-refund-outbox` emergency spool;本机文件不能替代库内队列,必须持久挂载、告警、恢复演练并支持人工补偿。
- 影响范围:`profile_wallet_refund_outbox` 表及 bindings、runtime enqueue/process procedure、外部生成失败事务、inline 资产退款、api-server 跨节点 worker 和 emergency spool、后端架构与开发运维文档。
- 验证方式:运行 `npm run spacetime:generate``npm run check:spacetime-schema``npm run check:server-rs-ddd``cargo check -p spacetime-module -p spacetime-client -p api-server --manifest-path server-rs/Cargo.toml`、退款 outbox / asset billing / external generation 定向测试、`npm run check:encoding``git diff --check`
## 2026-08-27 release 内存增长修复与备份 OOM 恢复兜底
- 决策:外部生成 worker 每轮主动 `try_join_next` 回收已完成 `JoinHandle`,避免持续有队列任务时只归还 semaphore permit 却让 `JoinSet` 句柄集合无界增长;超时脱管任务在 abort 后等待句柄结束,执行许可保持到 work 真正结束或被取消。
File diff suppressed because one or more lines are too long
@@ -779,7 +779,7 @@ node scripts/test-ve-llm.mjs
### 手机验证码短信
手机验证码发送走阿里云普通短信 `SendSms`,验证码由 `module-auth` 在当前 `api-server` 进程内生成哈希存储和校验不再调用阿里云托管验证码的 `SendSmsVerifyCode` / `CheckSmsVerifyCode`。因此 `api-server` 重启后,已发送但未校验的验证码失效。
手机验证码发送走阿里云普通短信 `SendSms`,验证码由 `module-auth` 在当前 `api-server` 进程内生成哈希,短期验证码投影随 `auth_store_projection_meta` 同步到 SpacetimeDB 后由任一 API 节点恢复和校验不再调用阿里云托管验证码的 `SendSmsVerifyCode` / `CheckSmsVerifyCode`。因此只要 SpacetimeDB 正常,`api-server` 重启不会使已发送但未过期的验证码失效。
生产默认短信配置:
@@ -855,7 +855,9 @@ GENARRATIVE_TRACKING_OUTBOX_MAX_BYTES=268435456
GENARRATIVE_API_SHUTDOWN_OUTBOX_FLUSH_TIMEOUT_MS=5000
```
outbox 采用 NDJSON 文件保存原始事件。达到 `BATCH_SIZE` 时会立刻把当前 active 文件原子封存为 sealed 文件,并马上切到新的 active 继续写入;后台 worker 异步 flush sealed 文件,HTTP 请求线程不等待 SpacetimeDB。`FLUSH_INTERVAL_MS` 只负责兜底封存长时间未满批的 active 文件。SpacetimeDB 批量 procedure 返回成功后删除 sealed 文件,失败则保留文件并重试。`MAX_BYTES` 是每个 outbox 实例的磁盘保护阈值,不是 flush 阈值;超过后低价值 route tracking 和 BgFilter provider 失败审计可以被丢弃并记录日志 / 指标,关键同步事件不进入该丢弃路径。api-server 使用配置目录本身,BgFilter worker 固定使用其 `bgfilter-worker/` 子目录,两个进程不得操作同一个 active 文件。sealed 文件若出现无法解析的坏行,会重命名为 `corrupt-*` 隔离并记录 `genarrative.tracking_outbox.files.corrupt` 指标,避免一个坏文件阻塞后续批量入库。进程收到退出信号后会在 `GENARRATIVE_API_SHUTDOWN_OUTBOX_FLUSH_TIMEOUT_MS` 窗口内封存各自 active 文件并尽力 flush sealed 文件,超时或 SpacetimeDB 暂不可用时保留本地文件给下次同角色启动继续投递。该机制对已 enqueue 记录提供至少一次投递语义,依赖 `tracking_event.event_id` 幂等跳过重复事件;BgFilter 尚未 enqueue 或因硬上限 / 保护阈值被丢弃的审计不在该保证内。
outbox 采用 NDJSON 文件保存原始事件。达到 `BATCH_SIZE` 时会立刻把当前 active 文件原子封存为 sealed 文件,并马上切到新的 active 继续写入;后台 worker 异步 flush sealed 文件,HTTP 请求线程不等待 SpacetimeDB。worker 启动时会先封存并 flush 已存在的 active / sealed 文件,恢复窗口内 SpacetimeDB 暂不可用则保留文件并按后续周期重试;`FLUSH_INTERVAL_MS` 只负责兜底封存长时间未满批的 active 文件。SpacetimeDB 批量 procedure 返回成功后删除 sealed 文件,失败则保留文件并重试。`MAX_BYTES` 是每个 outbox 实例的磁盘保护阈值,不是 flush 阈值;超过后低价值 route tracking 和 BgFilter provider 失败审计可以被丢弃并记录日志 / 指标,关键同步事件不进入该丢弃路径。api-server 使用配置目录本身,BgFilter worker 固定使用其 `bgfilter-worker/` 子目录,两个进程不得操作同一个 active 文件。sealed 文件若出现无法解析的坏行,会重命名为 `corrupt-*` 隔离并记录 `genarrative.tracking_outbox.files.corrupt` 指标,避免一个坏文件阻塞后续批量入库。进程收到退出信号后会在 `GENARRATIVE_API_SHUTDOWN_OUTBOX_FLUSH_TIMEOUT_MS` 窗口内封存各自 active 文件并尽力 flush sealed 文件,超时或 SpacetimeDB 暂不可用时保留本地文件给下次同角色启动继续投递。该机制对已 enqueue 记录提供至少一次投递语义,依赖 `tracking_event.event_id` 幂等跳过重复事件;BgFilter 尚未 enqueue 或因硬上限 / 保护阈值被丢弃的审计不在该保证内。
钱包退款正式 pending 队列在 SpacetimeDB 的 `profile_wallet_refund_outbox` 表中,由每个 API 节点的 worker 共同处理;worker 启动即扫描库内 pending 行,成功在同一事务内写钱包账本并删除 outbox 行,失败按库内 `available_at` / `attempts` 重试。只有 SpacetimeDB 完全不可达时才写本机 `wallet-refund-outbox` emergency spool;如果进程在“临时文件写完但尚未改名”阶段崩溃,启动恢复会校验 `tmp-*` 内容并原子提升为按 ledger id 命名的 pending 文件,损坏或冲突文件移入 `corrupt-*` 隔离目录。达到 `MAX_BYTES` 时不再静默丢弃退款,而是写入同一持久目录下的 `refund-overflow-*` 溢出文件并继续重放;溢出文件不计入普通容量阈值,但必须接入容量告警和人工补偿预案,底层磁盘写入失败仍按关键退款告警处理。worker 连接失败、库内 retry、emergency spool 写入 / 容量失败和 `corrupt-*` 出现都必须接入告警;人工补偿先按 refund ledger id 对账 `profile_wallet_ledger``asset_operation_wallet_settlement` 与两类 outbox,再通过受控退款 procedure 幂等重放,禁止直接手写钱包表。该目录不能替代库内 outbox;发布和主机替换必须保留 `/var/lib/genarrative/wallet-refund-outbox` 并纳入节点恢复 / 备份演练。容器 loadtest / 预览环境必须分别为 `api-server``external-generation-worker` 挂载各自的 tracking 与 wallet refund 命名卷,不能让节点重建清空本机恢复队列。
release 机器如果日志每秒刷 `tracking outbox ... Permission denied (os error 13)`,先检查 `/etc/genarrative/api-server.env` 是否缺少 `GENARRATIVE_TRACKING_OUTBOX_DIR`。缺少时 `api-server` 会回退到本地开发默认相对路径 `server-rs/.data/tracking-outbox`,而 systemd 的工作目录是只读发布目录 `/opt/genarrative/releases/<version>``genarrative` 用户无法在其中创建 `server-rs`。修复顺序:
@@ -866,7 +868,9 @@ systemctl restart genarrative-api.service
journalctl -u genarrative-api.service --since '30 seconds ago' --no-pager | grep -E 'tracking outbox|Permission denied|os error 13'
```
`Genarrative-Server-Provision``Genarrative-Api-Deploy` 会在保留旧 `/etc/genarrative/api-server.env` 的前提下补齐缺失的 tracking outbox 运行态路径,并确保 `/var/lib/genarrative/tracking-outbox` 归属 `genarrative:genarrative`。用户认证真相源只允许 SpacetimeDB 正式认证表(`user_account` / `auth_identity` / `refresh_session`恢复;不要再配置或依赖 `GENARRATIVE_AUTH_STORE_PATH` / `auth-store.json``module-auth` 也不再维护本地文件持久化;`auth_store_snapshot` 不再作为备查或运行期恢复源,只在正式认证表为空时一次性转移最新旧快照并清空,且旧 `get_auth_store_snapshot` / `upsert_auth_store_snapshot` / `import_auth_store_snapshot` 入口已经删除。如果 `api-server` 启动时连不上 SpacetimeDB,会持续重试启动恢复,直到认证工作集从 SpacetimeDB 正式表恢复成功后才开始监听 HTTP,以避免用空本地状态或旧快照覆盖认证表。
`Genarrative-Server-Provision``Genarrative-Api-Deploy` 会在保留旧 `/etc/genarrative/api-server.env` 的前提下补齐缺失的 tracking outbox 运行态路径,并确保 `/var/lib/genarrative/tracking-outbox` 归属 `genarrative:genarrative`。用户认证真相源只允许 SpacetimeDB 正式认证表(`user_account` / `auth_identity` / `refresh_session``auth_store_projection_meta` 中的短期验证码 / 微信 state 投影恢复;所有会读取或变更本机认证工作集的认证主链路在领域操作前都会从正式投影做一次只读刷新,刷新失败即 fail closed,不依赖粘性会话。不要再配置或依赖 `GENARRATIVE_AUTH_STORE_PATH` / `auth-store.json``module-auth` 也不再维护本地文件持久化;`auth_store_snapshot` 不再作为备查或运行期恢复源,只在正式认证表为空时一次性转移最新旧快照并清空,且旧 `get_auth_store_snapshot` / `upsert_auth_store_snapshot` / `import_auth_store_snapshot` 入口已经删除。所有 API 节点必须使用相同的 `GENARRATIVE_JWT_SECRET`,它也作为验证码哈希盐;轮换后尚未消费的验证码会失效。如果 `api-server` 启动时连不上 SpacetimeDB,会持续重试启动恢复,直到认证工作集从 SpacetimeDB 正式表和短期投影恢复成功后才开始监听 HTTP,以避免用空本地状态或旧快照覆盖认证表。
发短信运维门禁:handler 会先刷新正式认证投影,再通过 projection CAS 写入不可消费的占位验证码来占用跨节点冷却窗口;占用失败时不得调用短信 provider。微信 OAuth state 活动数量有上限,命中上限应返回服务错误并触发限流 / 入口告警;不要通过调大单个 `auth_store_projection_meta` JSON 字段来绕过该保护。
前端登录态恢复只把 `/api/auth/refresh``401` / `403` 当成权威失效信号;服务器重启窗口里的 `502` / `503` / `504`、浏览器 `Failed to fetch` 或 refresh 响应契约异常都必须保留已有本地 access token,不触发全局 auth 变化。refresh 成功响应以共享契约 `RefreshSessionResponse { token }` 为准,前端不要额外要求业务 `ok` 字段。排查“重启后用户都掉线”时,先区分前端是否被暂时不可用清掉本地 token,再检查 SpacetimeDB 正式认证表是否缺 `user_account` / `refresh_session` 数据。
+1
View File
@@ -5435,6 +5435,7 @@ dependencies = [
"shared-contracts",
"spacetimedb",
"spacetimedb-lib",
"time",
]
[[package]]
+4
View File
@@ -657,6 +657,10 @@ pub async fn admin_list_editor_assets(
Extension(_admin): Extension<AuthenticatedAdmin>,
Query(query): Query<AdminEditorAssetListQuery>,
) -> Result<Json<Value>, AppError> {
state
.refresh_auth_store_from_spacetime()
.await
.map_err(map_admin_spacetime_error)?;
let page_size = query
.limit
.unwrap_or(ADMIN_EDITOR_ASSET_DEFAULT_LIMIT)
@@ -64,6 +64,7 @@ pub async fn admin_list_recharge_orders(
Extension(_admin): Extension<AuthenticatedAdmin>,
Query(query): Query<AdminRechargeOrderListQuery>,
) -> Result<Json<Value>, Response> {
refresh_auth_projection(&state, &request_context).await?;
let user_id = resolve_optional_user_id(&state, query.user_id, query.public_user_code)
.map_err(|error| error_response(&request_context, error))?;
let input = build_runtime_profile_recharge_order_admin_list_input(
@@ -112,6 +113,7 @@ pub async fn admin_get_user_detail(
Extension(admin): Extension<AuthenticatedAdmin>,
Query(query): Query<AdminUserDetailQuery>,
) -> Result<Json<Value>, Response> {
refresh_auth_projection(&state, &request_context).await?;
let user = resolve_user(&state, query.user_id, query.public_user_code)
.map_err(|error| error_response(&request_context, error))?;
let wallet_detail = state
@@ -181,6 +183,7 @@ pub async fn admin_reconcile_user_consumption(
Extension(admin): Extension<AuthenticatedAdmin>,
Json(payload): Json<AdminUserConsumptionReconcileRequest>,
) -> Result<Json<Value>, Response> {
refresh_auth_projection(&state, &request_context).await?;
let user = resolve_user(&state, Some(payload.user_id), None)
.map_err(|error| error_response(&request_context, error))?;
let input = build_runtime_profile_wallet_consumption_reconcile_input(
@@ -589,6 +592,7 @@ pub async fn admin_update_wallet_restriction(
Extension(admin): Extension<AuthenticatedAdmin>,
Json(payload): Json<AdminWalletRestrictionRequest>,
) -> Result<Json<Value>, Response> {
refresh_auth_projection(&state, &request_context).await?;
let user = resolve_user(&state, Some(payload.user_id), None)
.map_err(|error| error_response(&request_context, error))?;
let input = build_runtime_profile_wallet_manual_restriction_upsert_input(
@@ -1040,6 +1044,22 @@ fn map_hold(hold: RuntimeProfileRechargeRefundHoldSnapshot) -> AdminRechargeRefu
}
}
async fn refresh_auth_projection(
state: &AppState,
request_context: &RequestContext,
) -> Result<(), Response> {
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
error_response(
request_context,
AppError::from_status(StatusCode::BAD_GATEWAY)
.with_message(format!("刷新用户认证信息失败:{error}")),
)
})
}
fn resolve_optional_user_id(
state: &AppState,
user_id: Option<String>,
+143 -69
View File
@@ -499,87 +499,127 @@ async fn refund_asset_operation_points_with_job_id(
external_generation_claim_attempt: Option<u32>,
) -> Result<(), AppError> {
let created_at_micros = current_utc_micros();
let metadata_json = wallet_metadata_json(
external_generation_job_id.as_deref(),
let current_attempt_is_owned_by_failure_transaction =
external_generation_job_id.as_deref().is_some_and(|job_id| {
current_external_generation_billing_context().is_some_and(|context| {
context.job_id == job_id
&& Some(context.claim_attempt) == external_generation_claim_attempt
})
});
if current_attempt_is_owned_by_failure_transaction {
// 队列当前 attempt 的 refund 由 fail_external_generation_job transaction 原子写入
// SpacetimeDB outbox;这里不能先写另一笔独立退款,避免任务成功写回后被误退。
return Ok(());
}
let settlement_reason = if external_generation_job_id.is_some() {
"stale_attempt_recovery"
} else {
"asset_operation_failed"
};
let enqueue_input = module_runtime::build_runtime_profile_wallet_refund_outbox_enqueue_input(
owner_user_id.clone(),
points_cost,
ledger_id.clone(),
created_at_micros,
asset_kind.clone(),
asset_id.clone(),
settlement_reason.to_string(),
external_generation_job_id.clone(),
external_generation_claim_attempt,
);
let result = state
)
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR).with_details(json!({
"provider": "profile-wallet-refund-outbox",
"message": error.to_string(),
}))
})?;
let enqueue_result = state
.spacetime_client()
.refund_profile_wallet_points_with_metadata(
owner_user_id.clone(),
points_cost,
ledger_id.clone(),
created_at_micros,
metadata_json,
)
.enqueue_profile_wallet_refund_outbox(enqueue_input)
.await;
if let Err(error) = result {
let refund_error = error.to_string();
let app_error = map_asset_operation_wallet_error(error);
if let Some(outbox) = state.wallet_refund_outbox() {
match outbox
.enqueue(WalletRefundOutboxRecord {
owner_user_id: owner_user_id.clone(),
amount: points_cost,
ledger_id: ledger_id.clone(),
created_at_micros,
asset_kind: asset_kind.clone(),
asset_id: asset_id.clone(),
external_generation_job_id: external_generation_job_id.clone(),
})
.await
{
Ok(WalletRefundOutboxEnqueueOutcome::Enqueued) => {
tracing::warn!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
error = %refund_error,
"资产操作失败后的泥点退款立即执行失败,已写入 wallet refund outbox"
);
}
Ok(WalletRefundOutboxEnqueueOutcome::Dropped { reason }) => {
tracing::error!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
reason,
error = %refund_error,
"资产操作失败后的泥点退款立即执行失败,且 wallet refund outbox 因容量限制丢弃"
);
}
Err(outbox_error) => {
tracing::error!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
refund_error = %refund_error,
outbox_error = %outbox_error,
"资产操作失败后的泥点退款立即执行失败,且写入 wallet refund outbox 失败"
);
}
}
} else {
tracing::error!(
match enqueue_result {
Ok(_) => {
tracing::info!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
external_generation_claim_attempt,
ledger_id,
error = %refund_error,
"资产操作失败后的泥点退款失败,且 wallet refund outbox 未启用"
"资产操作失败后的泥点退款已写入 SpacetimeDB refund outbox"
);
Ok(())
}
return Err(app_error);
Err(error) if should_use_wallet_refund_emergency_spool(&error) => {
let refund_error = error.to_string();
let app_error = map_asset_operation_wallet_error(error);
if let Some(outbox) = state.wallet_refund_outbox() {
match outbox
.enqueue(WalletRefundOutboxRecord {
owner_user_id: owner_user_id.clone(),
amount: points_cost,
ledger_id: ledger_id.clone(),
created_at_micros,
asset_kind: asset_kind.clone(),
asset_id: asset_id.clone(),
settlement_reason: settlement_reason.to_string(),
external_generation_job_id: external_generation_job_id.clone(),
external_generation_claim_attempt,
})
.await
{
Ok(WalletRefundOutboxEnqueueOutcome::Enqueued) => {
tracing::warn!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
error = %refund_error,
"SpacetimeDB refund outbox 不可达,已写入本机 emergency spool"
);
}
Ok(WalletRefundOutboxEnqueueOutcome::OverflowEnqueued { reason }) => {
tracing::error!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
reason,
error = %refund_error,
"SpacetimeDB refund outbox 不可达,退款已写入本机 emergency spool overflow 文件;需监控并尽快恢复库内队列"
);
}
Err(outbox_error) => {
tracing::error!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
ledger_id,
refund_error = %refund_error,
outbox_error = %outbox_error,
"SpacetimeDB refund outbox 不可达,且写入本机 emergency spool 失败"
);
}
}
} else {
tracing::error!(
owner_user_id,
asset_kind,
asset_id,
external_generation_job_id,
external_generation_claim_attempt,
ledger_id,
error = %refund_error,
"SpacetimeDB refund outbox 不可达,且本机 emergency spool 未启用"
);
}
Err(app_error)
}
Err(error) => Err(map_asset_operation_wallet_error(error)),
}
Ok(())
}
fn current_external_generation_billing_context() -> Option<ExternalGenerationBillingContext> {
@@ -683,6 +723,22 @@ pub(crate) fn should_skip_asset_operation_billing_for_connectivity(
}
}
fn should_use_wallet_refund_emergency_spool(error: &SpacetimeClientError) -> bool {
match error {
SpacetimeClientError::ConnectDropped | SpacetimeClientError::Timeout(_) => true,
SpacetimeClientError::Build(message)
| SpacetimeClientError::Procedure(message)
| SpacetimeClientError::Runtime(message) => {
message.contains("503")
|| message.contains("Service Unavailable")
|| message.contains("Failed to connect")
|| message.contains("WebSocket")
|| message.contains("连接已断开")
|| message.contains("连接在返回结果前已断开")
}
}
}
fn current_utc_micros() -> i64 {
time::OffsetDateTime::now_utc().unix_timestamp_nanos() as i64 / 1_000
}
@@ -838,6 +894,24 @@ mod tests {
));
}
#[test]
fn wallet_refund_emergency_spool_requires_database_unavailability() {
assert!(should_use_wallet_refund_emergency_spool(
&SpacetimeClientError::ConnectDropped
));
assert!(should_use_wallet_refund_emergency_spool(
&SpacetimeClientError::Runtime("503 Service Unavailable".to_string())
));
assert!(!should_use_wallet_refund_emergency_spool(
&SpacetimeClientError::Procedure(
"No such procedure: enqueue_profile_wallet_refund_outbox_and_return".to_string(),
)
));
assert!(!should_use_wallet_refund_emergency_spool(
&SpacetimeClientError::Procedure("泥点余额不足".to_string())
));
}
#[test]
fn asset_operation_wallet_insufficient_balance_is_public_message() {
for domain_message in [
+72 -38
View File
@@ -19,6 +19,7 @@ use serde_json::{Value, json};
use shared_contracts::auth::RuntimeGuestTokenResponse;
#[cfg(any())]
use shared_kernel::{format_rfc3339, new_uuid_simple_string};
#[cfg(test)]
use time::OffsetDateTime;
use tracing::warn;
@@ -145,6 +146,16 @@ pub async fn require_bearer_auth(
let Some(authenticated) = authenticate_request(&state, headers, request_id).await? else {
return Err(AppError::from_status(StatusCode::UNAUTHORIZED));
};
// JWT 会话校验走 SpacetimeDB;随后刷新用户/身份投影,保证所有受保护路由在
// 读取进程内工作集时都不会依赖粘性会话命中创建或更新它的 API 节点。
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
warn!(error = %error, "受保护请求刷新认证投影失败");
AppError::from_status(StatusCode::SERVICE_UNAVAILABLE)
.with_message("认证状态服务暂不可用")
})?;
request.extensions_mut().insert(authenticated.clone());
let mut response = next.run(request).await;
@@ -236,54 +247,77 @@ async fn authenticate_request(
);
AppError::from_status(StatusCode::UNAUTHORIZED)
})?;
let current_user = state
.auth_user_service()
.get_user_by_id(claims.user_id())
.map_err(|error| {
warn!(
%request_id,
error = %error,
"Bearer JWT 用户快照读取失败"
);
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
})?;
let Some(current_user) = current_user else {
warn!(
%request_id,
user_id = %claims.user_id(),
"Bearer JWT 对应用户不存在"
);
return Err(AppError::from_status(StatusCode::UNAUTHORIZED));
};
if current_user.token_version != claims.token_version() {
warn!(
%request_id,
user_id = %claims.user_id(),
token_version = claims.token_version(),
current_token_version = current_user.token_version,
"Bearer JWT 版本已失效"
);
return Err(AppError::from_status(StatusCode::UNAUTHORIZED)
.with_message("当前登录态已失效,请重新登录"));
}
#[cfg(not(test))]
let session_is_active = state
.refresh_session_service()
.is_session_active_for_user(
claims.user_id(),
claims.session_id(),
OffsetDateTime::now_utc(),
)
.spacetime_client()
.validate_auth_session(spacetime_client::AuthSessionValidationRecordInput {
user_id: claims.user_id().to_string(),
session_id: claims.session_id().to_string(),
token_version: claims.token_version(),
})
.await
.map_err(|error| {
warn!(
%request_id,
user_id = %claims.user_id(),
session_id = %claims.session_id(),
error = %error,
"Bearer JWT refresh session 状态读取失败"
"Bearer JWT SpacetimeDB 会话状态读取失败"
);
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
})?;
#[cfg(test)]
let session_is_active = {
let current_user = state
.auth_user_service()
.get_user_by_id(claims.user_id())
.map_err(|error| {
warn!(
%request_id,
error = %error,
"Bearer JWT 用户快照读取失败"
);
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
})?;
let Some(current_user) = current_user else {
warn!(
%request_id,
user_id = %claims.user_id(),
"Bearer JWT 对应用户不存在"
);
return Err(AppError::from_status(StatusCode::UNAUTHORIZED));
};
if current_user.token_version != claims.token_version() {
warn!(
%request_id,
user_id = %claims.user_id(),
token_version = claims.token_version(),
current_token_version = current_user.token_version,
"Bearer JWT 版本已失效"
);
return Err(AppError::from_status(StatusCode::UNAUTHORIZED)
.with_message("当前登录态已失效,请重新登录"));
}
state
.refresh_session_service()
.is_session_active_for_user(
claims.user_id(),
claims.session_id(),
OffsetDateTime::now_utc(),
)
.map_err(|error| {
warn!(
%request_id,
user_id = %claims.user_id(),
session_id = %claims.session_id(),
error = %error,
"Bearer JWT refresh session 状态读取失败"
);
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
})?
};
if !session_is_active {
warn!(
%request_id,
@@ -15,6 +15,13 @@ pub async fn get_public_user_by_code(
Extension(request_context): Extension<RequestContext>,
Path(code): Path<String>,
) -> Result<Json<serde_json::Value>, AppError> {
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新认证状态失败:{error}"))
})?;
let user = state
.password_entry_service()
.get_user_by_public_user_code(&code)
@@ -41,6 +48,14 @@ pub async fn get_public_user_by_id(
return Err(AppError::from_status(StatusCode::BAD_REQUEST).with_message("用户 ID 不能为空"));
}
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新认证状态失败:{error}"))
})?;
let user = state
.auth_user_service()
.get_user_by_id(user_id)
+8 -4
View File
@@ -529,21 +529,24 @@ async fn finalize_shutdown(context: ShutdownContext) {
}
if let Some(outbox) = context.wallet_refund_outbox {
info!(timeout_ms, "api-server 退出前 flush wallet refund outbox");
info!(
timeout_ms,
"api-server 退出前 flush wallet refund emergency spool"
);
match timeout(context.outbox_flush_timeout, outbox.flush_for_shutdown()).await {
Ok(Ok(())) => {
info!("api-server 退出前 wallet refund outbox flush 完成");
info!("api-server 退出前 wallet refund emergency spool flush 完成");
}
Ok(Err(error)) => {
warn!(
error = %error,
"api-server 退出前 wallet refund outbox flush 未完成,已保留本地文件等待下次启动重试"
"api-server 退出前 wallet refund emergency spool flush 未完成,已保留本地文件等待下次启动重试"
);
}
Err(_) => {
warn!(
timeout_ms,
"api-server 退出前 wallet refund outbox flush 超时,已保留本地文件等待下次启动重试"
"api-server 退出前 wallet refund emergency spool flush 超时,已保留本地文件等待下次启动重试"
);
}
}
@@ -557,6 +560,7 @@ fn spawn_common_app_state_background_workers(state: &AppState) {
if let Some(outbox) = state.wallet_refund_outbox() {
outbox.spawn_worker();
}
state.profile_wallet_refund_outbox_worker().spawn_worker();
}
fn spawn_http_app_state_background_workers(state: &AppState, process_role: ProcessRole) {
@@ -27,6 +27,13 @@ pub async fn password_entry(
headers: HeaderMap,
Json(payload): Json<PasswordEntryRequest>,
) -> Result<impl IntoResponse, AppError> {
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新认证状态失败:{error}"))
})?;
let input = PasswordEntryInput {
country_code: payload.country_code,
pure_phone_number: payload.pure_phone_number,
@@ -9,6 +9,7 @@ use shared_contracts::auth::{
PasswordChangeRequest, PasswordChangeResponse, PasswordResetRequest, PasswordResetResponse,
};
use time::OffsetDateTime;
use tracing::warn;
use crate::{
api_response::json_success_body,
@@ -81,7 +82,17 @@ pub async fn reset_password(
);
}
let result = state
// reset_password 消费的是跨节点共享的短期验证码;先恢复正式投影,
// 避免发码节点与消费节点的本机工作集不一致。
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新短信验证码状态失败:{error}"))
})?;
let result = match state
.phone_auth_service()
.reset_password(
ResetPasswordInput {
@@ -93,7 +104,17 @@ pub async fn reset_password(
OffsetDateTime::now_utc(),
)
.await
.map_err(map_phone_auth_error)?;
{
Ok(result) => result,
Err(error) => {
if let Err(sync_error) = state.sync_auth_store_tables_to_spacetime().await {
warn!(error = %sync_error, "重置密码失败后的短信验证码状态同步失败");
return Err(AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message("同步短信验证码状态失败"));
}
return Err(map_phone_auth_error(error));
}
};
let session_client = resolve_session_client_context(&headers);
let signed_session = create_auth_session(
&state,
+80 -20
View File
@@ -50,16 +50,33 @@ pub async fn send_phone_code(
phone_input_masked = phone_input_masked.as_str(),
"收到手机号验证码发送请求"
);
let send_input = SendPhoneCodeInput {
country_code: payload.country_code,
pure_phone_number: payload.pure_phone_number,
scene: scene.clone(),
};
let send_now = OffsetDateTime::now_utc();
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新短信验证码状态失败:{error}"))
})?;
state
.phone_auth_service()
.reserve_code_send(&send_input, send_now)
.map_err(map_phone_auth_error)?;
state
.sync_auth_store_tables_to_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("占用短信验证码发送窗口失败:{error}"))
})?;
let result = match state
.phone_auth_service()
.send_code(
SendPhoneCodeInput {
country_code: payload.country_code,
pure_phone_number: payload.pure_phone_number,
scene: scene.clone(),
},
OffsetDateTime::now_utc(),
)
.send_code_after_authoritative_reservation(send_input, send_now)
.await
{
Ok(result) => {
@@ -91,6 +108,14 @@ pub async fn send_phone_code(
}
};
state
.sync_auth_store_tables_to_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("同步短信验证码状态失败:{error}"))
})?;
Ok(json_success_body(
Some(&request_context),
PhoneSendCodeResponse {
@@ -114,19 +139,44 @@ pub async fn phone_login(
AppError::from_status(StatusCode::BAD_REQUEST).with_message("手机号登录暂未启用")
);
}
let invite_code = payload.invite_code.clone();
let result = match state
.phone_auth_service()
.login(
PhoneLoginInput {
country_code: payload.country_code,
pure_phone_number: payload.pure_phone_number,
verify_code: payload.code,
},
OffsetDateTime::now_utc(),
)
state
.refresh_auth_store_from_spacetime()
.await
{
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新短信验证码状态失败:{error}"))
})?;
let invite_code = payload.invite_code.clone();
let login_input = PhoneLoginInput {
country_code: payload.country_code,
pure_phone_number: payload.pure_phone_number,
verify_code: payload.code,
};
let login_now = OffsetDateTime::now_utc();
let login_result = state
.phone_auth_service()
.login(login_input.clone(), login_now)
.await;
let login_result = match login_result {
Err(PhoneAuthError::VerifyCodeNotFound) => {
if let Err(sync_error) = state.refresh_auth_store_from_spacetime().await {
warn!(
request_id = request_context.request_id(),
operation = request_context.operation(),
error = %sync_error,
"手机号验证码未命中后的认证投影刷新失败"
);
return Err(AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message("刷新短信验证码状态失败"));
}
state
.phone_auth_service()
.login(login_input, login_now)
.await
}
result => result,
};
let result = match login_result {
Ok(result) => {
info!(
request_id = request_context.request_id(),
@@ -142,6 +192,16 @@ pub async fn phone_login(
result
}
Err(error) => {
if let Err(sync_error) = state.sync_auth_store_tables_to_spacetime().await {
warn!(
request_id = request_context.request_id(),
operation = request_context.operation(),
error = %sync_error,
"手机号验证码登录失败后的状态同步失败"
);
return Err(AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message("同步短信验证码状态失败"));
}
warn!(
request_id = request_context.request_id(),
operation = request_context.operation(),
@@ -265,6 +265,14 @@ async fn process_expired_virtual_payment_order(
state: &AppState,
order: &RuntimeProfileRechargeOrderRecord,
) -> Result<(), ExpirationCompensationError> {
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
ExpirationCompensationError::Runtime(format!(
"failed to refresh auth projection for virtual payment query: {error}"
))
})?;
let identity = state
.wechat_auth_service()
.get_identity_by_user_id(&order.user_id)
@@ -39,6 +39,16 @@ pub async fn refresh_session(
let next_refresh_token = platform_auth::create_refresh_session_token();
let next_refresh_token_hash = hash_refresh_session_token(&next_refresh_token);
// refresh_session 是跨节点的正式认证入口;先加载最新投影,再在本机工作集执行领域轮换,
// 避免请求落到旧节点时把合法 refresh cookie 误判为不存在。
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(axum::http::StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新认证状态失败:{error}"))
})?;
let rotated = match state.refresh_session_service().rotate_session(
RotateRefreshSessionInput {
refresh_token_hash: refresh_token_hash.clone(),
@@ -1650,6 +1650,13 @@ async fn resolve_wechat_identity_for_payment(
state: &AppState,
user_id: &str,
) -> Result<String, AppError> {
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新微信认证状态失败:{error}"))
})?;
if let Some(identity) = state
.wechat_auth_service()
.get_identity_by_user_id(user_id)
+203 -27
View File
@@ -8,7 +8,7 @@ use std::{
fmt,
sync::{
Arc,
atomic::{AtomicBool, Ordering},
atomic::{AtomicBool, AtomicI64, AtomicU64, Ordering},
},
};
@@ -35,7 +35,7 @@ use spacetime_client::{
SpacetimeClient, SpacetimeClientConfig, SpacetimeClientError, SpacetimeClientHealthSnapshot,
};
use time::OffsetDateTime;
use tokio::sync::{Semaphore, broadcast};
use tokio::sync::{Mutex as AsyncMutex, Semaphore, broadcast};
use tracing::{info, warn};
use crate::config::AppConfig;
@@ -45,7 +45,7 @@ use crate::editor_generation_config::{
EditorGenerationPricingUnit,
};
use crate::tracking_outbox::TrackingOutbox;
use crate::wallet_refund_outbox::WalletRefundOutbox;
use crate::wallet_refund_outbox::{ProfileWalletRefundOutboxWorker, WalletRefundOutbox};
use crate::wechat::pay::{build_wechat_pay_config, map_wechat_pay_init_error};
use crate::wechat::provider::build_wechat_provider;
use crate::work_author::{
@@ -273,6 +273,14 @@ pub struct AppStateInner {
oss_client: Option<OssClient>,
#[cfg_attr(test, allow(dead_code))]
auth_store: InMemoryAuthStore,
/// 当前进程工作集所基于的正式认证投影版本;跨节点写入使用它做 CAS。
#[cfg_attr(test, allow(dead_code))]
auth_projection_version: AtomicI64,
/// 最近一次确认写入正式投影时对应的工作集 revision;不一致表示有待重试的本地变更。
#[cfg_attr(test, allow(dead_code))]
auth_projection_synced_revision: AtomicU64,
#[cfg_attr(test, allow(dead_code))]
auth_projection_sync_lock: AsyncMutex<()>,
password_entry_service: PasswordEntryService,
refresh_session_service: RefreshSessionService,
auth_user_service: AuthUserService,
@@ -289,6 +297,7 @@ pub struct AppStateInner {
puzzle_gallery_cache: PuzzleGalleryCache,
tracking_outbox: Option<Arc<TrackingOutbox>>,
wallet_refund_outbox: Option<Arc<WalletRefundOutbox>>,
profile_wallet_refund_outbox_worker: Arc<ProfileWalletRefundOutboxWorker>,
editor_generation_pricing_store: EditorGenerationPricingStore,
llm_client: Option<LlmClient>,
vector_engine_llm_client: Option<LlmClient>,
@@ -505,12 +514,13 @@ impl AppState {
pub fn new_with_empty_auth_store(config: AppConfig) -> Result<Self, AppStateInitError> {
// 中文注释:api-server 不再把本地 auth-store.json 当作用户认证真相源,启动恢复只允许来自 SpacetimeDB。
Self::new_with_auth_store(config, InMemoryAuthStore::default())
Self::new_with_auth_store(config, InMemoryAuthStore::default(), 0)
}
fn new_with_auth_store(
config: AppConfig,
auth_store: InMemoryAuthStore,
auth_projection_version: i64,
) -> Result<Self, AppStateInitError> {
let auth_jwt_config = JwtConfig::new(
config.jwt_issuer.clone(),
@@ -561,7 +571,11 @@ impl AppState {
ORPHAN_WORK_AUTHOR_PUBLIC_USER_CODE,
)
.map_err(|error| AppStateInitError::AuthStore(error.to_string()))?;
let phone_auth_service = PhoneAuthService::new(auth_store.clone(), sms_provider);
let phone_auth_service = PhoneAuthService::new_with_verify_code_salt(
auth_store.clone(),
sms_provider,
config.jwt_secret.clone(),
);
let wechat_auth_state_service =
WechatAuthStateService::new(auth_store.clone(), config.wechat_state_ttl_minutes);
let wechat_auth_service = WechatAuthService::new(auth_store.clone());
@@ -577,6 +591,8 @@ impl AppState {
let tracking_outbox = TrackingOutbox::from_config(&config, spacetime_client.clone());
let wallet_refund_outbox =
WalletRefundOutbox::from_config(&config, spacetime_client.clone());
let profile_wallet_refund_outbox_worker =
ProfileWalletRefundOutboxWorker::from_config(&config, spacetime_client.clone());
let editor_generation_pricing_store = EditorGenerationPricingStore::load(
config.editor_generation_pricing_override_path.clone(),
)
@@ -602,6 +618,10 @@ impl AppState {
let editor_oss_http_client = build_editor_oss_http_client()?;
let http_request_permit_pools = HttpRequestPermitPools::from_config(&config);
let (profile_recharge_order_updates, _) = broadcast::channel(128);
// `ensure_orphan_work_owner_user` 只为公开作品作者回退提供进程内占位账号,
// 不属于正式认证投影;将当前工作集 revision 作为已同步起点,首次认证请求会
// 先按正式投影刷新并自然丢弃该占位账号,避免把它误当成待提交认证变更。
let initial_auth_store_revision = auth_store.revision();
Ok(Self(Arc::new(AppStateInner {
config,
@@ -629,6 +649,9 @@ impl AppState {
test_external_background_removal_enqueue: Arc::new(Mutex::new(None)),
oss_client,
auth_store,
auth_projection_version: AtomicI64::new(auth_projection_version),
auth_projection_synced_revision: AtomicU64::new(initial_auth_store_revision),
auth_projection_sync_lock: AsyncMutex::new(()),
password_entry_service,
refresh_session_service,
auth_user_service,
@@ -644,6 +667,7 @@ impl AppState {
puzzle_gallery_cache: PuzzleGalleryCache::new(),
tracking_outbox,
wallet_refund_outbox,
profile_wallet_refund_outbox_worker,
editor_generation_pricing_store,
llm_client,
vector_engine_llm_client,
@@ -1288,30 +1312,138 @@ impl AppState {
return Ok(());
#[cfg(not(test))]
let updated_at_micros = i64::try_from(
OffsetDateTime::now_utc().unix_timestamp_nanos() / 1_000,
)
.map_err(|_| SpacetimeClientError::Runtime("认证状态更新时间超出 i64 范围".to_string()))?;
let _sync_guard = self.auth_projection_sync_lock.lock().await;
#[cfg(not(test))]
let projection = self
.auth_store
.export_projection_view(updated_at_micros)
.map_err(SpacetimeClientError::Runtime)?;
// 当前仍由 module-auth 的进程内工作集执行业务规则;这里只用 typed projection 同步正式认证表。
#[cfg(not(test))]
if let Err(error) = self
.spacetime_client
.sync_auth_store_projection(projection)
.await
{
warn!(
error = %error,
"认证投影同步 SpacetimeDB 正式表失败,当前认证流程中止"
);
return Err(error);
for attempt in 0..3 {
let base_updated_at_micros = self.auth_projection_version.load(Ordering::Acquire);
let now_updated_at_micros =
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos() / 1_000).map_err(
|_| SpacetimeClientError::Runtime("认证状态更新时间超出 i64 范围".to_string()),
)?;
let updated_at_micros = if now_updated_at_micros > base_updated_at_micros {
now_updated_at_micros
} else {
base_updated_at_micros.checked_add(1).ok_or_else(|| {
SpacetimeClientError::Runtime("认证状态版本超出 i64 范围".to_string())
})?
};
let (mut projection, attempted_revision) = self
.auth_store
.export_projection_view_with_revision(updated_at_micros)
.map_err(SpacetimeClientError::Runtime)?;
projection.base_updated_at_micros = base_updated_at_micros;
// 当前仍由 module-auth 的进程内工作集执行业务规则;这里只用 typed projection 同步正式认证表。
match self
.spacetime_client
.sync_auth_store_projection(projection)
.await
{
Ok(_) => {
self.auth_projection_version
.store(updated_at_micros, Ordering::Release);
if self.auth_store.revision() == attempted_revision {
self.auth_projection_synced_revision
.store(attempted_revision, Ordering::Release);
return Ok(());
}
warn!(
attempt,
"认证投影同步期间工作集发生变化,将继续同步最新工作集"
);
continue;
}
Err(error) => {
warn!(
error = %error,
"认证投影同步 SpacetimeDB 正式表失败,当前认证流程中止"
);
// 当前请求已经失败;只要同步尝试期间没有新的本地变更,恢复为
// 数据库快照,避免一次 CAS 冲突把本节点永久留在“待同步”状态。
if self.auth_store.revision() != attempted_revision {
warn!(
"认证投影同步失败期间工作集发生并发变化,跳过自动恢复以避免覆盖未提交变更"
);
} else if let Ok(current_projection) = self
.spacetime_client
.export_auth_store_projection_from_tables()
.await
{
match self.auth_store.refresh_from_projection_view_if_revision(
current_projection.clone(),
attempted_revision,
) {
Ok(true) => {
self.auth_projection_version
.store(current_projection.updated_at_micros, Ordering::Release);
self.auth_projection_synced_revision
.store(self.auth_store.revision(), Ordering::Release);
}
Ok(false) => {
warn!(
"认证投影同步冲突期间工作集发生并发变化,跳过自动恢复以避免覆盖未提交变更"
);
}
Err(refresh_error) => {
warn!(
error = %refresh_error,
"认证投影同步冲突后恢复进程内工作集失败"
);
}
}
}
return Err(error);
}
}
}
#[cfg(not(test))]
Ok(())
Err(SpacetimeClientError::Runtime(
"认证工作集在同步期间持续发生变化,未能完成投影同步".to_string(),
))
}
/// 在认证主链路执行前,从正式投影刷新一次本地工作集,避免请求落到另一节点后
/// 因本机工作集滞后而必须依赖粘性会话才能成功。
pub async fn refresh_auth_store_from_spacetime(&self) -> Result<(), SpacetimeClientError> {
#[cfg(test)]
return Ok(());
#[cfg(not(test))]
{
// 上一次业务操作可能已经改了工作集,但在返回响应前遇到数据库暂时不可用。
// 先重试提交这份待同步变更,避免只读请求把节点永久卡在 pending 状态。
if self.auth_projection_synced_revision.load(Ordering::Acquire)
!= self.auth_store.revision()
{
self.sync_auth_store_tables_to_spacetime().await?;
}
let _sync_guard = self.auth_projection_sync_lock.lock().await;
let expected_revision = self.auth_store.revision();
if self.auth_projection_synced_revision.load(Ordering::Acquire) != expected_revision {
return Err(SpacetimeClientError::Runtime(
"认证工作集存在待同步变更,跳过只读刷新".to_string(),
));
}
let projection = self
.spacetime_client
.export_auth_store_projection_from_tables()
.await?;
let updated_at_micros = projection.updated_at_micros;
let refreshed = self
.auth_store
.refresh_from_projection_view_if_revision(projection, expected_revision)
.map_err(SpacetimeClientError::Runtime)?;
if !refreshed {
return Err(SpacetimeClientError::Runtime(
"认证工作集刷新期间发生并发变更".to_string(),
));
}
self.auth_projection_version
.store(updated_at_micros, Ordering::Release);
self.auth_projection_synced_revision
.store(self.auth_store.revision(), Ordering::Release);
Ok(())
}
}
pub async fn try_restore_auth_store_from_spacetime(
@@ -1319,6 +1451,11 @@ impl AppState {
) -> Result<Self, AppStateInitError> {
let spacetime_client =
SpacetimeClient::new(spacetime_client_config_for_startup_restore(&config));
initialize_editor_generation_runtime_service_identity_for_startup(
&config,
&spacetime_client,
)
.await?;
let mut spacetime_restore_available = false;
let mut restore_errors = Vec::new();
@@ -1332,7 +1469,11 @@ impl AppState {
projection,
AuthStoreRestoreSource::SpacetimeTables,
)? {
let state = Self::new_with_auth_store(config, candidate.auth_store)?;
let state = Self::new_with_auth_store(
config,
candidate.auth_store,
candidate.updated_at_micros.unwrap_or_default(),
)?;
info!(
source = candidate.source.as_str(),
updated_at_micros = candidate.updated_at_micros,
@@ -1415,6 +1556,10 @@ impl AppState {
self.wallet_refund_outbox.clone()
}
pub fn profile_wallet_refund_outbox_worker(&self) -> Arc<ProfileWalletRefundOutboxWorker> {
self.profile_wallet_refund_outbox_worker.clone()
}
pub fn llm_client(&self) -> Option<&LlmClient> {
self.llm_client.as_ref()
}
@@ -1845,6 +1990,9 @@ fn auth_store_candidate_from_projection_view(
if projection.users.is_empty()
&& projection.identities.is_empty()
&& projection.refresh_sessions.is_empty()
&& projection.phone_codes.is_empty()
&& projection.wechat_states.is_empty()
&& projection.updated_at_micros == 0
{
return Ok(None);
}
@@ -1886,6 +2034,34 @@ fn spacetime_client_config_for_startup_restore(config: &AppConfig) -> SpacetimeC
}
}
async fn initialize_editor_generation_runtime_service_identity_for_startup(
config: &AppConfig,
spacetime_client: &SpacetimeClient,
) -> Result<(), AppStateInitError> {
let pricing_store =
EditorGenerationPricingStore::load(config.editor_generation_pricing_override_path.clone())
.map_err(|error| AppStateInitError::DependencyUnavailable(error.to_string()))?;
let fallback = pricing_store
.snapshot()
.map_err(|error| AppStateInitError::DependencyUnavailable(error.to_string()))?;
let models = editor_generation_pricing_to_records(&fallback)
.map_err(|error| AppStateInitError::DependencyUnavailable(error.to_string()))?;
spacetime_client
.initialize_editor_generation_pricing_config_if_missing(
editor_generation_pricing_upsert_input(
config,
"system:editor-generation-pricing".to_string(),
models,
crate::editor_project::current_utc_micros(),
),
)
.await
.map_err(|error| {
AppStateInitError::DependencyUnavailable(format!("初始化模型定价服务身份失败:{error}"))
})?;
Ok(())
}
impl fmt::Display for AppStateInitError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
@@ -140,6 +140,10 @@ impl TrackingOutbox {
pub fn spawn_worker(self: Arc<Self>) {
tokio::spawn(async move {
if let Err(error) = self.flush_sealed_files_once().await {
warn!(error = %error, "tracking outbox 启动恢复写入 SpacetimeDB 失败,将保留文件等待重试");
}
loop {
tokio::select! {
_ = sleep(self.flush_interval) => {
@@ -657,6 +661,45 @@ mod tests {
let _ = std::fs::remove_dir_all(dir);
}
#[tokio::test]
async fn worker_flushes_existing_active_file_immediately_on_startup() {
let dir = test_dir("worker-startup");
std::fs::create_dir_all(&dir).unwrap();
let active_path = dir.join(ACTIVE_FILE_NAME);
let record = TrackingOutboxRecord {
event: sample_event("startup-event"),
};
std::fs::write(&active_path, serde_json::to_vec(&record).unwrap()).unwrap();
let outbox = test_outbox(dir.clone(), 500, 1024 * 1024);
outbox.spawn_worker();
for _ in 0..100 {
if !active_path.exists() {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
assert!(
!active_path.exists(),
"worker should recover active file without waiting for interval"
);
let sealed_count = std::fs::read_dir(&dir)
.unwrap()
.filter_map(Result::ok)
.filter(|entry| {
entry
.file_name()
.to_str()
.is_some_and(|name| name.starts_with(SEALED_FILE_PREFIX))
})
.count();
assert_eq!(sealed_count, 1);
let _ = std::fs::remove_dir_all(dir);
}
#[test]
fn directory_size_excludes_quarantined_corrupt_files() {
let dir = test_dir("directory-size");
File diff suppressed because it is too large Load Diff
+108 -5
View File
@@ -16,6 +16,7 @@ use shared_contracts::auth::{
};
use shared_kernel::normalize_optional_string;
use time::OffsetDateTime;
use tracing::warn;
use url::Url;
use crate::{
@@ -42,6 +43,13 @@ pub async fn start_wechat_login(
if !state.config.wechat_auth_enabled {
return Err(AppError::from_status(StatusCode::BAD_REQUEST).with_message("微信登录暂未启用"));
}
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新微信认证状态失败:{error}"))
})?;
let user_agent = headers
.get("user-agent")
.and_then(|value| value.to_str().ok())
@@ -62,6 +70,13 @@ pub async fn start_wechat_login(
OffsetDateTime::now_utc(),
)
.map_err(map_wechat_auth_error)?;
state
.sync_auth_store_tables_to_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("同步微信登录状态失败:{error}"))
})?;
let authorization_url = state
.wechat_provider()
.build_authorization_url(
@@ -107,6 +122,13 @@ pub async fn start_wechat_bind(
OffsetDateTime::now_utc(),
)
.map_err(map_wechat_auth_error)?;
state
.sync_auth_store_tables_to_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("同步微信绑定状态失败:{error}"))
})?;
let authorization_url = state
.wechat_provider()
.build_authorization_url(
@@ -149,11 +171,59 @@ pub async fn handle_wechat_callback(
.into_response());
}
let consumed = match state
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
warn!(
request_id = request_context.request_id(),
operation = request_context.operation(),
error = %error,
"微信回调前刷新认证投影失败"
);
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message("刷新微信认证状态失败")
})?;
let consume_result = state
.wechat_auth_state_service()
.consume_state(&state_token, OffsetDateTime::now_utc())
{
.consume_state(&state_token, OffsetDateTime::now_utc());
let consumed = match consume_result {
Ok(value) => value,
Err(WechatAuthError::StateNotFound) => {
if let Err(error) = state.refresh_auth_store_from_spacetime().await {
warn!(
request_id = request_context.request_id(),
operation = request_context.operation(),
error = %error,
"微信 state 未命中后的认证投影刷新失败"
);
return Ok(Redirect::to(&build_auth_result_redirect_url(
&fallback_redirect,
&[
("auth_provider", "wechat"),
("auth_error", "微信登录状态已失效,请重新发起登录。"),
],
))
.into_response());
}
match state
.wechat_auth_state_service()
.consume_state(&state_token, OffsetDateTime::now_utc())
{
Ok(value) => value,
Err(_) => {
return Ok(Redirect::to(&build_auth_result_redirect_url(
&fallback_redirect,
&[
("auth_provider", "wechat"),
("auth_error", "微信登录状态已失效,请重新发起登录。"),
],
))
.into_response());
}
}
}
Err(_) => {
return Ok(Redirect::to(&build_auth_result_redirect_url(
&fallback_redirect,
@@ -165,6 +235,22 @@ pub async fn handle_wechat_callback(
.into_response());
}
};
if let Err(error) = state.sync_auth_store_tables_to_spacetime().await {
warn!(
request_id = request_context.request_id(),
operation = request_context.operation(),
error = %error,
"微信回调消费 state 后同步失败"
);
return Ok(Redirect::to(&build_auth_result_redirect_url(
&fallback_redirect,
&[
("auth_provider", "wechat"),
("auth_error", "微信登录服务暂时不可用,请稍后重试。"),
],
))
.into_response());
}
let redirect_path = consumed.state.redirect_path.clone();
let session_client = resolve_session_client_context(&headers);
@@ -293,7 +379,7 @@ pub async fn bind_wechat_phone(
.ok_or_else(|| {
AppError::from_status(StatusCode::BAD_REQUEST).with_message("缺少短信验证码")
})?;
state
match state
.phone_auth_service()
.bind_wechat_phone(
BindWechatPhoneInput {
@@ -306,7 +392,17 @@ pub async fn bind_wechat_phone(
OffsetDateTime::now_utc(),
)
.await
.map_err(map_wechat_bind_phone_error)?
{
Ok(result) => result,
Err(error) => {
if let Err(sync_error) = state.sync_auth_store_tables_to_spacetime().await {
warn!(error = %sync_error, "微信绑定手机号失败后的短信验证码状态同步失败");
return Err(AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message("同步短信验证码状态失败"));
}
return Err(map_wechat_bind_phone_error(error));
}
}
};
let session_client = resolve_session_client_context(&headers);
let signed_session = create_auth_session(
@@ -365,6 +461,13 @@ pub async fn login_wechat_mini_program(
if !state.config.wechat_auth_enabled {
return Err(AppError::from_status(StatusCode::BAD_REQUEST).with_message("微信登录暂未启用"));
}
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新微信认证状态失败:{error}"))
})?;
let code = payload.code.trim();
if code.is_empty() {
return Err(
@@ -239,6 +239,10 @@ async fn confirm_virtual_payment_recharge_order(
));
}
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| WechatPayError::Upstream(format!("刷新认证状态失败:{error}")))?;
let identity = state
.wechat_auth_service()
.get_identity_by_user_id(&order.user_id)
@@ -56,6 +56,13 @@ async fn send_generation_result_subscribe_message(
AppError::from_status(StatusCode::SERVICE_UNAVAILABLE)
.with_message("微信订阅消息模板 ID 未配置")
})?;
state
.refresh_auth_store_from_spacetime()
.await
.map_err(|error| {
AppError::from_status(StatusCode::INTERNAL_SERVER_ERROR)
.with_message(format!("刷新微信认证状态失败:{error}"))
})?;
let user = state
.auth_user_service()
.get_user_by_id(&message.owner_user_id)
+34 -2
View File
@@ -81,7 +81,7 @@ pub struct PhoneNumberSnapshot {
}
/// 手机验证码使用场景。
#[derive(Clone, Debug, PartialEq, Eq)]
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum PhoneAuthScene {
Login,
BindPhone,
@@ -101,7 +101,7 @@ impl PhoneAuthScene {
}
/// 微信授权入口场景。
#[derive(Clone, Debug, PartialEq, Eq)]
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub enum WechatAuthScene {
Desktop,
WechatInApp,
@@ -187,6 +187,13 @@ pub struct AuthStoreProjectionView {
pub users: Vec<AuthStoreProjectionUser>,
pub identities: Vec<AuthStoreProjectionIdentity>,
pub refresh_sessions: Vec<AuthStoreProjectionRefreshSession>,
#[serde(default)]
pub phone_codes: Vec<AuthStoreProjectionPhoneCode>,
#[serde(default)]
pub wechat_states: Vec<AuthStoreProjectionWechatState>,
/// 当前进程工作集所基于的正式投影版本,用于事务内 CAS。
#[serde(default)]
pub base_updated_at_micros: i64,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
@@ -230,6 +237,31 @@ pub struct AuthStoreProjectionRefreshSession {
pub last_seen_at: String,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct AuthStoreProjectionPhoneCode {
pub phone_number: String,
pub scene: String,
pub verify_code_hash: String,
pub expires_at: String,
pub last_sent_at: String,
pub failed_attempts: u32,
pub provider_out_id: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct AuthStoreProjectionWechatState {
pub wechat_state_id: String,
pub state_token: String,
pub redirect_path: String,
pub scene: String,
pub request_user_agent: Option<String>,
pub bind_user_id: Option<String>,
pub expires_at: String,
pub consumed_at: Option<String>,
pub created_at: String,
pub updated_at: String,
}
pub fn validate_password(password: &str) -> Result<(), PasswordEntryError> {
let length = password.chars().count();
if !(PASSWORD_MIN_LENGTH..=PASSWORD_MAX_LENGTH).contains(&length) {
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More