宿主终态由事实判定:模型自报失败投影进既有错误通道,失败载荷不再被收尾阶段吞掉
Project CI / AI game creator shell Rust smoke (pull_request) Successful in 1m57s
Project CI / Backend tests (pull_request) Failing after 11s
Project CI / AI game creator shell Rust crates (pull_request) Successful in 1m6s
Project CI / Frontend tests (pull_request) Successful in 2m2s
Project CI / Repository checks (pull_request) Failing after 12s
Project CI / AI game creator shell web tests (pull_request) Successful in 1m43s
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Successful in 8m33s
Project CI / Native shell tests (pull_request) Successful in 6m12s
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Successful in 9m3s
Project CI / AI game creator shell Rust smoke (pull_request) Successful in 1m57s
Project CI / Backend tests (pull_request) Failing after 11s
Project CI / AI game creator shell Rust crates (pull_request) Successful in 1m6s
Project CI / Frontend tests (pull_request) Successful in 2m2s
Project CI / Repository checks (pull_request) Failing after 12s
Project CI / AI game creator shell web tests (pull_request) Successful in 1m43s
Project CI / AI game creator shell Rust lane 1/2 (pull_request) Successful in 8m33s
Project CI / Native shell tests (pull_request) Successful in 6m12s
Project CI / AI game creator shell Rust lane 2/2 (pull_request) Successful in 9m3s
- codex_app_server 的 failed 分支先投影原生 turn.error(复用 game_creator_codex_app_server_failed_turn_error),把它当作本回合的错误结果返回:载荷形状不变,RepairRequired 保持自己的原语义,交付报告不再顶掉原因 - direct_turn_terminal 去掉 model_status 入参:判定改为「宿主当场记下的失败 -> 本回合错误结果是 Err -> 只有账本读不出来时才用交付报告」,有载荷一定写 status="failed",没载荷才用收尾阶段推出来的 status - 执行适配器把宿主观察到的失败记在适配器上(fail_turn / turn_failure / host_stop_requested):看门狗与终态判定共用同一条事实,不用调用点局部变量 - 单测:投影后的原生失败压过被收尾改写的会话状态、账本读不出来仍带载荷、宿主自己关的连接不算失败(断言改用真实原因)
This commit is contained in:
@@ -147,9 +147,11 @@ pub(super) struct ExecutionAdapter {
|
||||
changed: Notify,
|
||||
shutdown_gate: tokio::sync::Mutex<()>,
|
||||
outcome: watch::Sender<Option<HostOutcome>>,
|
||||
/// 宿主自己观察到的"执行通道断开":有值就代表本轮不是被主动收束的,终态必须是失败。
|
||||
/// 内容是给用户看的完整原因(策略句 + 宿主诊断),与交付报告同一份文本。
|
||||
transport_failure: Mutex<Option<String>>,
|
||||
/// 宿主自己判定的"本轮以失败收口":`(分类, 原因)`。有值就代表本轮终态必须是失败,
|
||||
/// 原因与交付报告同一份文本。
|
||||
turn_failure: Mutex<Option<(String, String)>>,
|
||||
/// 用户/宿主是否主动要求终止这一轮(界面的「终止」按钮)。用户主动终止不是失败。
|
||||
host_stop_requested: AtomicBool,
|
||||
}
|
||||
|
||||
fn identity(value: Option<&Value>) -> Option<&str> {
|
||||
@@ -266,7 +268,8 @@ impl ExecutionAdapter {
|
||||
changed: Notify::new(),
|
||||
shutdown_gate: tokio::sync::Mutex::new(()),
|
||||
outcome,
|
||||
transport_failure: Mutex::new(None),
|
||||
turn_failure: Mutex::new(None),
|
||||
host_stop_requested: AtomicBool::new(false),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -658,6 +661,7 @@ impl ExecutionAdapter {
|
||||
}
|
||||
|
||||
pub(super) fn cancel_from_host(self: &Arc<Self>) {
|
||||
self.request_host_stop();
|
||||
if self.background_done.load(Ordering::Acquire) || self.closed.load(Ordering::Acquire) {
|
||||
return;
|
||||
}
|
||||
@@ -676,38 +680,46 @@ impl ExecutionAdapter {
|
||||
let _ = tokio::task::spawn_blocking(move || session.interrupt(message)).await;
|
||||
}
|
||||
|
||||
/// 执行通道断开(app-server 进程退出 / 流断 / 回合事件通道关闭)时的收口入口:调用方只给
|
||||
/// 宿主诊断,文案、"这算不算失败"、报告都归这里管。
|
||||
/// 宿主判定"这一轮以失败收口":记下 `(分类, 原因)`,再把同一条原因写进宿主交付报告。
|
||||
///
|
||||
/// 谁调用:宿主亲眼看到或亲手判定的异常收场——执行通道断开(app-server 进程退出 / 流断 / 回合
|
||||
/// 事件通道关闭)、等待模型回执超时、app-server 单方面把这一轮判成中断。终态判定会读这份事实,
|
||||
/// 于是这些收场不会再被收尾阶段(`ExecutionPhase::Interrupted`)抹成一次没有原因的"已结束"。
|
||||
///
|
||||
/// **宿主自己关的连接不算失败。** 正常终态、用户主动停止、预算与交付收尾都会把连接关掉,回合
|
||||
/// 事件通道上看到的是同一个 `TransportClosed`;区分判据是 [`Self::is_closed`]——适配器先于连接
|
||||
/// 置位就说明这一轮是宿主在收束,只按既有口径中断收口(原因照样写进报告,便于核对)。
|
||||
/// 事件通道上看到的是同一个 `TransportClosed`;判据是 [`Self::is_closed`]——适配器先于连接置位
|
||||
/// 就说明这一轮是宿主在收束,只按既有口径中断收口(原因照样写进报告,便于核对)。
|
||||
///
|
||||
/// **连接自己断的才算失败,且事实要落在适配器上。** 调用点局部变量不行:回合还开着的时候,
|
||||
/// 看门狗会在同一个 `inner.closed` 标志上把本轮收束掉(见 [`Self::start_watchdog`]),谁先谁后
|
||||
/// 取决于调度,而终态判定发生在收束之后。记不下原因,界面就只能看到"本轮已结束"、看不到为什么。
|
||||
/// 所以顺序是:先**同步**记事实(终态随之判失败),再写报告。
|
||||
/// **事实要落在适配器上,不能落在调用点的局部变量里。** 回合还开着的时候,看门狗会在同一个
|
||||
/// `inner.closed` 标志上把本轮收束掉(见 [`Self::start_watchdog`]),谁先谁后取决于调度,而终态
|
||||
/// 判定发生在收束之后;记不下原因,界面就只能看到"本轮已结束"、看不到为什么。
|
||||
///
|
||||
/// 只记第一份原因:第一份最接近现场(连接终止时带 exitStatus / stderr 摘要),后面更粗的收束
|
||||
/// 理由(事件通道关闭、看门狗收尾)不得覆盖它。
|
||||
pub(super) async fn transport_failed(&self, diagnostic: &str) {
|
||||
let reason = format!("执行通道已断开,不能自动重放未确认操作:{diagnostic}");
|
||||
/// 只记第一份:第一份最接近现场(连接终止时带 exitStatus / stderr 摘要),后面更粗的收束理由
|
||||
/// 不得覆盖它。
|
||||
pub(super) async fn fail_turn(&self, kind: &str, reason: &str) {
|
||||
if !self.is_closed() {
|
||||
if let Ok(mut slot) = self.transport_failure.lock() {
|
||||
if let Ok(mut slot) = self.turn_failure.lock() {
|
||||
if slot.is_none() {
|
||||
*slot = Some(reason.clone());
|
||||
*slot = Some((kind.to_string(), reason.to_string()));
|
||||
}
|
||||
}
|
||||
}
|
||||
self.interrupt(&reason).await;
|
||||
self.interrupt(reason).await;
|
||||
}
|
||||
|
||||
/// 本轮是否以"执行通道断开"收场;有值就是宿主记下的那份原因。终态判定只读这一次。
|
||||
pub(super) fn transport_failure(&self) -> Option<String> {
|
||||
self.transport_failure
|
||||
.lock()
|
||||
.ok()
|
||||
.and_then(|slot| slot.clone())
|
||||
/// 本轮以什么理由失败;有值就是宿主记下的 `(分类, 原因)`。终态判定只读这一次。
|
||||
pub(super) fn turn_failure(&self) -> Option<(String, String)> {
|
||||
self.turn_failure.lock().ok().and_then(|slot| slot.clone())
|
||||
}
|
||||
|
||||
/// 记下"用户主动要求终止这一轮"。用来把用户主动终止与 app-server 自己中断分开:
|
||||
/// 前者不是失败,后者是(判据不能被事件到达的先后顺序左右,所以用标志而不是看阶段)。
|
||||
pub(super) fn request_host_stop(&self) {
|
||||
self.host_stop_requested.store(true, Ordering::Release);
|
||||
}
|
||||
|
||||
pub(super) fn host_stop_requested(&self) -> bool {
|
||||
self.host_stop_requested.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
pub(super) fn start_watchdog(self: &Arc<Self>, inner: Weak<CodexAppServerInner>) {
|
||||
@@ -792,11 +804,9 @@ impl ExecutionAdapter {
|
||||
}
|
||||
|
||||
pub(super) fn lifecycle_status(&self, fallback: &str) -> String {
|
||||
// 执行通道断开过的回合一律是失败:那不是本轮主动收束,也没有"用户主动停止"这层授权,
|
||||
// 报成 `interrupted` 只会让界面停在"本轮已结束"却不给原因(这就是连接被强杀时的老现象)。
|
||||
if self.transport_failure().is_some() {
|
||||
return "failed".to_string();
|
||||
}
|
||||
// 只按收尾阶段归类。失败事实(`fail_turn` 记下的)不在这里翻案:终态由
|
||||
// `direct_turn_terminal` 拿事实判定——否则"模型已经判失败"的一轮会被这里的
|
||||
// `Interrupted` 抹成一次没有原因的"已结束"。
|
||||
match self.session.snapshot().map(|state| state.phase) {
|
||||
Ok(ExecutionPhase::Completed) => "completed",
|
||||
Ok(ExecutionPhase::Exhausted | ExecutionPhase::Interrupted) => "interrupted",
|
||||
@@ -1089,6 +1099,7 @@ pub(super) async fn wait_outcome(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::{DIRECT_TURN_FAILURE_TIMEOUT_KIND, DIRECT_TURN_FAILURE_TRANSPORT_KIND};
|
||||
use super::*;
|
||||
|
||||
fn fixture() -> (tempfile::TempDir, Arc<ExecutionAdapter>) {
|
||||
@@ -1137,47 +1148,54 @@ mod tests {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn transport_failure_is_a_failed_terminal_with_the_host_diagnostic() {
|
||||
async fn host_observed_failure_is_recorded_with_its_kind_and_reason() {
|
||||
let (_temp, adapter) = fixture();
|
||||
assert_eq!(adapter.lifecycle_status("completed"), "completed");
|
||||
assert!(adapter.turn_failure().is_none());
|
||||
assert!(!adapter.host_stop_requested());
|
||||
|
||||
adapter
|
||||
.transport_failed("Codex app-server 已退出;exitStatus=signal: 9 (SIGKILL)")
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_TRANSPORT_KIND,
|
||||
"执行通道已断开:Codex app-server 已退出;exitStatus=signal: 9 (SIGKILL)",
|
||||
)
|
||||
.await;
|
||||
|
||||
// 终态判成失败:界面才有理由把它当失败讲,而不是"本轮已结束"。
|
||||
assert_eq!(adapter.lifecycle_status("completed"), "failed");
|
||||
let reason = adapter
|
||||
.transport_failure()
|
||||
.expect("host diagnostic must be recorded");
|
||||
// 终态判定读这份事实,界面才有理由把它当失败讲,而不是"本轮已结束"。
|
||||
let (kind, reason) = adapter.turn_failure().expect("host fact must be recorded");
|
||||
assert_eq!(kind, DIRECT_TURN_FAILURE_TRANSPORT_KIND);
|
||||
assert!(reason.contains("SIGKILL"));
|
||||
// 报告与事件载荷同一份原因:用户看到的现象和交付状态要对得上。
|
||||
// 报告与事件载荷同一份原因:用户看到的现象和交付状态对得上。
|
||||
assert!(adapter.report().contains("SIGKILL"));
|
||||
|
||||
// 只认第一份原因:后续更粗的收束理由不得覆盖真实诊断。
|
||||
adapter
|
||||
.transport_failed("Codex app-server turn 事件通道已关闭")
|
||||
.fail_turn(DIRECT_TURN_FAILURE_TIMEOUT_KIND, "等待模型执行回执超时")
|
||||
.await;
|
||||
let reason = adapter.transport_failure().expect("first reason is kept");
|
||||
let (kind, reason) = adapter.turn_failure().expect("first reason is kept");
|
||||
assert_eq!(kind, DIRECT_TURN_FAILURE_TRANSPORT_KIND);
|
||||
assert!(reason.contains("SIGKILL"));
|
||||
assert!(!reason.contains("事件通道已关闭"));
|
||||
assert!(!reason.contains("超时"));
|
||||
}
|
||||
|
||||
/// 宿主自己关的连接不算传输失败:正常终态、用户主动停止、预算与交付收尾都会关掉连接,回合事件
|
||||
/// 通道上看到的是同一个 `TransportClosed`。判据是适配器先于连接置位 `closed`。
|
||||
/// 宿主自己关的连接不算失败:正常终态、用户主动停止、预算与交付收尾都会关掉连接,回合事件通道
|
||||
/// 上看到的是同一个 `TransportClosed`。判据是适配器先于连接置位 `closed`。
|
||||
#[tokio::test]
|
||||
async fn host_ended_turn_is_not_a_transport_failure() {
|
||||
async fn host_ended_turn_is_not_a_failure() {
|
||||
let (_temp, adapter) = fixture();
|
||||
adapter.request_host_stop();
|
||||
adapter.closed.store(true, Ordering::Release);
|
||||
|
||||
adapter
|
||||
.transport_failed("模型本次执行结束,回收原生后台子树")
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_TRANSPORT_KIND,
|
||||
"执行通道已断开:模型本次执行结束,回收原生后台子树",
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(adapter.transport_failure().is_none());
|
||||
assert_ne!(adapter.lifecycle_status("completed"), "failed");
|
||||
assert!(adapter.turn_failure().is_none());
|
||||
assert!(adapter.host_stop_requested());
|
||||
// 原因照样进报告:不算失败不等于不用记。
|
||||
assert!(adapter.report().contains("不能自动重放未确认操作"));
|
||||
assert!(adapter.report().contains("模型本次执行结束"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -3602,8 +3602,12 @@ impl CodexAppServerConnection {
|
||||
hard_deadline.saturating_duration_since(tokio::time::Instant::now());
|
||||
if remaining.is_zero() {
|
||||
if let Some(adapter) = approval_adapter.as_ref() {
|
||||
// 等不到终态就是这一轮失败:只收口不留原因等于界面静默结束。
|
||||
adapter
|
||||
.interrupt("等待模型回合结束达到硬上限,已停止本轮并核对后台操作。")
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_TIMEOUT_KIND,
|
||||
"等待模型回合结束达到硬上限,已停止本轮并核对后台操作。",
|
||||
)
|
||||
.await;
|
||||
return execution::outcome_text(adapter.wait_outcome().await);
|
||||
}
|
||||
@@ -3631,7 +3635,10 @@ impl CodexAppServerConnection {
|
||||
Err(_) => {
|
||||
if let Some(adapter) = approval_adapter.as_ref() {
|
||||
adapter
|
||||
.interrupt("等待模型执行回执超时,不能自动重放未确认操作。")
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_TIMEOUT_KIND,
|
||||
"等待模型执行回执超时,不能自动重放未确认操作。",
|
||||
)
|
||||
.await;
|
||||
return execution::outcome_text(adapter.wait_outcome().await);
|
||||
}
|
||||
@@ -3961,9 +3968,15 @@ impl CodexAppServerConnection {
|
||||
}
|
||||
"interrupted" => {
|
||||
if let Some(adapter) = approval_adapter.as_ref() {
|
||||
if !adapter.is_host_ending() {
|
||||
// app-server 自己把这一轮判成中断,而宿主没有请求过终止(用户点
|
||||
// 「终止」会先置 `host_stop_requested`、并把阶段推成终态):这是异常
|
||||
// 收场,必须让界面看到原因,不能只是把回合静默收口。
|
||||
if !adapter.is_host_ending() && !adapter.host_stop_requested() {
|
||||
adapter
|
||||
.interrupt("本轮模型执行已中断,正在核对自有后台进程。")
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_INTERRUPTED_KIND,
|
||||
"本轮模型执行被中断,正在核对自有后台进程。",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
return execution::outcome_text(adapter.wait_outcome().await);
|
||||
@@ -3976,14 +3989,23 @@ impl CodexAppServerConnection {
|
||||
));
|
||||
}
|
||||
"failed" => {
|
||||
// 原生 `turn.error` 是这一轮最准的原因:先把它投影成 `LlmError`,
|
||||
// 再作为 `collect` 的错误结果走既有的 collect_result 通道。投影之后
|
||||
// 载荷形状(`{kind, message}`)和终态判定都不用为此多一个入参,
|
||||
// 原因文本里带着 `codex-app-server-error:<kind>` 前缀交给界面归类。
|
||||
// 交付报告只说明"收束到哪一步",不能顶掉原因;返修请求
|
||||
// (`RepairRequired`)是宿主复核要求,保持它自己的原语义。
|
||||
let native = game_creator_codex_app_server_failed_turn_error(turn);
|
||||
if let Some(adapter) = approval_adapter.as_ref() {
|
||||
if let Some(outcome) =
|
||||
adapter.finish_model_attempt(&self.inner, false).await
|
||||
{
|
||||
return execution::outcome_text(outcome);
|
||||
if let Err(repair) = execution::outcome_text(outcome) {
|
||||
return Err(repair);
|
||||
}
|
||||
}
|
||||
}
|
||||
return Err(game_creator_codex_app_server_failed_turn_error(turn));
|
||||
return Err(native);
|
||||
}
|
||||
status => {
|
||||
return Err(platform_llm::LlmError::Deserialize(format!(
|
||||
@@ -3997,8 +4019,13 @@ impl CodexAppServerConnection {
|
||||
if !adapter.is_host_ending() {
|
||||
// 事件带的 `error` 就是连接终止时那份诊断。通道断开是不是'失败'由
|
||||
// 适配器判(宿主自己关的连接不算),失败事实也记在它上面,回合终态
|
||||
// 判定之后才读得到:见 `ExecutionAdapter::transport_failed`。
|
||||
adapter.transport_failed(&error).await;
|
||||
// 判定之后才读得到:见 `ExecutionAdapter::fail_turn`。
|
||||
adapter
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_TRANSPORT_KIND,
|
||||
&execution_channel_failure_reason(&error),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
return execution::outcome_text(adapter.wait_outcome().await);
|
||||
}
|
||||
@@ -4014,7 +4041,12 @@ impl CodexAppServerConnection {
|
||||
// 事件通道在没有终态的情况下关掉,和连接断掉是同一件事:本轮只可能
|
||||
// 以失败收口,不能报成"被中断"。
|
||||
adapter
|
||||
.transport_failed("Codex app-server turn 事件通道已关闭")
|
||||
.fail_turn(
|
||||
DIRECT_TURN_FAILURE_TRANSPORT_KIND,
|
||||
&execution_channel_failure_reason(
|
||||
"Codex app-server turn 事件通道已关闭",
|
||||
),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
return execution::outcome_text(adapter.wait_outcome().await);
|
||||
@@ -4076,29 +4108,23 @@ impl CodexAppServerConnection {
|
||||
// 脱敏 + 截断后写进去),其余(`completed` / `interrupted` / `aborted`)不带载荷。
|
||||
// 失败不再只写一个 `status="failed"`:那让失败与正常结束在协议上长得一样,前端只能
|
||||
// 另开一条通道(命令返回 / 另一条 IPC)去拿原因,也就等于承认事件流讲不清一轮怎么结束。
|
||||
// 通道断开的失败原因取自执行适配器(宿主亲眼看到的断连事实),不从交付报告里猜:
|
||||
// 报告只说明收束状态,说不清连接为什么没了。
|
||||
let transport_failure = approval_adapter
|
||||
// 判定拿的是**事实**(模型终态 / 交付结果 / 宿主记下的失败),不是收尾阶段推出来的
|
||||
// `status`:收尾自己会把阶段推成 `Interrupted`,用它判就会把已经失败的回合讲成"已结束"。
|
||||
let turn_failure = approval_adapter
|
||||
.as_ref()
|
||||
.and_then(|adapter| adapter.transport_failure());
|
||||
let failure = direct_turn_failure(
|
||||
.and_then(|adapter| adapter.turn_failure());
|
||||
let terminal = direct_turn_terminal(
|
||||
&status,
|
||||
collect_result.as_ref().map(String::as_str),
|
||||
transport_failure.as_deref(),
|
||||
turn_failure
|
||||
.as_ref()
|
||||
.map(|(kind, reason)| (kind.as_str(), reason.as_str())),
|
||||
history_root,
|
||||
);
|
||||
match failure {
|
||||
Some(failure) => append_direct_thread_event(
|
||||
&direct_thread_id,
|
||||
DirectThreadEvent::turn_completed_failed(failure, completed_at)
|
||||
.with_user_item_id(direct_turn_user_item_id.as_deref()),
|
||||
),
|
||||
None => append_direct_thread_event(
|
||||
&direct_thread_id,
|
||||
DirectThreadEvent::turn_completed(status, completed_at)
|
||||
.with_user_item_id(direct_turn_user_item_id.as_deref()),
|
||||
),
|
||||
};
|
||||
append_direct_thread_event(
|
||||
&direct_thread_id,
|
||||
terminal.event(completed_at, direct_turn_user_item_id.as_deref()),
|
||||
);
|
||||
if let Some(guard) = direct_turn_failure_guard.as_mut() {
|
||||
guard.disarm();
|
||||
}
|
||||
@@ -5003,7 +5029,12 @@ async fn fail_game_creator_codex_app_server_connection(
|
||||
// 连接是在回合进行中断掉的:先把"本轮以传输失败收口"和这份诊断记到执行适配器上,再去收束
|
||||
// 连接。顺序不能反——执行适配器的看门狗盯着同一个 `closed` 标志,它可能先一步把本轮收束成
|
||||
// "被中断";终态一旦算出来,失败原因就只剩日志,界面只会看到"本轮已结束、没有原因"。
|
||||
record_execution_transport_failure(&inner, &diagnostic).await;
|
||||
record_execution_turn_failure(
|
||||
&inner,
|
||||
DIRECT_TURN_FAILURE_TRANSPORT_KIND,
|
||||
&execution_channel_failure_reason(&diagnostic),
|
||||
)
|
||||
.await;
|
||||
match shutdown_game_creator_codex_app_server_inner(&inner, &diagnostic).await {
|
||||
Ok(proof) if proof.confirmed() => {}
|
||||
Ok(_) => app_log!("Codex app-server 连接终止:process-group-only,完整子树退出未确认"),
|
||||
@@ -5011,9 +5042,10 @@ async fn fail_game_creator_codex_app_server_connection(
|
||||
}
|
||||
}
|
||||
|
||||
/// 把"执行通道断开"这个失败事实记到当前回合的执行适配器上:连接级故障与回合事件通道关闭共用
|
||||
/// 这一条路径,别在两处各写一份。没有进行中的 DirectProject 回合(适配器已释放)就是空操作。
|
||||
async fn record_execution_transport_failure(inner: &Arc<CodexAppServerInner>, diagnostic: &str) {
|
||||
/// 把"这一轮以失败收口"的事实记到当前回合的执行适配器上:连接级故障、等待超时、app-server
|
||||
/// 单方面中断都走这一条路径,别在多处各写一份。没有进行中的 DirectProject 回合(适配器已释放)
|
||||
/// 就是空操作。
|
||||
async fn record_execution_turn_failure(inner: &Arc<CodexAppServerInner>, kind: &str, reason: &str) {
|
||||
let adapter = {
|
||||
let slot = match inner.execution.lock() {
|
||||
Ok(slot) => slot,
|
||||
@@ -5025,7 +5057,13 @@ async fn record_execution_transport_failure(inner: &Arc<CodexAppServerInner>, di
|
||||
let Some(adapter) = adapter else {
|
||||
return;
|
||||
};
|
||||
adapter.transport_failed(diagnostic).await;
|
||||
adapter.fail_turn(kind, reason).await;
|
||||
}
|
||||
|
||||
/// 执行通道断开的统一说明:策略句(未确认操作禁止自动重放)+ 宿主诊断。回合失败载荷与宿主交付
|
||||
/// 报告共用这一份文本:用户看到的现象和交付状态必须对得上。
|
||||
fn execution_channel_failure_reason(diagnostic: &str) -> String {
|
||||
format!("执行通道已断开,不能自动重放未确认操作:{diagnostic}")
|
||||
}
|
||||
|
||||
async fn shutdown_game_creator_codex_app_server_inner(
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
//!
|
||||
//! 这个模块只有三件事,别再往里加第四件:
|
||||
//! 1. [`direct_turn_failure_kind`]:把 `LlmError` 归到稳定分类(只给界面选语气);
|
||||
//! 2. [`direct_turn_failure`]:判定这一轮的终态是不是失败,是的话给出脱敏后的原因;
|
||||
//! 2. [`direct_turn_terminal`]:拿这一轮的事实判定终态——是不是失败、原因是什么、状态写什么;
|
||||
//! 3. [`DirectTurnFailureGuard`]:`turn.started` 之后武装、写完终态解除的 Drop 兜底。
|
||||
//!
|
||||
//! 失败载荷的**形状**属于线上协议,定义在 `direct_thread_wire.rs`(`DirectTurnFailure`);
|
||||
@@ -29,9 +29,17 @@ const DIRECT_TURN_FAILURE_HOST_DROPPED_MESSAGE: &str =
|
||||
|
||||
/// 执行通道断开的分类:宿主自己看到的事实(app-server 进程退出 / 流断 / 回合事件通道关闭),
|
||||
/// 不由 `LlmError` 反推——那种情况下宿主手里只有一份交付报告,报告里没有"连接没了"这句真话。
|
||||
const DIRECT_TURN_FAILURE_TRANSPORT_KIND: &str = "transport-failed";
|
||||
pub(crate) const DIRECT_TURN_FAILURE_TRANSPORT_KIND: &str = "transport-failed";
|
||||
|
||||
/// 稳定失败分类:`timeout` / `model-failed` / `transport-failed` / `request-rejected`。
|
||||
/// app-server 单方面把这一轮判成中断(用户没要求停止、宿主也没在收尾)时的分类:这是异常收场,
|
||||
/// 不是"被主动终止",界面必须给原因。
|
||||
pub(crate) const DIRECT_TURN_FAILURE_INTERRUPTED_KIND: &str = "turn-interrupted";
|
||||
|
||||
/// 宿主等待模型回执超时(空闲上限 / 回合硬上限)时的分类。
|
||||
pub(crate) const DIRECT_TURN_FAILURE_TIMEOUT_KIND: &str = "timeout";
|
||||
|
||||
/// 稳定失败分类:`timeout` / `model-failed` / `transport-failed` / `request-rejected` /
|
||||
/// `turn-interrupted` / `host-dropped`。
|
||||
///
|
||||
/// 分类只影响界面语气,前端不得拿它做流程分支(流程判据只有"收到终态事件"这一条)。
|
||||
fn direct_turn_failure_kind(error: &LlmError) -> &'static str {
|
||||
@@ -47,48 +55,70 @@ fn direct_turn_failure_kind(error: &LlmError) -> &'static str {
|
||||
}
|
||||
}
|
||||
|
||||
/// 这一轮的终态是不是「失败」?是的话给出失败载荷(原因已脱敏并截断)。
|
||||
/// 一轮的终态:写进事件的 `status` 与(失败时的)载荷。**状态由载荷反推**,不由收尾阶段推。
|
||||
pub(crate) struct DirectTurnTerminal {
|
||||
pub(crate) status: String,
|
||||
pub(crate) failure: Option<DirectTurnFailure>,
|
||||
}
|
||||
|
||||
impl DirectTurnTerminal {
|
||||
/// 终态事件:失败时同一个 `turn.completed` 带载荷,其余只带 `status`。
|
||||
pub(crate) fn event(self, completed_at: u64, user_item_id: Option<&str>) -> DirectThreadEvent {
|
||||
let event = match self.failure {
|
||||
Some(failure) => DirectThreadEvent::turn_completed_failed(failure, completed_at),
|
||||
None => DirectThreadEvent::turn_completed(self.status, completed_at),
|
||||
};
|
||||
event.with_user_item_id(user_item_id)
|
||||
}
|
||||
}
|
||||
|
||||
/// 拿这一轮的**事实**判定终态。判据按优先级:
|
||||
/// 1. `host_failure`:宿主自己观察 / 判定的失败(执行通道断开、等待超时、app-server 单方面中断…),
|
||||
/// 原因就用宿主当场写下的那句——它比交付报告更接近现场,报告只说明"收束到哪一步";
|
||||
/// 2. `collect_result` 是错误:真失败(模型 / 传输 / 历史落盘),原因直接从错误里取。模型自报失败
|
||||
/// 也走这一档:原生 `turn/completed.status="failed"` 的 `error` 由调用点投影成 `LlmError`,
|
||||
/// 于是原因带着 `codex-app-server-error:<kind>` 前缀进来,不用在这里多认一种输入;
|
||||
/// 3. `session_status` 已经判成 `failed`、而拿到的只是一份交付报告:原因用那份报告兜底——收尾
|
||||
/// 阶段的账本读不出来时只有它可用。
|
||||
///
|
||||
/// 失败有三个来源,都必须进 `turn.completed(status="failed")` 的 `failure` 载荷,按优先级:
|
||||
/// 1. `transport_failure` 有值:宿主亲眼看到执行通道断开(app-server 进程退出、流断、回合事件
|
||||
/// 通道关闭)。原因就是宿主记下的那份诊断(含 exitStatus / stderr 摘要),它比交付报告更接近
|
||||
/// 现场;报告只说明"收束到哪一步",说不清连接为什么没了;
|
||||
/// 2. `collect_result` 是错误:真失败(模型 / 传输 / 历史落盘),原因直接从错误里取;
|
||||
/// 3. `collect_result` 是交付报告、但 `status` 已经判成 `failed`:宿主收束了一个失败的回合,
|
||||
/// 原因用那份报告本身(它本来就是给用户看的失败说明)。
|
||||
///
|
||||
/// 其余终态(`completed` / `interrupted` / `aborted`)都不是失败,返回 `None`,事件不带载荷。
|
||||
/// 注意这里只负责"原因写什么":把 `status` 判成 `failed` 是调用方的事,通道断开必须让宿主把本轮
|
||||
/// 判失败(`ExecutionAdapter::lifecycle_status` 就是这么做的)——只补一条载荷而状态还是
|
||||
/// `interrupted`,界面照样不会把它当成失败来讲。
|
||||
pub(crate) fn direct_turn_failure(
|
||||
status: &str,
|
||||
/// **有载荷就一定是 `failed`,没载荷就用收尾阶段的 `session_status`。** 这条反推关系是这个模块存在
|
||||
/// 的理由:`session_status` 是宿主收尾时按 ledger 阶段推的,收尾本身会把阶段推成 `Interrupted`,
|
||||
/// 于是"模型已经判失败"的一轮会被写成 `status="interrupted"` 且不带载荷——界面只剩"本轮已结束",
|
||||
/// 用户看不到任何原因(连接/上游断开时就是这个现象)。事实判失败就必须报失败。
|
||||
pub(crate) fn direct_turn_terminal(
|
||||
session_status: &str,
|
||||
collect_result: Result<&str, &LlmError>,
|
||||
transport_failure: Option<&str>,
|
||||
host_failure: Option<(&str, &str)>,
|
||||
history_root: &Path,
|
||||
) -> Option<DirectTurnFailure> {
|
||||
let (kind, message) = match (transport_failure, collect_result) {
|
||||
(Some(diagnostic), _) => (
|
||||
DIRECT_TURN_FAILURE_TRANSPORT_KIND.to_string(),
|
||||
diagnostic.to_string(),
|
||||
),
|
||||
(None, Err(error)) => (
|
||||
) -> DirectTurnTerminal {
|
||||
let failure = match (host_failure, collect_result) {
|
||||
(Some((kind, reason)), _) => Some((kind.to_string(), reason.to_string())),
|
||||
(None, Err(error)) => Some((
|
||||
direct_turn_failure_kind(error).to_string(),
|
||||
error.to_string(),
|
||||
),
|
||||
(None, Ok(report)) if status == "failed" => {
|
||||
("model-failed".to_string(), report.to_string())
|
||||
)),
|
||||
(None, Ok(report)) if session_status == "failed" => {
|
||||
Some(("model-failed".to_string(), report.to_string()))
|
||||
}
|
||||
(None, Ok(_)) => return None,
|
||||
(None, Ok(_)) => None,
|
||||
};
|
||||
Some(DirectTurnFailure::new(
|
||||
kind,
|
||||
redact_agent_runtime_error(
|
||||
history_root,
|
||||
&message,
|
||||
DIRECT_TURN_FAILURE_MESSAGE_MAX_CHARS,
|
||||
),
|
||||
))
|
||||
match failure {
|
||||
Some((kind, message)) => DirectTurnTerminal {
|
||||
status: "failed".to_string(),
|
||||
failure: Some(DirectTurnFailure::new(
|
||||
kind,
|
||||
redact_agent_runtime_error(
|
||||
history_root,
|
||||
&message,
|
||||
DIRECT_TURN_FAILURE_MESSAGE_MAX_CHARS,
|
||||
),
|
||||
)),
|
||||
},
|
||||
None => DirectTurnTerminal {
|
||||
status: session_status.to_string(),
|
||||
failure: None,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// 回合终态兜底守卫:`turn.started` 发出去之后,这一轮在宿主侧只剩两条收场路径——正常路径
|
||||
@@ -199,97 +229,114 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// 正常收场:不带载荷,`status` 就用收尾阶段推出来的那个。
|
||||
#[test]
|
||||
fn only_failed_terminals_carry_a_failure_payload() {
|
||||
// 正常终态:无论交付报告写了什么都不是失败。
|
||||
assert_eq!(
|
||||
direct_turn_failure("completed", Ok("本轮交付已完成"), None, &history_root()),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
direct_turn_failure("interrupted", Ok("本轮已被终止"), None, &history_root()),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
direct_turn_failure("aborted", Ok("已结束这一轮占用"), None, &history_root()),
|
||||
None
|
||||
);
|
||||
|
||||
// 失败且拿得到错误:分类取自错误,原因取自错误文本。
|
||||
let error =
|
||||
LlmError::Transport("DirectProject 收尾历史失败:写入 project.jsonl 失败".into());
|
||||
let failure = direct_turn_failure("failed", Err(&error), None, &history_root())
|
||||
.expect("transport error must produce a failure payload");
|
||||
assert_eq!(failure.kind, "transport-failed");
|
||||
assert!(failure.message.contains("收尾历史失败"));
|
||||
|
||||
// 失败但拿到的是交付报告:宿主已经收束了这一轮,报告本身就是失败说明。
|
||||
let failure = direct_turn_failure(
|
||||
"failed",
|
||||
Ok("宿主尚未确认交付完成;请核对未完成项。"),
|
||||
None,
|
||||
&history_root(),
|
||||
)
|
||||
.expect("failed status must produce a failure payload");
|
||||
assert_eq!(failure.kind, "model-failed");
|
||||
assert_eq!(failure.message, "宿主尚未确认交付完成;请核对未完成项。");
|
||||
fn non_failure_terminals_keep_the_session_status() {
|
||||
for status in ["completed", "interrupted", "aborted"] {
|
||||
let terminal = direct_turn_terminal(status, Ok("报告不重要"), None, &history_root());
|
||||
assert!(terminal.failure.is_none(), "{status} 不该带失败载荷");
|
||||
assert_eq!(terminal.status, status);
|
||||
}
|
||||
}
|
||||
|
||||
/// 执行通道断开:连接被强杀 / 流断时宿主手里只有交付报告,但真相是连接没了。原因必须用宿主
|
||||
/// 记下的诊断,而不是那份只说"收束到哪一步"的报告——否则界面只能看到一句泛泛的收尾说明。
|
||||
/// 拿得到错误:分类与原因都取自错误。
|
||||
#[test]
|
||||
fn host_observed_transport_failure_outranks_the_delivery_report() {
|
||||
fn collect_error_becomes_a_failure_terminal() {
|
||||
let error =
|
||||
LlmError::Transport("DirectProject 收尾历史失败:写入 project.jsonl 失败".into());
|
||||
let terminal = direct_turn_terminal("completed", Err(&error), None, &history_root());
|
||||
let failure = terminal
|
||||
.failure
|
||||
.expect("transport error must fail the turn");
|
||||
assert_eq!(terminal.status, "failed");
|
||||
assert_eq!(failure.kind, "transport-failed");
|
||||
assert!(failure.message.contains("收尾历史失败"));
|
||||
}
|
||||
|
||||
/// **收尾阶段的中断不能把已经失败的一轮讲成"已结束"。** 模型自报失败在调用点被投影成
|
||||
/// `LlmError`(原因带 `codex-app-server-error:<kind>` 前缀),宿主收尾自己又把 ledger 阶段推成
|
||||
/// `Interrupted`(`session_status` 因此是 `interrupted`):事实就是失败、原因就是那份投影,
|
||||
/// 必须原样发出去——否则界面只剩"本轮已结束",用户看不到任何东西。
|
||||
#[test]
|
||||
fn projected_native_failure_outranks_the_interrupted_session_status() {
|
||||
let error =
|
||||
LlmError::InvalidRequest("codex-app-server-error:context-window-exceeded".into());
|
||||
let terminal = direct_turn_terminal("interrupted", Err(&error), None, &history_root());
|
||||
let failure = terminal.failure.expect("native failure must fail the turn");
|
||||
assert_eq!(terminal.status, "failed");
|
||||
assert_eq!(failure.kind, "request-rejected");
|
||||
assert_eq!(
|
||||
failure.message,
|
||||
"codex-app-server-error:context-window-exceeded"
|
||||
);
|
||||
}
|
||||
|
||||
/// 收尾阶段的账本读不出来(`session_status` 只能是 `failed`)时没有错误可用:用交付报告兜底,
|
||||
/// 但照样要带载荷发出去,不能让界面停在"已结束、没原因"。
|
||||
#[test]
|
||||
fn unreadable_session_ledger_still_reports_a_payload() {
|
||||
let terminal = direct_turn_terminal("failed", Ok("报告"), None, &history_root());
|
||||
assert_eq!(terminal.status, "failed");
|
||||
let failure = terminal
|
||||
.failure
|
||||
.expect("unreadable ledger must fail the turn");
|
||||
assert_eq!(failure.kind, "model-failed");
|
||||
assert_eq!(failure.message, "报告");
|
||||
}
|
||||
|
||||
/// 宿主自己记下的失败排在最前面:它比交付报告更接近现场。
|
||||
#[test]
|
||||
fn host_recorded_failure_outranks_every_other_source() {
|
||||
let diagnostic = "执行通道已断开,不能自动重放未确认操作:Codex app-server 已退出;\
|
||||
exitStatus=signal: 9 (SIGKILL);stderrClass=nonempty;stderrBytes=1000";
|
||||
let failure = direct_turn_failure(
|
||||
"failed",
|
||||
let terminal = direct_turn_terminal(
|
||||
"interrupted",
|
||||
Ok("执行连接已结束,正在核对自有子进程与在途操作。"),
|
||||
Some(diagnostic),
|
||||
Some(("transport-failed", diagnostic)),
|
||||
&history_root(),
|
||||
)
|
||||
.expect("transport failure must produce a failure payload");
|
||||
assert_eq!(failure.kind, DIRECT_TURN_FAILURE_TRANSPORT_KIND);
|
||||
);
|
||||
let failure = terminal.failure.expect("host fact must fail the turn");
|
||||
assert_eq!(terminal.status, "failed");
|
||||
assert_eq!(failure.kind, "transport-failed");
|
||||
assert!(failure.message.contains("SIGKILL"));
|
||||
assert!(!failure.message.contains("正在核对自有子进程"));
|
||||
|
||||
// 即使同时拿到了错误,通道断开仍是本轮的第一事实。
|
||||
// 即使同时拿到了错误,宿主亲眼看到的事实仍然是第一顺位。
|
||||
let error = LlmError::Transport("DirectProject 收尾历史失败".into());
|
||||
let failure = direct_turn_failure(
|
||||
"failed",
|
||||
let terminal = direct_turn_terminal(
|
||||
"interrupted",
|
||||
Err(&error),
|
||||
Some("执行通道已断开,不能自动重放未确认操作:Codex app-server 已退出"),
|
||||
Some(("turn-interrupted", "本轮模型执行被中断")),
|
||||
&history_root(),
|
||||
)
|
||||
.expect("transport failure must produce a failure payload");
|
||||
assert_eq!(failure.kind, DIRECT_TURN_FAILURE_TRANSPORT_KIND);
|
||||
assert!(failure.message.contains("Codex app-server 已退出"));
|
||||
);
|
||||
let failure = terminal.failure.expect("host fact must fail the turn");
|
||||
assert_eq!(failure.kind, "turn-interrupted");
|
||||
assert!(failure.message.contains("本轮模型执行被中断"));
|
||||
}
|
||||
|
||||
/// 终态事件的形状:失败时同一个 `turn.completed` 带载荷,其余只带 `status`。
|
||||
#[test]
|
||||
fn failure_message_is_redacted_and_truncated() {
|
||||
let root = history_root();
|
||||
let with_path = format!("落盘失败:{} 不可写", root.display());
|
||||
let failure = direct_turn_failure("failed", Ok(&with_path), None, &history_root())
|
||||
.expect("failed status must produce a failure payload");
|
||||
assert!(!failure.message.contains("/tmp/direct-turn-failure-test"));
|
||||
assert!(failure.message.contains("$PROJECT_ROOT"));
|
||||
|
||||
let long = "x".repeat(4_000);
|
||||
let failure = direct_turn_failure("failed", Ok(&long), None, &history_root())
|
||||
.expect("failed status must produce a failure payload");
|
||||
// 按字符截断,最多再多一个省略号标记。
|
||||
assert!(failure.message.chars().count() <= DIRECT_TURN_FAILURE_MESSAGE_MAX_CHARS + 1);
|
||||
assert!(failure.message.ends_with('…'));
|
||||
|
||||
// 通道断开的诊断同样要脱敏 + 截断:它比报告长得多,且可能带本机路径。
|
||||
let diagnostic = format!(
|
||||
"执行通道已断开:Codex app-server 已退出;路径 {}",
|
||||
root.display()
|
||||
fn terminal_event_carries_the_payload_and_the_opening_identity() {
|
||||
let error = LlmError::Upstream {
|
||||
status_code: 502,
|
||||
message: "上游 502".into(),
|
||||
};
|
||||
let failing = direct_turn_terminal("interrupted", Err(&error), None, &history_root());
|
||||
let event = failing.event(2_000, Some("direct-codex:turn-1:user"));
|
||||
assert_eq!(
|
||||
event.failure().map(|failure| failure.kind.as_str()),
|
||||
Some("model-failed")
|
||||
);
|
||||
let failure = direct_turn_failure("failed", Ok("报告"), Some(&diagnostic), &history_root())
|
||||
.expect("transport failure must produce a failure payload");
|
||||
assert!(!failure.message.contains("/tmp/direct-turn-failure-test"));
|
||||
assert!(failure.message.contains("$PROJECT_ROOT"));
|
||||
assert_eq!(event.user_item_id(), Some("direct-codex:turn-1:user"));
|
||||
assert_eq!(event.at(), Some(2_000));
|
||||
|
||||
let quiet = direct_turn_terminal("completed", Ok("本轮交付已完成"), None, &history_root());
|
||||
let event = quiet.event(3_000, None);
|
||||
assert!(event.failure().is_none());
|
||||
assert!(matches!(
|
||||
event,
|
||||
DirectThreadEvent::TurnCompleted { ref status, .. } if status == "completed"
|
||||
));
|
||||
}
|
||||
|
||||
/// 兜底:守卫武装后没被解除就 Drop,必须补一条失败终态(panic / future 被丢弃走的就是这条)。
|
||||
|
||||
Reference in New Issue
Block a user