为 sandbox 启动失败补保留源路径的宿主诊断日志

- `command.exec` 在 sandbox-ready 失败时先取走子进程 stdout/stderr,再终止受控进程组,避免 `Child` 回收时丢掉 bwrap/trampoline 的真实报错;
- 采集结果在有界时间(2s)和字节(8 KiB/流)内读空;新增 `sanitize_diagnostic_message_preserving_paths`、`append_application_log_line_preserving_paths` 与 `app_log_with_paths!`,只做凭据脱敏、保留绝对路径(bwrap `Can't find source path` 的路径就是唯一证据),写入宿主 `application.log`;
- 默认 `sanitize_diagnostic_message` 仍整行脱敏绝对路径,新增项按 `target_os = "linux"` 门控;诊断只进宿主诊断包,不进入模型可见错误、command log、receipt 或 Agent DB;
- 新增 `sandbox_launch_diagnostics_keep_source_paths_and_redact_secrets`,覆盖源路径保留、凭据脱敏与空白流不产出行;
- `pitfalls.md` 记录 sandbox 启动失败丢弃子进程 stderr 的排障经验。
This commit is contained in:
2026-10-06 15:30:36 +08:00
parent 2cfc0d5bb4
commit 6311bb5837
3 changed files with 176 additions and 7 deletions
@@ -1853,18 +1853,18 @@ where
let (mut gate, ready) = match ready_task {
Ok(result) => result,
Err(error) => {
let termination = terminate_project_command_process_group(&mut child).await;
let cause = format!("等待 sandbox-ready 任务失败:{error}");
let termination =
terminate_project_command_launch_and_log_diagnostics(&mut child, &cause).await;
return Err(ProjectCommandError::new(
ProjectCommandErrorStage::Preflight,
project_command_launch_error_with_termination(
format!("等待 sandbox-ready 任务失败:{error}"),
termination,
),
project_command_launch_error_with_termination(cause, termination),
));
}
};
if let Err(error) = ready {
let termination = terminate_project_command_process_group(&mut child).await;
let termination =
terminate_project_command_launch_and_log_diagnostics(&mut child, &error).await;
return Err(ProjectCommandError::new(
ProjectCommandErrorStage::Preflight,
project_command_launch_error_with_termination(error, termination),
@@ -2085,6 +2085,103 @@ fn terminate_project_command_process_group_after_commit(
))
}
/// sandbox 启动失败时,bwrap / trampoline 子进程的 stderr / stdout 是唯一能说明 setup
/// 失败原因的证据。启动失败路径原先直接 kill + wait,管道里的诊断随 `Child` 一起被丢掉,
/// 只剩 `Connection reset by peer` 这类不透明的 OS 错误。这里只做有界采集,凭据脱敏但保留
/// 绝对路径(bwrap 的 `Can't find source path` 里路径就是唯一证据),写入宿主
/// application.log 供诊断包使用;不进入模型可见错误、command log、receipt 或 Agent DB。
#[cfg(target_os = "linux")]
const SANDBOX_LAUNCH_DIAGNOSTIC_MAX_BYTES: usize = 8 * 1024;
#[cfg(target_os = "linux")]
const SANDBOX_LAUNCH_DIAGNOSTIC_MAX_WAIT: Duration = Duration::from_secs(2);
#[cfg(target_os = "linux")]
async fn read_bounded_sandbox_launch_stream<R>(mut reader: R, budget: usize) -> Vec<u8>
where
R: tokio::io::AsyncRead + Unpin,
{
let mut collected = Vec::new();
let mut buffer = [0_u8; 1024];
while collected.len() < budget {
match reader.read(&mut buffer).await {
Ok(0) | Err(_) => break,
Ok(count) => {
let remaining = budget - collected.len();
collected.extend_from_slice(&buffer[..count.min(remaining)]);
}
}
}
collected
}
#[cfg(target_os = "linux")]
async fn collect_sandbox_launch_streams(
stdout: Option<tokio::process::ChildStdout>,
stderr: Option<tokio::process::ChildStderr>,
) -> (Vec<u8>, Vec<u8>) {
let read_stdout = async {
match stdout {
Some(reader) => {
read_bounded_sandbox_launch_stream(reader, SANDBOX_LAUNCH_DIAGNOSTIC_MAX_BYTES)
.await
}
None => Vec::new(),
}
};
let read_stderr = async {
match stderr {
Some(reader) => {
read_bounded_sandbox_launch_stream(reader, SANDBOX_LAUNCH_DIAGNOSTIC_MAX_BYTES)
.await
}
None => Vec::new(),
}
};
tokio::join!(read_stdout, read_stderr)
}
/// 把 sandbox 子进程的输出整理成一条宿主日志:非空、单行、凭据已脱敏。
/// 绝对路径必须保留——bwrap `Can't find source path` 的路径就是唯一证据。
#[cfg(target_os = "linux")]
fn format_sandbox_launch_diagnostics(stdout: &[u8], stderr: &[u8]) -> Option<String> {
let mut sections = Vec::new();
for (stream_name, bytes) in [("stdout", stdout), ("stderr", stderr)] {
let text = String::from_utf8_lossy(bytes);
let text = text.trim();
if text.is_empty() {
continue;
}
let sanitized = crate::sanitize_diagnostic_message_preserving_paths(text, None);
let sanitized = sanitized.trim();
if !sanitized.is_empty() {
sections.push(format!("{stream_name}={sanitized}"));
}
}
(!sections.is_empty()).then(|| sections.join(";"))
}
/// 启动失败路径统一入口:先取走管道(否则 `wait()` 回收后诊断随 `Child` 一起被丢掉),
/// 再终止受控进程组,最后在有界时间内读空管道并写入宿主日志。
#[cfg(target_os = "linux")]
async fn terminate_project_command_launch_and_log_diagnostics(
child: &mut tokio::process::Child,
cause: &str,
) -> Result<String, String> {
let stdout = child.stdout.take();
let stderr = child.stderr.take();
let termination = terminate_project_command_process_group(child).await;
let (stdout_bytes, stderr_bytes) = tokio::time::timeout(
SANDBOX_LAUNCH_DIAGNOSTIC_MAX_WAIT,
collect_sandbox_launch_streams(stdout, stderr),
)
.await
.unwrap_or_default();
let diagnostics = format_sandbox_launch_diagnostics(&stdout_bytes, &stderr_bytes)
.unwrap_or_else(|| "stdout/stderr 无输出".to_string());
app_log_with_paths!("command.exec sandbox 启动失败诊断:cause={cause};{diagnostics}");
termination
}
async fn read_bounded_project_command_output<R>(
mut reader: R,
) -> Result<BoundedCommandOutput, String>
@@ -3567,4 +3664,26 @@ raise SystemExit(code)'
assert!(output.text.contains("omitted"));
assert!(output.truncated);
}
#[cfg(target_os = "linux")]
#[test]
fn sandbox_launch_diagnostics_keep_source_paths_and_redact_secrets() {
let diagnostics = format_sandbox_launch_diagnostics(
b"",
b"bwrap: Can't find source path /home/alice/project/game No such file or directory\n",
)
.expect("non-empty stderr yields diagnostics");
assert!(diagnostics.contains("bwrap"), "{diagnostics}");
assert!(
diagnostics.contains("/home/alice/project/game"),
"source path is the diagnostic evidence and must survive: {diagnostics}"
);
assert!(
format_sandbox_launch_diagnostics(b"", b" \n \t").is_none(),
"whitespace-only streams must not produce a diagnostic line"
);
let secret = format_sandbox_launch_diagnostics(b"", b"authorization: Bearer abc123\n")
.expect("secret-bearing stderr still yields a redacted diagnostic");
assert!(!secret.to_ascii_lowercase().contains("bearer"), "{secret}");
}
}
@@ -68,6 +68,17 @@ macro_rules! app_log {
}};
}
/// 与 `app_log!` 相同,但保留绝对路径。只给「路径本身就是诊断证据」的宿主失败用
/// (例如 bwrap `Can't find source path`);凭据标记仍整行脱敏。
#[cfg(target_os = "linux")]
macro_rules! app_log_with_paths {
($($arg:tt)*) => {{
let message = format!($($arg)*);
let _ = $crate::append_application_log_line_preserving_paths(&format!("RUST {}: {}", module_path!(), message));
std::eprintln!("{}", message);
}};
}
// 调试落盘模块(保存 LLM 原始输出 / 失败输入,排查截断、空返回等)放在 debug_drafts.rs。
// 用 #[cfg] 编译期门控:仅开发(debug)且非测试构建编入;生产 release 与 cargo test 下整体剔除。
include!(concat!(env!("OUT_DIR"), "/agent_runtime_prompt_bundle.rs"));
@@ -1645,6 +1656,17 @@ pub(crate) fn append_application_log_line(line: &str) -> std::io::Result<()> {
append_bounded_diagnostic_line(&config_dir.join("diagnostics/application.log"), &sanitized)
}
/// 与 `append_application_log_line` 相同,但保留绝对路径。只给「路径本身就是诊断证据」的
/// 宿主失败用;凭据标记仍由 `sanitize_diagnostic_message_preserving_paths` 整行脱敏。
#[cfg(target_os = "linux")]
pub(crate) fn append_application_log_line_preserving_paths(line: &str) -> std::io::Result<()> {
let Some(config_dir) = game_creator_runtime_config_dir() else {
return Ok(());
};
let sanitized = sanitize_diagnostic_message_preserving_paths(line, Some(&config_dir));
append_bounded_diagnostic_line(&config_dir.join("diagnostics/application.log"), &sanitized)
}
fn redact_windows_absolute_paths(value: &str) -> String {
let bytes = value.as_bytes();
let mut output = String::with_capacity(value.len());
@@ -1704,6 +1726,24 @@ fn redact_unix_absolute_paths(value: &str) -> String {
}
pub(crate) fn sanitize_diagnostic_message(value: &str, private_root: Option<&Path>) -> String {
sanitize_diagnostic_message_inner(value, private_root, true)
}
/// 与 `sanitize_diagnostic_message` 相同,但保留绝对路径。路径本身就是诊断证据的宿主失败
/// (例如 bwrap `Can't find source path`)必须走这一条;凭据标记仍然整行脱敏。
#[cfg(target_os = "linux")]
pub(crate) fn sanitize_diagnostic_message_preserving_paths(
value: &str,
private_root: Option<&Path>,
) -> String {
sanitize_diagnostic_message_inner(value, private_root, false)
}
fn sanitize_diagnostic_message_inner(
value: &str,
private_root: Option<&Path>,
redact_paths: bool,
) -> String {
let mut sanitized = value.replace(['\r', '\n'], " ");
if let Some(root) = private_root {
let root = root.to_string_lossy();
@@ -1728,7 +1768,9 @@ pub(crate) fn sanitize_diagnostic_message(value: &str, private_root: Option<&Pat
{
return "<sensitive diagnostic details redacted>".to_string();
}
sanitized = redact_unix_absolute_paths(&redact_windows_absolute_paths(&sanitized));
if redact_paths {
sanitized = redact_unix_absolute_paths(&redact_windows_absolute_paths(&sanitized));
}
sanitized.chars().take(2_048).collect()
}
@@ -4149,6 +4149,14 @@ Cocos Creator 根目录由 `package.json.creator.version` 与普通 `assets/`
- 验证:并发执行全部 project verification 测试;同时真实运行 bwrap staged marker 用例,确认 child-created、block、ready、commit、exec 和目标退出链均稳定,目标 argv/env/FD 不含 nonce 或控制 socket。
- 关联:`apps/ai-game-creator-shell/src-tauri/src/command_sandbox_trampoline.rs`、`command_sandbox.rs`、`command_exec.rs`、`project.rs`。
## sandbox 启动失败不能把子进程 stderr 一起丢掉
- 现象:`command.exec` 在 `wait_sandbox_ready` 阶段失败时,模型和日志只看到 `等待 command sandbox ready 失败:Connection reset by peer (os error 104)`;无法判断是 userns、AppArmor、mount 还是 trampoline 自身失败,真实 bwrap 报错完全不可见。
- 原因:bwrap/trampoline 以 `Stdio::piped()` 启动,但启动失败路径只调用 `terminate_project_command_process_group`(`kill` + `wait`)后即返回;`Child` 被回收时管道里的 stderr/stdout 一并丢失。项目自己的真实 launcher 测试(`command_sandbox.rs` 的 opt-in staged gate 用例)反而会 `wait_with_output()` 并把 stderr 打进 panic。
- 处理:启动失败统一走 `terminate_project_command_launch_and_log_diagnostics`:先 `take()` stdout/stderr,再 kill + wait,然后在有界时间(2s)和字节(8 KiB/流)内读空,经 `sanitize_diagnostic_message_preserving_paths` 只做凭据脱敏、保留绝对路径(bwrap 的 `Can't find source path` 里路径就是唯一证据),写入宿主 `application.log`;诊断只进宿主诊断包,不进入模型可见错误、command log、receipt 或 Agent DB。默认的 `sanitize_diagnostic_message` 仍然整行脱敏绝对路径,只有这条宿主失败日志走保留路径的变体。
- 验证:`sandbox_launch_diagnostics_keep_source_paths_and_redact_secrets` 断言源路径保留、凭据整体脱敏、纯空白流不产出行;`GENARRATIVE_COMMAND_SANDBOX_REAL_TEST=1 cargo test ... real_linux_opt_in` 的真实 bwrap 握手用例保持通过。
- 关联:`apps/ai-game-creator-shell/src-tauri/src/command_exec.rs`、`command_sandbox.rs`、`command_sandbox_trampoline.rs`。
## bwrap 的命令分隔符不能从目标 argv 末尾反查
- 现象:普通命令握手正常,但 `cargo test -- --nocapture`、npm forwarded args 等包含独立 `--` 的合法目标参数可能在 sandbox preflight 或 staged launch 期间提前执行原目标。