策划审批的两个 GUI 入口等过项目锁瞬时争用,不再把成功的审批画成失败
线上症状:点「批准」后卡里弹出 `项目正在被其他写操作占用`,但 agent.db 显示 `plan.gdd_decided action=approve` 已落盘、approvals/v1.json 已写、index 里 approvedVersion=1、run 随后正常收束,且没有任何 gdd_projection_gap —— 审批本身是成功的,报错来自它之后那一次刷新。 成因是决定放行 run 续跑,而 GUI 在 decide 返回后紧接着 hydrate: `decide → await hydratePlanGddState()` 与 runner 的 `agent.schedule_ready`、 会话写入、planning 投影同时伸手拿 `.agent/project.lock`。hydrate 那一支是 一次性取锁、不重试,撞上就返回 PLAN_STORAGE_IO,前端原样贴进审批卡。 仓库里本来就有这个约定:project_gates 的 `..._with_wait` 只认 `项目正在被其他写操作占用:` 这一个前缀并退避重试,运行时侧 18 个点都在用。 策划的两个 GUI 入口是漏网的。按「丢了这一次要付什么代价」分档接上: - planning.gdd-decision 是一次性用户意图,丢了用户得重新找到卡片 → 等满窗口。 - planning.hydrate 是轮询刷新,下一拍还会来 → 只等 1 秒(新增 short wait 档), 免得为一次刷新把面板卡住整个窗口。 前端再兜一层:hydrate 撞上这条错误时保留上一份状态、不写错误位。后端等过短 窗口仍拿不到,只说明此刻运行时正在写盘;这条 effect 每次监工状态变化都会重跑。 用 includes 而不是 startsWith,因为 hydrate 的错误带 `PLAN_STORAGE_IO: ` 前缀。 两条新测试都做过 A/B(换回一次性取锁即失败): - decision_rides_out_a_briefly_held_project_lock - hydrate_rides_out_a_briefly_held_project_lock 锁由后台线程持有 120ms 再释放;pid 与 createdAt 都写真值,否则会被失效锁回收 顺手删掉、根本占不住。 hydrate 原有的争用用例(考脱敏与错误码形状)保持红,只更新了那条已失效的 「这一支不重试」注释。 做游戏链路未触及:改的两个函数都是策划专属入口,共用的 `acquire_project_write_lock` 语义一行未动。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1779,18 +1779,31 @@ pub(crate) fn project_verification_completion_blocker_at(
|
||||
project_verification_completion_blocker_at_locked(root, agent_id, run_id, observations)
|
||||
}
|
||||
|
||||
pub(crate) fn acquire_game_creator_agent_runtime_project_write_lock_with_wait(
|
||||
const AGENT_RUNTIME_PROJECT_WRITE_LOCK_RETRY_INTERVAL: Duration = Duration::from_millis(5);
|
||||
const AGENT_RUNTIME_PROJECT_WRITE_LOCK_WAIT_ATTEMPTS: usize = 2_000;
|
||||
const AGENT_RUNTIME_PROJECT_WRITE_LOCK_SHORT_WAIT_ATTEMPTS: usize = 200;
|
||||
|
||||
/// Take the project write lock, riding out transient contention for at most
|
||||
/// `max_attempts` polls.
|
||||
///
|
||||
/// `项目正在被其他写操作占用:` is the one lock error that means
|
||||
/// "nothing is broken, the current holder is mid-write" — every other variant
|
||||
/// (a torn lock file, a denied path) is returned immediately. Callers pick the
|
||||
/// budget from what a lost race costs them: a one-shot user intent waits out the
|
||||
/// full window, a poll that will run again shortly waits far less.
|
||||
fn acquire_game_creator_agent_runtime_project_write_lock_within(
|
||||
root: &Path,
|
||||
command_id: &str,
|
||||
max_attempts: usize,
|
||||
) -> Result<ProjectWriteLock, String> {
|
||||
const MAX_ATTEMPTS: usize = 2_000;
|
||||
for attempt in 0..MAX_ATTEMPTS {
|
||||
let max_attempts = max_attempts.max(1);
|
||||
for attempt in 0..max_attempts {
|
||||
match acquire_project_write_lock(root, command_id) {
|
||||
Err(error)
|
||||
if error.starts_with("项目正在被其他写操作占用:")
|
||||
&& attempt + 1 < MAX_ATTEMPTS =>
|
||||
&& attempt + 1 < max_attempts =>
|
||||
{
|
||||
std::thread::sleep(Duration::from_millis(5));
|
||||
std::thread::sleep(AGENT_RUNTIME_PROJECT_WRITE_LOCK_RETRY_INTERVAL);
|
||||
}
|
||||
result => return result,
|
||||
}
|
||||
@@ -1798,6 +1811,32 @@ pub(crate) fn acquire_game_creator_agent_runtime_project_write_lock_with_wait(
|
||||
unreachable!("project write lock retry loop always returns")
|
||||
}
|
||||
|
||||
pub(crate) fn acquire_game_creator_agent_runtime_project_write_lock_with_wait(
|
||||
root: &Path,
|
||||
command_id: &str,
|
||||
) -> Result<ProjectWriteLock, String> {
|
||||
acquire_game_creator_agent_runtime_project_write_lock_within(
|
||||
root,
|
||||
command_id,
|
||||
AGENT_RUNTIME_PROJECT_WRITE_LOCK_WAIT_ATTEMPTS,
|
||||
)
|
||||
}
|
||||
|
||||
/// Same wait, sized for a caller that re-runs on its own — a GUI refresh poll
|
||||
/// rather than a user's one-shot decision. Blocking such a caller for the full
|
||||
/// window would stall the panel it feeds; losing the race only costs it the
|
||||
/// current tick.
|
||||
pub(crate) fn acquire_game_creator_agent_runtime_project_write_lock_with_short_wait(
|
||||
root: &Path,
|
||||
command_id: &str,
|
||||
) -> Result<ProjectWriteLock, String> {
|
||||
acquire_game_creator_agent_runtime_project_write_lock_within(
|
||||
root,
|
||||
command_id,
|
||||
AGENT_RUNTIME_PROJECT_WRITE_LOCK_SHORT_WAIT_ATTEMPTS,
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) fn acquire_game_creator_agent_provider_plan_project_write_lock_with_wait(
|
||||
root: &Path,
|
||||
command_id: &str,
|
||||
|
||||
+9
-1
@@ -1358,7 +1358,15 @@ pub(crate) fn decide_plan_gdd_at(
|
||||
));
|
||||
}
|
||||
validate_decision_transport(input)?;
|
||||
let _lock = acquire_project_write_lock(root, "planning.gdd-decision").map_err(|error| {
|
||||
// A decision is a one-shot user intent: the click either lands or the user
|
||||
// has to find the card again. The runner is writing concurrently for the
|
||||
// whole life of the plan run, so a one-shot acquire hands the button a
|
||||
// failure whenever it happens to land mid-write. Wait the full window.
|
||||
let _lock = acquire_game_creator_agent_runtime_project_write_lock_with_wait(
|
||||
root,
|
||||
"planning.gdd-decision",
|
||||
)
|
||||
.map_err(|error| {
|
||||
approval_error(
|
||||
"PLAN_DURABILITY_FAILED",
|
||||
redact_agent_runtime_project_paths(root, &error, 500),
|
||||
|
||||
+55
-9
@@ -469,13 +469,23 @@ pub(crate) fn hydrate_game_creator_plan_gdd_state_at(
|
||||
// §18.3:先在同一把项目锁内只读校验所有 authority 的 projectId,之后才允许
|
||||
// session/index/pending recovery 写入。这样复制到另一个项目的 sidecar 只能失败关闭,
|
||||
// 不会在发现错绑前改写任何投影。
|
||||
let _lock =
|
||||
crate::project::acquire_project_write_lock(root, "planning.hydrate").map_err(|error| {
|
||||
plan_gdd_state_error(
|
||||
"PLAN_STORAGE_IO",
|
||||
redact_agent_runtime_project_paths(root, &error, 500),
|
||||
)
|
||||
})?;
|
||||
// Ride out transient contention instead of failing the caller. The GUI
|
||||
// re-hydrates right after a decision lands, and the decision is exactly what
|
||||
// releases the run to resume writing, so the refresh and the resumed runner
|
||||
// reach for this lock at the same moment. Losing that race used to paint
|
||||
// `项目正在被其他写操作占用` into the approval card of an approval that had
|
||||
// already committed. The wait is the short one: this call runs again on the
|
||||
// next supervisor poll, so it must never stall the panel for the full window.
|
||||
let _lock = acquire_game_creator_agent_runtime_project_write_lock_with_short_wait(
|
||||
root,
|
||||
"planning.hydrate",
|
||||
)
|
||||
.map_err(|error| {
|
||||
plan_gdd_state_error(
|
||||
"PLAN_STORAGE_IO",
|
||||
redact_agent_runtime_project_paths(root, &error, 500),
|
||||
)
|
||||
})?;
|
||||
let gdds = read_plan_gdd_chain_locked(root)?;
|
||||
let approvals = read_plan_gdd_approvals_locked(root)?;
|
||||
let session_read_only = read_plan_session_read_only_locked(root)?;
|
||||
@@ -675,6 +685,41 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// GUI 在决定落盘后紧接着重灌卡片,而决定本身正是放行 run 继续写盘的那一下——
|
||||
/// 刷新和续跑的 runner 会同时伸手拿同一把项目锁。这条断言的是短窗口内的争用要被
|
||||
/// 等过去:否则一次刚成功的审批会在卡里显示成 `项目正在被其他写操作占用`。
|
||||
#[test]
|
||||
fn hydrate_rides_out_a_briefly_held_project_lock() {
|
||||
let temporary = tempfile::tempdir().expect("create hydrate wait fixture");
|
||||
let root = temporary.path().join("project");
|
||||
crate::project::init_local_game_project_at(&root, "hydrate-wait", "锁等待项目")
|
||||
.expect("initialize hydrate wait fixture");
|
||||
|
||||
// 两个字段都必须写真值,否则会被失效锁回收顺手删掉、锁根本占不住。
|
||||
let lock_path = root.join(".agent/project.lock");
|
||||
let held = serde_json::json!({
|
||||
"commandId": "test.hold",
|
||||
"pid": std::process::id(),
|
||||
"createdAt": unix_timestamp(),
|
||||
"nonce": 0,
|
||||
});
|
||||
std::fs::write(
|
||||
&lock_path,
|
||||
serde_json::to_vec(&held).expect("serialize held lock"),
|
||||
)
|
||||
.expect("hold project lock");
|
||||
let holder = std::thread::spawn(move || {
|
||||
std::thread::sleep(std::time::Duration::from_millis(120));
|
||||
std::fs::remove_file(&lock_path).expect("release project lock");
|
||||
});
|
||||
|
||||
let view = hydrate_game_creator_plan_gdd_state_at(&root)
|
||||
.expect("hydrate 必须等过瞬时锁争用,而不是把失败画进审批卡");
|
||||
holder.join().expect("lock holder thread");
|
||||
|
||||
assert_eq!(view.state, "not_started");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hydrate_redacts_project_paths_from_contended_project_lock_errors() {
|
||||
let temporary = tempfile::tempdir().expect("create hydrate lock fixture");
|
||||
@@ -682,8 +727,9 @@ mod tests {
|
||||
crate::project::init_local_game_project_at(&root, "hydrate-lock", "锁竞争项目")
|
||||
.expect("initialize hydrate lock fixture");
|
||||
|
||||
// `.agent/project.lock` 是 `create_new(true)` 的文件锁且这一支不重试,预先占住它,
|
||||
// hydrate 的第一次取锁(`reconcile` 内那次)就确定性失败。
|
||||
// `.agent/project.lock` 是 `create_new(true)` 的文件锁。hydrate 现在会等一个短
|
||||
// 窗口再放弃(见 `..._with_short_wait`),所以这里占住不放:等窗口耗尽,hydrate
|
||||
// 的取锁确定性失败,这条用例考的是那条失败路径的脱敏与错误码形状。
|
||||
//
|
||||
// 两个字段都必须写真值,否则会被失效锁回收顺手删掉、锁根本占不住:
|
||||
// `pid` 供 unix 侧判 owner 是否存活;`createdAt` 供年龄判定——
|
||||
|
||||
@@ -3794,6 +3794,60 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Hold `.agent/project.lock` for `hold_millis`, then release it from a
|
||||
/// background thread. Both fields must carry real values or the stale-lock
|
||||
/// reclaim in `acquire_project_write_lock` deletes the file on the first
|
||||
/// contended attempt and nothing is actually held: `pid` is what the unix
|
||||
/// liveness check reads, `createdAt` is what the age check reads (it is
|
||||
/// preferred over the file mtime).
|
||||
fn hold_project_lock_briefly(root: &Path, hold_millis: u64) -> std::thread::JoinHandle<()> {
|
||||
let lock_path = root.join(".agent/project.lock");
|
||||
let held = serde_json::json!({
|
||||
"commandId": "test.hold",
|
||||
"pid": std::process::id(),
|
||||
"createdAt": unix_timestamp(),
|
||||
"nonce": 0,
|
||||
});
|
||||
fs::write(
|
||||
&lock_path,
|
||||
serde_json::to_vec(&held).expect("serialize held lock"),
|
||||
)
|
||||
.expect("hold project lock");
|
||||
std::thread::spawn(move || {
|
||||
std::thread::sleep(std::time::Duration::from_millis(hold_millis));
|
||||
fs::remove_file(&lock_path).expect("release project lock");
|
||||
})
|
||||
}
|
||||
|
||||
/// 审批按钮是一次性意图:点下去要么落盘,要么用户得重新找到卡片。批准放行后
|
||||
/// runner 立刻续跑并开始写盘,所以决定命令和运行时会同时伸手去拿同一把项目锁。
|
||||
/// 这里断言的是决定不能因为撞上这种瞬时争用而失败——线上症状是点了批准、审批
|
||||
/// 其实成功了,卡里却弹出 `项目正在被其他写操作占用`。
|
||||
#[test]
|
||||
fn decision_rides_out_a_briefly_held_project_lock() {
|
||||
let (root, context, input) = submit_fixture();
|
||||
execute_plan_submit_gdd(&root, &context, &input).expect("submit v1");
|
||||
let gdd = read_plan_gdd_chain(&root)
|
||||
.expect("read submitted GDD")
|
||||
.pop()
|
||||
.expect("GDD exists");
|
||||
create_plan_gdd_approval_pending_at(&root, &gdd).expect("create approval pending");
|
||||
let decision_input = approval_input(
|
||||
&gdd,
|
||||
"approve",
|
||||
"gdd-response-00000000-0000-4000-8000-000000000041",
|
||||
None,
|
||||
);
|
||||
|
||||
let holder = hold_project_lock_briefly(&root, 120);
|
||||
let decision = decide_plan_gdd_at(&root, &decision_input)
|
||||
.expect("决定必须等过瞬时锁争用,而不是把失败甩回按钮");
|
||||
holder.join().expect("lock holder thread");
|
||||
|
||||
assert_eq!(decision.outcome, "committed");
|
||||
cleanup_fixture(root);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn approval_pending_is_single_latest_unreceipted_projection() {
|
||||
let (root, context, input) = submit_fixture();
|
||||
|
||||
@@ -746,7 +746,14 @@ export function App({
|
||||
setPlanGddState(nextState);
|
||||
}
|
||||
} catch (error) {
|
||||
// 项目写锁争用是瞬时的:后端已经等过一个短窗口,仍然没抢到只说明此刻
|
||||
// 运行时正在写盘。这条 effect 每次监工状态变化都会再跑一次,下一拍就能
|
||||
// 拿到,所以保留上一份状态、不动错误位——否则一次刚落盘成功的审批会在
|
||||
// 卡里显示成失败(decide 成功后紧跟的这次 hydrate 正是最容易撞上的时刻)。
|
||||
const transientContention =
|
||||
String(error).includes('项目正在被其他写操作占用:');
|
||||
if (
|
||||
!transientContention &&
|
||||
requestSequence === planGddHydrateSequenceRef.current &&
|
||||
localProjectPathRef.current === targetProjectPath
|
||||
) {
|
||||
|
||||
Reference in New Issue
Block a user