修复试玩回包遗漏双端玩法失败证据(#548) (#598)
Project CI / AI game creator shell Rust crates (push) Successful in 2m58s
Project CI / Backend tests (push) Failing after 4m25s
Project CI / AI game creator shell Rust lane 2/2 (push) Successful in 5m34s
Project CI / AI game creator shell Rust lane 1/2 (push) Successful in 5m44s
Project CI / Frontend tests (push) Successful in 2m50s
Project CI / AI game creator shell web tests (push) Successful in 2m45s
Project CI / Repository checks (push) Failing after 4m7s
Project CI / Native shell tests (push) Successful in 6m49s
Project CI / AI game creator shell Rust crates (push) Successful in 2m58s
Project CI / Backend tests (push) Failing after 4m25s
Project CI / AI game creator shell Rust lane 2/2 (push) Successful in 5m34s
Project CI / AI game creator shell Rust lane 1/2 (push) Successful in 5m44s
Project CI / Frontend tests (push) Successful in 2m50s
Project CI / AI game creator shell web tests (push) Successful in 2m45s
Project CI / Repository checks (push) Failing after 4m7s
Project CI / Native shell tests (push) Successful in 6m49s
试玩报告已记录玩法阻断,但工具回包只展示视觉结果,Agent 无法从“画面正常、整体失败”定位原因;返回的私有报告路径也不能通过通用文件工具读取。 本次增加逐视口 gameplayResults,返回有界诊断、断言和初末状态,并从同一份投影生成摘要。明确区分视觉检查与玩法检查、未通过与未执行;保留现有通过判定、验证预算和私有目录保护。同步更新证据合同、提示词与 skill pack。 验证: - Rust 定向测试、提示词构建与源码边界检查通过。 - 最终 MCP 成功/失败回包均检查逐视口状态、原因、断言及截图。 - 显式运行 4 个真实浏览器用例:双端成功、禁用控件、真实点击后 phase 未进入 playing、仅移动端失败;对照持久报告核对回包与摘要。 - rustfmt、编码、文档索引、skill pack、git diff --check 通过。 - 未运行全量测试或真实模型端到端验证。 Closes #548 Reviewed-on: #598
This commit was merged in pull request #598.
This commit is contained in:
@@ -17,8 +17,14 @@
|
||||
"browser.status": "整体 passed={}",
|
||||
"browser.notStarted": "本次未启动 Chromium",
|
||||
"browser.evidence": "[AGC 浏览器事实证据]\nattempt={attempt}; completionError={completion_status}; browser={browser_status}; codeFingerprintChanged={output_changed}\n{}\n诊断与平台素材运行时观察:{}\n交互探针:{}\n客户端已保存结构化证据;它是事实输入,不代表 Codex 已阅读截图或已经完成修复。当前 cwd 是项目根目录;按需读取实际文件,并自行决定是否修改、再次试玩或直接回复。AGC 只负责启动浏览器、采集证据和执行项目边界,Codex 负责解释结果。",
|
||||
"browser.noFailureDetails": "无额外硬失败详情",
|
||||
"browser.noVisibleControls": "未找到可执行的可见控件",
|
||||
"browser.noFailureDetails": "无额外视觉检查失败详情",
|
||||
"browser.noVisibleControls": "无建议性交互探针记录",
|
||||
"browser.gameplayNotRun": "玩法检查:未执行(visual 仅提供视觉证据)。",
|
||||
"browser.gameplayMissing": "固定场景玩法结果缺失,不能确认通过。",
|
||||
"browser.gameplayNoDiagnostic": "无诊断记录",
|
||||
"browser.gameplayNoUnmetAssertion": "无",
|
||||
"browser.gameplayViewport": "{viewport}: gameplayPassed={passed};首条玩法诊断:{diagnostic};首个未通过断言:{assertion}",
|
||||
"browser.gameplayEvidence": "[固定场景玩法证据]\n{}\n断言 passed=false 表示未通过,也可能因前序阻断而未执行;修复以具体诊断为准,不把所有未通过项视为独立故障。固定场景通过不代表完整关卡已验证;reportPath 仅为宿主证据引用,无需读取私有报告。",
|
||||
"system.role": "你是陶泥儿,是 Genarrative 面向用户的游戏创作助手,负责当前任务的执行。先理解用户意图:普通对话直接回答,项目请求按需要检查、修改、运行和验证,并用简洁中文报告真实结果。",
|
||||
"system.workspaceBoundary": "工作区边界:只在当前项目目录内工作;不要读取或输出凭据、Token、Cookie、auth.json、.env 或宿主私密路径。遇到阻断必须说明具体原因、文件和下一步,不要声称未验证的成功。",
|
||||
"system.toolAuthorization": "AGC 工具授权:agc_tools 使用当前登录会话。工具返回 401/403 时,报告登录或权限状态异常并停止,交由用户处理登录和权限。",
|
||||
|
||||
+2
@@ -10,6 +10,8 @@ Canvas/WebGL probes can show whether registered image sources entered a core ren
|
||||
|
||||
`mode=visual` captures desktop/mobile screenshots, loading, layout and diagnostics without clicking gameplay controls. It does not prove that the game loop works. `mode=gameplay` uses the fixed approved scenario (`generic-v1` by default, or the existing `tetris-v1` / `lane-defense-v1` contract), not an agent-written autopilot. It exercises the scenario independently in desktop and mobile viewports and requires both results. The receipt declares `gameplayViewports` separately from `visualViewports`. Older reports with only a desktop `playtest` result do not prove mobile gameplay. These bounded scenarios do not establish completion of an arbitrary long level.
|
||||
|
||||
Use each viewport's `gameplayResults` to diagnose failures: start with its first `diagnostics` entry, then inspect `assertions` and the recorded initial/final phase, sequence and level. An assertion with `passed=false` may not have run because an earlier step blocked; do not treat every unmet assertion as an independent defect. Missing viewport evidence is not a pass. In visual mode, `gameplayResults` is empty because gameplay was not run. `reportPath` is a private host evidence reference; use the returned diagnostics instead of trying to read that report.
|
||||
|
||||
For `generic-v1`, expose exactly one `script#playable-web-game-state[type="application/json"]` whose JSON is a projection of actual game state:
|
||||
|
||||
```json
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schemaVersion": "agc-skill-pack.v1",
|
||||
"version": "2026-08-26.35",
|
||||
"version": "2026-08-26.37",
|
||||
"skills": [
|
||||
{
|
||||
"name": "agc-unity-editor",
|
||||
@@ -140,7 +140,7 @@
|
||||
"references/browser-evidence-contract.md",
|
||||
"references/runner-physics.mjs"
|
||||
],
|
||||
"sha256": "e0e7bdca529e068ce2974a5e312a267108ab5acb20e8f9d7c0c22d049d90b796"
|
||||
"sha256": "6800059c4e7bae70b0b42ec47175fc85e38d789e1759dca5df2067f715dc0657"
|
||||
},
|
||||
{
|
||||
"name": "agc-client-projection",
|
||||
|
||||
@@ -30,6 +30,8 @@ mod direct_runtime;
|
||||
mod direct_tool_bridge;
|
||||
mod direct_tools_mcp;
|
||||
mod direct_validation;
|
||||
#[cfg(test)]
|
||||
pub(crate) use direct_validation::browser_tool_result;
|
||||
mod generation;
|
||||
pub(crate) mod json_sidecar;
|
||||
pub(crate) mod redaction;
|
||||
|
||||
@@ -4421,6 +4421,103 @@ pub(super) fn drain_analytics_test_writer(
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn browser_gameplay_results_reach_model_with_matching_viewports_and_screenshots() {
|
||||
for passed in [false, true] {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let evidence = if passed {
|
||||
super::super::direct_validation::tests::browser_evidence_fixture(temp.path())
|
||||
} else {
|
||||
super::super::direct_validation::tests::failed_browser_evidence_fixture(temp.path())
|
||||
};
|
||||
for viewport in &evidence.viewport_results {
|
||||
image::RgbaImage::new(2, 2)
|
||||
.save(&viewport.screenshot_path)
|
||||
.unwrap();
|
||||
}
|
||||
let receipt = super::super::direct_validation::browser_tool_result(
|
||||
temp.path(),
|
||||
&evidence,
|
||||
Some(crate::browser::BrowserPlaytestScenario::GenericV1),
|
||||
1,
|
||||
);
|
||||
assert!(evidence.viewport_results.iter().all(|result| result.passed));
|
||||
assert_eq!(receipt["passed"], passed);
|
||||
let outcome = bridge_validation_result(
|
||||
temp.path(),
|
||||
Ok(receipt.clone()),
|
||||
|report, screenshots| BrowserPlaytestError::PlaytestNotPassed {
|
||||
report,
|
||||
screenshots,
|
||||
},
|
||||
)
|
||||
.map_err(|failure| {
|
||||
let error = ToolCallError::from(&failure);
|
||||
assert_eq!(error.error["PlaytestNotPassed"]["report"], receipt);
|
||||
assert!(error.error["PlaytestNotPassed"]
|
||||
.get("screenshots")
|
||||
.is_none());
|
||||
error
|
||||
});
|
||||
let state = direct_tool_bridge_state(temp.path().to_path_buf());
|
||||
let response =
|
||||
compose_direct_tool_outcome(&state, "agc_browser_playtest", "{}", false, outcome);
|
||||
assert_eq!(response["isError"], !passed);
|
||||
let text = response["content"][0]["text"].as_str().unwrap();
|
||||
let model_receipt: Value =
|
||||
serde_json::from_str(&text[text.find('{').unwrap()..]).unwrap();
|
||||
assert_eq!(model_receipt, receipt);
|
||||
assert_eq!(
|
||||
response["content"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|block| block["type"] == "image")
|
||||
.count(),
|
||||
2
|
||||
);
|
||||
let summary = model_receipt["summary"].as_str().unwrap();
|
||||
for (index, entry) in evidence.viewport_playtests.iter().enumerate() {
|
||||
let report = &model_receipt["gameplayResults"][index];
|
||||
assert_eq!(
|
||||
report["viewport"],
|
||||
serde_json::to_value(entry.viewport).unwrap()
|
||||
);
|
||||
assert_eq!(report["passed"], entry.result.passed);
|
||||
assert_eq!(
|
||||
report["assertions"],
|
||||
serde_json::to_value(&entry.result.assertions).unwrap()
|
||||
);
|
||||
assert_eq!(report["finalPhase"], "ready");
|
||||
assert_eq!(
|
||||
report["finalSequence"],
|
||||
entry.result.final_sequence.unwrap()
|
||||
);
|
||||
let viewport = report["viewport"].as_str().unwrap();
|
||||
let line = summary
|
||||
.lines()
|
||||
.find(|line| line.starts_with(&format!("{viewport}: gameplayPassed=")))
|
||||
.unwrap();
|
||||
assert!(line.starts_with(&format!(
|
||||
"{viewport}: gameplayPassed={}",
|
||||
entry.result.passed
|
||||
)));
|
||||
if let Some(reason) = entry.result.diagnostics.first() {
|
||||
assert_eq!(report["diagnostics"][0], reason.as_str());
|
||||
assert!(line.contains(reason));
|
||||
}
|
||||
if let Some(assertion) = entry
|
||||
.result
|
||||
.assertions
|
||||
.iter()
|
||||
.find(|assertion| !assertion.passed)
|
||||
{
|
||||
assert!(line.contains(&assertion.name));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn host_cancellation_releases_queued_image_without_a_new_post() {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
@@ -395,6 +395,137 @@ async fn finish_async(
|
||||
.map_err(|_| "验证回执任务退出".to_string())?
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct GameplayViewportEvidence {
|
||||
viewport: crate::browser::BrowserValidationViewport,
|
||||
passed: bool,
|
||||
diagnostics: Vec<String>,
|
||||
assertions: Vec<crate::browser::BrowserPlaytestAssertion>,
|
||||
initial_phase: Option<crate::browser::BrowserPlaytestPhase>,
|
||||
initial_sequence: Option<u64>,
|
||||
initial_level: Option<u64>,
|
||||
final_phase: Option<crate::browser::BrowserPlaytestPhase>,
|
||||
final_sequence: Option<u64>,
|
||||
final_level: Option<u64>,
|
||||
}
|
||||
|
||||
/// 只投影模型修复所需的玩法事实;完整报告继续留在宿主私有目录。
|
||||
pub(crate) fn browser_tool_result(
|
||||
root: &Path,
|
||||
evidence: &crate::browser::BrowserValidationResult,
|
||||
scenario: Option<crate::browser::BrowserPlaytestScenario>,
|
||||
sequence: usize,
|
||||
) -> Value {
|
||||
use crate::browser::BrowserValidationViewport::{Desktop, Mobile};
|
||||
|
||||
let gameplay_results = if scenario.is_some() {
|
||||
[Desktop, Mobile]
|
||||
.into_iter()
|
||||
.map(|viewport| {
|
||||
let result = evidence
|
||||
.viewport_playtests
|
||||
.iter()
|
||||
.find(|entry| entry.viewport == viewport)
|
||||
.map(|entry| &entry.result);
|
||||
GameplayViewportEvidence {
|
||||
viewport,
|
||||
passed: result.is_some_and(|result| result.passed),
|
||||
diagnostics: result
|
||||
.map(|result| {
|
||||
result
|
||||
.diagnostics
|
||||
.iter()
|
||||
.take(4)
|
||||
.map(|message| truncate_agent_runtime_text(message, 511))
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_else(|| {
|
||||
vec![prompt_text!("direct.browser.gameplayMissing").into()]
|
||||
}),
|
||||
assertions: result
|
||||
.map(|result| result.assertions.clone())
|
||||
.unwrap_or_default(),
|
||||
initial_phase: result.and_then(|result| result.initial_phase),
|
||||
initial_sequence: result.and_then(|result| result.initial_sequence),
|
||||
initial_level: result.and_then(|result| result.initial_level),
|
||||
final_phase: result.and_then(|result| result.final_phase),
|
||||
final_sequence: result.and_then(|result| result.final_sequence),
|
||||
final_level: result.and_then(|result| result.final_level),
|
||||
}
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
let gameplay_summary = if scenario.is_none() {
|
||||
prompt_text!("direct.browser.gameplayNotRun").to_string()
|
||||
} else {
|
||||
let lines = gameplay_results
|
||||
.iter()
|
||||
.map(|result| {
|
||||
format!(
|
||||
prompt_text!("direct.browser.gameplayViewport"),
|
||||
viewport = match result.viewport {
|
||||
Desktop => "desktop",
|
||||
Mobile => "mobile",
|
||||
},
|
||||
passed = result.passed,
|
||||
diagnostic = result
|
||||
.diagnostics
|
||||
.first()
|
||||
.map(String::as_str)
|
||||
.unwrap_or(prompt_text!("direct.browser.gameplayNoDiagnostic")),
|
||||
assertion = result
|
||||
.assertions
|
||||
.iter()
|
||||
.find(|assertion| !assertion.passed)
|
||||
.map(|assertion| assertion.name.as_str())
|
||||
.unwrap_or(prompt_text!("direct.browser.gameplayNoUnmetAssertion")),
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
format!(
|
||||
prompt_text!("direct.browser.gameplayEvidence"),
|
||||
lines.join("\n")
|
||||
)
|
||||
};
|
||||
let summary = format!(
|
||||
"{}\n{}",
|
||||
render_direct_browser_tool_evidence(root, evidence, sequence),
|
||||
gameplay_summary
|
||||
);
|
||||
let screenshots: Vec<_> = evidence
|
||||
.viewport_results
|
||||
.iter()
|
||||
.filter_map(|viewport| {
|
||||
viewport
|
||||
.screenshot_path
|
||||
.strip_prefix(root)
|
||||
.ok()
|
||||
.map(|path| path.to_string_lossy().replace('\\', "/"))
|
||||
})
|
||||
.collect();
|
||||
let passed = evidence.passed
|
||||
&& scenario.is_none_or(|scenario| {
|
||||
crate::browser::required_viewport_playtests_passed(
|
||||
scenario,
|
||||
&evidence.viewport_playtests,
|
||||
)
|
||||
});
|
||||
json!({
|
||||
"kind":"browser", "mode":if scenario.is_some() {"gameplay"} else {"visual"},
|
||||
"scenario":scenario, "passed":passed,
|
||||
"scenarioFingerprint":scenario.map(crate::browser::browser_playtest_scenario_fingerprint),
|
||||
"summary":summary, "screenshots":screenshots,
|
||||
"reportPath":evidence.evidence.report_path.strip_prefix(root).ok().map(|path|path.to_string_lossy().replace('\\', "/")),
|
||||
"visualViewports":["desktop","mobile"],
|
||||
"gameplayViewports":evidence.viewport_playtests.iter().map(|entry|entry.viewport).collect::<Vec<_>>(),
|
||||
"gameplayResults":gameplay_results,
|
||||
"coverage":if scenario.is_some() {"dual-viewport-fixed-scenario"} else {"visual-only"}
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) async fn run_browser(root: &Path, arguments: &Value) -> Result<Value, String> {
|
||||
run_browser_with_budget(root, arguments, true).await
|
||||
}
|
||||
@@ -466,33 +597,9 @@ async fn run_browser_with_budget(
|
||||
.await;
|
||||
let (result, passed) = match evidence {
|
||||
Ok(evidence) => {
|
||||
let screenshots: Vec<_> = evidence
|
||||
.viewport_results
|
||||
.iter()
|
||||
.filter_map(|v| {
|
||||
v.screenshot_path
|
||||
.strip_prefix(root)
|
||||
.ok()
|
||||
.map(|p| p.to_string_lossy().replace('\\', "/"))
|
||||
})
|
||||
.collect();
|
||||
let gameplay_passed = scenario.is_none_or(|scenario| {
|
||||
crate::browser::required_viewport_playtests_passed(
|
||||
scenario,
|
||||
&evidence.viewport_playtests,
|
||||
)
|
||||
});
|
||||
let passed = evidence.passed && gameplay_passed;
|
||||
let summary = render_direct_browser_tool_evidence(root, &evidence, sequence);
|
||||
(
|
||||
json!({"kind":"browser", "mode":input.mode, "scenario":scenario,
|
||||
"scenarioFingerprint":scenario.map(crate::browser::browser_playtest_scenario_fingerprint), "summary":summary, "screenshots":screenshots,
|
||||
"reportPath":evidence.evidence.report_path.strip_prefix(root).ok().map(|p|p.to_string_lossy().replace('\\', "/")),
|
||||
"visualViewports":["desktop","mobile"],
|
||||
"gameplayViewports":evidence.viewport_playtests.iter().map(|v|v.viewport).collect::<Vec<_>>(),
|
||||
"coverage":if scenario.is_some() {"dual-viewport-fixed-scenario"} else {"visual-only"}}),
|
||||
passed,
|
||||
)
|
||||
let result = browser_tool_result(root, &evidence, scenario, sequence);
|
||||
let passed = result["passed"] == true;
|
||||
(result, passed)
|
||||
}
|
||||
Err(error) => (
|
||||
json!({"mode":input.mode,"needsReconciliation":error.starts_with("browser-cleanup-unconfirmed:"),"error":truncate_agent_runtime_text(&error, 1800)}),
|
||||
@@ -599,9 +706,147 @@ pub(super) async fn run_command(root: &Path, arguments: &Value) -> Result<Value,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
pub(super) mod tests {
|
||||
use super::*;
|
||||
|
||||
pub(crate) fn browser_evidence_fixture(root: &Path) -> crate::browser::BrowserValidationResult {
|
||||
let mut viewports = Vec::new();
|
||||
let mut playtests = Vec::new();
|
||||
for viewport in ["desktop", "mobile"] {
|
||||
viewports.push(json!({
|
||||
"viewport":viewport, "width":1280, "height":720,
|
||||
"finalUrl":"http://127.0.0.1/", "title":"fixture", "readyState":"complete",
|
||||
"visibleTextSummary":"", "visibleTextCharacterCount":0, "domCharacterCount":0,
|
||||
"expectedText":[], "consoleErrors":[], "consoleWarnings":[], "exceptions":[],
|
||||
"failedRequests":[], "canvases":[], "diagnostics":[], "passed":true,
|
||||
"blockedPopupCount":0, "blockedDialogCount":0, "blockedDownloadCount":0,
|
||||
"blockedPermissionCount":0, "blockedServiceWorkerCount":0,
|
||||
"screenshotPath":root.join(format!("{viewport}.png"))
|
||||
}));
|
||||
playtests.push(json!({"viewport":viewport, "result":{
|
||||
"scenario":"generic-v1", "scenarioFingerprint":"fixture", "passed":true,
|
||||
"initialSequence":0, "initialPhase":"ready", "initialLevel":1,
|
||||
"finalSequence":3, "finalPhase":"ready", "finalLevel":1,
|
||||
"assertions":[
|
||||
{"name":"start-control-clicked", "passed":true},
|
||||
{"name":"start-phase-playing", "passed":true},
|
||||
{"name":"restart-control-clicked", "passed":true}
|
||||
], "diagnostics":[]
|
||||
}}));
|
||||
}
|
||||
serde_json::from_value(json!({
|
||||
"schemaVersion":"browser-validation.v1", "url":"http://127.0.0.1/",
|
||||
"browser":{"kind":"chrome", "product":"fixture", "protocolVersion":"1.3"},
|
||||
"passed":true, "viewportResults":viewports, "viewportPlaytests":playtests,
|
||||
"diagnostics":[], "completedAtUnixMs":1,
|
||||
"evidence":{"root":root, "reportPath":root.join("validation.json")}
|
||||
}))
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
pub(crate) fn failed_browser_evidence_fixture(
|
||||
root: &Path,
|
||||
) -> crate::browser::BrowserValidationResult {
|
||||
let mut evidence = browser_evidence_fixture(root);
|
||||
evidence.passed = false;
|
||||
for (index, entry) in evidence.viewport_playtests.iter_mut().enumerate() {
|
||||
entry.result.passed = false;
|
||||
entry.result.final_phase = Some(crate::browser::BrowserPlaytestPhase::Ready);
|
||||
entry.result.final_sequence = Some(index as u64);
|
||||
entry.result.diagnostics = vec![if index == 0 {
|
||||
"固定试玩控件 start 处于 disabled 状态".into()
|
||||
} else {
|
||||
"generic-v1 start 后必须进入 playing".into()
|
||||
}];
|
||||
for (assertion_index, assertion) in entry.result.assertions.iter_mut().enumerate() {
|
||||
assertion.passed = index == 1 && assertion_index == 0;
|
||||
}
|
||||
}
|
||||
evidence
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn browser_tool_projection_distinguishes_gameplay_success_visual_only_and_missing_viewport() {
|
||||
use crate::browser::BrowserPlaytestScenario::GenericV1;
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
for mode in ["gameplay", "visual", "missing-mobile"] {
|
||||
let mut evidence = browser_evidence_fixture(root.path());
|
||||
let scenario = if mode == "visual" {
|
||||
evidence.viewport_playtests.clear();
|
||||
None
|
||||
} else {
|
||||
Some(GenericV1)
|
||||
};
|
||||
if mode == "missing-mobile" {
|
||||
evidence.viewport_playtests.pop();
|
||||
evidence.passed = false;
|
||||
}
|
||||
let receipt = browser_tool_result(root.path(), &evidence, scenario, 1);
|
||||
let summary = receipt["summary"].as_str().unwrap();
|
||||
assert_eq!(receipt["passed"], mode != "missing-mobile");
|
||||
if mode == "visual" {
|
||||
assert_eq!(receipt["gameplayResults"], json!([]));
|
||||
assert!(summary
|
||||
.lines()
|
||||
.any(|line| line.starts_with("玩法检查:未执行")));
|
||||
assert_eq!(receipt["coverage"], "visual-only");
|
||||
continue;
|
||||
}
|
||||
for (index, viewport) in ["desktop", "mobile"].iter().enumerate() {
|
||||
let result = &receipt["gameplayResults"][index];
|
||||
assert_eq!(result["viewport"], *viewport);
|
||||
if mode == "missing-mobile" && index == 1 {
|
||||
assert_eq!(result["passed"], false);
|
||||
assert!(result["diagnostics"][0].as_str().unwrap().contains("缺失"));
|
||||
assert!(summary.contains("mobile: gameplayPassed=false"));
|
||||
assert_eq!(result["assertions"], json!([]));
|
||||
assert_eq!(result["finalPhase"], Value::Null);
|
||||
} else {
|
||||
assert_eq!(result["passed"], true);
|
||||
assert_eq!(result["diagnostics"], json!([]));
|
||||
assert!(result["assertions"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.all(|a| a["passed"] == true));
|
||||
assert!(summary.contains(&format!("{viewport}: gameplayPassed=true")));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn browser_tool_projection_bounds_diagnostics_without_losing_either_blocker() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let mut evidence = failed_browser_evidence_fixture(root.path());
|
||||
for viewport in &mut evidence.viewport_playtests {
|
||||
viewport
|
||||
.result
|
||||
.diagnostics
|
||||
.extend((0..8).map(|_| "额外诊断".repeat(200)));
|
||||
}
|
||||
let receipt = browser_tool_result(
|
||||
root.path(),
|
||||
&evidence,
|
||||
Some(crate::browser::BrowserPlaytestScenario::GenericV1),
|
||||
1,
|
||||
);
|
||||
for (index, entry) in evidence.viewport_playtests.iter().enumerate() {
|
||||
let diagnostics = receipt["gameplayResults"][index]["diagnostics"]
|
||||
.as_array()
|
||||
.unwrap();
|
||||
assert_eq!(diagnostics.len(), 4);
|
||||
assert_eq!(diagnostics[0], entry.result.diagnostics[0]);
|
||||
assert!(diagnostics
|
||||
.iter()
|
||||
.all(|d| d.as_str().unwrap().chars().count() <= 512));
|
||||
assert!(receipt["summary"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.contains(&entry.result.diagnostics[0]));
|
||||
}
|
||||
}
|
||||
|
||||
fn project() -> (tempfile::TempDir, std::path::PathBuf) {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("project");
|
||||
|
||||
@@ -1514,6 +1514,41 @@ async fn real_chrome_generic_playtest_pointer_events_keeps_unavailable_controls_
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[ignore = "requires an installed Chrome/Chromium/Edge and explicit local browser execution"]
|
||||
async fn real_chrome_generic_playtest_reports_start_click_without_playing_phase() {
|
||||
let result = run_stable_generic_viewport_fixture(false, Some(
|
||||
r#"<button type="button" data-playtest-id="start" data-start-phase="ready">Start</button>"#,
|
||||
)).await;
|
||||
assert!(!result.passed);
|
||||
assert!(result
|
||||
.viewport_results
|
||||
.iter()
|
||||
.all(|viewport| viewport.passed));
|
||||
for entry in &result.viewport_playtests {
|
||||
let playtest = &entry.result;
|
||||
assert!(!playtest.passed);
|
||||
assert_eq!(playtest.final_phase, Some(BrowserPlaytestPhase::Ready));
|
||||
assert_eq!(playtest.final_sequence, Some(1));
|
||||
for (name, expected) in [
|
||||
("start-control-clicked", true),
|
||||
("start-phase-playing", false),
|
||||
] {
|
||||
assert_eq!(
|
||||
playtest
|
||||
.assertions
|
||||
.iter()
|
||||
.find(|assertion| assertion.name == name)
|
||||
.unwrap()
|
||||
.passed,
|
||||
expected
|
||||
);
|
||||
}
|
||||
let reason = &playtest.diagnostics[0];
|
||||
assert!(reason.contains("必须进入 playing"), "{reason}");
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_stable_generic_viewport_fixture(
|
||||
mobile_action_broken: bool,
|
||||
start_markup: Option<&str>,
|
||||
@@ -1556,7 +1591,7 @@ async fn run_stable_generic_viewport_fixture(
|
||||
};
|
||||
document.querySelector('[data-playtest-id="start"]').addEventListener('click', (event) => {
|
||||
if (event.isTrusted && state.phase === 'ready') {
|
||||
advance(() => { state.phase = 'playing'; });
|
||||
advance(() => { state.phase = event.currentTarget.dataset.startPhase || 'playing'; });
|
||||
}
|
||||
});
|
||||
const primary = document.querySelector('[data-playtest-id="primary-action"]');
|
||||
@@ -1641,6 +1676,57 @@ async fn run_stable_generic_viewport_fixture(
|
||||
let persisted: serde_json::Value =
|
||||
serde_json::from_slice(&fs::read(&result.evidence.report_path).unwrap()).unwrap();
|
||||
assert_eq!(persisted["viewportPlaytests"].as_array().unwrap().len(), 2);
|
||||
let receipt = crate::agent::browser_tool_result(
|
||||
evidence.path(),
|
||||
&result,
|
||||
Some(BrowserPlaytestScenario::GenericV1),
|
||||
1,
|
||||
);
|
||||
assert_eq!(receipt["passed"], persisted["passed"]);
|
||||
for (index, entry) in persisted["viewportPlaytests"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.enumerate()
|
||||
{
|
||||
let projected = &receipt["gameplayResults"][index];
|
||||
assert_eq!(projected["viewport"], entry["viewport"]);
|
||||
let viewport = entry["viewport"].as_str().unwrap();
|
||||
let line = receipt["summary"]
|
||||
.as_str()
|
||||
.unwrap()
|
||||
.lines()
|
||||
.find(|line| line.starts_with(&format!("{viewport}: gameplayPassed=")))
|
||||
.unwrap();
|
||||
assert!(line.starts_with(&format!(
|
||||
"{viewport}: gameplayPassed={}",
|
||||
entry["result"]["passed"]
|
||||
)));
|
||||
for field in [
|
||||
"passed",
|
||||
"assertions",
|
||||
"initialPhase",
|
||||
"initialSequence",
|
||||
"initialLevel",
|
||||
"finalPhase",
|
||||
"finalSequence",
|
||||
"finalLevel",
|
||||
] {
|
||||
assert_eq!(projected[field], entry["result"][field], "{field}");
|
||||
}
|
||||
if let Some(reason) = entry["result"]["diagnostics"][0].as_str() {
|
||||
assert_eq!(projected["diagnostics"][0], reason);
|
||||
assert!(line.contains(reason));
|
||||
}
|
||||
if let Some(assertion) = entry["result"]["assertions"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.find(|assertion| assertion["passed"] == false)
|
||||
{
|
||||
assert!(line.contains(assertion["name"].as_str().unwrap()));
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
|
||||
@@ -195,6 +195,8 @@ UI 编辑器的“分析参考图”步骤、Rust 命令 `suggest_ui_design_sema
|
||||
### 分层验证与预算
|
||||
|
||||
- 复用客户端浏览器与现有固定玩法场景。视觉检查采集双端画面/布局/资源/诊断;玩法检查分别在 desktop/mobile 执行明确的固定场景和真实输入,按视口保存结果。旧报告缺少移动端玩法结果时保持未知,不补通过。报告必须声明检查层级;视觉通过不能宣称玩法通过,固定场景通过也不能宣称覆盖未执行的完整关卡。
|
||||
- 浏览器工具回执在 `mode=gameplay` 时新增 `gameplayResults` 双端有界投影,每个视口包含 `viewport`、`passed`、`diagnostics`、`assertions` 及 `initialPhase` / `initialSequence` / `initialLevel`、`finalPhase` / `finalSequence` / `finalLevel`。`assertions` 仅列当前固定场景的断言名称和通过状态;每端最多公开 4 条诊断,每条最多 512 字符。缺少某端结果时仍返回该视口 `passed=false`,诊断说明证据缺失,断言为空且六个状态字段为 `null`;断言的 `passed=false` 可包含未执行,不能据此断言该断言已独立执行且失败。`mode=visual` 的 `gameplayResults` 为空,并在摘要中明确玩法未执行。摘要从同一投影给出每端首条诊断和首个未通过断言,不能与结构化结果相矛盾。工具回执不得展开完整宿主报告;`reportPath` 只作宿主证据引用。宿主报告结构、验收门禁、持久化预算和私有路径保护保持原合同,旧回执不回填新字段。
|
||||
- 回执验收覆盖 start 禁用、点击后 phase 不符合和玩法成功;逐视口核对持久报告与摘要中的状态、首条诊断和首个未通过断言,并检查成功、失败两条最终 MCP 回包路径。
|
||||
- 输入或碰撞改变先做定点玩法检查;纯图像/颜色变化做视觉检查;首次交付和影响闭环的修改做所需玩法验证。新增失败或相关代码变化才重跑对应层,不因改说明文字重复完整验证。
|
||||
- 内置试玩和客户端托管的外部 Node/npm 验证共用当前 clientTurnId 的持久化预算。客户端分配递增执行序号,模型提供的旧 attempt 仅作兼容输入,不能减少计数或重置预算;同一轮错误反馈、工具切换和进程重启均不能刷新已消费次数。
|
||||
- 新增本地 validation.maxRuns(默认 3,正整数)独立于 llm.maxRetries;显式配置原样使用,不按角色或运行模式改写。超限直接返回已用/上限和最近证据,停止新的验证。预检与正常构建不计作重复试玩。
|
||||
|
||||
Reference in New Issue
Block a user