experiment-audit

实验审计：跨模型完整性验证

每个审查环节开始前，请遵循 可选审查者： 通过规范的辅助解析器解析调用方式 → .aris/reviewer.json → Claude Code 默认设置（Codex MCP）， 并使用 reviewer_route.py，同时保留此环节的级别、新鲜上下文、源文件／图像输入和确定性门控。 下方的 Codex 示例／固定配置仅适用于 backend=codex；其他路由使用各自的首次调用／回复约定。 保存所选路由和实际返回的身份；无法使用的选择标记为 REVIEW_UNAVAILABLE。根据实际的 模型系列／完成情况／范围判定是否接受。将调用方指定的审查者／模型／effort 覆盖项 传递给包含审查者的子任务。生成工作进程保持不变。

🔒 请勿在 /loop、/schedule 或 CronCreate 中运行此技能。 它会给出判定——判断实验完整性。 定时重复运行这一判定不会增加任何新信号，而一个接受自身输出来决定何时停止的循环会构成自我开脱（acceptance-gate.md）。请安排 在此之前的外部等待——实验完成 → 然后只审计一次。参见 shared-references/external-cadence.md。

审计以下实验的完整性：$ARGUMENTS

存在的原因

LLM Agent 可能会通过以下方式产出虚假的实验结果：

1. 伪造真实值——根据模型输出创建合成“参考”，再将高度一致报告为性能
2. 分数归一化——用模型自身的最大值去除指标，从而得到 0.99+
3. 虚构结果——声称的数字来自不存在的文件或从未调用的函数
4. 范围不足——将 2-scene 试点报告为“全面评估”

这些并非蓄意欺骗——而是缺乏完整性约束、以优化为目标的 Agent 所产生的失败模式。此技能会添加这一约束。

核心原则

执行方收集文件路径。外部审查者后端读取代码并判断完整性。执行方不参与完整性判断。

这遵循 shared-references/reviewer-independence.md 和 shared-references/experiment-integrity.md。

常量

• REVIEWER_BACKEND = codex — 默认值：Codex MCP（ultra）。使用 — reviewer: oracle-pro 可切换为 Oracle MCP，使用 — reviewer: manual 可切换为 Manual Review MCP。如果 Manual Review MCP 不可用，请停止并打印安装命令；不要回退到 Codex。参见 shared-references/reviewer-routing.md。

审查者调用约定

调用审查者时，根据 REVIEWER_BACKEND 分支处理：

如果 REVIEWER_BACKEND = codex： 对新的审查线程使用 mcp__codex__codex。 对后续轮次使用 mcp__codex__codex-reply（复用 threadId）。

如果 REVIEWER_BACKEND = manual： 对新的审查线程使用 mcp__manual_review__review，并传入： prompt：[与发给 Codex 完全相同的提示词] config：{"model_reasoning_effort": "xhigh", "executor_model": "<实际执行方模型>", "require_reviewer_model": true} 保存返回的 threadId。 对后续轮次使用 mcp__manual_review__review_reply，并传入： threadId：[保存的 manual-review threadId] prompt：[后续提示词] config：{"model_reasoning_effort": "xhigh", "executor_model": "<实际执行方模型>", "require_reviewer_model": true}

提示词一致性：发给 Manual Review 的提示词必须与 Codex 收到的文本完全相同。 两个后端均适用审查追踪。

工作流

步骤 1：收集产物（执行器 — Claude）

定位并列出这些文件，但不要阅读或概述其内容：

Scan project directory for:
1. Evaluation scripts:    *eval*.py, *metric*.py, *test*.py, *benchmark*.py
2. Result files:          *.json, *.csv in results/, outputs/, logs/
3. Ground truth paths:    look in eval scripts for data loading (dataset paths, GT references)
4. Experiment tracker:    EXPERIMENT_TRACKER.md, EXPERIMENT_LOG.md
5. Paper claims:          NARRATIVE_REPORT.md, paper/sections/*.tex, PAPER_PLAN.md
6. Config files:          *.yaml, *.toml, *.json configs with metric definitions

带有明确裁决结论的人工审核回复必须以 Reviewer-Model: <exact-model-id> 开头——将本会话实际运行的模型作为 executor_model 传入。缺失、未知或属于同一模型系列的身份信息都不能作为无罪依据； 应输出 REVIEW_UNAVAILABLE，而不是猜测。如果无法说出执行器模型的名称， 则人工审核的跨系列声明无法证实——请在报告中说明这一点，而不要断言该声明成立。

请勿概述、解读或解释任何文件内容。 只收集路径。

步骤 2：发送给审核者

根据选定的审核后端（参见 Reviewer Calling Convention），只向审核者传递文件路径和审核清单。审核者会直接读取所有内容。

对于 codex，调用 mcp__codex__codex，并设置：

• model: gpt-6-astra
• config: {"model_reasoning_effort": "ultra"}
• sandbox: read-only
• cwd: [project directory]
• prompt: [the exact full prompt below]

对于 manual，调用 mcp__manual_review__review，并设置：

• config: {"model_reasoning_effort": "xhigh", "executor_model": "<actual executor model>", "require_reviewer_model": true}
• prompt: [the exact full prompt below]

人工审核不能使用仅限 Codex 的 model、sandbox 或 cwd；请在提示词中包含相同的文件路径，方便用户检查。

两个后端均使用以下完整提示词：

You are an experiment integrity auditor. Start from the assumption that the
    evaluation is compromised somewhere — your job is to find where. Be
    adversarial. Trust nothing the author tells you — verify everything
    yourself. Read ALL files listed below and check for the following fraud
    patterns.

    Files to read:
    - Evaluation scripts: [list paths]
    - Result files: [list paths]
    - Experiment tracker: [list paths]
    - Paper claims: [list paths]
    - Config files: [list paths]

    ## Audit Checklist

    ### A. Ground Truth Provenance
    For each evaluation script:
    1. Where does "ground truth" / "reference" / "target" come from?
    2. Is it loaded from the DATASET, or generated/derived from MODEL OUTPUTS?
    3. If derived: is it explicitly labeled as proxy evaluation?
    4. Are official eval scripts used when available for this benchmark?
    FAIL if: GT is derived from model outputs without explicit proxy labeling.

    ### B. Score Normalization
    For each metric computation:
    1. Is any metric divided by max/min/mean of the model's OWN output?
    2. Are raw scores reported alongside any normalized scores?
    3. Are any scores suspiciously close to 1.0 or 100%?
    FAIL if: Normalization denominator comes from prediction statistics.

    ### C. Result File Existence
    For each claim in the paper/narrative:
    1. Does the referenced result file actually exist?
    2. Does the claimed metric key exist in that file?
    3. Does the claimed NUMBER match what's in the file?
    4. Is the experiment tracker status DONE (not TODO/IN_PROGRESS)?
    FAIL if: Claimed results reference nonexistent files or mismatched numbers.

    ### D. Dead Code Detection
    For each metric function defined in eval scripts:
    1. Is it actually CALLED in any evaluation pipeline?
    2. Does its output appear in any result file?
    WARN if: Metric functions exist but are never called.

    ### E. Scope Assessment
    1. How many scenes/datasets/configurations were actually tested?
    2. How many seeds/runs per configuration?
    3. Does the paper use words like "comprehensive", "extensive", "robust"?
    4. Is the actual scope sufficient for those claims?
    WARN if: Scope language exceeds actual evidence.

    ### F. Evaluation Type Classification
    Classify each evaluation as:
    - real_gt: uses dataset-provided ground truth
    - synthetic_proxy: uses model-generated reference
    - self_supervised_proxy: no GT by design
    - simulation_only: simulated environment
    - human_eval: human judges

    ## Output Format

    For each check (A-F), report:
    - Status: PASS | WARN | FAIL
    - Evidence: exact file:line references
    - Details: what specifically was found

    Overall verdict: PASS | WARN | FAIL
    
    Be thorough. Read every eval script line by line.

步骤 3：解析并撰写报告（执行器 — Claude）

解析审核者的回复，并撰写 EXPERIMENT_AUDIT.md：

# Experiment Audit Report

**Date**: [today]
**Auditor**: External reviewer backend, ultra reasoning (cross-model, read-only)
**Project**: [project name]

## Overall Verdict: [PASS | WARN | FAIL]

## Integrity Status: [pass | warn | fail]

## Checks

### A. Ground Truth Provenance: [PASS|WARN|FAIL]
[details + file:line evidence]

### B. Score Normalization: [PASS|WARN|FAIL]
[details]

### C. Result File Existence: [PASS|WARN|FAIL]
[details]

### D. Dead Code Detection: [PASS|WARN|FAIL]
[details]

### E. Scope Assessment: [PASS|WARN|FAIL]
[details]

### F. Evaluation Type: [real_gt | synthetic_proxy | ...]
[classification + evidence]

## Action Items
- [specific fixes if WARN or FAIL]

## Claim Impact
- Claim 1: [supported | needs qualifier | unsupported]
- Claim 2: ...

同时撰写供机器读取的 EXPERIMENT_AUDIT.json：

{
  "date": "2026-04-10",
  "auditor": "external-reviewer-ultra",
  "overall_verdict": "warn",
  "integrity_status": "warn",
  "checks": {
    "gt_provenance": {"status": "pass", "details": "..."},
    "score_normalization": {"status": "warn", "details": "..."},
    "result_existence": {"status": "pass", "details": "..."},
    "dead_code": {"status": "pass", "details": "..."},
    "scope": {"status": "warn", "details": "..."},
    "eval_type": "real_gt"
  },
  "claims": [
    {"id": "C1", "impact": "supported"},
    {"id": "C2", "impact": "needs_qualifier"}
  ]
}

步骤 4：打印摘要

🔬 Experiment Audit Complete

  GT Provenance:      ✅ PASS — real dataset GT used
  Score Normalization: ⚠️ WARN — boundary metric uses self-reference
  Result Existence:    ✅ PASS — all files exist, numbers match
  Dead Code:           ✅ PASS — all metric functions called
  Scope:               ⚠️ WARN — 2 scenes, paper says "comprehensive"

  Overall: ⚠️ WARN
  
  See EXPERIMENT_AUDIT.md for details.

与其他技能的集成

在 /research-pipeline 中自动运行（仅提供建议，绝不阻塞）

集成到流水线后，此技能会在 /experiment-bridge 之后、/auto-review-loop 之前自动运行：

/experiment-bridge → results ready
    ↓
/experiment-audit (automatic, advisory)
    ├── PASS  → continue normally
    ├── WARN  → print ⚠️ warning, continue, tag claims as [INTEGRITY: WARN]
    └── FAIL  → print 🔴 alert, continue, tag claims as [INTEGRITY CONCERN]
    ↓
/auto-review-loop → proceeds with integrity tags visible to reviewer

绝不阻塞流水线。 即使结果为 FAIL，流水线也会继续运行——但相关声明会带有醒目的完整性标签。

由 /result-to-claim 读取（如果存在）

if EXPERIMENT_AUDIT.json exists:
    read integrity_status
    attach to verdict: {claim_supported: "yes", integrity_status: "warn"}
    if integrity_status == "fail":
        downgrade verdict display: "yes [INTEGRITY CONCERN]"
else:
    verdict as normal, integrity_status = "unavailable"
    mark as "provisional — no integrity audit"

由 /paper-write 读取（如果存在）

if EXPERIMENT_AUDIT.json exists AND integrity_status == "fail":
    add footnote to affected claims: "Note: integrity audit flagged concerns with this evaluation"

关键规则

• 审核者独立性：执行器收集路径，审核者负责判断。仅此而已。
• 绝不阻塞：明确发出警告，但绝不停止流水线。
• 文件开关：没有 EXPERIMENT_AUDIT.md = 此技能从未运行 = 对现有行为没有影响。
• 跨模型：审核者必须与执行器属于不同模型系列。
• 如实说明限制：审核可捕捉常见模式，但无法发现所有可能的欺诈行为。它是安全网，而非保证。

致谢

本项目受到社区报告的完整性问题（#57、#131）启发；相关问题中，执行器 Agent 伪造了真实依据并自我规范化评分。

审核追踪

每次调用审核者后（mcp__codex__codex、mcp__codex__codex-reply、mcp__manual_review__review 或 mcp__manual_review__review_reply），都要按照 shared-references/review-tracing.md 保存追踪记录（Policy C — 取证；绝不可悄然跳过）。使用 save_trace.sh（根据 shared-references/integration-contract.md §2 中的链条解析），或直接将文件写入 .aris/traces/<skill>/<date>_run<NN>/。遵循 --- trace: 参数（默认值：full）。