judge-with-debate

judge-with-debate

通过多 Agent 辩论评估解决方案：独立评审者分析并质疑彼此的评估，然后反复完善评估，直至达成共识或达到最大轮数。 此命令采用多 Agent 辩论模式进行高质量评估，通过多种视角和严谨论证提升评估准确性。与单轮评估不同，辩论要求评审者以证据为依据捍卫自己的观点，并考虑反方论据。

主要优势：

• 结构化评估 - 元评审者会在评审开始前制定量身定制的评分细则和标准
• 多种视角 - 三位独立评审者可减少个人偏见
• 基于证据的辩论 - 评审者以解决方案和评估规范中的具体证据捍卫自己的观点
• 迭代完善 - 最多 3 轮辩论，推动评分逐渐趋于准确
• 共享规范 - 元评审者只运行一次；所有轮次中的所有评审者共用同一评估规范

模式：基于辩论的评估

此命令采用迭代式多评审者辩论：

Phase 0: Setup
         mkdir -p .specs/reports
                  |
Phase 0.5: Dispatch Meta-Judge
         Meta-Judge (Opus)
              |
         Evaluation Specification YAML
              |
Phase 1: Independent Analysis (3 judges in parallel)
         +- Judge 1 -> {name}.1.md -+
Solution +- Judge 2 -> {name}.2.md -+-+
         +- Judge 3 -> {name}.3.md -+ |
                                      |
Phase 2: Debate Round (iterative)     |
    Each judge reads others' reports  |
         |                            |
    Argue + Defend + Challenge        |
    (grounded in eval specification)  |
         |                            |
    Revise if convinced --------------+
         |                            |
    Check consensus                   |
         +- Yes -> Final Report       |
         +- No -> Next Round ---------+

流程

设置：创建报告目录

开始评估前，确保报告目录存在：

mkdir -p .specs/reports

报告命名约定： .specs/reports/{solution-name}-{YYYY-MM-DD}.[1|2|3].md

其中：

• {solution-name} - 从解决方案文件名中提取（例如，从 src/api/users.ts 提取 users-api）
• {YYYY-MM-DD} - 当前日期
• [1|2|3] - 评审者编号

阶段 0.5：派遣元评审者

在进行独立分析前，派遣一个元评审者 Agent，生成量身定制的评估规范。元评审者只运行一次，并生成评分细则、检查清单和评分标准，供所有评审者在所有轮次中使用。

元评审者提示词模板：

## Task

Generate an evaluation specification yaml for the following evaluation task. You will produce rubrics, checklists, and scoring criteria that multiple judge agents will use to evaluate the solution through independent analysis and multi-round debate.

CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`

## User Prompt
{task description - what the solution was supposed to accomplish}

## Context
{Any relevant context about the solution being evaluated}

## Artifact Type
{code | documentation | configuration | etc.}

## Evaluation Mode
Multi-judge debate with consensus-seeking across rounds

## Instructions
Return only the final evaluation specification YAML in your response.
The specification should support both independent analysis and debate-based refinement.

派遣：

Use Task tool:
  - description: "Meta-judge: generate evaluation specification for {solution-name}"
  - prompt: {meta-judge prompt}
  - model: opus
  - subagent_type: "sadd:meta-judge"

等待元评审者完成，并从其输出中提取评估规范 YAML，然后再进入阶段 1。

阶段 1：独立分析

并行启动 3 个独立评审者 Agent（为保证严谨性，使用 Opus）：

1. 每位评审者都会收到：
   • 正在评估的解决方案路径
   • 元评审者的评估规范 YAML
   • 任务说明
2. 每位评审者都会生成独立评估报告，并保存到 .specs/reports/{solution-name}-{date}.[1|2|3].md
3. 报告必须包含：
   • 各项标准的评分及证据
   • 支持评分的具体引文或示例
   • 加权总分
   • 主要优势和不足

关键原则： 初步分析保持独立，以避免群体思维。

初始评审者的提示词模板：

You are Judge {N} evaluating a solution independently against an evaluation specification produced by the meta judge.

CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`

## Solution
{path to solution file(s)}

## Task Description
{what the solution was supposed to accomplish}

## Evaluation Specification

```yaml
{meta-judge's evaluation specification YAML}

输出文件

.specs/reports/{solution-name}-{date}.{N}.md

说明

请遵循 Agent 说明中定义的完整评审流程！

补充说明：

1. 彻底阅读解决方案
2. 对于评估规范中的每项标准：
   • 找出具体证据（引用原文）
   • 按照规定的量表评分
   • 使用具体示例说明评分理由
3. 计算加权总分
4. 将完整报告写入 {output_file}

在报告开头添加 Done by Judge {N}

**Dispatch each judge:**

使用 Task 工具：

• description: "Judge {N}: 对 {solution-name} 进行独立分析"
• prompt: {包含评估规范 YAML 的评审者提示词}
• model: opus
• subagent_type: "sadd:judge"

### Phase 2: Debate Rounds (Iterative)

For each debate round (max 3 rounds):

Launch **3 debate agents in parallel**:

1. Each judge agent receives:
   - Path to their own previous report (`.specs/reports/{solution-name}-{date}.[1|2|3].md`)
   - Paths to other judges' reports (`.specs/reports/{solution-name}-{date}.[1|2|3].md`)
   - The original solution
   - The meta-judge's evaluation specification YAML
2. Each judge:
   - Identifies disagreements with other judges (>1 point score gap on any criterion)
   - Defends their own ratings with evidence from the solution and evaluation specification
   - Challenges other judges' ratings they disagree with
   - Considers counter-arguments
   - Revises their assessment if convinced
3. Updates their report file with new section: `## Debate Round {R}`
4. After they reply, if they reached agreement move to Phase 3: Consensus Report

**Key principle:** Judges communicate only through filesystem - orchestrator doesn't mediate and don't read reports files itself, it can overflow your context.

**Prompt template for debate judges:**

```markdown
You are Judge {N} in debate round {R}.

CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`

## Your Previous Report
{path to .specs/reports/{solution-name}-{date}.{N}.md}

## Other Judges' Reports
Judge 1: .specs/reports/{solution-name}-{date}.1.md
...

## Task Description
{what the solution was supposed to accomplish}

## Solution
{path to solution}

## Evaluation Specification

```yaml
{meta-judge's evaluation specification YAML}

输出文件

.specs/reports/{solution-name}-{date}.{N}.md（追加到现有文件）

说明

按照 Agent 指令中定义的完整评审流程执行！

其他辩论说明：

1. 阅读你之前的评估，路径为 {your_previous_report}
2. 阅读所有其他评审员的报告
3. 找出分歧（评分相差超过 1 分的情况）
4. 对于每个重大分歧：
   • 清楚说明分歧
   • 根据解决方案和评估规范中的证据为自己的立场辩护
   • 用反面证据质疑其他评审员的立场
   • 考虑他们提供的证据是否会改变你的看法
5. 通过追加辩论轮次部分来更新你的报告文件
6. 回复是否达成一致，以及与哪位评审员达成一致。列出重新审视后的总分和各项标准得分。

**Dispatch each debate judge:**

使用 Task 工具：

• description: "Judge {N}: debate round {R} for {solution-name}"
• prompt: {含评估规范 YAML 的辩论评审提示词}
• model: opus
• subagent_type: "sadd:judge"

### Consensus Check

After each debate round, check for consensus:

**Consensus achieved if:**
- All judges' overall scores within 0.5 points of each other
- No criterion has >1 point disagreement across any two judges
- All judges explicitly state they accept the consensus

**If no consensus after 3 rounds:**
- Report persistent disagreements
- Provide all judge reports for human review
- Flag that automated evaluation couldn't reach consensus

**Orchestration Instructions:**

**Step 1: Dispatch Meta-Judge (Phase 0.5)**

1. Launch meta-judge agent
2. Wait for meta-judge to complete
3. Extract the evaluation specification YAML from meta-judge output

**Step 2: Run Independent Analysis (Phase 1)**

1. Launch 3 judge agents in parallel (Judge 1, 2, 3) with the evaluation specification YAML
2. Each writes their independent assessment to `.specs/reports/{solution-name}-{date}.[1|2|3].md`
3. Wait for all 3 agents to complete

**Step 3: Check for Consensus**

Let's work through this systematically to ensure accurate consensus detection.

Read all three reports and extract:
- Each judge's overall weighted score
- Each judge's score for every criterion

Check consensus step by step:
1. First, extract all overall scores from each report and list them explicitly
2. Calculate the difference between the highest and lowest overall scores
   - If difference <= 0.5 points -> overall consensus achieved
   - If difference > 0.5 points -> no consensus yet
3. Next, for each criterion, list all three judges' scores side by side
4. For each criterion, calculate the difference between highest and lowest scores
   - If any criterion has difference > 1.0 point -> no consensus on that criterion
5. Finally, verify consensus is achieved only if BOTH conditions are met:
   - Overall scores within 0.5 points
   - All criterion scores within 1.0 point

**Step 4: Decision Point**

- **If consensus achieved**: Go to Step 6 (Generate Consensus Report)
- **If no consensus AND round < 3**: Go to Step 5 (Run Debate Round)
- **If no consensus AND round = 3**: Go to Step 7 (Report No Consensus)

**Step 5: Run Debate Round**

1. Increment round counter (round = round + 1)
2. Launch 3 judge agents in parallel with the same evaluation specification YAML
3. Each agent reads:
   - Their own previous report from filesystem
   - Other judges' reports from filesystem
   - Original solution
4. Each agent appends "Debate Round {R}" section to their own report file
5. Wait for all 3 agents to complete
6. Go back to Step 3 (Check for Consensus)

**Step 6: Reply with Report**

Let's synthesize the evaluation results step by step.

1. Read all final reports carefully
2. Before generating the report, analyze the following:
   - What is the consensus status (achieved or not)?
   - What were the key points of agreement across all judges?
   - What were the main areas of disagreement, if any?
   - How did the debate rounds change the evaluations?
3. Reply to user with a report that contains:
   - If there is consensus:
     - Consensus scores (average of all judges)
     - Consensus strengths/weaknesses
     - Number of rounds to reach consensus
     - Final recommendation with clear justification
   - If there is no consensus:
       - All judges' final scores showing disagreements
       - Specific criteria where consensus wasn't reached
       - Analysis of why consensus couldn't be reached
       - Flag for human review
4. Command complete

**Step 7: Report No Consensus**

- Report persistent disagreements
- Provide all judge reports for human review
- Flag that automated evaluation couldn't reach consensus

### Phase 3: Consensus Report

If consensus achieved, synthesize the final report by working through each section methodically:

```markdown
# Consensus Evaluation Report

Let's compile the final consensus by analyzing each component systematically.

## Consensus Scores

First, let's consolidate all judges' final scores:

| Criterion | Judge 1 | Judge 2 | Judge 3 | Final |
|-----------|---------|---------|---------|-------|
| {Name}    | {X}/5   | {X}/5   | {X}/5   | {X}/5 |
...

**Consensus Overall Score**: {avg}/5.0

## Consensus Strengths
[Review each judge's identified strengths and extract the common themes that all judges agreed upon]

## Consensus Weaknesses
[Review each judge's identified weaknesses and extract the common themes that all judges agreed upon]

## Debate Summary
Let's trace how consensus was reached:
- Rounds to consensus: {N}
- Initial disagreements: {list with specific criteria and score gaps}
- How resolved: {for each disagreement, explain what evidence or argument led to resolution}

## Final Recommendation
Based on the consensus scores and the key strengths/weaknesses identified:
{Pass/Fail/Needs Revision with clear justification tied to the evidence}

该命令会生成：

1. 报告目录：.specs/reports/（如果不存在则创建）
2. 初始报告：.specs/reports/{solution-name}-{date}.1.md、.specs/reports/{solution-name}-{date}.2.md、.specs/reports/{solution-name}-{date}.3.md
3. 辩论更新：每轮辩论都会在各报告文件中追加相应部分
4. 最终综合结果：回复用户（共识或分歧总结）

最佳实践

元评审员 + 评审员核验

• 切勿跳过元评审员——量身定制的评估标准能带来更好的评审和更有依据的辩论
• 元评审员只运行一次——在所有辩论轮次中，3 位评审员使用相同的规范
• 包含 CLAUDE_PLUGIN_ROOT——元评审员和评审员都需要解析后的插件根路径
• 元评审员 YAML——只将 YAML 传给评审员，不要修改
• 基于规范开展辩论——评审员在为自己的立场辩护时，应引用评估规范中的标准

常见问题

• 评审员新建报告，而非追加内容——会丢失辩论历史
• 编排器在评审员之间传递报告——违反基于文件系统的通信原则
• 初始评估薄弱——输入差，输出也会差
• 辩论轮次过多——超过 3 轮后收益递减
• 辩论中的谄媚迎合——没有真正证据就轻易同意
• 修改元评审员 YAML——必须将规范原样传给所有评审员
• 在轮次之间重新运行元评审员——规范只生成一次，并在各轮之间共享

请这样做

• 评审员将内容追加到自己的报告文件
• 评审员直接从文件系统读取其他报告
• 进行有充分证据支持的初始评估
• 最多进行 3 轮辩论
• 立场变更必须有证据支持
• 辩论论点应以评估规范中的标准为依据
• 所有轮次使用相同的评估规范

使用示例

评估 API 实现

/judge-with-debate Implement REST API for user management --solution "src/api/users.ts" 

阶段 0.5 - 元评审（假设日期为 2025-01-15）：

• 元评审生成包含以下标准的评估规范 YAML：
  • 正确性（30%）、设计（25%）、安全性（20%）、性能（15%）、文档（10%）
  • 每项标准的评分细则、检查清单和评分定义

阶段 1 - 独立分析（3 位评审收到规范）：

• .specs/reports/users-api-2025-01-15.1.md - 评审员 1 对正确性打 4/5 分，对安全性打 3/5 分
• .specs/reports/users-api-2025-01-15.2.md - 评审员 2 对正确性打 4/5 分，对安全性打 5/5 分
• .specs/reports/users-api-2025-01-15.3.md - 评审员 3 对正确性打 5/5 分，对安全性打 4/5 分

**检测到分歧：**安全性评分在 3-5 分之间

阶段 2 - 辩论第 1 轮（评审员参考评估规范）：

• 评审员 1 为 3/5 分辩护：“缺少速率限制，且根据规范检查清单第 4 项，输入验证不完整”
• 评审员 2 提出异议：“中间件中已有速率限制（第 45 行），符合规范评分细则”
• 评审员 1 修改为 4/5 分：“我漏看了中间件，但根据规范，输入验证仍然薄弱”
• 评审员 3 为 4/5 分辩护：“对于规范定义的要求，输入验证已经足够”

第 1 轮辩论结果：

• 现在所有评审员对安全性的评分均为 4-5/5（相差不超过 1 分）
• 对输入验证仍有分歧

辩论第 2 轮（使用同一份评估规范）：

• 评审员根据规范标准检查具体的验证代码
• 评审员 2 修改为 4/5 分：“重新检查后发现，根据规范检查清单，电子邮件验证正则表达式不够严谨”
• 达成共识：安全性 = 4/5

最终共识：

Correctness: 4.3/5
Design: 4.5/5
Security: 4.0/5 (2 debate rounds to consensus)
Performance: 4.7/5
Documentation: 4.0/5

Overall: 4.3/5 - PASS