simmer-judge-board

Simmer Judge Board

派遣一个评审团，让评审者独立打分、审议并达成共识分数和单一 ASI。评审团的输出与单个评审者的输出完全相同——编排器无法分辨两者。

为什么需要评审团

单个评审者有盲点。它会锚定在最先注意到的内容上，其 ASI 也只反映一种视角。三位视角各异的评审者能发现不同问题，并相互质疑。经过审议得出的 ASI 更有力，因为盲点会被揭示出来。

这在陷入平台期时尤其重要——单个评审者看不到真正的瓶颈，因此总是建议同一类修复方案。

接收的上下文

评审团接收与单个评审者相同的上下文（由编排器传入）：

• 当前候选内容：完整的产物文本或工作区关键文件
• 评估标准评分细则：2–3 项标准，并说明达到 10/10 的表现
• 迭代编号：当前轮次
• 种子校准（迭代 1+）：原始种子 + 第 0 次迭代的分数
• 评估器输出（评估器模式下）：评估器命令的 stdout/stderr
• 上下文纪律补充信息（仅限代码/管线）：先前的 ASI、迭代历史、搜索空间、探索状态

此外，还包括评审团专属上下文：

• JUDGE_PANEL（可选）：设置说明中的自定义评审者定义
• 问题类别：文本/创意、代码/可测试或管线/工程
• ARTIFACT_TYPE：单文件或工作区——这决定了生成器可以更改哪些内容
• SEARCH_SPACE（如果已定义）：明确界定可探索范围（模型、拓扑、提示词文件、配置参数）
• BACKGROUND：限制条件、可用资源、执行环境（模型规模、基础设施、预算）。评审者需要这些信息，才能根据执行者实际可行的操作校准其 ASI。
• 先前审议摘要（迭代 2+）：结构为 WORKING / NOT WORKING / DIRECTION。评审者必须遵循 WORKING 列表——其中的内容在多次迭代中一直保持稳定，不应删除或更改。NOT WORKING 列表可避免重试失败的方法。DIRECTION 是评审团进行策略推理的依据。应在先前结论的基础上继续推进，而不是每次迭代都从头推理。

修改范围

评审者必须了解生成器实际可以更改哪些内容。 如果 ASI 建议的内容超出生成器的范围，就毫无价值。

产物类型	生成器可以更改	生成器不能更改
单文件	产物的文本内容（提示词、文档、配置文件）	模型选择、代码、基础设施、管线拓扑、添加新文件
工作区	工作区目录中的任意文件——代码、配置、提示词、脚本，以及新增文件	工作区之外的内容、搜索空间中未包含的外部基础设施

如果定义了 SEARCH_SPACE，它会进一步限制可操作的范围。生成器只能在搜索空间界限内更改内容。

评审团给出的每条 ASI 都必须能在这些范围内执行。 如果评审团认为“模型是瓶颈”，但产物是单文件（无法更换模型），那么 ASI 应该说“仅修改提示词无法突破这个上限——建议切换到工作区模式或提前终止”，而不是建议生成器无法执行的模型更换。

评审者必须在打分前阅读相关产物。 阅读候选内容、评估器脚本、配置文件和先前的候选版本。了解系统的工作方式以及得分的原因。如果发现不熟悉如何修复的失败模式，应研究相关方法。ASI 应源于深入理解，而不是只看指标后凭空猜测。

三个阶段

阶段 1：独立评分（并行）

将 3 名评审作为并行子 Agent 分派。每位评审都会收到完整的评审上下文，以及一项独特的视角分配。

每位评审调用 simmer:simmer-judge —— 复用现有的评审技能。视角是用于设定其评审角度的前置上下文。

评审小组成员提示词模板：

You are one of three judges on a simmer judge board. Your role is to
score from your specific lens — the other judges cover other angles.

YOUR LENS: [name]
[lens description — what to focus on, what perspective you bring]

ARTIFACT_TYPE: [single-file | workspace]
SEARCH_SPACE: [what's in scope to change — omit if unconstrained]
WHAT THE GENERATOR CAN CHANGE: [single-file: text only | workspace: files in scope]

BACKGROUND:
[constraints, model size, infrastructure, available resources]

PREVIOUS PANEL DELIBERATION:
[WORKING / NOT WORKING / DIRECTION from last round — omit on iteration 0]

FILES YOU SHOULD READ:
- Candidate: [e.g., ./docs/simmer/iteration-2-candidate.md]
- Evaluator script: [e.g., ./evaluate.sh — omit if judge-only]
- Ground truth / test data: [e.g., ./test-data/expected.json — omit if unknown]
- Prior candidates: [e.g., ./docs/simmer/iteration-0-candidate.md, iteration-1-candidate.md — code/pipeline only]
- Config: [e.g., ./config.json — omit if none]

─── STEP 1: INVESTIGATE (required, before scoring) ───

Read the files listed above. Understand the problem before judging it.

On iteration 0 (seed):
- Read the evaluator script — understand HOW it scores (exact match?
  fuzzy? case-sensitive? what format does it expect?)
- Read the ground truth if accessible — what's the theoretical maximum?
  Are there unreachable targets?
- Read the background constraints — what can the model actually do?

Every iteration:
- Read the candidate file — structure and formatting matter, not just
  the text summary in this prompt
- [Code/pipeline only] Read prior candidates — what structural changes
  were tried and what was their effect? (Text/creative judges do NOT
  read prior candidates — this prevents anchoring to previous versions)
- When you see a failure pattern you don't know how to fix, SEARCH
  for solutions before proposing your ASI

─── STEP 2: SCORE (with full understanding) ───

Score ALL criteria from your lens — not just one. Your lens frames
HOW you analyze, not WHAT you analyze. Every judge scores every
criterion from their unique perspective. This gives cross-criterion
insight — one criterion improving while another regresses is a
trade-off the board needs to surface.

Your scores should be grounded in what you found during investigation,
not just observation of the evaluator output.

─── STEP 3: ASI (informed by research) ───

Your ASI candidate must:
- Be actionable within the generator's bounds
- Cite what you found during investigation — reference the evaluator
  mechanics, prior iteration results, or research you did
- Reference prior iterations when relevant — what was tried, why it
  worked or didn't, and how your suggestion differs
- If you searched for solutions, cite what you found

If the bottleneck is outside the generator's bounds, say so — the
clerk will recommend early termination or mode change.

调用技能：simmer:simmer-judge

[... 来自编排器的标准评审上下文 ...]

Each panelist produces the standard judge output format:

第 [N] 次迭代评分： [标准]：[N]/10 — [推理] — [具体改进建议] 综合评分：[N.N]/10

ASI（影响最大的改进方向）： [其 ASI 候选项]

### Phase 2: Deliberation (One Round)

After collecting all independent scores, run one deliberation round. Each judge sees the other judges' scores + reasoning (but NOT their ASI candidates yet — ASI deliberation happens in synthesis).

**Deliberation prompt per panelist:**

你正在参与 simmer 评审团的讨论。

你的独立评分（来自阶段 1）： [此评审的完整输出]

评审 B 的评分： [评审 B 的评分和推理，不包括其 ASI]

评审 C 的评分： [评审 C 的评分和推理，不包括其 ASI]

审阅其他评审的评分和推理。针对每项标准：

1. 同意——如果同意，请简要说明
2. 质疑——如果不同意，请说明其他评审遗漏或判断有误之处。 引用候选内容或评估器输出中的证据。
3. 让步——如果其他评审的推理改变了你的看法，请修改 评分并说明原因

仅进行一轮讨论。直截了当。

讨论： [标准]：[同意/质疑/让步] — [推理] — [修改后的评分（如有变更），或“维持原分”]

**Why one round:** Iterative deliberation risks convergence to the loudest voice. One round of "see and react" surfaces blind spots without losing independence.

**Why withhold ASI candidates:** If judges see each other's ASI during deliberation, they'll anchor on someone else's direction instead of defending their own perspective. ASI candidates go to the synthesis step.

### Phase 3: Synthesis (Clerk)

The board orchestrator acts as clerk. No subagent needed — this runs inline.

**Step 1: Consensus scores.**

For each criterion, take the post-deliberation scores (revised where judges conceded, original where they held):
- **All within 1 point**: use the median. Note agreement.
- **2+ point spread after deliberation**: use the median. Note the tension and why judges disagreed — this informs the ASI.

Compute composite as the average, one decimal place.

**Step 2: Synthesize ASI.**

Read all three judges' independent ASI candidates + the deliberation highlights (challenges, concessions, tensions). Then **distill** — don't select.

The board's value is in its analysis. The ASI's value is in its focus. Don't let the breadth of the analysis dilute the focus of the ASI.

1. **Identify the single highest-leverage move.** Not the most thorough analysis — the one change that would move scores the most. If all three judges point at different things, pick the one that addresses the primary criterion's biggest bottleneck. If you find yourself writing "also" or listing multiple changes, you're diluting. Pick one.

2. **Check alignment with panel conclusions.** If the deliberation concluded "current approach has hit a ceiling," the ASI must propose a structural change (different model, topology, post-processing) — not more of the same approach. If no structural change is possible in the current mode, recommend early termination with a specific next step the user can take.

3. **Check generator feasibility.** In single-file mode, the generator can only edit text. Don't propose code changes, model swaps, or architecture changes unless the artifact type is workspace. Filter for what the generator can actually do in the current mode.

4. **Keep it focused.** The ASI is ONE direction with ONE primary change. The generator should know exactly what to do. A compound ASI ("fix the correction table AND expand the taxonomy AND add self-check") will overwhelm the generator, especially with smaller models. The single judge's strength was focus — the board must match that.

**Step 3: Deliberation summary (for next iteration's panel).**

Write a structured summary with three sections. This gets passed to the next iteration's judges and to the generator as execution context.

有效项（保留，不要删除或更改）：

• [多轮迭代中持续有效的要素，例如：“更正查询表——自第 1 次迭代起持续有效，连续 3 次迭代均保留”]
• [另一项稳定有效的要素]

无效项（不要重复尝试相同方法）：

• [尝试后效果变差的要素，例如：“冗长的规则列表——第 2 次迭代尝试后导致效果退步”]
• [另一种失败的方法]

方向： [评审小组关于下一步工作的结论——用一句话说明。 例如：“将品牌推断作为明确的检查项加入清单——模型擅长处理二元检查。”]

Incorporate the STABLE WINS from the reflect subskill if available — those are tracked across the full trajectory, not just the current deliberation. The deliberation summary adds the panel's reasoning about *why* things worked or didn't.

This structured format ensures:
- The next iteration's judges know what to preserve (stops the pivoting problem)
- Future iterations' judges get richer context about what's actually happening
- The generator knows what's load-bearing and won't strip it when implementing the ASI

**Step 4: Output.**

Produce the standard judge output format:

第 [N] 次迭代评分： [标准]：[N]/10 — [共识推理，如有分歧请注明] — [具体改进建议] 综合评分：[N.N]/10

ASI（影响最大的改进方向）： [单一 ASI——评审小组证据最充分的方向]

This is identical to what a single judge produces. The orchestrator, reflect, and generator cannot tell the difference.

## Judge Composition

The board composes judges **once at the start of the run**, tailored to the specific problem. It reads the artifact, criteria, background, evaluator, and constraints, then builds 3 judges with diverse lenses. These judges are reused for every iteration — they get updated context each round (evaluator output, stable wins, deliberation summaries) but their lens and capabilities stay fixed.

The board uses the primitive library below to construct judges tailored to the specific problem.

### How Composition Works

**At the start of the run (once):**

1. **Read the full problem context.** The artifact, criteria, evaluator output format (if any), background constraints, model capabilities, what's mutable. Understand the problem before designing judges for it.

2. **Design 3 judges with diverse perspectives.** Each judge needs a unique angle on the problem. Ask: "What are the 3 most different ways to evaluate this artifact?" The answer depends on the problem:
   - A prompt with an evaluator might get: **Evaluator Analyst** (deep-dive the metrics), **Constraint Realist** (what can the executor actually handle?), **Downstream User** (does the output work for its intended use?)
   - A DND adventure hook might get: **Craft** (structure, pacing), **Player** (agency, decision points), **DM** (runnability, stat completeness)
   - A cold outreach email might get: **Copywriter** (hook, flow, CTA), **Recipient** (cold-read as a busy VP), **Deliverability** (length, spam triggers, reply friction)

   The judges should be specific to the problem, not generic "Metrics/Strategy/Integration."

3. **Give each judge relevant capabilities from the primitive library.** These are proven capabilities that make judges more effective — apply the relevant ones based on what each judge needs to do.

### Judge Primitive Library

Building blocks for constructing judges. Apply relevant ones to each judge based on their role.

**Core (apply to all judges):**
- Score via seed calibration — score the original first, anchor all subsequent iterations to it
- Diagnose before scoring — read the candidate, evaluator output, and relevant code/config. Understand *why* things are the way they are before writing scores.
- Protect high-scoring elements — identify what's working and constrain your ASI to preserve it
- Score ALL criteria from your lens — every judge scores every criterion from their perspective, not one criterion per judge

**When evaluator is present:**
- Cluster evaluator failures by type — near-misses (spelling), systematic gaps (whole category), noise (hallucinations). The pattern determines the fix.
- Verify proper nouns from lossy sources — transcripts, OCR, and auto-captions garble names
- Flag evaluator stochasticity — if the same config produces different results, small score changes may be noise

**When the problem involves exploration:**
- Review what's been tried — check iteration history before suggesting more of the same
- Flag ceilings — if 2+ iterations tried the same type of change with no improvement, the bottleneck is structural
- Research if stuck — look up how similar problems are solved rather than guessing

### Example Compositions

These show how judges are constructed for different problems — they're examples, not templates.

**Code/pipeline with evaluator:**
| Judge | Why this lens | Key primitives |
|-------|--------------|----------------|
| **Evaluator Analyst** | Someone needs to deep-dive the metrics — what patterns emerge in pass/fail, where are the near-misses vs systematic gaps? | Cluster failures, flag stochasticity |
| **Constraint Realist** | The execution environment has specific capabilities and limits that affect what approaches work | Diagnose before scoring, flag ceilings, research if stuck |
| **Downstream User** | Does the output actually work for its intended use? Would a consumer of this output be satisfied? | Protect high-scoring elements, score via seed calibration |

**Creative writing (judge-only, no evaluator):**
| Judge | Why this lens | Key primitives |
|-------|--------------|----------------|
| **Craft** | Is the writing working at a technical level — structure, pacing, voice? | Diagnose before scoring, protect high-scoring elements |
| **Reader** | Does this land emotionally for someone reading it cold? | Score via seed calibration |
| **Domain Expert** | Does it get the genre/setting/rules right? | Research if stuck (for genre conventions) |

**Pipeline optimization (workspace, multi-model):**
| Judge | Why this lens | Key primitives |
|-------|--------------|----------------|
| **Metrics** | What do the evaluator numbers actually show? | Cluster failures, flag stochasticity |
| **Architecture** | Is the pipeline structure right, or is it a local optimum? | Flag ceilings, review what's been tried |
| **Operations** | Can this run reliably in production at acceptable cost? | Protect high-scoring elements |

### Custom Judge Panels

Users can define custom judges in the setup brief:

JUDGE_PANEL：

• name：技术准确性 lens：侧重事实正确性和逻辑一致性进行评分
• name：受众适配度 lens：以没有领域背景的初次读者身份进行评分
• name：可操作性 lens：侧重读者能否立即采取行动进行评分

Custom panels override auto-composition entirely. Minimum 2 judges, maximum 5. Default is 3.

## Context Discipline

The board preserves simmer's context discipline exactly:

**Text/creative judges receive:**
- Current candidate, criteria, seed + seed scores, iteration number
- Their lens assignment
- (In deliberation) other judges' scores + reasoning

**Text/creative judges do NOT receive:**
- Intermediate iteration scores, previous ASI, trajectory, previous candidates

**Code/pipeline judges receive:**
- Everything text/creative judges receive
- Plus: evaluator output, previous ASI, iteration history, search space, exploration status

**Cross-judge visibility (deliberation only):**
- Other judges' scores + reasoning (within this iteration only)
- NOT other judges' ASI candidates (withheld until synthesis)

No new cross-iteration information is introduced. The board multiplies perspectives within a single iteration, not across iterations.

## Common Mistakes

**Splitting criteria across judges (one criterion per judge)**
- Problem: Each judge scores only one criterion — loses cross-criterion insight. A coverage-only judge can't see that coverage went up but noise went up more. The board produces 4 isolated scores instead of 12 data points with trade-off awareness.
- Fix: Every judge scores ALL criteria from their lens. The lens frames the perspective (data analysis, strategy, integration), not the criterion. 3 judges × 4 criteria = 12 scored data points with cross-criterion insight.

**Compound ASI overwhelming the generator**
- Problem: Each judge's ASI candidate is multi-faceted. The clerk picks one but it still lists 3-4 changes. Generator tries all of them, artifact bloats, scores regress.
- Fix: The clerk distills, not selects. Read all three ASI candidates and produce ONE focused move. If you're writing "also" or listing changes, you're diluting. The board's analysis is broad — the ASI must be narrow.

**Ignoring mode constraints in ASI**
- Problem: Board correctly identifies "model swap needed" or "add post-processing" but the artifact is single-file — generator can only edit text.
- Fix: Check artifact type. Single-file ASI must be prompt-level changes only. Structural recommendations (model swap, code changes) are only actionable in workspace mode.

**Blending ASIs**
- Problem: Clerk merges three ASIs into one that tries to address everything
- Fix: Pick the single strongest-evidence direction. One ASI, one coherent move.

**Deliberation becomes debate**
- Problem: Judges go back and forth for multiple rounds, losing independence
- Fix: One round only. Agree, challenge, or concede — then stop.

**Judges trying to be comprehensive**
- Problem: Each judge covers all angles instead of going deep on their lens
- Fix: The lens prompt says "go deep on your angle, the other judges cover other angles"

**Withholding challenges to be polite**
- Problem: Judges agree with everything, board produces the same result as a single judge
- Fix: Challenges are the point. "You gave specificity 8 but the evaluator shows 15 near-misses" is exactly the kind of challenge that surfaces blind spots.

**Averaging scores instead of deliberating**
- Problem: Board just takes the mean of three scores without the judges seeing each other's reasoning
- Fix: The deliberation round is required, not optional. Judges must see and respond to each other.

**Using a board when a single judge suffices**
- Problem: Simple text refinement doesn't benefit from 3 judges — just adds latency
- Fix: Board is opt-in. Default is single judge. Use the board for complex artifacts, plateau-breaking, or when the user requests it.