benchmark-runner

基准测试运行器

统一性能比较方法：指标选择、测试用例设计、环境记录、结果格式化和权衡分析。生成可复现的基准测试报告，支持明智决策——不只是“A 比 B 快”，还要说明“A 在小输入下更快，而 B 的扩展性更好”。

参考文件

文件	内容	加载时机
references/metric-selection.md	指标目录（延迟百分位数、吞吐量、内存、准确率），各类任务的选择标准	始终
references/test-case-design.md	代表性输入选择、规模变化、边界情况覆盖、预热策略	始终
references/environment-capture.md	硬件／软件环境记录、可复现性要求、方差控制	始终
references/statistical-rigor.md	样本量、方差测量、显著性检验、异常值处理	结果需要统计验证

前置条件

• 明确要比较的候选对象（至少 2 个）
• 能够运行或观察候选对象（代码、API 或现有结果）
• 定义具有代表性的工作负载

工作流

阶段 1：定义范围

1. 要比较哪些候选对象？ — 精确说出每个候选对象的名称，包括版本。 “Python dict 对比 Redis”太笼统。“Python 3.12 dict（进程内）对比 Redis 7.2 （本地主机，TCP）”才可测试。
2. 要验证哪些主张？ — “A 更快”→在哪方面更快？针对多大输入？ 在什么负载下？基准测试设计应围绕具体主张展开。
3. 决策背景是什么？ — 为什么这项比较很重要？这将决定哪些指标最重要。

阶段 2：选择指标

选择符合决策背景的指标：

指标类别	具体指标	重要场景
延迟	P50、P95、P99、平均值、标准差	面向用户的操作、API 调用
吞吐量	ops/sec、tokens/sec、MB/sec	批处理、流式处理
内存	峰值 RSS、平均 RSS、分配速率	资源受限环境
准确率	F1、BLEU、精确匹配、精确率／召回率	ML 模型、存在质量权衡的算法
成本	$/1K 次操作、$/小时、$/GB	云服务、API 比较
启动	首次操作耗时、冷启动	Serverless、CLI 工具

选择 2-4 项指标。超过 4 项会让比较表难以阅读。

阶段 3：设计测试用例

创建一组输入矩阵，以揭示性能特征：

1. 规模变化 — 小、中、大输入。性能常常会随规模呈非线性变化。
2. 代表性数据 — 使用真实的输入，而非合成的最佳情况数据。
3. 边界情况 — 空输入、最大尺寸、对抗性输入。
4. 预热 — 测量时排除 JIT 编译、缓存预热和连接建立的耗时。先运行 N 次预热迭代，再记录数据。

阶段 4：说明环境

记录重现结果所需的一切信息：

1. 硬件 — CPU 型号、核心数、RAM 容量、GPU 型号（如适用）
2. 软件 — OS 版本、语言运行时版本、依赖项版本
3. 配置 — 线程数、批次大小、连接池大小、缓存设置
4. 隔离 — 还有哪些程序正在运行？后台进程会影响结果。

阶段 5：整理结果

制作对比表，明确标出各项指标的优胜者，然后分析权衡。

输出格式

# Benchmark: {Descriptive Title}

**Date:** {YYYY-MM-DD}
**Hardware:** {CPU}, {RAM}, {GPU if applicable}
**Software:** {runtime versions}
**Configuration:** {key settings that affect results}

## Candidates

| # | Candidate | Version | Configuration |
|---|-----------|---------|---------------|
| A | {name} | {version} | {relevant config} |
| B | {name} | {version} | {relevant config} |

## Test Cases

| # | Name | Input Size | Description | Warmup | Iterations |
|---|------|------------|-------------|--------|------------|
| 1 | Small | {size} | {what it represents} | {N} | {N} |
| 2 | Medium | {size} | {what it represents} | {N} | {N} |
| 3 | Large | {size} | {what it represents} | {N} | {N} |

## Results

### Latency (ms, lower is better)

| Test Case | A (P50 / P95 / P99) | B (P50 / P95 / P99) | Winner |
|-----------|---------------------|---------------------|--------|
| Small | {values} | {values} | {A or B} |
| Medium | {values} | {values} | {A or B} |
| Large | {values} | {values} | {A or B} |

### Memory (MB, lower is better)

| Test Case | A (Peak) | B (Peak) | Winner |
|-----------|----------|----------|--------|
| Small | {value} | {value} | {A or B} |
| Medium | {value} | {value} | {A or B} |
| Large | {value} | {value} | {A or B} |

## Analysis

### Overall Winner
**{Candidate}** wins on {N} of {M} metrics across all test cases.

### Tradeoff Summary
- **Choose A when:** {conditions where A is the better choice}
- **Choose B when:** {conditions where B is the better choice}

### Caveats
- {Limitation of this benchmark}
- {Condition under which results may differ}

## Reproduction

```bash
# Environment setup
{commands to recreate the environment}

# Run benchmark
{commands to execute the benchmark}

## Configuring Scope

| Mode | Candidates | Depth | When to Use |
|------|-----------|-------|-------------|
| `quick` | 2 candidates, 1-2 metrics | Single test case, no statistics | Rough comparison, sanity check |
| `standard` | 2-3 candidates, 2-4 metrics | 3 test cases, mean + std dev | Default for most comparisons |
| `rigorous` | Any count, full metric suite | Multiple test cases, percentiles, significance tests | Publication, critical decisions |

## Calibration Rules

1. **Measure, don't guess.** Intuition about performance is unreliable. "Obviously
   faster" is not a benchmark result.
2. **Apples to apples.** Candidates must be compared under identical conditions.
   Different hardware, configuration, or input data invalidates the comparison.
3. **Report variance, not just means.** A mean of 50ms with std dev of 100ms is not
   the same as a mean of 50ms with std dev of 2ms. Always report spread.
4. **Warm up before measuring.** First-run performance includes JIT, cache warming,
   and connection setup. Exclude warmup iterations from results.
5. **Representative inputs only.** Benchmarking with synthetic best-case input is
   misleading. Use data that resembles production workloads.
6. **State the winner per metric, not overall.** "A is better" is lazy. "A has lower
   latency; B uses less memory" is useful.

## Error Handling

| Problem | Resolution |
|---------|------------|
| Cannot run candidates locally | Design the benchmark specification. Document what to measure and how. The user executes separately. |
| Results are noisy (high variance) | Increase iteration count. Check for background processes. Use dedicated hardware or containers for isolation. |
| Candidates serve different purposes | Acknowledge that the comparison is partial. Benchmark only the overlapping functionality. |
| No baseline exists | Establish one candidate as the baseline. Report relative performance (e.g., "B is 1.3x faster than A"). |
| Hardware context unavailable | Document what is known. Note that results may not be reproducible without full context. |

## When NOT to Benchmark

Push back if:
- The comparison is not performance-related (feature comparison → use a decision matrix or ADR instead)
- The candidates are fundamentally different tools (comparing a database to a message queue)
- The user wants to benchmark trivial operations (comparing two string concatenation methods in Python)
- Results from others already exist and conditions match — link to existing benchmarks instead