统计学验证器
用于验证统计学主张、核验研究方法并发现分析错误与偏差的结构化框架。
统计学主张验证清单
快速评估主张
CLAIM VERIFICATION PROTOCOL:
1. SOURCE CHECK
- Who made the claim?
- What is their expertise and incentive?
- Where was it published (peer-reviewed, preprint, press release)?
- Is the original data or study accessible?
2. METHODOLOGY CHECK
- What type of study (RCT, observational, survey, meta-analysis)?
- What was the sample size and population?
- What was the measurement method?
- Is the statistical test appropriate for the data type?
3. NUMBER SENSE CHECK
- Does the claim pass a basic plausibility test?
- Are units and denominators clearly stated?
- Absolute vs relative numbers — which is being used?
- Is the base rate provided for context?
4. REPLICATION CHECK
- Have other studies found similar results?
- Are the findings consistent across populations?
- Has anyone attempted and failed to replicate?
5. CONCLUSION CHECK
- Does the conclusion follow from the data?
- Are alternative explanations addressed?
- Is the scope of the claim proportional to the evidence?
主张中的危险信号
| 危险信号 | 含义 | 行动 |
|---|
| 未提供样本量 | 无法评估可靠性 | 要求提供或估算 N |
| 仅报告相对风险 | 可能掩盖较小的绝对效应 | 计算绝对差异 |
| 使用“最高达 X%”的表述 | 挑选了最佳情况 | 询问中位数或均值 |
| 未提供置信区间 | 精确度未知 | 持怀疑态度 |
| 将相关性表述为因果关系 | 可能忽视了混杂因素 | 检查研究设计 |
| 样本由受访者自行选择 | 可能存在选择偏差 | 说明局限性 |
| 复合终点 | 可能掩盖单项结果的微弱表现 | 分解终点 |
| 突出显示亚组分析 | 可能是事后筛选 | 要求预注册 |
常见统计错误
错误检测框架
CATEGORY 1: DESIGN ERRORS
- Sampling bias (convenience, voluntary response, survivorship)
- Confounding variables not controlled
- Insufficient sample size (underpowered study)
- No control group or inappropriate comparator
- Measurement instrument not validated
CATEGORY 2: ANALYSIS ERRORS
- Multiple comparisons without correction (p-hacking)
- Treating ordinal data as interval
- Assuming normality without checking
- Ignoring missing data patterns (MCAR vs MNAR)
- Using parametric tests on non-parametric data
CATEGORY 3: INTERPRETATION ERRORS
- Confusing statistical significance with practical significance
- Interpreting non-significant result as "no effect"
- Ecological fallacy (group-level applied to individuals)
- Simpson's paradox not checked
- Ignoring effect size and confidence intervals
CATEGORY 4: REPORTING ERRORS
- Selective reporting of favorable results
- Omitting negative or null findings
- Misleading axis scales in visualizations
- Presenting percentages without base numbers
- Switching between absolute and relative metrics
错误严重程度评估
| 错误类型 | 严重程度 | 对结论的影响 |
|---|
| P-hacking / HARKing | 严重 | 使研究结果无效 |
| 选择偏差 | 严重 | 样本存在根本缺陷 |
| 未处理混杂因素 | 高 | 仍存在其他解释 |
| 统计检验错误 | 高 | 结果可能是人为造成的 |
| 未校正多重比较 | 高 | 假阳性率升高 |
| 样本量小且未进行功效分析 | 中 | 可能漏掉真实效应 |
| 缺少置信区间 | 中 | 无法判断精确度 |
| 可视化具有误导性 | 中 | 错误呈现效应大小 |
| 轻微舍入误差 | 低 | 影响极小 |
显著性检验框架
检验选择指南
CHOOSING THE RIGHT TEST:
DATA TYPE → COMPARISON → TEST
Continuous + 2 groups + independent → Independent t-test (or Mann-Whitney)
Continuous + 2 groups + paired → Paired t-test (or Wilcoxon signed-rank)
Continuous + 3+ groups + independent → One-way ANOVA (or Kruskal-Wallis)
Continuous + 2+ factors → Two-way ANOVA (or Friedman)
Continuous + continuous → Pearson correlation (or Spearman)
Categorical + 2 groups → Chi-square test (or Fisher's exact)
Categorical + ordered → Cochran-Armitage trend test
Binary outcome + predictors → Logistic regression
Time-to-event + groups → Log-rank test / Cox regression
Count data → Poisson regression
Proportion + large sample → Z-test for proportions
P 值解读指南
P-VALUE CONTEXT:
p-value = P(data this extreme | null hypothesis is true)
COMMON MISINTERPRETATIONS:
p = 0.03 does NOT mean:
- "There is a 3% chance the result is due to chance"
- "There is a 97% probability the hypothesis is true"
- "The effect is large or important"
- "The study will replicate"
p = 0.03 DOES mean:
- If the null hypothesis were true, data this extreme
would occur about 3% of the time by chance alone.
THRESHOLDS (conventional, not absolute):
p < 0.001 — strong evidence against null
p < 0.01 — moderate evidence against null
p < 0.05 — conventional threshold (context-dependent)
p > 0.05 — insufficient evidence to reject null
(NOT evidence of no effect)
ALWAYS COMPLEMENT WITH:
- Effect size (Cohen's d, odds ratio, etc.)
- Confidence interval (range of plausible values)
- Practical significance (is the effect meaningful?)
- Study power (could it have detected a real effect?)
多重比较校正
| 方法 | 适用场景 | 保守程度 |
|---|
| Bonferroni | 比较次数较少，需要严格控制 | 非常保守 |
| Holm-Bonferroni | 比较次数中等，采用逐步法 | 保守程度较低 |
| Benjamini-Hochberg | 比较次数较多（控制 FDR） | 宽松 |
| Tukey's HSD | ANOVA 后进行所有成对比较 | 中等 |
| Dunnett's | 多个处理组与一个对照组比较 | 中等 |
样本量验证
快速参考表
MINIMUM SAMPLE SIZE GUIDELINES:
Survey (population estimate):
±3% margin, 95% CI → n ≈ 1,067
±5% margin, 95% CI → n ≈ 385
±10% margin, 95% CI → n ≈ 97
A/B Test (detecting 5% relative lift):
Baseline 10% conversion → n ≈ 3,200 per group
Baseline 5% conversion → n ≈ 6,400 per group
Baseline 2% conversion → n ≈ 16,000 per group
Clinical trial (medium effect d=0.5):
Two-group comparison, 80% power → n ≈ 64 per group
Two-group comparison, 90% power → n ≈ 86 per group
Correlation (detecting r=0.3):
80% power, alpha=0.05 → n ≈ 85
90% power, alpha=0.05 → n ≈ 113
功效分析清单
| 参数 | 必须指定 | 来源 |
|---|
| Alpha（I 类错误率） | 是 | 惯例（通常为 0.05） |
| 功效（1 - II 类错误） | 是 | 通常为 0.80 或 0.90 |
| 效应量 | 是 | 既往研究或 MCID |
| 方差 / SD | 是 | 试点数据或文献 |
| 样本量 | 计算得出 | 功效分析的输出 |
| 脱落率 | 建议指定 | 根据预期脱落率增加 N |
相关性与因果关系清单
Bradford Hill 因果关系标准
DOES CORRELATION IMPLY CAUSATION? CHECK:
1. STRENGTH Is the association large?
Larger effects harder to explain away.
2. CONSISTENCY Replicated across settings, populations?
Multiple studies, same finding.
3. SPECIFICITY Is X linked specifically to Y (not everything)?
Less useful for multifactorial diseases.
4. TEMPORALITY Does X precede Y in time?
REQUIRED — cause must come before effect.
5. BIOLOGICAL GRADIENT Does more X produce more Y (dose-response)?
Strong support for causation.
6. PLAUSIBILITY Is there a credible mechanism?
Based on current knowledge.
7. COHERENCE Consistent with known biology/theory?
No conflict with established facts.
8. EXPERIMENT Does removing X reduce Y?
Strongest evidence (RCT).
9. ANALOGY Similar exposures cause similar effects?
Weakest criterion, supporting only.
VERDICT:
Criteria 1-3 met + Temporality → Suggestive of causation
Criteria 1-6 met + Experiment → Strong evidence of causation
Only correlation observed → Association only, cannot infer cause
常见的第三变量混杂因素
| 观察到的关联 | 可能的混杂因素 |
|---|
| 冰淇淋销量与溺水事件 | 天气炎热（季节） |
| 鞋码与阅读能力 | 年龄 |
| 就诊次数与死亡率 | 病情严重程度 |
| 有机食品与健康 | 社会经济地位 |
| 屏幕使用时间与抑郁 | 社交孤立、睡眠 |
调查方法审查
调查质量评估
SURVEY METHODOLOGY CHECKLIST:
SAMPLING:
- [ ] Probability sampling method described?
- [ ] Sampling frame defined and appropriate?
- [ ] Response rate reported (acceptable: >60% mail, >80% in-person)?
- [ ] Non-response bias assessed?
QUESTIONNAIRE:
- [ ] Questions validated or adapted from validated instruments?
- [ ] Leading or double-barreled questions absent?
- [ ] Response options balanced and exhaustive?
- [ ] Pilot tested with target population?
ADMINISTRATION:
- [ ] Mode (online, phone, in-person) appropriate?
- [ ] Anonymity/confidentiality assured?
- [ ] Informed consent obtained?
- [ ] Social desirability bias mitigated?
ANALYSIS:
- [ ] Weighting applied for non-response or oversampling?
- [ ] Margin of error and confidence level reported?
- [ ] Subgroup analyses pre-specified (not exploratory)?
数据可视化完整性检查
图表审计清单
| 检查项 | 检查内容 | 不通过条件 |
|---|
| Y 轴从零开始（柱状图） | 截断坐标轴会夸大差异 | 坐标轴从零以上开始，且未明确标注 |
| 比例一致 | 两个坐标轴的刻度增量成比例 | 使用非线性比例，且未作说明 |
| 面积与数据成比例 | 气泡／图标大小与数值相符 | 面积无法准确表示数值大小 |
| 时间轴间距均匀 | 数据点之间的间隔相等 | 间距不均匀，导致趋势被压缩／拉伸 |
| 图表类型恰当 | 数据类型与可视化方式相符 | 饼图包含 20 个以上类别 |
| 提供上下文 | 基准、比较对象、基线 | 只有单个数据点，且没有参考信息 |
| 注明来源 | 数据来源可追溯 | 未注明来源 |
| 谨慎使用双轴 | 两个 Y 轴可能造成虚假相关性 | 任意缩放暗示两者存在关系 |
误导性可视化模式
WATCH FOR THESE TRICKS:
1. TRUNCATED AXIS
Small differences look dramatic when baseline removed.
FIX: Always check if y-axis starts at zero for bar charts.
2. CHERRY-PICKED TIME WINDOW
Start/end dates chosen to show desired trend.
FIX: Ask for longer time series with consistent intervals.
3. 3D EFFECTS
Perspective distortion makes sizes unequal.
FIX: Use flat 2D charts for accurate comparison.
4. DUAL AXIS MANIPULATION
Two y-axes scaled to create apparent correlation.
FIX: Normalize data or use separate panels.
5. CUMULATIVE VS DAILY
Cumulative charts always go up — hides declining rates.
FIX: Show rate of change alongside cumulative.
偏差检测框架
数据分析中的认知偏差
BIAS DETECTION CHECKLIST:
CONFIRMATION BIAS
- Are they only presenting data that supports their hypothesis?
- Were negative results reported?
- Was the analysis plan pre-registered?
ANCHORING BIAS
- Is the first number presented influencing interpretation of later data?
- Are comparisons made to appropriate benchmarks?
SURVIVORSHIP BIAS
- Are only successful cases included (ignoring failures)?
- Is the denominator complete (not just survivors)?
AVAILABILITY BIAS
- Are dramatic or recent events overweighted?
- Is systematic data used rather than anecdotal evidence?
PUBLICATION BIAS
- Is there a funnel plot asymmetry (meta-analyses)?
- Are null results published or only significant ones?
TEXAS SHARPSHOOTER FALLACY
- Were clusters or patterns found after looking at data?
- Was the hypothesis formed before or after seeing results?
偏差严重程度矩阵
| 偏差 | 检测方法 | 缓解措施 |
|---|
| 选择偏差 | 将样本与总体的人口统计特征进行比较 | 概率抽样、加权 |
| 测量偏差 | 检查测量工具的有效性和校准情况 | 使用经过验证的工具、盲法 |
| 报告偏差 | 检查漏斗图是否不对称 | 预注册、开放数据 |
| 回忆偏差 | 与客观记录进行比较 | 前瞻性数据收集 |
| 观察者偏差 | 检查评估人员是否采用盲法 | 双盲设计 |
| 脱落偏差 | 比较完成研究者与退出者 | 意向性治疗分析 |
可复现性清单
研究可复现性评估
REPRODUCIBILITY REQUIREMENTS:
DATA AVAILABILITY:
- [ ] Raw data accessible (repository, supplement, on request)?
- [ ] Data dictionary / codebook provided?
- [ ] Data collection protocol documented?
CODE / ANALYSIS:
- [ ] Analysis code shared (GitHub, OSF, supplement)?
- [ ] Software versions and packages specified?
- [ ] Random seeds set for reproducible computation?
- [ ] Pipeline documented end-to-end?
METHODOLOGY:
- [ ] Study pre-registered (OSF, ClinicalTrials.gov)?
- [ ] Deviations from protocol documented?
- [ ] All outcome measures reported (not just significant ones)?
- [ ] Sensitivity analyses included?
REPORTING:
- [ ] Follows reporting guidelines (CONSORT, STROBE, PRISMA)?
- [ ] Effect sizes and confidence intervals reported?
- [ ] Power analysis or sample size justification provided?
- [ ] Limitations section thorough and honest?
不同研究类型的报告规范
| 研究类型 | 指南 | 关键要素 |
|---|
| 随机试验 | CONSORT | 流程图、ITT 分析、盲法 |
| 观察性研究 | STROBE | 选择标准、混杂因素、缺失数据 |
| 系统综述 | PRISMA | 检索策略、纳入标准、偏倚风险 |
| 诊断准确性研究 | STARD | 指标检验、参考标准、流程图 |
| 定性研究 | COREQ | 研究团队、研究设计、数据分析 |
| 预测模型 | TRIPOD | 模型开发、验证、性能 |
快速验证工作流
FAST VERIFICATION (5 minutes):
1. Read the claim carefully — what exactly is being stated?
2. Check: source, sample size, study type
3. Ask: absolute or relative? What is the base rate?
4. Check: confidence interval or margin of error given?
5. Search: has this been replicated independently?
VERDICT CATEGORIES:
VERIFIED — multiple strong sources, robust methodology
PLAUSIBLE — reasonable evidence, some limitations
UNCERTAIN — mixed evidence, methodology concerns
MISLEADING — technically true but presented deceptively
FALSE — contradicted by strong evidence
UNVERIFIABLE — cannot assess with available information
另请参阅
