Building Paper Screening Rubrics

构建论文筛选标准

概述

核心原则： 通过头脑风暴 → 测试 → 优化 → 自动化 → 审查 → 迭代，协作制定筛选标准。

优秀的筛选标准源于提前了解边缘情况，并在批量筛选前使用真实论文进行测试。

适用场景

在以下情况下使用此技能：

• 开始一项需要筛选 50 篇以上论文的新文献检索
• 当前筛选标准会错误分类论文（误报/漏报）
• 需要在自动筛选前定义“相关性”标准
• 希望更新标准并重新筛选缓存的论文
• 为 evaluating-paper-relevance 构建辅助脚本

不适用场景：

• 小规模检索（少于 20 篇论文）——手动筛选即可
• 筛选标准已经效果良好——无需重建
• 一次性的探索性检索

两阶段流程

阶段 1：协作设计筛选标准

步骤 1：头脑风暴相关性标准

提出与领域无关的问题，了解哪些因素决定论文是否相关：

核心概念：

• “你的研究问题涉及哪些关键术语/概念？”
  • 示例：特定基因、蛋白质、化合物、疾病、方法、生物、理论
• “是否有同义词或其他名称？”
• “有没有应当排除论文的术语（误报）？”

数据类型与成果：

• “什么类型的信息会让论文具有价值？”
  • 定量测量值（IC50、表达水平、种群规模等）
  • 协议或方法
  • 带有登录号的数据集（GEO、SRA、PDB 等）
  • 代码或软件
  • 化学结构
  • 序列或基因组
  • 理论模型
• “你需要论文中包含实际数据，还是只需确认此类数据存在？”

论文类型：

• “哪些类型的论文相关？”
  • 仅限原创研究？
  • 综述或荟萃分析？
  • 方法学论文？
  • 临床试验？
  • 是否接受预印本？

关联与背景：

• “关于相关/类似概念的论文是否相关？”
  • 示例：“如果研究蛋白质 X，关于同源蛋白/旁系同源蛋白的论文是否相关？”
  • 示例：“如果研究化合物 A，关于类似物/衍生物的论文是否相关？”
  • 示例：“如果研究疾病 X，关于相关疾病的论文是否相关？”
• “论文必须以你的主题为主要研究内容，还是只需提及该主题？”
• “即使没有活性数据，合成/方法类论文是否相关？”

边缘情况：

• “你能想到看起来相关、但实际上并不相关的论文吗？”
• “有没有看起来可能不相关、但实际上相关的论文？”

将回答记录在 screening-criteria.json 中

步骤 2：构建初始筛选标准

根据头脑风暴结果，提出评分逻辑：

Scoring (0-10):

Keywords Match (0-3 pts):
  - Core term 1: +1 pt
  - Core term 2 OR synonym: +1 pt
  - Related term: +1 pt

Data Type Match (0-4 pts):
  - Measurement type (IC50, Ki, EC50, etc.): +2 pts
  - Dataset/code available: +1 pt
  - Methods described: +1 pt

Specificity (0-3 pts):
  - Primary research: +3 pts
  - Methods paper: +2 pts
  - Review: +1 pt

Special Rules:
  - If mentions exclusion term: score = 0

Threshold: ≥7 = relevant, 5-6 = possibly relevant, <5 = not relevant

向用户展示并询问：“这个逻辑符合你的预期吗？”

将初始筛选标准保存到 screening-criteria.json：

{
  "version": "1.0.0",
  "created": "2025-10-11T15:30:00Z",
  "keywords": {
    "core_terms": ["term1", "term2"],
    "synonyms": {"term1": ["alt1", "alt2"]},
    "related_terms": ["related1", "related2"],
    "exclusion_terms": ["exclude1", "exclude2"]
  },
  "data_types": {
    "measurements": ["IC50", "Ki", "MIC"],
    "datasets": ["GEO:", "SRA:", "PDB:"],
    "methods": ["protocol", "synthesis", "assay"]
  },
  "scoring": {
    "keywords_max": 3,
    "data_type_max": 4,
    "specificity_max": 3,
    "relevance_threshold": 7
  },
  "special_rules": [
    {
      "name": "scaffold_analogs",
      "condition": "mentions target scaffold AND (analog OR derivative)",
      "action": "add 3 points"
    }
  ]
}

阶段 2：测试驱动的优化

步骤 1：创建测试集

快速执行一次 PubMed 检索，获取候选论文：

# Search for 20 papers using initial keywords
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=YOUR_QUERY&retmax=20&retmode=json"

获取前 10～15 篇论文的摘要：

curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=PMID1,PMID2,...&retmode=xml&rettype=abstract"

逐篇向用户展示摘要：

Paper 1/10:
Title: [Title]
PMID: [12345678]
DOI: [10.1234/example]

Abstract:
[Full abstract text]

Is this paper RELEVANT to your research question? (y/n/maybe)

将用户的判断记录在 test-set.json 中：

{
  "test_papers": [
    {
      "pmid": "12345678",
      "doi": "10.1234/example",
      "title": "Paper title",
      "abstract": "Full abstract text...",
      "user_judgment": "relevant",
      "timestamp": "2025-10-11T15:45:00Z"
    }
  ]
}

继续操作，直到获得 5～10 篇判断明确的论文

步骤 2：使用筛选标准为测试论文评分

对每篇测试论文应用筛选标准：

for paper in test_papers:
    score = calculate_score(paper['abstract'], rubric)
    predicted_status = "relevant" if score >= 7 else "not_relevant"
    paper['predicted_score'] = score
    paper['predicted_status'] = predicted_status

计算准确率：

correct = sum(1 for p in test_papers
              if p['predicted_status'] == p['user_judgment'])
accuracy = correct / len(test_papers)

步骤 3：向用户展示结果

展示分类报告：

RUBRIC TEST RESULTS (5 papers):

✓ PMID 12345678: Score 9 → relevant (user: relevant) ✓
✗ PMID 23456789: Score 4 → not_relevant (user: relevant) ← FALSE NEGATIVE
✓ PMID 34567890: Score 8 → relevant (user: relevant) ✓
✓ PMID 45678901: Score 3 → not_relevant (user: not_relevant) ✓
✗ PMID 56789012: Score 7 → relevant (user: not_relevant) ← FALSE POSITIVE

Accuracy: 60% (3/5 correct)
Target: ≥80%

--- FALSE NEGATIVE: PMID 23456789 ---
Title: "Novel analogs of compound X with improved potency"
Score breakdown:
  - Keywords: 1 pt (matched "compound X")
  - Data type: 2 pts (mentioned IC50 values)
  - Specificity: 1 pt (primary research)
  - Total: 4 pts → not_relevant

Why missed: Paper discusses "analogs" but didn't trigger scaffold_analogs rule
Abstract excerpt: "We synthesized 12 analogs of compound X..."

--- FALSE POSITIVE: PMID 56789012 ---
Title: "Review of kinase inhibitors"
Score breakdown:
  - Keywords: 2 pts
  - Data type: 3 pts
  - Specificity: 2 pts (review, not primary)
  - Total: 7 pts → relevant

Why wrong: Review paper, user wants primary research only

步骤 4：迭代优化

询问用户需要调整哪些内容：

Current accuracy: 60% (below 80% threshold)

Suggestions to improve rubric:
1. Strengthen scaffold_analogs rule - should "synthesized N analogs" always trigger?
2. Lower points for review papers (currently 2 pts, maybe 0 pts?)
3. Add more synonym terms for core concepts?

What would you like to adjust?

根据反馈更新 screening-criteria.json

更新示例：

{
  "special_rules": [
    {
      "name": "scaffold_analogs",
      "condition": "mentions target scaffold AND (analog OR derivative OR synthesized)",
      "action": "add 3 points"
    }
  ],
  "paper_types": {
    "primary_research": 3,
    "methods": 2,
    "review": 0  // Changed from 1
  }
}

步骤 5：重复测试，直到满意

使用更新后的筛选标准重新为测试论文评分

展示新结果：

UPDATED RUBRIC TEST RESULTS (5 papers):

✓ PMID 12345678: Score 9 → relevant (user: relevant) ✓
✓ PMID 23456789: Score 7 → relevant (user: relevant) ✓ (FIXED!)
✓ PMID 34567890: Score 8 → relevant (user: relevant) ✓
✓ PMID 45678901: Score 3 → not_relevant (user: not_relevant) ✓
✓ PMID 56789012: Score 5 → not_relevant (user: not_relevant) ✓ (FIXED!)

Accuracy: 100% (5/5 correct) ✓
Target: ≥80% ✓

Rubric is ready for bulk screening!

如果准确率 ≥80%： 开始批量筛选
如果准确率 <80%： 继续迭代

阶段 3：批量筛选

筛选标准通过测试集验证后：

1. 在完整的 PubMed 检索结果上运行
2. 将所有摘要保存到 abstracts-cache.json：

{
  "10.1234/example": {
    "pmid": "12345678",
    "title": "Paper title",
    "abstract": "Full abstract text...",
    "fetched": "2025-10-11T16:00:00Z"
  }
}

3. 为所有论文评分，并保存到 papers-reviewed.json：

{
  "10.1234/example": {
    "pmid": "12345678",
    "status": "relevant",
    "score": 9,
    "source": "pubmed_search",
    "timestamp": "2025-10-11T16:00:00Z",
    "rubric_version": "1.0.0"
  }
}

4. 生成摘要报告：

Screened 127 papers using validated rubric:
- Highly relevant (≥8): 12 papers
- Relevant (7): 18 papers
- Possibly relevant (5-6): 23 papers
- Not relevant (<5): 74 papers

All abstracts cached for re-screening.
Results saved to papers-reviewed.json.

Review offline and provide feedback if any misclassifications found.

阶段 4：离线审查与重新筛选

用户离线审查论文并发现问题：

User: "I reviewed the results. Three papers were misclassified:
- PMID 23456789 scored 4 but is actually relevant (discusses scaffold analogs)
- PMID 34567890 scored 8 but not relevant (wrong target)
- PMID 45678901 scored 6 but is highly relevant (has key dataset)

Can we update the rubric?"

根据反馈更新评分标准：

1. 分析误分类的原因
2. 提出评分标准调整建议
3. 使用新评分标准重新评估所有缓存的论文
4. 显示变更差异

重新筛选工作流：

# Load all abstracts from abstracts-cache.json
# Apply updated rubric to each
# Generate change report

RUBRIC UPDATE: v1.0.0 → v1.1.0

Changes:
- Added "derivative" to scaffold_analogs rule
- Increased dataset bonus from +1 to +2 pts

Re-screening 127 cached papers...

Status changes:
  not_relevant → relevant: 3 papers
    - PMID 23456789 (score 4→7)
    - PMID 45678901 (score 6→8)
  relevant → not_relevant: 1 paper
    - PMID 34567890 (score 8→6)

Updated papers-reviewed.json with new scores.
New summary:
- Highly relevant: 13 papers (+1)
- Relevant: 19 papers (+1)

文件结构

research-sessions/YYYY-MM-DD-topic/
├── screening-criteria.json      # Rubric definition (weights, rules, version)
├── test-set.json               # Ground truth papers used for validation
├── abstracts-cache.json        # Full abstracts for all screened papers
├── papers-reviewed.json        # Simple tracking: DOI, score, status
└── rubric-changelog.md         # History of rubric changes and why

与其他技能集成

在使用 evaluating-paper-relevance 之前：

• 先使用此技能构建并验证评分标准
• 创建 screening-criteria.json 和 abstracts-cache.json
• 然后使用经验证的评分标准运行 evaluating-paper-relevance

创建辅助脚本时：

• 使用 screening-criteria.json 配置评分逻辑
• 参考 abstracts-cache.json，避免重新获取数据
• 无需重写脚本即可轻松更新评分标准

使用 answering-research-questions 时：

• 在初始化阶段构建评分标准（解析查询阶段之后）
• 批量筛选前，在测试集上进行验证
• 将评分标准与研究会话一同保存，以确保可复现性

评分标准设计模式

模式 1：加法评分（默认）

score = 0
score += count_keyword_matches(abstract, keywords)  # 0-3 pts
score += count_data_type_matches(abstract, data_types)  # 0-4 pts
score += specificity_score(paper_type)  # 0-3 pts

# Apply special rules
if matches_special_rule(abstract, rule):
    score += rule['bonus_points']

return score

模式 2：特定领域规则

药物化学：

{
  "special_rules": [
    {
      "name": "scaffold_analogs",
      "keywords": ["target_scaffold", "analog|derivative|series"],
      "bonus": 3
    },
    {
      "name": "sar_data",
      "keywords": ["IC50|Ki|MIC", "structure-activity|SAR"],
      "bonus": 2
    }
  ]
}

基因组学：

{
  "special_rules": [
    {
      "name": "public_data",
      "keywords": ["GEO:|SRA:|ENA:", "accession"],
      "bonus": 3
    },
    {
      "name": "differential_expression",
      "keywords": ["DEG|differentially expressed", "RNA-seq|microarray"],
      "bonus": 2
    }
  ]
}

计算方法：

{
  "special_rules": [
    {
      "name": "code_available",
      "keywords": ["github|gitlab|bitbucket", "code available|software"],
      "bonus": 3
    },
    {
      "name": "benchmark",
      "keywords": ["benchmark|comparison", "performance|accuracy"],
      "bonus": 2
    }
  ]
}

常见错误

跳过测试驱动的验证： 未测试评分标准就进行批量筛选 → 误分类很多，浪费时间 不缓存摘要： 重新筛选时从 PubMed 重新获取数据 → 速度慢，触及速率限制 不进行基准真值测试： 无法衡量评分标准的准确性 → 不知道它是否有效 测试论文太少： 仅用 2-3 篇论文进行测试 → 评分标准过拟合，无法泛化 评分标准过于复杂： 带有 10 条以上规则的布尔逻辑 → 难以调试、更新和解释 不记录变更： 更新评分标准却不记录原因 → 无法复现，也无法从错误中学习 阈值设得太高： 要求达到 95% 的准确率 → 不断迭代，收益递减 不在会话中保存评分标准： 会话结束后评分标准丢失 → 无法复现筛选过程

成功标准

满足以下条件时，评分标准即可投入使用：

• 在测试集（5-10 篇基准真值论文）上的准确率 ≥80%
• 用户确信评分标准涵盖了边缘情况
• 已记录特定领域模式的特殊规则
• 已创建所有文件（screening-criteria.json、test-set.json、abstracts-cache.json）
• 已准备好运行批量筛选

快速参考

任务	文件	内容
定义评分标准	screening-criteria.json	关键词、权重、特殊规则
测试评分标准	test-set.json	包含用户判断结果的基准真值论文
缓存摘要	abstracts-cache.json	用于重新筛选的完整摘要文本
跟踪结果	papers-reviewed.json	DOI、得分、状态、评分标准版本
记录变更	rubric-changelog.md	更新历史及理由

会话示例

Agent: "I'll help you build a screening rubric. Let's start with some questions about what makes papers relevant..."

[Brainstorming phase - ask 5 question categories]

User: "I'm looking for papers on tuberculosis drug resistance with MIC data for bedaquiline and clofazimine. Papers about analogs are also relevant."

Agent: "Got it. Here's the initial rubric:
- Keywords: tuberculosis, drug resistance, bedaquiline, clofazimine (0-3 pts)
- Data: MIC values, IC50, in vitro activity (0-4 pts)
- Specificity: Primary research (0-3 pts)
- Special rule: If mentions target drugs + 'analog/derivative' → +3 bonus pts
- Threshold: ≥7 = relevant

Does this look right?"

User: "Yes, let's test it."

Agent: "I'll search PubMed for 20 papers and show you abstracts to classify..."

[Shows 10 abstracts, user classifies them]

Agent: "Testing rubric on your classifications...
Accuracy: 70% (7/10 correct)
- 2 false negatives (missed relevant papers)
- 1 false positive (marked review as relevant)

False negative example: Paper discusses 'derivatives of bedaquiline' but scored only 5 pts because didn't explicitly say 'analog'.

Suggestion: Add 'derivative' and 'compound' to special rule trigger terms?"

User: "Yes, and lower review paper points."

Agent: "Updated rubric. Re-testing...
Accuracy: 90% (9/10 correct) ✓

Ready to run bulk screening on 127 papers?"

User: "Yes, go ahead."

Agent: [Screens 127 papers, saves results]

"Done! Found 18 relevant papers. All abstracts cached for re-screening.
Review offline and let me know if any misclassifications."

[Later...]

User: "Found 2 misclassifications. Can we update the rubric?"

Agent: "Sure! What did I miss?"

User: "PMID 12345 discusses pretomanid combination therapy with bedaquiline - should be relevant."

Agent: "Got it. Adding 'combination therapy' as related term with +2 bonus pts.
Re-screening all 127 cached papers...
Status changes: 3 papers now relevant (including PMID 12345).
Updated papers-reviewed.json."

后续步骤

构建评分标准后：

• 在 evaluating-paper-relevance 中用于批量筛选
• 使用 screening-criteria.json 配置辅助脚本
• 发现边缘情况时更新评分标准
• 标准变更后重新筛选缓存的论文
• 在研究会话的 README 中记录评分标准，以确保可复现性