transformers-bio-nlp

生物医学 NLP 的 Transformers

概述

HuggingFace Transformers 提供统一的 API，可加载、运行和微调 500 多种生物医学语言模型。关键的生物医学模型——BioBERT（基于 PubMed 摘要和 PMC 全文训练）、PubMedBERT（从头开始基于 PubMed 训练）、BioGPT（基于 PubMed 训练的生成式模型）和 BioMedLM——在生物医学 NER、关系抽取和问答任务上的表现显著优于通用 BERT。pipeline() 抽象可在一次调用中处理分词、推理和后处理。在特定任务的标注数据（例如用于化学物质／疾病 NER 的 BC5CDR）上进行微调，在单块 GPU 上不到一小时即可完成。datasets 库可直接访问标准生物医学基准数据集。

适用场景

• 从生物医学摘要中提取基因名称、疾病提及、药物名称或化学实体（NER）
• 按主题、临床结果情感或系统综述的 PICO 要素对摘要进行分类
• 使用抽取式 QA（BioASQ 格式）回答生物医学文献中的具体问题
• 使用 BioGPT 或 BioMedLM 从生物医学文本中生成假设或摘要
• 在自定义标注数据集（例如实验室的标注数据）上微调预训练的生物医学模型
• 为生物医学句子生成嵌入，以便跨文献进行语义相似度搜索
• 使用 spaCy + en_core_sci_lg 进行快速的规则增强型 NER；使用 Stanza 进行依存句法分析

前置条件

• Python 包：transformers、torch、datasets、accelerate、sentencepiece
• GPU：强烈建议使用 GPU 进行微调；在 CPU 上对单条文本进行推理也可行
• 数据要求：纯文本生物医学字符串；微调时需要采用 BIO／IOB 格式的标注数据

pip install transformers torch datasets accelerate sentencepiece
# For GPU (CUDA 11.8)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

快速开始

from transformers import pipeline

# Named entity recognition with BioBERT
ner = pipeline("ner", model="allenai/scibert_scivocab_cased",
               aggregation_strategy="simple")

text = "BRCA1 mutations are associated with increased risk of breast cancer and ovarian cancer."
entities = ner(text)
for ent in entities:
    print(f"  {ent['word']:20s} {ent['entity_group']:10s} score={ent['score']:.3f}")

核心 API

模块 1：命名实体识别（NER）

使用预训练的 NER 模型提取生物医学实体。

from transformers import pipeline, AutoTokenizer, AutoModelForTokenClassification

# BioBERT fine-tuned for NER (genes, diseases, chemicals)
# Common choices:
#   "allenai/scibert_scivocab_cased"  — scientific NER
#   "d4data/biomedical-ner-all"       — multi-entity biomedical NER
#   "pruas/BENT-PubMedBERT-NER-Gene"  — gene-specific NER
ner_pipe = pipeline(
    "ner",
    model="d4data/biomedical-ner-all",
    aggregation_strategy="simple",  # merge subword tokens into words
    device=-1  # -1=CPU, 0=GPU
)

abstracts = [
    "Imatinib inhibits the BCR-ABL1 tyrosine kinase and is first-line treatment for CML.",
    "EGFR mutations in non-small cell lung cancer predict response to erlotinib.",
]

for text in abstracts:
    entities = ner_pipe(text)
    print(f"\nText: {text[:60]}...")
    for e in entities:
        print(f"  [{e['entity_group']}] '{e['word']}' (score={e['score']:.2f})")

# Manual tokenization + inference for batch processing
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

model_name = "allenai/scibert_scivocab_cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
model.eval()

text = "Metformin activates AMPK and reduces hepatic glucose production in type 2 diabetes."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)

logits = outputs.logits  # shape: (1, seq_len, n_labels)
predictions = logits.argmax(dim=-1)[0]
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
labels = [model.config.id2label[p.item()] for p in predictions]

for token, label in zip(tokens[1:-1], labels[1:-1]):  # skip [CLS] and [SEP]
    if label != "O":
        print(f"  {token:20s} {label}")

模块 2：文本分类

对生物医学摘要或句子进行分类。

from transformers import pipeline

# Zero-shot classification — no fine-tuning needed
zs_clf = pipeline("zero-shot-classification",
                  model="facebook/bart-large-mnli",
                  device=-1)

abstract = """
This randomized controlled trial evaluated the efficacy of pembrolizumab versus
chemotherapy in patients with advanced non-small-cell lung cancer. Overall survival
was significantly improved in the pembrolizumab arm (HR=0.60, 95% CI 0.41-0.89).
"""

candidate_labels = ["clinical trial", "basic research", "meta-analysis", "review"]
result = zs_clf(abstract, candidate_labels)
print("Zero-shot classification:")
for label, score in zip(result["labels"], result["scores"]):
    print(f"  {label:20s}: {score:.3f}")

# Fine-tuned sentiment/outcome classification
from transformers import pipeline

# Example: classify clinical outcome sentiment
clf = pipeline("text-classification",
               model="pruas/BENT-PubMedBERT-NER-Gene",  # use appropriate task-specific model
               device=-1)

sentences = [
    "Treatment significantly improved overall survival (p<0.001).",
    "No statistically significant difference was observed between groups.",
]
results = clf(sentences)
for sent, result in zip(sentences, results):
    print(f"  [{result['label']} | {result['score']:.2f}] {sent[:50]}...")

模块 3：生物医学问答

从生物医学文本段落中提取答案。

from transformers import pipeline

# Extractive QA: find answer span within context
qa_pipe = pipeline(
    "question-answering",
    model="sultan/BioM-ELECTRA-Large-SQuAD2",  # biomedical QA model
    device=-1
)

context = """
BRCA1 is a tumor suppressor gene located on chromosome 17q21. Pathogenic variants
in BRCA1 confer a lifetime breast cancer risk of 50-72% and ovarian cancer risk
of 44-46%. BRCA1 protein functions in DNA double-strand break repair via
homologous recombination.
"""

questions = [
    "What chromosome is BRCA1 located on?",
    "What is the lifetime breast cancer risk from BRCA1 variants?",
    "What DNA repair pathway does BRCA1 participate in?",
]

for q in questions:
    result = qa_pipe(question=q, context=context)
    print(f"Q: {q}")
    print(f"A: {result['answer']} (score={result['score']:.3f})\n")

模块 4：使用 BioGPT 生成文本

生成生物医学文本、假设和摘要。

from transformers import AutoTokenizer, BioGptForCausalLM
import torch

model_name = "microsoft/biogpt"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = BioGptForCausalLM.from_pretrained(model_name)
model.eval()

prompt = "The role of VEGF in tumor angiogenesis"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        num_beams=5,
        early_stopping=True,
        no_repeat_ngram_size=3,
        pad_token_id=tokenizer.eos_token_id,
    )

generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Generated:\n{generated}")

模块 5：用于语义搜索的句子嵌入

为生物医学文本生成嵌入，以便进行相似度搜索和聚类。

from transformers import AutoTokenizer, AutoModel
import torch
import numpy as np

def mean_pooling(model_output, attention_mask):
    """Mean pooling across token embeddings."""
    token_embeddings = model_output.last_hidden_state
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return (token_embeddings * input_mask_expanded).sum(1) / input_mask_expanded.sum(1)

# PubMedBERT for biomedical sentence embeddings
model_name = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
model.eval()

sentences = [
    "BRCA1 is involved in DNA double-strand break repair.",
    "Homologous recombination requires BRCA1 and BRCA2.",
    "Metformin inhibits hepatic gluconeogenesis via AMPK.",
]

inputs = tokenizer(sentences, padding=True, truncation=True,
                   max_length=512, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

embeddings = mean_pooling(outputs, inputs["attention_mask"])
embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1).numpy()

# Compute cosine similarity
from numpy.linalg import norm
sim_01 = np.dot(embeddings[0], embeddings[1])
sim_02 = np.dot(embeddings[0], embeddings[2])
print(f"Similarity (BRCA1 repair vs. HR): {sim_01:.3f}")
print(f"Similarity (BRCA1 repair vs. Metformin): {sim_02:.3f}")

模块 6：使用自定义数据进行微调

在带标注的 NER 数据集上微调生物医学模型。

from transformers import (AutoTokenizer, AutoModelForTokenClassification,
                           TrainingArguments, Trainer, DataCollatorForTokenClassification)
from datasets import Dataset
import numpy as np

# Example: minimal NER fine-tuning setup
model_name = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract"
label_list = ["O", "B-GENE", "I-GENE", "B-DISEASE", "I-DISEASE"]
id2label = {i: l for i, l in enumerate(label_list)}
label2id = {l: i for i, l in enumerate(label_list)}

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(
    model_name, num_labels=len(label_list), id2label=id2label, label2id=label2id
)

# Training arguments
training_args = TrainingArguments(
    output_dir="./biomed_ner_finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    warmup_steps=100,
    weight_decay=0.01,
    logging_dir="./logs",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

print(f"Model ready for fine-tuning: {model_name}")
print(f"Labels: {label_list}")
# trainer = Trainer(model=model, args=training_args, ...)
# trainer.train()

核心概念

生物医学文本的分词

生物医学文本包含特殊词元（基因符号、药物名称、化学 SMILES、数值），WordPiece 和 BPE 分词器可能会以意外方式拆分这些词元。例如，“BRCA1” → ["BR", "##CA", "##1"]。这种子词拆分不会影响分类任务，但会影响 NER——在 pipeline() 中使用 aggregation_strategy="simple" 或 "first"，将子词级预测合并回词级。

BIO 标注方案

NER 使用 BIO（Begin-Inside-Outside）标注：B-GENE 表示基因名称的首个词元，I-GENE 表示后续词元，O 表示非实体词元。微调时，将非首个子词的标签设为 -100（由损失函数忽略），以便将标签与子词词元对齐。

常见工作流

工作流 1：批量摘要 NER 和实体聚合

from transformers import pipeline
import pandas as pd

ner_pipe = pipeline("ner", model="d4data/biomedical-ner-all",
                    aggregation_strategy="simple", device=-1)

abstracts = [
    "Pembrolizumab combined with chemotherapy significantly improved progression-free survival in HER2-positive breast cancer.",
    "Inhibition of EGFR by gefitinib is effective in patients with activating EGFR mutations in exons 19 and 21.",
    "CRISPR-Cas9 editing of the PCSK9 gene in hepatocytes reduces LDL cholesterol in murine models.",
]

records = []
for i, text in enumerate(abstracts):
    entities = ner_pipe(text)
    for e in entities:
        records.append({
            "abstract_id": i,
            "entity": e["word"],
            "type": e["entity_group"],
            "score": round(e["score"], 3),
        })

df = pd.DataFrame(records)
print(df.groupby("type")["entity"].apply(list).to_string())
df.to_csv("extracted_entities.csv", index=False)
print(f"\nExtracted {len(df)} entity mentions across {len(abstracts)} abstracts")

工作流 2：文献检索的语义相似度排序

from transformers import AutoTokenizer, AutoModel
import torch
import numpy as np

model_name = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
model.eval()

def embed(texts):
    enc = tokenizer(texts, padding=True, truncation=True,
                    max_length=512, return_tensors="pt")
    with torch.no_grad():
        out = model(**enc)
    vecs = out.last_hidden_state[:, 0, :]  # [CLS] token
    return torch.nn.functional.normalize(vecs, dim=1).numpy()

query = "CRISPR base editing for correction of point mutations in genetic disease"
corpus = [
    "Base editing enables precise single-base changes in genomic DNA without double-strand breaks.",
    "CAR-T cell therapy targets CD19 in B-cell acute lymphoblastic leukemia.",
    "Prime editing uses reverse transcriptase to install targeted edits at specific loci.",
    "RNA interference silences gene expression via RISC-mediated mRNA cleavage.",
]

q_emb = embed([query])
c_emb = embed(corpus)
scores = (q_emb @ c_emb.T).flatten()
ranked = sorted(zip(scores, corpus), reverse=True)

print("Top results:")
for score, text in ranked:
    print(f"  [{score:.3f}] {text[:70]}...")

关键参数

参数	模块/函数	默认值	范围 / 选项	作用
model	pipeline()	—	HuggingFace 模型 ID 字符串	要加载的预训练模型；必须与任务匹配
aggregation_strategy	NER pipeline	"none"	"none"、"simple"、"first"、"average"	合并 NER 子词预测；使用 "simple" 获得词级输出
device	pipeline()	-1	-1（CPU）、0（GPU 0）、1（GPU 1）	推理设备
max_length	tokenizer	512	128–2048（取决于模型）	最大 token 长度；会截断更长的输入
max_new_tokens	model.generate()	20	1–1000	文本生成模型要生成的 token 数
num_beams	model.generate()	1	1–10	束搜索宽度；数值越大，质量越好，速度越慢
num_train_epochs	TrainingArguments	3	1–10	微调轮数
per_device_train_batch_size	TrainingArguments	8	4–32	每个 GPU 的批次大小；显存不足时调低
weight_decay	TrainingArguments	0.0	0.01–0.1	微调的 L2 正则化

最佳实践

1. 使用特定领域模型，而非通用 BERT：在 PubMed 上从头训练的 PubMedBERT，在生物医学 NER 上比 BERT-base 高出 5–15%。在使用特定任务数据微调前，始终先使用生物医学预训练模型。

2. 生产环境使用前核实模型许可证：有些模型（BioGPT、BioMedLM）的许可证仅允许用于研究。将模型部署到商业应用前，请查看 HuggingFace 模型卡中的许可证字段。

3. 使用 aggregation_strategy="simple" 获取词级 NER 输出：默认的 "none" 会返回子词 token，导致后处理困难。"simple" 会使用首 token 策略合并子词 token。

4. 在句子边界处截断，而非在句中截断：超过 512 个 token 的生物医学长摘要应在编码前按句子边界拆分。在句中截断会降低截断位置附近实体的 NER 准确率。

常用方案

方案：从 PubMed 摘要中提取药物—疾病配对

from transformers import pipeline
from itertools import product

ner = pipeline("ner", model="d4data/biomedical-ner-all",
               aggregation_strategy="simple", device=-1)

def extract_drug_disease_pairs(text):
    entities = ner(text)
    drugs    = [e["word"] for e in entities if e["entity_group"] in ("DRUG", "CHEMICAL")]
    diseases = [e["word"] for e in entities if e["entity_group"] in ("DISEASE", "CONDITION")]
    return list(product(drugs, diseases))

text = "Imatinib and nilotinib both target BCR-ABL1 in chronic myeloid leukemia and Philadelphia chromosome-positive ALL."
pairs = extract_drug_disease_pairs(text)
print("Drug-Disease pairs:")
for drug, disease in pairs:
    print(f"  {drug} → {disease}")

方案：句子级摘要筛选

from transformers import pipeline

clf = pipeline("zero-shot-classification",
               model="facebook/bart-large-mnli", device=-1)

abstracts = [
    "We present a phase 3 randomized controlled trial of semaglutide in type 2 diabetes.",
    "Structural analysis of the SARS-CoV-2 spike protein RBD domain by cryo-EM.",
    "A retrospective cohort study of 1,200 ICU patients during the COVID-19 pandemic.",
]

label_options = ["randomized controlled trial", "observational study", "structural biology", "computational study"]

for abstract in abstracts:
    result = clf(abstract, label_options)
    print(f"Type: {result['labels'][0]} ({result['scores'][0]:.2f})")
    print(f"  {abstract[:70]}...\n")

故障排除

问题	原因	解决方案
推理期间出现 CUDA out of memory	批次过大，超出 GPU 显存容量	减小批次大小；使用 device=-1 切换到 CPU；使用 model.half() 启用 FP16
NER 返回子词 token（##CA）	未设置 aggregation_strategy	在 pipeline() 中设置 aggregation_strategy="simple"
模型下载超时	模型文件较大（1–10 GB）；网络连接较慢	设置 HF_HUB_OFFLINE=1，并使用 huggingface-cli download 手动下载
NER 漏掉长摘要末尾的实体	输入在 512 个 token 处被截断	将摘要拆分为句子；分别处理每个句子
微调损失为 NaN	学习率过高或梯度爆炸	将 learning_rate 调低至 2e-5；启用梯度裁剪 max_grad_norm=1.0
特定领域的实体识别错误	通用生物医学模型不适用于该子领域	使用领域标注数据进行微调；使用更具体的模型（例如，仅识别基因的 NER 模型）
BioGPT 生成重复文本	no_repeat_ngram_size 过小	将 no_repeat_ngram_size 设为 3 或 4；增大 num_beams

相关技能

• pubmed-database — 获取 PubMed 摘要，作为生物医学 NLP 工作流的输入
• biorxiv-database — 获取预印本，在同行评审前进行 NLP 分析
• scientific-critical-thinking — 在将 NLP 提取的证据用于研究结论前评估其质量

参考资料

• HuggingFace Transformers 文档 — pipeline、tokenizer 和训练 API
• BioBERT 论文：Lee 等人（2020），Bioinformatics — 在 PubMed 和 PMC 上进行预训练
• PubMedBERT 论文：Gu 等人（2021），ACL — 在 PubMed 上从头开始预训练
• BioGPT 论文：Luo 等人（2022），Briefings in Bioinformatics — 生成式生物医学语言模型
• BioCreative 基准 — 标准 NER 和关系抽取数据集