hypogenic-hypothesis-generation

HypoGeniC 假设生成

概述

HypoGeniC 使用大模型在表格数据集上自动生成并检验科学假设。给定带标签的数据（例如欺骗检测、AI 内容识别），它会生成可检验的假设，根据验证集表现迭代优化，并运行推理，对新样本进行分类。它支持三种方法：纯数据驱动（HypoGeniC）、整合文献（HypoRefine），以及将两者结合的机制。

适用场景

• 从带标签的观察性数据集中生成可检验的假设，无需预先理论
• 系统地在实证数据上检验多个相互竞争的假设
• 将研究论文中的见解与数据驱动的模式发现相结合
• 加快欺骗检测、内容分析、心理健康指标等领域的假设构思
• 将基于大模型的假设生成方法与少样本基线进行基准测试
• 如需手动构建假设的方法框架，请使用 hypothesis-generation knowhow
• 如需不强调假设可解释性的通用 ML 分类，请使用 scikit-learn-machine-learning

先决条件

• Python 包：hypogenic
• 可选：用于缓存 LLM 响应的 Redis 服务（端口 6832）；用于处理 PDF 文献的 GROBID
• API 密钥：环境变量中配置 OpenAI、Anthropic 或兼容的大模型 API 密钥
• 数据：符合 HypoGeniC 格式的带标签 JSON 数据集（见关键概念）

pip install hypogenic

# Optional: clone example datasets
git clone https://github.com/ChicagoHAI/HypoGeniC-datasets.git ./data
git clone https://github.com/ChicagoHAI/Hypothesis-agent-datasets.git ./data_lit

快速开始

from hypogenic import BaseTask
import re

# Custom label extractor (must match dataset label format)
def extract_label(text: str) -> str:
    match = re.search(r'final answer:\s+(.*)', text, re.IGNORECASE)
    return match.group(1).strip() if match else text.strip()

# 1. Load task from config
task = BaseTask(
    config_path="./data/your_task/config.yaml",
    extract_label=extract_label
)

# 2. Generate hypotheses (data-driven)
task.generate_hypotheses(
    method="hypogenic",
    num_hypotheses=20,
    output_path="./output/hypotheses.json"
)

# 3. Run inference on test set
results = task.inference(
    hypothesis_bank="./output/hypotheses.json",
    test_data="./data/your_task/your_task_test.json"
)
print(f"Accuracy: {results['accuracy']:.3f}")

工作流

步骤 1：准备数据集

创建包含文本特征和标签的 train/val/test JSON 文件。

import json

# Dataset: each key maps to a list of equal length
dataset = {
    "headline_1": [
        "What Up, Comet? You Just Got *PROBED*",
        "Scientists Made a Breakthrough in Quantum Computing"
    ],
    "headline_2": [
        "Scientists Were Holding Their Breath Today. Here's Why.",
        "New Quantum Computer Achieves Milestone"
    ],
    "label": [
        "Headline 2 has more clicks than Headline 1",
        "Headline 1 has more clicks than Headline 2"
    ]
}

# All lists must have equal length; labels must match extract_label output
for split in ["train", "val", "test"]:
    with open(f"my_task_{split}.json", "w") as f:
        json.dump(dataset, f, indent=2)
print(f"Created dataset with {len(dataset['label'])} samples")

步骤 2：创建任务配置

编写一个定义数据集路径和提示词模板的 config.yaml。

# config.yaml structure (write as YAML file)
config = """
task_name: my_task

train_data_path: ./my_task_train.json
val_data_path: ./my_task_val.json
test_data_path: ./my_task_test.json

prompt_templates:
  observations: |
    Feature 1: ${text_features_1}
    Feature 2: ${text_features_2}
    Observation: ${label}

  batched_generation:
    system: "You are a research scientist generating hypotheses."
    user: "Generate ${num_hypotheses} testable hypotheses from these observations."

  inference:
    system: "You are evaluating a hypothesis against data."
    user: "Hypothesis: ${hypothesis}\\nSample: ${sample_text}\\nFinal answer: ${label}"

  is_relevant:
    system: "Check hypothesis relevance."
    user: "Is this hypothesis relevant? ${hypothesis}"
"""

with open("config.yaml", "w") as f:
    f.write(config)
print("Configuration written to config.yaml")

步骤 3：实现标签提取

定义自定义的 extract_label 函数，使其与标签格式匹配。

import re

def extract_label(llm_output: str) -> str:
    """Parse LLM output to extract predicted label.

    Must return labels matching the 'label' field values in the dataset.
    Default: searches for 'final answer: <label>' pattern.
    """
    match = re.search(r'final answer:\s+(.*)', llm_output, re.IGNORECASE)
    if match:
        return match.group(1).strip()
    # Domain-specific fallback
    if "Final prediction:" in llm_output:
        return llm_output.split("Final prediction:")[-1].strip()
    return llm_output.strip()

# Test against expected labels
assert extract_label("Final answer: Headline 1") == "Headline 1"
print("Label extractor validated")

步骤 4：生成假设（HypoGeniC）

运行数据驱动的假设生成，并进行迭代优化。

from hypogenic import BaseTask

task = BaseTask(
    config_path="./config.yaml",
    extract_label=extract_label
)

# Generate hypotheses: initializes from data subset, iteratively refines
task.generate_hypotheses(
    method="hypogenic",       # Data-driven generation
    num_hypotheses=20,        # Target number of hypotheses
    output_path="./output/hypotheses.json"
)
# CLI equivalent:
# hypogenic_generation --config config.yaml --method hypogenic --num_hypotheses 20
print("Hypothesis bank saved to ./output/hypotheses.json")

步骤 5：运行推理

使用测试集检验生成的假设。

results = task.inference(
    hypothesis_bank="./output/hypotheses.json",
    test_data="./my_task_test.json"
)
print(f"Test accuracy: {results['accuracy']:.3f}")
print(f"Predictions: {results['predictions'][:5]}")
# CLI equivalent:
# hypogenic_inference --config config.yaml --hypotheses output/hypotheses.json

步骤 6：整合文献生成（HypoRefine）

将文献见解与数据驱动的假设相结合。

# Requires GROBID setup and preprocessed PDFs
# bash ./modules/setup_grobid.sh  # first time
# bash ./modules/run_grobid.sh    # start GROBID service
# python pdf_preprocess.py --task_name my_task

task.generate_hypotheses(
    method="hyporefine",
    num_hypotheses=15,
    literature_path="./literature/my_task/",
    output_path="./output/"
)
# Generates 3 hypothesis banks:
# - HypoRefine (integrated literature+data)
# - Literature-only hypotheses
# - Literature union HypoRefine
print("HypoRefine generation complete: 3 hypothesis banks created")

步骤 7：多假设推理

同时检验多个假设，以进行集成分类。

from examples.multi_hyp_inference import run_multi_hypothesis_inference

results = run_multi_hypothesis_inference(
    config_path="./config.yaml",
    hypothesis_bank="./output/hypotheses.json",
    test_data="./my_task_test.json"
)
print(f"Multi-hypothesis accuracy: {results['accuracy']:.3f}")

关键参数

参数	默认值	范围 / 选项	作用
method	"hypogenic"	"hypogenic"、"hyporefine"、"union"	生成策略
num_hypotheses	20	5-50	要生成的假设数量
batch_size	5	3-10	每批生成的样本数
max_iterations	10	1-50	优化迭代次数
temperature	0.7	0.0-1.0	LLM 采样温度
confidence_threshold	0.7	0.5-0.95	推理置信度阈值
num_papers	10	5-30	HypoRefine 文献提取的论文数量
inference_method	"voting"	"voting"、"weighted"、"ensemble"	多个假设组合预测的方式

关键概念

数据集格式

HypoGeniC 需要包含并行列表的 JSON 文件：

{
  "text_features_1": ["sample_1_feat1", "sample_2_feat1"],
  "text_features_2": ["sample_1_feat2", "sample_2_feat2"],
  "label": ["class_A", "class_B"]
}

• 所有列表的长度必须相同
• 特征键可自定义（review_text、post_content 等）
• 标签必须与 extract_label() 的输出格式完全一致
• 需要三个数据拆分：<TASK>_train.json、<TASK>_val.json、<TASK>_test.json

三种生成方法

方法	输入	流程	最适合
HypoGeniC	仅数据	从子集初始化，在验证集上迭代优化	探索性研究、缺少相关文献的新数据集
HypoRefine	数据 + PDF	提取文献见解，与数据模式合并，并优化两者	拓展或验证现有理论
Union	文献 + HypoGeniC	机制性组合、去重	最大限度拓展假设多样性和覆盖范围

配置模板

config.yaml 的最简必需结构：

task_name: my_task
train_data_path: ./my_task_train.json
val_data_path: ./my_task_val.json
test_data_path: ./my_task_test.json

model:
  name: "gpt-4"                    # or claude-3, gpt-3.5-turbo
  api_key_env: "OPENAI_API_KEY"
  temperature: 0.7

generation:
  method: "hypogenic"
  num_hypotheses: 20
  batch_size: 5
  max_iterations: 10

cache:
  enabled: true                    # Redis on localhost:6832
  host: "localhost"
  port: 6832

prompt_templates:
  observations: |
    Feature 1: ${text_features_1}
    Observation: ${label}
  batched_generation:
    system: "Generate testable hypotheses."
    user: "Generate ${num_hypotheses} hypotheses."
  inference:
    system: "Evaluate hypothesis against sample."
    user: "Hypothesis: ${hypothesis}\nSample: ${sample_text}"
  is_relevant:
    system: "Check relevance."
    user: "Is ${hypothesis} relevant?"

常见方案

方案：从头创建自定义任务

适用场景：使用特定领域数据创建新的分类任务。

import json
from hypogenic import BaseTask

# 1. Prepare data splits
for split_name, data in [("train", train_data), ("val", val_data), ("test", test_data)]:
    with open(f"my_task_{split_name}.json", "w") as f:
        json.dump(data, f)

# 2. Define domain-specific label extractor
def my_extractor(text):
    if "positive" in text.lower():
        return "positive"
    elif "negative" in text.lower():
        return "negative"
    return text.strip()

# 3. Create task and run full pipeline
task = BaseTask(config_path="./my_task/config.yaml", extract_label=my_extractor)
task.generate_hypotheses(method="hypogenic", num_hypotheses=15, output_path="./output/")
results = task.inference(hypothesis_bank="./output/hypotheses.json")
print(f"Custom task accuracy: {results['accuracy']:.3f}")

方案：文献处理设置

适用场景：在使用 HypoRefine 之前，设置 GROBID，将 PDF 转换为结构化文本。

# 1. Setup GROBID (first time only)
bash ./modules/setup_grobid.sh

# 2. Place PDFs in literature directory
mkdir -p literature/my_task/raw/
cp papers/*.pdf literature/my_task/raw/

# 3. Start GROBID and process
bash ./modules/run_grobid.sh
cd examples && python pdf_preprocess.py --task_name my_task
# Output: structured text files in literature/my_task/processed/

方案：使用并集方法实现最大覆盖率

适用场景：结合文献和数据驱动的假设，实现全面覆盖。

# Generate literature hypotheses first (via HypoRefine)
task.generate_hypotheses(
    method="hyporefine",
    num_hypotheses=15,
    literature_path="./literature/my_task/",
    output_path="./output/"
)

# Union combines and deduplicates both banks
# CLI alternative:
# hypogenic_generation --config config.yaml --method union \
#   --literature_hypotheses output/lit_hypotheses.json

# Compare all three approaches
for bank in ["hypogenic", "hyporefine", "union"]:
    r = task.inference(hypothesis_bank=f"./output/{bank}_hypotheses.json")
    print(f"{bank}: accuracy={r['accuracy']:.3f}")

预期输出

• hypotheses.json —— 假设库，包含经过排序且可检验的假设（通常为 10-20 个）
• 推理结果，包含每个样本的预测和整体准确率
• 对于 HypoRefine：三个假设库（仅文献、整合、并集）
• 报告的改进：比少样本基线高约 9%，比仅使用文献的方法高约 16%
• 80-84% 的假设对多样性（非冗余见解）

故障排除

问题	原因	解决方案
ModuleNotFoundError: hypogenic	未安装软件包	pip install hypogenic
假设过于笼统或不可检验	提示词模板过于含糊	在 batched_generation 提示词中添加特定领域的上下文
推理准确率低	训练样例过少或标签提取有误	增加训练数据；确认 extract_label 与数据集标签匹配
GROBID PDF 处理失败	GROBID 服务未运行	bash ./modules/run_grobid.sh；确保 PDF 是有效论文
标签提取不匹配	extract_label 的输出与数据集标签不同	打印两种格式并使其一致；使用 assert extract_label(sample) == expected 进行测试
Redis 连接错误	Redis 未运行或端口错误	在端口 6832 上启动 Redis，或设置 cache.enabled: false
API 速率限制错误	并发 LLM 调用过多	减小 batch_size；启用 Redis 缓存以避免重复调用
假设库为空	配置缺少必需的提示词模板	包含全部四个模板：observations、batched_generation、inference、is_relevant

捆绑资源

此条目内容完整。原始 references/config_template.yaml（151 行）已整合到“关键概念”的“配置模板”小节中，并保留了必要的 YAML 结构、模型/缓存/生成参数以及提示词模板模式。模板中省略的内容：评估指标部分、日志配置、特定任务的特征/标签元数据说明——这些都是标准 YAML 模式，用户可按需添加。

相关技能

• hypothesis-generation —— 手动构建假设框架和科学方法的知识
• scikit-learn-machine-learning —— 无需假设可解释性时用于分类的经典机器学习
• pubmed-database —— 搜索文献，为 HypoRefine 查找输入论文

参考资料

• Zhou, Y., Liu, H., Srivastava, T., Mei, H., & Tan, C. (2024). “使用大模型生成假设。” EMNLP Workshop NLP for Science. https://aclanthology.org/2024.nlp4science-1.10/
• Liu, H., Zhou, Y., Li, M., Yuan, C., & Tan, C. (2024). “文献与数据相遇：一种协同式假设生成方法。” arXiv:2410.17309. https://arxiv.org/abs/2410.17309
• Liu, H., Huang, S., Hu, J., Zhou, Y., & Tan, C. (2025). “HypoBench：迈向系统化、基于原则的假设生成基准测试。” arXiv:2504.11524. https://arxiv.org/abs/2504.11524
• GitHub 仓库：https://github.com/ChicagoHAI/hypothesis-generation
• PyPI 软件包：https://pypi.org/project/hypogenic/