llm-fine-tuning-guide

大模型微调指南

掌握大模型微调技巧，创建针对特定用例、领域和性能要求优化的专用模型。

概述

微调通过使用精心整理的数据集进行训练，将预训练大模型适配到特定任务、领域或风格。这能提高准确性、减少幻觉并优化成本。

何时进行微调

• 领域专业化：法律文档、医疗记录、财务报告
• 特定任务表现：在特定任务上的表现优于基础模型
• 成本优化：用较小的微调模型替代昂贵的大模型
• 风格适配：匹配特定的写作风格或语气
• 合规要求：将敏感数据保留在自己的基础设施内
• 延迟要求：较小的模型部署速度更快

何时不应进行微调

• 一次性查询（改用提示词）
• 快速变化的信息（改用 RAG）
• 训练数据有限（通常少于 100 个示例是不够的）
• 通用知识问题（基础模型已足够）

快速开始

全量微调：

python examples/full_fine_tuning.py

LoRA（大多数情况下推荐）：

python examples/lora_fine_tuning.py

QLoRA（单 GPU）：

python examples/qlora_fine_tuning.py

数据准备：

python scripts/data_preparation.py

微调方法

1. 全量微调

在训练过程中更新所有模型参数。

优点：

• 最大程度提升性能
• 可以彻底改变模型行为
• 最适合领域差异显著的情况

缺点：

• 计算成本高
• 需要大量数据集（1000 个以上示例）
• 存在灾难性遗忘的风险
• 训练时间长

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

model_id = "meta-llama/Llama-2-7b"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

training_args = TrainingArguments(
    output_dir="./fine-tuned-llama",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    weight_decay=0.01,
    logging_steps=10,
    save_steps=100,
    eval_strategy="steps",
    eval_steps=50,
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

2. 参数高效微调（PEFT）

仅训练一小部分参数。

LoRA（低秩适配）

向现有权重添加可训练的低秩矩阵。

优点：

• 可训练参数减少 99%
• 保留基础模型知识
• 训练速度快（快 10-20 倍）
• 适配器之间切换方便

缺点：

• 性能略低于全量微调
• 推理时需要基础模型

from peft import get_peft_model, LoraConfig, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "meta-llama/Llama-2-7b"
model = AutoModelForCausalLM.from_pretrained(base_model_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Configure LoRA
lora_config = LoraConfig(
    r=8,  # Rank of low-rank matrices
    lora_alpha=16,  # Scaling factor
    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

# Wrap model with LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,738,415,616 || trainable%: 0.06

# Train as normal
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)
trainer.train()

# Save only LoRA weights
model.save_pretrained("./llama-lora-adapter")

QLoRA（量化 LoRA）

将 LoRA 与量化结合，实现极高效率。

from peft import prepare_model_for_kbit_training, get_peft_model, LoraConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# Quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b",
    quantization_config=bnb_config,
    device_map="auto"
)

# Prepare for training
model = prepare_model_for_kbit_training(model)

# Apply LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)

# Train on single GPU
trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./qlora-output",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        learning_rate=5e-4,
        num_train_epochs=3,
    ),
    train_dataset=train_dataset,
)
trainer.train()

前缀微调

在输入前添加可训练的词元。

from peft import get_peft_model, PrefixTuningConfig

config = PrefixTuningConfig(
    num_virtual_tokens=20,
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, config)
# Only 20 * embedding_dim parameters trained

3. 指令微调

使用示例训练模型遵循指令。

# Training data format
training_data = [
    {
        "instruction": "Translate to French",
        "input": "Hello, how are you?",
        "output": "Bonjour, comment allez-vous?"
    },
    {
        "instruction": "Summarize this text",
        "input": "Long document...",
        "output": "Summary..."
    }
]

# Template for training
template = """Below is an instruction that describes a task, paired with an input that provides further context.

### Instruction:
{instruction}

### Input:
{input}

### Response:
{output}"""

# Create formatted dataset
formatted_data = [
    template.format(**example) for example in training_data
]

4. 领域专用微调

针对特定行业或领域定制模型。

法律领域示例

legal_training_data = [
    {
        "prompt": "What are the key clauses in an NDA?",
        "completion": """Key clauses typically include:
1. Definition of Confidential Information
2. Non-Disclosure Obligations
3. Permitted Disclosures
4. Term and Termination
5. Return of Information
6. Remedies"""
    },
    # ... more legal examples
]

# Train on legal domain
model = fine_tune_on_domain(
    base_model="gpt-3.5-turbo",
    training_data=legal_training_data,
    epochs=3,
    learning_rate=0.0002,
)

数据准备

1. 数据集质量

class DatasetValidator:
    def validate_dataset(self, data):
        issues = {
            "empty_samples": 0,
            "duplicates": 0,
            "outliers": 0,
            "imbalance": {}
        }

        # Check for empty samples
        for sample in data:
            if not sample.get("text"):
                issues["empty_samples"] += 1

        # Check for duplicates
        texts = [s.get("text") for s in data]
        issues["duplicates"] = len(texts) - len(set(texts))

        # Check for length outliers
        lengths = [len(t.split()) for t in texts]
        mean_length = sum(lengths) / len(lengths)
        issues["outliers"] = sum(1 for l in lengths if l > mean_length * 3)

        return issues

# Validate before training
validator = DatasetValidator()
issues = validator.validate_dataset(training_data)
print(f"Dataset Issues: {issues}")

2. 数据增强

from nlpaug.augmenter.word import SynonymAug, RandomWordAug
import nlpaug.flow as naf

# Create augmentation pipeline
text = "The quick brown fox jumps over the lazy dog"

# Synonym replacement
aug_syn = SynonymAug(aug_p=0.3)
augmented_syn = aug_syn.augment(text)

# Random word insertion
aug_insert = RandomWordAug(action="insert", aug_p=0.3)
augmented_insert = aug_insert.augment(text)

# Combine augmentations
flow = naf.Sequential([
    SynonymAug(aug_p=0.2),
    RandomWordAug(action="swap", aug_p=0.2)
])
augmented = flow.augment(text)

3. 训练集/验证集划分

from sklearn.model_selection import train_test_split

# Create splits
train_data, eval_data = train_test_split(
    data,
    test_size=0.2,
    random_state=42
)

eval_data, test_data = train_test_split(
    eval_data,
    test_size=0.5,
    random_state=42
)

print(f"Train: {len(train_data)}, Eval: {len(eval_data)}, Test: {len(test_data)}")

训练技巧

1. 学习率调度

from torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR

# Linear warmup + cosine annealing
def get_scheduler(optimizer, num_steps):
    lr_scheduler = get_linear_schedule_with_warmup(
        optimizer,
        num_warmup_steps=500,
        num_training_steps=num_steps
    )
    return lr_scheduler

training_args = TrainingArguments(
    learning_rate=1e-4,
    lr_scheduler_type="cosine",
    warmup_steps=500,
    warmup_ratio=0.1,
)

2. 梯度累积

training_args = TrainingArguments(
    gradient_accumulation_steps=4,  # Accumulate gradients over 4 steps
    per_device_train_batch_size=1,   # Effective batch size: 1 * 4 = 4
)

# Simulates larger batch on limited GPU memory

3. 混合精度训练

training_args = TrainingArguments(
    fp16=True,  # Use 16-bit floats
    bf16=False,
)

# Reduces memory usage by 50%, speeds up training

4. 多 GPU 训练

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    gradient_accumulation_steps=4,
    dataloader_pin_memory=True,
    dataloader_num_workers=4,
)

# Automatically uses all available GPUs
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

热门微调模型

开源模型

Llama 3.2（Meta）

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-7b")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-7b")

# Fine-tune on custom data
# ... training code

特点：

• 7B、70B 参数版本
• 指令遵循能力强
• 非常适合领域适配
• Apache 2.0 许可证

Gemma 3（Google）

model = AutoModelForCausalLM.from_pretrained("google/gemma-3-2b")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-2b")

# Gemma 3 sizes: 2B, 7B, 27B
# Very efficient, great for fine-tuning

特点：

• 小型、中型和大型版本
• 高效的架构
• 适合边缘部署
• 基于前沿研究成果构建

Mistral 7B

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

# Strong performance, efficient architecture

特点：

• 滑动窗口注意力
• 推理效率高
• 性能与模型大小的比率出色

商业模型

OpenAI 微调 API

import openai

# Prepare training data
training_file = openai.File.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
fine_tune_job = openai.FineTuningJob.create(
    training_file=training_file.id,
    model="gpt-3.5-turbo",
    hyperparameters={
        "n_epochs": 3,
        "learning_rate_multiplier": 0.1,
    }
)

# Wait for completion
fine_tuned_model = openai.FineTuningJob.retrieve(fine_tune_job.id)
print(f"Status: {fine_tuned_model.status}")

# Use fine-tuned model
response = openai.ChatCompletion.create(
    model=fine_tuned_model.fine_tuned_model,
    messages=[{"role": "user", "content": "Hello"}]
)

评估与指标

1. 困惑度

import torch
from math import exp

def calculate_perplexity(model, eval_dataset):
    model.eval()
    total_loss = 0
    total_tokens = 0

    with torch.no_grad():
        for batch in eval_dataset:
            outputs = model(**batch)
            loss = outputs.loss
            total_loss += loss.item() * batch["input_ids"].shape[0]
            total_tokens += batch["input_ids"].shape[0]

    perplexity = exp(total_loss / total_tokens)
    return perplexity

perplexity = calculate_perplexity(model, eval_dataset)
print(f"Perplexity: {perplexity:.2f}")

2. 特定任务指标

from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

def evaluate_task(predictions, ground_truth):
    return {
        "accuracy": accuracy_score(ground_truth, predictions),
        "precision": precision_score(ground_truth, predictions, average='weighted'),
        "recall": recall_score(ground_truth, predictions, average='weighted'),
        "f1": f1_score(ground_truth, predictions, average='weighted'),
    }

# Evaluate on task
predictions = [model.predict(x) for x in test_data]
metrics = evaluate_task(predictions, test_labels)
print(f"Metrics: {metrics}")

3. 人工评估

class HumanEvaluator:
    def evaluate_response(self, prompt, response):
        criteria = {
            "relevance": self._score_relevance(prompt, response),
            "coherence": self._score_coherence(response),
            "factuality": self._score_factuality(response),
            "helpfulness": self._score_helpfulness(response),
        }
        return sum(criteria.values()) / len(criteria)

    def _score_relevance(self, prompt, response):
        # Score 1-5
        pass

    def _score_coherence(self, response):
        # Score 1-5
        pass

常见挑战与解决方案

挑战：灾难性遗忘

模型在适配新领域时忘记预训练知识。

解决方案：

• 使用较低的学习率（2e-5 到 5e-5）
• 减少训练轮数（1-3）
• 使用正则化技术
• 采用持续学习方法

# Conservative training settings
training_args = TrainingArguments(
    learning_rate=2e-5,  # Lower learning rate
    num_train_epochs=2,   # Few epochs
    weight_decay=0.01,    # L2 regularization
    warmup_steps=500,
    save_total_limit=3,
    load_best_model_at_end=True,
)

挑战：过拟合

模型在训练数据上表现良好，但在新数据上表现不佳。

解决方案：

• 使用更多训练数据
• 应用 dropout
• 提前停止
• 监控验证结果

training_args = TrainingArguments(
    eval_strategy="steps",
    eval_steps=50,
    load_best_model_at_end=True,
    early_stopping_patience=3,
    metric_for_best_model="eval_loss",
)

挑战：训练数据不足

用于微调的示例太少。

解决方案：

• 数据增强
• 使用 PEFT（LoRA），而非完整微调
• 使用提示词进行少样本学习
• 迁移学习

# Use LoRA when data is limited
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)

最佳实践

微调前

• ✓ 从强大的基础模型开始
• ✓ 准备高质量的训练数据（建议至少 100 个示例）
• ✓ 明确评估指标
• ✓ 合理划分训练集和验证集
• ✓ 记录目标

微调期间

• ✓ 监控训练损失和验证损失
• ✓ 使用合适的学习率
• ✓ 定期保存检查点
• ✓ 使用留出数据进行验证
• ✓ 留意过拟合和欠拟合

微调后

• ✓ 在测试集上评估
• ✓ 与基线进行比较
• ✓ 进行定性分析
• ✓ 记录配置和结果
• ✓ 为微调后的模型进行版本管理

实施检查清单

• 确定微调方法（完整微调、LoRA、QLoRA、指令微调）
• 准备并验证训练数据集（至少 100 个示例）
• 选择基础模型（Llama 3.2、Gemma 3、Mistral 等）
• 如果使用参数高效方法，则设置 PEFT
• 配置训练参数和超参数
• 实现数据加载和预处理
• 设置评估指标
• 监控并训练模型
• 在测试集上评估
• 保存微调后的模型并进行版本管理
• 在生产环境中测试
• 记录流程和结果

资源

框架

• Hugging Face Transformers：https://huggingface.co/transformers/
• PEFT（参数高效微调）：https://github.com/huggingface/peft
• Hugging Face Datasets：https://huggingface.co/datasets

模型

• Llama 3.2：https://www.meta.com/llama/
• Gemma 3：https://deepmind.google/technologies/gemma/
• Mistral：https://mistral.ai/

论文

• “LoRA：大语言模型的低秩适配”（Hu 等）
• “QLoRA：量化大语言模型的高效微调”（Dettmers 等）