ai-sre-incident-response

AI SRE 事件响应

将 SRE 的严谨性应用于 AI 系统，处理质量回归、不安全输出和预算激增等事件。

适用场景

• LLM 端点开始返回质量下降或含有幻觉的答案
• Token 支出激增，超出预算阈值
• 模型提供商宕机，需要切换流量
• 安全护栏触发率异常升高
• 新模型部署导致延迟或准确率回归

前置条件

• 已部署 Prometheus 和 Alertmanager，并配置 AI 服务的抓取目标
• 已为关键指标（延迟、错误率、成本、质量）配置 Grafana 仪表盘
• 已在 PagerDuty、Opsgenie 或同类工具中配置值班轮换
• 响应人员可以访问运行手册仓库
• 已配置模型和提示词版本的回滚机制（GitOps 或功能标志）

AI 事件类别

• 可用性事件：模型或提供商不可用、超时激增。
• 质量事件：答案准确率或工具成功率低于 SLO。
• 安全事件：有害或违反策略的输出增加。
• 成本事件：Token 或提供商支出意外激增。

严重级别框架

严重级别	标准	响应时间	通知
SEV1	面向用户的服务中断、合规风险、数据泄露	5 分钟	呼叫值班人员和事件指挥官
SEV2	关键流程严重降级	15 分钟	呼叫值班人员
SEV3	影响有限或仅影响内部的问题	1 小时	Slack 警报
SEV4	外观问题或低优先级回归	下一个工作日	工单

AI 服务的关键指标

• 请求成功率
• 延迟（排队 + 生成 + 工具执行）
• 幻觉／有依据性代理指标
• 每分钟和每个租户的成本
• 护栏违规率

Prometheus 警报规则

# prometheus-ai-alerts.yaml
groups:
  - name: ai-service-alerts
    rules:
      - alert: ModelEndpointDown
        expr: up{job="llm-inference"} == 0
        for: 2m
        labels:
          severity: sev1
        annotations:
          summary: "LLM inference endpoint {{ $labels.instance }} is down"
          runbook_url: "https://runbooks.internal/ai/model-outage"

      - alert: HighHallucinationRate
        expr: |
          rate(llm_hallucination_detected_total[10m])
          / rate(llm_requests_total[10m]) > 0.15
        for: 5m
        labels:
          severity: sev2
        annotations:
          summary: "Hallucination rate above 15% for {{ $labels.model }}"
          runbook_url: "https://runbooks.internal/ai/quality-regression"

      - alert: TokenCostExplosion
        expr: |
          sum(rate(llm_token_cost_dollars[5m])) by (tenant)
          > 0.50
        for: 3m
        labels:
          severity: sev2
        annotations:
          summary: "Token spend exceeds $0.50/min for tenant {{ $labels.tenant }}"
          runbook_url: "https://runbooks.internal/ai/cost-spike"

      - alert: LatencyP95Exceeded
        expr: |
          histogram_quantile(0.95,
            rate(llm_request_duration_seconds_bucket[5m])
          ) > 5
        for: 5m
        labels:
          severity: sev2
        annotations:
          summary: "LLM p95 latency exceeds 5s for {{ $labels.service }}"

      - alert: GuardrailViolationSpike
        expr: |
          rate(llm_guardrail_violations_total[10m])
          / rate(llm_requests_total[10m]) > 0.05
        for: 5m
        labels:
          severity: sev1
        annotations:
          summary: "Guardrail violations above 5% for {{ $labels.model }}"
          runbook_url: "https://runbooks.internal/ai/safety-incident"

      - alert: ModelQualityDrop
        expr: |
          llm_eval_score{metric="groundedness"} < 0.70
        for: 10m
        labels:
          severity: sev2
        annotations:
          summary: "Groundedness score dropped below 0.70 for {{ $labels.model }}"

      - alert: ProviderErrorRateHigh
        expr: |
          rate(llm_provider_errors_total[5m])
          / rate(llm_provider_requests_total[5m]) > 0.10
        for: 3m
        labels:
          severity: sev2
        annotations:
          summary: "Provider {{ $labels.provider }} error rate above 10%"

响应预案

模型宕机运行手册

TRIGGER: ModelEndpointDown fires for > 2 minutes
RESPONDER: On-call AI platform engineer

1. Acknowledge alert in PagerDuty.
2. Check provider status page (e.g., status.openai.com).
3. Verify network connectivity:
     curl -s -o /dev/null -w "%{http_code}" https://api.provider.com/health
4. If provider is down:
     a. Enable fallback model route in gateway config.
     b. kubectl set env deployment/llm-gateway FALLBACK_ENABLED=true
     c. Verify fallback traffic is flowing via Grafana dashboard.
5. If self-hosted model is down:
     a. Check pod status: kubectl get pods -l app=llm-inference -n ai
     b. Check GPU health: kubectl logs -l app=llm-inference --tail=50
     c. Restart if OOM: kubectl rollout restart deployment/llm-inference -n ai
6. Freeze all deployments:
     kubectl annotate deployment --all deploy-freeze=true -n ai
7. Communicate ETA in #incident-channel.
8. When resolved, unfreeze and run smoke tests.

质量回归运行手册（幻觉激增）

TRIGGER: HighHallucinationRate or ModelQualityDrop fires
RESPONDER: On-call AI engineer + ML lead

1. Acknowledge alert. Open incident ticket.
2. Identify scope:
     - Which model version? Check deployment metadata.
     - Which routes/tenants affected? Filter by labels in Grafana.
3. Check recent changes:
     - Model version promotion in last 24h?
     - Prompt template changes in last 24h?
     - Retrieval index rebuild in last 24h?
4. If recent model change:
     kubectl rollout undo deployment/llm-inference -n ai
5. If recent prompt change:
     git revert <commit> && git push  # triggers GitOps redeploy
6. Increase trace sampling to 100% for affected route:
     kubectl set env deployment/llm-gateway TRACE_SAMPLE_RATE=1.0
7. Run offline eval suite against current production:
     python run_evals.py --target prod --suite quality --compare baseline
8. Confirm metrics return to baseline before closing.

Token 成本激增运行手册

TRIGGER: TokenCostExplosion fires
RESPONDER: On-call platform engineer

1. Identify top consumers:
     Query: topk(10, sum(rate(llm_token_cost_dollars[15m])) by (tenant, model, route))
2. Check for runaway loops:
     - Agent retry storms (exponential token growth per request)
     - Missing max_tokens caps on new routes
     - Cache bypass due to config change
3. Apply immediate caps:
     kubectl patch configmap llm-quotas -n ai --patch '
       data:
         max_tokens_per_request: "4096"
         rpm_limit: "60"
     '
4. Enable semantic cache if disabled:
     kubectl set env deployment/llm-gateway CACHE_ENABLED=true
5. Route traffic to cheaper model tier:
     kubectl set env deployment/llm-gateway DEFAULT_MODEL=gpt-4o-mini
6. Notify affected tenants of temporary limits.
7. Open postmortem with cost attribution analysis.

升级处理流程

Level 1 (0-15 min):  On-call AI platform engineer
Level 2 (15-30 min): AI platform team lead + affected product owner
Level 3 (30-60 min): Engineering director + security (if safety incident)
Level 4 (60+ min):   VP Engineering + legal (if compliance/data incident)

Safety incidents always start at Level 2 minimum.
Provider-side incidents: open support ticket immediately at Level 1.

检测查询（PromQL）

# Request success rate by model
1 - (
  sum(rate(llm_requests_total{status="error"}[5m])) by (model)
  / sum(rate(llm_requests_total[5m])) by (model)
)

# Cost per successful answer
sum(rate(llm_token_cost_dollars[5m])) by (route)
/ sum(rate(llm_requests_total{status="success"}[5m])) by (route)

# Hallucination rate trend (1h window, 5m steps)
rate(llm_hallucination_detected_total[1h])
/ rate(llm_requests_total[1h])

# Latency breakdown by stage
histogram_quantile(0.95, rate(llm_retrieval_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(llm_generation_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(llm_tool_execution_duration_seconds_bucket[5m]))

# Tenant cost leaderboard
topk(10, sum(rate(llm_token_cost_dollars[1h])) by (tenant))

事后复盘要求

• 包含检测器和响应人员时间戳的时间线
• 按租户和功能划分的影响范围
• 遗漏的信号和警报调优操作
• 明确的加固任务、负责人和截止日期
• 成本影响（金额、Token 数量、受影响请求）
• 客户沟通记录

事后复盘模板

## Incident Summary
- **Severity**: SEVx
- **Duration**: start_time - end_time (Xh Ym)
- **Detection**: How was it detected? (alert / customer report / manual)
- **Impact**: X tenants, Y requests, $Z cost

## Timeline
| Time (UTC) | Event |
|------------|-------|
| HH:MM | Alert fired |
| HH:MM | Responder acknowledged |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Incident resolved |

## Root Cause
[Description]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Tune alert threshold | @engineer | YYYY-MM-DD | Open |
| Add fallback route | @platform | YYYY-MM-DD | Open |

AI 系统的混沌工程

定期测试事件响应准备情况：

• 提供商故障切换演练：在网络层阻断提供商 API，验证回退能否在 SLO 时间内启用。
• 模型回滚演练：部署已知存在问题的模型版本，验证自动质量门禁能否发现问题。
• 成本上限演练：模拟 Token 使用失控，验证配额能否在达到预算阈值前触发。
• 缓存故障演练：禁用语义缓存，验证系统能否平稳降级。

故障排查

症状	检查项	修复方法
所有请求均超时	提供商状态页、DNS 解析	启用备用提供商
质量逐渐下降	最近的模型／提示词部署	回滚到上一个已知正常版本
成本突然激增	按租户划分的 Token 使用情况仪表盘	应用紧急 Token 上限
护栏违规激增	模型版本、提示词注入日志	启用更严格的输入过滤
间歇性 503 错误	Pod 重启、GPU OOM 事件	提高内存限制或减少批量大小

相关技能

• 事件响应 - 标准事件流程和证据
• 警报与值班 - 呼叫和升级策略
• LLM 成本优化 - 支出控制和效率模式
• Agent 可观测性 - 为请求、追踪和成本添加检测
• RAG 可观测性评估 - RAG 质量监控