alerting-irm

Grafana 告警与 IRM

文档：https://grafana.com/docs/grafana/latest/alerting.md

常见工作流

端到端配置新告警

1. 创建联系点（通知的发送位置）：

   curl -X POST https://grafana.example.com/api/v1/provisioning/contact-points \
     -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
     -d @contact-points.json
   

   验证：

   curl https://grafana.example.com/api/v1/provisioning/contact-points \
     -H 'Authorization: Bearer <token>' | jq '.[].name'
   

2. 添加通知策略（哪些告警发送到哪里）——匹配器模式请参阅下文的§ 通知策略。

3. 编写告警规则——选择类型：

   • Grafana 管理 → 请参阅 references/alerting.md § Grafana-managed alert rule
   • Prometheus/Mimir ruler → 请参阅 references/alerting.md § Prometheus / Mimir alert rule
   • Loki LogQL → 请参阅 references/alerting.md § Loki alert rule

4. 上线前验证路由：

   # Force-fire a test alert from the rule's UI, then check Alertmanager's view
   curl https://grafana.example.com/api/alertmanager/grafana/api/v2/alerts \
     -H 'Authorization: Bearer <token>' | jq '.[] | {alertname: .labels.alertname, receiver: .receivers}'
   

   应显示预期的接收器。如果显示的接收器不正确，请重新检查策略的匹配器。

将告警路由到 IRM / 值班系统

1. 在 IRM 中创建类型为“Grafana Alerting webhook”的集成 → 复制集成 URL
2. 在 Grafana Alerting 中添加一个指向该 URL 的 webhook 联系点（完整 YAML 请参阅 references/irm.md § Routing）
3. 添加通知策略匹配器，将相应严重级别的告警路由到新的联系点
4. 验证：触发测试告警；它应在约 30 秒内显示在 IRM 中。完整调试流程请参阅 references/irm.md § Verifying the IRM integration。

定义 SLO

1. 通过 UI 或 API 创建 SLO → Grafana 会自动生成记录规则、仪表板和消耗率告警（生成的 YAML 位于 references/slo.md）
2. 使用多窗口消耗率告警，而非单窗口告警——单窗口告警为何会因噪声触发，请参阅 references/slo.md § Multi-window burn-rate alerts
3. 按照 references/slo.md § Validating SLO config 中的 4 步模式验证

联系点（YAML 配置）

# provisioning/alerting/contact_points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: pagerduty-critical
    receivers:
      - uid: pd-receiver
        type: pagerduty
        settings:
          integrationKey: YOUR_PAGERDUTY_KEY
          severity: critical

  - orgId: 1
    name: slack-alerts
    receivers:
      - uid: slack-receiver
        type: slack
        settings:
          url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
          channel: '#alerts'

电子邮件、webhook、Teams、Telegram、OnCall 和其他接收器类型，请参阅 references/alerting.md § Contact point receiver types。

通知策略

带有标签匹配器的分层路由树：

# provisioning/alerting/notification_policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: default-receiver
    group_by: ['alertname', 'cluster', 'service']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    routes:
      # Critical alerts → PagerDuty
      - receiver: pagerduty-critical
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 4h

      # Platform team → Slack, but page on critical
      - receiver: slack-alerts
        matchers:
          - team = platform
        routes:
          - receiver: pagerduty-critical
            matchers:
              - severity = critical

      # Everything else → email
      - receiver: email-alerts
        matchers:
          - severity =~ "warning|info"

静默

抑制匹配告警的通知，但不停止评估：

curl -X POST https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighErrorRate", "isRegex": false},
      {"name": "env", "value": "staging", "isRegex": false}
    ],
    "startsAt": "2024-01-01T00:00:00Z",
    "endsAt": "2024-01-01T02:00:00Z",
    "comment": "Maintenance window",
    "createdBy": "admin"
  }'

# Verify it was created
curl https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' | jq '.[] | select(.status.state == "active")'

告警规则状态

状态	描述
Normal	条件未满足
Pending	条件已满足，等待 for 时长结束
Firing	条件在完整的 for 时长内持续满足
NoData	查询未返回数据
Error	查询/评估错误
Recovering	先前处于触发状态，当前条件不再满足

配置目录结构

provisioning/alerting/
├── alert_rules.yaml          # Alert and recording rules
├── contact_points.yaml       # Notification destinations
├── notification_policies.yaml  # Routing tree
├── templates.yaml            # Message templates
└── mute_timings.yaml         # Recurring mute windows

API 配置（保持 UI 可编辑）

添加 X-Disable-Provenance: true，以便通过 API 配置后仍可在 UI 中编辑资源：

curl -X PUT https://grafana.example.com/api/v1/provisioning/policies \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @policy.json

curl -X POST https://grafana.example.com/api/v1/provisioning/alert-rules \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @rule.json

参考资料

• references/alerting.md — 完整告警规则 YAML（Grafana 管理 / Prometheus / Loki）+ 通知模板
• references/slo.md — 生成的 SLO 记录规则 + 多窗口消耗率告警模式 + 验证步骤
• references/irm.md — IRM 功能、集成来源、告警到 IRM 的路由 + 验证 + 常见故障模式