mcpField
O
observability-monitoring

observability-monitoring

github.com/manutej云与 DevOps62455 次安装40 周前更新
代码仓库

关于 observability-monitoring

介绍使用 Prometheus、Grafana 及相关云原生工具开展生产可观测性与监控的技能，面向构建指标、告警、仪表板和可靠性实践的工程师。

主要功能

  1. 01Prometheus 指标与 PromQL
  2. 02Grafana 仪表板模式
  3. 03告警规则与 Alertmanager
  4. 04SLI、SLO 与错误预算
  5. 05Exporter 配置与自定义指标
  6. 06标签与基数管理建议

适用场景

  1. 01监控生产服务健康状况
  2. 02构建 API 与基础设施仪表板
  3. 03创建可执行的可靠性告警
  4. 04分析时间序列指标以规划容量

安装

复制下面的命令，在终端中运行。

$npx skills add https://github.com/manutej/luxor-claude-marketplace --skill observability-monitoring

安装后在 Claude Code 中输入/observability-monitoring确认已加载。

附带文件EXAMPLES.mdREADME.md

详细介绍

可观测性与监控

使用 Prometheus、Grafana 及更广泛的云原生监控生态系统，实现生产级可观测性与监控的综合指南。本指南涵盖指标采集、时间序列分析、告警、可视化和卓越运维模式。

何时使用本指南

在以下情形中使用本指南：

  • 为生产系统和应用设置监控
  • 为微服务实现指标采集与可观测性
  • 创建用于监控系统健康状况的仪表板和可视化
  • 定义告警规则和事件响应自动化流程
  • 使用时间序列数据分析系统性能和容量
  • 为服务可靠性实现 SLI、SLO 和 SLA
  • 使用指标和追踪信息调试生产问题
  • 为应用专属指标构建自定义导出器
  • 为多集群监控设置联邦
  • 从旧版监控迁移到云原生解决方案
  • 实现成本监控和优化追踪
  • 为 DevOps 团队创建实时运维仪表板

核心概念

可观测性的四大支柱

现代可观测性建立在四个基本支柱之上：

  1. 指标：随时间变化的系统行为数值测量

    • Counter：单调递增的值（已处理的请求、错误）
    • Gauge：某个时间点上可升可降的值（内存使用量、温度）
    • Histogram：值的分布（请求时长区间）
    • Summary：与 Histogram 类似，但在客户端计算分位数

  2. 日志：带有上下文信息的离散事件

    • 结构化日志记录（JSON、键值对）
    • 集中式日志聚合（ELK、Loki）
    • 与指标和追踪信息关联

  3. 追踪：请求在分布式系统中的流转过程

    • Span：具有开始和结束时间的单个工作单元
    • Trace：表示端到端请求的 Span 集合
    • 使用 OpenTelemetry 进行分布式追踪

  4. 事件：系统生命周期中的重要情况

    • 部署、配置变更
    • 扩缩容事件、故障
    • 业务事件和用户操作
Prometheus 架构

Prometheus 是一个基于拉取的监控系统，包含以下关键组件：

时间序列数据库（TSDB）

  • 将指标存储为时间序列数据
  • 高效的压缩和保留策略
  • 本地存储，可选远程存储

抓取目标

  • 服务发现（Kubernetes、Consul、EC2 等）
  • 静态配置
  • 通过重标记灵活选择目标

PromQL 查询引擎

  • 用于指标分析的强大查询语言
  • 聚合、筛选和数学运算
  • 范围向量和瞬时向量

Alertmanager

  • 告警规则评估
  • 分组、静默和路由
  • 与 PagerDuty、Slack、电子邮件和 Webhook 集成

导出器

  • 充当 Prometheus 与系统之间的桥梁
  • Node exporter、cAdvisor、自定义导出器
  • 用于数据库和服务的第三方导出器
指标标签与基数

标签是附加到指标上的键值对：

http_requests_total{method="GET", endpoint="/api/users", status="200"}

标签最佳实践：

  • 将标签用于你会按其查询或聚合的维度
  • 避免使用高基数标签（用户 ID、时间戳）
  • 在各项指标中保持标签名称一致
  • 使用重标记规范化外部标签

基数注意事项：

  • 每种唯一的标签组合都会生成一个新的时间序列
  • 高基数会增加内存和存储开销
  • 使用 prometheus_tsdb_symbol_table_size_bytes 监控基数
  • 使用记录规则预聚合高基数指标
记录规则

预先计算常用或开销较大的查询：

groups:
  - name: api_performance
    interval: 30s
    rules:
      - record: api:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
      - record: api:http_requests:rate5m
        expr: rate(http_requests_total[5m])

优势：

  • 加快仪表板加载速度
  • 减轻 Prometheus 的查询负载
  • 统一指标命名规范
  • 支持复杂聚合
服务级别目标（SLO）

定义并跟踪可靠性目标：

SLI（服务级别指标）：衡量服务质量的指标

  • 可用性：成功请求的百分比
  • 延迟：低于阈值的请求百分比
  • 吞吐量：每秒请求数

SLO（服务级别目标）：SLI 的目标

  • 99.9% 的可用性（每月停机 43.8 分钟）
  • 95% 的请求耗时 < 200ms
  • 持续达到 1000 RPS

SLA（服务级别协议）：包含违约后果的合同

  • 对客户的外部承诺
  • 违反 SLO 时的财务处罚

错误预算：可接受的故障率

  • 错误预算 = 100% - SLO
  • 99.9% 的 SLO = 0.1% 的错误预算
  • 在创新与可靠性之间权衡，使用错误预算

Prometheus 设置与配置

Prometheus 基本配置
# prometheus.yml
global:
  scrape_interval: 15s      # Default scrape interval
  evaluation_interval: 15s  # Alert rule evaluation interval
  external_labels:
    cluster: 'production'
    region: 'us-west-2'

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load rules
rule_files:
  - 'rules/*.yml'
  - 'alerts/*.yml'

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter for system metrics
  - job_name: 'node'
    static_configs:
      - targets:
          - 'node1:9100'
          - 'node2:9100'
          - 'node3:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
        replacement: '${1}'

  # Application metrics
  - job_name: 'api'
    static_configs:
      - targets: ['api-1:8080', 'api-2:8080', 'api-3:8080']
        labels:
          env: 'production'
          tier: 'backend'
Kubernetes 服务发现
scrape_configs:
  # Kubernetes API server
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

  # Kubernetes pods with prometheus.io annotations
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with prometheus.io/scrape: "true" annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use the port from prometheus.io/port annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (\d+)
        target_label: __address__
        replacement: ${1}:${2}
      # Use the path from prometheus.io/path annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
      # Add namespace label
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      # Add pod name label
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

  # Kubernetes services
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service
    metrics_path: /probe
    params:
      module: [http_2xx]
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance
存储与保留
# Storage configuration
storage:
  tsdb:
    path: /prometheus/data
    retention.time: 15d
    retention.size: 50GB

# Remote write for long-term storage
remote_write:
  - url: "https://prometheus-remote-storage.example.com/api/v1/write"
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/remote_storage_password
    queue_config:
      capacity: 10000
      max_shards: 50
      max_samples_per_send: 5000
    write_relabel_configs:
      # Drop high-cardinality metrics
      - source_labels: [__name__]
        regex: 'container_network_.*'
        action: drop

# Remote read for querying historical data
remote_read:
  - url: "https://prometheus-remote-storage.example.com/api/v1/read"
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/remote_storage_password
    read_recent: true

PromQL：Prometheus 查询语言

瞬时向量与选择器
# Basic metric selection
http_requests_total

# Filter by label
http_requests_total{job="api", status="200"}

# Regex matching
http_requests_total{status=~"2..|3.."}

# Negative matching
http_requests_total{status!="500"}

# Multiple label matchers
http_requests_total{job="api", method="GET", status=~"2.."}
范围向量与聚合
# 5-minute range vector
http_requests_total[5m]

# Rate of increase per second
rate(http_requests_total[5m])

# Increase over time window
increase(http_requests_total[1h])

# Average over time
avg_over_time(cpu_usage[5m])

# Max/Min over time
max_over_time(response_time_seconds[10m])
min_over_time(response_time_seconds[10m])

# Standard deviation
stddev_over_time(response_time_seconds[5m])
聚合运算符
# Sum across all instances
sum(rate(http_requests_total[5m]))

# Sum grouped by job
sum by (job) (rate(http_requests_total[5m]))

# Average grouped by multiple labels
avg by (job, instance) (cpu_usage)

# Count number of series
count(up == 1)

# Topk and bottomk
topk(5, rate(http_requests_total[5m]))
bottomk(3, node_memory_available_bytes)

# Quantile across instances
quantile(0.95, http_request_duration_seconds)
数学运算
# Arithmetic operations
(node_memory_total_bytes - node_memory_available_bytes) / node_memory_total_bytes * 100

# Comparison operators
http_request_duration_seconds > 0.5

# Logical operators
up == 1 and rate(http_requests_total[5m]) > 100

# Vector matching
rate(http_requests_total[5m]) / on(instance) group_left rate(http_responses_total[5m])
高级 PromQL 模式
# Request success rate
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# Latency percentiles (histogram)
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Predict linear growth
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600)

# Detect anomalies with standard deviation
abs(cpu_usage - avg_over_time(cpu_usage[1h]))
>
3 * stddev_over_time(cpu_usage[1h])

# Calculate saturation (RED method)
sum(rate(cpu_seconds_total{mode!="idle"}[5m])) by (instance)
/
count(cpu_seconds_total{mode="idle"}) by (instance)

# Burn rate for SLO
(
  sum(rate(http_requests_total{status=~"5.."}[1h]))
  /
  sum(rate(http_requests_total[1h]))
)
>
(14.4 * (1 - 0.999))  # For 99.9% SLO

使用 Prometheus 和 Alertmanager 进行告警

告警规则定义
# alerts/api_alerts.yml
groups:
  - name: api_alerts
    interval: 30s
    rules:
      # High error rate alert
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }} on {{ $labels.service }}"
          runbook_url: "https://runbooks.example.com/HighErrorRate"

      # High latency alert
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 1
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "High latency on {{ $labels.service }}"
          description: "P99 latency is {{ $value }}s on {{ $labels.service }}"

      # Service down alert
      - alert: ServiceDown
        expr: up{job="api"} == 0
        for: 2m
        labels:
          severity: critical
          team: sre
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          description: "{{ $labels.job }} on {{ $labels.instance }} has been down for more than 2 minutes"

      # Disk space alert
      - alert: DiskSpaceLow
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"}
          /
          node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is {{ $value | humanize }}% on {{ $labels.instance }}"

      # Memory pressure alert
      - alert: HighMemoryUsage
        expr: |
          (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 10m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | humanize }}% on {{ $labels.instance }}"

      # CPU saturation alert
      - alert: HighCPUUsage
        expr: |
          100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 15m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | humanize }}% on {{ $labels.instance }}"
Alertmanager 配置
# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# Templates for notifications
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# Route tree for alert distribution
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'team-default'

  routes:
    # Critical alerts go to PagerDuty
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true

    # Critical alerts also go to Slack
    - match:
        severity: critical
      receiver: 'slack-critical'
      group_wait: 0s

    # Warning alerts to Slack only
    - match:
        severity: warning
      receiver: 'slack-warnings'

    # Team-specific routing
    - match:
        team: backend
      receiver: 'team-backend'

    - match:
        team: frontend
      receiver: 'team-frontend'

    # Database alerts to DBA team
    - match_re:
        service: 'postgres|mysql|mongodb'
      receiver: 'team-dba'

# Alert receivers/integrations
receivers:
  - name: 'team-default'
    slack_configs:
      - channel: '#alerts'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        send_resolved: true

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        description: '{{ .GroupLabels.alertname }}: {{ .GroupLabels.service }}'
        severity: '{{ .CommonLabels.severity }}'

  - name: 'slack-critical'
    slack_configs:
      - channel: '#incidents'
        title: 'CRITICAL: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ .Annotations.description }}{{ end }}'
        color: 'danger'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#monitoring'
        title: 'Warning: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        color: 'warning'
        send_resolved: true

  - name: 'team-backend'
    slack_configs:
      - channel: '#team-backend'
        send_resolved: true
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.gmail.com:587'
        auth_username: '[email protected]'
        auth_password_file: '/etc/alertmanager/email_password'

  - name: 'team-frontend'
    slack_configs:
      - channel: '#team-frontend'
        send_resolved: true

  - name: 'team-dba'
    slack_configs:
      - channel: '#team-dba'
        send_resolved: true
    pagerduty_configs:
      - service_key: 'DBA_PAGERDUTY_KEY'

# Inhibition rules (suppress alerts)
inhibit_rules:
  # Inhibit warnings if critical alert is firing
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

  # Don't alert on instance down if cluster is down
  - source_match:
      alertname: 'ClusterDown'
    target_match_re:
      alertname: 'InstanceDown|ServiceDown'
    equal: ['cluster']
用于 SLO 的多窗口多燃烧率告警
# SLO-based alerting using burn rate
groups:
  - name: slo_alerts
    interval: 30s
    rules:
      # Fast burn (1h window, 5m burn)
      - alert: ErrorBudgetBurnFast
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (14.4 * (1 - 0.999))
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (14.4 * (1 - 0.999))
        for: 2m
        labels:
          severity: critical
          slo: "99.9%"
        annotations:
          summary: "Fast error budget burn detected"
          description: "Error rate is burning through 99.9% SLO budget 14.4x faster than normal"

      # Slow burn (6h window, 30m burn)
      - alert: ErrorBudgetBurnSlow
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > (6 * (1 - 0.999))
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[30m]))
            /
            sum(rate(http_requests_total[30m]))
          ) > (6 * (1 - 0.999))
        for: 15m
        labels:
          severity: warning
          slo: "99.9%"
        annotations:
          summary: "Slow error budget burn detected"
          description: "Error rate is burning through 99.9% SLO budget 6x faster than normal"

Grafana 仪表盘与可视化

仪表盘 JSON 结构
{
  "dashboard": {
    "title": "API Performance Dashboard",
    "tags": ["api", "performance", "production"],
    "timezone": "browser",
    "editable": true,
    "graphTooltip": 1,
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "timepicker": {
      "refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m"],
      "time_options": ["5m", "15m", "1h", "6h", "12h", "24h", "7d"]
    },
    "templating": {
      "list": [
        {
          "name": "cluster",
          "type": "query",
          "datasource": "Prometheus",
          "query": "label_values(up, cluster)",
          "refresh": 1,
          "multi": false,
          "includeAll": false
        },
        {
          "name": "service",
          "type": "query",
          "datasource": "Prometheus",
          "query": "label_values(up{cluster=\"$cluster\"}, service)",
          "refresh": 1,
          "multi": true,
          "includeAll": true
        },
        {
          "name": "interval",
          "type": "interval",
          "query": "1m,5m,10m,30m,1h",
          "auto": true,
          "auto_count": 30,
          "auto_min": "10s"
        }
      ]
    },
    "panels": [
      {
        "id": 1,
        "title": "Request Rate",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (service)",
            "legendFormat": "{{ service }}",
            "refId": "A"
          }
        ],
        "yaxes": [
          {"format": "reqps", "label": "Requests/sec"},
          {"format": "short"}
        ],
        "legend": {
          "show": true,
          "values": true,
          "current": true,
          "avg": true,
          "max": true
        }
      },
      {
        "id": 2,
        "title": "Error Rate",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\",status=~\"5..\"}[$interval])) by (service) / sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (service) * 100",
            "legendFormat": "{{ service }} error %",
            "refId": "A"
          }
        ],
        "yaxes": [
          {"format": "percent", "label": "Error Rate"},
          {"format": "short"}
        ],
        "alert": {
          "conditions": [
            {
              "evaluator": {"params": [5], "type": "gt"},
              "operator": {"type": "and"},
              "query": {"params": ["A", "5m", "now"]},
              "reducer": {"params": [], "type": "avg"},
              "type": "query"
            }
          ],
          "executionErrorState": "alerting",
          "frequency": "1m",
          "handler": 1,
          "name": "High Error Rate",
          "noDataState": "no_data",
          "notifications": []
        }
      },
      {
        "id": 3,
        "title": "Latency Percentiles",
        "type": "graph",
        "gridPos": {"h": 8, "w": 24, "x": 0, "y": 8},
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (service, le))",
            "legendFormat": "{{ service }} p99",
            "refId": "A"
          },
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (service, le))",
            "legendFormat": "{{ service }} p95",
            "refId": "B"
          },
          {
            "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (service, le))",
            "legendFormat": "{{ service }} p50",
            "refId": "C"
          }
        ],
        "yaxes": [
          {"format": "s", "label": "Duration"},
          {"format": "short"}
        ]
      }
    ]
  }
}
RED 方法仪表盘

RED 方法关注请求速率、错误率和持续时间：

{
  "panels": [
    {
      "title": "Request Rate (per service)",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total[$__rate_interval])) by (service)"
        }
      ]
    },
    {
      "title": "Error Rate % (per service)",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[$__rate_interval])) by (service) / sum(rate(http_requests_total[$__rate_interval])) by (service) * 100"
        }
      ]
    },
    {
      "title": "Duration p99 (per service)",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (service, le))"
        }
      ]
    }
  ]
}
USE 方法仪表盘

USE 方法监控利用率、饱和度和错误：

{
  "panels": [
    {
      "title": "CPU Utilization %",
      "targets": [
        {
          "expr": "100 - (avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])) * 100)"
        }
      ]
    },
    {
      "title": "CPU Saturation (Load Average)",
      "targets": [
        {
          "expr": "node_load1"
        }
      ]
    },
    {
      "title": "Memory Utilization %",
      "targets": [
        {
          "expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100"
        }
      ]
    },
    {
      "title": "Disk I/O Utilization %",
      "targets": [
        {
          "expr": "rate(node_disk_io_time_seconds_total[$__rate_interval]) * 100"
        }
      ]
    },
    {
      "title": "Network Errors",
      "targets": [
        {
          "expr": "rate(node_network_receive_errs_total[$__rate_interval]) + rate(node_network_transmit_errs_total[$__rate_interval])"
        }
      ]
    }
  ]
}

Exporter 与指标采集

用于系统指标的 Node Exporter
# Install node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz
cd node_exporter-1.6.1.linux-amd64
./node_exporter --web.listen-address=":9100" \
  --collector.filesystem.mount-points-exclude="^/(dev|proc|sys|var/lib/docker/.+)($|/)" \
  --collector.netclass.ignored-devices="^(veth.*|br.*|docker.*|lo)$"

Node Exporter 的关键指标：

  • node_cpu_seconds_total：按模式统计的 CPU 使用情况
  • node_memory_MemTotal_bytes、node_memory_MemAvailable_bytes：内存
  • node_disk_io_time_seconds_total：磁盘 I/O
  • node_network_receive_bytes_total、node_network_transmit_bytes_total：网络
  • node_filesystem_size_bytes、node_filesystem_avail_bytes：磁盘空间
自定义应用 Exporter（Python）
# app_exporter.py
from prometheus_client import start_http_server, Counter, Gauge, Histogram, Summary
import time
import random

# Define metrics
REQUEST_COUNT = Counter(
    'app_requests_total',
    'Total app requests',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'app_request_duration_seconds',
    'Request duration in seconds',
    ['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 2.5, 5.0, 10.0]
)

ACTIVE_USERS = Gauge(
    'app_active_users',
    'Number of active users'
)

QUEUE_SIZE = Gauge(
    'app_queue_size',
    'Current queue size',
    ['queue_name']
)

DATABASE_CONNECTIONS = Gauge(
    'app_database_connections',
    'Number of database connections',
    ['pool', 'state']
)

CACHE_HITS = Counter(
    'app_cache_hits_total',
    'Total cache hits',
    ['cache_name']
)

CACHE_MISSES = Counter(
    'app_cache_misses_total',
    'Total cache misses',
    ['cache_name']
)

def simulate_metrics():
    """Simulate application metrics"""
    while True:
        # Simulate requests
        method = random.choice(['GET', 'POST', 'PUT', 'DELETE'])
        endpoint = random.choice(['/api/users', '/api/products', '/api/orders'])
        status = random.choice(['200', '200', '200', '400', '500'])

        REQUEST_COUNT.labels(method=method, endpoint=endpoint, status=status).inc()

        # Simulate request duration
        duration = random.uniform(0.01, 2.0)
        REQUEST_DURATION.labels(method=method, endpoint=endpoint).observe(duration)

        # Update gauges
        ACTIVE_USERS.set(random.randint(100, 1000))
        QUEUE_SIZE.labels(queue_name='jobs').set(random.randint(0, 50))
        QUEUE_SIZE.labels(queue_name='emails').set(random.randint(0, 20))

        # Database connection pool
        DATABASE_CONNECTIONS.labels(pool='main', state='active').set(random.randint(5, 20))
        DATABASE_CONNECTIONS.labels(pool='main', state='idle').set(random.randint(10, 30))

        # Cache metrics
        if random.random() > 0.3:
            CACHE_HITS.labels(cache_name='redis').inc()
        else:
            CACHE_MISSES.labels(cache_name='redis').inc()

        time.sleep(1)

if __name__ == '__main__':
    # Start metrics server on port 8000
    start_http_server(8000)
    print("Metrics server started on port 8000")
    simulate_metrics()
自定义 Exporter（Go）
package main

import (
    "log"
    "math/rand"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "app_requests_total",
            Help: "Total number of requests",
        },
        []string{"method", "endpoint", "status"},
    )

    requestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "app_request_duration_seconds",
            Help:    "Request duration in seconds",
            Buckets: prometheus.ExponentialBuckets(0.01, 2, 10),
        },
        []string{"method", "endpoint"},
    )

    activeUsers = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "app_active_users",
            Help: "Number of active users",
        },
    )

    databaseConnections = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "app_database_connections",
            Help: "Number of database connections",
        },
        []string{"pool", "state"},
    )
)

func init() {
    prometheus.MustRegister(requestsTotal)
    prometheus.MustRegister(requestDuration)
    prometheus.MustRegister(activeUsers)
    prometheus.MustRegister(databaseConnections)
}

func simulateMetrics() {
    ticker := time.NewTicker(1 * time.Second)
    defer ticker.Stop()

    for range ticker.C {
        // Simulate requests
        methods := []string{"GET", "POST", "PUT", "DELETE"}
        endpoints := []string{"/api/users", "/api/products", "/api/orders"}
        statuses := []string{"200", "200", "200", "400", "500"}

        method := methods[rand.Intn(len(methods))]
        endpoint := endpoints[rand.Intn(len(endpoints))]
        status := statuses[rand.Intn(len(statuses))]

        requestsTotal.WithLabelValues(method, endpoint, status).Inc()
        requestDuration.WithLabelValues(method, endpoint).Observe(rand.Float64() * 2)

        // Update gauges
        activeUsers.Set(float64(rand.Intn(900) + 100))
        databaseConnections.WithLabelValues("main", "active").Set(float64(rand.Intn(15) + 5))
        databaseConnections.WithLabelValues("main", "idle").Set(float64(rand.Intn(20) + 10))
    }
}

func main() {
    go simulateMetrics()

    http.Handle("/metrics", promhttp.Handler())
    log.Println("Starting metrics server on :8000")
    log.Fatal(http.ListenAndServe(":8000", nil))
}
PostgreSQL Exporter
# docker-compose.yml for postgres_exporter
version: '3.8'
services:
  postgres-exporter:
    image: prometheuscommunity/postgres-exporter
    environment:
      DATA_SOURCE_NAME: "postgresql://user:password@postgres:5432/dbname?sslmode=disable"
    ports:
      - "9187:9187"
    command:
      - '--collector.stat_statements'
      - '--collector.stat_database'
      - '--collector.replication'

PostgreSQL 的关键指标：

  • pg_up：数据库可达性
  • pg_stat_database_tup_returned：读取的行数
  • pg_stat_database_tup_inserted：插入的行数
  • pg_stat_database_deadlocks：死锁数
  • pg_stat_replication_lag：复制延迟（秒）
  • pg_locks_count：活动锁数量
用于探测的 Blackbox Exporter
# blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      preferred_ip_protocol: "ip4"

  http_post_json:
    prober: http
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{"key":"value"}'
      valid_status_codes: [200, 201]

  tcp_connect:
    prober: tcp
    timeout: 5s

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

# Prometheus config for blackbox exporter
scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

最佳实践

指标命名约定

遵循 Prometheus 指标命名最佳实践：

# Format: <namespace>_<subsystem>_<metric>_<unit>

# Good examples
http_requests_total              # Counter
http_request_duration_seconds    # Histogram
database_connections_active      # Gauge
cache_hits_total                 # Counter
memory_usage_bytes              # Gauge

# Include unit suffixes
_seconds, _bytes, _total, _ratio, _percentage

# Avoid
RequestCount                     # Use snake_case
http_requests                    # Missing _total for counter
request_time                     # Missing unit (should be _seconds)
标签指南
# Good: Low cardinality labels
http_requests_total{method="GET", endpoint="/api/users", status="200"}

# Bad: High cardinality labels (avoid)
http_requests_total{user_id="12345", session_id="abc-def-ghi"}

# Good: Use bounded label values
http_requests_total{status_class="2xx"}

# Bad: Unbounded label values
http_requests_total{response_size="1234567"}
记录规则模式
groups:
  - name: performance_rules
    interval: 30s
    rules:
      # Pre-aggregate expensive queries
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      # Namespace aggregations
      - record: namespace:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (namespace)

      # SLI calculations
      - record: job:http_requests_success:rate5m
        expr: sum(rate(http_requests_total{status=~"2.."}[5m])) by (job)

      - record: job:http_requests_error_rate:ratio
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)
告警设计原则
  1. 针对症状而非原因发出告警：针对用户可感知的问题发出告警
  2. 确保告警可执行：包含运行手册链接
  3. 使用适当的严重级别：严重、警告、信息
  4. 设置合理的阈值：基于历史数据
  5. 在注释中包含上下文：帮助值班工程师
  6. 对相关告警进行分组：减少告警疲劳
  7. 使用抑制规则：抑制冗余告警
  8. 测试告警规则：确认它们会在预期情况下触发
仪表盘最佳实践
  1. 为每类受众设置一个仪表盘：SRE、开发人员、业务人员
  2. 使用一致的时间范围：便于比较
  3. 包含 SLI/SLO 指标：展示业务影响
  4. 为部署添加注释：将变更与指标关联起来
  5. 使用模板变量：让仪表盘可复用
  6. 展示趋势和聚合数据：不只展示原始指标
  7. 包含运行手册链接：便于快速响应
  8. 使用适当的可视化方式：图表、仪表、表格
高可用设置
# Prometheus HA with Thanos
# Deploy multiple Prometheus instances with same config
# Use Thanos to deduplicate and provide global view

# prometheus-1.yml
global:
  external_labels:
    cluster: 'prod'
    replica: '1'

# prometheus-2.yml
global:
  external_labels:
    cluster: 'prod'
    replica: '2'

# Thanos sidecar configuration
# Uploads blocks to object storage
# Provides StoreAPI for querying
容量规划查询
# Disk space exhaustion prediction
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0

# Memory growth trend
predict_linear(node_memory_MemAvailable_bytes[1h], 24 * 3600)

# Request rate growth
predict_linear(sum(rate(http_requests_total[1h]))[24h:1h], 7 * 24 * 3600)

# Storage capacity planning
prometheus_tsdb_storage_blocks_bytes / (30 * 24 * 3600)

高级模式

用于多集群监控的联邦
# Global Prometheus federating from cluster Prometheus instances
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'  # All recording rules
    static_configs:
      - targets:
          - 'prometheus-us-west:9090'
          - 'prometheus-us-east:9090'
          - 'prometheus-eu-central:9090'
成本监控模式
# Track cloud costs with custom metrics
groups:
  - name: cost_tracking
    rules:
      - record: cloud:cost:hourly_rate
        expr: |
          (
            sum(kube_pod_container_resource_requests{resource="cpu"}) * 0.03  # CPU cost/hour
            +
            sum(kube_pod_container_resource_requests{resource="memory"} / 1024 / 1024 / 1024) * 0.005  # Memory cost/hour
          )

      - record: cloud:cost:monthly_estimate
        expr: cloud:cost:hourly_rate * 730  # Hours in average month
自定义 SLO 实现
# SLO: 99.9% availability for API
groups:
  - name: api_slo
    interval: 30s
    rules:
      # Success rate SLI
      - record: api:sli:success_rate
        expr: |
          sum(rate(http_requests_total{job="api",status=~"2.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))

      # Error budget remaining (30 days)
      - record: api:error_budget:remaining
        expr: |
          1 - (
            (1 - api:sli:success_rate)
            /
            (1 - 0.999)
          )

      # Latency SLI (p99 < 500ms)
      - record: api:sli:latency_success_rate
        expr: |
          (
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le)
            ) < 0.5
          )

示例摘要

此技能包含 20 多个全面的示例，涵盖：

  1. Prometheus 配置（基本配置、Kubernetes SD、存储）
  2. PromQL 查询（瞬时向量、范围向量、聚合）
  3. 数学运算与高级模式
  4. 告警规则定义（错误率、延迟、资源使用情况）
  5. Alertmanager 配置（路由、接收器、抑制）
  6. 多窗口多燃烧率 SLO 告警
  7. Grafana 仪表板 JSON（完整仪表板、RED 方法、USE 方法）
  8. 自定义导出器（Python、Go）
  9. 第三方导出器（PostgreSQL、Blackbox）
  10. 用于提升性能的记录规则
  11. 用于多集群监控的联邦
  12. 成本监控与 SLO 实施
  13. 高可用模式
  14. 容量规划查询

技能版本：1.0.0 上次更新：2025 年 10 月 技能类别：可观测性、监控、SRE、DevOps 兼容：Prometheus、Grafana、Alertmanager、OpenTelemetry、Kubernetes

评论

支持 Markdown