mcpField
S
slo-implementation

slo-implementation

github.com/hermeticormus云与 DevOps12376 次安装4 周前更新
代码仓库

关于 slo-implementation

slo-implementation 是帮助团队制定和落实服务可靠性目标的框架，将 SLI、SLO 与错误预算连接起来，并涵盖 Prometheus 规则、告警、仪表板和定期评审。

主要功能

  1. 01定义可用性、延迟和持久性 SLI
  2. 02设定 SLO 目标与错误预算
  3. 03编写 Prometheus 记录和告警规则
  4. 04规划多窗口消耗速率告警
  5. 05组织 SLO 仪表板与评审

适用场景

  1. 01制定可衡量的服务可靠性目标
  2. 02跟踪剩余错误预算
  3. 03在预算快速消耗时触发告警
  4. 04评审可靠性趋势与事故

安装

复制下面的命令，在终端中运行。

$npx skills add https://github.com/hermeticormus/libreuiux-claude-code --skill slo-implementation

安装后在 Claude Code 中输入/slo-implementation确认已加载。

详细介绍

SLO 实施

用于定义和实施服务级别指标（SLI）、服务级别目标（SLO）和错误预算的框架。

目的

使用 SLI、SLO 和错误预算设定可衡量的可靠性目标，在可靠性与创新速度之间取得平衡。

适用场景

  • 定义服务可靠性目标
  • 衡量用户感知的可靠性
  • 实施错误预算
  • 创建基于 SLO 的告警
  • 跟踪可靠性目标

SLI/SLO/SLA 层级

SLA (Service Level Agreement)
  ↓ Contract with customers
SLO (Service Level Objective)
  ↓ Internal reliability target
SLI (Service Level Indicator)
  ↓ Actual measurement

定义 SLI

常见的 SLI 类型
1. 可用性 SLI
# Successful requests / Total requests
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
2. 延迟 SLI
# Requests below latency threshold / Total requests
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
3. 持久性 SLI
# Successful writes / Total writes
sum(storage_writes_successful_total)
/
sum(storage_writes_total)

**参考：**请参阅 references/slo-definitions.md

设定 SLO 目标

可用性 SLO 示例
SLO %每月停机时间每年停机时间
99%7.2 hours3.65 days
99.9%43.2 minutes8.76 hours
99.95%21.6 minutes4.38 hours
99.99%4.32 minutes52.56 minutes
选择合适的 SLO

请考虑：

  • 用户期望
  • 业务需求
  • 当前性能
  • 可靠性成本
  • 竞争对手基准

SLO 示例：

slos:
  - name: api_availability
    target: 99.9
    window: 28d
    sli: |
      sum(rate(http_requests_total{status!~"5.."}[28d]))
      /
      sum(rate(http_requests_total[28d]))

  - name: api_latency_p95
    target: 99
    window: 28d
    sli: |
      sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
      /
      sum(rate(http_request_duration_seconds_count[28d]))

错误预算计算

错误预算公式
Error Budget = 1 - SLO Target

示例：

  • SLO：99.9% 可用性
  • 错误预算：0.1% = 每月 43.2 分钟
  • 当前错误：0.05% = 每月 21.6 分钟
  • 剩余预算：50%
错误预算策略
error_budget_policy:
  - remaining_budget: 100%
    action: Normal development velocity
  - remaining_budget: 50%
    action: Consider postponing risky changes
  - remaining_budget: 10%
    action: Freeze non-critical changes
  - remaining_budget: 0%
    action: Feature freeze, focus on reliability

**参考：**请参阅 references/error-budget.md

SLO 实施

Prometheus 记录规则
# SLI Recording Rules
groups:
  - name: sli_rules
    interval: 30s
    rules:
      # Availability SLI
      - record: sli:http_availability:ratio
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[28d]))
          /
          sum(rate(http_requests_total[28d]))

      # Latency SLI (requests < 500ms)
      - record: sli:http_latency:ratio
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
          /
          sum(rate(http_request_duration_seconds_count[28d]))

  - name: slo_rules
    interval: 5m
    rules:
      # SLO compliance (1 = meeting SLO, 0 = violating)
      - record: slo:http_availability:compliance
        expr: sli:http_availability:ratio >= bool 0.999

      - record: slo:http_latency:compliance
        expr: sli:http_latency:ratio >= bool 0.99

      # Error budget remaining (percentage)
      - record: slo:http_availability:error_budget_remaining
        expr: |
          (sli:http_availability:ratio - 0.999) / (1 - 0.999) * 100

      # Error budget burn rate
      - record: slo:http_availability:burn_rate_5m
        expr: |
          (1 - (
            sum(rate(http_requests_total{status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          )) / (1 - 0.999)
SLO 告警规则
groups:
  - name: slo_alerts
    interval: 1m
    rules:
      # Fast burn: 14.4x rate, 1 hour window
      # Consumes 2% error budget in 1 hour
      - alert: SLOErrorBudgetBurnFast
        expr: |
          slo:http_availability:burn_rate_1h > 14.4
          and
          slo:http_availability:burn_rate_5m > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn detected"
          description: "Error budget burning at {{ $value }}x rate"

      # Slow burn: 6x rate, 6 hour window
      # Consumes 5% error budget in 6 hours
      - alert: SLOErrorBudgetBurnSlow
        expr: |
          slo:http_availability:burn_rate_6h > 6
          and
          slo:http_availability:burn_rate_30m > 6
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow error budget burn detected"
          description: "Error budget burning at {{ $value }}x rate"

      # Error budget exhausted
      - alert: SLOErrorBudgetExhausted
        expr: slo:http_availability:error_budget_remaining < 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SLO error budget exhausted"
          description: "Error budget remaining: {{ $value }}%"

SLO 仪表板

Grafana 仪表板结构：

┌────────────────────────────────────┐
│ SLO Compliance (Current)           │
│ ✓ 99.95% (Target: 99.9%)          │
├────────────────────────────────────┤
│ Error Budget Remaining: 65%        │
│ ████████░░ 65%                     │
├────────────────────────────────────┤
│ SLI Trend (28 days)                │
│ [Time series graph]                │
├────────────────────────────────────┤
│ Burn Rate Analysis                 │
│ [Burn rate by time window]         │
└────────────────────────────────────┘

查询示例：

# Current SLO compliance
sli:http_availability:ratio * 100

# Error budget remaining
slo:http_availability:error_budget_remaining

# Days until error budget exhausted (at current burn rate)
(slo:http_availability:error_budget_remaining / 100)
*
28
/
(1 - sli:http_availability:ratio) * (1 - 0.999)

多窗口燃烧率告警

# Combination of short and long windows reduces false positives
rules:
  - alert: SLOBurnRateHigh
    expr: |
      (
        slo:http_availability:burn_rate_1h > 14.4
        and
        slo:http_availability:burn_rate_5m > 14.4
      )
      or
      (
        slo:http_availability:burn_rate_6h > 6
        and
        slo:http_availability:burn_rate_30m > 6
      )
    labels:
      severity: critical

SLO 审查流程

每周审查
  • 当前 SLO 达标情况
  • 错误预算状态
  • 趋势分析
  • 事故影响
每月审查
  • SLO 达成情况
  • 错误预算使用情况
  • 事故复盘
  • SLO 调整
每季度审查
  • SLO 相关性
  • 目标调整
  • 流程改进
  • 工具增强

最佳实践

  1. 从面向用户的服务入手
  2. 使用多个 SLI（可用性、延迟等）
  3. 设定可实现的 SLO（不要以 100% 为目标）
  4. 实施多窗口告警以减少噪声
  5. 持续跟踪错误预算
  6. 定期审查 SLO
  7. 记录 SLO 决策
  8. 与业务目标保持一致
  9. 自动化 SLO 报告
  10. 使用 SLO 进行优先级排序

参考文件

  • assets/slo-template.md - SLO 定义模板
  • references/slo-definitions.md - SLO 定义模式
  • references/error-budget.md - 错误预算计算

相关技能

  • prometheus-configuration - 用于指标收集
  • grafana-dashboards - 用于 SLO 可视化

评论

支持 Markdown