mcpField
S
slo-implementation

slo-implementation

github.com/ormus-solutions云与 DevOps12377 次安装4 周前更新
代码仓库

关于 slo-implementation

面向可靠性与运维团队的 SLI、SLO 和错误预算实施指南，提供指标示例、Prometheus 规则、告警、仪表板及定期审查方法。

主要功能

  1. 01SLI 和 SLO 定义模式
  2. 02错误预算计算与策略
  3. 03Prometheus 记录和告警规则
  4. 04多窗口消耗速率告警
  5. 05SLO 仪表板与审查周期

适用场景

  1. 01设定可衡量的服务可靠性目标
  2. 02跟踪可用性和延迟是否达标
  3. 03针对错误预算消耗触发告警
  4. 04审查可靠性趋势与事故

安装

复制下面的命令，在终端中运行。

$npx skills add https://github.com/ormus-solutions/libreuiux-claude-code --skill slo-implementation

安装后在 Claude Code 中输入/slo-implementation确认已加载。

详细介绍

SLO 实施

用于定义和实施服务级别指标（SLI）、服务级别目标（SLO）和错误预算的框架。

目的

使用 SLI、SLO 和错误预算设定可衡量的可靠性目标，在可靠性与创新速度之间取得平衡。

适用场景

  • 定义服务可靠性目标
  • 衡量用户感知的可靠性
  • 实施错误预算
  • 创建基于 SLO 的告警
  • 跟踪可靠性目标

SLI/SLO/SLA 层级

SLA (Service Level Agreement)
  ↓ Contract with customers
SLO (Service Level Objective)
  ↓ Internal reliability target
SLI (Service Level Indicator)
  ↓ Actual measurement

定义 SLI

常见的 SLI 类型
1. 可用性 SLI
# Successful requests / Total requests
sum(rate(http_requests_total{status!~"5.."}[28d]))
/
sum(rate(http_requests_total[28d]))
2. 延迟 SLI
# Requests below latency threshold / Total requests
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
/
sum(rate(http_request_duration_seconds_count[28d]))
3. 持久性 SLI
# Successful writes / Total writes
sum(storage_writes_successful_total)
/
sum(storage_writes_total)

**参考：**请参阅 references/slo-definitions.md

设定 SLO 目标

可用性 SLO 示例
SLO %每月停机时间每年停机时间
99%7.2 hours3.65 days
99.9%43.2 minutes8.76 hours
99.95%21.6 minutes4.38 hours
99.99%4.32 minutes52.56 minutes
选择合适的 SLO

请考虑：

  • 用户期望
  • 业务需求
  • 当前性能
  • 可靠性成本
  • 竞争对手基准

SLO 示例：

slos:
  - name: api_availability
    target: 99.9
    window: 28d
    sli: |
      sum(rate(http_requests_total{status!~"5.."}[28d]))
      /
      sum(rate(http_requests_total[28d]))

  - name: api_latency_p95
    target: 99
    window: 28d
    sli: |
      sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
      /
      sum(rate(http_request_duration_seconds_count[28d]))

错误预算计算

错误预算公式
Error Budget = 1 - SLO Target

示例：

  • SLO：99.9% 可用性
  • 错误预算：0.1% = 每月 43.2 分钟
  • 当前错误：0.05% = 每月 21.6 分钟
  • 剩余预算：50%
错误预算策略
error_budget_policy:
  - remaining_budget: 100%
    action: Normal development velocity
  - remaining_budget: 50%
    action: Consider postponing risky changes
  - remaining_budget: 10%
    action: Freeze non-critical changes
  - remaining_budget: 0%
    action: Feature freeze, focus on reliability

**参考：**请参阅 references/error-budget.md

SLO 实施

Prometheus 记录规则
# SLI Recording Rules
groups:
  - name: sli_rules
    interval: 30s
    rules:
      # Availability SLI
      - record: sli:http_availability:ratio
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[28d]))
          /
          sum(rate(http_requests_total[28d]))

      # Latency SLI (requests < 500ms)
      - record: sli:http_latency:ratio
        expr: |
          sum(rate(http_request_duration_seconds_bucket{le="0.5"}[28d]))
          /
          sum(rate(http_request_duration_seconds_count[28d]))

  - name: slo_rules
    interval: 5m
    rules:
      # SLO compliance (1 = meeting SLO, 0 = violating)
      - record: slo:http_availability:compliance
        expr: sli:http_availability:ratio >= bool 0.999

      - record: slo:http_latency:compliance
        expr: sli:http_latency:ratio >= bool 0.99

      # Error budget remaining (percentage)
      - record: slo:http_availability:error_budget_remaining
        expr: |
          (sli:http_availability:ratio - 0.999) / (1 - 0.999) * 100

      # Error budget burn rate
      - record: slo:http_availability:burn_rate_5m
        expr: |
          (1 - (
            sum(rate(http_requests_total{status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          )) / (1 - 0.999)
SLO 告警规则
groups:
  - name: slo_alerts
    interval: 1m
    rules:
      # Fast burn: 14.4x rate, 1 hour window
      # Consumes 2% error budget in 1 hour
      - alert: SLOErrorBudgetBurnFast
        expr: |
          slo:http_availability:burn_rate_1h > 14.4
          and
          slo:http_availability:burn_rate_5m > 14.4
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Fast error budget burn detected"
          description: "Error budget burning at {{ $value }}x rate"

      # Slow burn: 6x rate, 6 hour window
      # Consumes 5% error budget in 6 hours
      - alert: SLOErrorBudgetBurnSlow
        expr: |
          slo:http_availability:burn_rate_6h > 6
          and
          slo:http_availability:burn_rate_30m > 6
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Slow error budget burn detected"
          description: "Error budget burning at {{ $value }}x rate"

      # Error budget exhausted
      - alert: SLOErrorBudgetExhausted
        expr: slo:http_availability:error_budget_remaining < 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SLO error budget exhausted"
          description: "Error budget remaining: {{ $value }}%"

SLO 仪表板

Grafana 仪表板结构：

┌────────────────────────────────────┐
│ SLO Compliance (Current)           │
│ ✓ 99.95% (Target: 99.9%)          │
├────────────────────────────────────┤
│ Error Budget Remaining: 65%        │
│ ████████░░ 65%                     │
├────────────────────────────────────┤
│ SLI Trend (28 days)                │
│ [Time series graph]                │
├────────────────────────────────────┤
│ Burn Rate Analysis                 │
│ [Burn rate by time window]         │
└────────────────────────────────────┘

查询示例：

# Current SLO compliance
sli:http_availability:ratio * 100

# Error budget remaining
slo:http_availability:error_budget_remaining

# Days until error budget exhausted (at current burn rate)
(slo:http_availability:error_budget_remaining / 100)
*
28
/
(1 - sli:http_availability:ratio) * (1 - 0.999)

多窗口燃烧率告警

# Combination of short and long windows reduces false positives
rules:
  - alert: SLOBurnRateHigh
    expr: |
      (
        slo:http_availability:burn_rate_1h > 14.4
        and
        slo:http_availability:burn_rate_5m > 14.4
      )
      or
      (
        slo:http_availability:burn_rate_6h > 6
        and
        slo:http_availability:burn_rate_30m > 6
      )
    labels:
      severity: critical

SLO 审查流程

每周审查
  • 当前 SLO 达标情况
  • 错误预算状态
  • 趋势分析
  • 事故影响
每月审查
  • SLO 达成情况
  • 错误预算使用情况
  • 事故复盘
  • SLO 调整
每季度审查
  • SLO 相关性
  • 目标调整
  • 流程改进
  • 工具增强

最佳实践

  1. 从面向用户的服务入手
  2. 使用多个 SLI（可用性、延迟等）
  3. 设定可实现的 SLO（不要以 100% 为目标）
  4. 实施多窗口告警以减少噪声
  5. 持续跟踪错误预算
  6. 定期审查 SLO
  7. 记录 SLO 决策
  8. 与业务目标保持一致
  9. 自动化 SLO 报告
  10. 使用 SLO 进行优先级排序

参考文件

  • assets/slo-template.md - SLO 定义模板
  • references/slo-definitions.md - SLO 定义模式
  • references/error-budget.md - 错误预算计算

相关技能

  • prometheus-configuration - 用于指标收集
  • grafana-dashboards - 用于 SLO 可视化

评论

支持 Markdown