reliability-strategy-builder

可靠性策略构建器

通过妥善处理故障和制定 SLO，构建高韧性系统。

可靠性模式

1. 熔断器

通过停止向故障服务发送请求，防止故障级联。

class CircuitBreaker {
  private state: "closed" | "open" | "half-open" = "closed";
  private failureCount = 0;
  private lastFailureTime?: Date;

  async execute<T>(operation: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (this.shouldAttemptReset()) {
        this.state = "half-open";
      } else {
        throw new Error("Circuit breaker is OPEN");
      }
    }

    try {
      const result = await operation();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    this.failureCount = 0;
    this.state = "closed";
  }

  private onFailure() {
    this.failureCount++;
    this.lastFailureTime = new Date();

    if (this.failureCount >= 5) {
      this.state = "open";
    }
  }

  private shouldAttemptReset(): boolean {
    if (!this.lastFailureTime) return false;
    const now = Date.now();
    const elapsed = now - this.lastFailureTime.getTime();
    return elapsed > 60000; // 1 minute
  }
}

2. 带退避的重试

通过指数退避处理瞬时故障。

async function retryWithBackoff<T>(
  operation: () => Promise<T>,
  maxRetries = 3,
  baseDelay = 1000
): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxRetries) throw error;

      // Exponential backoff: 1s, 2s, 4s
      const delay = baseDelay * Math.pow(2, attempt);
      await sleep(delay);
    }
  }
  throw new Error("Max retries exceeded");
}

3. 降级模式

在主要功能失效时提供降级功能。

async function getUserWithFallback(userId: string): Promise<User> {
  try {
    // Try primary database
    return await primaryDb.users.findById(userId);
  } catch (error) {
    logger.warn("Primary DB failed, using cache");

    // Fallback to cache
    const cached = await cache.get(`user:${userId}`);
    if (cached) return cached;

    // Final fallback: return minimal user object
    return {
      id: userId,
      name: "Unknown User",
      email: "unavailable",
    };
  }
}

4. 舱壁模式

隔离故障，防止资源耗尽。

class ThreadPool {
  private pools = new Map<string, Semaphore>();

  constructor() {
    // Separate pools for different operations
    this.pools.set("critical", new Semaphore(100));
    this.pools.set("standard", new Semaphore(50));
    this.pools.set("background", new Semaphore(10));
  }

  async execute(priority: string, operation: () => Promise<any>) {
    const pool = this.pools.get(priority);
    await pool.acquire();

    try {
      return await operation();
    } finally {
      pool.release();
    }
  }
}

SLO 定义

SLO 模板

service: user-api
slos:
  - name: Availability
    description: API should be available for successful requests
    target: 99.9%
    measurement:
      type: ratio
      success: status_code < 500
      total: all_requests
    window: 30 days

  - name: Latency
    description: 95% of requests complete within 500ms
    target: 95%
    measurement:
      type: percentile
      metric: request_duration_ms
      threshold: 500
      percentile: 95
    window: 7 days

  - name: Error Rate
    description: Less than 1% of requests result in errors
    target: 99%
    measurement:
      type: ratio
      success: status_code < 400 OR status_code IN [401, 403, 404]
      total: all_requests
    window: 24 hours

错误预算

Error Budget = 100% - SLO

Example:
SLO: 99.9% availability
Error Budget: 0.1% = 43.2 minutes/month downtime allowed

故障模式分析

| Component   | Failure Mode | Impact | Probability | Detection               | Mitigation                     |
| ----------- | ------------ | ------ | ----------- | ----------------------- | ------------------------------ |
| Database    | Unresponsive | HIGH   | Medium      | Health checks every 10s | Circuit breaker, read replicas |
| API Gateway | Overload     | HIGH   | Low         | Request queue depth     | Rate limiting, auto-scaling    |
| Cache       | Eviction     | MEDIUM | High        | Cache hit rate          | Fallback to DB, larger cache   |
| Queue       | Backed up    | LOW    | Medium      | Queue depth metric      | Add workers, DLQ               |

可靠性检查清单

基础设施

• 配置带健康检查的负载均衡器
• 配置多个可用区
• 配置自动扩缩容
• 配置数据库复制
• 定期备份（并进行测试！）

应用程序

• 对外部调用使用熔断器
• 配置带退避的重试逻辑
• 对所有 I/O 设置超时
• 配置降级机制
• 实现优雅降级

监控

• SLO 仪表板
• 跟踪错误预算
• 在违反 SLO 时发出告警
• 延迟百分位数（p50、p95、p99）
• 依赖项健康检查

运维

• 事件响应操作手册
• 值班轮换
• 事后复盘模板
• 灾难恢复计划
• 混沌工程测试

事件响应计划

严重级别

SEV1 (Critical): Complete service outage, data loss
  - Response time: <15 minutes
  - Page on-call immediately

SEV2 (High): Partial outage, degraded performance
  - Response time: <1 hour
  - Alert on-call

SEV3 (Medium): Minor issues, workarounds available
  - Response time: <4 hours
  - Create ticket

SEV4 (Low): Cosmetic issues, no user impact
  - Response time: Next business day
  - Backlog

事件响应步骤

1. 确认：在 SLA 时限内确认收到
2. 评估：确定严重程度和影响
3. 沟通：更新状态页面
4. 缓解：止住问题（回滚、扩容、禁用）
5. 解决：修复根本原因
6. 记录：撰写事后复盘

最佳实践

1. 为故障而设计：假设组件会发生故障
2. 快速失败：避免缓慢故障引发级联
3. 隔离故障：采用舱壁模式
4. 优雅降级：减少功能，而不是崩溃
5. 监控 SLO：跟踪错误预算
6. 测试故障模式：进行混沌工程测试
7. 编写操作手册：明确事件响应流程

输出检查清单

• 已实现熔断器
• 已配置带退避的重试逻辑
• 已配置降级机制
• 已实现舱壁隔离
• 已定义 SLO（可用性、延迟、错误）
• 已计算错误预算
• 已完成故障模式分析
• 已配置监控仪表板
• 已制定事件响应计划
• 已编写操作手册