observability-logging

可观测性日志记录

将日志作为全面可观测性策略的核心组成部分，并与指标、追踪、告警和仪表板集成，以深入了解系统并实现卓越运维。

何时使用我

在以下情况下使用此技能：

• 构建全面的可观测性平台
• 将日志与指标和追踪集成，实现完整可观测性
• 设计基于日志模式的告警和监控系统
• 创建结合日志洞察与其他遥测数据的仪表板
• 使用日志数据实施 SLO/SLA 监控
• 构建基于日志分析的事件响应工作流
• 建立卓越运维实践
• 设计值班流程和操作手册
• 使用日志模式实施预测性维护
• 基于可观测性信号构建自愈系统

我能做什么

1. 日志驱动的指标

• 从日志中提取指标（错误率、延迟百分位数、吞吐量）
• 根据日志模式和相关性创建派生指标
• 针对业务和运维事件实施基于日志的计数器
• 根据日志数据计算服务级别指标（SLI）
• 使用日志派生指标监控服务级别目标（SLO）
• 针对错误预算消耗实施燃尽速率告警
• 根据历史日志模式创建趋势分析

2. 日志增强的追踪

• 为追踪补充日志上下文，以获得更深入的洞察
• 将追踪跨度与日志事件关联，全面了解请求
• 为旧系统或未追踪的系统实施基于日志的跨度创建
• 使用日志填补分布式系统中的追踪缺口
• 创建结合日志和追踪的统一可观测性视图
• 实施日志到追踪的链接，便于无缝调查
• 在日志中使用追踪上下文进行关联和分析

3. 告警与监控

• 针对关键模式和异常设计基于日志的告警
• 对不同日志源实施告警去重和关联
• 根据日志模式的严重程度创建升级策略
• 设计告警路由，将告警发送给相应团队和人员
• 通过日志上下文丰富告警，加快诊断
• 针对已知问题和维护时段创建抑制规则
• 监控告警效果，并根据历史数据调整阈值

4. 仪表板与可视化

• 创建结合日志、指标和追踪的运维仪表板
• 设计包含日志派生健康指标的服务健康仪表板
• 实现实时日志流可视化
• 创建展示日志模式演变的趋势仪表板
• 设计包含关联数据的事件调查仪表板
• 为不同利益相关方的需求实现可自定义视图
• 使用机器学习分析日志数据，创建预测型仪表板

5. 事件响应

• 为常见问题设计日志驱动的操作手册
• 根据日志模式实施自动修复
• 根据日志关联创建事件时间线
• 使用全面的日志数据设计事后分析
• 利用可观测性数据开展无责回顾
• 根据已解决事件和日志模式创建知识库
• 根据可观测性信号设计升级流程

可观测性支柱集成

日志 + 指标 + 追踪 = 完整可观测性

示例：API 服务可观测性

Logs (What happened):
- "API call to /api/users failed with 500 error"
- "Database connection timeout after 5000ms"
- "Cache miss for user:123"

Metrics (How much/how often):
- Error rate: 5.2%
- P95 latency: 245ms
- Throughput: 1,234 requests/second

Traces (Where in the flow):
- Request flow: API Gateway → Auth Service → User Service → Database
- Time spent: 45ms in Auth, 120ms in User Service, 80ms in Database
- Bottleneck identified: Database query in User Service

统一数据模型

observability_data:
  logs:
    source: application, infrastructure, audit
    format: structured (JSON)
    fields: [timestamp, level, service, message, context]
    
  metrics:
    source: logs (derived), application (direct), infrastructure
    types: counter, gauge, histogram, summary
    dimensions: [service, endpoint, status_code, user_type]
    
  traces:
    source: instrumentation, log-derived
    context: trace_id, span_id, parent_span_id
    attributes: [service.name, operation.name, duration, status]
    
  correlations:
    log_to_metric: "error logs → error rate metric"
    log_to_trace: "trace_id field links logs to traces"
    metric_to_trace: "high latency metric → trace analysis"

日志驱动的 SLO 监控

根据日志计算错误预算

def calculate_error_budget_from_logs(logs, slo_target, time_window):
    """
    Calculate error budget consumption from log data
    
    Args:
        logs: List of log entries with timestamp and success status
        slo_target: SLO target (e.g., 0.999 for 99.9%)
        time_window: Time window for calculation in seconds
    
    Returns:
        error_budget_consumption: Percentage of error budget consumed
    """
    total_requests = len(logs)
    successful_requests = sum(1 for log in logs if log.get('status') != 'error')
    
    success_rate = successful_requests / total_requests if total_requests > 0 else 1.0
    error_rate = 1.0 - success_rate
    
    # Calculate error budget consumption
    allowed_errors = (1.0 - slo_target) * total_requests
    actual_errors = total_requests - successful_requests
    error_budget_consumption = actual_errors / allowed_errors if allowed_errors > 0 else float('inf')
    
    return {
        'total_requests': total_requests,
        'successful_requests': successful_requests,
        'success_rate': success_rate,
        'error_rate': error_rate,
        'slo_target': slo_target,
        'allowed_errors': allowed_errors,
        'actual_errors': actual_errors,
        'error_budget_consumption': error_budget_consumption,
        'error_budget_remaining': max(0, 1.0 - error_budget_consumption)
    }

燃尽速率告警

alerting:
  error_budget_burn_rate:
    # Alert when burning error budget too quickly
    - name: "high_error_budget_burn_rate"
      condition: "error_budget_burn_rate > 10"
      # Burning 10x faster than allowed
      window: "1h"
      severity: "critical"
    
    - name: "medium_error_budget_burn_rate"
      condition: "error_budget_burn_rate > 2"
      # Burning 2x faster than allowed
      window: "6h"
      severity: "warning"
    
  slo_violation:
    - name: "slo_violation_imminent"
      condition: "error_budget_remaining < 0.1"
      # Less than 10% error budget remaining
      window: "7d"
      severity: "warning"
    
    - name: "slo_violation_occurred"
      condition: "success_rate < slo_target"
      # Actual violation occurring
      window: "1h"
      severity: "critical"

示例

# Extract metrics from logs for SLO monitoring
npm run observability:extract-metrics -- --slo-target 0.999 --window 7d --output slo-metrics.json

# Create unified observability dashboard
npm run observability:create-dashboard -- --services "api,auth,db" --data-sources "logs,metrics,traces"

# Design log-based alerting rules
npm run observability:design-alerts -- --patterns "error_rate > 5%,latency_p95 > 1000ms" --escalation-policy "team-rotation"

# Implement incident response workflow
npm run observability:incident-workflow -- --trigger "error_spike" --actions "page,create-incident,notify-slack"

# Correlate logs with metrics and traces
npm run observability:correlate -- --time-range "last-1h" --output correlation-analysis.json

输出格式

可观测性平台配置：

observability:
  data_sources:
    logs:
      collection: [filebeat, fluentd, otel-collector]
      processing: [parsing, enrichment, correlation]
      storage: [elasticsearch, s3]
      
    metrics:
      collection: [prometheus, otel-collector]
      processing: [aggregation, derivation]
      storage: [prometheus, thanos]
      
    traces:
      collection: [otel-collector, jaeger-agent]
      processing: [sampling, enrichment]
      storage: [jaeger, tempo]
  
  correlation:
    fields:
      trace_id: ["trace_id", "trace.id", "X-Trace-Id"]
      service: ["service", "service.name", "component"]
      user_id: ["user_id", "user.id", "userId"]
    
    rules:
      - when: "log.level == 'ERROR'"
        then: "increment_metric('errors_total', labels=log.labels)"
      - when: "trace.duration > 1000"
        then: "log.warning('slow_trace', trace_attributes)"
      - when: "metric.name == 'error_rate' and metric.value > 0.05"
        then: "create_alert('high_error_rate', severity='warning')"
  
  dashboards:
    - name: "Service Health"
      panels:
        - type: "timeseries"
          title: "Error Rate"
          query: "rate(error_logs_total[5m])"
        - type: "histogram"
          title: "Request Latency"
          query: "histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))"
        - type: "log_stream"
          title: "Recent Errors"
          query: "level:ERROR"
    
    - name: "Business Metrics"
      panels:
        - type: "counter"
          title: "User Signups"
          query: "log.message:'User signed up'"
        - type: "timeseries"
          title: "Payment Success Rate"
          query: "successful_payments / total_payments"
  
  alerting:
    rules:
      - alert: "HighErrorRate"
        expr: "rate(error_logs_total[5m]) > 0.05"
        for: "5m"
        labels:
          severity: "critical"
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} which is above 5% threshold"
        
      - alert: "SLOErrorBudgetBurn"
        expr: "error_budget_burn_rate > 10"
        for: "1h"
        labels:
          severity: "warning"
        annotations:
          summary: "Error budget burning too fast"
          description: "Error budget burn rate is {{ $value }}x faster than allowed"
  
  incident_response:
    workflows:
      - trigger: "alert.severity == 'critical'"
        actions:
          - "create_incident"
          - "page_on_call"
          - "notify_slack('#alerts')"
          - "start_zoom_war_room"
      
      - trigger: "incident.created"
        actions:
          - "gather_observability_data"
          - "correlate_logs_metrics_traces"
          - "suggest_runbooks"
          - "update_status_page"

可观测性成熟度评估：

Observability Maturity Assessment
────────────────────────────────
Organization: Example Corp
Assessment Date: 2026-02-26
Overall Score: 72/100

Pillar Scores:
- Logging: 85/100 (Structured, correlated, well-managed)
- Metrics: 65/100 (Basic metrics, limited derivation)
- Tracing: 56/100 (Partial implementation, gaps in coverage)
- Alerting: 70/100 (Effective but could be smarter)
- Visualization: 74/100 (Good dashboards, could be more unified)

Integration Assessment:
✅ Logs include trace context (trace_id, span_id)
✅ Metrics derived from logs (error rates, throughput)
⚠️  Traces not fully correlated with logs (60% coverage)
⚠️  Alerting not using derived SLO metrics
✅ Dashboards combine multiple data sources

Gap Analysis:
1. Missing: Unified observability data model
2. Missing: Automated correlation across pillars
3. Missing: Predictive analytics on observability data
4. Missing: Self-healing based on observability signals
5. Missing: Comprehensive SLO monitoring

Observability ROI Analysis:
- Current MTTR (Mean Time To Resolution): 45 minutes
- Target MTTR with improved observability: 15 minutes
- Estimated reduction in incident impact: $15,000/month
- Estimated improvement in developer productivity: 20%
- Estimated reduction in on-call burden: 30%

Recommendations:
1. HIGH PRIORITY: Implement unified observability data model
2. HIGH PRIORITY: Improve trace coverage and correlation
3. MEDIUM PRIORITY: Implement SLO-based alerting
4. MEDIUM PRIORITY: Add predictive analytics capabilities
5. LOW PRIORITY: Explore self-healing automation

Implementation Roadmap:
- Phase 1 (1 month): Unified data model and correlation
- Phase 2 (2 months): SLO monitoring and alerting
- Phase 3 (3 months): Predictive analytics
- Phase 4 (6 months): Self-healing capabilities
- Ongoing: Continuous improvement and optimization

注意事项

• 可观测性是一段旅程，而非终点——持续改进至关重要
• 从需要解答的问题入手——围绕这些问题设计可观测性
• 跨数据源关联比单个数据源的深度更有价值
• 考虑可观测性的成本——在价值与费用之间取得平衡
• 让所有利益相关方参与——开发人员、运维人员、业务团队和高管
• 衡量可观测性的成效——跟踪 MTTR、事件频率等指标
• 记录可观测性实践——操作手册、仪表板和告警定义
• 定期审查和改进——可观测性需求会随系统演进
• 平衡自动化与人工洞察——不要通过自动化排除必要的人工判断
• 考虑安全与合规要求——可观测性数据可能包含敏感信息