aws-cloudwatch-investigation

AWS CloudWatch 故障排查技能

使用 CloudWatch Logs、Metrics 和 Alarms 排查生产事故的可复用模式。这些模式设计为在事故分诊期间可组合使用。

---

模式 1：Logs Insights 查询模板

错误激增检测

按错误类型分组，查找时间窗口内的主要错误：

fields @timestamp, @message, @logStream
| filter @message like /(?i)(error|exception|fatal|critical)/
| stats count(*) as errorCount by bin(5m), @logStream
| sort errorCount desc
| limit 20

按操作划分的 P99 延迟分解

识别哪些操作导致了延迟激增：

fields @timestamp, @duration, operation
| filter ispresent(@duration)
| stats avg(@duration) as avgMs,
        pct(@duration, 50) as p50Ms,
        pct(@duration, 95) as p95Ms,
        pct(@duration, 99) as p99Ms,
        count(*) as invocations
  by operation
| sort p99Ms desc
| limit 15

Lambda 冷启动检测

量化事故期间冷启动的影响：

fields @timestamp, @duration, @initDuration, @memorySize, @maxMemoryUsed
| filter ispresent(@initDuration)
| stats count(*) as coldStarts,
        avg(@initDuration) as avgInitMs,
        max(@initDuration) as maxInitMs,
        avg(@duration) as avgDurationMs
  by bin(5m)
| sort @timestamp desc

内存溢出 (OOM) 检测

查找因内存压力而被终止的 Lambda 函数或容器：

fields @timestamp, @message, @logStream, @memorySize, @maxMemoryUsed
| filter @message like /Runtime exited|out of memory|OOMKilled|Cannot allocate memory|MemoryError/
| stats count(*) as oomEvents by @logStream, bin(10m)
| sort oomEvents desc
| limit 10

如需了解 OOM 前的内存使用趋势：

fields @timestamp, @maxMemoryUsed, @memorySize
| filter ispresent(@maxMemoryUsed)
| stats max(@maxMemoryUsed / @memorySize * 100) as peakMemPct,
        avg(@maxMemoryUsed / @memorySize * 100) as avgMemPct
  by bin(5m)
| sort @timestamp desc

超时检测

查找达到配置超时时间的调用：

fields @timestamp, @duration, @logStream, @requestId
| filter @message like /Task timed out/ or @duration > 28000
| stats count(*) as timeouts by @logStream, bin(5m)
| sort timeouts desc

---

模式 2：告警历史与部署事件关联

流程

1. 获取告警状态转换时间——记录告警进入 ALARM 状态的确切时间戳。
2. 查询 CloudTrail，查找 [alarm_time - 30min, alarm_time] 时间窗口内与部署相关的事件：

# CloudTrail Lake query for deployment events
SELECT eventTime, eventName, userIdentity.arn, requestParameters
FROM <event-data-store-id>
WHERE eventTime > '<alarm_time_minus_30m>'
  AND eventTime < '<alarm_time>'
  AND eventName IN (
    'UpdateFunctionCode', 'UpdateFunctionConfiguration',
    'UpdateService', 'CreateDeployment', 'RegisterTaskDefinition',
    'CreateChangeSet', 'ExecuteChangeSet',
    'StartPipelineExecution', 'PutImage'
  )
ORDER BY eventTime DESC

3. 关联标准——满足以下条件时，部署被视为“已关联”：

   • 其目标服务/资源与告警相同
   • 其在告警状态转换前 15 分钟内完成
   • 部署者身份与 CI/CD 角色匹配（而非人工应用热修复）

4. 增强关联性：

   • 检查同一告警在上一个部署周期是否处于健康状态
   • 验证同一时间窗口内没有其他环境变更（扩缩容事件、配置变更）
   • 查找同时开始出现的金丝雀/合成监控失败

输出格式

Deploy Correlation:
  Event: UpdateFunctionCode
  Time: 2024-03-15T14:23:07Z (12 min before alarm)
  Actor: arn:aws:sts::123456789012:assumed-role/github-actions-deploy/session
  Resource: arn:aws:lambda:us-east-1:123456789012:function:payment-processor
  Correlation: STRONG — same resource, CI/CD actor, alarm was OK prior cycle

---

模式 3：缩小影响范围的决策树

使用此决策树，从最宽泛到最具体，系统性地界定事故范围：

START
  |
  v
[1] ACCOUNT — Which account(s) show the alarm?
  |  - Check: Are alarms firing in multiple accounts?
  |  - If yes → suspect shared service (SSO, networking, shared deployment pipeline)
  |  - If no → proceed to Region
  v
[2] REGION — Which region(s) are affected?
  |  - Check: Same alarm in other regions?
  |  - If multi-region → suspect global service (IAM, Route53, S3 global)
  |  - If single-region → proceed to Service
  v
[3] SERVICE — Which service namespace shows degradation?
  |  - Check CloudWatch namespace: AWS/Lambda, AWS/ECS, AWS/ApiGateway, etc.
  |  - If multiple services → suspect shared dependency (VPC, NAT, DNS, IAM)
  |  - If single service → proceed to Operation
  v
[4] OPERATION — Which API action or function is failing?
  |  - For Lambda: which function name?
  |  - For ECS: which service/task definition?
  |  - For API GW: which stage/resource/method?
  |  - If all operations → suspect service-level issue (throttling, quota)
  |  - If specific operation → proceed to Resource
  v
[5] RESOURCE — Which specific resource instance?
     - Function ARN, Task ID, DB instance identifier
     - This is your investigation target
     - Proceed to log and trace analysis scoped to this resource

共享依赖排查

当影响范围横跨多个服务时，按以下顺序排查：

1. VPC/网络——NAT Gateway ErrorPortAllocation、丢包、DNS 解析失败
2. IAM/STS——AssumeRole 上的 ThrottlingException、令牌颁发延迟
3. 下游依赖——共享数据库、缓存或外部 API
4. 部署流水线——同一流水线运行中跨服务的同步部署
5. AWS 服务事件——检查该区域的 AWS Health Dashboard 和 Service Health

---

模式 4：PromQL 风格的指标查询模式

这些模式使用 CloudWatch 指标数学和 GetMetricData 构建复合信号。将其表达为指标查询，用于仪表盘或程序化检索。

错误率百分比

MetricDataQueries:
  - Id: errors
    MetricStat:
      Metric:
        Namespace: AWS/Lambda
        MetricName: Errors
        Dimensions: [{Name: FunctionName, Value: TARGET}]
      Period: 60
      Stat: Sum
  - Id: invocations
    MetricStat:
      Metric:
        Namespace: AWS/Lambda
        MetricName: Invocations
        Dimensions: [{Name: FunctionName, Value: TARGET}]
      Period: 60
      Stat: Sum
  - Id: error_rate
    Expression: "errors / invocations * 100"
    Label: "Error Rate %"

延迟异常检测（与基线对比）

MetricDataQueries:
  - Id: current_p99
    MetricStat:
      Metric:
        Namespace: AWS/Lambda
        MetricName: Duration
        Dimensions: [{Name: FunctionName, Value: TARGET}]
      Period: 300
      Stat: p99
  - Id: baseline_p99
    MetricStat:
      Metric:
        Namespace: AWS/Lambda
        MetricName: Duration
        Dimensions: [{Name: FunctionName, Value: TARGET}]
      Period: 300
      Stat: p99
    # Use StartTime/EndTime set to same window last week
  - Id: anomaly_ratio
    Expression: "current_p99 / baseline_p99"
    Label: "Latency vs Baseline (ratio > 2 = anomaly)"

限流压力评分

将多个限流信号合并为单一压力指标：

MetricDataQueries:
  - Id: lambda_throttles
    MetricStat:
      Metric: {Namespace: AWS/Lambda, MetricName: Throttles}
      Period: 60
      Stat: Sum
  - Id: api_gw_429s
    MetricStat:
      Metric: {Namespace: AWS/ApiGateway, MetricName: 4XXError, Dimensions: [{Name: ApiName, Value: TARGET}]}
      Period: 60
      Stat: Sum
  - Id: dynamo_throttles
    MetricStat:
      Metric: {Namespace: AWS/DynamoDB, MetricName: ThrottledRequests, Dimensions: [{Name: TableName, Value: TARGET}]}
      Period: 60
      Stat: Sum
  - Id: throttle_pressure
    Expression: "lambda_throttles + api_gw_429s + dynamo_throttles"
    Label: "Combined Throttle Pressure"

并发执行余量

MetricDataQueries:
  - Id: concurrent
    MetricStat:
      Metric: {Namespace: AWS/Lambda, MetricName: ConcurrentExecutions}
      Period: 60
      Stat: Maximum
  - Id: headroom
    Expression: "1000 - concurrent"
    Label: "Remaining Concurrency (account limit 1000)"

---

模式 5：事故时间线重建

流程

通过合并多个来源的数据，重建精确的时间线：

1. 收集时间戳：

来源	查询	产出
CloudWatch Alarms	告警历史 API	状态转换时间
CloudWatch Metrics	使用 1 分钟周期的 GetMetricData	首个异常点
CloudWatch Logs	使用 earliest(@timestamp) 的 Logs Insights	首次错误发生
CloudTrail	按时间过滤的 LookupEvents	部署/变更事件
AWS Health	DescribeEvents	AWS 侧事故

2. 构建时间线：

fields @timestamp, @message
| filter @message like /ERROR|WARN|timeout|refused|denied/
| stats earliest(@timestamp) as firstSeen, latest(@timestamp) as lastSeen, count(*) as occurrences
  by @message
| sort firstSeen asc
| limit 20

3. 识别序列：

Timeline:
  T-15m: CloudTrail — UpdateFunctionCode by CI/CD role
  T-12m: Logs — first error "Connection refused to payments-api.internal"
  T-10m: Metrics — Error count crosses 5/min threshold
  T-8m:  Alarm — PaymentProcessorErrors enters ALARM
  T-5m:  Metrics — p99 latency spikes to 28s (timeout)
  T-0:   Current — error rate at 45%, alarm still firing

4. 确定根因事件——先于所有症状出现的最早变更。从第一个症状开始向前回溯，直至最近的变更（部署、配置变更、扩缩容事件或外部依赖变化）。

注意事项

• CloudWatch 指标时间戳为周期结束时间。14:05 的 1 分钟数据点覆盖 14:04-14:05。
• CloudTrail 事件最多可能有 15 分钟的投递延迟。请使用 eventTime，而非摄取时间。
• 日志组时间戳取决于 Agent/SDK 的刷新间隔。请预留 30-60 秒的时钟偏差。
• 告警状态变更存在内置的评估延迟（周期数 x 评估周期数）。实际异常发生得更早。