monitoring-observability

监控与可观测性

概述

本技能涵盖可观测性的三大支柱——追踪、指标和日志，以及告警、仪表板和健康检查模式。内容聚焦于作为厂商中立标准的 OpenTelemetry、便于查询的结构化日志、用于微服务调试的分布式追踪，以及基于 SLO 的告警，以减少噪声。

在为应用添加生产环境可视性、搭建监控基础设施、调试分布式系统、配置有意义的告警，或为运维和产品团队构建仪表板时，请使用本技能。

---

核心原则

1. 在边界处添加检测 - 追踪每次外部调用（HTTP、数据库、队列、缓存）。追踪内部函数会增加噪声；边界追踪能揭示系统行为。
2. 优先使用结构化日志 - 每条日志都必须是包含关联 ID、服务名称和上下文的 JSON。非结构化日志在大规模场景下无法搜索。
3. 针对症状而非原因告警 - 在用户受到影响时告警（错误率、延迟违反 SLO），而不是在某个特定服务器指标飙升时告警。基于症状的告警可减少 80% 的噪声。
4. 关联各类信号 - 同一请求的日志、追踪和指标应通过追踪 ID 关联。没有关联信息，调试分布式问题就只能靠猜测。
5. 管理错误预算 - 定义 SLO（可用性为 99.9% = 每月 43 分钟的停机预算）。当错误预算消耗速率过快时告警，而不是针对单个错误告警。

---

关键模式

模式 1：OpenTelemetry 检测（Node.js）

适用场景： 任何需要追踪、指标和日志关联的生产环境 Node.js 服务。

实现：

// tracing.ts - Must be imported BEFORE any other module
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { Resource } from "@opentelemetry/resources";
import {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
  ATTR_DEPLOYMENT_ENVIRONMENT,
} from "@opentelemetry/semantic-conventions";

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME ?? "my-service",
    [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION ?? "0.0.0",
    [ATTR_DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV ?? "development",
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/metrics",
    }),
    exportIntervalMillis: 30_000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable noisy fs instrumentation
      "@opentelemetry/instrumentation-fs": { enabled: false },
      // Configure HTTP to capture request/response headers
      "@opentelemetry/instrumentation-http": {
        requestHook: (span, request) => {
          span.setAttribute("http.request.header.x-request-id",
            request.headers?.["x-request-id"] ?? "unknown"
          );
        },
      },
    }),
  ],
});

sdk.start();

process.on("SIGTERM", () => {
  sdk.shutdown().then(() => process.exit(0));
});

// Custom span creation for business logic
import { trace, SpanStatusCode, context } from "@opentelemetry/api";

const tracer = trace.getTracer("order-service");

async function processOrder(orderId: string): Promise<Order> {
  return tracer.startActiveSpan("process_order", async (span) => {
    try {
      span.setAttribute("order.id", orderId);

      const order = await tracer.startActiveSpan("fetch_order", async (fetchSpan) => {
        const result = await db.orders.findUnique({ where: { id: orderId } });
        fetchSpan.setAttribute("order.total", result?.total ?? 0);
        fetchSpan.end();
        return result;
      });

      if (!order) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: "Order not found" });
        throw new OrderNotFoundError(orderId);
      }

      await tracer.startActiveSpan("charge_payment", async (paymentSpan) => {
        paymentSpan.setAttribute("payment.amount", order.total);
        await paymentService.charge(order);
        paymentSpan.end();
      });

      span.setStatus({ code: SpanStatusCode.OK });
      return order;
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) });
      span.recordException(error as Error);
      throw error;
    } finally {
      span.end();
    }
  });
}

原因： OpenTelemetry 提供厂商中立的检测方案。自动检测会自动捕获 HTTP、数据库和 gRPC 调用。自定义 Span 可添加业务上下文（订单 ID、支付金额），让追踪信息能够切实用于调试。

---

模式 2：带关联信息的结构化日志

适用场景： 任何会生成日志的应用（也就是所有应用）。

实现：

import pino from "pino";
import { context, trace } from "@opentelemetry/api";

// Create logger with trace correlation
const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  formatters: {
    level: (label) => ({ level: label }),
  },
  mixin() {
    // Automatically inject trace context into every log line
    const span = trace.getSpan(context.active());
    if (span) {
      const spanContext = span.spanContext();
      return {
        traceId: spanContext.traceId,
        spanId: spanContext.spanId,
        traceFlags: spanContext.traceFlags,
      };
    }
    return {};
  },
  // Redact sensitive fields
  redact: ["req.headers.authorization", "password", "token", "apiKey"],
});

// Usage - always log structured data, not string interpolation
logger.info({ orderId, userId, total: order.total }, "Order processed successfully");

// NOT this:
// logger.info(`Order ${orderId} processed for user ${userId} with total ${order.total}`);

// Error logging with context
logger.error(
  {
    err: error,
    orderId,
    operation: "payment_charge",
    paymentProvider: "stripe",
  },
  "Payment processing failed"
);

// Child loggers for request-scoped context
function createRequestLogger(req: Request) {
  return logger.child({
    requestId: req.headers.get("x-request-id") ?? crypto.randomUUID(),
    path: req.url,
    method: req.method,
    userAgent: req.headers.get("user-agent"),
  });
}

原因： 结构化日志可供查询。你可以按 orderId 筛选，通过 traceId 关联追踪信息，并按 operation 汇总错误率。使用字符串插值的日志需要通过正则表达式提取字段，在大规模场景下会失效。

---

模式 3：基于 SLO 的告警

适用场景： 设置生产环境告警，以减少噪声并聚焦于用户影响。

实现：

# Prometheus alerting rules based on SLOs
groups:
  - name: slo-alerts
    rules:
      # Availability SLO: 99.9% success rate
      # Alert when burning through error budget too fast
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (1 - 0.999) * 14.4
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error budget burn rate is 14.4x (will exhaust in 1 hour)"
          dashboard: "https://grafana.internal/d/slo-overview"

      # Latency SLO: 99% of requests under 500ms
      - alert: HighLatencyBudgetBurn
        expr: |
          (
            sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
            /
            sum(rate(http_request_duration_seconds_count[5m]))
          ) < 0.99
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Latency SLO breach: >1% of requests exceeding 500ms"

      # Multi-window multi-burn-rate alert (Google SRE book pattern)
      - alert: SLOBreach_MultiWindow
        expr: |
          (
            error_ratio:rate1h > (14.4 * 0.001)
            and
            error_ratio:rate5m > (14.4 * 0.001)
          )
          or
          (
            error_ratio:rate6h > (6 * 0.001)
            and
            error_ratio:rate30m > (6 * 0.001)
          )
        labels:
          severity: critical

// Application-level SLO tracking with custom metrics
import { metrics } from "@opentelemetry/api";

const meter = metrics.getMeter("slo-metrics");

const requestCounter = meter.createCounter("http.requests.total", {
  description: "Total HTTP requests",
});

const requestDuration = meter.createHistogram("http.request.duration", {
  description: "HTTP request duration in seconds",
  unit: "s",
});

// Middleware to track SLO metrics
function sloMiddleware(req: Request, res: Response, next: NextFunction) {
  const start = performance.now();

  res.on("finish", () => {
    const duration = (performance.now() - start) / 1000;
    const attributes = {
      method: req.method,
      route: req.route?.path ?? "unknown",
      status: String(res.statusCode),
      success: String(res.statusCode < 500),
    };

    requestCounter.add(1, attributes);
    requestDuration.record(duration, attributes);
  });

  next();
}

原因： 传统的基于阈值的告警（CPU > 80%、错误数 > 10）会产生噪声。基于 SLO 的告警会关注“用户是否受到影响？”以及“错误预算消耗得有多快？”。这种方法只会在需要采取行动时触发页面通知。

---

模式 4：健康检查和就绪探针

适用场景： 任何部署在 Kubernetes 中或位于负载均衡器后的服务。

实现：

import { Router } from "express";

interface HealthCheckResult {
  status: "healthy" | "degraded" | "unhealthy";
  checks: Record<string, {
    status: "pass" | "fail" | "warn";
    latencyMs: number;
    message?: string;
  }>;
  uptime: number;
  version: string;
}

const healthRouter = Router();

// Liveness probe - is the process alive?
// Should NEVER check dependencies. Only checks if the process can respond.
healthRouter.get("/healthz", (req, res) => {
  res.status(200).json({ status: "alive" });
});

// Readiness probe - can this instance serve traffic?
// Checks critical dependencies.
healthRouter.get("/readyz", async (req, res) => {
  const checks: HealthCheckResult["checks"] = {};

  // Check database
  const dbStart = performance.now();
  try {
    await db.$queryRaw`SELECT 1`;
    checks.database = { status: "pass", latencyMs: performance.now() - dbStart };
  } catch (err) {
    checks.database = {
      status: "fail",
      latencyMs: performance.now() - dbStart,
      message: (err as Error).message,
    };
  }

  // Check Redis
  const redisStart = performance.now();
  try {
    await redis.ping();
    checks.redis = { status: "pass", latencyMs: performance.now() - redisStart };
  } catch (err) {
    checks.redis = {
      status: "fail",
      latencyMs: performance.now() - redisStart,
      message: (err as Error).message,
    };
  }

  const allPassing = Object.values(checks).every((c) => c.status === "pass");
  const anyFailing = Object.values(checks).some((c) => c.status === "fail");

  const result: HealthCheckResult = {
    status: anyFailing ? "unhealthy" : allPassing ? "healthy" : "degraded",
    checks,
    uptime: process.uptime(),
    version: process.env.SERVICE_VERSION ?? "unknown",
  };

  res.status(anyFailing ? 503 : 200).json(result);
});

原因： Kubernetes 使用存活探针重启卡住的进程，并使用就绪探针停止向未就绪的实例发送流量。配置不当会导致级联故障：检查数据库的存活探针会在数据库中断期间重启健康的 Pod，令情况恶化。

---

Grafana 仪表板速查

{
  "panels": [
    {
      "title": "Request Rate",
      "type": "timeseries",
      "targets": [{ "expr": "sum(rate(http_requests_total[5m])) by (status)" }]
    },
    {
      "title": "Error Rate (%)",
      "type": "stat",
      "targets": [{
        "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100"
      }],
      "thresholds": [
        { "value": 0, "color": "green" },
        { "value": 0.1, "color": "yellow" },
        { "value": 1, "color": "red" }
      ]
    },
    {
      "title": "P99 Latency",
      "type": "timeseries",
      "targets": [{
        "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))"
      }]
    }
  ]
}

---

反模式

反模式	为什么不好	更好的做法
在生产环境中使用 console.log 记录日志	缺乏结构、关联信息和级别	使用 pino 或 winston 输出 JSON
每次出现 5xx 错误都触发告警	告警疲劳，团队会忽视告警	在 SLO 违约／错误预算消耗速率超标时触发告警
存活探针检查数据库	数据库故障时会重启 Pod（引发级联故障）	存活探针只检查进程；就绪探针检查依赖项
未传播跟踪上下文	无法跨服务追踪请求	使用 W3C traceparent 标头，并在所有客户端中注入
对 100% 的跟踪采样	规模扩大时存储成本会激增	使用基于头部的采样（10-20%），或针对错误使用基于尾部的采样
记录 PII（电子邮件、IP）	违反 GDPR／隐私规定	在日志记录器配置中删除敏感字段
仪表板包含 50 个面板	信息过载且加载缓慢	关注四个黄金信号：速率、错误、耗时和饱和度

---

检查清单

• 在所有其他导入之前初始化 OpenTelemetry SDK
• 为 HTTP、数据库和队列客户端启用自动插桩
• 在业务逻辑边界处添加带有相关属性的自定义跨度
• 使用结构化 JSON 日志，并关联跟踪 ID
• 在日志记录器配置中删除敏感字段
• 存活探针：只检查进程（不检查依赖项）
• 就绪探针：检查所有关键依赖项
• 为可用性和延迟定义 SLO
• 根据错误预算消耗速率触发告警，而非基于原始阈值
• 为每个服务配置包含四个黄金信号的仪表板
• 配置适用于生产规模的跟踪采样策略
• 配置日志聚合管道，将日志发送到集中式存储

---

相关资源

• 技能： performance-engineering（延迟优化）、application-security（安全日志记录）
• 规则： docs/reference/stacks/fullstack-nextjs-nestjs.md（NestJS 插桩模式）
• 规则： docs/reference/tooling/troubleshooting.md（使用日志进行调试）