opentelemetry

OpenTelemetry

采用与供应商无关的遥测方案，在各项服务中使用一致的埋点。

适用场景

• 调试微服务间的延迟
• 统一可观测性数据模型和命名
• 将遥测数据发送到 Prometheus、Grafana、Datadog 或 OTLP 后端
• 构建支持追踪与日志关联的 SLO 仪表板
• 为 Python 或 Node.js 应用添加追踪和指标埋点
• 为现有服务设置自动埋点，无需更改代码

先决条件

• 在容器或虚拟机上运行的应用服务
• 追踪数据后端（Jaeger、Tempo、Datadog 或任何 OTLP 接收器）
• 指标数据后端（Prometheus、Mimir 或 OTLP 接收器）
• Kubernetes 集群（用于部署 Collector）或运行 systemd 的虚拟机
• 服务到 Collector，以及 Collector 到后端的网络访问

核心工作流

1. 为服务、环境和版本定义语义约定。
2. 在每项服务中添加 SDK 或自动埋点。
3. 运行 OpenTelemetry Collector，以接收、转换和导出遥测数据。
4. 验证基数和采样设置，以控制成本。
5. 根据采集的数据创建黄金信号仪表板和告警。

Collector 生产环境配置

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape Prometheus endpoints
  prometheus:
    config:
      scrape_configs:
        - job_name: "kubernetes-pods"
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: $$1

  # Host metrics for infrastructure monitoring
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      network: {}
      load: {}

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s

  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  attributes:
    actions:
      - key: deployment.environment
        value: production
        action: upsert

  # Drop high-cardinality attributes to control cost
  filter/drop-debug:
    traces:
      span:
        - 'attributes["http.request.header.x-debug"] == "true"'

  # Reduce cardinality on URL paths
  transform/normalize-routes:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}")
          - replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")

  # Resource detection for cloud environments
  resourcedetection:
    detectors: [env, system, gcp, aws, azure]
    timeout: 5s

exporters:
  # Send traces to Tempo/Jaeger
  otlp/traces:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Send metrics to Prometheus via remote write
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push
    tls:
      insecure: true

  # Send logs to Loki
  otlp/logs:
    endpoint: loki:4317
    tls:
      insecure: true

  # Debug exporter for development
  debug:
    verbosity: basic

service:
  telemetry:
    logs:
      level: info
    metrics:
      address: 0.0.0.0:8888

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, transform/normalize-routes, batch, attributes]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [memory_limiter, resourcedetection, batch, attributes]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch, attributes]
      exporters: [otlp/logs]

Collector Kubernetes 部署

# otel-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: observability
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.98.0
          args: ["--config=/etc/otel/config.yaml"]
          ports:
            - containerPort: 4317
              name: otlp-grpc
            - containerPort: 4318
              name: otlp-http
            - containerPort: 8888
              name: metrics
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          volumeMounts:
            - name: config
              mountPath: /etc/otel
          livenessProbe:
            httpGet:
              path: /
              port: 13133
          readinessProbe:
            httpGet:
              path: /
              port: 13133
      volumes:
        - name: config
          configMap:
            name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    app: otel-collector
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
    - name: metrics
      port: 8888
      targetPort: 8888

Python SDK 埋点

# tracing_setup.py
"""Initialize OpenTelemetry tracing and metrics for a Python service."""
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import os

def init_telemetry(service_name: str, service_version: str):
    """Initialize OTel SDK with traces and metrics."""
    resource = Resource.create({
        "service.name": service_name,
        "service.version": service_version,
        "deployment.environment": os.getenv("DEPLOY_ENV", "development"),
    })

    # Traces
    trace_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
        insecure=True,
    )
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
    trace.set_tracer_provider(tracer_provider)

    # Metrics
    metric_exporter = OTLPMetricExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
        insecure=True,
    )
    metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=15000)
    meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
    metrics.set_meter_provider(meter_provider)

    # Auto-instrument common libraries
    RequestsInstrumentor().instrument()
    SQLAlchemyInstrumentor().instrument()

    return trace.get_tracer(service_name), metrics.get_meter(service_name)

# Usage example
tracer, meter = init_telemetry("order-service", "1.2.0")

# Custom span
with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.total", total)
    # ... business logic ...

# Custom metric
request_counter = meter.create_counter(
    "app.requests",
    description="Total application requests",
)
request_counter.add(1, {"route": "/api/orders", "method": "POST"})

Node.js SDK 埋点

// tracing.js
// Initialize OpenTelemetry for a Node.js service.
// Load this file BEFORE any other imports: node -r ./tracing.js app.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { Resource } = require("@opentelemetry/resources");
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require("@opentelemetry/semantic-conventions");

const resource = new Resource({
  [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || "node-service",
  [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || "1.0.0",
  "deployment.environment": process.env.DEPLOY_ENV || "development",
});

const sdk = new NodeSDK({
  resource,
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
    }),
    exportIntervalMillis: 15000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-http": {
        ignoreIncomingPaths: ["/health", "/ready"],
      },
      "@opentelemetry/instrumentation-express": { enabled: true },
      "@opentelemetry/instrumentation-pg": { enabled: true },
      "@opentelemetry/instrumentation-redis": { enabled: true },
    }),
  ],
});

sdk.start();
process.on("SIGTERM", () => sdk.shutdown());

使用 Kubernetes Operator 进行自动埋点

# otel-auto-instrumentation.yaml
# Install the OTel Operator first:
#   helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
#     --namespace observability --create-namespace

# Define instrumentation for Python services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0
    env:
      - name: OTEL_PYTHON_LOG_CORRELATION
        value: "true"
---
# Define instrumentation for Node.js services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: nodejs-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.49.1

为 Pod 添加埋点时，请添加以下注解：

# For Python:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

# For Node.js:
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "true"

采样策略

# Tail-based sampling config (in collector)
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      # Always keep error traces
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always keep slow traces (> 2s)
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 2000

      # Sample 10% of successful traces
      - name: normal-traffic
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

      # Always keep traces with specific attributes
      - name: important-users
        type: string_attribute
        string_attribute:
          key: user.tier
          values: [enterprise, premium]

      # Rate limit per service to prevent one service from dominating
      - name: rate-limit
        type: rate_limiting
        rate_limiting:
          spans_per_second: 500

最佳实践

• 对高流量生产环境中的追踪使用尾部采样。
• 为遥测数据添加 service.name、service.version 和 deployment.environment 标签。
• 在 Collector 中尽早丢弃噪声属性。
• 控制指标标签的基数，确保查询性能稳定。
• 使用资源检测器自动填充云环境元数据。
• 如果数据量需要，分别为追踪和指标配置独立的 Collector 池。
• 在每个 Collector 流水线中设置 memory_limiter，以防止 OOM。
• 生产环境中使用 contrib Collector 镜像（包含更多接收器和导出器）。

故障排查

症状	检查项	修复方法
后端未收到追踪数据	检查 Collector 日志中是否存在导出错误	验证端点 URL 和网络策略
追踪中缺少 Span	代理剥除了传播头	配置代理传递 traceparent 请求头
Collector 内存占用过高	尾部采样中未完成的追踪过多	减少 num_traces 或增加内存限制
指标基数激增	标签值无界增长（用户 ID、URL）	添加转换处理器以规范化值
自动埋点未生效	Pod 缺少注解或 Operator 未运行	确认 Operator 运行正常且注解正确
指标重复	SDK 和自动埋点同时启用	每种信号只使用一种埋点方式

相关技能

• prometheus-grafana - 仪表板和告警
• datadog - 托管式可观测性后端
• alerting-oncall - 值班路由和升级
• rag-observability-evals - RAG 专用可观测性
• agent-observability - AI Agent 追踪