opentelemetry

使用 Grafana 的 OpenTelemetry

文档：https://grafana.com/docs/opentelemetry/

与供应商无关的埋点管道。应用通过 OTLP 通信 → Alloy（或直接通信）→ Grafana Cloud（Mimir / Loki / Tempo / Pyroscope）。

后端

信号	后端
指标	Grafana Mimir
日志	Grafana Loki
跟踪	Grafana Tempo
性能剖析	Grafana Pyroscope

先决条件

• Grafana Cloud 堆栈，或可自托管的 Mimir / Loki / Tempo
• Cloud OTLP 端点：https://otlp-gateway-<region>.grafana.net/otlp
• Basic-auth 凭据：数字型实例 ID + 具有 MetricsPublisher + LogsPublisher + TracesPublisher 权限的 API 令牌
• 要进行埋点的应用

常见工作流

1. 向 Grafana Cloud OTLP 端点进行身份验证

# 1. Build the auth header
INSTANCE_ID=123456
API_KEY="glc_eyJ..."
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n "${INSTANCE_ID}:${API_KEY}" | base64)"
export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod"

# 2. Smoke-test creds with a curl POST against the OTLP traces endpoint (empty body)
curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST -H "Content-Type: application/x-protobuf" \
  -H "Authorization: Basic $(echo -n "${INSTANCE_ID}:${API_KEY}" | base64)" \
  "$OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces" --data-binary '\n'
# Expect 400 (malformed payload) — NOT 401 (auth) or 404 (wrong endpoint).

2. 对 Java 应用进行自动埋点并验证

# 1. Download the Grafana JVM agent (single jar)
curl -sLO https://github.com/grafana/grafana-opentelemetry-java/releases/latest/download/grafana-opentelemetry-java.jar

# 2. Run with the agent + env from step 1
java -javaagent:./grafana-opentelemetry-java.jar -jar myapp.jar

# 3. Generate traffic, then verify in Grafana → Explore → Tempo:
#    TraceQL: { resource.service.name = "myapp" }
#    Expect spans within ~30s. Also verify metrics:
#    PromQL: count by (service_name)({service_name="myapp"})

3. 对 Python 应用进行自动埋点

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install

# Same env vars as step 1, then:
opentelemetry-instrument python app.py

# Verify the same way — Explore → Traces filter service.name=myapp.

4. 添加 Alloy 作为缓冲 / 采样收集器

# Application points at local Alloy (gRPC fastest)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Alloy environment for forwarding to Cloud
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=$INSTANCE_ID
export GRAFANA_CLOUD_API_KEY=$API_KEY
alloy run /etc/alloy/config.alloy

# Verify Alloy received and forwarded
curl -s http://localhost:12345/metrics | grep otelcol_exporter_sent_spans

完整 Alloy 配置 + 尾部采样配置块 + OTel Collector YAML + K8s Operator 安装说明：references/collector-config.md。

各语言 SDK 详情（Go 完整代码、Node 手动设置、.NET ASP.NET Core、所有环境变量的特殊之处）：references/instrumentation.md。

5. Kubernetes — 通过 Operator 自动注入

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: my-instrumentation }
spec:
  exporter: { endpoint: http://otelcol:4317 }
  propagators: [tracecontext, baggage]
  java:
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
  nodejs: {}
  python: {}

然后为 Pod 添加注解：

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # or: inject-nodejs, inject-python, inject-dotnet

# Verify the operator injected the agent
kubectl describe pod <pod> | grep -A2 'opentelemetry-auto-instrumentation'
# Then run the same Grafana Explore checks.

采样 — 如何选择

# Head sampling (cheap, decided at start; may lose rare errors)
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # 10%

尾部采样（查看完整跟踪后再做决定——保留错误并对其余部分采样）需要 Alloy / OTel-Collector tail_sampling 处理器；完整配置块见 references/collector-config.md。

关键环境变量

变量	示例
OTEL_EXPORTER_OTLP_ENDPOINT	https://otlp-gateway-prod-us-east-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_PROTOCOL	grpc 或 http/protobuf
OTEL_EXPORTER_OTLP_HEADERS	Authorization=Basic <base64>
OTEL_RESOURCE_ATTRIBUTES	service.name=app,service.namespace=team,deployment.environment=prod
OTEL_SERVICE_NAME	service.name 的简写
OTEL_TRACES_SAMPLER / _ARG	parentbased_traceidratio / 0.1

故障排查

• OTLP 网关返回 401 → 实例 ID 不是数字，或 API 密钥缺少发布者权限
• 返回 404 → 端点 URL 错误（必须以 /otlp 结尾）
• 缺少 Span → 检查 OTEL_EXPORTER_OTLP_PROTOCOL 是否与传输方式匹配（Cloud OTLP 网关 = http/protobuf，本地 Alloy = grpc）
• Node.js 自动埋点在打包后失效 → @vercel/ncc 等打包工具会破坏 require 钩子
• Gunicorn / uWSGI 下的 Python 没有 Span → 在 post-fork 钩子中重新初始化 OTel providers

资源

• Grafana OTel 文档
• Grafana Cloud OTLP
• Grafana JVM Agent
• Grafana .NET SDK
• OTel Operator