tempo

Grafana Tempo

文档：https://grafana.com/docs/tempo/latest/

经济高效的分布式追踪方案。支持接收 OTLP / Jaeger / Zipkin / OpenCensus / Kafka 数据，并将 Parquet 块存储在 S3/GCS/Azure 中。

先决条件

• Docker（快速入门）或 Kubernetes（生产环境）
• 分布式部署所需的对象存储桶（S3/GCS/Azure）
• 可发送 OTLP 数据的应用，或用于生成合成流量的 tempo-cli
• 配置了 Tempo 数据源的 Grafana 堆栈，用于查询

常见工作流

1. 在本地启动 Tempo + 验证数据接收

# 1. Start the official Docker Compose example
git clone https://github.com/grafana/tempo.git
cd tempo/example/docker-compose/local
mkdir -p tempo-data
docker compose up -d

# 2. Verify readiness
curl -sf http://localhost:3200/ready                              # → "ready"

# 3. Send a synthetic OTLP span (full payload in scratch terminal)
curl -X POST -H 'Content-Type: application/json' \
  http://localhost:4318/v1/traces \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"my-service"}}]},
       "scopeSpans":[{"spans":[{"traceId":"5B8EFFF798038103D269B633813FC700","spanId":"EEE19B7EC3C1B100",
       "name":"my-op","startTimeUnixNano":1689969302000000000,"endTimeUnixNano":1689969302500000000,"kind":2}]}]}]}'

# 4. Verify the trace landed (ingestion-counter > 0 and the trace is fetchable)
curl -s http://localhost:3200/metrics | grep tempo_distributor_spans_received_total | head
curl -s http://localhost:3200/api/v2/traces/5B8EFFF798038103D269B633813FC700 | jq '.batches | length'
# Expect > 0.

# 5. In Grafana → Explore → Tempo, run TraceQL: {resource.service.name="my-service"}

2. 通过 Alloy 从应用发送追踪数据

// alloy.river
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls { insecure = true }
  }
}

# Verify Alloy forwarded successfully
curl -s http://localhost:12345/metrics | grep otelcol_exporter_sent_spans
# Then: same Grafana → Explore → Tempo check.

3. 编写并运行 TraceQL

# Slow requests from a service
{ resource.service.name = "frontend" && duration > 1s }

# Server span that has a downstream error (structural)
{ kind = server } >> { status = error }

# Error rate per service (metrics)
{ status = error } | rate() by (resource.service.name)

完整的运算符和作用域速查表、内置字段列表及指标函数：references/traceql.md。

# Via the API
curl -sG --data-urlencode 'q={resource.service.name="frontend" && duration > 1s}' \
  --data-urlencode "start=$(date -d '1h ago' +%s)" --data-urlencode "end=$(date +%s)" \
  http://localhost:3200/api/search | jq '.traces | length'

4. 在 Kubernetes 上部署（Helm）

helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo-distributed --version 1.61.3 \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-tempo-bucket \
  --set storage.trace.s3.region=us-east-1

# Verify every pod is Ready (distributor, ingester, querier, query-frontend, compactor)
kubectl get pods -n default -l app.kubernetes.io/instance=tempo
kubectl port-forward svc/tempo-query-frontend 3200:3200 &
curl -sf http://localhost:3200/ready

多租户

multitenancy_enabled: true
# All requests must include header: X-Scope-OrgID: <tenant-id>

完整的架构、端口、性能调优、指标生成器配置、多租户客户端代码片段，以及追踪到日志/指标/剖析数据源的说明：references/architecture-and-operations.md。

故障排除

• /ready → 503 → Ingester 仍在加入；检查 tempo_ingester_* 指标和日志
• 推送时出现 429 → 提高 max_outstanding_per_tenant 或各租户的数据接收限制
• “Explore 中未显示追踪数据” → 确认写入端和 Grafana 数据源的 X-Scope-OrgID 相匹配
• TraceQL 运行缓慢 → 缩小 start/end 范围，添加 service.name 筛选条件，并为常用属性启用专用 Parquet 列

资源

• Tempo 文档
• TraceQL 参考
• references/traceql.md — 完整的 TraceQL 速查表
• references/architecture-and-operations.md — 组件、端口、Helm、调优和数据源链接