loki

Grafana Loki 技能

Grafana Loki 综合指南——受 Prometheus 启发、经济高效且可水平扩展的日志聚合系统。

什么是 Loki？

Loki 是一种可水平扩展、高可用、多租户的日志聚合系统，它：

• 仅索引元数据（标签）——不像传统系统那样索引完整日志内容
• 在经济实惠的对象存储（S3、GCS、Azure Blob）中存储压缩后的数据块
• 使用 Prometheus 风格的标签来组织日志流
• 默认支持多租户，并内置租户隔离机制
• 成本高效——索引规模大幅缩小，运营成本更低

架构概述

核心组件

组件	用途
Distributor	验证请求、预处理标签，并将请求路由到 ingester
Ingester	在内存中缓冲日志，将其压缩为数据块并写入存储
Querier	执行来自 ingester 和存储的 LogQL 查询
Query Frontend	通过拆分、缓存和调度加速查询
Query Scheduler	管理每个租户的查询队列，确保公平性
Index Gateway	为 TSDB 存储提供索引查询服务
Compactor	合并索引文件、管理保留期限并处理删除操作
Ruler	评估告警规则和记录规则

数据流

写入路径：

Log Source → Distributor → Ingester → Object Storage
                                    ↓
                              Chunks + Indexes

读取路径：

Query → Query Frontend → Query Scheduler → Querier
                                             ↓
                                    Ingesters + Storage

部署模式

1. 单体模式（-target=all）

• 所有组件都在单个进程中运行
• 适用于：初期试用、小规模部署（每天约 20GB 日志）
• 最简单的方案

2. 简单可扩展部署（SSD）——推荐的默认模式

deploymentMode: SimpleScalable

write:
  replicas: 3   # Distributor + Ingester

read:
  replicas: 2   # Query Frontend + Querier

backend:
  replicas: 2   # Compactor + Index Gateway + Query Scheduler + Ruler

3. 微服务模式（分布式）

deploymentMode: Distributed

ingester:
  replicas: 3
  zoneAwareReplication:
    enabled: true

distributor:
  replicas: 3

querier:
  replicas: 3

queryFrontend:
  replicas: 2

queryScheduler:
  replicas: 2

compactor:
  replicas: 1

indexGateway:
  replicas: 2

架构配置

推荐：使用 Schema v13 的 TSDB

loki:
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: azure  # or s3, gcs
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

存储配置

Azure Blob Storage（Azure 推荐）

loki:
  storage:
    type: azure
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    azure:
      accountName: <storage-account-name>
      # Option 1: User-Assigned Managed Identity (Recommended)
      useManagedIdentity: true
      useFederatedToken: false
      userAssignedId: <identity-client-id>
      # Option 2: Account Key (Dev only)
      # accountKey: <account-key>
      requestTimeout: 30s

AWS S3

loki:
  storage:
    type: s3
    bucketNames:
      chunks: my-loki-chunks-2024
      ruler: my-loki-ruler-2024
      admin: my-loki-admin-2024
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      # Use IAM roles or access keys
      accessKeyId: <access-key>
      secretAccessKey: <secret-key>
      s3ForcePathStyle: false

Google Cloud Storage

loki:
  storage:
    type: gcs
    bucketNames:
      chunks: my-loki-gcs-bucket
    gcs:
      bucketName: my-loki-gcs-bucket
      # Uses Workload Identity or service account

数据块配置最佳实践

loki:
  ingester:
    chunk_encoding: snappy        # Recommended (fast + efficient)
    chunk_target_size: 1572864    # ~1.5MB compressed
    max_chunk_age: 2h             # Max time before flush
    chunk_idle_period: 30m        # Flush idle chunks
    flush_check_period: 30s
    flush_op_timeout: 10m

设置	推荐值	用途
chunk_encoding	snappy	在速度与压缩率之间实现最佳平衡
chunk_target_size	1.5MB	目标压缩数据块大小
max_chunk_age	2h	限制内存占用和数据丢失风险
chunk_idle_period	30m	刷新不活跃的流

限制配置

loki:
  limits_config:
    # Retention
    retention_period: 744h              # 31 days

    # Ingestion limits
    ingestion_rate_mb: 50
    ingestion_burst_size_mb: 100
    per_stream_rate_limit: 3MB
    per_stream_rate_limit_burst: 15MB

    # Query limits
    max_query_series: 10000
    max_query_lookback: 720h
    max_entries_limit_per_query: 10000

    # Required for OTLP
    allow_structured_metadata: true
    volume_enabled: true

    # Sample rejection
    reject_old_samples: true
    reject_old_samples_max_age: 168h    # 7 days
    max_label_names_per_series: 25

Compactor 配置

loki:
  compactor:
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 50
    compaction_interval: 10m
    delete_request_store: azure         # Match your storage type

缓存配置

推荐：使用独立的 Memcached 实例

# Helm values for Loki caching
memcached:
  # Results cache
  frontend:
    replicas: 3
    memcached:
      maxItemMemory: 1024               # 1GB
      maxItemSize: 5m
      connectionLimit: 1024

  # Chunks cache
  chunks:
    replicas: 3
    memcached:
      maxItemMemory: 4096               # 4GB
      maxItemSize: 2m
      connectionLimit: 1024

# Enable caching in Loki config
loki:
  chunk_store_config:
    chunk_cache_config:
      memcached_client:
        host: loki-memcached-chunks.monitoring.svc
        service: memcached-client

LogQL 查询语言

基本查询

# Stream selector
{job="api-server"}

# Multiple labels
{job="api-server", env="prod"}

# Label matchers
{namespace=~".*-prod"}           # Regex match
{level!="debug"}                  # Not equal

# Filter expressions
{job="api-server"} |= "error"     # Contains
{job="api-server"} != "debug"     # Not contains
{job="api-server"} |~ "err.*"     # Regex match
{job="api-server"} !~ "debug.*"   # Regex not match

管道阶段

# JSON parsing
{job="api-server"} | json

# Extract specific fields
{job="api-server"} | json | line_format "{{.message}}"

# Label extraction
{job="api-server"} | logfmt | level="error"

# Pattern matching
{job="api-server"} | pattern "<ip> - - [<_>] \"<method> <path>\"" | method="POST"

指标查询

# Count logs per minute
count_over_time({job="api-server"}[1m])

# Rate of errors
rate({job="api-server"} |= "error" [5m])

# Bytes rate
bytes_rate({job="api-server"}[5m])

# Sum by label
sum by (namespace) (rate({job="api-server"}[5m]))

# Top 10 by volume
topk(10, sum by (namespace) (bytes_rate({}[5m])))

OpenTelemetry 集成

原生 OTLP（推荐——Loki 3.0+）

OpenTelemetry Collector 配置：

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp
    headers:
      X-Scope-OrgID: "my-tenant"

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

Loki 配置：

loki:
  limits_config:
    allow_structured_metadata: true    # Required for OTLP

主要优势：

• 日志正文以纯文本形式存储（不编码为 JSON）
• 自动索引 17 个默认资源属性
• 无需解析 JSON，查询更简单
• 存储效率更高

资源属性映射

OTLP 属性	Loki 标签
service.name	service_name
service.namespace	service_namespace
k8s.pod.name	k8s_pod_name
k8s.namespace.name	k8s_namespace_name
cloud.region	cloud_region

Kubernetes Helm 部署

添加仓库

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

使用 Values 安装

helm install loki grafana/loki \
  --namespace monitoring \
  --values values.yaml

生产环境 Values 示例

deploymentMode: Distributed

loki:
  auth_enabled: true

  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: azure
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  storage:
    type: azure
    azure:
      accountName: mystorageaccount
      useManagedIdentity: true
      userAssignedId: <client-id>
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin

  limits_config:
    retention_period: 2160h             # 90 days
    allow_structured_metadata: true

ingester:
  replicas: 3
  zoneAwareReplication:
    enabled: true
  resources:
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      cpu: 4
      memory: 16Gi

querier:
  replicas: 3
  maxUnavailable: 2

queryFrontend:
  replicas: 2

distributor:
  replicas: 3

compactor:
  replicas: 1

indexGateway:
  replicas: 2
  maxUnavailable: 1

# Gateway for external access
gateway:
  service:
    type: LoadBalancer

# Monitoring
monitoring:
  serviceMonitor:
    enabled: true

Azure 身份配置

用户分配的托管身份（推荐）

1. 创建身份：

az identity create \
  --name loki-identity \
  --resource-group <rg>

IDENTITY_CLIENT_ID=$(az identity show --name loki-identity --resource-group <rg> --query clientId -o tsv)
IDENTITY_PRINCIPAL_ID=$(az identity show --name loki-identity --resource-group <rg> --query principalId -o tsv)

2. 分配给节点池：

az vmss identity assign \
  --resource-group <aks-node-rg> \
  --name <vmss-name> \
  --identities /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/loki-identity

3. 授予存储权限：

az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id $IDENTITY_PRINCIPAL_ID \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>

4. 配置 Loki：

loki:
  storage:
    azure:
      useManagedIdentity: true
      userAssignedId: <IDENTITY_CLIENT_ID>

多租户

loki:
  auth_enabled: true

# Query with tenant header
curl -H "X-Scope-OrgID: tenant-a" \
  "http://loki:3100/loki/api/v1/query?query={job=\"app\"}"

# Multi-tenant queries (if enabled)
# X-Scope-OrgID: tenant-a|tenant-b

故障排查

常见问题

1. 找不到容器（Azure）

# Create required containers
az storage container create --name loki-chunks --account-name <storage>
az storage container create --name loki-ruler --account-name <storage>
az storage container create --name loki-admin --account-name <storage>

2. 授权失败（Azure）

# Verify RBAC assignment
az role assignment list --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>

# Assign if missing
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

# Restart pod to refresh token
kubectl delete pod -n monitoring <ingester-pod>

3. Ingester 内存溢出

# Increase memory limits
ingester:
  resources:
    limits:
      memory: 16Gi

4. 查询超时

loki:
  querier:
    query_timeout: 5m
    max_concurrent: 8
  query_scheduler:
    max_outstanding_requests_per_tenant: 2048

诊断命令

# Check pod status
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki

# Check ingester logs
kubectl logs -n monitoring -l app.kubernetes.io/component=ingester --tail=100

# Check compactor logs
kubectl logs -n monitoring -l app.kubernetes.io/component=compactor --tail=100

# Verify readiness
kubectl exec -it <loki-pod> -n monitoring -- wget -qO- http://localhost:3100/ready

# Check configuration
kubectl exec -it <loki-pod> -n monitoring -- cat /etc/loki/config/config.yaml

API 参考

写入

# Push logs
POST /loki/api/v1/push

# OTLP logs
POST /otlp/v1/logs

查询

# Instant query
GET /loki/api/v1/query?query={job="app"}&time=<timestamp>

# Range query
GET /loki/api/v1/query_range?query={job="app"}&start=<start>&end=<end>

# Labels
GET /loki/api/v1/labels
GET /loki/api/v1/label/<name>/values

# Series
GET /loki/api/v1/series

# Tail (WebSocket)
GET /loki/api/v1/tail?query={job="app"}

健康状态

GET /ready
GET /metrics

参考文档

按主题查看详细配置：

• 存储配置：对象存储、保留策略、WAL
• LogQL 参考：查询语法和示例
• OpenTelemetry 集成：OTLP 配置

外部资源

• Loki 官方文档
• Loki Helm Chart
• LogQL 文档
• Loki GitHub 仓库

---

注意事项

• Ingester 会拒绝流中途发生基数变化的标签——从低基数切换为高基数的标签会悄无声息地将流拆分，形成一个幽灵缺口。
• LogQL |=（行过滤器）比标签过滤器更快——查询计划只会优化行过滤器；标签过滤器会在解析后执行。
• 通过 X-Scope-OrgID 隔离租户：缺少此标头时，写入会悄无声息地进入租户 "fake"——日志并没有丢失，而是进入了错误的租户。
• 保留策略按租户设置；全局保留环境变量仅作为回退——配置错误的租户会悄无声息地覆盖全局设置。
• Compactor 未运行 = 孤立块不断堆积——存储会无限增长；Compactor 的故障记录在另一个组件的日志中。
• 从 Promtail 迁移到 Alloy：标签规范化存在细微差异（例如 __path__ 语义）；在流量高峰期间迁移会导致日志丢失。