mcpField
L
loki-logging

loki-logging

github.com/bagelhole云与 DevOps1.2k331 次安装23 周前更新
代码仓库

关于 loki-logging

loki-logging 是面向平台与应用运维人员的 Grafana Loki 日志聚合和查询技能，介绍 Docker、Kubernetes 部署及 LogQL 分析。

主要功能

  1. 01使用 Promtail 采集日志
  2. 02通过标签和 LogQL 查询日志
  3. 03在 Grafana 中查看日志与仪表板
  4. 04配置记录规则和告警
  5. 05排查采集、查询与内存问题

适用场景

  1. 01为 Kubernetes 环境集中收集日志
  2. 02按标签筛选并分析应用日志
  3. 03在 Grafana 仪表板中查看日志
  4. 04基于日志错误率配置告警

安装

复制下面的命令，在终端中运行。

$npx skills add https://github.com/bagelhole/devops-security-agent-skills --skill loki-logging

安装后在 Claude Code 中输入/loki-logging确认已加载。

详细介绍

Grafana Loki

使用 Grafana Loki（受 Prometheus 启发的日志系统）聚合和查询日志。

何时使用此技能

在以下情况下使用此技能：

  • 实现经济高效的日志聚合
  • 为 Kubernetes 环境构建日志系统
  • 将日志与 Grafana 仪表板集成
  • 使用基于标签的筛选查询日志
  • 倾向于使用比 ELK 更轻量的替代方案

先决条件

  • Docker 或 Kubernetes
  • 用于可视化的 Grafana
  • Promtail 或其他日志采集器

架构概览

┌─────────────┐     ┌──────────┐     ┌──────────┐
│ Application │────▶│ Promtail │────▶│   Loki   │
└─────────────┘     └──────────┘     └──────────┘
                                          │
                                          ▼
                                     ┌──────────┐
                                     │ Grafana  │
                                     └──────────┘

Docker 部署

# docker-compose.yml
version: '3.8'

services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - ./promtail-config.yaml:/etc/promtail/config.yaml
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yaml

  grafana:
    image: grafana/grafana:10.2.0
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

volumes:
  loki-data:
  grafana-data:

Loki 配置

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/cache
    shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_query_series: 5000
  max_query_parallelism: 2

chunk_store_config:
  max_look_back_period: 168h

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h

Promtail 配置

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # System logs
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

  # Docker container logs
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'stream'

  # Application logs with parsing
  - job_name: application
    static_configs:
      - targets:
          - localhost
        labels:
          job: application
          __path__: /var/log/app/*.log
    pipeline_stages:
      - json:
          expressions:
            level: level
            message: message
            timestamp: timestamp
      - labels:
          level:
      - timestamp:
          source: timestamp
          format: RFC3339

Kubernetes 部署

# Using Helm
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.enabled=true \
  --set promtail.enabled=true
Promtail DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      containers:
        - name: promtail
          image: grafana/promtail:2.9.0
          args:
            - -config.file=/etc/promtail/promtail.yaml
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

LogQL 查询

基本查询
# All logs from a job
{job="application"}

# Filter by label
{job="application", level="error"}

# Multiple labels
{namespace="production", container="api"}

# Regex match
{job=~"app.*"}
日志处理流水线
# Filter by content
{job="application"} |= "error"

# Exclude content
{job="application"} != "debug"

# Regex filter
{job="application"} |~ "user_id=[0-9]+"

# JSON parsing
{job="application"} | json | level="error"

# Line format
{job="application"} | json | line_format "{{.level}}: {{.message}}"
指标查询
# Count logs per second
count_over_time({job="application"}[5m])

# Rate of errors
rate({job="application", level="error"}[5m])

# Sum by label
sum by (level) (count_over_time({job="application"}[5m]))

# Top services by error count
topk(5, sum by (service) (count_over_time({level="error"}[1h])))
聚合
# Average log line length
avg_over_time({job="application"} | unwrap line_length [5m])

# Percentile of numeric field
quantile_over_time(0.95, {job="application"} | json | unwrap response_time [5m])

# Error percentage
sum(rate({job="application", level="error"}[5m])) 
/ 
sum(rate({job="application"}[5m])) * 100

流水线阶段

# promtail-config.yaml
pipeline_stages:
  # Parse JSON logs
  - json:
      expressions:
        level: level
        message: msg
        trace_id: trace_id

  # Extract with regex
  - regex:
      expression: 'user_id=(?P<user_id>\d+)'

  # Add labels from parsed fields
  - labels:
      level:
      user_id:

  # Modify timestamp
  - timestamp:
      source: timestamp
      format: '2006-01-02T15:04:05.000Z'

  # Filter logs
  - match:
      selector: '{level="debug"}'
      action: drop

  # Add static labels
  - static_labels:
      environment: production

  # Modify log line
  - template:
      source: message
      template: '{{ ToUpper .Value }}'

Grafana 集成

数据源配置
# grafana/provisioning/datasources/loki.yaml
apiVersion: 1

datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    isDefault: false
    jsonData:
      maxLines: 1000
仪表板面板
{
  "title": "Application Logs",
  "type": "logs",
  "datasource": "Loki",
  "targets": [
    {
      "expr": "{job=\"application\"} | json",
      "refId": "A"
    }
  ],
  "options": {
    "showTime": true,
    "showLabels": true,
    "wrapLogMessage": true
  }
}

记录规则

# loki-rules.yaml
groups:
  - name: error_rates
    interval: 1m
    rules:
      - record: job:log_errors:rate5m
        expr: |
          sum by (job) (rate({level="error"}[5m]))

告警

# loki-alerts.yaml
groups:
  - name: log_alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({level="error"}[5m])) > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate in logs"
          description: "Error rate is {{ $value }} errors/second"

常见问题

问题：内存使用率过高

问题：Loki 占用过多内存 解决方案：降低 max_query_series，并限制查询的时间范围

问题：日志未显示

问题：Promtail 未发送日志 解决方案：检查 positions 文件，确认文件路径，并检查标签配置

问题：查询超时

问题：LogQL 查询超时 解决方案：添加更具体的标签筛选条件，并缩短时间范围

问题：摄取速率限制

问题：日志被丢弃 解决方案：提高 limits_config 中的 per_stream_rate_limit

最佳实践

  • 使用有意义的标签（避免高基数）
  • 先按标签筛选，再筛选日志内容
  • 使用 Promtail 在采集时解析日志
  • 设置适当的保留期限
  • 为常见查询使用记录规则
  • 为大型部署实现适当的多租户
  • 监控 Loki 自身的指标
  • 使用块缓存以提升性能

相关技能

评论

支持 Markdown