关于 datadog
面向基础设施和应用可观测性的 Datadog 使用技能,介绍 Agent、日志、APM、指标、仪表板和告警配置,适用于负责监控的团队。
主要功能
- 01安装和配置 Datadog Agent
- 02采集文件、容器和 Kubernetes 日志
- 03配置 APM 与分布式追踪
- 04提交自定义指标
- 05创建仪表板和告警监视器
适用场景
- 01监控云基础设施和应用
- 02追踪跨服务请求
- 03汇总应用与基础设施数据
- 04为错误率或延迟配置告警
安装
复制下面的命令,在终端中运行。
$
npx skills add https:/ / github.com/ bagelhole/ devops-security-agent-skills --skill datadog安装后在 Claude Code 中输入/datadog确认已加载。
附带文件references/datadog-integrations.md
详细介绍
Datadog
使用 Datadog 统一可观测性平台监控基础设施和应用。
何时使用此技能
在以下情况下使用此技能:
- 实施企业级监控
- 设置 APM 和分布式追踪
- 为基础设施和应用创建统一仪表板
- 配置智能告警
- 监控云基础设施(AWS、Azure、GCP)
前提条件
- Datadog 账号和 API 密钥
- Agent 安装权限
- 用于 APM 的应用代码访问权限
Agent 安装
Linux
# Install agent
DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"
# Or via package manager
apt-get update && apt-get install datadog-agent
# Configure API key
echo "api_key: YOUR_API_KEY" >> /etc/datadog-agent/datadog.yaml
# Start agent
systemctl start datadog-agent
systemctl enable datadog-agent
Docker
# docker-compose.yml
version: '3.8'
services:
datadog-agent:
image: gcr.io/datadoghq/agent:7
environment:
- DD_API_KEY=${DD_API_KEY}
- DD_SITE=datadoghq.com
- DD_LOGS_ENABLED=true
- DD_APM_ENABLED=true
- DD_PROCESS_AGENT_ENABLED=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /proc/:/host/proc/:ro
- /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
ports:
- "8126:8126" # APM
- "8125:8125/udp" # DogStatsD
Kubernetes
# Using Helm
helm repo add datadog https://helm.datadoghq.com
helm install datadog datadog/datadog \
--set datadog.apiKey=${DD_API_KEY} \
--set datadog.site=datadoghq.com \
--set datadog.logs.enabled=true \
--set datadog.apm.portEnabled=true \
--set datadog.processAgent.enabled=true \
--namespace datadog \
--create-namespace
Agent 配置
# /etc/datadog-agent/datadog.yaml
api_key: YOUR_API_KEY
site: datadoghq.com
# Hostname
hostname: myserver.example.com
# Tags applied to all metrics
tags:
- env:production
- service:myapp
- team:platform
# Log collection
logs_enabled: true
# APM
apm_config:
enabled: true
apm_dd_url: https://trace.agent.datadoghq.com
# Process monitoring
process_config:
enabled: true
# Container monitoring
container_collect_all: true
docker_labels_as_tags:
app: service
environment: env
集成配置
MySQL
# /etc/datadog-agent/conf.d/mysql.d/conf.yaml
init_config:
instances:
- host: localhost
port: 3306
username: datadog
password: <PASSWORD>
tags:
- env:production
options:
replication: true
extra_status_metrics: true
PostgreSQL
# /etc/datadog-agent/conf.d/postgres.d/conf.yaml
init_config:
instances:
- host: localhost
port: 5432
username: datadog
password: <PASSWORD>
dbname: mydb
collect_activity_metrics: true
collect_database_size_metrics: true
NGINX
# /etc/datadog-agent/conf.d/nginx.d/conf.yaml
init_config:
instances:
- nginx_status_url: http://localhost:80/nginx_status
tags:
- env:production
日志收集
基于文件的日志
# /etc/datadog-agent/conf.d/myapp.d/conf.yaml
logs:
- type: file
path: /var/log/myapp/*.log
service: myapp
source: python
sourcecategory: custom
tags:
- env:production
- type: file
path: /var/log/nginx/access.log
service: nginx
source: nginx
log_processing_rules:
- type: exclude_at_match
name: exclude_healthchecks
pattern: health_check
Docker 日志
# docker-compose.yml
services:
myapp:
labels:
com.datadoghq.ad.logs: '[{"source": "python", "service": "myapp"}]'
Kubernetes 日志
# Pod annotation
apiVersion: v1
kind: Pod
metadata:
annotations:
ad.datadoghq.com/myapp.logs: |
[{
"source": "python",
"service": "myapp",
"log_processing_rules": [{
"type": "multi_line",
"name": "python_tracebacks",
"pattern": "^Traceback"
}]
}]
APM 配置
Python
from ddtrace import patch_all, tracer
# Automatic instrumentation
patch_all()
# Configure tracer
tracer.configure(
hostname='localhost',
port=8126,
service='myapp',
env='production',
version='1.0.0'
)
# Manual instrumentation
@tracer.wrap(service='myapp', resource='process_order')
def process_order(order_id):
with tracer.trace('validate_order') as span:
span.set_tag('order_id', order_id)
# Validation logic
with tracer.trace('save_order'):
# Save logic
pass
# Install library
pip install ddtrace
# Run with auto-instrumentation
ddtrace-run python app.py
Node.js
const tracer = require('dd-trace').init({
service: 'myapp',
env: 'production',
version: '1.0.0',
logInjection: true
});
// Manual instrumentation
const span = tracer.startSpan('custom_operation');
span.setTag('user_id', userId);
// ... operation
span.finish();
# Install library
npm install dd-trace
# Run with auto-instrumentation
DD_TRACE_ENABLED=true node --require dd-trace/init app.js
Go
import (
"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)
func main() {
tracer.Start(
tracer.WithService("myapp"),
tracer.WithEnv("production"),
tracer.WithServiceVersion("1.0.0"),
)
defer tracer.Stop()
// Manual span
span, ctx := tracer.StartSpanFromContext(ctx, "process_request")
defer span.Finish()
span.SetTag("user_id", userID)
}
自定义指标
DogStatsD
from datadog import DogStatsd
statsd = DogStatsd(host='localhost', port=8125)
# Counter
statsd.increment('myapp.orders.count', tags=['env:production'])
# Gauge
statsd.gauge('myapp.queue.size', queue_size, tags=['queue:orders'])
# Histogram
statsd.histogram('myapp.request.duration', response_time)
# Distribution
statsd.distribution('myapp.response_time', duration, tags=['endpoint:/api/orders'])
API 提交
from datadog_api_client import Configuration, ApiClient
from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client.v2.model.metric_payload import MetricPayload
from datadog_api_client.v2.model.metric_series import MetricSeries
from datadog_api_client.v2.model.metric_point import MetricPoint
configuration = Configuration()
with ApiClient(configuration) as api_client:
api = MetricsApi(api_client)
payload = MetricPayload(
series=[
MetricSeries(
metric="custom.metric.name",
type=MetricSeries.GAUGE,
points=[MetricPoint(value=42.0, timestamp=int(time.time()))],
tags=["env:production"]
)
]
)
api.submit_metrics(body=payload)
仪表板
仪表板 JSON
{
"title": "Application Overview",
"widgets": [
{
"definition": {
"type": "timeseries",
"title": "Request Rate",
"requests": [
{
"q": "sum:trace.http.request.hits{service:myapp}.as_rate()",
"display_type": "line"
}
]
}
},
{
"definition": {
"type": "query_value",
"title": "Error Rate",
"requests": [
{
"q": "sum:trace.http.request.errors{service:myapp}.as_rate() / sum:trace.http.request.hits{service:myapp}.as_rate() * 100"
}
],
"precision": 2
}
}
]
}
监控器(告警)
指标监控器
{
"name": "High Error Rate",
"type": "metric alert",
"query": "sum(last_5m):sum:trace.http.request.errors{service:myapp}.as_count() / sum:trace.http.request.hits{service:myapp}.as_count() > 0.05",
"message": "Error rate is {{value}}% for {{service.name}}. @slack-alerts",
"tags": ["service:myapp", "env:production"],
"options": {
"thresholds": {
"critical": 0.05,
"warning": 0.02
},
"notify_no_data": true,
"no_data_timeframe": 10
}
}
APM 监控器
{
"name": "High Latency Alert",
"type": "trace-analytics alert",
"query": "trace-analytics(\"service:myapp @http.status_code:2*\").rollup(\"avg\", \"@duration\").last(\"5m\") > 2000000000",
"message": "Average latency is above 2 seconds. @pagerduty",
"options": {
"thresholds": {
"critical": 2000000000
}
}
}
常见问题
问题:Agent 未上报
问题:Datadog 中没有显示数据
解决方案:检查 API 密钥,使用 datadog-agent status 确认 Agent 状态
问题:缺少追踪数据
问题:没有显示 APM 追踪数据
解决方案:确认已启用 APM,检查追踪器配置,并确认端口 8126
问题:标签基数过高
问题:自定义指标被丢弃
解决方案:减少唯一标签值,并使用分布指标代替直方图
最佳实践
- 使用一致的服务和环境标签
- 实施规范的标签命名约定
- 使用统一服务标记(service、env、version)
- 设置服务级监控器
- 为每项服务创建仪表板
- 实施日志与追踪数据关联
- 对延迟指标使用分布指标
- 配置合理的告警升级机制
相关技能
- prometheus-grafana - 开源替代方案
- alerting-oncall - 告警管理
- aws-vpc - AWS 监控
评论
支持 Markdown