ml-ai

Grafana Cloud AI & ML

ドキュメント: https://grafana.com/docs/grafana-cloud/alerting-and-irm/machine-learning/

ML アラート＋自動 RCA＋LLM 搭載アシスタントを、1 つの Grafana Cloud スタックで利用できます。

前提条件

• Grafana Cloud スタック（Pro / Advanced — ほとんどの機能は GA、一部はプレビュー段階です）
• ML / Sift / LLM プラグインのエンドポイント用の plugins:write 権限を持つ API トークン
• Dynamic Alerting の場合：予測対象のメトリクスについて、少なくとも 14 日間（理想的には 90 日間）の履歴

一般的なワークフロー

1. Dynamic Alerting による予測アラート

# 1. Create forecast job (Prophet — learns daily/weekly seasonality)
curl -X POST https://<stack>.grafana.net/api/plugins/grafana-ml-app/resources/ml/v1/forecast \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "cpu-forecast",
    "metric": "avg(rate(node_cpu_seconds_total{mode=\"user\"}[5m]))",
    "datasourceId": 1,
    "interval": 300,
    "trainingWindow": "90d",
    "forecastWindow": "7d",
    "algorithm": { "name": "prophet", "config": {} }
  }'

# 2. Verify job is producing the predicted-value metric (may take a few minutes).
#    <datasourceId> must match the datasourceId used above (find it via
#    GET /api/datasources), or run the query from Explore instead.
curl -s -H "Authorization: Bearer <token>" \
  'https://<stack>.grafana.net/api/datasources/proxy/<datasourceId>/api/v1/query?query=ml_forecast_upper{job="cpu-forecast"}' \
  | jq '.data.result | length'
# Expect > 0

# 3. Add an alert that fires when actual exceeds the upper bound
# expr:  avg(rate(node_cpu_seconds_total{mode="user"}[5m]))
#         > ml_forecast_upper{job="cpu-forecast"} * 1.1

2. 外れ値アラート — 1 つのサービスが同種のサービスから逸脱する場合

# 1. Create outlier job (DBSCAN — groups peers, flags the odd one)
curl -X POST https://<stack>.grafana.net/api/plugins/grafana-ml-app/resources/ml/v1/outlier \
  -H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
  -d '{
    "name": "service-error-outliers",
    "metric": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (service)",
    "datasourceId": 1,
    "interval": 300,
    "algorithm": { "name": "dbscan", "sensitivity": 0.5, "config": { "epsilon": 0.5 } }
  }'

# 2. Verify the score metric exists (<datasourceId> must match the
#    datasourceId used above, or run the query from Explore instead)
curl -s -H "Authorization: Bearer <token>" \
  'https://<stack>.grafana.net/api/datasources/proxy/<datasourceId>/api/v1/query?query=ml_outlier_score{job="service-error-outliers"}' \
  | jq '.data.result | length'

# 3. Alert when ml_outlier_score{job="service-error-outliers"} > 0.8 for 5m

3. Sift の調査を実行する

# 1. Trigger from API (or from Explore / Incident / OnCall)
curl -X POST https://<stack>.grafana.net/api/plugins/grafana-sift-app/resources/sift/v1/investigations \
  -H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
  -d '{ "name":"checkout-spike","start":"2024-02-01T10:00:00Z","end":"2024-02-01T10:30:00Z",
        "filters":{"service":"checkout","namespace":"production"} }'

# 2. The response includes an investigation ID — open it in the UI:
#    https://<stack>.grafana.net/a/grafana-sift-app/investigations/<id>
# 3. Verify analyses ran — each of the 8 checks shows ✔ or ✖ with linked evidence.

8 種類の分析の完全な表については、references/sift.md を参照してください。

4. LLM Plugin を接続する

# 1. Provision (provisioning/plugins/llm.yaml — see references/llm-and-graph.md)
apiVersion: 1
apps:
  - type: grafana-llm-app
    jsonData: { openAIUrl: https://api.openai.com, openAIModel: gpt-4o }
    secureJsonData: { openAIKey: sk-... }

# 2. Restart Grafana, then verify the health endpoint reports the configured provider
curl -s -H "Authorization: Bearer <token>" \
  https://<stack>.grafana.net/api/plugins/grafana-llm-app/health | jq
# Expect: {"status":"ok", ...}

# 3. Verify in a panel — open any panel, click the Assistant icon, ask "what does this query do?"

アシスタントの機能、Knowledge Graph の検索構文、Adaptive Metrics の推奨事項については、references/llm-and-graph.md を参照してください。

リソース

• Machine Learning のドキュメント
• Sift
• Grafana Assistant
• LLM Plugin