prometheus

Prometheus API Skill

Query Prometheus monitoring systems via HTTP API at /api/v1.

Quick Reference

Instant Query

curl 'http://<prometheus>:9090/api/v1/query?query=<promql>&time=<timestamp>'

Range Query

curl 'http://<prometheus>:9090/api/v1/query_range?query=<promql>&start=<ts>&end=<ts>&step=<duration>'

Response Format

All responses return JSON:

{
  "status": "success" | "error",
  "data": <result>,
  "errorType": "<string>",
  "error": "<string>",
  "warnings": ["<string>"]
}

HTTP codes: 400 (bad params), 422 (expression error), 503 (timeout).

Query Endpoints

Endpoint	Purpose	Key Parameters
/api/v1/query	Instant query	query, time, timeout, limit
/api/v1/query_range	Range query	query, start, end, step, timeout, limit
/api/v1/format_query	Format PromQL	query
/api/v1/series	Find series by labels	match[], start, end, limit
/api/v1/labels	List label names	start, end, match[], limit
/api/v1/label/<name>/values	Label values	start, end, match[], limit
/api/v1/query_exemplars	Query exemplars	query, start, end

Metadata & Status Endpoints

Endpoint	Purpose
/api/v1/targets	Target discovery status (state=active|dropped|any)
/api/v1/targets/metadata	Metric metadata from targets
/api/v1/metadata	All metric metadata
/api/v1/rules	Alerting/recording rules
/api/v1/alerts	Active alerts
/api/v1/alertmanagers	Alertmanager discovery
/api/v1/status/config	Current config YAML
/api/v1/status/flags	CLI flags
/api/v1/status/runtimeinfo	Runtime info
/api/v1/status/buildinfo	Build info
/api/v1/status/tsdb	TSDB cardinality stats
/api/v1/status/walreplay	WAL replay progress

Admin Endpoints (require --web.enable-admin-api)

Endpoint	Method	Purpose
/api/v1/admin/tsdb/snapshot	POST	Create TSDB snapshot
/api/v1/admin/tsdb/delete_series	POST	Delete series (match[], start, end)
/api/v1/admin/tsdb/clean_tombstones	POST	Clean deleted data

Common PromQL Patterns

# Rate of counter over 5m
rate(http_requests_total[5m])

# Sum by label
sum by (job) (rate(http_requests_total[5m]))

# Percentile from histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Filter by label
up{job="prometheus", instance=~".*:9090"}

# Increase over time
increase(http_requests_total[1h])

# Average over time range
avg_over_time(process_cpu_seconds_total[5m])

Result Types

• vector: [{"metric": {...}, "value": [timestamp, "value"]}]
• matrix: [{"metric": {...}, "values": [[ts, "val"], ...]}]
• scalar: [timestamp, "value"]
• string: [timestamp, "string"]

Scripts

Query script: scripts/prom_query.py

# Instant query
python scripts/prom_query.py http://localhost:9090 'up'

# Range query
python scripts/prom_query.py http://localhost:9090 'rate(http_requests_total[5m])' \
  --start '2024-01-01T00:00:00Z' --end '2024-01-01T01:00:00Z' --step '1m'

# Output: table, json, csv
python scripts/prom_query.py http://localhost:9090 'up' --format table

Health check: scripts/prom_health.py

python scripts/prom_health.py http://localhost:9090

Detailed Reference

For complete API documentation: references/api_reference.md

For PromQL functions: references/promql_functions.md

---

Gotchas

• rate() over a counter that resets too often: math is correct but meaningless — use increase() and divide by interval explicitly when counters don't survive scrapes.
• up{} per-target gauge: a flaky target shows up=0 but doesn't trigger alerts unless for is met. Set short for for liveness, long for noise.
• Recording rules evaluate at fixed interval; missed evaluations don't backfill — gaps in the recording series during incidents.
• Federation match[] parameter requires ALL matchers to match — an empty matcher returns no series, which looks like a working query with no data.
• Stale-marker semantics: a series stops being scraped → stale marker after 5 min by default → queries see "no data" not "0". Affects alerts on absent().
• Service Discovery + relabel_config: a bad regex in keep action silently drops all targets — verify with /api/v1/targets after each config change.