mcpField
M
mlflow-trace-metrics

mlflow-trace-metrics

github.com/mlflowOther8216 installsUpdated 2d ago
Repository

About mlflow-trace-metrics

A skill for querying MLflow metrics across traces, spans, and assessments. It guides metric and aggregation selection, time-bounded queries, and interpretation of incomplete or missing results.

Key features

  1. 01Query traces, spans, or assessments
  2. 02Select metric and aggregation
  3. 03Set explicit time bounds
  4. 04Inspect dimensions and units
  5. 05Identify pagination and missing data

Use cases

  1. 01Measure token totals over a period
  2. 02Summarize latency distributions
  3. 03Count traces by status
  4. 04Review numeric assessment scores

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/mlflow/skills --skill mlflow-trace-metrics

After installing, type/mlflow-trace-metricsin Claude Code to confirm it loaded.

Bundled filesreferences/api_reference.mdscripts/fetch_metrics.py

Details

From the project README

Query MLflow metrics

Reuse server, experiment IDs and requested window. Select the exact metric, view, aggregation and grouping needed; token counts are not currency cost.

Prerequisites

For aggregate queries, read the metrics API reference. If the task also needs individual traces, read trace retrieval before searching; if unavailable, use the search guide.

Step 1: Check the client

Resolve scripts/fetch_metrics.py relative to this skill; use --help for flags. The metrics endpoint and helper are not supported for Databricks-managed tracking. For OSS they accept HTTP(S), not databricks:// profiles, and add no authentication headers. Use an existing supported authenticated client or report the blocker; never convert a Databricks profile to a workspace URL or request pasted tokens.

Step 2: Choose metrics

Select TRACES (default), SPANS or ASSESSMENTS using the API reference. Use COUNT for entities, SUM for tokens, AVG/P95 for distributions. Do not silently substitute metrics.

Step 3: Query the time window

Query explicit bounds: epoch milliseconds, timezone-qualified ISO timestamps, now, or relative values like -24h. Buckets are seconds and require both bounds.

python <SKILL_DIR>/scripts/fetch_metrics.py -s <SERVER> -x <EXPERIMENT_ID> \
  -m total_tokens -a SUM -t 3600 --start-time="-24h" --end-time=now -o json

Latency: -m latency -a AVG,P95 -d trace_name. Numeric quality: -v ASSESSMENTS -m assessment_value -a AVG -d assessment_name. Status counts: -m trace_count -a COUNT -d trace_status. Keep the same explicit window.

Verification

Inspect actual metric_name, dimensions and values; report exact experiment/window, metric, aggregation, units, filters and coverage. Empty data_points is not zero usage; absent token instrumentation can undercount. Quality summaries need per-scorer scales and valid/error/missing denominators: zero/false can be valid, scorer failures are not zero scores, and unrelated scorers should not be averaged.

A next_page_token means partial results; the helper exposes it but cannot fetch subsequent pages. Use an existing pagination-capable client or disclose incompleteness. Counts by status are not an error rate: derive ERROR / all statuses only with a complete stated denominator. Distinguish request/auth failures from empty results. Aggregates need representative trace evidence for causal diagnosis.

References

API reference: wire fields, metric/view compatibility and metrics-specific filters.

Comments

Markdown supported