oodle-traces

Oodle Traces — Search and Analysis

This skill teaches the agent to search Oodle traces with the right service / time / duration filters, to fetch full trace detail by ID, and to run TraceQL.

Prerequisites

brew install oodle-ai/oodle/oodle
oodle configure

Confirm the traces endpoint works (use a recent time window):

oodle traces list --start -15m --end now -o json | jq 'length'

Command Execution Order

Before running any oodle command:

1. Check whether a service name and approximate time window are already in context.
2. If not, run oodle traces label-values resource::service.name --start -1h --end now -o json to discover services that emitted traces recently.
3. Build the search with --service, --start, --end and at least one of --tags '{"span_status":"Error"}' / --min-duration / --operation. For conditions on any span attribute, use oodle traces traceql search.
4. To inspect a single trace, run oodle traces get <trace-id> --start -1h --end now -o json.
5. Do not run oodle traces list without a --service filter on production environments.

Quick Reference

Task	Command
List traces in a service	oodle traces list --service <svc> --start -1h --end now -o json
List error traces only	oodle traces list --service <svc> --tags '{"span_status":"Error"}' --start -1h --end now -o json
List slow traces (>500ms)	oodle traces list --service <svc> --min-duration 500ms --start -1h --end now -o json
Filter by operation	oodle traces list --service <svc> --operation GET_/users --start -1h --end now -o json
Get full trace	oodle traces get <trace-id> --start -1h --end now -o json
TraceQL search	oodle traces traceql search '{ status = error }' --start -1h --end now
TraceQL metrics	oodle traces traceql metrics '{ status = error } | rate() by (resource.service.name)' --start -1h --end now --step 1m

Common Operations

Searching for errors in the last hour

# ✅ CORRECT — scoped to one service, narrow time window, error filter on
oodle traces list --service api --tags '{"span_status":"Error"}' --start -1h --end now -o json

# ❌ WRONG — no service, no error filter, huge result set
oodle traces list --start -1h --end now -o json

Searching for slow traces

# ✅ CORRECT — slow checkout traces in the last hour
oodle traces list --service checkout --min-duration 500ms --start -1h --end now -o json

# ✅ CORRECT — narrow further by operation when you know the endpoint
oodle traces list --service checkout --operation POST_/cart --min-duration 500ms --start -1h --end now -o json

# ❌ WRONG — listing everything and sorting client-side wastes the API budget
oodle traces list --service checkout --start -1h --end now -o json | jq '.data | sort_by(.spans[0].duration) | reverse | .[0:10]'

Fetching a full trace

# ✅ CORRECT — list to discover trace ids, then get the one of interest
TRACE_ID=$(oodle traces list --service api --tags '{"span_status":"Error"}' --start -15m --end now -o json | jq -r '.data[0].traceID')
oodle traces get "$TRACE_ID" --start -1h --end now -o json

# ❌ WRONG — `oodle traces get` without an id (it expects the id as a positional arg)
oodle traces get

# ❌ WRONG — `oodle traces get` without --start and --end (both are required)
oodle traces get "$TRACE_ID" -o json

Time window flags

--start and --end are required for list and get. They accept the same formats as the rest of the CLI:

Format	Example
Epoch seconds	1731628800
RFC 3339	2026-01-02T00:00:00Z
now	now
Relative duration	-1h, -30m, -7d

# ✅ CORRECT
oodle traces list --service api --start -30m --end now -o json

# ❌ WRONG — human strings are not parsed
oodle traces list --service api --start "30 minutes ago"

TraceQL

oodle traces traceql has the alias oodle traces tql. Put the query in single quotes so the shell does not change {, }, ", or !.

# Find traces (filter only; a metrics query is rejected)
oodle traces traceql search '<filter>' [--start -1h] [--end now] [--limit 20]

# Time series from spans (needs a metrics function; a plain filter is rejected)
oodle traces traceql metrics '<filter> | <function>' [--start -1h] [--end now] [--step 5m] [--exemplars N] [-o graph|stats|json]

# Attribute names and values (reads only the last hour; no --start/--end)
oodle traces traceql tags [--scope resource|span|intrinsic|all] [-q '<filter>']
oodle traces traceql tag-values <tag> [-q '<filter>']

Flag	Meaning
--start, --end	search and metrics: default -1h and now.
--limit	search: maximum traces (default 20).
--step	metrics: 30s, 5m, 1h, 1d, or a number of seconds. Default 1m, made larger to keep 300 points or fewer per series.
--exemplars	metrics: maximum exemplar traces. Shown only with -o json or -o yaml.
-q, --query	tags, tag-values: only look at spans that match this filter.

Output: search prints one row per trace; -o json adds the matching spans and attributes. metrics prints one row per series (point count, last, min, max, average); -o graph draws a terminal chart, -o stats gives a summary, -o json gives all points.

Supported syntax

Part	Supported
Spanset filter	{ <condition> }
Attribute scopes	resource.<name>, span.<name>, and unscoped .<name> (matches span or resource)
Intrinsics	name, status, duration, kind
Comparison	= != =~ !~ > >= < <=
Combine in one spanset	&&, `
Union of spansets	`{ A }
Attribute present / absent	span.foo != nil, span.foo = nil
Literals	strings "x", numbers, durations 10s 250ms, status = error / ok / unset, kind = server / client / internal / producer / consumer
Metrics functions	rate(), count_over_time(), avg_over_time(<field>), min_over_time(<field>), max_over_time(<field>), sum_over_time(<field>), histogram_over_time(<field>), quantile_over_time(<field>, 0.5, 0.95, ...)
Grouping	by (<attribute>, ...) after a metrics function

Not supported yet

Syntax	Use instead
Scalar filters: { ... } | count() > 2, | avg(duration) > 1s	traceql metrics with count_over_time() or avg_over_time(), and read the values
{ A } && { B } (two different spans of one trace)	One spanset { A && B } when one span must match both
Structural operators >>, <<, ~	Search for each span on its own, then compare trace ids
parent. and link. scopes	span. or resource. attributes
rootName, traceDuration	name and duration of a single span (for example, the root span by its name)
Bare existence: { span.foo }	{ span.foo != nil }

If a query fails with "not supported", rewrite it with the table above. Do not tell the user that TraceQL is not available.

Examples

# Errors in one service
oodle traces traceql search '{ resource.service.name = "checkout" && status = error }' --start -1h --limit 20

# GenAI tool calls over 10 s
oodle traces traceql search '{ span.gen_ai.tool.name != nil && duration > 10s }' --start -24h --limit 50

# Error rate by service
oodle traces traceql metrics '{ status = error } | rate() by (resource.service.name)' --start -6h --step 5m

# p50 and p95 latency per route, as a terminal chart
oodle traces traceql metrics '{ kind = server } | quantile_over_time(duration, 0.5, 0.95) by (span.http.route)' --start -1h -o graph

# Error rate with 5 example traces, for drill-down
oodle traces traceql metrics '{ status = error } | rate()' --start -1h --exemplars 5 -o json

# Attribute names and values
oodle traces traceql tags --scope span
oodle traces traceql tag-values span.http.route -q '{ resource.service.name = "checkout" }'

Monitors run PromQL, not TraceQL. To alert on traces, write the monitor query on the oodle_trace_metrics metric, or on the oodle_genai_* metrics for GenAI spans (see the oodle-monitors skill).

Best Practices

Always specify --service to scope the search

Trace volume can be massive; an unfiltered list is expensive and rate-limited.

# ✅ CORRECT
oodle traces list --service api --start -1h --end now -o json

# ❌ WRONG — full-tenant scan
oodle traces list --start -1h --end now -o json

Use --tags to find failing traces, not client-side filtering

--tags '{"span_status":"Error"}' pushes the predicate to the server and returns matches faster.

# ✅ CORRECT
oodle traces list --service api --tags '{"span_status":"Error"}' --start -1h --end now -o json

# ❌ WRONG — fetches every trace then filters client-side
oodle traces list --service api --start -1h --end now -o json | jq '.data[] | select(.spans[].tags[] | .key=="error")'

Use --min-duration to find slow traces, not client-side sort

# ✅ CORRECT — server-side filter, returns ~the right number of rows
oodle traces list --service api --min-duration 500ms --start -1h --end now -o json

# ❌ WRONG — pulls everything, sorts on the client
oodle traces list --service api --start -1h --end now -o json | jq '.data | sort_by(.spans[0].duration) | reverse'

Keep time windows ≤ 24h for list; widen by re-querying, not by lowering --start

A 7-day list returns too many rows and risks timeouts. Page by hour.

# ✅ CORRECT — bounded window
oodle traces list --service api --tags '{"span_status":"Error"}' --start -1h --end now -o json

# ❌ WRONG — multi-day list
oodle traces list --service api --tags '{"span_status":"Error"}' --start -7d --end now -o json

Failure Handling

Error	Cause	Fix
401 Unauthorized	Invalid or missing API key	Run oodle configure or set OODLE_API_KEY
404 Not Found	Trace ID does not exist or has aged out of retention	Re-list with a recent --start to find a current trace id
connection refused	Wrong OODLE_DEPLOYMENT URL	Check OODLE_DEPLOYMENT env var
Empty result	Wrong service name or time window outside retention	List recent services: oodle traces label-values resource::service.name --start -1h --end now; reduce the --start offset if it is past retention, or widen it if the window is too short (the CLI prints the range it read)
Request times out	Time window too large or no service filter	Narrow --start/--end, add --service, add --min-duration or --tags
429 Too Many Requests	Concurrent trace fetches	Add --retries 3; throttle to <5 requests / second
required flag(s) "end", "start" not set	list and get need both flags	Add --start -1h --end now
400 on --min-duration	Duration has no unit	Use 500ms, 2s
TraceQL ... is not supported	Syntax from "Not supported yet"	Rewrite with the replacements in that table
search rejects the query	It has a metrics function (| rate() …)	Use oodle traces traceql metrics
metrics rejects the query	It has no metrics function	Add one, or use oodle traces traceql search

References

• Oodle CLI repo
• Oodle docs
• OpenTelemetry trace concepts
• TraceQL