kubesense-traces

KubeSense Traces (APM)

Use for anything about requests: latency, p99/p95, error rate, HTTP method, status code, endpoint, service-to-service calls, downstream dependencies. (Log lines and panics → kubesense-logs.)

Requires the KubeSense MCP server; see kubesense-mcp. Tools resolve RBAC against the traces module.

Tools

Tool	Use when
get-trace-or-log-fields (signal: "traces")	Always, first
analyze-traces	Latency percentiles, error rates, counts, trends
search-traces	Read individual spans
get-distributed-trace	You have a trace_id and need the full call chain

get-distributed-trace is the highest-leverage tool here. A single span tells you something failed; the trace tells you which hop failed and where the time went.

Field Names — Catalog Labels Only

Storage column names are rejected at every input slot.

Concept	Use this label	Storage column (rejected)
service identity	service	app_service
k8s workload	workload	—
pod	instance	pod_name
container	container	container_name
node	node_name	—
HTTP method	method	subtype
HTTP status code	status_code	return_code
resource / endpoint	resource	clustered_resource
span role	role	kind
protocol	protocol	protocol_type
span outcome	status	—
latency	duration	—
external call?	request_type	is_external
issue reason	reason	issue_reason
customer / tenant	domain	customer_identifier
trace / span id	trace_id, span_id	—
issue id	issue_id	—
namespace, cluster, region, app_version, env_type, source, server, client, server_namespace, client_namespace, operation_name, partner_cluster	same	—

Also parse but hidden from discovery: timestamp (→ start_timestamp), duration, primary_workload/primary_namespace (→ perspective_*), associate_workload/associate_namespace (→ partner_*).

service vs workload

Prefer service for application identity — it is the cross-platform identifier (Kubernetes, Docker, legacy). Use workload only when you specifically need Kubernetes topology.

Note

This is the opposite of the guidance for alert rules, where workload groups more reliably. See kubesense-alerts — the alert engine resolves service against a rollup column that is only populated for SDK-instrumented services.

Warning

node is not valid on traces — it is node_name. (Logs are the reverse: node, not node_name.) Never carry a field name across signals.

There is no body field on traces. Text search over a message is logs-only.

Enum values are lowercase on traces

Field	Values
status	error, ok
role	server, client
protocol	HTTP, gRPC, TCP, MongoDB, Redis, MySQL, PostgreSQL
method	GET, POST, PUT, DELETE, PATCH
source	eBPF, OTel
status_code	200, 400, 401, 403, 404, 500, 502, 503, 504

status = error — lowercase, unlike logs' UPPERCASE type = ERROR.

status_code is a string field, so its values stay quoted strings even when they look numeric: status_code IN (500, 502) yields the strings "500", "502". Range comparisons therefore do not work — enumerate the codes, or use status_code LIKE "5%" for a class.

Read casing from the enum_values column, not example — the example generator returns a blank cell for method, protocol, and status_code.

Duration: ms in, ns out

Important

Filtering duration takes MILLISECONDS. Aggregating duration returns NANOSECONDS. These are different units in the same field, and getting it wrong is silent.

where: "duration > 500"        →  spans slower than 500 milliseconds

The parser multiplies your value by 1e6 to reach the nanosecond storage column. So:

You want	Write
slower than 250ms	duration > 250
slower than 1s	duration > 1000
slower than 1.5s	duration > 1.5e3 or duration > 1500

Warning

The search-traces tool description shows "duration > 250000" as an example. That is wrong — it means 250,000 ms = 250 seconds and will match nothing. Ignore it.

Aggregation is the other direction — p95/avg/max over fields: [{"field": "duration"}] come back in nanoseconds:

Result	Means
250000000	250 ms
1000000000	1 s

Divide by 1e6 to report milliseconds. There is no duration_ms field at the MCP layer — it is rejected as an unknown field.

Trace attributes: group-by only

Warning

@attr in a traces WHERE clause does not work. It is rejected with unknown field "<key>" for signal=traces, because the parser clears the attribute discriminator during the ARRAY_MAP remap and the field then fails column validation. The tool descriptions advertise @db.system = postgresql as working; it does not.

Trace attributes do work everywhere the field is passed structurally — set the flag instead of a prefix:

"group_by_fields": [ { "field": "db.system", "is_attribute": true } ]

Valid in group_by_fields, fields, required_fields, and sort_by. To filter by an attribute value, group by it and read the series labels, or filter on an equivalent top-level column.

(Log attributes are unaffected — @user.id = abc works fine on logs.)

analyze-traces

{
  "from_time": "2026-07-30T10:00:00Z",
  "to_time": "2026-07-30T11:00:00Z",
  "query_type": "range",
  "group_by_fields": [ { "field": "service", "is_attribute": false } ],
  "fields": [ { "field": "duration", "is_attribute": false } ],
  "value_operation": "p95"
}

value_operation: row_count, unique_count, avg, sum, min, max, p50, p90, p95, p99. count is not valid — it is row_count, despite what the schema's summary line suggests.

Question	Shape
p95 latency by service, over time	range, group service, fields: [duration], p95
top 5 services by errors, now	instant, where: "status = error", group service, row_count, sort_direction: DESC, limit: 5
errors by status code	instant, where: "status = error", group status_code, row_count
slowest endpoints	instant, group resource, fields: [duration], p99, sort_direction: DESC
server-side only	add where: "role = server" — otherwise client spans double-count

Error rate as a percentage needs two queries plus a formula → analyze-telemetry. See kubesense-mcp.

Output is columnar JSON: for range, one shared timestamps array plus a positionally aligned values array per series; for instant, one scalar value per series.

search-traces

Same input shape as search-logs. Output is TSV with a # total_count=<N> is_free_search=<bool> header.

Pagination — and its hard limit

Page by narrowing to_time to the minimum start_timestamp returned (not timestamp, which is the span's end time; start_timestamp is the sort and filter key). The window is half-open, so the cursor row is not repeated.

Warning

Exhaustive paging is not achievable for traces. Rows are ordered toStartOfHour(start_timestamp) DESC, observation_point DESC, cluster DESC, start_timestamp DESC. Both observation_point and cluster outrank the timestamp, so pinning clusters alone is not enough — spans at other observation points in the same hour get skipped. And observation_point is not in the trace field catalog, so the where clause cannot pin it (the tool description suggests "observation_point = s-app"; that is rejected as an unknown field).

Practical consequences: keep the window short enough that total_count fits in one page, or use analyze-traces. Treat any paged trace scan as a sample.

Touching duration, trace_id, span_id, or any attribute sets is_free_search and routes to the raw table — slower, no rollups.

get-distributed-trace

The span tree for one request. Start here when investigating a failing or slow request.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "from_time": "2026-07-30T10:20:00Z",
  "to_time": "2026-07-30T10:40:00Z",
  "max_spans": 200
}

trace_id, from_time, to_time are all required. The window must contain the trace's start_timestamp — traces are time-partitioned, so a window that misses it finds nothing. Use ~10 minutes either side. max_spans defaults to 200, max 500.

Note

Key on trace_id, not a span's kuid. One trace has many spans; they all share the trace_id.

Reading the output

Block 1 — one row per span in preorder (a span is immediately followed by its children), with a tree column drawing the call chain. Key columns:

Column	Read it for
tree	who called whom
status / return_code	which span failed
duration_us	total span time in microseconds, including everything it called
self_time_us	time in this service, excluding children — the bottleneck ranking
start_time / end_time	RFC3339 ms — paste straight into search-logs
workload / namespace / cluster	scope for follow-up log/metric queries
resource_name / protocol_type / subtype	what was called

Block 2 — ## services, one row per (cluster, workload), ordered worst-first by errors then self time. Read this before the span rows to pick your suspect. It always covers every span, even when block 1 was truncated.

Two diagnostic rules

• A trace with no error may still be the problem. The top row of ## services is the hop that burned the time.
• Large duration_us + small self_time_us = the service was waiting, not working. Investigate what it called (its children), not the service itself. Conversely, if a span errored but every child succeeded, the fault is in that service.

namespace may be blank when spans are attributed to a collector agent rather than the workload's pod; take it from a span row that has one, or from get-workload-detail. Rarely it lists several joined by |, meaning that cluster runs the same workload name in more than one namespace.

Investigation Flow

1. get-distributed-trace(trace_id, ±10min)   → the call chain, errors marked
2. read "## services"                        → who owns the errors / the time
3. search-logs                               → scoped to that workload + the failing
                                               span's start_time/end_time (widen a few s)
4. get-recent-changes / get-infra-issues     → why: deploy, OOM, probe failure
5. analyze-metrics                           → confirm resource pressure

Steps 4–5 use kubesense-infra and kubesense-metrics.

Rules

1. get-trace-or-log-fields with signal: "traces" first, same window as the query.
2. Catalog labels only: service, instance, method, status_code, resource, role, protocol. Never app_service, pod_name, subtype, return_code, clustered_resource, kind, protocol_type.
3. node_name on traces, node on logs. Never carry a field across signals.
4. Prefer service over workload for app identity — but the alert engine prefers workload.
5. duration in WHERE is milliseconds; aggregated duration is nanoseconds. Divide results by 1e6 to report ms. duration_ms does not exist here.
6. @attr filters do not work on traces. Use is_attribute: true in group_by_fields/fields instead.
7. Enum values are lowercase (status = error, role = server); status_code is a string, so enumerate codes or use LIKE "5%" rather than a range.
8. value_operation is row_count, not count.
9. Add role = server when counting requests, or client spans double-count.
10. get-distributed-trace needs a window containing the trace's start; widen it first if nothing is found.
11. Don't try to page traces exhaustively — observation_point can't be pinned. Shorten the window or aggregate.