KubeSense Traces (APM)

Use for anything about requests: latency, p99/p95, error rate, HTTP method, status code, endpoint, service-to-service calls, downstream dependencies. (Log lines and panics → [kubesense-logs](../kubesense-logs/SKILL.md).)

Requires the KubeSense MCP server; see [kubesense-mcp](../kubesense-mcp/SKILL.md). Tools resolve RBAC against the traces module.

Tools

get-distributed-trace is the highest-leverage tool here. A single span tells you something failed; the trace tells you which hop failed and where the time went.

Field Names — Catalog Labels Only

Storage column names are rejected at every input slot.

Also parse but hidden from discovery: timestamp (→ starttimestamp), duration, primaryworkload/primarynamespace (→ perspective), associateworkload/associatenamespace (→ partner_).

service vs workload

Prefer service for application identity — it is the cross-platform identifier (Kubernetes, Docker, legacy). Use workload only when you specifically need Kubernetes topology.

[!NOTE]
This is the opposite of the guidance for alert rules, where workload groups more
reliably. See [kubesense-alerts](../kubesense-alerts/SKILL.md) — the alert engine
resolves service against a rollup column that is only populated for
SDK-instrumented services.

[!WARNING]
node is not valid on traces — it is node_name. (Logs are the reverse: node,
not node_name.) Never carry a field name across signals.

There is no body field on traces. Text search over a message is logs-only.

Enum values are lowercase on traces

status = error — lowercase, unlike logs' UPPERCASE type = ERROR.

statuscode is a string field, so its values stay quoted strings even when they look numeric: statuscode IN (500, 502) yields the strings "500", "502". Range comparisons therefore do not work — enumerate the codes, or use status_code LIKE "5%" for a class.

Read casing from the enumvalues column, not example — the example generator returns a blank cell for method, protocol, and statuscode.

Duration: ms in, ns out

[!IMPORTANT]
Filtering duration takes MILLISECONDS. Aggregating duration returns NANOSECONDS.
These are different units in the same field, and getting it wrong is silent.

where: "duration > 500"        →  spans slower than 500 milliseconds

The parser multiplies your value by 1e6 to reach the nanosecond storage column. So:

[!WARNING]
The search-traces tool description shows "duration > 250000" as an example. That is
wrong — it means 250,000 ms = 250 seconds and will match nothing. Ignore it.

Aggregation is the other direction — p95/avg/max over fields: [{"field": "duration"}] come back in nanoseconds:

Divide by 1e6 to report milliseconds. There is no duration_ms field at the MCP layer — it is rejected as an unknown field.

Trace attributes: group-by only

[!WARNING]
@attr in a traces WHERE clause does not work. It is rejected with
unknown field "<key>" for signal=traces, because the parser clears the attribute
discriminator during the ARRAY_MAP remap and the field then fails column validation.
The tool descriptions advertise @db.system = postgresql as working; it does not.

Trace attributes do work everywhere the field is passed structurally — set the flag instead of a prefix:

"group_by_fields": [ { "field": "db.system", "is_attribute": true } ]

Valid in groupbyfields, fields, requiredfields, and sortby. To filter by an attribute value, group by it and read the series labels, or filter on an equivalent top-level column.

(Log attributes are unaffected — @user.id = abc works fine on logs.)

analyze-traces

{
  "from_time": "2026-07-30T10:00:00Z",
  "to_time": "2026-07-30T11:00:00Z",
  "query_type": "range",
  "group_by_fields": [ { "field": "service", "is_attribute": false } ],
  "fields": [ { "field": "duration", "is_attribute": false } ],
  "value_operation": "p95"
}

valueoperation: rowcount, uniquecount, avg, sum, min, max, p50, p90, p95, p99. count is not valid — it is rowcount, despite what the schema's summary line suggests.

Error rate as a percentage needs two queries plus a formula → analyze-telemetry. See [kubesense-mcp](../kubesense-mcp/SKILL.md).

Output is columnar JSON: for range, one shared timestamps array plus a positionally aligned values array per series; for instant, one scalar value per series.

search-traces

Same input shape as search-logs. Output is TSV with a # totalcount=<N> isfree_search=<bool> header.

Pagination — and its hard limit

Page by narrowing totime to the minimum starttimestamp returned (not timestamp, which is the span's end time; start_timestamp is the sort and filter key). The window is half-open, so the cursor row is not repeated.

[!WARNING]
Exhaustive paging is not achievable for traces. Rows are ordered
`toStartOfHour(starttimestamp) DESC, observationpoint DESC, cluster DESC,
starttimestamp DESC. Both observationpoint and cluster` outrank the timestamp,
so pinning clusters alone is not enough — spans at other observation points in the
same hour get skipped. And observation_point is not in the trace field catalog, so
the where clause cannot pin it (the tool description suggests
"observation_point = s-app"; that is rejected as an unknown field).

Practical consequences: keep the window short enough that total_count fits in one
page, or use analyze-traces. Treat any paged trace scan as a sample.

Touching duration, traceid, spanid, or any attribute sets isfreesearch and routes to the raw table — slower, no rollups.

get-distributed-trace

The span tree for one request. Start here when investigating a failing or slow request.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "from_time": "2026-07-30T10:20:00Z",
  "to_time": "2026-07-30T10:40:00Z",
  "max_spans": 200
}

traceid, fromtime, totime are all required. The window must contain the trace's starttimestamp — traces are time-partitioned, so a window that misses it finds nothing. Use ~10 minutes either side. max_spans defaults to 200, max 500.

[!NOTE]
Key on trace_id, not a span's kuid. One trace has many spans; they all share the
trace_id.

Reading the output

Block 1 — one row per span in preorder (a span is immediately followed by its children), with a tree column drawing the call chain. Key columns:

Block 2 — ## services, one row per (cluster, workload), ordered worst-first by errors then self time. Read this before the span rows to pick your suspect. It always covers every span, even when block 1 was truncated.

Two diagnostic rules

• A trace with no error may still be the problem. The top row of ## services is the

hop that burned the time.

• Large durationus + small selftime_us = the service was waiting, not working.

Investigate what it called (its children), not the service itself. Conversely, if a span errored but every child succeeded, the fault is in that service.

namespace may be blank when spans are attributed to a collector agent rather than the workload's pod; take it from a span row that has one, or from get-workload-detail. Rarely it lists several joined by |, meaning that cluster runs the same workload name in more than one namespace.

Investigation Flow

1. get-distributed-trace(trace_id, ±10min)   → the call chain, errors marked
2. read "## services"                        → who owns the errors / the time
3. search-logs                               → scoped to that workload + the failing
                                               span's start_time/end_time (widen a few s)
4. get-recent-changes / get-infra-issues     → why: deploy, OOM, probe failure
5. analyze-metrics                           → confirm resource pressure

Steps 4–5 use [kubesense-infra](../kubesense-infra/SKILL.md) and [kubesense-metrics](../kubesense-metrics/SKILL.md).

Rules

1. get-trace-or-log-fields with signal: "traces" first, same window as the query.
2. Catalog labels only: service, instance, method, status_code, resource,

role, protocol. Never appservice, podname, subtype, returncode, clusteredresource, kind, protocol_type.

1. node_name on traces, node on logs. Never carry a field across signals.
2. Prefer service over workload for app identity — but the alert engine prefers

workload.

1. duration in WHERE is milliseconds; aggregated duration is nanoseconds. Divide

results by 1e6 to report ms. duration_ms does not exist here.

1. @attr filters do not work on traces. Use is_attribute: true in

groupbyfields/fields instead.

1. Enum values are lowercase (status = error, role = server); status_code is a

string, so enumerate codes or use LIKE "5%" rather than a range.

1. valueoperation is rowcount, not count.
2. Add role = server when counting requests, or client spans double-count.
3. get-distributed-trace needs a window containing the trace's start; widen it first if

nothing is found.

1. Don't try to page traces exhaustively — observation_point can't be pinned. Shorten

the window or aggregate.