Observe

Transient, session-scoped monitoring and ad-hoc querying. Unlike manage-alerts (which creates a durable saved object in Kibana), observe polls in-process and returns once fired, once its window closes, or — in now / table mode — immediately.

Modes

Decision tree — pick based on tense FIRST, then on shape

Does the user phrase it as past or future?

PAST tense / windowed lookback                    FUTURE tense / live monitoring
("what WAS / over the past N /                    ("watch / monitor / wait until /
 in the last N / how did X look")                  live-sample / for the NEXT N")
       │                                                  │
       ▼                                                  ▼
Is it a single number or a series?                Is there a threshold to fire on?
       │                                                  │
   ┌───┴────┐                                          ┌──┴──┐
single   series                                       yes    no
   │        │                                          │      │
   ▼        ▼                                          ▼      ▼
 now     table                                      metric  metric
                                                    (with    (live-
                                                  condition) sample)

Other:
  • "tell me when anything unusual fires" → anomaly (default)
  • "list X / which X are Y / top N" (any tense) → table
  • "page me whenever..." → use manage-alerts instead

⚠️ Most common mistake: picking metric for a past-tense query. A prompt like "what was the frontend memory over the past 60 seconds?" is asking about data that already exists — it's a windowed lookup, not a request to wait 60 more seconds and watch. Always inspect the verb before choosing metric:

- was / were / averaged / hit / spiked / over the past / in the last / for the last → now or table. NEVER metric.
- watch / monitor / poll / wait until / wake me / for the next / until X happens / live-sample → metric.

If the user wants durable alerting ("page me whenever..."), use manage-alerts instead.

⚠️ Don't use observe for health rollups. "Show me the health of X", "status of the X
environment", "how is X doing" — even with a time qualifier like "over the past hour" — should
route to apm-health-summary, not observe. observe is for raw-metric / single-query investigations;
apm-health-summary returns the full rollup (degraded services, anomalies, pod resources). Pick
apm-health-summary whenever the user is asking a HEALTH question rather than a specific metric
question.

Prerequisites

How to call observe

Anomaly mode (default)

{
  "mode": "anomaly",
  "min_score": 75,
  "max_wait": 600,
  "namespace": "otel-demo"
}

• min_score: 75 default (major+), 50 for minor inclusion, 90 for critical-only.
• max_wait: generous (600s default). Returns immediately on trigger — long waits are free.
• namespace: only if the user scopes to a K8s namespace.

Metric mode — threshold condition

{
  "mode": "metric",
  "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.pod.name == \"frontend-7d4b8f9c5-x2k9m\" | STATS v = AVG(metrics.k8s.pod.memory.working_set)",
  "condition": "< 80000000",
  "description": "frontend pod memory working set",
  "max_wait": 300
}

Condition format: <comparator> <threshold> — valid comparators: <, <=, >, >=, ==.

Metric mode — live sample (no threshold)

Omit condition and the tool live-samples for the full max_wait window — use for "show me a live chart of X" prompts. The view renders an accumulating sparkline.

{
  "mode": "metric",
  "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.namespace.name == \"oteldemo-esyox-default\" | STATS v = AVG(metrics.k8s.pod.memory.working_set)",
  "description": "oteldemo-esyox-default avg pod memory",
  "max_wait": 60
}

Now mode — single read

{
  "mode": "now",
  "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.namespace.name == \"oteldemo-esyox-default\" | STATS v = AVG(metrics.k8s.pod.memory.working_set)",
  "description": "current avg pod memory in oteldemo-esyox-default"
}

Past-tense windowed read — when the user asks "what was X over the past N seconds/minutes/hours", put the window in the ES\|QL WHERE clause, not in max_wait. The tool returns immediately with the aggregate; nothing is polled.

{
  "mode": "now",
  "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.pod.name == \"frontend-7d4b8f9c5-x2k9m\" AND @timestamp > NOW() - 60 seconds | STATS v = AVG(metrics.k8s.pod.memory.working_set)",
  "description": "frontend memory, last 60 seconds (avg)"
}

If the user wants the time series (not just one number), use table mode with BUCKET() instead — that returns rows the view can chart.

Table mode — full ES|QL rows and columns

Use when the query groups, lists, or returns mixed-type rows (strings + numbers + dates). now mode discards everything except the first numeric cell — table mode preserves the whole result.

{
  "mode": "table",
  "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE metrics.k8s.pod.memory.working_set IS NOT NULL | STATS avg_mem = AVG(metrics.k8s.pod.memory.working_set) BY resource.attributes.k8s.pod.name, resource.attributes.k8s.namespace.name | SORT avg_mem DESC | LIMIT 10",
  "description": "top 10 pods by memory"
}

Past-tense time series — use table with BUCKET() to return one row per time slice the view can chart:

{
  "mode": "table",
  "esql": "FROM metrics-kubeletstatsreceiver.otel* | WHERE resource.attributes.k8s.pod.name == \"frontend-7d4b8f9c5-x2k9m\" AND @timestamp > NOW() - 60 seconds | STATS v = AVG(metrics.k8s.pod.memory.working_set) BY bucket = BUCKET(@timestamp, 5 second) | SORT bucket ASC",
  "description": "frontend memory · 60s · 5s buckets"
}

Rows are capped at 50 by default. Prefer tightening the ES|QL with LIMIT / SORT over raising row_cap — very wide tables clog the context window.

Picking the right index pattern

Fields live where the data is emitted — ES|QL rejects queries that reference a field the target index doesn't map (verification_exception). Before writing the query, match the user's question to the right layer:

Cross-layer questions ("which node runs the most services") need the index that carries both fields — that's almost always metrics-, because OTel resource attributes propagate through the Collector, so metrics docs carry k8s.node.name and service.name. Trace indices (traces-apm, traces-.otel-) don't carry infra attributes like k8s.node.name — don't reach for them when the question is about nodes.

Example: "which node is running the most services"

FROM metrics-*
| WHERE @timestamp > NOW() - 5m AND k8s.node.name IS NOT NULL AND service.name IS NOT NULL
| STATS service_count = COUNT_DISTINCT(service.name) BY k8s.node.name
| SORT service_count DESC
| LIMIT 20

Common query patterns

These are the field paths this deployment's data actually uses — prefer them over guessing.

⛔ BEFORE WRITING ANY QUERY — check the field type first:

Look up the field you intend to aggregate in the tables below. If its type is counterlong, stop — do not write FROM. Use TS + RATE() as described in the "Counter fields" section. This is a hard syntax requirement: FROM is rejected for counterlong fields regardless of which aggregation function you use, and the error cannot be fixed by changing AVG to MAX or SUM. The source command must be TS.

OTel Kubernetes (kubeletstats receiver)

Index: metrics-kubeletstatsreceiver.otel*

Each kubeletstats scrape emits separate documents per metric — a CPU doc, a memory doc, a network doc, etc. Always filter WHERE <field> IS NOT NULL for the field you're aggregating, otherwise most rows carry nulls for it.

Gauge fields — use AVG / MAX / MIN, never SUM:

For counter fields (network I/O, network errors, uptime), see the "Counter fields" section below — these require TS + RATE().

Fields that may NOT exist in every cluster's kubeletstats schema — depends on whether the OTel collector / kubelet is configured to scrape pod-spec data:

Schema-aware error recovery. When ESQL returns Unknown column [X], did you mean [Y, Z]?, read the suggestion and retry with one of Y or Z — don't blindly retry the same query or guess a different name from memory.

Dimension fields — use for filtering and BY grouping:

Both forms work on metrics-kubeletstatsreceiver.otel*. Prefer the unprefixed form — it's shorter and also works on counter-field queries via TS.

Common recipes:

Top pods by memory (last 5m, across all namespaces):

FROM metrics-kubeletstatsreceiver.otel*
| WHERE @timestamp > NOW() - 5 minutes AND k8s.pod.memory.working_set IS NOT NULL
| STATS avg_mem = AVG(k8s.pod.memory.working_set),
        max_mem = MAX(k8s.pod.memory.working_set)
  BY k8s.pod.name, k8s.namespace.name
| SORT max_mem DESC
| LIMIT 20

Which pods are on a specific node:

FROM metrics-kubeletstatsreceiver.otel*
| WHERE @timestamp > NOW() - 5 minutes
  AND k8s.node.name == "<node>" AND k8s.pod.name IS NOT NULL
| STATS last_seen = MAX(@timestamp) BY k8s.pod.name, k8s.namespace.name
| SORT last_seen DESC

Namespace-wide memory average (single scalar — works in now/metric mode):

FROM metrics-kubeletstatsreceiver.otel*
| WHERE k8s.namespace.name == "oteldemo-esyox-default"
  AND k8s.pod.memory.working_set IS NOT NULL
| STATS v = AVG(k8s.pod.memory.working_set)

Is this node under memory pressure (working-set vs available):

FROM metrics-kubeletstatsreceiver.otel*
| WHERE @timestamp > NOW() - 5 minutes
  AND k8s.node.name == "<node>" AND k8s.node.memory.working_set IS NOT NULL
| STATS working_set = AVG(k8s.node.memory.working_set),
        available = AVG(k8s.node.memory.available)

Pods above namespace memory average (requires ES 9.2+ INLINE STATS):

FROM metrics-kubeletstatsreceiver.otel*
| WHERE @timestamp > NOW() - 5 minutes
  AND k8s.pod.memory.working_set IS NOT NULL
  AND k8s.namespace.name == "<namespace>"
| STATS pod_mem = AVG(k8s.pod.memory.working_set) BY k8s.pod.name
| INLINE STATS ns_avg = AVG(pod_mem)
| WHERE pod_mem > ns_avg
| EVAL pct_above_avg = ROUND((pod_mem - ns_avg) * 100.0 / ns_avg, 1)
| SORT pod_mem DESC

Use INLINE STATS whenever the question involves comparing individual entities to a group aggregate ("above average", "top X% of namespace memory", "pods consuming more than their share"). Without it, this requires two separate queries and client-side joining.

Counter fields — require TS + RATE()

⛔ MANDATORY PRE-QUERY CHECKLIST — run this before writing any ES|QL:

1. Is any field I'm querying in the counter fields table below? (network I/O, network errors, uptime)
- YES → Stop. Use TS, not FROM. Do not write FROM under any circumstances. Go to the TS pattern below.
- NO → FROM is fine. Proceed normally.
2. Am I inside a TS query?
- Use TBUCKET(duration) for time bucketing — e.g. TBUCKET(1 hour).
- Never use BUCKET(@timestamp, interval) — that is FROM-query syntax and will fail with a parse error inside TS.
3. Am I aggregating across multiple pods/nodes?
- YES → wrap RATE() in SUM().
- NO (single entity) → wrap in AVG().

Skipping this checklist and defaulting to FROM on a counter field is the single most common query mistake. The error it produces (argument of [...] must be [...numeric except counter types]) is not recoverable by changing the aggregation function — the source command must be TS.

Network I/O, network errors, and uptime fields are stored as monotonically-increasing counters (counter_long), not instantaneous gauges. FROM + MAX/AVG/SUM/VALUES on a counter field is a hard error — ES|QL returns argument of [...] must be [...numeric except counter types].

Version requirement: TS, RATE(), and TBUCKET() require Elasticsearch 9.2+. On older
clusters, counter_long fields cannot be aggregated at all — there is no workaround. If TS <index>
returns a parse error, verify the cluster version before iterating on syntax.

Counter fields in this deployment:

Correct pattern: TS as the source command (replaces FROM), RATE() wrapped in an outer aggregation, counter field filtered IS NOT NULL, and BY direction whenever querying network fields.

// CORRECT — TS as source, SUM(RATE()) for cluster-wide total, BUCKET(@timestamp, interval) for time bucketing
// Always alias the BUCKET expression and SORT by the alias — SORT on the raw BUCKET(...) expression fails
TS metrics-kubeletstatsreceiver.otel*
| WHERE @timestamp > NOW() - 15 minutes
  AND k8s.node.network.io IS NOT NULL
| STATS total_bps = SUM(RATE(k8s.node.network.io))
  BY bucket = BUCKET(@timestamp, 1 minute), direction
| SORT bucket ASC

// WRONG — FROM is rejected for counter_long fields
FROM metrics-kubeletstatsreceiver.otel*
| STATS max_io = MAX(k8s.node.network.io) ...   ← hard error

// WRONG — sorting on the raw BUCKET expression (not the alias) fails in TS context
TS metrics-kubeletstatsreceiver.otel*
| STATS total_bps = SUM(RATE(k8s.node.network.io))
  BY bucket = BUCKET(@timestamp, 1 minute), direction
| SORT BUCKET(@timestamp, 1 minute) ASC   ← parse error; use alias instead

SUM vs AVG when wrapping RATE(): Use SUM(RATE(...)) when aggregating across multiple parallel time series (e.g. total cluster throughput across all nodes). Use AVG(RATE(...)) when you want the mean rate per entity (e.g. average per-pod transmit rate). Mixing them up produces silently wrong numbers — AVG of a cluster-wide roll-up understates throughput; SUM of a single-entity query overstates it.

Time bucketing in TS queries: Use BUCKET(@timestamp, interval) — the same syntax as FROM queries. Always assign it an alias (e.g. BY bucket = BUCKET(@timestamp, 1 hour)) and reference that alias in any subsequent SORT — sorting on the raw BUCKET(@timestamp, ...) expression is rejected in TS context. TBUCKET (duration-only form) does not work reliably on this deployment and should not be used.

// CORRECT — aliased BUCKET, sorted by alias
TS metrics-kubeletstatsreceiver.otel*
| STATS total_bps = SUM(RATE(k8s.node.network.io))
  BY bucket = BUCKET(@timestamp, 1 hour), direction
| SORT bucket ASC

// WRONG — TBUCKET is unreliable; fails with "@timestamp not found" on this deployment
TS metrics-kubeletstatsreceiver.otel*
| STATS total_bps = SUM(RATE(k8s.node.network.io))
  BY TBUCKET(1 hour), direction   ← verification_exception on @timestamp

// WRONG — passing @timestamp to TBUCKET is also rejected
| STATS ... BY TBUCKET(@timestamp, 1 hour)   ← argument type error

Rules:

• TS <index>, not FROM <index>. FROM will be rejected for counter_long fields.
• Wrap RATE() in SUM() for totals across entities, AVG() for per-entity averages — bare RATE(...) BY ... is rejected.
• Network counters are emitted as separate docs per direction. Without BY direction or a direction == "..." filter, transmit and receive aggregate into a meaningless combined number.
• Without IS NOT NULL the query spans many kubeletstats docs that carry a different metric — you get nulls, not errors.
• Use BUCKET(@timestamp, interval) for time bucketing inside TS queries, always aliased. Do not use TBUCKET.

Escape hatch — raw counter snapshot: if you want the current counter value (e.g. "how long has node X been up"), cast first. TO_LONG strips the counter type and unlocks standard aggregations:

FROM metrics-kubeletstatsreceiver.otel*
| WHERE @timestamp > NOW() - 5 minutes AND k8s.node.uptime IS NOT NULL
| EVAL u = TO_LONG(k8s.node.uptime)
| STATS uptime_s = MAX(u) BY k8s.node.name

APM traces

Primary index: traces-.otel- (OTel-native). Fallback: traces-apm* (classic APM — only if the OTel path returns empty).

In EDOT-ingested clusters, traces-.otel- carries both OTel-native fields (duration, kind, status.code) and classic-APM-compatible fields (processor.event, event.outcome, transaction.duration.us on transaction-level docs). The cluster's "APM-ness" isn't determined by the index — it's determined by which field shape you query.

Unit warning. OTel duration is in nanoseconds. Divide by 1,000,000 for milliseconds. Classic transaction.duration.us is in microseconds — divide by 1,000. Mixing these across a comparison produces wildly wrong numbers.

Error-field warning. On traces-.otel- the exception attributes use the exception. namespace, not error.. Querying error.message / error.type against an OTel-native index returns verification_exception: Unknown column [error.message], did you mean any of [exception.message, message]?. The error. family belongs only to classic-APM traces-apm documents. When you see the user ask "show me the error messages from X", reach for exception.message first.

Where exception data actually lives — three deployment shapes. OTel records exceptions as Span Events; where they end up indexed depends on the exporter pipeline:

1. Flattened onto the trace doc — the parent span carries exception.message, exception.type, exception.stacktrace. Common with EDOT + APM Server. Query traces-.otel-.
2. Exported as separate log records — logs-.otel- carries exception.message / exception.type, correlated to the trace via trace.id and span.id. This is the default OpenTelemetry Collector pipeline. The trace doc itself only carries status.message / status.code (Error on failed spans).
3. Classic APM error events — traces-apm* with processor.event == "error" carries error.message and error.exception.type. Only present when classic-APM agents are in use.

The trace doc always carries event.outcome == "failure" and status.message regardless. If your exception.message query against traces-.otel- returns Unknown column [exception.message], the deployment is shape 2 — switch to logs-.otel- with WHERE exception.message IS NOT NULL and join back to the trace via trace.id if needed. Don't assume the original index has the field just because the user said "from traces".

Service p95 latency (OTel-native), last 15m — result in ms:

FROM traces-*.otel-*
| WHERE service.name == "checkout" AND @timestamp > NOW() - 15 minutes
  AND kind == "Server"
| STATS p95_ms = PERCENTILE(duration, 95) / 1000000

Error rate for a service — event.outcome is reliable here:

FROM traces-*.otel-*
| WHERE service.name == "checkout" AND @timestamp > NOW() - 15 minutes
  AND kind == "Server"
| STATS errors = COUNT(*) WHERE event.outcome == "failure", total = COUNT(*)
| EVAL error_rate_pct = ROUND(errors * 100.0 / total, 2)
| KEEP error_rate_pct, errors, total

Recent exception messages — try the trace index first (flattened-event pipelines), then fall back to logs (default OTel Collector pipeline):

1) Trace doc carries the exception (EDOT-flattened pipelines):

FROM traces-*.otel-*
| WHERE service.name == "checkout" AND @timestamp > NOW() - 15 minutes
  AND event.outcome == "failure"
  AND exception.message IS NOT NULL
| KEEP @timestamp, exception.type, exception.message
| SORT @timestamp DESC
| LIMIT 50

2) Exception lives in logs (default OTel pipeline) — use this when (1) returns Unknown column [exception.message] or empty:

FROM logs-*.otel-*
| WHERE service.name == "checkout" AND @timestamp > NOW() - 15 minutes
  AND exception.message IS NOT NULL
| KEEP @timestamp, exception.type, exception.message, trace.id, span.id
| SORT @timestamp DESC
| LIMIT 50

3) Classic-APM equivalent — only when both OTel paths return empty:

FROM traces-apm*
| WHERE service.name == "checkout" AND @timestamp > NOW() - 15 minutes
  AND processor.event == "error"
| KEEP @timestamp, error.exception.type, error.message
| SORT @timestamp DESC
| LIMIT 50

If traces-.otel- returns empty, the deployment is classic-APM-only — fall back to traces-apm* with processor.event == "transaction" and transaction.duration.us.

Throughput trend — use the pre-aggregated rollup when possible. metrics-servicesummary.1m.otel-* carries per-minute request counts in servicesummary (a regular long, designed to SUM). Cheaper and faster than scanning raw traces for "how many requests/min over the last hour":

FROM metrics-service_summary.1m.otel-*
| WHERE service.name == "frontend" AND @timestamp > NOW() - 1 hour
| STATS throughput = SUM(service_summary)
  BY bucket = BUCKET(@timestamp, 1 minute)
| SORT bucket ASC

Log rate

Index: logs-*

FROM logs-*
| WHERE service.name == "cartservice" AND @timestamp > NOW() - 5m
| STATS v = COUNT(*)

Query-construction rules

• For now and metric mode, the query must return a single row with a numeric first column —

the tool reads the first numeric cell. For table mode this restriction doesn't apply: any shape is fine.

• Scope with @timestamp > NOW() - <window> when the user implies "right now" (default 5m is

usually fine; let the window match the user's language).

• When the user names a namespace, match it exactly (e.g. oteldemo-esyox-default, not

otel-demo). If unsure, call apm-health-summary first — its namespace_candidates field surfaces fuzzy matches.

• String literals use double quotes only. ES|QL does not accept single quotes — WHERE kind == "Server", never WHERE kind == 'Server'. Single quotes cause token recognition error failures.
• Match the aggregation to the field's storage shape. Three shapes to recognize:

- Gauges (memory.workingset, memory.available, cpu.usage, filesystem.usage in metrics-kubeletstatsreceiver.otel): use AVG / MAX / MIN. Do not SUM a gauge — it will add every ~15s kubelet sample over your window and inflate the value by hundreds or thousands. - Counters (k8s.pod.network.io, k8s.node.uptime, etc. — counterlong type): require TS + RATE(). See the "Counter fields" section above. FROM + MAX/AVG/SUM on a counter is a hard error, not a silent wrong number. - Pre-aggregated rollups (servicesummary on metrics-servicesummary.1m.otel-, span.destination.service.responsetime.count on metrics-servicedestination.1m.otel-*): designed for SUM across the window. Each doc is already a per-minute bucket count.

• Use CHANGE_POINT to detect regime shifts automatically on time-bucketed results rather than

eyeballing the table. Append it after any STATS ... BY BUCKET(@timestamp, interval) query when the user asks about spikes, dips, or anomalies in a metric over time:

`` FROM metrics-kubeletstatsreceiver.otel* | WHERE @timestamp > NOW() - 24 hours AND k8s.node.cpu.usage IS NOT NULL | STATS avgcpu = AVG(k8s.node.cpu.usage) BY bucket = BUCKET(@timestamp, 1 hour) | SORT bucket ASC | CHANGEPOINT avg_cpu ON bucket | WHERE type IS NOT NULL ``

CHANGEPOINT appends type (e.g. stepchange, spike, dip) and pvalue columns. Filter WHERE type IS NOT NULL to surface only the detected change points. Works on any numeric time-bucketed series — CPU, memory, error rates, throughput.

After the tool returns

setupnotice is view-side chrome — don't echo or summarize it in chat. There are three variants:

• welcome and skill-gap are user-facing nudges to install the skill packs. Ignore them.
• schema-hint is an actionable retry instruction the server emits when your query failed in a

way the latest skill expects on this deployment shape (e.g. exceptions live in logs-.otel- instead of on the trace doc). Read its message and adapt your next call accordingly — don't surface the banner text in chat, but do follow its guidance.

The observe MCP App view renders inline in one of several modes, picked automatically from the result:

• Now mode (status: NOW) — compact card: big unit-formatted number, ES|QL subtitle, "evaluated

Xs ago" stamp, and three follow-up actions (re-check, escalate to live observation, create alert rule).

• Metric mode — area + line + dots sparkline with optional threshold line; stat cards for

current / threshold / peak / baseline. Covers CONDITION_MET, TIMEOUT, and SAMPLED.

• Anomaly mode — severity-scored trigger card with affected entities and click-to-send

investigation prompts.

• Table mode (status: TABLE) — styled HTML table with column headers, type-aware alignment

(numeric right, text left), and zebra-striped rows. Row count + truncation notice in the subtitle.

• Error (status: ERROR) — red-toned card with the ES|QL failure message verbatim. Surfaces

instead of throwing when the query is bad (unknown field, index missing, syntax error).

All modes surface an investigation_actions list as buttons. Follow up in chat too — don't rely on the buttons alone.

Status-by-status guidance

• NOW — State the value plainly. Offer to escalate to a live observation if the user seems to

want ongoing visibility.

• TABLE — Summarize what the rows show (top entity, total count, any outliers). Don't just

dump the full table back — the user can read the widget. If the result was truncated, say so and offer to tighten the ES|QL.

• ERROR — Read the error message, explain what likely went wrong (unknown field, index

pattern, syntax), and propose a corrected query. Don't retry blindly.

• ALERT (anomaly fired) — The response includes affected entities, affected services, top

anomalies, and investigation_hints naming the next tool to reach for. Follow those hints immediately — don't just report the alert, start investigating and narrate your reasoning.

• CONDITION_MET (metric threshold satisfied) — Confirm to the user and describe the trend

from the returned history. If this was post-remediation validation, explicitly state the fix has been validated. Offer to graduate the condition into a durable rule via manage-alerts.

• SAMPLED (live sample completed without a condition) — Summarize the trend (trending up /

down / flat, peak, typical). Offer "keep observing" (extend window) or graduate to an alert rule.

• TIMEOUT (metric condition never met) — Tell the user the metric didn't stabilize. Suggest

follow-ups: check ml-anomalies, persist as alert rule, re-examine the ES|QL.

• QUIET (anomaly mode, nothing fired) — Suggest adjustments: lower min_score, widen

lookback, verify ML jobs are running.

Accumulating timelines

Every metric-mode response includes an observe_key derived from esql + condition. When Claude re-invokes observe with the same ES|QL (e.g. via the "Extend observation (+60s)" button), the view merges the new samples into the existing sparkline instead of resetting — so the user sees a continuous timeline across multiple tool calls. To keep this continuity, reuse the exact same ES|QL string and condition when extending. Capped at 240 points to keep the chart readable.

Tools

Key principles

• Observe is transient. Nothing is saved. If the user wants an ongoing rule, use manage-alerts.
• Pick the mode from the user's phrasing. "What is X right now" (scalar) → now. "Show me a

live chart of X" or "watch X for 60s" → metric (no condition). "Wait until X drops below Y" → metric (with condition). "Tell me when anything unusual fires" → anomaly. "List …", "which pods are on node X", "top N by Y" → table. If a user asks "what is X" and X is actually a list or grouping (not a single number), pick table, not now.

• Use the known field paths. Don't probe generic metrics-* patterns when the deployment

indexes under metrics-kubeletstatsreceiver.otel*. The cheat sheet above is authoritative for this environment.

• On ALERT, start investigating immediately. The investigation_hints are suggestions —

follow them and narrate your reasoning.

• Don't start with observe for vague triage. If the user reports a symptom without naming a

specific metric ("something feels slow", "what's wrong with prod"), reach for apm-health-summary first — it surfaces the worst-offender services without needing a query. observe needs a target metric to poll; use it to drill in after the rollup names something.

• Don't over-tune min_score. 75 catches the important stuff; dropping below 50 produces noise.

Investigation discipline

• One tool call per turn. After this tool returns, narrate the headline finding before the next call. Each call adds a widget — chaining several after one "yes" looks like a runaway agent.
• Sequential offers, not OR. "Want me to check X or Y?" → "Want me to start with X? If it's inconclusive I can follow up with Y." The user's "yes" then maps to one call, not both.
• Build queries from known fields. Use the cheat-sheet above — don't guess field names. If an "Unknown column" error returns with a "did you mean…" suggestion, read it and adjust; don't blindly retry.