Monitoring the capture service with Grafana MCP

The capture service (rust/capture/) is PostHog's Rust HTTP ingestion endpoint. It receives events from SDKs, applies quota/rate limits, and produces to Kafka. Five deployments run the same binary with different CAPTURE_MODE configs, each in its own K8s namespace.

This skill teaches how to discover live metrics using the Grafana MCP tools rather than memorizing metric names that change as the code evolves.

Environment context

The Grafana MCP is connected to a single Grafana instance scoped to one environment. If the user hasn't specified, ask which environment they want to investigate:

• prod-us — US production (us-east-1)
• prod-eu — EU production (eu-central-1)

Most capture app metrics (e.g. capture, httprequests, envoycluster_*) are environment-specific by virtue of which Grafana you're connected to — they don't carry an environment label. MSK and CloudWatch metrics do carry environment labels but are still scoped to the connected Grafana's AWS account.

Cross-environment comparison requires switching Grafana instances (not possible in one session).

Observability landscape

Capture spans seven telemetry domains. Each has a Grafana datasource and a discovery entry point.

Stable waypoints

These facts change infrequently and are hard to discover dynamically.

Deployment roles

Each capture variant runs as a separate K8s deployment in its own namespace. The primary scope labels are namespace and container (not role — that label contains pod names and is not useful for filtering).

All variants share the same Rust binary (ghcr.io/posthog/posthog/capture).

Scope capture metrics with namespace=~"capture-." or container=~"capture-.". For a single pipeline, scope by namespace (e.g., namespace="capture-analytics").

Envoy cluster naming

Envoy metrics use envoyclustername to identify the upstream backend. Pattern: posthog{deployment}{port}.

Capture-related clusters: posthogcapture-analytics3000, posthogcapture-ai3000, posthogcapture-replay3000, posthogcapture-mirrored3000, posthogcapture-logs4318, posthogcapture-logs-canary4318.

capture-replay also has a proxy-as-a-service cluster used in KEDA autoscaling: proxy-as-a-servicecapture-replay3000.

Scope with: envoyclustername=~"posthog_capture-.*".

Redis instance topology

Capture depends on up to three logical Redis instances, plus one external instance at the Envoy layer (not in the capture binary). None emit captureredis* metrics — Redis health is inferred from capture-side metrics and CloudWatch ElastiCache metrics.

1. Primary Redis (REDIS_URL env var)

• ElastiCache: posthog-solo (prod-us) or posthog-prod-redis-encripted (prod-eu; sic — typo in actual cluster name)
• Both envs use a read-only endpoint for the token cache (REDISREADERURL)
• Backs: billing/quota limits (CaptureQuotaLimiter), session replay overflow limiter
• Capture metrics: capturebillinglimitsloadedtokens (by cache_key),

capturequotalimit_exceeded (by resource)

• Quota resources: events, exceptions, llmevents, recordings, surveyresponses
• Cache keys: @posthog/quota-limits/{resource}, @posthog/capture-overflow/replay

2. Global Rate Limiter Redis (GLOBALRATELIMITREDISURL, optional)

• ElastiCache: capture-globalratelimit-{env}-redis (prod-us, prod-eu; not dev)
• Backs: per-(token, distinct_id) sliding-window rate limiter
• Falls back to primary Redis when URL is unset
• Optional read replica: GLOBALRATELIMITREDISREADER_URL
• Toggle: GLOBALRATELIMIT_ENABLED (may be off in some envs during rollout)
• Metrics: globalratelimiter* (direct), captureeventsreroutedoverflow{reason="rate_limited"} (proxy signal)
• globalratelimiterwindowseconds{scope} publishes each process's window. capture-analytics and capture-ai share the AI byte budget, so if they disagree on this gauge the shared budget has split
• CloudWatch cluster id: capture-globalratelimit-prod-redis

3. Event Restrictions Redis (EVENTRESTRICTIONSREDIS_URL)

• ElastiCache: ingestion-prod-redis (separate writable cluster in both envs)
• Stores Django-synced ingestion restriction configs
• Falls back to primary Redis when URL is unset
• Capture metrics: captureeventrestrictionsredisfetch (labels: restriction_type,

result in success/notfound/error/parseerror), captureeventrestrictionsstale, captureeventrestrictionsloaded_count

4. Contour Rate Limit Redis (ratelimit-{env}-redis) — NOT in capture binary

• Per-IP DoS protection at the Envoy ingress layer, in front of capture
• Metrics: ratelimitservice* (label: domain="posthog")

Metric prefixes

Every prefix here can be discovered live with listprometheusmetric_names using datasourceUid: "victoriametrics" and regex: "<prefix>.*".

Kafka topics

Topics capture produces to (discover live via topic label on capturekafkaproduceavgbatchsizebytes). Partition counts are encoded in topic names and differ by env (EU generally has fewer partitions).

Capture writes to two different Kafka backing systems depending on the pipeline:

• MSK ingestion cluster — analytics events (main, overflow, historical, turbo), heatmaps, error tracking, client warnings
• WarpStream — session replay (warpstream-replay-v2 VC), logs (warpstream-logs VC), traces (warpstream-traces VC)

Pyroscope services

Profile types: processcpu:cpu:nanoseconds:cpu:nanoseconds, wall:wall:nanoseconds:wall:nanoseconds, memory:inusespace:bytes:inusespace:bytes, memory:inuseobjects:count:inuse_space:bytes.

Grafana dashboards

The two details boards live under Capture/Ingestion/ and cross-link each other plus the overview. Sections for machinery shared by both pipelines (Global Rate Limiter, Event restrictions, Hyper server, Contour, billing/quota) appear on both details boards, so use whichever one you are already on.

One board you will find via search_dashboards but should not rely on: capture-3000-envoy-codes is a personal, US-only Envoy investigation board (not synced to EU).

Discovery workflows

Prometheus / VictoriaMetrics

1. listprometheusmetricnames — datasourceUid: "victoriametrics", regex: "capture.*" to enumerate app metrics
2. Pick a metric, then listprometheuslabel_names scoped to it — see available dimensions
3. listprometheuslabel_values — discover actual values for a label

(e.g. labelName: "cause" on captureeventsdropped_total)

1. query_prometheus with PromQL — always scope by namespace (or container) and set a time range

Loki (logs)

1. listlokilabel_names — datasourceUid: "P44D702D3E93867EC" (Loki-logs; do NOT use primary Loki P8E80F9AEF21F6940 which 502s intermittently)
2. listlokilabel_values for app or namespace — find capture containers
3. querylokilogs — e.g. {app=~"capture.*"} |= "error"

Pyroscope (profiling)

1. listpyroscopeprofiletypes — datasource_uid: "pyroscope"
2. fetchpyroscopeprofile — matchers: '{service_name="capture-analytics/capture-analytics"}',

profiletype: "processcpu:cpu:nanoseconds:cpu:nanoseconds"

Dashboards

1. search_dashboards — query "capture" or "ingestion"
2. getdashboardby_uid — use a known UID (e.g. "capture") to get panel details
3. getdashboardpanel_queries — extract PromQL from existing panels

Redis / ElastiCache

• Capture-side: discover capturebilling, captureeventrestrictions_,

capturequota* metrics in VictoriaMetrics

• Infrastructure: CloudWatch datasource P034F075C744B399F for ElastiCache

(CPU, memory, connections, latency). Cluster IDs: capture-globalratelimit-prod-redis, posthog-solo (prod-us primary)

Key metric domains

Categories of what to look for. Discover specific metrics live using the prefixes above.

HTTP layer — request rate, latency distribution (p50/p99), active connections, error rates by status code. Metrics: httprequests* scoped by namespace, captureactiveconnections.

Event lifecycle — the funnel from received to ingested to dropped/rerouted. captureeventsreceivedtotal -> captureeventsingestedtotal -> captureeventsdroppedtotal. The cause label on drops has 20+ values (discover live). Key additions since the golden-chart migration: eventrestrictiondrop, eventtoobig, otelquotadrop, oversizeevent, aioptin, gathering, invalidsession, nosessionid, nosnapshot. Rerouting: captureeventsreroutedoverflow with reason label (ratelimited, forcelimited, eventrestriction). Also: captureeventsreroutedcustomtopic for topic-redirect restrictions.

Kafka producer — broker connectivity (capturekafkaanybrokersdown, capturekafkabrokerconnected), queue saturation (queuedepth / queuedepthlimit), produce RTT (capturekafkaproducerttlatencyus by quantile and broker), delivery errors (capturekafkaproduceerrors_total).

Billing and quota — capturebillinglimitsloadedtokens by cachekey, capturequotalimitexceeded by resource (events, exceptions, llmevents, recordings, surveyresponses).

Event restrictions — captureeventrestrictions* for Redis fetch health, staleness, loaded count, applied restrictions by restrictiontype (dropevent, forceoverflow, redirecttotopic, skippersonprocessing).

Envoy proxy — upstream latency, response codes (2xx/4xx/5xx), connection health, circuit breakers (open gauges), backend membership (healthy vs total), timeouts, retries. Always filter: envoyclustername=~"posthogcapture-.*". For capture-replay, also check proxy-as-a-servicecapture-replay3000.

Contour rate limit — ratelimitservice* for per-IP DoS protection. ratelimitserviceratelimitover_limit = actively rate-limited IPs.

MSK broker-side — awsmsk* JMX metrics for capture-analytics, capture-ai, and other MSK-backed topics. Key signals: throttle time, network processor idle %, memory pool depletion, request queue size. Both envs have a dedicated ingestion MSK cluster separate from the events cluster (prod-us: c21; prod-eu: posthog-prod-eu-ingestion-2026-05-04).

WarpStream — warpstreamagent* metrics for capture-replay, capture-logs, and traces. These pipelines produce to in-cluster WarpStream agents, not MSK. Key signals: warpstreamagentcontrolplaneoperationcounter (by operation), warpstreamagentfilecacheclientfetchlocalorremotecounter (cache hit/miss ratio). Dashboards: warpstream (Agent Overview), dbfj5c31spa1ogf (MSK vs WarpStream — Active Produce Topics). US-only personal dashboards (not synced to EU): ws-coarse-lag-explore (Coarse Lag), 8e93b023-… (CH Consumer Lag). Per-VC KMinion instances: kminion-warpstream-replay, kminion-warpstream-logs, kminion-warpstream-traces.

K8s resources — container and kube for CPU, memory, restarts, HPA state. Scope: namespace=~"capture-.", pod=~"capture-.".

Investigation playbooks

See [references/investigation-playbooks.md](./references/investigation-playbooks.md) for step-by-step workflows for common questions: health checks, event loss, latency, Kafka backpressure, rate limiting, Redis, and cross-env comparison.