Opinionated OTel Collector — Kubernetes DaemonSet

Use this skill for the agent tier: one collector per node ingesting node-local OTLP, kubelet/host metrics, same-node Prometheus targets, and pod logs. Use a separate gateway or cluster Deployment for tailsampling, loadbalancing, k8scluster, and k8sevents.

This repository owns OllyGarden's decisions, not component facts. Consult otel-collector for current component keys, defaults, and stability; otel-ottl for syntax; and the relevant public OpenTelemetry language or instrumentation skill for source-side fixes.

Non-negotiable pipeline contract

Apply these to traces, metrics, and logs:

1. Put memory_limiter first so backpressure happens before downstream buffering.
2. Enrich with real identity; do not fabricate it. Scope k8s_attributes to the local node with

filter.nodefromenvvar: K8SNODE_NAME, disable the system detector's host.name, and verify detector order/override against the pinned Collector distribution.

1. Put resource-value truncation last among transforms. Kubernetes metadata can otherwise inflate

every record.

1. Persist filelog offsets with filestorage on host-backed storage. Container-local storage

loses offsets when the pod is recreated.

The OTLP listener binding alone does not make ingest node-local. The DaemonSet deployment must route each workload to the agent on its own node. Confirm the networking topology before claiming the agent/gateway boundary holds.

Metrics: reduce series and cadence

Metric cost is series count × datapoints per minute. The references implement these decisions:

• Curate kubeletstats and host_metrics; prefer bounded utilization measurements over redundant

per-state series.

• Split fast and slow groups into disjoint receiver instances. Keep container/pod/node metrics at

20s, volume metrics at 60s, CPU/memory at 60s, and filesystem at 300s. Do not slow a single receiver globally and lose useful CPU/memory resolution.

• Drop read-only secret, configMap, downwardAPI, and projected service-account-token volume

metrics. Retain emptyDir: its writable usage is a disk-pressure signal.

• Scrape each pod only from the agent on its node using

field: spec.nodeName=${env:K8SNODENAME}; use a separate slow scrape for expensive endpoints and discard terminal pods.

• Filter to monitored namespaces.

The preferred DPM pattern is separate receivers over disjoint subsets. For SDK-set OTLP cadence or receivers without a partition knob, consult the public otel-collector routing connector and interval processor references. Preserve unmatched passthrough and account for empty emissions.

Logs: cap and scope

• Cap individual pod-log records (maxlogsize: 100KiB) and exclude the collector's own logs.
• Drop low-value severities only after structured records have a usable severity_number.
• Deduplicate only known-chatty services. Immediately before scoped log_dedup, remove

log.file.path and log.file.recordnumber under the same condition or those changing offsets defeat the hash. The shipped YAML deliberately does not enable dedup without a service-specific scope; consult the otel-collector logdedup reference before adding it.

• Fix telemetry values embedded in message text at the application rather than relying on brittle

Collector regexes.

Traces: deterministic noise only

• Drop probe spans using the bounded route/path/name patterns in references/traces.yaml; the

filters cover current and legacy HTTP attributes plus framework handler names. Keep regexes anchored.

• Prefer source-side suppression for static assets following the relevant public OpenTelemetry

instrumentation guidance; use the Collector filter as a portable fallback.

• Do not probabilistically head-sample at the agent for cost. Keep the agent lossless except for

reviewed deterministic noise filters; whole-trace reduction requires gateway tailsampling behind loadbalancing.

Self-monitoring

Use detailed internal telemetry at a modest reporting interval, with views dropping the noisiest high-cardinality internal series. Retain queue, refusal, and export-failure visibility so savings do not hide an unhealthy collector.

Reference configuration

Copy the full set and search for CUSTOMIZE:

• references/common.yaml — shared receiver, processors, exporter, state, and self-telemetry.
• references/traces.yaml, metrics.yaml, logs.yaml — one complete signal pipeline each.
• references/prometheus/*.yaml — bare scrape-job fragments included by metrics.yaml.

Read references/decomposing-config.md before editing. Processor arrays replace rather than merge, and ${file:} paths depend on the Collector working directory.

Verify before shipping

Complete every gate below; a parser-only or single-fragment check is not verification:

1. Obtain common.yaml, traces.yaml, metrics.yaml, and logs.yaml; if one is missing, stop and

request it. From references/, validate all four together against the pinned distribution.

1. Supply non-secret synthetic K8SNODENAME, K8SCLUSTERNAME, and exporter endpoint values,

then inspect print-config output for all pipelines, processor order, and included scrape jobs.

1. Use sanitized positive and near-miss telemetry to prove each filter drops only its intended

target. Never use production ingest/export endpoints for verification.

Follow references/validating.md for the merged commands, off-cluster failure classification, and version limits.

Handoffs

• Component configuration and OTTL: otel-collector, otel-ottl.
• Generic deep-merge mechanics: ollygarden-otel-collector-config-decomposition.
• Source-side telemetry design and suppression: the relevant public OpenTelemetry language and

instrumentation skills.