Pipelines as Code

This skill is the entry point for the Materialize pipelines-as-code project. Stable conventions live in the repo docsite under [docs/content/reference/internal/pipelines/](../../../docs/content/reference/internal/pipelines/) — this file is intentionally slim and links into the docsite at heading-level granularity. The non-link content below is the state snapshot: what currently exists, what's in flight, and what's queued.

Audience reminder

The pipelines themselves target alloy (the binary), so authoring decisions favor contributors rather than end users. There is no panel-description-voice equivalent — readability of the YAML and the rendered .alloy is the goal.

The docsite reference pages target repo contributors (SRE, Field Engineering, CloudOps, Database Engineers) and AI agents reading this skill.

Where to find what

Frequently needed deep links into Authoring:

• [The strict-attributes / raw-escape policy](../../../docs/content/reference/internal/pipelines/authoring.md#the-strict-attributes--raw-escape-policy) — what to do when an undocumented attribute is rejected
• [Schema layout](../../../docs/content/reference/internal/pipelines/authoring.md#schema-layout) — file/$id map and how cross-file $refs resolve
• [Adding an attribute](../../../docs/content/reference/internal/pipelines/authoring.md#adding-an-attribute-to-an-existing-component) — minimal extension
• [Adding a typed sub-block](../../../docs/content/reference/internal/pipelines/authoring.md#adding-a-typed-sub-block) — the selectors-style pattern
• [Adding a typed component](../../../docs/content/reference/internal/pipelines/authoring.md#adding-a-typed-component) — Rust struct + schema + ComponentBlock variant
• [Reference-valued attributes](../../../docs/content/reference/internal/pipelines/authoring.md#reference-valued-attributes) — why forwardto / targets need Expression::refname, not String
• [Load-bearing invariants](../../../docs/content/reference/internal/pipelines/authoring.md#load-bearing-invariants) — AttributeValue order, alignment quirk, schemas-as-docs
• [Sub-block recursion and the raw escape](../../../docs/content/reference/internal/pipelines/authoring.md#sub-block-recursion-and-the-raw-escape)
• [How validation interacts with rendering](../../../docs/content/reference/internal/pipelines/authoring.md#how-validation-interacts-with-rendering) — YAML → serde_json::Value → schema validation → typed Pipeline → config.alloy

Schema and code map

packages/alloy-pipelines/                       ← YAML inputs
  ├── gateway.yaml                              – gateway processing pipeline
  ├── gateway-dest-stub.yaml                    – gateway default egress tail (swappable)
  └── agent.yaml

packages/mzmon-lib/schemas/alloy/               ← validation schemas (embedded into the binary)
  ├── mzmon-alloy.schema.yaml                   – entry
  ├── top.schema.yaml                           – description / logging / livedebugging / blocks
  ├── loki.schema.yaml                          – loki.*, stage.* $defs, shared `rule` $def
  ├── discovery.schema.yaml                     – discovery.*, cross-ref'ing `rule` from loki
  ├── prometheus.schema.yaml                    – prometheus.*, cross-ref'ing `rule` from loki + `target` from discovery
  ├── otelcol.schema.yaml                       – otelcol.processor.{batch,memory_limiter,attributes,groupbyattrs,filter,transform} + output/action/conditions/statements sub-blocks
  └── common/                                   – shared fragments. $id tail MUST match the file path
      ├── raw.schema.yaml                       – {raw: <block>} escape hatch + block primitives
      ├── attribute.schema.yaml                 – attributeValue (anyOf: literal | expression | …)
      └── expression.schema.yaml                – sys.env / function / operator / ref expression
  (each fragment's `$id` + the `ID_*` const in validate.rs + the relative `$ref`s must agree)

packages/mzmon-lib/src/alloy/                   ← AST, render, validate, pipeline (Rust)
  ├── ast.rs                                    – Block / AttributeValue / Expression / Expressable<T>
  ├── render.rs                                 – write_to + alloy-fmt-canonical formatting
  ├── validate.rs                               – embedded schemas + jsonschema validator + hints
  ├── pipeline.rs                               – Pipeline::from_yaml_str (YAML → Value → validate → typed)
  ├── test_support.rs                           – assert_renders (oracle: pipes through `alloy fmt`)
  └── components/                               ← typed sugar; tests colocated with impl
      ├── top.rs                                – LoggingBlock, LiveDebuggingBlock
      ├── capsule.rs                            – LogsReceiver, MetricsReceiver, OtelcolConsumer, RelabelRules, TargetEntry
                                                  (+ string_map, logs_receiver_list,
                                                  metrics_receiver_list, otelcol_consumer_list, target_list helpers)
      ├── loki.rs                               – LokiEchoBlock, LokiSourceJournalBlock, ...
      ├── relabel.rs                            – RelabelRule + RelabelSubBlock (shared by *.relabel)
      ├── discovery.rs                          – DiscoveryKubernetesBlock, DiscoveryRelabelBlock,
      │                                            KubernetesSubBlock variants
      ├── prometheus.rs                         – Prometheus{Echo,Relabel,Scrape,ReceiveHttp,RemoteWrite,
                                                  OperatorPodMonitors,OperatorServiceMonitors}Block
                                                  (+ endpoint/http/clustering/basic_auth/tls_config/
                                                  selector/scrape sub-blocks)
      └── otelcol.rs                            – OtelcolProcessor{Batch,MemoryLimiter,Attributes,GroupByAttrs,Filter,Transform}Block
                                                  (+ shared output block, attributes action block, Ottl type,
                                                  Processor/Attributes/Filter/Transform sub-block enums)

packages/mz-monitoring-build/                   ← CLI: `mz-monitoring-build gen-pipelines`

Build/test:

make pipelines                          # render YAML → .alloy + run `alloy validate` per target
cargo test -p mzmon-lib                 # unit + oracle tests (uses `alloy fmt` if present)
cargo clippy -p mzmon-lib --all-targets # lints
cargo fmt                               # the team runs this in pre-commit; run it before
                                        #   committing any Rust changes

Current Pipeline State

This section captures the live state so the next session has something concrete to start from. Update it when state changes meaningfully (new pipeline, new typed component, schema gap closed, etc).

Pipeline inventory

Both pass alloy validate. agent.yaml is a faithful port of the reference staging-agent.alloy (packages/ref-alloy-pipelines/, not checked in).

There are ZERO raw: blocks left in any pipeline. Every component in agent.yaml, gateway.yaml, gateway-metrics.yaml, and gateway-dest-stub.yaml is typed and schema-validated. raw: remains in the schemas as the escape hatch — the last oneOf branch of every sub-block list, and available for any component — but nothing currently uses it. Adding an untyped component should be a deliberate, temporary choice, not the default.

The conversion was verified by rendering before and after: agent.alloy, gateway-metrics.alloy, and gateway-dest-stub.alloy came out byte-identical, and gateway.alloy differs only in logs/metrics ordering inside one output block (the typed struct's canonical order vs. YAML order — semantically irrelevant to alloy, see the attribute-order note below). Both configs were also load-tested under a real alloy run, since alloy validate cannot see capsule/type mismatches.

Agent log sidechannel (gateway.yaml): the gateway tails the agent DaemonSet's own logs from the Kubernetes API — discovery.kubernetes "agentpods" (typed, server-side selectors.label = app.kubernetes.io/name=alloy-agent) → discovery.relabel "agentpods" (typed) → loki.source.kubernetes "agent_logs" → loki.process.inputProcessor. Points that are load-bearing rather than stylistic:

• The agent no longer tails its own logs — discovery.relabel.local_pods in agent.yaml drops targets whose app.kubernetes.io/name is alloy-agent. The sidechannel is always on, so self-collection would deliver every line twice. The two changes are a pair.
• clustering { enabled = true } on the tail is required, not an optimization. The gateway runs 2+ replicas and each would otherwise tail every agent pod.
• RBAC is already covered by the alloy subchart's default rbac.rules (pods, pods/log, namespaces), which the chart does not override.
• The {app="alloy"} selector in inputProcessor was dead. app comes from app.kubernetes.io/name, which the subchart aliases make alloy-agent/alloy-gateway, so the reference deployment's bare alloy never matched — meaning alloy's own logs were never logfmt-parsed and the loki.echo drop never ran. Now {app=~"alloy(-agent|-gateway)?"}. That drop is what stops the echo tap's re-emitted pod logs being ingested a second time via the sidechannel.
• Verified end-to-end against a real API server (kind), not just alloy validate: discovery → relabel → tail delivered live log lines, confirming targets = discovery.relabel.X.output is the right capsule shape for loki.source.kubernetes.

Container metrics come from the kubelet, not from a cAdvisor we run. gateway.yaml: discovery.kubernetes role=node → discovery.relabel (node-label allowlist) → prometheus.scrape "kubelet_cadvisor" (https, /metrics/cadvisor, SA bearer token, clustered) → the otelcol metrics pipeline. Everything below was measured on a live GKE cluster, not reasoned about:

• An in-agent prometheus.exporter.cadvisor cost ~750Mi per node against a 200Mi logs-only envelope, and OOM-crashlooped the DaemonSet (180-224 restarts). A heap profile attributed ~34% to cAdvisor housekeeping and ~31% to the scrape loop reading it. The kubelet already computes those stats for eviction and the metrics API, so running our own computed them twice per node.
• The completeness argument for running our own does not survive measurement. A GKE kubelet serves 69 distinct container metrics vs 70 from an in-process cAdvisor; the only real loss is containerhealthstate, which nothing queries. All 23 container metrics this repo's queries reference are present. Where GKE dashboards look sparse it is the collector scraping a subset, not the kubelet withholding data.
• The kubelet serves the process-collector family too (containerfiledescriptors, containersockets, containerthreads, containerulimitssoft, container_processes). Those are why the agent briefly ran hostPID: true; that privilege — and the /proc/<pid>/environ exposure it grants alongside uid 0 — is no longer needed.
• The kubelet labels series namespace / pod / container directly, so none of the containerlabeliokubernetes* remapping an in-process cAdvisor needs applies.
• TLS: the in-cluster CA verifies the kubelet certificate on GKE (measured), so cafile + the SA token is enough and insecureskipverify stays false. Other distributions may need it true, and the failure is silent — hence GATEWAYKUBELETTLSINSECURE.
• clustering { enabled = true } on the scrape is required, not an optimization: the gateway runs many replicas and each would otherwise scrape every node (~6.7k series/node).
• RBAC needs nothing added — the alloy subchart's default clusterRules already grant nodes, nodes/metrics, nodes/proxy.
• GOMEMLIMIT is set on both roles (~80% of the memory limit). Go's GC has no knowledge of a cgroup limit, so it grows the heap toward a ceiling the kernel enforces by killing the process; telling the runtime about it converts an OOM-kill into GC pressure. Keep it in step with the limit whenever either moves.

**Top-level loki.* components**: loki.echo, loki.process, loki.relabel, loki.source.journal, loki.source.file (incl. the filematch sub-block), loki.source.api (+http server), loki.source.kubernetes (+clustering), loki.source.kubernetesevents, loki.write (+endpoint).

**Top-level discovery.* components**: discovery.kubernetes, discovery.relabel.

**Top-level prometheus.* components**: prometheus.echo, prometheus.relabel, prometheus.scrape, prometheus.receivehttp, prometheus.remotewrite, prometheus.operator.podmonitors, prometheus.operator.servicemonitors. Scoped to the stable, in-cluster surface — see [Metrics](../../../docs/content/reference/internal/pipelines/metrics.md) for the per-component raw:-deferred list (remotewrite endpoint auth/TLS/queue tuning, operator client, scrape oauth2/authorization, selector matchexpression, receive_http tls).

**prometheus.* sub-blocks**: clustering (enabled, shared by scrape + operators), basicauth/tlsconfig (on scrape), http (receivehttp server), endpoint (remotewrite — url + scalars + a raw-only nested blocks for auth/queue), selector (operator — matchlabels + raw-only nested blocks for matchexpression), scrape (operator defaults), and the shared rule (cross-file $ref to loki.schema.yaml#/$defs/rule, reusing the RelabelRule sugar).

prometheus.scrape's scrapeinterval/scrapetimeout are Expressable<GoDuration> via the existing expressionOrDuration $def (the same one operatorScrape uses), so the interval can be env-driven — it is the dominant cost lever on a per-node cAdvisor. That widened the struct enough that ComponentBlock::PrometheusScrape is now Boxed for clippy::largeenumvariant, like memory_limiter.

**Top-level prometheus.exporter.* components**: prometheus.exporter.cadvisor (the only one so far). Its collector allowlists are [ExpressableList](#expressablelist) rather than Expressable, and its toblock emits attributes in declaration order rather than grouped by type, so storecontainer_labels renders next to the allowlist it gates.

**Top-level otelcol.* components**: otelcol.processor.{batch,memorylimiter,attributes,groupbyattrs,filter,transform}. filter is now wired into gateway.yaml (the metrics pipeline is otelcol-primary — see the inventory row above); the rest are schema+sugar only so far. memorylimiter's limit knobs (checkinterval/limit/spikelimit/limitpercentage/spikelimitpercentage) are all Expressable for env-driven sizing (its ComponentBlock variant is Boxed — five Expressable fields made it the outsized variant, clippy::largeenumvariant). Receivers and exporters are now typed too: otelcol.receiver.{otlp,prometheus} and otelcol.exporter.{otlp,loki,prometheus}. Sub-blocks: grpc/http (one shared server struct — the variant names the block, so its ToBlock is hand-written rather than the dispatch macro; an empty grpc: {} is meaningful, since presence is what enables the listener), client + nested tls on the OTLP exporter. Still deferred to raw:: connectors, the per-processor debugmetrics block, exporter sendingqueue/retryon_failure, and (on filter) the deprecated traces/metrics/logs blocks.

OTTL (filter/transform) is carried by a dedicated Ottl newtype (ast.rs, a sealed LiteralScalar so fields are Vec<Expressable<Ottl>> — literal OTTL string OR an expression). We do NOT model OTTL's grammar/function library; statements render verbatim as normal (escaped) alloy strings. filter's conditions blocks and transform's statements/statements blocks share one Rust struct each (FilterConditionsBlock, TransformStatementsBlock), so their ToBlock is hand-written (not the macro) to pass each block name.

GOTCHA — otelcol needs alloy >= 1.17 and a label on every component. Confirmed by a real alloy validate (the schema/renderer agreeing is NOT proof — these aren't wired into a pipeline, so make pipelines doesn't validate them): (1) filter's *_conditions blocks are 1.17+ only — older alloy has just the deprecated traces/metrics/logs and errors "unrecognized block name"; we pin 1.17.1. (2) EVERY otelcol component must have a label or alloy load fails "must have a label" — so label is a required field on the otelcol structs and schema (required: [label]), deliberately diverging from the loki/prometheus families which leave label optional.

**otelcol.* sub-blocks**: output (shared — metrics/logs/traces lists of the OtelcolConsumer capsule; bare refs), action (attributes), conditions (filter: context + conditions), statements (transform: context/statements/conditions/error_mode), statements (transform inferred: trace/metric/log). Sub-block enums: ProcessorSubBlock, AttributesSubBlock, FilterSubBlock, TransformSubBlock.

loki.process stages (and stage.match body, recursively): stage.match, stage.drop, stage.limit, stage.regex, stage.replace, stage.template, stage.logfmt, stage.json, stage.timestamp, stage.labels, stage.staticlabels, stage.labeldrop, stage.structuredmetadata, stage.structuredmetadata_drop, stage.sampling, stage.cri (empty, no attributes), stage.tenant (label/source/value, each Expressable<String>).

discovery.kubernetes sub-blocks: selectors (incl. field/label as Expressable<String>), attach_metadata (node + namespace). Other sub-blocks (e.g. namespaces) use raw:.

***.relabel sub-blocks**: rule (shared $def used by loki.relabel, discovery.relabel, and prometheus.relabel via cross-file $ref to loki.schema.yaml#/$defs/ruleBlock; the operator components' rule refs #/$defs/rule directly).

Literal-or-expression fields use Expressable<T> (ast.rs): a field typed Expressable<f64|String|bool> (or Option<…>) accepts either a scalar literal or an inline expression object ({env}, {function}, {operator}, {ref}). In use on stage.limit rate/burst, stage.drop olderthan, stage.staticlabels values, selectors field/label, and the operator scrape block's defaultscrapeinterval/defaultscrapetimeout. Schema side: anyOf: [{type: <scalar>}, {$ref: common/expression.schema.yaml}] (safe — scalar vs object are disjoint). Scalars only — never Expressable<map/object> (a literal map would collide with the expression object, the same overlap that forced anyOf in the raw attributeValue). This is an actively-expanding pattern; adopt per-field as needed.

<a id="expressablelist"></a>ExpressableList (ast.rs) is the list analog of Expressable, which stays sealed to scalars. The sealing exists because a literal map is shape-indistinguishable from the expression object; a literal list is not (array vs object are disjoint), so the untagged dispatch is unambiguous with Literal first. Used by cAdvisor's collector allowlists, which are wired to env vars via encoding.from_json.

Fields widened to accept expressions during the raw-block conversion: RelabelRule.replacement (stamping node from HOSTNAME), remoteWriteEndpoint.url and lokiWriteEndpoint.url (env-coalesced destinations), prometheus.scrape.scrapeinterval/scrapetimeout, loki.source.api's listen_port, and the otelcol client.endpoint / server.endpoint.

Every sub-block oneOf ends with a raw: branch — the escape hatch is non-negotiable in the design.

Only the attributes we routinely use are documented per component — not exhaustive vs. alloy upstream. Strict additionalProperties: false is enforced; undocumented attributes are rejected at validation time with a hint pointing at the raw: escape vs. schema-extension choice. See [Authoring §The strict-attributes / raw-escape policy](../../../docs/content/reference/internal/pipelines/authoring.md#the-strict-attributes--raw-escape-policy).

Rust sugar deserialization status

The ComponentBlock enum in pipeline.rs dispatches to typed sugar structs (via #[serde(rename = "loki.echo")] etc.). Each typed component has a pub struct …Block { fields } + an impl ToBlock that normalizes to the generic Block AST for rendering. Sub-blocks follow the same pattern via per-component sub-block enums (KubernetesSubBlock, RelabelSubBlock). The ToBlock enum impls are generated by the impltoblock_dispatch! macro (ast.rs, pub(crate), $crate:: paths) — list every variant in the invocation; a missed one is a compile error.

Done: every component in the typed-schema coverage list above round-trips through the typed path. No pending sugar work.

Capsule types (components/capsule.rs) make the bare-ref invariant structural: LogsReceiver (forwardto), MetricsReceiver (forwardto on prometheus. — the metrics analog of LogsReceiver, list-wrapped via metricsreceiverlist), RelabelRules (relabelrules), and TargetEntry (targets, untagged Ref(Identifier) | Literal(IndexMap); prometheus.scrape reuses it). targets refs are list-valued and must NOT be bare-array-wrapped. A discovery. export (discovery.kubernetes.pods.targets, discovery.relabel.x.output) is a list(discovery.Target); a literal is a single discovery.Target. targets = [ <ref> ] is list(list(Target)) and alloy rejects it at load (conversion from '[]discovery.Target' is not supported) — even though alloy validate accepts it. So targetlist (capsule.rs) emits: a single ref directly (targets = discovery.x.targets), literals-only as an array ([{…}]), and multiple/mixed via array.concat(ref, …, [{…}]). (Earlier this list-wrapped and "verified against alloy validate" — a false green that broke a live agent; see the alloy validate gotcha below.) forwardto/relabelrules are NOT affected — those refs are single capsules, so list-wrapping (forwardto) or direct assignment (relabelrules) is correct. Schema side: $defs/target (discovery.schema.yaml, generic) vs $defs/fileTargetEntry (loki.schema.yaml, required: [path]) — strictness lives in the schema, the Rust type stays generic. A new capsule type is a newtype + From<&T> for AttributeValue via Expression::nametoref + a schema $def — OtelcolConsumer (the otelcol output block's metrics/logs/traces, list-wrapped via otelcolconsumerlist) is the worked example.

Load-bearing invariants

These are non-obvious things that must stay true — flagged here so reorders or refactors don't silently break them. Detail in [Authoring §Load-bearing invariants](../../../docs/content/reference/internal/pipelines/authoring.md#load-bearing-invariants).

• AttributeValue variant order: Null/Bool/Number/String/Array must come before Expression. Serde's untagged struct deserializer accepts a sequence by positional-field assignment, so ["a", "b"] would misroute to Expression { raw: Some("a"), env: Some("b") }. Regression-tested in ast.rs.
• #[serde(denyunknownfields)] on Expression: keeps generic objects ({mapping: ...}) from silently matching Expression with all heads None. Also regression-tested.
• Ref-valued attributes render as bare refs, never quoted strings — and this is now enforced by type: declare the field with a capsule type (Vec<LogsReceiver>, Option<RelabelRules>, Vec<TargetEntry>) and convert via the capsule helpers / Expression::nametoref. Do NOT hand-build Expression { refname: ... } in new toblock impls; if a new capsule kind is needed, add it to components/capsule.rs.
• raw: escape is always the last oneOf branch in every sub-block list. Adding a new typed branch goes before raw.
• A nested block list is Vec<RawOnlySubBlock> (or a typed sub-block enum), NEVER Vec<Block>. The schemas express a nested list as $ref: common/raw.schema.yaml — the externally-tagged {raw: {component, …}} wrapper — while Vec<Block> deserializes the unwrapped {component, …}. Schema-valid YAML then passes validation and dies in serde with missing field 'component', so the escape hatch reads as documented and is unusable. Four fields shipped that way in the raw-block conversion, because no pipeline exercises those escapes and nothing else notices. RawOnlySubBlock lives in ast.rs; pipeline.rs::rawescaperoundtripsineveryrawonlylist covers every such list and fails with that exact error if one regresses.
• A schema anyOf/$ref that admits an expression requires the matching Rust field to be Expressable. The two drift independently and the mismatch is silent in both directions: an expression-capable schema over a plain String field accepts YAML that then fails serde, and the reverse quietly rejects valid config. Widen (or narrow) both together.
• raw.schema.yaml's attributeValue uses anyOf, NOT oneOf: an expression-shaped object ({ref}, {env}, {operator}, {function}) is also a valid generic attributeObject, and a {value: ...} is also a commentedValue — so exactly-one oneOf rejected every expression/commented raw value (this blocked the first real expression-valued raw block in agent.yaml). The Rust AttributeValue deserializer disambiguates by variant order + denyunknownfields; the schema just needs "is some legal raw value," which anyOf gives. Don't revert it to oneOf. Regression-tested in pipeline.rs::rawblockwithexpressionvaluesroundtrips.
• Error::Multiple renders its children (header + indented bullet per child, recursive). Earlier it displayed a bare "multiple errors" and swallowed the detail — gen-pipelines failures were undiagnosable. Tested in error.rs.
• A schema file's $id tail MUST match its path (and the ID key in validate.rs). The Rust Registry registers each resource under an explicit key so it tolerates a mismatch, but yaml-language-server keys on the in-file $id and breaks (refs point at a URI no schema claims). Keep filename ↔ $id ↔ ID in lockstep.
• Known tooling bug — yaml-language-server Maximum call stack size exceeded: fires on our (necessarily) recursive schemas during the LSP's meta-schema validation; it's upstream (fixed by redhat-developer/yaml-language-server#1269), NOT our content. The CLI validator is the source of truth — don't flatten the recursion to appease the editor. Full writeup + debugging recipe in [authoring.md](../../../docs/content/reference/internal/pipelines/authoring.md#known-tooling-issue-yaml-language-server-maximum-call-stack-size-exceeded).
• Schemas are the reference docs: descriptions render in IDE hover and (eventually) in published reference; keep them user-facing. Each $def ends with a See: link to canonical alloy upstream.
• GOTCHA — schemas are embedded at compile time (includestr!). After editing any schemas/alloy/.yaml, you MUST cargo build --bin mz-monitoring-build before a manual gen-pipelines, or it validates against the stale embedded schema and reports phantom "doesn't match any typed schema" errors. make pipelines rebuilds the binary as a dependency (safe); cargo test recompiles the lib so tests see the fresh schema — which means unit tests can pass while a manual render of the same construct fails. If a render rejects something your tests accept, rebuild the binary first. (A new schema fragment file also needs registering in validate.rs: a SCHEMA includestr!, an ID* const matching its $id, and an entry in the Registry list.)
• GOTCHA — assertrenders enforces alloy-fmt-canonical output. It does an exact-bytes match AND (when alloy is on PATH) asserts the rendered output is what alloy fmt would produce. So a test fails if the renderer's output isn't canonical, even when your expected string matches the renderer. The renderer's alignment quirk (see Cleanup) makes some shapes non-canonical — write tests around shapes known to be canonical (single-line attribute groups, object literals) until the quirk is fixed. For a shape the renderer can't make canonical (single-line attr beside a multi-line array → alignment quirk; an inline object literal inside array.concat(...)), byte-check with a plain asserteq!(pipeline.render().unwrap(), …) and a comment, rather than assert_renders — see the targets/array.concat tests in loki.rs/discovery.rs.
• GOTCHA — alloy validate does NOT catch value-type (capsule) mismatches; a real load does. alloy validate checks syntax and resolves bare component references (dangling refs error), but it does not verify that an expression assigned to a capsule-typed argument actually yields that capsule. It green-lights forwardto = [sys.env("X")] and targets = [discovery.x.targets] (both wrong) — the errors (should be capsule, got string; conversion from '[]discovery.Target' is not supported) only surface under alloy run (initial graph eval). This produced two field bugs. When touching capsule-valued attributes (forwardto, targets, relabelrules, future otelcol.Consumer), verify with an actual load, not just validate — e.g. perl -e 'alarm 5; exec @ARGV' alloy run --disable-reporting --storage.path=/tmp/x --server.http.listen-addr=127.0.0.1:PORT cfg.alloy and grep for convert|capsule|failed to evaluate. assertrenders' alloy fmt oracle does NOT load either, so passing tests are not proof of load-validity.

Cleanup / refactor candidates

• writeexpression uses a renderedexpr flag: the idiomatic shape is a tuple-match over (&env, &raw, &function, &ref_name, &operator), which encodes "exactly one head set" structurally. The flag pattern bit us once (forgotten assignments); a refactor would prevent recurrence.
• Renderer block-attribute alignment (task #15): current rule "any multi-line value disables alignment for the whole group" is too aggressive; alloy fmt aligns in more cases. Concretely, the only divergences in the rendered agent.alloy vs alloy fmt are (a) rule blocks where a single-line attr (action, separator, replacement) sits in a group that also contains the multi-line sourcelabels array, and (b) loki.source.file's targets (single-line) next to the multi-line forwardto array. Everything else is canonical. Two ways out (an open decision): fix the alignment rule, or post-process rendered output through alloy fmt (we have alloy in CI now). Until then, assert_renders will reject tests that exercise those non-canonical shapes.
• Schema drift watch: description blocks link to canonical alloy docs. When alloy upstream renames a field or adds one we use, the schema description and properties need a corresponding update. There's no automation here; it's a manual sweep when bumping alloy versions.
• withcapacity + push loops in toblock impls (~5 sites): idiomatic form is self.blocks.iter().map(ToBlock::toblock).collect::<Result<Vec<>>>()?. Cosmetic; sweep opportunistically.
• Capsule newtype fields are pub (LogsReceiver(pub Identifier)): if ref-path validation is ever added, switch to private field + fn new() -> Result<Self>.

Queued work (non-binding — directions, not commitments)

Rough backlog from recent sessions; shapes may change. In loose priority:

• Fold agent logs into the OTLP exporter: the agent now has otelcol.exporter.otlp "gateway" for metrics while logs still go out via loki.write "gateway". Unifying them is tracked separately.
• replacemap sugar on discovery.relabel/loki.relabel: a {sourcelabel: targetlabel} map expanding to one action: replace rule each, in source order (relies on the preserveorder already enabled). Covers only the 1:1 replace case — multi-source / separator / replacement rules (e.g. job, path) stay explicit. Biggest readability win for the relabel-heavy blocks.
• Renderer alignment vs alloy fmt (the open decision above): fix the rule, or post-process through alloy fmt.
• $comment / inline-comment rendering: the schema already declares $comment/commentedValue, but Block has no comment field and the renderer drops them — pure plumbing. Doc-gen from comments is a stretch follow-on.
• Ref-resolution pass: forwardto/targets/relabelrules are free strings; nothing checks the referenced component exists (alloy validate catches it, but later + with worse messages).
• Reusable Expressable schema $defs (numberOrExpr/stringOrExpr): fields currently inline the anyOf; a couple of named defs would DRY it. Cosmetic.
• Golden snapshot test for the full rendered agent.alloy, once the above settle (so it doesn't churn).
• CI freshness: the pipelines job in .github/workflows/test.yaml asserts committed .alloy matches a fresh render — keep rendered output committed.

A few decisions are deliberately deferred: typing gateway.yaml's ingress components (see Cleanup); loki.write remote/auth config (it's exceptional — shared by agents + gateways, may target remote sinks; the gateway sink currently renders basicauth-free with a GATEWAYLOKIDEST env-coalesced url); and a real (build- or runtime-) parameterization mechanism for numeric knobs like stage.limit rates (current encoding.fromjson env coercion is provisional).