migrate-required-input-dependency

Migrate an integration package to integrations with required input dependencies.

Authoritative guide: HOWTO: Migrate an integration package to use required input dependencies (docs/howto/migrateintegrationrequiredinputdependency.md in elastic-package on main)

At the start of Phase 1, read that guide from a local elastic-package checkout or from the URL above.

Reference implementation: elastic/integrations#19719 (packages/elasticpackageregistry). If main still shows legacy input: / inline collector templates, diff against the PR branch — do not copy pre-migration patterns from main.

Rules

1. Never edit package files until all decision gates in Phase 2 are answered and you have presented a migration plan summary for confirmation.
2. Prefer manifest variable overrides over hardcoding values in stream.yml.hbs (hardcoding causes Fleet UI values that have no effect). For each variable, be explicit about intent per the how-to guide Variable overrides section: who sets it (integration author vs end user), whether it appears in Fleet, and whether the rendered agent template references it via {{variable}} rather than a hardcoded literal.
3. Set dataset: on the data stream manifest when the integration dataset must differ from the input package default — do not expose data_stream.dataset as a user variable unless the developer explicitly chooses that approach.
4. Keep local stream.yml.hbs limited to integration-owned template fragments only.
5. Run elastic-package build and elastic-package test after migration; use elastic-package test policy --generate only after the developer reviews generated expectations.
6. Do not treat an unmigrated reference package on main as source of truth — use the guide and PR #19719 when packages/elasticpackageregistry is still legacy.

Phase 1 — Discover the package

1. Confirm elastic-package version succeeds. If missing, stop and point to the elastic-package install guide. Version should be minimum v0.125.1.

1. Locate the integration package root (manifest.yml, type: integration).
2. Read the legacy setup:

- manifest.yml — policytemplates, formatversion, conditions, existing requires - Each data stream's manifest.yml and agent/stream/*.hbs - fields/, ingest pipelines, dashboards tied to the current dataset/index name - _dev/test/config.yml, policy/system/pipeline tests

1. Identify the target input package — search local packages/ for type: input; if not found, check the package registry or ask the developer.
2. Diff legacy template vars/defaults against the input package manifest vars/defaults. Flag input-only variables (present on input, absent from legacy template) for Gate D.
3. Record the integration's historical dataset name(s) from policy tests, dashboards, outputpermissions, or datastream.dataset usage.

Present a short inventory: package name, data streams, legacy input type, proposed input package, variables that differ between legacy and input defaults, input-only variables, and common diffs (for example hosts path format).

Phase 2 — Gather developer decisions (required before migration)

Use AskQuestion when available; otherwise ask conversationally. Do not proceed to Phase 3 until every applicable gate below is resolved.

Gate 0 — Migration appropriateness

Gate A — Scope and dependency

Gate B — Stack and format version

Gate C — Dataset management

Explain the risk: without an explicit dataset, documents may index under the input package default (e.g. metrics-prometheus-*).

Default recommendation when unsure: dataset: on the data stream manifest.

Gate D — Variable overrides (per variable)

Follow the how-to guide Variable overrides section. The rendered agent policy merges three layers: input package template defaults, integration stream.yml.hbs, and user-selected values. Understanding which layer wins is critical.

Include every variable from the input package manifest, even if absent from the legacy template. For data_stream.dataset on the input package, prefer manifest dataset: (Gate C), not a stream var override.

Variables can be declared at stream level (streams[].vars in the data stream manifest) or input level (policy_templates[].inputs[].vars in the package manifest). Input-level declarations are promoted to input-scoped variables. Use stream-level vars for per-data-stream tuning; use input-level vars when the override applies to every data stream that references the input package in that policy template.

For each variable, ask the developer to classify:

For each A and B variable, also confirm variable intent:

• Who sets it: integration author default vs end user at policy creation?
• Fleet visibility: show_user: true (user-facing) or false (advanced/hidden)?
• Template binding: referenced via {{variable}} in stream.yml.hbs or merged from the input template — not a hardcoded literal that bypasses Fleet?
• Default value (confirm against legacy template)

Category C variables inherit from the input package during bundling with show_user: false by default (advanced options in Fleet) — no explicit redeclaration needed.

Explicitly ask whether any variable should be hardcoded in stream.yml.hbs. If yes, warn that Fleet may still show the input default in the UI and user edits will not apply. Document the choice in the migration plan.

Present the variable matrix (name → category → intent → default → show_user → template binding) and get confirmation before editing.

Gate E — Local development and tests

Gate F — Collateral changes

Gate G — Plan confirmation

Summarize the full plan:

• Manifest changes (requires.input, policytemplates, formatversion, version bump)
• Per data stream: remove legacy input: key, streams[].package, dataset:, category A/B streams[].vars only, slim template contents with {{variable}} bindings
• Variable intent matrix (categories A/B/C, Fleet visibility, template binding)
• Test and changelog changes

Ask the developer to confirm the plan before making any edits.

Phase 3 — Execute migration

Apply changes in this order (see migrateintegrationrequiredinputdependency.md):

1. manifest.yml — formatversion, requires.input, policytemplates → package: <input>, bump version per Gate F
2. datastream/<name>/manifest.yml — set dataset:; replace legacy input: with streams[].package; add templatepath: stream.yml.hbs; declare streams[].vars for categories A/B only; remove category C vars from local manifest
3. agent/stream/stream.yml.hbs — keep only integration-owned fragments; remove all collector config merged from the input package
4. _dev/test/config.yml — policy/system requires.source if Gate E applies
5. Policy tests — test-default.yml, test-overrides.yml; generate expectations only after developer approval; confirm every Fleet-visible variable maps to the rendered agent template and user-set values take effect; confirm every Fleet-visible variable maps to the rendered agent template and user-set values take effect
6. Ingest pipelines — re-run pipeline tests; add null and missing-field cases per Gate E/F
7. System tests — extend hit assertions and traffic fixtures per Gate E
8. changelog.yml — migration (enhancement), stack constraint, field fixes (bugfix) per Gate B/F
9. Docs — elastic-package build to regenerate docs; then manual input section in _dev/build/docs/ if Gate F requires it

Do not bump unrelated packages or refactor outside migration scope.

Phase 4 — Verify

From the package directory:

elastic-package build
elastic-package check
elastic-package test -v

If system tests need variants or traffic, run what the developer confirmed in Gate E.

Verify variables in Fleet

Per the how-to guide end-to-end verification step, install the built package in a local stack and create an agent policy when possible:

1. Fleet UI ↔ template binding — every variable shown in Fleet should have a corresponding entry in the rendered agent template ({{variable}} reference or merged input-template binding). Flag any variable visible in the UI whose effective value is a hardcoded literal in stream.yml.hbs — user edits to that field will not apply.
2. User overrides take effect — change a Fleet-visible variable in the policy UI and confirm the rendered agent policy updates (policy test overrides should cover this; Fleet spot-check when a variable is not exercised in tests).
3. Defaults match intent — Fleet defaults for categories A/B match the integration manifest; category C inherited vars appear under advanced options unless explicitly redeclared.

Report:

• Build/test pass/fail with relevant log excerpts
• Policy output: datastream.dataset and outputpermissions index names (e.g. metrics-<dataset>-ep); confirm every Fleet-visible variable maps to the rendered agent template
• Fleet variable spot-check results (UI fields shown, template bindings, user override behaviour) when a local stack was available
• Dashboard spot-check on target stack when Gate F confirmed
• Platform gaps still relevant after migration:

Verification checklist

Mark each item done or N/A:

• format_version ≥ 3.6.5 and requires.input pinned to a published input version
• dataset: explicitly set on the data stream manifest when it must differ from the input default
• Local stream.yml.hbs contains only integration-owned template fragments; integration-specific or overridden values use {{variable}} references, not hardcoded literals that bypass Fleet
• Variable intent is explicit per Variable overrides: manifest vars for categories A/B, inherit input defaults when acceptable (category C) — avoid silent template hardcoding that leaves misleading values in the Fleet UI
• Variable overrides use streams[].vars, not silent template hardcoding
• Legacy input: key removed; streams[].package in place
• _dev/test/config.yml declares requires for local input package during development
• Policy tests (default + overrides): expectations confirm every Fleet-visible variable maps to the rendered agent template and user-set values take effect — review dataset, overridden defaults, sibling streams (enabled: false where required); spot-check in Fleet when policy tests do not cover a variable
• System tests pass with realistic service traffic where needed
• Pipeline regression tests for edge cases found during migration
• Changelog entries: migration, stack constraint, field-mapping fixes (use breaking-change when mapping type updates affect existing indices)
• Docs manually updated if {{ inputDocs }} is empty
• Dashboards validated on the target stack version

Decision quick-reference

Suitable input package + stack 3.6+?     → Gate 0 must pass before migrating
Legacy var differs from input default?  → B: redeclare on streams[].vars
Var only in integration template?       → A: add var + {{variable}} in slim template
Input default matches legacy?           → C: inherit; remove from local template/manifest
Input-only var on input package?        → Classify in Gate D (often C or N/A)
Per-stream vs all-streams override?     → streams[].vars vs policy_templates[].inputs[].vars
Variable intent unclear?                → Who sets it, Fleet visibility, template binding — see how-to Variable overrides
Fleet UI shows var but template ignores?→ Hardcoding anti-pattern; use manifest override or document intentional
Dataset must stay stable?               → dataset: on data stream manifest
Unpublished input package?              → _dev/test/config.yml requires.source (tests only)
Bump input pins later?                  → elastic-package requires update

Anti-patterns

• Starting migration without Gate 0 — no suitable input package or unsupported stack
• Copying patterns from packages/elasticpackageregistry on main while PR #19719 is unmerged
• Migrating without confirming dataset name → silent index rename
• Skipping output_permissions index name review in policy expectations
• Hardcoding overrides in stream.yml.hbs without developer acknowledgement → Fleet UI mismatch; variable shown in UI but user edits ignored
• Using hardcoded literals in stream.yml.hbs for values that should be Fleet-configurable — use {{variable}} and manifest vars instead
• Using data_stream.dataset as a user variable when dataset: field suffices
• Leaving legacy input: alongside new streams[].package
• Running elastic-package test policy --generate and committing expectations without developer review
• Leaving full collector config in local template after switching to streams[].package
• Skipping ingest pipeline re-test after collector output shape changes
• Assuming requires.source in test config satisfies elastic-package build (build still uses registry)