Report Script to Grafana Jsonnet Dashboard

Migrate Python email reports (Elasticsearch queries) to real-time Grafana dashboards with dual-datasource support (ClickHouse + ES7/ES8). Preserve report calculations while enabling interactive visualization.

Not suitable for: Standard dashboard creation (use grafana-json-to-jsonnet for JSON imports), refactoring existing Jsonnet (use grafana-jsonnet-refactor), or single-datasource dashboards.

Workflow with progress tracking

Copy this checklist and track your progress:

Migration Progress:
- [ ] Step 1: Read datasource-mapping.md for ES/ClickHouse patterns
- [ ] Step 2: Extract report metrics and logic
- [ ] Step 3: Map report sections to panel types
- [ ] Step 4: Define dual datasource configuration
- [ ] Step 5: Implement panels with explicit datasource selection
- [ ] Step 6: Compile and verify against report outputs

Step 1: Read datasource-mapping.md

Load references/datasource-mapping.md to understand Elasticsearch and ClickHouse query target patterns.

Step 2: Extract report metrics and logic

From the Python script, identify:

• Queries (ES aggregations, filters)
• Time windows and date ranges
• Post-processing calculations
• Grouping and aggregations
• Metric formulas

Step 3: Map report sections to panel types

Use this mapping:

• Summary numbers → panels.statPanel
• Time trends → panels.timeseriesPanel
• Top-N rankings → panels.tablePanel
• Comparisons → panels.barGaugePanel or timeseries with bars theme

For detailed mapping examples, see references/examples.md.

Step 4: Define dual datasource configuration

Create config with both datasources:

local config = {
  datasources: {
    elasticsearch: { type: 'elasticsearch', uid: ES_UID },
    clickhouse: { type: 'grafana-clickhouse-datasource', uid: CH_UID },
  },
  pluginVersion: '12.3.0',
};

For manual import mode, use ${DSELASTICSEARCH} and ${DSCLICKHOUSE} UIDs, and add inputs and requires blocks so Grafana can prompt for datasources on import.

Step 5: Implement panels

Implement each panel using unified libraries. Select datasource explicitly per panel. Preserve report calculations and metric semantics. Use standards. units/thresholds, themes. for timeseries style, and layouts.* or panels.withIdAndPatches(...) for grid placement.

Step 6: Compile and verify

Run mixin/build.sh or mixin/build.ps1. Verify panel results match the report for a known time window. Test both ES7/ES8 and ClickHouse queries in Grafana.

Quality checks

• Build succeeds (mixin/build.sh or mixin/build.ps1).
• Panel results match the report for a known time window.
• ES7/ES8 and ClickHouse queries return data in Grafana.
• Jsonnet is kept in a single file with local helpers (no dashboard-specific libs).
• inputs / requires are present when manual import is supported.
• Variables return values in Grafana; no duplicate or extra variables.
• Regex filters preserved or added where needed.
• Row membership is correct (gridPos.y aligns to row gridPos.y, and rows include panels).

Dual datasource example

local ES_UID = 'elasticsearch-prod';
// local ES_UID = '${DS_ELASTICSEARCH}';

local CH_UID = 'clickhouse-prod';
// local CH_UID = '${DS_CLICKHOUSE}';

local config = {
  datasources: {
    elasticsearch: { type: 'elasticsearch', uid: ES_UID },
    clickhouse: { type: 'grafana-clickhouse-datasource', uid: CH_UID },
  },
  pluginVersion: '12.3.0',
};

// Panel using Elasticsearch
local errorCountPanel = panels.statPanel(
  title='Error Count',
  targets=[/* ES query */],
  datasource=config.datasources.elasticsearch,
  unit=standards.units.short,
  pluginVersion=config.pluginVersion
);

// Panel using ClickHouse
local requestsPanel = panels.timeseriesPanel(
  title='Requests',
  targets=[/* ClickHouse query */],
  datasource=config.datasources.clickhouse,
  unit=standards.units.qps,
  pluginVersion=config.pluginVersion
);

Formatting guardrail

• Do not run jsonnetfmt / jsonnet fmt on generated Jsonnet files. Keep formatting manual and consistent with grafana-code mixin style.

References (load as needed)

• references/datasource-mapping.md
• references/full-report-playbook.md
• references/examples.md