Add Datasource Type

Orchestrates adding a new datasource type plugin — from API discovery through verified implementation. Three stages, worked autonomously: the stage boundaries are checkpoints you satisfy, not approvals you wait for.

When to Use

• User wants gcx CLI support for a datasource type gcx does not yet support
• User says "add support for an unsupported datasource type", "new datasource type"
• A task references datasource type implementation

When NOT to use:

• The user wants a datasource instance, not a type. "Add a datasource" most

often means creating or configuring one in a Grafana stack — that is gcx datasources create / update, already shipped. This skill writes Go code to teach gcx a new kind. Confirm which one is meant before starting.

• The kind is already registered under internal/datasources/providers/ —

extend that implementation instead of adding a duplicate.

• The product is a Grafana Cloud product, not a datasource — use

/add-provider.

Entry paths

Invoked from integrate-with-gcx (the placement section already exists — necessity, command path, backend evidence, wiring, readiness):

• Skip the Stage 1 questions it already answers: the datasource kind and plugin

type string, the query/metadata endpoints, and the readiness verdict. Record them and move on rather than re-asking.

• The Stage 1 approval gate does not apply on this path. Build autonomously.

If a query-language or endpoint detail is genuinely missing, discover it from the vendor docs, or ask one targeted question carrying the evidence and a recommendation — never fall back to a blanket approval gate.

• Start at Stage 2, and use Stage 3 verification as written.

Invoked directly (no placement section): work through all three stages. Autonomy is the same as above — the Stage 1 gate is a checkpoint you satisfy, not an approval you wait for. Discover the plugin type, endpoints and response shapes from bin/gcx datasources list -o json, the vendor's API docs and bin/gcx api probes; present findings and keep going. Ask only where an unresolved answer would materially change the implementation — an unknown query-expression format, an endpoint you cannot verify. If no instance is reachable, report the live checks as UNVERIFIED with the reason rather than blocking or claiming them green.

The flow, however you got here

contract (proportional)  →  implementation  →  Review

Both entry paths run all three, and none of them is a document or a gate:

• Contract, before code — .claude/skills/integrate-with-gcx/references/contract-and-tests.md,

sized to the change. If you arrived from integrate-with-gcx the contract already exists; use it, don't redo it.

• Review, before calling it review-ready —

.claude/skills/integrate-with-gcx/references/self-review.md, re-run after every fix push.

That is where the naming, typed-input, output-class, completeness, error, token-cost and test-quality guidance lives. Read those two rather than restating them here.

Workflow

Discover ───────> Implement ──gate──> Verify
   │                    │                  │
   v                    v                  v
research findings   code per step      smoke tests

Prerequisites — discover these, don't ask for them

Settle each from the repo and the environment first. Ask only if what remains is materially insufficient, and then in one grouped question carrying the evidence and a recommendation:

• Datasource type — usually stated in the request. Confirm the plugin type

string yourself with bin/gcx datasources list -o json.

• Access — check for a configured context the same way. If none is reachable,

proceed against the vendor's API docs and report every live check as UNVERIFIED with the reason; do not stop.

• Scope — infer from the request (a "query client" means query first) and

state what you inferred. Extra verbs are additive later; a wrong frozen name is not.

Stage 1: Discover

1a. Gather User Context

1. Run bin/gcx datasources list -o json to find the datasource UID and plugin type

string. If the user has a configured context, do this yourself rather than asking them to do it.

1. Find the query language and endpoint shapes from the vendor's API docs or the

plugin's source before writing anything — do not guess them. If they cannot be settled that way, ask once, naming exactly what is missing and what you will assume otherwise.

1. Known quirks — special auth, pagination, response formats?

1b. Research

• Use bin/gcx api raw calls to probe the datasource proxy API surface

(/api/datasources/proxy/uid/{uid}/... or /api/datasources/uid/{uid}/resources/...)

• Identify query endpoints and response shapes from the vendor's API docs or the

plugin's source

• Identify metadata endpoints (labels, series, etc.) the same way; if a non-query

endpoint cannot be established, record it as UNVERIFIED rather than guessing

1c. Record findings

Keep them in your working notes and the PR description; write a standalone docs/research/ report only if the investigation has lasting repository value or staged work must resume from it. Either way, what you record must cover:

• API endpoints and response shapes
• Query request/response format
• Available metadata operations
• At least one successful probe result — or, if no instance is reachable, the

probe you would run, marked UNVERIFIED with the reason

Checkpoint: Research Complete

Direct-invocation path only — see [Entry paths](#entry-paths).

Stage 2: Implement

Step 1: Query Client

Create internal/query/{kind}/ with client.go and types.go. Add a formatter.go only if you define your own response type — if you reuse a shared one, its formatter and codecs already exist (see Step 3).

Start with the shared transport — it is the default, not an optimisation. AGENTS.md Key Conventions: a client that calls Grafana's unified datasource query API (/apis/query.grafana.app/.../query, with the /api/ds/query fallback) must reuse internal/query/grafanaquery for the HTTP transport (POST + fallback + response-size limiting) and internal/query/dataframe for the data-frame wire types. Do not duplicate that logic or re-declare GrafanaQueryResponse/DataFrame. Check the current set with grep -rl query/grafanaquery internal/query/ and copy the closest one.

If the datasource takes raw SQL, the request body and --limit enforcement are shared too — querysql.BuildRawQueryBody and querysql.EnforceLimit with a dialect-local bail predicate, never a hand-rolled clamp. Read references/raw-sql.md before writing either: it carries the plugin-format exception, the stderr disclosure capped owes the caller, the four statement shapes bail has to catch, and which of the existing dialects is safe to copy.

Pick the client shape from what your commands actually call — there are three, and the middle one is the common case. The two transports are not alternatives: unified query is a POST to /apis/query.grafana.app/.../query, while label, series, metadata and other discovery endpoints are proxy/resource requests (/api/datasources/uid/{uid}/resources/...) that grafanaquery.Client cannot reach — it only exposes Execute. The verb there is per-plugin, not always GET: prometheus and loki GET their label endpoints, athena POSTs a body to its resource endpoint (internal/query/athena/client.go:42).

1. Query-only — a query leaf and nothing else:

type Client struct {
    queryClient *grafanaquery.Client
}

func NewClient(cfg config.NamespacedRESTConfig) (*Client, error)
func (c *Client) Query(ctx context.Context, uid string, req QueryRequest) (*QueryResponse, error)

1. Hybrid query + discovery — a query leaf plus labels/metadata/

series. Hold both transports; this is what every reference client with non-query commands does (prometheus, loki, cloudwatch: restConfig + httpClient + queryClient; athena: host + httpClient + queryClient):

type Client struct {
    restConfig  config.NamespacedRESTConfig     // or a plain host string
    httpClient  *http.Client                    // rest.HTTPClientFor(&cfg.Config)
    queryClient *grafanaquery.Client
}

func (c *Client) Query(ctx context.Context, uid string, req QueryRequest) (*QueryResponse, error)  // queryClient
func (c *Client) Labels(ctx context.Context, uid string) ([]string, error)                         // httpClient

Build httpClient with rest.HTTPClientFor(&cfg.Config) — a fresh http.Client drops the kubeconfig auth and TLS wiring, so calls can fail authentication or TLS depending on how the context is configured.

1. Direct-HTTP only — the datasource is not served by the unified query API

at all: hold the config/host and httpClient, no queryClient.

Say in the PR which shape you used and why.

Wire types alias the shared package rather than redeclaring it (type GrafanaQueryResponse = dataframe.Response). Response-type reuse is a separate decision with a codec consequence — see Step 3.

Reference: internal/query/prometheus/, internal/query/loki/ (hybrid, built on grafanaquery + dataframe), internal/query/athena/client.go (hybrid over a plain host)

Step 1b: Command Constructors

Create internal/datasources/{kind}/ with command constructor files:

• query.go — QueryCmd(loader providers.ConfigLoader) cobra.Command
• labels.go — LabelsCmd(...) (if the datasource supports label discovery)
• Other commands as needed (metadata, series, etc.)

Each file follows this pattern:

package {kind}

import (
    "github.com/grafana/gcx/internal/agent"
    dsquery "github.com/grafana/gcx/internal/datasources/query"
    "github.com/grafana/gcx/internal/providers"
    "github.com/grafana/gcx/internal/query/{kind}"
    "github.com/spf13/cobra"
)

func QueryCmd(loader *providers.ConfigLoader) *cobra.Command {
    shared := &dsquery.SharedOpts{}
    var datasource string

    cmd := &cobra.Command{
        Use:   "query [EXPR]",
        Short: "Execute a query against a {Name} datasource",
        Long: `Execute a query against a {Name} datasource.

EXPR is the query expression to evaluate; --expr is accepted instead.
Datasource is resolved from -d flag or datasources.{kind} in your context.`,
        Example: `
  # Query using configured default datasource
  gcx datasources {kind} query 'EXPR'

  # Query with explicit datasource UID
  gcx datasources {kind} query -d UID 'EXPR' --since 1h

  # Output as JSON
  gcx datasources {kind} query -d UID 'EXPR' -o json`,
        // RangeArgs, not ExactArgs — the current family accepts the
        // expression positionally OR via --expr. Copy the newest kind
        // (internal/datasources/athena/query.go) rather than this sketch.
        Args: cobra.RangeArgs(0, 1),
        RunE: func(cmd *cobra.Command, args []string) error {
            // ... resolve datasource, create client, execute query
        },
    }

    cmd.Annotations = map[string]string{
        agent.AnnotationTokenCost: "medium",
        agent.AnnotationLLMHint:   "gcx datasources {kind} query -d UID 'EXPR' -o json",
    }

    // The bool is enableGraph. Default to false: it decides whether `graph`
    // is a valid -o value and advertised in the format help, so only pass true
    // once a graph codec arm actually handles your response type (Step 3).
    shared.Setup(cmd.Flags(), false)
    cmd.Flags().StringVarP(&datasource, "datasource", "d", "", "Datasource UID")
    return cmd
}

Command field conventions:

• Long: Include a description of what the command does plus how the datasource

is resolved. Mention datasources.{kind} as the config key.

• Example: Use gcx datasources {kind} <subcommand> format (not the top-level

provider path). Use UID as the placeholder for datasource UIDs.

• Annotations: Set agent.AnnotationTokenCost ("small" for metadata/labels,

"medium" for queries) and agent.AnnotationLLMHint (a representative one-liner using gcx datasources {kind} ... format). Import "github.com/grafana/gcx/internal/agent".

Reference: internal/datasources/prometheus/, internal/datasources/loki/

Step 2: DatasourceProvider

Add a registration file in internal/datasources/providers/. This package contains one registration file per built-in datasource — ls it for the current set and copy the most recently added one rather than trusting a list here.

// internal/datasources/providers/{kind}.go
package providers

import (
    "github.com/grafana/gcx/internal/datasources"
    "github.com/grafana/gcx/internal/datasources/{kind}"
    "github.com/grafana/gcx/internal/providers"
    "github.com/spf13/cobra"
)

func init() { //nolint:gochecknoinits // Self-registration pattern (like database/sql drivers).
    datasources.RegisterProvider(&{kind}DSProvider{})
}

type {kind}DSProvider struct{}

func (p *{kind}DSProvider) Kind() string      { return "{kind}" }
func (p *{kind}DSProvider) ShortDesc() string { return "Query {Name} datasources" }

func (p *{kind}DSProvider) QueryCmd(loader *providers.ConfigLoader) *cobra.Command {
    return {kind}.QueryCmd(loader)
}

func (p *{kind}DSProvider) ExtraCommands(loader *providers.ConfigLoader) []*cobra.Command {
    return []*cobra.Command{
        // {kind}.LabelsCmd(loader),
    }
}

The DatasourceProvider interface is defined in internal/datasources/provider.go. The loader is supplied by the mounting code in cmd/gcx/datasources/command.go, which binds --config on each provider sub-command. The root owns --context and passes its value through the command context. Forward the loader to each command constructor.

Reference: internal/datasources/providers/prometheus.go.

Step 3: Codec dispatch — the step that breaks the default invocation

internal/datasources/query/codecs.go ends RegisterCodecs with ioOpts.DefaultFormat("table"), so the plain gcx datasources {kind} query … invocation — no -o — goes through the table codec. Its Encode is a type switch that falls through to errors.New("invalid data type for query table codec") (line 59), so a new response type with no arm there fails at encode time. The wide and graph codecs have their own fallthrough errors (lines 92 and 156); JSON and YAML do not type switch at all — they delegate to the shared format codecs and serialize whatever they are handed, which is why the default invocation breaks while -o json looks fine. Lint and unit tests will not catch it.

Two ways out, cheapest first:

1. Reuse an existing response type. If your results are table-shaped, returning

internal/query/sql's response gets you the table and wide codecs for free. Verify the fit first: internal/query/sql/parse.go takes Frames[0] only, so reuse it for genuinely SQL/sqlds-backed one-frame results and not for any query that returns one frame per series. It also carries a graph consequence: the graph codec's *querysql.QueryResponse arm returns "graph output is not supported for SQL datasource queries", so pass shared.Setup(flags, false) and keep graph off your -o list — athena and clickhouse both do exactly that (internal/datasources/athena/query.go:130, internal/datasources/clickhouse/query.go:128). Leaving graph enabled advertises a format that fails with an error naming a datasource family the caller never asked about.

1. Add an arm to every codec your type can actually reach — table and wide

always, plus graph when a graph-enabled path can return your type.

On graph specifically, two different commands are in play, which is why "keep graph off" and "still write a graph arm" are both right. Your typed command controls its own -o list via shared.Setup(flags, enableGraph) — keep that false unless the arm genuinely renders a chart. The generic datasources query is separate and passes true (cmd/gcx/datasources/query.go:211), so if you also add a case to its switch, your response type reaches the graph codec no matter what your typed command did. Give it an explicit "not supported for {kind}" arm there: that is why querysql.QueryResponse has one (internal/datasources/query/codecs.go:148) even though athena and clickhouse both pass false. Without the arm the caller gets the generic "invalid data type for graph codec", which names nothing. If your kind is not* in the generic switch and your typed command disables graph, the arm is unreachable — skip it, per self-review T1 check 4.

Then trace each registration in RegisterCodecs to a reachable Encode (self-review T1 check 4).

Step 4: Registration & Wiring

1. The internal/datasources/providers/ package is already blank-imported in

cmd/gcx/root/command.go — new registrations in that package are automatically picked up. No import changes needed.

1. NormalizeKind() mapping — Grafana plugin IDs often differ from the short

kind name (e.g., grafana-pyroscope-datasource → pyroscope, prometheus → prometheus). Check the plugin ID via gcx datasources list -o json and add a mapping in internal/datasources/query/resolve.go if they don't match. Without this, auto-discovery and datasource type validation will fail silently.

1. Routing for the auto-detecting datasources query — registration mounts

your typed datasources <kind> subtree but does not reach the generic command, which routes through the tables in cmd/gcx/datasources/query_routes.go. Add exactly one entry, keyed by the normalized kind: - the generic <uid> <expr> form can honestly carry your query → add a dispatch entry plus a small handler alongside the existing ones; - it cannot, because your query takes structured parameters no single expression represents → add a redirects entry built with structuredQueryRedirect, naming your typed command. CloudWatch is the worked example.

Adding neither is also a choice: your kind then reports as unsupported. Make it deliberately — a caller who reasonably reaches for datasources query gets a dead end. The two tables must stay disjoint and keyed by normalized kinds; queryroutesinternal_test.go enforces both.

The supported-kind list in the unsupported-type error is derived from the tables, so you never edit that string. You do update the two places that pin its exact value, because it is user-visible text on a GA path and is deliberately not free to drift: TestQueryRoutesSupportedKindsIsTheSortedUnion in queryroutesinternaltest.go, and wantUnsupportedMessage in queryunsupported_test.go. Both fail with the old and new lists side by side, so the update is mechanical.

Step 5: Agent Annotations

Annotations should already be set on each command via cmd.Annotations in the constructor (see Step 1b). Verify every leaf command has both agent.AnnotationTokenCost and agent.AnnotationLLMHint set.

If the datasource also needs entries in internal/agent/command_annotations.go (for commands that exist outside the DatasourceProvider path), add them there too:

"gcx datasources {kind} query": {Cost: "large", Hint: "..."},
"gcx datasources {kind} labels": {Cost: "small"},

Gate: mise run gate per step, GCXAGENTMODE=false mise run all once before push

Stage 3: Verify

3a. Smoke Tests

Only test the subcommands that were actually added:

# Build
mise run build

# Verify the parent command and each subcommand exist
bin/gcx datasources {kind} --help

# Test each subcommand against real Grafana — only commands you implemented
bin/gcx datasources {kind} query '<expr>' --since 1h
# bin/gcx datasources {kind} labels -d UID  (if labels was added)
# etc.

3b. Run Checks

# Full quality gates
GCX_AGENT_MODE=false mise run all

# Agent annotation consistency
mise exec -- go test ./internal/agent/...

Checkpoint: Verified

Reference Implementations

Common Pitfalls