Setting up a Custom REST source

A Custom source imports any HTTP REST API into queryable warehouse tables from a JSON manifest — no per-source Python. The manifest is a RESTAPIConfig: the same shape that powers PostHog's built-in REST connectors (Intercom, Attio, Sentry, …), so the generic REST engine handles auth, pagination, JSONPath record extraction, and incremental cursors for you. Your job is to author a correct manifest and prove it against live data before creating the source.

This is an alpha capability. Caps: at most 50 resources per manifest, and at most 5 Custom sources per project.

When to use this skill

• The user wants to import an API that has no built-in connector — an internal service, a niche SaaS, a public

API — and can give you its docs URL or describe its endpoints.

• The user explicitly asks for a "custom REST source", "custom source manifest", or to "build a connector from docs".

A Custom source is the fallback — only correct when no native connector fits. PostHog ships hundreds of native connectors, far more than could be listed here, so never assume an API has no built-in — most well-known SaaS apps and databases do, and guessing wrong misroutes them into a hand-authored manifest that duplicates a battle-tested connector. Step 0 below makes you check the registry before drafting anything; if a native connector matches, hand off to setting-up-a-data-warehouse-source instead.

The grammar

Read [references/manifest-reference.md](references/manifest-reference.md) before drafting — it is the full RESTAPIConfig field reference (auth types, the six paginators, incremental cursors, parent/child fan-out) with worked examples for each. Draft the manifest from that grammar; don't guess field names.

The skeleton:

{
  "client": {
    "base_url": "https://api.example.com/v1",
    "auth": { "type": "bearer" }
  },
  "resources": [
    {
      "name": "users",
      "primary_key": "id",
      "endpoint": {
        "path": "/users",
        "data_selector": "data",
        "paginator": { "type": "json_response", "next_url_path": "next" },
        "incremental": { "cursor_path": "updated_at", "start_param": "since" }
      }
    }
  ]
}

Secrets never go inline in the manifest. manifestjson holds only the non-secret structure. The credential travels in a separate payload key chosen by the manifest's client.auth.type: authtoken (bearer), authapikey (apikey), authpassword (httpbasic), or authoauth2clientsecret for oauth2 (plus authoauth2refresh_token for the refresh-token grant only). The engine injects it at run time, and PostHog redacts it from every response. Putting a token inline is rejected at validation.

Available tools

Workflow

Step 0 — Check for a native connector first

Before drafting anything, call external-data-sources-wizard to list the native source types and check whether the target API is among them, matching on the service name. A Custom source is the fallback for APIs with no native connector; do not skip this check on the assumption that a well-known API isn't supported — most are.

If a native connector matches, stop and tell the user the built-in path is simpler and battle-tested (it handles auth, pagination, schema, and incremental sync for you), and hand off to setting-up-a-data-warehouse-source. Only continue with this skill when the user has no matching native connector, or explicitly wants to exercise the custom REST path despite one existing.

Step 1 — Gather the API shape

Get either a docs URL (fetch it and read the auth scheme, the list endpoints, their response envelopes, and any pagination) or a natural-language description of the endpoints. You need, per resource you'll import:

• the path (relative to a common base_url) and method (GET, or POST for query-style read endpoints),
• the auth scheme (bearer token / API key in header or query / HTTP basic / OAuth2 with a customer-owned client —

clientcredentials or a pre-obtained refresh token; the interactive authorizationcode flow is not supported),

• the record path — where the array of records sits in the JSON response (e.g. data, results, items),
• how the API paginates (next-URL, link header, cursor, offset, page number, or single page),
• a primary key field, and
• optionally an incremental cursor field (updated_at-style) so re-syncs only fetch new/changed rows.

Ask the user for the credential value, but tell them you'll only ever place it in the auth_* payload key, never in the manifest.

Step 2 — Draft the manifest

Author the RESTAPIConfig from [references/manifest-reference.md](references/manifest-reference.md). Match the auth block to the scheme, pick the paginator that matches the docs, set dataselector to the record path, and add an incremental block when the API has an updatedat-style cursor and a matching query param. For an endpoint whose rows must be fetched per parent (e.g. /forms/{form_id}/responses), use a parent/child resolve param — see the fan-out example. Keep it to one level of nesting.

Step 3 — Validate and list resources

Call external-data-sources-db-schema with { sourcetype: "Custom", manifestjson: "<stringified manifest>", authtoken: "<credential>" }. The credential key is **not literally auth*** — use the one for your auth type: authtoken (bearer), authapikey (apikey), authpassword (httpbasic), or authoauth2clientsecret (+ authoauth2refreshtoken for the refresh-token grant). It validates the manifest structure, the fan-out graph, and the credential (a bounded live probe), then returns one table entry per resource with detectedprimarykeys and incremental_fields. If it returns a 400, the message is plain English (e.g. resources[0].endpoint.path: must not be empty) — fix the manifest and retry. Loop here until it validates.

Step 4 — Test-read each resource

For each resource, call external-data-sources-preview-resource with { sourcetype: "Custom", payload: { manifestjson, authtoken }, resourcename: "<name>", limit: 10 } (the auth_token key varies by auth type, as in Step 3). It returns up to limit real rows plus the inferred columns. Check that:

• data_selector is right — rows are the records you expect, not a wrapper object. If rows looks like

[{ "data": [...] }] you pointed at the envelope, not the array; fix data_selector.

• primary_key exists in the rows and is unique.
• the incremental cursor_path field is present in the rows and looks like a sortable timestamp/id.

A live failure (unreachable host, auth rejected) comes back as error with empty rows — fix credentials or the URL and retry. Iterate Steps 2–4 until the sample looks right.

Step 5 — Create the source

Call data-warehouse-source-setup with { sourcetype: "Custom", payload: { manifestjson, authtoken }, prefix: "<shortname>" } (the authtoken key varies by auth type, as in Step 3). It enables every resource in the manifest with sensible sync defaults (incremental where the manifest declares a cursor, else full refresh) and creates the source. If the user only wants a subset of resources, use external-data-sources-create with a schemas array instead (see setting-up-a-data-warehouse-source for the schemas shape). Pick a short lowercase prefix — tables become {prefix}{resource_name} in HogQL.

After creation, call external-data-schemas-list to show the user the initial sync status, and tell them how to query: SELECT * FROM {prefix}{resourcename} LIMIT 10.

Important notes

• Always preview before creating. db-schema proves the manifest parses and the credential works; preview proves the

field mappings (dataselector / primarykey / cursor_path) against real rows. Skipping preview is the most common way to create a source that syncs zero or malformed rows.

• **Secrets only in auth*.** Never inline a token/key/password in manifestjson — it's rejected, and the manifest

is non-secret (it round-trips to the client).

• One level of fan-out. A child resource may depend on a top-level parent; a parent can't itself be a child.
• GET and POST only. The engine reads upstream data; PUT/PATCH/DELETE are rejected so a manifest can't mutate the

source API.

• Pick the cursor carefully. Prefer an updatedat-style field over createdat (it catches edits), and set

cursor_type when the cursor isn't a datetime (e.g. an integer id) so it's compared with the right type.

• OAuth2 secrets are adopted into a server-managed credential store on the first db-schema / preview / create

call, and any rotated single-use refresh token is persisted server-side — so keep the entire client.auth block identical across those calls within one setup, and re-submit the same secrets each time. Changing any auth-block field mid-setup discards the stored rotation, and providers that rotate single-use refresh tokens will then reject the next mint until the user fetches a fresh token. Never set authoauth2integrationid yourself (it is server-owned); to reconnect a source whose token broke, update it with re-entered authoauth2clientsecret / authoauth2refresh_token. See the OAuth2 section of the manifest reference for the auth block fields.