Stackbone debug & diagnostics

This skill helps an AI coding agent locate the source of a failure in a Stackbone workspace — a deep agent or a durable workflow — it does not propose fixes. Once the cause is identified, switch back to the stackbone skill (for SDK / agent / workflow code changes) or the stackbone-cli skill (for CLI / db / publish actions) to implement the fix.

A workspace is a multi-piece project: a deep-agents/ folder (one agent per subfolder, each a deep-agents/<name>/index.ts default-exporting defineDeepAgent(...)) and a workflows/ folder (one durable workflow per workflows/<name>.workflow.ts, its name the file basename without .workflow.ts, exporting <camelCase(name)>Workflow). The registry is discovered by convention from those files; a stackbone.config.ts can only override the workflow list. Agents run in-process and are served over the standard OpenAI/Anthropic chat endpoints (the model field selects the agent); workflows are durable functions ('use workflow' / 'use step') triggered through the runtime's HTTP contract or the CLI. Every invocation — an agent turn or a workflow run — produces a durable run with a log you inspect after the fact.

Deep agents, durable workflows and the workspace runtime are exercised by stackbone dev today. On a deployed cloud install the workflows / runs / HITL endpoints may still 404 until the prod port lands — a 404 there is expected, not a bug.

How to use this skill

1. Capture the symptom verbatim from the user (error message, HTTP status, stuck run, unexpected output).
2. Match the symptom to a row in the tables below.
3. Run the diagnostic command as-is (these are read-only, safe).
4. Report what you found in plain narrative — exact command output, not paraphrase.
5. Hand off to stackbone or stackbone-cli for the fix.

Read-only by design. No command in this skill mutates state. If a diagnostic requires a mutation (e.g. retrying a failed run), it's flagged with ⚠️ destructive and the agent must surface the proposal to the user before executing.

SDK errors — { data, error } envelope

Most surface reads on the ambient stackbone client return a { data, error } envelope. The error is null on success; otherwise it carries { code, message, cause?, meta? }. code is a literal from the catalog (<prefix><reason>, e.g. airate_limited, plus a handful of standalone setup-bug codes); meta is the structured escape hatch (paths, retry hints, AWS metadata, constraint names); cause carries the upstream error object. There is no details, retryable, or nextActions field on an SdkError — read meta/cause instead.

Three surfaces do NOT use the envelope. stackbone.database is native Drizzle — await returns rows and throws on error (catch it, there's no .error). callDeepAgent(name, input) (from @stackbone/sdk/workflow) returns { text } and throws on failure — the throw fails the enclosing 'use step', so the error shows in the run's log frames. And stackbone.connection(id) (the Stackbone Connect broker) returns the operation output directly and throws a ConnectorCallError on failure — match err.code, never instanceof (see the connectors note in the Secrets / config section).

The full catalog of <prefix>_<reason> codes is the public error surface; pattern-match on result.error.code to branch. New codes arrive behind a minor — if you see a code not in this table, read error.message and treat it by its prefix family.

HTTP errors from the control plane

When you call the platform API directly (the control plane at STACKBONEAPIURL), or when stackbone <command> reports a backend error:

Durable runs failing or timing out

Every invocation — an agent turn or a workflow run — produces a durable run. A run records a status (running / done / failed / interrupted), a trigger, timing, and a step log. Each 'use step' inside a workflow is a persisted, replay-safe step: it runs once, is recorded, and is retried on failure — so a failed run can be inspected step-by-step and retried from its input.

Locate the failing run from the shell (the local-dev install by default, or a cloud install via --agent <id> — stackbone agents list gives the ids). See the stackbone-cli skill for full flags.

stackbone runs list --status failed --agent <id> --json   # find the run id
stackbone runs get <runId> --agent <id> --json            # header: status, trigger, timing
stackbone logs tail --run <runId> --agent <id> --json     # the run's log frames (the step waterfall lands here)

There is no stackbone runs steps verb. Inspect a run's progression through its log frames: stackbone logs tail --run <runId> carries the per-step records (each 'use step' boundary, its inputs, retries and the error that failed it). Without --follow the tail prints the buffered frames (up to --limit, default 100) and exits; add --follow to keep watching a still-running run.

Studio's Runs tab (production, or the dev Studio served alongside stackbone dev) shows the same timeline visually. Common causes:

HITL runs stuck

HITL pauses exist at two levels, resolved through the same approvals inbox.

Tool-level — an agent turn interrupted on a gated tool

A deep agent with interruptOn: { <tool>: true } pauses before the gated tool runs: the chat turn ends as a well-formed standard response (Anthropic stopreason: 'tooluse' / OpenAI finishreason: 'toolcalls'), the run flips to interrupted, and an approvals row appears (topic tool-approval for <tool> on <agent>). Deciding it (stackbone hitl approve|reject <id> --yes) resumes the turn server-side on the same session — the client never re-POSTs.

Workflow-level — requestApproval()

Human-in-the-loop in a workflow is the requestApproval() gate from @stackbone/sdk/workflow. It writes an approvals row (so the run shows in the inbox), then PAUSES the run durably on a Workflow SDK hook, racing the human decision against the timeout (applying the fallback — 'approve' / 'reject' — if nobody decides). The caller gates the side-effect on decision.status === 'approved'. The raw defineHook + sleep (also re-exported from @stackbone/sdk/workflow) are the escape hatch for custom gates.

import { requestApproval } from '@stackbone/sdk/workflow';

const decision = await requestApproval({
  token: `refund-${orderId}`, // the resume key
  topic: 'refund',
  payload: { orderId, amount },
  timeout: '24h',
  fallback: 'reject',
});
if (decision.status !== 'approved') return { refunded: false };

When the org member says "the run is paused but nothing is in the inbox":

1. The approvals row likely failed to write and the run never registered. Look for approvalpersistfailed in the run's frames (stackbone logs tail --run <runId> --agent <id> --json) — usually a missing/unmigrated approvals table or a DB error (error.cause has the Postgres error). requestApproval() writes the row in its own 'use step', so a failed write surfaces as a failed step, not a silent swallow.
2. Or the row was created but the run was never resumed. The pause is a hook keyed by the token; the human decides in Studio (or via the CLI), and the host resumes the parked run by that token through POST /api/workflows/hooks/:token/resume. List the inbox from the shell with stackbone hitl list --agent <id> --json and inspect one with stackbone hitl get <hitlId> --agent <id> --json to see its current status and audit trail of decisions. Decide it from the shell with stackbone hitl approve <hitlId> --agent <id> --yes / stackbone hitl reject <hitlId> --agent <id> --yes (both ⚠️ destructive, require --yes, accept an optional --reason). Studio's HITL tab shows the same.
3. Or the approver role isn't assigned to any org member, so no one can decide. stackbone metadata --json shows the org's roles.
4. Or the pause already timed out: when the timeout elapses before a human decides, the fallback is applied and the run continues with decision.timedOut === true. A run that "resumed on its own and rejected" is the fallback firing — check the timeout/fallback on the requestApproval() call.

The CLI verb group is hitl; the SDK surface that writes the row is stackbone.approval (driven for you by requestApproval()). You rarely call stackbone.approval directly — author the gate with requestApproval() and let it manage the row + hook.

Workflows — input schema, triggering, discovery

A workflow declares optional sibling inputSchema / outputSchema Zod exports. The runtime validates the start/chat body against inputSchema at the frontier, before the run starts — a mismatch is the 400 workflowinputinvalid above (with issues[]), and no run is created.

Database — slow queries, missing indexes, pgvector

The CLI has a read-only SQL explorer against the targeted install: stackbone db query "<single SELECT>" --agent <id> --json, stackbone db schemas and stackbone db table <schema> <table> (see the stackbone-cli skill). It is read-only — EXPLAIN ANALYZE and any write still need standard Postgres tooling (psql or any client) pointed at DATABASE_URL:

• Slow SELECT — EXPLAIN (ANALYZE, BUFFERS) <query> via psql (the CLI db query only runs plain reads); look for sequential scans on large tables.
• pgvector distance returns wrong order — confirm the index uses the same distance operator as your query (<-> cosine, <#> inner product, <=> L2). A query with <-> against an index built for <#> does a sequential scan and returns wrong results.
• tsvector queries return nothing — confirm the column was populated (UPDATE … SET searchtsv = totsvector(…) on existing rows; the trigger only fires on writes after creation).
• Migration applied but the table isn't there — stackbone db migrate status shows the journal. Inspect with stackbone db schemas --agent <id> --json (does the table exist?) or read the journal directly via psql.

stackbone.database is native Drizzle — it throws on a query error (no { data, error } envelope), so wrap DB code in try/catch and inspect the thrown Postgres error.

Storage — missing files, signed URLs failing

Scheduled workflows not firing

A recurring trigger is a dynamic schedule registered with stackbone.workflows.schedule(name, input, cron) on the ambient client (or a declarative export const schedules next to the workflow). Each tick starts the named workflow as its own run, so you diagnose a missing tick like any run.

Publish / build — stackbone publish failed

stackbone publish packages the whole workspace (every agent's compiled output + the workflows) into a verifiable bundle. The build runs on your host; the most common failures are build-time guardrails that abort BEFORE producing a bundle, so a broken agent never ships and crash-loops in production.

The upload to the registry + build-pointer registration is the provisioning step's job under BYOC (the CLI has no registry credentials), so publish writes the verifiable artifact + its digest locally and prints them. A failure there is a control-plane/provisioning concern, not a CLI one.

Secrets / config

Connectors are a live surface, but stackbone.connection(id) (the Stackbone Connect broker) throws a ConnectorCallError on failure — it does not return a { data, error } envelope. Match err.code (never instanceof) against the broker taxonomy: invalidargs (args failed the operation schema, or a bad operation id), credentialerror (token revoked/expired, or a non-2xx from the broker), timeout, ambiguous (several connections of that connector matched and none was selected), unauthorized (caller identity rejected), and unavailable / executefailed (broker or provider unreachable). For ambiguous, resolve the id or unique name with stackbone connectors --json (see the stackbone-cli skill) and pass it: stackbone.connection('<id or unique name>'). The legacy connections* envelope codes belong to the deprecated client.legacyConnections proxy (not shipped as a live surface), not to stackbone.connection(id). See the stackbone skill's connections doc for the full table.

Tier / billing — 402 responses

These are control-plane HTTP 402 bodies, not SdkErrors. They surface when you call the platform API directly (or when a stackbone <command> hits a tier cap) — they are not in the SdkErrorCode catalog and you cannot pattern-match them with result.error.code from a stackbone.* call. The strings below are fields on the API's HTTP response body, not SDK error codes.

These are not retryable — surface the body's message (and any hints it carries) to the user verbatim. The closest SDK-side signal is aicreditsexhausted (an actual catalog code) when a model call hits a spent credit bundle.

What this skill does NOT cover

• Performance tuning of the runtime (cold start, machine sizing, scale-to-zero pauses). That's a feature with its own metrics surface.
• Cost analysis beyond the 402 codes above. The Studio Costs panel covers per-agent breakdown.
• Custom OTel queries against external tracing backends. For now Studio's Logs tab is the supported path.