eliasecchig/adk-skills · Archived

adk-observability-guide

MUST READ before setting up observability for ADK agents or when analyzing production traffic, debugging agent behavior, or improving agent performance. ADK observability guide — Cloud Trace, prompt-response logging, BigQuery Agent Analytics, third-party integrations, and troubleshooting. Use when configuring monitoring, tracing, or logging for agents, or when understanding how a deployed agent handles real traffic.

Installation

$ npx skills add eliasecchig/adk-skills --skill adk-observability-guide

Stronger alternatives

This repository is archived — consider an actively maintained alternative.

Similar popular skills

Related neighbors and high-traction skills in the same topics — useful to compare before installing.

Also in this package

Other skills from eliasecchig/adk-skills.

npx skills add eliasecchig/adk-skills

Browse all from eliasecchig/adk-skills

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Also listed on

Alternate registries and mirrors of this skill.

Repository health

Stars 1
Default branch main
Open issues 0
Status Archived

Skill metadata

Parsed from SKILL.md frontmatter.

Version0.1.0
More metadata
author
Google
version
0.1.0
managed-by
adk-skills-setup

Package contents

Files included with this skill beyond the listing page.

  • skill md SKILL.md 9,438 B
  • docs SUMMARY.md 452 B

History

  1. First recorded snapshot · 5 installs

SKILL.md

ADK Observability Guide

Scaffolded project? Cloud Trace and prompt-response logging are pre-configured by Terraform. See references/cloud-trace-and-logging.md for infrastructure details, env vars, and verification commands.

No scaffold? Follow the ADK docs links below for manual setup. For production infrastructure, scaffold with /adk-scaffold.

Reference Files

File Contents
references/cloud-trace-and-logging.md Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally
references/third-party.md Third-party integration setup patterns, trade-offs, and ADK docs links for each provider

Observability Tiers

Choose the right level of observability based on your needs:

Tier What It Does Scope Default State Best For
Cloud Trace Distributed tracing — execution flow, latency, errors via OpenTelemetry spans All templates, all environments Always enabled Debugging latency, understanding agent execution flow
Prompt-Response Logging GenAI interactions exported to GCS, BigQuery, and Cloud Logging ADK agents only Disabled locally, enabled when deployed Auditing LLM interactions, compliance
BigQuery Agent Analytics Structured agent events (LLM calls, tool use, outcomes) to BigQuery ADK agents with plugin enabled Opt-in (--bq-analytics at scaffold time) Conversational analytics, custom dashboards, LLM-as-judge evals
Third-Party Integrations External observability platforms (AgentOps, Phoenix, MLflow, etc.) Any ADK agent Opt-in, per-provider setup Team collaboration, specialized visualization, prompt management

Ask the user which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive.


Cloud Trace

ADK uses OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow.

Span Hierarchy

invocation
  └── agent_run (one per agent in the chain)
        ├── call_llm (model request/response)
        └── execute_tool (tool execution)

Setup by Deployment Type

Deployment Setup
Agent Engine Automatic — traces are exported to Cloud Trace by default
Cloud Run (scaffolded) Automatic — oteltocloud=True in the FastAPI app
Cloud Run (manual) Configure OpenTelemetry exporter in your app
Local dev Works with make playground; traces visible in Cloud Console

View traces: Cloud Console → Trace → Trace explorer

For detailed setup instructions (Agent Engine CLI/SDK, Cloud Run, custom deployments), fetch the ADK docs:


Prompt-Response Logging

Captures GenAI interactions (model name, tokens, timing) and exports to GCS (JSONL), BigQuery (external tables), and Cloud Logging (dedicated bucket).

Privacy Modes

Prompt-response logging is privacy-preserving by default — only metadata is logged. Controlled by OTELINSTRUMENTATIONGENAICAPTUREMESSAGE_CONTENT:

Value Behavior
false Logging disabled
NO_CONTENT Enabled, metadata only — tokens, model name, timing (default in deployed environments)
true Enabled with full prompt/response content (not recommended for production)

For Agent Engine: the platform requires true during deployment, but the app overrides to NO_CONTENT at runtime.

Behavior by Environment

Environment Prompt-Response Logging Why
Local dev (make playground) Disabled No LOGSBUCKETNAME set
Dev (Terraform deployed) Enabled (NO_CONTENT) Terraform sets env vars
Staging / Production Enabled (NO_CONTENT) Terraform sets env vars

To enable locally, set LOGSBUCKETNAME and OTELINSTRUMENTATIONGENAICAPTUREMESSAGECONTENT=NOCONTENT before running make playground.

To disable in a deployed environment, set OTELINSTRUMENTATIONGENAICAPTUREMESSAGE_CONTENT=false in deployment/terraform/service.tf and re-apply.

For scaffolded project infrastructure details (Terraform resources, env vars, verification), see references/cloud-trace-and-logging.md.

For ADK logging docs (log levels, configuration, debugging):


BigQuery Agent Analytics Plugin

An optional plugin that logs structured agent events directly to BigQuery via the Storage Write API. Enables:

  • Conversational analytics — session flows, user interaction patterns
  • LLM-as-judge evals — structured data for evaluation pipelines
  • Custom dashboards — Looker Studio integration
  • Tool provenance tracking — LOCAL, MCP, SUBAGENT, A2A, TRANSFERAGENT

Enabling

Method How
At scaffold time uvx agent-starter-pack create . --bq-analytics
Post-scaffold Add the plugin manually to app/agent.py (see ADK docs)

Infrastructure (BigQuery dataset, GCS offloading) is provisioned automatically by Terraform when enabled at scaffold time.

Key Features

  • Auto-schema upgrade (new fields added without migration)
  • GCS offloading for multimodal content (images, audio)
  • Distributed tracing via OpenTelemetry span context
  • SQL-queryable event log for all agent interactions

For full schema, SQL query examples, and Looker Studio setup:


Third-Party Integrations

ADK supports six third-party observability platforms. Each uses OpenTelemetry or custom instrumentation to capture agent behavior.

Platform Key Differentiator Setup Complexity Self-Hosted Option
AgentOps Session replays, 2-line setup, replaces native telemetry Minimal No (SaaS)
Phoenix Open-source, custom evaluators, experiment testing Low Yes
MLflow OTel traces to MLflow Tracking Server, span tree visualization Medium (needs SQL backend) Yes
Monocle 1-call setup, VS Code Gantt chart visualizer Minimal Yes (local files)
Weave W&B platform, team collaboration, timeline views Low No (SaaS)
Freeplay Prompt management + evals + observability in one platform Low No (SaaS)

Ask the user which platform they prefer — present the trade-offs and let them choose. For setup details on each, see references/third-party.md.


Troubleshooting

Issue Solution
No traces in Cloud Trace Verify oteltocloud=True in FastAPI app; check service account has cloudtrace.agent role
Prompt-response data not appearing Check LOGSBUCKETNAME is set; verify SA has storage.objectCreator on the bucket; check app logs for telemetry setup warnings
Privacy mode misconfigured Check OTELINSTRUMENTATIONGENAICAPTUREMESSAGECONTENT value — use NOCONTENT for metadata-only, false to disable
BigQuery Analytics not logging Verify plugin is configured in app/agent.py; check BQANALYTICSDATASET_ID env var is set
Third-party integration not capturing spans Check provider-specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry
Traces missing tool spans Tool execution spans appear under execute_tool — check trace explorer filters
High telemetry costs Switch to NO_CONTENT mode; reduce BigQuery retention; disable unused tiers

Deep Dive: ADK Docs (WebFetch URLs)

For detailed documentation beyond what this skill covers, fetch these pages:

Topic URL
Observability overview https://google.github.io/adk-docs/observability/index.md
Agent activity logging https://google.github.io/adk-docs/observability/logging/index.md
Cloud Trace integration https://google.github.io/adk-docs/integrations/cloud-trace/index.md
BigQuery Agent Analytics https://google.github.io/adk-docs/integrations/bigquery-agent-analytics/index.md
AgentOps https://google.github.io/adk-docs/integrations/agentops/index.md
Phoenix (Arize) https://google.github.io/adk-docs/integrations/phoenix/index.md
MLflow tracing https://google.github.io/adk-docs/integrations/mlflow/index.md
Monocle https://google.github.io/adk-docs/integrations/monocle/index.md
W&B Weave https://google.github.io/adk-docs/integrations/weave/index.md
Freeplay https://google.github.io/adk-docs/integrations/freeplay/index.md