Adk Observability Guide

1---2name: adk-observability-guide3description: >4  MUST READ before setting up observability for ADK agents or when5  analyzing production traffic, debugging agent behavior, or improving6  agent performance.7  ADK observability guide — Cloud Trace, prompt-response logging,8  BigQuery Agent Analytics, third-party integrations, and troubleshooting.9  Use when configuring monitoring, tracing, or logging for agents,10  or when understanding how a deployed agent handles real traffic.11metadata:12  license: Apache-2.013  author: Google14---15 16# ADK Observability Guide17 18> **Scaffolded project?** Cloud Trace and prompt-response logging are pre-configured by Terraform. See `references/cloud-trace-and-logging.md` for infrastructure details, env vars, and verification commands.19>20> **No scaffold?** Follow the ADK docs links below for manual setup. For production infrastructure, scaffold with `/adk-scaffold`.21 22### Reference Files23 24| File | Contents |25|------|----------|26| `references/cloud-trace-and-logging.md` | Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally |27| `references/bigquery-agent-analytics.md` | BQ Agent Analytics plugin — enabling, key features, GCS offloading, tool provenance |28 29---30 31## Observability Tiers32 33Choose the right level of observability based on your needs:34 35| Tier | What It Does | Scope | Default State | Best For |36|------|-------------|-------|---------------|----------|37| **Cloud Trace** | Distributed tracing — execution flow, latency, errors via OpenTelemetry spans | All templates, all environments | Always enabled | Debugging latency, understanding agent execution flow |38| **Prompt-Response Logging** | GenAI interactions exported to GCS, BigQuery, and Cloud Logging | ADK agents only | Disabled locally, enabled when deployed | Auditing LLM interactions, compliance |39| **BigQuery Agent Analytics** | Structured agent events (LLM calls, tool use, outcomes) to BigQuery | ADK agents with plugin enabled | Opt-in (`--bq-analytics` at scaffold time) | Conversational analytics, custom dashboards, LLM-as-judge evals |40| **Third-Party Integrations** | External observability platforms (AgentOps, Phoenix, MLflow, etc.) | Any ADK agent | Opt-in, per-provider setup | Team collaboration, specialized visualization, prompt management |41 42**Ask the user** which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive.43 44---45 46## Cloud Trace47 48ADK uses OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow.49 50### Span Hierarchy51 52```53invocation54  └── agent_run (one per agent in the chain)55        ├── call_llm (model request/response)56        └── execute_tool (tool execution)57```58 59### Setup by Deployment Type60 61| Deployment | Setup |62|-----------|-------|63| **Agent Engine** | Automatic — traces are exported to Cloud Trace by default |64| **Cloud Run (scaffolded)** | Automatic — `otel_to_cloud=True` in the FastAPI app |65| **GKE (scaffolded)** | Automatic — `otel_to_cloud=True` in the FastAPI app |66| **Cloud Run / GKE (manual)** | Configure OpenTelemetry exporter in your app |67| **Local dev** | Works with `make playground`; traces visible in Cloud Console |68 69View traces: **Cloud Console → Trace → Trace explorer**70 71For detailed setup instructions (Agent Engine CLI/SDK, Cloud Run, custom deployments), fetch `https://adk.dev/integrations/cloud-trace/index.md`.72 73---74 75## Prompt-Response Logging76 77Captures GenAI interactions (model name, tokens, timing) and exports to GCS (JSONL), BigQuery (external tables), and Cloud Logging (dedicated bucket). Privacy-preserving by default — only metadata is logged unless explicitly configured otherwise.78 79Key env var: `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` — set to `NO_CONTENT` (metadata only, default in deployed envs), `true` (full content), or `false` (disabled). Logging is disabled locally unless `LOGS_BUCKET_NAME` is set.80 81For scaffolded project details (Terraform resources, env vars, privacy modes, enabling/disabling, verification commands), see `references/cloud-trace-and-logging.md`.82 83For ADK logging docs (log levels, configuration, debugging), fetch `https://adk.dev/observability/logging/index.md`.84 85---86 87## BigQuery Agent Analytics Plugin88 89Optional plugin that logs structured agent events to BigQuery. Enable with `--bq-analytics` at scaffold time. See `references/bigquery-agent-analytics.md` for details.90 91---92 93## Third-Party Integrations94 95ADK supports several third-party observability platforms. Each uses OpenTelemetry or custom instrumentation to capture agent behavior.96 97| Platform | Key Differentiator | Setup Complexity | Self-Hosted Option |98|----------|-------------------|-----------------|-------------------|99| **AgentOps** | Session replays, 2-line setup, replaces native telemetry | Minimal | No (SaaS) |100| **Arize AX** | Commercial platform, production monitoring, evaluation dashboards | Low | No (SaaS) |101| **Phoenix** | Open-source, custom evaluators, experiment testing | Low | Yes |102| **MLflow** | OTel traces to MLflow Tracking Server, span tree visualization | Medium (needs SQL backend) | Yes |103| **Monocle** | 1-call setup, VS Code Gantt chart visualizer | Minimal | Yes (local files) |104| **Weave** | W&B platform, team collaboration, timeline views | Low | No (SaaS) |105| **Freeplay** | Prompt management + evals + observability in one platform | Low | No (SaaS) |106 107**Ask the user** which platform they prefer — present the trade-offs and let them choose. For setup details, fetch the relevant ADK docs page from the Deep Dive table below.108 109---110 111## Troubleshooting112 113| Issue | Solution |114|-------|----------|115| No traces in Cloud Trace | Verify `otel_to_cloud=True` in FastAPI app; check service account has `cloudtrace.agent` role |116| Prompt-response data not appearing | Check `LOGS_BUCKET_NAME` is set; verify SA has `storage.objectCreator` on the bucket; check app logs for telemetry setup warnings |117| Privacy mode misconfigured | Check `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` value — use `NO_CONTENT` for metadata-only, `false` to disable |118| BigQuery Analytics not logging | Verify plugin is configured in `app/agent.py`; check `BQ_ANALYTICS_DATASET_ID` env var is set |119| Third-party integration not capturing spans | Check provider-specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry |120| Traces missing tool spans | Tool execution spans appear under `execute_tool` — check trace explorer filters |121| High telemetry costs | Switch to `NO_CONTENT` mode; reduce BigQuery retention; disable unused tiers |122 123---124 125## Deep Dive: ADK Docs (WebFetch URLs)126 127For detailed documentation beyond what this skill covers, fetch these pages:128 129| Topic | URL |130|-------|-----|131| Observability overview | `https://adk.dev/observability/index.md` |132| Agent activity logging | `https://adk.dev/observability/logging/index.md` |133| Cloud Trace integration | `https://adk.dev/integrations/cloud-trace/index.md` |134| BigQuery Agent Analytics | `https://adk.dev/integrations/bigquery-agent-analytics/index.md` |135| AgentOps | `https://adk.dev/integrations/agentops/index.md` |136| Arize AX | `https://adk.dev/integrations/arize-ax/index.md` |137| Phoenix (Arize) | `https://adk.dev/integrations/phoenix/index.md` |138| MLflow tracing | `https://adk.dev/integrations/mlflow/index.md` |139| Monocle | `https://adk.dev/integrations/monocle/index.md` |140| W&B Weave | `https://adk.dev/integrations/weave/index.md` |141| Freeplay | `https://adk.dev/integrations/freeplay/index.md` |