When the docs assistant gives a slow or wrong answer in production, the final reply tells you nothing about why. You need to see each model call and tool call, how long it took and how many tokens it used. ADK already instruments all of that with OpenTelemetry, the vendor-neutral standard for traces and metrics; it only needs exporters to send the data somewhere. The steps below register file exporters on ADK’s providers, run one question on Gemini gemini-3.8-flash, and read the result. This is the span tree and the metric totals of the recorded run:
When you need this. A docs assistant that sometimes takes ten seconds instead of three, or whose token bill doubled after a prompt change, needs traces before anyone can fix it. The same spans also show which page the agent read, which matters when it cites the wrong one.
$ python traced.py 'How do I rebuild the search index?'
answer: You can rebuild the search index by running `docs-assistant index --rebuild` (cli.md).
spans:
invocation 5659.9 ms
invoke_agent docs_assistant 5653.6 ms
call_llm 1957.8 ms model=gemini-3.8-flash input_tokens=156 output_tokens=33
generate_content gemini-3.8-flash 1955.5 ms model=gemini-3.8-flash input_tokens=156 output_tokens=33
execute_tool list_pages 0.9 ms name=list_pages
call_llm 1969.6 ms model=gemini-3.8-flash input_tokens=218 output_tokens=53
generate_content gemini-3.8-flash 1967.4 ms model=gemini-3.8-flash input_tokens=218 output_tokens=53
execute_tool read_page 1.1 ms name=read_page
call_llm 1708.1 ms model=gemini-3.8-flash input_tokens=349 output_tokens=75
generate_content gemini-3.8-flash 1707.4 ms model=gemini-3.8-flash input_tokens=349 output_tokens=75
metrics:
gen_ai.execute_tool.duration docs_assistant,list_pages count=1 sum=0.001 s
gen_ai.execute_tool.duration docs_assistant,read_page count=1 sum=0.001 s
gen_ai.client.operation.duration docs_assistant count=3 sum=5.63 s
gen_ai.client.token.usage docs_assistant,input count=3 sum=723 {token}
gen_ai.client.token.usage docs_assistant,output count=3 sum=161 {token}
gen_ai.invoke_agent.duration docs_assistant count=1 sum=5.654 s
gen_ai.invoke_agent.inference_calls docs_assistant count=1 sum=3
gen_ai.invoke_agent.tool_calls docs_assistant count=1 sum=2Lab setup. Download the complete lab ZIP, extract it and open a terminal in adk-otel-traces/. Keep the bundled code, tests and fixtures together. Create the pinned environment before running the commands below:
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt1. Start from the docs assistant
The agent is the docs assistant from the first course: two function tools over three Markdown pages in docs/. Nothing in it mentions telemetry, because ADK creates its spans and metrics itself.
"""The docs assistant: it answers questions about the docs-assistant project from its docs pages."""
import os
from pathlib import Path
from google.adk import Agent
from google.adk.models.lite_llm import LiteLlm
from google.genai import types
# Gemini through ADK's native client; set DOCS_MODEL=ollama_chat/gemma4:e2b to run Gemma locally.
MODEL = os.environ.get("DOCS_MODEL", "gemini-3.8-flash")
DOCS = Path(__file__).resolve().parent.parent / "docs"
def list_pages() -> dict:
"""List the docs pages by path."""
return {"pages": sorted(page.name for page in DOCS.glob("*.md"))}
def read_page(path: str) -> dict:
"""Return the Markdown of one docs page, by the path list_pages returned."""
page = DOCS / path
if page.parent != DOCS or not page.is_file():
return {"error": f"No page {path!r}. Call list_pages for valid paths."}
return {"path": path, "content": page.read_text(encoding="utf-8")}
root_agent = Agent(
name="docs_assistant",
model=LiteLlm(model=MODEL) if MODEL.startswith("ollama_chat/") else MODEL,
instruction=("Answer questions about the docs-assistant project from its docs only. "
"Call list_pages, then read_page on the one page that answers. Reply in "
"one sentence and end it with the page path in parentheses."),
tools=[list_pages, read_page],
generate_content_config=types.GenerateContentConfig(
temperature=0.2, thinking_config=types.ThinkingConfig(thinking_level="low")),
)2. Register exporters on ADK’s providers
OpenTelemetry splits the work in two. Instrumented code (here, ADK) asks a global tracer provider and meter provider for tracers and meters; exporters attached to those providers decide where the data goes. A configured provider and exporter are needed to retain the spans ADK opens. maybe_set_otel_providers from google.adk.telemetry.setup builds both providers from the OTelHooks you pass, adds OTLP exporters when OTEL_EXPORTER_OTLP_ENDPOINT is set, and registers them globally.
"""Run the docs assistant with OpenTelemetry traces and metrics written to local files.
Run from this folder: python traced.py "QUESTION" [OUT_DIR]
Writes OUT_DIR/traces.jsonl (one span per line) and OUT_DIR/metrics.jsonl, then prints
the span tree and the metric totals. With OTEL_EXPORTER_OTLP_ENDPOINT set, ADK also adds
OTLP exporters to the same providers, so a collector such as Jaeger gets the same spans.
"""
import asyncio
import warnings
import json
import sys
from pathlib import Path
warnings.filterwarnings("ignore")
from opentelemetry import metrics, trace # noqa: E402
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from google.adk.runners import InMemoryRunner
from google.adk.telemetry.setup import OTelHooks, maybe_set_otel_providers
from google.genai import types
from docs_assistant.agent import root_agent
def set_up_telemetry(out: Path) -> None:
"""Register file exporters for spans and metrics on new global providers."""
out.mkdir(parents=True, exist_ok=True)
spans = ConsoleSpanExporter(out=(out / "traces.jsonl").open("w"),
formatter=lambda span: span.to_json(indent=None) + "\n")
points = ConsoleMetricExporter(out=(out / "metrics.jsonl").open("w"),
formatter=lambda data: data.to_json(indent=None) + "\n")
maybe_set_otel_providers([OTelHooks(
span_processors=[SimpleSpanProcessor(spans)],
# Export once, at force_flush, instead of every 60 seconds.
metric_readers=[PeriodicExportingMetricReader(points, export_interval_millis=3_600_000)],
)])
async def ask(question: str) -> str:
runner = InMemoryRunner(agent=root_agent, app_name="docs_assistant")
session = await runner.session_service.create_session(app_name="docs_assistant", user_id="dev")
message = types.Content(role="user", parts=[types.Part(text=question)])
answer = ""
async for event in runner.run_async(user_id="dev", session_id=session.id, new_message=message):
if event.is_final_response() and event.content:
answer = "".join(part.text or "" for part in event.content.parts if not part.thought)
return answer
def span_tree(out: Path) -> list[str]:
"""One line per span, indented under its parent, with duration and the attributes that matter."""
spans = [json.loads(line) for line in (out / "traces.jsonl").read_text().splitlines()]
children: dict = {}
for span in spans:
children.setdefault(span["parent_id"], []).append(span)
keep = ("gen_ai.request.model", "gen_ai.tool.name", "gen_ai.usage.input_tokens",
"gen_ai.usage.output_tokens", "error.type")
lines = []
def walk(parent, depth):
for span in sorted(children.get(parent, []), key=lambda s: s["start_time"]):
ms = (_ns(span["end_time"]) - _ns(span["start_time"])) / 1e6
attrs = " ".join(f"{k.split('.')[-1]}={v}" for k, v in span["attributes"].items() if k in keep)
lines.append(f"{' ' * depth}{span['name']:<{40 - 2 * depth}} {ms:8.1f} ms {attrs}".rstrip())
walk(span["context"]["span_id"], depth + 1)
walk(None, 0)
return lines
def _ns(iso: str) -> int:
from datetime import datetime
return int(datetime.fromisoformat(iso.replace("Z", "+00:00")).timestamp() * 1e9)
def metric_totals(out: Path) -> list[str]:
lines = []
for line in (out / "metrics.jsonl").read_text().splitlines():
for resource in json.loads(line)["resource_metrics"]:
for scope in resource["scope_metrics"]:
for metric in scope["metrics"]:
for point in metric["data"]["data_points"]:
labels = ",".join(f"{v}" for k, v in point["attributes"].items()
if k in ("gen_ai.agent.name", "gen_ai.tool.name", "gen_ai.token.type"))
unit = "" if metric["unit"] == "1" else f" {metric['unit']}" # "1" means a plain count
lines.append(f"{metric['name']:<42} {labels:<24} count={point.get('count', '-')} "
f"sum={round(point.get('sum', point.get('value', 0)), 3)}{unit}")
return lines
if __name__ == "__main__":
out = Path(sys.argv[2] if len(sys.argv) > 2 else "telemetry")
set_up_telemetry(out)
print("answer:", asyncio.run(ask(sys.argv[1])))
trace.get_tracer_provider().force_flush()
metrics.get_meter_provider().force_flush()
print("\n".join(["", "spans:", *span_tree(out), "", "metrics:", *metric_totals(out)]))set_up_telemetry points the SDK’s own ConsoleSpanExporter and ConsoleMetricExporter at files instead of standard output, one JSON object per line. A file exporter is enough to read a trace on one machine, and it needs no extra package. The recording uses file exporters; the Note below describes the untested switch to a collector.
Two choices in that function matter later:
SimpleSpanProcessorexports each span the moment it ends, on the calling thread. That is right for a script whose file you read straight after. A server should useBatchSpanProcessor, which ADK uses for its own OTLP exporter: it queues spans and exports them from a background thread, so a slow collector doesn’t add latency to every model call.- The metric reader would normally export every 60 seconds. A one-question script ends long before that, so the reader is set to an hour and
force_flush()exports once at the end. Without the flush,metrics.jsonlstays empty.
Prompts and tool results in spans. With no settings, call_llm spans carry the full request and response, and execute_tool spans carry the page text. Anything a user pastes reaches your trace store. Set ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false everywhere except a debugging session.
No service name. The resource has no service.name, so a backend that groups by service name cannot assign these spans to this agent’s service. Set OTEL_SERVICE_NAME in the deployment, not in code that might run twice.
OTLP endpoint without the exporter package. maybe_set_otel_providers imports the OTLP/HTTP exporter and the process stops with ModuleNotFoundError before the agent runs. Install opentelemetry-exporter-otlp-proto-http with the agent, or unset the endpoint.
3. Run one question and read the trace
Run traced.py directly so the local file exporters are attached before execution (adk run and adk web do not register traced.py’s custom file exporters):
python traced.py "How do I rebuild the search index?" telemetryHere is what happens in the recorded run. The invocation span covers the whole run_async call, and invoke_agent docs_assistant covers the agent’s part of it. Under the agent, each model round trip is a call_llm span with the GenAI SDK’s generate_content span inside, and each tool call is an execute_tool span. The model asked for list_pages, then read_page, then answered, so the tree reads model, tool, model, tool, model. Almost all of the time is in the three model calls; the two tools took about a millisecond each.
The metrics answer different questions from the same run. gen_ai.client.operation.duration and gen_ai.client.token.usage are histograms per model call; across many runs they give you latency percentiles and token spend per agent. gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls count calls per invocation, which is the first number to check when a prompt change makes the agent loop. Spans tell you what one run did; metrics tell you how a thousand runs behaved, without inspecting each trace individually. These recordings cover one question per configuration, not a service workload.
Output tokens look too high. The span’s output tokens include thinking tokens, so they don’t match the length of the reply. Read usage_metadata.thoughts_token_count from the response, or lower the thinking level.
Two more limits come from OpenTelemetry itself. A provider can be set once per process: if your web framework’s instrumentation sets a tracer provider first, maybe_set_otel_providers leaves it in place and logs a warning, and the hooks supplied by this call are not attached to the existing provider, so pass the hooks to whichever code sets the provider. And keep metric attributes to the ones ADK sets, the agent and tool names; adding a user or session ID to a metric creates one time series per user.
4. Record three telemetry configurations
record.py runs traced.py three times in fresh processes, because OpenTelemetry lets a process set its providers only once. Each run changes only environment variables.
python record.py{
"defaults": {
"service_name": null,
"spans": 10,
"content_attributes": [
"call_llm: gcp.vertex.agent.llm_request",
"call_llm: gcp.vertex.agent.llm_response",
"execute_tool read_page: gcp.vertex.agent.tool_response"
]
},
"production": {
"service_name": "docs-assistant",
"spans": 10,
"content_attributes": []
},
"otlp-without-exporter": {
"error": "ModuleNotFoundError: No module named 'opentelemetry.exporter'"
}
}With no OTEL_* variables, the spans carried the user’s question, the full model request and response, and the text of the page the tool returned. They also had no service.name. The local run labeled production set two variables; the label names an exporter configuration, not a deployed workload: the recorded resource gained a service name and the inspected content attributes were absent. This is not a general secret scrubber: tool arguments, custom attributes and other instrumentation need their own review. The third run set a collector endpoint and failed before the agent ran:
# Run otlp-without-exporter: OTEL_SERVICE_NAME=docs-assistant OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
$ python traced.py 'How do I rebuild the search index?'
ModuleNotFoundError: No module named 'opentelemetry.exporter'How spans and metrics leave the Runner
ADK writes two sets of attributes on its spans. The gen_ai.* ones (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.tool.name) follow the OpenTelemetry GenAI semantic conventions, so any backend that knows those conventions can chart them. ADK also writes its own gcp.vertex.agent.* attributes, and three of them carry content: llm_request (the system instruction, the conversation and the tool declarations as JSON), llm_response and, on execute_tool spans, tool_response. In google-adk 2.12 these are controlled by ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS, which defaults to on. The adk.dev logging page says prompt content is left out of logs by default, and that is true; it does not describe these span attributes, so the package source is the reference here. adk deploy agent_engine --otel_to_cloud sets the variable to false for you unless you set it, and nothing does that for a service you run yourself.
gen_ai.usage.output_tokens on a span is the model’s candidate tokens plus its thinking tokens. In the recorded run, each of the three model calls spent more tokens thinking than writing its reply, so the span totals are more than twice the reply sizes. A test checks the sum against usage_metadata for every model call. When you budget output tokens, budget for thinking too.
The resource, the attributes shared by every span and metric point from a process, comes from OTELResourceDetector, which reads only OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES. ADK does not fill in a default, so without the variable the recorded spans had no service.name at all. A backend that groups by service then shows them under a placeholder, mixed with every other unnamed process.
Per-run telemetry settings can also request content capture. Review those settings along with the process variables before exporting traces to a shared backend.
Test the traces without a model
test_traced.py takes the model responses stored in the recorded call_llm spans and replays them through the real agent, tools and ADK tracing into an in-memory exporter. It checks the span tree, the content attributes with capture on and off, the resource and the exporter import.
python -m unittest -v test_traced.pytest_an_otlp_endpoint_without_the_exporter_package_fails_at_startup (test_traced.Traces.test_an_otlp_endpoint_without_the_exporter_package_fails_at_startup) ... ok
test_by_default_spans_carry_the_prompt_and_tool_results (test_traced.Traces.test_by_default_spans_carry_the_prompt_and_tool_results) ... ok
test_output_tokens_in_spans_include_thinking_tokens (test_traced.Traces.test_output_tokens_in_spans_include_thinking_tokens) ... ok
test_the_recorded_trace_nests_model_and_tool_spans_under_the_agent (test_traced.Traces.test_the_recorded_trace_nests_model_and_tool_spans_under_the_agent) ... ok
test_turning_capture_off_removes_message_content (test_traced.Traces.test_turning_capture_off_removes_message_content) ... ok
test_without_otel_service_name_the_resource_has_no_service_name (test_traced.Traces.test_without_otel_service_name_the_resource_has_no_service_name) ... ok
----------------------------------------------------------------------
Ran 6 tests in 0.259s
OKAll 6 tests pass.
Read the test and recording scripts
"""Check the recorded traces, and replay the recorded model turns through a traced run offline.
Run: python -m unittest -v test_traced.py
The replay uses the model responses stored in the recorded spans; the agent, tools and ADK's
tracing are real, and spans go to an in-memory exporter.
"""
import asyncio
import json
import logging
import os
import re
import unittest
import warnings
from pathlib import Path
warnings.filterwarnings("ignore")
logging.disable(logging.CRITICAL)
os.environ.pop("OTEL_SDK_DISABLED", None) # the site's test runner disables the SDK by default
from opentelemetry import trace # noqa: E402
from opentelemetry.sdk.trace import TracerProvider # noqa: E402
from opentelemetry.sdk.trace.export import SimpleSpanProcessor # noqa: E402
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter # noqa: E402
from google.adk.models.base_llm import BaseLlm # noqa: E402
from google.adk.models.llm_response import LlmResponse # noqa: E402
from google.adk.telemetry import setup # noqa: E402
from google.genai import types # noqa: E402
import traced # noqa: E402
HERE = Path(__file__).resolve().parent
RECORD = json.loads((HERE / "recorded-runs.json").read_text())
SPANS = [json.loads(line) for line in (HERE / "runs/defaults/traces.jsonl").read_text().splitlines()]
TURNS = [json.loads(s["attributes"]["gcp.vertex.agent.llm_response"])["content"]
for s in sorted(SPANS, key=lambda s: s["start_time"]) if s["name"] == "call_llm"]
EXPORTER = InMemorySpanExporter()
PROVIDER = TracerProvider()
PROVIDER.add_span_processor(SimpleSpanProcessor(EXPORTER))
trace.set_tracer_provider(PROVIDER)
class ReplayLlm(BaseLlm):
turns: list
async def generate_content_async(self, llm_request, stream=False):
yield LlmResponse(content=types.Content.model_validate(self.turns.pop(0)))
def replay(capture: str) -> list:
EXPORTER.clear()
os.environ["ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS"] = capture
model = ReplayLlm(model="replay", turns=[dict(t) for t in TURNS])
original = traced.root_agent
traced.root_agent = original.clone(update={"model": model})
try:
answer = asyncio.run(traced.ask(RECORD["question"]))
finally:
traced.root_agent = original
return answer, EXPORTER.get_finished_spans()
def content_attributes(spans) -> list:
return sorted({f"{s.name}: {k}" for s in spans for k, v in s.attributes.items() if "search index" in str(v)})
def setUpModule():
warnings.simplefilter("ignore") # unittest resets the filter; ADK warns about experimental features
class Traces(unittest.TestCase):
def test_the_recorded_trace_nests_model_and_tool_spans_under_the_agent(self):
lines = traced.span_tree(HERE / "runs/defaults")
self.assertTrue(lines[0].startswith("invocation"))
self.assertTrue(lines[1].startswith(" invoke_agent docs_assistant"))
nested = [re.split(r"\s{2,}", line.strip())[0] for line in lines[2:] if line[4] != " "]
self.assertEqual(nested, ["call_llm", "execute_tool list_pages", "call_llm",
"execute_tool read_page", "call_llm"])
def test_output_tokens_in_spans_include_thinking_tokens(self):
for span in SPANS:
if span["name"] == "call_llm":
usage = json.loads(span["attributes"]["gcp.vertex.agent.llm_response"])["usage_metadata"]
self.assertEqual(span["attributes"]["gen_ai.usage.output_tokens"],
usage["candidates_token_count"] + usage.get("thoughts_token_count", 0))
self.assertGreater(usage["thoughts_token_count"], usage["candidates_token_count"])
def test_by_default_spans_carry_the_prompt_and_tool_results(self):
answer, spans = replay("true")
self.assertIn("(cli.md)", answer)
self.assertEqual(content_attributes(spans), RECORD["runs"]["defaults"]["content_attributes"])
def test_turning_capture_off_removes_message_content(self):
answer, spans = replay("false")
self.assertIn("(cli.md)", answer)
self.assertEqual(content_attributes(spans), [])
self.assertEqual(RECORD["runs"]["production"]["content_attributes"], [])
def test_without_otel_service_name_the_resource_has_no_service_name(self):
self.assertIsNone(RECORD["runs"]["defaults"]["service_name"])
os.environ.pop("OTEL_SERVICE_NAME", None)
self.assertNotIn("service.name", setup._get_otel_resource().attributes)
os.environ["OTEL_SERVICE_NAME"] = "docs-assistant"
self.assertEqual(setup._get_otel_resource().attributes["service.name"], "docs-assistant")
def test_an_otlp_endpoint_without_the_exporter_package_fails_at_startup(self):
self.assertIn("ModuleNotFoundError", RECORD["runs"]["otlp-without-exporter"]["error"])
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:4318"
try:
with self.assertRaises(ModuleNotFoundError):
setup._get_otel_exporters()
finally:
del os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"]
if __name__ == "__main__":
unittest.main()"""Record traced runs of the docs assistant under three telemetry configurations.
Run from this folder with GOOGLE_API_KEY set: python record.py
Each run is a fresh process, because OpenTelemetry providers can be set only once per process.
Writes runs/<name>/traces.jsonl and metrics.jsonl, recorded-runs.json and run-output.txt.
"""
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
QUESTION = "How do I rebuild the search index?"
RUNS = {
"defaults": {},
"production": {"OTEL_SERVICE_NAME": "docs-assistant", "ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS": "false"},
"otlp-without-exporter": {"OTEL_SERVICE_NAME": "docs-assistant", "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318"},
}
def checks(out: Path) -> dict:
"""What a trace backend would receive: span names, the service name and any message content."""
spans = [json.loads(line) for line in (out / "traces.jsonl").read_text().splitlines()]
content = sorted({f"{s['name']}: {k}" for s in spans for k, v in s["attributes"].items()
if "search index" in str(v) or "docs-assistant index --rebuild" in str(v)})
return {"spans": len(spans), "span_names": sorted({s["name"] for s in spans}),
"service_name": spans[0]["resource"]["attributes"].get("service.name"),
"content_attributes": content}
def main() -> None:
record, output = {"question": QUESTION, "runs": {}}, []
for name, extra in RUNS.items():
out = HERE / "runs" / name
env = {**os.environ, "PYTHONWARNINGS": "ignore", **extra}
result = subprocess.run([sys.executable, "traced.py", QUESTION, str(out)], cwd=HERE, env=env,
capture_output=True, text=True, timeout=300)
settings = " ".join(f"{k}={v}" for k, v in extra.items()) or "(no OTEL_* variables)"
text = result.stdout + (result.stderr.strip().splitlines()[-1] + "\n" if result.returncode else "")
output.append(f"# Run {name}: {settings}\n$ python traced.py {QUESTION!r}\n{text}")
entry = {"env": extra, "exit_code": result.returncode}
if result.returncode == 0:
entry.update(checks(out))
else:
entry["error"] = result.stderr.strip().splitlines()[-1]
shutil.rmtree(out) # the exporter files it opened stay empty
record["runs"][name] = entry
(HERE / "recorded-runs.json").write_text(json.dumps(record, indent=2) + "\n")
(HERE / "run-output.txt").write_text("\n".join(output))
print("\n".join(output))
if __name__ == "__main__":
main()A teammate exports traces from the docs assistant to a shared Jaeger instance and sets only OTEL_EXPORTER_OTLP_ENDPOINT, with the exporter package installed. A developer pastes a config file with a database password into a question.
Read the explanation
Span content capture is a separate switch from log content capture, and in google-adk 2.12 it is on by default.
- Test an agent against simulated users: drive multi-turn conversations with a user simulator and score them with
adk eval. - Guard model and tool calls with callbacks and plugins: redact secrets before they reach the session, the model or a span.
References
Official sources checked October 21, 2026, against google-adk 2.12.0.