{ "model": "gemini-3.8-flash", "runs": { "pin": { "question": "Is the google-adk pin on the dependencies page current?", "transcript": "call read_page {'page': 'dependencies'}\nresult read_page ['page', 'text']\nserver GOOGLE_SEARCH_WEB {'queries': ['site pypi org project google adk', 'google-adk pypi', 'google-adk python package latest release', 'google adk pypi']}\nsearch ['google-adk python package latest release']\nanswer **No**, the `google-adk` pin on the dependencies page is not current.\n\n* **Documentation (`dependencies` page):** Currently pinned to `2.10.0`.\n* **Latest Release (PyPI / GitHub `google/adk-python`):** The latest release is `2.12.0` (released October 13, 2026).\nsource github.com\nsource pypistats.org", "sources": [ { "title": "github.com", "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEygXpZyjV_RBSvCoxnQkKN9EzdnGbjPuQkuvn2sFoxD1ijW80Vw67rxjhnrvzCFuuVoCP-HjJ1YWF_YmT6p3myoDumXB6rRxBe-w1PT4JRlV7VLRJjxfkZlLIyX9OLXv8WLSQ=" }, { "title": "pypistats.org", "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFSxQ-52WZnRKb462D46YvhxHW67c_Y3LwtVHXqm4cJ512EZJ1lxr0kgDd7fckh5uaI4tpcQzdlJ0vOmq2mCsTg_PUH31F3qraGy4dNNmzVpjZIaOlweBgqJ9z62_N7fA==" } ], "citations": [ { "text": "**No**, the `google-adk` pin on the dependencies page is not current", "sources": [ "github.com", "pypistats.org" ] }, { "text": "**Latest Release (PyPI / GitHub `google/adk-python`):** The latest release is `2.12.0` (released October 13, 2026)", "sources": [ "github.com", "pypistats.org" ] } ], "requests": [ { "tools": [ [ "code_execution" ], [ "function_declarations" ], [ "google_search" ] ], "tool_config": { "include_server_side_tool_invocations": true } }, { "tools": [ [ "code_execution" ], [ "function_declarations" ], [ "google_search" ] ], "tool_config": { "include_server_side_tool_invocations": true } } ], "responses": [ { "content": { "parts": [ { "function_call": { "id": "call_609228", "args": { "page": "dependencies" }, "name": "read_page" } } ], "role": "model" } }, { "content": { "parts": [ { "tool_call": { "id": "call_423307", "tool_type": "GOOGLE_SEARCH_WEB", "args": { "queries": [ "site pypi org project google adk", "google-adk pypi", "google-adk python package latest release", "google adk pypi" ] } } }, { "tool_response": { "id": "call_423307", "tool_type": "GOOGLE_SEARCH_WEB", "response": { "search_suggestions": "\n
\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n
\n
\ngoogle-adk python package latest release\ngoogle-adk pypi\ngoogle adk pypi\nsite pypi org project google adk\n
\n
" } } }, { "text": "**No**, the `google-adk` pin on the dependencies page is not current.\n\n* **Documentation (`dependencies` page):** Currently pinned to `2.10.0`.\n* **Latest Release (PyPI / GitHub `google/adk-python`):** The latest release is `2.12.0` (released October 13, 2026)." } ], "role": "model" }, "grounding_metadata": { "grounding_chunks": [ { "web": { "title": "github.com", "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQEygXpZyjV_RBSvCoxnQkKN9EzdnGbjPuQkuvn2sFoxD1ijW80Vw67rxjhnrvzCFuuVoCP-HjJ1YWF_YmT6p3myoDumXB6rRxBe-w1PT4JRlV7VLRJjxfkZlLIyX9OLXv8WLSQ=" } }, { "web": { "title": "pypistats.org", "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQFSxQ-52WZnRKb462D46YvhxHW67c_Y3LwtVHXqm4cJ512EZJ1lxr0kgDd7fckh5uaI4tpcQzdlJ0vOmq2mCsTg_PUH31F3qraGy4dNNmzVpjZIaOlweBgqJ9z62_N7fA==" } } ], "grounding_supports": [ { "grounding_chunk_indices": [ 0, 1 ], "segment": { "end_index": 68, "text": "**No**, the `google-adk` pin on the dependencies page is not current" } }, { "grounding_chunk_indices": [ 0, 1 ], "segment": { "start_index": 146, "end_index": 260, "text": "**Latest Release (PyPI / GitHub `google/adk-python`):** The latest release is `2.12.0` (released October 13, 2026)" } } ], "search_entry_point": { "rendered_content": "\n
\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n
\n
\n
\ngoogle-adk python package latest release\n
\n
" }, "web_search_queries": [ "google-adk python package latest release" ] } } ], "rendered_content_chars": 4500 }, "slo": { "question": "Do the deploy timings meet the SLO?", "transcript": "call read_page {'page': 'deploy-timings'}\nresult read_page ['page', 'text']\ncode import numpy as np\n \n timings = [41, 38, 44, 40, 39, 122, 42, 43, 37, 45, 40, 41, 39, 118, 44, 42, 40, 38, 43, 41]\n timings_sorted = sorted(timings)\n print(\"Sorted timings:\", timings_sorted)\n print(\"Length:\", len(timings_sorted))\n \n # Let's check various common percentile methods\n for method in ['linear', 'lower', 'higher', 'midpoint', 'nearest', 'weibull']:\n p95 = np.percentile(timings, 95, method=method)\n print(f\"p95 ({method}): {p95}\")\noutput OUTCOME_OK Sorted timings: [37, 38, 38, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 43, 43, 44, 44, 45, 118, 122]\nLength: 20\np95 (linear): 118.2\np95 (lower): 118\np95 (higher): 122\np95 (midpoint): 120.0\np95 (nearest): 118\np95 (weibull): 121.8\nanswer **No, the deploy timings do not meet the SLO.**\n\n### Details\n\nAccording to the **`deploy-timings`** documentation:\n- **SLO target:** p95 under 90 seconds.\n- **Last 20 production deploys (sorted in seconds):** \n `37, 38, 38, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 43, 43, 44, 44, 45, 118, 122`\n\n### Calculation\nFor 20 samples, the 95th percentile covers up to the 19th value:\n- **p95:** **118.2 seconds** (standard linear interpolation; at least **118 seconds** across common percentile estimation methods).\n\nSince 118.2s (or ≥ 118s) is greater than 90 seconds, the deploys are currently violating the SLO.", "sources": [], "citations": [], "requests": [ { "tools": [ [ "code_execution" ], [ "function_declarations" ], [ "google_search" ] ], "tool_config": { "include_server_side_tool_invocations": true } }, { "tools": [ [ "code_execution" ], [ "function_declarations" ], [ "google_search" ] ], "tool_config": { "include_server_side_tool_invocations": true } } ], "responses": [ { "content": { "parts": [ { "function_call": { "id": "call_447774", "args": { "page": "deploy-timings" }, "name": "read_page" } } ], "role": "model" } }, { "content": { "parts": [ { "executable_code": { "code": "import numpy as np\n\ntimings = [41, 38, 44, 40, 39, 122, 42, 43, 37, 45, 40, 41, 39, 118, 44, 42, 40, 38, 43, 41]\ntimings_sorted = sorted(timings)\nprint(\"Sorted timings:\", timings_sorted)\nprint(\"Length:\", len(timings_sorted))\n\n# Let's check various common percentile methods\nfor method in ['linear', 'lower', 'higher', 'midpoint', 'nearest', 'weibull']:\n p95 = np.percentile(timings, 95, method=method)\n print(f\"p95 ({method}): {p95}\")\n", "language": "PYTHON", "id": "call_645744" } }, { "code_execution_result": { "outcome": "OUTCOME_OK", "output": "Sorted timings: [37, 38, 38, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 43, 43, 44, 44, 45, 118, 122]\nLength: 20\np95 (linear): 118.2\np95 (lower): 118\np95 (higher): 122\np95 (midpoint): 120.0\np95 (nearest): 118\np95 (weibull): 121.8\n", "id": "call_645744" } }, { "text": "**No, the deploy timings do not meet the SLO.**\n\n### Details\n\nAccording to the **`deploy-timings`** documentation:\n- **SLO target:** p95 under 90 seconds.\n- **Last 20 production deploys (sorted in seconds):** \n `37, 38, 38, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 43, 43, 44, 44, 45, 118, 122`\n\n### Calculation\nFor 20 samples, the 95th percentile covers up to the 19th value:\n- **p95:** **118.2 seconds** (standard linear interpolation; at least **118 seconds** across common percentile estimation methods).\n\nSince 118.2s (or ≥ 118s) is greater than 90 seconds, the deploys are currently violating the SLO." } ], "role": "model" } } ], "rendered_content_chars": 0 }, "bypass": { "question": "Is the google-adk pin on the dependencies page current?", "transcript": "call read_page {'page': 'dependencies'}\nresult read_page ['page', 'text']\ncall google_search_agent {'request': 'latest release version of google-adk on PyPI'}\nresult google_search_agent ['result']\nsearch ['\"google-adk\" \"2.12.\" site:pypi.org', 'site pypi org project google adk']\nanswer No, the `google-adk` pin is not current:\n\n* **Current Documentation (`dependencies` page):** Pins `google-adk` at version **2.10.0**.\n* **Latest Release (PyPI via Google Search):** The latest release version of `google-adk` is **2.12.0**.\nsource pypistats.org\nsource pypi.org", "sources": [ { "title": "pypistats.org", "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGXW5b1JBD3KhPR0EEWrxec0gNv4Tn1Hc-N_mNPZXD0rQPJX8k_D8oQmZhOgb5t0eEjPlrKsrGuoi4pHFPXRtx_1h6aOw_9Oobi8bTmBc5675xf1jhkWIiy3RuhoWXA" }, { "title": "pypi.org", "uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGHBBI5zIPPURGVGPRnuNmGS7FNv-t0-kkOI2AI_HbpMPr1sboLymLYFGpIRHHiHUQel7QsZOsRb1cvTa64yRZwtiooNyMpMixA5icCv2X8jmx1fcnMWmGv0zo=" } ], "citations": [ { "text": "The latest release version of **`google-adk`** on PyPI is **`2.12.0`**", "sources": [ "pypistats.org", "pypi.org" ] } ], "requests": [ { "tools": [ [ "function_declarations" ] ], "tool_config": null }, { "tools": [ [ "function_declarations" ] ], "tool_config": null }, { "tools": [ [ "function_declarations" ] ], "tool_config": null } ], "responses": [ { "content": { "parts": [ { "function_call": { "id": "call_411083", "args": { "page": "dependencies" }, "name": "read_page" } } ], "role": "model" } }, { "content": { "parts": [ { "function_call": { "id": "call_549597", "args": { "request": "latest release version of google-adk on PyPI" }, "name": "google_search_agent" } } ], "role": "model" } }, { "content": { "parts": [ { "text": "No, the `google-adk` pin is not current:\n\n* **Current Documentation (`dependencies` page):** Pins `google-adk` at version **2.10.0**.\n* **Latest Release (PyPI via Google Search):** The latest release version of `google-adk` is **2.12.0**." } ], "role": "model" } } ], "rendered_content_chars": 0 } }, "failures": { "no_flag": "ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Please enable tool_config.include_server_side_tool_invocations to use Built-in tools with Function calling.', 'status': 'INVALID_ARGUMENT'}}", "gemma": "ValueError: Google search tool is not supported for model ollama_chat/gemma4:e2b" } }