Call a local model and a hosted API

You send one rollback question to local Gemma and hosted Gemini. Both recorded answers say that database migrations are not rolled back; the returned usage fields and request times differ.

So far every request went to a model on your own machine. Many teams also use hosted models, which a provider runs behind an HTTP API and bills per token. The docs assistant asks one question both ways, Gemma through Ollama and Gemini through the Gemini API. Compare what each API returns and how each one fails.

When you need this. Your team wants the docs assistant in CI, where it must not send internal docs off the network, and in a public help page, where it must answer quickly for many users at once. Those two places may need different models, and the code should make switching cheap.

Recorded example. The hosted/local usage comparison records the earlier shipctl question; it does not establish token usage or latency for a new docuctl request.

1. Ask the local model

Download the local-models-and-hosted-apis lab and extract its code, docs, tests and README together.

unzip local-models-and-hosted-apis.zip
cd local-models-and-hosted-apis
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt
# The live comparison calls BOTH providers; set GOOGLE_API_KEY first.
export GEMINI_MODEL=gemini-3.8-flash

# Offline tests: no model calls
.venv/bin/python -m pytest -q

# Run after reading the implementation below
ollama pull gemma4:e2b
.venv/bin/python ask_both.py

Both clients are replaced in the tests. Running ask_both.py calls Ollama and the Gemini API; the hosted request uses your configured API key. Both functions send the same system page, rollback.md, and the same question:

"""Send one docs question to a local model (Ollama) and a hosted API (Gemini). Run: python ask_both.py
Needs Ollama with gemma4:e2b pulled, and GOOGLE_API_KEY in the environment for the hosted call."""
import os
import time
from pathlib import Path

import ollama
from google import genai
from google.genai import types

SYSTEM = "Answer from this docuctl page only, in one sentence.\n\n" + (Path(__file__).parent / "docs/rollback.md").read_text()
QUESTION = "Does a rollback undo database migrations?"


def ask_local(question: str, model: str = "gemma4:e2b") -> dict:
    start = time.perf_counter()
    reply = ollama.chat(model=model, think=False, options={"temperature": 0},
                        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": question}])
    return {"where": f"local {model}", "seconds": round(time.perf_counter() - start, 2),
            "prompt_tokens": reply.prompt_eval_count, "output_tokens": reply.eval_count,
            "answer": reply.message.content.strip()}


def ask_hosted(question: str, model: str | None = None) -> dict:
    model = model or os.environ.get("GEMINI_MODEL", "gemini-3.8-flash")
    client = genai.Client()  # reads GOOGLE_API_KEY
    start = time.perf_counter()
    reply = client.models.generate_content(
        model=model, contents=question,
        config=types.GenerateContentConfig(system_instruction=SYSTEM, temperature=0))
    usage = reply.usage_metadata
    return {"where": f"hosted {model}", "seconds": round(time.perf_counter() - start, 2),
            "prompt_tokens": usage.prompt_token_count, "output_tokens": usage.candidates_token_count,
            "thinking_tokens": usage.thoughts_token_count, "answer": reply.text.strip()}


def main() -> None:
    for result in (ask_local(QUESTION), ask_hosted(QUESTION)):
        print(result)


if __name__ == "__main__":
    main()

ask_local() is the call you already know. Ollama is a local server on localhost:11434; the Python client sends it HTTP requests, and the server runs the model from weights on disk. This lab targets a local runtime with local weights. Check the client URL and runtime configuration before treating that as a data-handling guarantee.

The local server is not running

A client pointed at a port where no Ollama server listens raises ConnectionError. In a service, check the local runtime at startup and report it as a dependency failure, not as a model error, so that whoever is on call knows to restart a process rather than change a prompt.

2. Ask the hosted model

GEMINI_MODEL selects the hosted model; an explicit model argument takes precedence. Its default is the model used in the saved comparison. Check your account’s available models before making a new request.

ask_hosted() uses Google’s google-genai client. genai.Client() reads the API key from the GOOGLE_API_KEY environment variable, so the example does not hard-code the key. The request has the same parts with different names: system_instruction instead of a system message, contents for the question, and a GenerateContentConfig for options.

The response reports usage in usage_metadata. Here is the recorded run:

{'where': 'local gemma4:e2b', 'seconds': 0.87, 'prompt_tokens': 169, 'output_tokens': 28, 'answer': 'Database migrations are not rolled back, and if the release you leave ran a migration, you must write a new migration that reverses it.'}
{'where': 'hosted gemini-3.8-flash', 'seconds': 2.79, 'prompt_tokens': 158, 'output_tokens': 9, 'thinking_tokens': 186, 'answer': 'No, database migrations are not rolled back.'}

Each side has its own precondition, and the recorded run of failures.py breaks one of each:

"""What each side fails with when its precondition is missing. Run: python failures.py"""
import os

import ollama

from ask_both import QUESTION, ask_hosted

try:  # the hosted API needs a key on every request
    os.environ.pop("GOOGLE_API_KEY", None)
    os.environ.pop("GEMINI_API_KEY", None)
    ask_hosted(QUESTION)
except ValueError as error:
    print("hosted, no API key ->", type(error).__name__, error)

try:  # the local API is a server on this machine; nothing answers if it is not running
    ollama.Client(host="http://127.0.0.1:11435").chat(model="gemma4:e2b", messages=[{"role": "user", "content": QUESTION}])
except ConnectionError as error:
    print("local, server not running ->", type(error).__name__, str(error).splitlines()[0])

hosted, no API key -> ValueError No API key was provided. Please pass a valid API key. Learn how to create an API key at https://ai.google.dev/gemini-api/docs/api-key.
local, server not running -> ConnectionError Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download

The API key is missing

With both GOOGLE_API_KEY and GEMINI_API_KEY unset, genai.Client() raises ValueError before it sends anything. Load the key from the environment or a secret manager, check it when the service starts, and never write it into code, logs or recorded output.

3. Compare what came back

Both answers are correct: migrations are not rolled back. The details differ in ways that matter to an application:

• Token counts differ for the same text. Each model has its own tokenizer and template, as the tokenizer lesson showed, so a budget for one model is not a budget for the other.
• The hosted model thought before it answered. The saved hosted request used a model that generates internal reasoning tokens before its visible response, and spent many more tokens on that reasoning than it spent on the visible nine-token answer. Those tokens add latency and appear in usage, so budget from usage_metadata, not from the length of the text.
• Latency includes the network. The hosted call crossed the internet and generated thinking tokens; the local one did neither. On a busy local machine, or with a larger local model, the comparison can go the other way, so measure with your own traffic.

Separate inference from the API boundary

Both clients turn a function call into an HTTP request with a JSON body and parse a JSON response. What differs is who runs the model:

Many teams use both: a local model for development, private data or offline CI, and a hosted model where quality or scale matters most. Keep the provider behind one function in your code, as ask_local() and ask_hosted() are here, so that switching is a configuration change. Libraries such as LangChain’s chat model wrappers do the same for many providers.

Try it: switch providers with one setting

Add a PROVIDER environment variable that selects ask_local or ask_hosted, and make main() call only that one. Run each provider five times and record the median latency and the token counts. Then lower the thinking effort with types.ThinkingConfig, as the Gemini thinking guide describes, and compare thinking tokens and latency. The offline tests replace both clients: .venv/bin/python -m pytest -q.

Get structured output and validate it with Pydantic makes the docs assistant return a typed object instead of prose, so that code can use its answer.

References

Checked October 21, 2026.

• Gemini API: text generation
• Gemini API: thinking
• Gemini API: models
• Gemini API: API keys
• Ollama API: chat