Build the developer knowledge assistant

Build a separate documentation assistant for docuctl that applies the retrieval concepts from earlier lessons: ask a question, choose a documentation version and inspect the sources behind the answer.

When you need this. Use this capstone when separate retrieval scripts work, but you need a usable application that keeps version selection, source access, answer validation and evaluation connected.

This capstone uses a fictional docuctl command-line tool. A developer can ask how to roll back staging, rotate an API token or find a configuration file. The assistant explains commands; it cannot execute them. Its browser UI and CLI call the same FastAPI service.

Application boundary. This is a separate docuctl application, with paragraph IDs, eligible-only lexical ranking and an extractive answer baseline. It reuses the course’s design decisions, not the earlier Python hybrid index or scripted controller. Its local release check evaluates its own cases. It does not clear the failed Python retrieval-task gate in the preceding release lesson.

The application example below was run locally with the current docuctl documents and the extractive provider. It makes no model request. The optional Gemini section explains how to add generated answers separately.

Before you start. Review collection authorization, retrieval caching and task evaluation. Download the complete capstone lab and extract it. The README lists the files and commands.

1. Start with a working answer baseline

Use a baseline you can inspect before adding generated explanations. The default extractive provider returns the selected documentation passages verbatim. That makes the retrieval result visible and needs no model or API key. Matching passages are a useful baseline; they do not establish that the passages fully answer the question.

Open a terminal in developer-knowledge-assistant/:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt
python -m uvicorn api:app --host 127.0.0.1 --port 8120

Open http://127.0.0.1:8120 in your browser. The included reader token opens the handbook. Choose docuctl 3.5 and ask How do I roll back a staging deployment? Read the quoted instructions, then open the source document beneath the answer.

In a second terminal, ask the same question through the CLI:

source .venv/bin/activate
python cli.py "How do I roll back a staging deployment?"
python cli.py "How do I deploy staging?" --version 3.4
python cli.py "Who won Wimbledon?"

rollback: HTTP 200 · answered
To undo a staging deployment in docuctl 3.5, list known releases with `docuctl releases --env staging`. Then run `docuctl rollback --env staging --to RELEASE_ID`.

To deploy a staging release with docuctl 3.5, run `docuctl deploy --env staging --release RELEASE_ID`. Confirm the release with `docuctl status --env staging`.
rotate-token: HTTP 200 · answered
To rotate an API token in docuctl 3.5, create a replacement with `docuctl token create --name rotation`. Update the application's secret store and check a request before revoking the old token with `docuctl token revoke TOKEN_ID`.

Never put an API token in documentation, a shell history or a support message. These instructions use TOKEN_ID as a placeholder, not a credential.
previous-version: HTTP 200 · answered
In docuctl 3.4, deploy with `docuctl deploy staging --tag RELEASE_ID`. Check the result with `docuctl status staging`. Version 3.5 replaces positional environments with `--env`.

In docuctl 3.4, undo a staging deployment with `docuctl rollback staging RELEASE_ID`. Check the active release with `docuctl status staging`.
production: HTTP 200 · answered
Production deployment requires a reviewed release and approval from the on-call maintainer. A maintainer runs `docuctl deploy --env production --release RELEASE_ID --approval APPROVAL_ID`.

If production health checks fail, the maintainer follows the approved rollback runbook and records the release ID in the incident. A reader token cannot retrieve this collection.
repeat-rollback: HTTP 200 · answered
To undo a staging deployment in docuctl 3.5, list known releases with `docuctl releases --env staging`. Then run `docuctl rollback --env staging --to RELEASE_ID`.

To deploy a staging release with docuctl 3.5, run `docuctl deploy --env staging --release RELEASE_ID`. Confirm the release with `docuctl status --env staging`.
unknown: HTTP 200 · abstain
I could not find supporting documentation for that question in this collection and version.
reader-denied: HTTP 403 · Collection access denied
Provider: extractive. Saved 7 real HTTP activities.

These are actual HTTP results from the extractive provider, which quotes source passages rather than writing a generated explanation. Read the raw responses and execution details. The rollback result also includes a related deployment passage; a retrieved passage is not necessarily needed for the answer.

The version selector matters. The fictional 3.4 docs use a positional environment and --tag; 3.5 uses --env and --release. Retrieving both versions and asking the model to resolve the conflict would lose the application’s explicit version choice.

2. Retrieve only sources the caller can read

docs/manifest.json assigns each document an ID, version and collection. The corpus loader splits its paragraphs into passages and retains their source offsets. The assistant filters those passages by the server’s grants and the requested version before computing BM25 statistics.

The demo tokens map to reader and maintainer roles on the server. A request cannot grant itself a role. Switch the collection to operations while keeping the reader token: the service returns HTTP 403 before a search or model request. A maintainer token can read the fictional production-deployment runbook.

Unauthorized cache hit. The retrieval cache includes principal, role, version, collection, question and corpus snapshot in its key. Access is still checked before lookup. Otherwise, an earlier permitted request could make a later unauthorized request appear safe. Opening a cited source calls a separate endpoint that checks access again.

The cache retains retrieval candidates, not generated answers. Ask the same question twice and inspect the request details: the second retrieval can hit the cache, while a Gemini answer would still require its own model request. Cache expiry uses a monotonic clock, and a changed corpus produces a different snapshot.

3. Admit bounded context and validate the answer

A query can perform one search and, when that finds nothing, one configured alias refinement. It reads at most three candidate passages. Each complete passage is admitted only if its serialized source ID and text fit the remaining context-byte allowance.

    def ask(self, principal, role, request):
        # Authorize before admission too: denied requests do not consume model capacity.
        self.corpus.eligible(role, request.version, request.collection)
        if not self.slots.acquire(blocking=False): raise Busy('Assistant is busy; try later')
        try:
            passages, trace = self.retrieve(principal, role, request)
            admitted, used = [], 0
            for p in passages:
                size = len(json.dumps({'id': p.id, 'text': p.text}, ensure_ascii=False).encode())
                if used+size <= self.context_bytes:
                    admitted.append(p); used += size
            trace.update(context_bytes=used, source_reads=len(admitted), snapshot=self.corpus.snapshot,
                         provider=self.provider.name, model_requests=int(bool(admitted) and self.provider.name!='extractive'))
            if not admitted:
                answer = Answer(status='abstain', answer='I could not find supporting documentation for that question in this collection and version.', citations=[])
            else:
                answer = validate_answer(self.provider.generate(request.question, admitted), admitted)
            return {**answer.model_dump(), 'sources': [{'id':p.id,'document':p.document,'title':p.title,
                'version':p.version,'quote':p.text,'start':p.start,'end':p.end} for p in admitted], 'trace':trace}
        finally:
            self.slots.release()

This is a UTF-8 byte bound for source context, not a token count for the complete model request. Instructions, the question and wrapper fields add input. The token-budget lesson shows how to replace this small-corpus bound with the model’s tokenizer when extending the application.

No supporting source. A question with no lexical matches produces an abstention without calling the provider.

Context too small. If the context allowance cannot admit any passage, the assistant abstains too; it does not silently send an unbounded request.

Fabricated citation. A generated answer must cite admitted source IDs and exact quotes from those passages. Unknown IDs, changed quotes and an answered status with no citations are refused. These checks do not prove that every sentence in a free-form explanation follows from the cited text. Inspect the explanation and its quotes together.

All slots busy. One process admits two answer requests. A third concurrent request receives HTTP 429 while both slots are occupied. The slot is released even when the provider fails. This is local concurrency admission, not a multi-worker or distributed rate limiter.

4. Add a generated explanation when you need one

The optional Gemini adapter uses the same retrieved context and returns structured JSON: an answer status, short explanation and citations. It has no shell, browsing or document-write tools. The prompt asks the model to quote the documentation and ignore instructions embedded in it.

Set GOOGLE_API_KEY or GEMINI_API_KEY in your shell, stop the extractive server, then start:

CAPSTONE_PROVIDER=gemini python -m uvicorn api:app --host 127.0.0.1 --port 8120

Each supported question makes one API request. The adapter permits four generation requests per server process, uses LOW thinking and caps each output at 1,024 tokens. It does not retry a failed or incomplete answer. The browser sends questions to your local API; the API key stays in the server process.

The bundled application checks below use the current docuctl extractive baseline. Running the Gemini command above is a separate model execution; review its answers against the returned passages.

How do I roll back a staging deployment?

In shipctl 3.5, you can roll back a staging deployment by first listing known releases with `shipctl releases --env staging` and then executing `shipctl rollback --env staging --to RELEASE_ID`.

Source: rollback-35:p1
To undo a staging deployment in shipctl 3.5, list known releases with `shipctl releases --env staging`. Then run `shipctl rollback --env staging --to RELEASE_ID`.

5. Evaluate the application before accepting a release

The development cases cover current and older commands, unsupported questions, allowed operations access and denied operations access. They were written while building this example. Their pass count describes these cases; it does not estimate accuracy on your team’s questions.

python -m unittest discover -v
python evaluate.py --output runs/local-docuctl/evaluation.json
python release_check.py seal --manifest runs/local-docuctl/release.json
python release_check.py verify --manifest runs/local-docuctl/release.json

PASS staging-deploy
PASS rollback
PASS token-rotation
PASS configuration
PASS old-deploy
PASS old-rollback
PASS unrelated
PASS denied-operations
PASS maintainer-production
PASS empty-version
10/10 application cases passed (extractive).

The recorded baseline passes 10/10 application cases. release_check.py reruns those checks and compares the current code, UI, documents and case definitions with the saved local manifest. Changing a source file or editing a saved pass count makes verification fail. Run the checks again and review the change before sealing a new baseline. This command verifies a local bundle; it does not deploy the service or evaluate Gemini’s answer quality.

Changed release inputs. An earlier evaluation is not enough after changing documentation, retrieval settings or the prompt. Keep the earlier manifest if you need to compare releases; replacing it accepts a new baseline rather than preserving the old result.

Changed pass count. Editing a saved success count does not make a release pass: verification reruns the application cases and compares their results.

How it works

The UI and CLI submit the same request shape. The API authenticates the demo token, validates a small JSON body and passes the server-side identity to the assistant. Permission and version filtering happen before retrieval and cache lookup. The assistant selects bounded source context, obtains an extractive or generated answer, checks its citations and returns the answer with source references and a compact request trace.

The source endpoint checks permissions again when the reader opens a document. The browser renders answers and source text as text, so a document cannot inject HTML into the interface. The corpus is loaded at startup, and its snapshot scopes cache entries. Restart the process after changing the documents.

Check your understanding

What’s next

Try your own small documentation set. Add questions from colleagues, including outdated commands and questions the docs cannot answer. Keep a separate set for evaluating later changes. Add vector retrieval when lexical matching misses the questions you actually observe, and use the earlier ingestion and token-budget labs as the corpus grows.

References

The FastAPI request-body guide describes typed request validation, and its lifespan guide covers startup and shutdown. The Gemini structured-output guide documents JSON-schema responses; its thinking guide describes thinking settings. Python’s bounded semaphore supports the process-local admission control. The docuctl commands and documents are fictional teaching fixtures.