Choose an architecture from task requirements

Build a version-aware command lookup and an approval-bound action proposal. Use their different input and output contracts to decide which decisions belong in application code before introducing a language model.

Problem

When you need this. A documentation lookup and an action request need different permission and output contracts.

A user asks, “How do I roll production back to r41?” A model could write a fluent answer while choosing the wrong version of the command. The engineering problem starts earlier: which documentation applies, who may request an action, and what would count as a correct result?

In this lab, docuctl is a fictional deployment tool. You will build a documentation lookup and an action-proposal workflow. Both use the same versioned pages. You will also run an intentionally inadequate baseline that always returns the staging command from the older version.

Concepts

A lookup answers a read request from an identified source. A workflow applies a known sequence of decisions, such as checking a caller role before constructing a proposal. An agent adds a policy that can choose its next action from observations; the fixed lookup in this lab has no such policy.

An application contract states which inputs are accepted, which output fields are returned and when the program must refuse. Source selection, input validation and permission checks remain application responsibilities even if a model later interprets the user’s wording. A correct command and an authorized proposal are separate requirements.

Architecture

The baseline ignores request metadata. The implemented path validates the request, selects exactly one version/environment page, then either returns its command template or checks the proposal permission and release identifier. Both branches return data. Neither invokes a shell or executes the proposed rollback.

Working code

Write the contract before choosing a model

A contract states the inputs the application accepts and the outputs it promises. This request includes an operation, documentation version and environment. A lookup returns a command template and a source identifier. A proposal also requires a maintainer role and a valid release identifier. No branch executes a command.

The distinction matters. Explaining a command is a read operation. Preparing an action introduces an authorization boundary, even before execution exists. If these are combined into one open-ended prompt, it becomes harder to say which component must refuse an unauthorized request.

The baseline in naive_answer() always returns docuctl rollback RELEASE --env staging. That is a valid template for one supplied page. It is wrong for the current production request because it ignores both version and environment. A syntactically valid answer can still violate the task contract.

Download and extract the complete lab using the link above. Open its ai-system-architecture directory, then run:

python3.12 -m venv .venv
source .venv/bin/activate
python architecture.py
python -m unittest -v test_architecture.py

The program uses Python’s standard library. It needs no API key or model download.

Bind the answer to one source

The source set is deliberately small enough to inspect. pages.json contains separate rollback instructions for version 2 staging, version 3 staging and version 3 production. The code selects on explicit metadata rather than hoping a text search ranks the right version first.

A request for version 3 production selects rollback-v3-production. The returned template uses docuctl releases rollback RELEASE --environment production. Its citation identifies the page that supplied those bytes. Checking that the citation exists checks source identity; it does not establish that an arbitrary generated claim is supported by that source.

"""A local documentation lookup and action-proposal workflow; no model or network."""
from dataclasses import dataclass
from pathlib import Path
import json
import re

@dataclass(frozen=True)
class Request:
    operation: str
    version: str
    environment: str
    role: str = "reader"
    release: str = ""

class Refused(ValueError):
    pass

def answer(request, pages):
    if request.role not in {"reader", "maintainer"}:
        raise Refused("unknown caller role")
    if request.environment not in {"staging", "production"}:
        raise Refused("choose staging or production")
    if request.operation not in {"lookup_rollback", "propose_rollback"}:
        raise Refused("unsupported operation: send for review")
    if request.operation == "propose_rollback" and request.role != "maintainer":
        raise Refused("maintainer role required for an action proposal")
    matches = [p for p in pages if p["version"] == request.version
               and p["environment"] == request.environment and p["topic"] == "rollback"]
    if len(matches) != 1:
        raise Refused("expected one matching page; review missing or conflicting evidence")
    page = matches[0]
    if request.operation == "lookup_rollback":
        return {"architecture": "filtered_lookup", "command_template": page["command"],
                "citation": page["id"], "executed": False}
    if not re.fullmatch(r"r[1-9][0-9]*", request.release):
        raise Refused("release must be an r-prefixed positive integer")
    return {"architecture": "bounded_workflow", "proposed_command":
            page["command"].replace("RELEASE", request.release), "citation": page["id"],
            "approval_required": True, "executed": False}

def naive_answer(request):
    """Deliberately inadequate baseline: ignores version and environment."""
    return "docuctl rollback RELEASE --env staging"

if __name__ == "__main__":
    root = Path(__file__).parent
    pages = json.loads((root / "pages.json").read_text())
    cases = json.loads((root / "cases.json").read_text())
    for case in cases:
        request = Request(**case["request"])
        try:
            result = answer(request, pages)
            result["baseline_matches_source"] = naive_answer(request) == next(
                p["command"] for p in pages if p["id"] == result["citation"])
        except Refused as error:
            result = {"refused": str(error)}
        print(json.dumps({"case": case["id"], **result}, sort_keys=True))

Read answer() in three parts: validate the request, select one page, then produce the result appropriate to the operation. Each part has a distinct reason to refuse. A missing or conflicting page stops the request; the code does not silently substitute another version.

This lookup is sufficient when the task is to return an exact documented command. A model becomes useful when the product needs to interpret varied natural-language requests, explain a passage, or combine several sources. Add that capability at a named boundary. Keep source selection and permission checks in application code, and evaluate interpretation mistakes separately.

Make the action boundary explicit

A maintainer can request a proposal for r41. The workflow substitutes that validated identifier into the selected template and returns approval_required: true. It also returns executed: false.

A reader requesting an action proposal is refused before a proposal is constructed. An invalid release identifier is also refused. The regular expression is an input contract for this fictional identifier; it is not a general shell-escaping solution. There is no shell invocation in the lab.

In a real service, the caller’s role must come from authenticated server context. The fixture puts it in a Python object so the test can exercise the policy. Do not copy this into an HTTP endpoint that trusts a client-supplied role field.

An approval flag alone would not protect a later executor. That executor would need its own authorization check, a binding between approval and exact proposed action, and a policy for repeated requests. Those are separate changes to the application contract.

Compare what each case establishes

The actual local output shows successful lookups, a maintainer proposal, a refused reader proposal and a missing-version refusal:

{"architecture": "filtered_lookup", "baseline_matches_source": true, "case": "older-staging-lookup", "citation": "rollback-v2-staging", "command_template": "docuctl rollback RELEASE --env staging", "executed": false}
{"architecture": "filtered_lookup", "baseline_matches_source": false, "case": "current-production-lookup", "citation": "rollback-v3-production", "command_template": "docuctl releases rollback RELEASE --environment production", "executed": false}
{"approval_required": true, "architecture": "bounded_workflow", "baseline_matches_source": false, "case": "maintainer-proposal", "citation": "rollback-v3-production", "executed": false, "proposed_command": "docuctl releases rollback r41 --environment production"}
{"case": "reader-action-refused", "refused": "maintainer role required for an action proposal"}
{"case": "unknown-version", "refused": "expected one matching page; review missing or conflicting evidence"}

The baseline matches the older staging page and fails the current production case. This comparison diagnoses a missing input dependency. It is not a benchmark showing that rules outperform language models: no language model participated.

Run the tests, then duplicate the version 3 production page with a conflicting command. Predict whether the program should pick the first result or refuse. test_conflicting_pages_refused requires refusal, because source order must not choose operational instructions by accident.

Decide what to build next

For an exact command lookup, keep the filtered lookup. For a multi-step operation with known transitions, add an explicit workflow. For tasks where the next useful investigation depends on previous observations, use the agent-design comparison to decide whether a bounded agent loop buys anything over the fixed workflow.

This program is a local function and CLI experiment. Before serving it, add authenticated identity, an input schema, a versioned source-loading process and request logs that omit secrets. Measure lookup cost with the intended corpus and indexing strategy; this list scan examines every supplied page.

Design exercise: a user omits the documentation version. Would you infer it, use an account setting, or ask the user? Write the rule and a counterexample before changing the implementation.

Failure cases

Compare a wrong result with a required refusal. The baseline returns a plausible command for the wrong request; the reviewed path refuses ambiguous sources and unauthorized proposals before constructing output. Run the supplied tests, then change one source or caller field to observe the corresponding boundary.

Deployment

The deployment unit here is a Python function and a local CLI, not a running service. If it becomes an HTTP application, derive the caller role from authenticated server context, validate the request at entry, and retain the exact source version used for each result. Keep action execution separate until approval can be bound to that exact action.

Scaling

The current selector scans the supplied pages. With a larger source set, index the version and environment metadata and check that duplicate matches still refuse. Measure source loading and selection separately from any future model call; the present experiment supplies no model latency or service-capacity measurement.

Key takeaways

• The naive answer fails because it ignores inputs that change the correct command.
• A citation identifies the selected page; it does not validate arbitrary explanation text.
• A proposal requires permission and a valid release identifier even though execution is absent.

Best practices

• Write an omitted-version case before choosing a default or clarification rule.
• Refuse conflicting matches instead of relying on file order.
• Keep role and approval enforcement outside generated model text.

References

• Google’s Rules of ML: start with a simple baseline and align evaluation with the product objective.
• OWASP Authorization Cheat Sheet: validate permissions on each request and deny by default.

Sources reviewed October 23, 2026. The printed commands and refusal cases come from the bundled local fixture.