Create an evaluation set and a failure taxonomy

Turn the documentation application’s requirements into development cases. Reuse the existing runner, add task-specific checks and introduce controlled faults to learn which failures the suite can actually detect.

Problem

When you need this. A single pass rate hides whether a failure came from source selection, missing context or an unsupported answer.

The architecture pilot returns the correct production command and refuses a reader’s action proposal. Two successful examples are useful, but they do not tell you whether a later change still handles the older documentation, a conflicting source or an unrecognized caller role.

You will turn those requirements into 12 development cases and application-specific checks. The existing generic evaluation runner executes the unchanged lookup/proposal function. Five deliberately injected faults then show whether the suite detects different boundaries. No language model or model judge participates.

Concepts

A case pairs a request with an independently authored expected outcome. A grader checks one requirement, such as output shape or the selected source. A failure taxonomy gives observed defects useful names so that the next investigation starts at a relevant boundary.

A required refusal can pass the task contract. An incomplete trial has no usable response and must remain separate from both a refusal and a wrong answer. Request slices group cases by fields such as environment or caller role. These authored development cases help diagnose the application; they do not estimate production error rates.

Architecture

The unchanged application receives each case through a task adapter. Ordered graders check shape, source, task result and action policy; the generic harness records the verdict and first failed boundary. Five controlled faults change one response each, allowing the same graders to demonstrate their detection behavior.

Working code

Derive cases from the application’s decisions

Start with the inputs that can change the correct behavior: documentation version, environment, caller role and operation. For each boundary, write a normal case and a plausible counterexample. Include a reason so a future editor knows what would be lost by deleting the case.

The set includes version 2 and version 3 staging lookups, a version 3 production lookup, a maintainer proposal and required refusals. It also covers conflicting sources, an unknown version, an invalid release and an unsupported execution operation.

{
  "name": "docuctl-application-contract-v1",
  "task": "Version/environment-bound lookup and role-bound unexecuted proposals",
  "cases": [
    {
      "id": "v2-staging-lookup",
      "split": "development",
      "class": "source-selection",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "2",
          "environment": "staging"
        }
      },
      "expected": {
        "architecture": "filtered_lookup",
        "command_template": "docuctl rollback RELEASE --env staging",
        "citation": "rollback-v2-staging",
        "executed": false
      },
      "fail_if": [],
      "why": "Older documentation uses a different command syntax."
    },
    {
      "id": "v3-staging-lookup",
      "split": "development",
      "class": "source-selection",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "3",
          "environment": "staging"
        }
      },
      "expected": {
        "architecture": "filtered_lookup",
        "command_template": "docuctl releases rollback RELEASE --environment staging",
        "citation": "rollback-v3-staging",
        "executed": false
      },
      "fail_if": [],
      "why": "Current staging still differs from production."
    },
    {
      "id": "v3-production-lookup",
      "split": "development",
      "class": "source-selection",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "3",
          "environment": "production"
        }
      },
      "expected": {
        "architecture": "filtered_lookup",
        "command_template": "docuctl releases rollback RELEASE --environment production",
        "citation": "rollback-v3-production",
        "executed": false
      },
      "fail_if": [],
      "why": "A production lookup must not select a staging source."
    },
    {
      "id": "maintainer-proposal",
      "split": "development",
      "class": "action-policy",
      "input": {
        "request": {
          "operation": "propose_rollback",
          "version": "3",
          "environment": "production",
          "role": "maintainer",
          "release": "r41"
        }
      },
      "expected": {
        "architecture": "bounded_workflow",
        "proposed_command": "docuctl releases rollback r41 --environment production",
        "citation": "rollback-v3-production",
        "approval_required": true,
        "executed": false
      },
      "fail_if": [],
      "why": "A valid proposal stays unexecuted and approval-bound."
    },
    {
      "id": "reader-proposal",
      "split": "development",
      "class": "authorization",
      "input": {
        "request": {
          "operation": "propose_rollback",
          "version": "3",
          "environment": "production",
          "role": "reader",
          "release": "r41"
        }
      },
      "expected": {
        "rejected": "Refused",
        "message": "maintainer role required"
      },
      "fail_if": [],
      "why": "The caller role must be checked before constructing a proposal."
    },
    {
      "id": "unknown-role",
      "split": "development",
      "class": "authorization",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "3",
          "environment": "production",
          "role": "administrator"
        }
      },
      "expected": {
        "rejected": "Refused",
        "message": "unknown caller role"
      },
      "fail_if": [],
      "why": "An unrecognized role does not inherit maintainer permission."
    },
    {
      "id": "unknown-version",
      "split": "development",
      "class": "source-selection",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "9",
          "environment": "production"
        }
      },
      "expected": {
        "rejected": "Refused",
        "message": "expected one matching page"
      },
      "fail_if": [],
      "why": "A missing version must not silently fall back."
    },
    {
      "id": "conflicting-source",
      "split": "development",
      "class": "source-selection",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "3",
          "environment": "production"
        },
        "source_mode": "conflicting"
      },
      "expected": {
        "rejected": "Refused",
        "message": "expected one matching page"
      },
      "fail_if": [],
      "why": "Source order must not choose between conflicting operational instructions."
    },
    {
      "id": "unknown-environment",
      "split": "development",
      "class": "input-contract",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "3",
          "environment": "prod"
        }
      },
      "expected": {
        "rejected": "Refused",
        "message": "choose staging or production"
      },
      "fail_if": [],
      "why": "The application does not infer an environment alias."
    },
    {
      "id": "invalid-release",
      "split": "development",
      "class": "input-contract",
      "input": {
        "request": {
          "operation": "propose_rollback",
          "version": "3",
          "environment": "production",
          "role": "maintainer",
          "release": "r0"
        }
      },
      "expected": {
        "rejected": "Refused",
        "message": "r-prefixed positive integer"
      },
      "fail_if": [],
      "why": "Release identifiers follow the stated bounded proposal contract."
    },
    {
      "id": "unsupported-operation",
      "split": "development",
      "class": "input-contract",
      "input": {
        "request": {
          "operation": "execute_rollback",
          "version": "3",
          "environment": "production"
        }
      },
      "expected": {
        "rejected": "Refused",
        "message": "unsupported operation"
      },
      "fail_if": [],
      "why": "There is no command execution branch."
    },
    {
      "id": "maintainer-lookup",
      "split": "development",
      "class": "source-selection",
      "input": {
        "request": {
          "operation": "lookup_rollback",
          "version": "3",
          "environment": "production",
          "role": "maintainer"
        }
      },
      "expected": {
        "architecture": "filtered_lookup",
        "command_template": "docuctl releases rollback RELEASE --environment production",
        "citation": "rollback-v3-production",
        "executed": false
      },
      "fail_if": [],
      "why": "Maintainer status does not turn a lookup into execution."
    }
  ]
}

Expected commands and citation IDs are authored in this file, independently of the function’s current output. Generating the expectation by calling the same lookup would merely preserve its existing mistake. test_expectation_is_independent_of_the_current_source_command changes a source command and requires the unchanged expectation to catch it.

All cases are visible development fixtures. Use them to build and diagnose the application. A later comparison needs an appropriately reviewed held-out set; relabeling these already inspected examples would not make them unseen evaluation data. The existing leakage lesson owns grouping related examples and checking contamination.

Download the complete lab and enter ai-system-evaluation/:

python3.12 -m venv .venv
source .venv/bin/activate
python evaluate_cases.py
python -m unittest -v test_evaluation.py

The standard-library lab includes the unchanged architecture example and a vendored copy of the existing generic harness. That harness owns case loading, version hashes, trials, verdicts and run records. You are adding the task adapter and interpretation, not another evaluation engine. The LLM Evals course supplies the prerequisite vocabulary and model-evaluation workflow.

Check the result in layers

A JSON-shaped response can still choose the wrong source. A source-bound command can still omit the approval requirement. Keep those checks separate so a failure points to a useful application boundary:

"""Task-specific checks; the existing generic harness owns trial execution."""
NAME, VERSION = 'docuctl-request-contract', '1'
OUTPUT_FIELDS = {'architecture': str, 'citation': str, 'executed': bool,
                 'command_template': str, 'proposed_command': str, 'approval_required': bool}


def validate_expected(expected):
    lookup = {'architecture', 'citation', 'executed', 'command_template'}
    proposal = {'architecture', 'citation', 'executed', 'proposed_command', 'approval_required'}
    if set(expected) not in (lookup, proposal):
        raise ValueError('expectations need the complete lookup or proposal contract')
    if any(type(value) is not OUTPUT_FIELDS[key] for key, value in expected.items()):
        raise ValueError('expected output fields have incorrect types')
    if expected['executed'] or ('approval_required' in expected and not expected['approval_required']):
        raise ValueError('this application only returns unexecuted approval-bound proposals')


def output_contract(case, output):
    expected = case['expected']
    if not isinstance(output, dict) or set(output) != set(expected):
        return False, 'output must have exactly the fields required by the operation'
    if any(type(value) is not OUTPUT_FIELDS[key] for key, value in output.items()):
        return False, 'output fields have incorrect types'
    return True, ''


def source_binding(case, output):
    return (output['citation'] == case['expected']['citation'], 'selected source differs from the expected version/environment source')


def task_result(case, output):
    expected = case['expected']
    keys = [key for key in ('architecture', 'command_template', 'proposed_command') if key in expected]
    return all(output[key] == expected[key] for key in keys), 'command or operation result differs from the task contract'


def action_policy(case, output):
    return (output['executed'] is False and output.get('approval_required', True) is True,
            'commands must remain unexecuted; proposals require approval')


GRADERS = tuple({'name': name, 'version': VERSION, 'kind': 'contract' if name == 'output-contract' else 'outcome', 'check': check}
                for name, check in [('output-contract', output_contract), ('source-binding', source_binding),
                                    ('task-result', task_result), ('action-policy', action_policy)])


def validate_inputs(case_set):
    """Validate fixture structure while preserving intentionally invalid values."""
    required = {'operation', 'version', 'environment'}
    allowed = required | {'role', 'release'}
    for case in case_set['cases']:
        value = case['input']
        if not isinstance(value, dict) or not {'request'} <= set(value) <= {'request', 'source_mode'}:
            raise ValueError('case input needs a request and optional source_mode')
        request = value['request']
        if not isinstance(request, dict) or not required <= set(request) <= allowed:
            raise ValueError('request fixture has missing or unknown fields')
        if any(not isinstance(v, str) or len(v) > 100 for v in request.values()):
            raise ValueError('request fixture values must be bounded strings')
        if 'source_mode' in value and value['source_mode'] != 'conflicting':
            raise ValueError('unknown source fixture variation')

output-contract checks exact fields and types before another checker indexes the response. source-binding checks the expected citation. task-result compares the command and operation result. action-policy requires an unexecuted result and approval for a proposal.

The source check is intentionally narrow: it checks which supplied page the result identifies. It would not establish that arbitrary generated prose is supported by that page. Free-text grounding needs the separate rubric and source-support work later in this course.

The harness stops at the first failed check. A malformed response does not also produce misleading source errors or an exception from a checker that assumed a missing field existed. This ordering aids diagnosis, but it means one tag does not enumerate every defect in an output.

Count a required refusal as a successful outcome

The reader-proposal case expects Refused with a maintainer-role message. Returning a well-formed proposal is a failure even if its command is correct. An unexpected exception is not equivalent to the required refusal.

The fixture’s caller role exercises application policy; it is not authentication. A deployed service must obtain that role from trusted request context and enforce permission on the action path. OWASP’s authorization guidance recommends checking permission for each request. Adding an approval field to model output cannot supply that permission check.

The runner also keeps a missing response as incomplete. It does not drop the case or convert the absence into a successful refusal. A timeout or dependency error needs investigation even when no wrong command was returned.

Prove the suite can detect the boundaries it names

The controlled systems change one target case each: choose the older staging source, bypass the reader restriction, remove approval, omit a required field or return no output.

"""The reviewed local function and five single-fault probes; no language model."""
import copy
import json
from pathlib import Path
from architecture import Request, answer

ROOT = Path(__file__).resolve().parent
PAGES = json.loads((ROOT / 'pages.json').read_text())


class ContractSystem:
    mode = 'local_function'
    model = prompt = tools = None
    version = '1'

    def __init__(self, fault='none'):
        if fault not in {'none', 'source', 'permission', 'approval', 'format', 'no_output'}:
            raise ValueError('unknown controlled fault')
        self.fault = fault
        self.name = 'filtered-lookup' if fault == 'none' else f'controlled-{fault}'

    def respond(self, case, trial):
        request = Request(**case['input']['request'])
        pages = copy.deepcopy(PAGES)
        if case['input'].get('source_mode') == 'conflicting':
            pages.append(dict(pages[-1], id='conflicting-production', command='conflicting command'))
        if self.fault == 'permission' and case['id'] == 'reader-proposal':
            request = Request(**{**case['input']['request'], 'role': 'maintainer'})
        if self.fault == 'source' and case['id'] == 'v3-production-lookup':
            request = Request(operation='lookup_rollback', version='2', environment='staging')
        if self.fault == 'no_output' and case['id'] == 'v3-staging-lookup':
            return None
        output = answer(request, pages)
        if self.fault == 'approval' and case['id'] == 'maintainer-proposal':
            output['approval_required'] = False
        if self.fault == 'format' and case['id'] == 'v2-staging-lookup':
            del output['citation']
        return output

12 authored development cases; local Python function, no model or judge
none: pass=12; fail=0; incomplete=0
source: pass=11; fail=1; incomplete=0
  v3-production-lookup: source-selection
permission: pass=11; fail=1; incomplete=0
  reader-proposal: authorization
approval: pass=11; fail=1; incomplete=0
  maintainer-proposal: action-policy
format: pass=11; fail=1; incomplete=0
  v2-staging-lookup: response-contract
no_output: pass=11; fail=0; incomplete=1
  v3-staging-lookup: dependency-or-no-output
Boundary tags identify the first failed check, not an automatically proven root cause.

The unchanged function passes all 12 authored cases, including the required refusals. Each fault probe produces one failed or incomplete case. This checks the suite against known defects. It is not a measured failure rate for a model or the deployed application.

The taxonomy groups the first observed failure into response contract, source selection, task result, authorization, action policy or dependency/no output. In these injected examples, you know the cause because you introduced it. In a real report, use the tag to choose the next investigation: source filtering, permission handling, response construction or dependency logs. The tag alone does not prove which implementation component caused the failure.

Inspect slices before interpreting one aggregate

A request slice is a subset sharing a relevant property. The report groups results by version, environment, role and operation. Inspect the environment slice for the wrong-source probe:

The production slice contains the source-selection failure. The staging slice does not. The prod row is a deliberately invalid environment input whose correct outcome is refusal; it is not a third supported deployment environment.

One case can appear in a production slice, a reader slice and a lookup slice. Do not sum counts across these dimensions or treat them as independent observations. Sparse authored slices expose missing coverage; they do not estimate how often those requests occur in real traffic.

Keep cases, checks and application files identifiable

python evaluate_cases.py --check
python evaluate_cases.py --save my-report.json

The full report preserves every case, verdict and reason, plus case, grader, harness and application hashes. --check ignores incidental execution timing, timestamp and environment descriptions while comparing those bindings and outcomes. Environment request slices remain part of the comparison; only the manifest’s execution environment is omitted. It makes fresh local function calls; it is not a saved-response model replay.

Duplicate case IDs, incomplete expectations and misspelled fixture request fields are refused before interpreting a report. The suite also checks that the missing-field probe stops at the output checker and that no-output remains incomplete.

Design exercise: add a source with a valid citation but an incorrect command. Predict which check should fail and why the citation check alone should pass. Then write a new case for an omitted version, decide whether clarification or refusal is the intended behavior, and review that contract before changing the application.

Failure cases

A well-formed command with the wrong source must fail the source check. A correct proposal returned to a reader must fail authorization. No response remains incomplete. These contrasts show why checking only JSON shape, or treating every exception as a successful refusal, would miss the application requirement.

Deployment

Keep cases, graders and the application revision with a release report. A CI job can rerun these deterministic contracts before an application change is accepted. Add representative deployment checks at the actual service boundary; this local function suite does not authenticate callers or exercise an HTTP endpoint.

Scaling

Adding cases increases executions and report rows. Keep stable case IDs and inspect coverage by relevant request groups before expanding the set. When adding model trials, reuse the existing runner’s trial and versioning facilities instead of changing this task adapter into a second evaluation engine.

Key takeaways

• The expected result must be independent of the current implementation.
• Ordered checks identify the first observed failure, not every possible defect.
• Controlled faults test detection; development-set pass counts do not measure real-world reliability.

Best practices

• Add a reason for each boundary case so coverage survives later edits.
• Keep required refusals distinct from unexpected exceptions and missing output.
• Inspect overlapping slices without adding their counts together.

References

• OWASP, Authorization Cheat Sheet: enforce request permissions in application code.
• Google, Rules of ML: establish metrics and a simple baseline before adding model complexity.

Sources checked October 24, 2026. All reported verdicts come from the bundled local Python function and authored fault probes.