Bound a local repair process

Evaluate a small boundary repair inside a restricted candidate interface. Validate its syntax, write a fresh candidate workspace and run an independent checker so a claimed fix cannot replace observed behavior.

Problem

When you need this. A proposed code repair needs execution limits before it is run locally.

A retry helper permits one attempt too many because its condition includes the upper boundary. A patch changes the comparison and reports success. Before trusting that claim, the application must execute the candidate against the required boundary behavior.

You will build a small repair harness around this task. It accepts only a comparison between two named variables, writes a fresh candidate file, runs an independent checker in a separate Python process and records the exit status. Candidate expressions are fixed local fixtures; no model generates or executes a patch in this lesson.

Concepts

A candidate interface defines what kind of change the application will accept. This lab permits a comparison of named variables, not arbitrary Python source. Syntax validation checks that interface before execution.

A harness creates the workspace, starts the checker and records its result. Process exit and timeout are execution outcomes; neither comes from the candidate’s completion claim. A separate process improves lifecycle control, but it is not a general isolation boundary for arbitrary untrusted code.

Architecture

A fixed candidate expression enters a syntax allowlist. An accepted expression is written into a fresh temporary program, then a separate Python checker exercises the required boundary behavior. The parent records exit status or timeout and cleans the workspace. Model generation is absent from this controlled fixture.

Working code

Define the behavior before accepting a patch

attempt is a zero-based nonnegative integer. max_attempts is a positive integer. The helper should permit an attempt only while its index is below the maximum. Equality belongs to the refusal boundary.

{
  "requirement": "attempt is a zero-based nonnegative integer; permit a retry while attempt is below max_attempts",
  "baseline": "attempt <= max_attempts",
  "checks": [
    [
      0,
      1
    ],
    [
      1,
      1
    ],
    [
      2,
      3
    ],
    [
      3,
      3
    ],
    [
      4,
      3
    ],
    [
      0,
      5
    ]
  ]
}

The candidate may change one expression in a fixed function. It cannot edit the checker, replace the task inputs or choose its own expected results. That separation is the central harness decision: a submitted change supplies behavior to test, not the definition of success.

Download the complete lab and enter agent-engineering/:

python3.12 -m venv .venv
source .venv/bin/activate
python terminal_harness.py
python -m unittest -v test_harness.py

The program creates and removes temporary local directories. It neither modifies a real repository nor runs shell commands supplied by a policy.

Validate the expression before creating executable code

validate_patch() first bounds the string length, parses it as an expression and inspects the abstract syntax tree. It accepts one comparison with attempt on the left and max_attempts on the right. The comparison operator must be one of the explicitly allowed operators.

That is a positive grammar, not a search for suspicious substrings. A call, attribute lookup, import or chained comparison has a different syntax-tree shape and is rejected. ast.unparse() produces the accepted expression used in the generated function.

"""A deliberately small repair task, with a restricted expression boundary."""
import ast
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
from core import load

CHECKS = [tuple(pair) for pair in load('repair-task.json')['checks']]


def validate_patch(expression):
    if not isinstance(expression, str) or len(expression) > 64:
        raise ValueError('patch must be at most 64 characters')
    tree = ast.parse(expression, mode='eval')
    node = tree.body
    if (not isinstance(node, ast.Compare) or len(node.ops) != 1 or len(node.comparators) != 1
            or not isinstance(node.left, ast.Name) or node.left.id != 'attempt'
            or not isinstance(node.comparators[0], ast.Name) or node.comparators[0].id != 'max_attempts'
            or not isinstance(node.ops[0], (ast.Lt, ast.LtE, ast.Gt, ast.GtE, ast.Eq, ast.NotEq))):
        raise ValueError('only a comparison between attempt and max_attempts is allowed')
    return ast.unparse(node)


def check_candidate(expression, directory=None, *, timeout=2):
    if type(timeout) not in (int, float) or not 0 < timeout <= 2:
        raise ValueError('timeout must be positive and at most 2 seconds')
    try:
        expression = validate_patch(expression)
    except (ValueError, SyntaxError):
        return {'outcome': 'rejected_patch', 'passed': 0, 'total': len(CHECKS)}
    with tempfile.TemporaryDirectory(prefix='docuctl-repair-', dir=directory) as work:
        path = Path(work)
        (path / 'candidate.py').write_text(
            'def can_retry(attempt,max_attempts):\n'
            '    if type(attempt) is not int or type(max_attempts) is not int or attempt<0 or max_attempts<1:\n'
            '        raise ValueError("invalid retry inputs")\n'
            f'    return {expression}\n')
        # The checker uses a specification independent of the submitted expression.
        (path / 'check.py').write_text(
            'import json,sys\n'
            'sys.path.insert(0,sys.argv[1])\n'
            'from candidate import can_retry\n'
            f'checks={CHECKS!r}\n'
            'outcomes=[can_retry(a,m)==(a in range(m)) for a,m in checks]\n'
            'print(json.dumps({"passed":sum(outcomes),"total":len(outcomes)}))\n'
            'sys.exit(0 if all(outcomes) else 1)\n')
        try:
            process = subprocess.run([sys.executable, '-I', str(path / 'check.py'), str(path)],
                                     cwd=path, env={'PATH': os.defpath}, text=True,
                                     capture_output=True, timeout=timeout, check=False)
        except subprocess.TimeoutExpired:
            return {'outcome': 'timeout', 'passed': 0, 'total': len(CHECKS)}
        if len(process.stdout) > 8192 or process.returncode not in (0, 1):
            return {'outcome': 'checker_error', 'passed': 0, 'total': len(CHECKS)}
        result = json.loads(process.stdout)
        return {'outcome': 'passed' if process.returncode == 0 else 'failed',
                **result, 'exit_status': process.returncode}


def main():
    for expression in ('attempt <= max_attempts', 'attempt < max_attempts',
                       '__import__("os").getcwd()'):
        r = check_candidate(expression)
        print(f"{expression}: {r['outcome']} ({r['passed']}/{r['total']})")
    print('Fresh temporary workspace per candidate. This restricted task is not a general OS sandbox.')


if __name__ == '__main__':
    main()

test_imports_cannot_enter_the_candidate submits an import-shaped call and expects rejected_patch. It checks the real candidate boundary before a process is started. Allowing general Python here would remove the assumption that makes this small harness appropriate.

Give every candidate a new execution directory

For an accepted comparison, check_candidate() writes candidate.py and check.py into a new temporary directory. The candidate contains a fixed input guard and the selected comparison. The checker imports that candidate and evaluates the task pairs against a in range(m), a separately written expression of the specification.

A new workspace prevents a candidate from inheriting files written by the previous candidate. test_workspace_is_removed_after_the_candidate also checks cleanup. The experiment does not depend on a prior repair having left behind a passing module or result file.

The child starts with python -I, an explicit working directory and a minimal supplied environment. Isolated Python mode reduces interpreter-path and environment contamination; it is not an operating-system sandbox. The checker intentionally adds the fresh candidate directory to its import path.

The process call has a timeout. A timed-out check returns timeout instead of a passing score. Python kills and waits for this child when that timeout expires; process creation itself can still add time. This is a narrow one-child task, not a general process-tree cancellation implementation.

Read the independent outcome and the process exit together

attempt <= max_attempts: failed (4/6)
attempt < max_attempts: passed (6/6)
__import__("os").getcwd(): rejected_patch (0/6)
Fresh temporary workspace per candidate. This restricted task is not a general OS sandbox.

The unchanged comparison fails the equality cases. The strict comparison passes the supplied checks. The import-shaped expression is rejected without behavioral execution. These outcomes locate different problems: a supported patch can be wrong, while an unsupported patch should never reach the checker.

The child exits successfully only when every check passes. Its JSON reports the number of successful checks; the parent records both that result and the exit status. An unexpected exit code indicates a checker error; it does not show that the submitted change failed the intended task in a meaningful way.

The parent also rejects excessively long captured stdout after the child returns. That is an acceptance limit on this fixed checker’s result, not a streaming memory cap. This implementation does not provide CPU quotas, filesystem permissions, network isolation or safe execution of arbitrary code. Its small accepted grammar is essential to its scope.

test_timeout_is_an_incomplete_trial injects the subprocess timeout exception and verifies reporting. This exercises the error-handling branch; it does not claim that a deliberately hanging candidate was executed. The valid and unchanged comparisons are actually executed by their behavioral tests.

Expand the task only with a matching execution boundary

A general coding agent needs a substantially different environment: repository access rules, immutable grader material, dependency policy, process-tree control and an appropriate sandbox. Reusing this function for arbitrary code would exceed its design.

Design exercise: add a boundary pair that distinguishes the unchanged comparison from the intended fix. Then propose a patch outside the accepted grammar and explain why it should be rejected even if it could print a passing result. Name the new execution controls needed before accepting arbitrary Python.

The independent edit benchmark will reuse this harness to compare actual behavior with completion claims.

Failure cases

The unchanged off-by-one expression must fail the real boundary checks. An expression outside the syntax contract refuses before execution. Timeout remains incomplete even if a candidate wrote something first; cleanup follows every attempt.

Deployment

Keep this harness attached to its narrow comparison-expression task. Before evaluating arbitrary repository edits, define permissions, isolated execution and process-tree supervision appropriate to that code. The following coding-loop lesson demonstrates a richer tool interface with a trusted disposable sample; it does not turn this expression runner into a general sandbox.

Scaling

Parallel attempts require separate workspaces, bounded child processes and retained results. Measure process startup and checker cost before increasing concurrency. A candidate count is a workload measure here; it is not a language model’s coding success rate.

Key takeaways

• The candidate contract is checked before execution.
• Behavioral checks decide whether the repair works.
• Timeout and cleanup are harness responsibilities.

Best practices

• Keep the checker independent of the submitted completion claim.
• Create a new workspace for each attempt.
• Reject unbounded execution settings.

References

• Python, ast: parsing and inspecting expression trees.
• Python, isolated mode: interpreter isolation behavior.
• Python, subprocess.run: captured output, return codes and timeout behavior.

References checked October 24, 2026. The lab executes only the restricted local comparisons described above.