Record a RAG baseline

A useful RAG baseline saves the questions, retrieved passages and answers before you change the application. Here we run the existing documentation pipeline on 8 frozen questions. One local recording produced 5 automatic passes, 2 failures and 1 incomplete response. Those are response-check outcomes, not an overall answer-quality score.

When you need this. Record a baseline before changing a retriever, prompt or model, so you can compare the same questions and inspect which stage changed. A single passing example cannot tell you whether the application retrieves the right version, handles missing information or returns a usable answer.

Bring the fixed RAG pipeline and frozen question dataset from the preceding lessons. This lab measures the same in-memory pipeline; it does not rebuild retrieval or retest PostgreSQL. Persistent index updates and deletion remain in the published PGVectorStore lifecycle lesson.

The successful example in How RAG works is not the score of this frozen case set. Here the UTF-8 writing question misses path-write and the model declines to answer. Keep this as an unresolved task when comparing later retrieval changes. Finding a better passage is a separate check from generating an answer that correctly uses it.

Freeze questions and run settings

A baseline is a recorded reference run of one application configuration on named inputs. Keep the corpus, passage IDs, embedding model, answer model, prompt and expected handling fixed. Otherwise a score difference may come from changed questions or source text rather than a changed application.

The dataset has 4 development and 4 public held-out questions. The two version-specific questions stay together in the held-out split. These are small, authored teaching sets, not unseen benchmark data. No settings were changed between the two splits in this recording.

The implementation retains the preceding pipeline’s splitter, cosine vector store, top-two selection, prompt, reply type and source-membership checks. The RAG adapter passes only the question and retrieved text into that prompt. Expected statuses, relevant-passage labels and version tags stay in the evaluator. Sending those labels to the model would change the task you are measuring.

{
  "splits": [
    "development",
    "held_out"
  ],
  "trials": 1,
  "case_set_sha256": "1603dde3e9e4d3fbafb14126e5c002584c6f945c29968488b01406618a6035b9",
  "config": {
    "embedding_model": "qwen3-embedding:0.6b",
    "answer_model": "gemma4:e2b",
    "query_instruction": "Retrieve Python documentation that answers the developer question",
    "k": 2,
    "splitter": {
      "name": "RecursiveCharacterTextSplitter",
      "chunk_size": 280,
      "chunk_overlap": 40
    },
    "temperature": 0,
    "num_predict": 400,
    "structured_output_method": "json_schema",
    "trials": 1,
    "embedding_http_timeout_seconds": 120,
    "generation_http_timeout_seconds": 180,
    "whole_child_timeout_seconds": 1200,
    "automatic_retries": 0,
    "judge_calls": 0
  },
  "model_digests": {
    "qwen3-embedding:0.6b": "ac6da0dfba84a81fdbfbaf330198c33cd77c4cdfc53e8bc50eb581914a15621d",
    "gemma4:e2b": "b37049369adfe3d2b653af0ab301a062ca5cbe96ace6aa0d3f2559ee9b563fc2"
  }
}

The tested local defaults are qwen3-embedding:0.6b for embeddings and gemma4:e2b for answers, pinned by the recorded Ollama digests. Both publisher model cards identify Apache 2.0 licensing. Other models are not selectable in this frozen run: changing the embedding model requires a matching index, and changing the answer model creates a separate baseline. No model weights are included in the download.

Changed source or saved result. The offline replay checks the source-file and result hashes before grading. A deliberate edited-result test refuses the changed file rather than filling a missing answer with a fresh generation. Keep the original run and make a separate, named experiment when inputs change.

Record each question once

Download and extract the complete lesson ZIP, then set up the pinned environment from its root folder. Planning, tests and saved replay do not need a running model or API key.

cd record-a-rag-baseline
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt

The downloadable lab reuses the existing Evals recorder and replay system. PipelineSystem.respond supplies the application’s question-level boundary: embed the question, call the unchanged pipeline, and retain intermediate events plus the raw generated message. The recorder catches a case error and saves it, so one unusable answer does not erase the other observations.

Run the offline plan before using the local models:

python baseline.py plan --split development --split held_out --include-held-out
python -m unittest -v test_dataset.py test_baseline.py

The plan defaults to development only unless the held-out split is explicitly requested. The named model digests must already exist locally. The adapter does not download weights or fall back to a hosted provider. Review the complete lab, then record into a new directory; existing output folders are refused.

python baseline.py record --fresh-local \
--reviewed-code-sha256 REVIEWED_CODE_HASH \
--split development --split held_out --include-held-out --trials 1 \
--out runs/NEW_UTC_DIRECTORY

{
  "status": "recorded; human semantic review pending",
  "observed_counts": {
    "documents": 1,
    "queries": {
      "utf8-write": 1,
      "replace-file": 1,
      "json-result-type": 1,
      "backup-schedule": 1,
      "read-newline-312": 1,
      "read-newline-313": 1,
      "json-throughput": 1,
      "create-parents": 1
    },
    "generation": {
      "utf8-write": 1,
      "replace-file": 1,
      "json-result-type": 1,
      "backup-schedule": 1,
      "read-newline-312": 1,
      "read-newline-313": 1,
      "json-throughput": 1,
      "create-parents": 1
    },
    "metadata": 1
  }
}

Replace REVIEWED_CODE_HASH with code_bundle_sha256 from the plan after reviewing the lab. The saved baseline made 17 inference requests: one document-embedding batch, then one query embedding and one answer attempt per question.

Second generation attempt. The call guard reserves each case’s attempt before sending the request and refuses another generation for that case. Invalid JSON is retained; there is no repair prompt, silent retry or judge-model call. A parent wall-time limit bounds the child process, while individual embedding and generation requests have their own timeouts. These operational limits do not establish general model reliability.

"""A bounded task-specific Ollama adapter. Imported clients are used only after review."""
from __future__ import annotations
from contextlib import contextmanager
from datetime import datetime,timezone
import json
import socket
import time
import httpx
from baseline_lib import CONFIG,EMBED,MODEL,MODEL_DIGESTS,Reply,query_text,digest

BASE='http://127.0.0.1:11434'


class LimitError(RuntimeError):pass


class CallGuard:
 def __init__(self,plan,passages,save=lambda:None):
  self.plan=plan;self.passages=passages;self.save=save
  self.stage=None;self.case_id=None;self.expected_input=None;self.expected_messages=None
  self.counts={'documents':0,'queries':{},'generation':{},'metadata':0}
  self.http=[];self.network=[]
 def expect(self,stage,case_id=None,text=None,messages=None):
  self.stage,self.case_id,self.expected_input,self.expected_messages=stage,case_id,text,messages
 def before(self,request):
  if request.url.scheme!='http' or request.url.host not in ('127.0.0.1','localhost','::1') or request.url.port!=11434 or request.url.query:
   raise LimitError('Only existing Ollama loopback11434 is allowed')
  body=json.loads(request.content)if request.content else None
  path=request.url.path
  if path in ('/api/embed','/api/chat'):
   total=self.counts['documents']+sum(self.counts['queries'].values())+sum(self.counts['generation'].values())
   if total>=self.plan['max_inference_http_calls']:raise LimitError('Total inference call cap')
  if (request.method,path)in (('GET','/api/tags'),('POST','/api/show')):
   if self.counts['metadata']>=3:raise LimitError('Metadata call cap')
   if path=='/api/show' and body.get('model',body.get('name'))not in (EMBED,MODEL):raise LimitError('Wrong metadata model')
   self.counts['metadata']+=1
  elif request.method=='POST' and path=='/api/embed':
   if body.get('model')!=EMBED:raise LimitError('Unexpected embedding model')
   inp=body.get('input');actual=inp if isinstance(inp,list)else[inp]
   if self.stage=='documents':
    if self.counts['documents'] or actual!=self.passages:raise LimitError('One exact four-passage batch only')
    self.counts['documents']+=1
   elif self.stage=='query':
    if self.case_id not in self.plan['case_ids'] or self.case_id in self.counts['queries'] or actual!=[self.expected_input]:raise LimitError('One exact query embedding per planned case')
    self.counts['queries'][self.case_id]=1
   else:raise LimitError('Embedding outside reviewed phase')
  elif request.method=='POST' and path=='/api/chat':
   if set(body)-{'model','messages','options','stream','format','tools'}:raise LimitError('Unreviewed generation fields')
   if self.stage!='generation' or self.case_id not in self.plan['case_ids'] or self.case_id in self.counts['generation']:raise LimitError('One generation attempt per planned case')
   if body.get('model')!=MODEL or body.get('tools') or body.get('options')!={'temperature':0,'num_predict':400}:raise LimitError('Generation options/tools/model changed')
   expected=[('user' if role=='human'else role,text)for role,text in self.expected_messages]
   actual=[(m.get('role'),m.get('content'))for m in body.get('messages',[])]
   if actual!=expected or any(m.get('images')or m.get('tool_calls')for m in body['messages']):raise LimitError('Exact input prompt changed')
   if body.get('format')!=Reply.model_json_schema():raise LimitError('Reply schema changed')
   self.counts['generation'][self.case_id]=1
  else:raise LimitError('Unapproved endpoint; no pulls, downloads, generate fallback or providers')
  total=self.counts['documents']+sum(self.counts['queries'].values())+sum(self.counts['generation'].values())
  if total>self.plan['max_inference_http_calls']:raise LimitError('Total inference call cap')
  row={'at_utc':datetime.now(timezone.utc).isoformat(),'stage':self.stage,'case_id':self.case_id,
   'method':request.method,'url':str(request.url),'body':body,'headers_saved':False}
  self.http.append(row);self.save();return row

 @contextmanager
 def installed(self):
  old_send,old_asend=httpx.Client.send,httpx.AsyncClient.send
  old_connect,old_connect_ex,old_dns=socket.socket.connect,socket.socket.connect_ex,socket.getaddrinfo
  def check(address):
   allowed=isinstance(address,tuple)and len(address)>1 and address[0]in('127.0.0.1','localhost','::1')and address[1]==11434
   self.network.append({'allowed':allowed,'address':str(address)});self.save()
   if not allowed:raise LimitError('Non-Ollama socket blocked')
  def connect(sock,address):check(address);return old_connect(sock,address)
  def connect_ex(sock,address):check(address);return old_connect_ex(sock,address)
  def dns(host,port,*a,**kw):
   if host not in ('127.0.0.1','localhost','::1',b'127.0.0.1',b'localhost',b'::1') or port!=11434:
    self.network.append({'allowed':False,'dns_host':str(host),'port':str(port)});self.save();raise LimitError('External DNS blocked')
   return old_dns(host,port,*a,**kw)
  def send(client,request,*a,**kw):
   request.read();row=self.before(request);start=time.monotonic()
   try:
    response=old_send(client,request,*a,**kw);response.read()
    row.update({'status':response.status_code,'response_text':response.text,'elapsed_seconds':time.monotonic()-start});return response
   except Exception as exc:row['error']={'type':type(exc).__name__,'message':str(exc)};raise
   finally:self.save()
  async def asend(client,request,*a,**kw):
   await request.aread();row=self.before(request);start=time.monotonic()
   try:
    response=await old_asend(client,request,*a,**kw);await response.aread()
    row.update({'status':response.status_code,'response_text':response.text,'elapsed_seconds':time.monotonic()-start});return response
   except Exception as exc:row['error']={'type':type(exc).__name__,'message':str(exc)};raise
   finally:self.save()
  httpx.Client.send,httpx.AsyncClient.send=send,asend
  socket.socket.connect,socket.socket.connect_ex,socket.getaddrinfo=connect,connect_ex,dns
  try:yield
  finally:
   httpx.Client.send,httpx.AsyncClient.send=old_send,old_asend
   socket.socket.connect,socket.socket.connect_ex,socket.getaddrinfo=old_connect,old_connect_ex,old_dns


class FreshModels:
 mode='fresh_local_inference'
 def __init__(self,guard):
  self.guard=guard
  with httpx.Client(trust_env=False,follow_redirects=False,timeout=15)as c:tags=c.get(BASE+'/api/tags').json()
  found={m.get('name',m.get('model')):m.get('digest')for m in tags.get('models',[])}
  if any(found.get(k)!=v for k,v in MODEL_DIGESTS.items()):raise LimitError('Installed model missing/digest drift; no pull or fallback')
  self.models=dict(MODEL_DIGESTS)
  from langchain_ollama import ChatOllama,OllamaEmbeddings
  self.embedder=OllamaEmbeddings(model=EMBED,base_url=BASE,client_kwargs={'timeout':120,'trust_env':False,'follow_redirects':False})
  self.model=ChatOllama(model=MODEL,base_url=BASE,temperature=0,num_predict=400,
   client_kwargs={'timeout':180,'trust_env':False,'follow_redirects':False}).with_structured_output(Reply,include_raw=True)
 def documents(self,texts):
  self.guard.expect('documents');return self.embedder.embed_documents(texts)
 def query(self,case_id,question):
  text=query_text(question);self.guard.expect('query',case_id,text=text);return self.embedder.embed_query(text)
 def generate(self,case_id,question,messages):
  self.guard.expect('generation',case_id,messages=messages)
  value=self.model.invoke(messages)
  return {'raw':value['raw'].model_dump(mode='json',exclude_none=True),'parsed':None if value['parsed']is None else value['parsed'].model_dump(),
   'parsing_error':None if value['parsing_error']is None else str(value['parsing_error'])}

Measure retrieval against versioned targets

Hit@2 asks whether either of the two retrieved chunks is a designated relevant target. Reciprocal rank is one divided by the first relevant rank: a target first gives one, second gives one half, and absent gives zero. The reused Evals helper compares exact chunk IDs, so the 3.13 source is not a positive hit for the 3.12 target.

Each supported case has one annotated target, so recall at two equals Hit@2 here. Reporting both as independent improvements would count the same event twice. Unsupported cases have no positive target and receive null retrieval metrics. A failure before retrieval would also be unmeasured, rather than silently counted as a correct result; supported denominators remain visible.

Wrong version looks relevant. The read-newline-312 target was retrieved second, behind the 3.13 passage. Its reciprocal rank is lower than a first-place hit even though both passages discuss Path.read_text. The question’s requested version decides the target, not whichever method name matches.

The UTF-8 writing question retrieved the two reading summaries and missed path-write. The model then declined to answer. That refusal is reasonable for its selected context, but the end-to-end task still failed: the corpus contains the requested writing fact. A baseline lets you locate this retrieval miss before proposing a prompt change. This lesson preserves the miss; it does not tune against these held-out outcomes or rerun the model to replace it.

{
  "case_id": "utf8-write",
  "retrieved_chunk_ids": [
    "path-read-3-13:3f86296ba6f04cc2",
    "path-read:1f1129b8d1011a82"
  ],
  "annotated_target_chunk_ids": [
    "path-write:b71a2d18a53d86bf"
  ],
  "retrieval": {
    "hit": false,
    "recall": 0,
    "reciprocal_rank": 0
  },
  "returned_status": "not_in_docs"
}

Separate answer checks from answer quality

The task-specific graders check the reply shape, expected status and designated source alignment. They reuse the original reply validation; the generic Evals runner remains unchanged. A real source ID is only a valid reference. It does not establish whether the factual claim follows from the passage.

"""Task-specific automatic answer contracts; no faithfulness grader or LLM judge."""
from pathlib import Path
import sys
sys.path.insert(0,str(Path(__file__).parent/'src'))
from langchain_core.documents import Document
from agentic_rag.concepts import Reply,check_reply
from eval_task import validate_expected
NAME='rag-baseline-contracts'
VERSION='1'
OUTPUT_FIELDS={'status':str}
def reply_contract(case,output):
 try:
  reply=Reply.model_validate({k:output[k]for k in ('status','answer','source_ids')})
  selected=[Document(page_content='',metadata={'id':id})for id in output['context_ids']]
  check_reply(reply,selected)
 except (ValueError,KeyError,TypeError)as exc:return False,str(exc)
 return True,''
def expected_status(case,output):
 return output.get('status')==case['expected']['status'],'Reply status differs from the annotation.'
def target_source(case,output):
 if case['expected']['status']=='not_in_docs':return True,''
 targets={t['source_id']for t in case['expected']['targets']}
 return bool(targets&set(output.get('source_ids',[]))),'No designated target source is linked; this is alignment, not faithfulness.'
GRADERS=({'name':'reply-contract','version':'1','kind':'contract','check':reply_contract},
 {'name':'expected-status','version':'1','kind':'outcome','check':expected_status},
 {'name':'target-source-alignment','version':'1','kind':'contract','check':target_source})

A valid source can support the wrong claim. An offline test supplies a legal path-write ID beside the false statement that write_text appends. The automatic checks pass that controlled reply. The human rubric still needs to compare the sentence with the passage, which says existing contents are overwritten. This test uses a fake response, not a model result, to demonstrate the limit of ID and schema checks.

The 3.12 newline case shows why the layers must stay separate. Its final sentence correctly says that the parameter was added in 3.13 and is unavailable in 3.12. Yet it returns not_in_docs with no source IDs. A supported negative answer should return answered and identify the relevant source. Correct factual wording does not fix the wrong structured status or missing link.

{
  "question": "Can Path.read_text take a newline argument in Python 3.12?",
  "context_ids": [
    "path-read-3-13",
    "path-read"
  ],
  "status": "not_in_docs",
  "answer": "The provided documentation states that the `newline` parameter was added in Python 3.13, meaning it is not available in Python 3.12 according to this context.",
  "source_ids": [],
  "source_links": []
}

Complete human-review.json using the supplied rubric: does each answer state the right fact for the requested version and stay within the corpus? The saved automatic checks leave these human judgments unfilled.

Keep incomplete responses and reported usage

The parent-directory question returned no final answer. Ollama stopped with done_reason="length"; the adapter retained empty final content and a structured-output parsing error. Its recorded output-token field reached the requested limit. Do not convert that missing response into not_in_docs, because the application did not actually deliver that status.

{
  "case_id": "create-parents",
  "final_content": "",
  "parsed": null,
  "parsing_error": "Invalid json output: \nFor troubleshooting, visit: https://docs.langchain.com/oss/python/langchain/errors/OUTPUT_PARSING_FAILURE ",
  "response_metadata": {
    "model": "gemma4:e2b",
    "created_at": "2026-10-23T00:21:21.724875Z",
    "done": true,
    "done_reason": "length",
    "total_duration": 4881118750,
    "load_duration": 228535416,
    "prompt_eval_count": 227,
    "prompt_eval_duration": 194719000,
    "eval_count": 400,
    "eval_duration": 4415164000,
    "logprobs": null,
    "model_name": "gemma4:e2b",
    "model_provider": "ollama"
  },
  "usage_metadata": {
    "input_tokens": 227,
    "output_tokens": 400,
    "total_tokens": 627
  }
}

The full request/response ledger also retains streamed reasoning fields. The installed adapter’s final content field does not include that text. No separate reasoning-token count was supplied, so it stays missing rather than becoming zero. Use the provider’s reported fields as recorded; do not infer an exact breakdown of all generated reasoning or compute from the final-answer count.

Per-question elapsed time comes from recording.json, measured around fresh query embedding and the pipeline call. The replay report has much shorter grading times because it reads saved responses. Using those replay times as model latency would describe a different operation. Document embedding and index construction are recorded separately; one run does not establish latency percentiles or capacity.

Replay the saved run without another model call

Replaying means loading the recorded responses and running the same graders again. The file hashes bind that recording to its source, corpus, question set and rubric. It does not regenerate missing answers or validate a changed prompt. Keep the original record when experimenting with a fix.

Use the archived source command below when replaying the saved baseline. Its file checks expect the source bundle stored with that recording:

python runs/20261023T002037.167760Z-fixed-eight-local-baseline/source/baseline.py replay \
--record runs/20261023T002037.167760Z-fixed-eight-local-baseline

Incomplete and unsupported are different. The output table keeps the length-stopped case incomplete, while unsupported questions that actually returned not_in_docs remain ordinary responses. Unsupported retrieval metrics stay null. The offline tests cover that distinction, wrong-version targets, duplicate generation attempts and modified replay files. The saved adapter log contains 45 offline tests. The current command also checks the archived source binding and exact saved replay. These tests use controlled fakes or saved responses and do not establish model answer quality.

To interpret a future change, keep the questions and rubric fixed, record a separate run and compare retrieval, response checks and answer support separately. If the corpus or targets change, name that dataset revision before comparing results. The published Evals course develops those comparison and release practices; it is not reimplemented here.

What’s next

Continue with Load Developer Documentation.

References

Implementation and model references checked on 22 October 2026: InMemoryVectorStore, ChatOllama, OllamaEmbeddings, Qwen3-Embedding-0.6B and Gemma 4 E2B. The Python source summaries remain those of the frozen dataset, including the documented-only 3.13 addition. No Python 3.13 execution, PostgreSQL rerun or broader benchmark result is claimed by this recording.