Apply the same documentation-version filter to lexical and vector retrieval, then inspect the eligible chunk IDs and actual results. The controlled comparison shows how filtering after a result limit can discard the only returned hit.
When you need this. Use it when every retriever must honor the same document-version constraint before choosing its bounded results.
Prerequisites. Use the versioned corpus from the preceding module. The vector course supplies the embedding and vector-search prerequisites. Install the lab dependencies from requirements.lock in a virtual environment.
Define one metadata policy
The supported policy is version with a $eq operator and a numeric major/minor string. None or an empty object means all versions. A valid version absent from the corpus has no eligible candidates. This is a source-selection rule, not an authorization boundary.
Both adapters validate the same versioned source snapshot and derive metadata from it. The recorded candidate_chunk_ids lists show which chunks satisfy the policy before ranking. Lexical matches and nearest vector results may still differ within that same eligible set.
Unsupported policy. An unknown field, operator or invalid value raises ValueError. For example, $ne is not implemented. Refusing it is safer than letting one retriever silently ignore a constraint that the other enforces.
Put the filter before the limit in each adapter
The lexical adapter adds a parameterized version condition to its SQL WHERE, then orders native BM25 scores and applies LIMIT. The FTS5 index still contains the complete corpus, so its corpus statistics remain those of that index. Filtering candidates does not rebuild the index or recompute a smaller corpus.
The vector adapter supplies a version callback to the actual LangChain store. In the pinned implementation the callback removes ineligible documents before cosine scoring. The application then sorts eligible results with a stable chunk-ID tie-breaker and takes the requested count.
The lexical query uses the term encoding; vector search uses the full recorded question. Compare their metadata eligibility and candidate membership, rather than expecting identical rankings.
Compare prefiltering with discarding limited results
The controlled comparison asks for the documentation version that the unfiltered first result does not have. Selecting that first result and discarding it afterwards leaves an empty list. Applying the policy first lets each retriever select its eligible passage instead.
Post-limit underfill. The recorded comparison shows the discarded result and the actual prefiltered result for each adapter. A later filter cannot recover candidates excluded by the earlier limit. Supply the constraint to the retriever before its result limit.
This does not promise that every eligible set contains a match. A lexical query may match no eligible terms. A vector query over an empty eligible set returns no candidates. The adapter still refuses an unrecorded question even when the version would have no candidates, preserving its exact replay-input contract.
Keep score units and absence states separate
SQLite’s native BM25 values sort ascending; cosine similarity sorts descending. Do not add them together or treat equal-looking numbers as equal relevance. Hybrid combination belongs to the next module.
Invalid lexical query. The lexical interface accepts a single ASCII alphanumeric token and quotes it as an FTS literal. Invalid syntax raises ValueError, while a valid missing term yields no matching rows. A missing version is an eligible-set result, not an invalid-operator exception.
All vector lookups reuse the development-only replay from the preceding lesson. No new embeddings, generated answers, held-out runs or model-quality measurements are involved.
Execute the local retrieval contract
Download the complete lab. It includes the README, fixtures, implementation, tests, dependency information and actual run record.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.lock"""One version equality policy applied before either retriever selects top-k."""
import re
import sqlite3
from vector_retriever import rows_from_snapshot, version_filter
class LexicalRetriever:
def __init__(self,snapshot):
self.rows=rows_from_snapshot(snapshot)
self.db=sqlite3.connect(':memory:')
self.db.execute('CREATE VIRTUAL TABLE passages USING fts5(chunk_id UNINDEXED,source_id UNINDEXED,version UNINDEXED,url UNINDEXED,text, tokenize="unicode61")')
self.db.executemany('INSERT INTO passages VALUES (?,?,?,?,?)',[(r['chunk_id'],r['source_id'],r['version'],r['url'],r['text']) for r in self.rows])
def search(self,query,k=2,policy=None):
if type(k) is not int or k<1:raise ValueError('invalid_k')
version=version_filter(policy)
if not isinstance(query,str) or re.fullmatch(r'[A-Za-z0-9]{1,64}',query) is None:raise ValueError('invalid_lexical_query')
sql='SELECT chunk_id,source_id,version,url,text,bm25(passages) FROM passages WHERE passages MATCH ?'
params=['"'+query+'"']
if version is not None:
sql+=' AND version=?';params.append(version)
sql+=' ORDER BY bm25(passages),chunk_id LIMIT ?';params.append(k)
return [dict(zip(('chunk_id','source_id','version','url','text','native_score'),row)) for row in self.db.execute(sql,params)]
def close(self):self.db.close()
def candidates(rows,policy):
version=version_filter(policy)
return sorted(r['chunk_id'] for r in rows if version is None or r['version']==version)python run.py
python -m unittest -v test_vector_retriever.py test_filters.py{
"policy": {
"version": {
"$eq": "3.13"
}
},
"vector_postfilter": [],
"vector_prefilter": [
"path-read-3-13"
],
"lexical_postfilter": [],
"lexical_prefilter": [
"path-read-3-13"
],
"invalid_operator_refusal": "unsupported_filter_operator"
}Inspect the complete saved result
{
"execution_mode": "offline-recorded-vector-and-fts5-replay",
"model_called": false,
"steps": [
{
"policy": {
"version": {
"$eq": "3.12"
}
},
"candidate_chunk_ids": [
"c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c"
],
"vector_query": "How do I explicitly select UTF-8 when saving text with pathlib?",
"vector_results": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"score": 0.637539995393079
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"score": 0.5762678845903342
}
],
"lexical_query": "encoding",
"lexical_results": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -1.4220532319391636e-06
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -1.3203883495145633e-06
}
]
},
{
"policy": {
"version": {
"$eq": "3.13"
}
},
"candidate_chunk_ids": [
"bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35"
],
"vector_query": "How do I explicitly select UTF-8 when saving text with pathlib?",
"vector_results": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"score": 0.6510841890363557
}
],
"lexical_query": "encoding",
"lexical_results": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"native_score": -1.0374479889042996e-06
}
]
},
{
"policy": {
"version": {
"$eq": "9.99"
}
},
"candidate_chunk_ids": [],
"vector_query": "How do I explicitly select UTF-8 when saving text with pathlib?",
"vector_results": [],
"lexical_query": "encoding",
"lexical_results": []
}
],
"prefilter_comparison": {
"vector_query": "Will write_text replace the contents of an existing file?",
"lexical_query": "encoding",
"policy": {
"version": {
"$eq": "3.13"
}
},
"vector_unfiltered_top1": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"score": 0.6357498076122466
}
],
"vector_postfilter_top1": [],
"vector_prefilter_top1": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"score": 0.5565245149667724
}
],
"lexical_unfiltered_top1": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -1.4220532319391636e-06
}
],
"lexical_postfilter_top1": [],
"lexical_prefilter_top1": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"native_score": -1.0374479889042996e-06
}
]
},
"invalid_operator_refusal": "unsupported_filter_operator",
"policy": "None or empty object means all versions; only version $eq with a numeric major.minor string is supported. Unknown versions return no candidates."
}Compare the recorded behavior
Compare candidate IDs and actual hits for the recorded supported versions and the absent version. Inspect prefilter_comparison: both post-top-one lists are empty, while applying the version rule first returns the eligible passage. Try $ne in a scratch call and compare its explicit refusal with the valid empty-version result.
Read the explanation
Keep each adapter's input, filtering and score contract explicit; a retrieved passage is not an answer-validation result.
What’s next
Continue with Build Hybrid Retrieval.
References
Official SQLite FTS5 documentation, the LangChain InMemoryVectorStore reference and installed LangChain Core 1.7.0 source were checked on 23 October 2026. SQL filtering and the native callback are executed by the supplied adapters; their distinct score directions are retained.