Rank documents with embeddings

You embed four docs pages and rank them against questions. In the recorded ranking, “how do I undo a bad deploy” puts rollback.md first; unrelated questions still receive scores.

Putting every docs page in every prompt works for four pages; a real docs site has hundreds, and they will not fit in the context window. The assistant needs to find the one or two pages that answer the question, even when the question uses different words from the page. Here you embed the current docuctl pages, rank them against questions by cosine similarity, and record where the ranking misleads.

When you need this. A developer asks the docs assistant “how do I undo a bad deploy”, and the page that answers it is titled “Roll back a release”. A keyword search can miss alternate wording. Embedding similarity provides another ranking signal; the results still need evaluation.

Recorded example. The saved vectors and rankings correspond to shipctl source text; embedding the current docuctl pages is a new run.

1. Embed the pages and the questions

An embedding model turns a text into a fixed-length list of numbers, a vector, placed so that texts with similar meaning get vectors that point in similar directions. It is a different kind of model from the chat model: it produces no text, only vectors.

Download the vectors-and-similarity lab and extract its code, docs, tests and README together.

unzip vectors-and-similarity.zip
cd vectors-and-similarity
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt

# Offline tests: no model calls
.venv/bin/python -m pytest -q

# Run after reading the implementation below
ollama pull qwen3-embedding:0.6b
.venv/bin/python rank_pages.py

The tests use saved vectors and replace embedding calls. Running rank_pages.py requests fresh embeddings and writes a local vector file.

"""Embed the docuctl docs pages and rank them against questions by cosine similarity. Run: python rank_pages.py"""
import json
from pathlib import Path

import numpy as np
import ollama

MODEL = "qwen3-embedding:0.6b"
HERE = Path(__file__).parent
PAGES = {page.name: page.read_text() for page in sorted((HERE / "docs").glob("*.md"))}
QUERIES = ["how do I undo a bad deploy", "where should the CI credential live", "what replaced release revert"]


def embed(client: ollama.Client, texts: list[str], model: str = MODEL) -> np.ndarray:
    return np.array(client.embed(model=model, input=texts).embeddings)


def cosine(query: np.ndarray, matrix: np.ndarray) -> np.ndarray:
    """Cosine similarity of one vector against each row: dot product of unit-length vectors."""
    return matrix @ query / (np.linalg.norm(matrix, axis=1) * np.linalg.norm(query))


def rank(query_vector: np.ndarray, page_vectors: np.ndarray, names: list[str]) -> list[tuple[str, float]]:
    scores = cosine(query_vector, page_vectors)
    return sorted(zip(names, scores.round(3).tolist()), key=lambda item: -item[1])


def main() -> None:
    client = ollama.Client()
    names = list(PAGES)
    pages = embed(client, list(PAGES.values()))
    queries = embed(client, QUERIES)
    print(f"{len(names)} pages, {pages.shape[1]} dimensions per vector")
    results = {}
    for query, vector in zip(QUERIES, queries):
        results[query] = rank(vector, pages, names)
        print(f"\n{query!r}")
        for name, score in results[query]:
            print(f"  {score:.3f}  {name}")
    output = HERE / "runs/local-docuctl"
    output.mkdir(parents=True, exist_ok=True)
    (output / "rankings.json").write_text(json.dumps({"model": MODEL, "rankings": results}, indent=1) + "\n")
    np.save(output / "page-vectors.npy", pages)


if __name__ == "__main__":
    main()

embed() sends a batch of texts to Ollama’s embed endpoint and returns one row per text: here, 1024 numbers each. The pages are embedded once and saved as page-vectors.npy, as an index would store them; each question is embedded when it arrives. Fresh files go under runs/local-docuctl/; the root files remain the original recording.

This small example embeds the question without a task prefix. The Qwen model card recommends a retrieval instruction on queries, while documents need none. Adding that instruction is a new embedding configuration and needs a separate comparison; the later RAG embedding pipeline uses that format.

2. Rank by cosine similarity

Cosine similarity is the cosine of the angle between two vectors: the dot product divided by the product of their lengths. It is 1 for vectors pointing the same way, 0 for perpendicular vectors, and it ignores how long the vectors are. cosine() computes it for the query against every page row at once.

4 pages, 1024 dimensions per vector

'how do I undo a bad deploy'
  0.592  rollback.md
  0.536  deploy.md
  0.429  tokens.md
  0.394  config.md

'where should the CI credential live'
  0.589  tokens.md
  0.477  config.md
  0.412  deploy.md
  0.333  rollback.md

'what replaced release revert'
  0.615  rollback.md
  0.427  deploy.md
  0.330  config.md
  0.318  tokens.md

All three questions rank the right page first. The question uses “undo” and “deploy” rather than the command name “rollback,” yet rollback.md ranks first. In the original recording, “what replaced release revert” finds the sentence about shipctl 1.x.

Here is the recorded run of failures.py:

"""Two ways similarity search misleads you. Run: python failures.py (after rank_pages.py)"""
import numpy as np
import ollama

from rank_pages import HERE, PAGES, embed, rank

client = ollama.Client()
pages = np.load(HERE / "page-vectors.npy")
names = list(PAGES)

# 1. The query must be embedded exactly as the pages were: same model, same settings.
try:
    other = np.array(client.embed(model="qwen3-embedding:0.6b", input=["how do I undo a bad deploy"], dimensions=512).embeddings[0])
    print(f"1. query embedded with dimensions=512: {other.shape[0]} values, pages have {pages.shape[1]}")
    rank(other, pages, names)
except ValueError as error:
    print("   ->", type(error).__name__, str(error).splitlines()[0][:100])
try:
    client.embed(model="qwen3.5:0.8b", input=["how do I undo a bad deploy"])
except ollama.ResponseError as error:
    print("1b. a chat model as the embedder ->", error.status_code, error.error)

# 2. Every query gets a top page, even one the docs cannot answer.
for query in ["what is the capital of France", "how do I pin a Python dependency"]:
    best = rank(embed(client, [query])[0], pages, names)[0]
    print(f"2. {query!r} -> top page {best[0]} at {best[1]:.3f}")

1. query embedded with dimensions=512: 512 values, pages have 1024
   -> ValueError matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)-
1b. a chat model as the embedder -> 501 This server does not support embeddings. Start it with `--embeddings`
2. 'what is the capital of France' -> top page config.md at 0.182
2. 'how do I pin a Python dependency' -> top page config.md at 0.383

The query was embedded differently

qwen3-embedding:0.6b can return shorter vectors with dimensions. A query embedded at 512 dimensions against pages stored at 1024 fails with a NumPy shape error, which is the lucky case: a different model with the same vector length could pass this shape check while producing scores that were not calibrated for these page vectors. A chat model is not an embedder at all; Ollama returned HTTP status 501 rather than an embedding. Store the embedding model name and settings next to the index, check them on every query, and re-embed every page when either changes.

3. Read the scores

The top score is around 0.6, not 1. A short question and a whole page need not point in the same direction. What matters is the order and the gap: rollback.md leads deploy.md by a clear margin for the first question, while the bottom pages are close together. Use the scores to rank candidates for this question. They are not calibrated relevance probabilities, and thresholds do not automatically transfer to other questions, corpora or models.

Every query gets a top page

“what is the capital of France” still returned config.md as its top page. “how do I pin a Python dependency”, which the docs do not cover, returned config.md with a score of 0.383, close to the scores of the lower-ranked real pages. This lab sorts a nonempty page set and takes its highest scores without an abstention rule. Do not send the top page to the model unconditionally: set a minimum score from labelled examples, keep the top few rather than one, and let the assistant say that the docs do not cover the question.

Compare vectors in the same embedding space

The embedding model was trained so that related texts end up close together: a question and the passage that answers it, two paraphrases, a title and its page. After training it maps any text to a point in a 1024-dimensional space, and closeness in that space stands in for relatedness. The mapping is a property of the model’s weights, which is why pages and queries must go through the same model.

Comparing a query with every page, as cosine() does, is exact and fine for thousands of vectors. Large collections use a vector store with an approximate nearest-neighbor index, such as HNSW in pgvector, which skips most comparisons. Cosine similarity still measures the angle between a returned vector and the query. An approximate index can miss a nearer vector, so its returned top neighbors need not match this exhaustive calculation. Compare returned IDs with exact search to measure that loss; neither ranking by itself establishes answer relevance.

Try it: choose a minimum score

Write ten questions: seven that the four pages answer and three that they do not. Rank them all and record the top score for each. Find the minimum score that keeps all seven answerable questions and rejects the three others, if one exists, and note how close the two groups come. The offline tests use the recorded vectors and rankings: .venv/bin/python -m pytest -q.

Validate a model-proposed tool call lets the model ask for the page itself, through a search tool and a read tool that the application runs.

References

Checked October 21, 2026.

• Ollama: embeddings
• Ollama API: embed
• Qwen3-Embedding-0.6B model card
• NumPy: linalg.norm