Vector Databases

The embeddings lesson produced a vector for a piece of text and measured the distance between two of them. That works for three sentences held in memory. A handbook, a set of finance procedures and two years of internal notes come to tens of thousands of chunks, and comparing a question against every one of them on every request is not a plan.

What you need is a store that holds vectors, finds the closest few to a query vector quickly, and lets you restrict the search to rows the caller is allowed to see. That is the whole job description. "Vector database" makes it sound like a new category of infrastructure; for most teams it is a column and an index on a database they already run.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunk (
    chunk_id        text PRIMARY KEY,
    document_id     text NOT NULL,
    document_title  text NOT NULL,
    heading_path    text NOT NULL,
    chunk_text      text NOT NULL,
    source_url      text NOT NULL,
    page            integer,
    audience        text NOT NULL,
    effective_from  date NOT NULL,
    superseded      boolean NOT NULL DEFAULT false,
    embedding_model text NOT NULL,
    embedding       vector(1536) NOT NULL
);

CREATE INDEX document_chunk_embedding_idx
    ON document_chunk USING hnsw (embedding vector_cosine_ops);

CREATE INDEX document_chunk_audience_idx ON document_chunk (audience, superseded);

-- The closest current chunks the caller is allowed to read
SELECT chunk_id,
       document_title,
       heading_path,
       chunk_text,
       source_url,
       1 - (embedding <=> $1) AS similarity
FROM document_chunk
WHERE audience = ANY($2)
  AND superseded = false
  AND effective_from <= current_date
  AND embedding_model = $3
ORDER BY embedding <=> $1
LIMIT $4;

• vector(1536) fixes the dimension count in the column type, so a vector from the wrong model is rejected by the database rather than quietly stored. The number has to match your embedding model.
• The <=> operator is cosine distance in pgvector. Smaller means closer, so ORDER BY on it ascending puts the nearest chunk first. Subtracting from 1 converts distance into the similarity score the embeddings lesson used, which is the friendlier number to log.
• HNSW is an approximate index. It trades a small amount of recall for a very large speed gain, and approximate is the right trade here — you are retrieving candidate passages, not settling a payroll figure. Without any index the query still works, by comparing against every row, which is fine while you have a few thousand.
• The WHERE clause is doing security work, not tidying. audience and superseded are checked inside the query, so a chunk the caller may not read cannot reach the prompt. Filter in the search, never in the code that formats the results.
• Passing the embedding model as a parameter and matching on it means a query using the wrong model returns nothing instead of returning confident nonsense. A loud empty result is much easier to diagnose than a silently wrong ranking.
• One honest caveat about approximate indexes plus filters: the index searches for near neighbours first and your conditions are applied around that, so a very restrictive filter can return fewer rows than you asked for even when more exist. Check what your version does, and test with your narrowest realistic filter rather than an unfiltered query.

def search(question: str, caller: User, top_k: int = 5) -> list[Passage]:
    query_vector = embed(question)          # same model that built the index

    rows = db.fetch(
        SEARCH_SQL,
        query_vector,
        caller.audience_tags,               # what this person may read
        EMBEDDING_MODEL,
        top_k,
    )

    passages = [Passage.from_row(row) for row in rows]

    log.info(
        "retrieval",
        extra={
            "question_chars": len(question),
            "returned": len(passages),
            "top_similarity": passages[0].similarity if passages else None,
            "chunk_ids": [p.chunk_id for p in passages],
        },
    )

    return [p for p in passages if p.similarity >= MIN_SIMILARITY]

• embed uses the same model constant the index was built with, read from one place. A literal model name at each call site is how half an index ends up in a different coordinate space.
• caller.audience_tags goes into the query, so permissions are applied by the database. The model never sees a passage the person could not have opened themselves.
• The log line is the one thing people leave out and later wish they had. When an answer is wrong, the first question is always which chunks were supplied, and this line answers it without a reproduction.
• The relevance floor is applied after ranking, and it is what lets the pipeline say it does not know. top-k always returns k rows if k rows exist, however unrelated they are — a question about parking scores something against an expenses corpus. Filtering by score is what turns "the five least bad rows" into "nothing relevant here".
• Pick MIN_SIMILARITY from evidence, not from an article. Run fifty real questions, including some your corpus genuinely does not cover, and look at where the scores fall. The number is specific to your model and your documents.