Embeddings and Vector Search: Similarity, Dimensionality, and Model Choice; Storing Text in pgvector

Today's Goals

1. Explain what an embedding turns into coordinates, and why cosine similarity becomes equivalent to a dot product after normalization
2. List the four things you must check when choosing an embedding model: multilingual capability, dimensionality and storage cost, maximum input length, and whether it needs separate query and document prefixes
3. Create a table in pgvector, insert vectors, build an index, query nearest neighbors, and read whether the query plan actually used the index

Yesterday's BM25 goes by the letter: rephrase "the file upload limit" as "attachment size" and not one term matches. Today the system gets a second eye, turning similar in meaning into near in coordinates. Once you have read this and finished the lab, scroll back to the top and tick off the three goals.

Plain-Language Walkthrough

Coordinates on a map

Look at a map. Beijing is a pair of numbers on it and Shanghai is another pair, and whether the two cities are close needs no reading of names, only measuring the distance between two points. What makes a map powerful is that it turns the unsayable matter of geographic position into two numbers you can subtract. You can even ask which three cities are nearest to Hangzhou — a hard question given only a list of names, and merely a sort given coordinates.

An embedding does the same thing, except its coordinates are not two-dimensional but hundreds or thousands of dimensions, and what they measure is not geographic distance but nearness in meaning. "How long does the recycle bin keep things" and "can I get back a file I deleted" share almost no characters, and on this map of meaning they nearly overlap; "how long does the recycle bin keep things" and "which data center is the server in" sit far apart. Turning a passage into such a set of coordinates is embedding, and the resulting set of numbers is a vector.

Why so many dimensions? Because meaning is not a position on one line. A sentence carries topic, tone, tense, professional domain, whether it is a question, and much else at once, and two coordinates cannot hold that. 1536 dimensions means the model uses 1536 numbers to describe the sentence's many facets. Those dimensions have no names — nobody can say what dimension 37 stands for, because it was trained rather than designed. That matters: a vector is not explainable, and you cannot decompose a score back into which term contributed how much as you could yesterday. That is its cost against BM25 and why BM25 came first.

One more counter-intuitive thing to settle now: a vector encodes resemblance, not correctness. Two opposite sentences — "export to PDF is supported" and "export to PDF is not supported" — sit very close in vector space, because they discuss the same thing. Expecting vectors to distinguish affirmation from negation will fail. Their job is fetching topically relevant material, and judging correctness belongs to the later generation step.

Three distances, and normalization as their precondition

With coordinates comes measuring distance. Three are common, and pgvector gives each an operator.

Euclidean distance (<->) is the most intuitive: the straight line between two points. It is affected by both direction and length — two vectors pointing identically, one long and one short, still have a non-zero Euclidean distance.

The dot product (<#>) multiplies dimension by dimension and sums. It carries two layers at once: how aligned the directions are and how long each vector is. Length is trouble here: the longer the text, the larger the model's output vector norm tends to be, so ordering by dot product systematically favors long documents — exactly the problem BM25's b parameter handles, surfacing in a new place.

Cosine similarity (with distance operator <=>) looks only at direction: treat both vectors as arrows from the origin and measure the angle. Smaller angle, more similar. It is inherently unaffected by length, which makes it the default for text retrieval.

The relationship among the three fits in one sentence: once every vector is L2-normalized (its length squeezed to 1), cosine similarity equals the dot product, and Euclidean distance becomes a monotonic function of cosine distance. The reason is simple: cosine similarity is defined as the dot product divided by the product of the norms, and with both norms at 1 the divisor is 1. That is not a mathematical game but an engineering convention that eliminates a whole class of bugs — normalize once at the embedding's exit and every operator ranks identically thereafter, with no more worrying about whether a store defaults to dot product or cosine.

export function l2Normalize(vec) {
  const norm = Math.sqrt(vec.reduce((sum, x) => sum + x * x, 0))
  return norm === 0 ? vec : vec.map((x) => x / norm) // return a zero vector as-is; never divide by 0
}
 
export function dot(a, b) {
  let sum = 0
  for (let i = 0; i < a.length; i += 1) sum += a[i] * b[i]
  return sum
}
 
// After normalization, cosine similarity is the dot product; cosine distance = 1 - similarity, smaller is nearer
export function cosineDistance(a, b) {
  return 1 - dot(a, b)
}

import math
 
 
def l2_normalize(vec: list[float]) -> list[float]:
    norm = math.sqrt(sum(x * x for x in vec))
    return vec if norm == 0 else [x / norm for x in vec]  # return a zero vector as-is; never divide by 0
 
 
def dot(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))
 
 
def cosine_distance(a: list[float], b: list[float]) -> float:
    """After normalization, cosine similarity is the dot product; cosine distance = 1 - similarity"""
    return 1 - dot(a, b)

More dimensions is not better

Leaderboard models run to 1536 or 3072 dimensions, which easily suggests that more dimensions is more accurate. Do the arithmetic first.

Thirty documents at 1536 dimensions and 4 bytes per dimension is under 200 KB, storable anywhere. Now a million-chunk knowledge base: 1,000,000 times 1536 times 4 bytes is about 6 GB. Those 6 GB are not only disk — an approximate nearest neighbor index such as HNSW needs the graph structure and the vectors in memory to be fast, so it is essentially one machine's memory budget. Dimensionality decides that bill directly.

Dimensionality affects two other things. One is computation: each comparison is a few thousand multiply-adds, so doubling the dimensions roughly doubles retrieval latency. The other is quality, with diminishing returns: going from 256 to 512 usually shows a clear gain, while 1536 to 3072 often buys very little for double the storage and latency.

The good news is that mainstream models now support Matryoshka representation: training packs the most important information into the leading dimensions, so you can truncate a 1536-dimensional vector to 512 and renormalize, and it still works. That is what text-embedding-3-small's dimensions parameter does — not a separately trained small model but the same vector with its tail cut. Note two things: truncation is inherently lossy and how much requires an evaluation on your own data (day 8's job); and every vector in an index must have the same dimensionality, so changing it midway means rebuilding the whole store.

Four questions for choosing a model

Facing a leaderboard, the four questions below are enough, without agonizing over rank.

First, language. Is your corpus Chinese, mixed Chinese and English, or multilingual? A model topping an English leaderboard may be quite ordinary at Chinese retrieval. Read the multilingual retrieval column rather than the overall score.

Second, dimensionality and storage. The arithmetic from the last section. While you are there, confirm whether the model supports reduced dimensions; if not, you are locked to its native dimensionality.

Third, maximum input length. Every model has a token limit, and exceeding it either errors or truncates silently — silent truncation is the worse kind, because you believe the whole document was encoded when the second half never entered the vector. This constrains directly how large day 4's chunks may be.

Fourth, whether queries and documents need different prefixes. This is the most easily missed. One family of open models (the e5 series being typical) was trained on paired data and requires prefixing queries with query: and documents with passage: . Without the prefixes it still returns vectors and still computes distances, only noticeably worse, and with no error at all. Read the model card before use and confirm whether it belongs to that family.

The API or a local model: make them look the same

Concretely, computing vectors has two roads.

Call an API (this course's main line uses OpenAI's text-embedding-3-small): no model downloads, no GPU, and a batch request computes dozens at once, with essentially no engineering. The price is paying for every passage, and the initial build is the most expensive one — a hundred thousand chunks is a hundred thousand billed calls; plus an often-overlooked issue: the content leaves your gateway, and whether compliance allows that has to be asked first.

Run an open model locally (Xenova/multilingual-e5-small at 384 dimensions, say): no network, nothing leaving your gateway, no marginal cost per call. The price is managing model loading and inference speed yourself, with quality usually a notch below the leading APIs.

Those two roads should not be written twice in business code. The right approach is one interface with a swappable backend:

embed(texts: string[], opts?: { dimensions?: number }): Promise<number[][]>

That signature serves all fourteen days, and every later lab copies it. Its three backends are the offline deterministic hash vector (used when MOCK=1), a local open model, and the default OpenAI API. Business code knows only this function and needs not one line changed to swap backends.

// The only backend selection point: business code gets an embed function and does not know who is behind it
export function createEmbedBackend() {
  if (process.env.MOCK === '1') {
    return { name: 'mock-hash', dimensions: 384, embed: mockEmbed }
  }
  if (process.env.EMBED_BACKEND === 'local') {
    // The e5 family needs different prefixes for queries and documents; the caller composes them
    return { name: 'local:e5-small', dimensions: 384, embed: localEmbed }
  }
  const dimensions = Number(process.env.EMBED_DIM ?? 1536)
  return { name: 'openai:text-embedding-3-small', dimensions, embed: openaiEmbed }
}

import os
 
 
def create_embed_backend() -> dict:
    """The only backend selection point: business code gets an embed function, not who is behind it"""
    if os.getenv("MOCK") == "1":
        return {"name": "mock-hash", "dimensions": 384, "embed": mock_embed}
    if os.getenv("EMBED_BACKEND") == "local":
        # The e5 family needs different prefixes for queries and documents; the caller composes them
        return {"name": "local:e5-small", "dimensions": 384, "embed": local_embed}
    dimensions = int(os.getenv("EMBED_DIM", "1536"))
    return {"name": "openai:text-embedding-3-small", "dimensions": dimensions, "embed": openai_embed}

That offline backend deserves a sentence more. It does not pretend a call succeeded; it is a set of deterministic hash vectors: cut the text into adjacent bigrams, FNV-1a hash each one, use the hash to decide which dimension to add to and with what sign, then normalize. The same text always gives the same vector, consistently across days. What it preserves is literal overlap, not meaning — so vector retrieval in offline mode behaves much like keyword retrieval. It validates the code path, not retrieval quality, and that sentence appears in every day's lab README.

Getting started with pgvector: create, insert, query nearest neighbors

Where do vectors live? There are plenty of dedicated vector stores, but if your data is already in PostgreSQL, the pgvector extension is often the best value: vectors and business data in one database and one transaction, permission filtering written as a plain where, and no consistency to maintain between two stores. Day 12's memory store uses it too, and day 12 of the 30-day course covers another use of the same extension.

Creating a table means adding a vector-typed column with the dimensionality fixed in the type:

CREATE EXTENSION IF NOT EXISTS vector;
 
CREATE TABLE chunk_embeddings (
  chunk_id       text PRIMARY KEY REFERENCES chunks(chunk_id) ON DELETE CASCADE,
  embedding      vector(1536) NOT NULL,
  embedding_half halfvec(1536) NOT NULL
);

halfvec is the half-precision type at 2 bytes per dimension instead of 4, halving the footprint outright. The retrieval quality loss is usually small — today's lab prints both columns' top three side by side so you can see how much.

Querying nearest neighbors is ordering by distance and taking the top few, with <=> as cosine distance:

SELECT c.chunk_id, c.doc_id, e.embedding <=> $1 AS distance
  FROM chunk_embeddings e
  JOIN chunks c ON c.chunk_id = e.chunk_id
 ORDER BY e.embedding <=> $1, c.chunk_id
 LIMIT 3;

Note the extra c.chunk_id after ORDER BY: two equal distances need a tie-breaker, or the ranking shifts with scan order and will not match on a rerun tomorrow.

With no index, that query is a sequential scan, which is fast enough below tens of thousands of rows. Indexing uses HNSW, and today only one thing matters — the index's operator class must match the operator in the query:

CREATE INDEX ON chunk_embeddings USING hnsw (embedding vector_cosine_ops);

Build it as vector_l2_ops and query with <=> and the index is wasted; the planner simply scans the table, with no error whatsoever. How do you confirm? Read the query plan:

default              ->  Seq Scan on chunk_embeddings
sequential scan off  ->  Index Scan using chunk_embeddings_hnsw on chunk_embeddings

At only 30 rows, the planner choosing a sequential scan is correct — building an index does not mean it will be used, and using one is slower when the row count is tiny. To confirm the index is usable at all, turn enable_seqscan off and look again; an Index Scan appearing means the operator classes match. How index parameters are tuned and how much recall drops is day 5's material; today reaches only "an index exists and it queries."

import { toSql } from 'pgvector'
 
// Inserting: toSql turns an array into text pgvector understands, and ::vector casts it to the vector type
export async function upsert(sql, row) {
  await sql.unsafe(
    `INSERT INTO chunk_embeddings (chunk_id, embedding, embedding_half)
     VALUES ($1, $2::vector, $2::vector::halfvec)
     ON CONFLICT (chunk_id) DO UPDATE SET embedding = EXCLUDED.embedding`,
    [row.chunkId, toSql(row.embedding)],
  )
}
 
export async function nearest(sql, queryVec, topK) {
  return sql.unsafe(
    `SELECT chunk_id, embedding <=> $1::vector AS distance
       FROM chunk_embeddings
      ORDER BY embedding <=> $1::vector, chunk_id
      LIMIT ${topK}`,
    [toSql(queryVec)],
  )
}

from pgvector.psycopg import register_vector
 
 
def upsert(conn, chunk_id: str, embedding: list[float]) -> None:
    """After register_vector you can pass a list directly and the driver converts it to pgvector's type"""
    register_vector(conn)
    conn.execute(
        """INSERT INTO chunk_embeddings (chunk_id, embedding, embedding_half)
           VALUES (%s, %s, %s::halfvec)
           ON CONFLICT (chunk_id) DO UPDATE SET embedding = EXCLUDED.embedding""",
        (chunk_id, embedding, embedding),
    )
 
 
def nearest(conn, query_vec: list[float], top_k: int) -> list[tuple]:
    register_vector(conn)
    return conn.execute(
        """SELECT chunk_id, embedding <=> %s AS distance
             FROM chunk_embeddings
            ORDER BY embedding <=> %s, chunk_id
            LIMIT %s""",
        (query_vec, query_vec, top_k),
    ).fetchall()

At this point you hold two different retrievers: one that goes by the letter and one that goes by meaning. The lab shows where each fails — asked whether exceeding the rate limit returns 429, BM25 steadily hits the document containing 429 while the vector path ranks a topically similar product manual that never mentions 429 above it. Neither is perfect, and they fail differently, which is precisely the reason day 9 fuses the two paths.

Source Reading

Hands-On Lab

The lab's corpus/ is a complete copy of day 1's, with questions overlapping the baseline questions so you can compare the two retrievers' rankings directly. starter/ has four exercise points cut out: hash vectors, L2 normalization, cosine distance, and half-precision quantization. Run as-is, every document's distance is 0.0000 and the top three are always doc-001 through doc-003 — because the distance function is unimplemented and every document is equally near. It can all be done offline without an API key or Docker.

1. Run only MOCK=1 pnpm start first and observe the starter's initial state: the BM25 column is normal and every distance in the vector column is 0.
2. Complete the hash vector and L2 normalization, run again, and watch the vector column's distances start to differ with a top three broadly matching BM25.
3. Complete cosine distance and half-precision quantization, confirming half precision takes half the space with an unchanged ranking and deviations only beyond the fifth decimal place.
4. Run docker compose up -d in the lab root to bring up pgvector, set DATABASE_URL, rerun, and verify the ranking against a real database matches the in-memory version.
5. Read the query plan section at the end: a sequential scan by default and an HNSW index scan after SET enable_seqscan = off, and work out why a sequential scan is correct for a 30-row table.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward the mathematical preconditions of vector similarity, the relationship between dimensionality and recall quality, and the trade-offs in choosing an embedding model. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Explain what an embedding turns into coordinates, and why cosine similarity becomes equivalent to a dot product after normalization
• List the four things you must check when choosing an embedding model: multilingual capability, dimensionality and storage cost, maximum input length, and whether it needs separate query and document prefixes
• Create a table in pgvector, insert vectors, build an index, query nearest neighbors, and read whether the query plan actually used the index
• Explain why a vector cannot tell supported from not supported, and which step has to cover that
• All 5 acceptance criteria of the lab pass, including the run against a real database
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D3) we step back to a stage further upstream: how documents get in. Today we computed vectors over a clean Markdown corpus, and reality brings PDFs, web pages, and scans — multi-column layouts scramble the reading order, tables collapse into one line, and headers and footers mix into the body. Parsing quality is the ceiling on retrieval quality: text already scrambled at parse time is beyond rescue however accurate the vectors. Getting today's path running and knowing what downstream needs is what tells you tomorrow what parsing must preserve.