Embeddings & vector search

Key takeaways An embedding turns a piece of text into a vector — a list of numbers positioned so that similar meanings land close together. Compare vectors and you get semantic search: retrieval by meaning, not exact keywords. The vectors live in a vector database built for fast nearest-neighbour lookup. This is the engine under grounding & retrieval — the mechanism that actually finds the right material to hand a model.

The last lesson said retrieval means finding the relevant material and putting it in front of the model. This lesson opens the box and shows how that finding happens. The trick is to convert text into numbers in a way that captures meaning, so that “finding relevant text” becomes “finding nearby points.”

What an embedding is

An embedding model takes a chunk of text and outputs a vector — a fixed-length list of numbers. The numbers aren’t random: the model is trained so that texts with similar meaning produce vectors that are close together, and unrelated texts produce vectors that are far apart.

Think of each vector as a point in space. Two sentences that say the same thing in different words land near each other; a sentence about something else lands far away. You never read the numbers yourself — they’re only useful for measuring distance between texts. That single property, meaning becomes proximity, is what everything else in this lesson is built on.

Searching by meaning

Here’s the move. You embed your documents once, ahead of time, and store the vectors. Then, when a question comes in, you embed the query with the same model and look for the document vectors closest to it. A common way to measure “close” is cosine similarity, which compares the direction two vectors point. The closest documents are your results.

Because closeness tracks meaning, this returns relevant passages even when they share no words with the query — ask about “losing signal lock” and it can surface a passage about “the decoder dropping sync.” That’s the payoff over keyword search (such as BM25), which matches the literal words and would miss it.

Neither approach is strictly better, so many systems run hybrid search: combine the keyword score and the vector score so exact terms and overall meaning both count.

Vector databases & indexes

Comparing your query to every stored vector one by one works for a few thousand documents but gets slow fast. A vector database solves this by building an index for approximate-nearest-neighbour (ANN) search — a structure that finds the closest vectors quickly without checking all of them, trading a tiny bit of accuracy for a big speed-up.

You have options, and they’re all the same idea underneath:

  • pgvector — an extension that adds vector columns and ANN indexes to Postgres, so your vectors live next to your regular data.
  • FAISS — a library for building an in-process vector index, good when you want vectors in your own application rather than a separate service.
  • Dedicated vector databases — standalone services built specifically for storing and querying vectors at scale.

Whatever you pick, a query asks for the top-k results — the k nearest vectors, say the top 5 — which become the material you pass along to the model.

The workflow

Two phases, and it helps to keep them separate. Indexing happens offline, once (or whenever documents change). Querying happens online, every time someone asks.

INDEX  (offline, once)
  document ─▶ chunk into passages ─▶ embed each chunk ─▶ store vectors in the index

QUERY  (online, per request)
  question ─▶ embed the question ─▶ search the index ─▶ return top-k nearest chunks

The indexing phase is the expensive, slow part, and you pay it up front. The query phase is cheap: one embedding call and one fast index lookup. Both phases must use the same embedding model, or the query vector and the document vectors won’t live in the same space.

Practical notes

  • The embedding model is separate and cheap. Embeddings come from a different, much smaller and cheaper model than the chat model that writes the answer — see the sibling lesson on types of models. You call it a lot (every chunk, every query), so its low cost matters.
  • Budget for cost and storage. Embedding a large corpus is many model calls, and every vector takes space. Neither is huge, but both scale with your document count, so plan for them.
  • Keep the model consistent. Use the same embedding model for documents and queries. Mixing models compares points from incompatible spaces and returns nonsense.
  • Re-embed when you change models. Switching to a newer embedding model means re-embedding your whole corpus so old and new vectors are comparable again.

Limits

Semantic search is powerful, but it isn’t the right tool for everything. Because it matches meaning rather than exact strings, it can miss exact-match needs — a specific part number, an error code, a talkgroup ID — where you want the literal token, not something “similar.” Hybrid search helps here by keeping a keyword path alongside the vector path.

And even perfect search can only return what you fed it. If your documents were chopped into unhelpful pieces, retrieval surfaces unhelpful pieces. Chunking quality is its own topic — we cover it in chunking & retrieval quality.

Quick check: Embeddings let you search by...

Recap

  • An embedding model turns text into a vector so that similar meanings sit close together.
  • Semantic search embeds the query and returns the nearest document vectors (e.g. by cosine similarity) — results by meaning, not exact keywords.
  • A vector database (pgvector, FAISS, or a dedicated service) stores vectors and runs fast approximate-nearest-neighbour search to fetch the top-k matches.
  • Index offline (chunk, embed, store); query online (embed, search, return top-k) — always with the same embedding model on both sides.
  • Semantic search can miss exact-match needs, so hybrid search keeps a keyword path too, and retrieval quality still depends on how you chunked.

Next up: building a RAG pipeline.