What is Vector Search?

Why It Matters

Keyword search finds documents that share words with the query. Vector search finds documents that share meaning, which is what users actually need, since they rarely phrase a question the way the document phrases the answer. It is the technology that makes RAG’s promise real: ask naturally, retrieve what matters.

Its limits are just as structural. Meaning-space has no notion of exactness: part numbers, error codes, proper nouns, and version identifiers all blur together, because “close in meaning” is the wrong axis for “exactly this string.”

How It Works

Each document chunk is stored as an embedding, a point in vector space. At query time the query gets embedded the same way, and the system returns the nearest points by distance. The index is built for approximate nearest-neighbor lookup, so search stays fast even over millions of chunks.

The output is a ranked candidate set, ordered by similarity, a score of semantic proximity, not correctness. The reranker applies the second, more careful pass before anything reaches the model.

Where It Breaks

Pure vector search fails on exact identifiers: query for a specific error code or clause number and similarity returns things that are semantically adjacent but factually wrong. It also fails quietly: the near-miss results read plausibly enough that nobody questions them, and the answer gets grounded in the wrong passage with confidence.

The other break is assuming similarity equals relevance. Two texts can be about the same topic while one answers the question and the other contradicts it; distance in vector space does not know the difference.

How Flytebit Handles It

Production retrieval in our RAG builds is never vector-only: hybrid search runs keyword alongside vector so exact identifiers survive, and the fused candidates go through a reranker before the model sees them. The failure modes this design answers are cataloged in TrustNLP’s taxonomy.