What is Reranker?

Why It Matters

First-pass retrieval optimizes for speed over a large index: it returns candidates that are probably relevant. Probably is the problem. The raw top-k from vector search includes near-misses that look right and ground the answer in the wrong place, and the model has no way to know.

The reranker exists to close that gap. It is a slower, more careful judgment applied to a small candidate set: the difference between documents that matched the query and documents that actually answer it.

How It Works

Retrieval returns tens of candidates; the reranker scores each candidate against the query with a cross-encoder or a dedicated ranking model, and re-orders the set. Only the top few after reranking enter the context window.

The economics are the point. Cross-encoder scoring is too expensive to run over the whole index, so the pipeline splits the work: cheap broad recall first, expensive narrow precision second. Each stage does the job it is priced for.

Where It Breaks

The most common failure is absence: teams ship raw top-k retrieval and never add the second pass, so ranking quality is whatever the embedding model happened to order. The second failure is a reranker tuned on the wrong distribution: generic relevance benchmarks instead of the actual corpus, so it promotes confidently wrong passages.

Rerankers also drift like every other model in the stack. A version update that re-orders differently can shift what the model sees without a single line of your code changing.

How Flytebit Handles It

Reranking is standard in our RAG builds, tuned against the client’s corpus rather than a benchmark set, and scored in the same eval harness as the rest of the pipeline, so a reranker change is a measurable event, not a silent one. The measurement approach is documented in Evaluating Agentic AI.