Adaptive Top-k Retrieval: 17% Fewer RAG Input Tokens With No Extra LLM Calls

Short answer: Adaptive top-k retrieval lets a RAG system choose how many passages to send to the LLM for each question, based on the drop in similarity scores. In my benchmark it used 17% fewer input tokens than a fixed top-k of 5, with no extra LLM calls and no loss in answer quality.

Most RAG pipelines retrieve a fixed number of passages for every question, usually five. That is simple, but it is wasteful. A question like "Who wrote this book?" needs one passage. A question that compares two things may need four or five. With a fixed k, the easy questions pay for context they do not need, and the extra passages can distract the model.

I tested a small change: let the retriever decide how many passages to keep for each query, using only the similarity scores it already has. This post covers how it works, what the numbers were, and where the evidence is still thin.

What is adaptive top-k retrieval?

When you search a vector store, the results come back with similarity scores. For many questions those scores have a visible shape: one or two passages score high, then there is a sharp drop, then a long tail of weak matches. The passages after the drop are usually noise.

The adaptive router does three things:

  1. Retrieve a maximum number of candidates with their scores.
  2. Walk down the sorted scores and find the first sharp drop (the elbow).
  3. Keep only the passages before that drop.
How adaptive top-k picks k: cut at the first sharp dropSimilarity score0.82Passage 10.79Passage 20.61Passage 30.58Passage 40.55Passage 5Elbow: drop 0.18 > margin 0.04Kept: k = 2Dropped as noise

The diagram above uses made-up scores to illustrate the idea, not benchmark data. There is no extra LLM call and no second model. The decision uses numbers the retriever already produced, so it adds no cost.

How does the elbow detection work in code?

This is a simplified version of the logic:

def adaptive_top_k(scores, max_k=5, margin=0.04):
    """scores: similarity scores sorted high to low."""
    scores = scores[:max_k]
    for i in range(1, len(scores)):
        if scores[i - 1] - scores[i] > margin:
            return i          # keep passages before the drop
    return len(scores)        # no clear drop, keep all

The margin (0.04 in my setup) decides how big a drop counts as an elbow. I calibrated it on embedding-only score curves before running any paid generation calls, so tuning it cost nothing. The full code is in the rag-cost-latency-router repository on GitHub.

Benchmark setup

  • Dataset: 50 questions from the rag-mini-wikipedia QA test set
  • Corpus: about 3,200 passages
  • Embeddings: gemini-embedding-001
  • Generation: gemini-3-flash
  • Baseline: static retrieval with k=5

Results: adaptive top-k vs fixed top-k

MetricStatic (k=5)Adaptive (avg k=4.0)
Exact Match0.320.40
F10.5400.561
Avg input tokens per query569.4472.4 (17.0% lower)
Avg latency (seconds)1.9341.934
Benchmark results for adaptive vs static top-k retrieval: F1 score, average latency in seconds and average input tokens per query

What the numbers do and do not show

The token saving is real. Input tokens dropped 17% across the run. Token counts are deterministic, so this result does not depend on luck. At scale, 17% fewer input tokens is a direct cut in the bill.

The accuracy gain is not proven. Exact Match went from 0.32 to 0.40, which on 50 questions is 16 correct against 20. A gap of four questions can easily be noise. The fair reading is that accuracy did not get worse while cost went down. I would not claim it improved until I rerun this on a few hundred questions.

Latency did not change. I expected fewer tokens to mean faster responses. At this prompt size, generation time is dominated by output, so the saving did not show up in latency.

When should you use adaptive top-k?

  • Your questions vary in difficulty, with many simple lookups mixed with harder ones.
  • Input tokens are a meaningful part of your cost, for example long passages or high traffic.
  • You want a cost cut that needs no new model and no extra call.

It helps less when your similarity scores are flat, with no clear drop. In that case the router keeps all the candidates and you are back to fixed k.

Frequently asked questions

Does adaptive top-k need an extra LLM call?

No. It only uses the similarity scores the retriever already returns, so it adds no model calls and no measurable latency.

How much can adaptive top-k reduce RAG cost?

In this benchmark it cut input tokens by 17% compared with a fixed top-k of 5. The saving depends on your data: the more simple questions you get, the more passages it can drop.

Does it hurt answer accuracy?

Not in this test. Exact Match and F1 were both slightly higher with the adaptive router, but the sample was 50 questions, so the safe conclusion is that accuracy held steady.

How do I choose the margin value?

Plot the sorted similarity scores for a sample of your own queries and look at how large the drops are. The margin depends on the embedding model, so recalibrate it if you change models.

Is this the same as reranking?

No. A reranker rescores passages with a second model. Adaptive top-k only decides where to cut the list. The two can be combined.

What I am doing next

  • Rerunning on 300 to 500 questions across two datasets, with confidence intervals
  • Testing a second embedding model, since the margin depends on the score distribution
  • Comparing against a reranker-based cutoff

If you are running RAG in production and want help measuring or cutting its cost, see Work With Me.

Comments

Popular posts from this blog

Humanoid Robots 2026: Tesla Optimus, Figure AI & 1X NEO

Humanoid Robots Are Working in Real Factories — Here's What's Actually Happening

Agentic AI & Autonomous Systems: The Complete 2026 Guide (Gartner's #1 Strategic Trend)