AI Explainedby Milad Fahmy

AI Explained

Retrieval metrics: recall@k, MRR and nDCG for RAG

How to measure whether your RAG retrieval finds the right documents, with precision and recall at k, mean reciprocal rank and nDCG, plus a worked example.

Milad FahmyNov 17, 20262 min read

On this page
  1. Test retrieval on its own
  2. Precision and recall
  3. Mean reciprocal rank (MRR)
  4. nDCG
  5. Which one to use

Your RAG answer is wrong. Was it the model, or the search? Retrieval metrics tell you.

Test retrieval on its own

Build a test set: real queries, each labelled with the documents that should come back. Then run only the retriever and score what it returns.

Precision and recall

  • Precision is the fraction of retrieved documents that are relevant.
  • Recall is the fraction of relevant documents you actually retrieved.

A RAG pipeline only keeps the top k results, so you measure at a cutoff k.

Recall@k asks: of the documents that matter, how many made it into the top k? If a query has 4 relevant documents and 3 of them are in the top 5, recall@5 is 3/4. For RAG this is often the one to watch: if the right chunk isn't retrieved, the model never sees it.

Precision@k counts how much of the top k is relevant. But it ignores where in the top k those documents sit: a hit at position 1 and a hit at position 5 count the same.

Mean reciprocal rank (MRR)

MRR cares about position. For each query, take one over the rank of the first correct result, then average across queries:

Query First correct result Score
A rank 3 1/3
B rank 2 1/2
C rank 1 1

(1/3 + 1/2 + 1) / 3 ≈ 0.61.

Two pieces of fine print: a query with no correct result scores zero, and any extra relevant results after the first are ignored.

nDCG

nDCG (normalized discounted cumulative gain) goes further. It handles graded relevance (very relevant, somewhat relevant, not relevant) rather than just yes or no, and it rewards putting the best documents near the top. It compares your ranking with the ideal ranking, so a perfect ranking scores 1.

The BEIR retrieval benchmark reports nDCG@10 for every dataset, and calls plain precision and recall "rank unaware".

Which one to use

  • Recall@k: what actually reaches the model.
  • MRR: when the first good answer is what matters.
  • nDCG: when you care about the whole ranking.

Track them whenever you change chunking, embeddings or rerankers, always on the same query set, so you know whether a change helped retrieval or just moved the problem.

Don't guess where RAG fails. Measure the retrieval first.

Sources

#rag#retrieval#evaluation#mrr#ndcg

Watch