Learning how to evaluate a RAG pipeline – really evaluate it – starts by splitting retrieval from generation, then measuring both with the rigor your users deserve. When we built Parcoursup Zen, an AI orientation platform matching French students to 23,000+ post‑bac formations, a retrieval slip wasn’t a minor UX annoyance; it could push a student toward a dead‑end degree or miss the perfect fit entirely. So we treated evaluation as a first‑class engineering discipline, not an afterthought.

Building a golden question set before you measure anything

Without ground truth, your metrics are floating numbers. For Parcoursup Zen we created a 240‑case evaluation suite with domain experts. Every case pairs a student profile (interests, grades, location) with a known set of formation IDs that should appear in the top‑10 results. This golden set gates every prompt or embedding change and is the only reason we can trust that our pipeline won’t silently degrade during admission season.

How to evaluate a RAG pipeline: retrieval metrics that catch silent drift

The takeaway: Track recall@k and mean reciprocal rank (MRR) on your golden set. A chunk‑size tweak that passes a vibe check can tank recall by double digits — unless you measure it.

In Parcoursup Zen, retrieval quality is non‑negotiable: a science‑profile student must see the relevant engineering programmes in the top 5, not art history. We instrument every retriever call against the golden suite:

def evaluate_retrieval(test_cases, retriever, k=10):
    recall_at_k = []
    mrr = []
    for case in test_cases:
        retrieved = retriever.get_relevant_docs(case['query'], top_k=k)
        retrieved_ids = [doc.id for doc in retrieved]
        hits = set(case['relevant_doc_ids']) & set(retrieved_ids)
        recall_at_k.append(len(hits) / len(case['relevant_doc_ids']))
        for rank, doc_id in enumerate(retrieved_ids, start=1):
            if doc_id in case['relevant_doc_ids']:
                mrr.append(1.0 / rank)
                break
        else:
            mrr.append(0.0)
    return {'recall@k': np.mean(recall_at_k), 'mrr': np.mean(mrr)}

The Braintrust evaluation guide recommends pairing recall with MRR to penalise late‑appearing relevant documents.

How to evaluate a RAG pipeline: generation‑side metrics when answers must be truthful

The takeaway: A fluent answer that hallucinates facts is worse than silence. Use an LLM‑as‑judge to score faithfulness and citation accuracy against the retrieved context.

In Parcoursup Zen, when the model explains why a formation fits a student, every claim must be traceable to the formation’s official description. A fabricated “high employment rate” could mislead a family. We built a faithfulness check that feeds the retrieved chunks and the answer to a judge prompt:

prompt = """
Given the retrieved context:
{context}

And the answer:
{answer}

Is the answer fully supported by the context? Reply with YES or NO and a brief reason.
"""

The Ragas tutorial and Hugging Face’s evaluation cookbook provide formalised faithfulness scoring. For stricter compliance, we also validate that citations in the answer point to correct source identifiers.

Cost and latency: the operational metrics nobody budgets for

The takeaway: A perfect answer that costs 10× more than expected can sink your margins. In Parcoursup Zen, keeping the average AI cost below €4 per student was a hard requirement; we track token consumption and wall‑clock time for every interaction.

We instrumented the LangChain pipeline with a custom callback handler:

from langchain.callbacks import BaseCallbackHandler
import time, tiktoken

class CostLatencyTracker(BaseCallbackHandler):
    def on_llm_start(self, serialized, prompts, **kwargs):
        self.start_time = time.time()
    def on_llm_end(self, response, **kwargs):
        latency = time.time() - self.start_time
        tokens = response.llm_output['token_usage']['total_tokens']
        log_metric('llm_latency_ms', latency * 1000, tags={'model': response.model_name})
        log_metric('tokens_per_query', tokens)

These metrics feed a dashboard that alerts if per‑query cost drifts upward after a prompt engineering sprint. The Redis RAG evaluation guide highlights similar infrastructure‑level observability.

Wire your evaluation into CI so every change reports its own grade

The takeaway: An evaluation suite is useless if it only runs when someone remembers. We made the 240‑case Parcoursup Zen eval a gated step in our GitHub Actions pipeline — a chunking parameter change must prove it doesn’t degrade recall or faithfulness before it can merge.

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run RAG evaluation suite
        run: |
          pip install -r requirements-eval.txt
          python -m pytest tests/eval_suite.py --junitxml=results.xml
      - name: Assert metric thresholds
        uses: dorny/test-reporter@v1
        with:
          name: RAG Eval
          path: results.xml
          fail-on-error: true
          fail-on-empty: true

Our eval suite reports whether recall@10 ≥ 0.85 and faithfulness ≥ 0.95 — if a new chunking strategy drops recall by even three points, the CI job fails and the pull request turns red. This is the safety net that let us ship changes weekly through peak admission season without breaking a single student session.

If you want a team that builds RAG systems with evaluation baked into the CI pipeline from day zero, our AI agent development company designs these safeguards as part of the core architecture. (Book a deep dive on our AI services to see how we instrument it.)

FAQ

What’s the most critical metric for evaluating a RAG pipeline?

It depends on the stakes. For retrieval‑heavy apps like Parcoursup Zen, recall@k and MRR are life‑critical — missing a relevant formation can change a student’s future. For generative tasks, faithfulness is non‑negotiable: a fluent hallucination erodes trust. Most production systems need all three.

How often should I re‑evaluate my RAG pipeline?

On every change. We run our 240‑case evaluation suite in CI on every pull request that touches embeddings, chunking, or prompts. A periodic manual audit on fresh real‑world queries also catches distribution shift.

Can I evaluate a RAG pipeline without a golden dataset?

You can bootstrap with synthetic queries generated by an LLM and ask a judge model to score effectiveness, but that inherits biases. Domain‑expert‑curated golden sets, like the 240‑profile suite we built for Parcoursup Zen, give the highest signal and catch edge cases no synthetic generator will invent.