RAG vs MCP: What Your Agent Actually Needs

RAG pipeline vs MCP is not a fork in the road

The rag pipeline vs mcp question gets asked like you're picking a database, and that framing is why so many agents give confident wrong answers. RAG is a retrieval pattern over a copy of text you own. MCP is a protocol for asking a system that owns the data itself. Get the ownership boundary wrong and no amount of prompt engineering saves you.

This distinction is not a hot take. The clearest formulation is that RAG is a pattern for finding relevant content while MCP is a transport for connecting an agent to sources, and the two compose: you can run retrieval behind an MCP endpoint. Others frame it the same way — the RAG vs MCP choice is a false one — while practitioners tend to split it as RAG for grounding, MCP for structured tool use, and agents for orchestration. Most teams still start by treating them as two competing architectural options, build the index first, hit staleness, then bolt on tools with no way to tell which layer broke a given answer.

That last part is the real cost. Not picking wrong — picking without instrumentation.

RAG pipeline vs MCP: the dividing line

Own the copy, use retrieval. Ask the owner, use a tool call.

That last row is the whole article. Retrieval failures return a well-formatted chunk of the wrong thing. Tool failures usually surface as an error, a timeout, or a duplicate. You debug them completely differently.

Where RAG pipelines actually fail

RAG fails quietly. Nothing throws.

• Stale index. The index faithfully returns what was true last Tuesday. The model has no signal that the chunk is outdated, so it answers with total confidence. This is the single most common cause of "our agent lies about pricing."
• Chunking. A fixed-size splitter cuts the qualifier away from the claim. The chunk that survives says "refunds are available for 30 days" and drops the "unless the order shipped from the EU" that was in the next paragraph. The retriever did its job. The chunker destroyed the answer.
• Ranking. Top-k by vector similarity rewards topical overlap, not answer-bearing content. The chunk that actually contains the number often ranks fifth or seventh behind four chunks that merely mention the topic.
• Permissions. The index is a copy of the document, which means it's a copy of the ACL you remembered to sync. A tenant filter that lives in application code rather than in the index itself is one refactor away from serving another customer's document. That's a retrieval bug that presents as a model bug, and it's the one that gets you a phone call.

None of these produce an exception. All of them produce fluent output. If you want the full evaluation methodology for this layer, it's its own problem — see how to evaluate a RAG pipeline.

Where MCP actually fails

MCP fails loudly but late — usually after the agent has committed to a plan.

• Latency. One vector query is roughly a single round trip to your own infrastructure. A tool call is a round trip to a server that is frequently calling a third-party API on your behalf. An agent that plans three steps and calls two tools at each step multiplies that, and users experience it as "the agent is thinking" for twenty seconds.
• Auth. Whose permissions does the tool run under? If the answer is a service account, the agent can read everything that service account can read, and your careful per-user ACL story is gone. Delegated, per-user tokens are the correct answer and the harder one to build.
• Side effects. Reads are safe to retry. create_refund is not. Any tool with a write needs an idempotency key, and most MCP servers shipped in a hurry don't have one.
• Schema drift. The upstream API changes a field name and the tool returns a payload the model misreads. The tool call succeeded with a 200, so nothing alerts.

The combined setup that actually works

Index for discovery, call the source for truth.

Retrieval is very good at one thing: narrowing a large corpus down to a handful of candidate identifiers. It is bad at asserting current state. Tool calls are the inverse. So split the job:

1. Retrieval returns candidate IDs plus a freshness stamp (indexed_at, source_updated_at), never a final answer.
2. An MCP tool fetches current state by ID from the system that owns it.
3. The answer is assembled from both, and the model is told which is which.

A retrieval tool exposed over MCP looks like this — one interface for the agent, one place for you to log:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("docs")

@mcp.tool()
def search_docs(query: str, k: int = 8) -> list[dict]:
    """Find candidate documents. Returns IDs and timestamps, not answers."""
    hits = index.search(query, k=k, acl=current_principal())
    return [
        {
            "id": h.id,
            "snippet": h.text[:400],
            "score": h.score,
            "indexed_at": h.indexed_at,
        }
        for h in hits
    ]

@mcp.tool()
def get_document(doc_id: str) -> dict:
    """Fetch the current, authoritative version of a document."""
    return source.fetch(doc_id, acl=current_principal())

The current_principal() call is the load-bearing line in both tools. If it's a hardcoded service account, you've built a very fast way to leak documents.

When retrieval is the wrong answer

If the data changes faster than you can reindex, retrieval is the wrong tool — no matter how good the embeddings are.

Reindexing every minute isn't a freshness strategy, it's a load test. Reach for a live lookup instead when:

• The value changes every minute. Inventory, prices, balances, ticket status, on-call rotation. Anything where a five-minute-old answer is a wrong answer.
• The answer is a computation, not a passage. "How many", "what's the total", "which is earliest" — these are queries. No chunk contains the answer.
• The answer depends on the caller's permissions. "Can this user see it" cannot be answered from a copy, because the copy doesn't know who's asking.
• The corpus fits in the context window. Just put it in the prompt. Retrieval over 40 documents is ceremony.

Where teams get burned is the middle case: data that's mostly static with a few volatile fields. Index the prose, fetch the volatile fields. Don't reindex the whole document to update a stock count.

How to tell which one caused the bad answer

Split the pipeline and score each layer separately. Aggregate scores hide the cause.

Score retrieval on whether the right evidence arrived (recall@k against a gold passage), score the tool on whether it returned the right data at acceptable latency, and only then score the final answer. Three numbers, three different fixes.

To run that table you need a trace per turn, not a log line. Record both layers in the same record:

trace = {
    "turn_id": turn_id,
    "retrieval": {
        "query": query,
        "chunks": [{"id": c.id, "score": c.score, "indexed_at": c.indexed_at} for c in hits],
    },
    "tool_calls": [
        {"name": name, "args": args, "status": status,
         "latency_ms": ms, "as_of": source_timestamp}
    ],
    "answer": answer,
    "verdict": "ok | retrieval_miss | stale_index | chunk_boundary | tool_error | generation",
}

The verdict field is the point. A human labels a sample of failures once, and the distribution tells you which layer to fix first. Without it you're guessing, and the guess is usually "the model needs a better prompt."

Then run the same eval set against three configurations: retrieval only, tools only, both. The delta between them is the actual answer to "which one does my agent need" — for your data, not for a blog post. The broader methodology lives in our RAG evaluation walkthrough, and if you'd rather not build the trace ledger and eval harness from scratch, that's the kind of thing we do in our AI engineering work.

One last note on the horizon: this is becoming an empirical question rather than an architectural argument. There are now direct benchmarks comparing agent interfaces — HTML browsing, retrieval, and newer protocols — and tooling that deliberately bundles MCP, GraphRAG, and agents in one system. The convergence is real. Your mileage will still depend on how fast your data moves.

FAQ

Is MCP a replacement for RAG?

No. MCP is a transport for reaching tools and data sources; RAG is a retrieval pattern for finding relevant content. You can expose a RAG pipeline as an MCP tool, which is usually the right move once you have both — one interface for the agent, one place to log retrieval quality.

Can I run my RAG pipeline behind an MCP server?

Yes, and it's often the cleanest composition. Wrap retrieval as a tool that returns candidate IDs and freshness timestamps, then add a second tool that fetches current state from the source system by ID. The retriever narrows, the source system asserts.

How do I know if a bad answer came from retrieval or from a tool call?

Instrument both layers in the same per-turn trace: retrieved chunk IDs with scores and indexed_at, plus every tool call with arguments, status, latency, and the source timestamp it returned. Then check whether the correct evidence was ever present. If it was and the answer is still wrong, it's generation. If it wasn't, you know which layer to open first.