A rag pipeline diagram is usually drawn as a clean left-to-right sequence of boxes—ingest, chunk, embed, index, retrieve, rerank, generate. It looks finished. It isn't. The diagram is the floorplan for a production system where every single stage carries one distinct, recurring failure mode that no amount of prompt tweaking will paper over. This guide walks through the stages in architectural order, attaches the mode that actually breaks at each step, and ends with the guardrail that keeps a correctly retrieved document from being weaponized as a hallucination cover story.

Our team at techpotions builds retrieval-augmented systems that go into customer-facing products, and the lesson that keeps re-emerging is that retrieval quality and generation faithfulness are separate failures. They need separate fixes. Everything below is drawn from what we've had to rearchitect after watching a pipeline succeed on the demo and fail on real queries.

We'll assemble the rag pipeline diagram stage by stage, defining what each box does and naming the breakage that hides inside it. The full diagram—with failure modes attached—is available in our AI services overview, but the explanation that follows is what makes the diagram legible.

Rag pipeline diagram: ingestion and chunking

This stage silently decides retrieval quality, and nothing downstream can recover the context lost here.

Ingestion pulls raw source material—documentation pages, code files, markdown repos, support tickets—into the pipeline. Chunking slices that material into pieces small enough to embed and retrieve. The failure mode is a chunk boundary that splits a table, a code block, or a definition from its heading. The resulting chunk is individually meaningless. Vector similarity will match it; reranking will promote it; the generator will still see a fragment that makes no sense without its neighbor.

Chunking strategies that look identical on a whiteboard produce very different recall in practice:

The fix isn't a single strategy—it's a preprocessing step that marks structural boundaries before chunking so the splitter can't cut through a ` ``` ` fence or a markdown table. If your source documents aren't clean enough for structure-aware chunking, consider our content pipeline approach, which enforces formatting upstream so chunking downstream stops breaking.

Rag pipeline diagram: embedding and indexing

Embedding converts chunks into vectors. Indexing stores those vectors in a structure that supports approximate nearest neighbor search. The failure mode here is a model swap that invalidates everything.

Embeddings from different models—or even different versions of the same model—are not comparable. If you start with text-embedding-ada-002, index a million documents, and then switch to text-embedding-3-small without reindexing, your index now contains vectors that mean two different things in the same space. A query embedded with the new model will return nearest neighbors that are semantically irrelevant because the old vectors were measured on a different ruler.

A partial reindex makes this worse. Half of your vectors use one distance metric; half use another. The system degrades in ways that look like retrieval quality problems but are actually index corruption. The pipeline diagram should show an explicit version pin on the embedding step, and the ops runbook should mandate a full reindex on any embedding model change. No exceptions.

This is also the stage where LLM feature evals need a benchmark that measures embedding drift, not just retrieval accuracy—if you can't detect when your vector space silently split, you'll ship broken results without knowing.

Rag pipeline diagram: retrieval

Retrieval is the stage everyone tunes and the one where relevance and recall trade against each other directly.

Given a query embedding, the retriever searches the vector index and returns the k nearest chunks. The failure mode: pure vector similarity retrieves things that are semantically near and factually wrong. "Quarterly revenue grew 14%" and "Quarterly revenue declined 14%" embed close together. The retriever doesn't know which one is true; it only knows they're neighbors in embedding space.

The tradeoff:

The fix is acknowledging that retrieval is a recall engine, not an answer engine. It should return enough context to cover the question, and then hand off to a separate stage that discriminates. Over-tuning retrieval to also be the discriminator produces a brittle system that works on the eval set and breaks on real queries.

Rag pipeline diagram: reranking and assembly

Reranking re-scores the retrieved chunks using a model that can compare relevance more precisely than embedding similarity alone. Assembly packs the highest-scoring chunks into the context window that will be sent to the generator. The failure mode here is that the context window budget gets spent on chunks that actively degrade the answer.

A chunk that scored high on reranking but is factually irrelevant doesn't just waste tokens—it pollutes the generator's attention. The model tries to reconcile irrelevant context with the query and produces answers that blend sources incorrectly. This is worse than retrieving nothing. A generator with an empty context will sometimes refuse to answer or state uncertainty. A generator with retrieved-but-irrelevant chunks will produce confident wrong answers that cite your documents.

Assembly is therefore a pruning step, not a packing step. The rule we build into our AI pipelines is: if a chunk can't answer a concrete sub-question extracted from the user's query, it doesn't go into the context window, regardless of its rerank score.

Rag pipeline diagram: generation and citation

This is where you find out whether the model is grounding in retrieved context or falling back on parametric memory while citing your documents as cover.

The model receives a system prompt, the assembled context, and the user query. It generates an answer. The citations tell you where each claim came from—ostensibly.

The failure we kept hitting is invented statistics delivered with total confidence and linked to retrieved documents that don't contain them. The retriever fetched a document about pricing. The document had no pricing numbers in it. The model quoted a specific dollar figure, cited the document, and the figure was pure parametric hallucination wrapped in a citation.

Retrieval worked. Generation failed. Retrieval quality and generation faithfulness are separate failures and need separate fixes.

The guardrail we ship with every pipeline

From our own content pipeline, the system prompt is explicit on two points:

1. Grounding over recall: If a fact isn't explicitly present in the retrieved context, do not state it. Unsupported facts must be omitted entirely.
2. Citation format as a constraint: Citations must appear as inline contextual links with descriptive anchor text, not as a trailing reference list. This forces the model to connect each claim to a specific source chunk at generation time rather than appending a bibliography after the fact.

Here is the core of that system instruction:

You are answering from a set of retrieved source chunks.

Rules:
- State only facts explicitly present in the chunks that follow.
- If a chunk does not support a claim, do not make the claim.
- Cite inline with descriptive anchor text that names the source document, 
  never as a bracketed number or trailing reference list.
- If the retrieved context is insufficient to answer accurately, 
  state that limitation rather than guessing.

This prompt pattern works because it constrains the model at two levels: the content boundary (no unsupported facts) and the format boundary (inline citations as a forcing function). The trailing reference list is easy for a model to generate after fabricating an answer; inline contextual links require it to hold each claim and its source together during generation.

For teams that want to measure whether this guardrail is actually working, eval design for LLM features includes a citation-faithfulness metric we run per-release, separate from retrieval recall.

The full rag pipeline diagram with failure modes

Putting the stages together, here is the pipeline as it should be drawn:

[Source Documents]
       |
       v
[Ingestion + Chunking]  <- FAILURE: boundary splits tables/code/definitions
       |
       v
[Embedding + Indexing]  <- FAILURE: model swap invalidates vector space
       |
       v
[Retrieval]            <- FAILURE: semantic proximity ≠ factual accuracy
       |
       v
[Reranking + Assembly] <- FAILURE: irrelevant chunks consume budget, 
       |                          degrade answers
       v
[Generation + Citation] <- FAILURE: parametric memory cloaked in citations

Every stage is necessary. Every stage breaks in its own way. A rag pipeline diagram that omits the failure modes is a wish, not an architecture. The version with failure modes labeled is what we use to scope work and to debug production incidents—because the incident always maps to one of those five labels.

FAQ

Why attach failure modes to a rag pipeline diagram?

A pipeline diagram that stops at the happy path misses the real engineering. Each stage—chunking, embedding, retrieval, reranking, generation—has a distinct failure mode that a clean workflow box doesn't show. Attaching those modes to the diagram makes the architecture actionable rather than aspirational, and it gives the on-call engineer an immediate map from symptom to stage.

How do you stop a RAG system from inventing facts while citing your documents?

Use a strict 1:1 mapping between source paragraphs and citation markers. Set an explicit system prompt rule: "If a fact isn't explicitly present in the retrieved context, omit it. Cite inline with contextual anchor text, never as a trailing reference list." This penalizes parametric invention because the model can't backfill a bibliography after generating unsupported claims—it has to bind each claim to a source at generation time.

Is retrieval quality the same thing as generation faithfulness?

No—they're separate failure surfaces. Retrieval fails when the vector store returns semantically near but factually wrong documents. Generation fails when the model ignores correctly retrieved context and substitutes parametric memory. Tuning retrieval won't fix generation hallucination. A pipeline needs separate metrics and separate guardrails for each.