Build a RAG Pipeline with Citations

Retrieval-augmented generation (RAG) answers a question from retrieved documents instead of relying only on model weights. A production pipeline has five jobs: ingest sources, retrieve the right passages, generate an answer that cites those passages, refuse when retrieval is empty, and measure faithfulness.

This is the canonical GenAIWiki how-to. It is not a PII-redaction legal clone and not a vendor bakeoff. Use RAG for the definition. Use this page to ship the loop.

1. Confirm you actually need RAG

Use RAG when answers must stay grounded in your corpus: internal docs, tickets, policies, or product manuals.

Skip a custom pipeline when:

• The corpus is a modest file set the model can search with a hosted tool. OpenAI documents this as file search.
• The task is style or format, not facts. Prompting or structured outputs may be enough.
• You need the model to acquire a skill that is not in the documents. That is fine-tuning, not RAG.

Do not mix those three jobs in one eval.

2. Pick one retrieval path

Official docs describe three supported starting points. Pick one and stay there until the gold set is green.

1. LangChain RAG tutorial. LangChain's RAG tutorial walks through load, split, embed, retrieve, and generate. Use it when you want a composable Python graph and your own vector store.
2. LlamaIndex RAG. LlamaIndex's RAG guide is the query-engine path: index documents, retrieve nodes, synthesize an answer. Use it when the product is a document Q&A engine.
3. OpenAI file search. The file search tool lets the model retrieve from hosted files without you operating an index. Use it when the corpus fits the tool's limits and you want OpenAI to run retrieval.

Do not start from a scraped blog that copies LangChain snippets without the current APIs.

3. Ingest with metadata you can filter later

Whatever path you pick:

1. Keep source identity (URL, doc id, tenant, ACL group) on every chunk.
2. Chunk by document structure, not a random character count.
3. Embed with a pinned model ID. Changing the embedding model later means re-indexing.
4. Enforce tenancy in metadata now. Query-time filters cannot recover a mixed index.

Official LangChain and LlamaIndex tutorials both treat splitting and metadata as required steps, not optional polish.

4. Retrieve, then require citations in the prompt

Retrieval without citations is a chatbot with extra latency.

1. Retrieve top-k passages with metadata filters for the caller.
2. Pass only those passages to the generator.
3. Instruct the model to answer from the passages and to cite them. Use Grounded RAG answer with citations.
4. If retrieval returns nothing, return a refusal. Do not let the model fill the gap from weights.

Hosted file search still needs this contract: the answer must point at the retrieved files, not invent a footnote.

5. Evaluate faithfulness, not demo quality

Ship a small gold set before you scale the corpus.

1. Collect 30–50 questions whose answers exist in the documents.
2. Score answer faithfulness to retrieved text, citation coverage, and empty-retrieval behavior.
3. Log query, retrieved ids, answer, and latency.
4. Re-run the set when you change chunking, embeddings, or the generator model.

Vendor RAG tutorials demonstrate the loop. They do not replace your gold set.

6. Production controls that the tutorials assume you add

Official getting-started pages do not operate your access control.

• Apply ACLs at query time. Do not retrieve another tenant's chunks.
• Version the embedding model and the index together.
• Keep secrets out of the documents you ingest.
• Treat citation links as part of the product, not a debug overlay.

Official sources

• LangChain RAG tutorial: https://python.langchain.com/docs/tutorials/rag/
• LlamaIndex RAG: https://docs.llamaindex.ai/en/stable/understanding/rag/
• OpenAI file search: https://developers.openai.com/api/docs/guides/tools-file-search

Related GenAIWiki pages

• RAG
• Grounded RAG answer with citations
• LangChain vs LlamaIndex
• Vector database