What Is RAG? Retrieval-Augmented Generation Explained
Learn how Retrieval-Augmented Generation (RAG) connects AI models to your own documents, so answers are current, grounded and can cite where they came from.

If you ask a standard AI model to summarise your company's new HR policy, it will fail. It might invent a plausible-sounding policy, or apologise and say it doesn't know.
That's because a model only knows what was in its training data, up to a cut-off date. It has never seen the documents in your Google Drive, your help centre or your internal wiki.
RAG (retrieval-augmented generation) fixes that. Before the model answers, the system looks up the most relevant passages in your own documents and hands them to the model along with the question. The model then answers from that material instead of from memory.
RAG turns a closed-book exam into an open-book one. The AI still writes the answer, but it gets to look up the right pages first, and it can tell you which pages it used.
How RAG works
RAG has two halves: preparing your documents once, and answering each question.
Preparing the documents (once, then whenever they change)
- Collect: gather the sources the AI may use: PDFs, help articles, Notion pages, support tickets, product specs.
- Chunk: split each document into short passages, typically a few paragraphs each. Small chunks make it possible to pull out just the part that answers a question.
- Embed: turn each chunk into an embedding, a list of numbers that captures what the text means.
- Store: save the chunks and their embeddings in a vector database, along with details like the source, date and who's allowed to see it.
Answering a question (every time)
- Embed the question the same way the chunks were embedded.
- Retrieve: find the chunks whose meaning is closest to the question. This is semantic search: it matches meaning, not just exact words, so "Can I get my money back?" finds a passage titled "Refund policy". Closeness is usually measured with cosine similarity.
- Augment: put the best few chunks into the prompt, together with an instruction such as "Answer only from these sources and cite them."
- Generate: the model writes the answer, ideally pointing to the chunks it used.
A worked example
Say a customer asks your support assistant: "I bought the annual plan three weeks ago. Can I still get a refund?"
- Without RAG, the model guesses from general knowledge about software refunds. It might say 14 days, or 30, or "contact support".
- With RAG, retrieval finds two chunks: your refund policy ("Annual plans can be refunded in full within 30 days of purchase") and a note on how refunds are processed. The model answers: "Yes. Annual plans can be refunded within 30 days, so you're still eligible. Refunds go back to the original payment method within 5–10 business days." It links to the policy page it used.
If you change the policy tomorrow, you update one document. The next answer uses the new wording, with no retraining.
Why teams use RAG
Answers grounded in your own information
The model answers from documents you control, which cuts down hallucinations (confident, made-up answers). It doesn't remove them entirely: a model can still misread a passage or fill gaps around it, so important answers still need checking.
Always up to date
Training or fine-tuning a model is slow and costly, and the knowledge is frozen when it finishes. With RAG, updating the knowledge means updating a document.
Sources you can check
Because the system knows which chunks it retrieved, it can show citations. A reader can click through and confirm the answer, which builds trust. (Read more in Can you trust AI-generated citations?)
Respecting permissions
A well-built RAG system filters by access before retrieving, so a new employee's question never pulls in documents only finance is allowed to see.
RAG vs. Fine-Tuning
Wondering whether to use RAG or train your own model? Read our comparison of RAG, fine-tuning and prompt engineering.
Where RAG goes wrong
Most problems with RAG come from retrieval, not from the model.
- The right passage isn't found. If chunks are too long, too short or split in the wrong place, the answer may be spread across chunks that never get retrieved together.
- The wrong passage is found. An outdated policy that's still in the index can outrank the current one. Old and duplicate documents need cleaning up.
- Too much is retrieved. Stuffing many chunks into the prompt uses up the context window and costs more tokens, and the model may focus on the wrong one.
- Some questions aren't search questions. "How many refunds did we issue last month?" needs a database query, not a paragraph. "Summarise every contract" needs all the documents, not the top three.
- Retrieved text can carry instructions. A document that says "ignore previous instructions" is a prompt injection risk. Treat retrieved text as information, never as commands.
Teams improve results by tuning chunk size, combining semantic search with keyword search, re-ranking the results before they reach the model, and testing with real questions.
When not to use RAG
- The model already knows the answer. General knowledge ("What is a CSV file?") doesn't need retrieval.
- You need a new skill or style, not new facts. Teaching a model a format, tone or task is a job for prompting or fine-tuning.
- The data is structured. Numbers in a database are better answered with a query (often through tool calling) than by searching text.
- There's very little content. If everything fits comfortably in the prompt, simply include it.
Agentic RAG
Early RAG was a single pass: search once, then answer. Many systems now let an AI agent drive retrieval. The agent decides what it needs, searches, reads the results, rephrases and searches again if the first attempt missed, and can combine sources, such as a vector database, a live SQL database and a knowledge graph of how things relate.
This handles harder questions but is slower and costs more, so it's worth using where a single search regularly fails. It's also how agents keep a long-term memory; see How AI agent memory works.
Frequently asked questions
What does RAG stand for?
Retrieval-augmented generation: the system retrieves relevant text, augments the prompt with it, then the model generates the answer.
Does RAG stop AI hallucinations?
It reduces them by giving the model the right information, but it doesn't eliminate them. Good RAG systems also tell the model to say "I don't know" when the sources don't contain the answer, and show citations so people can check.
Do I need a vector database for RAG?
Most RAG systems use one, because it makes semantic search fast. Small projects can start with a simple keyword search, or a database that supports both.
Is RAG better than fine-tuning?
They solve different problems. RAG adds knowledge that changes; fine-tuning changes how a model behaves. Many products use both.
Does RAG keep my data private?
RAG itself doesn't send your data anywhere new beyond the model you call, but the retrieved passages are part of each request. Check how your model provider handles data, and filter by permissions before retrieval.
Summary
- RAG looks up relevant passages in your own documents and gives them to the model before it answers.
- It relies on chunking, embeddings, a vector database and semantic search.
- It makes answers current and checkable and reduces made-up facts, but doesn't remove them.
- Most failures are retrieval failures: wrong, missing, outdated or too many chunks.
- Use it for knowledge that changes; use prompting or fine-tuning for behaviour and style.

Stephen Gbolagade
Software Engineer and Technical Writer at Ploba
Writes Ploba's guides to AI tools, agents and protocols, explaining how they work in plain language.
Keep learning
Plain-language explanations, technical guides and practical comparisons.


