What is RAG?
RAG stands for retrieval augmented generation. It is the standard way to make a language model answer questions about information it was never trained on, which in practice means almost everything specific to a company: contracts, procedures, product data, support history, internal documentation.
The principle is simple. Instead of hoping the model knows something, you find the relevant passages yourself and hand them over with the question. The model then has one job, writing a good answer from evidence it has been given, instead of two, remembering and then writing.
That separation is the whole value of the approach. It also explains the most common misunderstanding about RAG: the quality of the answer is decided before the model is even called.
Why RAG matters
Four reasons made it the default architecture in a business context.
- It works on information that moves. Update a document and the next answer changes. No retraining, no waiting, no cost per update.
- It makes answers verifiable. Because passages are retrieved explicitly, the system can cite the document it used, which turns an assistant into something an auditor will accept.
- It limits exposure. Only the retrieved passages leave your systems for a given question, not the whole document set.
- It costs far less than the alternatives. Retrieving five passages costs a fraction of sending an entire corpus with every question, and the gap widens with each additional user.
The consequence is organisational as much as technical. A RAG system that works forces a company to decide which document is authoritative, a question most organisations have quietly avoided answering.
How it works
Four stages, and each one is a place where quality is won or lost.
Chunking splits documents into passages. Too small, and a passage loses the context that gives it meaning. Too large, and the model receives noise around the useful sentence. This is the most underestimated decision in the pipeline.
Embedding converts each passage into a numerical representation that captures its meaning, so similarity can be measured rather than guessed from keywords. Those representations are stored in an index built for that kind of comparison.
Retrieval compares the incoming question against the index and returns the closest passages. Good systems filter as much as they rank, using metadata such as date, client, language or document version, so that a superseded procedure never competes with the current one.
Generation injects the retrieved passages into the prompt alongside the question, with an instruction to answer only from what was provided and to say so when the evidence is missing.
The failure mode follows directly from that order. When a RAG system gives a wrong answer, retrieval is at fault far more often than the model. If the right passage never reaches the top results, no model can recover the situation.
Implementation
The order matters more than the tooling.
- Pick one document set with a clear owner and a known level of authority. Not the whole intranet.
- Clean it before indexing. Remove duplicates and superseded versions. A retrieval system will happily return an obsolete document with complete confidence.
- Add metadata for date, version, language and scope, because filtering is what prevents the majority of wrong answers.
- Tune chunking against real questions, not a generic setting copied from a tutorial. Measure whether the right passage appears in the top results.
- Allow the model to abstain. An answer of the form I do not have that information is a feature, not a failure.
- Show the sources in the interface, so a user can verify in one click and trust builds instead of eroding.
Related technologies and tools
- Vector storage: the index that holds passage representations and answers similarity queries efficiently.
- Embedding models: the component that turns text into the numerical form compared during retrieval.
- Rerankers: a second pass that reorders retrieved passages by relevance, often the cheapest available quality improvement.
- Hybrid search: combining keyword matching with similarity search, which catches exact references such as product codes that similarity alone misses.
- Evaluation sets: a fixed set of real questions with their expected sources, replayed after every change, so quality becomes measurable rather than anecdotal.
Conclusion
RAG is a document retrieval problem disguised as an AI topic. Teams that treat it as a model problem spend months switching providers and tuning prompts while the actual fault sits in chunking, in metadata, or in an index nobody refreshed.
The reliable path is unspectacular: one document set with a clear owner, cleaned before indexing, enriched with metadata, chunked against real questions, with sources shown in the interface and permission for the system to admit it does not know. That version earns trust within weeks. The clever version that skips document preparation usually never earns it at all.
