AI · Learning

What is RAG?

A simple explanation of Retrieval-Augmented Generation, how it works, why it is useful, and when you actually need it.

By Renaldy Prasetio · · 4 min read

Caveman TL;DR

AI forget. AI guess. RAG make AI look at right page first. Then AI answer.

What is RAG?

RAG, or Retrieval-Augmented Generation, is a way of giving an AI relevant information before it generates an answer.

LLMs are always giving answers confidently. Sometimes it’s useful, sometimes it’s a problem. (Something that Inkling is trying to solve, but that’s a story for a different day.)

Ask them a question and they can hallucinate and give you the wrong answer. At the end of the day, they are basically autocomplete on steroids.

This is where RAG comes in handy.

RAG stands for Retrieval-Augmented Generation, which simply means: before giving an answer, look for the right information first.

Go back to school homework. Your biology book asks you, “What is mitochondria?”

You open the book, find the useful page, read it, and then find the answer.

This is basically how RAG works.

With RAG, LLMs are asked to find the answer from a knowledge base. This could be anything: books, documents, notes, or company guides.

Using the same example as before, the system first searches the book, finds the right part, and only that part is given to the AI to help it write the answer.

LLMs are smart, so we could probably just give them the whole book. But imagine if you have 100 books or 10,000 company documents.

You probably wouldn’t give your new intern all 10,000 documents just to find out:

“How many days of personal leave do I get?”

You would just give them the page about personal leave.

Ask about personal leave? Give them that part.

Ask about getting a new monitor? Give them the part about office equipment.

Your intern doesn’t need all 10,000 documents. They just need the right page.

Same with AI.

How does RAG work?

Basically, we cut the book or document into smaller sections. This is called chunking.

By splitting the document into smaller sections, we make it easier for the system to search and find the right information.

The simple RAG flow:
  1. A user asks a question.
  2. The system searches the knowledge base.
  3. It finds the most relevant chunks of information.
  4. Those chunks are given to the LLM as context.
  5. The LLM uses that information to generate an answer.

So instead of asking the AI to answer entirely from memory, RAG lets it look up relevant information first.

Why is RAG useful?

  1. Give the AI information it was never trained on. For example, the technical docs for your own app.
  2. Make answers easier to verify. The system can tell us where the information came from.
  3. Update information easily. You can update the knowledge base without retraining the AI.

When should you use RAG?

RAG is useful when the AI needs access to information that is private, large, frequently updated, or not already available in its training data.

  • Company guidebooks
  • Technical documentation
  • Private documents or notes
  • Large collections of books or research papers
  • Policies and procedures
  • Product manuals
  • Internal company knowledge bases

When should you not use RAG?

You usually don’t need RAG when all the information needed to complete the task is already available in the prompt.

  • Short data, like one email
  • Summarising a short document
  • Rewriting or fixing grammar
  • Translating text
  • Analysing a transcript that you already gave to the AI
  • Extracting information from a small piece of text
  • Simple questions where all the information is already in the prompt

The simple way to think about RAG

RAG doesn’t make the LLM magically know everything.

It gives the LLM a way to find relevant information before answering.

Instead of giving your AI 10,000 documents and hoping it figures everything out, find the right page first and let the AI answer using that.