Here’s one. Monday morning, someone new to the team drops a question in the general channel: “How many days ahead do I need to request unpaid leave?” Easy question. It got answered on Wednesday.
The answer lived in three places. The main rule sat inside a PDF nobody could find the link to. The exception lived in a Slack thread from ten months ago. And how things actually work lived in the HR lead’s head — she was travelling for two days.
Sound familiar?
That’s one of the reasons we started building an internal RAG system rather than another pile of shared drives. Not to have a chatbot answer for show, but so questions like that one get an answer in thirty seconds — with a source, with permissions respected, and updated when the documents change.
What is RAG?
RAG stands for Retrieval-Augmented Generation. The name sounds technical. The idea is ordinary.
Picture a genuinely smart new hire. Reads fast, writes well, explains things clearly. But they joined this week, so they have no idea how your company handles refunds.
You have two ways to use that person. Option one: ask them to memorise every policy, contract and meeting note from the last three years. Not happening. Option two: hand them a very good index and teach them how to look things up. That works.
RAG picks option two. The system breaks your documents into small pieces (chunks), turns each chunk into a numeric vector (embedding), and stores those in a vector database. When a question arrives, it finds the closest-matching chunks, puts them into the prompt, and the language model answers — using only what was just handed to it. That’s it.
So why not paste every document into a chatbot and tell it to remember?
- The context window is finite. A few thousand pages in a single question is not a thing.
- Documents change weekly. Re-learning everything after each edit is the fastest route to answering with stale information.
- You cannot trace the source. When the bot says “employees get 12 days”, you need to know where that number came from.
- You cannot enforce permissions. A new hire and someone in payroll should not see the same document set.
RAG splits the job in two: retrieval and generation. Whichever part breaks, you fix that part. Far easier than trying to correct a model that learned the wrong thing.
Building an internal RAG system: the 5 steps we usually take
1. Scope the documents — and say no to most of them.
This is the most underrated step, and the one that decides everything. We’ve watched teams want to load it all: the internal wiki, personal Drives, eight years of email, the scanned contracts folder. The result answers a lot of things, none of them trustworthily.
Our approach is to start from questions, not from documents. Ask employees: what did you go looking for last week? Then pick the three to five topics with the highest volume of repeat questions — HR, internal process, product specs, sales policy. Load only that.
2. Clean and chunk the documents.
Real internal documents are messy. Headings are inconsistent. Tables break when they leave Word. Screenshots have no text. Tables of contents float free. This step consumes the most time and there is no way around it.
Chunking takes some judgement too. Chunks that are too large dilute the signal and the model latches onto the wrong point. Chunks that are too small lose context — one sentence pulled out of a clause can mean the opposite of the clause. We usually chunk along the original structure (section, article, part) and let chunks overlap slightly, so answers don’t get sliced mid-sentence.
3. Pick the embedding model and where vectors live.
For Vietnamese content, this is where things go wrong fastest if you default to an English-first embedding model. The quick test we still run: take a few dozen real questions with their known correct answers, and check whether the model ranks the right chunk first.
Where the vectors live depends on scale. A few thousand chunks need nothing fancy. Once you’re into hundreds of thousands, or you need to filter by department before searching, that’s when the heavier options earn their place.
4. Design the answering prompt.
The prompt is where we set the ground rules. Three matter most:
- Answer only from the passages provided.
- If the passages aren’t enough, say so plainly — and point to who to ask instead.
- Always name the document and section the answer came from.
The second rule is the hardest to sell. Nobody wants an assistant that says “I don’t know”. But an assistant that admits it doesn’t know stays useful for years. An assistant that guesses gets abandoned the first time it’s confidently wrong.
5. Measure with real employee questions.
Don’t measure with a question set you invented yourself. Pull real questions from internal chat logs, tickets and shared inboxes. Track two separate things: did the system retrieve the right chunk, and was the answer correct?
These are different failures. If retrieval is wrong, the problem is in chunking or embeddings. If retrieval is right but the answer is wrong, the problem is in the prompt. Telling those two apart saves an enormous amount of debugging time.
RAG for querying internal documents: always cite the source
A system that answers correctly but won’t show its work is still not usable. The person asking needs to know where the answer came from — because internal documents come in versions, and the newest version is the correct one.
So in every RAG for querying internal documents setup we build, citations are attached to the answer itself: document name, section number, last updated date, and a link that opens the original. One click and the claim is verified.
Alongside citations sits permissions. This is where pretty demos collapse inside a real company. A system that simply searches and answers will happily answer a payroll question for anyone who types the right keywords.
Our approach: filter by role at the retrieval layer, before the model ever sees the data — not by instructing the prompt to “please don’t reveal”. A prompt is guidance, not a fence. The fence is the candidate list being narrowed to what the asker is actually allowed to read, before it enters the context.
There’s one more thing people rarely plan for: expired documents. Superseded policies, old templates, replaced rules. Left alone, the system will answer from exactly the version the company retired. We tag each chunk with an effective date, and when documents conflict, prefer the newer one while noting that an older version still exists somewhere.
Traps we have run into
Duplicate documents. The same process saved in four places, each slightly edited. The system retrieves all four and produces a muddled answer. The fix isn’t in the code — it’s picking one official source per topic.
Chunks too big or too small. Too big, and the model rambles, treating a minor detail as the main point. Too small, and it loses context, sometimes answering with the opposite meaning. There is no correct number that works for every document set — you have to test.
Nobody prunes old documents. This is the most expensive mistake. A rule from two years ago, still sitting in the folder, still retrievable, still cited as fact.
Wrong expectations. This is the line we repeat to clients: RAG does not fix dirty data. If a process was never written down, or everyone does it differently, no model can guess your intent. Clarifying the process usually takes longer than building the system — and it’s also the part that creates the most value.
When RAG alone is not enough
RAG is very good at one thing: finding information that exists in your documents. It isn’t good at other things, and knowing that boundary saves a lot of money.
RAG is not enough when:
- The answer requires computation over data that changes constantly — stock levels, today’s revenue, delivery schedules.
- The job is to actually do something, not just answer: create an order, send an email, update the CRM.
- One task needs several steps across several systems, where each step depends on the last.
- The data is too sensitive to leave your infrastructure, so the model has to run locally.
For those cases, RAG is usually one part of a larger system — wired to tools, to multi-step agents, or to a smaller model running on your own hardware. We went deeper in our post on local SLMs and multi-agent systems if you want the details.
Where to start
If you’re considering this, don’t start by picking a model. Start with one folder of the documents your colleagues ask each other about most, and a list of twenty real questions employees keep repeating.
With just those two things, you’ll already know whether your system will be good.
If you’d rather move faster, we can look at it with you. At Mon AI we design and build internal knowledge retrieval systems like this for companies — starting from your process, not from a model that happens to be available. Take a look at how we build agents, or send us a question — even if you’re only curious whether your company’s documents are clean enough to start.
