"Can we have a ChatGPT that knows our company?" It is one of the most common requests we hear. Teams want to ask questions in plain language and get answers drawn from their own manuals, policies, contracts, and wikis, with a link to the source.

The usual way to build this is called retrieval-augmented generation, or RAG. This guide explains how it works without the jargon and, more importantly, what you need to have in place before it can work well.

How RAG works, in plain terms

A language model on its own only knows what it learned during training. It does not know your pricing, your internal processes, or last month's policy change. RAG solves this in three steps:

  1. Prepare. Your documents are split into small passages and indexed so they can be searched by meaning, not just by keywords.
  2. Retrieve. When someone asks a question, the system finds the passages most relevant to it.
  3. Answer. The model receives the question together with those passages and writes an answer based on them, ideally citing where each part came from.

The model is not memorizing your documents. It is reading the right pages at the moment of the question, which means you can update the content at any time without retraining anything.

Good use cases

  • Internal knowledge assistants for HR policies, IT help, onboarding, and how-to questions
  • Support teams finding the right answer across a large help center
  • Sales and pre-sales looking up product specifications and approved wording
  • Operations and compliance searching procedures, contracts, and standards
  • Customer-facing help that answers from your public documentation

What decides whether it works

In practice, the biggest factor is rarely the model. It is the content and the setup around it.

1. Content quality

If two documents disagree, the assistant may quote either one. Before building, identify the authoritative sources, archive outdated versions, and fill obvious gaps. A smaller set of trustworthy documents beats a large pile of mixed-quality ones.

2. Document formats

Clean text, web pages, and well-structured documents are the easiest to work with. Scanned PDFs, complex tables, slides, and images need extra processing and are worth testing early, because they are where answers most often go wrong.

3. Access permissions

Not everyone should see everything. If your documents have different audiences, such as management-only files or specific departments, the system must respect those permissions when it retrieves content. Decide this before you build, not after.

4. Keeping it fresh

Plan how new and changed documents will flow into the index. Manual uploads work for a pilot. For anything long-term, connect to where the documents actually live so updates happen automatically.

5. A way to measure quality

Write down a set of real questions with their correct answers before you start. Run them every time you change something. This simple test set is the most reliable way to know whether a change made answers better or worse.

Questions to answer before you build

  • Who will use it, and what questions do they ask most often?
  • Which documents are the source of truth, and who owns them?
  • Are there permission rules the system must follow?
  • Should every answer show its sources? (For most business uses, yes.)
  • What should the assistant say when it cannot find an answer? ("I don't know" is far better than a confident guess.)
  • Where will people use it: a web page, a chat tool your team already uses, or inside an existing product?

A practical preparation checklist

Before choosing a tool or model, make a small inventory for the first pilot:

  1. Choose one audience and task. For example, a support team checking the current returns policy.
  2. Name the source of truth. Identify which approved documents answer those questions, who owns them, and how updates are published.
  3. Map access rules. List which people may see each collection and test that restricted content stays restricted.
  4. Collect a test set. Include real, anonymized questions with expected sources, ambiguous wording, and questions with no answer in the documents.
  5. Agree on review and handoff. Decide how answers show sources, what happens when evidence is missing, and who reviews mistakes.

For example, if a policy does not state whether an exception applies, the assistant should say the source does not answer the question and direct the user to a person instead of inventing a rule.

Common pitfalls

  • Indexing everything at once. Start with one well-understood collection of documents and expand once answers are reliable.
  • No source citations. Without them, users cannot check answers and trust erodes quickly.
  • Testing only with easy questions. Include tricky, ambiguous, and out-of-scope questions in your test set.
  • Treating it as a one-off project. Content changes, so someone needs to own quality over time.

Getting started

A focused pilot is the fastest way to learn: one team, one document collection, a few dozen real test questions, and a simple interface. Within a short time you will know how well your content supports AI answers and what needs fixing before a wider rollout.

If you are considering an assistant built on your own documents, we can help you assess your content, design the pilot, and build something your team will trust.

Explore our AI agents and knowledge assistant service or discuss your knowledge-base project.