How to Turn Your Company Documents Into an AI Knowledge Assistant

Most AI knowledge assistants don't fail because of the model. They fail because of the documents.

Here's how it usually goes. A team connects an assistant to the company drive, asks it about the refund policy, and gets a clean, confident answer. Everyone's impressed. Two weeks later a salesperson asks about a discount exception and the assistant quotes a policy that was retired in the spring, word for word, completely sure of itself. Nobody can work out what went wrong, because nothing did. The assistant found a document, read it, and answered. It just found the wrong one.

That's the part nobody mentions in the demo. The language model is the piece you'll spend the least time on. What decides whether people trust an assistant, or quietly stop opening it, is everything around the model: which documents it reads, how it finds the right paragraph, who it's allowed to show it to, what it does when it doesn't know, and who keeps it current. We build these systems, and in nearly every project that's where the real work turns out to be. Here's how we think about each piece.

Start with the documents, not the technology

Before anyone picks a tool, it's worth opening the shared drive and looking at what's actually in it. Three versions of the approval process. A handbook that contradicts last quarter's slide deck. A scanned contract from 2019 that, to a computer, is just a photograph. A new hire handed that drive would at least ask which file to trust. An assistant won't ask. It'll pick one and answer.

So the first job is triage, and it's less technical than people expect. Decide which document is the source of truth for each topic, give it an owner, and retire the old versions instead of leaving them lying around. Keep informal sources like Slack threads and email chains out of the first release. They hold real knowledge, but nobody has approved them and they often contradict each other, which is a bad foundation for a system that answers with authority.

After that, it's a matter of getting each kind of file into a shape the assistant can read. Notion pages and Google Docs can be pulled in through their APIs. PDFs with columns and tables need a smarter reader, Unstructured is one tool built for exactly this or the tables come out as soup. Scanned files need optical character recognition, such as the open-source Tesseract or Amazon Textract, and a person should check the output, because a misread part number is the kind of error nobody notices until it costs money.

The most important step is also the least exciting one: label everything as it goes in. Every piece of text should carry its file name, link, page, department, version, status, last-modified date, and who's allowed to see it. That labelling is what makes citations, version control and security possible later, and adding it after the fact to a hundred thousand fragments of text is a miserable job.

Why the right paragraph matters more than the right model

There's a name for the approach that works: retrieval-augmented generation, or RAG, first described in a 2020 research paper. The idea is simpler than the name. You don't retrain the model on your documents, that's slow, expensive to update, and the model still makes things up. Instead, when someone asks a question, the system searches your documents, pulls out the few passages that matter, and hands them to the model along with the question. The model's job is to read and write. It looks things up every time instead of trying to remember.

Microsoft's RAG design guide is worth a skim for one reason: notice how little of it is about the model. Almost all of it is about preparing content, searching well, and measuring results. That tells you where the effort belongs.

Two details make a surprising difference. The first is how documents get cut up for searching. Slice them into arbitrary pieces and you split sentences in half, so the fragment mentioning the exception never says what it's an exception to. A good starting point is pieces of a few hundred words, split along paragraphs and headings, with a little overlap so nothing falls through the gap, then adjust based on real questions. The second is that one kind of search isn't enough. Search by meaning is great when someone types "SLA" and the document says "response commitment," but it stumbles on exact things like part numbers and clause references. Plain keyword search is the opposite. Good systems run both and let a re-ranking step pick the best few passages, using a tool like Cohere Rerank. Whatever the model sees is all it has to work with, so if the right passage doesn't make that short list, no clever prompt will rescue the answer.

Who gets to see what

An assistant that's helpful to everyone is a data leak with good manners. Imagine an intern asking an HR question and getting an answer lifted from an executive compensation file. Nobody broke in. A keyword just happened to match. And telling the model, in its instructions, to keep confidential things private doesn't help, because by then it has already read them.

The fix is to handle permissions before the model ever sees anything. Each piece of text carries a note about who's allowed to read it, and when someone asks a question the search is filtered by their role first. Modern search databases support this directly, Qdrant's filtering is one example and the same trick can limit results to documents marked as current, which quietly fixes a lot of the stale-policy problem. Security researchers treat this as a leading risk for AI applications; the OWASP GenAI Security Project lists sensitive information disclosure among its top concerns. It's also worth deciding early how long conversations are logged and which AI providers get to see your content.

Teach it to say "I don't know"

Language models sound sure of themselves even when they shouldn't be. They misread passages, blend two documents together, and when the documents say nothing, they'll produce something that sounds exactly like a rule would. So the instructions need to be blunt: answer only from the passages provided, point to where each claim came from, and if the answer isn't there, say "I can't find this in our documentation." Then test it. Ask questions you know aren't answered anywhere and see whether it admits that or improvises.

Citations are what turn a plausible answer into a checkable one. Link to the page or section, not just the file name, so someone can verify it in ten seconds. When two documents disagree, the assistant should say so and name the owner instead of quietly picking one. And give people a one-click way to flag a wrong answer, those reports are the cheapest improvement you'll ever get.

The part that decides whether it lasts

Launch day is the easy bit. The real test comes around month six, when the assistant is still quoting a policy that changed in the spring because the documents were loaded once by a script someone ran and forgot. The fix is to connect the assistant to your document sources so changes flow through automatically: an edited document gets re-read, a deleted one disappears from the assistant straight away, a new version replaces the old. Then give someone a name and a job as owner of the content. An assistant makes approved knowledge easier to reach. It can't invent an approval process you don't have.

You'll also want to know if it's working, which is harder than it sounds. "Documents indexed" and "questions answered" tell you it's busy, not that it's good. Build a small test set of real questions where you already know the right source, and check whether it finds it, answers correctly and cites the right page. Keep a list of what it couldn't answer, because that list is a map of the gaps in your documentation. And time the original problem before launch. If support staff spend ages hunting for the right procedure today, measure that now, or you'll end up defending the project with a feeling.

Where to start

Don't launch company-wide. Pick two or three well-organized sets of documents with clear owners, build a pilot, lock down permissions and citations before anyone outside the pilot group sees it, and let a few people try to break it. Frameworks like LlamaIndex or LangChain make the plumbing quicker, but the plumbing was never the hard part.

And before you spend anything, try a cheap test. Pull fifty real questions from your team's chat history and see how many your current documents can actually answer. If it's most of them, you have a retrieval problem, and everything above will help. If many can't be answered at all, you have a documentation problem and no assistant, however clever, will fix that. It's worth finding out which one you have before you build anything.

Murphy Repos builds AI knowledge assistants around the documents, permissions and workflows a business already has. If your team keeps losing afternoons to answers that exist somewhere, book a consultation and we'll walk through where to start.

Frequently asked questions