The model is the easy part
A language model answers from what it is given. In a company knowledge assistant, that means the passages the search pulls out of your documents. If the file share contradicts itself, the assistant repeats the contradiction. It does so in a calm, confident tone, which is the dangerous bit.
Your service desk colleague knew not to trust the 2019 PDF. The assistant doesn't. It quotes the old PDF with the same confidence as last week's data sheet. An assistant that sounds sure and is wrong does more damage than having no assistant at all.
Outdated versions. "Pricelist_2024_final_new.xlsx" sits next to "Pricelist_2025.xlsx". A person reads the file names and understands. The search sees two hits with similar content and picks the one whose wording matches the question better. That is not always the current one.
Documents nobody owns. Someone wrote the manual in the "Engineering general" folder, and nobody has opened it since. While it just sits there, it does little harm. The moment an assistant quotes from it, it becomes your company's official answer again.
Permissions nobody remembers. On most file shares, access rights have grown for years without anyone looking at the whole picture. The salary spreadsheet in the wrong folder goes unnoticed because nobody searches there. An assistant searches everywhere it is allowed to.
Start with one question area
The most common mistake is planning to connect all company knowledge at once. Twenty years of file share go into the index, the first answers are patchy, and after two weeks people stop asking. Pick one area instead. Good candidates have three things in common: the same questions come up again and again, the answers already exist in documents, and a wrong answer is annoying rather than harmful.
Define the area in one sentence. For example, the service desk asks about technical specifications, spare parts and availability of our products. Everything else stays out, however politely someone asks. HR topics, contracts and complaints come later, or never.
Choose the documents together. Sit down for two hours with the person who answers these questions today. Go through the folders and ask one question about each document: would you answer a customer from this today? Yes means in. No means out. For a well-defined area you usually end up with 50 to 200 documents. If you have a lot more, the area is too broad.
Clean up before you index. Old versions move to an archive folder the assistant cannot see. What nobody needs any more, you delete. It feels like janitorial work because it is janitorial work. Every old version you leave in place will eventually become a wrong answer given to someone who believes it.
Give every document an owner. Each document gets a named person, not a department, plus a date on which they will look at it again. When the assistant quotes a document wrongly or can't answer a question in its area, that goes to the owner. Without a name, nobody takes care of it and the collection goes stale all over again.
Access rights, sources and unanswered questions
Three rules decide whether your people still trust the assistant after a month. A fourth applies only in Germany, but there it applies for certain.
Reuse the permissions you already have. For every question, the assistant checks which documents the person asking is allowed to open, and searches only those. In practice it reads the permissions from SharePoint, Google Drive or the file server rather than keeping its own list. A separately maintained access list drifts away from the real one within months. The rule is that nobody sees a document through the assistant that they couldn't open in their own file browser. Before launch, check the permissions on the selected documents with whoever is responsible for IT.
Every answer names its source. Below each answer sit the document, the passage and a link. Anyone can check the answer with one click. If there is no source, there is no answer. When the assistant finds nothing relevant, it says so: "I can't find anything on this in the documents I have access to." That sentence is worth more than any clever guess.
Collect the questions it couldn't answer. Every question without an answer goes on a list, and the document owners go through it once a week. Either a document is missing, or one is badly written, or the question doesn't belong in this area. For the first time you get a written record of what your people want to know and can't look up anywhere.
Talk to the works council early, in Germany. German companies with a Betriebsrat, the elected works council, need to watch the question log. If you store each question with the name of the person who asked it, you have a technical system capable of recording employee behaviour or performance. Under section 87, paragraph 1, number 6 of the German Works Constitution Act, the BetrVG, the works council has a right of co-determination over such systems. The Federal Labour Court treats objective suitability for monitoring as enough. Whether you intend to monitor anyone doesn't matter. Log questions without names, and talk to the works council before launch, not after. What a works agreement on tools like this usually contains is laid out in the guide to AI and works councils.
Test it with 30 real questions
Before the assistant goes to everyone, it sits an exam. The exam uses questions people actually asked, not questions someone makes up at their desk. Invented questions are too tidy. They use exactly the words that appear in the documents, which makes the search look better than it is.
Write down questions for two weeks. The service desk notes every question in the area, in the asker's own words, abbreviations and typos included. Pick 30 of them. Add a few that have no answer anywhere in the documents, and a few aimed at a document the tester isn't allowed to open.
Write the correct answers first. Before the test runs, the document owners record the correct answer and the correct source for each question. If you judge afterwards, almost every answer looks plausible.
Grade strictly. Each answer goes into one of three columns. Correct with the correct source. An honest "I don't know". Wrong, or correct with the wrong source. The third column is the one that matters. For the questions with no answer in the documents, "I don't know" is the right result. For the restricted documents, the assistant must reveal nothing from them.
Fix the document, then run it again. Trace every wrong answer back to the document that caused it. Usually it is an old version or two files that contradict each other. Correct the document rather than the instructions you give the model, then run all 30 questions again.
Give it to the people who answer first. The first users should be the people who answer these questions today, because they spot a wrong answer immediately. Only once they trust it does it go to the people who used to ask them.
A company knowledge assistant is a clean-up project with a language model on top. That sounds less exciting than the demo, and it decides whether your people are still asking it questions four weeks in. Start with one area and with documents that a named person stands behind. No assistant knows more than the file share it reads from.