Skip to content
caadoo
de

AI assistant for company documents: fix files first

Most companies planning an AI assistant for their own documents spend their time choosing a model. The real work is in the file share, with its outdated versions, documents nobody owns and access rights nobody has checked in years. Here is the order to do it in, from one bounded question area to a test with 30 real questions before anyone else gets access.

By T. Krause8 min read

Every company over a certain size has a person who knows where things are. Someone in the service desk who can tell you, without looking, which spare part fits the pump from the old product line and whether it still ships. When that person is on holiday, the questions pile up. When they retire, the answers go with them.

So the idea comes up in a management meeting: an AI assistant that has read all our documents and answers these questions in seconds. The technique behind it is called retrieval. For each question, the system searches your files for the most relevant passages and has the language model answer from those passages only. It works. Building it is the smaller part of the job.

The bigger part is your file share. If it holds three versions of the price list, the assistant finds all three and has no idea which one is current. Quite often nobody in the building knows for sure either. Most of what follows is about files.

Photo: Unsplash

The model is the easy part

A language model answers from what it is given. In a company knowledge assistant, that means the passages the search pulls out of your documents. If the file share contradicts itself, the assistant repeats the contradiction. It does so in a calm, confident tone, which is the dangerous bit.

Your service desk colleague knew not to trust the 2019 PDF. The assistant doesn't. It quotes the old PDF with the same confidence as last week's data sheet. An assistant that sounds sure and is wrong does more damage than having no assistant at all.

Outdated versions. "Pricelist_2024_final_new.xlsx" sits next to "Pricelist_2025.xlsx". A person reads the file names and understands. The search sees two hits with similar content and picks the one whose wording matches the question better. That is not always the current one.

Documents nobody owns. Someone wrote the manual in the "Engineering general" folder, and nobody has opened it since. While it just sits there, it does little harm. The moment an assistant quotes from it, it becomes your company's official answer again.

Permissions nobody remembers. On most file shares, access rights have grown for years without anyone looking at the whole picture. The salary spreadsheet in the wrong folder goes unnoticed because nobody searches there. An assistant searches everywhere it is allowed to.

Start with one question area

The most common mistake is planning to connect all company knowledge at once. Twenty years of file share go into the index, the first answers are patchy, and after two weeks people stop asking. Pick one area instead. Good candidates have three things in common: the same questions come up again and again, the answers already exist in documents, and a wrong answer is annoying rather than harmful.

Define the area in one sentence. For example, the service desk asks about technical specifications, spare parts and availability of our products. Everything else stays out, however politely someone asks. HR topics, contracts and complaints come later, or never.

Choose the documents together. Sit down for two hours with the person who answers these questions today. Go through the folders and ask one question about each document: would you answer a customer from this today? Yes means in. No means out. For a well-defined area you usually end up with 50 to 200 documents. If you have a lot more, the area is too broad.

Clean up before you index. Old versions move to an archive folder the assistant cannot see. What nobody needs any more, you delete. It feels like janitorial work because it is janitorial work. Every old version you leave in place will eventually become a wrong answer given to someone who believes it.

Give every document an owner. Each document gets a named person, not a department, plus a date on which they will look at it again. When the assistant quotes a document wrongly or can't answer a question in its area, that goes to the owner. Without a name, nobody takes care of it and the collection goes stale all over again.

Access rights, sources and unanswered questions

Three rules decide whether your people still trust the assistant after a month. A fourth applies only in Germany, but there it applies for certain.

Reuse the permissions you already have. For every question, the assistant checks which documents the person asking is allowed to open, and searches only those. In practice it reads the permissions from SharePoint, Google Drive or the file server rather than keeping its own list. A separately maintained access list drifts away from the real one within months. The rule is that nobody sees a document through the assistant that they couldn't open in their own file browser. Before launch, check the permissions on the selected documents with whoever is responsible for IT.

Every answer names its source. Below each answer sit the document, the passage and a link. Anyone can check the answer with one click. If there is no source, there is no answer. When the assistant finds nothing relevant, it says so: "I can't find anything on this in the documents I have access to." That sentence is worth more than any clever guess.

Collect the questions it couldn't answer. Every question without an answer goes on a list, and the document owners go through it once a week. Either a document is missing, or one is badly written, or the question doesn't belong in this area. For the first time you get a written record of what your people want to know and can't look up anywhere.

Talk to the works council early, in Germany. German companies with a Betriebsrat, the elected works council, need to watch the question log. If you store each question with the name of the person who asked it, you have a technical system capable of recording employee behaviour or performance. Under section 87, paragraph 1, number 6 of the German Works Constitution Act, the BetrVG, the works council has a right of co-determination over such systems. The Federal Labour Court treats objective suitability for monitoring as enough. Whether you intend to monitor anyone doesn't matter. Log questions without names, and talk to the works council before launch, not after. What a works agreement on tools like this usually contains is laid out in the guide to AI and works councils.

Test it with 30 real questions

Before the assistant goes to everyone, it sits an exam. The exam uses questions people actually asked, not questions someone makes up at their desk. Invented questions are too tidy. They use exactly the words that appear in the documents, which makes the search look better than it is.

Write down questions for two weeks. The service desk notes every question in the area, in the asker's own words, abbreviations and typos included. Pick 30 of them. Add a few that have no answer anywhere in the documents, and a few aimed at a document the tester isn't allowed to open.

Write the correct answers first. Before the test runs, the document owners record the correct answer and the correct source for each question. If you judge afterwards, almost every answer looks plausible.

Grade strictly. Each answer goes into one of three columns. Correct with the correct source. An honest "I don't know". Wrong, or correct with the wrong source. The third column is the one that matters. For the questions with no answer in the documents, "I don't know" is the right result. For the restricted documents, the assistant must reveal nothing from them.

Fix the document, then run it again. Trace every wrong answer back to the document that caused it. Usually it is an old version or two files that contradict each other. Correct the document rather than the instructions you give the model, then run all 30 questions again.

Give it to the people who answer first. The first users should be the people who answer these questions today, because they spot a wrong answer immediately. Only once they trust it does it go to the people who used to ask them.

A company knowledge assistant is a clean-up project with a language model on top. That sounds less exciting than the demo, and it decides whether your people are still asking it questions four weeks in. Start with one area and with documents that a named person stands behind. No assistant knows more than the file share it reads from.

Topics

  • AI assistant
  • Knowledge management
  • Documents
  • Customer service
  • Retrieval
All articles

Keep reading

  • 7 min read

    Shadow AI in the workplace: find it, don't ban it

    Many managing directors either assume nobody in their company uses AI or plan to ban it. Meanwhile staff are pasting customer data into private accounts, often on their own phones. This piece shows how to find out what's really happening and bring it under control with one sanctioned tool, short rules and, in Germany, the works council.

  • 8 min read

    AI literacy under Article 4: what companies must do

    Most mid-sized companies file the EU AI Act under problems for software vendors. Article 4 applies to any company whose staff use ChatGPT or Copilot at work. This piece covers what the obligation asks for, what it doesn't, and how to meet it without hiring a consultancy.

  • 7 min read

    Replacing Excel with an internal tool, step by step

    Plenty of companies depend on one Excel file that nobody dares to touch, and the usual plan is to replace it with new software in one go someday. That plan tends to break things. This article shows how to tell when a spreadsheet has become a system, what to pin down before you change anything, and how to move it into a small internal tool step by step.

End of call

You can hang up now.

Or book 15 minutes. I'll ask you the questions I'd ask on the phone, and tell you at the end whether it fits. If it doesn't, we've both lost a quarter of an hour instead of a month.

Or call. Cold, if you like. My AI assistant picks up and I call you back. +49 5136 9414999

Analytics

With your consent I use Google Analytics to see which pages get read. Without it, no analytics code loads. Your answer is stored locally in your browser.

Privacy notice