Choose a narrow knowledge domain, preserve source ownership and permissions, evaluate retrieval before answers, and give outdated content an explicit retirement path.
1. Start with a domain and real questions
Choose a collection with an owner and a clear audience: product manuals for support agents, approved operating procedures, or employee guidance for a particular team. Ask those users which questions take time to answer and where the answers should come from.
Gather representative documents and a set of questions in parallel. Include short articles, long documents, tables, and the formats people actually maintain. Record which passage should support an answer. Questions that cannot be answered from the collection are equally important; the assistant needs a useful “not found” behavior.
2. Prepare content without losing its meaning
Give each source a stable identifier, title, owner, version or update date, and canonical location. Extract text while preserving useful headings and table relationships. Repeated headers, footers, or unrelated navigation text can make search results harder to use.
Split content into retrievable passages based on its structure and the questions you expect. A paragraph without its product name or section heading can become misleading. Keep enough surrounding context to interpret the passage, and keep the source identifier with every indexed unit. Try different splitting approaches on the same test questions before choosing one.
3. Apply permissions before information reaches the answer
Determine which user or team can read each document and preserve that information through ingestion and retrieval. Filter the searchable material for the signed-in user, or enforce an equivalent boundary in the retrieval service. Do not retrieve another team’s confidential passage and rely on a prompt to suppress it.
Plan updates and deletion across the whole system. Removing an original file may leave extracted text, embeddings, caches, or conversation history elsewhere. Document where copies exist and how a permission change or deleted source affects them. These decisions also shape the operating workload.
4. Evaluate retrieval and generation separately
First check whether the expected passage appears in the search results. If it does not, investigate ingestion, vocabulary, metadata, filters, and ranking. A more elaborate answer prompt cannot recover a passage the application never supplied.
Then review the answer: does it address the question, reflect the source accurately, and show useful references? A citation alone does not prove that every claim is supported. Test ambiguous names, conflicting versions, missing information, and questions posed by users with different permissions. Repeat a sample of cases to understand output variation.
On small screens, scroll the table sideways to see every column.
| Case | Expected result | Check |
|---|---|---|
| Current policy question | Supported answer with the current source | Correct version and passage |
| Restricted team document | No unauthorized content | Retrieval access boundary |
| Unsupported question | Useful clarification or not-found response | No invented company policy |
5. Give knowledge an operating owner
Define how new sources are approved, how updates trigger reindexing, and how failed ingestion becomes visible. Show users the source title and an accessible location, and provide a way to report a wrong or stale answer.
Review patterns in unsuccessful searches with the knowledge owner. Sometimes the best fix is improving a confusing document or adding missing information. Track index freshness, retrieval failures, answer-review results, and usage costs so the assistant remains useful after its launch.
6. Frequently asked questions
Do we need a vector database?
You need a retrieval approach that works for your content and queries. Compare keyword, vector, and hybrid search where appropriate. The evaluation result matters more than choosing a particular database category.
Can we upload every company file at once?
A smaller, owned collection is easier to evaluate and maintain. Expand after you understand source quality, permissions, update handling, and the actual questions users ask.
Will source references make every answer correct?
No. The assistant can retrieve the wrong source or interpret a correct passage badly. Check the passage and the claims separately, and accept not-found responses when evidence is insufficient.
Explore free planning tools or Discuss a knowledge assistant.
7. Related articles and next steps
Bring representative inputs, expected outputs, and the systems involved. Discuss your requirements with OdooMatrix.