How an AI knowledge base works, RAG vs. fine-tuning vs. knowledge graphs, internal vs. external use, a 4-step build plan, and a free golden question set.

An AI knowledge base is a collection of your organization's documents that an AI system can search by meaning and use to write direct answers, with links back to the sources it used. Most AI knowledge bases run on retrieval-augmented generation (RAG): the system first finds the passages most relevant to a question, then a large language model (LLM) writes the answer from those passages.
Key takeaways
A traditional knowledge base is a library of articles that people browse or search with keywords. An AI knowledge base keeps that library, but adds a layer that reads it for you. You ask a question in plain language. The system finds the relevant passages, writes a short answer from them, and cites the pages it used so you can check the answer.
The same content can serve people and software. Employees can ask it questions in a chat window, a support widget can use it to answer customers, and AI agents can query it before they act.
| Traditional knowledge base | AI knowledge base | |
|---|---|---|
| How you search | Keywords and categories | A question in plain language |
| What you get back | A list of articles | A written answer with source links |
| How matching works | Exact or near-exact words | Meaning, usually combined with keywords |
| Who uses it | People | People, chatbots, and AI agents |
| What breaks it | Missing keywords, poor navigation | Outdated pages, conflicting pages, weak retrieval |
| What it still needs | Writers and owners | Writers and owners, plus testing of the answers |
The last row matters most. An AI layer does not remove the editorial work. It makes the quality of that work more visible, because a wrong page now turns into a confident wrong answer.
Every AI knowledge base, whether you buy it or build it, has the same five parts:
Vendors compete on parts 2 to 4. Projects usually fail on parts 1 and 5.
The pattern behind most AI knowledge bases is called retrieval-augmented generation. The term comes from a 2020 research paper by Lewis and colleagues, which combined a language model with a searchable document index so the model could look facts up instead of relying on what it memorized during training.
A RAG pipeline has two halves. The first half runs whenever content is added or changed:
The second half runs every time someone asks a question:
Vector search finds meaning. A question about "getting back into my account" can match a page titled "Reset your password" even though the words differ. Keyword search finds exact strings. If someone searches for error code "TS-999", a vector search may return pages about error codes in general, while a keyword search finds the exact match.
That is why strong systems use both. In Anthropic's contextual retrieval experiments, adding a short context sentence to each chunk before embedding cut failed retrievals by 35%. Combining that with keyword (BM25) search cut them by 49%, and adding a reranker brought the reduction to 67%. The tests covered codebases, fiction, and research papers. Your numbers will differ, but the direction holds: hybrid search plus reranking beats vector search alone.
The language model does not hold your policies or product specs. It only sees the passages retrieval hands it for each question. This is the main advantage of RAG: when a policy changes, you update the page, re-index it, and the next answer uses the new version. No retraining is needed.
It is also the main risk. If retrieval returns the wrong passages, or nothing relevant exists, the model may still produce a fluent answer from its general training. Instruct the system to say "I don't know" when the sources do not cover a question, and test that it does.
The cost of getting this wrong is real. In Moffatt v. Air Canada (2024 BCCRT 149), a British Columbia tribunal found the airline liable for negligent misrepresentation after its website chatbot described a bereavement refund process that contradicted the airline's own policy page. The tribunal rejected the argument that the customer should have checked the other page. For a public AI knowledge base, the answer is the policy as far as your customer is concerned.
There are three common ways to give a language model your organization's knowledge. Most teams should start with RAG and add the others only for specific problems.
| RAG | Fine-tuning | RAG plus a knowledge graph | |
|---|---|---|---|
| How knowledge gets in | Retrieved from your documents at question time | Trained into the model's weights | Retrieved from documents and from a map of entities and their relationships |
| Updating facts | Edit the page and re-index | Retrain the model | Edit the page; update the graph on a schedule |
| Tracing an answer to a source | Yes, via the retrieved passages | No | Yes, via passages and graph paths |
| Upkeep | Index and retrieval tuning | Training data and retraining cycles | Index tuning plus graph design and curation |
| Best for | Facts that change; answers that need citations | Fixed formats, tone, and domain style | Questions that connect several entities, such as "which policy applies to this team in this region?" |
RAG is the default for good reasons. Menlo Ventures' 2024 enterprise survey reported RAG at 51% adoption among enterprise AI design patterns, up from 31% the year before, while only 9% of production models were fine-tuned.
Fine-tuning changes how a model writes, not what it can look up. It helps when outputs must follow a strict structure or a specialized style. OpenAI's announcement of GPT-4o fine-tuning cites Cosine reaching state-of-the-art scores on a software engineering benchmark with a fine-tuned model. For a knowledge base, though, facts baked into weights go stale, cannot be cited, and cannot be removed without retraining.
Knowledge graphs store entities, such as products, teams, and policies, together with named relationships like "applies to" or "replaced by." They help with multi-step questions that pure similarity search handles poorly. In an evaluation Lettria published on the AWS Machine Learning Blog, adding graph-based retrieval to RAG improved answer precision by up to 35% across finance, healthcare, industrial, and legal test sets. Lettria sells GraphRAG tooling, so treat the figure as a vendor result. The cost is design work: someone has to define the entity types and keep the graph current.
The same pipeline can serve employees or customers, but the two audiences need different content, permissions, and tone. Decide which one you are building first.
An internal AI knowledge base answers questions from staff using wikis, policies, runbooks, and onboarding guides. Common uses:
The hard requirement is permissions. An employee should never receive an answer built from a page they cannot open directly. If you plan to run models and data inside your own environment, see our guides to private AI for business and self-hosting an enterprise knowledge base.
An external AI knowledge base answers customers in a help center, a support widget, or an agent-assist panel that suggests answers to support staff. It needs a consistent brand voice, strict filtering so internal-only content never appears, answers in the customer's language, and a clear path to a human. For the help center side, see our guides to help desk knowledge bases and customer knowledge bases.
| Requirement | Customer support | Internal operations | Product documentation |
|---|---|---|---|
| How often content changes | Often, with releases and policy changes | Quarterly, or with organizational changes | With every release and API change |
| Access control | Public articles, plus internal-only agent notes | Restricted by team for sensitive topics such as pay or legal matters | Public docs, plus partner or internal-only specs |
| Main AI job | Answer customers directly, with sources | Find answers and draft summaries, guides, and recaps | Draft and update docs to keep pace with releases |
| Success signal | Self-service resolution and customer satisfaction | Fewer repeat questions and faster onboarding | Coverage of shipped features and time to publish |
The strongest setups share one source of truth and add an audience layer on top: separate retrieval filters, permissions, and answer rules for each audience. That avoids maintaining two copies of the same policy.
If you do not have a knowledge base yet, start with our guide on how to create a knowledge base. The steps below assume you have content and want AI answers on top of it.
List where the knowledge for one use case actually lives: wiki pages, help articles, policy PDFs, ticket resolutions, chat threads, and meeting notes. For each source, note the owner, the last review date, and whether it conflicts with another source. Then pick a single starting use case, such as "answer IT questions for new hires." A narrow scope gives you a test set you can finish and a result you can measure.
Also write down what the system must not answer. Pricing exceptions, legal advice, and anything involving personal data are common examples.
You have three practical paths:
Whichever path you choose, confirm two locations separately: where documents are stored, and where AI processing runs. Local or self-hosted storage does not keep content in your environment if the AI feature sends it to an external model provider.
Most retrieval problems are content problems. Merge duplicate pages, retire outdated ones, and rewrite sections that only make sense in context. The next section covers the specific rules. If the library itself is not organized yet, start by building a team knowledge base with owners and review dates before you connect AI.
A golden question set is a fixed list of real questions with the answer and source you expect for each. Run it before launch and after every significant change, so you can tell whether a change helped or hurt.
Build the set from real tickets, chat threads, and search logs, and include more than easy lookups:
| Question type | What it checks | Expected behavior |
|---|---|---|
| Common questions | Everyday accuracy | Correct answer with the right source |
| Rare, high-risk questions | Policy, pricing, and safety topics | Correct answer, or a hand-off to a person |
| Not covered by the sources | Hallucination | "I don't know", with a suggestion of where to ask |
| Permission checks | Restricted content | No answer built from pages the user cannot open |
| Outdated-content traps | A retired page and its replacement both exist | Answer from the current page only |
| Ambiguous questions | Vague or multi-part wording | A clarifying question, or an answer that covers each part |
Score retrieval and the answer separately. A system can retrieve the right page and still write a wrong answer, and the fix is different in each case. Record a failure code for every miss, such as wrong source, missing source, outdated source, or permission leak. Anthropic's guidance on evaluations makes a related point: test cases where a behavior should happen and cases where it should not, or you will optimize for one side only.
Our two downloads follow this method:
The template contains example questions only. It does not include test results, and we do not report results from our own runs in this article.
After the test set passes, pilot with one team for a few weeks. Turn every confirmed failure into a new test case, so the same mistake cannot return unnoticed.
Retrieval never sees your knowledge base the way a reader does. It sees isolated chunks, cut out of the page. A paragraph that starts with "As mentioned above, the limit is 30 days" loses its meaning once "above" is gone. Most disappointing AI knowledge base projects trace back to content written for scrolling, not for retrieval.
Clear structure also makes chunking easier. In NVIDIA's chunking experiments, page-level chunking had the highest average accuracy (0.648) and the lowest variation (0.107) across the datasets tested. Natural boundaries such as pages and well-defined sections tend to beat fixed token counts.
When five pages each describe the refund policy slightly differently, the model blends them. Pick one page as the canonical answer, mark it as such, and have every other page link to it instead of restating it. When the policy changes, you edit one page.
Give every page a small set of fields that retrieval can filter on:
Use a controlled vocabulary for topics so "password reset," "credential recovery," and "account access" all map to one tag. MatrixFlows recommends keeping universal content separate from audience-specific filters, which avoids duplicating pages for every audience.
If three pages cover network troubleshooting for three product lines, say which one each page covers in its first sentence: "This guide covers connection errors for the Pro series on firmware 4.x." That sentence gives both the embedding and the keyword index a clear signal, and it is the cheapest accuracy fix available.
Most organizations are at stage 2 or early stage 3. Each stage depends on the habits built in the one before it, so skipping ahead tends to produce a system people stop trusting.
| Stage | What it looks like | Next move |
|---|---|---|
| 1. Scattered | Knowledge sits in drives, chat threads, and people's heads | Consolidate one use case's sources in one place and name owners |
| 2. Searchable | One repository with keyword search and categories | Add templates, metadata, and review dates |
| 3. AI-assisted | Questions in plain language get answers with sources | Build the golden question set and a feedback loop |
| 4. Self-maintaining | The system flags gaps and stale pages, and drafts updates for review | Track answer quality by topic and expand to the next team |
| 5. Proactive | Relevant knowledge is suggested before people ask, based on role and task | Keep human review and clear rules about what the system may do on its own |
Moving from stage 2 to 3 is mostly content work. Moving from 3 to 4 is mostly process: someone has to own the feedback loop and act on it.
Use a few metrics that point to a specific layer, so you know what to fix when a number moves:
| Metric | What it tells you | How to measure it |
|---|---|---|
| Retrieval hit rate | Whether the right source reaches the model | Share of golden-set questions where the expected source is in the top results |
| Grounded answer rate | Whether answers stick to the sources | Share of sampled answers where every claim is supported by a cited passage |
| Correct "I don't know" rate | Whether the system admits gaps | Share of uncovered questions that are declined instead of answered |
| Citation accuracy | Whether the links prove the answer | Share of citations that actually contain the stated fact |
| Escalation and repeat-question rate | Whether real users got what they needed | Hand-offs to a person, and the same question asked again within a session |
Record a baseline before launch and re-run the golden set after every change to content, prompts, or models. For broader knowledge base metrics such as search success and article health, see our guide to knowledge base metrics.
Tools fall into four groups. The examples below show what each group looks like; they are not rankings. For a tested, side-by-side comparison of products, see our comparison of AI knowledge base software.
| Tool type | Examples | Good fit when | Watch for |
|---|---|---|---|
| Workspace-native | AFFiNE, Notion | Knowledge is written and maintained by the same team that queries it | Fewer connectors to outside systems |
| Support AI and help centers | Intercom Fin, Zendesk AI | Customers need answers from your public help content | Mostly built for external support use |
| Enterprise search and knowledge management | Glean, Guru | Knowledge is spread across many existing apps | Connector setup and permission mapping across systems |
| Open-source frameworks | LlamaIndex, LangChain | You need full control over retrieval, models, and hosting | You build and maintain every part |
AFFiNE is a workspace-native option. Docs, whiteboards, and databases live in one workspace, so the place where people write knowledge is also where they look it up. Workspace content can be stored locally with optional cloud sync, or on a self-hosted server. AFFiNE AI runs through AFFiNE Cloud, or through your own provider key: self-hosted workspaces can bring keys for OpenAI, Anthropic, Gemini, or FAL. Self-hosted workspace indexing for AI uses PostgreSQL with pgvector. When you use AFFiNE AI, related content is sent to the AI provider, so review the AI terms before using it with sensitive material. See AFFiNE as a team knowledge base, or read how an AI workspace and an LLM wiki differ from a classic knowledge base.
An AI knowledge base is a collection of an organization's documents that an AI system can search by meaning and use to write direct answers with links to the sources. Most are built on retrieval-augmented generation, which retrieves relevant passages first and then has a language model answer from those passages.
A traditional knowledge base returns a list of articles that match your keywords, and you read them to find the answer. An AI knowledge base understands a question in plain language, finds the relevant passages across many articles, and writes a short answer with citations. Both still depend on accurate, well-owned content.
The knowledge base is the content and the system around it. RAG is the technique most AI knowledge bases use to answer questions from that content: retrieve the relevant passages, then generate an answer from them. You can have a knowledge base without RAG, but most AI knowledge bases use some form of it.
You can attach files to a general chat assistant and ask questions about them, which works for personal or small one-off tasks. A team knowledge base needs more: content that stays in sync with the source, answers that respect each person's permissions, named owners, review dates, and tests you can re-run. Those are the gaps dedicated or workspace-native tools fill.
Yes, if you have engineering time. Open-source frameworks such as LlamaIndex or LangChain, plus PostgreSQL with pgvector, cover indexing and retrieval. You still pay for compute, for model usage unless you host a model yourself, and for the people who maintain the pipeline and the content.
You cannot remove the risk entirely, but you can reduce it. Use hybrid search and a reranker so the right passages reach the model, instruct it to answer only from the sources and to say "I don't know" otherwise, require citations, and test with a golden question set that includes questions the sources do not cover.
Start internal if your main problem is repeat questions from employees, because mistakes are cheaper and staff can flag them. Start external if support volume is the main cost, but only after your public help content is current and you have tested the answers. Either way, build on one shared source of truth.