Original research brief
How to Measure a RAG System When You Can't Trust Its Answers
A practical method to measure a RAG knowledge system for small business. Define source quality, retrieval checks, citation gaps, and a maintenance plan…
OMNIASSIST / FIELD NOTESblog · source-led editorial
The visual file
Read the signal before the detail.
Every image is selected for a distinct editorial role, then checked for source, rights and fit before it enters the story.
What this piece is grounded in
01
According to 'Retrieval-Augmented Generation (RAG) | Pinecone', foundation models can produce confidently inaccurate and irrelevant output, a behaviour known as hallucination.
02
According to 'RAG makes LLMs better and equal | Pinecone', research demonstrates that RAG significantly improves LLM performance, even on questions within their training domain.
03
According to 'Retrieve data and generate AI responses with Amazon Bedrock Knowledge Bases - Amazon Bedrock', RAG is a technique that uses information from data sources to improve the relevancy and accuracy of generated responses.
04
According to 'Full-Text Search is Now Generally Available In Pinecone Database | Pinecone', queries come in more than one kind, and identifiers and exact phrases like SKUs and error codes are a different kind of query where semantic search often fails.
01 / FIELD NOTE
Define the reader problem and intended outcome
You have a RAG system. It answers questions from your documents. The problem is you don't know if those answers are right, complete, or even safe to rely on. According to 'Retrieval-Augmented Generation (RAG) | Pinecone', foundation models have a known limitation: they can produce confidently inaccurate and irrelevant output, a behaviour known as hallucination. Your intended outcome isn't a perfect system; it's a measurement plan that tells you where the system is failing and what to fix first. Start by listing the types of questions your system must answer. Are they about product specifications, support procedures, or internal policies? For each type, write down the consequence of a wrong answer. A wrong support step wastes a customer's time. A wrong specification could lead to a compliance issue. That consequence defines your measurement priority. This isn't about chasing a score; it's about managing a specific business risk that your RAG system either mitigates or introduces.
02 / FIELD NOTE
Choose trustworthy evidence before drafting
Your measurement plan is only as good as the evidence you use to judge the system. According to 'RAG makes LLMs better and equal | Pinecone', research demonstrates that RAG significantly improves LLM performance, even on questions within their training domain. This is a useful signal: your evidence should test whether your RAG system is actually providing that improvement over a raw model. But the evidence itself must be trustworthy. Don't use the system's own outputs to verify itself. Instead, build a small, curated evaluation set from your source materials. Pick a dozen documents that represent critical knowledge areas. From each, extract a few clear facts or procedures. These become your 'known correct' answers. The trick is to ensure these source snippets are themselves accurate and up-to-date. If your source is a three-year-old policy document, your test is measuring fidelity to outdated information. Your first measurement, then, is a simple audit of your source material's freshness and authority. If you can't vouch for the source, you can't trust any answer derived from it.
03 / FIELD NOTE
Choose sources and define permission boundaries
Not all documents are equal candidates for your knowledge base. The 'Retrieve data and generate AI responses with Amazon Bedrock Knowledge Bases - Amazon Bedrock' guide notes that RAG is a technique that uses information from data sources to improve the relevancy and accuracy of generated responses. The practical question is which sources actually improve relevancy. Start by categorising your potential sources: internal wikis, official product manuals, third-party blog posts, regulatory texts. For each category, define a permission boundary. Internal wikis might be editable by many, so you need a rule about which version to ingest—perhaps only pages marked 'approved'. Third-party content requires a check for copyright and a decision on whether to link out or ingest. According to 'Full-Text Search is Now Generally Available In Pinecone Database | Pinecone', identifiers and exact phrases like SKUs and error codes are a different kind of query where semantic search often fails. This is a critical boundary: if your source is full of precise codes, you must plan for hybrid retrieval that can match them literally. Your source list should come with a one-sentence reason for inclusion and a clear rule for when it should be re-ingested or removed.
04 / FIELD NOTE
Design retrieval checks around realistic questions
The most common failure is testing with questions the system will never see. Your retrieval checks must mirror real user intent. According to 'Full-Text Search is Now Generally Available In Pinecone Database | Pinecone', queries come in more than one kind. Semantic search finds meaning, but literal strings like 'PROD-001' need exact matching. Write down ten real questions your team or customers have asked in the last month. Categorise them: are they seeking a definition, a step-by-step process, a specific identifier, or a judgment? Then, for each category, design a different check. For a process question, the check is whether the retrieved chunk contains all necessary steps in order. For an identifier, the check is a perfect string match. Run these checks manually first, before automation. You'll discover if your chunking strategy—splitting documents into pieces—is breaking logical flow. If a user asks 'how do I reset the router?' and the retrieval returns a chunk that starts with 'step three', your measurement plan has just found its first critical bug. Document the failure mode, not just the pass/fail score.
05 / FIELD NOTE
Inspect citations gaps and stale knowledge
A RAG system should cite its sources. When it doesn't, or when it cites irrelevant passages, you have a citation gap. This is a direct measurement of retrieval quality. Set up a simple review: for each answer in your evaluation set, ask if the provided citation actually supports the claim made. If the system says 'the warranty lasts two years' but cites a paragraph about shipping costs, that's a failure. The second, subtler failure is stale knowledge. According to 'Retrieval-Augmented Generation (RAG) | Pinecone', foundation models suffer from knowledge cutoffs and may provide outdated information. Your RAG system might retrieve a perfectly cited, but obsolete, policy. Therefore, your measurement plan must include a freshness audit. Tag your source documents with a 'last reviewed' date. Any answer drawn from a document older than a threshold you define—say, twelve months for product specs—triggers a manual review flag. This turns a vague worry about 'old data' into a countable, actionable item on a maintenance report.
06 / FIELD NOTE
Assign ownership for updates and evaluation
A measurement plan that no one owns is a forgotten document. The work here is operational. Assign a single person to be responsible for the evaluation set. Their job is to run the retrieval checks monthly, update source documents when they change, and add new question types as the business evolves. This isn't a full-time role; it's a recurring calendar task with a clear output: a one-page report listing the number of tests passed, citation gaps found, and stale sources flagged. The second assignment is for source ingestion. When a new official manual is published, who ensures it replaces the old one in the knowledge base? Define that handoff. Without ownership, your system will drift, and your measurements will become meaningless because they're measuring an abandoned system. Use the report to decide when to retrain embeddings or adjust chunking rules. If questions about troubleshooting start failing because the answers are split across chunks, that's a signal to revisit your document splitting logic, not just to accept poor performance.
07 / FIELD NOTE
Turn the method into a measurable next step
You now have a method, not a theory. The next step is to execute the first cycle. Pick one critical knowledge area. Assemble your source documents for that area. Extract ten 'known correct' facts to form your initial evaluation set. Write five realistic user questions. Run them through your system manually, checking retrieval and citations. Document each failure mode you find. That list is your first measurement report. The goal isn't to fix everything now; it's to establish a baseline. Next, assign the monthly review task to someone's calendar. The measurable outcome is not a performance score but the completion of that first review cycle and the actionable items it produces. This approach turns an overwhelming quality assurance problem into a scheduled, repeatable audit. It accepts that the system will have flaws but ensures you know what they are and can prioritise them. That's the only practical measurement that matters for a small business: knowing what you don't know, and having a plan to close the gap.
Questions readers ask
What is the first thing I should measure in my RAG system?
Measure whether your system can retrieve and cite the correct source passage for a simple, verifiable fact. Start with a small evaluation set of known facts from your most important documents. If it cannot accurately cite the source of a basic answer, more complex measurements like answer quality are premature. This tests the foundational retrieval step.
How do I know if my source documents are good enough?
Audit your sources for freshness, authority, and clarity. If a document is outdated, contradicts another source, or is written in ambiguous language, it will degrade your system's outputs. A simple rule: if you wouldn't trust a human new hire to learn from that document alone, don't feed it to your RAG system without cleaning or annotating it first.
What's a common failure mode in RAG retrieval?
A common failure is when your document chunking strategy breaks a logical sequence. If a user asks for a step-by-step process and the retrieval returns a fragment starting at step three, the answer will be incomplete or confusing. Design retrieval checks that use real multi-step questions to see if your chunks preserve necessary context.
Who should be responsible for maintaining the measurement plan?
Assign a single person, perhaps a product owner or a lead developer, to run the monthly evaluation cycle and update the test set. Their output is a brief report highlighting gaps, not a technical deep dive. This turns measurement from a project into an operational routine with clear accountability.
How often should I update my evaluation set of questions?
Review and potentially expand your evaluation set every time you add a significant new source or knowledge domain to the system. Also, add new questions quarterly based on real user queries or support tickets. The set should evolve with your business needs, not remain static, or it will miss emerging failure points.