How to Evaluate AI Agents and RAG Systems
Quick answer: To evaluate an AI agent or a RAG (retrieval-augmented generation) system, write a small set of real questions with known good answers, then score two things separately: did it find the right information, and did it use that information correctly? Re-run the same set after every change.
Last checked 7 November 2026. The methods below are general practice and are described in the linked documentation. We give no benchmark numbers, because results depend entirely on your data.
Why demos mislead
A demo shows the three questions that work. A real user asks the thousandth question, which nobody tried. Many teams find that their first version looks great in a demo and fails on everyday inputs. Evaluation, often shortened to "evals", is how you replace a good feeling with evidence. It is also the most useful habit you can learn if you build with AI, because it tells you whether a change helped or hurt.
What RAG is, in one minute
RAG means the system looks up relevant passages from your documents and gives them to the model along with the question, so the answer is based on your material instead of the model's memory alone. The idea was described in the research paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. A RAG system has two jobs that can each fail:
- Retrieval: finding the right passages.
- Generation: writing an answer that is faithful to those passages.
If you only judge the final answer, you cannot tell which job broke. That is why the next section scores them separately.
Step 1: Build a small test set
Start with 20 to 50 questions. Quality matters more than quantity.
- Collect real questions. Use what users actually ask, from support emails, search logs or your own notes. Do not write only easy ones.
- Add the expected answer in a sentence or two, written by someone who knows the subject.
- Record where the answer lives. Note the document and section that contains it. This lets you test retrieval directly.
- Include awkward cases: questions the documents cannot answer, ambiguous questions, and questions with a misspelling. A good system should say "I don't know" when the answer is not there.
- Keep a private holdout. Do not tune your system on every question, or you will only be measuring how well you memorised the test.
Step 2: Score retrieval
For each question, check whether the right passage was in what the system retrieved. Simple measures work well:
- Hit rate: in what share of questions did the correct passage appear in the top results?
- Rank: how high up was it? A correct passage buried at position ten may be cut off or ignored.
- Noise: how many irrelevant passages came back with it? Too much noise can distract the model.
Common fixes for weak retrieval are splitting documents into better-sized chunks, adding the document title or context to each chunk, combining keyword and meaning-based search, and re-ranking results. Anthropic describes one such approach in Contextual Retrieval. Change one thing at a time and re-run the test set.
Step 3: Score the answer
Given the retrieved passages, judge the answer on separate questions:
- Correct: does it match the expected answer?
- Grounded (faithful): is every claim supported by the passages, with nothing invented?
- Complete: did it leave out something important?
- Appropriate refusal: when the documents do not contain the answer, did it say so instead of guessing?
You can mark these by hand for a small set. For larger sets, some teams use a second model as a judge. If you do, check the judge against your own marks on a sample first, because a judge can be wrong or biased towards longer answers. Open-source tools such as Ragas package several of these measures. Read their documentation to see exactly how each score is calculated before trusting it.
Step 4: Evaluate agents, not just answers
An agent takes many steps, so the final answer is not enough. Also record:
- Did it finish the task? A simple pass or fail per test case.
- Were the tool calls right? Right tool, right inputs, no needless repeats.
- How many steps and how much time or cost? A correct answer that takes forty calls may be unusable.
- Did it stay safe? Test that it refuses actions it should not take, and does not follow instructions hidden in the content it reads.
- Is it consistent? Run the same case several times. If the result changes each time, note how often.
For agents that change things in the world, run tests against a copy or a sandbox, never against live accounts.
Step 5: Make it a habit
- Save the test set and the scores in a file that is tracked with your project.
- Run it before every change that affects prompts, models, chunking or tools.
- Compare against the previous run and read the cases that got worse, not just the average.
- When a real user finds a failure, add that case to the set.
- Keep a person reviewing a sample of real conversations, even after the numbers look good.
Common mistakes
- Testing only with questions you already know the system handles.
- Reporting one average and ignoring which cases failed.
- Changing three things at once, so you cannot tell what helped.
- Trusting an automatic judge without checking it.
- Putting private documents into a tool without checking where the data goes.
Practice idea
Take a PDF you own, such as a course handbook. Questions about it make a good first test set. If you need to cut it into smaller parts first, our free split PDF tool runs in your browser, so the file is not uploaded. For broader AI courses, see the free learning paths on Brynflow.
Sources
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Anthropic: Contextual Retrieval
- Ragas documentation
Keep reading: AI agents for beginners and MCP explained for beginners.