LEARN SOMETHING. BUILD SOMETHING.
Turn a small set of documents into a question-answering system that can cite its evidence and admit when the answer is missing.
6 short lessons · 1 project · video resources · no login for this guide
Before you start: Basic Python, lists, dictionaries, and familiarity with text generation.
Your toolkit: Python · retrieval · grounded answers
A language model predicts tokens from its context. Fluent text is not proof that a statement is correct. A prompt controls the task and output format, while retrieved context supplies facts. Retrieval augments context; it does not retrain the model.
Try it: Ask a question whose answer is absent from your documents. Write the refusal behavior you expect.
Split documents at meaningful boundaries such as headings and paragraphs. Preserve title, source, and section with every chunk. Very small chunks lose context; very large chunks waste the context window. Test boundaries with actual questions rather than choosing a universal chunk size.
Try it: Split a three-section FAQ into chunks and attach a stable source ID to each one.
Keyword search is a useful first baseline. Embeddings encode text as vectors so similar meanings can be compared. Hybrid retrieval combines keyword and vector signals. The best choice depends on your documents: exact product codes often need keywords.
Try it: For five questions, manually identify the expected chunk. Measure how often your retriever includes it in its top results.
Send only the authorized retrieved passages to the model. Ask for an answer supported by those passages and source IDs that exist in the retrieved set. Validate IDs before displaying citations. When evidence is insufficient, return a useful admission rather than inventing a source.
Try it: Create one answerable and one unanswerable question. Check each factual sentence against the supplied passage.
If the correct passage was never retrieved, a better answer prompt will not fix the failure. Track retrieval hits separately from answer correctness and citation accuracy. Keep a small held-out question set so prompt tuning does not simply memorize your examples.
Try it: Record question, expected source, retrieved source, answer, and pass/fail in a local table.
Filter documents by the current user’s permissions before retrieval. Treat document instructions as untrusted, avoid logging private passages, and cache only where access boundaries permit it. Keep model credentials on the server.
Try it: Design a two-user test: a private document belonging to user A must never reach user B’s context.
Use this small example to connect the lessons to code. Read the comments before running it.
from collections import Counter
def retrieve(question, chunks):
terms = set(question.lower().split())
ranked = sorted(chunks, key=lambda c:
sum(Counter(c["text"].lower().split())[t] for t in terms),
reverse=True)
return ranked[:2]
# A keyword baseline, not an embedding model or production retriever.
PUT IT INTO PRACTICE
Build the project described in the lessons, then verify these acceptance criteria:
Deliverable: your source code, a short README, and evidence for each check. Explain one tradeoff you made.
Videos open externally and do not autoplay. Provider exercises, certificates, and account requirements are separate from this free guide.
No. Retrieval can miss relevant evidence and generation can misinterpret it. Test both stages and verify citations.
Hugging Face LLM Course — continue with the original documentation and exercises.
Go deeper: continue to our existing Generative AI & RAG course.
Before paying for embeddings or connecting a language model, make a tiny evidence lookup work. This Python 3 example uses three fictional store policies so you can inspect every result. Save it as retrieval_demo.py and run python3 retrieval_demo.py. It needs no packages, account, network connection or API key.
import re
DOCUMENTS = [
{"id": "delivery", "text": "Delivery takes five working days."},
{"id": "returns", "text": "Returns are accepted within thirty days."},
{"id": "support", "text": "Support is available by email."},
]
STOP_WORDS = {"a", "an", "the", "is", "are", "by", "what", "how", "do", "i"}
def tokens(text):
return set(re.findall(r"[a-z0-9]+", text.lower())) - STOP_WORDS
def retrieve(question, limit=2):
query = tokens(question)
ranked = []
for document in DOCUMENTS:
score = len(query & tokens(document["text"]))
if score:
ranked.append((score, document))
ranked.sort(key=lambda item: (-item[0], item[1]["id"]))
return [document for _, document in ranked[:limit]]
def answer(question):
evidence = retrieve(question)
if not evidence:
return "No matching evidence. Please contact support."
return "\n".join(
f'{document["text"]} [{document["id"]}]' for document in evidence
)
if __name__ == "__main__":
checks = [
("How many days does delivery take?", "delivery"),
("What is the returns policy?", "returns"),
("How do I contact support?", "support"),
("What does gift wrapping cost?", None),
]
for question, expected_id in checks:
results = retrieve(question)
found = [item["id"] for item in results]
assert (expected_id in found) if expected_id else not found
print(question)
print(answer(question))
print()
print("4 retrieval checks passed")
The delivery question should include [delivery], the returns question [returns], and the support question [support]. The gift-wrapping question should return no evidence. The final line should say 4 retrieval checks passed. A delivery query can also retrieve the returns policy because both contain “days”; the check measures whether the relevant source is included, not whether every result is relevant.
Try “Can I send an item back?” The baseline cannot match “send back” to “returns,” so it refuses even though a human would find the answer. Next try “Are returns accepted after ninety days?” Retrieval will find the returns source, but it has no reasoning step to compare ninety with thirty. These are different problems: the first is retrieval vocabulary, the second is answer interpretation.
This deliberately returns source sentences rather than generated prose. It is an evidence lookup, not a complete RAG system. A nonzero token overlap is not a confidence score and must not be treated as proof that a question is answered. Before connecting a model, label a larger question set, including ambiguous queries and questions with no answer, and measure irrelevant matches as well as missing sources.
For private documents, filter by the requesting user's access before ranking or sending context to a model. Never treat instructions embedded in retrieved documents as instructions to your application. Use fictional data while experimenting and keep API credentials on the server.
Completion check: Explain one false positive, one missed answer and one out-of-scope question using your recorded results. A fluent answer alone does not demonstrate retrieval quality.