Intermediate · Lesson 5 of 6 · 18 min
RAG fundamentals
Ground model answers in your own documents: embeddings, vector databases, chunking, retrieval and cited answers.
The problem RAG solves
A model does not know your price list, your SOPs or last week's policy change. You could paste every document into each prompt, but that costs tokens, adds latency and, as Anthropic notes, accuracy degrades as context grows.
Retrieval-augmented generation (RAG) fetches only the passages relevant to a question and puts those in the prompt. The pipeline has two halves. Ingestion runs ahead of time: split documents into chunks, turn each chunk into an embedding and store it. Answering runs per question: embed the question, find the closest chunks, and have the model answer from them with citations.
Embeddings and vector databases
An embedding is a list of numbers that represents the meaning of a piece of text. Texts with similar meaning produce vectors that sit close together, which is what makes search by meaning possible: 'cancel my booking' finds the chunk about cancellations even though it never uses the word cancel.
You need an embedding model for this. Anthropic does not offer its own and points to providers such as Voyage AI; open models such as gte-small also work. Whatever you choose, use the same model for documents and questions, and size your vector column to that model's dimensions.
A vector database stores the vectors and finds the nearest ones fast. With Supabase you enable the pgvector extension and get a vector column type plus three distance operators: <-> for Euclidean distance, <#> for negative inner product and <=> for cosine distance. Supabase's client cannot call those operators directly, so you wrap the query in a Postgres function.
-- After enabling the "vector" extension in the Supabase dashboard
create table documents (
id bigserial primary key,
source_url text not null, -- kept for citations
body text not null,
embedding extensions.vector(384) -- match your embedding model's dimensions
);
create or replace function match_documents (
query_embedding extensions.vector(384),
match_threshold float,
match_count int
)
returns table (id bigint, source_url text, body text, similarity float)
language sql stable
as $$
select id, source_url, body, 1 - (embedding <=> query_embedding) as similarity
from documents
where 1 - (embedding <=> query_embedding) > match_threshold
order by embedding <=> query_embedding asc
limit match_count;
$$;Chunking
Chunking decides what a retrieval hit looks like. Chunks that are too large drag in unrelated text and dilute the match. Chunks that are too small lose the context that made them meaningful, such as which product a sentence refers to.
Start with natural units, such as a section under a heading or a single FAQ answer, and include the heading in the chunk text. Store metadata with every chunk: the source URL or file, the title and the last-updated date. That metadata is what lets you cite sources and re-ingest a document when it changes.
Retrieval and grounded answers
At question time, embed the question with the same model and call your match function with a similarity threshold and a result count. Supabase's docs stress that the right threshold varies by application and has to be found by testing on your own data.
Put the retrieved chunks into the prompt inside tags with their IDs and source URLs, instruct the model to answer only from them, to cite the chunk it used and to say so when the answer is not there. If nothing clears the threshold, skip the model call and return 'I couldn't find that in our documents' with a route to a person.
const { data: chunks, error } = await supabase.rpc("match_documents", {
query_embedding: questionEmbedding, // same embedding model as ingestion
match_threshold: 0.78, // tune on your own data
match_count: 5,
});
if (error) throw new Error("Retrieval failed: " + error.message);
if (chunks.length === 0) return { answer: null, reason: "no_match" };Measure retrieval and answers separately
When a RAG bot gives a wrong answer, the cause is either retrieval (the right chunk was not found) or generation (it was found and the model misused it). Fix the wrong one and nothing improves.
Write twenty real questions with the document section that answers each. Check whether that section appears in the retrieved chunks, then check the answer. Re-run the set every time you change chunk size, the embedding model, the threshold or the prompt.
Key points
- ◆RAG retrieves only the relevant passages instead of stuffing every document into context.
- ◆Use the same embedding model for documents and questions, with matching vector dimensions.
- ◆pgvector offers Euclidean, negative inner product and cosine distance operators.
- ◆Chunk on natural sections and store source metadata for citations and re-ingestion.
- ◆Tune the similarity threshold on your data and evaluate retrieval separately from answers.
Terms in this lesson
Try it · exercise
See vector similarity work in Postgres using tiny hand-made vectors, with no embedding API needed.
- Create a free Supabase project and enable the vector extension from Database, then Extensions.
- In the SQL editor, create a table with a body text column and an embedding extensions.vector(3) column.
- Insert three rows: 'refund policy' with [1,0,0], 'refund timeline' with [0.9,0.1,0] and 'office hours' with [0,0,1].
- Run a query that selects body and embedding <=> '[1,0,0]'::extensions.vector as distance, ordered by distance ascending.
You're done when: The query returns 'refund policy' first with a distance of 0, 'refund timeline' second with a small distance, and 'office hours' last with the largest distance.