Dictionary term
LLM-as-a-Judge
Also called LLM-as-judge · LLM-based grading · model-graded evals
An evaluation method in which a language model scores or compares another model's outputs against a rubric, used where exact-match or code-based checks cannot capture quality.
In plain English
Using one AI to grade another AI's answers against a checklist, so you can test hundreds of responses without a person reading every one.
Analogy
Like a senior editor marking a stack of drafts against a style guide: fast and consistent, but you still spot-check the editor.
Why it matters
It makes evals affordable for open-ended outputs like support replies or summaries, but judges have known biases, so you validate the judge against human-graded samples before trusting it at scale.
In practice
Scoring 200 AI-drafted follow-up emails on tone, accuracy and whether they include the booking link, using a judge prompt with a 1-5 rubric and checking 20 of them by hand.
Related terms
Sources
Browse all terms →Last verified