LLM-as-a-Judge
Also called LLM Judge, Model-Based Evaluation
Using a strong language model to grade or compare the outputs of other AI systems, as a cheaper, faster alternative to human reviewers.
What is LLM-as-a-Judge?
LLM-as-a-Judge is a method for evaluating AI output in which one language model scores or compares the answers of other models, usually against a written rubric. It is used because having people review thousands of answers is slow and costly.
How does it work?
The judge model is given the question, one or more candidate answers and instructions, such as "pick the more accurate answer" or "rate helpfulness from 1 to 5". Its verdicts are collected across many examples. The 2023 paper that named the approach found that a strong judge such as GPT-4 agreed with human preferences about as often as humans agree with each other.
What is a simple example?
A team comparing two versions of a support bot has a judge model read 500 pairs of replies and pick the better one in each. The version that wins more often is kept, after a person spot-checks a sample.
Why does it matter?
It makes model evaluation scalable. Judges have known weaknesses, such as favouring longer answers or their own style, so teams check them against human ratings and keep a person in the loop for important decisions.