All articles

LLM as a judge: how to test the judge before you trust it

In short. LLM as a judge means one language model assesses the work of another against written criteria. Published studies show it can come close to human agreement on chat answers, and they also document position, length and self-preference biases. Before you rely on a judge, test it: plant known errors in its input and count how many it catches. In one of our tests, one model caught 0 of 3 and another 3 of 3.

Proofreading desk in a small print shop in Nice, with marked proofs and a red pencil under the morning light
A review is only worth something if the reviewer can be shown to catch errors.

LLM as a judge is the usual answer when an AI system produces more text than anyone can read: translations, support replies, summaries, the results of an agent. A second model reads the output and gives a verdict. This guide is for people who build or buy such systems and need to know whether that verdict can be trusted. It covers what the research says, which biases to expect, and the test we now run on every AI reviewer in our own pipelines.

What is LLM as a judge?

LLM as a judge is a method where a language model assesses an output against written criteria and returns a score, a pass or fail, or a preference between two answers. The output may come from another model, an agent or a person. The method replaces part of the human reading, which makes it possible to review thousands of items.

Three forms are common:

  • Pairwise comparison: the judge sees two answers and picks the better one.
  • Scoring: the judge gives one answer a grade on a rubric.
  • Pass or fail: the judge checks one precise condition, for example “does this translation keep every date?”.

The third form is the easiest to verify, because a person can quickly say whether the verdict was right.

How reliable is an LLM judge?

An LLM judge can be close to human agreement on some tasks, but that figure does not transfer to yours. In the study that popularised the method, strong judges such as GPT-4 reached over 80% agreement with human preferences, the level at which humans agree with each other. That was measured on open-ended questions put to chat assistants.

Other tasks give a less comfortable picture. G-Eval, a scoring method built on GPT-4, reported a Spearman correlation of 0.514 with human ratings on summarisation. That is a useful signal, and it is far from a perfect match. Anthropic’s guide to evaluating agents describes model graders as flexible and scalable, but also non-deterministic, more expensive than code checks, and in need of calibration against human graders.

Which biases does an LLM judge have?

The published studies name four main problems, and each has a practical answer.

  • Position bias. The order of the answers changes the verdict. In one study, simply changing the order let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the evaluator. Answer: run the comparison in both orders and combine the results.
  • Length bias. The MT-Bench authors list verbosity among the judge’s biases: a longer answer can look better. Answer: state in the rubric that length earns nothing, and check the verdicts on short correct answers.
  • Self-preference. A model can score its own outputs higher than others’ while human annotators rate them as equal. Answer: the judge should not be the model that wrote the text.
  • Limited reasoning. A judge can fail on the same maths or logic it is meant to check. Answer: give such checks to code.

A panel can also help. In the Replacing Judges with Juries study, several smaller models from different families outperformed one large judge across six datasets, at over seven times less cost. That result comes from the datasets tested, so it is a reason to try a panel and measure it, not a rule.

How do you test an AI reviewer with planted errors?

You test an AI reviewer by putting errors you already know about into its real input and counting how many it reports. If it misses them, its “no issues found” means nothing. This takes an hour or two and needs no research team.

  1. Take a real batch of the work the judge will review, in the real format.
  2. Plant two or three known errors of the kind you fear: swapped translations, a wrong date, an invented fact.
  3. Add one trap of the opposite kind: a correct but unusual line that must not be changed. This measures false alarms.
  4. Run the judge in the same pass, with the same prompt it will use in production, and do not tell it that errors were planted.
  5. Count: planted errors found, false alarms raised, items that could not be read.
  6. Set the rule in code: one missed planted error means the whole pass fails, and an item that could not be read counts as a failure, never as clean.

Repeat the test when you change the model, the prompt or the type of content. A judge chosen for one task has proved nothing about the next one.

Market stall on the Cours Saleya in Nice with crates of lemons being sorted and a few bad fruits set aside
Planted errors work like known bad fruit in a crate: if the sorter misses them, the crate has not been sorted.

What did this test show in our own pipelines?

We run AI reviewers on translation and media pipelines, and three cases changed how we work. These are small tests on specific tasks, so they say nothing general about any model.

A reviewer that caught nothing. On a multilingual questionnaire of 341 strings, we planted three errors. One model caught 0 of 3. Another caught 3 of 3 in each of the four languages tested, and it also found a real error nobody had seen: a medical word for “stool” translated as the piece of furniture. Smaller versions of that second model caught the planted errors too, but raised dozens of false alarms on Uzbek and Kazakh. So a good catch rate still needs a look at the false alarms.

A clean report with nothing read. A quality gate for video reported 0 defects out of 21 frames. The vision service was down, no frame had been read, and “unread” was treated as “no defect”. We found the problem by eye in a finished video. Since then, a tool that is unavailable means a failure.

A reviewer that invented facts. A model reviewing translations passed its test, with 6 of 6 planted language errors and no needless edits. In production it then replaced demo contact details with a plausible invented address, phone number and email. The test had covered language, not facts. Now code rejects any edit that changes numbers, emails or links, and the test includes a line with unfamiliar contact details that must stay untouched.

What should go to code, to a judge and to a person?

Most checks do not need a judge at all. Give each check to the cheapest thing that can do it reliably.

Check Who does it Why
Numbers, dates, emails and links are unchanged Code The answer is exact, and a model may “improve” them
Format, length, required fields Code Free, repeatable, the same result every time
Tone, clarity, whether the answer addresses the question Judge Open-ended, too much volume for a person
Meaning kept between two languages Judge, with planted errors Needs language skill, and the catch rate can be measured
Medical, legal or financial content Person An error has real consequences for someone
A sample of the judge’s verdicts Person This is how you see whether the judge drifts

How should you write the judge’s instructions?

Write one narrow question per judge, with the criteria spelled out. Anthropic’s guide advises grading each dimension with its own isolated judge and giving the model a way out, such as answering “Unknown” when it lacks information. We add three habits:

  • Ask for the exact line and the reason, so a person can check the verdict in seconds.
  • Give the judge the full text. In one of our pipelines a judge rejected good translations because our own code had cut the input at 3,000 characters.
  • Read a sample of the verdicts yourself. The same guide says you cannot tell whether graders work without reading the transcripts and grades.

When is LLM as a judge the wrong tool?

It is the wrong tool when a rule can be written as code, when a mistake would harm someone and no person reads the result, and when volume is low enough for a person to read everything. A judge also adds cost and time on each item, so a few dozen documents a month rarely justify one. If the data is confidential, the judge sees that data too, and a local model may be the better place to run it.

How does Castel Studio use LLM judges?

We build reviewers into AI agents with the same rules: code for exact checks, a judge from another model for open-ended ones, a planted-error test, and failure when a check cannot run. If you are still deciding what your agent should do, start with our seven questions on AI agents. For the building blocks, see agents and skills in Claude Code. A first piece of work starts from 150 €.

Tell us what your AI system produces and who reads it today. We reply in writing within 48 hours.

Questions

What does LLM as a judge mean?

It means using one language model to assess the output of another system against written criteria: a score, a pass or fail, or a choice between two answers. It replaces part of the human review, not the responsibility for the result.

How reliable is an LLM judge?

In the 2023 MT-Bench study, strong judges such as GPT-4 agreed with human preferences over 80% of the time, about the level at which humans agree with each other. That result concerns chat answers. On your task, reliability has to be measured, for instance with planted errors.

Can a model judge its own output?

It can, but research has found self-preference: a model may score its own text higher than texts that human annotators rate as equal. A judge from a different model family is the safer default.

How much does it cost to add a judge to an AI agent?

At Castel Studio a first piece of work on an agent system (a one-page map, tested rules, confidential files blocked) starts from 150 €. Connecting it to your tools is priced by written quote. Model fees and maintenance are extra.

Contact

One person reads this inbox — the same person who builds what you order.

Describe your task

Tell us what you have and what result you need — even a single small task. No brief required: within 48 hours you get a plan and a price in writing, a question, or an honest “not a fit”.

Include the situation, files or links, deadline, language, privacy limits and budget boundary if you have one.

File (optional)
No file chosen

A brief, a photo, a spreadsheet — whatever explains the task faster than words. Up to 8 MB, stored with our hosting provider. Please don't send sensitive personal or medical documents through the site.

Sending creates no order and no payment. We reply within 48 hours. Prefer email? contact@castel.studio

Talk to us

Prefer a call or a messenger? Write or call — the studio's founder answers.

Connect on LinkedIn

Prefer a freelance platform?

Order the same work there, with the platform's secure payment.

A one-person AI studio in Nice. Whoever answers your message is the one who does the work — no account managers in between. Files are processed on our own machine, not in someone else's cloud.

Nice 06100, France · Open in maps →

SIREN 103 115 705

Describe your task