LLM as a judge, pairwise comparison and rubrics

How it works: LLM as a judge

1. Pick a judge model: usually a strong model, ideally from a different family than the one you're testing (models tend to favour their own style).
2. Write a judging prompt: the task, the inputs (question, sources, answer), the criteria, and examples of passes and fails.
3. Ask for structured output: a verdict per criterion plus a short reason, as JSON (structured outputs). Reasons make results debuggable; asking for the reason before the verdict often improves consistency.
4. Run it over your eval set and aggregate.
5. Calibrate before trusting it: have humans label 50–200 cases; compare with the judge using agreement and Cohen's kappa (agreement corrected for chance); adjust the prompt and examples until agreement is high.
6. Watch for biases: position (in pairwise), length (longer looks better), self-preference, leniency, and being fooled by confident tone.
7. Re-check calibration when you change the judge model or prompt.

How it works: pairwise comparison

1. Show the judge two outputs for the same input (A and B) and ask which is better against the criteria (or a tie).
2. Swap the order and ask again to cancel position bias; count only wins that hold both ways, otherwise a tie.
3. Aggregate win rates across cases: "version B wins 58%, loses 30%, ties 12%".
4. Best for: comparing two prompts or models when absolute scores are hard to define. Relative judgements are often more reliable than absolute ones.
5. Rank many versions with pairwise results (Elo-style ratings), like human-preference leaderboards (benchmarks).

How it works: rubric scoring

1. List specific criteria that define a good answer for this product: grounded in the sources, answers every part of the question, follows the format, appropriate tone, no unsafe advice.
2. Make each criterion binary (pass/fail) with a clear definition and examples. Binary criteria are applied far more consistently than 1–10 scales.
3. Score each criterion separately, then combine (count of passes, weighted score, or "must pass all critical criteria").
4. Report per criterion, not just the total, so you know what to fix.
5. Grow the rubric from error analysis: each important failure mode becomes a criterion.

import json
from sklearn.metrics import cohen_kappa_score
 
RUBRIC = {"grounded": "Every claim is supported by the sources.",
          "complete": "Answers every part of the question.",
          "concise": "No more than 4 sentences."}
 
def judge_prompt(question: str, answer: str, sources: str) -> str:
    """What you send to the judge model. Ask for JSON so code can read the verdict."""
    criteria = "\n".join(f"- {k}: {v}" for k, v in RUBRIC.items())
    return (f"Grade the answer. For each criterion reply pass or fail with a reason.\n{criteria}\n"
            f"<sources>{sources}</sources>\n<question>{question}</question>\n<answer>{answer}</answer>\n"
            'Reply as JSON: {"grounded": {"verdict": ..., "reason": ...}, ...}')
 
# A judge reply (a model writes this in practice) -> rubric score.
reply = '{"grounded": {"verdict": "pass", "reason": "matches [1]"}, "complete": {"verdict": "fail", "reason": "ignores carry-over"}, "concise": {"verdict": "pass", "reason": "2 sentences"}}'
verdicts = {k: v["verdict"] == "pass" for k, v in json.loads(reply).items()}
print("rubric:", verdicts, "-> score", f"{sum(verdicts.values())}/{len(verdicts)}")
 
# Pairwise: ask twice with the order swapped to cancel position bias; count only consistent wins.
def pairwise(first_run_winner: str, swapped_run_winner: str) -> str:
    swapped_back = {"A": "B", "B": "A"}[swapped_run_winner]
    return first_run_winner if first_run_winner == swapped_back else "tie (inconsistent)"
print("pairwise:", pairwise("A", "B"), "|", pairwise("A", "A"))
 
# Calibrate the judge against human labels before trusting it.
human = [1, 1, 0, 1, 0, 0, 1, 1, 0, 1]
judge = [1, 1, 0, 1, 1, 0, 1, 1, 0, 1]
agree = sum(h == j for h, j in zip(human, judge)) / len(human)
print(f"judge vs human: agreement {agree:.0%}, Cohen's kappa {cohen_kappa_score(human, judge):.2f}")

rubric: {'grounded': True, 'complete': False, 'concise': True} -> score 2/3
pairwise: A | tie (inconsistent)
judge vs human: agreement 90%, Cohen's kappa 0.78