LLM-as-a-Judge.

🎯Why you end up here

You tweaked your RAG prompt. Is it better? You can't diff free text. BLEU/ROUGE score wording, not correctness. Hand-grading 200 examples every release is never going to happen.

So you have a model grade the model. Fine — but a judge is just another model with an accuracy you haven't measured yet.

🛠The 12 lines

RUBRIC = """Question: {q}
Reference answer: {ref}
Candidate answer: {cand}

Score the candidate 1-5 on factual agreement with the reference.
1 = contradicts it. 3 = partly right. 5 = fully consistent.
Length and style do not matter.
Reply as JSON: {{"reasoning": "<20 words", "score": <int>}}"""

def judge(q, ref, cand):
    r = client.messages.create(model="claude-sonnet-5", max_tokens=200,
        messages=[{"role": "user", "content": RUBRIC.format(q=q, ref=ref, cand=cand)}])
    return json.loads(r.content[0].text)

scores = [judge(**row)["score"] for row in dataset]
print(sum(scores) / len(scores))

🔍What each choice teaches

• A reference answer is in the prompt. Reference-based grading beats "is this good?" — an unanchored judge scores vibes.
• "reasoning" comes BEFORE "score" in the JSON. Swap the key order and accuracy drops: the model emits a number first, then rationalizes it. Order of keys = order of thinking.
• 1-5 with anchored labels, not 0-100. Unanchored scales give you a wall of 85s.
• "Length and style do not matter" is load-bearing. Keep reading.

⚠️The 4 ways it lies

1. Position bias — in A/B comparisons the judge favors whichever answer came first, often by 10-25 points. Fix: run both orders, keep only the agreements, count flips as ties.
2. Self-preference — a judge tends to score its own model family higher. Where you can, judge with a different model than the one that generated.
3. Verbosity bias — longer answers score higher regardless of correctness. Cheap check: log answer length next to score. If they correlate above ~0.5, your judge is measuring word count.
4. Scale compression — everything lands on 4. If stddev across obviously-different outputs is under 0.5, the judge isn't discriminating, it's agreeing.

📏Validate the judge before you trust the numbers

Hand-label 50 examples. Compare against the judge and compute Cohen's kappa. Below ~0.6, fix the rubric — don't ship the harness. A judge you haven't calibrated is a dashboard, not a signal.

⬆️Cheapest upgrade

Make it binary: "Does the candidate contradict the reference? yes/no." Binary judgments agree with humans far more than 1-5 scales, and a pass-rate is a much better regression alarm than the mean of a pile of noisy 4s. Reserve graded scales for things that genuinely have degrees.

🏋️Your exercise

Grab 20 outputs from something you've built. Grade them 1-5 yourself first. Then run the rubric above. How many land within 1 point of your score?

Then take 10 pairwise A/B comparisons and run each twice with the order swapped. How many verdicts flip? That flip rate is your judge's noise floor — any "improvement" smaller than it is you reading static.