Lincoln AI-Judge Evals
An LLM judge, and the work of validating the judge
A pipeline that asks whether what biographers say about Lincoln matches what Lincoln himself wrote. It scrapes primary sources from the Library of Congress and secondary accounts from Project Gutenberg, extracts claims about five events, and scores consistency with a DSPy judge. The part I care about is the second half: measuring the judge against human labels and reporting where it disagrees.
Verified: each of these is checkable in the repository
- 0.41
- Cohen's kappa against human labelsdata/evaluated/kappa_report.json, moderate agreement
- 0.80
- raw agreement with human labelssame report, over 10 labelled pairs
- 5
- historical events across primary and secondary sources1860 election through Ford's Theatre
- 0.7
- hybrid retrieval alpha, keyword versus embeddinghybrid_retriever.py
LLM-as-judge is the default way to evaluate generative systems, and it is usually taken on faith. A judge produces a number, the number goes in a report, and nobody asks whether the judge agrees with a human. If it does not, every downstream conclusion is decoration.
The pipeline has five stages: scrape primary and secondary sources, normalise them, extract claims about five events using hybrid retrieval that blends keyword scoring with embedding similarity, score consistency with a DSPy judge that classifies contradictions as factual, interpretive or omission with a severity, and present the results. Two validators branch off that: one measures the judge against human labels with Cohen's kappa and tests self-consistency by running the same comparison five times at non-zero temperature, and one compares zero-shot, chain-of-thought and few-shot prompting by variance.
The number I am most willing to defend, because it is mediocre
Ten pairs were labelled by hand and compared against the judge's verdicts. Reporting the kappa rather than the raw agreement is the whole point, raw agreement flatters a judge that mostly says yes.
0.41
Cohen's kappa
Moderate agreement
0.80
Raw agreement
the misleading one
10
Human-labelled pairs
small, wide confidence intervals
The failure mode · The judge called 2 of 3 human-negative pairs consistent. It under-detects exactly the contradictions the system exists to find.
Why it is built this way
01Constrain the judge to the provided text
The judge may only reason from the passages given to it, not from what the model already knows about Lincoln. Otherwise it grades against its pretraining and the exercise measures nothing.
02Typed contradictions, not a single score
A factual contradiction, a difference of interpretation and an omission are different findings. Collapsing them into one consistency number throws away the distinction that matters most to a historian.
03Measure the judge against humans
Ten pairs were labelled by hand and compared to the judge's verdicts. Kappa came out at 0.41 (moderate agreement), with raw agreement at 0.80. Reporting the kappa rather than the raw agreement is the point; raw agreement flatters a judge that mostly says yes.
04Test the judge against itself
Running the same comparison five times at temperature 0.7 shows how much of a score is signal and how much is sampling noise. A judge that disagrees with itself cannot arbitrate anything.
05Compare prompting strategies by variance
Zero-shot, chain-of-thought and few-shot are compared on the standard deviation of their scores, not just the mean. Stability is what makes a judge usable for regression detection.
The kappa of 0.41 is the most useful number I have produced, precisely because it is mediocre. Raw agreement of 0.80 would have looked fine in a summary and would have been misleading, the judge agrees because it usually says consistent, and kappa is what exposes that. I now build the judge-validation step before trusting any judge-derived metric.
Next project
DocIntel
A serverless GCP platform, then a multi-agent assistant over its data