What My LLM Judge Caught, and What It Missed

Escape is a social app for discovering events and communities, on iOS, Android and the web in 15 languages. One of its paid features is a personality report: your archetype, how you connect, who you click with, even your role in the group chat. It's generated from your profile, your answers and your activity, in your language.

The pipeline has five stages:

1. Outline. The model reads a digest of the profile and finds one narrative thread, a named contradiction such as “an organizer who's uneasy with strangers”, with a hook for each section.
2. Sections. Each section is generated separately from its own bundle of profile data. The three core narrative sections get disjoint bundles, enforced by a contract test, so the same fact can't turn up in three places.
3. Coherence pass. A low-temperature call smooths the narrative across sections. It only sees and returns prose, so it can't corrupt the structured parts.
4. Stat guard. A deterministic check: every percentage in the output must come from the set of numbers we actually injected. Invented ones are stripped from the stat blocks and logged when they appear in prose.
5. Assembly into the app's widgets.

The flagship sections run on Claude Sonnet. Real rarity and percentile numbers are computed from the database and injected with one rule: quote them verbatim or say nothing.

The hard part is that none of this yields to ordinary tests. The service has about 900 of them, and they tell me a report is well formed. They can't tell me it got blander, repeated itself, or read like translated English.

eval_report_quality.py is one script, under 300 lines.

• Three synthetic personas, each built to fail differently. Rich fills every field and is the showcase. Sparse sits at the 30% profile-completeness floor, where a report is most likely to disappoint. Privacy marks several fields private; it's a canary, because none of those values may ever appear in prose.
• The real pipeline, with no database: outline, sections, coherence, stat guard, assembly.
• A judge for every section. Claude Sonnet scores each section against the persona's data on a fixed rubric.

The judge also returns the best and worst line of each section, so a low score always comes with an example.

• Baseline and compare. A run writes its averages to a JSON file that I commit. Later runs pass --compare with that file and exit non-zero if any score falls more than 0.3 below the baseline, or if stat-fidelity failures rise at all.
• Cost. About $0.59 a run for three personas in English and Korean (34 sections judged), with a $5 cap. It needs an API key and costs money, so it's a manual gate: I run it before merging a prompt change, not on every commit.

The first run on the rebuilt pipeline found two problems, one in the product and one in the judge.

The product was repeating itself. Repetition scored 2.26 out of 5. I'd been injecting a recurring motif into every section template to tie the report together, and every section dutifully re-wove the same image. The motif now feeds only the opening and closing sections, and the coherence pass prunes it elsewhere instead of adding more.

The judge was wrong about statistics. It flagged 19 sections for unsupported numbers, but many of those flags were for data we had provided: spectrum values between 0 and 1, and raw counts. I rescoped the check to percentage claims, which is what the production guard enforces. So the drop from 19 flags to 4 is partly a better judge, not a better product, and I count it that way.

Next I read full sample reports myself and found two problems the scores hadn't pointed at.

A scale inversion. The model saw “stranger comfort: 2” (on a 1 to 5 scale) next to “connected: 0.6” (on a 0 to 1 scale), compared the raw numbers, and told someone uneasy with strangers that they're “easier with new people”. Nothing in the prompt said which scale was which. Every 1 to 5 value now reaches the model self-describing, as “2/5 (low)”.

Parroted numbers. A number in the outline's narrative thread was being repeated by about five sections per report. My fix was to ban digits from the thread and its hooks entirely.

The compare failed. Specificity fell from 4.0 to 3.53, and to 3.62 on a second run, both past the threshold. Repetition, the thing the change was meant to fix, didn't improve. Without numbers to anchor them, the hooks went vague, and the sections got shy about figures they were allowed to use.

I reverted the digit ban and kept the rest: the scale glosses, localized rarity text, and a softer rule that encourages each section to quote its own figures and only forbids re-quoting the thread's.

The final compare passed on every score but failed on statistics: 8 flags against the baseline's 4. I audited all eight by hand before merging. One was a real invented percentage, which the production guard had already caught and logged. The other seven were the judge's mistakes: a ratio we pass the model that the allowed-numbers list didn't include, a real local percentile, and four flagged lines with no percentage in them at all. I merged with that audit written on the pull request, and added the ratio to the allowed list in the next one.

That's the part of the process I trust most. A failing gate means stop and look, and the looking gets written down.

The scores never flagged these three. Reading the output and the code did.

English inside Korean sentences. In the baseline, Korean sections averaged 4.59 for nativeness against 5.0 for English, so a gap was visible but not its cause. Reading a Korean sample myself, I found English labels and stat phrasing spliced into Korean sentences, like “Active 약점” and “higher than 95% of all Escape users 수준으로”. The judge even picked a Korean line containing the raw field value close_few as that section's worst line, and still scored the section 5 out of 5 for nativeness:

The fix was localized labels in all 15 languages and a rule to phrase numbers natively. The lesson: a score can see a problem without weighing it, and the per-line evidence is where the signal was.

Numbers the model computed itself. A Korean stats section turned raw counts into “60% 매칭 성공률”, a 60% match success rate. No such statistic exists. The stat-bearing templates now forbid derived figures: the model may quote a number we give it, and never compute a new one.

A bug the fixtures hid. In production, sections were ordered alphabetically, so the closing section played third and an opener played eighth. The eval never saw it, because my fixture personas listed their sections in hand-written order. The fix was an explicit section order. The lesson is that fixtures cleaner than production data end up testing the fixtures.

I build Escape with Claude Code, and I hold that setup to the same rule: an automated check you haven't tested is a hope, not a check.

• 12 custom subagents carry domain checklists that general reviewers lack: GraphQL resolver templates against the schema, CloudFormation's 500-resource limit, DynamoDB tables against the services that read them, payments and personal data, translations across 15 languages. A review skill maps a branch's changed files to the right agents and runs them in parallel.
• 29 skills cover releases, deploys, incident triage and similar routines.
• 8 hooks enforce the rules I don't want to rely on remembering. Edits to secrets and lock files are blocked, and so is one AppSync setting that would recreate every resolver. A shell guard stops sed -i or a heredoc from routing around those blocks, and asks before anything that changes production.

The shell guard is tested like any other code: 87 table cases, plus a mutation test that deletes one rule at a time and checks that the table catches every deletion. All 12 mutants are caught today. One of them re-creates a real bug I shipped: the first version treated any mention of the production account ID as production access, so it prompted on a mkdir and on editing a CDK file. Now “production” means selecting production credentials, and that mutant keeps the old behavior from coming back.

The last lesson was about silence. A set of scheduled agent routines had failed on all 37 logged runs with “command not found”, and nothing told me. A startup hook now reads their logs and surfaces any routine that failed or stopped running. A check with no failure signal isn't a check.