eval-design

You shipped an AI feature. It's okay. How do you make it better?

Not by rewriting the prompt and seeing if it feels sharper. By finding out exactly what it gets wrong, fixing that specific thing, and proving the fix worked — then repeating until the remaining problems are ones you can live with.

This is a skill for Claude Code that sets up that loop, plus a script that renders each run as a scorecard.

Let's use an example

You run a flight and hotel booking site. You've replaced the old FAQ widget with a chatbot, and it's live. Customers seem happy enough. Now what?

Take one message: "cancel my flight to Lisbon and refund me." Your bot can end up in very different places:

flowchart LR
    A["Cancel my flight to Lisbon"] --> B["Chatbot replies"]
    B --> C["Handles it correctly"]
    B --> D["Invents a refund policy"]
    B --> E["Cancels without asking first"]

A single "accuracy: 87%" treats those last two as ordinary mistakes. They aren't. One creates a refund you must honour or publicly refuse; the other cancels a flight that can't be uncancelled. And crucially — neither one tells you what to go and build next.

So you score the things that actually differ, and the eval becomes a to-do list.

Round one: what's actually broken?

You take 140 real conversations, write down what good looks like, and run your current bot against them. Here's what comes back:

Look at the bottom row — that's v1, the version with the FAQ pasted into the prompt that everyone builds first. Out of 140 conversations it invented a policy 25 times and changed or cancelled a booking without asking 40 times.

That's not a vague quality problem. That's two engineering tasks:

1. It's guessing at policy because it doesn't have the policy documents.
2. It's acting without permission because nothing forces it to ask.

Round two: fix those two things, prove it worked

You give it the real policy documents (v2), then read access to the actual booking plus a mandatory confirm step (v3). You re-run the same 140 conversations each time:

Each fix is attributable to a change you made, on the same examples, so nobody has to take your word for it. That's the loop.

Two versions are struck through. They failed a floor — a minimum you don't trade away no matter how good the rest of the score is. v1 and v2 both changed bookings without asking, so neither ships at any price or any score. That's a gate, and it's the most useful structure on the page.

Round three: the target moves

v3 clears both floors. So is it done?

No — the eval now points somewhere completely different. Go back to the failure chart and look at v3's row: its biggest bucket by far is 31 unnecessary handoffs, conversations it passed to a human without needing to. The safety problems are solved; the remaining problem is that it gives up too easily.

That's the next sprint, and you'd never have found it by reading transcripts and feeling good about how much safer the bot got.

Under each criterion is the disagreement it was written to settle — the argument two reasonable people had before the rule existed. That's the test of a real criterion. "Is the bot helpful?" fails it. "Does it quote the real policy?" passes.

And look at the last column: the 2019 FAQ widget you're replacing beats every model on three of five criteria. It has never invented a policy in its life, because it can only return canned text. It scores 18% on resolving anything, which is exactly why it's being replaced. Stated plainly — what you're buying is coverage, not correctness. The old widget was already correct.

Knowing that changes what you work on next.

Know when to stop

The same setup on a cheap model scores 86.5% against v3's 91.3% — but the margin of error on both is around ±5, so on this evidence you cannot say the expensive one is better overall. Improvement inside the noise isn't improvement; it's a coin flip you're about to spend money on.

What justifies the 6x here is one specific number, not the headline: v3 resolves 78% of conversations without a human against 71%. Seven points of deflection pays for it. If that gap closes, the cheap model becomes correct — and writing that down is part of the job.

The words, in plain English

The loop

flowchart TD
    A["1. Name the decision"] --> B["2. Read 40 real chats, group the mistakes"]
    B --> C["3. Write the rubric"]
    C --> D["4. Pick 140 example chats"]
    D --> E["5. Run every version, plus the old FAQ widget"]
    E --> F["Scorecard: what to fix next"]
    F --> G["Ship the fix"]
    G --> E

Step 1 is the one people skip and the most valuable. Here it was: can this thing cancel a booking on its own? If no result would change your plan, you don't need an eval — and saying so out loud is the cheapest thing this will ever save you.

Step 2 comes before the rubric on purpose. Criteria invented in a meeting measure what you imagined going wrong. Criteria taken from real transcripts measure what does.

The arrow back from step 5 is the part that matters. A scorecard you run once is a report; a scorecard you run every time you change the prompt is a safety net — it's how you find out that the fix which raised resolution rate quietly broke the confirm step.

Why gates exist

A weighted average will happily let a system buy its way past a safety failure by being charming everywhere else. v1 was charming:

flowchart TD
    A["A chatbot version"] --> B{"Ever cancelled a booking without asking?"}
    B -->|Yes| C["Out. No score buys that back"]
    B -->|No| D["Score it, then compare on cost"]

The four rules

These are opinions. They're the reason this exists rather than being another test harness.

1. No vibes. If a change can't be shown as movement on a defined criterion over a fixed set, it wasn't evaluated. Ship it anyway if you like — just don't say it was tested.

2. Every criterion must be arguable. If two careful people couldn't land on opposite sides of it, it isn't measuring anything.

3. Don't let an AI grade the homework by default. An AI judge has its own error rate. Humans score first, the judge gets checked against those scores one criterion at a time, and its agreement is published next to every number it produced.

4. Always include the boring baseline. A regex, a keyword rule, whatever you run today. In three of the four examples here, the boring option wins a column outright.

Try it

You don't need Claude Code to see it work. The renderer is plain Python with no dependencies:

git clone https://github.com/YoursSarcastically/eval-design
cd eval-design
python3 skills/eval-design/scripts/render_scorecard.py examples/travel-booking-chatbot/results.json
open examples/travel-booking-chatbot/scorecard.html

To use it as a skill, copy it into your project:

cp -r eval-design/skills/eval-design .claude/skills/

Or into ~/.claude/skills/ to have it everywhere. Then ask for what you need:

• "Our booking chatbot is live and I don't know what to fix next. Set up an eval."
• "Here are 40 transcripts — group the failures for me."
• "Is this rubric any good?" — and paste it
• "Did my prompt change actually improve anything, or is it inside the noise?"
• "We want to use an AI judge for tone. Should we?"

Check your work

python3 skills/eval-design/scripts/validate_rubric.py results.json

Fails on the things that quietly make a scorecard dishonest: weights that don't add up, a criterion with no stated disagreement, an AI judge with no accuracy figure, a recommendation another option beats outright. Warns about a missing baseline and too few examples.

It caught three real problems in this repo's own examples while they were being written. Those got fixed, rather than the checks loosened.

The examples

Each is built around a finding a single accuracy number would have hidden.

Every folder has the rubric, the results file, the rendered page, and a markdown summary you can read without leaving GitHub.

The numbers in these examples are made up, and systems are named generically ("v3 on a small model") rather than after real products — see examples/NOTE.md. Publishing invented benchmark scores against named commercial models would be a fake benchmark, and fake benchmarks get screenshotted and quoted as fact. The patterns are real; the measurements are illustrations.

When you shouldn't use this

• A quick prompt tweak with no decision attached. Just look at the output.
• No users and no real examples yet. Get 30 real inputs first. An eval built on imagined inputs measures your imagination.
• The answer is checkable by a program — valid JSON, correct arithmetic, a build that compiles. Write the check, not a rubric.
• Someone wants a number to justify a decision already made. Say that out loud instead of building the eval.

Under the hood

Standard-library Python and self-contained HTML. No dependencies, no CDN, no fonts to fetch, no network calls of any kind — a scorecard usually ends up pasted into a ticket or mailed to an exec, and it has to work offline.

The whole visual design lives in one TOKENS dictionary at the top of render_scorecard.py. Change eleven lines and the system restyles. Light and dark come from one stylesheet.

Contributing

A new domain example is the most useful thing you can add — one results file, one rubric, and one finding a plain accuracy score would have hidden. See CONTRIBUTING.md.

MIT.