skill-receipts

Agent skills for Claude Code where every entry ships with its receipts — a reproducible, accuracy-gated benchmark against a no-skill baseline and a placebo prompt. Skills that fail are published too, with their numbers.

Featured — Audit #4: we ran the ≈196k★ Karpathy Guidelines skill on our exact source-LOC metric. It beats a placebo (−20%) but barely beats no instructions at all (−4.8%) — and our 20-line underkill writes 20% less code than it does, at the same 60/60 hold-out accuracy.

Narrative writeup of what came out of all this: I benchmarked Claude Code skills against a placebo — and half of mine failed

The admission rule (pre-registered in this repo's first commit, 347cb54): no skill gets merged unless it beats both baseline and placebo on its pre-registered target metric, measured on tasks with hidden hold-out acceptance tests, with accuracy not allowed to drop. Rejects go in the table below.

Install

As a Claude Code plugin (all admitted skills at once):

/plugin marketplace add sjh9714/skill-receipts
/plugin install skill-receipts

Or take a single skill: the admitted skills ship as skills/<name>/SKILL.md, which is what npx skills add sjh9714/skill-receipts installs. Rejected skills deliberately ship no SKILL.md — both installers discover skills by scanning for that file, so keeping one would install a reject alongside the winners. Admission is computed from the committed run logs rather than declared, and CI fails if a skill that missed the gate has an installable artifact.

Every skill, admitted or rejected, still publishes its rules as a copy-paste skills/<name>/dist/CLAUDE.md.snippet — byte-identical to what the benchmark measured, which CI enforces. Paste a reject in if you want it; just do it knowing what its table says.

Why a placebo arm

Most "battle-tested" skill collections publish no tests. The few that measure anything compare skill-on vs skill-off — which can't distinguish "this skill works" from "adding any plausible-sounding prompt changes behavior." Every receipt here is a three-way comparison — off / placebo / on — on tasks with hidden hold-out acceptance tests the agent never sees, K=5+ trials per task per arm, isolated workspaces, raw logs and per-run patches committed.

Popular third-party rulesets get the same treatment, published win-or-lose: Audit #4 — Karpathy Guidelines (196k★) ran a widely-installed general behavioral ruleset on underkill's exact metric — it beats a placebo (−20% LOC) but barely beats no instructions at all (−4.8%), and our 20-line skill writes 20% less code than it does. Audit #1 — ponytail (89k★) ran a purpose-built anti-over-engineering ruleset under the same protocol (it won on LOC, at a cost); Audit #2 — caveman (92k★) measured whether terse-mode compression cuts agentic run cost (it didn't — it raised it); Audit #3 — superpowers systematic-debugging (260k★) found the Iron Law produces verified repros at 4× baseline, at +80% cost — and Audit #5 re-ran it at v6.2.0 when the ruleset shrank, finding no detectable change (a published null).

Control note: a reader asked whether the placebo's effect is dose-dependent — the control was shorter than the skill, so part of the measured effect could be the skill offsetting its own padding. A pre-registered length-matched control settles it: 2.2x the control text produced indistinguishable output (CI straddles zero), and underkill beats the length-matched placebo by the same −36.0% it beats the short one. Answering the question also turned up that our own disclosure of the length gap was wrong in our favour; both are corrected there.

Model note: every receipt below is measured on claude-opus-4-8. A pre-registered model-sensitivity pilot re-ran underkill's 12 tasks on claude-opus-5: the LOC effect holds, and the cost effect flips sign — the same rule that cost +5.2% more than doing nothing on 4-8 costs 22.8% less on 5, because the unconstrained baseline there takes 67% more turns for the same amount of code.

Receipts

Everything between the markers below is generated from the raw run logs in bench/results/ and verified byte-identical in CI — nothing hand-written.

✅ repro-first — admitted

verified-repro rate: on 10/20 runs vs off 2/20 vs placebo 0/20. Hold-out acceptance: off 20/20, placebo 20/20, on 20/20. 60 runs, claude-opus-4-8, CLI 2.1.216 (Claude Code), total cost $12.94.

Compliance receipt via the pre-registered pilot-gate fallback (docs/DESIGN.md): the off arm fixed every misdiagnosis task (pilot 12/12, sweep 20/20), so the accuracy axis was unearnable. What the skill measurably changes is that fixes arrive with a machine-verified regression test — one that fails on the pre-fix baseline and passes after — at +$0.07/run and +11 test LOC median. Detector counts added test files only, so this understates compliance where the repro belongs in an existing file (tasks 01/04).

Comparison arm vs-superpowers-debugging (vendored ruleset — same tasks, model, harness, and gates; trial count may differ from the primary arms; never part of admission): verified-repro rate median 0.5, acceptance 20/20, 20 runs. See docs/audits/superpowers-debugging.md.

Comparison arm vs-superpowers-debugging-620 (vendored ruleset — same tasks, model, harness, and gates; trial count may differ from the primary arms; never part of admission): verified-repro rate median 0.5, acceptance 20/20, 20 runs. See docs/audits/superpowers-debugging-620.md.

✅ underkill — admitted

src LOC added: -23.8% vs baseline, -36% vs placebo (medians 10.5 / 12.5 / 8). Hold-out acceptance: off 60/60, placebo 60/60, on 60/60. 180 runs, claude-opus-4-8, CLI 2.1.216 (Claude Code), total cost $22.69.

Comparison arm vs-karpathy (vendored ruleset — same tasks, model, harness, and gates; trial count may differ from the primary arms; never part of admission): src LOC added median 10, acceptance 60/60, 60 runs. See docs/audits/karpathy.md.

❌ Did not make the cut

Skills we wrote, measured, and rejected under the same protocol — full tables included. Published because a results log that only contains wins would not be worth trusting.

❌ tests-that-bite — rejected: does not beat baseline on mutant kill rate (0.944 vs 1); does not beat placebo on mutant kill rate (0.944 vs 1)

mutant kill rate: -5.6% vs baseline, -5.6% vs placebo (medians 1 / 1 / 0.944). Hold-out acceptance: off 16/20, placebo 15/20, on 20/20. 60 runs, claude-opus-4-8, CLI 2.1.216 (Claude Code), total cost $19.98.

❌ thrift — rejected: does not beat baseline on cost per run (USD) (0.168 vs 0.163)

cost per run (USD): +3.3% vs baseline, -3.5% vs placebo (medians 0.163 / 0.174 / 0.168). Hold-out acceptance: off 32/32, placebo 32/32, on 32/32. 96 runs, claude-opus-4-8, CLI 2.1.216 (Claude Code), total cost $25.00.

Comparison arm vs-caveman (vendored ruleset — same tasks, model, harness, and gates; trial count may differ from the primary arms; never part of admission): cost per run (USD) median 0.183, acceptance 20/20, 20 runs. See docs/audits/caveman.md.

Methodology

Full protocol, per-skill pre-registered targets, amendments, and known limitations: docs/DESIGN.md. The harness was built and validated in underkill (300 published runs); this repo generalizes it to N skills with per-skill task sets, a placebo arm by default, mutation-testing receipts, and a repro-verified detector.

Want a skill admitted (yours or a famous one audited)? Open an issue — the gate is the same for everyone.

License

MIT