Driftproof

Someone says this AI skill makes the assistant better. Does it?

An AI skill is a file of instructions that people add to an AI assistant, such as Claude, to make it better at a job. Driftproof tests whether a skill really helps, the way a medicine is tested against a sugar pill. The same test works on custom instructions, saved prompts and rules files for Claude Code, Cursor, Copilot and Windsurf. The latest reports give the AI the same tasks three ways: with the skill, with a fake of the same length, and with nothing added. Then it scores every answer, again and again. Every result is a dated receipt with its own page; anyone with a terminal can verify the file.

18 reports published

319 receipts published

12 models measured on

Comparing two benchmark scores? The gap calculator says whether the gap is bigger than the benchmark's own sampling noise.

For the curious

How a skill is tested

  1. Run the skill's test tasks twice: with the skill, and without it.
  2. Draw each answer several times and score each draw several times, so every result is a range, not one fragile number.
  3. Call it a separation only when the ranges don't overlap and the move clears the floor. When the ranges overlap, say no separation detected at this sample size; when the draws were too few to tell, say so.

That range is a band. The dated file that records all of it is a receipt. What changes between two receipts is drift.

Two horizontal ranges, drawn apart.
Separated The two ranges do not overlap, and the move clears the floor. A separation is detected under the rule, which is not proof that the skill moved.
Two horizontal ranges, drawn crossing each other.
Overlapping The ranges share space. We say no separation detected at this sample size, and we say it plainly.
Two horizontal ranges, drawn with one hatched because nothing was measured.
Refused The run could not stand behind a number, so it reports no result instead of guessing one.
The same two ranges, drawn from the bundled example receipt.

Move the floor and change what counts as a band to see the verdict change. This is the bundled example receipt's own per-case data.

Walk through one real test, step by step, or try to tell which answer had the skill.

The reports

Latest report

Placebo comparison report

Report 019: Skill or sugar pill, on Fable 5.2? Twenty skills against a same-length placebo

What moved: a same-length placebo text

claude-fable-5-2

2026-11-11

Band plot for the first cell of Report 019. 0 0.25 0.5 0.75 1 separated without the skill 0.738 with the skill 0.963

The latest report reads each skill against a same-length placebo as well as against no skill: a text of the skill's length in bytes, in its place, drawn from a fixed corpus chosen without reference to any task. See your file's fake shows the text Driftproof would use for your file, in your browser, before you measure it. On a run that kept its answers, the results page and driftproof blind show each task's two answers without saying which used your file, ask you to pick, and after your picks say how often you were right beside the judge's scores. In Claude Code, /driftproof:blind opens it.

All 18 reports

The same skill on different models and dates: what each receipt read, model by model.

Writing: What happens to agent skills when the model changes, the launch essay reading all eighteen reports on what a moving model does to encoded expertise.

For people who use AI skills

Using a skill someone else wrote?

Paste its link and see whether it has been measured, or browse the skills already on record. Look up a skill. Do popular AI skills help? Browse skill badges.

Before you measure a file of your own, open See your file's fake: the same-length text that stands in for the sugar pill.

For people who make skills

What a receipt can say

A skill's own tests passing is one answer. Driftproof asks two more: whether the scores with the skill and without it separate beyond their spread, and whether that held when the model changed. Each answer is a dated, hash-verified receipt, and when there were too few draws to tell, the receipt says so.

  • Clearly helped. Receipt label: Passing. A case separated upward under the band rule: the two ranges do not overlap and the move clears the effect floor. That is a separation detected under the rule, not proof that the skill moved the score.
  • Clearly hurt. Receipt label: Regressed. A case separated downward under the same rule.
  • No clear difference. Receipt label: No separation detected. The ranges overlap at this sample size, which is not evidence that nothing changed.
  • Too few answers to tell. Receipt label: Not enough draws. Not enough draws to conclude at this effect floor. The receipt prints how many draws per arm would have been needed, or that no number of draws reaches the floor at the spreads it measured.
  • Too few test tasks to tell. Receipt label: More cases needed. More cases, not more draws, are needed to conclude at this effect floor. The cases score too far apart from each other for more draws to be expected to bring the comparison band under the floor, and the receipt prints the band with each case's draw-to-draw spread taken out.
  • Not measured. The receipt carries no verdict, and its page names every reason, such as a run that does not record that a model answered it.

Every receipt has its own page, with its verdict, its two arms and a way to verify it yourself. How this is built describes how the tool itself is checked.

A receipt, in full

Verdict Passing

Model claude-haiku-4-5

Date 2026-09-17

Skill commit-message-conventions

Range 0.434 without, 0.822 with

commit-message-conventions-claude-haiku-4-5-20251001-2026-09-17. Passing. Without the skill, 0.434 plus or minus 0.188. With the skill, 0.822 plus or minus 0.108. 0 0.25 0.5 0.75 1 Passing without the skill 0.434 with the skill 0.822

the two ranges do not overlap: separated

Receipt hash fad883d1018d3a4b3c75bc60b1eaec50032256cbf69fba3299999edf7978589c

Skills don't break. Models move.

18reports published
319receipts published
12models measured on
1574eval cases scored
0refusals published
243runs with no separation detected at their sample size

On the report that measured it, draw-to-draw spread reaches sd 0.355 on a single case (semver-hidden-breaking-change). That is the across-draw spread of the model writing a different answer, not of the scorer re-reading one, and it is wider than the scorer's.

We publish refusals instead of guesses, and we publish our own instrument defects, including what outside audits found. Read the methodology, or read what we will not claim.

Who it's for

People who use AI chat apps

See whether your custom instructions do anything, with no install.

Skill authors

See whether it helps before you publish, and on which model.

Catalog maintainers

See which of your skills stopped separating when the model changed.

Platform teams

An evidence trail for the skills your agents run in production.

Install in Claude Code

You need Claude Code, Node.js 22 or newer, and your project in a git repository. You do not need an API key: runs use your Claude Code plan's usage.

claude plugin marketplace add driftproofhq/driftproof

claude plugin install driftproof@driftproofhq

Each command starts the pinned runner with npx, so a first run has no extra install step. Two gaps remain, and the plugin's words cannot close them: a program in a parent folder's node_modules/.bin can run before the runner, and the line Claude runs can be altered. The safer alternative is to install the runner with npm install -g, at the version the command names, and use the installed line each command file gives. npm then starts nothing, so the first gap closes. The line Claude runs can still be altered in either form.

That installs /driftproof:start, /driftproof:init, /driftproof:run, /driftproof:blind, /driftproof:badge, /driftproof:compare, /driftproof:inspect, /driftproof:recheck and /driftproof:measure. The plugin hands your arguments to the pinned runner and writes the receipt that runner would have written; /driftproof:run spends your Claude Code subscription rather than an API key. It measures and never changes a skill's instructions. The one file it adds is a new evals/evals.json, from /driftproof:start or /driftproof:init, to a skill that has none, after you say yes in the run.

Compare and recheck need a git repository, and compare takes a rules file: CLAUDE.md, AGENTS.md, .cursor/rules/*.mdc, .cursorrules, .github/copilot-instructions.md, .github/instructions/*.instructions.md, .windsurfrules, .windsurf/rules/*.md.

A full run can take up to 120 model calls for each test task from your plan's usage. A quick check takes far fewer.

Start with a skill you already have. In Claude Code, from the folder above it, in a git repository, run /driftproof:start. Claude lists your skills and asks which to check. If the skill has no test cases, Claude reads your SKILL.md and drafts test cases with you, aimed at outcomes you would check. It writes only after you say yes, then runs a quick check and opens a results page: your first receipt. Drafted cases are labelled as drafted from the skill and approved by you. A quick check is a first look and cannot produce a verdict. For a direct measurement, use /driftproof:run path/to/your-skill. On a skill with no test cases, it offers the same draft. For a new skill, scaffold one with npx driftproof init my-skill, as the next section shows. /driftproof:start works on these rules files too: CLAUDE.md, AGENTS.md, .cursor/rules/*.mdc, .cursorrules, .github/copilot-instructions.md, .github/instructions/*.instructions.md, .windsurfrules, .windsurf/rules/*.md. Measured as always-on text. The editor's system prompt and rule selection are not reproduced.

Tested on Linux and macOS. On Windows the plugin also needs Node, npm's npx and Git for Windows on PATH, and Windows has only been checked without calling a model. The findings page has the test details.

In CI: npx and the GitHub Action

You need Node 22 or newer and an ANTHROPIC_API_KEY. A Claude Pro or Max plan cannot be used in CI: the Action needs an API key, billed per call, so set max-usd. From a clean checkout to a receipt for a new skill, on the command line:

npx driftproof init my-skill          # scaffold SKILL.md + evals/evals.json + .driftproofrc
# edit the 3 example cases so each is grounded in a claim your SKILL.md makes
export CLAUDE_PROVIDER=api
read -rsp "Anthropic API key: " ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY
npx driftproof run my-skill --models claude-haiku-4-5
cat receipts/*.summary.md              # read the receipt + human summary

Wire it into a repository with the GitHub Action (uses: driftproofhq/driftproof@v0.30.0): it re-runs the suite on every push, uploads the receipts, and fails the job when a case separated downward. driftproof badge turns a receipt into a badge that carries the model and the date it was measured.

Copy the pull request workflow into .github/workflows/ and add an ANTHROPIC_API_KEY repository secret. It checks changed skills and rules files with test cases, then updates a receipt table on each push. The budget is shared across files. Files without cases are named as not measured.

The step that matters, with its settings:

uses: driftproofhq/driftproof@v0.30.0
with:
  api-key: ${{ secrets.ANTHROPIC_API_KEY }}
  placebo: 'true'
  max-usd: '20'
  fail-on-regression: 'true'
  fail-on-fake: 'false'

Skills need evals/evals.json in their folder. Rules files need it beside the file. The workflow pins a release. Update the pin when you choose to adopt a newer release. Forks make no model calls and get a notice in the job summary; GitHub may block their comment. The receipt links open the job's artifacts, which follow GitHub's retention period.

fail-on-regression is on by default. Turn on fail-on-fake to fail on worse than the fake, never just because it did not beat the fake. That setting needs the placebo arm enabled, as in the example. For OpenAI models, set models and openai-api-key. Only API keys are supported. A green check does not prove a file helps.

To compare a pull request's supported rules file with the version on its base branch, on the same tasks, set the Action's compare-with-base input.

driftproof badge: passing on claude-haiku-4-5

Get the next one

Your address goes to Buttondown, our newsletter provider, and nowhere else. One email per model release. Unsubscribe any time.

Every report is also in the Atom feed.

Common questions

Roadmap

Words with a dotted line link to the glossary, where each is explained in plain words.