Verify Regression Tests

Test the test, not only the code.

skills.sh directory · 한국어

Your regression test is green. But does it actually catch the bug?

A green result proves the test passed — not that it guards anything. verify-regression-tests is an open Agent Skill that closes that gap with one focused counterfactual, keeping the regression test frozen throughout:

1. candidate implementation   + frozen target test  -> GREEN
2. faithful known-bad control + frozen target test  -> RED at the intended witness
3. restored candidate         + frozen target test  -> GREEN

If step 2 never turns RED — or turns RED for the wrong reason, such as an earlier validation gate or an unrelated exception — the test was not protecting you, and the skill reports that instead of trusting the green.

All six synthetic evaluation cases shipped with this repository completed that loop end to end. The verified case table, host runs, environment, and evidence limits are published in EVALS.md.

The skill is framework-neutral: Claude Code and Codex run the same SKILL.md.

Quick start

Run this from the project where you want the skill available:

npx skills add aiopshwang/verify-regression-tests \
  --skill verify-regression-tests \
  -a claude-code \
  -a codex

Both hosts can also select the skill implicitly when a request matches its narrow description. Try requests such as:

Use the regression-test verifier on this bug guard. Prove the unchanged test
fails for the original defect, not an earlier validation error.

This retry regression test is suspiciously green. Check whether the cache or
mock bypasses the retry branch, and strengthen only the test if needed.

Verify this existing idempotency regression test. If it already kills a faithful
known-bad control, do not change files; report the evidence.

Why green tests lie

A regression test can stay green while the original bug is present because:

• an earlier validation gate rejects the fixture first;
• a mock, cache, fallback, or default bypasses the target branch;
• the test is skipped, filtered out, not awaited, or running a stale artifact;
• another detector satisfies a broad assertion;
• the assertion accepts both the broken and correct behavior.

A passing suite does not reveal any of those failures. Only a deliberate counterfactual does — and the known-bad control must reproduce the target behavior independently of the test. A compile error, unrelated exception, unreachable mutation, or equivalent change is not proof.

What a verification run does

The skill:

1. defines the trigger, bad behavior, expected contract, decision point, and observable witness;
2. confirms the exact target test is discovered and executed;
3. creates one safe, reversible, faithful known-bad control;
4. validates that control independently of the target test;
5. rejects false greens and wrong reds caused by gates, fixtures, mocks, fallbacks, artifacts, or broad assertions;
6. restores the candidate exactly and reruns the relevant checks;
7. reports Verified, Not verified, or Inconclusive without extending the claim beyond the exercised defect and environment.

How it differs

The skill can use a targeted mutation when it faithfully recreates the defect, but it does not install mutation tools or optimize a mutation score.

Evaluation you can rerun

The repository includes six dependency-free synthetic fixtures across Node.js, Python, and Dart — one per failure mode:

• wrong upstream gate;
• retry path hidden by cache fallback;
• target detector masked by another detector;
• overly broad exception assertion;
• weak invariant assertion;
• an already-strong idempotency test that should remain unchanged.

All six baselines are intentionally green, and in the recorded v0.1.0 evaluation every case finished Verified through the full candidate GREEN → known-bad RED → restored GREEN loop. Replay the proofs locally:

python3 evals/check_baselines.py
python3 evals/run_proofs.py

These are six synthetic verified cases, not a statistical benchmark. Behavioral results, recorded real-host invocations, routing checks, host versions, and evidence limits are in EVALS.md.

Safety boundaries

Known-bad controls are temporary evidence. The skill requires an isolated copy or worktree when exact in-place restoration is uncertain and never authorizes a commit, push, deployment, production service call, shared-data migration, or external account change.

It preserves unrelated work, keeps the test frozen during candidate/control comparison, and reports an inconclusive result when a safe control cannot be constructed.

See SECURITY.md and PRIVACY.md.

Limits

This skill verifies a named regression guard at the boundary actually exercised. It does not prove that the full suite is strong, that no other bugs exist, or that unit evidence applies to integration, device, production, or deployment behavior.

It is designed to work with any repository that has an executable test command, including Jest, Vitest, pytest, JUnit, Dart, and Flutter projects, but v0.1.0's published synthetic evaluations cover Node.js, Python, and Dart only.

Open standard

The skill follows the open Agent Skills format documented by OpenAI, Claude Code, and the Agent Skills specification.

aiopshwang skill family

Independent, evidence-first Agent Skills that work well together:

• goal-to-proof — the general completion gate: finish authorized work and prove the requested outcome.
• ship-mobile-app — production mobile work across domain, state, lifecycle, platform, and release boundaries.
• data-analysis-ml-agent-skills — decision-grade data analysis and ML: audits, leakage-safe experiments, validation, reproducible handoff.
• fresh-eyes-check — a context-free second model checks whether an earlier instruction still fits before you act on it.

Contributing and license

See CONTRIBUTING.md for evidence and privacy requirements. Licensed under the MIT License.