red-green-mode

Your coding agent says the tests are green. These seven CLIs check whether it cheated to get there.

English · 中文

Coding agents are graded by their own test suite, so the cheapest way to "pass" is to attack the grader: delete the assertion, @pytest.mark.skip the failure, sprinkle # type: ignore, or write a test that never had teeth in the first place. You get a green checkmark and a broken product.

red-green-mode is a referee. Zero dependencies, pure Python stdlib, no network, no LLM calls — just deterministic exit codes that any agent, any language, any CI can consume.

30 seconds, on your machine

git clone https://github.com/hidevinliu/red-green-mode ~/rgm && export RGM=~/rgm
mkdir -p /tmp/demo && cd /tmp/demo && git init -q .

A function with a real bug, and a test that catches it:

# billing.py
def apply_discount(price, pct):
    return price - price * pct / 100

# test_billing.py
def test_rejects_over_100():
    try:
        apply_discount(100, 150)
    except ValueError:
        return
    raise AssertionError("should reject pct > 100")

Gate 1 — did it fix the code, or fix the test?

A lazy agent "fixes" it without touching billing.py:

[email protected](reason="flaky")
 def test_rejects_over_100():
-    try:
-        apply_discount(100, 150)
-    except ValueError:
-        return
-    raise AssertionError("should reject pct > 100")
+    assert True

$ python3 -m pytest -q
1 passed, 1 skipped          # ← the agent reports success

$ git diff > /tmp/d.diff
$ python3 $RGM/tools/rgm_anticheat.py scan --diff-file /tmp/d.diff --format sentinel
ANTICHEAT=FAIL
FINDINGS=3
WARNINGS=0
ALLOWS=0
$ echo $?
1

Now the honest fix — add the missing guard clause to billing.py, leave the test alone:

$ python3 -m pytest -q
2 passed

$ git diff > /tmp/honest.diff
$ python3 $RGM/tools/rgm_anticheat.py scan --diff-file /tmp/honest.diff
{"clean": true, "findings": [], "allows": []}
$ echo $?
0

Same green suite. Opposite verdict.

Gate 2 — is the test even biting the code?

Anti-cheat only sees what changed. A test that was toothless from birth passes it clean. So the second gate mutates your production code and demands the test go red:

# test_dead.py — green forever, tests nothing
def test_it_runs():
    try:
        apply_discount(100, 10)
    except Exception:
        pass

$ python3 $RGM/tools/rgm_mutation.py check-pair \
    --verifier "python3 -m pytest -q test_dead.py" \
    --target "billing.py::apply_discount" --format sentinel
RGM_MUTATION=FAIL
pair target=billing.py::apply_discount DEAD (survived 6/6)
$ echo $?
1

$ python3 $RGM/tools/rgm_mutation.py check-pair \
    --verifier "python3 -m pytest -q test_billing.py" \
    --target "billing.py::apply_discount" --format sentinel
RGM_MUTATION=PASS
pair target=billing.py::apply_discount ALIVE (killed by: L2:'if not 0 <= pct <= 100:')
$ echo $?
0

Six mutations injected, six restored byte-for-byte (finally + on-disk sidecar + a restore subcommand for crash recovery). Your working tree is unchanged afterward.

What's in the box

Seven tools that return an exit code. Nothing here asks a model for an opinion.

Anti-cheat rules (9 total — 7 blocking, 2 advisory): test skips in Python / JS-TS / Go-Rust, static-analysis suppressions (# noqa, @ts-ignore, eslint-disable, #[allow(...)]), tautological assertions, deleted assertions and test functions, linter-strictness downgrades, plus two warn-only smells (mocking the thing under test, "hardcoded to pass the test" comments).

It only reads the added/removed lines of a diff — a # type: ignore that was already in your codebase is not this run's crime.

Escape hatch. Sometimes skipping a test is legitimate. Write the reason inline:

@pytest.mark.skip(reason="upstream API down")  # rgm-allow: vendor outage, ticket OPS-412

The finding is downgraded to allowed and the reason is written into the run ledger. The difference between cheating and a judgment call is whether you left an auditable reason.

Install

git clone https://github.com/hidevinliu/red-green-mode
python3 -m pytest red-green-mode/tests/ -q      # 262 passed in ~20s

Requirements: Python 3.9+ and git. That's it — the tools import nothing outside the standard library. pytest is only needed to run this repo's own test suite, not to use the tools on your project. There is no package to install, no server to start, no config file to write.

As a Claude Code / Codex skill

git clone https://github.com/hidevinliu/red-green-mode ~/.claude/skills/red-green-mode

SKILL.md then triggers on phrases like "keep fixing until the tests pass" / "红绿灯模式" and drives the loop: pick a verifier → run → classify the red → fix → re-run → run the gate before declaring done.

Skills alone are advice. A model that decides to stop can stop. For an actual block, wire the Stop hook (tools/rgm_stop_hook.sh) into settings.json — it exits 2 when the gate fails, and it is fail-closed: if the gate itself can't run, you still don't get to finish. See tools/STOP-HOOK-INSTALL.md.

In CI, with no agent involved

rgm_anticheat.py is just a diff scanner. It works on human pull requests too:

- name: Block test-tampering in this PR
  run: |
    git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff
    python3 tools/rgm_anticheat.py scan --diff-file /tmp/pr.diff --format sentinel

What this does not do

Read tools/ANTICHEAT-LIMITATIONS.md before you trust it. The short version of the two that matter most:

• Production-code gaming is largely undetected. If the agent hardcodes return 90 in billing.py so the test passes, regex over a diff will not catch it. Mutation testing partly covers this by asking whether the test has teeth, but "agent writes wrong-but-tested code" is an open problem, not a solved one.
• The ledger is trusted input. rgm_gate.py executes the verifier commands it finds there via a shell. Anyone who can write your ledger can make the gate run arbitrary commands and return PASS. Treat the ledger exactly like a Makefile: it is code, review it as code.

Also honest about scope: the evals/ directory holds 53 hand-written scenario rubrics with no automated runner. They are a human review checklist, not a passing benchmark. Don't read them as "53 evals green".

Prior art, and where this sits

• nizos/tdd-guard blocks the agent before it writes code without a failing test. This project judges after: the suite is green — is it green for a real reason? Complementary, not competing.
• Anthropic's built-in verification loop gets an agent to re-run its own checks. It does not ask whether the agent tampered with the checks. That gap is the entire point of this repo.
• Academic work is converging on the same problem from the benchmark side — SpecBench, EvilGenie, TRACE. This is the small, boring, exit-code-shaped version you can put in a hook today.

License

MIT — see LICENSE.