🐀 ratctl

A Static + Dynamic Auditor for Detecting Reward-Hacking Vulnerabilities in RL Environments

Fuzz your verifier before an RL agent does.

🤗 Hugging Face Live Demo — The live demo is a browser-based JavaScript port of ratctl's regex detection engine (covering all 6 exploit classes). For full AST-based analysis, multi-file directory scanning, format auto-detection, and dynamic LLM fuzzing, install the CLI: pip install ratctl.

⚡ Quick Demo

Catching a multi-vector reward hack and enforcing a fail-closed CI gate:

$ ratctl audit ./vulnerable_env --fail-on 'gameability>0.3'

============================================================
  RATCTL AUDIT REPORT
============================================================

  Gameability Score: 52/100
  Total Findings:   6
  Files Scanned:    11
  Format Detected:  openenv (99%)

------------------------------------------------------------
  FINDINGS BY EXPLOIT CLASS
------------------------------------------------------------

  [GRADER_MANIPULATION] - 2 finding(s)
    Max severity: critical

    1. [CRITICAL] Private frame access
       File: server\app.py:6
       Using sys._getframe() to access interpreter stack frames. Enables grader detection and conditional cheating.
       Evidence: 2. Stack frame introspection via sys._getframe()
       Fix: Block sys._getframe by running in a restricted Python environment.

    2. [CRITICAL] Private frame access
       File: server\app.py:27
       Evidence: caller = sys._getframe(1)
       Fix: Block sys._getframe by running in a restricted Python environment.

  [PREMATURE_TERMINATION] - 2 finding(s)
    Max severity: critical

    1. [CRITICAL] sys.exit(0) - premature success
       File: server\app.py:7
       Calling sys.exit(0) could terminate the grading process with a success exit code before actual verification completes.
       Evidence: 3. Premature exit via sys.exit(0)
       Fix: Trap sys.exit() in the grading harness. Run agent in an isolated subprocess.

    2. [CRITICAL] sys.exit(0) - premature success
       File: server\app.py:34
       Evidence: sys.exit(0)
       Fix: Trap sys.exit() in the grading harness.

  [TEST_TAMPERING] - 1 finding(s)
    Max severity: critical

    1. [CRITICAL] Test file deletion
       File: server\app.py:24
       Code attempts to delete test/verification files.
       Evidence: os.remove("tests/test_solution.py")
       Fix: Mount test files in a read-only filesystem layer.

  [ENV_HIJACKING] - 1 finding(s)
    Max severity: critical

    1. [CRITICAL] Git history access
       File: server\app.py:30
       Accessing git history could reveal previous solutions, test changes, or answer keys.
       Evidence: subprocess.run(["git", "log", "-n", "1"])
       Fix: Remove .git directory from the agent's sandbox.

============================================================
FAIL: Gameability score 52/100 exceeds threshold 30%
Exit code: 1

💡 Why ratctl?

In Reinforcement Learning post-training (RLHF, RLAIF, GRPO), agents optimize strictly for the reward signal. If a grading environment has logic flaws, the agent will learn to hack the verifier instead of solving the task.

Recent research highlights how common this is:

• Terminal Wrench (Bercovich et al., 2026): Cataloged 331 hackable environments and 3,632 exploit trajectories across terminal-agent benchmarks — over 15% of standard benchmark tasks were bypassable without solving the core task.
• SWE-bench Verified Audit (Rajan et al., 2026): Showed that 28.5% of audited code-generation tasks were Docker-verified hackable (e.g. agents reading ground-truth fixes directly from local .git logs).

ratctl provides a practical CLI tool and CI scanner to catch these common reward-hacking patterns before you deploy environments for agent training or publish them to hubs.

📊 Empirical Security Audit: 112 Environments Scanned

We conducted an empirical security audit using ratctl across a diverse dataset of 112 RL post-training environments (spanning OpenEnv Hub tasks, Prime Intellect verifiers spec environments, Gymnasium wrappers, and SWE-bench tasks).

See the standalone technical paper in PAPER.md and full dataset findings in AUDIT_REPORT.md.

$ python scripts/generate_real_world_audit.py

Empirical Audit Metrics

Framework Audit Breakdown

🧪 Initial Canonical Validation Suite

In addition to the 112-environment audit, we maintain a canonical 10-environment unit suite in benchmarks/ representing explicit exploit reproductions from Terminal Wrench and SWE-bench Verified alongside hardened controls:

$ ratctl benchmark benchmarks --format markdown

🏗️ System Architecture

ratctl combines static AST analysis with optional dynamic LLM red-teaming:

flowchart LR
    subgraph INGESTION["1. Ingestion and Detection"]
        A["Target Directory"] --> B["Format Detector"]
        B --> C1["OpenEnv Adapter"]
        B --> C2["Verifiers-Spec Adapter"]
        B --> C3["Gymnasium Adapter"]
        B --> C4["Raw Adapter"]
    end

    subgraph AUDIT["2. Dual-Mode Audit Pipeline"]
        C1 --> D["Source File Graph"]
        C2 --> D
        C3 --> D
        C4 --> D
        D --> E["Static AST Engine (6 Detectors)"]
        D --> F["Dynamic LLM Fuzzer (Subprocess Sandbox)"]
    end

    subgraph SCORING["3. Scoring and Enforcement"]
        E --> G["Weighted Scoring Engine"]
        F --> G
        G --> H["Report Renderers (Rich / JSON / Text)"]
        G --> I["CI/CD Gate (--fail-on)"]
    end

🎯 Exploit Taxonomy

ratctl checks for 6 core categories of verifier vulnerabilities:

🚀 Quick Start

Installation

Choose the method that fits your setup:

# Method 1: Install from PyPI (adds `ratctl` to your PATH)
pip install ratctl

# Method 2: Install in dev mode from source (editable, adds `ratctl` to PATH)
git clone https://github.com/FreakyAdy/Reward-Hackability-Auditor--CLI---Claude-Skill-.git
cd Reward-Hackability-Auditor--CLI---Claude-Skill-
pip install -e .

# Method 3: Run directly without installing (no PATH setup needed)
python -m ratctl.cli audit ./my_environment

Optional extras for dynamic LLM fuzzing:

pip install "ratctl[ollama]"    # Local Ollama red-teaming (free)
pip install "ratctl[frontier]"  # GPT-4o / Claude 3.7 red-teaming

Basic Commands

# Run a static audit on any environment directory
ratctl audit ./my_environment

# CI Gate: block PRs if gameability score exceeds 30%
ratctl audit ./my_environment --fail-on 'gameability>0.3'

# Dynamic LLM Red-Teaming (uses local Ollama — 100% free)
ratctl audit ./my_environment --dynamic

# Launch full-screen animated Terminal TUI Dashboard
ratctl tui ./examples/vulnerable_env

# Launch interactive Web Dashboard UI in browser
ratctl ui ./my_environment

# Output structured JSON for security telemetry
ratctl audit ./my_environment --format json -o audit-report.json

💡 If ratctl is not recognized, use python -m ratctl.cli instead:

python -m ratctl.cli tui ./examples/vulnerable_env
python -m ratctl.cli audit ./my_environment

Runnable Examples

Explore the examples/ directory to test ratctl against clean vs. vulnerable environments back-to-back:

# Audit clean hardened environment (0 findings, score: 0/100)
ratctl tui ./examples/hardened_env

# Audit vulnerable environment (6 findings, score: 52/100)
ratctl tui ./examples/vulnerable_env

📡 Live In-Training Monitoring (ratctl watch) (Beta)

While ratctl audit secures verifiers pre-deployment, ratctl watch monitors verifiers in-training during RL policy optimization (GRPO, PPO, TRL).

Drop-in Python Decorator

import ratctl

@ratctl.watch
def verify_solution(completion, ground_truth):
    return 1.0 if completion.strip() == ground_truth.strip() else 0.0

TRL / GRPO Integrations (GRPOSpy)

from ratctl.integrations import GRPOSpy
from trl import GRPOTrainer

spy = GRPOSpy(reward_funcs=[accuracy_reward, format_reward])
trainer = GRPOTrainer(
    model=model,
    reward_funcs=spy.wrapped_reward_funcs,
    ...
)

CLI Trajectory Inspection

# 1. Stream live verifier trajectory logs
ratctl show logs/run.jsonl

# 2. Display aggregate ceiling & anomaly statistics
ratctl summary logs/run.jsonl

🔄 GitHub Actions CI/CD Integration

Fail-close your CI pipeline when a verifier exceeds a gameability score:

name: RL Verifier Security Gate
on: [push, pull_request]

jobs:
  audit-verifier:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Run ratctl Auditor
        uses: FreakyAdy/Reward-Hackability-Auditor--CLI---Claude-Skill-@main
        with:
          path: "."
          fail-on: "gameability>0.3"
          format: "text"

Sample $GITHUB_STEP_SUMMARY Output

When triggered in GitHub Actions, ratctl renders rich Markdown summaries directly into the Action job log:

### 🐀 RATCTL AUDIT SUMMARY
* **Status**: 🔴 FAILED (Gameability Score: 85/100)
* **Format Detected**: `openenv` (99% confidence)
* **Total Findings**: 4 (3 Critical, 1 High)

| Exploit Class | Severity | Finding | Location |
| :--- | :---: | :--- | :--- |
| `TEST_TAMPERING` | 🔴 Critical | Deleting test file | `server/app.py:18` |
| `GRADER_MANIPULATION` | 🔴 Critical | Stack frame introspection | `server/environment.py:42` |
| `PREMATURE_TERMINATION` | 🔴 Critical | `sys.exit(0)` early exit | `verifier.py:28` |
| `ENV_HIJACKING` | 🟠 High | Git history leak | `verifier.py:65` |

⚖️ Related Tools & Ecosystem

ratctl complements existing tools in the RL safety and monitoring ecosystem:

ratctl and rewardspy address complementary lifecycle stages: use ratctl audit pre-deployment to fix verifier security bugs, and ratctl watch or rewardspy in-training to track reward dynamics.

🤝 Contributing & Community

ratctl is an open-source community effort. We welcome custom detectors, bug reports, and research extensions!

• CONTRIBUTING.md: Setup guide, custom detector tutorial, and PR guidelines.
• CHANGELOG.md: Version history and feature updates.
• CODE_OF_CONDUCT.md: Community health standards.
• docs/detectors.md: Mathematical specifications of the detection engine.
• docs/hack_patterns.md: Code gallery of exploit patterns with before/after fixes.

📄 License

MIT License — see LICENSE for details.

Distributed under the MIT License.