CERES: Coursework Evidence Review & Evaluation System

English | 简体中文

Evidence-first review for coursework, code projects, reports, and project showcases.

CERES turns teacher-provided requirements, rubrics, starter templates, and tests into consistent reviews of student code, experiment results, reports, and presentation artifacts. Three isolated roles interpret the assignment, score each criterion, and audit the result. Deterministic scripts enforce archive safety, score arithmetic, output formats, and resumable batch state.

CERES is particularly useful for AI coursework involving memory, tool calling, ReAct, RAG, planning, and multi-agent systems, but its rubric-driven workflow also applies to general software projects.

Why CERES

Quick Start

1. Install the Skill

git clone https://github.com/sanwu-maizi/CERES.git
cd CERES

SKILL_HOME="${CODEX_HOME:-$HOME/.codex}/skills/ceres-review"
mkdir -p "$SKILL_HOME"
rsync -a SKILL.md agents assets references scripts evals "$SKILL_HOME/"

Restart Codex after installing the Skill so it can discover $ceres-review.

2. Create a Grading Configuration

cp assets/grading-config.example.json grading-config.json
python scripts/validate_inputs.py --config grading-config.json

Define the assignment requirements, allowed student changes, rubric criteria, required artifacts, evaluation scope, and any teacher-controlled tests. For artifact-only review, use:

{
  "evaluation": {
    "scope": "static",
    "course_grade_authoritative": false
  },
  "runtime": {
    "backend": "none"
  }
}

3. Use the Recommended Prompt

Replace the three placeholders with local directories.

Use the $ceres-review Skill from <CERES_SKILL_PATH> to review every submission
in <SUBMISSIONS_DIR> against the assignment requirements, rubric, and teacher
materials in <TEACHER_MATERIALS_DIR>. Use separate, isolated subagents for the
Assignment Analyst, Evidence Grader, and Independent Auditor roles, and inspect all
scoring-relevant code, PDFs, images, results, reports, and presentation materials.
Unless I explicitly authorize execution, use static scope. Score only the supplied
rubric. Treat security and API findings as redacted diagnostics, not deductions,
unless the rubric explicitly assigns points to them.

4. Prepare a Submission

python scripts/safe_extract.py \
  --archive submission.zip \
  --output-dir submission \
  --out archive-manifest.json

python scripts/grade_pipeline.py \
  --phase prepare \
  --config grading-config.json \
  --submission-root submission \
  --work-dir grading-work

safe_extract.py accepts both ZIP and RAR archives. In static scope, the pipeline writes a not_run record and does not import or execute student code.

5. Assemble the Reviewed Result

After the scoring and audit roles produce rubric-scores.json and audit.json, run:

python scripts/grade_pipeline.py \
  --phase assemble \
  --config grading-config.json \
  --work-dir grading-work \
  --scores rubric-scores.json \
  --audit audit.json \
  --final-dir final

The final directory contains:

final/final-report.md
final/final-report.json
final/grade-summary.csv

How It Works

The parent agent coordinates the workflow, while three isolated subagents perform the semantic review. The subagents exchange structured files rather than conversation history. Deterministic scripts handle safety checks, state, arithmetic, and aggregation; they do not replace subagent reading or assign semantic scores. This keeps each decision reviewable and ensures that evidence in PDFs, images, screenshots, and diagrams is not silently ignored.

Teacher materials ──> Analyst subagent ──> Frozen review instructions
                                                   │
Student submission ──> Safety checks ──> Grader subagent ──> Criterion scores
                                                   │
                                                   v
                                         Auditor subagent ──> Final reports

Evidence and Feedback Model

Every criterion must record:

1. the teacher's requirement;
2. the code, result, report, or test evidence that was observed;
3. the exact difference between the evidence and the requirement;
4. each point deduction and its supporting reference; and
5. the change or evidence that would restore the points.

For cohort exports, CERES creates adjacent columns for each criterion:

A non-full criterion without a concrete deduction basis is routed to human_review instead of being published as a final result.

Evaluation Scopes

Only formal scope can be configured as course-grade authoritative. Every report states what CERES did and did not verify.

Safety and Failure Handling

• CERES rejects path traversal, links, duplicate members, Unicode or case collisions, and configured archive limits before extraction.
• RAR support fails closed when a reliable backend is unavailable. The submission becomes not_assessed and enters the human-review queue; it does not receive a numeric zero.
• content_exclude_globs can exclude large, irrelevant data directories. Excluded payloads are not extracted, hashed, scanned, or sent to a grading role.
• Likely credentials are replaced with [REDACTED]. The report records only the detector, file, and line, and recommends revocation.
• Security, credential, API, and network findings are advisory diagnostics by default. They may trigger human review or remediation guidance, but they do not change the numeric score unless the supplied rubric explicitly assigns points to that issue.
• Environment or dependency failures remain separate from student-caused failures until a teacher-controlled reproduction proves otherwise.
• Teacher solutions, hidden tests, and other students' files remain outside the submission runtime and grading context.

Batch Review

The scheduler keeps each submission and role in a separate directory. It writes complete state updates and records which worker owns each task, preventing duplicate grading and preventing an auditor from starting before scoring finishes.

discovered -> prepared -> agent2_ready -> agent2_done
           -> agent3_ready -> assembled_ready -> assembled

Failed preparation, timeouts, and human-review decisions are recorded separately. Completed reports can be aggregated with:

python scripts/aggregate_reports.py \
  --reports submissions/*/final/final-report.json \
  --csv cohort-grades.csv \
  --jsonl cohort-grades.jsonl

Use scripts/batch_grade.py requeue when corrected evidence or policy requires a new review. CERES archives the prior attempt instead of overwriting it.

Validation and Trust

Every push and pull request runs the Validate CERES workflow with Python 3.12. The workflow verifies:

• the required Skill name and description;
• deterministic safety, static-scope, scoring, CSV, batch-state, worker-recovery, RAR-failure, and redaction fixtures;
• Waza Eval and Task schema conformance; and
• generation of a downloadable validation manifest.

Run the same checks locally:

python -m pip install -r requirements-dev.txt
python scripts/self_test.py
python scripts/validate_waza_evals.py --eval evals/eval.yaml

Passing these checks proves the deterministic behaviors covered by the fixtures. It does not prove that a model-generated grade is correct; the independent audit and teacher review remain part of the grading process.

Repository Layout

.
├── SKILL.md                     # Agent instructions and safety boundaries
├── agents/openai.yaml           # Skill display metadata and default prompt
├── assets/                      # Example configs and report templates
├── references/                  # Detailed input, scoring, and runtime guidance
├── scripts/                     # Deterministic intake, scoring, and batch tools
├── evals/                       # Waza-compatible evaluation fixtures
├── docs/diagrams/               # Workflow diagram assets
└── .github/workflows/           # Continuous validation

Limitations

• CERES cannot resolve a missing or ambiguous rubric, teacher template, or report requirement without teacher input.
• Static review cannot prove that external APIs, models, databases, or stochastic behavior will reproduce in the teacher's environment.
• RAR extraction depends on a compatible local backend; unavailable support triggers human review.
• Cohort ranking criteria remain provisional until cohort data is available.
• A teacher remains responsible for approving course grades and any policy-specific exceptions.

Standards and Acknowledgements

CERES follows the Agent Skills folder format. Its evaluation fixtures use schema and validation concepts from Microsoft Waza. Bundled Waza license and provenance records are stored in references/.

CERES is released under the MIT License. Third-party files remain under their original licenses.