Skill-Driven Extraction

A production-grade pipeline for extracting structured data from unstructured oil & gas reports (10-Ks, earnings PPTs, press releases, AIFs) into a 39-column analyst-ready schema — powered by a versioned rule registry, blocking pre-flight checks, and a two-layer eval loop.

What this repo is really about: a case study in getting an LLM-based extraction pipeline from "usually works" to demonstrably stable. The domain (upstream O&G filings) is the harness; the interesting part is the engineering pattern.

The problem this solves

Financial and reserves data in oil & gas filings is scattered across dozens of tables, footnotes, charts, and narrative paragraphs. Human analysts spend hours per report copying numbers into a fixed 39-column schema so the data becomes queryable. A naive LLM pass looks great on the first PDF and quietly regresses on the fifth, because:

1. Rules drift — corrections made in one session don't reach the next.
2. Context skips — critical registries get dropped from long-context runs and nobody notices until QC.
3. No eval loop — "the output looks right" isn't a test.

This pipeline addresses all three with an architectural pattern, not a bigger model.

The pattern (the "AI skills" part)

1. Separation of what, how, and which

The SOP describes the human process. The SKILL describes the pipeline. The rules describe the enforceable corrections learned across runs. No file restates another. Rules are edited only in rules.yaml; rules.md is regenerated and is the file the model reads at extraction time.

2. Rule lifecycle: candidate → confirmed → locked

Every rule earns enforcement. A new pattern lands as candidate (do not apply — just observe). After it holds for a second run it becomes confirmed (flag deviations). After a third run it becomes locked (auto-fix by validator). This prevents overfitting a single report's quirks into the global registry.

See docs/rules-evolution/ for archived snapshots of rules.yaml showing how the registry grew.

3. Blocking STOP gate

Every run begins with a required-reading checklist. Extraction cannot start until four files are confirmed loaded: SOP, SKILL, rules.md, and the memory index. This gate exists because it was skipped once and cost 20 percentage points of accuracy on the same PDF.

4. Two-layer eval loop

• scripts/validate_output.py — enforces LOCKED rules against the output XLSX. Auto-fixes what it can, flags what it can't.
• scripts/audit_coverage.py — the miss-catcher: re-scans the source PDF and flags numbers the extraction skipped. Every flag must be resolved individually. Bulk-dismissal is forbidden (that failure mode caused 28 misses on one run).
• docs/QC_prompt.md — a manual re-audit prompt used as a third-layer safety net.

Results

Full write-up: docs/results.md. The regression story itself is the single most useful artifact in this repo — it's a worked example of how eval loops earn their keep.

Repo tour

skill-driven-extraction/
├── README.md                    ← you are here
├── ARCHITECTURE.md              ← deeper dive on the pattern
├── docs/
│   ├── diagram.svg              ← architecture picture
│   ├── SOP.md                   ← the domain SOP (what to extract)
│   ├── walkthrough.md           ← end-to-end on a public 10-K
│   ├── results.md               ← accuracy numbers + the regression story
│   ├── QC_prompt.md             ← manual re-audit prompt
│   ├── engineering-notes/       ← anonymized doctrine memories (the "why" behind every rule)
│   └── rules-evolution/         ← archived rules.yaml diffs
├── skill/
│   ├── SKILL.md                 ← pipeline definition + STOP gate
│   ├── rules/
│   │   ├── rules.yaml           ← source of truth
│   │   ├── rules.md             ← generated, loaded by the model
│   │   └── history.yaml         ← version log
│   ├── references/              ← domain reference docs
│   └── examples/                ← few-shot extraction samples
├── scripts/                     ← validator, auditor, transforms
├── tests/                       ← regression harness + scoring
├── templates/                   ← blank 39-column skeleton + parameter window
└── examples/public_10k_sample/  ← runnable example on an SEC-EDGAR filing

Try it

The pipeline is designed to be invoked as a Claude skill. To run it against your own PDF:

# 1. Drop a US or Canadian upstream O&G report PDF into examples/
# 2. Point Claude at skill/SKILL.md and let the STOP gate run
# 3. Validate the output
python3 scripts/validate_output.py \
    --xlsx examples/public_10k_sample/output.xlsx \
    --report-type "AR" \
    --pdf-pages 120 \
    --country USA \
    --auto-fix

# 4. Audit coverage
python3 scripts/audit_coverage.py \
    --pdf examples/public_10k_sample/source.pdf \
    --xlsx examples/public_10k_sample/output.xlsx

Full walkthrough on a public Devon Energy 10-K: docs/walkthrough.md.

Why this matters beyond O&G

The three-layer separation, the rule lifecycle, the STOP gate, and the two-layer eval loop generalize to any high-accuracy structured-extraction task where:

• The output schema is fixed and non-negotiable
• Ground truth exists (or can be produced by an expert)
• The cost of a silent miss > the cost of an extra flag

Insurance claim processing, medical coding, contract abstracting, and regulatory filing intake all fit the shape.

License

Apache 2.0 — see LICENSE.

Acknowledgements

Built iteratively across ~40 real-world extraction runs. Every rule in rules.yaml traces back to a specific miss on a specific report. Every engineering note in docs/engineering-notes/ is scrubbed doctrine from those runs. Client and operator names have been anonymized.