watch2 — Video Analysis for AI Agents

It watches. It listens. It verifies. Rust-powered video analysis skill for Hermes Agent — transcript-first with scene detection. Cross-references frames against captions to catch what either one misses.

Works with: Hermes Agent · Claude Code · Codex · Any AI agent that reads files

Paste a URL or a local path. Hermes fetches captions, downloads the video, detects scene changes, extracts frames at key moments, and cross-references the transcript against what's actually on screen. Auto-captions misspell names? It catches that. A claim doesn't match the visual? It flags that.

Zero config to start. yt-dlp, ffmpeg, and av-scenechange are the only runtime dependencies. Captions cover most public videos for free. Whisper API key is only needed when a video has no captions.

Quick Install

Hermes Agent (recommended):

hermes skill install watch2

From source:

git clone https://github.com/m1crodevil/hermes-video-rs && cd hermes-video-rs
cargo build --release
sudo cp target/release/watch2 /usr/local/bin/

Runtime dependencies: yt-dlp, ffmpeg, ffprobe, av-scenechange

What People Use It For

Analyze someone else's content.

watch2 https://youtu.be/ what hook did they open with?

Diagnose a bug from a video.

watch2 bug-repro.mov what's going wrong?

Summarize a video.

watch2 https://youtu.be/ summarize this

Catch what captions get wrong.

watch2 https://youtu.be/ are any names or terms misspelled in the captions?

How It Works

watch2 runs a single linear pipeline — no mode branching, no configuration trees:

Video URL / local path
    ↓
1. Detect language via yt-dlp metadata (quick, no download)
    ↓
2. Download video (720p) + targeted subtitles (JSON3) via yt-dlp
    ↓
3. Parse transcript from best-matching subtitle file
    ↓
4. Whisper fallback (if no captions and API key available)
    ↓
5. Scene detection via av-scenechange
    ↓
6. Agent reads report.json → selects key moments via tiered LLM priority (hook-first, impact-scored) → extracts frames
    ↓
7. Cleanup video file (save disk space)
    ↓
8. Build WatchReport (markdown/JSON)

The binary handles data extraction only. All intelligence (LLM calls, moment selection, analysis) is handled by the agent. No moments_prompt.txt, no key_moments.json — the agent reads report.json and decides what to analyze.

Cross-Reference Methodology

Every vision finding is classified:

• ✅ confirmed — vision matches transcript
• 🔧 corrected — vision shows different spelling/entity
• ❓ fabrication — claim has no visual evidence
• ⚠️ unverified — cannot determine from visual alone
• 🔸 partial — partially shown on screen

Skill Architecture

The skill uses progressive disclosure — the agent loads only what it needs:

Tier 1: skill_view('watch2') → 216-line core (~800 tokens)
Tier 2: skill_view('watch2', file_path) → reference files on-demand

Core (SKILL.md): Quick reference, output format, CLI options, configuration References (references/): Detailed workflows, pitfalls, visual verification rules (16 files)

This keeps token cost minimal (~800 tokens per invocation vs ~12K before) while providing full context when needed. Based on Hermes Agent's 4-tier progressive disclosure model.

Key Features

CLI Reference

# Basic usage
watch2 https://youtu.be/dQw4w9WgXcQ what happens at the 30 second mark?
watch2 https://www.tiktok.com/@user/video/123 summarize this
watch2 ~/Movies/screen-recording.mp4 when does the UI break?
watch2 https://vimeo.com/123 what tools does she mention?

Output Formats

The WatchReport includes: video metadata, extracted frames with timestamps, full transcript with word-level timing (when available from JSON3 captions), scene boundaries from av-scenechange, key moment metadata (LLM-selected moments with reasons), and warnings for sparse coverage or missing transcript.

API Keys & Configuration

Captions cover the majority of public videos for free. The Whisper fallback only kicks in when a video has no caption track.

Config file: ~/.config/watch/.env

GROQ_API_KEY=gsk_...        # Optional — for Whisper fallback
OPENAI_API_KEY=sk-...        # Optional — alternative Whisper provider
SETUP_COMPLETE=true

Architecture

watch2/
├── skill/watch2/
│   ├── SKILL.md              # Core skill (216 lines, ~800 tokens)
│   └── references/           # On-demand reference files (16 files)
│       ├── agent-workflow.md # Full 8-step workflow
│       ├── pitfalls.md       # Debugging reference
│       ├── moment-selection.md   # Tiered priority + impact scoring
│       ├── transcript-moments-pipeline.md
│       ├── visual-verification.md
│       ├── reading-report.md
│       ├── scene-detection.md
│       └── ...
├── src/
│   ├── main.rs             # Entry point — CLI, cache init, pipeline run
│   ├── cli.rs              # clap CLI definition
│   ├── config.rs           # Config loading (.env)
│   ├── pipeline.rs         # Linear pipeline
│   ├── download.rs         # yt-dlp wrapper with retry + caching
│   ├── transcript.rs       # JSON3/VTT subtitle parser
│   ├── frames/
│   │   ├── mod.rs          # auto-fps, scale filter, FrameMeta
│   │   ├── metadata.rs     # ffprobe video metadata
│   │   └── timestamp.rs    # Frame extraction at timestamps
│   ├── scene_detect.rs     # av-scenechange integration
│   ├── output.rs           # WatchReport structs + output
│   ├── cache.rs            # Download cache (SHA256)
│   └── whisper.rs          # Groq/OpenAI Whisper API
└── tests/                  # Integration tests

Why Rust?

Development

# Run all tests
cargo test

# Run specific test suites
cargo test test_cli
cargo test test_transcript
cargo test test_output
cargo test test_frames

# Build release
cargo build --release

# Install
sudo cp target/release/watch2 /usr/local/bin/

# Run with verbose output
RUST_LOG=debug cargo run -- --help

# Check installed version
watch2 --version

Versioning & Releases

• Source of truth: the version field in Cargo.toml (SemVer 2.0.0).
• Tags: annotated git tags vX.Y.Z point to the released commit (e.g. v8.1.0).
• Releases: GitHub Releases mirror the tags, with notes generated from CHANGELOG.md.
• Changelog: maintained with git-cliff (git-cliff --output CHANGELOG.md) from conventional commits.
• Automation: cargo-release with release.toml (publish disabled — binary only). Bump + tag + push:

cargo install cargo-release
cargo release patch --execute   # or minor / major

Check the current version: watch2 --version

Related Projects

Note: This is a Rust rewrite of hermes-video — same features, 100× faster startup, single binary.

• bradautomates/claude-video — Original inspiration (7.6k stars)
• m1crodevil/hermes-video — Python version (same features, slower startup)

License

MIT. Built on yt-dlp, ffmpeg, av-scenechange. Whisper transcription via Groq or OpenAI.

Original: bradautomates/claude-video