LIA Assistant

Table of Contents

• Why LIA?
• Try LIA Online
• Built by an AI, Directed by a Human
• Screenshots
• Features
• Administration & Monitoring
• Quick Start
• Architecture
• Technologies
• Documentation
• Tests
• CI/CD
• Performance
• Security
• Contributing
• Support
• License
• Acknowledgments

Why LIA?

LIA solves the fundamental problems of today's AI assistants:

Primary Use Cases

📅 "Find my meetings for tomorrow and send a reminder to all participants"
📧 "Summarize my unread emails from this week that have attachments"
👥 "Update the companies of my contacts who work at startups"
🔔 "Remind me tomorrow at 9am to call Marie for her birthday"

Try LIA Online

LIA is available as a hosted service at https://lia.jeyswork.com/ — no installation required.

Closed beta: Access is currently limited to a restricted number of users, at the administrator's discretion. To request an invitation, contact [email protected].

Built by an AI, Directed by a Human

"Speed comes from the AI. Quality comes from the framework."

Nearly 100% of this codebase was written by an AI, under human direction: a written engineering rulebook, blocking automated checks, systematic review, adversarial audits. The result is measured, not proclaimed:

• The full story — method, trade-offs, results and what remains to be done, weaknesses included: lia.jeyswork.com/story
• The audit itself — 24 normalized areas mapped to ISO/IEC 25010:2023, every score backed by executed evidence, 7 open worksites included, with the protocol and the full standalone report: docs/audit/

Screenshots

Features

Multi-Agent Intelligence (LangGraph 1.x)

• 20+ Specialized Agents: Contacts, Emails, Calendar, Drive, Tasks, Reminders, Places, Routes, Weather, Wikipedia, Perplexity, Brave, Web Search, Web Fetch, Browser Control (with progressive screenshot streaming), Smart Home (Philips Hue), Context, Query + dynamic MCP agents
• ReAct Execution Mode (ADR-070): Alternative to the pipeline — the LLM iteratively reasons about tool outputs and decides next steps autonomously. User-toggleable preference, 4-node LangGraph architecture with native HITL support, timeout enforcement, cross-domain initiative via prompt engineering. Supports all tools including MCP and Skills
• MCP (Model Context Protocol): Per-user external tool servers with OAuth 2.1, SSRF protection, structured items parsing, MCP Apps (interactive iframe widgets), Iterative Mode (ReAct) for complex servers — a dedicated agent reads docs then calls tools correctly
• Agent Initiative Phase: Post-execution cross-domain enrichment — the assistant proactively verifies related information (e.g., weather shows rain → checks calendar for outdoor events). Prompt-driven, read-only, fully configurable
• Skills (agentskills.io) with Rich Outputs: Open standard for expert instructions (SKILL.md), model-driven activation, progressive disclosure (L1/L2/L3), sandboxed scripts, marketplace import, auto-translated multi-language descriptions, ZIP download, admin management. Rich Skill Outputs (v1.16.8): skills can return interactive HTML frames (iframe srcDoc or external URL) and/or images in addition to text, via a simple JSON contract (SkillScriptOutput). Automatic theme & locale sync (theme switch propagates live to frames via postMessage), iframe auto-resize, CSP-sandboxed client-side interactivity (addEventListener, crypto.getRandomValues), bundled segno for QR codes. Seven built-in rich skills: interactive-map, weather-dashboard, calendar-month, qr-code, pomodoro-timer, unit-converter, dice-roller. Planner skill guard: multi-domain deterministic skills are protected from false-positive early clarification requests via domain overlap detection (_has_potential_skill_match). Built-in Skill Generator: create custom skills in natural language — the assistant guides you through need analysis and archetype selection (the dialogue keeps its context across turns), then validates and installs the finished skill directly into My Skills, announced by name and immediately usable. Every import path (chat-generated or manual upload) goes through one hardened pipeline: strict name validation, zip-expansion caps, name-conflict rejection, atomic install with automatic rollback
• Agentic Telephony (ADR-127): LIA places real outbound phone calls on your behalf via your own per-user ElevenLabs + Twilio connector (BYO — zero cost on LIA's side). Every call is HITL-confirmed before dialing; the goal-driven voice agent greets the instant the line opens, resolves relative dates against a live temporal anchor, and hangs up when done. Privacy by capability: the call agent can only read free/busy availability — never event titles or contents; no recording, no stored transcript. A strict mandate boundary forbids any expense or commitment beyond the objective (offers are captured with their price and deferred to you), and the asynchronous post-call summary must state every cost and flag every open point. Config self-heals: fingerprint-based lazy re-sync of the vendor agent, self-healing one-active-call guard (vendor status probe, deleted-conversation 404 handling), pinned thinking-free agent LLM, telephony-native ulaw_8000 audio
• AI Image Generation & Editing: Generate images from text prompts (gpt-image-1), edit existing images with natural language instructions. Multi-provider factory architecture, per-user quality/size preferences, cost tracking with DB-cached pricing, attachment-based storage with cascade cleanup
• File Attachments (Images, PDF): Upload with client-side compression, configurable LLM vision analysis, PDF text extraction, strict per-user isolation
• Response Grounding on Recent Entities (ADR-147): on a turn that produces no tool data, the response model is re-grounded on the most recent entities already in state (zero I/O, age-bounded, explicitly non-authoritative) instead of paraphrasing older prose — and the prompt forbids inventing an entity attribute rather than admitting it is unknown
• Semantic Routing: Binary classification with confidence scoring (high >0.85, medium >0.65)
• Multi-Step Planning: ExecutionPlan DSL with dependencies and conditions
• Parallel Execution: asyncio.gather for independent domains
• Intelligent Context Compaction: LLM-based conversation history summarization when token count exceeds dynamic threshold (ratio of response model context window). Preserves identifiers (UUIDs, URLs, emails). /resume command for manual trigger. 4 HITL safety conditions prevent compaction during active approval flows
• Scroll-up History Pagination: GET /conversations/me/messages exposes a keyset cursor (?before=<created_at>) with has_more / next_cursor. The chat UI binds an IntersectionObserver on a top sentinel — older pages prepend with id-based dedup, scroll position preserved via a shared wasPrependRef that skips the auto-scroll-to-bottom for that cycle. Conversations of any length stay fully reachable; the existing (conversation_id, created_at DESC) composite index makes each page an index-only seek. Bounds env-tunable (CONVERSATION_HISTORY_DEFAULT_LIMIT / _MAX_LIMIT)

Psyche Engine — Dynamic Emotional Intelligence

• 5-Layer Psychological State: Big Five personality traits (permanent) → PAD mood space with 14 moods (hours) → 22 discrete emotions with cross-suppression (minutes) → 4-stage relationship progression (weeks) → curiosity/engagement drives (per-session)
• Show, Don't Tell: Mood and emotions subtly influence word choice, sentence rhythm, energy level, and relational tone — the assistant never declares "I'm feeling happy"
• Emotional Avatar: Mood-responsive emoji with colored ring on each message. Historical avatars persisted per-message for reload consistency
• Evolution Awareness: The assistant knows how its mood shifted since the last message, providing narrative continuity
• 4-Chart Dashboard: Interactive recharts visualization of mood (PAD), emotions, relationship, and drives over time (24h to 90 days)
• Education Guide: 7-section interactive documentation explaining every layer, with descriptive tables for 14 moods and 22 emotions
• Customizable Temperament: Expressiveness (stoic → highly expressive) and stability (volatile → very stable) sliders. Soft reset (mood only) and full reset (everything) with explicit scope descriptions
• Global Injection: Behavioral directives injected via template variables into all user-facing text generation (response, notifications, reminders, voice) within semantic XML blocks (<InnerState purpose="tone-calibration">)
• Safety Guardrail: Explicit instruction prevents the LLM from projecting its own emotional state onto the user
• Self-Report: Zero-cost emotion tracking via hidden <psyche_eval/> tag — no additional LLM call

Voice: Input & Output

Voice Input (STT)

• Push-to-Talk: Hold microphone button to speak, release to transcribe. Optimized for mobile (anti-long-press CSS, touch gesture handling)
• Wake Word: Say "OK Guy" to activate hands-free recording. Sherpa-onnx WASM (Whisper Tiny.en) runs entirely in-browser — no audio sent externally for wake word detection
• Per-User Language: STT transcription uses the user's preferred language setting (Whisper Small, 99+ languages, fully offline)
• Latency Optimized: Mic stream reuse, WebSocket pre-warming, parallel setup, cached AudioWorklet (~50-100ms wake-to-record)

Voice Output (TTS)

• Catalogue-driven (ADR-081): provider/model/voice are admin-controlled via Configuration LLM (LLM type voice_tts). Voice + tuning live in provider_config JSONB. No env vars to maintain across deployments.
• Sentence streaming (ADR-082): TTS runs sentence-by-sentence pipelined with the LLM stream. First audio lands in ~1 s on chat mode (was ~5 s).
• Per-message cost transparency: 🔊 N chars · €X.XXX badge on the assistant bubble (paid providers only — Edge stays badge-free as it's $0).
• Graceful degradation: missing API key on a paid provider transparently falls back to Edge with a structured warning log.
• Persistent HTTP pool on ElevenLabs: keep-alive across sentences saves ~100–300 ms TLS handshake per call.

FOR_EACH Iteration Pattern

# DSL Syntax
ExecutionStep(
    tool_name="send_email",
    for_each="$steps.get_contacts.contacts",
    for_each_max=10
)

• Per-parent correlation identity: an enrichment fetched for one iteration belongs to that iteration — results keyed on content alone (weather → place + day) used to collide between two parents sharing them, silently costing one of them its data; ids are now derived per parent and stay stable across replays and resumed checkpoints
• HITL Thresholds: Mutations >= 1 trigger mandatory approval
• Bulk Operations: Send emails, update contacts, mass deletions

Smart Services (Token Savings 89%)

Google Integrations (OAuth 2.1 + PKCE)

• Gmail: Search, read, send, reply, trash
• Contacts: Fuzzy search, list, details (14+ schemas)
• Calendar: Search, create, update events
• Drive: Search, file/folder listing
• Tasks: Full CRUD with completion

Apple iCloud Integrations

• Apple Mail: Search, read, send, reply, forward, trash (IMAP/SMTP)
• Apple Calendar: Search, create, update, delete events (CalDAV)
• Apple Contacts: Search, list, create, update, delete (CardDAV)

Microsoft 365 Integrations (OAuth 2.0 + PKCE)

• Outlook: Search, read, send, reply, forward, trash (Graph API)
• Calendar: Search, create, update, delete events (calendarView)
• Contacts: Search, list, create, update, delete
• To Do: Full CRUD with completion (task lists + tasks)
• Multi-tenant: Personal accounts (outlook.com) and business accounts (Azure AD) via tenant=common

3-Way Mutual Exclusivity

• Only one provider per functional category (email, calendar, contacts, tasks)
• 3 supported providers: Google, Apple, Microsoft
• Activating a new provider automatically deactivates the active competitor

Smart Home — Philips Hue

• Voice-controlled lighting: Turn lights on/off, adjust brightness and colors via natural language
• Room & scene management: Control entire rooms or activate predefined scenes ("dim the living room", "activate movie mode")
• Local or cloud connection: Connect via local bridge IP or Philips Hue cloud API
• Feature flag: PHILIPS_HUE_ENABLED=true to enable

Human-in-the-Loop (HITL)

Note: the plan-approval level is currently auto-approved — tool-level HITL supersedes it (see ADR-106); the other five levels interrupt execution and wait for the user.

Enterprise Observability

• Prometheus: 425 custom metrics (agents, LLM, infrastructure)
• Grafana: 22 production-ready dashboards
• Langfuse: LLM-specific tracing with prompt versions
• Loki: Structured JSON logs with PII filtering
• Tempo: Distributed cross-service tracing
• Probes: liveness (GET /health, always 200 while the process serves — what Docker healthchecks poll) split from readiness (GET /ready, 503 unless PostgreSQL and Redis answer) — ADR-115
• Alerting: a 14-alert vital core (service/DB/Redis down, disk, container OOM, 5xx rate, SSE latency, backup failure, public-endpoint & TLS-certificate probes, chain self-monitoring) evaluated by Prometheus and emailed by a dedicated Alertmanager — unit-tested with promtool test rules, every alert linking its runbook — ADR-119

Cost Tracking & Billing

• Google Maps Platform: Places, Routes, Geocoding, Static Maps
• Dynamic Pricing: Admin UI for full LLM catalogue CRUD — provider, 8 capability flags (max input/output tokens, tools, structured output, strict mode, streaming, vision, reasoning) and pricing per model, all stored in the database. Same surface for image generation models (provider + quality/size/pricing). Cross-worker cache invalidation via Redis Pub/Sub (ADR-063), live cross-sibling refresh in the frontend — no code change, no redeploy
• ContextVar Pattern: Implicit tracking without explicit parameter passing
• Admin CSV Exports: Token usage, Google API usage, Consumption summary (all users or filtered by user)
• User CSV Exports (v1.9.1): Personal consumption export in Settings > Features — users export their own data only (user_id forced server-side, IDOR-safe)

Security & Compliance

• OAuth 2.1: PKCE (S256), single-use state token
• BFF Pattern: HTTP-only cookies, server-side Redis sessions (fixed 7-day lifetime, 30 with remember-me)
• Strong authentication: WebAuthn passkeys (discoverable credentials, conditional UI on the login form, single-use challenges, clone detection, zero enumeration) + TOTP second factor (two-step login, matched-timestep anti-replay, 10 single-use backup codes revealed once); step-up re-authentication guards every sensitive action behind a 5-minute sudo window (typed 403 contract; password, code, passkey, or provider re-sign-in). Feature flag: MFA_ENABLED=true — ADR-143
• Device sessions: "My devices" lists every live session with deliberately bounded metadata (browser/OS families, truncated IP, coarse last-seen), per-device revocation and step-up-guarded revoke-others; a revoked session's SSE streams close within one keepalive tick; new-login push alerts with FCM device attestation — ADR-144
• Encryption: Fernet (credentials), bcrypt (passwords)
• GDPR: Automatic PII filtering, pseudonymization, and full-account export (Art. 20): durable jobs build a ZIP (JSON + readable Markdown + uploaded files) from a total data classification where secret tables are unexportable by construction. Feature flag: ACCOUNT_EXPORT_ENABLED=true — ADR-145
• Offline PWA: one unified service worker serves push and a branded 6-language offline page; /api/ is never cached — ADR-146
• Per-User Usage Limits: Token, message, and cost quotas (period/global) with 5-layer defense-in-depth enforcement, admin kill switch, real-time dashboard with WebSocket gauges. Feature flag: USAGE_LIMITS_ENABLED=true
• Backups: Automated daily PostgreSQL dumps (pg_dump sidecar, daily/weekly/monthly rotation, all .env-driven) with a tested one-command restore and a verification drill (task backup:verify) — ADR-109, runbook in docs/runbooks/DATABASE_BACKUP_RESTORE.md

MCP (Model Context Protocol)

• Per-user external servers: Each user connects their own MCP servers (third-party tools)
• Flexible authentication: None, API Key, Bearer Token, OAuth 2.1 (DCR + PKCE S256)
• Enhanced security: HTTPS-only, SSRF prevention (DNS resolution + IP blocklist), encrypted credentials (Fernet)
• Structured Items Parsing: Automatic JSON array detection into individual items with McpResultCard HTML
• Auto-generated descriptions: LLM analysis of discovered tools to generate domain descriptions optimized for intelligent routing
• Per-server rate limiting: Redis sliding window per server/tool
• Feature flag: MCP_USER_ENABLED=true to enable per-user

Multi-Channel Messaging (Telegram)

• Bidirectional Telegram: Full chat with LIA via Telegram (text, voice, HITL)
• OTP Linking: Secure account-to-Telegram linking via 6-digit OTP code (single-use, 5min TTL, brute-force protection)
• HITL Inline Keyboards: Approval/rejection buttons localized in 6 languages directly in Telegram
• Voice Transcription: Telegram voice messages to STT (Sherpa Whisper) to text processing
• Proactive Notifications: Reminders and interest alerts also sent via Telegram
• Extensible Architecture: BaseChannelSender/BaseChannelWebhookHandler abstraction for future channels (Discord, WhatsApp)
• Observability: 12 dedicated Prometheus RED metrics (latency, errors, volumes)
• Feature flag: CHANNELS_ENABLED=true to enable

Autonomous Heartbeat — Proactive Notifications

• LLM-driven proactivity: LIA takes the initiative to inform you when relevant (weather, calendar, interests)
• Multi-source aggregation: Calendar, Weather (with change detection), Tasks, Interests, Memories, Activity — parallel fetch
• 2-phase LLM decision: Phase 1 (structured output, cost-effective model) decides whether to notify, Phase 2 rewrites with user personality and language
• Intelligent anti-redundancy: Recent history + cross-type dedup (heartbeat vs. interests) in the decision prompt
• User control: Push notifications (FCM/Telegram) independently toggleable, configurable daily max (1-8), dedicated time windows (independent from interests)
• Feedback loop: thumbs up/down on every notification card, persisted on the archived message so a verdict is given once, across devices
• Weather change detection: Rain start/end, temperature drops, wind alerts — truly actionable notifications
• Feature flag: HEARTBEAT_ENABLED=true to enable

Interest Learning — Admission Doctrine

• A creation requires a named ground (ADR-166): stated passion, own practice, prior knowledge, or digging into the same subject across an exchange — and the model must quote the user's words that carry it. Asking about a subject is a task, not a taste.
• Six exclusion classes, stated as classes rather than examples: the subject of a request, a remark about the assistant, a third party's taste, something tried once, a daily action, anything the assistant introduced itself
• Deduplication reads every status: a blocked subject cannot be re-created under a neighbouring name, renamed back into place, or deleted to free the slot; a dormant one is revived instead of duplicated
• Cap on destructive actions: beyond EXTRACTION_MAX_DELETES_PER_RUN (2), the batch's deletions are dropped and counted — one replayed production window proposed 19
• Anchored confidence floor (INTEREST_EXTRACTION_MIN_CONFIDENCE, 0.75): the prompt's scale is tied to the ground it can name, so the written rule is enforceable
• Measured, not asserted: scripts/measure_extraction_selectivity.py replays a production-derived battery plus a held-out one — noise on negatives 0.50 → 0.00, recall 0.75 → 1.00, reproduced on two model providers

Scheduled Actions

• Recurring actions: Schedule repetitive actions executed automatically (send emails, checks, reminders)
• Timezone-aware: Correct timezone handling per user
• Retry logic: Automatic retries on failure with back-off
• Auto-disable: Automatic deactivation after N consecutive failures
• Multi-channel integration: Result notifications via FCM, SSE, and Telegram
• Feature flag: SCHEDULED_ACTIONS_ENABLED=true to enable

Sub-Agents (F6)

• Persistent specialized agents: Create sub-agents with custom instructions, skills, and LLM configuration
• Read-only V1: Sub-agents perform research, analysis, and synthesis — no write operations
• Template-based creation: Pre-defined templates (Research Assistant, Writing Assistant, Data Analyst)
• Invisible to user: The principal assistant orchestrates sub-agents and presents results naturally
• Token guard-rails: Per-execution budget, daily budget, auto-disable after consecutive failures
• Feature flag: SUB_AGENTS_ENABLED=true to enable (default: false)

RAG Knowledge Spaces

• Personal knowledge bases: Create spaces, upload documents in 15+ formats (PDF, DOCX, PPTX, XLSX, CSV, RTF, HTML, EPUB, and more), automatic chunking and embedding
• Google Drive folder sync: Link Google Drive folders to spaces for automatic file vectorization with incremental change detection (new, modified, deleted). Feature flag: RAG_SPACES_DRIVE_SYNC_ENABLED
• Hybrid search: Semantic similarity (pgvector cosine) + BM25 keyword matching with configurable alpha fusion
• Response enrichment: RAG context automatically injected into assistant responses when active spaces exist
• Full cost transparency: Embedding costs tracked per document and per query, visible in chat bubbles and dashboard
• System knowledge spaces: Built-in FAQ knowledge base (119+ Q/A across 17 sections) indexed from Markdown files (docs/knowledge/). is_app_help_query detection by QueryAnalyzer, RoutingDecider Rule 0 override, App Identity Prompt injection with lazy loading (zero overhead on normal queries). Auto-indexed at startup with SHA-256 hash-based staleness. Admin UI for reindex and staleness monitoring. ADR-058
• Admin reindexation: Full reindex when embedding model changes, with Redis mutual exclusion and automatic dimension ALTER. System spaces have independent reindex via admin API
• Observability: 17 Prometheus metrics (14 user + 3 system), dedicated Grafana dashboard
• Feature flags: RAG_SPACES_ENABLED=true (user spaces), RAG_SPACES_SYSTEM_ENABLED=true (system FAQ spaces)

Personal Journals (Carnets de Bord) — Stratified consciousness

• Introspective notebooks: The assistant maintains thematic journals (self-reflection, user observations, ideas & analyses, learnings) written in first person, colored by its active personality
• Four abstraction levels: Each entry carries a level — L0 raw observation, L1 operational directive (WHEN→DO BECAUSE), L2 transversal pattern, L3 portrait facet. L2/L3 are produced exclusively at consolidation through active topic clustering (ADR-079)
• Epistemic status: confidence ∈ {low, medium, high} plus evidence_count and contradiction_count counters per entry. The journal distinguishes hypotheses still in test from observations validated across many turns
• Deferred self-evaluation T → T+1: MessagesState.injected_journal_ids carries IDs across turns; the post-conversation extractor sees the previous turn's directives + the current user reaction, signals evidence_outcome="evidence" | "contradiction", and the service atomically increments the counters. Zero added LLM cost (same extractor call, enriched prompt). Anti-hallucination layer 4: LLM never writes absolute counter values.
• Dual trigger: Post-conversation extraction (fire-and-forget) + periodic consolidation (APScheduler, 4–12 h cooldown)
• Gemini dual-vector embeddings: gemini-embedding-001 (1536d) — one vector on title+content, one on search_hints keywords. Search uses LEAST(dist_content, dist_keyword) per row to bridge the assistant's introspective vocabulary and the user's vocabulary (ADR-069)
• Ambient diffusion of the user-model portrait: Consolidation produces, in the same LLM call, a portrait_full (~200 tokens) for conversation/planner and a portrait_brief (~60 tokens) diffused across 6 secondary flows (ReAct setup, interest proactive, reminder notification, voice, heartbeat, fallback sync+async). Standalone builder build_journal_user_model_block(user_id, format, flow) mirrors build_psyche_prompt_block.
• Three corrective levers on the portrait (never directly editable): edit L3 source entries, POST /journals/portrait/feedback (free text → L0 user_correction + synchronous re-consolidation), POST /journals/consolidate (manual, bypasses cooldown).
• Prompt-driven lifecycle: The assistant manages its own journals — no hardcoded auto-archival. Mandatory pairwise dedup at consolidation STEP 1, classification audit, active L1→L2 clustering at STEP 5
• Heartbeat integration: Journal entries enrich proactive notifications via dynamic second-pass query built from aggregated context. The compiled portrait brief is also injected so the notification voice is aligned with the same