Celes Renata
88ad1e8d99
- Add scheduler and ingestion unit tests (test_scheduler_unit.py, test_ingestion_unit.py) - Add all 13 app services + dashboard to docker-compose.yml - Add full documentation suite: API reference, Helm reference, Docker deployment guide, 3 architecture diagrams (K8s, Docker Compose, data pipeline), AI agent guide, backup/restore guide, observability/metrics reference, per-service docs - Add intelligence pipeline deep-dive docs with Mermaid diagrams - Update README with documentation index and links - Add specs for comprehensive-quality-docs, intelligence-pipeline-deep-dive, sanitized-pipeline-docs
3.9 KiB
Intelligence Pipeline Deep Dive
This document series provides a narrative walkthrough of the full intelligence-to-decision pipeline in the platform. Unlike the existing service reference and API documentation, these pages tell the story of how raw data enters the system, gets processed by AI agents, produces structured signals, accumulates into trend summaries, and ultimately drives autonomous decision execution.
Each page covers one stage of the pipeline and ends with a transition to the next, so you can read the series end-to-end or jump directly to the stage you need. Diagrams are stored as standalone Mermaid files that can be rendered independently or embedded in other documents.
Table of Contents
- Data Ingestion and Preparation — How raw data from an external data provider, a public records API, and macro news APIs enters the system, gets deduplicated, stored, parsed, and routed for AI processing.
- AI Agent Processing and Structured Extraction — How the Document Intelligence Extractor and Global Event Classifier agents use LLM inference to produce structured JSON intelligence from documents.
- Signal Scoring and the WeightedSignal Abstraction — How raw extraction output is transformed into weighted signals through confidence gating, recency decay, source credibility, novelty bonuses, and environmental context multipliers.
- Trend Aggregation and Accumulating Signals — How the aggregation engine merges weighted signals across five time windows, detects contradictions, ranks evidence, and escalates trend strength as consecutive signals accumulate.
- Recommendation Generation — How trend summaries pass through data quality suppression, eligibility evaluation, commitment sizing, thesis generation, and risk classification to produce actionable recommendations.
- Decision Execution — How the decision execution engine polls recommendations, runs pre-execution checks, sizes commitments, enforces circuit breakers, and submits execution requests through the execution adapter.
Diagrams
The following Mermaid diagram files can be rendered independently or referenced from the narrative pages:
- Ingestion to Extraction Flow — Flowchart from Scheduler through Ingestion, Parser, to Extractor with all queues and storage.
- Three-Layer Signal Merging — Entity-specific, Environmental, and Relational signal layers converging into the Aggregation engine.
- Weighted Signal Computation — Component breakdown of the composite weight formula.
- Trend Accumulation and Escalation — How consecutive signals strengthen trends and escalate actions across time windows.
- Recommendation Generation Flow — From TrendSummary through suppression, eligibility, thesis, risk classification, to persistence.
- Decision Engine Loop — Pre-execution check sequence, commitment sizing, and execution request submission flow.
Related Documentation
For reference-level detail on individual services, AI agent configuration, and infrastructure, see the existing documentation:
- Services Reference — Per-service configuration, database tables, queues, and runtime behaviors.
- AI Agents Guide — AI agent configuration, variants, A/B testing, and the agent management API.
- Data Pipeline Architecture — Queue topology, data store summary, and Mermaid flow diagrams for the full data pipeline.