feat: Intelligence Pipeline v3 — full implementation

Multi-stage evidence-grounded inference architecture replacing the monolithic 9B model extraction pipeline. CPU-first specialist services handle routine extraction while the 9B vLLM model is preserved for semantic adjudication of ambiguous cases. Key components: - Capability-aware inference gateway (OpenAI-compatible + Ollama) - Endpoint registry with DB migrations and REST API - Sentence-aware document segmenter (property tests) - Deterministic financial parsing with offset integrity - Symbol resolution with ambiguity detection - Specialist service (GLiNER2, dynamic batching, K8s deployment) - Company-specific sentiment (FinBERT, calibration) - Retrieval-based novelty and duplicate detection - Confidence calibration pipeline - Deterministic routing engine (property tests) - 9B adjudication layer with VRAM gating - Stock-specific impact model (features, labels, baseline, trained) - Pipeline orchestrator (state machine, queues, leases, feature flags) - Bounded parallelism (async workers, semaphore, load shedding) - Observability (tracing, metrics, alerts) - Compatibility adapter (v3→v2 golden mapping tests) - Shadow/canary promotion framework - Active learning and fine-tuning pipeline Test results: 1,161 tests pass, ruff lint clean. All 282 spec tasks completed.
2026-07-13 02:14:59 +00:00
parent 84634a365e
commit a72f336ad1
227 changed files with 50403 additions and 0 deletions
@@ -0,0 +1,130 @@
+"""Calibrated fast-path thresholds by document type and event type.
+
+Thresholds represent the minimum calibrated confidence required for
+fast-path acceptance. Documents/events below these thresholds are routed
+to adjudication. Thresholds are versioned and can be updated as
+calibration data improves.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from services.intelligence_pipeline_v3.routing.reasons import RouteDecision
+
+# Default confidence thresholds per document type.
+# These are initial conservative values; calibration on the Gold Corpus
+# will refine them over time.
+DEFAULT_DOCUMENT_THRESHOLDS: dict[str, float] = {
+    "article": 0.80,
+    "press_release": 0.80,
+    "filing": 0.70,
+    "transcript": 0.75,
+    "macro_event": 0.75,
+}
+
+# Default confidence thresholds per event type (override document-type defaults).
+DEFAULT_EVENT_THRESHOLDS: dict[str, float] = {
+    "earnings_beat": 0.75,
+    "earnings_miss": 0.75,
+    "guidance_change": 0.65,
+    "management_change": 0.70,
+    "merger_acquisition": 0.60,
+    "regulatory_action": 0.65,
+    "product_launch": 0.80,
+    "legal_action": 0.65,
+    "rating_change": 0.75,
+    "supply_chain": 0.70,
+}
+
+# Fallback threshold when document_type or event_type is unknown.
+DEFAULT_FALLBACK_THRESHOLD: float = 0.80
+
+
+@dataclass(frozen=True)
+class FastPathThresholds:
+    """Configuration for fast-path acceptance thresholds.
+
+    Resolution order:
+    1. Event-type-specific threshold (if event_type is provided and known).
+    2. Document-type-specific threshold.
+    3. Fallback threshold.
+
+    Higher thresholds are more conservative (more documents go to adjudication).
+    """
+
+    document_thresholds: dict[str, float] = field(
+        default_factory=lambda: dict(DEFAULT_DOCUMENT_THRESHOLDS)
+    )
+    event_thresholds: dict[str, float] = field(
+        default_factory=lambda: dict(DEFAULT_EVENT_THRESHOLDS)
+    )
+    fallback_threshold: float = DEFAULT_FALLBACK_THRESHOLD
+    version: str = "1.0.0"
+
+    def resolve_threshold(
+        self,
+        document_type: str,
+        event_type: str | None = None,
+    ) -> float:
+        """Resolve the applicable threshold for a document/event combination.
+
+        Parameters
+        ----------
+        document_type:
+            The document type (article, filing, transcript, etc.).
+        event_type:
+            Optional event type detected in the document.
+
+        Returns
+        -------
+        float
+            The minimum calibrated confidence required for fast-path acceptance.
+        """
+        # Event-type threshold takes priority when available
+        if event_type and event_type in self.event_thresholds:
+            return self.event_thresholds[event_type]
+
+        # Document-type threshold
+        if document_type in self.document_thresholds:
+            return self.document_thresholds[document_type]
+
+        # Fallback
+        return self.fallback_threshold
+
+
+def evaluate_thresholds(
+    confidence: float,
+    document_type: str,
+    event_type: str | None,
+    thresholds: FastPathThresholds,
+) -> RouteDecision:
+    """Evaluate whether calibrated confidence meets the fast-path threshold.
+
+    Parameters
+    ----------
+    confidence:
+        Calibrated confidence score (0.0 to 1.0).
+    document_type:
+        The document type being processed.
+    event_type:
+        Optional event type detected in the document.
+    thresholds:
+        Threshold configuration to use.
+
+    Returns
+    -------
+    RouteDecision
+        FAST_PATH if confidence >= threshold, ADJUDICATION otherwise.
+
+    Notes
+    -----
+    The comparison uses ``>=`` (greater-than-or-equal). A confidence value
+    exactly at the threshold is accepted on the fast path. This boundary
+    behavior is deterministic and tested by property tests.
+    """
+    threshold = thresholds.resolve_threshold(document_type, event_type)
+
+    if confidence >= threshold:
+        return RouteDecision.FAST_PATH
+    return RouteDecision.ADJUDICATION